2024-02-28T15:54:44.697 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-28T15:54:44.697 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_vshankar@teuthology --verbose --name vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589 -- /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589/orig.config.yaml 2024-02-28T15:54:44.751 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-28T15:54:44.838 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589 branch: main description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: vshankar@redhat.com first_in_suite: false job_id: '7575589' kernel: flavor: default kdb: true sha1: bd7d00fb724897dda560070d01bc6e5c80881f15 last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_vshankar@teuthology priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2226 sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJhQDYXyDZaLzE/d7Nsps3cOMWi1ygHuaOzm+e+/sACBuqlTUUchOEbsP1h+nN4ONrGGRgTxPgmmnbAC232Nco= smithi066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCcFfzZm+EyMI205iYFlk8i1qEFqGB7M/9JUZv1Pn6faQKtShms8EsSV8ZQC4+J5ZAE2rGAB2+DeVQ0bP+XIIBY= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-28_15:21:56 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-28T15:54:44.839 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-02-28T15:54:44.839 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-02-28T15:54:44.840 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-28T15:54:44.841 INFO:teuthology.task.internal:Checking packages... 2024-02-28T15:54:44.863 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1fb5d10608dccee73edd6ef8a2c7b804a097c324' 2024-02-28T15:54:44.863 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-28T15:54:44.863 INFO:teuthology.packaging:ref: None 2024-02-28T15:54:44.863 INFO:teuthology.packaging:tag: None 2024-02-28T15:54:44.864 INFO:teuthology.packaging:branch: main 2024-02-28T15:54:44.864 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T15:54:44.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-28T15:54:45.188 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1716.g4a1c26b5 2024-02-28T15:54:45.190 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-28T15:54:45.196 INFO:teuthology.task.internal:no buildpackages task found 2024-02-28T15:54:45.197 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-28T15:54:45.218 INFO:teuthology.task.internal:Saving configuration 2024-02-28T15:54:45.238 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-28T15:54:45.244 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-28T15:54:45.270 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-28 15:48:21.484039', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJhQDYXyDZaLzE/d7Nsps3cOMWi1ygHuaOzm+e+/sACBuqlTUUchOEbsP1h+nN4ONrGGRgTxPgmmnbAC232Nco='} 2024-02-28T15:54:45.296 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-28 15:48:21.485960', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCcFfzZm+EyMI205iYFlk8i1qEFqGB7M/9JUZv1Pn6faQKtShms8EsSV8ZQC4+J5ZAE2rGAB2+DeVQ0bP+XIIBY='} 2024-02-28T15:54:45.296 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-28T15:54:45.303 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-28T15:54:45.303 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-28T15:54:45.303 INFO:teuthology.run_tasks:Running task console_log... 2024-02-28T15:54:45.377 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3364424160>, signals=[15]) 2024-02-28T15:54:45.378 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-28T15:54:45.386 INFO:teuthology.task.internal:Opening connections... 2024-02-28T15:54:45.386 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T15:54:45.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T15:54:45.500 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2024-02-28T15:54:45.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T15:54:45.604 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-28T15:54:45.611 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-02-28T15:54:45.629 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-02-28T15:54:45.629 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-02-28T15:54:45.686 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-02-28T15:54:45.686 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-02-28T15:54:45.686 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-28T15:54:45.687 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-28T15:54:45.688 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-02-28T15:54:45.709 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2024-02-28T15:54:45.726 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2024-02-28T15:54:45.727 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon" 2024-02-28T15:54:45.784 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-28T15:54:45.785 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/" 2024-02-28T15:54:45.785 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-28T15:54:45.785 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-28T15:54:45.785 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-28T15:54:45.785 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2024-02-28T15:54:45.809 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-28T15:54:45.819 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-28T15:54:45.824 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-28T15:54:45.824 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-02-28T15:54:45.827 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2024-02-28T15:54:45.842 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-28T15:54:45.849 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-28T15:54:45.849 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-02-28T15:54:45.884 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2024-02-28T15:54:45.911 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-28T15:54:45.961 INFO:teuthology.run_tasks:Running task kernel... 2024-02-28T15:54:45.975 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': 'bd7d00fb724897dda560070d01bc6e5c80881f15'} 2024-02-28T15:54:45.975 INFO:teuthology.task.kernel:config {'host.a': {'flavor': 'default', 'kdb': True, 'sha1': 'bd7d00fb724897dda560070d01bc6e5c80881f15'}, 'host.b': {'flavor': 'default', 'kdb': True, 'sha1': 'bd7d00fb724897dda560070d01bc6e5c80881f15'}}, timeout 300 2024-02-28T15:54:45.975 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-28T15:54:45.976 DEBUG:teuthology.orchestra.run.smithi066:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-28T15:54:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T15:54:45.992 DEBUG:teuthology.task.kernel:sha1 for host.b is bd7d00fb724897dda560070d01bc6e5c80881f15 2024-02-28T15:54:45.992 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "bd7d00fb724897dda560070d01bc6e5c80881f15"... 2024-02-28T15:54:45.992 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-02-28T15:54:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T15:54:45.994 DEBUG:teuthology.task.kernel:sha1 for host.a is bd7d00fb724897dda560070d01bc6e5c80881f15 2024-02-28T15:54:45.994 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "bd7d00fb724897dda560070d01bc6e5c80881f15"... 2024-02-28T15:54:45.994 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-02-28T15:54:46.048 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-361.el9.x86_64 2024-02-28T15:54:46.048 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs bd7d00fb724897dda560070d01bc6e5c80881f15 2024-02-28T15:54:46.048 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-02-28T15:54:46.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bd7d00fb724897dda560070d01bc6e5c80881f15 2024-02-28T15:54:46.052 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-361.el9.x86_64 2024-02-28T15:54:46.053 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs bd7d00fb724897dda560070d01bc6e5c80881f15 2024-02-28T15:54:46.053 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-02-28T15:54:46.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bd7d00fb724897dda560070d01bc6e5c80881f15 2024-02-28T15:54:46.277 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum upgrade -y linux-firmware 2024-02-28T15:54:46.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum upgrade -y linux-firmware 2024-02-28T15:54:48.581 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 12 MB/s | 8.1 MB 00:00 2024-02-28T15:54:48.733 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 9.7 MB/s | 8.1 MB 00:00 2024-02-28T15:54:50.273 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.5 MB 00:00 2024-02-28T15:54:50.400 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.5 MB 00:00 2024-02-28T15:54:51.382 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-02-28T15:54:51.449 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-02-28T15:54:54.700 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 19 MB 00:02 2024-02-28T15:54:55.665 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 19 MB 00:03 2024-02-28T15:55:01.098 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-02-28T15:55:01.300 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-02-28T15:55:02.832 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2024-02-28T15:55:03.152 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 15 kB 00:00 2024-02-28T15:55:04.484 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-02-28T15:55:09.644 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-28T15:55:11.193 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 20 MB 00:07 2024-02-28T15:55:11.955 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T15:55:11.955 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T15:55:11.955 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T15:55:11.955 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.955 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T15:55:11.955 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.955 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T15:55:11.955 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.956 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.957 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T15:55:11.958 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.959 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.960 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.961 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.962 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.963 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T15:55:11.963 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.963 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.963 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T15:55:11.963 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T15:55:12.517 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: linux-firmware noarch 20240219-143.el9 baseos 376 M 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 1 Package 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T15:55:12.531 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 376 M 2024-02-28T15:55:12.532 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T15:55:16.335 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-28T15:55:18.660 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T15:55:18.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T15:55:18.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T15:55:19.213 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T15:55:19.225 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T15:55:19.225 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-02-28T15:55:19.225 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T15:55:19.225 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrading: 2024-02-28T15:55:19.225 INFO:teuthology.orchestra.run.smithi066.stdout: linux-firmware noarch 20240219-143.el9 baseos 376 M 2024-02-28T15:55:19.226 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T15:55:19.226 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T15:55:19.226 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T15:55:19.226 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrade 1 Package 2024-02-28T15:55:19.226 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T15:55:19.226 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 376 M 2024-02-28T15:55:19.226 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T15:55:20.626 INFO:teuthology.orchestra.run.smithi001.stdout:linux-firmware-20240219-143.el9.noarch.rpm 47 MB/s | 376 MB 00:08 2024-02-28T15:55:20.627 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T15:55:20.628 INFO:teuthology.orchestra.run.smithi001.stdout:Total 46 MB/s | 376 MB 00:08 2024-02-28T15:55:23.813 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T15:55:23.916 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T15:55:23.916 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T15:55:27.249 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T15:55:27.249 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T15:55:34.788 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T15:55:35.124 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-02-28T15:55:35.225 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-02-28T15:55:36.374 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-02-28T15:55:36.374 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-02-28T15:55:36.809 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-02-28T15:55:36.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T15:55:36.809 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-02-28T15:55:36.809 INFO:teuthology.orchestra.run.smithi001.stdout: linux-firmware-20240219-143.el9.noarch 2024-02-28T15:55:36.809 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T15:55:36.809 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T15:55:37.131 INFO:teuthology.task.kernel:Downloading kernel bd7d00fb724897dda560070d01bc6e5c80881f15 on host.a... 2024-02-28T15:55:37.131 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bd7d00fb724897dda560070d01bc6e5c80881f15 2024-02-28T15:55:37.382 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/bd7d00fb724897dda560070d01bc6e5c80881f15/centos/9/flavors/default/x86_64/ 2024-02-28T15:55:37.382 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gbd7d00fb7248-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/bd7d00fb724897dda560070d01bc6e5c80881f15/centos/9/flavors/default/x86_64/ --input-file=- 2024-02-28T15:55:37.385 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi001.front.sepia.ceph.com to complete... 2024-02-28T15:55:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28 15:55:48 URL:https://4.chacra.ceph.com/r/kernel/testing/bd7d00fb724897dda560070d01bc6e5c80881f15/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gbd7d00fb7248-1.x86_64.rpm [335852831/335852831] -> "/tmp/kernel.x86_64.rpm" [1] 2024-02-28T15:55:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:FINISHED --2024-02-28 15:55:48-- 2024-02-28T15:55:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:Total wall clock time: 11s 2024-02-28T15:55:48.771 INFO:teuthology.orchestra.run.smithi001.stderr:Downloaded: 1 files, 320M in 11s (28.9 MB/s) 2024-02-28T15:55:48.774 INFO:teuthology.task.kernel:Installing kernel bd7d00fb724897dda560070d01bc6e5c80881f15 on host.a... 2024-02-28T15:55:48.774 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-02-28T15:55:50.807 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying... ######################################## 2024-02-28T15:55:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing... ######################################## 2024-02-28T15:55:50.952 INFO:teuthology.orchestra.run.smithi001.stdout:Updating / installing... 2024-02-28T15:55:58.795 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-6.8.0_rc6_gbd7d00fb7248-1 ######################################## 2024-02-28T15:56:14.346 INFO:teuthology.orchestra.run.smithi001.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-28T15:56:37.474 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-02-28T15:56:37.475 DEBUG:teuthology.orchestra.run.smithi001:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-02-28T15:56:37.544 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248 2024-02-28T15:56:37.544 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/System.map 2024-02-28T15:56:37.544 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/config 2024-02-28T15:56:37.544 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/crc32c-intel.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-02-28T15:56:37.545 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/serpent-avx2.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kernel 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kernel/cpu 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kernel/cpu/mce 2024-02-28T15:56:37.546 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kvm 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kvm/kvm-amd.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kvm/kvm-intel.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kvm/kvm.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/ansi_cprng.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/anubis.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/arc4.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_memcpy.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_pq.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_raid6_recov.ko 2024-02-28T15:56:37.547 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_tx.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_xor.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/raid6test.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/authenc.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/authencesn.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/blake2b_generic.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/blowfish_common.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/blowfish_generic.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/camellia_generic.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cast5_generic.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cast6_generic.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cast_common.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/ccm.ko 2024-02-28T15:56:37.548 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cmac.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/crc32_generic.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/crypto_engine.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/crypto_user.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cts.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/des_generic.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/dh_generic.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/echainiv.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/essiv.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/fcrypt.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/khazad.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/lz4.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/lz4hc.ko 2024-02-28T15:56:37.549 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/md4.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/michael_mic.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/pcbc.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/pcrypt.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/rmd160.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/seed.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/serpent_generic.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/tcrypt.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/tea.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/twofish_common.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/twofish_generic.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/vmac.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/wp512.ko 2024-02-28T15:56:37.550 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/xcbc.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/xor.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/xxhash_generic.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/acpi_ipmi.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/acpi_pad.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/custom_method.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/ec_sys.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/platform_profile.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/sbs.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/sbshc.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/video.ko 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata 2024-02-28T15:56:37.551 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/acard-ahci.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/ahci_platform.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/ata_generic.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/libahci_platform.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_acpi.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_ali.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_amd.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_artop.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_atiixp.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_atp867x.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_cmd640.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_cmd64x.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_cypress.ko 2024-02-28T15:56:37.552 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_efar.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_hpt366.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_hpt37x.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_hpt3x2n.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_hpt3x3.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_it8213.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_it821x.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_jmicron.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_marvell.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_mpiix.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_netcell.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_ninja32.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_ns87410.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_ns87415.ko 2024-02-28T15:56:37.553 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_oldpiix.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_opti.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_optidma.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_pcmcia.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_pdc2027x.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_piccolo.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_rdc.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_sch.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_serverworks.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_sil680.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_sis.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_sl82c105.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_triflex.ko 2024-02-28T15:56:37.554 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_via.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pdc_adma.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_inic162x.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_mv.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_nv.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_promise.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_qstor.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_sil.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_sil24.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_sis.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_svw.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_sx4.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_uli.ko 2024-02-28T15:56:37.555 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_via.ko 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_vsc.ko 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/auxdisplay 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/auxdisplay/ks0108.ko 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/base 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/base/regmap 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/base/regmap/regmap-i2c.ko 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/base/regmap/regmap-mmio.ko 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/bcma 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/bcma/bcma.ko 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/aoe 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/aoe/aoe.ko 2024-02-28T15:56:37.556 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/brd.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/drbd 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/drbd/drbd.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/floppy.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/loop.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/mtip32xx 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/nbd.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/null_blk 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/null_blk/null_blk.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/pktcdvd.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/rbd.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/virtio_blk.ko 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/xen-blkback 2024-02-28T15:56:37.557 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/xen-blkfront.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/zram 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/zram/zram.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hangcheck-timer.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/amd-rng.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/intel-rng.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/via-rng.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/virtio-rng.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-02-28T15:56:37.558 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_si.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/lp.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/mwave 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/mwave/mwave.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ppdev.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tlclk.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_atmel.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_infineon.ko 2024-02-28T15:56:37.559 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_nsc.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_tis.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/uv_mmtimer.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/virtio_console.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/p4-clockmod.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/powernow-k8.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/speedstep-lib.ko 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto 2024-02-28T15:56:37.560 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/ccp 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/ccp/ccp.ko 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat/qat_common 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/padlock-aes.ko 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/padlock-sha.ko 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/virtio 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-02-28T15:56:37.561 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dca 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dca/dca.ko 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/devfreq 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma/dw 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma/dw/dw_dmac.ko 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma/ioat 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma/ioat/ioatdma.ko 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/amd64_edac.ko 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/e752x_edac.ko 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/edac_mce_amd.ko 2024-02-28T15:56:37.562 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i3000_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i3200_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i5100_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i5400_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i7300_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i7core_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i82975x_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/ie31200_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/sb_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/x38_edac.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/firmware 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/firmware/edd.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/firmware/iscsi_ibft.ko 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/gpio 2024-02-28T15:56:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/gpio/gpio-ich.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/gpio/gpio-viperboard.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-appleir.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-aureal.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-axff.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-dr.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-elecom.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-elo.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-emsff.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-gaff.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-gt683r.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-gyration.ko 2024-02-28T15:56:37.564 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-holtek-kbd.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-holtek-mouse.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-holtekff.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-hyperv.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-icade.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-keytouch.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-kye.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-lcpower.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-led.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-lenovo.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-logitech-dj.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-multitouch.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-ortek.ko 2024-02-28T15:56:37.565 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-penmount.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-petalynx.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-picolcd.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-pl.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-plantronics.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-primax.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-rmi.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-arvo.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-common.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-isku.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-kone.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-konepure.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-02-28T15:56:37.566 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-lua.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-pyra.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-ryos.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-savu.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-saitek.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-samsung.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-sensor-hub.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-sjoy.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-sony.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-speedlink.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-steelseries.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-sunplus.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-thrustmaster.ko 2024-02-28T15:56:37.567 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-tivo.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-tmff.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-topseed.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-twinhan.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-uclogic.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-waltop.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-wiimote.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-xinmo.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-zpff.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-zydacron.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/uhid.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/wacom.ko 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hv 2024-02-28T15:56:37.568 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hv/hv_balloon.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hv/hv_utils.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hv/hv_vmbus.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/abituguru.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/abituguru3.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/acpi_power_meter.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ad7414.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ad7418.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adc128d818.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm1025.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm1026.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm1029.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm1031.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm9240.ko 2024-02-28T15:56:37.569 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ads7828.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7410.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7411.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7462.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7470.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7475.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7x10.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/amc6821.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/applesmc.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/asb100.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/asc7621.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/asus_atk0110.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/atxp1.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/coretemp.ko 2024-02-28T15:56:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/dme1737.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ds1621.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ds620.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/emc1403.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/emc6w201.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/f71805f.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/f71882fg.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/f75375s.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/fam15h_power.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/fschmd.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/g760a.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/g762.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/gl518sm.ko 2024-02-28T15:56:37.571 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/gl520sm.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/hwmon-vid.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/i5500_temp.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/i5k_amb.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ibmaem.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ibmpex.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ina209.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ina2xx.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/it87.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/k10temp.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/k8temp.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lineage-pem.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm63.ko 2024-02-28T15:56:37.572 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm73.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm75.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm77.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm78.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm80.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm83.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm85.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm87.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm90.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm92.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm93.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm95234.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm95241.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm95245.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc2945.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4151.ko 2024-02-28T15:56:37.573 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4215.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4222.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4245.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4260.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4261.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max16065.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max1619.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max1668.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max197.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max6639.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max6650.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max6697.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/mcp3021.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/nct6683.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/nct6775-core.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/nct6775.ko 2024-02-28T15:56:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/nct7802.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pc87360.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pc87427.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pcf8591.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/max16064.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/max34440.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/max8688.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-02-28T15:56:37.575 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/powr1220.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sch5627.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sch5636.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sch56xx-common.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sht15.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sht21.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/shtc1.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sis5595.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/smsc47b397.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/smsc47m1.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/smsc47m192.ko 2024-02-28T15:56:37.576 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/thmc50.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/tmp102.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/tmp103.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/tmp401.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/tmp421.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/via-cputemp.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/via686a.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/vt1211.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/vt8231.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83627ehf.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83627hf.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83781d.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83791d.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83792d.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83793.ko 2024-02-28T15:56:37.577 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83795.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83l785ts.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83l786ng.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/algos 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-02-28T15:56:37.578 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-i801.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-isch.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-parport.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-via.ko 2024-02-28T15:56:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-dev.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-mux.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-smbus.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-stub.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/ib_cm.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/ib_core.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/ib_umad.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-02-28T15:56:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/iw_cm.ko 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/rdma_cm.ko 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/cxgb4 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mlx4 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mlx5 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mthca 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp 2024-02-28T15:56:37.581 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/ipoib 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/iser 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/isert 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/srp 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/srpt 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/gameport 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-02-28T15:56:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/gameport/fm801-gp.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/gameport/gameport.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/joydev.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/keyboard 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/keyboard/gpio_keys.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/appletouch.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/bcm5974.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/cyapatp.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/elan_i2c.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/sermouse.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/synaptics_usb.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/vsxxxaa.ko 2024-02-28T15:56:37.583 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/rmi4 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/rmi4/rmi_core.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio/altera_ps2.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio/arc_ps2.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio/serio_raw.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/sparse-keymap.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/leds-blinkm.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/leds-lm3530.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/leds-lp3944.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/leds-ss4200.ko 2024-02-28T15:56:37.584 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/bcache 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/bcache/bcache.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-bio-prison.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-cache-smq.ko 2024-02-28T15:56:37.585 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-cache.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-crypt.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-delay.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-flakey.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-log-userspace.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-multipath.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-queue-length.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-raid.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-round-robin.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-service-time.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-switch.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-thin-pool.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-verity.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/persistent-data 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-02-28T15:56:37.586 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/raid0.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/raid1.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/raid10.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/raid456.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptbase.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptctl.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptfc.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptlan.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptsas.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptscsih.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptspi.ko 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd 2024-02-28T15:56:37.587 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/lpc_ich.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/lpc_sch.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/mfd-core.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/sm501.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/viperboard.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/vx855.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/wl1273-core.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/altera-stapl 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/apds9802als.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/apds990x.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/bh1770glc.ko 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/cb710 2024-02-28T15:56:37.588 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/cb710/cb710.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/eeprom 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/eeprom/at24.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/eeprom/max6875.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/enclosure.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/hpilo.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/isl29003.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/isl29020.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/lis3lv02d 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/mei 2024-02-28T15:56:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/mei/mei-me.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/mei/mei-txe.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/mei/mei.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-gru 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-gru/gru.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-xp 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-xp/xp.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-xp/xpc.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/tifm_7xx1.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/tifm_core.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/tsl2550.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/vmw_balloon.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/vmw_vmci 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net 2024-02-28T15:56:37.590 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/bonding 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/bonding/bonding.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/b53 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/b53/b53_common.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/bcm-sf2.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/mv88e6060.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/mv88e6xxx 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dummy.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/eql.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-02-28T15:56:37.591 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390/8390.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/adaptec 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/agere 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/agere/et131x.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/alteon 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/altera 2024-02-28T15:56:37.592 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/amd 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/alx 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atl1c 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atl1e 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atlx 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-02-28T15:56:37.593 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/genet 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/brocade 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/brocade/bna 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio 2024-02-28T15:56:37.594 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/cisco 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/cisco/enic 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip 2024-02-28T15:56:37.595 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dlink 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dnet.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/emulex 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/emulex/benet 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/ethoc.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/fealnx.ko 2024-02-28T15:56:37.596 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e100.ko 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e1000 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e1000e 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/fm10k 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/i40e 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/iavf 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/igb 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/igbvf 2024-02-28T15:56:37.597 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/ixgbe 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/ixgbevf 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/jme.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/marvell 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/marvell/skge.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx4 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-02-28T15:56:37.598 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx5 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/micrel 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/myricom 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/myricom/myri10ge 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/natsemi 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/neterion 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-02-28T15:56:37.599 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/nvidia 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/packetengines 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/netxen 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/rdc 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-02-28T15:56:37.600 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek/atp.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/rocker 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sfc 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/silan 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sis 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sis/sis190.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sis/sis900.ko 2024-02-28T15:56:37.601 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/stmicro 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/stmicro/stmmac 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun/cassini.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun/niu.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun/sungem.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/tehuti 2024-02-28T15:56:37.602 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/ti 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/ti/tlan.ko 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/via 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/wiznet 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/xircom 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-02-28T15:56:37.603 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/geneve.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/hyperv 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ieee802154 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ieee802154/fakelb.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ifb.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ipvlan 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ipvlan/ipvlan.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/macvlan.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/macvtap.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-02-28T15:56:37.604 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio/mdio-regmap.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mii.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/net_failover.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/netconsole.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/nlmon.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/pcs 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/pcs/pcs-lynx.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/amd.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/bcm7xxx.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/bcm87xx.ko 2024-02-28T15:56:37.605 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/broadcom.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/cicada.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/davicom.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/dp83640.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/et1011c.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/icplus.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/lxt.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/marvell.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/micrel.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/national.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/phylink.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/qsemi.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/realtek.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/smsc.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/ste10Xp.ko 2024-02-28T15:56:37.606 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/vitesse.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/bsd_comp.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_async.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_deflate.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_generic.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_mppe.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_synctty.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/pppoe.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/pppox.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/pptp.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/slip 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/slip/slhc.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/slip/slip.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/sungem_phy.ko 2024-02-28T15:56:37.607 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/tap.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_activebackup.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_broadcast.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_random.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/tun.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/veth.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/virtio_net.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/vmxnet3 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/vxlan 2024-02-28T15:56:37.608 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/vxlan/vxlan.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/xen-netback 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/xen-netback/xen-netback.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/xen-netfront.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ntb 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ntb/ntb.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/nvme 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/nvme/host 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/nvme/host/nvme-core.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/nvme/host/nvme.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport/parport.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport/parport_cs.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport/parport_pc.ko 2024-02-28T15:56:37.609 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport/parport_serial.ko 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/hotplug 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/pcie 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/pcie/aer_inject.ko 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/xen-pcifront.ko 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pcmcia 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pcmcia/i82092.ko 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pcmcia/pd6729.ko 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pcmcia/yenta_socket.ko 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pinctrl 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pinctrl/intel 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-02-28T15:56:37.610 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/acer-wmi.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/acerhdf.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/apple-gmux.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/asus-laptop.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/asus-wmi.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/classmate-laptop.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/hdaps.ko 2024-02-28T15:56:37.611 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/intel 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/intel_ips.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/msi-wmi.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/mxm-wmi.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/samsung-laptop.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/samsung-q10.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/topstar-laptop.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/toshiba_haps.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/wmi-bmof.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/wmi.ko 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/powercap 2024-02-28T15:56:37.612 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/powercap/intel_rapl_common.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/powercap/intel_rapl_msr.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/clients 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/clients/pps-gpio.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/clients/pps-ldisc.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/clients/pps_parport.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/pps_core.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ptp 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ptp/ptp.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ptp/ptp_kvm.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-bq32k.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1286.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1307.ko 2024-02-28T15:56:37.613 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1374.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1511.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1553.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1672.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1742.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds2404.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds3232.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-em3027.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-fm3130.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-isl12022.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-isl1208.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-m41t80.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-m48t35.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-m48t59.ko 2024-02-28T15:56:37.614 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-max6900.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-msm6242.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf2127.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf85063.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf8523.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf8563.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf8583.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rp5c01.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rs5c372.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rx8025.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rx8581.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-x1205.ko 2024-02-28T15:56:37.615 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/3w-9xxx.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/3w-sas.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/3w-xxxx.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/BusLogic.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/a100u2w.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aacraid 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aacraid/aacraid.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/advansys.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aic7xxx 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/am53c974.ko 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/arcmsr 2024-02-28T15:56:37.616 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/atp870u.ko 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/be2iscsi 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bfa 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bfa/bfa.ko 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bnx2fc 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bnx2i 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/ch.ko 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/csiostor 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/csiostor/csiostor.ko 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi 2024-02-28T15:56:37.617 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/cxgb3i 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/cxgb4i 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/dc395x.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/dmx3191d.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/esas2r 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/esas2r/esas2r.ko 2024-02-28T15:56:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/esp_scsi.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fcoe 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fcoe/fcoe.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fnic 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fnic/fnic.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/hpsa.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/hptiop.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/hv_storvsc.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/initio.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/ipr.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/ips.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/isci 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/isci/isci.ko 2024-02-28T15:56:37.619 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/iscsi_tcp.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libfc 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libfc/libfc.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libiscsi.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libiscsi_tcp.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libsas 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libsas/libsas.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/lpfc 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/lpfc/lpfc.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-02-28T15:56:37.620 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mpt3sas 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mvsas 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mvsas/mvsas.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mvumi.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/pm8001 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/pmcraid.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla1280.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla2xxx 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-02-28T15:56:37.621 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla4xxx 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/raid_class.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_debug.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_fc.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_sas.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_spi.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_srp.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/ses.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/st.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/stex.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/sym53c8xx_2 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/virtio_scsi.ko 2024-02-28T15:56:37.622 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/vmw_pvscsi.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/wd719x.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ssb 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ssb/ssb.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/iscsi 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/loopback 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/loopback/tcm_loop.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_file.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_iblock.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_mod.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_pscsi.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_user.ko 2024-02-28T15:56:37.623 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/tcm_fc 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-02-28T15:56:37.624 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/ipwireless 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/n_gsm.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/n_hdlc.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/nozomi.ko 2024-02-28T15:56:37.625 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/8250 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/arc_uart.ko 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/jsm 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/jsm/jsm.ko 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs/core 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs/core/ufshcd-core.ko 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs/host 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio.ko 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio_aec.ko 2024-02-28T15:56:37.626 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio_cif.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio_pci_generic.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio_sercos3.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class/cdc-acm.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class/cdc-wdm.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class/usblp.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class/usbtmc.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/host 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/host/sl811-hcd.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/image 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/image/mdc800.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/image/microtek.ko 2024-02-28T15:56:37.627 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/adutux.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/appledisplay.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/emi26.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/emi62.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/ezusb.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/idmouse.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/iowarrior.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/isight_firmware.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/ldusb.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/legousbtower.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/sisusbvga 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/trancevibrator.ko 2024-02-28T15:56:37.628 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/usb3503.ko 2024-02-28T15:56:37.629 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/usblcd.ko 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/usbsevseg.ko 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/uss720.ko 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/yurex.ko 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/phy 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/phy/phy-generic.ko 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/aircable.ko 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ark3116.ko 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/belkin_sa.ko 2024-02-28T15:56:37.631 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ch341.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/cp210x.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/cyberjack.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/cypress_m8.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/digi_acceleport.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/empeg.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ftdi_sio.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/garmin_gps.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/io_edgeport.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/io_ti.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ipaq.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ipw.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ir-usb.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-02-28T15:56:37.632 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/keyspan.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/keyspan_pda.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/kl5kusb105.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/kobil_sct.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/mct_u232.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/mos7720.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/mos7840.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/navman.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/omninet.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/opticon.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/option.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/oti6858.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/pl2303.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/qcaux.ko 2024-02-28T15:56:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/qcserial.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/quatech2.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/safe_serial.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/sierra.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/spcp8x5.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ssu100.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/symbolserial.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/usb_debug.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/usb_wwan.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/visor.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/whiteheat.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/xsens_mt.ko 2024-02-28T15:56:37.634 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/uas.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-alauda.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-cypress.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-datafab.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-freecom.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-isd200.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-karma.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-onetouch.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-realtek.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-sddr09.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-sddr55.ko 2024-02-28T15:56:37.635 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-usbat.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/usb-storage.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/usbip 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/usbip/usbip-core.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/usbip/usbip-host.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/pci 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/pci/vfio-pci.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/vfio.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost 2024-02-28T15:56:37.636 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost/vhost.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost/vhost_iotlb.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost/vhost_net.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost/vhost_scsi.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video/backlight 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video/backlight/apple_bl.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video/backlight/lcd.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video/backlight/platform_lcd.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_balloon.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_mmio.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_pci.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-02-28T15:56:37.637 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_ring.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/masters 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/masters/ds2482.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/masters/ds2490.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-02-28T15:56:37.638 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_smem.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_therm.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/wire.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/alim1535_wdt.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/alim7101_wdt.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/f71808e_wdt.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/hpwdt.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/i6300esb.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/iTCO_wdt.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/ib700wdt.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/ibmasr.ko 2024-02-28T15:56:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/it8712f_wdt.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/it87_wdt.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/machzwd.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/nv_tco.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/pcwd_pci.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/pcwd_usb.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/sch311x_wdt.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/softdog.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/sp5100_tco.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/via_wdt.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/w83877f_wdt.ko 2024-02-28T15:56:37.640 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/w83977f_wdt.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/wdt_pci.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/xen_wdt.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-acpi-processor.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-evtchn.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-gntalloc.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-gntdev.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-pciback 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-privcmd.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xenfs 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xenfs/xenfs.ko 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs 2024-02-28T15:56:37.641 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/9p 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/9p/9p.ko 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/affs 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/affs/affs.ko 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/befs 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/befs/befs.ko 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/binfmt_misc.ko 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/btrfs 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/btrfs/btrfs.ko 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/cachefiles 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/cachefiles/cachefiles.ko 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ceph 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ceph/ceph.ko 2024-02-28T15:56:37.642 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/coda 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/coda/coda.ko 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/cramfs 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/cramfs/cramfs.ko 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/dlm 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/dlm/dlm.ko 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ecryptfs 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ecryptfs/ecryptfs.ko 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/f2fs 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/f2fs/f2fs.ko 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fat 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fat/fat.ko 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fat/msdos.ko 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fat/vfat.ko 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fuse 2024-02-28T15:56:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fuse/cuse.ko 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fuse/fuse.ko 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/gfs2 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/gfs2/gfs2.ko 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/hfs 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/hfs/hfs.ko 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/hfsplus 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/hfsplus/hfsplus.ko 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/isofs 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/isofs/isofs.ko 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/jfs 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/jfs/jfs.ko 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/lockd 2024-02-28T15:56:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/lockd/lockd.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/minix 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/minix/minix.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/netfs 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/netfs/netfs.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/blocklayout 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/filelayout 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/flexfilelayout 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/nfs.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/nfsv3.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/nfsv4.ko 2024-02-28T15:56:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs_common 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs_common/grace.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs_common/nfs_acl.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfsd 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfsd/nfsd.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nilfs2 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nilfs2/nilfs2.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-celtic.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-centeuro.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-croatian.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-cyrillic.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-gaelic.ko 2024-02-28T15:56:37.646 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-greek.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-iceland.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-inuit.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-roman.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-romanian.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-turkish.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp1250.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp1251.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp1255.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp737.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp775.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp850.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp852.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp855.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp857.ko 2024-02-28T15:56:37.647 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp860.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp861.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp862.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp863.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp864.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp865.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp866.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp869.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp874.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp932.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp936.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp949.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp950.ko 2024-02-28T15:56:37.648 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_euc-jp.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-1.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-13.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-14.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-15.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-2.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-3.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-4.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-5.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-6.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-7.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-9.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_koi8-r.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_koi8-ru.ko 2024-02-28T15:56:37.649 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_koi8-u.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_ucs2_utils.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_utf8.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/cluster 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/dlm 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/dlmfs 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/ocfs2.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/overlayfs 2024-02-28T15:56:37.650 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/overlayfs/overlay.ko 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/pstore 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/pstore/ramoops.ko 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/reiserfs 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/reiserfs/reiserfs.ko 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/romfs 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/romfs/romfs.ko 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/client 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/client/cifs.ko 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/common 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/common/cifs_arc4.ko 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/common/cifs_md4.ko 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/squashfs 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/squashfs/squashfs.ko 2024-02-28T15:56:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/sysv 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/sysv/sysv.ko 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/udf 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/udf/udf.ko 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ufs 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ufs/ufs.ko 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/xfs 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/xfs/xfs.ko 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/rcu 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/rcu/rcutorture.ko 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/torture.ko 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/trace 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib 2024-02-28T15:56:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/asn1_encoder.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crc-itu-t.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crc8.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crypto 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crypto/libarc4.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crypto/libdes.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/libcrc32c.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/lru_cache.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/lz4 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/lz4/lz4_compress.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/lz4/lz4hc_compress.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/math 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/math/cordic.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/raid6 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/raid6/raid6_pq.ko 2024-02-28T15:56:37.653 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/reed_solomon 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/reed_solomon/reed_solomon.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/ts_bm.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/ts_fsm.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/ts_kmp.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/zstd 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/zstd/zstd_compress.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/mm 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/mm/hwpoison-inject.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/6lowpan.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_dest.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_fragment.ko 2024-02-28T15:56:37.654 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_hop.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_ipv6.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_mobility.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_routing.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_udp.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/garp.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/mrp.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/p8022.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/psnap.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/stp.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/8021q 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/8021q/8021q.ko 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p 2024-02-28T15:56:37.655 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p/9pnet.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p/9pnet_fd.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p/9pnet_rdma.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p/9pnet_virtio.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/batman-adv 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/batman-adv/batman-adv.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/br_netfilter.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/bridge.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_among.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_arp.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-02-28T15:56:37.656 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_ip.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_limit.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_log.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_mark.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_snat.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_stp.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-02-28T15:56:37.657 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebtables.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ceph 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ceph/libceph.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/core 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/core/failover.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/core/pktgen.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp/dccp.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp/dccp_diag.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp/dccp_ipv4.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp/dccp_ipv6.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dns_resolver 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dns_resolver/dns_resolver.ko 2024-02-28T15:56:37.658 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/dsa_core.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/tag_brcm.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/tag_dsa.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/tag_none.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/tag_trailer.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154/6lowpan 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154/ieee802154.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154/ieee802154_socket.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ah4.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/esp4.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/fou.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/gre.ko 2024-02-28T15:56:37.659 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/inet_diag.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ip_gre.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ip_tunnel.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ip_vti.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ipcomp.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ipip.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/arp_tables.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-02-28T15:56:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/iptable_security.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_bic.ko 2024-02-28T15:56:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_dctcp.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_diag.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_highspeed.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_htcp.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_hybla.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_illinois.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_lp.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_scalable.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_vegas.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_veno.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_westwood.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_yeah.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tunnel4.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/udp_diag.ko 2024-02-28T15:56:37.662 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/udp_tunnel.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/xfrm4_tunnel.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ah6.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/esp6.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/fou6.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ip6_tunnel.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ip6_vti.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ipcomp6.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-02-28T15:56:37.663 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-02-28T15:56:37.664 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/sit.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/tunnel6.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/xfrm6_tunnel.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/key 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/key/af_key.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_core.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_debugfs.ko 2024-02-28T15:56:37.665 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_eth.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_ip.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_ip6.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_netlink.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_ppp.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/llc 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/llc/llc.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/mac802154 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/mac802154/mac802154.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/mpls 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/mpls/mpls_gso.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set.ko 2024-02-28T15:56:37.666 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-02-28T15:56:37.667 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-02-28T15:56:37.668 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conncount.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_h323.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_irc.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_sane.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_sip.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_dup_netdev.ko 2024-02-28T15:56:37.669 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_flow_table.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_flow_table_inet.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_log_syslog.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_amanda.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_ftp.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_irc.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_sip.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_tftp.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_synproxy_core.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_tables.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink_acct.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink_log.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink_osf.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink_queue.ko 2024-02-28T15:56:37.670 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_chain_nat.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_compat.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_ct.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_dup_netdev.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_fib.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_fwd_netdev.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_hash.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_limit.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_log.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_masq.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_nat.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_numgen.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_queue.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_quota.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_redir.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_reject.ko 2024-02-28T15:56:37.671 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_reject_inet.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_socket.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_tunnel.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_AUDIT.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_CHECKSUM.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_CLASSIFY.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_CT.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_DSCP.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_HL.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_HMARK.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_IDLETIMER.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_LED.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_LOG.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_MASQUERADE.ko 2024-02-28T15:56:37.672 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_NETMAP.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_NFLOG.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_NFQUEUE.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_RATEEST.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_REDIRECT.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_SECMARK.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TCPMSS.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TEE.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TPROXY.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TRACE.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_addrtype.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_bpf.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_cgroup.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_cluster.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_comment.ko 2024-02-28T15:56:37.673 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_connbytes.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_connlabel.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_connlimit.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_connmark.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_conntrack.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_cpu.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_dccp.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_devgroup.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_dscp.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_ecn.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_esp.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_hashlimit.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_helper.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_hl.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_ipcomp.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_iprange.ko 2024-02-28T15:56:37.674 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_ipvs.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_l2tp.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_length.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_limit.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_mac.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_mark.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_multiport.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_nat.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_nfacct.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_osf.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_owner.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_physdev.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_pkttype.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_policy.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_quota.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_rateest.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_realm.ko 2024-02-28T15:56:37.675 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_recent.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_sctp.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_set.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_socket.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_state.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_statistic.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_string.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_tcpmss.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_time.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_u32.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netlink 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netlink/netlink_diag.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/nsh 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/nsh/nsh.ko 2024-02-28T15:56:37.676 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch/openvswitch.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch/vport-geneve.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch/vport-gre.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch/vport-vxlan.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/packet 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/packet/af_packet_diag.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/rds 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/rds/rds.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/rds/rds_rdma.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/rds/rds_tcp.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_csum.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_gact.ko 2024-02-28T15:56:37.677 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_mirred.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_nat.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_pedit.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_police.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_simple.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_skbedit.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_vlan.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_basic.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_bpf.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_flow.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_fw.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_route.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_u32.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_cmp.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_ipset.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_meta.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_nbyte.ko 2024-02-28T15:56:37.678 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_text.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_u32.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_choke.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_codel.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_drr.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_fq.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_gred.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_hfsc.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_hhf.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_htb.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_ingress.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_mqprio.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_mqprio_lib.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_multiq.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_netem.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_pie.ko 2024-02-28T15:56:37.679 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_plug.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_prio.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_qfq.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_red.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_sfb.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_sfq.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_tbf.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_teql.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sctp 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sctp/sctp.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sctp/sctp_diag.ko 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/auth_gss 2024-02-28T15:56:37.680 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/sunrpc.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/xprtrdma 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/tipc 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/tipc/diag.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/tipc/tipc.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/unix 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/unix/unix_diag.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vsock.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vsock_diag.ko 2024-02-28T15:56:37.681 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vsock_loopback.ko 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/xfrm 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/xfrm/xfrm_ipcomp.ko 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys/encrypted-keys 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys/trusted-keys 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys/trusted-keys/trusted.ko 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/virt 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/virt/lib 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/virt/lib/irqbypass.ko 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.alias 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.alias.bin 2024-02-28T15:56:37.682 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.builtin 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.builtin.alias.bin 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.builtin.bin 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.builtin.modinfo 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.dep 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.dep.bin 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.devname 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.order 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.softdep 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.symbols 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.symbols.bin 2024-02-28T15:56:37.683 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/vmlinuz 2024-02-28T15:56:37.688 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gbd7d00fb7248 2024-02-28T15:56:37.688 DEBUG:teuthology.orchestra.run.smithi001:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout:postinstall scriptlet (using /bin/sh): 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gbd7d00fb7248 /lib/modules/6.8.0-rc6-gbd7d00fb7248/vmlinuz 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout:fi 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout:for file in vmlinuz System.map config; do 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gbd7d00fb7248/${file}" "/boot/${file}-6.8.0-rc6-gbd7d00fb7248"; then 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: cp "/lib/modules/6.8.0-rc6-gbd7d00fb7248/${file}" "/boot/${file}-6.8.0-rc6-gbd7d00fb7248" 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: fi 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout:done 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout:preuninstall scriptlet (using /bin/sh): 2024-02-28T15:56:37.712 INFO:teuthology.orchestra.run.smithi001.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-02-28T15:56:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:new-kernel-pkg --remove 6.8.0-rc6-gbd7d00fb7248 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gbd7d00fb7248.img 2024-02-28T15:56:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-02-28T15:56:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-install remove 6.8.0-rc6-gbd7d00fb7248 2024-02-28T15:56:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:fi 2024-02-28T15:56:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:postuninstall scriptlet (using /bin/sh): 2024-02-28T15:56:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:if [ -x /sbin/update-bootloader ]; then 2024-02-28T15:56:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gbd7d00fb7248 2024-02-28T15:56:37.713 INFO:teuthology.orchestra.run.smithi001.stdout:fi 2024-02-28T15:56:37.714 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-02-28T15:56:37.794 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-02-28T15:56:37.794 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-02-28T15:56:37.794 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 68.el9 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8264254 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-02-28T15:56:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-02-28T15:56:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T15:56:37.796 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-28T15:56:37.796 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-28T15:56:37.796 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-28T15:56:37.796 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-02-28T15:56:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T15:56:37.796 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-02-28T15:56:37.797 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-28T15:56:37.797 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 6.8.0-rc6-gbd7d00fb7248 2024-02-28T15:56:37.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-28T15:56:38.383 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-02-28T15:56:39.834 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-28T15:56:39.875 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-02-28T15:56:39.878 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-02-28T15:56:39.893 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.xjemxhUfAh 2024-02-28T15:56:39.893 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xjemxhUfAh 2024-02-28T15:56:39.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.xjemxhUfAh 2024-02-28T15:56:40.109 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.xjemxhUfAh is 6KB 2024-02-28T15:56:40.159 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.xjemxhUfAh 2024-02-28T15:56:40.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-02-28T15:56:40.242 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-28T15:56:40.242 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-28T15:56:40.242 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-28T15:56:40.242 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gbd7d00fb7248.conf 2024-02-28T15:56:40.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gbd7d00fb7248 2024-02-28T15:56:40.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-02-28T15:57:10.414 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-28T15:57:10.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-02-28T15:57:10.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T15:57:28.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-02-28T15:57:37.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-02-28T15:57:37.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T15:58:37.769 DEBUG:teuthology.orchestra.remote:timed out 2024-02-28T15:58:49.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-02-28T15:58:49.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T15:58:50.330 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-02-28T15:58:50.944 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-02-28T15:58:50.944 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "6.8.0-rc6-gbd7d00fb7248"... 2024-02-28T15:58:50.944 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-02-28T15:58:50.999 INFO:teuthology.orchestra.run.smithi001.stdout:6.8.0-rc6-gbd7d00fb7248 2024-02-28T15:58:50.999 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gbd7d00fb7248 vs 6.8.0-rc6-gbd7d00fb7248 2024-02-28T15:58:50.999 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-28T15:58:51.000 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-28T15:58:52.001 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-28T15:58:52.001 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-28T15:58:52.147 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-02-28T15:58:52.217 DEBUG:teuthology.parallel:result is None 2024-02-28T16:01:25.540 INFO:teuthology.orchestra.run.smithi066.stdout:linux-firmware-20240219-143.el9.noarch.rpm 1.0 MB/s | 376 MB 06:06 2024-02-28T16:01:25.540 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:01:25.541 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.0 MB/s | 376 MB 06:06 2024-02-28T16:01:28.721 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:01:28.815 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:01:28.815 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:01:32.157 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:01:32.157 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:01:39.514 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:01:39.929 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-02-28T16:01:40.019 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-02-28T16:01:40.952 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-02-28T16:01:40.952 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-02-28T16:01:41.350 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-02-28T16:01:41.350 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:01:41.350 INFO:teuthology.orchestra.run.smithi066.stdout:Upgraded: 2024-02-28T16:01:41.350 INFO:teuthology.orchestra.run.smithi066.stdout: linux-firmware-20240219-143.el9.noarch 2024-02-28T16:01:41.350 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:01:41.350 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:01:41.649 INFO:teuthology.task.kernel:Downloading kernel bd7d00fb724897dda560070d01bc6e5c80881f15 on host.b... 2024-02-28T16:01:41.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bd7d00fb724897dda560070d01bc6e5c80881f15 2024-02-28T16:01:41.988 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/bd7d00fb724897dda560070d01bc6e5c80881f15/centos/9/flavors/default/x86_64/ 2024-02-28T16:01:41.989 DEBUG:teuthology.orchestra.run.smithi066:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gbd7d00fb7248-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/bd7d00fb724897dda560070d01bc6e5c80881f15/centos/9/flavors/default/x86_64/ --input-file=- 2024-02-28T16:01:41.991 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi066.front.sepia.ceph.com to complete... 2024-02-28T16:02:14.501 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28 16:02:14 URL:https://4.chacra.ceph.com/r/kernel/testing/bd7d00fb724897dda560070d01bc6e5c80881f15/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gbd7d00fb7248-1.x86_64.rpm [335852831/335852831] -> "/tmp/kernel.x86_64.rpm" [1] 2024-02-28T16:02:14.501 INFO:teuthology.orchestra.run.smithi066.stderr:FINISHED --2024-02-28 16:02:14-- 2024-02-28T16:02:14.501 INFO:teuthology.orchestra.run.smithi066.stderr:Total wall clock time: 32s 2024-02-28T16:02:14.501 INFO:teuthology.orchestra.run.smithi066.stderr:Downloaded: 1 files, 320M in 32s (10.1 MB/s) 2024-02-28T16:02:14.504 INFO:teuthology.task.kernel:Installing kernel bd7d00fb724897dda560070d01bc6e5c80881f15 on host.b... 2024-02-28T16:02:14.505 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-02-28T16:02:16.530 INFO:teuthology.orchestra.run.smithi066.stdout:Verifying... ######################################## 2024-02-28T16:02:16.585 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing... ######################################## 2024-02-28T16:02:16.671 INFO:teuthology.orchestra.run.smithi066.stdout:Updating / installing... 2024-02-28T16:02:24.530 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-6.8.0_rc6_gbd7d00fb7248-1 ######################################## 2024-02-28T16:02:38.150 INFO:teuthology.orchestra.run.smithi066.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-28T16:03:01.273 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-02-28T16:03:01.273 DEBUG:teuthology.orchestra.run.smithi066:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-02-28T16:03:01.336 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248 2024-02-28T16:03:01.336 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/System.map 2024-02-28T16:03:01.336 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/config 2024-02-28T16:03:01.336 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel 2024-02-28T16:03:01.336 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/crc32c-intel.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-02-28T16:03:01.337 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/serpent-avx2.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kernel 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kernel/cpu 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kernel/cpu/mce 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-02-28T16:03:01.338 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kvm 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kvm/kvm-amd.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kvm/kvm-intel.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/arch/x86/kvm/kvm.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/ansi_cprng.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/anubis.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/arc4.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_memcpy.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_pq.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_raid6_recov.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_tx.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/async_xor.ko 2024-02-28T16:03:01.339 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/async_tx/raid6test.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/authenc.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/authencesn.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/blake2b_generic.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/blowfish_common.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/blowfish_generic.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/camellia_generic.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cast5_generic.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cast6_generic.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cast_common.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/ccm.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cmac.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/crc32_generic.ko 2024-02-28T16:03:01.340 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/crypto_engine.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/crypto_user.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/cts.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/des_generic.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/dh_generic.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/echainiv.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/essiv.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/fcrypt.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/khazad.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/lz4.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/lz4hc.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/md4.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/michael_mic.ko 2024-02-28T16:03:01.341 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/pcbc.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/pcrypt.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/rmd160.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/seed.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/serpent_generic.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/tcrypt.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/tea.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/twofish_common.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/twofish_generic.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/vmac.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/wp512.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/xcbc.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/xor.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/crypto/xxhash_generic.ko 2024-02-28T16:03:01.342 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/acpi_ipmi.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/acpi_pad.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/custom_method.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/ec_sys.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/platform_profile.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/sbs.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/sbshc.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/acpi/video.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/acard-ahci.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/ahci_platform.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/ata_generic.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/libahci_platform.ko 2024-02-28T16:03:01.343 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_acpi.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_ali.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_amd.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_artop.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_atiixp.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_atp867x.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_cmd640.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_cmd64x.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_cypress.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_efar.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_hpt366.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_hpt37x.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_hpt3x2n.ko 2024-02-28T16:03:01.344 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_hpt3x3.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_it8213.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_it821x.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_jmicron.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_marvell.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_mpiix.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_netcell.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_ninja32.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_ns87410.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_ns87415.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_oldpiix.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_opti.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_optidma.ko 2024-02-28T16:03:01.345 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_pcmcia.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_pdc2027x.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_piccolo.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_rdc.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_sch.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_serverworks.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_sil680.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_sis.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_sl82c105.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_triflex.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pata_via.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/pdc_adma.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_inic162x.ko 2024-02-28T16:03:01.346 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_mv.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_nv.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_promise.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_qstor.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_sil.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_sil24.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_sis.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_svw.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_sx4.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_uli.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_via.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ata/sata_vsc.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/auxdisplay 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/auxdisplay/ks0108.ko 2024-02-28T16:03:01.347 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/base 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/base/regmap 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/base/regmap/regmap-i2c.ko 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/base/regmap/regmap-mmio.ko 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/bcma 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/bcma/bcma.ko 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/aoe 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/aoe/aoe.ko 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/brd.ko 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/drbd 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/drbd/drbd.ko 2024-02-28T16:03:01.348 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/floppy.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/loop.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/mtip32xx 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/nbd.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/null_blk 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/null_blk/null_blk.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/pktcdvd.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/rbd.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/virtio_blk.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/xen-blkback 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/xen-blkfront.ko 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/zram 2024-02-28T16:03:01.349 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/block/zram/zram.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hangcheck-timer.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/amd-rng.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/intel-rng.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/via-rng.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/hw_random/virtio-rng.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-02-28T16:03:01.350 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_si.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/lp.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/mwave 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/mwave/mwave.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/ppdev.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tlclk.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_atmel.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_infineon.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_nsc.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_tis.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-02-28T16:03:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/uv_mmtimer.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/char/virtio_console.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/p4-clockmod.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/powernow-k8.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/cpufreq/speedstep-lib.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/ccp 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/ccp/ccp.ko 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel 2024-02-28T16:03:01.352 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat/qat_common 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/padlock-aes.ko 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/padlock-sha.ko 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/virtio 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dca 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dca/dca.ko 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/devfreq 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-02-28T16:03:01.353 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma/dw 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma/dw/dw_dmac.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma/ioat 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/dma/ioat/ioatdma.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/amd64_edac.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/e752x_edac.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/edac_mce_amd.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i3000_edac.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i3200_edac.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i5100_edac.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i5400_edac.ko 2024-02-28T16:03:01.354 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i7300_edac.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i7core_edac.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/i82975x_edac.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/ie31200_edac.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/sb_edac.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/edac/x38_edac.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/firmware 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/firmware/edd.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/firmware/iscsi_ibft.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/gpio 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/gpio/gpio-ich.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/gpio/gpio-viperboard.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-appleir.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-aureal.ko 2024-02-28T16:03:01.355 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-axff.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-dr.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-elecom.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-elo.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-emsff.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-gaff.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-gt683r.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-gyration.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-holtek-kbd.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-holtek-mouse.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-holtekff.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-hyperv.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-icade.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-keytouch.ko 2024-02-28T16:03:01.356 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-kye.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-lcpower.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-led.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-lenovo.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-logitech-dj.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-multitouch.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-ortek.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-penmount.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-petalynx.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-picolcd.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-pl.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-plantronics.ko 2024-02-28T16:03:01.357 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-primax.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-rmi.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-arvo.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-common.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-isku.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-kone.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-konepure.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-lua.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-pyra.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-ryos.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat-savu.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-roccat.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-saitek.ko 2024-02-28T16:03:01.358 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-samsung.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-sensor-hub.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-sjoy.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-sony.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-speedlink.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-steelseries.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-sunplus.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-thrustmaster.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-tivo.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-tmff.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-topseed.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-twinhan.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-uclogic.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-waltop.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-wiimote.ko 2024-02-28T16:03:01.359 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-xinmo.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-zpff.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/hid-zydacron.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/uhid.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hid/wacom.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hv 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hv/hv_balloon.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hv/hv_utils.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hv/hv_vmbus.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/abituguru.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/abituguru3.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/acpi_power_meter.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ad7414.ko 2024-02-28T16:03:01.360 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ad7418.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adc128d818.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm1025.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm1026.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm1029.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm1031.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adm9240.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ads7828.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7410.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7411.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7462.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7470.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7475.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/adt7x10.ko 2024-02-28T16:03:01.361 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/amc6821.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/applesmc.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/asb100.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/asc7621.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/asus_atk0110.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/atxp1.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/coretemp.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/dme1737.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ds1621.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ds620.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/emc1403.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/emc6w201.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/f71805f.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/f71882fg.ko 2024-02-28T16:03:01.362 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/f75375s.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/fam15h_power.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/fschmd.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/g760a.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/g762.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/gl518sm.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/gl520sm.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/hwmon-vid.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/i5500_temp.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/i5k_amb.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ibmaem.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ibmpex.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ina209.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ina2xx.ko 2024-02-28T16:03:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/it87.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/k10temp.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/k8temp.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lineage-pem.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm63.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm73.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm75.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm77.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm78.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm80.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm83.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm85.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm87.ko 2024-02-28T16:03:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm90.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm92.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm93.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm95234.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm95241.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/lm95245.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc2945.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4151.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4215.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4222.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4245.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4260.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/ltc4261.ko 2024-02-28T16:03:01.365 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max16065.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max1619.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max1668.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max197.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max6639.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max6650.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/max6697.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/mcp3021.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/nct6683.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/nct6775-core.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/nct6775.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/nct7802.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pc87360.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pc87427.ko 2024-02-28T16:03:01.366 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pcf8591.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/max16064.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/max34440.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/max8688.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-02-28T16:03:01.367 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/powr1220.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sch5627.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sch5636.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sch56xx-common.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sht15.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sht21.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/shtc1.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/sis5595.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/smsc47b397.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/smsc47m1.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/smsc47m192.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/thmc50.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/tmp102.ko 2024-02-28T16:03:01.368 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/tmp103.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/tmp401.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/tmp421.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/via-cputemp.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/via686a.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/vt1211.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/vt8231.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83627ehf.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83627hf.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83781d.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83791d.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83792d.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83793.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83795.ko 2024-02-28T16:03:01.369 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83l785ts.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/hwmon/w83l786ng.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/algos 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-02-28T16:03:01.370 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-i801.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-isch.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-parport.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-02-28T16:03:01.371 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-via.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-dev.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-mux.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-smbus.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/i2c/i2c-stub.ko 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core 2024-02-28T16:03:01.372 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/ib_cm.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/ib_core.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/ib_umad.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/iw_cm.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/rdma_cm.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/cxgb4 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mlx4 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mlx5 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mthca 2024-02-28T16:03:01.373 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/ipoib 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/iser 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/isert 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/srp 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/srpt 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input 2024-02-28T16:03:01.374 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/gameport 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/gameport/fm801-gp.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/gameport/gameport.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/joydev.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/keyboard 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/keyboard/gpio_keys.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/appletouch.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/bcm5974.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/cyapatp.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/elan_i2c.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/sermouse.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/synaptics_usb.ko 2024-02-28T16:03:01.375 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/mouse/vsxxxaa.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/rmi4 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/rmi4/rmi_core.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio/altera_ps2.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio/arc_ps2.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/serio/serio_raw.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/input/sparse-keymap.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/leds-blinkm.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/leds-lm3530.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/leds-lp3944.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/leds-ss4200.ko 2024-02-28T16:03:01.376 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/bcache 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/bcache/bcache.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-bio-prison.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-cache-smq.ko 2024-02-28T16:03:01.377 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-cache.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-crypt.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-delay.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-flakey.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-log-userspace.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-multipath.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-queue-length.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-raid.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-round-robin.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-service-time.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-switch.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-thin-pool.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/dm-verity.ko 2024-02-28T16:03:01.378 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/persistent-data 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/raid0.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/raid1.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/raid10.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/md/raid456.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptbase.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptctl.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptfc.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptlan.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptsas.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptscsih.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/message/fusion/mptspi.ko 2024-02-28T16:03:01.379 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/lpc_ich.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/lpc_sch.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/mfd-core.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/sm501.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/viperboard.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/vx855.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/mfd/wl1273-core.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/altera-stapl 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/apds9802als.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/apds990x.ko 2024-02-28T16:03:01.380 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/bh1770glc.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/cb710 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/cb710/cb710.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/eeprom 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/eeprom/at24.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/eeprom/max6875.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/enclosure.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/hpilo.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/isl29003.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/isl29020.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/lis3lv02d 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-02-28T16:03:01.381 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/mei 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/mei/mei-me.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/mei/mei-txe.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/mei/mei.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-gru 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-gru/gru.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-xp 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-xp/xp.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-xp/xpc.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/tifm_7xx1.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/tifm_core.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/tsl2550.ko 2024-02-28T16:03:01.382 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/vmw_balloon.ko 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/vmw_vmci 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/bonding 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/bonding/bonding.ko 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/b53 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/b53/b53_common.ko 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/bcm-sf2.ko 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/mv88e6060.ko 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/mv88e6xxx 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-02-28T16:03:01.383 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/dummy.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/eql.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390/8390.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-02-28T16:03:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/adaptec 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/agere 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/agere/et131x.ko 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/alteon 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/altera 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/amd 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-02-28T16:03:01.385 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/alx 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atl1c 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atl1e 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atlx 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-02-28T16:03:01.386 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/genet 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/brocade 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/brocade/bna 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-02-28T16:03:01.387 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/cisco 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/cisco/enic 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-02-28T16:03:01.388 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dlink 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/dnet.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/emulex 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/emulex/benet 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-02-28T16:03:01.389 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/ethoc.ko 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/fealnx.ko 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e100.ko 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e1000 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e1000e 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/fm10k 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/i40e 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-02-28T16:03:01.390 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/iavf 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/igb 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/igbvf 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/ixgbe 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/ixgbevf 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/jme.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/marvell 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/marvell/skge.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-02-28T16:03:01.391 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox 2024-02-28T16:03:01.392 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx4 2024-02-28T16:03:01.392 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-02-28T16:03:01.392 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-02-28T16:03:01.392 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx5 2024-02-28T16:03:01.392 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-02-28T16:03:01.392 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-02-28T16:03:01.392 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/micrel 2024-02-28T16:03:01.392 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-02-28T16:03:01.395 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/myricom 2024-02-28T16:03:01.395 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/myricom/myri10ge 2024-02-28T16:03:01.395 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-02-28T16:03:01.395 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/natsemi 2024-02-28T16:03:01.395 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/neterion 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/nvidia 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/packetengines 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/netxen 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-02-28T16:03:01.396 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/rdc 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek/atp.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/rocker 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sfc 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/silan 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-02-28T16:03:01.397 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sis 2024-02-28T16:03:01.398 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sis/sis190.ko 2024-02-28T16:03:01.398 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sis/sis900.ko 2024-02-28T16:03:01.398 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc 2024-02-28T16:03:01.398 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-02-28T16:03:01.398 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-02-28T16:03:01.398 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-02-28T16:03:01.398 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/stmicro 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/stmicro/stmmac 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun/cassini.ko 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun/niu.ko 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun/sungem.ko 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/tehuti 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-02-28T16:03:01.399 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/ti 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/ti/tlan.ko 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/via 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/wiznet 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/xircom 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/geneve.ko 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/hyperv 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ieee802154 2024-02-28T16:03:01.400 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ieee802154/fakelb.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ifb.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ipvlan 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ipvlan/ipvlan.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/macvlan.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/macvtap.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mdio/mdio-regmap.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/mii.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/net_failover.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/netconsole.ko 2024-02-28T16:03:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/nlmon.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/pcs 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/pcs/pcs-lynx.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/amd.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/bcm7xxx.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/bcm87xx.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/broadcom.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/cicada.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/davicom.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/dp83640.ko 2024-02-28T16:03:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/et1011c.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/icplus.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/lxt.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/marvell.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/micrel.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/national.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/phylink.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/qsemi.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/realtek.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/smsc.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/ste10Xp.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/phy/vitesse.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/bsd_comp.ko 2024-02-28T16:03:01.403 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_async.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_deflate.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_generic.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_mppe.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/ppp_synctty.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/pppoe.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/pppox.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/ppp/pptp.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/slip 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/slip/slhc.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/slip/slip.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/sungem_phy.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/tap.ko 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team 2024-02-28T16:03:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_activebackup.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_broadcast.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_random.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/tun.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/veth.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/virtio_net.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/vmxnet3 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/vxlan 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/vxlan/vxlan.ko 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/xen-netback 2024-02-28T16:03:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/xen-netback/xen-netback.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/net/xen-netfront.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ntb 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ntb/ntb.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/nvme 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/nvme/host 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/nvme/host/nvme-core.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/nvme/host/nvme.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport/parport.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport/parport_cs.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport/parport_pc.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/parport/parport_serial.ko 2024-02-28T16:03:01.406 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/hotplug 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/pcie 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/pcie/aer_inject.ko 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pci/xen-pcifront.ko 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pcmcia 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pcmcia/i82092.ko 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pcmcia/pd6729.ko 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pcmcia/yenta_socket.ko 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pinctrl 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pinctrl/intel 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform 2024-02-28T16:03:01.407 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/acer-wmi.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/acerhdf.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/apple-gmux.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/asus-laptop.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/asus-wmi.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/classmate-laptop.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/hdaps.ko 2024-02-28T16:03:01.408 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/intel 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/intel_ips.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/msi-wmi.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/mxm-wmi.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/samsung-laptop.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/samsung-q10.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/topstar-laptop.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/toshiba_haps.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/wmi-bmof.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/platform/x86/wmi.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/powercap 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/powercap/intel_rapl_common.ko 2024-02-28T16:03:01.409 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/powercap/intel_rapl_msr.ko 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/clients 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/clients/pps-gpio.ko 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/clients/pps-ldisc.ko 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/clients/pps_parport.ko 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/pps/pps_core.ko 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ptp 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ptp/ptp.ko 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ptp/ptp_kvm.ko 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-bq32k.ko 2024-02-28T16:03:01.410 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1286.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1307.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1374.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1511.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1553.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1672.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds1742.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds2404.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-ds3232.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-em3027.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-fm3130.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-isl12022.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-isl1208.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-m41t80.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-m48t35.ko 2024-02-28T16:03:01.411 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-m48t59.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-max6900.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-msm6242.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf2127.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf85063.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf8523.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf8563.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-pcf8583.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rp5c01.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rs5c372.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rx8025.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-rx8581.ko 2024-02-28T16:03:01.412 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/rtc/rtc-x1205.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/3w-9xxx.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/3w-sas.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/3w-xxxx.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/BusLogic.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/a100u2w.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aacraid 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aacraid/aacraid.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/advansys.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aic7xxx 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-02-28T16:03:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/am53c974.ko 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/arcmsr 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/atp870u.ko 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/be2iscsi 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bfa 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bfa/bfa.ko 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bnx2fc 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bnx2i 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-02-28T16:03:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/ch.ko 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/csiostor 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/csiostor/csiostor.ko 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/cxgb3i 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/cxgb4i 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/dc395x.ko 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-02-28T16:03:01.415 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/dmx3191d.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/esas2r 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/esas2r/esas2r.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/esp_scsi.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fcoe 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fcoe/fcoe.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fnic 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/fnic/fnic.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/hpsa.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/hptiop.ko 2024-02-28T16:03:01.416 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/hv_storvsc.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/initio.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/ipr.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/ips.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/isci 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/isci/isci.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/iscsi_tcp.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libfc 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libfc/libfc.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libiscsi.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libiscsi_tcp.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libsas 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/libsas/libsas.ko 2024-02-28T16:03:01.417 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/lpfc 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/lpfc/lpfc.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mpt3sas 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mvsas 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mvsas/mvsas.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/mvumi.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/pm8001 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/pmcraid.ko 2024-02-28T16:03:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla1280.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla2xxx 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla4xxx 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/raid_class.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_debug.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_fc.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_sas.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_spi.ko 2024-02-28T16:03:01.419 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/scsi_transport_srp.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/ses.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/st.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/stex.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/sym53c8xx_2 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/virtio_scsi.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/vmw_pvscsi.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/scsi/wd719x.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ssb 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ssb/ssb.ko 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/iscsi 2024-02-28T16:03:01.420 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/loopback 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/loopback/tcm_loop.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_file.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_iblock.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_mod.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_pscsi.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/target_core_user.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/tcm_fc 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-02-28T16:03:01.421 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-02-28T16:03:01.422 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/ipwireless 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/n_gsm.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/n_hdlc.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/nozomi.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/8250 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/arc_uart.ko 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/jsm 2024-02-28T16:03:01.423 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/tty/serial/jsm/jsm.ko 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs/core 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs/core/ufshcd-core.ko 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs/host 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio.ko 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio_aec.ko 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio_cif.ko 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio_pci_generic.ko 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/uio/uio_sercos3.ko 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb 2024-02-28T16:03:01.424 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class/cdc-acm.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class/cdc-wdm.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class/usblp.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/class/usbtmc.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/host 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/host/sl811-hcd.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/image 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/image/mdc800.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/image/microtek.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/adutux.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/appledisplay.ko 2024-02-28T16:03:01.425 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/emi26.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/emi62.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/ezusb.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/idmouse.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/iowarrior.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/isight_firmware.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/ldusb.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/legousbtower.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/sisusbvga 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/trancevibrator.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/usb3503.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/usblcd.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/usbsevseg.ko 2024-02-28T16:03:01.426 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/uss720.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/misc/yurex.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/phy 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/phy/phy-generic.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/aircable.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ark3116.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/belkin_sa.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ch341.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/cp210x.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/cyberjack.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/cypress_m8.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/digi_acceleport.ko 2024-02-28T16:03:01.427 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/empeg.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ftdi_sio.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/garmin_gps.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/io_edgeport.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/io_ti.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ipaq.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ipw.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ir-usb.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/keyspan.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/keyspan_pda.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/kl5kusb105.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/kobil_sct.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/mct_u232.ko 2024-02-28T16:03:01.428 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/mos7720.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/mos7840.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/navman.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/omninet.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/opticon.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/option.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/oti6858.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/pl2303.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/qcaux.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/qcserial.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/quatech2.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/safe_serial.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/sierra.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/spcp8x5.ko 2024-02-28T16:03:01.429 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ssu100.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/symbolserial.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/usb_debug.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/usb_wwan.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/visor.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/whiteheat.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/serial/xsens_mt.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/uas.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-alauda.ko 2024-02-28T16:03:01.430 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-cypress.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-datafab.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-freecom.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-isd200.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-karma.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-onetouch.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-realtek.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-sddr09.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-sddr55.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/ums-usbat.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/storage/usb-storage.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/usbip 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/usbip/usbip-core.ko 2024-02-28T16:03:01.431 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/usbip/usbip-host.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/pci 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/pci/vfio-pci.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/vfio.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost/vhost.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost/vhost_iotlb.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost/vhost_net.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/vhost/vhost_scsi.ko 2024-02-28T16:03:01.432 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video/backlight 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video/backlight/apple_bl.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video/backlight/lcd.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/video/backlight/platform_lcd.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_balloon.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_mmio.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_pci.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/virtio/virtio_ring.ko 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1 2024-02-28T16:03:01.433 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/masters 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/masters/ds2482.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/masters/ds2490.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_smem.ko 2024-02-28T16:03:01.434 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/slaves/w1_therm.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/w1/wire.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/alim1535_wdt.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/alim7101_wdt.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/f71808e_wdt.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/hpwdt.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/i6300esb.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/iTCO_wdt.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/ib700wdt.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/ibmasr.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-02-28T16:03:01.435 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/it8712f_wdt.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/it87_wdt.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/machzwd.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/nv_tco.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/pcwd_pci.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/pcwd_usb.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/sch311x_wdt.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/softdog.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/sp5100_tco.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/via_wdt.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/w83877f_wdt.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/w83977f_wdt.ko 2024-02-28T16:03:01.436 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/wdt_pci.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/watchdog/xen_wdt.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-acpi-processor.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-evtchn.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-gntalloc.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-gntdev.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-pciback 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xen-privcmd.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xenfs 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/drivers/xen/xenfs/xenfs.ko 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/9p 2024-02-28T16:03:01.437 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/9p/9p.ko 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/affs 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/affs/affs.ko 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/befs 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/befs/befs.ko 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/binfmt_misc.ko 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/btrfs 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/btrfs/btrfs.ko 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/cachefiles 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/cachefiles/cachefiles.ko 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ceph 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ceph/ceph.ko 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/coda 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/coda/coda.ko 2024-02-28T16:03:01.438 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/cramfs 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/cramfs/cramfs.ko 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/dlm 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/dlm/dlm.ko 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ecryptfs 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ecryptfs/ecryptfs.ko 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/f2fs 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/f2fs/f2fs.ko 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fat 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fat/fat.ko 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fat/msdos.ko 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fat/vfat.ko 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fuse 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fuse/cuse.ko 2024-02-28T16:03:01.439 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/fuse/fuse.ko 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/gfs2 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/gfs2/gfs2.ko 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/hfs 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/hfs/hfs.ko 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/hfsplus 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/hfsplus/hfsplus.ko 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/isofs 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/isofs/isofs.ko 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/jfs 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/jfs/jfs.ko 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/lockd 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/lockd/lockd.ko 2024-02-28T16:03:01.440 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/minix 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/minix/minix.ko 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/netfs 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/netfs/netfs.ko 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/blocklayout 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/filelayout 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/flexfilelayout 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/nfs.ko 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/nfsv3.ko 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs/nfsv4.ko 2024-02-28T16:03:01.441 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs_common 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs_common/grace.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfs_common/nfs_acl.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfsd 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nfsd/nfsd.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nilfs2 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nilfs2/nilfs2.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-celtic.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-centeuro.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-croatian.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-cyrillic.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-gaelic.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-greek.ko 2024-02-28T16:03:01.442 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-iceland.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-inuit.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-roman.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-romanian.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/mac-turkish.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp1250.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp1251.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp1255.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp737.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp775.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp850.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp852.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp855.ko 2024-02-28T16:03:01.443 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp857.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp860.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp861.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp862.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp863.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp864.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp865.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp866.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp869.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp874.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp932.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp936.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp949.ko 2024-02-28T16:03:01.444 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_cp950.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_euc-jp.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-1.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-13.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-14.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-15.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-2.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-3.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-4.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-5.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-6.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-7.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_iso8859-9.ko 2024-02-28T16:03:01.445 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_koi8-r.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_koi8-ru.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_koi8-u.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_ucs2_utils.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/nls/nls_utf8.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/cluster 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/dlm 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/dlmfs 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/ocfs2.ko 2024-02-28T16:03:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/overlayfs 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/overlayfs/overlay.ko 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/pstore 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/pstore/ramoops.ko 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/reiserfs 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/reiserfs/reiserfs.ko 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/romfs 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/romfs/romfs.ko 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb 2024-02-28T16:03:01.447 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/client 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/client/cifs.ko 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/common 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/common/cifs_arc4.ko 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/smb/common/cifs_md4.ko 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/squashfs 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/squashfs/squashfs.ko 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/sysv 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/sysv/sysv.ko 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/udf 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/udf/udf.ko 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ufs 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/ufs/ufs.ko 2024-02-28T16:03:01.448 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/xfs 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/fs/xfs/xfs.ko 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/rcu 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/rcu/rcutorture.ko 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/torture.ko 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/trace 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/asn1_encoder.ko 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crc-itu-t.ko 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crc8.ko 2024-02-28T16:03:01.449 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crypto 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crypto/libarc4.ko 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/crypto/libdes.ko 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/libcrc32c.ko 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/lru_cache.ko 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/lz4 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/lz4/lz4_compress.ko 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/lz4/lz4hc_compress.ko 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/math 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/math/cordic.ko 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/raid6 2024-02-28T16:03:01.450 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/raid6/raid6_pq.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/reed_solomon 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/reed_solomon/reed_solomon.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/ts_bm.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/ts_fsm.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/ts_kmp.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/zstd 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/lib/zstd/zstd_compress.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/mm 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/mm/hwpoison-inject.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/6lowpan.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_dest.ko 2024-02-28T16:03:01.451 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_fragment.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_hop.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_ipv6.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_mobility.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_routing.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/6lowpan/nhc_udp.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/garp.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/mrp.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/p8022.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/psnap.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/802/stp.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/8021q 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/8021q/8021q.ko 2024-02-28T16:03:01.452 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p/9pnet.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p/9pnet_fd.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p/9pnet_rdma.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/9p/9pnet_virtio.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/batman-adv 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/batman-adv/batman-adv.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/br_netfilter.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/bridge.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_among.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_arp.ko 2024-02-28T16:03:01.453 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_ip.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_limit.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_log.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_mark.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_snat.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_stp.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-02-28T16:03:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/ebtables.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ceph 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ceph/libceph.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/core 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/core/failover.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/core/pktgen.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp/dccp.ko 2024-02-28T16:03:01.455 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp/dccp_diag.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp/dccp_ipv4.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dccp/dccp_ipv6.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dns_resolver 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dns_resolver/dns_resolver.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/dsa_core.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/tag_brcm.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/tag_dsa.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/tag_none.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/dsa/tag_trailer.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154/6lowpan 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-02-28T16:03:01.456 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154/ieee802154.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ieee802154/ieee802154_socket.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ah4.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/esp4.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/fou.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/gre.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/inet_diag.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ip_gre.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ip_tunnel.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ip_vti.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ipcomp.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/ipip.ko 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter 2024-02-28T16:03:01.457 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/arp_tables.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/iptable_security.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-02-28T16:03:01.458 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_bic.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_dctcp.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_diag.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_highspeed.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_htcp.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_hybla.ko 2024-02-28T16:03:01.459 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_illinois.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_lp.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_scalable.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_vegas.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_veno.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_westwood.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tcp_yeah.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/tunnel4.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/udp_diag.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/udp_tunnel.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv4/xfrm4_tunnel.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ah6.ko 2024-02-28T16:03:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/esp6.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/fou6.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ip6_tunnel.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ip6_vti.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/ipcomp6.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-02-28T16:03:01.461 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-02-28T16:03:01.462 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/sit.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/tunnel6.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/ipv6/xfrm6_tunnel.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/key 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/key/af_key.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_core.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_debugfs.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_eth.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_ip.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_ip6.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_netlink.ko 2024-02-28T16:03:01.463 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/l2tp/l2tp_ppp.ko 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/llc 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/llc/llc.ko 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/mac802154 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/mac802154/mac802154.ko 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/mpls 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/mpls/mpls_gso.ko 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set.ko 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-02-28T16:03:01.464 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs 2024-02-28T16:03:01.465 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-02-28T16:03:01.466 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conncount.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_h323.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_irc.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_sane.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_sip.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-02-28T16:03:01.467 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_dup_netdev.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_flow_table.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_flow_table_inet.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_log_syslog.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_amanda.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_ftp.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_irc.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_sip.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_nat_tftp.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_synproxy_core.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nf_tables.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink.ko 2024-02-28T16:03:01.468 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink_acct.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink_log.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink_osf.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nfnetlink_queue.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_chain_nat.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_compat.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_ct.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_dup_netdev.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_fib.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_fwd_netdev.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_hash.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_limit.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_log.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_masq.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_nat.ko 2024-02-28T16:03:01.469 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_numgen.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_queue.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_quota.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_redir.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_reject.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_reject_inet.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_socket.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/nft_tunnel.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_AUDIT.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_CHECKSUM.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_CLASSIFY.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_CT.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_DSCP.ko 2024-02-28T16:03:01.470 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_HL.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_HMARK.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_IDLETIMER.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_LED.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_LOG.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_MASQUERADE.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_NETMAP.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_NFLOG.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_NFQUEUE.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_RATEEST.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_REDIRECT.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_SECMARK.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TCPMSS.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-02-28T16:03:01.471 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TEE.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TPROXY.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_TRACE.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_addrtype.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_bpf.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_cgroup.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_cluster.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_comment.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_connbytes.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_connlabel.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_connlimit.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_connmark.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_conntrack.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_cpu.ko 2024-02-28T16:03:01.472 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_dccp.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_devgroup.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_dscp.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_ecn.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_esp.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_hashlimit.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_helper.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_hl.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_ipcomp.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_iprange.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_ipvs.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_l2tp.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_length.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_limit.ko 2024-02-28T16:03:01.473 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_mac.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_mark.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_multiport.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_nat.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_nfacct.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_osf.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_owner.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_physdev.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_pkttype.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_policy.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_quota.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_rateest.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_realm.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_recent.ko 2024-02-28T16:03:01.474 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_sctp.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_set.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_socket.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_state.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_statistic.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_string.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_tcpmss.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_time.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netfilter/xt_u32.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netlink 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/netlink/netlink_diag.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/nsh 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/nsh/nsh.ko 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch 2024-02-28T16:03:01.475 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch/openvswitch.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch/vport-geneve.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch/vport-gre.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/openvswitch/vport-vxlan.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/packet 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/packet/af_packet_diag.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/rds 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/rds/rds.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/rds/rds_rdma.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/rds/rds_tcp.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_csum.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_gact.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_mirred.ko 2024-02-28T16:03:01.476 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_nat.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_pedit.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_police.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_simple.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_skbedit.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/act_vlan.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_basic.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_bpf.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_flow.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_fw.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_route.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/cls_u32.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_cmp.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_ipset.ko 2024-02-28T16:03:01.477 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_meta.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_nbyte.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_text.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/em_u32.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_choke.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_codel.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_drr.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_fq.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_gred.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_hfsc.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_hhf.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_htb.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_ingress.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_mqprio.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_mqprio_lib.ko 2024-02-28T16:03:01.478 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_multiq.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_netem.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_pie.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_plug.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_prio.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_qfq.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_red.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_sfb.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_sfq.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_tbf.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sched/sch_teql.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sctp 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sctp/sctp.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sctp/sctp_diag.ko 2024-02-28T16:03:01.479 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/auth_gss 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/sunrpc.ko 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/xprtrdma 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/tipc 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/tipc/diag.ko 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/tipc/tipc.ko 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/unix 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/unix/unix_diag.ko 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-02-28T16:03:01.480 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vsock.ko 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vsock_diag.ko 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/vmw_vsock/vsock_loopback.ko 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/xfrm 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/net/xfrm/xfrm_ipcomp.ko 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys/encrypted-keys 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys/trusted-keys 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/security/keys/trusted-keys/trusted.ko 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/virt 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/virt/lib 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/kernel/virt/lib/irqbypass.ko 2024-02-28T16:03:01.481 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.alias 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.alias.bin 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.builtin 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.builtin.alias.bin 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.builtin.bin 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.builtin.modinfo 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.dep 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.dep.bin 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.devname 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.order 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.softdep 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.symbols 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/modules.symbols.bin 2024-02-28T16:03:01.482 INFO:teuthology.orchestra.run.smithi066.stdout:/lib/modules/6.8.0-rc6-gbd7d00fb7248/vmlinuz 2024-02-28T16:03:01.486 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gbd7d00fb7248 2024-02-28T16:03:01.487 DEBUG:teuthology.orchestra.run.smithi066:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-02-28T16:03:01.510 INFO:teuthology.orchestra.run.smithi066.stdout:postinstall scriptlet (using /bin/sh): 2024-02-28T16:03:01.510 INFO:teuthology.orchestra.run.smithi066.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-02-28T16:03:01.510 INFO:teuthology.orchestra.run.smithi066.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gbd7d00fb7248 /lib/modules/6.8.0-rc6-gbd7d00fb7248/vmlinuz 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout:fi 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout:for file in vmlinuz System.map config; do 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gbd7d00fb7248/${file}" "/boot/${file}-6.8.0-rc6-gbd7d00fb7248"; then 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout: cp "/lib/modules/6.8.0-rc6-gbd7d00fb7248/${file}" "/boot/${file}-6.8.0-rc6-gbd7d00fb7248" 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout: fi 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout:done 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout:preuninstall scriptlet (using /bin/sh): 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout:new-kernel-pkg --remove 6.8.0-rc6-gbd7d00fb7248 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gbd7d00fb7248.img 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-02-28T16:03:01.511 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-install remove 6.8.0-rc6-gbd7d00fb7248 2024-02-28T16:03:01.512 INFO:teuthology.orchestra.run.smithi066.stdout:fi 2024-02-28T16:03:01.512 INFO:teuthology.orchestra.run.smithi066.stdout:postuninstall scriptlet (using /bin/sh): 2024-02-28T16:03:01.512 INFO:teuthology.orchestra.run.smithi066.stdout:if [ -x /sbin/update-bootloader ]; then 2024-02-28T16:03:01.512 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gbd7d00fb7248 2024-02-28T16:03:01.512 INFO:teuthology.orchestra.run.smithi066.stdout:fi 2024-02-28T16:03:01.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -qi grub2-tools 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:Name : grub2-tools 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:Epoch : 1 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:Version : 2.06 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:Release : 68.el9 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: x86_64 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:Group : Unspecified 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:Size : 8264254 2024-02-28T16:03:01.593 INFO:teuthology.orchestra.run.smithi066.stdout:License : GPLv3+ 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:Packager : builder@centos.org 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:Vendor : CentOS 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:Summary : Support tools for GRUB. 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:Description : 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout:hardware devices. 2024-02-28T16:03:01.594 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:03:01.595 INFO:teuthology.orchestra.run.smithi066.stdout:This subpackage provides tools for support of all platforms. 2024-02-28T16:03:01.595 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-28T16:03:01.596 INFO:teuthology.task.kernel:Updating grub on smithi066 to boot 6.8.0-rc6-gbd7d00fb7248 2024-02-28T16:03:01.596 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-28T16:03:02.097 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2024-02-28T16:03:03.684 INFO:teuthology.orchestra.run.smithi066.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-28T16:03:03.723 INFO:teuthology.orchestra.run.smithi066.stderr:done 2024-02-28T16:03:03.725 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-02-28T16:03:03.741 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.urw8R8MLe0 2024-02-28T16:03:03.742 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.urw8R8MLe0 2024-02-28T16:03:03.816 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.urw8R8MLe0 2024-02-28T16:03:03.965 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.urw8R8MLe0 is 6KB 2024-02-28T16:03:04.014 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.urw8R8MLe0 2024-02-28T16:03:04.028 DEBUG:teuthology.orchestra.run.smithi066:> sudo /bin/ls /boot/loader/entries || true 2024-02-28T16:03:04.098 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-28T16:03:04.098 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-28T16:03:04.098 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-28T16:03:04.098 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gbd7d00fb7248.conf 2024-02-28T16:03:04.099 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gbd7d00fb7248 2024-02-28T16:03:04.261 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2024-02-28T16:03:34.263 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-28T16:03:34.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-02-28T16:03:34.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T16:04:08.896 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-02-28T16:04:17.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-02-28T16:04:17.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T16:04:20.960 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-02-28T16:04:32.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-02-28T16:04:32.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T16:04:36.032 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-02-28T16:04:51.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-02-28T16:04:51.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T16:05:51.037 DEBUG:teuthology.orchestra.remote:timed out 2024-02-28T16:06:09.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-02-28T16:06:09.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T16:06:09.510 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-02-28T16:06:10.196 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-02-28T16:06:10.196 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "6.8.0-rc6-gbd7d00fb7248"... 2024-02-28T16:06:10.197 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-02-28T16:06:10.253 INFO:teuthology.orchestra.run.smithi066.stdout:6.8.0-rc6-gbd7d00fb7248 2024-02-28T16:06:10.253 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gbd7d00fb7248 vs 6.8.0-rc6-gbd7d00fb7248 2024-02-28T16:06:10.253 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-28T16:06:10.254 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-28T16:06:11.254 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-28T16:06:11.255 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-28T16:06:11.523 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2024-02-28T16:06:11.594 DEBUG:teuthology.parallel:result is None 2024-02-28T16:06:11.594 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-28T16:06:11.602 INFO:teuthology.task.internal:Creating test directory... 2024-02-28T16:06:11.602 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-28T16:06:11.611 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-28T16:06:11.672 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-28T16:06:11.686 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-28T16:06:11.692 INFO:teuthology.task.internal:Creating archive directory... 2024-02-28T16:06:11.693 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-28T16:06:11.700 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-28T16:06:11.774 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-28T16:06:11.788 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-28T16:06:11.789 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-28T16:06:11.798 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-28T16:06:11.906 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-28T16:06:11.913 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-28T16:06:11.949 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-28T16:06:11.962 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-28T16:06:11.967 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-28T16:06:11.975 INFO:teuthology.task.internal:Configuring sudo... 2024-02-28T16:06:11.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-28T16:06:11.983 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-28T16:06:12.091 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-28T16:06:12.101 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-28T16:06:12.102 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-28T16:06:12.109 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-28T16:06:12.169 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-28T16:06:12.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-28T16:06:12.380 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-28T16:06:12.466 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-28T16:06:12.570 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:06:12.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-28T16:06:12.673 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-28T16:06:12.761 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-28T16:06:12.891 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-28T16:06:12.978 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-28T16:06:13.083 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:06:13.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-28T16:06:13.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-02-28T16:06:13.192 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2024-02-28T16:06:13.322 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-28T16:06:13.368 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-28T16:06:13.846 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-28T16:06:13.854 INFO:teuthology.task.internal:Starting timer... 2024-02-28T16:06:13.854 INFO:teuthology.run_tasks:Running task pcp... 2024-02-28T16:06:13.874 INFO:teuthology.run_tasks:Running task selinux... 2024-02-28T16:06:13.882 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-28T16:06:13.883 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-02-28T16:06:14.013 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-02-28T16:06:14.020 DEBUG:teuthology.orchestra.run.smithi066:> sudo service auditd rotate 2024-02-28T16:06:14.175 INFO:teuthology.orchestra.run.smithi066.stdout:Rotating logs: 2024-02-28T16:06:14.180 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-28T16:06:14.181 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-02-28T16:06:14.247 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-02-28T16:06:14.247 DEBUG:teuthology.orchestra.run.smithi066:> /usr/sbin/getenforce 2024-02-28T16:06:14.302 INFO:teuthology.orchestra.run.smithi066.stdout:Permissive 2024-02-28T16:06:14.303 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi066.front.sepia.ceph.com': 'permissive'} 2024-02-28T16:06:14.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-28T16:06:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T16:06:14.391 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-28T16:06:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T16:06:14.476 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-28T16:06:14.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-02-28T16:06:14.568 DEBUG:teuthology.orchestra.run.smithi066:> sudo /usr/sbin/setenforce permissive 2024-02-28T16:06:14.676 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-28T16:06:14.686 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-28T16:06:14.710 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-28T16:06:14.711 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi066.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-28T16:12:46.833 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi066.front.sepia.ceph.com')] 2024-02-28T16:12:46.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-02-28T16:12:46.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T16:12:46.990 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-02-28T16:12:47.146 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-02-28T16:12:47.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-02-28T16:12:47.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-28T16:12:47.306 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-02-28T16:12:47.465 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-02-28T16:12:47.465 INFO:teuthology.run_tasks:Running task clock... 2024-02-28T16:12:47.476 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-28T16:12:47.476 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-28T16:12:47.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-28T16:12:47.484 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-28T16:12:47.484 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-28T16:12:47.588 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-28T16:12:47.589 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-28T16:12:47.644 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-28T16:12:47.645 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-28T16:12:47.746 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-02-28T16:12:47.758 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: ntpd: command not found 2024-02-28T16:12:47.790 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-02-28T16:12:47.809 INFO:teuthology.orchestra.run.smithi066.stdout:506 Cannot talk to daemon 2024-02-28T16:12:47.844 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-28T16:12:47.872 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-28T16:12:47.902 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-28T16:12:47.934 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-28T16:12:48.039 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-02-28T16:12:48.048 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-28T16:12:48.049 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-02-28T16:12:48.049 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T16:12:48.049 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T16:12:48.049 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T16:12:48.049 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T16:12:48.081 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-02-28T16:12:48.090 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-28T16:12:48.090 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-02-28T16:12:48.090 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T16:12:48.090 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T16:12:48.090 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T16:12:48.090 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T16:12:48.091 INFO:teuthology.run_tasks:Running task install... 2024-02-28T16:12:48.099 DEBUG:teuthology.task.install:project ceph 2024-02-28T16:12:48.100 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}} 2024-02-28T16:12:48.100 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'} 2024-02-28T16:12:48.100 INFO:teuthology.task.install:Using flavor: default 2024-02-28T16:12:48.108 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-28T16:12:48.108 INFO:teuthology.task.install:extra packages: [] 2024-02-28T16:12:48.109 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'tag': None, 'wait_for_package': False} 2024-02-28T16:12:48.109 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-28T16:12:48.109 INFO:teuthology.packaging:ref: None 2024-02-28T16:12:48.109 INFO:teuthology.packaging:tag: None 2024-02-28T16:12:48.109 INFO:teuthology.packaging:branch: quincy 2024-02-28T16:12:48.109 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:12:48.110 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-28T16:12:48.112 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'tag': None, 'wait_for_package': False} 2024-02-28T16:12:48.112 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-28T16:12:48.112 INFO:teuthology.packaging:ref: None 2024-02-28T16:12:48.112 INFO:teuthology.packaging:tag: None 2024-02-28T16:12:48.112 INFO:teuthology.packaging:branch: quincy 2024-02-28T16:12:48.112 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:12:48.112 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-28T16:12:48.433 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/c0f02e608ed955f887728ffa27c686f52f9fd400/centos/9/flavors/default/ 2024-02-28T16:12:48.433 INFO:teuthology.task.install.rpm:Package version is 17.2.7-535.gc0f02e60 2024-02-28T16:12:48.452 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/c0f02e608ed955f887728ffa27c686f52f9fd400/centos/9/flavors/default/ 2024-02-28T16:12:48.452 INFO:teuthology.task.install.rpm:Package version is 17.2.7-535.gc0f02e60 2024-02-28T16:12:48.682 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/c0f02e608ed955f887728ffa27c686f52f9fd400/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/c0f02e608ed955f887728ffa27c686f52f9fd400/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/c0f02e608ed955f887728ffa27c686f52f9fd400/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-28T16:12:48.682 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:12:48.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-28T16:12:48.782 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-28T16:12:48.782 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-28T16:12:48.782 INFO:teuthology.packaging:ref: None 2024-02-28T16:12:48.782 INFO:teuthology.packaging:tag: None 2024-02-28T16:12:48.782 INFO:teuthology.packaging:branch: quincy 2024-02-28T16:12:48.782 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:12:48.782 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-28T16:12:48.841 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/c0f02e608ed955f887728ffa27c686f52f9fd400/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/c0f02e608ed955f887728ffa27c686f52f9fd400/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/c0f02e608ed955f887728ffa27c686f52f9fd400/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-28T16:12:48.841 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:12:48.841 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-28T16:12:48.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-28T16:12:48.953 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-28T16:12:48.953 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-28T16:12:48.953 INFO:teuthology.packaging:ref: None 2024-02-28T16:12:48.953 INFO:teuthology.packaging:tag: None 2024-02-28T16:12:48.953 INFO:teuthology.packaging:branch: quincy 2024-02-28T16:12:48.953 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:12:48.954 DEBUG:teuthology.orchestra.run.smithi066:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-28T16:12:49.069 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-28T16:12:49.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-28T16:12:49.161 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-02-28T16:12:49.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-02-28T16:12:49.254 DEBUG:teuthology.orchestra.run.smithi066:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-28T16:12:49.357 INFO:teuthology.orchestra.run.smithi066.stdout:check_obsoletes = 1 2024-02-28T16:12:49.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum clean all 2024-02-28T16:12:49.522 INFO:teuthology.orchestra.run.smithi001.stdout:74 files removed 2024-02-28T16:12:49.563 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw 2024-02-28T16:12:49.751 INFO:teuthology.orchestra.run.smithi066.stdout:74 files removed 2024-02-28T16:12:49.796 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-radosgw 2024-02-28T16:12:50.733 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 117 kB/s | 77 kB 00:00 2024-02-28T16:12:50.928 INFO:teuthology.orchestra.run.smithi066.stdout:ceph packages for x86_64 131 kB/s | 77 kB 00:00 2024-02-28T16:12:51.399 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 16 kB/s | 9.2 kB 00:00 2024-02-28T16:12:51.412 INFO:teuthology.orchestra.run.smithi066.stdout:ceph noarch packages 24 kB/s | 9.2 kB 00:00 2024-02-28T16:12:51.979 INFO:teuthology.orchestra.run.smithi066.stdout:ceph source packages 3.6 kB/s | 1.8 kB 00:00 2024-02-28T16:12:51.983 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 3.5 kB/s | 1.8 kB 00:00 2024-02-28T16:12:52.732 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 12 MB/s | 8.1 MB 00:00 2024-02-28T16:12:52.788 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 11 MB/s | 8.1 MB 00:00 2024-02-28T16:12:54.738 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.5 MB 00:00 2024-02-28T16:12:54.784 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.5 MB 00:00 2024-02-28T16:12:57.392 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-02-28T16:12:58.204 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.0 MB 00:02 2024-02-28T16:13:00.035 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-02-28T16:13:00.890 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-28T16:13:05.923 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.1 MB 00:00 2024-02-28T16:13:07.574 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 15 kB 00:00 2024-02-28T16:13:09.356 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2024-02-28T16:13:15.500 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-02-28T16:13:18.244 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:13:18.244 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:13:18.244 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:13:18.244 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.245 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.246 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.247 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.248 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.249 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.250 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.251 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.252 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.253 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:13:18.253 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.253 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.253 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.253 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:13:18.934 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:13:18.944 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:13:18.944 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:13:18.944 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:13:18.944 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:13:18.944 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 14 M 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrading: 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: librados2 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 3.1 M 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 2.9 M 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 5.0 M 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 23 M 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 25 k 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 636 k 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-02-28T16:13:18.945 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 458 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 4.9 M 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 45 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 116 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 156 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 320 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 293 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 99 k 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:18.946 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:13:18.947 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:13:18.947 INFO:teuthology.orchestra.run.smithi066.stdout:Install 22 Packages 2024-02-28T16:13:18.947 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrade 2 Packages 2024-02-28T16:13:18.947 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:18.947 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 58 M 2024-02-28T16:13:18.948 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:13:20.293 INFO:teuthology.orchestra.run.smithi066.stdout:(1/24): ceph-base-17.2.7-535.gc0f02e60.el9.x86_ 4.5 MB/s | 5.0 MB 00:01 2024-02-28T16:13:20.519 INFO:teuthology.orchestra.run.smithi066.stdout:(2/24): ceph-selinux-17.2.7-535.gc0f02e60.el9.x 112 kB/s | 25 kB 00:00 2024-02-28T16:13:20.769 INFO:teuthology.orchestra.run.smithi066.stdout:(3/24): libcephfs2-17.2.7-535.gc0f02e60.el9.x86 2.5 MB/s | 636 kB 00:00 2024-02-28T16:13:20.937 INFO:teuthology.orchestra.run.smithi066.stdout:(4/24): libradosstriper1-17.2.7-535.gc0f02e60.e 2.7 MB/s | 458 kB 00:00 2024-02-28T16:13:22.003 INFO:teuthology.orchestra.run.smithi066.stdout:(5/24): librgw2-17.2.7-535.gc0f02e60.el9.x86_64 4.6 MB/s | 4.9 MB 00:01 2024-02-28T16:13:22.236 INFO:teuthology.orchestra.run.smithi066.stdout:(6/24): ceph-radosgw-17.2.7-535.gc0f02e60.el9.x 4.6 MB/s | 14 MB 00:03 2024-02-28T16:13:22.262 INFO:teuthology.orchestra.run.smithi066.stdout:(7/24): python3-ceph-argparse-17.2.7-535.gc0f02 175 kB/s | 45 kB 00:00 2024-02-28T16:13:22.337 INFO:teuthology.orchestra.run.smithi066.stdout:(8/24): python3-ceph-common-17.2.7-535.gc0f02e6 1.1 MB/s | 116 kB 00:00 2024-02-28T16:13:22.370 INFO:teuthology.orchestra.run.smithi066.stdout:(9/24): python3-cephfs-17.2.7-535.gc0f02e60.el9 1.4 MB/s | 156 kB 00:00 2024-02-28T16:13:22.462 INFO:teuthology.orchestra.run.smithi066.stdout:(10/24): python3-rados-17.2.7-535.gc0f02e60.el9 2.5 MB/s | 320 kB 00:00 2024-02-28T16:13:22.496 INFO:teuthology.orchestra.run.smithi066.stdout:(11/24): python3-rbd-17.2.7-535.gc0f02e60.el9.x 2.3 MB/s | 293 kB 00:00 2024-02-28T16:13:22.579 INFO:teuthology.orchestra.run.smithi066.stdout:(12/24): python3-rgw-17.2.7-535.gc0f02e60.el9.x 856 kB/s | 99 kB 00:00 2024-02-28T16:13:22.596 INFO:teuthology.orchestra.run.smithi066.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-02-28T16:13:22.888 INFO:teuthology.orchestra.run.smithi066.stdout:(14/24): ceph-common-17.2.7-535.gc0f02e60.el9.x 6.1 MB/s | 23 MB 00:03 2024-02-28T16:13:22.913 INFO:teuthology.orchestra.run.smithi066.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 136 kB/s | 45 kB 00:00 2024-02-28T16:13:22.989 INFO:teuthology.orchestra.run.smithi066.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 569 kB/s | 42 kB 00:00 2024-02-28T16:13:23.022 INFO:teuthology.orchestra.run.smithi066.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 292 kB 00:00 2024-02-28T16:13:23.056 INFO:teuthology.orchestra.run.smithi066.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.4 MB/s | 662 kB 00:00 2024-02-28T16:13:23.390 INFO:teuthology.orchestra.run.smithi066.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 133 kB/s | 49 kB 00:00 2024-02-28T16:13:23.465 INFO:teuthology.orchestra.run.smithi066.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 165 kB/s | 67 kB 00:00 2024-02-28T16:13:23.498 INFO:teuthology.orchestra.run.smithi066.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 608 kB/s | 309 kB 00:00 2024-02-28T16:13:23.782 INFO:teuthology.orchestra.run.smithi066.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-02-28T16:13:24.660 INFO:teuthology.orchestra.run.smithi066.stdout:(23/24): librbd1-17.2.7-535.gc0f02e60.el9.x86_6 2.5 MB/s | 2.9 MB 00:01 2024-02-28T16:13:24.743 INFO:teuthology.orchestra.run.smithi066.stdout:(24/24): librados2-17.2.7-535.gc0f02e60.el9.x86 2.4 MB/s | 3.1 MB 00:01 2024-02-28T16:13:24.754 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:13:24.755 INFO:teuthology.orchestra.run.smithi066.stdout:Total 10 MB/s | 58 MB 00:05 2024-02-28T16:13:24.946 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:13:25.029 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:13:25.029 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:13:25.574 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:13:25.574 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:13:27.009 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:13:27.407 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-28T16:13:27.461 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 2/26 2024-02-28T16:13:27.630 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 2/26 2024-02-28T16:13:27.726 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rados-2:17.2.7-535.gc0f02e60.el9.x86_64 3/26 2024-02-28T16:13:27.755 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 4/26 2024-02-28T16:13:27.893 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 4/26 2024-02-28T16:13:27.986 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-28T16:13:28.022 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-28T16:13:28.580 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-28T16:13:28.616 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librgw2-2:17.2.7-535.gc0f02e60.el9.x86_64 8/26 2024-02-28T16:13:28.699 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librgw2-2:17.2.7-535.gc0f02e60.el9.x86_64 8/26 2024-02-28T16:13:28.739 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-28T16:13:28.809 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-argparse-2:17.2.7-535.gc0f02e60.el9.x 10/26 2024-02-28T16:13:28.853 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cephfs-2:17.2.7-535.gc0f02e60.el9.x86_64 11/26 2024-02-28T16:13:28.981 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rgw-2:17.2.7-535.gc0f02e60.el9.x86_64 12/26 2024-02-28T16:13:29.010 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libradosstriper1-2:17.2.7-535.gc0f02e60.el9.x86_64 13/26 2024-02-28T16:13:29.121 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libradosstriper1-2:17.2.7-535.gc0f02e60.el9.x86_64 13/26 2024-02-28T16:13:29.199 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-28T16:13:29.265 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-28T16:13:29.314 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-28T16:13:29.714 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-28T16:13:29.739 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 18/26 2024-02-28T16:13:29.866 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 18/26 2024-02-28T16:13:29.984 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rbd-2:17.2.7-535.gc0f02e60.el9.x86_64 19/26 2024-02-28T16:13:30.017 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-common-2:17.2.7-535.gc0f02e60.el9.x86 20/26 2024-02-28T16:13:32.976 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 21/26 2024-02-28T16:13:33.013 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 21/26 2024-02-28T16:13:33.679 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 21/26 2024-02-28T16:13:33.717 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 22/26 2024-02-28T16:13:33.934 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 22/26 2024-02-28T16:13:33.935 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-28T16:13:33.935 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-28T16:13:33.935 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:33.976 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-selinux-2:17.2.7-535.gc0f02e60.el9.x86_64 23/26 2024-02-28T16:13:45.531 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-selinux-2:17.2.7-535.gc0f02e60.el9.x86_64 23/26 2024-02-28T16:13:45.532 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /sys 2024-02-28T16:13:45.532 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /proc 2024-02-28T16:13:45.532 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /mnt 2024-02-28T16:13:45.532 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /var/tmp 2024-02-28T16:13:45.532 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /home 2024-02-28T16:13:45.532 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /root 2024-02-28T16:13:45.532 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /tmp 2024-02-28T16:13:45.532 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:47.003 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-radosgw-2:17.2.7-535.gc0f02e60.el9.x86_64 24/26 2024-02-28T16:13:47.156 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-535.gc0f02e60.el9.x86_64 24/26 2024-02-28T16:13:47.157 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:13:47.157 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-28T16:13:47.157 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-28T16:13:47.157 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-28T16:13:47.157 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:47.162 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-28T16:13:47.275 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-28T16:13:47.275 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-28T16:13:48.920 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 1/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 2/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-radosgw-2:17.2.7-535.gc0f02e60.el9.x86_64 3/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-selinux-2:17.2.7-535.gc0f02e60.el9.x86_64 4/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 5/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libradosstriper1-2:17.2.7-535.gc0f02e60.el9.x86_64 6/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librgw2-2:17.2.7-535.gc0f02e60.el9.x86_64 7/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-argparse-2:17.2.7-535.gc0f02e60.el9.x 8/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-common-2:17.2.7-535.gc0f02e60.el9.x86 9/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cephfs-2:17.2.7-535.gc0f02e60.el9.x86_64 10/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rados-2:17.2.7-535.gc0f02e60.el9.x86_64 11/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rbd-2:17.2.7-535.gc0f02e60.el9.x86_64 12/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rgw-2:17.2.7-535.gc0f02e60.el9.x86_64 13/26 2024-02-28T16:13:48.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 23/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-28T16:13:48.922 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 25/26 2024-02-28T16:13:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-28T16:13:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:50.015 INFO:teuthology.orchestra.run.smithi066.stdout:Upgraded: 2024-02-28T16:13:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.015 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.016 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:50.017 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:13:50.478 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-test 2024-02-28T16:13:51.410 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:36 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:13:51.734 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:13:51.734 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:13:51.734 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:13:51.734 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:13:51.735 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.736 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.737 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.738 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.739 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.740 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.741 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.742 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.743 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:13:51.809 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 48 M 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 162 k 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-02-28T16:13:51.810 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-02-28T16:13:51.811 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:51.811 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:13:51.811 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:13:51.811 INFO:teuthology.orchestra.run.smithi066.stdout:Install 6 Packages 2024-02-28T16:13:51.811 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:13:51.811 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 49 M 2024-02-28T16:13:51.811 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 196 M 2024-02-28T16:13:51.812 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:13:52.692 INFO:teuthology.orchestra.run.smithi066.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 457 kB/s | 187 kB 00:00 2024-02-28T16:13:52.817 INFO:teuthology.orchestra.run.smithi066.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.7 MB/s | 218 kB 00:00 2024-02-28T16:13:52.892 INFO:teuthology.orchestra.run.smithi066.stdout:(3/6): libcephsqlite-17.2.7-535.gc0f02e60.el9.x 266 kB/s | 162 kB 00:00 2024-02-28T16:13:53.001 INFO:teuthology.orchestra.run.smithi066.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 593 kB/s | 64 kB 00:00 2024-02-28T16:13:53.034 INFO:teuthology.orchestra.run.smithi066.stdout:(5/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 305 kB 00:00 2024-02-28T16:13:53.297 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 128 kB/s | 6.1 MB 00:49 2024-02-28T16:13:54.832 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 15 kB 00:00 2024-02-28T16:13:56.059 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 20 MB/s | 20 MB 00:01 2024-02-28T16:14:00.126 INFO:teuthology.orchestra.run.smithi066.stdout:(6/6): ceph-test-17.2.7-535.gc0f02e60.el9.x86_6 6.1 MB/s | 48 MB 00:07 2024-02-28T16:14:00.130 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:14:00.131 INFO:teuthology.orchestra.run.smithi066.stdout:Total 5.9 MB/s | 49 MB 00:08 2024-02-28T16:14:00.288 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:14:00.346 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:14:00.346 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:14:00.839 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:14:00.839 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:14:01.535 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:14:01.621 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-02-28T16:14:01.665 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-02-28T16:14:01.705 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-02-28T16:14:01.805 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-02-28T16:14:01.829 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephsqlite-2:17.2.7-535.gc0f02e60.el9.x86_64 5/6 2024-02-28T16:14:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:14:04.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:14:04.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.928 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.929 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.930 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.931 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.932 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:04.933 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:05.594 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:14:05.616 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 14 M 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 3.1 M 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 2.9 M 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 5.0 M 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 23 M 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 25 k 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 636 k 2024-02-28T16:14:05.617 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 458 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 4.9 M 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 45 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 116 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 156 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-02-28T16:14:05.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 320 k 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 293 k 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 99 k 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout:Install 22 Packages 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 2 Packages 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:05.619 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 58 M 2024-02-28T16:14:05.620 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:14:06.946 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephsqlite-2:17.2.7-535.gc0f02e60.el9.x86_64 5/6 2024-02-28T16:14:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-test-2:17.2.7-535.gc0f02e60.el9.x86_64 6/6 2024-02-28T16:14:07.501 INFO:teuthology.orchestra.run.smithi001.stdout:(1/24): ceph-base-17.2.7-535.gc0f02e60.el9.x86_ 3.6 MB/s | 5.0 MB 00:01 2024-02-28T16:14:07.551 INFO:teuthology.orchestra.run.smithi001.stdout:(2/24): ceph-selinux-17.2.7-535.gc0f02e60.el9.x 516 kB/s | 25 kB 00:00 2024-02-28T16:14:07.718 INFO:teuthology.orchestra.run.smithi001.stdout:(3/24): libcephfs2-17.2.7-535.gc0f02e60.el9.x86 3.7 MB/s | 636 kB 00:00 2024-02-28T16:14:07.868 INFO:teuthology.orchestra.run.smithi001.stdout:(4/24): libradosstriper1-17.2.7-535.gc0f02e60.e 3.0 MB/s | 458 kB 00:00 2024-02-28T16:14:08.744 INFO:teuthology.orchestra.run.smithi001.stdout:(5/24): librgw2-17.2.7-535.gc0f02e60.el9.x86_64 5.6 MB/s | 4.9 MB 00:00 2024-02-28T16:14:08.794 INFO:teuthology.orchestra.run.smithi001.stdout:(6/24): python3-ceph-argparse-17.2.7-535.gc0f02 921 kB/s | 45 kB 00:00 2024-02-28T16:14:08.853 INFO:teuthology.orchestra.run.smithi001.stdout:(7/24): python3-ceph-common-17.2.7-535.gc0f02e6 2.0 MB/s | 116 kB 00:00 2024-02-28T16:14:08.928 INFO:teuthology.orchestra.run.smithi001.stdout:(8/24): python3-cephfs-17.2.7-535.gc0f02e60.el9 2.1 MB/s | 156 kB 00:00 2024-02-28T16:14:09.170 INFO:teuthology.orchestra.run.smithi001.stdout:(9/24): ceph-radosgw-17.2.7-535.gc0f02e60.el9.x 4.6 MB/s | 14 MB 00:03 2024-02-28T16:14:09.211 INFO:teuthology.orchestra.run.smithi001.stdout:(10/24): python3-rados-17.2.7-535.gc0f02e60.el9 1.1 MB/s | 320 kB 00:00 2024-02-28T16:14:09.245 INFO:teuthology.orchestra.run.smithi001.stdout:(11/24): python3-rbd-17.2.7-535.gc0f02e60.el9.x 3.9 MB/s | 293 kB 00:00 2024-02-28T16:14:09.287 INFO:teuthology.orchestra.run.smithi001.stdout:(12/24): python3-rgw-17.2.7-535.gc0f02e60.el9.x 1.3 MB/s | 99 kB 00:00 2024-02-28T16:14:09.437 INFO:teuthology.orchestra.run.smithi001.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 303 kB/s | 45 kB 00:00 2024-02-28T16:14:09.462 INFO:teuthology.orchestra.run.smithi001.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 740 kB/s | 160 kB 00:00 2024-02-28T16:14:09.597 INFO:teuthology.orchestra.run.smithi001.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-02-28T16:14:09.597 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-test-2:17.2.7-535.gc0f02e60.el9.x86_64 6/6 2024-02-28T16:14:09.597 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-test-2:17.2.7-535.gc0f02e60.el9.x86_64 1/6 2024-02-28T16:14:09.597 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephsqlite-2:17.2.7-535.gc0f02e60.el9.x86_64 2/6 2024-02-28T16:14:09.598 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-02-28T16:14:09.598 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-02-28T16:14:09.598 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-02-28T16:14:09.638 INFO:teuthology.orchestra.run.smithi001.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-02-28T16:14:09.663 INFO:teuthology.orchestra.run.smithi001.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 640 kB/s | 42 kB 00:00 2024-02-28T16:14:09.981 INFO:teuthology.orchestra.run.smithi001.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 154 kB/s | 49 kB 00:00 2024-02-28T16:14:09.986 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout: jq-1.6-16.el9.x86_64 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:09.987 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:14:10.232 INFO:teuthology.orchestra.run.smithi001.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 521 kB/s | 309 kB 00:00 2024-02-28T16:14:10.264 INFO:teuthology.orchestra.run.smithi001.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 238 kB/s | 67 kB 00:00 2024-02-28T16:14:10.357 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph 2024-02-28T16:14:10.581 INFO:teuthology.orchestra.run.smithi001.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2024-02-28T16:14:10.932 INFO:teuthology.orchestra.run.smithi001.stdout:(22/24): ceph-common-17.2.7-535.gc0f02e60.el9.x 4.7 MB/s | 23 MB 00:04 2024-02-28T16:14:11.040 INFO:teuthology.orchestra.run.smithi001.stdout:(23/24): librados2-17.2.7-535.gc0f02e60.el9.x86 4.0 MB/s | 3.1 MB 00:00 2024-02-28T16:14:11.165 INFO:teuthology.orchestra.run.smithi001.stdout:(24/24): librbd1-17.2.7-535.gc0f02e60.el9.x86_6 5.0 MB/s | 2.9 MB 00:00 2024-02-28T16:14:11.171 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:14:11.172 INFO:teuthology.orchestra.run.smithi001.stdout:Total 10 MB/s | 58 MB 00:05 2024-02-28T16:14:11.275 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:56 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:14:11.370 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:14:11.452 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:14:11.452 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.595 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.596 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.597 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:14:11.598 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.599 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:14:11.600 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.601 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:11.602 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.603 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.604 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:11.604 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.604 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.604 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.604 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:11.676 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:14:11.679 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:11.679 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:14:11.679 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:11.679 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:14:11.679 INFO:teuthology.orchestra.run.smithi066.stdout: ceph x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 6.5 k 2024-02-28T16:14:11.679 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-02-28T16:14:11.679 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 2.1 M 2024-02-28T16:14:11.679 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 1.4 M 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 239 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 3.7 M 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 15 M 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-28T16:14:11.680 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-02-28T16:14:11.681 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout:Install 33 Packages 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 27 M 2024-02-28T16:14:11.682 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 99 M 2024-02-28T16:14:11.683 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:14:11.991 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:14:11.992 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:14:12.545 INFO:teuthology.orchestra.run.smithi066.stdout:(1/33): ceph-17.2.7-535.gc0f02e60.el9.x86_64.rp 17 kB/s | 6.5 kB 00:00 2024-02-28T16:14:13.514 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:14:13.925 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-28T16:14:14.009 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 2/26 2024-02-28T16:14:14.020 INFO:teuthology.orchestra.run.smithi066.stdout:(2/33): ceph-mgr-17.2.7-535.gc0f02e60.el9.x86_6 772 kB/s | 1.4 MB 00:01 2024-02-28T16:14:14.123 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 2/26 2024-02-28T16:14:14.137 INFO:teuthology.orchestra.run.smithi066.stdout:(3/33): ceph-mon-17.2.7-535.gc0f02e60.el9.x86_6 2.3 MB/s | 3.7 MB 00:01 2024-02-28T16:14:14.196 INFO:teuthology.orchestra.run.smithi066.stdout:(4/33): ceph-mds-17.2.7-535.gc0f02e60.el9.x86_6 1.0 MB/s | 2.1 MB 00:02 2024-02-28T16:14:14.233 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:17.2.7-535.gc0f02e60.el9.x86_64 3/26 2024-02-28T16:14:14.260 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 4/26 2024-02-28T16:14:14.279 INFO:teuthology.orchestra.run.smithi066.stdout:(5/33): ceph-mgr-modules-core-17.2.7-535.gc0f02 1.7 MB/s | 239 kB 00:00 2024-02-28T16:14:14.391 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 4/26 2024-02-28T16:14:14.469 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-28T16:14:14.535 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-28T16:14:14.580 INFO:teuthology.orchestra.run.smithi066.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 662 kB/s | 253 kB 00:00 2024-02-28T16:14:14.738 INFO:teuthology.orchestra.run.smithi066.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 676 kB/s | 106 kB 00:00 2024-02-28T16:14:15.006 INFO:teuthology.orchestra.run.smithi066.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 508 kB/s | 135 kB 00:00 2024-02-28T16:14:15.087 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-28T16:14:15.181 INFO:teuthology.orchestra.run.smithi066.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 716 kB/s | 125 kB 00:00 2024-02-28T16:14:15.221 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:17.2.7-535.gc0f02e60.el9.x86_64 8/26 2024-02-28T16:14:15.306 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:17.2.7-535.gc0f02e60.el9.x86_64 8/26 2024-02-28T16:14:15.352 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-28T16:14:15.442 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:17.2.7-535.gc0f02e60.el9.x 10/26 2024-02-28T16:14:15.464 INFO:teuthology.orchestra.run.smithi066.stdout:(10/33): python3-urllib3-1.26.5-5.el9.noarch.rp 760 kB/s | 215 kB 00:00 2024-02-28T16:14:15.515 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:17.2.7-535.gc0f02e60.el9.x86_64 11/26 2024-02-28T16:14:15.625 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:17.2.7-535.gc0f02e60.el9.x86_64 12/26 2024-02-28T16:14:15.673 INFO:teuthology.orchestra.run.smithi066.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 831 kB/s | 172 kB 00:00 2024-02-28T16:14:15.683 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:17.2.7-535.gc0f02e60.el9.x86_64 13/26 2024-02-28T16:14:15.740 INFO:teuthology.orchestra.run.smithi066.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 525 kB/s | 35 kB 00:00 2024-02-28T16:14:15.778 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:17.2.7-535.gc0f02e60.el9.x86_64 13/26 2024-02-28T16:14:15.807 INFO:teuthology.orchestra.run.smithi066.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 775 kB/s | 51 kB 00:00 2024-02-28T16:14:15.855 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-28T16:14:15.874 INFO:teuthology.orchestra.run.smithi066.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 633 kB/s | 42 kB 00:00 2024-02-28T16:14:15.913 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-28T16:14:15.963 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-28T16:14:16.050 INFO:teuthology.orchestra.run.smithi066.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 248 kB/s | 43 kB 00:00 2024-02-28T16:14:16.191 INFO:teuthology.orchestra.run.smithi066.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-02-28T16:14:16.325 INFO:teuthology.orchestra.run.smithi066.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-02-28T16:14:16.361 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-28T16:14:16.375 INFO:teuthology.orchestra.run.smithi066.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 217 kB/s | 11 kB 00:00 2024-02-28T16:14:16.395 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 18/26 2024-02-28T16:14:16.425 INFO:teuthology.orchestra.run.smithi066.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 358 kB/s | 18 kB 00:00 2024-02-28T16:14:16.475 INFO:teuthology.orchestra.run.smithi066.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 471 kB/s | 23 kB 00:00 2024-02-28T16:14:16.511 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 18/26 2024-02-28T16:14:16.567 INFO:teuthology.orchestra.run.smithi066.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 214 kB/s | 19 kB 00:00 2024-02-28T16:14:16.607 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:17.2.7-535.gc0f02e60.el9.x86_64 19/26 2024-02-28T16:14:16.639 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:17.2.7-535.gc0f02e60.el9.x86 20/26 2024-02-28T16:14:16.643 INFO:teuthology.orchestra.run.smithi066.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 265 kB/s | 20 kB 00:00 2024-02-28T16:14:16.693 INFO:teuthology.orchestra.run.smithi066.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 943 kB/s | 46 kB 00:00 2024-02-28T16:14:16.752 INFO:teuthology.orchestra.run.smithi066.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-02-28T16:14:16.827 INFO:teuthology.orchestra.run.smithi066.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-02-28T16:14:16.878 INFO:teuthology.orchestra.run.smithi066.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 334 kB/s | 16 kB 00:00 2024-02-28T16:14:16.968 INFO:teuthology.orchestra.run.smithi066.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-02-28T16:14:17.028 INFO:teuthology.orchestra.run.smithi066.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 602 kB/s | 36 kB 00:00 2024-02-28T16:14:17.103 INFO:teuthology.orchestra.run.smithi066.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-02-28T16:14:17.187 INFO:teuthology.orchestra.run.smithi066.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.1 MB/s | 427 kB 00:00 2024-02-28T16:14:17.246 INFO:teuthology.orchestra.run.smithi066.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 347 kB/s | 20 kB 00:00 2024-02-28T16:14:18.629 INFO:teuthology.orchestra.run.smithi066.stdout:(32/33): python3-cryptography-36.0.1-4.el9.x86_ 289 kB/s | 1.2 MB 00:04 2024-02-28T16:14:19.461 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 21/26 2024-02-28T16:14:19.502 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 21/26 2024-02-28T16:14:20.154 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 21/26 2024-02-28T16:14:20.214 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 22/26 2024-02-28T16:14:20.395 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 22/26 2024-02-28T16:14:20.395 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-28T16:14:20.395 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-28T16:14:20.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:20.439 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:17.2.7-535.gc0f02e60.el9.x86_64 23/26 2024-02-28T16:14:23.280 INFO:teuthology.orchestra.run.smithi066.stdout:(33/33): ceph-osd-17.2.7-535.gc0f02e60.el9.x86_ 1.7 MB/s | 15 MB 00:09 2024-02-28T16:14:23.285 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:14:23.286 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.3 MB/s | 27 MB 00:11 2024-02-28T16:14:23.668 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:14:23.760 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:14:23.760 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:14:24.182 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:14:24.183 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:14:25.825 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:14:25.917 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-28T16:14:25.942 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-28T16:14:25.983 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-28T16:14:26.127 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-28T16:14:26.310 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-28T16:14:26.366 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-28T16:14:26.407 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-28T16:14:26.444 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-02-28T16:14:26.480 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-02-28T16:14:26.524 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-02-28T16:14:26.616 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-02-28T16:14:26.684 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-02-28T16:14:26.727 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-02-28T16:14:26.784 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-02-28T16:14:26.942 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-02-28T16:14:27.151 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-02-28T16:14:27.259 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-02-28T16:14:27.327 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-02-28T16:14:27.377 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-02-28T16:14:27.445 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-02-28T16:14:27.519 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-02-28T16:14:27.805 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-02-28T16:14:27.865 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-02-28T16:14:27.999 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-02-28T16:14:28.121 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-02-28T16:14:28.167 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-02-28T16:14:28.348 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-02-28T16:14:28.527 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-535.gc0f02e60.el9.n 28/33 2024-02-28T16:14:28.586 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 29/33 2024-02-28T16:14:28.682 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 29/33 2024-02-28T16:14:28.682 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:14:28.682 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-28T16:14:28.682 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-28T16:14:28.682 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-28T16:14:28.682 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:30.287 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-osd-2:17.2.7-535.gc0f02e60.el9.x86_64 30/33 2024-02-28T16:14:30.381 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-osd-2:17.2.7-535.gc0f02e60.el9.x86_64 30/33 2024-02-28T16:14:30.381 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:14:30.381 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-28T16:14:30.381 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-28T16:14:30.381 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-28T16:14:30.382 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:30.838 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mon-2:17.2.7-535.gc0f02e60.el9.x86_64 31/33 2024-02-28T16:14:30.940 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mon-2:17.2.7-535.gc0f02e60.el9.x86_64 31/33 2024-02-28T16:14:30.940 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:14:30.940 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-28T16:14:30.940 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-28T16:14:30.940 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-28T16:14:30.940 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:31.180 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mds-2:17.2.7-535.gc0f02e60.el9.x86_64 32/33 2024-02-28T16:14:31.283 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mds-2:17.2.7-535.gc0f02e60.el9.x86_64 32/33 2024-02-28T16:14:31.283 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:14:31.283 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-28T16:14:31.283 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-28T16:14:31.283 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-28T16:14:31.283 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:31.364 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-2:17.2.7-535.gc0f02e60.el9.x86_64 33/33 2024-02-28T16:14:31.489 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-2:17.2.7-535.gc0f02e60.el9.x86_64 33/33 2024-02-28T16:14:31.489 INFO:teuthology.orchestra.run.smithi066.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-02-28T16:14:31.489 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:31.805 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:17.2.7-535.gc0f02e60.el9.x86_64 23/26 2024-02-28T16:14:31.806 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-02-28T16:14:31.806 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-02-28T16:14:31.806 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-02-28T16:14:31.806 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-02-28T16:14:31.806 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-02-28T16:14:31.806 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-02-28T16:14:31.806 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-02-28T16:14:31.806 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:33.262 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:17.2.7-535.gc0f02e60.el9.x86_64 24/26 2024-02-28T16:14:33.328 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-2:17.2.7-535.gc0f02e60.el9.x86_64 1/33 2024-02-28T16:14:33.328 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mds-2:17.2.7-535.gc0f02e60.el9.x86_64 2/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 3/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mon-2:17.2.7-535.gc0f02e60.el9.x86_64 4/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-osd-2:17.2.7-535.gc0f02e60.el9.x86_64 5/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-535.gc0f02e60.el9.n 6/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-02-28T16:14:33.329 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-28T16:14:33.330 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-28T16:14:33.331 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-28T16:14:33.331 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-28T16:14:33.331 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-28T16:14:33.331 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-28T16:14:33.331 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-28T16:14:33.331 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-28T16:14:33.331 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-28T16:14:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-535.gc0f02e60.el9.x86_64 24/26 2024-02-28T16:14:33.396 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:14:33.396 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-28T16:14:33.396 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-28T16:14:33.396 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-28T16:14:33.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:33.401 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-28T16:14:33.516 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-28T16:14:33.516 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-28T16:14:34.306 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-28T16:14:34.306 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:34.306 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:14:34.306 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:34.306 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-28T16:14:34.307 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply-3.11-14.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-28T16:14:34.308 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-28T16:14:34.309 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-28T16:14:34.309 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-28T16:14:34.309 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-28T16:14:34.309 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-28T16:14:34.309 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-28T16:14:34.309 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:34.309 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:14:34.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-base 2024-02-28T16:14:35.297 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-28T16:14:35.297 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 1/26 2024-02-28T16:14:35.297 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 2/26 2024-02-28T16:14:35.297 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:17.2.7-535.gc0f02e60.el9.x86_64 3/26 2024-02-28T16:14:35.297 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:17.2.7-535.gc0f02e60.el9.x86_64 4/26 2024-02-28T16:14:35.297 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 5/26 2024-02-28T16:14:35.297 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:17.2.7-535.gc0f02e60.el9.x86_64 6/26 2024-02-28T16:14:35.297 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:17.2.7-535.gc0f02e60.el9.x86_64 7/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:17.2.7-535.gc0f02e60.el9.x 8/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:17.2.7-535.gc0f02e60.el9.x86 9/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:17.2.7-535.gc0f02e60.el9.x86_64 10/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:17.2.7-535.gc0f02e60.el9.x86_64 11/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:17.2.7-535.gc0f02e60.el9.x86_64 12/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:17.2.7-535.gc0f02e60.el9.x86_64 13/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-28T16:14:35.298 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-28T16:14:35.299 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-28T16:14:35.299 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-28T16:14:35.299 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 23/26 2024-02-28T16:14:35.299 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-28T16:14:35.299 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 25/26 2024-02-28T16:14:35.604 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:20 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:14:35.939 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:14:35.940 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:14:35.941 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:14:35.942 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:14:35.943 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:14:35.944 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:14:35.945 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.946 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:35.947 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.948 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.949 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:35.950 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:14:36.017 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:14:36.019 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:14:36.019 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.063 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.064 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-28T16:14:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-28T16:14:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:36.065 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:14:36.099 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install cephadm 2024-02-28T16:14:36.449 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-test 2024-02-28T16:14:36.983 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:21 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:14:37.310 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:14:37.310 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.311 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.313 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.314 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.315 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:14:37.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.346 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:35 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:14:37.388 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 80 k 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-02-28T16:14:37.389 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:37.390 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 80 k 2024-02-28T16:14:37.390 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 364 k 2024-02-28T16:14:37.390 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:14:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:14:37.674 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:14:37.674 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:14:37.675 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:14:37.676 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.677 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.678 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.679 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.684 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:37.750 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:14:37.751 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:14:37.751 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:37.751 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:14:37.751 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 48 M 2024-02-28T16:14:37.751 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 162 k 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-02-28T16:14:37.752 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:37.753 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 49 M 2024-02-28T16:14:37.753 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 196 M 2024-02-28T16:14:37.753 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:14:38.104 INFO:teuthology.orchestra.run.smithi066.stdout:cephadm-17.2.7-535.gc0f02e60.el9.noarch.rpm 113 kB/s | 80 kB 00:00 2024-02-28T16:14:38.105 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:14:38.106 INFO:teuthology.orchestra.run.smithi066.stdout:Total 112 kB/s | 80 kB 00:00 2024-02-28T16:14:38.106 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:14:38.125 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:14:38.125 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:14:38.132 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:14:38.134 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:14:38.271 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:14:38.434 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 350 kB/s | 187 kB 00:00 2024-02-28T16:14:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 218 kB 00:00 2024-02-28T16:14:38.726 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): libcephsqlite-17.2.7-535.gc0f02e60.el9.x 197 kB/s | 162 kB 00:00 2024-02-28T16:14:38.826 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-02-28T16:14:38.956 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/1 2024-02-28T16:14:39.093 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 174 kB/s | 64 kB 00:00 2024-02-28T16:14:39.162 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/1 2024-02-28T16:14:39.707 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/1 2024-02-28T16:14:40.061 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/1 2024-02-28T16:14:40.061 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:40.061 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:14:40.061 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:14:40.062 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:40.062 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:14:40.382 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-immutable-object-cache 2024-02-28T16:14:41.286 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:26 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.611 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:14:41.612 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:14:41.613 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:14:41.614 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.615 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.616 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:41.617 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.618 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.619 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:41.684 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:14:41.685 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:41.685 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-02-28T16:14:41.685 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:41.685 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 137 k 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 137 k 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 379 k 2024-02-28T16:14:41.686 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:14:42.444 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-immutable-object-cache-17.2.7-535.gc0f02e6 181 kB/s | 137 kB 00:00 2024-02-28T16:14:42.446 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:14:42.446 INFO:teuthology.orchestra.run.smithi066.stdout:Total 180 kB/s | 137 kB 00:00 2024-02-28T16:14:42.447 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:14:42.474 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:14:42.475 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:14:42.579 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:14:42.579 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:14:42.776 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:14:42.853 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-535.gc0f02e60.e 1/1 2024-02-28T16:14:42.937 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-535.gc0f02e60.e 1/1 2024-02-28T16:14:42.937 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:14:42.937 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-28T16:14:42.937 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:44.332 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-535.gc0f02e60.e 1/1 2024-02-28T16:14:44.332 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:44.332 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:14:44.332 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:44.332 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:44.332 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:14:44.662 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr 2024-02-28T16:14:45.599 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:30 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:14:45.928 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:14:45.928 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:14:45.928 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.929 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.930 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:14:45.931 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.932 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:14:45.933 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.934 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:45.935 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.936 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.937 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:45.938 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:14:46.006 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:14:46.007 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:14:46.007 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:14:46.109 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-dashboard 2024-02-28T16:14:46.260 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): ceph-test-17.2.7-535.gc0f02e60.el9.x86_6 5.7 MB/s | 48 MB 00:08 2024-02-28T16:14:46.261 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:14:46.262 INFO:teuthology.orchestra.run.smithi001.stdout:Total 5.7 MB/s | 49 MB 00:08 2024-02-28T16:14:46.400 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:14:46.449 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:14:46.449 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:14:46.950 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:14:46.951 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:14:46.992 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:32 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:14:47.316 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:14:47.316 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:14:47.316 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:14:47.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.316 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:14:47.316 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:14:47.317 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:14:47.318 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:14:47.319 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.320 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.321 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:47.322 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.323 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.324 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:47.390 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:14:47.391 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 1.7 M 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 23 k 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 15 k 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout:Installing weak dependencies: 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-28T16:14:47.392 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:47.393 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:14:47.393 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:14:47.393 INFO:teuthology.orchestra.run.smithi066.stdout:Install 7 Packages 2024-02-28T16:14:47.393 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:47.393 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 2.0 M 2024-02-28T16:14:47.393 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 8.1 M 2024-02-28T16:14:47.393 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:14:47.671 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:14:47.759 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-02-28T16:14:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-02-28T16:14:47.848 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-02-28T16:14:47.912 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-02-28T16:14:47.946 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:17.2.7-535.gc0f02e60.el9.x86_64 5/6 2024-02-28T16:14:47.956 INFO:teuthology.orchestra.run.smithi066.stdout:(1/7): ceph-grafana-dashboards-17.2.7-535.gc0f0 58 kB/s | 23 kB 00:00 2024-02-28T16:14:48.081 INFO:teuthology.orchestra.run.smithi066.stdout:(2/7): ceph-prometheus-alerts-17.2.7-535.gc0f02 28 kB/s | 15 kB 00:00 2024-02-28T16:14:48.198 INFO:teuthology.orchestra.run.smithi066.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 38 kB/s | 9.0 kB 00:00 2024-02-28T16:14:48.257 INFO:teuthology.orchestra.run.smithi066.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 556 kB/s | 31 kB 00:00 2024-02-28T16:14:48.290 INFO:teuthology.orchestra.run.smithi066.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 199 kB/s | 41 kB 00:00 2024-02-28T16:14:48.382 INFO:teuthology.orchestra.run.smithi066.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-02-28T16:14:48.957 INFO:teuthology.orchestra.run.smithi066.stdout:(7/7): ceph-mgr-dashboard-17.2.7-535.gc0f02e60. 1.2 MB/s | 1.7 MB 00:01 2024-02-28T16:14:48.966 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:14:48.966 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-02-28T16:14:48.967 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:14:48.995 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:14:48.995 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:14:49.131 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:14:49.131 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:14:49.571 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:14:49.667 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-28T16:14:49.718 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-28T16:14:49.822 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-28T16:14:49.890 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-28T16:14:49.939 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-535.gc0f02e60.el9.no 5/7 2024-02-28T16:14:50.289 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-535.gc0f02e60.el9.n 6/7 2024-02-28T16:14:50.352 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-535.gc0f02e60.el9.noarch 7/7 2024-02-28T16:14:50.985 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-535.gc0f02e60.el9.noarch 7/7 2024-02-28T16:14:50.985 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-535.gc0f02e60.el9.n 1/7 2024-02-28T16:14:50.986 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-535.gc0f02e60.el9.noarch 2/7 2024-02-28T16:14:50.986 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-535.gc0f02e60.el9.no 3/7 2024-02-28T16:14:50.986 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-28T16:14:50.986 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-28T16:14:50.986 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-28T16:14:51.377 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-28T16:14:51.377 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:51.378 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:14:51.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-28T16:14:52.649 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:37 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:14:52.978 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.979 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.980 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.981 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.982 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:14:52.983 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.984 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:52.985 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.986 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:52.987 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:53.058 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:14:53.059 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-02-28T16:14:53.059 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:14:53.059 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-02-28T16:14:53.059 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:14:53.059 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 7.4 M 2024-02-28T16:14:53.059 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:53.060 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:14:53.061 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-02-28T16:14:53.061 INFO:teuthology.orchestra.run.smithi066.stdout:Install 12 Packages 2024-02-28T16:14:53.061 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:14:53.061 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 38 M 2024-02-28T16:14:53.061 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 209 M 2024-02-28T16:14:53.061 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:14:53.092 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:17.2.7-535.gc0f02e60.el9.x86_64 5/6 2024-02-28T16:14:53.180 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:17.2.7-535.gc0f02e60.el9.x86_64 6/6 2024-02-28T16:14:54.121 INFO:teuthology.orchestra.run.smithi066.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 218 kB/s | 195 kB 00:00 2024-02-28T16:14:54.339 INFO:teuthology.orchestra.run.smithi066.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 150 kB/s | 33 kB 00:00 2024-02-28T16:14:54.465 INFO:teuthology.orchestra.run.smithi066.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 648 kB/s | 803 kB 00:01 2024-02-28T16:14:54.698 INFO:teuthology.orchestra.run.smithi066.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.4 MB/s | 3.0 MB 00:00 2024-02-28T16:14:54.723 INFO:teuthology.orchestra.run.smithi066.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 71 kB/s | 18 kB 00:00 2024-02-28T16:14:54.792 INFO:teuthology.orchestra.run.smithi066.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 377 kB/s | 35 kB 00:00 2024-02-28T16:14:54.891 INFO:teuthology.orchestra.run.smithi066.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.5 MB/s | 245 kB 00:00 2024-02-28T16:14:55.121 INFO:teuthology.orchestra.run.smithi066.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-02-28T16:14:55.233 INFO:teuthology.orchestra.run.smithi066.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-02-28T16:14:55.266 INFO:teuthology.orchestra.run.smithi066.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.7 MB/s | 250 kB 00:00 2024-02-28T16:14:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-test-2:17.2.7-535.gc0f02e60.el9.x86_64 6/6 2024-02-28T16:14:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:17.2.7-535.gc0f02e60.el9.x86_64 1/6 2024-02-28T16:14:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:17.2.7-535.gc0f02e60.el9.x86_64 2/6 2024-02-28T16:14:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-02-28T16:14:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-02-28T16:14:55.335 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-02-28T16:14:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-02-28T16:14:55.748 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: jq-1.6-16.el9.x86_64 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:55.749 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:14:55.800 INFO:teuthology.orchestra.run.smithi066.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-02-28T16:14:56.086 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph 2024-02-28T16:14:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:54 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:14:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:14:57.308 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:14:57.308 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:14:57.308 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.308 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:14:57.308 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.309 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.310 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.311 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:14:57.312 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.313 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.314 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.315 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.316 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:14:57.391 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:14:57.394 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:57.394 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 6.5 k 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 2.1 M 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 1.4 M 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 239 k 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 3.7 M 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 15 M 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-02-28T16:14:57.395 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-02-28T16:14:57.396 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:14:57.397 INFO:teuthology.orchestra.run.smithi001.stdout:Install 33 Packages 2024-02-28T16:14:57.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:14:57.398 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 27 M 2024-02-28T16:14:57.398 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 99 M 2024-02-28T16:14:57.398 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:14:58.380 INFO:teuthology.orchestra.run.smithi001.stdout:(1/33): ceph-17.2.7-535.gc0f02e60.el9.x86_64.rp 14 kB/s | 6.5 kB 00:00 2024-02-28T16:14:58.500 INFO:teuthology.orchestra.run.smithi066.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 1.4 MB/s | 7.4 MB 00:05 2024-02-28T16:14:58.507 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:14:58.507 INFO:teuthology.orchestra.run.smithi066.stdout:Total 7.0 MB/s | 38 MB 00:05 2024-02-28T16:14:59.180 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:14:59.263 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:14:59.263 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:14:59.480 INFO:teuthology.orchestra.run.smithi001.stdout:(2/33): ceph-mgr-17.2.7-535.gc0f02e60.el9.x86_6 922 kB/s | 1.4 MB 00:01 2024-02-28T16:14:59.730 INFO:teuthology.orchestra.run.smithi001.stdout:(3/33): ceph-mds-17.2.7-535.gc0f02e60.el9.x86_6 1.1 MB/s | 2.1 MB 00:01 2024-02-28T16:14:59.876 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:14:59.877 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:00.064 INFO:teuthology.orchestra.run.smithi001.stdout:(4/33): ceph-mon-17.2.7-535.gc0f02e60.el9.x86_6 2.2 MB/s | 3.7 MB 00:01 2024-02-28T16:15:00.097 INFO:teuthology.orchestra.run.smithi001.stdout:(5/33): ceph-mgr-modules-core-17.2.7-535.gc0f02 652 kB/s | 239 kB 00:00 2024-02-28T16:15:00.423 INFO:teuthology.orchestra.run.smithi001.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 708 kB/s | 253 kB 00:00 2024-02-28T16:15:00.490 INFO:teuthology.orchestra.run.smithi001.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-02-28T16:15:00.557 INFO:teuthology.orchestra.run.smithi001.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 2.7 MB/s | 1.2 MB 00:00 2024-02-28T16:15:00.590 INFO:teuthology.orchestra.run.smithi001.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.3 MB/s | 135 kB 00:00 2024-02-28T16:15:00.670 INFO:teuthology.orchestra.run.smithi001.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-02-28T16:15:00.708 INFO:teuthology.orchestra.run.smithi001.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-02-28T16:15:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-02-28T16:15:00.900 INFO:teuthology.orchestra.run.smithi001.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 182 kB/s | 35 kB 00:00 2024-02-28T16:15:00.925 INFO:teuthology.orchestra.run.smithi001.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 678 kB/s | 172 kB 00:00 2024-02-28T16:15:00.959 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-02-28T16:15:00.967 INFO:teuthology.orchestra.run.smithi001.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 773 kB/s | 51 kB 00:00 2024-02-28T16:15:00.984 INFO:teuthology.orchestra.run.smithi001.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 723 kB/s | 42 kB 00:00 2024-02-28T16:15:01.134 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-02-28T16:15:01.151 INFO:teuthology.orchestra.run.smithi001.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 238 kB/s | 43 kB 00:00 2024-02-28T16:15:01.185 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-28T16:15:01.209 INFO:teuthology.orchestra.run.smithi001.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 765 kB/s | 172 kB 00:00 2024-02-28T16:15:01.251 INFO:teuthology.orchestra.run.smithi001.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 262 kB/s | 11 kB 00:00 2024-02-28T16:15:01.318 INFO:teuthology.orchestra.run.smithi001.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-02-28T16:15:01.335 INFO:teuthology.orchestra.run.smithi001.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 213 kB/s | 18 kB 00:00 2024-02-28T16:15:01.360 INFO:teuthology.orchestra.run.smithi001.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 566 kB/s | 23 kB 00:00 2024-02-28T16:15:01.377 INFO:teuthology.orchestra.run.smithi001.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 472 kB/s | 19 kB 00:00 2024-02-28T16:15:01.411 INFO:teuthology.orchestra.run.smithi001.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 398 kB/s | 20 kB 00:00 2024-02-28T16:15:01.436 INFO:teuthology.orchestra.run.smithi001.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 798 kB/s | 46 kB 00:00 2024-02-28T16:15:01.454 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-28T16:15:01.470 INFO:teuthology.orchestra.run.smithi001.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-02-28T16:15:01.499 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-28T16:15:01.512 INFO:teuthology.orchestra.run.smithi001.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-02-28T16:15:01.537 INFO:teuthology.orchestra.run.smithi001.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 248 kB/s | 16 kB 00:00 2024-02-28T16:15:01.554 INFO:teuthology.orchestra.run.smithi001.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-02-28T16:15:01.596 INFO:teuthology.orchestra.run.smithi001.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 621 kB/s | 36 kB 00:00 2024-02-28T16:15:01.617 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-28T16:15:01.621 INFO:teuthology.orchestra.run.smithi001.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-02-28T16:15:01.663 INFO:teuthology.orchestra.run.smithi001.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 490 kB/s | 20 kB 00:00 2024-02-28T16:15:01.747 INFO:teuthology.orchestra.run.smithi001.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-02-28T16:15:02.537 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-28T16:15:02.681 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-28T16:15:04.346 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-28T16:15:05.580 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-28T16:15:05.605 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-535.gc0f02e 12/12 2024-02-28T16:15:07.938 INFO:teuthology.orchestra.run.smithi001.stdout:(33/33): ceph-osd-17.2.7-535.gc0f02e60.el9.x86_ 1.8 MB/s | 15 MB 00:08 2024-02-28T16:15:07.945 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:07.946 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.5 MB/s | 27 MB 00:10 2024-02-28T16:15:08.309 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-535.gc0f02e 12/12 2024-02-28T16:15:08.309 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-535.gc0f02e 1/12 2024-02-28T16:15:08.309 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-02-28T16:15:08.310 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-02-28T16:15:08.324 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:15:08.415 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:15:08.416 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:15:08.876 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:15:08.877 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:15:08.900 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-02-28T16:15:08.900 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:08.900 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:08.900 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:08.900 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-28T16:15:08.900 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-28T16:15:08.900 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-28T16:15:08.900 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:08.901 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:09.352 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-rook 2024-02-28T16:15:10.327 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:55 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:10.365 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:15:10.425 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-28T16:15:10.463 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-28T16:15:10.506 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-28T16:15:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.654 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:10.655 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:10.656 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.657 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.658 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:10.659 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.660 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.661 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.662 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:10.730 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:10.731 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 47 k 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-02-28T16:15:10.732 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout:Install 13 Packages 2024-02-28T16:15:10.733 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:10.734 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 2.2 M 2024-02-28T16:15:10.734 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 26 M 2024-02-28T16:15:10.734 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:15:10.752 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-28T16:15:10.806 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-28T16:15:10.898 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-28T16:15:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-02-28T16:15:10.975 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-02-28T16:15:11.019 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-02-28T16:15:11.072 INFO:teuthology.orchestra.run.smithi066.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 147 kB/s | 26 kB 00:00 2024-02-28T16:15:11.083 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-02-28T16:15:11.097 INFO:teuthology.orchestra.run.smithi066.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 96 kB/s | 19 kB 00:00 2024-02-28T16:15:11.148 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-02-28T16:15:11.193 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-02-28T16:15:11.197 INFO:teuthology.orchestra.run.smithi066.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-02-28T16:15:11.223 INFO:teuthology.orchestra.run.smithi066.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-02-28T16:15:11.252 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-02-28T16:15:11.315 INFO:teuthology.orchestra.run.smithi066.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 591 kB/s | 54 kB 00:00 2024-02-28T16:15:11.357 INFO:teuthology.orchestra.run.smithi066.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.7 MB/s | 279 kB 00:00 2024-02-28T16:15:11.397 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-02-28T16:15:11.474 INFO:teuthology.orchestra.run.smithi066.stdout:(7/13): ceph-mgr-rook-17.2.7-535.gc0f02e60.el9. 82 kB/s | 47 kB 00:00 2024-02-28T16:15:11.595 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-02-28T16:15:11.649 INFO:teuthology.orchestra.run.smithi066.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 97 kB/s | 32 kB 00:00 2024-02-28T16:15:11.701 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-02-28T16:15:11.708 INFO:teuthology.orchestra.run.smithi066.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-02-28T16:15:11.777 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-02-28T16:15:11.783 INFO:teuthology.orchestra.run.smithi066.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 798 kB/s | 59 kB 00:00 2024-02-28T16:15:11.819 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-02-28T16:15:11.825 INFO:teuthology.orchestra.run.smithi066.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 728 kB/s | 203 kB 00:00 2024-02-28T16:15:11.850 INFO:teuthology.orchestra.run.smithi066.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-02-28T16:15:11.892 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-02-28T16:15:11.968 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-02-28T16:15:12.025 INFO:teuthology.orchestra.run.smithi066.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-02-28T16:15:12.033 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:12.034 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-02-28T16:15:12.220 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-02-28T16:15:12.261 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:15:12.281 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-02-28T16:15:12.311 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:15:12.311 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:15:12.372 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-02-28T16:15:12.483 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:15:12.484 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:12.493 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-02-28T16:15:12.546 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-02-28T16:15:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-02-28T16:15:12.876 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-535.gc0f02e60.el9.n 28/33 2024-02-28T16:15:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 29/33 2024-02-28T16:15:12.982 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 29/33 2024-02-28T16:15:12.982 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:15:12.982 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-28T16:15:12.982 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-28T16:15:12.983 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-28T16:15:12.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:13.185 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:13.437 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-28T16:15:13.503 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-28T16:15:13.577 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-28T16:15:13.639 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-28T16:15:13.688 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-28T16:15:13.822 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-28T16:15:13.960 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-02-28T16:15:14.038 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-28T16:15:14.545 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:17.2.7-535.gc0f02e60.el9.x86_64 30/33 2024-02-28T16:15:14.629 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:17.2.7-535.gc0f02e60.el9.x86_64 30/33 2024-02-28T16:15:14.629 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:15:14.629 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-28T16:15:14.629 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-28T16:15:14.629 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-28T16:15:14.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:15.023 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-28T16:15:15.071 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:17.2.7-535.gc0f02e60.el9.x86_64 31/33 2024-02-28T16:15:15.078 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-28T16:15:15.111 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-28T16:15:15.144 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:17.2.7-535.gc0f02e60.el9.x86_64 31/33 2024-02-28T16:15:15.144 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:15:15.145 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-28T16:15:15.145 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-28T16:15:15.145 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-28T16:15:15.145 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:15.164 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-28T16:15:15.187 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-rook-2:17.2.7-535.gc0f02e60.el9.noarch 13/13 2024-02-28T16:15:15.396 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:17.2.7-535.gc0f02e60.el9.x86_64 32/33 2024-02-28T16:15:15.471 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:17.2.7-535.gc0f02e60.el9.x86_64 32/33 2024-02-28T16:15:15.471 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:15:15.471 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-28T16:15:15.471 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-28T16:15:15.471 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-28T16:15:15.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:15.674 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:17.2.7-535.gc0f02e60.el9.x86_64 33/33 2024-02-28T16:15:15.799 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-2:17.2.7-535.gc0f02e60.el9.x86_64 33/33 2024-02-28T16:15:15.800 INFO:teuthology.orchestra.run.smithi001.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-02-28T16:15:15.800 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:16.183 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-535.gc0f02e60.el9.noarch 13/13 2024-02-28T16:15:16.183 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-rook-2:17.2.7-535.gc0f02e60.el9.noarch 1/13 2024-02-28T16:15:16.183 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-28T16:15:16.184 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-28T16:15:16.819 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-28T16:15:16.819 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:16.819 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:16.819 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:16.819 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-28T16:15:16.819 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-28T16:15:16.819 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:16.820 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:17.194 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-cephadm 2024-02-28T16:15:17.464 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:17.2.7-535.gc0f02e60.el9.x86_64 1/33 2024-02-28T16:15:17.464 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:17.2.7-535.gc0f02e60.el9.x86_64 2/33 2024-02-28T16:15:17.464 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 3/33 2024-02-28T16:15:17.464 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:17.2.7-535.gc0f02e60.el9.x86_64 4/33 2024-02-28T16:15:17.464 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:17.2.7-535.gc0f02e60.el9.x86_64 5/33 2024-02-28T16:15:17.464 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-535.gc0f02e60.el9.n 6/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-28T16:15:17.465 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-28T16:15:17.466 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-28T16:15:17.467 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-28T16:15:17.467 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-28T16:15:17.467 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-28T16:15:17.467 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-28T16:15:17.467 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-28T16:15:18.138 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:03 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:18.466 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.467 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.468 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.469 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:18.470 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.471 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.472 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.473 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.474 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.475 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.475 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:18.524 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-28T16:15:18.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:18.524 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:15:18.524 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-28T16:15:18.525 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-28T16:15:18.526 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-28T16:15:18.527 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-28T16:15:18.527 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-28T16:15:18.527 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-28T16:15:18.527 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-28T16:15:18.527 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-28T16:15:18.527 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-28T16:15:18.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:18.527 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:15:18.545 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 121 k 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-28T16:15:18.546 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-28T16:15:18.547 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:18.547 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:15:18.547 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:18.547 INFO:teuthology.orchestra.run.smithi066.stdout:Install 6 Packages 2024-02-28T16:15:18.547 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:18.547 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 7.0 M 2024-02-28T16:15:18.547 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 33 M 2024-02-28T16:15:18.548 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:15:18.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base 2024-02-28T16:15:19.185 INFO:teuthology.orchestra.run.smithi066.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 669 kB/s | 248 kB 00:00 2024-02-28T16:15:19.419 INFO:teuthology.orchestra.run.smithi066.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.9 MB/s | 6.0 MB 00:00 2024-02-28T16:15:19.719 INFO:teuthology.orchestra.run.smithi066.stdout:(3/6): ceph-mgr-cephadm-17.2.7-535.gc0f02e60.el 134 kB/s | 121 kB 00:00 2024-02-28T16:15:19.744 INFO:teuthology.orchestra.run.smithi066.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 178 kB/s | 58 kB 00:00 2024-02-28T16:15:19.803 INFO:teuthology.orchestra.run.smithi066.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 888 kB/s | 548 kB 00:00 2024-02-28T16:15:19.811 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:17 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:15:20.053 INFO:teuthology.orchestra.run.smithi066.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 153 kB/s | 51 kB 00:00 2024-02-28T16:15:20.059 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:20.060 INFO:teuthology.orchestra.run.smithi066.stdout:Total 4.6 MB/s | 7.0 MB 00:01 2024-02-28T16:15:20.141 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:15:20.141 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:15:20.141 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:15:20.141 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.142 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.143 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.144 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.145 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.146 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.147 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.148 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.149 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.150 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.151 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.151 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.151 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:20.151 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.151 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.151 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.151 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:20.152 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-base-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:20.194 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:15:20.219 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:15:20.221 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:15:20.221 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:15:20.236 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:15:20.236 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:15:20.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install cephadm 2024-02-28T16:15:20.403 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:15:20.405 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:20.822 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:21.056 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-28T16:15:21.119 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-28T16:15:21.189 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:19 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:15:21.515 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:15:21.515 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:15:21.515 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:15:21.516 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:15:21.517 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:15:21.518 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.519 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:15:21.520 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.521 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.522 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.523 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.524 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.525 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.525 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.525 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:21.525 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.525 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.525 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.525 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:21.593 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:15:21.593 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 80 k 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 80 k 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 364 k 2024-02-28T16:15:21.594 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:15:21.851 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-28T16:15:21.966 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-28T16:15:22.054 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-28T16:15:22.091 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 6/6 2024-02-28T16:15:22.179 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm-17.2.7-535.gc0f02e60.el9.noarch.rpm 138 kB/s | 80 kB 00:00 2024-02-28T16:15:22.180 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:22.181 INFO:teuthology.orchestra.run.smithi001.stdout:Total 137 kB/s | 80 kB 00:00 2024-02-28T16:15:22.182 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:15:22.200 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:15:22.200 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:15:22.207 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:15:22.208 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:15:22.343 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:15:23.001 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/1 2024-02-28T16:15:23.030 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 6/6 2024-02-28T16:15:23.030 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/6 2024-02-28T16:15:23.030 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-28T16:15:23.030 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-28T16:15:23.030 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-28T16:15:23.030 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-28T16:15:23.131 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/1 2024-02-28T16:15:23.449 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-28T16:15:23.449 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:23.449 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:23.449 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:23.449 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-28T16:15:23.450 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-28T16:15:23.450 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-28T16:15:23.450 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-28T16:15:23.450 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-28T16:15:23.450 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:23.450 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/1 2024-02-28T16:15:23.817 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-fuse 2024-02-28T16:15:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/1 2024-02-28T16:15:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:23.970 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:15:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:23.970 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:15:24.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-immutable-object-cache 2024-02-28T16:15:24.728 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:09 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:25.055 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:25.055 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:25.055 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:25.055 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.055 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:25.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:25.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.062 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.064 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:25.064 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.064 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.064 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.064 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.130 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 812 k 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:25.131 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-02-28T16:15:25.132 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:25.132 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 892 k 2024-02-28T16:15:25.132 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 2.6 M 2024-02-28T16:15:25.132 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:15:25.153 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:23 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:15:25.471 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 305 kB/s | 80 kB 00:00 2024-02-28T16:15:25.477 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:15:25.477 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:15:25.477 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:15:25.477 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:15:25.478 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.479 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:15:25.480 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:15:25.481 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:15:25.482 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.483 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:25.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:25.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:25.552 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 137 k 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 137 k 2024-02-28T16:15:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 379 k 2024-02-28T16:15:25.554 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:15:26.252 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-immutable-object-cache-17.2.7-535.gc0f02e6 196 kB/s | 137 kB 00:00 2024-02-28T16:15:26.254 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:26.255 INFO:teuthology.orchestra.run.smithi001.stdout:Total 195 kB/s | 137 kB 00:00 2024-02-28T16:15:26.255 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:15:26.282 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:15:26.282 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:15:26.371 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-fuse-17.2.7-535.gc0f02e60.el9.x86_6 698 kB/s | 812 kB 00:01 2024-02-28T16:15:26.375 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:26.376 INFO:teuthology.orchestra.run.smithi066.stdout:Total 717 kB/s | 892 kB 00:01 2024-02-28T16:15:26.385 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:15:26.385 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:15:26.418 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:15:26.494 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:15:26.494 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:15:26.611 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:15:26.612 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:26.620 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:15:26.670 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-535.gc0f02e60.e 1/1 2024-02-28T16:15:26.745 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-535.gc0f02e60.e 1/1 2024-02-28T16:15:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:15:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-28T16:15:26.746 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:26.895 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:27.096 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-28T16:15:27.179 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 2/2 2024-02-28T16:15:28.048 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-535.gc0f02e60.e 1/1 2024-02-28T16:15:28.048 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:28.048 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:15:28.048 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:28.048 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:28.048 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:15:28.196 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 2/2 2024-02-28T16:15:28.196 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 1/2 2024-02-28T16:15:28.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr 2024-02-28T16:15:28.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-28T16:15:28.658 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:28.658 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:28.658 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-28T16:15:28.658 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:28.658 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:29.000 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados-devel 2024-02-28T16:15:29.351 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:27 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:15:29.680 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:15:29.680 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:15:29.680 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:15:29.680 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.681 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.682 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:15:29.683 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.684 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:15:29.685 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:15:29.686 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.687 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.687 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.687 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.687 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.687 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.687 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:29.687 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.687 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:29.688 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:29.689 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.690 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.690 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.690 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.690 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:29.690 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.690 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.690 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.690 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:29.691 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-mgr-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:29.758 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:15:29.759 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:15:29.759 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:15:29.846 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-dashboard 2024-02-28T16:15:29.960 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:14 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:30.286 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:30.287 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.288 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.289 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.290 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.291 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.292 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.293 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.294 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:30.295 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.295 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.295 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.295 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:30.363 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 126 k 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:30.364 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-02-28T16:15:30.365 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:30.365 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 126 k 2024-02-28T16:15:30.365 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 448 k 2024-02-28T16:15:30.365 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:15:30.739 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:28 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:15:30.934 INFO:teuthology.orchestra.run.smithi066.stdout:librados-devel-17.2.7-535.gc0f02e60.el9.x86_64. 222 kB/s | 126 kB 00:00 2024-02-28T16:15:30.936 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:30.937 INFO:teuthology.orchestra.run.smithi066.stdout:Total 221 kB/s | 126 kB 00:00 2024-02-28T16:15:30.937 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:15:30.958 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:15:30.958 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:15:31.060 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:15:31.061 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.062 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.063 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.064 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.065 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.066 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.067 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.068 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.069 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:31.069 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.069 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.069 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.069 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:31.073 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:15:31.074 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:31.138 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 1.7 M 2024-02-28T16:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-02-28T16:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 23 k 2024-02-28T16:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 15 k 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:Install 7 Packages 2024-02-28T16:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.0 M 2024-02-28T16:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 8.1 M 2024-02-28T16:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:15:31.376 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:31.507 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librados-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:31.698 INFO:teuthology.orchestra.run.smithi001.stdout:(1/7): ceph-grafana-dashboards-17.2.7-535.gc0f0 50 kB/s | 23 kB 00:00 2024-02-28T16:15:31.757 INFO:teuthology.orchestra.run.smithi001.stdout:(2/7): ceph-prometheus-alerts-17.2.7-535.gc0f02 28 kB/s | 15 kB 00:00 2024-02-28T16:15:31.987 INFO:teuthology.orchestra.run.smithi001.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 31 kB/s | 9.0 kB 00:00 2024-02-28T16:15:32.065 INFO:teuthology.orchestra.run.smithi001.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 133 kB/s | 41 kB 00:00 2024-02-28T16:15:32.091 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:32.166 INFO:teuthology.orchestra.run.smithi001.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 173 kB/s | 31 kB 00:00 2024-02-28T16:15:32.383 INFO:teuthology.orchestra.run.smithi001.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 596 kB/s | 188 kB 00:00 2024-02-28T16:15:32.490 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:32.491 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:32.491 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:32.491 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:32.491 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:32.491 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:32.869 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs2 2024-02-28T16:15:33.433 INFO:teuthology.orchestra.run.smithi001.stdout:(7/7): ceph-mgr-dashboard-17.2.7-535.gc0f02e60. 789 kB/s | 1.7 MB 00:02 2024-02-28T16:15:33.438 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:33.439 INFO:teuthology.orchestra.run.smithi001.stdout:Total 889 kB/s | 2.0 MB 00:02 2024-02-28T16:15:33.439 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:15:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:15:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:15:33.612 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:15:33.613 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:15:33.821 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:18 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:34.065 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:15:34.151 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-28T16:15:34.157 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:34.157 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:34.157 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:34.157 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.158 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.159 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:34.160 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.161 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:34.162 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.163 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.164 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.165 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:34.166 INFO:teuthology.orchestra.run.smithi066.stdout:Package libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:34.199 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-28T16:15:34.238 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:34.239 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:15:34.239 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:34.282 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-28T16:15:34.310 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs-devel 2024-02-28T16:15:34.332 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-28T16:15:34.410 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-535.gc0f02e60.el9.no 5/7 2024-02-28T16:15:34.750 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-535.gc0f02e60.el9.n 6/7 2024-02-28T16:15:34.787 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-535.gc0f02e60.el9.noarch 7/7 2024-02-28T16:15:35.258 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:20 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:35.412 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-535.gc0f02e60.el9.noarch 7/7 2024-02-28T16:15:35.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-535.gc0f02e60.el9.n 1/7 2024-02-28T16:15:35.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-535.gc0f02e60.el9.noarch 2/7 2024-02-28T16:15:35.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-535.gc0f02e60.el9.no 3/7 2024-02-28T16:15:35.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-28T16:15:35.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-28T16:15:35.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-28T16:15:35.586 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:35.586 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:35.586 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:35.586 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.586 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:35.587 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.588 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:35.589 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:35.590 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:35.591 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.592 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:35.593 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.594 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.595 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:35.595 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.595 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.595 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.595 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:35.664 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:35.665 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:35.665 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-02-28T16:15:35.665 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:35.665 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:15:35.665 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 25 k 2024-02-28T16:15:35.665 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:35.665 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:15:35.666 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:35.666 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-02-28T16:15:35.666 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:35.666 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 25 k 2024-02-28T16:15:35.666 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 110 k 2024-02-28T16:15:35.666 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:15:35.848 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-28T16:15:35.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:35.848 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:15:35.848 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:35.848 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:35.848 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:35.849 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-28T16:15:35.849 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-28T16:15:35.849 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-28T16:15:35.849 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-28T16:15:35.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:35.849 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:15:35.980 INFO:teuthology.orchestra.run.smithi066.stdout:libcephfs-devel-17.2.7-535.gc0f02e60.el9.x86_64 80 kB/s | 25 kB 00:00 2024-02-28T16:15:35.981 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:35.982 INFO:teuthology.orchestra.run.smithi066.stdout:Total 79 kB/s | 25 kB 00:00 2024-02-28T16:15:35.982 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:15:35.994 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:15:35.995 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:15:36.001 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:15:36.002 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:36.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-28T16:15:36.182 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:36.353 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:36.829 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:37.079 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:35 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:15:37.180 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:37.180 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:37.180 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:37.181 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:37.181 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:37.181 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.402 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:15:37.403 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.404 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.405 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.406 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.407 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.408 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.409 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.410 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.411 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:37.412 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados2 2024-02-28T16:15:37.482 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:15:37.483 INFO:teuthology.orchestra.run.smithi001.stdout:====================================================================================== 2024-02-28T16:15:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:15:37.483 INFO:teuthology.orchestra.run.smithi001.stdout:====================================================================================== 2024-02-28T16:15:37.483 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:15:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 7.4 M 2024-02-28T16:15:37.483 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-02-28T16:15:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-02-28T16:15:37.483 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:15:37.484 INFO:teuthology.orchestra.run.smithi001.stdout:====================================================================================== 2024-02-28T16:15:37.485 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-02-28T16:15:37.485 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:37.485 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 38 M 2024-02-28T16:15:37.485 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 209 M 2024-02-28T16:15:37.485 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:15:38.248 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 497 kB/s | 195 kB 00:00 2024-02-28T16:15:38.298 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-02-28T16:15:38.334 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:23 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:38.349 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 329 kB/s | 33 kB 00:00 2024-02-28T16:15:38.416 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 276 kB/s | 18 kB 00:00 2024-02-28T16:15:38.483 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 535 kB/s | 35 kB 00:00 2024-02-28T16:15:38.600 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-02-28T16:15:38.663 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:38.663 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:38.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:38.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:38.663 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.663 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.664 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.665 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.666 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.667 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.668 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:38.669 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.670 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.671 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.672 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:38.673 INFO:teuthology.orchestra.run.smithi066.stdout:Package librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:38.675 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-02-28T16:15:38.739 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:38.740 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:15:38.740 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:38.822 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librbd1 2024-02-28T16:15:38.850 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-02-28T16:15:38.926 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-02-28T16:15:39.034 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-02-28T16:15:39.226 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-5 5.4 MB/s | 7.4 MB 00:01 2024-02-28T16:15:39.535 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-02-28T16:15:39.540 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:39.541 INFO:teuthology.orchestra.run.smithi001.stdout:Total 18 MB/s | 38 MB 00:02 2024-02-28T16:15:39.717 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:24 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:40.040 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:40.040 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:40.040 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:40.040 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.040 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.041 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.042 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.043 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:40.044 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:40.045 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:40.046 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.047 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:40.048 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:40.049 INFO:teuthology.orchestra.run.smithi066.stdout:Package librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:40.118 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:40.119 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:15:40.119 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:40.192 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rados 2024-02-28T16:15:40.203 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:15:40.285 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:15:40.286 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:15:40.891 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:15:40.892 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:15:41.111 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:26 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:41.445 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:41.445 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:41.445 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:41.445 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:41.446 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:41.447 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:41.448 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.449 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.450 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.451 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.452 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.453 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.454 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.454 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:41.454 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.454 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.454 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.454 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:41.454 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rados-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:41.525 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:41.526 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:15:41.527 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:41.605 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rgw 2024-02-28T16:15:41.816 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:15:41.886 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-02-28T16:15:41.954 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-02-28T16:15:42.117 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-02-28T16:15:42.162 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-28T16:15:42.421 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-28T16:15:42.459 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-28T16:15:42.511 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:27 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-28T16:15:42.847 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:42.847 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:42.847 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:42.847 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.847 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:42.847 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.847 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.848 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:42.849 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:42.850 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.851 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:42.852 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.853 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.854 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.855 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.856 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.856 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:42.856 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rgw-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:42.928 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:42.929 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:15:42.929 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:43.019 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-cephfs 2024-02-28T16:15:43.519 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-28T16:15:43.658 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-28T16:15:43.952 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:28 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:44.276 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:44.276 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:44.277 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.278 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.279 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.280 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:44.281 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.282 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.283 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.284 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.285 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.286 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:44.286 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-cephfs-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:44.354 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:44.355 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:15:44.355 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:44.428 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rbd 2024-02-28T16:15:45.304 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-28T16:15:45.333 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:30 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.673 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:45.674 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:45.675 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:45.676 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.677 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.678 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.679 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.680 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:45.681 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:45.682 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.683 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:45.684 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rbd-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:15:45.754 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:45.756 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-02-28T16:15:45.756 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:45.833 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-fuse 2024-02-28T16:15:46.513 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-28T16:15:46.540 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-535.gc0f02e 12/12 2024-02-28T16:15:46.726 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:31 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:47.055 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:47.055 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:47.055 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:47.055 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.055 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:47.055 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.055 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:47.055 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:47.056 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:47.057 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.058 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.059 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.060 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:47.061 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.062 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.063 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:47.130 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:47.130 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:47.130 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 81 k 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 81 k 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 222 k 2024-02-28T16:15:47.131 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:15:47.508 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-fuse-17.2.7-535.gc0f02e60.el9.x86_64.rpm 217 kB/s | 81 kB 00:00 2024-02-28T16:15:47.510 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:47.511 INFO:teuthology.orchestra.run.smithi066.stdout:Total 215 kB/s | 81 kB 00:00 2024-02-28T16:15:47.511 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:15:47.535 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:15:47.536 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:15:47.648 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:15:47.648 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:47.845 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:48.555 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:48.927 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:48.928 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:48.928 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:48.928 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:48.928 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:48.928 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-535.gc0f02e 12/12 2024-02-28T16:15:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-535.gc0f02e 1/12 2024-02-28T16:15:49.084 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-02-28T16:15:49.085 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-02-28T16:15:49.266 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-mirror 2024-02-28T16:15:49.521 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-28T16:15:49.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-28T16:15:49.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-28T16:15:49.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-28T16:15:49.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:49.523 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:15:49.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-rook 2024-02-28T16:15:50.219 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:35 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:50.552 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:50.552 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:50.552 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:50.552 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.552 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.553 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.554 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.555 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.556 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.557 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.558 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.559 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.561 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.561 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.561 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.561 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:50.561 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.561 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.561 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.561 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:50.631 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 2.9 M 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-02-28T16:15:50.632 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:50.633 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 2.9 M 2024-02-28T16:15:50.633 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 12 M 2024-02-28T16:15:50.633 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:48 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:51.271 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:15:51.272 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:51.272 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:15:51.272 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:51.272 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:15:51.272 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 47 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-28T16:15:51.273 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-28T16:15:51.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:51.274 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:15:51.274 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:51.274 INFO:teuthology.orchestra.run.smithi001.stdout:Install 13 Packages 2024-02-28T16:15:51.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:51.274 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.2 M 2024-02-28T16:15:51.274 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 26 M 2024-02-28T16:15:51.275 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:15:51.361 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-mirror-17.2.7-535.gc0f02e60.el9.x86_64.rpm 4.0 MB/s | 2.9 MB 00:00 2024-02-28T16:15:51.363 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:51.364 INFO:teuthology.orchestra.run.smithi066.stdout:Total 4.0 MB/s | 2.9 MB 00:00 2024-02-28T16:15:51.364 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:15:51.398 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:15:51.398 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:15:51.519 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:15:51.520 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:51.875 INFO:teuthology.orchestra.run.smithi001.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 143 kB/s | 26 kB 00:00 2024-02-28T16:15:51.900 INFO:teuthology.orchestra.run.smithi001.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 94 kB/s | 19 kB 00:00 2024-02-28T16:15:51.967 INFO:teuthology.orchestra.run.smithi001.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-02-28T16:15:51.992 INFO:teuthology.orchestra.run.smithi001.stdout:(4/13): ceph-mgr-rook-17.2.7-535.gc0f02e60.el9. 160 kB/s | 47 kB 00:00 2024-02-28T16:15:52.026 INFO:teuthology.orchestra.run.smithi001.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-02-28T16:15:52.051 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:52.101 INFO:teuthology.orchestra.run.smithi001.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 279 kB 00:00 2024-02-28T16:15:52.126 INFO:teuthology.orchestra.run.smithi001.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 403 kB/s | 54 kB 00:00 2024-02-28T16:15:52.128 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-mirror-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:52.151 INFO:teuthology.orchestra.run.smithi001.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 258 kB/s | 32 kB 00:00 2024-02-28T16:15:52.177 INFO:teuthology.orchestra.run.smithi001.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 201 kB/s | 15 kB 00:00 2024-02-28T16:15:52.219 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-mirror-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:52.219 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:15:52.219 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-28T16:15:52.219 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-28T16:15:52.219 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-28T16:15:52.220 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:52.235 INFO:teuthology.orchestra.run.smithi001.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-02-28T16:15:52.269 INFO:teuthology.orchestra.run.smithi001.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-02-28T16:15:52.294 INFO:teuthology.orchestra.run.smithi001.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-02-28T16:15:52.336 INFO:teuthology.orchestra.run.smithi001.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.6 MB/s | 1.0 MB 00:00 2024-02-28T16:15:52.337 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:52.337 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-02-28T16:15:52.563 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:15:52.611 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:15:52.611 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:15:52.777 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:15:52.778 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:15:53.467 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:15:53.691 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-28T16:15:53.764 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-28T16:15:53.837 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-28T16:15:53.875 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-28T16:15:53.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-mirror-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:53.884 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:53.885 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:53.885 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:53.885 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:53.885 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:53.922 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-28T16:15:54.062 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-28T16:15:54.198 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-02-28T16:15:54.233 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-nbd 2024-02-28T16:15:54.284 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-28T16:15:55.163 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:40 ago on Wed 28 Feb 2024 04:13:15 PM UTC. 2024-02-28T16:15:55.255 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-28T16:15:55.314 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-28T16:15:55.348 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-28T16:15:55.429 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-28T16:15:55.456 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:17.2.7-535.gc0f02e60.el9.noarch 13/13 2024-02-28T16:15:55.485 INFO:teuthology.orchestra.run.smithi066.stderr:Modular dependency problems: 2024-02-28T16:15:55.485 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 1: conflicting requests 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 2: conflicting requests 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 3: conflicting requests 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 4: conflicting requests 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 5: conflicting requests 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 6: conflicting requests 2024-02-28T16:15:55.486 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 7: conflicting requests 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 8: conflicting requests 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 9: conflicting requests 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 10: conflicting requests 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 11: conflicting requests 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 12: conflicting requests 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.487 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 13: conflicting requests 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 14: conflicting requests 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 15: conflicting requests 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 16: conflicting requests 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 17: conflicting requests 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 18: conflicting requests 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 19: conflicting requests 2024-02-28T16:15:55.488 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 20: conflicting requests 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 21: conflicting requests 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 22: conflicting requests 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 23: conflicting requests 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 24: conflicting requests 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 25: conflicting requests 2024-02-28T16:15:55.489 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 26: conflicting requests 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 27: conflicting requests 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 28: conflicting requests 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 29: conflicting requests 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 30: conflicting requests 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 31: conflicting requests 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.490 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 32: conflicting requests 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 33: conflicting requests 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 34: conflicting requests 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 35: conflicting requests 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 36: conflicting requests 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.491 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 38: conflicting requests 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.492 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.493 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:55.494 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.495 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.495 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.495 INFO:teuthology.orchestra.run.smithi066.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:55.561 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-02-28T16:15:55.561 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:55.561 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 164 k 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 164 k 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 477 k 2024-02-28T16:15:55.562 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-02-28T16:15:55.806 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-nbd-17.2.7-535.gc0f02e60.el9.x86_64.rpm 679 kB/s | 164 kB 00:00 2024-02-28T16:15:55.807 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:55.808 INFO:teuthology.orchestra.run.smithi066.stdout:Total 669 kB/s | 164 kB 00:00 2024-02-28T16:15:55.808 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-02-28T16:15:55.836 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-02-28T16:15:55.836 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-02-28T16:15:55.944 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-02-28T16:15:55.945 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-02-28T16:15:56.152 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-02-28T16:15:56.287 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-nbd-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-535.gc0f02e60.el9.noarch 13/13 2024-02-28T16:15:56.425 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:17.2.7-535.gc0f02e60.el9.noarch 1/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-28T16:15:56.426 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-28T16:15:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-28T16:15:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:56.930 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:15:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:15:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-28T16:15:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-28T16:15:56.930 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:56.931 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:15:56.950 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-nbd-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:57.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-cephadm 2024-02-28T16:15:57.294 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-nbd-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:15:57.294 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:57.294 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-02-28T16:15:57.294 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:15:57.294 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:15:57.294 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-02-28T16:15:57.607 DEBUG:teuthology.parallel:result is None 2024-02-28T16:15:58.160 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:56 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:15:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:15:58.483 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:15:58.483 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:15:58.484 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:15:58.485 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.486 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:15:58.487 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:15:58.488 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.489 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.490 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.491 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:15:58.492 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.493 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.493 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.493 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:15:58.563 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:15:58.569 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:15:58.569 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:58.569 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:15:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:17.2.7-535.gc0f02e60.el9 ceph-noarch 121 k 2024-02-28T16:15:58.569 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 7.0 M 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 33 M 2024-02-28T16:15:58.570 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:15:59.052 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 931 kB/s | 248 kB 00:00 2024-02-28T16:15:59.088 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): ceph-mgr-cephadm-17.2.7-535.gc0f02e60.el 400 kB/s | 121 kB 00:00 2024-02-28T16:15:59.389 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.9 MB/s | 6.0 MB 00:00 2024-02-28T16:15:59.455 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 158 kB/s | 58 kB 00:00 2024-02-28T16:15:59.664 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 898 kB/s | 548 kB 00:00 2024-02-28T16:15:59.764 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 136 kB/s | 51 kB 00:00 2024-02-28T16:15:59.772 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:15:59.773 INFO:teuthology.orchestra.run.smithi001.stdout:Total 5.8 MB/s | 7.0 MB 00:01 2024-02-28T16:15:59.904 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:15:59.946 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:15:59.946 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:16:00.106 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:16:00.107 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:16:00.503 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:16:00.725 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-28T16:16:00.790 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-28T16:16:01.523 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-28T16:16:01.638 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-28T16:16:01.716 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-28T16:16:01.745 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 6/6 2024-02-28T16:16:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 6/6 2024-02-28T16:16:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 1/6 2024-02-28T16:16:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-28T16:16:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-28T16:16:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-28T16:16:02.664 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-28T16:16:03.071 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:17.2.7-535.gc0f02e60.el9.noarch 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:03.072 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:03.424 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-fuse 2024-02-28T16:16:04.320 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:02 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.642 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.643 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.644 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.645 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:04.646 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.647 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:04.648 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.649 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.650 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.651 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:04.719 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:04.720 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:04.720 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-02-28T16:16:04.720 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:04.720 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:16:04.720 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 812 k 2024-02-28T16:16:04.720 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-02-28T16:16:04.720 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-02-28T16:16:04.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:04.721 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:16:04.721 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:04.721 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-02-28T16:16:04.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:04.721 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 892 k 2024-02-28T16:16:04.721 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.6 M 2024-02-28T16:16:04.721 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:16:05.325 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 176 kB/s | 80 kB 00:00 2024-02-28T16:16:05.400 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): ceph-fuse-17.2.7-535.gc0f02e60.el9.x86_6 1.5 MB/s | 812 kB 00:00 2024-02-28T16:16:05.401 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:16:05.402 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.3 MB/s | 892 kB 00:00 2024-02-28T16:16:05.443 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:16:05.518 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:16:05.518 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:16:05.631 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:16:05.631 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:16:05.916 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:16:06.084 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-28T16:16:06.169 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 2/2 2024-02-28T16:16:07.169 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 2/2 2024-02-28T16:16:07.169 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 1/2 2024-02-28T16:16:07.558 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-28T16:16:07.558 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:07.559 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:16:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-28T16:16:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:07.559 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:07.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados-devel 2024-02-28T16:16:08.843 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:06 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:09.169 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:09.169 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:09.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:09.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:09.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:09.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.176 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.177 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.178 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:09.246 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 126 k 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-02-28T16:16:09.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:09.248 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 126 k 2024-02-28T16:16:09.248 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 448 k 2024-02-28T16:16:09.248 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:16:09.496 INFO:teuthology.orchestra.run.smithi001.stdout:librados-devel-17.2.7-535.gc0f02e60.el9.x86_64. 510 kB/s | 126 kB 00:00 2024-02-28T16:16:09.498 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:16:09.498 INFO:teuthology.orchestra.run.smithi001.stdout:Total 503 kB/s | 126 kB 00:00 2024-02-28T16:16:09.499 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:16:09.518 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:16:09.518 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:16:09.632 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:16:09.632 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:16:09.842 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:16:09.964 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:10.569 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:10.937 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:16:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:16:10.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:10.937 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:11.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs2 2024-02-28T16:16:12.227 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:10 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:12.556 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:12.557 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.558 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:12.559 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.559 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:12.559 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.559 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:12.559 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.559 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:12.559 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.559 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.560 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:12.561 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.562 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:12.563 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.564 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.565 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:12.566 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:12.567 INFO:teuthology.orchestra.run.smithi001.stdout:Package libcephfs2-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:16:12.636 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:12.637 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:16:12.638 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:12.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs-devel 2024-02-28T16:16:13.616 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:11 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.941 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:13.942 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:13.943 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:13.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:13.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:13.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:14.018 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-02-28T16:16:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:16:14.019 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 25 k 2024-02-28T16:16:14.019 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:16:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-02-28T16:16:14.019 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:14.020 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 25 k 2024-02-28T16:16:14.020 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 110 k 2024-02-28T16:16:14.020 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:16:14.210 INFO:teuthology.orchestra.run.smithi001.stdout:libcephfs-devel-17.2.7-535.gc0f02e60.el9.x86_64 131 kB/s | 25 kB 00:00 2024-02-28T16:16:14.211 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:16:14.212 INFO:teuthology.orchestra.run.smithi001.stdout:Total 130 kB/s | 25 kB 00:00 2024-02-28T16:16:14.212 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:16:14.219 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:16:14.219 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:16:14.223 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:16:14.224 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:16:14.367 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:16:14.534 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:15.006 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:15.314 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:15.315 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:15.315 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:16:15.315 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:16:15.315 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:15.315 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:15.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados2 2024-02-28T16:16:16.488 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:14 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:16.813 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.814 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.815 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:16.816 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:16.817 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.818 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.819 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.820 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.821 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.822 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:16.822 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.822 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.822 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.822 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:16.822 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:16:16.888 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:16.890 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:16:16.890 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:16.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librbd1 2024-02-28T16:16:17.844 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:15 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:18.165 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:18.165 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:18.165 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:18.165 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.166 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:18.167 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:18.168 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:18.169 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:18.170 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.171 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.172 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.173 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.174 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.175 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:18.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.175 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:18.175 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:16:18.241 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:18.242 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:16:18.243 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:18.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rados 2024-02-28T16:16:19.216 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:17 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:19.545 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:19.629 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.630 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:19.631 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.632 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.633 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.634 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.635 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.636 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.637 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:19.638 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rados-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:16:19.638 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:19.639 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:16:19.639 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:19.723 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rgw 2024-02-28T16:16:20.614 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:18 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:20.943 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:20.943 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:20.943 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:20.943 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.943 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:20.944 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.945 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:20.946 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:20.947 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.948 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.949 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.950 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.951 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.952 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.952 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.952 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:20.952 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.952 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.952 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.952 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:20.953 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rgw-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:16:21.023 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:21.024 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:16:21.024 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:21.114 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-cephfs 2024-02-28T16:16:21.993 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:20 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:22.317 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.318 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:22.319 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.320 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:22.321 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.322 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.323 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.324 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.325 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:22.326 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-cephfs-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:16:22.394 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:22.395 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:16:22.395 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:22.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rbd 2024-02-28T16:16:23.377 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:21 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:23.705 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.706 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:23.707 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.708 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:23.709 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:23.710 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.711 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.712 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.713 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.714 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.715 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.715 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:23.715 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rbd-2:17.2.7-535.gc0f02e60.el9.x86_64 is already installed. 2024-02-28T16:16:23.785 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:23.786 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-02-28T16:16:23.786 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:23.877 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-fuse 2024-02-28T16:16:24.792 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:22 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.119 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.120 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.121 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.122 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:25.123 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.124 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.125 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.126 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.127 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.128 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.128 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:25.195 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:25.195 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 81 k 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-02-28T16:16:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:25.197 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 81 k 2024-02-28T16:16:25.197 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 222 k 2024-02-28T16:16:25.197 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:16:25.546 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-fuse-17.2.7-535.gc0f02e60.el9.x86_64.rpm 233 kB/s | 81 kB 00:00 2024-02-28T16:16:25.548 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:16:25.549 INFO:teuthology.orchestra.run.smithi001.stdout:Total 231 kB/s | 81 kB 00:00 2024-02-28T16:16:25.549 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:16:25.573 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:16:25.573 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:16:25.684 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:16:25.684 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:16:25.896 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:16:26.033 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:26.541 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:26.920 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:26.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:26.920 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:16:26.920 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:16:26.920 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:26.920 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:27.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-02-28T16:16:28.155 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:26 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:28.499 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.500 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.501 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.502 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:28.503 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.504 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.505 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.506 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.507 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.508 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:28.578 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:28.578 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:28.578 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-02-28T16:16:28.578 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:28.579 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:16:28.579 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 2.9 M 2024-02-28T16:16:28.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:28.579 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:16:28.579 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:28.579 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-02-28T16:16:28.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:28.580 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.9 M 2024-02-28T16:16:28.580 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 12 M 2024-02-28T16:16:28.580 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:16:30.029 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-mirror-17.2.7-535.gc0f02e60.el9.x86_64.rpm 2.0 MB/s | 2.9 MB 00:01 2024-02-28T16:16:30.031 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:16:30.032 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.0 MB/s | 2.9 MB 00:01 2024-02-28T16:16:30.033 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:16:30.067 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:16:30.067 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:16:30.189 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:16:30.189 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:16:30.732 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:16:30.801 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:30.887 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:30.887 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-28T16:16:30.887 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-28T16:16:30.888 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-28T16:16:30.888 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-28T16:16:30.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:32.338 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:32.338 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:32.338 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:16:32.338 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:16:32.338 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:32.338 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:32.683 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-nbd 2024-02-28T16:16:33.595 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:02:31 ago on Wed 28 Feb 2024 04:14:02 PM UTC. 2024-02-28T16:16:33.918 INFO:teuthology.orchestra.run.smithi001.stderr:Modular dependency problems: 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 1: conflicting requests 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240221162200:20125149.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 2: conflicting requests 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 3: conflicting requests 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 4: conflicting requests 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.919 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 5: conflicting requests 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 6: conflicting requests 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 7: conflicting requests 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 8: conflicting requests 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 9: conflicting requests 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 10: conflicting requests 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.920 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 11: conflicting requests 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 12: conflicting requests 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 13: conflicting requests 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 14: conflicting requests 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 15: conflicting requests 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 16: conflicting requests 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 17: conflicting requests 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 18: conflicting requests 2024-02-28T16:16:33.921 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 19: conflicting requests 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 20: conflicting requests 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 21: conflicting requests 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 22: conflicting requests 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 23: conflicting requests 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 24: conflicting requests 2024-02-28T16:16:33.922 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 25: conflicting requests 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 26: conflicting requests 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 27: conflicting requests 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 28: conflicting requests 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 29: conflicting requests 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 30: conflicting requests 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.923 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 31: conflicting requests 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 32: conflicting requests 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 33: conflicting requests 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 34: conflicting requests 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 35: conflicting requests 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240226094445:9edba152.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 36: conflicting requests 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.924 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 37: cannot install the best candidate for the job 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 38: conflicting requests 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.925 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 39: cannot install the best candidate for the job 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 40: cannot install the best candidate for the job 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 41: cannot install the best candidate for the job 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.926 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 42: cannot install the best candidate for the job 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 43: cannot install the best candidate for the job 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: Problem 44: cannot install the best candidate for the job 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.927 INFO:teuthology.orchestra.run.smithi001.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-28T16:16:33.993 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-02-28T16:16:33.994 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:33.994 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-02-28T16:16:33.994 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:33.994 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-02-28T16:16:33.994 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:17.2.7-535.gc0f02e60.el9 ceph 164 k 2024-02-28T16:16:33.994 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:33.994 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-02-28T16:16:33.994 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-02-28T16:16:33.995 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-02-28T16:16:33.995 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:33.995 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 164 k 2024-02-28T16:16:33.995 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 477 k 2024-02-28T16:16:33.995 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-02-28T16:16:34.245 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-nbd-17.2.7-535.gc0f02e60.el9.x86_64.rpm 658 kB/s | 164 kB 00:00 2024-02-28T16:16:34.246 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:16:34.247 INFO:teuthology.orchestra.run.smithi001.stdout:Total 651 kB/s | 164 kB 00:00 2024-02-28T16:16:34.247 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-02-28T16:16:34.274 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-02-28T16:16:34.274 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-02-28T16:16:34.382 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-02-28T16:16:34.382 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-02-28T16:16:34.583 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-02-28T16:16:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:35.346 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-nbd-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:35.710 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:17.2.7-535.gc0f02e60.el9.x86_64 1/1 2024-02-28T16:16:35.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:35.710 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-02-28T16:16:35.710 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:17.2.7-535.gc0f02e60.el9.x86_64 2024-02-28T16:16:35.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:16:35.710 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-02-28T16:16:36.011 DEBUG:teuthology.parallel:result is None 2024-02-28T16:16:36.011 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-28T16:16:36.011 INFO:teuthology.packaging:ref: None 2024-02-28T16:16:36.011 INFO:teuthology.packaging:tag: None 2024-02-28T16:16:36.011 INFO:teuthology.packaging:branch: quincy 2024-02-28T16:16:36.011 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:16:36.012 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-28T16:16:36.313 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-28T16:16:36.385 INFO:teuthology.orchestra.run.smithi001.stdout:17.2.7-535.gc0f02e60.el9 2024-02-28T16:16:36.386 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-535.gc0f02e60.el9 2024-02-28T16:16:36.386 INFO:teuthology.task.install:The correct ceph version 17.2.7-535.gc0f02e60 is installed. 2024-02-28T16:16:36.388 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-28T16:16:36.388 INFO:teuthology.packaging:ref: None 2024-02-28T16:16:36.388 INFO:teuthology.packaging:tag: None 2024-02-28T16:16:36.388 INFO:teuthology.packaging:branch: quincy 2024-02-28T16:16:36.388 INFO:teuthology.packaging:sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:16:36.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-28T16:16:36.591 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-28T16:16:36.667 INFO:teuthology.orchestra.run.smithi066.stdout:17.2.7-535.gc0f02e60.el9 2024-02-28T16:16:36.667 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-535.gc0f02e60.el9 2024-02-28T16:16:36.668 INFO:teuthology.task.install:The correct ceph version 17.2.7-535.gc0f02e60 is installed. 2024-02-28T16:16:36.670 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-28T16:16:36.670 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:16:36.670 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-28T16:16:36.770 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:16:36.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-28T16:16:36.882 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-28T16:16:36.882 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:16:36.882 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-02-28T16:16:36.987 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-28T16:16:37.086 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:16:37.086 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/daemon-helper 2024-02-28T16:16:37.198 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-28T16:16:37.296 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-28T16:16:37.297 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:16:37.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-28T16:16:37.402 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-28T16:16:37.544 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:16:37.545 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-28T16:16:37.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-28T16:16:37.773 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-28T16:16:37.773 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:16:37.774 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-02-28T16:16:37.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-28T16:16:37.980 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:16:37.981 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/stdin-killer 2024-02-28T16:16:38.079 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-28T16:16:38.177 INFO:teuthology.run_tasks:Running task print... 2024-02-28T16:16:38.187 INFO:teuthology.task.print:**** done install task... 2024-02-28T16:16:38.187 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-28T16:16:38.302 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'} 2024-02-28T16:16:38.303 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-02-28T16:16:38.303 INFO:tasks.cephadm:Cluster fsid is c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:16:38.303 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-28T16:16:38.303 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-28T16:16:38.303 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi001': '172.21.15.1', 'mon.smithi066': '172.21.15.66'} 2024-02-28T16:16:38.304 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-28T16:16:38.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo hostname $(hostname -s) 2024-02-28T16:16:38.400 DEBUG:teuthology.orchestra.run.smithi066:> sudo hostname $(hostname -s) 2024-02-28T16:16:38.499 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-02-28T16:16:38.500 DEBUG:teuthology.orchestra.run.smithi001:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-28T16:16:38.818 INFO:teuthology.orchestra.run.smithi001.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 28 16:16 /home/ubuntu/cephtest/cephadm 2024-02-28T16:16:38.818 DEBUG:teuthology.orchestra.run.smithi066:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-28T16:16:39.054 INFO:teuthology.orchestra.run.smithi066.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 28 16:16 /home/ubuntu/cephtest/cephadm 2024-02-28T16:16:39.054 DEBUG:teuthology.orchestra.run.smithi001:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-28T16:16:39.118 DEBUG:teuthology.orchestra.run.smithi066:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-28T16:16:39.190 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-02-28T16:16:39.190 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-28T16:16:39.198 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-28T16:16:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-28T16:16:40.549 INFO:teuthology.orchestra.run.smithi066.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-28T16:17:15.593 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:17:15.593 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)", 2024-02-28T16:17:15.593 INFO:teuthology.orchestra.run.smithi001.stdout: "image_id": "1c37ed3713489866ca403a4faaa38eb6c635a16f541c47d5bb849a203de99e6f", 2024-02-28T16:17:15.593 INFO:teuthology.orchestra.run.smithi001.stdout: "repo_digests": [ 2024-02-28T16:17:15.593 INFO:teuthology.orchestra.run.smithi001.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13" 2024-02-28T16:17:15.593 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-02-28T16:17:15.594 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:17:20.265 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-02-28T16:17:20.265 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_version": "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)", 2024-02-28T16:17:20.265 INFO:teuthology.orchestra.run.smithi066.stdout: "image_id": "1c37ed3713489866ca403a4faaa38eb6c635a16f541c47d5bb849a203de99e6f", 2024-02-28T16:17:20.265 INFO:teuthology.orchestra.run.smithi066.stdout: "repo_digests": [ 2024-02-28T16:17:20.266 INFO:teuthology.orchestra.run.smithi066.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13" 2024-02-28T16:17:20.266 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-02-28T16:17:20.266 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-02-28T16:17:20.318 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph 2024-02-28T16:17:20.427 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph 2024-02-28T16:17:20.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /etc/ceph 2024-02-28T16:17:20.647 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 777 /etc/ceph 2024-02-28T16:17:20.757 INFO:tasks.cephadm:Writing seed config... 2024-02-28T16:17:20.758 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-28T16:17:20.758 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-28T16:17:20.758 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-28T16:17:20.759 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-28T16:17:20.760 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-28T16:17:20.761 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-28T16:17:20.762 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:17:20.762 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-28T16:17:20.820 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c0118718-d654-11ee-95c3-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-28T16:17:20.821 DEBUG:teuthology.orchestra.run.smithi001:mon.smithi001> sudo journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service 2024-02-28T16:17:20.868 INFO:tasks.cephadm:Bootstrapping... 2024-02-28T16:17:20.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid c0118718-d654-11ee-95c3-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.1 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-28T16:17:21.193 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-02-28T16:17:21.194 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'c0118718-d654-11ee-95c3-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.1', '--skip-admin-label'] 2024-02-28T16:17:21.253 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-28T16:17:21.253 INFO:teuthology.orchestra.run.smithi001.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-28T16:17:21.254 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying podman|docker is present... 2024-02-28T16:17:21.315 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-28T16:17:21.315 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying lvm2 is present... 2024-02-28T16:17:21.315 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying time synchronization is in place... 2024-02-28T16:17:21.334 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-28T16:17:21.334 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-28T16:17:21.351 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-28T16:17:21.351 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-02-28T16:17:21.371 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-02-28T16:17:21.388 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-02-28T16:17:21.388 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-02-28T16:17:21.388 INFO:teuthology.orchestra.run.smithi001.stdout:Repeating the final host check... 2024-02-28T16:17:21.439 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-28T16:17:21.439 INFO:teuthology.orchestra.run.smithi001.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-28T16:17:21.439 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl is present 2024-02-28T16:17:21.439 INFO:teuthology.orchestra.run.smithi001.stdout:lvcreate is present 2024-02-28T16:17:21.454 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-28T16:17:21.454 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-28T16:17:21.471 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-28T16:17:21.472 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-02-28T16:17:21.496 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-02-28T16:17:21.518 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-02-28T16:17:21.518 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-02-28T16:17:21.518 INFO:teuthology.orchestra.run.smithi001.stdout:Host looks OK 2024-02-28T16:17:21.518 INFO:teuthology.orchestra.run.smithi001.stdout:Cluster fsid: c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:17:21.519 INFO:teuthology.orchestra.run.smithi001.stdout:Acquiring lock 140183867406848 on /run/cephadm/c0118718-d654-11ee-95c3-87774f69a715.lock 2024-02-28T16:17:21.519 INFO:teuthology.orchestra.run.smithi001.stdout:Lock 140183867406848 acquired on /run/cephadm/c0118718-d654-11ee-95c3-87774f69a715.lock 2024-02-28T16:17:21.521 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 3300 ... 2024-02-28T16:17:21.522 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 6789 ... 2024-02-28T16:17:21.522 INFO:teuthology.orchestra.run.smithi001.stdout:Base mon IP(s) is [172.21.15.1:3300, 172.21.15.1:6789], mon addrv is [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-02-28T16:17:21.533 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.1 metric 100 2024-02-28T16:17:21.533 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.1 metric 100 2024-02-28T16:17:21.546 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-28T16:17:21.546 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-28T16:17:21.547 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-28T16:17:21.559 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-28T16:17:21.559 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-28T16:17:21.560 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-28T16:17:21.560 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-28T16:17:21.560 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-02-28T16:17:21.560 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-28T16:17:21.562 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-02-28T16:17:21.562 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-02-28T16:17:21.563 INFO:teuthology.orchestra.run.smithi001.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-28T16:17:21.565 INFO:teuthology.orchestra.run.smithi001.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-28T16:17:21.568 INFO:teuthology.orchestra.run.smithi001.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-28T16:17:22.309 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 1c37ed3713489866ca403a4faaa38eb6c635a16f541c47d5bb849a203de99e6f 2024-02-28T16:17:22.309 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-28T16:17:22.309 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Getting image source signatures 2024-02-28T16:17:22.309 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:71fe8a47461413e338d0747cd84d29e39028a8869d35ba198bca4ef00bae3782 2024-02-28T16:17:22.309 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-02-28T16:17:22.309 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying config sha256:1c37ed3713489866ca403a4faaa38eb6c635a16f541c47d5bb849a203de99e6f 2024-02-28T16:17:22.310 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-28T16:17:23.499 INFO:teuthology.orchestra.run.smithi001.stdout:ceph: stdout ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable) 2024-02-28T16:17:23.499 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph version: ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable) 2024-02-28T16:17:23.499 INFO:teuthology.orchestra.run.smithi001.stdout:Extracting ceph user uid/gid from container image... 2024-02-28T16:17:33.642 INFO:teuthology.orchestra.run.smithi001.stdout:stat: stdout 167 167 2024-02-28T16:17:33.642 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial keys... 2024-02-28T16:17:34.776 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQCeXN9lluKcBxAAKXBKW1T28v94M2yNs/oAxg== 2024-02-28T16:17:35.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQCfXN9lG3NhDRAA0k8piXOCO+gfcHtwwxrcdQ== 2024-02-28T16:17:37.020 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQCgXN9l39giGBAAXed2/cz8PTnfrBwqdrBR0g== 2024-02-28T16:17:37.022 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial monmap... 2024-02-28T16:17:38.151 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-28T16:17:38.152 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-02-28T16:17:38.152 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:17:38.152 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-28T16:17:38.152 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool for smithi001 [v2:172.21.15.1:3300,v1:172.21.15.1:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-28T16:17:38.152 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = octopus 2024-02-28T16:17:38.152 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: set fsid to c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:17:38.152 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-28T16:17:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:17:38.153 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mon... 2024-02-28T16:17:40.280 INFO:teuthology.orchestra.run.smithi001.stdout:create mon.smithi001 on 2024-02-28T16:17:40.638 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-28T16:17:40.867 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-28T16:17:41.127 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c0118718-d654-11ee-95c3-87774f69a715.target → /etc/systemd/system/ceph-c0118718-d654-11ee-95c3-87774f69a715.target. 2024-02-28T16:17:41.127 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c0118718-d654-11ee-95c3-87774f69a715.target → /etc/systemd/system/ceph-c0118718-d654-11ee-95c3-87774f69a715.target. 2024-02-28T16:17:41.422 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001 2024-02-28T16:17:41.422 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service: Unit ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service not loaded. 2024-02-28T16:17:41.731 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c0118718-d654-11ee-95c3-87774f69a715.target.wants/ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service → /etc/systemd/system/ceph-c0118718-d654-11ee-95c3-87774f69a715@.service. 2024-02-28T16:17:42.370 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:42 smithi001 podman[28595]: 2024-02-28 16:17:42.379106384 +0000 UTC m=+0.382376865 container start eb136374ecd73fb3ef27dc869497b02d6ae1fc1f7e3596e2a510ca8e809ae4d6 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=quincy-c0f02e6) 2024-02-28T16:17:42.493 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-28T16:17:42.493 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-02-28T16:17:42.511 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-28T16:17:42.511 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-02-28T16:17:42.512 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-02-28T16:17:42.512 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-02-28T16:17:42.513 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon to start... 2024-02-28T16:17:42.513 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon... 2024-02-28T16:17:42.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:42 smithi001 ceph-mon[28614]: mkfs c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:17:42.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:42 smithi001 ceph-mon[28614]: mon.smithi001 is new leader, mons smithi001 in quorum (ranks 0) 2024-02-28T16:17:42.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:42 smithi001 bash[28595]: eb136374ecd73fb3ef27dc869497b02d6ae1fc1f7e3596e2a510ca8e809ae4d6 2024-02-28T16:17:42.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:42 smithi001 systemd[1]: Started Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:17:43.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:43 smithi001 ceph-mon[28614]: mon.smithi001 is new leader, mons smithi001 in quorum (ranks 0) 2024-02-28T16:17:43.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:43 smithi001 ceph-mon[28614]: monmap e1: 1 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-28T16:17:43.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:43 smithi001 ceph-mon[28614]: fsmap 2024-02-28T16:17:43.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:43 smithi001 ceph-mon[28614]: osdmap e1: 0 total, 0 up, 0 in 2024-02-28T16:17:43.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:43 smithi001 ceph-mon[28614]: mgrmap e1: no daemons active 2024-02-28T16:17:43.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:43 smithi001 ceph-mon[28614]: from='client.? 172.21.15.1:0/1890902047' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-28T16:17:43.814 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout cluster: 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout id: c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout services: 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi001 (age 0.74592s) 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout data: 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-28T16:17:43.815 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pgs: 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.104+0000 7f9f62200700 1 Processor -- start 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.104+0000 7f9f62200700 1 -- start start 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.104+0000 7f9f62200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.104+0000 7f9f62200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c1058c0 con 0x7f9f5c071480 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.105+0000 7f9f5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.105+0000 7f9f5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45056/0 (socket says 172.21.15.1:45056) 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.105+0000 7f9f5b400700 1 -- 172.21.15.1:0/4189697158 learned_addr learned my addr 172.21.15.1:0/4189697158 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.106+0000 7f9f5b400700 1 -- 172.21.15.1:0/4189697158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c105a00 con 0x7f9f5c071480 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.106+0000 7f9f5b400700 1 --2- 172.21.15.1:0/4189697158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c105380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0098e0 tx=0x7f9f4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6586bc5b4b56b3b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:43.816 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f5a000700 1 -- 172.21.15.1:0/4189697158 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4c00b860 con 0x7f9f5c071480 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f5a000700 1 -- 172.21.15.1:0/4189697158 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c00b9c0 con 0x7f9f5c071480 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f5a000700 1 -- 172.21.15.1:0/4189697158 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4c00bc90 con 0x7f9f5c071480 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f62200700 1 -- 172.21.15.1:0/4189697158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 msgr2=0x7f9f5c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f62200700 1 --2- 172.21.15.1:0/4189697158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c105380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0098e0 tx=0x7f9f4c009250 comp rx=0 tx=0).stop 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f62200700 1 -- 172.21.15.1:0/4189697158 shutdown_connections 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f62200700 1 --2- 172.21.15.1:0/4189697158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c105380 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f62200700 1 -- 172.21.15.1:0/4189697158 >> 172.21.15.1:0/4189697158 conn(0x7f9f5c0fcdd0 msgr2=0x7f9f5c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.107+0000 7f9f62200700 1 -- 172.21.15.1:0/4189697158 shutdown_connections 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.108+0000 7f9f62200700 1 -- 172.21.15.1:0/4189697158 wait complete. 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.108+0000 7f9f62200700 1 Processor -- start 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.108+0000 7f9f62200700 1 -- start start 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.109+0000 7f9f62200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c110e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.109+0000 7f9f62200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c111340 con 0x7f9f5c071480 2024-02-28T16:17:43.817 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.109+0000 7f9f5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c110e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.109+0000 7f9f5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c110e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45060/0 (socket says 172.21.15.1:45060) 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.109+0000 7f9f5b400700 1 -- 172.21.15.1:0/1890902047 learned_addr learned my addr 172.21.15.1:0/1890902047 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.110+0000 7f9f5b400700 1 -- 172.21.15.1:0/1890902047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0094f0 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.110+0000 7f9f5b400700 1 --2- 172.21.15.1:0/1890902047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c110e00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009290 tx=0x7f9f4c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.110+0000 7f9f53400700 1 -- 172.21.15.1:0/1890902047 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4c01c5c0 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.110+0000 7f9f53400700 1 -- 172.21.15.1:0/1890902047 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c01c740 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.110+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c111540 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.110+0000 7f9f53400700 1 -- 172.21.15.1:0/1890902047 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4c01ca50 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.110+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c112100 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.111+0000 7f9f53400700 1 -- 172.21.15.1:0/1890902047 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9f4c01a450 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.111+0000 7f9f53400700 1 -- 172.21.15.1:0/1890902047 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9f4c02e790 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.111+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c05f5c0 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.115+0000 7f9f53400700 1 -- 172.21.15.1:0/1890902047 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9f4c01ce30 con 0x7f9f5c071480 2024-02-28T16:17:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.194+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f9f5c105d10 con 0x7f9f5c071480 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.195+0000 7f9f53400700 1 -- 172.21.15.1:0/1890902047 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7f9f4c01a840 con 0x7f9f5c071480 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.196+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 msgr2=0x7f9f5c110e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.196+0000 7f9f62200700 1 --2- 172.21.15.1:0/1890902047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c110e00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009290 tx=0x7f9f4c016000 comp rx=0 tx=0).stop 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.196+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 shutdown_connections 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.196+0000 7f9f62200700 1 --2- 172.21.15.1:0/1890902047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f5c071480 0x7f9f5c110e00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.196+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 >> 172.21.15.1:0/1890902047 conn(0x7f9f5c0fcdd0 msgr2=0x7f9f5c0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.196+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 shutdown_connections 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:43.196+0000 7f9f62200700 1 -- 172.21.15.1:0/1890902047 wait complete. 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:mon is available 2024-02-28T16:17:43.819 INFO:teuthology.orchestra.run.smithi001.stdout:Assimilating anything we can from ceph.conf... 2024-02-28T16:17:44.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:44 smithi001 ceph-mon[28614]: from='client.? 172.21.15.1:0/3314472329' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-28T16:17:44.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:44 smithi001 ceph-mon[28614]: from='client.? 172.21.15.1:0/3314472329' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-28T16:17:45.351 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:17:45.351 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-02-28T16:17:45.351 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:17:45.351 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-28T16:17:45.352 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.408+0000 7fa4dee00700 1 Processor -- start 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.409+0000 7fa4dee00700 1 -- start start 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.409+0000 7fa4dee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.409+0000 7fa4dee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d81030c0 con 0x7fa4d8102770 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.410+0000 7fa4d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.410+0000 7fa4d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45062/0 (socket says 172.21.15.1:45062) 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.410+0000 7fa4d7e00700 1 -- 172.21.15.1:0/472477362 learned_addr learned my addr 172.21.15.1:0/472477362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.411+0000 7fa4d7e00700 1 -- 172.21.15.1:0/472477362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d8103200 con 0x7fa4d8102770 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.411+0000 7fa4d7e00700 1 --2- 172.21.15.1:0/472477362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8102b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80098e0 tx=0x7fa4c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319aaa097bb8b96d server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.411+0000 7fa4d6a00700 1 -- 172.21.15.1:0/472477362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4c800b860 con 0x7fa4d8102770 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.411+0000 7fa4d6a00700 1 -- 172.21.15.1:0/472477362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c800b9c0 con 0x7fa4d8102770 2024-02-28T16:17:45.353 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.411+0000 7fa4d6a00700 1 -- 172.21.15.1:0/472477362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4c800bc90 con 0x7fa4d8102770 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.412+0000 7fa4dee00700 1 -- 172.21.15.1:0/472477362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 msgr2=0x7fa4d8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.412+0000 7fa4dee00700 1 --2- 172.21.15.1:0/472477362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8102b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80098e0 tx=0x7fa4c8009250 comp rx=0 tx=0).stop 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.412+0000 7fa4dee00700 1 -- 172.21.15.1:0/472477362 shutdown_connections 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.412+0000 7fa4dee00700 1 --2- 172.21.15.1:0/472477362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8102b80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.412+0000 7fa4dee00700 1 -- 172.21.15.1:0/472477362 >> 172.21.15.1:0/472477362 conn(0x7fa4d80fded0 msgr2=0x7fa4d8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.412+0000 7fa4dee00700 1 -- 172.21.15.1:0/472477362 shutdown_connections 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.412+0000 7fa4dee00700 1 -- 172.21.15.1:0/472477362 wait complete. 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.413+0000 7fa4dee00700 1 Processor -- start 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.413+0000 7fa4dee00700 1 -- start start 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.414+0000 7fa4dee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8112f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.414+0000 7fa4dee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d81134b0 con 0x7fa4d8102770 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.414+0000 7fa4d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8112f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.414+0000 7fa4d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8112f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45070/0 (socket says 172.21.15.1:45070) 2024-02-28T16:17:45.354 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.414+0000 7fa4d7e00700 1 -- 172.21.15.1:0/3314472329 learned_addr learned my addr 172.21.15.1:0/3314472329 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.414+0000 7fa4d7e00700 1 -- 172.21.15.1:0/3314472329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c80094f0 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.415+0000 7fa4d7e00700 1 --2- 172.21.15.1:0/3314472329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8112f70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8003080 tx=0x7fa4c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.415+0000 7fa4cfe00700 1 -- 172.21.15.1:0/3314472329 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4c801c5c0 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.415+0000 7fa4cfe00700 1 -- 172.21.15.1:0/3314472329 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c801c720 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.415+0000 7fa4cfe00700 1 -- 172.21.15.1:0/3314472329 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4c801ca30 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.415+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d81136b0 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.415+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d8114350 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.415+0000 7fa4cfe00700 1 -- 172.21.15.1:0/3314472329 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa4c801a450 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.416+0000 7fa4cfe00700 1 -- 172.21.15.1:0/3314472329 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa4c802e790 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.416+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d805f5c0 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.419+0000 7fa4cfe00700 1 -- 172.21.15.1:0/3314472329 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa4c801a890 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.496+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa4d8103c70 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.504+0000 7fa4cfe00700 1 -- 172.21.15.1:0/3314472329 <== mon.0 v2:172.21.15.1:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa4c802ee80 con 0x7fa4d8102770 2024-02-28T16:17:45.355 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.504+0000 7fa4cfe00700 1 -- 172.21.15.1:0/3314472329 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7fa4d8103c70 con 0x7fa4d8102770 2024-02-28T16:17:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.505+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 msgr2=0x7fa4d8112f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.505+0000 7fa4dee00700 1 --2- 172.21.15.1:0/3314472329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8112f70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8003080 tx=0x7fa4c8016000 comp rx=0 tx=0).stop 2024-02-28T16:17:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.506+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 shutdown_connections 2024-02-28T16:17:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.506+0000 7fa4dee00700 1 --2- 172.21.15.1:0/3314472329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d8102770 0x7fa4d8112f70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.506+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 >> 172.21.15.1:0/3314472329 conn(0x7fa4d80fded0 msgr2=0x7fa4d8078b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.506+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 shutdown_connections 2024-02-28T16:17:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:44.506+0000 7fa4dee00700 1 -- 172.21.15.1:0/3314472329 wait complete. 2024-02-28T16:17:45.356 INFO:teuthology.orchestra.run.smithi001.stdout:Generating new minimal ceph.conf... 2024-02-28T16:17:46.217 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:46 smithi001 ceph-mon[28614]: from='client.? 172.21.15.1:0/603926885' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:17:46.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.943+0000 7f63af600700 1 Processor -- start 2024-02-28T16:17:46.654 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.943+0000 7f63af600700 1 -- start start 2024-02-28T16:17:46.654 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.944+0000 7f63af600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a8071460 0x7f63a81077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:46.654 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.944+0000 7f63af600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8103190 con 0x7f63a8071460 2024-02-28T16:17:46.654 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.944+0000 7f63acc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a8071460 0x7f63a81077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.945+0000 7f63acc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a8071460 0x7f63a81077a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45080/0 (socket says 172.21.15.1:45080) 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.945+0000 7f63acc00700 1 -- 172.21.15.1:0/3538136190 learned_addr learned my addr 172.21.15.1:0/3538136190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.945+0000 7f63acc00700 1 -- 172.21.15.1:0/3538136190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a8107ce0 con 0x7f63a8071460 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.945+0000 7f63acc00700 1 --2- 172.21.15.1:0/3538136190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a8071460 0x7f63a81077a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f639c0098e0 tx=0x7f639c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27074d6c04bb508d server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.946+0000 7f63a7400700 1 -- 172.21.15.1:0/3538136190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f639c00b860 con 0x7f63a8071460 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.946+0000 7f63a7400700 1 -- 172.21.15.1:0/3538136190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f639c00be50 con 0x7f63a8071460 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.946+0000 7f63a7400700 1 -- 172.21.15.1:0/3538136190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f639c010380 con 0x7f63a8071460 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.946+0000 7f63af600700 1 -- 172.21.15.1:0/3538136190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a8071460 msgr2=0x7f63a81077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.946+0000 7f63af600700 1 --2- 172.21.15.1:0/3538136190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a8071460 0x7f63a81077a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f639c0098e0 tx=0x7f639c009250 comp rx=0 tx=0).stop 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.947+0000 7f63af600700 1 -- 172.21.15.1:0/3538136190 shutdown_connections 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.947+0000 7f63af600700 1 --2- 172.21.15.1:0/3538136190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a8071460 0x7f63a81077a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:46.655 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.947+0000 7f63af600700 1 -- 172.21.15.1:0/3538136190 >> 172.21.15.1:0/3538136190 conn(0x7f63a80fcdd0 msgr2=0x7f63a80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.947+0000 7f63af600700 1 -- 172.21.15.1:0/3538136190 shutdown_connections 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.947+0000 7f63af600700 1 -- 172.21.15.1:0/3538136190 wait complete. 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.948+0000 7f63af600700 1 Processor -- start 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.948+0000 7f63af600700 1 -- start start 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.949+0000 7f63af600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a81138e0 0x7f63a8113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.949+0000 7f63af600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8114230 con 0x7f63a81138e0 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.949+0000 7f63acc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a81138e0 0x7f63a8113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.949+0000 7f63acc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a81138e0 0x7f63a8113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45096/0 (socket says 172.21.15.1:45096) 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.949+0000 7f63acc00700 1 -- 172.21.15.1:0/603926885 learned_addr learned my addr 172.21.15.1:0/603926885 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.949+0000 7f63acc00700 1 -- 172.21.15.1:0/603926885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c0094f0 con 0x7f63a81138e0 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.950+0000 7f63acc00700 1 --2- 172.21.15.1:0/603926885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a81138e0 0x7f63a8113cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63a80711a0 tx=0x7f639c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.950+0000 7f63a4c00700 1 -- 172.21.15.1:0/603926885 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f639c010380 con 0x7f63a81138e0 2024-02-28T16:17:46.656 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.950+0000 7f63a4c00700 1 -- 172.21.15.1:0/603926885 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f639c01f070 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.950+0000 7f63a4c00700 1 -- 172.21.15.1:0/603926885 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f639c01aa10 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.950+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a8114430 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.950+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a8114f40 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.950+0000 7f63a4c00700 1 -- 172.21.15.1:0/603926885 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f639c01ab70 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.951+0000 7f63a4c00700 1 -- 172.21.15.1:0/603926885 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f639c021070 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.952+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a805f5c0 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:45.954+0000 7f63a4c00700 1 -- 172.21.15.1:0/603926885 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f639c01a440 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.031+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f63a8115a90 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.031+0000 7f63a4c00700 1 -- 172.21.15.1:0/603926885 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f639c026480 con 0x7f63a81138e0 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.033+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a81138e0 msgr2=0x7f63a8113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.033+0000 7f63af600700 1 --2- 172.21.15.1:0/603926885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a81138e0 0x7f63a8113cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63a80711a0 tx=0x7f639c009250 comp rx=0 tx=0).stop 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.033+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 shutdown_connections 2024-02-28T16:17:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.033+0000 7f63af600700 1 --2- 172.21.15.1:0/603926885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a81138e0 0x7f63a8113cf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:46.658 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.033+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 >> 172.21.15.1:0/603926885 conn(0x7f63a80fcdd0 msgr2=0x7f63a80fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:46.658 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.033+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 shutdown_connections 2024-02-28T16:17:46.658 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:46.033+0000 7f63af600700 1 -- 172.21.15.1:0/603926885 wait complete. 2024-02-28T16:17:46.658 INFO:teuthology.orchestra.run.smithi001.stdout:Restarting the monitor... 2024-02-28T16:17:47.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:46 smithi001 systemd[1]: Stopping Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:17:47.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[28610]: 2024-02-28T16:17:47.141+0000 7f729fa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi001 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:17:47.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[28610]: 2024-02-28T16:17:47.141+0000 7f729fa00700 -1 mon.smithi001@0(leader) e1 *** Got Signal Terminated *** 2024-02-28T16:17:47.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:47 smithi001 podman[28867]: 2024-02-28 16:17:47.264823705 +0000 UTC m=+0.252055371 container died eb136374ecd73fb3ef27dc869497b02d6ae1fc1f7e3596e2a510ca8e809ae4d6 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, GIT_BRANCH=HEAD, RELEASE=quincy-c0f02e6, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-28T16:17:47.859 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:47 smithi001 podman[28867]: 2024-02-28 16:17:47.666755812 +0000 UTC m=+0.653987468 container cleanup eb136374ecd73fb3ef27dc869497b02d6ae1fc1f7e3596e2a510ca8e809ae4d6 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=quincy-c0f02e6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-28T16:17:47.859 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:47 smithi001 bash[28867]: ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001 2024-02-28T16:17:48.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:47 smithi001 podman[28882]: 2024-02-28 16:17:47.867431914 +0000 UTC m=+0.591546696 container remove eb136374ecd73fb3ef27dc869497b02d6ae1fc1f7e3596e2a510ca8e809ae4d6 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=quincy-c0f02e6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-28T16:17:48.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:48 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service: Deactivated successfully. 2024-02-28T16:17:48.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:48 smithi001 systemd[1]: Stopped Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:17:48.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:48 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service: Consumed 1.904s CPU time. 2024-02-28T16:17:48.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:48 smithi001 systemd[1]: Starting Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:17:49.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:48 smithi001 podman[28972]: 2024-02-28 16:17:48.639447087 +0000 UTC m=+0.036947036 image pull 1c37ed3713489866ca403a4faaa38eb6c635a16f541c47d5bb849a203de99e6f quay.ceph.io/ceph-ci/ceph:quincy 2024-02-28T16:17:49.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:48 smithi001 podman[28972]: 2024-02-28 16:17:48.751824541 +0000 UTC m=+0.149324532 container create 4d5c15828f14e4d1fcfbdd566b05a1e164b76046386d9ea2029cede00c0fd2fc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, RELEASE=quincy-c0f02e6, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-28T16:17:49.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 podman[28972]: 2024-02-28 16:17:49.030749898 +0000 UTC m=+0.428249899 container init 4d5c15828f14e4d1fcfbdd566b05a1e164b76046386d9ea2029cede00c0fd2fc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=quincy-c0f02e6, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-28T16:17:49.118 INFO:teuthology.orchestra.run.smithi001.stdout:Setting mon public_network to 172.21.0.0/20 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 podman[28972]: 2024-02-28 16:17:49.042257229 +0000 UTC m=+0.439757200 container start 4d5c15828f14e4d1fcfbdd566b05a1e164b76046386d9ea2029cede00c0fd2fc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, RELEASE=quincy-c0f02e6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: set uid:gid to 167:167 (ceph:ceph) 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable), process ceph-mon, pid 2 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: pidfile_write: ignore empty --pid-file 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: load: jerasure load: lrc 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: RocksDB version: 6.15.5 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Compile date Feb 26 2024 2024-02-28T16:17:49.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: DB SUMMARY 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: DB Session ID: 3AMN3BQGJGTB1PALNXOX 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: CURRENT file: CURRENT 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: IDENTITY file: IDENTITY 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi001/store.db dir, Total Num: 1, files: 000008.sst 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi001/store.db: 000010.log size: 97254 ; 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.error_if_exists: 0 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.create_if_missing: 0 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.paranoid_checks: 1 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.env: 0x55a34d9d7a80 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.fs: Posix File System 2024-02-28T16:17:49.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.info_log: 0x55a34f23bcc0 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_file_opening_threads: 16 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.statistics: (nil) 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.use_fsync: 0 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_log_file_size: 0 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.keep_log_file_num: 1000 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.recycle_log_file_num: 0 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.allow_fallocate: 1 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.allow_mmap_reads: 0 2024-02-28T16:17:49.275 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.allow_mmap_writes: 0 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.use_direct_reads: 0 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.create_missing_column_families: 0 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.db_log_dir: 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi001/store.db 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-28T16:17:49.276 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.advise_random_on_open: 1 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.db_write_buffer_size: 0 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.write_buffer_manager: 0x55a34ffb0b40 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.rate_limiter: (nil) 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-28T16:17:49.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.wal_recovery_mode: 2 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.enable_thread_tracking: 0 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.enable_pipelined_write: 0 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.unordered_write: 0 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.row_cache: None 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.wal_filter: None 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.allow_ingest_behind: 0 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.preserve_deletes: 0 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.two_write_queues: 0 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.manual_wal_flush: 0 2024-02-28T16:17:49.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.atomic_flush: 0 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.log_readahead_size: 0 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.best_efforts_recovery: 0 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.allow_data_in_errors: 0 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.db_host_id: __hostname__ 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_background_jobs: 2 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_background_compactions: -1 2024-02-28T16:17:49.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_subcompactions: 1 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_total_wal_size: 0 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_open_files: -1 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bytes_per_sync: 0 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-28T16:17:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_readahead_size: 0 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_background_flushes: -1 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Compression algorithms supported: 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: kZSTD supported: 0 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: kXpressCompression supported: 0 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: kLZ4HCCompression supported: 1 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: kZlibCompression supported: 1 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: kSnappyCompression supported: 1 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: kLZ4Compression supported: 1 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: kBZip2Compression supported: 0 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi001/store.db/MANIFEST-000009 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-28T16:17:49.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.merge_operator: 2024-02-28T16:17:49.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_filter: None 2024-02-28T16:17:49.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_filter_factory: None 2024-02-28T16:17:49.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.sst_partitioner_factory: None 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a34f290f58) 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: cache_index_and_filter_blocks: 1 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: pin_top_level_index_and_filter: 1 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_type: 0 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: data_block_index_type: 0 2024-02-28T16:17:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_shortening: 1 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: hash_index_allow_collision: 1 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: checksum: 1 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: no_block_cache: 0 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache: 0x55a34f2c3090 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_name: BinnedLRUCache 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_options: 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: capacity : 536870912 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: num_shard_bits : 4 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: strict_capacity_limit : 0 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: high_pri_pool_ratio: 0.000 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_compressed: (nil) 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: persistent_cache: (nil) 2024-02-28T16:17:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_size: 4096 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_size_deviation: 10 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_restart_interval: 16 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_block_restart_interval: 1 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: metadata_block_size: 4096 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: partition_filters: 0 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: use_delta_encoding: 1 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: whole_key_filtering: 1 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: verify_compression: 0 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: read_amp_bytes_per_bit: 0 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: format_version: 4 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: enable_index_compression: 1 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_align: 0 2024-02-28T16:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.write_buffer_size: 33554432 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_write_buffer_number: 2 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compression: NoCompression 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bottommost_compression: Disabled 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.prefix_extractor: nullptr 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.num_levels: 7 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-28T16:17:49.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compression_opts.level: 32767 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compression_opts.strategy: 0 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compression_opts.enabled: false 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-28T16:17:49.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.target_file_size_base: 67108864 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-28T16:17:49.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.arena_block_size: 4194304 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.disable_auto_compactions: 0 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-28T16:17:49.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.table_properties_collectors: 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.inplace_update_support: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.bloom_locality: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.max_successive_merges: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.paranoid_file_checks: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.force_consistency_checks: 1 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.report_bg_io_stats: 0 2024-02-28T16:17:49.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.ttl: 2592000 2024-02-28T16:17:49.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-28T16:17:49.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.enable_blob_files: false 2024-02-28T16:17:49.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.min_blob_size: 0 2024-02-28T16:17:49.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.blob_file_size: 268435456 2024-02-28T16:17:49.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi001/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709137069097197, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-28T16:17:49.292 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709137069099649, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92526, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90621, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83336, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1709137069, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "dd7c6d90-7103-461d-9dee-f19804439cf4", "db_session_id": "3AMN3BQGJGTB1PALNXOX"}} 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709137069100741, "job": 1, "event": "recovery_finished"} 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi001/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55a34f257180 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: DB pointer 0x55a34f32c000 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** DB Stats ** 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-28T16:17:49.293 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: L0 2/0 92.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.2 0.00 0.00 1 0.002 0 0 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Sum 2/0 92.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.2 0.00 0.00 1 0.002 0 0 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.2 0.00 0.00 1 0.002 0 0 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-28T16:17:49.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 58.2 0.00 0.00 1 0.002 0 0 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative compaction: 0.00 GB write, 9.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval compaction: 0.00 GB write, 9.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-28T16:17:49.295 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: L0 2/0 92.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.2 0.00 0.00 1 0.002 0 0 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Sum 2/0 92.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.2 0.00 0.00 1 0.002 0 0 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 58.2 0.00 0.00 1 0.002 0 0 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-28T16:17:49.296 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative compaction: 0.00 GB write, 9.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: starting mon.smithi001 rank 0 at public addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] at bind addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi001 fsid c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001@-1(???) e1 preinit fsid c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001@-1(???).mds e1 new map 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001@-1(???).mds e1 print_map 2024-02-28T16:17:49.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: e1 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: legacy client fscid: -1 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: No filesystems configured 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 bash[28972]: 4d5c15828f14e4d1fcfbdd566b05a1e164b76046386d9ea2029cede00c0fd2fc 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 systemd[1]: Started Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mon.smithi001 is new leader, mons smithi001 in quorum (ranks 0) 2024-02-28T16:17:49.298 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: monmap e1: 1 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-28T16:17:49.299 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: fsmap 2024-02-28T16:17:49.299 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: osdmap e1: 0 total, 0 up, 0 in 2024-02-28T16:17:49.299 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:49 smithi001 ceph-mon[28991]: mgrmap e1: no daemons active 2024-02-28T16:17:50.531 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.792+0000 7f48da400700 1 Processor -- start 2024-02-28T16:17:50.533 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.792+0000 7f48da400700 1 -- start start 2024-02-28T16:17:50.533 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.793+0000 7f48da400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d40760c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:50.533 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.793+0000 7f48da400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4077fc0 con 0x7f48d4077bb0 2024-02-28T16:17:50.533 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.794+0000 7f48d3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d40760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:50.533 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.794+0000 7f48d3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d40760c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45112/0 (socket says 172.21.15.1:45112) 2024-02-28T16:17:50.533 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.794+0000 7f48d3400700 1 -- 172.21.15.1:0/3503126631 learned_addr learned my addr 172.21.15.1:0/3503126631 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:50.534 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.795+0000 7f48d3400700 1 -- 172.21.15.1:0/3503126631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d4076600 con 0x7f48d4077bb0 2024-02-28T16:17:50.534 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.795+0000 7f48d3400700 1 --2- 172.21.15.1:0/3503126631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d40760c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f48c40098e0 tx=0x7f48c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246cfe5dcdac0830 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:50.534 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.795+0000 7f48d2000700 1 -- 172.21.15.1:0/3503126631 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c400b860 con 0x7f48d4077bb0 2024-02-28T16:17:50.534 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.795+0000 7f48d2000700 1 -- 172.21.15.1:0/3503126631 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f48c400be50 con 0x7f48d4077bb0 2024-02-28T16:17:50.534 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.795+0000 7f48d2000700 1 -- 172.21.15.1:0/3503126631 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c4010380 con 0x7f48d4077bb0 2024-02-28T16:17:50.534 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.796+0000 7f48da400700 1 -- 172.21.15.1:0/3503126631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 msgr2=0x7f48d40760c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:50.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.796+0000 7f48da400700 1 --2- 172.21.15.1:0/3503126631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d40760c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f48c40098e0 tx=0x7f48c4009250 comp rx=0 tx=0).stop 2024-02-28T16:17:50.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.796+0000 7f48da400700 1 -- 172.21.15.1:0/3503126631 shutdown_connections 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.796+0000 7f48da400700 1 --2- 172.21.15.1:0/3503126631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d40760c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.796+0000 7f48da400700 1 -- 172.21.15.1:0/3503126631 >> 172.21.15.1:0/3503126631 conn(0x7f48d40fded0 msgr2=0x7f48d4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.796+0000 7f48da400700 1 -- 172.21.15.1:0/3503126631 shutdown_connections 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.796+0000 7f48da400700 1 -- 172.21.15.1:0/3503126631 wait complete. 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.797+0000 7f48da400700 1 Processor -- start 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.797+0000 7f48da400700 1 -- start start 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.798+0000 7f48da400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d4113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.798+0000 7f48da400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4113fc0 con 0x7f48d4077bb0 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.798+0000 7f48d3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d4113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.798+0000 7f48d3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d4113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45122/0 (socket says 172.21.15.1:45122) 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.798+0000 7f48d3400700 1 -- 172.21.15.1:0/3568932490 learned_addr learned my addr 172.21.15.1:0/3568932490 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:50.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.799+0000 7f48d3400700 1 -- 172.21.15.1:0/3568932490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c40094f0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.799+0000 7f48d3400700 1 --2- 172.21.15.1:0/3568932490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d4113a80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f48c4006fd0 tx=0x7f48c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.799+0000 7f48cb400700 1 -- 172.21.15.1:0/3568932490 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c40107c0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.799+0000 7f48cb400700 1 -- 172.21.15.1:0/3568932490 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f48c4010db0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.799+0000 7f48cb400700 1 -- 172.21.15.1:0/3568932490 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c401add0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.799+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d41141c0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.799+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d4114e60 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.800+0000 7f48cb400700 1 -- 172.21.15.1:0/3568932490 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f48c40216e0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.800+0000 7f48cb400700 1 -- 172.21.15.1:0/3568932490 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f48c401e070 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.801+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d405f5c0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.804+0000 7f48cb400700 1 -- 172.21.15.1:0/3568932490 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f48c401a5c0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.882+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f48d404bed0 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.888+0000 7f48cb400700 1 -- 172.21.15.1:0/3568932490 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f48c4010920 con 0x7f48d4077bb0 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.889+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 msgr2=0x7f48d4113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:50.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.889+0000 7f48da400700 1 --2- 172.21.15.1:0/3568932490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d4113a80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f48c4006fd0 tx=0x7f48c4009250 comp rx=0 tx=0).stop 2024-02-28T16:17:50.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.889+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 shutdown_connections 2024-02-28T16:17:50.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.889+0000 7f48da400700 1 --2- 172.21.15.1:0/3568932490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48d4077bb0 0x7f48d4113a80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:50.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.889+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 >> 172.21.15.1:0/3568932490 conn(0x7f48d40fded0 msgr2=0x7f48d40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:50.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.889+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 shutdown_connections 2024-02-28T16:17:50.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:49.889+0000 7f48da400700 1 -- 172.21.15.1:0/3568932490 wait complete. 2024-02-28T16:17:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-28T16:17:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-28T16:17:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mgr... 2024-02-28T16:17:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying port 9283 ... 2024-02-28T16:17:50.937 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c0118718-d654-11ee-95c3-87774f69a715@mgr.smithi001.rmthtx 2024-02-28T16:17:50.937 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c0118718-d654-11ee-95c3-87774f69a715@mgr.smithi001.rmthtx.service: Unit ceph-c0118718-d654-11ee-95c3-87774f69a715@mgr.smithi001.rmthtx.service not loaded. 2024-02-28T16:17:51.210 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c0118718-d654-11ee-95c3-87774f69a715.target.wants/ceph-c0118718-d654-11ee-95c3-87774f69a715@mgr.smithi001.rmthtx.service → /etc/systemd/system/ceph-c0118718-d654-11ee-95c3-87774f69a715@.service. 2024-02-28T16:17:51.238 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:50 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3568932490' entity='client.admin' 2024-02-28T16:17:52.056 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-28T16:17:52.056 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-02-28T16:17:52.078 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-28T16:17:52.078 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-02-28T16:17:52.079 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-02-28T16:17:52.079 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-02-28T16:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-28T16:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-02-28T16:17:52.129 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-28T16:17:52.129 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-02-28T16:17:52.129 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-02-28T16:17:52.129 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-28T16:17:52.129 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr to start... 2024-02-28T16:17:52.130 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr... 2024-02-28T16:17:53.236 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:52 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/427281512' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-28T16:17:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:17:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-02-28T16:17:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "c0118718-d654-11ee-95c3-87774f69a715", 2024-02-28T16:17:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "smithi001" 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-02-28T16:17:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-28T16:17:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-28T16:17:53.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-28T16:17:53.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-02-28T16:17:42.450989+0000", 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.819+0000 7fb6fac00700 1 Processor -- start 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.819+0000 7fb6fac00700 1 -- start start 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.819+0000 7fb6fac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.819+0000 7fb6fac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f41030c0 con 0x7fb6f4102770 2024-02-28T16:17:53.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.820+0000 7fb6f3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.820+0000 7fb6f3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44500/0 (socket says 172.21.15.1:44500) 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.820+0000 7fb6f3e00700 1 -- 172.21.15.1:0/1489039870 learned_addr learned my addr 172.21.15.1:0/1489039870 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.820+0000 7fb6f3e00700 1 -- 172.21.15.1:0/1489039870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f4103200 con 0x7fb6f4102770 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.821+0000 7fb6f3e00700 1 --2- 172.21.15.1:0/1489039870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4102b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009e50 tx=0x7fb6e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7295b4b8bac2a642 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.821+0000 7fb6f2a00700 1 -- 172.21.15.1:0/1489039870 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6e40092e0 con 0x7fb6f4102770 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.821+0000 7fb6f2a00700 1 -- 172.21.15.1:0/1489039870 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb6e4012430 con 0x7fb6f4102770 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.821+0000 7fb6f2a00700 1 -- 172.21.15.1:0/1489039870 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6e40075c0 con 0x7fb6f4102770 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.822+0000 7fb6fac00700 1 -- 172.21.15.1:0/1489039870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 msgr2=0x7fb6f4102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.822+0000 7fb6fac00700 1 --2- 172.21.15.1:0/1489039870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4102b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009e50 tx=0x7fb6e4009840 comp rx=0 tx=0).stop 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.822+0000 7fb6fac00700 1 -- 172.21.15.1:0/1489039870 shutdown_connections 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.822+0000 7fb6fac00700 1 --2- 172.21.15.1:0/1489039870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4102b80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.822+0000 7fb6fac00700 1 -- 172.21.15.1:0/1489039870 >> 172.21.15.1:0/1489039870 conn(0x7fb6f40fded0 msgr2=0x7fb6f4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:53.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.822+0000 7fb6fac00700 1 -- 172.21.15.1:0/1489039870 shutdown_connections 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.822+0000 7fb6fac00700 1 -- 172.21.15.1:0/1489039870 wait complete. 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.823+0000 7fb6fac00700 1 Processor -- start 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.823+0000 7fb6fac00700 1 -- start start 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.824+0000 7fb6fac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4077c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.824+0000 7fb6fac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f4076080 con 0x7fb6f4102770 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.824+0000 7fb6f3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4077c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.824+0000 7fb6f3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4077c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44516/0 (socket says 172.21.15.1:44516) 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.824+0000 7fb6f3e00700 1 -- 172.21.15.1:0/427281512 learned_addr learned my addr 172.21.15.1:0/427281512 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.824+0000 7fb6f3e00700 1 -- 172.21.15.1:0/427281512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4009ae0 con 0x7fb6f4102770 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.825+0000 7fb6f3e00700 1 --2- 172.21.15.1:0/427281512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4077c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4000f80 tx=0x7fb6e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.825+0000 7fb6ebe00700 1 -- 172.21.15.1:0/427281512 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6e40092e0 con 0x7fb6f4102770 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.825+0000 7fb6ebe00700 1 -- 172.21.15.1:0/427281512 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb6e4007cc0 con 0x7fb6f4102770 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.825+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f4076280 con 0x7fb6f4102770 2024-02-28T16:17:53.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.825+0000 7fb6ebe00700 1 -- 172.21.15.1:0/427281512 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6e401c990 con 0x7fb6f4102770 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.825+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f4076770 con 0x7fb6f4102770 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.826+0000 7fb6ebe00700 1 -- 172.21.15.1:0/427281512 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb6e4019070 con 0x7fb6f4102770 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.826+0000 7fb6ebe00700 1 -- 172.21.15.1:0/427281512 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb6e4025b60 con 0x7fb6f4102770 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.827+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f410b670 con 0x7fb6f4102770 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.829+0000 7fb6ebe00700 1 -- 172.21.15.1:0/427281512 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb6e402a060 con 0x7fb6f4102770 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.907+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb6f404bed0 con 0x7fb6f4102770 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.907+0000 7fb6ebe00700 1 -- 172.21.15.1:0/427281512 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fb6e4020070 con 0x7fb6f4102770 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.909+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 msgr2=0x7fb6f4077c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.909+0000 7fb6fac00700 1 --2- 172.21.15.1:0/427281512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4077c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4000f80 tx=0x7fb6e4009840 comp rx=0 tx=0).stop 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.909+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 shutdown_connections 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.909+0000 7fb6fac00700 1 --2- 172.21.15.1:0/427281512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6f4102770 0x7fb6f4077c00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.909+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 >> 172.21.15.1:0/427281512 conn(0x7fb6f40fded0 msgr2=0x7fb6f40fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.909+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 shutdown_connections 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:52.909+0000 7fb6fac00700 1 -- 172.21.15.1:0/427281512 wait complete. 2024-02-28T16:17:53.503 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (1/15)... 2024-02-28T16:17:56.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:17:56 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/337872075' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-28T16:17:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:17:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-02-28T16:17:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "c0118718-d654-11ee-95c3-87774f69a715", 2024-02-28T16:17:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-02-28T16:17:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-28T16:17:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-28T16:17:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-28T16:17:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "smithi001" 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:56.987 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-28T16:17:56.988 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-28T16:17:56.989 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-02-28T16:17:42.450989+0000", 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.123+0000 7f3a10200700 1 Processor -- start 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.123+0000 7f3a10200700 1 -- start start 2024-02-28T16:17:56.990 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.124+0000 7f3a10200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a080760a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.124+0000 7f3a10200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08077fa0 con 0x7f3a08077b90 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.124+0000 7f3a0d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a080760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.124+0000 7f3a0d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a080760a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44530/0 (socket says 172.21.15.1:44530) 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.124+0000 7f3a0d800700 1 -- 172.21.15.1:0/1226486091 learned_addr learned my addr 172.21.15.1:0/1226486091 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.125+0000 7f3a0d800700 1 -- 172.21.15.1:0/1226486091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a080765e0 con 0x7f3a08077b90 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.125+0000 7f3a0d800700 1 --2- 172.21.15.1:0/1226486091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a080760a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0098e0 tx=0x7f39fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb68cdff4dcb11d server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.125+0000 7f3a07e00700 1 -- 172.21.15.1:0/1226486091 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc00b860 con 0x7f3a08077b90 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.125+0000 7f3a07e00700 1 -- 172.21.15.1:0/1226486091 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f39fc00be50 con 0x7f3a08077b90 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.125+0000 7f3a07e00700 1 -- 172.21.15.1:0/1226486091 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc010380 con 0x7f3a08077b90 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.126+0000 7f3a10200700 1 -- 172.21.15.1:0/1226486091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 msgr2=0x7f3a080760a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:56.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.126+0000 7f3a10200700 1 --2- 172.21.15.1:0/1226486091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a080760a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0098e0 tx=0x7f39fc009250 comp rx=0 tx=0).stop 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.126+0000 7f3a10200700 1 -- 172.21.15.1:0/1226486091 shutdown_connections 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.126+0000 7f3a10200700 1 --2- 172.21.15.1:0/1226486091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a080760a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.126+0000 7f3a10200700 1 -- 172.21.15.1:0/1226486091 >> 172.21.15.1:0/1226486091 conn(0x7f3a080fded0 msgr2=0x7f3a08100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.126+0000 7f3a10200700 1 -- 172.21.15.1:0/1226486091 shutdown_connections 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.126+0000 7f3a10200700 1 -- 172.21.15.1:0/1226486091 wait complete. 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.127+0000 7f3a10200700 1 Processor -- start 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.128+0000 7f3a10200700 1 -- start start 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.128+0000 7f3a10200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a08113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.128+0000 7f3a10200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08113f70 con 0x7f3a08077b90 2024-02-28T16:17:56.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.128+0000 7f3a0d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a08113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.128+0000 7f3a0d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a08113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44532/0 (socket says 172.21.15.1:44532) 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.128+0000 7f3a0d800700 1 -- 172.21.15.1:0/337872075 learned_addr learned my addr 172.21.15.1:0/337872075 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.129+0000 7f3a0d800700 1 -- 172.21.15.1:0/337872075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc0094f0 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.129+0000 7f3a0d800700 1 --2- 172.21.15.1:0/337872075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a08113a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00b440 tx=0x7f39fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.129+0000 7f3a05600700 1 -- 172.21.15.1:0/337872075 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc0107c0 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.129+0000 7f3a05600700 1 -- 172.21.15.1:0/337872075 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f39fc010db0 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.129+0000 7f3a05600700 1 -- 172.21.15.1:0/337872075 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc01ad30 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.129+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a08114170 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.130+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a08114e10 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.130+0000 7f3a05600700 1 -- 172.21.15.1:0/337872075 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f39fc02e440 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.130+0000 7f3a05600700 1 -- 172.21.15.1:0/337872075 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f39fc024740 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.131+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0805f5c0 con 0x7f3a08077b90 2024-02-28T16:17:56.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.133+0000 7f3a05600700 1 -- 172.21.15.1:0/337872075 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f39fc010ba0 con 0x7f3a08077b90 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.212+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3a08108a70 con 0x7f3a08077b90 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.213+0000 7f3a05600700 1 -- 172.21.15.1:0/337872075 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f39fc01a430 con 0x7f3a08077b90 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.214+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 msgr2=0x7f3a08113a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.214+0000 7f3a10200700 1 --2- 172.21.15.1:0/337872075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a08113a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00b440 tx=0x7f39fc009250 comp rx=0 tx=0).stop 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.214+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 shutdown_connections 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.214+0000 7f3a10200700 1 --2- 172.21.15.1:0/337872075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a08077b90 0x7f3a08113a30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.214+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 >> 172.21.15.1:0/337872075 conn(0x7f3a080fded0 msgr2=0x7f3a081001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.214+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 shutdown_connections 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:56.214+0000 7f3a10200700 1 -- 172.21.15.1:0/337872075 wait complete. 2024-02-28T16:17:56.997 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (2/15)... 2024-02-28T16:18:00.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:00 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1069705645' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-28T16:18:00.592 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:18:00.592 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-02-28T16:18:00.592 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "c0118718-d654-11ee-95c3-87774f69a715", 2024-02-28T16:18:00.592 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-02-28T16:18:00.592 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "smithi001" 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:18:00.593 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-28T16:18:00.594 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-28T16:18:00.595 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-02-28T16:17:42.450989+0000", 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-28T16:18:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.545+0000 7f4d65e00700 1 Processor -- start 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.545+0000 7f4d65e00700 1 -- start start 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.546+0000 7f4d65e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d600760e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.546+0000 7f4d65e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60077fe0 con 0x7f4d60077bd0 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.546+0000 7f4d5ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d600760e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.546+0000 7f4d5ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d600760e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44536/0 (socket says 172.21.15.1:44536) 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.546+0000 7f4d5ea00700 1 -- 172.21.15.1:0/1916725950 learned_addr learned my addr 172.21.15.1:0/1916725950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.547+0000 7f4d5ea00700 1 -- 172.21.15.1:0/1916725950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60076620 con 0x7f4d60077bd0 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.547+0000 7f4d5ea00700 1 --2- 172.21.15.1:0/1916725950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d600760e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4d500098e0 tx=0x7f4d50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72cbbe817197644 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.547+0000 7f4d5d600700 1 -- 172.21.15.1:0/1916725950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d5000b860 con 0x7f4d60077bd0 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.547+0000 7f4d5d600700 1 -- 172.21.15.1:0/1916725950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4d5000be50 con 0x7f4d60077bd0 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.547+0000 7f4d5d600700 1 -- 172.21.15.1:0/1916725950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d50010380 con 0x7f4d60077bd0 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.548+0000 7f4d65e00700 1 -- 172.21.15.1:0/1916725950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 msgr2=0x7f4d600760e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.548+0000 7f4d65e00700 1 --2- 172.21.15.1:0/1916725950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d600760e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4d500098e0 tx=0x7f4d50009250 comp rx=0 tx=0).stop 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.548+0000 7f4d65e00700 1 -- 172.21.15.1:0/1916725950 shutdown_connections 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.548+0000 7f4d65e00700 1 --2- 172.21.15.1:0/1916725950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d600760e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.548+0000 7f4d65e00700 1 -- 172.21.15.1:0/1916725950 >> 172.21.15.1:0/1916725950 conn(0x7f4d600fded0 msgr2=0x7f4d60100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.548+0000 7f4d65e00700 1 -- 172.21.15.1:0/1916725950 shutdown_connections 2024-02-28T16:18:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.548+0000 7f4d65e00700 1 -- 172.21.15.1:0/1916725950 wait complete. 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.549+0000 7f4d65e00700 1 Processor -- start 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.549+0000 7f4d65e00700 1 -- start start 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.550+0000 7f4d65e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d60113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.550+0000 7f4d65e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60113fb0 con 0x7f4d60077bd0 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.550+0000 7f4d5ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d60113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.550+0000 7f4d5ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d60113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44542/0 (socket says 172.21.15.1:44542) 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.550+0000 7f4d5ea00700 1 -- 172.21.15.1:0/1069705645 learned_addr learned my addr 172.21.15.1:0/1069705645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.550+0000 7f4d5ea00700 1 -- 172.21.15.1:0/1069705645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500094f0 con 0x7f4d60077bd0 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.551+0000 7f4d5ea00700 1 --2- 172.21.15.1:0/1069705645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d60113a70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d50006fd0 tx=0x7f4d50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.551+0000 7f4d56a00700 1 -- 172.21.15.1:0/1069705645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d500107c0 con 0x7f4d60077bd0 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.551+0000 7f4d56a00700 1 -- 172.21.15.1:0/1069705645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4d50010db0 con 0x7f4d60077bd0 2024-02-28T16:18:00.599 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.551+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d601141b0 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.551+0000 7f4d56a00700 1 -- 172.21.15.1:0/1069705645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d5001add0 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.551+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d60114e50 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.551+0000 7f4d56a00700 1 -- 172.21.15.1:0/1069705645 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4d50021700 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.551+0000 7f4d56a00700 1 -- 172.21.15.1:0/1069705645 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4d5001e070 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.552+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6005f5c0 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.555+0000 7f4d56a00700 1 -- 172.21.15.1:0/1069705645 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4d5001a5c0 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.634+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4d60064ad0 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.635+0000 7f4d56a00700 1 -- 172.21.15.1:0/1069705645 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f4d50021b90 con 0x7f4d60077bd0 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.636+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 msgr2=0x7f4d60113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.636+0000 7f4d65e00700 1 --2- 172.21.15.1:0/1069705645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d60113a70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d50006fd0 tx=0x7f4d50009250 comp rx=0 tx=0).stop 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.636+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 shutdown_connections 2024-02-28T16:18:00.600 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.636+0000 7f4d65e00700 1 --2- 172.21.15.1:0/1069705645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d60077bd0 0x7f4d60113a70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:00.601 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.636+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 >> 172.21.15.1:0/1069705645 conn(0x7f4d600fded0 msgr2=0x7f4d601001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:00.601 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.636+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 shutdown_connections 2024-02-28T16:18:00.601 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:17:59.636+0000 7f4d65e00700 1 -- 172.21.15.1:0/1069705645 wait complete. 2024-02-28T16:18:00.601 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (3/15)... 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: Activating manager daemon smithi001.rmthtx 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: mgrmap e2: smithi001.rmthtx(active, starting, since 0.0111323s) 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:18:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:18:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:18:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: Manager daemon smithi001.rmthtx is now available 2024-02-28T16:18:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:18:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/trash_purge_schedule"}]: dispatch 2024-02-28T16:18:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:01 smithi001 ceph-mon[28991]: from='mgr.14100 172.21.15.1:0/171648685' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:02.949 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:02 smithi001 ceph-mon[28991]: mgrmap e3: smithi001.rmthtx(active, since 1.02206s) 2024-02-28T16:18:03.851 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:03 smithi001 ceph-mon[28991]: mgrmap e4: smithi001.rmthtx(active, since 2s) 2024-02-28T16:18:03.851 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:03 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3855094629' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-28T16:18:04.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:18:04.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-02-28T16:18:04.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "c0118718-d654-11ee-95c3-87774f69a715", 2024-02-28T16:18:04.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "smithi001" 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:18:04.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-28T16:18:04.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-28T16:18:04.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-28T16:18:04.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-28T16:18:04.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-28T16:18:04.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-28T16:18:04.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-28T16:18:04.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:04.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-28T16:18:04.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-02-28T16:17:42.450989+0000", 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.246+0000 7f2bcc600700 1 Processor -- start 2024-02-28T16:18:04.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.246+0000 7f2bcc600700 1 -- start start 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.247+0000 7f2bcc600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc407cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.247+0000 7f2bcc600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc407d0f0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.247+0000 7f2bc9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc407cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.247+0000 7f2bc9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc407cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54478/0 (socket says 172.21.15.1:54478) 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.247+0000 7f2bc9c00700 1 -- 172.21.15.1:0/3907391194 learned_addr learned my addr 172.21.15.1:0/3907391194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.248+0000 7f2bc9c00700 1 -- 172.21.15.1:0/3907391194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc407d230 con 0x7f2bc407c7a0 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.248+0000 7f2bc9c00700 1 --2- 172.21.15.1:0/3907391194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc407cbb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009e50 tx=0x7f2bb8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29989ef79f774802 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.248+0000 7f2bc3e00700 1 -- 172.21.15.1:0/3907391194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb80092e0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bc3e00700 1 -- 172.21.15.1:0/3907391194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2bb8012430 con 0x7f2bc407c7a0 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bc3e00700 1 -- 172.21.15.1:0/3907391194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb80075c0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bcc600700 1 -- 172.21.15.1:0/3907391194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 msgr2=0x7f2bc407cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bcc600700 1 --2- 172.21.15.1:0/3907391194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc407cbb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009e50 tx=0x7f2bb8009840 comp rx=0 tx=0).stop 2024-02-28T16:18:04.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bcc600700 1 -- 172.21.15.1:0/3907391194 shutdown_connections 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bcc600700 1 --2- 172.21.15.1:0/3907391194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc407cbb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bcc600700 1 -- 172.21.15.1:0/3907391194 >> 172.21.15.1:0/3907391194 conn(0x7f2bc4078030 msgr2=0x7f2bc407a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bcc600700 1 -- 172.21.15.1:0/3907391194 shutdown_connections 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.249+0000 7f2bcc600700 1 -- 172.21.15.1:0/3907391194 wait complete. 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.250+0000 7f2bcc600700 1 Processor -- start 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.250+0000 7f2bcc600700 1 -- start start 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.251+0000 7f2bcc600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc4117bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.251+0000 7f2bcc600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4118110 con 0x7f2bc407c7a0 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.251+0000 7f2bc9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc4117bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.251+0000 7f2bc9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc4117bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54488/0 (socket says 172.21.15.1:54488) 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.251+0000 7f2bc9c00700 1 -- 172.21.15.1:0/3855094629 learned_addr learned my addr 172.21.15.1:0/3855094629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.251+0000 7f2bc9c00700 1 -- 172.21.15.1:0/3855094629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb8009ae0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.252+0000 7f2bc9c00700 1 --2- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc4117bd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009fd0 tx=0x7f2bb8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:04.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.252+0000 7f2bc1600700 1 -- 172.21.15.1:0/3855094629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb80092e0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.252+0000 7f2bc1600700 1 -- 172.21.15.1:0/3855094629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2bb8007cc0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.252+0000 7f2bc1600700 1 -- 172.21.15.1:0/3855094629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb801c880 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.252+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc4118310 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.252+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc4118fb0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.253+0000 7f2bc1600700 1 -- 172.21.15.1:0/3855094629 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f2bb8019070 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.254+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc405f5c0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.254+0000 7f2bc1600700 1 --2- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f2bac036790 0x7f2bac038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.254+0000 7f2bc1600700 1 -- 172.21.15.1:0/3855094629 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2bb8049da0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.254+0000 7f2bc9200700 1 --2- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f2bac036790 0x7f2bac038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.255+0000 7f2bc9200700 1 --2- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f2bac036790 0x7f2bac038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009990 tx=0x7f2bb0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.260+0000 7f2bc1600700 1 -- 172.21.15.1:0/3855094629 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2bb8011390 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.549+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2bc4119350 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.550+0000 7f2bc1600700 1 -- 172.21.15.1:0/3855094629 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f2bb80204f0 con 0x7f2bc407c7a0 2024-02-28T16:18:04.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f2bac036790 msgr2=0x7f2bac038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 --2- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f2bac036790 0x7f2bac038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009990 tx=0x7f2bb0006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 msgr2=0x7f2bc4117bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 --2- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc4117bd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009fd0 tx=0x7f2bb8009840 comp rx=0 tx=0).stop 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 shutdown_connections 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 --2- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f2bac036790 0x7f2bac038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 --2- 172.21.15.1:0/3855094629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bc407c7a0 0x7f2bc4117bd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 >> 172.21.15.1:0/3855094629 conn(0x7f2bc4078030 msgr2=0x7f2bc407a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 shutdown_connections 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:03.552+0000 7f2bcc600700 1 -- 172.21.15.1:0/3855094629 wait complete. 2024-02-28T16:18:04.160 INFO:teuthology.orchestra.run.smithi001.stdout:mgr is available 2024-02-28T16:18:05.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:05 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/382881416' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-28T16:18:05.670 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:18:05.670 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-02-28T16:18:05.670 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-28T16:18:05.671 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.780+0000 7f3e5a800700 1 Processor -- start 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.780+0000 7f3e5a800700 1 -- start start 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.781+0000 7f3e5a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54109330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.781+0000 7f3e5a800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54109870 con 0x7f3e54108f20 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.781+0000 7f3e53400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.781+0000 7f3e53400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54498/0 (socket says 172.21.15.1:54498) 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.781+0000 7f3e53400700 1 -- 172.21.15.1:0/2955949839 learned_addr learned my addr 172.21.15.1:0/2955949839 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.782+0000 7f3e53400700 1 -- 172.21.15.1:0/2955949839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e541099b0 con 0x7f3e54108f20 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.782+0000 7f3e53400700 1 --2- 172.21.15.1:0/2955949839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54109330 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e440098e0 tx=0x7f3e44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4eb172e78e7b2a server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.782+0000 7f3e52a00700 1 -- 172.21.15.1:0/2955949839 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e4400b860 con 0x7f3e54108f20 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.782+0000 7f3e52a00700 1 -- 172.21.15.1:0/2955949839 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3e4400be50 con 0x7f3e54108f20 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.782+0000 7f3e52a00700 1 -- 172.21.15.1:0/2955949839 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e44010380 con 0x7f3e54108f20 2024-02-28T16:18:05.672 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.783+0000 7f3e5a800700 1 -- 172.21.15.1:0/2955949839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 msgr2=0x7f3e54109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.783+0000 7f3e5a800700 1 --2- 172.21.15.1:0/2955949839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54109330 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e440098e0 tx=0x7f3e44009250 comp rx=0 tx=0).stop 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.783+0000 7f3e5a800700 1 -- 172.21.15.1:0/2955949839 shutdown_connections 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.783+0000 7f3e5a800700 1 --2- 172.21.15.1:0/2955949839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54109330 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.783+0000 7f3e5a800700 1 -- 172.21.15.1:0/2955949839 >> 172.21.15.1:0/2955949839 conn(0x7f3e54078030 msgr2=0x7f3e5407a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.783+0000 7f3e5a800700 1 -- 172.21.15.1:0/2955949839 shutdown_connections 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.783+0000 7f3e5a800700 1 -- 172.21.15.1:0/2955949839 wait complete. 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.784+0000 7f3e5a800700 1 Processor -- start 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.784+0000 7f3e5a800700 1 -- start start 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.784+0000 7f3e5a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54117b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.784+0000 7f3e5a800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e541180c0 con 0x7f3e54108f20 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.785+0000 7f3e53400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54117b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.785+0000 7f3e53400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54117b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54508/0 (socket says 172.21.15.1:54508) 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.785+0000 7f3e53400700 1 -- 172.21.15.1:0/382881416 learned_addr learned my addr 172.21.15.1:0/382881416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:05.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.785+0000 7f3e53400700 1 -- 172.21.15.1:0/382881416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e440094f0 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.785+0000 7f3e53400700 1 --2- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54117b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3e44000c00 tx=0x7f3e44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.786+0000 7f3e4b400700 1 -- 172.21.15.1:0/382881416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e44010380 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.786+0000 7f3e4b400700 1 -- 172.21.15.1:0/382881416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3e44010c50 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.786+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e541182c0 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.786+0000 7f3e4b400700 1 -- 172.21.15.1:0/382881416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e4401ac50 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.786+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e54118f60 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.787+0000 7f3e4b400700 1 -- 172.21.15.1:0/382881416 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f3e440107c0 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.787+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5405f5c0 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.787+0000 7f3e4b400700 1 --2- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f3e38036790 0x7f3e38038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.787+0000 7f3e4b400700 1 -- 172.21.15.1:0/382881416 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3e4404a4d0 con 0x7f3e54108f20 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.788+0000 7f3e4be00700 1 --2- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f3e38036790 0x7f3e38038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.788+0000 7f3e4be00700 1 --2- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f3e38036790 0x7f3e38038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009990 tx=0x7f3e3c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:05.674 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:04.793+0000 7f3e4b400700 1 -- 172.21.15.1:0/382881416 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3e4401a430 con 0x7f3e54108f20 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.006+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3e5404bed0 con 0x7f3e54108f20 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.009+0000 7f3e4b400700 1 -- 172.21.15.1:0/382881416 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f3e4401a430 con 0x7f3e54108f20 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.011+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f3e38036790 msgr2=0x7f3e38038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.011+0000 7f3e5a800700 1 --2- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f3e38036790 0x7f3e38038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009990 tx=0x7f3e3c006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.011+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 msgr2=0x7f3e54117b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.011+0000 7f3e5a800700 1 --2- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54117b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3e44000c00 tx=0x7f3e44009250 comp rx=0 tx=0).stop 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.012+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 shutdown_connections 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.012+0000 7f3e5a800700 1 --2- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f3e38036790 0x7f3e38038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.012+0000 7f3e5a800700 1 --2- 172.21.15.1:0/382881416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e54108f20 0x7f3e54117b80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.012+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 >> 172.21.15.1:0/382881416 conn(0x7f3e54078030 msgr2=0x7f3e54079e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.012+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 shutdown_connections 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:05.012+0000 7f3e5a800700 1 -- 172.21.15.1:0/382881416 wait complete. 2024-02-28T16:18:05.675 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling cephadm module... 2024-02-28T16:18:06.956 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:06 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2698594506' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-28T16:18:07.263 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.254+0000 7fdf80c00700 1 Processor -- start 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.255+0000 7fdf80c00700 1 -- start start 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.255+0000 7fdf80c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.255+0000 7fdf80c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c1030c0 con 0x7fdf7c102770 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.256+0000 7fdf79e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.256+0000 7fdf79e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54520/0 (socket says 172.21.15.1:54520) 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.256+0000 7fdf79e00700 1 -- 172.21.15.1:0/2795007810 learned_addr learned my addr 172.21.15.1:0/2795007810 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.256+0000 7fdf79e00700 1 -- 172.21.15.1:0/2795007810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c103200 con 0x7fdf7c102770 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.256+0000 7fdf79e00700 1 --2- 172.21.15.1:0/2795007810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c102b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098e0 tx=0x7fdf6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829225ba078f673e server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf78a00700 1 -- 172.21.15.1:0/2795007810 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf6c00b860 con 0x7fdf7c102770 2024-02-28T16:18:07.264 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf78a00700 1 -- 172.21.15.1:0/2795007810 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdf6c00be50 con 0x7fdf7c102770 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf78a00700 1 -- 172.21.15.1:0/2795007810 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf6c010380 con 0x7fdf7c102770 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf80c00700 1 -- 172.21.15.1:0/2795007810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 msgr2=0x7fdf7c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf80c00700 1 --2- 172.21.15.1:0/2795007810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c102b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098e0 tx=0x7fdf6c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf80c00700 1 -- 172.21.15.1:0/2795007810 shutdown_connections 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf80c00700 1 --2- 172.21.15.1:0/2795007810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c102b80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf80c00700 1 -- 172.21.15.1:0/2795007810 >> 172.21.15.1:0/2795007810 conn(0x7fdf7c0fded0 msgr2=0x7fdf7c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf80c00700 1 -- 172.21.15.1:0/2795007810 shutdown_connections 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.257+0000 7fdf80c00700 1 -- 172.21.15.1:0/2795007810 wait complete. 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.258+0000 7fdf80c00700 1 Processor -- start 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.258+0000 7fdf80c00700 1 -- start start 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.259+0000 7fdf80c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c111840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.259+0000 7fdf80c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c111d80 con 0x7fdf7c102770 2024-02-28T16:18:07.265 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.259+0000 7fdf79e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c111840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.259+0000 7fdf79e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c111840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54530/0 (socket says 172.21.15.1:54530) 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.259+0000 7fdf79e00700 1 -- 172.21.15.1:0/2698594506 learned_addr learned my addr 172.21.15.1:0/2698594506 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.259+0000 7fdf79e00700 1 -- 172.21.15.1:0/2698594506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c0094f0 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.260+0000 7fdf79e00700 1 --2- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c111840 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00b440 tx=0x7fdf6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.260+0000 7fdf72000700 1 -- 172.21.15.1:0/2698594506 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf6c0107c0 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.260+0000 7fdf72000700 1 -- 172.21.15.1:0/2698594506 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdf6c010db0 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.260+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c111f80 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.260+0000 7fdf72000700 1 -- 172.21.15.1:0/2698594506 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf6c01ada0 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.260+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c112c20 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.261+0000 7fdf72000700 1 -- 172.21.15.1:0/2698594506 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fdf6c010920 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.262+0000 7fdf72000700 1 --2- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7fdf60036790 0x7fdf60038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.262+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c05f5c0 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.262+0000 7fdf72000700 1 -- 172.21.15.1:0/2698594506 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdf6c04abf0 con 0x7fdf7c102770 2024-02-28T16:18:07.266 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.262+0000 7fdf79400700 1 --2- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7fdf60036790 0x7fdf60038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.263+0000 7fdf79400700 1 --2- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7fdf60036790 0x7fdf60038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdf68009990 tx=0x7fdf68006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.268+0000 7fdf72000700 1 -- 172.21.15.1:0/2698594506 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdf6c02d3e0 con 0x7fdf7c102770 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.535+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fdf7c04bed0 con 0x7fdf7c102770 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.591+0000 7fdf72000700 1 -- 172.21.15.1:0/2698594506 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fdf6c02ece0 con 0x7fdf7c102770 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.593+0000 7fdf72000700 1 -- 172.21.15.1:0/2698594506 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fdf6c017e00 con 0x7fdf7c102770 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.596+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7fdf60036790 msgr2=0x7fdf60038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.596+0000 7fdf80c00700 1 --2- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7fdf60036790 0x7fdf60038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdf68009990 tx=0x7fdf68006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.596+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 msgr2=0x7fdf7c111840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.596+0000 7fdf80c00700 1 --2- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c111840 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00b440 tx=0x7fdf6c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.596+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 shutdown_connections 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.596+0000 7fdf80c00700 1 --2- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7fdf60036790 0x7fdf60038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.596+0000 7fdf80c00700 1 --2- 172.21.15.1:0/2698594506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf7c102770 0x7fdf7c111840 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:07.267 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.596+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 >> 172.21.15.1:0/2698594506 conn(0x7fdf7c0fded0 msgr2=0x7fdf7c0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:07.268 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.597+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 shutdown_connections 2024-02-28T16:18:07.268 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:06.597+0000 7fdf80c00700 1 -- 172.21.15.1:0/2698594506 wait complete. 2024-02-28T16:18:07.889 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:07 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2698594506' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-28T16:18:07.889 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:07 smithi001 ceph-mon[28991]: mgrmap e5: smithi001.rmthtx(active, since 5s) 2024-02-28T16:18:08.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:08 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3630789936' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "smithi001.rmthtx", 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.898+0000 7f47ebe00700 1 Processor -- start 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.898+0000 7f47ebe00700 1 -- start start 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.898+0000 7f47ebe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:08.923 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.898+0000 7f47ebe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4101fc0 con 0x7f47e4101670 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.899+0000 7f47e9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.899+0000 7f47e9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54556/0 (socket says 172.21.15.1:54556) 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.899+0000 7f47e9400700 1 -- 172.21.15.1:0/2022809360 learned_addr learned my addr 172.21.15.1:0/2022809360 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.899+0000 7f47e9400700 1 -- 172.21.15.1:0/2022809360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e4102100 con 0x7f47e4101670 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.900+0000 7f47e9400700 1 --2- 172.21.15.1:0/2022809360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4101a80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47d80098e0 tx=0x7f47d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e707052b63a98a server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.900+0000 7f47e3e00700 1 -- 172.21.15.1:0/2022809360 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47d800b860 con 0x7f47e4101670 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.900+0000 7f47e3e00700 1 -- 172.21.15.1:0/2022809360 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f47d800be50 con 0x7f47e4101670 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.900+0000 7f47e3e00700 1 -- 172.21.15.1:0/2022809360 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47d8010380 con 0x7f47e4101670 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.901+0000 7f47ebe00700 1 -- 172.21.15.1:0/2022809360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 msgr2=0x7f47e4101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.901+0000 7f47ebe00700 1 --2- 172.21.15.1:0/2022809360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4101a80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47d80098e0 tx=0x7f47d8009250 comp rx=0 tx=0).stop 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.901+0000 7f47ebe00700 1 -- 172.21.15.1:0/2022809360 shutdown_connections 2024-02-28T16:18:08.924 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.901+0000 7f47ebe00700 1 --2- 172.21.15.1:0/2022809360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4101a80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.901+0000 7f47ebe00700 1 -- 172.21.15.1:0/2022809360 >> 172.21.15.1:0/2022809360 conn(0x7f47e40fcdd0 msgr2=0x7f47e40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.901+0000 7f47ebe00700 1 -- 172.21.15.1:0/2022809360 shutdown_connections 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.901+0000 7f47ebe00700 1 -- 172.21.15.1:0/2022809360 wait complete. 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.902+0000 7f47ebe00700 1 Processor -- start 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.902+0000 7f47ebe00700 1 -- start start 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.903+0000 7f47ebe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.903+0000 7f47ebe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4113cc0 con 0x7f47e4101670 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.903+0000 7f47e9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.903+0000 7f47e9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54568/0 (socket says 172.21.15.1:54568) 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.903+0000 7f47e9400700 1 -- 172.21.15.1:0/3630789936 learned_addr learned my addr 172.21.15.1:0/3630789936 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.903+0000 7f47e9400700 1 -- 172.21.15.1:0/3630789936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d80094f0 con 0x7f47e4101670 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.904+0000 7f47e9400700 1 --2- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4113780 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f47d8000c00 tx=0x7f47d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.904+0000 7f47e1600700 1 -- 172.21.15.1:0/3630789936 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47d80107c0 con 0x7f47e4101670 2024-02-28T16:18:08.925 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.904+0000 7f47e1600700 1 -- 172.21.15.1:0/3630789936 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f47d8010db0 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.904+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e4113ec0 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.904+0000 7f47e1600700 1 -- 172.21.15.1:0/3630789936 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47d801ad00 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.904+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e4114b60 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.905+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c40052e0 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.906+0000 7f47e1600700 1 -- 172.21.15.1:0/3630789936 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f47d80193c0 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.906+0000 7f47e1600700 1 --2- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f47cc0367e0 0x7f47cc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.906+0000 7f47e1600700 1 -- 172.21.15.1:0/3630789936 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f47d804a610 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.906+0000 7f47e8a00700 1 -- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f47cc0367e0 msgr2=0x7f47cc038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3844925093 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.906+0000 7f47e8a00700 1 --2- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f47cc0367e0 0x7f47cc038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:07.911+0000 7f47e1600700 1 -- 172.21.15.1:0/3630789936 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f47d8022930 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.107+0000 7f47e8a00700 1 -- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f47cc0367e0 msgr2=0x7f47cc038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3844925093 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.107+0000 7f47e8a00700 1 --2- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f47cc0367e0 0x7f47cc038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.175+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f47c4005da0 con 0x7f47e4101670 2024-02-28T16:18:08.926 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.175+0000 7f47e1600700 1 -- 172.21.15.1:0/3630789936 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f47d8022340 con 0x7f47e4101670 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f47cc0367e0 msgr2=0x7f47cc038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 --2- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f47cc0367e0 0x7f47cc038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 msgr2=0x7f47e4113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 --2- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4113780 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f47d8000c00 tx=0x7f47d8009250 comp rx=0 tx=0).stop 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 shutdown_connections 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 --2- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7f47cc0367e0 0x7f47cc038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 --2- 172.21.15.1:0/3630789936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47e4101670 0x7f47e4113780 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 >> 172.21.15.1:0/3630789936 conn(0x7f47e40fcdd0 msgr2=0x7f47e40710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 shutdown_connections 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:08.179+0000 7f47ebe00700 1 -- 172.21.15.1:0/3630789936 wait complete. 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-02-28T16:18:08.927 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 5... 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: Active manager daemon smithi001.rmthtx restarted 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: Activating manager daemon smithi001.rmthtx 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: osdmap e2: 0 total, 0 up, 0 in 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: mgrmap e6: smithi001.rmthtx(active, starting, since 0.00859015s) 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:18:15.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: Manager daemon smithi001.rmthtx is now available 2024-02-28T16:18:15.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:15 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:16.601 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:16 smithi001 ceph-mon[28991]: Found migration_current of "None". Setting to last migration. 2024-02-28T16:18:16.601 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:16 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:16.601 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:16 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:18:16.601 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:16 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:18:16.602 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:16 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:18:16.602 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:16 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:18:16.602 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:16 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/trash_purge_schedule"}]: dispatch 2024-02-28T16:18:16.602 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:16 smithi001 ceph-mon[28991]: mgrmap e7: smithi001.rmthtx(active, since 1.01417s) 2024-02-28T16:18:16.928 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.547+0000 7febe7e00700 1 Processor -- start 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.547+0000 7febe7e00700 1 -- start start 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.548+0000 7febe7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.548+0000 7febe7e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe01030c0 con 0x7febe0102770 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.548+0000 7febe5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.548+0000 7febe5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54574/0 (socket says 172.21.15.1:54574) 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.548+0000 7febe5400700 1 -- 172.21.15.1:0/3990712584 learned_addr learned my addr 172.21.15.1:0/3990712584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.549+0000 7febe5400700 1 -- 172.21.15.1:0/3990712584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0103200 con 0x7febe0102770 2024-02-28T16:18:16.936 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.549+0000 7febe5400700 1 --2- 172.21.15.1:0/3990712584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0102b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febd40098e0 tx=0x7febd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c38a5ec857b108 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.549+0000 7febdfe00700 1 -- 172.21.15.1:0/3990712584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd400b860 con 0x7febe0102770 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.549+0000 7febdfe00700 1 -- 172.21.15.1:0/3990712584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7febd400be50 con 0x7febe0102770 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.549+0000 7febdfe00700 1 -- 172.21.15.1:0/3990712584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd4010380 con 0x7febe0102770 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.550+0000 7febe7e00700 1 -- 172.21.15.1:0/3990712584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 msgr2=0x7febe0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.550+0000 7febe7e00700 1 --2- 172.21.15.1:0/3990712584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0102b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febd40098e0 tx=0x7febd4009250 comp rx=0 tx=0).stop 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.550+0000 7febe7e00700 1 -- 172.21.15.1:0/3990712584 shutdown_connections 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.550+0000 7febe7e00700 1 --2- 172.21.15.1:0/3990712584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0102b80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.550+0000 7febe7e00700 1 -- 172.21.15.1:0/3990712584 >> 172.21.15.1:0/3990712584 conn(0x7febe00fded0 msgr2=0x7febe0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.550+0000 7febe7e00700 1 -- 172.21.15.1:0/3990712584 shutdown_connections 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.550+0000 7febe7e00700 1 -- 172.21.15.1:0/3990712584 wait complete. 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.551+0000 7febe7e00700 1 Processor -- start 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.551+0000 7febe7e00700 1 -- start start 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.552+0000 7febe7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:16.937 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.552+0000 7febe7e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe0113f40 con 0x7febe0102770 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.552+0000 7febe5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.552+0000 7febe5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54576/0 (socket says 172.21.15.1:54576) 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.552+0000 7febe5400700 1 -- 172.21.15.1:0/1666986391 learned_addr learned my addr 172.21.15.1:0/1666986391 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.552+0000 7febe5400700 1 -- 172.21.15.1:0/1666986391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd40094f0 con 0x7febe0102770 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.553+0000 7febe5400700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0113a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7febd400b440 tx=0x7febd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.553+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd40107c0 con 0x7febe0102770 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.553+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7febd4010db0 con 0x7febe0102770 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.553+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd401ada0 con 0x7febe0102770 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.553+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe0114140 con 0x7febe0102770 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.553+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe0114de0 con 0x7febe0102770 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.554+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7febd4010920 con 0x7febe0102770 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.554+0000 7febdd600700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 0x7febc8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.554+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 --> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7febc80393a0 con 0x7febc80367e0 2024-02-28T16:18:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.555+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7febd404ac50 con 0x7febe0102770 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.555+0000 7febe4a00700 1 -- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 msgr2=0x7febc8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3844925093 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.555+0000 7febe4a00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 0x7febc8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.756+0000 7febe4a00700 1 -- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 msgr2=0x7febc8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3844925093 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:09.756+0000 7febe4a00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 0x7febc8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:10.157+0000 7febe4a00700 1 -- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 msgr2=0x7febc8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3844925093 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:10.157+0000 7febe4a00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 0x7febc8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:10.958+0000 7febe4a00700 1 -- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 msgr2=0x7febc8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3844925093 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:10.958+0000 7febe4a00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 0x7febc8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:12.560+0000 7febe4a00700 1 -- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 msgr2=0x7febc8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3844925093 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:12.560+0000 7febe4a00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 0x7febc8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:15.281+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7febd402d3e0 con 0x7febe0102770 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:15.281+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 msgr2=0x7febc8038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-28T16:18:16.939 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:15.281+0000 7febdd600700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 0x7febc8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.283+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7febd4018d40 con 0x7febe0102770 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.283+0000 7febdd600700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7febc8039fc0 0x7febc803c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.283+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7febc80393a0 con 0x7febc8039fc0 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.286+0000 7febe4a00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7febc8039fc0 0x7febc803c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.289+0000 7febe4a00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7febc8039fc0 0x7febc803c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7febcc0039d0 tx=0x7febcc009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.290+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7febc80393a0 con 0x7febc8039fc0 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.297+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7febe01145d0 con 0x7febc8039fc0 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.298+0000 7febdd600700 1 -- 172.21.15.1:0/1666986391 <== mgr.14120 v2:172.21.15.1:6800/3320028482 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7febe01145d0 con 0x7febc8039fc0 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7febc8039fc0 msgr2=0x7febc803c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7febc8039fc0 0x7febc803c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7febcc0039d0 tx=0x7febcc009350 comp rx=0 tx=0).stop 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 msgr2=0x7febe0113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0113a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7febd400b440 tx=0x7febd4009250 comp rx=0 tx=0).stop 2024-02-28T16:18:16.940 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 shutdown_connections 2024-02-28T16:18:16.941 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7febc8039fc0 0x7febc803c3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:16.941 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe0102770 0x7febe0113a00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:16.941 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 --2- 172.21.15.1:0/1666986391 >> [v2:172.21.15.1:6800/3844925093,v1:172.21.15.1:6801/3844925093] conn(0x7febc80367e0 0x7febc8038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:16.941 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 >> 172.21.15.1:0/1666986391 conn(0x7febe00fded0 msgr2=0x7febe00fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:16.941 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 shutdown_connections 2024-02-28T16:18:16.941 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:16.299+0000 7febe7e00700 1 -- 172.21.15.1:0/1666986391 wait complete. 2024-02-28T16:18:16.941 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 5 is available 2024-02-28T16:18:16.941 INFO:teuthology.orchestra.run.smithi001.stdout:Setting orchestrator backend to cephadm... 2024-02-28T16:18:17.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:17 smithi001 ceph-mon[28991]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-28T16:18:17.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:17 smithi001 ceph-mon[28991]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-28T16:18:17.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:17 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:17.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:17 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:18:18.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:18 smithi001 ceph-mon[28991]: [28/Feb/2024:16:18:16] ENGINE Bus STARTING 2024-02-28T16:18:18.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:18 smithi001 ceph-mon[28991]: [28/Feb/2024:16:18:16] ENGINE Serving on https://172.21.15.1:7150 2024-02-28T16:18:18.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:18 smithi001 ceph-mon[28991]: [28/Feb/2024:16:18:16] ENGINE Bus STARTED 2024-02-28T16:18:18.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:18 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:18.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:18 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:18:18.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:18 smithi001 ceph-mon[28991]: mgrmap e8: smithi001.rmthtx(active, since 2s) 2024-02-28T16:18:18.566 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.555+0000 7fa625e00700 1 Processor -- start 2024-02-28T16:18:18.566 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.555+0000 7fa625e00700 1 -- start start 2024-02-28T16:18:18.566 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.556+0000 7fa625e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620104d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:18.566 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.556+0000 7fa625e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6201052d0 con 0x7fa620104980 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.556+0000 7fa61ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620104d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.557+0000 7fa61ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620104d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59558/0 (socket says 172.21.15.1:59558) 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.557+0000 7fa61ea00700 1 -- 172.21.15.1:0/1095736808 learned_addr learned my addr 172.21.15.1:0/1095736808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.557+0000 7fa61ea00700 1 -- 172.21.15.1:0/1095736808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620105410 con 0x7fa620104980 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.557+0000 7fa61ea00700 1 --2- 172.21.15.1:0/1095736808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620104d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa6100098e0 tx=0x7fa610009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c384bff772b1b8c server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.558+0000 7fa61d600700 1 -- 172.21.15.1:0/1095736808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa61000b860 con 0x7fa620104980 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.558+0000 7fa61d600700 1 -- 172.21.15.1:0/1095736808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa61000be50 con 0x7fa620104980 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.558+0000 7fa61d600700 1 -- 172.21.15.1:0/1095736808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa610010380 con 0x7fa620104980 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.558+0000 7fa625e00700 1 -- 172.21.15.1:0/1095736808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 msgr2=0x7fa620104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.558+0000 7fa625e00700 1 --2- 172.21.15.1:0/1095736808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620104d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa6100098e0 tx=0x7fa610009250 comp rx=0 tx=0).stop 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.558+0000 7fa625e00700 1 -- 172.21.15.1:0/1095736808 shutdown_connections 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.558+0000 7fa625e00700 1 --2- 172.21.15.1:0/1095736808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620104d90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.558+0000 7fa625e00700 1 -- 172.21.15.1:0/1095736808 >> 172.21.15.1:0/1095736808 conn(0x7fa620078030 msgr2=0x7fa620078450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:18.567 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.559+0000 7fa625e00700 1 -- 172.21.15.1:0/1095736808 shutdown_connections 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.559+0000 7fa625e00700 1 -- 172.21.15.1:0/1095736808 wait complete. 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.559+0000 7fa625e00700 1 Processor -- start 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.560+0000 7fa625e00700 1 -- start start 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.560+0000 7fa625e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620113770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.560+0000 7fa625e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa620113cb0 con 0x7fa620104980 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.560+0000 7fa61ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.560+0000 7fa61ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620113770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59574/0 (socket says 172.21.15.1:59574) 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.560+0000 7fa61ea00700 1 -- 172.21.15.1:0/727109227 learned_addr learned my addr 172.21.15.1:0/727109227 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.561+0000 7fa61ea00700 1 -- 172.21.15.1:0/727109227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6100094f0 con 0x7fa620104980 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.561+0000 7fa61ea00700 1 --2- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620113770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa610000c00 tx=0x7fa610009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.561+0000 7fa616a00700 1 -- 172.21.15.1:0/727109227 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6100107c0 con 0x7fa620104980 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.561+0000 7fa616a00700 1 -- 172.21.15.1:0/727109227 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa610010db0 con 0x7fa620104980 2024-02-28T16:18:18.568 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.561+0000 7fa616a00700 1 -- 172.21.15.1:0/727109227 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa61001ad00 con 0x7fa620104980 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.561+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa620113eb0 con 0x7fa620104980 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.561+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa620114b50 con 0x7fa620104980 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.563+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62005f5c0 con 0x7fa620104980 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.563+0000 7fa616a00700 1 -- 172.21.15.1:0/727109227 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fa61001a530 con 0x7fa620104980 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.563+0000 7fa616a00700 1 --2- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fa6040366b0 0x7fa604038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.563+0000 7fa616a00700 1 -- 172.21.15.1:0/727109227 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa61004a480 con 0x7fa620104980 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.563+0000 7fa61e000700 1 --2- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fa6040366b0 0x7fa604038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.564+0000 7fa61e000700 1 --2- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fa6040366b0 0x7fa604038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa60c009990 tx=0x7fa60c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.569+0000 7fa616a00700 1 -- 172.21.15.1:0/727109227 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa610010920 con 0x7fa620104980 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.783+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fa620103cf0 con 0x7fa6040366b0 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.793+0000 7fa616a00700 1 -- 172.21.15.1:0/727109227 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa620103cf0 con 0x7fa6040366b0 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fa6040366b0 msgr2=0x7fa604038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:18.569 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 --2- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fa6040366b0 0x7fa604038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa60c009990 tx=0x7fa60c006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:18.570 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 msgr2=0x7fa620113770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:18.570 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 --2- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620113770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa610000c00 tx=0x7fa610009250 comp rx=0 tx=0).stop 2024-02-28T16:18:18.570 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 shutdown_connections 2024-02-28T16:18:18.570 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 --2- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fa6040366b0 0x7fa604038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:18.570 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 --2- 172.21.15.1:0/727109227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa620104980 0x7fa620113770 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:18.570 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 >> 172.21.15.1:0/727109227 conn(0x7fa620078030 msgr2=0x7fa6201030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:18.570 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.795+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 shutdown_connections 2024-02-28T16:18:18.570 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:17.796+0000 7fa625e00700 1 -- 172.21.15.1:0/727109227 wait complete. 2024-02-28T16:18:19.372 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:19 smithi001 ceph-mon[28991]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-28T16:18:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.154+0000 7fbeaca00700 1 Processor -- start 2024-02-28T16:18:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.154+0000 7fbeaca00700 1 -- start start 2024-02-28T16:18:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.154+0000 7fbeaca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.154+0000 7fbeaca00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8079130 con 0x7fbea8078d20 2024-02-28T16:18:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.155+0000 7fbea5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:20.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.155+0000 7fbea5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59582/0 (socket says 172.21.15.1:59582) 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.155+0000 7fbea5e00700 1 -- 172.21.15.1:0/987574563 learned_addr learned my addr 172.21.15.1:0/987574563 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.155+0000 7fbea5e00700 1 -- 172.21.15.1:0/987574563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea8077770 con 0x7fbea8078d20 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.156+0000 7fbea5e00700 1 --2- 172.21.15.1:0/987574563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8077230 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbe980098e0 tx=0x7fbe98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80d810afcf3ae58 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.156+0000 7fbea5400700 1 -- 172.21.15.1:0/987574563 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9800b860 con 0x7fbea8078d20 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.156+0000 7fbea5400700 1 -- 172.21.15.1:0/987574563 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbe9800be50 con 0x7fbea8078d20 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.156+0000 7fbea5400700 1 -- 172.21.15.1:0/987574563 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe98010380 con 0x7fbea8078d20 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.157+0000 7fbeaca00700 1 -- 172.21.15.1:0/987574563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 msgr2=0x7fbea8077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.157+0000 7fbeaca00700 1 --2- 172.21.15.1:0/987574563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8077230 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbe980098e0 tx=0x7fbe98009250 comp rx=0 tx=0).stop 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.157+0000 7fbeaca00700 1 -- 172.21.15.1:0/987574563 shutdown_connections 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.157+0000 7fbeaca00700 1 --2- 172.21.15.1:0/987574563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8077230 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.157+0000 7fbeaca00700 1 -- 172.21.15.1:0/987574563 >> 172.21.15.1:0/987574563 conn(0x7fbea80fdf00 msgr2=0x7fbea8100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.157+0000 7fbeaca00700 1 -- 172.21.15.1:0/987574563 shutdown_connections 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.157+0000 7fbeaca00700 1 -- 172.21.15.1:0/987574563 wait complete. 2024-02-28T16:18:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.158+0000 7fbeaca00700 1 Processor -- start 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.158+0000 7fbeaca00700 1 -- start start 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.158+0000 7fbeaca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.158+0000 7fbeaca00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8113d50 con 0x7fbea8078d20 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.158+0000 7fbea5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.158+0000 7fbea5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59586/0 (socket says 172.21.15.1:59586) 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.158+0000 7fbea5e00700 1 -- 172.21.15.1:0/191571724 learned_addr learned my addr 172.21.15.1:0/191571724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.159+0000 7fbea5e00700 1 -- 172.21.15.1:0/191571724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980094f0 con 0x7fbea8078d20 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.159+0000 7fbea5e00700 1 --2- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8113810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe98006fd0 tx=0x7fbe98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.159+0000 7fbe9e000700 1 -- 172.21.15.1:0/191571724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe980107c0 con 0x7fbea8078d20 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.159+0000 7fbe9e000700 1 -- 172.21.15.1:0/191571724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbe98010db0 con 0x7fbea8078d20 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.159+0000 7fbe9e000700 1 -- 172.21.15.1:0/191571724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9801ada0 con 0x7fbea8078d20 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.159+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea8113f50 con 0x7fbea8078d20 2024-02-28T16:18:20.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.160+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea8114bf0 con 0x7fbea8078d20 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.161+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea805f5c0 con 0x7fbea8078d20 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.161+0000 7fbe9e000700 1 -- 172.21.15.1:0/191571724 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fbe98010920 con 0x7fbea8078d20 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.161+0000 7fbe9e000700 1 --2- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fbe8c0367e0 0x7fbe8c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.161+0000 7fbe9e000700 1 -- 172.21.15.1:0/191571724 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbe98049de0 con 0x7fbea8078d20 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.162+0000 7fbe9fe00700 1 --2- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fbe8c0367e0 0x7fbe8c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.163+0000 7fbe9fe00700 1 --2- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fbe8c0367e0 0x7fbe8c038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009990 tx=0x7fbe90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.167+0000 7fbe9e000700 1 -- 172.21.15.1:0/191571724 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbe98023360 con 0x7fbea8078d20 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.381+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fbea8100250 con 0x7fbe8c0367e0 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.383+0000 7fbe9e000700 1 -- 172.21.15.1:0/191571724 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fbea8100250 con 0x7fbe8c0367e0 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.386+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fbe8c0367e0 msgr2=0x7fbe8c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.386+0000 7fbeaca00700 1 --2- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fbe8c0367e0 0x7fbe8c038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009990 tx=0x7fbe90006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.387+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 msgr2=0x7fbea8113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:20.109 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.387+0000 7fbeaca00700 1 --2- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8113810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe98006fd0 tx=0x7fbe98009250 comp rx=0 tx=0).stop 2024-02-28T16:18:20.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.387+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 shutdown_connections 2024-02-28T16:18:20.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.387+0000 7fbeaca00700 1 --2- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fbe8c0367e0 0x7fbe8c038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:20.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.387+0000 7fbeaca00700 1 --2- 172.21.15.1:0/191571724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea8078d20 0x7fbea8113810 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:20.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.387+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 >> 172.21.15.1:0/191571724 conn(0x7fbea80fdf00 msgr2=0x7fbea80ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:20.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.387+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 shutdown_connections 2024-02-28T16:18:20.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:19.387+0000 7fbeaca00700 1 -- 172.21.15.1:0/191571724 wait complete. 2024-02-28T16:18:20.110 INFO:teuthology.orchestra.run.smithi001.stdout:Generating ssh key... 2024-02-28T16:18:20.399 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:20 smithi001 ceph-mon[28991]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:22.292 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.709+0000 7f0d6b600700 1 Processor -- start 2024-02-28T16:18:22.292 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.709+0000 7f0d6b600700 1 -- start start 2024-02-28T16:18:22.292 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.709+0000 7f0d6b600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:22.292 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.709+0000 7f0d6b600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d64101fc0 con 0x7f0d64101670 2024-02-28T16:18:22.293 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.710+0000 7f0d68c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:22.293 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.710+0000 7f0d68c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59598/0 (socket says 172.21.15.1:59598) 2024-02-28T16:18:22.293 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.710+0000 7f0d68c00700 1 -- 172.21.15.1:0/2933834700 learned_addr learned my addr 172.21.15.1:0/2933834700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:22.293 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.711+0000 7f0d68c00700 1 -- 172.21.15.1:0/2933834700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d64102100 con 0x7f0d64101670 2024-02-28T16:18:22.294 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.711+0000 7f0d68c00700 1 --2- 172.21.15.1:0/2933834700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64101a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d540098e0 tx=0x7f0d54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4268966db736cede server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.711+0000 7f0d63e00700 1 -- 172.21.15.1:0/2933834700 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d5400b860 con 0x7f0d64101670 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.711+0000 7f0d63e00700 1 -- 172.21.15.1:0/2933834700 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0d5400be50 con 0x7f0d64101670 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.711+0000 7f0d63e00700 1 -- 172.21.15.1:0/2933834700 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d54010380 con 0x7f0d64101670 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.712+0000 7f0d6b600700 1 -- 172.21.15.1:0/2933834700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 msgr2=0x7f0d64101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.712+0000 7f0d6b600700 1 --2- 172.21.15.1:0/2933834700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64101a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d540098e0 tx=0x7f0d54009250 comp rx=0 tx=0).stop 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.712+0000 7f0d6b600700 1 -- 172.21.15.1:0/2933834700 shutdown_connections 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.712+0000 7f0d6b600700 1 --2- 172.21.15.1:0/2933834700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64101a80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.712+0000 7f0d6b600700 1 -- 172.21.15.1:0/2933834700 >> 172.21.15.1:0/2933834700 conn(0x7f0d640fcdd0 msgr2=0x7f0d640ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:22.295 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.712+0000 7f0d6b600700 1 -- 172.21.15.1:0/2933834700 shutdown_connections 2024-02-28T16:18:22.296 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.712+0000 7f0d6b600700 1 -- 172.21.15.1:0/2933834700 wait complete. 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.713+0000 7f0d6b600700 1 Processor -- start 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.713+0000 7f0d6b600700 1 -- start start 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.714+0000 7f0d6b600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.714+0000 7f0d6b600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d64111ac0 con 0x7f0d64101670 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.714+0000 7f0d68c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.714+0000 7f0d68c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59614/0 (socket says 172.21.15.1:59614) 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.714+0000 7f0d68c00700 1 -- 172.21.15.1:0/3732428299 learned_addr learned my addr 172.21.15.1:0/3732428299 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.714+0000 7f0d68c00700 1 -- 172.21.15.1:0/3732428299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d540094f0 con 0x7f0d64101670 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.715+0000 7f0d68c00700 1 --2- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64111580 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0d54000c00 tx=0x7f0d54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.715+0000 7f0d61600700 1 -- 172.21.15.1:0/3732428299 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d540107c0 con 0x7f0d64101670 2024-02-28T16:18:22.298 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.715+0000 7f0d61600700 1 -- 172.21.15.1:0/3732428299 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0d54010db0 con 0x7f0d64101670 2024-02-28T16:18:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.715+0000 7f0d61600700 1 -- 172.21.15.1:0/3732428299 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d5401ad00 con 0x7f0d64101670 2024-02-28T16:18:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.715+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d64111cc0 con 0x7f0d64101670 2024-02-28T16:18:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.715+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d64112960 con 0x7f0d64101670 2024-02-28T16:18:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.716+0000 7f0d61600700 1 -- 172.21.15.1:0/3732428299 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f0d54010920 con 0x7f0d64101670 2024-02-28T16:18:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.716+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6405f5c0 con 0x7f0d64101670 2024-02-28T16:18:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.717+0000 7f0d61600700 1 --2- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f0d480367e0 0x7f0d48038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.717+0000 7f0d61600700 1 -- 172.21.15.1:0/3732428299 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0d5404a6c0 con 0x7f0d64101670 2024-02-28T16:18:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.717+0000 7f0d5be00700 1 --2- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f0d480367e0 0x7f0d48038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.718+0000 7f0d5be00700 1 --2- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f0d480367e0 0x7f0d48038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009990 tx=0x7f0d4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.722+0000 7f0d61600700 1 -- 172.21.15.1:0/3732428299 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0d5401a530 con 0x7f0d64101670 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:20.932+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d640fef60 con 0x7f0d480367e0 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.656+0000 7f0d61600700 1 -- 172.21.15.1:0/3732428299 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0d640fef60 con 0x7f0d480367e0 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f0d480367e0 msgr2=0x7f0d48038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 --2- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f0d480367e0 0x7f0d48038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009990 tx=0x7f0d4c006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 msgr2=0x7f0d64111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 --2- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64111580 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0d54000c00 tx=0x7f0d54009250 comp rx=0 tx=0).stop 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 shutdown_connections 2024-02-28T16:18:22.301 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 --2- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f0d480367e0 0x7f0d48038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:22.302 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 --2- 172.21.15.1:0/3732428299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d64101670 0x7f0d64111580 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:22.302 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 >> 172.21.15.1:0/3732428299 conn(0x7f0d640fcdd0 msgr2=0x7f0d640710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:22.302 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 shutdown_connections 2024-02-28T16:18:22.302 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:21.658+0000 7f0d6b600700 1 -- 172.21.15.1:0/3732428299 wait complete. 2024-02-28T16:18:22.907 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:22 smithi001 ceph-mon[28991]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:22.907 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:22 smithi001 ceph-mon[28991]: Generating ssh key... 2024-02-28T16:18:22.907 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:22 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:22.907 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:22 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:23.787 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbe61e3rMHFvhXRNG0ybqMHp9Oy+O72pvqcv7GpD4+n1+ylhGViFCn6Z+SMc9sowHUN4Q4wfQbRWGdhK+PmxVd67YAOPmj9wlsH3ty0TITfG5/7vBw7T3gxSDNWlFP4fiOB6lPnKlEMvP1H+95kd5cEnkYjjJMU6NijMFsJFAgjMKzeV8gWjU1XTm8DLds419PhZLw0BWWUivvayqbyIiNjMiPRNdaSUdg48hYdt3dX04fOR2Fn1lq4yRdIzQUNK/n4uhfPmH/4xONi6+eiQQ63VHQxPcv5XCA17d1qHnPwP/A9kcAH7GrUtaq1TYddA+zvNzT8TrFmfuMosi4/DDciyCjdoCgwHMQaTe8Q/w6F1ChLLkyFGzAZwYx5FNB9GAWau3bc/SdYLCUvDJkDPquZ8cwqfP+doAWC9DAioMSxGKpqk1p4uyWgamyhZGa+hI5TYyhkAeIGwUlXlZ7NGcCRkmPK1CCWJmrFasqEu2lJpfA0CtLbxFbndz/UedsnqE= ceph-c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:18:23.787 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.915+0000 7f45a5a00700 1 Processor -- start 2024-02-28T16:18:23.787 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.916+0000 7f45a5a00700 1 -- start start 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.916+0000 7f45a5a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a00760e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.916+0000 7f45a5a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0077fe0 con 0x7f45a0077bd0 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.917+0000 7f459ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a00760e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.917+0000 7f459ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a00760e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45982/0 (socket says 172.21.15.1:45982) 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.917+0000 7f459ea00700 1 -- 172.21.15.1:0/2149728130 learned_addr learned my addr 172.21.15.1:0/2149728130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.917+0000 7f459ea00700 1 -- 172.21.15.1:0/2149728130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0076620 con 0x7f45a0077bd0 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.918+0000 7f459ea00700 1 --2- 172.21.15.1:0/2149728130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a00760e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45880098e0 tx=0x7f4588009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45542fa21eb6f88a server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.918+0000 7f459e000700 1 -- 172.21.15.1:0/2149728130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f458800b860 con 0x7f45a0077bd0 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.918+0000 7f459e000700 1 -- 172.21.15.1:0/2149728130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f458800be50 con 0x7f45a0077bd0 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.918+0000 7f459e000700 1 -- 172.21.15.1:0/2149728130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4588010380 con 0x7f45a0077bd0 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.919+0000 7f45a5a00700 1 -- 172.21.15.1:0/2149728130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 msgr2=0x7f45a00760e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:23.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.919+0000 7f45a5a00700 1 --2- 172.21.15.1:0/2149728130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a00760e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45880098e0 tx=0x7f4588009250 comp rx=0 tx=0).stop 2024-02-28T16:18:23.789 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.919+0000 7f45a5a00700 1 -- 172.21.15.1:0/2149728130 shutdown_connections 2024-02-28T16:18:23.789 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.919+0000 7f45a5a00700 1 --2- 172.21.15.1:0/2149728130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a00760e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:23.789 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.919+0000 7f45a5a00700 1 -- 172.21.15.1:0/2149728130 >> 172.21.15.1:0/2149728130 conn(0x7f45a00fded0 msgr2=0x7f45a0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.919+0000 7f45a5a00700 1 -- 172.21.15.1:0/2149728130 shutdown_connections 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.919+0000 7f45a5a00700 1 -- 172.21.15.1:0/2149728130 wait complete. 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.920+0000 7f45a5a00700 1 Processor -- start 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.920+0000 7f45a5a00700 1 -- start start 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.920+0000 7f45a5a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a0113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.920+0000 7f45a5a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0113fb0 con 0x7f45a0077bd0 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.921+0000 7f459ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a0113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.921+0000 7f459ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a0113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45994/0 (socket says 172.21.15.1:45994) 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.921+0000 7f459ea00700 1 -- 172.21.15.1:0/1550272246 learned_addr learned my addr 172.21.15.1:0/1550272246 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.921+0000 7f459ea00700 1 -- 172.21.15.1:0/1550272246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45880094f0 con 0x7f45a0077bd0 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.921+0000 7f459ea00700 1 --2- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a0113a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4588006fd0 tx=0x7f4588009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.922+0000 7f4597400700 1 -- 172.21.15.1:0/1550272246 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45880107c0 con 0x7f45a0077bd0 2024-02-28T16:18:23.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.922+0000 7f4597400700 1 -- 172.21.15.1:0/1550272246 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4588010db0 con 0x7f45a0077bd0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.922+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a01141b0 con 0x7f45a0077bd0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.922+0000 7f4597400700 1 -- 172.21.15.1:0/1550272246 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f458801add0 con 0x7f45a0077bd0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.922+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a0114e50 con 0x7f45a0077bd0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.923+0000 7f4597400700 1 -- 172.21.15.1:0/1550272246 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4588010920 con 0x7f45a0077bd0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.923+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a005f5c0 con 0x7f45a0077bd0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.923+0000 7f4597400700 1 --2- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4584036790 0x7f4584038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.923+0000 7f4597400700 1 -- 172.21.15.1:0/1550272246 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f458801e070 con 0x7f45a0077bd0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.924+0000 7f4596000700 1 --2- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4584036790 0x7f4584038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.924+0000 7f4596000700 1 --2- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4584036790 0x7f4584038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4590009990 tx=0x7f4590006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:22.929+0000 7f4597400700 1 -- 172.21.15.1:0/1550272246 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4588010bd0 con 0x7f45a0077bd0 2024-02-28T16:18:23.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.140+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f45a0108bb0 con 0x7f4584036790 2024-02-28T16:18:23.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.141+0000 7f4597400700 1 -- 172.21.15.1:0/1550272246 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f45a0108bb0 con 0x7f4584036790 2024-02-28T16:18:23.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.144+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4584036790 msgr2=0x7f4584038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:23.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.144+0000 7f45a5a00700 1 --2- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4584036790 0x7f4584038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4590009990 tx=0x7f4590006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:23.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.144+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 msgr2=0x7f45a0113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:23.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.144+0000 7f45a5a00700 1 --2- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a0113a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4588006fd0 tx=0x7f4588009250 comp rx=0 tx=0).stop 2024-02-28T16:18:23.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.144+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 shutdown_connections 2024-02-28T16:18:23.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.144+0000 7f45a5a00700 1 --2- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4584036790 0x7f4584038c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:23.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.144+0000 7f45a5a00700 1 --2- 172.21.15.1:0/1550272246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45a0077bd0 0x7f45a0113a70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:23.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.144+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 >> 172.21.15.1:0/1550272246 conn(0x7f45a00fded0 msgr2=0x7f45a01001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:23.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.145+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 shutdown_connections 2024-02-28T16:18:23.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:23.145+0000 7f45a5a00700 1 -- 172.21.15.1:0/1550272246 wait complete. 2024-02-28T16:18:23.796 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-28T16:18:23.797 INFO:teuthology.orchestra.run.smithi001.stdout:Adding key to root@localhost authorized_keys... 2024-02-28T16:18:23.797 INFO:teuthology.orchestra.run.smithi001.stdout:Adding host smithi001... 2024-02-28T16:18:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:23 smithi001 ceph-mon[28991]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:25.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:25 smithi001 ceph-mon[28991]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:27.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:26 smithi001 ceph-mon[28991]: Deploying cephadm binary to smithi001 2024-02-28T16:18:29.547 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Added host 'smithi001' with addr '172.21.15.1' 2024-02-28T16:18:29.547 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.365+0000 7f5906400700 1 Processor -- start 2024-02-28T16:18:29.547 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.365+0000 7f5906400700 1 -- start start 2024-02-28T16:18:29.547 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.365+0000 7f5906400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f5900104b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:29.547 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.365+0000 7f5906400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59001050d0 con 0x7f5900104780 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.366+0000 7f5905000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f5900104b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.366+0000 7f5905000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f5900104b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45996/0 (socket says 172.21.15.1:45996) 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.366+0000 7f5905000700 1 -- 172.21.15.1:0/1376696988 learned_addr learned my addr 172.21.15.1:0/1376696988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.366+0000 7f5905000700 1 -- 172.21.15.1:0/1376696988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900105210 con 0x7f5900104780 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.367+0000 7f5905000700 1 --2- 172.21.15.1:0/1376696988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f5900104b90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58f40098e0 tx=0x7f58f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e8124c2272d619 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.367+0000 7f58ff400700 1 -- 172.21.15.1:0/1376696988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58f400b860 con 0x7f5900104780 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.367+0000 7f58ff400700 1 -- 172.21.15.1:0/1376696988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f58f400be50 con 0x7f5900104780 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.367+0000 7f58ff400700 1 -- 172.21.15.1:0/1376696988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58f4010380 con 0x7f5900104780 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.368+0000 7f5906400700 1 -- 172.21.15.1:0/1376696988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 msgr2=0x7f5900104b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.368+0000 7f5906400700 1 --2- 172.21.15.1:0/1376696988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f5900104b90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58f40098e0 tx=0x7f58f4009250 comp rx=0 tx=0).stop 2024-02-28T16:18:29.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.368+0000 7f5906400700 1 -- 172.21.15.1:0/1376696988 shutdown_connections 2024-02-28T16:18:29.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.368+0000 7f5906400700 1 --2- 172.21.15.1:0/1376696988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f5900104b90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:29.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.368+0000 7f5906400700 1 -- 172.21.15.1:0/1376696988 >> 172.21.15.1:0/1376696988 conn(0x7f5900077dd0 msgr2=0x7f59000781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.368+0000 7f5906400700 1 -- 172.21.15.1:0/1376696988 shutdown_connections 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.368+0000 7f5906400700 1 -- 172.21.15.1:0/1376696988 wait complete. 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.369+0000 7f5906400700 1 Processor -- start 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.369+0000 7f5906400700 1 -- start start 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.370+0000 7f5906400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f59001135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.370+0000 7f5906400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5900113ae0 con 0x7f5900104780 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.370+0000 7f5905000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f59001135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.370+0000 7f5905000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f59001135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46000/0 (socket says 172.21.15.1:46000) 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.370+0000 7f5905000700 1 -- 172.21.15.1:0/3589435493 learned_addr learned my addr 172.21.15.1:0/3589435493 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.370+0000 7f5905000700 1 -- 172.21.15.1:0/3589435493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f40094f0 con 0x7f5900104780 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.371+0000 7f5905000700 1 --2- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f59001135a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58f4006b00 tx=0x7f58f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.371+0000 7f58fcc00700 1 -- 172.21.15.1:0/3589435493 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58f40107c0 con 0x7f5900104780 2024-02-28T16:18:29.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.371+0000 7f58fcc00700 1 -- 172.21.15.1:0/3589435493 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f58f4010db0 con 0x7f5900104780 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.371+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5900113ce0 con 0x7f5900104780 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.371+0000 7f58fcc00700 1 -- 172.21.15.1:0/3589435493 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58f401acf0 con 0x7f5900104780 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.371+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5900114980 con 0x7f5900104780 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.372+0000 7f58fcc00700 1 -- 172.21.15.1:0/3589435493 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f58f4010920 con 0x7f5900104780 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.372+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590005f5c0 con 0x7f5900104780 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.373+0000 7f58fcc00700 1 --2- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f58e8036830 0x7f58e8038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.373+0000 7f58fcc00700 1 -- 172.21.15.1:0/3589435493 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f58f404a6a0 con 0x7f5900104780 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.373+0000 7f58ffe00700 1 --2- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f58e8036830 0x7f58e8038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.374+0000 7f58ffe00700 1 --2- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f58e8036830 0x7f58e8038ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009990 tx=0x7f58f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.378+0000 7f58fcc00700 1 -- 172.21.15.1:0/3589435493 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f58f401a530 con 0x7f5900104780 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:24.594+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5900115500 con 0x7f58e8036830 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.850+0000 7f58fcc00700 1 -- 172.21.15.1:0/3589435493 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5900115500 con 0x7f58e8036830 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.854+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f58e8036830 msgr2=0x7f58e8038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:29.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.854+0000 7f5906400700 1 --2- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f58e8036830 0x7f58e8038ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009990 tx=0x7f58f0006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.854+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 msgr2=0x7f59001135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.854+0000 7f5906400700 1 --2- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f59001135a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58f4006b00 tx=0x7f58f4009250 comp rx=0 tx=0).stop 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.854+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 shutdown_connections 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.854+0000 7f5906400700 1 --2- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f58e8036830 0x7f58e8038ce0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.854+0000 7f5906400700 1 --2- 172.21.15.1:0/3589435493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5900104780 0x7f59001135a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.854+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 >> 172.21.15.1:0/3589435493 conn(0x7f5900077dd0 msgr2=0x7f59001030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.855+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 shutdown_connections 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:28.855+0000 7f5906400700 1 -- 172.21.15.1:0/3589435493 wait complete. 2024-02-28T16:18:29.555 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying mon service with default placement... 2024-02-28T16:18:30.162 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:29 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:30.162 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:29 smithi001 ceph-mon[28991]: Added host smithi001 2024-02-28T16:18:30.162 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:29 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:18:31.133 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-28T16:18:31.133 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.217+0000 7f6b22600700 1 Processor -- start 2024-02-28T16:18:31.133 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.217+0000 7f6b22600700 1 -- start start 2024-02-28T16:18:31.134 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.218+0000 7f6b22600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c104d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:31.134 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.218+0000 7f6b22600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c1052b0 con 0x7f6b1c104960 2024-02-28T16:18:31.134 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.218+0000 7f6b1b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c104d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:31.134 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.218+0000 7f6b1b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c104d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46010/0 (socket says 172.21.15.1:46010) 2024-02-28T16:18:31.135 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.218+0000 7f6b1b400700 1 -- 172.21.15.1:0/2065103818 learned_addr learned my addr 172.21.15.1:0/2065103818 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:31.135 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.218+0000 7f6b1b400700 1 -- 172.21.15.1:0/2065103818 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c1053f0 con 0x7f6b1c104960 2024-02-28T16:18:31.135 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.219+0000 7f6b1b400700 1 --2- 172.21.15.1:0/2065103818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c104d70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0098e0 tx=0x7f6b0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc094af5bc22ef8 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:31.136 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.219+0000 7f6b1a000700 1 -- 172.21.15.1:0/2065103818 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b0c00b860 con 0x7f6b1c104960 2024-02-28T16:18:31.136 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.219+0000 7f6b1a000700 1 -- 172.21.15.1:0/2065103818 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6b0c00be50 con 0x7f6b1c104960 2024-02-28T16:18:31.136 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.219+0000 7f6b1a000700 1 -- 172.21.15.1:0/2065103818 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b0c010380 con 0x7f6b1c104960 2024-02-28T16:18:31.136 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.220+0000 7f6b22600700 1 -- 172.21.15.1:0/2065103818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 msgr2=0x7f6b1c104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:31.136 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.220+0000 7f6b22600700 1 --2- 172.21.15.1:0/2065103818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c104d70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0098e0 tx=0x7f6b0c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:31.136 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.220+0000 7f6b22600700 1 -- 172.21.15.1:0/2065103818 shutdown_connections 2024-02-28T16:18:31.137 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.220+0000 7f6b22600700 1 --2- 172.21.15.1:0/2065103818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c104d70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:31.137 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.220+0000 7f6b22600700 1 -- 172.21.15.1:0/2065103818 >> 172.21.15.1:0/2065103818 conn(0x7f6b1c100340 msgr2=0x7f6b1c102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:31.137 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.220+0000 7f6b22600700 1 -- 172.21.15.1:0/2065103818 shutdown_connections 2024-02-28T16:18:31.137 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.220+0000 7f6b22600700 1 -- 172.21.15.1:0/2065103818 wait complete. 2024-02-28T16:18:31.137 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.221+0000 7f6b22600700 1 Processor -- start 2024-02-28T16:18:31.137 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.221+0000 7f6b22600700 1 -- start start 2024-02-28T16:18:31.138 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.221+0000 7f6b22600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.222+0000 7f6b22600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c113f60 con 0x7f6b1c104960 2024-02-28T16:18:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.222+0000 7f6b1b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.222+0000 7f6b1b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c113a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46022/0 (socket says 172.21.15.1:46022) 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.222+0000 7f6b1b400700 1 -- 172.21.15.1:0/1024018998 learned_addr learned my addr 172.21.15.1:0/1024018998 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.222+0000 7f6b1b400700 1 -- 172.21.15.1:0/1024018998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c0094f0 con 0x7f6b1c104960 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.223+0000 7f6b1b400700 1 --2- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c113a20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00b440 tx=0x7f6b0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.223+0000 7f6b13400700 1 -- 172.21.15.1:0/1024018998 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b0c0107c0 con 0x7f6b1c104960 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.223+0000 7f6b13400700 1 -- 172.21.15.1:0/1024018998 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6b0c010db0 con 0x7f6b1c104960 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.223+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c114160 con 0x7f6b1c104960 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.223+0000 7f6b13400700 1 -- 172.21.15.1:0/1024018998 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b0c01ada0 con 0x7f6b1c104960 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.223+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c114e00 con 0x7f6b1c104960 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.224+0000 7f6b13400700 1 -- 172.21.15.1:0/1024018998 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6b0c010920 con 0x7f6b1c104960 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.224+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c05f5c0 con 0x7f6b1c104960 2024-02-28T16:18:31.142 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.225+0000 7f6b13400700 1 --2- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f6b000367e0 0x7f6b00038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.225+0000 7f6b13400700 1 -- 172.21.15.1:0/1024018998 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6b0c04ad50 con 0x7f6b1c104960 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.225+0000 7f6b1aa00700 1 --2- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f6b000367e0 0x7f6b00038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.226+0000 7f6b1aa00700 1 --2- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f6b000367e0 0x7f6b00038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009990 tx=0x7f6b08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.230+0000 7f6b13400700 1 -- 172.21.15.1:0/1024018998 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6b0c02e440 con 0x7f6b1c104960 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.455+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b1c106940 con 0x7f6b000367e0 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.467+0000 7f6b13400700 1 -- 172.21.15.1:0/1024018998 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6b1c106940 con 0x7f6b000367e0 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.470+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f6b000367e0 msgr2=0x7f6b00038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.470+0000 7f6b22600700 1 --2- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f6b000367e0 0x7f6b00038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009990 tx=0x7f6b08006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.470+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 msgr2=0x7f6b1c113a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.470+0000 7f6b22600700 1 --2- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c113a20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00b440 tx=0x7f6b0c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.470+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 shutdown_connections 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.471+0000 7f6b22600700 1 --2- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f6b000367e0 0x7f6b00038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:31.143 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.471+0000 7f6b22600700 1 --2- 172.21.15.1:0/1024018998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c104960 0x7f6b1c113a20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:31.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.471+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 >> 172.21.15.1:0/1024018998 conn(0x7f6b1c100340 msgr2=0x7f6b1c102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:31.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.471+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 shutdown_connections 2024-02-28T16:18:31.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:30.471+0000 7f6b22600700 1 -- 172.21.15.1:0/1024018998 wait complete. 2024-02-28T16:18:31.144 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying mgr service with default placement... 2024-02-28T16:18:31.639 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:31 smithi001 ceph-mon[28991]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:31.640 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:31 smithi001 ceph-mon[28991]: Saving service mon spec with placement count:5 2024-02-28T16:18:31.640 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:31 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:33.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-28T16:18:33.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.165+0000 7f4a62000700 1 Processor -- start 2024-02-28T16:18:33.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.165+0000 7f4a62000700 1 -- start start 2024-02-28T16:18:33.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.165+0000 7f4a62000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c104d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:33.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.165+0000 7f4a62000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c1052d0 con 0x7f4a5c104980 2024-02-28T16:18:33.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.166+0000 7f4a5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c104d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:33.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.166+0000 7f4a5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c104d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59110/0 (socket says 172.21.15.1:59110) 2024-02-28T16:18:33.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.166+0000 7f4a5b400700 1 -- 172.21.15.1:0/1524296999 learned_addr learned my addr 172.21.15.1:0/1524296999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.166+0000 7f4a5b400700 1 -- 172.21.15.1:0/1524296999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c105410 con 0x7f4a5c104980 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.167+0000 7f4a5b400700 1 --2- 172.21.15.1:0/1524296999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c104d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0098e0 tx=0x7f4a4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e2673e0e47e034 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.167+0000 7f4a5a000700 1 -- 172.21.15.1:0/1524296999 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a4c00b860 con 0x7f4a5c104980 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.167+0000 7f4a5a000700 1 -- 172.21.15.1:0/1524296999 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4a4c00be50 con 0x7f4a5c104980 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.167+0000 7f4a5a000700 1 -- 172.21.15.1:0/1524296999 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a4c010380 con 0x7f4a5c104980 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.168+0000 7f4a62000700 1 -- 172.21.15.1:0/1524296999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 msgr2=0x7f4a5c104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.168+0000 7f4a62000700 1 --2- 172.21.15.1:0/1524296999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c104d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0098e0 tx=0x7f4a4c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.168+0000 7f4a62000700 1 -- 172.21.15.1:0/1524296999 shutdown_connections 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.168+0000 7f4a62000700 1 --2- 172.21.15.1:0/1524296999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c104d90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.168+0000 7f4a62000700 1 -- 172.21.15.1:0/1524296999 >> 172.21.15.1:0/1524296999 conn(0x7f4a5c078030 msgr2=0x7f4a5c078450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:33.241 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.168+0000 7f4a62000700 1 -- 172.21.15.1:0/1524296999 shutdown_connections 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.168+0000 7f4a62000700 1 -- 172.21.15.1:0/1524296999 wait complete. 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.169+0000 7f4a62000700 1 Processor -- start 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.169+0000 7f4a62000700 1 -- start start 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.169+0000 7f4a62000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.169+0000 7f4a62000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c113ca0 con 0x7f4a5c104980 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.169+0000 7f4a5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.169+0000 7f4a5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59126/0 (socket says 172.21.15.1:59126) 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.169+0000 7f4a5b400700 1 -- 172.21.15.1:0/1061423908 learned_addr learned my addr 172.21.15.1:0/1061423908 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.170+0000 7f4a5b400700 1 -- 172.21.15.1:0/1061423908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c0094f0 con 0x7f4a5c104980 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.170+0000 7f4a5b400700 1 --2- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c113760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c000c00 tx=0x7f4a4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.170+0000 7f4a53400700 1 -- 172.21.15.1:0/1061423908 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a4c0107c0 con 0x7f4a5c104980 2024-02-28T16:18:33.242 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.170+0000 7f4a53400700 1 -- 172.21.15.1:0/1061423908 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4a4c010db0 con 0x7f4a5c104980 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.170+0000 7f4a53400700 1 -- 172.21.15.1:0/1061423908 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a4c01ad00 con 0x7f4a5c104980 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.171+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c113ea0 con 0x7f4a5c104980 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.171+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c114b40 con 0x7f4a5c104980 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.172+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c05f5c0 con 0x7f4a5c104980 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.172+0000 7f4a53400700 1 -- 172.21.15.1:0/1061423908 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4a4c01a530 con 0x7f4a5c104980 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.172+0000 7f4a53400700 1 --2- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4a400367e0 0x7f4a40038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.172+0000 7f4a53400700 1 -- 172.21.15.1:0/1061423908 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4a4c04a6c0 con 0x7f4a5c104980 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.173+0000 7f4a5aa00700 1 --2- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4a400367e0 0x7f4a40038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.173+0000 7f4a5aa00700 1 --2- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4a400367e0 0x7f4a40038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009990 tx=0x7f4a48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.177+0000 7f4a53400700 1 -- 172.21.15.1:0/1061423908 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4a4c010920 con 0x7f4a5c104980 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.405+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a5c102f30 con 0x7f4a400367e0 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.413+0000 7f4a53400700 1 -- 172.21.15.1:0/1061423908 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4a5c102f30 con 0x7f4a400367e0 2024-02-28T16:18:33.243 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4a400367e0 msgr2=0x7f4a40038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 --2- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4a400367e0 0x7f4a40038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009990 tx=0x7f4a48006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 msgr2=0x7f4a5c113760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 --2- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c113760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c000c00 tx=0x7f4a4c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 shutdown_connections 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 --2- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f4a400367e0 0x7f4a40038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 --2- 172.21.15.1:0/1061423908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a5c104980 0x7f4a5c113760 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 >> 172.21.15.1:0/1061423908 conn(0x7f4a5c078030 msgr2=0x7f4a5c102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.416+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 shutdown_connections 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:32.417+0000 7f4a62000700 1 -- 172.21.15.1:0/1061423908 wait complete. 2024-02-28T16:18:33.244 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying crash service with default placement... 2024-02-28T16:18:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:33 smithi001 ceph-mon[28991]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:33 smithi001 ceph-mon[28991]: Saving service mgr spec with placement count:2 2024-02-28T16:18:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:33 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:33 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:34 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:34.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:34 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:34.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-28T16:18:34.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.856+0000 7fb62f000700 1 Processor -- start 2024-02-28T16:18:34.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.856+0000 7fb62f000700 1 -- start start 2024-02-28T16:18:34.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.856+0000 7fb62f000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:34.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.857+0000 7fb62f000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6281030c0 con 0x7fb628102770 2024-02-28T16:18:34.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.857+0000 7fb627e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:34.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.857+0000 7fb627e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59128/0 (socket says 172.21.15.1:59128) 2024-02-28T16:18:34.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.857+0000 7fb627e00700 1 -- 172.21.15.1:0/577811138 learned_addr learned my addr 172.21.15.1:0/577811138 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.857+0000 7fb627e00700 1 -- 172.21.15.1:0/577811138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb628103200 con 0x7fb628102770 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.858+0000 7fb627e00700 1 --2- 172.21.15.1:0/577811138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628102b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6180098e0 tx=0x7fb618009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f25ba452f671d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.858+0000 7fb626a00700 1 -- 172.21.15.1:0/577811138 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb61800b860 con 0x7fb628102770 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.858+0000 7fb626a00700 1 -- 172.21.15.1:0/577811138 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb61800be90 con 0x7fb628102770 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.858+0000 7fb626a00700 1 -- 172.21.15.1:0/577811138 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6180103c0 con 0x7fb628102770 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.859+0000 7fb62f000700 1 -- 172.21.15.1:0/577811138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 msgr2=0x7fb628102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.859+0000 7fb62f000700 1 --2- 172.21.15.1:0/577811138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628102b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6180098e0 tx=0x7fb618009250 comp rx=0 tx=0).stop 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.859+0000 7fb62f000700 1 -- 172.21.15.1:0/577811138 shutdown_connections 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.859+0000 7fb62f000700 1 --2- 172.21.15.1:0/577811138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628102b80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.859+0000 7fb62f000700 1 -- 172.21.15.1:0/577811138 >> 172.21.15.1:0/577811138 conn(0x7fb6280fded0 msgr2=0x7fb628100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.859+0000 7fb62f000700 1 -- 172.21.15.1:0/577811138 shutdown_connections 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.859+0000 7fb62f000700 1 -- 172.21.15.1:0/577811138 wait complete. 2024-02-28T16:18:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.861+0000 7fb62f000700 1 Processor -- start 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.861+0000 7fb62f000700 1 -- start start 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.861+0000 7fb62f000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628077c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.861+0000 7fb62f000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb628076080 con 0x7fb628102770 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.862+0000 7fb627e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628077c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.862+0000 7fb627e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628077c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59130/0 (socket says 172.21.15.1:59130) 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.862+0000 7fb627e00700 1 -- 172.21.15.1:0/3259653189 learned_addr learned my addr 172.21.15.1:0/3259653189 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.862+0000 7fb627e00700 1 -- 172.21.15.1:0/3259653189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6180094f0 con 0x7fb628102770 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.862+0000 7fb627e00700 1 --2- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628077c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb6180090b0 tx=0x7fb618009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.863+0000 7fb61fe00700 1 -- 172.21.15.1:0/3259653189 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb618010840 con 0x7fb628102770 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.863+0000 7fb61fe00700 1 -- 172.21.15.1:0/3259653189 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb618010e70 con 0x7fb628102770 2024-02-28T16:18:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.863+0000 7fb61fe00700 1 -- 172.21.15.1:0/3259653189 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb618024e40 con 0x7fb628102770 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.863+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb628076280 con 0x7fb628102770 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.863+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb628076710 con 0x7fb628102770 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.864+0000 7fb61fe00700 1 -- 172.21.15.1:0/3259653189 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb618024430 con 0x7fb628102770 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.864+0000 7fb61fe00700 1 --2- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb60c0367e0 0x7fb60c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.865+0000 7fb61fe00700 1 -- 172.21.15.1:0/3259653189 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb61804a820 con 0x7fb628102770 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.865+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62804bed0 con 0x7fb628102770 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.865+0000 7fb627400700 1 --2- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb60c0367e0 0x7fb60c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.866+0000 7fb627400700 1 --2- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb60c0367e0 0x7fb60c038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb614009990 tx=0x7fb614006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:33.870+0000 7fb61fe00700 1 -- 172.21.15.1:0/3259653189 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb61802c930 con 0x7fb628102770 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.131+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb628076b80 con 0x7fb60c0367e0 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.137+0000 7fb61fe00700 1 -- 172.21.15.1:0/3259653189 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb628076b80 con 0x7fb60c0367e0 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.139+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb60c0367e0 msgr2=0x7fb60c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.139+0000 7fb62f000700 1 --2- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb60c0367e0 0x7fb60c038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb614009990 tx=0x7fb614006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.139+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 msgr2=0x7fb628077c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.139+0000 7fb62f000700 1 --2- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628077c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb6180090b0 tx=0x7fb618009250 comp rx=0 tx=0).stop 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.139+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 shutdown_connections 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.139+0000 7fb62f000700 1 --2- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb60c0367e0 0x7fb60c038c90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.139+0000 7fb62f000700 1 --2- 172.21.15.1:0/3259653189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb628102770 0x7fb628077c00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.139+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 >> 172.21.15.1:0/3259653189 conn(0x7fb6280fded0 msgr2=0x7fb6280fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.140+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 shutdown_connections 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:34.140+0000 7fb62f000700 1 -- 172.21.15.1:0/3259653189 wait complete. 2024-02-28T16:18:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying prometheus service with default placement... 2024-02-28T16:18:35.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:35 smithi001 ceph-mon[28991]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:35.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:35 smithi001 ceph-mon[28991]: Saving service crash spec with placement * 2024-02-28T16:18:36.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:36 smithi001 ceph-mon[28991]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:18:36.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:36 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:36.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-28T16:18:36.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.580+0000 7eff7c000700 1 Processor -- start 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.580+0000 7eff7c000700 1 -- start start 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.580+0000 7eff7c000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff74102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.580+0000 7eff7c000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff741030d0 con 0x7eff74102780 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.581+0000 7eff79600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff74102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.581+0000 7eff79600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff74102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59142/0 (socket says 172.21.15.1:59142) 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.581+0000 7eff79600700 1 -- 172.21.15.1:0/4150989134 learned_addr learned my addr 172.21.15.1:0/4150989134 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.581+0000 7eff79600700 1 -- 172.21.15.1:0/4150989134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff74103210 con 0x7eff74102780 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.581+0000 7eff79600700 1 --2- 172.21.15.1:0/4150989134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff74102b90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff680098e0 tx=0x7eff68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a42626cef9093df server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.582+0000 7eff73e00700 1 -- 172.21.15.1:0/4150989134 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff6800b860 con 0x7eff74102780 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.582+0000 7eff73e00700 1 -- 172.21.15.1:0/4150989134 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff6800be90 con 0x7eff74102780 2024-02-28T16:18:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.582+0000 7eff73e00700 1 -- 172.21.15.1:0/4150989134 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff680103c0 con 0x7eff74102780 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.582+0000 7eff7c000700 1 -- 172.21.15.1:0/4150989134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 msgr2=0x7eff74102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.582+0000 7eff7c000700 1 --2- 172.21.15.1:0/4150989134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff74102b90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff680098e0 tx=0x7eff68009250 comp rx=0 tx=0).stop 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.582+0000 7eff7c000700 1 -- 172.21.15.1:0/4150989134 shutdown_connections 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.582+0000 7eff7c000700 1 --2- 172.21.15.1:0/4150989134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff74102b90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.582+0000 7eff7c000700 1 -- 172.21.15.1:0/4150989134 >> 172.21.15.1:0/4150989134 conn(0x7eff740fdf00 msgr2=0x7eff74100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.583+0000 7eff7c000700 1 -- 172.21.15.1:0/4150989134 shutdown_connections 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.583+0000 7eff7c000700 1 -- 172.21.15.1:0/4150989134 wait complete. 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.583+0000 7eff7c000700 1 Processor -- start 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.584+0000 7eff7c000700 1 -- start start 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.584+0000 7eff7c000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff741137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:36.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.584+0000 7eff7c000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff74113d10 con 0x7eff74102780 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.584+0000 7eff79600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff741137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.584+0000 7eff79600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff741137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59152/0 (socket says 172.21.15.1:59152) 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.584+0000 7eff79600700 1 -- 172.21.15.1:0/1662799720 learned_addr learned my addr 172.21.15.1:0/1662799720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.585+0000 7eff79600700 1 -- 172.21.15.1:0/1662799720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff680094f0 con 0x7eff74102780 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.585+0000 7eff79600700 1 --2- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff741137d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7eff680090b0 tx=0x7eff68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.585+0000 7eff71600700 1 -- 172.21.15.1:0/1662799720 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff68010840 con 0x7eff74102780 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.585+0000 7eff71600700 1 -- 172.21.15.1:0/1662799720 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff68010e70 con 0x7eff74102780 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.585+0000 7eff71600700 1 -- 172.21.15.1:0/1662799720 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff68024e40 con 0x7eff74102780 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.585+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff74113f10 con 0x7eff74102780 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.585+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff74114bb0 con 0x7eff74102780 2024-02-28T16:18:36.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.586+0000 7eff71600700 1 -- 172.21.15.1:0/1662799720 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7eff68024430 con 0x7eff74102780 2024-02-28T16:18:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.586+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7405f5c0 con 0x7eff74102780 2024-02-28T16:18:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.587+0000 7eff71600700 1 --2- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7eff5c036790 0x7eff5c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.587+0000 7eff71600700 1 -- 172.21.15.1:0/1662799720 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7eff6804a810 con 0x7eff74102780 2024-02-28T16:18:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.587+0000 7eff78c00700 1 --2- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7eff5c036790 0x7eff5c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.588+0000 7eff78c00700 1 --2- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7eff5c036790 0x7eff5c038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7eff60009990 tx=0x7eff60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.592+0000 7eff71600700 1 -- 172.21.15.1:0/1662799720 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7eff6802c930 con 0x7eff74102780 2024-02-28T16:18:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.818+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7eff740ff3e0 con 0x7eff5c036790 2024-02-28T16:18:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.828+0000 7eff71600700 1 -- 172.21.15.1:0/1662799720 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7eff740ff3e0 con 0x7eff5c036790 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7eff5c036790 msgr2=0x7eff5c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 --2- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7eff5c036790 0x7eff5c038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7eff60009990 tx=0x7eff60006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 msgr2=0x7eff741137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 --2- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff741137d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7eff680090b0 tx=0x7eff68009250 comp rx=0 tx=0).stop 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 shutdown_connections 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 --2- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7eff5c036790 0x7eff5c038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 --2- 172.21.15.1:0/1662799720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff74102780 0x7eff741137d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 >> 172.21.15.1:0/1662799720 conn(0x7eff740fdf00 msgr2=0x7eff740fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.831+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 shutdown_connections 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:35.832+0000 7eff7c000700 1 -- 172.21.15.1:0/1662799720 wait complete. 2024-02-28T16:18:36.845 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying grafana service with default placement... 2024-02-28T16:18:37.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:37 smithi001 ceph-mon[28991]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:37.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:37 smithi001 ceph-mon[28991]: Saving service prometheus spec with placement count:1 2024-02-28T16:18:37.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:37 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:38.340 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-28T16:18:38.340 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.428+0000 7f8dd8c00700 1 Processor -- start 2024-02-28T16:18:38.340 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.428+0000 7f8dd8c00700 1 -- start start 2024-02-28T16:18:38.340 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.429+0000 7f8dd8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.429+0000 7f8dd8c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd41030c0 con 0x7f8dd4102770 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.429+0000 7f8dd1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.429+0000 7f8dd1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59162/0 (socket says 172.21.15.1:59162) 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.429+0000 7f8dd1e00700 1 -- 172.21.15.1:0/3263220416 learned_addr learned my addr 172.21.15.1:0/3263220416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.430+0000 7f8dd1e00700 1 -- 172.21.15.1:0/3263220416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd4103200 con 0x7f8dd4102770 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.430+0000 7f8dd1e00700 1 --2- 172.21.15.1:0/3263220416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4102b80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40098e0 tx=0x7f8dc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377675b7267ba9bc server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.430+0000 7f8dd0a00700 1 -- 172.21.15.1:0/3263220416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc400b860 con 0x7f8dd4102770 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.431+0000 7f8dd0a00700 1 -- 172.21.15.1:0/3263220416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8dc400be90 con 0x7f8dd4102770 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.431+0000 7f8dd0a00700 1 -- 172.21.15.1:0/3263220416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc40103c0 con 0x7f8dd4102770 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.431+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3263220416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 msgr2=0x7f8dd4102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.431+0000 7f8dd8c00700 1 --2- 172.21.15.1:0/3263220416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4102b80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40098e0 tx=0x7f8dc4009250 comp rx=0 tx=0).stop 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.431+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3263220416 shutdown_connections 2024-02-28T16:18:38.341 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.431+0000 7f8dd8c00700 1 --2- 172.21.15.1:0/3263220416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4102b80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:38.342 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.431+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3263220416 >> 172.21.15.1:0/3263220416 conn(0x7f8dd40fded0 msgr2=0x7f8dd4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:38.342 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.432+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3263220416 shutdown_connections 2024-02-28T16:18:38.342 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.432+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3263220416 wait complete. 2024-02-28T16:18:38.342 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.432+0000 7f8dd8c00700 1 Processor -- start 2024-02-28T16:18:38.342 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.433+0000 7f8dd8c00700 1 -- start start 2024-02-28T16:18:38.342 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.433+0000 7f8dd8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4077c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:38.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.433+0000 7f8dd8c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd4076080 con 0x7f8dd4102770 2024-02-28T16:18:38.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.433+0000 7f8dd1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4077c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:38.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.433+0000 7f8dd1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4077c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59174/0 (socket says 172.21.15.1:59174) 2024-02-28T16:18:38.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.433+0000 7f8dd1e00700 1 -- 172.21.15.1:0/3298219443 learned_addr learned my addr 172.21.15.1:0/3298219443 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.434+0000 7f8dd1e00700 1 -- 172.21.15.1:0/3298219443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc40094f0 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.434+0000 7f8dd1e00700 1 --2- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4077c00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4006b00 tx=0x7f8dc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.434+0000 7f8dca000700 1 -- 172.21.15.1:0/3298219443 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc40109a0 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.434+0000 7f8dca000700 1 -- 172.21.15.1:0/3298219443 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8dc401a430 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.434+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd4076280 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.434+0000 7f8dca000700 1 -- 172.21.15.1:0/3298219443 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc40245b0 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.435+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd4076710 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.436+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd410b690 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.436+0000 7f8dca000700 1 -- 172.21.15.1:0/3298219443 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8dc401aaa0 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.436+0000 7f8dca000700 1 --2- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8db80367e0 0x7f8db8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.436+0000 7f8dca000700 1 -- 172.21.15.1:0/3298219443 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8dc404a680 con 0x7f8dd4102770 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.437+0000 7f8dd1400700 1 --2- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8db80367e0 0x7f8db8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.437+0000 7f8dd1400700 1 --2- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8db80367e0 0x7f8db8038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009990 tx=0x7f8dc0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:38.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.442+0000 7f8dca000700 1 -- 172.21.15.1:0/3298219443 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8dc4018940 con 0x7f8dd4102770 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.667+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dd41046c0 con 0x7f8db80367e0 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.681+0000 7f8dca000700 1 -- 172.21.15.1:0/3298219443 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8dd41046c0 con 0x7f8db80367e0 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.684+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8db80367e0 msgr2=0x7f8db8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.684+0000 7f8dd8c00700 1 --2- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8db80367e0 0x7f8db8038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009990 tx=0x7f8dc0006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.685+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 msgr2=0x7f8dd4077c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.685+0000 7f8dd8c00700 1 --2- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4077c00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4006b00 tx=0x7f8dc4009250 comp rx=0 tx=0).stop 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.685+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 shutdown_connections 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.685+0000 7f8dd8c00700 1 --2- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8db80367e0 0x7f8db8038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.685+0000 7f8dd8c00700 1 --2- 172.21.15.1:0/3298219443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd4102770 0x7f8dd4077c00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.685+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 >> 172.21.15.1:0/3298219443 conn(0x7f8dd40fded0 msgr2=0x7f8dd40fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.685+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 shutdown_connections 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:37.685+0000 7f8dd8c00700 1 -- 172.21.15.1:0/3298219443 wait complete. 2024-02-28T16:18:38.347 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying node-exporter service with default placement... 2024-02-28T16:18:38.949 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:38 smithi001 ceph-mon[28991]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:18:38.950 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:38 smithi001 ceph-mon[28991]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:38.950 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:38 smithi001 ceph-mon[28991]: Saving service grafana spec with placement count:1 2024-02-28T16:18:38.950 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:38 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:38.950 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:38 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:40.449 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:40 smithi001 ceph-mon[28991]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:40.450 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:40 smithi001 ceph-mon[28991]: Saving service node-exporter spec with placement * 2024-02-28T16:18:40.450 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:40 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:40.450 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:40 smithi001 ceph-mon[28991]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:18:40.462 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.957+0000 7f955cc00700 1 Processor -- start 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.958+0000 7f955cc00700 1 -- start start 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.958+0000 7f955cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.958+0000 7f955cc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9558101fc0 con 0x7f9558101670 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.958+0000 7f9555e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.958+0000 7f9555e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59184/0 (socket says 172.21.15.1:59184) 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.958+0000 7f9555e00700 1 -- 172.21.15.1:0/2907707213 learned_addr learned my addr 172.21.15.1:0/2907707213 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.959+0000 7f9555e00700 1 -- 172.21.15.1:0/2907707213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9558102100 con 0x7f9558101670 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.959+0000 7f9555e00700 1 --2- 172.21.15.1:0/2907707213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558101a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f954800bb80 tx=0x7f954800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3ead4090c92820 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.960+0000 7f9554a00700 1 -- 172.21.15.1:0/2907707213 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f954800dd00 con 0x7f9558101670 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.960+0000 7f9554a00700 1 -- 172.21.15.1:0/2907707213 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f954800de60 con 0x7f9558101670 2024-02-28T16:18:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.960+0000 7f9554a00700 1 -- 172.21.15.1:0/2907707213 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9548018920 con 0x7f9558101670 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.961+0000 7f955cc00700 1 -- 172.21.15.1:0/2907707213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 msgr2=0x7f9558101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.961+0000 7f955cc00700 1 --2- 172.21.15.1:0/2907707213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558101a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f954800bb80 tx=0x7f954800b4f0 comp rx=0 tx=0).stop 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.961+0000 7f955cc00700 1 -- 172.21.15.1:0/2907707213 shutdown_connections 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.961+0000 7f955cc00700 1 --2- 172.21.15.1:0/2907707213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558101a80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.961+0000 7f955cc00700 1 -- 172.21.15.1:0/2907707213 >> 172.21.15.1:0/2907707213 conn(0x7f95580fcdd0 msgr2=0x7f95580ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.961+0000 7f955cc00700 1 -- 172.21.15.1:0/2907707213 shutdown_connections 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.961+0000 7f955cc00700 1 -- 172.21.15.1:0/2907707213 wait complete. 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.962+0000 7f955cc00700 1 Processor -- start 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.962+0000 7f955cc00700 1 -- start start 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.962+0000 7f955cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558113750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.962+0000 7f955cc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9558113c90 con 0x7f9558101670 2024-02-28T16:18:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.963+0000 7f9555e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558113750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.963+0000 7f9555e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558113750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59190/0 (socket says 172.21.15.1:59190) 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.963+0000 7f9555e00700 1 -- 172.21.15.1:0/512067160 learned_addr learned my addr 172.21.15.1:0/512067160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.963+0000 7f9555e00700 1 -- 172.21.15.1:0/512067160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954800b790 con 0x7f9558101670 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.964+0000 7f9555e00700 1 --2- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558113750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95480106a0 tx=0x7f954800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.964+0000 7f954e000700 1 -- 172.21.15.1:0/512067160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f954800dc70 con 0x7f9558101670 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.964+0000 7f954e000700 1 -- 172.21.15.1:0/512067160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9548010c30 con 0x7f9558101670 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.964+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9558113e90 con 0x7f9558101670 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.964+0000 7f954e000700 1 -- 172.21.15.1:0/512067160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9548021550 con 0x7f9558101670 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.964+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9558114b30 con 0x7f9558101670 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.965+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955805f5c0 con 0x7f9558101670 2024-02-28T16:18:40.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.965+0000 7f954e000700 1 -- 172.21.15.1:0/512067160 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f95480223f0 con 0x7f9558101670 2024-02-28T16:18:40.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.966+0000 7f954e000700 1 --2- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f953c036740 0x7f953c038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:40.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.966+0000 7f954e000700 1 -- 172.21.15.1:0/512067160 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9548014070 con 0x7f9558101670 2024-02-28T16:18:40.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.966+0000 7f9555400700 1 --2- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f953c036740 0x7f953c038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:40.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.967+0000 7f9555400700 1 --2- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f953c036740 0x7f953c038bf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9544009990 tx=0x7f9544006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:40.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:38.971+0000 7f954e000700 1 -- 172.21.15.1:0/512067160 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9548021810 con 0x7f9558101670 2024-02-28T16:18:40.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.201+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f9558115710 con 0x7f953c036740 2024-02-28T16:18:40.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.207+0000 7f954e000700 1 -- 172.21.15.1:0/512067160 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f9558115710 con 0x7f953c036740 2024-02-28T16:18:40.467 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.209+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f953c036740 msgr2=0x7f953c038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:40.467 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.210+0000 7f955cc00700 1 --2- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f953c036740 0x7f953c038bf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9544009990 tx=0x7f9544006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:40.467 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.210+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 msgr2=0x7f9558113750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:40.467 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.210+0000 7f955cc00700 1 --2- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558113750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95480106a0 tx=0x7f954800b4f0 comp rx=0 tx=0).stop 2024-02-28T16:18:40.467 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.210+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 shutdown_connections 2024-02-28T16:18:40.467 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.210+0000 7f955cc00700 1 --2- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f953c036740 0x7f953c038bf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:40.467 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.210+0000 7f955cc00700 1 --2- 172.21.15.1:0/512067160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9558101670 0x7f9558113750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:40.468 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.210+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 >> 172.21.15.1:0/512067160 conn(0x7f95580fcdd0 msgr2=0x7f95580ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:40.468 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.211+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 shutdown_connections 2024-02-28T16:18:40.468 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:39.211+0000 7f955cc00700 1 -- 172.21.15.1:0/512067160 wait complete. 2024-02-28T16:18:40.468 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying alertmanager service with default placement... 2024-02-28T16:18:42.449 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:42 smithi001 ceph-mon[28991]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:18:42.450 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:42 smithi001 ceph-mon[28991]: from='mgr.14120 172.21.15.1:0/794717220' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:42.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-28T16:18:42.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.518+0000 7f487e200700 1 Processor -- start 2024-02-28T16:18:42.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.518+0000 7f487e200700 1 -- start start 2024-02-28T16:18:42.466 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.519+0000 7f487e200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f4878105400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.519+0000 7f487e200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4878105940 con 0x7f4878076ba0 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.519+0000 7f4877400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f4878105400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.519+0000 7f4877400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f4878105400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59194/0 (socket says 172.21.15.1:59194) 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.519+0000 7f4877400700 1 -- 172.21.15.1:0/972108084 learned_addr learned my addr 172.21.15.1:0/972108084 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.520+0000 7f4877400700 1 -- 172.21.15.1:0/972108084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878105a80 con 0x7f4878076ba0 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.520+0000 7f4877400700 1 --2- 172.21.15.1:0/972108084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f4878105400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f48680098e0 tx=0x7f4868009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a727bb5ba664164 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.520+0000 7f4876000700 1 -- 172.21.15.1:0/972108084 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f486800b8f0 con 0x7f4878076ba0 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.521+0000 7f4876000700 1 -- 172.21.15.1:0/972108084 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4868011420 con 0x7f4878076ba0 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.521+0000 7f4876000700 1 -- 172.21.15.1:0/972108084 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4868010540 con 0x7f4878076ba0 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.521+0000 7f487e200700 1 -- 172.21.15.1:0/972108084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 msgr2=0x7f4878105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.521+0000 7f487e200700 1 --2- 172.21.15.1:0/972108084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f4878105400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f48680098e0 tx=0x7f4868009250 comp rx=0 tx=0).stop 2024-02-28T16:18:42.469 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.521+0000 7f487e200700 1 -- 172.21.15.1:0/972108084 shutdown_connections 2024-02-28T16:18:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.521+0000 7f487e200700 1 --2- 172.21.15.1:0/972108084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f4878105400 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.521+0000 7f487e200700 1 -- 172.21.15.1:0/972108084 >> 172.21.15.1:0/972108084 conn(0x7f48780fcdd0 msgr2=0x7f48780ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.522+0000 7f487e200700 1 -- 172.21.15.1:0/972108084 shutdown_connections 2024-02-28T16:18:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.522+0000 7f487e200700 1 -- 172.21.15.1:0/972108084 wait complete. 2024-02-28T16:18:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.522+0000 7f487e200700 1 Processor -- start 2024-02-28T16:18:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.523+0000 7f487e200700 1 -- start start 2024-02-28T16:18:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.523+0000 7f487e200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f48781137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.523+0000 7f487e200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4878113d10 con 0x7f4878076ba0 2024-02-28T16:18:42.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.523+0000 7f4877400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f48781137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:42.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.523+0000 7f4877400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f48781137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59204/0 (socket says 172.21.15.1:59204) 2024-02-28T16:18:42.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.523+0000 7f4877400700 1 -- 172.21.15.1:0/1519468993 learned_addr learned my addr 172.21.15.1:0/1519468993 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:42.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.524+0000 7f4877400700 1 -- 172.21.15.1:0/1519468993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680094f0 con 0x7f4878076ba0 2024-02-28T16:18:42.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.524+0000 7f4877400700 1 --2- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f48781137d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4868011b40 tx=0x7f4868009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:42.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.524+0000 7f486f400700 1 -- 172.21.15.1:0/1519468993 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f486800beb0 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.524+0000 7f486f400700 1 -- 172.21.15.1:0/1519468993 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4868010c80 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.524+0000 7f486f400700 1 -- 172.21.15.1:0/1519468993 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4868024d70 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.524+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4878113f10 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.525+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4878114bb0 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.526+0000 7f486f400700 1 -- 172.21.15.1:0/1519468993 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4868024650 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.526+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487805f5c0 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.526+0000 7f486f400700 1 --2- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f485c036870 0x7f485c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.526+0000 7f486f400700 1 -- 172.21.15.1:0/1519468993 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f486804a5e0 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.526+0000 7f4876a00700 1 --2- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f485c036870 0x7f485c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.527+0000 7f4876a00700 1 --2- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f485c036870 0x7f485c038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4864009990 tx=0x7f4864006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.532+0000 7f486f400700 1 -- 172.21.15.1:0/1519468993 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4868024430 con 0x7f4878076ba0 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.755+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f48780fe2e0 con 0x7f485c036870 2024-02-28T16:18:42.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.765+0000 7f486f400700 1 -- 172.21.15.1:0/1519468993 <== mgr.14120 v2:172.21.15.1:6800/3320028482 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f48780fe2e0 con 0x7f485c036870 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f485c036870 msgr2=0x7f485c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 --2- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f485c036870 0x7f485c038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4864009990 tx=0x7f4864006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 msgr2=0x7f48781137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 --2- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f48781137d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4868011b40 tx=0x7f4868009250 comp rx=0 tx=0).stop 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 shutdown_connections 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 --2- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f485c036870 0x7f485c038d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 --2- 172.21.15.1:0/1519468993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4878076ba0 0x7f48781137d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 >> 172.21.15.1:0/1519468993 conn(0x7f48780fcdd0 msgr2=0x7f48780fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.768+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 shutdown_connections 2024-02-28T16:18:42.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:41.769+0000 7f487e200700 1 -- 172.21.15.1:0/1519468993 wait complete. 2024-02-28T16:18:43.667 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:43 smithi001 ceph-mon[28991]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:18:43.667 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:43 smithi001 ceph-mon[28991]: Saving service alertmanager spec with placement count:1 2024-02-28T16:18:43.667 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:43 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2579671449' entity='client.admin' 2024-02-28T16:18:43.972 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.053+0000 7fb3ea400700 1 Processor -- start 2024-02-28T16:18:43.972 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.053+0000 7fb3ea400700 1 -- start start 2024-02-28T16:18:43.972 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.054+0000 7fb3ea400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e40760c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:43.972 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.054+0000 7fb3ea400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e4077fc0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.972 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.054+0000 7fb3e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e40760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:43.972 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.054+0000 7fb3e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e40760c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42666/0 (socket says 172.21.15.1:42666) 2024-02-28T16:18:43.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.054+0000 7fb3e3400700 1 -- 172.21.15.1:0/2725265191 learned_addr learned my addr 172.21.15.1:0/2725265191 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:43.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.055+0000 7fb3e3400700 1 -- 172.21.15.1:0/2725265191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e4076600 con 0x7fb3e4077bb0 2024-02-28T16:18:43.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.055+0000 7fb3e3400700 1 --2- 172.21.15.1:0/2725265191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e40760c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40098e0 tx=0x7fb3d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5c0b2dea5696c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:43.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.055+0000 7fb3e2000700 1 -- 172.21.15.1:0/2725265191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d400b860 con 0x7fb3e4077bb0 2024-02-28T16:18:43.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.055+0000 7fb3e2000700 1 -- 172.21.15.1:0/2725265191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3d400be90 con 0x7fb3e4077bb0 2024-02-28T16:18:43.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.055+0000 7fb3e2000700 1 -- 172.21.15.1:0/2725265191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d40103c0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.056+0000 7fb3ea400700 1 -- 172.21.15.1:0/2725265191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 msgr2=0x7fb3e40760c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.056+0000 7fb3ea400700 1 --2- 172.21.15.1:0/2725265191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e40760c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40098e0 tx=0x7fb3d4009250 comp rx=0 tx=0).stop 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.056+0000 7fb3ea400700 1 -- 172.21.15.1:0/2725265191 shutdown_connections 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.056+0000 7fb3ea400700 1 --2- 172.21.15.1:0/2725265191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e40760c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.056+0000 7fb3ea400700 1 -- 172.21.15.1:0/2725265191 >> 172.21.15.1:0/2725265191 conn(0x7fb3e40fded0 msgr2=0x7fb3e4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.057+0000 7fb3ea400700 1 -- 172.21.15.1:0/2725265191 shutdown_connections 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.057+0000 7fb3ea400700 1 -- 172.21.15.1:0/2725265191 wait complete. 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.057+0000 7fb3ea400700 1 Processor -- start 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.058+0000 7fb3ea400700 1 -- start start 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.058+0000 7fb3ea400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e4113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.058+0000 7fb3ea400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e4113fe0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.058+0000 7fb3e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e4113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.058+0000 7fb3e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e4113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42676/0 (socket says 172.21.15.1:42676) 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.058+0000 7fb3e3400700 1 -- 172.21.15.1:0/2579671449 learned_addr learned my addr 172.21.15.1:0/2579671449 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:43.976 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.059+0000 7fb3e3400700 1 -- 172.21.15.1:0/2579671449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d40094f0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.977 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.059+0000 7fb3e3400700 1 --2- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e4113aa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40090b0 tx=0x7fb3d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:43.977 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.059+0000 7fb3db400700 1 -- 172.21.15.1:0/2579671449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d40118f0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.979 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.059+0000 7fb3db400700 1 -- 172.21.15.1:0/2579671449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3d4010d10 con 0x7fb3e4077bb0 2024-02-28T16:18:43.979 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.059+0000 7fb3db400700 1 -- 172.21.15.1:0/2579671449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d4024e70 con 0x7fb3e4077bb0 2024-02-28T16:18:43.979 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.059+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e41141e0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.979 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.059+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e4114e80 con 0x7fb3e4077bb0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.060+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e405f5c0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.061+0000 7fb3db400700 1 -- 172.21.15.1:0/2579671449 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb3d4024440 con 0x7fb3e4077bb0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.061+0000 7fb3db400700 1 --2- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb3c8036790 0x7fb3c8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.061+0000 7fb3db400700 1 -- 172.21.15.1:0/2579671449 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb3d404a5c0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.062+0000 7fb3e2a00700 1 --2- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb3c8036790 0x7fb3c8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.062+0000 7fb3e2a00700 1 --2- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb3c8036790 0x7fb3c8038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0009990 tx=0x7fb3d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.066+0000 7fb3db400700 1 -- 172.21.15.1:0/2579671449 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb3d4010e80 con 0x7fb3e4077bb0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.280+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb3e4108af0 con 0x7fb3e4077bb0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.288+0000 7fb3db400700 1 -- 172.21.15.1:0/2579671449 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb3d4022340 con 0x7fb3e4077bb0 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.292+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb3c8036790 msgr2=0x7fb3c8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.292+0000 7fb3ea400700 1 --2- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb3c8036790 0x7fb3c8038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0009990 tx=0x7fb3d0006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.292+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 msgr2=0x7fb3e4113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:43.980 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.292+0000 7fb3ea400700 1 --2- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e4113aa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40090b0 tx=0x7fb3d4009250 comp rx=0 tx=0).stop 2024-02-28T16:18:43.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.292+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 shutdown_connections 2024-02-28T16:18:43.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.293+0000 7fb3ea400700 1 --2- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fb3c8036790 0x7fb3c8038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:43.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.293+0000 7fb3ea400700 1 --2- 172.21.15.1:0/2579671449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3e4077bb0 0x7fb3e4113aa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:43.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.293+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 >> 172.21.15.1:0/2579671449 conn(0x7fb3e40fded0 msgr2=0x7fb3e41001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:43.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.293+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 shutdown_connections 2024-02-28T16:18:43.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:43.293+0000 7fb3ea400700 1 -- 172.21.15.1:0/2579671449 wait complete. 2024-02-28T16:18:44.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:44 smithi001 ceph-mon[28991]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:18:45.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.587+0000 7f58bc600700 1 Processor -- start 2024-02-28T16:18:45.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.588+0000 7f58bc600700 1 -- start start 2024-02-28T16:18:45.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.588+0000 7f58bc600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b40771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:45.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.588+0000 7f58bc600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b40790d0 con 0x7f58b4078cc0 2024-02-28T16:18:45.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.588+0000 7f58b9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b40771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:45.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.589+0000 7f58b9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b40771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42686/0 (socket says 172.21.15.1:42686) 2024-02-28T16:18:45.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.589+0000 7f58b9c00700 1 -- 172.21.15.1:0/1190211685 learned_addr learned my addr 172.21.15.1:0/1190211685 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:45.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.589+0000 7f58b9c00700 1 -- 172.21.15.1:0/1190211685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4077710 con 0x7f58b4078cc0 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.589+0000 7f58b9c00700 1 --2- 172.21.15.1:0/1190211685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b40771d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58a80098e0 tx=0x7f58a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab02495f1a378f10 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58b3e00700 1 -- 172.21.15.1:0/1190211685 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a800b860 con 0x7f58b4078cc0 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58b3e00700 1 -- 172.21.15.1:0/1190211685 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58a800be90 con 0x7f58b4078cc0 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58b3e00700 1 -- 172.21.15.1:0/1190211685 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a80103c0 con 0x7f58b4078cc0 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58bc600700 1 -- 172.21.15.1:0/1190211685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 msgr2=0x7f58b40771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58bc600700 1 --2- 172.21.15.1:0/1190211685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b40771d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58a80098e0 tx=0x7f58a8009250 comp rx=0 tx=0).stop 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58bc600700 1 -- 172.21.15.1:0/1190211685 shutdown_connections 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58bc600700 1 --2- 172.21.15.1:0/1190211685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b40771d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58bc600700 1 -- 172.21.15.1:0/1190211685 >> 172.21.15.1:0/1190211685 conn(0x7f58b40fdf00 msgr2=0x7f58b4100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.590+0000 7f58bc600700 1 -- 172.21.15.1:0/1190211685 shutdown_connections 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.591+0000 7f58bc600700 1 -- 172.21.15.1:0/1190211685 wait complete. 2024-02-28T16:18:45.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.591+0000 7f58bc600700 1 Processor -- start 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.592+0000 7f58bc600700 1 -- start start 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.592+0000 7f58bc600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b41137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.592+0000 7f58bc600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b4113d20 con 0x7f58b4078cc0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.592+0000 7f58b9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b41137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.592+0000 7f58b9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b41137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42688/0 (socket says 172.21.15.1:42688) 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.592+0000 7f58b9c00700 1 -- 172.21.15.1:0/3346268793 learned_addr learned my addr 172.21.15.1:0/3346268793 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.593+0000 7f58b9c00700 1 -- 172.21.15.1:0/3346268793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a80094f0 con 0x7f58b4078cc0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.593+0000 7f58b9c00700 1 --2- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b41137e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f58a8006d40 tx=0x7f58a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.593+0000 7f58b1600700 1 -- 172.21.15.1:0/3346268793 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a8010840 con 0x7f58b4078cc0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.593+0000 7f58b1600700 1 -- 172.21.15.1:0/3346268793 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58a8010e70 con 0x7f58b4078cc0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.593+0000 7f58b1600700 1 -- 172.21.15.1:0/3346268793 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a8024e60 con 0x7f58b4078cc0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.593+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b4113f20 con 0x7f58b4078cc0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.593+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b4114bc0 con 0x7f58b4078cc0 2024-02-28T16:18:45.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.595+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b405f5c0 con 0x7f58b4078cc0 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.595+0000 7f58b1600700 1 -- 172.21.15.1:0/3346268793 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f58a80109a0 con 0x7f58b4078cc0 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.595+0000 7f58b1600700 1 --2- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f589c036790 0x7f589c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.595+0000 7f58b1600700 1 -- 172.21.15.1:0/3346268793 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f58a804a750 con 0x7f58b4078cc0 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.596+0000 7f58b9200700 1 --2- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f589c036790 0x7f589c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.596+0000 7f58b9200700 1 --2- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f589c036790 0x7f589c038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009990 tx=0x7f58a0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.601+0000 7f58b1600700 1 -- 172.21.15.1:0/3346268793 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f58a802c520 con 0x7f58b4078cc0 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.814+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f58b404bed0 con 0x7f58b4078cc0 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.819+0000 7f58b1600700 1 -- 172.21.15.1:0/3346268793 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f58a802c340 con 0x7f58b4078cc0 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.822+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f589c036790 msgr2=0x7f589c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.822+0000 7f58bc600700 1 --2- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f589c036790 0x7f589c038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009990 tx=0x7f58a0006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.822+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 msgr2=0x7f58b41137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.822+0000 7f58bc600700 1 --2- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b41137e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f58a8006d40 tx=0x7f58a8009250 comp rx=0 tx=0).stop 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.822+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 shutdown_connections 2024-02-28T16:18:45.502 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.822+0000 7f58bc600700 1 --2- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f589c036790 0x7f589c038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:45.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.822+0000 7f58bc600700 1 --2- 172.21.15.1:0/3346268793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58b4078cc0 0x7f58b41137e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:45.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.822+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 >> 172.21.15.1:0/3346268793 conn(0x7f58b40fdf00 msgr2=0x7f58b40ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:45.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.823+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 shutdown_connections 2024-02-28T16:18:45.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:44.823+0000 7f58bc600700 1 -- 172.21.15.1:0/3346268793 wait complete. 2024-02-28T16:18:45.503 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling the dashboard module... 2024-02-28T16:18:46.078 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:45 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3346268793' entity='client.admin' 2024-02-28T16:18:46.078 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:45 smithi001 ceph-mon[28991]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:18:47.171 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:46 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1433541017' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-28T16:18:47.471 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.086+0000 7fe951e00700 1 Processor -- start 2024-02-28T16:18:47.471 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.086+0000 7fe951e00700 1 -- start start 2024-02-28T16:18:47.471 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.087+0000 7fe951e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:47.471 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.087+0000 7fe951e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c1030c0 con 0x7fe94c102770 2024-02-28T16:18:47.471 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.088+0000 7fe94aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.088+0000 7fe94aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42694/0 (socket says 172.21.15.1:42694) 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.088+0000 7fe94aa00700 1 -- 172.21.15.1:0/1190458808 learned_addr learned my addr 172.21.15.1:0/1190458808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.088+0000 7fe94aa00700 1 -- 172.21.15.1:0/1190458808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c103200 con 0x7fe94c102770 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.088+0000 7fe94aa00700 1 --2- 172.21.15.1:0/1190458808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c102b80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0098e0 tx=0x7fe93c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf4c6caac7dd295 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.089+0000 7fe949600700 1 -- 172.21.15.1:0/1190458808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe93c00b860 con 0x7fe94c102770 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.089+0000 7fe949600700 1 -- 172.21.15.1:0/1190458808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe93c00be90 con 0x7fe94c102770 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.089+0000 7fe949600700 1 -- 172.21.15.1:0/1190458808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe93c0103c0 con 0x7fe94c102770 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.090+0000 7fe951e00700 1 -- 172.21.15.1:0/1190458808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 msgr2=0x7fe94c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.090+0000 7fe951e00700 1 --2- 172.21.15.1:0/1190458808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c102b80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0098e0 tx=0x7fe93c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.090+0000 7fe951e00700 1 -- 172.21.15.1:0/1190458808 shutdown_connections 2024-02-28T16:18:47.472 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.090+0000 7fe951e00700 1 --2- 172.21.15.1:0/1190458808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c102b80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.090+0000 7fe951e00700 1 -- 172.21.15.1:0/1190458808 >> 172.21.15.1:0/1190458808 conn(0x7fe94c0fded0 msgr2=0x7fe94c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.090+0000 7fe951e00700 1 -- 172.21.15.1:0/1190458808 shutdown_connections 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.090+0000 7fe951e00700 1 -- 172.21.15.1:0/1190458808 wait complete. 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.091+0000 7fe951e00700 1 Processor -- start 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.091+0000 7fe951e00700 1 -- start start 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.092+0000 7fe951e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.092+0000 7fe951e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c113fc0 con 0x7fe94c102770 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.092+0000 7fe94aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.092+0000 7fe94aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42706/0 (socket says 172.21.15.1:42706) 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.092+0000 7fe94aa00700 1 -- 172.21.15.1:0/1433541017 learned_addr learned my addr 172.21.15.1:0/1433541017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.092+0000 7fe94aa00700 1 -- 172.21.15.1:0/1433541017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0094f0 con 0x7fe94c102770 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.093+0000 7fe94aa00700 1 --2- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c113a80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe93c006b00 tx=0x7fe93c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:47.473 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.093+0000 7fe942a00700 1 -- 172.21.15.1:0/1433541017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe93c0109a0 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.093+0000 7fe942a00700 1 -- 172.21.15.1:0/1433541017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe93c01a430 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.093+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c1141c0 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.093+0000 7fe942a00700 1 -- 172.21.15.1:0/1433541017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe93c0245b0 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.093+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c114e60 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.094+0000 7fe942a00700 1 -- 172.21.15.1:0/1433541017 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe93c01aaa0 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.095+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c05f5c0 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.095+0000 7fe942a00700 1 --2- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fe930036830 0x7fe930038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.095+0000 7fe942a00700 1 -- 172.21.15.1:0/1433541017 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe93c04a740 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.095+0000 7fe94a000700 1 --2- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fe930036830 0x7fe930038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.096+0000 7fe94a000700 1 --2- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fe930036830 0x7fe930038ce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe938009990 tx=0x7fe938006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.100+0000 7fe942a00700 1 -- 172.21.15.1:0/1433541017 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe93c018940 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.377+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe94c04bed0 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.827+0000 7fe942a00700 1 -- 172.21.15.1:0/1433541017 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fe93c010b00 con 0x7fe94c102770 2024-02-28T16:18:47.474 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.831+0000 7fe942a00700 1 -- 172.21.15.1:0/1433541017 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe93c019780 con 0x7fe94c102770 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fe930036830 msgr2=0x7fe930038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 --2- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fe930036830 0x7fe930038ce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe938009990 tx=0x7fe938006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 msgr2=0x7fe94c113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 --2- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c113a80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe93c006b00 tx=0x7fe93c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 shutdown_connections 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 --2- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7fe930036830 0x7fe930038ce0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 --2- 172.21.15.1:0/1433541017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c102770 0x7fe94c113a80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 >> 172.21.15.1:0/1433541017 conn(0x7fe94c0fded0 msgr2=0x7fe94c0fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 shutdown_connections 2024-02-28T16:18:47.475 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:46.835+0000 7fe951e00700 1 -- 172.21.15.1:0/1433541017 wait complete. 2024-02-28T16:18:48.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:47 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1433541017' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-28T16:18:48.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:47 smithi001 ceph-mon[28991]: mgrmap e9: smithi001.rmthtx(active, since 31s) 2024-02-28T16:18:49.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:48 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/789957610' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-28T16:18:49.148 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "smithi001.rmthtx", 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.151+0000 7f7895400700 1 Processor -- start 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.151+0000 7f7895400700 1 -- start start 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.152+0000 7f7895400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f7890102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.152+0000 7f7895400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78901030c0 con 0x7f7890102770 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.152+0000 7f788e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f7890102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:49.153 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.152+0000 7f788e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f7890102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42730/0 (socket says 172.21.15.1:42730) 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.152+0000 7f788e800700 1 -- 172.21.15.1:0/1795886532 learned_addr learned my addr 172.21.15.1:0/1795886532 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.153+0000 7f788e800700 1 -- 172.21.15.1:0/1795886532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7890103200 con 0x7f7890102770 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.153+0000 7f788e800700 1 --2- 172.21.15.1:0/1795886532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f7890102b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78800098e0 tx=0x7f7880009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6961f7c3981c5d5a server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.153+0000 7f788d400700 1 -- 172.21.15.1:0/1795886532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f788000b860 con 0x7f7890102770 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.153+0000 7f788d400700 1 -- 172.21.15.1:0/1795886532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f788000be90 con 0x7f7890102770 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.153+0000 7f788d400700 1 -- 172.21.15.1:0/1795886532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78800103c0 con 0x7f7890102770 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.154+0000 7f7895400700 1 -- 172.21.15.1:0/1795886532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 msgr2=0x7f7890102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.154+0000 7f7895400700 1 --2- 172.21.15.1:0/1795886532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f7890102b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78800098e0 tx=0x7f7880009250 comp rx=0 tx=0).stop 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.154+0000 7f7895400700 1 -- 172.21.15.1:0/1795886532 shutdown_connections 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.154+0000 7f7895400700 1 --2- 172.21.15.1:0/1795886532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f7890102b80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.154+0000 7f7895400700 1 -- 172.21.15.1:0/1795886532 >> 172.21.15.1:0/1795886532 conn(0x7f78900fded0 msgr2=0x7f7890100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.154+0000 7f7895400700 1 -- 172.21.15.1:0/1795886532 shutdown_connections 2024-02-28T16:18:49.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.154+0000 7f7895400700 1 -- 172.21.15.1:0/1795886532 wait complete. 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.155+0000 7f7895400700 1 Processor -- start 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.155+0000 7f7895400700 1 -- start start 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.155+0000 7f7895400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f78901072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.155+0000 7f7895400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7890107800 con 0x7f7890102770 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.155+0000 7f788e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f78901072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.155+0000 7f788e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f78901072c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42738/0 (socket says 172.21.15.1:42738) 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.155+0000 7f788e800700 1 -- 172.21.15.1:0/789957610 learned_addr learned my addr 172.21.15.1:0/789957610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.156+0000 7f788e800700 1 -- 172.21.15.1:0/789957610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78800094f0 con 0x7f7890102770 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.156+0000 7f788e800700 1 --2- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f78901072c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7880006b00 tx=0x7f7880009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:49.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.156+0000 7f7886a00700 1 -- 172.21.15.1:0/789957610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78800109a0 con 0x7f7890102770 2024-02-28T16:18:49.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.156+0000 7f7886a00700 1 -- 172.21.15.1:0/789957610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f788001a430 con 0x7f7890102770 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.156+0000 7f7886a00700 1 -- 172.21.15.1:0/789957610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78800245b0 con 0x7f7890102770 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.156+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7890107a00 con 0x7f7890102770 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.157+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7890107e90 con 0x7f7890102770 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.157+0000 7f7886a00700 1 -- 172.21.15.1:0/789957610 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f7880010b00 con 0x7f7890102770 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.158+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789004bed0 con 0x7f7890102770 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.158+0000 7f7886a00700 1 --2- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f7874036830 0x7f7874038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.158+0000 7f7886a00700 1 -- 172.21.15.1:0/789957610 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f788004a780 con 0x7f7890102770 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.158+0000 7f788de00700 1 -- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f7874036830 msgr2=0x7f7874038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3320028482 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.158+0000 7f788de00700 1 --2- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f7874036830 0x7f7874038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.163+0000 7f7886a00700 1 -- 172.21.15.1:0/789957610 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7880018b40 con 0x7f7890102770 2024-02-28T16:18:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.359+0000 7f788de00700 1 -- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f7874036830 msgr2=0x7f7874038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3320028482 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.359+0000 7f788de00700 1 --2- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f7874036830 0x7f7874038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.426+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f78901086f0 con 0x7f7890102770 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.427+0000 7f7886a00700 1 -- 172.21.15.1:0/789957610 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f7880018b40 con 0x7f7890102770 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f7874036830 msgr2=0x7f7874038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 --2- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f7874036830 0x7f7874038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 msgr2=0x7f78901072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 --2- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f78901072c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7880006b00 tx=0x7f7880009250 comp rx=0 tx=0).stop 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 shutdown_connections 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 --2- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f7874036830 0x7f7874038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 --2- 172.21.15.1:0/789957610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7890102770 0x7f78901072c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 >> 172.21.15.1:0/789957610 conn(0x7f78900fded0 msgr2=0x7f78900fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.430+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 shutdown_connections 2024-02-28T16:18:49.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:48.431+0000 7f7895400700 1 -- 172.21.15.1:0/789957610 wait complete. 2024-02-28T16:18:49.161 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-02-28T16:18:49.161 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 9... 2024-02-28T16:18:56.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: Active manager daemon smithi001.rmthtx restarted 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: Activating manager daemon smithi001.rmthtx 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: osdmap e3: 0 total, 0 up, 0 in 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: mgrmap e10: smithi001.rmthtx(active, starting, since 0.0040315s) 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: Manager daemon smithi001.rmthtx is now available 2024-02-28T16:18:56.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:18:56.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:55 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:18:56.847 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:56 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:18:56.847 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:56 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/trash_purge_schedule"}]: dispatch 2024-02-28T16:18:56.847 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:56 smithi001 ceph-mon[28991]: mgrmap e11: smithi001.rmthtx(active, since 1.00924s) 2024-02-28T16:18:57.608 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-02-28T16:18:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-28T16:18:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-28T16:18:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-02-28T16:18:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.812+0000 7f8e81000700 1 Processor -- start 2024-02-28T16:18:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.812+0000 7f8e81000700 1 -- start start 2024-02-28T16:18:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.813+0000 7f8e81000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.813+0000 7f8e81000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c1030c0 con 0x7f8e7c102770 2024-02-28T16:18:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.813+0000 7f8e79e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.813+0000 7f8e79e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42742/0 (socket says 172.21.15.1:42742) 2024-02-28T16:18:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.813+0000 7f8e79e00700 1 -- 172.21.15.1:0/1836669112 learned_addr learned my addr 172.21.15.1:0/1836669112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.814+0000 7f8e79e00700 1 -- 172.21.15.1:0/1836669112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c103200 con 0x7f8e7c102770 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.814+0000 7f8e79e00700 1 --2- 172.21.15.1:0/1836669112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c102b80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0098e0 tx=0x7f8e6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a3726278c9df0f server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.814+0000 7f8e78a00700 1 -- 172.21.15.1:0/1836669112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6c00b860 con 0x7f8e7c102770 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.814+0000 7f8e78a00700 1 -- 172.21.15.1:0/1836669112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e6c00be90 con 0x7f8e7c102770 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.814+0000 7f8e78a00700 1 -- 172.21.15.1:0/1836669112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6c0103c0 con 0x7f8e7c102770 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.815+0000 7f8e81000700 1 -- 172.21.15.1:0/1836669112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 msgr2=0x7f8e7c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.815+0000 7f8e81000700 1 --2- 172.21.15.1:0/1836669112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c102b80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0098e0 tx=0x7f8e6c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.815+0000 7f8e81000700 1 -- 172.21.15.1:0/1836669112 shutdown_connections 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.815+0000 7f8e81000700 1 --2- 172.21.15.1:0/1836669112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c102b80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.815+0000 7f8e81000700 1 -- 172.21.15.1:0/1836669112 >> 172.21.15.1:0/1836669112 conn(0x7f8e7c0fded0 msgr2=0x7f8e7c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.816+0000 7f8e81000700 1 -- 172.21.15.1:0/1836669112 shutdown_connections 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.816+0000 7f8e81000700 1 -- 172.21.15.1:0/1836669112 wait complete. 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.817+0000 7f8e81000700 1 Processor -- start 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.817+0000 7f8e81000700 1 -- start start 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.817+0000 7f8e81000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c111860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.817+0000 7f8e81000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c111da0 con 0x7f8e7c102770 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.818+0000 7f8e79e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c111860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.818+0000 7f8e79e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c111860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42746/0 (socket says 172.21.15.1:42746) 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.818+0000 7f8e79e00700 1 -- 172.21.15.1:0/2587231050 learned_addr learned my addr 172.21.15.1:0/2587231050 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.818+0000 7f8e79e00700 1 -- 172.21.15.1:0/2587231050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c0094f0 con 0x7f8e7c102770 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.818+0000 7f8e79e00700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c111860 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c006b00 tx=0x7f8e6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.819+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6c0109a0 con 0x7f8e7c102770 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.819+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e6c01a430 con 0x7f8e7c102770 2024-02-28T16:18:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.819+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7c111fa0 con 0x7f8e7c102770 2024-02-28T16:18:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.819+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6c0245b0 con 0x7f8e7c102770 2024-02-28T16:18:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.819+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7c112c40 con 0x7f8e7c102770 2024-02-28T16:18:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.820+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f8e6c01aaa0 con 0x7f8e7c102770 2024-02-28T16:18:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.820+0000 7f8e72000700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 0x7f8e60038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.820+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 --> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e600393f0 con 0x7f8e60036830 2024-02-28T16:18:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.821+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8e6c04a6f0 con 0x7f8e7c102770 2024-02-28T16:18:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.821+0000 7f8e79400700 1 -- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 msgr2=0x7f8e60038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3320028482 2024-02-28T16:18:57.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:49.821+0000 7f8e79400700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 0x7f8e60038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:50.022+0000 7f8e79400700 1 -- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 msgr2=0x7f8e60038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3320028482 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:50.022+0000 7f8e79400700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 0x7f8e60038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:50.423+0000 7f8e79400700 1 -- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 msgr2=0x7f8e60038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3320028482 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:50.423+0000 7f8e79400700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 0x7f8e60038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:51.224+0000 7f8e79400700 1 -- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 msgr2=0x7f8e60038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3320028482 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:51.224+0000 7f8e79400700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 0x7f8e60038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:52.826+0000 7f8e79400700 1 -- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 msgr2=0x7f8e60038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3320028482 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:52.826+0000 7f8e79400700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 0x7f8e60038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:55.596+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7f8e6c010b00 con 0x7f8e7c102770 2024-02-28T16:18:57.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:55.596+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 msgr2=0x7f8e60038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:55.596+0000 7f8e72000700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 0x7f8e60038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.599+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f8e6c04bfa0 con 0x7f8e7c102770 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.599+0000 7f8e72000700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8e6003a070 0x7f8e6003c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.599+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 --> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e600393f0 con 0x7f8e6003a070 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.600+0000 7f8e79400700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8e6003a070 0x7f8e6003c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.602+0000 7f8e79400700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8e6003a070 0x7f8e6003c450 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8e680039d0 tx=0x7f8e68009350 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.603+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mgr.14162 v2:172.21.15.1:6800/701199259 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f8e600393f0 con 0x7f8e6003a070 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.609+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 --> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8e7c112430 con 0x7f8e6003a070 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.609+0000 7f8e72000700 1 -- 172.21.15.1:0/2587231050 <== mgr.14162 v2:172.21.15.1:6800/701199259 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8e7c112430 con 0x7f8e6003a070 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8e6003a070 msgr2=0x7f8e6003c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8e6003a070 0x7f8e6003c450 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8e680039d0 tx=0x7f8e68009350 comp rx=0 tx=0).stop 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 msgr2=0x7f8e7c111860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c111860 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c006b00 tx=0x7f8e6c009250 comp rx=0 tx=0).stop 2024-02-28T16:18:57.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 shutdown_connections 2024-02-28T16:18:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8e6003a070 0x7f8e6003c450 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7c102770 0x7f8e7c111860 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 --2- 172.21.15.1:0/2587231050 >> [v2:172.21.15.1:6800/3320028482,v1:172.21.15.1:6801/3320028482] conn(0x7f8e60036830 0x7f8e60038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 >> 172.21.15.1:0/2587231050 conn(0x7f8e7c0fded0 msgr2=0x7f8e7c0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 shutdown_connections 2024-02-28T16:18:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:56.610+0000 7f8e81000700 1 -- 172.21.15.1:0/2587231050 wait complete. 2024-02-28T16:18:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 9 is available 2024-02-28T16:18:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:Generating a dashboard self-signed certificate... 2024-02-28T16:18:58.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:57 smithi001 ceph-mon[28991]: [28/Feb/2024:16:18:56] ENGINE Bus STARTING 2024-02-28T16:18:58.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:57 smithi001 ceph-mon[28991]: [28/Feb/2024:16:18:56] ENGINE Serving on https://172.21.15.1:7150 2024-02-28T16:18:58.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:57 smithi001 ceph-mon[28991]: [28/Feb/2024:16:18:56] ENGINE Bus STARTED 2024-02-28T16:18:58.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:57 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:59.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:58 smithi001 ceph-mon[28991]: mgrmap e12: smithi001.rmthtx(active, since 2s) 2024-02-28T16:18:59.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:58 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.556+0000 7ffb8c400700 1 Processor -- start 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.556+0000 7ffb8c400700 1 -- start start 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.556+0000 7ffb8c400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841077c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.556+0000 7ffb8c400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb841031b0 con 0x7ffb84071440 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.557+0000 7ffb89a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.557+0000 7ffb89a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56240/0 (socket says 172.21.15.1:56240) 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.557+0000 7ffb89a00700 1 -- 172.21.15.1:0/2014436735 learned_addr learned my addr 172.21.15.1:0/2014436735 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:59.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.557+0000 7ffb89a00700 1 -- 172.21.15.1:0/2014436735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb84107d00 con 0x7ffb84071440 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.558+0000 7ffb89a00700 1 --2- 172.21.15.1:0/2014436735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841077c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb780098e0 tx=0x7ffb78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b9430db6dde0e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.558+0000 7ffb83e00700 1 -- 172.21.15.1:0/2014436735 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb7800b860 con 0x7ffb84071440 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.558+0000 7ffb83e00700 1 -- 172.21.15.1:0/2014436735 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb7800be90 con 0x7ffb84071440 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.558+0000 7ffb83e00700 1 -- 172.21.15.1:0/2014436735 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb780103c0 con 0x7ffb84071440 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.559+0000 7ffb8c400700 1 -- 172.21.15.1:0/2014436735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 msgr2=0x7ffb841077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.559+0000 7ffb8c400700 1 --2- 172.21.15.1:0/2014436735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841077c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb780098e0 tx=0x7ffb78009250 comp rx=0 tx=0).stop 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.559+0000 7ffb8c400700 1 -- 172.21.15.1:0/2014436735 shutdown_connections 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.559+0000 7ffb8c400700 1 --2- 172.21.15.1:0/2014436735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841077c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.559+0000 7ffb8c400700 1 -- 172.21.15.1:0/2014436735 >> 172.21.15.1:0/2014436735 conn(0x7ffb840fcdd0 msgr2=0x7ffb840ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.559+0000 7ffb8c400700 1 -- 172.21.15.1:0/2014436735 shutdown_connections 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.559+0000 7ffb8c400700 1 -- 172.21.15.1:0/2014436735 wait complete. 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.560+0000 7ffb8c400700 1 Processor -- start 2024-02-28T16:18:59.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.560+0000 7ffb8c400700 1 -- start start 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.560+0000 7ffb8c400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841137a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.560+0000 7ffb8c400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb84113ce0 con 0x7ffb84071440 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.561+0000 7ffb89a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.561+0000 7ffb89a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841137a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56244/0 (socket says 172.21.15.1:56244) 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.561+0000 7ffb89a00700 1 -- 172.21.15.1:0/2697307008 learned_addr learned my addr 172.21.15.1:0/2697307008 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.561+0000 7ffb89a00700 1 -- 172.21.15.1:0/2697307008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb780094f0 con 0x7ffb84071440 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.561+0000 7ffb89a00700 1 --2- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841137a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb780094c0 tx=0x7ffb78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.562+0000 7ffb81600700 1 -- 172.21.15.1:0/2697307008 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb78010890 con 0x7ffb84071440 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.562+0000 7ffb81600700 1 -- 172.21.15.1:0/2697307008 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb78010ec0 con 0x7ffb84071440 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.562+0000 7ffb81600700 1 -- 172.21.15.1:0/2697307008 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb780243f0 con 0x7ffb84071440 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.562+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb84113ee0 con 0x7ffb84071440 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.562+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb84114b80 con 0x7ffb84071440 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.563+0000 7ffb81600700 1 -- 172.21.15.1:0/2697307008 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7ffb7801a7e0 con 0x7ffb84071440 2024-02-28T16:18:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.564+0000 7ffb81600700 1 --2- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ffb6c036830 0x7ffb6c038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.564+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8405f5c0 con 0x7ffb84071440 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.564+0000 7ffb81600700 1 -- 172.21.15.1:0/2697307008 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ffb7804a470 con 0x7ffb84071440 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.564+0000 7ffb89000700 1 --2- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ffb6c036830 0x7ffb6c038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.565+0000 7ffb89000700 1 --2- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ffb6c036830 0x7ffb6c038ce0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009990 tx=0x7ffb70006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.574+0000 7ffb81600700 1 -- 172.21.15.1:0/2697307008 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ffb7802b8e0 con 0x7ffb84071440 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.797+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 --> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb840ff0c0 con 0x7ffb6c036830 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.964+0000 7ffb81600700 1 -- 172.21.15.1:0/2697307008 <== mgr.14162 v2:172.21.15.1:6800/701199259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7ffb840ff0c0 con 0x7ffb6c036830 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ffb6c036830 msgr2=0x7ffb6c038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 --2- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ffb6c036830 0x7ffb6c038ce0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009990 tx=0x7ffb70006e40 comp rx=0 tx=0).stop 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 msgr2=0x7ffb841137a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 --2- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841137a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb780094c0 tx=0x7ffb78009250 comp rx=0 tx=0).stop 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 shutdown_connections 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 --2- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ffb6c036830 0x7ffb6c038ce0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 --2- 172.21.15.1:0/2697307008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb84071440 0x7ffb841137a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:18:59.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 >> 172.21.15.1:0/2697307008 conn(0x7ffb840fcdd0 msgr2=0x7ffb840fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:18:59.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 shutdown_connections 2024-02-28T16:18:59.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:18:58.967+0000 7ffb8c400700 1 -- 172.21.15.1:0/2697307008 wait complete. 2024-02-28T16:18:59.619 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial admin user... 2024-02-28T16:19:00.214 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:59 smithi001 ceph-mon[28991]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:19:00.214 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:59 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:00.214 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:59 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:00.214 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:18:59 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:02.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:01 smithi001 ceph-mon[28991]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:19:02.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:01 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:02.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Bl436hp5Cp9APCekARqpjefrESbjo/Ptx2tXyVxhF/YMaoJrhDnNi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709137140, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-28T16:19:02.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.224+0000 7fa91b400700 1 Processor -- start 2024-02-28T16:19:02.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.224+0000 7fa91b400700 1 -- start start 2024-02-28T16:19:02.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.224+0000 7fa91b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa9140760c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:02.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.224+0000 7fa91b400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914077fc0 con 0x7fa914077bb0 2024-02-28T16:19:02.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.225+0000 7fa918a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa9140760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:02.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.225+0000 7fa918a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa9140760c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56248/0 (socket says 172.21.15.1:56248) 2024-02-28T16:19:02.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.225+0000 7fa918a00700 1 -- 172.21.15.1:0/3198888969 learned_addr learned my addr 172.21.15.1:0/3198888969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:02.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.225+0000 7fa918a00700 1 -- 172.21.15.1:0/3198888969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa914076600 con 0x7fa914077bb0 2024-02-28T16:19:02.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.226+0000 7fa918a00700 1 --2- 172.21.15.1:0/3198888969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa9140760c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9080098e0 tx=0x7fa908009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6af1745c78802c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:02.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.226+0000 7fa913400700 1 -- 172.21.15.1:0/3198888969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa90800b860 con 0x7fa914077bb0 2024-02-28T16:19:02.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.226+0000 7fa913400700 1 -- 172.21.15.1:0/3198888969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa90800be90 con 0x7fa914077bb0 2024-02-28T16:19:02.108 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.226+0000 7fa913400700 1 -- 172.21.15.1:0/3198888969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9080103c0 con 0x7fa914077bb0 2024-02-28T16:19:02.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.227+0000 7fa91b400700 1 -- 172.21.15.1:0/3198888969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 msgr2=0x7fa9140760c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:02.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.227+0000 7fa91b400700 1 --2- 172.21.15.1:0/3198888969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa9140760c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9080098e0 tx=0x7fa908009250 comp rx=0 tx=0).stop 2024-02-28T16:19:02.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.227+0000 7fa91b400700 1 -- 172.21.15.1:0/3198888969 shutdown_connections 2024-02-28T16:19:02.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.227+0000 7fa91b400700 1 --2- 172.21.15.1:0/3198888969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa9140760c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:02.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.227+0000 7fa91b400700 1 -- 172.21.15.1:0/3198888969 >> 172.21.15.1:0/3198888969 conn(0x7fa9140fded0 msgr2=0x7fa914100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:02.110 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.227+0000 7fa91b400700 1 -- 172.21.15.1:0/3198888969 shutdown_connections 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.228+0000 7fa91b400700 1 -- 172.21.15.1:0/3198888969 wait complete. 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.228+0000 7fa91b400700 1 Processor -- start 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.229+0000 7fa91b400700 1 -- start start 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.229+0000 7fa91b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa914113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.229+0000 7fa91b400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914113fe0 con 0x7fa914077bb0 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.229+0000 7fa918a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa914113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.229+0000 7fa918a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa914113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56262/0 (socket says 172.21.15.1:56262) 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.229+0000 7fa918a00700 1 -- 172.21.15.1:0/1617705503 learned_addr learned my addr 172.21.15.1:0/1617705503 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.230+0000 7fa918a00700 1 -- 172.21.15.1:0/1617705503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9080094f0 con 0x7fa914077bb0 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.230+0000 7fa918a00700 1 --2- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa914113aa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9080090b0 tx=0x7fa908009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.230+0000 7fa910c00700 1 -- 172.21.15.1:0/1617705503 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9080118f0 con 0x7fa914077bb0 2024-02-28T16:19:02.111 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.230+0000 7fa910c00700 1 -- 172.21.15.1:0/1617705503 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa908010d10 con 0x7fa914077bb0 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.230+0000 7fa910c00700 1 -- 172.21.15.1:0/1617705503 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa908024e70 con 0x7fa914077bb0 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.230+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9141141e0 con 0x7fa914077bb0 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.230+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa914114e80 con 0x7fa914077bb0 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.231+0000 7fa910c00700 1 -- 172.21.15.1:0/1617705503 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa908024440 con 0x7fa914077bb0 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.232+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91405f5c0 con 0x7fa914077bb0 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.232+0000 7fa910c00700 1 --2- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fa8fc0367e0 0x7fa8fc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.232+0000 7fa910c00700 1 -- 172.21.15.1:0/1617705503 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa90804a660 con 0x7fa914077bb0 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.232+0000 7fa913e00700 1 --2- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fa8fc0367e0 0x7fa8fc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:02.114 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.233+0000 7fa913e00700 1 --2- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fa8fc0367e0 0x7fa8fc038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa900009990 tx=0x7fa900006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.238+0000 7fa910c00700 1 -- 172.21.15.1:0/1617705503 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa908010e80 con 0x7fa914077bb0 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.476+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 --> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa91405f3e0 con 0x7fa8fc0367e0 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.746+0000 7fa910c00700 1 -- 172.21.15.1:0/1617705503 <== mgr.14162 v2:172.21.15.1:6800/701199259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fa91405f3e0 con 0x7fa8fc0367e0 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.749+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fa8fc0367e0 msgr2=0x7fa8fc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.749+0000 7fa91b400700 1 --2- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fa8fc0367e0 0x7fa8fc038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa900009990 tx=0x7fa900006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.749+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 msgr2=0x7fa914113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.749+0000 7fa91b400700 1 --2- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa914113aa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9080090b0 tx=0x7fa908009250 comp rx=0 tx=0).stop 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.749+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 shutdown_connections 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.749+0000 7fa91b400700 1 --2- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fa8fc0367e0 0x7fa8fc038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.749+0000 7fa91b400700 1 --2- 172.21.15.1:0/1617705503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa914077bb0 0x7fa914113aa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.749+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 >> 172.21.15.1:0/1617705503 conn(0x7fa9140fded0 msgr2=0x7fa9141001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.750+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 shutdown_connections 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:00.750+0000 7fa91b400700 1 -- 172.21.15.1:0/1617705503 wait complete. 2024-02-28T16:19:02.115 INFO:teuthology.orchestra.run.smithi001.stdout:Fetching dashboard port number... 2024-02-28T16:19:03.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:02 smithi001 ceph-mon[28991]: mgrmap e13: smithi001.rmthtx(active, since 6s) 2024-02-28T16:19:03.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 8443 2024-02-28T16:19:03.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.717+0000 7fb18ae00700 1 Processor -- start 2024-02-28T16:19:03.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.717+0000 7fb18ae00700 1 -- start start 2024-02-28T16:19:03.673 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.717+0000 7fb18ae00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1840771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.717+0000 7fb18ae00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1840790f0 con 0x7fb184078ce0 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.718+0000 7fb189a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1840771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.718+0000 7fb189a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1840771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35296/0 (socket says 172.21.15.1:35296) 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.718+0000 7fb189a00700 1 -- 172.21.15.1:0/773444803 learned_addr learned my addr 172.21.15.1:0/773444803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.719+0000 7fb189a00700 1 -- 172.21.15.1:0/773444803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb184077730 con 0x7fb184078ce0 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.719+0000 7fb189a00700 1 --2- 172.21.15.1:0/773444803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1840771f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb1780098e0 tx=0x7fb178009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7fc061d26333f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.719+0000 7fb183e00700 1 -- 172.21.15.1:0/773444803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb17800b860 con 0x7fb184078ce0 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.719+0000 7fb183e00700 1 -- 172.21.15.1:0/773444803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb17800be90 con 0x7fb184078ce0 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.719+0000 7fb183e00700 1 -- 172.21.15.1:0/773444803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1780103c0 con 0x7fb184078ce0 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.720+0000 7fb18ae00700 1 -- 172.21.15.1:0/773444803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 msgr2=0x7fb1840771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:03.675 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.720+0000 7fb18ae00700 1 --2- 172.21.15.1:0/773444803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1840771f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb1780098e0 tx=0x7fb178009250 comp rx=0 tx=0).stop 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.720+0000 7fb18ae00700 1 -- 172.21.15.1:0/773444803 shutdown_connections 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.720+0000 7fb18ae00700 1 --2- 172.21.15.1:0/773444803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1840771f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.720+0000 7fb18ae00700 1 -- 172.21.15.1:0/773444803 >> 172.21.15.1:0/773444803 conn(0x7fb1840fdd20 msgr2=0x7fb184100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.720+0000 7fb18ae00700 1 -- 172.21.15.1:0/773444803 shutdown_connections 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.720+0000 7fb18ae00700 1 -- 172.21.15.1:0/773444803 wait complete. 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.721+0000 7fb18ae00700 1 Processor -- start 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.721+0000 7fb18ae00700 1 -- start start 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.721+0000 7fb18ae00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1841135d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:03.679 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.722+0000 7fb18ae00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb184113b10 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.722+0000 7fb189a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1841135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.722+0000 7fb189a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1841135d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35312/0 (socket says 172.21.15.1:35312) 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.722+0000 7fb189a00700 1 -- 172.21.15.1:0/128158311 learned_addr learned my addr 172.21.15.1:0/128158311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.722+0000 7fb189a00700 1 -- 172.21.15.1:0/128158311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1780094f0 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.722+0000 7fb189a00700 1 --2- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1841135d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb1780090b0 tx=0x7fb178009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.723+0000 7fb181600700 1 -- 172.21.15.1:0/128158311 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1780118f0 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.723+0000 7fb181600700 1 -- 172.21.15.1:0/128158311 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb178010d10 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.723+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb184113d10 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.723+0000 7fb181600700 1 -- 172.21.15.1:0/128158311 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb178024e70 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.723+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1841149b0 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.724+0000 7fb181600700 1 -- 172.21.15.1:0/128158311 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fb178024440 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.724+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18405f5c0 con 0x7fb184078ce0 2024-02-28T16:19:03.680 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.724+0000 7fb181600700 1 --2- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fb16c0368d0 0x7fb16c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.725+0000 7fb181600700 1 -- 172.21.15.1:0/128158311 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb17804a770 con 0x7fb184078ce0 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.725+0000 7fb189000700 1 --2- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fb16c0368d0 0x7fb16c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.726+0000 7fb189000700 1 --2- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fb16c0368d0 0x7fb16c038d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb170009990 tx=0x7fb170006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.730+0000 7fb181600700 1 -- 172.21.15.1:0/128158311 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb178010e80 con 0x7fb184078ce0 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.941+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb18404bed0 con 0x7fb184078ce0 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.942+0000 7fb181600700 1 -- 172.21.15.1:0/128158311 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb178022340 con 0x7fb184078ce0 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fb16c0368d0 msgr2=0x7fb16c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 --2- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fb16c0368d0 0x7fb16c038d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb170009990 tx=0x7fb170006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 msgr2=0x7fb1841135d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:03.681 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 --2- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1841135d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb1780090b0 tx=0x7fb178009250 comp rx=0 tx=0).stop 2024-02-28T16:19:03.682 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 shutdown_connections 2024-02-28T16:19:03.682 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 --2- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fb16c0368d0 0x7fb16c038d80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:03.682 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 --2- 172.21.15.1:0/128158311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb184078ce0 0x7fb1841135d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:03.682 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 >> 172.21.15.1:0/128158311 conn(0x7fb1840fdd20 msgr2=0x7fb1840ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:03.682 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.945+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 shutdown_connections 2024-02-28T16:19:03.682 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:02.946+0000 7fb18ae00700 1 -- 172.21.15.1:0/128158311 wait complete. 2024-02-28T16:19:03.710 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-28T16:19:03.710 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-02-28T16:19:03.734 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-28T16:19:03.734 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-02-28T16:19:03.734 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-02-28T16:19:03.734 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-28T16:19:03.739 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph Dashboard is now available at: 2024-02-28T16:19:03.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:03.739 INFO:teuthology.orchestra.run.smithi001.stdout: URL: https://smithi001.front.sepia.ceph.com:8443/ 2024-02-28T16:19:03.739 INFO:teuthology.orchestra.run.smithi001.stdout: User: admin 2024-02-28T16:19:03.739 INFO:teuthology.orchestra.run.smithi001.stdout: Password: ovybq5pie6 2024-02-28T16:19:03.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:03.739 INFO:teuthology.orchestra.run.smithi001.stdout:Saving cluster configuration to /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config directory 2024-02-28T16:19:03.740 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling autotune for osd_memory_target 2024-02-28T16:19:03.920 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:03 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/128158311' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-28T16:19:05.219 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.373+0000 7f6ed5a00700 1 Processor -- start 2024-02-28T16:19:05.219 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.374+0000 7f6ed5a00700 1 -- start start 2024-02-28T16:19:05.219 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.374+0000 7f6ed5a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed007ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:05.219 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.374+0000 7f6ed5a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed007d010 con 0x7f6ed007c630 2024-02-28T16:19:05.219 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.375+0000 7f6ecfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed007ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:05.219 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.375+0000 7f6ecfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed007ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35324/0 (socket says 172.21.15.1:35324) 2024-02-28T16:19:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.375+0000 7f6ecfe00700 1 -- 172.21.15.1:0/4228815305 learned_addr learned my addr 172.21.15.1:0/4228815305 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.375+0000 7f6ecfe00700 1 -- 172.21.15.1:0/4228815305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0109e10 con 0x7f6ed007c630 2024-02-28T16:19:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.376+0000 7f6ecfe00700 1 --2- 172.21.15.1:0/4228815305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed007ca40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00098e0 tx=0x7f6ec0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c679be63acfed7 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.376+0000 7f6ecea00700 1 -- 172.21.15.1:0/4228815305 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec000b860 con 0x7f6ed007c630 2024-02-28T16:19:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.376+0000 7f6ecea00700 1 -- 172.21.15.1:0/4228815305 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ec000be90 con 0x7f6ed007c630 2024-02-28T16:19:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.376+0000 7f6ecea00700 1 -- 172.21.15.1:0/4228815305 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec00103c0 con 0x7f6ed007c630 2024-02-28T16:19:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.377+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4228815305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 msgr2=0x7f6ed007ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.377+0000 7f6ed5a00700 1 --2- 172.21.15.1:0/4228815305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed007ca40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00098e0 tx=0x7f6ec0009250 comp rx=0 tx=0).stop 2024-02-28T16:19:05.222 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.377+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4228815305 shutdown_connections 2024-02-28T16:19:05.222 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.377+0000 7f6ed5a00700 1 --2- 172.21.15.1:0/4228815305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed007ca40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:05.222 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.377+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4228815305 >> 172.21.15.1:0/4228815305 conn(0x7f6ed0077dd0 msgr2=0x7f6ed007a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:05.222 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.377+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4228815305 shutdown_connections 2024-02-28T16:19:05.222 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.377+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4228815305 wait complete. 2024-02-28T16:19:05.222 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.378+0000 7f6ed5a00700 1 Processor -- start 2024-02-28T16:19:05.225 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.378+0000 7f6ed5a00700 1 -- start start 2024-02-28T16:19:05.225 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.379+0000 7f6ed5a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed0117a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.379+0000 7f6ed5a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed0117f50 con 0x7f6ed007c630 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.379+0000 7f6ecfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed0117a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.379+0000 7f6ecfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed0117a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35340/0 (socket says 172.21.15.1:35340) 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.379+0000 7f6ecfe00700 1 -- 172.21.15.1:0/4183572062 learned_addr learned my addr 172.21.15.1:0/4183572062 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.379+0000 7f6ecfe00700 1 -- 172.21.15.1:0/4183572062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec00094f0 con 0x7f6ed007c630 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.379+0000 7f6ecfe00700 1 --2- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed0117a10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00090b0 tx=0x7f6ec0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.380+0000 7f6ec7e00700 1 -- 172.21.15.1:0/4183572062 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec0010840 con 0x7f6ed007c630 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.380+0000 7f6ec7e00700 1 -- 172.21.15.1:0/4183572062 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ec0010e70 con 0x7f6ed007c630 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.380+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed0118150 con 0x7f6ed007c630 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.380+0000 7f6ec7e00700 1 -- 172.21.15.1:0/4183572062 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec0024e40 con 0x7f6ed007c630 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.380+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed0118df0 con 0x7f6ed007c630 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.381+0000 7f6ec7e00700 1 -- 172.21.15.1:0/4183572062 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f6ec001a4e0 con 0x7f6ed007c630 2024-02-28T16:19:05.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.381+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb00052e0 con 0x7f6ed007c630 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.381+0000 7f6ec7e00700 1 --2- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f6eb40368d0 0x7f6eb4038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.382+0000 7f6ec7e00700 1 -- 172.21.15.1:0/4183572062 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6ec004aa30 con 0x7f6ed007c630 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.382+0000 7f6ecf400700 1 --2- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f6eb40368d0 0x7f6eb4038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.382+0000 7f6ecf400700 1 --2- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f6eb40368d0 0x7f6eb4038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc009990 tx=0x7f6ebc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.387+0000 7f6ec7e00700 1 -- 172.21.15.1:0/4183572062 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6ec0018940 con 0x7f6ed007c630 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.600+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f6eb0005b10 con 0x7f6ed007c630 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.601+0000 7f6ec7e00700 1 -- 172.21.15.1:0/4183572062 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f6ec0018350 con 0x7f6ed007c630 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f6eb40368d0 msgr2=0x7f6eb4038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 --2- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f6eb40368d0 0x7f6eb4038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc009990 tx=0x7f6ebc006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 msgr2=0x7f6ed0117a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 --2- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed0117a10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00090b0 tx=0x7f6ec0009250 comp rx=0 tx=0).stop 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 shutdown_connections 2024-02-28T16:19:05.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 --2- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f6eb40368d0 0x7f6eb4038d80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:05.228 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 --2- 172.21.15.1:0/4183572062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ed007c630 0x7f6ed0117a10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:05.228 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 >> 172.21.15.1:0/4183572062 conn(0x7f6ed0077dd0 msgr2=0x7f6ed0079480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:05.228 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 shutdown_connections 2024-02-28T16:19:05.228 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:04.604+0000 7f6ed5a00700 1 -- 172.21.15.1:0/4183572062 wait complete. 2024-02-28T16:19:07.064 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.855+0000 7f27f3800700 1 Processor -- start 2024-02-28T16:19:07.064 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.856+0000 7f27f3800700 1 -- start start 2024-02-28T16:19:07.064 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.856+0000 7f27f3800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:07.064 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.856+0000 7f27f3800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec105960 con 0x7f27ec0713e0 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.856+0000 7f27f0e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.856+0000 7f27f0e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35350/0 (socket says 172.21.15.1:35350) 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.856+0000 7f27f0e00700 1 -- 172.21.15.1:0/1970453791 learned_addr learned my addr 172.21.15.1:0/1970453791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.857+0000 7f27f0e00700 1 -- 172.21.15.1:0/1970453791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec105aa0 con 0x7f27ec0713e0 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.857+0000 7f27f0e00700 1 --2- 172.21.15.1:0/1970453791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec105420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f27dc0098e0 tx=0x7f27dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f507ade7c2fae4 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.858+0000 7f27ebe00700 1 -- 172.21.15.1:0/1970453791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27dc00b860 con 0x7f27ec0713e0 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.858+0000 7f27ebe00700 1 -- 172.21.15.1:0/1970453791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27dc00be90 con 0x7f27ec0713e0 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.858+0000 7f27ebe00700 1 -- 172.21.15.1:0/1970453791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27dc0103c0 con 0x7f27ec0713e0 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.858+0000 7f27f3800700 1 -- 172.21.15.1:0/1970453791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 msgr2=0x7f27ec105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.858+0000 7f27f3800700 1 --2- 172.21.15.1:0/1970453791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec105420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f27dc0098e0 tx=0x7f27dc009250 comp rx=0 tx=0).stop 2024-02-28T16:19:07.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.858+0000 7f27f3800700 1 -- 172.21.15.1:0/1970453791 shutdown_connections 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.858+0000 7f27f3800700 1 --2- 172.21.15.1:0/1970453791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec105420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.859+0000 7f27f3800700 1 -- 172.21.15.1:0/1970453791 >> 172.21.15.1:0/1970453791 conn(0x7f27ec0fcdd0 msgr2=0x7f27ec0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.859+0000 7f27f3800700 1 -- 172.21.15.1:0/1970453791 shutdown_connections 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.859+0000 7f27f3800700 1 -- 172.21.15.1:0/1970453791 wait complete. 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.859+0000 7f27f3800700 1 Processor -- start 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.860+0000 7f27f3800700 1 -- start start 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.860+0000 7f27f3800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.860+0000 7f27f3800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec113d60 con 0x7f27ec0713e0 2024-02-28T16:19:07.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.860+0000 7f27f0e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.860+0000 7f27f0e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35366/0 (socket says 172.21.15.1:35366) 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.860+0000 7f27f0e00700 1 -- 172.21.15.1:0/1477248237 learned_addr learned my addr 172.21.15.1:0/1477248237 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.861+0000 7f27f0e00700 1 -- 172.21.15.1:0/1477248237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc0094f0 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.861+0000 7f27f0e00700 1 --2- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec113820 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27dc0105b0 tx=0x7f27dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.861+0000 7f27e9600700 1 -- 172.21.15.1:0/1477248237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27dc010840 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.861+0000 7f27e9600700 1 -- 172.21.15.1:0/1477248237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27dc010e70 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.861+0000 7f27e9600700 1 -- 172.21.15.1:0/1477248237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27dc0243f0 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.861+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec113f60 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.862+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec114c00 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.863+0000 7f27e9600700 1 -- 172.21.15.1:0/1477248237 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f27dc01a7c0 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.863+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec05f5c0 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.863+0000 7f27e9600700 1 --2- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f27d0036920 0x7f27d0038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.863+0000 7f27e9600700 1 -- 172.21.15.1:0/1477248237 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f27dc04a810 con 0x7f27ec0713e0 2024-02-28T16:19:07.067 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.863+0000 7f27e3e00700 1 --2- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f27d0036920 0x7f27d0038dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.864+0000 7f27e3e00700 1 --2- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f27d0036920 0x7f27d0038dd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009990 tx=0x7f27d4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:05.868+0000 7f27e9600700 1 -- 172.21.15.1:0/1477248237 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f27dc018b80 con 0x7f27ec0713e0 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.192+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f27ec04bed0 con 0x7f27ec0713e0 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.196+0000 7f27e9600700 1 -- 172.21.15.1:0/1477248237 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f27dc018770 con 0x7f27ec0713e0 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f27d0036920 msgr2=0x7f27d0038dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 --2- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f27d0036920 0x7f27d0038dd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009990 tx=0x7f27d4006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 msgr2=0x7f27ec113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 --2- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec113820 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27dc0105b0 tx=0x7f27dc009250 comp rx=0 tx=0).stop 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 shutdown_connections 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 --2- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f27d0036920 0x7f27d0038dd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 --2- 172.21.15.1:0/1477248237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27ec0713e0 0x7f27ec113820 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 >> 172.21.15.1:0/1477248237 conn(0x7f27ec0fcdd0 msgr2=0x7f27ec0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:07.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 shutdown_connections 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-02-28T16:19:06.199+0000 7f27f3800700 1 -- 172.21.15.1:0/1477248237 wait complete. 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout:Or, if you are only running a single cluster on this host: 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:07.069 INFO:teuthology.orchestra.run.smithi001.stdout: ceph telemetry on 2024-02-28T16:19:07.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:07.070 INFO:teuthology.orchestra.run.smithi001.stdout:For more information see: 2024-02-28T16:19:07.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:07.070 INFO:teuthology.orchestra.run.smithi001.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-28T16:19:07.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:07.070 INFO:teuthology.orchestra.run.smithi001.stdout:Bootstrap complete. 2024-02-28T16:19:07.152 INFO:tasks.cephadm:Fetching config... 2024-02-28T16:19:07.153 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:19:07.153 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-28T16:19:07.201 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-28T16:19:07.201 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:19:07.201 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-28T16:19:07.280 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-28T16:19:07.280 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:19:07.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/keyring of=/dev/stdout 2024-02-28T16:19:07.335 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:07 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1477248237' entity='client.admin' 2024-02-28T16:19:07.411 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-28T16:19:07.411 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:19:07.411 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-28T16:19:07.510 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-28T16:19:07.510 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbe61e3rMHFvhXRNG0ybqMHp9Oy+O72pvqcv7GpD4+n1+ylhGViFCn6Z+SMc9sowHUN4Q4wfQbRWGdhK+PmxVd67YAOPmj9wlsH3ty0TITfG5/7vBw7T3gxSDNWlFP4fiOB6lPnKlEMvP1H+95kd5cEnkYjjJMU6NijMFsJFAgjMKzeV8gWjU1XTm8DLds419PhZLw0BWWUivvayqbyIiNjMiPRNdaSUdg48hYdt3dX04fOR2Fn1lq4yRdIzQUNK/n4uhfPmH/4xONi6+eiQQ63VHQxPcv5XCA17d1qHnPwP/A9kcAH7GrUtaq1TYddA+zvNzT8TrFmfuMosi4/DDciyCjdoCgwHMQaTe8Q/w6F1ChLLkyFGzAZwYx5FNB9GAWau3bc/SdYLCUvDJkDPquZ8cwqfP+doAWC9DAioMSxGKpqk1p4uyWgamyhZGa+hI5TYyhkAeIGwUlXlZ7NGcCRkmPK1CCWJmrFasqEu2lJpfA0CtLbxFbndz/UedsnqE= ceph-c0118718-d654-11ee-95c3-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-28T16:19:07.653 INFO:teuthology.orchestra.run.smithi001.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbe61e3rMHFvhXRNG0ybqMHp9Oy+O72pvqcv7GpD4+n1+ylhGViFCn6Z+SMc9sowHUN4Q4wfQbRWGdhK+PmxVd67YAOPmj9wlsH3ty0TITfG5/7vBw7T3gxSDNWlFP4fiOB6lPnKlEMvP1H+95kd5cEnkYjjJMU6NijMFsJFAgjMKzeV8gWjU1XTm8DLds419PhZLw0BWWUivvayqbyIiNjMiPRNdaSUdg48hYdt3dX04fOR2Fn1lq4yRdIzQUNK/n4uhfPmH/4xONi6+eiQQ63VHQxPcv5XCA17d1qHnPwP/A9kcAH7GrUtaq1TYddA+zvNzT8TrFmfuMosi4/DDciyCjdoCgwHMQaTe8Q/w6F1ChLLkyFGzAZwYx5FNB9GAWau3bc/SdYLCUvDJkDPquZ8cwqfP+doAWC9DAioMSxGKpqk1p4uyWgamyhZGa+hI5TYyhkAeIGwUlXlZ7NGcCRkmPK1CCWJmrFasqEu2lJpfA0CtLbxFbndz/UedsnqE= ceph-c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:19:07.726 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbe61e3rMHFvhXRNG0ybqMHp9Oy+O72pvqcv7GpD4+n1+ylhGViFCn6Z+SMc9sowHUN4Q4wfQbRWGdhK+PmxVd67YAOPmj9wlsH3ty0TITfG5/7vBw7T3gxSDNWlFP4fiOB6lPnKlEMvP1H+95kd5cEnkYjjJMU6NijMFsJFAgjMKzeV8gWjU1XTm8DLds419PhZLw0BWWUivvayqbyIiNjMiPRNdaSUdg48hYdt3dX04fOR2Fn1lq4yRdIzQUNK/n4uhfPmH/4xONi6+eiQQ63VHQxPcv5XCA17d1qHnPwP/A9kcAH7GrUtaq1TYddA+zvNzT8TrFmfuMosi4/DDciyCjdoCgwHMQaTe8Q/w6F1ChLLkyFGzAZwYx5FNB9GAWau3bc/SdYLCUvDJkDPquZ8cwqfP+doAWC9DAioMSxGKpqk1p4uyWgamyhZGa+hI5TYyhkAeIGwUlXlZ7NGcCRkmPK1CCWJmrFasqEu2lJpfA0CtLbxFbndz/UedsnqE= ceph-c0118718-d654-11ee-95c3-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-28T16:19:07.882 INFO:teuthology.orchestra.run.smithi066.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbe61e3rMHFvhXRNG0ybqMHp9Oy+O72pvqcv7GpD4+n1+ylhGViFCn6Z+SMc9sowHUN4Q4wfQbRWGdhK+PmxVd67YAOPmj9wlsH3ty0TITfG5/7vBw7T3gxSDNWlFP4fiOB6lPnKlEMvP1H+95kd5cEnkYjjJMU6NijMFsJFAgjMKzeV8gWjU1XTm8DLds419PhZLw0BWWUivvayqbyIiNjMiPRNdaSUdg48hYdt3dX04fOR2Fn1lq4yRdIzQUNK/n4uhfPmH/4xONi6+eiQQ63VHQxPcv5XCA17d1qHnPwP/A9kcAH7GrUtaq1TYddA+zvNzT8TrFmfuMosi4/DDciyCjdoCgwHMQaTe8Q/w6F1ChLLkyFGzAZwYx5FNB9GAWau3bc/SdYLCUvDJkDPquZ8cwqfP+doAWC9DAioMSxGKpqk1p4uyWgamyhZGa+hI5TYyhkAeIGwUlXlZ7NGcCRkmPK1CCWJmrFasqEu2lJpfA0CtLbxFbndz/UedsnqE= ceph-c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:19:07.960 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-28T16:19:08.422 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:19:11.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.105+0000 7f4f56c00700 1 -- 172.21.15.1:0/2461016872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 msgr2=0x7f4f500ff920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:11.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.105+0000 7f4f56c00700 1 --2- 172.21.15.1:0/2461016872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 0x7f4f500ff920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f440048c0 tx=0x7f4f44009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:11.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.106+0000 7f4f56c00700 1 -- 172.21.15.1:0/2461016872 shutdown_connections 2024-02-28T16:19:11.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.106+0000 7f4f56c00700 1 --2- 172.21.15.1:0/2461016872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 0x7f4f500ff920 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:11.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.106+0000 7f4f56c00700 1 -- 172.21.15.1:0/2461016872 >> 172.21.15.1:0/2461016872 conn(0x7f4f500f6fc0 msgr2=0x7f4f500f93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:11.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.106+0000 7f4f56c00700 1 -- 172.21.15.1:0/2461016872 shutdown_connections 2024-02-28T16:19:11.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.106+0000 7f4f56c00700 1 -- 172.21.15.1:0/2461016872 wait complete. 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.107+0000 7f4f56c00700 1 Processor -- start 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.107+0000 7f4f56c00700 1 -- start start 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.107+0000 7f4f56c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 0x7f4f500fcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.107+0000 7f4f56c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f500fb450 con 0x7f4f500fd540 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.107+0000 7f4f55800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 0x7f4f500fcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.107+0000 7f4f55800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 0x7f4f500fcfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35390/0 (socket says 172.21.15.1:35390) 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.107+0000 7f4f55800700 1 -- 172.21.15.1:0/2589297467 learned_addr learned my addr 172.21.15.1:0/2589297467 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.108+0000 7f4f55800700 1 -- 172.21.15.1:0/2589297467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44005040 con 0x7f4f500fd540 2024-02-28T16:19:11.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.108+0000 7f4f55800700 1 --2- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 0x7f4f500fcfd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009b80 tx=0x7f4f44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:11.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.108+0000 7f4f4d600700 1 -- 172.21.15.1:0/2589297467 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4400b5c0 con 0x7f4f500fd540 2024-02-28T16:19:11.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.108+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f500fb650 con 0x7f4f500fd540 2024-02-28T16:19:11.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.108+0000 7f4f4d600700 1 -- 172.21.15.1:0/2589297467 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f4400be70 con 0x7f4f500fd540 2024-02-28T16:19:11.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.108+0000 7f4f4d600700 1 -- 172.21.15.1:0/2589297467 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4401c660 con 0x7f4f500fd540 2024-02-28T16:19:11.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.109+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f500fbac0 con 0x7f4f500fd540 2024-02-28T16:19:11.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.110+0000 7f4f4d600700 1 -- 172.21.15.1:0/2589297467 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f4f44004d30 con 0x7f4f500fd540 2024-02-28T16:19:11.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.110+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f50106700 con 0x7f4f500fd540 2024-02-28T16:19:11.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.110+0000 7f4f4d600700 1 --2- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4f38036870 0x7f4f38038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:11.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.110+0000 7f4f4d600700 1 -- 172.21.15.1:0/2589297467 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4f44049a70 con 0x7f4f500fd540 2024-02-28T16:19:11.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.111+0000 7f4f54e00700 1 --2- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4f38036870 0x7f4f38038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:11.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.111+0000 7f4f54e00700 1 --2- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4f38036870 0x7f4f38038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009990 tx=0x7f4f3c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:11.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.116+0000 7f4f4d600700 1 -- 172.21.15.1:0/2589297467 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4f44025400 con 0x7f4f500fd540 2024-02-28T16:19:11.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.337+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4f501027c0 con 0x7f4f500fd540 2024-02-28T16:19:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.341+0000 7f4f4d600700 1 -- 172.21.15.1:0/2589297467 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4f44011370 con 0x7f4f500fd540 2024-02-28T16:19:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.345+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4f38036870 msgr2=0x7f4f38038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.345+0000 7f4f56c00700 1 --2- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4f38036870 0x7f4f38038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009990 tx=0x7f4f3c006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.345+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 msgr2=0x7f4f500fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.345+0000 7f4f56c00700 1 --2- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 0x7f4f500fcfd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009b80 tx=0x7f4f44009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.346+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 shutdown_connections 2024-02-28T16:19:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.346+0000 7f4f56c00700 1 --2- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4f38036870 0x7f4f38038d20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.346+0000 7f4f56c00700 1 --2- 172.21.15.1:0/2589297467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f500fd540 0x7f4f500fcfd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.346+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 >> 172.21.15.1:0/2589297467 conn(0x7f4f500f6fc0 msgr2=0x7f4f500f7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.346+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 shutdown_connections 2024-02-28T16:19:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:11.346+0000 7f4f56c00700 1 -- 172.21.15.1:0/2589297467 wait complete. 2024-02-28T16:19:11.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:11.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:11.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi001", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:19:11.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:11.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:19:11.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-28T16:19:11.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:19:11.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: Deploying daemon crash.smithi001 on smithi001 2024-02-28T16:19:11.727 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:11 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2589297467' entity='client.admin' 2024-02-28T16:19:12.098 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-28T16:19:12.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-28T16:19:12.571 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:19:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.117+0000 7f8e1a400700 1 -- 172.21.15.1:0/2395679827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 msgr2=0x7f8e140fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.117+0000 7f8e1a400700 1 --2- 172.21.15.1:0/2395679827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 0x7f8e140fcfd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8e040048c0 tx=0x7f8e04009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.117+0000 7f8e1a400700 1 -- 172.21.15.1:0/2395679827 shutdown_connections 2024-02-28T16:19:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.117+0000 7f8e1a400700 1 --2- 172.21.15.1:0/2395679827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 0x7f8e140fcfd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.117+0000 7f8e1a400700 1 -- 172.21.15.1:0/2395679827 >> 172.21.15.1:0/2395679827 conn(0x7f8e140f82a0 msgr2=0x7f8e140fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.117+0000 7f8e1a400700 1 -- 172.21.15.1:0/2395679827 shutdown_connections 2024-02-28T16:19:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.117+0000 7f8e1a400700 1 -- 172.21.15.1:0/2395679827 wait complete. 2024-02-28T16:19:15.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.118+0000 7f8e1a400700 1 Processor -- start 2024-02-28T16:19:15.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.118+0000 7f8e1a400700 1 -- start start 2024-02-28T16:19:15.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.118+0000 7f8e1a400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 0x7f8e14071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:15.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.118+0000 7f8e1a400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140702e0 con 0x7f8e140fcbc0 2024-02-28T16:19:15.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.119+0000 7f8e13400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 0x7f8e14071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:15.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.119+0000 7f8e13400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 0x7f8e14071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45276/0 (socket says 172.21.15.1:45276) 2024-02-28T16:19:15.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.119+0000 7f8e13400700 1 -- 172.21.15.1:0/2769524834 learned_addr learned my addr 172.21.15.1:0/2769524834 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:15.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.119+0000 7f8e13400700 1 -- 172.21.15.1:0/2769524834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e04005040 con 0x7f8e140fcbc0 2024-02-28T16:19:15.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.119+0000 7f8e13400700 1 --2- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 0x7f8e14071e60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009b80 tx=0x7f8e04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:15.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.120+0000 7f8e0b400700 1 -- 172.21.15.1:0/2769524834 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e0400b5c0 con 0x7f8e140fcbc0 2024-02-28T16:19:15.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.120+0000 7f8e0b400700 1 -- 172.21.15.1:0/2769524834 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e0400be70 con 0x7f8e140fcbc0 2024-02-28T16:19:15.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.120+0000 7f8e0b400700 1 -- 172.21.15.1:0/2769524834 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e0401c660 con 0x7f8e140fcbc0 2024-02-28T16:19:15.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.120+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e140704e0 con 0x7f8e140fcbc0 2024-02-28T16:19:15.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.120+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e140709d0 con 0x7f8e140fcbc0 2024-02-28T16:19:15.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.121+0000 7f8e0b400700 1 -- 172.21.15.1:0/2769524834 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f8e0401c7c0 con 0x7f8e140fcbc0 2024-02-28T16:19:15.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.121+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e14106990 con 0x7f8e140fcbc0 2024-02-28T16:19:15.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.122+0000 7f8e0b400700 1 --2- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8df80368c0 0x7f8df8038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:15.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.122+0000 7f8e0b400700 1 -- 172.21.15.1:0/2769524834 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8e04049b50 con 0x7f8e140fcbc0 2024-02-28T16:19:15.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.122+0000 7f8e12a00700 1 --2- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8df80368c0 0x7f8df8038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:15.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.123+0000 7f8e12a00700 1 --2- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8df80368c0 0x7f8df8038d70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009990 tx=0x7f8e00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:15.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.128+0000 7f8e0b400700 1 -- 172.21.15.1:0/2769524834 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8e04011550 con 0x7f8e140fcbc0 2024-02-28T16:19:15.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.349+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 --> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e1402b310 con 0x7f8df80368c0 2024-02-28T16:19:15.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.356+0000 7f8e0b400700 1 -- 172.21.15.1:0/2769524834 <== mgr.14162 v2:172.21.15.1:6800/701199259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8e1402b310 con 0x7f8df80368c0 2024-02-28T16:19:15.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8df80368c0 msgr2=0x7f8df8038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:15.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 --2- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8df80368c0 0x7f8df8038d70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009990 tx=0x7f8e00006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:15.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 msgr2=0x7f8e14071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:15.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 --2- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 0x7f8e14071e60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009b80 tx=0x7f8e04009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:15.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 shutdown_connections 2024-02-28T16:19:15.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 --2- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f8df80368c0 0x7f8df8038d70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:15.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 --2- 172.21.15.1:0/2769524834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e140fcbc0 0x7f8e14071e60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:15.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 >> 172.21.15.1:0/2769524834 conn(0x7f8e140f82a0 msgr2=0x7f8e140f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:15.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 shutdown_connections 2024-02-28T16:19:15.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:15.359+0000 7f8e1a400700 1 -- 172.21.15.1:0/2769524834 wait complete. 2024-02-28T16:19:16.693 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:16 smithi001 ceph-mon[28991]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:19:16.693 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:16 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:16.693 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:16 smithi001 ceph-mon[28991]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:16.737 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi066 2024-02-28T16:19:16.738 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:19:16.738 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/etc/ceph/ceph.conf 2024-02-28T16:19:16.800 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:19:16.800 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:19:16.899 INFO:tasks.cephadm:Adding host smithi066 to orchestrator... 2024-02-28T16:19:16.900 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch host add smithi066 2024-02-28T16:19:17.496 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:19:18.932 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:18 smithi001 ceph-mon[28991]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.820+0000 7f1b60e00700 1 -- 172.21.15.1:0/755875898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 msgr2=0x7f1b5c0fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.820+0000 7f1b60e00700 1 --2- 172.21.15.1:0/755875898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 0x7f1b5c0fcfd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0048c0 tx=0x7f1b4c009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.820+0000 7f1b60e00700 1 -- 172.21.15.1:0/755875898 shutdown_connections 2024-02-28T16:19:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.820+0000 7f1b60e00700 1 --2- 172.21.15.1:0/755875898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 0x7f1b5c0fcfd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.820+0000 7f1b60e00700 1 -- 172.21.15.1:0/755875898 >> 172.21.15.1:0/755875898 conn(0x7f1b5c0f82a0 msgr2=0x7f1b5c0fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.821+0000 7f1b60e00700 1 -- 172.21.15.1:0/755875898 shutdown_connections 2024-02-28T16:19:19.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.821+0000 7f1b60e00700 1 -- 172.21.15.1:0/755875898 wait complete. 2024-02-28T16:19:19.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.822+0000 7f1b60e00700 1 Processor -- start 2024-02-28T16:19:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.822+0000 7f1b60e00700 1 -- start start 2024-02-28T16:19:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.822+0000 7f1b60e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 0x7f1b5c10f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.822+0000 7f1b60e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c10f5e0 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.822+0000 7f1b59e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 0x7f1b5c10f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.822+0000 7f1b59e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 0x7f1b5c10f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45306/0 (socket says 172.21.15.1:45306) 2024-02-28T16:19:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.822+0000 7f1b59e00700 1 -- 172.21.15.1:0/4062388061 learned_addr learned my addr 172.21.15.1:0/4062388061 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.823+0000 7f1b59e00700 1 -- 172.21.15.1:0/4062388061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c005040 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.823+0000 7f1b59e00700 1 --2- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 0x7f1b5c10f0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009b80 tx=0x7f1b4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.823+0000 7f1b52000700 1 -- 172.21.15.1:0/4062388061 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4c00b5c0 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.823+0000 7f1b52000700 1 -- 172.21.15.1:0/4062388061 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b4c00be70 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.824+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5c10f7e0 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.824+0000 7f1b52000700 1 -- 172.21.15.1:0/4062388061 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4c01c660 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.824+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5c110480 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.825+0000 7f1b52000700 1 -- 172.21.15.1:0/4062388061 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f1b4c004d30 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.825+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5c05f5c0 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.825+0000 7f1b52000700 1 --2- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1b400368c0 0x7f1b40038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.826+0000 7f1b52000700 1 -- 172.21.15.1:0/4062388061 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1b4c049ae0 con 0x7f1b5c0fcbc0 2024-02-28T16:19:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.826+0000 7f1b59400700 1 --2- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1b400368c0 0x7f1b40038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.827+0000 7f1b59400700 1 --2- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1b400368c0 0x7f1b40038d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009990 tx=0x7f1b48006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:19.831+0000 7f1b52000700 1 -- 172.21.15.1:0/4062388061 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1b4c011370 con 0x7f1b5c0fcbc0 2024-02-28T16:19:20.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:20.077+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 --> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi066", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b5c111060 con 0x7f1b400368c0 2024-02-28T16:19:20.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:19 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:20.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:19 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:20.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:19 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:20.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:19 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:20.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:19 smithi001 ceph-mon[28991]: Deploying daemon node-exporter.smithi001 on smithi001 2024-02-28T16:19:21.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:20 smithi001 ceph-mon[28991]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:21.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:20 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:22.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:21 smithi001 ceph-mon[28991]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:19:23.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:22 smithi001 ceph-mon[28991]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:23.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:22 smithi001 ceph-mon[28991]: Deploying cephadm binary to smithi066 2024-02-28T16:19:24.733 INFO:teuthology.orchestra.run.smithi001.stdout:Added host 'smithi066' with addr '172.21.15.66' 2024-02-28T16:19:24.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.740+0000 7f1b52000700 1 -- 172.21.15.1:0/4062388061 <== mgr.14162 v2:172.21.15.1:6800/701199259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f1b5c111060 con 0x7f1b400368c0 2024-02-28T16:19:24.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.744+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1b400368c0 msgr2=0x7f1b40038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:24.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.744+0000 7f1b60e00700 1 --2- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1b400368c0 0x7f1b40038d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009990 tx=0x7f1b48006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:24.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.744+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 msgr2=0x7f1b5c10f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:24.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.744+0000 7f1b60e00700 1 --2- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 0x7f1b5c10f0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009b80 tx=0x7f1b4c009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:24.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.745+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 shutdown_connections 2024-02-28T16:19:24.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.745+0000 7f1b60e00700 1 --2- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1b400368c0 0x7f1b40038d70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:24.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.745+0000 7f1b60e00700 1 --2- 172.21.15.1:0/4062388061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b5c0fcbc0 0x7f1b5c10f0a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:24.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.745+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 >> 172.21.15.1:0/4062388061 conn(0x7f1b5c0f82a0 msgr2=0x7f1b5c0f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:24.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.745+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 shutdown_connections 2024-02-28T16:19:24.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:24.745+0000 7f1b60e00700 1 -- 172.21.15.1:0/4062388061 wait complete. 2024-02-28T16:19:25.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:24 smithi001 ceph-mon[28991]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:25.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:24 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:25.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch host ls --format=json 2024-02-28T16:19:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:25 smithi001 ceph-mon[28991]: Added host smithi066 2024-02-28T16:19:26.271 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:19:26.943 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:26 smithi001 ceph-mon[28991]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:26.943 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:26 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:26.943 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:26 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:26.943 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:26 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:26.944 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:26 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:28.083 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:27 smithi001 ceph-mon[28991]: Deploying daemon alertmanager.smithi001 on smithi001 2024-02-28T16:19:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.261+0000 7f4ff4600700 1 -- 172.21.15.1:0/1207025162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 msgr2=0x7f4fec0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.261+0000 7f4ff4600700 1 --2- 172.21.15.1:0/1207025162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 0x7f4fec0fb910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00048c0 tx=0x7f4fe0009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.261+0000 7f4ff4600700 1 -- 172.21.15.1:0/1207025162 shutdown_connections 2024-02-28T16:19:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.261+0000 7f4ff4600700 1 --2- 172.21.15.1:0/1207025162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 0x7f4fec0fb910 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.261+0000 7f4ff4600700 1 -- 172.21.15.1:0/1207025162 >> 172.21.15.1:0/1207025162 conn(0x7f4fec0f7190 msgr2=0x7f4fec0f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.261+0000 7f4ff4600700 1 -- 172.21.15.1:0/1207025162 shutdown_connections 2024-02-28T16:19:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.261+0000 7f4ff4600700 1 -- 172.21.15.1:0/1207025162 wait complete. 2024-02-28T16:19:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.262+0000 7f4ff4600700 1 Processor -- start 2024-02-28T16:19:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.263+0000 7f4ff4600700 1 -- start start 2024-02-28T16:19:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.263+0000 7f4ff4600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 0x7f4fec10f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.263+0000 7f4ff4600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec10f5c0 con 0x7f4fec0fb500 2024-02-28T16:19:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.263+0000 7f4ff1c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 0x7f4fec10f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.263+0000 7f4ff1c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 0x7f4fec10f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55918/0 (socket says 172.21.15.1:55918) 2024-02-28T16:19:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.263+0000 7f4ff1c00700 1 -- 172.21.15.1:0/4121690141 learned_addr learned my addr 172.21.15.1:0/4121690141 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:28.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.264+0000 7f4ff1c00700 1 -- 172.21.15.1:0/4121690141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0005040 con 0x7f4fec0fb500 2024-02-28T16:19:28.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.264+0000 7f4ff1c00700 1 --2- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 0x7f4fec10f080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009680 tx=0x7f4fe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.265+0000 7f4fe9600700 1 -- 172.21.15.1:0/4121690141 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fe000b5c0 con 0x7f4fec0fb500 2024-02-28T16:19:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.265+0000 7f4fe9600700 1 -- 172.21.15.1:0/4121690141 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fe0004e60 con 0x7f4fec0fb500 2024-02-28T16:19:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.265+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec10f7c0 con 0x7f4fec0fb500 2024-02-28T16:19:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.265+0000 7f4fe9600700 1 -- 172.21.15.1:0/4121690141 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fe001c590 con 0x7f4fec0fb500 2024-02-28T16:19:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.265+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec110460 con 0x7f4fec0fb500 2024-02-28T16:19:28.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.266+0000 7f4fe9600700 1 -- 172.21.15.1:0/4121690141 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f4fe0013a80 con 0x7f4fec0fb500 2024-02-28T16:19:28.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.266+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec05f5c0 con 0x7f4fec0fb500 2024-02-28T16:19:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.266+0000 7f4fe9600700 1 --2- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4fd4036870 0x7f4fd4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.267+0000 7f4fe9600700 1 -- 172.21.15.1:0/4121690141 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4fe0049c00 con 0x7f4fec0fb500 2024-02-28T16:19:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.267+0000 7f4ff1200700 1 --2- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4fd4036870 0x7f4fd4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.268+0000 7f4ff1200700 1 --2- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4fd4036870 0x7f4fd4038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009990 tx=0x7f4fd8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:28.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.272+0000 7f4fe9600700 1 -- 172.21.15.1:0/4121690141 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4fe0025400 con 0x7f4fec0fb500 2024-02-28T16:19:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.498+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 --> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4fec111040 con 0x7f4fd4036870 2024-02-28T16:19:28.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.501+0000 7f4fe9600700 1 -- 172.21.15.1:0/4121690141 <== mgr.14162 v2:172.21.15.1:6800/701199259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7f4fec111040 con 0x7f4fd4036870 2024-02-28T16:19:28.491 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:19:28.491 INFO:teuthology.orchestra.run.smithi001.stdout:[{"addr": "172.21.15.1", "hostname": "smithi001", "labels": [], "status": ""}, {"addr": "172.21.15.66", "hostname": "smithi066", "labels": [], "status": ""}] 2024-02-28T16:19:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4fd4036870 msgr2=0x7f4fd4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 --2- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4fd4036870 0x7f4fd4038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009990 tx=0x7f4fd8006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 msgr2=0x7f4fec10f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 --2- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 0x7f4fec10f080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009680 tx=0x7f4fe0009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 shutdown_connections 2024-02-28T16:19:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 --2- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f4fd4036870 0x7f4fd4038d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 --2- 172.21.15.1:0/4121690141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fec0fb500 0x7f4fec10f080 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 >> 172.21.15.1:0/4121690141 conn(0x7f4fec0f7190 msgr2=0x7f4fec0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:28.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 shutdown_connections 2024-02-28T16:19:28.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:28.504+0000 7f4ff4600700 1 -- 172.21.15.1:0/4121690141 wait complete. 2024-02-28T16:19:29.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:28 smithi001 ceph-mon[28991]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:29.476 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-28T16:19:29.476 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd crush tunables default 2024-02-28T16:19:30.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:29 smithi001 ceph-mon[28991]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-28T16:19:30.712 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:19:31.149 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:30 smithi001 ceph-mon[28991]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:31.149 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:30 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:33.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:32 smithi001 ceph-mon[28991]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.487+0000 7fc276c00700 1 -- 172.21.15.1:0/393923330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 msgr2=0x7fc2700ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:33.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.487+0000 7fc276c00700 1 --2- 172.21.15.1:0/393923330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 0x7fc2700ff1e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc2600048c0 tx=0x7fc260009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.487+0000 7fc276c00700 1 -- 172.21.15.1:0/393923330 shutdown_connections 2024-02-28T16:19:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.487+0000 7fc276c00700 1 --2- 172.21.15.1:0/393923330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 0x7fc2700ff1e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.487+0000 7fc276c00700 1 -- 172.21.15.1:0/393923330 >> 172.21.15.1:0/393923330 conn(0x7fc2700fa5a0 msgr2=0x7fc2700fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.487+0000 7fc276c00700 1 -- 172.21.15.1:0/393923330 shutdown_connections 2024-02-28T16:19:33.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.487+0000 7fc276c00700 1 -- 172.21.15.1:0/393923330 wait complete. 2024-02-28T16:19:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.488+0000 7fc276c00700 1 Processor -- start 2024-02-28T16:19:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.488+0000 7fc276c00700 1 -- start start 2024-02-28T16:19:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.488+0000 7fc276c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 0x7fc27010f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.488+0000 7fc276c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27010f600 con 0x7fc2700fedd0 2024-02-28T16:19:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.489+0000 7fc26fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 0x7fc27010f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.489+0000 7fc26fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 0x7fc27010f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51236/0 (socket says 172.21.15.1:51236) 2024-02-28T16:19:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.489+0000 7fc26fe00700 1 -- 172.21.15.1:0/228311644 learned_addr learned my addr 172.21.15.1:0/228311644 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:19:33.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.489+0000 7fc26fe00700 1 -- 172.21.15.1:0/228311644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc260005040 con 0x7fc2700fedd0 2024-02-28T16:19:33.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.489+0000 7fc26fe00700 1 --2- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 0x7fc27010f0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc260009b80 tx=0x7fc260009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:33.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.490+0000 7fc267e00700 1 -- 172.21.15.1:0/228311644 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26000b5c0 con 0x7fc2700fedd0 2024-02-28T16:19:33.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.490+0000 7fc267e00700 1 -- 172.21.15.1:0/228311644 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc26000be70 con 0x7fc2700fedd0 2024-02-28T16:19:33.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.490+0000 7fc267e00700 1 -- 172.21.15.1:0/228311644 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26001c660 con 0x7fc2700fedd0 2024-02-28T16:19:33.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.490+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27010f800 con 0x7fc2700fedd0 2024-02-28T16:19:33.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.490+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2701104a0 con 0x7fc2700fedd0 2024-02-28T16:19:33.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.491+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27005f5c0 con 0x7fc2700fedd0 2024-02-28T16:19:33.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.491+0000 7fc267e00700 1 -- 172.21.15.1:0/228311644 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fc260004d30 con 0x7fc2700fedd0 2024-02-28T16:19:33.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.492+0000 7fc267e00700 1 --2- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fc2540368c0 0x7fc254038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:33.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.492+0000 7fc267e00700 1 -- 172.21.15.1:0/228311644 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc260049a90 con 0x7fc2700fedd0 2024-02-28T16:19:33.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.492+0000 7fc26f400700 1 --2- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fc2540368c0 0x7fc254038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:33.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.493+0000 7fc26f400700 1 --2- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fc2540368c0 0x7fc254038d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009990 tx=0x7fc25c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:33.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.497+0000 7fc267e00700 1 -- 172.21.15.1:0/228311644 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc260025400 con 0x7fc2700fedd0 2024-02-28T16:19:33.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.722+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc270111010 con 0x7fc2700fedd0 2024-02-28T16:19:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.973+0000 7fc267e00700 1 -- 172.21.15.1:0/228311644 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc260011370 con 0x7fc2700fedd0 2024-02-28T16:19:33.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.977+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fc2540368c0 msgr2=0x7fc254038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:33.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.977+0000 7fc276c00700 1 --2- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fc2540368c0 0x7fc254038d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009990 tx=0x7fc25c006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:33.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.977+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 msgr2=0x7fc27010f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:33.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.977+0000 7fc276c00700 1 --2- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 0x7fc27010f0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc260009b80 tx=0x7fc260009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.977+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 shutdown_connections 2024-02-28T16:19:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.977+0000 7fc276c00700 1 --2- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fc2540368c0 0x7fc254038d70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.977+0000 7fc276c00700 1 --2- 172.21.15.1:0/228311644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2700fedd0 0x7fc27010f0c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.977+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 >> 172.21.15.1:0/228311644 conn(0x7fc2700fa5a0 msgr2=0x7fc2700fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.978+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 shutdown_connections 2024-02-28T16:19:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:19:33.978+0000 7fc276c00700 1 -- 172.21.15.1:0/228311644 wait complete. 2024-02-28T16:19:33.969 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-02-28T16:19:34.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:33 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/228311644' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-28T16:19:35.009 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:34 smithi001 ceph-mon[28991]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:35.009 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:34 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/228311644' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-28T16:19:35.009 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:34 smithi001 ceph-mon[28991]: osdmap e4: 0 total, 0 up, 0 in 2024-02-28T16:19:35.063 INFO:tasks.cephadm:Adding mon.smithi001 on smithi001 2024-02-28T16:19:35.063 INFO:tasks.cephadm:Adding mon.smithi066 on smithi066 2024-02-28T16:19:35.063 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch apply mon '2;smithi001:172.21.15.1=smithi001;smithi066:172.21.15.66=smithi066' 2024-02-28T16:19:35.539 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:35.645 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:37.203 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:36 smithi001 ceph-mon[28991]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:37.854 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.862+0000 7f5307c00700 1 -- 172.21.15.66:0/3485824816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 msgr2=0x7f5300070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:37.854 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.862+0000 7f5307c00700 1 --2- 172.21.15.66:0/3485824816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 0x7f5300070380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f52f40048c0 tx=0x7f52f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:37.854 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.863+0000 7f5307c00700 1 -- 172.21.15.66:0/3485824816 shutdown_connections 2024-02-28T16:19:37.854 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.863+0000 7f5307c00700 1 --2- 172.21.15.66:0/3485824816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 0x7f5300070380 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:37.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.863+0000 7f5307c00700 1 -- 172.21.15.66:0/3485824816 >> 172.21.15.66:0/3485824816 conn(0x7f53000f82a0 msgr2=0x7f53000fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:37.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.863+0000 7f5307c00700 1 -- 172.21.15.66:0/3485824816 shutdown_connections 2024-02-28T16:19:37.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.863+0000 7f5307c00700 1 -- 172.21.15.66:0/3485824816 wait complete. 2024-02-28T16:19:37.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.864+0000 7f5307c00700 1 Processor -- start 2024-02-28T16:19:37.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.864+0000 7f5307c00700 1 -- start start 2024-02-28T16:19:37.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.864+0000 7f5307c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 0x7f530010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:37.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.864+0000 7f5307c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530010f5c0 con 0x7f5300071f10 2024-02-28T16:19:37.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.865+0000 7f5305200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 0x7f530010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:37.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.865+0000 7f5305200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 0x7f530010f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:58220/0 (socket says 172.21.15.66:58220) 2024-02-28T16:19:37.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.865+0000 7f5305200700 1 -- 172.21.15.66:0/256489573 learned_addr learned my addr 172.21.15.66:0/256489573 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:19:37.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.865+0000 7f5305200700 1 -- 172.21.15.66:0/256489573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f4005040 con 0x7f5300071f10 2024-02-28T16:19:37.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.866+0000 7f5305200700 1 --2- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 0x7f530010f080 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009b80 tx=0x7f52f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:37.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.866+0000 7f52fcc00700 1 -- 172.21.15.66:0/256489573 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52f400b5c0 con 0x7f5300071f10 2024-02-28T16:19:37.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.866+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530010f7c0 con 0x7f5300071f10 2024-02-28T16:19:37.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.867+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5300110460 con 0x7f5300071f10 2024-02-28T16:19:37.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.867+0000 7f52fcc00700 1 -- 172.21.15.66:0/256489573 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52f400be70 con 0x7f5300071f10 2024-02-28T16:19:37.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.867+0000 7f52fcc00700 1 -- 172.21.15.66:0/256489573 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52f40255c0 con 0x7f5300071f10 2024-02-28T16:19:37.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.868+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530005f5c0 con 0x7f5300071f10 2024-02-28T16:19:37.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.868+0000 7f52fcc00700 1 -- 172.21.15.66:0/256489573 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f52f4004d30 con 0x7f5300071f10 2024-02-28T16:19:37.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.869+0000 7f52fcc00700 1 --2- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f52e8036820 0x7f52e8038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:37.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.869+0000 7f52fcc00700 1 -- 172.21.15.66:0/256489573 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f52f4049b90 con 0x7f5300071f10 2024-02-28T16:19:37.860 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.869+0000 7f52ffe00700 1 --2- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f52e8036820 0x7f52e8038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:37.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.871+0000 7f52ffe00700 1 --2- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f52e8036820 0x7f52e8038cd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009990 tx=0x7f52ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:37.874 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:37.884+0000 7f52fcc00700 1 -- 172.21.15.66:0/256489573 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f52f40238f0 con 0x7f5300071f10 2024-02-28T16:19:38.099 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.109+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 --> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi001:172.21.15.1=smithi001;smithi066:172.21.15.66=smithi066", "target": ["mon-mgr", ""]}) v1 -- 0x7f5300110ff0 con 0x7f52e8036820 2024-02-28T16:19:38.107 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.117+0000 7f52fcc00700 1 -- 172.21.15.66:0/256489573 <== mgr.14162 v2:172.21.15.1:6800/701199259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5300110ff0 con 0x7f52e8036820 2024-02-28T16:19:38.108 INFO:teuthology.orchestra.run.smithi066.stdout:Scheduled mon update... 2024-02-28T16:19:38.110 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f52e8036820 msgr2=0x7f52e8038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:38.110 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 --2- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f52e8036820 0x7f52e8038cd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009990 tx=0x7f52ec006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:38.110 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 msgr2=0x7f530010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:38.111 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 --2- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 0x7f530010f080 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009b80 tx=0x7f52f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:38.111 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 shutdown_connections 2024-02-28T16:19:38.111 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 --2- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f52e8036820 0x7f52e8038cd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:38.111 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 --2- 172.21.15.66:0/256489573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5300071f10 0x7f530010f080 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:38.111 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 >> 172.21.15.66:0/256489573 conn(0x7f53000f82a0 msgr2=0x7f53000f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:38.111 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 shutdown_connections 2024-02-28T16:19:38.111 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:38.120+0000 7f5307c00700 1 -- 172.21.15.66:0/256489573 wait complete. 2024-02-28T16:19:38.944 DEBUG:teuthology.orchestra.run.smithi066:mon.smithi066> sudo journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi066.service 2024-02-28T16:19:38.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:19:38.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:19:39.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:38 smithi001 ceph-mon[28991]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:39.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:38 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:39.442 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:39.542 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:40.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi001:172.21.15.1=smithi001;smithi066:172.21.15.66=smithi066", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: Saving service mon spec with placement smithi001:172.21.15.1=smithi001;smithi066:172.21.15.66=smithi066;count:2 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-28T16:19:40.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:39 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:40 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-28T16:19:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:40 smithi001 ceph-mon[28991]: Deploying daemon grafana.smithi001 on smithi001 2024-02-28T16:19:41.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:40 smithi001 ceph-mon[28991]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:41.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:40 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:19:42.031 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.040+0000 7f65e2e00700 1 -- 172.21.15.66:0/4131369280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 msgr2=0x7f65dc0ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:42.031 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.040+0000 7f65e2e00700 1 --2- 172.21.15.66:0/4131369280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 0x7f65dc0ffaf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0048c0 tx=0x7f65cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:42.031 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.040+0000 7f65e2e00700 1 -- 172.21.15.66:0/4131369280 shutdown_connections 2024-02-28T16:19:42.031 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.040+0000 7f65e2e00700 1 --2- 172.21.15.66:0/4131369280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 0x7f65dc0ffaf0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:42.031 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.040+0000 7f65e2e00700 1 -- 172.21.15.66:0/4131369280 >> 172.21.15.66:0/4131369280 conn(0x7f65dc0f7190 msgr2=0x7f65dc0f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:42.031 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.040+0000 7f65e2e00700 1 -- 172.21.15.66:0/4131369280 shutdown_connections 2024-02-28T16:19:42.032 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.040+0000 7f65e2e00700 1 -- 172.21.15.66:0/4131369280 wait complete. 2024-02-28T16:19:42.032 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.042+0000 7f65e2e00700 1 Processor -- start 2024-02-28T16:19:42.032 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.042+0000 7f65e2e00700 1 -- start start 2024-02-28T16:19:42.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.042+0000 7f65e2e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 0x7f65dc0fd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:42.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.042+0000 7f65e2e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc0fb620 con 0x7f65dc0fd710 2024-02-28T16:19:42.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.043+0000 7f65dbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 0x7f65dc0fd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:42.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.043+0000 7f65dbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 0x7f65dc0fd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53358/0 (socket says 172.21.15.66:53358) 2024-02-28T16:19:42.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.043+0000 7f65dbe00700 1 -- 172.21.15.66:0/2883185640 learned_addr learned my addr 172.21.15.66:0/2883185640 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:19:42.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.043+0000 7f65dbe00700 1 -- 172.21.15.66:0/2883185640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc005040 con 0x7f65dc0fd710 2024-02-28T16:19:42.034 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.044+0000 7f65dbe00700 1 --2- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 0x7f65dc0fd1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009b80 tx=0x7f65cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:42.034 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.044+0000 7f65d3e00700 1 -- 172.21.15.66:0/2883185640 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65cc00b5c0 con 0x7f65dc0fd710 2024-02-28T16:19:42.035 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.044+0000 7f65d3e00700 1 -- 172.21.15.66:0/2883185640 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65cc00be70 con 0x7f65dc0fd710 2024-02-28T16:19:42.035 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.044+0000 7f65e2e00700 1 -- 172.21.15.66:0/2883185640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65dc0fb820 con 0x7f65dc0fd710 2024-02-28T16:19:42.035 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.045+0000 7f65d3e00700 1 -- 172.21.15.66:0/2883185640 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65cc01c660 con 0x7f65dc0fd710 2024-02-28T16:19:42.035 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.045+0000 7f65e2e00700 1 -- 172.21.15.66:0/2883185640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65dc0fbd10 con 0x7f65dc0fd710 2024-02-28T16:19:42.036 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.046+0000 7f65d1600700 1 -- 172.21.15.66:0/2883185640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b40052b0 con 0x7f65dc0fd710 2024-02-28T16:19:42.036 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.046+0000 7f65d3e00700 1 -- 172.21.15.66:0/2883185640 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f65cc004d30 con 0x7f65dc0fd710 2024-02-28T16:19:42.037 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.046+0000 7f65d3e00700 1 --2- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f65c0036870 0x7f65c0038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:42.037 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.047+0000 7f65d3e00700 1 -- 172.21.15.66:0/2883185640 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f65cc049a70 con 0x7f65dc0fd710 2024-02-28T16:19:42.037 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.047+0000 7f65db400700 1 --2- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f65c0036870 0x7f65c0038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:42.038 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.048+0000 7f65db400700 1 --2- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f65c0036870 0x7f65c0038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009990 tx=0x7f65c8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:42.063 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.073+0000 7f65d3e00700 1 -- 172.21.15.66:0/2883185640 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f65cc025400 con 0x7f65dc0fd710 2024-02-28T16:19:42.350 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.360+0000 7f65d1600700 1 -- 172.21.15.66:0/2883185640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65b4005560 con 0x7f65dc0fd710 2024-02-28T16:19:42.351 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.361+0000 7f65d3e00700 1 -- 172.21.15.66:0/2883185640 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f65cc011370 con 0x7f65dc0fd710 2024-02-28T16:19:42.351 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:19:42.352 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.363+0000 7f65d1600700 1 -- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f65c0036870 msgr2=0x7f65c0038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.363+0000 7f65d1600700 1 --2- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f65c0036870 0x7f65c0038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009990 tx=0x7f65c8006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.363+0000 7f65d1600700 1 -- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 msgr2=0x7f65dc0fd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.363+0000 7f65d1600700 1 --2- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 0x7f65dc0fd1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009b80 tx=0x7f65cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.363+0000 7f65d1600700 1 -- 172.21.15.66:0/2883185640 shutdown_connections 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.363+0000 7f65d1600700 1 --2- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f65c0036870 0x7f65c0038d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.363+0000 7f65d1600700 1 --2- 172.21.15.66:0/2883185640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc0fd710 0x7f65dc0fd1a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.363+0000 7f65d1600700 1 -- 172.21.15.66:0/2883185640 >> 172.21.15.66:0/2883185640 conn(0x7f65dc0f7190 msgr2=0x7f65dc0f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.364+0000 7f65d1600700 1 -- 172.21.15.66:0/2883185640 shutdown_connections 2024-02-28T16:19:42.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:42.364+0000 7f65d1600700 1 -- 172.21.15.66:0/2883185640 wait complete. 2024-02-28T16:19:42.355 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:19:43.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:42 smithi001 ceph-mon[28991]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:43.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:42 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/2883185640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:19:44.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:19:44.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:19:44.710 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:44.817 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:45.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:44 smithi001 ceph-mon[28991]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:47.112 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.121+0000 7fe855a00700 1 -- 172.21.15.66:0/2180288286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 msgr2=0x7fe850070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:47.112 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.121+0000 7fe855a00700 1 --2- 172.21.15.66:0/2180288286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 0x7fe850070300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe840009c60 tx=0x7fe840004d10 comp rx=0 tx=0).stop 2024-02-28T16:19:47.113 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.121+0000 7fe855a00700 1 -- 172.21.15.66:0/2180288286 shutdown_connections 2024-02-28T16:19:47.113 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.121+0000 7fe855a00700 1 --2- 172.21.15.66:0/2180288286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 0x7fe850070300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:47.113 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.121+0000 7fe855a00700 1 -- 172.21.15.66:0/2180288286 >> 172.21.15.66:0/2180288286 conn(0x7fe8500f82a0 msgr2=0x7fe8500fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:47.113 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.122+0000 7fe855a00700 1 -- 172.21.15.66:0/2180288286 shutdown_connections 2024-02-28T16:19:47.113 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.122+0000 7fe855a00700 1 -- 172.21.15.66:0/2180288286 wait complete. 2024-02-28T16:19:47.113 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.123+0000 7fe855a00700 1 Processor -- start 2024-02-28T16:19:47.114 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.123+0000 7fe855a00700 1 -- start start 2024-02-28T16:19:47.114 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.123+0000 7fe855a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 0x7fe85010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:47.114 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.123+0000 7fe855a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85010f5c0 con 0x7fe850071e90 2024-02-28T16:19:47.114 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.124+0000 7fe84ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 0x7fe85010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:47.114 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.124+0000 7fe84ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 0x7fe85010f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53380/0 (socket says 172.21.15.66:53380) 2024-02-28T16:19:47.114 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.124+0000 7fe84ea00700 1 -- 172.21.15.66:0/665453877 learned_addr learned my addr 172.21.15.66:0/665453877 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:19:47.115 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.124+0000 7fe84ea00700 1 -- 172.21.15.66:0/665453877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840009940 con 0x7fe850071e90 2024-02-28T16:19:47.115 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.125+0000 7fe84ea00700 1 --2- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 0x7fe85010f080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe840004e60 tx=0x7fe840004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:47.116 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.125+0000 7fe846a00700 1 -- 172.21.15.66:0/665453877 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe84001d440 con 0x7fe850071e90 2024-02-28T16:19:47.116 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.125+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85010f7c0 con 0x7fe850071e90 2024-02-28T16:19:47.116 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.126+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe850110460 con 0x7fe850071e90 2024-02-28T16:19:47.116 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.126+0000 7fe846a00700 1 -- 172.21.15.66:0/665453877 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe84001da70 con 0x7fe850071e90 2024-02-28T16:19:47.116 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.126+0000 7fe846a00700 1 -- 172.21.15.66:0/665453877 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe840012840 con 0x7fe850071e90 2024-02-28T16:19:47.117 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.127+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8300052e0 con 0x7fe850071e90 2024-02-28T16:19:47.117 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.127+0000 7fe846a00700 1 -- 172.21.15.66:0/665453877 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fe840019040 con 0x7fe850071e90 2024-02-28T16:19:47.117 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.127+0000 7fe846a00700 1 --2- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fe8340368c0 0x7fe834038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:47.118 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.128+0000 7fe846a00700 1 -- 172.21.15.66:0/665453877 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe840049e80 con 0x7fe850071e90 2024-02-28T16:19:47.118 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.128+0000 7fe84e000700 1 --2- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fe8340368c0 0x7fe834038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:47.120 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.129+0000 7fe84e000700 1 --2- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fe8340368c0 0x7fe834038d70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe83c009990 tx=0x7fe83c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:47.134 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.144+0000 7fe846a00700 1 -- 172.21.15.66:0/665453877 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe8400232c0 con 0x7fe850071e90 2024-02-28T16:19:47.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:46 smithi001 ceph-mon[28991]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:47.432 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.441+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe830005590 con 0x7fe850071e90 2024-02-28T16:19:47.433 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.443+0000 7fe846a00700 1 -- 172.21.15.66:0/665453877 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fe840017030 con 0x7fe850071e90 2024-02-28T16:19:47.433 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:19:47.433 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.445+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fe8340368c0 msgr2=0x7fe834038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.445+0000 7fe855a00700 1 --2- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fe8340368c0 0x7fe834038d70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe83c009990 tx=0x7fe83c006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.445+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 msgr2=0x7fe85010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.445+0000 7fe855a00700 1 --2- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 0x7fe85010f080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe840004e60 tx=0x7fe840004d10 comp rx=0 tx=0).stop 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.445+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 shutdown_connections 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.445+0000 7fe855a00700 1 --2- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fe8340368c0 0x7fe834038d70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.445+0000 7fe855a00700 1 --2- 172.21.15.66:0/665453877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe850071e90 0x7fe85010f080 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.445+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 >> 172.21.15.66:0/665453877 conn(0x7fe8500f82a0 msgr2=0x7fe8500f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:47.436 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.446+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 shutdown_connections 2024-02-28T16:19:47.437 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:47.446+0000 7fe855a00700 1 -- 172.21.15.66:0/665453877 wait complete. 2024-02-28T16:19:47.437 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:19:48.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:47 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/665453877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:19:49.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:19:49.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:19:49.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:48 smithi001 ceph-mon[28991]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:49.646 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:49.748 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:50.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:50 smithi001 ceph-mon[28991]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:51.896 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.905+0000 7f1f55200700 1 -- 172.21.15.66:0/3612954151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 msgr2=0x7f1f500fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:51.896 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.905+0000 7f1f55200700 1 --2- 172.21.15.66:0/3612954151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 0x7f1f500fcfd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1f400048c0 tx=0x7f1f40009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:51.896 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.905+0000 7f1f55200700 1 -- 172.21.15.66:0/3612954151 shutdown_connections 2024-02-28T16:19:51.896 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.905+0000 7f1f55200700 1 --2- 172.21.15.66:0/3612954151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 0x7f1f500fcfd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:51.896 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.905+0000 7f1f55200700 1 -- 172.21.15.66:0/3612954151 >> 172.21.15.66:0/3612954151 conn(0x7f1f500f82a0 msgr2=0x7f1f500fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:51.896 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.905+0000 7f1f55200700 1 -- 172.21.15.66:0/3612954151 shutdown_connections 2024-02-28T16:19:51.896 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.905+0000 7f1f55200700 1 -- 172.21.15.66:0/3612954151 wait complete. 2024-02-28T16:19:51.897 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.906+0000 7f1f55200700 1 Processor -- start 2024-02-28T16:19:51.897 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.907+0000 7f1f55200700 1 -- start start 2024-02-28T16:19:51.897 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.907+0000 7f1f55200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 0x7f1f5010f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:51.897 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.907+0000 7f1f55200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5010f610 con 0x7f1f500fcbc0 2024-02-28T16:19:51.898 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.908+0000 7f1f4de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 0x7f1f5010f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:51.898 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.908+0000 7f1f4de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 0x7f1f5010f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53410/0 (socket says 172.21.15.66:53410) 2024-02-28T16:19:51.898 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.908+0000 7f1f4de00700 1 -- 172.21.15.66:0/2165002603 learned_addr learned my addr 172.21.15.66:0/2165002603 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:19:51.898 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.908+0000 7f1f4de00700 1 -- 172.21.15.66:0/2165002603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f40005040 con 0x7f1f500fcbc0 2024-02-28T16:19:51.899 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.909+0000 7f1f4de00700 1 --2- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 0x7f1f5010f0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009b80 tx=0x7f1f40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:51.900 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.909+0000 7f1f46000700 1 -- 172.21.15.66:0/2165002603 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f4000b5c0 con 0x7f1f500fcbc0 2024-02-28T16:19:51.900 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.910+0000 7f1f46000700 1 -- 172.21.15.66:0/2165002603 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f4000be70 con 0x7f1f500fcbc0 2024-02-28T16:19:51.900 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.910+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5010f810 con 0x7f1f500fcbc0 2024-02-28T16:19:51.900 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.910+0000 7f1f46000700 1 -- 172.21.15.66:0/2165002603 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f4001c660 con 0x7f1f500fcbc0 2024-02-28T16:19:51.900 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.910+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f501104b0 con 0x7f1f500fcbc0 2024-02-28T16:19:51.901 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.911+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5005f5c0 con 0x7f1f500fcbc0 2024-02-28T16:19:51.901 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.911+0000 7f1f46000700 1 -- 172.21.15.66:0/2165002603 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f1f40004d30 con 0x7f1f500fcbc0 2024-02-28T16:19:51.902 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.912+0000 7f1f46000700 1 --2- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1f340368c0 0x7f1f34038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:51.902 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.912+0000 7f1f46000700 1 -- 172.21.15.66:0/2165002603 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1f400499d0 con 0x7f1f500fcbc0 2024-02-28T16:19:51.902 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.913+0000 7f1f4d400700 1 --2- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1f340368c0 0x7f1f34038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:51.904 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.914+0000 7f1f4d400700 1 --2- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1f340368c0 0x7f1f34038d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c009990 tx=0x7f1f3c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:51.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:51.923+0000 7f1f46000700 1 -- 172.21.15.66:0/2165002603 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1f40025400 con 0x7f1f500fcbc0 2024-02-28T16:19:52.211 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.220+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f50110830 con 0x7f1f500fcbc0 2024-02-28T16:19:52.212 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.222+0000 7f1f46000700 1 -- 172.21.15.66:0/2165002603 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f1f40011370 con 0x7f1f500fcbc0 2024-02-28T16:19:52.212 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:19:52.212 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:19:52.215 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1f340368c0 msgr2=0x7f1f34038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:52.215 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 --2- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1f340368c0 0x7f1f34038d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c009990 tx=0x7f1f3c006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:52.215 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 msgr2=0x7f1f5010f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:52.215 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 --2- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 0x7f1f5010f0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009b80 tx=0x7f1f40009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:52.215 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 shutdown_connections 2024-02-28T16:19:52.215 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 --2- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f1f340368c0 0x7f1f34038d70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:52.215 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 --2- 172.21.15.66:0/2165002603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f500fcbc0 0x7f1f5010f0d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:52.216 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 >> 172.21.15.66:0/2165002603 conn(0x7f1f500f82a0 msgr2=0x7f1f500701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:52.216 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 shutdown_connections 2024-02-28T16:19:52.216 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:52.225+0000 7f1f55200700 1 -- 172.21.15.66:0/2165002603 wait complete. 2024-02-28T16:19:52.217 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:19:53.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:52 smithi001 ceph-mon[28991]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:53.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:52 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/2165002603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:19:54.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:19:54.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:19:54.562 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:54.670 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:55.014 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:54 smithi001 ceph-mon[28991]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:56.710 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.720+0000 7fbea6600700 1 -- 172.21.15.66:0/1857093183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 msgr2=0x7fbea01035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:56.711 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.720+0000 7fbea6600700 1 --2- 172.21.15.66:0/1857093183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 0x7fbea01035e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbe900048c0 tx=0x7fbe90009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:56.711 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.720+0000 7fbea6600700 1 -- 172.21.15.66:0/1857093183 shutdown_connections 2024-02-28T16:19:56.711 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.720+0000 7fbea6600700 1 --2- 172.21.15.66:0/1857093183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 0x7fbea01035e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:56.711 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.720+0000 7fbea6600700 1 -- 172.21.15.66:0/1857093183 >> 172.21.15.66:0/1857093183 conn(0x7fbea00722d0 msgr2=0x7fbea00746e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:56.711 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.720+0000 7fbea6600700 1 -- 172.21.15.66:0/1857093183 shutdown_connections 2024-02-28T16:19:56.711 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.720+0000 7fbea6600700 1 -- 172.21.15.66:0/1857093183 wait complete. 2024-02-28T16:19:56.712 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.721+0000 7fbea6600700 1 Processor -- start 2024-02-28T16:19:56.712 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.721+0000 7fbea6600700 1 -- start start 2024-02-28T16:19:56.712 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.722+0000 7fbea6600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 0x7fbea01134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:56.712 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.722+0000 7fbea6600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea0113a20 con 0x7fbea01031d0 2024-02-28T16:19:56.713 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.722+0000 7fbe9f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 0x7fbea01134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:56.713 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.722+0000 7fbe9f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 0x7fbea01134e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:33894/0 (socket says 172.21.15.66:33894) 2024-02-28T16:19:56.713 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.722+0000 7fbe9f400700 1 -- 172.21.15.66:0/662370485 learned_addr learned my addr 172.21.15.66:0/662370485 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:19:56.713 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.723+0000 7fbe9f400700 1 -- 172.21.15.66:0/662370485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90005040 con 0x7fbea01031d0 2024-02-28T16:19:56.715 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.723+0000 7fbe9f400700 1 --2- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 0x7fbea01134e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009680 tx=0x7fbe90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:56.715 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.724+0000 7fbe97400700 1 -- 172.21.15.66:0/662370485 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9000bed0 con 0x7fbea01031d0 2024-02-28T16:19:56.715 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.724+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea0113c20 con 0x7fbea01031d0 2024-02-28T16:19:56.715 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.724+0000 7fbe97400700 1 -- 172.21.15.66:0/662370485 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe9000b5c0 con 0x7fbea01031d0 2024-02-28T16:19:56.715 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.724+0000 7fbe97400700 1 -- 172.21.15.66:0/662370485 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9001c5b0 con 0x7fbea01031d0 2024-02-28T16:19:56.715 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.724+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea01148c0 con 0x7fbea01031d0 2024-02-28T16:19:56.716 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.725+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea005f5c0 con 0x7fbea01031d0 2024-02-28T16:19:56.716 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.726+0000 7fbe97400700 1 -- 172.21.15.66:0/662370485 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fbe90004d30 con 0x7fbea01031d0 2024-02-28T16:19:56.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.726+0000 7fbe97400700 1 --2- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fbe84036820 0x7fbe84038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:19:56.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.726+0000 7fbe97400700 1 -- 172.21.15.66:0/662370485 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbe90049ae0 con 0x7fbea01031d0 2024-02-28T16:19:56.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.727+0000 7fbe9ea00700 1 --2- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fbe84036820 0x7fbe84038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:19:56.718 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.728+0000 7fbe9ea00700 1 --2- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fbe84036820 0x7fbe84038cd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009990 tx=0x7fbe8c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:19:56.732 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:56.742+0000 7fbe97400700 1 -- 172.21.15.66:0/662370485 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbe90011960 con 0x7fbea01031d0 2024-02-28T16:19:57.011 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:56 smithi001 ceph-mon[28991]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:57.028 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.038+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbea0115410 con 0x7fbea01031d0 2024-02-28T16:19:57.029 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.039+0000 7fbe97400700 1 -- 172.21.15.66:0/662370485 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fbe90011370 con 0x7fbea01031d0 2024-02-28T16:19:57.030 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:19:57.030 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:19:57.032 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fbe84036820 msgr2=0x7fbe84038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:57.032 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 --2- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fbe84036820 0x7fbe84038cd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009990 tx=0x7fbe8c006e40 comp rx=0 tx=0).stop 2024-02-28T16:19:57.032 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 msgr2=0x7fbea01134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:19:57.032 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 --2- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 0x7fbea01134e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009680 tx=0x7fbe90009e30 comp rx=0 tx=0).stop 2024-02-28T16:19:57.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 shutdown_connections 2024-02-28T16:19:57.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 --2- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fbe84036820 0x7fbe84038cd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:57.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 --2- 172.21.15.66:0/662370485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbea01031d0 0x7fbea01134e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:19:57.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 >> 172.21.15.66:0/662370485 conn(0x7fbea00722d0 msgr2=0x7fbea0073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:19:57.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 shutdown_connections 2024-02-28T16:19:57.033 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:19:57.042+0000 7fbea6600700 1 -- 172.21.15.66:0/662370485 wait complete. 2024-02-28T16:19:57.033 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:19:57.954 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:57 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/662370485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:19:58.800 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:19:58.801 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:19:58.905 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:19:58 smithi001 ceph-mon[28991]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:19:59.273 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:19:59.374 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:00.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:00 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:00.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:00 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:00.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:00 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:00.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:00 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:00.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:00 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:00.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:00 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:00.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:00 smithi001 ceph-mon[28991]: Deploying daemon prometheus.smithi001 on smithi001 2024-02-28T16:20:00.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:00 smithi001 ceph-mon[28991]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:01.687 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.696+0000 7f9e4d600700 1 -- 172.21.15.66:0/3933288378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 msgr2=0x7f9e480ff010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:01.687 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.696+0000 7f9e4d600700 1 --2- 172.21.15.66:0/3933288378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 0x7f9e480ff010 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9e380048c0 tx=0x7f9e38009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:01.688 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.697+0000 7f9e4d600700 1 -- 172.21.15.66:0/3933288378 shutdown_connections 2024-02-28T16:20:01.688 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.697+0000 7f9e4d600700 1 --2- 172.21.15.66:0/3933288378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 0x7f9e480ff010 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:01.688 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.697+0000 7f9e4d600700 1 -- 172.21.15.66:0/3933288378 >> 172.21.15.66:0/3933288378 conn(0x7f9e480fa5a0 msgr2=0x7f9e480fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:01.688 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.697+0000 7f9e4d600700 1 -- 172.21.15.66:0/3933288378 shutdown_connections 2024-02-28T16:20:01.688 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.697+0000 7f9e4d600700 1 -- 172.21.15.66:0/3933288378 wait complete. 2024-02-28T16:20:01.689 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.698+0000 7f9e4d600700 1 Processor -- start 2024-02-28T16:20:01.689 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.698+0000 7f9e4d600700 1 -- start start 2024-02-28T16:20:01.689 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.698+0000 7f9e4d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 0x7f9e4810ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:01.689 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.698+0000 7f9e4d600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4810f460 con 0x7f9e480fec00 2024-02-28T16:20:01.689 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.699+0000 7f9e47e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 0x7f9e4810ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:01.689 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.699+0000 7f9e47e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 0x7f9e4810ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:33918/0 (socket says 172.21.15.66:33918) 2024-02-28T16:20:01.689 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.699+0000 7f9e47e00700 1 -- 172.21.15.66:0/1885176418 learned_addr learned my addr 172.21.15.66:0/1885176418 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:01.690 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.699+0000 7f9e47e00700 1 -- 172.21.15.66:0/1885176418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e38005040 con 0x7f9e480fec00 2024-02-28T16:20:01.690 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.700+0000 7f9e47e00700 1 --2- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 0x7f9e4810ef20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009b80 tx=0x7f9e38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:01.690 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.700+0000 7f9e44c00700 1 -- 172.21.15.66:0/1885176418 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e3800b5c0 con 0x7f9e480fec00 2024-02-28T16:20:01.691 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.700+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4810f660 con 0x7f9e480fec00 2024-02-28T16:20:01.691 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.700+0000 7f9e44c00700 1 -- 172.21.15.66:0/1885176418 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e3800be70 con 0x7f9e480fec00 2024-02-28T16:20:01.691 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.700+0000 7f9e44c00700 1 -- 172.21.15.66:0/1885176418 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e3801c660 con 0x7f9e480fec00 2024-02-28T16:20:01.691 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.701+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e48110300 con 0x7f9e480fec00 2024-02-28T16:20:01.692 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.702+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4805f5c0 con 0x7f9e480fec00 2024-02-28T16:20:01.692 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.702+0000 7f9e44c00700 1 -- 172.21.15.66:0/1885176418 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f9e38004d30 con 0x7f9e480fec00 2024-02-28T16:20:01.693 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.703+0000 7f9e44c00700 1 --2- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f9e2c036870 0x7f9e2c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:01.693 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.703+0000 7f9e44c00700 1 -- 172.21.15.66:0/1885176418 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9e380497d0 con 0x7f9e480fec00 2024-02-28T16:20:01.693 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.703+0000 7f9e3fe00700 1 --2- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f9e2c036870 0x7f9e2c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:01.694 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.704+0000 7f9e3fe00700 1 --2- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f9e2c036870 0x7f9e2c038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009990 tx=0x7f9e30006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:01.704 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:01.714+0000 7f9e44c00700 1 -- 172.21.15.66:0/1885176418 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9e38011760 con 0x7f9e480fec00 2024-02-28T16:20:02.004 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.013+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e48110e50 con 0x7f9e480fec00 2024-02-28T16:20:02.005 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.014+0000 7f9e44c00700 1 -- 172.21.15.66:0/1885176418 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f9e38011370 con 0x7f9e480fec00 2024-02-28T16:20:02.005 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:02.006 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:02.009 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.017+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f9e2c036870 msgr2=0x7f9e2c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:02.009 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.017+0000 7f9e4d600700 1 --2- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f9e2c036870 0x7f9e2c038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009990 tx=0x7f9e30006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:02.009 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.017+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 msgr2=0x7f9e4810ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:02.009 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.017+0000 7f9e4d600700 1 --2- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 0x7f9e4810ef20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009b80 tx=0x7f9e38009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:02.009 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.017+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 shutdown_connections 2024-02-28T16:20:02.009 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.017+0000 7f9e4d600700 1 --2- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f9e2c036870 0x7f9e2c038d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:02.010 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.017+0000 7f9e4d600700 1 --2- 172.21.15.66:0/1885176418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e480fec00 0x7f9e4810ef20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:02.010 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.017+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 >> 172.21.15.66:0/1885176418 conn(0x7f9e480fa5a0 msgr2=0x7f9e480fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:02.010 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.018+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 shutdown_connections 2024-02-28T16:20:02.010 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:02.018+0000 7f9e4d600700 1 -- 172.21.15.66:0/1885176418 wait complete. 2024-02-28T16:20:02.011 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:02.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:01 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:03.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:02 smithi001 ceph-mon[28991]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:03.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:02 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/1885176418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:03.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:03.933 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:04.410 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:04.520 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:05.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:04 smithi001 ceph-mon[28991]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:06.699 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.709+0000 7ff182400700 1 -- 172.21.15.66:0/121718207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 msgr2=0x7ff17c0ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:06.700 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.709+0000 7ff182400700 1 --2- 172.21.15.66:0/121718207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 0x7ff17c0ff1e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff16c0048c0 tx=0x7ff16c009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:06.700 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.709+0000 7ff182400700 1 -- 172.21.15.66:0/121718207 shutdown_connections 2024-02-28T16:20:06.700 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.709+0000 7ff182400700 1 --2- 172.21.15.66:0/121718207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 0x7ff17c0ff1e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:06.700 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.709+0000 7ff182400700 1 -- 172.21.15.66:0/121718207 >> 172.21.15.66:0/121718207 conn(0x7ff17c0fa5a0 msgr2=0x7ff17c0fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:06.700 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.709+0000 7ff182400700 1 -- 172.21.15.66:0/121718207 shutdown_connections 2024-02-28T16:20:06.700 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.709+0000 7ff182400700 1 -- 172.21.15.66:0/121718207 wait complete. 2024-02-28T16:20:06.701 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.710+0000 7ff182400700 1 Processor -- start 2024-02-28T16:20:06.701 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.710+0000 7ff182400700 1 -- start start 2024-02-28T16:20:06.701 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.710+0000 7ff182400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 0x7ff17c10f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:06.701 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.710+0000 7ff182400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c10f600 con 0x7ff17c0fedd0 2024-02-28T16:20:06.702 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.711+0000 7ff17b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 0x7ff17c10f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:06.702 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.711+0000 7ff17b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 0x7ff17c10f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53118/0 (socket says 172.21.15.66:53118) 2024-02-28T16:20:06.702 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.711+0000 7ff17b400700 1 -- 172.21.15.66:0/3137867745 learned_addr learned my addr 172.21.15.66:0/3137867745 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:06.702 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.712+0000 7ff17b400700 1 -- 172.21.15.66:0/3137867745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16c005040 con 0x7ff17c0fedd0 2024-02-28T16:20:06.702 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.712+0000 7ff17b400700 1 --2- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 0x7ff17c10f0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009b80 tx=0x7ff16c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:06.703 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.713+0000 7ff173e00700 1 -- 172.21.15.66:0/3137867745 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff16c00b5c0 con 0x7ff17c0fedd0 2024-02-28T16:20:06.703 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.713+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17c10f800 con 0x7ff17c0fedd0 2024-02-28T16:20:06.703 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.713+0000 7ff173e00700 1 -- 172.21.15.66:0/3137867745 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff16c00be70 con 0x7ff17c0fedd0 2024-02-28T16:20:06.703 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.713+0000 7ff173e00700 1 -- 172.21.15.66:0/3137867745 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff16c01c660 con 0x7ff17c0fedd0 2024-02-28T16:20:06.703 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.713+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff17c1104a0 con 0x7ff17c0fedd0 2024-02-28T16:20:06.704 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.714+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1580052e0 con 0x7ff17c0fedd0 2024-02-28T16:20:06.704 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.714+0000 7ff173e00700 1 -- 172.21.15.66:0/3137867745 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7ff16c004d30 con 0x7ff17c0fedd0 2024-02-28T16:20:06.705 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.715+0000 7ff173e00700 1 --2- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ff160036870 0x7ff160038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:06.705 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.715+0000 7ff173e00700 1 -- 172.21.15.66:0/3137867745 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff16c049a00 con 0x7ff17c0fedd0 2024-02-28T16:20:06.706 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.716+0000 7ff172a00700 1 --2- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ff160036870 0x7ff160038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:06.707 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.717+0000 7ff172a00700 1 --2- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ff160036870 0x7ff160038d20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff164009990 tx=0x7ff164006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:06.722 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:06.731+0000 7ff173e00700 1 -- 172.21.15.66:0/3137867745 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff16c025400 con 0x7ff17c0fedd0 2024-02-28T16:20:07.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:06 smithi001 ceph-mon[28991]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:07.022 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.032+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff158005590 con 0x7ff17c0fedd0 2024-02-28T16:20:07.023 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.033+0000 7ff173e00700 1 -- 172.21.15.66:0/3137867745 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7ff16c011370 con 0x7ff17c0fedd0 2024-02-28T16:20:07.024 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:07.024 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:07.026 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.035+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ff160036870 msgr2=0x7ff160038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:07.026 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.035+0000 7ff182400700 1 --2- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ff160036870 0x7ff160038d20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff164009990 tx=0x7ff164006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:07.027 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.035+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 msgr2=0x7ff17c10f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:07.027 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.035+0000 7ff182400700 1 --2- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 0x7ff17c10f0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009b80 tx=0x7ff16c009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:07.027 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.035+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 shutdown_connections 2024-02-28T16:20:07.027 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.035+0000 7ff182400700 1 --2- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7ff160036870 0x7ff160038d20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:07.027 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.035+0000 7ff182400700 1 --2- 172.21.15.66:0/3137867745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff17c0fedd0 0x7ff17c10f0c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:07.028 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.035+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 >> 172.21.15.66:0/3137867745 conn(0x7ff17c0fa5a0 msgr2=0x7ff17c0fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:07.028 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.036+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 shutdown_connections 2024-02-28T16:20:07.028 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:07.036+0000 7ff182400700 1 -- 172.21.15.66:0/3137867745 wait complete. 2024-02-28T16:20:07.029 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:08.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:07 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/3137867745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:08.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:08.748 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:09.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:08 smithi001 ceph-mon[28991]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:09.241 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:09.363 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:10.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:10 smithi001 ceph-mon[28991]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:11.404 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.413+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1961047597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 msgr2=0x7f0ddc0ff070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:11.405 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.413+0000 7f0ddbe00700 1 --2- 172.21.15.66:0/1961047597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 0x7f0ddc0ff070 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0048c0 tx=0x7f0dcc009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:11.405 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.413+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1961047597 shutdown_connections 2024-02-28T16:20:11.405 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.413+0000 7f0ddbe00700 1 --2- 172.21.15.66:0/1961047597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 0x7f0ddc0ff070 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:11.405 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.413+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1961047597 >> 172.21.15.66:0/1961047597 conn(0x7f0ddc0fa5a0 msgr2=0x7f0ddc0fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:11.405 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.414+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1961047597 shutdown_connections 2024-02-28T16:20:11.405 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.414+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1961047597 wait complete. 2024-02-28T16:20:11.406 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.415+0000 7f0ddbe00700 1 Processor -- start 2024-02-28T16:20:11.406 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.415+0000 7f0ddbe00700 1 -- start start 2024-02-28T16:20:11.406 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.415+0000 7f0ddbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 0x7f0ddc10eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:11.406 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.415+0000 7f0ddbe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc10f410 con 0x7f0ddc0fec60 2024-02-28T16:20:11.407 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.416+0000 7f0ddaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 0x7f0ddc10eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:11.407 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.416+0000 7f0ddaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 0x7f0ddc10eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53136/0 (socket says 172.21.15.66:53136) 2024-02-28T16:20:11.407 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.416+0000 7f0ddaa00700 1 -- 172.21.15.66:0/1798192744 learned_addr learned my addr 172.21.15.66:0/1798192744 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:11.407 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.417+0000 7f0ddaa00700 1 -- 172.21.15.66:0/1798192744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc005040 con 0x7f0ddc0fec60 2024-02-28T16:20:11.408 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.417+0000 7f0ddaa00700 1 --2- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 0x7f0ddc10eed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009680 tx=0x7f0dcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:11.408 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.418+0000 7f0dd2a00700 1 -- 172.21.15.66:0/1798192744 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dcc00bed0 con 0x7f0ddc0fec60 2024-02-28T16:20:11.408 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.418+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc10f610 con 0x7f0ddc0fec60 2024-02-28T16:20:11.408 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.418+0000 7f0dd2a00700 1 -- 172.21.15.66:0/1798192744 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dcc00b5c0 con 0x7f0ddc0fec60 2024-02-28T16:20:11.408 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.418+0000 7f0dd2a00700 1 -- 172.21.15.66:0/1798192744 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dcc01c5b0 con 0x7f0ddc0fec60 2024-02-28T16:20:11.408 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.418+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc1102b0 con 0x7f0ddc0fec60 2024-02-28T16:20:11.409 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.419+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ddc05f5c0 con 0x7f0ddc0fec60 2024-02-28T16:20:11.409 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.419+0000 7f0dd2a00700 1 -- 172.21.15.66:0/1798192744 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f0dcc004d30 con 0x7f0ddc0fec60 2024-02-28T16:20:11.410 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.420+0000 7f0dd2a00700 1 --2- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f0dc0036820 0x7f0dc0038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:11.410 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.420+0000 7f0dd2a00700 1 -- 172.21.15.66:0/1798192744 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0dcc04a050 con 0x7f0ddc0fec60 2024-02-28T16:20:11.410 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.420+0000 7f0dda000700 1 --2- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f0dc0036820 0x7f0dc0038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:11.412 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.421+0000 7f0dda000700 1 --2- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f0dc0036820 0x7f0dc0038cd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8009990 tx=0x7f0dc8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:11.424 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.434+0000 7f0dd2a00700 1 -- 172.21.15.66:0/1798192744 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0dcc011bd0 con 0x7f0ddc0fec60 2024-02-28T16:20:11.718 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.727+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ddc110e20 con 0x7f0ddc0fec60 2024-02-28T16:20:11.719 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.728+0000 7f0dd2a00700 1 -- 172.21.15.66:0/1798192744 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f0dcc01be90 con 0x7f0ddc0fec60 2024-02-28T16:20:11.719 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:11.719 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:11.721 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f0dc0036820 msgr2=0x7f0dc0038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:11.721 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 --2- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f0dc0036820 0x7f0dc0038cd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8009990 tx=0x7f0dc8006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:11.721 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 msgr2=0x7f0ddc10eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:11.722 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 --2- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 0x7f0ddc10eed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009680 tx=0x7f0dcc009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:11.722 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 shutdown_connections 2024-02-28T16:20:11.722 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 --2- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f0dc0036820 0x7f0dc0038cd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:11.722 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 --2- 172.21.15.66:0/1798192744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ddc0fec60 0x7f0ddc10eed0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:11.722 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 >> 172.21.15.66:0/1798192744 conn(0x7f0ddc0fa5a0 msgr2=0x7f0ddc0fc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:11.722 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 shutdown_connections 2024-02-28T16:20:11.722 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:11.731+0000 7f0ddbe00700 1 -- 172.21.15.66:0/1798192744 wait complete. 2024-02-28T16:20:11.723 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:12.963 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:12 smithi001 ceph-mon[28991]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:12.963 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:12 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/1798192744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:13.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:13.485 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:13.975 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:14.087 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:14.905 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:14 smithi001 ceph-mon[28991]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:16.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.074+0000 7fddf7200700 1 -- 172.21.15.66:0/4168044969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 msgr2=0x7fddf00ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:16.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.074+0000 7fddf7200700 1 --2- 172.21.15.66:0/4168044969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 0x7fddf00ff1e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdde00048c0 tx=0x7fdde0009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:16.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.074+0000 7fddf7200700 1 -- 172.21.15.66:0/4168044969 shutdown_connections 2024-02-28T16:20:16.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.074+0000 7fddf7200700 1 --2- 172.21.15.66:0/4168044969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 0x7fddf00ff1e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:16.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.074+0000 7fddf7200700 1 -- 172.21.15.66:0/4168044969 >> 172.21.15.66:0/4168044969 conn(0x7fddf00fa5a0 msgr2=0x7fddf00fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:16.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.074+0000 7fddf7200700 1 -- 172.21.15.66:0/4168044969 shutdown_connections 2024-02-28T16:20:16.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.075+0000 7fddf7200700 1 -- 172.21.15.66:0/4168044969 wait complete. 2024-02-28T16:20:16.066 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.076+0000 7fddf7200700 1 Processor -- start 2024-02-28T16:20:16.066 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.076+0000 7fddf7200700 1 -- start start 2024-02-28T16:20:16.066 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.076+0000 7fddf7200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 0x7fddf010f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:16.067 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.076+0000 7fddf7200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf010f650 con 0x7fddf00fedd0 2024-02-28T16:20:16.067 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.077+0000 7fddefe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 0x7fddf010f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:16.067 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.077+0000 7fddefe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 0x7fddf010f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53590/0 (socket says 172.21.15.66:53590) 2024-02-28T16:20:16.067 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.077+0000 7fddefe00700 1 -- 172.21.15.66:0/3607320930 learned_addr learned my addr 172.21.15.66:0/3607320930 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:16.067 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.077+0000 7fddefe00700 1 -- 172.21.15.66:0/3607320930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde0005040 con 0x7fddf00fedd0 2024-02-28T16:20:16.068 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.078+0000 7fddefe00700 1 --2- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 0x7fddf010f110 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009b80 tx=0x7fdde0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:16.068 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.078+0000 7fddecc00700 1 -- 172.21.15.66:0/3607320930 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdde000b5c0 con 0x7fddf00fedd0 2024-02-28T16:20:16.068 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.078+0000 7fddecc00700 1 -- 172.21.15.66:0/3607320930 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdde000be70 con 0x7fddf00fedd0 2024-02-28T16:20:16.068 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.078+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf010f850 con 0x7fddf00fedd0 2024-02-28T16:20:16.068 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.078+0000 7fddecc00700 1 -- 172.21.15.66:0/3607320930 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdde001c660 con 0x7fddf00fedd0 2024-02-28T16:20:16.069 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.078+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf01104f0 con 0x7fddf00fedd0 2024-02-28T16:20:16.069 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.079+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf005f5c0 con 0x7fddf00fedd0 2024-02-28T16:20:16.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.079+0000 7fddecc00700 1 -- 172.21.15.66:0/3607320930 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fdde0004d30 con 0x7fddf00fedd0 2024-02-28T16:20:16.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.080+0000 7fddecc00700 1 --2- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fddd4036870 0x7fddd4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:16.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.080+0000 7fddecc00700 1 -- 172.21.15.66:0/3607320930 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdde0049ae0 con 0x7fddf00fedd0 2024-02-28T16:20:16.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.080+0000 7fdde7e00700 1 --2- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fddd4036870 0x7fddd4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:16.071 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.081+0000 7fdde7e00700 1 --2- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fddd4036870 0x7fddd4038d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009990 tx=0x7fddd8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:16.078 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.087+0000 7fddecc00700 1 -- 172.21.15.66:0/3607320930 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdde0025400 con 0x7fddf00fedd0 2024-02-28T16:20:16.376 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.385+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fddf0111040 con 0x7fddf00fedd0 2024-02-28T16:20:16.377 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.386+0000 7fddecc00700 1 -- 172.21.15.66:0/3607320930 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fdde0011370 con 0x7fddf00fedd0 2024-02-28T16:20:16.377 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:16.377 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:16.379 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fddd4036870 msgr2=0x7fddd4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 --2- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fddd4036870 0x7fddd4038d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009990 tx=0x7fddd8006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 msgr2=0x7fddf010f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 --2- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 0x7fddf010f110 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009b80 tx=0x7fdde0009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 shutdown_connections 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 --2- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fddd4036870 0x7fddd4038d20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 --2- 172.21.15.66:0/3607320930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddf00fedd0 0x7fddf010f110 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 >> 172.21.15.66:0/3607320930 conn(0x7fddf00fa5a0 msgr2=0x7fddf00fc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.389+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 shutdown_connections 2024-02-28T16:20:16.380 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:16.390+0000 7fddf7200700 1 -- 172.21.15.66:0/3607320930 wait complete. 2024-02-28T16:20:16.381 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:17.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:16 smithi001 ceph-mon[28991]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:17.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:16 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/3607320930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:17.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:16 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:17.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:16 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:17.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:16 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:17.986 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:17 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-28T16:20:18.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:18.091 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:18.578 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:18.694 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:18.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:18 smithi001 ceph-mon[28991]: from='mgr.14162 172.21.15.1:0/608125996' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-28T16:20:18.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:18 smithi001 ceph-mon[28991]: mgrmap e14: smithi001.rmthtx(active, since 82s) 2024-02-28T16:20:20.716 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.725+0000 7f944f200700 1 -- 172.21.15.66:0/2133776567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 msgr2=0x7f94500ff010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:20.716 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.725+0000 7f944f200700 1 --2- 172.21.15.66:0/2133776567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 0x7f94500ff010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f94400048c0 tx=0x7f9440009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:20.716 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.726+0000 7f944f200700 1 -- 172.21.15.66:0/2133776567 shutdown_connections 2024-02-28T16:20:20.716 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.726+0000 7f944f200700 1 --2- 172.21.15.66:0/2133776567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 0x7f94500ff010 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:20.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.726+0000 7f944f200700 1 -- 172.21.15.66:0/2133776567 >> 172.21.15.66:0/2133776567 conn(0x7f94500fa5a0 msgr2=0x7f94500fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:20.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.726+0000 7f944f200700 1 -- 172.21.15.66:0/2133776567 shutdown_connections 2024-02-28T16:20:20.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.726+0000 7f944f200700 1 -- 172.21.15.66:0/2133776567 wait complete. 2024-02-28T16:20:20.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.727+0000 7f944f200700 1 Processor -- start 2024-02-28T16:20:20.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.727+0000 7f944f200700 1 -- start start 2024-02-28T16:20:20.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.727+0000 7f944f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 0x7f945010ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:20.717 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.727+0000 7f944f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945010f460 con 0x7f94500fec00 2024-02-28T16:20:20.718 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.728+0000 7f944de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 0x7f945010ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:20.718 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.728+0000 7f944de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 0x7f945010ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53602/0 (socket says 172.21.15.66:53602) 2024-02-28T16:20:20.718 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.728+0000 7f944de00700 1 -- 172.21.15.66:0/2434024409 learned_addr learned my addr 172.21.15.66:0/2434024409 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:20.718 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.728+0000 7f944de00700 1 -- 172.21.15.66:0/2434024409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9440005040 con 0x7f94500fec00 2024-02-28T16:20:20.719 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.728+0000 7f944de00700 1 --2- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 0x7f945010ef20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9440009b80 tx=0x7f9440009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:20.719 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.729+0000 7f9446000700 1 -- 172.21.15.66:0/2434024409 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f944000b5c0 con 0x7f94500fec00 2024-02-28T16:20:20.719 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.729+0000 7f9446000700 1 -- 172.21.15.66:0/2434024409 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f944000be70 con 0x7f94500fec00 2024-02-28T16:20:20.719 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.729+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945010f660 con 0x7f94500fec00 2024-02-28T16:20:20.719 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.729+0000 7f9446000700 1 -- 172.21.15.66:0/2434024409 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f944001c660 con 0x7f94500fec00 2024-02-28T16:20:20.719 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.729+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9450110300 con 0x7f94500fec00 2024-02-28T16:20:20.720 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.730+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945005f5c0 con 0x7f94500fec00 2024-02-28T16:20:20.720 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.730+0000 7f9446000700 1 -- 172.21.15.66:0/2434024409 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 43437+0+0 (secure 0 0 0) 0x7f9440004d30 con 0x7f94500fec00 2024-02-28T16:20:20.721 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.731+0000 7f9446000700 1 --2- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f94340368c0 0x7f9434038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:20.721 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.731+0000 7f9446000700 1 -- 172.21.15.66:0/2434024409 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9440049b50 con 0x7f94500fec00 2024-02-28T16:20:20.721 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.731+0000 7f944d400700 1 -- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f94340368c0 msgr2=0x7f9434038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/701199259 2024-02-28T16:20:20.721 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.731+0000 7f944d400700 1 --2- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f94340368c0 0x7f9434038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-28T16:20:20.735 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.745+0000 7f9446000700 1 -- 172.21.15.66:0/2434024409 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9440025400 con 0x7f94500fec00 2024-02-28T16:20:20.922 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.932+0000 7f944d400700 1 -- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f94340368c0 msgr2=0x7f9434038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/701199259 2024-02-28T16:20:20.923 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:20.932+0000 7f944d400700 1 --2- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f94340368c0 0x7f9434038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-28T16:20:21.036 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.045+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9450110e50 con 0x7f94500fec00 2024-02-28T16:20:21.037 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.046+0000 7f9446000700 1 -- 172.21.15.66:0/2434024409 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f9440023300 con 0x7f94500fec00 2024-02-28T16:20:21.037 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:21.037 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:21.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f94340368c0 msgr2=0x7f9434038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-28T16:20:21.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 --2- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f94340368c0 0x7f9434038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 msgr2=0x7f945010ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 --2- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 0x7f945010ef20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9440009b80 tx=0x7f9440009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 shutdown_connections 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 --2- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7f94340368c0 0x7f9434038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 --2- 172.21.15.66:0/2434024409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94500fec00 0x7f945010ef20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 >> 172.21.15.66:0/2434024409 conn(0x7f94500fa5a0 msgr2=0x7f94500fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 shutdown_connections 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:21.050+0000 7f944f200700 1 -- 172.21.15.66:0/2434024409 wait complete. 2024-02-28T16:20:21.041 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:21.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:21 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/2434024409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:22.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:22.836 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:23.323 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:23.427 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:25.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.479+0000 7fdc67200700 1 -- 172.21.15.66:0/3497305246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 msgr2=0x7fdc601035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:25.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.479+0000 7fdc67200700 1 --2- 172.21.15.66:0/3497305246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 0x7fdc601035e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdc500048c0 tx=0x7fdc50009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:25.471 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.480+0000 7fdc67200700 1 -- 172.21.15.66:0/3497305246 shutdown_connections 2024-02-28T16:20:25.471 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.480+0000 7fdc67200700 1 --2- 172.21.15.66:0/3497305246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 0x7fdc601035e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:25.471 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.480+0000 7fdc67200700 1 -- 172.21.15.66:0/3497305246 >> 172.21.15.66:0/3497305246 conn(0x7fdc600722d0 msgr2=0x7fdc600746e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:25.471 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.480+0000 7fdc67200700 1 -- 172.21.15.66:0/3497305246 shutdown_connections 2024-02-28T16:20:25.471 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.480+0000 7fdc67200700 1 -- 172.21.15.66:0/3497305246 wait complete. 2024-02-28T16:20:25.471 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.481+0000 7fdc67200700 1 Processor -- start 2024-02-28T16:20:25.471 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.481+0000 7fdc67200700 1 -- start start 2024-02-28T16:20:25.472 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.482+0000 7fdc67200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 0x7fdc601134f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:25.472 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.482+0000 7fdc67200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc60113a30 con 0x7fdc601031d0 2024-02-28T16:20:25.472 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.482+0000 7fdc5fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 0x7fdc601134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:25.472 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.482+0000 7fdc5fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 0x7fdc601134f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:47158/0 (socket says 172.21.15.66:47158) 2024-02-28T16:20:25.472 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.482+0000 7fdc5fe00700 1 -- 172.21.15.66:0/1946804301 learned_addr learned my addr 172.21.15.66:0/1946804301 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:25.473 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.483+0000 7fdc5fe00700 1 -- 172.21.15.66:0/1946804301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc50005040 con 0x7fdc601031d0 2024-02-28T16:20:25.473 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.483+0000 7fdc5fe00700 1 --2- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 0x7fdc601134f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009680 tx=0x7fdc50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:25.474 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.484+0000 7fdc57e00700 1 -- 172.21.15.66:0/1946804301 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc5000bed0 con 0x7fdc601031d0 2024-02-28T16:20:25.474 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.484+0000 7fdc57e00700 1 -- 172.21.15.66:0/1946804301 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc5000b5c0 con 0x7fdc601031d0 2024-02-28T16:20:25.474 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.484+0000 7fdc57e00700 1 -- 172.21.15.66:0/1946804301 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc5001c5b0 con 0x7fdc601031d0 2024-02-28T16:20:25.475 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.484+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc60113c30 con 0x7fdc601031d0 2024-02-28T16:20:25.475 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.484+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc601148d0 con 0x7fdc601031d0 2024-02-28T16:20:25.476 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.485+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6005f5c0 con 0x7fdc601031d0 2024-02-28T16:20:25.476 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.486+0000 7fdc57e00700 1 -- 172.21.15.66:0/1946804301 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 43437+0+0 (secure 0 0 0) 0x7fdc50013a80 con 0x7fdc601031d0 2024-02-28T16:20:25.476 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.486+0000 7fdc57e00700 1 --2- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fdc44036870 0x7fdc44038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:25.476 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.486+0000 7fdc57e00700 1 -- 172.21.15.66:0/1946804301 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdc50049b10 con 0x7fdc601031d0 2024-02-28T16:20:25.476 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.486+0000 7fdc5f400700 1 -- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fdc44036870 msgr2=0x7fdc44038d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/701199259 2024-02-28T16:20:25.477 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.486+0000 7fdc5f400700 1 --2- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fdc44036870 0x7fdc44038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-28T16:20:25.491 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.501+0000 7fdc57e00700 1 -- 172.21.15.66:0/1946804301 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdc50011940 con 0x7fdc601031d0 2024-02-28T16:20:25.677 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.687+0000 7fdc5f400700 1 -- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fdc44036870 msgr2=0x7fdc44038d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/701199259 2024-02-28T16:20:25.677 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.687+0000 7fdc5f400700 1 --2- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fdc44036870 0x7fdc44038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-28T16:20:25.799 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.808+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc60115420 con 0x7fdc601031d0 2024-02-28T16:20:25.800 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.810+0000 7fdc57e00700 1 -- 172.21.15.66:0/1946804301 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fdc50011370 con 0x7fdc601031d0 2024-02-28T16:20:25.800 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:25.800 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:25.803 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.812+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fdc44036870 msgr2=0x7fdc44038d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-28T16:20:25.803 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.812+0000 7fdc67200700 1 --2- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fdc44036870 0x7fdc44038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:25.803 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.812+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 msgr2=0x7fdc601134f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:25.803 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.812+0000 7fdc67200700 1 --2- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 0x7fdc601134f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009680 tx=0x7fdc50009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:25.803 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.813+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 shutdown_connections 2024-02-28T16:20:25.804 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.813+0000 7fdc67200700 1 --2- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:6800/701199259,v1:172.21.15.1:6801/701199259] conn(0x7fdc44036870 0x7fdc44038d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:25.804 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.813+0000 7fdc67200700 1 --2- 172.21.15.66:0/1946804301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601031d0 0x7fdc601134f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:25.804 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.813+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 >> 172.21.15.66:0/1946804301 conn(0x7fdc600722d0 msgr2=0x7fdc60073070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:25.804 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.813+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 shutdown_connections 2024-02-28T16:20:25.804 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:25.813+0000 7fdc67200700 1 -- 172.21.15.66:0/1946804301 wait complete. 2024-02-28T16:20:25.804 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:25 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/1946804301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: Active manager daemon smithi001.rmthtx restarted 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: Activating manager daemon smithi001.rmthtx 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: osdmap e5: 0 total, 0 up, 0 in 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: mgrmap e15: smithi001.rmthtx(active, starting, since 0.00348325s) 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: Manager daemon smithi001.rmthtx is now available 2024-02-28T16:20:27.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:20:27.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:20:27.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:27.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:20:27.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:20:27.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/trash_purge_schedule"}]: dispatch 2024-02-28T16:20:27.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:26 smithi001 ceph-mon[28991]: [28/Feb/2024:16:20:26] ENGINE Bus STARTING 2024-02-28T16:20:27.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:27.524 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:27.976 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:28.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:27 smithi001 ceph-mon[28991]: [28/Feb/2024:16:20:26] ENGINE Serving on https://172.21.15.1:7150 2024-02-28T16:20:28.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:27 smithi001 ceph-mon[28991]: [28/Feb/2024:16:20:26] ENGINE Bus STARTED 2024-02-28T16:20:28.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:27 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:28.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:27 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:28.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:27 smithi001 ceph-mon[28991]: mgrmap e16: smithi001.rmthtx(active, since 1.01262s) 2024-02-28T16:20:28.077 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:29.058 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:29 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.217+0000 7fc707c00700 1 -- 172.21.15.66:0/1729029945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 msgr2=0x7fc7000ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.217+0000 7fc707c00700 1 --2- 172.21.15.66:0/1729029945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 0x7fc7000ffaf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40048c0 tx=0x7fc6f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.218+0000 7fc707c00700 1 -- 172.21.15.66:0/1729029945 shutdown_connections 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.218+0000 7fc707c00700 1 --2- 172.21.15.66:0/1729029945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 0x7fc7000ffaf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.218+0000 7fc707c00700 1 -- 172.21.15.66:0/1729029945 >> 172.21.15.66:0/1729029945 conn(0x7fc7000f7190 msgr2=0x7fc7000f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.218+0000 7fc707c00700 1 -- 172.21.15.66:0/1729029945 shutdown_connections 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.218+0000 7fc707c00700 1 -- 172.21.15.66:0/1729029945 wait complete. 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.219+0000 7fc707c00700 1 Processor -- start 2024-02-28T16:20:30.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.219+0000 7fc707c00700 1 -- start start 2024-02-28T16:20:30.210 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.219+0000 7fc707c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 0x7fc7000fd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:30.210 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.219+0000 7fc707c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000fb620 con 0x7fc7000fd710 2024-02-28T16:20:30.210 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.220+0000 7fc705200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 0x7fc7000fd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:30.210 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.220+0000 7fc705200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 0x7fc7000fd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:47182/0 (socket says 172.21.15.66:47182) 2024-02-28T16:20:30.210 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.220+0000 7fc705200700 1 -- 172.21.15.66:0/730413487 learned_addr learned my addr 172.21.15.66:0/730413487 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:30.211 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.220+0000 7fc705200700 1 -- 172.21.15.66:0/730413487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f4005040 con 0x7fc7000fd710 2024-02-28T16:20:30.211 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.220+0000 7fc705200700 1 --2- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 0x7fc7000fd1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009680 tx=0x7fc6f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:30.211 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.221+0000 7fc6fcc00700 1 -- 172.21.15.66:0/730413487 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f400b5c0 con 0x7fc7000fd710 2024-02-28T16:20:30.211 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.221+0000 7fc6fcc00700 1 -- 172.21.15.66:0/730413487 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6f4004e60 con 0x7fc7000fd710 2024-02-28T16:20:30.211 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.221+0000 7fc6fcc00700 1 -- 172.21.15.66:0/730413487 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f401c590 con 0x7fc7000fd710 2024-02-28T16:20:30.212 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.221+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7000fb820 con 0x7fc7000fd710 2024-02-28T16:20:30.212 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.221+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7000fbd10 con 0x7fc7000fd710 2024-02-28T16:20:30.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.223+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7001069a0 con 0x7fc7000fd710 2024-02-28T16:20:30.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.223+0000 7fc6fcc00700 1 -- 172.21.15.66:0/730413487 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7fc6f4013a80 con 0x7fc7000fd710 2024-02-28T16:20:30.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.223+0000 7fc6fcc00700 1 --2- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fc6e80369b0 0x7fc6e8038e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:30.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.223+0000 7fc6fcc00700 1 -- 172.21.15.66:0/730413487 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fc6f4049d90 con 0x7fc7000fd710 2024-02-28T16:20:30.214 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.224+0000 7fc6ffe00700 1 --2- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fc6e80369b0 0x7fc6e8038e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:30.215 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.225+0000 7fc6ffe00700 1 --2- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fc6e80369b0 0x7fc6e8038e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009990 tx=0x7fc6ec006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:30.222 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.232+0000 7fc6fcc00700 1 -- 172.21.15.66:0/730413487 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc6f404f020 con 0x7fc7000fd710 2024-02-28T16:20:30.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:30 smithi001 ceph-mon[28991]: mgrmap e17: smithi001.rmthtx(active, since 2s) 2024-02-28T16:20:30.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:30.543 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.552+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc70004bed0 con 0x7fc7000fd710 2024-02-28T16:20:30.544 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.554+0000 7fc6fcc00700 1 -- 172.21.15.66:0/730413487 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fc6f40119c0 con 0x7fc7000fd710 2024-02-28T16:20:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:30.544 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.556+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fc6e80369b0 msgr2=0x7fc6e8038e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.556+0000 7fc707c00700 1 --2- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fc6e80369b0 0x7fc6e8038e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009990 tx=0x7fc6ec006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.556+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 msgr2=0x7fc7000fd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.556+0000 7fc707c00700 1 --2- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 0x7fc7000fd1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009680 tx=0x7fc6f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.556+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 shutdown_connections 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.556+0000 7fc707c00700 1 --2- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fc6e80369b0 0x7fc6e8038e60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.556+0000 7fc707c00700 1 --2- 172.21.15.66:0/730413487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7000fd710 0x7fc7000fd1a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.556+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 >> 172.21.15.66:0/730413487 conn(0x7fc7000f7190 msgr2=0x7fc7000f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.557+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 shutdown_connections 2024-02-28T16:20:30.547 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:30.557+0000 7fc707c00700 1 -- 172.21.15.66:0/730413487 wait complete. 2024-02-28T16:20:30.548 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:31.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:31 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/730413487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:33.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:33.154 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:33.653 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:33.763 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-28T16:20:35.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:34 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:35.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:34 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:35.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:34 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi001", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:20:36.360 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.369+0000 7eff46200700 1 -- 172.21.15.66:0/314193881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 msgr2=0x7eff400fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:36.360 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.369+0000 7eff46200700 1 --2- 172.21.15.66:0/314193881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 0x7eff400fb910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff300048c0 tx=0x7eff30009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:36.360 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.369+0000 7eff46200700 1 -- 172.21.15.66:0/314193881 shutdown_connections 2024-02-28T16:20:36.360 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.369+0000 7eff46200700 1 --2- 172.21.15.66:0/314193881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 0x7eff400fb910 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:36.360 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.369+0000 7eff46200700 1 -- 172.21.15.66:0/314193881 >> 172.21.15.66:0/314193881 conn(0x7eff400f7190 msgr2=0x7eff400f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:36.360 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.369+0000 7eff46200700 1 -- 172.21.15.66:0/314193881 shutdown_connections 2024-02-28T16:20:36.360 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.369+0000 7eff46200700 1 -- 172.21.15.66:0/314193881 wait complete. 2024-02-28T16:20:36.361 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.370+0000 7eff46200700 1 Processor -- start 2024-02-28T16:20:36.361 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.371+0000 7eff46200700 1 -- start start 2024-02-28T16:20:36.361 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.371+0000 7eff46200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 0x7eff4010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:36.361 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.371+0000 7eff46200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4010f5c0 con 0x7eff400fb500 2024-02-28T16:20:36.362 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.371+0000 7eff3f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 0x7eff4010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:36.362 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.371+0000 7eff3f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 0x7eff4010f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:51912/0 (socket says 172.21.15.66:51912) 2024-02-28T16:20:36.362 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.371+0000 7eff3f400700 1 -- 172.21.15.66:0/2996761191 learned_addr learned my addr 172.21.15.66:0/2996761191 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:36.362 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.372+0000 7eff3f400700 1 -- 172.21.15.66:0/2996761191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30005040 con 0x7eff400fb500 2024-02-28T16:20:36.362 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.372+0000 7eff3f400700 1 --2- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 0x7eff4010f080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7eff30009b80 tx=0x7eff30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:36.363 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.373+0000 7eff37400700 1 -- 172.21.15.66:0/2996761191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3000b5c0 con 0x7eff400fb500 2024-02-28T16:20:36.363 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.373+0000 7eff37400700 1 -- 172.21.15.66:0/2996761191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff3000be70 con 0x7eff400fb500 2024-02-28T16:20:36.363 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.373+0000 7eff37400700 1 -- 172.21.15.66:0/2996761191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3001c660 con 0x7eff400fb500 2024-02-28T16:20:36.363 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.373+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4010f7c0 con 0x7eff400fb500 2024-02-28T16:20:36.363 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.373+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff40110460 con 0x7eff400fb500 2024-02-28T16:20:36.364 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.374+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4005f5c0 con 0x7eff400fb500 2024-02-28T16:20:36.364 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.374+0000 7eff37400700 1 -- 172.21.15.66:0/2996761191 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7eff30004d30 con 0x7eff400fb500 2024-02-28T16:20:36.365 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.375+0000 7eff37400700 1 --2- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7eff240369b0 0x7eff24038e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:36.365 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.375+0000 7eff37400700 1 -- 172.21.15.66:0/2996761191 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7eff30049ca0 con 0x7eff400fb500 2024-02-28T16:20:36.365 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.375+0000 7eff3ea00700 1 --2- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7eff240369b0 0x7eff24038e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:36.366 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.376+0000 7eff3ea00700 1 --2- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7eff240369b0 0x7eff24038e60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009990 tx=0x7eff2c006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:36.371 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.381+0000 7eff37400700 1 -- 172.21.15.66:0/2996761191 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7eff30025400 con 0x7eff400fb500 2024-02-28T16:20:36.671 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.681+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff40110fb0 con 0x7eff400fb500 2024-02-28T16:20:36.672 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.682+0000 7eff37400700 1 -- 172.21.15.66:0/2996761191 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7eff30011370 con 0x7eff400fb500 2024-02-28T16:20:36.672 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:36.672 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:36.675 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7eff240369b0 msgr2=0x7eff24038e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:36.675 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 --2- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7eff240369b0 0x7eff24038e60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009990 tx=0x7eff2c006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:36.675 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 msgr2=0x7eff4010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:36.675 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 --2- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 0x7eff4010f080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7eff30009b80 tx=0x7eff30009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:36.676 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 shutdown_connections 2024-02-28T16:20:36.676 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 --2- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7eff240369b0 0x7eff24038e60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:36.676 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 --2- 172.21.15.66:0/2996761191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400fb500 0x7eff4010f080 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:36.676 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 >> 172.21.15.66:0/2996761191 conn(0x7eff400f7190 msgr2=0x7eff400f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:36.676 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 shutdown_connections 2024-02-28T16:20:36.676 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:36.685+0000 7eff46200700 1 -- 172.21.15.66:0/2996761191 wait complete. 2024-02-28T16:20:36.677 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:37.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:37.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:37.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:37.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:37.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi066", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:20:37.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:20:37.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:20:37.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:20:37.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:20:37.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:20:37.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:20:37.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:36 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/2996761191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:38.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:37 smithi001 ceph-mon[28991]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:20:38.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:37 smithi001 ceph-mon[28991]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:20:38.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:38.465 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:38.916 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:20:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi066", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi066", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-28T16:20:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:20:41.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:40 smithi001 ceph-mon[28991]: Deploying daemon crash.smithi066 on smithi066 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.209+0000 7fec71600700 1 -- 172.21.15.66:0/4162592334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 msgr2=0x7fec6c1051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.209+0000 7fec71600700 1 --2- 172.21.15.66:0/4162592334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 0x7fec6c1051b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0048c0 tx=0x7fec5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.209+0000 7fec71600700 1 -- 172.21.15.66:0/4162592334 shutdown_connections 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.209+0000 7fec71600700 1 --2- 172.21.15.66:0/4162592334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 0x7fec6c1051b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.209+0000 7fec71600700 1 -- 172.21.15.66:0/4162592334 >> 172.21.15.66:0/4162592334 conn(0x7fec6c0fa950 msgr2=0x7fec6c0fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.210+0000 7fec71600700 1 -- 172.21.15.66:0/4162592334 shutdown_connections 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.210+0000 7fec71600700 1 -- 172.21.15.66:0/4162592334 wait complete. 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.210+0000 7fec71600700 1 Processor -- start 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.211+0000 7fec71600700 1 -- start start 2024-02-28T16:20:41.278 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.211+0000 7fec71600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 0x7fec6c115150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.211+0000 7fec71600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c115690 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.211+0000 7fec6aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 0x7fec6c115150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.212+0000 7fec6aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 0x7fec6c115150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:51918/0 (socket says 172.21.15.66:51918) 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.212+0000 7fec6aa00700 1 -- 172.21.15.66:0/3074830227 learned_addr learned my addr 172.21.15.66:0/3074830227 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.212+0000 7fec6aa00700 1 -- 172.21.15.66:0/3074830227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c005040 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.213+0000 7fec6aa00700 1 --2- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 0x7fec6c115150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fec5c009330 tx=0x7fec5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.213+0000 7fec62a00700 1 -- 172.21.15.66:0/3074830227 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec5c00be50 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.213+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6c115890 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.213+0000 7fec62a00700 1 -- 172.21.15.66:0/3074830227 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec5c009630 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.213+0000 7fec62a00700 1 -- 172.21.15.66:0/3074830227 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec5c01c3f0 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.213+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6c115d80 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.214+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6c05f5c0 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.215+0000 7fec62a00700 1 -- 172.21.15.66:0/3074830227 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7fec5c0097a0 con 0x7fec6c104de0 2024-02-28T16:20:41.279 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.215+0000 7fec62a00700 1 --2- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec500369b0 0x7fec50038e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:41.280 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.215+0000 7fec62a00700 1 -- 172.21.15.66:0/3074830227 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fec5c049c50 con 0x7fec6c104de0 2024-02-28T16:20:41.280 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.216+0000 7fec6a000700 1 --2- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec500369b0 0x7fec50038e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:41.280 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.217+0000 7fec6a000700 1 --2- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec500369b0 0x7fec50038e60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fec58009990 tx=0x7fec58006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:41.280 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.230+0000 7fec62a00700 1 -- 172.21.15.66:0/3074830227 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fec5c011960 con 0x7fec6c104de0 2024-02-28T16:20:41.518 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.528+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec6c110900 con 0x7fec6c104de0 2024-02-28T16:20:41.519 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.529+0000 7fec62a00700 1 -- 172.21.15.66:0/3074830227 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fec5c011370 con 0x7fec6c104de0 2024-02-28T16:20:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:41.520 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:41.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.532+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec500369b0 msgr2=0x7fec50038e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:41.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.532+0000 7fec71600700 1 --2- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec500369b0 0x7fec50038e60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fec58009990 tx=0x7fec58006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:41.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.532+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 msgr2=0x7fec6c115150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:41.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.532+0000 7fec71600700 1 --2- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 0x7fec6c115150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fec5c009330 tx=0x7fec5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:41.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.532+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 shutdown_connections 2024-02-28T16:20:41.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.533+0000 7fec71600700 1 --2- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec500369b0 0x7fec50038e60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:41.524 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.533+0000 7fec71600700 1 --2- 172.21.15.66:0/3074830227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec6c104de0 0x7fec6c115150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:41.524 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.533+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 >> 172.21.15.66:0/3074830227 conn(0x7fec6c0fa950 msgr2=0x7fec6c0fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:41.524 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.533+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 shutdown_connections 2024-02-28T16:20:41.524 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:41.533+0000 7fec71600700 1 -- 172.21.15.66:0/3074830227 wait complete. 2024-02-28T16:20:41.524 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:42.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:20:42.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:41 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/3074830227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:43.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:43.890 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:44.654 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:20:46.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:45 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:46.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:45 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:46.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:45 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:46.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:45 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:46.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:45 smithi001 ceph-mon[28991]: Deploying daemon node-exporter.smithi066 on smithi066 2024-02-28T16:20:46.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.863+0000 7f1717400700 1 -- 172.21.15.66:0/3213830659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 msgr2=0x7f1710102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:46.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.863+0000 7f1717400700 1 --2- 172.21.15.66:0/3213830659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 0x7f1710102f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f17040048c0 tx=0x7f1704009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:46.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.864+0000 7f1717400700 1 -- 172.21.15.66:0/3213830659 shutdown_connections 2024-02-28T16:20:46.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.864+0000 7f1717400700 1 --2- 172.21.15.66:0/3213830659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 0x7f1710102f80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:46.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.864+0000 7f1717400700 1 -- 172.21.15.66:0/3213830659 >> 172.21.15.66:0/3213830659 conn(0x7f17100f8760 msgr2=0x7f17100fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:46.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.864+0000 7f1717400700 1 -- 172.21.15.66:0/3213830659 shutdown_connections 2024-02-28T16:20:46.856 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.864+0000 7f1717400700 1 -- 172.21.15.66:0/3213830659 wait complete. 2024-02-28T16:20:46.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.865+0000 7f1717400700 1 Processor -- start 2024-02-28T16:20:46.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.865+0000 7f1717400700 1 -- start start 2024-02-28T16:20:46.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.865+0000 7f1717400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 0x7f1710112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:46.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.865+0000 7f1717400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1710113480 con 0x7f1710102bb0 2024-02-28T16:20:46.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.866+0000 7f1714a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 0x7f1710112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:46.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.866+0000 7f1714a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 0x7f1710112f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:47176/0 (socket says 172.21.15.66:47176) 2024-02-28T16:20:46.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.866+0000 7f1714a00700 1 -- 172.21.15.66:0/3681759143 learned_addr learned my addr 172.21.15.66:0/3681759143 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:46.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.867+0000 7f1714a00700 1 -- 172.21.15.66:0/3681759143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704005040 con 0x7f1710102bb0 2024-02-28T16:20:46.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.867+0000 7f1714a00700 1 --2- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 0x7f1710112f40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1704009330 tx=0x7f1704009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:46.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.867+0000 7f170cc00700 1 -- 172.21.15.66:0/3681759143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f170400be50 con 0x7f1710102bb0 2024-02-28T16:20:46.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.867+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1710113680 con 0x7f1710102bb0 2024-02-28T16:20:46.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.867+0000 7f170cc00700 1 -- 172.21.15.66:0/3681759143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1704009630 con 0x7f1710102bb0 2024-02-28T16:20:46.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.867+0000 7f170cc00700 1 -- 172.21.15.66:0/3681759143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f170401c3f0 con 0x7f1710102bb0 2024-02-28T16:20:46.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.868+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1710113b70 con 0x7f1710102bb0 2024-02-28T16:20:46.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.869+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171005f5c0 con 0x7f1710102bb0 2024-02-28T16:20:46.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.869+0000 7f170cc00700 1 -- 172.21.15.66:0/3681759143 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7f17040097a0 con 0x7f1710102bb0 2024-02-28T16:20:46.860 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.870+0000 7f170cc00700 1 --2- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16f80369b0 0x7f16f8038e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:46.860 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.870+0000 7f170cc00700 1 -- 172.21.15.66:0/3681759143 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1704049b90 con 0x7f1710102bb0 2024-02-28T16:20:46.860 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.870+0000 7f170fe00700 1 --2- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16f80369b0 0x7f16f8038e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:46.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.871+0000 7f170fe00700 1 --2- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16f80369b0 0x7f16f8038e60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009990 tx=0x7f16fc006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:46.876 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:46.885+0000 7f170cc00700 1 -- 172.21.15.66:0/3681759143 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1704011940 con 0x7f1710102bb0 2024-02-28T16:20:47.208 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.218+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1710106ed0 con 0x7f1710102bb0 2024-02-28T16:20:47.209 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.219+0000 7f170cc00700 1 -- 172.21.15.66:0/3681759143 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f1704011370 con 0x7f1710102bb0 2024-02-28T16:20:47.209 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:47.209 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:47.212 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16f80369b0 msgr2=0x7f16f8038e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:47.212 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 --2- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16f80369b0 0x7f16f8038e60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009990 tx=0x7f16fc006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:47.212 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 msgr2=0x7f1710112f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:47.212 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 --2- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 0x7f1710112f40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1704009330 tx=0x7f1704009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:47.212 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 shutdown_connections 2024-02-28T16:20:47.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 --2- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16f80369b0 0x7f16f8038e60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:47.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 --2- 172.21.15.66:0/3681759143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1710102bb0 0x7f1710112f40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:47.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 >> 172.21.15.66:0/3681759143 conn(0x7f17100f8760 msgr2=0x7f17100fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:47.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 shutdown_connections 2024-02-28T16:20:47.213 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:47.222+0000 7f1717400700 1 -- 172.21.15.66:0/3681759143 wait complete. 2024-02-28T16:20:47.213 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:47.683 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:47 smithi001 ceph-mon[28991]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:47.683 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:47.683 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:47 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/3681759143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:49.330 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:49.330 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:49.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:49 smithi001 ceph-mon[28991]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:49.850 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:20:51.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:51.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:51.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:51.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:51.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:51.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:20:51.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-28T16:20:51.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:20:51.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:20:52.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:52 smithi001 ceph-mon[28991]: Deploying daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:20:52.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:52 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:52.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.847+0000 7f2a41e00700 1 -- 172.21.15.66:0/3917641542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 msgr2=0x7f2a3c102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:52.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.847+0000 7f2a41e00700 1 --2- 172.21.15.66:0/3917641542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 0x7f2a3c102de0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2a300048c0 tx=0x7f2a30009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:52.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.847+0000 7f2a41e00700 1 -- 172.21.15.66:0/3917641542 shutdown_connections 2024-02-28T16:20:52.839 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.847+0000 7f2a41e00700 1 --2- 172.21.15.66:0/3917641542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 0x7f2a3c102de0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:52.839 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.847+0000 7f2a41e00700 1 -- 172.21.15.66:0/3917641542 >> 172.21.15.66:0/3917641542 conn(0x7f2a3c0f8550 msgr2=0x7f2a3c0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:52.839 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.847+0000 7f2a41e00700 1 -- 172.21.15.66:0/3917641542 shutdown_connections 2024-02-28T16:20:52.839 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.847+0000 7f2a41e00700 1 -- 172.21.15.66:0/3917641542 wait complete. 2024-02-28T16:20:52.839 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.848+0000 7f2a41e00700 1 Processor -- start 2024-02-28T16:20:52.842 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.848+0000 7f2a41e00700 1 -- start start 2024-02-28T16:20:52.842 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.849+0000 7f2a41e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 0x7f2a3c114f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:52.842 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.849+0000 7f2a41e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c115480 con 0x7f2a3c102a10 2024-02-28T16:20:52.843 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.849+0000 7f2a40a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 0x7f2a3c114f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:52.843 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.850+0000 7f2a40a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 0x7f2a3c114f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53214/0 (socket says 172.21.15.66:53214) 2024-02-28T16:20:52.843 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.850+0000 7f2a40a00700 1 -- 172.21.15.66:0/2739970637 learned_addr learned my addr 172.21.15.66:0/2739970637 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:52.843 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.850+0000 7f2a40a00700 1 -- 172.21.15.66:0/2739970637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a30005040 con 0x7f2a3c102a10 2024-02-28T16:20:52.843 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.851+0000 7f2a40a00700 1 --2- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 0x7f2a3c114f40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009330 tx=0x7f2a30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:52.843 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.851+0000 7f2a38c00700 1 -- 172.21.15.66:0/2739970637 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a3000be50 con 0x7f2a3c102a10 2024-02-28T16:20:52.844 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.851+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c115680 con 0x7f2a3c102a10 2024-02-28T16:20:52.844 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.851+0000 7f2a38c00700 1 -- 172.21.15.66:0/2739970637 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a30009630 con 0x7f2a3c102a10 2024-02-28T16:20:52.844 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.851+0000 7f2a38c00700 1 -- 172.21.15.66:0/2739970637 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a3001c3f0 con 0x7f2a3c102a10 2024-02-28T16:20:52.844 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.851+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c115b70 con 0x7f2a3c102a10 2024-02-28T16:20:52.845 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.852+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3c05f5c0 con 0x7f2a3c102a10 2024-02-28T16:20:52.846 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.853+0000 7f2a38c00700 1 -- 172.21.15.66:0/2739970637 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7f2a300097a0 con 0x7f2a3c102a10 2024-02-28T16:20:52.846 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.853+0000 7f2a38c00700 1 --2- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2a240369c0 0x7f2a24038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:52.846 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.853+0000 7f2a38c00700 1 -- 172.21.15.66:0/2739970637 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f2a30049c00 con 0x7f2a3c102a10 2024-02-28T16:20:52.846 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.854+0000 7f2a3be00700 1 --2- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2a240369c0 0x7f2a24038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:52.846 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.855+0000 7f2a3be00700 1 --2- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2a240369c0 0x7f2a24038e70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009990 tx=0x7f2a28006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:52.869+0000 7f2a38c00700 1 -- 172.21.15.66:0/2739970637 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2a30011960 con 0x7f2a3c102a10 2024-02-28T16:20:53.168 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.178+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a3c1106b0 con 0x7f2a3c102a10 2024-02-28T16:20:53.170 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.180+0000 7f2a38c00700 1 -- 172.21.15.66:0/2739970637 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f2a30049e60 con 0x7f2a3c102a10 2024-02-28T16:20:53.170 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:53.170 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:53.173 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2a240369c0 msgr2=0x7f2a24038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 --2- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2a240369c0 0x7f2a24038e70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009990 tx=0x7f2a28006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 msgr2=0x7f2a3c114f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 --2- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 0x7f2a3c114f40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009330 tx=0x7f2a30009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 shutdown_connections 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 --2- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2a240369c0 0x7f2a24038e70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 --2- 172.21.15.66:0/2739970637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c102a10 0x7f2a3c114f40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 >> 172.21.15.66:0/2739970637 conn(0x7f2a3c0f8550 msgr2=0x7f2a3c071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 shutdown_connections 2024-02-28T16:20:53.174 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:53.183+0000 7f2a41e00700 1 -- 172.21.15.66:0/2739970637 wait complete. 2024-02-28T16:20:53.175 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:20:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:53 smithi001 ceph-mon[28991]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:53 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/2739970637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:20:55.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:20:55.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:20:55.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:55 smithi001 ceph-mon[28991]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:55.861 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:20:57.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:20:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:20:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: Deploying daemon mon.smithi066 on smithi066 2024-02-28T16:20:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:20:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:20:58.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:57 smithi001 ceph-mon[28991]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:20:59.635 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.549+0000 7f9ec9e00700 1 -- 172.21.15.66:0/356519542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 msgr2=0x7f9ec4070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:59.635 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.549+0000 7f9ec9e00700 1 --2- 172.21.15.66:0/356519542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 0x7f9ec4070700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40048c0 tx=0x7f9eb4009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:59.635 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.549+0000 7f9ec9e00700 1 -- 172.21.15.66:0/356519542 shutdown_connections 2024-02-28T16:20:59.635 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.549+0000 7f9ec9e00700 1 --2- 172.21.15.66:0/356519542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 0x7f9ec4070700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.549+0000 7f9ec9e00700 1 -- 172.21.15.66:0/356519542 >> 172.21.15.66:0/356519542 conn(0x7f9ec40f8760 msgr2=0x7f9ec40fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.549+0000 7f9ec9e00700 1 -- 172.21.15.66:0/356519542 shutdown_connections 2024-02-28T16:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.549+0000 7f9ec9e00700 1 -- 172.21.15.66:0/356519542 wait complete. 2024-02-28T16:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.550+0000 7f9ec9e00700 1 Processor -- start 2024-02-28T16:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.550+0000 7f9ec9e00700 1 -- start start 2024-02-28T16:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.551+0000 7f9ec9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 0x7f9ec4115190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.551+0000 7f9ec9e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec41156d0 con 0x7f9ec4070330 2024-02-28T16:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.551+0000 7f9ec2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 0x7f9ec4115190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.551+0000 7f9ec2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 0x7f9ec4115190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:53246/0 (socket says 172.21.15.66:53246) 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.551+0000 7f9ec2a00700 1 -- 172.21.15.66:0/2744888726 learned_addr learned my addr 172.21.15.66:0/2744888726 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.552+0000 7f9ec2a00700 1 -- 172.21.15.66:0/2744888726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb4005040 con 0x7f9ec4070330 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.552+0000 7f9ec2a00700 1 --2- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 0x7f9ec4115190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009300 tx=0x7f9eb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.553+0000 7f9ebaa00700 1 -- 172.21.15.66:0/2744888726 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9eb4004e60 con 0x7f9ec4070330 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.553+0000 7f9ebaa00700 1 -- 172.21.15.66:0/2744888726 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9eb4013420 con 0x7f9ec4070330 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.553+0000 7f9ebaa00700 1 -- 172.21.15.66:0/2744888726 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9eb401c5b0 con 0x7f9ec4070330 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.553+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec41158d0 con 0x7f9ec4070330 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.554+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec4115d40 con 0x7f9ec4070330 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.555+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec405f5c0 con 0x7f9ec4070330 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.555+0000 7f9ebaa00700 1 -- 172.21.15.66:0/2744888726 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7f9eb4013a90 con 0x7f9ec4070330 2024-02-28T16:20:59.637 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.556+0000 7f9ebaa00700 1 --2- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f9ea8036960 0x7f9ea8038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:20:59.638 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.556+0000 7f9ebaa00700 1 -- 172.21.15.66:0/2744888726 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9eb4049660 con 0x7f9ec4070330 2024-02-28T16:20:59.638 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.557+0000 7f9ec2000700 1 --2- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f9ea8036960 0x7f9ea8038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:20:59.638 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.558+0000 7f9ec2000700 1 --2- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f9ea8036960 0x7f9ea8038e10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009990 tx=0x7f9eb0006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:20:59.638 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.571+0000 7f9ebaa00700 1 -- 172.21.15.66:0/2744888726 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9eb4011960 con 0x7f9ec4070330 2024-02-28T16:20:59.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.867+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ec4109110 con 0x7f9ec4070330 2024-02-28T16:20:59.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.868+0000 7f9ebaa00700 1 -- 172.21.15.66:0/2744888726 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f9eb4023570 con 0x7f9ec4070330 2024-02-28T16:20:59.858 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:20:59.858 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":1,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:17:37.519004Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-28T16:20:59.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f9ea8036960 msgr2=0x7f9ea8038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:59.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 --2- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f9ea8036960 0x7f9ea8038e10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009990 tx=0x7f9eb0006e40 comp rx=0 tx=0).stop 2024-02-28T16:20:59.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 msgr2=0x7f9ec4115190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:20:59.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 --2- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 0x7f9ec4115190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009300 tx=0x7f9eb4009e30 comp rx=0 tx=0).stop 2024-02-28T16:20:59.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 shutdown_connections 2024-02-28T16:20:59.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 --2- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f9ea8036960 0x7f9ea8038e10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:59.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 --2- 172.21.15.66:0/2744888726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec4070330 0x7f9ec4115190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:20:59.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 >> 172.21.15.66:0/2744888726 conn(0x7f9ec40f8760 msgr2=0x7f9ec4103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:20:59.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 shutdown_connections 2024-02-28T16:20:59.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:20:59.871+0000 7f9ec9e00700 1 -- 172.21.15.66:0/2744888726 wait complete. 2024-02-28T16:20:59.862 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 1 2024-02-28T16:21:00.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:20:59 smithi001 ceph-mon[28991]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:01.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:00 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/2744888726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:21:01.870 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-28T16:21:01.870 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-28T16:21:02.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:01 smithi001 ceph-mon[28991]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:02.774 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T16:21:02.920 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:02 smithi066 ceph-mon[36236]: mon.smithi066@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-28T16:21:07.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.918+0000 7f90d8a00700 1 -- 172.21.15.66:0/917738077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 msgr2=0x7f90b8005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:07.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.918+0000 7f90d8a00700 1 --2- 172.21.15.66:0/917738077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 0x7f90b8005a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90d40fcc10 tx=0x7f90c4009040 comp rx=0 tx=0).stop 2024-02-28T16:21:07.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.918+0000 7f90d8a00700 1 -- 172.21.15.66:0/917738077 shutdown_connections 2024-02-28T16:21:07.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.918+0000 7f90d8a00700 1 --2- 172.21.15.66:0/917738077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 0x7f90b8005a00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:07.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.918+0000 7f90d8a00700 1 -- 172.21.15.66:0/917738077 >> 172.21.15.66:0/917738077 conn(0x7f90d40f8860 msgr2=0x7f90d40fac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:21:07.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.919+0000 7f90d8a00700 1 -- 172.21.15.66:0/917738077 shutdown_connections 2024-02-28T16:21:07.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.919+0000 7f90d8a00700 1 -- 172.21.15.66:0/917738077 wait complete. 2024-02-28T16:21:07.911 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.920+0000 7f90d8a00700 1 Processor -- start 2024-02-28T16:21:07.911 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.920+0000 7f90d8a00700 1 -- start start 2024-02-28T16:21:07.911 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.921+0000 7f90d8a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 0x7f90d41153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:07.911 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.921+0000 7f90d8a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 0x7f90d410f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:07.911 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.921+0000 7f90d8a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4115ef0 con 0x7f90b8003580 2024-02-28T16:21:07.911 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.921+0000 7f90d8a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d410f780 con 0x7f90d41158e0 2024-02-28T16:21:07.912 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 0x7f90d41153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 0x7f90d41153a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.66:44170/0 (socket says 172.21.15.66:44170) 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90d1e00700 1 -- 172.21.15.66:0/2905427105 learned_addr learned my addr 172.21.15.66:0/2905427105 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90c9400700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 0x7f90d410f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90c9400700 1 -- 172.21.15.66:0/2905427105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 msgr2=0x7f90d410f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90c9400700 1 -- 172.21.15.66:0/2905427105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 msgr2=0x7f90d410f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90c9400700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 0x7f90d410f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90c9400700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 0x7f90d410f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90d1e00700 1 -- 172.21.15.66:0/2905427105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 msgr2=0x7f90d410f240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.922+0000 7f90d1e00700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 0x7f90d410f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:07.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.923+0000 7f90d1e00700 1 -- 172.21.15.66:0/2905427105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c40079e0 con 0x7f90b8003580 2024-02-28T16:21:07.914 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.923+0000 7f90d1e00700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 0x7f90d41153a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f90c4004920 tx=0x7f90c4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:21:07.914 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.924+0000 7f90caa00700 1 -- 172.21.15.66:0/2905427105 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90c4004610 con 0x7f90b8003580 2024-02-28T16:21:07.914 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.924+0000 7f90caa00700 1 -- 172.21.15.66:0/2905427105 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90c4017b50 con 0x7f90b8003580 2024-02-28T16:21:07.914 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.924+0000 7f90caa00700 1 -- 172.21.15.66:0/2905427105 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90c4020c90 con 0x7f90b8003580 2024-02-28T16:21:07.914 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.924+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d410fa00 con 0x7f90b8003580 2024-02-28T16:21:07.914 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.924+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d410ffc0 con 0x7f90b8003580 2024-02-28T16:21:07.915 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.925+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d405f5c0 con 0x7f90b8003580 2024-02-28T16:21:07.916 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.926+0000 7f90caa00700 1 -- 172.21.15.66:0/2905427105 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f90c4017cc0 con 0x7f90b8003580 2024-02-28T16:21:07.919 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.927+0000 7f90caa00700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f90b0069360 0x7f90b006b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:07.919 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.927+0000 7f90caa00700 1 -- 172.21.15.66:0/2905427105 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f90c4088970 con 0x7f90b8003580 2024-02-28T16:21:07.919 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.928+0000 7f90c9400700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f90b0069360 0x7f90b006b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:07.919 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.928+0000 7f90c9400700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f90b0069360 0x7f90b006b810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f90bc006f30 tx=0x7f90bc008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:21:07.924 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:07.934+0000 7f90caa00700 1 -- 172.21.15.66:0/2905427105 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f90c4059770 con 0x7f90b8003580 2024-02-28T16:21:08.233 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.243+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90d4110980 con 0x7f90b8003580 2024-02-28T16:21:08.233 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: mon.smithi001 calling monitor election 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/239923070' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: mon.smithi066 calling monitor election 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: mon.smithi001 is new leader, mons smithi001,smithi066 in quorum (ranks 0,1) 2024-02-28T16:21:08.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: monmap e2: 2 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],smithi066=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: fsmap 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: osdmap e5: 0 total, 0 up, 0 in 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: mgrmap e17: smithi001.rmthtx(active, since 41s) 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/239923070' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: overall HEALTH_OK 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/239923070' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/239923070' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:08.235 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:08.236 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-02-28T16:21:08.236 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":2,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","modified":"2024-02-28T16:21:02.739489Z","created":"2024-02-28T16:17:37.519004Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi066","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:3300","nonce":0},{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-28T16:21:08.236 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.244+0000 7f90caa00700 1 -- 172.21.15.66:0/2905427105 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1009 (secure 0 0 0) 0x7f90c402f090 con 0x7f90b8003580 2024-02-28T16:21:08.237 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f90b0069360 msgr2=0x7f90b006b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:08.237 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f90b0069360 0x7f90b006b810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f90bc006f30 tx=0x7f90bc008040 comp rx=0 tx=0).stop 2024-02-28T16:21:08.237 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 msgr2=0x7f90d41153a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:08.237 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 0x7f90d41153a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f90c4004920 tx=0x7f90c4009040 comp rx=0 tx=0).stop 2024-02-28T16:21:08.237 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 shutdown_connections 2024-02-28T16:21:08.238 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f90b0069360 0x7f90b006b810 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:08.238 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90b8003580 0x7f90d41153a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:08.238 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 --2- 172.21.15.66:0/2905427105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90d41158e0 0x7f90d410f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:08.238 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 >> 172.21.15.66:0/2905427105 conn(0x7f90d40f8860 msgr2=0x7f90d40fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:21:08.238 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 shutdown_connections 2024-02-28T16:21:08.238 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:21:08.247+0000 7f90d8a00700 1 -- 172.21.15.66:0/2905427105 wait complete. 2024-02-28T16:21:08.238 INFO:teuthology.orchestra.run.smithi066.stderr:dumped monmap epoch 2 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: mon.smithi001 calling monitor election 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/239923070' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: mon.smithi066 calling monitor election 2024-02-28T16:21:08.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: mon.smithi001 is new leader, mons smithi001,smithi066 in quorum (ranks 0,1) 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: monmap e2: 2 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],smithi066=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: fsmap 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: osdmap e5: 0 total, 0 up, 0 in 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: mgrmap e17: smithi001.rmthtx(active, since 41s) 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/239923070' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: overall HEALTH_OK 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/239923070' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/239923070' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:08.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:08.914 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: mgrmap e18: smithi001.rmthtx(active, since 41s), standbys: smithi066.ztraxe 2024-02-28T16:21:08.914 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi066.ztraxe", "id": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:21:08.914 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:08.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:08.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:08.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:21:08.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: from='client.? 172.21.15.66:0/2905427105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:21:08.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:08.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:08 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:09.171 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-28T16:21:09.171 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph config generate-minimal-conf 2024-02-28T16:21:09.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: mgrmap e18: smithi001.rmthtx(active, since 41s), standbys: smithi066.ztraxe 2024-02-28T16:21:09.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi066.ztraxe", "id": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:21:09.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:09.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:09.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:09.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:21:09.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/2905427105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-28T16:21:09.265 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:09.265 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:08 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:21:09.670 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:21:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:11 smithi001 ceph-mon[28991]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:21:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:21:11.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.734+0000 7ff780000700 1 -- 172.21.15.1:0/3913116228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7780fdc30 msgr2=0x7ff7780fe000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:11.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.734+0000 7ff780000700 1 --2- 172.21.15.1:0/3913116228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7780fdc30 0x7ff7780fe000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0048c0 tx=0x7ff76c009e30 comp rx=0 tx=0).stop 2024-02-28T16:21:11.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.735+0000 7ff780000700 1 -- 172.21.15.1:0/3913116228 shutdown_connections 2024-02-28T16:21:11.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.735+0000 7ff780000700 1 --2- 172.21.15.1:0/3913116228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7780fdc30 0x7ff7780fe000 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:11.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.735+0000 7ff780000700 1 -- 172.21.15.1:0/3913116228 >> 172.21.15.1:0/3913116228 conn(0x7ff7780f7640 msgr2=0x7ff7780f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:21:11.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.735+0000 7ff780000700 1 -- 172.21.15.1:0/3913116228 shutdown_connections 2024-02-28T16:21:11.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.735+0000 7ff780000700 1 -- 172.21.15.1:0/3913116228 wait complete. 2024-02-28T16:21:11.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.736+0000 7ff780000700 1 Processor -- start 2024-02-28T16:21:11.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.736+0000 7ff780000700 1 -- start start 2024-02-28T16:21:11.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.736+0000 7ff780000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7780fdc30 0x7ff7781153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:11.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.736+0000 7ff780000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7781158f0 0x7ff77810f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:11.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.736+0000 7ff780000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff778115f00 con 0x7ff7781158f0 2024-02-28T16:21:11.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.736+0000 7ff780000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77810f790 con 0x7ff7780fdc30 2024-02-28T16:21:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.737+0000 7ff77cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7781158f0 0x7ff77810f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.737+0000 7ff77cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7781158f0 0x7ff77810f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38546/0 (socket says 172.21.15.1:38546) 2024-02-28T16:21:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.737+0000 7ff77cc00700 1 -- 172.21.15.1:0/4247158172 learned_addr learned my addr 172.21.15.1:0/4247158172 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.737+0000 7ff77d600700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7780fdc30 0x7ff7781153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.737+0000 7ff77cc00700 1 -- 172.21.15.1:0/4247158172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7780fdc30 msgr2=0x7ff7781153b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.737+0000 7ff77cc00700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7780fdc30 0x7ff7781153b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.737+0000 7ff77cc00700 1 -- 172.21.15.1:0/4247158172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c005040 con 0x7ff7781158f0 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.738+0000 7ff77cc00700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7781158f0 0x7ff77810f250 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff76400b790 tx=0x7ff764009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.738+0000 7ff775600700 1 -- 172.21.15.1:0/4247158172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff764010870 con 0x7ff7781158f0 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.738+0000 7ff775600700 1 -- 172.21.15.1:0/4247158172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff76400f3e0 con 0x7ff7781158f0 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.738+0000 7ff775600700 1 -- 172.21.15.1:0/4247158172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff764010ea0 con 0x7ff7781158f0 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.738+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77810fa70 con 0x7ff7781158f0 2024-02-28T16:21:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.738+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77810ffb0 con 0x7ff7781158f0 2024-02-28T16:21:11.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.739+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77805f5c0 con 0x7ff7781158f0 2024-02-28T16:21:11.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.740+0000 7ff775600700 1 -- 172.21.15.1:0/4247158172 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7ff76400f550 con 0x7ff7781158f0 2024-02-28T16:21:11.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.742+0000 7ff775600700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff760069380 0x7ff76006b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:11.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.742+0000 7ff775600700 1 -- 172.21.15.1:0/4247158172 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff764086650 con 0x7ff7781158f0 2024-02-28T16:21:11.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.742+0000 7ff77d600700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff760069380 0x7ff76006b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:11.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.743+0000 7ff77d600700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff760069380 0x7ff76006b830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00b3b0 tx=0x7ff76c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:21:11.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.746+0000 7ff775600700 1 -- 172.21.15.1:0/4247158172 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff764057450 con 0x7ff7781158f0 2024-02-28T16:21:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:11 smithi066 ceph-mon[36236]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:21:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:21:11.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.984+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff778106f10 con 0x7ff7781158f0 2024-02-28T16:21:11.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.985+0000 7ff775600700 1 -- 172.21.15.1:0/4247158172 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+221 (secure 0 0 0) 0x7ff7640140f0 con 0x7ff7781158f0 2024-02-28T16:21:11.976 INFO:teuthology.orchestra.run.smithi001.stdout:# minimal ceph.conf for c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:21:11.976 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-02-28T16:21:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:21:11.976 INFO:teuthology.orchestra.run.smithi001.stdout: mon_host = [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff760069380 msgr2=0x7ff76006b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff760069380 0x7ff76006b830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00b3b0 tx=0x7ff76c009e30 comp rx=0 tx=0).stop 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7781158f0 msgr2=0x7ff77810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7781158f0 0x7ff77810f250 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff76400b790 tx=0x7ff764009250 comp rx=0 tx=0).stop 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 shutdown_connections 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff760069380 0x7ff76006b830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7780fdc30 0x7ff7781153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 --2- 172.21.15.1:0/4247158172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7781158f0 0x7ff77810f250 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 >> 172.21.15.1:0/4247158172 conn(0x7ff7780f7640 msgr2=0x7ff778103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:21:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 shutdown_connections 2024-02-28T16:21:11.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:11.989+0000 7ff780000700 1 -- 172.21.15.1:0/4247158172 wait complete. 2024-02-28T16:21:12.723 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:12 smithi001 ceph-mon[28991]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:21:12.723 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:12 smithi001 ceph-mon[28991]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:21:12.723 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:12 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:12.723 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:12 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/4247158172' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:12.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:12 smithi066 ceph-mon[36236]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:21:12.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:12 smithi066 ceph-mon[36236]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:21:12.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:12 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:12.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:12 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/4247158172' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:12.842 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-28T16:21:12.842 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:21:12.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.conf 2024-02-28T16:21:12.932 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:21:12.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:21:13.017 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:21:13.018 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.conf 2024-02-28T16:21:13.130 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:21:13.130 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:21:13.266 INFO:tasks.cephadm:Deploying OSDs... 2024-02-28T16:21:13.266 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:21:13.267 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-02-28T16:21:13.311 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-28T16:21:13.311 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-02-28T16:21:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-28T16:21:13.393 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-28T16:21:13.394 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-02-28T16:21:13.394 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:21:13.394 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-02-28T16:21:13.394 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-02-28 16:20:15.661475210 +0000 2024-02-28T16:21:13.394 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-02-28 16:12:05.894212894 +0000 2024-02-28T16:21:13.394 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-02-28 16:12:05.894212894 +0000 2024-02-28T16:21:13.394 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-02-28 16:12:05.894212894 +0000 2024-02-28T16:21:13.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-28T16:21:13.528 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-02-28T16:21:13.528 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-02-28T16:21:13.528 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000229493 s, 2.2 MB/s 2024-02-28T16:21:13.532 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-28T16:21:13.584 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-02-28T16:21:13.624 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-28T16:21:13.624 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-28T16:21:13.625 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-02-28T16:21:13.625 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:21:13.625 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-02-28T16:21:13.625 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-02-28 16:20:15.662475197 +0000 2024-02-28T16:21:13.625 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-02-28 16:12:06.478204875 +0000 2024-02-28T16:21:13.625 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-02-28 16:12:06.478204875 +0000 2024-02-28T16:21:13.625 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-02-28 16:12:06.478204875 +0000 2024-02-28T16:21:13.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-28T16:21:13.734 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-02-28T16:21:13.734 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-02-28T16:21:13.734 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000181357 s, 2.8 MB/s 2024-02-28T16:21:13.737 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:21:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:21:13.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:13.776 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-02-28T16:21:13.869 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-28T16:21:13.870 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-28T16:21:13.870 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-02-28T16:21:13.870 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:21:13.870 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-02-28T16:21:13.870 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-02-28 16:20:15.662475197 +0000 2024-02-28T16:21:13.870 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-02-28 16:12:07.050197021 +0000 2024-02-28T16:21:13.870 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-02-28 16:12:07.050197021 +0000 2024-02-28T16:21:13.870 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-02-28 16:12:07.050197021 +0000 2024-02-28T16:21:13.870 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-28T16:21:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-02-28T16:21:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-02-28T16:21:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000200123 s, 2.6 MB/s 2024-02-28T16:21:13.948 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-28T16:21:13.995 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-02-28T16:21:14.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:21:14.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:21:14.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:14.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:14.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:14.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:14.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:14.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:14.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:21:14.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:21:14.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:14.042 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-28T16:21:14.042 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-28T16:21:14.042 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-02-28T16:21:14.042 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:21:14.042 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-02-28T16:21:14.043 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-02-28 16:20:15.663475183 +0000 2024-02-28T16:21:14.043 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-02-28 16:12:07.633189015 +0000 2024-02-28T16:21:14.043 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-02-28 16:12:07.633189015 +0000 2024-02-28T16:21:14.043 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-02-28 16:12:07.633189015 +0000 2024-02-28T16:21:14.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-28T16:21:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-02-28T16:21:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-02-28T16:21:14.115 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000229047 s, 2.2 MB/s 2024-02-28T16:21:14.117 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-28T16:21:14.167 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:21:14.167 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-02-28T16:21:14.227 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-28T16:21:14.227 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 2024-02-28T16:21:14.284 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-28T16:21:14.285 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-28T16:21:14.285 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-02-28T16:21:14.285 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:21:14.285 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-02-28T16:21:14.285 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-02-28 16:21:01.827952003 +0000 2024-02-28T16:21:14.285 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-02-28 16:12:11.120356200 +0000 2024-02-28T16:21:14.285 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-02-28 16:12:11.120356200 +0000 2024-02-28T16:21:14.285 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-02-28 16:12:11.120356200 +0000 2024-02-28T16:21:14.285 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-28T16:21:14.383 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-02-28T16:21:14.384 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-02-28T16:21:14.384 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000428823 s, 1.2 MB/s 2024-02-28T16:21:14.388 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-28T16:21:14.451 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 2024-02-28T16:21:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-28T16:21:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-28T16:21:14.506 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-02-28T16:21:14.506 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:21:14.506 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-02-28T16:21:14.506 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-02-28 16:21:01.828951996 +0000 2024-02-28T16:21:14.506 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-02-28 16:12:11.750351575 +0000 2024-02-28T16:21:14.506 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-02-28 16:12:11.750351575 +0000 2024-02-28T16:21:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-02-28 16:12:11.750351575 +0000 2024-02-28T16:21:14.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-28T16:21:14.605 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-02-28T16:21:14.605 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-02-28T16:21:14.605 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000371008 s, 1.4 MB/s 2024-02-28T16:21:14.610 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-28T16:21:14.668 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 2024-02-28T16:21:14.718 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-28T16:21:14.718 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-28T16:21:14.718 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-02-28T16:21:14.718 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:21:14.719 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-02-28T16:21:14.719 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-02-28 16:21:01.828951996 +0000 2024-02-28T16:21:14.719 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-02-28 16:12:12.341347235 +0000 2024-02-28T16:21:14.719 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-02-28 16:12:12.341347235 +0000 2024-02-28T16:21:14.719 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-02-28 16:12:12.341347235 +0000 2024-02-28T16:21:14.719 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-28T16:21:14.817 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-02-28T16:21:14.818 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-02-28T16:21:14.818 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.0003621 s, 1.4 MB/s 2024-02-28T16:21:14.821 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-28T16:21:14.878 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 2024-02-28T16:21:14.932 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-28T16:21:14.933 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-28T16:21:14.933 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-02-28T16:21:14.933 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:21:14.933 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-02-28T16:21:14.933 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-02-28 16:21:01.830951981 +0000 2024-02-28T16:21:14.933 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-02-28 16:12:12.959342699 +0000 2024-02-28T16:21:14.933 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-02-28 16:12:12.959342699 +0000 2024-02-28T16:21:14.933 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-02-28 16:12:12.959342699 +0000 2024-02-28T16:21:14.933 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-28T16:21:14.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:14 smithi066 ceph-mon[36236]: Reconfiguring mon.smithi001 (unknown last config time)... 2024-02-28T16:21:14.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:14 smithi066 ceph-mon[36236]: Reconfiguring daemon mon.smithi001 on smithi001 2024-02-28T16:21:15.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:14 smithi001 ceph-mon[28991]: Reconfiguring mon.smithi001 (unknown last config time)... 2024-02-28T16:21:15.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:14 smithi001 ceph-mon[28991]: Reconfiguring daemon mon.smithi001 on smithi001 2024-02-28T16:21:15.037 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-02-28T16:21:15.037 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-02-28T16:21:15.038 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000362049 s, 1.4 MB/s 2024-02-28T16:21:15.042 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-28T16:21:15.101 INFO:tasks.cephadm:Deploying osd.0 on smithi001 with /dev/vg_nvme/lv_4... 2024-02-28T16:21:15.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-28T16:21:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:21:15.964 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:15 smithi001 ceph-mon[28991]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:15.964 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:15.964 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:15.964 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.rmthtx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:21:15.964 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:21:15.964 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:15 smithi066 ceph-mon[36236]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.rmthtx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:21:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:21:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:16.795 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:16 smithi001 ceph-mon[28991]: Reconfiguring mgr.smithi001.rmthtx (unknown last config time)... 2024-02-28T16:21:16.795 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:16 smithi001 ceph-mon[28991]: Reconfiguring daemon mgr.smithi001.rmthtx on smithi001 2024-02-28T16:21:16.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:16 smithi066 ceph-mon[36236]: Reconfiguring mgr.smithi001.rmthtx (unknown last config time)... 2024-02-28T16:21:16.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:16 smithi066 ceph-mon[36236]: Reconfiguring daemon mgr.smithi001.rmthtx on smithi001 2024-02-28T16:21:17.971 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:17 smithi001 ceph-mon[28991]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:17.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:17 smithi066 ceph-mon[36236]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:18.350 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-28T16:21:18.350 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-28T16:21:18.350 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-02-28T16:21:18.350 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-02-28T16:21:18.350 INFO:teuthology.orchestra.run.smithi001.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.033661 s, 312 MB/s 2024-02-28T16:21:18.350 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 2024-02-28T16:21:18.350 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-02-28T16:21:19.256 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi001:vg_nvme/lv_4 2024-02-28T16:21:19.472 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:19 smithi001 ceph-mon[28991]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:19.472 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:19.472 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:19.472 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:19 smithi001 ceph-mon[28991]: Reconfiguring crash.smithi001 (monmap changed)... 2024-02-28T16:21:19.472 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:21:19.472 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:19.472 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:19 smithi001 ceph-mon[28991]: Reconfiguring daemon crash.smithi001 on smithi001 2024-02-28T16:21:19.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:19 smithi066 ceph-mon[36236]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:19.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:19.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:19.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:19 smithi066 ceph-mon[36236]: Reconfiguring crash.smithi001 (monmap changed)... 2024-02-28T16:21:19.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:21:19.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:19.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:19 smithi066 ceph-mon[36236]: Reconfiguring daemon crash.smithi001 on smithi001 2024-02-28T16:21:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:21:21.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:21 smithi001 ceph-mon[28991]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:21.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:21 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:21.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:21 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:21.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:21 smithi001 ceph-mon[28991]: Reconfiguring alertmanager.smithi001 (dependencies changed)... 2024-02-28T16:21:21.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:21 smithi001 ceph-mon[28991]: Reconfiguring daemon alertmanager.smithi001 on smithi001 2024-02-28T16:21:21.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:21 smithi066 ceph-mon[36236]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:21.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:21 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:21.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:21 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:21.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:21 smithi066 ceph-mon[36236]: Reconfiguring alertmanager.smithi001 (dependencies changed)... 2024-02-28T16:21:21.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:21 smithi066 ceph-mon[36236]: Reconfiguring daemon alertmanager.smithi001 on smithi001 2024-02-28T16:21:23.635 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:23 smithi001 ceph-mon[28991]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:23.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:23 smithi066 ceph-mon[36236]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:24.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.493+0000 7f1228c00700 1 -- 172.21.15.1:0/2208950325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224104750 msgr2=0x7f1224104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:24.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.493+0000 7f1228c00700 1 --2- 172.21.15.1:0/2208950325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224104750 0x7f1224104ba0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1210004940 tx=0x7f1210009e30 comp rx=0 tx=0).stop 2024-02-28T16:21:24.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.493+0000 7f1228c00700 1 -- 172.21.15.1:0/2208950325 shutdown_connections 2024-02-28T16:21:24.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.493+0000 7f1228c00700 1 --2- 172.21.15.1:0/2208950325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224104750 0x7f1224104ba0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:24.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.493+0000 7f1228c00700 1 --2- 172.21.15.1:0/2208950325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1224103540 0x7f1224103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:24.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.493+0000 7f1228c00700 1 -- 172.21.15.1:0/2208950325 >> 172.21.15.1:0/2208950325 conn(0x7f12240727d0 msgr2=0x7f1224074c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:21:24.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.493+0000 7f1228c00700 1 -- 172.21.15.1:0/2208950325 shutdown_connections 2024-02-28T16:21:24.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.493+0000 7f1228c00700 1 -- 172.21.15.1:0/2208950325 wait complete. 2024-02-28T16:21:24.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.494+0000 7f1228c00700 1 Processor -- start 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.494+0000 7f1228c00700 1 -- start start 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1228c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224103540 0x7f1224113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1228c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1224104750 0x7f12241144e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1228c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1224114b70 con 0x7f1224103540 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1228c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12241153b0 con 0x7f1224104750 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1221e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224103540 0x7f1224113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1221e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224103540 0x7f1224113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56006/0 (socket says 172.21.15.1:56006) 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1221e00700 1 -- 172.21.15.1:0/2125162238 learned_addr learned my addr 172.21.15.1:0/2125162238 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1221400700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1224104750 0x7f12241144e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1221e00700 1 -- 172.21.15.1:0/2125162238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1224104750 msgr2=0x7f12241144e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1221e00700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1224104750 0x7f12241144e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:21:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.495+0000 7f1221e00700 1 -- 172.21.15.1:0/2125162238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1210005040 con 0x7f1224103540 2024-02-28T16:21:24.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.496+0000 7f1221e00700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224103540 0x7f1224113fa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f121400b790 tx=0x7f1214009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:21:24.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.496+0000 7f121a000700 1 -- 172.21.15.1:0/2125162238 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f121400f6f0 con 0x7f1224103540 2024-02-28T16:21:24.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.496+0000 7f121a000700 1 -- 172.21.15.1:0/2125162238 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f121400fd20 con 0x7f1224103540 2024-02-28T16:21:24.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.496+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1224115690 con 0x7f1224103540 2024-02-28T16:21:24.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.496+0000 7f121a000700 1 -- 172.21.15.1:0/2125162238 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12140103a0 con 0x7f1224103540 2024-02-28T16:21:24.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.496+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1224115bd0 con 0x7f1224103540 2024-02-28T16:21:24.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.497+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122404bed0 con 0x7f1224103540 2024-02-28T16:21:24.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.499+0000 7f121a000700 1 -- 172.21.15.1:0/2125162238 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f1214011400 con 0x7f1224103540 2024-02-28T16:21:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.500+0000 7f121a000700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f1208069210 0x7f120806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:21:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.500+0000 7f121a000700 1 -- 172.21.15.1:0/2125162238 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1214086910 con 0x7f1224103540 2024-02-28T16:21:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.501+0000 7f1221400700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f1208069210 0x7f120806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:21:24.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.501+0000 7f1221400700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f1208069210 0x7f120806b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f121000b3b0 tx=0x7f1210009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:21:24.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.504+0000 7f121a000700 1 -- 172.21.15.1:0/2125162238 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1214057790 con 0x7f1224103540 2024-02-28T16:21:24.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:21:24.734+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1224116180 con 0x7f1208069210 2024-02-28T16:21:25.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:25 smithi001 ceph-mon[28991]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:25.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:25 smithi001 ceph-mon[28991]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:21:25.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:25 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:21:25.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:25 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:21:25.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:25 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:25.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:25 smithi066 ceph-mon[36236]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:25.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:25 smithi066 ceph-mon[36236]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:21:25.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:25 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:21:25.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:25 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:21:25.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:25 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:26.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:21:26.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:21:27.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:27 smithi066 ceph-mon[36236]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:27.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:27 smithi001 ceph-mon[28991]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:29.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:29 smithi001 ceph-mon[28991]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:29.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:29 smithi066 ceph-mon[36236]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:30.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:30.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:30.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:30 smithi066 ceph-mon[36236]: Reconfiguring grafana.smithi001 (dependencies changed)... 2024-02-28T16:21:30.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:30 smithi066 ceph-mon[36236]: Reconfiguring daemon grafana.smithi001 on smithi001 2024-02-28T16:21:30.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:30.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:30.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:30 smithi001 ceph-mon[28991]: Reconfiguring grafana.smithi001 (dependencies changed)... 2024-02-28T16:21:30.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:30 smithi001 ceph-mon[28991]: Reconfiguring daemon grafana.smithi001 on smithi001 2024-02-28T16:21:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:31 smithi066 ceph-mon[36236]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:31 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/256541847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac92369f-8144-4695-87fe-f9cd73fb5f29"}]: dispatch 2024-02-28T16:21:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:31 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/256541847' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac92369f-8144-4695-87fe-f9cd73fb5f29"}]': finished 2024-02-28T16:21:31.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:31 smithi066 ceph-mon[36236]: osdmap e6: 1 total, 0 up, 1 in 2024-02-28T16:21:31.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:31 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:21:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:31 smithi001 ceph-mon[28991]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:31 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/256541847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac92369f-8144-4695-87fe-f9cd73fb5f29"}]: dispatch 2024-02-28T16:21:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:31 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/256541847' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac92369f-8144-4695-87fe-f9cd73fb5f29"}]': finished 2024-02-28T16:21:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:31 smithi001 ceph-mon[28991]: osdmap e6: 1 total, 0 up, 1 in 2024-02-28T16:21:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:31 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:21:32.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:32 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/414272150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:21:32.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:32 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/414272150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:21:33.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:33 smithi066 ceph-mon[36236]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:33.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:33 smithi001 ceph-mon[28991]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:35.446 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:35 smithi001 ceph-mon[28991]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:35.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:35 smithi066 ceph-mon[36236]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:37.736 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:37 smithi001 ceph-mon[28991]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:37 smithi066 ceph-mon[36236]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:39.666 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:39 smithi001 ceph-mon[28991]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:39.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:39 smithi066 ceph-mon[36236]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:41.697 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:41 smithi001 ceph-mon[28991]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:41.697 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:21:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:41 smithi066 ceph-mon[36236]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:21:43.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:42 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:43.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:42 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:43.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:42 smithi066 ceph-mon[36236]: Reconfiguring prometheus.smithi001 (dependencies changed)... 2024-02-28T16:21:43.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:42 smithi066 ceph-mon[36236]: Reconfiguring daemon prometheus.smithi001 on smithi001 2024-02-28T16:21:43.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:42 smithi066 ceph-mon[36236]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:43.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:42 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:43.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:42 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:43.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:42 smithi001 ceph-mon[28991]: Reconfiguring prometheus.smithi001 (dependencies changed)... 2024-02-28T16:21:43.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:42 smithi001 ceph-mon[28991]: Reconfiguring daemon prometheus.smithi001 on smithi001 2024-02-28T16:21:43.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:42 smithi001 ceph-mon[28991]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:44.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:43 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-28T16:21:44.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:43 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:44.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:43 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-28T16:21:44.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:43 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:45 smithi066 ceph-mon[36236]: Deploying daemon osd.0 on smithi001 2024-02-28T16:21:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:45 smithi066 ceph-mon[36236]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:45.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:45 smithi001 ceph-mon[28991]: Deploying daemon osd.0 on smithi001 2024-02-28T16:21:45.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:45 smithi001 ceph-mon[28991]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:47.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:47 smithi066 ceph-mon[36236]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:47.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:47 smithi001 ceph-mon[28991]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:49.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:49 smithi066 ceph-mon[36236]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:49.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:49 smithi001 ceph-mon[28991]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:50.968 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:50 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:50.969 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:50 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:50.969 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:50 smithi001 ceph-mon[28991]: Reconfiguring crash.smithi066 (monmap changed)... 2024-02-28T16:21:50.969 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:50 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi066", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:21:50.969 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:50 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:50.969 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:50 smithi001 ceph-mon[28991]: Reconfiguring daemon crash.smithi066 on smithi066 2024-02-28T16:21:50.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:50 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:50.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:50 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:50.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:50 smithi066 ceph-mon[36236]: Reconfiguring crash.smithi066 (monmap changed)... 2024-02-28T16:21:50.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:50 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi066", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:21:50.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:50 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:50.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:50 smithi066 ceph-mon[36236]: Reconfiguring daemon crash.smithi066 on smithi066 2024-02-28T16:21:51.645 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:51 smithi066 ceph-mon[36236]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:51.645 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:51 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:51.645 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:51 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:51.645 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:51 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:21:51.645 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:51 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:21:51.645 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:51 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:51.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:51 smithi001 ceph-mon[28991]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:51.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:51.688 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:51.688 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:21:51.688 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:21:51.688 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:52.765 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:52 smithi001 ceph-mon[28991]: Reconfiguring mgr.smithi066.ztraxe (monmap changed)... 2024-02-28T16:21:52.766 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:52 smithi001 ceph-mon[28991]: Reconfiguring daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:21:52.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:52 smithi066 ceph-mon[36236]: Reconfiguring mgr.smithi066.ztraxe (monmap changed)... 2024-02-28T16:21:52.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:52 smithi066 ceph-mon[36236]: Reconfiguring daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:21:53.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:53 smithi001 ceph-mon[28991]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:53.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:53.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:53.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:21:53.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:21:53.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:53.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:53 smithi066 ceph-mon[36236]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:53.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:53.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:53.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:21:53.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:21:53.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:21:54.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:54 smithi066 ceph-mon[36236]: Reconfiguring mon.smithi066 (monmap changed)... 2024-02-28T16:21:54.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:54 smithi066 ceph-mon[36236]: Reconfiguring daemon mon.smithi066 on smithi066 2024-02-28T16:21:55.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:54 smithi001 ceph-mon[28991]: Reconfiguring mon.smithi066 (monmap changed)... 2024-02-28T16:21:55.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:54 smithi001 ceph-mon[28991]: Reconfiguring daemon mon.smithi066 on smithi066 2024-02-28T16:21:55.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:55 smithi066 ceph-mon[36236]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:55.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:55 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:55.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:55 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:55.956 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:55 smithi001 ceph-mon[28991]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:55.956 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:55 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:55.956 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:55 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:57.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:21:57.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:21:57.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi001.front.sepia.ceph.com:9093"}]: dispatch 2024-02-28T16:21:57.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi001.front.sepia.ceph.com:9093"}]: dispatch 2024-02-28T16:21:57.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi001.front.sepia.ceph.com:3000"}]: dispatch 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi001.front.sepia.ceph.com:3000"}]: dispatch 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi001.front.sepia.ceph.com:9095"}]: dispatch 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi001.front.sepia.ceph.com:9095"}]: dispatch 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:21:57.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:21:57.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:21:57.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:21:57.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi001.front.sepia.ceph.com:9093"}]: dispatch 2024-02-28T16:21:57.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi001.front.sepia.ceph.com:9093"}]: dispatch 2024-02-28T16:21:57.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:57.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi001.front.sepia.ceph.com:3000"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi001.front.sepia.ceph.com:3000"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi001.front.sepia.ceph.com:9095"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi001.front.sepia.ceph.com:9095"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:21:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:21:57.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:57 smithi066 ceph-mon[36236]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:57.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:57.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:58.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:57 smithi001 ceph-mon[28991]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:58.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:58.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:59.861 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:59 smithi001 ceph-mon[28991]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:59.861 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:59 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:59.861 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:21:59 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:59.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:59 smithi066 ceph-mon[36236]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:21:59.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:59 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:21:59.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:21:59 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:01.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.678+0000 7f121a000700 1 -- 172.21.15.1:0/2125162238 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1224116180 con 0x7f1208069210 2024-02-28T16:22:01.668 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 0 on host 'smithi001' 2024-02-28T16:22:01.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.681+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f1208069210 msgr2=0x7f120806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:22:01.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.681+0000 7f1228c00700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f1208069210 0x7f120806b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f121000b3b0 tx=0x7f1210009e30 comp rx=0 tx=0).stop 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.681+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224103540 msgr2=0x7f1224113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.681+0000 7f1228c00700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224103540 0x7f1224113fa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f121400b790 tx=0x7f1214009220 comp rx=0 tx=0).stop 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.682+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 shutdown_connections 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.682+0000 7f1228c00700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f1208069210 0x7f120806b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.682+0000 7f1228c00700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1224103540 0x7f1224113fa0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.682+0000 7f1228c00700 1 --2- 172.21.15.1:0/2125162238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1224104750 0x7f12241144e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.682+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 >> 172.21.15.1:0/2125162238 conn(0x7f12240727d0 msgr2=0x7f1224107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.682+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 shutdown_connections 2024-02-28T16:22:01.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:01.682+0000 7f1228c00700 1 -- 172.21.15.1:0/2125162238 wait complete. 2024-02-28T16:22:01.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:01 smithi066 ceph-mon[36236]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:22:01.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:01 smithi066 ceph-mon[36236]: from='osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-28T16:22:02.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:01 smithi001 ceph-mon[28991]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:22:02.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:01 smithi001 ceph-mon[28991]: from='osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-28T16:22:02.731 DEBUG:teuthology.orchestra.run.smithi001:osd.0> sudo journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.0.service 2024-02-28T16:22:02.738 INFO:tasks.cephadm:Deploying osd.1 on smithi001 with /dev/vg_nvme/lv_3... 2024-02-28T16:22:02.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-28T16:22:02.818 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:02 smithi001 ceph-mon[28991]: from='osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-28T16:22:02.900 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:02 smithi001 ceph-mon[28991]: osdmap e7: 1 total, 0 up, 1 in 2024-02-28T16:22:02.900 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:02 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:22:02.900 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:02 smithi001 ceph-mon[28991]: from='osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:22:02.900 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:02 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:02.900 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:02 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:02.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:02 smithi066 ceph-mon[36236]: from='osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-28T16:22:02.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:02 smithi066 ceph-mon[36236]: osdmap e7: 1 total, 0 up, 1 in 2024-02-28T16:22:02.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:02 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:22:02.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:02 smithi066 ceph-mon[36236]: from='osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:22:03.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:02 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:03.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:02 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:22:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:03 smithi066 ceph-mon[36236]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:22:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:03 smithi066 ceph-mon[36236]: from='osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-02-28T16:22:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:03 smithi066 ceph-mon[36236]: osdmap e8: 1 total, 0 up, 1 in 2024-02-28T16:22:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:03 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:22:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:03 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:22:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:03 smithi001 ceph-mon[28991]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:22:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:03 smithi001 ceph-mon[28991]: from='osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-02-28T16:22:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:03 smithi001 ceph-mon[28991]: osdmap e8: 1 total, 0 up, 1 in 2024-02-28T16:22:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:03 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:22:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:03 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:22:04.979 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:04 smithi001 ceph-mon[28991]: purged_snaps scrub starts 2024-02-28T16:22:04.980 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:04 smithi001 ceph-mon[28991]: purged_snaps scrub ok 2024-02-28T16:22:04.980 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:04 smithi001 ceph-mon[28991]: osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] boot 2024-02-28T16:22:04.980 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:04 smithi001 ceph-mon[28991]: osdmap e9: 1 total, 1 up, 1 in 2024-02-28T16:22:04.980 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:04 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:22:04.980 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:04 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:04.980 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:04 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:04 smithi066 ceph-mon[36236]: purged_snaps scrub starts 2024-02-28T16:22:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:04 smithi066 ceph-mon[36236]: purged_snaps scrub ok 2024-02-28T16:22:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:04 smithi066 ceph-mon[36236]: osd.0 [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] boot 2024-02-28T16:22:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:04 smithi066 ceph-mon[36236]: osdmap e9: 1 total, 1 up, 1 in 2024-02-28T16:22:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:04 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:22:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:04 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:05.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:04 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:05.985 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:05 smithi001 ceph-mon[28991]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:22:05.986 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:05 smithi001 ceph-mon[28991]: osdmap e10: 1 total, 1 up, 1 in 2024-02-28T16:22:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:05 smithi066 ceph-mon[36236]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-28T16:22:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:05 smithi066 ceph-mon[36236]: osdmap e10: 1 total, 1 up, 1 in 2024-02-28T16:22:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-28T16:22:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-28T16:22:06.521 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-02-28T16:22:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-02-28T16:22:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0325392 s, 322 MB/s 2024-02-28T16:22:06.521 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 2024-02-28T16:22:06.521 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-02-28T16:22:07.735 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:07 smithi001 ceph-mon[28991]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:07.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi001:vg_nvme/lv_3 2024-02-28T16:22:07.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:07 smithi066 ceph-mon[36236]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:22:09.823 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:09 smithi001 ceph-mon[28991]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:09.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:09 smithi066 ceph-mon[36236]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:10.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.492+0000 7fec31000700 1 -- 172.21.15.1:0/3203630193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fe0e0 msgr2=0x7fec2c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:22:10.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.492+0000 7fec31000700 1 --2- 172.21.15.1:0/3203630193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fe0e0 0x7fec2c0fe550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec14004940 tx=0x7fec14009e30 comp rx=0 tx=0).stop 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.492+0000 7fec31000700 1 -- 172.21.15.1:0/3203630193 shutdown_connections 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.492+0000 7fec31000700 1 --2- 172.21.15.1:0/3203630193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fe0e0 0x7fec2c0fe550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.492+0000 7fec31000700 1 --2- 172.21.15.1:0/3203630193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec2c0fcef0 0x7fec2c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.492+0000 7fec31000700 1 -- 172.21.15.1:0/3203630193 >> 172.21.15.1:0/3203630193 conn(0x7fec2c0f8620 msgr2=0x7fec2c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.492+0000 7fec31000700 1 -- 172.21.15.1:0/3203630193 shutdown_connections 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.492+0000 7fec31000700 1 -- 172.21.15.1:0/3203630193 wait complete. 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec31000700 1 Processor -- start 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec31000700 1 -- start start 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec31000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fcef0 0x7fec2c10fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec31000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec2c0fe0e0 0x7fec2c1100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:22:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec31000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c1106a0 con 0x7fec2c0fcef0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec31000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c110f40 con 0x7fec2c0fe0e0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec29e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fcef0 0x7fec2c10fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec29e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fcef0 0x7fec2c10fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37144/0 (socket says 172.21.15.1:37144) 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.493+0000 7fec29e00700 1 -- 172.21.15.1:0/3477738664 learned_addr learned my addr 172.21.15.1:0/3477738664 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.494+0000 7fec23e00700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec2c0fe0e0 0x7fec2c1100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.494+0000 7fec29e00700 1 -- 172.21.15.1:0/3477738664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec2c0fe0e0 msgr2=0x7fec2c1100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.494+0000 7fec29e00700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec2c0fe0e0 0x7fec2c1100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.494+0000 7fec29e00700 1 -- 172.21.15.1:0/3477738664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec14005040 con 0x7fec2c0fcef0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.494+0000 7fec29e00700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fcef0 0x7fec2c10fb60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00c8b0 tx=0x7fec1c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.495+0000 7fec22000700 1 -- 172.21.15.1:0/3477738664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec1c0106f0 con 0x7fec2c0fcef0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.495+0000 7fec22000700 1 -- 172.21.15.1:0/3477738664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec1c010d20 con 0x7fec2c0fcef0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.495+0000 7fec22000700 1 -- 172.21.15.1:0/3477738664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec1c012e90 con 0x7fec2c0fcef0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.495+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c111220 con 0x7fec2c0fcef0 2024-02-28T16:22:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.495+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c111710 con 0x7fec2c0fcef0 2024-02-28T16:22:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.496+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c102210 con 0x7fec2c0fcef0 2024-02-28T16:22:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.497+0000 7fec22000700 1 -- 172.21.15.1:0/3477738664 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7fec1c010850 con 0x7fec2c0fcef0 2024-02-28T16:22:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.498+0000 7fec22000700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec100692f0 0x7fec1006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:22:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.498+0000 7fec22000700 1 -- 172.21.15.1:0/3477738664 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fec1c0862b0 con 0x7fec2c0fcef0 2024-02-28T16:22:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.498+0000 7fec23e00700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec100692f0 0x7fec1006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:22:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.499+0000 7fec23e00700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec100692f0 0x7fec1006b7a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fec14004f90 tx=0x7fec14009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:22:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.503+0000 7fec22000700 1 -- 172.21.15.1:0/3477738664 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fec1c056e50 con 0x7fec2c0fcef0 2024-02-28T16:22:10.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:10.729+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fec2c111a80 con 0x7fec100692f0 2024-02-28T16:22:11.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:11.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:22:11.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:22:11.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:22:11.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:11.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:11.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:11.747 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:22:11.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:11.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:11.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:22:11.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:11.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:22:11.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:22:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:11.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:22:11.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:22:12.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:12 smithi066 ceph-mon[36236]: Detected new or changed devices on smithi001 2024-02-28T16:22:12.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:12 smithi066 ceph-mon[36236]: Adjusting osd_memory_target on smithi001 to 12871M 2024-02-28T16:22:12.753 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:12 smithi001 ceph-mon[28991]: Detected new or changed devices on smithi001 2024-02-28T16:22:12.753 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:12 smithi001 ceph-mon[28991]: Adjusting osd_memory_target on smithi001 to 12871M 2024-02-28T16:22:13.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:13 smithi066 ceph-mon[36236]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:14.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:13 smithi001 ceph-mon[28991]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:15.215 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:14 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:15.215 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:14 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:15.215 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:14 smithi001 ceph-mon[28991]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:15.215 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:14 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/133589707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfd6ea17-fd9b-46ae-b3ff-547d736cd232"}]: dispatch 2024-02-28T16:22:15.215 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:14 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/133589707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfd6ea17-fd9b-46ae-b3ff-547d736cd232"}]': finished 2024-02-28T16:22:15.215 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:14 smithi001 ceph-mon[28991]: osdmap e11: 2 total, 1 up, 2 in 2024-02-28T16:22:15.215 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:14 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:15.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:14 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:15.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:14 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:15.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:14 smithi066 ceph-mon[36236]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:15.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:14 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/133589707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfd6ea17-fd9b-46ae-b3ff-547d736cd232"}]: dispatch 2024-02-28T16:22:15.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:14 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/133589707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfd6ea17-fd9b-46ae-b3ff-547d736cd232"}]': finished 2024-02-28T16:22:15.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:14 smithi066 ceph-mon[36236]: osdmap e11: 2 total, 1 up, 2 in 2024-02-28T16:22:15.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:14 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:16.393 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:16 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2419003742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:22:16.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:16 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2419003742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:22:17.476 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:17 smithi001 ceph-mon[28991]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:17.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:17 smithi066 ceph-mon[36236]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:19.224 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:19.224 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:19.224 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:19.224 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:22:19.224 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:19.225 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:19 smithi001 ceph-mon[28991]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:19.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:19.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:19.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:19.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:22:19.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:19.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:19 smithi066 ceph-mon[36236]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:21.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:21 smithi001 ceph-mon[28991]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:21.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:21 smithi066 ceph-mon[36236]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:23.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:23 smithi066 ceph-mon[36236]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:23 smithi001 ceph-mon[28991]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:24.743 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-28T16:22:24.743 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:24.743 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:24 smithi001 ceph-mon[28991]: Deploying daemon osd.1 on smithi001 2024-02-28T16:22:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-28T16:22:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:24 smithi066 ceph-mon[36236]: Deploying daemon osd.1 on smithi001 2024-02-28T16:22:25.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:25 smithi066 ceph-mon[36236]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:25.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:25 smithi001 ceph-mon[28991]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:26.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:22:26.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:22:27.618 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:27 smithi001 ceph-mon[28991]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:27.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:27 smithi066 ceph-mon[36236]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:29.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:29 smithi066 ceph-mon[36236]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:29.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:29 smithi001 ceph-mon[28991]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:31 smithi066 ceph-mon[36236]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:31.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:31 smithi001 ceph-mon[28991]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:32.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:32.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:32.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:22:32.822 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:32.822 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:32.822 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:22:33.798 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:33 smithi066 ceph-mon[36236]: pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:33.899 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:33 smithi001 ceph-mon[28991]: pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:35.549 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:35 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:35.550 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:35 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:35.550 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:35 smithi001 ceph-mon[28991]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:35 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:35 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:35 smithi066 ceph-mon[36236]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:36.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.272+0000 7fec22000700 1 -- 172.21.15.1:0/3477738664 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fec2c111a80 con 0x7fec100692f0 2024-02-28T16:22:36.263 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 1 on host 'smithi001' 2024-02-28T16:22:36.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec100692f0 msgr2=0x7fec1006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:22:36.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec100692f0 0x7fec1006b7a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fec14004f90 tx=0x7fec14009e30 comp rx=0 tx=0).stop 2024-02-28T16:22:36.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fcef0 msgr2=0x7fec2c10fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:22:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fcef0 0x7fec2c10fb60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00c8b0 tx=0x7fec1c004470 comp rx=0 tx=0).stop 2024-02-28T16:22:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 shutdown_connections 2024-02-28T16:22:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fec100692f0 0x7fec1006b7a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec2c0fcef0 0x7fec2c10fb60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 --2- 172.21.15.1:0/3477738664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec2c0fe0e0 0x7fec2c1100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:36.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.276+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 >> 172.21.15.1:0/3477738664 conn(0x7fec2c0f8620 msgr2=0x7fec2c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:22:36.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.277+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 shutdown_connections 2024-02-28T16:22:36.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:36.277+0000 7fec31000700 1 -- 172.21.15.1:0/3477738664 wait complete. 2024-02-28T16:22:36.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:36 smithi001 ceph-mon[28991]: from='osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-28T16:22:36.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:36.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:36 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:36 smithi066 ceph-mon[36236]: from='osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-28T16:22:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:36 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:36 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:37.126 DEBUG:teuthology.orchestra.run.smithi001:osd.1> sudo journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.1.service 2024-02-28T16:22:37.133 INFO:tasks.cephadm:Deploying osd.2 on smithi001 with /dev/vg_nvme/lv_2... 2024-02-28T16:22:37.134 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-28T16:22:37.576 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:37 smithi001 ceph-mon[28991]: from='osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-28T16:22:37.576 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:37 smithi001 ceph-mon[28991]: osdmap e12: 2 total, 1 up, 2 in 2024-02-28T16:22:37.576 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:37 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:37.576 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:37 smithi001 ceph-mon[28991]: from='osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:22:37.577 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:37 smithi001 ceph-mon[28991]: pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:37.577 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:22:37 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[54704]: 2024-02-28T16:22:37.337+0000 7f0f4f400700 -1 osd.1 0 waiting for initial osdmap 2024-02-28T16:22:37.577 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:22:37 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[54704]: 2024-02-28T16:22:37.346+0000 7f0f4a400700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-28T16:22:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:37 smithi066 ceph-mon[36236]: from='osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-28T16:22:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:37 smithi066 ceph-mon[36236]: osdmap e12: 2 total, 1 up, 2 in 2024-02-28T16:22:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:37 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:37 smithi066 ceph-mon[36236]: from='osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:22:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:37 smithi066 ceph-mon[36236]: pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:37.898 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:22:38.696 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:38 smithi001 ceph-mon[28991]: from='osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-02-28T16:22:38.696 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:38 smithi001 ceph-mon[28991]: osdmap e13: 2 total, 1 up, 2 in 2024-02-28T16:22:38.696 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:38.697 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:38.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:38 smithi066 ceph-mon[36236]: from='osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-02-28T16:22:38.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:38 smithi066 ceph-mon[36236]: osdmap e13: 2 total, 1 up, 2 in 2024-02-28T16:22:38.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:38.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:39.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:39 smithi066 ceph-mon[36236]: purged_snaps scrub starts 2024-02-28T16:22:39.652 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:39 smithi066 ceph-mon[36236]: purged_snaps scrub ok 2024-02-28T16:22:39.652 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:39 smithi066 ceph-mon[36236]: osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] boot 2024-02-28T16:22:39.652 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:39 smithi066 ceph-mon[36236]: osdmap e14: 2 total, 2 up, 2 in 2024-02-28T16:22:39.652 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:39.652 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:39 smithi066 ceph-mon[36236]: pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:39.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:39 smithi001 ceph-mon[28991]: purged_snaps scrub starts 2024-02-28T16:22:39.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:39 smithi001 ceph-mon[28991]: purged_snaps scrub ok 2024-02-28T16:22:39.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:39 smithi001 ceph-mon[28991]: osd.1 [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] boot 2024-02-28T16:22:39.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:39 smithi001 ceph-mon[28991]: osdmap e14: 2 total, 2 up, 2 in 2024-02-28T16:22:39.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:22:39.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:39 smithi001 ceph-mon[28991]: pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-28T16:22:40.446 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:40 smithi001 ceph-mon[28991]: osdmap e15: 2 total, 2 up, 2 in 2024-02-28T16:22:40.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:40 smithi066 ceph-mon[36236]: osdmap e15: 2 total, 2 up, 2 in 2024-02-28T16:22:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:41 smithi066 ceph-mon[36236]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:41.753 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:41 smithi001 ceph-mon[28991]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:41.753 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:41.753 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-28T16:22:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-28T16:22:41.753 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-02-28T16:22:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-02-28T16:22:41.753 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0339365 s, 309 MB/s 2024-02-28T16:22:41.754 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-02-28T16:22:42.479 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:42 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:22:42.546 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi001:vg_nvme/lv_2 2024-02-28T16:22:42.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:42 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:22:43.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:43 smithi001 ceph-mon[28991]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:43.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:43 smithi066 ceph-mon[36236]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:43.874 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:22:45.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:45 smithi001 ceph-mon[28991]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:45 smithi066 ceph-mon[36236]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.613+0000 7f3929e00700 1 -- 172.21.15.1:0/2917793201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 msgr2=0x7f39240fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:22:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.613+0000 7f3929e00700 1 --2- 172.21.15.1:0/2917793201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 0x7f39240fd320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3914004940 tx=0x7f3914009e30 comp rx=0 tx=0).stop 2024-02-28T16:22:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.613+0000 7f3929e00700 1 -- 172.21.15.1:0/2917793201 shutdown_connections 2024-02-28T16:22:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.613+0000 7f3929e00700 1 --2- 172.21.15.1:0/2917793201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f39240fe0e0 0x7f39240fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.613+0000 7f3929e00700 1 --2- 172.21.15.1:0/2917793201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 0x7f39240fd320 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.613+0000 7f3929e00700 1 -- 172.21.15.1:0/2917793201 >> 172.21.15.1:0/2917793201 conn(0x7f39240f8620 msgr2=0x7f39240faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:22:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.613+0000 7f3929e00700 1 -- 172.21.15.1:0/2917793201 shutdown_connections 2024-02-28T16:22:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.614+0000 7f3929e00700 1 -- 172.21.15.1:0/2917793201 wait complete. 2024-02-28T16:22:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.614+0000 7f3929e00700 1 Processor -- start 2024-02-28T16:22:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.615+0000 7f3929e00700 1 -- start start 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.615+0000 7f3929e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 0x7f392410fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.615+0000 7f3929e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f39240fe0e0 0x7f39241100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.615+0000 7f3929e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241106d0 con 0x7f39240fcef0 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.615+0000 7f3929e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3924110f70 con 0x7f39240fe0e0 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.615+0000 7f3922a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 0x7f392410fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.615+0000 7f3922a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 0x7f392410fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44670/0 (socket says 172.21.15.1:44670) 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.615+0000 7f3922a00700 1 -- 172.21.15.1:0/920033124 learned_addr learned my addr 172.21.15.1:0/920033124 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.616+0000 7f3922000700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f39240fe0e0 0x7f39241100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:22:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.616+0000 7f3922a00700 1 -- 172.21.15.1:0/920033124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f39240fe0e0 msgr2=0x7f39241100d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:22:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.616+0000 7f3922a00700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f39240fe0e0 0x7f39241100d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:22:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.616+0000 7f3922a00700 1 -- 172.21.15.1:0/920033124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3914005040 con 0x7f39240fcef0 2024-02-28T16:22:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.616+0000 7f3922a00700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 0x7f392410fb90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3914000c00 tx=0x7f3914009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:22:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.617+0000 7f391aa00700 1 -- 172.21.15.1:0/920033124 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f391400e040 con 0x7f39240fcef0 2024-02-28T16:22:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.617+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39241111f0 con 0x7f39240fcef0 2024-02-28T16:22:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.617+0000 7f391aa00700 1 -- 172.21.15.1:0/920033124 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f391400bcc0 con 0x7f39240fcef0 2024-02-28T16:22:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.617+0000 7f391aa00700 1 -- 172.21.15.1:0/920033124 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3914013ea0 con 0x7f39240fcef0 2024-02-28T16:22:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.617+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3924111730 con 0x7f39240fcef0 2024-02-28T16:22:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.618+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924102210 con 0x7f39240fcef0 2024-02-28T16:22:46.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.619+0000 7f391aa00700 1 -- 172.21.15.1:0/920033124 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f3914013420 con 0x7f39240fcef0 2024-02-28T16:22:46.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.620+0000 7f391aa00700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3908069210 0x7f390806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:22:46.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.621+0000 7f391aa00700 1 -- 172.21.15.1:0/920033124 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f39140878d0 con 0x7f39240fcef0 2024-02-28T16:22:46.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.621+0000 7f3922000700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3908069210 0x7f390806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:22:46.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.622+0000 7f3922000700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3908069210 0x7f390806b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3910007fd0 tx=0x7f3910009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:22:46.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.626+0000 7f391aa00700 1 -- 172.21.15.1:0/920033124 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3914058240 con 0x7f39240fcef0 2024-02-28T16:22:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:22:46.874+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f392402b310 con 0x7f3908069210 2024-02-28T16:22:47.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:47 smithi066 ceph-mon[36236]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:47.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:22:47.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:22:47.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:47.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:47 smithi001 ceph-mon[28991]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:47.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:22:47.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:22:47.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:48.894 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:22:48.894 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: Detected new or changed devices on smithi001 2024-02-28T16:22:48.894 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:48.894 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:48.894 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:22:48.894 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:22:48.895 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: Adjusting osd_memory_target on smithi001 to 6435M 2024-02-28T16:22:48.895 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:48.895 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:48.895 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:22:48.895 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:48.895 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:22:48.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:22:48.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: Detected new or changed devices on smithi001 2024-02-28T16:22:48.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: Adjusting osd_memory_target on smithi001 to 6435M 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:22:49.830 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:49 smithi001 ceph-mon[28991]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:49.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:49 smithi066 ceph-mon[36236]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:51.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:51 smithi066 ceph-mon[36236]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:51.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:51 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2767936647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b713f513-6604-4e88-8cc1-0f963d9f8599"}]: dispatch 2024-02-28T16:22:51.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:51 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2767936647' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b713f513-6604-4e88-8cc1-0f963d9f8599"}]': finished 2024-02-28T16:22:51.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:51 smithi066 ceph-mon[36236]: osdmap e16: 3 total, 2 up, 3 in 2024-02-28T16:22:51.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:51 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:22:51.931 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:51 smithi001 ceph-mon[28991]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:51.931 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:51 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2767936647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b713f513-6604-4e88-8cc1-0f963d9f8599"}]: dispatch 2024-02-28T16:22:51.931 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:51 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2767936647' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b713f513-6604-4e88-8cc1-0f963d9f8599"}]': finished 2024-02-28T16:22:51.931 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:51 smithi001 ceph-mon[28991]: osdmap e16: 3 total, 2 up, 3 in 2024-02-28T16:22:51.931 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:51 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:22:52.903 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:52 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3695420298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:22:52.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:52 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3695420298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:22:53.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:53 smithi066 ceph-mon[36236]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:54.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:53 smithi001 ceph-mon[28991]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:55.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:55 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:55.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:55 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:55.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:55 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:55.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:55 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:22:55.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:55 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:55.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:55 smithi001 ceph-mon[28991]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:55.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:55 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:55.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:55 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:55.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:55 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:22:55.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:55 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:22:55.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:55 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:22:55.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:55 smithi066 ceph-mon[36236]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:56.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:22:56.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:22:57.718 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:57 smithi001 ceph-mon[28991]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:57 smithi066 ceph-mon[36236]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:59.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:22:59 smithi001 ceph-mon[28991]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:22:59.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:22:59 smithi066 ceph-mon[36236]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:00.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:00 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-28T16:23:00.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:00 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:00 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-28T16:23:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:00 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:01.718 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:01 smithi001 ceph-mon[28991]: Deploying daemon osd.2 on smithi001 2024-02-28T16:23:01.719 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:01 smithi001 ceph-mon[28991]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:01 smithi066 ceph-mon[36236]: Deploying daemon osd.2 on smithi001 2024-02-28T16:23:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:01 smithi066 ceph-mon[36236]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:03 smithi066 ceph-mon[36236]: pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:03 smithi001 ceph-mon[28991]: pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:05.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:05 smithi001 ceph-mon[28991]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:05 smithi066 ceph-mon[36236]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:07 smithi001 ceph-mon[28991]: pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:07.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:07 smithi066 ceph-mon[36236]: pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:09.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:09.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:23:09.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:09.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:09 smithi066 ceph-mon[36236]: pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:09.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:09.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:23:09.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:09.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:09 smithi001 ceph-mon[28991]: pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:11.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:11 smithi066 ceph-mon[36236]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:11.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:23:11.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:11 smithi001 ceph-mon[28991]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:11.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:23:12.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:12 smithi066 ceph-mon[36236]: from='osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-28T16:23:12.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:12 smithi001 ceph-mon[28991]: from='osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-28T16:23:12.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.918+0000 7f391aa00700 1 -- 172.21.15.1:0/920033124 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f392402b310 con 0x7f3908069210 2024-02-28T16:23:12.908 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 2 on host 'smithi001' 2024-02-28T16:23:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3908069210 msgr2=0x7f390806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:23:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3908069210 0x7f390806b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3910007fd0 tx=0x7f3910009040 comp rx=0 tx=0).stop 2024-02-28T16:23:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 msgr2=0x7f392410fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:23:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 0x7f392410fb90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3914000c00 tx=0x7f3914009e30 comp rx=0 tx=0).stop 2024-02-28T16:23:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 shutdown_connections 2024-02-28T16:23:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3908069210 0x7f390806b6c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39240fcef0 0x7f392410fb90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 --2- 172.21.15.1:0/920033124 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f39240fe0e0 0x7f39241100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.922+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 >> 172.21.15.1:0/920033124 conn(0x7f39240f8620 msgr2=0x7f39240ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:23:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.923+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 shutdown_connections 2024-02-28T16:23:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:23:12.923+0000 7f3929e00700 1 -- 172.21.15.1:0/920033124 wait complete. 2024-02-28T16:23:13.710 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:13 smithi001 ceph-mon[28991]: pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:13.710 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:13 smithi001 ceph-mon[28991]: from='osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-28T16:23:13.710 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:13 smithi001 ceph-mon[28991]: osdmap e17: 3 total, 2 up, 3 in 2024-02-28T16:23:13.710 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:23:13.710 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:13 smithi001 ceph-mon[28991]: from='osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:23:13.710 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:13.710 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:13 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:13 smithi066 ceph-mon[36236]: pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:13 smithi066 ceph-mon[36236]: from='osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-28T16:23:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:13 smithi066 ceph-mon[36236]: osdmap e17: 3 total, 2 up, 3 in 2024-02-28T16:23:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:23:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:13 smithi066 ceph-mon[36236]: from='osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:23:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:13.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:13 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:14.044 DEBUG:teuthology.orchestra.run.smithi001:osd.2> sudo journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.2.service 2024-02-28T16:23:14.048 INFO:tasks.cephadm:Deploying osd.3 on smithi066 with /dev/vg_nvme/lv_4... 2024-02-28T16:23:14.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-28T16:23:14.557 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T16:23:14.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:14 smithi001 ceph-mon[28991]: from='osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-02-28T16:23:14.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:14 smithi001 ceph-mon[28991]: osdmap e18: 3 total, 2 up, 3 in 2024-02-28T16:23:14.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:14 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:23:14.723 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:14 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:23:14.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:14 smithi066 ceph-mon[36236]: from='osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-02-28T16:23:14.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:14 smithi066 ceph-mon[36236]: osdmap e18: 3 total, 2 up, 3 in 2024-02-28T16:23:14.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:14 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:23:14.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:14 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:23:15.653 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:15 smithi001 ceph-mon[28991]: purged_snaps scrub starts 2024-02-28T16:23:15.654 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:15 smithi001 ceph-mon[28991]: purged_snaps scrub ok 2024-02-28T16:23:15.654 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:15 smithi001 ceph-mon[28991]: pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:15.654 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:15 smithi001 ceph-mon[28991]: osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] boot 2024-02-28T16:23:15.654 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:15 smithi001 ceph-mon[28991]: osdmap e19: 3 total, 3 up, 3 in 2024-02-28T16:23:15.654 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:23:15.654 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-28T16:23:15.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:15 smithi066 ceph-mon[36236]: purged_snaps scrub starts 2024-02-28T16:23:15.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:15 smithi066 ceph-mon[36236]: purged_snaps scrub ok 2024-02-28T16:23:15.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:15 smithi066 ceph-mon[36236]: pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-28T16:23:15.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:15 smithi066 ceph-mon[36236]: osd.2 [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] boot 2024-02-28T16:23:15.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:15 smithi066 ceph-mon[36236]: osdmap e19: 3 total, 3 up, 3 in 2024-02-28T16:23:15.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:23:15.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-28T16:23:16.733 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:16 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-28T16:23:16.733 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:16 smithi066 ceph-mon[36236]: osdmap e20: 3 total, 3 up, 3 in 2024-02-28T16:23:16.733 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:16 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-28T16:23:16.733 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:16 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:16.733 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:16 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:16.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:16 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-28T16:23:16.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:16 smithi001 ceph-mon[28991]: osdmap e20: 3 total, 3 up, 3 in 2024-02-28T16:23:16.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:16 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-28T16:23:16.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:16 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:16.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:16 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:17.676 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-28T16:23:17.676 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-28T16:23:17.676 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-02-28T16:23:17.676 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-02-28T16:23:17.676 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0331859 s, 316 MB/s 2024-02-28T16:23:17.677 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-02-28T16:23:17.769 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:17 smithi066 ceph-mon[36236]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:17.769 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:17 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-28T16:23:17.769 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:17 smithi066 ceph-mon[36236]: osdmap e21: 3 total, 3 up, 3 in 2024-02-28T16:23:17.769 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:17 smithi066 ceph-mon[36236]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-28T16:23:17.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:17 smithi001 ceph-mon[28991]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:17.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:17 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-28T16:23:17.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:17 smithi001 ceph-mon[28991]: osdmap e21: 3 total, 3 up, 3 in 2024-02-28T16:23:17.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:17 smithi001 ceph-mon[28991]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-28T16:23:17.771 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:23:17 smithi001 sudo[64283]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-28T16:23:17.771 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:23:17 smithi001 sudo[64283]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:17.771 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:23:17 smithi001 sudo[64283]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:17.771 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:23:17 smithi001 sudo[64283]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:18.345 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:23:18 smithi001 sudo[64357]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-28T16:23:18.345 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:23:18 smithi001 sudo[64357]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:18.345 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:23:18 smithi001 sudo[64357]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:18.345 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:23:18 smithi001 sudo[64357]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:18.624 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_4 2024-02-28T16:23:18.716 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:18 smithi001 ceph-mon[28991]: osdmap e22: 3 total, 3 up, 3 in 2024-02-28T16:23:18.716 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:18 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:18.972 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:23:18 smithi001 sudo[64382]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-28T16:23:18.972 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:23:18 smithi001 sudo[64382]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:18.972 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:23:18 smithi001 sudo[64382]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:18.972 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:23:18 smithi001 sudo[64382]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:18.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:18 smithi066 ceph-mon[36236]: osdmap e22: 3 total, 3 up, 3 in 2024-02-28T16:23:18.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:18 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:19.125 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T16:23:19.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:19 smithi066 ceph-mon[36236]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:19.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:19 smithi066 ceph-mon[36236]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-28T16:23:19.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:19 smithi066 ceph-mon[36236]: Cluster is now healthy 2024-02-28T16:23:19.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:19 smithi066 ceph-mon[36236]: mgrmap e19: smithi001.rmthtx(active, since 2m), standbys: smithi066.ztraxe 2024-02-28T16:23:19.665 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:19 smithi001 ceph-mon[28991]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:19.666 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:19 smithi001 ceph-mon[28991]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-28T16:23:19.666 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:19 smithi001 ceph-mon[28991]: Cluster is now healthy 2024-02-28T16:23:19.666 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:19 smithi001 ceph-mon[28991]: mgrmap e19: smithi001.rmthtx(active, since 2m), standbys: smithi066.ztraxe 2024-02-28T16:23:19.666 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:23:19 smithi001 sudo[64385]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-28T16:23:19.666 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:23:19 smithi001 sudo[64385]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:19.666 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:23:19 smithi001 sudo[64385]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:19.666 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:23:19 smithi001 sudo[64385]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:19.926 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:23:19 smithi001 sudo[64414]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-28T16:23:19.926 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:23:19 smithi001 sudo[64414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:19.926 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:23:19 smithi001 sudo[64414]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:19.926 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:23:19 smithi001 sudo[64414]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:20.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:23:20 smithi001 sudo[64424]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-28T16:23:20.771 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:23:20 smithi001 sudo[64424]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:20.771 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:23:20 smithi001 sudo[64424]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:20.771 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:23:20 smithi001 sudo[64424]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:21.344 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.353+0000 7f03cfe00700 1 -- 172.21.15.66:0/14538248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 msgr2=0x7f03c8102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:23:21.344 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.353+0000 7f03cfe00700 1 --2- 172.21.15.66:0/14538248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 0x7f03c8102780 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f03b4004940 tx=0x7f03b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:23:21.344 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.353+0000 7f03cfe00700 1 -- 172.21.15.66:0/14538248 shutdown_connections 2024-02-28T16:23:21.344 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.353+0000 7f03cfe00700 1 --2- 172.21.15.66:0/14538248 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 0x7f03c8102780 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:21.344 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.353+0000 7f03cfe00700 1 --2- 172.21.15.66:0/14538248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fda80 0x7f03c80ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:21.344 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.353+0000 7f03cfe00700 1 -- 172.21.15.66:0/14538248 >> 172.21.15.66:0/14538248 conn(0x7f03c80f74e0 msgr2=0x7f03c80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:23:21.344 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.353+0000 7f03cfe00700 1 -- 172.21.15.66:0/14538248 shutdown_connections 2024-02-28T16:23:21.344 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.353+0000 7f03cfe00700 1 -- 172.21.15.66:0/14538248 wait complete. 2024-02-28T16:23:21.345 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.354+0000 7f03cfe00700 1 Processor -- start 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cfe00700 1 -- start start 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fda80 0x7f03c810fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cfe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 0x7f03c8110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cfe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8110710 con 0x7f03c80fda80 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cfe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8110f50 con 0x7f03c81003a0 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cca00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 0x7f03c8110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cca00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 0x7f03c8110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.66:44112/0 (socket says 172.21.15.66:44112) 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cca00700 1 -- 172.21.15.66:0/3883908818 learned_addr learned my addr 172.21.15.66:0/3883908818 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.355+0000 7f03cd400700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fda80 0x7f03c810fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.356+0000 7f03cca00700 1 -- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fda80 msgr2=0x7f03c810fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.356+0000 7f03cca00700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fda80 0x7f03c810fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.356+0000 7f03cca00700 1 -- 172.21.15.66:0/3883908818 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b4005040 con 0x7f03c81003a0 2024-02-28T16:23:21.346 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.356+0000 7f03cca00700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 0x7f03c8110080 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009fd0 tx=0x7f03b4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:23:21.348 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.356+0000 7f03c5600700 1 -- 172.21.15.66:0/3883908818 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03b400e040 con 0x7f03c81003a0 2024-02-28T16:23:21.348 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.356+0000 7f03c5600700 1 -- 172.21.15.66:0/3883908818 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03b400bcc0 con 0x7f03c81003a0 2024-02-28T16:23:21.348 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.356+0000 7f03c5600700 1 -- 172.21.15.66:0/3883908818 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03b401c630 con 0x7f03c81003a0 2024-02-28T16:23:21.348 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.356+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c81111d0 con 0x7f03c81003a0 2024-02-28T16:23:21.348 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.357+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c8111710 con 0x7f03c81003a0 2024-02-28T16:23:21.348 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.358+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c804bed0 con 0x7f03c81003a0 2024-02-28T16:23:21.349 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.359+0000 7f03c5600700 1 -- 172.21.15.66:0/3883908818 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f03b4013980 con 0x7f03c81003a0 2024-02-28T16:23:21.350 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.360+0000 7f03c5600700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f03b00692b0 0x7f03b006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:23:21.350 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.360+0000 7f03c5600700 1 -- 172.21.15.66:0/3883908818 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f03b4087d80 con 0x7f03c81003a0 2024-02-28T16:23:21.350 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.360+0000 7f03cd400700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f03b00692b0 0x7f03b006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:23:21.351 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.361+0000 7f03cd400700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f03b00692b0 0x7f03b006b760 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03bc006fd0 tx=0x7f03bc008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:23:21.354 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.365+0000 7f03c5600700 1 -- 172.21.15.66:0/3883908818 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f03b40583e0 con 0x7f03c81003a0 2024-02-28T16:23:21.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:21 smithi001 sudo[64450]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-28T16:23:21.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:21 smithi001 sudo[64450]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:21.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:21 smithi001 sudo[64450]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:21.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:21 smithi001 sudo[64450]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:21.578 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:21.588+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f03c80008d0 con 0x7f03b00692b0 2024-02-28T16:23:22.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:21 smithi001 sudo[66595]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-28T16:23:22.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:21 smithi001 sudo[66595]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:22.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:21 smithi001 sudo[66595]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:22.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:21 smithi001 sudo[66595]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:22.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:22 smithi066 sudo[39530]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-28T16:23:22.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:22 smithi066 sudo[39530]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:22.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:22 smithi066 sudo[39530]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:22.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:22 smithi066 sudo[39530]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 sudo[39554]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 sudo[39554]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 sudo[39554]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 sudo[39554]: pam_unix(sudo:session): session closed for user root 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:23:23.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:23:23.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:23:23.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:23:23.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:23.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:23:23.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:23 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:23:23.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:23.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-28T16:23:23.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-28T16:23:23.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:23:23.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:23:23.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:23:23.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:23:23.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:23.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:23:23.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:23 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:23:24.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='client.24123 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:23:24.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-28T16:23:24.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:24.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: Detected new or changed devices on smithi001 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: Adjusting osd_memory_target on smithi001 to 4290M 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:23:24.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:23:24.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='client.24123 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:23:24.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-28T16:23:24.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:24.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: Detected new or changed devices on smithi001 2024-02-28T16:23:24.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-28T16:23:24.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: Adjusting osd_memory_target on smithi001 to 4290M 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:24.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:23:25.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:25 smithi001 ceph-mon[28991]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-28T16:23:25.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:25 smithi066 ceph-mon[36236]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-28T16:23:26.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:26 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/1781201100' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f425b10-672f-4265-a314-5a9a15237257"}]: dispatch 2024-02-28T16:23:26.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:26 smithi001 ceph-mon[28991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f425b10-672f-4265-a314-5a9a15237257"}]: dispatch 2024-02-28T16:23:26.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:26 smithi001 ceph-mon[28991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f425b10-672f-4265-a314-5a9a15237257"}]': finished 2024-02-28T16:23:26.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:26 smithi001 ceph-mon[28991]: osdmap e23: 4 total, 3 up, 4 in 2024-02-28T16:23:26.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:26.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:26 smithi066 ceph-mon[36236]: from='client.? 172.21.15.66:0/1781201100' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f425b10-672f-4265-a314-5a9a15237257"}]: dispatch 2024-02-28T16:23:26.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:26 smithi066 ceph-mon[36236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f425b10-672f-4265-a314-5a9a15237257"}]: dispatch 2024-02-28T16:23:26.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:26 smithi066 ceph-mon[36236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f425b10-672f-4265-a314-5a9a15237257"}]': finished 2024-02-28T16:23:26.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:26 smithi066 ceph-mon[36236]: osdmap e23: 4 total, 3 up, 4 in 2024-02-28T16:23:26.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:27.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:27 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/2727265576' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:23:27.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:27 smithi001 ceph-mon[28991]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-28T16:23:27.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:27 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:23:27.656 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:27 smithi066 ceph-mon[36236]: from='client.? 172.21.15.66:0/2727265576' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:23:27.656 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:27 smithi066 ceph-mon[36236]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-28T16:23:27.656 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:27 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:23:29.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:29 smithi066 ceph-mon[36236]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-28T16:23:29.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:29 smithi001 ceph-mon[28991]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-28T16:23:30.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:30.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:30.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:30.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:23:30.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:23:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:31 smithi066 ceph-mon[36236]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-28T16:23:32.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:31 smithi001 ceph-mon[28991]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-28T16:23:33.955 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:33 smithi066 ceph-mon[36236]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-28T16:23:34.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:33 smithi001 ceph-mon[28991]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-28T16:23:35.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:35 smithi066 ceph-mon[36236]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:35.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:35 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-28T16:23:35.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:35 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:35.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:35 smithi066 ceph-mon[36236]: Deploying daemon osd.3 on smithi066 2024-02-28T16:23:35.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:35 smithi001 ceph-mon[28991]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:35.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:35 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-28T16:23:35.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:35 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:35.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:35 smithi001 ceph-mon[28991]: Deploying daemon osd.3 on smithi066 2024-02-28T16:23:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:37 smithi066 ceph-mon[36236]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:37.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:37 smithi001 ceph-mon[28991]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:39.714 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:39 smithi066 ceph-mon[36236]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:39.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:39 smithi001 ceph-mon[28991]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:41 smithi066 ceph-mon[36236]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:23:41.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:41 smithi001 ceph-mon[28991]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:41.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:23:43.382 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:43 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:43.382 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:43 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:23:43.382 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:43 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:43.383 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:43 smithi066 ceph-mon[36236]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:43.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:43 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:43.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:43 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:23:43.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:43 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:43.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:43 smithi001 ceph-mon[28991]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:45 smithi066 ceph-mon[36236]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:45.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:45 smithi001 ceph-mon[28991]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:46.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:46 smithi066 ceph-mon[36236]: from='osd.3 [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-28T16:23:46.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:46 smithi066 ceph-mon[36236]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-28T16:23:46.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:46 smithi001 ceph-mon[28991]: from='osd.3 [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-28T16:23:46.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:46 smithi001 ceph-mon[28991]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-28T16:23:47.389 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 3 on host 'smithi066' 2024-02-28T16:23:47.390 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.399+0000 7f03c5600700 1 -- 172.21.15.66:0/3883908818 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f03c80008d0 con 0x7f03b00692b0 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f03b00692b0 msgr2=0x7f03b006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f03b00692b0 0x7f03b006b760 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03bc006fd0 tx=0x7f03bc008040 comp rx=0 tx=0).stop 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 msgr2=0x7f03c8110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 0x7f03c8110080 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009fd0 tx=0x7f03b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 shutdown_connections 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f03b00692b0 0x7f03b006b760 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fda80 0x7f03c810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 --2- 172.21.15.66:0/3883908818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c81003a0 0x7f03c8110080 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 >> 172.21.15.66:0/3883908818 conn(0x7f03c80f74e0 msgr2=0x7f03c80fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 shutdown_connections 2024-02-28T16:23:47.396 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:47.402+0000 7f03cfe00700 1 -- 172.21.15.66:0/3883908818 wait complete. 2024-02-28T16:23:47.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:47 smithi001 ceph-mon[28991]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-28T16:23:47.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:47 smithi001 ceph-mon[28991]: osdmap e24: 4 total, 3 up, 4 in 2024-02-28T16:23:47.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:47.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:47 smithi001 ceph-mon[28991]: from='osd.3 [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:23:47.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:47 smithi001 ceph-mon[28991]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:23:47.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:47 smithi001 ceph-mon[28991]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:47.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:47.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:47.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:47 smithi066 ceph-mon[36236]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-28T16:23:47.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:47 smithi066 ceph-mon[36236]: osdmap e24: 4 total, 3 up, 4 in 2024-02-28T16:23:47.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:47.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:47 smithi066 ceph-mon[36236]: from='osd.3 [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:23:47.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:47 smithi066 ceph-mon[36236]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:23:47.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:47 smithi066 ceph-mon[36236]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:47.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:47.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:48.269 DEBUG:teuthology.orchestra.run.smithi066:osd.3> sudo journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.3.service 2024-02-28T16:23:48.274 INFO:tasks.cephadm:Deploying osd.4 on smithi066 with /dev/vg_nvme/lv_3... 2024-02-28T16:23:48.274 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-28T16:23:48.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:48 smithi066 ceph-mon[36236]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-02-28T16:23:48.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:48 smithi066 ceph-mon[36236]: osdmap e25: 4 total, 3 up, 4 in 2024-02-28T16:23:48.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:48.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:48.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:48.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:48.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:48 smithi001 ceph-mon[28991]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-02-28T16:23:48.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:48 smithi001 ceph-mon[28991]: osdmap e25: 4 total, 3 up, 4 in 2024-02-28T16:23:48.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:48.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:48.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:48.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:49.693 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:49 smithi066 ceph-mon[36236]: purged_snaps scrub starts 2024-02-28T16:23:49.693 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:49 smithi066 ceph-mon[36236]: purged_snaps scrub ok 2024-02-28T16:23:49.693 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:49 smithi066 ceph-mon[36236]: osd.3 [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] boot 2024-02-28T16:23:49.694 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:49 smithi066 ceph-mon[36236]: osdmap e26: 4 total, 4 up, 4 in 2024-02-28T16:23:49.694 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:49 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:49.694 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:49 smithi066 ceph-mon[36236]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:49.694 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:49 smithi066 ceph-mon[36236]: osdmap e27: 4 total, 4 up, 4 in 2024-02-28T16:23:49.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:49 smithi001 ceph-mon[28991]: purged_snaps scrub starts 2024-02-28T16:23:49.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:49 smithi001 ceph-mon[28991]: purged_snaps scrub ok 2024-02-28T16:23:49.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:49 smithi001 ceph-mon[28991]: osd.3 [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] boot 2024-02-28T16:23:49.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:49 smithi001 ceph-mon[28991]: osdmap e26: 4 total, 4 up, 4 in 2024-02-28T16:23:49.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:49 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:23:49.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:49 smithi001 ceph-mon[28991]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-28T16:23:49.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:49 smithi001 ceph-mon[28991]: osdmap e27: 4 total, 4 up, 4 in 2024-02-28T16:23:49.776 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T16:23:51.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:51 smithi066 ceph-mon[36236]: osdmap e28: 4 total, 4 up, 4 in 2024-02-28T16:23:51.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:51 smithi066 ceph-mon[36236]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:23:51.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:51 smithi001 ceph-mon[28991]: osdmap e28: 4 total, 4 up, 4 in 2024-02-28T16:23:51.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:51 smithi001 ceph-mon[28991]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:23:52.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:52 smithi066 ceph-mon[36236]: osdmap e29: 4 total, 4 up, 4 in 2024-02-28T16:23:52.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:52 smithi001 ceph-mon[28991]: osdmap e29: 4 total, 4 up, 4 in 2024-02-28T16:23:53.185 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-28T16:23:53.185 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-28T16:23:53.185 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-02-28T16:23:53.185 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-02-28T16:23:53.185 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0403173 s, 260 MB/s 2024-02-28T16:23:53.185 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-02-28T16:23:53.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:53 smithi066 ceph-mon[36236]: osdmap e30: 4 total, 4 up, 4 in 2024-02-28T16:23:53.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:53 smithi066 ceph-mon[36236]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:23:53.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:53 smithi001 ceph-mon[28991]: osdmap e30: 4 total, 4 up, 4 in 2024-02-28T16:23:53.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:53 smithi001 ceph-mon[28991]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:23:54.126 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_3 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: Detected new or changed devices on smithi066 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: Adjusting osd_memory_target on smithi066 to 15939M 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:23:54.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:54.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:23:54.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:54 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.641 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T16:23:54.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: Detected new or changed devices on smithi066 2024-02-28T16:23:54.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:23:54.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: Adjusting osd_memory_target on smithi066 to 15939M 2024-02-28T16:23:54.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:54.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:23:54.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:54.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:23:54.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:54.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:23:54.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:54 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:23:55.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:55 smithi066 ceph-mon[36236]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-28T16:23:55.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:55 smithi001 ceph-mon[28991]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-02-28T16:23:56.623 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:23:56.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:23:56.905 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.915+0000 7fa4f4400700 1 -- 172.21.15.66:0/3281097457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec103540 msgr2=0x7fa4ec103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:23:56.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.915+0000 7fa4f4400700 1 --2- 172.21.15.66:0/3281097457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec103540 0x7fa4ec103950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0004940 tx=0x7fa4e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:23:56.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.915+0000 7fa4f4400700 1 -- 172.21.15.66:0/3281097457 shutdown_connections 2024-02-28T16:23:56.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.915+0000 7fa4f4400700 1 --2- 172.21.15.66:0/3281097457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec104750 0x7fa4ec104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:56.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.915+0000 7fa4f4400700 1 --2- 172.21.15.66:0/3281097457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec103540 0x7fa4ec103950 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:56.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.915+0000 7fa4f4400700 1 -- 172.21.15.66:0/3281097457 >> 172.21.15.66:0/3281097457 conn(0x7fa4ec0727d0 msgr2=0x7fa4ec074c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:23:56.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.915+0000 7fa4f4400700 1 -- 172.21.15.66:0/3281097457 shutdown_connections 2024-02-28T16:23:56.906 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.915+0000 7fa4f4400700 1 -- 172.21.15.66:0/3281097457 wait complete. 2024-02-28T16:23:56.907 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.917+0000 7fa4f4400700 1 Processor -- start 2024-02-28T16:23:56.907 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.917+0000 7fa4f4400700 1 -- start start 2024-02-28T16:23:56.907 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f4400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec104750 0x7fa4ec114100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:23:56.908 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f4400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec114640 0x7fa4ec1154e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:23:56.908 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f4400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec114b00 con 0x7fa4ec114640 2024-02-28T16:23:56.908 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f4400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec114c40 con 0x7fa4ec104750 2024-02-28T16:23:56.908 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f1a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec104750 0x7fa4ec114100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:23:56.908 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f1a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec104750 0x7fa4ec114100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.66:60254/0 (socket says 172.21.15.66:60254) 2024-02-28T16:23:56.908 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f1a00700 1 -- 172.21.15.66:0/129072411 learned_addr learned my addr 172.21.15.66:0/129072411 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f1000700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec114640 0x7fa4ec1154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f1a00700 1 -- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec114640 msgr2=0x7fa4ec1154e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f1a00700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec114640 0x7fa4ec1154e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.918+0000 7fa4f1a00700 1 -- 172.21.15.66:0/129072411 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e0005040 con 0x7fa4ec104750 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.919+0000 7fa4f1a00700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec104750 0x7fa4ec114100 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0000c00 tx=0x7fa4e0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.919+0000 7fa4e9600700 1 -- 172.21.15.66:0/129072411 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4e000e040 con 0x7fa4ec104750 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.919+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec115ae0 con 0x7fa4ec104750 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.919+0000 7fa4e9600700 1 -- 172.21.15.66:0/129072411 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4e0004290 con 0x7fa4ec104750 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.919+0000 7fa4e9600700 1 -- 172.21.15.66:0/129072411 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4e001c3f0 con 0x7fa4ec104750 2024-02-28T16:23:56.910 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.919+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec115ff0 con 0x7fa4ec104750 2024-02-28T16:23:56.911 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.920+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec10e060 con 0x7fa4ec104750 2024-02-28T16:23:56.911 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.922+0000 7fa4e9600700 1 -- 172.21.15.66:0/129072411 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa4e000bbf0 con 0x7fa4ec104750 2024-02-28T16:23:56.912 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.923+0000 7fa4e9600700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa4d40692b0 0x7fa4d406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:23:56.912 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.923+0000 7fa4e9600700 1 -- 172.21.15.66:0/129072411 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fa4e0087f20 con 0x7fa4ec104750 2024-02-28T16:23:56.913 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.923+0000 7fa4f1000700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa4d40692b0 0x7fa4d406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:23:56.914 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.924+0000 7fa4f1000700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa4d40692b0 0x7fa4d406b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8004990 tx=0x7fa4d8009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:23:56.918 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:56.928+0000 7fa4e9600700 1 -- 172.21.15.66:0/129072411 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa4e00582e0 con 0x7fa4ec104750 2024-02-28T16:23:57.140 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:23:57.150+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4ec05e3d0 con 0x7fa4d40692b0 2024-02-28T16:23:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:57 smithi066 ceph-mon[36236]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-28T16:23:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:23:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:23:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:57.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:57 smithi001 ceph-mon[28991]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-28T16:23:57.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:23:57.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:23:57.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:23:58.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:58 smithi066 ceph-mon[36236]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:23:58.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:58 smithi001 ceph-mon[28991]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:23:59.690 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:23:59 smithi066 ceph-mon[36236]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-28T16:23:59.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:23:59 smithi001 ceph-mon[28991]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-28T16:24:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:01 smithi066 ceph-mon[36236]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-28T16:24:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:01 smithi066 ceph-mon[36236]: from='client.? 172.21.15.66:0/1324801174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f2651b-47f3-444e-9325-22a0a962c3b0"}]: dispatch 2024-02-28T16:24:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:01 smithi066 ceph-mon[36236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f2651b-47f3-444e-9325-22a0a962c3b0"}]: dispatch 2024-02-28T16:24:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:01 smithi066 ceph-mon[36236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f2651b-47f3-444e-9325-22a0a962c3b0"}]': finished 2024-02-28T16:24:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:01 smithi066 ceph-mon[36236]: osdmap e31: 5 total, 4 up, 5 in 2024-02-28T16:24:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:01 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:01 smithi001 ceph-mon[28991]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-28T16:24:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:01 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/1324801174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f2651b-47f3-444e-9325-22a0a962c3b0"}]: dispatch 2024-02-28T16:24:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:01 smithi001 ceph-mon[28991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f2651b-47f3-444e-9325-22a0a962c3b0"}]: dispatch 2024-02-28T16:24:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:01 smithi001 ceph-mon[28991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f2651b-47f3-444e-9325-22a0a962c3b0"}]': finished 2024-02-28T16:24:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:01 smithi001 ceph-mon[28991]: osdmap e31: 5 total, 4 up, 5 in 2024-02-28T16:24:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:01 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:02.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:02 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/4254034634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:24:02.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:02 smithi066 ceph-mon[36236]: from='client.? 172.21.15.66:0/4254034634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:24:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:03 smithi001 ceph-mon[28991]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-28T16:24:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:03 smithi066 ceph-mon[36236]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-28T16:24:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:05 smithi066 ceph-mon[36236]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:05 smithi001 ceph-mon[28991]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:07.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:07 smithi066 ceph-mon[36236]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:07.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:07 smithi001 ceph-mon[28991]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:09.638 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:09 smithi066 ceph-mon[36236]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:09.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:09 smithi001 ceph-mon[28991]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:10.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:10 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-28T16:24:10.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:10 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:10 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-28T16:24:10.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:10 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:11 smithi066 ceph-mon[36236]: Deploying daemon osd.4 on smithi066 2024-02-28T16:24:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:11 smithi066 ceph-mon[36236]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:24:11.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:11 smithi001 ceph-mon[28991]: Deploying daemon osd.4 on smithi066 2024-02-28T16:24:11.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:11 smithi001 ceph-mon[28991]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:11.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:24:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:13 smithi066 ceph-mon[36236]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:13.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:13 smithi001 ceph-mon[28991]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:15.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:15 smithi066 ceph-mon[36236]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:15.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:15 smithi001 ceph-mon[28991]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:17.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:17 smithi066 ceph-mon[36236]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:17.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:17 smithi001 ceph-mon[28991]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:19.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:19 smithi001 ceph-mon[28991]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:19.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:19.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:19.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:19 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:24:19.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:19 smithi066 ceph-mon[36236]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:19.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:19.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:19.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:19 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:24:21.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:21 smithi066 ceph-mon[36236]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:22.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:21 smithi001 ceph-mon[28991]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:22.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:22 smithi066 ceph-mon[36236]: from='osd.4 [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-28T16:24:22.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:22 smithi066 ceph-mon[36236]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-28T16:24:23.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:22 smithi001 ceph-mon[28991]: from='osd.4 [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-28T16:24:23.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:22 smithi001 ceph-mon[28991]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-28T16:24:23.597 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.606+0000 7fa4e9600700 1 -- 172.21.15.66:0/129072411 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa4ec05e3d0 con 0x7fa4d40692b0 2024-02-28T16:24:23.597 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 4 on host 'smithi066' 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa4d40692b0 msgr2=0x7fa4d406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa4d40692b0 0x7fa4d406b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8004990 tx=0x7fa4d8009ee0 comp rx=0 tx=0).stop 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec104750 msgr2=0x7fa4ec114100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec104750 0x7fa4ec114100 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0000c00 tx=0x7fa4e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 shutdown_connections 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa4d40692b0 0x7fa4d406b760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4ec104750 0x7fa4ec114100 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 --2- 172.21.15.66:0/129072411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec114640 0x7fa4ec1154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:24:23.600 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 >> 172.21.15.66:0/129072411 conn(0x7fa4ec0727d0 msgr2=0x7fa4ec075b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:24:23.601 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 shutdown_connections 2024-02-28T16:24:23.601 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:23.610+0000 7fa4f4400700 1 -- 172.21.15.66:0/129072411 wait complete. 2024-02-28T16:24:23.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:23 smithi066 ceph-mon[36236]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:23.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:23 smithi066 ceph-mon[36236]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-28T16:24:23.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:23 smithi066 ceph-mon[36236]: osdmap e32: 5 total, 4 up, 5 in 2024-02-28T16:24:23.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:23 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:23.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:23 smithi066 ceph-mon[36236]: from='osd.4 [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:24:23.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:23 smithi066 ceph-mon[36236]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:24:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:23 smithi001 ceph-mon[28991]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:23 smithi001 ceph-mon[28991]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-28T16:24:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:23 smithi001 ceph-mon[28991]: osdmap e32: 5 total, 4 up, 5 in 2024-02-28T16:24:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:23 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:23 smithi001 ceph-mon[28991]: from='osd.4 [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:24:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:23 smithi001 ceph-mon[28991]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:24:24.476 DEBUG:teuthology.orchestra.run.smithi066:osd.4> sudo journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.4.service 2024-02-28T16:24:24.487 INFO:tasks.cephadm:Deploying osd.5 on smithi066 with /dev/vg_nvme/lv_2... 2024-02-28T16:24:24.487 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-28T16:24:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:24 smithi066 ceph-mon[36236]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-02-28T16:24:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:24 smithi066 ceph-mon[36236]: osdmap e33: 5 total, 4 up, 5 in 2024-02-28T16:24:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:24.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:24 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:25.007 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T16:24:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:24 smithi001 ceph-mon[28991]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-02-28T16:24:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:24 smithi001 ceph-mon[28991]: osdmap e33: 5 total, 4 up, 5 in 2024-02-28T16:24:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:24 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:25.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:25 smithi066 ceph-mon[36236]: purged_snaps scrub starts 2024-02-28T16:24:25.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:25 smithi066 ceph-mon[36236]: purged_snaps scrub ok 2024-02-28T16:24:25.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:25 smithi066 ceph-mon[36236]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:25.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:25 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:25.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:25 smithi066 ceph-mon[36236]: osd.4 [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] boot 2024-02-28T16:24:25.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:25 smithi066 ceph-mon[36236]: osdmap e34: 5 total, 5 up, 5 in 2024-02-28T16:24:25.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:25 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:25 smithi001 ceph-mon[28991]: purged_snaps scrub starts 2024-02-28T16:24:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:25 smithi001 ceph-mon[28991]: purged_snaps scrub ok 2024-02-28T16:24:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:25 smithi001 ceph-mon[28991]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-28T16:24:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:25 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:25 smithi001 ceph-mon[28991]: osd.4 [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] boot 2024-02-28T16:24:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:25 smithi001 ceph-mon[28991]: osdmap e34: 5 total, 5 up, 5 in 2024-02-28T16:24:26.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:25 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:24:27.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:27 smithi001 ceph-mon[28991]: osdmap e35: 5 total, 5 up, 5 in 2024-02-28T16:24:27.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:27 smithi001 ceph-mon[28991]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:27.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:27 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:24:27.632 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:27 smithi066 ceph-mon[36236]: osdmap e35: 5 total, 5 up, 5 in 2024-02-28T16:24:27.632 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:27 smithi066 ceph-mon[36236]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:27.632 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:27 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:24:28.095 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-28T16:24:28.095 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-28T16:24:28.095 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-02-28T16:24:28.095 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-02-28T16:24:28.095 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0335581 s, 312 MB/s 2024-02-28T16:24:28.095 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-02-28T16:24:29.250 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_2 2024-02-28T16:24:29.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:29 smithi066 ceph-mon[36236]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:29.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:29 smithi001 ceph-mon[28991]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:29.910 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T16:24:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:31 smithi066 ceph-mon[36236]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:31.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:31 smithi001 ceph-mon[28991]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:32.834 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.844+0000 7f11a5000700 1 -- 172.21.15.66:0/1532107386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0072890 msgr2=0x7f11a0072ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:24:32.834 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.844+0000 7f11a5000700 1 --2- 172.21.15.66:0/1532107386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0072890 0x7f11a0072ca0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1190004940 tx=0x7f1190009e30 comp rx=0 tx=0).stop 2024-02-28T16:24:32.835 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.844+0000 7f11a5000700 1 -- 172.21.15.66:0/1532107386 shutdown_connections 2024-02-28T16:24:32.835 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.844+0000 7f11a5000700 1 --2- 172.21.15.66:0/1532107386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0073af0 0x7f11a0073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:24:32.835 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.844+0000 7f11a5000700 1 --2- 172.21.15.66:0/1532107386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0072890 0x7f11a0072ca0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:24:32.835 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.844+0000 7f11a5000700 1 -- 172.21.15.66:0/1532107386 >> 172.21.15.66:0/1532107386 conn(0x7f11a00fa920 msgr2=0x7f11a00fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:24:32.835 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.844+0000 7f11a5000700 1 -- 172.21.15.66:0/1532107386 shutdown_connections 2024-02-28T16:24:32.835 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.844+0000 7f11a5000700 1 -- 172.21.15.66:0/1532107386 wait complete. 2024-02-28T16:24:32.835 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.845+0000 7f11a5000700 1 Processor -- start 2024-02-28T16:24:32.836 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.846+0000 7f11a5000700 1 -- start start 2024-02-28T16:24:32.836 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.846+0000 7f11a5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0072890 0x7f11a0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:24:32.836 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.846+0000 7f11a5000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0073af0 0x7f11a01144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:24:32.836 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.846+0000 7f11a5000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a0114aa0 con 0x7f11a0072890 2024-02-28T16:24:32.836 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.846+0000 7f11a5000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a0115340 con 0x7f11a0073af0 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.846+0000 7f1197e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0073af0 0x7f11a01144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.846+0000 7f1197e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0073af0 0x7f11a01144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.66:60898/0 (socket says 172.21.15.66:60898) 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.846+0000 7f1197e00700 1 -- 172.21.15.66:0/2701305793 learned_addr learned my addr 172.21.15.66:0/2701305793 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.847+0000 7f119de00700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0072890 0x7f11a0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.847+0000 7f1197e00700 1 -- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0072890 msgr2=0x7f11a0113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.847+0000 7f1197e00700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0072890 0x7f11a0113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.847+0000 7f1197e00700 1 -- 172.21.15.66:0/2701305793 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1190005040 con 0x7f11a0073af0 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.847+0000 7f119de00700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0072890 0x7f11a0113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:24:32.837 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.847+0000 7f1197e00700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0073af0 0x7f11a01144a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f118800c830 tx=0x7f1188004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:24:32.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.848+0000 7f1196000700 1 -- 172.21.15.66:0/2701305793 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f118800a760 con 0x7f11a0073af0 2024-02-28T16:24:32.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.848+0000 7f1196000700 1 -- 172.21.15.66:0/2701305793 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f118800ad90 con 0x7f11a0073af0 2024-02-28T16:24:32.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.848+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a0115620 con 0x7f11a0073af0 2024-02-28T16:24:32.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.848+0000 7f1196000700 1 -- 172.21.15.66:0/2701305793 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11880113a0 con 0x7f11a0073af0 2024-02-28T16:24:32.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.848+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a0115b60 con 0x7f11a0073af0 2024-02-28T16:24:32.838 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.849+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a004bed0 con 0x7f11a0073af0 2024-02-28T16:24:32.840 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.850+0000 7f1196000700 1 -- 172.21.15.66:0/2701305793 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f1188012400 con 0x7f11a0073af0 2024-02-28T16:24:32.841 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.852+0000 7f1196000700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f11840692b0 0x7f118406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:24:32.841 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.852+0000 7f1196000700 1 -- 172.21.15.66:0/2701305793 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f1188086d10 con 0x7f11a0073af0 2024-02-28T16:24:32.842 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.853+0000 7f119de00700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f11840692b0 0x7f118406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:24:32.843 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.854+0000 7f119de00700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f11840692b0 0x7f118406b760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f119000b840 tx=0x7f1190009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:24:32.846 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:32.856+0000 7f1196000700 1 -- 172.21.15.66:0/2701305793 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1188056fa0 con 0x7f11a0073af0 2024-02-28T16:24:33.068 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:24:33.078+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f11a010ab80 con 0x7f11840692b0 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: Detected new or changed devices on smithi066 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: Adjusting osd_memory_target on smithi066 to 7969M 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:33.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:24:33.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:33.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:24:33.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:24:33.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:24:33.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:33 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: Detected new or changed devices on smithi066 2024-02-28T16:24:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:24:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:24:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: Adjusting osd_memory_target on smithi066 to 7969M 2024-02-28T16:24:33.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-28T16:24:33.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:33 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:34.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:34 smithi066 ceph-mon[36236]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:24:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:34 smithi001 ceph-mon[28991]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:24:35.386 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:35 smithi066 ceph-mon[36236]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:35 smithi001 ceph-mon[28991]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:37 smithi066 ceph-mon[36236]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:37.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:37 smithi066 ceph-mon[36236]: from='client.? 172.21.15.66:0/1809481611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e83df14f-7d81-4bb7-9617-024ad94de1a4"}]: dispatch 2024-02-28T16:24:37.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:37 smithi066 ceph-mon[36236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e83df14f-7d81-4bb7-9617-024ad94de1a4"}]: dispatch 2024-02-28T16:24:37.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:37 smithi066 ceph-mon[36236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e83df14f-7d81-4bb7-9617-024ad94de1a4"}]': finished 2024-02-28T16:24:37.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:37 smithi066 ceph-mon[36236]: osdmap e36: 6 total, 5 up, 6 in 2024-02-28T16:24:37.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:37 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:24:37.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:37 smithi001 ceph-mon[28991]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:37.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:37 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/1809481611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e83df14f-7d81-4bb7-9617-024ad94de1a4"}]: dispatch 2024-02-28T16:24:37.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:37 smithi001 ceph-mon[28991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e83df14f-7d81-4bb7-9617-024ad94de1a4"}]: dispatch 2024-02-28T16:24:37.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:37 smithi001 ceph-mon[28991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e83df14f-7d81-4bb7-9617-024ad94de1a4"}]': finished 2024-02-28T16:24:37.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:37 smithi001 ceph-mon[28991]: osdmap e36: 6 total, 5 up, 6 in 2024-02-28T16:24:37.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:37 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:24:38.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:38 smithi066 ceph-mon[36236]: from='client.? 172.21.15.66:0/3748555268' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:24:38.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:38 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/3748555268' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-28T16:24:39.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:39 smithi066 ceph-mon[36236]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:39.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:39 smithi001 ceph-mon[28991]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:41 smithi066 ceph-mon[36236]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:24:41.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:41 smithi001 ceph-mon[28991]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:41.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:24:43.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:43 smithi066 ceph-mon[36236]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:43.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:43 smithi001 ceph-mon[28991]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:45 smithi066 ceph-mon[36236]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:45.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:45 smithi001 ceph-mon[28991]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:47.740 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:47 smithi066 ceph-mon[36236]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:47.740 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-28T16:24:47.740 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:47.740 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:47 smithi066 ceph-mon[36236]: Deploying daemon osd.5 on smithi066 2024-02-28T16:24:47.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:47 smithi001 ceph-mon[28991]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:47.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-28T16:24:47.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:24:47.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:47 smithi001 ceph-mon[28991]: Deploying daemon osd.5 on smithi066 2024-02-28T16:24:49.579 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:49 smithi066 ceph-mon[36236]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:49.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:49 smithi001 ceph-mon[28991]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:51.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:51 smithi066 ceph-mon[36236]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:51.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:51 smithi001 ceph-mon[28991]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:53.735 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:53 smithi066 ceph-mon[36236]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:53.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:53 smithi001 ceph-mon[28991]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:55.369 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:55 smithi066 ceph-mon[36236]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:55.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:55 smithi001 ceph-mon[28991]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:56.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:24:56.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:56.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:56.557 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:24:56.557 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:56.557 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:24:57.486 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:57 smithi066 ceph-mon[36236]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:57.487 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:24:57.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:57 smithi001 ceph-mon[28991]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:57.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:24:59.725 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:24:59 smithi066 ceph-mon[36236]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:24:59.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:24:59 smithi001 ceph-mon[28991]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:25:00.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:00 smithi066 ceph-mon[36236]: from='osd.5 [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-28T16:25:00.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:00 smithi066 ceph-mon[36236]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-28T16:25:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:00 smithi001 ceph-mon[28991]: from='osd.5 [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-28T16:25:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:00 smithi001 ceph-mon[28991]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-28T16:25:01.143 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.153+0000 7f1196000700 1 -- 172.21.15.66:0/2701305793 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f11a010ab80 con 0x7f11840692b0 2024-02-28T16:25:01.143 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 5 on host 'smithi066' 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.156+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f11840692b0 msgr2=0x7f118406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.156+0000 7f11a5000700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f11840692b0 0x7f118406b760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f119000b840 tx=0x7f1190009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.156+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0073af0 msgr2=0x7f11a01144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.156+0000 7f11a5000700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0073af0 0x7f11a01144a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f118800c830 tx=0x7f1188004470 comp rx=0 tx=0).stop 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.157+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 shutdown_connections 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.157+0000 7f11a5000700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f11840692b0 0x7f118406b760 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.157+0000 7f11a5000700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11a0072890 0x7f11a0113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.157+0000 7f11a5000700 1 --2- 172.21.15.66:0/2701305793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11a0073af0 0x7f11a01144a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.157+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 >> 172.21.15.66:0/2701305793 conn(0x7f11a00fa920 msgr2=0x7f11a0109460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.157+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 shutdown_connections 2024-02-28T16:25:01.147 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:01.157+0000 7f11a5000700 1 -- 172.21.15.66:0/2701305793 wait complete. 2024-02-28T16:25:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:01 smithi066 ceph-mon[36236]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:25:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:01 smithi066 ceph-mon[36236]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-28T16:25:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:01 smithi066 ceph-mon[36236]: osdmap e37: 6 total, 5 up, 6 in 2024-02-28T16:25:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:01 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:25:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:01 smithi066 ceph-mon[36236]: from='osd.5 [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:25:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:01 smithi066 ceph-mon[36236]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:25:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:01 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:01 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:01 smithi001 ceph-mon[28991]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:25:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:01 smithi001 ceph-mon[28991]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-28T16:25:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:01 smithi001 ceph-mon[28991]: osdmap e37: 6 total, 5 up, 6 in 2024-02-28T16:25:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:01 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:25:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:01 smithi001 ceph-mon[28991]: from='osd.5 [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:25:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:01 smithi001 ceph-mon[28991]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:25:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:01 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:01.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:01 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:02.039 DEBUG:teuthology.orchestra.run.smithi066:osd.5> sudo journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.5.service 2024-02-28T16:25:02.048 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-28T16:25:02.049 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd stat -f json 2024-02-28T16:25:02.546 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:02.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:02 smithi001 ceph-mon[28991]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-02-28T16:25:02.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:02 smithi001 ceph-mon[28991]: osdmap e38: 6 total, 5 up, 6 in 2024-02-28T16:25:02.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:02 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:25:02.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:02 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:25:02.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:02 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:02.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:02 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:02.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:02 smithi066 ceph-mon[36236]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-02-28T16:25:02.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:02 smithi066 ceph-mon[36236]: osdmap e38: 6 total, 5 up, 6 in 2024-02-28T16:25:02.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:02 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:25:02.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:02 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:25:02.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:02 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:02.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:02 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:03 smithi066 ceph-mon[36236]: purged_snaps scrub starts 2024-02-28T16:25:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:03 smithi066 ceph-mon[36236]: purged_snaps scrub ok 2024-02-28T16:25:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:03 smithi066 ceph-mon[36236]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:25:03.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:03 smithi066 ceph-mon[36236]: osd.5 [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] boot 2024-02-28T16:25:03.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:03 smithi066 ceph-mon[36236]: osdmap e39: 6 total, 6 up, 6 in 2024-02-28T16:25:03.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:03 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:25:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:03 smithi001 ceph-mon[28991]: purged_snaps scrub starts 2024-02-28T16:25:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:03 smithi001 ceph-mon[28991]: purged_snaps scrub ok 2024-02-28T16:25:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:03 smithi001 ceph-mon[28991]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-28T16:25:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:03 smithi001 ceph-mon[28991]: osd.5 [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] boot 2024-02-28T16:25:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:03 smithi001 ceph-mon[28991]: osdmap e39: 6 total, 6 up, 6 in 2024-02-28T16:25:03.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:03 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:25:04.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:04 smithi001 ceph-mon[28991]: osdmap e40: 6 total, 6 up, 6 in 2024-02-28T16:25:04.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:04 smithi066 ceph-mon[36236]: osdmap e40: 6 total, 6 up, 6 in 2024-02-28T16:25:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.008+0000 7f3d09c00700 1 -- 172.21.15.1:0/276962701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 msgr2=0x7f3d041093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.008+0000 7f3d09c00700 1 --2- 172.21.15.1:0/276962701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 0x7f3d041093f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0004940 tx=0x7f3cf0009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.008+0000 7f3d09c00700 1 -- 172.21.15.1:0/276962701 shutdown_connections 2024-02-28T16:25:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.008+0000 7f3d09c00700 1 --2- 172.21.15.1:0/276962701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 0x7f3d041093f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.008+0000 7f3d09c00700 1 --2- 172.21.15.1:0/276962701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3d04070380 0x7f3d04070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.008+0000 7f3d09c00700 1 -- 172.21.15.1:0/276962701 >> 172.21.15.1:0/276962701 conn(0x7f3d040f8400 msgr2=0x7f3d040fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.008+0000 7f3d09c00700 1 -- 172.21.15.1:0/276962701 shutdown_connections 2024-02-28T16:25:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.008+0000 7f3d09c00700 1 -- 172.21.15.1:0/276962701 wait complete. 2024-02-28T16:25:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.009+0000 7f3d09c00700 1 Processor -- start 2024-02-28T16:25:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.009+0000 7f3d09c00700 1 -- start start 2024-02-28T16:25:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.009+0000 7f3d09c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3d04070380 0x7f3d04113d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.009+0000 7f3d09c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 0x7f3d041142c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.009+0000 7f3d09c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d041150b0 con 0x7f3d04070cd0 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.009+0000 7f3d09c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d04115220 con 0x7f3d04070380 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.010+0000 7f3d03400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 0x7f3d041142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.010+0000 7f3d03400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 0x7f3d041142c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41432/0 (socket says 172.21.15.1:41432) 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.010+0000 7f3d03400700 1 -- 172.21.15.1:0/2415197030 learned_addr learned my addr 172.21.15.1:0/2415197030 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.010+0000 7f3d03e00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3d04070380 0x7f3d04113d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.010+0000 7f3d03400700 1 -- 172.21.15.1:0/2415197030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3d04070380 msgr2=0x7f3d04113d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.010+0000 7f3d03400700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3d04070380 0x7f3d04113d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.010+0000 7f3d03400700 1 -- 172.21.15.1:0/2415197030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0005040 con 0x7f3d04070cd0 2024-02-28T16:25:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.010+0000 7f3d03400700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 0x7f3d041142c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00093b0 tx=0x7f3cf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.011+0000 7f3cfbe00700 1 -- 172.21.15.1:0/2415197030 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cf000e040 con 0x7f3d04070cd0 2024-02-28T16:25:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.011+0000 7f3cfbe00700 1 -- 172.21.15.1:0/2415197030 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cf000bcc0 con 0x7f3d04070cd0 2024-02-28T16:25:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.011+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d041154a0 con 0x7f3d04070cd0 2024-02-28T16:25:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.011+0000 7f3cfbe00700 1 -- 172.21.15.1:0/2415197030 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cf001c630 con 0x7f3d04070cd0 2024-02-28T16:25:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.011+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d04115a60 con 0x7f3d04070cd0 2024-02-28T16:25:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.012+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0405f5c0 con 0x7f3d04070cd0 2024-02-28T16:25:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.013+0000 7f3cfbe00700 1 -- 172.21.15.1:0/2415197030 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3cf00139f0 con 0x7f3d04070cd0 2024-02-28T16:25:05.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.014+0000 7f3cfbe00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3ce80692a0 0x7f3ce806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:05.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.014+0000 7f3cfbe00700 1 -- 172.21.15.1:0/2415197030 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3cf00880f0 con 0x7f3d04070cd0 2024-02-28T16:25:05.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.014+0000 7f3d03e00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3ce80692a0 0x7f3ce806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:05.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.015+0000 7f3d03e00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3ce80692a0 0x7f3ce806b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d04114b40 tx=0x7f3cf4009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.019+0000 7f3cfbe00700 1 -- 172.21.15.1:0/2415197030 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3cf0054740 con 0x7f3d04070cd0 2024-02-28T16:25:05.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.232+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3d04072230 con 0x7f3d04070cd0 2024-02-28T16:25:05.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.233+0000 7f3cfbe00700 1 -- 172.21.15.1:0/2415197030 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f3d04072230 con 0x7f3d04070cd0 2024-02-28T16:25:05.224 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:25:05.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.236+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3ce80692a0 msgr2=0x7f3ce806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:05.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.236+0000 7f3d09c00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3ce80692a0 0x7f3ce806b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d04114b40 tx=0x7f3cf4009ee0 comp rx=0 tx=0).stop 2024-02-28T16:25:05.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.236+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 msgr2=0x7f3d041142c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:05.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.236+0000 7f3d09c00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 0x7f3d041142c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00093b0 tx=0x7f3cf0009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:05.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.237+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 shutdown_connections 2024-02-28T16:25:05.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.237+0000 7f3d09c00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3ce80692a0 0x7f3ce806b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:05.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.237+0000 7f3d09c00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3d04070380 0x7f3d04113d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:05.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.237+0000 7f3d09c00700 1 --2- 172.21.15.1:0/2415197030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d04070cd0 0x7f3d041142c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:05.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.237+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 >> 172.21.15.1:0/2415197030 conn(0x7f3d040f8400 msgr2=0x7f3d040fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:05.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.237+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 shutdown_connections 2024-02-28T16:25:05.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:05.237+0000 7f3d09c00700 1 -- 172.21.15.1:0/2415197030 wait complete. 2024-02-28T16:25:05.869 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:05 smithi001 ceph-mon[28991]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:05.869 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:05 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2415197030' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-28T16:25:05.922 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1709137502,"num_in_osds":6,"osd_in_since":1709137476,"num_remapped_pgs":0} 2024-02-28T16:25:05.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-28T16:25:05.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:05 smithi066 ceph-mon[36236]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:05 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2415197030' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-28T16:25:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:07.626 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:07 smithi001 ceph-mon[28991]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:07.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:07 smithi066 ceph-mon[36236]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.407+0000 7f1671e00700 1 -- 172.21.15.1:0/373578085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c103dc0 msgr2=0x7f166c104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.407+0000 7f1671e00700 1 --2- 172.21.15.1:0/373578085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c103dc0 0x7f166c104210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1654004940 tx=0x7f1654009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.407+0000 7f1671e00700 1 -- 172.21.15.1:0/373578085 shutdown_connections 2024-02-28T16:25:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.407+0000 7f1671e00700 1 --2- 172.21.15.1:0/373578085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c103dc0 0x7f166c104210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:08.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.407+0000 7f1671e00700 1 --2- 172.21.15.1:0/373578085 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f166c1095f0 0x7f166c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:08.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.407+0000 7f1671e00700 1 -- 172.21.15.1:0/373578085 >> 172.21.15.1:0/373578085 conn(0x7f166c072d50 msgr2=0x7f166c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:08.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.407+0000 7f1671e00700 1 -- 172.21.15.1:0/373578085 shutdown_connections 2024-02-28T16:25:08.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.407+0000 7f1671e00700 1 -- 172.21.15.1:0/373578085 wait complete. 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.408+0000 7f1671e00700 1 Processor -- start 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.408+0000 7f1671e00700 1 -- start start 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.408+0000 7f1671e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f166c103dc0 0x7f166c11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.408+0000 7f1671e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c1095f0 0x7f166c11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.408+0000 7f1671e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c11abd0 con 0x7f166c1095f0 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.408+0000 7f1671e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c113ee0 con 0x7f166c103dc0 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.409+0000 7f1663e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c1095f0 0x7f166c11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.409+0000 7f1663e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c1095f0 0x7f166c11a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41448/0 (socket says 172.21.15.1:41448) 2024-02-28T16:25:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.409+0000 7f1663e00700 1 -- 172.21.15.1:0/984748657 learned_addr learned my addr 172.21.15.1:0/984748657 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.409+0000 7f166aa00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f166c103dc0 0x7f166c11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.409+0000 7f1663e00700 1 -- 172.21.15.1:0/984748657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f166c103dc0 msgr2=0x7f166c11a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.409+0000 7f1663e00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f166c103dc0 0x7f166c11a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.409+0000 7f1663e00700 1 -- 172.21.15.1:0/984748657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1654005040 con 0x7f166c1095f0 2024-02-28T16:25:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.410+0000 7f1663e00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c1095f0 0x7f166c11a580 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1654009fd0 tx=0x7f1654009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.410+0000 7f1662a00700 1 -- 172.21.15.1:0/984748657 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f165400e040 con 0x7f166c1095f0 2024-02-28T16:25:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.410+0000 7f1662a00700 1 -- 172.21.15.1:0/984748657 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f165400bbf0 con 0x7f166c1095f0 2024-02-28T16:25:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.410+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c114160 con 0x7f166c1095f0 2024-02-28T16:25:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.410+0000 7f1662a00700 1 -- 172.21.15.1:0/984748657 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f165401c3f0 con 0x7f166c1095f0 2024-02-28T16:25:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.410+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c1146a0 con 0x7f166c1095f0 2024-02-28T16:25:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.411+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c10b9c0 con 0x7f166c1095f0 2024-02-28T16:25:08.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.412+0000 7f1662a00700 1 -- 172.21.15.1:0/984748657 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f165400bd60 con 0x7f166c1095f0 2024-02-28T16:25:08.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.413+0000 7f1662a00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16500692b0 0x7f165006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:08.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.413+0000 7f1662a00700 1 -- 172.21.15.1:0/984748657 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f16540880a0 con 0x7f166c1095f0 2024-02-28T16:25:08.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.413+0000 7f166aa00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16500692b0 0x7f165006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.414+0000 7f166aa00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16500692b0 0x7f165006b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f165c007fd0 tx=0x7f165c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:08.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.418+0000 7f1662a00700 1 -- 172.21.15.1:0/984748657 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1654058240 con 0x7f166c1095f0 2024-02-28T16:25:08.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.637+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f166c075870 con 0x7f166c1095f0 2024-02-28T16:25:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.638+0000 7f1662a00700 1 -- 172.21.15.1:0/984748657 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10970 (secure 0 0 0) 0x7f165406ae60 con 0x7f166c1095f0 2024-02-28T16:25:08.629 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:25:08.629 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","created":"2024-02-28T16:17:42.450467+0000","modified":"2024-02-28T16:25:03.518590+0000","last_up_change":"2024-02-28T16:25:02.506452+0000","last_in_change":"2024-02-28T16:24:36.954989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-28T16:23:15.048529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ac92369f-8144-4695-87fe-f9cd73fb5f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6803","nonce":3270329297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6805","nonce":3270329297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6809","nonce":3270329297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6807","nonce":3270329297}]},"public_addr":"172.21.15.1:6803/3270329297","cluster_addr":"172.21.15.1:6805/3270329297","heartbeat_back_addr":"172.21.15.1:6809/3270329297","heartbeat_front_addr":"172.21.15.1:6807/3270329297","state":["exists","up"]},{"osd":1,"uuid":"bfd6ea17-fd9b-46ae-b3ff-547d736cd232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6811","nonce":1496879960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6813","nonce":1496879960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6817","nonce":1496879960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6815","nonce":1496879960}]},"public_addr":"172.21.15.1:6811/1496879960","cluster_addr":"172.21.15.1:6813/1496879960","heartbeat_back_addr":"172.21.15.1:6817/1496879960","heartbeat_front_addr":"172.21.15.1:6815/1496879960","state":["exists","up"]},{"osd":2,"uuid":"b713f513-6604-4e88-8cc1-0f963d9f8599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6819","nonce":131562471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6821","nonce":131562471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6825","nonce":131562471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6823","nonce":131562471}]},"public_addr":"172.21.15.1:6819/131562471","cluster_addr":"172.21.15.1:6821/131562471","heartbeat_back_addr":"172.21.15.1:6825/131562471","heartbeat_front_addr":"172.21.15.1:6823/131562471","state":["exists","up"]},{"osd":3,"uuid":"4f425b10-672f-4265-a314-5a9a15237257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6801","nonce":863585173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6803","nonce":863585173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6807","nonce":863585173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6805","nonce":863585173}]},"public_addr":"172.21.15.66:6801/863585173","cluster_addr":"172.21.15.66:6803/863585173","heartbeat_back_addr":"172.21.15.66:6807/863585173","heartbeat_front_addr":"172.21.15.66:6805/863585173","state":["exists","up"]},{"osd":4,"uuid":"c1f2651b-47f3-444e-9325-22a0a962c3b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6809","nonce":376874414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6811","nonce":376874414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6815","nonce":376874414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6813","nonce":376874414}]},"public_addr":"172.21.15.66:6809/376874414","cluster_addr":"172.21.15.66:6811/376874414","heartbeat_back_addr":"172.21.15.66:6815/376874414","heartbeat_front_addr":"172.21.15.66:6813/376874414","state":["exists","up"]},{"osd":5,"uuid":"e83df14f-7d81-4bb7-9617-024ad94de1a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6817","nonce":2849488501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6819","nonce":2849488501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6823","nonce":2849488501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6821","nonce":2849488501}]},"public_addr":"172.21.15.66:6817/2849488501","cluster_addr":"172.21.15.66:6819/2849488501","heartbeat_back_addr":"172.21.15.66:6823/2849488501","heartbeat_front_addr":"172.21.15.66:6821/2849488501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:01.831777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:36.505002+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:13.091284+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:47.260219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:24:23.366141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:25:00.532840+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6800/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/298653290":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/3865282714":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/4195777758":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/2061955559":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/4223980442":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1339161018":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/833315191":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6801/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:6801/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6801/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/2780166226":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/1224289276":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3320028482":"2024-02-29T16:18:55.593679+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-28T16:25:08.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.640+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16500692b0 msgr2=0x7f165006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:08.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.640+0000 7f1671e00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16500692b0 0x7f165006b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f165c007fd0 tx=0x7f165c009040 comp rx=0 tx=0).stop 2024-02-28T16:25:08.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.640+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c1095f0 msgr2=0x7f166c11a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:08.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.640+0000 7f1671e00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c1095f0 0x7f166c11a580 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1654009fd0 tx=0x7f1654009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:08.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.641+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 shutdown_connections 2024-02-28T16:25:08.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.641+0000 7f1671e00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f16500692b0 0x7f165006b760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:08.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.641+0000 7f1671e00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f166c103dc0 0x7f166c11a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:08.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.641+0000 7f1671e00700 1 --2- 172.21.15.1:0/984748657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f166c1095f0 0x7f166c11a580 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:08.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.641+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 >> 172.21.15.1:0/984748657 conn(0x7f166c072d50 msgr2=0x7f166c074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:08.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.641+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 shutdown_connections 2024-02-28T16:25:08.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:08.641+0000 7f1671e00700 1 -- 172.21.15.1:0/984748657 wait complete. 2024-02-28T16:25:09.241 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: Detected new or changed devices on smithi066 2024-02-28T16:25:09.241 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.241 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: Adjusting osd_memory_target on smithi066 to 5313M 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:09 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/984748657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T16:25:09.314 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-28T16:23:15.048529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-28T16:25:09.314 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-28T16:25:09.403 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: Detected new or changed devices on smithi066 2024-02-28T16:25:09.403 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.403 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: Adjusting osd_memory_target on smithi066 to 5313M 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:25:09.404 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:25:09.405 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:25:09.405 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:09 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/984748657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T16:25:09.804 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:11.604 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:11 smithi001 ceph-mon[28991]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:11.604 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:25:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:11 smithi066 ceph-mon[36236]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:25:11.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.895+0000 7f7ce8a00700 1 -- 172.21.15.1:0/1214303309 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ce4105b00 msgr2=0x7f7ce40ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:11.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.895+0000 7f7ce8a00700 1 --2- 172.21.15.1:0/1214303309 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ce4105b00 0x7f7ce40ff170 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0004940 tx=0x7f7cd0009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.895+0000 7f7ce8a00700 1 -- 172.21.15.1:0/1214303309 shutdown_connections 2024-02-28T16:25:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.895+0000 7f7ce8a00700 1 --2- 172.21.15.1:0/1214303309 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ce4105b00 0x7f7ce40ff170 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.895+0000 7f7ce8a00700 1 --2- 172.21.15.1:0/1214303309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ce4105160 0x7f7ce4105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.895+0000 7f7ce8a00700 1 -- 172.21.15.1:0/1214303309 >> 172.21.15.1:0/1214303309 conn(0x7f7ce40facb0 msgr2=0x7f7ce40fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.895+0000 7f7ce8a00700 1 -- 172.21.15.1:0/1214303309 shutdown_connections 2024-02-28T16:25:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.895+0000 7f7ce8a00700 1 -- 172.21.15.1:0/1214303309 wait complete. 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.896+0000 7f7ce8a00700 1 Processor -- start 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.896+0000 7f7ce8a00700 1 -- start start 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ce4105160 0x7f7ce4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ce4105b00 0x7f7ce4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce8a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4116790 con 0x7f7ce4105b00 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce8a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce410fac0 con 0x7f7ce4105160 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ce4105b00 0x7f7ce4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ce4105b00 0x7f7ce4116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32808/0 (socket says 172.21.15.1:32808) 2024-02-28T16:25:11.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce1400700 1 -- 172.21.15.1:0/3126620388 learned_addr learned my addr 172.21.15.1:0/3126620388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:11.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce1e00700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ce4105160 0x7f7ce4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:11.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.897+0000 7f7ce1400700 1 -- 172.21.15.1:0/3126620388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ce4105160 msgr2=0x7f7ce4115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:11.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.898+0000 7f7ce1400700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ce4105160 0x7f7ce4115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:11.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.898+0000 7f7ce1400700 1 -- 172.21.15.1:0/3126620388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd0005040 con 0x7f7ce4105b00 2024-02-28T16:25:11.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.898+0000 7f7ce1400700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ce4105b00 0x7f7ce4116140 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009150 tx=0x7f7cd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:11.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.898+0000 7f7cda000700 1 -- 172.21.15.1:0/3126620388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cd000e040 con 0x7f7ce4105b00 2024-02-28T16:25:11.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.898+0000 7f7cda000700 1 -- 172.21.15.1:0/3126620388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cd000bbf0 con 0x7f7ce4105b00 2024-02-28T16:25:11.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.898+0000 7f7cda000700 1 -- 172.21.15.1:0/3126620388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cd001c630 con 0x7f7ce4105b00 2024-02-28T16:25:11.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.898+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce410fd40 con 0x7f7ce4105b00 2024-02-28T16:25:11.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.898+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce4110230 con 0x7f7ce4105b00 2024-02-28T16:25:11.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.900+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce40fe8f0 con 0x7f7ce4105b00 2024-02-28T16:25:11.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.901+0000 7f7cda000700 1 -- 172.21.15.1:0/3126620388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7cd000bd60 con 0x7f7ce4105b00 2024-02-28T16:25:11.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.902+0000 7f7cda000700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7cc80692a0 0x7f7cc806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:11.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.902+0000 7f7cda000700 1 -- 172.21.15.1:0/3126620388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7cd00882f0 con 0x7f7ce4105b00 2024-02-28T16:25:11.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.902+0000 7f7ce1e00700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7cc80692a0 0x7f7cc806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:11.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.902+0000 7f7ce1e00700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7cc80692a0 0x7f7cc806b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009fd0 tx=0x7f7cd4009f60 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:11.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:11.907+0000 7f7cda000700 1 -- 172.21.15.1:0/3126620388 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7cd0058350 con 0x7f7ce4105b00 2024-02-28T16:25:12.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.120+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ce41111e0 con 0x7f7ce4105b00 2024-02-28T16:25:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.122+0000 7f7cda000700 1 -- 172.21.15.1:0/3126620388 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7cd0057ee0 con 0x7f7ce4105b00 2024-02-28T16:25:12.111 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7cc80692a0 msgr2=0x7f7cc806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7cc80692a0 0x7f7cc806b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009fd0 tx=0x7f7cd4009f60 comp rx=0 tx=0).stop 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ce4105b00 msgr2=0x7f7ce4116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ce4105b00 0x7f7ce4116140 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009150 tx=0x7f7cd0009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 shutdown_connections 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7cc80692a0 0x7f7cc806b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ce4105160 0x7f7ce4115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 --2- 172.21.15.1:0/3126620388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ce4105b00 0x7f7ce4116140 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.124+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 >> 172.21.15.1:0/3126620388 conn(0x7f7ce40facb0 msgr2=0x7f7ce40fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.125+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 shutdown_connections 2024-02-28T16:25:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:12.125+0000 7f7ce8a00700 1 -- 172.21.15.1:0/3126620388 wait complete. 2024-02-28T16:25:12.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:12 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3126620388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T16:25:12.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:12 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3126620388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T16:25:12.949 INFO:tasks.cephadm:Setting up client nodes... 2024-02-28T16:25:12.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-28T16:25:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:13 smithi066 ceph-mon[36236]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:13.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:13 smithi001 ceph-mon[28991]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:15.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:15 smithi001 ceph-mon[28991]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:15.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:15 smithi066 ceph-mon[36236]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:15.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.802+0000 7efd83000700 1 -- 172.21.15.1:0/2510512433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 msgr2=0x7efd7c0fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:15.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.802+0000 7efd83000700 1 --2- 172.21.15.1:0/2510512433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 0x7efd7c0fe700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd68004940 tx=0x7efd68009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:15.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.802+0000 7efd83000700 1 -- 172.21.15.1:0/2510512433 shutdown_connections 2024-02-28T16:25:15.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.802+0000 7efd83000700 1 --2- 172.21.15.1:0/2510512433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 0x7efd7c0fe700 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:15.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.802+0000 7efd83000700 1 --2- 172.21.15.1:0/2510512433 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd7c0fb220 0x7efd7c0fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:15.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.802+0000 7efd83000700 1 -- 172.21.15.1:0/2510512433 >> 172.21.15.1:0/2510512433 conn(0x7efd7c0f6ca0 msgr2=0x7efd7c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:15.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.803+0000 7efd83000700 1 -- 172.21.15.1:0/2510512433 shutdown_connections 2024-02-28T16:25:15.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.803+0000 7efd83000700 1 -- 172.21.15.1:0/2510512433 wait complete. 2024-02-28T16:25:15.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.804+0000 7efd83000700 1 Processor -- start 2024-02-28T16:25:15.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.804+0000 7efd83000700 1 -- start start 2024-02-28T16:25:15.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd83000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd7c0fb220 0x7efd7c10d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:15.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd83000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 0x7efd7c10dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:15.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd83000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c10e4c0 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd83000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c10ed60 con 0x7efd7c0fb220 2024-02-28T16:25:15.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd7b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 0x7efd7c10dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:15.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd7b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 0x7efd7c10dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32820/0 (socket says 172.21.15.1:32820) 2024-02-28T16:25:15.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd7b400700 1 -- 172.21.15.1:0/3794086480 learned_addr learned my addr 172.21.15.1:0/3794086480 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:15.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd7be00700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd7c0fb220 0x7efd7c10d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:15.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd7b400700 1 -- 172.21.15.1:0/3794086480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd7c0fb220 msgr2=0x7efd7c10d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:15.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd7b400700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd7c0fb220 0x7efd7c10d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:15.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.805+0000 7efd7b400700 1 -- 172.21.15.1:0/3794086480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68005040 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.806+0000 7efd7b400700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 0x7efd7c10dec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efd68000c00 tx=0x7efd68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:15.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.806+0000 7efd73e00700 1 -- 172.21.15.1:0/3794086480 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd68021070 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.806+0000 7efd73e00700 1 -- 172.21.15.1:0/3794086480 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd6801c400 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.806+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c10efe0 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.806+0000 7efd73e00700 1 -- 172.21.15.1:0/3794086480 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd6800e070 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.806+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c10f4c0 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.808+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c04bed0 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.808+0000 7efd73e00700 1 -- 172.21.15.1:0/3794086480 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7efd6801c570 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.809+0000 7efd73e00700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7efd600692b0 0x7efd6006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:15.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.810+0000 7efd73e00700 1 -- 172.21.15.1:0/3794086480 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efd680880e0 con 0x7efd7c0fe2b0 2024-02-28T16:25:15.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.810+0000 7efd7be00700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7efd600692b0 0x7efd6006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:15.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.810+0000 7efd7be00700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7efd600692b0 0x7efd6006b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd6c004600 tx=0x7efd6c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:15.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:15.815+0000 7efd73e00700 1 -- 172.21.15.1:0/3794086480 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7efd680581c0 con 0x7efd7c0fe2b0 2024-02-28T16:25:16.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.112+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7efd7c105a40 con 0x7efd7c0fe2b0 2024-02-28T16:25:16.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.120+0000 7efd73e00700 1 -- 172.21.15.1:0/3794086480 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7efd68057d50 con 0x7efd7c0fe2b0 2024-02-28T16:25:16.109 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-02-28T16:25:16.110 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBsXt9lvdTIBhAAtSMvckRdkPCGk7rQNrVzEw== 2024-02-28T16:25:16.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7efd600692b0 msgr2=0x7efd6006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7efd600692b0 0x7efd6006b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd6c004600 tx=0x7efd6c00b040 comp rx=0 tx=0).stop 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 msgr2=0x7efd7c10dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 0x7efd7c10dec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efd68000c00 tx=0x7efd68009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 shutdown_connections 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7efd600692b0 0x7efd6006b760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd7c0fb220 0x7efd7c10d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 --2- 172.21.15.1:0/3794086480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c0fe2b0 0x7efd7c10dec0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 >> 172.21.15.1:0/3794086480 conn(0x7efd7c0f6ca0 msgr2=0x7efd7c1014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.123+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 shutdown_connections 2024-02-28T16:25:16.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:16.124+0000 7efd83000700 1 -- 172.21.15.1:0/3794086480 wait complete. 2024-02-28T16:25:16.491 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:16 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3794086480' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-28T16:25:16.492 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:16 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3794086480' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-28T16:25:16.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:16 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3794086480' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-28T16:25:16.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:16 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3794086480' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-28T16:25:16.892 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:25:16.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-28T16:25:16.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-28T16:25:17.053 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-28T16:25:17.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:17 smithi066 ceph-mon[36236]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:17.572 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T16:25:17.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:17 smithi001 ceph-mon[28991]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:19.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:19 smithi066 ceph-mon[36236]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:19.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:19 smithi001 ceph-mon[28991]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:20.074 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.084+0000 7f7d71e00700 1 -- 172.21.15.66:0/4193114664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c100360 msgr2=0x7f7d6c1007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:20.074 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.084+0000 7f7d71e00700 1 --2- 172.21.15.66:0/4193114664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c100360 0x7f7d6c1007b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7d58004940 tx=0x7f7d58009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:20.074 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.084+0000 7f7d71e00700 1 -- 172.21.15.66:0/4193114664 shutdown_connections 2024-02-28T16:25:20.074 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.084+0000 7f7d71e00700 1 --2- 172.21.15.66:0/4193114664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c100360 0x7f7d6c1007b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:20.075 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.084+0000 7f7d71e00700 1 --2- 172.21.15.66:0/4193114664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0ff150 0x7f7d6c0ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:20.075 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.084+0000 7f7d71e00700 1 -- 172.21.15.66:0/4193114664 >> 172.21.15.66:0/4193114664 conn(0x7f7d6c0fa920 msgr2=0x7f7d6c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:20.075 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.084+0000 7f7d71e00700 1 -- 172.21.15.66:0/4193114664 shutdown_connections 2024-02-28T16:25:20.075 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.084+0000 7f7d71e00700 1 -- 172.21.15.66:0/4193114664 wait complete. 2024-02-28T16:25:20.076 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.085+0000 7f7d71e00700 1 Processor -- start 2024-02-28T16:25:20.076 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.085+0000 7f7d71e00700 1 -- start start 2024-02-28T16:25:20.076 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d71e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c0ff150 0x7f7d6c10fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:20.076 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d71e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c100360 0x7f7d6c1100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:20.076 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d71e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c1106d0 con 0x7f7d6c100360 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d71e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c110f70 con 0x7f7d6c0ff150 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d6aa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c0ff150 0x7f7d6c10fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d6aa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c0ff150 0x7f7d6c10fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.66:58320/0 (socket says 172.21.15.66:58320) 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d6aa00700 1 -- 172.21.15.66:0/3538185291 learned_addr learned my addr 172.21.15.66:0/3538185291 (peer_addr_for_me v2:172.21.15.66:0/0) 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d6a000700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c100360 0x7f7d6c1100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d6aa00700 1 -- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c100360 msgr2=0x7f7d6c1100d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.086+0000 7f7d6aa00700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c100360 0x7f7d6c1100d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.087+0000 7f7d6aa00700 1 -- 172.21.15.66:0/3538185291 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d58005040 con 0x7f7d6c0ff150 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.087+0000 7f7d6a000700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c100360 0x7f7d6c1100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:25:20.077 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.087+0000 7f7d6aa00700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c0ff150 0x7f7d6c10fb90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c002a80 tx=0x7f7d5c004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:20.078 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.087+0000 7f7d62a00700 1 -- 172.21.15.66:0/3538185291 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d5c007cc0 con 0x7f7d6c0ff150 2024-02-28T16:25:20.078 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.087+0000 7f7d62a00700 1 -- 172.21.15.66:0/3538185291 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d5c00f040 con 0x7f7d6c0ff150 2024-02-28T16:25:20.078 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.087+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6c111250 con 0x7f7d6c0ff150 2024-02-28T16:25:20.078 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.087+0000 7f7d62a00700 1 -- 172.21.15.66:0/3538185291 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d5c00dad0 con 0x7f7d6c0ff150 2024-02-28T16:25:20.078 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.087+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6c111790 con 0x7f7d6c0ff150 2024-02-28T16:25:20.078 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.089+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6c04bed0 con 0x7f7d6c0ff150 2024-02-28T16:25:20.079 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.089+0000 7f7d62a00700 1 -- 172.21.15.66:0/3538185291 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7d5c00dc30 con 0x7f7d6c0ff150 2024-02-28T16:25:20.080 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.090+0000 7f7d62a00700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7d500692b0 0x7f7d5006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:20.080 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.091+0000 7f7d62a00700 1 -- 172.21.15.66:0/3538185291 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7d5c086c00 con 0x7f7d6c0ff150 2024-02-28T16:25:20.080 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.091+0000 7f7d6a000700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7d500692b0 0x7f7d5006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:20.081 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.092+0000 7f7d6a000700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7d500692b0 0x7f7d5006b760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800b7a0 tx=0x7f7d58009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:20.085 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.095+0000 7f7d62a00700 1 -- 172.21.15.66:0/3538185291 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7d5c08c020 con 0x7f7d6c0ff150 2024-02-28T16:25:20.383 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.393+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f7d6c107c30 con 0x7f7d6c0ff150 2024-02-28T16:25:20.393 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.404+0000 7f7d62a00700 1 -- 172.21.15.66:0/3538185291 <== mon.1 v2:172.21.15.66:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f7d5c056c80 con 0x7f7d6c0ff150 2024-02-28T16:25:20.394 INFO:teuthology.orchestra.run.smithi066.stdout:[client.1] 2024-02-28T16:25:20.394 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBwXt9lMImnFxAAzii8qPfXc8MMvjM48AyUmg== 2024-02-28T16:25:20.397 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.407+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7d500692b0 msgr2=0x7f7d5006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:20.397 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.407+0000 7f7d71e00700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7d500692b0 0x7f7d5006b760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800b7a0 tx=0x7f7d58009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:20.397 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.407+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c0ff150 msgr2=0x7f7d6c10fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:20.397 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.407+0000 7f7d71e00700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c0ff150 0x7f7d6c10fb90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c002a80 tx=0x7f7d5c004600 comp rx=0 tx=0).stop 2024-02-28T16:25:20.398 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.408+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 shutdown_connections 2024-02-28T16:25:20.398 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.408+0000 7f7d71e00700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7d500692b0 0x7f7d5006b760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:20.398 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.408+0000 7f7d71e00700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d6c0ff150 0x7f7d6c10fb90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:20.398 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.408+0000 7f7d71e00700 1 --2- 172.21.15.66:0/3538185291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c100360 0x7f7d6c1100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:20.398 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.408+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 >> 172.21.15.66:0/3538185291 conn(0x7f7d6c0fa920 msgr2=0x7f7d6c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:20.398 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.408+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 shutdown_connections 2024-02-28T16:25:20.399 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:25:20.408+0000 7f7d71e00700 1 -- 172.21.15.66:0/3538185291 wait complete. 2024-02-28T16:25:20.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:20 smithi066 ceph-mon[36236]: from='client.? 172.21.15.66:0/3538185291' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-28T16:25:20.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:20 smithi066 ceph-mon[36236]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-28T16:25:20.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:20 smithi066 ceph-mon[36236]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-28T16:25:20.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:20 smithi001 ceph-mon[28991]: from='client.? 172.21.15.66:0/3538185291' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-28T16:25:20.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:20 smithi001 ceph-mon[28991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-28T16:25:20.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:20 smithi001 ceph-mon[28991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-28T16:25:21.362 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:25:21.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-28T16:25:21.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-28T16:25:21.538 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:21 smithi066 ceph-mon[36236]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:21.543 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-28T16:25:21.543 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-28T16:25:21.544 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph mgr dump --format=json 2024-02-28T16:25:21.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:21 smithi001 ceph-mon[28991]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:22.056 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:23.708 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:23 smithi001 ceph-mon[28991]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:23.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:23 smithi066 ceph-mon[36236]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:24.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.188+0000 7f2d9d000700 1 -- 172.21.15.1:0/3825787972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 msgr2=0x7f2d981099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:24.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.188+0000 7f2d9d000700 1 --2- 172.21.15.1:0/3825787972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 0x7f2d981099c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2d88004940 tx=0x7f2d88009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:24.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.188+0000 7f2d9d000700 1 -- 172.21.15.1:0/3825787972 shutdown_connections 2024-02-28T16:25:24.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.188+0000 7f2d9d000700 1 --2- 172.21.15.1:0/3825787972 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d98103dc0 0x7f2d98104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:24.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.188+0000 7f2d9d000700 1 --2- 172.21.15.1:0/3825787972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 0x7f2d981099c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:24.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.188+0000 7f2d9d000700 1 -- 172.21.15.1:0/3825787972 >> 172.21.15.1:0/3825787972 conn(0x7f2d98072d50 msgr2=0x7f2d98075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:24.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.188+0000 7f2d9d000700 1 -- 172.21.15.1:0/3825787972 shutdown_connections 2024-02-28T16:25:24.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.188+0000 7f2d9d000700 1 -- 172.21.15.1:0/3825787972 wait complete. 2024-02-28T16:25:24.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.189+0000 7f2d9d000700 1 Processor -- start 2024-02-28T16:25:24.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.189+0000 7f2d9d000700 1 -- start start 2024-02-28T16:25:24.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.190+0000 7f2d9d000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d98103dc0 0x7f2d98119ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:24.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.190+0000 7f2d9d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 0x7f2d9811a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.190+0000 7f2d9d000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9811ab80 con 0x7f2d981095f0 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.190+0000 7f2d9d000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98113f00 con 0x7f2d98103dc0 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.190+0000 7f2d95400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 0x7f2d9811a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.190+0000 7f2d95400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 0x7f2d9811a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48122/0 (socket says 172.21.15.1:48122) 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.190+0000 7f2d95400700 1 -- 172.21.15.1:0/1978442791 learned_addr learned my addr 172.21.15.1:0/1978442791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.190+0000 7f2d95e00700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d98103dc0 0x7f2d98119ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d95400700 1 -- 172.21.15.1:0/1978442791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d98103dc0 msgr2=0x7f2d98119ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d95400700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d98103dc0 0x7f2d98119ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d95400700 1 -- 172.21.15.1:0/1978442791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d88005040 con 0x7f2d981095f0 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d95e00700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d98103dc0 0x7f2d98119ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d95400700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 0x7f2d9811a530 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d84002a50 tx=0x7f2d840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d8e000700 1 -- 172.21.15.1:0/1978442791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d84007cc0 con 0x7f2d981095f0 2024-02-28T16:25:24.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d8e000700 1 -- 172.21.15.1:0/1978442791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d8400f040 con 0x7f2d981095f0 2024-02-28T16:25:24.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d8e000700 1 -- 172.21.15.1:0/1978442791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d8400db70 con 0x7f2d981095f0 2024-02-28T16:25:24.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d981141e0 con 0x7f2d981095f0 2024-02-28T16:25:24.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.191+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d98114720 con 0x7f2d981095f0 2024-02-28T16:25:24.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.193+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d98075870 con 0x7f2d981095f0 2024-02-28T16:25:24.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.194+0000 7f2d8e000700 1 -- 172.21.15.1:0/1978442791 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2d8400dd10 con 0x7f2d981095f0 2024-02-28T16:25:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.195+0000 7f2d8e000700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d7c0692b0 0x7f2d7c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.195+0000 7f2d8e000700 1 -- 172.21.15.1:0/1978442791 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2d84086550 con 0x7f2d981095f0 2024-02-28T16:25:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.195+0000 7f2d95e00700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d7c0692b0 0x7f2d7c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.196+0000 7f2d95e00700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d7c0692b0 0x7f2d7c06b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d88000c00 tx=0x7f2d88009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:24.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.200+0000 7f2d8e000700 1 -- 172.21.15.1:0/1978442791 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2d840565b0 con 0x7f2d981095f0 2024-02-28T16:25:24.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.469+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2d9804bed0 con 0x7f2d981095f0 2024-02-28T16:25:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.480+0000 7f2d8e000700 1 -- 172.21.15.1:0/1978442791 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167157 (secure 0 0 0) 0x7f2d84056140 con 0x7f2d981095f0 2024-02-28T16:25:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:25:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d7c0692b0 msgr2=0x7f2d7c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d7c0692b0 0x7f2d7c06b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d88000c00 tx=0x7f2d88009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 msgr2=0x7f2d9811a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 0x7f2d9811a530 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d84002a50 tx=0x7f2d840046b0 comp rx=0 tx=0).stop 2024-02-28T16:25:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 shutdown_connections 2024-02-28T16:25:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d7c0692b0 0x7f2d7c06b760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:24.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d98103dc0 0x7f2d98119ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:24.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 --2- 172.21.15.1:0/1978442791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d981095f0 0x7f2d9811a530 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:24.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 >> 172.21.15.1:0/1978442791 conn(0x7f2d98072d50 msgr2=0x7f2d98074760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:24.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 shutdown_connections 2024-02-28T16:25:24.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:24.485+0000 7f2d9d000700 1 -- 172.21.15.1:0/1978442791 wait complete. 2024-02-28T16:25:25.330 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":19,"active_gid":14212,"active_name":"smithi001.rmthtx","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":762266793},{"type":"v1","addr":"172.21.15.1:6801","nonce":762266793}]},"active_addr":"172.21.15.1:6801/762266793","active_change":"2024-02-28T16:20:26.347340+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14234,"name":"smithi066.ztraxe","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.1:8443/","prometheus":"http://172.21.15.1:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":968413644}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":1492016386}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2866890740}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":202779563}]}]} 2024-02-28T16:25:25.334 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-28T16:25:25.334 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-28T16:25:25.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-28T16:25:25.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:25 smithi001 ceph-mon[28991]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:25.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:25 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1978442791' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-28T16:25:25.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:25 smithi066 ceph-mon[36236]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:25.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:25 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1978442791' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-28T16:25:25.848 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:26.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:25:26.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:25:27.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:27 smithi001 ceph-mon[28991]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:27.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:27 smithi066 ceph-mon[36236]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:27.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.902+0000 7ff92f200700 1 -- 172.21.15.1:0/4101354159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930105930 msgr2=0x7ff9300fefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.902+0000 7ff92f200700 1 --2- 172.21.15.1:0/4101354159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930105930 0x7ff9300fefa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff91c004940 tx=0x7ff91c009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.902+0000 7ff92f200700 1 -- 172.21.15.1:0/4101354159 shutdown_connections 2024-02-28T16:25:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.902+0000 7ff92f200700 1 --2- 172.21.15.1:0/4101354159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930105930 0x7ff9300fefa0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.902+0000 7ff92f200700 1 --2- 172.21.15.1:0/4101354159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff930104f90 0x7ff930105360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.902+0000 7ff92f200700 1 -- 172.21.15.1:0/4101354159 >> 172.21.15.1:0/4101354159 conn(0x7ff9300fab40 msgr2=0x7ff9300fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.903+0000 7ff92f200700 1 -- 172.21.15.1:0/4101354159 shutdown_connections 2024-02-28T16:25:27.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.903+0000 7ff92f200700 1 -- 172.21.15.1:0/4101354159 wait complete. 2024-02-28T16:25:27.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.903+0000 7ff92f200700 1 Processor -- start 2024-02-28T16:25:27.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.903+0000 7ff92f200700 1 -- start start 2024-02-28T16:25:27.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.904+0000 7ff92f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930104f90 0x7ff930115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:27.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.904+0000 7ff92f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff930105930 0x7ff930115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:27.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.904+0000 7ff92f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9301165e0 con 0x7ff930104f90 2024-02-28T16:25:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.904+0000 7ff92f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93010f910 con 0x7ff930105930 2024-02-28T16:25:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.904+0000 7ff92de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930104f90 0x7ff930115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.904+0000 7ff92de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930104f90 0x7ff930115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48134/0 (socket says 172.21.15.1:48134) 2024-02-28T16:25:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.904+0000 7ff92de00700 1 -- 172.21.15.1:0/3876865372 learned_addr learned my addr 172.21.15.1:0/3876865372 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff92d400700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff930105930 0x7ff930115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff92de00700 1 -- 172.21.15.1:0/3876865372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff930105930 msgr2=0x7ff930115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff92de00700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff930105930 0x7ff930115f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:27.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff92de00700 1 -- 172.21.15.1:0/3876865372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c005040 con 0x7ff930104f90 2024-02-28T16:25:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff92de00700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930104f90 0x7ff930115a50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff92000c8c0 tx=0x7ff9200044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff926000700 1 -- 172.21.15.1:0/3876865372 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff92000a980 con 0x7ff930104f90 2024-02-28T16:25:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff926000700 1 -- 172.21.15.1:0/3876865372 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff92000ce60 con 0x7ff930104f90 2024-02-28T16:25:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93010fbf0 con 0x7ff930104f90 2024-02-28T16:25:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff926000700 1 -- 172.21.15.1:0/3876865372 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9200115e0 con 0x7ff930104f90 2024-02-28T16:25:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.905+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9301101b0 con 0x7ff930104f90 2024-02-28T16:25:27.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.907+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff930107460 con 0x7ff930104f90 2024-02-28T16:25:27.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.908+0000 7ff926000700 1 -- 172.21.15.1:0/3876865372 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff920012400 con 0x7ff930104f90 2024-02-28T16:25:27.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.908+0000 7ff926000700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff9140692b0 0x7ff91406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:27.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.909+0000 7ff926000700 1 -- 172.21.15.1:0/3876865372 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff920086f10 con 0x7ff930104f90 2024-02-28T16:25:27.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.909+0000 7ff92d400700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff9140692b0 0x7ff91406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:27.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.909+0000 7ff92d400700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff9140692b0 0x7ff91406b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff930111280 tx=0x7ff91c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:27.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:27.913+0000 7ff926000700 1 -- 172.21.15.1:0/3876865372 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff920056ff0 con 0x7ff930104f90 2024-02-28T16:25:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.128+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff93004bed0 con 0x7ff930104f90 2024-02-28T16:25:28.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.130+0000 7ff926000700 1 -- 172.21.15.1:0/3876865372 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10970 (secure 0 0 0) 0x7ff9200126b0 con 0x7ff930104f90 2024-02-28T16:25:28.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:25:28.120 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","created":"2024-02-28T16:17:42.450467+0000","modified":"2024-02-28T16:25:03.518590+0000","last_up_change":"2024-02-28T16:25:02.506452+0000","last_in_change":"2024-02-28T16:24:36.954989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-28T16:23:15.048529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ac92369f-8144-4695-87fe-f9cd73fb5f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6803","nonce":3270329297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6805","nonce":3270329297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6809","nonce":3270329297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6807","nonce":3270329297}]},"public_addr":"172.21.15.1:6803/3270329297","cluster_addr":"172.21.15.1:6805/3270329297","heartbeat_back_addr":"172.21.15.1:6809/3270329297","heartbeat_front_addr":"172.21.15.1:6807/3270329297","state":["exists","up"]},{"osd":1,"uuid":"bfd6ea17-fd9b-46ae-b3ff-547d736cd232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6811","nonce":1496879960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6813","nonce":1496879960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6817","nonce":1496879960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6815","nonce":1496879960}]},"public_addr":"172.21.15.1:6811/1496879960","cluster_addr":"172.21.15.1:6813/1496879960","heartbeat_back_addr":"172.21.15.1:6817/1496879960","heartbeat_front_addr":"172.21.15.1:6815/1496879960","state":["exists","up"]},{"osd":2,"uuid":"b713f513-6604-4e88-8cc1-0f963d9f8599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6819","nonce":131562471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6821","nonce":131562471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6825","nonce":131562471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6823","nonce":131562471}]},"public_addr":"172.21.15.1:6819/131562471","cluster_addr":"172.21.15.1:6821/131562471","heartbeat_back_addr":"172.21.15.1:6825/131562471","heartbeat_front_addr":"172.21.15.1:6823/131562471","state":["exists","up"]},{"osd":3,"uuid":"4f425b10-672f-4265-a314-5a9a15237257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6801","nonce":863585173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6803","nonce":863585173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6807","nonce":863585173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6805","nonce":863585173}]},"public_addr":"172.21.15.66:6801/863585173","cluster_addr":"172.21.15.66:6803/863585173","heartbeat_back_addr":"172.21.15.66:6807/863585173","heartbeat_front_addr":"172.21.15.66:6805/863585173","state":["exists","up"]},{"osd":4,"uuid":"c1f2651b-47f3-444e-9325-22a0a962c3b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6809","nonce":376874414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6811","nonce":376874414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6815","nonce":376874414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6813","nonce":376874414}]},"public_addr":"172.21.15.66:6809/376874414","cluster_addr":"172.21.15.66:6811/376874414","heartbeat_back_addr":"172.21.15.66:6815/376874414","heartbeat_front_addr":"172.21.15.66:6813/376874414","state":["exists","up"]},{"osd":5,"uuid":"e83df14f-7d81-4bb7-9617-024ad94de1a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6817","nonce":2849488501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6819","nonce":2849488501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6823","nonce":2849488501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6821","nonce":2849488501}]},"public_addr":"172.21.15.66:6817/2849488501","cluster_addr":"172.21.15.66:6819/2849488501","heartbeat_back_addr":"172.21.15.66:6823/2849488501","heartbeat_front_addr":"172.21.15.66:6821/2849488501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:01.831777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:36.505002+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:13.091284+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:47.260219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:24:23.366141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:25:00.532840+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6800/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/298653290":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/3865282714":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/4195777758":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/2061955559":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/4223980442":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1339161018":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/833315191":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6801/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:6801/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6801/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/2780166226":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/1224289276":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3320028482":"2024-02-29T16:18:55.593679+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-28T16:25:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff9140692b0 msgr2=0x7ff91406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff9140692b0 0x7ff91406b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff930111280 tx=0x7ff91c009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930104f90 msgr2=0x7ff930115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930104f90 0x7ff930115a50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff92000c8c0 tx=0x7ff9200044c0 comp rx=0 tx=0).stop 2024-02-28T16:25:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 shutdown_connections 2024-02-28T16:25:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff9140692b0 0x7ff91406b760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff930104f90 0x7ff930115a50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 --2- 172.21.15.1:0/3876865372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff930105930 0x7ff930115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 >> 172.21.15.1:0/3876865372 conn(0x7ff9300fab40 msgr2=0x7ff9300fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 shutdown_connections 2024-02-28T16:25:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:28.133+0000 7ff92f200700 1 -- 172.21.15.1:0/3876865372 wait complete. 2024-02-28T16:25:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:28 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3876865372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T16:25:28.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:28 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3876865372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T16:25:28.892 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-28T16:25:28.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-28T16:25:29.399 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:29.726 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:29 smithi001 ceph-mon[28991]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:29.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:29 smithi066 ceph-mon[36236]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:31.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.523+0000 7f59d3200700 1 -- 172.21.15.1:0/48063267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 msgr2=0x7f59cc0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:31.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.523+0000 7f59d3200700 1 --2- 172.21.15.1:0/48063267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 0x7f59cc0fcf60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f59b8004940 tx=0x7f59b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:31.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.523+0000 7f59d3200700 1 -- 172.21.15.1:0/48063267 shutdown_connections 2024-02-28T16:25:31.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.523+0000 7f59d3200700 1 --2- 172.21.15.1:0/48063267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 0x7f59cc0fcf60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.523+0000 7f59d3200700 1 --2- 172.21.15.1:0/48063267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc102f50 0x7f59cc103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.523+0000 7f59d3200700 1 -- 172.21.15.1:0/48063267 >> 172.21.15.1:0/48063267 conn(0x7f59cc0f8ae0 msgr2=0x7f59cc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.523+0000 7f59d3200700 1 -- 172.21.15.1:0/48063267 shutdown_connections 2024-02-28T16:25:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.523+0000 7f59d3200700 1 -- 172.21.15.1:0/48063267 wait complete. 2024-02-28T16:25:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.524+0000 7f59d3200700 1 Processor -- start 2024-02-28T16:25:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.524+0000 7f59d3200700 1 -- start start 2024-02-28T16:25:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.525+0000 7f59d3200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc102f50 0x7f59cc070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:31.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.525+0000 7f59d3200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 0x7f59cc0707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.525+0000 7f59d3200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc070e40 con 0x7f59cc1038f0 2024-02-28T16:25:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.525+0000 7f59d3200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc070fb0 con 0x7f59cc102f50 2024-02-28T16:25:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.525+0000 7f59cb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 0x7f59cc0707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.525+0000 7f59cb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 0x7f59cc0707f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48144/0 (socket says 172.21.15.1:48144) 2024-02-28T16:25:31.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.525+0000 7f59cb400700 1 -- 172.21.15.1:0/1650999180 learned_addr learned my addr 172.21.15.1:0/1650999180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.525+0000 7f59cbe00700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc102f50 0x7f59cc070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59cb400700 1 -- 172.21.15.1:0/1650999180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc102f50 msgr2=0x7f59cc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59cb400700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc102f50 0x7f59cc070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59cb400700 1 -- 172.21.15.1:0/1650999180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8005040 con 0x7f59cc1038f0 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59cbe00700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc102f50 0x7f59cc070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59cb400700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 0x7f59cc0707f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f59b8000c00 tx=0x7f59b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59c3e00700 1 -- 172.21.15.1:0/1650999180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59b8021070 con 0x7f59cc1038f0 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59c3e00700 1 -- 172.21.15.1:0/1650999180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59b8013dd0 con 0x7f59cc1038f0 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59c3e00700 1 -- 172.21.15.1:0/1650999180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59b800e070 con 0x7f59cc1038f0 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.526+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc119cb0 con 0x7f59cc1038f0 2024-02-28T16:25:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.527+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc11a140 con 0x7f59cc1038f0 2024-02-28T16:25:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.528+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc105420 con 0x7f59cc1038f0 2024-02-28T16:25:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.529+0000 7f59c3e00700 1 -- 172.21.15.1:0/1650999180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f59b801cbf0 con 0x7f59cc1038f0 2024-02-28T16:25:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.530+0000 7f59c3e00700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f59b00692a0 0x7f59b006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.530+0000 7f59c3e00700 1 -- 172.21.15.1:0/1650999180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f59b80883a0 con 0x7f59cc1038f0 2024-02-28T16:25:31.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.530+0000 7f59cbe00700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f59b00692a0 0x7f59b006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:31.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.530+0000 7f59cbe00700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f59b00692a0 0x7f59b006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f59bc004b90 tx=0x7f59bc00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.535+0000 7f59c3e00700 1 -- 172.21.15.1:0/1650999180 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f59b80549f0 con 0x7f59cc1038f0 2024-02-28T16:25:31.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.755+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59cc04bed0 con 0x7f59cc1038f0 2024-02-28T16:25:31.745 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:31 smithi001 ceph-mon[28991]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:31.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.756+0000 7f59c3e00700 1 -- 172.21.15.1:0/1650999180 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10970 (secure 0 0 0) 0x7f59b8058010 con 0x7f59cc1038f0 2024-02-28T16:25:31.746 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:25:31.746 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","created":"2024-02-28T16:17:42.450467+0000","modified":"2024-02-28T16:25:03.518590+0000","last_up_change":"2024-02-28T16:25:02.506452+0000","last_in_change":"2024-02-28T16:24:36.954989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-28T16:23:15.048529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ac92369f-8144-4695-87fe-f9cd73fb5f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6803","nonce":3270329297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6805","nonce":3270329297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6809","nonce":3270329297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6807","nonce":3270329297}]},"public_addr":"172.21.15.1:6803/3270329297","cluster_addr":"172.21.15.1:6805/3270329297","heartbeat_back_addr":"172.21.15.1:6809/3270329297","heartbeat_front_addr":"172.21.15.1:6807/3270329297","state":["exists","up"]},{"osd":1,"uuid":"bfd6ea17-fd9b-46ae-b3ff-547d736cd232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6811","nonce":1496879960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6813","nonce":1496879960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6817","nonce":1496879960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6815","nonce":1496879960}]},"public_addr":"172.21.15.1:6811/1496879960","cluster_addr":"172.21.15.1:6813/1496879960","heartbeat_back_addr":"172.21.15.1:6817/1496879960","heartbeat_front_addr":"172.21.15.1:6815/1496879960","state":["exists","up"]},{"osd":2,"uuid":"b713f513-6604-4e88-8cc1-0f963d9f8599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6819","nonce":131562471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6821","nonce":131562471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6825","nonce":131562471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6823","nonce":131562471}]},"public_addr":"172.21.15.1:6819/131562471","cluster_addr":"172.21.15.1:6821/131562471","heartbeat_back_addr":"172.21.15.1:6825/131562471","heartbeat_front_addr":"172.21.15.1:6823/131562471","state":["exists","up"]},{"osd":3,"uuid":"4f425b10-672f-4265-a314-5a9a15237257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6801","nonce":863585173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6803","nonce":863585173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6807","nonce":863585173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6805","nonce":863585173}]},"public_addr":"172.21.15.66:6801/863585173","cluster_addr":"172.21.15.66:6803/863585173","heartbeat_back_addr":"172.21.15.66:6807/863585173","heartbeat_front_addr":"172.21.15.66:6805/863585173","state":["exists","up"]},{"osd":4,"uuid":"c1f2651b-47f3-444e-9325-22a0a962c3b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6809","nonce":376874414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6811","nonce":376874414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6815","nonce":376874414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6813","nonce":376874414}]},"public_addr":"172.21.15.66:6809/376874414","cluster_addr":"172.21.15.66:6811/376874414","heartbeat_back_addr":"172.21.15.66:6815/376874414","heartbeat_front_addr":"172.21.15.66:6813/376874414","state":["exists","up"]},{"osd":5,"uuid":"e83df14f-7d81-4bb7-9617-024ad94de1a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6817","nonce":2849488501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6819","nonce":2849488501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6823","nonce":2849488501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6821","nonce":2849488501}]},"public_addr":"172.21.15.66:6817/2849488501","cluster_addr":"172.21.15.66:6819/2849488501","heartbeat_back_addr":"172.21.15.66:6823/2849488501","heartbeat_front_addr":"172.21.15.66:6821/2849488501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:01.831777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:36.505002+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:13.091284+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:47.260219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:24:23.366141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:25:00.532840+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6800/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/298653290":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/3865282714":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/4195777758":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/2061955559":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/4223980442":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1339161018":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/833315191":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6801/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:6801/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6801/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/2780166226":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/1224289276":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3320028482":"2024-02-29T16:18:55.593679+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-28T16:25:31.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f59b00692a0 msgr2=0x7f59b006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:31.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f59b00692a0 0x7f59b006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f59bc004b90 tx=0x7f59bc00b040 comp rx=0 tx=0).stop 2024-02-28T16:25:31.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 msgr2=0x7f59cc0707f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:31.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 0x7f59cc0707f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f59b8000c00 tx=0x7f59b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:31.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 shutdown_connections 2024-02-28T16:25:31.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f59b00692a0 0x7f59b006b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:31.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc102f50 0x7f59cc070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:31.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 --2- 172.21.15.1:0/1650999180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc1038f0 0x7f59cc0707f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:31.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.759+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 >> 172.21.15.1:0/1650999180 conn(0x7f59cc0f8ae0 msgr2=0x7f59cc0fa520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:31.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.760+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 shutdown_connections 2024-02-28T16:25:31.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:31.760+0000 7f59d3200700 1 -- 172.21.15.1:0/1650999180 wait complete. 2024-02-28T16:25:31.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:31 smithi066 ceph-mon[36236]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:32.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-28T16:25:32.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-28T16:25:32.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-28T16:25:32.489 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-28T16:25:32.489 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-28T16:25:32.490 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-28T16:25:32.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:32 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1650999180' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T16:25:32.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:32 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1650999180' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T16:25:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:33.059 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:33.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:33 smithi001 ceph-mon[28991]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:33.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:33 smithi066 ceph-mon[36236]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:35.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:35 smithi001 ceph-mon[28991]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:35.526 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:35.528 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:35.534 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:35 smithi066 ceph-mon[36236]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:37.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.946+0000 7f0f07a00700 1 -- 172.21.15.1:0/4049357075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00102f50 msgr2=0x7f0f00103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.946+0000 7f0f07a00700 1 --2- 172.21.15.1:0/4049357075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00102f50 0x7f0f00103320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009ce0 tx=0x7f0ef4004d10 comp rx=0 tx=0).stop 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.947+0000 7f0f07a00700 1 -- 172.21.15.1:0/4049357075 shutdown_connections 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.947+0000 7f0f07a00700 1 --2- 172.21.15.1:0/4049357075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f001038f0 0x7f0f000fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.947+0000 7f0f07a00700 1 --2- 172.21.15.1:0/4049357075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00102f50 0x7f0f00103320 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.947+0000 7f0f07a00700 1 -- 172.21.15.1:0/4049357075 >> 172.21.15.1:0/4049357075 conn(0x7f0f000f8ae0 msgr2=0x7f0f000faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.947+0000 7f0f07a00700 1 -- 172.21.15.1:0/4049357075 shutdown_connections 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.947+0000 7f0f07a00700 1 -- 172.21.15.1:0/4049357075 wait complete. 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.948+0000 7f0f07a00700 1 Processor -- start 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.948+0000 7f0f07a00700 1 -- start start 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0f07a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00102f50 0x7f0f00070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0f07a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f001038f0 0x7f0f000707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0f07a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00111b80 con 0x7f0f001038f0 2024-02-28T16:25:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0f07a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00070d10 con 0x7f0f00102f50 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0effe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f001038f0 0x7f0f000707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0effe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f001038f0 0x7f0f000707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57402/0 (socket says 172.21.15.1:57402) 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0effe00700 1 -- 172.21.15.1:0/56809079 learned_addr learned my addr 172.21.15.1:0/56809079 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0f05000700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00102f50 0x7f0f00070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0effe00700 1 -- 172.21.15.1:0/56809079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00102f50 msgr2=0x7f0f00070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0effe00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00102f50 0x7f0f00070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.949+0000 7f0effe00700 1 -- 172.21.15.1:0/56809079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef4009940 con 0x7f0f001038f0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.950+0000 7f0effe00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f001038f0 0x7f0f000707d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00b790 tx=0x7f0eec009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.950+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0eec00f6f0 con 0x7f0f001038f0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.950+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0eec00fd20 con 0x7f0f001038f0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.950+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f00070ff0 con 0x7f0f001038f0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.950+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0eec0103a0 con 0x7f0f001038f0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.950+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0011e7f0 con 0x7f0f001038f0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.951+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f000faee0 con 0x7f0f001038f0 2024-02-28T16:25:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.952+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0eec011400 con 0x7f0f001038f0 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.953+0000 7f0efcc00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0ee80692b0 0x7f0ee806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.953+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0eec086d40 con 0x7f0f001038f0 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.953+0000 7f0f05000700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0ee80692b0 0x7f0ee806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.954+0000 7f0efcc00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] conn(0x7f0ee806ef80 0x7f0ee8071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.954+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 --> [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ee8071a40 con 0x7f0ee806ef80 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.954+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0eec089290 con 0x7f0f001038f0 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.954+0000 7f0f05a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] conn(0x7f0ee806ef80 0x7f0ee8071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.954+0000 7f0f05000700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0ee80692b0 0x7f0ee806b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400b740 tx=0x7f0ef4004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.954+0000 7f0f05a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] conn(0x7f0ee806ef80 0x7f0ee8071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:37.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.955+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 <== osd.0 v2:172.21.15.1:6802/3270329297 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0ee8071a40 con 0x7f0ee806ef80 2024-02-28T16:25:37.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.971+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 --> [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f000fc6e0 con 0x7f0ee806ef80 2024-02-28T16:25:37.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0efcc00700 1 -- 172.21.15.1:0/56809079 <== osd.0 v2:172.21.15.1:6802/3270329297 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0f000fc6e0 con 0x7f0ee806ef80 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] conn(0x7f0ee806ef80 msgr2=0x7f0ee8071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] conn(0x7f0ee806ef80 0x7f0ee8071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0ee80692b0 msgr2=0x7f0ee806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0ee80692b0 0x7f0ee806b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400b740 tx=0x7f0ef4004d10 comp rx=0 tx=0).stop 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f001038f0 msgr2=0x7f0f000707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f001038f0 0x7f0f000707d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00b790 tx=0x7f0eec009220 comp rx=0 tx=0).stop 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 shutdown_connections 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0ee80692b0 0x7f0ee806b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:6802/3270329297,v1:172.21.15.1:6803/3270329297] conn(0x7f0ee806ef80 0x7f0ee8071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00102f50 0x7f0f00070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 --2- 172.21.15.1:0/56809079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f001038f0 0x7f0f000707d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.972+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 >> 172.21.15.1:0/56809079 conn(0x7f0f000f8ae0 msgr2=0x7f0f000fa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:37.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.973+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 shutdown_connections 2024-02-28T16:25:37.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:36.973+0000 7f0f07a00700 1 -- 172.21.15.1:0/56809079 wait complete. 2024-02-28T16:25:37.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.590+0000 7fbfd3c00700 1 -- 172.21.15.1:0/2933626802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070c40 msgr2=0x7fbfcc109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.590+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/2933626802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070c40 0x7fbfcc109260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8004970 tx=0x7fbfb8009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:37.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.590+0000 7fbfd3c00700 1 -- 172.21.15.1:0/2933626802 shutdown_connections 2024-02-28T16:25:37.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.590+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/2933626802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070c40 0x7fbfcc109260 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.590+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/2933626802 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbfcc070330 0x7fbfcc070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.590+0000 7fbfd3c00700 1 -- 172.21.15.1:0/2933626802 >> 172.21.15.1:0/2933626802 conn(0x7fbfcc0f8ae0 msgr2=0x7fbfcc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:37.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.590+0000 7fbfd3c00700 1 -- 172.21.15.1:0/2933626802 shutdown_connections 2024-02-28T16:25:37.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.591+0000 7fbfd3c00700 1 -- 172.21.15.1:0/2933626802 wait complete. 2024-02-28T16:25:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.591+0000 7fbfd3c00700 1 Processor -- start 2024-02-28T16:25:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.592+0000 7fbfd3c00700 1 -- start start 2024-02-28T16:25:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.592+0000 7fbfd3c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070330 0x7fbfcc115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.592+0000 7fbfd3c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbfcc070c40 0x7fbfcc116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.592+0000 7fbfd3c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc116790 con 0x7fbfcc070330 2024-02-28T16:25:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.592+0000 7fbfd3c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc10fae0 con 0x7fbfcc070c40 2024-02-28T16:25:37.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.592+0000 7fbfd1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070330 0x7fbfcc115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:37.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.592+0000 7fbfd1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070330 0x7fbfcc115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57432/0 (socket says 172.21.15.1:57432) 2024-02-28T16:25:37.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.592+0000 7fbfd1200700 1 -- 172.21.15.1:0/3416893689 learned_addr learned my addr 172.21.15.1:0/3416893689 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:37.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.593+0000 7fbfcbe00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbfcc070c40 0x7fbfcc116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.593+0000 7fbfd1200700 1 -- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbfcc070c40 msgr2=0x7fbfcc116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.593+0000 7fbfd1200700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbfcc070c40 0x7fbfcc116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.593+0000 7fbfd1200700 1 -- 172.21.15.1:0/3416893689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb8005040 con 0x7fbfcc070330 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.593+0000 7fbfd1200700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070330 0x7fbfcc115c00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000cac0 tx=0x7fbfc0007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.594+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfc000a980 con 0x7fbfcc070330 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.594+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfc0012440 con 0x7fbfcc070330 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.594+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfcc10fdc0 con 0x7fbfcc070330 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.594+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfc00115e0 con 0x7fbfcc070330 2024-02-28T16:25:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.594+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfcc1102d0 con 0x7fbfcc070330 2024-02-28T16:25:37.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.595+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbfcc1107d0 con 0x7fbfcc070330 2024-02-28T16:25:37.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.596+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fbfc00125b0 con 0x7fbfcc070330 2024-02-28T16:25:37.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.597+0000 7fbfc8c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fbfb40692a0 0x7fbfb406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:37.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.597+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbfc00875a0 con 0x7fbfcc070330 2024-02-28T16:25:37.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.597+0000 7fbfcbe00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fbfb40692a0 0x7fbfb406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:37.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.597+0000 7fbfc8c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] conn(0x7fbfb406ef70 0x7fbfb4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:37.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.597+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 --> [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbfb4071a30 con 0x7fbfb406ef70 2024-02-28T16:25:37.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.597+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfc0089bf0 con 0x7fbfcc070330 2024-02-28T16:25:37.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.598+0000 7fbfcbe00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fbfb40692a0 0x7fbfb406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8000c00 tx=0x7fbfb8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:37.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.598+0000 7fbfd1c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] conn(0x7fbfb406ef70 0x7fbfb4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:37.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.599+0000 7fbfd1c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] conn(0x7fbfb406ef70 0x7fbfb4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:37.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.601+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 <== osd.4 v2:172.21.15.66:6808/376874414 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fbfb4071a30 con 0x7fbfb406ef70 2024-02-28T16:25:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.617+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 --> [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbfcc05e690 con 0x7fbfb406ef70 2024-02-28T16:25:37.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.618+0000 7fbfc8c00700 1 -- 172.21.15.1:0/3416893689 <== osd.4 v2:172.21.15.66:6808/376874414 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbfcc05e690 con 0x7fbfb406ef70 2024-02-28T16:25:37.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] conn(0x7fbfb406ef70 msgr2=0x7fbfb4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] conn(0x7fbfb406ef70 0x7fbfb4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fbfb40692a0 msgr2=0x7fbfb406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fbfb40692a0 0x7fbfb406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8000c00 tx=0x7fbfb8009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070330 msgr2=0x7fbfcc115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070330 0x7fbfcc115c00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000cac0 tx=0x7fbfc0007ee0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 shutdown_connections 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:6808/376874414,v1:172.21.15.66:6809/376874414] conn(0x7fbfb406ef70 0x7fbfb4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fbfb40692a0 0x7fbfb406b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfcc070330 0x7fbfcc115c00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 --2- 172.21.15.1:0/3416893689 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbfcc070c40 0x7fbfcc116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 >> 172.21.15.1:0/3416893689 conn(0x7fbfcc0f8ae0 msgr2=0x7fbfcc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.619+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 shutdown_connections 2024-02-28T16:25:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:37.620+0000 7fbfd3c00700 1 -- 172.21.15.1:0/3416893689 wait complete. 2024-02-28T16:25:37.674 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:37 smithi001 ceph-mon[28991]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:37 smithi066 ceph-mon[36236]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:38.695 INFO:teuthology.orchestra.run.smithi001.stdout:146028888080 2024-02-28T16:25:38.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-28T16:25:39.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:39 smithi001 ceph-mon[28991]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:39.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:39 smithi066 ceph-mon[36236]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:39.895 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:41.016 INFO:teuthology.orchestra.run.smithi001.stdout:38654705709 2024-02-28T16:25:41.017 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-28T16:25:41.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:41 smithi066 ceph-mon[36236]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:41.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:25:41.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:41 smithi001 ceph-mon[28991]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:41.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:25:42.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.994+0000 7f00f6a00700 1 -- 172.21.15.1:0/2542050874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fbdd0 msgr2=0x7f00f00fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:42.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.994+0000 7f00f6a00700 1 --2- 172.21.15.1:0/2542050874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fbdd0 0x7f00f00fc1a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f00e0004940 tx=0x7f00e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:42.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.994+0000 7f00f6a00700 1 -- 172.21.15.1:0/2542050874 shutdown_connections 2024-02-28T16:25:42.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.994+0000 7f00f6a00700 1 --2- 172.21.15.1:0/2542050874 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00f00fc6e0 0x7f00f0109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.994+0000 7f00f6a00700 1 --2- 172.21.15.1:0/2542050874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fbdd0 0x7f00f00fc1a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.994+0000 7f00f6a00700 1 -- 172.21.15.1:0/2542050874 >> 172.21.15.1:0/2542050874 conn(0x7f00f0071d90 msgr2=0x7f00f0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:42.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.994+0000 7f00f6a00700 1 -- 172.21.15.1:0/2542050874 shutdown_connections 2024-02-28T16:25:42.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.994+0000 7f00f6a00700 1 -- 172.21.15.1:0/2542050874 wait complete. 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.995+0000 7f00f6a00700 1 Processor -- start 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.995+0000 7f00f6a00700 1 -- start start 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00f6a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00f00fbdd0 0x7f00f0115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00f6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fc6e0 0x7f00f0116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00f6a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0116860 con 0x7f00f00fc6e0 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00f6a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f010fb70 con 0x7f00f00fbdd0 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00ef400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fc6e0 0x7f00f0116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00ef400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fc6e0 0x7f00f0116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50198/0 (socket says 172.21.15.1:50198) 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00ef400700 1 -- 172.21.15.1:0/373071974 learned_addr learned my addr 172.21.15.1:0/373071974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00efe00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00f00fbdd0 0x7f00f0115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00ef400700 1 -- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00f00fbdd0 msgr2=0x7f00f0115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00ef400700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00f00fbdd0 0x7f00f0115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.996+0000 7f00ef400700 1 -- 172.21.15.1:0/373071974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e0005040 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.997+0000 7f00ef400700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fc6e0 0x7f00f0116210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f00dc002a50 tx=0x7f00dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.997+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00dc007cc0 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.997+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00dc00f040 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.997+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00dc009640 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.997+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f010fe50 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.997+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f0110390 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:41.998+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f00f0110840 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.000+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f00dc0097a0 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.000+0000 7f00e7e00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f00d40692b0 0x7f00d406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.001+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f00dc086b70 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.001+0000 7f00efe00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f00d40692b0 0x7f00d406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.001+0000 7f00e7e00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] conn(0x7f00d406ef80 0x7f00d4071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.001+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 --> [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00d4071a40 con 0x7f00d406ef80 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.001+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00dc08b220 con 0x7f00f00fc6e0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.001+0000 7f00efe00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f00d40692b0 0x7f00d406b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f00e000b3b0 tx=0x7f00e0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:42.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.001+0000 7f00f4a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] conn(0x7f00d406ef80 0x7f00d4071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:42.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.002+0000 7f00f4a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] conn(0x7f00d406ef80 0x7f00d4071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:42.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.005+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 <== osd.3 v2:172.21.15.66:6800/863585173 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f00d4071a40 con 0x7f00d406ef80 2024-02-28T16:25:42.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.022+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 --> [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f00f005e690 con 0x7f00d406ef80 2024-02-28T16:25:42.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.023+0000 7f00e7e00700 1 -- 172.21.15.1:0/373071974 <== osd.3 v2:172.21.15.66:6800/863585173 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f00f005e690 con 0x7f00d406ef80 2024-02-28T16:25:42.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.023+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] conn(0x7f00d406ef80 msgr2=0x7f00d4071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:42.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.023+0000 7f00f6a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] conn(0x7f00d406ef80 0x7f00d4071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f00d40692b0 msgr2=0x7f00d406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:42.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f00d40692b0 0x7f00d406b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f00e000b3b0 tx=0x7f00e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:42.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fc6e0 msgr2=0x7f00f0116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:42.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fc6e0 0x7f00f0116210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f00dc002a50 tx=0x7f00dc0046b0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 shutdown_connections 2024-02-28T16:25:42.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f00d40692b0 0x7f00d406b760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00f00fbdd0 0x7f00f0115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.66:6800/863585173,v1:172.21.15.66:6801/863585173] conn(0x7f00d406ef80 0x7f00d4071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 --2- 172.21.15.1:0/373071974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00f00fc6e0 0x7f00f0116210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:42.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 >> 172.21.15.1:0/373071974 conn(0x7f00f0071d90 msgr2=0x7f00f00faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:42.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 shutdown_connections 2024-02-28T16:25:42.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:42.024+0000 7f00f6a00700 1 -- 172.21.15.1:0/373071974 wait complete. 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.153+0000 7f2f79600700 1 -- 172.21.15.1:0/2548312443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74070870 msgr2=0x7f2f74070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.153+0000 7f2f79600700 1 --2- 172.21.15.1:0/2548312443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74070870 0x7f2f74070ce0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2f60004970 tx=0x7f2f60009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.153+0000 7f2f79600700 1 -- 172.21.15.1:0/2548312443 shutdown_connections 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.153+0000 7f2f79600700 1 --2- 172.21.15.1:0/2548312443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74070870 0x7f2f74070ce0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.153+0000 7f2f79600700 1 --2- 172.21.15.1:0/2548312443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f74071f60 0x7f2f74070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.153+0000 7f2f79600700 1 -- 172.21.15.1:0/2548312443 >> 172.21.15.1:0/2548312443 conn(0x7f2f740f8af0 msgr2=0x7f2f740faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.153+0000 7f2f79600700 1 -- 172.21.15.1:0/2548312443 shutdown_connections 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.153+0000 7f2f79600700 1 -- 172.21.15.1:0/2548312443 wait complete. 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.154+0000 7f2f79600700 1 Processor -- start 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.154+0000 7f2f79600700 1 -- start start 2024-02-28T16:25:43.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.155+0000 7f2f79600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f74070870 0x7f2f7411a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.155+0000 7f2f79600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74071f60 0x7f2f7411a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.155+0000 7f2f79600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7411abd0 con 0x7f2f74071f60 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.155+0000 7f2f79600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74113ee0 con 0x7f2f74070870 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.155+0000 7f2f72000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74071f60 0x7f2f7411a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.155+0000 7f2f72000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74071f60 0x7f2f7411a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50216/0 (socket says 172.21.15.1:50216) 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.155+0000 7f2f72000700 1 -- 172.21.15.1:0/1849895299 learned_addr learned my addr 172.21.15.1:0/1849895299 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.155+0000 7f2f72a00700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f74070870 0x7f2f7411a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.156+0000 7f2f72000700 1 -- 172.21.15.1:0/1849895299 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f74070870 msgr2=0x7f2f7411a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.156+0000 7f2f72000700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f74070870 0x7f2f7411a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.156+0000 7f2f72000700 1 -- 172.21.15.1:0/1849895299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60005040 con 0x7f2f74071f60 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.156+0000 7f2f72000700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74071f60 0x7f2f7411a580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2f60004f50 tx=0x7f2f60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.156+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f6000e040 con 0x7f2f74071f60 2024-02-28T16:25:43.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.156+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f6000bc10 con 0x7f2f74071f60 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.156+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f74114160 con 0x7f2f74071f60 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.157+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f6001c630 con 0x7f2f74071f60 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.157+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f741146a0 con 0x7f2f74071f60 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.158+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f74114c10 con 0x7f2f74071f60 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.159+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2f6000bd80 con 0x7f2f74071f60 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.160+0000 7f2f6aa00700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2f580692b0 0x7f2f5806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.160+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2f60088cd0 con 0x7f2f74071f60 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.160+0000 7f2f72a00700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2f580692b0 0x7f2f5806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.160+0000 7f2f6aa00700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] conn(0x7f2f5806ef80 0x7f2f58071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.161+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 --> [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f58071a40 con 0x7f2f5806ef80 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.161+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f60089010 con 0x7f2f74071f60 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.161+0000 7f2f73400700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] conn(0x7f2f5806ef80 0x7f2f58071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.161+0000 7f2f72a00700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2f580692b0 0x7f2f5806b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f64007fd0 tx=0x7f2f64009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:43.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.162+0000 7f2f73400700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] conn(0x7f2f5806ef80 0x7f2f58071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:43.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.162+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 <== osd.1 v2:172.21.15.1:6810/1496879960 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2f58071a40 con 0x7f2f5806ef80 2024-02-28T16:25:43.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.179+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 --> [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2f7405e640 con 0x7f2f5806ef80 2024-02-28T16:25:43.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.180+0000 7f2f6aa00700 1 -- 172.21.15.1:0/1849895299 <== osd.1 v2:172.21.15.1:6810/1496879960 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2f7405e640 con 0x7f2f5806ef80 2024-02-28T16:25:43.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] conn(0x7f2f5806ef80 msgr2=0x7f2f58071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] conn(0x7f2f5806ef80 0x7f2f58071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2f580692b0 msgr2=0x7f2f5806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2f580692b0 0x7f2f5806b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f64007fd0 tx=0x7f2f64009040 comp rx=0 tx=0).stop 2024-02-28T16:25:43.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74071f60 msgr2=0x7f2f7411a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74071f60 0x7f2f7411a580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2f60004f50 tx=0x7f2f60009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:43.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 shutdown_connections 2024-02-28T16:25:43.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6810/1496879960,v1:172.21.15.1:6811/1496879960] conn(0x7f2f5806ef80 0x7f2f58071390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2f580692b0 0x7f2f5806b760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f74070870 0x7f2f7411a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 --2- 172.21.15.1:0/1849895299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f74071f60 0x7f2f7411a580 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 >> 172.21.15.1:0/1849895299 conn(0x7f2f740f8af0 msgr2=0x7f2f740fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:43.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 shutdown_connections 2024-02-28T16:25:43.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.181+0000 7f2f79600700 1 -- 172.21.15.1:0/1849895299 wait complete. 2024-02-28T16:25:43.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.450+0000 7fa413a00700 1 -- 172.21.15.1:0/3717611832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 msgr2=0x7fa40c070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.450+0000 7fa413a00700 1 --2- 172.21.15.1:0/3717611832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 0x7fa40c070ce0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8004970 tx=0x7fa3f8009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.450+0000 7fa413a00700 1 -- 172.21.15.1:0/3717611832 shutdown_connections 2024-02-28T16:25:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.450+0000 7fa413a00700 1 --2- 172.21.15.1:0/3717611832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 0x7fa40c070ce0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.450+0000 7fa413a00700 1 --2- 172.21.15.1:0/3717611832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa40c071f60 0x7fa40c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.450+0000 7fa413a00700 1 -- 172.21.15.1:0/3717611832 >> 172.21.15.1:0/3717611832 conn(0x7fa40c0f8ae0 msgr2=0x7fa40c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.450+0000 7fa413a00700 1 -- 172.21.15.1:0/3717611832 shutdown_connections 2024-02-28T16:25:43.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.450+0000 7fa413a00700 1 -- 172.21.15.1:0/3717611832 wait complete. 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.451+0000 7fa413a00700 1 Processor -- start 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.451+0000 7fa413a00700 1 -- start start 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.452+0000 7fa413a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 0x7fa40c11a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.452+0000 7fa413a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa40c071f60 0x7fa40c11a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.452+0000 7fa413a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c11aba0 con 0x7fa40c070870 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.452+0000 7fa413a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c113eb0 con 0x7fa40c071f60 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.452+0000 7fa411000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 0x7fa40c11a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.452+0000 7fa411000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 0x7fa40c11a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50234/0 (socket says 172.21.15.1:50234) 2024-02-28T16:25:43.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.452+0000 7fa411000700 1 -- 172.21.15.1:0/149432994 learned_addr learned my addr 172.21.15.1:0/149432994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:43.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa40be00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa40c071f60 0x7fa40c11a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:43.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa411000700 1 -- 172.21.15.1:0/149432994 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa40c071f60 msgr2=0x7fa40c11a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa411000700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa40c071f60 0x7fa40c11a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa411000700 1 -- 172.21.15.1:0/149432994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f8005040 con 0x7fa40c070870 2024-02-28T16:25:43.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa411000700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 0x7fa40c11a010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa40000c830 tx=0x7fa400004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40000a760 con 0x7fa40c070870 2024-02-28T16:25:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa40000ad90 con 0x7fa40c070870 2024-02-28T16:25:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4000113a0 con 0x7fa40c070870 2024-02-28T16:25:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.453+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40c114190 con 0x7fa40c070870 2024-02-28T16:25:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.454+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40c1146d0 con 0x7fa40c070870 2024-02-28T16:25:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.455+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa40c114be0 con 0x7fa40c070870 2024-02-28T16:25:43.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.456+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa400012400 con 0x7fa40c070870 2024-02-28T16:25:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.457+0000 7fa408c00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3f4069370 0x7fa3f406b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.457+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa400086ec0 con 0x7fa40c070870 2024-02-28T16:25:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.457+0000 7fa408c00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] conn(0x7fa3f406f040 0x7fa3f4071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.457+0000 7fa40be00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3f4069370 0x7fa3f406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.457+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 --> [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3f4071b00 con 0x7fa3f406f040 2024-02-28T16:25:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.457+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa400089400 con 0x7fa40c070870 2024-02-28T16:25:43.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.458+0000 7fa411a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] conn(0x7fa3f406f040 0x7fa3f4071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:43.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.458+0000 7fa40be00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3f4069370 0x7fa3f406b820 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa40c1158b0 tx=0x7fa3f8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:43.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.459+0000 7fa411a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] conn(0x7fa3f406f040 0x7fa3f4071450 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:43.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.460+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 <== osd.2 v2:172.21.15.1:6818/131562471 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa3f4071b00 con 0x7fa3f406f040 2024-02-28T16:25:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.477+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 --> [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa40c05e640 con 0x7fa3f406f040 2024-02-28T16:25:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.478+0000 7fa408c00700 1 -- 172.21.15.1:0/149432994 <== osd.2 v2:172.21.15.1:6818/131562471 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa40c05e640 con 0x7fa3f406f040 2024-02-28T16:25:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.478+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] conn(0x7fa3f406f040 msgr2=0x7fa3f4071450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.478+0000 7fa413a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] conn(0x7fa3f406f040 0x7fa3f4071450 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3f4069370 msgr2=0x7fa3f406b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3f4069370 0x7fa3f406b820 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa40c1158b0 tx=0x7fa3f8009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 msgr2=0x7fa40c11a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 0x7fa40c11a010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa40000c830 tx=0x7fa400004470 comp rx=0 tx=0).stop 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 shutdown_connections 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6818/131562471,v1:172.21.15.1:6819/131562471] conn(0x7fa3f406f040 0x7fa3f4071450 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3f4069370 0x7fa3f406b820 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa40c070870 0x7fa40c11a010 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 --2- 172.21.15.1:0/149432994 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa40c071f60 0x7fa40c11a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.479+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 >> 172.21.15.1:0/149432994 conn(0x7fa40c0f8ae0 msgr2=0x7fa40c0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.480+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 shutdown_connections 2024-02-28T16:25:43.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:43.480+0000 7fa413a00700 1 -- 172.21.15.1:0/149432994 wait complete. 2024-02-28T16:25:43.689 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:43 smithi001 ceph-mon[28991]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:43.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:43 smithi066 ceph-mon[36236]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:44.326 INFO:teuthology.orchestra.run.smithi001.stdout:111669149721 2024-02-28T16:25:44.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-28T16:25:44.722 INFO:teuthology.orchestra.run.smithi001.stdout:60129542182 2024-02-28T16:25:44.723 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-28T16:25:45.054 INFO:teuthology.orchestra.run.smithi001.stdout:81604378656 2024-02-28T16:25:45.054 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-28T16:25:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:45.691 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:45 smithi001 ceph-mon[28991]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:45.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:45 smithi066 ceph-mon[36236]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:46.431 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:47.555 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:47 smithi001 ceph-mon[28991]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:47.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:47 smithi066 ceph-mon[36236]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:49.718 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:49 smithi001 ceph-mon[28991]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:49.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:49 smithi066 ceph-mon[36236]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:51.746 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:51 smithi001 ceph-mon[28991]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:51.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:51 smithi066 ceph-mon[36236]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:52.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.108+0000 7f7420a00700 1 -- 172.21.15.1:0/3035505239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 msgr2=0x7f741c0ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.108+0000 7f7420a00700 1 --2- 172.21.15.1:0/3035505239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 0x7f741c0ff160 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7408004940 tx=0x7f7408009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.109+0000 7f7420a00700 1 -- 172.21.15.1:0/3035505239 shutdown_connections 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.109+0000 7f7420a00700 1 --2- 172.21.15.1:0/3035505239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 0x7f741c0ff160 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.109+0000 7f7420a00700 1 --2- 172.21.15.1:0/3035505239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f741c105150 0x7f741c105520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.109+0000 7f7420a00700 1 -- 172.21.15.1:0/3035505239 >> 172.21.15.1:0/3035505239 conn(0x7f741c0face0 msgr2=0x7f741c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.109+0000 7f7420a00700 1 -- 172.21.15.1:0/3035505239 shutdown_connections 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.109+0000 7f7420a00700 1 -- 172.21.15.1:0/3035505239 wait complete. 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.110+0000 7f7420a00700 1 Processor -- start 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.110+0000 7f7420a00700 1 -- start start 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7420a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f741c105150 0x7f741c115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7420a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 0x7f741c1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7420a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c1167f0 con 0x7f741c105af0 2024-02-28T16:25:52.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7420a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c10fb40 con 0x7f741c105150 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7419400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 0x7f741c1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7419400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 0x7f741c1161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45166/0 (socket says 172.21.15.1:45166) 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7419400700 1 -- 172.21.15.1:0/1825509466 learned_addr learned my addr 172.21.15.1:0/1825509466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7419e00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f741c105150 0x7f741c115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7419400700 1 -- 172.21.15.1:0/1825509466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f741c105150 msgr2=0x7f741c115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7419400700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f741c105150 0x7f741c115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7419400700 1 -- 172.21.15.1:0/1825509466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408005040 con 0x7f741c105af0 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.111+0000 7f7419e00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f741c105150 0x7f741c115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.112+0000 7f7419400700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 0x7f741c1161a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7408009100 tx=0x7f7408009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.112+0000 7f7412000700 1 -- 172.21.15.1:0/1825509466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f740800e040 con 0x7f741c105af0 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.112+0000 7f7412000700 1 -- 172.21.15.1:0/1825509466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7408004290 con 0x7f741c105af0 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.112+0000 7f7412000700 1 -- 172.21.15.1:0/1825509466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f740801c3f0 con 0x7f741c105af0 2024-02-28T16:25:52.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.112+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741c10fdc0 con 0x7f741c105af0 2024-02-28T16:25:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.112+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741c110300 con 0x7f741c105af0 2024-02-28T16:25:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.114+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741c107620 con 0x7f741c105af0 2024-02-28T16:25:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.115+0000 7f7412000700 1 -- 172.21.15.1:0/1825509466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f740800bbf0 con 0x7f741c105af0 2024-02-28T16:25:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.117+0000 7f7412000700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f74000692b0 0x7f740006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.117+0000 7f7412000700 1 -- 172.21.15.1:0/1825509466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7408088060 con 0x7f741c105af0 2024-02-28T16:25:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.117+0000 7f7419e00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f74000692b0 0x7f740006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.118+0000 7f7419e00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f74000692b0 0x7f740006b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f740c009fd0 tx=0x7f740c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.123+0000 7f7412000700 1 -- 172.21.15.1:0/1825509466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7408054770 con 0x7f741c105af0 2024-02-28T16:25:52.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.345+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f741c04bed0 con 0x7f741c105af0 2024-02-28T16:25:52.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.346+0000 7f7412000700 1 -- 172.21.15.1:0/1825509466 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7408057d90 con 0x7f741c105af0 2024-02-28T16:25:52.336 INFO:teuthology.orchestra.run.smithi001.stdout:60129542184 2024-02-28T16:25:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f74000692b0 msgr2=0x7f740006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f74000692b0 0x7f740006b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f740c009fd0 tx=0x7f740c00b040 comp rx=0 tx=0).stop 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 msgr2=0x7f741c1161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 0x7f741c1161a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7408009100 tx=0x7f7408009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 shutdown_connections 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f74000692b0 0x7f740006b760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f741c105150 0x7f741c115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 --2- 172.21.15.1:0/1825509466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f741c105af0 0x7f741c1161a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.348+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 >> 172.21.15.1:0/1825509466 conn(0x7f741c0face0 msgr2=0x7f741c0fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.349+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 shutdown_connections 2024-02-28T16:25:52.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.349+0000 7f7420a00700 1 -- 172.21.15.1:0/1825509466 wait complete. 2024-02-28T16:25:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.488+0000 7fb9d3200700 1 -- 172.21.15.1:0/2497454604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 msgr2=0x7fb9cc070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.488+0000 7fb9d3200700 1 --2- 172.21.15.1:0/2497454604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 0x7fb9cc070700 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc004970 tx=0x7fb9bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.489+0000 7fb9d3200700 1 -- 172.21.15.1:0/2497454604 shutdown_connections 2024-02-28T16:25:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.489+0000 7fb9d3200700 1 --2- 172.21.15.1:0/2497454604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9cc070c40 0x7fb9cc109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.489+0000 7fb9d3200700 1 --2- 172.21.15.1:0/2497454604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 0x7fb9cc070700 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.489+0000 7fb9d3200700 1 -- 172.21.15.1:0/2497454604 >> 172.21.15.1:0/2497454604 conn(0x7fb9cc0f8ae0 msgr2=0x7fb9cc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.489+0000 7fb9d3200700 1 -- 172.21.15.1:0/2497454604 shutdown_connections 2024-02-28T16:25:52.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.489+0000 7fb9d3200700 1 -- 172.21.15.1:0/2497454604 wait complete. 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.490+0000 7fb9d3200700 1 Processor -- start 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.490+0000 7fb9d3200700 1 -- start start 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.490+0000 7fb9d3200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 0x7fb9cc115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9d3200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9cc070c40 0x7fb9cc116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9d3200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc1168e0 con 0x7fb9cc070330 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9d3200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc10fb60 con 0x7fb9cc070c40 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9cbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 0x7fb9cc115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9cbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 0x7fb9cc115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45182/0 (socket says 172.21.15.1:45182) 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9cbe00700 1 -- 172.21.15.1:0/979409651 learned_addr learned my addr 172.21.15.1:0/979409651 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9cb400700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9cc070c40 0x7fb9cc116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9cbe00700 1 -- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9cc070c40 msgr2=0x7fb9cc116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9cbe00700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9cc070c40 0x7fb9cc116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9cbe00700 1 -- 172.21.15.1:0/979409651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc005040 con 0x7fb9cc070330 2024-02-28T16:25:52.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.491+0000 7fb9cb400700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9cc070c40 0x7fb9cc116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.492+0000 7fb9cbe00700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 0x7fb9cc115cc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc004f50 tx=0x7fb9bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.492+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9bc00e040 con 0x7fb9cc070330 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.492+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9bc004290 con 0x7fb9cc070330 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.492+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc10fd80 con 0x7fb9cc070330 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.492+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9bc01c630 con 0x7fb9cc070330 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.492+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc1102c0 con 0x7fb9cc070330 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.493+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9cc110720 con 0x7fb9cc070330 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.495+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fb9bc00bc10 con 0x7fb9cc070330 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.496+0000 7fb9c3e00700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fb9b00692b0 0x7fb9b006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.496+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb9bc088cb0 con 0x7fb9cc070330 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.496+0000 7fb9c3e00700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] conn(0x7fb9b006ef80 0x7fb9b0071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.496+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 --> [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9b0071a40 con 0x7fb9b006ef80 2024-02-28T16:25:52.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.496+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9bc088ff0 con 0x7fb9cc070330 2024-02-28T16:25:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.496+0000 7fb9cb400700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fb9b00692b0 0x7fb9b006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.497+0000 7fb9d1200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] conn(0x7fb9b006ef80 0x7fb9b0071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.497+0000 7fb9cb400700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fb9b00692b0 0x7fb9b006b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8006fd0 tx=0x7fb9b8008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.498+0000 7fb9d1200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] conn(0x7fb9b006ef80 0x7fb9b0071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.500+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 <== osd.5 v2:172.21.15.66:6816/2849488501 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb9b0071a40 con 0x7fb9b006ef80 2024-02-28T16:25:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.521+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 --> [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb9cc0fc6e0 con 0x7fb9b006ef80 2024-02-28T16:25:52.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.522+0000 7fb9c3e00700 1 -- 172.21.15.1:0/979409651 <== osd.5 v2:172.21.15.66:6816/2849488501 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb9cc0fc6e0 con 0x7fb9b006ef80 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.522+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] conn(0x7fb9b006ef80 msgr2=0x7fb9b0071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.522+0000 7fb9d3200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] conn(0x7fb9b006ef80 0x7fb9b0071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fb9b00692b0 msgr2=0x7fb9b006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fb9b00692b0 0x7fb9b006b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8006fd0 tx=0x7fb9b8008040 comp rx=0 tx=0).stop 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 msgr2=0x7fb9cc115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 0x7fb9cc115cc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc004f50 tx=0x7fb9bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 shutdown_connections 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:6816/2849488501,v1:172.21.15.66:6817/2849488501] conn(0x7fb9b006ef80 0x7fb9b0071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fb9b00692b0 0x7fb9b006b760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc070330 0x7fb9cc115cc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 --2- 172.21.15.1:0/979409651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9cc070c40 0x7fb9cc116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 >> 172.21.15.1:0/979409651 conn(0x7fb9cc0f8ae0 msgr2=0x7fb9cc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 shutdown_connections 2024-02-28T16:25:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:52.523+0000 7fb9d3200700 1 -- 172.21.15.1:0/979409651 wait complete. 2024-02-28T16:25:52.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:52 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1825509466' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-28T16:25:52.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:52 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1825509466' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-28T16:25:53.588 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:53 smithi001 ceph-mon[28991]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:53.588 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:53 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2506182747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-28T16:25:53.908 INFO:teuthology.orchestra.run.smithi001.stdout:167503724555 2024-02-28T16:25:53.909 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-28T16:25:53.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.211+0000 7f91a9c00700 1 -- 172.21.15.1:0/3237764904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a4103dc0 msgr2=0x7f91a4104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:53.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.211+0000 7f91a9c00700 1 --2- 172.21.15.1:0/3237764904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a4103dc0 0x7f91a4104210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9190004940 tx=0x7f9190009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:53.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.211+0000 7f91a9c00700 1 -- 172.21.15.1:0/3237764904 shutdown_connections 2024-02-28T16:25:53.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.211+0000 7f91a9c00700 1 --2- 172.21.15.1:0/3237764904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a4103dc0 0x7f91a4104210 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:53.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.211+0000 7f91a9c00700 1 --2- 172.21.15.1:0/3237764904 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f91a41095f0 0x7f91a41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:53.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.211+0000 7f91a9c00700 1 -- 172.21.15.1:0/3237764904 >> 172.21.15.1:0/3237764904 conn(0x7f91a4072d50 msgr2=0x7f91a4075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:53.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.211+0000 7f91a9c00700 1 -- 172.21.15.1:0/3237764904 shutdown_connections 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.211+0000 7f91a9c00700 1 -- 172.21.15.1:0/3237764904 wait complete. 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.212+0000 7f91a9c00700 1 Processor -- start 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.212+0000 7f91a9c00700 1 -- start start 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.212+0000 7f91a9c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f91a4103dc0 0x7f91a411a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a41095f0 0x7f91a411a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a9c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a411abd0 con 0x7f91a41095f0 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a9c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a4113ee0 con 0x7f91a4103dc0 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a41095f0 0x7f91a411a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a41095f0 0x7f91a411a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45202/0 (socket says 172.21.15.1:45202) 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a2000700 1 -- 172.21.15.1:0/2506182747 learned_addr learned my addr 172.21.15.1:0/2506182747 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a2a00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f91a4103dc0 0x7f91a411a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a2000700 1 -- 172.21.15.1:0/2506182747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f91a4103dc0 msgr2=0x7f91a411a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a2000700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f91a4103dc0 0x7f91a411a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:53.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.213+0000 7f91a2000700 1 -- 172.21.15.1:0/2506182747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9190005040 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.214+0000 7f91a2000700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a41095f0 0x7f91a411a580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9190009fd0 tx=0x7f9190009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.214+0000 7f919aa00700 1 -- 172.21.15.1:0/2506182747 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919000e040 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.214+0000 7f919aa00700 1 -- 172.21.15.1:0/2506182747 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f919000bbf0 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.214+0000 7f919aa00700 1 -- 172.21.15.1:0/2506182747 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919001c3f0 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.214+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a4114160 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.214+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a41146a0 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.216+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a410b9c0 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.216+0000 7f919aa00700 1 -- 172.21.15.1:0/2506182747 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f919000bd60 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.217+0000 7f919aa00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f91880692b0 0x7f918806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.217+0000 7f919aa00700 1 -- 172.21.15.1:0/2506182747 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f91900880b0 con 0x7f91a41095f0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.218+0000 7f91a2a00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f91880692b0 0x7f918806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:53.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.218+0000 7f91a2a00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f91880692b0 0x7f918806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9194004990 tx=0x7f9194009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:53.951 INFO:teuthology.orchestra.run.smithi001.stdout:146028888083 2024-02-28T16:25:53.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.222+0000 7f919aa00700 1 -- 172.21.15.1:0/2506182747 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9190058250 con 0x7f91a41095f0 2024-02-28T16:25:53.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.456+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f91a4075870 con 0x7f91a41095f0 2024-02-28T16:25:53.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.456+0000 7f919aa00700 1 -- 172.21.15.1:0/2506182747 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9190057de0 con 0x7f91a41095f0 2024-02-28T16:25:53.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.459+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f91880692b0 msgr2=0x7f918806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:53.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.459+0000 7f91a9c00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f91880692b0 0x7f918806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9194004990 tx=0x7f9194009ee0 comp rx=0 tx=0).stop 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.459+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a41095f0 msgr2=0x7f91a411a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.459+0000 7f91a9c00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a41095f0 0x7f91a411a580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9190009fd0 tx=0x7f9190009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.459+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 shutdown_connections 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.459+0000 7f91a9c00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f91880692b0 0x7f918806b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.459+0000 7f91a9c00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f91a4103dc0 0x7f91a411a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.459+0000 7f91a9c00700 1 --2- 172.21.15.1:0/2506182747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a41095f0 0x7f91a411a580 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.460+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 >> 172.21.15.1:0/2506182747 conn(0x7f91a4072d50 msgr2=0x7f91a4074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.460+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 shutdown_connections 2024-02-28T16:25:53.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:53.460+0000 7f91a9c00700 1 -- 172.21.15.1:0/2506182747 wait complete. 2024-02-28T16:25:53.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:53 smithi066 ceph-mon[36236]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:54.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:53 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2506182747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-28T16:25:54.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.399+0000 7fe29aa00700 1 -- 172.21.15.1:0/2147385334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2941038f0 msgr2=0x7fe2940fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:54.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.399+0000 7fe29aa00700 1 --2- 172.21.15.1:0/2147385334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2941038f0 0x7fe2940fcf60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe280004940 tx=0x7fe280009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:54.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.399+0000 7fe29aa00700 1 -- 172.21.15.1:0/2147385334 shutdown_connections 2024-02-28T16:25:54.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.399+0000 7fe29aa00700 1 --2- 172.21.15.1:0/2147385334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2941038f0 0x7fe2940fcf60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.399+0000 7fe29aa00700 1 --2- 172.21.15.1:0/2147385334 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe294102f50 0x7fe294103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.399+0000 7fe29aa00700 1 -- 172.21.15.1:0/2147385334 >> 172.21.15.1:0/2147385334 conn(0x7fe2940f8ae0 msgr2=0x7fe2940faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.399+0000 7fe29aa00700 1 -- 172.21.15.1:0/2147385334 shutdown_connections 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.399+0000 7fe29aa00700 1 -- 172.21.15.1:0/2147385334 wait complete. 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.400+0000 7fe29aa00700 1 Processor -- start 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.400+0000 7fe29aa00700 1 -- start start 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.401+0000 7fe29aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe294102f50 0x7fe294113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.401+0000 7fe29aa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2941038f0 0x7fe294113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.401+0000 7fe29aa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe294114600 con 0x7fe294102f50 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.401+0000 7fe29aa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29410d930 con 0x7fe2941038f0 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.401+0000 7fe293e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe294102f50 0x7fe294113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.401+0000 7fe293e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe294102f50 0x7fe294113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45208/0 (socket says 172.21.15.1:45208) 2024-02-28T16:25:54.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.401+0000 7fe293e00700 1 -- 172.21.15.1:0/1278793302 learned_addr learned my addr 172.21.15.1:0/1278793302 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.402+0000 7fe293400700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2941038f0 0x7fe294113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.402+0000 7fe293e00700 1 -- 172.21.15.1:0/1278793302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2941038f0 msgr2=0x7fe294113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.402+0000 7fe293e00700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2941038f0 0x7fe294113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.402+0000 7fe293e00700 1 -- 172.21.15.1:0/1278793302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280005040 con 0x7fe294102f50 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.402+0000 7fe293e00700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe294102f50 0x7fe294113a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe28400c830 tx=0x7fe284004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.402+0000 7fe28be00700 1 -- 172.21.15.1:0/1278793302 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe28400a760 con 0x7fe294102f50 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.403+0000 7fe28be00700 1 -- 172.21.15.1:0/1278793302 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe28400ad90 con 0x7fe294102f50 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.403+0000 7fe28be00700 1 -- 172.21.15.1:0/1278793302 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2840113a0 con 0x7fe294102f50 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.403+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29410dc10 con 0x7fe294102f50 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.403+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29410e150 con 0x7fe294102f50 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.404+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe294105420 con 0x7fe294102f50 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.405+0000 7fe28be00700 1 -- 172.21.15.1:0/1278793302 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe284012400 con 0x7fe294102f50 2024-02-28T16:25:54.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.406+0000 7fe28be00700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe2780692b0 0x7fe27806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:54.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.406+0000 7fe28be00700 1 -- 172.21.15.1:0/1278793302 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe284086ef0 con 0x7fe294102f50 2024-02-28T16:25:54.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.406+0000 7fe293400700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe2780692b0 0x7fe27806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:54.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.407+0000 7fe293400700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe2780692b0 0x7fe27806b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe29410f2a0 tx=0x7fe280009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:54.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.410+0000 7fe28be00700 1 -- 172.21.15.1:0/1278793302 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe284056ff0 con 0x7fe294102f50 2024-02-28T16:25:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.631+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe29404bed0 con 0x7fe294102f50 2024-02-28T16:25:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.631+0000 7fe28be00700 1 -- 172.21.15.1:0/1278793302 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe284056b80 con 0x7fe294102f50 2024-02-28T16:25:54.622 INFO:teuthology.orchestra.run.smithi001.stdout:111669149723 2024-02-28T16:25:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe2780692b0 msgr2=0x7fe27806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe2780692b0 0x7fe27806b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe29410f2a0 tx=0x7fe280009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe294102f50 msgr2=0x7fe294113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe294102f50 0x7fe294113a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe28400c830 tx=0x7fe284004470 comp rx=0 tx=0).stop 2024-02-28T16:25:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 shutdown_connections 2024-02-28T16:25:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe2780692b0 0x7fe27806b760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe294102f50 0x7fe294113a70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 --2- 172.21.15.1:0/1278793302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2941038f0 0x7fe294113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 >> 172.21.15.1:0/1278793302 conn(0x7fe2940f8ae0 msgr2=0x7fe2940fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 shutdown_connections 2024-02-28T16:25:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:54.635+0000 7fe29aa00700 1 -- 172.21.15.1:0/1278793302 wait complete. 2024-02-28T16:25:54.825 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888080 got 146028888083 for osd.4 2024-02-28T16:25:54.825 DEBUG:teuthology.parallel:result is None 2024-02-28T16:25:55.419 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:55 smithi001 ceph-mon[28991]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:55.419 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:55 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1278793302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-28T16:25:55.491 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149721 got 111669149723 for osd.3 2024-02-28T16:25:55.491 DEBUG:teuthology.parallel:result is None 2024-02-28T16:25:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.703+0000 7f3adb400700 1 -- 172.21.15.1:0/457982948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad41027e0 msgr2=0x7f3ad40fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.703+0000 7f3adb400700 1 --2- 172.21.15.1:0/457982948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad41027e0 0x7f3ad40fbea0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0004940 tx=0x7f3ac0009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.703+0000 7f3adb400700 1 -- 172.21.15.1:0/457982948 shutdown_connections 2024-02-28T16:25:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.703+0000 7f3adb400700 1 --2- 172.21.15.1:0/457982948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad41027e0 0x7f3ad40fbea0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.703+0000 7f3adb400700 1 --2- 172.21.15.1:0/457982948 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ad4101e40 0x7f3ad4102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.703+0000 7f3adb400700 1 -- 172.21.15.1:0/457982948 >> 172.21.15.1:0/457982948 conn(0x7f3ad4071d90 msgr2=0x7f3ad4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.703+0000 7f3adb400700 1 -- 172.21.15.1:0/457982948 shutdown_connections 2024-02-28T16:25:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.703+0000 7f3adb400700 1 -- 172.21.15.1:0/457982948 wait complete. 2024-02-28T16:25:55.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.704+0000 7f3adb400700 1 Processor -- start 2024-02-28T16:25:55.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.704+0000 7f3adb400700 1 -- start start 2024-02-28T16:25:55.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3adb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad4101e40 0x7f3ad40fdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:55.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3adb400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ad41027e0 0x7f3ad40fe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:55.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3adb400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad40feb20 con 0x7f3ad4101e40 2024-02-28T16:25:55.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3adb400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad40fec90 con 0x7f3ad41027e0 2024-02-28T16:25:55.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3ad8a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad4101e40 0x7f3ad40fdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3ad8a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad4101e40 0x7f3ad40fdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45222/0 (socket says 172.21.15.1:45222) 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3ad8a00700 1 -- 172.21.15.1:0/1699170089 learned_addr learned my addr 172.21.15.1:0/1699170089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3ad3e00700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ad41027e0 0x7f3ad40fe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.705+0000 7f3ad8a00700 1 -- 172.21.15.1:0/1699170089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ad41027e0 msgr2=0x7f3ad40fe4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.706+0000 7f3ad8a00700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ad41027e0 0x7f3ad40fe4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.706+0000 7f3ad8a00700 1 -- 172.21.15.1:0/1699170089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0005040 con 0x7f3ad4101e40 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.706+0000 7f3ad8a00700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad4101e40 0x7f3ad40fdf70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800cb10 tx=0x7f3ac8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.706+0000 7f3ad0c00700 1 -- 172.21.15.1:0/1699170089 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ac8004200 con 0x7f3ad4101e40 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.706+0000 7f3ad0c00700 1 -- 172.21.15.1:0/1699170089 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ac800acc0 con 0x7f3ad4101e40 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.706+0000 7f3ad0c00700 1 -- 172.21.15.1:0/1699170089 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ac8003d90 con 0x7f3ad4101e40 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.706+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad4119ca0 con 0x7f3ad4101e40 2024-02-28T16:25:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.706+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad411a1e0 con 0x7f3ad4101e40 2024-02-28T16:25:55.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.708+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad40738b0 con 0x7f3ad4101e40 2024-02-28T16:25:55.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.709+0000 7f3ad0c00700 1 -- 172.21.15.1:0/1699170089 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3ac8003430 con 0x7f3ad4101e40 2024-02-28T16:25:55.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.709+0000 7f3ad0c00700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3abc0692a0 0x7f3abc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:55.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.709+0000 7f3ad0c00700 1 -- 172.21.15.1:0/1699170089 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3ac8087410 con 0x7f3ad4101e40 2024-02-28T16:25:55.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.710+0000 7f3ad3e00700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3abc0692a0 0x7f3abc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:55.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.710+0000 7f3ad3e00700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3abc0692a0 0x7f3abc06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000b3b0 tx=0x7f3ac0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.714+0000 7f3ad0c00700 1 -- 172.21.15.1:0/1699170089 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3ac8053a00 con 0x7f3ad4101e40 2024-02-28T16:25:55.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:55 smithi066 ceph-mon[36236]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:55.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:55 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1278793302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-28T16:25:55.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.935+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3ad40ff6b0 con 0x7f3ad4101e40 2024-02-28T16:25:55.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.935+0000 7f3ad0c00700 1 -- 172.21.15.1:0/1699170089 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ac8057020 con 0x7f3ad4101e40 2024-02-28T16:25:55.926 INFO:teuthology.orchestra.run.smithi001.stdout:38654705712 2024-02-28T16:25:55.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.938+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3abc0692a0 msgr2=0x7f3abc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:55.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.938+0000 7f3adb400700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3abc0692a0 0x7f3abc06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000b3b0 tx=0x7f3ac0009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:55.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.938+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad4101e40 msgr2=0x7f3ad40fdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:55.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.938+0000 7f3adb400700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad4101e40 0x7f3ad40fdf70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800cb10 tx=0x7f3ac8007f90 comp rx=0 tx=0).stop 2024-02-28T16:25:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.939+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 shutdown_connections 2024-02-28T16:25:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.939+0000 7f3adb400700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3abc0692a0 0x7f3abc06b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.939+0000 7f3adb400700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad4101e40 0x7f3ad40fdf70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.939+0000 7f3adb400700 1 --2- 172.21.15.1:0/1699170089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ad41027e0 0x7f3ad40fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.939+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 >> 172.21.15.1:0/1699170089 conn(0x7f3ad4071d90 msgr2=0x7f3ad4100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.939+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 shutdown_connections 2024-02-28T16:25:55.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:55.939+0000 7f3adb400700 1 -- 172.21.15.1:0/1699170089 wait complete. 2024-02-28T16:25:56.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:56 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1699170089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-28T16:25:56.759 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:56 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1699170089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-28T16:25:56.833 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542182 got 60129542184 for osd.1 2024-02-28T16:25:56.833 DEBUG:teuthology.parallel:result is None 2024-02-28T16:25:57.242 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:25:57.278 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705712 for osd.0 2024-02-28T16:25:57.279 DEBUG:teuthology.parallel:result is None 2024-02-28T16:25:57.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.322+0000 7f4eb2800700 1 -- 172.21.15.1:0/2183932609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fc6e0 msgr2=0x7f4eac109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:57.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.322+0000 7f4eb2800700 1 --2- 172.21.15.1:0/2183932609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fc6e0 0x7f4eac109440 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e98004940 tx=0x7f4e98009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:57.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.322+0000 7f4eb2800700 1 -- 172.21.15.1:0/2183932609 shutdown_connections 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.322+0000 7f4eb2800700 1 --2- 172.21.15.1:0/2183932609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fc6e0 0x7f4eac109440 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.322+0000 7f4eb2800700 1 --2- 172.21.15.1:0/2183932609 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4eac0fbdd0 0x7f4eac0fc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.322+0000 7f4eb2800700 1 -- 172.21.15.1:0/2183932609 >> 172.21.15.1:0/2183932609 conn(0x7f4eac071d90 msgr2=0x7f4eac072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.323+0000 7f4eb2800700 1 -- 172.21.15.1:0/2183932609 shutdown_connections 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.323+0000 7f4eb2800700 1 -- 172.21.15.1:0/2183932609 wait complete. 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.323+0000 7f4eb2800700 1 Processor -- start 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.324+0000 7f4eb2800700 1 -- start start 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.324+0000 7f4eb2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fbdd0 0x7f4eac115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.324+0000 7f4eb2800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4eac0fc6e0 0x7f4eac1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.324+0000 7f4eb2800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac116890 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.324+0000 7f4eb2800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac10fb30 con 0x7f4eac0fc6e0 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.324+0000 7f4eab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fbdd0 0x7f4eac115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.324+0000 7f4eab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fbdd0 0x7f4eac115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45232/0 (socket says 172.21.15.1:45232) 2024-02-28T16:25:57.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.324+0000 7f4eab400700 1 -- 172.21.15.1:0/353204977 learned_addr learned my addr 172.21.15.1:0/353204977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4eaaa00700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4eac0fc6e0 0x7f4eac1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4eab400700 1 -- 172.21.15.1:0/353204977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4eac0fc6e0 msgr2=0x7f4eac1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4eab400700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4eac0fc6e0 0x7f4eac1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4eab400700 1 -- 172.21.15.1:0/353204977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98005040 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4eab400700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fbdd0 0x7f4eac115c70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00b790 tx=0x7f4e9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4ea3400700 1 -- 172.21.15.1:0/353204977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e9c00f770 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4ea3400700 1 -- 172.21.15.1:0/353204977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e9c00fda0 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4ea3400700 1 -- 172.21.15.1:0/353204977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e9c00d310 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac10fe10 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.325+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac110320 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.326+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac106b70 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.327+0000 7f4ea3400700 1 -- 172.21.15.1:0/353204977 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f4e9c010420 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.328+0000 7f4ea3400700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f4e900692b0 0x7f4e9006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:57.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.328+0000 7f4ea3400700 1 -- 172.21.15.1:0/353204977 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4e9c086e20 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.328+0000 7f4eaaa00700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f4e900692b0 0x7f4e9006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:57.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.329+0000 7f4eaaa00700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f4e900692b0 0x7f4e9006b760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4eac110e60 tx=0x7f4e98009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:57.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.333+0000 7f4ea3400700 1 -- 172.21.15.1:0/353204977 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4e9c056f00 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.453 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:25:57.453 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:57 smithi001 ceph-mon[28991]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:57.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.568+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4eac1111e0 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.569+0000 7f4ea3400700 1 -- 172.21.15.1:0/353204977 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e9c056a90 con 0x7f4eac0fbdd0 2024-02-28T16:25:57.559 INFO:teuthology.orchestra.run.smithi001.stdout:81604378659 2024-02-28T16:25:57.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.571+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f4e900692b0 msgr2=0x7f4e9006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:57.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.571+0000 7f4eb2800700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f4e900692b0 0x7f4e9006b760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4eac110e60 tx=0x7f4e98009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.571+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fbdd0 msgr2=0x7f4eac115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.572+0000 7f4eb2800700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fbdd0 0x7f4eac115c70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00b790 tx=0x7f4e9c009250 comp rx=0 tx=0).stop 2024-02-28T16:25:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.572+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 shutdown_connections 2024-02-28T16:25:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.572+0000 7f4eb2800700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f4e900692b0 0x7f4e9006b760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.572+0000 7f4eb2800700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4eac0fbdd0 0x7f4eac115c70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.572+0000 7f4eb2800700 1 --2- 172.21.15.1:0/353204977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4eac0fc6e0 0x7f4eac1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.572+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 >> 172.21.15.1:0/353204977 conn(0x7f4eac071d90 msgr2=0x7f4eac0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.572+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 shutdown_connections 2024-02-28T16:25:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:57.572+0000 7f4eb2800700 1 -- 172.21.15.1:0/353204977 wait complete. 2024-02-28T16:25:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:25:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:57 smithi066 ceph-mon[36236]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:58.298 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378656 got 81604378659 for osd.2 2024-02-28T16:25:58.298 DEBUG:teuthology.parallel:result is None 2024-02-28T16:25:58.511 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:58 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/353204977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-28T16:25:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:58 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/353204977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-28T16:25:59.465 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:25:59 smithi001 ceph-mon[28991]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:59.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:25:59 smithi066 ceph-mon[36236]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:25:59.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.936+0000 7f9569000700 1 -- 172.21.15.1:0/1213889860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9564103dc0 msgr2=0x7f9564104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:59.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.936+0000 7f9569000700 1 --2- 172.21.15.1:0/1213889860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9564103dc0 0x7f9564104210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9550004940 tx=0x7f9550009e30 comp rx=0 tx=0).stop 2024-02-28T16:25:59.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.937+0000 7f9569000700 1 -- 172.21.15.1:0/1213889860 shutdown_connections 2024-02-28T16:25:59.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.937+0000 7f9569000700 1 --2- 172.21.15.1:0/1213889860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9564103dc0 0x7f9564104210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:59.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.937+0000 7f9569000700 1 --2- 172.21.15.1:0/1213889860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95641095f0 0x7f95641099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.937+0000 7f9569000700 1 -- 172.21.15.1:0/1213889860 >> 172.21.15.1:0/1213889860 conn(0x7f9564072d50 msgr2=0x7f9564075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.937+0000 7f9569000700 1 -- 172.21.15.1:0/1213889860 shutdown_connections 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.937+0000 7f9569000700 1 -- 172.21.15.1:0/1213889860 wait complete. 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.937+0000 7f9569000700 1 Processor -- start 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.938+0000 7f9569000700 1 -- start start 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.938+0000 7f9569000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9564103dc0 0x7f956411a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.938+0000 7f9569000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95641095f0 0x7f956411a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.938+0000 7f9569000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956411abe0 con 0x7f95641095f0 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.938+0000 7f9569000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564113ef0 con 0x7f9564103dc0 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.938+0000 7f9561400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95641095f0 0x7f956411a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.938+0000 7f9561400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95641095f0 0x7f956411a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45238/0 (socket says 172.21.15.1:45238) 2024-02-28T16:25:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.938+0000 7f9561400700 1 -- 172.21.15.1:0/4088228842 learned_addr learned my addr 172.21.15.1:0/4088228842 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f9561e00700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9564103dc0 0x7f956411a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f9561400700 1 -- 172.21.15.1:0/4088228842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9564103dc0 msgr2=0x7f956411a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:25:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f9561400700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9564103dc0 0x7f956411a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f9561400700 1 -- 172.21.15.1:0/4088228842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9550005040 con 0x7f95641095f0 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f9561e00700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9564103dc0 0x7f956411a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f9561400700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95641095f0 0x7f956411a590 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9550009150 tx=0x7f9550009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f955a000700 1 -- 172.21.15.1:0/4088228842 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f955000e040 con 0x7f95641095f0 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f955a000700 1 -- 172.21.15.1:0/4088228842 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f955000bbf0 con 0x7f95641095f0 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9564114170 con 0x7f95641095f0 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.939+0000 7f955a000700 1 -- 172.21.15.1:0/4088228842 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f955001c630 con 0x7f95641095f0 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.940+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95641146b0 con 0x7f95641095f0 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.941+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956410b9c0 con 0x7f95641095f0 2024-02-28T16:25:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.942+0000 7f955a000700 1 -- 172.21.15.1:0/4088228842 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f955000bd60 con 0x7f95641095f0 2024-02-28T16:25:59.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.942+0000 7f955a000700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f95480692b0 0x7f954806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:25:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.943+0000 7f955a000700 1 -- 172.21.15.1:0/4088228842 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9550088260 con 0x7f95641095f0 2024-02-28T16:25:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.943+0000 7f9561e00700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f95480692b0 0x7f954806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:25:59.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.943+0000 7f9561e00700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f95480692b0 0x7f954806b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9554007fd0 tx=0x7f9554009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:25:59.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:25:59.948+0000 7f955a000700 1 -- 172.21.15.1:0/4088228842 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9550058340 con 0x7f95641095f0 2024-02-28T16:26:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.167+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f956404bed0 con 0x7f95641095f0 2024-02-28T16:26:00.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.168+0000 7f955a000700 1 -- 172.21.15.1:0/4088228842 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9550057ed0 con 0x7f95641095f0 2024-02-28T16:26:00.158 INFO:teuthology.orchestra.run.smithi001.stdout:167503724557 2024-02-28T16:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f95480692b0 msgr2=0x7f954806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f95480692b0 0x7f954806b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9554007fd0 tx=0x7f9554009040 comp rx=0 tx=0).stop 2024-02-28T16:26:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95641095f0 msgr2=0x7f956411a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95641095f0 0x7f956411a590 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9550009150 tx=0x7f9550009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 shutdown_connections 2024-02-28T16:26:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f95480692b0 0x7f954806b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9564103dc0 0x7f956411a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 --2- 172.21.15.1:0/4088228842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95641095f0 0x7f956411a590 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 >> 172.21.15.1:0/4088228842 conn(0x7f9564072d50 msgr2=0x7f9564074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 shutdown_connections 2024-02-28T16:26:00.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:00.171+0000 7f9569000700 1 -- 172.21.15.1:0/4088228842 wait complete. 2024-02-28T16:26:00.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:00 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/4088228842' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-28T16:26:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:00 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/4088228842' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-28T16:26:00.985 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724555 got 167503724557 for osd.5 2024-02-28T16:26:00.986 DEBUG:teuthology.parallel:result is None 2024-02-28T16:26:00.986 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-28T16:26:00.986 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph pg dump --format=json 2024-02-28T16:26:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:01.648 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:01 smithi001 ceph-mon[28991]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:01 smithi066 ceph-mon[36236]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.503+0000 7f2b62800700 1 -- 172.21.15.1:0/2450633896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 msgr2=0x7f2b5c0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:03.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.503+0000 7f2b62800700 1 --2- 172.21.15.1:0/2450633896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 0x7f2b5c0fe3b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c004970 tx=0x7f2b4c009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.503+0000 7f2b62800700 1 -- 172.21.15.1:0/2450633896 shutdown_connections 2024-02-28T16:26:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.503+0000 7f2b62800700 1 --2- 172.21.15.1:0/2450633896 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b5c0fe8f0 0x7f2b5c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.503+0000 7f2b62800700 1 --2- 172.21.15.1:0/2450633896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 0x7f2b5c0fe3b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.503+0000 7f2b62800700 1 -- 172.21.15.1:0/2450633896 >> 172.21.15.1:0/2450633896 conn(0x7f2b5c071d90 msgr2=0x7f2b5c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.504+0000 7f2b62800700 1 -- 172.21.15.1:0/2450633896 shutdown_connections 2024-02-28T16:26:03.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.504+0000 7f2b62800700 1 -- 172.21.15.1:0/2450633896 wait complete. 2024-02-28T16:26:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.504+0000 7f2b62800700 1 Processor -- start 2024-02-28T16:26:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.505+0000 7f2b62800700 1 -- start start 2024-02-28T16:26:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.505+0000 7f2b62800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 0x7f2b5c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.505+0000 7f2b62800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b5c0fe8f0 0x7f2b5c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.505+0000 7f2b62800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c116790 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.505+0000 7f2b62800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c10fae0 con 0x7f2b5c0fe8f0 2024-02-28T16:26:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.506+0000 7f2b5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 0x7f2b5c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.506+0000 7f2b5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 0x7f2b5c115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58646/0 (socket says 172.21.15.1:58646) 2024-02-28T16:26:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.506+0000 7f2b5b400700 1 -- 172.21.15.1:0/4274942096 learned_addr learned my addr 172.21.15.1:0/4274942096 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.506+0000 7f2b5aa00700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b5c0fe8f0 0x7f2b5c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.506+0000 7f2b5b400700 1 -- 172.21.15.1:0/4274942096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b5c0fe8f0 msgr2=0x7f2b5c116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.506+0000 7f2b5b400700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b5c0fe8f0 0x7f2b5c116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.506+0000 7f2b5b400700 1 -- 172.21.15.1:0/4274942096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c005040 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.507+0000 7f2b5b400700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 0x7f2b5c115c00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c005010 tx=0x7f2b4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.507+0000 7f2b53400700 1 -- 172.21.15.1:0/4274942096 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b4c00e040 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.507+0000 7f2b53400700 1 -- 172.21.15.1:0/4274942096 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b4c004290 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.507+0000 7f2b62800700 1 -- 172.21.15.1:0/4274942096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c10fd60 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.507+0000 7f2b53400700 1 -- 172.21.15.1:0/4274942096 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b4c01c3f0 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.507+0000 7f2b62800700 1 -- 172.21.15.1:0/4274942096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c1102a0 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.508+0000 7f2b50c00700 1 -- 172.21.15.1:0/4274942096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b340052b0 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.509+0000 7f2b53400700 1 -- 172.21.15.1:0/4274942096 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2b4c013820 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.511+0000 7f2b53400700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2b400692b0 0x7f2b4006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.511+0000 7f2b53400700 1 -- 172.21.15.1:0/4274942096 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2b4c0880e0 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.511+0000 7f2b5aa00700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2b400692b0 0x7f2b4006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.512+0000 7f2b5aa00700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2b400692b0 0x7f2b4006b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c1114a0 tx=0x7f2b4800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.515+0000 7f2b53400700 1 -- 172.21.15.1:0/4274942096 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2b4c054730 con 0x7f2b5c0fdfe0 2024-02-28T16:26:03.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.723+0000 7f2b50c00700 1 -- 172.21.15.1:0/4274942096 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b34000bc0 con 0x7f2b400692b0 2024-02-28T16:26:03.716 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:03 smithi001 ceph-mon[28991]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.725+0000 7f2b53400700 1 -- 172.21.15.1:0/4274942096 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27101 (secure 0 0 0) 0x7f2b34000bc0 con 0x7f2b400692b0 2024-02-28T16:26:03.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:26:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 -- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2b400692b0 msgr2=0x7f2b4006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2b400692b0 0x7f2b4006b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c1114a0 tx=0x7f2b4800b040 comp rx=0 tx=0).stop 2024-02-28T16:26:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 -- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 msgr2=0x7f2b5c115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 0x7f2b5c115c00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c005010 tx=0x7f2b4c009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 -- 172.21.15.1:0/4274942096 shutdown_connections 2024-02-28T16:26:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2b400692b0 0x7f2b4006b760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b5c0fdfe0 0x7f2b5c115c00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 --2- 172.21.15.1:0/4274942096 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b5c0fe8f0 0x7f2b5c116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.728+0000 7f2b50c00700 1 -- 172.21.15.1:0/4274942096 >> 172.21.15.1:0/4274942096 conn(0x7f2b5c071d90 msgr2=0x7f2b5c0fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:03.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.729+0000 7f2b50c00700 1 -- 172.21.15.1:0/4274942096 shutdown_connections 2024-02-28T16:26:03.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:03.729+0000 7f2b50c00700 1 -- 172.21.15.1:0/4274942096 wait complete. 2024-02-28T16:26:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-02-28T16:26:03.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:03 smithi066 ceph-mon[36236]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:04.478 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-02-28T16:26:02.440856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-28T16:25:02.514940+0000","last_change":"2024-02-28T16:24:24.591817+0000","last_active":"2024-02-28T16:25:02.514940+0000","last_peered":"2024-02-28T16:25:02.514940+0000","last_clean":"2024-02-28T16:25:02.514940+0000","last_became_active":"2024-02-28T16:24:24.591434+0000","last_became_peered":"2024-02-28T16:24:24.591434+0000","last_unstale":"2024-02-28T16:25:02.514940+0000","last_undegraded":"2024-02-28T16:25:02.514940+0000","last_fullsized":"2024-02-28T16:25:02.514940+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-28T16:23:15.482151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-28T16:23:15.482151+0000","last_clean_scrub_stamp":"2024-02-28T16:23:15.482151+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-01T01:04:31.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888085,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.96299999999999997}]},{"osd":1,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.995},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.3009999999999999}]},{"osd":2,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":11.913,"5min":11.913,"15min":11.913},"last":1.198}]},{"osd":3,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":1.369},{"interface":"front","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":1.264}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 28 16:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.92400000000000004,"15min":0.92200000000000004},"min":{"1min":0.70299999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.081},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.96199999999999997,"15min":0.95899999999999996},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.0429999999999999}]},{"osd":1,"last update":"Wed Feb 28 16:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.95899999999999996,"15min":0.95299999999999996},"min":{"1min":0.751,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.226,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.012,"5min":1.024,"15min":1.026},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.3149999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":1.3049999999999999}]},{"osd":2,"last update":"Wed Feb 28 16:25:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0760000000000001,"15min":1.073},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1579999999999999,"15min":1.1519999999999999},"min":{"1min":0.93000000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.4119999999999999}]},{"osd":4,"last update":"Wed Feb 28 16:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.143},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.2450000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378660,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 28 16:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.84899999999999998,"15min":0.83899999999999997},"min":{"1min":0.67200000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.133},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.88900000000000001,"15min":0.86899999999999999},"min":{"1min":0.70399999999999996,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":1.351}]},{"osd":1,"last update":"Wed Feb 28 16:25:20 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.89900000000000002,"15min":0.879},"min":{"1min":0.78400000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":0.97899999999999998,"15min":0.95499999999999996},"min":{"1min":0.84299999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.97899999999999998}]},{"osd":3,"last update":"Wed Feb 28 16:25:56 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.1910000000000001,"15min":1.1819999999999999},"min":{"1min":0.82299999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.125,"15min":1.0900000000000001},"min":{"1min":0.90400000000000003,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.3039999999999998,"5min":2.3039999999999998,"15min":2.3039999999999998},"last":1.2370000000000001}]},{"osd":4,"last update":"Wed Feb 28 16:25:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.276},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.6779999999999999,"5min":2.6779999999999999,"15min":2.6779999999999999},"last":1.304}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.421}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 28 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.79400000000000004,"15min":0.746},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.0330000000000004,"5min":4.0330000000000004,"15min":4.0330000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.81799999999999995,"15min":0.79400000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.647,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.78600000000000003}]},{"osd":2,"last update":"Wed Feb 28 16:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.879,"15min":0.872},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.01,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.65300000000000002}]},{"osd":3,"last update":"Wed Feb 28 16:25:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.3460000000000001,"15min":1.3879999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.512,"5min":7.0789999999999997,"15min":7.0789999999999997},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1220000000000001,"15min":1.1259999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.52,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.996}]},{"osd":4,"last update":"Wed Feb 28 16:25:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.484,"15min":1.484},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":7.0289999999999999,"5min":7.0289999999999999,"15min":7.0289999999999999},"last":1.3799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 28 16:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.73199999999999998,"15min":0.69199999999999995},"min":{"1min":0.61399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0800000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.81499999999999995,"15min":0.77100000000000002},"min":{"1min":0.58599999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":1.141}]},{"osd":2,"last update":"Wed Feb 28 16:25:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.998,"15min":0.98399999999999999},"min":{"1min":0.80800000000000005,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.05},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.96399999999999997,"15min":0.96099999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.298,"5min":1.361,"15min":1.361},"last":1.0740000000000001}]},{"osd":3,"last update":"Wed Feb 28 16:25:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1240000000000001,"15min":1.1140000000000001},"min":{"1min":0.95999999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4019999999999999,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.331},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.135,"15min":1.135},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.468,"5min":1.552,"15min":1.552},"last":1.3}]},{"osd":4,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.1910000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-28T16:26:04.479 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph pg dump --format=json 2024-02-28T16:26:04.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:04 smithi066 ceph-mon[36236]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-28T16:26:04.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:04 smithi001 ceph-mon[28991]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-28T16:26:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:05.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:05 smithi001 ceph-mon[28991]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:05.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:05 smithi066 ceph-mon[36236]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:07.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.140+0000 7f611f400700 1 -- 172.21.15.1:0/3023557118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 msgr2=0x7f6118102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:07.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.140+0000 7f611f400700 1 --2- 172.21.15.1:0/3023557118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 0x7f6118102210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f610c004940 tx=0x7f610c009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:07.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.140+0000 7f611f400700 1 -- 172.21.15.1:0/3023557118 shutdown_connections 2024-02-28T16:26:07.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.140+0000 7f611f400700 1 --2- 172.21.15.1:0/3023557118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61181027e0 0x7f61180fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:07.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.140+0000 7f611f400700 1 --2- 172.21.15.1:0/3023557118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 0x7f6118102210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:07.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.140+0000 7f611f400700 1 -- 172.21.15.1:0/3023557118 >> 172.21.15.1:0/3023557118 conn(0x7f6118071d90 msgr2=0x7f6118072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:07.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.140+0000 7f611f400700 1 -- 172.21.15.1:0/3023557118 shutdown_connections 2024-02-28T16:26:07.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.140+0000 7f611f400700 1 -- 172.21.15.1:0/3023557118 wait complete. 2024-02-28T16:26:07.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.141+0000 7f611f400700 1 Processor -- start 2024-02-28T16:26:07.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.141+0000 7f611f400700 1 -- start start 2024-02-28T16:26:07.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 0x7f6118102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:07.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611f400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61181027e0 0x7f61181031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:07.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611f400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61181037e0 con 0x7f6118101e40 2024-02-28T16:26:07.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611f400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6118103950 con 0x7f61181027e0 2024-02-28T16:26:07.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 0x7f6118102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:07.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 0x7f6118102ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58660/0 (socket says 172.21.15.1:58660) 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611ca00700 1 -- 172.21.15.1:0/3839366112 learned_addr learned my addr 172.21.15.1:0/3839366112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611ca00700 1 -- 172.21.15.1:0/3839366112 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61181027e0 msgr2=0x7f61181031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f6117e00700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61181027e0 0x7f61181031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611ca00700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61181027e0 0x7f61181031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.142+0000 7f611ca00700 1 -- 172.21.15.1:0/3839366112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c005040 con 0x7f6118101e40 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.143+0000 7f611ca00700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 0x7f6118102ca0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f610c00b3b0 tx=0x7f610c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.143+0000 7f6114c00700 1 -- 172.21.15.1:0/3839366112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610c00e040 con 0x7f6118101e40 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.143+0000 7f6114c00700 1 -- 172.21.15.1:0/3839366112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f610c00bbf0 con 0x7f6118101e40 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.143+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6118119cc0 con 0x7f6118101e40 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.143+0000 7f6114c00700 1 -- 172.21.15.1:0/3839366112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610c01c630 con 0x7f6118101e40 2024-02-28T16:26:07.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.143+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611811a120 con 0x7f6118101e40 2024-02-28T16:26:07.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.144+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61180738b0 con 0x7f6118101e40 2024-02-28T16:26:07.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.146+0000 7f6114c00700 1 -- 172.21.15.1:0/3839366112 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f610c00bd60 con 0x7f6118101e40 2024-02-28T16:26:07.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.148+0000 7f6114c00700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f61000692a0 0x7f610006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:07.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.148+0000 7f6114c00700 1 -- 172.21.15.1:0/3839366112 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f610c088c60 con 0x7f6118101e40 2024-02-28T16:26:07.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.148+0000 7f6117e00700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f61000692a0 0x7f610006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:07.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.149+0000 7f6117e00700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f61000692a0 0x7f610006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f61181046a0 tx=0x7f6108009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:07.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.152+0000 7f6114c00700 1 -- 172.21.15.1:0/3839366112 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f610c055370 con 0x7f6118101e40 2024-02-28T16:26:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.359+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6118065dd0 con 0x7f61000692a0 2024-02-28T16:26:07.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.363+0000 7f6114c00700 1 -- 172.21.15.1:0/3839366112 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28410 (secure 0 0 0) 0x7f6118065dd0 con 0x7f61000692a0 2024-02-28T16:26:07.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f61000692a0 msgr2=0x7f610006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f61000692a0 0x7f610006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f61181046a0 tx=0x7f6108009040 comp rx=0 tx=0).stop 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 msgr2=0x7f6118102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 0x7f6118102ca0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f610c00b3b0 tx=0x7f610c009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 shutdown_connections 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f61000692a0 0x7f610006b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6118101e40 0x7f6118102ca0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 --2- 172.21.15.1:0/3839366112 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61181027e0 0x7f61181031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 >> 172.21.15.1:0/3839366112 conn(0x7f6118071d90 msgr2=0x7f61180656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:07.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.366+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 shutdown_connections 2024-02-28T16:26:07.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:07.367+0000 7f611f400700 1 -- 172.21.15.1:0/3839366112 wait complete. 2024-02-28T16:26:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-02-28T16:26:07.735 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:07 smithi001 ceph-mon[28991]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:07.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:07 smithi066 ceph-mon[36236]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:08.088 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-02-28T16:26:06.442279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003698"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-28T16:25:02.514940+0000","last_change":"2024-02-28T16:24:24.591817+0000","last_active":"2024-02-28T16:25:02.514940+0000","last_peered":"2024-02-28T16:25:02.514940+0000","last_clean":"2024-02-28T16:25:02.514940+0000","last_became_active":"2024-02-28T16:24:24.591434+0000","last_became_peered":"2024-02-28T16:24:24.591434+0000","last_unstale":"2024-02-28T16:25:02.514940+0000","last_undegraded":"2024-02-28T16:25:02.514940+0000","last_fullsized":"2024-02-28T16:25:02.514940+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-28T16:23:15.482151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-28T16:23:15.482151+0000","last_clean_scrub_stamp":"2024-02-28T16:23:15.482151+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-01T01:04:31.583736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888086,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.81000000000000005}]},{"osd":1,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.175}]},{"osd":2,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.091,"15min":1.091},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":11.913,"5min":11.913,"15min":11.913},"last":1.127}]},{"osd":3,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":1.0780000000000001}]},{"osd":5,"last update":"Wed Feb 28 16:26:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.2849999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 28 16:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.92400000000000004,"15min":0.92200000000000004},"min":{"1min":0.70299999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.96199999999999997,"15min":0.95899999999999996},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.92000000000000004}]},{"osd":1,"last update":"Wed Feb 28 16:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.95899999999999996,"15min":0.95299999999999996},"min":{"1min":0.751,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.226,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.024,"15min":1.026},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.3149999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":1.246}]},{"osd":2,"last update":"Wed Feb 28 16:25:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.0760000000000001,"15min":1.073},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1579999999999999,"15min":1.1519999999999999},"min":{"1min":0.93000000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.4239999999999999}]},{"osd":4,"last update":"Wed Feb 28 16:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.3080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]}]},{"osd":2,"up_from":19,"seq":81604378661,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 28 16:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.84899999999999998,"15min":0.83899999999999997},"min":{"1min":0.67200000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.88900000000000001,"15min":0.86899999999999999},"min":{"1min":0.70399999999999996,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":0.746}]},{"osd":1,"last update":"Wed Feb 28 16:25:20 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.89900000000000002,"15min":0.879},"min":{"1min":0.78400000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":0.97899999999999998,"15min":0.95499999999999996},"min":{"1min":0.84299999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.84899999999999998}]},{"osd":3,"last update":"Wed Feb 28 16:25:56 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.1910000000000001,"15min":1.1819999999999999},"min":{"1min":0.82299999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.125,"15min":1.0900000000000001},"min":{"1min":0.90400000000000003,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.3039999999999998,"5min":2.3039999999999998,"15min":2.3039999999999998},"last":1.1659999999999999}]},{"osd":4,"last update":"Wed Feb 28 16:25:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.014},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.6779999999999999,"5min":2.6779999999999999,"15min":2.6779999999999999},"last":1.196}]},{"osd":5,"last update":"Wed Feb 28 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":2.4489999999999998,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":1.288},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.2229999999999999,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.2529999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 28 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.79400000000000004,"15min":0.746},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.0330000000000004,"5min":4.0330000000000004,"15min":4.0330000000000004},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.81799999999999995,"15min":0.79400000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.647,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.1140000000000001}]},{"osd":2,"last update":"Wed Feb 28 16:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.879,"15min":0.872},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":1.282},{"interface":"front","average":{"1min":1.01,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":1.0649999999999999}]},{"osd":3,"last update":"Wed Feb 28 16:25:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.3460000000000001,"15min":1.3879999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.512,"5min":7.0789999999999997,"15min":7.0789999999999997},"last":1.363},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1220000000000001,"15min":1.1259999999999999},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.52,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.4890000000000001}]},{"osd":4,"last update":"Wed Feb 28 16:25:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.484,"5min":1.484,"15min":1.484},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":7.0289999999999999,"5min":7.0289999999999999,"15min":7.0289999999999999},"last":1.766}]},{"osd":5,"last update":"Wed Feb 28 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":1.8480000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 28 16:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.73199999999999998,"15min":0.69199999999999995},"min":{"1min":0.61399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0800000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.627},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.81499999999999995,"15min":0.77100000000000002},"min":{"1min":0.58599999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Feb 28 16:25:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.998,"15min":0.98399999999999999},"min":{"1min":0.80800000000000005,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.96399999999999997,"15min":0.96099999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.298,"5min":1.361,"15min":1.361},"last":0.59999999999999998}]},{"osd":3,"last update":"Wed Feb 28 16:25:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1240000000000001,"15min":1.1140000000000001},"min":{"1min":0.95999999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4019999999999999,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.123},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.135,"15min":1.135},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.468,"5min":1.552,"15min":1.552},"last":1.073}]},{"osd":4,"last update":"Wed Feb 28 16:25:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":1.02},{"interface":"front","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.91800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-28T16:26:08.089 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-28T16:26:08.089 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-28T16:26:08.089 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-28T16:26:08.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph health --format=json 2024-02-28T16:26:08.602 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:08.766 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:08 smithi001 ceph-mon[28991]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-28T16:26:08.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:08 smithi066 ceph-mon[36236]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-28T16:26:09.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:09 smithi001 ceph-mon[28991]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:09.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:26:09.900 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:09 smithi066 ceph-mon[36236]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:09.900 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:26:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.741+0000 7f46d2e00700 1 -- 172.21.15.1:0/363162300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105b10 msgr2=0x7f46cc0ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.741+0000 7f46d2e00700 1 --2- 172.21.15.1:0/363162300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105b10 0x7f46cc0ff180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f46b8004940 tx=0x7f46b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.741+0000 7f46d2e00700 1 -- 172.21.15.1:0/363162300 shutdown_connections 2024-02-28T16:26:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.741+0000 7f46d2e00700 1 --2- 172.21.15.1:0/363162300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105b10 0x7f46cc0ff180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.741+0000 7f46d2e00700 1 --2- 172.21.15.1:0/363162300 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46cc105170 0x7f46cc105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.741+0000 7f46d2e00700 1 -- 172.21.15.1:0/363162300 >> 172.21.15.1:0/363162300 conn(0x7f46cc0face0 msgr2=0x7f46cc0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.741+0000 7f46d2e00700 1 -- 172.21.15.1:0/363162300 shutdown_connections 2024-02-28T16:26:10.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.741+0000 7f46d2e00700 1 -- 172.21.15.1:0/363162300 wait complete. 2024-02-28T16:26:10.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.742+0000 7f46d2e00700 1 Processor -- start 2024-02-28T16:26:10.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.742+0000 7f46d2e00700 1 -- start start 2024-02-28T16:26:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.743+0000 7f46d2e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105170 0x7f46cc115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.743+0000 7f46d2e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46cc105b10 0x7f46cc116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.743+0000 7f46d2e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc116870 con 0x7f46cc105170 2024-02-28T16:26:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.743+0000 7f46d2e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc10fb80 con 0x7f46cc105b10 2024-02-28T16:26:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.743+0000 7f46cbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105170 0x7f46cc115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.743+0000 7f46cbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105170 0x7f46cc115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58684/0 (socket says 172.21.15.1:58684) 2024-02-28T16:26:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.743+0000 7f46cbe00700 1 -- 172.21.15.1:0/2554171249 learned_addr learned my addr 172.21.15.1:0/2554171249 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:10.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.744+0000 7f46cb400700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46cc105b10 0x7f46cc116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:10.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.744+0000 7f46cbe00700 1 -- 172.21.15.1:0/2554171249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46cc105b10 msgr2=0x7f46cc116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:10.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.744+0000 7f46cbe00700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46cc105b10 0x7f46cc116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:10.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.744+0000 7f46cbe00700 1 -- 172.21.15.1:0/2554171249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b8005040 con 0x7f46cc105170 2024-02-28T16:26:10.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.744+0000 7f46cbe00700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105170 0x7f46cc115ce0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00c830 tx=0x7f46bc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:10.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.744+0000 7f46c3e00700 1 -- 172.21.15.1:0/2554171249 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46bc00a760 con 0x7f46cc105170 2024-02-28T16:26:10.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.745+0000 7f46c3e00700 1 -- 172.21.15.1:0/2554171249 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46bc00ad90 con 0x7f46cc105170 2024-02-28T16:26:10.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.745+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc10fe60 con 0x7f46cc105170 2024-02-28T16:26:10.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.745+0000 7f46c3e00700 1 -- 172.21.15.1:0/2554171249 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46bc0113a0 con 0x7f46cc105170 2024-02-28T16:26:10.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.745+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc1103a0 con 0x7f46cc105170 2024-02-28T16:26:10.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.746+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc107640 con 0x7f46cc105170 2024-02-28T16:26:10.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.747+0000 7f46c3e00700 1 -- 172.21.15.1:0/2554171249 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f46bc012400 con 0x7f46cc105170 2024-02-28T16:26:10.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.748+0000 7f46c3e00700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46b00692a0 0x7f46b006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.748+0000 7f46c3e00700 1 -- 172.21.15.1:0/2554171249 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f46bc086ec0 con 0x7f46cc105170 2024-02-28T16:26:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.748+0000 7f46cb400700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46b00692a0 0x7f46b006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:10.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.749+0000 7f46cb400700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46b00692a0 0x7f46b006b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f46b800b3b0 tx=0x7f46b8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:10.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:10.752+0000 7f46c3e00700 1 -- 172.21.15.1:0/2554171249 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f46bc056fa0 con 0x7f46cc105170 2024-02-28T16:26:10.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:10 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:10.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:10 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:26:10.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:10 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:10 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:10 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:26:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:10 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:11.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.013+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f46cc04bf00 con 0x7f46cc105170 2024-02-28T16:26:11.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.014+0000 7f46c3e00700 1 -- 172.21.15.1:0/2554171249 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f46bc056b30 con 0x7f46cc105170 2024-02-28T16:26:11.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:26:11.004 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46b00692a0 msgr2=0x7f46b006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46b00692a0 0x7f46b006b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f46b800b3b0 tx=0x7f46b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105170 msgr2=0x7f46cc115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105170 0x7f46cc115ce0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00c830 tx=0x7f46bc004470 comp rx=0 tx=0).stop 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 shutdown_connections 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46b00692a0 0x7f46b006b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46cc105170 0x7f46cc115ce0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 --2- 172.21.15.1:0/2554171249 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46cc105b10 0x7f46cc116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:11.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 >> 172.21.15.1:0/2554171249 conn(0x7f46cc0face0 msgr2=0x7f46cc0fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 shutdown_connections 2024-02-28T16:26:11.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:11.017+0000 7f46d2e00700 1 -- 172.21.15.1:0/2554171249 wait complete. 2024-02-28T16:26:11.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:11 smithi001 ceph-mon[28991]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:11.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:11 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2554171249' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-28T16:26:11.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:26:11.892 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-28T16:26:11.895 INFO:tasks.cephadm:Setup complete, yielding 2024-02-28T16:26:11.895 INFO:teuthology.run_tasks:Running task print... 2024-02-28T16:26:11.906 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-02-28T16:26:11.906 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-28T16:26:11.927 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:26:11.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-28T16:26:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:11 smithi066 ceph-mon[36236]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:11.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:11 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2554171249' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-28T16:26:11.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:26:12.415 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:13 smithi001 ceph-mon[28991]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:13.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:13 smithi066 ceph-mon[36236]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:14.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.582+0000 7f6c8f400700 1 -- 172.21.15.1:0/643656963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c88100360 msgr2=0x7f6c881007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:14.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.582+0000 7f6c8f400700 1 --2- 172.21.15.1:0/643656963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c88100360 0x7f6c881007b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6c74004940 tx=0x7f6c74009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:14.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.582+0000 7f6c8f400700 1 -- 172.21.15.1:0/643656963 shutdown_connections 2024-02-28T16:26:14.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.582+0000 7f6c8f400700 1 --2- 172.21.15.1:0/643656963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c88100360 0x7f6c881007b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:14.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.582+0000 7f6c8f400700 1 --2- 172.21.15.1:0/643656963 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c880ff150 0x7f6c880ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:14.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.582+0000 7f6c8f400700 1 -- 172.21.15.1:0/643656963 >> 172.21.15.1:0/643656963 conn(0x7f6c880fa920 msgr2=0x7f6c880fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:14.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.582+0000 7f6c8f400700 1 -- 172.21.15.1:0/643656963 shutdown_connections 2024-02-28T16:26:14.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.582+0000 7f6c8f400700 1 -- 172.21.15.1:0/643656963 wait complete. 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.583+0000 7f6c8f400700 1 Processor -- start 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.583+0000 7f6c8f400700 1 -- start start 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.583+0000 7f6c8f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c880ff150 0x7f6c8810fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.583+0000 7f6c8f400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c88100360 0x7f6c881100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.583+0000 7f6c8f400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c881106b0 con 0x7f6c880ff150 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.583+0000 7f6c8f400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c88110f50 con 0x7f6c88100360 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.584+0000 7f6c8ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c880ff150 0x7f6c8810fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.584+0000 7f6c8ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c880ff150 0x7f6c8810fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35900/0 (socket says 172.21.15.1:35900) 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.584+0000 7f6c8ca00700 1 -- 172.21.15.1:0/1765260326 learned_addr learned my addr 172.21.15.1:0/1765260326 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.584+0000 7f6c87e00700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c88100360 0x7f6c881100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.584+0000 7f6c8ca00700 1 -- 172.21.15.1:0/1765260326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c88100360 msgr2=0x7f6c881100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.584+0000 7f6c8ca00700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c88100360 0x7f6c881100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:14.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.584+0000 7f6c8ca00700 1 -- 172.21.15.1:0/1765260326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74005040 con 0x7f6c880ff150 2024-02-28T16:26:14.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.585+0000 7f6c8ca00700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c880ff150 0x7f6c8810fb70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c8e0 tx=0x7f6c7c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:14.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.585+0000 7f6c84c00700 1 -- 172.21.15.1:0/1765260326 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c7c00a940 con 0x7f6c880ff150 2024-02-28T16:26:14.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.585+0000 7f6c84c00700 1 -- 172.21.15.1:0/1765260326 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c7c00ce80 con 0x7f6c880ff150 2024-02-28T16:26:14.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.585+0000 7f6c84c00700 1 -- 172.21.15.1:0/1765260326 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c7c0113a0 con 0x7f6c880ff150 2024-02-28T16:26:14.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.585+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c88111230 con 0x7f6c880ff150 2024-02-28T16:26:14.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.585+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c88111770 con 0x7f6c880ff150 2024-02-28T16:26:14.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.586+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8804bed0 con 0x7f6c880ff150 2024-02-28T16:26:14.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.587+0000 7f6c84c00700 1 -- 172.21.15.1:0/1765260326 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f6c7c012400 con 0x7f6c880ff150 2024-02-28T16:26:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.588+0000 7f6c84c00700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6c700692b0 0x7f6c7006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.588+0000 7f6c84c00700 1 -- 172.21.15.1:0/1765260326 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6c7c086f30 con 0x7f6c880ff150 2024-02-28T16:26:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.588+0000 7f6c87e00700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6c700692b0 0x7f6c7006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:14.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.589+0000 7f6c87e00700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6c700692b0 0x7f6c7006b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400b840 tx=0x7f6c74009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:14.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.593+0000 7f6c84c00700 1 -- 172.21.15.1:0/1765260326 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6c7c057010 con 0x7f6c880ff150 2024-02-28T16:26:14.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.805+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f6c88107c30 con 0x7f6c880ff150 2024-02-28T16:26:14.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.811+0000 7f6c84c00700 1 -- 172.21.15.1:0/1765260326 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f6c7c056ba0 con 0x7f6c880ff150 2024-02-28T16:26:14.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.814+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6c700692b0 msgr2=0x7f6c7006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:14.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.814+0000 7f6c8f400700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6c700692b0 0x7f6c7006b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400b840 tx=0x7f6c74009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:14.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.814+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c880ff150 msgr2=0x7f6c8810fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:14.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.814+0000 7f6c8f400700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c880ff150 0x7f6c8810fb70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c8e0 tx=0x7f6c7c0044c0 comp rx=0 tx=0).stop 2024-02-28T16:26:14.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.815+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 shutdown_connections 2024-02-28T16:26:14.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.815+0000 7f6c8f400700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6c700692b0 0x7f6c7006b760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:14.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.815+0000 7f6c8f400700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c880ff150 0x7f6c8810fb70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:14.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.815+0000 7f6c8f400700 1 --2- 172.21.15.1:0/1765260326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c88100360 0x7f6c881100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:14.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.815+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 >> 172.21.15.1:0/1765260326 conn(0x7f6c880fa920 msgr2=0x7f6c88103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:14.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.815+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 shutdown_connections 2024-02-28T16:26:14.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:14.815+0000 7f6c8f400700 1 -- 172.21.15.1:0/1765260326 wait complete. 2024-02-28T16:26:15.696 INFO:teuthology.run_tasks:Running task print... 2024-02-28T16:26:15.709 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-28T16:26:15.710 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-28T16:26:15.718 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:26:15.718 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph orch status' 2024-02-28T16:26:16.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:15 smithi001 ceph-mon[28991]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:16.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:15 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1765260326' entity='client.admin' 2024-02-28T16:26:16.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:26:16.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:16.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:26:16.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:15 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:16.209 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:16.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:15 smithi066 ceph-mon[36236]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:16.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:15 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1765260326' entity='client.admin' 2024-02-28T16:26:16.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:26:16.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:16.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:26:16.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:15 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:18.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:17 smithi001 ceph-mon[28991]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:18.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:17 smithi066 ceph-mon[36236]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:18.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.532+0000 7f19cfc00700 1 -- 172.21.15.1:0/4051290803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8071eb0 msgr2=0x7f19c8070320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:18.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.532+0000 7f19cfc00700 1 --2- 172.21.15.1:0/4051290803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8071eb0 0x7f19c8070320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f19bc004940 tx=0x7f19bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:18.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.532+0000 7f19cfc00700 1 -- 172.21.15.1:0/4051290803 shutdown_connections 2024-02-28T16:26:18.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.532+0000 7f19cfc00700 1 --2- 172.21.15.1:0/4051290803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c8070860 0x7f19c8070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:18.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.532+0000 7f19cfc00700 1 --2- 172.21.15.1:0/4051290803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8071eb0 0x7f19c8070320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:18.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.532+0000 7f19cfc00700 1 -- 172.21.15.1:0/4051290803 >> 172.21.15.1:0/4051290803 conn(0x7f19c80f8620 msgr2=0x7f19c80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:18.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.532+0000 7f19cfc00700 1 -- 172.21.15.1:0/4051290803 shutdown_connections 2024-02-28T16:26:18.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.532+0000 7f19cfc00700 1 -- 172.21.15.1:0/4051290803 wait complete. 2024-02-28T16:26:18.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.533+0000 7f19cfc00700 1 Processor -- start 2024-02-28T16:26:18.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.533+0000 7f19cfc00700 1 -- start start 2024-02-28T16:26:18.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.533+0000 7f19cfc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8070860 0x7f19c810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:18.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.533+0000 7f19cfc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c8071eb0 0x7f19c81100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:18.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.533+0000 7f19cfc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c81106a0 con 0x7f19c8070860 2024-02-28T16:26:18.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.533+0000 7f19cfc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c8110f40 con 0x7f19c8071eb0 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.533+0000 7f19cd200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8070860 0x7f19c810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.534+0000 7f19cd200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8070860 0x7f19c810fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35918/0 (socket says 172.21.15.1:35918) 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.534+0000 7f19cd200700 1 -- 172.21.15.1:0/991594803 learned_addr learned my addr 172.21.15.1:0/991594803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.534+0000 7f19c7e00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c8071eb0 0x7f19c81100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.534+0000 7f19cd200700 1 -- 172.21.15.1:0/991594803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c8071eb0 msgr2=0x7f19c81100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.534+0000 7f19cd200700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c8071eb0 0x7f19c81100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.534+0000 7f19cd200700 1 -- 172.21.15.1:0/991594803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc005040 con 0x7f19c8070860 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.534+0000 7f19c7e00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c8071eb0 0x7f19c81100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.534+0000 7f19cd200700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8070860 0x7f19c810fb60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009fd0 tx=0x7f19bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:18.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.535+0000 7f19c4c00700 1 -- 172.21.15.1:0/991594803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19bc00e040 con 0x7f19c8070860 2024-02-28T16:26:18.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.535+0000 7f19c4c00700 1 -- 172.21.15.1:0/991594803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19bc00bcc0 con 0x7f19c8070860 2024-02-28T16:26:18.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.535+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c81111c0 con 0x7f19c8070860 2024-02-28T16:26:18.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.535+0000 7f19c4c00700 1 -- 172.21.15.1:0/991594803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19bc01c630 con 0x7f19c8070860 2024-02-28T16:26:18.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.535+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c8111700 con 0x7f19c8070860 2024-02-28T16:26:18.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.536+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c804bed0 con 0x7f19c8070860 2024-02-28T16:26:18.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.537+0000 7f19c4c00700 1 -- 172.21.15.1:0/991594803 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f19bc013980 con 0x7f19c8070860 2024-02-28T16:26:18.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.539+0000 7f19c4c00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f19b00692b0 0x7f19b006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:18.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.539+0000 7f19c4c00700 1 -- 172.21.15.1:0/991594803 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f19bc0882e0 con 0x7f19c8070860 2024-02-28T16:26:18.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.539+0000 7f19c7e00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f19b00692b0 0x7f19b006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:18.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.540+0000 7f19c7e00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f19b00692b0 0x7f19b006b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f19b4007fd0 tx=0x7f19b4009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:18.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.543+0000 7f19c4c00700 1 -- 172.21.15.1:0/991594803 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f19bc0583c0 con 0x7f19c8070860 2024-02-28T16:26:18.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.774+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19c8104550 con 0x7f19b00692b0 2024-02-28T16:26:18.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.776+0000 7f19c4c00700 1 -- 172.21.15.1:0/991594803 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f19c8104550 con 0x7f19b00692b0 2024-02-28T16:26:18.766 INFO:teuthology.orchestra.run.smithi001.stdout:Backend: cephadm 2024-02-28T16:26:18.766 INFO:teuthology.orchestra.run.smithi001.stdout:Available: Yes 2024-02-28T16:26:18.767 INFO:teuthology.orchestra.run.smithi001.stdout:Paused: No 2024-02-28T16:26:18.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f19b00692b0 msgr2=0x7f19b006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:18.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f19b00692b0 0x7f19b006b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f19b4007fd0 tx=0x7f19b4009040 comp rx=0 tx=0).stop 2024-02-28T16:26:18.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8070860 msgr2=0x7f19c810fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:18.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8070860 0x7f19c810fb60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009fd0 tx=0x7f19bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:18.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 shutdown_connections 2024-02-28T16:26:18.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f19b00692b0 0x7f19b006b760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:18.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c8070860 0x7f19c810fb60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:18.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 --2- 172.21.15.1:0/991594803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f19c8071eb0 0x7f19c81100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:18.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 >> 172.21.15.1:0/991594803 conn(0x7f19c80f8620 msgr2=0x7f19c8102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:18.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.779+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 shutdown_connections 2024-02-28T16:26:18.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:18.780+0000 7f19cfc00700 1 -- 172.21.15.1:0/991594803 wait complete. 2024-02-28T16:26:19.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-28T16:26:20.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:19 smithi001 ceph-mon[28991]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:20.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:19 smithi001 ceph-mon[28991]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:20.160 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:20.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:19 smithi066 ceph-mon[36236]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:20.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:19 smithi066 ceph-mon[36236]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:21.887 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:21 smithi001 ceph-mon[28991]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:22.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:21 smithi066 ceph-mon[36236]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:22.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.365+0000 7f06f1c00700 1 -- 172.21.15.1:0/1312953222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec100380 msgr2=0x7f06ec1007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:22.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.365+0000 7f06f1c00700 1 --2- 172.21.15.1:0/1312953222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec100380 0x7f06ec1007d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f06d8004940 tx=0x7f06d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:22.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.366+0000 7f06f1c00700 1 -- 172.21.15.1:0/1312953222 shutdown_connections 2024-02-28T16:26:22.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.366+0000 7f06f1c00700 1 --2- 172.21.15.1:0/1312953222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec100380 0x7f06ec1007d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:22.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.366+0000 7f06f1c00700 1 --2- 172.21.15.1:0/1312953222 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f06ec0ff170 0x7f06ec0ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:22.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.366+0000 7f06f1c00700 1 -- 172.21.15.1:0/1312953222 >> 172.21.15.1:0/1312953222 conn(0x7f06ec0fa920 msgr2=0x7f06ec0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:22.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.366+0000 7f06f1c00700 1 -- 172.21.15.1:0/1312953222 shutdown_connections 2024-02-28T16:26:22.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.366+0000 7f06f1c00700 1 -- 172.21.15.1:0/1312953222 wait complete. 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.367+0000 7f06f1c00700 1 Processor -- start 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.367+0000 7f06f1c00700 1 -- start start 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.367+0000 7f06f1c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec0ff170 0x7f06ec10fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.367+0000 7f06f1c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f06ec100380 0x7f06ec1100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.367+0000 7f06f1c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1106d0 con 0x7f06ec0ff170 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.367+0000 7f06f1c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec110f70 con 0x7f06ec100380 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.367+0000 7f06eaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec0ff170 0x7f06ec10fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.368+0000 7f06eaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec0ff170 0x7f06ec10fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34156/0 (socket says 172.21.15.1:34156) 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.368+0000 7f06eaa00700 1 -- 172.21.15.1:0/3394552459 learned_addr learned my addr 172.21.15.1:0/3394552459 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.368+0000 7f06ea000700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f06ec100380 0x7f06ec1100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:22.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.368+0000 7f06eaa00700 1 -- 172.21.15.1:0/3394552459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f06ec100380 msgr2=0x7f06ec1100d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:22.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.368+0000 7f06eaa00700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f06ec100380 0x7f06ec1100d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:22.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.368+0000 7f06eaa00700 1 -- 172.21.15.1:0/3394552459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d8005040 con 0x7f06ec0ff170 2024-02-28T16:26:22.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.369+0000 7f06eaa00700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec0ff170 0x7f06ec10fb90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00cb10 tx=0x7f06dc00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:22.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.369+0000 7f06e2a00700 1 -- 172.21.15.1:0/3394552459 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06dc00a980 con 0x7f06ec0ff170 2024-02-28T16:26:22.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.369+0000 7f06e2a00700 1 -- 172.21.15.1:0/3394552459 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06dc012420 con 0x7f06ec0ff170 2024-02-28T16:26:22.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.369+0000 7f06e2a00700 1 -- 172.21.15.1:0/3394552459 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06dc0115c0 con 0x7f06ec0ff170 2024-02-28T16:26:22.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.369+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec111250 con 0x7f06ec0ff170 2024-02-28T16:26:22.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.369+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec111740 con 0x7f06ec0ff170 2024-02-28T16:26:22.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.370+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec04bed0 con 0x7f06ec0ff170 2024-02-28T16:26:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.372+0000 7f06e2a00700 1 -- 172.21.15.1:0/3394552459 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f06dc003020 con 0x7f06ec0ff170 2024-02-28T16:26:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.373+0000 7f06e2a00700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f06d00692b0 0x7f06d006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.373+0000 7f06e2a00700 1 -- 172.21.15.1:0/3394552459 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f06dc087790 con 0x7f06ec0ff170 2024-02-28T16:26:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.374+0000 7f06ea000700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f06d00692b0 0x7f06d006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.374+0000 7f06ea000700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f06d00692b0 0x7f06d006b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f06d800b020 tx=0x7f06d8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:22.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.377+0000 7f06e2a00700 1 -- 172.21.15.1:0/3394552459 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f06dc057870 con 0x7f06ec0ff170 2024-02-28T16:26:22.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.600+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f06ec111e00 con 0x7f06d00692b0 2024-02-28T16:26:22.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.624+0000 7f06e2a00700 1 -- 172.21.15.1:0/3394552459 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f06d800b5c0 con 0x7f06d00692b0 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (4m) 2m ago 6m 13.5M - 0.25.0 c8568f914cd2 3d10852c3022 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (7m) 2m ago 7m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (5m) 80s ago 5m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (4m) 2m ago 6m 74.0M - 9.4.7 954c08fa6188 2a03709b4c9c 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:9283 running (8m) 2m ago 8m 474M - 17.2.7-535-gc0f02e60 1c37ed371348 8eb0fd290742 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283 running (5m) 80s ago 5m 431M - 17.2.7-535-gc0f02e60 1c37ed371348 d7f37fd2495f 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (8m) 2m ago 8m 41.3M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (5m) 80s ago 5m 33.6M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (6m) 2m ago 6m 7763k - 1.5.0 0da6a335fe13 3dac2d3d42b2 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (5m) 80s ago 5m 7453k - 1.5.0 0da6a335fe13 31630ee26c1c 2024-02-28T16:26:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (4m) 2m ago 4m 37.3M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:26:22.616 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (3m) 2m ago 3m 37.7M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:26:22.616 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (3m) 2m ago 3m 36.3M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:26:22.616 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (2m) 80s ago 2m 38.8M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:26:22.616 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (2m) 80s ago 2m 36.4M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:26:22.616 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (87s) 80s ago 87s 10.5M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:26:22.616 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (4m) 2m ago 6m 34.7M - 2.43.0 a07b618ecd1d 21713d7e702b 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f06d00692b0 msgr2=0x7f06d006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f06d00692b0 0x7f06d006b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f06d800b020 tx=0x7f06d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec0ff170 msgr2=0x7f06ec10fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec0ff170 0x7f06ec10fb90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00cb10 tx=0x7f06dc00ced0 comp rx=0 tx=0).stop 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 shutdown_connections 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f06d00692b0 0x7f06d006b760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06ec0ff170 0x7f06ec10fb90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 --2- 172.21.15.1:0/3394552459 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f06ec100380 0x7f06ec1100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 >> 172.21.15.1:0/3394552459 conn(0x7f06ec0fa920 msgr2=0x7f06ec1035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.627+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 shutdown_connections 2024-02-28T16:26:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:22.628+0000 7f06f1c00700 1 -- 172.21.15.1:0/3394552459 wait complete. 2024-02-28T16:26:23.454 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-28T16:26:23.982 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:24.133 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:23 smithi001 ceph-mon[28991]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:24.133 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:23 smithi001 ceph-mon[28991]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:24.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:23 smithi066 ceph-mon[36236]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:24.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:23 smithi066 ceph-mon[36236]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:25.655 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:25 smithi001 ceph-mon[28991]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:25.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:25 smithi066 ceph-mon[36236]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:26.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 -- 172.21.15.1:0/3494027010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ff150 msgr2=0x7fdda80ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 --2- 172.21.15.1:0/3494027010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ff150 0x7fdda80ff560 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004940 tx=0x7fdd98009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 -- 172.21.15.1:0/3494027010 shutdown_connections 2024-02-28T16:26:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 --2- 172.21.15.1:0/3494027010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdda8100360 0x7fdda81007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 --2- 172.21.15.1:0/3494027010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ff150 0x7fdda80ff560 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 -- 172.21.15.1:0/3494027010 >> 172.21.15.1:0/3494027010 conn(0x7fdda80fa920 msgr2=0x7fdda80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 -- 172.21.15.1:0/3494027010 shutdown_connections 2024-02-28T16:26:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 -- 172.21.15.1:0/3494027010 wait complete. 2024-02-28T16:26:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.192+0000 7fddace00700 1 Processor -- start 2024-02-28T16:26:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.193+0000 7fddace00700 1 -- start start 2024-02-28T16:26:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.193+0000 7fddace00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdda80ff150 0x7fdda810fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.193+0000 7fddace00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda8100360 0x7fdda81100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.193+0000 7fddace00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda8110780 con 0x7fdda8100360 2024-02-28T16:26:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.193+0000 7fddace00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda8110fc0 con 0x7fdda80ff150 2024-02-28T16:26:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.193+0000 7fdda5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda8100360 0x7fdda81100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.193+0000 7fdda5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda8100360 0x7fdda81100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34178/0 (socket says 172.21.15.1:34178) 2024-02-28T16:26:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.193+0000 7fdda5400700 1 -- 172.21.15.1:0/1297844653 learned_addr learned my addr 172.21.15.1:0/1297844653 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.194+0000 7fdda5e00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdda80ff150 0x7fdda810fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:26.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.194+0000 7fdda5400700 1 -- 172.21.15.1:0/1297844653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdda80ff150 msgr2=0x7fdda810fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:26.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.194+0000 7fdda5400700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdda80ff150 0x7fdda810fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:26.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.194+0000 7fdda5400700 1 -- 172.21.15.1:0/1297844653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd98005040 con 0x7fdda8100360 2024-02-28T16:26:26.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.194+0000 7fdda5e00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdda80ff150 0x7fdda810fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:26:26.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.194+0000 7fdda5400700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda8100360 0x7fdda81100f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400b790 tx=0x7fdd94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:26.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.195+0000 7fdd9e000700 1 -- 172.21.15.1:0/1297844653 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd9400f770 con 0x7fdda8100360 2024-02-28T16:26:26.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.195+0000 7fdd9e000700 1 -- 172.21.15.1:0/1297844653 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd9400fda0 con 0x7fdda8100360 2024-02-28T16:26:26.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.195+0000 7fdd9e000700 1 -- 172.21.15.1:0/1297844653 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd9400d310 con 0x7fdda8100360 2024-02-28T16:26:26.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.195+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda81112a0 con 0x7fdda8100360 2024-02-28T16:26:26.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.195+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda81117e0 con 0x7fdda8100360 2024-02-28T16:26:26.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.197+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda804bed0 con 0x7fdda8100360 2024-02-28T16:26:26.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.198+0000 7fdd9e000700 1 -- 172.21.15.1:0/1297844653 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fdd94010420 con 0x7fdda8100360 2024-02-28T16:26:26.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.199+0000 7fdd9e000700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fdd8c0692b0 0x7fdd8c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:26.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.199+0000 7fdd9e000700 1 -- 172.21.15.1:0/1297844653 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdd94086e20 con 0x7fdda8100360 2024-02-28T16:26:26.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.199+0000 7fdda5e00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fdd8c0692b0 0x7fdd8c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:26.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.199+0000 7fdda5e00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fdd8c0692b0 0x7fdd8c06b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800b840 tx=0x7fdd98009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:26.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.204+0000 7fdd9e000700 1 -- 172.21.15.1:0/1297844653 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdd94056f00 con 0x7fdda8100360 2024-02-28T16:26:26.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.428+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdda8111ce0 con 0x7fdd8c0692b0 2024-02-28T16:26:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.446+0000 7fdd9e000700 1 -- 172.21.15.1:0/1297844653 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1152 (secure 0 0 0) 0x7fdda8111ce0 con 0x7fdd8c0692b0 2024-02-28T16:26:26.435 INFO:teuthology.orchestra.run.smithi001.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-28T16:26:26.435 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager ?:9093,9094 1/1 2m ago 7m count:1 2024-02-28T16:26:26.435 INFO:teuthology.orchestra.run.smithi001.stdout:crash 2/2 2m ago 7m * 2024-02-28T16:26:26.436 INFO:teuthology.orchestra.run.smithi001.stdout:grafana ?:3000 1/1 2m ago 7m count:1 2024-02-28T16:26:26.436 INFO:teuthology.orchestra.run.smithi001.stdout:mgr 2/2 2m ago 7m count:2 2024-02-28T16:26:26.436 INFO:teuthology.orchestra.run.smithi001.stdout:mon 2/2 2m ago 6m smithi001:172.21.15.1=smithi001;smithi066:172.21.15.66=smithi066;count:2 2024-02-28T16:26:26.436 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter ?:9100 2/2 2m ago 7m * 2024-02-28T16:26:26.436 INFO:teuthology.orchestra.run.smithi001.stdout:osd 6 2m ago - 2024-02-28T16:26:26.436 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus ?:9095 1/1 2m ago 7m count:1 2024-02-28T16:26:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fdd8c0692b0 msgr2=0x7fdd8c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fdd8c0692b0 0x7fdd8c06b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800b840 tx=0x7fdd98009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda8100360 msgr2=0x7fdda81100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda8100360 0x7fdda81100f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400b790 tx=0x7fdd94009250 comp rx=0 tx=0).stop 2024-02-28T16:26:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 shutdown_connections 2024-02-28T16:26:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fdd8c0692b0 0x7fdd8c06b760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdda80ff150 0x7fdda810fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 --2- 172.21.15.1:0/1297844653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda8100360 0x7fdda81100f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.449+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 >> 172.21.15.1:0/1297844653 conn(0x7fdda80fa920 msgr2=0x7fdda8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.450+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 shutdown_connections 2024-02-28T16:26:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:26.450+0000 7fddace00700 1 -- 172.21.15.1:0/1297844653 wait complete. 2024-02-28T16:26:26.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:26:26.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:26:27.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-28T16:26:27.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:27 smithi066 ceph-mon[36236]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:27.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:27 smithi066 ceph-mon[36236]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:27.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:27 smithi001 ceph-mon[28991]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:27.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:27 smithi001 ceph-mon[28991]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:27.773 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:29.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:29 smithi001 ceph-mon[28991]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:29.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.888+0000 7f29ba800700 1 -- 172.21.15.1:0/3219168759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fcef0 msgr2=0x7f29b40fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.888+0000 7f29ba800700 1 --2- 172.21.15.1:0/3219168759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fcef0 0x7f29b40fd320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009ce0 tx=0x7f29a4004d10 comp rx=0 tx=0).stop 2024-02-28T16:26:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.888+0000 7f29ba800700 1 -- 172.21.15.1:0/3219168759 shutdown_connections 2024-02-28T16:26:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.888+0000 7f29ba800700 1 --2- 172.21.15.1:0/3219168759 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b40fe0e0 0x7f29b40fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.888+0000 7f29ba800700 1 --2- 172.21.15.1:0/3219168759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fcef0 0x7f29b40fd320 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.888+0000 7f29ba800700 1 -- 172.21.15.1:0/3219168759 >> 172.21.15.1:0/3219168759 conn(0x7f29b40f8620 msgr2=0x7f29b40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.888+0000 7f29ba800700 1 -- 172.21.15.1:0/3219168759 shutdown_connections 2024-02-28T16:26:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.889+0000 7f29ba800700 1 -- 172.21.15.1:0/3219168759 wait complete. 2024-02-28T16:26:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.890+0000 7f29ba800700 1 Processor -- start 2024-02-28T16:26:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.890+0000 7f29ba800700 1 -- start start 2024-02-28T16:26:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.890+0000 7f29ba800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b40fcef0 0x7f29b4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.890+0000 7f29ba800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fe0e0 0x7f29b40702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.890+0000 7f29ba800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b40708e0 con 0x7f29b40fe0e0 2024-02-28T16:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.890+0000 7f29ba800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b4070a50 con 0x7f29b40fcef0 2024-02-28T16:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.891+0000 7f29b2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fe0e0 0x7f29b40702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.891+0000 7f29b2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fe0e0 0x7f29b40702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34198/0 (socket says 172.21.15.1:34198) 2024-02-28T16:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.891+0000 7f29b2a00700 1 -- 172.21.15.1:0/546953595 learned_addr learned my addr 172.21.15.1:0/546953595 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.891+0000 7f29b3400700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b40fcef0 0x7f29b4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.891+0000 7f29b2a00700 1 -- 172.21.15.1:0/546953595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b40fcef0 msgr2=0x7f29b4071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.891+0000 7f29b2a00700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b40fcef0 0x7f29b4071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.891+0000 7f29b2a00700 1 -- 172.21.15.1:0/546953595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a4009940 con 0x7f29b40fe0e0 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.892+0000 7f29b2a00700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fe0e0 0x7f29b40702e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f29a0002a50 tx=0x7f29a00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.892+0000 7f29ab400700 1 -- 172.21.15.1:0/546953595 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29a0007cc0 con 0x7f29b40fe0e0 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.892+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b4070d30 con 0x7f29b40fe0e0 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.892+0000 7f29ab400700 1 -- 172.21.15.1:0/546953595 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29a000f040 con 0x7f29b40fe0e0 2024-02-28T16:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.892+0000 7f29ab400700 1 -- 172.21.15.1:0/546953595 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29a0009640 con 0x7f29b40fe0e0 2024-02-28T16:26:29.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.892+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b4115b10 con 0x7f29b40fe0e0 2024-02-28T16:26:29.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.893+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b404bed0 con 0x7f29b40fe0e0 2024-02-28T16:26:29.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.895+0000 7f29ab400700 1 -- 172.21.15.1:0/546953595 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f29a00097a0 con 0x7f29b40fe0e0 2024-02-28T16:26:29.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.896+0000 7f29ab400700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29980692a0 0x7f299806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:29.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.896+0000 7f29ab400700 1 -- 172.21.15.1:0/546953595 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29a0086c10 con 0x7f29b40fe0e0 2024-02-28T16:26:29.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.897+0000 7f29b3400700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29980692a0 0x7f299806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:29.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.897+0000 7f29b3400700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29980692a0 0x7f299806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009cb0 tx=0x7f29a4004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:29.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:29.901+0000 7f29ab400700 1 -- 172.21.15.1:0/546953595 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f29a0056c70 con 0x7f29b40fe0e0 2024-02-28T16:26:29.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:29 smithi066 ceph-mon[36236]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:30.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.126+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f29b4102ac0 con 0x7f29980692a0 2024-02-28T16:26:30.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.129+0000 7f29ab400700 1 -- 172.21.15.1:0/546953595 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f29b4102ac0 con 0x7f29980692a0 2024-02-28T16:26:30.118 INFO:teuthology.orchestra.run.smithi001.stdout:HOST ADDR LABELS STATUS 2024-02-28T16:26:30.119 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 172.21.15.1 2024-02-28T16:26:30.119 INFO:teuthology.orchestra.run.smithi001.stdout:smithi066 172.21.15.66 2024-02-28T16:26:30.119 INFO:teuthology.orchestra.run.smithi001.stdout:2 hosts in cluster 2024-02-28T16:26:30.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29980692a0 msgr2=0x7f299806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:30.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29980692a0 0x7f299806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009cb0 tx=0x7f29a4004d10 comp rx=0 tx=0).stop 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fe0e0 msgr2=0x7f29b40702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fe0e0 0x7f29b40702e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f29a0002a50 tx=0x7f29a00046b0 comp rx=0 tx=0).stop 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 shutdown_connections 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29980692a0 0x7f299806b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b40fcef0 0x7f29b4071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 --2- 172.21.15.1:0/546953595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b40fe0e0 0x7f29b40702e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 >> 172.21.15.1:0/546953595 conn(0x7f29b40f8620 msgr2=0x7f29b41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 shutdown_connections 2024-02-28T16:26:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:30.132+0000 7f29ba800700 1 -- 172.21.15.1:0/546953595 wait complete. 2024-02-28T16:26:30.878 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-28T16:26:31.389 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:31.757 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:31 smithi001 ceph-mon[28991]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:31.757 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:31 smithi001 ceph-mon[28991]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:31.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:31 smithi066 ceph-mon[36236]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:31.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:31 smithi066 ceph-mon[36236]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:33.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.434+0000 7f8443200700 1 -- 172.21.15.1:0/2997727271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 msgr2=0x7f8444102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:33.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.434+0000 7f8443200700 1 --2- 172.21.15.1:0/2997727271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 0x7f8444102580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f842c004940 tx=0x7f842c009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:33.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.435+0000 7f8443200700 1 -- 172.21.15.1:0/2997727271 shutdown_connections 2024-02-28T16:26:33.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.435+0000 7f8443200700 1 --2- 172.21.15.1:0/2997727271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 0x7f8444102580 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:33.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.435+0000 7f8443200700 1 --2- 172.21.15.1:0/2997727271 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84440fd880 0x7f84440ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:33.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.435+0000 7f8443200700 1 -- 172.21.15.1:0/2997727271 >> 172.21.15.1:0/2997727271 conn(0x7f84440f72c0 msgr2=0x7f84440f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:33.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.435+0000 7f8443200700 1 -- 172.21.15.1:0/2997727271 shutdown_connections 2024-02-28T16:26:33.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.435+0000 7f8443200700 1 -- 172.21.15.1:0/2997727271 wait complete. 2024-02-28T16:26:33.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.436+0000 7f8443200700 1 Processor -- start 2024-02-28T16:26:33.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.436+0000 7f8443200700 1 -- start start 2024-02-28T16:26:33.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.436+0000 7f8443200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84440fd880 0x7f84440fd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:33.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.436+0000 7f8443200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 0x7f84440fb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f8443200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440fbd90 con 0x7f84441001a0 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f8443200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440fbf00 con 0x7f84440fd880 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f843be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 0x7f84440fb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f843be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 0x7f84440fb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34784/0 (socket says 172.21.15.1:34784) 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f843be00700 1 -- 172.21.15.1:0/407971907 learned_addr learned my addr 172.21.15.1:0/407971907 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f8441e00700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84440fd880 0x7f84440fd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f843be00700 1 -- 172.21.15.1:0/407971907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84440fd880 msgr2=0x7f84440fd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f843be00700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84440fd880 0x7f84440fd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.437+0000 7f843be00700 1 -- 172.21.15.1:0/407971907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c005040 con 0x7f84441001a0 2024-02-28T16:26:33.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.438+0000 7f843be00700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 0x7f84440fb790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f842c0093b0 tx=0x7f842c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:33.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.438+0000 7f843a000700 1 -- 172.21.15.1:0/407971907 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f842c00e040 con 0x7f84441001a0 2024-02-28T16:26:33.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.438+0000 7f843a000700 1 -- 172.21.15.1:0/407971907 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f842c00bcc0 con 0x7f84441001a0 2024-02-28T16:26:33.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.438+0000 7f843a000700 1 -- 172.21.15.1:0/407971907 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f842c01c630 con 0x7f84441001a0 2024-02-28T16:26:33.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.438+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84440fc180 con 0x7f84441001a0 2024-02-28T16:26:33.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.438+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8444115700 con 0x7f84441001a0 2024-02-28T16:26:33.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.439+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844404bed0 con 0x7f84441001a0 2024-02-28T16:26:33.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.440+0000 7f843a000700 1 -- 172.21.15.1:0/407971907 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f842c0139f0 con 0x7f84441001a0 2024-02-28T16:26:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.441+0000 7f843a000700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f84280692a0 0x7f842806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.441+0000 7f843a000700 1 -- 172.21.15.1:0/407971907 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f842c0882d0 con 0x7f84441001a0 2024-02-28T16:26:33.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.441+0000 7f8441e00700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f84280692a0 0x7f842806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:33.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.442+0000 7f8441e00700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f84280692a0 0x7f842806b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84340041d0 tx=0x7f8434009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.446+0000 7f843a000700 1 -- 172.21.15.1:0/407971907 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f842c0583b0 con 0x7f84441001a0 2024-02-28T16:26:33.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.669+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8444100e00 con 0x7f84280692a0 2024-02-28T16:26:33.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.672+0000 7f843a000700 1 -- 172.21.15.1:0/407971907 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f8444100e00 con 0x7f84280692a0 2024-02-28T16:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-28T16:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513000D480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-02-28T16:26:33.661 INFO:teuthology.orchestra.run.smithi001.stdout:smithi066 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53430091400BGN 372G No 85s ago Has a FileSystem, LVM detected 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.674+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f84280692a0 msgr2=0x7f842806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.674+0000 7f8443200700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f84280692a0 0x7f842806b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84340041d0 tx=0x7f8434009040 comp rx=0 tx=0).stop 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.674+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 msgr2=0x7f84440fb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.674+0000 7f8443200700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 0x7f84440fb790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f842c0093b0 tx=0x7f842c009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.674+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 shutdown_connections 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.674+0000 7f8443200700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f84280692a0 0x7f842806b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.675+0000 7f8443200700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84440fd880 0x7f84440fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.675+0000 7f8443200700 1 --2- 172.21.15.1:0/407971907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84441001a0 0x7f84440fb790 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.675+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 >> 172.21.15.1:0/407971907 conn(0x7f84440f72c0 msgr2=0x7f84440f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.675+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 shutdown_connections 2024-02-28T16:26:33.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:33.675+0000 7f8443200700 1 -- 172.21.15.1:0/407971907 wait complete. 2024-02-28T16:26:33.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:33 smithi001 ceph-mon[28991]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:33.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:33 smithi066 ceph-mon[36236]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:34.445 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-28T16:26:34.455 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:26:34.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-28T16:26:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:34 smithi001 ceph-mon[28991]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:34.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:34 smithi066 ceph-mon[36236]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:35.690 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:35 smithi001 ceph-mon[28991]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:35.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:35 smithi066 ceph-mon[36236]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.021+0000 7f838e800700 1 -- 172.21.15.1:0/607279192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83881003a0 msgr2=0x7f8388102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.021+0000 7f838e800700 1 --2- 172.21.15.1:0/607279192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83881003a0 0x7f8388102780 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8374004940 tx=0x7f8374009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.021+0000 7f838e800700 1 -- 172.21.15.1:0/607279192 shutdown_connections 2024-02-28T16:26:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.021+0000 7f838e800700 1 --2- 172.21.15.1:0/607279192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83881003a0 0x7f8388102780 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.021+0000 7f838e800700 1 --2- 172.21.15.1:0/607279192 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83880fda80 0x7f83880ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.021+0000 7f838e800700 1 -- 172.21.15.1:0/607279192 >> 172.21.15.1:0/607279192 conn(0x7f83880f74e0 msgr2=0x7f83880f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.021+0000 7f838e800700 1 -- 172.21.15.1:0/607279192 shutdown_connections 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.022+0000 7f838e800700 1 -- 172.21.15.1:0/607279192 wait complete. 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.022+0000 7f838e800700 1 Processor -- start 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.022+0000 7f838e800700 1 -- start start 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.023+0000 7f838e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83880fda80 0x7f838810d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.023+0000 7f838e800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83881003a0 0x7f838810de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.023+0000 7f838e800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838810e480 con 0x7f83880fda80 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.023+0000 7f838e800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838810ed20 con 0x7f83881003a0 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.023+0000 7f8387400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83880fda80 0x7f838810d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.023+0000 7f8387400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83880fda80 0x7f838810d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34804/0 (socket says 172.21.15.1:34804) 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.023+0000 7f8387400700 1 -- 172.21.15.1:0/3856661829 learned_addr learned my addr 172.21.15.1:0/3856661829 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.023+0000 7f8386a00700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83881003a0 0x7f838810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.024+0000 7f8387400700 1 -- 172.21.15.1:0/3856661829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83881003a0 msgr2=0x7f838810de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:37.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.024+0000 7f8387400700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83881003a0 0x7f838810de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.024+0000 7f8387400700 1 -- 172.21.15.1:0/3856661829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8374005040 con 0x7f83880fda80 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.024+0000 7f8386a00700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83881003a0 0x7f838810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.024+0000 7f8387400700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83880fda80 0x7f838810d940 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8378002a50 tx=0x7f83780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.025+0000 7f837f400700 1 -- 172.21.15.1:0/3856661829 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8378007cc0 con 0x7f83880fda80 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.025+0000 7f837f400700 1 -- 172.21.15.1:0/3856661829 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f837800f040 con 0x7f83880fda80 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.025+0000 7f837f400700 1 -- 172.21.15.1:0/3856661829 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8378009640 con 0x7f83880fda80 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.025+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838810f000 con 0x7f83880fda80 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.025+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838810f540 con 0x7f83880fda80 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.026+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838804bed0 con 0x7f83880fda80 2024-02-28T16:26:37.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.027+0000 7f837f400700 1 -- 172.21.15.1:0/3856661829 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f83780097a0 con 0x7f83880fda80 2024-02-28T16:26:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.028+0000 7f837f400700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f836c0692b0 0x7f836c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.028+0000 7f837f400700 1 -- 172.21.15.1:0/3856661829 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8378086c10 con 0x7f83880fda80 2024-02-28T16:26:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.028+0000 7f8386a00700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f836c0692b0 0x7f836c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:37.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.029+0000 7f8386a00700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f836c0692b0 0x7f836c06b760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f837400b7a0 tx=0x7f8374009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:37.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.033+0000 7f837f400700 1 -- 172.21.15.1:0/3856661829 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8378056c70 con 0x7f83880fda80 2024-02-28T16:26:37.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:37.274+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f838810f9b0 con 0x7f836c0692b0 2024-02-28T16:26:38.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:37 smithi066 ceph-mon[36236]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:38.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:37 smithi001 ceph-mon[28991]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.939+0000 7f837f400700 1 -- 172.21.15.1:0/3856661829 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f838810f9b0 con 0x7f836c0692b0 2024-02-28T16:26:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f836c0692b0 msgr2=0x7f836c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f836c0692b0 0x7f836c06b760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f837400b7a0 tx=0x7f8374009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83880fda80 msgr2=0x7f838810d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83880fda80 0x7f838810d940 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8378002a50 tx=0x7f83780046b0 comp rx=0 tx=0).stop 2024-02-28T16:26:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 shutdown_connections 2024-02-28T16:26:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f836c0692b0 0x7f836c06b760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83880fda80 0x7f838810d940 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 --2- 172.21.15.1:0/3856661829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83881003a0 0x7f838810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.942+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 >> 172.21.15.1:0/3856661829 conn(0x7f83880f74e0 msgr2=0x7f83880f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:38.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.943+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 shutdown_connections 2024-02-28T16:26:38.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:38.943+0000 7f838e800700 1 -- 172.21.15.1:0/3856661829 wait complete. 2024-02-28T16:26:39.151 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:38 smithi066 ceph-mon[36236]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:39.151 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-28T16:26:39.151 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-28T16:26:39.151 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:38 smithi066 ceph-mon[36236]: osdmap e41: 6 total, 6 up, 6 in 2024-02-28T16:26:39.151 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-28T16:26:39.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:38 smithi001 ceph-mon[28991]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:26:39.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-28T16:26:39.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-28T16:26:39.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:38 smithi001 ceph-mon[28991]: osdmap e41: 6 total, 6 up, 6 in 2024-02-28T16:26:39.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-28T16:26:39.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:38 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[28987]: 2024-02-28T16:26:38.900+0000 7f949c600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-28T16:26:39.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-28T16:26:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: pgmap v215: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-28T16:26:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: osdmap e42: 6 total, 6 up, 6 in 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: osdmap e43: 6 total, 6 up, 6 in 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: fsmap cephfs:0 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.awxxdo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:26:40.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.awxxdo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-28T16:26:40.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:39 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: pgmap v215: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: osdmap e42: 6 total, 6 up, 6 in 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: osdmap e43: 6 total, 6 up, 6 in 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-28T16:26:40.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: fsmap cephfs:0 2024-02-28T16:26:40.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:40.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:26:40.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:40.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:26:40.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:40.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.awxxdo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:26:40.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.awxxdo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-28T16:26:40.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:39 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:41.088 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:40 smithi001 ceph-mon[28991]: Saving service mds.cephfs spec with placement count:4 2024-02-28T16:26:41.088 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:40 smithi001 ceph-mon[28991]: Deploying daemon mds.cephfs.smithi001.awxxdo on smithi001 2024-02-28T16:26:41.088 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:40 smithi001 ceph-mon[28991]: osdmap e44: 6 total, 6 up, 6 in 2024-02-28T16:26:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:40 smithi066 ceph-mon[36236]: Saving service mds.cephfs spec with placement count:4 2024-02-28T16:26:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:40 smithi066 ceph-mon[36236]: Deploying daemon mds.cephfs.smithi001.awxxdo on smithi001 2024-02-28T16:26:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:40 smithi066 ceph-mon[36236]: osdmap e44: 6 total, 6 up, 6 in 2024-02-28T16:26:42.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:41 smithi001 ceph-mon[28991]: pgmap v219: 65 pgs: 17 creating+peering, 28 unknown, 20 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:42.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:41 smithi001 ceph-mon[28991]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-28T16:26:42.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:41 smithi001 ceph-mon[28991]: osdmap e45: 6 total, 6 up, 6 in 2024-02-28T16:26:42.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:42.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:26:42.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:41 smithi066 ceph-mon[36236]: pgmap v219: 65 pgs: 17 creating+peering, 28 unknown, 20 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:42.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:41 smithi066 ceph-mon[36236]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-28T16:26:42.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:41 smithi066 ceph-mon[36236]: osdmap e45: 6 total, 6 up, 6 in 2024-02-28T16:26:42.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:42.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:26:43.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.570+0000 7fe486c00700 1 -- 172.21.15.1:0/832866816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 msgr2=0x7fe480070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.570+0000 7fe486c00700 1 --2- 172.21.15.1:0/832866816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 0x7fe480070380 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe470004940 tx=0x7fe470009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.570+0000 7fe486c00700 1 -- 172.21.15.1:0/832866816 shutdown_connections 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.570+0000 7fe486c00700 1 --2- 172.21.15.1:0/832866816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4800708c0 0x7fe480070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.570+0000 7fe486c00700 1 --2- 172.21.15.1:0/832866816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 0x7fe480070380 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.570+0000 7fe486c00700 1 -- 172.21.15.1:0/832866816 >> 172.21.15.1:0/832866816 conn(0x7fe4800f8620 msgr2=0x7fe4800faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.570+0000 7fe486c00700 1 -- 172.21.15.1:0/832866816 shutdown_connections 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.570+0000 7fe486c00700 1 -- 172.21.15.1:0/832866816 wait complete. 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.571+0000 7fe486c00700 1 Processor -- start 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.571+0000 7fe486c00700 1 -- start start 2024-02-28T16:26:43.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.571+0000 7fe486c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4800708c0 0x7fe48010fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.571+0000 7fe486c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 0x7fe480110100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.571+0000 7fe486c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe480110700 con 0x7fe480071f10 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.571+0000 7fe486c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe480110fa0 con 0x7fe4800708c0 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.572+0000 7fe47f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 0x7fe480110100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.572+0000 7fe47f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 0x7fe480110100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52688/0 (socket says 172.21.15.1:52688) 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.572+0000 7fe47f400700 1 -- 172.21.15.1:0/1346590325 learned_addr learned my addr 172.21.15.1:0/1346590325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.572+0000 7fe47fe00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4800708c0 0x7fe48010fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.572+0000 7fe47f400700 1 -- 172.21.15.1:0/1346590325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4800708c0 msgr2=0x7fe48010fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.572+0000 7fe47f400700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4800708c0 0x7fe48010fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.572+0000 7fe47f400700 1 -- 172.21.15.1:0/1346590325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470005040 con 0x7fe480071f10 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.573+0000 7fe47f400700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 0x7fe480110100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe46c002a50 tx=0x7fe46c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.573+0000 7fe477e00700 1 -- 172.21.15.1:0/1346590325 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe46c007cc0 con 0x7fe480071f10 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.573+0000 7fe477e00700 1 -- 172.21.15.1:0/1346590325 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe46c00f040 con 0x7fe480071f10 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.573+0000 7fe477e00700 1 -- 172.21.15.1:0/1346590325 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe46c00db70 con 0x7fe480071f10 2024-02-28T16:26:43.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.573+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe480111280 con 0x7fe480071f10 2024-02-28T16:26:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.573+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4801117c0 con 0x7fe480071f10 2024-02-28T16:26:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.575+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48004bed0 con 0x7fe480071f10 2024-02-28T16:26:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.577+0000 7fe477e00700 1 -- 172.21.15.1:0/1346590325 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe46c00dd10 con 0x7fe480071f10 2024-02-28T16:26:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.578+0000 7fe477e00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe4640692b0 0x7fe46406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.578+0000 7fe477e00700 1 -- 172.21.15.1:0/1346590325 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe46c086890 con 0x7fe480071f10 2024-02-28T16:26:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.582+0000 7fe47fe00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe4640692b0 0x7fe46406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.584+0000 7fe47fe00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe4640692b0 0x7fe46406b760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe470004f90 tx=0x7fe470009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.587+0000 7fe477e00700 1 -- 172.21.15.1:0/1346590325 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe46c0565b0 con 0x7fe480071f10 2024-02-28T16:26:43.861 INFO:teuthology.orchestra.run.smithi001.stdout:e4 2024-02-28T16:26:43.861 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:26:43.861 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:26:43.861 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:26:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 4 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:26:43.843193+0000 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:26:43.862 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 1 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:in 0 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464} 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data disabled 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 0 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:creating seq 1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:26:43.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:26:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.871+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe480107c30 con 0x7fe480071f10 2024-02-28T16:26:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.871+0000 7fe477e00700 1 -- 172.21.15.1:0/1346590325 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1245 (secure 0 0 0) 0x7fe46c0174f0 con 0x7fe480071f10 2024-02-28T16:26:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe4640692b0 msgr2=0x7fe46406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe4640692b0 0x7fe46406b760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe470004f90 tx=0x7fe470009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 msgr2=0x7fe480110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 0x7fe480110100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe46c002a50 tx=0x7fe46c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 shutdown_connections 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fe4640692b0 0x7fe46406b760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe4800708c0 0x7fe48010fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 --2- 172.21.15.1:0/1346590325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe480071f10 0x7fe480110100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.875+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 >> 172.21.15.1:0/1346590325 conn(0x7fe4800f8620 msgr2=0x7fe480102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.876+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 shutdown_connections 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:43.876+0000 7fe486c00700 1 -- 172.21.15.1:0/1346590325 wait complete. 2024-02-28T16:26:43.865 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 4 2024-02-28T16:26:44.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:43 smithi001 ceph-mon[28991]: pgmap v221: 65 pgs: 17 creating+peering, 13 unknown, 35 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:43 smithi066 ceph-mon[36236]: pgmap v221: 65 pgs: 17 creating+peering, 13 unknown, 35 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:26:44.747 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-28T16:26:44.757 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:26:44.758 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] up:boot 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: daemon mds.cephfs.smithi001.awxxdo assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: Cluster is now healthy 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: fsmap cephfs:0 1 up:standby 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: fsmap cephfs:1 {0=cephfs.smithi001.awxxdo=up:creating} 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1346590325' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: daemon mds.cephfs.smithi001.awxxdo is now active in filesystem cephfs as rank 0 2024-02-28T16:26:44.989 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:44.990 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.wfbdor", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:26:44.990 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.wfbdor", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-28T16:26:44.990 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:44 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:45.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] up:boot 2024-02-28T16:26:45.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: daemon mds.cephfs.smithi001.awxxdo assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: Cluster is now healthy 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: fsmap cephfs:0 1 up:standby 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: fsmap cephfs:1 {0=cephfs.smithi001.awxxdo=up:creating} 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1346590325' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: daemon mds.cephfs.smithi001.awxxdo is now active in filesystem cephfs as rank 0 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.wfbdor", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.wfbdor", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-28T16:26:45.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:44 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:45.272 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:45.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:45 smithi066 ceph-mon[36236]: Deploying daemon mds.cephfs.smithi066.wfbdor on smithi066 2024-02-28T16:26:45.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:45 smithi066 ceph-mon[36236]: pgmap v222: 65 pgs: 6 creating+peering, 59 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 920 B/s wr, 3 op/s 2024-02-28T16:26:45.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:45 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] up:active 2024-02-28T16:26:45.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:45 smithi066 ceph-mon[36236]: fsmap cephfs:1 {0=cephfs.smithi001.awxxdo=up:active} 2024-02-28T16:26:46.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:45 smithi001 ceph-mon[28991]: Deploying daemon mds.cephfs.smithi066.wfbdor on smithi066 2024-02-28T16:26:46.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:45 smithi001 ceph-mon[28991]: pgmap v222: 65 pgs: 6 creating+peering, 59 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 920 B/s wr, 3 op/s 2024-02-28T16:26:46.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:45 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] up:active 2024-02-28T16:26:46.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:45 smithi001 ceph-mon[28991]: fsmap cephfs:1 {0=cephfs.smithi001.awxxdo=up:active} 2024-02-28T16:26:47.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.561+0000 7fd72d800700 1 -- 172.21.15.1:0/306675040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fcef0 msgr2=0x7fd7280fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:47.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.561+0000 7fd72d800700 1 --2- 172.21.15.1:0/306675040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fcef0 0x7fd7280fd320 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd718004940 tx=0x7fd718009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:47.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.561+0000 7fd72d800700 1 -- 172.21.15.1:0/306675040 shutdown_connections 2024-02-28T16:26:47.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.561+0000 7fd72d800700 1 --2- 172.21.15.1:0/306675040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd7280fe0e0 0x7fd7280fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:47.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.561+0000 7fd72d800700 1 --2- 172.21.15.1:0/306675040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fcef0 0x7fd7280fd320 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:47.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.561+0000 7fd72d800700 1 -- 172.21.15.1:0/306675040 >> 172.21.15.1:0/306675040 conn(0x7fd7280f8620 msgr2=0x7fd7280faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:47.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.561+0000 7fd72d800700 1 -- 172.21.15.1:0/306675040 shutdown_connections 2024-02-28T16:26:47.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.561+0000 7fd72d800700 1 -- 172.21.15.1:0/306675040 wait complete. 2024-02-28T16:26:47.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.562+0000 7fd72d800700 1 Processor -- start 2024-02-28T16:26:47.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.562+0000 7fd72d800700 1 -- start start 2024-02-28T16:26:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.563+0000 7fd72d800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd7280fcef0 0x7fd72810d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.563+0000 7fd72d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fe0e0 0x7fd72810de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.563+0000 7fd72d800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72810e420 con 0x7fd7280fe0e0 2024-02-28T16:26:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.563+0000 7fd72d800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72810ecc0 con 0x7fd7280fcef0 2024-02-28T16:26:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.563+0000 7fd726000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fe0e0 0x7fd72810de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.563+0000 7fd726000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fe0e0 0x7fd72810de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52708/0 (socket says 172.21.15.1:52708) 2024-02-28T16:26:47.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.563+0000 7fd726000700 1 -- 172.21.15.1:0/3929680742 learned_addr learned my addr 172.21.15.1:0/3929680742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.563+0000 7fd726a00700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd7280fcef0 0x7fd72810d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd726000700 1 -- 172.21.15.1:0/3929680742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd7280fcef0 msgr2=0x7fd72810d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd726000700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd7280fcef0 0x7fd72810d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd726000700 1 -- 172.21.15.1:0/3929680742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd718005040 con 0x7fd7280fe0e0 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd726000700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fe0e0 0x7fd72810de20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd71400eb40 tx=0x7fd71400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd71ea00700 1 -- 172.21.15.1:0/3929680742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd714007cc0 con 0x7fd7280fe0e0 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd71ea00700 1 -- 172.21.15.1:0/3929680742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd71400f040 con 0x7fd7280fe0e0 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72810ef40 con 0x7fd7280fe0e0 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd71ea00700 1 -- 172.21.15.1:0/3929680742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd71400ca10 con 0x7fd7280fe0e0 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.564+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72810f480 con 0x7fd7280fe0e0 2024-02-28T16:26:47.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.566+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72804bed0 con 0x7fd7280fe0e0 2024-02-28T16:26:47.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.567+0000 7fd71ea00700 1 -- 172.21.15.1:0/3929680742 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd71400cb70 con 0x7fd7280fe0e0 2024-02-28T16:26:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.568+0000 7fd71ea00700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd70c0692a0 0x7fd70c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.568+0000 7fd71ea00700 1 -- 172.21.15.1:0/3929680742 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd714086ca0 con 0x7fd7280fe0e0 2024-02-28T16:26:47.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.568+0000 7fd726a00700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd70c0692a0 0x7fd70c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:47.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.569+0000 7fd726a00700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd70c0692a0 0x7fd70c06b750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd718000c00 tx=0x7fd718009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:47.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.573+0000 7fd71ea00700 1 -- 172.21.15.1:0/3929680742 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd71404dbd0 con 0x7fd7280fe0e0 2024-02-28T16:26:47.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.836+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fd72810f810 con 0x7fd7280fe0e0 2024-02-28T16:26:47.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.879+0000 7fd71ea00700 1 -- 172.21.15.1:0/3929680742 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd71408b070 con 0x7fd7280fe0e0 2024-02-28T16:26:47.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.882+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd70c0692a0 msgr2=0x7fd70c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:47.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.882+0000 7fd72d800700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd70c0692a0 0x7fd70c06b750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd718000c00 tx=0x7fd718009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:47.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.882+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fe0e0 msgr2=0x7fd72810de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:47.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.882+0000 7fd72d800700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fe0e0 0x7fd72810de20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd71400eb40 tx=0x7fd71400b040 comp rx=0 tx=0).stop 2024-02-28T16:26:47.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.882+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 shutdown_connections 2024-02-28T16:26:47.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.882+0000 7fd72d800700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd70c0692a0 0x7fd70c06b750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:47.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.882+0000 7fd72d800700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd7280fcef0 0x7fd72810d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:47.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.882+0000 7fd72d800700 1 --2- 172.21.15.1:0/3929680742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7280fe0e0 0x7fd72810de20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:47.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.883+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 >> 172.21.15.1:0/3929680742 conn(0x7fd7280f8620 msgr2=0x7fd7281013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:47.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.883+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 shutdown_connections 2024-02-28T16:26:47.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:47.884+0000 7fd72d800700 1 -- 172.21.15.1:0/3929680742 wait complete. 2024-02-28T16:26:48.195 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:47 smithi066 ceph-mon[36236]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-02-28T16:26:48.195 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:48.195 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:48.195 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:48.195 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:47 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3929680742' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-28T16:26:48.195 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.rtdwhc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:26:48.195 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.rtdwhc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-28T16:26:48.196 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:47 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:48.227 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:47 smithi001 ceph-mon[28991]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-02-28T16:26:48.227 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:48.227 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:48.227 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:48.227 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:47 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3929680742' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-28T16:26:48.227 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.rtdwhc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:26:48.227 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.rtdwhc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-28T16:26:48.228 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:47 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:48.704 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-28T16:26:48.717 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:26:48.718 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-02-28T16:26:49.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: Deploying daemon mds.cephfs.smithi001.rtdwhc on smithi001 2024-02-28T16:26:49.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-28T16:26:49.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] up:boot 2024-02-28T16:26:49.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3929680742' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-28T16:26:49.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: daemon mds.cephfs.smithi066.wfbdor assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-28T16:26:49.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-28T16:26:49.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-28T16:26:49.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: fsmap cephfs:1 {0=cephfs.smithi001.awxxdo=up:active} 1 up:standby 2024-02-28T16:26:49.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:26:49.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:creating} 2024-02-28T16:26:49.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:48 smithi001 ceph-mon[28991]: daemon mds.cephfs.smithi066.wfbdor is now active in filesystem cephfs as rank 1 2024-02-28T16:26:49.145 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: Deploying daemon mds.cephfs.smithi001.rtdwhc on smithi001 2024-02-28T16:26:49.145 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-28T16:26:49.145 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] up:boot 2024-02-28T16:26:49.145 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3929680742' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-28T16:26:49.145 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: daemon mds.cephfs.smithi066.wfbdor assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-28T16:26:49.146 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-28T16:26:49.146 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-28T16:26:49.146 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: fsmap cephfs:1 {0=cephfs.smithi001.awxxdo=up:active} 1 up:standby 2024-02-28T16:26:49.146 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:26:49.146 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:creating} 2024-02-28T16:26:49.146 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:48 smithi066 ceph-mon[36236]: daemon mds.cephfs.smithi066.wfbdor is now active in filesystem cephfs as rank 1 2024-02-28T16:26:50.169 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:49 smithi001 ceph-mon[28991]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-02-28T16:26:50.170 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:49 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] up:active 2024-02-28T16:26:50.170 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:49 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2024-02-28T16:26:50.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:49 smithi066 ceph-mon[36236]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-02-28T16:26:50.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:49 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] up:active 2024-02-28T16:26:50.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:49 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2024-02-28T16:26:50.266 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:51.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:50 smithi066 ceph-mon[36236]: pgmap v225: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s wr, 7 op/s 2024-02-28T16:26:51.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:50 smithi001 ceph-mon[28991]: pgmap v225: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s wr, 7 op/s 2024-02-28T16:26:52.658 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:52 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:52.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.935+0000 7ff404600700 1 -- 172.21.15.1:0/213343994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 msgr2=0x7ff3fc0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.935+0000 7ff404600700 1 --2- 172.21.15.1:0/213343994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 0x7ff3fc0fe550 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8004940 tx=0x7ff3e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.935+0000 7ff404600700 1 -- 172.21.15.1:0/213343994 shutdown_connections 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.935+0000 7ff404600700 1 --2- 172.21.15.1:0/213343994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 0x7ff3fc0fe550 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.935+0000 7ff404600700 1 --2- 172.21.15.1:0/213343994 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3fc0fcef0 0x7ff3fc0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.935+0000 7ff404600700 1 -- 172.21.15.1:0/213343994 >> 172.21.15.1:0/213343994 conn(0x7ff3fc0f8620 msgr2=0x7ff3fc0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.935+0000 7ff404600700 1 -- 172.21.15.1:0/213343994 shutdown_connections 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.935+0000 7ff404600700 1 -- 172.21.15.1:0/213343994 wait complete. 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.936+0000 7ff404600700 1 Processor -- start 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff404600700 1 -- start start 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff404600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3fc0fcef0 0x7ff3fc10fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff404600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 0x7ff3fc1100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:52.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff404600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc1106a0 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff404600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc110f40 con 0x7ff3fc0fcef0 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff401200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 0x7ff3fc1100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff401200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 0x7ff3fc1100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43390/0 (socket says 172.21.15.1:43390) 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff401200700 1 -- 172.21.15.1:0/3369702319 learned_addr learned my addr 172.21.15.1:0/3369702319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.937+0000 7ff401c00700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3fc0fcef0 0x7ff3fc10fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.938+0000 7ff401200700 1 -- 172.21.15.1:0/3369702319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3fc0fcef0 msgr2=0x7ff3fc10fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.938+0000 7ff401200700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3fc0fcef0 0x7ff3fc10fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.938+0000 7ff401200700 1 -- 172.21.15.1:0/3369702319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e8005040 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.938+0000 7ff401200700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 0x7ff3fc1100a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800b3b0 tx=0x7ff3e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.939+0000 7ff3f9600700 1 -- 172.21.15.1:0/3369702319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3e800e040 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.939+0000 7ff3f9600700 1 -- 172.21.15.1:0/3369702319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3e800bd90 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.939+0000 7ff3f9600700 1 -- 172.21.15.1:0/3369702319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3e801c630 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.939+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc1111c0 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.939+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc111700 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.940+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3fc102210 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.941+0000 7ff3f9600700 1 -- 172.21.15.1:0/3369702319 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff3e8013980 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.942+0000 7ff3f9600700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff3e40692b0 0x7ff3e406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:52.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.942+0000 7ff3f9600700 1 -- 172.21.15.1:0/3369702319 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff3e80885d0 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.943+0000 7ff401c00700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff3e40692b0 0x7ff3e406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:52.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.943+0000 7ff401c00700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff3e40692b0 0x7ff3e406b760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0fcc30 tx=0x7ff3f000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:52.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:52.947+0000 7ff3f9600700 1 -- 172.21.15.1:0/3369702319 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff3e8058370 con 0x7ff3fc0fe0e0 2024-02-28T16:26:52.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:52 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:53.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.207+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff3fc04bed0 con 0x7ff3fc0fe0e0 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: pgmap v226: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 6 op/s 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: Cluster is now healthy 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] up:boot 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 1 up:standby 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:53.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:53.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.pivimo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:26:53.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.pivimo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-28T16:26:53.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:53.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: Deploying daemon mds.cephfs.smithi066.pivimo on smithi066 2024-02-28T16:26:53.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:53 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3369702319' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-02-28T16:26:53.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.796+0000 7ff3f9600700 1 -- 172.21.15.1:0/3369702319 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff3e8057f00 con 0x7ff3fc0fe0e0 2024-02-28T16:26:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff3e40692b0 msgr2=0x7ff3e406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff3e40692b0 0x7ff3e406b760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0fcc30 tx=0x7ff3f000b040 comp rx=0 tx=0).stop 2024-02-28T16:26:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 msgr2=0x7ff3fc1100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 0x7ff3fc1100a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800b3b0 tx=0x7ff3e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 shutdown_connections 2024-02-28T16:26:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7ff3e40692b0 0x7ff3e406b760 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3fc0fcef0 0x7ff3fc10fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:53.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 --2- 172.21.15.1:0/3369702319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3fc0fe0e0 0x7ff3fc1100a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:53.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 >> 172.21.15.1:0/3369702319 conn(0x7ff3fc0f8620 msgr2=0x7ff3fc0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:53.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 shutdown_connections 2024-02-28T16:26:53.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:53.800+0000 7ff404600700 1 -- 172.21.15.1:0/3369702319 wait complete. 2024-02-28T16:26:53.935 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: pgmap v226: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 6 op/s 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: Cluster is now healthy 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] up:boot 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 1 up:standby 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.pivimo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.pivimo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: Deploying daemon mds.cephfs.smithi066.pivimo on smithi066 2024-02-28T16:26:53.936 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:53 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3369702319' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-02-28T16:26:54.637 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-28T16:26:54.648 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:26:54.649 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-28T16:26:54.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:54 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3369702319' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-02-28T16:26:54.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:54 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 1 up:standby 2024-02-28T16:26:55.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:54 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3369702319' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-02-28T16:26:55.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:54 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 1 up:standby 2024-02-28T16:26:55.151 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:26:55.934 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:55 smithi066 ceph-mon[36236]: pgmap v227: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 7 op/s 2024-02-28T16:26:56.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:55 smithi001 ceph-mon[28991]: pgmap v227: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 7 op/s 2024-02-28T16:26:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.454+0000 7fd3e0400700 1 -- 172.21.15.1:0/1976527175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d80708c0 msgr2=0x7fd3d8070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.454+0000 7fd3e0400700 1 --2- 172.21.15.1:0/1976527175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d80708c0 0x7fd3d8070d10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4004940 tx=0x7fd3c4009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.454+0000 7fd3e0400700 1 -- 172.21.15.1:0/1976527175 shutdown_connections 2024-02-28T16:26:57.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.454+0000 7fd3e0400700 1 --2- 172.21.15.1:0/1976527175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d80708c0 0x7fd3d8070d10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:57.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.454+0000 7fd3e0400700 1 --2- 172.21.15.1:0/1976527175 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d8071f10 0x7fd3d8070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:57.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.454+0000 7fd3e0400700 1 -- 172.21.15.1:0/1976527175 >> 172.21.15.1:0/1976527175 conn(0x7fd3d80f8620 msgr2=0x7fd3d80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:57.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.454+0000 7fd3e0400700 1 -- 172.21.15.1:0/1976527175 shutdown_connections 2024-02-28T16:26:57.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.454+0000 7fd3e0400700 1 -- 172.21.15.1:0/1976527175 wait complete. 2024-02-28T16:26:57.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.455+0000 7fd3e0400700 1 Processor -- start 2024-02-28T16:26:57.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.455+0000 7fd3e0400700 1 -- start start 2024-02-28T16:26:57.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3e0400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d80708c0 0x7fd3d810fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3e0400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8071f10 0x7fd3d8110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3e0400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d8110650 con 0x7fd3d8071f10 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3e0400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d8110ef0 con 0x7fd3d80708c0 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3dd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8071f10 0x7fd3d8110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3dda00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d80708c0 0x7fd3d810fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3dd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8071f10 0x7fd3d8110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43418/0 (socket says 172.21.15.1:43418) 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3dd000700 1 -- 172.21.15.1:0/2935946808 learned_addr learned my addr 172.21.15.1:0/2935946808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3dda00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d80708c0 0x7fd3d810fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:48712/0 (socket says 172.21.15.1:48712) 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3dd000700 1 -- 172.21.15.1:0/2935946808 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d80708c0 msgr2=0x7fd3d810fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3dd000700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d80708c0 0x7fd3d810fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.456+0000 7fd3dd000700 1 -- 172.21.15.1:0/2935946808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c4005040 con 0x7fd3d8071f10 2024-02-28T16:26:57.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.457+0000 7fd3dd000700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8071f10 0x7fd3d8110050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4000c00 tx=0x7fd3c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:57.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.457+0000 7fd3d5600700 1 -- 172.21.15.1:0/2935946808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3c4021070 con 0x7fd3d8071f10 2024-02-28T16:26:57.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.457+0000 7fd3d5600700 1 -- 172.21.15.1:0/2935946808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3c401c400 con 0x7fd3d8071f10 2024-02-28T16:26:57.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.457+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d8111170 con 0x7fd3d8071f10 2024-02-28T16:26:57.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.457+0000 7fd3d5600700 1 -- 172.21.15.1:0/2935946808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3c400e070 con 0x7fd3d8071f10 2024-02-28T16:26:57.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.457+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d8111650 con 0x7fd3d8071f10 2024-02-28T16:26:57.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.459+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d804bed0 con 0x7fd3d8071f10 2024-02-28T16:26:57.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.459+0000 7fd3d5600700 1 -- 172.21.15.1:0/2935946808 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd3c401ca50 con 0x7fd3d8071f10 2024-02-28T16:26:57.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.460+0000 7fd3d5600700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd3c00692b0 0x7fd3c006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:26:57.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.461+0000 7fd3d5600700 1 -- 172.21.15.1:0/2935946808 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd3c40882d0 con 0x7fd3d8071f10 2024-02-28T16:26:57.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.461+0000 7fd3dda00700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd3c00692b0 0x7fd3c006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:26:57.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.461+0000 7fd3dda00700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd3c00692b0 0x7fd3c006b760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009b80 tx=0x7fd3cc00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:26:57.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.465+0000 7fd3d5600700 1 -- 172.21.15.1:0/2935946808 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd3c4058070 con 0x7fd3d8071f10 2024-02-28T16:26:57.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:26:57.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: pgmap v228: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-02-28T16:26:57.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:57 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:26:57.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:57.731+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd3d8107c10 con 0x7fd3d8071f10 2024-02-28T16:26:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:26:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: pgmap v228: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-02-28T16:26:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:26:57.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:57 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:26:58.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.455+0000 7fd3d5600700 1 -- 172.21.15.1:0/2935946808 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v12) v1 ==== 168+0+0 (secure 0 0 0) 0x7fd3c4057c00 con 0x7fd3d8071f10 2024-02-28T16:26:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd3c00692b0 msgr2=0x7fd3c006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd3c00692b0 0x7fd3c006b760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009b80 tx=0x7fd3cc00b040 comp rx=0 tx=0).stop 2024-02-28T16:26:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8071f10 msgr2=0x7fd3d8110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:26:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8071f10 0x7fd3d8110050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4000c00 tx=0x7fd3c4009e30 comp rx=0 tx=0).stop 2024-02-28T16:26:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 shutdown_connections 2024-02-28T16:26:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fd3c00692b0 0x7fd3c006b760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d80708c0 0x7fd3d810fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 --2- 172.21.15.1:0/2935946808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8071f10 0x7fd3d8110050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:26:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 >> 172.21.15.1:0/2935946808 conn(0x7fd3d80f8620 msgr2=0x7fd3d8102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:26:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 shutdown_connections 2024-02-28T16:26:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:26:58.459+0000 7fd3e0400700 1 -- 172.21.15.1:0/2935946808 wait complete. 2024-02-28T16:26:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:inline data enabled 2024-02-28T16:26:58.698 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:58 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] up:boot 2024-02-28T16:26:58.698 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:58 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:26:58.698 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:58 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T16:26:58.699 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:58 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2935946808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-28T16:26:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:58 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] up:boot 2024-02-28T16:26:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:58 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:26:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:58 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T16:26:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:58 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2935946808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-28T16:26:59.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:59 smithi066 ceph-mon[36236]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-28T16:26:59.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:59 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2935946808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-28T16:26:59.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:59 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:26:59.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:26:59 smithi066 ceph-mon[36236]: pgmap v229: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 3 op/s 2024-02-28T16:26:59.547 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-28T16:26:59.558 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:26:59.558 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-28T16:26:59.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:59 smithi001 ceph-mon[28991]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-28T16:26:59.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:59 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2935946808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-28T16:26:59.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:59 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:26:59.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:26:59 smithi001 ceph-mon[28991]: pgmap v229: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 3 op/s 2024-02-28T16:27:00.606 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:01.741 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:01 smithi001 ceph-mon[28991]: pgmap v230: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 3 op/s 2024-02-28T16:27:01.741 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:01 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] up:active 2024-02-28T16:27:01.741 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:01 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] up:active 2024-02-28T16:27:01.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:01 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:27:01.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:01 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:01.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:01 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:01.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:01 smithi066 ceph-mon[36236]: pgmap v230: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 3 op/s 2024-02-28T16:27:01.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:01 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] up:active 2024-02-28T16:27:01.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:01 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] up:active 2024-02-28T16:27:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:01 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:27:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:01 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:01.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:01 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:02 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] up:standby 2024-02-28T16:27:02.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:02 smithi066 ceph-mon[36236]: mds.? [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] up:standby 2024-02-28T16:27:02.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:02 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:27:02.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:02 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:02.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:02 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] up:standby 2024-02-28T16:27:02.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:02 smithi001 ceph-mon[28991]: mds.? [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] up:standby 2024-02-28T16:27:02.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:02 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:27:02.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:02 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:03.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:03 smithi001 ceph-mon[28991]: pgmap v231: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 1 op/s 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.813+0000 7f3ca3e00700 1 -- 172.21.15.1:0/1897052716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c0ff150 msgr2=0x7f3c9c0ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.813+0000 7f3ca3e00700 1 --2- 172.21.15.1:0/1897052716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c0ff150 0x7f3c9c0ff560 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c90004940 tx=0x7f3c90009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.813+0000 7f3ca3e00700 1 -- 172.21.15.1:0/1897052716 shutdown_connections 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.813+0000 7f3ca3e00700 1 --2- 172.21.15.1:0/1897052716 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c9c100360 0x7f3c9c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.813+0000 7f3ca3e00700 1 --2- 172.21.15.1:0/1897052716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c0ff150 0x7f3c9c0ff560 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.813+0000 7f3ca3e00700 1 -- 172.21.15.1:0/1897052716 >> 172.21.15.1:0/1897052716 conn(0x7f3c9c0fa920 msgr2=0x7f3c9c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.813+0000 7f3ca3e00700 1 -- 172.21.15.1:0/1897052716 shutdown_connections 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.813+0000 7f3ca3e00700 1 -- 172.21.15.1:0/1897052716 wait complete. 2024-02-28T16:27:03.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.814+0000 7f3ca3e00700 1 Processor -- start 2024-02-28T16:27:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.814+0000 7f3ca3e00700 1 -- start start 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca3e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c9c0ff150 0x7f3c9c10fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c100360 0x7f3c9c110100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca3e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c110790 con 0x7f3c9c100360 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca3e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c110fd0 con 0x7f3c9c0ff150 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca0a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c100360 0x7f3c9c110100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca0a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c100360 0x7f3c9c110100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57004/0 (socket says 172.21.15.1:57004) 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca0a00700 1 -- 172.21.15.1:0/392843959 learned_addr learned my addr 172.21.15.1:0/392843959 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca1400700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c9c0ff150 0x7f3c9c10fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:03.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca0a00700 1 -- 172.21.15.1:0/392843959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c9c0ff150 msgr2=0x7f3c9c10fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca0a00700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c9c0ff150 0x7f3c9c10fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca0a00700 1 -- 172.21.15.1:0/392843959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90005040 con 0x7f3c9c100360 2024-02-28T16:27:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.815+0000 7f3ca1400700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c9c0ff150 0x7f3c9c10fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:27:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.816+0000 7f3ca0a00700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c100360 0x7f3c9c110100 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800b790 tx=0x7f3c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.816+0000 7f3c99600700 1 -- 172.21.15.1:0/392843959 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c8800f770 con 0x7f3c9c100360 2024-02-28T16:27:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.816+0000 7f3c99600700 1 -- 172.21.15.1:0/392843959 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c8800fda0 con 0x7f3c9c100360 2024-02-28T16:27:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.816+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c1112b0 con 0x7f3c9c100360 2024-02-28T16:27:03.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.816+0000 7f3c99600700 1 -- 172.21.15.1:0/392843959 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c8800d310 con 0x7f3c9c100360 2024-02-28T16:27:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.816+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c1117f0 con 0x7f3c9c100360 2024-02-28T16:27:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.817+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c04bed0 con 0x7f3c9c100360 2024-02-28T16:27:03.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.818+0000 7f3c99600700 1 -- 172.21.15.1:0/392843959 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3c88010420 con 0x7f3c9c100360 2024-02-28T16:27:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.820+0000 7f3c99600700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3c840692a0 0x7f3c8406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:03.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.820+0000 7f3c99600700 1 -- 172.21.15.1:0/392843959 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3c88087240 con 0x7f3c9c100360 2024-02-28T16:27:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.820+0000 7f3ca1400700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3c840692a0 0x7f3c8406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:03.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.821+0000 7f3ca1400700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3c840692a0 0x7f3c8406b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009fd0 tx=0x7f3c90009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:03.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:03.824+0000 7f3c99600700 1 -- 172.21.15.1:0/392843959 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3c88056f60 con 0x7f3c9c100360 2024-02-28T16:27:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:03 smithi066 ceph-mon[36236]: pgmap v231: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 1 op/s 2024-02-28T16:27:04.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.095+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3c9c111c50 con 0x7f3c9c100360 2024-02-28T16:27:04.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.096+0000 7f3c99600700 1 -- 172.21.15.1:0/392843959 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f3c88056af0 con 0x7f3c9c100360 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:e14 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 13 2024-02-28T16:27:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:27:00.455017+0000 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:27:04.086 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:27:04.088 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:27:04.088 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3c840692a0 msgr2=0x7f3c8406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3c840692a0 0x7f3c8406b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009fd0 tx=0x7f3c90009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c100360 msgr2=0x7f3c9c110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c100360 0x7f3c9c110100 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800b790 tx=0x7f3c88009250 comp rx=0 tx=0).stop 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 shutdown_connections 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3c840692a0 0x7f3c8406b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c9c0ff150 0x7f3c9c10fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 --2- 172.21.15.1:0/392843959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c100360 0x7f3c9c110100 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.098+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 >> 172.21.15.1:0/392843959 conn(0x7f3c9c0fa920 msgr2=0x7f3c9c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.099+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 shutdown_connections 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:04.099+0000 7f3ca3e00700 1 -- 172.21.15.1:0/392843959 wait complete. 2024-02-28T16:27:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 14 2024-02-28T16:27:04.974 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:04 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/392843959' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:27:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:04 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/392843959' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:27:05.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-28T16:27:05.571 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:05.847 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:05 smithi001 ceph-mon[28991]: pgmap v232: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-28T16:27:05.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:05 smithi066 ceph-mon[36236]: pgmap v232: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-28T16:27:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.669+0000 7f3026a00700 1 -- 172.21.15.1:0/706173203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fe0e0 msgr2=0x7f30200fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.669+0000 7f3026a00700 1 --2- 172.21.15.1:0/706173203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fe0e0 0x7f30200fe550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f300c004940 tx=0x7f300c009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.669+0000 7f3026a00700 1 -- 172.21.15.1:0/706173203 shutdown_connections 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.669+0000 7f3026a00700 1 --2- 172.21.15.1:0/706173203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fe0e0 0x7f30200fe550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.669+0000 7f3026a00700 1 --2- 172.21.15.1:0/706173203 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30200fcef0 0x7f30200fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.669+0000 7f3026a00700 1 -- 172.21.15.1:0/706173203 >> 172.21.15.1:0/706173203 conn(0x7f30200f8620 msgr2=0x7f30200faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.669+0000 7f3026a00700 1 -- 172.21.15.1:0/706173203 shutdown_connections 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.669+0000 7f3026a00700 1 -- 172.21.15.1:0/706173203 wait complete. 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.670+0000 7f3026a00700 1 Processor -- start 2024-02-28T16:27:07.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.671+0000 7f3026a00700 1 -- start start 2024-02-28T16:27:07.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.671+0000 7f3026a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fcef0 0x7f3020071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:07.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.671+0000 7f3026a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30200fe0e0 0x7f30200702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:07.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.671+0000 7f3026a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3020111650 con 0x7f30200fcef0 2024-02-28T16:27:07.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.671+0000 7f3026a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30201117c0 con 0x7f30200fe0e0 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.671+0000 7f301fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fcef0 0x7f3020071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.671+0000 7f301fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fcef0 0x7f3020071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57026/0 (socket says 172.21.15.1:57026) 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.671+0000 7f301fe00700 1 -- 172.21.15.1:0/3795553465 learned_addr learned my addr 172.21.15.1:0/3795553465 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f301fe00700 1 -- 172.21.15.1:0/3795553465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30200fe0e0 msgr2=0x7f30200702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f301fe00700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30200fe0e0 0x7f30200702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f301fe00700 1 -- 172.21.15.1:0/3795553465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c005040 con 0x7f30200fcef0 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f301fe00700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fcef0 0x7f3020071e60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3010002a80 tx=0x7f3010004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f3017e00700 1 -- 172.21.15.1:0/3795553465 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3010007cc0 con 0x7f30200fcef0 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3020070940 con 0x7f30200fcef0 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f3017e00700 1 -- 172.21.15.1:0/3795553465 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f301000f040 con 0x7f30200fcef0 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f3017e00700 1 -- 172.21.15.1:0/3795553465 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f301000dad0 con 0x7f30200fcef0 2024-02-28T16:27:07.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.672+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3020070e80 con 0x7f30200fcef0 2024-02-28T16:27:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.674+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302004bed0 con 0x7f30200fcef0 2024-02-28T16:27:07.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.675+0000 7f3017e00700 1 -- 172.21.15.1:0/3795553465 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f301000dc30 con 0x7f30200fcef0 2024-02-28T16:27:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.676+0000 7f3017e00700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f30040692b0 0x7f300406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:07.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.676+0000 7f3017e00700 1 -- 172.21.15.1:0/3795553465 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3010086fe0 con 0x7f30200fcef0 2024-02-28T16:27:07.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.676+0000 7f301f400700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f30040692b0 0x7f300406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:07.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.677+0000 7f301f400700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f30040692b0 0x7f300406b760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f300c004f90 tx=0x7f300c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:07.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.682+0000 7f3017e00700 1 -- 172.21.15.1:0/3795553465 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3010056c80 con 0x7f30200fcef0 2024-02-28T16:27:07.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:07 smithi066 ceph-mon[36236]: pgmap v233: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:07.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:07.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:07.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:27:07.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:27:07.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:07.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:07 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:27:07.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:07 smithi001 ceph-mon[28991]: pgmap v233: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:07.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:07.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:07.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:27:07.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:27:07.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:07.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:07 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:27:07.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.960+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30200710a0 con 0x7f30200fcef0 2024-02-28T16:27:07.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.961+0000 7f3017e00700 1 -- 172.21.15.1:0/3795553465 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4695 (secure 0 0 0) 0x7f3010056810 con 0x7f30200fcef0 2024-02-28T16:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f30040692b0 msgr2=0x7f300406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f30040692b0 0x7f300406b760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f300c004f90 tx=0x7f300c009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fcef0 msgr2=0x7f3020071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fcef0 0x7f3020071e60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3010002a80 tx=0x7f3010004600 comp rx=0 tx=0).stop 2024-02-28T16:27:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 shutdown_connections 2024-02-28T16:27:07.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f30040692b0 0x7f300406b760 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:07.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30200fcef0 0x7f3020071e60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:07.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 --2- 172.21.15.1:0/3795553465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30200fe0e0 0x7f30200702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:07.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 >> 172.21.15.1:0/3795553465 conn(0x7f30200f8620 msgr2=0x7f30201013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:07.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.964+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 shutdown_connections 2024-02-28T16:27:07.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:07.965+0000 7f3026a00700 1 -- 172.21.15.1:0/3795553465 wait complete. 2024-02-28T16:27:07.955 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 14 2024-02-28T16:27:07.968 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:27:08.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:08 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3795553465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-28T16:27:08.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:08 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3795553465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-28T16:27:09.013 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-28T16:27:09.516 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:09.639 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:09 smithi066 ceph-mon[36236]: pgmap v234: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:09.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:09 smithi001 ceph-mon[28991]: pgmap v234: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:11.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:11 smithi001 ceph-mon[28991]: pgmap v235: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:11.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:11.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:11 smithi066 ceph-mon[36236]: pgmap v235: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:12.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.156+0000 7f79cd800700 1 -- 172.21.15.1:0/2538350238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c80fef60 msgr2=0x7f79c80ff370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:12.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.156+0000 7f79cd800700 1 --2- 172.21.15.1:0/2538350238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c80fef60 0x7f79c80ff370 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f79b8004940 tx=0x7f79b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:12.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.156+0000 7f79cd800700 1 -- 172.21.15.1:0/2538350238 shutdown_connections 2024-02-28T16:27:12.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.156+0000 7f79cd800700 1 --2- 172.21.15.1:0/2538350238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79c8100170 0x7f79c81005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:12.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.156+0000 7f79cd800700 1 --2- 172.21.15.1:0/2538350238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c80fef60 0x7f79c80ff370 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:12.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.156+0000 7f79cd800700 1 -- 172.21.15.1:0/2538350238 >> 172.21.15.1:0/2538350238 conn(0x7f79c80fa760 msgr2=0x7f79c80fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:12.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.157+0000 7f79cd800700 1 -- 172.21.15.1:0/2538350238 shutdown_connections 2024-02-28T16:27:12.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.157+0000 7f79cd800700 1 -- 172.21.15.1:0/2538350238 wait complete. 2024-02-28T16:27:12.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.158+0000 7f79cd800700 1 Processor -- start 2024-02-28T16:27:12.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.158+0000 7f79cd800700 1 -- start start 2024-02-28T16:27:12.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.158+0000 7f79cd800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79c80fef60 0x7f79c810f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:12.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.158+0000 7f79cd800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c8100170 0x7f79c810feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:12.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.158+0000 7f79cd800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c81104b0 con 0x7f79c8100170 2024-02-28T16:27:12.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.158+0000 7f79cd800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c8110d50 con 0x7f79c80fef60 2024-02-28T16:27:12.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.159+0000 7f79c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c8100170 0x7f79c810feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:12.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.159+0000 7f79c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c8100170 0x7f79c810feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52762/0 (socket says 172.21.15.1:52762) 2024-02-28T16:27:12.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.159+0000 7f79c7400700 1 -- 172.21.15.1:0/249474751 learned_addr learned my addr 172.21.15.1:0/249474751 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:12.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.159+0000 7f79c7e00700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79c80fef60 0x7f79c810f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:12.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.159+0000 7f79c7400700 1 -- 172.21.15.1:0/249474751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79c80fef60 msgr2=0x7f79c810f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:12.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.159+0000 7f79c7400700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79c80fef60 0x7f79c810f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:12.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.159+0000 7f79c7400700 1 -- 172.21.15.1:0/249474751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b8005040 con 0x7f79c8100170 2024-02-28T16:27:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.160+0000 7f79c7400700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c8100170 0x7f79c810feb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f79b400c8c0 tx=0x7f79b40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.160+0000 7f79bfe00700 1 -- 172.21.15.1:0/249474751 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79b400a980 con 0x7f79c8100170 2024-02-28T16:27:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.160+0000 7f79bfe00700 1 -- 172.21.15.1:0/249474751 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79b400ce60 con 0x7f79c8100170 2024-02-28T16:27:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.160+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c8111030 con 0x7f79c8100170 2024-02-28T16:27:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.160+0000 7f79bfe00700 1 -- 172.21.15.1:0/249474751 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79b40115e0 con 0x7f79c8100170 2024-02-28T16:27:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.160+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c8111570 con 0x7f79c8100170 2024-02-28T16:27:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.161+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c804bed0 con 0x7f79c8100170 2024-02-28T16:27:12.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.163+0000 7f79bfe00700 1 -- 172.21.15.1:0/249474751 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f79b4012400 con 0x7f79c8100170 2024-02-28T16:27:12.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.164+0000 7f79bfe00700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f79ac0692a0 0x7f79ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:12.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.164+0000 7f79bfe00700 1 -- 172.21.15.1:0/249474751 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f79b4087290 con 0x7f79c8100170 2024-02-28T16:27:12.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.164+0000 7f79c7e00700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f79ac0692a0 0x7f79ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:12.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.165+0000 7f79c7e00700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f79ac0692a0 0x7f79ac06b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f79b8009fd0 tx=0x7f79b8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:12.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.169+0000 7f79bfe00700 1 -- 172.21.15.1:0/249474751 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f79b4057050 con 0x7f79c8100170 2024-02-28T16:27:12.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.480+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f79c81119f0 con 0x7f79c8100170 2024-02-28T16:27:12.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.482+0000 7f79bfe00700 1 -- 172.21.15.1:0/249474751 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) v1 ==== 78+0+99 (secure 0 0 0) 0x7f79b40126b0 con 0x7f79c8100170 2024-02-28T16:27:12.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.484+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f79ac0692a0 msgr2=0x7f79ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:12.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.484+0000 7f79cd800700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f79ac0692a0 0x7f79ac06b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f79b8009fd0 tx=0x7f79b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.484+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c8100170 msgr2=0x7f79c810feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.484+0000 7f79cd800700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c8100170 0x7f79c810feb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f79b400c8c0 tx=0x7f79b40044c0 comp rx=0 tx=0).stop 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.485+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 shutdown_connections 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.485+0000 7f79cd800700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f79ac0692a0 0x7f79ac06b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.485+0000 7f79cd800700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79c80fef60 0x7f79c810f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.485+0000 7f79cd800700 1 --2- 172.21.15.1:0/249474751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c8100170 0x7f79c810feb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.485+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 >> 172.21.15.1:0/249474751 conn(0x7f79c80fa760 msgr2=0x7f79c81033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.485+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 shutdown_connections 2024-02-28T16:27:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:12.485+0000 7f79cd800700 1 -- 172.21.15.1:0/249474751 wait complete. 2024-02-28T16:27:12.490 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:27:12.542 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:12 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:12.542 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:12 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:27:12.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:12 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:12.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:12 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:27:13.519 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-28T16:27:13.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-28T16:27:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:13 smithi066 ceph-mon[36236]: pgmap v236: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:13.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:13 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/249474751' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-28T16:27:13.755 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:13 smithi001 ceph-mon[28991]: pgmap v236: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:13.755 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:13 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/249474751' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-28T16:27:13.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.786+0000 7f46dec00640 1 -- 172.21.15.1:0/3183067472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 msgr2=0x7f46d80fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:13.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.786+0000 7f46dec00640 1 --2- 172.21.15.1:0/3183067472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 0x7f46d80fcc80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f46c80048d0 tx=0x7f46c8009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:13.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.786+0000 7f46dec00640 1 -- 172.21.15.1:0/3183067472 shutdown_connections 2024-02-28T16:27:13.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.786+0000 7f46dec00640 1 --2- 172.21.15.1:0/3183067472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46d80fd1c0 0x7f46d80ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:13.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.786+0000 7f46dec00640 1 --2- 172.21.15.1:0/3183067472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 0x7f46d80fcc80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:13.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.786+0000 7f46dec00640 1 -- 172.21.15.1:0/3183067472 >> 172.21.15.1:0/3183067472 conn(0x7f46d80f6940 msgr2=0x7f46d80f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:13.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.787+0000 7f46dec00640 1 -- 172.21.15.1:0/3183067472 shutdown_connections 2024-02-28T16:27:13.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.787+0000 7f46dec00640 1 -- 172.21.15.1:0/3183067472 wait complete. 2024-02-28T16:27:13.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.787+0000 7f46dec00640 1 Processor -- start 2024-02-28T16:27:13.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46dec00640 1 -- start start 2024-02-28T16:27:13.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46dec00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 0x7f46d8113180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:13.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46dec00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46d80fd1c0 0x7f46d81136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:13.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46dec00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8113d10 con 0x7f46d80663b0 2024-02-28T16:27:13.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46dec00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8118760 con 0x7f46d80fd1c0 2024-02-28T16:27:13.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46d7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 0x7f46d8113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:13.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46d7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 0x7f46d8113180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52782/0 (socket says 172.21.15.1:52782) 2024-02-28T16:27:13.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46d7e00640 1 -- 172.21.15.1:0/3611693919 learned_addr learned my addr 172.21.15.1:0/3611693919 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:13.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.788+0000 7f46cfe00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46d80fd1c0 0x7f46d81136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.789+0000 7f46d7e00640 1 -- 172.21.15.1:0/3611693919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46d80fd1c0 msgr2=0x7f46d81136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.789+0000 7f46d7e00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46d80fd1c0 0x7f46d81136c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.789+0000 7f46d7e00640 1 -- 172.21.15.1:0/3611693919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c8005040 con 0x7f46d80663b0 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.789+0000 7f46cfe00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46d80fd1c0 0x7f46d81136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.789+0000 7f46d7e00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 0x7f46d8113180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f46c802bda0 tx=0x7f46c8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.790+0000 7f46d4c00640 1 -- 172.21.15.1:0/3611693919 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c80094c0 con 0x7f46d80663b0 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.790+0000 7f46d4c00640 1 -- 172.21.15.1:0/3611693919 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46c8009620 con 0x7f46d80663b0 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.790+0000 7f46d4c00640 1 -- 172.21.15.1:0/3611693919 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c80435f0 con 0x7f46d80663b0 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.790+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d81188a0 con 0x7f46d80663b0 2024-02-28T16:27:13.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.790+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d8118d30 con 0x7f46d80663b0 2024-02-28T16:27:13.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.791+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d810d380 con 0x7f46d80663b0 2024-02-28T16:27:13.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.792+0000 7f46d4c00640 1 -- 172.21.15.1:0/3611693919 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f46c802f070 con 0x7f46d80663b0 2024-02-28T16:27:13.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.793+0000 7f46d4c00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46bc069220 0x7f46bc06b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:13.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.793+0000 7f46d4c00640 1 -- 172.21.15.1:0/3611693919 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f46c80a9090 con 0x7f46d80663b0 2024-02-28T16:27:13.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.793+0000 7f46cfe00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46bc069220 0x7f46bc06b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:13.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.794+0000 7f46cfe00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46bc069220 0x7f46bc06b6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009fd0 tx=0x7f46c0009e50 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:13.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:13.801+0000 7f46d4c00640 1 -- 172.21.15.1:0/3611693919 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f46c8075340 con 0x7f46d80663b0 2024-02-28T16:27:14.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.024+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46d802ab50 con 0x7f46d80663b0 2024-02-28T16:27:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.025+0000 7f46d4c00640 1 -- 172.21.15.1:0/3611693919 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+14155 (secure 0 0 0) 0x7f46c8078960 con 0x7f46d80663b0 2024-02-28T16:27:14.016 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:27:14.016 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":45,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","created":"2024-02-28T16:17:42.450467+0000","modified":"2024-02-28T16:26:40.917386+0000","last_up_change":"2024-02-28T16:25:02.506452+0000","last_in_change":"2024-02-28T16:24:36.954989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-28T16:23:15.048529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-28T16:26:37.820888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-28T16:26:38.430385+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ac92369f-8144-4695-87fe-f9cd73fb5f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6803","nonce":3270329297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6805","nonce":3270329297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6809","nonce":3270329297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3270329297},{"type":"v1","addr":"172.21.15.1:6807","nonce":3270329297}]},"public_addr":"172.21.15.1:6803/3270329297","cluster_addr":"172.21.15.1:6805/3270329297","heartbeat_back_addr":"172.21.15.1:6809/3270329297","heartbeat_front_addr":"172.21.15.1:6807/3270329297","state":["exists","up"]},{"osd":1,"uuid":"bfd6ea17-fd9b-46ae-b3ff-547d736cd232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6811","nonce":1496879960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6813","nonce":1496879960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6817","nonce":1496879960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1496879960},{"type":"v1","addr":"172.21.15.1:6815","nonce":1496879960}]},"public_addr":"172.21.15.1:6811/1496879960","cluster_addr":"172.21.15.1:6813/1496879960","heartbeat_back_addr":"172.21.15.1:6817/1496879960","heartbeat_front_addr":"172.21.15.1:6815/1496879960","state":["exists","up"]},{"osd":2,"uuid":"b713f513-6604-4e88-8cc1-0f963d9f8599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6819","nonce":131562471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6821","nonce":131562471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6825","nonce":131562471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":131562471},{"type":"v1","addr":"172.21.15.1:6823","nonce":131562471}]},"public_addr":"172.21.15.1:6819/131562471","cluster_addr":"172.21.15.1:6821/131562471","heartbeat_back_addr":"172.21.15.1:6825/131562471","heartbeat_front_addr":"172.21.15.1:6823/131562471","state":["exists","up"]},{"osd":3,"uuid":"4f425b10-672f-4265-a314-5a9a15237257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6801","nonce":863585173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6803","nonce":863585173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6807","nonce":863585173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":863585173},{"type":"v1","addr":"172.21.15.66:6805","nonce":863585173}]},"public_addr":"172.21.15.66:6801/863585173","cluster_addr":"172.21.15.66:6803/863585173","heartbeat_back_addr":"172.21.15.66:6807/863585173","heartbeat_front_addr":"172.21.15.66:6805/863585173","state":["exists","up"]},{"osd":4,"uuid":"c1f2651b-47f3-444e-9325-22a0a962c3b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6809","nonce":376874414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6811","nonce":376874414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6815","nonce":376874414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":376874414},{"type":"v1","addr":"172.21.15.66:6813","nonce":376874414}]},"public_addr":"172.21.15.66:6809/376874414","cluster_addr":"172.21.15.66:6811/376874414","heartbeat_back_addr":"172.21.15.66:6815/376874414","heartbeat_front_addr":"172.21.15.66:6813/376874414","state":["exists","up"]},{"osd":5,"uuid":"e83df14f-7d81-4bb7-9617-024ad94de1a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6817","nonce":2849488501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6819","nonce":2849488501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6823","nonce":2849488501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2849488501},{"type":"v1","addr":"172.21.15.66:6821","nonce":2849488501}]},"public_addr":"172.21.15.66:6817/2849488501","cluster_addr":"172.21.15.66:6819/2849488501","heartbeat_back_addr":"172.21.15.66:6823/2849488501","heartbeat_front_addr":"172.21.15.66:6821/2849488501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:01.831777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:36.505002+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:13.091284+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:47.260219+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:24:23.366141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:25:00.532840+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6800/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/298653290":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/3865282714":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/4195777758":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/2061955559":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/4223980442":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1339161018":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/833315191":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6801/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:6801/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6801/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/2780166226":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/1224289276":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3320028482":"2024-02-29T16:18:55.593679+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-28T16:27:14.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.028+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46bc069220 msgr2=0x7f46bc06b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.028+0000 7f46dec00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46bc069220 0x7f46bc06b6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009fd0 tx=0x7f46c0009e50 comp rx=0 tx=0).stop 2024-02-28T16:27:14.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.028+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 msgr2=0x7f46d8113180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.028+0000 7f46dec00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 0x7f46d8113180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f46c802bda0 tx=0x7f46c8009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.029+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 shutdown_connections 2024-02-28T16:27:14.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.029+0000 7f46dec00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f46bc069220 0x7f46bc06b6d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.029+0000 7f46dec00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46d80fd1c0 0x7f46d81136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.029+0000 7f46dec00640 1 --2- 172.21.15.1:0/3611693919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46d80663b0 0x7f46d8113180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.029+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 >> 172.21.15.1:0/3611693919 conn(0x7f46d80f6940 msgr2=0x7f46d80f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:14.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.029+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 shutdown_connections 2024-02-28T16:27:14.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.029+0000 7f46dec00640 1 -- 172.21.15.1:0/3611693919 wait complete. 2024-02-28T16:27:14.038 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-28T16:23:15.048529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-28T16:26:37.820888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-28T16:26:38.430385+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-28T16:27:14.038 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-28T16:27:14.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.240+0000 7fa3d5e00640 1 -- 172.21.15.1:0/3481616334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fd360 msgr2=0x7fa3d00fd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.240+0000 7fa3d5e00640 1 --2- 172.21.15.1:0/3481616334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fd360 0x7fa3d00fd7d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00048d0 tx=0x7fa3c0009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.240+0000 7fa3d5e00640 1 -- 172.21.15.1:0/3481616334 shutdown_connections 2024-02-28T16:27:14.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.240+0000 7fa3d5e00640 1 --2- 172.21.15.1:0/3481616334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fd360 0x7fa3d00fd7d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.240+0000 7fa3d5e00640 1 --2- 172.21.15.1:0/3481616334 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3d00fc130 0x7fa3d00fc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.240+0000 7fa3d5e00640 1 -- 172.21.15.1:0/3481616334 >> 172.21.15.1:0/3481616334 conn(0x7fa3d00f7810 msgr2=0x7fa3d00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:14.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.240+0000 7fa3d5e00640 1 -- 172.21.15.1:0/3481616334 shutdown_connections 2024-02-28T16:27:14.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.240+0000 7fa3d5e00640 1 -- 172.21.15.1:0/3481616334 wait complete. 2024-02-28T16:27:14.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.241+0000 7fa3d5e00640 1 Processor -- start 2024-02-28T16:27:14.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.241+0000 7fa3d5e00640 1 -- start start 2024-02-28T16:27:14.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.241+0000 7fa3d5e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fc130 0x7fa3d010eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:14.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d5e00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3d00fd360 0x7fa3d010f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:14.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d5e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d010f720 con 0x7fa3d00fc130 2024-02-28T16:27:14.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d5e00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d0114170 con 0x7fa3d00fd360 2024-02-28T16:27:14.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d4a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fc130 0x7fa3d010eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:14.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d4a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fc130 0x7fa3d010eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52792/0 (socket says 172.21.15.1:52792) 2024-02-28T16:27:14.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d4a00640 1 -- 172.21.15.1:0/1853697803 learned_addr learned my addr 172.21.15.1:0/1853697803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:14.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3cfe00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3d00fd360 0x7fa3d010f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:14.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d4a00640 1 -- 172.21.15.1:0/1853697803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3d00fd360 msgr2=0x7fa3d010f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d4a00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3d00fd360 0x7fa3d010f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.242+0000 7fa3d4a00640 1 -- 172.21.15.1:0/1853697803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c0005040 con 0x7fa3d00fc130 2024-02-28T16:27:14.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.243+0000 7fa3d4a00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fc130 0x7fa3d010eb90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c810 tx=0x7fa3c4004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:14.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.243+0000 7fa3ccc00640 1 -- 172.21.15.1:0/1853697803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3c400a850 con 0x7fa3d00fc130 2024-02-28T16:27:14.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.243+0000 7fa3ccc00640 1 -- 172.21.15.1:0/1853697803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3c400ae80 con 0x7fa3d00fc130 2024-02-28T16:27:14.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.243+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d0114310 con 0x7fa3d00fc130 2024-02-28T16:27:14.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.243+0000 7fa3ccc00640 1 -- 172.21.15.1:0/1853697803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3c40115e0 con 0x7fa3d00fc130 2024-02-28T16:27:14.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.243+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d0114800 con 0x7fa3d00fc130 2024-02-28T16:27:14.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.245+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d0101370 con 0x7fa3d00fc130 2024-02-28T16:27:14.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.245+0000 7fa3ccc00640 1 -- 172.21.15.1:0/1853697803 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa3c4012400 con 0x7fa3d00fc130 2024-02-28T16:27:14.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.246+0000 7fa3ccc00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3b8069210 0x7fa3b806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:14.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.246+0000 7fa3ccc00640 1 -- 172.21.15.1:0/1853697803 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fa3c40873c0 con 0x7fa3d00fc130 2024-02-28T16:27:14.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.246+0000 7fa3cfe00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3b8069210 0x7fa3b806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:14.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.247+0000 7fa3cfe00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3b8069210 0x7fa3b806b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa3d01100e0 tx=0x7fa3c0009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:14.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.250+0000 7fa3ccc00640 1 -- 172.21.15.1:0/1853697803 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa3c40536f0 con 0x7fa3d00fc130 2024-02-28T16:27:14.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.469+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3d002ab10 con 0x7fa3d00fc130 2024-02-28T16:27:14.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.470+0000 7fa3ccc00640 1 -- 172.21.15.1:0/1853697803 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3c4056d10 con 0x7fa3d00fc130 2024-02-28T16:27:14.460 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-02-28T16:27:14.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3b8069210 msgr2=0x7fa3b806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3b8069210 0x7fa3b806b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa3d01100e0 tx=0x7fa3c0009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fc130 msgr2=0x7fa3d010eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fc130 0x7fa3d010eb90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c810 tx=0x7fa3c4004430 comp rx=0 tx=0).stop 2024-02-28T16:27:14.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 shutdown_connections 2024-02-28T16:27:14.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fa3b8069210 0x7fa3b806b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3d00fd360 0x7fa3d010f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 --2- 172.21.15.1:0/1853697803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3d00fc130 0x7fa3d010eb90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 >> 172.21.15.1:0/1853697803 conn(0x7fa3d00f7810 msgr2=0x7fa3d00fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:14.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.472+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 shutdown_connections 2024-02-28T16:27:14.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.473+0000 7fa3d5e00640 1 -- 172.21.15.1:0/1853697803 wait complete. 2024-02-28T16:27:14.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-28T16:27:14.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.689+0000 7fabee400640 1 -- 172.21.15.1:0/3690022900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe8066350 msgr2=0x7fabe80fcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.689+0000 7fabee400640 1 --2- 172.21.15.1:0/3690022900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe8066350 0x7fabe80fcca0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fabd80048d0 tx=0x7fabd8009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.689+0000 7fabee400640 1 -- 172.21.15.1:0/3690022900 shutdown_connections 2024-02-28T16:27:14.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.689+0000 7fabee400640 1 --2- 172.21.15.1:0/3690022900 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabe80fd1e0 0x7fabe80ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.689+0000 7fabee400640 1 --2- 172.21.15.1:0/3690022900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe8066350 0x7fabe80fcca0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.689+0000 7fabee400640 1 -- 172.21.15.1:0/3690022900 >> 172.21.15.1:0/3690022900 conn(0x7fabe80f6960 msgr2=0x7fabe80f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:14.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.689+0000 7fabee400640 1 -- 172.21.15.1:0/3690022900 shutdown_connections 2024-02-28T16:27:14.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.689+0000 7fabee400640 1 -- 172.21.15.1:0/3690022900 wait complete. 2024-02-28T16:27:14.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.690+0000 7fabee400640 1 Processor -- start 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.690+0000 7fabee400640 1 -- start start 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.690+0000 7fabee400640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabe8066350 0x7fabe8113140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.690+0000 7fabee400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe80fd1e0 0x7fabe8113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.690+0000 7fabee400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8113cd0 con 0x7fabe80fd1e0 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.690+0000 7fabee400640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8118720 con 0x7fabe8066350 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe6a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe80fd1e0 0x7fabe8113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe6a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe80fd1e0 0x7fabe8113680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52806/0 (socket says 172.21.15.1:52806) 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe6a00640 1 -- 172.21.15.1:0/3662528326 learned_addr learned my addr 172.21.15.1:0/3662528326 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe7400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabe8066350 0x7fabe8113140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe6a00640 1 -- 172.21.15.1:0/3662528326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabe8066350 msgr2=0x7fabe8113140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe6a00640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabe8066350 0x7fabe8113140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe6a00640 1 -- 172.21.15.1:0/3662528326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd8005040 con 0x7fabe80fd1e0 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe7400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabe8066350 0x7fabe8113140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.691+0000 7fabe6a00640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe80fd1e0 0x7fabe8113680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fabd000cae0 tx=0x7fabd0007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.692+0000 7fabdf400640 1 -- 172.21.15.1:0/3662528326 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd000ab00 con 0x7fabe80fd1e0 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.692+0000 7fabdf400640 1 -- 172.21.15.1:0/3662528326 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabd000ac60 con 0x7fabe80fd1e0 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.692+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe81188c0 con 0x7fabe80fd1e0 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.692+0000 7fabdf400640 1 -- 172.21.15.1:0/3662528326 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd00116f0 con 0x7fabe80fd1e0 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.692+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe8118db0 con 0x7fabe80fd1e0 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.693+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe810b3f0 con 0x7fabe80fd1e0 2024-02-28T16:27:14.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.694+0000 7fabdf400640 1 -- 172.21.15.1:0/3662528326 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fabd0012440 con 0x7fabe80fd1e0 2024-02-28T16:27:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.695+0000 7fabdf400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fabc8069220 0x7fabc806b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.695+0000 7fabdf400640 1 -- 172.21.15.1:0/3662528326 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fabd0087a60 con 0x7fabe80fd1e0 2024-02-28T16:27:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.696+0000 7fabe7400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fabc8069220 0x7fabc806b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:14.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.696+0000 7fabe7400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fabc8069220 0x7fabc806b6d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fabd802bda0 tx=0x7fabd8009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:14.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.700+0000 7fabdf400640 1 -- 172.21.15.1:0/3662528326 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fabd0053d30 con 0x7fabe80fd1e0 2024-02-28T16:27:14.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:14 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3611693919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T16:27:14.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:14 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3611693919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T16:27:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.896+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fabe8119210 con 0x7fabe80fd1e0 2024-02-28T16:27:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.896+0000 7fabdf400640 1 -- 172.21.15.1:0/3662528326 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7fabd0057350 con 0x7fabe80fd1e0 2024-02-28T16:27:14.886 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fabc8069220 msgr2=0x7fabc806b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fabc8069220 0x7fabc806b6d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fabd802bda0 tx=0x7fabd8009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe80fd1e0 msgr2=0x7fabe8113680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe80fd1e0 0x7fabe8113680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fabd000cae0 tx=0x7fabd0007e50 comp rx=0 tx=0).stop 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 shutdown_connections 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fabc8069220 0x7fabc806b6d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabe80fd1e0 0x7fabe8113680 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 --2- 172.21.15.1:0/3662528326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabe8066350 0x7fabe8113140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 >> 172.21.15.1:0/3662528326 conn(0x7fabe80f6960 msgr2=0x7fabe80f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 shutdown_connections 2024-02-28T16:27:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:14.899+0000 7fabee400640 1 -- 172.21.15.1:0/3662528326 wait complete. 2024-02-28T16:27:14.914 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-28T16:27:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.131+0000 7fae55a00640 1 -- 172.21.15.1:0/4284287017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500fe320 msgr2=0x7fae500fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.131+0000 7fae55a00640 1 --2- 172.21.15.1:0/4284287017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500fe320 0x7fae500fe730 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fae400048d0 tx=0x7fae40009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.131+0000 7fae55a00640 1 -- 172.21.15.1:0/4284287017 shutdown_connections 2024-02-28T16:27:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.131+0000 7fae55a00640 1 --2- 172.21.15.1:0/4284287017 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae500ff550 0x7fae500ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.131+0000 7fae55a00640 1 --2- 172.21.15.1:0/4284287017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500fe320 0x7fae500fe730 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.131+0000 7fae55a00640 1 -- 172.21.15.1:0/4284287017 >> 172.21.15.1:0/4284287017 conn(0x7fae500f9ab0 msgr2=0x7fae500fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:15.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.132+0000 7fae55a00640 1 -- 172.21.15.1:0/4284287017 shutdown_connections 2024-02-28T16:27:15.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.132+0000 7fae55a00640 1 -- 172.21.15.1:0/4284287017 wait complete. 2024-02-28T16:27:15.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.132+0000 7fae55a00640 1 Processor -- start 2024-02-28T16:27:15.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae55a00640 1 -- start start 2024-02-28T16:27:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae55a00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae500fe320 0x7fae5010ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae55a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500ff550 0x7fae5010f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae55a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5010f770 con 0x7fae500ff550 2024-02-28T16:27:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae55a00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae50114180 con 0x7fae500fe320 2024-02-28T16:27:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae4f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500ff550 0x7fae5010f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae4f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500ff550 0x7fae5010f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52820/0 (socket says 172.21.15.1:52820) 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae4f400640 1 -- 172.21.15.1:0/2570203607 learned_addr learned my addr 172.21.15.1:0/2570203607 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae4f400640 1 -- 172.21.15.1:0/2570203607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae500fe320 msgr2=0x7fae5010ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae4f400640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae500fe320 0x7fae5010ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.133+0000 7fae4f400640 1 -- 172.21.15.1:0/2570203607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae40005040 con 0x7fae500ff550 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.134+0000 7fae4f400640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500ff550 0x7fae5010f120 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00c830 tx=0x7fae3c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.134+0000 7fae47e00640 1 -- 172.21.15.1:0/2570203607 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae3c00a850 con 0x7fae500ff550 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.134+0000 7fae47e00640 1 -- 172.21.15.1:0/2570203607 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae3c00ae80 con 0x7fae500ff550 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.134+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae50114320 con 0x7fae500ff550 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.134+0000 7fae47e00640 1 -- 172.21.15.1:0/2570203607 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae3c012ea0 con 0x7fae500ff550 2024-02-28T16:27:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.134+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae501147f0 con 0x7fae500ff550 2024-02-28T16:27:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.135+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae50106e70 con 0x7fae500ff550 2024-02-28T16:27:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.136+0000 7fae47e00640 1 -- 172.21.15.1:0/2570203607 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fae3c012400 con 0x7fae500ff550 2024-02-28T16:27:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.137+0000 7fae47e00640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fae34069220 0x7fae3406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:15.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.138+0000 7fae47e00640 1 -- 172.21.15.1:0/2570203607 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fae3c087400 con 0x7fae500ff550 2024-02-28T16:27:15.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.138+0000 7fae4fe00640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fae34069220 0x7fae3406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:15.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.138+0000 7fae4fe00640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fae34069220 0x7fae3406b6d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae400048a0 tx=0x7fae40009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:15.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.141+0000 7fae47e00640 1 -- 172.21.15.1:0/2570203607 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fae3c053730 con 0x7fae500ff550 2024-02-28T16:27:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.338+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fae50114a70 con 0x7fae500ff550 2024-02-28T16:27:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.339+0000 7fae47e00640 1 -- 172.21.15.1:0/2570203607 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7fae3c0126b0 con 0x7fae500ff550 2024-02-28T16:27:15.328 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-02-28T16:27:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fae34069220 msgr2=0x7fae3406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fae34069220 0x7fae3406b6d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae400048a0 tx=0x7fae40009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500ff550 msgr2=0x7fae5010f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500ff550 0x7fae5010f120 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00c830 tx=0x7fae3c004430 comp rx=0 tx=0).stop 2024-02-28T16:27:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 shutdown_connections 2024-02-28T16:27:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7fae34069220 0x7fae3406b6d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae500ff550 0x7fae5010f120 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 --2- 172.21.15.1:0/2570203607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae500fe320 0x7fae5010ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.341+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 >> 172.21.15.1:0/2570203607 conn(0x7fae500f9ab0 msgr2=0x7fae50102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.342+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 shutdown_connections 2024-02-28T16:27:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.342+0000 7fae55a00640 1 -- 172.21.15.1:0/2570203607 wait complete. 2024-02-28T16:27:15.350 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-28T16:27:15.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.565+0000 7f57d3600640 1 -- 172.21.15.1:0/569923195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 msgr2=0x7f57cc070220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:15.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.565+0000 7f57d3600640 1 --2- 172.21.15.1:0/569923195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 0x7f57cc070220 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f57b80048d0 tx=0x7f57b8009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.565+0000 7f57d3600640 1 -- 172.21.15.1:0/569923195 shutdown_connections 2024-02-28T16:27:15.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.565+0000 7f57d3600640 1 --2- 172.21.15.1:0/569923195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 0x7f57cc070220 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.565+0000 7f57d3600640 1 --2- 172.21.15.1:0/569923195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57cc071400 0x7f57cc06f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.565+0000 7f57d3600640 1 -- 172.21.15.1:0/569923195 >> 172.21.15.1:0/569923195 conn(0x7f57cc0f7980 msgr2=0x7f57cc0f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:15.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.565+0000 7f57d3600640 1 -- 172.21.15.1:0/569923195 shutdown_connections 2024-02-28T16:27:15.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.565+0000 7f57d3600640 1 -- 172.21.15.1:0/569923195 wait complete. 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.566+0000 7f57d3600640 1 Processor -- start 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.566+0000 7f57d3600640 1 -- start start 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.566+0000 7f57d3600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 0x7f57cc113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.566+0000 7f57d3600640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57cc071400 0x7f57cc113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.566+0000 7f57d3600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc113cc0 con 0x7f57cc06fdd0 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.566+0000 7f57d3600640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc118710 con 0x7f57cc071400 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.567+0000 7f57d0c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 0x7f57cc113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.567+0000 7f57d0c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 0x7f57cc113130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52826/0 (socket says 172.21.15.1:52826) 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.567+0000 7f57d0c00640 1 -- 172.21.15.1:0/3353357064 learned_addr learned my addr 172.21.15.1:0/3353357064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.567+0000 7f57cbe00640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57cc071400 0x7f57cc113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:15.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.567+0000 7f57d0c00640 1 -- 172.21.15.1:0/3353357064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57cc071400 msgr2=0x7f57cc113670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.567+0000 7f57d0c00640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57cc071400 0x7f57cc113670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.567+0000 7f57d0c00640 1 -- 172.21.15.1:0/3353357064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b8005040 con 0x7f57cc06fdd0 2024-02-28T16:27:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.568+0000 7f57d0c00640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 0x7f57cc113130 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f57c0002a50 tx=0x7f57c0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.568+0000 7f57c8c00640 1 -- 172.21.15.1:0/3353357064 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c0007c30 con 0x7f57cc06fdd0 2024-02-28T16:27:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.568+0000 7f57c8c00640 1 -- 172.21.15.1:0/3353357064 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57c000f040 con 0x7f57cc06fdd0 2024-02-28T16:27:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.568+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc1188b0 con 0x7f57cc06fdd0 2024-02-28T16:27:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.568+0000 7f57c8c00640 1 -- 172.21.15.1:0/3353357064 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c000db50 con 0x7f57cc06fdd0 2024-02-28T16:27:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.568+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc118da0 con 0x7f57cc06fdd0 2024-02-28T16:27:15.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.570+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc10b3f0 con 0x7f57cc06fdd0 2024-02-28T16:27:15.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.572+0000 7f57c8c00640 1 -- 172.21.15.1:0/3353357064 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f57c001c400 con 0x7f57cc06fdd0 2024-02-28T16:27:15.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.573+0000 7f57c8c00640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f57b4069210 0x7f57b406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.573+0000 7f57c8c00640 1 -- 172.21.15.1:0/3353357064 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f57c0086a80 con 0x7f57cc06fdd0 2024-02-28T16:27:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.573+0000 7f57cbe00640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f57b4069210 0x7f57b406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:15.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.574+0000 7f57cbe00640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f57b4069210 0x7f57b406b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57cc114680 tx=0x7f57b8009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:15.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.577+0000 7f57c8c00640 1 -- 172.21.15.1:0/3353357064 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f57c004cbd0 con 0x7f57cc06fdd0 2024-02-28T16:27:15.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:15 smithi066 ceph-mon[36236]: pgmap v237: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:15.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:15 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1853697803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T16:27:15.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:15 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3662528326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-28T16:27:15.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:15 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2570203607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-28T16:27:15.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:15 smithi001 ceph-mon[28991]: pgmap v237: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:15.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:15 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1853697803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T16:27:15.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:15 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3662528326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-28T16:27:15.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:15 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2570203607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-28T16:27:15.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.812+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57cc118fc0 con 0x7f57cc06fdd0 2024-02-28T16:27:15.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.813+0000 7f57c8c00640 1 -- 172.21.15.1:0/3353357064 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4695 (secure 0 0 0) 0x7f57c0052d30 con 0x7f57cc06fdd0 2024-02-28T16:27:15.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:27:15.803 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:27:00.455017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14464,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14464":{"gid":14464,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6827/1940162712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1940162712},{"type":"v1","addr":"172.21.15.1:6827","nonce":1940162712}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi066.wfbdor","rank":1,"incarnation":7,"state":"up:active","state_seq":5,"addr":"172.21.15.66:6825/1481832428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1481832428},{"type":"v1","addr":"172.21.15.66:6825","nonce":1481832428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-02-28T16:27:15.803 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 14 2024-02-28T16:27:15.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.815+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f57b4069210 msgr2=0x7f57b406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.815+0000 7f57d3600640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f57b4069210 0x7f57b406b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57cc114680 tx=0x7f57b8009da0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 msgr2=0x7f57cc113130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 0x7f57cc113130 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f57c0002a50 tx=0x7f57c0004600 comp rx=0 tx=0).stop 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 shutdown_connections 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f57b4069210 0x7f57b406b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57cc071400 0x7f57cc113670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 --2- 172.21.15.1:0/3353357064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57cc06fdd0 0x7f57cc113130 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 >> 172.21.15.1:0/3353357064 conn(0x7f57cc0f7980 msgr2=0x7f57cc0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 shutdown_connections 2024-02-28T16:27:15.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:15.816+0000 7f57d3600640 1 -- 172.21.15.1:0/3353357064 wait complete. 2024-02-28T16:27:15.831 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 13, 'max_mds': 2, 'flags': 18} 2024-02-28T16:27:15.831 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-02-28T16:27:15.857 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-02-28T16:27:15.857 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-02-28T16:27:15.858 INFO:tasks.ceph_fuse:client.0 config is {} 2024-02-28T16:27:15.858 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-28T16:27:15.858 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:15.858 INFO:tasks.ceph_fuse:client.1 config is {} 2024-02-28T16:27:15.858 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-28T16:27:15.858 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:15.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:15.859 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-02-28T16:27:15.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:15.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-02-28T16:27:16.002 INFO:teuthology.orchestra.run.smithi001.stderr:Cannot find device "ceph-brx" 2024-02-28T16:27:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T16:27:16.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:16.007 DEBUG:teuthology.orchestra.run.smithi066:> ip netns list 2024-02-28T16:27:16.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:16.069 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip link delete ceph-brx 2024-02-28T16:27:16.183 INFO:teuthology.orchestra.run.smithi066.stderr:Cannot find device "ceph-brx" 2024-02-28T16:27:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T16:27:16.188 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-02-28T16:27:16.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph fs ls 2024-02-28T16:27:16.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:16 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3353357064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-28T16:27:16.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:16 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:16.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:16 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:16.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:16 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3353357064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-28T16:27:16.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:16 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:16.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:16 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:17.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:17 smithi066 ceph-mon[36236]: pgmap v238: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:17.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:17 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:27:17.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:17 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:27:17.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:17 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:17.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:17 smithi001 ceph-mon[28991]: pgmap v238: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:17.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:17 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:27:17.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:17 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:27:17.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:17 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:27:19.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.138+0000 7f68f7600700 1 -- 172.21.15.1:0/2294034777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 msgr2=0x7f68f0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.138+0000 7f68f7600700 1 --2- 172.21.15.1:0/2294034777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 0x7f68f0103320 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f68e4004940 tx=0x7f68e4009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.138+0000 7f68f7600700 1 -- 172.21.15.1:0/2294034777 shutdown_connections 2024-02-28T16:27:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.138+0000 7f68f7600700 1 --2- 172.21.15.1:0/2294034777 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68f01038f0 0x7f68f00fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.138+0000 7f68f7600700 1 --2- 172.21.15.1:0/2294034777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 0x7f68f0103320 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.138+0000 7f68f7600700 1 -- 172.21.15.1:0/2294034777 >> 172.21.15.1:0/2294034777 conn(0x7f68f00f8ae0 msgr2=0x7f68f00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.138+0000 7f68f7600700 1 -- 172.21.15.1:0/2294034777 shutdown_connections 2024-02-28T16:27:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.138+0000 7f68f7600700 1 -- 172.21.15.1:0/2294034777 wait complete. 2024-02-28T16:27:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.139+0000 7f68f7600700 1 Processor -- start 2024-02-28T16:27:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.140+0000 7f68f7600700 1 -- start start 2024-02-28T16:27:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.140+0000 7f68f7600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 0x7f68f0115be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:19.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.140+0000 7f68f7600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68f01038f0 0x7f68f0116120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.140+0000 7f68f7600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f0116770 con 0x7f68f0102f50 2024-02-28T16:27:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.140+0000 7f68f7600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f010faa0 con 0x7f68f01038f0 2024-02-28T16:27:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.140+0000 7f68f4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 0x7f68f0115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.141+0000 7f68f4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 0x7f68f0115be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52840/0 (socket says 172.21.15.1:52840) 2024-02-28T16:27:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.141+0000 7f68f4c00700 1 -- 172.21.15.1:0/2626848580 learned_addr learned my addr 172.21.15.1:0/2626848580 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.141+0000 7f68efe00700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68f01038f0 0x7f68f0116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.141+0000 7f68f4c00700 1 -- 172.21.15.1:0/2626848580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68f01038f0 msgr2=0x7f68f0116120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.141+0000 7f68f4c00700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68f01038f0 0x7f68f0116120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.141+0000 7f68f4c00700 1 -- 172.21.15.1:0/2626848580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e4005040 con 0x7f68f0102f50 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.141+0000 7f68f4c00700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 0x7f68f0115be0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f68e4000c00 tx=0x7f68e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.142+0000 7f68ecc00700 1 -- 172.21.15.1:0/2626848580 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68e4021070 con 0x7f68f0102f50 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.142+0000 7f68ecc00700 1 -- 172.21.15.1:0/2626848580 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68e4013dd0 con 0x7f68f0102f50 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.142+0000 7f68ecc00700 1 -- 172.21.15.1:0/2626848580 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68e400e070 con 0x7f68f0102f50 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.142+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f010fd20 con 0x7f68f0102f50 2024-02-28T16:27:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.142+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f0110200 con 0x7f68f0102f50 2024-02-28T16:27:19.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.143+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f00fc6e0 con 0x7f68f0102f50 2024-02-28T16:27:19.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.144+0000 7f68ecc00700 1 -- 172.21.15.1:0/2626848580 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f68e401c400 con 0x7f68f0102f50 2024-02-28T16:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.145+0000 7f68ecc00700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f68d80692b0 0x7f68d806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.145+0000 7f68ecc00700 1 -- 172.21.15.1:0/2626848580 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f68e4088640 con 0x7f68f0102f50 2024-02-28T16:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.146+0000 7f68efe00700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f68d80692b0 0x7f68d806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.146+0000 7f68efe00700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f68d80692b0 0x7f68d806b760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f68dc004600 tx=0x7f68dc00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:19.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.150+0000 7f68ecc00700 1 -- 172.21.15.1:0/2626848580 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f68e40584a0 con 0x7f68f0102f50 2024-02-28T16:27:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.408+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f68f01111f0 con 0x7f68f0102f50 2024-02-28T16:27:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.409+0000 7f68ecc00700 1 -- 172.21.15.1:0/2626848580 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f68e4058030 con 0x7f68f0102f50 2024-02-28T16:27:19.399 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-28T16:27:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f68d80692b0 msgr2=0x7f68d806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f68d80692b0 0x7f68d806b760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f68dc004600 tx=0x7f68dc00b040 comp rx=0 tx=0).stop 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 msgr2=0x7f68f0115be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 0x7f68f0115be0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f68e4000c00 tx=0x7f68e4009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 shutdown_connections 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f68d80692b0 0x7f68d806b760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f0102f50 0x7f68f0115be0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 --2- 172.21.15.1:0/2626848580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68f01038f0 0x7f68f0116120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 >> 172.21.15.1:0/2626848580 conn(0x7f68f00f8ae0 msgr2=0x7f68f00fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 shutdown_connections 2024-02-28T16:27:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:19.412+0000 7f68f7600700 1 -- 172.21.15.1:0/2626848580 wait complete. 2024-02-28T16:27:19.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:19 smithi066 ceph-mon[36236]: pgmap v239: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:19.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:19 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2626848580' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-28T16:27:19.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:19 smithi001 ceph-mon[28991]: pgmap v239: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:19.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:19 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2626848580' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-28T16:27:20.243 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-28T16:27:20.243 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-28T16:27:20.243 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-02-28T16:27:20.243 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-28T16:27:20.243 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:20.243 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-28T16:27:20.243 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-28T16:27:20.244 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-28T16:27:20.244 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-28T16:27:20.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:20.244 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-02-28T16:27:20.304 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-28T16:27:20.304 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-28T16:27:20.304 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-02-28T16:27:20.304 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-02-28T16:27:20.304 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-02-28T16:27:20.304 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-02-28T16:27:20.304 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-28T16:27:20.304 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41459sec preferred_lft 41459sec 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-02-28T16:27:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-02-28T16:27:20.307 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-28T16:27:20.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:20.307 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-02-28T16:27:20.307 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-02-28T16:27:20.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-02-28T16:27:20.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-02-28T16:27:20.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-02-28T16:27:20.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-28T16:27:20.307 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-02-28T16:27:20.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:20.805 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-28T16:27:20.891 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-02-28T16:27:20.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:20.895 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-02-28T16:27:20.949 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-02-28T16:27:20.949 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-28T16:27:20.949 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-02-28T16:27:20.949 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-02-28T16:27:20.949 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-28T16:27:20.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:20.950 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-02-28T16:27:20.950 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-02-28T16:27:20.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-28T16:27:20.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-28T16:27:20.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-28T16:27:20.950 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-02-28T16:27:21.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:21 smithi066 ceph-mon[36236]: pgmap v240: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:21.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:21 smithi001 ceph-mon[28991]: pgmap v240: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:21.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:21.892 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-02-28T16:27:21.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:21.952 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-02-28T16:27:22.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:22.009 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-02-28T16:27:22.010 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-02-28T16:27:22.010 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-28T16:27:22.010 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-28T16:27:22.010 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-02-28T16:27:22.168 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-28T16:27:22.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:22.169 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-02-28T16:27:22.169 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-02-28T16:27:22.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-28T16:27:22.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-28T16:27:22.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-28T16:27:22.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-28T16:27:22.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-28T16:27:22.169 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-02-28T16:27:22.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:22.458 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-02-28T16:27:22.458 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-02-28T16:27:22.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-02-28T16:27:22.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-02-28T16:27:22.458 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-02-28T16:27:22.638 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-02-28T16:27:22.638 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-28T16:27:22.638 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:22.694 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-28T16:27:22.695 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-28T16:27:22.695 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:22.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-02-28T16:27:22.845 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-02-28T16:27:22.897 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-02-28T16:27:22.897 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-02-28T16:27:22.897 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-02-28T16:27:22.897 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-02-28T16:27:22.897 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-02-28T16:27:22.897 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-02-28T16:27:22.897 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-02-28T16:27:22.898 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay 2024-02-28T16:27:22.899 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/81a6afdd779cfe195ed3a9fb29944fd9f7085c76cc7034bd62f01765fd3e7a43/merged 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/6577110bce2f82986e4dc05d63eaa00654c373c9ae5e3ec3ffdba68622cb7693/merged 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/0 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/fc98c0895d5bde0a1bba0d4da502655ac577d7a9e398ad5a37bc80e73810f414/merged 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/e6915f7e6fe5de7b366a3b7c33836101f8ab24aa66ca73feb2f567279dfa66d7/merged 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/8411f1a79c394c1efef10098de69d81e5b157daf7fb15c09636bbe573f4f3917/merged 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/68149801d45dbdc8d1de38289cfc7ea10b2a965ac14914b0eca434e8c480b930/merged 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/29a4b224c8c6c9511db19de36b5e2c33d1eab1e078e79cab345d2d0b5b7db91b/merged 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/5af5b863b17276f7b8e66da9ace4b04fe263bbe08704bd80eda185328502002e/merged 2024-02-28T16:27:22.900 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/3efbc13a96f1c27b1fdcfebd1ee15cf7aaa0cb68920c88035cc244d756b36178/merged 2024-02-28T16:27:22.901 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/cf943bd460bafd93931580f43f8b6669171d9b11ac923e1b5da5c750df28f0c5/merged 2024-02-28T16:27:22.901 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/827af50bbe632ea335f7d7d535c2c94c43f2274e473ebdcb77c52b5d0620bf4a/merged 2024-02-28T16:27:22.901 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/85deb54ca446f54a76c7d4f000d56e0299874d2384a70cfcb7bd44496f4e8e7b/merged 2024-02-28T16:27:22.901 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-02-28T16:27:22.901 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-28T16:27:22.901 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-28T16:27:22.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:22.902 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-02-28T16:27:22.952 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-02-28T16:27:22.953 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-02-28T16:27:22.960 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-02-28T16:27:23.049 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-02-28T16:27:23.101 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-28T16:27:23.102 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/81a6afdd779cfe195ed3a9fb29944fd9f7085c76cc7034bd62f01765fd3e7a43/merged 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/6577110bce2f82986e4dc05d63eaa00654c373c9ae5e3ec3ffdba68622cb7693/merged 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/0 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/fc98c0895d5bde0a1bba0d4da502655ac577d7a9e398ad5a37bc80e73810f414/merged 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/e6915f7e6fe5de7b366a3b7c33836101f8ab24aa66ca73feb2f567279dfa66d7/merged 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/8411f1a79c394c1efef10098de69d81e5b157daf7fb15c09636bbe573f4f3917/merged 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/68149801d45dbdc8d1de38289cfc7ea10b2a965ac14914b0eca434e8c480b930/merged 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/29a4b224c8c6c9511db19de36b5e2c33d1eab1e078e79cab345d2d0b5b7db91b/merged 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/5af5b863b17276f7b8e66da9ace4b04fe263bbe08704bd80eda185328502002e/merged 2024-02-28T16:27:23.103 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/3efbc13a96f1c27b1fdcfebd1ee15cf7aaa0cb68920c88035cc244d756b36178/merged 2024-02-28T16:27:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/cf943bd460bafd93931580f43f8b6669171d9b11ac923e1b5da5c750df28f0c5/merged 2024-02-28T16:27:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/827af50bbe632ea335f7d7d535c2c94c43f2274e473ebdcb77c52b5d0620bf4a/merged 2024-02-28T16:27:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/85deb54ca446f54a76c7d4f000d56e0299874d2384a70cfcb7bd44496f4e8e7b/merged 2024-02-28T16:27:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-02-28T16:27:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-28T16:27:23.104 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-28T16:27:23.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:23.105 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-02-28T16:27:23.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-02-28T16:27:23.434+0000 7f95582c4300 -1 init, newargv = 0x559b27e471b0 newargc=15 2024-02-28T16:27:23.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[76738]: starting ceph client 2024-02-28T16:27:23.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[76738]: starting fuse 2024-02-28T16:27:23.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:23 smithi066 ceph-mon[36236]: pgmap v241: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:23 smithi001 ceph-mon[28991]: pgmap v241: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:27:24.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-02-28T16:27:24.363 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-02-28T16:27:24.423 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-02-28T16:27:24.423 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-02-28T16:27:24.423 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-02-28T16:27:24.423 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-02-28T16:27:24.423 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-02-28T16:27:24.423 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-02-28T16:27:24.423 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-02-28T16:27:24.423 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-28T16:27:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/81a6afdd779cfe195ed3a9fb29944fd9f7085c76cc7034bd62f01765fd3e7a43/merged 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/6577110bce2f82986e4dc05d63eaa00654c373c9ae5e3ec3ffdba68622cb7693/merged 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/0 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/fc98c0895d5bde0a1bba0d4da502655ac577d7a9e398ad5a37bc80e73810f414/merged 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/e6915f7e6fe5de7b366a3b7c33836101f8ab24aa66ca73feb2f567279dfa66d7/merged 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/8411f1a79c394c1efef10098de69d81e5b157daf7fb15c09636bbe573f4f3917/merged 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/68149801d45dbdc8d1de38289cfc7ea10b2a965ac14914b0eca434e8c480b930/merged 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/29a4b224c8c6c9511db19de36b5e2c33d1eab1e078e79cab345d2d0b5b7db91b/merged 2024-02-28T16:27:24.425 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/5af5b863b17276f7b8e66da9ace4b04fe263bbe08704bd80eda185328502002e/merged 2024-02-28T16:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/3efbc13a96f1c27b1fdcfebd1ee15cf7aaa0cb68920c88035cc244d756b36178/merged 2024-02-28T16:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/cf943bd460bafd93931580f43f8b6669171d9b11ac923e1b5da5c750df28f0c5/merged 2024-02-28T16:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/827af50bbe632ea335f7d7d535c2c94c43f2274e473ebdcb77c52b5d0620bf4a/merged 2024-02-28T16:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/85deb54ca446f54a76c7d4f000d56e0299874d2384a70cfcb7bd44496f4e8e7b/merged 2024-02-28T16:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-02-28T16:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-28T16:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-28T16:27:24.426 INFO:teuthology.orchestra.run.smithi001.stdout:/home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:24.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:24.427 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-02-28T16:27:24.479 INFO:teuthology.orchestra.run.smithi001.stdout:147 2024-02-28T16:27:24.479 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [147] 2024-02-28T16:27:24.479 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-02-28T16:27:24.479 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-02-28T16:27:24.479 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-02-28T16:27:24.479 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-02-28T16:27:24.479 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-02-28T16:27:24.479 DEBUG:teuthology.orchestra.run.smithi001:> 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-02-28T16:27:24.480 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-02-28T16:27:24.481 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-02-28T16:27:24.666 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.76738.asok 2024-02-28T16:27:24.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-28T16:27:24.685 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76738.asok 2024-02-28T16:27:24.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:24.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76738.asok status 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "c0f02e608ed955f887728ffa27c686f52f9fd400", 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)", 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "76738", 2024-02-28T16:27:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 14524, 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 14524 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-02-28T16:27:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 612538778 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 612538778 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.14524 192.168.0.1:0/612538778", 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/612538778", 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 13, 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 45, 2024-02-28T16:27:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-02-28T16:27:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-02-28T16:27:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-02-28T16:27:24.931 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:27:24.943 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c0118718-d654-11ee-95c3-87774f69a715 -- ceph fs ls 2024-02-28T16:27:25.427 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:25.582 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:25 smithi001 ceph-mon[28991]: pgmap v242: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-02-28T16:27:25.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:25 smithi066 ceph-mon[36236]: pgmap v242: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-02-28T16:27:26.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:27:26.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:27:27.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:27 smithi001 ceph-mon[28991]: pgmap v243: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-02-28T16:27:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.820+0000 7f364c800700 1 -- 172.21.15.1:0/447239036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 msgr2=0x7f36440ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.820+0000 7f364c800700 1 --2- 172.21.15.1:0/447239036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 0x7f36440ff140 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3634004940 tx=0x7f3634009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.820+0000 7f364c800700 1 -- 172.21.15.1:0/447239036 shutdown_connections 2024-02-28T16:27:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.820+0000 7f364c800700 1 --2- 172.21.15.1:0/447239036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 0x7f36440ff140 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.821+0000 7f364c800700 1 --2- 172.21.15.1:0/447239036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3644105130 0x7f3644105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.821+0000 7f364c800700 1 -- 172.21.15.1:0/447239036 >> 172.21.15.1:0/447239036 conn(0x7f36440fad00 msgr2=0x7f36440fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.821+0000 7f364c800700 1 -- 172.21.15.1:0/447239036 shutdown_connections 2024-02-28T16:27:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.821+0000 7f364c800700 1 -- 172.21.15.1:0/447239036 wait complete. 2024-02-28T16:27:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.821+0000 7f364c800700 1 Processor -- start 2024-02-28T16:27:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.822+0000 7f364c800700 1 -- start start 2024-02-28T16:27:27.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.822+0000 7f364c800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3644105130 0x7f3644115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:27.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.822+0000 7f364c800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 0x7f36441161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:27.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.822+0000 7f364c800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644116840 con 0x7f3644105ad0 2024-02-28T16:27:27.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.822+0000 7f364c800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364410fb50 con 0x7f3644105130 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.822+0000 7f3649400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 0x7f36441161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.822+0000 7f3649400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 0x7f36441161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59736/0 (socket says 172.21.15.1:59736) 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.822+0000 7f3649400700 1 -- 172.21.15.1:0/2923464271 learned_addr learned my addr 172.21.15.1:0/2923464271 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.823+0000 7f3649e00700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3644105130 0x7f3644115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.823+0000 7f3649400700 1 -- 172.21.15.1:0/2923464271 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3644105130 msgr2=0x7f3644115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.823+0000 7f3649400700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3644105130 0x7f3644115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.823+0000 7f3649400700 1 -- 172.21.15.1:0/2923464271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3634005040 con 0x7f3644105ad0 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.823+0000 7f3649400700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 0x7f36441161f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3634009100 tx=0x7f3634009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.823+0000 7f363e000700 1 -- 172.21.15.1:0/2923464271 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363400e040 con 0x7f3644105ad0 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.824+0000 7f363e000700 1 -- 172.21.15.1:0/2923464271 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3634004290 con 0x7f3644105ad0 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.824+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364410fdd0 con 0x7f3644105ad0 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.824+0000 7f363e000700 1 -- 172.21.15.1:0/2923464271 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363401c3f0 con 0x7f3644105ad0 2024-02-28T16:27:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.824+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3644110310 con 0x7f3644105ad0 2024-02-28T16:27:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.825+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36440fe900 con 0x7f3644105ad0 2024-02-28T16:27:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.826+0000 7f363e000700 1 -- 172.21.15.1:0/2923464271 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f363400bbf0 con 0x7f3644105ad0 2024-02-28T16:27:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.827+0000 7f363e000700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f362c0692b0 0x7f362c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.827+0000 7f363e000700 1 -- 172.21.15.1:0/2923464271 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3634088460 con 0x7f3644105ad0 2024-02-28T16:27:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.827+0000 7f3649e00700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f362c0692b0 0x7f362c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.828+0000 7f3649e00700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f362c0692b0 0x7f362c06b760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3638007fd0 tx=0x7f3638009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:27.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:27.832+0000 7f363e000700 1 -- 172.21.15.1:0/2923464271 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3634058200 con 0x7f3644105ad0 2024-02-28T16:27:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:27 smithi066 ceph-mon[36236]: pgmap v243: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-02-28T16:27:28.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.097+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f364404bed0 con 0x7f3644105ad0 2024-02-28T16:27:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.098+0000 7f363e000700 1 -- 172.21.15.1:0/2923464271 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f3634057d90 con 0x7f3644105ad0 2024-02-28T16:27:28.088 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-28T16:27:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.101+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f362c0692b0 msgr2=0x7f362c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.101+0000 7f364c800700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f362c0692b0 0x7f362c06b760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3638007fd0 tx=0x7f3638009040 comp rx=0 tx=0).stop 2024-02-28T16:27:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.101+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 msgr2=0x7f36441161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.101+0000 7f364c800700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 0x7f36441161f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3634009100 tx=0x7f3634009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.102+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 shutdown_connections 2024-02-28T16:27:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.102+0000 7f364c800700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f362c0692b0 0x7f362c06b760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.102+0000 7f364c800700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3644105130 0x7f3644115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.102+0000 7f364c800700 1 --2- 172.21.15.1:0/2923464271 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3644105ad0 0x7f36441161f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.102+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 >> 172.21.15.1:0/2923464271 conn(0x7f36440fad00 msgr2=0x7f36440fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.102+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 shutdown_connections 2024-02-28T16:27:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:28.102+0000 7f364c800700 1 -- 172.21.15.1:0/2923464271 wait complete. 2024-02-28T16:27:28.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:28 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2923464271' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi066.front.sepia.ceph.com 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-28T16:27:28.924 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-28T16:27:28.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:28.925 DEBUG:teuthology.orchestra.run.smithi066:> ip addr 2024-02-28T16:27:28.987 INFO:teuthology.orchestra.run.smithi066.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: inet 127.0.0.1/8 scope host lo 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: valid_lft forever preferred_lft forever 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: inet6 ::1/128 scope host 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: valid_lft forever preferred_lft forever 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: link/ether 0c:c4:7a:a8:b4:84 brd ff:ff:ff:ff:ff:ff 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: altname enp7s0f0 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: link/ether 0c:c4:7a:a8:b4:85 brd ff:ff:ff:ff:ff:ff 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout: altname enp7s0f1 2024-02-28T16:27:28.988 INFO:teuthology.orchestra.run.smithi066.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-28T16:27:28.989 INFO:teuthology.orchestra.run.smithi066.stdout: link/ether 0c:c4:7a:b6:f8:bc brd ff:ff:ff:ff:ff:ff 2024-02-28T16:27:28.989 INFO:teuthology.orchestra.run.smithi066.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-28T16:27:28.989 INFO:teuthology.orchestra.run.smithi066.stdout: link/ether 0c:c4:7a:b6:f8:bd brd ff:ff:ff:ff:ff:ff 2024-02-28T16:27:28.989 INFO:teuthology.orchestra.run.smithi066.stdout: inet 172.21.15.66/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-28T16:27:28.989 INFO:teuthology.orchestra.run.smithi066.stdout: valid_lft 41885sec preferred_lft 41885sec 2024-02-28T16:27:28.989 INFO:teuthology.orchestra.run.smithi066.stdout: inet6 fe80::ec4:7aff:feb6:f8bd/64 scope link 2024-02-28T16:27:28.989 INFO:teuthology.orchestra.run.smithi066.stdout: valid_lft forever preferred_lft forever 2024-02-28T16:27:28.990 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-28T16:27:28.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:28.990 DEBUG:teuthology.orchestra.run.smithi066:> (cd / && exec bash -c ' 2024-02-28T16:27:28.991 DEBUG:teuthology.orchestra.run.smithi066:> set -e 2024-02-28T16:27:28.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip link add name ceph-brx type bridge 2024-02-28T16:27:28.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip addr flush dev ceph-brx 2024-02-28T16:27:28.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip link set ceph-brx up 2024-02-28T16:27:28.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-28T16:27:28.991 DEBUG:teuthology.orchestra.run.smithi066:> ') 2024-02-28T16:27:28.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:28 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2923464271' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-28T16:27:29.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:29.475 DEBUG:teuthology.orchestra.run.smithi066:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-28T16:27:29.558 INFO:teuthology.orchestra.run.smithi066.stdout:1 2024-02-28T16:27:29.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:29.561 DEBUG:teuthology.orchestra.run.smithi066:> route 2024-02-28T16:27:29.608 INFO:teuthology.orchestra.run.smithi066.stdout:Kernel IP routing table 2024-02-28T16:27:29.608 INFO:teuthology.orchestra.run.smithi066.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-28T16:27:29.609 INFO:teuthology.orchestra.run.smithi066.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-28T16:27:29.609 INFO:teuthology.orchestra.run.smithi066.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-28T16:27:29.609 INFO:teuthology.orchestra.run.smithi066.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-28T16:27:29.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:29.609 DEBUG:teuthology.orchestra.run.smithi066:> (cd / && exec bash -c ' 2024-02-28T16:27:29.609 DEBUG:teuthology.orchestra.run.smithi066:> set -e 2024-02-28T16:27:29.609 DEBUG:teuthology.orchestra.run.smithi066:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-28T16:27:29.609 DEBUG:teuthology.orchestra.run.smithi066:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-28T16:27:29.609 DEBUG:teuthology.orchestra.run.smithi066:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-28T16:27:29.610 DEBUG:teuthology.orchestra.run.smithi066:> ') 2024-02-28T16:27:29.686 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:29 smithi066 ceph-mon[36236]: pgmap v244: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:27:30.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:29 smithi001 ceph-mon[28991]: pgmap v244: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:27:30.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:30.331 DEBUG:teuthology.orchestra.run.smithi066:> ip netns list 2024-02-28T16:27:30.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:30.385 DEBUG:teuthology.orchestra.run.smithi066:> ip netns list-id 2024-02-28T16:27:30.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:30.431 DEBUG:teuthology.orchestra.run.smithi066:> (cd / && exec bash -c ' 2024-02-28T16:27:30.432 DEBUG:teuthology.orchestra.run.smithi066:> set -e 2024-02-28T16:27:30.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-28T16:27:30.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-28T16:27:30.432 DEBUG:teuthology.orchestra.run.smithi066:> ') 2024-02-28T16:27:30.624 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-28T16:27:30.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:30.625 DEBUG:teuthology.orchestra.run.smithi066:> (cd / && exec bash -c ' 2024-02-28T16:27:30.625 DEBUG:teuthology.orchestra.run.smithi066:> set -e 2024-02-28T16:27:30.625 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-28T16:27:30.625 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-28T16:27:30.625 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-28T16:27:30.625 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-28T16:27:30.625 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-28T16:27:30.625 DEBUG:teuthology.orchestra.run.smithi066:> ') 2024-02-28T16:27:30.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:30.929 DEBUG:teuthology.orchestra.run.smithi066:> (cd / && exec bash -c ' 2024-02-28T16:27:30.929 DEBUG:teuthology.orchestra.run.smithi066:> set -e 2024-02-28T16:27:30.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip link set brx.0 up 2024-02-28T16:27:30.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo ip link set dev brx.0 master ceph-brx 2024-02-28T16:27:30.930 DEBUG:teuthology.orchestra.run.smithi066:> ') 2024-02-28T16:27:31.110 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-02-28T16:27:31.110 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-28T16:27:31.110 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:31.153 INFO:teuthology.orchestra.run.smithi066.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-28T16:27:31.154 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-28T16:27:31.154 DEBUG:teuthology.orchestra.run.smithi066:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:31.247 DEBUG:teuthology.orchestra.run.smithi066:> sudo modprobe fuse 2024-02-28T16:27:31.344 DEBUG:teuthology.orchestra.run.smithi066:> cat /proc/self/mounts | awk '{print $2}' 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/proc 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/sys 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/dev 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/security 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/shm 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/pts 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/run 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/cgroup 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/pstore 2024-02-28T16:27:31.470 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/bpf 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/ 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/selinux 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/mqueue 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/hugepages 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/debug 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/tracing 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/fuse/connections 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/config 2024-02-28T16:27:31.471 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-sysctl.service 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/nfs/rpc_pipefs 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/run/user/1000 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/ceph 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/run/user/0 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/b6942e15b129bfdafa5711726c2812c156882f840483caff8e9f59de64e46131/merged 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/1e9417cd3d88a42e13a231f98d5e0f531d0ef0813f7b5cae58c4d4dda1eb434c/merged 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/5f0b292e699314dfbbfcb6bbe0dc9e191d7a012d300509e1b37d933fd7adadcd/merged 2024-02-28T16:27:31.472 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/3e2f4e27a4ee1d909f8fa0250bf0fcfc803f28dced874ff96b7bf481c5deed66/merged 2024-02-28T16:27:31.473 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/4d822b6ca9e36576a5a985dbb021b8185546ed26c70f2b17658608a196c6cac0/merged 2024-02-28T16:27:31.473 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/81bba177f11393b3c0fbb660fdfdb25829116cf22a16242fdbb8395c4d503675/merged 2024-02-28T16:27:31.473 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/4f525fe10b3d69dfa3e2d39ddedc5df5575d5c6ee897f4f92a55cbf2e68a1962/merged 2024-02-28T16:27:31.473 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/b6f8b14b476f1f17a110013fbf13d7ebf2effbf7938cbf0b829a58e1ca41a098/merged 2024-02-28T16:27:31.473 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/c288402d07ea9aeaee9674ecc5826d17120c1ae2da53f7df5df401ff7b135cc9/merged 2024-02-28T16:27:31.473 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns 2024-02-28T16:27:31.473 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-28T16:27:31.473 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-28T16:27:31.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:31.474 DEBUG:teuthology.orchestra.run.smithi066:> ls /sys/fs/fuse/connections 2024-02-28T16:27:31.531 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-02-28T16:27:31.531 DEBUG:teuthology.orchestra.run.smithi066:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-02-28T16:27:31.540 DEBUG:teuthology.orchestra.run.smithi066:> sudo modprobe fuse 2024-02-28T16:27:31.611 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:31 smithi066 ceph-mon[36236]: pgmap v245: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:27:31.624 DEBUG:teuthology.orchestra.run.smithi066:> cat /proc/self/mounts | awk '{print $2}' 2024-02-28T16:27:31.671 INFO:teuthology.orchestra.run.smithi066.stdout:/proc 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/sys 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/dev 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/security 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/shm 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/pts 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/run 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/cgroup 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/pstore 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/bpf 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/ 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/selinux 2024-02-28T16:27:31.672 INFO:teuthology.orchestra.run.smithi066.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/mqueue 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/hugepages 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/debug 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/tracing 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/fuse/connections 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/config 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-sysctl.service 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/nfs/rpc_pipefs 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/run/user/1000 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/ceph 2024-02-28T16:27:31.673 INFO:teuthology.orchestra.run.smithi066.stdout:/run/user/0 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/b6942e15b129bfdafa5711726c2812c156882f840483caff8e9f59de64e46131/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/1e9417cd3d88a42e13a231f98d5e0f531d0ef0813f7b5cae58c4d4dda1eb434c/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/5f0b292e699314dfbbfcb6bbe0dc9e191d7a012d300509e1b37d933fd7adadcd/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/3e2f4e27a4ee1d909f8fa0250bf0fcfc803f28dced874ff96b7bf481c5deed66/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/4d822b6ca9e36576a5a985dbb021b8185546ed26c70f2b17658608a196c6cac0/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/81bba177f11393b3c0fbb660fdfdb25829116cf22a16242fdbb8395c4d503675/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/4f525fe10b3d69dfa3e2d39ddedc5df5575d5c6ee897f4f92a55cbf2e68a1962/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/b6f8b14b476f1f17a110013fbf13d7ebf2effbf7938cbf0b829a58e1ca41a098/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/c288402d07ea9aeaee9674ecc5826d17120c1ae2da53f7df5df401ff7b135cc9/merged 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-28T16:27:31.674 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-28T16:27:31.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:31.675 DEBUG:teuthology.orchestra.run.smithi066:> ls /sys/fs/fuse/connections 2024-02-28T16:27:31.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi066.stderr:2024-02-28T16:27:31.817+0000 7f1717cc1300 -1 init, newargv = 0x565142ae4330 newargc=15 2024-02-28T16:27:31.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi066.stderr:ceph-fuse[61458]: starting ceph client 2024-02-28T16:27:31.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi066.stderr:ceph-fuse[61458]: starting fuse 2024-02-28T16:27:32.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:31 smithi001 ceph-mon[28991]: pgmap v245: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:27:32.824 DEBUG:teuthology.orchestra.run.smithi066:> sudo modprobe fuse 2024-02-28T16:27:32.941 DEBUG:teuthology.orchestra.run.smithi066:> cat /proc/self/mounts | awk '{print $2}' 2024-02-28T16:27:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:/proc 2024-02-28T16:27:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:/sys 2024-02-28T16:27:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:/dev 2024-02-28T16:27:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/security 2024-02-28T16:27:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/shm 2024-02-28T16:27:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/pts 2024-02-28T16:27:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:/run 2024-02-28T16:27:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/cgroup 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/pstore 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/bpf 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/ 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/selinux 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/mqueue 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/hugepages 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/debug 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/tracing 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/fs/fuse/connections 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/sys/kernel/config 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-sysctl.service 2024-02-28T16:27:33.004 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/nfs/rpc_pipefs 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/run/user/1000 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/ceph 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/run/user/0 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/proc/sys/fs/binfmt_misc 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/b6942e15b129bfdafa5711726c2812c156882f840483caff8e9f59de64e46131/merged 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/1e9417cd3d88a42e13a231f98d5e0f531d0ef0813f7b5cae58c4d4dda1eb434c/merged 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/5f0b292e699314dfbbfcb6bbe0dc9e191d7a012d300509e1b37d933fd7adadcd/merged 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/3e2f4e27a4ee1d909f8fa0250bf0fcfc803f28dced874ff96b7bf481c5deed66/merged 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/4d822b6ca9e36576a5a985dbb021b8185546ed26c70f2b17658608a196c6cac0/merged 2024-02-28T16:27:33.005 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/81bba177f11393b3c0fbb660fdfdb25829116cf22a16242fdbb8395c4d503675/merged 2024-02-28T16:27:33.006 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/4f525fe10b3d69dfa3e2d39ddedc5df5575d5c6ee897f4f92a55cbf2e68a1962/merged 2024-02-28T16:27:33.006 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/b6f8b14b476f1f17a110013fbf13d7ebf2effbf7938cbf0b829a58e1ca41a098/merged 2024-02-28T16:27:33.006 INFO:teuthology.orchestra.run.smithi066.stdout:/var/lib/containers/storage/overlay/c288402d07ea9aeaee9674ecc5826d17120c1ae2da53f7df5df401ff7b135cc9/merged 2024-02-28T16:27:33.006 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns 2024-02-28T16:27:33.006 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-28T16:27:33.006 INFO:teuthology.orchestra.run.smithi066.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-28T16:27:33.006 INFO:teuthology.orchestra.run.smithi066.stdout:/home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:33.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:33.007 DEBUG:teuthology.orchestra.run.smithi066:> ls /sys/fs/fuse/connections 2024-02-28T16:27:33.063 INFO:teuthology.orchestra.run.smithi066.stdout:114 2024-02-28T16:27:33.064 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo stdin-killer -- python3 -c ' 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> import glob 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> import re 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> import os 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> import subprocess 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> def _find_admin_socket(client_name): 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-02-28T16:27:33.064 DEBUG:teuthology.orchestra.run.smithi066:> files = glob.glob(asok_path) 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> # Given a non-glob path, it better be there 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> if "*" not in asok_path: 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> assert(len(files) == 1) 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> return files[0] 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> for f in files: 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> if os.path.exists("/proc/{0}".format(pid)): 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> contents = proc_f.read() 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> if mountpoint in contents: 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> return f 2024-02-28T16:27:33.065 DEBUG:teuthology.orchestra.run.smithi066:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-02-28T16:27:33.066 DEBUG:teuthology.orchestra.run.smithi066:> 2024-02-28T16:27:33.066 DEBUG:teuthology.orchestra.run.smithi066:> print(_find_admin_socket("client.1")) 2024-02-28T16:27:33.066 DEBUG:teuthology.orchestra.run.smithi066:> ' 2024-02-28T16:27:33.258 INFO:teuthology.orchestra.run.smithi066.stdout:/var/run/ceph/ceph-client.1.61458.asok 2024-02-28T16:27:33.263 INFO:teuthology.orchestra.run.smithi066.stderr:2024-02-28T16:27:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-28T16:27:33.277 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.61458.asok 2024-02-28T16:27:33.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:33.277 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.61458.asok status 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout: "metadata": { 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_sha1": "c0f02e608ed955f887728ffa27c686f52f9fd400", 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_version": "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)", 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout: "entity_id": "1", 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout: "hostname": "smithi066", 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout: "pid": "61458", 2024-02-28T16:27:33.521 INFO:teuthology.orchestra.run.smithi066.stdout: "root": "/" 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "dentry_count": 0, 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "dentry_pinned_count": 0, 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "id": 24299, 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "inst": { 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "name": { 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "client", 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "num": 24299 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": { 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "192.168.0.1:0", 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 790488749 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-02-28T16:27:33.522 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": { 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "192.168.0.1:0", 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 790488749 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "inst_str": "client.24299 192.168.0.1:0/790488749", 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "addr_str": "192.168.0.1:0/790488749", 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "inode_count": 1, 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_epoch": 13, 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_epoch": 45, 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_epoch_barrier": 0, 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklisted": false, 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout: "fs_name": "cephfs" 2024-02-28T16:27:33.523 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-02-28T16:27:33.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:33.537 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-02-28T16:27:33.537 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:33.599 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-02-28T16:27:33.599 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:33.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:33.599 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:33.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:33.718 DEBUG:teuthology.orchestra.run.smithi066:> stat --file-system '--printf=%T 2024-02-28T16:27:33.718 DEBUG:teuthology.orchestra.run.smithi066:> ' -- /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:33.781 INFO:teuthology.orchestra.run.smithi066.stdout:fuseblk 2024-02-28T16:27:33.781 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:33.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T16:27:33.781 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:33.873 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:33 smithi066 ceph-mon[36236]: pgmap v246: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:27:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:33 smithi001 ceph-mon[28991]: pgmap v246: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:27:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:35 smithi066 ceph-mon[36236]: pgmap v247: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-28T16:27:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:35 smithi001 ceph-mon[28991]: pgmap v247: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-28T16:27:37.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:37 smithi001 ceph-mon[28991]: pgmap v248: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-02-28T16:27:37.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:37 smithi066 ceph-mon[36236]: pgmap v248: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-02-28T16:27:38.607 INFO:teuthology.run_tasks:Running task print... 2024-02-28T16:27:38.616 INFO:teuthology.task.print:**** done client 2024-02-28T16:27:38.617 INFO:teuthology.run_tasks:Running task parallel... 2024-02-28T16:27:38.627 INFO:teuthology.task.parallel:starting parallel... 2024-02-28T16:27:38.627 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-28T16:27:38.628 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-28T16:27:38.628 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:27:38.628 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-28T16:27:38.629 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-28T16:27:38.629 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-28T16:27:38.633 INFO:tasks.workunit:Pulling workunits from ref 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:27:38.633 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-28T16:27:38.633 INFO:tasks.workunit:timeout=3h 2024-02-28T16:27:38.633 INFO:tasks.workunit:cleanup=True 2024-02-28T16:27:38.633 DEBUG:teuthology.orchestra.run.smithi001:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:38.694 INFO:teuthology.orchestra.run.smithi001.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:38.694 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-02-28T16:27:38.694 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 93h/147d Inode: 1 Links: 2 2024-02-28T16:27:38.694 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:27:38.695 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:fusefs_t:s0 2024-02-28T16:27:38.695 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-28T16:27:38.695 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-02-28 16:26:43.856322042 +0000 2024-02-28T16:27:38.695 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-02-28 16:27:33.717715632 +0000 2024-02-28T16:27:38.695 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-02-28T16:27:38.695 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-28T16:27:38.695 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-28T16:27:38.854 DEBUG:teuthology.orchestra.run.smithi066:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:38.920 INFO:teuthology.orchestra.run.smithi066.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:38.920 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-02-28T16:27:38.920 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 72h/114d Inode: 1 Links: 3 2024-02-28T16:27:38.920 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-28T16:27:38.921 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:fusefs_t:s0 2024-02-28T16:27:38.921 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-28T16:27:38.921 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-02-28 16:27:38.846073604 +0000 2024-02-28T16:27:38.921 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-02-28 16:27:38.846073604 +0000 2024-02-28T16:27:38.921 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2024-02-28T16:27:38.921 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-28T16:27:38.921 DEBUG:teuthology.orchestra.run.smithi066:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-28T16:27:39.042 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:27:39.043 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:27:39.086 INFO:tasks.workunit.client.0.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-28T16:27:39.106 INFO:tasks.workunit.client.1.smithi066.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-28T16:27:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:39.622 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:39 smithi001 ceph-mon[28991]: pgmap v249: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-02-28T16:27:39.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:39 smithi066 ceph-mon[36236]: pgmap v249: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-02-28T16:27:41.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.423+0000 7f2bda400700 1 -- 172.21.15.1:0/475144667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd40708c0 msgr2=0x7f2bd4070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.423+0000 7f2bda400700 1 --2- 172.21.15.1:0/475144667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd40708c0 0x7f2bd4070d10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc004940 tx=0x7f2bbc009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.423+0000 7f2bda400700 1 -- 172.21.15.1:0/475144667 shutdown_connections 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.423+0000 7f2bda400700 1 --2- 172.21.15.1:0/475144667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd40708c0 0x7f2bd4070d10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.423+0000 7f2bda400700 1 --2- 172.21.15.1:0/475144667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2bd4071f10 0x7f2bd4070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.423+0000 7f2bda400700 1 -- 172.21.15.1:0/475144667 >> 172.21.15.1:0/475144667 conn(0x7f2bd40f8620 msgr2=0x7f2bd40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.423+0000 7f2bda400700 1 -- 172.21.15.1:0/475144667 shutdown_connections 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.423+0000 7f2bda400700 1 -- 172.21.15.1:0/475144667 wait complete. 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.424+0000 7f2bda400700 1 Processor -- start 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.424+0000 7f2bda400700 1 -- start start 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.425+0000 7f2bda400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2bd40708c0 0x7f2bd410fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.425+0000 7f2bda400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd4071f10 0x7f2bd41100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.425+0000 7f2bda400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41106b0 con 0x7f2bd4071f10 2024-02-28T16:27:41.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.425+0000 7f2bda400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4110f50 con 0x7f2bd40708c0 2024-02-28T16:27:41.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.425+0000 7f2bcbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd4071f10 0x7f2bd41100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.425+0000 7f2bcbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd4071f10 0x7f2bd41100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57664/0 (socket says 172.21.15.1:57664) 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.425+0000 7f2bcbe00700 1 -- 172.21.15.1:0/4002280178 learned_addr learned my addr 172.21.15.1:0/4002280178 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.425+0000 7f2bd3400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2bd40708c0 0x7f2bd410fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.426+0000 7f2bcbe00700 1 -- 172.21.15.1:0/4002280178 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2bd40708c0 msgr2=0x7f2bd410fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.426+0000 7f2bcbe00700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2bd40708c0 0x7f2bd410fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.426+0000 7f2bcbe00700 1 -- 172.21.15.1:0/4002280178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc005040 con 0x7f2bd4071f10 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.426+0000 7f2bcbe00700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd4071f10 0x7f2bd41100b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009fd0 tx=0x7f2bbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.426+0000 7f2bcb400700 1 -- 172.21.15.1:0/4002280178 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bbc00e040 con 0x7f2bd4071f10 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.426+0000 7f2bcb400700 1 -- 172.21.15.1:0/4002280178 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bbc004290 con 0x7f2bd4071f10 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.426+0000 7f2bcb400700 1 -- 172.21.15.1:0/4002280178 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bbc01c630 con 0x7f2bd4071f10 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.426+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd41111d0 con 0x7f2bd4071f10 2024-02-28T16:27:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.427+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd4111710 con 0x7f2bd4071f10 2024-02-28T16:27:41.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.428+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd404bed0 con 0x7f2bd4071f10 2024-02-28T16:27:41.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.429+0000 7f2bcb400700 1 -- 172.21.15.1:0/4002280178 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2bbc013420 con 0x7f2bd4071f10 2024-02-28T16:27:41.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.430+0000 7f2bcb400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2bb80692b0 0x7f2bb806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:41.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.430+0000 7f2bcb400700 1 -- 172.21.15.1:0/4002280178 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2bbc088390 con 0x7f2bd4071f10 2024-02-28T16:27:41.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.430+0000 7f2bd3400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2bb80692b0 0x7f2bb806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:41.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.431+0000 7f2bd3400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2bb80692b0 0x7f2bb806b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4004990 tx=0x7f2bc4009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.435+0000 7f2bcb400700 1 -- 172.21.15.1:0/4002280178 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2bbc0580b0 con 0x7f2bd4071f10 2024-02-28T16:27:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.655+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f2bd4107c10 con 0x7f2bd4071f10 2024-02-28T16:27:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.656+0000 7f2bcb400700 1 -- 172.21.15.1:0/4002280178 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f2bbc057c40 con 0x7f2bd4071f10 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2bb80692b0 msgr2=0x7f2bb806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2bb80692b0 0x7f2bb806b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4004990 tx=0x7f2bc4009ee0 comp rx=0 tx=0).stop 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd4071f10 msgr2=0x7f2bd41100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd4071f10 0x7f2bd41100b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009fd0 tx=0x7f2bbc009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 shutdown_connections 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2bb80692b0 0x7f2bb806b760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2bd40708c0 0x7f2bd410fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 --2- 172.21.15.1:0/4002280178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bd4071f10 0x7f2bd41100b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 >> 172.21.15.1:0/4002280178 conn(0x7f2bd40f8620 msgr2=0x7f2bd4102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 shutdown_connections 2024-02-28T16:27:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:41.659+0000 7f2bda400700 1 -- 172.21.15.1:0/4002280178 wait complete. 2024-02-28T16:27:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:41 smithi066 ceph-mon[36236]: pgmap v250: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 852 B/s wr, 0 op/s 2024-02-28T16:27:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:27:42.003 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:41 smithi001 ceph-mon[28991]: pgmap v250: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 852 B/s wr, 0 op/s 2024-02-28T16:27:42.003 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:27:42.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-28T16:27:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:43.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:43 smithi001 ceph-mon[28991]: pgmap v251: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 852 B/s wr, 0 op/s 2024-02-28T16:27:43.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:43 smithi066 ceph-mon[36236]: pgmap v251: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 852 B/s wr, 0 op/s 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.309+0000 7f5a91800700 1 -- 172.21.15.1:0/3691096915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c104750 msgr2=0x7f5a8c104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.309+0000 7f5a91800700 1 --2- 172.21.15.1:0/3691096915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c104750 0x7f5a8c104ba0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5a78004940 tx=0x7f5a78009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.309+0000 7f5a91800700 1 -- 172.21.15.1:0/3691096915 shutdown_connections 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.309+0000 7f5a91800700 1 --2- 172.21.15.1:0/3691096915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c104750 0x7f5a8c104ba0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.309+0000 7f5a91800700 1 --2- 172.21.15.1:0/3691096915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a8c103540 0x7f5a8c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.309+0000 7f5a91800700 1 -- 172.21.15.1:0/3691096915 >> 172.21.15.1:0/3691096915 conn(0x7f5a8c0727d0 msgr2=0x7f5a8c074c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.309+0000 7f5a91800700 1 -- 172.21.15.1:0/3691096915 shutdown_connections 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.309+0000 7f5a91800700 1 -- 172.21.15.1:0/3691096915 wait complete. 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.310+0000 7f5a91800700 1 Processor -- start 2024-02-28T16:27:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.310+0000 7f5a91800700 1 -- start start 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.310+0000 7f5a91800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c103540 0x7f5a8c113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a91800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a8c104750 0x7f5a8c1144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a91800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c114aa0 con 0x7f5a8c103540 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a91800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c115340 con 0x7f5a8c104750 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a8aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c103540 0x7f5a8c113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a8aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c103540 0x7f5a8c113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34020/0 (socket says 172.21.15.1:34020) 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a8aa00700 1 -- 172.21.15.1:0/3006497791 learned_addr learned my addr 172.21.15.1:0/3006497791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a8aa00700 1 -- 172.21.15.1:0/3006497791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a8c104750 msgr2=0x7f5a8c1144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a8aa00700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a8c104750 0x7f5a8c1144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.311+0000 7f5a8aa00700 1 -- 172.21.15.1:0/3006497791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a78005040 con 0x7f5a8c103540 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.312+0000 7f5a8aa00700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c103540 0x7f5a8c113f60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c002a80 tx=0x7f5a7c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.312+0000 7f5a82a00700 1 -- 172.21.15.1:0/3006497791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a7c007cc0 con 0x7f5a8c103540 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.312+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8c115620 con 0x7f5a8c103540 2024-02-28T16:27:45.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.312+0000 7f5a82a00700 1 -- 172.21.15.1:0/3006497791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a7c00f040 con 0x7f5a8c103540 2024-02-28T16:27:45.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.312+0000 7f5a82a00700 1 -- 172.21.15.1:0/3006497791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a7c00dad0 con 0x7f5a8c103540 2024-02-28T16:27:45.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.312+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8c115b60 con 0x7f5a8c103540 2024-02-28T16:27:45.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.313+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8c04bed0 con 0x7f5a8c103540 2024-02-28T16:27:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.314+0000 7f5a82a00700 1 -- 172.21.15.1:0/3006497791 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5a7c00dc30 con 0x7f5a8c103540 2024-02-28T16:27:45.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.315+0000 7f5a82a00700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f5a700692b0 0x7f5a7006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:45.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.315+0000 7f5a82a00700 1 -- 172.21.15.1:0/3006497791 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5a7c086fe0 con 0x7f5a8c103540 2024-02-28T16:27:45.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.315+0000 7f5a8a000700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f5a700692b0 0x7f5a7006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:45.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.316+0000 7f5a8a000700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f5a700692b0 0x7f5a7006b760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a78004f90 tx=0x7f5a78009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:45.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.320+0000 7f5a82a00700 1 -- 172.21.15.1:0/3006497791 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5a7c056c80 con 0x7f5a8c103540 2024-02-28T16:27:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.532+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f5a8c116160 con 0x7f5a8c103540 2024-02-28T16:27:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.533+0000 7f5a82a00700 1 -- 172.21.15.1:0/3006497791 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f5a7c056810 con 0x7f5a8c103540 2024-02-28T16:27:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f5a700692b0 msgr2=0x7f5a7006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f5a700692b0 0x7f5a7006b760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a78004f90 tx=0x7f5a78009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c103540 msgr2=0x7f5a8c113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c103540 0x7f5a8c113f60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c002a80 tx=0x7f5a7c004600 comp rx=0 tx=0).stop 2024-02-28T16:27:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 shutdown_connections 2024-02-28T16:27:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f5a700692b0 0x7f5a7006b760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c103540 0x7f5a8c113f60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 --2- 172.21.15.1:0/3006497791 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a8c104750 0x7f5a8c1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.536+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 >> 172.21.15.1:0/3006497791 conn(0x7f5a8c0727d0 msgr2=0x7f5a8c107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.537+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 shutdown_connections 2024-02-28T16:27:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:45.537+0000 7f5a91800700 1 -- 172.21.15.1:0/3006497791 wait complete. 2024-02-28T16:27:45.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:45 smithi066 ceph-mon[36236]: pgmap v252: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 1 op/s 2024-02-28T16:27:45.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:45 smithi001 ceph-mon[28991]: pgmap v252: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 1 op/s 2024-02-28T16:27:46.378 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-28T16:27:46.912 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:47.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:47 smithi001 ceph-mon[28991]: pgmap v253: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-28T16:27:47.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:47 smithi066 ceph-mon[36236]: pgmap v253: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-28T16:27:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.245+0000 7f29afe00700 1 -- 172.21.15.1:0/1967556542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 msgr2=0x7f29b00fd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.245+0000 7f29afe00700 1 --2- 172.21.15.1:0/1967556542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 0x7f29b00fd2a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f299c004940 tx=0x7f299c009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.245+0000 7f29afe00700 1 -- 172.21.15.1:0/1967556542 shutdown_connections 2024-02-28T16:27:49.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.245+0000 7f29afe00700 1 --2- 172.21.15.1:0/1967556542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 0x7f29b00fd2a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:49.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.245+0000 7f29afe00700 1 --2- 172.21.15.1:0/1967556542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b00fbc40 0x7f29b00fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:49.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.245+0000 7f29afe00700 1 -- 172.21.15.1:0/1967556542 >> 172.21.15.1:0/1967556542 conn(0x7f29b00f72c0 msgr2=0x7f29b00f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:49.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.245+0000 7f29afe00700 1 -- 172.21.15.1:0/1967556542 shutdown_connections 2024-02-28T16:27:49.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.245+0000 7f29afe00700 1 -- 172.21.15.1:0/1967556542 wait complete. 2024-02-28T16:27:49.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.246+0000 7f29afe00700 1 Processor -- start 2024-02-28T16:27:49.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.247+0000 7f29afe00700 1 -- start start 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.247+0000 7f29afe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b00fbc40 0x7f29b010f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.247+0000 7f29afe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 0x7f29b010fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.247+0000 7f29afe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b01104c0 con 0x7f29b00fce50 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.247+0000 7f29afe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0110d60 con 0x7f29b00fbc40 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.247+0000 7f29ae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 0x7f29b010fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.247+0000 7f29ae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 0x7f29b010fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34034/0 (socket says 172.21.15.1:34034) 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.248+0000 7f29ae000700 1 -- 172.21.15.1:0/1214655785 learned_addr learned my addr 172.21.15.1:0/1214655785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.248+0000 7f29aea00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b00fbc40 0x7f29b010f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:49.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.248+0000 7f29ae000700 1 -- 172.21.15.1:0/1214655785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b00fbc40 msgr2=0x7f29b010f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.248+0000 7f29ae000700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b00fbc40 0x7f29b010f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.248+0000 7f29ae000700 1 -- 172.21.15.1:0/1214655785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c005040 con 0x7f29b00fce50 2024-02-28T16:27:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.248+0000 7f29ae000700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 0x7f29b010fec0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f299c0093b0 tx=0x7f299c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.249+0000 7f29a6a00700 1 -- 172.21.15.1:0/1214655785 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f299c00e040 con 0x7f29b00fce50 2024-02-28T16:27:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.249+0000 7f29a6a00700 1 -- 172.21.15.1:0/1214655785 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f299c00bcc0 con 0x7f29b00fce50 2024-02-28T16:27:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.249+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b0110fe0 con 0x7f29b00fce50 2024-02-28T16:27:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.249+0000 7f29a6a00700 1 -- 172.21.15.1:0/1214655785 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f299c01c630 con 0x7f29b00fce50 2024-02-28T16:27:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.249+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b0111520 con 0x7f29b00fce50 2024-02-28T16:27:49.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.250+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b004bed0 con 0x7f29b00fce50 2024-02-28T16:27:49.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.251+0000 7f29a6a00700 1 -- 172.21.15.1:0/1214655785 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f299c0139f0 con 0x7f29b00fce50 2024-02-28T16:27:49.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.252+0000 7f29a6a00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29940692a0 0x7f299406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:49.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.252+0000 7f29a6a00700 1 -- 172.21.15.1:0/1214655785 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f299c088550 con 0x7f29b00fce50 2024-02-28T16:27:49.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.252+0000 7f29aea00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29940692a0 0x7f299406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:49.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.253+0000 7f29aea00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29940692a0 0x7f299406b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f29a0007fd0 tx=0x7f29a0009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:49.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.257+0000 7f29a6a00700 1 -- 172.21.15.1:0/1214655785 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f299c0583b0 con 0x7f29b00fce50 2024-02-28T16:27:49.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.470+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f29b0104720 con 0x7f29b00fce50 2024-02-28T16:27:49.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.471+0000 7f29a6a00700 1 -- 172.21.15.1:0/1214655785 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f299c057f40 con 0x7f29b00fce50 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.473+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29940692a0 msgr2=0x7f299406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.473+0000 7f29afe00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29940692a0 0x7f299406b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f29a0007fd0 tx=0x7f29a0009040 comp rx=0 tx=0).stop 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 msgr2=0x7f29b010fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 0x7f29b010fec0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f299c0093b0 tx=0x7f299c009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 shutdown_connections 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f29940692a0 0x7f299406b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29b00fbc40 0x7f29b010f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 --2- 172.21.15.1:0/1214655785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29b00fce50 0x7f29b010fec0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 >> 172.21.15.1:0/1214655785 conn(0x7f29b00f72c0 msgr2=0x7f29b00656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 shutdown_connections 2024-02-28T16:27:49.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:49.474+0000 7f29afe00700 1 -- 172.21.15.1:0/1214655785 wait complete. 2024-02-28T16:27:49.873 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:49 smithi001 ceph-mon[28991]: pgmap v254: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-28T16:27:49.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:49 smithi066 ceph-mon[36236]: pgmap v254: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-28T16:27:50.286 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-02-28T16:27:50.792 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:27:51.934 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:51 smithi001 ceph-mon[28991]: pgmap v255: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-02-28T16:27:51.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:51 smithi066 ceph-mon[36236]: pgmap v255: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-02-28T16:27:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.786+0000 7f2d48e00700 1 -- 172.21.15.1:0/2084394614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fcef0 msgr2=0x7f2d440fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.786+0000 7f2d48e00700 1 --2- 172.21.15.1:0/2084394614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fcef0 0x7f2d440fd320 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2d34004940 tx=0x7f2d34009e30 comp rx=0 tx=0).stop 2024-02-28T16:27:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.786+0000 7f2d48e00700 1 -- 172.21.15.1:0/2084394614 shutdown_connections 2024-02-28T16:27:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.786+0000 7f2d48e00700 1 --2- 172.21.15.1:0/2084394614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d440fe0e0 0x7f2d440fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.786+0000 7f2d48e00700 1 --2- 172.21.15.1:0/2084394614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fcef0 0x7f2d440fd320 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.786+0000 7f2d48e00700 1 -- 172.21.15.1:0/2084394614 >> 172.21.15.1:0/2084394614 conn(0x7f2d440f8620 msgr2=0x7f2d440faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:27:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.786+0000 7f2d48e00700 1 -- 172.21.15.1:0/2084394614 shutdown_connections 2024-02-28T16:27:52.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.786+0000 7f2d48e00700 1 -- 172.21.15.1:0/2084394614 wait complete. 2024-02-28T16:27:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.787+0000 7f2d48e00700 1 Processor -- start 2024-02-28T16:27:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.787+0000 7f2d48e00700 1 -- start start 2024-02-28T16:27:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.788+0000 7f2d48e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d440fcef0 0x7f2d4410d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:52.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.788+0000 7f2d48e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fe0e0 0x7f2d4410def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:52.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.788+0000 7f2d48e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4410e4f0 con 0x7f2d440fe0e0 2024-02-28T16:27:52.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.788+0000 7f2d48e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4410ed90 con 0x7f2d440fcef0 2024-02-28T16:27:52.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.788+0000 7f2d41400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fe0e0 0x7f2d4410def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:52.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.788+0000 7f2d41400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fe0e0 0x7f2d4410def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39140/0 (socket says 172.21.15.1:39140) 2024-02-28T16:27:52.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.788+0000 7f2d41400700 1 -- 172.21.15.1:0/3972656737 learned_addr learned my addr 172.21.15.1:0/3972656737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:27:52.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.788+0000 7f2d41e00700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d440fcef0 0x7f2d4410d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:52.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.789+0000 7f2d41400700 1 -- 172.21.15.1:0/3972656737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d440fcef0 msgr2=0x7f2d4410d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:27:52.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.789+0000 7f2d41400700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d440fcef0 0x7f2d4410d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:27:52.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.789+0000 7f2d41400700 1 -- 172.21.15.1:0/3972656737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d34005040 con 0x7f2d440fe0e0 2024-02-28T16:27:52.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.789+0000 7f2d41400700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fe0e0 0x7f2d4410def0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2d30002a50 tx=0x7f2d300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:52.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.789+0000 7f2d3a000700 1 -- 172.21.15.1:0/3972656737 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d30007cc0 con 0x7f2d440fe0e0 2024-02-28T16:27:52.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.789+0000 7f2d3a000700 1 -- 172.21.15.1:0/3972656737 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d3000f040 con 0x7f2d440fe0e0 2024-02-28T16:27:52.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.789+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4410f070 con 0x7f2d440fe0e0 2024-02-28T16:27:52.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.789+0000 7f2d3a000700 1 -- 172.21.15.1:0/3972656737 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d3000db70 con 0x7f2d440fe0e0 2024-02-28T16:27:52.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.790+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4410f5b0 con 0x7f2d440fe0e0 2024-02-28T16:27:52.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.791+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4404bed0 con 0x7f2d440fe0e0 2024-02-28T16:27:52.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.792+0000 7f2d3a000700 1 -- 172.21.15.1:0/3972656737 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2d3000dd10 con 0x7f2d440fe0e0 2024-02-28T16:27:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.793+0000 7f2d3a000700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d280692a0 0x7f2d2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:27:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.793+0000 7f2d3a000700 1 -- 172.21.15.1:0/3972656737 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2d300869a0 con 0x7f2d440fe0e0 2024-02-28T16:27:52.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.794+0000 7f2d41e00700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d280692a0 0x7f2d2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:27:52.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.794+0000 7f2d41e00700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d280692a0 0x7f2d2806b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400b3b0 tx=0x7f2d34009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:27:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:52.798+0000 7f2d3a000700 1 -- 172.21.15.1:0/3972656737 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2d30056640 con 0x7f2d440fe0e0 2024-02-28T16:27:53.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:27:53.032+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d4410f9c0 con 0x7f2d280692a0 2024-02-28T16:27:53.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:53 smithi066 ceph-mon[36236]: pgmap v256: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-28T16:27:54.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:53 smithi001 ceph-mon[28991]: pgmap v256: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-28T16:27:54.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:54 smithi066 ceph-mon[36236]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:27:55.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:54 smithi001 ceph-mon[28991]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:27:55.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:55 smithi066 ceph-mon[36236]: pgmap v257: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-28T16:27:56.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:55 smithi001 ceph-mon[28991]: pgmap v257: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-28T16:27:56.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:27:57.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:27:57.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:57 smithi001 ceph-mon[28991]: pgmap v258: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:27:57.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:57 smithi066 ceph-mon[36236]: pgmap v258: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:27:59.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:27:59 smithi066 ceph-mon[36236]: pgmap v259: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:28:00.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:27:59 smithi001 ceph-mon[28991]: pgmap v259: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:28:01.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:01 smithi066 ceph-mon[36236]: pgmap v260: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:28:02.215 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:01 smithi001 ceph-mon[28991]: pgmap v260: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:28:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:03 smithi066 ceph-mon[36236]: pgmap v261: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:03 smithi001 ceph-mon[28991]: pgmap v261: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:05.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:05 smithi066 ceph-mon[36236]: pgmap v262: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:05.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:05 smithi001 ceph-mon[28991]: pgmap v262: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:07.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:07 smithi066 ceph-mon[36236]: pgmap v263: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:08.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:07 smithi001 ceph-mon[28991]: pgmap v263: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:09.801 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:09 smithi066 ceph-mon[36236]: pgmap v264: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:10.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:09 smithi001 ceph-mon[28991]: pgmap v264: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:11 smithi066 ceph-mon[36236]: pgmap v265: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:28:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:11 smithi001 ceph-mon[28991]: pgmap v265: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:28:13.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:13 smithi066 ceph-mon[36236]: pgmap v266: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:14.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:13 smithi001 ceph-mon[28991]: pgmap v266: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:15.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:15 smithi066 ceph-mon[36236]: pgmap v267: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:15.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:15 smithi001 ceph-mon[28991]: pgmap v267: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:17 smithi066 ceph-mon[36236]: pgmap v268: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:17 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:28:18.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:17 smithi001 ceph-mon[28991]: pgmap v268: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:18.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:17 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:28:19.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:19 smithi001 ceph-mon[28991]: pgmap v269: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:19.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:19 smithi066 ceph-mon[36236]: pgmap v269: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:21.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:21 smithi066 ceph-mon[36236]: pgmap v270: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:22.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:21 smithi001 ceph-mon[28991]: pgmap v270: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:23.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:23 smithi066 ceph-mon[36236]: pgmap v271: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:24.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:23 smithi001 ceph-mon[28991]: pgmap v271: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:25.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:25 smithi066 ceph-mon[36236]: pgmap v272: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:25.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:25 smithi001 ceph-mon[28991]: pgmap v272: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:26.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:28:26.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:28:27.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:27 smithi066 ceph-mon[36236]: pgmap v273: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:27.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:27 smithi001 ceph-mon[28991]: pgmap v273: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:29.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:29 smithi001 ceph-mon[28991]: pgmap v274: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:29.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:29 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:28:29.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:29 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:28:29.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:29 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:29.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:29 smithi066 ceph-mon[36236]: pgmap v274: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:29.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:29 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:28:29.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:29 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:28:29.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:29 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:31.507 INFO:teuthology.orchestra.run.smithi001.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:28:31.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.517+0000 7f2d3a000700 1 -- 172.21.15.1:0/3972656737 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f2d4410f9c0 con 0x7f2d280692a0 2024-02-28T16:28:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.519+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d280692a0 msgr2=0x7f2d2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.519+0000 7f2d48e00700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d280692a0 0x7f2d2806b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400b3b0 tx=0x7f2d34009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.519+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fe0e0 msgr2=0x7f2d4410def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.519+0000 7f2d48e00700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fe0e0 0x7f2d4410def0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2d30002a50 tx=0x7f2d300046b0 comp rx=0 tx=0).stop 2024-02-28T16:28:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.520+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 shutdown_connections 2024-02-28T16:28:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.520+0000 7f2d48e00700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f2d280692a0 0x7f2d2806b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.520+0000 7f2d48e00700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d440fcef0 0x7f2d4410d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.520+0000 7f2d48e00700 1 --2- 172.21.15.1:0/3972656737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d440fe0e0 0x7f2d4410def0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.520+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 >> 172.21.15.1:0/3972656737 conn(0x7f2d440f8620 msgr2=0x7f2d441013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.520+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 shutdown_connections 2024-02-28T16:28:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:31.520+0000 7f2d48e00700 1 -- 172.21.15.1:0/3972656737 wait complete. 2024-02-28T16:28:31.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:31 smithi001 ceph-mon[28991]: pgmap v275: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:31.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:31 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:31.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:31 smithi066 ceph-mon[36236]: pgmap v275: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:31.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:31 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:32.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-28T16:28:32.923 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:32 smithi001 ceph-mon[28991]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:28:32.923 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:28:32.923 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:28:32.923 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:28:32.923 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:32.924 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:32 smithi001 ceph-mon[28991]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:28:32.924 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:28:32.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:32 smithi066 ceph-mon[36236]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:28:32.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:28:32.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:28:32.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:28:32.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:32.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:32 smithi066 ceph-mon[36236]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:28:33.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:33 smithi066 ceph-mon[36236]: pgmap v276: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:33 smithi001 ceph-mon[28991]: pgmap v276: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:35.497 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:35 smithi001 ceph-mon[28991]: pgmap v277: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:35.497 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:35 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:35.497 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:35 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:28:35.497 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:35 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:35 smithi066 ceph-mon[36236]: pgmap v277: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:35 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:35 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:28:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:35 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:28:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.878+0000 7f540ca00700 1 -- 172.21.15.1:0/3590847369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54081003a0 msgr2=0x7f5408102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.878+0000 7f540ca00700 1 --2- 172.21.15.1:0/3590847369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54081003a0 0x7f5408102780 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f53f4004940 tx=0x7f53f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.878+0000 7f540ca00700 1 -- 172.21.15.1:0/3590847369 shutdown_connections 2024-02-28T16:28:35.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.878+0000 7f540ca00700 1 --2- 172.21.15.1:0/3590847369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54081003a0 0x7f5408102780 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:35.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.878+0000 7f540ca00700 1 --2- 172.21.15.1:0/3590847369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54080fda80 0x7f54080ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:35.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.878+0000 7f540ca00700 1 -- 172.21.15.1:0/3590847369 >> 172.21.15.1:0/3590847369 conn(0x7f54080f74e0 msgr2=0x7f54080f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:35.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.878+0000 7f540ca00700 1 -- 172.21.15.1:0/3590847369 shutdown_connections 2024-02-28T16:28:35.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.878+0000 7f540ca00700 1 -- 172.21.15.1:0/3590847369 wait complete. 2024-02-28T16:28:35.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.879+0000 7f540ca00700 1 Processor -- start 2024-02-28T16:28:35.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.879+0000 7f540ca00700 1 -- start start 2024-02-28T16:28:35.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.880+0000 7f540ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54080fda80 0x7f540810b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:35.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.880+0000 7f540ca00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54081003a0 0x7f540810bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:35.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.880+0000 7f540ca00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540810c210 con 0x7f54080fda80 2024-02-28T16:28:35.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.880+0000 7f540ca00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540810cab0 con 0x7f54081003a0 2024-02-28T16:28:35.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.880+0000 7f5405e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54080fda80 0x7f540810b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:35.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.880+0000 7f5405e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54080fda80 0x7f540810b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39186/0 (socket says 172.21.15.1:39186) 2024-02-28T16:28:35.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.880+0000 7f5405e00700 1 -- 172.21.15.1:0/46204306 learned_addr learned my addr 172.21.15.1:0/46204306 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:28:35.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.880+0000 7f5405400700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54081003a0 0x7f540810bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:35.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.881+0000 7f5405e00700 1 -- 172.21.15.1:0/46204306 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54081003a0 msgr2=0x7f540810bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:35.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.881+0000 7f5405e00700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54081003a0 0x7f540810bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:35.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.881+0000 7f5405e00700 1 -- 172.21.15.1:0/46204306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f4005040 con 0x7f54080fda80 2024-02-28T16:28:35.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.881+0000 7f5405e00700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54080fda80 0x7f540810b680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f53f800eb40 tx=0x7f53f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:35.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.882+0000 7f53fe000700 1 -- 172.21.15.1:0/46204306 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53f8007cc0 con 0x7f54080fda80 2024-02-28T16:28:35.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.882+0000 7f53fe000700 1 -- 172.21.15.1:0/46204306 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53f800f040 con 0x7f54080fda80 2024-02-28T16:28:35.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.882+0000 7f53fe000700 1 -- 172.21.15.1:0/46204306 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53f8009620 con 0x7f54080fda80 2024-02-28T16:28:35.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.882+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540810cd30 con 0x7f54080fda80 2024-02-28T16:28:35.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.882+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540810d270 con 0x7f54080fda80 2024-02-28T16:28:35.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.883+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f540804bed0 con 0x7f54080fda80 2024-02-28T16:28:35.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.884+0000 7f53fe000700 1 -- 172.21.15.1:0/46204306 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f53f8009780 con 0x7f54080fda80 2024-02-28T16:28:35.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.885+0000 7f53fe000700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f53ec0692b0 0x7f53ec06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:35.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.885+0000 7f53fe000700 1 -- 172.21.15.1:0/46204306 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f53f8086e30 con 0x7f54080fda80 2024-02-28T16:28:35.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.886+0000 7f5405400700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f53ec0692b0 0x7f53ec06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:35.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.886+0000 7f5405400700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f53ec0692b0 0x7f53ec06b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f53f400b0d0 tx=0x7f53f4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:35.890+0000 7f53fe000700 1 -- 172.21.15.1:0/46204306 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f53f804dbd0 con 0x7f54080fda80 2024-02-28T16:28:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.150+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f54080fbf80 con 0x7f53ec0692b0 2024-02-28T16:28:36.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.152+0000 7f53fe000700 1 -- 172.21.15.1:0/46204306 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f54080fbf80 con 0x7f53ec0692b0 2024-02-28T16:28:36.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f53ec0692b0 msgr2=0x7f53ec06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:36.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f53ec0692b0 0x7f53ec06b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f53f400b0d0 tx=0x7f53f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:36.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54080fda80 msgr2=0x7f540810b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:36.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54080fda80 0x7f540810b680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f53f800eb40 tx=0x7f53f800b040 comp rx=0 tx=0).stop 2024-02-28T16:28:36.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 shutdown_connections 2024-02-28T16:28:36.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f53ec0692b0 0x7f53ec06b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54080fda80 0x7f540810b680 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 --2- 172.21.15.1:0/46204306 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54081003a0 0x7f540810bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.155+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 >> 172.21.15.1:0/46204306 conn(0x7f54080f74e0 msgr2=0x7f54080fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:36.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.156+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 shutdown_connections 2024-02-28T16:28:36.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.156+0000 7f540ca00700 1 -- 172.21.15.1:0/46204306 wait complete. 2024-02-28T16:28:36.167 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:28:36.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.324+0000 7f0a24000700 1 -- 172.21.15.1:0/287927632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a1c0fe0e0 msgr2=0x7f0a1c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:36.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.324+0000 7f0a24000700 1 --2- 172.21.15.1:0/287927632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a1c0fe0e0 0x7f0a1c0fe550 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0a08004940 tx=0x7f0a08009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:36.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.324+0000 7f0a24000700 1 -- 172.21.15.1:0/287927632 shutdown_connections 2024-02-28T16:28:36.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.324+0000 7f0a24000700 1 --2- 172.21.15.1:0/287927632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a1c0fe0e0 0x7f0a1c0fe550 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.324+0000 7f0a24000700 1 --2- 172.21.15.1:0/287927632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a1c0fcef0 0x7f0a1c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.324+0000 7f0a24000700 1 -- 172.21.15.1:0/287927632 >> 172.21.15.1:0/287927632 conn(0x7f0a1c0f8620 msgr2=0x7f0a1c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.324+0000 7f0a24000700 1 -- 172.21.15.1:0/287927632 shutdown_connections 2024-02-28T16:28:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.324+0000 7f0a24000700 1 -- 172.21.15.1:0/287927632 wait complete. 2024-02-28T16:28:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.325+0000 7f0a24000700 1 Processor -- start 2024-02-28T16:28:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.325+0000 7f0a24000700 1 -- start start 2024-02-28T16:28:36.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.326+0000 7f0a24000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a1c0fcef0 0x7f0a1c071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:36.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.326+0000 7f0a24000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a1c0fe0e0 0x7f0a1c0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:36.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.326+0000 7f0a24000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0708e0 con 0x7f0a1c0fcef0 2024-02-28T16:28:36.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.326+0000 7f0a24000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c070a50 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.326+0000 7f0a20c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a1c0fe0e0 0x7f0a1c0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:36.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.326+0000 7f0a20c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a1c0fe0e0 0x7f0a1c0702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:39154/0 (socket says 172.21.15.1:39154) 2024-02-28T16:28:36.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.326+0000 7f0a20c00700 1 -- 172.21.15.1:0/2986307174 learned_addr learned my addr 172.21.15.1:0/2986307174 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:28:36.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.327+0000 7f0a20c00700 1 -- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a1c0fcef0 msgr2=0x7f0a1c071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:36.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.327+0000 7f0a20c00700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a1c0fcef0 0x7f0a1c071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.327+0000 7f0a20c00700 1 -- 172.21.15.1:0/2986307174 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a08005040 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.327+0000 7f0a20c00700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a1c0fe0e0 0x7f0a1c0702e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0a08000c00 tx=0x7f0a08009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:36.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.328+0000 7f0a19600700 1 -- 172.21.15.1:0/2986307174 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a0800e070 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.328+0000 7f0a19600700 1 -- 172.21.15.1:0/2986307174 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a0801c400 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.328+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c070cd0 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.328+0000 7f0a19600700 1 -- 172.21.15.1:0/2986307174 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a0800b650 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.328+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1c115920 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.329+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc0052e0 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.331+0000 7f0a19600700 1 -- 172.21.15.1:0/2986307174 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0a0801ca00 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.333+0000 7f0a19600700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0a040692a0 0x7f0a0406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:36.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.333+0000 7f0a19600700 1 -- 172.21.15.1:0/2986307174 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0a08088ab0 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.334+0000 7f0a21600700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0a040692a0 0x7f0a0406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:36.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.335+0000 7f0a21600700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0a040692a0 0x7f0a0406b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c071bf0 tx=0x7f0a1000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:36.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.340+0000 7f0a19600700 1 -- 172.21.15.1:0/2986307174 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0a08058850 con 0x7f0a1c0fe0e0 2024-02-28T16:28:36.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:36 smithi001 ceph-mon[28991]: Upgrade: Target is version 19.0.0-1670-g1fb5d106 (unknown) 2024-02-28T16:28:36.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:36 smithi001 ceph-mon[28991]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc'] 2024-02-28T16:28:36.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:36 smithi001 ceph-mon[28991]: Upgrade: Need to upgrade myself (mgr.smithi001.rmthtx) 2024-02-28T16:28:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:36 smithi066 ceph-mon[36236]: Upgrade: Target is version 19.0.0-1670-g1fb5d106 (unknown) 2024-02-28T16:28:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:36 smithi066 ceph-mon[36236]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc'] 2024-02-28T16:28:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:36 smithi066 ceph-mon[36236]: Upgrade: Need to upgrade myself (mgr.smithi001.rmthtx) 2024-02-28T16:28:36.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.779+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f09fc000bf0 con 0x7f0a040692a0 2024-02-28T16:28:36.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.781+0000 7f0a19600700 1 -- 172.21.15.1:0/2986307174 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f09fc000bf0 con 0x7f0a040692a0 2024-02-28T16:28:36.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0a040692a0 msgr2=0x7f0a0406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:36.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0a040692a0 0x7f0a0406b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c071bf0 tx=0x7f0a1000b040 comp rx=0 tx=0).stop 2024-02-28T16:28:36.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a1c0fe0e0 msgr2=0x7f0a1c0702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a1c0fe0e0 0x7f0a1c0702e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0a08000c00 tx=0x7f0a08009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 shutdown_connections 2024-02-28T16:28:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f0a040692a0 0x7f0a0406b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a1c0fcef0 0x7f0a1c071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 --2- 172.21.15.1:0/2986307174 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a1c0fe0e0 0x7f0a1c0702e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 >> 172.21.15.1:0/2986307174 conn(0x7f0a1c0f8620 msgr2=0x7f0a1c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 shutdown_connections 2024-02-28T16:28:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:36.784+0000 7f0a24000700 1 -- 172.21.15.1:0/2986307174 wait complete. 2024-02-28T16:28:36.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.009+0000 7f6515c00700 1 -- 172.21.15.1:0/3382503993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fcef0 msgr2=0x7f65100fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:37.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.009+0000 7f6515c00700 1 --2- 172.21.15.1:0/3382503993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fcef0 0x7f65100fd320 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6500004940 tx=0x7f6500009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:37.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.009+0000 7f6515c00700 1 -- 172.21.15.1:0/3382503993 shutdown_connections 2024-02-28T16:28:37.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.009+0000 7f6515c00700 1 --2- 172.21.15.1:0/3382503993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65100fe0e0 0x7f65100fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.009+0000 7f6515c00700 1 --2- 172.21.15.1:0/3382503993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fcef0 0x7f65100fd320 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.009+0000 7f6515c00700 1 -- 172.21.15.1:0/3382503993 >> 172.21.15.1:0/3382503993 conn(0x7f65100f8620 msgr2=0x7f65100faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:37.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.010+0000 7f6515c00700 1 -- 172.21.15.1:0/3382503993 shutdown_connections 2024-02-28T16:28:37.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.010+0000 7f6515c00700 1 -- 172.21.15.1:0/3382503993 wait complete. 2024-02-28T16:28:37.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.011+0000 7f6515c00700 1 Processor -- start 2024-02-28T16:28:37.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.011+0000 7f6515c00700 1 -- start start 2024-02-28T16:28:37.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.011+0000 7f6515c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65100fcef0 0x7f6510071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:37.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.011+0000 7f6515c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fe0e0 0x7f6510071d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:37.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.011+0000 7f6515c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6510070370 con 0x7f65100fe0e0 2024-02-28T16:28:37.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.011+0000 7f6515c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65100704b0 con 0x7f65100fcef0 2024-02-28T16:28:37.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.012+0000 7f650e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fe0e0 0x7f6510071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:37.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.012+0000 7f650e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fe0e0 0x7f6510071d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39234/0 (socket says 172.21.15.1:39234) 2024-02-28T16:28:37.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.012+0000 7f650e000700 1 -- 172.21.15.1:0/2935752450 learned_addr learned my addr 172.21.15.1:0/2935752450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:28:37.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.013+0000 7f650e000700 1 -- 172.21.15.1:0/2935752450 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65100fcef0 msgr2=0x7f6510071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:37.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.013+0000 7f650e000700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65100fcef0 0x7f6510071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.013+0000 7f650e000700 1 -- 172.21.15.1:0/2935752450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6500005040 con 0x7f65100fe0e0 2024-02-28T16:28:37.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.013+0000 7f650e000700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fe0e0 0x7f6510071d80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00eb40 tx=0x7f64fc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:37.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.014+0000 7f6506a00700 1 -- 172.21.15.1:0/2935752450 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64fc007cc0 con 0x7f65100fe0e0 2024-02-28T16:28:37.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.014+0000 7f6506a00700 1 -- 172.21.15.1:0/2935752450 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64fc00f040 con 0x7f65100fe0e0 2024-02-28T16:28:37.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.014+0000 7f6506a00700 1 -- 172.21.15.1:0/2935752450 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64fc009620 con 0x7f65100fe0e0 2024-02-28T16:28:37.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.014+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6510070730 con 0x7f65100fe0e0 2024-02-28T16:28:37.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.014+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6510070c70 con 0x7f65100fe0e0 2024-02-28T16:28:37.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.016+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651004bed0 con 0x7f65100fe0e0 2024-02-28T16:28:37.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.017+0000 7f6506a00700 1 -- 172.21.15.1:0/2935752450 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f64fc009780 con 0x7f65100fe0e0 2024-02-28T16:28:37.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.018+0000 7f6506a00700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f64f40692b0 0x7f64f406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:37.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.019+0000 7f6506a00700 1 -- 172.21.15.1:0/2935752450 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f64fc086da0 con 0x7f65100fe0e0 2024-02-28T16:28:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.019+0000 7f650ea00700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f64f40692b0 0x7f64f406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:37.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.020+0000 7f650ea00700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f64f40692b0 0x7f64f406b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6500009fd0 tx=0x7f6500009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:37.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.027+0000 7f6506a00700 1 -- 172.21.15.1:0/2935752450 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f64fc04dbd0 con 0x7f65100fe0e0 2024-02-28T16:28:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.455+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6510102ac0 con 0x7f64f40692b0 2024-02-28T16:28:37.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.474+0000 7f6506a00700 1 -- 172.21.15.1:0/2935752450 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f6510102ac0 con 0x7f64f40692b0 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (7m) 81s ago 8m 13.1M - 0.25.0 c8568f914cd2 3d10852c3022 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (9m) 81s ago 9m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (7m) 87s ago 7m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (6m) 81s ago 8m 74.2M - 9.4.7 954c08fa6188 2a03709b4c9c 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (114s) 81s ago 113s 13.2M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (105s) 81s ago 104s 10.5M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (101s) 87s ago 100s 10.5M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (110s) 87s ago 109s 12.9M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:28:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:9283 running (10m) 81s ago 10m 481M - 17.2.7-535-gc0f02e60 1c37ed371348 8eb0fd290742 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283 running (7m) 87s ago 7m 432M - 17.2.7-535-gc0f02e60 1c37ed371348 d7f37fd2495f 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (10m) 81s ago 10m 42.0M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (7m) 87s ago 7m 31.1M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (9m) 81s ago 9m 7956k - 1.5.0 0da6a335fe13 3dac2d3d42b2 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (7m) 87s ago 7m 7620k - 1.5.0 0da6a335fe13 31630ee26c1c 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (6m) 81s ago 6m 43.7M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (6m) 81s ago 6m 45.5M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (5m) 81s ago 5m 43.4M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (4m) 87s ago 4m 43.3M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (4m) 87s ago 4m 41.1M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (3m) 87s ago 3m 40.5M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:28:37.465 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (6m) 81s ago 8m 45.0M - 2.43.0 a07b618ecd1d 21713d7e702b 2024-02-28T16:28:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f64f40692b0 msgr2=0x7f64f406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f64f40692b0 0x7f64f406b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6500009fd0 tx=0x7f6500009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fe0e0 msgr2=0x7f6510071d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fe0e0 0x7f6510071d80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00eb40 tx=0x7f64fc00b040 comp rx=0 tx=0).stop 2024-02-28T16:28:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 shutdown_connections 2024-02-28T16:28:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f64f40692b0 0x7f64f406b760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65100fcef0 0x7f6510071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 --2- 172.21.15.1:0/2935752450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65100fe0e0 0x7f6510071d80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 >> 172.21.15.1:0/2935752450 conn(0x7f65100f8620 msgr2=0x7f65101013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.477+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 shutdown_connections 2024-02-28T16:28:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.478+0000 7f6515c00700 1 -- 172.21.15.1:0/2935752450 wait complete. 2024-02-28T16:28:37.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.699+0000 7f7697000700 1 -- 172.21.15.1:0/3553787023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 msgr2=0x7f7690070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:37.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.699+0000 7f7697000700 1 --2- 172.21.15.1:0/3553787023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 0x7f7690070cb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f767c004940 tx=0x7f767c009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:37.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.699+0000 7f7697000700 1 -- 172.21.15.1:0/3553787023 shutdown_connections 2024-02-28T16:28:37.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.699+0000 7f7697000700 1 --2- 172.21.15.1:0/3553787023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 0x7f7690070cb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.699+0000 7f7697000700 1 --2- 172.21.15.1:0/3553787023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7690071e90 0x7f7690070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.699+0000 7f7697000700 1 -- 172.21.15.1:0/3553787023 >> 172.21.15.1:0/3553787023 conn(0x7f76900f8620 msgr2=0x7f76900faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:37.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.699+0000 7f7697000700 1 -- 172.21.15.1:0/3553787023 shutdown_connections 2024-02-28T16:28:37.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.699+0000 7f7697000700 1 -- 172.21.15.1:0/3553787023 wait complete. 2024-02-28T16:28:37.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.700+0000 7f7697000700 1 Processor -- start 2024-02-28T16:28:37.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.701+0000 7f7697000700 1 -- start start 2024-02-28T16:28:37.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.701+0000 7f7697000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 0x7f769010fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:37.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.701+0000 7f7697000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7690071e90 0x7f7690110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:37.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.701+0000 7f7697000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7690110600 con 0x7f7690070840 2024-02-28T16:28:37.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.701+0000 7f7697000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7690110ea0 con 0x7f7690071e90 2024-02-28T16:28:37.691 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:37 smithi001 ceph-mon[28991]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc on smithi066 2024-02-28T16:28:37.691 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:37 smithi001 ceph-mon[28991]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:28:37.691 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:37 smithi001 ceph-mon[28991]: pgmap v278: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:37.691 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:37 smithi001 ceph-mon[28991]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:28:37.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.702+0000 7f768fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 0x7f769010fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.702+0000 7f768f400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7690071e90 0x7f7690110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.702+0000 7f768fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 0x7f769010fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39264/0 (socket says 172.21.15.1:39264) 2024-02-28T16:28:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.702+0000 7f768fe00700 1 -- 172.21.15.1:0/2656879020 learned_addr learned my addr 172.21.15.1:0/2656879020 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:28:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.703+0000 7f768fe00700 1 -- 172.21.15.1:0/2656879020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7690071e90 msgr2=0x7f7690110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.703+0000 7f768fe00700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7690071e90 0x7f7690110000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.703+0000 7f768fe00700 1 -- 172.21.15.1:0/2656879020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c005040 con 0x7f7690070840 2024-02-28T16:28:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.703+0000 7f768f400700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7690071e90 0x7f7690110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-28T16:28:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.704+0000 7f768fe00700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 0x7f769010fac0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f768000eb40 tx=0x7f768000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.704+0000 7f7687e00700 1 -- 172.21.15.1:0/2656879020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7680007cc0 con 0x7f7690070840 2024-02-28T16:28:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.704+0000 7f7687e00700 1 -- 172.21.15.1:0/2656879020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f768000f040 con 0x7f7690070840 2024-02-28T16:28:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.704+0000 7f7687e00700 1 -- 172.21.15.1:0/2656879020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f768000ca10 con 0x7f7690070840 2024-02-28T16:28:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.705+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7690111120 con 0x7f7690070840 2024-02-28T16:28:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.705+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7690111660 con 0x7f7690070840 2024-02-28T16:28:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.706+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769004bed0 con 0x7f7690070840 2024-02-28T16:28:37.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.709+0000 7f7687e00700 1 -- 172.21.15.1:0/2656879020 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f768000cb70 con 0x7f7690070840 2024-02-28T16:28:37.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.711+0000 7f7687e00700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7674069380 0x7f767406b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:37.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.711+0000 7f7687e00700 1 -- 172.21.15.1:0/2656879020 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7680086d50 con 0x7f7690070840 2024-02-28T16:28:37.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.712+0000 7f768f400700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7674069380 0x7f767406b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:37.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.713+0000 7f768f400700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7674069380 0x7f767406b830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f767c0045a0 tx=0x7f767c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:37.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:37.719+0000 7f7687e00700 1 -- 172.21.15.1:0/2656879020 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f768004dbd0 con 0x7f7690070840 2024-02-28T16:28:37.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:37 smithi066 ceph-mon[36236]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc on smithi066 2024-02-28T16:28:37.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:37 smithi066 ceph-mon[36236]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:28:37.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:37 smithi066 ceph-mon[36236]: pgmap v278: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:37.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:37 smithi066 ceph-mon[36236]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:28:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.289+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f76901160e0 con 0x7f7690070840 2024-02-28T16:28:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.289+0000 7f7687e00700 1 -- 172.21.15.1:0/2656879020 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f7680056b20 con 0x7f7690070840 2024-02-28T16:28:38.279 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:28:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:28:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:28:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:28:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:28:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 14 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:28:38.280 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:28:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.292+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7674069380 msgr2=0x7f767406b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.292+0000 7f7697000700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7674069380 0x7f767406b830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f767c0045a0 tx=0x7f767c009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.292+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 msgr2=0x7f769010fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.292+0000 7f7697000700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 0x7f769010fac0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f768000eb40 tx=0x7f768000b040 comp rx=0 tx=0).stop 2024-02-28T16:28:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.293+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 shutdown_connections 2024-02-28T16:28:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.293+0000 7f7697000700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7674069380 0x7f767406b830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.293+0000 7f7697000700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7690070840 0x7f769010fac0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.293+0000 7f7697000700 1 --2- 172.21.15.1:0/2656879020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7690071e90 0x7f7690110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.293+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 >> 172.21.15.1:0/2656879020 conn(0x7f76900f8620 msgr2=0x7f7690102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.293+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 shutdown_connections 2024-02-28T16:28:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.293+0000 7f7697000700 1 -- 172.21.15.1:0/2656879020 wait complete. 2024-02-28T16:28:38.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.431+0000 7f6b23000700 1 -- 172.21.15.1:0/1729853773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fe1d0 msgr2=0x7f6b1c100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:38.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.431+0000 7f6b23000700 1 --2- 172.21.15.1:0/1729853773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fe1d0 0x7f6b1c100650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6b08004940 tx=0x7f6b08009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:38.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.431+0000 7f6b23000700 1 -- 172.21.15.1:0/1729853773 shutdown_connections 2024-02-28T16:28:38.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.431+0000 7f6b23000700 1 --2- 172.21.15.1:0/1729853773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fe1d0 0x7f6b1c100650 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.431+0000 7f6b23000700 1 --2- 172.21.15.1:0/1729853773 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6b1c0fb870 0x7f6b1c0fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.431+0000 7f6b23000700 1 -- 172.21.15.1:0/1729853773 >> 172.21.15.1:0/1729853773 conn(0x7f6b1c0f74e0 msgr2=0x7f6b1c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:38.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.432+0000 7f6b23000700 1 -- 172.21.15.1:0/1729853773 shutdown_connections 2024-02-28T16:28:38.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.432+0000 7f6b23000700 1 -- 172.21.15.1:0/1729853773 wait complete. 2024-02-28T16:28:38.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.433+0000 7f6b23000700 1 Processor -- start 2024-02-28T16:28:38.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.433+0000 7f6b23000700 1 -- start start 2024-02-28T16:28:38.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.433+0000 7f6b23000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fb870 0x7f6b1c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:38.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.434+0000 7f6b23000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6b1c0fe1d0 0x7f6b1c114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:38.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.434+0000 7f6b23000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c114a10 con 0x7f6b1c0fb870 2024-02-28T16:28:38.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.434+0000 7f6b23000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c1152b0 con 0x7f6b1c0fe1d0 2024-02-28T16:28:38.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.434+0000 7f6b1be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fb870 0x7f6b1c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:38.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.434+0000 7f6b1be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fb870 0x7f6b1c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39282/0 (socket says 172.21.15.1:39282) 2024-02-28T16:28:38.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.434+0000 7f6b1be00700 1 -- 172.21.15.1:0/4170210627 learned_addr learned my addr 172.21.15.1:0/4170210627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:28:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.435+0000 7f6b1be00700 1 -- 172.21.15.1:0/4170210627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6b1c0fe1d0 msgr2=0x7f6b1c114410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-28T16:28:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.435+0000 7f6b1be00700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6b1c0fe1d0 0x7f6b1c114410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.435+0000 7f6b1be00700 1 -- 172.21.15.1:0/4170210627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08005040 con 0x7f6b1c0fb870 2024-02-28T16:28:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.436+0000 7f6b1be00700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fb870 0x7f6b1c113ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00eb40 tx=0x7f6b0c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.436+0000 7f6b13e00700 1 -- 172.21.15.1:0/4170210627 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b0c007cc0 con 0x7f6b1c0fb870 2024-02-28T16:28:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.436+0000 7f6b13e00700 1 -- 172.21.15.1:0/4170210627 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b0c00f040 con 0x7f6b1c0fb870 2024-02-28T16:28:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.436+0000 7f6b13e00700 1 -- 172.21.15.1:0/4170210627 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b0c009620 con 0x7f6b1c0fb870 2024-02-28T16:28:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.436+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c115530 con 0x7f6b1c0fb870 2024-02-28T16:28:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.437+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c115a70 con 0x7f6b1c0fb870 2024-02-28T16:28:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.438+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6afc0052e0 con 0x7f6b1c0fb870 2024-02-28T16:28:38.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.440+0000 7f6b13e00700 1 -- 172.21.15.1:0/4170210627 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f6b0c009780 con 0x7f6b1c0fb870 2024-02-28T16:28:38.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.441+0000 7f6b13e00700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6b000692b0 0x7f6b0006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:28:38.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.441+0000 7f6b13e00700 1 -- 172.21.15.1:0/4170210627 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6b0c086d10 con 0x7f6b1c0fb870 2024-02-28T16:28:38.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.441+0000 7f6b1b400700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6b000692b0 0x7f6b0006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:28:38.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.442+0000 7f6b1b400700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6b000692b0 0x7f6b0006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800b0d0 tx=0x7f6b08009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:28:38.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.449+0000 7f6b13e00700 1 -- 172.21.15.1:0/4170210627 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6b0c04dbd0 con 0x7f6b1c0fb870 2024-02-28T16:28:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.712+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6afc000bf0 con 0x7f6b000692b0 2024-02-28T16:28:38.702 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:38 smithi001 ceph-mon[28991]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:28:38.702 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:38 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2656879020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:28:38.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.714+0000 7f6b13e00700 1 -- 172.21.15.1:0/4170210627 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f6afc000bf0 con 0x7f6b000692b0 2024-02-28T16:28:38.703 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:28:38.704 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:28:38.704 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:28:38.704 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:28:38.704 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [], 2024-02-28T16:28:38.704 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "0/2 daemons upgraded", 2024-02-28T16:28:38.704 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc image on host smithi066", 2024-02-28T16:28:38.704 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:28:38.704 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:28:38.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.717+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6b000692b0 msgr2=0x7f6b0006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:38.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.717+0000 7f6b23000700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6b000692b0 0x7f6b0006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800b0d0 tx=0x7f6b08009e30 comp rx=0 tx=0).stop 2024-02-28T16:28:38.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.717+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fb870 msgr2=0x7f6b1c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:28:38.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.717+0000 7f6b23000700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fb870 0x7f6b1c113ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00eb40 tx=0x7f6b0c00b040 comp rx=0 tx=0).stop 2024-02-28T16:28:38.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.718+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 shutdown_connections 2024-02-28T16:28:38.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.718+0000 7f6b23000700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f6b000692b0 0x7f6b0006b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.718+0000 7f6b23000700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b1c0fb870 0x7f6b1c113ed0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.718+0000 7f6b23000700 1 --2- 172.21.15.1:0/4170210627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6b1c0fe1d0 0x7f6b1c114410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:28:38.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.718+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 >> 172.21.15.1:0/4170210627 conn(0x7f6b1c0f74e0 msgr2=0x7f6b1c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:28:38.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.718+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 shutdown_connections 2024-02-28T16:28:38.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:28:38.719+0000 7f6b23000700 1 -- 172.21.15.1:0/4170210627 wait complete. 2024-02-28T16:28:38.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:38 smithi066 ceph-mon[36236]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:28:38.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:38 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2656879020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:28:39.650 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:39 smithi066 ceph-mon[36236]: pgmap v279: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:39.650 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:39 smithi066 ceph-mon[36236]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:28:39.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:39 smithi001 ceph-mon[28991]: pgmap v279: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:39.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:39 smithi001 ceph-mon[28991]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:28:41.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:41 smithi066 ceph-mon[36236]: pgmap v280: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:41.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:41 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:28:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:41 smithi001 ceph-mon[28991]: pgmap v280: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:41 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:28:43.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:43 smithi066 ceph-mon[36236]: pgmap v281: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:43 smithi001 ceph-mon[28991]: pgmap v281: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:45.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:45 smithi066 ceph-mon[36236]: pgmap v282: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:45.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:45 smithi001 ceph-mon[28991]: pgmap v282: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:47.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:47 smithi066 ceph-mon[36236]: pgmap v283: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:48.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:47 smithi001 ceph-mon[28991]: pgmap v283: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:49.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:49 smithi001 ceph-mon[28991]: pgmap v284: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:49.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:49 smithi066 ceph-mon[36236]: pgmap v284: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:51.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:51 smithi066 ceph-mon[36236]: pgmap v285: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:52.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:51 smithi001 ceph-mon[28991]: pgmap v285: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:53.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:53 smithi066 ceph-mon[36236]: pgmap v286: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:53 smithi001 ceph-mon[28991]: pgmap v286: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:55.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:55 smithi066 ceph-mon[36236]: pgmap v287: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:55.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:55 smithi001 ceph-mon[28991]: pgmap v287: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:56.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:56 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:28:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:56 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:28:57.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:57 smithi066 ceph-mon[36236]: pgmap v288: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:28:57.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:57 smithi001 ceph-mon[28991]: pgmap v288: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:00.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:28:59 smithi066 ceph-mon[36236]: pgmap v289: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:00.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:28:59 smithi001 ceph-mon[28991]: pgmap v289: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:01.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:01 smithi066 ceph-mon[36236]: pgmap v290: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:02.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:01 smithi001 ceph-mon[28991]: pgmap v290: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:03 smithi066 ceph-mon[36236]: pgmap v291: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:04.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:03 smithi001 ceph-mon[28991]: pgmap v291: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:04.468 INFO:tasks.workunit.client.0.smithi001.stderr:Updating files: 59% (7269/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 98% (12038/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-02-28T16:29:04.842 INFO:tasks.workunit.client.0.smithi001.stderr:Note: switching to '1fb5d10608dccee73edd6ef8a2c7b804a097c324'. 2024-02-28T16:29:04.842 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-02-28T16:29:04.842 INFO:tasks.workunit.client.0.smithi001.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-28T16:29:04.842 INFO:tasks.workunit.client.0.smithi001.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-28T16:29:04.842 INFO:tasks.workunit.client.0.smithi001.stderr:state without impacting any branches by switching back to a branch. 2024-02-28T16:29:04.842 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-02-28T16:29:04.842 INFO:tasks.workunit.client.0.smithi001.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-28T16:29:04.842 INFO:tasks.workunit.client.0.smithi001.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr: git switch -c 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr:Or undo this operation with: 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr: git switch - 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-02-28T16:29:04.843 INFO:tasks.workunit.client.0.smithi001.stderr:HEAD is now at 1fb5d10608d Merge pull request #55565 from BBoozmen/oozmen_avoid_path_normalization 2024-02-28T16:29:04.846 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-28T16:29:04.904 INFO:tasks.workunit.client.0.smithi001.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-28T16:29:04.912 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-28T16:29:04.912 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-28T16:29:05.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:05 smithi066 ceph-mon[36236]: pgmap v292: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:05.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:05 smithi001 ceph-mon[28991]: pgmap v292: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:06.033 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-28T16:29:06.153 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-28T16:29:06.241 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-28T16:29:06.247 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-28T16:29:06.247 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-28T16:29:06.333 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-28T16:29:06.343 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T16:29:06.343 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-28T16:29:06.397 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-28T16:29:06.400 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-28T16:29:06.400 DEBUG:teuthology.orchestra.run.smithi001:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-28T16:29:06.498 INFO:tasks.workunit.client.0.smithi001.stderr:+ mkdir -p fsstress 2024-02-28T16:29:06.506 INFO:tasks.workunit.client.0.smithi001.stderr:+ pushd fsstress 2024-02-28T16:29:06.513 INFO:tasks.workunit.client.0.smithi001.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-28T16:29:06.513 INFO:tasks.workunit.client.0.smithi001.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-28T16:29:07.867 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:07 smithi066 ceph-mon[36236]: pgmap v293: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:08.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:07 smithi001 ceph-mon[28991]: pgmap v293: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:08.417 INFO:tasks.workunit.client.0.smithi001.stderr:+ tar xzf ltp-full.tgz 2024-02-28T16:29:08.504 INFO:tasks.workunit.client.1.smithi066.stderr:Updating files: 58% (7157/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 94% (11584/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.904+0000 7f705f600700 1 -- 172.21.15.1:0/2724743700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 msgr2=0x7f7058102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.904+0000 7f705f600700 1 --2- 172.21.15.1:0/2724743700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 0x7f7058102520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7044004940 tx=0x7f7044009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.904+0000 7f705f600700 1 -- 172.21.15.1:0/2724743700 shutdown_connections 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.904+0000 7f705f600700 1 --2- 172.21.15.1:0/2724743700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 0x7f7058102520 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.904+0000 7f705f600700 1 --2- 172.21.15.1:0/2724743700 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70580663c0 0x7f70580ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.904+0000 7f705f600700 1 -- 172.21.15.1:0/2724743700 >> 172.21.15.1:0/2724743700 conn(0x7f70580f74e0 msgr2=0x7f70580f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.904+0000 7f705f600700 1 -- 172.21.15.1:0/2724743700 shutdown_connections 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.905+0000 7f705f600700 1 -- 172.21.15.1:0/2724743700 wait complete. 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.905+0000 7f705f600700 1 Processor -- start 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.906+0000 7f705f600700 1 -- start start 2024-02-28T16:29:08.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.906+0000 7f705f600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70580663c0 0x7f705810fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:08.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.906+0000 7f705f600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 0x7f7058110100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:08.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.906+0000 7f705f600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7058110700 con 0x7f70581000e0 2024-02-28T16:29:08.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.906+0000 7f705f600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7058110fa0 con 0x7f70580663c0 2024-02-28T16:29:08.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.906+0000 7f7057e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 0x7f7058110100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:08.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.906+0000 7f7057e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 0x7f7058110100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57066/0 (socket says 172.21.15.1:57066) 2024-02-28T16:29:08.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.906+0000 7f7057e00700 1 -- 172.21.15.1:0/2250884354 learned_addr learned my addr 172.21.15.1:0/2250884354 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:08.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.907+0000 7f705cc00700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70580663c0 0x7f705810fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:08.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.907+0000 7f7057e00700 1 -- 172.21.15.1:0/2250884354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70580663c0 msgr2=0x7f705810fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:08.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.907+0000 7f7057e00700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70580663c0 0x7f705810fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:08.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.907+0000 7f7057e00700 1 -- 172.21.15.1:0/2250884354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7044005040 con 0x7f70581000e0 2024-02-28T16:29:08.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.907+0000 7f7057e00700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 0x7f7058110100 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7044000c00 tx=0x7f7044009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:08.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.908+0000 7f7054c00700 1 -- 172.21.15.1:0/2250884354 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f704400e040 con 0x7f70581000e0 2024-02-28T16:29:08.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.908+0000 7f7054c00700 1 -- 172.21.15.1:0/2250884354 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f704400bcc0 con 0x7f70581000e0 2024-02-28T16:29:08.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.908+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7058111220 con 0x7f70581000e0 2024-02-28T16:29:08.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.908+0000 7f7054c00700 1 -- 172.21.15.1:0/2250884354 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7044013ea0 con 0x7f70581000e0 2024-02-28T16:29:08.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.908+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7058111760 con 0x7f70581000e0 2024-02-28T16:29:08.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.909+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70580f9080 con 0x7f70581000e0 2024-02-28T16:29:08.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.910+0000 7f7054c00700 1 -- 172.21.15.1:0/2250884354 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7044013420 con 0x7f70581000e0 2024-02-28T16:29:08.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.911+0000 7f7054c00700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f70400692b0 0x7f704006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:08.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.911+0000 7f7054c00700 1 -- 172.21.15.1:0/2250884354 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f70440884c0 con 0x7f70581000e0 2024-02-28T16:29:08.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.912+0000 7f705cc00700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f70400692b0 0x7f704006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:08.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.913+0000 7f705cc00700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f70400692b0 0x7f704006b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f704c004ec0 tx=0x7f704c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:08.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:08.917+0000 7f7054c00700 1 -- 172.21.15.1:0/2250884354 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f70440581e0 con 0x7f70581000e0 2024-02-28T16:29:08.919 INFO:tasks.workunit.client.1.smithi066.stderr:Note: switching to '1fb5d10608dccee73edd6ef8a2c7b804a097c324'. 2024-02-28T16:29:08.919 INFO:tasks.workunit.client.1.smithi066.stderr: 2024-02-28T16:29:08.919 INFO:tasks.workunit.client.1.smithi066.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-28T16:29:08.919 INFO:tasks.workunit.client.1.smithi066.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-28T16:29:08.919 INFO:tasks.workunit.client.1.smithi066.stderr:state without impacting any branches by switching back to a branch. 2024-02-28T16:29:08.919 INFO:tasks.workunit.client.1.smithi066.stderr: 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr: 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr: git switch -c 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr: 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr:Or undo this operation with: 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr: 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr: git switch - 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr: 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr: 2024-02-28T16:29:08.920 INFO:tasks.workunit.client.1.smithi066.stderr:HEAD is now at 1fb5d10608d Merge pull request #55565 from BBoozmen/oozmen_avoid_path_normalization 2024-02-28T16:29:08.923 DEBUG:teuthology.orchestra.run.smithi066:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-28T16:29:09.029 INFO:tasks.workunit.client.1.smithi066.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-28T16:29:09.038 INFO:tasks.workunit.client.1.smithi066.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-28T16:29:09.038 INFO:tasks.workunit.client.1.smithi066.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-28T16:29:09.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.207+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7058111a90 con 0x7f70400692b0 2024-02-28T16:29:09.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.209+0000 7f7054c00700 1 -- 172.21.15.1:0/2250884354 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f7058111a90 con 0x7f70400692b0 2024-02-28T16:29:09.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f70400692b0 msgr2=0x7f704006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:09.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f70400692b0 0x7f704006b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f704c004ec0 tx=0x7f704c009040 comp rx=0 tx=0).stop 2024-02-28T16:29:09.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 msgr2=0x7f7058110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:09.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 0x7f7058110100 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7044000c00 tx=0x7f7044009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:09.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 shutdown_connections 2024-02-28T16:29:09.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f70400692b0 0x7f704006b760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70580663c0 0x7f705810fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 --2- 172.21.15.1:0/2250884354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70581000e0 0x7f7058110100 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 >> 172.21.15.1:0/2250884354 conn(0x7f70580f74e0 msgr2=0x7f7058100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:09.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 shutdown_connections 2024-02-28T16:29:09.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.212+0000 7f705f600700 1 -- 172.21.15.1:0/2250884354 wait complete. 2024-02-28T16:29:09.218 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:29:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.387+0000 7f3e90000700 1 -- 172.21.15.1:0/1047988971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e880fe1d0 msgr2=0x7f3e88100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.387+0000 7f3e90000700 1 --2- 172.21.15.1:0/1047988971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e880fe1d0 0x7f3e88100650 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e74004880 tx=0x7f3e74009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.387+0000 7f3e90000700 1 -- 172.21.15.1:0/1047988971 shutdown_connections 2024-02-28T16:29:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.387+0000 7f3e90000700 1 --2- 172.21.15.1:0/1047988971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e880fe1d0 0x7f3e88100650 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.387+0000 7f3e90000700 1 --2- 172.21.15.1:0/1047988971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e880fb870 0x7f3e880fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.387+0000 7f3e90000700 1 -- 172.21.15.1:0/1047988971 >> 172.21.15.1:0/1047988971 conn(0x7f3e880f74e0 msgr2=0x7f3e880f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.387+0000 7f3e90000700 1 -- 172.21.15.1:0/1047988971 shutdown_connections 2024-02-28T16:29:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.387+0000 7f3e90000700 1 -- 172.21.15.1:0/1047988971 wait complete. 2024-02-28T16:29:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.388+0000 7f3e90000700 1 Processor -- start 2024-02-28T16:29:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.388+0000 7f3e90000700 1 -- start start 2024-02-28T16:29:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.388+0000 7f3e90000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e880fb870 0x7f3e88113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.388+0000 7f3e90000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e880fe1d0 0x7f3e88114480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.388+0000 7f3e90000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e88114a80 con 0x7f3e880fb870 2024-02-28T16:29:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.388+0000 7f3e90000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e88115320 con 0x7f3e880fe1d0 2024-02-28T16:29:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.389+0000 7f3e8d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e880fb870 0x7f3e88113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:09.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.389+0000 7f3e8d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e880fb870 0x7f3e88113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57070/0 (socket says 172.21.15.1:57070) 2024-02-28T16:29:09.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.389+0000 7f3e8d600700 1 -- 172.21.15.1:0/668446643 learned_addr learned my addr 172.21.15.1:0/668446643 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.389+0000 7f3e8cc00700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e880fe1d0 0x7f3e88114480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.390+0000 7f3e8d600700 1 -- 172.21.15.1:0/668446643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e880fe1d0 msgr2=0x7f3e88114480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.390+0000 7f3e8d600700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e880fe1d0 0x7f3e88114480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.390+0000 7f3e8d600700 1 -- 172.21.15.1:0/668446643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74005040 con 0x7f3e880fb870 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.390+0000 7f3e8d600700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e880fb870 0x7f3e88113f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0041d0 tx=0x7f3e7c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.391+0000 7f3e85600700 1 -- 172.21.15.1:0/668446643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7c007cc0 con 0x7f3e880fb870 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.391+0000 7f3e85600700 1 -- 172.21.15.1:0/668446643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e7c00f040 con 0x7f3e880fb870 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.391+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e881155a0 con 0x7f3e880fb870 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.391+0000 7f3e85600700 1 -- 172.21.15.1:0/668446643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7c00ca50 con 0x7f3e880fb870 2024-02-28T16:29:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.391+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e88115ae0 con 0x7f3e880fb870 2024-02-28T16:29:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.393+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8810e050 con 0x7f3e880fb870 2024-02-28T16:29:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.394+0000 7f3e85600700 1 -- 172.21.15.1:0/668446643 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3e7c00cbb0 con 0x7f3e880fb870 2024-02-28T16:29:09.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.395+0000 7f3e85600700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3e700692b0 0x7f3e7006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:09.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.395+0000 7f3e85600700 1 -- 172.21.15.1:0/668446643 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3e7c086c20 con 0x7f3e880fb870 2024-02-28T16:29:09.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.396+0000 7f3e8cc00700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3e700692b0 0x7f3e7006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.396+0000 7f3e8cc00700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3e700692b0 0x7f3e7006b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3e74000c00 tx=0x7f3e74009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.402+0000 7f3e85600700 1 -- 172.21.15.1:0/668446643 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3e7c04dbd0 con 0x7f3e880fb870 2024-02-28T16:29:09.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.668+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e88115e30 con 0x7f3e700692b0 2024-02-28T16:29:09.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.671+0000 7f3e85600700 1 -- 172.21.15.1:0/668446643 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f3e88115e30 con 0x7f3e700692b0 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3e700692b0 msgr2=0x7f3e7006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3e700692b0 0x7f3e7006b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3e74000c00 tx=0x7f3e74009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e880fb870 msgr2=0x7f3e88113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e880fb870 0x7f3e88113f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0041d0 tx=0x7f3e7c00b040 comp rx=0 tx=0).stop 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 shutdown_connections 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f3e700692b0 0x7f3e7006b760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e880fb870 0x7f3e88113f40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 --2- 172.21.15.1:0/668446643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e880fe1d0 0x7f3e88114480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 >> 172.21.15.1:0/668446643 conn(0x7f3e880f74e0 msgr2=0x7f3e880f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 shutdown_connections 2024-02-28T16:29:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.674+0000 7f3e90000700 1 -- 172.21.15.1:0/668446643 wait complete. 2024-02-28T16:29:09.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.843+0000 7f7dd5200700 1 -- 172.21.15.1:0/2160241846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 msgr2=0x7f7dd00ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:09.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.843+0000 7f7dd5200700 1 --2- 172.21.15.1:0/2160241846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 0x7f7dd00ffe60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0004940 tx=0x7f7dc0009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:09.832 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:09 smithi001 ceph-mon[28991]: pgmap v294: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:09.832 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:09.832 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:29:09.832 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:29:09.832 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:09 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:29:09.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.843+0000 7f7dd5200700 1 -- 172.21.15.1:0/2160241846 shutdown_connections 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.843+0000 7f7dd5200700 1 --2- 172.21.15.1:0/2160241846 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7dd01003a0 0x7f7dd0102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.843+0000 7f7dd5200700 1 --2- 172.21.15.1:0/2160241846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 0x7f7dd00ffe60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.843+0000 7f7dd5200700 1 -- 172.21.15.1:0/2160241846 >> 172.21.15.1:0/2160241846 conn(0x7f7dd00f74e0 msgr2=0x7f7dd00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.843+0000 7f7dd5200700 1 -- 172.21.15.1:0/2160241846 shutdown_connections 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.843+0000 7f7dd5200700 1 -- 172.21.15.1:0/2160241846 wait complete. 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.844+0000 7f7dd5200700 1 Processor -- start 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.844+0000 7f7dd5200700 1 -- start start 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.844+0000 7f7dd5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 0x7f7dd010fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.844+0000 7f7dd5200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7dd01003a0 0x7f7dd0110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.844+0000 7f7dd5200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0110670 con 0x7f7dd00fda80 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.844+0000 7f7dd5200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0110f10 con 0x7f7dd01003a0 2024-02-28T16:29:09.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.845+0000 7f7dcde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 0x7f7dd010fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:09.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.845+0000 7f7dcde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 0x7f7dd010fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57098/0 (socket says 172.21.15.1:57098) 2024-02-28T16:29:09.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.845+0000 7f7dcde00700 1 -- 172.21.15.1:0/1571836745 learned_addr learned my addr 172.21.15.1:0/1571836745 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.845+0000 7f7dcd400700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7dd01003a0 0x7f7dd0110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.845+0000 7f7dcde00700 1 -- 172.21.15.1:0/1571836745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7dd01003a0 msgr2=0x7f7dd0110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.845+0000 7f7dcde00700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7dd01003a0 0x7f7dd0110070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.846+0000 7f7dcde00700 1 -- 172.21.15.1:0/1571836745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc0005040 con 0x7f7dd00fda80 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.846+0000 7f7dcde00700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 0x7f7dd010fb30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7dc00093b0 tx=0x7f7dc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.846+0000 7f7dc6000700 1 -- 172.21.15.1:0/1571836745 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dc000e040 con 0x7f7dd00fda80 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.846+0000 7f7dc6000700 1 -- 172.21.15.1:0/1571836745 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7dc000bcc0 con 0x7f7dd00fda80 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.846+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd0111190 con 0x7f7dd00fda80 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.847+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd01116d0 con 0x7f7dd00fda80 2024-02-28T16:29:09.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.847+0000 7f7dc6000700 1 -- 172.21.15.1:0/1571836745 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dc001c630 con 0x7f7dd00fda80 2024-02-28T16:29:09.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.848+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd00f9080 con 0x7f7dd00fda80 2024-02-28T16:29:09.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.849+0000 7f7dc6000700 1 -- 172.21.15.1:0/1571836745 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7dc00139f0 con 0x7f7dd00fda80 2024-02-28T16:29:09.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.850+0000 7f7dc6000700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7db40692a0 0x7f7db406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:09.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.850+0000 7f7dc6000700 1 -- 172.21.15.1:0/1571836745 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7dc00885c0 con 0x7f7dd00fda80 2024-02-28T16:29:09.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.851+0000 7f7dcd400700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7db40692a0 0x7f7db406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:09.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.851+0000 7f7dcd400700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7db40692a0 0x7f7db406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc004990 tx=0x7f7dbc009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:09.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:09.856+0000 7f7dc6000700 1 -- 172.21.15.1:0/1571836745 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7dc0058360 con 0x7f7dd00fda80 2024-02-28T16:29:09.981 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:09 smithi066 ceph-mon[36236]: pgmap v294: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:09.982 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:09.982 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:29:09.982 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:29:09.982 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:09 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:29:10.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.107+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dd01119b0 con 0x7f7db40692a0 2024-02-28T16:29:10.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.140+0000 7f7dc6000700 1 -- 172.21.15.1:0/1571836745 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f7dd01119b0 con 0x7f7db40692a0 2024-02-28T16:29:10.129 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:29:10.129 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (7m) 114s ago 9m 13.1M - 0.25.0 c8568f914cd2 3d10852c3022 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (9m) 114s ago 9m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (8m) 119s ago 8m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (7m) 114s ago 9m 74.2M - 9.4.7 954c08fa6188 2a03709b4c9c 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (2m) 114s ago 2m 13.2M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (2m) 114s ago 2m 10.5M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (2m) 119s ago 2m 10.5M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (2m) 119s ago 2m 12.9M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:9283 running (11m) 114s ago 11m 481M - 17.2.7-535-gc0f02e60 1c37ed371348 8eb0fd290742 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283 running (8m) 119s ago 8m 432M - 17.2.7-535-gc0f02e60 1c37ed371348 d7f37fd2495f 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (11m) 114s ago 11m 42.0M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (8m) 119s ago 8m 31.1M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (9m) 114s ago 9m 7956k - 1.5.0 0da6a335fe13 3dac2d3d42b2 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (8m) 119s ago 8m 7620k - 1.5.0 0da6a335fe13 31630ee26c1c 2024-02-28T16:29:10.130 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (7m) 114s ago 7m 43.7M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:29:10.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (6m) 114s ago 6m 45.5M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:29:10.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (6m) 114s ago 6m 43.4M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:29:10.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (5m) 119s ago 5m 43.3M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:29:10.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (4m) 119s ago 4m 41.1M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:29:10.131 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (4m) 119s ago 4m 40.5M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:29:10.131 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (7m) 114s ago 8m 45.0M - 2.43.0 a07b618ecd1d 21713d7e702b 2024-02-28T16:29:10.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7db40692a0 msgr2=0x7f7db406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7db40692a0 0x7f7db406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc004990 tx=0x7f7dbc009ee0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 msgr2=0x7f7dd010fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 0x7f7dd010fb30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7dc00093b0 tx=0x7f7dc0009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 shutdown_connections 2024-02-28T16:29:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7db40692a0 0x7f7db406b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dd00fda80 0x7f7dd010fb30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 --2- 172.21.15.1:0/1571836745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7dd01003a0 0x7f7dd0110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.143+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 >> 172.21.15.1:0/1571836745 conn(0x7f7dd00f74e0 msgr2=0x7f7dd0101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.144+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 shutdown_connections 2024-02-28T16:29:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.144+0000 7f7dd5200700 1 -- 172.21.15.1:0/1571836745 wait complete. 2024-02-28T16:29:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.304+0000 7f7c71600700 1 -- 172.21.15.1:0/433930992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fcef0 msgr2=0x7f7c6c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:10.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.304+0000 7f7c71600700 1 --2- 172.21.15.1:0/433930992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fcef0 0x7f7c6c0fd320 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c004940 tx=0x7f7c5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:10.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.304+0000 7f7c71600700 1 -- 172.21.15.1:0/433930992 shutdown_connections 2024-02-28T16:29:10.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.304+0000 7f7c71600700 1 --2- 172.21.15.1:0/433930992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7c6c0fe0e0 0x7f7c6c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.304+0000 7f7c71600700 1 --2- 172.21.15.1:0/433930992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fcef0 0x7f7c6c0fd320 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.304+0000 7f7c71600700 1 -- 172.21.15.1:0/433930992 >> 172.21.15.1:0/433930992 conn(0x7f7c6c0f8620 msgr2=0x7f7c6c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:10.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.304+0000 7f7c71600700 1 -- 172.21.15.1:0/433930992 shutdown_connections 2024-02-28T16:29:10.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.304+0000 7f7c71600700 1 -- 172.21.15.1:0/433930992 wait complete. 2024-02-28T16:29:10.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.305+0000 7f7c71600700 1 Processor -- start 2024-02-28T16:29:10.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.305+0000 7f7c71600700 1 -- start start 2024-02-28T16:29:10.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.305+0000 7f7c71600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7c6c0fcef0 0x7f7c6c10fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:10.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.305+0000 7f7c71600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fe0e0 0x7f7c6c1100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:10.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.305+0000 7f7c71600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c1106d0 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.305+0000 7f7c71600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c110f70 con 0x7f7c6c0fcef0 2024-02-28T16:29:10.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.306+0000 7f7c6a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fe0e0 0x7f7c6c1100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:10.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.306+0000 7f7c6a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fe0e0 0x7f7c6c1100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57124/0 (socket says 172.21.15.1:57124) 2024-02-28T16:29:10.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.306+0000 7f7c6a000700 1 -- 172.21.15.1:0/424560094 learned_addr learned my addr 172.21.15.1:0/424560094 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:10.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.306+0000 7f7c6aa00700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7c6c0fcef0 0x7f7c6c10fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:10.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.306+0000 7f7c6a000700 1 -- 172.21.15.1:0/424560094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7c6c0fcef0 msgr2=0x7f7c6c10fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:10.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.306+0000 7f7c6a000700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7c6c0fcef0 0x7f7c6c10fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.306+0000 7f7c6a000700 1 -- 172.21.15.1:0/424560094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c005040 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.307+0000 7f7c6a000700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fe0e0 0x7f7c6c1100d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800c8c0 tx=0x7f7c580044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:10.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.307+0000 7f7c62a00700 1 -- 172.21.15.1:0/424560094 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c5800a980 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.307+0000 7f7c62a00700 1 -- 172.21.15.1:0/424560094 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c5800ce60 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.307+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6c111250 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.307+0000 7f7c62a00700 1 -- 172.21.15.1:0/424560094 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c580115e0 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.307+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6c111790 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.309+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6c102210 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.310+0000 7f7c62a00700 1 -- 172.21.15.1:0/424560094 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7c58012400 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.311+0000 7f7c62a00700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7c500692b0 0x7f7c5006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:10.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.311+0000 7f7c62a00700 1 -- 172.21.15.1:0/424560094 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7c58087250 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.312+0000 7f7c6aa00700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7c500692b0 0x7f7c5006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:10.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.312+0000 7f7c6aa00700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7c500692b0 0x7f7c5006b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00b020 tx=0x7f7c5c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:10.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.316+0000 7f7c62a00700 1 -- 172.21.15.1:0/424560094 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7c58056ff0 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.336 INFO:tasks.workunit.client.1.smithi066.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-28T16:29:10.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.656+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7c6c105a30 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.657+0000 7f7c62a00700 1 -- 172.21.15.1:0/424560094 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f7c580126b0 con 0x7f7c6c0fe0e0 2024-02-28T16:29:10.646 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:29:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:29:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:29:10.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 14 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:29:10.647 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:29:10.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.660+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7c500692b0 msgr2=0x7f7c5006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:10.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.660+0000 7f7c71600700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7c500692b0 0x7f7c5006b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00b020 tx=0x7f7c5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:10.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.660+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fe0e0 msgr2=0x7f7c6c1100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:10.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.660+0000 7f7c71600700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fe0e0 0x7f7c6c1100d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800c8c0 tx=0x7f7c580044c0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.661+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 shutdown_connections 2024-02-28T16:29:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.661+0000 7f7c71600700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f7c500692b0 0x7f7c5006b760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.661+0000 7f7c71600700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7c6c0fcef0 0x7f7c6c10fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.661+0000 7f7c71600700 1 --2- 172.21.15.1:0/424560094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c6c0fe0e0 0x7f7c6c1100d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.661+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 >> 172.21.15.1:0/424560094 conn(0x7f7c6c0f8620 msgr2=0x7f7c6c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.661+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 shutdown_connections 2024-02-28T16:29:10.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.661+0000 7f7c71600700 1 -- 172.21.15.1:0/424560094 wait complete. 2024-02-28T16:29:10.739 INFO:tasks.workunit.client.1.smithi066.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-28T16:29:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.829+0000 7f8d4ec00700 1 -- 172.21.15.1:0/1248453818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 msgr2=0x7f8d48070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.829+0000 7f8d4ec00700 1 --2- 172.21.15.1:0/1248453818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 0x7f8d48070cd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8d30004940 tx=0x7f8d30009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.829+0000 7f8d4ec00700 1 -- 172.21.15.1:0/1248453818 shutdown_connections 2024-02-28T16:29:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.829+0000 7f8d4ec00700 1 --2- 172.21.15.1:0/1248453818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 0x7f8d48070cd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.829+0000 7f8d4ec00700 1 --2- 172.21.15.1:0/1248453818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d48071eb0 0x7f8d48070320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.829+0000 7f8d4ec00700 1 -- 172.21.15.1:0/1248453818 >> 172.21.15.1:0/1248453818 conn(0x7f8d480f8620 msgr2=0x7f8d480faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.829+0000 7f8d4ec00700 1 -- 172.21.15.1:0/1248453818 shutdown_connections 2024-02-28T16:29:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.830+0000 7f8d4ec00700 1 -- 172.21.15.1:0/1248453818 wait complete. 2024-02-28T16:29:10.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.830+0000 7f8d4ec00700 1 Processor -- start 2024-02-28T16:29:10.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d4ec00700 1 -- start start 2024-02-28T16:29:10.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d4ec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 0x7f8d4810fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:10.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d4ec00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d48071eb0 0x7f8d48110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d4ec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d48110620 con 0x7f8d48070860 2024-02-28T16:29:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d4ec00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d48110ec0 con 0x7f8d48071eb0 2024-02-28T16:29:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d47e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 0x7f8d4810fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d47e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 0x7f8d4810fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57136/0 (socket says 172.21.15.1:57136) 2024-02-28T16:29:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d47e00700 1 -- 172.21.15.1:0/416168982 learned_addr learned my addr 172.21.15.1:0/416168982 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.831+0000 7f8d3f400700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d48071eb0 0x7f8d48110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.832+0000 7f8d47e00700 1 -- 172.21.15.1:0/416168982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d48071eb0 msgr2=0x7f8d48110020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.832+0000 7f8d47e00700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d48071eb0 0x7f8d48110020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.832+0000 7f8d47e00700 1 -- 172.21.15.1:0/416168982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30005040 con 0x7f8d48070860 2024-02-28T16:29:10.821 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:10 smithi001 ceph-mon[28991]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:10.821 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:10 smithi001 ceph-mon[28991]: Upgrade: Updating mgr.smithi066.ztraxe 2024-02-28T16:29:10.821 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:10 smithi001 ceph-mon[28991]: Deploying daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:29:10.821 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:10 smithi001 ceph-mon[28991]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.832+0000 7f8d47e00700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 0x7f8d4810fae0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8d3800eb40 tx=0x7f8d3800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.833+0000 7f8d44c00700 1 -- 172.21.15.1:0/416168982 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d38007cc0 con 0x7f8d48070860 2024-02-28T16:29:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.833+0000 7f8d44c00700 1 -- 172.21.15.1:0/416168982 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d3800f040 con 0x7f8d48070860 2024-02-28T16:29:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.833+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d48111140 con 0x7f8d48070860 2024-02-28T16:29:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.833+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d48111680 con 0x7f8d48070860 2024-02-28T16:29:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.833+0000 7f8d44c00700 1 -- 172.21.15.1:0/416168982 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d3800ca10 con 0x7f8d48070860 2024-02-28T16:29:10.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.835+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d240052e0 con 0x7f8d48070860 2024-02-28T16:29:10.824 INFO:tasks.workunit.client.1.smithi066.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-28T16:29:10.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.836+0000 7f8d44c00700 1 -- 172.21.15.1:0/416168982 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f8d3800cb70 con 0x7f8d48070860 2024-02-28T16:29:10.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.837+0000 7f8d44c00700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f8d280692b0 0x7f8d2806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:10.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.837+0000 7f8d44c00700 1 -- 172.21.15.1:0/416168982 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8d38086ba0 con 0x7f8d48070860 2024-02-28T16:29:10.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.837+0000 7f8d3f400700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f8d280692b0 0x7f8d2806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:10.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.838+0000 7f8d3f400700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f8d280692b0 0x7f8d2806b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8d30013740 tx=0x7f8d30009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:10.829 INFO:tasks.workunit.client.1.smithi066.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-28T16:29:10.829 INFO:tasks.workunit.client.1.smithi066.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-28T16:29:10.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:10.842+0000 7f8d44c00700 1 -- 172.21.15.1:0/416168982 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8d38056960 con 0x7f8d48070860 2024-02-28T16:29:10.911 INFO:tasks.workunit.client.1.smithi066.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-28T16:29:10.923 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-02-28T16:29:10.923 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-28T16:29:10.975 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-28T16:29:10.976 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-28T16:29:10.976 DEBUG:teuthology.orchestra.run.smithi066:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-28T16:29:10.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:10 smithi066 ceph-mon[36236]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:10.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:10 smithi066 ceph-mon[36236]: Upgrade: Updating mgr.smithi066.ztraxe 2024-02-28T16:29:10.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:10 smithi066 ceph-mon[36236]: Deploying daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:29:10.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:10 smithi066 ceph-mon[36236]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:11.055 INFO:tasks.workunit.client.1.smithi066.stderr:+ mkdir -p fsstress 2024-02-28T16:29:11.066 INFO:tasks.workunit.client.1.smithi066.stderr:+ pushd fsstress 2024-02-28T16:29:11.070 INFO:tasks.workunit.client.1.smithi066.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-28T16:29:11.070 INFO:tasks.workunit.client.1.smithi066.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-28T16:29:11.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.107+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 --> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d24000bf0 con 0x7f8d280692b0 2024-02-28T16:29:11.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.108+0000 7f8d44c00700 1 -- 172.21.15.1:0/416168982 <== mgr.14212 v2:172.21.15.1:6800/762266793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f8d24000bf0 con 0x7f8d280692b0 2024-02-28T16:29:11.097 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:29:11.097 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:29:11.098 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:29:11.098 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:29:11.098 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [], 2024-02-28T16:29:11.098 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "0/2 daemons upgraded", 2024-02-28T16:29:11.098 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mgr daemons", 2024-02-28T16:29:11.098 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:29:11.098 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:29:11.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f8d280692b0 msgr2=0x7f8d2806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:11.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f8d280692b0 0x7f8d2806b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8d30013740 tx=0x7f8d30009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:11.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 msgr2=0x7f8d4810fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:11.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 0x7f8d4810fae0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8d3800eb40 tx=0x7f8d3800b040 comp rx=0 tx=0).stop 2024-02-28T16:29:11.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 shutdown_connections 2024-02-28T16:29:11.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:6800/762266793,v1:172.21.15.1:6801/762266793] conn(0x7f8d280692b0 0x7f8d2806b760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:11.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d48070860 0x7f8d4810fae0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:11.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 --2- 172.21.15.1:0/416168982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d48071eb0 0x7f8d48110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:11.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.111+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 >> 172.21.15.1:0/416168982 conn(0x7f8d480f8620 msgr2=0x7f8d48102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:11.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.112+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 shutdown_connections 2024-02-28T16:29:11.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:11.112+0000 7f8d4ec00700 1 -- 172.21.15.1:0/416168982 wait complete. 2024-02-28T16:29:11.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:11 smithi066 ceph-mon[36236]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:11.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:11 smithi066 ceph-mon[36236]: pgmap v295: 65 pgs: 65 active+clean; 7.2 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 565 KiB/s wr, 3 op/s 2024-02-28T16:29:11.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:11 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/424560094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:29:11.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:11 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:29:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:11 smithi001 ceph-mon[28991]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:11 smithi001 ceph-mon[28991]: pgmap v295: 65 pgs: 65 active+clean; 7.2 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 565 KiB/s wr, 3 op/s 2024-02-28T16:29:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:11 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/424560094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:29:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:11 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:29:12.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:12 smithi066 ceph-mon[36236]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:13.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:12 smithi001 ceph-mon[28991]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:13.091 INFO:tasks.workunit.client.1.smithi066.stderr:+ tar xzf ltp-full.tgz 2024-02-28T16:29:13.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:13 smithi066 ceph-mon[36236]: pgmap v296: 65 pgs: 65 active+clean; 7.3 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 572 KiB/s wr, 4 op/s 2024-02-28T16:29:14.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:13 smithi001 ceph-mon[28991]: pgmap v296: 65 pgs: 65 active+clean; 7.3 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 572 KiB/s wr, 4 op/s 2024-02-28T16:29:15.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:15 smithi066 ceph-mon[36236]: pgmap v297: 65 pgs: 65 active+clean; 20 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.6 MiB/s wr, 19 op/s 2024-02-28T16:29:15.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:15 smithi001 ceph-mon[28991]: pgmap v297: 65 pgs: 65 active+clean; 20 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.6 MiB/s wr, 19 op/s 2024-02-28T16:29:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:17 smithi066 ceph-mon[36236]: pgmap v298: 65 pgs: 65 active+clean; 21 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.7 MiB/s wr, 26 op/s 2024-02-28T16:29:18.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:17 smithi001 ceph-mon[28991]: pgmap v298: 65 pgs: 65 active+clean; 21 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.7 MiB/s wr, 26 op/s 2024-02-28T16:29:19.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:19 smithi066 ceph-mon[36236]: pgmap v299: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 35 op/s 2024-02-28T16:29:20.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:19 smithi001 ceph-mon[28991]: pgmap v299: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 35 op/s 2024-02-28T16:29:21.731 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:21 smithi066 ceph-mon[36236]: pgmap v300: 65 pgs: 65 active+clean; 29 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.4 MiB/s wr, 54 op/s 2024-02-28T16:29:22.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:21 smithi001 ceph-mon[28991]: pgmap v300: 65 pgs: 65 active+clean; 29 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.4 MiB/s wr, 54 op/s 2024-02-28T16:29:23.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:23 smithi066 ceph-mon[36236]: pgmap v301: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 57 op/s 2024-02-28T16:29:24.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:23 smithi001 ceph-mon[28991]: pgmap v301: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 57 op/s 2024-02-28T16:29:25.946 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:25 smithi066 ceph-mon[36236]: pgmap v302: 65 pgs: 65 active+clean; 35 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.3 MiB/s wr, 70 op/s 2024-02-28T16:29:25.946 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:25 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:29:25.946 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:25 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:25.946 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:25 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:25.946 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:25 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:29:26.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:25 smithi001 ceph-mon[28991]: pgmap v302: 65 pgs: 65 active+clean; 35 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2.3 MiB/s wr, 70 op/s 2024-02-28T16:29:26.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:25 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:29:26.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:25 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:26.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:25 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:26.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:25 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:29:26.940 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:26.940 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:26.940 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:26 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:29:27.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:27.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:27.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:26 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:29:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:27 smithi066 ceph-mon[36236]: pgmap v303: 65 pgs: 65 active+clean; 35 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 64 op/s 2024-02-28T16:29:28.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:27 smithi001 ceph-mon[28991]: pgmap v303: 65 pgs: 65 active+clean; 35 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 64 op/s 2024-02-28T16:29:30.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:29 smithi066 ceph-mon[36236]: pgmap v304: 65 pgs: 65 active+clean; 35 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 63 op/s 2024-02-28T16:29:30.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:29 smithi001 ceph-mon[28991]: pgmap v304: 65 pgs: 65 active+clean; 35 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 63 op/s 2024-02-28T16:29:31.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:31.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:30 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:31.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:30 smithi066 ceph-mon[36236]: pgmap v305: 65 pgs: 65 active+clean; 42 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.4 MiB/s wr, 68 op/s 2024-02-28T16:29:31.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:31.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:30 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:31.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:30 smithi001 ceph-mon[28991]: pgmap v305: 65 pgs: 65 active+clean; 42 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.4 MiB/s wr, 68 op/s 2024-02-28T16:29:32.905 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:32.905 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:32 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:33.139 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:33.139 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:32 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:34.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:33 smithi066 ceph-mon[36236]: pgmap v306: 65 pgs: 65 active+clean; 42 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.1 MiB/s wr, 52 op/s 2024-02-28T16:29:34.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:33 smithi001 ceph-mon[28991]: pgmap v306: 65 pgs: 65 active+clean; 42 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.1 MiB/s wr, 52 op/s 2024-02-28T16:29:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:35 smithi066 ceph-mon[36236]: pgmap v307: 65 pgs: 65 active+clean; 45 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 66 op/s 2024-02-28T16:29:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:35 smithi001 ceph-mon[28991]: pgmap v307: 65 pgs: 65 active+clean; 45 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 66 op/s 2024-02-28T16:29:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:36 smithi066 ceph-mon[36236]: Standby manager daemon smithi066.ztraxe restarted 2024-02-28T16:29:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:36 smithi066 ceph-mon[36236]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:29:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:36 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:29:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:36 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:29:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:36 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:29:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:36 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:29:36.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:36 smithi001 ceph-mon[28991]: Standby manager daemon smithi066.ztraxe restarted 2024-02-28T16:29:36.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:36 smithi001 ceph-mon[28991]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:29:36.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:36 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:29:36.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:36 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:29:36.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:36 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:29:36.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:36 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:29:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:37 smithi066 ceph-mon[36236]: mgrmap e20: smithi001.rmthtx(active, since 9m), standbys: smithi066.ztraxe 2024-02-28T16:29:37.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:37 smithi066 ceph-mon[36236]: pgmap v308: 65 pgs: 65 active+clean; 49 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 71 op/s 2024-02-28T16:29:37.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:37 smithi001 ceph-mon[28991]: mgrmap e20: smithi001.rmthtx(active, since 9m), standbys: smithi066.ztraxe 2024-02-28T16:29:37.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:37 smithi001 ceph-mon[28991]: pgmap v308: 65 pgs: 65 active+clean; 49 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 71 op/s 2024-02-28T16:29:38.751 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:38.751 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:38.751 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:29:38.751 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:29:38.751 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:38.751 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:29:38.751 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr fail", "who": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: osdmap e46: 6 total, 6 up, 6 in 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "mgr fail", "who": "smithi001.rmthtx"}]': finished 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: mgrmap e21: smithi066.ztraxe(active, starting, since 0.020016s) 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr metadata", "who": "smithi066.ztraxe", "id": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:29:38.752 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:29:38.753 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:29:38.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr fail", "who": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: osdmap e46: 6 total, 6 up, 6 in 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.14212 172.21.15.1:0/3420427157' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "mgr fail", "who": "smithi001.rmthtx"}]': finished 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: mgrmap e21: smithi066.ztraxe(active, starting, since 0.020016s) 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:29:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr metadata", "who": "smithi066.ztraxe", "id": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: Manager daemon smithi066.ztraxe is now available 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:39.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:29:39.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:29:39.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:29:39.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi066.ztraxe/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:29:39.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi066.ztraxe/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:29:39.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi066.ztraxe/trash_purge_schedule"}]: dispatch 2024-02-28T16:29:39.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:38 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi066.ztraxe/trash_purge_schedule"}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: Manager daemon smithi066.ztraxe is now available 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi066.ztraxe/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:29:39.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi066.ztraxe/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:29:39.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi066.ztraxe/trash_purge_schedule"}]: dispatch 2024-02-28T16:29:39.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:38 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi066.ztraxe/trash_purge_schedule"}]: dispatch 2024-02-28T16:29:40.429 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:40 smithi001 ceph-mon[28991]: Deploying cephadm binary to smithi001 2024-02-28T16:29:40.430 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:40 smithi001 ceph-mon[28991]: Deploying cephadm binary to smithi066 2024-02-28T16:29:40.430 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:40 smithi001 ceph-mon[28991]: mgrmap e22: smithi066.ztraxe(active, since 1.0157s) 2024-02-28T16:29:40.430 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:40 smithi001 ceph-mon[28991]: pgmap v3: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:40.430 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:40 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:40.430 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:40 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:40 smithi066 ceph-mon[36236]: Deploying cephadm binary to smithi001 2024-02-28T16:29:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:40 smithi066 ceph-mon[36236]: Deploying cephadm binary to smithi066 2024-02-28T16:29:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:40 smithi066 ceph-mon[36236]: mgrmap e22: smithi066.ztraxe(active, since 1.0157s) 2024-02-28T16:29:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:40 smithi066 ceph-mon[36236]: pgmap v3: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:40 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:40 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:41.143 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:41 smithi066 ceph-mon[36236]: pgmap v4: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:41.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.287+0000 7f11f9a00700 1 -- 172.21.15.1:0/3401233019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11f4102250 msgr2=0x7f11f4104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:41.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.287+0000 7f11f9a00700 1 --2- 172.21.15.1:0/3401233019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11f4102250 0x7f11f4104630 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f11e0004940 tx=0x7f11e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.287+0000 7f11f9a00700 1 -- 172.21.15.1:0/3401233019 shutdown_connections 2024-02-28T16:29:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.287+0000 7f11f9a00700 1 --2- 172.21.15.1:0/3401233019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11f4102250 0x7f11f4104630 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.287+0000 7f11f9a00700 1 --2- 172.21.15.1:0/3401233019 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11f40663c0 0x7f11f4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.287+0000 7f11f9a00700 1 -- 172.21.15.1:0/3401233019 >> 172.21.15.1:0/3401233019 conn(0x7f11f40f72c0 msgr2=0x7f11f40f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.288+0000 7f11f9a00700 1 -- 172.21.15.1:0/3401233019 shutdown_connections 2024-02-28T16:29:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.288+0000 7f11f9a00700 1 -- 172.21.15.1:0/3401233019 wait complete. 2024-02-28T16:29:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.289+0000 7f11f9a00700 1 Processor -- start 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.289+0000 7f11f9a00700 1 -- start start 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.289+0000 7f11f9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11f40663c0 0x7f11f4113d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.289+0000 7f11f9a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11f4102250 0x7f11f4114290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.289+0000 7f11f9a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f4115050 con 0x7f11f40663c0 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.289+0000 7f11f9a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f41151c0 con 0x7f11f4102250 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.290+0000 7f11f3400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11f4102250 0x7f11f4114290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.290+0000 7f11f3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11f40663c0 0x7f11f4113d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.290+0000 7f11f3400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11f4102250 0x7f11f4114290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:51610/0 (socket says 172.21.15.1:51610) 2024-02-28T16:29:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.290+0000 7f11f3400700 1 -- 172.21.15.1:0/2489919853 learned_addr learned my addr 172.21.15.1:0/2489919853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.291+0000 7f11f3400700 1 -- 172.21.15.1:0/2489919853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11f40663c0 msgr2=0x7f11f4113d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.291+0000 7f11f3400700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11f40663c0 0x7f11f4113d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.291+0000 7f11f3400700 1 -- 172.21.15.1:0/2489919853 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e0005040 con 0x7f11f4102250 2024-02-28T16:29:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.291+0000 7f11f3400700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11f4102250 0x7f11f4114290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009fd0 tx=0x7f11e0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.291+0000 7f11ebe00700 1 -- 172.21.15.1:0/2489919853 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11e0021070 con 0x7f11f4102250 2024-02-28T16:29:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.291+0000 7f11ebe00700 1 -- 172.21.15.1:0/2489919853 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11e0013de0 con 0x7f11f4102250 2024-02-28T16:29:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.291+0000 7f11ebe00700 1 -- 172.21.15.1:0/2489919853 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11e000e070 con 0x7f11f4102250 2024-02-28T16:29:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.291+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f41153e0 con 0x7f11f4102250 2024-02-28T16:29:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.292+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f41159a0 con 0x7f11f4102250 2024-02-28T16:29:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.293+0000 7f11ebe00700 1 -- 172.21.15.1:0/2489919853 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 23) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f11e0026410 con 0x7f11f4102250 2024-02-28T16:29:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.294+0000 7f11ebe00700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f11d8038a70 0x7f11d803af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.294+0000 7f11ebe00700 1 -- 172.21.15.1:0/2489919853 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f11e004d780 con 0x7f11f4102250 2024-02-28T16:29:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.294+0000 7f11f3e00700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f11d8038a70 0x7f11d803af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.294+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f40ff060 con 0x7f11f4102250 2024-02-28T16:29:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.295+0000 7f11f3e00700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f11d8038a70 0x7f11d803af20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11f4113ac0 tx=0x7f11e400b040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:41.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.305+0000 7f11ebe00700 1 -- 172.21.15.1:0/2489919853 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f11e0024310 con 0x7f11f4102250 2024-02-28T16:29:41.465 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:41 smithi001 ceph-mon[28991]: pgmap v4: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:41.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.604+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11f4115c80 con 0x7f11d8038a70 2024-02-28T16:29:41.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.609+0000 7f11ebe00700 1 -- 172.21.15.1:0/2489919853 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f11f4115c80 con 0x7f11d8038a70 2024-02-28T16:29:41.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f11d8038a70 msgr2=0x7f11d803af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f11d8038a70 0x7f11d803af20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11f4113ac0 tx=0x7f11e400b040 comp rx=0 tx=0).stop 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11f4102250 msgr2=0x7f11f4114290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11f4102250 0x7f11f4114290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009fd0 tx=0x7f11e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 shutdown_connections 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f11d8038a70 0x7f11d803af20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11f40663c0 0x7f11f4113d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 --2- 172.21.15.1:0/2489919853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11f4102250 0x7f11f4114290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 >> 172.21.15.1:0/2489919853 conn(0x7f11f40f72c0 msgr2=0x7f11f40f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 shutdown_connections 2024-02-28T16:29:41.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.612+0000 7f11f9a00700 1 -- 172.21.15.1:0/2489919853 wait complete. 2024-02-28T16:29:41.618 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:29:41.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.805+0000 7f655c800700 1 -- 172.21.15.1:0/796282223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65540fdeb0 msgr2=0x7f6554104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:41.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.805+0000 7f655c800700 1 --2- 172.21.15.1:0/796282223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65540fdeb0 0x7f6554104770 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6544004940 tx=0x7f6544009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:41.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.806+0000 7f655c800700 1 -- 172.21.15.1:0/796282223 shutdown_connections 2024-02-28T16:29:41.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.806+0000 7f655c800700 1 --2- 172.21.15.1:0/796282223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65540fdeb0 0x7f6554104770 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.806+0000 7f655c800700 1 --2- 172.21.15.1:0/796282223 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65540fd4d0 0x7f65540fd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.806+0000 7f655c800700 1 -- 172.21.15.1:0/796282223 >> 172.21.15.1:0/796282223 conn(0x7f65540f6ca0 msgr2=0x7f65540f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:41.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.806+0000 7f655c800700 1 -- 172.21.15.1:0/796282223 shutdown_connections 2024-02-28T16:29:41.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.806+0000 7f655c800700 1 -- 172.21.15.1:0/796282223 wait complete. 2024-02-28T16:29:41.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.807+0000 7f655c800700 1 Processor -- start 2024-02-28T16:29:41.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.807+0000 7f655c800700 1 -- start start 2024-02-28T16:29:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.807+0000 7f655c800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65540fd4d0 0x7f655410fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.807+0000 7f655c800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6554110200 0x7f6554110f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.807+0000 7f655c800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65541106c0 con 0x7f6554110200 2024-02-28T16:29:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.807+0000 7f655c800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6554110800 con 0x7f65540fd4d0 2024-02-28T16:29:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.808+0000 7f6559e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65540fd4d0 0x7f655410fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.808+0000 7f6559e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65540fd4d0 0x7f655410fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:51626/0 (socket says 172.21.15.1:51626) 2024-02-28T16:29:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.808+0000 7f6559e00700 1 -- 172.21.15.1:0/240602089 learned_addr learned my addr 172.21.15.1:0/240602089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.808+0000 7f6559400700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6554110200 0x7f6554110f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.809+0000 7f6559400700 1 -- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65540fd4d0 msgr2=0x7f655410fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.809+0000 7f6559400700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65540fd4d0 0x7f655410fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.809+0000 7f6559400700 1 -- 172.21.15.1:0/240602089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6544005040 con 0x7f6554110200 2024-02-28T16:29:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.809+0000 7f6559400700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6554110200 0x7f6554110f60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6554065600 tx=0x7f6544009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.810+0000 7f654e000700 1 -- 172.21.15.1:0/240602089 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f654400e040 con 0x7f6554110200 2024-02-28T16:29:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.810+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65541115e0 con 0x7f6554110200 2024-02-28T16:29:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.810+0000 7f654e000700 1 -- 172.21.15.1:0/240602089 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6544004290 con 0x7f6554110200 2024-02-28T16:29:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.810+0000 7f654e000700 1 -- 172.21.15.1:0/240602089 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f654401c3f0 con 0x7f6554110200 2024-02-28T16:29:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.810+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6554111b20 con 0x7f6554110200 2024-02-28T16:29:41.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.811+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655404bed0 con 0x7f6554110200 2024-02-28T16:29:41.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.812+0000 7f654e000700 1 -- 172.21.15.1:0/240602089 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 23) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f6544013420 con 0x7f6554110200 2024-02-28T16:29:41.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.812+0000 7f654e000700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f653c038a70 0x7f653c03af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:41.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.812+0000 7f654e000700 1 -- 172.21.15.1:0/240602089 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f654404d6b0 con 0x7f6554110200 2024-02-28T16:29:41.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.813+0000 7f6559e00700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f653c038a70 0x7f653c03af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:41.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.814+0000 7f6559e00700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f653c038a70 0x7f653c03af20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f65480041d0 tx=0x7f654800b040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:41.820+0000 7f654e000700 1 -- 172.21.15.1:0/240602089 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f654400bbf0 con 0x7f6554110200 2024-02-28T16:29:42.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.123+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f65540fe060 con 0x7f653c038a70 2024-02-28T16:29:42.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.132+0000 7f654e000700 1 -- 172.21.15.1:0/240602089 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f65540fe060 con 0x7f653c038a70 2024-02-28T16:29:42.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f653c038a70 msgr2=0x7f653c03af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f653c038a70 0x7f653c03af20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f65480041d0 tx=0x7f654800b040 comp rx=0 tx=0).stop 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6554110200 msgr2=0x7f6554110f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6554110200 0x7f6554110f60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6554065600 tx=0x7f6544009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 shutdown_connections 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f653c038a70 0x7f653c03af20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65540fd4d0 0x7f655410fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 --2- 172.21.15.1:0/240602089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6554110200 0x7f6554110f60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 >> 172.21.15.1:0/240602089 conn(0x7f65540f6ca0 msgr2=0x7f65540f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 shutdown_connections 2024-02-28T16:29:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.135+0000 7f655c800700 1 -- 172.21.15.1:0/240602089 wait complete. 2024-02-28T16:29:42.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.310+0000 7f02a2a00700 1 -- 172.21.15.1:0/2812568377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 msgr2=0x7f029c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:42.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.310+0000 7f02a2a00700 1 --2- 172.21.15.1:0/2812568377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 0x7f029c0fd320 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f028c004940 tx=0x7f028c009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:42.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.311+0000 7f02a2a00700 1 -- 172.21.15.1:0/2812568377 shutdown_connections 2024-02-28T16:29:42.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.311+0000 7f02a2a00700 1 --2- 172.21.15.1:0/2812568377 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f029c0fe0e0 0x7f029c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.311+0000 7f02a2a00700 1 --2- 172.21.15.1:0/2812568377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 0x7f029c0fd320 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.311+0000 7f02a2a00700 1 -- 172.21.15.1:0/2812568377 >> 172.21.15.1:0/2812568377 conn(0x7f029c0f8620 msgr2=0x7f029c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:42.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.311+0000 7f02a2a00700 1 -- 172.21.15.1:0/2812568377 shutdown_connections 2024-02-28T16:29:42.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.311+0000 7f02a2a00700 1 -- 172.21.15.1:0/2812568377 wait complete. 2024-02-28T16:29:42.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.312+0000 7f02a2a00700 1 Processor -- start 2024-02-28T16:29:42.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.312+0000 7f02a2a00700 1 -- start start 2024-02-28T16:29:42.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.312+0000 7f02a2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 0x7f029c10fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:42.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.312+0000 7f02a2a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f029c0fe0e0 0x7f029c1100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:42.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.312+0000 7f02a2a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c1106c0 con 0x7f029c0fcef0 2024-02-28T16:29:42.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.312+0000 7f02a2a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c110f60 con 0x7f029c0fe0e0 2024-02-28T16:29:42.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.313+0000 7f029be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 0x7f029c10fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:42.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.313+0000 7f029be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 0x7f029c10fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41144/0 (socket says 172.21.15.1:41144) 2024-02-28T16:29:42.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.313+0000 7f029be00700 1 -- 172.21.15.1:0/306217123 learned_addr learned my addr 172.21.15.1:0/306217123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:42.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.313+0000 7f029b400700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f029c0fe0e0 0x7f029c1100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:42.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.313+0000 7f029be00700 1 -- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f029c0fe0e0 msgr2=0x7f029c1100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:42.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.313+0000 7f029be00700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f029c0fe0e0 0x7f029c1100c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.313+0000 7f029be00700 1 -- 172.21.15.1:0/306217123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c005040 con 0x7f029c0fcef0 2024-02-28T16:29:42.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.314+0000 7f029be00700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 0x7f029c10fb80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f028c000c00 tx=0x7f028c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:42.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.314+0000 7f0293e00700 1 -- 172.21.15.1:0/306217123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f028c00e040 con 0x7f029c0fcef0 2024-02-28T16:29:42.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.314+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c1111e0 con 0x7f029c0fcef0 2024-02-28T16:29:42.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.315+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c111720 con 0x7f029c0fcef0 2024-02-28T16:29:42.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.315+0000 7f0293e00700 1 -- 172.21.15.1:0/306217123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f028c00bbf0 con 0x7f029c0fcef0 2024-02-28T16:29:42.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.315+0000 7f0293e00700 1 -- 172.21.15.1:0/306217123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f028c01c630 con 0x7f029c0fcef0 2024-02-28T16:29:42.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.316+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c0052e0 con 0x7f029c0fcef0 2024-02-28T16:29:42.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.318+0000 7f0293e00700 1 -- 172.21.15.1:0/306217123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 23) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f028c00bd60 con 0x7f029c0fcef0 2024-02-28T16:29:42.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.319+0000 7f0293e00700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0280038a60 0x7f028003af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:42.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.319+0000 7f0293e00700 1 -- 172.21.15.1:0/306217123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f028c04e3f0 con 0x7f029c0fcef0 2024-02-28T16:29:42.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.320+0000 7f029b400700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0280038a60 0x7f028003af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:42.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.321+0000 7f029b400700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0280038a60 0x7f028003af10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0288004f40 tx=0x7f028800b040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:42.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.324+0000 7f0293e00700 1 -- 172.21.15.1:0/306217123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f028c01b8a0 con 0x7f029c0fcef0 2024-02-28T16:29:42.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:42 smithi066 ceph-mon[36236]: [28/Feb/2024:16:29:41] ENGINE Bus STARTING 2024-02-28T16:29:42.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:42 smithi066 ceph-mon[36236]: [28/Feb/2024:16:29:41] ENGINE Serving on http://172.21.15.66:8765 2024-02-28T16:29:42.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:42 smithi066 ceph-mon[36236]: mgrmap e23: smithi066.ztraxe(active, since 3s) 2024-02-28T16:29:42.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:42 smithi066 ceph-mon[36236]: [28/Feb/2024:16:29:41] ENGINE Serving on https://172.21.15.66:7150 2024-02-28T16:29:42.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:42 smithi066 ceph-mon[36236]: [28/Feb/2024:16:29:41] ENGINE Bus STARTED 2024-02-28T16:29:42.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:42 smithi066 ceph-mon[36236]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:42.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:42 smithi001 ceph-mon[28991]: [28/Feb/2024:16:29:41] ENGINE Bus STARTING 2024-02-28T16:29:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:42 smithi001 ceph-mon[28991]: [28/Feb/2024:16:29:41] ENGINE Serving on http://172.21.15.66:8765 2024-02-28T16:29:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:42 smithi001 ceph-mon[28991]: mgrmap e23: smithi066.ztraxe(active, since 3s) 2024-02-28T16:29:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:42 smithi001 ceph-mon[28991]: [28/Feb/2024:16:29:41] ENGINE Serving on https://172.21.15.66:7150 2024-02-28T16:29:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:42 smithi001 ceph-mon[28991]: [28/Feb/2024:16:29:41] ENGINE Bus STARTED 2024-02-28T16:29:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:42 smithi001 ceph-mon[28991]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:42.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.592+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f027c000bf0 con 0x7f0280038a60 2024-02-28T16:29:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.614+0000 7f0293e00700 1 -- 172.21.15.1:0/306217123 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f027c000bf0 con 0x7f0280038a60 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (8m) 2m ago 10m 13.1M - 0.25.0 c8568f914cd2 3d10852c3022 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (10m) 2m ago 10m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (8m) 12s ago 8m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (8m) 2m ago 9m 74.2M - 9.4.7 954c08fa6188 2a03709b4c9c 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (2m) 2m ago 2m 13.2M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (2m) 2m ago 2m 10.5M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (2m) 12s ago 2m 11.3M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (2m) 12s ago 2m 32.5M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:9283 running (11m) 2m ago 11m 481M - 17.2.7-535-gc0f02e60 1c37ed371348 8eb0fd290742 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283 running (20s) 12s ago 8m 56.3M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 30a2de341a4e 2024-02-28T16:29:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (11m) 2m ago 12m 42.0M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (8m) 12s ago 8m 32.5M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (10m) 2m ago 10m 7956k - 1.5.0 0da6a335fe13 3dac2d3d42b2 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (8m) 12s ago 8m 7784k - 1.5.0 0da6a335fe13 31630ee26c1c 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (7m) 2m ago 7m 43.7M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (7m) 2m ago 7m 45.5M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (6m) 2m ago 6m 43.4M 4290M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (6m) 12s ago 6m 54.3M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (5m) 12s ago 5m 51.7M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (4m) 12s ago 4m 51.6M 5313M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:29:42.605 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (7m) 2m ago 9m 45.0M - 2.43.0 a07b618ecd1d 21713d7e702b 2024-02-28T16:29:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.618+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0280038a60 msgr2=0x7f028003af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.618+0000 7f02a2a00700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0280038a60 0x7f028003af10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0288004f40 tx=0x7f028800b040 comp rx=0 tx=0).stop 2024-02-28T16:29:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.618+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 msgr2=0x7f029c10fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:42.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.618+0000 7f02a2a00700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 0x7f029c10fb80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f028c000c00 tx=0x7f028c009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:42.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.620+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 shutdown_connections 2024-02-28T16:29:42.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.620+0000 7f02a2a00700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0280038a60 0x7f028003af10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.620+0000 7f02a2a00700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f029c0fcef0 0x7f029c10fb80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.620+0000 7f02a2a00700 1 --2- 172.21.15.1:0/306217123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f029c0fe0e0 0x7f029c1100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.620+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 >> 172.21.15.1:0/306217123 conn(0x7f029c0f8620 msgr2=0x7f029c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:42.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.620+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 shutdown_connections 2024-02-28T16:29:42.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.620+0000 7f02a2a00700 1 -- 172.21.15.1:0/306217123 wait complete. 2024-02-28T16:29:42.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.832+0000 7fac0b600700 1 -- 172.21.15.1:0/927699013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac04070840 msgr2=0x7fac04070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:42.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.832+0000 7fac0b600700 1 --2- 172.21.15.1:0/927699013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac04070840 0x7fac04070cb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fabf0004940 tx=0x7fabf0009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:42.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.832+0000 7fac0b600700 1 -- 172.21.15.1:0/927699013 shutdown_connections 2024-02-28T16:29:42.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.832+0000 7fac0b600700 1 --2- 172.21.15.1:0/927699013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac04070840 0x7fac04070cb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.832+0000 7fac0b600700 1 --2- 172.21.15.1:0/927699013 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac04071e90 0x7fac04070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.832+0000 7fac0b600700 1 -- 172.21.15.1:0/927699013 >> 172.21.15.1:0/927699013 conn(0x7fac040f8620 msgr2=0x7fac040faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:42.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.833+0000 7fac0b600700 1 -- 172.21.15.1:0/927699013 shutdown_connections 2024-02-28T16:29:42.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.833+0000 7fac0b600700 1 -- 172.21.15.1:0/927699013 wait complete. 2024-02-28T16:29:42.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.834+0000 7fac0b600700 1 Processor -- start 2024-02-28T16:29:42.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.834+0000 7fac0b600700 1 -- start start 2024-02-28T16:29:42.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.834+0000 7fac0b600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac04070840 0x7fac0410fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:42.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.835+0000 7fac0b600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac04071e90 0x7fac04110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:42.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.835+0000 7fac0b600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac041106f0 con 0x7fac04070840 2024-02-28T16:29:42.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.835+0000 7fac0b600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac04110f30 con 0x7fac04071e90 2024-02-28T16:29:42.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.835+0000 7fac03e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac04071e90 0x7fac04110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:42.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.835+0000 7fac03e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac04071e90 0x7fac04110060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:34184/0 (socket says 172.21.15.1:34184) 2024-02-28T16:29:42.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.835+0000 7fac03e00700 1 -- 172.21.15.1:0/1803569961 learned_addr learned my addr 172.21.15.1:0/1803569961 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:42.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.836+0000 7fac08c00700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac04070840 0x7fac0410fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.836+0000 7fac03e00700 1 -- 172.21.15.1:0/1803569961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac04070840 msgr2=0x7fac0410fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.836+0000 7fac03e00700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac04070840 0x7fac0410fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.836+0000 7fac03e00700 1 -- 172.21.15.1:0/1803569961 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf0005040 con 0x7fac04071e90 2024-02-28T16:29:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.836+0000 7fac03e00700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac04071e90 0x7fac04110060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fabf000b3b0 tx=0x7fabf0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:42.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.837+0000 7fac00c00700 1 -- 172.21.15.1:0/1803569961 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabf000e040 con 0x7fac04071e90 2024-02-28T16:29:42.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.837+0000 7fac00c00700 1 -- 172.21.15.1:0/1803569961 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabf0004290 con 0x7fac04071e90 2024-02-28T16:29:42.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.837+0000 7fac00c00700 1 -- 172.21.15.1:0/1803569961 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabf001c630 con 0x7fac04071e90 2024-02-28T16:29:42.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.838+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac041111b0 con 0x7fac04071e90 2024-02-28T16:29:42.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.839+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac04111670 con 0x7fac04071e90 2024-02-28T16:29:42.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.840+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0404bed0 con 0x7fac04071e90 2024-02-28T16:29:42.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.841+0000 7fac00c00700 1 -- 172.21.15.1:0/1803569961 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 23) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fabf0013420 con 0x7fac04071e90 2024-02-28T16:29:42.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.842+0000 7fac00c00700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fabec038a70 0x7fabec03af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:42.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.842+0000 7fac00c00700 1 -- 172.21.15.1:0/1803569961 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fabf004d540 con 0x7fac04071e90 2024-02-28T16:29:42.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.842+0000 7fac08c00700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fabec038a70 0x7fabec03af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:42.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.843+0000 7fac08c00700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fabec038a70 0x7fabec03af20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fabf8006fd0 tx=0x7fabf8008040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:42.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:42.859+0000 7fac00c00700 1 -- 172.21.15.1:0/1803569961 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fabf0011960 con 0x7fac04071e90 2024-02-28T16:29:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.385+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fac04107c80 con 0x7fac04071e90 2024-02-28T16:29:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.387+0000 7fac00c00700 1 -- 172.21.15.1:0/1803569961 <== mon.1 v2:172.21.15.66:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fabf0021210 con 0x7fac04071e90 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:29:43.377 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:29:43.378 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:29:43.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:29:43.378 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:29:43.378 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 12, 2024-02-28T16:29:43.378 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:29:43.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:29:43.378 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:29:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.390+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fabec038a70 msgr2=0x7fabec03af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.390+0000 7fac0b600700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fabec038a70 0x7fabec03af20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fabf8006fd0 tx=0x7fabf8008040 comp rx=0 tx=0).stop 2024-02-28T16:29:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.390+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac04071e90 msgr2=0x7fac04110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.390+0000 7fac0b600700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac04071e90 0x7fac04110060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fabf000b3b0 tx=0x7fabf0009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.391+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 shutdown_connections 2024-02-28T16:29:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.391+0000 7fac0b600700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fabec038a70 0x7fabec03af20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.391+0000 7fac0b600700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac04070840 0x7fac0410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.391+0000 7fac0b600700 1 --2- 172.21.15.1:0/1803569961 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac04071e90 0x7fac04110060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.391+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 >> 172.21.15.1:0/1803569961 conn(0x7fac040f8620 msgr2=0x7fac04102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.391+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 shutdown_connections 2024-02-28T16:29:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.391+0000 7fac0b600700 1 -- 172.21.15.1:0/1803569961 wait complete. 2024-02-28T16:29:43.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:43 smithi066 ceph-mon[36236]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:43.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:43 smithi066 ceph-mon[36236]: pgmap v5: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:43.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:43 smithi066 ceph-mon[36236]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:43.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:43 smithi001 ceph-mon[28991]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:43.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:43 smithi001 ceph-mon[28991]: pgmap v5: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:29:43.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:43 smithi001 ceph-mon[28991]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.561+0000 7f076b800700 1 -- 172.21.15.1:0/1811717254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fe0e0 msgr2=0x7f07640fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.561+0000 7f076b800700 1 --2- 172.21.15.1:0/1811717254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fe0e0 0x7f07640fe550 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0750004940 tx=0x7f0750009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.562+0000 7f076b800700 1 -- 172.21.15.1:0/1811717254 shutdown_connections 2024-02-28T16:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.562+0000 7f076b800700 1 --2- 172.21.15.1:0/1811717254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fe0e0 0x7f07640fe550 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.562+0000 7f076b800700 1 --2- 172.21.15.1:0/1811717254 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07640fcef0 0x7f07640fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.562+0000 7f076b800700 1 -- 172.21.15.1:0/1811717254 >> 172.21.15.1:0/1811717254 conn(0x7f07640f8620 msgr2=0x7f07640faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:43.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.562+0000 7f076b800700 1 -- 172.21.15.1:0/1811717254 shutdown_connections 2024-02-28T16:29:43.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.562+0000 7f076b800700 1 -- 172.21.15.1:0/1811717254 wait complete. 2024-02-28T16:29:43.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.563+0000 7f076b800700 1 Processor -- start 2024-02-28T16:29:43.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f076b800700 1 -- start start 2024-02-28T16:29:43.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f076b800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fcef0 0x7f0764113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:43.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f076b800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07640fe0e0 0x7f07641144c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:43.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f076b800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764114ac0 con 0x7f07640fcef0 2024-02-28T16:29:43.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f076b800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764115360 con 0x7f07640fe0e0 2024-02-28T16:29:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f0768e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fcef0 0x7f0764113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f0768e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fcef0 0x7f0764113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41174/0 (socket says 172.21.15.1:41174) 2024-02-28T16:29:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f0768e00700 1 -- 172.21.15.1:0/4131243596 learned_addr learned my addr 172.21.15.1:0/4131243596 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:29:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.564+0000 7f0763e00700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07640fe0e0 0x7f07641144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:43.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.565+0000 7f0768e00700 1 -- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07640fe0e0 msgr2=0x7f07641144c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.565+0000 7f0768e00700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07640fe0e0 0x7f07641144c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.565+0000 7f0768e00700 1 -- 172.21.15.1:0/4131243596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0750005040 con 0x7f07640fcef0 2024-02-28T16:29:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.566+0000 7f0768e00700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fcef0 0x7f0764113f80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0758002a50 tx=0x7f0758004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.566+0000 7f0760c00700 1 -- 172.21.15.1:0/4131243596 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0758007cc0 con 0x7f07640fcef0 2024-02-28T16:29:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.566+0000 7f0760c00700 1 -- 172.21.15.1:0/4131243596 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f075800f040 con 0x7f07640fcef0 2024-02-28T16:29:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.566+0000 7f0760c00700 1 -- 172.21.15.1:0/4131243596 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f075800d940 con 0x7f07640fcef0 2024-02-28T16:29:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.567+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0764115640 con 0x7f07640fcef0 2024-02-28T16:29:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.567+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0764115b80 con 0x7f07640fcef0 2024-02-28T16:29:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.568+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0764102210 con 0x7f07640fcef0 2024-02-28T16:29:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.569+0000 7f0760c00700 1 -- 172.21.15.1:0/4131243596 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 23) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f075800dbb0 con 0x7f07640fcef0 2024-02-28T16:29:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.570+0000 7f0760c00700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f074c038a70 0x7f074c03af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:29:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.570+0000 7f0760c00700 1 -- 172.21.15.1:0/4131243596 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f075804c880 con 0x7f07640fcef0 2024-02-28T16:29:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.570+0000 7f0763e00700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f074c038a70 0x7f074c03af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:29:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.572+0000 7f0763e00700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f074c038a70 0x7f074c03af20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f07500043d0 tx=0x7f0750009e30 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:29:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.576+0000 7f0760c00700 1 -- 172.21.15.1:0/4131243596 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f0758024900 con 0x7f07640fcef0 2024-02-28T16:29:43.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.933+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0764115f20 con 0x7f074c038a70 2024-02-28T16:29:43.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.944+0000 7f0760c00700 1 -- 172.21.15.1:0/4131243596 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f0764115f20 con 0x7f074c038a70 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [], 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "1/2 daemons upgraded", 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "", 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:29:43.934 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:29:43.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.948+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f074c038a70 msgr2=0x7f074c03af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:43.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.948+0000 7f076b800700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f074c038a70 0x7f074c03af20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f07500043d0 tx=0x7f0750009e30 comp rx=0 tx=0).stop 2024-02-28T16:29:43.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.948+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fcef0 msgr2=0x7f0764113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:29:43.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.948+0000 7f076b800700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fcef0 0x7f0764113f80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0758002a50 tx=0x7f0758004600 comp rx=0 tx=0).stop 2024-02-28T16:29:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.949+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 shutdown_connections 2024-02-28T16:29:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.949+0000 7f076b800700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f074c038a70 0x7f074c03af20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.949+0000 7f076b800700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640fcef0 0x7f0764113f80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.949+0000 7f076b800700 1 --2- 172.21.15.1:0/4131243596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07640fe0e0 0x7f07641144c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:29:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.949+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 >> 172.21.15.1:0/4131243596 conn(0x7f07640f8620 msgr2=0x7f07640ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:29:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.949+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 shutdown_connections 2024-02-28T16:29:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:29:43.949+0000 7f076b800700 1 -- 172.21.15.1:0/4131243596 wait complete. 2024-02-28T16:29:44.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:44 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1803569961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:29:44.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:44 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1803569961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:29:45.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:45 smithi001 ceph-mon[28991]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:45.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:45 smithi001 ceph-mon[28991]: pgmap v6: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 483 KiB/s wr, 22 op/s 2024-02-28T16:29:45.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:45 smithi066 ceph-mon[36236]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:29:45.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:45 smithi066 ceph-mon[36236]: pgmap v6: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 483 KiB/s wr, 22 op/s 2024-02-28T16:29:47.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:47 smithi066 ceph-mon[36236]: pgmap v7: 65 pgs: 65 active+clean; 71 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.7 MiB/s wr, 71 op/s 2024-02-28T16:29:47.761 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:47 smithi001 ceph-mon[28991]: pgmap v7: 65 pgs: 65 active+clean; 71 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.7 MiB/s wr, 71 op/s 2024-02-28T16:29:49.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:49 smithi001 ceph-mon[28991]: pgmap v8: 65 pgs: 65 active+clean; 71 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-02-28T16:29:49.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:49 smithi001 ceph-mon[28991]: Standby manager daemon smithi001.rmthtx started 2024-02-28T16:29:49.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:49 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.1:0/4101980120' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi001.rmthtx/crt"}]: dispatch 2024-02-28T16:29:49.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:49 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.1:0/4101980120' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:29:49.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:49 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.1:0/4101980120' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi001.rmthtx/key"}]: dispatch 2024-02-28T16:29:49.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:49 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.1:0/4101980120' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:29:49.692 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:49 smithi066 ceph-mon[36236]: pgmap v8: 65 pgs: 65 active+clean; 71 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-02-28T16:29:49.692 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:49 smithi066 ceph-mon[36236]: Standby manager daemon smithi001.rmthtx started 2024-02-28T16:29:49.692 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:49 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.1:0/4101980120' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi001.rmthtx/crt"}]: dispatch 2024-02-28T16:29:49.692 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:49 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.1:0/4101980120' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:29:49.692 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:49 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.1:0/4101980120' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi001.rmthtx/key"}]: dispatch 2024-02-28T16:29:49.693 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:49 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.1:0/4101980120' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:29:50.703 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:50 smithi001 ceph-mon[28991]: mgrmap e24: smithi066.ztraxe(active, since 11s), standbys: smithi001.rmthtx 2024-02-28T16:29:50.703 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:50 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:29:50.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:50 smithi066 ceph-mon[36236]: mgrmap e24: smithi066.ztraxe(active, since 11s), standbys: smithi001.rmthtx 2024-02-28T16:29:50.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:50 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:29:51.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:51 smithi066 ceph-mon[36236]: pgmap v9: 65 pgs: 65 active+clean; 93 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 3.1 MiB/s wr, 131 op/s 2024-02-28T16:29:51.749 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:51 smithi001 ceph-mon[28991]: pgmap v9: 65 pgs: 65 active+clean; 93 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 3.1 MiB/s wr, 131 op/s 2024-02-28T16:29:53.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:53 smithi066 ceph-mon[36236]: pgmap v10: 65 pgs: 65 active+clean; 93 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-02-28T16:29:53.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:53 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:53.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:53 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:53.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:53 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:53.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:53 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:29:53.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:53 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:54.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:53 smithi001 ceph-mon[28991]: pgmap v10: 65 pgs: 65 active+clean; 93 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-02-28T16:29:54.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:53 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:54.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:53 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:54.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:53 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:54.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:53 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:29:54.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:53 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:55.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: pgmap v11: 65 pgs: 65 active+clean; 94 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 2.9 MiB/s wr, 132 op/s 2024-02-28T16:29:55.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:55.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:55.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:55.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:55.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:55.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:55.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:55.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:55.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:55.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:55.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:55 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:56.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: pgmap v11: 65 pgs: 65 active+clean; 94 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 2.9 MiB/s wr, 132 op/s 2024-02-28T16:29:56.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:56.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:56.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:56.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:56.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:56.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:56.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:56.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:29:56.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:56.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:56.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:55 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:29:56.900 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:56 smithi066 ceph-mon[36236]: Adjusting osd_memory_target on smithi001 to 1559M 2024-02-28T16:29:57.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:56 smithi001 ceph-mon[28991]: Adjusting osd_memory_target on smithi001 to 1559M 2024-02-28T16:29:57.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:57 smithi066 ceph-mon[36236]: pgmap v12: 65 pgs: 65 active+clean; 101 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 3.3 MiB/s wr, 138 op/s 2024-02-28T16:29:58.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:57 smithi001 ceph-mon[28991]: pgmap v12: 65 pgs: 65 active+clean; 101 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 3.3 MiB/s wr, 138 op/s 2024-02-28T16:29:59.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:29:59 smithi066 ceph-mon[36236]: pgmap v13: 65 pgs: 65 active+clean; 101 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.5 MiB/s wr, 105 op/s 2024-02-28T16:30:00.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:29:59 smithi001 ceph-mon[28991]: pgmap v13: 65 pgs: 65 active+clean; 101 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.5 MiB/s wr, 105 op/s 2024-02-28T16:30:01.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:00 smithi001 ceph-mon[28991]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:30:01.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:00 smithi001 ceph-mon[28991]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:30:01.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:00 smithi001 ceph-mon[28991]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:30:01.030 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:00 smithi066 ceph-mon[36236]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:30:01.030 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:00 smithi066 ceph-mon[36236]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:30:01.031 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:00 smithi066 ceph-mon[36236]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:30:02.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:01 smithi066 ceph-mon[36236]: pgmap v14: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 3.1 MiB/s wr, 126 op/s 2024-02-28T16:30:02.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:01 smithi001 ceph-mon[28991]: pgmap v14: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 3.1 MiB/s wr, 126 op/s 2024-02-28T16:30:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: pgmap v15: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 47 op/s 2024-02-28T16:30:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:03.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: Adjusting osd_memory_target on smithi066 to 2582M 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:30:03.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:03 smithi066 ceph-mon[36236]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:30:04.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: pgmap v15: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 47 op/s 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: Adjusting osd_memory_target on smithi066 to 2582M 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:30:04.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:03 smithi001 ceph-mon[28991]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:30:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:04 smithi066 ceph-mon[36236]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:30:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:04 smithi066 ceph-mon[36236]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:30:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:04 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:30:05.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:04 smithi001 ceph-mon[28991]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:30:05.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:04 smithi001 ceph-mon[28991]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:30:05.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:04 smithi001 ceph-mon[28991]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:30:05.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: pgmap v16: 65 pgs: 65 active+clean; 112 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.6 MiB/s wr, 71 op/s 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:05.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:05 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:06.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:30:06.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:30:06.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: pgmap v16: 65 pgs: 65 active+clean; 112 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.6 MiB/s wr, 71 op/s 2024-02-28T16:30:06.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:30:06.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:30:06.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:06.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:06.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:06.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:06.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:05 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:06 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:06 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:06 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:06 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:06 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:06 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:06 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:06 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:07.907 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:07 smithi001 ceph-mon[28991]: Reconfiguring node-exporter.smithi001 (dependencies changed)... 2024-02-28T16:30:07.907 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:07 smithi001 ceph-mon[28991]: Reconfiguring daemon node-exporter.smithi001 on smithi001 2024-02-28T16:30:07.907 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:07 smithi001 ceph-mon[28991]: pgmap v17: 65 pgs: 65 active+clean; 120 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.2 MiB/s wr, 83 op/s 2024-02-28T16:30:08.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:07 smithi066 ceph-mon[36236]: Reconfiguring node-exporter.smithi001 (dependencies changed)... 2024-02-28T16:30:08.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:07 smithi066 ceph-mon[36236]: Reconfiguring daemon node-exporter.smithi001 on smithi001 2024-02-28T16:30:08.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:07 smithi066 ceph-mon[36236]: pgmap v17: 65 pgs: 65 active+clean; 120 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.2 MiB/s wr, 83 op/s 2024-02-28T16:30:09.150 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:08 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:30:09.199 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:08 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:30:10.171 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:09 smithi066 ceph-mon[36236]: pgmap v18: 65 pgs: 65 active+clean; 120 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.5 MiB/s wr, 68 op/s 2024-02-28T16:30:10.171 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:09 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:10.171 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:09 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:10.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:09 smithi001 ceph-mon[28991]: pgmap v18: 65 pgs: 65 active+clean; 120 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.5 MiB/s wr, 68 op/s 2024-02-28T16:30:10.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:09 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:10.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:09 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:11.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:10 smithi001 ceph-mon[28991]: Reconfiguring alertmanager.smithi001 (dependencies changed)... 2024-02-28T16:30:11.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:10 smithi001 ceph-mon[28991]: Reconfiguring daemon alertmanager.smithi001 on smithi001 2024-02-28T16:30:11.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:10 smithi066 ceph-mon[36236]: Reconfiguring alertmanager.smithi001 (dependencies changed)... 2024-02-28T16:30:11.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:10 smithi066 ceph-mon[36236]: Reconfiguring daemon alertmanager.smithi001 on smithi001 2024-02-28T16:30:12.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:11 smithi066 ceph-mon[36236]: pgmap v19: 65 pgs: 65 active+clean; 127 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.2 MiB/s wr, 98 op/s 2024-02-28T16:30:12.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:11 smithi001 ceph-mon[28991]: pgmap v19: 65 pgs: 65 active+clean; 127 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.2 MiB/s wr, 98 op/s 2024-02-28T16:30:13.140 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:12 smithi001 ceph-mon[28991]: pgmap v20: 65 pgs: 65 active+clean; 127 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.6 MiB/s wr, 77 op/s 2024-02-28T16:30:13.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:12 smithi066 ceph-mon[36236]: pgmap v20: 65 pgs: 65 active+clean; 127 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.6 MiB/s wr, 77 op/s 2024-02-28T16:30:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.141+0000 7f15c0400700 1 -- 172.21.15.1:0/4007313418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 msgr2=0x7f15b80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.141+0000 7f15c0400700 1 --2- 172.21.15.1:0/4007313418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 0x7f15b80fc250 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f15ac004940 tx=0x7f15ac009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.141+0000 7f15c0400700 1 -- 172.21.15.1:0/4007313418 shutdown_connections 2024-02-28T16:30:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.141+0000 7f15c0400700 1 --2- 172.21.15.1:0/4007313418 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b80fcfb0 0x7f15b80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.141+0000 7f15c0400700 1 --2- 172.21.15.1:0/4007313418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 0x7f15b80fc250 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.141+0000 7f15c0400700 1 -- 172.21.15.1:0/4007313418 >> 172.21.15.1:0/4007313418 conn(0x7f15b80f74e0 msgr2=0x7f15b80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.141+0000 7f15c0400700 1 -- 172.21.15.1:0/4007313418 shutdown_connections 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.141+0000 7f15c0400700 1 -- 172.21.15.1:0/4007313418 wait complete. 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.142+0000 7f15c0400700 1 Processor -- start 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.143+0000 7f15c0400700 1 -- start start 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.143+0000 7f15c0400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 0x7f15b8065d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.143+0000 7f15c0400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b80fcfb0 0x7f15b806ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.143+0000 7f15c0400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80662a0 con 0x7f15b80fbe40 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.143+0000 7f15c0400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8066410 con 0x7f15b80fcfb0 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.143+0000 7f15bda00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 0x7f15b8065d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.143+0000 7f15bda00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 0x7f15b8065d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44298/0 (socket says 172.21.15.1:44298) 2024-02-28T16:30:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.143+0000 7f15bda00700 1 -- 172.21.15.1:0/595080605 learned_addr learned my addr 172.21.15.1:0/595080605 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:14.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.144+0000 7f15bd000700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b80fcfb0 0x7f15b806ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:14.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.144+0000 7f15bda00700 1 -- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b80fcfb0 msgr2=0x7f15b806ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:14.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.144+0000 7f15bda00700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b80fcfb0 0x7f15b806ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.144+0000 7f15bda00700 1 -- 172.21.15.1:0/595080605 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac005040 con 0x7f15b80fbe40 2024-02-28T16:30:14.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.144+0000 7f15bda00700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 0x7f15b8065d60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0093b0 tx=0x7f15ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:14.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.144+0000 7f15b5600700 1 -- 172.21.15.1:0/595080605 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15ac00e040 con 0x7f15b80fbe40 2024-02-28T16:30:14.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.144+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b8066630 con 0x7f15b80fbe40 2024-02-28T16:30:14.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.144+0000 7f15b5600700 1 -- 172.21.15.1:0/595080605 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15ac00bcc0 con 0x7f15b80fbe40 2024-02-28T16:30:14.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.145+0000 7f15b5600700 1 -- 172.21.15.1:0/595080605 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15ac01c630 con 0x7f15b80fbe40 2024-02-28T16:30:14.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.145+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b806f410 con 0x7f15b80fbe40 2024-02-28T16:30:14.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.146+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b804bed0 con 0x7f15b80fbe40 2024-02-28T16:30:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.147+0000 7f15b5600700 1 -- 172.21.15.1:0/595080605 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7f15ac0139f0 con 0x7f15b80fbe40 2024-02-28T16:30:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.148+0000 7f15b5600700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f15a006b2e0 0x7f15a006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.148+0000 7f15b5600700 1 -- 172.21.15.1:0/595080605 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f15ac08b0b0 con 0x7f15b80fbe40 2024-02-28T16:30:14.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.149+0000 7f15bd000700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f15a006b2e0 0x7f15a006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:14.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.151+0000 7f15bd000700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f15a006b2e0 0x7f15a006d790 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f15a4004d60 tx=0x7f15a400b040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:14.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.155+0000 7f15b5600700 1 -- 172.21.15.1:0/595080605 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f15ac0550f0 con 0x7f15b80fbe40 2024-02-28T16:30:14.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.461+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f15b8066630 con 0x7f15a006b2e0 2024-02-28T16:30:14.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.465+0000 7f15b5600700 1 -- 172.21.15.1:0/595080605 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f15b8066630 con 0x7f15a006b2e0 2024-02-28T16:30:14.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f15a006b2e0 msgr2=0x7f15a006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:14.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f15a006b2e0 0x7f15a006d790 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f15a4004d60 tx=0x7f15a400b040 comp rx=0 tx=0).stop 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 msgr2=0x7f15b8065d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 0x7f15b8065d60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0093b0 tx=0x7f15ac009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 shutdown_connections 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f15a006b2e0 0x7f15a006d790 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b80fbe40 0x7f15b8065d60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 --2- 172.21.15.1:0/595080605 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15b80fcfb0 0x7f15b806ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 >> 172.21.15.1:0/595080605 conn(0x7f15b80f74e0 msgr2=0x7f15b8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 shutdown_connections 2024-02-28T16:30:14.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.469+0000 7f15c0400700 1 -- 172.21.15.1:0/595080605 wait complete. 2024-02-28T16:30:14.480 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:30:14.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.657+0000 7f49e8000700 1 -- 172.21.15.1:0/1958709861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e00fda80 msgr2=0x7f49e00ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:14.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.657+0000 7f49e8000700 1 --2- 172.21.15.1:0/1958709861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e00fda80 0x7f49e00ffe60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f49d4004940 tx=0x7f49d4009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:14.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.657+0000 7f49e8000700 1 -- 172.21.15.1:0/1958709861 shutdown_connections 2024-02-28T16:30:14.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.657+0000 7f49e8000700 1 --2- 172.21.15.1:0/1958709861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49e01003a0 0x7f49e0102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.657+0000 7f49e8000700 1 --2- 172.21.15.1:0/1958709861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e00fda80 0x7f49e00ffe60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.657+0000 7f49e8000700 1 -- 172.21.15.1:0/1958709861 >> 172.21.15.1:0/1958709861 conn(0x7f49e00f74e0 msgr2=0x7f49e00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:14.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.658+0000 7f49e8000700 1 -- 172.21.15.1:0/1958709861 shutdown_connections 2024-02-28T16:30:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.658+0000 7f49e8000700 1 -- 172.21.15.1:0/1958709861 wait complete. 2024-02-28T16:30:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.659+0000 7f49e8000700 1 Processor -- start 2024-02-28T16:30:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.659+0000 7f49e8000700 1 -- start start 2024-02-28T16:30:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.659+0000 7f49e8000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49e00fda80 0x7f49e006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.659+0000 7f49e8000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e01003a0 0x7f49e006f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.659+0000 7f49e8000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e006faa0 con 0x7f49e01003a0 2024-02-28T16:30:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.659+0000 7f49e8000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0117480 con 0x7f49e00fda80 2024-02-28T16:30:14.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.660+0000 7f49e4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e01003a0 0x7f49e006f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:14.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.660+0000 7f49e4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e01003a0 0x7f49e006f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44324/0 (socket says 172.21.15.1:44324) 2024-02-28T16:30:14.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.660+0000 7f49e4c00700 1 -- 172.21.15.1:0/587151915 learned_addr learned my addr 172.21.15.1:0/587151915 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:14.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.660+0000 7f49e5600700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49e00fda80 0x7f49e006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:14.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.660+0000 7f49e4c00700 1 -- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49e00fda80 msgr2=0x7f49e006ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.660+0000 7f49e4c00700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49e00fda80 0x7f49e006ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.660+0000 7f49e4c00700 1 -- 172.21.15.1:0/587151915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4005040 con 0x7f49e01003a0 2024-02-28T16:30:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.661+0000 7f49e4c00700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e01003a0 0x7f49e006f4a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00eb40 tx=0x7f49cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.661+0000 7f49dd600700 1 -- 172.21.15.1:0/587151915 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49cc007cc0 con 0x7f49e01003a0 2024-02-28T16:30:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.661+0000 7f49dd600700 1 -- 172.21.15.1:0/587151915 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49cc00f040 con 0x7f49e01003a0 2024-02-28T16:30:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.661+0000 7f49dd600700 1 -- 172.21.15.1:0/587151915 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49cc009620 con 0x7f49e01003a0 2024-02-28T16:30:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.661+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e0117700 con 0x7f49e01003a0 2024-02-28T16:30:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.661+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e0117c40 con 0x7f49e01003a0 2024-02-28T16:30:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.663+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e0069110 con 0x7f49e01003a0 2024-02-28T16:30:14.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.664+0000 7f49dd600700 1 -- 172.21.15.1:0/587151915 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7f49cc009780 con 0x7f49e01003a0 2024-02-28T16:30:14.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.665+0000 7f49dd600700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f49c806b2e0 0x7f49c806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:14.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.665+0000 7f49dd600700 1 -- 172.21.15.1:0/587151915 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f49cc089930 con 0x7f49e01003a0 2024-02-28T16:30:14.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.666+0000 7f49e5600700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f49c806b2e0 0x7f49c806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:14.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.668+0000 7f49e5600700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f49c806b2e0 0x7f49c806d790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49d400b0d0 tx=0x7f49d4009e30 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:14.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:14.670+0000 7f49dd600700 1 -- 172.21.15.1:0/587151915 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f49cc04f620 con 0x7f49e01003a0 2024-02-28T16:30:15.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.014+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49e002b310 con 0x7f49c806b2e0 2024-02-28T16:30:15.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.017+0000 7f49dd600700 1 -- 172.21.15.1:0/587151915 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f49e002b310 con 0x7f49c806b2e0 2024-02-28T16:30:15.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f49c806b2e0 msgr2=0x7f49c806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:15.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f49c806b2e0 0x7f49c806d790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49d400b0d0 tx=0x7f49d4009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e01003a0 msgr2=0x7f49e006f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e01003a0 0x7f49e006f4a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00eb40 tx=0x7f49cc00b040 comp rx=0 tx=0).stop 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 shutdown_connections 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f49c806b2e0 0x7f49c806d790 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49e00fda80 0x7f49e006ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 --2- 172.21.15.1:0/587151915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49e01003a0 0x7f49e006f4a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 >> 172.21.15.1:0/587151915 conn(0x7f49e00f74e0 msgr2=0x7f49e00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 shutdown_connections 2024-02-28T16:30:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.020+0000 7f49e8000700 1 -- 172.21.15.1:0/587151915 wait complete. 2024-02-28T16:30:15.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.189+0000 7ff640000700 1 -- 172.21.15.1:0/2965464049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 msgr2=0x7ff638104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:15.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.189+0000 7ff640000700 1 --2- 172.21.15.1:0/2965464049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 0x7ff638104ba0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff624004940 tx=0x7ff624009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:15.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.189+0000 7ff640000700 1 -- 172.21.15.1:0/2965464049 shutdown_connections 2024-02-28T16:30:15.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.189+0000 7ff640000700 1 --2- 172.21.15.1:0/2965464049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 0x7ff638104ba0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.189+0000 7ff640000700 1 --2- 172.21.15.1:0/2965464049 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff638103540 0x7ff638103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.189+0000 7ff640000700 1 -- 172.21.15.1:0/2965464049 >> 172.21.15.1:0/2965464049 conn(0x7ff6380727d0 msgr2=0x7ff638074c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:15.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.190+0000 7ff640000700 1 -- 172.21.15.1:0/2965464049 shutdown_connections 2024-02-28T16:30:15.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.190+0000 7ff640000700 1 -- 172.21.15.1:0/2965464049 wait complete. 2024-02-28T16:30:15.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.190+0000 7ff640000700 1 Processor -- start 2024-02-28T16:30:15.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.191+0000 7ff640000700 1 -- start start 2024-02-28T16:30:15.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.191+0000 7ff640000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff638103540 0x7ff638113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:15.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.191+0000 7ff640000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 0x7ff6381144c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:15.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.191+0000 7ff640000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff638114ac0 con 0x7ff638104750 2024-02-28T16:30:15.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.191+0000 7ff640000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff638115360 con 0x7ff638103540 2024-02-28T16:30:15.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.191+0000 7ff63cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 0x7ff6381144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:15.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.192+0000 7ff63cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 0x7ff6381144c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44328/0 (socket says 172.21.15.1:44328) 2024-02-28T16:30:15.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.192+0000 7ff63cc00700 1 -- 172.21.15.1:0/647391071 learned_addr learned my addr 172.21.15.1:0/647391071 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:15.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.192+0000 7ff63d600700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff638103540 0x7ff638113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:15.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.192+0000 7ff63cc00700 1 -- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff638103540 msgr2=0x7ff638113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:15.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.192+0000 7ff63cc00700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff638103540 0x7ff638113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.192+0000 7ff63cc00700 1 -- 172.21.15.1:0/647391071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff624005040 con 0x7ff638104750 2024-02-28T16:30:15.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.192+0000 7ff63cc00700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 0x7ff6381144c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff6240093b0 tx=0x7ff624009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:15.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.193+0000 7ff635600700 1 -- 172.21.15.1:0/647391071 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff62400e040 con 0x7ff638104750 2024-02-28T16:30:15.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.193+0000 7ff635600700 1 -- 172.21.15.1:0/647391071 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff62400bcc0 con 0x7ff638104750 2024-02-28T16:30:15.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.193+0000 7ff635600700 1 -- 172.21.15.1:0/647391071 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff62401c630 con 0x7ff638104750 2024-02-28T16:30:15.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.193+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6381155e0 con 0x7ff638104750 2024-02-28T16:30:15.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.193+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff638115b20 con 0x7ff638104750 2024-02-28T16:30:15.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.195+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63804bed0 con 0x7ff638104750 2024-02-28T16:30:15.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.196+0000 7ff635600700 1 -- 172.21.15.1:0/647391071 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7ff6240139f0 con 0x7ff638104750 2024-02-28T16:30:15.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.198+0000 7ff635600700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ff62006b2e0 0x7ff62006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:15.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.198+0000 7ff635600700 1 -- 172.21.15.1:0/647391071 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7ff62408b1f0 con 0x7ff638104750 2024-02-28T16:30:15.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.199+0000 7ff63d600700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ff62006b2e0 0x7ff62006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:15.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.200+0000 7ff63d600700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ff62006b2e0 0x7ff62006d790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff62c007fd0 tx=0x7ff62c009040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:15.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.203+0000 7ff635600700 1 -- 172.21.15.1:0/647391071 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7ff624058d60 con 0x7ff638104750 2024-02-28T16:30:15.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.494+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff638115f40 con 0x7ff62006b2e0 2024-02-28T16:30:15.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.540+0000 7ff635600700 1 -- 172.21.15.1:0/647391071 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7ff638115f40 con 0x7ff62006b2e0 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (8m) 22s ago 10m 13.3M - 0.25.0 c8568f914cd2 3d10852c3022 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (10m) 22s ago 10m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (9m) 22s ago 9m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (8m) 22s ago 10m 74.1M - 9.4.7 954c08fa6188 2a03709b4c9c 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (3m) 22s ago 3m 67.7M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (3m) 22s ago 3m 11.8M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (3m) 22s ago 3m 11.9M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:30:15.529 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (3m) 22s ago 3m 68.6M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:9283 running (12m) 22s ago 12m 336M - 17.2.7-535-gc0f02e60 1c37ed371348 8eb0fd290742 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283 running (53s) 22s ago 9m 492M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 30a2de341a4e 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (12m) 22s ago 12m 47.6M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (9m) 22s ago 9m 36.1M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 starting - - - - 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (9m) 22s ago 9m 8036k - 1.5.0 0da6a335fe13 31630ee26c1c 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (8m) 22s ago 8m 62.2M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (7m) 22s ago 7m 66.5M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (7m) 22s ago 7m 59.1M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (6m) 22s ago 6m 69.1M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (5m) 22s ago 5m 65.0M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (5m) 22s ago 5m 60.9M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:30:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (8m) 22s ago 9m 52.1M - 2.43.0 a07b618ecd1d 21713d7e702b 2024-02-28T16:30:15.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ff62006b2e0 msgr2=0x7ff62006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:15.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ff62006b2e0 0x7ff62006d790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff62c007fd0 tx=0x7ff62c009040 comp rx=0 tx=0).stop 2024-02-28T16:30:15.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 msgr2=0x7ff6381144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:15.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 0x7ff6381144c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff6240093b0 tx=0x7ff624009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:15.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 shutdown_connections 2024-02-28T16:30:15.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ff62006b2e0 0x7ff62006d790 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff638103540 0x7ff638113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 --2- 172.21.15.1:0/647391071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff638104750 0x7ff6381144c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.543+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 >> 172.21.15.1:0/647391071 conn(0x7ff6380727d0 msgr2=0x7ff638107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:15.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.544+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 shutdown_connections 2024-02-28T16:30:15.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.544+0000 7ff640000700 1 -- 172.21.15.1:0/647391071 wait complete. 2024-02-28T16:30:15.617 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:15 smithi001 ceph-mon[28991]: pgmap v21: 65 pgs: 65 active+clean; 132 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 100 op/s 2024-02-28T16:30:15.617 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:15 smithi001 ceph-mon[28991]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.728+0000 7f462c600700 1 -- 172.21.15.1:0/380253672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 msgr2=0x7f46240fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.728+0000 7f462c600700 1 --2- 172.21.15.1:0/380253672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 0x7f46240fd320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4618004940 tx=0x7f4618009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.729+0000 7f462c600700 1 -- 172.21.15.1:0/380253672 shutdown_connections 2024-02-28T16:30:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.729+0000 7f462c600700 1 --2- 172.21.15.1:0/380253672 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46240fe0e0 0x7f46240fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.729+0000 7f462c600700 1 --2- 172.21.15.1:0/380253672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 0x7f46240fd320 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.729+0000 7f462c600700 1 -- 172.21.15.1:0/380253672 >> 172.21.15.1:0/380253672 conn(0x7f46240f8620 msgr2=0x7f46240faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.729+0000 7f462c600700 1 -- 172.21.15.1:0/380253672 shutdown_connections 2024-02-28T16:30:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.729+0000 7f462c600700 1 -- 172.21.15.1:0/380253672 wait complete. 2024-02-28T16:30:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.730+0000 7f462c600700 1 Processor -- start 2024-02-28T16:30:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.730+0000 7f462c600700 1 -- start start 2024-02-28T16:30:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f462c600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 0x7f462410fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f462c600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46240fe0e0 0x7f46241100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f462c600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46241106b0 con 0x7f46240fcef0 2024-02-28T16:30:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f462c600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4624110f50 con 0x7f46240fe0e0 2024-02-28T16:30:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f4629c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 0x7f462410fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f4629c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 0x7f462410fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44346/0 (socket says 172.21.15.1:44346) 2024-02-28T16:30:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f4629c00700 1 -- 172.21.15.1:0/1651264555 learned_addr learned my addr 172.21.15.1:0/1651264555 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f4629c00700 1 -- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46240fe0e0 msgr2=0x7f46241100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.731+0000 7f4629200700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46240fe0e0 0x7f46241100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.732+0000 7f4629c00700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46240fe0e0 0x7f46241100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.732+0000 7f4629c00700 1 -- 172.21.15.1:0/1651264555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4618005040 con 0x7f46240fcef0 2024-02-28T16:30:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.732+0000 7f4629c00700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 0x7f462410fb70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4618000c00 tx=0x7f4618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:15.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.732+0000 7f4621600700 1 -- 172.21.15.1:0/1651264555 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f461800e040 con 0x7f46240fcef0 2024-02-28T16:30:15.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.732+0000 7f4621600700 1 -- 172.21.15.1:0/1651264555 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f461800bcc0 con 0x7f46240fcef0 2024-02-28T16:30:15.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.732+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46241111d0 con 0x7f46240fcef0 2024-02-28T16:30:15.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.732+0000 7f4621600700 1 -- 172.21.15.1:0/1651264555 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f461801c630 con 0x7f46240fcef0 2024-02-28T16:30:15.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.733+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4624111710 con 0x7f46240fcef0 2024-02-28T16:30:15.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.734+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462404bed0 con 0x7f46240fcef0 2024-02-28T16:30:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.735+0000 7f4621600700 1 -- 172.21.15.1:0/1651264555 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7f4618013980 con 0x7f46240fcef0 2024-02-28T16:30:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.736+0000 7f4621600700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f460c06b2e0 0x7f460c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:15.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.736+0000 7f4621600700 1 -- 172.21.15.1:0/1651264555 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f461808b200 con 0x7f46240fcef0 2024-02-28T16:30:15.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.737+0000 7f4629200700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f460c06b2e0 0x7f460c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:15.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.738+0000 7f4629200700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f460c06b2e0 0x7f460c06d790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f46100041d0 tx=0x7f4610009040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:15.742+0000 7f4621600700 1 -- 172.21.15.1:0/1651264555 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f4618058cf0 con 0x7f46240fcef0 2024-02-28T16:30:15.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:15 smithi066 ceph-mon[36236]: pgmap v21: 65 pgs: 65 active+clean; 132 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 100 op/s 2024-02-28T16:30:15.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:15 smithi066 ceph-mon[36236]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:16.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.157+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4624111a40 con 0x7f46240fcef0 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.157+0000 7f4621600700 1 -- 172.21.15.1:0/1651264555 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f4618058880 con 0x7f46240fcef0 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:30:16.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 13, 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:30:16.147 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:30:16.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f460c06b2e0 msgr2=0x7f460c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:16.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f460c06b2e0 0x7f460c06d790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f46100041d0 tx=0x7f4610009040 comp rx=0 tx=0).stop 2024-02-28T16:30:16.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 msgr2=0x7f462410fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:16.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 0x7f462410fb70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4618000c00 tx=0x7f4618009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:16.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 shutdown_connections 2024-02-28T16:30:16.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f460c06b2e0 0x7f460c06d790 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46240fcef0 0x7f462410fb70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 --2- 172.21.15.1:0/1651264555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46240fe0e0 0x7f46241100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 >> 172.21.15.1:0/1651264555 conn(0x7f46240f8620 msgr2=0x7f46241013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:16.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 shutdown_connections 2024-02-28T16:30:16.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.161+0000 7f462c600700 1 -- 172.21.15.1:0/1651264555 wait complete. 2024-02-28T16:30:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.339+0000 7f6abe800700 1 -- 172.21.15.1:0/609108430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 msgr2=0x7f6ab8070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.339+0000 7f6abe800700 1 --2- 172.21.15.1:0/609108430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 0x7f6ab8070380 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8004940 tx=0x7f6aa8009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.340+0000 7f6abe800700 1 -- 172.21.15.1:0/609108430 shutdown_connections 2024-02-28T16:30:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.340+0000 7f6abe800700 1 --2- 172.21.15.1:0/609108430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ab80708c0 0x7f6ab8070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.340+0000 7f6abe800700 1 --2- 172.21.15.1:0/609108430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 0x7f6ab8070380 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.340+0000 7f6abe800700 1 -- 172.21.15.1:0/609108430 >> 172.21.15.1:0/609108430 conn(0x7f6ab80f8620 msgr2=0x7f6ab80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.340+0000 7f6abe800700 1 -- 172.21.15.1:0/609108430 shutdown_connections 2024-02-28T16:30:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.340+0000 7f6abe800700 1 -- 172.21.15.1:0/609108430 wait complete. 2024-02-28T16:30:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.341+0000 7f6abe800700 1 Processor -- start 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.341+0000 7f6abe800700 1 -- start start 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.342+0000 7f6abe800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ab80708c0 0x7f6ab8113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.342+0000 7f6abe800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 0x7f6ab8114480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.342+0000 7f6abe800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8114b10 con 0x7f6ab8071f10 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.342+0000 7f6abe800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8115350 con 0x7f6ab80708c0 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.342+0000 7f6ab6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 0x7f6ab8114480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.342+0000 7f6ab6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 0x7f6ab8114480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44364/0 (socket says 172.21.15.1:44364) 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.342+0000 7f6ab6a00700 1 -- 172.21.15.1:0/4130440710 learned_addr learned my addr 172.21.15.1:0/4130440710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.343+0000 7f6ab7400700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ab80708c0 0x7f6ab8113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.343+0000 7f6ab6a00700 1 -- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ab80708c0 msgr2=0x7f6ab8113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.343+0000 7f6ab6a00700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ab80708c0 0x7f6ab8113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.343+0000 7f6ab6a00700 1 -- 172.21.15.1:0/4130440710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa8005040 con 0x7f6ab8071f10 2024-02-28T16:30:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.343+0000 7f6ab6a00700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 0x7f6ab8114480 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400b790 tx=0x7f6aa4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.344+0000 7f6aaf400700 1 -- 172.21.15.1:0/4130440710 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6aa400f770 con 0x7f6ab8071f10 2024-02-28T16:30:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.344+0000 7f6aaf400700 1 -- 172.21.15.1:0/4130440710 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6aa400fda0 con 0x7f6ab8071f10 2024-02-28T16:30:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.344+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab8115630 con 0x7f6ab8071f10 2024-02-28T16:30:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.344+0000 7f6aaf400700 1 -- 172.21.15.1:0/4130440710 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6aa400d310 con 0x7f6ab8071f10 2024-02-28T16:30:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.344+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab8115b70 con 0x7f6ab8071f10 2024-02-28T16:30:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.346+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a980052e0 con 0x7f6ab8071f10 2024-02-28T16:30:16.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.347+0000 7f6aaf400700 1 -- 172.21.15.1:0/4130440710 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7f6aa400f8d0 con 0x7f6ab8071f10 2024-02-28T16:30:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.348+0000 7f6aaf400700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f6a9c06b2e0 0x7f6a9c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.348+0000 7f6aaf400700 1 -- 172.21.15.1:0/4130440710 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f6aa4089c10 con 0x7f6ab8071f10 2024-02-28T16:30:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.349+0000 7f6ab7400700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f6a9c06b2e0 0x7f6a9c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.351+0000 7f6ab7400700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f6a9c06b2e0 0x7f6a9c06d790 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6aa800b020 tx=0x7f6aa8009e30 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:16.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.353+0000 7f6aaf400700 1 -- 172.21.15.1:0/4130440710 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f6aa4057890 con 0x7f6ab8071f10 2024-02-28T16:30:16.443 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:16 smithi001 ceph-mon[28991]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:16.443 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:16 smithi001 ceph-mon[28991]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:16.443 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:16 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1651264555' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:30:16.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.690+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a98000bf0 con 0x7f6a9c06b2e0 2024-02-28T16:30:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.694+0000 7f6aaf400700 1 -- 172.21.15.1:0/4130440710 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f6a98000bf0 con 0x7f6a9c06b2e0 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [], 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "1/2 daemons upgraded", 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "", 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:30:16.683 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:30:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.697+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f6a9c06b2e0 msgr2=0x7f6a9c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.697+0000 7f6abe800700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f6a9c06b2e0 0x7f6a9c06d790 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6aa800b020 tx=0x7f6aa8009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.697+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 msgr2=0x7f6ab8114480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.697+0000 7f6abe800700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 0x7f6ab8114480 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400b790 tx=0x7f6aa4009250 comp rx=0 tx=0).stop 2024-02-28T16:30:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.698+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 shutdown_connections 2024-02-28T16:30:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.698+0000 7f6abe800700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f6a9c06b2e0 0x7f6a9c06d790 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.698+0000 7f6abe800700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ab80708c0 0x7f6ab8113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.698+0000 7f6abe800700 1 --2- 172.21.15.1:0/4130440710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ab8071f10 0x7f6ab8114480 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.698+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 >> 172.21.15.1:0/4130440710 conn(0x7f6ab80f8620 msgr2=0x7f6ab80fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.698+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 shutdown_connections 2024-02-28T16:30:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:16.698+0000 7f6abe800700 1 -- 172.21.15.1:0/4130440710 wait complete. 2024-02-28T16:30:16.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:16 smithi066 ceph-mon[36236]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:16.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:16 smithi066 ceph-mon[36236]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:16.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:16 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1651264555' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:30:17.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:17 smithi001 ceph-mon[28991]: pgmap v22: 65 pgs: 65 active+clean; 135 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 83 op/s 2024-02-28T16:30:17.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:17 smithi001 ceph-mon[28991]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:17.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:17 smithi066 ceph-mon[36236]: pgmap v22: 65 pgs: 65 active+clean; 135 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 83 op/s 2024-02-28T16:30:17.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:17 smithi066 ceph-mon[36236]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:19.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:18 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:19.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:18 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:19.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:18 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:19.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:18 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:20.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:19 smithi001 ceph-mon[28991]: Reconfiguring grafana.smithi001 (dependencies changed)... 2024-02-28T16:30:20.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:19 smithi001 ceph-mon[28991]: Reconfiguring daemon grafana.smithi001 on smithi001 2024-02-28T16:30:20.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:19 smithi001 ceph-mon[28991]: pgmap v23: 65 pgs: 65 active+clean; 135 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 58 op/s 2024-02-28T16:30:20.224 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:19 smithi066 ceph-mon[36236]: Reconfiguring grafana.smithi001 (dependencies changed)... 2024-02-28T16:30:20.224 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:19 smithi066 ceph-mon[36236]: Reconfiguring daemon grafana.smithi001 on smithi001 2024-02-28T16:30:20.224 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:19 smithi066 ceph-mon[36236]: pgmap v23: 65 pgs: 65 active+clean; 135 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.3 MiB/s wr, 58 op/s 2024-02-28T16:30:22.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:21 smithi066 ceph-mon[36236]: pgmap v24: 65 pgs: 65 active+clean; 146 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.3 MiB/s wr, 123 op/s 2024-02-28T16:30:22.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:21 smithi001 ceph-mon[28991]: pgmap v24: 65 pgs: 65 active+clean; 146 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.3 MiB/s wr, 123 op/s 2024-02-28T16:30:24.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:23 smithi066 ceph-mon[36236]: pgmap v25: 65 pgs: 65 active+clean; 146 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.6 MiB/s wr, 93 op/s 2024-02-28T16:30:24.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:23 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:30:24.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:23 smithi001 ceph-mon[28991]: pgmap v25: 65 pgs: 65 active+clean; 146 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.6 MiB/s wr, 93 op/s 2024-02-28T16:30:24.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:23 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:30:26.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:25 smithi066 ceph-mon[36236]: pgmap v26: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.6 MiB/s wr, 158 op/s 2024-02-28T16:30:26.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:25 smithi001 ceph-mon[28991]: pgmap v26: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.6 MiB/s wr, 158 op/s 2024-02-28T16:30:28.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:27 smithi066 ceph-mon[36236]: pgmap v27: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.2 MiB/s wr, 136 op/s 2024-02-28T16:30:28.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:27 smithi001 ceph-mon[28991]: pgmap v27: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.2 MiB/s wr, 136 op/s 2024-02-28T16:30:29.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:28 smithi066 ceph-mon[36236]: pgmap v28: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 130 op/s 2024-02-28T16:30:29.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:28 smithi001 ceph-mon[28991]: pgmap v28: 65 pgs: 65 active+clean; 158 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 130 op/s 2024-02-28T16:30:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:31 smithi066 ceph-mon[36236]: pgmap v29: 65 pgs: 65 active+clean; 168 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.8 MiB/s wr, 200 op/s 2024-02-28T16:30:31.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:31 smithi001 ceph-mon[28991]: pgmap v29: 65 pgs: 65 active+clean; 168 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.8 MiB/s wr, 200 op/s 2024-02-28T16:30:33.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:33 smithi066 ceph-mon[36236]: pgmap v30: 65 pgs: 65 active+clean; 168 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.8 MiB/s wr, 135 op/s 2024-02-28T16:30:33.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:33 smithi001 ceph-mon[28991]: pgmap v30: 65 pgs: 65 active+clean; 168 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.8 MiB/s wr, 135 op/s 2024-02-28T16:30:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:35 smithi066 ceph-mon[36236]: pgmap v31: 65 pgs: 65 active+clean; 180 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.9 MiB/s wr, 215 op/s 2024-02-28T16:30:35.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:35 smithi001 ceph-mon[28991]: pgmap v31: 65 pgs: 65 active+clean; 180 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.9 MiB/s wr, 215 op/s 2024-02-28T16:30:37.581 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:37 smithi001 ceph-mon[28991]: pgmap v32: 65 pgs: 65 active+clean; 180 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 150 op/s 2024-02-28T16:30:37.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:37 smithi066 ceph-mon[36236]: pgmap v32: 65 pgs: 65 active+clean; 180 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 150 op/s 2024-02-28T16:30:38.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:38 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:30:38.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:38 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:30:39.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:39 smithi066 ceph-mon[36236]: pgmap v33: 65 pgs: 65 active+clean; 180 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 150 op/s 2024-02-28T16:30:39.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:39 smithi001 ceph-mon[28991]: pgmap v33: 65 pgs: 65 active+clean; 180 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.9 MiB/s wr, 150 op/s 2024-02-28T16:30:41.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:41 smithi001 ceph-mon[28991]: pgmap v34: 65 pgs: 65 active+clean; 192 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.9 MiB/s wr, 217 op/s 2024-02-28T16:30:41.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:41 smithi066 ceph-mon[36236]: pgmap v34: 65 pgs: 65 active+clean; 192 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.9 MiB/s wr, 217 op/s 2024-02-28T16:30:43.454 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:43 smithi001 ceph-mon[28991]: pgmap v35: 65 pgs: 65 active+clean; 192 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.1 MiB/s wr, 147 op/s 2024-02-28T16:30:43.454 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:43 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:43.454 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:43 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:43.454 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:43 smithi001 ceph-mon[28991]: Reconfiguring prometheus.smithi001 (dependencies changed)... 2024-02-28T16:30:43.454 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:43 smithi001 ceph-mon[28991]: Reconfiguring daemon prometheus.smithi001 on smithi001 2024-02-28T16:30:43.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:43 smithi066 ceph-mon[36236]: pgmap v35: 65 pgs: 65 active+clean; 192 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.1 MiB/s wr, 147 op/s 2024-02-28T16:30:43.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:43 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:43.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:43 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:43.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:43 smithi066 ceph-mon[36236]: Reconfiguring prometheus.smithi001 (dependencies changed)... 2024-02-28T16:30:43.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:43 smithi066 ceph-mon[36236]: Reconfiguring daemon prometheus.smithi001 on smithi001 2024-02-28T16:30:45.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:45 smithi066 ceph-mon[36236]: pgmap v36: 65 pgs: 65 active+clean; 204 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 3.1 MiB/s wr, 238 op/s 2024-02-28T16:30:45.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:45 smithi001 ceph-mon[28991]: pgmap v36: 65 pgs: 65 active+clean; 204 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 3.1 MiB/s wr, 238 op/s 2024-02-28T16:30:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.895+0000 7f1e01000700 1 -- 172.21.15.1:0/3734285081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc0ff150 msgr2=0x7f1dfc0ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.895+0000 7f1e01000700 1 --2- 172.21.15.1:0/3734285081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc0ff150 0x7f1dfc0ff560 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1dec004940 tx=0x7f1dec009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.896+0000 7f1e01000700 1 -- 172.21.15.1:0/3734285081 shutdown_connections 2024-02-28T16:30:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.896+0000 7f1e01000700 1 --2- 172.21.15.1:0/3734285081 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1dfc100360 0x7f1dfc1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.896+0000 7f1e01000700 1 --2- 172.21.15.1:0/3734285081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc0ff150 0x7f1dfc0ff560 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.896+0000 7f1e01000700 1 -- 172.21.15.1:0/3734285081 >> 172.21.15.1:0/3734285081 conn(0x7f1dfc0fa920 msgr2=0x7f1dfc0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.896+0000 7f1e01000700 1 -- 172.21.15.1:0/3734285081 shutdown_connections 2024-02-28T16:30:46.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.896+0000 7f1e01000700 1 -- 172.21.15.1:0/3734285081 wait complete. 2024-02-28T16:30:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.897+0000 7f1e01000700 1 Processor -- start 2024-02-28T16:30:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.898+0000 7f1e01000700 1 -- start start 2024-02-28T16:30:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.898+0000 7f1e01000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1dfc0ff150 0x7f1dfc10fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.898+0000 7f1e01000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc100360 0x7f1dfc1100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.898+0000 7f1e01000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc1106d0 con 0x7f1dfc100360 2024-02-28T16:30:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.898+0000 7f1e01000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc110f70 con 0x7f1dfc0ff150 2024-02-28T16:30:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.899+0000 7f1df9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc100360 0x7f1dfc1100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.899+0000 7f1df9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc100360 0x7f1dfc1100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44710/0 (socket says 172.21.15.1:44710) 2024-02-28T16:30:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.899+0000 7f1df9400700 1 -- 172.21.15.1:0/994351341 learned_addr learned my addr 172.21.15.1:0/994351341 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.899+0000 7f1df9e00700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1dfc0ff150 0x7f1dfc10fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.899+0000 7f1df9400700 1 -- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1dfc0ff150 msgr2=0x7f1dfc10fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.899+0000 7f1df9400700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1dfc0ff150 0x7f1dfc10fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.899+0000 7f1df9400700 1 -- 172.21.15.1:0/994351341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec005040 con 0x7f1dfc100360 2024-02-28T16:30:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.899+0000 7f1df9e00700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1dfc0ff150 0x7f1dfc10fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:30:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.900+0000 7f1df9400700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc100360 0x7f1dfc1100d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1de8002a50 tx=0x7f1de80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.900+0000 7f1df2000700 1 -- 172.21.15.1:0/994351341 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1de8007cc0 con 0x7f1dfc100360 2024-02-28T16:30:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.900+0000 7f1df2000700 1 -- 172.21.15.1:0/994351341 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1de800f040 con 0x7f1dfc100360 2024-02-28T16:30:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.900+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dfc111250 con 0x7f1dfc100360 2024-02-28T16:30:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.900+0000 7f1df2000700 1 -- 172.21.15.1:0/994351341 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1de8009640 con 0x7f1dfc100360 2024-02-28T16:30:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.900+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dfc111790 con 0x7f1dfc100360 2024-02-28T16:30:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.902+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dfc04bed0 con 0x7f1dfc100360 2024-02-28T16:30:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.903+0000 7f1df2000700 1 -- 172.21.15.1:0/994351341 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7f1de80097a0 con 0x7f1dfc100360 2024-02-28T16:30:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.904+0000 7f1df2000700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f1de006b2e0 0x7f1de006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.904+0000 7f1df2000700 1 -- 172.21.15.1:0/994351341 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f1de8089ae0 con 0x7f1dfc100360 2024-02-28T16:30:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.905+0000 7f1df9e00700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f1de006b2e0 0x7f1de006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.906+0000 7f1df9e00700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f1de006b2e0 0x7f1de006d790 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00b7a0 tx=0x7f1dec009e30 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:46.909+0000 7f1df2000700 1 -- 172.21.15.1:0/994351341 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f1de8057670 con 0x7f1dfc100360 2024-02-28T16:30:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.248+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dfc111c30 con 0x7f1de006b2e0 2024-02-28T16:30:47.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.251+0000 7f1df2000700 1 -- 172.21.15.1:0/994351341 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f1dfc111c30 con 0x7f1de006b2e0 2024-02-28T16:30:47.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.254+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f1de006b2e0 msgr2=0x7f1de006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:47.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.254+0000 7f1e01000700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f1de006b2e0 0x7f1de006d790 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00b7a0 tx=0x7f1dec009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:47.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.254+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc100360 msgr2=0x7f1dfc1100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:47.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.254+0000 7f1e01000700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc100360 0x7f1dfc1100d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1de8002a50 tx=0x7f1de80046b0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.254+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 shutdown_connections 2024-02-28T16:30:47.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.254+0000 7f1e01000700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f1de006b2e0 0x7f1de006d790 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.254+0000 7f1e01000700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1dfc0ff150 0x7f1dfc10fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.254+0000 7f1e01000700 1 --2- 172.21.15.1:0/994351341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc100360 0x7f1dfc1100d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.255+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 >> 172.21.15.1:0/994351341 conn(0x7f1dfc0fa920 msgr2=0x7f1dfc103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:47.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.255+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 shutdown_connections 2024-02-28T16:30:47.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.255+0000 7f1e01000700 1 -- 172.21.15.1:0/994351341 wait complete. 2024-02-28T16:30:47.260 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:30:47.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.435+0000 7f243c200700 1 -- 172.21.15.1:0/896801886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 msgr2=0x7f24340fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:47.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.435+0000 7f243c200700 1 --2- 172.21.15.1:0/896801886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 0x7f24340fe550 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2424004940 tx=0x7f2424009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:47.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.435+0000 7f243c200700 1 -- 172.21.15.1:0/896801886 shutdown_connections 2024-02-28T16:30:47.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.435+0000 7f243c200700 1 --2- 172.21.15.1:0/896801886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 0x7f24340fe550 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.435+0000 7f243c200700 1 --2- 172.21.15.1:0/896801886 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f24340fcef0 0x7f24340fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.435+0000 7f243c200700 1 -- 172.21.15.1:0/896801886 >> 172.21.15.1:0/896801886 conn(0x7f24340f8620 msgr2=0x7f24340faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:47.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.436+0000 7f243c200700 1 -- 172.21.15.1:0/896801886 shutdown_connections 2024-02-28T16:30:47.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.436+0000 7f243c200700 1 -- 172.21.15.1:0/896801886 wait complete. 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.437+0000 7f243c200700 1 Processor -- start 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.437+0000 7f243c200700 1 -- start start 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.437+0000 7f243c200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f24340fcef0 0x7f24340718c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.437+0000 7f243c200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 0x7f2434071e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.437+0000 7f243c200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2434070500 con 0x7f24340fe0e0 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.437+0000 7f243c200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2434070640 con 0x7f24340fcef0 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.438+0000 7f2438e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 0x7f2434071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.438+0000 7f2438e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 0x7f2434071e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44730/0 (socket says 172.21.15.1:44730) 2024-02-28T16:30:47.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.438+0000 7f2438e00700 1 -- 172.21.15.1:0/3135519915 learned_addr learned my addr 172.21.15.1:0/3135519915 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:47.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.438+0000 7f2439800700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f24340fcef0 0x7f24340718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:47.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.438+0000 7f2438e00700 1 -- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f24340fcef0 msgr2=0x7f24340718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:47.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.438+0000 7f2438e00700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f24340fcef0 0x7f24340718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.438+0000 7f2438e00700 1 -- 172.21.15.1:0/3135519915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2424005040 con 0x7f24340fe0e0 2024-02-28T16:30:47.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.439+0000 7f2438e00700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 0x7f2434071e00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f24240093b0 tx=0x7f2424009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:47.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.439+0000 7f2431600700 1 -- 172.21.15.1:0/3135519915 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f242400e040 con 0x7f24340fe0e0 2024-02-28T16:30:47.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.439+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24340708c0 con 0x7f24340fe0e0 2024-02-28T16:30:47.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.439+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2434070e00 con 0x7f24340fe0e0 2024-02-28T16:30:47.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.440+0000 7f2431600700 1 -- 172.21.15.1:0/3135519915 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f242400bcc0 con 0x7f24340fe0e0 2024-02-28T16:30:47.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.440+0000 7f2431600700 1 -- 172.21.15.1:0/3135519915 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f242401c3f0 con 0x7f24340fe0e0 2024-02-28T16:30:47.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.442+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24180052e0 con 0x7f24340fe0e0 2024-02-28T16:30:47.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.443+0000 7f2431600700 1 -- 172.21.15.1:0/3135519915 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7f242401c650 con 0x7f24340fe0e0 2024-02-28T16:30:47.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.444+0000 7f2431600700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f241c06b3b0 0x7f241c06d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:47.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.445+0000 7f2431600700 1 -- 172.21.15.1:0/3135519915 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f242408b1a0 con 0x7f24340fe0e0 2024-02-28T16:30:47.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.445+0000 7f2439800700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f241c06b3b0 0x7f241c06d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.447+0000 7f2439800700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f241c06b3b0 0x7f241c06d860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2428009fd0 tx=0x7f2428009f00 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:47.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.450+0000 7f2431600700 1 -- 172.21.15.1:0/3135519915 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f2424058d10 con 0x7f24340fe0e0 2024-02-28T16:30:47.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:47 smithi001 ceph-mon[28991]: pgmap v37: 65 pgs: 65 active+clean; 204 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 158 op/s 2024-02-28T16:30:47.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:47 smithi066 ceph-mon[36236]: pgmap v37: 65 pgs: 65 active+clean; 204 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 158 op/s 2024-02-28T16:30:47.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.789+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2418000bf0 con 0x7f241c06b3b0 2024-02-28T16:30:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.791+0000 7f2431600700 1 -- 172.21.15.1:0/3135519915 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f2418000bf0 con 0x7f241c06b3b0 2024-02-28T16:30:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f241c06b3b0 msgr2=0x7f241c06d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f241c06b3b0 0x7f241c06d860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2428009fd0 tx=0x7f2428009f00 comp rx=0 tx=0).stop 2024-02-28T16:30:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 msgr2=0x7f2434071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 0x7f2434071e00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f24240093b0 tx=0x7f2424009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 shutdown_connections 2024-02-28T16:30:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f241c06b3b0 0x7f241c06d860 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f24340fcef0 0x7f24340718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 --2- 172.21.15.1:0/3135519915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24340fe0e0 0x7f2434071e00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 >> 172.21.15.1:0/3135519915 conn(0x7f24340f8620 msgr2=0x7f24341013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.794+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 shutdown_connections 2024-02-28T16:30:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.795+0000 7f243c200700 1 -- 172.21.15.1:0/3135519915 wait complete. 2024-02-28T16:30:47.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.982+0000 7f0ffd600700 1 -- 172.21.15.1:0/2594255351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 msgr2=0x7f0ff80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.982+0000 7f0ffd600700 1 --2- 172.21.15.1:0/2594255351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 0x7f0ff80fd320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8004940 tx=0x7f0fe8009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.983+0000 7f0ffd600700 1 -- 172.21.15.1:0/2594255351 shutdown_connections 2024-02-28T16:30:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.983+0000 7f0ffd600700 1 --2- 172.21.15.1:0/2594255351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ff80fe0e0 0x7f0ff80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.983+0000 7f0ffd600700 1 --2- 172.21.15.1:0/2594255351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 0x7f0ff80fd320 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.983+0000 7f0ffd600700 1 -- 172.21.15.1:0/2594255351 >> 172.21.15.1:0/2594255351 conn(0x7f0ff80f8620 msgr2=0x7f0ff80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.983+0000 7f0ffd600700 1 -- 172.21.15.1:0/2594255351 shutdown_connections 2024-02-28T16:30:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.983+0000 7f0ffd600700 1 -- 172.21.15.1:0/2594255351 wait complete. 2024-02-28T16:30:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.984+0000 7f0ffd600700 1 Processor -- start 2024-02-28T16:30:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.984+0000 7f0ffd600700 1 -- start start 2024-02-28T16:30:47.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.984+0000 7f0ffd600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 0x7f0ff810fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.984+0000 7f0ffd600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ff80fe0e0 0x7f0ff8110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.984+0000 7f0ffd600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8110640 con 0x7f0ff80fcef0 2024-02-28T16:30:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.984+0000 7f0ffd600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8110ee0 con 0x7f0ff80fe0e0 2024-02-28T16:30:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.985+0000 7f0ff6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 0x7f0ff810fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.985+0000 7f0ff6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 0x7f0ff810fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44740/0 (socket says 172.21.15.1:44740) 2024-02-28T16:30:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.985+0000 7f0ff6a00700 1 -- 172.21.15.1:0/2751528182 learned_addr learned my addr 172.21.15.1:0/2751528182 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:47.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.985+0000 7f0ff6000700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ff80fe0e0 0x7f0ff8110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.986+0000 7f0ff6a00700 1 -- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ff80fe0e0 msgr2=0x7f0ff8110040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.986+0000 7f0ff6a00700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ff80fe0e0 0x7f0ff8110040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.986+0000 7f0ff6a00700 1 -- 172.21.15.1:0/2751528182 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8005040 con 0x7f0ff80fcef0 2024-02-28T16:30:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.986+0000 7f0ff6a00700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 0x7f0ff810fb00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009fd0 tx=0x7f0fe8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.986+0000 7f0feea00700 1 -- 172.21.15.1:0/2751528182 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe800e070 con 0x7f0ff80fcef0 2024-02-28T16:30:47.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.986+0000 7f0feea00700 1 -- 172.21.15.1:0/2751528182 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fe8009430 con 0x7f0ff80fcef0 2024-02-28T16:30:47.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.986+0000 7f0feea00700 1 -- 172.21.15.1:0/2751528182 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe801ce30 con 0x7f0ff80fcef0 2024-02-28T16:30:47.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.986+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff8111160 con 0x7f0ff80fcef0 2024-02-28T16:30:47.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.987+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff8111640 con 0x7f0ff80fcef0 2024-02-28T16:30:47.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.988+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff8102210 con 0x7f0ff80fcef0 2024-02-28T16:30:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.990+0000 7f0feea00700 1 -- 172.21.15.1:0/2751528182 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7f0fe801c690 con 0x7f0ff80fcef0 2024-02-28T16:30:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.991+0000 7f0feea00700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0fdc06b2e0 0x7f0fdc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:47.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.991+0000 7f0feea00700 1 -- 172.21.15.1:0/2751528182 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f0fe808a910 con 0x7f0ff80fcef0 2024-02-28T16:30:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.992+0000 7f0ff6000700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0fdc06b2e0 0x7f0fdc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:47.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.993+0000 7f0ff6000700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0fdc06b2e0 0x7f0fdc06d790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009b80 tx=0x7f0fe400b040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:47.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:47.997+0000 7f0feea00700 1 -- 172.21.15.1:0/2751528182 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f0fe8058480 con 0x7f0ff80fcef0 2024-02-28T16:30:48.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.328+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ff8111990 con 0x7f0fdc06b2e0 2024-02-28T16:30:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.363+0000 7f0feea00700 1 -- 172.21.15.1:0/2751528182 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f0ff8111990 con 0x7f0fdc06b2e0 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 starting - - - - 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (11m) 55s ago 11m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (10m) 55s ago 10m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 starting - - - - 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (4m) 55s ago 4m 67.7M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (3m) 55s ago 3m 11.8M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (3m) 55s ago 3m 11.9M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (4m) 55s ago 4m 68.6M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:30:48.353 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:9283 running (12m) 55s ago 12m 336M - 17.2.7-535-gc0f02e60 1c37ed371348 8eb0fd290742 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283 running (85s) 55s ago 9m 492M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 30a2de341a4e 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (12m) 55s ago 13m 47.6M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (9m) 55s ago 9m 36.1M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 starting - - - - 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (9m) 55s ago 9m 8036k - 1.5.0 0da6a335fe13 31630ee26c1c 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (8m) 55s ago 8m 62.2M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (8m) 55s ago 8m 66.5M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (7m) 55s ago 7m 59.1M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (7m) 55s ago 7m 69.1M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (6m) 55s ago 6m 65.0M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (5m) 55s ago 5m 60.9M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:30:48.354 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (8m) 55s ago 10m 52.1M - 2.43.0 a07b618ecd1d 21713d7e702b 2024-02-28T16:30:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.367+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0fdc06b2e0 msgr2=0x7f0fdc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.367+0000 7f0ffd600700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0fdc06b2e0 0x7f0fdc06d790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009b80 tx=0x7f0fe400b040 comp rx=0 tx=0).stop 2024-02-28T16:30:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.367+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 msgr2=0x7f0ff810fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.367+0000 7f0ffd600700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 0x7f0ff810fb00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009fd0 tx=0x7f0fe8009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.368+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 shutdown_connections 2024-02-28T16:30:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.368+0000 7f0ffd600700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0fdc06b2e0 0x7f0fdc06d790 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.368+0000 7f0ffd600700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ff80fcef0 0x7f0ff810fb00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.368+0000 7f0ffd600700 1 --2- 172.21.15.1:0/2751528182 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ff80fe0e0 0x7f0ff8110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.368+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 >> 172.21.15.1:0/2751528182 conn(0x7f0ff80f8620 msgr2=0x7f0ff80ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.368+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 shutdown_connections 2024-02-28T16:30:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.369+0000 7f0ffd600700 1 -- 172.21.15.1:0/2751528182 wait complete. 2024-02-28T16:30:48.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:48 smithi001 ceph-mon[28991]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:48.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:48 smithi001 ceph-mon[28991]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:48.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.552+0000 7ffb8c800700 1 -- 172.21.15.1:0/2415712133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 msgr2=0x7ffb84102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:48.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.552+0000 7ffb8c800700 1 --2- 172.21.15.1:0/2415712133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 0x7ffb84102780 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ffb74004940 tx=0x7ffb74009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:48.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.552+0000 7ffb8c800700 1 -- 172.21.15.1:0/2415712133 shutdown_connections 2024-02-28T16:30:48.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.552+0000 7ffb8c800700 1 --2- 172.21.15.1:0/2415712133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 0x7ffb84102780 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:48.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.552+0000 7ffb8c800700 1 --2- 172.21.15.1:0/2415712133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb840fda80 0x7ffb840ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:48.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.552+0000 7ffb8c800700 1 -- 172.21.15.1:0/2415712133 >> 172.21.15.1:0/2415712133 conn(0x7ffb840f74e0 msgr2=0x7ffb840f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:48.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.552+0000 7ffb8c800700 1 -- 172.21.15.1:0/2415712133 shutdown_connections 2024-02-28T16:30:48.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.552+0000 7ffb8c800700 1 -- 172.21.15.1:0/2415712133 wait complete. 2024-02-28T16:30:48.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.553+0000 7ffb8c800700 1 Processor -- start 2024-02-28T16:30:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.553+0000 7ffb8c800700 1 -- start start 2024-02-28T16:30:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.554+0000 7ffb8c800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb840fda80 0x7ffb8410fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.554+0000 7ffb8c800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 0x7ffb84110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.554+0000 7ffb8c800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb84110720 con 0x7ffb841003a0 2024-02-28T16:30:48.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.554+0000 7ffb8c800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb84110f60 con 0x7ffb840fda80 2024-02-28T16:30:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.554+0000 7ffb89400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 0x7ffb84110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.554+0000 7ffb89400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 0x7ffb84110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44764/0 (socket says 172.21.15.1:44764) 2024-02-28T16:30:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.554+0000 7ffb89400700 1 -- 172.21.15.1:0/540813968 learned_addr learned my addr 172.21.15.1:0/540813968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:48.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.555+0000 7ffb89e00700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb840fda80 0x7ffb8410fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.555+0000 7ffb89400700 1 -- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb840fda80 msgr2=0x7ffb8410fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.555+0000 7ffb89400700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb840fda80 0x7ffb8410fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.555+0000 7ffb89400700 1 -- 172.21.15.1:0/540813968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb74005040 con 0x7ffb841003a0 2024-02-28T16:30:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.556+0000 7ffb89400700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 0x7ffb84110090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009fd0 tx=0x7ffb74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.556+0000 7ffb7e000700 1 -- 172.21.15.1:0/540813968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb7400e040 con 0x7ffb841003a0 2024-02-28T16:30:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.556+0000 7ffb7e000700 1 -- 172.21.15.1:0/540813968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb7400bcc0 con 0x7ffb841003a0 2024-02-28T16:30:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.556+0000 7ffb7e000700 1 -- 172.21.15.1:0/540813968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb7401c630 con 0x7ffb841003a0 2024-02-28T16:30:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.556+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb841111e0 con 0x7ffb841003a0 2024-02-28T16:30:48.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.557+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb84111720 con 0x7ffb841003a0 2024-02-28T16:30:48.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.558+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb840f9080 con 0x7ffb841003a0 2024-02-28T16:30:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.560+0000 7ffb7e000700 1 -- 172.21.15.1:0/540813968 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7ffb74013980 con 0x7ffb841003a0 2024-02-28T16:30:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.561+0000 7ffb7e000700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ffb6c06b2e0 0x7ffb6c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.561+0000 7ffb7e000700 1 -- 172.21.15.1:0/540813968 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7ffb7408b480 con 0x7ffb841003a0 2024-02-28T16:30:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.562+0000 7ffb89e00700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ffb6c06b2e0 0x7ffb6c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:48.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.564+0000 7ffb89e00700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ffb6c06b2e0 0x7ffb6c06d790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffb78006fd0 tx=0x7ffb78008040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:48.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:48.565+0000 7ffb7e000700 1 -- 172.21.15.1:0/540813968 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7ffb74058ff0 con 0x7ffb841003a0 2024-02-28T16:30:48.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:48 smithi066 ceph-mon[36236]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:48.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:48 smithi066 ceph-mon[36236]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:49.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.023+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffb84105a90 con 0x7ffb841003a0 2024-02-28T16:30:49.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.024+0000 7ffb7e000700 1 -- 172.21.15.1:0/540813968 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7ffb74058b80 con 0x7ffb841003a0 2024-02-28T16:30:49.014 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:30:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:30:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:30:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:30:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:30:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:30:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:30:49.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 13, 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:30:49.015 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:30:49.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ffb6c06b2e0 msgr2=0x7ffb6c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:49.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ffb6c06b2e0 0x7ffb6c06d790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffb78006fd0 tx=0x7ffb78008040 comp rx=0 tx=0).stop 2024-02-28T16:30:49.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 msgr2=0x7ffb84110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:49.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 0x7ffb84110090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009fd0 tx=0x7ffb74009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:49.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 shutdown_connections 2024-02-28T16:30:49.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7ffb6c06b2e0 0x7ffb6c06d790 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb840fda80 0x7ffb8410fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 --2- 172.21.15.1:0/540813968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb841003a0 0x7ffb84110090 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.029+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 >> 172.21.15.1:0/540813968 conn(0x7ffb840f74e0 msgr2=0x7ffb84101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:49.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.031+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 shutdown_connections 2024-02-28T16:30:49.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.031+0000 7ffb8c800700 1 -- 172.21.15.1:0/540813968 wait complete. 2024-02-28T16:30:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.229+0000 7f0980a00700 1 -- 172.21.15.1:0/2884211826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fcfb0 msgr2=0x7f097c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.229+0000 7f0980a00700 1 --2- 172.21.15.1:0/2884211826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fcfb0 0x7f097c0fd420 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0968004940 tx=0x7f0968009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.229+0000 7f0980a00700 1 -- 172.21.15.1:0/2884211826 shutdown_connections 2024-02-28T16:30:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.229+0000 7f0980a00700 1 --2- 172.21.15.1:0/2884211826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fcfb0 0x7f097c0fd420 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.229+0000 7f0980a00700 1 --2- 172.21.15.1:0/2884211826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f097c0fbe40 0x7f097c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.229+0000 7f0980a00700 1 -- 172.21.15.1:0/2884211826 >> 172.21.15.1:0/2884211826 conn(0x7f097c0f74e0 msgr2=0x7f097c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.229+0000 7f0980a00700 1 -- 172.21.15.1:0/2884211826 shutdown_connections 2024-02-28T16:30:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.230+0000 7f0980a00700 1 -- 172.21.15.1:0/2884211826 wait complete. 2024-02-28T16:30:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.231+0000 7f0980a00700 1 Processor -- start 2024-02-28T16:30:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.231+0000 7f0980a00700 1 -- start start 2024-02-28T16:30:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.231+0000 7f0980a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fbe40 0x7f097c0fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.232+0000 7f0980a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f097c0fcfb0 0x7f097c0fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.232+0000 7f0980a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0fe980 con 0x7f097c0fbe40 2024-02-28T16:30:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.232+0000 7f0980a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0feaf0 con 0x7f097c0fcfb0 2024-02-28T16:30:49.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.232+0000 7f0979e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fbe40 0x7f097c0fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:49.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.232+0000 7f0979e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fbe40 0x7f097c0fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44782/0 (socket says 172.21.15.1:44782) 2024-02-28T16:30:49.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.232+0000 7f0979e00700 1 -- 172.21.15.1:0/1371986541 learned_addr learned my addr 172.21.15.1:0/1371986541 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:30:49.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.232+0000 7f0979400700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f097c0fcfb0 0x7f097c0fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:49.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.233+0000 7f0979e00700 1 -- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f097c0fcfb0 msgr2=0x7f097c0fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:49.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.233+0000 7f0979e00700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f097c0fcfb0 0x7f097c0fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.233+0000 7f0979e00700 1 -- 172.21.15.1:0/1371986541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0968005040 con 0x7f097c0fbe40 2024-02-28T16:30:49.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.233+0000 7f0979e00700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fbe40 0x7f097c0fff00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f096c002a50 tx=0x7f096c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:49.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.234+0000 7f0972000700 1 -- 172.21.15.1:0/1371986541 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f096c007cc0 con 0x7f097c0fbe40 2024-02-28T16:30:49.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.234+0000 7f0972000700 1 -- 172.21.15.1:0/1371986541 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f096c00f040 con 0x7f097c0fbe40 2024-02-28T16:30:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.234+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c0fedd0 con 0x7f097c0fbe40 2024-02-28T16:30:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.234+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c115990 con 0x7f097c0fbe40 2024-02-28T16:30:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.234+0000 7f0972000700 1 -- 172.21.15.1:0/1371986541 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f096c00da30 con 0x7f097c0fbe40 2024-02-28T16:30:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.235+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c0052e0 con 0x7f097c0fbe40 2024-02-28T16:30:49.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.237+0000 7f0972000700 1 -- 172.21.15.1:0/1371986541 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 24) v1 ==== 89012+0+0 (secure 0 0 0) 0x7f096c00db90 con 0x7f097c0fbe40 2024-02-28T16:30:49.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.238+0000 7f0972000700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f096006b2e0 0x7f096006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:30:49.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.238+0000 7f0972000700 1 -- 172.21.15.1:0/1371986541 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f096c089bd0 con 0x7f097c0fbe40 2024-02-28T16:30:49.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.239+0000 7f0979400700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f096006b2e0 0x7f096006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:30:49.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.240+0000 7f0979400700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f096006b2e0 0x7f096006d790 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0968004f90 tx=0x7f0968009e30 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:30:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.246+0000 7f0972000700 1 -- 172.21.15.1:0/1371986541 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f096c057640 con 0x7f097c0fbe40 2024-02-28T16:30:49.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.586+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f095c000bf0 con 0x7f096006b2e0 2024-02-28T16:30:49.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.590+0000 7f0972000700 1 -- 172.21.15.1:0/1371986541 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f095c000bf0 con 0x7f096006b2e0 2024-02-28T16:30:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:30:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:30:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:30:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:30:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [], 2024-02-28T16:30:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "1/2 daemons upgraded", 2024-02-28T16:30:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "", 2024-02-28T16:30:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:30:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:30:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f096006b2e0 msgr2=0x7f096006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f096006b2e0 0x7f096006d790 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0968004f90 tx=0x7f0968009e30 comp rx=0 tx=0).stop 2024-02-28T16:30:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fbe40 msgr2=0x7f097c0fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:30:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fbe40 0x7f097c0fff00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f096c002a50 tx=0x7f096c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 shutdown_connections 2024-02-28T16:30:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f096006b2e0 0x7f096006d790 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f097c0fbe40 0x7f097c0fff00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 --2- 172.21.15.1:0/1371986541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f097c0fcfb0 0x7f097c0fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:30:49.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 >> 172.21.15.1:0/1371986541 conn(0x7f097c0f74e0 msgr2=0x7f097c0656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:30:49.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 shutdown_connections 2024-02-28T16:30:49.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:30:49.593+0000 7f0980a00700 1 -- 172.21.15.1:0/1371986541 wait complete. 2024-02-28T16:30:49.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:49 smithi066 ceph-mon[36236]: pgmap v38: 65 pgs: 65 active+clean; 204 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 158 op/s 2024-02-28T16:30:49.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:49 smithi066 ceph-mon[36236]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:49.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:49 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/540813968' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:30:49.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:49 smithi001 ceph-mon[28991]: pgmap v38: 65 pgs: 65 active+clean; 204 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.0 MiB/s wr, 158 op/s 2024-02-28T16:30:49.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:49 smithi001 ceph-mon[28991]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:49.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:49 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/540813968' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:30:50.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:50 smithi066 ceph-mon[36236]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:50.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:50 smithi001 ceph-mon[28991]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:30:51.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:51 smithi066 ceph-mon[36236]: pgmap v39: 65 pgs: 65 active+clean; 212 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.7 MiB/s wr, 214 op/s 2024-02-28T16:30:51.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:51 smithi001 ceph-mon[28991]: pgmap v39: 65 pgs: 65 active+clean; 212 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2.7 MiB/s wr, 214 op/s 2024-02-28T16:30:51.890 INFO:tasks.workunit.client.0.smithi001.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-28T16:30:51.925 INFO:tasks.workunit.client.0.smithi001.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-28T16:30:51.925 INFO:tasks.workunit.client.0.smithi001.stderr:+ make 2024-02-28T16:30:52.293 INFO:tasks.workunit.client.0.smithi001.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-28T16:30:53.512 INFO:tasks.workunit.client.0.smithi001.stderr:++ readlink -f fsstress 2024-02-28T16:30:53.518 INFO:tasks.workunit.client.0.smithi001.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-02-28T16:30:53.518 INFO:tasks.workunit.client.0.smithi001.stderr:+ popd 2024-02-28T16:30:53.524 INFO:tasks.workunit.client.0.smithi001.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-28T16:30:53.524 INFO:tasks.workunit.client.0.smithi001.stderr:+ popd 2024-02-28T16:30:53.528 INFO:tasks.workunit.client.0.smithi001.stdout:~/cephtest/mnt.0/client.0/tmp 2024-02-28T16:30:53.530 INFO:tasks.workunit.client.0.smithi001.stderr:++ mktemp -d -p . 2024-02-28T16:30:53.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ T=./tmp.r3r2qJkLtM 2024-02-28T16:30:53.539 INFO:tasks.workunit.client.0.smithi001.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.r3r2qJkLtM -l 1 -n 1000 -p 10 -v 2024-02-28T16:30:53.552 INFO:tasks.workunit.client.0.smithi001.stdout:seed = 1708638790 2024-02-28T16:30:53.565 INFO:tasks.workunit.client.0.smithi001.stdout:9/0: write - no filename 2024-02-28T16:30:53.565 INFO:tasks.workunit.client.0.smithi001.stdout:9/1: fdatasync - no filename 2024-02-28T16:30:53.565 INFO:tasks.workunit.client.0.smithi001.stdout:9/2: stat - no entries 2024-02-28T16:30:53.565 INFO:tasks.workunit.client.0.smithi001.stdout:9/3: write - no filename 2024-02-28T16:30:53.568 INFO:tasks.workunit.client.0.smithi001.stdout:1/0: dread - no filename 2024-02-28T16:30:53.568 INFO:tasks.workunit.client.0.smithi001.stdout:1/1: rmdir - no directory 2024-02-28T16:30:53.568 INFO:tasks.workunit.client.0.smithi001.stdout:1/2: dwrite - no filename 2024-02-28T16:30:53.568 INFO:tasks.workunit.client.0.smithi001.stdout:9/4: creat f0 x:0 0 0 2024-02-28T16:30:53.568 INFO:tasks.workunit.client.0.smithi001.stdout:1/3: chown . 27982080 1 2024-02-28T16:30:53.569 INFO:tasks.workunit.client.0.smithi001.stdout:1/4: dwrite - no filename 2024-02-28T16:30:53.569 INFO:tasks.workunit.client.0.smithi001.stdout:1/5: dwrite - no filename 2024-02-28T16:30:53.569 INFO:tasks.workunit.client.0.smithi001.stdout:1/6: unlink - no file 2024-02-28T16:30:53.569 INFO:tasks.workunit.client.0.smithi001.stdout:1/7: truncate - no filename 2024-02-28T16:30:53.569 INFO:tasks.workunit.client.0.smithi001.stdout:1/8: dwrite - no filename 2024-02-28T16:30:53.569 INFO:tasks.workunit.client.0.smithi001.stdout:1/9: dread - no filename 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/10: chown . 58374 1 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/11: dread - no filename 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/12: dread - no filename 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/13: chown . 12901231 1 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/14: chown . 5 1 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/15: write - no filename 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/16: dread - no filename 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/17: link - no file 2024-02-28T16:30:53.570 INFO:tasks.workunit.client.0.smithi001.stdout:1/18: readlink - no filename 2024-02-28T16:30:53.571 INFO:tasks.workunit.client.0.smithi001.stdout:2/0: dwrite - no filename 2024-02-28T16:30:53.571 INFO:tasks.workunit.client.0.smithi001.stdout:2/1: write - no filename 2024-02-28T16:30:53.571 INFO:tasks.workunit.client.0.smithi001.stdout:9/5: symlink l1 0 2024-02-28T16:30:53.573 INFO:tasks.workunit.client.0.smithi001.stdout:1/19: mkdir d0 0 2024-02-28T16:30:53.573 INFO:tasks.workunit.client.0.smithi001.stdout:9/6: fdatasync f0 0 2024-02-28T16:30:53.574 INFO:tasks.workunit.client.0.smithi001.stdout:3/0: chown . 24 1 2024-02-28T16:30:53.580 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:53 smithi001 ceph-mon[28991]: pgmap v40: 65 pgs: 65 active+clean; 212 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.7 MiB/s wr, 147 op/s 2024-02-28T16:30:53.580 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:53 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:30:53.581 INFO:tasks.workunit.client.0.smithi001.stdout:9/7: creat f2 x:0 0 0 2024-02-28T16:30:53.584 INFO:tasks.workunit.client.0.smithi001.stdout:5/0: unlink - no file 2024-02-28T16:30:53.586 INFO:tasks.workunit.client.0.smithi001.stdout:1/20: creat d0/f1 x:0 0 0 2024-02-28T16:30:53.588 INFO:tasks.workunit.client.0.smithi001.stdout:3/1: mkdir d0 0 2024-02-28T16:30:53.590 INFO:tasks.workunit.client.0.smithi001.stdout:3/2: rename d0 to d0/d1 22 2024-02-28T16:30:53.590 INFO:tasks.workunit.client.0.smithi001.stdout:3/3: write - no filename 2024-02-28T16:30:53.590 INFO:tasks.workunit.client.0.smithi001.stdout:3/4: link - no file 2024-02-28T16:30:53.590 INFO:tasks.workunit.client.0.smithi001.stdout:3/5: write - no filename 2024-02-28T16:30:53.591 INFO:tasks.workunit.client.0.smithi001.stdout:9/8: rename l1 to l3 0 2024-02-28T16:30:53.593 INFO:tasks.workunit.client.0.smithi001.stdout:4/0: symlink l0 0 2024-02-28T16:30:53.595 INFO:tasks.workunit.client.0.smithi001.stdout:5/1: mknod c0 0 2024-02-28T16:30:53.595 INFO:tasks.workunit.client.0.smithi001.stdout:5/2: write - no filename 2024-02-28T16:30:53.595 INFO:tasks.workunit.client.0.smithi001.stdout:5/3: truncate - no filename 2024-02-28T16:30:53.598 INFO:tasks.workunit.client.0.smithi001.stdout:4/1: mknod c1 0 2024-02-28T16:30:53.598 INFO:tasks.workunit.client.0.smithi001.stdout:4/2: write - no filename 2024-02-28T16:30:53.598 INFO:tasks.workunit.client.0.smithi001.stdout:4/3: write - no filename 2024-02-28T16:30:53.598 INFO:tasks.workunit.client.0.smithi001.stdout:4/4: rmdir - no directory 2024-02-28T16:30:53.599 INFO:tasks.workunit.client.0.smithi001.stdout:7/0: read - no filename 2024-02-28T16:30:53.599 INFO:tasks.workunit.client.0.smithi001.stdout:7/1: dwrite - no filename 2024-02-28T16:30:53.599 INFO:tasks.workunit.client.0.smithi001.stdout:7/2: stat - no entries 2024-02-28T16:30:53.599 INFO:tasks.workunit.client.0.smithi001.stdout:3/6: getdents d0 0 2024-02-28T16:30:53.600 INFO:tasks.workunit.client.0.smithi001.stdout:6/0: symlink l0 0 2024-02-28T16:30:53.600 INFO:tasks.workunit.client.0.smithi001.stdout:9/9: mknod c4 0 2024-02-28T16:30:53.601 INFO:tasks.workunit.client.0.smithi001.stdout:5/4: rename c0 to c1 0 2024-02-28T16:30:53.601 INFO:tasks.workunit.client.0.smithi001.stdout:5/5: dread - no filename 2024-02-28T16:30:53.602 INFO:tasks.workunit.client.0.smithi001.stdout:5/6: read - no filename 2024-02-28T16:30:53.602 INFO:tasks.workunit.client.0.smithi001.stdout:3/7: rename d0 to d0/d2 22 2024-02-28T16:30:53.602 INFO:tasks.workunit.client.0.smithi001.stdout:3/8: unlink - no file 2024-02-28T16:30:53.603 INFO:tasks.workunit.client.0.smithi001.stdout:4/5: creat f2 x:0 0 0 2024-02-28T16:30:53.604 INFO:tasks.workunit.client.0.smithi001.stdout:6/1: creat f1 x:0 0 0 2024-02-28T16:30:53.604 INFO:tasks.workunit.client.0.smithi001.stdout:9/10: mkdir d5 0 2024-02-28T16:30:53.605 INFO:tasks.workunit.client.0.smithi001.stdout:8/0: chown . 751136763 1 2024-02-28T16:30:53.605 INFO:tasks.workunit.client.0.smithi001.stdout:8/1: truncate - no filename 2024-02-28T16:30:53.605 INFO:tasks.workunit.client.0.smithi001.stdout:8/2: write - no filename 2024-02-28T16:30:53.605 INFO:tasks.workunit.client.0.smithi001.stdout:8/3: dwrite - no filename 2024-02-28T16:30:53.605 INFO:tasks.workunit.client.0.smithi001.stdout:8/4: link - no file 2024-02-28T16:30:53.605 INFO:tasks.workunit.client.0.smithi001.stdout:8/5: dread - no filename 2024-02-28T16:30:53.605 INFO:tasks.workunit.client.0.smithi001.stdout:8/6: write - no filename 2024-02-28T16:30:53.606 INFO:tasks.workunit.client.0.smithi001.stdout:8/7: rename - no filename 2024-02-28T16:30:53.606 INFO:tasks.workunit.client.0.smithi001.stdout:8/8: dread - no filename 2024-02-28T16:30:53.606 INFO:tasks.workunit.client.0.smithi001.stdout:3/9: rename d0 to d0/d3 22 2024-02-28T16:30:53.606 INFO:tasks.workunit.client.0.smithi001.stdout:3/10: dwrite - no filename 2024-02-28T16:30:53.606 INFO:tasks.workunit.client.0.smithi001.stdout:3/11: dread - no filename 2024-02-28T16:30:53.606 INFO:tasks.workunit.client.0.smithi001.stdout:3/12: write - no filename 2024-02-28T16:30:53.606 INFO:tasks.workunit.client.0.smithi001.stdout:3/13: write - no filename 2024-02-28T16:30:53.607 INFO:tasks.workunit.client.0.smithi001.stdout:6/2: mknod c2 0 2024-02-28T16:30:53.611 INFO:tasks.workunit.client.0.smithi001.stdout:4/6: mkdir d3 0 2024-02-28T16:30:53.614 INFO:tasks.workunit.client.0.smithi001.stdout:4/7: chown d3 135 1 2024-02-28T16:30:53.615 INFO:tasks.workunit.client.0.smithi001.stdout:8/9: symlink l0 0 2024-02-28T16:30:53.615 INFO:tasks.workunit.client.0.smithi001.stdout:6/3: mknod c3 0 2024-02-28T16:30:53.618 INFO:tasks.workunit.client.0.smithi001.stdout:9/11: creat d5/f6 x:0 0 0 2024-02-28T16:30:53.618 INFO:tasks.workunit.client.0.smithi001.stdout:3/14: rmdir d0 0 2024-02-28T16:30:53.618 INFO:tasks.workunit.client.0.smithi001.stdout:8/10: readlink l0 0 2024-02-28T16:30:53.621 INFO:tasks.workunit.client.0.smithi001.stdout:8/11: readlink l0 0 2024-02-28T16:30:53.625 INFO:tasks.workunit.client.0.smithi001.stdout:4/8: rename l0 to d3/l4 0 2024-02-28T16:30:53.625 INFO:tasks.workunit.client.0.smithi001.stdout:3/15: mkdir d4 0 2024-02-28T16:30:53.625 INFO:tasks.workunit.client.0.smithi001.stdout:3/16: dwrite - no filename 2024-02-28T16:30:53.630 INFO:tasks.workunit.client.0.smithi001.stdout:1/21: dwrite d0/f1 [0,4194304] 0 2024-02-28T16:30:53.630 INFO:tasks.workunit.client.0.smithi001.stdout:8/12: creat f1 x:0 0 0 2024-02-28T16:30:53.633 INFO:tasks.workunit.client.0.smithi001.stdout:4/9: mkdir d3/d5 0 2024-02-28T16:30:53.635 INFO:tasks.workunit.client.0.smithi001.stdout:8/13: symlink l2 0 2024-02-28T16:30:53.636 INFO:tasks.workunit.client.0.smithi001.stdout:3/17: getdents d4 0 2024-02-28T16:30:53.637 INFO:tasks.workunit.client.0.smithi001.stdout:4/10: dread - f2 zero size 2024-02-28T16:30:53.640 INFO:tasks.workunit.client.0.smithi001.stdout:8/14: creat f3 x:0 0 0 2024-02-28T16:30:53.640 INFO:tasks.workunit.client.0.smithi001.stdout:1/22: link d0/f1 d0/f2 0 2024-02-28T16:30:53.646 INFO:tasks.workunit.client.0.smithi001.stdout:4/11: creat d3/d5/f6 x:0 0 0 2024-02-28T16:30:53.647 INFO:tasks.workunit.client.0.smithi001.stdout:8/15: mkdir d4 0 2024-02-28T16:30:53.654 INFO:tasks.workunit.client.0.smithi001.stdout:4/12: mkdir d3/d5/d7 0 2024-02-28T16:30:53.655 INFO:tasks.workunit.client.0.smithi001.stdout:8/16: write f1 [113757,67188] 0 2024-02-28T16:30:53.662 INFO:tasks.workunit.client.0.smithi001.stdout:4/13: symlink d3/d5/l8 0 2024-02-28T16:30:53.669 INFO:tasks.workunit.client.0.smithi001.stdout:4/14: chown d3/d5/d7 0 1 2024-02-28T16:30:53.671 INFO:tasks.workunit.client.0.smithi001.stdout:8/17: link l2 d4/l5 0 2024-02-28T16:30:53.673 INFO:tasks.workunit.client.0.smithi001.stdout:9/12: dwrite d5/f6 [0,4194304] 0 2024-02-28T16:30:53.678 INFO:tasks.workunit.client.0.smithi001.stdout:4/15: mkdir d3/d5/d7/d9 0 2024-02-28T16:30:53.680 INFO:tasks.workunit.client.0.smithi001.stdout:8/18: rename l2 to d4/l6 0 2024-02-28T16:30:53.682 INFO:tasks.workunit.client.0.smithi001.stdout:9/13: readlink l3 0 2024-02-28T16:30:53.683 INFO:tasks.workunit.client.0.smithi001.stdout:4/16: creat d3/fa x:0 0 0 2024-02-28T16:30:53.685 INFO:tasks.workunit.client.0.smithi001.stdout:4/17: fdatasync d3/fa 0 2024-02-28T16:30:53.686 INFO:tasks.workunit.client.0.smithi001.stdout:9/14: mknod d5/c7 0 2024-02-28T16:30:53.688 INFO:tasks.workunit.client.0.smithi001.stdout:4/18: truncate f2 986456 0 2024-02-28T16:30:53.693 INFO:tasks.workunit.client.0.smithi001.stdout:6/4: fsync f1 0 2024-02-28T16:30:53.695 INFO:tasks.workunit.client.0.smithi001.stdout:4/19: write d3/d5/f6 [966583,125791] 0 2024-02-28T16:30:53.695 INFO:tasks.workunit.client.0.smithi001.stdout:5/7: stat c1 0 2024-02-28T16:30:53.703 INFO:tasks.workunit.client.0.smithi001.stdout:5/8: chown c1 19337868 1 2024-02-28T16:30:53.703 INFO:tasks.workunit.client.0.smithi001.stdout:5/9: dread - no filename 2024-02-28T16:30:53.703 INFO:tasks.workunit.client.0.smithi001.stdout:6/5: rename l0 to l4 0 2024-02-28T16:30:53.711 INFO:tasks.workunit.client.0.smithi001.stdout:6/6: fsync f1 0 2024-02-28T16:30:53.716 INFO:tasks.workunit.client.0.smithi001.stdout:5/10: creat f2 x:0 0 0 2024-02-28T16:30:53.727 INFO:tasks.workunit.client.0.smithi001.stdout:5/11: chown c1 88 1 2024-02-28T16:30:53.729 INFO:tasks.workunit.client.0.smithi001.stdout:8/19: dwrite f1 [0,4194304] 0 2024-02-28T16:30:53.732 INFO:tasks.workunit.client.0.smithi001.stdout:5/12: dread - f2 zero size 2024-02-28T16:30:53.738 INFO:tasks.workunit.client.0.smithi001.stdout:1/23: dwrite d0/f2 [4194304,4194304] 0 2024-02-28T16:30:53.741 INFO:tasks.workunit.client.0.smithi001.stdout:5/13: write f2 [677950,18792] 0 2024-02-28T16:30:53.744 INFO:tasks.workunit.client.0.smithi001.stdout:9/15: dwrite d5/f6 [0,4194304] 0 2024-02-28T16:30:53.746 INFO:tasks.workunit.client.0.smithi001.stdout:5/14: truncate f2 1488051 0 2024-02-28T16:30:53.747 INFO:tasks.workunit.client.0.smithi001.stdout:8/20: getdents d4 0 2024-02-28T16:30:53.752 INFO:tasks.workunit.client.0.smithi001.stdout:4/20: dwrite d3/fa [0,4194304] 0 2024-02-28T16:30:53.777 INFO:tasks.workunit.client.0.smithi001.stdout:1/24: dwrite d0/f1 [4194304,4194304] 0 2024-02-28T16:30:53.781 INFO:tasks.workunit.client.0.smithi001.stdout:9/16: mknod d5/c8 0 2024-02-28T16:30:53.783 INFO:tasks.workunit.client.0.smithi001.stdout:5/15: creat f3 x:0 0 0 2024-02-28T16:30:53.785 INFO:tasks.workunit.client.0.smithi001.stdout:8/21: mkdir d4/d7 0 2024-02-28T16:30:53.788 INFO:tasks.workunit.client.0.smithi001.stdout:8/22: dread - f3 zero size 2024-02-28T16:30:53.793 INFO:tasks.workunit.client.0.smithi001.stdout:9/17: rename f0 to d5/f9 0 2024-02-28T16:30:53.797 INFO:tasks.workunit.client.0.smithi001.stdout:1/25: unlink d0/f2 0 2024-02-28T16:30:53.802 INFO:tasks.workunit.client.0.smithi001.stdout:9/18: chown l3 22240555 1 2024-02-28T16:30:53.805 INFO:tasks.workunit.client.0.smithi001.stdout:1/26: symlink d0/l3 0 2024-02-28T16:30:53.806 INFO:tasks.workunit.client.0.smithi001.stdout:8/23: dread f1 [0,4194304] 0 2024-02-28T16:30:53.820 INFO:tasks.workunit.client.0.smithi001.stdout:5/16: getdents . 0 2024-02-28T16:30:53.820 INFO:tasks.workunit.client.0.smithi001.stdout:5/17: readlink - no filename 2024-02-28T16:30:53.822 INFO:tasks.workunit.client.0.smithi001.stdout:9/19: rmdir d5 39 2024-02-28T16:30:53.824 INFO:tasks.workunit.client.0.smithi001.stdout:9/20: write f2 [368602,24738] 0 2024-02-28T16:30:53.826 INFO:tasks.workunit.client.0.smithi001.stdout:5/18: creat f4 x:0 0 0 2024-02-28T16:30:53.828 INFO:tasks.workunit.client.0.smithi001.stdout:5/19: chown f2 13305 1 2024-02-28T16:30:53.832 INFO:tasks.workunit.client.0.smithi001.stdout:9/21: dread f2 [0,4194304] 0 2024-02-28T16:30:53.835 INFO:tasks.workunit.client.0.smithi001.stdout:1/27: dread d0/f1 [4194304,4194304] 0 2024-02-28T16:30:53.837 INFO:tasks.workunit.client.0.smithi001.stdout:8/24: dwrite f1 [0,4194304] 0 2024-02-28T16:30:53.839 INFO:tasks.workunit.client.0.smithi001.stdout:9/22: symlink d5/la 0 2024-02-28T16:30:53.841 INFO:tasks.workunit.client.0.smithi001.stdout:8/25: dread - f3 zero size 2024-02-28T16:30:53.841 INFO:tasks.workunit.client.0.smithi001.stdout:1/28: symlink d0/l4 0 2024-02-28T16:30:53.841 INFO:tasks.workunit.client.0.smithi001.stdout:5/20: dread f2 [0,4194304] 0 2024-02-28T16:30:53.850 INFO:tasks.workunit.client.0.smithi001.stdout:5/21: creat f5 x:0 0 0 2024-02-28T16:30:53.861 INFO:tasks.workunit.client.0.smithi001.stdout:5/22: dread f2 [0,4194304] 0 2024-02-28T16:30:53.861 INFO:tasks.workunit.client.0.smithi001.stdout:1/29: truncate d0/f1 7848229 0 2024-02-28T16:30:53.865 INFO:tasks.workunit.client.0.smithi001.stdout:5/23: chown f4 222729 1 2024-02-28T16:30:53.874 INFO:tasks.workunit.client.0.smithi001.stdout:8/26: dwrite f1 [4194304,4194304] 0 2024-02-28T16:30:53.874 INFO:tasks.workunit.client.0.smithi001.stdout:9/23: dwrite d5/f6 [0,4194304] 0 2024-02-28T16:30:53.877 INFO:tasks.workunit.client.0.smithi001.stdout:9/24: stat d5/f9 0 2024-02-28T16:30:53.879 INFO:tasks.workunit.client.0.smithi001.stdout:8/27: truncate f3 348306 0 2024-02-28T16:30:53.888 INFO:tasks.workunit.client.0.smithi001.stdout:9/25: creat d5/fb x:0 0 0 2024-02-28T16:30:53.891 INFO:tasks.workunit.client.0.smithi001.stdout:9/26: write d5/f6 [3960876,22994] 0 2024-02-28T16:30:53.891 INFO:tasks.workunit.client.0.smithi001.stdout:8/28: symlink d4/l8 0 2024-02-28T16:30:53.894 INFO:tasks.workunit.client.0.smithi001.stdout:9/27: symlink d5/lc 0 2024-02-28T16:30:53.898 INFO:tasks.workunit.client.0.smithi001.stdout:9/28: rename f2 to d5/fd 0 2024-02-28T16:30:53.899 INFO:tasks.workunit.client.0.smithi001.stdout:8/29: rename d4/l6 to d4/l9 0 2024-02-28T16:30:53.901 INFO:tasks.workunit.client.0.smithi001.stdout:9/29: creat d5/fe x:0 0 0 2024-02-28T16:30:53.901 INFO:tasks.workunit.client.0.smithi001.stdout:8/30: truncate f3 1204758 0 2024-02-28T16:30:53.901 INFO:tasks.workunit.client.0.smithi001.stdout:9/30: readlink l3 0 2024-02-28T16:30:53.903 INFO:tasks.workunit.client.0.smithi001.stdout:8/31: rmdir d4 39 2024-02-28T16:30:53.905 INFO:tasks.workunit.client.0.smithi001.stdout:9/31: dread d5/fd [0,4194304] 0 2024-02-28T16:30:53.906 INFO:tasks.workunit.client.0.smithi001.stdout:9/32: dread - d5/f9 zero size 2024-02-28T16:30:53.908 INFO:tasks.workunit.client.0.smithi001.stdout:9/33: fdatasync d5/fe 0 2024-02-28T16:30:53.912 INFO:tasks.workunit.client.0.smithi001.stdout:8/32: link d4/l5 d4/d7/la 0 2024-02-28T16:30:53.918 INFO:tasks.workunit.client.0.smithi001.stdout:1/30: dwrite d0/f1 [4194304,4194304] 0 2024-02-28T16:30:53.922 INFO:tasks.workunit.client.0.smithi001.stdout:8/33: link d4/d7/la d4/d7/lb 0 2024-02-28T16:30:53.926 INFO:tasks.workunit.client.0.smithi001.stdout:9/34: getdents d5 0 2024-02-28T16:30:53.929 INFO:tasks.workunit.client.0.smithi001.stdout:1/31: getdents d0 0 2024-02-28T16:30:53.931 INFO:tasks.workunit.client.0.smithi001.stdout:8/34: getdents d4 0 2024-02-28T16:30:53.933 INFO:tasks.workunit.client.0.smithi001.stdout:8/35: write f3 [643935,88459] 0 2024-02-28T16:30:53.937 INFO:tasks.workunit.client.0.smithi001.stdout:9/35: link l3 d5/lf 0 2024-02-28T16:30:53.937 INFO:tasks.workunit.client.0.smithi001.stdout:1/32: getdents d0 0 2024-02-28T16:30:53.943 INFO:tasks.workunit.client.0.smithi001.stdout:1/33: creat d0/f5 x:0 0 0 2024-02-28T16:30:53.945 INFO:tasks.workunit.client.0.smithi001.stdout:8/36: truncate f1 1605481 0 2024-02-28T16:30:53.948 INFO:tasks.workunit.client.0.smithi001.stdout:1/34: unlink d0/f1 0 2024-02-28T16:30:53.950 INFO:tasks.workunit.client.0.smithi001.stdout:8/37: mknod d4/cc 0 2024-02-28T16:30:53.955 INFO:tasks.workunit.client.0.smithi001.stdout:8/38: chown d4/d7/la 1 1 2024-02-28T16:30:53.956 INFO:tasks.workunit.client.0.smithi001.stdout:8/39: readlink d4/l9 0 2024-02-28T16:30:53.956 INFO:tasks.workunit.client.0.smithi001.stdout:9/36: dwrite d5/fb [0,4194304] 0 2024-02-28T16:30:53.959 INFO:tasks.workunit.client.0.smithi001.stdout:1/35: getdents d0 0 2024-02-28T16:30:53.961 INFO:tasks.workunit.client.0.smithi001.stdout:1/36: fsync d0/f5 0 2024-02-28T16:30:53.962 INFO:tasks.workunit.client.0.smithi001.stdout:1/37: dread - d0/f5 zero size 2024-02-28T16:30:53.967 INFO:tasks.workunit.client.0.smithi001.stdout:1/38: creat d0/f6 x:0 0 0 2024-02-28T16:30:53.967 INFO:tasks.workunit.client.0.smithi001.stdout:1/39: fdatasync d0/f6 0 2024-02-28T16:30:53.970 INFO:tasks.workunit.client.0.smithi001.stdout:1/40: write d0/f6 [158048,117226] 0 2024-02-28T16:30:53.977 INFO:tasks.workunit.client.0.smithi001.stdout:1/41: getdents d0 0 2024-02-28T16:30:53.984 INFO:tasks.workunit.client.0.smithi001.stdout:9/37: dwrite d5/fd [0,4194304] 0 2024-02-28T16:30:53.985 INFO:tasks.workunit.client.0.smithi001.stdout:1/42: rename d0/f6 to d0/f7 0 2024-02-28T16:30:53.988 INFO:tasks.workunit.client.0.smithi001.stdout:1/43: read d0/f7 [159601,35807] 0 2024-02-28T16:30:53.991 INFO:tasks.workunit.client.0.smithi001.stdout:1/44: write d0/f5 [190198,52397] 0 2024-02-28T16:30:53.992 INFO:tasks.workunit.client.0.smithi001.stdout:1/45: stat d0 0 2024-02-28T16:30:53.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:53 smithi066 ceph-mon[36236]: pgmap v40: 65 pgs: 65 active+clean; 212 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1.7 MiB/s wr, 147 op/s 2024-02-28T16:30:53.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:53 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:30:53.999 INFO:tasks.workunit.client.0.smithi001.stdout:1/46: write d0/f5 [1083455,46133] 0 2024-02-28T16:30:54.011 INFO:tasks.workunit.client.0.smithi001.stdout:9/38: dread d5/f6 [0,4194304] 0 2024-02-28T16:30:54.014 INFO:tasks.workunit.client.0.smithi001.stdout:8/40: dwrite f1 [0,4194304] 0 2024-02-28T16:30:54.016 INFO:tasks.workunit.client.0.smithi001.stdout:8/41: write f3 [832303,109556] 0 2024-02-28T16:30:54.026 INFO:tasks.workunit.client.0.smithi001.stdout:8/42: getdents d4 0 2024-02-28T16:30:54.049 INFO:tasks.workunit.client.0.smithi001.stdout:1/47: dwrite d0/f7 [0,4194304] 0 2024-02-28T16:30:54.054 INFO:tasks.workunit.client.0.smithi001.stdout:1/48: write d0/f5 [2137150,71191] 0 2024-02-28T16:30:54.056 INFO:tasks.workunit.client.0.smithi001.stdout:9/39: dwrite d5/f6 [0,4194304] 0 2024-02-28T16:30:54.056 INFO:tasks.workunit.client.0.smithi001.stdout:8/43: dwrite f3 [0,4194304] 0 2024-02-28T16:30:54.058 INFO:tasks.workunit.client.0.smithi001.stdout:1/49: chown d0/f7 1797011 1 2024-02-28T16:30:54.060 INFO:tasks.workunit.client.0.smithi001.stdout:1/50: rename d0 to d0/d8 22 2024-02-28T16:30:54.065 INFO:tasks.workunit.client.0.smithi001.stdout:9/40: mknod d5/c10 0 2024-02-28T16:30:54.070 INFO:tasks.workunit.client.0.smithi001.stdout:9/41: unlink c4 0 2024-02-28T16:30:54.074 INFO:tasks.workunit.client.0.smithi001.stdout:9/42: symlink d5/l11 0 2024-02-28T16:30:54.082 INFO:tasks.workunit.client.0.smithi001.stdout:9/43: creat d5/f12 x:0 0 0 2024-02-28T16:30:54.087 INFO:tasks.workunit.client.0.smithi001.stdout:9/44: rename d5/la to d5/l13 0 2024-02-28T16:30:54.089 INFO:tasks.workunit.client.0.smithi001.stdout:9/45: truncate d5/fe 999712 0 2024-02-28T16:30:54.093 INFO:tasks.workunit.client.0.smithi001.stdout:9/46: symlink d5/l14 0 2024-02-28T16:30:54.094 INFO:tasks.workunit.client.0.smithi001.stdout:9/47: chown d5/f6 248702 1 2024-02-28T16:30:54.096 INFO:tasks.workunit.client.0.smithi001.stdout:9/48: fdatasync d5/fb 0 2024-02-28T16:30:54.099 INFO:tasks.workunit.client.0.smithi001.stdout:9/49: creat d5/f15 x:0 0 0 2024-02-28T16:30:54.100 INFO:tasks.workunit.client.0.smithi001.stdout:8/44: fsync f3 0 2024-02-28T16:30:54.101 INFO:tasks.workunit.client.0.smithi001.stdout:1/51: fsync d0/f7 0 2024-02-28T16:30:54.105 INFO:tasks.workunit.client.0.smithi001.stdout:9/50: mknod d5/c16 0 2024-02-28T16:30:54.107 INFO:tasks.workunit.client.0.smithi001.stdout:1/52: mkdir d0/d9 0 2024-02-28T16:30:54.110 INFO:tasks.workunit.client.0.smithi001.stdout:9/51: creat d5/f17 x:0 0 0 2024-02-28T16:30:54.114 INFO:tasks.workunit.client.0.smithi001.stdout:1/53: creat d0/d9/fa x:0 0 0 2024-02-28T16:30:54.117 INFO:tasks.workunit.client.0.smithi001.stdout:8/45: getdents d4/d7 0 2024-02-28T16:30:54.122 INFO:tasks.workunit.client.0.smithi001.stdout:1/54: symlink d0/lb 0 2024-02-28T16:30:54.137 INFO:tasks.workunit.client.0.smithi001.stdout:1/55: link d0/f5 d0/d9/fc 0 2024-02-28T16:30:54.144 INFO:tasks.workunit.client.0.smithi001.stdout:8/46: dwrite f3 [0,4194304] 0 2024-02-28T16:30:54.208 INFO:tasks.workunit.client.0.smithi001.stdout:1/56: getdents d0 0 2024-02-28T16:30:54.248 INFO:tasks.workunit.client.0.smithi001.stdout:8/47: truncate f1 1747586 0 2024-02-28T16:30:54.251 INFO:tasks.workunit.client.0.smithi001.stdout:8/48: creat d4/d7/fd x:0 0 0 2024-02-28T16:30:54.252 INFO:tasks.workunit.client.0.smithi001.stdout:8/49: readlink d4/d7/lb 0 2024-02-28T16:30:54.256 INFO:tasks.workunit.client.0.smithi001.stdout:1/57: dwrite d0/d9/fc [0,4194304] 0 2024-02-28T16:30:54.278 INFO:tasks.workunit.client.0.smithi001.stdout:1/58: dwrite d0/f5 [0,4194304] 0 2024-02-28T16:30:54.281 INFO:tasks.workunit.client.0.smithi001.stdout:8/50: dread f1 [0,4194304] 0 2024-02-28T16:30:54.281 INFO:tasks.workunit.client.0.smithi001.stdout:1/59: write d0/d9/fc [2195537,58382] 0 2024-02-28T16:30:54.287 INFO:tasks.workunit.client.0.smithi001.stdout:1/60: mknod d0/cd 0 2024-02-28T16:30:54.290 INFO:tasks.workunit.client.0.smithi001.stdout:1/61: mknod d0/d9/ce 0 2024-02-28T16:30:54.292 INFO:tasks.workunit.client.0.smithi001.stdout:1/62: truncate d0/f5 4630391 0 2024-02-28T16:30:54.303 INFO:tasks.workunit.client.0.smithi001.stdout:8/51: dwrite f3 [0,4194304] 0 2024-02-28T16:30:54.308 INFO:tasks.workunit.client.0.smithi001.stdout:1/63: dwrite d0/f7 [0,4194304] 0 2024-02-28T16:30:54.310 INFO:tasks.workunit.client.0.smithi001.stdout:1/64: dread - d0/d9/fa zero size 2024-02-28T16:30:54.323 INFO:tasks.workunit.client.0.smithi001.stdout:1/65: truncate d0/f7 845352 0 2024-02-28T16:30:54.326 INFO:tasks.workunit.client.0.smithi001.stdout:1/66: creat d0/d9/ff x:0 0 0 2024-02-28T16:30:54.330 INFO:tasks.workunit.client.0.smithi001.stdout:1/67: mkdir d0/d9/d10 0 2024-02-28T16:30:54.332 INFO:tasks.workunit.client.0.smithi001.stdout:1/68: write d0/d9/ff [467280,16686] 0 2024-02-28T16:30:54.335 INFO:tasks.workunit.client.0.smithi001.stdout:1/69: creat d0/d9/f11 x:0 0 0 2024-02-28T16:30:54.339 INFO:tasks.workunit.client.0.smithi001.stdout:1/70: unlink d0/f5 0 2024-02-28T16:30:54.355 INFO:tasks.workunit.client.0.smithi001.stdout:1/71: dwrite d0/d9/ff [0,4194304] 0 2024-02-28T16:30:54.368 INFO:tasks.workunit.client.0.smithi001.stdout:1/72: write d0/d9/fc [4116400,88037] 0 2024-02-28T16:30:54.372 INFO:tasks.workunit.client.0.smithi001.stdout:1/73: read - d0/d9/fa zero size 2024-02-28T16:30:54.373 INFO:tasks.workunit.client.0.smithi001.stdout:1/74: dread - d0/d9/fa zero size 2024-02-28T16:30:54.380 INFO:tasks.workunit.client.0.smithi001.stdout:1/75: truncate d0/f7 250312 0 2024-02-28T16:30:54.386 INFO:tasks.workunit.client.0.smithi001.stdout:1/76: rename d0/cd to d0/d9/d10/c12 0 2024-02-28T16:30:54.388 INFO:tasks.workunit.client.0.smithi001.stdout:1/77: truncate d0/d9/fa 72281 0 2024-02-28T16:30:54.391 INFO:tasks.workunit.client.0.smithi001.stdout:1/78: unlink d0/d9/fc 0 2024-02-28T16:30:54.394 INFO:tasks.workunit.client.0.smithi001.stdout:1/79: read d0/d9/fa [4917,19896] 0 2024-02-28T16:30:54.405 INFO:tasks.workunit.client.0.smithi001.stdout:1/80: rmdir d0 39 2024-02-28T16:30:54.408 INFO:tasks.workunit.client.0.smithi001.stdout:1/81: chown d0/lb 1 1 2024-02-28T16:30:54.413 INFO:tasks.workunit.client.0.smithi001.stdout:1/82: link d0/d9/ce d0/d9/d10/c13 0 2024-02-28T16:30:54.415 INFO:tasks.workunit.client.0.smithi001.stdout:1/83: symlink d0/d9/d10/l14 0 2024-02-28T16:30:54.417 INFO:tasks.workunit.client.0.smithi001.stdout:1/84: symlink d0/d9/d10/l15 0 2024-02-28T16:30:54.419 INFO:tasks.workunit.client.0.smithi001.stdout:1/85: chown d0/d9/d10/l14 109 1 2024-02-28T16:30:54.421 INFO:tasks.workunit.client.0.smithi001.stdout:1/86: symlink d0/l16 0 2024-02-28T16:30:54.422 INFO:tasks.workunit.client.0.smithi001.stdout:1/87: chown d0/d9/d10/l14 430337 1 2024-02-28T16:30:54.431 INFO:tasks.workunit.client.0.smithi001.stdout:1/88: write d0/d9/ff [4989084,68383] 0 2024-02-28T16:30:54.433 INFO:tasks.workunit.client.0.smithi001.stdout:1/89: write d0/d9/f11 [50869,111238] 0 2024-02-28T16:30:54.434 INFO:tasks.workunit.client.0.smithi001.stdout:1/90: readlink d0/d9/d10/l14 0 2024-02-28T16:30:54.437 INFO:tasks.workunit.client.0.smithi001.stdout:1/91: dread d0/d9/fa [0,4194304] 0 2024-02-28T16:30:54.439 INFO:tasks.workunit.client.0.smithi001.stdout:1/92: creat d0/d9/d10/f17 x:0 0 0 2024-02-28T16:30:54.440 INFO:tasks.workunit.client.0.smithi001.stdout:1/93: dread d0/d9/fa [0,4194304] 0 2024-02-28T16:30:54.442 INFO:tasks.workunit.client.0.smithi001.stdout:1/94: unlink d0/d9/d10/l15 0 2024-02-28T16:30:54.444 INFO:tasks.workunit.client.0.smithi001.stdout:1/95: fdatasync d0/d9/fa 0 2024-02-28T16:30:54.449 INFO:tasks.workunit.client.0.smithi001.stdout:1/96: fsync d0/d9/d10/f17 0 2024-02-28T16:30:54.463 INFO:tasks.workunit.client.0.smithi001.stdout:1/97: dwrite d0/d9/f11 [0,4194304] 0 2024-02-28T16:30:54.465 INFO:tasks.workunit.client.0.smithi001.stdout:1/98: symlink d0/l18 0 2024-02-28T16:30:54.468 INFO:tasks.workunit.client.0.smithi001.stdout:1/99: chown d0/d9/d10/c12 1358191 1 2024-02-28T16:30:54.486 INFO:tasks.workunit.client.0.smithi001.stdout:1/100: dwrite d0/d9/d10/f17 [0,4194304] 0 2024-02-28T16:30:54.488 INFO:tasks.workunit.client.0.smithi001.stdout:1/101: fdatasync d0/d9/ff 0 2024-02-28T16:30:54.508 INFO:tasks.workunit.client.0.smithi001.stdout:1/102: dwrite d0/d9/d10/f17 [0,4194304] 0 2024-02-28T16:30:54.512 INFO:tasks.workunit.client.0.smithi001.stdout:1/103: getdents d0/d9 0 2024-02-28T16:30:54.533 INFO:tasks.workunit.client.0.smithi001.stdout:1/104: dwrite d0/d9/d10/f17 [0,4194304] 0 2024-02-28T16:30:54.534 INFO:tasks.workunit.client.0.smithi001.stdout:1/105: readlink d0/l16 0 2024-02-28T16:30:54.537 INFO:tasks.workunit.client.0.smithi001.stdout:1/106: symlink d0/d9/l19 0 2024-02-28T16:30:54.546 INFO:tasks.workunit.client.0.smithi001.stdout:1/107: dread d0/d9/fa [0,4194304] 0 2024-02-28T16:30:54.548 INFO:tasks.workunit.client.0.smithi001.stdout:1/108: fdatasync d0/d9/f11 0 2024-02-28T16:30:54.549 INFO:tasks.workunit.client.0.smithi001.stdout:1/109: stat d0/l3 0 2024-02-28T16:30:54.561 INFO:tasks.workunit.client.0.smithi001.stdout:1/110: dwrite d0/d9/f11 [0,4194304] 0 2024-02-28T16:30:54.564 INFO:tasks.workunit.client.0.smithi001.stdout:1/111: mkdir d0/d9/d1a 0 2024-02-28T16:30:54.567 INFO:tasks.workunit.client.0.smithi001.stdout:1/112: unlink d0/f7 0 2024-02-28T16:30:54.569 INFO:tasks.workunit.client.0.smithi001.stdout:1/113: chown d0/d9/d10/c12 1005227 1 2024-02-28T16:30:54.571 INFO:tasks.workunit.client.0.smithi001.stdout:1/114: mknod d0/d9/d10/c1b 0 2024-02-28T16:30:54.574 INFO:tasks.workunit.client.0.smithi001.stdout:1/115: unlink d0/d9/d10/l14 0 2024-02-28T16:30:54.580 INFO:tasks.workunit.client.0.smithi001.stdout:1/116: link d0/l18 d0/d9/l1c 0 2024-02-28T16:30:54.589 INFO:tasks.workunit.client.0.smithi001.stdout:1/117: getdents d0/d9 0 2024-02-28T16:30:54.591 INFO:tasks.workunit.client.0.smithi001.stdout:1/118: symlink d0/l1d 0 2024-02-28T16:30:54.593 INFO:tasks.workunit.client.0.smithi001.stdout:1/119: rmdir d0/d9 39 2024-02-28T16:30:54.597 INFO:tasks.workunit.client.0.smithi001.stdout:1/120: truncate d0/d9/fa 217164 0 2024-02-28T16:30:54.609 INFO:tasks.workunit.client.0.smithi001.stdout:1/121: dwrite d0/d9/ff [4194304,4194304] 0 2024-02-28T16:30:55.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:55 smithi001 ceph-mon[28991]: pgmap v41: 65 pgs: 65 active+clean; 262 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 85 KiB/s rd, 8.3 MiB/s wr, 222 op/s 2024-02-28T16:30:55.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:55 smithi066 ceph-mon[36236]: pgmap v41: 65 pgs: 65 active+clean; 262 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 85 KiB/s rd, 8.3 MiB/s wr, 222 op/s 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:4/21: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:5/24: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:6/7: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:7/3: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:0/0: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:3/18: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:3/19: dwrite - no filename 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:9/52: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:2/2: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:8/52: sync 2024-02-28T16:30:56.584 INFO:tasks.workunit.client.0.smithi001.stdout:1/122: sync 2024-02-28T16:30:56.585 INFO:tasks.workunit.client.0.smithi001.stdout:0/1: chown . 128909 1 2024-02-28T16:30:56.585 INFO:tasks.workunit.client.0.smithi001.stdout:0/2: dwrite - no filename 2024-02-28T16:30:56.586 INFO:tasks.workunit.client.0.smithi001.stdout:3/20: chown d4 618522820 1 2024-02-28T16:30:56.587 INFO:tasks.workunit.client.0.smithi001.stdout:6/8: dread - f1 zero size 2024-02-28T16:30:56.591 INFO:tasks.workunit.client.0.smithi001.stdout:5/25: creat f6 x:0 0 0 2024-02-28T16:30:56.591 INFO:tasks.workunit.client.0.smithi001.stdout:9/53: chown d5/l13 246560 1 2024-02-28T16:30:56.592 INFO:tasks.workunit.client.0.smithi001.stdout:5/26: dread - f4 zero size 2024-02-28T16:30:56.593 INFO:tasks.workunit.client.0.smithi001.stdout:7/4: creat f0 x:0 0 0 2024-02-28T16:30:56.596 INFO:tasks.workunit.client.0.smithi001.stdout:7/5: chown f0 15 1 2024-02-28T16:30:56.599 INFO:tasks.workunit.client.0.smithi001.stdout:8/53: creat d4/fe x:0 0 0 2024-02-28T16:30:56.600 INFO:tasks.workunit.client.0.smithi001.stdout:6/9: readlink l4 0 2024-02-28T16:30:56.602 INFO:tasks.workunit.client.0.smithi001.stdout:2/3: mkdir d0 0 2024-02-28T16:30:56.605 INFO:tasks.workunit.client.0.smithi001.stdout:3/21: mkdir d4/d5 0 2024-02-28T16:30:56.605 INFO:tasks.workunit.client.0.smithi001.stdout:3/22: dread - no filename 2024-02-28T16:30:56.606 INFO:tasks.workunit.client.0.smithi001.stdout:0/3: getdents . 0 2024-02-28T16:30:56.606 INFO:tasks.workunit.client.0.smithi001.stdout:0/4: dread - no filename 2024-02-28T16:30:56.606 INFO:tasks.workunit.client.0.smithi001.stdout:0/5: dwrite - no filename 2024-02-28T16:30:56.606 INFO:tasks.workunit.client.0.smithi001.stdout:0/6: dwrite - no filename 2024-02-28T16:30:56.606 INFO:tasks.workunit.client.0.smithi001.stdout:0/7: truncate - no filename 2024-02-28T16:30:56.608 INFO:tasks.workunit.client.0.smithi001.stdout:6/10: creat f5 x:0 0 0 2024-02-28T16:30:56.612 INFO:tasks.workunit.client.0.smithi001.stdout:9/54: link d5/f9 d5/f18 0 2024-02-28T16:30:56.612 INFO:tasks.workunit.client.0.smithi001.stdout:2/4: symlink d0/l1 0 2024-02-28T16:30:56.613 INFO:tasks.workunit.client.0.smithi001.stdout:6/11: fdatasync f1 0 2024-02-28T16:30:56.615 INFO:tasks.workunit.client.0.smithi001.stdout:0/8: creat f0 x:0 0 0 2024-02-28T16:30:56.618 INFO:tasks.workunit.client.0.smithi001.stdout:3/23: rename d4/d5 to d4/d6 0 2024-02-28T16:30:56.618 INFO:tasks.workunit.client.0.smithi001.stdout:2/5: chown d0 670612147 1 2024-02-28T16:30:56.618 INFO:tasks.workunit.client.0.smithi001.stdout:2/6: dread - no filename 2024-02-28T16:30:56.619 INFO:tasks.workunit.client.0.smithi001.stdout:2/7: truncate - no filename 2024-02-28T16:30:56.620 INFO:tasks.workunit.client.0.smithi001.stdout:6/12: chown c2 29563 1 2024-02-28T16:30:56.620 INFO:tasks.workunit.client.0.smithi001.stdout:9/55: unlink d5/lc 0 2024-02-28T16:30:56.621 INFO:tasks.workunit.client.0.smithi001.stdout:0/9: dread - f0 zero size 2024-02-28T16:30:56.623 INFO:tasks.workunit.client.0.smithi001.stdout:3/24: rmdir d4 39 2024-02-28T16:30:56.630 INFO:tasks.workunit.client.0.smithi001.stdout:2/8: chown d0/l1 7 1 2024-02-28T16:30:56.630 INFO:tasks.workunit.client.0.smithi001.stdout:2/9: write - no filename 2024-02-28T16:30:56.630 INFO:tasks.workunit.client.0.smithi001.stdout:1/123: dwrite d0/d9/f11 [4194304,4194304] 0 2024-02-28T16:30:56.633 INFO:tasks.workunit.client.0.smithi001.stdout:9/56: creat d5/f19 x:0 0 0 2024-02-28T16:30:56.634 INFO:tasks.workunit.client.0.smithi001.stdout:4/22: dwrite d3/d5/f6 [0,4194304] 0 2024-02-28T16:30:56.640 INFO:tasks.workunit.client.0.smithi001.stdout:5/27: dwrite f4 [0,4194304] 0 2024-02-28T16:30:56.642 INFO:tasks.workunit.client.0.smithi001.stdout:3/25: mkdir d4/d6/d7 0 2024-02-28T16:30:56.642 INFO:tasks.workunit.client.0.smithi001.stdout:3/26: fdatasync - no filename 2024-02-28T16:30:56.642 INFO:tasks.workunit.client.0.smithi001.stdout:3/27: read - no filename 2024-02-28T16:30:56.642 INFO:tasks.workunit.client.0.smithi001.stdout:3/28: dwrite - no filename 2024-02-28T16:30:56.643 INFO:tasks.workunit.client.0.smithi001.stdout:2/10: mkdir d0/d2 0 2024-02-28T16:30:56.643 INFO:tasks.workunit.client.0.smithi001.stdout:2/11: write - no filename 2024-02-28T16:30:56.643 INFO:tasks.workunit.client.0.smithi001.stdout:2/12: fdatasync - no filename 2024-02-28T16:30:56.643 INFO:tasks.workunit.client.0.smithi001.stdout:2/13: fsync - no filename 2024-02-28T16:30:56.646 INFO:tasks.workunit.client.0.smithi001.stdout:4/23: mkdir d3/d5/d7/db 0 2024-02-28T16:30:56.647 INFO:tasks.workunit.client.0.smithi001.stdout:4/24: rename d3 to d3/d5/d7/dc 22 2024-02-28T16:30:56.650 INFO:tasks.workunit.client.0.smithi001.stdout:4/25: chown d3/d5/f6 65 1 2024-02-28T16:30:56.650 INFO:tasks.workunit.client.0.smithi001.stdout:5/28: write f2 [1679100,23285] 0 2024-02-28T16:30:56.652 INFO:tasks.workunit.client.0.smithi001.stdout:3/29: mknod d4/d6/c8 0 2024-02-28T16:30:56.654 INFO:tasks.workunit.client.0.smithi001.stdout:1/124: mknod d0/d9/d1a/c1e 0 2024-02-28T16:30:56.655 INFO:tasks.workunit.client.0.smithi001.stdout:2/14: mkdir d0/d2/d3 0 2024-02-28T16:30:56.655 INFO:tasks.workunit.client.0.smithi001.stdout:9/57: dread d5/fd [0,4194304] 0 2024-02-28T16:30:56.657 INFO:tasks.workunit.client.0.smithi001.stdout:5/29: creat f7 x:0 0 0 2024-02-28T16:30:56.660 INFO:tasks.workunit.client.0.smithi001.stdout:3/30: mknod d4/d6/c9 0 2024-02-28T16:30:56.660 INFO:tasks.workunit.client.0.smithi001.stdout:3/31: readlink - no filename 2024-02-28T16:30:56.660 INFO:tasks.workunit.client.0.smithi001.stdout:3/32: read - no filename 2024-02-28T16:30:56.662 INFO:tasks.workunit.client.0.smithi001.stdout:1/125: mkdir d0/d9/d1a/d1f 0 2024-02-28T16:30:56.664 INFO:tasks.workunit.client.0.smithi001.stdout:2/15: creat d0/f4 x:0 0 0 2024-02-28T16:30:56.667 INFO:tasks.workunit.client.0.smithi001.stdout:9/58: write d5/fd [161590,44928] 0 2024-02-28T16:30:56.667 INFO:tasks.workunit.client.0.smithi001.stdout:4/26: dread d3/d5/f6 [0,4194304] 0 2024-02-28T16:30:56.669 INFO:tasks.workunit.client.0.smithi001.stdout:4/27: stat d3/fa 0 2024-02-28T16:30:56.672 INFO:tasks.workunit.client.0.smithi001.stdout:4/28: write d3/fa [814416,91457] 0 2024-02-28T16:30:56.673 INFO:tasks.workunit.client.0.smithi001.stdout:5/30: link f6 f8 0 2024-02-28T16:30:56.674 INFO:tasks.workunit.client.0.smithi001.stdout:2/16: rename d0/f4 to d0/d2/d3/f5 0 2024-02-28T16:30:56.677 INFO:tasks.workunit.client.0.smithi001.stdout:2/17: dread - d0/d2/d3/f5 zero size 2024-02-28T16:30:56.681 INFO:tasks.workunit.client.0.smithi001.stdout:5/31: symlink l9 0 2024-02-28T16:30:56.683 INFO:tasks.workunit.client.0.smithi001.stdout:9/59: truncate d5/fb 3981578 0 2024-02-28T16:30:56.686 INFO:tasks.workunit.client.0.smithi001.stdout:2/18: mkdir d0/d2/d6 0 2024-02-28T16:30:56.688 INFO:tasks.workunit.client.0.smithi001.stdout:5/32: rename f6 to fa 0 2024-02-28T16:30:56.688 INFO:tasks.workunit.client.0.smithi001.stdout:5/33: rmdir - no directory 2024-02-28T16:30:56.690 INFO:tasks.workunit.client.0.smithi001.stdout:5/34: write f5 [938835,120332] 0 2024-02-28T16:30:56.691 INFO:tasks.workunit.client.0.smithi001.stdout:5/35: dread - f3 zero size 2024-02-28T16:30:56.693 INFO:tasks.workunit.client.0.smithi001.stdout:4/29: rmdir d3/d5/d7/db 0 2024-02-28T16:30:56.721 INFO:tasks.workunit.client.0.smithi001.stdout:5/36: dwrite f8 [0,4194304] 0 2024-02-28T16:30:56.722 INFO:tasks.workunit.client.0.smithi001.stdout:4/30: dwrite d3/d5/f6 [0,4194304] 0 2024-02-28T16:30:56.724 INFO:tasks.workunit.client.0.smithi001.stdout:5/37: write f2 [1037927,66949] 0 2024-02-28T16:30:56.741 INFO:tasks.workunit.client.0.smithi001.stdout:4/31: dread d3/fa [0,4194304] 0 2024-02-28T16:30:56.741 INFO:tasks.workunit.client.0.smithi001.stdout:5/38: dread f8 [0,4194304] 0 2024-02-28T16:30:56.742 INFO:tasks.workunit.client.0.smithi001.stdout:5/39: stat f8 0 2024-02-28T16:30:56.744 INFO:tasks.workunit.client.0.smithi001.stdout:4/32: chown d3/fa 176 1 2024-02-28T16:30:56.756 INFO:tasks.workunit.client.0.smithi001.stdout:4/33: rename d3/d5/d7/d9 to d3/d5/dd 0 2024-02-28T16:30:56.760 INFO:tasks.workunit.client.0.smithi001.stdout:4/34: mknod d3/ce 0 2024-02-28T16:30:56.761 INFO:tasks.workunit.client.0.smithi001.stdout:4/35: write f2 [944267,123201] 0 2024-02-28T16:30:56.764 INFO:tasks.workunit.client.0.smithi001.stdout:7/6: getdents . 0 2024-02-28T16:30:56.767 INFO:tasks.workunit.client.0.smithi001.stdout:4/36: creat d3/d5/dd/ff x:0 0 0 2024-02-28T16:30:56.769 INFO:tasks.workunit.client.0.smithi001.stdout:7/7: creat f1 x:0 0 0 2024-02-28T16:30:56.770 INFO:tasks.workunit.client.0.smithi001.stdout:4/37: symlink d3/d5/d7/l10 0 2024-02-28T16:30:56.771 INFO:tasks.workunit.client.0.smithi001.stdout:5/40: fsync f4 0 2024-02-28T16:30:56.772 INFO:tasks.workunit.client.0.smithi001.stdout:5/41: write f2 [1118057,62632] 0 2024-02-28T16:30:56.773 INFO:tasks.workunit.client.0.smithi001.stdout:7/8: unlink f0 0 2024-02-28T16:30:56.773 INFO:tasks.workunit.client.0.smithi001.stdout:9/60: fsync d5/f9 0 2024-02-28T16:30:56.774 INFO:tasks.workunit.client.0.smithi001.stdout:9/61: fdatasync d5/f15 0 2024-02-28T16:30:56.777 INFO:tasks.workunit.client.0.smithi001.stdout:5/42: write f4 [4489822,47172] 0 2024-02-28T16:30:56.779 INFO:tasks.workunit.client.0.smithi001.stdout:9/62: creat d5/f1a x:0 0 0 2024-02-28T16:30:56.780 INFO:tasks.workunit.client.0.smithi001.stdout:5/43: unlink f8 0 2024-02-28T16:30:56.783 INFO:tasks.workunit.client.0.smithi001.stdout:9/63: readlink l3 0 2024-02-28T16:30:56.784 INFO:tasks.workunit.client.0.smithi001.stdout:5/44: symlink lb 0 2024-02-28T16:30:56.787 INFO:tasks.workunit.client.0.smithi001.stdout:9/64: mknod d5/c1b 0 2024-02-28T16:30:56.789 INFO:tasks.workunit.client.0.smithi001.stdout:9/65: symlink d5/l1c 0 2024-02-28T16:30:56.789 INFO:tasks.workunit.client.0.smithi001.stdout:1/126: rmdir d0 39 2024-02-28T16:30:56.790 INFO:tasks.workunit.client.0.smithi001.stdout:9/66: readlink d5/lf 0 2024-02-28T16:30:56.792 INFO:tasks.workunit.client.0.smithi001.stdout:9/67: dread - d5/f17 zero size 2024-02-28T16:30:56.793 INFO:tasks.workunit.client.0.smithi001.stdout:9/68: fsync d5/f6 0 2024-02-28T16:30:56.798 INFO:tasks.workunit.client.0.smithi001.stdout:2/19: getdents d0/d2 0 2024-02-28T16:30:56.799 INFO:tasks.workunit.client.0.smithi001.stdout:9/69: write d5/fd [3850888,26337] 0 2024-02-28T16:30:56.804 INFO:tasks.workunit.client.0.smithi001.stdout:2/20: symlink d0/l7 0 2024-02-28T16:30:56.806 INFO:tasks.workunit.client.0.smithi001.stdout:5/45: dwrite f7 [0,4194304] 0 2024-02-28T16:30:56.807 INFO:tasks.workunit.client.0.smithi001.stdout:9/70: creat d5/f1d x:0 0 0 2024-02-28T16:30:56.808 INFO:tasks.workunit.client.0.smithi001.stdout:5/46: readlink lb 0 2024-02-28T16:30:56.808 INFO:tasks.workunit.client.0.smithi001.stdout:2/21: mkdir d0/d8 0 2024-02-28T16:30:56.811 INFO:tasks.workunit.client.0.smithi001.stdout:9/71: read d5/fb [1105127,3828] 0 2024-02-28T16:30:56.811 INFO:tasks.workunit.client.0.smithi001.stdout:2/22: read - d0/d2/d3/f5 zero size 2024-02-28T16:30:56.813 INFO:tasks.workunit.client.0.smithi001.stdout:9/72: fdatasync d5/fd 0 2024-02-28T16:30:56.815 INFO:tasks.workunit.client.0.smithi001.stdout:4/38: getdents d3 0 2024-02-28T16:30:56.816 INFO:tasks.workunit.client.0.smithi001.stdout:9/73: chown d5/fb 80 1 2024-02-28T16:30:56.824 INFO:tasks.workunit.client.0.smithi001.stdout:5/47: dread f4 [0,4194304] 0 2024-02-28T16:30:56.825 INFO:tasks.workunit.client.0.smithi001.stdout:5/48: readlink lb 0 2024-02-28T16:30:56.826 INFO:tasks.workunit.client.0.smithi001.stdout:1/127: truncate d0/d9/d10/f17 2849118 0 2024-02-28T16:30:56.831 INFO:tasks.workunit.client.0.smithi001.stdout:1/128: symlink d0/d9/d1a/d1f/l20 0 2024-02-28T16:30:56.839 INFO:tasks.workunit.client.0.smithi001.stdout:1/129: creat d0/f21 x:0 0 0 2024-02-28T16:30:56.847 INFO:tasks.workunit.client.0.smithi001.stdout:9/74: dwrite d5/f1d [0,4194304] 0 2024-02-28T16:30:56.848 INFO:tasks.workunit.client.0.smithi001.stdout:2/23: dwrite d0/d2/d3/f5 [0,4194304] 0 2024-02-28T16:30:56.850 INFO:tasks.workunit.client.0.smithi001.stdout:2/24: readlink d0/l7 0 2024-02-28T16:30:56.857 INFO:tasks.workunit.client.0.smithi001.stdout:1/130: link d0/d9/d10/c1b d0/d9/d10/c22 0 2024-02-28T16:30:56.859 INFO:tasks.workunit.client.0.smithi001.stdout:2/25: link d0/l7 d0/l9 0 2024-02-28T16:30:56.861 INFO:tasks.workunit.client.0.smithi001.stdout:1/131: mknod d0/d9/d1a/c23 0 2024-02-28T16:30:56.861 INFO:tasks.workunit.client.0.smithi001.stdout:9/75: dread d5/fb [0,4194304] 0 2024-02-28T16:30:56.864 INFO:tasks.workunit.client.0.smithi001.stdout:9/76: dread - d5/f18 zero size 2024-02-28T16:30:56.864 INFO:tasks.workunit.client.0.smithi001.stdout:2/26: creat d0/d8/fa x:0 0 0 2024-02-28T16:30:56.865 INFO:tasks.workunit.client.0.smithi001.stdout:2/27: dread - d0/d8/fa zero size 2024-02-28T16:30:56.867 INFO:tasks.workunit.client.0.smithi001.stdout:9/77: creat d5/f1e x:0 0 0 2024-02-28T16:30:56.870 INFO:tasks.workunit.client.0.smithi001.stdout:2/28: creat d0/d2/fb x:0 0 0 2024-02-28T16:30:56.870 INFO:tasks.workunit.client.0.smithi001.stdout:9/78: creat d5/f1f x:0 0 0 2024-02-28T16:30:56.874 INFO:tasks.workunit.client.0.smithi001.stdout:2/29: mkdir d0/d8/dc 0 2024-02-28T16:30:56.875 INFO:tasks.workunit.client.0.smithi001.stdout:9/79: mknod d5/c20 0 2024-02-28T16:30:56.878 INFO:tasks.workunit.client.0.smithi001.stdout:2/30: creat d0/fd x:0 0 0 2024-02-28T16:30:56.880 INFO:tasks.workunit.client.0.smithi001.stdout:2/31: fdatasync d0/d2/fb 0 2024-02-28T16:30:56.887 INFO:tasks.workunit.client.0.smithi001.stdout:1/132: dwrite d0/d9/f11 [8388608,4194304] 0 2024-02-28T16:30:56.904 INFO:tasks.workunit.client.0.smithi001.stdout:2/32: truncate d0/d2/d3/f5 4118775 0 2024-02-28T16:30:56.907 INFO:tasks.workunit.client.0.smithi001.stdout:9/80: dwrite d5/f17 [0,4194304] 0 2024-02-28T16:30:56.909 INFO:tasks.workunit.client.0.smithi001.stdout:9/81: stat d5/f1a 0 2024-02-28T16:30:56.922 INFO:tasks.workunit.client.0.smithi001.stdout:9/82: getdents d5 0 2024-02-28T16:30:56.927 INFO:tasks.workunit.client.0.smithi001.stdout:1/133: dwrite d0/f21 [0,4194304] 0 2024-02-28T16:30:56.927 INFO:tasks.workunit.client.0.smithi001.stdout:9/83: read - d5/f18 zero size 2024-02-28T16:30:56.938 INFO:tasks.workunit.client.0.smithi001.stdout:9/84: link d5/f1d d5/f21 0 2024-02-28T16:30:56.939 INFO:tasks.workunit.client.0.smithi001.stdout:2/33: dwrite d0/d8/fa [0,4194304] 0 2024-02-28T16:30:56.942 INFO:tasks.workunit.client.0.smithi001.stdout:1/134: truncate d0/f21 270592 0 2024-02-28T16:30:56.942 INFO:tasks.workunit.client.0.smithi001.stdout:2/34: truncate d0/fd 1000389 0 2024-02-28T16:30:56.942 INFO:tasks.workunit.client.0.smithi001.stdout:9/85: rmdir d5 39 2024-02-28T16:30:56.951 INFO:tasks.workunit.client.0.smithi001.stdout:1/135: creat d0/d9/d10/f24 x:0 0 0 2024-02-28T16:30:56.951 INFO:tasks.workunit.client.0.smithi001.stdout:2/35: rename d0/d8 to d0/d2/d3/de 0 2024-02-28T16:30:56.955 INFO:tasks.workunit.client.0.smithi001.stdout:9/86: rename l3 to d5/l22 0 2024-02-28T16:30:56.956 INFO:tasks.workunit.client.0.smithi001.stdout:9/87: dread - d5/f9 zero size 2024-02-28T16:30:56.956 INFO:tasks.workunit.client.0.smithi001.stdout:2/36: creat d0/ff x:0 0 0 2024-02-28T16:30:56.958 INFO:tasks.workunit.client.0.smithi001.stdout:2/37: truncate d0/ff 772086 0 2024-02-28T16:30:56.959 INFO:tasks.workunit.client.0.smithi001.stdout:1/136: symlink d0/l25 0 2024-02-28T16:30:56.961 INFO:tasks.workunit.client.0.smithi001.stdout:1/137: read d0/d9/f11 [12081873,87867] 0 2024-02-28T16:30:56.962 INFO:tasks.workunit.client.0.smithi001.stdout:2/38: chown d0/l9 5621101 1 2024-02-28T16:30:56.964 INFO:tasks.workunit.client.0.smithi001.stdout:1/138: rmdir d0/d9/d1a/d1f 39 2024-02-28T16:30:56.968 INFO:tasks.workunit.client.0.smithi001.stdout:1/139: read d0/d9/fa [203315,13765] 0 2024-02-28T16:30:56.969 INFO:tasks.workunit.client.0.smithi001.stdout:2/39: creat d0/d2/d3/de/dc/f10 x:0 0 0 2024-02-28T16:30:56.970 INFO:tasks.workunit.client.0.smithi001.stdout:9/88: dread d5/fb [0,4194304] 0 2024-02-28T16:30:56.970 INFO:tasks.workunit.client.0.smithi001.stdout:1/140: rmdir d0/d9/d1a 39 2024-02-28T16:30:56.973 INFO:tasks.workunit.client.0.smithi001.stdout:2/40: read d0/d2/d3/f5 [1055481,80712] 0 2024-02-28T16:30:56.973 INFO:tasks.workunit.client.0.smithi001.stdout:1/141: stat d0/d9/d1a 0 2024-02-28T16:30:56.977 INFO:tasks.workunit.client.0.smithi001.stdout:9/89: getdents d5 0 2024-02-28T16:30:56.977 INFO:tasks.workunit.client.0.smithi001.stdout:2/41: creat d0/d2/f11 x:0 0 0 2024-02-28T16:30:56.977 INFO:tasks.workunit.client.0.smithi001.stdout:1/142: mkdir d0/d9/d10/d26 0 2024-02-28T16:30:56.978 INFO:tasks.workunit.client.0.smithi001.stdout:9/90: fdatasync d5/f15 0 2024-02-28T16:30:56.980 INFO:tasks.workunit.client.0.smithi001.stdout:2/42: mkdir d0/d2/d6/d12 0 2024-02-28T16:30:56.983 INFO:tasks.workunit.client.0.smithi001.stdout:9/91: unlink d5/l22 0 2024-02-28T16:30:56.985 INFO:tasks.workunit.client.0.smithi001.stdout:2/43: symlink d0/d2/d3/de/dc/l13 0 2024-02-28T16:30:56.985 INFO:tasks.workunit.client.0.smithi001.stdout:9/92: chown d5/f1d 2931784 1 2024-02-28T16:30:56.987 INFO:tasks.workunit.client.0.smithi001.stdout:2/44: read - d0/d2/fb zero size 2024-02-28T16:30:56.987 INFO:tasks.workunit.client.0.smithi001.stdout:1/143: getdents d0/d9/d10 0 2024-02-28T16:30:56.987 INFO:tasks.workunit.client.0.smithi001.stdout:9/93: chown d5/fd 1293 1 2024-02-28T16:30:56.988 INFO:tasks.workunit.client.0.smithi001.stdout:1/144: dread - d0/d9/d10/f24 zero size 2024-02-28T16:30:56.989 INFO:tasks.workunit.client.0.smithi001.stdout:9/94: chown d5/f15 162252 1 2024-02-28T16:30:56.990 INFO:tasks.workunit.client.0.smithi001.stdout:2/45: creat d0/d2/f14 x:0 0 0 2024-02-28T16:30:56.995 INFO:tasks.workunit.client.0.smithi001.stdout:2/46: dread d0/ff [0,4194304] 0 2024-02-28T16:30:56.995 INFO:tasks.workunit.client.0.smithi001.stdout:1/145: mkdir d0/d9/d10/d26/d27 0 2024-02-28T16:30:56.995 INFO:tasks.workunit.client.0.smithi001.stdout:2/47: readlink d0/l9 0 2024-02-28T16:30:56.999 INFO:tasks.workunit.client.0.smithi001.stdout:9/95: mknod d5/c23 0 2024-02-28T16:30:57.001 INFO:tasks.workunit.client.0.smithi001.stdout:1/146: creat d0/f28 x:0 0 0 2024-02-28T16:30:57.003 INFO:tasks.workunit.client.0.smithi001.stdout:2/48: creat d0/d2/d6/f15 x:0 0 0 2024-02-28T16:30:57.004 INFO:tasks.workunit.client.0.smithi001.stdout:2/49: chown d0/d2/d6/d12 2929414 1 2024-02-28T16:30:57.005 INFO:tasks.workunit.client.0.smithi001.stdout:2/50: dread - d0/d2/f14 zero size 2024-02-28T16:30:57.006 INFO:tasks.workunit.client.0.smithi001.stdout:2/51: read d0/d2/d3/f5 [3716266,112916] 0 2024-02-28T16:30:57.008 INFO:tasks.workunit.client.0.smithi001.stdout:1/147: creat d0/d9/d1a/d1f/f29 x:0 0 0 2024-02-28T16:30:57.009 INFO:tasks.workunit.client.0.smithi001.stdout:1/148: chown d0/d9/d10 11994 1 2024-02-28T16:30:57.010 INFO:tasks.workunit.client.0.smithi001.stdout:9/96: creat d5/f24 x:0 0 0 2024-02-28T16:30:57.010 INFO:tasks.workunit.client.0.smithi001.stdout:2/52: mknod d0/d2/c16 0 2024-02-28T16:30:57.011 INFO:tasks.workunit.client.0.smithi001.stdout:1/149: chown d0/d9/d10/c12 1259169 1 2024-02-28T16:30:57.012 INFO:tasks.workunit.client.0.smithi001.stdout:2/53: creat d0/f17 x:0 0 0 2024-02-28T16:30:57.018 INFO:tasks.workunit.client.0.smithi001.stdout:2/54: creat d0/d2/d3/de/f18 x:0 0 0 2024-02-28T16:30:57.020 INFO:tasks.workunit.client.0.smithi001.stdout:2/55: dread - d0/d2/f11 zero size 2024-02-28T16:30:57.021 INFO:tasks.workunit.client.0.smithi001.stdout:1/150: getdents d0/d9/d1a/d1f 0 2024-02-28T16:30:57.022 INFO:tasks.workunit.client.0.smithi001.stdout:2/56: dread - d0/f17 zero size 2024-02-28T16:30:57.025 INFO:tasks.workunit.client.0.smithi001.stdout:2/57: dread - d0/f17 zero size 2024-02-28T16:30:57.031 INFO:tasks.workunit.client.0.smithi001.stdout:9/97: link d5/l1c d5/l25 0 2024-02-28T16:30:57.032 INFO:tasks.workunit.client.0.smithi001.stdout:1/151: link d0/d9/l19 d0/d9/d10/d26/l2a 0 2024-02-28T16:30:57.034 INFO:tasks.workunit.client.0.smithi001.stdout:1/152: chown d0/d9/fa 2 1 2024-02-28T16:30:57.034 INFO:tasks.workunit.client.0.smithi001.stdout:9/98: symlink d5/l26 0 2024-02-28T16:30:57.035 INFO:tasks.workunit.client.0.smithi001.stdout:1/153: fsync d0/f28 0 2024-02-28T16:30:57.043 INFO:tasks.workunit.client.0.smithi001.stdout:2/58: dwrite d0/d2/d3/de/f18 [0,4194304] 0 2024-02-28T16:30:57.046 INFO:tasks.workunit.client.0.smithi001.stdout:1/154: rename d0/d9/d10 to d0/d9/d1a/d2b 0 2024-02-28T16:30:57.047 INFO:tasks.workunit.client.0.smithi001.stdout:2/59: symlink d0/d2/d6/l19 0 2024-02-28T16:30:57.052 INFO:tasks.workunit.client.0.smithi001.stdout:2/60: truncate d0/d2/fb 958594 0 2024-02-28T16:30:57.052 INFO:tasks.workunit.client.0.smithi001.stdout:1/155: creat d0/f2c x:0 0 0 2024-02-28T16:30:57.056 INFO:tasks.workunit.client.0.smithi001.stdout:1/156: write d0/d9/ff [6812714,21976] 0 2024-02-28T16:30:57.057 INFO:tasks.workunit.client.0.smithi001.stdout:2/61: symlink d0/d2/l1a 0 2024-02-28T16:30:57.060 INFO:tasks.workunit.client.0.smithi001.stdout:1/157: chown d0/d9/d1a/d1f 2175 1 2024-02-28T16:30:57.063 INFO:tasks.workunit.client.0.smithi001.stdout:9/99: dwrite d5/f9 [0,4194304] 0 2024-02-28T16:30:57.063 INFO:tasks.workunit.client.0.smithi001.stdout:2/62: write d0/d2/d3/de/fa [3788598,128633] 0 2024-02-28T16:30:57.069 INFO:tasks.workunit.client.0.smithi001.stdout:9/100: mkdir d5/d27 0 2024-02-28T16:30:57.071 INFO:tasks.workunit.client.0.smithi001.stdout:1/158: link d0/d9/d1a/d2b/c12 d0/d9/d1a/d2b/c2d 0 2024-02-28T16:30:57.093 INFO:tasks.workunit.client.0.smithi001.stdout:2/63: dwrite d0/fd [0,4194304] 0 2024-02-28T16:30:57.096 INFO:tasks.workunit.client.0.smithi001.stdout:1/159: dread d0/d9/ff [4194304,4194304] 0 2024-02-28T16:30:57.099 INFO:tasks.workunit.client.0.smithi001.stdout:9/101: dwrite d5/f15 [0,4194304] 0 2024-02-28T16:30:57.104 INFO:tasks.workunit.client.0.smithi001.stdout:1/160: rename d0/d9/f11 to d0/d9/d1a/d1f/f2e 0 2024-02-28T16:30:57.106 INFO:tasks.workunit.client.0.smithi001.stdout:2/64: link d0/d2/f14 d0/d2/d3/de/dc/f1b 0 2024-02-28T16:30:57.107 INFO:tasks.workunit.client.0.smithi001.stdout:9/102: symlink d5/l28 0 2024-02-28T16:30:57.108 INFO:tasks.workunit.client.0.smithi001.stdout:1/161: stat d0/d9/ce 0 2024-02-28T16:30:57.110 INFO:tasks.workunit.client.0.smithi001.stdout:1/162: chown d0/l4 3634453 1 2024-02-28T16:30:57.110 INFO:tasks.workunit.client.0.smithi001.stdout:9/103: chown d5/l25 5 1 2024-02-28T16:30:57.116 INFO:tasks.workunit.client.0.smithi001.stdout:9/104: chown d5/l1c 45464 1 2024-02-28T16:30:57.117 INFO:tasks.workunit.client.0.smithi001.stdout:9/105: fdatasync d5/f9 0 2024-02-28T16:30:57.119 INFO:tasks.workunit.client.0.smithi001.stdout:9/106: stat d5/f9 0 2024-02-28T16:30:57.123 INFO:tasks.workunit.client.0.smithi001.stdout:2/65: truncate d0/d2/d3/de/fa 2981111 0 2024-02-28T16:30:57.138 INFO:tasks.workunit.client.0.smithi001.stdout:1/163: dwrite d0/d9/fa [0,4194304] 0 2024-02-28T16:30:57.144 INFO:tasks.workunit.client.0.smithi001.stdout:1/164: write d0/d9/ff [5697029,67521] 0 2024-02-28T16:30:57.150 INFO:tasks.workunit.client.0.smithi001.stdout:1/165: mkdir d0/d9/d1a/d1f/d2f 0 2024-02-28T16:30:57.151 INFO:tasks.workunit.client.0.smithi001.stdout:2/66: dwrite d0/ff [0,4194304] 0 2024-02-28T16:30:57.153 INFO:tasks.workunit.client.0.smithi001.stdout:1/166: read d0/d9/d1a/d2b/f17 [472830,608] 0 2024-02-28T16:30:57.157 INFO:tasks.workunit.client.0.smithi001.stdout:1/167: chown d0/l18 453207067 1 2024-02-28T16:30:57.160 INFO:tasks.workunit.client.0.smithi001.stdout:2/67: link d0/d2/l1a d0/d2/d3/de/l1c 0 2024-02-28T16:30:57.161 INFO:tasks.workunit.client.0.smithi001.stdout:1/168: fdatasync d0/d9/d1a/d1f/f2e 0 2024-02-28T16:30:57.161 INFO:tasks.workunit.client.0.smithi001.stdout:2/68: stat d0/d2/f11 0 2024-02-28T16:30:57.164 INFO:tasks.workunit.client.0.smithi001.stdout:2/69: creat d0/f1d x:0 0 0 2024-02-28T16:30:57.164 INFO:tasks.workunit.client.0.smithi001.stdout:1/169: mkdir d0/d9/d1a/d2b/d26/d27/d30 0 2024-02-28T16:30:57.170 INFO:tasks.workunit.client.0.smithi001.stdout:8/54: sync 2024-02-28T16:30:57.209 INFO:tasks.workunit.client.0.smithi001.stdout:8/55: mknod d4/d7/cf 0 2024-02-28T16:30:57.211 INFO:tasks.workunit.client.0.smithi001.stdout:1/170: symlink d0/d9/d1a/d2b/d26/d27/d30/l31 0 2024-02-28T16:30:57.216 INFO:tasks.workunit.client.0.smithi001.stdout:1/171: fdatasync d0/f2c 0 2024-02-28T16:30:57.216 INFO:tasks.workunit.client.0.smithi001.stdout:1/172: fsync d0/d9/ff 0 2024-02-28T16:30:57.234 INFO:tasks.workunit.client.0.smithi001.stdout:8/56: dwrite f3 [0,4194304] 0 2024-02-28T16:30:57.237 INFO:tasks.workunit.client.0.smithi001.stdout:1/173: dwrite d0/f28 [0,4194304] 0 2024-02-28T16:30:57.241 INFO:tasks.workunit.client.0.smithi001.stdout:8/57: chown d4/d7/lb 2084459 1 2024-02-28T16:30:57.243 INFO:tasks.workunit.client.0.smithi001.stdout:8/58: truncate f3 4652275 0 2024-02-28T16:30:57.247 INFO:tasks.workunit.client.0.smithi001.stdout:8/59: dread - d4/d7/fd zero size 2024-02-28T16:30:57.253 INFO:tasks.workunit.client.0.smithi001.stdout:8/60: creat d4/d7/f10 x:0 0 0 2024-02-28T16:30:57.259 INFO:tasks.workunit.client.0.smithi001.stdout:1/174: dread d0/d9/fa [0,4194304] 0 2024-02-28T16:30:57.264 INFO:tasks.workunit.client.0.smithi001.stdout:1/175: unlink d0/d9/d1a/d2b/c1b 0 2024-02-28T16:30:57.266 INFO:tasks.workunit.client.0.smithi001.stdout:1/176: write d0/f2c [85816,46288] 0 2024-02-28T16:30:57.271 INFO:tasks.workunit.client.0.smithi001.stdout:1/177: symlink d0/d9/d1a/l32 0 2024-02-28T16:30:57.276 INFO:tasks.workunit.client.0.smithi001.stdout:8/61: dwrite f1 [0,4194304] 0 2024-02-28T16:30:57.276 INFO:tasks.workunit.client.0.smithi001.stdout:1/178: creat d0/d9/d1a/f33 x:0 0 0 2024-02-28T16:30:57.279 INFO:tasks.workunit.client.0.smithi001.stdout:1/179: chown d0/d9/d1a/d1f/l20 577843 1 2024-02-28T16:30:57.294 INFO:tasks.workunit.client.0.smithi001.stdout:1/180: chown d0/d9/d1a/d2b/d26/l2a 8677 1 2024-02-28T16:30:57.298 INFO:tasks.workunit.client.0.smithi001.stdout:8/62: fsync f1 0 2024-02-28T16:30:57.299 INFO:tasks.workunit.client.0.smithi001.stdout:1/181: creat d0/d9/d1a/d2b/d26/f34 x:0 0 0 2024-02-28T16:30:57.300 INFO:tasks.workunit.client.0.smithi001.stdout:8/63: fsync d4/d7/f10 0 2024-02-28T16:30:57.301 INFO:tasks.workunit.client.0.smithi001.stdout:8/64: creat d4/f11 x:0 0 0 2024-02-28T16:30:57.302 INFO:tasks.workunit.client.0.smithi001.stdout:1/182: mknod d0/d9/d1a/d2b/c35 0 2024-02-28T16:30:57.305 INFO:tasks.workunit.client.0.smithi001.stdout:8/65: rename d4/fe to d4/f12 0 2024-02-28T16:30:57.306 INFO:tasks.workunit.client.0.smithi001.stdout:8/66: chown d4/d7/f10 248 1 2024-02-28T16:30:57.307 INFO:tasks.workunit.client.0.smithi001.stdout:1/183: creat d0/f36 x:0 0 0 2024-02-28T16:30:57.315 INFO:tasks.workunit.client.0.smithi001.stdout:1/184: creat d0/d9/d1a/d2b/d26/f37 x:0 0 0 2024-02-28T16:30:57.325 INFO:tasks.workunit.client.0.smithi001.stdout:1/185: truncate d0/f2c 115826 0 2024-02-28T16:30:57.326 INFO:tasks.workunit.client.0.smithi001.stdout:8/67: dwrite d4/d7/f10 [0,4194304] 0 2024-02-28T16:30:57.330 INFO:tasks.workunit.client.0.smithi001.stdout:1/186: symlink d0/d9/d1a/d2b/d26/l38 0 2024-02-28T16:30:57.330 INFO:tasks.workunit.client.0.smithi001.stdout:8/68: write d4/f12 [947741,41162] 0 2024-02-28T16:30:57.349 INFO:tasks.workunit.client.0.smithi001.stdout:0/10: sync 2024-02-28T16:30:57.349 INFO:tasks.workunit.client.0.smithi001.stdout:6/13: sync 2024-02-28T16:30:57.349 INFO:tasks.workunit.client.0.smithi001.stdout:7/9: sync 2024-02-28T16:30:57.349 INFO:tasks.workunit.client.0.smithi001.stdout:3/33: sync 2024-02-28T16:30:57.349 INFO:tasks.workunit.client.0.smithi001.stdout:3/34: write - no filename 2024-02-28T16:30:57.349 INFO:tasks.workunit.client.0.smithi001.stdout:5/49: sync 2024-02-28T16:30:57.349 INFO:tasks.workunit.client.0.smithi001.stdout:3/35: write - no filename 2024-02-28T16:30:57.349 INFO:tasks.workunit.client.0.smithi001.stdout:4/39: sync 2024-02-28T16:30:57.351 INFO:tasks.workunit.client.0.smithi001.stdout:8/69: dwrite d4/f12 [0,4194304] 0 2024-02-28T16:30:57.352 INFO:tasks.workunit.client.0.smithi001.stdout:7/10: truncate f1 527906 0 2024-02-28T16:30:57.352 INFO:tasks.workunit.client.0.smithi001.stdout:4/40: dread - d3/d5/dd/ff zero size 2024-02-28T16:30:57.353 INFO:tasks.workunit.client.0.smithi001.stdout:0/11: write f0 [929941,124137] 0 2024-02-28T16:30:57.354 INFO:tasks.workunit.client.0.smithi001.stdout:8/70: chown f3 156863410 1 2024-02-28T16:30:57.360 INFO:tasks.workunit.client.0.smithi001.stdout:7/11: write f1 [288476,31168] 0 2024-02-28T16:30:57.360 INFO:tasks.workunit.client.0.smithi001.stdout:0/12: chown f0 54762 1 2024-02-28T16:30:57.362 INFO:tasks.workunit.client.0.smithi001.stdout:7/12: readlink - no filename 2024-02-28T16:30:57.362 INFO:tasks.workunit.client.0.smithi001.stdout:3/36: getdents d4/d6/d7 0 2024-02-28T16:30:57.362 INFO:tasks.workunit.client.0.smithi001.stdout:3/37: readlink - no filename 2024-02-28T16:30:57.362 INFO:tasks.workunit.client.0.smithi001.stdout:3/38: fdatasync - no filename 2024-02-28T16:30:57.362 INFO:tasks.workunit.client.0.smithi001.stdout:3/39: fdatasync - no filename 2024-02-28T16:30:57.362 INFO:tasks.workunit.client.0.smithi001.stdout:3/40: dread - no filename 2024-02-28T16:30:57.363 INFO:tasks.workunit.client.0.smithi001.stdout:0/13: write f0 [1413651,114323] 0 2024-02-28T16:30:57.364 INFO:tasks.workunit.client.0.smithi001.stdout:3/41: stat d4 0 2024-02-28T16:30:57.364 INFO:tasks.workunit.client.0.smithi001.stdout:3/42: dread - no filename 2024-02-28T16:30:57.365 INFO:tasks.workunit.client.0.smithi001.stdout:4/41: rename d3/d5/l8 to d3/l11 0 2024-02-28T16:30:57.368 INFO:tasks.workunit.client.0.smithi001.stdout:0/14: mknod c1 0 2024-02-28T16:30:57.370 INFO:tasks.workunit.client.0.smithi001.stdout:7/13: fsync f1 0 2024-02-28T16:30:57.372 INFO:tasks.workunit.client.0.smithi001.stdout:8/71: link d4/d7/fd d4/f13 0 2024-02-28T16:30:57.374 INFO:tasks.workunit.client.0.smithi001.stdout:0/15: symlink l2 0 2024-02-28T16:30:57.376 INFO:tasks.workunit.client.0.smithi001.stdout:3/43: rename d4/d6/c9 to d4/d6/ca 0 2024-02-28T16:30:57.376 INFO:tasks.workunit.client.0.smithi001.stdout:3/44: write - no filename 2024-02-28T16:30:57.377 INFO:tasks.workunit.client.0.smithi001.stdout:7/14: unlink f1 0 2024-02-28T16:30:57.378 INFO:tasks.workunit.client.0.smithi001.stdout:8/72: mknod d4/d7/c14 0 2024-02-28T16:30:57.381 INFO:tasks.workunit.client.0.smithi001.stdout:0/16: creat f3 x:0 0 0 2024-02-28T16:30:57.381 INFO:tasks.workunit.client.0.smithi001.stdout:8/73: fdatasync f1 0 2024-02-28T16:30:57.382 INFO:tasks.workunit.client.0.smithi001.stdout:3/45: creat d4/fb x:0 0 0 2024-02-28T16:30:57.383 INFO:tasks.workunit.client.0.smithi001.stdout:7/15: symlink l2 0 2024-02-28T16:30:57.388 INFO:tasks.workunit.client.0.smithi001.stdout:8/74: fdatasync f1 0 2024-02-28T16:30:57.389 INFO:tasks.workunit.client.0.smithi001.stdout:6/14: dwrite f5 [0,4194304] 0 2024-02-28T16:30:57.389 INFO:tasks.workunit.client.0.smithi001.stdout:7/16: mkdir d3 0 2024-02-28T16:30:57.390 INFO:tasks.workunit.client.0.smithi001.stdout:0/17: truncate f0 1620321 0 2024-02-28T16:30:57.390 INFO:tasks.workunit.client.0.smithi001.stdout:3/46: rmdir d4/d6 39 2024-02-28T16:30:57.397 INFO:tasks.workunit.client.0.smithi001.stdout:0/18: symlink l4 0 2024-02-28T16:30:57.397 INFO:tasks.workunit.client.0.smithi001.stdout:0/19: rmdir - no directory 2024-02-28T16:30:57.401 INFO:tasks.workunit.client.0.smithi001.stdout:3/47: creat d4/d6/fc x:0 0 0 2024-02-28T16:30:57.403 INFO:tasks.workunit.client.0.smithi001.stdout:0/20: symlink l5 0 2024-02-28T16:30:57.403 INFO:tasks.workunit.client.0.smithi001.stdout:4/42: dwrite f2 [0,4194304] 0 2024-02-28T16:30:57.405 INFO:tasks.workunit.client.0.smithi001.stdout:4/43: read d3/fa [150296,13948] 0 2024-02-28T16:30:57.406 INFO:tasks.workunit.client.0.smithi001.stdout:8/75: symlink d4/d7/l15 0 2024-02-28T16:30:57.410 INFO:tasks.workunit.client.0.smithi001.stdout:0/21: unlink f3 0 2024-02-28T16:30:57.412 INFO:tasks.workunit.client.0.smithi001.stdout:4/44: creat d3/f12 x:0 0 0 2024-02-28T16:30:57.415 INFO:tasks.workunit.client.0.smithi001.stdout:3/48: creat d4/d6/d7/fd x:0 0 0 2024-02-28T16:30:57.416 INFO:tasks.workunit.client.0.smithi001.stdout:8/76: creat d4/f16 x:0 0 0 2024-02-28T16:30:57.416 INFO:tasks.workunit.client.0.smithi001.stdout:3/49: read - d4/d6/fc zero size 2024-02-28T16:30:57.421 INFO:tasks.workunit.client.0.smithi001.stdout:6/15: truncate f5 2837700 0 2024-02-28T16:30:57.422 INFO:tasks.workunit.client.0.smithi001.stdout:3/50: truncate d4/d6/fc 175216 0 2024-02-28T16:30:57.423 INFO:tasks.workunit.client.0.smithi001.stdout:6/16: chown f1 5 1 2024-02-28T16:30:57.423 INFO:tasks.workunit.client.0.smithi001.stdout:6/17: readlink l4 0 2024-02-28T16:30:57.426 INFO:tasks.workunit.client.0.smithi001.stdout:0/22: dread f0 [0,4194304] 0 2024-02-28T16:30:57.426 INFO:tasks.workunit.client.0.smithi001.stdout:6/18: creat f6 x:0 0 0 2024-02-28T16:30:57.427 INFO:tasks.workunit.client.0.smithi001.stdout:3/51: mknod d4/ce 0 2024-02-28T16:30:57.429 INFO:tasks.workunit.client.0.smithi001.stdout:4/45: dread d3/fa [0,4194304] 0 2024-02-28T16:30:57.430 INFO:tasks.workunit.client.0.smithi001.stdout:0/23: rename l2 to l6 0 2024-02-28T16:30:57.432 INFO:tasks.workunit.client.0.smithi001.stdout:6/19: mknod c7 0 2024-02-28T16:30:57.432 INFO:tasks.workunit.client.0.smithi001.stdout:3/52: dread - d4/d6/d7/fd zero size 2024-02-28T16:30:57.433 INFO:tasks.workunit.client.0.smithi001.stdout:4/46: unlink d3/ce 0 2024-02-28T16:30:57.434 INFO:tasks.workunit.client.0.smithi001.stdout:0/24: mknod c7 0 2024-02-28T16:30:57.435 INFO:tasks.workunit.client.0.smithi001.stdout:6/20: fdatasync f5 0 2024-02-28T16:30:57.435 INFO:tasks.workunit.client.0.smithi001.stdout:3/53: mkdir d4/d6/df 0 2024-02-28T16:30:57.436 INFO:tasks.workunit.client.0.smithi001.stdout:8/77: dwrite f3 [0,4194304] 0 2024-02-28T16:30:57.437 INFO:tasks.workunit.client.0.smithi001.stdout:4/47: write d3/f12 [964530,65140] 0 2024-02-28T16:30:57.442 INFO:tasks.workunit.client.0.smithi001.stdout:3/54: dread d4/d6/fc [0,4194304] 0 2024-02-28T16:30:57.449 INFO:tasks.workunit.client.0.smithi001.stdout:4/48: symlink d3/l13 0 2024-02-28T16:30:57.450 INFO:tasks.workunit.client.0.smithi001.stdout:6/21: read f5 [1720112,76209] 0 2024-02-28T16:30:57.456 INFO:tasks.workunit.client.0.smithi001.stdout:3/55: creat d4/d6/df/f10 x:0 0 0 2024-02-28T16:30:57.462 INFO:tasks.workunit.client.0.smithi001.stdout:3/56: write d4/d6/df/f10 [438652,120533] 0 2024-02-28T16:30:57.464 INFO:tasks.workunit.client.0.smithi001.stdout:0/25: dwrite f0 [0,4194304] 0 2024-02-28T16:30:57.464 INFO:tasks.workunit.client.0.smithi001.stdout:8/78: dread d4/d7/f10 [0,4194304] 0 2024-02-28T16:30:57.466 INFO:tasks.workunit.client.0.smithi001.stdout:0/26: read f0 [3476832,33980] 0 2024-02-28T16:30:57.467 INFO:tasks.workunit.client.0.smithi001.stdout:8/79: read - d4/f13 zero size 2024-02-28T16:30:57.468 INFO:tasks.workunit.client.0.smithi001.stdout:8/80: readlink d4/l8 0 2024-02-28T16:30:57.470 INFO:tasks.workunit.client.0.smithi001.stdout:8/81: write f1 [4024056,78883] 0 2024-02-28T16:30:57.470 INFO:tasks.workunit.client.0.smithi001.stdout:4/49: dread d3/fa [0,4194304] 0 2024-02-28T16:30:57.478 INFO:tasks.workunit.client.0.smithi001.stdout:3/57: getdents d4/d6/df 0 2024-02-28T16:30:57.481 INFO:tasks.workunit.client.0.smithi001.stdout:8/82: link d4/l5 d4/d7/l17 0 2024-02-28T16:30:57.482 INFO:tasks.workunit.client.0.smithi001.stdout:0/27: dread f0 [0,4194304] 0 2024-02-28T16:30:57.484 INFO:tasks.workunit.client.0.smithi001.stdout:3/58: truncate d4/d6/fc 853735 0 2024-02-28T16:30:57.485 INFO:tasks.workunit.client.0.smithi001.stdout:0/28: mknod c8 0 2024-02-28T16:30:57.486 INFO:tasks.workunit.client.0.smithi001.stdout:8/83: unlink d4/d7/l17 0 2024-02-28T16:30:57.487 INFO:tasks.workunit.client.0.smithi001.stdout:0/29: fsync f0 0 2024-02-28T16:30:57.488 INFO:tasks.workunit.client.0.smithi001.stdout:3/59: creat d4/d6/df/f11 x:0 0 0 2024-02-28T16:30:57.489 INFO:tasks.workunit.client.0.smithi001.stdout:8/84: readlink d4/d7/lb 0 2024-02-28T16:30:57.490 INFO:tasks.workunit.client.0.smithi001.stdout:0/30: creat f9 x:0 0 0 2024-02-28T16:30:57.492 INFO:tasks.workunit.client.0.smithi001.stdout:0/31: creat fa x:0 0 0 2024-02-28T16:30:57.494 INFO:tasks.workunit.client.0.smithi001.stdout:0/32: symlink lb 0 2024-02-28T16:30:57.494 INFO:tasks.workunit.client.0.smithi001.stdout:0/33: fsync f0 0 2024-02-28T16:30:57.496 INFO:tasks.workunit.client.0.smithi001.stdout:0/34: creat fc x:0 0 0 2024-02-28T16:30:57.496 INFO:tasks.workunit.client.0.smithi001.stdout:0/35: dread - f9 zero size 2024-02-28T16:30:57.498 INFO:tasks.workunit.client.0.smithi001.stdout:0/36: dread - fa zero size 2024-02-28T16:30:57.499 INFO:tasks.workunit.client.0.smithi001.stdout:8/85: write d4/d7/fd [680808,61425] 0 2024-02-28T16:30:57.500 INFO:tasks.workunit.client.0.smithi001.stdout:3/60: truncate d4/d6/df/f10 350979 0 2024-02-28T16:30:57.503 INFO:tasks.workunit.client.0.smithi001.stdout:3/61: mknod d4/d6/df/c12 0 2024-02-28T16:30:57.505 INFO:tasks.workunit.client.0.smithi001.stdout:3/62: mkdir d4/d6/d7/d13 0 2024-02-28T16:30:57.511 INFO:tasks.workunit.client.0.smithi001.stdout:0/37: dread f0 [0,4194304] 0 2024-02-28T16:30:57.511 INFO:tasks.workunit.client.0.smithi001.stdout:8/86: dread f3 [0,4194304] 0 2024-02-28T16:30:57.512 INFO:tasks.workunit.client.0.smithi001.stdout:0/38: dread - fc zero size 2024-02-28T16:30:57.534 INFO:tasks.workunit.client.0.smithi001.stdout:3/63: dwrite d4/fb [0,4194304] 0 2024-02-28T16:30:57.544 INFO:tasks.workunit.client.0.smithi001.stdout:0/39: dwrite fc [0,4194304] 0 2024-02-28T16:30:57.546 INFO:tasks.workunit.client.0.smithi001.stdout:8/87: dwrite d4/f12 [4194304,4194304] 0 2024-02-28T16:30:57.548 INFO:tasks.workunit.client.0.smithi001.stdout:8/88: read - d4/f16 zero size 2024-02-28T16:30:57.550 INFO:tasks.workunit.client.0.smithi001.stdout:8/89: chown f1 384 1 2024-02-28T16:30:57.551 INFO:tasks.workunit.client.0.smithi001.stdout:0/40: write fc [2498052,7680] 0 2024-02-28T16:30:57.555 INFO:tasks.workunit.client.0.smithi001.stdout:0/41: creat fd x:0 0 0 2024-02-28T16:30:57.555 INFO:tasks.workunit.client.0.smithi001.stdout:8/90: write f3 [1540922,13863] 0 2024-02-28T16:30:57.562 INFO:tasks.workunit.client.0.smithi001.stdout:0/42: creat fe x:0 0 0 2024-02-28T16:30:57.563 INFO:tasks.workunit.client.0.smithi001.stdout:8/91: creat d4/d7/f18 x:0 0 0 2024-02-28T16:30:57.564 INFO:tasks.workunit.client.0.smithi001.stdout:9/107: sync 2024-02-28T16:30:57.564 INFO:tasks.workunit.client.0.smithi001.stdout:2/70: sync 2024-02-28T16:30:57.566 INFO:tasks.workunit.client.0.smithi001.stdout:0/43: mknod cf 0 2024-02-28T16:30:57.569 INFO:tasks.workunit.client.0.smithi001.stdout:2/71: dread - d0/f1d zero size 2024-02-28T16:30:57.569 INFO:tasks.workunit.client.0.smithi001.stdout:0/44: chown fe 3496 1 2024-02-28T16:30:57.571 INFO:tasks.workunit.client.0.smithi001.stdout:8/92: write d4/f12 [6991170,88232] 0 2024-02-28T16:30:57.574 INFO:tasks.workunit.client.0.smithi001.stdout:2/72: dread - d0/d2/f11 zero size 2024-02-28T16:30:57.577 INFO:tasks.workunit.client.0.smithi001.stdout:0/45: rename fc to f10 0 2024-02-28T16:30:57.578 INFO:tasks.workunit.client.0.smithi001.stdout:3/64: dwrite d4/d6/d7/fd [0,4194304] 0 2024-02-28T16:30:57.580 INFO:tasks.workunit.client.0.smithi001.stdout:2/73: chown d0/d2/d3/de/dc/l13 1207 1 2024-02-28T16:30:57.581 INFO:tasks.workunit.client.0.smithi001.stdout:8/93: rename d4/d7/f18 to d4/d7/f19 0 2024-02-28T16:30:57.583 INFO:tasks.workunit.client.0.smithi001.stdout:9/108: link d5/c8 d5/d27/c29 0 2024-02-28T16:30:57.584 INFO:tasks.workunit.client.0.smithi001.stdout:8/94: write d4/f16 [318214,79763] 0 2024-02-28T16:30:57.586 INFO:tasks.workunit.client.0.smithi001.stdout:9/109: write d5/f24 [219974,128680] 0 2024-02-28T16:30:57.587 INFO:tasks.workunit.client.0.smithi001.stdout:2/74: link d0/ff d0/d2/d6/d12/f1e 0 2024-02-28T16:30:57.588 INFO:tasks.workunit.client.0.smithi001.stdout:3/65: creat d4/f14 x:0 0 0 2024-02-28T16:30:57.589 INFO:tasks.workunit.client.0.smithi001.stdout:3/66: read d4/fb [4083077,60664] 0 2024-02-28T16:30:57.595 INFO:tasks.workunit.client.0.smithi001.stdout:8/95: link d4/d7/l15 d4/l1a 0 2024-02-28T16:30:57.595 INFO:tasks.workunit.client.0.smithi001.stdout:2/75: write d0/d2/d6/d12/f1e [1859738,4357] 0 2024-02-28T16:30:57.604 INFO:tasks.workunit.client.0.smithi001.stdout:8/96: rmdir d4/d7 39 2024-02-28T16:30:57.605 INFO:tasks.workunit.client.0.smithi001.stdout:8/97: chown f1 17319883 1 2024-02-28T16:30:57.606 INFO:tasks.workunit.client.0.smithi001.stdout:9/110: truncate d5/f17 2848815 0 2024-02-28T16:30:57.608 INFO:tasks.workunit.client.0.smithi001.stdout:0/46: dwrite f10 [0,4194304] 0 2024-02-28T16:30:57.609 INFO:tasks.workunit.client.0.smithi001.stdout:8/98: mknod d4/c1b 0 2024-02-28T16:30:57.610 INFO:tasks.workunit.client.0.smithi001.stdout:9/111: read d5/f1d [3382066,114112] 0 2024-02-28T16:30:57.620 INFO:tasks.workunit.client.0.smithi001.stdout:2/76: dwrite d0/d2/d3/de/dc/f10 [0,4194304] 0 2024-02-28T16:30:57.691 INFO:tasks.workunit.client.0.smithi001.stdout:8/99: unlink d4/d7/cf 0 2024-02-28T16:30:57.699 INFO:tasks.workunit.client.0.smithi001.stdout:9/112: chown d5/f17 1120981 1 2024-02-28T16:30:57.724 INFO:tasks.workunit.client.0.smithi001.stdout:1/187: sync 2024-02-28T16:30:57.736 INFO:tasks.workunit.client.0.smithi001.stdout:1/188: mknod d0/d9/d1a/d2b/c39 0 2024-02-28T16:30:57.740 INFO:tasks.workunit.client.0.smithi001.stdout:9/113: dwrite d5/f24 [0,4194304] 0 2024-02-28T16:30:57.741 INFO:tasks.workunit.client.0.smithi001.stdout:8/100: dwrite d4/f11 [0,4194304] 0 2024-02-28T16:30:57.741 INFO:tasks.workunit.client.0.smithi001.stdout:1/189: rename d0/d9/d1a/d1f to d0/d9/d1a/d1f/d3a 22 2024-02-28T16:30:57.744 INFO:tasks.workunit.client.0.smithi001.stdout:1/190: dread - d0/d9/d1a/d1f/f29 zero size 2024-02-28T16:30:57.748 INFO:tasks.workunit.client.0.smithi001.stdout:9/114: chown d5/f1e 37855165 1 2024-02-28T16:30:57.749 INFO:tasks.workunit.client.0.smithi001.stdout:1/191: truncate d0/d9/d1a/d2b/d26/f34 669346 0 2024-02-28T16:30:57.750 INFO:tasks.workunit.client.0.smithi001.stdout:8/101: mknod d4/c1c 0 2024-02-28T16:30:57.751 INFO:tasks.workunit.client.0.smithi001.stdout:1/192: chown d0/l4 1 1 2024-02-28T16:30:57.755 INFO:tasks.workunit.client.0.smithi001.stdout:9/115: truncate d5/fb 3784212 0 2024-02-28T16:30:57.762 INFO:tasks.workunit.client.0.smithi001.stdout:1/193: rename d0/d9/d1a/d1f/l20 to d0/l3b 0 2024-02-28T16:30:57.765 INFO:tasks.workunit.client.0.smithi001.stdout:9/116: creat d5/f2a x:0 0 0 2024-02-28T16:30:57.767 INFO:tasks.workunit.client.0.smithi001.stdout:8/102: getdents d4 0 2024-02-28T16:30:57.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:57 smithi001 ceph-mon[28991]: pgmap v42: 65 pgs: 65 active+clean; 262 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 85 KiB/s rd, 7.3 MiB/s wr, 131 op/s 2024-02-28T16:30:57.774 INFO:tasks.workunit.client.0.smithi001.stdout:1/194: symlink d0/d9/l3c 0 2024-02-28T16:30:57.781 INFO:tasks.workunit.client.0.smithi001.stdout:1/195: link d0/l18 d0/l3d 0 2024-02-28T16:30:57.793 INFO:tasks.workunit.client.0.smithi001.stdout:9/117: dwrite d5/f12 [0,4194304] 0 2024-02-28T16:30:57.796 INFO:tasks.workunit.client.0.smithi001.stdout:9/118: dread - d5/f1e zero size 2024-02-28T16:30:57.801 INFO:tasks.workunit.client.0.smithi001.stdout:9/119: getdents d5/d27 0 2024-02-28T16:30:57.802 INFO:tasks.workunit.client.0.smithi001.stdout:1/196: dwrite d0/f28 [0,4194304] 0 2024-02-28T16:30:57.804 INFO:tasks.workunit.client.0.smithi001.stdout:9/120: write d5/f18 [3675735,77160] 0 2024-02-28T16:30:57.805 INFO:tasks.workunit.client.0.smithi001.stdout:1/197: write d0/d9/d1a/d2b/d26/f37 [464077,38585] 0 2024-02-28T16:30:57.808 INFO:tasks.workunit.client.0.smithi001.stdout:9/121: dread - d5/f19 zero size 2024-02-28T16:30:57.810 INFO:tasks.workunit.client.0.smithi001.stdout:1/198: symlink d0/l3e 0 2024-02-28T16:30:57.812 INFO:tasks.workunit.client.0.smithi001.stdout:9/122: creat d5/d27/f2b x:0 0 0 2024-02-28T16:30:57.814 INFO:tasks.workunit.client.0.smithi001.stdout:9/123: write d5/f24 [2089744,73002] 0 2024-02-28T16:30:57.819 INFO:tasks.workunit.client.0.smithi001.stdout:1/199: rename d0/d9/d1a/d2b/c2d to d0/d9/d1a/d2b/d26/c3f 0 2024-02-28T16:30:57.827 INFO:tasks.workunit.client.0.smithi001.stdout:1/200: rename d0/d9/d1a/d2b/c22 to d0/d9/d1a/d1f/d2f/c40 0 2024-02-28T16:30:57.834 INFO:tasks.workunit.client.0.smithi001.stdout:1/201: write d0/f2c [1095889,107709] 0 2024-02-28T16:30:57.835 INFO:tasks.workunit.client.0.smithi001.stdout:5/50: sync 2024-02-28T16:30:57.836 INFO:tasks.workunit.client.0.smithi001.stdout:9/124: dwrite d5/f1d [0,4194304] 0 2024-02-28T16:30:57.838 INFO:tasks.workunit.client.0.smithi001.stdout:9/125: write d5/f1e [75300,83206] 0 2024-02-28T16:30:57.840 INFO:tasks.workunit.client.0.smithi001.stdout:1/202: symlink d0/l41 0 2024-02-28T16:30:57.842 INFO:tasks.workunit.client.0.smithi001.stdout:1/203: dread - d0/f36 zero size 2024-02-28T16:30:57.847 INFO:tasks.workunit.client.0.smithi001.stdout:9/126: truncate d5/f6 1499799 0 2024-02-28T16:30:57.848 INFO:tasks.workunit.client.0.smithi001.stdout:5/51: dread f7 [0,4194304] 0 2024-02-28T16:30:57.848 INFO:tasks.workunit.client.0.smithi001.stdout:9/127: write d5/f18 [2187910,21857] 0 2024-02-28T16:30:57.851 INFO:tasks.workunit.client.0.smithi001.stdout:5/52: mkdir dc 0 2024-02-28T16:30:57.853 INFO:tasks.workunit.client.0.smithi001.stdout:5/53: write f4 [1110446,38977] 0 2024-02-28T16:30:57.863 INFO:tasks.workunit.client.0.smithi001.stdout:1/204: dwrite d0/d9/d1a/d2b/d26/f34 [0,4194304] 0 2024-02-28T16:30:57.863 INFO:tasks.workunit.client.0.smithi001.stdout:9/128: truncate d5/f12 3413497 0 2024-02-28T16:30:57.866 INFO:tasks.workunit.client.0.smithi001.stdout:1/205: rename d0/d9/d1a to d0/d9/d1a/d2b/d26/d27/d42 22 2024-02-28T16:30:57.867 INFO:tasks.workunit.client.0.smithi001.stdout:5/54: truncate fa 2027993 0 2024-02-28T16:30:57.868 INFO:tasks.workunit.client.0.smithi001.stdout:1/206: fdatasync d0/d9/d1a/f33 0 2024-02-28T16:30:57.869 INFO:tasks.workunit.client.0.smithi001.stdout:5/55: chown lb 30 1 2024-02-28T16:30:57.871 INFO:tasks.workunit.client.0.smithi001.stdout:1/207: stat d0/d9/d1a/d1f/f2e 0 2024-02-28T16:30:57.872 INFO:tasks.workunit.client.0.smithi001.stdout:5/56: getdents dc 0 2024-02-28T16:30:57.875 INFO:tasks.workunit.client.0.smithi001.stdout:5/57: rename l9 to dc/ld 0 2024-02-28T16:30:57.878 INFO:tasks.workunit.client.0.smithi001.stdout:5/58: write f7 [3186502,101720] 0 2024-02-28T16:30:57.878 INFO:tasks.workunit.client.0.smithi001.stdout:9/129: dread d5/f21 [0,4194304] 0 2024-02-28T16:30:57.879 INFO:tasks.workunit.client.0.smithi001.stdout:5/59: write f2 [2189333,23170] 0 2024-02-28T16:30:57.879 INFO:tasks.workunit.client.0.smithi001.stdout:1/208: rename d0/l1d to d0/d9/d1a/d1f/d2f/l43 0 2024-02-28T16:30:57.881 INFO:tasks.workunit.client.0.smithi001.stdout:1/209: chown d0/d9/fa 365247047 1 2024-02-28T16:30:57.882 INFO:tasks.workunit.client.0.smithi001.stdout:9/130: unlink d5/f17 0 2024-02-28T16:30:57.883 INFO:tasks.workunit.client.0.smithi001.stdout:5/60: creat dc/fe x:0 0 0 2024-02-28T16:30:57.884 INFO:tasks.workunit.client.0.smithi001.stdout:1/210: mknod d0/d9/d1a/d1f/d2f/c44 0 2024-02-28T16:30:57.885 INFO:tasks.workunit.client.0.smithi001.stdout:9/131: creat d5/d27/f2c x:0 0 0 2024-02-28T16:30:57.886 INFO:tasks.workunit.client.0.smithi001.stdout:5/61: mkdir dc/df 0 2024-02-28T16:30:57.887 INFO:tasks.workunit.client.0.smithi001.stdout:5/62: chown f4 2 1 2024-02-28T16:30:57.889 INFO:tasks.workunit.client.0.smithi001.stdout:5/63: symlink dc/l10 0 2024-02-28T16:30:57.892 INFO:tasks.workunit.client.0.smithi001.stdout:5/64: creat dc/df/f11 x:0 0 0 2024-02-28T16:30:57.896 INFO:tasks.workunit.client.0.smithi001.stdout:1/211: readlink d0/d9/d1a/d1f/d2f/l43 0 2024-02-28T16:30:57.902 INFO:tasks.workunit.client.0.smithi001.stdout:5/65: truncate f7 3828122 0 2024-02-28T16:30:57.917 INFO:tasks.workunit.client.0.smithi001.stdout:5/66: dwrite dc/df/f11 [0,4194304] 0 2024-02-28T16:30:57.920 INFO:tasks.workunit.client.0.smithi001.stdout:5/67: write dc/df/f11 [2934612,43147] 0 2024-02-28T16:30:57.929 INFO:tasks.workunit.client.0.smithi001.stdout:5/68: write fa [1252236,71727] 0 2024-02-28T16:30:57.932 INFO:tasks.workunit.client.0.smithi001.stdout:5/69: mkdir dc/d12 0 2024-02-28T16:30:57.934 INFO:tasks.workunit.client.0.smithi001.stdout:5/70: creat dc/df/f13 x:0 0 0 2024-02-28T16:30:57.942 INFO:tasks.workunit.client.0.smithi001.stdout:5/71: getdents dc/df 0 2024-02-28T16:30:57.946 INFO:tasks.workunit.client.0.smithi001.stdout:5/72: rename dc/fe to dc/d12/f14 0 2024-02-28T16:30:57.946 INFO:tasks.workunit.client.0.smithi001.stdout:5/73: chown c1 962349 1 2024-02-28T16:30:57.948 INFO:tasks.workunit.client.0.smithi001.stdout:5/74: symlink dc/df/l15 0 2024-02-28T16:30:57.951 INFO:tasks.workunit.client.0.smithi001.stdout:5/75: creat dc/d12/f16 x:0 0 0 2024-02-28T16:30:57.953 INFO:tasks.workunit.client.0.smithi001.stdout:5/76: mknod dc/d12/c17 0 2024-02-28T16:30:57.957 INFO:tasks.workunit.client.0.smithi001.stdout:5/77: rename dc/df/l15 to dc/l18 0 2024-02-28T16:30:57.960 INFO:tasks.workunit.client.0.smithi001.stdout:5/78: creat dc/d12/f19 x:0 0 0 2024-02-28T16:30:57.960 INFO:tasks.workunit.client.0.smithi001.stdout:5/79: stat dc/df 0 2024-02-28T16:30:57.974 INFO:tasks.workunit.client.0.smithi001.stdout:5/80: dread f7 [0,4194304] 0 2024-02-28T16:30:57.982 INFO:tasks.workunit.client.0.smithi001.stdout:0/47: sync 2024-02-28T16:30:57.983 INFO:tasks.workunit.client.0.smithi001.stdout:4/50: sync 2024-02-28T16:30:57.983 INFO:tasks.workunit.client.0.smithi001.stdout:2/77: sync 2024-02-28T16:30:57.983 INFO:tasks.workunit.client.0.smithi001.stdout:6/22: sync 2024-02-28T16:30:57.983 INFO:tasks.workunit.client.0.smithi001.stdout:3/67: sync 2024-02-28T16:30:57.983 INFO:tasks.workunit.client.0.smithi001.stdout:7/17: sync 2024-02-28T16:30:57.986 INFO:tasks.workunit.client.0.smithi001.stdout:7/18: chown d3 29 1 2024-02-28T16:30:57.987 INFO:tasks.workunit.client.0.smithi001.stdout:6/23: read - f1 zero size 2024-02-28T16:30:57.987 INFO:tasks.workunit.client.0.smithi001.stdout:2/78: chown d0/l7 5265452 1 2024-02-28T16:30:57.988 INFO:tasks.workunit.client.0.smithi001.stdout:0/48: unlink l4 0 2024-02-28T16:30:57.990 INFO:tasks.workunit.client.0.smithi001.stdout:3/68: rmdir d4/d6/df 39 2024-02-28T16:30:57.990 INFO:tasks.workunit.client.0.smithi001.stdout:6/24: chown l4 1 1 2024-02-28T16:30:57.990 INFO:tasks.workunit.client.0.smithi001.stdout:4/51: rmdir d3/d5/d7 39 2024-02-28T16:30:57.993 INFO:tasks.workunit.client.0.smithi001.stdout:0/49: truncate f9 609403 0 2024-02-28T16:30:57.993 INFO:tasks.workunit.client.0.smithi001.stdout:7/19: rename l2 to d3/l4 0 2024-02-28T16:30:57.996 INFO:tasks.workunit.client.0.smithi001.stdout:3/69: truncate d4/f14 386042 0 2024-02-28T16:30:57.997 INFO:tasks.workunit.client.0.smithi001.stdout:0/50: write f10 [3820137,57996] 0 2024-02-28T16:30:57.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:57 smithi066 ceph-mon[36236]: pgmap v42: 65 pgs: 65 active+clean; 262 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 85 KiB/s rd, 7.3 MiB/s wr, 131 op/s 2024-02-28T16:30:58.000 INFO:tasks.workunit.client.0.smithi001.stdout:5/81: dwrite f3 [0,4194304] 0 2024-02-28T16:30:58.001 INFO:tasks.workunit.client.0.smithi001.stdout:0/51: write f10 [184271,101693] 0 2024-02-28T16:30:58.001 INFO:tasks.workunit.client.0.smithi001.stdout:5/82: fdatasync fa 0 2024-02-28T16:30:58.035 INFO:tasks.workunit.client.0.smithi001.stdout:7/20: creat d3/f5 x:0 0 0 2024-02-28T16:30:58.037 INFO:tasks.workunit.client.0.smithi001.stdout:4/52: creat d3/d5/f14 x:0 0 0 2024-02-28T16:30:58.039 INFO:tasks.workunit.client.0.smithi001.stdout:7/21: write d3/f5 [78044,7278] 0 2024-02-28T16:30:58.042 INFO:tasks.workunit.client.0.smithi001.stdout:4/53: fsync d3/d5/f6 0 2024-02-28T16:30:58.043 INFO:tasks.workunit.client.0.smithi001.stdout:2/79: truncate d0/d2/d3/de/fa 2690228 0 2024-02-28T16:30:58.045 INFO:tasks.workunit.client.0.smithi001.stdout:2/80: readlink d0/d2/d6/l19 0 2024-02-28T16:30:58.046 INFO:tasks.workunit.client.0.smithi001.stdout:0/52: creat f11 x:0 0 0 2024-02-28T16:30:58.047 INFO:tasks.workunit.client.0.smithi001.stdout:5/83: mkdir dc/d1a 0 2024-02-28T16:30:58.048 INFO:tasks.workunit.client.0.smithi001.stdout:3/70: symlink d4/d6/df/l15 0 2024-02-28T16:30:58.050 INFO:tasks.workunit.client.0.smithi001.stdout:3/71: chown d4/fb 93720 1 2024-02-28T16:30:58.052 INFO:tasks.workunit.client.0.smithi001.stdout:2/81: stat d0/d2/l1a 0 2024-02-28T16:30:58.052 INFO:tasks.workunit.client.0.smithi001.stdout:0/53: creat f12 x:0 0 0 2024-02-28T16:30:58.053 INFO:tasks.workunit.client.0.smithi001.stdout:5/84: readlink dc/l18 0 2024-02-28T16:30:58.053 INFO:tasks.workunit.client.0.smithi001.stdout:2/82: chown d0/f1d 1153 1 2024-02-28T16:30:58.054 INFO:tasks.workunit.client.0.smithi001.stdout:4/54: symlink d3/d5/l15 0 2024-02-28T16:30:58.057 INFO:tasks.workunit.client.0.smithi001.stdout:0/54: creat f13 x:0 0 0 2024-02-28T16:30:58.058 INFO:tasks.workunit.client.0.smithi001.stdout:2/83: creat d0/d2/d6/f1f x:0 0 0 2024-02-28T16:30:58.062 INFO:tasks.workunit.client.0.smithi001.stdout:0/55: rename c7 to c14 0 2024-02-28T16:30:58.064 INFO:tasks.workunit.client.0.smithi001.stdout:3/72: rmdir d4/d6/d7/d13 0 2024-02-28T16:30:58.066 INFO:tasks.workunit.client.0.smithi001.stdout:4/55: symlink d3/d5/d7/l16 0 2024-02-28T16:30:58.068 INFO:tasks.workunit.client.0.smithi001.stdout:2/84: mknod d0/d2/d3/de/dc/c20 0 2024-02-28T16:30:58.071 INFO:tasks.workunit.client.0.smithi001.stdout:3/73: symlink d4/l16 0 2024-02-28T16:30:58.072 INFO:tasks.workunit.client.0.smithi001.stdout:4/56: creat d3/f17 x:0 0 0 2024-02-28T16:30:58.075 INFO:tasks.workunit.client.0.smithi001.stdout:4/57: read - d3/f17 zero size 2024-02-28T16:30:58.078 INFO:tasks.workunit.client.0.smithi001.stdout:3/74: fsync d4/d6/df/f10 0 2024-02-28T16:30:58.081 INFO:tasks.workunit.client.0.smithi001.stdout:4/58: write d3/d5/dd/ff [805373,51490] 0 2024-02-28T16:30:58.082 INFO:tasks.workunit.client.0.smithi001.stdout:8/103: sync 2024-02-28T16:30:58.084 INFO:tasks.workunit.client.0.smithi001.stdout:4/59: fsync d3/f12 0 2024-02-28T16:30:58.086 INFO:tasks.workunit.client.0.smithi001.stdout:0/56: dwrite f13 [0,4194304] 0 2024-02-28T16:30:58.092 INFO:tasks.workunit.client.0.smithi001.stdout:3/75: rmdir d4 39 2024-02-28T16:30:58.095 INFO:tasks.workunit.client.0.smithi001.stdout:8/104: rename d4/f16 to d4/d7/f1d 0 2024-02-28T16:30:58.096 INFO:tasks.workunit.client.0.smithi001.stdout:0/57: creat f15 x:0 0 0 2024-02-28T16:30:58.097 INFO:tasks.workunit.client.0.smithi001.stdout:0/58: readlink l5 0 2024-02-28T16:30:58.101 INFO:tasks.workunit.client.0.smithi001.stdout:4/60: dread d3/d5/f6 [0,4194304] 0 2024-02-28T16:30:58.102 INFO:tasks.workunit.client.0.smithi001.stdout:2/85: write d0/d2/d3/de/fa [105542,92791] 0 2024-02-28T16:30:58.103 INFO:tasks.workunit.client.0.smithi001.stdout:2/86: stat d0/d2/d6/d12/f1e 0 2024-02-28T16:30:58.109 INFO:tasks.workunit.client.0.smithi001.stdout:2/87: getdents d0/d2/d3 0 2024-02-28T16:30:58.110 INFO:tasks.workunit.client.0.smithi001.stdout:8/105: dread - d4/d7/f19 zero size 2024-02-28T16:30:58.112 INFO:tasks.workunit.client.0.smithi001.stdout:0/59: truncate f10 3322492 0 2024-02-28T16:30:58.112 INFO:tasks.workunit.client.0.smithi001.stdout:3/76: write d4/d6/df/f10 [958882,27370] 0 2024-02-28T16:30:58.117 INFO:tasks.workunit.client.0.smithi001.stdout:0/60: truncate f9 1217133 0 2024-02-28T16:30:58.118 INFO:tasks.workunit.client.0.smithi001.stdout:2/88: truncate d0/d2/d6/f15 768616 0 2024-02-28T16:30:58.118 INFO:tasks.workunit.client.0.smithi001.stdout:8/106: read d4/d7/f10 [573135,130249] 0 2024-02-28T16:30:58.118 INFO:tasks.workunit.client.0.smithi001.stdout:3/77: creat d4/f17 x:0 0 0 2024-02-28T16:30:58.119 INFO:tasks.workunit.client.0.smithi001.stdout:2/89: creat d0/d2/f21 x:0 0 0 2024-02-28T16:30:58.121 INFO:tasks.workunit.client.0.smithi001.stdout:8/107: symlink d4/l1e 0 2024-02-28T16:30:58.123 INFO:tasks.workunit.client.0.smithi001.stdout:2/90: mkdir d0/d2/d3/de/d22 0 2024-02-28T16:30:58.124 INFO:tasks.workunit.client.0.smithi001.stdout:8/108: write d4/f13 [877976,67931] 0 2024-02-28T16:30:58.125 INFO:tasks.workunit.client.0.smithi001.stdout:3/78: rename d4/d6/df to d4/d18 0 2024-02-28T16:30:58.128 INFO:tasks.workunit.client.0.smithi001.stdout:8/109: write f3 [1344206,70504] 0 2024-02-28T16:30:58.135 INFO:tasks.workunit.client.0.smithi001.stdout:8/110: symlink d4/l1f 0 2024-02-28T16:30:58.147 INFO:tasks.workunit.client.0.smithi001.stdout:0/61: dwrite fe [0,4194304] 0 2024-02-28T16:30:58.151 INFO:tasks.workunit.client.0.smithi001.stdout:0/62: dread - f11 zero size 2024-02-28T16:30:58.152 INFO:tasks.workunit.client.0.smithi001.stdout:0/63: stat c8 0 2024-02-28T16:30:58.158 INFO:tasks.workunit.client.0.smithi001.stdout:2/91: dwrite d0/f1d [0,4194304] 0 2024-02-28T16:30:58.159 INFO:tasks.workunit.client.0.smithi001.stdout:3/79: dread d4/d6/d7/fd [0,4194304] 0 2024-02-28T16:30:58.167 INFO:tasks.workunit.client.0.smithi001.stdout:3/80: rename d4/fb to d4/d18/f19 0 2024-02-28T16:30:58.174 INFO:tasks.workunit.client.0.smithi001.stdout:0/64: fsync fe 0 2024-02-28T16:30:58.176 INFO:tasks.workunit.client.0.smithi001.stdout:3/81: getdents d4/d6/d7 0 2024-02-28T16:30:58.176 INFO:tasks.workunit.client.0.smithi001.stdout:8/111: dwrite d4/d7/f19 [0,4194304] 0 2024-02-28T16:30:58.182 INFO:tasks.workunit.client.0.smithi001.stdout:8/112: creat d4/d7/f20 x:0 0 0 2024-02-28T16:30:58.183 INFO:tasks.workunit.client.0.smithi001.stdout:3/82: mknod d4/c1a 0 2024-02-28T16:30:58.185 INFO:tasks.workunit.client.0.smithi001.stdout:0/65: getdents . 0 2024-02-28T16:30:58.187 INFO:tasks.workunit.client.0.smithi001.stdout:8/113: creat d4/d7/f21 x:0 0 0 2024-02-28T16:30:58.188 INFO:tasks.workunit.client.0.smithi001.stdout:0/66: truncate f15 20555 0 2024-02-28T16:30:58.188 INFO:tasks.workunit.client.0.smithi001.stdout:2/92: dwrite d0/ff [4194304,4194304] 0 2024-02-28T16:30:58.192 INFO:tasks.workunit.client.0.smithi001.stdout:2/93: readlink d0/d2/d3/de/l1c 0 2024-02-28T16:30:58.193 INFO:tasks.workunit.client.0.smithi001.stdout:0/67: symlink l16 0 2024-02-28T16:30:58.201 INFO:tasks.workunit.client.0.smithi001.stdout:3/83: truncate d4/d18/f19 4267942 0 2024-02-28T16:30:58.202 INFO:tasks.workunit.client.0.smithi001.stdout:9/132: sync 2024-02-28T16:30:58.203 INFO:tasks.workunit.client.0.smithi001.stdout:1/212: sync 2024-02-28T16:30:58.206 INFO:tasks.workunit.client.0.smithi001.stdout:8/114: dread f3 [0,4194304] 0 2024-02-28T16:30:58.208 INFO:tasks.workunit.client.0.smithi001.stdout:9/133: rmdir d5 39 2024-02-28T16:30:58.209 INFO:tasks.workunit.client.0.smithi001.stdout:1/213: read - d0/f36 zero size 2024-02-28T16:30:58.211 INFO:tasks.workunit.client.0.smithi001.stdout:3/84: truncate d4/d18/f11 23987 0 2024-02-28T16:30:58.216 INFO:tasks.workunit.client.0.smithi001.stdout:8/115: mkdir d4/d7/d22 0 2024-02-28T16:30:58.217 INFO:tasks.workunit.client.0.smithi001.stdout:1/214: chown d0/d9/d1a/d2b/d26/d27 7768117 1 2024-02-28T16:30:58.220 INFO:tasks.workunit.client.0.smithi001.stdout:3/85: mknod d4/c1b 0 2024-02-28T16:30:58.222 INFO:tasks.workunit.client.0.smithi001.stdout:2/94: dwrite d0/ff [4194304,4194304] 0 2024-02-28T16:30:58.224 INFO:tasks.workunit.client.0.smithi001.stdout:9/134: getdents d5 0 2024-02-28T16:30:58.226 INFO:tasks.workunit.client.0.smithi001.stdout:0/68: dwrite fa [0,4194304] 0 2024-02-28T16:30:58.227 INFO:tasks.workunit.client.0.smithi001.stdout:9/135: chown d5/fd 1 1 2024-02-28T16:30:58.229 INFO:tasks.workunit.client.0.smithi001.stdout:1/215: mkdir d0/d9/d1a/d2b/d26/d45 0 2024-02-28T16:30:58.231 INFO:tasks.workunit.client.0.smithi001.stdout:0/69: creat f17 x:0 0 0 2024-02-28T16:30:58.235 INFO:tasks.workunit.client.0.smithi001.stdout:2/95: truncate d0/d2/d6/f15 1327431 0 2024-02-28T16:30:58.235 INFO:tasks.workunit.client.0.smithi001.stdout:3/86: creat d4/d6/f1c x:0 0 0 2024-02-28T16:30:58.236 INFO:tasks.workunit.client.0.smithi001.stdout:1/216: readlink d0/l3d 0 2024-02-28T16:30:58.244 INFO:tasks.workunit.client.0.smithi001.stdout:1/217: mknod d0/d9/d1a/d1f/d2f/c46 0 2024-02-28T16:30:58.253 INFO:tasks.workunit.client.0.smithi001.stdout:3/87: write d4/d6/d7/fd [5018677,62278] 0 2024-02-28T16:30:58.262 INFO:tasks.workunit.client.0.smithi001.stdout:8/116: dwrite d4/d7/f19 [4194304,4194304] 0 2024-02-28T16:30:58.271 INFO:tasks.workunit.client.0.smithi001.stdout:3/88: mknod d4/d6/c1d 0 2024-02-28T16:30:58.282 INFO:tasks.workunit.client.0.smithi001.stdout:9/136: dwrite d5/d27/f2b [0,4194304] 0 2024-02-28T16:30:58.283 INFO:tasks.workunit.client.0.smithi001.stdout:6/25: sync 2024-02-28T16:30:58.283 INFO:tasks.workunit.client.0.smithi001.stdout:7/22: sync 2024-02-28T16:30:58.285 INFO:tasks.workunit.client.0.smithi001.stdout:8/117: creat d4/d7/f23 x:0 0 0 2024-02-28T16:30:58.286 INFO:tasks.workunit.client.0.smithi001.stdout:3/89: mkdir d4/d1e 0 2024-02-28T16:30:58.289 INFO:tasks.workunit.client.0.smithi001.stdout:6/26: read f5 [678544,74601] 0 2024-02-28T16:30:58.290 INFO:tasks.workunit.client.0.smithi001.stdout:9/137: rmdir d5 39 2024-02-28T16:30:58.292 INFO:tasks.workunit.client.0.smithi001.stdout:0/70: dwrite f15 [0,4194304] 0 2024-02-28T16:30:58.294 INFO:tasks.workunit.client.0.smithi001.stdout:7/23: rename d3/f5 to d3/f6 0 2024-02-28T16:30:58.295 INFO:tasks.workunit.client.0.smithi001.stdout:2/96: dwrite d0/f1d [0,4194304] 0 2024-02-28T16:30:58.302 INFO:tasks.workunit.client.0.smithi001.stdout:8/118: chown d4/d7/l15 93 1 2024-02-28T16:30:58.302 INFO:tasks.workunit.client.0.smithi001.stdout:1/218: dwrite d0/f36 [0,4194304] 0 2024-02-28T16:30:58.302 INFO:tasks.workunit.client.0.smithi001.stdout:9/138: write d5/f15 [2658267,19631] 0 2024-02-28T16:30:58.303 INFO:tasks.workunit.client.0.smithi001.stdout:3/90: creat d4/d6/d7/f1f x:0 0 0 2024-02-28T16:30:58.304 INFO:tasks.workunit.client.0.smithi001.stdout:0/71: mkdir d18 0 2024-02-28T16:30:58.306 INFO:tasks.workunit.client.0.smithi001.stdout:0/72: read - fd zero size 2024-02-28T16:30:58.308 INFO:tasks.workunit.client.0.smithi001.stdout:8/119: symlink d4/d7/d22/l24 0 2024-02-28T16:30:58.308 INFO:tasks.workunit.client.0.smithi001.stdout:3/91: write d4/d18/f11 [968145,71124] 0 2024-02-28T16:30:58.310 INFO:tasks.workunit.client.0.smithi001.stdout:9/139: fdatasync d5/d27/f2c 0 2024-02-28T16:30:58.312 INFO:tasks.workunit.client.0.smithi001.stdout:0/73: rename c14 to d18/c19 0 2024-02-28T16:30:58.316 INFO:tasks.workunit.client.0.smithi001.stdout:9/140: creat d5/f2d x:0 0 0 2024-02-28T16:30:58.316 INFO:tasks.workunit.client.0.smithi001.stdout:3/92: creat d4/d6/d7/f20 x:0 0 0 2024-02-28T16:30:58.323 INFO:tasks.workunit.client.0.smithi001.stdout:1/219: getdents d0/d9/d1a/d2b/d26/d27/d30 0 2024-02-28T16:30:58.332 INFO:tasks.workunit.client.0.smithi001.stdout:6/27: dwrite f5 [0,4194304] 0 2024-02-28T16:30:58.338 INFO:tasks.workunit.client.0.smithi001.stdout:6/28: fsync f6 0 2024-02-28T16:30:58.338 INFO:tasks.workunit.client.0.smithi001.stdout:0/74: getdents d18 0 2024-02-28T16:30:58.339 INFO:tasks.workunit.client.0.smithi001.stdout:3/93: dread d4/d6/fc [0,4194304] 0 2024-02-28T16:30:58.344 INFO:tasks.workunit.client.0.smithi001.stdout:2/97: dwrite d0/d2/d6/f15 [0,4194304] 0 2024-02-28T16:30:58.347 INFO:tasks.workunit.client.0.smithi001.stdout:1/220: truncate d0/d9/d1a/d2b/f24 687015 0 2024-02-28T16:30:58.347 INFO:tasks.workunit.client.0.smithi001.stdout:6/29: write f5 [1846098,116676] 0 2024-02-28T16:30:58.347 INFO:tasks.workunit.client.0.smithi001.stdout:0/75: creat d18/f1a x:0 0 0 2024-02-28T16:30:58.351 INFO:tasks.workunit.client.0.smithi001.stdout:8/120: dwrite d4/f13 [0,4194304] 0 2024-02-28T16:30:58.352 INFO:tasks.workunit.client.0.smithi001.stdout:3/94: write d4/f14 [1287228,82828] 0 2024-02-28T16:30:58.352 INFO:tasks.workunit.client.0.smithi001.stdout:9/141: creat d5/f2e x:0 0 0 2024-02-28T16:30:58.353 INFO:tasks.workunit.client.0.smithi001.stdout:2/98: chown d0/l9 107154 1 2024-02-28T16:30:58.354 INFO:tasks.workunit.client.0.smithi001.stdout:6/30: dread - f6 zero size 2024-02-28T16:30:58.355 INFO:tasks.workunit.client.0.smithi001.stdout:1/221: chown d0/l16 887003 1 2024-02-28T16:30:58.356 INFO:tasks.workunit.client.0.smithi001.stdout:2/99: fdatasync d0/ff 0 2024-02-28T16:30:58.357 INFO:tasks.workunit.client.0.smithi001.stdout:6/31: chown l4 5456944 1 2024-02-28T16:30:58.357 INFO:tasks.workunit.client.0.smithi001.stdout:9/142: unlink d5/f24 0 2024-02-28T16:30:58.359 INFO:tasks.workunit.client.0.smithi001.stdout:6/32: dread - f6 zero size 2024-02-28T16:30:58.361 INFO:tasks.workunit.client.0.smithi001.stdout:1/222: rmdir d0/d9/d1a/d1f 39 2024-02-28T16:30:58.361 INFO:tasks.workunit.client.0.smithi001.stdout:2/100: truncate d0/d2/d6/f1f 16455 0 2024-02-28T16:30:58.364 INFO:tasks.workunit.client.0.smithi001.stdout:1/223: fsync d0/d9/d1a/d2b/d26/f37 0 2024-02-28T16:30:58.367 INFO:tasks.workunit.client.0.smithi001.stdout:0/76: rmdir d18 39 2024-02-28T16:30:58.367 INFO:tasks.workunit.client.0.smithi001.stdout:9/143: stat d5/d27/f2c 0 2024-02-28T16:30:58.372 INFO:tasks.workunit.client.0.smithi001.stdout:0/77: fdatasync fa 0 2024-02-28T16:30:58.372 INFO:tasks.workunit.client.0.smithi001.stdout:3/95: creat d4/d1e/f21 x:0 0 0 2024-02-28T16:30:58.373 INFO:tasks.workunit.client.0.smithi001.stdout:3/96: write d4/d1e/f21 [559347,72335] 0 2024-02-28T16:30:58.374 INFO:tasks.workunit.client.0.smithi001.stdout:8/121: link d4/l1e d4/d7/l25 0 2024-02-28T16:30:58.375 INFO:tasks.workunit.client.0.smithi001.stdout:8/122: readlink l0 0 2024-02-28T16:30:58.381 INFO:tasks.workunit.client.0.smithi001.stdout:6/33: dread f5 [0,4194304] 0 2024-02-28T16:30:58.384 INFO:tasks.workunit.client.0.smithi001.stdout:8/123: fdatasync d4/f11 0 2024-02-28T16:30:58.384 INFO:tasks.workunit.client.0.smithi001.stdout:9/144: mkdir d5/d2f 0 2024-02-28T16:30:58.385 INFO:tasks.workunit.client.0.smithi001.stdout:2/101: dread d0/d2/d3/de/dc/f10 [0,4194304] 0 2024-02-28T16:30:58.386 INFO:tasks.workunit.client.0.smithi001.stdout:6/34: creat f8 x:0 0 0 2024-02-28T16:30:58.391 INFO:tasks.workunit.client.0.smithi001.stdout:1/224: truncate d0/d9/d1a/d2b/f17 2769138 0 2024-02-28T16:30:58.393 INFO:tasks.workunit.client.0.smithi001.stdout:2/102: rmdir d0/d2/d3/de 39 2024-02-28T16:30:58.396 INFO:tasks.workunit.client.0.smithi001.stdout:0/78: dwrite f13 [0,4194304] 0 2024-02-28T16:30:58.400 INFO:tasks.workunit.client.0.smithi001.stdout:3/97: dwrite d4/d6/d7/f20 [0,4194304] 0 2024-02-28T16:30:58.402 INFO:tasks.workunit.client.0.smithi001.stdout:2/103: rmdir d0/d2 39 2024-02-28T16:30:58.411 INFO:tasks.workunit.client.0.smithi001.stdout:9/145: getdents d5 0 2024-02-28T16:30:58.415 INFO:tasks.workunit.client.0.smithi001.stdout:1/225: creat d0/d9/d1a/d2b/f47 x:0 0 0 2024-02-28T16:30:58.418 INFO:tasks.workunit.client.0.smithi001.stdout:0/79: fdatasync f13 0 2024-02-28T16:30:58.421 INFO:tasks.workunit.client.0.smithi001.stdout:9/146: creat d5/f30 x:0 0 0 2024-02-28T16:30:58.423 INFO:tasks.workunit.client.0.smithi001.stdout:0/80: dread - fd zero size 2024-02-28T16:30:58.428 INFO:tasks.workunit.client.0.smithi001.stdout:5/85: sync 2024-02-28T16:30:58.428 INFO:tasks.workunit.client.0.smithi001.stdout:4/61: sync 2024-02-28T16:30:58.429 INFO:tasks.workunit.client.0.smithi001.stdout:4/62: chown f2 0 1 2024-02-28T16:30:58.429 INFO:tasks.workunit.client.0.smithi001.stdout:8/124: dwrite d4/f11 [4194304,4194304] 0 2024-02-28T16:30:58.435 INFO:tasks.workunit.client.0.smithi001.stdout:6/35: dwrite f5 [0,4194304] 0 2024-02-28T16:30:58.437 INFO:tasks.workunit.client.0.smithi001.stdout:0/81: mkdir d18/d1b 0 2024-02-28T16:30:58.439 INFO:tasks.workunit.client.0.smithi001.stdout:0/82: write f0 [1166711,126277] 0 2024-02-28T16:30:58.439 INFO:tasks.workunit.client.0.smithi001.stdout:4/63: write d3/f12 [2002167,66818] 0 2024-02-28T16:30:58.443 INFO:tasks.workunit.client.0.smithi001.stdout:5/86: truncate dc/d12/f14 675728 0 2024-02-28T16:30:58.445 INFO:tasks.workunit.client.0.smithi001.stdout:5/87: chown dc 20676867 1 2024-02-28T16:30:58.450 INFO:tasks.workunit.client.0.smithi001.stdout:3/98: dwrite d4/d18/f11 [0,4194304] 0 2024-02-28T16:30:58.451 INFO:tasks.workunit.client.0.smithi001.stdout:4/64: rmdir d3 39 2024-02-28T16:30:58.452 INFO:tasks.workunit.client.0.smithi001.stdout:8/125: write d4/d7/f19 [6757740,89660] 0 2024-02-28T16:30:58.452 INFO:tasks.workunit.client.0.smithi001.stdout:2/104: dread d0/d2/d3/de/fa [0,4194304] 0 2024-02-28T16:30:58.452 INFO:tasks.workunit.client.0.smithi001.stdout:5/88: chown fa 14 1 2024-02-28T16:30:58.459 INFO:tasks.workunit.client.0.smithi001.stdout:1/226: dwrite d0/d9/d1a/f33 [0,4194304] 0 2024-02-28T16:30:58.464 INFO:tasks.workunit.client.0.smithi001.stdout:8/126: chown d4/d7/f1d 33075 1 2024-02-28T16:30:58.465 INFO:tasks.workunit.client.0.smithi001.stdout:9/147: creat d5/d27/f31 x:0 0 0 2024-02-28T16:30:58.465 INFO:tasks.workunit.client.0.smithi001.stdout:5/89: mknod dc/c1b 0 2024-02-28T16:30:58.466 INFO:tasks.workunit.client.0.smithi001.stdout:2/105: chown d0/d2/d3/de/dc/l13 1362376 1 2024-02-28T16:30:58.470 INFO:tasks.workunit.client.0.smithi001.stdout:0/83: mkdir d18/d1b/d1c 0 2024-02-28T16:30:58.471 INFO:tasks.workunit.client.0.smithi001.stdout:8/127: mknod d4/d7/d22/c26 0 2024-02-28T16:30:58.472 INFO:tasks.workunit.client.0.smithi001.stdout:9/148: chown d5/d27/f31 23 1 2024-02-28T16:30:58.473 INFO:tasks.workunit.client.0.smithi001.stdout:5/90: symlink dc/l1c 0 2024-02-28T16:30:58.474 INFO:tasks.workunit.client.0.smithi001.stdout:0/84: chown d18/f1a 56220620 1 2024-02-28T16:30:58.475 INFO:tasks.workunit.client.0.smithi001.stdout:4/65: chown d3/d5/d7/l16 14000421 1 2024-02-28T16:30:58.477 INFO:tasks.workunit.client.0.smithi001.stdout:6/36: dwrite f1 [0,4194304] 0 2024-02-28T16:30:58.480 INFO:tasks.workunit.client.0.smithi001.stdout:4/66: chown d3/d5/dd/ff 1635 1 2024-02-28T16:30:58.482 INFO:tasks.workunit.client.0.smithi001.stdout:8/128: chown d4/cc 686 1 2024-02-28T16:30:58.484 INFO:tasks.workunit.client.0.smithi001.stdout:9/149: chown d5/c8 324 1 2024-02-28T16:30:58.484 INFO:tasks.workunit.client.0.smithi001.stdout:0/85: creat d18/f1d x:0 0 0 2024-02-28T16:30:58.485 INFO:tasks.workunit.client.0.smithi001.stdout:6/37: mkdir d9 0 2024-02-28T16:30:58.487 INFO:tasks.workunit.client.0.smithi001.stdout:4/67: creat d3/f18 x:0 0 0 2024-02-28T16:30:58.489 INFO:tasks.workunit.client.0.smithi001.stdout:8/129: mkdir d4/d7/d22/d27 0 2024-02-28T16:30:58.490 INFO:tasks.workunit.client.0.smithi001.stdout:1/227: dwrite d0/d9/ff [4194304,4194304] 0 2024-02-28T16:30:58.491 INFO:tasks.workunit.client.0.smithi001.stdout:0/86: chown f11 460146007 1 2024-02-28T16:30:58.495 INFO:tasks.workunit.client.0.smithi001.stdout:9/150: rename d5/c10 to d5/d27/c32 0 2024-02-28T16:30:58.499 INFO:tasks.workunit.client.0.smithi001.stdout:0/87: truncate f11 370088 0 2024-02-28T16:30:58.503 INFO:tasks.workunit.client.0.smithi001.stdout:0/88: fdatasync fa 0 2024-02-28T16:30:58.503 INFO:tasks.workunit.client.0.smithi001.stdout:6/38: creat d9/fa x:0 0 0 2024-02-28T16:30:58.503 INFO:tasks.workunit.client.0.smithi001.stdout:8/130: write f1 [973450,69115] 0 2024-02-28T16:30:58.505 INFO:tasks.workunit.client.0.smithi001.stdout:5/91: dwrite dc/df/f13 [0,4194304] 0 2024-02-28T16:30:58.507 INFO:tasks.workunit.client.0.smithi001.stdout:6/39: fsync f5 0 2024-02-28T16:30:58.507 INFO:tasks.workunit.client.0.smithi001.stdout:2/106: dwrite d0/d2/f11 [0,4194304] 0 2024-02-28T16:30:58.517 INFO:tasks.workunit.client.0.smithi001.stdout:6/40: mkdir d9/db 0 2024-02-28T16:30:58.518 INFO:tasks.workunit.client.0.smithi001.stdout:8/131: creat d4/d7/d22/f28 x:0 0 0 2024-02-28T16:30:58.519 INFO:tasks.workunit.client.0.smithi001.stdout:5/92: creat dc/f1d x:0 0 0 2024-02-28T16:30:58.520 INFO:tasks.workunit.client.0.smithi001.stdout:0/89: rename d18/f1a to d18/f1e 0 2024-02-28T16:30:58.524 INFO:tasks.workunit.client.0.smithi001.stdout:0/90: chown cf 234 1 2024-02-28T16:30:58.526 INFO:tasks.workunit.client.0.smithi001.stdout:5/93: rename dc to dc/d1a/d1e 22 2024-02-28T16:30:58.532 INFO:tasks.workunit.client.0.smithi001.stdout:9/151: write d5/f1d [5066020,62354] 0 2024-02-28T16:30:58.532 INFO:tasks.workunit.client.0.smithi001.stdout:4/68: dwrite d3/f17 [0,4194304] 0 2024-02-28T16:30:58.533 INFO:tasks.workunit.client.0.smithi001.stdout:0/91: creat d18/d1b/f1f x:0 0 0 2024-02-28T16:30:58.535 INFO:tasks.workunit.client.0.smithi001.stdout:4/69: stat d3/l4 0 2024-02-28T16:30:58.540 INFO:tasks.workunit.client.0.smithi001.stdout:1/228: dread d0/d9/d1a/d2b/f17 [0,4194304] 0 2024-02-28T16:30:58.547 INFO:tasks.workunit.client.0.smithi001.stdout:9/152: truncate d5/f6 84064 0 2024-02-28T16:30:58.552 INFO:tasks.workunit.client.0.smithi001.stdout:2/107: dread d0/ff [0,4194304] 0 2024-02-28T16:30:58.606 INFO:tasks.workunit.client.0.smithi001.stdout:9/153: mkdir d5/d2f/d33 0 2024-02-28T16:30:58.606 INFO:tasks.workunit.client.0.smithi001.stdout:2/108: truncate d0/d2/d6/f1f 316407 0 2024-02-28T16:30:58.608 INFO:tasks.workunit.client.0.smithi001.stdout:6/41: dwrite f6 [0,4194304] 0 2024-02-28T16:30:58.619 INFO:tasks.workunit.client.0.smithi001.stdout:5/94: dwrite dc/f1d [0,4194304] 0 2024-02-28T16:30:58.620 INFO:tasks.workunit.client.0.smithi001.stdout:8/132: dwrite d4/d7/f20 [0,4194304] 0 2024-02-28T16:30:58.623 INFO:tasks.workunit.client.0.smithi001.stdout:1/229: dwrite d0/f36 [0,4194304] 0 2024-02-28T16:30:58.626 INFO:tasks.workunit.client.0.smithi001.stdout:0/92: dwrite fa [0,4194304] 0 2024-02-28T16:30:58.628 INFO:tasks.workunit.client.0.smithi001.stdout:2/109: creat d0/d2/d6/f23 x:0 0 0 2024-02-28T16:30:58.631 INFO:tasks.workunit.client.0.smithi001.stdout:8/133: chown d4/d7/f20 0 1 2024-02-28T16:30:58.631 INFO:tasks.workunit.client.0.smithi001.stdout:1/230: fdatasync d0/f21 0 2024-02-28T16:30:58.633 INFO:tasks.workunit.client.0.smithi001.stdout:6/42: rename d9/fa to d9/db/fc 0 2024-02-28T16:30:58.634 INFO:tasks.workunit.client.0.smithi001.stdout:0/93: write d18/f1d [299624,75877] 0 2024-02-28T16:30:58.638 INFO:tasks.workunit.client.0.smithi001.stdout:5/95: mknod dc/d1a/c1f 0 2024-02-28T16:30:58.638 INFO:tasks.workunit.client.0.smithi001.stdout:6/43: dread - d9/db/fc zero size 2024-02-28T16:30:58.643 INFO:tasks.workunit.client.0.smithi001.stdout:0/94: unlink f17 0 2024-02-28T16:30:58.643 INFO:tasks.workunit.client.0.smithi001.stdout:2/110: creat d0/d2/d3/de/d22/f24 x:0 0 0 2024-02-28T16:30:58.647 INFO:tasks.workunit.client.0.smithi001.stdout:9/154: truncate d5/f15 2429916 0 2024-02-28T16:30:58.647 INFO:tasks.workunit.client.0.smithi001.stdout:6/44: creat d9/db/fd x:0 0 0 2024-02-28T16:30:58.649 INFO:tasks.workunit.client.0.smithi001.stdout:0/95: symlink d18/d1b/d1c/l20 0 2024-02-28T16:30:58.654 INFO:tasks.workunit.client.0.smithi001.stdout:6/45: mkdir d9/de 0 2024-02-28T16:30:58.655 INFO:tasks.workunit.client.0.smithi001.stdout:1/231: truncate d0/f36 1213364 0 2024-02-28T16:30:58.656 INFO:tasks.workunit.client.0.smithi001.stdout:0/96: creat d18/d1b/d1c/f21 x:0 0 0 2024-02-28T16:30:58.658 INFO:tasks.workunit.client.0.smithi001.stdout:3/99: sync 2024-02-28T16:30:58.658 INFO:tasks.workunit.client.0.smithi001.stdout:7/24: sync 2024-02-28T16:30:58.661 INFO:tasks.workunit.client.0.smithi001.stdout:6/46: creat d9/ff x:0 0 0 2024-02-28T16:30:58.667 INFO:tasks.workunit.client.0.smithi001.stdout:5/96: dwrite fa [0,4194304] 0 2024-02-28T16:30:58.673 INFO:tasks.workunit.client.0.smithi001.stdout:8/134: dwrite d4/d7/f10 [0,4194304] 0 2024-02-28T16:30:58.676 INFO:tasks.workunit.client.0.smithi001.stdout:3/100: creat d4/d1e/f22 x:0 0 0 2024-02-28T16:30:58.676 INFO:tasks.workunit.client.0.smithi001.stdout:6/47: mkdir d9/db/d10 0 2024-02-28T16:30:58.680 INFO:tasks.workunit.client.0.smithi001.stdout:6/48: fdatasync f1 0 2024-02-28T16:30:58.681 INFO:tasks.workunit.client.0.smithi001.stdout:0/97: getdents d18/d1b 0 2024-02-28T16:30:58.682 INFO:tasks.workunit.client.0.smithi001.stdout:2/111: dwrite d0/d2/d6/d12/f1e [4194304,4194304] 0 2024-02-28T16:30:58.687 INFO:tasks.workunit.client.0.smithi001.stdout:3/101: getdents d4 0 2024-02-28T16:30:58.692 INFO:tasks.workunit.client.0.smithi001.stdout:1/232: dread d0/d9/d1a/d2b/d26/f37 [0,4194304] 0 2024-02-28T16:30:58.692 INFO:tasks.workunit.client.0.smithi001.stdout:4/70: sync 2024-02-28T16:30:58.694 INFO:tasks.workunit.client.0.smithi001.stdout:9/155: dwrite d5/d27/f2b [0,4194304] 0 2024-02-28T16:30:58.695 INFO:tasks.workunit.client.0.smithi001.stdout:6/49: creat d9/db/d10/f11 x:0 0 0 2024-02-28T16:30:58.695 INFO:tasks.workunit.client.0.smithi001.stdout:3/102: stat d4/d18/l15 0 2024-02-28T16:30:58.696 INFO:tasks.workunit.client.0.smithi001.stdout:2/112: write d0/d2/d6/f23 [881675,64260] 0 2024-02-28T16:30:58.701 INFO:tasks.workunit.client.0.smithi001.stdout:3/103: readlink d4/d18/l15 0 2024-02-28T16:30:58.702 INFO:tasks.workunit.client.0.smithi001.stdout:9/156: mkdir d5/d27/d34 0 2024-02-28T16:30:58.707 INFO:tasks.workunit.client.0.smithi001.stdout:6/50: dread - d9/ff zero size 2024-02-28T16:30:58.708 INFO:tasks.workunit.client.0.smithi001.stdout:4/71: rmdir d3/d5/d7 39 2024-02-28T16:30:58.711 INFO:tasks.workunit.client.0.smithi001.stdout:0/98: truncate f11 358093 0 2024-02-28T16:30:58.711 INFO:tasks.workunit.client.0.smithi001.stdout:3/104: write d4/f17 [325072,81704] 0 2024-02-28T16:30:58.711 INFO:tasks.workunit.client.0.smithi001.stdout:4/72: stat d3/f18 0 2024-02-28T16:30:58.716 INFO:tasks.workunit.client.0.smithi001.stdout:1/233: creat d0/d9/f48 x:0 0 0 2024-02-28T16:30:58.718 INFO:tasks.workunit.client.0.smithi001.stdout:2/113: mknod d0/d2/d6/c25 0 2024-02-28T16:30:58.718 INFO:tasks.workunit.client.0.smithi001.stdout:7/25: dwrite d3/f6 [0,4194304] 0 2024-02-28T16:30:58.721 INFO:tasks.workunit.client.0.smithi001.stdout:6/51: creat d9/db/f12 x:0 0 0 2024-02-28T16:30:58.725 INFO:tasks.workunit.client.0.smithi001.stdout:1/234: truncate d0/f2c 2123475 0 2024-02-28T16:30:58.725 INFO:tasks.workunit.client.0.smithi001.stdout:6/52: readlink l4 0 2024-02-28T16:30:58.726 INFO:tasks.workunit.client.0.smithi001.stdout:8/135: dwrite d4/d7/f21 [0,4194304] 0 2024-02-28T16:30:58.726 INFO:tasks.workunit.client.0.smithi001.stdout:9/157: read d5/f6 [15266,34397] 0 2024-02-28T16:30:58.727 INFO:tasks.workunit.client.0.smithi001.stdout:0/99: write f9 [367048,97819] 0 2024-02-28T16:30:58.727 INFO:tasks.workunit.client.0.smithi001.stdout:5/97: dwrite dc/df/f11 [0,4194304] 0 2024-02-28T16:30:58.728 INFO:tasks.workunit.client.0.smithi001.stdout:7/26: write d3/f6 [4792593,28752] 0 2024-02-28T16:30:58.729 INFO:tasks.workunit.client.0.smithi001.stdout:4/73: symlink d3/l19 0 2024-02-28T16:30:58.730 INFO:tasks.workunit.client.0.smithi001.stdout:6/53: readlink l4 0 2024-02-28T16:30:58.731 INFO:tasks.workunit.client.0.smithi001.stdout:4/74: dread - d3/f18 zero size 2024-02-28T16:30:58.732 INFO:tasks.workunit.client.0.smithi001.stdout:3/105: dread d4/f17 [0,4194304] 0 2024-02-28T16:30:58.732 INFO:tasks.workunit.client.0.smithi001.stdout:2/114: truncate d0/d2/f14 563272 0 2024-02-28T16:30:58.735 INFO:tasks.workunit.client.0.smithi001.stdout:6/54: dread - d9/db/fd zero size 2024-02-28T16:30:58.736 INFO:tasks.workunit.client.0.smithi001.stdout:8/136: chown d4/d7/f19 60930565 1 2024-02-28T16:30:58.737 INFO:tasks.workunit.client.0.smithi001.stdout:3/106: chown d4/c1b 7615 1 2024-02-28T16:30:58.737 INFO:tasks.workunit.client.0.smithi001.stdout:7/27: mknod d3/c7 0 2024-02-28T16:30:58.741 INFO:tasks.workunit.client.0.smithi001.stdout:5/98: link f4 dc/f20 0 2024-02-28T16:30:58.741 INFO:tasks.workunit.client.0.smithi001.stdout:6/55: write d9/db/f12 [765328,109856] 0 2024-02-28T16:30:58.742 INFO:tasks.workunit.client.0.smithi001.stdout:0/100: truncate f13 1033199 0 2024-02-28T16:30:58.746 INFO:tasks.workunit.client.0.smithi001.stdout:7/28: chown d3/l4 7663235 1 2024-02-28T16:30:58.746 INFO:tasks.workunit.client.0.smithi001.stdout:9/158: rename d5/f2a to d5/d2f/d33/f35 0 2024-02-28T16:30:58.747 INFO:tasks.workunit.client.0.smithi001.stdout:4/75: rename d3 to d3/d5/dd/d1a 22 2024-02-28T16:30:58.751 INFO:tasks.workunit.client.0.smithi001.stdout:8/137: unlink d4/d7/d22/f28 0 2024-02-28T16:30:58.751 INFO:tasks.workunit.client.0.smithi001.stdout:9/159: chown d5/d27/f2c 3835 1 2024-02-28T16:30:58.757 INFO:tasks.workunit.client.0.smithi001.stdout:8/138: write d4/d7/f23 [958506,21791] 0 2024-02-28T16:30:58.760 INFO:tasks.workunit.client.0.smithi001.stdout:1/235: dread d0/d9/d1a/d2b/f17 [0,4194304] 0 2024-02-28T16:30:58.771 INFO:tasks.workunit.client.0.smithi001.stdout:7/29: dwrite d3/f6 [4194304,4194304] 0 2024-02-28T16:30:58.772 INFO:tasks.workunit.client.0.smithi001.stdout:5/99: mkdir dc/d1a/d21 0 2024-02-28T16:30:58.773 INFO:tasks.workunit.client.0.smithi001.stdout:9/160: dwrite d5/f30 [0,4194304] 0 2024-02-28T16:30:58.777 INFO:tasks.workunit.client.0.smithi001.stdout:0/101: mkdir d18/d22 0 2024-02-28T16:30:58.780 INFO:tasks.workunit.client.0.smithi001.stdout:1/236: dread d0/d9/d1a/d2b/f17 [0,4194304] 0 2024-02-28T16:30:58.780 INFO:tasks.workunit.client.0.smithi001.stdout:0/102: chown f10 2 1 2024-02-28T16:30:58.783 INFO:tasks.workunit.client.0.smithi001.stdout:1/237: write d0/d9/d1a/d2b/d26/f37 [1547725,116710] 0 2024-02-28T16:30:58.785 INFO:tasks.workunit.client.0.smithi001.stdout:1/238: chown d0/lb 2622 1 2024-02-28T16:30:58.786 INFO:tasks.workunit.client.0.smithi001.stdout:3/107: unlink d4/d6/ca 0 2024-02-28T16:30:58.790 INFO:tasks.workunit.client.0.smithi001.stdout:6/56: creat d9/db/f13 x:0 0 0 2024-02-28T16:30:58.793 INFO:tasks.workunit.client.0.smithi001.stdout:7/30: dread d3/f6 [0,4194304] 0 2024-02-28T16:30:58.797 INFO:tasks.workunit.client.0.smithi001.stdout:8/139: write d4/d7/f1d [804498,27033] 0 2024-02-28T16:30:58.804 INFO:tasks.workunit.client.0.smithi001.stdout:0/103: dwrite fd [0,4194304] 0 2024-02-28T16:30:58.805 INFO:tasks.workunit.client.0.smithi001.stdout:1/239: dread d0/d9/d1a/f33 [0,4194304] 0 2024-02-28T16:30:58.808 INFO:tasks.workunit.client.0.smithi001.stdout:0/104: chown d18/f1d 1834 1 2024-02-28T16:30:58.812 INFO:tasks.workunit.client.0.smithi001.stdout:5/100: mknod dc/d1a/d21/c22 0 2024-02-28T16:30:58.814 INFO:tasks.workunit.client.0.smithi001.stdout:9/161: mknod d5/c36 0 2024-02-28T16:30:58.817 INFO:tasks.workunit.client.0.smithi001.stdout:9/162: chown d5/d27/f2b 188 1 2024-02-28T16:30:58.819 INFO:tasks.workunit.client.0.smithi001.stdout:8/140: dwrite d4/d7/f19 [0,4194304] 0 2024-02-28T16:30:58.820 INFO:tasks.workunit.client.0.smithi001.stdout:6/57: creat d9/de/f14 x:0 0 0 2024-02-28T16:30:58.821 INFO:tasks.workunit.client.0.smithi001.stdout:6/58: chown d9/de 15747 1 2024-02-28T16:30:58.822 INFO:tasks.workunit.client.0.smithi001.stdout:7/31: rename d3/f6 to d3/f8 0 2024-02-28T16:30:58.822 INFO:tasks.workunit.client.0.smithi001.stdout:6/59: write d9/db/f13 [374099,82565] 0 2024-02-28T16:30:58.824 INFO:tasks.workunit.client.0.smithi001.stdout:8/141: write d4/d7/f19 [3386727,15397] 0 2024-02-28T16:30:58.825 INFO:tasks.workunit.client.0.smithi001.stdout:5/101: mknod dc/d1a/c23 0 2024-02-28T16:30:58.826 INFO:tasks.workunit.client.0.smithi001.stdout:6/60: chown d9/db/fd 827 1 2024-02-28T16:30:58.829 INFO:tasks.workunit.client.0.smithi001.stdout:9/163: creat d5/d2f/f37 x:0 0 0 2024-02-28T16:30:58.829 INFO:tasks.workunit.client.0.smithi001.stdout:5/102: read dc/f1d [464811,94477] 0 2024-02-28T16:30:58.829 INFO:tasks.workunit.client.0.smithi001.stdout:1/240: symlink d0/d9/d1a/d1f/l49 0 2024-02-28T16:30:58.831 INFO:tasks.workunit.client.0.smithi001.stdout:5/103: chown dc/d1a/d21 1637 1 2024-02-28T16:30:58.831 INFO:tasks.workunit.client.0.smithi001.stdout:0/105: link fa d18/d1b/f23 0 2024-02-28T16:30:58.832 INFO:tasks.workunit.client.0.smithi001.stdout:1/241: read d0/f28 [2259116,89160] 0 2024-02-28T16:30:58.834 INFO:tasks.workunit.client.0.smithi001.stdout:0/106: chown d18/d1b/d1c/f21 27630 1 2024-02-28T16:30:58.835 INFO:tasks.workunit.client.0.smithi001.stdout:0/107: chown f10 11157743 1 2024-02-28T16:30:58.837 INFO:tasks.workunit.client.0.smithi001.stdout:6/61: mkdir d9/db/d10/d15 0 2024-02-28T16:30:58.837 INFO:tasks.workunit.client.0.smithi001.stdout:4/76: rename d3/d5/f6 to d3/f1b 0 2024-02-28T16:30:58.839 INFO:tasks.workunit.client.0.smithi001.stdout:6/62: dread - d9/db/fd zero size 2024-02-28T16:30:58.839 INFO:tasks.workunit.client.0.smithi001.stdout:5/104: rmdir dc/d12 39 2024-02-28T16:30:58.841 INFO:tasks.workunit.client.0.smithi001.stdout:6/63: write f1 [165499,124340] 0 2024-02-28T16:30:58.842 INFO:tasks.workunit.client.0.smithi001.stdout:8/142: getdents d4/d7/d22/d27 0 2024-02-28T16:30:58.844 INFO:tasks.workunit.client.0.smithi001.stdout:6/64: write d9/db/f12 [768694,126622] 0 2024-02-28T16:30:58.844 INFO:tasks.workunit.client.0.smithi001.stdout:1/242: truncate d0/d9/d1a/d1f/f2e 8937529 0 2024-02-28T16:30:58.846 INFO:tasks.workunit.client.0.smithi001.stdout:0/108: creat d18/d1b/f24 x:0 0 0 2024-02-28T16:30:58.847 INFO:tasks.workunit.client.0.smithi001.stdout:0/109: read fd [4042413,86940] 0 2024-02-28T16:30:58.850 INFO:tasks.workunit.client.0.smithi001.stdout:3/108: write d4/d18/f19 [4850987,99185] 0 2024-02-28T16:30:58.852 INFO:tasks.workunit.client.0.smithi001.stdout:4/77: mkdir d3/d1c 0 2024-02-28T16:30:58.854 INFO:tasks.workunit.client.0.smithi001.stdout:7/32: link d3/c7 d3/c9 0 2024-02-28T16:30:58.858 INFO:tasks.workunit.client.0.smithi001.stdout:8/143: unlink d4/d7/fd 0 2024-02-28T16:30:58.859 INFO:tasks.workunit.client.0.smithi001.stdout:1/243: write d0/f36 [720773,48607] 0 2024-02-28T16:30:58.860 INFO:tasks.workunit.client.0.smithi001.stdout:2/115: sync 2024-02-28T16:30:58.860 INFO:tasks.workunit.client.0.smithi001.stdout:8/144: write d4/d7/f21 [1778021,77890] 0 2024-02-28T16:30:58.861 INFO:tasks.workunit.client.0.smithi001.stdout:8/145: fdatasync d4/d7/f23 0 2024-02-28T16:30:58.866 INFO:tasks.workunit.client.0.smithi001.stdout:3/109: mkdir d4/d6/d7/d23 0 2024-02-28T16:30:58.866 INFO:tasks.workunit.client.0.smithi001.stdout:1/244: write d0/d9/d1a/d2b/d26/f37 [836648,8932] 0 2024-02-28T16:30:58.867 INFO:tasks.workunit.client.0.smithi001.stdout:1/245: chown d0/d9/d1a/d1f/l49 1 1 2024-02-28T16:30:58.868 INFO:tasks.workunit.client.0.smithi001.stdout:1/246: chown d0/d9/d1a/d2b/d26/f34 816024 1 2024-02-28T16:30:58.870 INFO:tasks.workunit.client.0.smithi001.stdout:1/247: chown d0/d9/ff 2 1 2024-02-28T16:30:58.874 INFO:tasks.workunit.client.0.smithi001.stdout:1/248: truncate d0/d9/d1a/d2b/d26/f34 5194825 0 2024-02-28T16:30:58.875 INFO:tasks.workunit.client.0.smithi001.stdout:9/164: rename d5/d27/c29 to d5/c38 0 2024-02-28T16:30:58.878 INFO:tasks.workunit.client.0.smithi001.stdout:4/78: mknod d3/d5/c1d 0 2024-02-28T16:30:58.879 INFO:tasks.workunit.client.0.smithi001.stdout:3/110: creat d4/d1e/f24 x:0 0 0 2024-02-28T16:30:58.882 INFO:tasks.workunit.client.0.smithi001.stdout:7/33: truncate d3/f8 4778381 0 2024-02-28T16:30:58.884 INFO:tasks.workunit.client.0.smithi001.stdout:1/249: write d0/d9/fa [4473542,57022] 0 2024-02-28T16:30:58.885 INFO:tasks.workunit.client.0.smithi001.stdout:9/165: dread - d5/f1a zero size 2024-02-28T16:30:58.886 INFO:tasks.workunit.client.0.smithi001.stdout:1/250: write d0/f2c [2648779,67224] 0 2024-02-28T16:30:58.887 INFO:tasks.workunit.client.0.smithi001.stdout:6/65: rename f6 to d9/db/d10/f16 0 2024-02-28T16:30:58.888 INFO:tasks.workunit.client.0.smithi001.stdout:6/66: fsync f1 0 2024-02-28T16:30:58.889 INFO:tasks.workunit.client.0.smithi001.stdout:6/67: dread - f8 zero size 2024-02-28T16:30:58.890 INFO:tasks.workunit.client.0.smithi001.stdout:4/79: chown d3/d5/d7 162508226 1 2024-02-28T16:30:58.892 INFO:tasks.workunit.client.0.smithi001.stdout:3/111: creat d4/d1e/f25 x:0 0 0 2024-02-28T16:30:58.894 INFO:tasks.workunit.client.0.smithi001.stdout:2/116: rename d0/l9 to d0/l26 0 2024-02-28T16:30:58.895 INFO:tasks.workunit.client.0.smithi001.stdout:7/34: creat d3/fa x:0 0 0 2024-02-28T16:30:58.896 INFO:tasks.workunit.client.0.smithi001.stdout:7/35: truncate d3/fa 382785 0 2024-02-28T16:30:58.898 INFO:tasks.workunit.client.0.smithi001.stdout:7/36: write d3/fa [166225,6110] 0 2024-02-28T16:30:58.899 INFO:tasks.workunit.client.0.smithi001.stdout:2/117: truncate d0/d2/d3/de/dc/f1b 1257774 0 2024-02-28T16:30:58.901 INFO:tasks.workunit.client.0.smithi001.stdout:0/110: rename l16 to d18/d1b/d1c/l25 0 2024-02-28T16:30:58.901 INFO:tasks.workunit.client.0.smithi001.stdout:2/118: fdatasync d0/ff 0 2024-02-28T16:30:58.905 INFO:tasks.workunit.client.0.smithi001.stdout:6/68: symlink d9/l17 0 2024-02-28T16:30:58.906 INFO:tasks.workunit.client.0.smithi001.stdout:0/111: dread - d18/f1e zero size 2024-02-28T16:30:58.906 INFO:tasks.workunit.client.0.smithi001.stdout:1/251: mkdir d0/d9/d1a/d2b/d26/d27/d30/d4a 0 2024-02-28T16:30:58.908 INFO:tasks.workunit.client.0.smithi001.stdout:0/112: dread - d18/d1b/f1f zero size 2024-02-28T16:30:58.917 INFO:tasks.workunit.client.0.smithi001.stdout:7/37: dwrite d3/fa [0,4194304] 0 2024-02-28T16:30:58.919 INFO:tasks.workunit.client.0.smithi001.stdout:7/38: chown d3 7 1 2024-02-28T16:30:58.923 INFO:tasks.workunit.client.0.smithi001.stdout:5/105: rename f3 to dc/d12/f24 0 2024-02-28T16:30:58.925 INFO:tasks.workunit.client.0.smithi001.stdout:5/106: write dc/df/f11 [3748747,89034] 0 2024-02-28T16:30:58.931 INFO:tasks.workunit.client.0.smithi001.stdout:1/252: creat d0/d9/d1a/d1f/f4b x:0 0 0 2024-02-28T16:30:58.936 INFO:tasks.workunit.client.0.smithi001.stdout:7/39: rmdir d3 39 2024-02-28T16:30:58.944 INFO:tasks.workunit.client.0.smithi001.stdout:6/69: creat d9/db/d10/d15/f18 x:0 0 0 2024-02-28T16:30:58.945 INFO:tasks.workunit.client.0.smithi001.stdout:4/80: truncate d3/f12 1199607 0 2024-02-28T16:30:58.945 INFO:tasks.workunit.client.0.smithi001.stdout:1/253: dread d0/f2c [0,4194304] 0 2024-02-28T16:30:58.949 INFO:tasks.workunit.client.0.smithi001.stdout:3/112: truncate d4/d18/f10 933995 0 2024-02-28T16:30:58.955 INFO:tasks.workunit.client.0.smithi001.stdout:9/166: truncate d5/fe 662700 0 2024-02-28T16:30:58.955 INFO:tasks.workunit.client.0.smithi001.stdout:0/113: creat d18/d22/f26 x:0 0 0 2024-02-28T16:30:58.958 INFO:tasks.workunit.client.0.smithi001.stdout:6/70: mkdir d9/db/d19 0 2024-02-28T16:30:58.959 INFO:tasks.workunit.client.0.smithi001.stdout:2/119: truncate d0/d2/d6/f23 929400 0 2024-02-28T16:30:58.962 INFO:tasks.workunit.client.0.smithi001.stdout:5/107: write f7 [3422030,8281] 0 2024-02-28T16:30:58.969 INFO:tasks.workunit.client.0.smithi001.stdout:9/167: mknod d5/d2f/d33/c39 0 2024-02-28T16:30:58.971 INFO:tasks.workunit.client.0.smithi001.stdout:4/81: creat d3/d1c/f1e x:0 0 0 2024-02-28T16:30:58.976 INFO:tasks.workunit.client.0.smithi001.stdout:6/71: write d9/db/d10/f16 [4539174,75262] 0 2024-02-28T16:30:58.977 INFO:tasks.workunit.client.0.smithi001.stdout:9/168: truncate d5/f2d 99191 0 2024-02-28T16:30:58.980 INFO:tasks.workunit.client.0.smithi001.stdout:2/120: creat d0/d2/d6/d12/f27 x:0 0 0 2024-02-28T16:30:58.985 INFO:tasks.workunit.client.0.smithi001.stdout:0/114: dread d18/d1b/f23 [0,4194304] 0 2024-02-28T16:30:58.985 INFO:tasks.workunit.client.0.smithi001.stdout:7/40: dread d3/fa [0,4194304] 0 2024-02-28T16:30:58.985 INFO:tasks.workunit.client.0.smithi001.stdout:1/254: link d0/d9/d1a/d2b/d26/f34 d0/d9/d1a/f4c 0 2024-02-28T16:30:58.990 INFO:tasks.workunit.client.0.smithi001.stdout:0/115: write f12 [647667,25680] 0 2024-02-28T16:30:59.000 INFO:tasks.workunit.client.0.smithi001.stdout:7/41: mknod d3/cb 0 2024-02-28T16:30:59.000 INFO:tasks.workunit.client.0.smithi001.stdout:5/108: dwrite dc/d12/f16 [0,4194304] 0 2024-02-28T16:30:59.000 INFO:tasks.workunit.client.0.smithi001.stdout:7/42: readlink d3/l4 0 2024-02-28T16:30:59.007 INFO:tasks.workunit.client.0.smithi001.stdout:9/169: link d5/f6 d5/d2f/f3a 0 2024-02-28T16:30:59.008 INFO:tasks.workunit.client.0.smithi001.stdout:2/121: rename d0/d2/d3/de/dc/c20 to d0/d2/d3/de/d22/c28 0 2024-02-28T16:30:59.014 INFO:tasks.workunit.client.0.smithi001.stdout:5/109: write dc/d12/f16 [3712020,123563] 0 2024-02-28T16:30:59.016 INFO:tasks.workunit.client.0.smithi001.stdout:9/170: mknod d5/d2f/c3b 0 2024-02-28T16:30:59.017 INFO:tasks.workunit.client.0.smithi001.stdout:7/43: chown d3/f8 9656405 1 2024-02-28T16:30:59.028 INFO:tasks.workunit.client.0.smithi001.stdout:4/82: dwrite d3/d1c/f1e [0,4194304] 0 2024-02-28T16:30:59.031 INFO:tasks.workunit.client.0.smithi001.stdout:9/171: chown d5/fe 52 1 2024-02-28T16:30:59.040 INFO:tasks.workunit.client.0.smithi001.stdout:3/113: dwrite d4/d18/f10 [0,4194304] 0 2024-02-28T16:30:59.041 INFO:tasks.workunit.client.0.smithi001.stdout:4/83: fsync d3/f17 0 2024-02-28T16:30:59.041 INFO:tasks.workunit.client.0.smithi001.stdout:2/122: rename d0/d2/d6/d12/f1e to d0/d2/d3/f29 0 2024-02-28T16:30:59.041 INFO:tasks.workunit.client.0.smithi001.stdout:6/72: dwrite d9/db/f12 [0,4194304] 0 2024-02-28T16:30:59.043 INFO:tasks.workunit.client.0.smithi001.stdout:5/110: rename dc/ld to dc/l25 0 2024-02-28T16:30:59.044 INFO:tasks.workunit.client.0.smithi001.stdout:3/114: rmdir d4 39 2024-02-28T16:30:59.050 INFO:tasks.workunit.client.0.smithi001.stdout:1/255: dwrite d0/d9/ff [4194304,4194304] 0 2024-02-28T16:30:59.053 INFO:tasks.workunit.client.0.smithi001.stdout:9/172: getdents d5/d27/d34 0 2024-02-28T16:30:59.053 INFO:tasks.workunit.client.0.smithi001.stdout:6/73: read d9/db/d10/f16 [3910448,9696] 0 2024-02-28T16:30:59.054 INFO:tasks.workunit.client.0.smithi001.stdout:5/111: chown dc/d12/f16 1181697 1 2024-02-28T16:30:59.057 INFO:tasks.workunit.client.0.smithi001.stdout:6/74: readlink l4 0 2024-02-28T16:30:59.059 INFO:tasks.workunit.client.0.smithi001.stdout:0/116: dwrite f13 [0,4194304] 0 2024-02-28T16:30:59.060 INFO:tasks.workunit.client.0.smithi001.stdout:1/256: chown d0/d9/d1a/d2b/f24 10364 1 2024-02-28T16:30:59.064 INFO:tasks.workunit.client.0.smithi001.stdout:4/84: mkdir d3/d5/d7/d1f 0 2024-02-28T16:30:59.064 INFO:tasks.workunit.client.0.smithi001.stdout:5/112: symlink dc/d1a/l26 0 2024-02-28T16:30:59.065 INFO:tasks.workunit.client.0.smithi001.stdout:7/44: dwrite d3/f8 [0,4194304] 0 2024-02-28T16:30:59.071 INFO:tasks.workunit.client.0.smithi001.stdout:9/173: fsync d5/fb 0 2024-02-28T16:30:59.071 INFO:tasks.workunit.client.0.smithi001.stdout:0/117: symlink d18/d1b/d1c/l27 0 2024-02-28T16:30:59.073 INFO:tasks.workunit.client.0.smithi001.stdout:3/115: getdents d4/d6/d7/d23 0 2024-02-28T16:30:59.079 INFO:tasks.workunit.client.0.smithi001.stdout:2/123: rename d0/d2/d3/de/f18 to d0/f2a 0 2024-02-28T16:30:59.080 INFO:tasks.workunit.client.0.smithi001.stdout:4/85: mkdir d3/d5/d7/d1f/d20 0 2024-02-28T16:30:59.080 INFO:tasks.workunit.client.0.smithi001.stdout:8/146: sync 2024-02-28T16:30:59.081 INFO:tasks.workunit.client.0.smithi001.stdout:0/118: truncate d18/d1b/f24 485668 0 2024-02-28T16:30:59.085 INFO:tasks.workunit.client.0.smithi001.stdout:9/174: rmdir d5 39 2024-02-28T16:30:59.086 INFO:tasks.workunit.client.0.smithi001.stdout:6/75: dwrite d9/db/f12 [0,4194304] 0 2024-02-28T16:30:59.093 INFO:tasks.workunit.client.0.smithi001.stdout:5/113: creat dc/f27 x:0 0 0 2024-02-28T16:30:59.093 INFO:tasks.workunit.client.0.smithi001.stdout:3/116: creat d4/f26 x:0 0 0 2024-02-28T16:30:59.093 INFO:tasks.workunit.client.0.smithi001.stdout:0/119: mknod d18/d22/c28 0 2024-02-28T16:30:59.100 INFO:tasks.workunit.client.0.smithi001.stdout:6/76: mknod d9/db/d10/d15/c1a 0 2024-02-28T16:30:59.105 INFO:tasks.workunit.client.0.smithi001.stdout:1/257: dwrite d0/d9/d1a/d2b/d26/f34 [4194304,4194304] 0 2024-02-28T16:30:59.110 INFO:tasks.workunit.client.0.smithi001.stdout:0/120: symlink d18/d1b/d1c/l29 0 2024-02-28T16:30:59.115 INFO:tasks.workunit.client.0.smithi001.stdout:7/45: truncate d3/f8 4071510 0 2024-02-28T16:30:59.118 INFO:tasks.workunit.client.0.smithi001.stdout:0/121: write d18/d1b/f24 [1520218,77714] 0 2024-02-28T16:30:59.122 INFO:tasks.workunit.client.0.smithi001.stdout:2/124: dwrite d0/d2/d6/f1f [0,4194304] 0 2024-02-28T16:30:59.125 INFO:tasks.workunit.client.0.smithi001.stdout:6/77: creat d9/db/f1b x:0 0 0 2024-02-28T16:30:59.131 INFO:tasks.workunit.client.0.smithi001.stdout:5/114: dwrite dc/d12/f19 [0,4194304] 0 2024-02-28T16:30:59.131 INFO:tasks.workunit.client.0.smithi001.stdout:3/117: fdatasync d4/d18/f10 0 2024-02-28T16:30:59.134 INFO:tasks.workunit.client.0.smithi001.stdout:7/46: read d3/f8 [282301,60062] 0 2024-02-28T16:30:59.136 INFO:tasks.workunit.client.0.smithi001.stdout:6/78: truncate d9/db/f1b 383552 0 2024-02-28T16:30:59.136 INFO:tasks.workunit.client.0.smithi001.stdout:8/147: rename d4/cc to d4/c29 0 2024-02-28T16:30:59.142 INFO:tasks.workunit.client.0.smithi001.stdout:0/122: rename d18/d22/f26 to d18/d1b/f2a 0 2024-02-28T16:30:59.145 INFO:tasks.workunit.client.0.smithi001.stdout:4/86: dwrite d3/f12 [0,4194304] 0 2024-02-28T16:30:59.146 INFO:tasks.workunit.client.0.smithi001.stdout:9/175: dwrite d5/d2f/f37 [0,4194304] 0 2024-02-28T16:30:59.148 INFO:tasks.workunit.client.0.smithi001.stdout:6/79: unlink d9/ff 0 2024-02-28T16:30:59.149 INFO:tasks.workunit.client.0.smithi001.stdout:4/87: chown d3/d5/c1d 6986966 1 2024-02-28T16:30:59.152 INFO:tasks.workunit.client.0.smithi001.stdout:2/125: mknod d0/d2/d3/c2b 0 2024-02-28T16:30:59.157 INFO:tasks.workunit.client.0.smithi001.stdout:3/118: creat d4/d1e/f27 x:0 0 0 2024-02-28T16:30:59.160 INFO:tasks.workunit.client.0.smithi001.stdout:1/258: getdents d0/d9/d1a/d2b 0 2024-02-28T16:30:59.163 INFO:tasks.workunit.client.0.smithi001.stdout:0/123: mkdir d18/d22/d2b 0 2024-02-28T16:30:59.166 INFO:tasks.workunit.client.0.smithi001.stdout:6/80: creat d9/de/f1c x:0 0 0 2024-02-28T16:30:59.168 INFO:tasks.workunit.client.0.smithi001.stdout:6/81: stat d9/de 0 2024-02-28T16:30:59.172 INFO:tasks.workunit.client.0.smithi001.stdout:9/176: dwrite d5/d27/f31 [0,4194304] 0 2024-02-28T16:30:59.174 INFO:tasks.workunit.client.0.smithi001.stdout:4/88: dread d3/f12 [0,4194304] 0 2024-02-28T16:30:59.174 INFO:tasks.workunit.client.0.smithi001.stdout:1/259: creat d0/d9/d1a/d2b/f4d x:0 0 0 2024-02-28T16:30:59.174 INFO:tasks.workunit.client.0.smithi001.stdout:0/124: write fa [3101586,86404] 0 2024-02-28T16:30:59.176 INFO:tasks.workunit.client.0.smithi001.stdout:0/125: fdatasync d18/d1b/f24 0 2024-02-28T16:30:59.182 INFO:tasks.workunit.client.0.smithi001.stdout:7/47: truncate d3/fa 3067601 0 2024-02-28T16:30:59.184 INFO:tasks.workunit.client.0.smithi001.stdout:0/126: read f9 [386989,48444] 0 2024-02-28T16:30:59.187 INFO:tasks.workunit.client.0.smithi001.stdout:5/115: truncate dc/df/f11 4065665 0 2024-02-28T16:30:59.190 INFO:tasks.workunit.client.0.smithi001.stdout:2/126: rename d0/d2/d3/de/dc to d0/d2/d3/d2c 0 2024-02-28T16:30:59.192 INFO:tasks.workunit.client.0.smithi001.stdout:1/260: fsync d0/f28 0 2024-02-28T16:30:59.192 INFO:tasks.workunit.client.0.smithi001.stdout:5/116: chown dc/d12/f19 23748 1 2024-02-28T16:30:59.195 INFO:tasks.workunit.client.0.smithi001.stdout:0/127: truncate d18/d1b/f2a 152145 0 2024-02-28T16:30:59.196 INFO:tasks.workunit.client.0.smithi001.stdout:7/48: fsync d3/f8 0 2024-02-28T16:30:59.197 INFO:tasks.workunit.client.0.smithi001.stdout:8/148: dwrite d4/d7/f10 [4194304,4194304] 0 2024-02-28T16:30:59.198 INFO:tasks.workunit.client.0.smithi001.stdout:0/128: read f13 [602294,52264] 0 2024-02-28T16:30:59.199 INFO:tasks.workunit.client.0.smithi001.stdout:2/127: dread d0/d2/d3/d2c/f1b [0,4194304] 0 2024-02-28T16:30:59.199 INFO:tasks.workunit.client.0.smithi001.stdout:6/82: rename c2 to d9/db/d10/c1d 0 2024-02-28T16:30:59.199 INFO:tasks.workunit.client.0.smithi001.stdout:0/129: fdatasync f0 0 2024-02-28T16:30:59.200 INFO:tasks.workunit.client.0.smithi001.stdout:3/119: mknod d4/d6/d7/d23/c28 0 2024-02-28T16:30:59.200 INFO:tasks.workunit.client.0.smithi001.stdout:0/130: chown lb 19 1 2024-02-28T16:30:59.205 INFO:tasks.workunit.client.0.smithi001.stdout:7/49: symlink d3/lc 0 2024-02-28T16:30:59.217 INFO:tasks.workunit.client.0.smithi001.stdout:8/149: creat d4/d7/f2a x:0 0 0 2024-02-28T16:30:59.220 INFO:tasks.workunit.client.0.smithi001.stdout:6/83: mknod d9/de/c1e 0 2024-02-28T16:30:59.229 INFO:tasks.workunit.client.0.smithi001.stdout:9/177: dwrite d5/f1f [0,4194304] 0 2024-02-28T16:30:59.230 INFO:tasks.workunit.client.0.smithi001.stdout:5/117: creat dc/f28 x:0 0 0 2024-02-28T16:30:59.231 INFO:tasks.workunit.client.0.smithi001.stdout:1/261: link d0/f21 d0/f4e 0 2024-02-28T16:30:59.231 INFO:tasks.workunit.client.0.smithi001.stdout:6/84: write d9/de/f14 [752845,7647] 0 2024-02-28T16:30:59.231 INFO:tasks.workunit.client.0.smithi001.stdout:3/120: dread d4/d6/d7/f20 [0,4194304] 0 2024-02-28T16:30:59.239 INFO:tasks.workunit.client.0.smithi001.stdout:9/178: rmdir d5/d2f/d33 39 2024-02-28T16:30:59.253 INFO:tasks.workunit.client.0.smithi001.stdout:6/85: creat d9/db/d19/f1f x:0 0 0 2024-02-28T16:30:59.254 INFO:tasks.workunit.client.0.smithi001.stdout:2/128: dwrite d0/d2/d3/de/fa [0,4194304] 0 2024-02-28T16:30:59.255 INFO:tasks.workunit.client.0.smithi001.stdout:5/118: dread f5 [0,4194304] 0 2024-02-28T16:30:59.255 INFO:tasks.workunit.client.0.smithi001.stdout:1/262: rename d0/d9/d1a/d2b/c35 to d0/d9/d1a/d2b/d26/d27/c4f 0 2024-02-28T16:30:59.258 INFO:tasks.workunit.client.0.smithi001.stdout:0/131: dread f10 [0,4194304] 0 2024-02-28T16:30:59.258 INFO:tasks.workunit.client.0.smithi001.stdout:8/150: dread d4/f11 [4194304,4194304] 0 2024-02-28T16:30:59.264 INFO:tasks.workunit.client.0.smithi001.stdout:0/132: write fa [2772502,12014] 0 2024-02-28T16:30:59.267 INFO:tasks.workunit.client.0.smithi001.stdout:5/119: creat dc/df/f29 x:0 0 0 2024-02-28T16:30:59.269 INFO:tasks.workunit.client.0.smithi001.stdout:6/86: symlink d9/l20 0 2024-02-28T16:30:59.273 INFO:tasks.workunit.client.0.smithi001.stdout:6/87: dread - f8 zero size 2024-02-28T16:30:59.274 INFO:tasks.workunit.client.0.smithi001.stdout:2/129: symlink d0/d2/l2d 0 2024-02-28T16:30:59.276 INFO:tasks.workunit.client.0.smithi001.stdout:7/50: dwrite d3/fa [0,4194304] 0 2024-02-28T16:30:59.276 INFO:tasks.workunit.client.0.smithi001.stdout:0/133: creat d18/f2c x:0 0 0 2024-02-28T16:30:59.280 INFO:tasks.workunit.client.0.smithi001.stdout:3/121: dwrite d4/d6/d7/f1f [0,4194304] 0 2024-02-28T16:30:59.281 INFO:tasks.workunit.client.0.smithi001.stdout:0/134: chown d18/d1b/d1c/l20 6701 1 2024-02-28T16:30:59.284 INFO:tasks.workunit.client.0.smithi001.stdout:6/88: dread - d9/db/d10/d15/f18 zero size 2024-02-28T16:30:59.290 INFO:tasks.workunit.client.0.smithi001.stdout:3/122: fsync d4/d6/fc 0 2024-02-28T16:30:59.293 INFO:tasks.workunit.client.0.smithi001.stdout:2/130: fsync d0/d2/f14 0 2024-02-28T16:30:59.298 INFO:tasks.workunit.client.0.smithi001.stdout:0/135: write d18/d1b/f2a [197273,130294] 0 2024-02-28T16:30:59.302 INFO:tasks.workunit.client.0.smithi001.stdout:9/179: dread d5/f1d [0,4194304] 0 2024-02-28T16:30:59.303 INFO:tasks.workunit.client.0.smithi001.stdout:7/51: rename d3/c9 to d3/cd 0 2024-02-28T16:30:59.303 INFO:tasks.workunit.client.0.smithi001.stdout:6/89: write d9/db/fc [517060,108053] 0 2024-02-28T16:30:59.304 INFO:tasks.workunit.client.0.smithi001.stdout:5/120: link dc/l18 dc/df/l2a 0 2024-02-28T16:30:59.304 INFO:tasks.workunit.client.0.smithi001.stdout:2/131: truncate d0/f17 593626 0 2024-02-28T16:30:59.311 INFO:tasks.workunit.client.0.smithi001.stdout:3/123: getdents d4 0 2024-02-28T16:30:59.312 INFO:tasks.workunit.client.0.smithi001.stdout:1/263: dwrite d0/d9/d1a/d1f/f4b [0,4194304] 0 2024-02-28T16:30:59.314 INFO:tasks.workunit.client.0.smithi001.stdout:7/52: write d3/f8 [2405820,98675] 0 2024-02-28T16:30:59.316 INFO:tasks.workunit.client.0.smithi001.stdout:8/151: dwrite d4/d7/f1d [0,4194304] 0 2024-02-28T16:30:59.316 INFO:tasks.workunit.client.0.smithi001.stdout:9/180: write d5/f18 [2464118,63350] 0 2024-02-28T16:30:59.321 INFO:tasks.workunit.client.0.smithi001.stdout:6/90: write d9/db/d10/d15/f18 [532370,84106] 0 2024-02-28T16:30:59.322 INFO:tasks.workunit.client.0.smithi001.stdout:7/53: chown d3/l4 0 1 2024-02-28T16:30:59.331 INFO:tasks.workunit.client.0.smithi001.stdout:8/152: write d4/d7/f21 [2890860,125392] 0 2024-02-28T16:30:59.335 INFO:tasks.workunit.client.0.smithi001.stdout:2/132: link d0/d2/d6/f1f d0/d2/d3/d2c/f2e 0 2024-02-28T16:30:59.335 INFO:tasks.workunit.client.0.smithi001.stdout:1/264: creat d0/d9/d1a/d2b/d26/d27/d30/f50 x:0 0 0 2024-02-28T16:30:59.338 INFO:tasks.workunit.client.0.smithi001.stdout:2/133: write d0/d2/f14 [1595322,44212] 0 2024-02-28T16:30:59.340 INFO:tasks.workunit.client.0.smithi001.stdout:8/153: creat d4/d7/f2b x:0 0 0 2024-02-28T16:30:59.344 INFO:tasks.workunit.client.0.smithi001.stdout:5/121: dread fa [0,4194304] 0 2024-02-28T16:30:59.344 INFO:tasks.workunit.client.0.smithi001.stdout:1/265: fdatasync d0/d9/d1a/d2b/f17 0 2024-02-28T16:30:59.345 INFO:tasks.workunit.client.0.smithi001.stdout:2/134: mknod d0/d2/d6/d12/c2f 0 2024-02-28T16:30:59.347 INFO:tasks.workunit.client.0.smithi001.stdout:8/154: mknod d4/d7/c2c 0 2024-02-28T16:30:59.349 INFO:tasks.workunit.client.0.smithi001.stdout:5/122: symlink dc/df/l2b 0 2024-02-28T16:30:59.350 INFO:tasks.workunit.client.0.smithi001.stdout:9/181: dwrite d5/f1f [0,4194304] 0 2024-02-28T16:30:59.359 INFO:tasks.workunit.client.0.smithi001.stdout:8/155: write d4/d7/f2b [750009,121282] 0 2024-02-28T16:30:59.361 INFO:tasks.workunit.client.0.smithi001.stdout:2/135: chown d0/d2/d6/f23 184683894 1 2024-02-28T16:30:59.363 INFO:tasks.workunit.client.0.smithi001.stdout:5/123: truncate dc/f27 1025851 0 2024-02-28T16:30:59.365 INFO:tasks.workunit.client.0.smithi001.stdout:6/91: dwrite d9/db/f13 [0,4194304] 0 2024-02-28T16:30:59.366 INFO:tasks.workunit.client.0.smithi001.stdout:7/54: dwrite d3/f8 [0,4194304] 0 2024-02-28T16:30:59.368 INFO:tasks.workunit.client.0.smithi001.stdout:5/124: chown dc 1 1 2024-02-28T16:30:59.368 INFO:tasks.workunit.client.0.smithi001.stdout:9/182: stat d5/l1c 0 2024-02-28T16:30:59.370 INFO:tasks.workunit.client.0.smithi001.stdout:5/125: readlink dc/l1c 0 2024-02-28T16:30:59.370 INFO:tasks.workunit.client.0.smithi001.stdout:8/156: write d4/d7/f10 [2586504,112578] 0 2024-02-28T16:30:59.377 INFO:tasks.workunit.client.0.smithi001.stdout:2/136: dread - d0/d2/d3/de/d22/f24 zero size 2024-02-28T16:30:59.381 INFO:tasks.workunit.client.0.smithi001.stdout:0/136: dread f0 [0,4194304] 0 2024-02-28T16:30:59.381 INFO:tasks.workunit.client.0.smithi001.stdout:6/92: truncate d9/db/d10/f11 206068 0 2024-02-28T16:30:59.382 INFO:tasks.workunit.client.0.smithi001.stdout:5/126: creat dc/df/f2c x:0 0 0 2024-02-28T16:30:59.383 INFO:tasks.workunit.client.0.smithi001.stdout:8/157: write d4/d7/f1d [1254886,80991] 0 2024-02-28T16:30:59.386 INFO:tasks.workunit.client.0.smithi001.stdout:9/183: creat d5/f3c x:0 0 0 2024-02-28T16:30:59.389 INFO:tasks.workunit.client.0.smithi001.stdout:0/137: stat d18/d1b/d1c/l27 0 2024-02-28T16:30:59.390 INFO:tasks.workunit.client.0.smithi001.stdout:1/266: dwrite d0/d9/d1a/d2b/d26/f37 [0,4194304] 0 2024-02-28T16:30:59.392 INFO:tasks.workunit.client.0.smithi001.stdout:6/93: creat d9/de/f21 x:0 0 0 2024-02-28T16:30:59.393 INFO:tasks.workunit.client.0.smithi001.stdout:8/158: fsync d4/f13 0 2024-02-28T16:30:59.395 INFO:tasks.workunit.client.0.smithi001.stdout:6/94: readlink d9/l20 0 2024-02-28T16:30:59.397 INFO:tasks.workunit.client.0.smithi001.stdout:9/184: fdatasync d5/f21 0 2024-02-28T16:30:59.398 INFO:tasks.workunit.client.0.smithi001.stdout:6/95: truncate f8 544436 0 2024-02-28T16:30:59.400 INFO:tasks.workunit.client.0.smithi001.stdout:6/96: chown d9/de 6 1 2024-02-28T16:30:59.400 INFO:tasks.workunit.client.0.smithi001.stdout:7/55: dread d3/fa [0,4194304] 0 2024-02-28T16:30:59.402 INFO:tasks.workunit.client.0.smithi001.stdout:5/127: rmdir dc/df 39 2024-02-28T16:30:59.402 INFO:tasks.workunit.client.0.smithi001.stdout:6/97: fsync f5 0 2024-02-28T16:30:59.404 INFO:tasks.workunit.client.0.smithi001.stdout:1/267: rmdir d0/d9/d1a/d2b/d26/d45 0 2024-02-28T16:30:59.405 INFO:tasks.workunit.client.0.smithi001.stdout:9/185: chown d5/f9 1681 1 2024-02-28T16:30:59.406 INFO:tasks.workunit.client.0.smithi001.stdout:6/98: write d9/db/fd [413178,127598] 0 2024-02-28T16:30:59.406 INFO:tasks.workunit.client.0.smithi001.stdout:5/128: mkdir dc/d1a/d21/d2d 0 2024-02-28T16:30:59.407 INFO:tasks.workunit.client.0.smithi001.stdout:1/268: read - d0/d9/f48 zero size 2024-02-28T16:30:59.410 INFO:tasks.workunit.client.0.smithi001.stdout:6/99: symlink d9/db/d19/l22 0 2024-02-28T16:30:59.411 INFO:tasks.workunit.client.0.smithi001.stdout:2/137: dread d0/d2/f14 [0,4194304] 0 2024-02-28T16:30:59.411 INFO:tasks.workunit.client.0.smithi001.stdout:5/129: write fa [4739556,14503] 0 2024-02-28T16:30:59.413 INFO:tasks.workunit.client.0.smithi001.stdout:2/138: fdatasync d0/f1d 0 2024-02-28T16:30:59.415 INFO:tasks.workunit.client.0.smithi001.stdout:9/186: rename d5/f9 to d5/d2f/d33/f3d 0 2024-02-28T16:30:59.417 INFO:tasks.workunit.client.0.smithi001.stdout:2/139: read d0/d2/f14 [1040950,50326] 0 2024-02-28T16:30:59.417 INFO:tasks.workunit.client.0.smithi001.stdout:6/100: creat d9/de/f23 x:0 0 0 2024-02-28T16:30:59.420 INFO:tasks.workunit.client.0.smithi001.stdout:2/140: readlink d0/d2/l2d 0 2024-02-28T16:30:59.421 INFO:tasks.workunit.client.0.smithi001.stdout:9/187: mknod d5/c3e 0 2024-02-28T16:30:59.423 INFO:tasks.workunit.client.0.smithi001.stdout:6/101: mkdir d9/db/d10/d24 0 2024-02-28T16:30:59.423 INFO:tasks.workunit.client.0.smithi001.stdout:8/159: dwrite d4/d7/f1d [0,4194304] 0 2024-02-28T16:30:59.424 INFO:tasks.workunit.client.0.smithi001.stdout:2/141: write d0/f17 [882440,128074] 0 2024-02-28T16:30:59.427 INFO:tasks.workunit.client.0.smithi001.stdout:1/269: rename d0/d9/d1a/d2b/d26/l2a to d0/d9/d1a/d2b/l51 0 2024-02-28T16:30:59.433 INFO:tasks.workunit.client.0.smithi001.stdout:9/188: truncate d5/f6 333343 0 2024-02-28T16:30:59.433 INFO:tasks.workunit.client.0.smithi001.stdout:7/56: dwrite d3/fa [4194304,4194304] 0 2024-02-28T16:30:59.439 INFO:tasks.workunit.client.0.smithi001.stdout:1/270: mkdir d0/d9/d1a/d2b/d26/d27/d30/d52 0 2024-02-28T16:30:59.440 INFO:tasks.workunit.client.0.smithi001.stdout:8/160: mknod d4/d7/d22/d27/c2d 0 2024-02-28T16:30:59.443 INFO:tasks.workunit.client.0.smithi001.stdout:7/57: readlink d3/l4 0 2024-02-28T16:30:59.450 INFO:tasks.workunit.client.0.smithi001.stdout:1/271: creat d0/d9/d1a/d2b/d26/d27/d30/d52/f53 x:0 0 0 2024-02-28T16:30:59.451 INFO:tasks.workunit.client.0.smithi001.stdout:7/58: creat d3/fe x:0 0 0 2024-02-28T16:30:59.454 INFO:tasks.workunit.client.0.smithi001.stdout:1/272: dread - d0/d9/f48 zero size 2024-02-28T16:30:59.456 INFO:tasks.workunit.client.0.smithi001.stdout:6/102: truncate d9/db/f13 3942645 0 2024-02-28T16:30:59.456 INFO:tasks.workunit.client.0.smithi001.stdout:4/89: sync 2024-02-28T16:30:59.458 INFO:tasks.workunit.client.0.smithi001.stdout:2/142: dwrite d0/ff [4194304,4194304] 0 2024-02-28T16:30:59.458 INFO:tasks.workunit.client.0.smithi001.stdout:4/90: chown d3/d5/f14 432090 1 2024-02-28T16:30:59.459 INFO:tasks.workunit.client.0.smithi001.stdout:9/189: dwrite d5/f21 [0,4194304] 0 2024-02-28T16:30:59.459 INFO:tasks.workunit.client.0.smithi001.stdout:7/59: link d3/fa d3/ff 0 2024-02-28T16:30:59.468 INFO:tasks.workunit.client.0.smithi001.stdout:6/103: dread d9/db/f1b [0,4194304] 0 2024-02-28T16:30:59.468 INFO:tasks.workunit.client.0.smithi001.stdout:4/91: write d3/f18 [488133,67087] 0 2024-02-28T16:30:59.468 INFO:tasks.workunit.client.0.smithi001.stdout:7/60: creat d3/f10 x:0 0 0 2024-02-28T16:30:59.470 INFO:tasks.workunit.client.0.smithi001.stdout:1/273: write d0/f2c [1435245,50102] 0 2024-02-28T16:30:59.473 INFO:tasks.workunit.client.0.smithi001.stdout:7/61: stat d3/fa 0 2024-02-28T16:30:59.475 INFO:tasks.workunit.client.0.smithi001.stdout:4/92: creat d3/d5/d7/f21 x:0 0 0 2024-02-28T16:30:59.475 INFO:tasks.workunit.client.0.smithi001.stdout:6/104: dread d9/db/d10/f11 [0,4194304] 0 2024-02-28T16:30:59.476 INFO:tasks.workunit.client.0.smithi001.stdout:4/93: fsync d3/d1c/f1e 0 2024-02-28T16:30:59.478 INFO:tasks.workunit.client.0.smithi001.stdout:7/62: creat d3/f11 x:0 0 0 2024-02-28T16:30:59.480 INFO:tasks.workunit.client.0.smithi001.stdout:7/63: rename d3 to d3/d12 22 2024-02-28T16:30:59.482 INFO:tasks.workunit.client.0.smithi001.stdout:6/105: creat d9/db/f25 x:0 0 0 2024-02-28T16:30:59.488 INFO:tasks.workunit.client.0.smithi001.stdout:6/106: rmdir d9/db/d10/d24 0 2024-02-28T16:30:59.493 INFO:tasks.workunit.client.0.smithi001.stdout:6/107: getdents d9/de 0 2024-02-28T16:30:59.494 INFO:tasks.workunit.client.0.smithi001.stdout:4/94: dwrite d3/f17 [0,4194304] 0 2024-02-28T16:30:59.494 INFO:tasks.workunit.client.0.smithi001.stdout:6/108: chown d9/db/d19 14851 1 2024-02-28T16:30:59.495 INFO:tasks.workunit.client.0.smithi001.stdout:2/143: write d0/d2/d3/d2c/f2e [5231819,88037] 0 2024-02-28T16:30:59.501 INFO:tasks.workunit.client.0.smithi001.stdout:6/109: symlink d9/db/d10/d15/l26 0 2024-02-28T16:30:59.507 INFO:tasks.workunit.client.0.smithi001.stdout:7/64: write d3/ff [6581451,98571] 0 2024-02-28T16:30:59.511 INFO:tasks.workunit.client.0.smithi001.stdout:6/110: mknod d9/c27 0 2024-02-28T16:30:59.512 INFO:tasks.workunit.client.0.smithi001.stdout:6/111: stat d9/de/f21 0 2024-02-28T16:30:59.513 INFO:tasks.workunit.client.0.smithi001.stdout:7/65: rename d3/f11 to d3/f13 0 2024-02-28T16:30:59.513 INFO:tasks.workunit.client.0.smithi001.stdout:2/144: link d0/d2/d6/l19 d0/d2/d6/l30 0 2024-02-28T16:30:59.515 INFO:tasks.workunit.client.0.smithi001.stdout:7/66: stat d3/fa 0 2024-02-28T16:30:59.517 INFO:tasks.workunit.client.0.smithi001.stdout:2/145: chown d0/d2/l1a 77052 1 2024-02-28T16:30:59.519 INFO:tasks.workunit.client.0.smithi001.stdout:6/112: mknod d9/db/c28 0 2024-02-28T16:30:59.524 INFO:tasks.workunit.client.0.smithi001.stdout:4/95: dwrite d3/d5/f14 [0,4194304] 0 2024-02-28T16:30:59.525 INFO:tasks.workunit.client.0.smithi001.stdout:6/113: dread d9/db/f1b [0,4194304] 0 2024-02-28T16:30:59.526 INFO:tasks.workunit.client.0.smithi001.stdout:2/146: dread d0/d2/d6/f23 [0,4194304] 0 2024-02-28T16:30:59.528 INFO:tasks.workunit.client.0.smithi001.stdout:7/67: getdents d3 0 2024-02-28T16:30:59.529 INFO:tasks.workunit.client.0.smithi001.stdout:4/96: rmdir d3/d1c 39 2024-02-28T16:30:59.529 INFO:tasks.workunit.client.0.smithi001.stdout:7/68: fdatasync d3/f10 0 2024-02-28T16:30:59.529 INFO:tasks.workunit.client.0.smithi001.stdout:2/147: chown d0/d2/d3/d2c/f10 14 1 2024-02-28T16:30:59.530 INFO:tasks.workunit.client.0.smithi001.stdout:1/274: dread d0/d9/d1a/f4c [0,4194304] 0 2024-02-28T16:30:59.532 INFO:tasks.workunit.client.0.smithi001.stdout:4/97: fdatasync f2 0 2024-02-28T16:30:59.533 INFO:tasks.workunit.client.0.smithi001.stdout:6/114: mkdir d9/d29 0 2024-02-28T16:30:59.534 INFO:tasks.workunit.client.0.smithi001.stdout:7/69: symlink d3/l14 0 2024-02-28T16:30:59.539 INFO:tasks.workunit.client.0.smithi001.stdout:7/70: unlink d3/cb 0 2024-02-28T16:30:59.540 INFO:tasks.workunit.client.0.smithi001.stdout:2/148: unlink d0/d2/d3/de/d22/c28 0 2024-02-28T16:30:59.542 INFO:tasks.workunit.client.0.smithi001.stdout:2/149: write d0/f17 [1334211,12890] 0 2024-02-28T16:30:59.543 INFO:tasks.workunit.client.0.smithi001.stdout:1/275: creat d0/d9/f54 x:0 0 0 2024-02-28T16:30:59.545 INFO:tasks.workunit.client.0.smithi001.stdout:4/98: creat d3/d1c/f22 x:0 0 0 2024-02-28T16:30:59.545 INFO:tasks.workunit.client.0.smithi001.stdout:7/71: fdatasync d3/f13 0 2024-02-28T16:30:59.549 INFO:tasks.workunit.client.0.smithi001.stdout:1/276: creat d0/d9/d1a/d2b/d26/d27/d30/f55 x:0 0 0 2024-02-28T16:30:59.550 INFO:tasks.workunit.client.0.smithi001.stdout:7/72: unlink d3/c7 0 2024-02-28T16:30:59.553 INFO:tasks.workunit.client.0.smithi001.stdout:4/99: creat d3/d5/d7/d1f/d20/f23 x:0 0 0 2024-02-28T16:30:59.554 INFO:tasks.workunit.client.0.smithi001.stdout:6/115: dwrite d9/db/f25 [0,4194304] 0 2024-02-28T16:30:59.558 INFO:tasks.workunit.client.0.smithi001.stdout:1/277: rename d0/d9/d1a/d1f to d0/d9/d56 0 2024-02-28T16:30:59.565 INFO:tasks.workunit.client.0.smithi001.stdout:4/100: getdents d3/d5/d7 0 2024-02-28T16:30:59.571 INFO:tasks.workunit.client.0.smithi001.stdout:1/278: mkdir d0/d9/d1a/d2b/d26/d27/d30/d57 0 2024-02-28T16:30:59.574 INFO:tasks.workunit.client.0.smithi001.stdout:2/150: dwrite d0/d2/d3/d2c/f1b [0,4194304] 0 2024-02-28T16:30:59.577 INFO:tasks.workunit.client.0.smithi001.stdout:4/101: symlink d3/d5/d7/d1f/d20/l24 0 2024-02-28T16:30:59.577 INFO:tasks.workunit.client.0.smithi001.stdout:1/279: read d0/d9/d56/f4b [3495700,25809] 0 2024-02-28T16:30:59.578 INFO:tasks.workunit.client.0.smithi001.stdout:4/102: write d3/f18 [1307651,67682] 0 2024-02-28T16:30:59.579 INFO:tasks.workunit.client.0.smithi001.stdout:7/73: dwrite d3/ff [4194304,4194304] 0 2024-02-28T16:30:59.582 INFO:tasks.workunit.client.0.smithi001.stdout:6/116: dread d9/db/f13 [0,4194304] 0 2024-02-28T16:30:59.584 INFO:tasks.workunit.client.0.smithi001.stdout:4/103: truncate d3/d5/d7/f21 864139 0 2024-02-28T16:30:59.585 INFO:tasks.workunit.client.0.smithi001.stdout:2/151: rename d0/d2/d6/l19 to d0/d2/d3/d2c/l31 0 2024-02-28T16:30:59.595 INFO:tasks.workunit.client.0.smithi001.stdout:7/74: dread - d3/f10 zero size 2024-02-28T16:30:59.596 INFO:tasks.workunit.client.0.smithi001.stdout:1/280: link d0/d9/f54 d0/d9/d1a/d2b/d26/d27/d30/f58 0 2024-02-28T16:30:59.597 INFO:tasks.workunit.client.0.smithi001.stdout:6/117: symlink d9/d29/l2a 0 2024-02-28T16:30:59.601 INFO:tasks.workunit.client.0.smithi001.stdout:2/152: creat d0/f32 x:0 0 0 2024-02-28T16:30:59.602 INFO:tasks.workunit.client.0.smithi001.stdout:6/118: mkdir d9/d29/d2b 0 2024-02-28T16:30:59.603 INFO:tasks.workunit.client.0.smithi001.stdout:6/119: chown d9/de 0 1 2024-02-28T16:30:59.604 INFO:tasks.workunit.client.0.smithi001.stdout:7/75: chown d3/l14 0 1 2024-02-28T16:30:59.605 INFO:tasks.workunit.client.0.smithi001.stdout:4/104: link c1 d3/d5/c25 0 2024-02-28T16:30:59.608 INFO:tasks.workunit.client.0.smithi001.stdout:2/153: rmdir d0/d2/d6/d12 39 2024-02-28T16:30:59.608 INFO:tasks.workunit.client.0.smithi001.stdout:6/120: creat d9/de/f2c x:0 0 0 2024-02-28T16:30:59.613 INFO:tasks.workunit.client.0.smithi001.stdout:4/105: unlink d3/d5/l15 0 2024-02-28T16:30:59.615 INFO:tasks.workunit.client.0.smithi001.stdout:4/106: chown d3/f17 0 1 2024-02-28T16:30:59.616 INFO:tasks.workunit.client.0.smithi001.stdout:6/121: dread d9/db/fc [0,4194304] 0 2024-02-28T16:30:59.619 INFO:tasks.workunit.client.0.smithi001.stdout:4/107: rmdir d3 39 2024-02-28T16:30:59.621 INFO:tasks.workunit.client.0.smithi001.stdout:6/122: write d9/db/fc [600270,5422] 0 2024-02-28T16:30:59.625 INFO:tasks.workunit.client.0.smithi001.stdout:7/76: dread d3/fa [0,4194304] 0 2024-02-28T16:30:59.626 INFO:tasks.workunit.client.0.smithi001.stdout:4/108: read - d3/d5/d7/d1f/d20/f23 zero size 2024-02-28T16:30:59.626 INFO:tasks.workunit.client.0.smithi001.stdout:6/123: unlink d9/de/f21 0 2024-02-28T16:30:59.627 INFO:tasks.workunit.client.0.smithi001.stdout:7/77: dread - d3/f10 zero size 2024-02-28T16:30:59.629 INFO:tasks.workunit.client.0.smithi001.stdout:1/281: dwrite d0/d9/d1a/d2b/d26/d27/d30/f58 [0,4194304] 0 2024-02-28T16:30:59.629 INFO:tasks.workunit.client.0.smithi001.stdout:6/124: write d9/de/f1c [863071,127710] 0 2024-02-28T16:30:59.635 INFO:tasks.workunit.client.0.smithi001.stdout:7/78: chown d3/f8 646218 1 2024-02-28T16:30:59.636 INFO:tasks.workunit.client.0.smithi001.stdout:7/79: truncate d3/f10 693068 0 2024-02-28T16:30:59.640 INFO:tasks.workunit.client.0.smithi001.stdout:6/125: rmdir d9 39 2024-02-28T16:30:59.641 INFO:tasks.workunit.client.0.smithi001.stdout:2/154: dread d0/d2/d6/f15 [0,4194304] 0 2024-02-28T16:30:59.647 INFO:tasks.workunit.client.0.smithi001.stdout:1/282: truncate d0/d9/d56/f2e 4713268 0 2024-02-28T16:30:59.648 INFO:tasks.workunit.client.0.smithi001.stdout:7/80: write d3/fe [670330,566] 0 2024-02-28T16:30:59.649 INFO:tasks.workunit.client.0.smithi001.stdout:6/126: symlink d9/db/l2d 0 2024-02-28T16:30:59.650 INFO:tasks.workunit.client.0.smithi001.stdout:6/127: chown d9/l17 4324845 1 2024-02-28T16:30:59.650 INFO:tasks.workunit.client.0.smithi001.stdout:7/81: chown d3/lc 5101123 1 2024-02-28T16:30:59.652 INFO:tasks.workunit.client.0.smithi001.stdout:2/155: creat d0/f33 x:0 0 0 2024-02-28T16:30:59.653 INFO:tasks.workunit.client.0.smithi001.stdout:6/128: symlink d9/d29/l2e 0 2024-02-28T16:30:59.654 INFO:tasks.workunit.client.0.smithi001.stdout:7/82: write d3/ff [7304275,24291] 0 2024-02-28T16:30:59.656 INFO:tasks.workunit.client.0.smithi001.stdout:7/83: write d3/fa [1256024,92307] 0 2024-02-28T16:30:59.659 INFO:tasks.workunit.client.0.smithi001.stdout:7/84: write d3/f10 [256133,78359] 0 2024-02-28T16:30:59.659 INFO:tasks.workunit.client.0.smithi001.stdout:6/129: truncate d9/db/d10/f11 793247 0 2024-02-28T16:30:59.662 INFO:tasks.workunit.client.0.smithi001.stdout:6/130: fdatasync d9/db/d19/f1f 0 2024-02-28T16:30:59.667 INFO:tasks.workunit.client.0.smithi001.stdout:7/85: getdents d3 0 2024-02-28T16:30:59.668 INFO:tasks.workunit.client.0.smithi001.stdout:7/86: write d3/f8 [1740823,96327] 0 2024-02-28T16:30:59.669 INFO:tasks.workunit.client.0.smithi001.stdout:1/283: dwrite d0/d9/d1a/d2b/d26/f34 [0,4194304] 0 2024-02-28T16:30:59.671 INFO:tasks.workunit.client.0.smithi001.stdout:1/284: chown d0/d9/fa 966609 1 2024-02-28T16:30:59.673 INFO:tasks.workunit.client.0.smithi001.stdout:7/87: write d3/ff [3868480,24157] 0 2024-02-28T16:30:59.674 INFO:tasks.workunit.client.0.smithi001.stdout:1/285: readlink d0/d9/d1a/d2b/d26/d27/d30/l31 0 2024-02-28T16:30:59.675 INFO:tasks.workunit.client.0.smithi001.stdout:7/88: dread - d3/f13 zero size 2024-02-28T16:30:59.677 INFO:tasks.workunit.client.0.smithi001.stdout:1/286: write d0/d9/d1a/d2b/d26/d27/d30/f58 [717326,6356] 0 2024-02-28T16:30:59.678 INFO:tasks.workunit.client.0.smithi001.stdout:6/131: truncate d9/db/fc 431660 0 2024-02-28T16:30:59.678 INFO:tasks.workunit.client.0.smithi001.stdout:7/89: mknod d3/c15 0 2024-02-28T16:30:59.681 INFO:tasks.workunit.client.0.smithi001.stdout:3/124: sync 2024-02-28T16:30:59.683 INFO:tasks.workunit.client.0.smithi001.stdout:3/125: chown d4/d18/f10 479 1 2024-02-28T16:30:59.686 INFO:tasks.workunit.client.0.smithi001.stdout:1/287: link d0/d9/d1a/d2b/d26/d27/d30/f55 d0/d9/d1a/d2b/d26/d27/d30/f59 0 2024-02-28T16:30:59.687 INFO:tasks.workunit.client.0.smithi001.stdout:3/126: mkdir d4/d1e/d29 0 2024-02-28T16:30:59.690 INFO:tasks.workunit.client.0.smithi001.stdout:2/156: dread d0/d2/d3/f5 [0,4194304] 0 2024-02-28T16:30:59.693 INFO:tasks.workunit.client.0.smithi001.stdout:6/132: dread d9/db/f13 [0,4194304] 0 2024-02-28T16:30:59.693 INFO:tasks.workunit.client.0.smithi001.stdout:3/127: truncate d4/d6/fc 1696827 0 2024-02-28T16:30:59.695 INFO:tasks.workunit.client.0.smithi001.stdout:7/90: write d3/f13 [957212,114096] 0 2024-02-28T16:30:59.696 INFO:tasks.workunit.client.0.smithi001.stdout:6/133: rmdir d9 39 2024-02-28T16:30:59.696 INFO:tasks.workunit.client.0.smithi001.stdout:3/128: write d4/d1e/f25 [972122,77895] 0 2024-02-28T16:30:59.698 INFO:tasks.workunit.client.0.smithi001.stdout:3/129: fdatasync d4/d18/f11 0 2024-02-28T16:30:59.699 INFO:tasks.workunit.client.0.smithi001.stdout:7/91: truncate d3/f13 1934666 0 2024-02-28T16:30:59.699 INFO:tasks.workunit.client.0.smithi001.stdout:2/157: rename d0/ff to d0/d2/d3/f34 0 2024-02-28T16:30:59.701 INFO:tasks.workunit.client.0.smithi001.stdout:6/134: unlink c3 0 2024-02-28T16:30:59.703 INFO:tasks.workunit.client.0.smithi001.stdout:1/288: rename d0/d9/d1a/d2b/d26/d27/d30/f50 to d0/d9/d1a/d2b/d26/d27/d30/d52/f5a 0 2024-02-28T16:30:59.704 INFO:tasks.workunit.client.0.smithi001.stdout:3/130: symlink d4/d6/d7/d23/l2a 0 2024-02-28T16:30:59.705 INFO:tasks.workunit.client.0.smithi001.stdout:7/92: chown d3/fa 2699 1 2024-02-28T16:30:59.706 INFO:tasks.workunit.client.0.smithi001.stdout:1/289: write d0/d9/d1a/d2b/d26/f34 [1834329,15694] 0 2024-02-28T16:30:59.706 INFO:tasks.workunit.client.0.smithi001.stdout:7/93: write d3/fa [8926434,128973] 0 2024-02-28T16:30:59.708 INFO:tasks.workunit.client.0.smithi001.stdout:2/158: symlink d0/d2/d3/d2c/l35 0 2024-02-28T16:30:59.709 INFO:tasks.workunit.client.0.smithi001.stdout:2/159: stat d0/l1 0 2024-02-28T16:30:59.712 INFO:tasks.workunit.client.0.smithi001.stdout:7/94: dread d3/f10 [0,4194304] 0 2024-02-28T16:30:59.714 INFO:tasks.workunit.client.0.smithi001.stdout:6/135: creat d9/db/d10/d15/f2f x:0 0 0 2024-02-28T16:30:59.715 INFO:tasks.workunit.client.0.smithi001.stdout:1/290: mknod d0/d9/d1a/d2b/d26/d27/d30/d4a/c5b 0 2024-02-28T16:30:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:59 smithi001 ceph-mon[28991]: pgmap v43: 65 pgs: 65 active+clean; 262 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 85 KiB/s rd, 7.3 MiB/s wr, 131 op/s 2024-02-28T16:30:59.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:59 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:59.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:30:59 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:59.828 INFO:tasks.workunit.client.0.smithi001.stdout:7/95: mknod d3/c16 0 2024-02-28T16:30:59.830 INFO:tasks.workunit.client.0.smithi001.stdout:6/136: rename d9/de/f1c to d9/d29/d2b/f30 0 2024-02-28T16:30:59.831 INFO:tasks.workunit.client.0.smithi001.stdout:2/160: getdents d0 0 2024-02-28T16:30:59.836 INFO:tasks.workunit.client.0.smithi001.stdout:7/96: mkdir d3/d17 0 2024-02-28T16:30:59.841 INFO:tasks.workunit.client.0.smithi001.stdout:6/137: rename d9/de/f23 to d9/db/d10/d15/f31 0 2024-02-28T16:30:59.842 INFO:tasks.workunit.client.0.smithi001.stdout:6/138: fdatasync f5 0 2024-02-28T16:30:59.843 INFO:tasks.workunit.client.0.smithi001.stdout:1/291: getdents d0/d9/d1a/d2b/d26 0 2024-02-28T16:30:59.849 INFO:tasks.workunit.client.0.smithi001.stdout:7/97: getdents d3/d17 0 2024-02-28T16:30:59.853 INFO:tasks.workunit.client.0.smithi001.stdout:2/161: getdents d0/d2/d6 0 2024-02-28T16:30:59.855 INFO:tasks.workunit.client.0.smithi001.stdout:1/292: creat d0/d9/f5c x:0 0 0 2024-02-28T16:30:59.863 INFO:tasks.workunit.client.0.smithi001.stdout:7/98: dread d3/fe [0,4194304] 0 2024-02-28T16:30:59.863 INFO:tasks.workunit.client.0.smithi001.stdout:2/162: mkdir d0/d2/d3/d36 0 2024-02-28T16:30:59.867 INFO:tasks.workunit.client.0.smithi001.stdout:1/293: rename d0/d9/l19 to d0/d9/d56/l5d 0 2024-02-28T16:30:59.871 INFO:tasks.workunit.client.0.smithi001.stdout:6/139: dwrite d9/de/f14 [0,4194304] 0 2024-02-28T16:30:59.872 INFO:tasks.workunit.client.0.smithi001.stdout:7/99: symlink d3/d17/l18 0 2024-02-28T16:30:59.875 INFO:tasks.workunit.client.0.smithi001.stdout:2/163: link d0/d2/fb d0/d2/d3/d2c/f37 0 2024-02-28T16:30:59.883 INFO:tasks.workunit.client.0.smithi001.stdout:6/140: mkdir d9/db/d32 0 2024-02-28T16:30:59.884 INFO:tasks.workunit.client.0.smithi001.stdout:6/141: stat c7 0 2024-02-28T16:30:59.888 INFO:tasks.workunit.client.0.smithi001.stdout:2/164: creat d0/d2/d3/de/d22/f38 x:0 0 0 2024-02-28T16:30:59.890 INFO:tasks.workunit.client.0.smithi001.stdout:7/100: rename d3/f13 to d3/d17/f19 0 2024-02-28T16:30:59.892 INFO:tasks.workunit.client.0.smithi001.stdout:2/165: write d0/fd [3313952,125600] 0 2024-02-28T16:30:59.894 INFO:tasks.workunit.client.0.smithi001.stdout:0/138: sync 2024-02-28T16:30:59.896 INFO:tasks.workunit.client.0.smithi001.stdout:7/101: rmdir d3 39 2024-02-28T16:30:59.897 INFO:tasks.workunit.client.0.smithi001.stdout:2/166: fdatasync d0/d2/d6/f15 0 2024-02-28T16:30:59.898 INFO:tasks.workunit.client.0.smithi001.stdout:0/139: symlink d18/d1b/l2d 0 2024-02-28T16:30:59.901 INFO:tasks.workunit.client.0.smithi001.stdout:2/167: write d0/d2/d3/d2c/f1b [3876226,11121] 0 2024-02-28T16:30:59.909 INFO:tasks.workunit.client.0.smithi001.stdout:1/294: dwrite d0/d9/d1a/d2b/d26/d27/d30/f59 [0,4194304] 0 2024-02-28T16:30:59.912 INFO:tasks.workunit.client.0.smithi001.stdout:2/168: write d0/d2/d3/de/fa [3151887,8418] 0 2024-02-28T16:30:59.918 INFO:tasks.workunit.client.0.smithi001.stdout:6/142: dread f5 [0,4194304] 0 2024-02-28T16:30:59.925 INFO:tasks.workunit.client.0.smithi001.stdout:7/102: dwrite d3/f8 [0,4194304] 0 2024-02-28T16:30:59.927 INFO:tasks.workunit.client.0.smithi001.stdout:0/140: dwrite f11 [0,4194304] 0 2024-02-28T16:30:59.928 INFO:tasks.workunit.client.0.smithi001.stdout:6/143: link d9/db/f25 d9/d29/d2b/f33 0 2024-02-28T16:30:59.933 INFO:tasks.workunit.client.0.smithi001.stdout:6/144: unlink d9/db/l2d 0 2024-02-28T16:30:59.937 INFO:tasks.workunit.client.0.smithi001.stdout:6/145: write f8 [1526897,28963] 0 2024-02-28T16:30:59.938 INFO:tasks.workunit.client.0.smithi001.stdout:4/109: sync 2024-02-28T16:30:59.939 INFO:tasks.workunit.client.0.smithi001.stdout:8/161: sync 2024-02-28T16:30:59.939 INFO:tasks.workunit.client.0.smithi001.stdout:5/130: sync 2024-02-28T16:30:59.939 INFO:tasks.workunit.client.0.smithi001.stdout:9/190: sync 2024-02-28T16:30:59.946 INFO:tasks.workunit.client.0.smithi001.stdout:9/191: stat d5/l28 0 2024-02-28T16:30:59.947 INFO:tasks.workunit.client.0.smithi001.stdout:7/103: link d3/l4 d3/l1a 0 2024-02-28T16:30:59.948 INFO:tasks.workunit.client.0.smithi001.stdout:2/169: dwrite d0/d2/fb [0,4194304] 0 2024-02-28T16:30:59.952 INFO:tasks.workunit.client.0.smithi001.stdout:8/162: read d4/d7/f21 [1383000,56381] 0 2024-02-28T16:30:59.953 INFO:tasks.workunit.client.0.smithi001.stdout:4/110: dread d3/d5/d7/f21 [0,4194304] 0 2024-02-28T16:30:59.955 INFO:tasks.workunit.client.0.smithi001.stdout:0/141: dread d18/d1b/f23 [0,4194304] 0 2024-02-28T16:30:59.955 INFO:tasks.workunit.client.0.smithi001.stdout:7/104: read d3/d17/f19 [1041175,122506] 0 2024-02-28T16:30:59.956 INFO:tasks.workunit.client.0.smithi001.stdout:2/170: write d0/fd [1683564,110710] 0 2024-02-28T16:30:59.957 INFO:tasks.workunit.client.0.smithi001.stdout:8/163: write d4/d7/f1d [2517686,1787] 0 2024-02-28T16:30:59.957 INFO:tasks.workunit.client.0.smithi001.stdout:5/131: rename dc/l18 to dc/d1a/d21/l2e 0 2024-02-28T16:30:59.960 INFO:tasks.workunit.client.0.smithi001.stdout:2/171: chown d0/d2/d3 209938250 1 2024-02-28T16:30:59.962 INFO:tasks.workunit.client.0.smithi001.stdout:5/132: chown dc/f28 79755 1 2024-02-28T16:30:59.964 INFO:tasks.workunit.client.0.smithi001.stdout:0/142: write d18/d1b/d1c/f21 [573251,87565] 0 2024-02-28T16:30:59.968 INFO:tasks.workunit.client.0.smithi001.stdout:6/146: link d9/db/fc d9/db/d10/d15/f34 0 2024-02-28T16:30:59.971 INFO:tasks.workunit.client.0.smithi001.stdout:7/105: write d3/d17/f19 [2177101,13001] 0 2024-02-28T16:30:59.973 INFO:tasks.workunit.client.0.smithi001.stdout:0/143: symlink d18/l2e 0 2024-02-28T16:30:59.974 INFO:tasks.workunit.client.0.smithi001.stdout:0/144: stat f11 0 2024-02-28T16:30:59.976 INFO:tasks.workunit.client.0.smithi001.stdout:1/295: dread d0/d9/d56/f2e [0,4194304] 0 2024-02-28T16:30:59.976 INFO:tasks.workunit.client.0.smithi001.stdout:9/192: rename d5/l25 to d5/l3f 0 2024-02-28T16:30:59.978 INFO:tasks.workunit.client.0.smithi001.stdout:1/296: stat d0/d9/d56 0 2024-02-28T16:30:59.978 INFO:tasks.workunit.client.0.smithi001.stdout:0/145: write f12 [227431,17818] 0 2024-02-28T16:30:59.979 INFO:tasks.workunit.client.0.smithi001.stdout:2/172: creat d0/d2/d3/d36/f39 x:0 0 0 2024-02-28T16:30:59.979 INFO:tasks.workunit.client.0.smithi001.stdout:5/133: rmdir dc/df 39 2024-02-28T16:30:59.982 INFO:tasks.workunit.client.0.smithi001.stdout:7/106: rename d3/fe to d3/d17/f1b 0 2024-02-28T16:30:59.986 INFO:tasks.workunit.client.0.smithi001.stdout:4/111: dwrite d3/d1c/f1e [0,4194304] 0 2024-02-28T16:30:59.986 INFO:tasks.workunit.client.0.smithi001.stdout:2/173: dread - d0/d2/f21 zero size 2024-02-28T16:30:59.993 INFO:tasks.workunit.client.0.smithi001.stdout:2/174: fdatasync d0/fd 0 2024-02-28T16:30:59.993 INFO:tasks.workunit.client.0.smithi001.stdout:9/193: creat d5/d2f/f40 x:0 0 0 2024-02-28T16:30:59.993 INFO:tasks.workunit.client.0.smithi001.stdout:6/147: dread f5 [0,4194304] 0 2024-02-28T16:30:59.993 INFO:tasks.workunit.client.0.smithi001.stdout:1/297: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d52/f5a 0 2024-02-28T16:30:59.995 INFO:tasks.workunit.client.0.smithi001.stdout:7/107: write d3/f10 [1735268,54200] 0 2024-02-28T16:30:59.997 INFO:tasks.workunit.client.0.smithi001.stdout:2/175: dread - d0/d2/f21 zero size 2024-02-28T16:30:59.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:59 smithi066 ceph-mon[36236]: pgmap v43: 65 pgs: 65 active+clean; 262 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 85 KiB/s rd, 7.3 MiB/s wr, 131 op/s 2024-02-28T16:30:59.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:59 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:59.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:30:59 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:30:59.998 INFO:tasks.workunit.client.0.smithi001.stdout:6/148: read f8 [524621,17088] 0 2024-02-28T16:30:59.999 INFO:tasks.workunit.client.0.smithi001.stdout:9/194: chown d5/c1b 11 1 2024-02-28T16:31:00.002 INFO:tasks.workunit.client.0.smithi001.stdout:6/149: stat d9/de 0 2024-02-28T16:31:00.003 INFO:tasks.workunit.client.0.smithi001.stdout:8/164: dwrite d4/f12 [0,4194304] 0 2024-02-28T16:31:00.006 INFO:tasks.workunit.client.0.smithi001.stdout:6/150: chown d9/db/f25 6729 1 2024-02-28T16:31:00.006 INFO:tasks.workunit.client.0.smithi001.stdout:0/146: dwrite f15 [0,4194304] 0 2024-02-28T16:31:00.009 INFO:tasks.workunit.client.0.smithi001.stdout:2/176: mknod d0/d2/d3/d2c/c3a 0 2024-02-28T16:31:00.009 INFO:tasks.workunit.client.0.smithi001.stdout:0/147: chown d18/d22/c28 0 1 2024-02-28T16:31:00.011 INFO:tasks.workunit.client.0.smithi001.stdout:7/108: mkdir d3/d1c 0 2024-02-28T16:31:00.016 INFO:tasks.workunit.client.0.smithi001.stdout:1/298: symlink d0/d9/l5e 0 2024-02-28T16:31:00.017 INFO:tasks.workunit.client.0.smithi001.stdout:7/109: write d3/d17/f19 [3075407,94036] 0 2024-02-28T16:31:00.021 INFO:tasks.workunit.client.0.smithi001.stdout:6/151: truncate f5 2866331 0 2024-02-28T16:31:00.023 INFO:tasks.workunit.client.0.smithi001.stdout:8/165: rename d4/l5 to d4/d7/d22/d27/l2e 0 2024-02-28T16:31:00.029 INFO:tasks.workunit.client.0.smithi001.stdout:7/110: dread d3/d17/f1b [0,4194304] 0 2024-02-28T16:31:00.030 INFO:tasks.workunit.client.0.smithi001.stdout:0/148: rename d18/d1b to d18/d22/d2f 0 2024-02-28T16:31:00.032 INFO:tasks.workunit.client.0.smithi001.stdout:4/112: dwrite d3/f17 [4194304,4194304] 0 2024-02-28T16:31:00.038 INFO:tasks.workunit.client.0.smithi001.stdout:9/195: write d5/f6 [596496,102842] 0 2024-02-28T16:31:00.040 INFO:tasks.workunit.client.0.smithi001.stdout:2/177: dwrite d0/d2/f21 [0,4194304] 0 2024-02-28T16:31:00.044 INFO:tasks.workunit.client.0.smithi001.stdout:7/111: symlink d3/d17/l1d 0 2024-02-28T16:31:00.045 INFO:tasks.workunit.client.0.smithi001.stdout:9/196: write d5/d2f/f3a [619052,58198] 0 2024-02-28T16:31:00.052 INFO:tasks.workunit.client.0.smithi001.stdout:1/299: link d0/d9/d1a/c1e d0/d9/d1a/d2b/d26/d27/d30/d4a/c5f 0 2024-02-28T16:31:00.052 INFO:tasks.workunit.client.0.smithi001.stdout:8/166: write d4/f13 [1478764,83367] 0 2024-02-28T16:31:00.065 INFO:tasks.workunit.client.0.smithi001.stdout:7/112: creat d3/f1e x:0 0 0 2024-02-28T16:31:00.066 INFO:tasks.workunit.client.0.smithi001.stdout:8/167: creat d4/d7/d22/d27/f2f x:0 0 0 2024-02-28T16:31:00.067 INFO:tasks.workunit.client.0.smithi001.stdout:2/178: dread d0/f17 [0,4194304] 0 2024-02-28T16:31:00.067 INFO:tasks.workunit.client.0.smithi001.stdout:1/300: creat d0/d9/d56/f60 x:0 0 0 2024-02-28T16:31:00.070 INFO:tasks.workunit.client.0.smithi001.stdout:6/152: dwrite d9/db/d10/d15/f34 [0,4194304] 0 2024-02-28T16:31:00.074 INFO:tasks.workunit.client.0.smithi001.stdout:8/168: write f3 [1469531,103435] 0 2024-02-28T16:31:00.077 INFO:tasks.workunit.client.0.smithi001.stdout:2/179: creat d0/d2/f3b x:0 0 0 2024-02-28T16:31:00.083 INFO:tasks.workunit.client.0.smithi001.stdout:7/113: getdents d3 0 2024-02-28T16:31:00.083 INFO:tasks.workunit.client.0.smithi001.stdout:4/113: dwrite d3/d5/d7/f21 [0,4194304] 0 2024-02-28T16:31:00.085 INFO:tasks.workunit.client.0.smithi001.stdout:2/180: write d0/d2/d3/d2c/f2e [3812688,70369] 0 2024-02-28T16:31:00.090 INFO:tasks.workunit.client.0.smithi001.stdout:4/114: mknod d3/d5/d7/d1f/d20/c26 0 2024-02-28T16:31:00.094 INFO:tasks.workunit.client.0.smithi001.stdout:9/197: dwrite d5/f1e [0,4194304] 0 2024-02-28T16:31:00.102 INFO:tasks.workunit.client.0.smithi001.stdout:2/181: unlink d0/d2/d3/d2c/f1b 0 2024-02-28T16:31:00.105 INFO:tasks.workunit.client.0.smithi001.stdout:9/198: truncate d5/f6 1543700 0 2024-02-28T16:31:00.109 INFO:tasks.workunit.client.0.smithi001.stdout:7/114: truncate d3/ff 8970476 0 2024-02-28T16:31:00.110 INFO:tasks.workunit.client.0.smithi001.stdout:6/153: dwrite d9/db/d10/f16 [4194304,4194304] 0 2024-02-28T16:31:00.111 INFO:tasks.workunit.client.0.smithi001.stdout:3/131: sync 2024-02-28T16:31:00.114 INFO:tasks.workunit.client.0.smithi001.stdout:2/182: symlink d0/d2/d3/de/d22/l3c 0 2024-02-28T16:31:00.115 INFO:tasks.workunit.client.0.smithi001.stdout:6/154: chown d9/db/f13 0 1 2024-02-28T16:31:00.129 INFO:tasks.workunit.client.0.smithi001.stdout:2/183: symlink d0/d2/d3/l3d 0 2024-02-28T16:31:00.130 INFO:tasks.workunit.client.0.smithi001.stdout:4/115: dwrite d3/f12 [0,4194304] 0 2024-02-28T16:31:00.190 INFO:tasks.workunit.client.0.smithi001.stdout:7/115: link d3/ff d3/f1f 0 2024-02-28T16:31:00.192 INFO:tasks.workunit.client.0.smithi001.stdout:6/155: link d9/db/d10/c1d d9/d29/c35 0 2024-02-28T16:31:00.193 INFO:tasks.workunit.client.0.smithi001.stdout:2/184: truncate d0/fd 1484914 0 2024-02-28T16:31:00.198 INFO:tasks.workunit.client.0.smithi001.stdout:6/156: write d9/db/fd [534760,96038] 0 2024-02-28T16:31:00.198 INFO:tasks.workunit.client.0.smithi001.stdout:9/199: dwrite d5/fe [0,4194304] 0 2024-02-28T16:31:00.199 INFO:tasks.workunit.client.0.smithi001.stdout:7/116: getdents d3/d1c 0 2024-02-28T16:31:00.202 INFO:tasks.workunit.client.0.smithi001.stdout:8/169: dread d4/d7/f21 [0,4194304] 0 2024-02-28T16:31:00.205 INFO:tasks.workunit.client.0.smithi001.stdout:4/116: dwrite d3/d5/d7/f21 [0,4194304] 0 2024-02-28T16:31:00.206 INFO:tasks.workunit.client.0.smithi001.stdout:6/157: creat d9/d29/d2b/f36 x:0 0 0 2024-02-28T16:31:00.207 INFO:tasks.workunit.client.0.smithi001.stdout:4/117: fsync d3/f12 0 2024-02-28T16:31:00.210 INFO:tasks.workunit.client.0.smithi001.stdout:8/170: creat d4/d7/d22/d27/f30 x:0 0 0 2024-02-28T16:31:00.211 INFO:tasks.workunit.client.0.smithi001.stdout:6/158: fdatasync d9/db/d10/d15/f34 0 2024-02-28T16:31:00.213 INFO:tasks.workunit.client.0.smithi001.stdout:9/200: creat d5/d2f/f41 x:0 0 0 2024-02-28T16:31:00.215 INFO:tasks.workunit.client.0.smithi001.stdout:4/118: unlink d3/d1c/f1e 0 2024-02-28T16:31:00.219 INFO:tasks.workunit.client.0.smithi001.stdout:2/185: truncate d0/d2/d3/d2c/f37 1241604 0 2024-02-28T16:31:00.220 INFO:tasks.workunit.client.0.smithi001.stdout:6/159: unlink c7 0 2024-02-28T16:31:00.223 INFO:tasks.workunit.client.0.smithi001.stdout:9/201: mkdir d5/d27/d42 0 2024-02-28T16:31:00.225 INFO:tasks.workunit.client.0.smithi001.stdout:2/186: dread d0/f17 [0,4194304] 0 2024-02-28T16:31:00.226 INFO:tasks.workunit.client.0.smithi001.stdout:9/202: fdatasync d5/d27/f2c 0 2024-02-28T16:31:00.226 INFO:tasks.workunit.client.0.smithi001.stdout:8/171: symlink d4/l31 0 2024-02-28T16:31:00.227 INFO:tasks.workunit.client.0.smithi001.stdout:2/187: stat d0/d2 0 2024-02-28T16:31:00.229 INFO:tasks.workunit.client.0.smithi001.stdout:6/160: rename d9/d29/d2b/f36 to d9/db/d19/f37 0 2024-02-28T16:31:00.229 INFO:tasks.workunit.client.0.smithi001.stdout:7/117: dwrite d3/fa [4194304,4194304] 0 2024-02-28T16:31:00.232 INFO:tasks.workunit.client.0.smithi001.stdout:4/119: truncate d3/f12 4051416 0 2024-02-28T16:31:00.233 INFO:tasks.workunit.client.0.smithi001.stdout:8/172: rmdir d4 39 2024-02-28T16:31:00.235 INFO:tasks.workunit.client.0.smithi001.stdout:6/161: rmdir d9/db/d10/d15 39 2024-02-28T16:31:00.238 INFO:tasks.workunit.client.0.smithi001.stdout:7/118: dread d3/f10 [0,4194304] 0 2024-02-28T16:31:00.239 INFO:tasks.workunit.client.0.smithi001.stdout:8/173: fsync d4/d7/f1d 0 2024-02-28T16:31:00.241 INFO:tasks.workunit.client.0.smithi001.stdout:6/162: mknod d9/db/c38 0 2024-02-28T16:31:00.245 INFO:tasks.workunit.client.0.smithi001.stdout:2/188: rename d0/d2/d6/d12/c2f to d0/d2/d3/c3e 0 2024-02-28T16:31:00.247 INFO:tasks.workunit.client.0.smithi001.stdout:7/119: mknod d3/d17/c20 0 2024-02-28T16:31:00.247 INFO:tasks.workunit.client.0.smithi001.stdout:5/134: sync 2024-02-28T16:31:00.251 INFO:tasks.workunit.client.0.smithi001.stdout:9/203: getdents d5 0 2024-02-28T16:31:00.253 INFO:tasks.workunit.client.0.smithi001.stdout:2/189: dread - d0/d2/d3/d36/f39 zero size 2024-02-28T16:31:00.253 INFO:tasks.workunit.client.0.smithi001.stdout:7/120: chown d3/c15 9441 1 2024-02-28T16:31:00.254 INFO:tasks.workunit.client.0.smithi001.stdout:8/174: write d4/d7/f21 [1235167,95610] 0 2024-02-28T16:31:00.257 INFO:tasks.workunit.client.0.smithi001.stdout:4/120: rename d3/f18 to d3/d5/dd/f27 0 2024-02-28T16:31:00.260 INFO:tasks.workunit.client.0.smithi001.stdout:7/121: creat d3/f21 x:0 0 0 2024-02-28T16:31:00.262 INFO:tasks.workunit.client.0.smithi001.stdout:5/135: rmdir dc/df 39 2024-02-28T16:31:00.264 INFO:tasks.workunit.client.0.smithi001.stdout:4/121: creat d3/d5/dd/f28 x:0 0 0 2024-02-28T16:31:00.265 INFO:tasks.workunit.client.0.smithi001.stdout:9/204: mknod d5/d27/d42/c43 0 2024-02-28T16:31:00.268 INFO:tasks.workunit.client.0.smithi001.stdout:4/122: read d3/f1b [2092933,55733] 0 2024-02-28T16:31:00.269 INFO:tasks.workunit.client.0.smithi001.stdout:7/122: fsync d3/ff 0 2024-02-28T16:31:00.269 INFO:tasks.workunit.client.0.smithi001.stdout:9/205: symlink d5/d2f/d33/l44 0 2024-02-28T16:31:00.271 INFO:tasks.workunit.client.0.smithi001.stdout:8/175: getdents d4 0 2024-02-28T16:31:00.273 INFO:tasks.workunit.client.0.smithi001.stdout:9/206: write d5/f1a [575549,39758] 0 2024-02-28T16:31:00.274 INFO:tasks.workunit.client.0.smithi001.stdout:8/176: fdatasync d4/d7/f21 0 2024-02-28T16:31:00.278 INFO:tasks.workunit.client.0.smithi001.stdout:6/163: truncate d9/db/d10/f16 649314 0 2024-02-28T16:31:00.279 INFO:tasks.workunit.client.0.smithi001.stdout:9/207: mkdir d5/d45 0 2024-02-28T16:31:00.280 INFO:tasks.workunit.client.0.smithi001.stdout:7/123: dread d3/f10 [0,4194304] 0 2024-02-28T16:31:00.288 INFO:tasks.workunit.client.0.smithi001.stdout:7/124: creat d3/f22 x:0 0 0 2024-02-28T16:31:00.290 INFO:tasks.workunit.client.0.smithi001.stdout:2/190: write d0/d2/fb [2207350,129524] 0 2024-02-28T16:31:00.296 INFO:tasks.workunit.client.0.smithi001.stdout:8/177: dwrite d4/d7/f20 [0,4194304] 0 2024-02-28T16:31:00.299 INFO:tasks.workunit.client.0.smithi001.stdout:9/208: rename d5/c36 to d5/d45/c46 0 2024-02-28T16:31:00.299 INFO:tasks.workunit.client.0.smithi001.stdout:8/178: write d4/d7/f19 [9165142,981] 0 2024-02-28T16:31:00.301 INFO:tasks.workunit.client.0.smithi001.stdout:2/191: mknod d0/d2/d3/de/c3f 0 2024-02-28T16:31:00.305 INFO:tasks.workunit.client.0.smithi001.stdout:8/179: mkdir d4/d32 0 2024-02-28T16:31:00.305 INFO:tasks.workunit.client.0.smithi001.stdout:4/123: dread f2 [0,4194304] 0 2024-02-28T16:31:00.309 INFO:tasks.workunit.client.0.smithi001.stdout:5/136: dread dc/d12/f24 [0,4194304] 0 2024-02-28T16:31:00.309 INFO:tasks.workunit.client.0.smithi001.stdout:7/125: rename d3/l4 to d3/d1c/l23 0 2024-02-28T16:31:00.311 INFO:tasks.workunit.client.0.smithi001.stdout:4/124: fdatasync d3/fa 0 2024-02-28T16:31:00.312 INFO:tasks.workunit.client.0.smithi001.stdout:7/126: write d3/f8 [59539,84387] 0 2024-02-28T16:31:00.314 INFO:tasks.workunit.client.0.smithi001.stdout:5/137: fdatasync dc/f20 0 2024-02-28T16:31:00.317 INFO:tasks.workunit.client.0.smithi001.stdout:7/127: mknod d3/d1c/c24 0 2024-02-28T16:31:00.318 INFO:tasks.workunit.client.0.smithi001.stdout:4/125: symlink d3/d5/d7/d1f/l29 0 2024-02-28T16:31:00.320 INFO:tasks.workunit.client.0.smithi001.stdout:4/126: chown d3/fa 457796 1 2024-02-28T16:31:00.321 INFO:tasks.workunit.client.0.smithi001.stdout:6/164: dread f1 [0,4194304] 0 2024-02-28T16:31:00.324 INFO:tasks.workunit.client.0.smithi001.stdout:8/180: fdatasync d4/d7/f2a 0 2024-02-28T16:31:00.325 INFO:tasks.workunit.client.0.smithi001.stdout:6/165: mkdir d9/d39 0 2024-02-28T16:31:00.327 INFO:tasks.workunit.client.0.smithi001.stdout:8/181: write d4/d7/f21 [4704494,1812] 0 2024-02-28T16:31:00.328 INFO:tasks.workunit.client.0.smithi001.stdout:6/166: write d9/de/f14 [1286025,39168] 0 2024-02-28T16:31:00.333 INFO:tasks.workunit.client.0.smithi001.stdout:6/167: write d9/db/fd [185446,59852] 0 2024-02-28T16:31:00.334 INFO:tasks.workunit.client.0.smithi001.stdout:8/182: symlink d4/d7/d22/d27/l33 0 2024-02-28T16:31:00.335 INFO:tasks.workunit.client.0.smithi001.stdout:7/128: dread d3/f8 [0,4194304] 0 2024-02-28T16:31:00.337 INFO:tasks.workunit.client.0.smithi001.stdout:6/168: rmdir d9/db/d10 39 2024-02-28T16:31:00.338 INFO:tasks.workunit.client.0.smithi001.stdout:4/127: dread f2 [0,4194304] 0 2024-02-28T16:31:00.339 INFO:tasks.workunit.client.0.smithi001.stdout:9/209: dwrite d5/d2f/f3a [0,4194304] 0 2024-02-28T16:31:00.341 INFO:tasks.workunit.client.0.smithi001.stdout:4/128: write f2 [914248,31059] 0 2024-02-28T16:31:00.343 INFO:tasks.workunit.client.0.smithi001.stdout:7/129: unlink d3/d17/l1d 0 2024-02-28T16:31:00.344 INFO:tasks.workunit.client.0.smithi001.stdout:8/183: rename d4/l1e to d4/d32/l34 0 2024-02-28T16:31:00.346 INFO:tasks.workunit.client.0.smithi001.stdout:9/210: chown d5/f21 285 1 2024-02-28T16:31:00.346 INFO:tasks.workunit.client.0.smithi001.stdout:4/129: creat d3/d5/dd/f2a x:0 0 0 2024-02-28T16:31:00.350 INFO:tasks.workunit.client.0.smithi001.stdout:6/169: rmdir d9/db/d32 0 2024-02-28T16:31:00.353 INFO:tasks.workunit.client.0.smithi001.stdout:9/211: rename d5/d2f to d5/d47 0 2024-02-28T16:31:00.354 INFO:tasks.workunit.client.0.smithi001.stdout:6/170: write d9/d29/d2b/f30 [671592,125715] 0 2024-02-28T16:31:00.361 INFO:tasks.workunit.client.0.smithi001.stdout:6/171: creat d9/d39/f3a x:0 0 0 2024-02-28T16:31:00.364 INFO:tasks.workunit.client.0.smithi001.stdout:6/172: dread - d9/db/d19/f1f zero size 2024-02-28T16:31:00.364 INFO:tasks.workunit.client.0.smithi001.stdout:4/130: dwrite d3/d5/dd/f2a [0,4194304] 0 2024-02-28T16:31:00.365 INFO:tasks.workunit.client.0.smithi001.stdout:7/130: truncate d3/d17/f19 3161374 0 2024-02-28T16:31:00.368 INFO:tasks.workunit.client.0.smithi001.stdout:8/184: rmdir d4/d7/d22 39 2024-02-28T16:31:00.368 INFO:tasks.workunit.client.0.smithi001.stdout:7/131: truncate d3/d17/f1b 799187 0 2024-02-28T16:31:00.369 INFO:tasks.workunit.client.0.smithi001.stdout:8/185: fsync d4/d7/f19 0 2024-02-28T16:31:00.372 INFO:tasks.workunit.client.0.smithi001.stdout:9/212: chown d5/d27/d42 216710 1 2024-02-28T16:31:00.379 INFO:tasks.workunit.client.0.smithi001.stdout:7/132: mknod d3/d17/c25 0 2024-02-28T16:31:00.379 INFO:tasks.workunit.client.0.smithi001.stdout:8/186: unlink d4/d7/l15 0 2024-02-28T16:31:00.381 INFO:tasks.workunit.client.0.smithi001.stdout:7/133: chown d3/d1c 45 1 2024-02-28T16:31:00.382 INFO:tasks.workunit.client.0.smithi001.stdout:9/213: write d5/d47/f40 [638076,60815] 0 2024-02-28T16:31:00.384 INFO:tasks.workunit.client.0.smithi001.stdout:8/187: dread - d4/d7/d22/d27/f30 zero size 2024-02-28T16:31:00.384 INFO:tasks.workunit.client.0.smithi001.stdout:7/134: truncate d3/f1e 840880 0 2024-02-28T16:31:00.388 INFO:tasks.workunit.client.0.smithi001.stdout:4/131: write d3/f12 [2784134,78103] 0 2024-02-28T16:31:00.389 INFO:tasks.workunit.client.0.smithi001.stdout:9/214: mkdir d5/d27/d42/d48 0 2024-02-28T16:31:00.390 INFO:tasks.workunit.client.0.smithi001.stdout:8/188: chown d4/d7/f2b 3441 1 2024-02-28T16:31:00.393 INFO:tasks.workunit.client.0.smithi001.stdout:6/173: dwrite d9/d29/d2b/f33 [4194304,4194304] 0 2024-02-28T16:31:00.393 INFO:tasks.workunit.client.0.smithi001.stdout:7/135: rename d3/lc to d3/l26 0 2024-02-28T16:31:00.395 INFO:tasks.workunit.client.0.smithi001.stdout:7/136: stat d3/f1f 0 2024-02-28T16:31:00.397 INFO:tasks.workunit.client.0.smithi001.stdout:7/137: truncate d3/d17/f1b 1759195 0 2024-02-28T16:31:00.398 INFO:tasks.workunit.client.0.smithi001.stdout:4/132: symlink d3/d5/d7/d1f/d20/l2b 0 2024-02-28T16:31:00.399 INFO:tasks.workunit.client.0.smithi001.stdout:6/174: creat d9/d29/f3b x:0 0 0 2024-02-28T16:31:00.403 INFO:tasks.workunit.client.0.smithi001.stdout:4/133: creat d3/d5/dd/f2c x:0 0 0 2024-02-28T16:31:00.408 INFO:tasks.workunit.client.0.smithi001.stdout:8/189: fsync d4/d7/f23 0 2024-02-28T16:31:00.409 INFO:tasks.workunit.client.0.smithi001.stdout:6/175: mknod d9/d29/d2b/c3c 0 2024-02-28T16:31:00.411 INFO:tasks.workunit.client.0.smithi001.stdout:7/138: truncate d3/ff 5008955 0 2024-02-28T16:31:00.412 INFO:tasks.workunit.client.0.smithi001.stdout:8/190: write f1 [1774555,51487] 0 2024-02-28T16:31:00.416 INFO:tasks.workunit.client.0.smithi001.stdout:7/139: creat d3/f27 x:0 0 0 2024-02-28T16:31:00.417 INFO:tasks.workunit.client.0.smithi001.stdout:6/176: symlink d9/db/l3d 0 2024-02-28T16:31:00.417 INFO:tasks.workunit.client.0.smithi001.stdout:9/215: dwrite d5/d47/f41 [0,4194304] 0 2024-02-28T16:31:00.419 INFO:tasks.workunit.client.0.smithi001.stdout:7/140: fdatasync d3/f8 0 2024-02-28T16:31:00.421 INFO:tasks.workunit.client.0.smithi001.stdout:8/191: write d4/f11 [7006559,91746] 0 2024-02-28T16:31:00.422 INFO:tasks.workunit.client.0.smithi001.stdout:9/216: rename d5/d47/d33 to d5/d47/d33/d49 22 2024-02-28T16:31:00.424 INFO:tasks.workunit.client.0.smithi001.stdout:4/134: dread d3/f17 [0,4194304] 0 2024-02-28T16:31:00.426 INFO:tasks.workunit.client.0.smithi001.stdout:7/141: creat d3/f28 x:0 0 0 2024-02-28T16:31:00.430 INFO:tasks.workunit.client.0.smithi001.stdout:7/142: symlink d3/d1c/l29 0 2024-02-28T16:31:00.430 INFO:tasks.workunit.client.0.smithi001.stdout:9/217: chown d5/c1b 113 1 2024-02-28T16:31:00.440 INFO:tasks.workunit.client.0.smithi001.stdout:7/143: creat d3/d17/f2a x:0 0 0 2024-02-28T16:31:00.440 INFO:tasks.workunit.client.0.smithi001.stdout:8/192: creat d4/d7/f35 x:0 0 0 2024-02-28T16:31:00.440 INFO:tasks.workunit.client.0.smithi001.stdout:9/218: chown d5/c16 4049 1 2024-02-28T16:31:00.440 INFO:tasks.workunit.client.0.smithi001.stdout:7/144: dread - d3/d17/f2a zero size 2024-02-28T16:31:00.440 INFO:tasks.workunit.client.0.smithi001.stdout:9/219: creat d5/d45/f4a x:0 0 0 2024-02-28T16:31:00.440 INFO:tasks.workunit.client.0.smithi001.stdout:8/193: mknod d4/d32/c36 0 2024-02-28T16:31:00.441 INFO:tasks.workunit.client.0.smithi001.stdout:7/145: truncate d3/f10 2121363 0 2024-02-28T16:31:00.442 INFO:tasks.workunit.client.0.smithi001.stdout:1/301: sync 2024-02-28T16:31:00.442 INFO:tasks.workunit.client.0.smithi001.stdout:0/149: sync 2024-02-28T16:31:00.444 INFO:tasks.workunit.client.0.smithi001.stdout:9/220: truncate d5/f1e 4601317 0 2024-02-28T16:31:00.444 INFO:tasks.workunit.client.0.smithi001.stdout:8/194: mknod d4/d7/d22/c37 0 2024-02-28T16:31:00.445 INFO:tasks.workunit.client.0.smithi001.stdout:7/146: mknod d3/d1c/c2b 0 2024-02-28T16:31:00.448 INFO:tasks.workunit.client.0.smithi001.stdout:7/147: dread - d3/f27 zero size 2024-02-28T16:31:00.449 INFO:tasks.workunit.client.0.smithi001.stdout:0/150: rename d18/d22/d2f/f2a to d18/d22/d2f/d1c/f30 0 2024-02-28T16:31:00.450 INFO:tasks.workunit.client.0.smithi001.stdout:7/148: rename d3 to d3/d17/d2c 22 2024-02-28T16:31:00.451 INFO:tasks.workunit.client.0.smithi001.stdout:0/151: chown fd 11546680 1 2024-02-28T16:31:00.453 INFO:tasks.workunit.client.0.smithi001.stdout:6/177: dwrite d9/db/d10/f16 [0,4194304] 0 2024-02-28T16:31:00.455 INFO:tasks.workunit.client.0.smithi001.stdout:1/302: creat d0/d9/f61 x:0 0 0 2024-02-28T16:31:00.455 INFO:tasks.workunit.client.0.smithi001.stdout:7/149: readlink d3/l1a 0 2024-02-28T16:31:00.458 INFO:tasks.workunit.client.0.smithi001.stdout:9/221: mknod d5/d27/d42/d48/c4b 0 2024-02-28T16:31:00.458 INFO:tasks.workunit.client.0.smithi001.stdout:8/195: truncate d4/d7/f23 357859 0 2024-02-28T16:31:00.459 INFO:tasks.workunit.client.0.smithi001.stdout:0/152: mkdir d18/d31 0 2024-02-28T16:31:00.459 INFO:tasks.workunit.client.0.smithi001.stdout:6/178: mkdir d9/d29/d3e 0 2024-02-28T16:31:00.460 INFO:tasks.workunit.client.0.smithi001.stdout:9/222: write d5/f2d [568453,43594] 0 2024-02-28T16:31:00.462 INFO:tasks.workunit.client.0.smithi001.stdout:7/150: creat d3/d17/f2d x:0 0 0 2024-02-28T16:31:00.463 INFO:tasks.workunit.client.0.smithi001.stdout:8/196: truncate d4/d7/f2a 904134 0 2024-02-28T16:31:00.464 INFO:tasks.workunit.client.0.smithi001.stdout:1/303: symlink d0/d9/d1a/d2b/d26/l62 0 2024-02-28T16:31:00.466 INFO:tasks.workunit.client.0.smithi001.stdout:1/304: write d0/f36 [695234,10311] 0 2024-02-28T16:31:00.470 INFO:tasks.workunit.client.0.smithi001.stdout:7/151: mknod d3/d17/c2e 0 2024-02-28T16:31:00.471 INFO:tasks.workunit.client.0.smithi001.stdout:7/152: fsync d3/f21 0 2024-02-28T16:31:00.472 INFO:tasks.workunit.client.0.smithi001.stdout:9/223: creat d5/d27/d42/d48/f4c x:0 0 0 2024-02-28T16:31:00.475 INFO:tasks.workunit.client.0.smithi001.stdout:1/305: unlink d0/d9/ce 0 2024-02-28T16:31:00.476 INFO:tasks.workunit.client.0.smithi001.stdout:0/153: symlink d18/d31/l32 0 2024-02-28T16:31:00.479 INFO:tasks.workunit.client.0.smithi001.stdout:7/153: unlink d3/d1c/c2b 0 2024-02-28T16:31:00.480 INFO:tasks.workunit.client.0.smithi001.stdout:0/154: rmdir d18/d31 39 2024-02-28T16:31:00.480 INFO:tasks.workunit.client.0.smithi001.stdout:6/179: link d9/c27 d9/d29/d2b/c3f 0 2024-02-28T16:31:00.481 INFO:tasks.workunit.client.0.smithi001.stdout:0/155: read f10 [847986,9189] 0 2024-02-28T16:31:00.482 INFO:tasks.workunit.client.0.smithi001.stdout:9/224: mkdir d5/d4d 0 2024-02-28T16:31:00.484 INFO:tasks.workunit.client.0.smithi001.stdout:0/156: unlink l5 0 2024-02-28T16:31:00.489 INFO:tasks.workunit.client.0.smithi001.stdout:6/180: creat d9/db/d10/d15/f40 x:0 0 0 2024-02-28T16:31:00.489 INFO:tasks.workunit.client.0.smithi001.stdout:1/306: getdents d0/d9/d1a/d2b/d26/d27/d30/d4a 0 2024-02-28T16:31:00.494 INFO:tasks.workunit.client.0.smithi001.stdout:6/181: rename d9/d29/l2a to d9/db/d19/l41 0 2024-02-28T16:31:00.498 INFO:tasks.workunit.client.0.smithi001.stdout:1/307: rename d0/l4 to d0/d9/d1a/d2b/d26/d27/d30/d57/l63 0 2024-02-28T16:31:00.499 INFO:tasks.workunit.client.0.smithi001.stdout:6/182: symlink d9/d39/l42 0 2024-02-28T16:31:00.500 INFO:tasks.workunit.client.0.smithi001.stdout:9/225: getdents d5/d27/d42/d48 0 2024-02-28T16:31:00.501 INFO:tasks.workunit.client.0.smithi001.stdout:1/308: fsync d0/d9/d1a/d2b/d26/d27/d30/d52/f5a 0 2024-02-28T16:31:00.502 INFO:tasks.workunit.client.0.smithi001.stdout:1/309: fdatasync d0/d9/f5c 0 2024-02-28T16:31:00.503 INFO:tasks.workunit.client.0.smithi001.stdout:9/226: chown d5/f1e 110282 1 2024-02-28T16:31:00.505 INFO:tasks.workunit.client.0.smithi001.stdout:7/154: getdents d3/d1c 0 2024-02-28T16:31:00.507 INFO:tasks.workunit.client.0.smithi001.stdout:1/310: write d0/d9/d1a/d2b/f47 [106657,121530] 0 2024-02-28T16:31:00.514 INFO:tasks.workunit.client.0.smithi001.stdout:7/155: write d3/d17/f1b [518390,46155] 0 2024-02-28T16:31:00.516 INFO:tasks.workunit.client.0.smithi001.stdout:1/311: creat d0/d9/d56/d2f/f64 x:0 0 0 2024-02-28T16:31:00.520 INFO:tasks.workunit.client.0.smithi001.stdout:0/157: dwrite f9 [0,4194304] 0 2024-02-28T16:31:00.522 INFO:tasks.workunit.client.0.smithi001.stdout:0/158: truncate f12 1190904 0 2024-02-28T16:31:00.522 INFO:tasks.workunit.client.0.smithi001.stdout:6/183: dwrite f8 [0,4194304] 0 2024-02-28T16:31:00.526 INFO:tasks.workunit.client.0.smithi001.stdout:0/159: write d18/f2c [925088,73260] 0 2024-02-28T16:31:00.528 INFO:tasks.workunit.client.0.smithi001.stdout:0/160: stat d18/f2c 0 2024-02-28T16:31:00.529 INFO:tasks.workunit.client.0.smithi001.stdout:9/227: truncate d5/f21 4622468 0 2024-02-28T16:31:00.530 INFO:tasks.workunit.client.0.smithi001.stdout:0/161: dread - d18/f1e zero size 2024-02-28T16:31:00.536 INFO:tasks.workunit.client.0.smithi001.stdout:9/228: mknod d5/d27/d42/d48/c4e 0 2024-02-28T16:31:00.537 INFO:tasks.workunit.client.0.smithi001.stdout:0/162: write d18/d22/d2f/f24 [23947,80465] 0 2024-02-28T16:31:00.541 INFO:tasks.workunit.client.0.smithi001.stdout:9/229: fsync d5/fe 0 2024-02-28T16:31:00.543 INFO:tasks.workunit.client.0.smithi001.stdout:7/156: dwrite d3/f1e [0,4194304] 0 2024-02-28T16:31:00.551 INFO:tasks.workunit.client.0.smithi001.stdout:9/230: mkdir d5/d4d/d4f 0 2024-02-28T16:31:00.555 INFO:tasks.workunit.client.0.smithi001.stdout:7/157: rename d3/f28 to d3/d17/f2f 0 2024-02-28T16:31:00.556 INFO:tasks.workunit.client.0.smithi001.stdout:9/231: dread - d5/d45/f4a zero size 2024-02-28T16:31:00.557 INFO:tasks.workunit.client.0.smithi001.stdout:6/184: dwrite d9/db/fd [0,4194304] 0 2024-02-28T16:31:00.557 INFO:tasks.workunit.client.0.smithi001.stdout:1/312: dwrite d0/d9/fa [4194304,4194304] 0 2024-02-28T16:31:00.561 INFO:tasks.workunit.client.0.smithi001.stdout:7/158: read d3/ff [1252986,128614] 0 2024-02-28T16:31:00.562 INFO:tasks.workunit.client.0.smithi001.stdout:6/185: write d9/db/f12 [2266242,90737] 0 2024-02-28T16:31:00.564 INFO:tasks.workunit.client.0.smithi001.stdout:9/232: read d5/d47/f3a [611753,81414] 0 2024-02-28T16:31:00.565 INFO:tasks.workunit.client.0.smithi001.stdout:6/186: write d9/d29/d2b/f30 [1354998,117427] 0 2024-02-28T16:31:00.568 INFO:tasks.workunit.client.0.smithi001.stdout:7/159: creat d3/d1c/f30 x:0 0 0 2024-02-28T16:31:00.571 INFO:tasks.workunit.client.0.smithi001.stdout:0/163: truncate d18/d22/d2f/d1c/f30 122499 0 2024-02-28T16:31:00.583 INFO:tasks.workunit.client.0.smithi001.stdout:1/313: link d0/d9/d1a/d2b/d26/c3f d0/c65 0 2024-02-28T16:31:00.586 INFO:tasks.workunit.client.0.smithi001.stdout:9/233: dread d5/f1e [0,4194304] 0 2024-02-28T16:31:00.588 INFO:tasks.workunit.client.0.smithi001.stdout:7/160: dread d3/f1e [0,4194304] 0 2024-02-28T16:31:00.590 INFO:tasks.workunit.client.0.smithi001.stdout:7/161: rename d3/d17 to d3/d17/d31 22 2024-02-28T16:31:00.593 INFO:tasks.workunit.client.0.smithi001.stdout:1/314: creat d0/d9/f66 x:0 0 0 2024-02-28T16:31:00.595 INFO:tasks.workunit.client.0.smithi001.stdout:9/234: write d5/f18 [1973936,86305] 0 2024-02-28T16:31:00.596 INFO:tasks.workunit.client.0.smithi001.stdout:0/164: dread f9 [0,4194304] 0 2024-02-28T16:31:00.597 INFO:tasks.workunit.client.0.smithi001.stdout:7/162: creat d3/d17/f32 x:0 0 0 2024-02-28T16:31:00.600 INFO:tasks.workunit.client.0.smithi001.stdout:0/165: readlink d18/d22/d2f/d1c/l29 0 2024-02-28T16:31:00.602 INFO:tasks.workunit.client.0.smithi001.stdout:7/163: creat d3/d1c/f33 x:0 0 0 2024-02-28T16:31:00.604 INFO:tasks.workunit.client.0.smithi001.stdout:0/166: chown d18/d22/d2f/f1f 3428909 1 2024-02-28T16:31:00.604 INFO:tasks.workunit.client.0.smithi001.stdout:9/235: mkdir d5/d27/d34/d50 0 2024-02-28T16:31:00.605 INFO:tasks.workunit.client.0.smithi001.stdout:7/164: mknod d3/c34 0 2024-02-28T16:31:00.607 INFO:tasks.workunit.client.0.smithi001.stdout:7/165: dread - d3/d1c/f30 zero size 2024-02-28T16:31:00.608 INFO:tasks.workunit.client.0.smithi001.stdout:6/187: dwrite d9/db/f1b [0,4194304] 0 2024-02-28T16:31:00.611 INFO:tasks.workunit.client.0.smithi001.stdout:0/167: rename d18/f2c to d18/f33 0 2024-02-28T16:31:00.612 INFO:tasks.workunit.client.0.smithi001.stdout:7/166: mknod d3/d17/c35 0 2024-02-28T16:31:00.616 INFO:tasks.workunit.client.0.smithi001.stdout:9/236: rename d5/c7 to d5/d27/d34/d50/c51 0 2024-02-28T16:31:00.618 INFO:tasks.workunit.client.0.smithi001.stdout:1/315: truncate d0/d9/f54 2166202 0 2024-02-28T16:31:00.618 INFO:tasks.workunit.client.0.smithi001.stdout:3/132: sync 2024-02-28T16:31:00.621 INFO:tasks.workunit.client.0.smithi001.stdout:6/188: rename d9/d29/d2b/c3c to d9/db/d19/c43 0 2024-02-28T16:31:00.624 INFO:tasks.workunit.client.0.smithi001.stdout:9/237: creat d5/d27/d34/d50/f52 x:0 0 0 2024-02-28T16:31:00.624 INFO:tasks.workunit.client.0.smithi001.stdout:7/167: rmdir d3/d17 39 2024-02-28T16:31:00.626 INFO:tasks.workunit.client.0.smithi001.stdout:3/133: creat d4/d6/d7/d23/f2b x:0 0 0 2024-02-28T16:31:00.630 INFO:tasks.workunit.client.0.smithi001.stdout:6/189: creat d9/d29/f44 x:0 0 0 2024-02-28T16:31:00.630 INFO:tasks.workunit.client.0.smithi001.stdout:9/238: rmdir d5 39 2024-02-28T16:31:00.634 INFO:tasks.workunit.client.0.smithi001.stdout:9/239: chown d5/d45/f4a 18154377 1 2024-02-28T16:31:00.634 INFO:tasks.workunit.client.0.smithi001.stdout:7/168: truncate d3/d17/f32 417916 0 2024-02-28T16:31:00.634 INFO:tasks.workunit.client.0.smithi001.stdout:0/168: dwrite f13 [0,4194304] 0 2024-02-28T16:31:00.635 INFO:tasks.workunit.client.0.smithi001.stdout:6/190: rmdir d9/d29/d2b 39 2024-02-28T16:31:00.636 INFO:tasks.workunit.client.0.smithi001.stdout:1/316: write d0/f21 [33363,17866] 0 2024-02-28T16:31:00.637 INFO:tasks.workunit.client.0.smithi001.stdout:7/169: rmdir d3/d17 39 2024-02-28T16:31:00.640 INFO:tasks.workunit.client.0.smithi001.stdout:0/169: unlink fe 0 2024-02-28T16:31:00.641 INFO:tasks.workunit.client.0.smithi001.stdout:0/170: read f10 [1644836,128314] 0 2024-02-28T16:31:00.642 INFO:tasks.workunit.client.0.smithi001.stdout:7/170: truncate d3/f8 4353672 0 2024-02-28T16:31:00.647 INFO:tasks.workunit.client.0.smithi001.stdout:5/138: sync 2024-02-28T16:31:00.647 INFO:tasks.workunit.client.0.smithi001.stdout:8/197: sync 2024-02-28T16:31:00.648 INFO:tasks.workunit.client.0.smithi001.stdout:2/192: sync 2024-02-28T16:31:00.648 INFO:tasks.workunit.client.0.smithi001.stdout:4/135: sync 2024-02-28T16:31:00.649 INFO:tasks.workunit.client.0.smithi001.stdout:0/171: getdents d18 0 2024-02-28T16:31:00.650 INFO:tasks.workunit.client.0.smithi001.stdout:5/139: truncate dc/f27 1458711 0 2024-02-28T16:31:00.653 INFO:tasks.workunit.client.0.smithi001.stdout:0/172: write fa [2276363,118036] 0 2024-02-28T16:31:00.659 INFO:tasks.workunit.client.0.smithi001.stdout:0/173: write d18/d22/d2f/f24 [1009635,40680] 0 2024-02-28T16:31:00.661 INFO:tasks.workunit.client.0.smithi001.stdout:0/174: chown f13 198242580 1 2024-02-28T16:31:00.661 INFO:tasks.workunit.client.0.smithi001.stdout:7/171: dread d3/f1e [0,4194304] 0 2024-02-28T16:31:00.662 INFO:tasks.workunit.client.0.smithi001.stdout:9/240: truncate d5/d47/f37 2617724 0 2024-02-28T16:31:00.666 INFO:tasks.workunit.client.0.smithi001.stdout:1/317: dwrite d0/d9/d1a/d2b/d26/d27/d30/f55 [4194304,4194304] 0 2024-02-28T16:31:00.668 INFO:tasks.workunit.client.0.smithi001.stdout:4/136: mkdir d3/d2d 0 2024-02-28T16:31:00.673 INFO:tasks.workunit.client.0.smithi001.stdout:9/241: readlink d5/lf 0 2024-02-28T16:31:00.674 INFO:tasks.workunit.client.0.smithi001.stdout:5/140: link c1 dc/d12/c2f 0 2024-02-28T16:31:00.676 INFO:tasks.workunit.client.0.smithi001.stdout:2/193: dwrite d0/d2/f3b [0,4194304] 0 2024-02-28T16:31:00.678 INFO:tasks.workunit.client.0.smithi001.stdout:9/242: creat d5/d27/d42/f53 x:0 0 0 2024-02-28T16:31:00.680 INFO:tasks.workunit.client.0.smithi001.stdout:3/134: dread d4/d18/f11 [0,4194304] 0 2024-02-28T16:31:00.681 INFO:tasks.workunit.client.0.smithi001.stdout:4/137: write d3/d5/dd/f2a [1738753,122750] 0 2024-02-28T16:31:00.681 INFO:tasks.workunit.client.0.smithi001.stdout:5/141: write dc/f27 [423853,66810] 0 2024-02-28T16:31:00.682 INFO:tasks.workunit.client.0.smithi001.stdout:1/318: chown d0/d9/d1a/d2b/d26/d27/d30/f59 2364 1 2024-02-28T16:31:00.688 INFO:tasks.workunit.client.0.smithi001.stdout:1/319: fsync d0/d9/d56/d2f/f64 0 2024-02-28T16:31:00.697 INFO:tasks.workunit.client.0.smithi001.stdout:8/198: truncate d4/d7/f21 40737 0 2024-02-28T16:31:00.698 INFO:tasks.workunit.client.0.smithi001.stdout:6/191: dread d9/db/f12 [0,4194304] 0 2024-02-28T16:31:00.700 INFO:tasks.workunit.client.0.smithi001.stdout:0/175: dwrite fd [4194304,4194304] 0 2024-02-28T16:31:00.700 INFO:tasks.workunit.client.0.smithi001.stdout:6/192: fsync f8 0 2024-02-28T16:31:00.702 INFO:tasks.workunit.client.0.smithi001.stdout:0/176: write f11 [3171228,122063] 0 2024-02-28T16:31:00.704 INFO:tasks.workunit.client.0.smithi001.stdout:4/138: mkdir d3/d5/d7/d1f/d2e 0 2024-02-28T16:31:00.704 INFO:tasks.workunit.client.0.smithi001.stdout:9/243: truncate d5/f6 4527066 0 2024-02-28T16:31:00.705 INFO:tasks.workunit.client.0.smithi001.stdout:1/320: mkdir d0/d9/d1a/d2b/d26/d27/d30/d67 0 2024-02-28T16:31:00.711 INFO:tasks.workunit.client.0.smithi001.stdout:8/199: read d4/d7/f19 [1407461,35283] 0 2024-02-28T16:31:00.715 INFO:tasks.workunit.client.0.smithi001.stdout:5/142: creat dc/f30 x:0 0 0 2024-02-28T16:31:00.716 INFO:tasks.workunit.client.0.smithi001.stdout:9/244: chown d5/c1b 0 1 2024-02-28T16:31:00.716 INFO:tasks.workunit.client.0.smithi001.stdout:3/135: dwrite d4/d1e/f21 [0,4194304] 0 2024-02-28T16:31:00.717 INFO:tasks.workunit.client.0.smithi001.stdout:7/172: getdents d3/d17 0 2024-02-28T16:31:00.717 INFO:tasks.workunit.client.0.smithi001.stdout:1/321: mkdir d0/d9/d1a/d2b/d26/d27/d30/d4a/d68 0 2024-02-28T16:31:00.721 INFO:tasks.workunit.client.0.smithi001.stdout:9/245: dread - d5/d27/d42/d48/f4c zero size 2024-02-28T16:31:00.722 INFO:tasks.workunit.client.0.smithi001.stdout:6/193: creat d9/db/f45 x:0 0 0 2024-02-28T16:31:00.724 INFO:tasks.workunit.client.0.smithi001.stdout:0/177: dwrite f13 [0,4194304] 0 2024-02-28T16:31:00.726 INFO:tasks.workunit.client.0.smithi001.stdout:8/200: mknod d4/d7/c38 0 2024-02-28T16:31:00.728 INFO:tasks.workunit.client.0.smithi001.stdout:1/322: write d0/d9/d1a/d2b/f47 [988300,21134] 0 2024-02-28T16:31:00.731 INFO:tasks.workunit.client.0.smithi001.stdout:9/246: chown d5/f19 2684149 1 2024-02-28T16:31:00.743 INFO:tasks.workunit.client.0.smithi001.stdout:9/247: readlink d5/l14 0 2024-02-28T16:31:00.745 INFO:tasks.workunit.client.0.smithi001.stdout:6/194: write d9/d29/d2b/f30 [203479,114081] 0 2024-02-28T16:31:00.746 INFO:tasks.workunit.client.0.smithi001.stdout:1/323: mknod d0/d9/d1a/d2b/d26/d27/d30/d4a/c69 0 2024-02-28T16:31:00.748 INFO:tasks.workunit.client.0.smithi001.stdout:7/173: rename d3/f27 to d3/f36 0 2024-02-28T16:31:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:00 smithi066 ceph-mon[36236]: Reconfiguring node-exporter.smithi066 (dependencies changed)... 2024-02-28T16:31:00.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:00 smithi066 ceph-mon[36236]: Reconfiguring daemon node-exporter.smithi066 on smithi066 2024-02-28T16:31:00.750 INFO:tasks.workunit.client.0.smithi001.stdout:3/136: link d4/c1b d4/d1e/d29/c2c 0 2024-02-28T16:31:00.750 INFO:tasks.workunit.client.0.smithi001.stdout:1/324: dread - d0/d9/f5c zero size 2024-02-28T16:31:00.752 INFO:tasks.workunit.client.0.smithi001.stdout:8/201: dwrite d4/d7/f1d [0,4194304] 0 2024-02-28T16:31:00.754 INFO:tasks.workunit.client.0.smithi001.stdout:5/143: dread f7 [0,4194304] 0 2024-02-28T16:31:00.762 INFO:tasks.workunit.client.0.smithi001.stdout:5/144: mknod dc/d1a/c31 0 2024-02-28T16:31:00.762 INFO:tasks.workunit.client.0.smithi001.stdout:0/178: truncate fd 2159164 0 2024-02-28T16:31:00.766 INFO:tasks.workunit.client.0.smithi001.stdout:8/202: unlink d4/d7/c14 0 2024-02-28T16:31:00.766 INFO:tasks.workunit.client.0.smithi001.stdout:5/145: write dc/f28 [423523,61055] 0 2024-02-28T16:31:00.766 INFO:tasks.workunit.client.0.smithi001.stdout:9/248: rmdir d5/d4d/d4f 0 2024-02-28T16:31:00.766 INFO:tasks.workunit.client.0.smithi001.stdout:4/139: dread d3/fa [0,4194304] 0 2024-02-28T16:31:00.768 INFO:tasks.workunit.client.0.smithi001.stdout:1/325: symlink d0/d9/l6a 0 2024-02-28T16:31:00.768 INFO:tasks.workunit.client.0.smithi001.stdout:0/179: mknod d18/d22/d2f/d1c/c34 0 2024-02-28T16:31:00.768 INFO:tasks.workunit.client.0.smithi001.stdout:5/146: stat dc/d1a/l26 0 2024-02-28T16:31:00.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:00 smithi001 ceph-mon[28991]: Reconfiguring node-exporter.smithi066 (dependencies changed)... 2024-02-28T16:31:00.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:00 smithi001 ceph-mon[28991]: Reconfiguring daemon node-exporter.smithi066 on smithi066 2024-02-28T16:31:00.770 INFO:tasks.workunit.client.0.smithi001.stdout:6/195: rename d9/db/f12 to d9/db/d10/f46 0 2024-02-28T16:31:00.772 INFO:tasks.workunit.client.0.smithi001.stdout:8/203: mknod d4/d7/d22/c39 0 2024-02-28T16:31:00.774 INFO:tasks.workunit.client.0.smithi001.stdout:5/147: creat dc/d12/f32 x:0 0 0 2024-02-28T16:31:00.775 INFO:tasks.workunit.client.0.smithi001.stdout:0/180: chown d18/d22/d2f/f23 53 1 2024-02-28T16:31:00.776 INFO:tasks.workunit.client.0.smithi001.stdout:9/249: read d5/f18 [800632,62024] 0 2024-02-28T16:31:00.776 INFO:tasks.workunit.client.0.smithi001.stdout:4/140: rmdir d3/d5/dd 39 2024-02-28T16:31:00.781 INFO:tasks.workunit.client.0.smithi001.stdout:7/174: dwrite d3/f36 [0,4194304] 0 2024-02-28T16:31:00.782 INFO:tasks.workunit.client.0.smithi001.stdout:6/196: unlink d9/db/d10/d15/l26 0 2024-02-28T16:31:00.783 INFO:tasks.workunit.client.0.smithi001.stdout:6/197: readlink d9/d29/l2e 0 2024-02-28T16:31:00.784 INFO:tasks.workunit.client.0.smithi001.stdout:8/204: creat d4/d7/d22/d27/f3a x:0 0 0 2024-02-28T16:31:00.786 INFO:tasks.workunit.client.0.smithi001.stdout:8/205: fdatasync d4/d7/d22/d27/f2f 0 2024-02-28T16:31:00.786 INFO:tasks.workunit.client.0.smithi001.stdout:5/148: mknod dc/d1a/d21/c33 0 2024-02-28T16:31:00.789 INFO:tasks.workunit.client.0.smithi001.stdout:3/137: truncate d4/d6/d7/f20 3863854 0 2024-02-28T16:31:00.789 INFO:tasks.workunit.client.0.smithi001.stdout:5/149: stat dc/d1a/d21/d2d 0 2024-02-28T16:31:00.791 INFO:tasks.workunit.client.0.smithi001.stdout:3/138: dread - d4/d1e/f27 zero size 2024-02-28T16:31:00.792 INFO:tasks.workunit.client.0.smithi001.stdout:1/326: mkdir d0/d9/d1a/d2b/d26/d27/d30/d52/d6b 0 2024-02-28T16:31:00.794 INFO:tasks.workunit.client.0.smithi001.stdout:4/141: dread d3/f12 [0,4194304] 0 2024-02-28T16:31:00.795 INFO:tasks.workunit.client.0.smithi001.stdout:7/175: chown d3/cd 26 1 2024-02-28T16:31:00.796 INFO:tasks.workunit.client.0.smithi001.stdout:6/198: mkdir d9/de/d47 0 2024-02-28T16:31:00.799 INFO:tasks.workunit.client.0.smithi001.stdout:7/176: read d3/d17/f19 [211787,58795] 0 2024-02-28T16:31:00.801 INFO:tasks.workunit.client.0.smithi001.stdout:0/181: truncate f0 3778897 0 2024-02-28T16:31:00.803 INFO:tasks.workunit.client.0.smithi001.stdout:1/327: creat d0/d9/d56/f6c x:0 0 0 2024-02-28T16:31:00.805 INFO:tasks.workunit.client.0.smithi001.stdout:4/142: mknod d3/d5/d7/c2f 0 2024-02-28T16:31:00.815 INFO:tasks.workunit.client.0.smithi001.stdout:0/182: symlink d18/d22/l35 0 2024-02-28T16:31:00.819 INFO:tasks.workunit.client.0.smithi001.stdout:3/139: link d4/d6/d7/d23/f2b d4/d6/d7/d23/f2d 0 2024-02-28T16:31:00.821 INFO:tasks.workunit.client.0.smithi001.stdout:3/140: write d4/d1e/f22 [925126,107648] 0 2024-02-28T16:31:00.822 INFO:tasks.workunit.client.0.smithi001.stdout:8/206: creat d4/f3b x:0 0 0 2024-02-28T16:31:00.824 INFO:tasks.workunit.client.0.smithi001.stdout:9/250: getdents d5/d27 0 2024-02-28T16:31:00.826 INFO:tasks.workunit.client.0.smithi001.stdout:3/141: fsync d4/d1e/f21 0 2024-02-28T16:31:00.827 INFO:tasks.workunit.client.0.smithi001.stdout:5/150: dread dc/df/f13 [0,4194304] 0 2024-02-28T16:31:00.828 INFO:tasks.workunit.client.0.smithi001.stdout:0/183: unlink d18/f1d 0 2024-02-28T16:31:00.829 INFO:tasks.workunit.client.0.smithi001.stdout:6/199: creat d9/d29/f48 x:0 0 0 2024-02-28T16:31:00.830 INFO:tasks.workunit.client.0.smithi001.stdout:3/142: chown d4/d6/d7/d23/f2b 16324 1 2024-02-28T16:31:00.831 INFO:tasks.workunit.client.0.smithi001.stdout:2/194: sync 2024-02-28T16:31:00.833 INFO:tasks.workunit.client.0.smithi001.stdout:1/328: creat d0/d9/d1a/d2b/f6d x:0 0 0 2024-02-28T16:31:00.834 INFO:tasks.workunit.client.0.smithi001.stdout:9/251: rmdir d5/d47/d33 39 2024-02-28T16:31:00.835 INFO:tasks.workunit.client.0.smithi001.stdout:4/143: dread d3/d5/dd/f2a [0,4194304] 0 2024-02-28T16:31:00.836 INFO:tasks.workunit.client.0.smithi001.stdout:7/177: dwrite d3/d17/f19 [0,4194304] 0 2024-02-28T16:31:00.850 INFO:tasks.workunit.client.0.smithi001.stdout:0/184: dread fa [0,4194304] 0 2024-02-28T16:31:00.852 INFO:tasks.workunit.client.0.smithi001.stdout:3/143: rename d4/d6/d7/d23/l2a to d4/d18/l2e 0 2024-02-28T16:31:00.852 INFO:tasks.workunit.client.0.smithi001.stdout:7/178: creat d3/d1c/f37 x:0 0 0 2024-02-28T16:31:00.854 INFO:tasks.workunit.client.0.smithi001.stdout:5/151: write dc/df/f11 [1805536,46251] 0 2024-02-28T16:31:00.858 INFO:tasks.workunit.client.0.smithi001.stdout:2/195: dwrite d0/d2/f21 [0,4194304] 0 2024-02-28T16:31:00.858 INFO:tasks.workunit.client.0.smithi001.stdout:3/144: write d4/d18/f10 [4804639,57572] 0 2024-02-28T16:31:00.861 INFO:tasks.workunit.client.0.smithi001.stdout:5/152: write dc/f27 [2125948,97857] 0 2024-02-28T16:31:00.861 INFO:tasks.workunit.client.0.smithi001.stdout:4/144: rename d3/d5/d7/d1f/d20/c26 to d3/d5/d7/d1f/c30 0 2024-02-28T16:31:00.863 INFO:tasks.workunit.client.0.smithi001.stdout:6/200: getdents d9/db/d19 0 2024-02-28T16:31:00.867 INFO:tasks.workunit.client.0.smithi001.stdout:8/207: truncate d4/f13 2252285 0 2024-02-28T16:31:00.869 INFO:tasks.workunit.client.0.smithi001.stdout:7/179: symlink d3/d1c/l38 0 2024-02-28T16:31:00.872 INFO:tasks.workunit.client.0.smithi001.stdout:6/201: chown d9/d39/f3a 4149688 1 2024-02-28T16:31:00.877 INFO:tasks.workunit.client.0.smithi001.stdout:5/153: creat dc/df/f34 x:0 0 0 2024-02-28T16:31:00.878 INFO:tasks.workunit.client.0.smithi001.stdout:9/252: dread d5/d47/f37 [0,4194304] 0 2024-02-28T16:31:00.879 INFO:tasks.workunit.client.0.smithi001.stdout:6/202: fdatasync d9/db/d10/d15/f34 0 2024-02-28T16:31:00.879 INFO:tasks.workunit.client.0.smithi001.stdout:4/145: creat d3/d5/d7/d1f/d20/f31 x:0 0 0 2024-02-28T16:31:00.880 INFO:tasks.workunit.client.0.smithi001.stdout:1/329: dwrite d0/d9/d1a/d2b/f4d [0,4194304] 0 2024-02-28T16:31:00.888 INFO:tasks.workunit.client.0.smithi001.stdout:5/154: truncate dc/df/f29 747981 0 2024-02-28T16:31:00.890 INFO:tasks.workunit.client.0.smithi001.stdout:3/145: rename d4/d6/d7/f1f to d4/d6/d7/f2f 0 2024-02-28T16:31:00.890 INFO:tasks.workunit.client.0.smithi001.stdout:7/180: mknod d3/d1c/c39 0 2024-02-28T16:31:00.894 INFO:tasks.workunit.client.0.smithi001.stdout:6/203: mknod d9/c49 0 2024-02-28T16:31:00.896 INFO:tasks.workunit.client.0.smithi001.stdout:7/181: write d3/d17/f19 [1564509,39030] 0 2024-02-28T16:31:00.898 INFO:tasks.workunit.client.0.smithi001.stdout:2/196: dwrite d0/d2/f11 [0,4194304] 0 2024-02-28T16:31:00.902 INFO:tasks.workunit.client.0.smithi001.stdout:3/146: mkdir d4/d30 0 2024-02-28T16:31:00.902 INFO:tasks.workunit.client.0.smithi001.stdout:6/204: fsync d9/db/d10/d15/f18 0 2024-02-28T16:31:00.902 INFO:tasks.workunit.client.0.smithi001.stdout:0/185: mkdir d18/d22/d2b/d36 0 2024-02-28T16:31:00.903 INFO:tasks.workunit.client.0.smithi001.stdout:4/146: write d3/f17 [5923122,52674] 0 2024-02-28T16:31:00.904 INFO:tasks.workunit.client.0.smithi001.stdout:8/208: dwrite d4/d7/f35 [0,4194304] 0 2024-02-28T16:31:00.907 INFO:tasks.workunit.client.0.smithi001.stdout:1/330: creat d0/d9/d1a/d2b/d26/d27/d30/d52/d6b/f6e x:0 0 0 2024-02-28T16:31:00.911 INFO:tasks.workunit.client.0.smithi001.stdout:8/209: fdatasync d4/d7/d22/d27/f2f 0 2024-02-28T16:31:00.912 INFO:tasks.workunit.client.0.smithi001.stdout:6/205: mknod d9/de/c4a 0 2024-02-28T16:31:00.914 INFO:tasks.workunit.client.0.smithi001.stdout:3/147: chown d4 27336080 1 2024-02-28T16:31:00.914 INFO:tasks.workunit.client.0.smithi001.stdout:7/182: write d3/d17/f2f [770745,122894] 0 2024-02-28T16:31:00.920 INFO:tasks.workunit.client.0.smithi001.stdout:0/186: rename f10 to d18/d22/d2b/f37 0 2024-02-28T16:31:00.921 INFO:tasks.workunit.client.0.smithi001.stdout:1/331: symlink d0/d9/d56/l6f 0 2024-02-28T16:31:00.922 INFO:tasks.workunit.client.0.smithi001.stdout:8/210: dread - d4/d7/d22/d27/f2f zero size 2024-02-28T16:31:00.923 INFO:tasks.workunit.client.0.smithi001.stdout:4/147: creat d3/d2d/f32 x:0 0 0 2024-02-28T16:31:00.925 INFO:tasks.workunit.client.0.smithi001.stdout:6/206: fdatasync d9/db/d19/f1f 0 2024-02-28T16:31:00.927 INFO:tasks.workunit.client.0.smithi001.stdout:8/211: stat d4/d7 0 2024-02-28T16:31:00.935 INFO:tasks.workunit.client.0.smithi001.stdout:5/155: dwrite f5 [0,4194304] 0 2024-02-28T16:31:00.941 INFO:tasks.workunit.client.0.smithi001.stdout:8/212: chown d4/l9 259804 1 2024-02-28T16:31:00.942 INFO:tasks.workunit.client.0.smithi001.stdout:6/207: rename d9/db/f25 to d9/db/d19/f4b 0 2024-02-28T16:31:00.944 INFO:tasks.workunit.client.0.smithi001.stdout:2/197: dwrite d0/d2/d6/d12/f27 [0,4194304] 0 2024-02-28T16:31:00.949 INFO:tasks.workunit.client.0.smithi001.stdout:5/156: creat dc/df/f35 x:0 0 0 2024-02-28T16:31:00.950 INFO:tasks.workunit.client.0.smithi001.stdout:8/213: write f1 [1699315,71804] 0 2024-02-28T16:31:00.952 INFO:tasks.workunit.client.0.smithi001.stdout:6/208: chown d9/db/f1b 6 1 2024-02-28T16:31:00.953 INFO:tasks.workunit.client.0.smithi001.stdout:3/148: write d4/d6/d7/d23/f2d [516765,15865] 0 2024-02-28T16:31:00.956 INFO:tasks.workunit.client.0.smithi001.stdout:2/198: creat d0/f40 x:0 0 0 2024-02-28T16:31:00.956 INFO:tasks.workunit.client.0.smithi001.stdout:6/209: readlink d9/l17 0 2024-02-28T16:31:00.960 INFO:tasks.workunit.client.0.smithi001.stdout:0/187: dread d18/d22/d2f/f23 [0,4194304] 0 2024-02-28T16:31:00.961 INFO:tasks.workunit.client.0.smithi001.stdout:7/183: dwrite d3/f10 [0,4194304] 0 2024-02-28T16:31:00.963 INFO:tasks.workunit.client.0.smithi001.stdout:3/149: stat d4/f14 0 2024-02-28T16:31:00.963 INFO:tasks.workunit.client.0.smithi001.stdout:1/332: dread d0/f2c [0,4194304] 0 2024-02-28T16:31:00.964 INFO:tasks.workunit.client.0.smithi001.stdout:8/214: chown d4/d32/l34 1544 1 2024-02-28T16:31:00.964 INFO:tasks.workunit.client.0.smithi001.stdout:6/210: write d9/d39/f3a [12473,29987] 0 2024-02-28T16:31:00.966 INFO:tasks.workunit.client.0.smithi001.stdout:4/148: dwrite d3/f12 [0,4194304] 0 2024-02-28T16:31:00.971 INFO:tasks.workunit.client.0.smithi001.stdout:6/211: readlink d9/db/l3d 0 2024-02-28T16:31:00.973 INFO:tasks.workunit.client.0.smithi001.stdout:5/157: getdents dc/df 0 2024-02-28T16:31:00.974 INFO:tasks.workunit.client.0.smithi001.stdout:3/150: mknod d4/c31 0 2024-02-28T16:31:00.976 INFO:tasks.workunit.client.0.smithi001.stdout:8/215: symlink d4/d7/d22/l3c 0 2024-02-28T16:31:00.979 INFO:tasks.workunit.client.0.smithi001.stdout:0/188: rename d18/d22/d2f/f23 to d18/d22/d2f/d1c/f38 0 2024-02-28T16:31:00.981 INFO:tasks.workunit.client.0.smithi001.stdout:6/212: chown d9/db/d19/f1f 26610616 1 2024-02-28T16:31:00.983 INFO:tasks.workunit.client.0.smithi001.stdout:0/189: fsync f13 0 2024-02-28T16:31:00.983 INFO:tasks.workunit.client.0.smithi001.stdout:5/158: write dc/df/f34 [754440,94817] 0 2024-02-28T16:31:00.983 INFO:tasks.workunit.client.0.smithi001.stdout:1/333: creat d0/d9/d1a/d2b/d26/d27/d30/d4a/f70 x:0 0 0 2024-02-28T16:31:00.984 INFO:tasks.workunit.client.0.smithi001.stdout:6/213: fdatasync d9/db/f1b 0 2024-02-28T16:31:00.988 INFO:tasks.workunit.client.0.smithi001.stdout:2/199: truncate d0/d2/d3/de/d22/f38 150343 0 2024-02-28T16:31:00.988 INFO:tasks.workunit.client.0.smithi001.stdout:8/216: mknod d4/d7/d22/c3d 0 2024-02-28T16:31:00.991 INFO:tasks.workunit.client.0.smithi001.stdout:4/149: dread f2 [0,4194304] 0 2024-02-28T16:31:00.991 INFO:tasks.workunit.client.0.smithi001.stdout:7/184: truncate d3/d17/f2f 122918 0 2024-02-28T16:31:00.997 INFO:tasks.workunit.client.0.smithi001.stdout:2/200: write d0/f33 [115816,8613] 0 2024-02-28T16:31:01.000 INFO:tasks.workunit.client.0.smithi001.stdout:5/159: symlink dc/d1a/d21/d2d/l36 0 2024-02-28T16:31:01.002 INFO:tasks.workunit.client.0.smithi001.stdout:7/185: mknod d3/d17/c3a 0 2024-02-28T16:31:01.003 INFO:tasks.workunit.client.0.smithi001.stdout:1/334: rename d0/d9/d56/l5d to d0/d9/d1a/d2b/d26/d27/d30/d4a/d68/l71 0 2024-02-28T16:31:01.008 INFO:tasks.workunit.client.0.smithi001.stdout:7/186: creat d3/d17/f3b x:0 0 0 2024-02-28T16:31:01.010 INFO:tasks.workunit.client.0.smithi001.stdout:5/160: creat dc/d1a/d21/f37 x:0 0 0 2024-02-28T16:31:01.012 INFO:tasks.workunit.client.0.smithi001.stdout:1/335: write d0/d9/d1a/d2b/d26/f34 [2018257,36798] 0 2024-02-28T16:31:01.014 INFO:tasks.workunit.client.0.smithi001.stdout:3/151: write d4/d6/d7/f20 [3899332,125780] 0 2024-02-28T16:31:01.016 INFO:tasks.workunit.client.0.smithi001.stdout:5/161: creat dc/d12/f38 x:0 0 0 2024-02-28T16:31:01.018 INFO:tasks.workunit.client.0.smithi001.stdout:0/190: dwrite d18/d22/d2f/f24 [0,4194304] 0 2024-02-28T16:31:01.019 INFO:tasks.workunit.client.0.smithi001.stdout:8/217: dwrite d4/f12 [8388608,4194304] 0 2024-02-28T16:31:01.020 INFO:tasks.workunit.client.0.smithi001.stdout:7/187: symlink d3/d1c/l3c 0 2024-02-28T16:31:01.023 INFO:tasks.workunit.client.0.smithi001.stdout:2/201: dwrite d0/d2/d3/d2c/f2e [0,4194304] 0 2024-02-28T16:31:01.023 INFO:tasks.workunit.client.0.smithi001.stdout:3/152: readlink d4/d18/l15 0 2024-02-28T16:31:01.024 INFO:tasks.workunit.client.0.smithi001.stdout:8/218: chown d4/d7/f20 0 1 2024-02-28T16:31:01.026 INFO:tasks.workunit.client.0.smithi001.stdout:8/219: read d4/f12 [3606804,88624] 0 2024-02-28T16:31:01.034 INFO:tasks.workunit.client.0.smithi001.stdout:7/188: mkdir d3/d17/d3d 0 2024-02-28T16:31:01.037 INFO:tasks.workunit.client.0.smithi001.stdout:2/202: creat d0/d2/d6/d12/f41 x:0 0 0 2024-02-28T16:31:01.048 INFO:tasks.workunit.client.0.smithi001.stdout:2/203: mknod d0/d2/d3/c42 0 2024-02-28T16:31:01.057 INFO:tasks.workunit.client.0.smithi001.stdout:0/191: truncate f15 1858931 0 2024-02-28T16:31:01.058 INFO:tasks.workunit.client.0.smithi001.stdout:1/336: dread d0/f36 [0,4194304] 0 2024-02-28T16:31:01.060 INFO:tasks.workunit.client.0.smithi001.stdout:5/162: dwrite f7 [0,4194304] 0 2024-02-28T16:31:01.062 INFO:tasks.workunit.client.0.smithi001.stdout:8/220: dwrite d4/d7/d22/d27/f3a [0,4194304] 0 2024-02-28T16:31:01.064 INFO:tasks.workunit.client.0.smithi001.stdout:2/204: creat d0/d2/d3/f43 x:0 0 0 2024-02-28T16:31:01.065 INFO:tasks.workunit.client.0.smithi001.stdout:0/192: symlink d18/d22/d2b/l39 0 2024-02-28T16:31:01.066 INFO:tasks.workunit.client.0.smithi001.stdout:3/153: dwrite d4/d6/d7/d23/f2d [0,4194304] 0 2024-02-28T16:31:01.068 INFO:tasks.workunit.client.0.smithi001.stdout:5/163: write fa [3891822,73383] 0 2024-02-28T16:31:01.071 INFO:tasks.workunit.client.0.smithi001.stdout:2/205: chown d0/d2/d3/d2c/l35 227466818 1 2024-02-28T16:31:01.072 INFO:tasks.workunit.client.0.smithi001.stdout:8/221: mknod d4/d32/c3e 0 2024-02-28T16:31:01.077 INFO:tasks.workunit.client.0.smithi001.stdout:7/189: dwrite d3/f1f [0,4194304] 0 2024-02-28T16:31:01.077 INFO:tasks.workunit.client.0.smithi001.stdout:5/164: creat dc/d1a/d21/d2d/f39 x:0 0 0 2024-02-28T16:31:01.080 INFO:tasks.workunit.client.0.smithi001.stdout:1/337: mkdir d0/d9/d1a/d72 0 2024-02-28T16:31:01.085 INFO:tasks.workunit.client.0.smithi001.stdout:3/154: creat d4/d30/f32 x:0 0 0 2024-02-28T16:31:01.090 INFO:tasks.workunit.client.0.smithi001.stdout:7/190: symlink d3/d1c/l3e 0 2024-02-28T16:31:01.091 INFO:tasks.workunit.client.0.smithi001.stdout:5/165: creat dc/df/f3a x:0 0 0 2024-02-28T16:31:01.095 INFO:tasks.workunit.client.0.smithi001.stdout:1/338: unlink d0/d9/d1a/d2b/d26/d27/d30/f55 0 2024-02-28T16:31:01.096 INFO:tasks.workunit.client.0.smithi001.stdout:2/206: truncate d0/d2/f14 3359219 0 2024-02-28T16:31:01.096 INFO:tasks.workunit.client.0.smithi001.stdout:5/166: readlink dc/d1a/d21/d2d/l36 0 2024-02-28T16:31:01.096 INFO:tasks.workunit.client.0.smithi001.stdout:1/339: stat d0/f36 0 2024-02-28T16:31:01.096 INFO:tasks.workunit.client.0.smithi001.stdout:5/167: stat dc/df/f11 0 2024-02-28T16:31:01.097 INFO:tasks.workunit.client.0.smithi001.stdout:1/340: chown d0/d9/d1a/d2b 1118162 1 2024-02-28T16:31:01.098 INFO:tasks.workunit.client.0.smithi001.stdout:0/193: getdents d18/d22 0 2024-02-28T16:31:01.099 INFO:tasks.workunit.client.0.smithi001.stdout:8/222: dwrite d4/f11 [0,4194304] 0 2024-02-28T16:31:01.105 INFO:tasks.workunit.client.0.smithi001.stdout:5/168: mknod dc/d12/c3b 0 2024-02-28T16:31:01.105 INFO:tasks.workunit.client.0.smithi001.stdout:8/223: write d4/f12 [12915398,26682] 0 2024-02-28T16:31:01.106 INFO:tasks.workunit.client.0.smithi001.stdout:3/155: rename d4/d30/f32 to d4/d1e/f33 0 2024-02-28T16:31:01.106 INFO:tasks.workunit.client.0.smithi001.stdout:0/194: mknod d18/d22/d2f/c3a 0 2024-02-28T16:31:01.108 INFO:tasks.workunit.client.0.smithi001.stdout:1/341: mkdir d0/d73 0 2024-02-28T16:31:01.110 INFO:tasks.workunit.client.0.smithi001.stdout:8/224: read d4/d7/f2b [671829,65897] 0 2024-02-28T16:31:01.116 INFO:tasks.workunit.client.0.smithi001.stdout:5/169: mkdir dc/d1a/d21/d2d/d3c 0 2024-02-28T16:31:01.117 INFO:tasks.workunit.client.0.smithi001.stdout:0/195: mknod d18/d22/d2f/d1c/c3b 0 2024-02-28T16:31:01.117 INFO:tasks.workunit.client.0.smithi001.stdout:7/191: getdents d3/d17 0 2024-02-28T16:31:01.119 INFO:tasks.workunit.client.0.smithi001.stdout:5/170: chown dc/df/f29 848 1 2024-02-28T16:31:01.120 INFO:tasks.workunit.client.0.smithi001.stdout:7/192: fsync d3/f8 0 2024-02-28T16:31:01.124 INFO:tasks.workunit.client.0.smithi001.stdout:5/171: write dc/d12/f32 [953952,71498] 0 2024-02-28T16:31:01.124 INFO:tasks.workunit.client.0.smithi001.stdout:9/253: sync 2024-02-28T16:31:01.125 INFO:tasks.workunit.client.0.smithi001.stdout:1/342: unlink d0/d9/d56/l6f 0 2024-02-28T16:31:01.129 INFO:tasks.workunit.client.0.smithi001.stdout:5/172: dread - dc/d1a/d21/f37 zero size 2024-02-28T16:31:01.129 INFO:tasks.workunit.client.0.smithi001.stdout:1/343: dread - d0/d9/d56/f29 zero size 2024-02-28T16:31:01.133 INFO:tasks.workunit.client.0.smithi001.stdout:0/196: dread d18/d22/d2f/d1c/f30 [0,4194304] 0 2024-02-28T16:31:01.134 INFO:tasks.workunit.client.0.smithi001.stdout:9/254: symlink d5/d4d/l54 0 2024-02-28T16:31:01.140 INFO:tasks.workunit.client.0.smithi001.stdout:5/173: rename dc/d12/f38 to dc/d1a/d21/d2d/d3c/f3d 0 2024-02-28T16:31:01.142 INFO:tasks.workunit.client.0.smithi001.stdout:5/174: read - dc/d1a/d21/f37 zero size 2024-02-28T16:31:01.143 INFO:tasks.workunit.client.0.smithi001.stdout:3/156: dread d4/d6/d7/f2f [0,4194304] 0 2024-02-28T16:31:01.144 INFO:tasks.workunit.client.0.smithi001.stdout:7/193: symlink d3/d17/d3d/l3f 0 2024-02-28T16:31:01.146 INFO:tasks.workunit.client.0.smithi001.stdout:7/194: write d3/f22 [255427,103537] 0 2024-02-28T16:31:01.150 INFO:tasks.workunit.client.0.smithi001.stdout:8/225: dwrite d4/d7/f2b [0,4194304] 0 2024-02-28T16:31:01.155 INFO:tasks.workunit.client.0.smithi001.stdout:0/197: dread d18/f33 [0,4194304] 0 2024-02-28T16:31:01.158 INFO:tasks.workunit.client.0.smithi001.stdout:2/207: dwrite d0/d2/f14 [0,4194304] 0 2024-02-28T16:31:01.162 INFO:tasks.workunit.client.0.smithi001.stdout:8/226: creat d4/d7/d22/f3f x:0 0 0 2024-02-28T16:31:01.163 INFO:tasks.workunit.client.0.smithi001.stdout:2/208: write d0/f33 [592291,35360] 0 2024-02-28T16:31:01.167 INFO:tasks.workunit.client.0.smithi001.stdout:8/227: stat d4/d7/f2b 0 2024-02-28T16:31:01.169 INFO:tasks.workunit.client.0.smithi001.stdout:2/209: fsync d0/d2/d6/f15 0 2024-02-28T16:31:01.170 INFO:tasks.workunit.client.0.smithi001.stdout:5/175: unlink dc/df/l2a 0 2024-02-28T16:31:01.171 INFO:tasks.workunit.client.0.smithi001.stdout:3/157: dwrite d4/f26 [0,4194304] 0 2024-02-28T16:31:01.176 INFO:tasks.workunit.client.0.smithi001.stdout:9/255: link d5/l3f d5/d45/l55 0 2024-02-28T16:31:01.178 INFO:tasks.workunit.client.0.smithi001.stdout:3/158: chown d4/d1e/f21 72055 1 2024-02-28T16:31:01.179 INFO:tasks.workunit.client.0.smithi001.stdout:1/344: creat d0/d9/d1a/d2b/d26/d27/d30/f74 x:0 0 0 2024-02-28T16:31:01.180 INFO:tasks.workunit.client.0.smithi001.stdout:5/176: truncate dc/f20 3815054 0 2024-02-28T16:31:01.181 INFO:tasks.workunit.client.0.smithi001.stdout:7/195: dwrite d3/d17/f32 [0,4194304] 0 2024-02-28T16:31:01.190 INFO:tasks.workunit.client.0.smithi001.stdout:2/210: symlink d0/l44 0 2024-02-28T16:31:01.190 INFO:tasks.workunit.client.0.smithi001.stdout:3/159: creat d4/d6/d7/d23/f34 x:0 0 0 2024-02-28T16:31:01.191 INFO:tasks.workunit.client.0.smithi001.stdout:0/198: truncate f9 1540635 0 2024-02-28T16:31:01.192 INFO:tasks.workunit.client.0.smithi001.stdout:0/199: fdatasync f13 0 2024-02-28T16:31:01.195 INFO:tasks.workunit.client.0.smithi001.stdout:5/177: mknod dc/d1a/d21/c3e 0 2024-02-28T16:31:01.196 INFO:tasks.workunit.client.0.smithi001.stdout:3/160: truncate d4/d1e/f24 986334 0 2024-02-28T16:31:01.196 INFO:tasks.workunit.client.0.smithi001.stdout:8/228: rename d4/f12 to d4/d7/f40 0 2024-02-28T16:31:01.198 INFO:tasks.workunit.client.0.smithi001.stdout:7/196: creat d3/d1c/f40 x:0 0 0 2024-02-28T16:31:01.199 INFO:tasks.workunit.client.0.smithi001.stdout:9/256: mknod d5/d47/c56 0 2024-02-28T16:31:01.206 INFO:tasks.workunit.client.0.smithi001.stdout:2/211: truncate d0/d2/d6/d12/f41 160963 0 2024-02-28T16:31:01.208 INFO:tasks.workunit.client.0.smithi001.stdout:5/178: rename dc/d12/f19 to dc/d12/f3f 0 2024-02-28T16:31:01.210 INFO:tasks.workunit.client.0.smithi001.stdout:8/229: symlink d4/d7/d22/d27/l41 0 2024-02-28T16:31:01.215 INFO:tasks.workunit.client.0.smithi001.stdout:2/212: symlink d0/d2/d3/de/l45 0 2024-02-28T16:31:01.224 INFO:tasks.workunit.client.0.smithi001.stdout:2/213: fdatasync d0/d2/d3/f34 0 2024-02-28T16:31:01.229 INFO:tasks.workunit.client.0.smithi001.stdout:7/197: dwrite d3/f22 [0,4194304] 0 2024-02-28T16:31:01.230 INFO:tasks.workunit.client.0.smithi001.stdout:3/161: dwrite d4/d18/f19 [4194304,4194304] 0 2024-02-28T16:31:01.232 INFO:tasks.workunit.client.0.smithi001.stdout:5/179: write dc/f1d [1725958,30869] 0 2024-02-28T16:31:01.232 INFO:tasks.workunit.client.0.smithi001.stdout:9/257: dwrite d5/d27/d34/d50/f52 [0,4194304] 0 2024-02-28T16:31:01.233 INFO:tasks.workunit.client.0.smithi001.stdout:4/150: sync 2024-02-28T16:31:01.233 INFO:tasks.workunit.client.0.smithi001.stdout:6/214: sync 2024-02-28T16:31:01.233 INFO:tasks.workunit.client.0.smithi001.stdout:1/345: dread d0/d9/ff [4194304,4194304] 0 2024-02-28T16:31:01.235 INFO:tasks.workunit.client.0.smithi001.stdout:3/162: readlink d4/d18/l15 0 2024-02-28T16:31:01.238 INFO:tasks.workunit.client.0.smithi001.stdout:6/215: readlink d9/l17 0 2024-02-28T16:31:01.238 INFO:tasks.workunit.client.0.smithi001.stdout:7/198: dread - d3/d17/f2a zero size 2024-02-28T16:31:01.241 INFO:tasks.workunit.client.0.smithi001.stdout:0/200: dread f15 [0,4194304] 0 2024-02-28T16:31:01.242 INFO:tasks.workunit.client.0.smithi001.stdout:9/258: dread - d5/f19 zero size 2024-02-28T16:31:01.242 INFO:tasks.workunit.client.0.smithi001.stdout:1/346: fsync d0/d9/d1a/f4c 0 2024-02-28T16:31:01.244 INFO:tasks.workunit.client.0.smithi001.stdout:3/163: readlink d4/d18/l15 0 2024-02-28T16:31:01.250 INFO:tasks.workunit.client.0.smithi001.stdout:8/230: dwrite d4/d7/d22/d27/f30 [0,4194304] 0 2024-02-28T16:31:01.252 INFO:tasks.workunit.client.0.smithi001.stdout:3/164: truncate d4/d1e/f21 4796216 0 2024-02-28T16:31:01.253 INFO:tasks.workunit.client.0.smithi001.stdout:2/214: truncate d0/d2/d3/f29 603781 0 2024-02-28T16:31:01.257 INFO:tasks.workunit.client.0.smithi001.stdout:4/151: write d3/d5/dd/f2a [665881,48431] 0 2024-02-28T16:31:01.258 INFO:tasks.workunit.client.0.smithi001.stdout:9/259: mknod d5/d45/c57 0 2024-02-28T16:31:01.258 INFO:tasks.workunit.client.0.smithi001.stdout:2/215: fsync d0/d2/fb 0 2024-02-28T16:31:01.258 INFO:tasks.workunit.client.0.smithi001.stdout:0/201: fdatasync d18/f33 0 2024-02-28T16:31:01.260 INFO:tasks.workunit.client.0.smithi001.stdout:7/199: mkdir d3/d1c/d41 0 2024-02-28T16:31:01.262 INFO:tasks.workunit.client.0.smithi001.stdout:7/200: fdatasync d3/d1c/f33 0 2024-02-28T16:31:01.263 INFO:tasks.workunit.client.0.smithi001.stdout:1/347: symlink d0/d9/d1a/d2b/d26/l75 0 2024-02-28T16:31:01.268 INFO:tasks.workunit.client.0.smithi001.stdout:5/180: rename dc/l25 to dc/d12/l40 0 2024-02-28T16:31:01.269 INFO:tasks.workunit.client.0.smithi001.stdout:2/216: read d0/d2/d3/d2c/f37 [152260,24507] 0 2024-02-28T16:31:01.277 INFO:tasks.workunit.client.0.smithi001.stdout:3/165: creat d4/f35 x:0 0 0 2024-02-28T16:31:01.277 INFO:tasks.workunit.client.0.smithi001.stdout:2/217: mkdir d0/d2/d6/d12/d46 0 2024-02-28T16:31:01.278 INFO:tasks.workunit.client.0.smithi001.stdout:8/231: rename d4/c1c to d4/d7/d22/c42 0 2024-02-28T16:31:01.283 INFO:tasks.workunit.client.0.smithi001.stdout:8/232: write d4/d7/f10 [2099829,55052] 0 2024-02-28T16:31:01.283 INFO:tasks.workunit.client.0.smithi001.stdout:6/216: dwrite d9/db/d19/f4b [4194304,4194304] 0 2024-02-28T16:31:01.285 INFO:tasks.workunit.client.0.smithi001.stdout:2/218: rmdir d0 39 2024-02-28T16:31:01.285 INFO:tasks.workunit.client.0.smithi001.stdout:4/152: getdents d3 0 2024-02-28T16:31:01.287 INFO:tasks.workunit.client.0.smithi001.stdout:0/202: getdents d18/d22/d2b 0 2024-02-28T16:31:01.291 INFO:tasks.workunit.client.0.smithi001.stdout:4/153: readlink d3/d5/d7/d1f/l29 0 2024-02-28T16:31:01.293 INFO:tasks.workunit.client.0.smithi001.stdout:7/201: truncate d3/fa 3637947 0 2024-02-28T16:31:01.300 INFO:tasks.workunit.client.0.smithi001.stdout:3/166: symlink d4/d1e/d29/l36 0 2024-02-28T16:31:01.302 INFO:tasks.workunit.client.0.smithi001.stdout:6/217: creat d9/de/f4c x:0 0 0 2024-02-28T16:31:01.303 INFO:tasks.workunit.client.0.smithi001.stdout:9/260: dwrite d5/d27/d42/f53 [0,4194304] 0 2024-02-28T16:31:01.305 INFO:tasks.workunit.client.0.smithi001.stdout:0/203: creat d18/d22/d2b/d36/f3c x:0 0 0 2024-02-28T16:31:01.305 INFO:tasks.workunit.client.0.smithi001.stdout:1/348: getdents d0/d9/d1a/d2b/d26/d27/d30/d4a/d68 0 2024-02-28T16:31:01.309 INFO:tasks.workunit.client.0.smithi001.stdout:5/181: dwrite dc/df/f35 [0,4194304] 0 2024-02-28T16:31:01.309 INFO:tasks.workunit.client.0.smithi001.stdout:7/202: mkdir d3/d17/d3d/d42 0 2024-02-28T16:31:01.311 INFO:tasks.workunit.client.0.smithi001.stdout:0/204: dread - d18/d22/d2f/f1f zero size 2024-02-28T16:31:01.312 INFO:tasks.workunit.client.0.smithi001.stdout:3/167: creat d4/d18/f37 x:0 0 0 2024-02-28T16:31:01.312 INFO:tasks.workunit.client.0.smithi001.stdout:6/218: creat d9/de/f4d x:0 0 0 2024-02-28T16:31:01.317 INFO:tasks.workunit.client.0.smithi001.stdout:4/154: link d3/l19 d3/d1c/l33 0 2024-02-28T16:31:01.321 INFO:tasks.workunit.client.0.smithi001.stdout:6/219: chown d9/db/l3d 1878343 1 2024-02-28T16:31:01.322 INFO:tasks.workunit.client.0.smithi001.stdout:8/233: dwrite d4/d7/d22/d27/f2f [0,4194304] 0 2024-02-28T16:31:01.322 INFO:tasks.workunit.client.0.smithi001.stdout:4/155: fsync d3/d5/d7/f21 0 2024-02-28T16:31:01.325 INFO:tasks.workunit.client.0.smithi001.stdout:6/220: chown l4 0 1 2024-02-28T16:31:01.325 INFO:tasks.workunit.client.0.smithi001.stdout:3/168: truncate d4/d1e/f27 393424 0 2024-02-28T16:31:01.326 INFO:tasks.workunit.client.0.smithi001.stdout:3/169: chown d4/d18/f11 29385805 1 2024-02-28T16:31:01.327 INFO:tasks.workunit.client.0.smithi001.stdout:4/156: write d3/d5/d7/f21 [1666242,15337] 0 2024-02-28T16:31:01.329 INFO:tasks.workunit.client.0.smithi001.stdout:5/182: rmdir dc/df 39 2024-02-28T16:31:01.329 INFO:tasks.workunit.client.0.smithi001.stdout:0/205: mkdir d18/d22/d3d 0 2024-02-28T16:31:01.330 INFO:tasks.workunit.client.0.smithi001.stdout:5/183: dread - dc/f30 zero size 2024-02-28T16:31:01.331 INFO:tasks.workunit.client.0.smithi001.stdout:9/261: rmdir d5/d47/d33 39 2024-02-28T16:31:01.337 INFO:tasks.workunit.client.0.smithi001.stdout:6/221: mkdir d9/db/d10/d15/d4e 0 2024-02-28T16:31:01.339 INFO:tasks.workunit.client.0.smithi001.stdout:3/170: unlink d4/d18/l15 0 2024-02-28T16:31:01.341 INFO:tasks.workunit.client.0.smithi001.stdout:2/219: truncate d0/d2/d6/f1f 382415 0 2024-02-28T16:31:01.343 INFO:tasks.workunit.client.0.smithi001.stdout:5/184: mkdir dc/d12/d41 0 2024-02-28T16:31:01.345 INFO:tasks.workunit.client.0.smithi001.stdout:2/220: write d0/d2/d6/d12/f27 [3170012,45685] 0 2024-02-28T16:31:01.346 INFO:tasks.workunit.client.0.smithi001.stdout:6/222: creat d9/f4f x:0 0 0 2024-02-28T16:31:01.347 INFO:tasks.workunit.client.0.smithi001.stdout:7/203: link d3/d17/c25 d3/d17/c43 0 2024-02-28T16:31:01.348 INFO:tasks.workunit.client.0.smithi001.stdout:9/262: mkdir d5/d47/d58 0 2024-02-28T16:31:01.349 INFO:tasks.workunit.client.0.smithi001.stdout:1/349: link d0/d9/d56/f29 d0/d9/d1a/d2b/d26/d27/d30/f76 0 2024-02-28T16:31:01.351 INFO:tasks.workunit.client.0.smithi001.stdout:8/234: creat d4/f43 x:0 0 0 2024-02-28T16:31:01.352 INFO:tasks.workunit.client.0.smithi001.stdout:5/185: rmdir dc/d1a/d21 39 2024-02-28T16:31:01.354 INFO:tasks.workunit.client.0.smithi001.stdout:3/171: mkdir d4/d38 0 2024-02-28T16:31:01.355 INFO:tasks.workunit.client.0.smithi001.stdout:4/157: creat d3/d5/f34 x:0 0 0 2024-02-28T16:31:01.356 INFO:tasks.workunit.client.0.smithi001.stdout:2/221: mknod d0/d2/c47 0 2024-02-28T16:31:01.357 INFO:tasks.workunit.client.0.smithi001.stdout:3/172: chown d4/d6/d7/d23/f2b 3165218 1 2024-02-28T16:31:01.357 INFO:tasks.workunit.client.0.smithi001.stdout:4/158: fsync d3/d1c/f22 0 2024-02-28T16:31:01.358 INFO:tasks.workunit.client.0.smithi001.stdout:6/223: symlink d9/de/l50 0 2024-02-28T16:31:01.370 INFO:tasks.workunit.client.0.smithi001.stdout:7/204: dwrite d3/d17/f3b [0,4194304] 0 2024-02-28T16:31:01.371 INFO:tasks.workunit.client.0.smithi001.stdout:7/205: chown d3/d17/d3d/l3f 56714 1 2024-02-28T16:31:01.374 INFO:tasks.workunit.client.0.smithi001.stdout:2/222: unlink d0/f1d 0 2024-02-28T16:31:01.375 INFO:tasks.workunit.client.0.smithi001.stdout:6/224: symlink d9/db/d10/d15/d4e/l51 0 2024-02-28T16:31:01.376 INFO:tasks.workunit.client.0.smithi001.stdout:8/235: dread d4/d7/f23 [0,4194304] 0 2024-02-28T16:31:01.376 INFO:tasks.workunit.client.0.smithi001.stdout:7/206: chown d3/l14 8183 1 2024-02-28T16:31:01.380 INFO:tasks.workunit.client.0.smithi001.stdout:5/186: link dc/d1a/d21/d2d/d3c/f3d dc/d1a/d21/d2d/f42 0 2024-02-28T16:31:01.384 INFO:tasks.workunit.client.0.smithi001.stdout:5/187: chown dc/c1b 37212104 1 2024-02-28T16:31:01.384 INFO:tasks.workunit.client.0.smithi001.stdout:7/207: mkdir d3/d1c/d41/d44 0 2024-02-28T16:31:01.389 INFO:tasks.workunit.client.0.smithi001.stdout:2/223: write d0/f33 [754379,52184] 0 2024-02-28T16:31:01.389 INFO:tasks.workunit.client.0.smithi001.stdout:3/173: dwrite d4/d1e/f21 [0,4194304] 0 2024-02-28T16:31:01.391 INFO:tasks.workunit.client.0.smithi001.stdout:7/208: creat d3/f45 x:0 0 0 2024-02-28T16:31:01.394 INFO:tasks.workunit.client.0.smithi001.stdout:1/350: dwrite d0/d9/d1a/f33 [0,4194304] 0 2024-02-28T16:31:01.396 INFO:tasks.workunit.client.0.smithi001.stdout:3/174: mknod d4/d6/d7/c39 0 2024-02-28T16:31:01.397 INFO:tasks.workunit.client.0.smithi001.stdout:2/224: creat d0/d2/d3/de/d22/f48 x:0 0 0 2024-02-28T16:31:01.400 INFO:tasks.workunit.client.0.smithi001.stdout:1/351: write d0/d9/d56/f6c [806379,100370] 0 2024-02-28T16:31:01.404 INFO:tasks.workunit.client.0.smithi001.stdout:7/209: symlink d3/d17/d3d/d42/l46 0 2024-02-28T16:31:01.406 INFO:tasks.workunit.client.0.smithi001.stdout:2/225: symlink d0/d2/d6/d12/d46/l49 0 2024-02-28T16:31:01.409 INFO:tasks.workunit.client.0.smithi001.stdout:8/236: dwrite d4/d7/f40 [0,4194304] 0 2024-02-28T16:31:01.411 INFO:tasks.workunit.client.0.smithi001.stdout:7/210: creat d3/d17/d3d/d42/f47 x:0 0 0 2024-02-28T16:31:01.411 INFO:tasks.workunit.client.0.smithi001.stdout:6/225: dwrite d9/db/d10/d15/f2f [0,4194304] 0 2024-02-28T16:31:01.414 INFO:tasks.workunit.client.0.smithi001.stdout:1/352: rename d0/c65 to d0/d9/d1a/d72/c77 0 2024-02-28T16:31:01.416 INFO:tasks.workunit.client.0.smithi001.stdout:1/353: dread - d0/d9/d56/d2f/f64 zero size 2024-02-28T16:31:01.420 INFO:tasks.workunit.client.0.smithi001.stdout:6/226: rmdir d9/de/d47 0 2024-02-28T16:31:01.423 INFO:tasks.workunit.client.0.smithi001.stdout:5/188: dread dc/f20 [0,4194304] 0 2024-02-28T16:31:01.424 INFO:tasks.workunit.client.0.smithi001.stdout:6/227: creat d9/f52 x:0 0 0 2024-02-28T16:31:01.427 INFO:tasks.workunit.client.0.smithi001.stdout:5/189: truncate dc/d1a/d21/d2d/f39 647253 0 2024-02-28T16:31:01.431 INFO:tasks.workunit.client.0.smithi001.stdout:1/354: symlink d0/d9/d1a/d2b/d26/d27/d30/d52/l78 0 2024-02-28T16:31:01.432 INFO:tasks.workunit.client.0.smithi001.stdout:6/228: write d9/db/d10/f16 [1547214,94582] 0 2024-02-28T16:31:01.435 INFO:tasks.workunit.client.0.smithi001.stdout:3/175: write d4/d6/fc [1794102,79005] 0 2024-02-28T16:31:01.439 INFO:tasks.workunit.client.0.smithi001.stdout:6/229: dread - d9/de/f2c zero size 2024-02-28T16:31:01.441 INFO:tasks.workunit.client.0.smithi001.stdout:9/263: dread d5/f12 [0,4194304] 0 2024-02-28T16:31:01.441 INFO:tasks.workunit.client.0.smithi001.stdout:6/230: fdatasync d9/d29/f48 0 2024-02-28T16:31:01.446 INFO:tasks.workunit.client.0.smithi001.stdout:6/231: mkdir d9/db/d10/d15/d53 0 2024-02-28T16:31:01.449 INFO:tasks.workunit.client.0.smithi001.stdout:5/190: dread f5 [0,4194304] 0 2024-02-28T16:31:01.454 INFO:tasks.workunit.client.0.smithi001.stdout:1/355: dread d0/d9/ff [4194304,4194304] 0 2024-02-28T16:31:01.454 INFO:tasks.workunit.client.0.smithi001.stdout:5/191: mknod dc/df/c43 0 2024-02-28T16:31:01.455 INFO:tasks.workunit.client.0.smithi001.stdout:2/226: dread d0/d2/d3/f29 [0,4194304] 0 2024-02-28T16:31:01.457 INFO:tasks.workunit.client.0.smithi001.stdout:3/176: dwrite d4/d6/d7/d23/f2b [0,4194304] 0 2024-02-28T16:31:01.457 INFO:tasks.workunit.client.0.smithi001.stdout:7/211: write d3/f1f [3865762,74] 0 2024-02-28T16:31:01.460 INFO:tasks.workunit.client.0.smithi001.stdout:5/192: chown dc/d12/c17 1371 1 2024-02-28T16:31:01.462 INFO:tasks.workunit.client.0.smithi001.stdout:1/356: read d0/d9/d1a/d2b/f24 [21993,13596] 0 2024-02-28T16:31:01.464 INFO:tasks.workunit.client.0.smithi001.stdout:3/177: write d4/d18/f37 [721520,118133] 0 2024-02-28T16:31:01.465 INFO:tasks.workunit.client.0.smithi001.stdout:2/227: creat d0/d2/d6/d12/d46/f4a x:0 0 0 2024-02-28T16:31:01.468 INFO:tasks.workunit.client.0.smithi001.stdout:6/232: rmdir d9/de 39 2024-02-28T16:31:01.470 INFO:tasks.workunit.client.0.smithi001.stdout:9/264: dwrite d5/d47/d33/f3d [0,4194304] 0 2024-02-28T16:31:01.474 INFO:tasks.workunit.client.0.smithi001.stdout:2/228: creat d0/d2/d3/d36/f4b x:0 0 0 2024-02-28T16:31:01.475 INFO:tasks.workunit.client.0.smithi001.stdout:6/233: mknod d9/d39/c54 0 2024-02-28T16:31:01.480 INFO:tasks.workunit.client.0.smithi001.stdout:6/234: mknod d9/d29/c55 0 2024-02-28T16:31:01.481 INFO:tasks.workunit.client.0.smithi001.stdout:9/265: truncate d5/f1d 5545299 0 2024-02-28T16:31:01.481 INFO:tasks.workunit.client.0.smithi001.stdout:6/235: stat d9/db/d10/f16 0 2024-02-28T16:31:01.483 INFO:tasks.workunit.client.0.smithi001.stdout:3/178: rename d4/d1e/d29/c2c to d4/d6/c3a 0 2024-02-28T16:31:01.485 INFO:tasks.workunit.client.0.smithi001.stdout:7/212: write d3/d17/f2f [239290,104905] 0 2024-02-28T16:31:01.486 INFO:tasks.workunit.client.0.smithi001.stdout:5/193: truncate fa 3016939 0 2024-02-28T16:31:01.487 INFO:tasks.workunit.client.0.smithi001.stdout:2/229: write d0/d2/d6/f1f [964319,83570] 0 2024-02-28T16:31:01.494 INFO:tasks.workunit.client.0.smithi001.stdout:2/230: symlink d0/d2/d3/de/d22/l4c 0 2024-02-28T16:31:01.498 INFO:tasks.workunit.client.0.smithi001.stdout:9/266: link d5/d27/d34/d50/c51 d5/d27/c59 0 2024-02-28T16:31:01.498 INFO:tasks.workunit.client.0.smithi001.stdout:2/231: chown d0/l7 218 1 2024-02-28T16:31:01.504 INFO:tasks.workunit.client.0.smithi001.stdout:9/267: fsync d5/f2e 0 2024-02-28T16:31:01.504 INFO:tasks.workunit.client.0.smithi001.stdout:2/232: readlink d0/d2/d3/de/d22/l4c 0 2024-02-28T16:31:01.504 INFO:tasks.workunit.client.0.smithi001.stdout:3/179: truncate d4/d18/f37 763678 0 2024-02-28T16:31:01.505 INFO:tasks.workunit.client.0.smithi001.stdout:7/213: dread d3/f1f [0,4194304] 0 2024-02-28T16:31:01.509 INFO:tasks.workunit.client.0.smithi001.stdout:5/194: dwrite dc/df/f2c [0,4194304] 0 2024-02-28T16:31:01.511 INFO:tasks.workunit.client.0.smithi001.stdout:7/214: write d3/f22 [3949488,100346] 0 2024-02-28T16:31:01.514 INFO:tasks.workunit.client.0.smithi001.stdout:5/195: unlink dc/df/f2c 0 2024-02-28T16:31:01.514 INFO:tasks.workunit.client.0.smithi001.stdout:3/180: getdents d4/d38 0 2024-02-28T16:31:01.519 INFO:tasks.workunit.client.0.smithi001.stdout:7/215: unlink d3/f1f 0 2024-02-28T16:31:01.519 INFO:tasks.workunit.client.0.smithi001.stdout:2/233: dread d0/d2/d6/f1f [0,4194304] 0 2024-02-28T16:31:01.520 INFO:tasks.workunit.client.0.smithi001.stdout:3/181: dread d4/d1e/f24 [0,4194304] 0 2024-02-28T16:31:01.520 INFO:tasks.workunit.client.0.smithi001.stdout:7/216: stat d3/c16 0 2024-02-28T16:31:01.525 INFO:tasks.workunit.client.0.smithi001.stdout:9/268: write d5/d47/f3a [5154230,3502] 0 2024-02-28T16:31:01.525 INFO:tasks.workunit.client.0.smithi001.stdout:5/196: rmdir dc 39 2024-02-28T16:31:01.530 INFO:tasks.workunit.client.0.smithi001.stdout:5/197: readlink dc/d1a/d21/l2e 0 2024-02-28T16:31:01.531 INFO:tasks.workunit.client.0.smithi001.stdout:9/269: fdatasync d5/f6 0 2024-02-28T16:31:01.535 INFO:tasks.workunit.client.0.smithi001.stdout:7/217: dread d3/d17/f3b [0,4194304] 0 2024-02-28T16:31:01.536 INFO:tasks.workunit.client.0.smithi001.stdout:7/218: stat d3/d1c/l23 0 2024-02-28T16:31:01.537 INFO:tasks.workunit.client.0.smithi001.stdout:2/234: dwrite d0/d2/d3/de/d22/f24 [0,4194304] 0 2024-02-28T16:31:01.538 INFO:tasks.workunit.client.0.smithi001.stdout:7/219: readlink d3/d1c/l29 0 2024-02-28T16:31:01.548 INFO:tasks.workunit.client.0.smithi001.stdout:5/198: dwrite dc/d12/f14 [0,4194304] 0 2024-02-28T16:31:01.577 INFO:tasks.workunit.client.0.smithi001.stdout:0/206: sync 2024-02-28T16:31:01.601 INFO:tasks.workunit.client.0.smithi001.stdout:9/270: mkdir d5/d27/d42/d5a 0 2024-02-28T16:31:01.604 INFO:tasks.workunit.client.0.smithi001.stdout:2/235: mknod d0/d2/d3/d2c/c4d 0 2024-02-28T16:31:01.604 INFO:tasks.workunit.client.0.smithi001.stdout:9/271: write d5/f6 [1288973,102153] 0 2024-02-28T16:31:01.607 INFO:tasks.workunit.client.0.smithi001.stdout:5/199: fsync dc/d12/f3f 0 2024-02-28T16:31:01.608 INFO:tasks.workunit.client.0.smithi001.stdout:0/207: symlink d18/d22/l3e 0 2024-02-28T16:31:01.611 INFO:tasks.workunit.client.0.smithi001.stdout:5/200: write f5 [2539534,79550] 0 2024-02-28T16:31:01.617 INFO:tasks.workunit.client.0.smithi001.stdout:2/236: link d0/d2/d3/de/l45 d0/d2/d3/d36/l4e 0 2024-02-28T16:31:01.618 INFO:tasks.workunit.client.0.smithi001.stdout:9/272: truncate d5/f18 2495579 0 2024-02-28T16:31:01.620 INFO:tasks.workunit.client.0.smithi001.stdout:0/208: write f9 [556990,108373] 0 2024-02-28T16:31:01.621 INFO:tasks.workunit.client.0.smithi001.stdout:9/273: read d5/d47/f37 [1894587,116804] 0 2024-02-28T16:31:01.621 INFO:tasks.workunit.client.0.smithi001.stdout:2/237: mknod d0/d2/c4f 0 2024-02-28T16:31:01.625 INFO:tasks.workunit.client.0.smithi001.stdout:2/238: mknod d0/d2/d6/c50 0 2024-02-28T16:31:01.627 INFO:tasks.workunit.client.0.smithi001.stdout:0/209: link f9 d18/d22/f3f 0 2024-02-28T16:31:01.630 INFO:tasks.workunit.client.0.smithi001.stdout:5/201: dwrite dc/d12/f3f [4194304,4194304] 0 2024-02-28T16:31:01.647 INFO:tasks.workunit.client.0.smithi001.stdout:5/202: link dc/df/f13 dc/d1a/d21/d2d/f44 0 2024-02-28T16:31:01.649 INFO:tasks.workunit.client.0.smithi001.stdout:2/239: dwrite d0/d2/d3/de/fa [0,4194304] 0 2024-02-28T16:31:01.649 INFO:tasks.workunit.client.0.smithi001.stdout:9/274: dwrite d5/d27/f2b [4194304,4194304] 0 2024-02-28T16:31:01.652 INFO:tasks.workunit.client.0.smithi001.stdout:0/210: dwrite d18/d22/d2f/f1f [0,4194304] 0 2024-02-28T16:31:01.654 INFO:tasks.workunit.client.0.smithi001.stdout:9/275: write d5/d47/f3a [3569580,30645] 0 2024-02-28T16:31:01.655 INFO:tasks.workunit.client.0.smithi001.stdout:2/240: chown d0/d2/d3/d36 7 1 2024-02-28T16:31:01.658 INFO:tasks.workunit.client.0.smithi001.stdout:9/276: read - d5/d27/f2c zero size 2024-02-28T16:31:01.660 INFO:tasks.workunit.client.0.smithi001.stdout:5/203: symlink dc/d12/d41/l45 0 2024-02-28T16:31:01.660 INFO:tasks.workunit.client.0.smithi001.stdout:2/241: creat d0/d2/d3/d2c/f51 x:0 0 0 2024-02-28T16:31:01.664 INFO:tasks.workunit.client.0.smithi001.stdout:0/211: rename d18/l2e to d18/d22/l40 0 2024-02-28T16:31:01.675 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:01 smithi066 ceph-mon[36236]: pgmap v44: 65 pgs: 65 active+clean; 547 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 489 KiB/s rd, 44 MiB/s wr, 211 op/s 2024-02-28T16:31:01.676 INFO:tasks.workunit.client.0.smithi001.stdout:2/242: dwrite d0/f40 [0,4194304] 0 2024-02-28T16:31:01.678 INFO:tasks.workunit.client.0.smithi001.stdout:2/243: fdatasync d0/d2/d3/de/d22/f38 0 2024-02-28T16:31:01.691 INFO:tasks.workunit.client.0.smithi001.stdout:0/212: creat d18/d22/d2b/f41 x:0 0 0 2024-02-28T16:31:01.694 INFO:tasks.workunit.client.0.smithi001.stdout:0/213: symlink d18/d31/l42 0 2024-02-28T16:31:01.695 INFO:tasks.workunit.client.0.smithi001.stdout:0/214: chown d18/d31/l42 134 1 2024-02-28T16:31:01.698 INFO:tasks.workunit.client.0.smithi001.stdout:0/215: unlink d18/d22/d2f/f1f 0 2024-02-28T16:31:01.701 INFO:tasks.workunit.client.0.smithi001.stdout:0/216: rename f9 to d18/d31/f43 0 2024-02-28T16:31:01.703 INFO:tasks.workunit.client.0.smithi001.stdout:0/217: rename l6 to d18/d22/l44 0 2024-02-28T16:31:01.704 INFO:tasks.workunit.client.0.smithi001.stdout:0/218: chown lb 0 1 2024-02-28T16:31:01.706 INFO:tasks.workunit.client.0.smithi001.stdout:0/219: mkdir d18/d22/d45 0 2024-02-28T16:31:01.709 INFO:tasks.workunit.client.0.smithi001.stdout:0/220: mkdir d18/d22/d2b/d36/d46 0 2024-02-28T16:31:01.711 INFO:tasks.workunit.client.0.smithi001.stdout:0/221: mkdir d18/d22/d47 0 2024-02-28T16:31:01.712 INFO:tasks.workunit.client.0.smithi001.stdout:0/222: dread - d18/d22/d2b/f41 zero size 2024-02-28T16:31:01.714 INFO:tasks.workunit.client.0.smithi001.stdout:0/223: mkdir d18/d22/d2b/d36/d48 0 2024-02-28T16:31:01.730 INFO:tasks.workunit.client.0.smithi001.stdout:0/224: dread f0 [0,4194304] 0 2024-02-28T16:31:01.746 INFO:tasks.workunit.client.0.smithi001.stdout:0/225: dwrite d18/d22/d2f/f24 [0,4194304] 0 2024-02-28T16:31:01.767 INFO:tasks.workunit.client.0.smithi001.stdout:4/159: sync 2024-02-28T16:31:01.769 INFO:tasks.workunit.client.0.smithi001.stdout:4/160: chown d3/d1c/f22 1064876015 1 2024-02-28T16:31:01.771 INFO:tasks.workunit.client.0.smithi001.stdout:0/226: creat d18/d22/d2b/d36/f49 x:0 0 0 2024-02-28T16:31:01.774 INFO:tasks.workunit.client.0.smithi001.stdout:0/227: fdatasync d18/d22/d2f/d1c/f21 0 2024-02-28T16:31:01.774 INFO:tasks.workunit.client.0.smithi001.stdout:4/161: rmdir d3/d5 39 2024-02-28T16:31:01.784 INFO:tasks.workunit.client.0.smithi001.stdout:4/162: symlink d3/d5/d7/d1f/d2e/l35 0 2024-02-28T16:31:01.786 INFO:tasks.workunit.client.0.smithi001.stdout:4/163: dread - d3/d1c/f22 zero size 2024-02-28T16:31:01.792 INFO:tasks.workunit.client.0.smithi001.stdout:4/164: dread d3/d5/dd/ff [0,4194304] 0 2024-02-28T16:31:01.796 INFO:tasks.workunit.client.0.smithi001.stdout:0/228: dwrite d18/d22/d2f/d1c/f21 [0,4194304] 0 2024-02-28T16:31:01.797 INFO:tasks.workunit.client.0.smithi001.stdout:0/229: readlink d18/d22/l3e 0 2024-02-28T16:31:01.808 INFO:tasks.workunit.client.0.smithi001.stdout:4/165: dwrite d3/d5/dd/f27 [0,4194304] 0 2024-02-28T16:31:01.814 INFO:tasks.workunit.client.0.smithi001.stdout:4/166: symlink d3/d2d/l36 0 2024-02-28T16:31:01.817 INFO:tasks.workunit.client.0.smithi001.stdout:0/230: dwrite d18/d22/d2f/d1c/f30 [0,4194304] 0 2024-02-28T16:31:01.822 INFO:tasks.workunit.client.0.smithi001.stdout:0/231: symlink d18/d22/d47/l4a 0 2024-02-28T16:31:01.831 INFO:tasks.workunit.client.0.smithi001.stdout:4/167: dwrite d3/d5/d7/f21 [0,4194304] 0 2024-02-28T16:31:01.834 INFO:tasks.workunit.client.0.smithi001.stdout:4/168: write d3/d5/dd/f28 [652747,109103] 0 2024-02-28T16:31:01.838 INFO:tasks.workunit.client.0.smithi001.stdout:4/169: write d3/f17 [8018030,92860] 0 2024-02-28T16:31:01.850 INFO:tasks.workunit.client.0.smithi001.stdout:4/170: dread d3/f12 [0,4194304] 0 2024-02-28T16:31:01.854 INFO:tasks.workunit.client.0.smithi001.stdout:4/171: truncate d3/d2d/f32 381618 0 2024-02-28T16:31:01.855 INFO:tasks.workunit.client.0.smithi001.stdout:4/172: fsync d3/d2d/f32 0 2024-02-28T16:31:01.859 INFO:tasks.workunit.client.0.smithi001.stdout:4/173: mkdir d3/d5/d7/d1f/d20/d37 0 2024-02-28T16:31:01.860 INFO:tasks.workunit.client.0.smithi001.stdout:0/232: dwrite d18/d22/d2b/f37 [0,4194304] 0 2024-02-28T16:31:01.863 INFO:tasks.workunit.client.0.smithi001.stdout:4/174: unlink d3/d5/d7/d1f/d20/l24 0 2024-02-28T16:31:01.865 INFO:tasks.workunit.client.0.smithi001.stdout:0/233: creat d18/d22/d2b/d36/d46/f4b x:0 0 0 2024-02-28T16:31:01.867 INFO:tasks.workunit.client.0.smithi001.stdout:0/234: fdatasync d18/d22/f3f 0 2024-02-28T16:31:01.876 INFO:tasks.workunit.client.0.smithi001.stdout:4/175: dread f2 [0,4194304] 0 2024-02-28T16:31:01.878 INFO:tasks.workunit.client.0.smithi001.stdout:4/176: truncate d3/d5/dd/f2a 4620370 0 2024-02-28T16:31:01.881 INFO:tasks.workunit.client.0.smithi001.stdout:4/177: write d3/d5/d7/d1f/d20/f23 [75789,60052] 0 2024-02-28T16:31:01.882 INFO:tasks.workunit.client.0.smithi001.stdout:0/235: dread d18/d22/d2b/f37 [0,4194304] 0 2024-02-28T16:31:01.884 INFO:tasks.workunit.client.0.smithi001.stdout:4/178: unlink d3/d5/d7/l16 0 2024-02-28T16:31:01.886 INFO:tasks.workunit.client.0.smithi001.stdout:8/237: sync 2024-02-28T16:31:01.886 INFO:tasks.workunit.client.0.smithi001.stdout:0/236: creat d18/d22/d45/f4c x:0 0 0 2024-02-28T16:31:01.889 INFO:tasks.workunit.client.0.smithi001.stdout:8/238: mknod d4/d7/c44 0 2024-02-28T16:31:01.892 INFO:tasks.workunit.client.0.smithi001.stdout:4/179: rmdir d3/d5/d7/d1f/d20/d37 0 2024-02-28T16:31:01.895 INFO:tasks.workunit.client.0.smithi001.stdout:0/237: rename d18/d22/d45 to d18/d22/d2b/d36/d4d 0 2024-02-28T16:31:01.899 INFO:tasks.workunit.client.0.smithi001.stdout:4/180: rename d3/d5/d7/d1f/d20/f23 to d3/d5/f38 0 2024-02-28T16:31:01.972 INFO:tasks.workunit.client.0.smithi001.stdout:4/181: dread - d3/d5/f34 zero size 2024-02-28T16:31:01.976 INFO:tasks.workunit.client.0.smithi001.stdout:0/238: rename f0 to d18/d22/d2f/f4e 0 2024-02-28T16:31:01.977 INFO:tasks.workunit.client.0.smithi001.stdout:4/182: read d3/d5/f38 [31976,13976] 0 2024-02-28T16:31:01.978 INFO:tasks.workunit.client.0.smithi001.stdout:8/239: dwrite d4/f43 [0,4194304] 0 2024-02-28T16:31:01.985 INFO:tasks.workunit.client.0.smithi001.stdout:0/239: unlink f12 0 2024-02-28T16:31:02.007 INFO:tasks.workunit.client.0.smithi001.stdout:1/357: sync 2024-02-28T16:31:02.013 INFO:tasks.workunit.client.0.smithi001.stdout:4/183: creat d3/d5/d7/d1f/d2e/f39 x:0 0 0 2024-02-28T16:31:02.013 INFO:tasks.workunit.client.0.smithi001.stdout:0/240: truncate d18/d31/f43 1706167 0 2024-02-28T16:31:02.017 INFO:tasks.workunit.client.0.smithi001.stdout:0/241: chown d18/d22/d2b/d36/d46 72438 1 2024-02-28T16:31:02.018 INFO:tasks.workunit.client.0.smithi001.stdout:4/184: truncate d3/d5/f38 177930 0 2024-02-28T16:31:02.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:01 smithi001 ceph-mon[28991]: pgmap v44: 65 pgs: 65 active+clean; 547 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 489 KiB/s rd, 44 MiB/s wr, 211 op/s 2024-02-28T16:31:02.024 INFO:tasks.workunit.client.0.smithi001.stdout:0/242: mkdir d18/d31/d4f 0 2024-02-28T16:31:02.026 INFO:tasks.workunit.client.0.smithi001.stdout:1/358: mknod d0/d9/d1a/d2b/d26/d27/d30/d4a/c79 0 2024-02-28T16:31:02.026 INFO:tasks.workunit.client.0.smithi001.stdout:8/240: dwrite f1 [4194304,4194304] 0 2024-02-28T16:31:02.027 INFO:tasks.workunit.client.0.smithi001.stdout:4/185: mkdir d3/d5/d7/d1f/d3a 0 2024-02-28T16:31:02.030 INFO:tasks.workunit.client.0.smithi001.stdout:1/359: rmdir d0/d9/d1a/d2b 39 2024-02-28T16:31:02.032 INFO:tasks.workunit.client.0.smithi001.stdout:8/241: rename d4/d7/d22/d27/c2d to d4/d32/c45 0 2024-02-28T16:31:02.035 INFO:tasks.workunit.client.0.smithi001.stdout:4/186: creat d3/f3b x:0 0 0 2024-02-28T16:31:02.035 INFO:tasks.workunit.client.0.smithi001.stdout:8/242: stat d4/d7/d22/c39 0 2024-02-28T16:31:02.037 INFO:tasks.workunit.client.0.smithi001.stdout:8/243: fdatasync d4/f3b 0 2024-02-28T16:31:02.038 INFO:tasks.workunit.client.0.smithi001.stdout:1/360: dread - d0/d9/d1a/d2b/d26/d27/d30/f74 zero size 2024-02-28T16:31:02.044 INFO:tasks.workunit.client.0.smithi001.stdout:4/187: creat d3/d2d/f3c x:0 0 0 2024-02-28T16:31:02.047 INFO:tasks.workunit.client.0.smithi001.stdout:0/243: write d18/d22/f3f [2507242,25926] 0 2024-02-28T16:31:02.051 INFO:tasks.workunit.client.0.smithi001.stdout:8/244: truncate d4/d7/d22/d27/f3a 3510922 0 2024-02-28T16:31:02.053 INFO:tasks.workunit.client.0.smithi001.stdout:0/244: creat d18/d22/d3d/f50 x:0 0 0 2024-02-28T16:31:02.055 INFO:tasks.workunit.client.0.smithi001.stdout:0/245: readlink d18/d22/d2b/l39 0 2024-02-28T16:31:02.057 INFO:tasks.workunit.client.0.smithi001.stdout:1/361: dwrite d0/d9/fa [0,4194304] 0 2024-02-28T16:31:02.062 INFO:tasks.workunit.client.0.smithi001.stdout:1/362: mknod d0/d9/d1a/d2b/d26/c7a 0 2024-02-28T16:31:02.067 INFO:tasks.workunit.client.0.smithi001.stdout:3/182: sync 2024-02-28T16:31:02.067 INFO:tasks.workunit.client.0.smithi001.stdout:7/220: sync 2024-02-28T16:31:02.067 INFO:tasks.workunit.client.0.smithi001.stdout:6/236: sync 2024-02-28T16:31:02.070 INFO:tasks.workunit.client.0.smithi001.stdout:4/188: dread d3/f17 [0,4194304] 0 2024-02-28T16:31:02.074 INFO:tasks.workunit.client.0.smithi001.stdout:7/221: creat d3/d17/d3d/f48 x:0 0 0 2024-02-28T16:31:02.075 INFO:tasks.workunit.client.0.smithi001.stdout:3/183: mknod d4/d6/d7/c3b 0 2024-02-28T16:31:02.077 INFO:tasks.workunit.client.0.smithi001.stdout:1/363: mknod d0/d9/c7b 0 2024-02-28T16:31:02.078 INFO:tasks.workunit.client.0.smithi001.stdout:6/237: creat d9/d29/d2b/f56 x:0 0 0 2024-02-28T16:31:02.079 INFO:tasks.workunit.client.0.smithi001.stdout:4/189: mknod d3/d1c/c3d 0 2024-02-28T16:31:02.080 INFO:tasks.workunit.client.0.smithi001.stdout:8/245: dwrite d4/d7/d22/d27/f2f [0,4194304] 0 2024-02-28T16:31:02.080 INFO:tasks.workunit.client.0.smithi001.stdout:3/184: dread - d4/d1e/f33 zero size 2024-02-28T16:31:02.082 INFO:tasks.workunit.client.0.smithi001.stdout:6/238: read - d9/d29/f48 zero size 2024-02-28T16:31:02.083 INFO:tasks.workunit.client.0.smithi001.stdout:0/246: dwrite d18/d22/d3d/f50 [0,4194304] 0 2024-02-28T16:31:02.090 INFO:tasks.workunit.client.0.smithi001.stdout:8/246: creat d4/d32/f46 x:0 0 0 2024-02-28T16:31:02.090 INFO:tasks.workunit.client.0.smithi001.stdout:7/222: getdents d3 0 2024-02-28T16:31:02.091 INFO:tasks.workunit.client.0.smithi001.stdout:1/364: creat d0/d9/d56/d2f/f7c x:0 0 0 2024-02-28T16:31:02.092 INFO:tasks.workunit.client.0.smithi001.stdout:8/247: readlink d4/d7/lb 0 2024-02-28T16:31:02.097 INFO:tasks.workunit.client.0.smithi001.stdout:3/185: mkdir d4/d3c 0 2024-02-28T16:31:02.097 INFO:tasks.workunit.client.0.smithi001.stdout:1/365: stat d0/d9/d1a/d2b/d26/d27/d30/d52/d6b/f6e 0 2024-02-28T16:31:02.100 INFO:tasks.workunit.client.0.smithi001.stdout:8/248: mkdir d4/d47 0 2024-02-28T16:31:02.102 INFO:tasks.workunit.client.0.smithi001.stdout:0/247: dread d18/d22/d2f/d1c/f21 [0,4194304] 0 2024-02-28T16:31:02.106 INFO:tasks.workunit.client.0.smithi001.stdout:4/190: creat d3/f3e x:0 0 0 2024-02-28T16:31:02.108 INFO:tasks.workunit.client.0.smithi001.stdout:0/248: readlink d18/d22/d2f/d1c/l20 0 2024-02-28T16:31:02.108 INFO:tasks.workunit.client.0.smithi001.stdout:5/204: sync 2024-02-28T16:31:02.109 INFO:tasks.workunit.client.0.smithi001.stdout:8/249: creat d4/d7/d22/f48 x:0 0 0 2024-02-28T16:31:02.109 INFO:tasks.workunit.client.0.smithi001.stdout:9/277: sync 2024-02-28T16:31:02.109 INFO:tasks.workunit.client.0.smithi001.stdout:2/244: sync 2024-02-28T16:31:02.114 INFO:tasks.workunit.client.0.smithi001.stdout:4/191: creat d3/d5/d7/d1f/d2e/f3f x:0 0 0 2024-02-28T16:31:02.118 INFO:tasks.workunit.client.0.smithi001.stdout:2/245: rmdir d0/d2/d3/d2c 39 2024-02-28T16:31:02.119 INFO:tasks.workunit.client.0.smithi001.stdout:9/278: symlink d5/d47/l5b 0 2024-02-28T16:31:02.120 INFO:tasks.workunit.client.0.smithi001.stdout:3/186: dread d4/d6/fc [0,4194304] 0 2024-02-28T16:31:02.129 INFO:tasks.workunit.client.0.smithi001.stdout:6/239: dwrite d9/db/d19/f37 [0,4194304] 0 2024-02-28T16:31:02.130 INFO:tasks.workunit.client.0.smithi001.stdout:1/366: dread d0/d9/f54 [0,4194304] 0 2024-02-28T16:31:02.131 INFO:tasks.workunit.client.0.smithi001.stdout:7/223: dwrite d3/d17/f32 [0,4194304] 0 2024-02-28T16:31:02.132 INFO:tasks.workunit.client.0.smithi001.stdout:5/205: truncate dc/f1d 3951790 0 2024-02-28T16:31:02.134 INFO:tasks.workunit.client.0.smithi001.stdout:3/187: fsync d4/d18/f37 0 2024-02-28T16:31:02.135 INFO:tasks.workunit.client.0.smithi001.stdout:8/250: link d4/d7/d22/d27/l33 d4/l49 0 2024-02-28T16:31:02.135 INFO:tasks.workunit.client.0.smithi001.stdout:2/246: truncate d0/d2/d3/de/d22/f38 225030 0 2024-02-28T16:31:02.135 INFO:tasks.workunit.client.0.smithi001.stdout:4/192: creat d3/d5/f40 x:0 0 0 2024-02-28T16:31:02.138 INFO:tasks.workunit.client.0.smithi001.stdout:1/367: chown d0/d9/d56/f60 520460 1 2024-02-28T16:31:02.141 INFO:tasks.workunit.client.0.smithi001.stdout:5/206: write dc/f30 [646648,28638] 0 2024-02-28T16:31:02.143 INFO:tasks.workunit.client.0.smithi001.stdout:2/247: write d0/d2/f21 [227902,77923] 0 2024-02-28T16:31:02.148 INFO:tasks.workunit.client.0.smithi001.stdout:8/251: rmdir d4 39 2024-02-28T16:31:02.154 INFO:tasks.workunit.client.0.smithi001.stdout:9/279: getdents d5/d27/d42/d48 0 2024-02-28T16:31:02.155 INFO:tasks.workunit.client.0.smithi001.stdout:1/368: write d0/d9/d56/f60 [952589,40111] 0 2024-02-28T16:31:02.155 INFO:tasks.workunit.client.0.smithi001.stdout:5/207: truncate dc/df/f34 1227645 0 2024-02-28T16:31:02.160 INFO:tasks.workunit.client.0.smithi001.stdout:0/249: dwrite d18/d22/d2f/d1c/f38 [0,4194304] 0 2024-02-28T16:31:02.160 INFO:tasks.workunit.client.0.smithi001.stdout:8/252: creat d4/d7/d22/d27/f4a x:0 0 0 2024-02-28T16:31:02.165 INFO:tasks.workunit.client.0.smithi001.stdout:1/369: write d0/d9/d1a/d2b/d26/d27/d30/d4a/f70 [587493,119282] 0 2024-02-28T16:31:02.170 INFO:tasks.workunit.client.0.smithi001.stdout:5/208: creat dc/d1a/d21/d2d/d3c/f46 x:0 0 0 2024-02-28T16:31:02.170 INFO:tasks.workunit.client.0.smithi001.stdout:7/224: truncate d3/f10 823979 0 2024-02-28T16:31:02.174 INFO:tasks.workunit.client.0.smithi001.stdout:9/280: getdents d5 0 2024-02-28T16:31:02.176 INFO:tasks.workunit.client.0.smithi001.stdout:7/225: write d3/d17/f2a [172854,4472] 0 2024-02-28T16:31:02.177 INFO:tasks.workunit.client.0.smithi001.stdout:3/188: dwrite d4/f26 [0,4194304] 0 2024-02-28T16:31:02.188 INFO:tasks.workunit.client.0.smithi001.stdout:9/281: chown d5/c38 0 1 2024-02-28T16:31:02.192 INFO:tasks.workunit.client.0.smithi001.stdout:7/226: mkdir d3/d1c/d41/d49 0 2024-02-28T16:31:02.194 INFO:tasks.workunit.client.0.smithi001.stdout:3/189: write d4/d6/d7/d23/f34 [686442,5120] 0 2024-02-28T16:31:02.194 INFO:tasks.workunit.client.0.smithi001.stdout:6/240: dread d9/db/f13 [0,4194304] 0 2024-02-28T16:31:02.203 INFO:tasks.workunit.client.0.smithi001.stdout:7/227: stat d3/d17/d3d/d42 0 2024-02-28T16:31:02.206 INFO:tasks.workunit.client.0.smithi001.stdout:5/209: dread dc/df/f34 [0,4194304] 0 2024-02-28T16:31:02.210 INFO:tasks.workunit.client.0.smithi001.stdout:0/250: dwrite d18/d22/d2f/f24 [0,4194304] 0 2024-02-28T16:31:02.213 INFO:tasks.workunit.client.0.smithi001.stdout:7/228: fsync d3/fa 0 2024-02-28T16:31:02.214 INFO:tasks.workunit.client.0.smithi001.stdout:7/229: readlink d3/d1c/l3c 0 2024-02-28T16:31:02.217 INFO:tasks.workunit.client.0.smithi001.stdout:4/193: dread d3/fa [0,4194304] 0 2024-02-28T16:31:02.217 INFO:tasks.workunit.client.0.smithi001.stdout:2/248: dwrite d0/d2/d3/d2c/f51 [0,4194304] 0 2024-02-28T16:31:02.221 INFO:tasks.workunit.client.0.smithi001.stdout:4/194: fsync d3/d5/f34 0 2024-02-28T16:31:02.221 INFO:tasks.workunit.client.0.smithi001.stdout:5/210: creat dc/d1a/d21/d2d/d3c/f47 x:0 0 0 2024-02-28T16:31:02.222 INFO:tasks.workunit.client.0.smithi001.stdout:9/282: dread d5/f18 [0,4194304] 0 2024-02-28T16:31:02.225 INFO:tasks.workunit.client.0.smithi001.stdout:4/195: stat d3/d5/d7/d1f/d20/f31 0 2024-02-28T16:31:02.225 INFO:tasks.workunit.client.0.smithi001.stdout:5/211: chown dc/df/c43 21 1 2024-02-28T16:31:02.226 INFO:tasks.workunit.client.0.smithi001.stdout:1/370: dwrite d0/d9/d1a/d2b/f6d [0,4194304] 0 2024-02-28T16:31:02.229 INFO:tasks.workunit.client.0.smithi001.stdout:7/230: mknod d3/c4a 0 2024-02-28T16:31:02.233 INFO:tasks.workunit.client.0.smithi001.stdout:7/231: chown d3/d17/f1b 32325 1 2024-02-28T16:31:02.233 INFO:tasks.workunit.client.0.smithi001.stdout:2/249: creat d0/d2/d3/de/d22/f52 x:0 0 0 2024-02-28T16:31:02.234 INFO:tasks.workunit.client.0.smithi001.stdout:6/241: creat d9/db/d10/f57 x:0 0 0 2024-02-28T16:31:02.234 INFO:tasks.workunit.client.0.smithi001.stdout:9/283: creat d5/d27/f5c x:0 0 0 2024-02-28T16:31:02.236 INFO:tasks.workunit.client.0.smithi001.stdout:6/242: readlink d9/l20 0 2024-02-28T16:31:02.238 INFO:tasks.workunit.client.0.smithi001.stdout:5/212: creat dc/d1a/d21/f48 x:0 0 0 2024-02-28T16:31:02.243 INFO:tasks.workunit.client.0.smithi001.stdout:4/196: dread d3/d5/dd/f28 [0,4194304] 0 2024-02-28T16:31:02.243 INFO:tasks.workunit.client.0.smithi001.stdout:2/250: symlink d0/d2/l53 0 2024-02-28T16:31:02.244 INFO:tasks.workunit.client.0.smithi001.stdout:9/284: write d5/d27/d42/d48/f4c [856682,36981] 0 2024-02-28T16:31:02.244 INFO:tasks.workunit.client.0.smithi001.stdout:3/190: dwrite d4/d6/d7/f2f [0,4194304] 0 2024-02-28T16:31:02.246 INFO:tasks.workunit.client.0.smithi001.stdout:4/197: fsync d3/f3e 0 2024-02-28T16:31:02.248 INFO:tasks.workunit.client.0.smithi001.stdout:0/251: getdents d18/d22/d2b 0 2024-02-28T16:31:02.248 INFO:tasks.workunit.client.0.smithi001.stdout:4/198: stat d3/d5/f40 0 2024-02-28T16:31:02.249 INFO:tasks.workunit.client.0.smithi001.stdout:5/213: creat dc/d1a/d21/d2d/d3c/f49 x:0 0 0 2024-02-28T16:31:02.256 INFO:tasks.workunit.client.0.smithi001.stdout:3/191: mknod d4/d1e/c3d 0 2024-02-28T16:31:02.256 INFO:tasks.workunit.client.0.smithi001.stdout:9/285: creat d5/d47/d33/f5d x:0 0 0 2024-02-28T16:31:02.257 INFO:tasks.workunit.client.0.smithi001.stdout:0/252: truncate f15 2058629 0 2024-02-28T16:31:02.263 INFO:tasks.workunit.client.0.smithi001.stdout:0/253: write d18/d22/d2b/d36/f3c [13597,77925] 0 2024-02-28T16:31:02.264 INFO:tasks.workunit.client.0.smithi001.stdout:5/214: mkdir dc/d1a/d21/d2d/d3c/d4a 0 2024-02-28T16:31:02.272 INFO:tasks.workunit.client.0.smithi001.stdout:5/215: unlink dc/f28 0 2024-02-28T16:31:02.281 INFO:tasks.workunit.client.0.smithi001.stdout:2/251: dwrite d0/d2/d3/d2c/f2e [0,4194304] 0 2024-02-28T16:31:02.281 INFO:tasks.workunit.client.0.smithi001.stdout:0/254: link d18/d22/l44 d18/d22/d2b/l51 0 2024-02-28T16:31:02.285 INFO:tasks.workunit.client.0.smithi001.stdout:9/286: dread d5/d27/d34/d50/f52 [0,4194304] 0 2024-02-28T16:31:02.286 INFO:tasks.workunit.client.0.smithi001.stdout:5/216: creat dc/d1a/d21/f4b x:0 0 0 2024-02-28T16:31:02.288 INFO:tasks.workunit.client.0.smithi001.stdout:0/255: fsync f13 0 2024-02-28T16:31:02.293 INFO:tasks.workunit.client.0.smithi001.stdout:4/199: dwrite d3/d5/d7/d1f/d2e/f39 [0,4194304] 0 2024-02-28T16:31:02.296 INFO:tasks.workunit.client.0.smithi001.stdout:1/371: dread d0/d9/d1a/d2b/d26/d27/d30/f59 [0,4194304] 0 2024-02-28T16:31:02.300 INFO:tasks.workunit.client.0.smithi001.stdout:5/217: creat dc/d12/d41/f4c x:0 0 0 2024-02-28T16:31:02.302 INFO:tasks.workunit.client.0.smithi001.stdout:9/287: fsync d5/f1d 0 2024-02-28T16:31:02.306 INFO:tasks.workunit.client.0.smithi001.stdout:1/372: truncate d0/d9/d1a/d2b/d26/d27/d30/f76 376129 0 2024-02-28T16:31:02.308 INFO:tasks.workunit.client.0.smithi001.stdout:2/252: creat d0/d2/d3/f54 x:0 0 0 2024-02-28T16:31:02.309 INFO:tasks.workunit.client.0.smithi001.stdout:2/253: stat d0/d2/l1a 0 2024-02-28T16:31:02.311 INFO:tasks.workunit.client.0.smithi001.stdout:4/200: rename d3/d1c/f22 to d3/d5/f41 0 2024-02-28T16:31:02.312 INFO:tasks.workunit.client.0.smithi001.stdout:9/288: mkdir d5/d47/d5e 0 2024-02-28T16:31:02.321 INFO:tasks.workunit.client.0.smithi001.stdout:5/218: dwrite f5 [0,4194304] 0 2024-02-28T16:31:02.324 INFO:tasks.workunit.client.0.smithi001.stdout:5/219: truncate dc/d1a/d21/d2d/f42 127016 0 2024-02-28T16:31:02.325 INFO:tasks.workunit.client.0.smithi001.stdout:5/220: write dc/f30 [654902,31099] 0 2024-02-28T16:31:02.328 INFO:tasks.workunit.client.0.smithi001.stdout:4/201: dread d3/f17 [0,4194304] 0 2024-02-28T16:31:02.331 INFO:tasks.workunit.client.0.smithi001.stdout:1/373: mkdir d0/d9/d1a/d2b/d26/d27/d30/d57/d7d 0 2024-02-28T16:31:02.331 INFO:tasks.workunit.client.0.smithi001.stdout:0/256: getdents d18/d22 0 2024-02-28T16:31:02.333 INFO:tasks.workunit.client.0.smithi001.stdout:4/202: mkdir d3/d1c/d42 0 2024-02-28T16:31:02.333 INFO:tasks.workunit.client.0.smithi001.stdout:3/192: dread d4/d6/d7/fd [0,4194304] 0 2024-02-28T16:31:02.336 INFO:tasks.workunit.client.0.smithi001.stdout:0/257: write d18/d22/d2b/d36/d4d/f4c [758301,8340] 0 2024-02-28T16:31:02.336 INFO:tasks.workunit.client.0.smithi001.stdout:9/289: rmdir d5/d27/d42/d5a 0 2024-02-28T16:31:02.337 INFO:tasks.workunit.client.0.smithi001.stdout:3/193: chown d4/f35 1002279648 1 2024-02-28T16:31:02.337 INFO:tasks.workunit.client.0.smithi001.stdout:1/374: unlink d0/d9/d56/f29 0 2024-02-28T16:31:02.338 INFO:tasks.workunit.client.0.smithi001.stdout:5/221: link dc/d12/c3b dc/d1a/d21/d2d/d3c/c4d 0 2024-02-28T16:31:02.340 INFO:tasks.workunit.client.0.smithi001.stdout:4/203: mknod d3/d5/d7/c43 0 2024-02-28T16:31:02.342 INFO:tasks.workunit.client.0.smithi001.stdout:4/204: dread - d3/d2d/f3c zero size 2024-02-28T16:31:02.343 INFO:tasks.workunit.client.0.smithi001.stdout:1/375: chown d0/d9/ff 103493 1 2024-02-28T16:31:02.344 INFO:tasks.workunit.client.0.smithi001.stdout:9/290: creat d5/d47/f5f x:0 0 0 2024-02-28T16:31:02.345 INFO:tasks.workunit.client.0.smithi001.stdout:1/376: write d0/d9/d1a/f4c [6505744,28549] 0 2024-02-28T16:31:02.345 INFO:tasks.workunit.client.0.smithi001.stdout:4/205: mknod d3/d5/dd/c44 0 2024-02-28T16:31:02.346 INFO:tasks.workunit.client.0.smithi001.stdout:9/291: chown d5/c8 118702023 1 2024-02-28T16:31:02.352 INFO:tasks.workunit.client.0.smithi001.stdout:4/206: write d3/d5/dd/f28 [619693,72016] 0 2024-02-28T16:31:02.356 INFO:tasks.workunit.client.0.smithi001.stdout:1/377: rename d0/d9/d1a/d2b/f47 to d0/d9/f7e 0 2024-02-28T16:31:02.358 INFO:tasks.workunit.client.0.smithi001.stdout:1/378: readlink d0/d9/d56/d2f/l43 0 2024-02-28T16:31:02.362 INFO:tasks.workunit.client.0.smithi001.stdout:1/379: creat d0/d9/d56/f7f x:0 0 0 2024-02-28T16:31:02.364 INFO:tasks.workunit.client.0.smithi001.stdout:5/222: dread dc/df/f13 [0,4194304] 0 2024-02-28T16:31:02.365 INFO:tasks.workunit.client.0.smithi001.stdout:5/223: chown dc/df/f13 7833586 1 2024-02-28T16:31:02.366 INFO:tasks.workunit.client.0.smithi001.stdout:4/207: dread d3/f17 [4194304,4194304] 0 2024-02-28T16:31:02.369 INFO:tasks.workunit.client.0.smithi001.stdout:5/224: write dc/d1a/d21/d2d/d3c/f46 [298432,111966] 0 2024-02-28T16:31:02.374 INFO:tasks.workunit.client.0.smithi001.stdout:9/292: dwrite d5/f1d [0,4194304] 0 2024-02-28T16:31:02.376 INFO:tasks.workunit.client.0.smithi001.stdout:5/225: unlink dc/d1a/d21/d2d/f39 0 2024-02-28T16:31:02.378 INFO:tasks.workunit.client.0.smithi001.stdout:9/293: symlink d5/d47/d33/l60 0 2024-02-28T16:31:02.381 INFO:tasks.workunit.client.0.smithi001.stdout:1/380: write d0/d9/d56/f4b [4584121,36477] 0 2024-02-28T16:31:02.383 INFO:tasks.workunit.client.0.smithi001.stdout:4/208: dwrite d3/d5/f40 [0,4194304] 0 2024-02-28T16:31:02.388 INFO:tasks.workunit.client.0.smithi001.stdout:4/209: creat d3/d2d/f45 x:0 0 0 2024-02-28T16:31:02.396 INFO:tasks.workunit.client.0.smithi001.stdout:5/226: dwrite f5 [4194304,4194304] 0 2024-02-28T16:31:02.397 INFO:tasks.workunit.client.0.smithi001.stdout:8/253: sync 2024-02-28T16:31:02.400 INFO:tasks.workunit.client.0.smithi001.stdout:4/210: getdents d3/d5/d7 0 2024-02-28T16:31:02.401 INFO:tasks.workunit.client.0.smithi001.stdout:5/227: symlink dc/d1a/d21/l4e 0 2024-02-28T16:31:02.402 INFO:tasks.workunit.client.0.smithi001.stdout:8/254: mkdir d4/d7/d22/d27/d4b 0 2024-02-28T16:31:02.403 INFO:tasks.workunit.client.0.smithi001.stdout:1/381: getdents d0/d9/d1a/d2b/d26/d27/d30/d4a/d68 0 2024-02-28T16:31:02.405 INFO:tasks.workunit.client.0.smithi001.stdout:4/211: creat d3/d5/dd/f46 x:0 0 0 2024-02-28T16:31:02.409 INFO:tasks.workunit.client.0.smithi001.stdout:9/294: dwrite d5/fd [0,4194304] 0 2024-02-28T16:31:02.411 INFO:tasks.workunit.client.0.smithi001.stdout:5/228: rename dc/df/f35 to dc/d1a/d21/d2d/d3c/f4f 0 2024-02-28T16:31:02.415 INFO:tasks.workunit.client.0.smithi001.stdout:4/212: mkdir d3/d5/d47 0 2024-02-28T16:31:02.415 INFO:tasks.workunit.client.0.smithi001.stdout:9/295: symlink d5/d47/d58/l61 0 2024-02-28T16:31:02.418 INFO:tasks.workunit.client.0.smithi001.stdout:1/382: rmdir d0/d73 0 2024-02-28T16:31:02.419 INFO:tasks.workunit.client.0.smithi001.stdout:5/229: rename dc/d12/c17 to dc/df/c50 0 2024-02-28T16:31:02.420 INFO:tasks.workunit.client.0.smithi001.stdout:4/213: creat d3/d2d/f48 x:0 0 0 2024-02-28T16:31:02.421 INFO:tasks.workunit.client.0.smithi001.stdout:5/230: chown dc/d1a/d21/f48 1002 1 2024-02-28T16:31:02.422 INFO:tasks.workunit.client.0.smithi001.stdout:4/214: fdatasync d3/d5/dd/f28 0 2024-02-28T16:31:02.424 INFO:tasks.workunit.client.0.smithi001.stdout:5/231: chown dc/d1a/d21/d2d/f44 1581730 1 2024-02-28T16:31:02.426 INFO:tasks.workunit.client.0.smithi001.stdout:8/255: rmdir d4/d7/d22/d27 39 2024-02-28T16:31:02.426 INFO:tasks.workunit.client.0.smithi001.stdout:4/215: dread d3/d5/f38 [0,4194304] 0 2024-02-28T16:31:02.429 INFO:tasks.workunit.client.0.smithi001.stdout:1/383: symlink d0/d9/d1a/d2b/d26/d27/d30/d52/l80 0 2024-02-28T16:31:02.433 INFO:tasks.workunit.client.0.smithi001.stdout:1/384: rename d0/d9 to d0/d9/d1a/d2b/d26/d27/d30/d4a/d81 22 2024-02-28T16:31:02.433 INFO:tasks.workunit.client.0.smithi001.stdout:9/296: link d5/lf d5/d45/l62 0 2024-02-28T16:31:02.436 INFO:tasks.workunit.client.0.smithi001.stdout:9/297: write d5/fe [1146433,13831] 0 2024-02-28T16:31:02.437 INFO:tasks.workunit.client.0.smithi001.stdout:8/256: chown d4/d7/f21 10 1 2024-02-28T16:31:02.438 INFO:tasks.workunit.client.0.smithi001.stdout:9/298: dread - d5/d27/f2c zero size 2024-02-28T16:31:02.439 INFO:tasks.workunit.client.0.smithi001.stdout:8/257: chown d4/d7/d22/c26 38902746 1 2024-02-28T16:31:02.440 INFO:tasks.workunit.client.0.smithi001.stdout:9/299: stat d5/d47/d58 0 2024-02-28T16:31:02.440 INFO:tasks.workunit.client.0.smithi001.stdout:5/232: dread dc/d12/f14 [0,4194304] 0 2024-02-28T16:31:02.441 INFO:tasks.workunit.client.0.smithi001.stdout:1/385: rename d0/d9/d56/f2e to d0/d9/d1a/d2b/d26/f82 0 2024-02-28T16:31:02.443 INFO:tasks.workunit.client.0.smithi001.stdout:9/300: rmdir d5/d4d 39 2024-02-28T16:31:02.445 INFO:tasks.workunit.client.0.smithi001.stdout:8/258: symlink d4/l4c 0 2024-02-28T16:31:02.446 INFO:tasks.workunit.client.0.smithi001.stdout:9/301: chown d5/d27/d42/d48/f4c 73700 1 2024-02-28T16:31:02.448 INFO:tasks.workunit.client.0.smithi001.stdout:4/216: dwrite d3/d5/f14 [0,4194304] 0 2024-02-28T16:31:02.456 INFO:tasks.workunit.client.0.smithi001.stdout:4/217: dread - d3/d2d/f48 zero size 2024-02-28T16:31:02.463 INFO:tasks.workunit.client.0.smithi001.stdout:4/218: creat d3/d1c/f49 x:0 0 0 2024-02-28T16:31:02.464 INFO:tasks.workunit.client.0.smithi001.stdout:1/386: dwrite d0/d9/d56/f4b [0,4194304] 0 2024-02-28T16:31:02.467 INFO:tasks.workunit.client.0.smithi001.stdout:5/233: dread dc/f20 [0,4194304] 0 2024-02-28T16:31:02.467 INFO:tasks.workunit.client.0.smithi001.stdout:9/302: getdents d5/d4d 0 2024-02-28T16:31:02.471 INFO:tasks.workunit.client.0.smithi001.stdout:8/259: dwrite d4/d7/f2b [4194304,4194304] 0 2024-02-28T16:31:02.474 INFO:tasks.workunit.client.0.smithi001.stdout:8/260: chown d4/f3b 0 1 2024-02-28T16:31:02.478 INFO:tasks.workunit.client.0.smithi001.stdout:1/387: dread d0/d9/d56/f60 [0,4194304] 0 2024-02-28T16:31:02.489 INFO:tasks.workunit.client.0.smithi001.stdout:9/303: dread d5/f1d [4194304,4194304] 0 2024-02-28T16:31:02.491 INFO:tasks.workunit.client.0.smithi001.stdout:1/388: write d0/d9/d56/f60 [1415321,86903] 0 2024-02-28T16:31:02.496 INFO:tasks.workunit.client.0.smithi001.stdout:4/219: dwrite d3/d2d/f3c [0,4194304] 0 2024-02-28T16:31:02.496 INFO:tasks.workunit.client.0.smithi001.stdout:9/304: rename d5/d47/d58/l61 to d5/d27/l63 0 2024-02-28T16:31:02.498 INFO:tasks.workunit.client.0.smithi001.stdout:5/234: dwrite dc/df/f29 [0,4194304] 0 2024-02-28T16:31:02.503 INFO:tasks.workunit.client.0.smithi001.stdout:1/389: dread d0/f4e [0,4194304] 0 2024-02-28T16:31:02.507 INFO:tasks.workunit.client.0.smithi001.stdout:4/220: rename d3/d5/dd/f2c to d3/d1c/f4a 0 2024-02-28T16:31:02.508 INFO:tasks.workunit.client.0.smithi001.stdout:8/261: dwrite d4/d7/d22/d27/f4a [0,4194304] 0 2024-02-28T16:31:02.509 INFO:tasks.workunit.client.0.smithi001.stdout:7/232: sync 2024-02-28T16:31:02.509 INFO:tasks.workunit.client.0.smithi001.stdout:9/305: creat d5/f64 x:0 0 0 2024-02-28T16:31:02.513 INFO:tasks.workunit.client.0.smithi001.stdout:8/262: dread - d4/d7/d22/f48 zero size 2024-02-28T16:31:02.514 INFO:tasks.workunit.client.0.smithi001.stdout:1/390: chown d0/d9/d1a/d2b/c12 966063 1 2024-02-28T16:31:02.517 INFO:tasks.workunit.client.0.smithi001.stdout:1/391: dread - d0/d9/d56/d2f/f64 zero size 2024-02-28T16:31:02.518 INFO:tasks.workunit.client.0.smithi001.stdout:7/233: creat d3/d1c/f4b x:0 0 0 2024-02-28T16:31:02.518 INFO:tasks.workunit.client.0.smithi001.stdout:4/221: symlink d3/d5/dd/l4b 0 2024-02-28T16:31:02.522 INFO:tasks.workunit.client.0.smithi001.stdout:1/392: chown d0/d9/d1a/d2b/d26/d27/d30/d4a/c79 287795185 1 2024-02-28T16:31:02.522 INFO:tasks.workunit.client.0.smithi001.stdout:9/306: mkdir d5/d47/d65 0 2024-02-28T16:31:02.523 INFO:tasks.workunit.client.0.smithi001.stdout:7/234: rmdir d3/d17 39 2024-02-28T16:31:02.526 INFO:tasks.workunit.client.0.smithi001.stdout:4/222: write d3/fa [4411672,52981] 0 2024-02-28T16:31:02.526 INFO:tasks.workunit.client.0.smithi001.stdout:9/307: write d5/d27/f2c [712586,6494] 0 2024-02-28T16:31:02.526 INFO:tasks.workunit.client.0.smithi001.stdout:3/194: sync 2024-02-28T16:31:02.526 INFO:tasks.workunit.client.0.smithi001.stdout:6/243: sync 2024-02-28T16:31:02.527 INFO:tasks.workunit.client.0.smithi001.stdout:5/235: dread dc/d1a/d21/d2d/f44 [0,4194304] 0 2024-02-28T16:31:02.528 INFO:tasks.workunit.client.0.smithi001.stdout:2/254: sync 2024-02-28T16:31:02.529 INFO:tasks.workunit.client.0.smithi001.stdout:0/258: sync 2024-02-28T16:31:02.535 INFO:tasks.workunit.client.0.smithi001.stdout:9/308: symlink d5/d27/d42/l66 0 2024-02-28T16:31:02.535 INFO:tasks.workunit.client.0.smithi001.stdout:4/223: write d3/d5/dd/f28 [1544682,36378] 0 2024-02-28T16:31:02.536 INFO:tasks.workunit.client.0.smithi001.stdout:5/236: truncate dc/df/f11 4713295 0 2024-02-28T16:31:02.537 INFO:tasks.workunit.client.0.smithi001.stdout:8/263: dread d4/f43 [0,4194304] 0 2024-02-28T16:31:02.538 INFO:tasks.workunit.client.0.smithi001.stdout:7/235: symlink d3/d17/d3d/d42/l4c 0 2024-02-28T16:31:02.539 INFO:tasks.workunit.client.0.smithi001.stdout:0/259: creat d18/d22/d3d/f52 x:0 0 0 2024-02-28T16:31:02.540 INFO:tasks.workunit.client.0.smithi001.stdout:2/255: write d0/d2/d3/de/d22/f48 [901261,111905] 0 2024-02-28T16:31:02.541 INFO:tasks.workunit.client.0.smithi001.stdout:4/224: chown d3/d5/f40 0 1 2024-02-28T16:31:02.542 INFO:tasks.workunit.client.0.smithi001.stdout:7/236: dread - d3/d1c/f37 zero size 2024-02-28T16:31:02.546 INFO:tasks.workunit.client.0.smithi001.stdout:2/256: chown d0/d2/d6/d12 115 1 2024-02-28T16:31:02.548 INFO:tasks.workunit.client.0.smithi001.stdout:4/225: mkdir d3/d2d/d4c 0 2024-02-28T16:31:02.552 INFO:tasks.workunit.client.0.smithi001.stdout:5/237: dread dc/f30 [0,4194304] 0 2024-02-28T16:31:02.556 INFO:tasks.workunit.client.0.smithi001.stdout:0/260: rename d18/d31/l42 to d18/d22/l53 0 2024-02-28T16:31:02.558 INFO:tasks.workunit.client.0.smithi001.stdout:5/238: read - dc/d1a/d21/f4b zero size 2024-02-28T16:31:02.558 INFO:tasks.workunit.client.0.smithi001.stdout:3/195: dread d4/f14 [0,4194304] 0 2024-02-28T16:31:02.559 INFO:tasks.workunit.client.0.smithi001.stdout:7/237: creat d3/d17/d3d/f4d x:0 0 0 2024-02-28T16:31:02.568 INFO:tasks.workunit.client.0.smithi001.stdout:3/196: read d4/d18/f11 [2149376,104396] 0 2024-02-28T16:31:02.575 INFO:tasks.workunit.client.0.smithi001.stdout:6/244: dwrite d9/db/d10/d15/f40 [0,4194304] 0 2024-02-28T16:31:02.583 INFO:tasks.workunit.client.0.smithi001.stdout:9/309: dwrite d5/d27/d42/f53 [0,4194304] 0 2024-02-28T16:31:02.583 INFO:tasks.workunit.client.0.smithi001.stdout:2/257: dwrite d0/f32 [0,4194304] 0 2024-02-28T16:31:02.587 INFO:tasks.workunit.client.0.smithi001.stdout:8/264: dread d4/d7/f20 [0,4194304] 0 2024-02-28T16:31:02.593 INFO:tasks.workunit.client.0.smithi001.stdout:9/310: unlink d5/f21 0 2024-02-28T16:31:02.594 INFO:tasks.workunit.client.0.smithi001.stdout:4/226: dwrite d3/d5/d7/d1f/d20/f31 [0,4194304] 0 2024-02-28T16:31:02.600 INFO:tasks.workunit.client.0.smithi001.stdout:6/245: write d9/d29/d2b/f30 [2045565,44385] 0 2024-02-28T16:31:02.602 INFO:tasks.workunit.client.0.smithi001.stdout:0/261: dwrite d18/d22/d2b/d36/d4d/f4c [0,4194304] 0 2024-02-28T16:31:02.608 INFO:tasks.workunit.client.0.smithi001.stdout:4/227: chown d3/d5/dd/l4b 43 1 2024-02-28T16:31:02.608 INFO:tasks.workunit.client.0.smithi001.stdout:7/238: dwrite d3/d17/d3d/d42/f47 [0,4194304] 0 2024-02-28T16:31:02.609 INFO:tasks.workunit.client.0.smithi001.stdout:8/265: symlink d4/d7/d22/l4d 0 2024-02-28T16:31:02.611 INFO:tasks.workunit.client.0.smithi001.stdout:2/258: mkdir d0/d2/d6/d12/d55 0 2024-02-28T16:31:02.615 INFO:tasks.workunit.client.0.smithi001.stdout:9/311: mknod d5/c67 0 2024-02-28T16:31:02.617 INFO:tasks.workunit.client.0.smithi001.stdout:3/197: dwrite d4/d18/f10 [0,4194304] 0 2024-02-28T16:31:02.618 INFO:tasks.workunit.client.0.smithi001.stdout:9/312: write d5/f6 [3605879,64997] 0 2024-02-28T16:31:02.624 INFO:tasks.workunit.client.0.smithi001.stdout:6/246: symlink d9/db/d10/d15/d53/l58 0 2024-02-28T16:31:02.625 INFO:tasks.workunit.client.0.smithi001.stdout:9/313: creat d5/d27/d34/d50/f68 x:0 0 0 2024-02-28T16:31:02.628 INFO:tasks.workunit.client.0.smithi001.stdout:9/314: truncate d5/d27/f2c 1679941 0 2024-02-28T16:31:02.628 INFO:tasks.workunit.client.0.smithi001.stdout:4/228: mknod d3/d2d/d4c/c4d 0 2024-02-28T16:31:02.629 INFO:tasks.workunit.client.0.smithi001.stdout:3/198: getdents d4/d38 0 2024-02-28T16:31:02.635 INFO:tasks.workunit.client.0.smithi001.stdout:4/229: mkdir d3/d5/d4e 0 2024-02-28T16:31:02.638 INFO:tasks.workunit.client.0.smithi001.stdout:9/315: rename d5/l3f to d5/d47/d65/l69 0 2024-02-28T16:31:02.647 INFO:tasks.workunit.client.0.smithi001.stdout:3/199: creat d4/d3c/f3e x:0 0 0 2024-02-28T16:31:02.648 INFO:tasks.workunit.client.0.smithi001.stdout:4/230: rename d3/d5/d7/d1f/d2e/f3f to d3/d1c/f4f 0 2024-02-28T16:31:02.649 INFO:tasks.workunit.client.0.smithi001.stdout:9/316: mknod d5/d47/d5e/c6a 0 2024-02-28T16:31:02.649 INFO:tasks.workunit.client.0.smithi001.stdout:2/259: dread d0/f33 [0,4194304] 0 2024-02-28T16:31:02.652 INFO:tasks.workunit.client.0.smithi001.stdout:6/247: dwrite d9/db/f45 [0,4194304] 0 2024-02-28T16:31:02.654 INFO:tasks.workunit.client.0.smithi001.stdout:9/317: mknod d5/d27/d34/c6b 0 2024-02-28T16:31:02.655 INFO:tasks.workunit.client.0.smithi001.stdout:3/200: read d4/d1e/f25 [551764,103441] 0 2024-02-28T16:31:02.657 INFO:tasks.workunit.client.0.smithi001.stdout:2/260: dread d0/f33 [0,4194304] 0 2024-02-28T16:31:02.660 INFO:tasks.workunit.client.0.smithi001.stdout:5/239: dread fa [0,4194304] 0 2024-02-28T16:31:02.660 INFO:tasks.workunit.client.0.smithi001.stdout:6/248: rmdir d9/d29 39 2024-02-28T16:31:02.665 INFO:tasks.workunit.client.0.smithi001.stdout:9/318: unlink d5/d27/d42/f53 0 2024-02-28T16:31:02.666 INFO:tasks.workunit.client.0.smithi001.stdout:2/261: rename d0/d2/f21 to d0/d2/d3/de/d22/f56 0 2024-02-28T16:31:02.667 INFO:tasks.workunit.client.0.smithi001.stdout:9/319: write d5/d47/f40 [81709,52591] 0 2024-02-28T16:31:02.678 INFO:tasks.workunit.client.0.smithi001.stdout:5/240: dwrite dc/d1a/d21/f4b [0,4194304] 0 2024-02-28T16:31:02.682 INFO:tasks.workunit.client.0.smithi001.stdout:9/320: creat d5/d47/f6c x:0 0 0 2024-02-28T16:31:02.683 INFO:tasks.workunit.client.0.smithi001.stdout:2/262: getdents d0 0 2024-02-28T16:31:02.688 INFO:tasks.workunit.client.0.smithi001.stdout:3/201: dwrite d4/f17 [0,4194304] 0 2024-02-28T16:31:02.689 INFO:tasks.workunit.client.0.smithi001.stdout:9/321: fsync d5/fb 0 2024-02-28T16:31:02.689 INFO:tasks.workunit.client.0.smithi001.stdout:7/239: dread d3/ff [0,4194304] 0 2024-02-28T16:31:02.695 INFO:tasks.workunit.client.0.smithi001.stdout:5/241: dread dc/d1a/d21/d2d/d3c/f46 [0,4194304] 0 2024-02-28T16:31:02.695 INFO:tasks.workunit.client.0.smithi001.stdout:5/242: readlink dc/l1c 0 2024-02-28T16:31:02.695 INFO:tasks.workunit.client.0.smithi001.stdout:5/243: chown dc/d1a/d21/l2e 909797280 1 2024-02-28T16:31:02.698 INFO:tasks.workunit.client.0.smithi001.stdout:7/240: rmdir d3 39 2024-02-28T16:31:02.700 INFO:tasks.workunit.client.0.smithi001.stdout:3/202: symlink d4/d1e/l3f 0 2024-02-28T16:31:02.701 INFO:tasks.workunit.client.0.smithi001.stdout:1/393: sync 2024-02-28T16:31:02.704 INFO:tasks.workunit.client.0.smithi001.stdout:7/241: read - d3/d1c/f37 zero size 2024-02-28T16:31:02.709 INFO:tasks.workunit.client.0.smithi001.stdout:7/242: dread - d3/d17/d3d/f4d zero size 2024-02-28T16:31:02.709 INFO:tasks.workunit.client.0.smithi001.stdout:5/244: getdents dc/d1a/d21/d2d/d3c/d4a 0 2024-02-28T16:31:02.713 INFO:tasks.workunit.client.0.smithi001.stdout:2/263: dwrite d0/d2/d3/d36/f39 [0,4194304] 0 2024-02-28T16:31:02.714 INFO:tasks.workunit.client.0.smithi001.stdout:9/322: getdents d5/d47/d33 0 2024-02-28T16:31:02.717 INFO:tasks.workunit.client.0.smithi001.stdout:4/231: dread d3/d5/dd/f27 [0,4194304] 0 2024-02-28T16:31:02.721 INFO:tasks.workunit.client.0.smithi001.stdout:1/394: getdents d0/d9/d56/d2f 0 2024-02-28T16:31:02.722 INFO:tasks.workunit.client.0.smithi001.stdout:5/245: creat dc/d12/f51 x:0 0 0 2024-02-28T16:31:02.722 INFO:tasks.workunit.client.0.smithi001.stdout:2/264: dread d0/d2/d6/d12/f41 [0,4194304] 0 2024-02-28T16:31:02.723 INFO:tasks.workunit.client.0.smithi001.stdout:9/323: mknod d5/d27/d42/d48/c6d 0 2024-02-28T16:31:02.724 INFO:tasks.workunit.client.0.smithi001.stdout:4/232: chown d3/d2d/d4c/c4d 39076169 1 2024-02-28T16:31:02.724 INFO:tasks.workunit.client.0.smithi001.stdout:3/203: rmdir d4/d1e 39 2024-02-28T16:31:02.732 INFO:tasks.workunit.client.0.smithi001.stdout:3/204: write d4/f17 [3781216,60326] 0 2024-02-28T16:31:02.735 INFO:tasks.workunit.client.0.smithi001.stdout:7/243: dwrite d3/d17/f32 [4194304,4194304] 0 2024-02-28T16:31:02.736 INFO:tasks.workunit.client.0.smithi001.stdout:2/265: creat d0/d2/d6/d12/d46/f57 x:0 0 0 2024-02-28T16:31:02.737 INFO:tasks.workunit.client.0.smithi001.stdout:1/395: creat d0/d9/d1a/d2b/d26/d27/d30/d52/d6b/f83 x:0 0 0 2024-02-28T16:31:02.742 INFO:tasks.workunit.client.0.smithi001.stdout:1/396: stat d0/d9/d1a/l32 0 2024-02-28T16:31:02.743 INFO:tasks.workunit.client.0.smithi001.stdout:7/244: chown d3/d17/d3d/d42/f47 231615 1 2024-02-28T16:31:02.747 INFO:tasks.workunit.client.0.smithi001.stdout:1/397: read d0/d9/d1a/d2b/f6d [7571,70724] 0 2024-02-28T16:31:02.749 INFO:tasks.workunit.client.0.smithi001.stdout:3/205: truncate d4/f26 2085803 0 2024-02-28T16:31:02.752 INFO:tasks.workunit.client.0.smithi001.stdout:2/266: rename d0/d2/l1a to d0/d2/d6/d12/d55/l58 0 2024-02-28T16:31:02.752 INFO:tasks.workunit.client.0.smithi001.stdout:3/206: chown d4/d6/d7/fd 12617 1 2024-02-28T16:31:02.755 INFO:tasks.workunit.client.0.smithi001.stdout:7/245: write d3/f1e [1450678,68202] 0 2024-02-28T16:31:02.756 INFO:tasks.workunit.client.0.smithi001.stdout:4/233: dwrite d3/d2d/f32 [0,4194304] 0 2024-02-28T16:31:02.759 INFO:tasks.workunit.client.0.smithi001.stdout:3/207: write d4/d18/f19 [124778,55787] 0 2024-02-28T16:31:02.759 INFO:tasks.workunit.client.0.smithi001.stdout:9/324: dwrite d5/fd [0,4194304] 0 2024-02-28T16:31:02.760 INFO:tasks.workunit.client.0.smithi001.stdout:2/267: creat d0/f59 x:0 0 0 2024-02-28T16:31:02.766 INFO:tasks.workunit.client.0.smithi001.stdout:7/246: mknod d3/d17/c4e 0 2024-02-28T16:31:02.767 INFO:tasks.workunit.client.0.smithi001.stdout:7/247: write d3/d1c/f40 [28910,118507] 0 2024-02-28T16:31:02.769 INFO:tasks.workunit.client.0.smithi001.stdout:4/234: creat d3/d2d/d4c/f50 x:0 0 0 2024-02-28T16:31:02.772 INFO:tasks.workunit.client.0.smithi001.stdout:9/325: mknod d5/d27/d34/d50/c6e 0 2024-02-28T16:31:02.774 INFO:tasks.workunit.client.0.smithi001.stdout:7/248: symlink d3/d1c/l4f 0 2024-02-28T16:31:02.774 INFO:tasks.workunit.client.0.smithi001.stdout:9/326: write d5/f64 [516996,33569] 0 2024-02-28T16:31:02.776 INFO:tasks.workunit.client.0.smithi001.stdout:4/235: write d3/d5/f41 [196460,81535] 0 2024-02-28T16:31:02.777 INFO:tasks.workunit.client.0.smithi001.stdout:3/208: rmdir d4/d30 0 2024-02-28T16:31:02.783 INFO:tasks.workunit.client.0.smithi001.stdout:9/327: mkdir d5/d45/d6f 0 2024-02-28T16:31:02.783 INFO:tasks.workunit.client.0.smithi001.stdout:4/236: truncate d3/d2d/f45 494525 0 2024-02-28T16:31:02.784 INFO:tasks.workunit.client.0.smithi001.stdout:4/237: read d3/fa [1544179,8640] 0 2024-02-28T16:31:02.790 INFO:tasks.workunit.client.0.smithi001.stdout:5/246: dread dc/d12/f16 [0,4194304] 0 2024-02-28T16:31:02.790 INFO:tasks.workunit.client.0.smithi001.stdout:9/328: creat d5/d4d/f70 x:0 0 0 2024-02-28T16:31:02.791 INFO:tasks.workunit.client.0.smithi001.stdout:4/238: creat d3/d2d/d4c/f51 x:0 0 0 2024-02-28T16:31:02.793 INFO:tasks.workunit.client.0.smithi001.stdout:4/239: chown d3/d1c/f4f 0 1 2024-02-28T16:31:02.795 INFO:tasks.workunit.client.0.smithi001.stdout:1/398: write d0/d9/d1a/d2b/d26/d27/d30/f59 [7023934,101223] 0 2024-02-28T16:31:02.796 INFO:tasks.workunit.client.0.smithi001.stdout:4/240: truncate d3/d5/f38 1073730 0 2024-02-28T16:31:02.803 INFO:tasks.workunit.client.0.smithi001.stdout:5/247: mkdir dc/d1a/d52 0 2024-02-28T16:31:02.807 INFO:tasks.workunit.client.0.smithi001.stdout:4/241: rename d3/d1c/l33 to d3/d1c/d42/l52 0 2024-02-28T16:31:02.807 INFO:tasks.workunit.client.0.smithi001.stdout:5/248: write dc/df/f3a [473294,122457] 0 2024-02-28T16:31:02.813 INFO:tasks.workunit.client.0.smithi001.stdout:4/242: truncate d3/f3e 428394 0 2024-02-28T16:31:02.815 INFO:tasks.workunit.client.0.smithi001.stdout:2/268: rename d0/d2/c4f to d0/d2/d3/c5a 0 2024-02-28T16:31:02.815 INFO:tasks.workunit.client.0.smithi001.stdout:9/329: dwrite d5/d47/f3a [0,4194304] 0 2024-02-28T16:31:02.816 INFO:tasks.workunit.client.0.smithi001.stdout:3/209: truncate d4/f26 1012390 0 2024-02-28T16:31:02.818 INFO:tasks.workunit.client.0.smithi001.stdout:1/399: dwrite d0/d9/d1a/f33 [0,4194304] 0 2024-02-28T16:31:02.822 INFO:tasks.workunit.client.0.smithi001.stdout:9/330: unlink d5/f1f 0 2024-02-28T16:31:02.822 INFO:tasks.workunit.client.0.smithi001.stdout:1/400: fdatasync d0/d9/d56/d2f/f7c 0 2024-02-28T16:31:02.832 INFO:tasks.workunit.client.0.smithi001.stdout:9/331: creat d5/d45/f71 x:0 0 0 2024-02-28T16:31:02.834 INFO:tasks.workunit.client.0.smithi001.stdout:2/269: rmdir d0/d2 39 2024-02-28T16:31:02.834 INFO:tasks.workunit.client.0.smithi001.stdout:9/332: fdatasync d5/f1a 0 2024-02-28T16:31:02.837 INFO:tasks.workunit.client.0.smithi001.stdout:5/249: dwrite dc/d1a/d21/d2d/f44 [0,4194304] 0 2024-02-28T16:31:02.845 INFO:tasks.workunit.client.0.smithi001.stdout:3/210: dwrite d4/d18/f10 [4194304,4194304] 0 2024-02-28T16:31:02.845 INFO:tasks.workunit.client.0.smithi001.stdout:5/250: creat dc/d1a/d21/d2d/d3c/f53 x:0 0 0 2024-02-28T16:31:02.848 INFO:tasks.workunit.client.0.smithi001.stdout:4/243: dwrite d3/d5/d7/d1f/d2e/f39 [0,4194304] 0 2024-02-28T16:31:02.851 INFO:tasks.workunit.client.0.smithi001.stdout:9/333: dread d5/f64 [0,4194304] 0 2024-02-28T16:31:02.852 INFO:tasks.workunit.client.0.smithi001.stdout:4/244: chown d3/d5/d7/d1f/d2e/f39 2 1 2024-02-28T16:31:02.852 INFO:tasks.workunit.client.0.smithi001.stdout:5/251: mkdir dc/d12/d41/d54 0 2024-02-28T16:31:02.853 INFO:tasks.workunit.client.0.smithi001.stdout:3/211: chown d4/d1e/f22 60820029 1 2024-02-28T16:31:02.860 INFO:tasks.workunit.client.0.smithi001.stdout:9/334: mknod d5/c72 0 2024-02-28T16:31:02.861 INFO:tasks.workunit.client.0.smithi001.stdout:1/401: write d0/d9/d1a/d2b/f17 [3709147,21692] 0 2024-02-28T16:31:02.863 INFO:tasks.workunit.client.0.smithi001.stdout:4/245: creat d3/d1c/d42/f53 x:0 0 0 2024-02-28T16:31:02.865 INFO:tasks.workunit.client.0.smithi001.stdout:2/270: dread d0/d2/d3/de/d22/f48 [0,4194304] 0 2024-02-28T16:31:02.866 INFO:tasks.workunit.client.0.smithi001.stdout:5/252: stat dc/d1a/d21/d2d/d3c/c4d 0 2024-02-28T16:31:02.868 INFO:tasks.workunit.client.0.smithi001.stdout:9/335: creat d5/d27/f73 x:0 0 0 2024-02-28T16:31:02.868 INFO:tasks.workunit.client.0.smithi001.stdout:1/402: mkdir d0/d9/d1a/d2b/d26/d27/d30/d52/d84 0 2024-02-28T16:31:02.868 INFO:tasks.workunit.client.0.smithi001.stdout:4/246: mknod d3/d5/dd/c54 0 2024-02-28T16:31:02.869 INFO:tasks.workunit.client.0.smithi001.stdout:2/271: rmdir d0/d2 39 2024-02-28T16:31:02.872 INFO:tasks.workunit.client.0.smithi001.stdout:3/212: rename d4/d18/f37 to d4/d6/f40 0 2024-02-28T16:31:02.872 INFO:tasks.workunit.client.0.smithi001.stdout:5/253: write dc/df/f34 [808095,86952] 0 2024-02-28T16:31:02.876 INFO:tasks.workunit.client.0.smithi001.stdout:4/247: symlink d3/d5/d4e/l55 0 2024-02-28T16:31:02.878 INFO:tasks.workunit.client.0.smithi001.stdout:1/403: creat d0/d9/f85 x:0 0 0 2024-02-28T16:31:02.878 INFO:tasks.workunit.client.0.smithi001.stdout:5/254: chown dc/d1a/d21/f37 1022434201 1 2024-02-28T16:31:02.879 INFO:tasks.workunit.client.0.smithi001.stdout:9/336: rename d5/d27/d42/c43 to d5/d27/d34/c74 0 2024-02-28T16:31:02.881 INFO:tasks.workunit.client.0.smithi001.stdout:2/272: write d0/d2/d3/de/d22/f52 [108326,6802] 0 2024-02-28T16:31:02.882 INFO:tasks.workunit.client.0.smithi001.stdout:4/248: symlink d3/d1c/d42/l56 0 2024-02-28T16:31:02.884 INFO:tasks.workunit.client.0.smithi001.stdout:9/337: write d5/d27/f2b [3815344,127215] 0 2024-02-28T16:31:02.887 INFO:tasks.workunit.client.0.smithi001.stdout:4/249: write d3/d2d/f48 [55749,35334] 0 2024-02-28T16:31:02.888 INFO:tasks.workunit.client.0.smithi001.stdout:5/255: write dc/d12/f14 [4171857,12845] 0 2024-02-28T16:31:02.891 INFO:tasks.workunit.client.0.smithi001.stdout:2/273: symlink d0/d2/d3/de/d22/l5b 0 2024-02-28T16:31:02.891 INFO:tasks.workunit.client.0.smithi001.stdout:9/338: write d5/d47/f5f [109150,50801] 0 2024-02-28T16:31:02.893 INFO:tasks.workunit.client.0.smithi001.stdout:4/250: chown d3/d2d/f48 32466 1 2024-02-28T16:31:02.893 INFO:tasks.workunit.client.0.smithi001.stdout:5/256: readlink dc/d1a/d21/l4e 0 2024-02-28T16:31:02.894 INFO:tasks.workunit.client.0.smithi001.stdout:1/404: creat d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/f86 x:0 0 0 2024-02-28T16:31:02.895 INFO:tasks.workunit.client.0.smithi001.stdout:4/251: chown d3/d5/d7/d1f 837883353 1 2024-02-28T16:31:02.896 INFO:tasks.workunit.client.0.smithi001.stdout:2/274: symlink d0/d2/d3/d36/l5c 0 2024-02-28T16:31:02.896 INFO:tasks.workunit.client.0.smithi001.stdout:5/257: write dc/f27 [980929,111505] 0 2024-02-28T16:31:02.897 INFO:tasks.workunit.client.0.smithi001.stdout:3/213: dread d4/d6/d7/fd [0,4194304] 0 2024-02-28T16:31:02.900 INFO:tasks.workunit.client.0.smithi001.stdout:4/252: read d3/d2d/f45 [362129,110552] 0 2024-02-28T16:31:02.901 INFO:tasks.workunit.client.0.smithi001.stdout:9/339: link d5/d47/f6c d5/d45/d6f/f75 0 2024-02-28T16:31:02.904 INFO:tasks.workunit.client.0.smithi001.stdout:0/262: sync 2024-02-28T16:31:02.905 INFO:tasks.workunit.client.0.smithi001.stdout:2/275: symlink d0/d2/d3/d36/l5d 0 2024-02-28T16:31:02.910 INFO:tasks.workunit.client.0.smithi001.stdout:4/253: creat d3/d1c/d42/f57 x:0 0 0 2024-02-28T16:31:02.914 INFO:tasks.workunit.client.0.smithi001.stdout:5/258: mknod dc/d1a/d52/c55 0 2024-02-28T16:31:02.914 INFO:tasks.workunit.client.0.smithi001.stdout:1/405: dread d0/d9/d1a/d2b/d26/d27/d30/d4a/f70 [0,4194304] 0 2024-02-28T16:31:02.924 INFO:tasks.workunit.client.0.smithi001.stdout:4/254: rename d3/d5/dd to d3/d1c/d58 0 2024-02-28T16:31:02.924 INFO:tasks.workunit.client.0.smithi001.stdout:9/340: dwrite d5/d4d/f70 [0,4194304] 0 2024-02-28T16:31:02.929 INFO:tasks.workunit.client.0.smithi001.stdout:4/255: write d3/d5/f41 [380273,101191] 0 2024-02-28T16:31:02.934 INFO:tasks.workunit.client.0.smithi001.stdout:3/214: dread d4/d6/d7/d23/f2d [0,4194304] 0 2024-02-28T16:31:02.936 INFO:tasks.workunit.client.0.smithi001.stdout:0/263: dwrite d18/d22/d2f/f24 [0,4194304] 0 2024-02-28T16:31:02.941 INFO:tasks.workunit.client.0.smithi001.stdout:3/215: read d4/d6/fc [863667,70129] 0 2024-02-28T16:31:02.943 INFO:tasks.workunit.client.0.smithi001.stdout:2/276: getdents d0/d2/d3/de 0 2024-02-28T16:31:02.946 INFO:tasks.workunit.client.0.smithi001.stdout:0/264: truncate d18/d22/d2b/d36/f3c 157309 0 2024-02-28T16:31:02.946 INFO:tasks.workunit.client.0.smithi001.stdout:8/266: sync 2024-02-28T16:31:02.946 INFO:tasks.workunit.client.0.smithi001.stdout:6/249: sync 2024-02-28T16:31:02.946 INFO:tasks.workunit.client.0.smithi001.stdout:7/249: sync 2024-02-28T16:31:02.952 INFO:tasks.workunit.client.0.smithi001.stdout:6/250: chown d9/db/d10/d15/f40 25355 1 2024-02-28T16:31:02.953 INFO:tasks.workunit.client.0.smithi001.stdout:2/277: creat d0/d2/d3/de/f5e x:0 0 0 2024-02-28T16:31:02.957 INFO:tasks.workunit.client.0.smithi001.stdout:7/250: mknod d3/d17/c50 0 2024-02-28T16:31:02.959 INFO:tasks.workunit.client.0.smithi001.stdout:6/251: write d9/db/d10/f16 [808044,32920] 0 2024-02-28T16:31:02.961 INFO:tasks.workunit.client.0.smithi001.stdout:4/256: dread d3/d5/d7/d1f/d20/f31 [0,4194304] 0 2024-02-28T16:31:02.965 INFO:tasks.workunit.client.0.smithi001.stdout:1/406: dwrite d0/d9/ff [4194304,4194304] 0 2024-02-28T16:31:02.970 INFO:tasks.workunit.client.0.smithi001.stdout:7/251: write d3/d17/f1b [1564587,21157] 0 2024-02-28T16:31:02.971 INFO:tasks.workunit.client.0.smithi001.stdout:2/278: truncate d0/d2/d3/f43 521713 0 2024-02-28T16:31:02.976 INFO:tasks.workunit.client.0.smithi001.stdout:5/259: dwrite f4 [0,4194304] 0 2024-02-28T16:31:02.978 INFO:tasks.workunit.client.0.smithi001.stdout:9/341: dwrite d5/f1e [0,4194304] 0 2024-02-28T16:31:02.984 INFO:tasks.workunit.client.0.smithi001.stdout:5/260: write dc/d1a/d21/d2d/f44 [743416,7085] 0 2024-02-28T16:31:02.984 INFO:tasks.workunit.client.0.smithi001.stdout:0/265: dwrite f13 [0,4194304] 0 2024-02-28T16:31:02.987 INFO:tasks.workunit.client.0.smithi001.stdout:1/407: dread d0/d9/d56/f6c [0,4194304] 0 2024-02-28T16:31:02.996 INFO:tasks.workunit.client.0.smithi001.stdout:1/408: read d0/d9/d1a/d2b/d26/f37 [2361664,15599] 0 2024-02-28T16:31:03.005 INFO:tasks.workunit.client.0.smithi001.stdout:0/266: dwrite d18/d22/f3f [0,4194304] 0 2024-02-28T16:31:03.013 INFO:tasks.workunit.client.0.smithi001.stdout:4/257: creat d3/d5/d7/d1f/d2e/f59 x:0 0 0 2024-02-28T16:31:03.016 INFO:tasks.workunit.client.0.smithi001.stdout:4/258: chown d3/d5/d7/l10 10013 1 2024-02-28T16:31:03.016 INFO:tasks.workunit.client.0.smithi001.stdout:2/279: mknod d0/d2/d3/de/d22/c5f 0 2024-02-28T16:31:03.018 INFO:tasks.workunit.client.0.smithi001.stdout:9/342: unlink d5/d47/c3b 0 2024-02-28T16:31:03.020 INFO:tasks.workunit.client.0.smithi001.stdout:2/280: truncate d0/d2/f11 4715540 0 2024-02-28T16:31:03.023 INFO:tasks.workunit.client.0.smithi001.stdout:8/267: dwrite d4/f13 [0,4194304] 0 2024-02-28T16:31:03.023 INFO:tasks.workunit.client.0.smithi001.stdout:0/267: rmdir d18/d22 39 2024-02-28T16:31:03.030 INFO:tasks.workunit.client.0.smithi001.stdout:2/281: write d0/d2/d3/de/d22/f24 [3424684,72642] 0 2024-02-28T16:31:03.033 INFO:tasks.workunit.client.0.smithi001.stdout:1/409: rename d0/d9/d1a/d2b/d26/d27/d30/d4a/d68 to d0/d9/d1a/d2b/d26/d27/d30/d67/d87 0 2024-02-28T16:31:03.036 INFO:tasks.workunit.client.0.smithi001.stdout:7/252: rmdir d3/d1c/d41/d44 0 2024-02-28T16:31:03.038 INFO:tasks.workunit.client.0.smithi001.stdout:7/253: fsync d3/d17/f1b 0 2024-02-28T16:31:03.038 INFO:tasks.workunit.client.0.smithi001.stdout:6/252: getdents d9/db/d10/d15/d4e 0 2024-02-28T16:31:03.042 INFO:tasks.workunit.client.0.smithi001.stdout:8/268: dwrite d4/f13 [0,4194304] 0 2024-02-28T16:31:03.044 INFO:tasks.workunit.client.0.smithi001.stdout:9/343: rename d5/d47/d33/f35 to d5/d27/f76 0 2024-02-28T16:31:03.046 INFO:tasks.workunit.client.0.smithi001.stdout:9/344: chown d5/d47/d58 7 1 2024-02-28T16:31:03.047 INFO:tasks.workunit.client.0.smithi001.stdout:5/261: truncate dc/df/f29 1241200 0 2024-02-28T16:31:03.049 INFO:tasks.workunit.client.0.smithi001.stdout:4/259: link d3/d5/d7/c2f d3/d5/d47/c5a 0 2024-02-28T16:31:03.053 INFO:tasks.workunit.client.0.smithi001.stdout:0/268: rmdir d18/d22/d47 39 2024-02-28T16:31:03.054 INFO:tasks.workunit.client.0.smithi001.stdout:4/260: fdatasync d3/d5/d7/f21 0 2024-02-28T16:31:03.054 INFO:tasks.workunit.client.0.smithi001.stdout:9/345: chown d5/d27/d42/d48/c6d 0 1 2024-02-28T16:31:03.056 INFO:tasks.workunit.client.0.smithi001.stdout:5/262: mkdir dc/d12/d56 0 2024-02-28T16:31:03.056 INFO:tasks.workunit.client.0.smithi001.stdout:4/261: fdatasync d3/d5/d7/f21 0 2024-02-28T16:31:03.056 INFO:tasks.workunit.client.0.smithi001.stdout:9/346: dread - d5/d47/d33/f5d zero size 2024-02-28T16:31:03.058 INFO:tasks.workunit.client.0.smithi001.stdout:5/263: fdatasync dc/df/f3a 0 2024-02-28T16:31:03.059 INFO:tasks.workunit.client.0.smithi001.stdout:1/410: mkdir d0/d9/d1a/d2b/d26/d27/d30/d88 0 2024-02-28T16:31:03.062 INFO:tasks.workunit.client.0.smithi001.stdout:8/269: mknod d4/d47/c4e 0 2024-02-28T16:31:03.064 INFO:tasks.workunit.client.0.smithi001.stdout:7/254: dwrite d3/d1c/f37 [0,4194304] 0 2024-02-28T16:31:03.066 INFO:tasks.workunit.client.0.smithi001.stdout:2/282: write d0/d2/d6/d12/f41 [715127,119348] 0 2024-02-28T16:31:03.070 INFO:tasks.workunit.client.0.smithi001.stdout:9/347: creat d5/d4d/f77 x:0 0 0 2024-02-28T16:31:03.070 INFO:tasks.workunit.client.0.smithi001.stdout:3/216: sync 2024-02-28T16:31:03.071 INFO:tasks.workunit.client.0.smithi001.stdout:4/262: mknod d3/d5/c5b 0 2024-02-28T16:31:03.073 INFO:tasks.workunit.client.0.smithi001.stdout:8/270: chown d4/c29 155649342 1 2024-02-28T16:31:03.073 INFO:tasks.workunit.client.0.smithi001.stdout:0/269: creat d18/f54 x:0 0 0 2024-02-28T16:31:03.079 INFO:tasks.workunit.client.0.smithi001.stdout:6/253: dwrite d9/de/f4d [0,4194304] 0 2024-02-28T16:31:03.080 INFO:tasks.workunit.client.0.smithi001.stdout:9/348: write d5/f3c [107788,29544] 0 2024-02-28T16:31:03.082 INFO:tasks.workunit.client.0.smithi001.stdout:4/263: chown c1 2587107 1 2024-02-28T16:31:03.090 INFO:tasks.workunit.client.0.smithi001.stdout:7/255: rename d3/d17/d3d/d42/f47 to d3/d1c/f51 0 2024-02-28T16:31:03.096 INFO:tasks.workunit.client.0.smithi001.stdout:2/283: symlink d0/l60 0 2024-02-28T16:31:03.105 INFO:tasks.workunit.client.0.smithi001.stdout:8/271: creat d4/d7/d22/d27/f4f x:0 0 0 2024-02-28T16:31:03.105 INFO:tasks.workunit.client.0.smithi001.stdout:9/349: dwrite d5/f3c [0,4194304] 0 2024-02-28T16:31:03.113 INFO:tasks.workunit.client.0.smithi001.stdout:7/256: dread d3/d17/f2f [0,4194304] 0 2024-02-28T16:31:03.114 INFO:tasks.workunit.client.0.smithi001.stdout:3/217: dwrite d4/d3c/f3e [0,4194304] 0 2024-02-28T16:31:03.115 INFO:tasks.workunit.client.0.smithi001.stdout:8/272: write d4/f3b [744597,34833] 0 2024-02-28T16:31:03.117 INFO:tasks.workunit.client.0.smithi001.stdout:5/264: write dc/df/f29 [1360753,28178] 0 2024-02-28T16:31:03.118 INFO:tasks.workunit.client.0.smithi001.stdout:9/350: unlink d5/l14 0 2024-02-28T16:31:03.122 INFO:tasks.workunit.client.0.smithi001.stdout:7/257: readlink d3/d1c/l3e 0 2024-02-28T16:31:03.123 INFO:tasks.workunit.client.0.smithi001.stdout:2/284: rename d0/d2/d6 to d0/d2/d3/de/d61 0 2024-02-28T16:31:03.123 INFO:tasks.workunit.client.0.smithi001.stdout:6/254: dread d9/db/fc [0,4194304] 0 2024-02-28T16:31:03.127 INFO:tasks.workunit.client.0.smithi001.stdout:4/264: dwrite d3/d1c/f49 [0,4194304] 0 2024-02-28T16:31:03.127 INFO:tasks.workunit.client.0.smithi001.stdout:8/273: fdatasync d4/d7/f21 0 2024-02-28T16:31:03.128 INFO:tasks.workunit.client.0.smithi001.stdout:7/258: readlink d3/d17/l18 0 2024-02-28T16:31:03.130 INFO:tasks.workunit.client.0.smithi001.stdout:5/265: write dc/d1a/d21/f4b [5058552,6752] 0 2024-02-28T16:31:03.133 INFO:tasks.workunit.client.0.smithi001.stdout:9/351: write d5/d45/f71 [721917,103068] 0 2024-02-28T16:31:03.133 INFO:tasks.workunit.client.0.smithi001.stdout:3/218: write d4/f14 [1651592,46555] 0 2024-02-28T16:31:03.137 INFO:tasks.workunit.client.0.smithi001.stdout:6/255: chown d9/db/d19/c43 48824 1 2024-02-28T16:31:03.138 INFO:tasks.workunit.client.0.smithi001.stdout:1/411: dwrite d0/d9/d1a/d2b/f24 [0,4194304] 0 2024-02-28T16:31:03.144 INFO:tasks.workunit.client.0.smithi001.stdout:7/259: write d3/d1c/f30 [752457,5736] 0 2024-02-28T16:31:03.144 INFO:tasks.workunit.client.0.smithi001.stdout:2/285: creat d0/f62 x:0 0 0 2024-02-28T16:31:03.144 INFO:tasks.workunit.client.0.smithi001.stdout:4/265: symlink d3/d5/d7/l5c 0 2024-02-28T16:31:03.144 INFO:tasks.workunit.client.0.smithi001.stdout:0/270: dwrite f13 [0,4194304] 0 2024-02-28T16:31:03.147 INFO:tasks.workunit.client.0.smithi001.stdout:6/256: fdatasync f8 0 2024-02-28T16:31:03.148 INFO:tasks.workunit.client.0.smithi001.stdout:4/266: write d3/d5/f14 [3806027,90640] 0 2024-02-28T16:31:03.149 INFO:tasks.workunit.client.0.smithi001.stdout:7/260: write d3/d1c/f4b [34111,90363] 0 2024-02-28T16:31:03.149 INFO:tasks.workunit.client.0.smithi001.stdout:6/257: chown d9/db/fd 285380 1 2024-02-28T16:31:03.153 INFO:tasks.workunit.client.0.smithi001.stdout:8/274: link f1 d4/d32/f50 0 2024-02-28T16:31:03.163 INFO:tasks.workunit.client.0.smithi001.stdout:1/412: creat d0/d9/d1a/d2b/d26/d27/d30/d4a/f89 x:0 0 0 2024-02-28T16:31:03.168 INFO:tasks.workunit.client.0.smithi001.stdout:4/267: creat d3/d5/d7/d1f/d20/f5d x:0 0 0 2024-02-28T16:31:03.175 INFO:tasks.workunit.client.0.smithi001.stdout:7/261: unlink d3/d17/f2f 0 2024-02-28T16:31:03.196 INFO:tasks.workunit.client.0.smithi001.stdout:4/268: dwrite d3/f3b [0,4194304] 0 2024-02-28T16:31:03.198 INFO:tasks.workunit.client.0.smithi001.stdout:3/219: dread d4/d18/f19 [4194304,4194304] 0 2024-02-28T16:31:03.201 INFO:tasks.workunit.client.0.smithi001.stdout:8/275: mknod d4/d7/d22/d27/d4b/c51 0 2024-02-28T16:31:03.203 INFO:tasks.workunit.client.0.smithi001.stdout:9/352: getdents d5/d47/d65 0 2024-02-28T16:31:03.203 INFO:tasks.workunit.client.0.smithi001.stdout:5/266: link dc/d12/c2f dc/d12/d56/c57 0 2024-02-28T16:31:03.203 INFO:tasks.workunit.client.0.smithi001.stdout:0/271: write fd [1812765,77964] 0 2024-02-28T16:31:03.250 INFO:tasks.workunit.client.0.smithi001.stdout:6/258: dread d9/db/fd [0,4194304] 0 2024-02-28T16:31:03.254 INFO:tasks.workunit.client.0.smithi001.stdout:9/353: symlink d5/d27/d34/l78 0 2024-02-28T16:31:03.256 INFO:tasks.workunit.client.0.smithi001.stdout:1/413: dread d0/d9/d1a/f4c [4194304,4194304] 0 2024-02-28T16:31:03.256 INFO:tasks.workunit.client.0.smithi001.stdout:5/267: symlink dc/d1a/d21/d2d/d3c/l58 0 2024-02-28T16:31:03.257 INFO:tasks.workunit.client.0.smithi001.stdout:9/354: stat d5/d27/l63 0 2024-02-28T16:31:03.258 INFO:tasks.workunit.client.0.smithi001.stdout:0/272: creat d18/d22/d2b/d36/d48/f55 x:0 0 0 2024-02-28T16:31:03.259 INFO:tasks.workunit.client.0.smithi001.stdout:5/268: readlink dc/d1a/d21/l4e 0 2024-02-28T16:31:03.304 INFO:tasks.workunit.client.0.smithi001.stdout:2/286: sync 2024-02-28T16:31:03.310 INFO:tasks.workunit.client.0.smithi001.stdout:1/414: write d0/f4e [825242,130045] 0 2024-02-28T16:31:03.312 INFO:tasks.workunit.client.0.smithi001.stdout:0/273: fdatasync d18/d22/d2b/d36/f49 0 2024-02-28T16:31:03.315 INFO:tasks.workunit.client.0.smithi001.stdout:5/269: write dc/d12/f14 [2876504,57110] 0 2024-02-28T16:31:03.318 INFO:tasks.workunit.client.0.smithi001.stdout:2/287: creat d0/f63 x:0 0 0 2024-02-28T16:31:03.319 INFO:tasks.workunit.client.0.smithi001.stdout:1/415: creat d0/d9/d1a/d72/f8a x:0 0 0 2024-02-28T16:31:03.323 INFO:tasks.workunit.client.0.smithi001.stdout:7/262: dread d3/f8 [0,4194304] 0 2024-02-28T16:31:03.327 INFO:tasks.workunit.client.0.smithi001.stdout:2/288: mknod d0/d2/d3/de/d61/d12/c64 0 2024-02-28T16:31:03.328 INFO:tasks.workunit.client.0.smithi001.stdout:0/274: truncate d18/d22/d2b/d36/d48/f55 555643 0 2024-02-28T16:31:03.329 INFO:tasks.workunit.client.0.smithi001.stdout:3/220: dwrite d4/d1e/f33 [0,4194304] 0 2024-02-28T16:31:03.331 INFO:tasks.workunit.client.0.smithi001.stdout:7/263: write d3/d17/f1b [1840716,28838] 0 2024-02-28T16:31:03.331 INFO:tasks.workunit.client.0.smithi001.stdout:1/416: dread - d0/d9/d1a/d2b/d26/d27/d30/d52/d6b/f83 zero size 2024-02-28T16:31:03.335 INFO:tasks.workunit.client.0.smithi001.stdout:9/355: dwrite d5/f12 [0,4194304] 0 2024-02-28T16:31:03.336 INFO:tasks.workunit.client.0.smithi001.stdout:1/417: chown d0/d9/d56/f60 17169 1 2024-02-28T16:31:03.341 INFO:tasks.workunit.client.0.smithi001.stdout:0/275: unlink d18/c19 0 2024-02-28T16:31:03.341 INFO:tasks.workunit.client.0.smithi001.stdout:7/264: unlink d3/c16 0 2024-02-28T16:31:03.341 INFO:tasks.workunit.client.0.smithi001.stdout:5/270: truncate dc/df/f11 152539 0 2024-02-28T16:31:03.342 INFO:tasks.workunit.client.0.smithi001.stdout:0/276: write f13 [4122350,85758] 0 2024-02-28T16:31:03.347 INFO:tasks.workunit.client.0.smithi001.stdout:3/221: write d4/d18/f19 [4243390,8309] 0 2024-02-28T16:31:03.348 INFO:tasks.workunit.client.0.smithi001.stdout:5/271: write dc/df/f3a [998336,48076] 0 2024-02-28T16:31:03.348 INFO:tasks.workunit.client.0.smithi001.stdout:2/289: dread d0/d2/d3/de/d61/f23 [0,4194304] 0 2024-02-28T16:31:03.348 INFO:tasks.workunit.client.0.smithi001.stdout:7/265: mkdir d3/d17/d52 0 2024-02-28T16:31:03.349 INFO:tasks.workunit.client.0.smithi001.stdout:6/259: dread d9/db/d10/f16 [0,4194304] 0 2024-02-28T16:31:03.349 INFO:tasks.workunit.client.0.smithi001.stdout:9/356: creat d5/f79 x:0 0 0 2024-02-28T16:31:03.350 INFO:tasks.workunit.client.0.smithi001.stdout:7/266: fdatasync d3/f21 0 2024-02-28T16:31:03.359 INFO:tasks.workunit.client.0.smithi001.stdout:0/277: link d18/d22/d2b/d36/f3c d18/d22/d2b/d36/f56 0 2024-02-28T16:31:03.364 INFO:tasks.workunit.client.0.smithi001.stdout:0/278: stat d18/d22/d2f/d1c/c3b 0 2024-02-28T16:31:03.368 INFO:tasks.workunit.client.0.smithi001.stdout:1/418: write d0/f28 [3558109,42422] 0 2024-02-28T16:31:03.369 INFO:tasks.workunit.client.0.smithi001.stdout:2/290: link d0/d2/d3/de/d22/l3c d0/d2/d3/de/l65 0 2024-02-28T16:31:03.370 INFO:tasks.workunit.client.0.smithi001.stdout:6/260: creat d9/f59 x:0 0 0 2024-02-28T16:31:03.374 INFO:tasks.workunit.client.0.smithi001.stdout:5/272: dwrite dc/d12/f51 [0,4194304] 0 2024-02-28T16:31:03.375 INFO:tasks.workunit.client.0.smithi001.stdout:4/269: sync 2024-02-28T16:31:03.375 INFO:tasks.workunit.client.0.smithi001.stdout:8/276: sync 2024-02-28T16:31:03.379 INFO:tasks.workunit.client.0.smithi001.stdout:0/279: mkdir d18/d22/d57 0 2024-02-28T16:31:03.380 INFO:tasks.workunit.client.0.smithi001.stdout:2/291: chown d0/l1 3170519 1 2024-02-28T16:31:03.382 INFO:tasks.workunit.client.0.smithi001.stdout:5/273: readlink dc/d1a/d21/d2d/l36 0 2024-02-28T16:31:03.383 INFO:tasks.workunit.client.0.smithi001.stdout:6/261: rmdir d9/db/d10/d15/d4e 39 2024-02-28T16:31:03.386 INFO:tasks.workunit.client.0.smithi001.stdout:9/357: dwrite d5/d27/f5c [0,4194304] 0 2024-02-28T16:31:03.392 INFO:tasks.workunit.client.0.smithi001.stdout:1/419: creat d0/d9/d1a/d72/f8b x:0 0 0 2024-02-28T16:31:03.393 INFO:tasks.workunit.client.0.smithi001.stdout:7/267: dwrite d3/f21 [0,4194304] 0 2024-02-28T16:31:03.394 INFO:tasks.workunit.client.0.smithi001.stdout:9/358: dread - d5/d27/f73 zero size 2024-02-28T16:31:03.395 INFO:tasks.workunit.client.0.smithi001.stdout:0/280: symlink d18/d22/d2b/d36/l58 0 2024-02-28T16:31:03.397 INFO:tasks.workunit.client.0.smithi001.stdout:7/268: read d3/f21 [279885,96716] 0 2024-02-28T16:31:03.397 INFO:tasks.workunit.client.0.smithi001.stdout:2/292: creat d0/d2/d3/de/d61/d12/d46/f66 x:0 0 0 2024-02-28T16:31:03.398 INFO:tasks.workunit.client.0.smithi001.stdout:9/359: chown d5/f15 516500978 1 2024-02-28T16:31:03.402 INFO:tasks.workunit.client.0.smithi001.stdout:5/274: creat dc/d1a/d21/d2d/d3c/d4a/f59 x:0 0 0 2024-02-28T16:31:03.402 INFO:tasks.workunit.client.0.smithi001.stdout:3/222: dwrite d4/d1e/f24 [0,4194304] 0 2024-02-28T16:31:03.410 INFO:tasks.workunit.client.0.smithi001.stdout:8/277: creat d4/f52 x:0 0 0 2024-02-28T16:31:03.411 INFO:tasks.workunit.client.0.smithi001.stdout:1/420: creat d0/d9/d1a/d2b/d26/d27/d30/d57/f8c x:0 0 0 2024-02-28T16:31:03.414 INFO:tasks.workunit.client.0.smithi001.stdout:0/281: write d18/d22/d2b/f37 [3002802,89280] 0 2024-02-28T16:31:03.420 INFO:tasks.workunit.client.0.smithi001.stdout:4/270: dwrite d3/f3e [0,4194304] 0 2024-02-28T16:31:03.421 INFO:tasks.workunit.client.0.smithi001.stdout:1/421: creat d0/d9/d56/f8d x:0 0 0 2024-02-28T16:31:03.423 INFO:tasks.workunit.client.0.smithi001.stdout:6/262: link d9/d29/f44 d9/de/f5a 0 2024-02-28T16:31:03.423 INFO:tasks.workunit.client.0.smithi001.stdout:3/223: unlink d4/d6/c3a 0 2024-02-28T16:31:03.423 INFO:tasks.workunit.client.0.smithi001.stdout:0/282: rmdir d18/d22 39 2024-02-28T16:31:03.429 INFO:tasks.workunit.client.0.smithi001.stdout:8/278: link d4/d7/lb d4/d47/l53 0 2024-02-28T16:31:03.430 INFO:tasks.workunit.client.0.smithi001.stdout:9/360: getdents d5/d27 0 2024-02-28T16:31:03.431 INFO:tasks.workunit.client.0.smithi001.stdout:5/275: dread dc/d12/f51 [0,4194304] 0 2024-02-28T16:31:03.432 INFO:tasks.workunit.client.0.smithi001.stdout:7/269: dwrite d3/f36 [4194304,4194304] 0 2024-02-28T16:31:03.433 INFO:tasks.workunit.client.0.smithi001.stdout:1/422: mkdir d0/d9/d1a/d2b/d26/d27/d30/d57/d8e 0 2024-02-28T16:31:03.439 INFO:tasks.workunit.client.0.smithi001.stdout:9/361: write d5/d45/f71 [232830,63398] 0 2024-02-28T16:31:03.441 INFO:tasks.workunit.client.0.smithi001.stdout:1/423: chown d0/d9/d56/d2f/l43 17 1 2024-02-28T16:31:03.442 INFO:tasks.workunit.client.0.smithi001.stdout:4/271: getdents d3/d1c/d42 0 2024-02-28T16:31:03.442 INFO:tasks.workunit.client.0.smithi001.stdout:7/270: write d3/d1c/f40 [434351,81044] 0 2024-02-28T16:31:03.447 INFO:tasks.workunit.client.0.smithi001.stdout:3/224: getdents d4/d6/d7 0 2024-02-28T16:31:03.447 INFO:tasks.workunit.client.0.smithi001.stdout:2/293: write d0/d2/d3/f34 [522970,45584] 0 2024-02-28T16:31:03.448 INFO:tasks.workunit.client.0.smithi001.stdout:4/272: fsync d3/d5/f41 0 2024-02-28T16:31:03.450 INFO:tasks.workunit.client.0.smithi001.stdout:7/271: chown d3/d1c/f51 131846396 1 2024-02-28T16:31:03.450 INFO:tasks.workunit.client.0.smithi001.stdout:6/263: rename d9/db/d10/f57 to d9/db/d10/f5b 0 2024-02-28T16:31:03.451 INFO:tasks.workunit.client.0.smithi001.stdout:5/276: symlink dc/d12/l5a 0 2024-02-28T16:31:03.452 INFO:tasks.workunit.client.0.smithi001.stdout:1/424: chown d0/lb 2 1 2024-02-28T16:31:03.456 INFO:tasks.workunit.client.0.smithi001.stdout:9/362: rename d5/d27/d34/d50/c6e to d5/d27/c7a 0 2024-02-28T16:31:03.458 INFO:tasks.workunit.client.0.smithi001.stdout:2/294: creat d0/d2/d3/de/d61/f67 x:0 0 0 2024-02-28T16:31:03.459 INFO:tasks.workunit.client.0.smithi001.stdout:4/273: fsync d3/fa 0 2024-02-28T16:31:03.463 INFO:tasks.workunit.client.0.smithi001.stdout:1/425: creat d0/d9/d56/f8f x:0 0 0 2024-02-28T16:31:03.466 INFO:tasks.workunit.client.0.smithi001.stdout:7/272: creat d3/d17/d3d/d42/f53 x:0 0 0 2024-02-28T16:31:03.469 INFO:tasks.workunit.client.0.smithi001.stdout:0/283: dread f11 [0,4194304] 0 2024-02-28T16:31:03.470 INFO:tasks.workunit.client.0.smithi001.stdout:3/225: link d4/f14 d4/d38/f41 0 2024-02-28T16:31:03.471 INFO:tasks.workunit.client.0.smithi001.stdout:5/277: symlink dc/d1a/l5b 0 2024-02-28T16:31:03.478 INFO:tasks.workunit.client.0.smithi001.stdout:0/284: chown d18/d22/d2b/d36/f49 20266 1 2024-02-28T16:31:03.480 INFO:tasks.workunit.client.0.smithi001.stdout:1/426: link d0/l25 d0/d9/d1a/d72/l90 0 2024-02-28T16:31:03.485 INFO:tasks.workunit.client.0.smithi001.stdout:9/363: dwrite d5/d47/f41 [0,4194304] 0 2024-02-28T16:31:03.493 INFO:tasks.workunit.client.0.smithi001.stdout:0/285: write d18/d22/d2b/f37 [1288327,7232] 0 2024-02-28T16:31:03.494 INFO:tasks.workunit.client.0.smithi001.stdout:5/278: creat dc/f5c x:0 0 0 2024-02-28T16:31:03.500 INFO:tasks.workunit.client.0.smithi001.stdout:9/364: rmdir d5/d47/d65 39 2024-02-28T16:31:03.505 INFO:tasks.workunit.client.0.smithi001.stdout:7/273: dwrite d3/f8 [0,4194304] 0 2024-02-28T16:31:03.506 INFO:tasks.workunit.client.0.smithi001.stdout:2/295: dwrite d0/d2/d3/d2c/f2e [0,4194304] 0 2024-02-28T16:31:03.507 INFO:tasks.workunit.client.0.smithi001.stdout:6/264: dread d9/d29/d2b/f33 [4194304,4194304] 0 2024-02-28T16:31:03.508 INFO:tasks.workunit.client.0.smithi001.stdout:4/274: dwrite d3/d1c/d58/f46 [0,4194304] 0 2024-02-28T16:31:03.509 INFO:tasks.workunit.client.0.smithi001.stdout:9/365: write d5/d45/f71 [618529,52551] 0 2024-02-28T16:31:03.525 INFO:tasks.workunit.client.0.smithi001.stdout:3/226: dwrite d4/d38/f41 [0,4194304] 0 2024-02-28T16:31:03.532 INFO:tasks.workunit.client.0.smithi001.stdout:2/296: rmdir d0/d2/d3/de/d22 39 2024-02-28T16:31:03.532 INFO:tasks.workunit.client.0.smithi001.stdout:7/274: creat d3/d17/f54 x:0 0 0 2024-02-28T16:31:03.532 INFO:tasks.workunit.client.0.smithi001.stdout:4/275: mkdir d3/d2d/d4c/d5e 0 2024-02-28T16:31:03.532 INFO:tasks.workunit.client.0.smithi001.stdout:1/427: truncate d0/d9/d1a/d2b/d26/d27/d30/f58 2749669 0 2024-02-28T16:31:03.532 INFO:tasks.workunit.client.0.smithi001.stdout:6/265: symlink d9/l5c 0 2024-02-28T16:31:03.537 INFO:tasks.workunit.client.0.smithi001.stdout:6/266: readlink d9/d39/l42 0 2024-02-28T16:31:03.542 INFO:tasks.workunit.client.0.smithi001.stdout:2/297: chown d0/l7 34 1 2024-02-28T16:31:03.542 INFO:tasks.workunit.client.0.smithi001.stdout:0/286: dread d18/d22/d2f/f24 [0,4194304] 0 2024-02-28T16:31:03.544 INFO:tasks.workunit.client.0.smithi001.stdout:7/275: mkdir d3/d17/d52/d55 0 2024-02-28T16:31:03.549 INFO:tasks.workunit.client.0.smithi001.stdout:1/428: stat d0/d9/d1a/d2b/d26/d27/d30/f59 0 2024-02-28T16:31:03.549 INFO:tasks.workunit.client.0.smithi001.stdout:4/276: rename d3/d2d/d4c/c4d to d3/d5/c5f 0 2024-02-28T16:31:03.555 INFO:tasks.workunit.client.0.smithi001.stdout:0/287: write d18/d22/d2b/d36/d4d/f4c [1412723,125043] 0 2024-02-28T16:31:03.555 INFO:tasks.workunit.client.0.smithi001.stdout:2/298: creat d0/d2/d3/d36/f68 x:0 0 0 2024-02-28T16:31:03.556 INFO:tasks.workunit.client.0.smithi001.stdout:5/279: dwrite dc/d1a/d21/d2d/d3c/f3d [0,4194304] 0 2024-02-28T16:31:03.558 INFO:tasks.workunit.client.0.smithi001.stdout:9/366: dwrite d5/d4d/f77 [0,4194304] 0 2024-02-28T16:31:03.559 INFO:tasks.workunit.client.0.smithi001.stdout:7/276: rename d3/d1c/l23 to d3/d1c/d41/d49/l56 0 2024-02-28T16:31:03.561 INFO:tasks.workunit.client.0.smithi001.stdout:3/227: dread d4/d18/f19 [4194304,4194304] 0 2024-02-28T16:31:03.563 INFO:tasks.workunit.client.0.smithi001.stdout:3/228: chown d4/f35 426 1 2024-02-28T16:31:03.564 INFO:tasks.workunit.client.0.smithi001.stdout:4/277: symlink d3/d5/d4e/l60 0 2024-02-28T16:31:03.567 INFO:tasks.workunit.client.0.smithi001.stdout:1/429: symlink d0/d9/d1a/d2b/d26/d27/d30/l91 0 2024-02-28T16:31:03.567 INFO:tasks.workunit.client.0.smithi001.stdout:6/267: write f5 [1223449,25221] 0 2024-02-28T16:31:03.569 INFO:tasks.workunit.client.0.smithi001.stdout:6/268: stat d9/db/d10/d15/f18 0 2024-02-28T16:31:03.570 INFO:tasks.workunit.client.0.smithi001.stdout:5/280: creat dc/df/f5d x:0 0 0 2024-02-28T16:31:03.571 INFO:tasks.workunit.client.0.smithi001.stdout:8/279: sync 2024-02-28T16:31:03.573 INFO:tasks.workunit.client.0.smithi001.stdout:6/269: dread - d9/db/d10/d15/f31 zero size 2024-02-28T16:31:03.576 INFO:tasks.workunit.client.0.smithi001.stdout:2/299: symlink d0/d2/d3/de/d61/d12/d55/l69 0 2024-02-28T16:31:03.578 INFO:tasks.workunit.client.0.smithi001.stdout:4/278: symlink d3/d5/d4e/l61 0 2024-02-28T16:31:03.579 INFO:tasks.workunit.client.0.smithi001.stdout:1/430: symlink d0/d9/d1a/d2b/d26/d27/d30/d52/l92 0 2024-02-28T16:31:03.579 INFO:tasks.workunit.client.0.smithi001.stdout:8/280: write d4/d7/d22/d27/f2f [853512,87419] 0 2024-02-28T16:31:03.583 INFO:tasks.workunit.client.0.smithi001.stdout:9/367: mknod d5/d47/c7b 0 2024-02-28T16:31:03.585 INFO:tasks.workunit.client.0.smithi001.stdout:2/300: write d0/d2/f11 [277634,111673] 0 2024-02-28T16:31:03.586 INFO:tasks.workunit.client.0.smithi001.stdout:5/281: creat dc/d1a/d21/d2d/d3c/f5e x:0 0 0 2024-02-28T16:31:03.590 INFO:tasks.workunit.client.0.smithi001.stdout:4/279: dread - d3/d5/d7/d1f/d2e/f59 zero size 2024-02-28T16:31:03.591 INFO:tasks.workunit.client.0.smithi001.stdout:3/229: mknod d4/d6/c42 0 2024-02-28T16:31:03.591 INFO:tasks.workunit.client.0.smithi001.stdout:9/368: fdatasync d5/d27/f73 0 2024-02-28T16:31:03.592 INFO:tasks.workunit.client.0.smithi001.stdout:7/277: link d3/l1a d3/d17/d3d/l57 0 2024-02-28T16:31:03.594 INFO:tasks.workunit.client.0.smithi001.stdout:1/431: unlink d0/d9/d1a/d2b/d26/f37 0 2024-02-28T16:31:03.595 INFO:tasks.workunit.client.0.smithi001.stdout:8/281: truncate d4/d7/d22/d27/f4f 811893 0 2024-02-28T16:31:03.597 INFO:tasks.workunit.client.0.smithi001.stdout:0/288: dwrite f13 [0,4194304] 0 2024-02-28T16:31:03.598 INFO:tasks.workunit.client.0.smithi001.stdout:4/280: dread - d3/d5/d7/d1f/d2e/f59 zero size 2024-02-28T16:31:03.600 INFO:tasks.workunit.client.0.smithi001.stdout:8/282: fsync d4/f11 0 2024-02-28T16:31:03.602 INFO:tasks.workunit.client.0.smithi001.stdout:5/282: link dc/d1a/d21/f48 dc/d1a/f5f 0 2024-02-28T16:31:03.603 INFO:tasks.workunit.client.0.smithi001.stdout:9/369: creat d5/d47/f7c x:0 0 0 2024-02-28T16:31:03.612 INFO:tasks.workunit.client.0.smithi001.stdout:6/270: dwrite d9/de/f2c [0,4194304] 0 2024-02-28T16:31:03.616 INFO:tasks.workunit.client.0.smithi001.stdout:8/283: fdatasync d4/f43 0 2024-02-28T16:31:03.617 INFO:tasks.workunit.client.0.smithi001.stdout:9/370: unlink d5/d45/d6f/f75 0 2024-02-28T16:31:03.627 INFO:tasks.workunit.client.0.smithi001.stdout:4/281: dwrite d3/d2d/f32 [0,4194304] 0 2024-02-28T16:31:03.628 INFO:tasks.workunit.client.0.smithi001.stdout:3/230: dread d4/d1e/f33 [0,4194304] 0 2024-02-28T16:31:03.632 INFO:tasks.workunit.client.0.smithi001.stdout:1/432: creat d0/d9/d1a/d2b/d26/d27/d30/d57/d8e/f93 x:0 0 0 2024-02-28T16:31:03.637 INFO:tasks.workunit.client.0.smithi001.stdout:6/271: dread d9/db/d10/d15/f34 [0,4194304] 0 2024-02-28T16:31:03.638 INFO:tasks.workunit.client.0.smithi001.stdout:3/231: write d4/d6/d7/f2f [365768,17571] 0 2024-02-28T16:31:03.639 INFO:tasks.workunit.client.0.smithi001.stdout:1/433: chown d0/d9/d1a/d2b/d26/c3f 221 1 2024-02-28T16:31:03.640 INFO:tasks.workunit.client.0.smithi001.stdout:0/289: creat d18/f59 x:0 0 0 2024-02-28T16:31:03.640 INFO:tasks.workunit.client.0.smithi001.stdout:3/232: dread - d4/d6/f1c zero size 2024-02-28T16:31:03.642 INFO:tasks.workunit.client.0.smithi001.stdout:7/278: link d3/d1c/d41/d49/l56 d3/d1c/d41/d49/l58 0 2024-02-28T16:31:03.646 INFO:tasks.workunit.client.0.smithi001.stdout:2/301: truncate d0/d2/d3/f34 287900 0 2024-02-28T16:31:03.652 INFO:tasks.workunit.client.0.smithi001.stdout:5/283: rename dc/d1a/d21/l2e to dc/l60 0 2024-02-28T16:31:03.659 INFO:tasks.workunit.client.0.smithi001.stdout:6/272: mknod d9/d29/c5d 0 2024-02-28T16:31:03.660 INFO:tasks.workunit.client.0.smithi001.stdout:3/233: dread d4/d6/fc [0,4194304] 0 2024-02-28T16:31:03.661 INFO:tasks.workunit.client.0.smithi001.stdout:5/284: fsync dc/f30 0 2024-02-28T16:31:03.666 INFO:tasks.workunit.client.0.smithi001.stdout:4/282: rename d3/d2d/f48 to d3/d2d/d4c/d5e/f62 0 2024-02-28T16:31:03.667 INFO:tasks.workunit.client.0.smithi001.stdout:0/290: fdatasync d18/f1e 0 2024-02-28T16:31:03.668 INFO:tasks.workunit.client.0.smithi001.stdout:1/434: creat d0/d9/d1a/d2b/d26/d27/d30/d52/d84/f94 x:0 0 0 2024-02-28T16:31:03.670 INFO:tasks.workunit.client.0.smithi001.stdout:8/284: dwrite d4/d32/f50 [4194304,4194304] 0 2024-02-28T16:31:03.671 INFO:tasks.workunit.client.0.smithi001.stdout:9/371: getdents d5/d27/d34 0 2024-02-28T16:31:03.673 INFO:tasks.workunit.client.0.smithi001.stdout:4/283: write d3/d2d/d4c/f51 [895228,21015] 0 2024-02-28T16:31:03.675 INFO:tasks.workunit.client.0.smithi001.stdout:3/234: creat d4/d6/d7/f43 x:0 0 0 2024-02-28T16:31:03.675 INFO:tasks.workunit.client.0.smithi001.stdout:1/435: fsync d0/d9/d56/f4b 0 2024-02-28T16:31:03.676 INFO:tasks.workunit.client.0.smithi001.stdout:7/279: dwrite d3/d1c/f37 [0,4194304] 0 2024-02-28T16:31:03.677 INFO:tasks.workunit.client.0.smithi001.stdout:2/302: truncate d0/fd 1277259 0 2024-02-28T16:31:03.678 INFO:tasks.workunit.client.0.smithi001.stdout:1/436: stat d0/d9/f85 0 2024-02-28T16:31:03.680 INFO:tasks.workunit.client.0.smithi001.stdout:0/291: unlink lb 0 2024-02-28T16:31:03.684 INFO:tasks.workunit.client.0.smithi001.stdout:5/285: creat dc/d12/d56/f61 x:0 0 0 2024-02-28T16:31:03.685 INFO:tasks.workunit.client.0.smithi001.stdout:6/273: rename d9/d29/d2b/c3f to d9/d39/c5e 0 2024-02-28T16:31:03.688 INFO:tasks.workunit.client.0.smithi001.stdout:8/285: symlink d4/d7/l54 0 2024-02-28T16:31:03.692 INFO:tasks.workunit.client.0.smithi001.stdout:0/292: stat d18/d22/d2b/l51 0 2024-02-28T16:31:03.695 INFO:tasks.workunit.client.0.smithi001.stdout:1/437: mkdir d0/d9/d95 0 2024-02-28T16:31:03.696 INFO:tasks.workunit.client.0.smithi001.stdout:2/303: mkdir d0/d2/d3/de/d61/d12/d6a 0 2024-02-28T16:31:03.696 INFO:tasks.workunit.client.0.smithi001.stdout:9/372: rmdir d5/d47/d65 39 2024-02-28T16:31:03.698 INFO:tasks.workunit.client.0.smithi001.stdout:5/286: write dc/d12/f24 [2010394,68012] 0 2024-02-28T16:31:03.701 INFO:tasks.workunit.client.0.smithi001.stdout:3/235: creat d4/f44 x:0 0 0 2024-02-28T16:31:03.705 INFO:tasks.workunit.client.0.smithi001.stdout:6/274: mkdir d9/db/d5f 0 2024-02-28T16:31:03.706 INFO:tasks.workunit.client.0.smithi001.stdout:1/438: dread - d0/d9/d1a/d2b/d26/d27/d30/d57/f8c zero size 2024-02-28T16:31:03.708 INFO:tasks.workunit.client.0.smithi001.stdout:7/280: dwrite d3/d17/f32 [8388608,4194304] 0 2024-02-28T16:31:03.710 INFO:tasks.workunit.client.0.smithi001.stdout:0/293: mkdir d18/d5a 0 2024-02-28T16:31:03.717 INFO:tasks.workunit.client.0.smithi001.stdout:3/236: symlink d4/d1e/l45 0 2024-02-28T16:31:03.719 INFO:tasks.workunit.client.0.smithi001.stdout:8/286: creat d4/f55 x:0 0 0 2024-02-28T16:31:03.722 INFO:tasks.workunit.client.0.smithi001.stdout:6/275: creat d9/d39/f60 x:0 0 0 2024-02-28T16:31:03.725 INFO:tasks.workunit.client.0.smithi001.stdout:0/294: write f11 [855078,96890] 0 2024-02-28T16:31:03.726 INFO:tasks.workunit.client.0.smithi001.stdout:5/287: mkdir dc/d12/d41/d54/d62 0 2024-02-28T16:31:03.727 INFO:tasks.workunit.client.0.smithi001.stdout:7/281: mkdir d3/d17/d3d/d59 0 2024-02-28T16:31:03.731 INFO:tasks.workunit.client.0.smithi001.stdout:1/439: chown d0/d9/d1a/d2b/d26/d27/d30/d67/d87/l71 3522908 1 2024-02-28T16:31:03.733 INFO:tasks.workunit.client.0.smithi001.stdout:0/295: mknod d18/d22/d2f/c5b 0 2024-02-28T16:31:03.740 INFO:tasks.workunit.client.0.smithi001.stdout:3/237: rename d4/ce to d4/d6/d7/d23/c46 0 2024-02-28T16:31:03.743 INFO:tasks.workunit.client.0.smithi001.stdout:2/304: dwrite d0/d2/d3/de/d22/f52 [0,4194304] 0 2024-02-28T16:31:03.744 INFO:tasks.workunit.client.0.smithi001.stdout:7/282: rename d3/d17/d3d/d42/l46 to d3/d17/d3d/d42/l5a 0 2024-02-28T16:31:03.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:03 smithi066 ceph-mon[36236]: pgmap v45: 65 pgs: 65 active+clean; 547 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 489 KiB/s rd, 44 MiB/s wr, 154 op/s 2024-02-28T16:31:03.750 INFO:tasks.workunit.client.0.smithi001.stdout:5/288: dread dc/d1a/d21/d2d/d3c/f46 [0,4194304] 0 2024-02-28T16:31:03.750 INFO:tasks.workunit.client.0.smithi001.stdout:0/296: fsync d18/d22/d2f/d1c/f21 0 2024-02-28T16:31:03.752 INFO:tasks.workunit.client.0.smithi001.stdout:2/305: dread d0/d2/d3/f43 [0,4194304] 0 2024-02-28T16:31:03.755 INFO:tasks.workunit.client.0.smithi001.stdout:3/238: creat d4/d1e/d29/f47 x:0 0 0 2024-02-28T16:31:03.755 INFO:tasks.workunit.client.0.smithi001.stdout:9/373: dread d5/fb [0,4194304] 0 2024-02-28T16:31:03.756 INFO:tasks.workunit.client.0.smithi001.stdout:0/297: write d18/f1e [834990,104794] 0 2024-02-28T16:31:03.761 INFO:tasks.workunit.client.0.smithi001.stdout:2/306: chown d0/d2/d3/de/d22/l4c 813994405 1 2024-02-28T16:31:03.762 INFO:tasks.workunit.client.0.smithi001.stdout:8/287: dwrite d4/d32/f46 [0,4194304] 0 2024-02-28T16:31:03.773 INFO:tasks.workunit.client.0.smithi001.stdout:8/288: write d4/d7/f35 [654911,21523] 0 2024-02-28T16:31:03.776 INFO:tasks.workunit.client.0.smithi001.stdout:8/289: fsync d4/d7/f19 0 2024-02-28T16:31:03.785 INFO:tasks.workunit.client.0.smithi001.stdout:2/307: dwrite d0/d2/fb [0,4194304] 0 2024-02-28T16:31:03.789 INFO:tasks.workunit.client.0.smithi001.stdout:6/276: getdents d9/db/d10/d15 0 2024-02-28T16:31:03.801 INFO:tasks.workunit.client.0.smithi001.stdout:3/239: symlink d4/d18/l48 0 2024-02-28T16:31:03.804 INFO:tasks.workunit.client.0.smithi001.stdout:9/374: readlink d5/d45/l62 0 2024-02-28T16:31:03.808 INFO:tasks.workunit.client.0.smithi001.stdout:2/308: mknod d0/d2/d3/d2c/c6b 0 2024-02-28T16:31:03.815 INFO:tasks.workunit.client.0.smithi001.stdout:3/240: creat d4/d18/f49 x:0 0 0 2024-02-28T16:31:03.817 INFO:tasks.workunit.client.0.smithi001.stdout:2/309: unlink d0/d2/d3/de/d22/f56 0 2024-02-28T16:31:03.817 INFO:tasks.workunit.client.0.smithi001.stdout:3/241: write d4/d3c/f3e [1055033,79908] 0 2024-02-28T16:31:03.830 INFO:tasks.workunit.client.0.smithi001.stdout:9/375: truncate d5/d47/d33/f3d 3359869 0 2024-02-28T16:31:03.835 INFO:tasks.workunit.client.0.smithi001.stdout:8/290: getdents d4/d7/d22/d27 0 2024-02-28T16:31:03.839 INFO:tasks.workunit.client.0.smithi001.stdout:9/376: truncate d5/d27/d34/d50/f52 5222280 0 2024-02-28T16:31:03.841 INFO:tasks.workunit.client.0.smithi001.stdout:9/377: fdatasync d5/fd 0 2024-02-28T16:31:03.842 INFO:tasks.workunit.client.0.smithi001.stdout:9/378: dread - d5/f79 zero size 2024-02-28T16:31:03.843 INFO:tasks.workunit.client.0.smithi001.stdout:8/291: creat d4/d7/d22/d27/f56 x:0 0 0 2024-02-28T16:31:03.845 INFO:tasks.workunit.client.0.smithi001.stdout:2/310: dwrite d0/d2/fb [0,4194304] 0 2024-02-28T16:31:03.846 INFO:tasks.workunit.client.0.smithi001.stdout:9/379: fdatasync d5/f12 0 2024-02-28T16:31:03.846 INFO:tasks.workunit.client.0.smithi001.stdout:5/289: write dc/d1a/d21/f48 [104406,42560] 0 2024-02-28T16:31:03.855 INFO:tasks.workunit.client.0.smithi001.stdout:5/290: fdatasync dc/d1a/d21/d2d/d3c/f4f 0 2024-02-28T16:31:03.857 INFO:tasks.workunit.client.0.smithi001.stdout:5/291: fdatasync f5 0 2024-02-28T16:31:03.862 INFO:tasks.workunit.client.0.smithi001.stdout:4/284: sync 2024-02-28T16:31:03.863 INFO:tasks.workunit.client.0.smithi001.stdout:3/242: dread d4/d18/f10 [0,4194304] 0 2024-02-28T16:31:03.864 INFO:tasks.workunit.client.0.smithi001.stdout:2/311: creat d0/d2/d3/f6c x:0 0 0 2024-02-28T16:31:03.868 INFO:tasks.workunit.client.0.smithi001.stdout:5/292: symlink dc/d1a/d52/l63 0 2024-02-28T16:31:03.868 INFO:tasks.workunit.client.0.smithi001.stdout:8/292: rmdir d4/d7 39 2024-02-28T16:31:03.871 INFO:tasks.workunit.client.0.smithi001.stdout:4/285: creat d3/d2d/f63 x:0 0 0 2024-02-28T16:31:03.872 INFO:tasks.workunit.client.0.smithi001.stdout:3/243: fsync d4/d6/fc 0 2024-02-28T16:31:03.876 INFO:tasks.workunit.client.0.smithi001.stdout:0/298: dwrite d18/d22/d2b/d36/f3c [0,4194304] 0 2024-02-28T16:31:03.882 INFO:tasks.workunit.client.0.smithi001.stdout:9/380: dwrite d5/f1d [0,4194304] 0 2024-02-28T16:31:03.883 INFO:tasks.workunit.client.0.smithi001.stdout:4/286: unlink d3/d5/f40 0 2024-02-28T16:31:03.884 INFO:tasks.workunit.client.0.smithi001.stdout:9/381: readlink d5/l28 0 2024-02-28T16:31:03.889 INFO:tasks.workunit.client.0.smithi001.stdout:6/277: mknod d9/c61 0 2024-02-28T16:31:03.890 INFO:tasks.workunit.client.0.smithi001.stdout:2/312: rename d0/f2a to d0/d2/d3/de/d22/f6d 0 2024-02-28T16:31:03.896 INFO:tasks.workunit.client.0.smithi001.stdout:9/382: unlink d5/f2d 0 2024-02-28T16:31:03.898 INFO:tasks.workunit.client.0.smithi001.stdout:1/440: sync 2024-02-28T16:31:03.898 INFO:tasks.workunit.client.0.smithi001.stdout:7/283: sync 2024-02-28T16:31:03.900 INFO:tasks.workunit.client.0.smithi001.stdout:5/293: creat dc/f64 x:0 0 0 2024-02-28T16:31:03.901 INFO:tasks.workunit.client.0.smithi001.stdout:0/299: symlink d18/d5a/l5c 0 2024-02-28T16:31:03.902 INFO:tasks.workunit.client.1.smithi066.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-28T16:31:03.903 INFO:tasks.workunit.client.0.smithi001.stdout:1/441: chown d0/l3 1560604 1 2024-02-28T16:31:03.904 INFO:tasks.workunit.client.0.smithi001.stdout:9/383: chown d5/d27/f5c 40 1 2024-02-28T16:31:03.907 INFO:tasks.workunit.client.0.smithi001.stdout:4/287: unlink d3/d5/d47/c5a 0 2024-02-28T16:31:03.910 INFO:tasks.workunit.client.0.smithi001.stdout:5/294: chown dc/d1a/d21/d2d/l36 1518580063 1 2024-02-28T16:31:03.911 INFO:tasks.workunit.client.0.smithi001.stdout:7/284: mknod d3/d17/d3d/d59/c5b 0 2024-02-28T16:31:03.912 INFO:tasks.workunit.client.0.smithi001.stdout:8/293: dwrite d4/d7/d22/d27/f4a [4194304,4194304] 0 2024-02-28T16:31:03.915 INFO:tasks.workunit.client.0.smithi001.stdout:7/285: read - d3/d17/f54 zero size 2024-02-28T16:31:03.918 INFO:tasks.workunit.client.0.smithi001.stdout:1/442: chown d0/d9/d1a/d2b/d26/d27/d30/f59 42004534 1 2024-02-28T16:31:03.920 INFO:tasks.workunit.client.0.smithi001.stdout:8/294: fdatasync d4/f3b 0 2024-02-28T16:31:03.920 INFO:tasks.workunit.client.0.smithi001.stdout:5/295: write dc/f64 [729862,7460] 0 2024-02-28T16:31:03.925 INFO:tasks.workunit.client.0.smithi001.stdout:3/244: truncate d4/d1e/f33 3300157 0 2024-02-28T16:31:03.928 INFO:tasks.workunit.client.0.smithi001.stdout:6/278: dwrite d9/db/f45 [0,4194304] 0 2024-02-28T16:31:03.931 INFO:tasks.workunit.client.0.smithi001.stdout:4/288: write d3/d1c/d58/ff [216823,16675] 0 2024-02-28T16:31:03.931 INFO:tasks.workunit.client.0.smithi001.stdout:3/245: write d4/d6/fc [1663913,60537] 0 2024-02-28T16:31:03.932 INFO:tasks.workunit.client.0.smithi001.stdout:7/286: symlink d3/d17/d3d/l5c 0 2024-02-28T16:31:03.934 INFO:tasks.workunit.client.0.smithi001.stdout:8/295: mknod d4/d7/d22/d27/d4b/c57 0 2024-02-28T16:31:03.935 INFO:tasks.workunit.client.0.smithi001.stdout:6/279: write d9/d29/d2b/f30 [2054537,99764] 0 2024-02-28T16:31:03.937 INFO:tasks.workunit.client.0.smithi001.stdout:1/443: write d0/d9/f7e [1976653,17087] 0 2024-02-28T16:31:03.941 INFO:tasks.workunit.client.0.smithi001.stdout:5/296: mknod dc/d12/d56/c65 0 2024-02-28T16:31:03.946 INFO:tasks.workunit.client.1.smithi066.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-28T16:31:03.946 INFO:tasks.workunit.client.1.smithi066.stderr:+ make 2024-02-28T16:31:03.947 INFO:tasks.workunit.client.0.smithi001.stdout:2/313: write d0/d2/d3/f43 [801001,120525] 0 2024-02-28T16:31:03.947 INFO:tasks.workunit.client.0.smithi001.stdout:3/246: mknod d4/d18/c4a 0 2024-02-28T16:31:03.947 INFO:tasks.workunit.client.0.smithi001.stdout:8/296: write d4/d7/d22/d27/f4f [1584648,23519] 0 2024-02-28T16:31:03.950 INFO:tasks.workunit.client.0.smithi001.stdout:8/297: chown d4/d47 3 1 2024-02-28T16:31:03.951 INFO:tasks.workunit.client.0.smithi001.stdout:2/314: chown d0/d2/d3/f54 118414 1 2024-02-28T16:31:03.956 INFO:tasks.workunit.client.0.smithi001.stdout:2/315: fdatasync d0/d2/d3/d36/f68 0 2024-02-28T16:31:03.960 INFO:tasks.workunit.client.0.smithi001.stdout:8/298: read d4/f11 [614320,112198] 0 2024-02-28T16:31:03.964 INFO:tasks.workunit.client.0.smithi001.stdout:0/300: dwrite d18/d22/d3d/f52 [0,4194304] 0 2024-02-28T16:31:03.967 INFO:tasks.workunit.client.0.smithi001.stdout:4/289: getdents d3/d5/d7/d1f/d3a 0 2024-02-28T16:31:03.967 INFO:tasks.workunit.client.0.smithi001.stdout:2/316: dread - d0/d2/d3/f54 zero size 2024-02-28T16:31:03.970 INFO:tasks.workunit.client.0.smithi001.stdout:3/247: mknod d4/c4b 0 2024-02-28T16:31:03.976 INFO:tasks.workunit.client.0.smithi001.stdout:1/444: getdents d0 0 2024-02-28T16:31:03.983 INFO:tasks.workunit.client.0.smithi001.stdout:3/248: readlink d4/d18/l2e 0 2024-02-28T16:31:03.984 INFO:tasks.workunit.client.0.smithi001.stdout:2/317: write d0/d2/d3/de/fa [2759158,64002] 0 2024-02-28T16:31:03.985 INFO:tasks.workunit.client.0.smithi001.stdout:7/287: dwrite d3/d17/d3d/f4d [0,4194304] 0 2024-02-28T16:31:03.989 INFO:tasks.workunit.client.0.smithi001.stdout:9/384: dwrite d5/d47/f6c [0,4194304] 0 2024-02-28T16:31:03.990 INFO:tasks.workunit.client.0.smithi001.stdout:7/288: readlink d3/l26 0 2024-02-28T16:31:03.993 INFO:tasks.workunit.client.0.smithi001.stdout:5/297: dwrite dc/f30 [0,4194304] 0 2024-02-28T16:31:03.995 INFO:tasks.workunit.client.0.smithi001.stdout:3/249: chown d4/c1a 559 1 2024-02-28T16:31:03.999 INFO:tasks.workunit.client.0.smithi001.stdout:7/289: write d3/f1e [4780737,49337] 0 2024-02-28T16:31:04.001 INFO:tasks.workunit.client.0.smithi001.stdout:1/445: symlink d0/d9/d1a/d72/l96 0 2024-02-28T16:31:04.009 INFO:tasks.workunit.client.0.smithi001.stdout:9/385: creat d5/d45/d6f/f7d x:0 0 0 2024-02-28T16:31:04.011 INFO:tasks.workunit.client.0.smithi001.stdout:9/386: dread - d5/d45/f4a zero size 2024-02-28T16:31:04.013 INFO:tasks.workunit.client.0.smithi001.stdout:7/290: mkdir d3/d17/d3d/d5d 0 2024-02-28T16:31:04.015 INFO:tasks.workunit.client.0.smithi001.stdout:5/298: creat dc/d1a/d21/d2d/f66 x:0 0 0 2024-02-28T16:31:04.016 INFO:tasks.workunit.client.0.smithi001.stdout:7/291: dread - d3/d17/d3d/d42/f53 zero size 2024-02-28T16:31:04.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:03 smithi001 ceph-mon[28991]: pgmap v45: 65 pgs: 65 active+clean; 547 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 489 KiB/s rd, 44 MiB/s wr, 154 op/s 2024-02-28T16:31:04.020 INFO:tasks.workunit.client.0.smithi001.stdout:9/387: creat d5/d27/d42/f7e x:0 0 0 2024-02-28T16:31:04.022 INFO:tasks.workunit.client.0.smithi001.stdout:6/280: dwrite d9/de/f5a [0,4194304] 0 2024-02-28T16:31:04.022 INFO:tasks.workunit.client.0.smithi001.stdout:2/318: getdents d0/d2/d3/d36 0 2024-02-28T16:31:04.023 INFO:tasks.workunit.client.0.smithi001.stdout:9/388: read - d5/f2e zero size 2024-02-28T16:31:04.032 INFO:tasks.workunit.client.0.smithi001.stdout:4/290: dread d3/f17 [0,4194304] 0 2024-02-28T16:31:04.035 INFO:tasks.workunit.client.0.smithi001.stdout:5/299: symlink dc/d12/l67 0 2024-02-28T16:31:04.035 INFO:tasks.workunit.client.0.smithi001.stdout:8/299: dread d4/d7/d22/d27/f3a [0,4194304] 0 2024-02-28T16:31:04.038 INFO:tasks.workunit.client.0.smithi001.stdout:0/301: truncate f11 2803178 0 2024-02-28T16:31:04.040 INFO:tasks.workunit.client.0.smithi001.stdout:7/292: mknod d3/c5e 0 2024-02-28T16:31:04.040 INFO:tasks.workunit.client.0.smithi001.stdout:9/389: creat d5/d45/d6f/f7f x:0 0 0 2024-02-28T16:31:04.040 INFO:tasks.workunit.client.0.smithi001.stdout:2/319: dread d0/d2/d3/de/d61/f23 [0,4194304] 0 2024-02-28T16:31:04.041 INFO:tasks.workunit.client.0.smithi001.stdout:6/281: mkdir d9/db/d5f/d62 0 2024-02-28T16:31:04.043 INFO:tasks.workunit.client.0.smithi001.stdout:5/300: chown dc/l1c 166 1 2024-02-28T16:31:04.050 INFO:tasks.workunit.client.0.smithi001.stdout:0/302: symlink d18/d22/d57/l5d 0 2024-02-28T16:31:04.052 INFO:tasks.workunit.client.0.smithi001.stdout:2/320: truncate d0/f59 235401 0 2024-02-28T16:31:04.055 INFO:tasks.workunit.client.0.smithi001.stdout:8/300: mkdir d4/d7/d22/d27/d58 0 2024-02-28T16:31:04.055 INFO:tasks.workunit.client.0.smithi001.stdout:6/282: write d9/db/d10/d15/f34 [1352269,89771] 0 2024-02-28T16:31:04.058 INFO:tasks.workunit.client.0.smithi001.stdout:8/301: fdatasync d4/d7/f19 0 2024-02-28T16:31:04.058 INFO:tasks.workunit.client.0.smithi001.stdout:3/250: dwrite d4/f17 [0,4194304] 0 2024-02-28T16:31:04.062 INFO:tasks.workunit.client.0.smithi001.stdout:0/303: creat d18/d22/d3d/f5e x:0 0 0 2024-02-28T16:31:04.066 INFO:tasks.workunit.client.0.smithi001.stdout:0/304: write d18/f59 [747554,59547] 0 2024-02-28T16:31:04.068 INFO:tasks.workunit.client.0.smithi001.stdout:1/446: truncate d0/d9/d1a/d2b/f4d 2659419 0 2024-02-28T16:31:04.068 INFO:tasks.workunit.client.0.smithi001.stdout:8/302: fsync d4/d7/d22/f48 0 2024-02-28T16:31:04.075 INFO:tasks.workunit.client.0.smithi001.stdout:9/390: link d5/d27/l63 d5/d27/l80 0 2024-02-28T16:31:04.077 INFO:tasks.workunit.client.0.smithi001.stdout:4/291: dwrite d3/f3e [0,4194304] 0 2024-02-28T16:31:04.083 INFO:tasks.workunit.client.0.smithi001.stdout:5/301: rename dc/d1a/d21/c22 to dc/d1a/c68 0 2024-02-28T16:31:04.086 INFO:tasks.workunit.client.0.smithi001.stdout:0/305: dread d18/d22/d2b/d36/d48/f55 [0,4194304] 0 2024-02-28T16:31:04.089 INFO:tasks.workunit.client.1.smithi066.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-28T16:31:04.096 INFO:tasks.workunit.client.0.smithi001.stdout:2/321: dwrite d0/d2/d3/de/d22/f24 [0,4194304] 0 2024-02-28T16:31:04.096 INFO:tasks.workunit.client.0.smithi001.stdout:9/391: mknod d5/d45/d6f/c81 0 2024-02-28T16:31:04.109 INFO:tasks.workunit.client.0.smithi001.stdout:3/251: rename d4/c4b to d4/d1e/d29/c4c 0 2024-02-28T16:31:04.109 INFO:tasks.workunit.client.0.smithi001.stdout:5/302: chown c1 950467328 1 2024-02-28T16:31:04.109 INFO:tasks.workunit.client.0.smithi001.stdout:7/293: dwrite d3/d1c/f4b [0,4194304] 0 2024-02-28T16:31:04.112 INFO:tasks.workunit.client.0.smithi001.stdout:9/392: mknod d5/d45/d6f/c82 0 2024-02-28T16:31:04.113 INFO:tasks.workunit.client.0.smithi001.stdout:1/447: getdents d0/d9/d1a/d2b/d26/d27/d30/d67/d87 0 2024-02-28T16:31:04.116 INFO:tasks.workunit.client.0.smithi001.stdout:8/303: creat d4/f59 x:0 0 0 2024-02-28T16:31:04.118 INFO:tasks.workunit.client.0.smithi001.stdout:5/303: read dc/d12/f3f [4385599,7273] 0 2024-02-28T16:31:04.120 INFO:tasks.workunit.client.0.smithi001.stdout:1/448: truncate d0/d9/d56/f8d 582944 0 2024-02-28T16:31:04.125 INFO:tasks.workunit.client.0.smithi001.stdout:6/283: dwrite d9/db/d19/f1f [0,4194304] 0 2024-02-28T16:31:04.130 INFO:tasks.workunit.client.0.smithi001.stdout:5/304: symlink dc/df/l69 0 2024-02-28T16:31:04.136 INFO:tasks.workunit.client.0.smithi001.stdout:0/306: dread d18/d22/d2b/d36/f3c [0,4194304] 0 2024-02-28T16:31:04.138 INFO:tasks.workunit.client.0.smithi001.stdout:3/252: dread d4/d1e/f24 [0,4194304] 0 2024-02-28T16:31:04.139 INFO:tasks.workunit.client.0.smithi001.stdout:8/304: mknod d4/d7/d22/c5a 0 2024-02-28T16:31:04.143 INFO:tasks.workunit.client.0.smithi001.stdout:9/393: rename d5/d27/c7a to d5/d47/d5e/c83 0 2024-02-28T16:31:04.148 INFO:tasks.workunit.client.0.smithi001.stdout:1/449: symlink d0/d9/d1a/d2b/d26/d27/d30/d52/d84/l97 0 2024-02-28T16:31:04.152 INFO:tasks.workunit.client.0.smithi001.stdout:4/292: truncate d3/d1c/d58/f2a 3076991 0 2024-02-28T16:31:04.154 INFO:tasks.workunit.client.0.smithi001.stdout:5/305: dwrite dc/d1a/d21/f4b [0,4194304] 0 2024-02-28T16:31:04.155 INFO:tasks.workunit.client.0.smithi001.stdout:9/394: dread d5/d47/f40 [0,4194304] 0 2024-02-28T16:31:04.155 INFO:tasks.workunit.client.0.smithi001.stdout:4/293: write d3/d1c/d42/f53 [341756,98072] 0 2024-02-28T16:31:04.163 INFO:tasks.workunit.client.0.smithi001.stdout:9/395: write d5/d27/f31 [2715222,4625] 0 2024-02-28T16:31:04.163 INFO:tasks.workunit.client.0.smithi001.stdout:7/294: creat d3/f5f x:0 0 0 2024-02-28T16:31:04.164 INFO:tasks.workunit.client.0.smithi001.stdout:6/284: symlink d9/l63 0 2024-02-28T16:31:04.164 INFO:tasks.workunit.client.0.smithi001.stdout:5/306: truncate dc/d1a/d21/f37 622968 0 2024-02-28T16:31:04.164 INFO:tasks.workunit.client.0.smithi001.stdout:1/450: truncate d0/d9/d1a/d2b/d26/d27/d30/d4a/f89 937720 0 2024-02-28T16:31:04.164 INFO:tasks.workunit.client.0.smithi001.stdout:7/295: fsync d3/f1e 0 2024-02-28T16:31:04.165 INFO:tasks.workunit.client.0.smithi001.stdout:8/305: symlink d4/d7/d22/d27/d4b/l5b 0 2024-02-28T16:31:04.165 INFO:tasks.workunit.client.0.smithi001.stdout:3/253: readlink d4/l16 0 2024-02-28T16:31:04.167 INFO:tasks.workunit.client.0.smithi001.stdout:4/294: creat d3/d5/d7/d1f/d2e/f64 x:0 0 0 2024-02-28T16:31:04.169 INFO:tasks.workunit.client.0.smithi001.stdout:3/254: fdatasync d4/d6/d7/d23/f34 0 2024-02-28T16:31:04.169 INFO:tasks.workunit.client.0.smithi001.stdout:9/396: rmdir d5/d4d 39 2024-02-28T16:31:04.171 INFO:tasks.workunit.client.0.smithi001.stdout:8/306: creat d4/d32/f5c x:0 0 0 2024-02-28T16:31:04.175 INFO:tasks.workunit.client.0.smithi001.stdout:4/295: chown d3/d2d/d4c/d5e/f62 7336 1 2024-02-28T16:31:04.177 INFO:tasks.workunit.client.0.smithi001.stdout:4/296: chown d3/d1c/d42 1 1 2024-02-28T16:31:04.179 INFO:tasks.workunit.client.0.smithi001.stdout:1/451: truncate d0/d9/d1a/d2b/d26/d27/d30/f74 94101 0 2024-02-28T16:31:04.182 INFO:tasks.workunit.client.0.smithi001.stdout:0/307: dread d18/d22/d2f/d1c/f21 [0,4194304] 0 2024-02-28T16:31:04.184 INFO:tasks.workunit.client.0.smithi001.stdout:0/308: readlink d18/d22/d2b/l51 0 2024-02-28T16:31:04.188 INFO:tasks.workunit.client.0.smithi001.stdout:8/307: dread d4/d7/d22/d27/f3a [0,4194304] 0 2024-02-28T16:31:04.196 INFO:tasks.workunit.client.0.smithi001.stdout:7/296: write d3/f21 [4059338,126437] 0 2024-02-28T16:31:04.197 INFO:tasks.workunit.client.0.smithi001.stdout:9/397: rmdir d5/d27/d42/d48 39 2024-02-28T16:31:04.197 INFO:tasks.workunit.client.0.smithi001.stdout:4/297: dread d3/d5/d7/f21 [0,4194304] 0 2024-02-28T16:31:04.198 INFO:tasks.workunit.client.0.smithi001.stdout:9/398: chown d5/d27 584625 1 2024-02-28T16:31:04.199 INFO:tasks.workunit.client.0.smithi001.stdout:7/297: read d3/ff [2988562,15128] 0 2024-02-28T16:31:04.199 INFO:tasks.workunit.client.0.smithi001.stdout:3/255: mknod d4/d3c/c4d 0 2024-02-28T16:31:04.210 INFO:tasks.workunit.client.0.smithi001.stdout:1/452: mknod d0/d9/d1a/d2b/d26/d27/d30/d57/c98 0 2024-02-28T16:31:04.213 INFO:tasks.workunit.client.0.smithi001.stdout:8/308: chown d4/d32/c45 113 1 2024-02-28T16:31:04.214 INFO:tasks.workunit.client.0.smithi001.stdout:7/298: dread - d3/f45 zero size 2024-02-28T16:31:04.215 INFO:tasks.workunit.client.0.smithi001.stdout:1/453: dread - d0/d9/d56/d2f/f64 zero size 2024-02-28T16:31:04.216 INFO:tasks.workunit.client.0.smithi001.stdout:3/256: mknod d4/d18/c4e 0 2024-02-28T16:31:04.218 INFO:tasks.workunit.client.0.smithi001.stdout:5/307: rmdir dc/d12/d41/d54/d62 0 2024-02-28T16:31:04.222 INFO:tasks.workunit.client.0.smithi001.stdout:8/309: unlink d4/f55 0 2024-02-28T16:31:04.224 INFO:tasks.workunit.client.0.smithi001.stdout:3/257: read d4/d6/fc [1448737,13075] 0 2024-02-28T16:31:04.224 INFO:tasks.workunit.client.0.smithi001.stdout:5/308: chown dc/df/l2b 64989177 1 2024-02-28T16:31:04.227 INFO:tasks.workunit.client.0.smithi001.stdout:7/299: dread d3/d1c/f30 [0,4194304] 0 2024-02-28T16:31:04.227 INFO:tasks.workunit.client.0.smithi001.stdout:9/399: dwrite d5/f12 [0,4194304] 0 2024-02-28T16:31:04.231 INFO:tasks.workunit.client.0.smithi001.stdout:1/454: rename d0/d9/d56/l49 to d0/d9/d95/l99 0 2024-02-28T16:31:04.232 INFO:tasks.workunit.client.0.smithi001.stdout:4/298: dwrite d3/d1c/d58/f28 [0,4194304] 0 2024-02-28T16:31:04.236 INFO:tasks.workunit.client.0.smithi001.stdout:3/258: write d4/d6/d7/d23/f2b [2053979,10114] 0 2024-02-28T16:31:04.244 INFO:tasks.workunit.client.0.smithi001.stdout:5/309: rename dc/df/f29 to dc/d12/d41/d54/f6a 0 2024-02-28T16:31:04.244 INFO:tasks.workunit.client.0.smithi001.stdout:0/309: dwrite d18/d22/d2b/d36/d48/f55 [0,4194304] 0 2024-02-28T16:31:04.246 INFO:tasks.workunit.client.0.smithi001.stdout:5/310: stat dc/d12/d41/d54/f6a 0 2024-02-28T16:31:04.249 INFO:tasks.workunit.client.0.smithi001.stdout:0/310: readlink d18/d22/l3e 0 2024-02-28T16:31:04.250 INFO:tasks.workunit.client.0.smithi001.stdout:9/400: link d5/c8 d5/d47/d58/c84 0 2024-02-28T16:31:04.254 INFO:tasks.workunit.client.0.smithi001.stdout:1/455: creat d0/d9/d1a/d2b/d26/d27/d30/d67/d87/f9a x:0 0 0 2024-02-28T16:31:04.256 INFO:tasks.workunit.client.0.smithi001.stdout:0/311: chown d18/d22/d2b/d36/d48/f55 5 1 2024-02-28T16:31:04.260 INFO:tasks.workunit.client.0.smithi001.stdout:5/311: mkdir dc/d12/d56/d6b 0 2024-02-28T16:31:04.260 INFO:tasks.workunit.client.0.smithi001.stdout:9/401: symlink d5/d45/l85 0 2024-02-28T16:31:04.262 INFO:tasks.workunit.client.0.smithi001.stdout:0/312: write d18/f54 [50629,111512] 0 2024-02-28T16:31:04.263 INFO:tasks.workunit.client.0.smithi001.stdout:8/310: write d4/d7/d22/d27/f3a [3224266,27635] 0 2024-02-28T16:31:04.264 INFO:tasks.workunit.client.0.smithi001.stdout:3/259: dwrite d4/d6/d7/d23/f34 [0,4194304] 0 2024-02-28T16:31:04.268 INFO:tasks.workunit.client.0.smithi001.stdout:7/300: dwrite d3/d17/f2d [0,4194304] 0 2024-02-28T16:31:04.271 INFO:tasks.workunit.client.0.smithi001.stdout:7/301: write d3/d1c/f4b [2913250,5840] 0 2024-02-28T16:31:04.274 INFO:tasks.workunit.client.0.smithi001.stdout:1/456: mknod d0/d9/c9b 0 2024-02-28T16:31:04.282 INFO:tasks.workunit.client.0.smithi001.stdout:7/302: fdatasync d3/d1c/f4b 0 2024-02-28T16:31:04.285 INFO:tasks.workunit.client.0.smithi001.stdout:5/312: dwrite f5 [0,4194304] 0 2024-02-28T16:31:04.293 INFO:tasks.workunit.client.0.smithi001.stdout:5/313: write dc/df/f5d [269723,11324] 0 2024-02-28T16:31:04.298 INFO:tasks.workunit.client.0.smithi001.stdout:4/299: dread f2 [0,4194304] 0 2024-02-28T16:31:04.299 INFO:tasks.workunit.client.0.smithi001.stdout:9/402: rename d5/d45/l55 to d5/d27/d34/l86 0 2024-02-28T16:31:04.300 INFO:tasks.workunit.client.0.smithi001.stdout:0/313: dwrite d18/d22/d2b/d36/f56 [0,4194304] 0 2024-02-28T16:31:04.305 INFO:tasks.workunit.client.0.smithi001.stdout:3/260: link d4/d18/l48 d4/d18/l4f 0 2024-02-28T16:31:04.305 INFO:tasks.workunit.client.0.smithi001.stdout:1/457: mkdir d0/d9/d1a/d2b/d26/d27/d30/d52/d6b/d9c 0 2024-02-28T16:31:04.310 INFO:tasks.workunit.client.0.smithi001.stdout:0/314: dread d18/f1e [0,4194304] 0 2024-02-28T16:31:04.310 INFO:tasks.workunit.client.0.smithi001.stdout:1/458: chown d0/d9/d1a/d72/f8b 61475 1 2024-02-28T16:31:04.315 INFO:tasks.workunit.client.0.smithi001.stdout:0/315: write d18/f54 [897219,30351] 0 2024-02-28T16:31:04.316 INFO:tasks.workunit.client.0.smithi001.stdout:4/300: symlink d3/d1c/l65 0 2024-02-28T16:31:04.317 INFO:tasks.workunit.client.0.smithi001.stdout:3/261: rmdir d4 39 2024-02-28T16:31:04.320 INFO:tasks.workunit.client.0.smithi001.stdout:1/459: unlink d0/l3e 0 2024-02-28T16:31:04.323 INFO:tasks.workunit.client.0.smithi001.stdout:9/403: dwrite d5/d45/d6f/f7d [0,4194304] 0 2024-02-28T16:31:04.332 INFO:tasks.workunit.client.0.smithi001.stdout:0/316: mkdir d18/d22/d2b/d5f 0 2024-02-28T16:31:04.338 INFO:tasks.workunit.client.0.smithi001.stdout:4/301: unlink d3/d2d/f32 0 2024-02-28T16:31:04.340 INFO:tasks.workunit.client.0.smithi001.stdout:5/314: truncate dc/f1d 4524482 0 2024-02-28T16:31:04.344 INFO:tasks.workunit.client.0.smithi001.stdout:9/404: rename d5/d27/d34/d50/f68 to d5/d45/d6f/f87 0 2024-02-28T16:31:04.344 INFO:tasks.workunit.client.0.smithi001.stdout:5/315: write dc/d1a/d21/d2d/d3c/f47 [173454,20933] 0 2024-02-28T16:31:04.347 INFO:tasks.workunit.client.0.smithi001.stdout:8/311: truncate d4/d7/f2a 94907 0 2024-02-28T16:31:04.352 INFO:tasks.workunit.client.0.smithi001.stdout:8/312: write d4/d7/d22/d27/f3a [3161492,12845] 0 2024-02-28T16:31:04.353 INFO:tasks.workunit.client.0.smithi001.stdout:0/317: dwrite d18/d22/d3d/f52 [0,4194304] 0 2024-02-28T16:31:04.354 INFO:tasks.workunit.client.0.smithi001.stdout:4/302: write f2 [5047837,77767] 0 2024-02-28T16:31:04.357 INFO:tasks.workunit.client.0.smithi001.stdout:9/405: chown d5/d27/d34/d50/c51 0 1 2024-02-28T16:31:04.360 INFO:tasks.workunit.client.0.smithi001.stdout:3/262: truncate d4/d6/f40 1746513 0 2024-02-28T16:31:04.362 INFO:tasks.workunit.client.0.smithi001.stdout:3/263: write d4/d1e/d29/f47 [30874,112714] 0 2024-02-28T16:31:04.371 INFO:tasks.workunit.client.0.smithi001.stdout:9/406: symlink d5/d47/d58/l88 0 2024-02-28T16:31:04.377 INFO:tasks.workunit.client.0.smithi001.stdout:5/316: dwrite dc/d12/f16 [0,4194304] 0 2024-02-28T16:31:04.382 INFO:tasks.workunit.client.0.smithi001.stdout:8/313: rename d4/l4c to d4/d7/d22/l5d 0 2024-02-28T16:31:04.387 INFO:tasks.workunit.client.0.smithi001.stdout:9/407: creat d5/f89 x:0 0 0 2024-02-28T16:31:04.391 INFO:tasks.workunit.client.0.smithi001.stdout:0/318: rename d18/d22/d2b/d36/d4d/f4c to d18/d31/d4f/f60 0 2024-02-28T16:31:04.394 INFO:tasks.workunit.client.0.smithi001.stdout:1/460: dread d0/d9/d1a/d2b/f4d [0,4194304] 0 2024-02-28T16:31:04.394 INFO:tasks.workunit.client.0.smithi001.stdout:5/317: write f2 [825948,15876] 0 2024-02-28T16:31:04.399 INFO:tasks.workunit.client.0.smithi001.stdout:9/408: mkdir d5/d47/d33/d8a 0 2024-02-28T16:31:04.399 INFO:tasks.workunit.client.0.smithi001.stdout:0/319: mknod d18/d22/d2f/d1c/c61 0 2024-02-28T16:31:04.401 INFO:tasks.workunit.client.0.smithi001.stdout:5/318: unlink dc/d1a/d21/c33 0 2024-02-28T16:31:04.402 INFO:tasks.workunit.client.0.smithi001.stdout:3/264: dwrite d4/d1e/f33 [0,4194304] 0 2024-02-28T16:31:04.404 INFO:tasks.workunit.client.0.smithi001.stdout:9/409: mknod d5/d47/d33/d8a/c8b 0 2024-02-28T16:31:04.404 INFO:tasks.workunit.client.0.smithi001.stdout:0/320: write d18/d22/d2b/d36/f56 [4899723,20231] 0 2024-02-28T16:31:04.404 INFO:tasks.workunit.client.0.smithi001.stdout:3/265: chown d4 120914 1 2024-02-28T16:31:04.409 INFO:tasks.workunit.client.0.smithi001.stdout:9/410: write d5/fd [2813924,68295] 0 2024-02-28T16:31:04.410 INFO:tasks.workunit.client.0.smithi001.stdout:0/321: chown f11 1 1 2024-02-28T16:31:04.410 INFO:tasks.workunit.client.0.smithi001.stdout:3/266: symlink d4/d3c/l50 0 2024-02-28T16:31:04.411 INFO:tasks.workunit.client.0.smithi001.stdout:8/314: truncate d4/d32/f46 1020071 0 2024-02-28T16:31:04.413 INFO:tasks.workunit.client.0.smithi001.stdout:0/322: write d18/d22/d2b/f37 [1614400,111606] 0 2024-02-28T16:31:04.415 INFO:tasks.workunit.client.0.smithi001.stdout:8/315: creat d4/d32/f5e x:0 0 0 2024-02-28T16:31:04.417 INFO:tasks.workunit.client.0.smithi001.stdout:6/285: sync 2024-02-28T16:31:04.417 INFO:tasks.workunit.client.0.smithi001.stdout:2/322: sync 2024-02-28T16:31:04.418 INFO:tasks.workunit.client.0.smithi001.stdout:6/286: fdatasync d9/d39/f3a 0 2024-02-28T16:31:04.420 INFO:tasks.workunit.client.0.smithi001.stdout:3/267: creat d4/d6/f51 x:0 0 0 2024-02-28T16:31:04.422 INFO:tasks.workunit.client.0.smithi001.stdout:6/287: write d9/db/d10/d15/f40 [2314426,21133] 0 2024-02-28T16:31:04.423 INFO:tasks.workunit.client.0.smithi001.stdout:2/323: unlink d0/d2/f11 0 2024-02-28T16:31:04.424 INFO:tasks.workunit.client.0.smithi001.stdout:8/316: mkdir d4/d7/d22/d27/d5f 0 2024-02-28T16:31:04.426 INFO:tasks.workunit.client.0.smithi001.stdout:0/323: fdatasync d18/d22/d2f/f4e 0 2024-02-28T16:31:04.426 INFO:tasks.workunit.client.0.smithi001.stdout:3/268: chown d4/d1e/f33 1198628 1 2024-02-28T16:31:04.429 INFO:tasks.workunit.client.0.smithi001.stdout:3/269: chown d4/d3c/l50 55787 1 2024-02-28T16:31:04.431 INFO:tasks.workunit.client.0.smithi001.stdout:2/324: creat d0/d2/d3/d2c/f6e x:0 0 0 2024-02-28T16:31:04.432 INFO:tasks.workunit.client.0.smithi001.stdout:8/317: creat d4/d7/d22/d27/f60 x:0 0 0 2024-02-28T16:31:04.433 INFO:tasks.workunit.client.0.smithi001.stdout:0/324: symlink d18/d22/d2b/l62 0 2024-02-28T16:31:04.438 INFO:tasks.workunit.client.0.smithi001.stdout:2/325: dread - d0/d2/d3/de/d61/d12/d46/f57 zero size 2024-02-28T16:31:04.438 INFO:tasks.workunit.client.0.smithi001.stdout:3/270: read d4/d3c/f3e [885421,3394] 0 2024-02-28T16:31:04.442 INFO:tasks.workunit.client.0.smithi001.stdout:8/318: write d4/d7/d22/d27/f4a [2644042,127578] 0 2024-02-28T16:31:04.450 INFO:tasks.workunit.client.0.smithi001.stdout:2/326: write d0/d2/d3/de/d61/d12/d46/f57 [404411,23317] 0 2024-02-28T16:31:04.450 INFO:tasks.workunit.client.0.smithi001.stdout:3/271: dread d4/d1e/d29/f47 [0,4194304] 0 2024-02-28T16:31:04.450 INFO:tasks.workunit.client.0.smithi001.stdout:5/319: dread dc/d12/f3f [4194304,4194304] 0 2024-02-28T16:31:04.454 INFO:tasks.workunit.client.0.smithi001.stdout:6/288: link d9/db/d10/d15/d4e/l51 d9/db/d10/d15/d4e/l64 0 2024-02-28T16:31:04.455 INFO:tasks.workunit.client.0.smithi001.stdout:0/325: dread d18/d22/d3d/f52 [0,4194304] 0 2024-02-28T16:31:04.460 INFO:tasks.workunit.client.0.smithi001.stdout:9/411: dread d5/fd [0,4194304] 0 2024-02-28T16:31:04.463 INFO:tasks.workunit.client.0.smithi001.stdout:8/319: unlink d4/l49 0 2024-02-28T16:31:04.465 INFO:tasks.workunit.client.0.smithi001.stdout:2/327: creat d0/d2/d3/de/d22/f6f x:0 0 0 2024-02-28T16:31:04.467 INFO:tasks.workunit.client.0.smithi001.stdout:5/320: mkdir dc/d1a/d21/d2d/d3c/d4a/d6c 0 2024-02-28T16:31:04.471 INFO:tasks.workunit.client.0.smithi001.stdout:5/321: write dc/f5c [691068,53798] 0 2024-02-28T16:31:04.474 INFO:tasks.workunit.client.0.smithi001.stdout:3/272: getdents d4/d1e/d29 0 2024-02-28T16:31:04.477 INFO:tasks.workunit.client.0.smithi001.stdout:0/326: dwrite d18/f59 [0,4194304] 0 2024-02-28T16:31:04.480 INFO:tasks.workunit.client.0.smithi001.stdout:9/412: dread d5/f12 [0,4194304] 0 2024-02-28T16:31:04.482 INFO:tasks.workunit.client.0.smithi001.stdout:0/327: read d18/f54 [462948,59864] 0 2024-02-28T16:31:04.486 INFO:tasks.workunit.client.0.smithi001.stdout:9/413: symlink d5/d27/l8c 0 2024-02-28T16:31:04.488 INFO:tasks.workunit.client.0.smithi001.stdout:6/289: dread d9/d29/d2b/f33 [0,4194304] 0 2024-02-28T16:31:04.490 INFO:tasks.workunit.client.0.smithi001.stdout:6/290: readlink d9/l63 0 2024-02-28T16:31:04.490 INFO:tasks.workunit.client.0.smithi001.stdout:3/273: write d4/d1e/d29/f47 [330619,26827] 0 2024-02-28T16:31:04.495 INFO:tasks.workunit.client.0.smithi001.stdout:5/322: truncate dc/d1a/d21/f4b 4204021 0 2024-02-28T16:31:04.495 INFO:tasks.workunit.client.0.smithi001.stdout:6/291: write d9/db/d19/f1f [225272,82575] 0 2024-02-28T16:31:04.495 INFO:tasks.workunit.client.0.smithi001.stdout:3/274: truncate d4/d6/d7/f43 951513 0 2024-02-28T16:31:04.497 INFO:tasks.workunit.client.0.smithi001.stdout:9/414: link d5/d45/f71 d5/d27/d34/f8d 0 2024-02-28T16:31:04.501 INFO:tasks.workunit.client.0.smithi001.stdout:9/415: creat d5/d45/f8e x:0 0 0 2024-02-28T16:31:04.506 INFO:tasks.workunit.client.0.smithi001.stdout:9/416: rmdir d5/d45/d6f 39 2024-02-28T16:31:04.509 INFO:tasks.workunit.client.0.smithi001.stdout:9/417: write d5/d27/f2b [320962,37534] 0 2024-02-28T16:31:04.519 INFO:tasks.workunit.client.0.smithi001.stdout:3/275: dwrite d4/d1e/f25 [0,4194304] 0 2024-02-28T16:31:04.525 INFO:tasks.workunit.client.0.smithi001.stdout:9/418: dwrite d5/f3c [4194304,4194304] 0 2024-02-28T16:31:04.528 INFO:tasks.workunit.client.0.smithi001.stdout:9/419: dread - d5/f19 zero size 2024-02-28T16:31:04.535 INFO:tasks.workunit.client.0.smithi001.stdout:9/420: mknod d5/d45/d6f/c8f 0 2024-02-28T16:31:04.545 INFO:tasks.workunit.client.0.smithi001.stdout:3/276: dwrite d4/f26 [0,4194304] 0 2024-02-28T16:31:04.547 INFO:tasks.workunit.client.0.smithi001.stdout:3/277: write d4/f17 [1955393,68283] 0 2024-02-28T16:31:04.552 INFO:tasks.workunit.client.0.smithi001.stdout:9/421: dread d5/d27/d34/f8d [0,4194304] 0 2024-02-28T16:31:04.555 INFO:tasks.workunit.client.0.smithi001.stdout:3/278: creat d4/d6/f52 x:0 0 0 2024-02-28T16:31:04.566 INFO:tasks.workunit.client.0.smithi001.stdout:3/279: link d4/d6/d7/d23/c46 d4/d6/c53 0 2024-02-28T16:31:04.566 INFO:tasks.workunit.client.0.smithi001.stdout:9/422: dread d5/d27/f2c [0,4194304] 0 2024-02-28T16:31:04.566 INFO:tasks.workunit.client.0.smithi001.stdout:3/280: readlink d4/l16 0 2024-02-28T16:31:04.569 INFO:tasks.workunit.client.0.smithi001.stdout:3/281: readlink d4/d3c/l50 0 2024-02-28T16:31:04.569 INFO:tasks.workunit.client.0.smithi001.stdout:9/423: creat d5/d27/d34/f90 x:0 0 0 2024-02-28T16:31:04.570 INFO:tasks.workunit.client.0.smithi001.stdout:9/424: read - d5/d27/f73 zero size 2024-02-28T16:31:04.571 INFO:tasks.workunit.client.0.smithi001.stdout:9/425: chown d5 0 1 2024-02-28T16:31:04.575 INFO:tasks.workunit.client.0.smithi001.stdout:9/426: creat d5/d47/f91 x:0 0 0 2024-02-28T16:31:04.578 INFO:tasks.workunit.client.0.smithi001.stdout:3/282: dread d4/d18/f10 [4194304,4194304] 0 2024-02-28T16:31:04.578 INFO:tasks.workunit.client.0.smithi001.stdout:9/427: creat d5/d27/d34/d50/f92 x:0 0 0 2024-02-28T16:31:04.579 INFO:tasks.workunit.client.0.smithi001.stdout:3/283: chown d4/d1e/d29 904222 1 2024-02-28T16:31:04.579 INFO:tasks.workunit.client.0.smithi001.stdout:9/428: fdatasync d5/d27/f5c 0 2024-02-28T16:31:04.580 INFO:tasks.workunit.client.0.smithi001.stdout:3/284: chown d4/d18 10324784 1 2024-02-28T16:31:04.584 INFO:tasks.workunit.client.0.smithi001.stdout:9/429: dread d5/d27/d34/f8d [0,4194304] 0 2024-02-28T16:31:04.585 INFO:tasks.workunit.client.0.smithi001.stdout:9/430: chown d5/f3c 0 1 2024-02-28T16:31:04.595 INFO:tasks.workunit.client.0.smithi001.stdout:3/285: dwrite d4/f44 [0,4194304] 0 2024-02-28T16:31:04.601 INFO:tasks.workunit.client.0.smithi001.stdout:7/303: sync 2024-02-28T16:31:04.605 INFO:tasks.workunit.client.0.smithi001.stdout:7/304: mknod d3/d17/c60 0 2024-02-28T16:31:04.610 INFO:tasks.workunit.client.0.smithi001.stdout:3/286: dread d4/d1e/f22 [0,4194304] 0 2024-02-28T16:31:04.613 INFO:tasks.workunit.client.0.smithi001.stdout:3/287: symlink d4/d6/d7/d23/l54 0 2024-02-28T16:31:04.614 INFO:tasks.workunit.client.0.smithi001.stdout:3/288: fsync d4/f17 0 2024-02-28T16:31:04.628 INFO:tasks.workunit.client.0.smithi001.stdout:7/305: dwrite d3/f10 [0,4194304] 0 2024-02-28T16:31:04.631 INFO:tasks.workunit.client.0.smithi001.stdout:7/306: mkdir d3/d17/d61 0 2024-02-28T16:31:04.634 INFO:tasks.workunit.client.0.smithi001.stdout:7/307: write d3/f36 [9415040,33189] 0 2024-02-28T16:31:04.637 INFO:tasks.workunit.client.0.smithi001.stdout:7/308: fsync d3/d1c/f30 0 2024-02-28T16:31:04.638 INFO:tasks.workunit.client.0.smithi001.stdout:7/309: chown d3/d1c/l3e 11 1 2024-02-28T16:31:04.640 INFO:tasks.workunit.client.0.smithi001.stdout:3/289: dread d4/f14 [0,4194304] 0 2024-02-28T16:31:04.641 INFO:tasks.workunit.client.0.smithi001.stdout:7/310: creat d3/d1c/d41/f62 x:0 0 0 2024-02-28T16:31:04.645 INFO:tasks.workunit.client.0.smithi001.stdout:3/290: creat d4/d38/f55 x:0 0 0 2024-02-28T16:31:04.650 INFO:tasks.workunit.client.0.smithi001.stdout:3/291: rename d4/d1e/f24 to d4/d6/f56 0 2024-02-28T16:31:04.652 INFO:tasks.workunit.client.0.smithi001.stdout:3/292: mknod d4/d6/d7/c57 0 2024-02-28T16:31:04.654 INFO:tasks.workunit.client.0.smithi001.stdout:3/293: write d4/f14 [178155,14075] 0 2024-02-28T16:31:04.658 INFO:tasks.workunit.client.0.smithi001.stdout:3/294: creat d4/d6/d7/d23/f58 x:0 0 0 2024-02-28T16:31:04.658 INFO:tasks.workunit.client.0.smithi001.stdout:7/311: dwrite d3/f5f [0,4194304] 0 2024-02-28T16:31:04.659 INFO:tasks.workunit.client.0.smithi001.stdout:3/295: stat d4/d3c/l50 0 2024-02-28T16:31:04.660 INFO:tasks.workunit.client.0.smithi001.stdout:7/312: fdatasync d3/d17/d3d/f4d 0 2024-02-28T16:31:04.660 INFO:tasks.workunit.client.0.smithi001.stdout:3/296: readlink d4/l16 0 2024-02-28T16:31:04.665 INFO:tasks.workunit.client.0.smithi001.stdout:3/297: creat d4/d1e/d29/f59 x:0 0 0 2024-02-28T16:31:04.665 INFO:tasks.workunit.client.0.smithi001.stdout:7/313: mkdir d3/d1c/d41/d49/d63 0 2024-02-28T16:31:04.674 INFO:tasks.workunit.client.0.smithi001.stdout:3/298: link d4/d18/c12 d4/d6/d7/c5a 0 2024-02-28T16:31:04.677 INFO:tasks.workunit.client.0.smithi001.stdout:7/314: link d3/d17/d3d/l57 d3/l64 0 2024-02-28T16:31:04.680 INFO:tasks.workunit.client.0.smithi001.stdout:3/299: rename d4/d6/d7/d23/f58 to d4/d1e/d29/f5b 0 2024-02-28T16:31:04.682 INFO:tasks.workunit.client.0.smithi001.stdout:3/300: mknod d4/c5c 0 2024-02-28T16:31:04.692 INFO:tasks.workunit.client.0.smithi001.stdout:7/315: dwrite d3/d17/f3b [0,4194304] 0 2024-02-28T16:31:04.702 INFO:tasks.workunit.client.0.smithi001.stdout:3/301: dwrite d4/d6/d7/d23/f2b [0,4194304] 0 2024-02-28T16:31:04.718 INFO:tasks.workunit.client.0.smithi001.stdout:3/302: rmdir d4/d18 39 2024-02-28T16:31:04.719 INFO:tasks.workunit.client.0.smithi001.stdout:3/303: read d4/d6/fc [700120,85446] 0 2024-02-28T16:31:04.721 INFO:tasks.workunit.client.0.smithi001.stdout:3/304: write d4/d1e/d29/f59 [108907,101963] 0 2024-02-28T16:31:04.735 INFO:tasks.workunit.client.0.smithi001.stdout:3/305: dwrite d4/f17 [0,4194304] 0 2024-02-28T16:31:04.738 INFO:tasks.workunit.client.0.smithi001.stdout:3/306: creat d4/d6/d7/f5d x:0 0 0 2024-02-28T16:31:04.751 INFO:tasks.workunit.client.0.smithi001.stdout:3/307: dwrite d4/d1e/f33 [0,4194304] 0 2024-02-28T16:31:04.754 INFO:tasks.workunit.client.0.smithi001.stdout:3/308: mknod d4/d38/c5e 0 2024-02-28T16:31:04.758 INFO:tasks.workunit.client.0.smithi001.stdout:3/309: rename d4/d1e/d29 to d4/d18/d5f 0 2024-02-28T16:31:04.763 INFO:tasks.workunit.client.0.smithi001.stdout:3/310: symlink d4/d3c/l60 0 2024-02-28T16:31:04.765 INFO:tasks.workunit.client.0.smithi001.stdout:3/311: stat d4/d6/d7/c3b 0 2024-02-28T16:31:04.773 INFO:tasks.workunit.client.0.smithi001.stdout:3/312: creat d4/f61 x:0 0 0 2024-02-28T16:31:04.819 INFO:tasks.workunit.client.0.smithi001.stdout:4/303: sync 2024-02-28T16:31:04.820 INFO:tasks.workunit.client.0.smithi001.stdout:3/313: write d4/d6/f52 [951366,75691] 0 2024-02-28T16:31:04.823 INFO:tasks.workunit.client.0.smithi001.stdout:3/314: rmdir d4/d18 39 2024-02-28T16:31:04.825 INFO:tasks.workunit.client.0.smithi001.stdout:3/315: symlink d4/d1e/l62 0 2024-02-28T16:31:04.828 INFO:tasks.workunit.client.0.smithi001.stdout:3/316: creat d4/d6/d7/f63 x:0 0 0 2024-02-28T16:31:04.844 INFO:tasks.workunit.client.0.smithi001.stdout:3/317: dwrite d4/d1e/f27 [0,4194304] 0 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:04.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: Upgrade: Need to upgrade myself (mgr.smithi066.ztraxe) 2024-02-28T16:31:04.876 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:04 smithi001 ceph-mon[28991]: Upgrade: Need to upgrade myself (mgr.smithi066.ztraxe) 2024-02-28T16:31:04.888 INFO:tasks.workunit.client.0.smithi001.stdout:3/318: mknod d4/d18/d5f/c64 0 2024-02-28T16:31:04.891 INFO:tasks.workunit.client.0.smithi001.stdout:3/319: chown d4/d1e/l3f 11 1 2024-02-28T16:31:04.899 INFO:tasks.workunit.client.0.smithi001.stdout:3/320: rename d4/d6/d7/fd to d4/d6/d7/f65 0 2024-02-28T16:31:04.910 INFO:tasks.workunit.client.0.smithi001.stdout:3/321: link d4/d6/d7/c39 d4/d6/c66 0 2024-02-28T16:31:04.914 INFO:tasks.workunit.client.0.smithi001.stdout:3/322: mknod d4/d1e/c67 0 2024-02-28T16:31:04.924 INFO:tasks.workunit.client.0.smithi001.stdout:3/323: link d4/d18/d5f/f59 d4/d1e/f68 0 2024-02-28T16:31:04.929 INFO:tasks.workunit.client.0.smithi001.stdout:3/324: creat d4/d18/f69 x:0 0 0 2024-02-28T16:31:04.935 INFO:tasks.workunit.client.0.smithi001.stdout:3/325: mknod d4/c6a 0 2024-02-28T16:31:04.935 INFO:tasks.workunit.client.0.smithi001.stdout:1/461: sync 2024-02-28T16:31:04.951 INFO:tasks.workunit.client.0.smithi001.stdout:1/462: rename d0/d9/d1a/d2b/d26/d27/d30/d52/d6b to d0/d9/d1a/d72/d9d 0 2024-02-28T16:31:04.964 INFO:tasks.workunit.client.0.smithi001.stdout:1/463: unlink d0/d9/d1a/d2b/d26/c7a 0 2024-02-28T16:31:04.968 INFO:tasks.workunit.client.0.smithi001.stdout:1/464: readlink d0/l16 0 2024-02-28T16:31:04.970 INFO:tasks.workunit.client.0.smithi001.stdout:3/326: dwrite d4/f17 [0,4194304] 0 2024-02-28T16:31:04.973 INFO:tasks.workunit.client.0.smithi001.stdout:3/327: chown d4/d6/f52 37 1 2024-02-28T16:31:04.980 INFO:tasks.workunit.client.0.smithi001.stdout:1/465: symlink d0/d9/d1a/d72/l9e 0 2024-02-28T16:31:04.994 INFO:tasks.workunit.client.0.smithi001.stdout:1/466: truncate d0/d9/d1a/d2b/d26/d27/d30/d57/d8e/f93 900144 0 2024-02-28T16:31:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:31:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:31:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:31:04.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:31:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:31:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:31:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: Upgrade: Need to upgrade myself (mgr.smithi066.ztraxe) 2024-02-28T16:31:04.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:04 smithi066 ceph-mon[36236]: Upgrade: Need to upgrade myself (mgr.smithi066.ztraxe) 2024-02-28T16:31:04.999 INFO:tasks.workunit.client.0.smithi001.stdout:1/467: write d0/d9/d1a/d2b/f24 [262329,130905] 0 2024-02-28T16:31:05.006 INFO:tasks.workunit.client.0.smithi001.stdout:1/468: write d0/d9/f66 [601908,35932] 0 2024-02-28T16:31:05.011 INFO:tasks.workunit.client.0.smithi001.stdout:2/328: sync 2024-02-28T16:31:05.012 INFO:tasks.workunit.client.0.smithi001.stdout:8/320: sync 2024-02-28T16:31:05.012 INFO:tasks.workunit.client.0.smithi001.stdout:1/469: creat d0/d9/d56/d2f/f9f x:0 0 0 2024-02-28T16:31:05.013 INFO:tasks.workunit.client.0.smithi001.stdout:8/321: fsync d4/f3b 0 2024-02-28T16:31:05.017 INFO:tasks.workunit.client.0.smithi001.stdout:3/328: dread d4/d6/d7/f2f [0,4194304] 0 2024-02-28T16:31:05.020 INFO:tasks.workunit.client.0.smithi001.stdout:1/470: creat d0/d9/d1a/d2b/d26/d27/d30/d57/fa0 x:0 0 0 2024-02-28T16:31:05.021 INFO:tasks.workunit.client.0.smithi001.stdout:3/329: creat d4/d18/f6b x:0 0 0 2024-02-28T16:31:05.024 INFO:tasks.workunit.client.0.smithi001.stdout:1/471: getdents d0 0 2024-02-28T16:31:05.025 INFO:tasks.workunit.client.0.smithi001.stdout:2/329: dread d0/f33 [0,4194304] 0 2024-02-28T16:31:05.025 INFO:tasks.workunit.client.0.smithi001.stdout:3/330: write d4/d6/d7/f2f [198446,40053] 0 2024-02-28T16:31:05.029 INFO:tasks.workunit.client.0.smithi001.stdout:3/331: mknod d4/d1e/c6c 0 2024-02-28T16:31:05.031 INFO:tasks.workunit.client.0.smithi001.stdout:1/472: symlink d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/la1 0 2024-02-28T16:31:05.033 INFO:tasks.workunit.client.0.smithi001.stdout:3/332: creat d4/d18/f6d x:0 0 0 2024-02-28T16:31:05.034 INFO:tasks.workunit.client.0.smithi001.stdout:3/333: readlink d4/d1e/l45 0 2024-02-28T16:31:05.035 INFO:tasks.workunit.client.0.smithi001.stdout:1/473: symlink d0/la2 0 2024-02-28T16:31:05.038 INFO:tasks.workunit.client.0.smithi001.stdout:8/322: dwrite d4/d7/d22/d27/f60 [0,4194304] 0 2024-02-28T16:31:05.040 INFO:tasks.workunit.client.0.smithi001.stdout:8/323: stat d4/d7/d22/c26 0 2024-02-28T16:31:05.044 INFO:tasks.workunit.client.0.smithi001.stdout:8/324: chown d4/d7/d22/c5a 3382 1 2024-02-28T16:31:05.046 INFO:tasks.workunit.client.0.smithi001.stdout:8/325: chown d4/f3b 4342357 1 2024-02-28T16:31:05.049 INFO:tasks.workunit.client.0.smithi001.stdout:8/326: readlink d4/d7/d22/l4d 0 2024-02-28T16:31:05.051 INFO:tasks.workunit.client.0.smithi001.stdout:3/334: write d4/d1e/f68 [912396,126904] 0 2024-02-28T16:31:05.051 INFO:tasks.workunit.client.0.smithi001.stdout:8/327: chown d4/d7 16137 1 2024-02-28T16:31:05.052 INFO:tasks.workunit.client.0.smithi001.stdout:2/330: dwrite d0/d2/d3/de/d61/f67 [0,4194304] 0 2024-02-28T16:31:05.053 INFO:tasks.workunit.client.0.smithi001.stdout:1/474: fdatasync d0/d9/f7e 0 2024-02-28T16:31:05.055 INFO:tasks.workunit.client.0.smithi001.stdout:3/335: mknod d4/d18/c6e 0 2024-02-28T16:31:05.056 INFO:tasks.workunit.client.0.smithi001.stdout:1/475: write d0/d9/d56/f7f [470610,83097] 0 2024-02-28T16:31:05.056 INFO:tasks.workunit.client.0.smithi001.stdout:2/331: write d0/d2/d3/de/f5e [880532,93614] 0 2024-02-28T16:31:05.059 INFO:tasks.workunit.client.0.smithi001.stdout:8/328: rename f3 to d4/d7/d22/d27/d58/f61 0 2024-02-28T16:31:05.065 INFO:tasks.workunit.client.0.smithi001.stdout:1/476: link d0/d9/d1a/l32 d0/d9/d1a/d72/la3 0 2024-02-28T16:31:05.066 INFO:tasks.workunit.client.0.smithi001.stdout:8/329: read d4/d7/f2b [3363619,9056] 0 2024-02-28T16:31:05.068 INFO:tasks.workunit.client.0.smithi001.stdout:1/477: symlink d0/d9/d1a/d72/la4 0 2024-02-28T16:31:05.071 INFO:tasks.workunit.client.0.smithi001.stdout:8/330: mkdir d4/d7/d22/d27/d58/d62 0 2024-02-28T16:31:05.075 INFO:tasks.workunit.client.0.smithi001.stdout:8/331: rmdir d4/d7/d22 39 2024-02-28T16:31:05.077 INFO:tasks.workunit.client.0.smithi001.stdout:3/336: dwrite d4/d6/fc [0,4194304] 0 2024-02-28T16:31:05.081 INFO:tasks.workunit.client.0.smithi001.stdout:8/332: stat d4/d7/d22/d27/l33 0 2024-02-28T16:31:05.088 INFO:tasks.workunit.client.0.smithi001.stdout:3/337: rename d4/d18/c4e to d4/d1e/c6f 0 2024-02-28T16:31:05.089 INFO:tasks.workunit.client.0.smithi001.stdout:3/338: chown d4/d38/f55 8806 1 2024-02-28T16:31:05.096 INFO:tasks.workunit.client.0.smithi001.stdout:1/478: truncate d0/d9/f7e 1371392 0 2024-02-28T16:31:05.099 INFO:tasks.workunit.client.0.smithi001.stdout:8/333: dwrite d4/d7/d22/d27/f60 [0,4194304] 0 2024-02-28T16:31:05.102 INFO:tasks.workunit.client.0.smithi001.stdout:2/332: dread d0/d2/d3/d2c/f37 [0,4194304] 0 2024-02-28T16:31:05.103 INFO:tasks.workunit.client.0.smithi001.stdout:1/479: unlink d0/d9/d1a/d72/f8b 0 2024-02-28T16:31:05.106 INFO:tasks.workunit.client.0.smithi001.stdout:3/339: dread d4/d6/d7/d23/f2b [0,4194304] 0 2024-02-28T16:31:05.107 INFO:tasks.workunit.client.0.smithi001.stdout:8/334: creat d4/d7/d22/d27/d4b/f63 x:0 0 0 2024-02-28T16:31:05.108 INFO:tasks.workunit.client.0.smithi001.stdout:2/333: creat d0/d2/d3/d2c/f70 x:0 0 0 2024-02-28T16:31:05.110 INFO:tasks.workunit.client.0.smithi001.stdout:8/335: dread - d4/d7/d22/d27/f56 zero size 2024-02-28T16:31:05.110 INFO:tasks.workunit.client.0.smithi001.stdout:1/480: unlink d0/d9/d1a/c1e 0 2024-02-28T16:31:05.111 INFO:tasks.workunit.client.0.smithi001.stdout:3/340: symlink d4/d38/l70 0 2024-02-28T16:31:05.112 INFO:tasks.workunit.client.0.smithi001.stdout:2/334: write d0/d2/d3/de/d61/d12/d46/f66 [704687,68929] 0 2024-02-28T16:31:05.113 INFO:tasks.workunit.client.0.smithi001.stdout:3/341: readlink d4/d1e/l45 0 2024-02-28T16:31:05.115 INFO:tasks.workunit.client.0.smithi001.stdout:8/336: link d4/d7/d22/f48 d4/d47/f64 0 2024-02-28T16:31:05.116 INFO:tasks.workunit.client.0.smithi001.stdout:3/342: creat d4/d3c/f71 x:0 0 0 2024-02-28T16:31:05.117 INFO:tasks.workunit.client.0.smithi001.stdout:2/335: creat d0/d2/d3/d2c/f71 x:0 0 0 2024-02-28T16:31:05.120 INFO:tasks.workunit.client.0.smithi001.stdout:8/337: getdents d4/d7/d22/d27/d5f 0 2024-02-28T16:31:05.120 INFO:tasks.workunit.client.0.smithi001.stdout:1/481: getdents d0/d9/d1a/d2b/d26/d27/d30/d67/d87 0 2024-02-28T16:31:05.121 INFO:tasks.workunit.client.0.smithi001.stdout:5/323: sync 2024-02-28T16:31:05.122 INFO:tasks.workunit.client.0.smithi001.stdout:6/292: sync 2024-02-28T16:31:05.122 INFO:tasks.workunit.client.0.smithi001.stdout:9/431: sync 2024-02-28T16:31:05.122 INFO:tasks.workunit.client.0.smithi001.stdout:0/328: sync 2024-02-28T16:31:05.122 INFO:tasks.workunit.client.0.smithi001.stdout:7/316: sync 2024-02-28T16:31:05.122 INFO:tasks.workunit.client.0.smithi001.stdout:4/304: sync 2024-02-28T16:31:05.125 INFO:tasks.workunit.client.0.smithi001.stdout:6/293: fsync d9/d29/d2b/f56 0 2024-02-28T16:31:05.128 INFO:tasks.workunit.client.0.smithi001.stdout:7/317: unlink d3/cd 0 2024-02-28T16:31:05.129 INFO:tasks.workunit.client.0.smithi001.stdout:9/432: rmdir d5/d4d 39 2024-02-28T16:31:05.131 INFO:tasks.workunit.client.0.smithi001.stdout:9/433: dread - d5/d45/d6f/f7f zero size 2024-02-28T16:31:05.132 INFO:tasks.workunit.client.0.smithi001.stdout:3/343: dread d4/d6/d7/d23/f2d [0,4194304] 0 2024-02-28T16:31:05.132 INFO:tasks.workunit.client.0.smithi001.stdout:0/329: rename fa to d18/d5a/f63 0 2024-02-28T16:31:05.184 INFO:tasks.workunit.client.0.smithi001.stdout:4/305: mknod d3/d5/d7/c66 0 2024-02-28T16:31:05.184 INFO:tasks.workunit.client.0.smithi001.stdout:6/294: unlink d9/de/f4d 0 2024-02-28T16:31:05.185 INFO:tasks.workunit.client.0.smithi001.stdout:8/338: symlink d4/d7/d22/d27/d5f/l65 0 2024-02-28T16:31:05.186 INFO:tasks.workunit.client.0.smithi001.stdout:9/434: stat d5/c8 0 2024-02-28T16:31:05.186 INFO:tasks.workunit.client.0.smithi001.stdout:3/344: mknod d4/d38/c72 0 2024-02-28T16:31:05.186 INFO:tasks.workunit.client.0.smithi001.stdout:2/336: link d0/d2/d3/de/d61/c50 d0/d2/d3/de/d61/d12/d6a/c72 0 2024-02-28T16:31:05.189 INFO:tasks.workunit.client.0.smithi001.stdout:0/330: mknod d18/d22/d2b/d36/d48/c64 0 2024-02-28T16:31:05.194 INFO:tasks.workunit.client.0.smithi001.stdout:4/306: mknod d3/d1c/d42/c67 0 2024-02-28T16:31:05.199 INFO:tasks.workunit.client.0.smithi001.stdout:4/307: truncate d3/d1c/f4f 815661 0 2024-02-28T16:31:05.202 INFO:tasks.workunit.client.0.smithi001.stdout:8/339: dread d4/d7/d22/d27/f3a [0,4194304] 0 2024-02-28T16:31:05.204 INFO:tasks.workunit.client.0.smithi001.stdout:7/318: truncate d3/f8 607736 0 2024-02-28T16:31:05.212 INFO:tasks.workunit.client.0.smithi001.stdout:6/295: truncate f1 1078898 0 2024-02-28T16:31:05.213 INFO:tasks.workunit.client.0.smithi001.stdout:2/337: mkdir d0/d2/d3/d73 0 2024-02-28T16:31:05.215 INFO:tasks.workunit.client.0.smithi001.stdout:9/435: creat d5/d47/f93 x:0 0 0 2024-02-28T16:31:05.217 INFO:tasks.workunit.client.0.smithi001.stdout:7/319: creat d3/d1c/d41/d49/f65 x:0 0 0 2024-02-28T16:31:05.219 INFO:tasks.workunit.client.0.smithi001.stdout:0/331: dwrite d18/d22/d2b/f41 [0,4194304] 0 2024-02-28T16:31:05.223 INFO:tasks.workunit.client.0.smithi001.stdout:1/482: dwrite d0/d9/d1a/d2b/d26/d27/d30/f76 [0,4194304] 0 2024-02-28T16:31:05.223 INFO:tasks.workunit.client.0.smithi001.stdout:7/320: write d3/d17/d3d/f48 [980632,36152] 0 2024-02-28T16:31:05.227 INFO:tasks.workunit.client.0.smithi001.stdout:4/308: mkdir d3/d5/d7/d1f/d3a/d68 0 2024-02-28T16:31:05.231 INFO:tasks.workunit.client.0.smithi001.stdout:2/338: creat d0/d2/d3/de/d22/f74 x:0 0 0 2024-02-28T16:31:05.233 INFO:tasks.workunit.client.0.smithi001.stdout:7/321: write d3/d1c/f40 [632388,76930] 0 2024-02-28T16:31:05.234 INFO:tasks.workunit.client.0.smithi001.stdout:9/436: truncate d5/d45/f71 1517844 0 2024-02-28T16:31:05.235 INFO:tasks.workunit.client.0.smithi001.stdout:4/309: mkdir d3/d2d/d4c/d69 0 2024-02-28T16:31:05.236 INFO:tasks.workunit.client.0.smithi001.stdout:3/345: getdents d4/d1e 0 2024-02-28T16:31:05.240 INFO:tasks.workunit.client.0.smithi001.stdout:9/437: write d5/fe [4322796,8712] 0 2024-02-28T16:31:05.241 INFO:tasks.workunit.client.0.smithi001.stdout:3/346: dread - d4/f61 zero size 2024-02-28T16:31:05.245 INFO:tasks.workunit.client.0.smithi001.stdout:9/438: mknod d5/d27/d34/d50/c94 0 2024-02-28T16:31:05.248 INFO:tasks.workunit.client.0.smithi001.stdout:9/439: write d5/f79 [211480,71772] 0 2024-02-28T16:31:05.255 INFO:tasks.workunit.client.0.smithi001.stdout:6/296: dwrite d9/db/d19/f1f [0,4194304] 0 2024-02-28T16:31:05.258 INFO:tasks.workunit.client.0.smithi001.stdout:0/332: write f15 [231157,77728] 0 2024-02-28T16:31:05.258 INFO:tasks.workunit.client.0.smithi001.stdout:4/310: dread d3/d5/f38 [0,4194304] 0 2024-02-28T16:31:05.263 INFO:tasks.workunit.client.0.smithi001.stdout:3/347: rename d4/d18/c6e to d4/d6/d7/d23/c73 0 2024-02-28T16:31:05.264 INFO:tasks.workunit.client.0.smithi001.stdout:9/440: mknod d5/d4d/c95 0 2024-02-28T16:31:05.275 INFO:tasks.workunit.client.0.smithi001.stdout:3/348: write d4/d1e/f68 [2038339,47943] 0 2024-02-28T16:31:05.277 INFO:tasks.workunit.client.0.smithi001.stdout:6/297: rename d9/de/f4c to d9/db/d5f/f65 0 2024-02-28T16:31:05.277 INFO:tasks.workunit.client.0.smithi001.stdout:2/339: dwrite d0/d2/d3/de/d61/f1f [0,4194304] 0 2024-02-28T16:31:05.279 INFO:tasks.workunit.client.0.smithi001.stdout:9/441: mknod d5/d47/d65/c96 0 2024-02-28T16:31:05.279 INFO:tasks.workunit.client.0.smithi001.stdout:8/340: dread d4/d32/f50 [4194304,4194304] 0 2024-02-28T16:31:05.280 INFO:tasks.workunit.client.0.smithi001.stdout:6/298: fdatasync d9/db/fc 0 2024-02-28T16:31:05.284 INFO:tasks.workunit.client.0.smithi001.stdout:7/322: dwrite d3/d1c/f33 [0,4194304] 0 2024-02-28T16:31:05.290 INFO:tasks.workunit.client.0.smithi001.stdout:7/323: dread - d3/d17/f54 zero size 2024-02-28T16:31:05.293 INFO:tasks.workunit.client.0.smithi001.stdout:8/341: truncate d4/d7/d22/f48 550634 0 2024-02-28T16:31:05.296 INFO:tasks.workunit.client.0.smithi001.stdout:2/340: mknod d0/d2/d3/d36/c75 0 2024-02-28T16:31:05.305 INFO:tasks.workunit.client.0.smithi001.stdout:2/341: readlink d0/d2/d3/d36/l5d 0 2024-02-28T16:31:05.305 INFO:tasks.workunit.client.0.smithi001.stdout:4/311: dwrite d3/d5/f38 [0,4194304] 0 2024-02-28T16:31:05.308 INFO:tasks.workunit.client.0.smithi001.stdout:1/483: dread d0/d9/fa [0,4194304] 0 2024-02-28T16:31:05.308 INFO:tasks.workunit.client.0.smithi001.stdout:7/324: symlink d3/d17/d3d/d59/l66 0 2024-02-28T16:31:05.311 INFO:tasks.workunit.client.0.smithi001.stdout:3/349: rename d4/d38/c72 to d4/d6/c74 0 2024-02-28T16:31:05.312 INFO:tasks.workunit.client.0.smithi001.stdout:8/342: symlink d4/d7/l66 0 2024-02-28T16:31:05.313 INFO:tasks.workunit.client.0.smithi001.stdout:4/312: read - d3/d5/d7/d1f/d20/f5d zero size 2024-02-28T16:31:05.314 INFO:tasks.workunit.client.0.smithi001.stdout:2/342: mknod d0/d2/d3/d36/c76 0 2024-02-28T16:31:05.320 INFO:tasks.workunit.client.0.smithi001.stdout:3/350: write d4/d3c/f3e [2951605,93664] 0 2024-02-28T16:31:05.321 INFO:tasks.workunit.client.0.smithi001.stdout:4/313: symlink d3/d1c/d58/l6a 0 2024-02-28T16:31:05.322 INFO:tasks.workunit.client.0.smithi001.stdout:8/343: truncate d4/d7/f1d 4823116 0 2024-02-28T16:31:05.326 INFO:tasks.workunit.client.0.smithi001.stdout:2/343: stat d0/d2/d3/de/d61/d12/d46/l49 0 2024-02-28T16:31:05.328 INFO:tasks.workunit.client.0.smithi001.stdout:0/333: dread d18/d22/d2b/f37 [0,4194304] 0 2024-02-28T16:31:05.331 INFO:tasks.workunit.client.0.smithi001.stdout:9/442: rename d5/d27/d42/d48/c6d to d5/c97 0 2024-02-28T16:31:05.334 INFO:tasks.workunit.client.0.smithi001.stdout:4/314: read d3/d1c/f49 [2728595,50053] 0 2024-02-28T16:31:05.336 INFO:tasks.workunit.client.0.smithi001.stdout:9/443: chown d5/c1b 1133 1 2024-02-28T16:31:05.337 INFO:tasks.workunit.client.0.smithi001.stdout:6/299: dwrite d9/db/d10/f5b [0,4194304] 0 2024-02-28T16:31:05.340 INFO:tasks.workunit.client.0.smithi001.stdout:4/315: chown d3/d1c/f49 653 1 2024-02-28T16:31:05.353 INFO:tasks.workunit.client.0.smithi001.stdout:3/351: getdents d4/d18/d5f 0 2024-02-28T16:31:05.355 INFO:tasks.workunit.client.0.smithi001.stdout:9/444: truncate d5/d27/d42/d48/f4c 1638943 0 2024-02-28T16:31:05.359 INFO:tasks.workunit.client.0.smithi001.stdout:4/316: fdatasync d3/f12 0 2024-02-28T16:31:05.360 INFO:tasks.workunit.client.0.smithi001.stdout:2/344: write d0/d2/d3/de/d61/f23 [8762,52893] 0 2024-02-28T16:31:05.363 INFO:tasks.workunit.client.0.smithi001.stdout:4/317: fsync d3/d2d/f63 0 2024-02-28T16:31:05.363 INFO:tasks.workunit.client.0.smithi001.stdout:9/445: write d5/d27/f73 [822703,60105] 0 2024-02-28T16:31:05.367 INFO:tasks.workunit.client.0.smithi001.stdout:1/484: link d0/d9/d1a/d2b/d26/f34 d0/d9/d1a/d2b/d26/d27/d30/d4a/fa5 0 2024-02-28T16:31:05.367 INFO:tasks.workunit.client.0.smithi001.stdout:7/325: dread d3/d1c/f51 [0,4194304] 0 2024-02-28T16:31:05.367 INFO:tasks.workunit.client.0.smithi001.stdout:9/446: chown d5/d27/c59 2154676 1 2024-02-28T16:31:05.369 INFO:tasks.workunit.client.0.smithi001.stdout:2/345: read - d0/d2/d3/de/d22/f6f zero size 2024-02-28T16:31:05.372 INFO:tasks.workunit.client.0.smithi001.stdout:4/318: rename d3/d1c/f49 to d3/d1c/d58/f6b 0 2024-02-28T16:31:05.372 INFO:tasks.workunit.client.0.smithi001.stdout:2/346: read d0/d2/d3/de/d22/f48 [125685,71114] 0 2024-02-28T16:31:05.374 INFO:tasks.workunit.client.0.smithi001.stdout:2/347: fdatasync d0/d2/d3/de/d22/f74 0 2024-02-28T16:31:05.375 INFO:tasks.workunit.client.0.smithi001.stdout:7/326: creat d3/d17/d3d/d59/f67 x:0 0 0 2024-02-28T16:31:05.375 INFO:tasks.workunit.client.0.smithi001.stdout:6/300: truncate d9/d29/f44 4145597 0 2024-02-28T16:31:05.376 INFO:tasks.workunit.client.0.smithi001.stdout:9/447: mknod d5/d47/d33/c98 0 2024-02-28T16:31:05.377 INFO:tasks.workunit.client.0.smithi001.stdout:7/327: chown d3/d1c/f30 3 1 2024-02-28T16:31:05.379 INFO:tasks.workunit.client.0.smithi001.stdout:6/301: write d9/f4f [298576,48939] 0 2024-02-28T16:31:05.388 INFO:tasks.workunit.client.0.smithi001.stdout:4/319: unlink d3/l4 0 2024-02-28T16:31:05.388 INFO:tasks.workunit.client.0.smithi001.stdout:0/334: dread d18/d22/d2f/f24 [0,4194304] 0 2024-02-28T16:31:05.388 INFO:tasks.workunit.client.0.smithi001.stdout:9/448: mkdir d5/d99 0 2024-02-28T16:31:05.389 INFO:tasks.workunit.client.0.smithi001.stdout:1/485: dread d0/d9/d56/f6c [0,4194304] 0 2024-02-28T16:31:05.390 INFO:tasks.workunit.client.0.smithi001.stdout:4/320: mkdir d3/d1c/d58/d6c 0 2024-02-28T16:31:05.391 INFO:tasks.workunit.client.0.smithi001.stdout:0/335: creat d18/d22/d2b/d36/d4d/f65 x:0 0 0 2024-02-28T16:31:05.394 INFO:tasks.workunit.client.0.smithi001.stdout:1/486: write d0/d9/d56/d2f/f9f [856913,80813] 0 2024-02-28T16:31:05.394 INFO:tasks.workunit.client.0.smithi001.stdout:6/302: dread d9/db/d10/f11 [0,4194304] 0 2024-02-28T16:31:05.394 INFO:tasks.workunit.client.0.smithi001.stdout:9/449: mkdir d5/d9a 0 2024-02-28T16:31:05.396 INFO:tasks.workunit.client.0.smithi001.stdout:0/336: chown d18/d22/d2f/l2d 55749 1 2024-02-28T16:31:05.396 INFO:tasks.workunit.client.0.smithi001.stdout:9/450: fsync d5/d27/f2b 0 2024-02-28T16:31:05.398 INFO:tasks.workunit.client.0.smithi001.stdout:4/321: mknod d3/d5/d7/d1f/d3a/d68/c6d 0 2024-02-28T16:31:05.399 INFO:tasks.workunit.client.0.smithi001.stdout:6/303: creat d9/de/f66 x:0 0 0 2024-02-28T16:31:05.400 INFO:tasks.workunit.client.0.smithi001.stdout:4/322: chown d3/d5/d7 240234325 1 2024-02-28T16:31:05.401 INFO:tasks.workunit.client.0.smithi001.stdout:7/328: dwrite d3/fa [0,4194304] 0 2024-02-28T16:31:05.401 INFO:tasks.workunit.client.0.smithi001.stdout:9/451: creat d5/d47/d58/f9b x:0 0 0 2024-02-28T16:31:05.404 INFO:tasks.workunit.client.0.smithi001.stdout:0/337: creat d18/d22/f66 x:0 0 0 2024-02-28T16:31:05.405 INFO:tasks.workunit.client.0.smithi001.stdout:9/452: chown d5/d27/l80 2506 1 2024-02-28T16:31:05.407 INFO:tasks.workunit.client.0.smithi001.stdout:4/323: mkdir d3/d1c/d58/d6e 0 2024-02-28T16:31:05.408 INFO:tasks.workunit.client.0.smithi001.stdout:7/329: mknod d3/d17/d3d/d59/c68 0 2024-02-28T16:31:05.412 INFO:tasks.workunit.client.0.smithi001.stdout:4/324: mknod d3/c6f 0 2024-02-28T16:31:05.412 INFO:tasks.workunit.client.0.smithi001.stdout:2/348: dread d0/d2/d3/de/fa [0,4194304] 0 2024-02-28T16:31:05.414 INFO:tasks.workunit.client.0.smithi001.stdout:9/453: mknod d5/d47/c9c 0 2024-02-28T16:31:05.414 INFO:tasks.workunit.client.0.smithi001.stdout:1/487: rename d0/d9/d1a/d2b/d26/d27/d30/d57/l63 to d0/d9/d1a/d72/d9d/la6 0 2024-02-28T16:31:05.417 INFO:tasks.workunit.client.0.smithi001.stdout:9/454: mkdir d5/d27/d34/d50/d9d 0 2024-02-28T16:31:05.420 INFO:tasks.workunit.client.0.smithi001.stdout:9/455: dread - d5/f89 zero size 2024-02-28T16:31:05.421 INFO:tasks.workunit.client.0.smithi001.stdout:7/330: truncate d3/d17/d3d/f4d 335370 0 2024-02-28T16:31:05.425 INFO:tasks.workunit.client.0.smithi001.stdout:9/456: write d5/d47/f91 [231920,84175] 0 2024-02-28T16:31:05.428 INFO:tasks.workunit.client.0.smithi001.stdout:1/488: chown d0/d9/d1a/d2b/d26/d27/d30/d4a/c5f 103 1 2024-02-28T16:31:05.431 INFO:tasks.workunit.client.0.smithi001.stdout:7/331: mknod d3/c69 0 2024-02-28T16:31:05.434 INFO:tasks.workunit.client.0.smithi001.stdout:4/325: dwrite d3/d2d/d4c/f50 [0,4194304] 0 2024-02-28T16:31:05.435 INFO:tasks.workunit.client.0.smithi001.stdout:9/457: symlink d5/l9e 0 2024-02-28T16:31:05.437 INFO:tasks.workunit.client.0.smithi001.stdout:7/332: symlink d3/d17/d3d/l6a 0 2024-02-28T16:31:05.439 INFO:tasks.workunit.client.0.smithi001.stdout:7/333: read d3/d1c/f33 [3479314,49631] 0 2024-02-28T16:31:05.440 INFO:tasks.workunit.client.0.smithi001.stdout:2/349: getdents d0/d2/d3/de/d61/d12/d6a 0 2024-02-28T16:31:05.441 INFO:tasks.workunit.client.0.smithi001.stdout:9/458: symlink d5/d47/d58/l9f 0 2024-02-28T16:31:05.444 INFO:tasks.workunit.client.0.smithi001.stdout:0/338: truncate d18/d5a/f63 4294 0 2024-02-28T16:31:05.449 INFO:tasks.workunit.client.0.smithi001.stdout:7/334: rename d3/d1c/f30 to d3/d17/d3d/d59/f6b 0 2024-02-28T16:31:05.449 INFO:tasks.workunit.client.0.smithi001.stdout:0/339: write d18/d22/d2f/d1c/f30 [5209110,122602] 0 2024-02-28T16:31:05.452 INFO:tasks.workunit.client.0.smithi001.stdout:4/326: dread f2 [0,4194304] 0 2024-02-28T16:31:05.456 INFO:tasks.workunit.client.0.smithi001.stdout:2/350: dwrite d0/d2/d3/d36/f4b [0,4194304] 0 2024-02-28T16:31:05.458 INFO:tasks.workunit.client.0.smithi001.stdout:2/351: write d0/f40 [655852,44099] 0 2024-02-28T16:31:05.461 INFO:tasks.workunit.client.0.smithi001.stdout:2/352: read d0/d2/d3/d36/f39 [3262126,68146] 0 2024-02-28T16:31:05.464 INFO:tasks.workunit.client.0.smithi001.stdout:2/353: fsync d0/d2/d3/de/d61/d12/f41 0 2024-02-28T16:31:05.521 INFO:tasks.workunit.client.0.smithi001.stdout:5/324: sync 2024-02-28T16:31:05.523 INFO:tasks.workunit.client.0.smithi001.stdout:5/325: stat dc/d1a/l26 0 2024-02-28T16:31:05.526 INFO:tasks.workunit.client.0.smithi001.stdout:5/326: chown dc/d1a/d21/d2d/d3c/l58 16323 1 2024-02-28T16:31:05.548 INFO:tasks.workunit.client.0.smithi001.stdout:7/335: creat d3/d1c/d41/d49/f6c x:0 0 0 2024-02-28T16:31:05.559 INFO:tasks.workunit.client.0.smithi001.stdout:5/327: mknod dc/d12/c6d 0 2024-02-28T16:31:05.563 INFO:tasks.workunit.client.0.smithi001.stdout:6/304: dwrite d9/de/f5a [4194304,4194304] 0 2024-02-28T16:31:05.564 INFO:tasks.workunit.client.0.smithi001.stdout:9/459: dwrite d5/d45/f71 [0,4194304] 0 2024-02-28T16:31:05.564 INFO:tasks.workunit.client.0.smithi001.stdout:5/328: fdatasync dc/d1a/d21/d2d/f44 0 2024-02-28T16:31:05.567 INFO:tasks.workunit.client.0.smithi001.stdout:7/336: creat d3/d1c/d41/f6d x:0 0 0 2024-02-28T16:31:05.569 INFO:tasks.workunit.client.0.smithi001.stdout:2/354: mknod d0/c77 0 2024-02-28T16:31:05.575 INFO:tasks.workunit.client.0.smithi001.stdout:6/305: mknod d9/db/d19/c67 0 2024-02-28T16:31:05.577 INFO:tasks.workunit.client.0.smithi001.stdout:5/329: symlink dc/d1a/d21/l6e 0 2024-02-28T16:31:05.578 INFO:tasks.workunit.client.0.smithi001.stdout:6/306: chown d9/db/c38 15999286 1 2024-02-28T16:31:05.580 INFO:tasks.workunit.client.0.smithi001.stdout:6/307: fdatasync d9/db/f45 0 2024-02-28T16:31:05.580 INFO:tasks.workunit.client.0.smithi001.stdout:5/330: chown dc/f20 27421755 1 2024-02-28T16:31:05.584 INFO:tasks.workunit.client.0.smithi001.stdout:2/355: symlink d0/d2/l78 0 2024-02-28T16:31:05.585 INFO:tasks.workunit.client.0.smithi001.stdout:0/340: getdents d18/d22/d2f/d1c 0 2024-02-28T16:31:05.586 INFO:tasks.workunit.client.0.smithi001.stdout:4/327: dwrite d3/d2d/f45 [0,4194304] 0 2024-02-28T16:31:05.596 INFO:tasks.workunit.client.0.smithi001.stdout:5/331: creat dc/d12/f6f x:0 0 0 2024-02-28T16:31:05.596 INFO:tasks.workunit.client.0.smithi001.stdout:7/337: truncate d3/d17/d3d/f4d 534841 0 2024-02-28T16:31:05.599 INFO:tasks.workunit.client.0.smithi001.stdout:5/332: symlink dc/d1a/d52/l70 0 2024-02-28T16:31:05.599 INFO:tasks.workunit.client.0.smithi001.stdout:2/356: unlink d0/l44 0 2024-02-28T16:31:05.602 INFO:tasks.workunit.client.0.smithi001.stdout:6/308: dread d9/db/d19/f1f [0,4194304] 0 2024-02-28T16:31:05.602 INFO:tasks.workunit.client.0.smithi001.stdout:9/460: fdatasync d5/d27/d34/f8d 0 2024-02-28T16:31:05.605 INFO:tasks.workunit.client.0.smithi001.stdout:5/333: truncate fa 376227 0 2024-02-28T16:31:05.606 INFO:tasks.workunit.client.0.smithi001.stdout:5/334: dread - dc/d12/f6f zero size 2024-02-28T16:31:05.607 INFO:tasks.workunit.client.0.smithi001.stdout:2/357: readlink d0/d2/d3/de/d61/d12/d55/l58 0 2024-02-28T16:31:05.612 INFO:tasks.workunit.client.0.smithi001.stdout:2/358: symlink d0/l79 0 2024-02-28T16:31:05.618 INFO:tasks.workunit.client.0.smithi001.stdout:6/309: link d9/db/fd d9/db/f68 0 2024-02-28T16:31:05.619 INFO:tasks.workunit.client.0.smithi001.stdout:2/359: dread d0/d2/d3/de/d22/f48 [0,4194304] 0 2024-02-28T16:31:05.623 INFO:tasks.workunit.client.0.smithi001.stdout:9/461: rename d5/d27/d42/d48/f4c to d5/d45/fa0 0 2024-02-28T16:31:05.624 INFO:tasks.workunit.client.0.smithi001.stdout:4/328: truncate d3/d1c/d58/f28 2294748 0 2024-02-28T16:31:05.624 INFO:tasks.workunit.client.0.smithi001.stdout:0/341: fdatasync d18/d22/d2f/d1c/f38 0 2024-02-28T16:31:05.625 INFO:tasks.workunit.client.0.smithi001.stdout:9/462: fsync d5/f1a 0 2024-02-28T16:31:05.628 INFO:tasks.workunit.client.0.smithi001.stdout:6/310: mkdir d9/db/d5f/d69 0 2024-02-28T16:31:05.628 INFO:tasks.workunit.client.0.smithi001.stdout:5/335: truncate f7 1782510 0 2024-02-28T16:31:05.629 INFO:tasks.workunit.client.0.smithi001.stdout:6/311: read - d9/f52 zero size 2024-02-28T16:31:05.630 INFO:tasks.workunit.client.0.smithi001.stdout:2/360: symlink d0/d2/l7a 0 2024-02-28T16:31:05.631 INFO:tasks.workunit.client.0.smithi001.stdout:7/338: dread d3/f21 [0,4194304] 0 2024-02-28T16:31:05.631 INFO:tasks.workunit.client.0.smithi001.stdout:6/312: fsync f5 0 2024-02-28T16:31:05.632 INFO:tasks.workunit.client.0.smithi001.stdout:0/342: creat d18/d22/d3d/f67 x:0 0 0 2024-02-28T16:31:05.635 INFO:tasks.workunit.client.0.smithi001.stdout:4/329: creat d3/d5/d7/d1f/d3a/f70 x:0 0 0 2024-02-28T16:31:05.635 INFO:tasks.workunit.client.0.smithi001.stdout:0/343: chown d18/d22/d3d/f50 2496 1 2024-02-28T16:31:05.638 INFO:tasks.workunit.client.0.smithi001.stdout:8/344: sync 2024-02-28T16:31:05.638 INFO:tasks.workunit.client.0.smithi001.stdout:1/489: sync 2024-02-28T16:31:05.638 INFO:tasks.workunit.client.0.smithi001.stdout:3/352: sync 2024-02-28T16:31:05.638 INFO:tasks.workunit.client.0.smithi001.stdout:4/330: write d3/d5/f34 [694275,122027] 0 2024-02-28T16:31:05.639 INFO:tasks.workunit.client.0.smithi001.stdout:9/463: symlink d5/d47/la1 0 2024-02-28T16:31:05.641 INFO:tasks.workunit.client.0.smithi001.stdout:7/339: mkdir d3/d17/d3d/d59/d6e 0 2024-02-28T16:31:05.643 INFO:tasks.workunit.client.0.smithi001.stdout:7/340: stat d3/d17/d3d/f48 0 2024-02-28T16:31:05.645 INFO:tasks.workunit.client.0.smithi001.stdout:0/344: symlink d18/d31/l68 0 2024-02-28T16:31:05.646 INFO:tasks.workunit.client.0.smithi001.stdout:6/313: mkdir d9/d29/d6a 0 2024-02-28T16:31:05.651 INFO:tasks.workunit.client.0.smithi001.stdout:1/490: mknod d0/d9/d1a/d2b/d26/d27/d30/d52/d84/ca7 0 2024-02-28T16:31:05.652 INFO:tasks.workunit.client.0.smithi001.stdout:4/331: write d3/d1c/f4f [1054266,62964] 0 2024-02-28T16:31:05.656 INFO:tasks.workunit.client.0.smithi001.stdout:2/361: rename d0/d2/d3/de/d22/l4c to d0/d2/d3/de/l7b 0 2024-02-28T16:31:05.659 INFO:tasks.workunit.client.0.smithi001.stdout:5/336: dwrite dc/d1a/d21/d2d/d3c/f53 [0,4194304] 0 2024-02-28T16:31:05.662 INFO:tasks.workunit.client.0.smithi001.stdout:4/332: truncate d3/d1c/d58/ff 1462868 0 2024-02-28T16:31:05.662 INFO:tasks.workunit.client.0.smithi001.stdout:7/341: getdents d3/d17/d61 0 2024-02-28T16:31:05.663 INFO:tasks.workunit.client.0.smithi001.stdout:0/345: rename d18/d22/d2b/d36/f3c to d18/d22/d2b/d36/f69 0 2024-02-28T16:31:05.664 INFO:tasks.workunit.client.0.smithi001.stdout:9/464: dread d5/fd [0,4194304] 0 2024-02-28T16:31:05.666 INFO:tasks.workunit.client.0.smithi001.stdout:6/314: write d9/db/d10/f16 [1425369,17640] 0 2024-02-28T16:31:05.666 INFO:tasks.workunit.client.0.smithi001.stdout:5/337: write dc/d1a/d21/f37 [1116945,36232] 0 2024-02-28T16:31:05.670 INFO:tasks.workunit.client.0.smithi001.stdout:8/345: dwrite d4/f59 [0,4194304] 0 2024-02-28T16:31:05.676 INFO:tasks.workunit.client.0.smithi001.stdout:2/362: symlink d0/l7c 0 2024-02-28T16:31:05.679 INFO:tasks.workunit.client.0.smithi001.stdout:5/338: stat dc/f1d 0 2024-02-28T16:31:05.680 INFO:tasks.workunit.client.0.smithi001.stdout:8/346: mkdir d4/d7/d22/d27/d5f/d67 0 2024-02-28T16:31:05.682 INFO:tasks.workunit.client.0.smithi001.stdout:6/315: getdents d9/d29/d6a 0 2024-02-28T16:31:05.682 INFO:tasks.workunit.client.0.smithi001.stdout:2/363: mknod d0/d2/d3/de/d61/d12/d55/c7d 0 2024-02-28T16:31:05.684 INFO:tasks.workunit.client.0.smithi001.stdout:8/347: fsync d4/d7/f40 0 2024-02-28T16:31:05.688 INFO:tasks.workunit.client.0.smithi001.stdout:6/316: fsync d9/d29/d2b/f56 0 2024-02-28T16:31:05.689 INFO:tasks.workunit.client.0.smithi001.stdout:2/364: write d0/d2/f14 [3325764,42285] 0 2024-02-28T16:31:05.689 INFO:tasks.workunit.client.0.smithi001.stdout:5/339: creat dc/d12/d41/d54/f71 x:0 0 0 2024-02-28T16:31:05.692 INFO:tasks.workunit.client.0.smithi001.stdout:8/348: chown d4/d7/d22/d27/d58 6933 1 2024-02-28T16:31:05.698 INFO:tasks.workunit.client.0.smithi001.stdout:6/317: chown d9/de/f2c 320 1 2024-02-28T16:31:05.700 INFO:tasks.workunit.client.0.smithi001.stdout:2/365: fdatasync d0/d2/d3/de/d22/f24 0 2024-02-28T16:31:05.702 INFO:tasks.workunit.client.0.smithi001.stdout:1/491: dwrite d0/d9/d1a/d2b/d26/f34 [0,4194304] 0 2024-02-28T16:31:05.703 INFO:tasks.workunit.client.0.smithi001.stdout:5/340: rename f4 to dc/d1a/d52/f72 0 2024-02-28T16:31:05.704 INFO:tasks.workunit.client.0.smithi001.stdout:9/465: dwrite d5/d27/f5c [0,4194304] 0 2024-02-28T16:31:05.707 INFO:tasks.workunit.client.0.smithi001.stdout:5/341: readlink dc/d12/l5a 0 2024-02-28T16:31:05.707 INFO:tasks.workunit.client.0.smithi001.stdout:7/342: dwrite d3/d1c/d41/f6d [0,4194304] 0 2024-02-28T16:31:05.707 INFO:tasks.workunit.client.0.smithi001.stdout:2/366: stat d0/c77 0 2024-02-28T16:31:05.710 INFO:tasks.workunit.client.0.smithi001.stdout:0/346: dwrite d18/d22/d2b/d36/f56 [4194304,4194304] 0 2024-02-28T16:31:05.717 INFO:tasks.workunit.client.0.smithi001.stdout:0/347: rmdir d18/d22/d2b/d36/d4d 39 2024-02-28T16:31:05.717 INFO:tasks.workunit.client.0.smithi001.stdout:9/466: chown d5/f89 1453100 1 2024-02-28T16:31:05.721 INFO:tasks.workunit.client.0.smithi001.stdout:5/342: dread dc/d1a/d21/f48 [0,4194304] 0 2024-02-28T16:31:05.721 INFO:tasks.workunit.client.0.smithi001.stdout:6/318: link d9/db/d10/d15/f31 d9/db/d5f/d62/f6b 0 2024-02-28T16:31:05.725 INFO:tasks.workunit.client.0.smithi001.stdout:6/319: write d9/d29/f3b [118167,114010] 0 2024-02-28T16:31:05.725 INFO:tasks.workunit.client.0.smithi001.stdout:9/467: symlink d5/d27/d34/d50/la2 0 2024-02-28T16:31:05.727 INFO:tasks.workunit.client.0.smithi001.stdout:6/320: chown d9/d29/f48 101497358 1 2024-02-28T16:31:05.729 INFO:tasks.workunit.client.0.smithi001.stdout:0/348: symlink d18/d22/d2b/d36/d46/l6a 0 2024-02-28T16:31:05.730 INFO:tasks.workunit.client.0.smithi001.stdout:1/492: dread d0/d9/f7e [0,4194304] 0 2024-02-28T16:31:05.732 INFO:tasks.workunit.client.0.smithi001.stdout:6/321: mknod d9/d39/c6c 0 2024-02-28T16:31:05.735 INFO:tasks.workunit.client.0.smithi001.stdout:6/322: dread - d9/db/d10/d15/f31 zero size 2024-02-28T16:31:05.740 INFO:tasks.workunit.client.0.smithi001.stdout:0/349: rename d18/d22/d2f/d1c/l29 to d18/l6b 0 2024-02-28T16:31:05.743 INFO:tasks.workunit.client.0.smithi001.stdout:5/343: dread dc/d1a/d21/d2d/d3c/f46 [0,4194304] 0 2024-02-28T16:31:05.749 INFO:tasks.workunit.client.0.smithi001.stdout:8/349: write d4/d7/f23 [788636,105565] 0 2024-02-28T16:31:05.749 INFO:tasks.workunit.client.0.smithi001.stdout:0/350: creat d18/d31/f6c x:0 0 0 2024-02-28T16:31:05.749 INFO:tasks.workunit.client.0.smithi001.stdout:1/493: link d0/d9/d1a/d2b/d26/d27/d30/d67/d87/l71 d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/la8 0 2024-02-28T16:31:05.754 INFO:tasks.workunit.client.0.smithi001.stdout:5/344: fsync fa 0 2024-02-28T16:31:05.755 INFO:tasks.workunit.client.0.smithi001.stdout:7/343: write d3/f8 [893795,42936] 0 2024-02-28T16:31:05.757 INFO:tasks.workunit.client.0.smithi001.stdout:1/494: chown d0/d9/d1a/d2b/d26/d27/d30/d57/d8e 808781311 1 2024-02-28T16:31:05.757 INFO:tasks.workunit.client.0.smithi001.stdout:5/345: readlink dc/d1a/l5b 0 2024-02-28T16:31:05.761 INFO:tasks.workunit.client.0.smithi001.stdout:8/350: mknod d4/d7/d22/c68 0 2024-02-28T16:31:05.761 INFO:tasks.workunit.client.0.smithi001.stdout:5/346: stat dc/d1a/d21 0 2024-02-28T16:31:05.762 INFO:tasks.workunit.client.0.smithi001.stdout:2/367: dread d0/d2/d3/de/d61/f15 [0,4194304] 0 2024-02-28T16:31:05.763 INFO:tasks.workunit.client.0.smithi001.stdout:6/323: dwrite d9/d29/d2b/f30 [0,4194304] 0 2024-02-28T16:31:05.763 INFO:tasks.workunit.client.0.smithi001.stdout:0/351: link d18/d22/d3d/f5e d18/f6d 0 2024-02-28T16:31:05.766 INFO:tasks.workunit.client.0.smithi001.stdout:8/351: unlink d4/d32/f5e 0 2024-02-28T16:31:05.768 INFO:tasks.workunit.client.0.smithi001.stdout:8/352: dread - d4/d32/f5c zero size 2024-02-28T16:31:05.769 INFO:tasks.workunit.client.0.smithi001.stdout:1/495: creat d0/fa9 x:0 0 0 2024-02-28T16:31:05.770 INFO:tasks.workunit.client.0.smithi001.stdout:7/344: mknod d3/d17/d3d/d59/d6e/c6f 0 2024-02-28T16:31:05.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:05 smithi001 ceph-mon[28991]: pgmap v46: 65 pgs: 65 active+clean; 1.0 GiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 20 MiB/s rd, 117 MiB/s wr, 328 op/s 2024-02-28T16:31:05.776 INFO:tasks.workunit.client.0.smithi001.stdout:0/352: symlink d18/d22/d3d/l6e 0 2024-02-28T16:31:05.779 INFO:tasks.workunit.client.0.smithi001.stdout:8/353: mkdir d4/d32/d69 0 2024-02-28T16:31:05.781 INFO:tasks.workunit.client.0.smithi001.stdout:1/496: symlink d0/d9/laa 0 2024-02-28T16:31:05.782 INFO:tasks.workunit.client.0.smithi001.stdout:0/353: truncate d18/d22/d3d/f5e 762494 0 2024-02-28T16:31:05.787 INFO:tasks.workunit.client.0.smithi001.stdout:0/354: chown d18/d22/d2b/d36/d46 104 1 2024-02-28T16:31:05.788 INFO:tasks.workunit.client.0.smithi001.stdout:8/354: creat d4/d7/f6a x:0 0 0 2024-02-28T16:31:05.790 INFO:tasks.workunit.client.0.smithi001.stdout:1/497: mknod d0/d9/d1a/d72/d9d/d9c/cab 0 2024-02-28T16:31:05.792 INFO:tasks.workunit.client.0.smithi001.stdout:0/355: symlink d18/d5a/l6f 0 2024-02-28T16:31:05.793 INFO:tasks.workunit.client.0.smithi001.stdout:5/347: dread dc/d1a/d21/d2d/d3c/f53 [0,4194304] 0 2024-02-28T16:31:05.794 INFO:tasks.workunit.client.0.smithi001.stdout:6/324: write d9/db/d5f/d62/f6b [586465,40705] 0 2024-02-28T16:31:05.800 INFO:tasks.workunit.client.0.smithi001.stdout:2/368: dwrite d0/d2/d3/de/f5e [0,4194304] 0 2024-02-28T16:31:05.805 INFO:tasks.workunit.client.0.smithi001.stdout:1/498: symlink d0/d9/d1a/d2b/d26/d27/d30/lac 0 2024-02-28T16:31:05.805 INFO:tasks.workunit.client.0.smithi001.stdout:7/345: dwrite d3/d1c/d41/d49/f65 [0,4194304] 0 2024-02-28T16:31:05.805 INFO:tasks.workunit.client.0.smithi001.stdout:6/325: dread d9/db/d10/f11 [0,4194304] 0 2024-02-28T16:31:05.806 INFO:tasks.workunit.client.0.smithi001.stdout:8/355: rename d4/d7/d22/d27/d5f to d4/d7/d22/d27/d4b/d6b 0 2024-02-28T16:31:05.808 INFO:tasks.workunit.client.0.smithi001.stdout:8/356: chown d4/d7/f35 1920 1 2024-02-28T16:31:05.811 INFO:tasks.workunit.client.0.smithi001.stdout:2/369: creat d0/d2/d3/de/d61/d12/d6a/f7e x:0 0 0 2024-02-28T16:31:05.816 INFO:tasks.workunit.client.0.smithi001.stdout:5/348: rmdir dc/d12/d56/d6b 0 2024-02-28T16:31:05.816 INFO:tasks.workunit.client.0.smithi001.stdout:7/346: creat d3/d17/d3d/f70 x:0 0 0 2024-02-28T16:31:05.818 INFO:tasks.workunit.client.0.smithi001.stdout:1/499: mknod d0/d9/d1a/d2b/d26/d27/d30/d52/cad 0 2024-02-28T16:31:05.821 INFO:tasks.workunit.client.0.smithi001.stdout:2/370: write d0/d2/d3/d2c/f6e [2624,75159] 0 2024-02-28T16:31:05.821 INFO:tasks.workunit.client.0.smithi001.stdout:5/349: readlink dc/d12/l40 0 2024-02-28T16:31:05.821 INFO:tasks.workunit.client.0.smithi001.stdout:8/357: mknod d4/c6c 0 2024-02-28T16:31:05.824 INFO:tasks.workunit.client.0.smithi001.stdout:7/347: creat d3/d17/d61/f71 x:0 0 0 2024-02-28T16:31:05.825 INFO:tasks.workunit.client.0.smithi001.stdout:5/350: symlink dc/df/l73 0 2024-02-28T16:31:05.825 INFO:tasks.workunit.client.0.smithi001.stdout:7/348: stat d3/d17/c3a 0 2024-02-28T16:31:05.826 INFO:tasks.workunit.client.0.smithi001.stdout:8/358: readlink d4/l9 0 2024-02-28T16:31:05.831 INFO:tasks.workunit.client.0.smithi001.stdout:2/371: creat d0/f7f x:0 0 0 2024-02-28T16:31:05.834 INFO:tasks.workunit.client.0.smithi001.stdout:2/372: chown d0/d2/d3/d2c/l13 1385258 1 2024-02-28T16:31:05.835 INFO:tasks.workunit.client.0.smithi001.stdout:7/349: link d3/d17/d3d/d59/f67 d3/d17/d3d/f72 0 2024-02-28T16:31:05.836 INFO:tasks.workunit.client.0.smithi001.stdout:8/359: creat d4/d7/d22/d27/d4b/f6d x:0 0 0 2024-02-28T16:31:05.839 INFO:tasks.workunit.client.0.smithi001.stdout:6/326: dwrite d9/d29/d2b/f33 [4194304,4194304] 0 2024-02-28T16:31:05.839 INFO:tasks.workunit.client.0.smithi001.stdout:5/351: dread dc/df/f3a [0,4194304] 0 2024-02-28T16:31:05.844 INFO:tasks.workunit.client.0.smithi001.stdout:5/352: chown dc/d1a/d21/l4e 88679413 1 2024-02-28T16:31:05.844 INFO:tasks.workunit.client.0.smithi001.stdout:7/350: rmdir d3/d1c 39 2024-02-28T16:31:05.845 INFO:tasks.workunit.client.0.smithi001.stdout:8/360: mkdir d4/d6e 0 2024-02-28T16:31:05.848 INFO:tasks.workunit.client.0.smithi001.stdout:6/327: creat d9/d29/d3e/f6d x:0 0 0 2024-02-28T16:31:05.851 INFO:tasks.workunit.client.0.smithi001.stdout:0/356: dread d18/f59 [0,4194304] 0 2024-02-28T16:31:05.856 INFO:tasks.workunit.client.0.smithi001.stdout:1/500: dwrite d0/d9/d1a/d2b/d26/d27/d30/d52/d84/f94 [0,4194304] 0 2024-02-28T16:31:05.858 INFO:tasks.workunit.client.0.smithi001.stdout:7/351: creat d3/d17/d3d/d5d/f73 x:0 0 0 2024-02-28T16:31:05.864 INFO:tasks.workunit.client.0.smithi001.stdout:2/373: dwrite d0/d2/d3/de/d61/d12/f27 [0,4194304] 0 2024-02-28T16:31:05.870 INFO:tasks.workunit.client.0.smithi001.stdout:2/374: mkdir d0/d2/d3/de/d22/d80 0 2024-02-28T16:31:05.871 INFO:tasks.workunit.client.0.smithi001.stdout:5/353: dwrite dc/d1a/d21/d2d/d3c/f53 [0,4194304] 0 2024-02-28T16:31:05.871 INFO:tasks.workunit.client.0.smithi001.stdout:7/352: write d3/d1c/d41/f6d [4181270,69160] 0 2024-02-28T16:31:05.878 INFO:tasks.workunit.client.0.smithi001.stdout:0/357: getdents d18/d22/d47 0 2024-02-28T16:31:05.879 INFO:tasks.workunit.client.0.smithi001.stdout:8/361: dread d4/d7/d22/d27/f4f [0,4194304] 0 2024-02-28T16:31:05.887 INFO:tasks.workunit.client.0.smithi001.stdout:2/375: truncate d0/d2/d3/de/d22/f24 2691117 0 2024-02-28T16:31:05.899 INFO:tasks.workunit.client.0.smithi001.stdout:1/501: dread d0/d9/f54 [0,4194304] 0 2024-02-28T16:31:05.899 INFO:tasks.workunit.client.0.smithi001.stdout:6/328: write d9/db/d10/f11 [1185976,92138] 0 2024-02-28T16:31:05.905 INFO:tasks.workunit.client.0.smithi001.stdout:2/376: getdents d0/d2/d3/de/d61/d12/d6a 0 2024-02-28T16:31:05.913 INFO:tasks.workunit.client.0.smithi001.stdout:8/362: dwrite d4/d7/d22/d27/f3a [4194304,4194304] 0 2024-02-28T16:31:05.914 INFO:tasks.workunit.client.0.smithi001.stdout:7/353: dwrite d3/fa [4194304,4194304] 0 2024-02-28T16:31:05.919 INFO:tasks.workunit.client.0.smithi001.stdout:6/329: rename d9/db/d10/d15 to d9/d29/d6e 0 2024-02-28T16:31:05.920 INFO:tasks.workunit.client.0.smithi001.stdout:1/502: mkdir d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/dae 0 2024-02-28T16:31:05.921 INFO:tasks.workunit.client.0.smithi001.stdout:6/330: chown d9/f4f 4565 1 2024-02-28T16:31:05.922 INFO:tasks.workunit.client.0.smithi001.stdout:8/363: stat d4/l8 0 2024-02-28T16:31:05.926 INFO:tasks.workunit.client.0.smithi001.stdout:7/354: symlink d3/d1c/d41/d49/d63/l74 0 2024-02-28T16:31:05.927 INFO:tasks.workunit.client.0.smithi001.stdout:2/377: link d0/d2/d3/de/d61/d12/d46/f4a d0/d2/d3/de/d61/d12/d6a/f81 0 2024-02-28T16:31:05.928 INFO:tasks.workunit.client.0.smithi001.stdout:0/358: dread d18/d31/d4f/f60 [0,4194304] 0 2024-02-28T16:31:05.930 INFO:tasks.workunit.client.0.smithi001.stdout:5/354: dread dc/d1a/d52/f72 [0,4194304] 0 2024-02-28T16:31:05.932 INFO:tasks.workunit.client.0.smithi001.stdout:1/503: write d0/d9/d1a/d2b/d26/d27/d30/d52/d84/f94 [2268405,79297] 0 2024-02-28T16:31:05.936 INFO:tasks.workunit.client.0.smithi001.stdout:7/355: truncate d3/d1c/d41/d49/f6c 406385 0 2024-02-28T16:31:05.937 INFO:tasks.workunit.client.0.smithi001.stdout:8/364: creat d4/d32/d69/f6f x:0 0 0 2024-02-28T16:31:05.938 INFO:tasks.workunit.client.0.smithi001.stdout:8/365: chown d4/d32 52368668 1 2024-02-28T16:31:05.940 INFO:tasks.workunit.client.0.smithi001.stdout:0/359: write d18/d22/f3f [4195083,112402] 0 2024-02-28T16:31:05.940 INFO:tasks.workunit.client.0.smithi001.stdout:6/331: rename d9/c27 to d9/de/c6f 0 2024-02-28T16:31:05.944 INFO:tasks.workunit.client.0.smithi001.stdout:6/332: chown d9/db/f13 0 1 2024-02-28T16:31:05.945 INFO:tasks.workunit.client.0.smithi001.stdout:7/356: read - d3/d17/d3d/d42/f53 zero size 2024-02-28T16:31:05.947 INFO:tasks.workunit.client.0.smithi001.stdout:1/504: mknod d0/d9/d1a/caf 0 2024-02-28T16:31:05.954 INFO:tasks.workunit.client.0.smithi001.stdout:2/378: rename d0/d2/d3/de/d61/d12/d55/l58 to d0/d2/d3/d36/l82 0 2024-02-28T16:31:05.955 INFO:tasks.workunit.client.0.smithi001.stdout:6/333: mknod d9/db/d19/c70 0 2024-02-28T16:31:05.959 INFO:tasks.workunit.client.0.smithi001.stdout:2/379: read d0/d2/d3/de/d61/f1f [1017745,59071] 0 2024-02-28T16:31:05.960 INFO:tasks.workunit.client.0.smithi001.stdout:8/366: getdents d4/d7/d22/d27/d58/d62 0 2024-02-28T16:31:05.962 INFO:tasks.workunit.client.0.smithi001.stdout:7/357: dread d3/d1c/d41/d49/f65 [0,4194304] 0 2024-02-28T16:31:05.965 INFO:tasks.workunit.client.0.smithi001.stdout:8/367: dread - d4/d7/d22/f3f zero size 2024-02-28T16:31:05.972 INFO:tasks.workunit.client.0.smithi001.stdout:6/334: creat d9/db/d5f/d62/f71 x:0 0 0 2024-02-28T16:31:05.972 INFO:tasks.workunit.client.0.smithi001.stdout:0/360: getdents d18/d22 0 2024-02-28T16:31:05.972 INFO:tasks.workunit.client.0.smithi001.stdout:2/380: unlink d0/d2/d3/d36/l82 0 2024-02-28T16:31:05.973 INFO:tasks.workunit.client.0.smithi001.stdout:1/505: dwrite d0/d9/d1a/d2b/f17 [0,4194304] 0 2024-02-28T16:31:05.973 INFO:tasks.workunit.client.0.smithi001.stdout:8/368: symlink d4/d7/d22/d27/d58/l70 0 2024-02-28T16:31:05.976 INFO:tasks.workunit.client.0.smithi001.stdout:5/355: dread dc/d1a/d21/d2d/f42 [0,4194304] 0 2024-02-28T16:31:05.977 INFO:tasks.workunit.client.0.smithi001.stdout:0/361: chown d18/d22/d2b/f37 0 1 2024-02-28T16:31:05.978 INFO:tasks.workunit.client.0.smithi001.stdout:2/381: write d0/d2/d3/d36/f68 [507206,51047] 0 2024-02-28T16:31:05.978 INFO:tasks.workunit.client.0.smithi001.stdout:1/506: readlink d0/l18 0 2024-02-28T16:31:05.979 INFO:tasks.workunit.client.0.smithi001.stdout:6/335: creat d9/d29/d3e/f72 x:0 0 0 2024-02-28T16:31:05.982 INFO:tasks.workunit.client.0.smithi001.stdout:5/356: write dc/f30 [1834915,96351] 0 2024-02-28T16:31:05.986 INFO:tasks.workunit.client.0.smithi001.stdout:8/369: creat d4/d7/d22/d27/d58/f71 x:0 0 0 2024-02-28T16:31:05.988 INFO:tasks.workunit.client.0.smithi001.stdout:5/357: fdatasync dc/d1a/d21/d2d/d3c/f49 0 2024-02-28T16:31:05.988 INFO:tasks.workunit.client.0.smithi001.stdout:7/358: dwrite d3/ff [8388608,4194304] 0 2024-02-28T16:31:05.990 INFO:tasks.workunit.client.0.smithi001.stdout:0/362: mkdir d18/d22/d2f/d1c/d70 0 2024-02-28T16:31:05.993 INFO:tasks.workunit.client.0.smithi001.stdout:2/382: rmdir d0/d2/d3/de/d61/d12/d6a 39 2024-02-28T16:31:05.995 INFO:tasks.workunit.client.0.smithi001.stdout:8/370: creat d4/d32/d69/f72 x:0 0 0 2024-02-28T16:31:05.996 INFO:tasks.workunit.client.0.smithi001.stdout:7/359: truncate d3/d17/d3d/d42/f53 661796 0 2024-02-28T16:31:05.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:05 smithi066 ceph-mon[36236]: pgmap v46: 65 pgs: 65 active+clean; 1.0 GiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 20 MiB/s rd, 117 MiB/s wr, 328 op/s 2024-02-28T16:31:05.998 INFO:tasks.workunit.client.0.smithi001.stdout:2/383: write d0/d2/d3/f43 [930922,42283] 0 2024-02-28T16:31:06.002 INFO:tasks.workunit.client.0.smithi001.stdout:0/363: rename d18/f54 to d18/d31/f71 0 2024-02-28T16:31:06.003 INFO:tasks.workunit.client.0.smithi001.stdout:1/507: unlink d0/d9/d1a/d72/l90 0 2024-02-28T16:31:06.003 INFO:tasks.workunit.client.0.smithi001.stdout:9/468: sync 2024-02-28T16:31:06.004 INFO:tasks.workunit.client.0.smithi001.stdout:3/353: sync 2024-02-28T16:31:06.004 INFO:tasks.workunit.client.0.smithi001.stdout:4/333: sync 2024-02-28T16:31:06.007 INFO:tasks.workunit.client.0.smithi001.stdout:5/358: fdatasync dc/d1a/d21/d2d/f42 0 2024-02-28T16:31:06.007 INFO:tasks.workunit.client.0.smithi001.stdout:8/371: fdatasync d4/d7/f21 0 2024-02-28T16:31:06.010 INFO:tasks.workunit.client.0.smithi001.stdout:4/334: write d3/d1c/d42/f53 [104227,27906] 0 2024-02-28T16:31:06.014 INFO:tasks.workunit.client.0.smithi001.stdout:0/364: creat d18/d5a/f72 x:0 0 0 2024-02-28T16:31:06.016 INFO:tasks.workunit.client.0.smithi001.stdout:4/335: write d3/f12 [1045510,73004] 0 2024-02-28T16:31:06.021 INFO:tasks.workunit.client.0.smithi001.stdout:7/360: rename d3/l64 to d3/d17/l75 0 2024-02-28T16:31:06.024 INFO:tasks.workunit.client.0.smithi001.stdout:0/365: write d18/d22/d3d/f50 [1864314,70647] 0 2024-02-28T16:31:06.024 INFO:tasks.workunit.client.0.smithi001.stdout:1/508: mknod d0/d9/d1a/d2b/d26/d27/d30/cb0 0 2024-02-28T16:31:06.025 INFO:tasks.workunit.client.0.smithi001.stdout:4/336: fdatasync d3/d5/d7/d1f/d2e/f59 0 2024-02-28T16:31:06.030 INFO:tasks.workunit.client.0.smithi001.stdout:0/366: fsync d18/d31/f6c 0 2024-02-28T16:31:06.032 INFO:tasks.workunit.client.0.smithi001.stdout:6/336: truncate d9/d29/d2b/f30 2469961 0 2024-02-28T16:31:06.032 INFO:tasks.workunit.client.0.smithi001.stdout:9/469: creat d5/d47/fa3 x:0 0 0 2024-02-28T16:31:06.033 INFO:tasks.workunit.client.0.smithi001.stdout:7/361: write d3/d17/d3d/f48 [614399,121603] 0 2024-02-28T16:31:06.042 INFO:tasks.workunit.client.0.smithi001.stdout:4/337: read d3/d5/d7/d1f/d20/f31 [111980,36349] 0 2024-02-28T16:31:06.045 INFO:tasks.workunit.client.0.smithi001.stdout:1/509: truncate d0/d9/d1a/d2b/d26/d27/d30/d52/f5a 587170 0 2024-02-28T16:31:06.045 INFO:tasks.workunit.client.0.smithi001.stdout:5/359: dwrite dc/f20 [0,4194304] 0 2024-02-28T16:31:06.050 INFO:tasks.workunit.client.0.smithi001.stdout:6/337: creat d9/d29/d6a/f73 x:0 0 0 2024-02-28T16:31:06.054 INFO:tasks.workunit.client.0.smithi001.stdout:9/470: creat d5/d47/fa4 x:0 0 0 2024-02-28T16:31:06.054 INFO:tasks.workunit.client.0.smithi001.stdout:1/510: truncate d0/d9/f85 524047 0 2024-02-28T16:31:06.063 INFO:tasks.workunit.client.0.smithi001.stdout:2/384: dread d0/d2/d3/d2c/f10 [0,4194304] 0 2024-02-28T16:31:06.066 INFO:tasks.workunit.client.0.smithi001.stdout:9/471: symlink d5/d27/d34/d50/la5 0 2024-02-28T16:31:06.067 INFO:tasks.workunit.client.0.smithi001.stdout:1/511: read d0/f36 [790538,40176] 0 2024-02-28T16:31:06.068 INFO:tasks.workunit.client.0.smithi001.stdout:0/367: dwrite d18/d22/d3d/f5e [0,4194304] 0 2024-02-28T16:31:06.073 INFO:tasks.workunit.client.0.smithi001.stdout:9/472: dread - d5/f19 zero size 2024-02-28T16:31:06.074 INFO:tasks.workunit.client.0.smithi001.stdout:8/372: dread d4/d7/f35 [0,4194304] 0 2024-02-28T16:31:06.074 INFO:tasks.workunit.client.0.smithi001.stdout:4/338: truncate d3/d1c/f4f 748632 0 2024-02-28T16:31:06.077 INFO:tasks.workunit.client.0.smithi001.stdout:7/362: dwrite d3/d17/f2d [0,4194304] 0 2024-02-28T16:31:06.078 INFO:tasks.workunit.client.0.smithi001.stdout:6/338: rename d9/l63 to d9/d29/d3e/l74 0 2024-02-28T16:31:06.081 INFO:tasks.workunit.client.0.smithi001.stdout:6/339: write f8 [3644269,6314] 0 2024-02-28T16:31:06.082 INFO:tasks.workunit.client.0.smithi001.stdout:0/368: creat d18/d5a/f73 x:0 0 0 2024-02-28T16:31:06.086 INFO:tasks.workunit.client.0.smithi001.stdout:6/340: chown d9/db/d19 3 1 2024-02-28T16:31:06.086 INFO:tasks.workunit.client.0.smithi001.stdout:0/369: readlink d18/d22/d2f/l2d 0 2024-02-28T16:31:06.091 INFO:tasks.workunit.client.0.smithi001.stdout:1/512: mkdir d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/dae/db1 0 2024-02-28T16:31:06.094 INFO:tasks.workunit.client.0.smithi001.stdout:2/385: creat d0/f83 x:0 0 0 2024-02-28T16:31:06.102 INFO:tasks.workunit.client.0.smithi001.stdout:1/513: creat d0/d9/d1a/d72/fb2 x:0 0 0 2024-02-28T16:31:06.102 INFO:tasks.workunit.client.0.smithi001.stdout:7/363: getdents d3/d1c/d41 0 2024-02-28T16:31:06.103 INFO:tasks.workunit.client.0.smithi001.stdout:5/360: dwrite dc/d1a/d21/f4b [0,4194304] 0 2024-02-28T16:31:06.104 INFO:tasks.workunit.client.0.smithi001.stdout:1/514: readlink d0/l41 0 2024-02-28T16:31:06.106 INFO:tasks.workunit.client.0.smithi001.stdout:4/339: dread d3/d2d/f45 [0,4194304] 0 2024-02-28T16:31:06.108 INFO:tasks.workunit.client.0.smithi001.stdout:9/473: dread d5/f15 [0,4194304] 0 2024-02-28T16:31:06.115 INFO:tasks.workunit.client.0.smithi001.stdout:8/373: dwrite d4/d7/f20 [0,4194304] 0 2024-02-28T16:31:06.118 INFO:tasks.workunit.client.0.smithi001.stdout:5/361: mknod dc/d12/d56/c74 0 2024-02-28T16:31:06.126 INFO:tasks.workunit.client.0.smithi001.stdout:2/386: dwrite d0/f40 [0,4194304] 0 2024-02-28T16:31:06.130 INFO:tasks.workunit.client.0.smithi001.stdout:6/341: write d9/d29/d2b/f30 [2631313,74813] 0 2024-02-28T16:31:06.130 INFO:tasks.workunit.client.0.smithi001.stdout:8/374: creat d4/d7/d22/d27/f73 x:0 0 0 2024-02-28T16:31:06.133 INFO:tasks.workunit.client.0.smithi001.stdout:1/515: creat d0/d9/d1a/d2b/d26/d27/d30/d88/fb3 x:0 0 0 2024-02-28T16:31:06.136 INFO:tasks.workunit.client.0.smithi001.stdout:5/362: creat dc/d1a/d21/d2d/d3c/d4a/f75 x:0 0 0 2024-02-28T16:31:06.143 INFO:tasks.workunit.client.0.smithi001.stdout:9/474: dwrite d5/d27/d34/f90 [0,4194304] 0 2024-02-28T16:31:06.149 INFO:tasks.workunit.client.0.smithi001.stdout:7/364: write d3/d17/d3d/f72 [693485,103073] 0 2024-02-28T16:31:06.149 INFO:tasks.workunit.client.0.smithi001.stdout:6/342: rename d9/f59 to d9/db/d5f/d69/f75 0 2024-02-28T16:31:06.149 INFO:tasks.workunit.client.0.smithi001.stdout:5/363: write dc/d12/f32 [1418513,55190] 0 2024-02-28T16:31:06.157 INFO:tasks.workunit.client.0.smithi001.stdout:0/370: dread d18/d22/d3d/f50 [0,4194304] 0 2024-02-28T16:31:06.157 INFO:tasks.workunit.client.0.smithi001.stdout:8/375: mknod d4/d6e/c74 0 2024-02-28T16:31:06.159 INFO:tasks.workunit.client.0.smithi001.stdout:5/364: creat dc/d1a/d21/d2d/f76 x:0 0 0 2024-02-28T16:31:06.160 INFO:tasks.workunit.client.0.smithi001.stdout:1/516: symlink d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/dae/db1/lb4 0 2024-02-28T16:31:06.161 INFO:tasks.workunit.client.0.smithi001.stdout:7/365: symlink d3/d17/d3d/d59/l76 0 2024-02-28T16:31:06.161 INFO:tasks.workunit.client.1.smithi066.stderr:++ readlink -f fsstress 2024-02-28T16:31:06.162 INFO:tasks.workunit.client.0.smithi001.stdout:9/475: mkdir d5/d47/da6 0 2024-02-28T16:31:06.165 INFO:tasks.workunit.client.1.smithi066.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-02-28T16:31:06.166 INFO:tasks.workunit.client.1.smithi066.stderr:+ popd 2024-02-28T16:31:06.166 INFO:tasks.workunit.client.0.smithi001.stdout:4/340: dwrite d3/f1b [0,4194304] 0 2024-02-28T16:31:06.167 INFO:tasks.workunit.client.0.smithi001.stdout:2/387: mkdir d0/d2/d3/de/d61/d12/d6a/d84 0 2024-02-28T16:31:06.168 INFO:tasks.workunit.client.0.smithi001.stdout:1/517: truncate d0/d9/d56/f7f 1163900 0 2024-02-28T16:31:06.169 INFO:tasks.workunit.client.1.smithi066.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-28T16:31:06.169 INFO:tasks.workunit.client.1.smithi066.stderr:+ popd 2024-02-28T16:31:06.169 INFO:tasks.workunit.client.0.smithi001.stdout:0/371: write d18/f33 [306721,62322] 0 2024-02-28T16:31:06.172 INFO:tasks.workunit.client.1.smithi066.stdout:~/cephtest/mnt.1/client.1/tmp 2024-02-28T16:31:06.172 INFO:tasks.workunit.client.1.smithi066.stderr:++ mktemp -d -p . 2024-02-28T16:31:06.173 INFO:tasks.workunit.client.0.smithi001.stdout:4/341: write d3/d1c/d58/f46 [2043246,98590] 0 2024-02-28T16:31:06.174 INFO:tasks.workunit.client.0.smithi001.stdout:8/376: creat d4/d7/d22/d27/d58/f75 x:0 0 0 2024-02-28T16:31:06.179 INFO:tasks.workunit.client.0.smithi001.stdout:0/372: symlink d18/d31/l74 0 2024-02-28T16:31:06.180 INFO:tasks.workunit.client.1.smithi066.stderr:+ T=./tmp.IEDwljntxx 2024-02-28T16:31:06.180 INFO:tasks.workunit.client.1.smithi066.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.IEDwljntxx -l 1 -n 1000 -p 10 -v 2024-02-28T16:31:06.181 INFO:tasks.workunit.client.0.smithi001.stdout:1/518: chown d0/d9/d1a/d72/d9d/la6 12637751 1 2024-02-28T16:31:06.183 INFO:tasks.workunit.client.0.smithi001.stdout:3/354: sync 2024-02-28T16:31:06.183 INFO:tasks.workunit.client.0.smithi001.stdout:8/377: read - d4/d7/f6a zero size 2024-02-28T16:31:06.186 INFO:tasks.workunit.client.0.smithi001.stdout:5/365: link dc/d12/d41/f4c dc/d1a/d21/d2d/f77 0 2024-02-28T16:31:06.187 INFO:tasks.workunit.client.0.smithi001.stdout:6/343: dwrite d9/d29/d6e/f40 [0,4194304] 0 2024-02-28T16:31:06.188 INFO:tasks.workunit.client.1.smithi066.stdout:seed = 1708936821 2024-02-28T16:31:06.191 INFO:tasks.workunit.client.0.smithi001.stdout:1/519: readlink d0/lb 0 2024-02-28T16:31:06.192 INFO:tasks.workunit.client.0.smithi001.stdout:3/355: chown d4/d6/c42 0 1 2024-02-28T16:31:06.195 INFO:tasks.workunit.client.1.smithi066.stdout:0/0: fdatasync - no filename 2024-02-28T16:31:06.196 INFO:tasks.workunit.client.0.smithi001.stdout:7/366: dwrite d3/d17/d3d/f72 [0,4194304] 0 2024-02-28T16:31:06.197 INFO:tasks.workunit.client.0.smithi001.stdout:9/476: write d5/d47/d33/f3d [4188119,111896] 0 2024-02-28T16:31:06.198 INFO:tasks.workunit.client.1.smithi066.stdout:1/0: unlink - no file 2024-02-28T16:31:06.198 INFO:tasks.workunit.client.1.smithi066.stdout:1/1: write - no filename 2024-02-28T16:31:06.198 INFO:tasks.workunit.client.1.smithi066.stdout:1/2: dread - no filename 2024-02-28T16:31:06.198 INFO:tasks.workunit.client.1.smithi066.stdout:1/3: truncate - no filename 2024-02-28T16:31:06.198 INFO:tasks.workunit.client.1.smithi066.stdout:1/4: write - no filename 2024-02-28T16:31:06.201 INFO:tasks.workunit.client.0.smithi001.stdout:8/378: rename d4/l31 to d4/d47/l76 0 2024-02-28T16:31:06.201 INFO:tasks.workunit.client.0.smithi001.stdout:0/373: rename d18/d22/d2f/d1c to d18/d22/d2f/d1c/d70/d75 22 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:0/1: creat f0 x:0 0 0 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/0: unlink - no file 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/1: read - no filename 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/2: write - no filename 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/3: dwrite - no filename 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/4: chown . 23 1 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/5: dwrite - no filename 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/6: rename - no filename 2024-02-28T16:31:06.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/7: dwrite - no filename 2024-02-28T16:31:06.203 INFO:tasks.workunit.client.0.smithi001.stdout:2/388: dread d0/d2/d3/de/d61/f15 [0,4194304] 0 2024-02-28T16:31:06.203 INFO:tasks.workunit.client.1.smithi066.stdout:1/5: symlink l0 0 2024-02-28T16:31:06.203 INFO:tasks.workunit.client.1.smithi066.stdout:1/6: dread - no filename 2024-02-28T16:31:06.204 INFO:tasks.workunit.client.0.smithi001.stdout:0/374: chown d18/d22/d2f/d1c/l25 1791991099 1 2024-02-28T16:31:06.204 INFO:tasks.workunit.client.0.smithi001.stdout:5/366: creat dc/d1a/d21/d2d/f78 x:0 0 0 2024-02-28T16:31:06.204 INFO:tasks.workunit.client.1.smithi066.stdout:1/7: chown l0 7 1 2024-02-28T16:31:06.205 INFO:tasks.workunit.client.1.smithi066.stdout:0/2: symlink l1 0 2024-02-28T16:31:06.208 INFO:tasks.workunit.client.1.smithi066.stdout:1/8: symlink l1 0 2024-02-28T16:31:06.208 INFO:tasks.workunit.client.0.smithi001.stdout:3/356: symlink d4/d6/d7/d23/l75 0 2024-02-28T16:31:06.208 INFO:tasks.workunit.client.1.smithi066.stdout:2/8: mkdir d0 0 2024-02-28T16:31:06.209 INFO:tasks.workunit.client.1.smithi066.stdout:2/9: write - no filename 2024-02-28T16:31:06.209 INFO:tasks.workunit.client.1.smithi066.stdout:2/10: read - no filename 2024-02-28T16:31:06.211 INFO:tasks.workunit.client.0.smithi001.stdout:6/344: mknod d9/d29/c76 0 2024-02-28T16:31:06.212 INFO:tasks.workunit.client.0.smithi001.stdout:2/389: write d0/d2/d3/d2c/f70 [517965,15929] 0 2024-02-28T16:31:06.217 INFO:tasks.workunit.client.1.smithi066.stdout:1/9: mkdir d2 0 2024-02-28T16:31:06.217 INFO:tasks.workunit.client.1.smithi066.stdout:0/3: truncate f0 174349 0 2024-02-28T16:31:06.217 INFO:tasks.workunit.client.1.smithi066.stdout:0/4: chown f0 3921570 1 2024-02-28T16:31:06.217 INFO:tasks.workunit.client.1.smithi066.stdout:0/5: fdatasync f0 0 2024-02-28T16:31:06.219 INFO:tasks.workunit.client.1.smithi066.stdout:2/11: mkdir d0/d1 0 2024-02-28T16:31:06.220 INFO:tasks.workunit.client.1.smithi066.stdout:0/6: dread f0 [0,4194304] 0 2024-02-28T16:31:06.221 INFO:tasks.workunit.client.1.smithi066.stdout:2/12: chown d0 1549 1 2024-02-28T16:31:06.222 INFO:tasks.workunit.client.1.smithi066.stdout:0/7: readlink l1 0 2024-02-28T16:31:06.223 INFO:tasks.workunit.client.1.smithi066.stdout:4/0: mknod c0 0 2024-02-28T16:31:06.223 INFO:tasks.workunit.client.0.smithi001.stdout:6/345: chown d9/db/d19/l22 497 1 2024-02-28T16:31:06.224 INFO:tasks.workunit.client.1.smithi066.stdout:5/0: read - no filename 2024-02-28T16:31:06.227 INFO:tasks.workunit.client.1.smithi066.stdout:0/8: write f0 [997779,1897] 0 2024-02-28T16:31:06.228 INFO:tasks.workunit.client.1.smithi066.stdout:1/10: creat d2/f3 x:0 0 0 2024-02-28T16:31:06.228 INFO:tasks.workunit.client.1.smithi066.stdout:4/1: mkdir d1 0 2024-02-28T16:31:06.228 INFO:tasks.workunit.client.1.smithi066.stdout:2/13: rename d0/d1 to d0/d1/d2 22 2024-02-28T16:31:06.228 INFO:tasks.workunit.client.0.smithi001.stdout:6/346: read - d9/de/f66 zero size 2024-02-28T16:31:06.230 INFO:tasks.workunit.client.1.smithi066.stdout:5/1: symlink l0 0 2024-02-28T16:31:06.230 INFO:tasks.workunit.client.1.smithi066.stdout:5/2: fsync - no filename 2024-02-28T16:31:06.231 INFO:tasks.workunit.client.1.smithi066.stdout:5/3: dread - no filename 2024-02-28T16:31:06.232 INFO:tasks.workunit.client.1.smithi066.stdout:2/14: chown d0/d1 1 1 2024-02-28T16:31:06.232 INFO:tasks.workunit.client.1.smithi066.stdout:2/15: write - no filename 2024-02-28T16:31:06.232 INFO:tasks.workunit.client.1.smithi066.stdout:2/16: fsync - no filename 2024-02-28T16:31:06.233 INFO:tasks.workunit.client.0.smithi001.stdout:4/342: dread d3/fa [0,4194304] 0 2024-02-28T16:31:06.233 INFO:tasks.workunit.client.1.smithi066.stdout:7/0: rename - no filename 2024-02-28T16:31:06.233 INFO:tasks.workunit.client.0.smithi001.stdout:1/520: truncate d0/d9/d1a/d2b/f24 2371213 0 2024-02-28T16:31:06.234 INFO:tasks.workunit.client.1.smithi066.stdout:1/11: creat d2/f4 x:0 0 0 2024-02-28T16:31:06.234 INFO:tasks.workunit.client.1.smithi066.stdout:5/4: creat f1 x:0 0 0 2024-02-28T16:31:06.235 INFO:tasks.workunit.client.1.smithi066.stdout:5/5: fdatasync f1 0 2024-02-28T16:31:06.236 INFO:tasks.workunit.client.1.smithi066.stdout:8/0: dwrite - no filename 2024-02-28T16:31:06.236 INFO:tasks.workunit.client.1.smithi066.stdout:6/0: mknod c0 0 2024-02-28T16:31:06.237 INFO:tasks.workunit.client.1.smithi066.stdout:5/6: readlink l0 0 2024-02-28T16:31:06.237 INFO:tasks.workunit.client.1.smithi066.stdout:8/1: chown . 160773 1 2024-02-28T16:31:06.237 INFO:tasks.workunit.client.1.smithi066.stdout:8/2: readlink - no filename 2024-02-28T16:31:06.238 INFO:tasks.workunit.client.1.smithi066.stdout:5/7: truncate f1 365803 0 2024-02-28T16:31:06.239 INFO:tasks.workunit.client.1.smithi066.stdout:4/2: creat d1/f2 x:0 0 0 2024-02-28T16:31:06.240 INFO:tasks.workunit.client.1.smithi066.stdout:6/1: chown c0 525916079 1 2024-02-28T16:31:06.240 INFO:tasks.workunit.client.1.smithi066.stdout:5/8: truncate f1 644566 0 2024-02-28T16:31:06.241 INFO:tasks.workunit.client.0.smithi001.stdout:7/367: getdents d3/d1c 0 2024-02-28T16:31:06.241 INFO:tasks.workunit.client.0.smithi001.stdout:8/379: dwrite d4/d7/d22/d27/f60 [4194304,4194304] 0 2024-02-28T16:31:06.242 INFO:tasks.workunit.client.1.smithi066.stdout:6/2: stat c0 0 2024-02-28T16:31:06.242 INFO:tasks.workunit.client.1.smithi066.stdout:6/3: readlink - no filename 2024-02-28T16:31:06.242 INFO:tasks.workunit.client.1.smithi066.stdout:6/4: dwrite - no filename 2024-02-28T16:31:06.242 INFO:tasks.workunit.client.1.smithi066.stdout:6/5: dread - no filename 2024-02-28T16:31:06.242 INFO:tasks.workunit.client.1.smithi066.stdout:5/9: truncate f1 1138270 0 2024-02-28T16:31:06.244 INFO:tasks.workunit.client.1.smithi066.stdout:5/10: read f1 [204668,87463] 0 2024-02-28T16:31:06.244 INFO:tasks.workunit.client.0.smithi001.stdout:0/375: dwrite f13 [0,4194304] 0 2024-02-28T16:31:06.245 INFO:tasks.workunit.client.1.smithi066.stdout:5/11: fdatasync f1 0 2024-02-28T16:31:06.246 INFO:tasks.workunit.client.0.smithi001.stdout:9/477: dwrite d5/f19 [0,4194304] 0 2024-02-28T16:31:06.246 INFO:tasks.workunit.client.0.smithi001.stdout:7/368: chown d3/ff 1243953567 1 2024-02-28T16:31:06.247 INFO:tasks.workunit.client.1.smithi066.stdout:1/12: creat d2/f5 x:0 0 0 2024-02-28T16:31:06.247 INFO:tasks.workunit.client.0.smithi001.stdout:5/367: dwrite dc/d1a/d21/d2d/d3c/f47 [0,4194304] 0 2024-02-28T16:31:06.247 INFO:tasks.workunit.client.0.smithi001.stdout:4/343: read d3/d5/f38 [64730,11545] 0 2024-02-28T16:31:06.249 INFO:tasks.workunit.client.1.smithi066.stdout:6/6: creat f1 x:0 0 0 2024-02-28T16:31:06.251 INFO:tasks.workunit.client.0.smithi001.stdout:9/478: dread - d5/d47/d33/f5d zero size 2024-02-28T16:31:06.252 INFO:tasks.workunit.client.0.smithi001.stdout:3/357: creat d4/f76 x:0 0 0 2024-02-28T16:31:06.252 INFO:tasks.workunit.client.0.smithi001.stdout:1/521: mknod d0/d9/d1a/cb5 0 2024-02-28T16:31:06.255 INFO:tasks.workunit.client.0.smithi001.stdout:0/376: mkdir d18/d22/d76 0 2024-02-28T16:31:06.255 INFO:tasks.workunit.client.1.smithi066.stdout:1/13: write d2/f3 [98377,59619] 0 2024-02-28T16:31:06.255 INFO:tasks.workunit.client.1.smithi066.stdout:8/3: mkdir d0 0 2024-02-28T16:31:06.255 INFO:tasks.workunit.client.1.smithi066.stdout:8/4: dread - no filename 2024-02-28T16:31:06.257 INFO:tasks.workunit.client.1.smithi066.stdout:2/17: rmdir d0/d1 0 2024-02-28T16:31:06.257 INFO:tasks.workunit.client.1.smithi066.stdout:2/18: dwrite - no filename 2024-02-28T16:31:06.257 INFO:tasks.workunit.client.1.smithi066.stdout:8/5: chown d0 21038 1 2024-02-28T16:31:06.258 INFO:tasks.workunit.client.0.smithi001.stdout:0/377: chown f11 3803377 1 2024-02-28T16:31:06.258 INFO:tasks.workunit.client.0.smithi001.stdout:9/479: fdatasync d5/d47/f3a 0 2024-02-28T16:31:06.258 INFO:tasks.workunit.client.0.smithi001.stdout:2/390: dread d0/d2/d3/de/d61/d12/f27 [0,4194304] 0 2024-02-28T16:31:06.259 INFO:tasks.workunit.client.1.smithi066.stdout:1/14: write d2/f5 [151534,104246] 0 2024-02-28T16:31:06.260 INFO:tasks.workunit.client.1.smithi066.stdout:8/6: rename d0 to d0/d1 22 2024-02-28T16:31:06.260 INFO:tasks.workunit.client.0.smithi001.stdout:9/480: chown d5/d47/f91 7 1 2024-02-28T16:31:06.260 INFO:tasks.workunit.client.0.smithi001.stdout:3/358: chown d4/d18/d5f/l36 62 1 2024-02-28T16:31:06.261 INFO:tasks.workunit.client.0.smithi001.stdout:6/347: read d9/db/d10/f46 [2562398,12981] 0 2024-02-28T16:31:06.262 INFO:tasks.workunit.client.0.smithi001.stdout:3/359: readlink d4/d3c/l60 0 2024-02-28T16:31:06.262 INFO:tasks.workunit.client.0.smithi001.stdout:8/380: mkdir d4/d7/d22/d27/d4b/d77 0 2024-02-28T16:31:06.263 INFO:tasks.workunit.client.1.smithi066.stdout:6/7: mkdir d2 0 2024-02-28T16:31:06.265 INFO:tasks.workunit.client.1.smithi066.stdout:9/0: mknod c0 0 2024-02-28T16:31:06.266 INFO:tasks.workunit.client.1.smithi066.stdout:2/19: symlink d0/l3 0 2024-02-28T16:31:06.267 INFO:tasks.workunit.client.1.smithi066.stdout:8/7: rename d0 to d0/d2 22 2024-02-28T16:31:06.267 INFO:tasks.workunit.client.1.smithi066.stdout:8/8: dwrite - no filename 2024-02-28T16:31:06.269 INFO:tasks.workunit.client.0.smithi001.stdout:4/344: symlink d3/d5/d7/d1f/d3a/l71 0 2024-02-28T16:31:06.269 INFO:tasks.workunit.client.0.smithi001.stdout:5/368: mkdir dc/df/d79 0 2024-02-28T16:31:06.273 INFO:tasks.workunit.client.1.smithi066.stdout:9/1: chown c0 32510 1 2024-02-28T16:31:06.273 INFO:tasks.workunit.client.1.smithi066.stdout:9/2: truncate - no filename 2024-02-28T16:31:06.273 INFO:tasks.workunit.client.1.smithi066.stdout:1/15: truncate d2/f4 832291 0 2024-02-28T16:31:06.274 INFO:tasks.workunit.client.0.smithi001.stdout:5/369: chown dc/d1a/c23 54246 1 2024-02-28T16:31:06.277 INFO:tasks.workunit.client.0.smithi001.stdout:5/370: readlink dc/l1c 0 2024-02-28T16:31:06.278 INFO:tasks.workunit.client.1.smithi066.stdout:2/20: unlink d0/l3 0 2024-02-28T16:31:06.278 INFO:tasks.workunit.client.1.smithi066.stdout:2/21: dread - no filename 2024-02-28T16:31:06.278 INFO:tasks.workunit.client.1.smithi066.stdout:2/22: dwrite - no filename 2024-02-28T16:31:06.280 INFO:tasks.workunit.client.1.smithi066.stdout:2/23: chown d0 168 1 2024-02-28T16:31:06.280 INFO:tasks.workunit.client.1.smithi066.stdout:2/24: dread - no filename 2024-02-28T16:31:06.280 INFO:tasks.workunit.client.1.smithi066.stdout:2/25: dwrite - no filename 2024-02-28T16:31:06.280 INFO:tasks.workunit.client.1.smithi066.stdout:2/26: write - no filename 2024-02-28T16:31:06.283 INFO:tasks.workunit.client.1.smithi066.stdout:8/9: creat d0/f3 x:0 0 0 2024-02-28T16:31:06.285 INFO:tasks.workunit.client.1.smithi066.stdout:9/3: creat f1 x:0 0 0 2024-02-28T16:31:06.285 INFO:tasks.workunit.client.0.smithi001.stdout:3/360: creat d4/d18/d5f/f77 x:0 0 0 2024-02-28T16:31:06.288 INFO:tasks.workunit.client.0.smithi001.stdout:8/381: creat d4/d7/d22/d27/d4b/f78 x:0 0 0 2024-02-28T16:31:06.289 INFO:tasks.workunit.client.1.smithi066.stdout:2/27: rename d0 to d0/d4 22 2024-02-28T16:31:06.290 INFO:tasks.workunit.client.0.smithi001.stdout:8/382: fsync d4/f59 0 2024-02-28T16:31:06.290 INFO:tasks.workunit.client.0.smithi001.stdout:7/369: link d3/f45 d3/d17/d52/f77 0 2024-02-28T16:31:06.291 INFO:tasks.workunit.client.0.smithi001.stdout:6/348: dwrite d9/f4f [0,4194304] 0 2024-02-28T16:31:06.292 INFO:tasks.workunit.client.0.smithi001.stdout:5/371: chown dc/d1a/d21/d2d/d3c/f47 244 1 2024-02-28T16:31:06.295 INFO:tasks.workunit.client.0.smithi001.stdout:1/522: dwrite d0/d9/d56/d2f/f7c [0,4194304] 0 2024-02-28T16:31:06.296 INFO:tasks.workunit.client.1.smithi066.stdout:2/28: chown d0 5548 1 2024-02-28T16:31:06.296 INFO:tasks.workunit.client.1.smithi066.stdout:2/29: link - no file 2024-02-28T16:31:06.296 INFO:tasks.workunit.client.1.smithi066.stdout:2/30: truncate - no filename 2024-02-28T16:31:06.296 INFO:tasks.workunit.client.1.smithi066.stdout:2/31: dwrite - no filename 2024-02-28T16:31:06.296 INFO:tasks.workunit.client.1.smithi066.stdout:8/10: dread - d0/f3 zero size 2024-02-28T16:31:06.297 INFO:tasks.workunit.client.0.smithi001.stdout:4/345: symlink d3/d2d/d4c/d5e/l72 0 2024-02-28T16:31:06.297 INFO:tasks.workunit.client.1.smithi066.stdout:9/4: write f1 [836790,15352] 0 2024-02-28T16:31:06.299 INFO:tasks.workunit.client.0.smithi001.stdout:4/346: chown d3/d1c/d58/c54 936904080 1 2024-02-28T16:31:06.301 INFO:tasks.workunit.client.0.smithi001.stdout:1/523: fsync d0/d9/d56/f8f 0 2024-02-28T16:31:06.303 INFO:tasks.workunit.client.1.smithi066.stdout:0/9: dwrite f0 [0,4194304] 0 2024-02-28T16:31:06.303 INFO:tasks.workunit.client.0.smithi001.stdout:3/361: readlink d4/d18/l4f 0 2024-02-28T16:31:06.304 INFO:tasks.workunit.client.0.smithi001.stdout:9/481: dwrite d5/d27/f5c [0,4194304] 0 2024-02-28T16:31:06.306 INFO:tasks.workunit.client.0.smithi001.stdout:1/524: dread - d0/d9/d1a/d72/f8a zero size 2024-02-28T16:31:06.348 INFO:tasks.workunit.client.1.smithi066.stdout:2/32: chown d0 381877 1 2024-02-28T16:31:06.348 INFO:tasks.workunit.client.1.smithi066.stdout:0/10: read f0 [2829750,54197] 0 2024-02-28T16:31:06.348 INFO:tasks.workunit.client.1.smithi066.stdout:4/3: dwrite d1/f2 [0,4194304] 0 2024-02-28T16:31:06.348 INFO:tasks.workunit.client.1.smithi066.stdout:8/11: creat d0/f4 x:0 0 0 2024-02-28T16:31:06.348 INFO:tasks.workunit.client.1.smithi066.stdout:6/8: dwrite f1 [0,4194304] 0 2024-02-28T16:31:06.350 INFO:tasks.workunit.client.0.smithi001.stdout:8/383: dread d4/d7/f23 [0,4194304] 0 2024-02-28T16:31:06.350 INFO:tasks.workunit.client.0.smithi001.stdout:3/362: chown d4/d1e/l62 22842729 1 2024-02-28T16:31:06.350 INFO:tasks.workunit.client.0.smithi001.stdout:7/370: rmdir d3/d17/d52 39 2024-02-28T16:31:06.350 INFO:tasks.workunit.client.0.smithi001.stdout:1/525: stat d0/d9/d1a/d2b/d26/d27/d30/d88 0 2024-02-28T16:31:06.350 INFO:tasks.workunit.client.0.smithi001.stdout:3/363: readlink d4/d6/d7/d23/l54 0 2024-02-28T16:31:06.350 INFO:tasks.workunit.client.0.smithi001.stdout:1/526: stat d0/d9/d1a/cb5 0 2024-02-28T16:31:06.350 INFO:tasks.workunit.client.0.smithi001.stdout:5/372: dwrite dc/d12/f32 [0,4194304] 0 2024-02-28T16:31:06.350 INFO:tasks.workunit.client.0.smithi001.stdout:3/364: mkdir d4/d6/d7/d78 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:8/384: write d4/d7/f35 [850387,50793] 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:2/391: getdents d0 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:3/365: fdatasync d4/d6/d7/f5d 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:9/482: dwrite d5/d47/f7c [0,4194304] 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:0/378: symlink d18/d22/d2b/l77 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:8/385: chown d4/d7/d22/d27/d4b/f6d 4487733 1 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:5/373: truncate fa 420769 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:4/347: read d3/d5/d7/d1f/d20/f31 [1308175,112460] 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:9/483: dread - d5/d47/fa4 zero size 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:8/386: write d4/d7/d22/d27/d4b/f78 [71391,1242] 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:5/374: fsync dc/d1a/d21/d2d/d3c/f4f 0 2024-02-28T16:31:06.351 INFO:tasks.workunit.client.0.smithi001.stdout:7/371: mknod d3/d17/d52/d55/c78 0 2024-02-28T16:31:06.353 INFO:tasks.workunit.client.0.smithi001.stdout:0/379: readlink d18/d22/d2f/d1c/l20 0 2024-02-28T16:31:06.356 INFO:tasks.workunit.client.1.smithi066.stdout:2/33: mknod d0/c5 0 2024-02-28T16:31:06.356 INFO:tasks.workunit.client.1.smithi066.stdout:2/34: truncate - no filename 2024-02-28T16:31:06.356 INFO:tasks.workunit.client.1.smithi066.stdout:1/16: dwrite d2/f5 [0,4194304] 0 2024-02-28T16:31:06.364 INFO:tasks.workunit.client.1.smithi066.stdout:8/12: link d0/f4 d0/f5 0 2024-02-28T16:31:06.365 INFO:tasks.workunit.client.0.smithi001.stdout:9/484: symlink d5/d9a/la7 0 2024-02-28T16:31:06.366 INFO:tasks.workunit.client.1.smithi066.stdout:6/9: getdents d2 0 2024-02-28T16:31:06.366 INFO:tasks.workunit.client.0.smithi001.stdout:4/348: read d3/d1c/f4f [413871,51630] 0 2024-02-28T16:31:06.368 INFO:tasks.workunit.client.0.smithi001.stdout:0/380: chown d18/d22/c28 12134321 1 2024-02-28T16:31:06.368 INFO:tasks.workunit.client.1.smithi066.stdout:6/10: read f1 [2997910,114677] 0 2024-02-28T16:31:06.370 INFO:tasks.workunit.client.1.smithi066.stdout:0/11: dread f0 [0,4194304] 0 2024-02-28T16:31:06.372 INFO:tasks.workunit.client.1.smithi066.stdout:4/4: dread d1/f2 [0,4194304] 0 2024-02-28T16:31:06.372 INFO:tasks.workunit.client.1.smithi066.stdout:1/17: rename l0 to d2/l6 0 2024-02-28T16:31:06.375 INFO:tasks.workunit.client.0.smithi001.stdout:5/375: truncate dc/d1a/d21/d2d/d3c/f3d 3263751 0 2024-02-28T16:31:06.375 INFO:tasks.workunit.client.1.smithi066.stdout:1/18: write d2/f3 [1084253,71978] 0 2024-02-28T16:31:06.377 INFO:tasks.workunit.client.1.smithi066.stdout:8/13: mkdir d0/d6 0 2024-02-28T16:31:06.378 INFO:tasks.workunit.client.0.smithi001.stdout:9/485: fdatasync d5/d45/d6f/f87 0 2024-02-28T16:31:06.379 INFO:tasks.workunit.client.1.smithi066.stdout:8/14: dread - d0/f5 zero size 2024-02-28T16:31:06.379 INFO:tasks.workunit.client.1.smithi066.stdout:1/19: chown d2/f4 221803 1 2024-02-28T16:31:06.380 INFO:tasks.workunit.client.0.smithi001.stdout:0/381: write d18/d22/d3d/f67 [66918,80154] 0 2024-02-28T16:31:06.384 INFO:tasks.workunit.client.0.smithi001.stdout:9/486: truncate d5/f1a 955239 0 2024-02-28T16:31:06.384 INFO:tasks.workunit.client.0.smithi001.stdout:6/349: dwrite d9/db/fd [0,4194304] 0 2024-02-28T16:31:06.386 INFO:tasks.workunit.client.0.smithi001.stdout:9/487: read d5/d27/f5c [2275955,45188] 0 2024-02-28T16:31:06.389 INFO:tasks.workunit.client.1.smithi066.stdout:6/11: creat d2/f3 x:0 0 0 2024-02-28T16:31:06.390 INFO:tasks.workunit.client.0.smithi001.stdout:4/349: rename d3/d5/d7/d1f/c30 to d3/d2d/d4c/c73 0 2024-02-28T16:31:06.392 INFO:tasks.workunit.client.0.smithi001.stdout:5/376: symlink dc/d12/d41/d54/l7a 0 2024-02-28T16:31:06.394 INFO:tasks.workunit.client.0.smithi001.stdout:2/392: dread d0/d2/d3/f5 [0,4194304] 0 2024-02-28T16:31:06.394 INFO:tasks.workunit.client.0.smithi001.stdout:5/377: stat dc/d1a/d52/c55 0 2024-02-28T16:31:06.398 INFO:tasks.workunit.client.0.smithi001.stdout:4/350: mknod d3/d5/d7/c74 0 2024-02-28T16:31:06.405 INFO:tasks.workunit.client.1.smithi066.stdout:9/5: dread f1 [0,4194304] 0 2024-02-28T16:31:06.406 INFO:tasks.workunit.client.1.smithi066.stdout:0/12: fdatasync f0 0 2024-02-28T16:31:06.406 INFO:tasks.workunit.client.0.smithi001.stdout:3/366: dread d4/f17 [0,4194304] 0 2024-02-28T16:31:06.408 INFO:tasks.workunit.client.0.smithi001.stdout:3/367: chown d4/d38 394536 1 2024-02-28T16:31:06.417 INFO:tasks.workunit.client.1.smithi066.stdout:0/13: write f0 [4048030,53639] 0 2024-02-28T16:31:06.417 INFO:tasks.workunit.client.1.smithi066.stdout:1/20: link d2/f4 d2/f7 0 2024-02-28T16:31:06.418 INFO:tasks.workunit.client.0.smithi001.stdout:2/393: rename d0/d2/d3/de/d61/d12/d46/f4a to d0/d2/d3/d73/f85 0 2024-02-28T16:31:06.418 INFO:tasks.workunit.client.0.smithi001.stdout:3/368: write d4/d38/f55 [217345,941] 0 2024-02-28T16:31:06.418 INFO:tasks.workunit.client.1.smithi066.stdout:9/6: dread f1 [0,4194304] 0 2024-02-28T16:31:06.419 INFO:tasks.workunit.client.1.smithi066.stdout:0/14: mkdir d2 0 2024-02-28T16:31:06.420 INFO:tasks.workunit.client.0.smithi001.stdout:8/387: truncate d4/d7/d22/d27/f2f 2079124 0 2024-02-28T16:31:06.421 INFO:tasks.workunit.client.0.smithi001.stdout:7/372: truncate d3/f5f 359997 0 2024-02-28T16:31:06.422 INFO:tasks.workunit.client.0.smithi001.stdout:3/369: chown d4/d6/d7/d23/l54 191472 1 2024-02-28T16:31:06.422 INFO:tasks.workunit.client.0.smithi001.stdout:0/382: dread d18/d22/d2f/d1c/f21 [0,4194304] 0 2024-02-28T16:31:06.422 INFO:tasks.workunit.client.1.smithi066.stdout:1/21: symlink d2/l8 0 2024-02-28T16:31:06.424 INFO:tasks.workunit.client.1.smithi066.stdout:4/5: dwrite d1/f2 [0,4194304] 0 2024-02-28T16:31:06.425 INFO:tasks.workunit.client.1.smithi066.stdout:0/15: write f0 [1828694,106780] 0 2024-02-28T16:31:06.425 INFO:tasks.workunit.client.0.smithi001.stdout:9/488: getdents d5/d4d 0 2024-02-28T16:31:06.426 INFO:tasks.workunit.client.0.smithi001.stdout:3/370: dread - d4/d18/f49 zero size 2024-02-28T16:31:06.428 INFO:tasks.workunit.client.1.smithi066.stdout:9/7: dread f1 [0,4194304] 0 2024-02-28T16:31:06.431 INFO:tasks.workunit.client.0.smithi001.stdout:6/350: dwrite d9/db/d10/f16 [0,4194304] 0 2024-02-28T16:31:06.434 INFO:tasks.workunit.client.0.smithi001.stdout:4/351: getdents d3/d2d/d4c/d5e 0 2024-02-28T16:31:06.434 INFO:tasks.workunit.client.0.smithi001.stdout:5/378: rmdir dc/d12/d41 39 2024-02-28T16:31:06.438 INFO:tasks.workunit.client.1.smithi066.stdout:1/22: dread d2/f4 [0,4194304] 0 2024-02-28T16:31:06.438 INFO:tasks.workunit.client.0.smithi001.stdout:7/373: write d3/d1c/f37 [4445703,16888] 0 2024-02-28T16:31:06.438 INFO:tasks.workunit.client.0.smithi001.stdout:9/489: mkdir d5/d47/d33/d8a/da8 0 2024-02-28T16:31:06.441 INFO:tasks.workunit.client.0.smithi001.stdout:1/527: dwrite d0/d9/f48 [0,4194304] 0 2024-02-28T16:31:06.441 INFO:tasks.workunit.client.1.smithi066.stdout:9/8: write f1 [1108834,15722] 0 2024-02-28T16:31:06.442 INFO:tasks.workunit.client.0.smithi001.stdout:0/383: truncate d18/d5a/f73 153378 0 2024-02-28T16:31:06.442 INFO:tasks.workunit.client.0.smithi001.stdout:7/374: chown d3/d17/d3d/f72 52836 1 2024-02-28T16:31:06.443 INFO:tasks.workunit.client.0.smithi001.stdout:4/352: chown d3/d2d/d4c/f50 10 1 2024-02-28T16:31:06.443 INFO:tasks.workunit.client.1.smithi066.stdout:1/23: chown l1 31911 1 2024-02-28T16:31:06.445 INFO:tasks.workunit.client.0.smithi001.stdout:0/384: dread - d18/d22/d2b/d36/d46/f4b zero size 2024-02-28T16:31:06.446 INFO:tasks.workunit.client.1.smithi066.stdout:9/9: mknod c2 0 2024-02-28T16:31:06.451 INFO:tasks.workunit.client.1.smithi066.stdout:1/24: symlink d2/l9 0 2024-02-28T16:31:06.452 INFO:tasks.workunit.client.0.smithi001.stdout:9/490: read d5/d47/f37 [1741302,88285] 0 2024-02-28T16:31:06.453 INFO:tasks.workunit.client.0.smithi001.stdout:3/371: symlink d4/d38/l79 0 2024-02-28T16:31:06.455 INFO:tasks.workunit.client.1.smithi066.stdout:0/16: dwrite f0 [0,4194304] 0 2024-02-28T16:31:06.456 INFO:tasks.workunit.client.0.smithi001.stdout:6/351: rename d9/d29/d2b/f56 to d9/d29/d6a/f77 0 2024-02-28T16:31:06.461 INFO:tasks.workunit.client.0.smithi001.stdout:2/394: dwrite d0/d2/d3/de/d61/d12/d46/f66 [0,4194304] 0 2024-02-28T16:31:06.463 INFO:tasks.workunit.client.0.smithi001.stdout:3/372: creat d4/d6/d7/d23/f7a x:0 0 0 2024-02-28T16:31:06.464 INFO:tasks.workunit.client.1.smithi066.stdout:0/17: symlink d2/l3 0 2024-02-28T16:31:06.469 INFO:tasks.workunit.client.0.smithi001.stdout:9/491: rename d5/l11 to d5/d9a/la9 0 2024-02-28T16:31:06.474 INFO:tasks.workunit.client.0.smithi001.stdout:4/353: getdents d3/d2d/d4c/d69 0 2024-02-28T16:31:06.478 INFO:tasks.workunit.client.0.smithi001.stdout:3/373: symlink d4/d1e/l7b 0 2024-02-28T16:31:06.479 INFO:tasks.workunit.client.1.smithi066.stdout:0/18: mknod d2/c4 0 2024-02-28T16:31:06.481 INFO:tasks.workunit.client.0.smithi001.stdout:9/492: mknod d5/d47/d5e/caa 0 2024-02-28T16:31:06.487 INFO:tasks.workunit.client.1.smithi066.stdout:1/25: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:06.487 INFO:tasks.workunit.client.0.smithi001.stdout:7/375: write d3/d17/d52/f77 [1046447,81060] 0 2024-02-28T16:31:06.488 INFO:tasks.workunit.client.0.smithi001.stdout:5/379: creat dc/d1a/f7b x:0 0 0 2024-02-28T16:31:06.493 INFO:tasks.workunit.client.0.smithi001.stdout:0/385: dwrite d18/d22/d2f/f24 [4194304,4194304] 0 2024-02-28T16:31:06.493 INFO:tasks.workunit.client.0.smithi001.stdout:9/493: chown d5/d27/d42/d48/c4e 518785375 1 2024-02-28T16:31:06.501 INFO:tasks.workunit.client.0.smithi001.stdout:0/386: stat d18/d22/d2f/d1c/l20 0 2024-02-28T16:31:06.502 INFO:tasks.workunit.client.0.smithi001.stdout:8/388: dwrite d4/d7/d22/d27/f4f [0,4194304] 0 2024-02-28T16:31:06.504 INFO:tasks.workunit.client.0.smithi001.stdout:5/380: mknod dc/df/c7c 0 2024-02-28T16:31:06.506 INFO:tasks.workunit.client.1.smithi066.stdout:0/19: dwrite f0 [0,4194304] 0 2024-02-28T16:31:06.509 INFO:tasks.workunit.client.0.smithi001.stdout:7/376: unlink d3/d17/d3d/d59/c5b 0 2024-02-28T16:31:06.524 INFO:tasks.workunit.client.0.smithi001.stdout:2/395: dwrite d0/d2/d3/de/d61/d12/d6a/f81 [0,4194304] 0 2024-02-28T16:31:06.524 INFO:tasks.workunit.client.0.smithi001.stdout:0/387: symlink d18/d22/d2b/d36/d4d/l78 0 2024-02-28T16:31:06.528 INFO:tasks.workunit.client.0.smithi001.stdout:3/374: dread d4/d1e/f27 [0,4194304] 0 2024-02-28T16:31:06.528 INFO:tasks.workunit.client.0.smithi001.stdout:7/377: mknod d3/d1c/c79 0 2024-02-28T16:31:06.531 INFO:tasks.workunit.client.0.smithi001.stdout:4/354: link d3/d5/d7/d1f/d20/l2b d3/d5/d7/d1f/d2e/l75 0 2024-02-28T16:31:06.532 INFO:tasks.workunit.client.0.smithi001.stdout:6/352: dread d9/db/fc [0,4194304] 0 2024-02-28T16:31:06.534 INFO:tasks.workunit.client.0.smithi001.stdout:2/396: truncate d0/d2/d3/d2c/f51 4565453 0 2024-02-28T16:31:06.538 INFO:tasks.workunit.client.0.smithi001.stdout:8/389: rename d4/d7/d22/d27/f4f to d4/d6e/f79 0 2024-02-28T16:31:06.538 INFO:tasks.workunit.client.0.smithi001.stdout:4/355: chown d3/d1c/d58/c54 388 1 2024-02-28T16:31:06.539 INFO:tasks.workunit.client.0.smithi001.stdout:2/397: chown d0/d2/d3/c42 0 1 2024-02-28T16:31:06.540 INFO:tasks.workunit.client.0.smithi001.stdout:0/388: mkdir d18/d22/d2f/d1c/d79 0 2024-02-28T16:31:06.548 INFO:tasks.workunit.client.0.smithi001.stdout:8/390: read d4/d7/d22/d27/d58/f61 [4000164,122475] 0 2024-02-28T16:31:06.551 INFO:tasks.workunit.client.0.smithi001.stdout:5/381: getdents dc/d1a 0 2024-02-28T16:31:06.552 INFO:tasks.workunit.client.0.smithi001.stdout:4/356: creat d3/d1c/d58/f76 x:0 0 0 2024-02-28T16:31:06.552 INFO:tasks.workunit.client.0.smithi001.stdout:2/398: mknod d0/d2/d3/d73/c86 0 2024-02-28T16:31:06.560 INFO:tasks.workunit.client.0.smithi001.stdout:9/494: truncate d5/f19 510650 0 2024-02-28T16:31:06.562 INFO:tasks.workunit.client.0.smithi001.stdout:5/382: write dc/d1a/f7b [562290,130733] 0 2024-02-28T16:31:06.565 INFO:tasks.workunit.client.0.smithi001.stdout:9/495: fdatasync d5/f3c 0 2024-02-28T16:31:06.570 INFO:tasks.workunit.client.0.smithi001.stdout:8/391: creat d4/d7/d22/d27/d4b/d77/f7a x:0 0 0 2024-02-28T16:31:06.570 INFO:tasks.workunit.client.0.smithi001.stdout:5/383: readlink dc/d1a/d52/l63 0 2024-02-28T16:31:06.579 INFO:tasks.workunit.client.0.smithi001.stdout:9/496: creat d5/d27/d42/d48/fab x:0 0 0 2024-02-28T16:31:06.581 INFO:tasks.workunit.client.0.smithi001.stdout:5/384: mkdir dc/d1a/d7d 0 2024-02-28T16:31:06.587 INFO:tasks.workunit.client.0.smithi001.stdout:0/389: dwrite d18/d31/d4f/f60 [0,4194304] 0 2024-02-28T16:31:06.587 INFO:tasks.workunit.client.0.smithi001.stdout:3/375: dread d4/d6/fc [0,4194304] 0 2024-02-28T16:31:06.591 INFO:tasks.workunit.client.0.smithi001.stdout:3/376: chown d4/d38 123 1 2024-02-28T16:31:06.597 INFO:tasks.workunit.client.0.smithi001.stdout:7/378: dread d3/d1c/f37 [0,4194304] 0 2024-02-28T16:31:06.599 INFO:tasks.workunit.client.0.smithi001.stdout:6/353: dwrite d9/db/f13 [0,4194304] 0 2024-02-28T16:31:06.603 INFO:tasks.workunit.client.0.smithi001.stdout:9/497: dread d5/f79 [0,4194304] 0 2024-02-28T16:31:06.604 INFO:tasks.workunit.client.0.smithi001.stdout:8/392: dwrite d4/d7/f19 [4194304,4194304] 0 2024-02-28T16:31:06.605 INFO:tasks.workunit.client.0.smithi001.stdout:6/354: dread - d9/d29/d3e/f6d zero size 2024-02-28T16:31:06.606 INFO:tasks.workunit.client.0.smithi001.stdout:0/390: mkdir d18/d22/d76/d7a 0 2024-02-28T16:31:06.607 INFO:tasks.workunit.client.0.smithi001.stdout:5/385: stat dc/d1a/l5b 0 2024-02-28T16:31:06.608 INFO:tasks.workunit.client.0.smithi001.stdout:7/379: truncate d3/d17/d61/f71 898171 0 2024-02-28T16:31:06.614 INFO:tasks.workunit.client.0.smithi001.stdout:4/357: dread d3/f3e [0,4194304] 0 2024-02-28T16:31:06.614 INFO:tasks.workunit.client.0.smithi001.stdout:5/386: mknod dc/d12/d56/c7e 0 2024-02-28T16:31:06.620 INFO:tasks.workunit.client.0.smithi001.stdout:2/399: dread d0/d2/d3/de/d61/f67 [0,4194304] 0 2024-02-28T16:31:06.620 INFO:tasks.workunit.client.0.smithi001.stdout:6/355: creat d9/db/d19/f78 x:0 0 0 2024-02-28T16:31:06.621 INFO:tasks.workunit.client.0.smithi001.stdout:7/380: creat d3/d17/d3d/d5d/f7a x:0 0 0 2024-02-28T16:31:06.621 INFO:tasks.workunit.client.0.smithi001.stdout:0/391: mknod d18/d22/d2b/d5f/c7b 0 2024-02-28T16:31:06.623 INFO:tasks.workunit.client.0.smithi001.stdout:2/400: read - d0/f62 zero size 2024-02-28T16:31:06.623 INFO:tasks.workunit.client.0.smithi001.stdout:8/393: link d4/d6e/f79 d4/d7/d22/d27/d58/f7b 0 2024-02-28T16:31:06.623 INFO:tasks.workunit.client.0.smithi001.stdout:0/392: stat d18/d22/d2f/d1c/c3b 0 2024-02-28T16:31:06.624 INFO:tasks.workunit.client.0.smithi001.stdout:4/358: stat d3/d5/d7/l10 0 2024-02-28T16:31:06.628 INFO:tasks.workunit.client.0.smithi001.stdout:5/387: chown dc/d1a/d21/c3e 1 1 2024-02-28T16:31:06.628 INFO:tasks.workunit.client.0.smithi001.stdout:6/356: write d9/db/d5f/f65 [247179,27695] 0 2024-02-28T16:31:06.628 INFO:tasks.workunit.client.0.smithi001.stdout:5/388: chown dc/df/f13 103953237 1 2024-02-28T16:31:06.632 INFO:tasks.workunit.client.0.smithi001.stdout:2/401: write d0/d2/d3/de/d61/f67 [4136903,43396] 0 2024-02-28T16:31:06.632 INFO:tasks.workunit.client.0.smithi001.stdout:4/359: creat d3/d1c/d42/f77 x:0 0 0 2024-02-28T16:31:06.632 INFO:tasks.workunit.client.0.smithi001.stdout:6/357: chown d9/db 7 1 2024-02-28T16:31:06.633 INFO:tasks.workunit.client.0.smithi001.stdout:8/394: getdents d4/d7/d22/d27/d58/d62 0 2024-02-28T16:31:06.634 INFO:tasks.workunit.client.0.smithi001.stdout:7/381: getdents d3/d17/d3d 0 2024-02-28T16:31:06.635 INFO:tasks.workunit.client.0.smithi001.stdout:0/393: rename d18/d22/d2b/d5f/c7b to d18/d22/d3d/c7c 0 2024-02-28T16:31:06.637 INFO:tasks.workunit.client.0.smithi001.stdout:7/382: read d3/d17/d3d/d42/f53 [207009,39437] 0 2024-02-28T16:31:06.642 INFO:tasks.workunit.client.0.smithi001.stdout:4/360: write d3/d5/d7/d1f/d20/f5d [209615,96137] 0 2024-02-28T16:31:06.642 INFO:tasks.workunit.client.0.smithi001.stdout:0/394: write d18/d22/d2b/d36/d4d/f65 [155825,99295] 0 2024-02-28T16:31:06.644 INFO:tasks.workunit.client.0.smithi001.stdout:5/389: mknod dc/df/d79/c7f 0 2024-02-28T16:31:06.647 INFO:tasks.workunit.client.0.smithi001.stdout:0/395: unlink cf 0 2024-02-28T16:31:06.649 INFO:tasks.workunit.client.0.smithi001.stdout:8/395: rename d4/d7/f2a to d4/d32/f7c 0 2024-02-28T16:31:06.652 INFO:tasks.workunit.client.0.smithi001.stdout:4/361: readlink d3/d5/d7/d1f/d2e/l75 0 2024-02-28T16:31:06.655 INFO:tasks.workunit.client.0.smithi001.stdout:0/396: unlink d18/d22/d2f/d1c/c34 0 2024-02-28T16:31:06.657 INFO:tasks.workunit.client.1.smithi066.stdout:5/12: truncate f1 244196 0 2024-02-28T16:31:06.658 INFO:tasks.workunit.client.0.smithi001.stdout:5/390: symlink dc/l80 0 2024-02-28T16:31:06.659 INFO:tasks.workunit.client.0.smithi001.stdout:8/396: mknod d4/d7/d22/d27/d4b/d77/c7d 0 2024-02-28T16:31:06.661 INFO:tasks.workunit.client.0.smithi001.stdout:6/358: rename d9/d29/c35 to d9/d29/d6e/d53/c79 0 2024-02-28T16:31:06.661 INFO:tasks.workunit.client.0.smithi001.stdout:2/402: dwrite d0/f63 [0,4194304] 0 2024-02-28T16:31:06.662 INFO:tasks.workunit.client.0.smithi001.stdout:7/383: getdents d3/d1c/d41/d49/d63 0 2024-02-28T16:31:06.663 INFO:tasks.workunit.client.1.smithi066.stdout:4/6: write d1/f2 [4177676,53479] 0 2024-02-28T16:31:06.664 INFO:tasks.workunit.client.1.smithi066.stdout:4/7: stat d1/f2 0 2024-02-28T16:31:06.666 INFO:tasks.workunit.client.1.smithi066.stdout:4/8: read d1/f2 [1457507,35140] 0 2024-02-28T16:31:06.668 INFO:tasks.workunit.client.1.smithi066.stdout:4/9: write d1/f2 [4460510,17768] 0 2024-02-28T16:31:06.668 INFO:tasks.workunit.client.0.smithi001.stdout:8/397: creat d4/d7/d22/d27/d58/d62/f7e x:0 0 0 2024-02-28T16:31:06.672 INFO:tasks.workunit.client.1.smithi066.stdout:1/26: creat d2/fa x:0 0 0 2024-02-28T16:31:06.673 INFO:tasks.workunit.client.0.smithi001.stdout:6/359: chown d9/d29/d6e/d4e/l64 15 1 2024-02-28T16:31:06.673 INFO:tasks.workunit.client.0.smithi001.stdout:8/398: read d4/d7/f35 [1362334,90168] 0 2024-02-28T16:31:06.673 INFO:tasks.workunit.client.0.smithi001.stdout:7/384: mknod d3/d17/d61/c7b 0 2024-02-28T16:31:06.673 INFO:tasks.workunit.client.0.smithi001.stdout:2/403: truncate d0/d2/fb 4768413 0 2024-02-28T16:31:06.674 INFO:tasks.workunit.client.0.smithi001.stdout:5/391: fsync dc/d1a/f5f 0 2024-02-28T16:31:06.675 INFO:tasks.workunit.client.1.smithi066.stdout:0/20: rmdir d2 39 2024-02-28T16:31:06.681 INFO:tasks.workunit.client.0.smithi001.stdout:6/360: readlink d9/d29/d3e/l74 0 2024-02-28T16:31:06.682 INFO:tasks.workunit.client.0.smithi001.stdout:8/399: symlink d4/d7/d22/l7f 0 2024-02-28T16:31:06.683 INFO:tasks.workunit.client.0.smithi001.stdout:4/362: dwrite d3/d1c/d58/ff [0,4194304] 0 2024-02-28T16:31:06.683 INFO:tasks.workunit.client.1.smithi066.stdout:1/27: rename d2/f3 to d2/fb 0 2024-02-28T16:31:06.685 INFO:tasks.workunit.client.1.smithi066.stdout:6/12: dwrite f1 [4194304,4194304] 0 2024-02-28T16:31:06.685 INFO:tasks.workunit.client.1.smithi066.stdout:8/15: truncate d0/f4 961154 0 2024-02-28T16:31:06.686 INFO:tasks.workunit.client.0.smithi001.stdout:5/392: creat dc/df/f81 x:0 0 0 2024-02-28T16:31:06.686 INFO:tasks.workunit.client.0.smithi001.stdout:2/404: truncate d0/d2/d3/d2c/f71 504672 0 2024-02-28T16:31:06.688 INFO:tasks.workunit.client.1.smithi066.stdout:6/13: write f1 [2630141,10946] 0 2024-02-28T16:31:06.692 INFO:tasks.workunit.client.0.smithi001.stdout:6/361: getdents d9/d29/d6a 0 2024-02-28T16:31:06.693 INFO:tasks.workunit.client.1.smithi066.stdout:0/21: chown d2/c4 12537 1 2024-02-28T16:31:06.694 INFO:tasks.workunit.client.1.smithi066.stdout:5/13: getdents . 0 2024-02-28T16:31:06.695 INFO:tasks.workunit.client.1.smithi066.stdout:0/22: truncate f0 4708517 0 2024-02-28T16:31:06.695 INFO:tasks.workunit.client.1.smithi066.stdout:5/14: readlink l0 0 2024-02-28T16:31:06.696 INFO:tasks.workunit.client.0.smithi001.stdout:5/393: truncate dc/d1a/d21/f48 622277 0 2024-02-28T16:31:06.697 INFO:tasks.workunit.client.0.smithi001.stdout:2/405: truncate d0/d2/d3/d2c/f6e 1078602 0 2024-02-28T16:31:06.698 INFO:tasks.workunit.client.0.smithi001.stdout:4/363: truncate d3/d5/d7/f21 969865 0 2024-02-28T16:31:06.700 INFO:tasks.workunit.client.1.smithi066.stdout:4/10: getdents d1 0 2024-02-28T16:31:06.703 INFO:tasks.workunit.client.1.smithi066.stdout:5/15: unlink l0 0 2024-02-28T16:31:06.704 INFO:tasks.workunit.client.0.smithi001.stdout:4/364: write d3/d1c/f4a [789246,58105] 0 2024-02-28T16:31:06.704 INFO:tasks.workunit.client.1.smithi066.stdout:0/23: creat d2/f5 x:0 0 0 2024-02-28T16:31:06.705 INFO:tasks.workunit.client.0.smithi001.stdout:6/362: link d9/d29/d6a/f73 d9/d29/d6e/d53/f7a 0 2024-02-28T16:31:06.707 INFO:tasks.workunit.client.0.smithi001.stdout:5/394: creat dc/d1a/d21/d2d/d3c/d4a/f82 x:0 0 0 2024-02-28T16:31:06.708 INFO:tasks.workunit.client.0.smithi001.stdout:6/363: dread - d9/f52 zero size 2024-02-28T16:31:06.709 INFO:tasks.workunit.client.1.smithi066.stdout:0/24: mknod d2/c6 0 2024-02-28T16:31:06.711 INFO:tasks.workunit.client.1.smithi066.stdout:0/25: write f0 [3797610,49376] 0 2024-02-28T16:31:06.712 INFO:tasks.workunit.client.0.smithi001.stdout:4/365: fsync d3/d1c/d58/f76 0 2024-02-28T16:31:06.713 INFO:tasks.workunit.client.1.smithi066.stdout:0/26: chown d2/c6 28776969 1 2024-02-28T16:31:06.714 INFO:tasks.workunit.client.1.smithi066.stdout:0/27: fdatasync d2/f5 0 2024-02-28T16:31:06.715 INFO:tasks.workunit.client.0.smithi001.stdout:6/364: fsync d9/db/d10/f11 0 2024-02-28T16:31:06.717 INFO:tasks.workunit.client.0.smithi001.stdout:7/385: dread d3/f45 [0,4194304] 0 2024-02-28T16:31:06.717 INFO:tasks.workunit.client.1.smithi066.stdout:0/28: creat d2/f7 x:0 0 0 2024-02-28T16:31:06.717 INFO:tasks.workunit.client.1.smithi066.stdout:1/28: dread d2/fb [0,4194304] 0 2024-02-28T16:31:06.719 INFO:tasks.workunit.client.1.smithi066.stdout:0/29: write d2/f7 [249240,70832] 0 2024-02-28T16:31:06.720 INFO:tasks.workunit.client.0.smithi001.stdout:7/386: readlink d3/l1a 0 2024-02-28T16:31:06.720 INFO:tasks.workunit.client.1.smithi066.stdout:0/30: chown l1 61160 1 2024-02-28T16:31:06.721 INFO:tasks.workunit.client.1.smithi066.stdout:1/29: write d2/fb [489739,98101] 0 2024-02-28T16:31:06.723 INFO:tasks.workunit.client.0.smithi001.stdout:4/366: creat d3/f78 x:0 0 0 2024-02-28T16:31:06.723 INFO:tasks.workunit.client.1.smithi066.stdout:1/30: truncate d2/f5 4439438 0 2024-02-28T16:31:06.724 INFO:tasks.workunit.client.1.smithi066.stdout:0/31: mknod d2/c8 0 2024-02-28T16:31:06.726 INFO:tasks.workunit.client.0.smithi001.stdout:6/365: write d9/d29/d6a/f77 [778200,59100] 0 2024-02-28T16:31:06.726 INFO:tasks.workunit.client.0.smithi001.stdout:8/400: truncate d4/d7/d22/d27/f60 2963932 0 2024-02-28T16:31:06.727 INFO:tasks.workunit.client.1.smithi066.stdout:1/31: write d2/f5 [3980767,71823] 0 2024-02-28T16:31:06.728 INFO:tasks.workunit.client.1.smithi066.stdout:0/32: mkdir d2/d9 0 2024-02-28T16:31:06.729 INFO:tasks.workunit.client.0.smithi001.stdout:6/366: stat d9/db/d10/f16 0 2024-02-28T16:31:06.731 INFO:tasks.workunit.client.1.smithi066.stdout:1/32: creat d2/fc x:0 0 0 2024-02-28T16:31:06.732 INFO:tasks.workunit.client.0.smithi001.stdout:4/367: unlink d3/d1c/d42/f57 0 2024-02-28T16:31:06.733 INFO:tasks.workunit.client.0.smithi001.stdout:2/406: dwrite d0/d2/d3/de/d61/d12/f41 [0,4194304] 0 2024-02-28T16:31:06.734 INFO:tasks.workunit.client.0.smithi001.stdout:8/401: mkdir d4/d7/d22/d27/d4b/d80 0 2024-02-28T16:31:06.737 INFO:tasks.workunit.client.0.smithi001.stdout:2/407: read d0/f40 [2830781,120671] 0 2024-02-28T16:31:06.739 INFO:tasks.workunit.client.0.smithi001.stdout:5/395: dread dc/d12/f32 [0,4194304] 0 2024-02-28T16:31:06.745 INFO:tasks.workunit.client.0.smithi001.stdout:4/368: fsync d3/d1c/d58/f2a 0 2024-02-28T16:31:06.748 INFO:tasks.workunit.client.0.smithi001.stdout:8/402: rename d4/d7/f40 to d4/d32/d69/f81 0 2024-02-28T16:31:06.749 INFO:tasks.workunit.client.0.smithi001.stdout:6/367: creat d9/db/d10/f7b x:0 0 0 2024-02-28T16:31:06.752 INFO:tasks.workunit.client.1.smithi066.stdout:1/33: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:06.752 INFO:tasks.workunit.client.0.smithi001.stdout:6/368: write f8 [1904945,119303] 0 2024-02-28T16:31:06.754 INFO:tasks.workunit.client.0.smithi001.stdout:2/408: symlink d0/d2/d3/de/d61/d12/d55/l87 0 2024-02-28T16:31:06.758 INFO:tasks.workunit.client.1.smithi066.stdout:0/33: dwrite d2/f7 [0,4194304] 0 2024-02-28T16:31:06.759 INFO:tasks.workunit.client.0.smithi001.stdout:4/369: mkdir d3/d1c/d58/d6c/d79 0 2024-02-28T16:31:06.761 INFO:tasks.workunit.client.0.smithi001.stdout:6/369: fsync d9/db/d10/f46 0 2024-02-28T16:31:06.761 INFO:tasks.workunit.client.0.smithi001.stdout:8/403: symlink d4/d7/l82 0 2024-02-28T16:31:06.762 INFO:tasks.workunit.client.0.smithi001.stdout:7/387: dwrite d3/f5f [0,4194304] 0 2024-02-28T16:31:06.763 INFO:tasks.workunit.client.0.smithi001.stdout:2/409: creat d0/d2/d3/d2c/f88 x:0 0 0 2024-02-28T16:31:06.766 INFO:tasks.workunit.client.0.smithi001.stdout:2/410: readlink d0/d2/d3/l3d 0 2024-02-28T16:31:06.766 INFO:tasks.workunit.client.1.smithi066.stdout:0/34: dread - d2/f5 zero size 2024-02-28T16:31:06.766 INFO:tasks.workunit.client.1.smithi066.stdout:1/34: dread d2/f7 [0,4194304] 0 2024-02-28T16:31:06.768 INFO:tasks.workunit.client.1.smithi066.stdout:0/35: unlink f0 0 2024-02-28T16:31:06.769 INFO:tasks.workunit.client.0.smithi001.stdout:5/396: dwrite dc/d1a/d21/d2d/f78 [0,4194304] 0 2024-02-28T16:31:06.770 INFO:tasks.workunit.client.1.smithi066.stdout:0/36: chown d2/c4 5282 1 2024-02-28T16:31:06.770 INFO:tasks.workunit.client.1.smithi066.stdout:1/35: mknod d2/cd 0 2024-02-28T16:31:06.772 INFO:tasks.workunit.client.1.smithi066.stdout:0/37: dread - d2/f5 zero size 2024-02-28T16:31:06.774 INFO:tasks.workunit.client.0.smithi001.stdout:4/370: creat d3/d1c/d58/d6e/f7a x:0 0 0 2024-02-28T16:31:06.775 INFO:tasks.workunit.client.1.smithi066.stdout:1/36: symlink d2/le 0 2024-02-28T16:31:06.776 INFO:tasks.workunit.client.0.smithi001.stdout:4/371: chown d3/f12 11 1 2024-02-28T16:31:06.778 INFO:tasks.workunit.client.0.smithi001.stdout:7/388: dread d3/d17/d3d/f72 [0,4194304] 0 2024-02-28T16:31:06.779 INFO:tasks.workunit.client.0.smithi001.stdout:5/397: mknod dc/d1a/d52/c83 0 2024-02-28T16:31:06.779 INFO:tasks.workunit.client.1.smithi066.stdout:1/37: rename d2/fa to d2/ff 0 2024-02-28T16:31:06.783 INFO:tasks.workunit.client.0.smithi001.stdout:7/389: chown d3/d17/f54 94210719 1 2024-02-28T16:31:06.786 INFO:tasks.workunit.client.1.smithi066.stdout:1/38: truncate d2/ff 652588 0 2024-02-28T16:31:06.788 INFO:tasks.workunit.client.0.smithi001.stdout:2/411: dread d0/d2/d3/f34 [0,4194304] 0 2024-02-28T16:31:06.789 INFO:tasks.workunit.client.0.smithi001.stdout:7/390: mknod d3/d17/d3d/d59/d6e/c7c 0 2024-02-28T16:31:06.791 INFO:tasks.workunit.client.0.smithi001.stdout:4/372: getdents d3/d5/d7/d1f/d3a/d68 0 2024-02-28T16:31:06.792 INFO:tasks.workunit.client.0.smithi001.stdout:2/412: mknod d0/d2/d3/d36/c89 0 2024-02-28T16:31:06.794 INFO:tasks.workunit.client.0.smithi001.stdout:4/373: write d3/f3b [134399,76793] 0 2024-02-28T16:31:06.794 INFO:tasks.workunit.client.0.smithi001.stdout:7/391: creat d3/d17/d3d/d42/f7d x:0 0 0 2024-02-28T16:31:06.801 INFO:tasks.workunit.client.0.smithi001.stdout:4/374: creat d3/d5/d4e/f7b x:0 0 0 2024-02-28T16:31:06.804 INFO:tasks.workunit.client.0.smithi001.stdout:4/375: fsync d3/d1c/d58/ff 0 2024-02-28T16:31:06.807 INFO:tasks.workunit.client.1.smithi066.stdout:5/16: dread f1 [0,4194304] 0 2024-02-28T16:31:06.810 INFO:tasks.workunit.client.1.smithi066.stdout:5/17: chown f1 65 1 2024-02-28T16:31:06.812 INFO:tasks.workunit.client.0.smithi001.stdout:4/376: rename d3/d1c/f4f to d3/d1c/d42/f7c 0 2024-02-28T16:31:06.814 INFO:tasks.workunit.client.1.smithi066.stdout:5/18: rename f1 to f2 0 2024-02-28T16:31:06.815 INFO:tasks.workunit.client.1.smithi066.stdout:5/19: read f2 [28459,91396] 0 2024-02-28T16:31:06.815 INFO:tasks.workunit.client.1.smithi066.stdout:1/39: dwrite d2/ff [0,4194304] 0 2024-02-28T16:31:06.817 INFO:tasks.workunit.client.0.smithi001.stdout:6/370: dread d9/de/f5a [4194304,4194304] 0 2024-02-28T16:31:06.818 INFO:tasks.workunit.client.1.smithi066.stdout:4/11: truncate d1/f2 1395194 0 2024-02-28T16:31:06.819 INFO:tasks.workunit.client.0.smithi001.stdout:7/392: dwrite d3/d17/f1b [0,4194304] 0 2024-02-28T16:31:06.820 INFO:tasks.workunit.client.1.smithi066.stdout:5/20: mkdir d3 0 2024-02-28T16:31:06.822 INFO:tasks.workunit.client.1.smithi066.stdout:1/40: creat d2/f10 x:0 0 0 2024-02-28T16:31:06.822 INFO:tasks.workunit.client.0.smithi001.stdout:6/371: truncate d9/de/f2c 4932214 0 2024-02-28T16:31:06.823 INFO:tasks.workunit.client.0.smithi001.stdout:2/413: dwrite d0/d2/d3/de/f5e [0,4194304] 0 2024-02-28T16:31:06.828 INFO:tasks.workunit.client.1.smithi066.stdout:4/12: rmdir d1 39 2024-02-28T16:31:06.828 INFO:tasks.workunit.client.0.smithi001.stdout:7/393: getdents d3/d1c/d41/d49/d63 0 2024-02-28T16:31:06.828 INFO:tasks.workunit.client.0.smithi001.stdout:6/372: write d9/db/d19/f4b [9427615,89529] 0 2024-02-28T16:31:06.831 INFO:tasks.workunit.client.0.smithi001.stdout:4/377: truncate d3/d1c/d58/f6b 3495444 0 2024-02-28T16:31:06.831 INFO:tasks.workunit.client.0.smithi001.stdout:6/373: write d9/db/f45 [5004883,39259] 0 2024-02-28T16:31:06.833 INFO:tasks.workunit.client.0.smithi001.stdout:7/394: truncate d3/d17/f2a 829768 0 2024-02-28T16:31:06.833 INFO:tasks.workunit.client.1.smithi066.stdout:1/41: mkdir d2/d11 0 2024-02-28T16:31:06.833 INFO:tasks.workunit.client.0.smithi001.stdout:4/378: read d3/f1b [2239905,73958] 0 2024-02-28T16:31:06.836 INFO:tasks.workunit.client.1.smithi066.stdout:0/38: fsync d2/f7 0 2024-02-28T16:31:06.836 INFO:tasks.workunit.client.1.smithi066.stdout:5/21: mkdir d3/d4 0 2024-02-28T16:31:06.840 INFO:tasks.workunit.client.1.smithi066.stdout:0/39: write d2/f5 [494602,123475] 0 2024-02-28T16:31:06.842 INFO:tasks.workunit.client.0.smithi001.stdout:2/414: rename d0/d2/d3/d2c to d0/d2/d3/de/d8a 0 2024-02-28T16:31:06.843 INFO:tasks.workunit.client.0.smithi001.stdout:2/415: chown d0/d2/d3 0 1 2024-02-28T16:31:06.846 INFO:tasks.workunit.client.1.smithi066.stdout:4/13: symlink d1/l3 0 2024-02-28T16:31:06.850 INFO:tasks.workunit.client.0.smithi001.stdout:2/416: chown d0/d2/d3/de/d61/d12/d6a/c72 12024 1 2024-02-28T16:31:06.850 INFO:tasks.workunit.client.0.smithi001.stdout:6/374: getdents d9/d29/d3e 0 2024-02-28T16:31:06.853 INFO:tasks.workunit.client.1.smithi066.stdout:1/42: link d2/le d2/l12 0 2024-02-28T16:31:06.853 INFO:tasks.workunit.client.1.smithi066.stdout:5/22: creat d3/d4/f5 x:0 0 0 2024-02-28T16:31:06.854 INFO:tasks.workunit.client.1.smithi066.stdout:4/14: creat d1/f4 x:0 0 0 2024-02-28T16:31:06.856 INFO:tasks.workunit.client.0.smithi001.stdout:2/417: symlink d0/d2/d3/de/d61/d12/d46/l8b 0 2024-02-28T16:31:06.856 INFO:tasks.workunit.client.1.smithi066.stdout:1/43: creat d2/f13 x:0 0 0 2024-02-28T16:31:06.857 INFO:tasks.workunit.client.1.smithi066.stdout:1/44: stat d2/d11 0 2024-02-28T16:31:06.858 INFO:tasks.workunit.client.0.smithi001.stdout:2/418: dread - d0/d2/d3/f6c zero size 2024-02-28T16:31:06.863 INFO:tasks.workunit.client.1.smithi066.stdout:0/40: fsync d2/f5 0 2024-02-28T16:31:06.864 INFO:tasks.workunit.client.0.smithi001.stdout:2/419: write d0/d2/d3/de/d8a/f37 [2101842,108994] 0 2024-02-28T16:31:06.868 INFO:tasks.workunit.client.0.smithi001.stdout:1/528: sync 2024-02-28T16:31:06.869 INFO:tasks.workunit.client.0.smithi001.stdout:4/379: dwrite d3/d5/d7/d1f/d20/f31 [0,4194304] 0 2024-02-28T16:31:06.870 INFO:tasks.workunit.client.0.smithi001.stdout:2/420: creat d0/d2/f8c x:0 0 0 2024-02-28T16:31:06.872 INFO:tasks.workunit.client.0.smithi001.stdout:1/529: mkdir d0/d9/d1a/d2b/d26/d27/d30/d88/db6 0 2024-02-28T16:31:06.873 INFO:tasks.workunit.client.0.smithi001.stdout:7/395: dread d3/d1c/f33 [0,4194304] 0 2024-02-28T16:31:06.875 INFO:tasks.workunit.client.0.smithi001.stdout:4/380: write d3/d1c/d58/f76 [168272,582] 0 2024-02-28T16:31:06.878 INFO:tasks.workunit.client.0.smithi001.stdout:7/396: chown d3/d1c/d41/d49/d63/l74 39446 1 2024-02-28T16:31:06.878 INFO:tasks.workunit.client.0.smithi001.stdout:2/421: rename d0/l7c to d0/d2/d3/de/d61/d12/d46/l8d 0 2024-02-28T16:31:06.879 INFO:tasks.workunit.client.1.smithi066.stdout:4/15: getdents d1 0 2024-02-28T16:31:06.880 INFO:tasks.workunit.client.0.smithi001.stdout:1/530: mkdir d0/d9/d1a/d2b/d26/d27/d30/d67/db7 0 2024-02-28T16:31:06.881 INFO:tasks.workunit.client.0.smithi001.stdout:4/381: write d3/d2d/d4c/d5e/f62 [1023036,131046] 0 2024-02-28T16:31:06.883 INFO:tasks.workunit.client.1.smithi066.stdout:1/45: dwrite d2/f13 [0,4194304] 0 2024-02-28T16:31:06.884 INFO:tasks.workunit.client.0.smithi001.stdout:6/375: dread d9/de/f2c [0,4194304] 0 2024-02-28T16:31:06.886 INFO:tasks.workunit.client.0.smithi001.stdout:7/397: rename d3/d17/d3d/d5d/f7a to d3/d17/f7e 0 2024-02-28T16:31:06.887 INFO:tasks.workunit.client.0.smithi001.stdout:6/376: readlink d9/d29/l2e 0 2024-02-28T16:31:06.888 INFO:tasks.workunit.client.1.smithi066.stdout:1/46: read d2/fb [781738,73584] 0 2024-02-28T16:31:06.889 INFO:tasks.workunit.client.0.smithi001.stdout:4/382: creat d3/d5/d7/d1f/f7d x:0 0 0 2024-02-28T16:31:06.889 INFO:tasks.workunit.client.0.smithi001.stdout:6/377: chown d9/d29/d6a/f77 1114 1 2024-02-28T16:31:06.894 INFO:tasks.workunit.client.1.smithi066.stdout:1/47: getdents d2/d11 0 2024-02-28T16:31:06.895 INFO:tasks.workunit.client.1.smithi066.stdout:0/41: dwrite d2/f5 [0,4194304] 0 2024-02-28T16:31:06.898 INFO:tasks.workunit.client.0.smithi001.stdout:2/422: write d0/d2/d3/de/d8a/f71 [1269994,74815] 0 2024-02-28T16:31:06.903 INFO:tasks.workunit.client.0.smithi001.stdout:7/398: getdents d3/d17/d52/d55 0 2024-02-28T16:31:06.903 INFO:tasks.workunit.client.0.smithi001.stdout:1/531: dread d0/f2c [0,4194304] 0 2024-02-28T16:31:06.905 INFO:tasks.workunit.client.0.smithi001.stdout:6/378: getdents d9/db/d19 0 2024-02-28T16:31:06.906 INFO:tasks.workunit.client.0.smithi001.stdout:2/423: rename d0/d2/d3/d36 to d0/d2/d3/d36/d8e 22 2024-02-28T16:31:06.908 INFO:tasks.workunit.client.1.smithi066.stdout:0/42: mknod d2/d9/ca 0 2024-02-28T16:31:06.909 INFO:tasks.workunit.client.1.smithi066.stdout:1/48: rmdir d2/d11 0 2024-02-28T16:31:06.909 INFO:tasks.workunit.client.0.smithi001.stdout:7/399: mkdir d3/d17/d52/d7f 0 2024-02-28T16:31:06.912 INFO:tasks.workunit.client.0.smithi001.stdout:7/400: dread - d3/d17/d3d/d42/f7d zero size 2024-02-28T16:31:06.912 INFO:tasks.workunit.client.1.smithi066.stdout:1/49: dread - d2/f10 zero size 2024-02-28T16:31:06.913 INFO:tasks.workunit.client.1.smithi066.stdout:0/43: symlink d2/d9/lb 0 2024-02-28T16:31:06.914 INFO:tasks.workunit.client.1.smithi066.stdout:4/16: dwrite d1/f4 [0,4194304] 0 2024-02-28T16:31:06.914 INFO:tasks.workunit.client.1.smithi066.stdout:1/50: stat d2/l9 0 2024-02-28T16:31:06.916 INFO:tasks.workunit.client.0.smithi001.stdout:4/383: dwrite d3/d5/d7/d1f/d2e/f59 [0,4194304] 0 2024-02-28T16:31:06.916 INFO:tasks.workunit.client.0.smithi001.stdout:6/379: rename d9/db/l3d to d9/db/l7c 0 2024-02-28T16:31:06.919 INFO:tasks.workunit.client.1.smithi066.stdout:4/17: unlink d1/l3 0 2024-02-28T16:31:06.919 INFO:tasks.workunit.client.0.smithi001.stdout:7/401: mkdir d3/d17/d3d/d59/d6e/d80 0 2024-02-28T16:31:06.920 INFO:tasks.workunit.client.0.smithi001.stdout:1/532: read d0/d9/d1a/d2b/f6d [559409,18769] 0 2024-02-28T16:31:06.920 INFO:tasks.workunit.client.1.smithi066.stdout:1/51: readlink d2/le 0 2024-02-28T16:31:06.922 INFO:tasks.workunit.client.0.smithi001.stdout:2/424: link d0/d2/d3/d73/f85 d0/d2/d3/de/f8f 0 2024-02-28T16:31:06.923 INFO:tasks.workunit.client.0.smithi001.stdout:2/425: stat d0/d2 0 2024-02-28T16:31:06.923 INFO:tasks.workunit.client.0.smithi001.stdout:6/380: creat d9/db/d5f/d62/f7d x:0 0 0 2024-02-28T16:31:06.925 INFO:tasks.workunit.client.1.smithi066.stdout:4/18: fsync d1/f2 0 2024-02-28T16:31:06.925 INFO:tasks.workunit.client.0.smithi001.stdout:1/533: write d0/f28 [4616239,95323] 0 2024-02-28T16:31:06.927 INFO:tasks.workunit.client.0.smithi001.stdout:7/402: unlink d3/d17/c60 0 2024-02-28T16:31:06.932 INFO:tasks.workunit.client.0.smithi001.stdout:1/534: chown d0/d9/d1a/d2b/d26/d27/d30/f58 883068 1 2024-02-28T16:31:06.932 INFO:tasks.workunit.client.1.smithi066.stdout:4/19: chown d1/f4 1041837 1 2024-02-28T16:31:06.932 INFO:tasks.workunit.client.1.smithi066.stdout:1/52: rename d2/fb to d2/f14 0 2024-02-28T16:31:06.933 INFO:tasks.workunit.client.0.smithi001.stdout:6/381: mkdir d9/db/d5f/d62/d7e 0 2024-02-28T16:31:06.935 INFO:tasks.workunit.client.1.smithi066.stdout:1/53: creat d2/f15 x:0 0 0 2024-02-28T16:31:06.937 INFO:tasks.workunit.client.0.smithi001.stdout:7/403: creat d3/d17/d52/d7f/f81 x:0 0 0 2024-02-28T16:31:06.937 INFO:tasks.workunit.client.0.smithi001.stdout:2/426: dread d0/d2/d3/de/d61/d12/d46/f57 [0,4194304] 0 2024-02-28T16:31:06.938 INFO:tasks.workunit.client.0.smithi001.stdout:1/535: write d0/d9/d1a/d2b/d26/d27/d30/d57/f8c [674509,30341] 0 2024-02-28T16:31:06.939 INFO:tasks.workunit.client.0.smithi001.stdout:7/404: fdatasync d3/f22 0 2024-02-28T16:31:06.939 INFO:tasks.workunit.client.0.smithi001.stdout:6/382: fsync d9/db/f1b 0 2024-02-28T16:31:06.940 INFO:tasks.workunit.client.1.smithi066.stdout:1/54: fsync d2/fc 0 2024-02-28T16:31:06.942 INFO:tasks.workunit.client.0.smithi001.stdout:4/384: write d3/d5/f38 [3250853,75252] 0 2024-02-28T16:31:06.944 INFO:tasks.workunit.client.0.smithi001.stdout:6/383: symlink d9/d29/d6e/d53/l7f 0 2024-02-28T16:31:06.945 INFO:tasks.workunit.client.1.smithi066.stdout:1/55: write d2/f13 [1372944,77900] 0 2024-02-28T16:31:06.945 INFO:tasks.workunit.client.1.smithi066.stdout:0/44: dwrite d2/f5 [0,4194304] 0 2024-02-28T16:31:06.949 INFO:tasks.workunit.client.1.smithi066.stdout:0/45: stat d2 0 2024-02-28T16:31:06.950 INFO:tasks.workunit.client.0.smithi001.stdout:6/384: creat d9/d29/d3e/f80 x:0 0 0 2024-02-28T16:31:06.951 INFO:tasks.workunit.client.0.smithi001.stdout:2/427: getdents d0/d2/d3/de/d61 0 2024-02-28T16:31:06.952 INFO:tasks.workunit.client.0.smithi001.stdout:6/385: chown d9/db/f68 12881094 1 2024-02-28T16:31:06.953 INFO:tasks.workunit.client.1.smithi066.stdout:0/46: chown d2/f5 115425 1 2024-02-28T16:31:06.955 INFO:tasks.workunit.client.1.smithi066.stdout:1/56: rename d2/f14 to d2/f16 0 2024-02-28T16:31:06.957 INFO:tasks.workunit.client.0.smithi001.stdout:4/385: creat d3/f7e x:0 0 0 2024-02-28T16:31:06.960 INFO:tasks.workunit.client.0.smithi001.stdout:6/386: symlink d9/d29/d2b/l81 0 2024-02-28T16:31:06.962 INFO:tasks.workunit.client.1.smithi066.stdout:1/57: creat d2/f17 x:0 0 0 2024-02-28T16:31:06.963 INFO:tasks.workunit.client.1.smithi066.stdout:4/20: dwrite d1/f2 [0,4194304] 0 2024-02-28T16:31:06.966 INFO:tasks.workunit.client.0.smithi001.stdout:6/387: rmdir d9/db 39 2024-02-28T16:31:06.968 INFO:tasks.workunit.client.0.smithi001.stdout:2/428: link d0/d2/d3/de/d8a/l35 d0/d2/d3/d73/l90 0 2024-02-28T16:31:06.968 INFO:tasks.workunit.client.0.smithi001.stdout:7/405: dwrite d3/d17/d3d/d42/f7d [0,4194304] 0 2024-02-28T16:31:06.969 INFO:tasks.workunit.client.1.smithi066.stdout:1/58: dread d2/f4 [0,4194304] 0 2024-02-28T16:31:06.969 INFO:tasks.workunit.client.0.smithi001.stdout:4/386: getdents d3/d5/d47 0 2024-02-28T16:31:06.971 INFO:tasks.workunit.client.1.smithi066.stdout:4/21: link d1/f4 d1/f5 0 2024-02-28T16:31:06.972 INFO:tasks.workunit.client.1.smithi066.stdout:1/59: mkdir d2/d18 0 2024-02-28T16:31:06.972 INFO:tasks.workunit.client.0.smithi001.stdout:1/536: dwrite d0/d9/d1a/d2b/d26/d27/d30/d52/f53 [0,4194304] 0 2024-02-28T16:31:06.973 INFO:tasks.workunit.client.0.smithi001.stdout:4/387: fsync d3/f12 0 2024-02-28T16:31:06.975 INFO:tasks.workunit.client.1.smithi066.stdout:1/60: fsync d2/f5 0 2024-02-28T16:31:06.976 INFO:tasks.workunit.client.1.smithi066.stdout:4/22: link d1/f2 d1/f6 0 2024-02-28T16:31:06.976 INFO:tasks.workunit.client.1.smithi066.stdout:0/47: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:06.977 INFO:tasks.workunit.client.1.smithi066.stdout:1/61: read d2/f13 [1483767,20103] 0 2024-02-28T16:31:06.979 INFO:tasks.workunit.client.1.smithi066.stdout:1/62: chown d2/f7 3 1 2024-02-28T16:31:06.980 INFO:tasks.workunit.client.1.smithi066.stdout:0/48: creat d2/fc x:0 0 0 2024-02-28T16:31:06.980 INFO:tasks.workunit.client.0.smithi001.stdout:2/429: creat d0/f91 x:0 0 0 2024-02-28T16:31:06.982 INFO:tasks.workunit.client.0.smithi001.stdout:4/388: mknod d3/d5/d7/d1f/d2e/c7f 0 2024-02-28T16:31:06.984 INFO:tasks.workunit.client.0.smithi001.stdout:4/389: read - d3/d5/d7/d1f/f7d zero size 2024-02-28T16:31:06.985 INFO:tasks.workunit.client.1.smithi066.stdout:1/63: mkdir d2/d18/d19 0 2024-02-28T16:31:06.987 INFO:tasks.workunit.client.1.smithi066.stdout:1/64: write d2/f15 [144004,112550] 0 2024-02-28T16:31:06.989 INFO:tasks.workunit.client.0.smithi001.stdout:7/406: getdents d3/d1c 0 2024-02-28T16:31:06.989 INFO:tasks.workunit.client.0.smithi001.stdout:1/537: symlink d0/d9/d1a/lb8 0 2024-02-28T16:31:06.990 INFO:tasks.workunit.client.0.smithi001.stdout:6/388: write d9/db/f1b [1360265,107822] 0 2024-02-28T16:31:06.991 INFO:tasks.workunit.client.0.smithi001.stdout:4/390: creat d3/d5/d4e/f80 x:0 0 0 2024-02-28T16:31:06.991 INFO:tasks.workunit.client.1.smithi066.stdout:4/23: dread d1/f2 [0,4194304] 0 2024-02-28T16:31:06.993 INFO:tasks.workunit.client.0.smithi001.stdout:2/430: dread d0/d2/d3/de/d61/d12/d46/f57 [0,4194304] 0 2024-02-28T16:31:06.993 INFO:tasks.workunit.client.0.smithi001.stdout:4/391: readlink d3/d5/d7/d1f/l29 0 2024-02-28T16:31:06.993 INFO:tasks.workunit.client.1.smithi066.stdout:1/65: read d2/f15 [229313,45508] 0 2024-02-28T16:31:06.994 INFO:tasks.workunit.client.0.smithi001.stdout:1/538: creat d0/d9/d1a/d72/fb9 x:0 0 0 2024-02-28T16:31:06.994 INFO:tasks.workunit.client.0.smithi001.stdout:7/407: chown d3/d17/d3d/d59/l76 27874 1 2024-02-28T16:31:06.996 INFO:tasks.workunit.client.0.smithi001.stdout:4/392: mkdir d3/d2d/d81 0 2024-02-28T16:31:06.997 INFO:tasks.workunit.client.1.smithi066.stdout:0/49: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:06.997 INFO:tasks.workunit.client.1.smithi066.stdout:4/24: fdatasync d1/f6 0 2024-02-28T16:31:06.998 INFO:tasks.workunit.client.0.smithi001.stdout:7/408: rename d3/d17/c3a to d3/d1c/c82 0 2024-02-28T16:31:06.999 INFO:tasks.workunit.client.0.smithi001.stdout:7/409: chown d3/d17/d3d/d59/d6e/d80 18290821 1 2024-02-28T16:31:07.000 INFO:tasks.workunit.client.1.smithi066.stdout:0/50: truncate d2/f5 4241994 0 2024-02-28T16:31:07.000 INFO:tasks.workunit.client.1.smithi066.stdout:1/66: link d2/cd d2/d18/d19/c1a 0 2024-02-28T16:31:07.002 INFO:tasks.workunit.client.0.smithi001.stdout:7/410: symlink d3/d17/d52/d55/l83 0 2024-02-28T16:31:07.008 INFO:tasks.workunit.client.0.smithi001.stdout:7/411: symlink d3/d1c/d41/l84 0 2024-02-28T16:31:07.009 INFO:tasks.workunit.client.1.smithi066.stdout:1/67: getdents d2/d18 0 2024-02-28T16:31:07.010 INFO:tasks.workunit.client.0.smithi001.stdout:7/412: stat d3/c15 0 2024-02-28T16:31:07.012 INFO:tasks.workunit.client.1.smithi066.stdout:4/25: write d1/f5 [2449115,49354] 0 2024-02-28T16:31:07.013 INFO:tasks.workunit.client.1.smithi066.stdout:0/51: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:07.013 INFO:tasks.workunit.client.1.smithi066.stdout:1/68: rmdir d2/d18/d19 39 2024-02-28T16:31:07.014 INFO:tasks.workunit.client.0.smithi001.stdout:7/413: unlink d3/d17/f54 0 2024-02-28T16:31:07.014 INFO:tasks.workunit.client.0.smithi001.stdout:2/431: dwrite d0/d2/d3/de/d8a/f70 [0,4194304] 0 2024-02-28T16:31:07.014 INFO:tasks.workunit.client.1.smithi066.stdout:0/52: dread - d2/fc zero size 2024-02-28T16:31:07.016 INFO:tasks.workunit.client.1.smithi066.stdout:0/53: chown d2/c8 160651 1 2024-02-28T16:31:07.018 INFO:tasks.workunit.client.1.smithi066.stdout:0/54: mknod d2/cd 0 2024-02-28T16:31:07.019 INFO:tasks.workunit.client.1.smithi066.stdout:0/55: fsync d2/fc 0 2024-02-28T16:31:07.023 INFO:tasks.workunit.client.1.smithi066.stdout:0/56: creat d2/fe x:0 0 0 2024-02-28T16:31:07.023 INFO:tasks.workunit.client.0.smithi001.stdout:2/432: read d0/d2/d3/de/d61/d12/d6a/f81 [1863561,15561] 0 2024-02-28T16:31:07.025 INFO:tasks.workunit.client.0.smithi001.stdout:7/414: rename d3/d17/d3d/l3f to d3/d17/d52/l85 0 2024-02-28T16:31:07.025 INFO:tasks.workunit.client.0.smithi001.stdout:4/393: dwrite d3/d2d/d4c/f50 [0,4194304] 0 2024-02-28T16:31:07.028 INFO:tasks.workunit.client.0.smithi001.stdout:6/389: dread d9/d29/f44 [0,4194304] 0 2024-02-28T16:31:07.032 INFO:tasks.workunit.client.1.smithi066.stdout:0/57: creat d2/d9/ff x:0 0 0 2024-02-28T16:31:07.033 INFO:tasks.workunit.client.1.smithi066.stdout:4/26: dread d1/f4 [0,4194304] 0 2024-02-28T16:31:07.034 INFO:tasks.workunit.client.0.smithi001.stdout:6/390: symlink d9/de/l82 0 2024-02-28T16:31:07.034 INFO:tasks.workunit.client.0.smithi001.stdout:7/415: rename d3/d1c/l3e to d3/d17/d52/l86 0 2024-02-28T16:31:07.034 INFO:tasks.workunit.client.0.smithi001.stdout:6/391: dread - d9/db/d5f/d62/f7d zero size 2024-02-28T16:31:07.034 INFO:tasks.workunit.client.0.smithi001.stdout:2/433: mkdir d0/d2/d3/de/d22/d80/d92 0 2024-02-28T16:31:07.036 INFO:tasks.workunit.client.0.smithi001.stdout:7/416: symlink d3/d1c/d41/d49/l87 0 2024-02-28T16:31:07.038 INFO:tasks.workunit.client.0.smithi001.stdout:6/392: mkdir d9/db/d5f/d62/d83 0 2024-02-28T16:31:07.039 INFO:tasks.workunit.client.0.smithi001.stdout:2/434: readlink d0/d2/d3/de/d61/d12/d46/l8d 0 2024-02-28T16:31:07.040 INFO:tasks.workunit.client.0.smithi001.stdout:7/417: creat d3/d17/d3d/d59/d6e/f88 x:0 0 0 2024-02-28T16:31:07.043 INFO:tasks.workunit.client.1.smithi066.stdout:4/27: link d1/f4 d1/f7 0 2024-02-28T16:31:07.043 INFO:tasks.workunit.client.1.smithi066.stdout:1/69: dwrite d2/f4 [0,4194304] 0 2024-02-28T16:31:07.053 INFO:tasks.workunit.client.1.smithi066.stdout:1/70: truncate d2/f16 65871 0 2024-02-28T16:31:07.055 INFO:tasks.workunit.client.1.smithi066.stdout:1/71: chown d2/d18/d19 616 1 2024-02-28T16:31:07.058 INFO:tasks.workunit.client.1.smithi066.stdout:0/58: dwrite d2/fc [0,4194304] 0 2024-02-28T16:31:07.060 INFO:tasks.workunit.client.1.smithi066.stdout:4/28: truncate d1/f4 2885332 0 2024-02-28T16:31:07.061 INFO:tasks.workunit.client.1.smithi066.stdout:4/29: readlink - no filename 2024-02-28T16:31:07.061 INFO:tasks.workunit.client.0.smithi001.stdout:6/393: dwrite d9/db/d10/f5b [0,4194304] 0 2024-02-28T16:31:07.065 INFO:tasks.workunit.client.1.smithi066.stdout:0/59: mkdir d2/d9/d10 0 2024-02-28T16:31:07.065 INFO:tasks.workunit.client.0.smithi001.stdout:4/394: dread d3/d1c/d58/f6b [0,4194304] 0 2024-02-28T16:31:07.066 INFO:tasks.workunit.client.0.smithi001.stdout:4/395: chown d3/d5 236505 1 2024-02-28T16:31:07.068 INFO:tasks.workunit.client.1.smithi066.stdout:1/72: rmdir d2 39 2024-02-28T16:31:07.071 INFO:tasks.workunit.client.0.smithi001.stdout:4/396: readlink d3/d2d/l36 0 2024-02-28T16:31:07.071 INFO:tasks.workunit.client.1.smithi066.stdout:4/30: truncate d1/f6 1904818 0 2024-02-28T16:31:07.072 INFO:tasks.workunit.client.1.smithi066.stdout:4/31: stat d1 0 2024-02-28T16:31:07.074 INFO:tasks.workunit.client.1.smithi066.stdout:4/32: chown d1/f2 0 1 2024-02-28T16:31:07.077 INFO:tasks.workunit.client.1.smithi066.stdout:0/60: rename d2/f7 to d2/d9/f11 0 2024-02-28T16:31:07.077 INFO:tasks.workunit.client.0.smithi001.stdout:2/435: dwrite d0/fd [0,4194304] 0 2024-02-28T16:31:07.079 INFO:tasks.workunit.client.0.smithi001.stdout:4/397: rmdir d3/d2d/d4c 39 2024-02-28T16:31:07.079 INFO:tasks.workunit.client.1.smithi066.stdout:4/33: symlink d1/l8 0 2024-02-28T16:31:07.083 INFO:tasks.workunit.client.1.smithi066.stdout:4/34: mknod d1/c9 0 2024-02-28T16:31:07.086 INFO:tasks.workunit.client.1.smithi066.stdout:1/73: rename d2/l12 to d2/d18/l1b 0 2024-02-28T16:31:07.088 INFO:tasks.workunit.client.0.smithi001.stdout:6/394: dwrite d9/d29/d6e/f2f [0,4194304] 0 2024-02-28T16:31:07.089 INFO:tasks.workunit.client.1.smithi066.stdout:0/61: link d2/l3 d2/l12 0 2024-02-28T16:31:07.089 INFO:tasks.workunit.client.0.smithi001.stdout:4/398: rename d3/d1c/d42 to d3/d2d/d4c/d69/d82 0 2024-02-28T16:31:07.090 INFO:tasks.workunit.client.1.smithi066.stdout:1/74: write d2/f5 [4608387,18942] 0 2024-02-28T16:31:07.093 INFO:tasks.workunit.client.1.smithi066.stdout:1/75: write d2/f17 [359795,129084] 0 2024-02-28T16:31:07.093 INFO:tasks.workunit.client.0.smithi001.stdout:4/399: write d3/d1c/f4a [1566619,52027] 0 2024-02-28T16:31:07.096 INFO:tasks.workunit.client.1.smithi066.stdout:0/62: link d2/c8 d2/d9/c13 0 2024-02-28T16:31:07.097 INFO:tasks.workunit.client.0.smithi001.stdout:4/400: rmdir d3/d1c/d58/d6e 39 2024-02-28T16:31:07.099 INFO:tasks.workunit.client.1.smithi066.stdout:0/63: truncate d2/fc 4297081 0 2024-02-28T16:31:07.102 INFO:tasks.workunit.client.1.smithi066.stdout:0/64: chown d2/d9/d10 42355019 1 2024-02-28T16:31:07.102 INFO:tasks.workunit.client.1.smithi066.stdout:0/65: fsync d2/fc 0 2024-02-28T16:31:07.102 INFO:tasks.workunit.client.1.smithi066.stdout:4/35: unlink d1/f2 0 2024-02-28T16:31:07.105 INFO:tasks.workunit.client.0.smithi001.stdout:2/436: dwrite d0/d2/d3/de/d22/f38 [0,4194304] 0 2024-02-28T16:31:07.108 INFO:tasks.workunit.client.1.smithi066.stdout:1/76: dread d2/f7 [0,4194304] 0 2024-02-28T16:31:07.112 INFO:tasks.workunit.client.1.smithi066.stdout:0/66: mkdir d2/d9/d10/d14 0 2024-02-28T16:31:07.112 INFO:tasks.workunit.client.0.smithi001.stdout:6/395: dwrite d9/db/d5f/d62/f6b [0,4194304] 0 2024-02-28T16:31:07.112 INFO:tasks.workunit.client.0.smithi001.stdout:2/437: write d0/d2/d3/de/d22/f48 [469447,122923] 0 2024-02-28T16:31:07.117 INFO:tasks.workunit.client.0.smithi001.stdout:2/438: mknod d0/d2/d3/d73/c93 0 2024-02-28T16:31:07.118 INFO:tasks.workunit.client.1.smithi066.stdout:1/77: link d2/f7 d2/d18/d19/f1c 0 2024-02-28T16:31:07.118 INFO:tasks.workunit.client.1.smithi066.stdout:0/67: rename d2/cd to d2/d9/c15 0 2024-02-28T16:31:07.119 INFO:tasks.workunit.client.1.smithi066.stdout:0/68: stat d2/d9/ff 0 2024-02-28T16:31:07.119 INFO:tasks.workunit.client.0.smithi001.stdout:2/439: chown d0/d2/f14 109610 1 2024-02-28T16:31:07.120 INFO:tasks.workunit.client.1.smithi066.stdout:0/69: fsync d2/d9/ff 0 2024-02-28T16:31:07.121 INFO:tasks.workunit.client.0.smithi001.stdout:6/396: rename d9/d29/d6e/d53/l58 to d9/l84 0 2024-02-28T16:31:07.122 INFO:tasks.workunit.client.1.smithi066.stdout:4/36: write d1/f4 [1956989,40090] 0 2024-02-28T16:31:07.123 INFO:tasks.workunit.client.0.smithi001.stdout:6/397: readlink d9/de/l50 0 2024-02-28T16:31:07.124 INFO:tasks.workunit.client.0.smithi001.stdout:2/440: write d0/f59 [964841,64000] 0 2024-02-28T16:31:07.127 INFO:tasks.workunit.client.1.smithi066.stdout:0/70: rmdir d2/d9/d10/d14 0 2024-02-28T16:31:07.130 INFO:tasks.workunit.client.1.smithi066.stdout:0/71: write d2/fe [762787,70583] 0 2024-02-28T16:31:07.133 INFO:tasks.workunit.client.1.smithi066.stdout:0/72: write d2/d9/ff [673643,24464] 0 2024-02-28T16:31:07.136 INFO:tasks.workunit.client.1.smithi066.stdout:0/73: mknod d2/c16 0 2024-02-28T16:31:07.137 INFO:tasks.workunit.client.0.smithi001.stdout:2/441: rmdir d0/d2/d3/de/d22/d80/d92 0 2024-02-28T16:31:07.143 INFO:tasks.workunit.client.1.smithi066.stdout:4/37: rename d1/c9 to d1/ca 0 2024-02-28T16:31:07.144 INFO:tasks.workunit.client.1.smithi066.stdout:1/78: dwrite d2/f16 [0,4194304] 0 2024-02-28T16:31:07.146 INFO:tasks.workunit.client.0.smithi001.stdout:2/442: getdents d0/d2/d3/de/d61/d12/d46 0 2024-02-28T16:31:07.148 INFO:tasks.workunit.client.1.smithi066.stdout:1/79: stat d2/d18/d19/c1a 0 2024-02-28T16:31:07.152 INFO:tasks.workunit.client.1.smithi066.stdout:1/80: creat d2/f1d x:0 0 0 2024-02-28T16:31:07.152 INFO:tasks.workunit.client.0.smithi001.stdout:6/398: dwrite d9/d29/d6e/d53/f7a [0,4194304] 0 2024-02-28T16:31:07.155 INFO:tasks.workunit.client.1.smithi066.stdout:1/81: write d2/f17 [167752,94007] 0 2024-02-28T16:31:07.156 INFO:tasks.workunit.client.1.smithi066.stdout:4/38: dread d1/f5 [0,4194304] 0 2024-02-28T16:31:07.160 INFO:tasks.workunit.client.0.smithi001.stdout:6/399: rename d9/db/fd to d9/d29/d6a/f85 0 2024-02-28T16:31:07.162 INFO:tasks.workunit.client.1.smithi066.stdout:0/74: dwrite d2/d9/f11 [0,4194304] 0 2024-02-28T16:31:07.164 INFO:tasks.workunit.client.0.smithi001.stdout:6/400: creat d9/f86 x:0 0 0 2024-02-28T16:31:07.166 INFO:tasks.workunit.client.1.smithi066.stdout:0/75: rmdir d2/d9 39 2024-02-28T16:31:07.167 INFO:tasks.workunit.client.0.smithi001.stdout:2/443: truncate d0/d2/d3/d36/f68 257031 0 2024-02-28T16:31:07.171 INFO:tasks.workunit.client.0.smithi001.stdout:6/401: creat d9/d29/f87 x:0 0 0 2024-02-28T16:31:07.172 INFO:tasks.workunit.client.0.smithi001.stdout:2/444: mkdir d0/d2/d3/d36/d94 0 2024-02-28T16:31:07.173 INFO:tasks.workunit.client.0.smithi001.stdout:2/445: fsync d0/f59 0 2024-02-28T16:31:07.176 INFO:tasks.workunit.client.0.smithi001.stdout:6/402: creat d9/db/f88 x:0 0 0 2024-02-28T16:31:07.178 INFO:tasks.workunit.client.0.smithi001.stdout:6/403: chown d9/db/d19/c70 93801879 1 2024-02-28T16:31:07.179 INFO:tasks.workunit.client.0.smithi001.stdout:6/404: chown d9/db/d19 225044 1 2024-02-28T16:31:07.180 INFO:tasks.workunit.client.1.smithi066.stdout:1/82: dwrite d2/f1d [0,4194304] 0 2024-02-28T16:31:07.180 INFO:tasks.workunit.client.0.smithi001.stdout:6/405: fsync d9/db/f13 0 2024-02-28T16:31:07.185 INFO:tasks.workunit.client.1.smithi066.stdout:4/39: dwrite d1/f7 [0,4194304] 0 2024-02-28T16:31:07.186 INFO:tasks.workunit.client.1.smithi066.stdout:0/76: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:07.187 INFO:tasks.workunit.client.1.smithi066.stdout:1/83: creat d2/d18/d19/f1e x:0 0 0 2024-02-28T16:31:07.189 INFO:tasks.workunit.client.0.smithi001.stdout:2/446: write d0/f40 [2248080,119343] 0 2024-02-28T16:31:07.192 INFO:tasks.workunit.client.0.smithi001.stdout:6/406: dread d9/de/f5a [4194304,4194304] 0 2024-02-28T16:31:07.194 INFO:tasks.workunit.client.0.smithi001.stdout:2/447: stat d0/d2/d3/de/d22/f24 0 2024-02-28T16:31:07.196 INFO:tasks.workunit.client.0.smithi001.stdout:6/407: mkdir d9/db/d5f/d89 0 2024-02-28T16:31:07.197 INFO:tasks.workunit.client.0.smithi001.stdout:6/408: stat d9/d29/d3e 0 2024-02-28T16:31:07.199 INFO:tasks.workunit.client.1.smithi066.stdout:1/84: truncate d2/f13 658783 0 2024-02-28T16:31:07.200 INFO:tasks.workunit.client.0.smithi001.stdout:6/409: write d9/d29/f87 [368034,110942] 0 2024-02-28T16:31:07.201 INFO:tasks.workunit.client.1.smithi066.stdout:1/85: truncate d2/f16 5078702 0 2024-02-28T16:31:07.203 INFO:tasks.workunit.client.0.smithi001.stdout:2/448: rename d0/d2/d3/d36/c89 to d0/d2/d3/de/c95 0 2024-02-28T16:31:07.204 INFO:tasks.workunit.client.0.smithi001.stdout:6/410: readlink d9/l84 0 2024-02-28T16:31:07.204 INFO:tasks.workunit.client.1.smithi066.stdout:4/40: dread d1/f5 [0,4194304] 0 2024-02-28T16:31:07.205 INFO:tasks.workunit.client.0.smithi001.stdout:6/411: fdatasync d9/d29/d3e/f6d 0 2024-02-28T16:31:07.207 INFO:tasks.workunit.client.1.smithi066.stdout:4/41: mknod d1/cb 0 2024-02-28T16:31:07.208 INFO:tasks.workunit.client.0.smithi001.stdout:2/449: rename d0/d2/d3/de/d8a/f10 to d0/d2/d3/d36/d94/f96 0 2024-02-28T16:31:07.210 INFO:tasks.workunit.client.1.smithi066.stdout:0/77: dwrite d2/d9/ff [0,4194304] 0 2024-02-28T16:31:07.211 INFO:tasks.workunit.client.1.smithi066.stdout:0/78: fsync d2/d9/f11 0 2024-02-28T16:31:07.214 INFO:tasks.workunit.client.0.smithi001.stdout:2/450: rmdir d0/d2/d3/de/d22/d80 0 2024-02-28T16:31:07.214 INFO:tasks.workunit.client.0.smithi001.stdout:2/451: fdatasync d0/d2/f8c 0 2024-02-28T16:31:07.218 INFO:tasks.workunit.client.1.smithi066.stdout:4/42: truncate d1/f5 1276490 0 2024-02-28T16:31:07.224 INFO:tasks.workunit.client.0.smithi001.stdout:2/452: truncate d0/d2/f3b 222626 0 2024-02-28T16:31:07.224 INFO:tasks.workunit.client.1.smithi066.stdout:4/43: link c0 d1/cc 0 2024-02-28T16:31:07.227 INFO:tasks.workunit.client.1.smithi066.stdout:4/44: mknod d1/cd 0 2024-02-28T16:31:07.227 INFO:tasks.workunit.client.1.smithi066.stdout:0/79: dwrite d2/fc [0,4194304] 0 2024-02-28T16:31:07.228 INFO:tasks.workunit.client.0.smithi001.stdout:6/412: dwrite d9/db/d5f/d69/f75 [0,4194304] 0 2024-02-28T16:31:07.228 INFO:tasks.workunit.client.0.smithi001.stdout:2/453: unlink d0/d2/d3/f29 0 2024-02-28T16:31:07.231 INFO:tasks.workunit.client.1.smithi066.stdout:0/80: symlink d2/d9/d10/l17 0 2024-02-28T16:31:07.234 INFO:tasks.workunit.client.0.smithi001.stdout:2/454: symlink d0/d2/l97 0 2024-02-28T16:31:07.236 INFO:tasks.workunit.client.1.smithi066.stdout:0/81: write d2/f5 [5044056,89638] 0 2024-02-28T16:31:07.241 INFO:tasks.workunit.client.1.smithi066.stdout:0/82: write d2/d9/f11 [4893184,87881] 0 2024-02-28T16:31:07.242 INFO:tasks.workunit.client.0.smithi001.stdout:2/455: mknod d0/d2/d3/c98 0 2024-02-28T16:31:07.243 INFO:tasks.workunit.client.1.smithi066.stdout:0/83: creat d2/d9/f18 x:0 0 0 2024-02-28T16:31:07.244 INFO:tasks.workunit.client.1.smithi066.stdout:4/45: read d1/f7 [771981,7678] 0 2024-02-28T16:31:07.258 INFO:tasks.workunit.client.1.smithi066.stdout:0/84: dwrite d2/d9/f11 [0,4194304] 0 2024-02-28T16:31:07.259 INFO:tasks.workunit.client.1.smithi066.stdout:4/46: dread d1/f4 [0,4194304] 0 2024-02-28T16:31:07.260 INFO:tasks.workunit.client.0.smithi001.stdout:2/456: dwrite d0/d2/d3/de/d22/f48 [0,4194304] 0 2024-02-28T16:31:07.260 INFO:tasks.workunit.client.1.smithi066.stdout:0/85: rename d2/d9/d10 to d2/d9/d10/d19 22 2024-02-28T16:31:07.262 INFO:tasks.workunit.client.0.smithi001.stdout:2/457: rmdir d0/d2/d3/d36 39 2024-02-28T16:31:07.265 INFO:tasks.workunit.client.0.smithi001.stdout:2/458: chown d0/d2/d3/de/d61/f23 0 1 2024-02-28T16:31:07.265 INFO:tasks.workunit.client.1.smithi066.stdout:0/86: symlink d2/d9/d10/l1a 0 2024-02-28T16:31:07.267 INFO:tasks.workunit.client.1.smithi066.stdout:4/47: write d1/f6 [2714326,91283] 0 2024-02-28T16:31:07.267 INFO:tasks.workunit.client.0.smithi001.stdout:3/377: sync 2024-02-28T16:31:07.270 INFO:tasks.workunit.client.0.smithi001.stdout:2/459: fdatasync d0/d2/d3/de/d22/f24 0 2024-02-28T16:31:07.271 INFO:tasks.workunit.client.1.smithi066.stdout:0/87: symlink d2/d9/d10/l1b 0 2024-02-28T16:31:07.272 INFO:tasks.workunit.client.0.smithi001.stdout:3/378: unlink d4/d38/l70 0 2024-02-28T16:31:07.274 INFO:tasks.workunit.client.0.smithi001.stdout:2/460: chown d0/d2/d3/de/d22/f48 16 1 2024-02-28T16:31:07.275 INFO:tasks.workunit.client.0.smithi001.stdout:3/379: mkdir d4/d1e/d7c 0 2024-02-28T16:31:07.275 INFO:tasks.workunit.client.1.smithi066.stdout:4/48: creat d1/fe x:0 0 0 2024-02-28T16:31:07.278 INFO:tasks.workunit.client.0.smithi001.stdout:3/380: mkdir d4/d1e/d7c/d7d 0 2024-02-28T16:31:07.279 INFO:tasks.workunit.client.1.smithi066.stdout:4/49: fdatasync d1/f7 0 2024-02-28T16:31:07.281 INFO:tasks.workunit.client.0.smithi001.stdout:3/381: mkdir d4/d1e/d7e 0 2024-02-28T16:31:07.282 INFO:tasks.workunit.client.0.smithi001.stdout:2/461: dread d0/f59 [0,4194304] 0 2024-02-28T16:31:07.283 INFO:tasks.workunit.client.1.smithi066.stdout:4/50: link d1/f5 d1/ff 0 2024-02-28T16:31:07.285 INFO:tasks.workunit.client.1.smithi066.stdout:0/88: dread d2/d9/ff [0,4194304] 0 2024-02-28T16:31:07.285 INFO:tasks.workunit.client.0.smithi001.stdout:3/382: mkdir d4/d6/d7f 0 2024-02-28T16:31:07.286 INFO:tasks.workunit.client.1.smithi066.stdout:4/51: chown d1/ca 32796836 1 2024-02-28T16:31:07.287 INFO:tasks.workunit.client.0.smithi001.stdout:3/383: chown d4/d1e/f21 3137 1 2024-02-28T16:31:07.288 INFO:tasks.workunit.client.1.smithi066.stdout:0/89: fsync d2/d9/ff 0 2024-02-28T16:31:07.291 INFO:tasks.workunit.client.0.smithi001.stdout:3/384: mkdir d4/d1e/d7c/d7d/d80 0 2024-02-28T16:31:07.291 INFO:tasks.workunit.client.1.smithi066.stdout:4/52: chown d1/ca 132 1 2024-02-28T16:31:07.291 INFO:tasks.workunit.client.1.smithi066.stdout:4/53: read d1/f5 [741731,123356] 0 2024-02-28T16:31:07.292 INFO:tasks.workunit.client.1.smithi066.stdout:0/90: unlink d2/d9/ff 0 2024-02-28T16:31:07.294 INFO:tasks.workunit.client.0.smithi001.stdout:3/385: mkdir d4/d6/d7/d78/d81 0 2024-02-28T16:31:07.295 INFO:tasks.workunit.client.1.smithi066.stdout:4/54: dread d1/f4 [0,4194304] 0 2024-02-28T16:31:07.296 INFO:tasks.workunit.client.1.smithi066.stdout:0/91: rename d2/d9/d10/l1a to d2/d9/l1c 0 2024-02-28T16:31:07.299 INFO:tasks.workunit.client.0.smithi001.stdout:2/462: dwrite d0/d2/d3/de/d8a/f6e [0,4194304] 0 2024-02-28T16:31:07.300 INFO:tasks.workunit.client.1.smithi066.stdout:0/92: creat d2/d9/d10/f1d x:0 0 0 2024-02-28T16:31:07.301 INFO:tasks.workunit.client.1.smithi066.stdout:4/55: creat d1/f10 x:0 0 0 2024-02-28T16:31:07.302 INFO:tasks.workunit.client.0.smithi001.stdout:2/463: mknod d0/d2/c99 0 2024-02-28T16:31:07.304 INFO:tasks.workunit.client.1.smithi066.stdout:4/56: write d1/f4 [1806600,112328] 0 2024-02-28T16:31:07.310 INFO:tasks.workunit.client.0.smithi001.stdout:2/464: symlink d0/d2/d3/d36/d94/l9a 0 2024-02-28T16:31:07.312 INFO:tasks.workunit.client.0.smithi001.stdout:3/386: truncate d4/d18/f10 370324 0 2024-02-28T16:31:07.313 INFO:tasks.workunit.client.1.smithi066.stdout:4/57: dread d1/f7 [0,4194304] 0 2024-02-28T16:31:07.314 INFO:tasks.workunit.client.0.smithi001.stdout:3/387: truncate d4/d18/f49 2215 0 2024-02-28T16:31:07.316 INFO:tasks.workunit.client.0.smithi001.stdout:2/465: link d0/d2/d3/d36/f39 d0/d2/d3/d73/f9b 0 2024-02-28T16:31:07.316 INFO:tasks.workunit.client.1.smithi066.stdout:0/93: dwrite d2/fe [0,4194304] 0 2024-02-28T16:31:07.317 INFO:tasks.workunit.client.1.smithi066.stdout:4/58: creat d1/f11 x:0 0 0 2024-02-28T16:31:07.317 INFO:tasks.workunit.client.0.smithi001.stdout:3/388: mkdir d4/d6/d7f/d82 0 2024-02-28T16:31:07.318 INFO:tasks.workunit.client.1.smithi066.stdout:4/59: chown d1/f5 44 1 2024-02-28T16:31:07.320 INFO:tasks.workunit.client.1.smithi066.stdout:0/94: creat d2/d9/f1e x:0 0 0 2024-02-28T16:31:07.321 INFO:tasks.workunit.client.1.smithi066.stdout:4/60: stat d1/l8 0 2024-02-28T16:31:07.325 INFO:tasks.workunit.client.1.smithi066.stdout:4/61: creat d1/f12 x:0 0 0 2024-02-28T16:31:07.326 INFO:tasks.workunit.client.1.smithi066.stdout:4/62: write d1/f6 [3828957,42007] 0 2024-02-28T16:31:07.333 INFO:tasks.workunit.client.0.smithi001.stdout:3/389: dwrite d4/f17 [0,4194304] 0 2024-02-28T16:31:07.337 INFO:tasks.workunit.client.1.smithi066.stdout:4/63: dread d1/ff [0,4194304] 0 2024-02-28T16:31:07.338 INFO:tasks.workunit.client.0.smithi001.stdout:3/390: rename d4/f17 to d4/f83 0 2024-02-28T16:31:07.339 INFO:tasks.workunit.client.1.smithi066.stdout:4/64: write d1/f6 [442860,34148] 0 2024-02-28T16:31:07.339 INFO:tasks.workunit.client.1.smithi066.stdout:0/95: dwrite d2/fe [0,4194304] 0 2024-02-28T16:31:07.348 INFO:tasks.workunit.client.1.smithi066.stdout:0/96: rename d2/f5 to d2/d9/d10/f1f 0 2024-02-28T16:31:07.349 INFO:tasks.workunit.client.1.smithi066.stdout:4/65: rename d1/ff to d1/f13 0 2024-02-28T16:31:07.350 INFO:tasks.workunit.client.1.smithi066.stdout:0/97: creat d2/d9/f20 x:0 0 0 2024-02-28T16:31:07.354 INFO:tasks.workunit.client.0.smithi001.stdout:3/391: truncate d4/d6/d7/f65 4934376 0 2024-02-28T16:31:07.355 INFO:tasks.workunit.client.1.smithi066.stdout:4/66: mknod d1/c14 0 2024-02-28T16:31:07.356 INFO:tasks.workunit.client.0.smithi001.stdout:3/392: fsync d4/f26 0 2024-02-28T16:31:07.362 INFO:tasks.workunit.client.1.smithi066.stdout:4/67: dread d1/f7 [0,4194304] 0 2024-02-28T16:31:07.362 INFO:tasks.workunit.client.1.smithi066.stdout:0/98: rmdir d2 39 2024-02-28T16:31:07.366 INFO:tasks.workunit.client.1.smithi066.stdout:0/99: write d2/d9/f20 [833245,74134] 0 2024-02-28T16:31:07.366 INFO:tasks.workunit.client.0.smithi001.stdout:3/393: rename d4/d6/d7/d23/c73 to d4/d3c/c84 0 2024-02-28T16:31:07.367 INFO:tasks.workunit.client.0.smithi001.stdout:3/394: write d4/d18/d5f/f47 [1400540,100686] 0 2024-02-28T16:31:07.369 INFO:tasks.workunit.client.0.smithi001.stdout:3/395: write d4/d18/d5f/f47 [1020634,55930] 0 2024-02-28T16:31:07.370 INFO:tasks.workunit.client.1.smithi066.stdout:0/100: rename d2/d9/f18 to d2/d9/f21 0 2024-02-28T16:31:07.371 INFO:tasks.workunit.client.1.smithi066.stdout:4/68: dread d1/f7 [0,4194304] 0 2024-02-28T16:31:07.371 INFO:tasks.workunit.client.0.smithi001.stdout:3/396: write d4/d6/d7/d23/f34 [2658495,3651] 0 2024-02-28T16:31:07.372 INFO:tasks.workunit.client.1.smithi066.stdout:0/101: write d2/fe [4879745,63235] 0 2024-02-28T16:31:07.373 INFO:tasks.workunit.client.0.smithi001.stdout:3/397: read d4/d6/fc [816189,100511] 0 2024-02-28T16:31:07.374 INFO:tasks.workunit.client.0.smithi001.stdout:3/398: read d4/f83 [2275443,130405] 0 2024-02-28T16:31:07.376 INFO:tasks.workunit.client.1.smithi066.stdout:0/102: creat d2/f22 x:0 0 0 2024-02-28T16:31:07.378 INFO:tasks.workunit.client.1.smithi066.stdout:0/103: rmdir d2/d9 39 2024-02-28T16:31:07.382 INFO:tasks.workunit.client.1.smithi066.stdout:0/104: mkdir d2/d9/d10/d23 0 2024-02-28T16:31:07.383 INFO:tasks.workunit.client.1.smithi066.stdout:0/105: write d2/d9/f1e [999257,68496] 0 2024-02-28T16:31:07.387 INFO:tasks.workunit.client.0.smithi001.stdout:3/399: dwrite d4/d3c/f3e [0,4194304] 0 2024-02-28T16:31:07.389 INFO:tasks.workunit.client.1.smithi066.stdout:0/106: link d2/d9/d10/l1b d2/d9/d10/d23/l24 0 2024-02-28T16:31:07.391 INFO:tasks.workunit.client.0.smithi001.stdout:3/400: rmdir d4/d6/d7/d23 39 2024-02-28T16:31:07.394 INFO:tasks.workunit.client.1.smithi066.stdout:0/107: mknod d2/d9/d10/c25 0 2024-02-28T16:31:07.394 INFO:tasks.workunit.client.1.smithi066.stdout:4/69: dwrite d1/f4 [0,4194304] 0 2024-02-28T16:31:07.397 INFO:tasks.workunit.client.1.smithi066.stdout:0/108: write d2/fc [2400525,71102] 0 2024-02-28T16:31:07.402 INFO:tasks.workunit.client.1.smithi066.stdout:4/70: rmdir d1 39 2024-02-28T16:31:07.406 INFO:tasks.workunit.client.1.smithi066.stdout:0/109: creat d2/d9/d10/f26 x:0 0 0 2024-02-28T16:31:07.409 INFO:tasks.workunit.client.1.smithi066.stdout:4/71: symlink d1/l15 0 2024-02-28T16:31:07.410 INFO:tasks.workunit.client.1.smithi066.stdout:4/72: stat d1/cd 0 2024-02-28T16:31:07.411 INFO:tasks.workunit.client.0.smithi001.stdout:3/401: dwrite d4/d18/d5f/f77 [0,4194304] 0 2024-02-28T16:31:07.413 INFO:tasks.workunit.client.1.smithi066.stdout:4/73: creat d1/f16 x:0 0 0 2024-02-28T16:31:07.415 INFO:tasks.workunit.client.0.smithi001.stdout:3/402: rename d4/d1e/c6f to d4/d1e/c85 0 2024-02-28T16:31:07.417 INFO:tasks.workunit.client.0.smithi001.stdout:3/403: stat d4/d6/d7/f2f 0 2024-02-28T16:31:07.421 INFO:tasks.workunit.client.0.smithi001.stdout:3/404: creat d4/d6/d7/d78/d81/f86 x:0 0 0 2024-02-28T16:31:07.421 INFO:tasks.workunit.client.1.smithi066.stdout:4/74: rmdir d1 39 2024-02-28T16:31:07.422 INFO:tasks.workunit.client.0.smithi001.stdout:3/405: chown d4/d3c/c4d 0 1 2024-02-28T16:31:07.424 INFO:tasks.workunit.client.1.smithi066.stdout:4/75: symlink d1/l17 0 2024-02-28T16:31:07.426 INFO:tasks.workunit.client.0.smithi001.stdout:3/406: creat d4/d6/f87 x:0 0 0 2024-02-28T16:31:07.431 INFO:tasks.workunit.client.0.smithi001.stdout:3/407: write d4/f35 [607197,72192] 0 2024-02-28T16:31:07.444 INFO:tasks.workunit.client.1.smithi066.stdout:4/76: dwrite d1/f16 [0,4194304] 0 2024-02-28T16:31:07.444 INFO:tasks.workunit.client.0.smithi001.stdout:3/408: dwrite d4/f76 [0,4194304] 0 2024-02-28T16:31:07.446 INFO:tasks.workunit.client.0.smithi001.stdout:3/409: rmdir d4/d1e 39 2024-02-28T16:31:07.446 INFO:tasks.workunit.client.1.smithi066.stdout:4/77: mkdir d1/d18 0 2024-02-28T16:31:07.448 INFO:tasks.workunit.client.0.smithi001.stdout:3/410: write d4/d6/f87 [784930,80151] 0 2024-02-28T16:31:07.458 INFO:tasks.workunit.client.1.smithi066.stdout:4/78: truncate d1/f6 687227 0 2024-02-28T16:31:07.464 INFO:tasks.workunit.client.1.smithi066.stdout:4/79: truncate d1/f16 2119672 0 2024-02-28T16:31:07.466 INFO:tasks.workunit.client.1.smithi066.stdout:4/80: truncate d1/f10 255718 0 2024-02-28T16:31:07.467 INFO:tasks.workunit.client.1.smithi066.stdout:4/81: stat d1/f7 0 2024-02-28T16:31:07.471 INFO:tasks.workunit.client.0.smithi001.stdout:0/397: sync 2024-02-28T16:31:07.471 INFO:tasks.workunit.client.0.smithi001.stdout:8/404: sync 2024-02-28T16:31:07.471 INFO:tasks.workunit.client.0.smithi001.stdout:9/498: sync 2024-02-28T16:31:07.480 INFO:tasks.workunit.client.0.smithi001.stdout:0/398: creat d18/d22/d57/f7d x:0 0 0 2024-02-28T16:31:07.484 INFO:tasks.workunit.client.0.smithi001.stdout:0/399: getdents d18/d22/d2b 0 2024-02-28T16:31:07.486 INFO:tasks.workunit.client.0.smithi001.stdout:0/400: chown d18/f33 10655669 1 2024-02-28T16:31:07.491 INFO:tasks.workunit.client.0.smithi001.stdout:0/401: mkdir d18/d22/d57/d7e 0 2024-02-28T16:31:07.491 INFO:tasks.workunit.client.0.smithi001.stdout:9/499: dwrite d5/d27/f73 [0,4194304] 0 2024-02-28T16:31:07.495 INFO:tasks.workunit.client.0.smithi001.stdout:0/402: write d18/d22/d3d/f67 [251337,16667] 0 2024-02-28T16:31:07.495 INFO:tasks.workunit.client.0.smithi001.stdout:8/405: dwrite d4/d32/f46 [0,4194304] 0 2024-02-28T16:31:07.518 INFO:tasks.workunit.client.0.smithi001.stdout:9/500: dwrite d5/d27/d42/f7e [0,4194304] 0 2024-02-28T16:31:07.524 INFO:tasks.workunit.client.0.smithi001.stdout:9/501: chown d5/d45/f4a 217389656 1 2024-02-28T16:31:07.527 INFO:tasks.workunit.client.0.smithi001.stdout:0/403: dwrite f15 [0,4194304] 0 2024-02-28T16:31:07.528 INFO:tasks.workunit.client.0.smithi001.stdout:9/502: readlink d5/d27/d42/l66 0 2024-02-28T16:31:07.531 INFO:tasks.workunit.client.0.smithi001.stdout:0/404: readlink d18/d22/l3e 0 2024-02-28T16:31:07.532 INFO:tasks.workunit.client.0.smithi001.stdout:8/406: dwrite d4/d32/f7c [0,4194304] 0 2024-02-28T16:31:07.533 INFO:tasks.workunit.client.0.smithi001.stdout:8/407: chown d4/f3b 848 1 2024-02-28T16:31:07.534 INFO:tasks.workunit.client.0.smithi001.stdout:0/405: write d18/d31/d4f/f60 [3323121,71866] 0 2024-02-28T16:31:07.540 INFO:tasks.workunit.client.0.smithi001.stdout:8/408: rename d4/d7/f19 to d4/d7/d22/d27/d4b/d77/f83 0 2024-02-28T16:31:07.542 INFO:tasks.workunit.client.0.smithi001.stdout:0/406: read d18/d22/d2b/d36/f56 [6513362,64789] 0 2024-02-28T16:31:07.545 INFO:tasks.workunit.client.0.smithi001.stdout:8/409: mkdir d4/d7/d22/d27/d4b/d84 0 2024-02-28T16:31:07.548 INFO:tasks.workunit.client.0.smithi001.stdout:0/407: creat d18/d31/f7f x:0 0 0 2024-02-28T16:31:07.549 INFO:tasks.workunit.client.0.smithi001.stdout:8/410: dread - d4/d7/d22/d27/d58/d62/f7e zero size 2024-02-28T16:31:07.551 INFO:tasks.workunit.client.0.smithi001.stdout:9/503: dwrite d5/d45/f71 [0,4194304] 0 2024-02-28T16:31:07.553 INFO:tasks.workunit.client.0.smithi001.stdout:9/504: chown d5/fe 122031 1 2024-02-28T16:31:07.554 INFO:tasks.workunit.client.0.smithi001.stdout:0/408: symlink d18/d22/d2b/d36/d4d/l80 0 2024-02-28T16:31:07.558 INFO:tasks.workunit.client.0.smithi001.stdout:0/409: symlink d18/d22/d57/l81 0 2024-02-28T16:31:07.560 INFO:tasks.workunit.client.0.smithi001.stdout:8/411: fsync d4/d7/d22/d27/f2f 0 2024-02-28T16:31:07.564 INFO:tasks.workunit.client.0.smithi001.stdout:0/410: fdatasync d18/d31/f71 0 2024-02-28T16:31:07.568 INFO:tasks.workunit.client.0.smithi001.stdout:0/411: getdents d18/d22/d2f 0 2024-02-28T16:31:07.573 INFO:tasks.workunit.client.0.smithi001.stdout:0/412: link d18/d22/d2b/d36/f56 d18/d31/d4f/f82 0 2024-02-28T16:31:07.573 INFO:tasks.workunit.client.0.smithi001.stdout:0/413: chown f13 47737942 1 2024-02-28T16:31:07.576 INFO:tasks.workunit.client.0.smithi001.stdout:9/505: truncate d5/d27/d34/f8d 3958658 0 2024-02-28T16:31:07.577 INFO:tasks.workunit.client.0.smithi001.stdout:9/506: dread - d5/d45/f4a zero size 2024-02-28T16:31:07.579 INFO:tasks.workunit.client.0.smithi001.stdout:9/507: chown d5/d47/d33/c39 6594 1 2024-02-28T16:31:07.582 INFO:tasks.workunit.client.0.smithi001.stdout:9/508: write d5/f12 [1806281,110010] 0 2024-02-28T16:31:07.583 INFO:tasks.workunit.client.0.smithi001.stdout:5/398: sync 2024-02-28T16:31:07.583 INFO:tasks.workunit.client.0.smithi001.stdout:2/466: sync 2024-02-28T16:31:07.583 INFO:tasks.workunit.client.0.smithi001.stdout:1/539: sync 2024-02-28T16:31:07.583 INFO:tasks.workunit.client.0.smithi001.stdout:4/401: sync 2024-02-28T16:31:07.583 INFO:tasks.workunit.client.0.smithi001.stdout:7/418: sync 2024-02-28T16:31:07.583 INFO:tasks.workunit.client.0.smithi001.stdout:6/413: sync 2024-02-28T16:31:07.585 INFO:tasks.workunit.client.0.smithi001.stdout:5/399: stat dc/d12/f32 0 2024-02-28T16:31:07.586 INFO:tasks.workunit.client.0.smithi001.stdout:9/509: fsync d5/d47/d33/f5d 0 2024-02-28T16:31:07.587 INFO:tasks.workunit.client.0.smithi001.stdout:7/419: write d3/f1e [5037436,5460] 0 2024-02-28T16:31:07.589 INFO:tasks.workunit.client.0.smithi001.stdout:6/414: write d9/db/d10/f16 [2977319,124414] 0 2024-02-28T16:31:07.590 INFO:tasks.workunit.client.0.smithi001.stdout:2/467: symlink d0/l9c 0 2024-02-28T16:31:07.591 INFO:tasks.workunit.client.0.smithi001.stdout:4/402: mkdir d3/d2d/d83 0 2024-02-28T16:31:07.594 INFO:tasks.workunit.client.0.smithi001.stdout:6/415: write d9/db/f1b [1266997,85528] 0 2024-02-28T16:31:07.595 INFO:tasks.workunit.client.0.smithi001.stdout:7/420: mknod d3/d17/d52/c89 0 2024-02-28T16:31:07.596 INFO:tasks.workunit.client.0.smithi001.stdout:5/400: rename dc/df/c7c to dc/d12/c84 0 2024-02-28T16:31:07.599 INFO:tasks.workunit.client.0.smithi001.stdout:8/412: dread d4/d7/d22/d27/d4b/d77/f83 [0,4194304] 0 2024-02-28T16:31:07.602 INFO:tasks.workunit.client.0.smithi001.stdout:0/414: dread d18/d22/d2b/f37 [0,4194304] 0 2024-02-28T16:31:07.602 INFO:tasks.workunit.client.0.smithi001.stdout:5/401: fsync dc/d1a/d21/d2d/d3c/f4f 0 2024-02-28T16:31:07.604 INFO:tasks.workunit.client.0.smithi001.stdout:5/402: chown dc/d12/f32 16 1 2024-02-28T16:31:07.607 INFO:tasks.workunit.client.0.smithi001.stdout:0/415: read d18/d22/d2f/d1c/f21 [794348,89167] 0 2024-02-28T16:31:07.609 INFO:tasks.workunit.client.0.smithi001.stdout:1/540: rename d0/d9/f54 to d0/d9/d1a/d2b/fba 0 2024-02-28T16:31:07.610 INFO:tasks.workunit.client.0.smithi001.stdout:6/416: mkdir d9/db/d5f/d89/d8a 0 2024-02-28T16:31:07.610 INFO:tasks.workunit.client.0.smithi001.stdout:0/416: readlink d18/d22/d57/l5d 0 2024-02-28T16:31:07.610 INFO:tasks.workunit.client.0.smithi001.stdout:4/403: dread d3/d5/d7/f21 [0,4194304] 0 2024-02-28T16:31:07.614 INFO:tasks.workunit.client.0.smithi001.stdout:9/510: rename d5/d27/f2c to d5/d99/fac 0 2024-02-28T16:31:07.615 INFO:tasks.workunit.client.0.smithi001.stdout:4/404: dread - d3/f78 zero size 2024-02-28T16:31:07.617 INFO:tasks.workunit.client.0.smithi001.stdout:3/411: sync 2024-02-28T16:31:07.621 INFO:tasks.workunit.client.0.smithi001.stdout:9/511: chown d5/d45/d6f/c82 141803451 1 2024-02-28T16:31:07.621 INFO:tasks.workunit.client.0.smithi001.stdout:0/417: unlink d18/d22/l40 0 2024-02-28T16:31:07.623 INFO:tasks.workunit.client.0.smithi001.stdout:2/468: rename d0/d2/d3/de/d61/d12/d46/l49 to d0/d2/d3/d36/l9d 0 2024-02-28T16:31:07.629 INFO:tasks.workunit.client.0.smithi001.stdout:8/413: dwrite d4/f3b [0,4194304] 0 2024-02-28T16:31:07.630 INFO:tasks.workunit.client.0.smithi001.stdout:1/541: truncate d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/f86 526590 0 2024-02-28T16:31:07.634 INFO:tasks.workunit.client.0.smithi001.stdout:6/417: rename d9/d29/d3e to d9/db/d5f/d8b 0 2024-02-28T16:31:07.640 INFO:tasks.workunit.client.0.smithi001.stdout:2/469: symlink d0/d2/d3/d36/d94/l9e 0 2024-02-28T16:31:07.645 INFO:tasks.workunit.client.0.smithi001.stdout:7/421: dwrite d3/fa [4194304,4194304] 0 2024-02-28T16:31:07.645 INFO:tasks.workunit.client.0.smithi001.stdout:5/403: dread dc/d1a/d21/f4b [0,4194304] 0 2024-02-28T16:31:07.657 INFO:tasks.workunit.client.0.smithi001.stdout:1/542: truncate d0/d9/d1a/d2b/f6d 1873831 0 2024-02-28T16:31:07.657 INFO:tasks.workunit.client.0.smithi001.stdout:7/422: chown d3/f36 13014 1 2024-02-28T16:31:07.657 INFO:tasks.workunit.client.0.smithi001.stdout:4/405: write d3/d1c/d58/f28 [2614845,85820] 0 2024-02-28T16:31:07.661 INFO:tasks.workunit.client.0.smithi001.stdout:5/404: stat dc/d12/d41/l45 0 2024-02-28T16:31:07.663 INFO:tasks.workunit.client.0.smithi001.stdout:3/412: rename d4/d6/d7/d23/f2b to d4/d6/d7/f88 0 2024-02-28T16:31:07.663 INFO:tasks.workunit.client.0.smithi001.stdout:0/418: dwrite d18/d22/d3d/f5e [0,4194304] 0 2024-02-28T16:31:07.665 INFO:tasks.workunit.client.0.smithi001.stdout:7/423: creat d3/d1c/d41/d49/d63/f8a x:0 0 0 2024-02-28T16:31:07.670 INFO:tasks.workunit.client.0.smithi001.stdout:9/512: truncate d5/d45/f71 1727835 0 2024-02-28T16:31:07.674 INFO:tasks.workunit.client.0.smithi001.stdout:1/543: mknod d0/d9/d1a/d72/d9d/cbb 0 2024-02-28T16:31:07.676 INFO:tasks.workunit.client.0.smithi001.stdout:8/414: truncate d4/d32/f46 2366082 0 2024-02-28T16:31:07.677 INFO:tasks.workunit.client.0.smithi001.stdout:6/418: dwrite d9/db/d5f/d8b/f6d [0,4194304] 0 2024-02-28T16:31:07.678 INFO:tasks.workunit.client.0.smithi001.stdout:4/406: symlink d3/d5/d7/d1f/d2e/l84 0 2024-02-28T16:31:07.687 INFO:tasks.workunit.client.0.smithi001.stdout:7/424: symlink d3/d17/d52/l8b 0 2024-02-28T16:31:07.688 INFO:tasks.workunit.client.0.smithi001.stdout:2/470: dwrite d0/d2/d3/de/d8a/f70 [0,4194304] 0 2024-02-28T16:31:07.694 INFO:tasks.workunit.client.0.smithi001.stdout:3/413: symlink d4/d1e/d7c/d7d/l89 0 2024-02-28T16:31:07.698 INFO:tasks.workunit.client.0.smithi001.stdout:4/407: dread d3/d1c/f4a [0,4194304] 0 2024-02-28T16:31:07.704 INFO:tasks.workunit.client.0.smithi001.stdout:5/405: link dc/d1a/d21/f48 dc/d1a/d21/d2d/d3c/d4a/f85 0 2024-02-28T16:31:07.704 INFO:tasks.workunit.client.0.smithi001.stdout:1/544: creat d0/d9/d1a/d2b/d26/d27/d30/d88/fbc x:0 0 0 2024-02-28T16:31:07.706 INFO:tasks.workunit.client.0.smithi001.stdout:0/419: mkdir d18/d22/d2b/d36/d83 0 2024-02-28T16:31:07.710 INFO:tasks.workunit.client.0.smithi001.stdout:1/545: readlink d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/dae/db1/lb4 0 2024-02-28T16:31:07.710 INFO:tasks.workunit.client.0.smithi001.stdout:9/513: dread d5/d99/fac [0,4194304] 0 2024-02-28T16:31:07.711 INFO:tasks.workunit.client.0.smithi001.stdout:3/414: mkdir d4/d6/d7/d78/d81/d8a 0 2024-02-28T16:31:07.714 INFO:tasks.workunit.client.0.smithi001.stdout:8/415: creat d4/d7/d22/d27/d4b/d80/f85 x:0 0 0 2024-02-28T16:31:07.716 INFO:tasks.workunit.client.0.smithi001.stdout:9/514: write d5/f3c [5002962,60081] 0 2024-02-28T16:31:07.721 INFO:tasks.workunit.client.0.smithi001.stdout:4/408: dwrite d3/d2d/f63 [0,4194304] 0 2024-02-28T16:31:07.722 INFO:tasks.workunit.client.0.smithi001.stdout:7/425: creat d3/f8c x:0 0 0 2024-02-28T16:31:07.722 INFO:tasks.workunit.client.0.smithi001.stdout:8/416: read d4/d7/d22/d27/d4b/d77/f83 [3310761,2143] 0 2024-02-28T16:31:07.725 INFO:tasks.workunit.client.0.smithi001.stdout:5/406: mknod dc/c86 0 2024-02-28T16:31:07.731 INFO:tasks.workunit.client.0.smithi001.stdout:2/471: link d0/d2/d3/d73/c93 d0/d2/d3/de/d22/c9f 0 2024-02-28T16:31:07.739 INFO:tasks.workunit.client.0.smithi001.stdout:9/515: mknod d5/d4d/cad 0 2024-02-28T16:31:07.740 INFO:tasks.workunit.client.0.smithi001.stdout:6/419: getdents d9/db/d5f/d89 0 2024-02-28T16:31:07.744 INFO:tasks.workunit.client.0.smithi001.stdout:3/415: dwrite d4/f76 [4194304,4194304] 0 2024-02-28T16:31:07.744 INFO:tasks.workunit.client.0.smithi001.stdout:4/409: mknod d3/d2d/d4c/d69/c85 0 2024-02-28T16:31:07.748 INFO:tasks.workunit.client.0.smithi001.stdout:6/420: read d9/d39/f3a [5509,106167] 0 2024-02-28T16:31:07.749 INFO:tasks.workunit.client.0.smithi001.stdout:2/472: unlink d0/d2/d3/de/d22/f74 0 2024-02-28T16:31:07.751 INFO:tasks.workunit.client.0.smithi001.stdout:5/407: mkdir dc/d1a/d7d/d87 0 2024-02-28T16:31:07.752 INFO:tasks.workunit.client.0.smithi001.stdout:6/421: dread - d9/db/d5f/d62/f71 zero size 2024-02-28T16:31:07.753 INFO:tasks.workunit.client.0.smithi001.stdout:9/516: creat d5/d47/d65/fae x:0 0 0 2024-02-28T16:31:07.756 INFO:tasks.workunit.client.0.smithi001.stdout:3/416: mknod d4/d6/d7/d23/c8b 0 2024-02-28T16:31:07.757 INFO:tasks.workunit.client.0.smithi001.stdout:5/408: write dc/d1a/d21/d2d/d3c/d4a/f59 [303784,29987] 0 2024-02-28T16:31:07.757 INFO:tasks.workunit.client.0.smithi001.stdout:4/410: creat d3/d5/d7/d1f/d2e/f86 x:0 0 0 2024-02-28T16:31:07.758 INFO:tasks.workunit.client.0.smithi001.stdout:0/420: getdents d18/d22/d2b/d36/d46 0 2024-02-28T16:31:07.759 INFO:tasks.workunit.client.0.smithi001.stdout:1/546: dread d0/d9/d1a/d2b/f4d [0,4194304] 0 2024-02-28T16:31:07.761 INFO:tasks.workunit.client.0.smithi001.stdout:3/417: readlink d4/d6/d7/d23/l54 0 2024-02-28T16:31:07.763 INFO:tasks.workunit.client.0.smithi001.stdout:1/547: write d0/d9/d1a/d2b/f17 [187036,107941] 0 2024-02-28T16:31:07.765 INFO:tasks.workunit.client.0.smithi001.stdout:7/426: getdents d3/d1c/d41 0 2024-02-28T16:31:07.766 INFO:tasks.workunit.client.0.smithi001.stdout:2/473: symlink d0/d2/d3/la0 0 2024-02-28T16:31:07.767 INFO:tasks.workunit.client.0.smithi001.stdout:1/548: truncate d0/d9/d56/f7f 1932622 0 2024-02-28T16:31:07.770 INFO:tasks.workunit.client.0.smithi001.stdout:4/411: mkdir d3/d5/d7/d1f/d3a/d68/d87 0 2024-02-28T16:31:07.770 INFO:tasks.workunit.client.0.smithi001.stdout:5/409: creat dc/d1a/d7d/f88 x:0 0 0 2024-02-28T16:31:07.773 INFO:tasks.workunit.client.0.smithi001.stdout:6/422: link d9/d29/d6e/f31 d9/f8c 0 2024-02-28T16:31:07.774 INFO:tasks.workunit.client.0.smithi001.stdout:0/421: dread d18/d22/d2f/d1c/f38 [0,4194304] 0 2024-02-28T16:31:07.778 INFO:tasks.workunit.client.0.smithi001.stdout:1/549: write d0/d9/fa [356409,36257] 0 2024-02-28T16:31:07.779 INFO:tasks.workunit.client.0.smithi001.stdout:3/418: truncate d4/d6/f40 1249860 0 2024-02-28T16:31:07.781 INFO:tasks.workunit.client.0.smithi001.stdout:0/422: chown d18/d22/d2f/d1c/l20 49360 1 2024-02-28T16:31:07.785 INFO:tasks.workunit.client.0.smithi001.stdout:4/412: getdents d3/d5/d47 0 2024-02-28T16:31:07.785 INFO:tasks.workunit.client.0.smithi001.stdout:1/550: chown d0/d9/d1a/d72/c77 189 1 2024-02-28T16:31:07.785 INFO:tasks.workunit.client.0.smithi001.stdout:5/410: rmdir dc/d1a/d21/d2d/d3c/d4a/d6c 0 2024-02-28T16:31:07.786 INFO:tasks.workunit.client.0.smithi001.stdout:0/423: dread d18/d5a/f73 [0,4194304] 0 2024-02-28T16:31:07.788 INFO:tasks.workunit.client.0.smithi001.stdout:2/474: link d0/d2/d3/c42 d0/ca1 0 2024-02-28T16:31:07.789 INFO:tasks.workunit.client.0.smithi001.stdout:9/517: dwrite d5/d47/f93 [0,4194304] 0 2024-02-28T16:31:07.789 INFO:tasks.workunit.client.0.smithi001.stdout:3/419: rename d4/d38/c5e to d4/d1e/c8c 0 2024-02-28T16:31:07.793 INFO:tasks.workunit.client.0.smithi001.stdout:9/518: chown d5/d47/f6c 87 1 2024-02-28T16:31:07.798 INFO:tasks.workunit.client.0.smithi001.stdout:1/551: mknod d0/d9/d1a/d2b/d26/cbd 0 2024-02-28T16:31:07.801 INFO:tasks.workunit.client.0.smithi001.stdout:4/413: mkdir d3/d1c/d58/d88 0 2024-02-28T16:31:07.803 INFO:tasks.workunit.client.0.smithi001.stdout:9/519: creat d5/d99/faf x:0 0 0 2024-02-28T16:31:07.803 INFO:tasks.workunit.client.0.smithi001.stdout:2/475: mknod d0/d2/d3/ca2 0 2024-02-28T16:31:07.805 INFO:tasks.workunit.client.0.smithi001.stdout:0/424: mknod d18/d22/d57/c84 0 2024-02-28T16:31:07.809 INFO:tasks.workunit.client.0.smithi001.stdout:4/414: symlink d3/d2d/d83/l89 0 2024-02-28T16:31:07.814 INFO:tasks.workunit.client.0.smithi001.stdout:1/552: creat d0/d9/d1a/fbe x:0 0 0 2024-02-28T16:31:07.818 INFO:tasks.workunit.client.0.smithi001.stdout:9/520: creat d5/d27/fb0 x:0 0 0 2024-02-28T16:31:07.818 INFO:tasks.workunit.client.0.smithi001.stdout:0/425: creat d18/d22/d2b/d5f/f85 x:0 0 0 2024-02-28T16:31:07.823 INFO:tasks.workunit.client.0.smithi001.stdout:6/423: dread d9/f4f [0,4194304] 0 2024-02-28T16:31:07.824 INFO:tasks.workunit.client.0.smithi001.stdout:3/420: dwrite d4/d6/f1c [0,4194304] 0 2024-02-28T16:31:07.830 INFO:tasks.workunit.client.0.smithi001.stdout:2/476: mknod d0/d2/d3/de/d61/d12/ca3 0 2024-02-28T16:31:07.830 INFO:tasks.workunit.client.0.smithi001.stdout:0/426: write d18/d22/d2f/d1c/f30 [2266226,105879] 0 2024-02-28T16:31:07.831 INFO:tasks.workunit.client.0.smithi001.stdout:4/415: rename d3/f1b to d3/d1c/d58/d88/f8a 0 2024-02-28T16:31:07.832 INFO:tasks.workunit.client.0.smithi001.stdout:1/553: write d0/d9/d1a/d2b/d26/d27/d30/d57/fa0 [590391,117053] 0 2024-02-28T16:31:07.833 INFO:tasks.workunit.client.0.smithi001.stdout:0/427: write d18/d22/d2b/d36/d4d/f65 [640012,117625] 0 2024-02-28T16:31:07.835 INFO:tasks.workunit.client.0.smithi001.stdout:5/411: dwrite dc/d1a/d21/d2d/d3c/f47 [0,4194304] 0 2024-02-28T16:31:07.836 INFO:tasks.workunit.client.0.smithi001.stdout:7/427: dread d3/d17/f19 [0,4194304] 0 2024-02-28T16:31:07.843 INFO:tasks.workunit.client.0.smithi001.stdout:9/521: rename d5/d47 to d5/d27/db1 0 2024-02-28T16:31:07.852 INFO:tasks.workunit.client.0.smithi001.stdout:3/421: dwrite d4/d6/d7/d23/f7a [0,4194304] 0 2024-02-28T16:31:07.859 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:07 smithi001 ceph-mon[28991]: pgmap v47: 65 pgs: 65 active+clean; 1.0 GiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 20 MiB/s rd, 110 MiB/s wr, 252 op/s 2024-02-28T16:31:07.861 INFO:tasks.workunit.client.0.smithi001.stdout:1/554: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d57/fa0 0 2024-02-28T16:31:07.865 INFO:tasks.workunit.client.0.smithi001.stdout:0/428: creat d18/d22/d47/f86 x:0 0 0 2024-02-28T16:31:07.867 INFO:tasks.workunit.client.0.smithi001.stdout:0/429: write d18/d31/d4f/f60 [3316322,27113] 0 2024-02-28T16:31:07.870 INFO:tasks.workunit.client.0.smithi001.stdout:6/424: dwrite d9/d29/f44 [4194304,4194304] 0 2024-02-28T16:31:07.871 INFO:tasks.workunit.client.0.smithi001.stdout:5/412: fdatasync dc/d1a/d21/d2d/d3c/f3d 0 2024-02-28T16:31:07.877 INFO:tasks.workunit.client.0.smithi001.stdout:1/555: unlink d0/d9/d1a/d2b/d26/l62 0 2024-02-28T16:31:07.877 INFO:tasks.workunit.client.0.smithi001.stdout:0/430: read d18/d22/d2f/d1c/f21 [263901,128395] 0 2024-02-28T16:31:07.879 INFO:tasks.workunit.client.0.smithi001.stdout:6/425: write d9/db/f45 [2632985,81897] 0 2024-02-28T16:31:07.886 INFO:tasks.workunit.client.0.smithi001.stdout:4/416: getdents d3/d5/d7 0 2024-02-28T16:31:07.892 INFO:tasks.workunit.client.0.smithi001.stdout:1/556: symlink d0/d9/d1a/d2b/d26/d27/d30/d57/lbf 0 2024-02-28T16:31:07.892 INFO:tasks.workunit.client.0.smithi001.stdout:7/428: getdents d3/d17/d3d 0 2024-02-28T16:31:07.892 INFO:tasks.workunit.client.0.smithi001.stdout:6/426: mknod d9/db/d19/c8d 0 2024-02-28T16:31:07.895 INFO:tasks.workunit.client.0.smithi001.stdout:6/427: chown d9/db/d5f/d69 758 1 2024-02-28T16:31:07.895 INFO:tasks.workunit.client.0.smithi001.stdout:1/557: dread - d0/fa9 zero size 2024-02-28T16:31:07.906 INFO:tasks.workunit.client.0.smithi001.stdout:1/558: unlink d0/d9/d1a/d72/d9d/d9c/cab 0 2024-02-28T16:31:07.909 INFO:tasks.workunit.client.0.smithi001.stdout:7/429: write d3/d17/d3d/d5d/f73 [3990,60609] 0 2024-02-28T16:31:07.910 INFO:tasks.workunit.client.0.smithi001.stdout:5/413: dwrite dc/f27 [0,4194304] 0 2024-02-28T16:31:07.915 INFO:tasks.workunit.client.0.smithi001.stdout:3/422: write d4/d6/f40 [1517889,2475] 0 2024-02-28T16:31:07.916 INFO:tasks.workunit.client.0.smithi001.stdout:9/522: dread d5/f18 [0,4194304] 0 2024-02-28T16:31:07.917 INFO:tasks.workunit.client.0.smithi001.stdout:1/559: rmdir d0/d9/d1a/d2b/d26/d27/d30 39 2024-02-28T16:31:07.918 INFO:tasks.workunit.client.0.smithi001.stdout:0/431: dwrite d18/d22/d3d/f5e [0,4194304] 0 2024-02-28T16:31:07.927 INFO:tasks.workunit.client.0.smithi001.stdout:7/430: write d3/d1c/d41/d49/f65 [3992844,67056] 0 2024-02-28T16:31:07.930 INFO:tasks.workunit.client.0.smithi001.stdout:6/428: dwrite d9/db/d5f/d69/f75 [0,4194304] 0 2024-02-28T16:31:07.930 INFO:tasks.workunit.client.0.smithi001.stdout:6/429: fsync d9/d29/d6a/f77 0 2024-02-28T16:31:07.930 INFO:tasks.workunit.client.0.smithi001.stdout:3/423: rename d4/d6/d7/d23/l75 to d4/d18/l8d 0 2024-02-28T16:31:07.931 INFO:tasks.workunit.client.0.smithi001.stdout:9/523: mkdir d5/d27/db1/da6/db2 0 2024-02-28T16:31:07.932 INFO:tasks.workunit.client.0.smithi001.stdout:6/430: readlink d9/db/d19/l41 0 2024-02-28T16:31:07.933 INFO:tasks.workunit.client.0.smithi001.stdout:9/524: chown d5/d27/c59 2425476 1 2024-02-28T16:31:07.933 INFO:tasks.workunit.client.0.smithi001.stdout:7/431: chown d3/d17/c25 45 1 2024-02-28T16:31:07.935 INFO:tasks.workunit.client.0.smithi001.stdout:3/424: symlink d4/d1e/d7c/l8e 0 2024-02-28T16:31:07.938 INFO:tasks.workunit.client.0.smithi001.stdout:3/425: chown d4/d1e/c3d 14576 1 2024-02-28T16:31:07.938 INFO:tasks.workunit.client.0.smithi001.stdout:0/432: read d18/d22/d2f/d1c/f21 [3540902,17392] 0 2024-02-28T16:31:07.939 INFO:tasks.workunit.client.0.smithi001.stdout:9/525: mkdir d5/d45/d6f/db3 0 2024-02-28T16:31:07.941 INFO:tasks.workunit.client.0.smithi001.stdout:9/526: chown d5/f6 22 1 2024-02-28T16:31:07.949 INFO:tasks.workunit.client.0.smithi001.stdout:7/432: rmdir d3/d17/d3d/d59/d6e/d80 0 2024-02-28T16:31:07.951 INFO:tasks.workunit.client.0.smithi001.stdout:0/433: rename d18/d22/d57/f7d to d18/d22/d2b/f87 0 2024-02-28T16:31:07.952 INFO:tasks.workunit.client.0.smithi001.stdout:7/433: stat d3/d1c/l3c 0 2024-02-28T16:31:07.952 INFO:tasks.workunit.client.0.smithi001.stdout:1/560: dwrite d0/d9/d1a/f33 [0,4194304] 0 2024-02-28T16:31:07.960 INFO:tasks.workunit.client.0.smithi001.stdout:7/434: mkdir d3/d1c/d41/d49/d8d 0 2024-02-28T16:31:07.963 INFO:tasks.workunit.client.0.smithi001.stdout:4/417: dread d3/d1c/d58/f27 [0,4194304] 0 2024-02-28T16:31:07.964 INFO:tasks.workunit.client.0.smithi001.stdout:3/426: getdents d4/d6/d7 0 2024-02-28T16:31:07.966 INFO:tasks.workunit.client.0.smithi001.stdout:6/431: dwrite d9/d29/d6a/f85 [0,4194304] 0 2024-02-28T16:31:07.966 INFO:tasks.workunit.client.0.smithi001.stdout:5/414: dread f5 [0,4194304] 0 2024-02-28T16:31:07.972 INFO:tasks.workunit.client.0.smithi001.stdout:9/527: dwrite d5/f15 [0,4194304] 0 2024-02-28T16:31:07.978 INFO:tasks.workunit.client.0.smithi001.stdout:6/432: creat d9/db/d19/f8e x:0 0 0 2024-02-28T16:31:07.979 INFO:tasks.workunit.client.0.smithi001.stdout:1/561: creat d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/dae/fc0 x:0 0 0 2024-02-28T16:31:07.979 INFO:tasks.workunit.client.0.smithi001.stdout:5/415: creat dc/d1a/d52/f89 x:0 0 0 2024-02-28T16:31:07.980 INFO:tasks.workunit.client.0.smithi001.stdout:3/427: write d4/d1e/f22 [519890,58721] 0 2024-02-28T16:31:07.980 INFO:tasks.workunit.client.0.smithi001.stdout:7/435: dwrite d3/d17/f32 [8388608,4194304] 0 2024-02-28T16:31:07.980 INFO:tasks.workunit.client.0.smithi001.stdout:0/434: link f11 d18/d22/d2b/f88 0 2024-02-28T16:31:07.981 INFO:tasks.workunit.client.0.smithi001.stdout:0/435: fdatasync fd 0 2024-02-28T16:31:07.983 INFO:tasks.workunit.client.0.smithi001.stdout:6/433: dread - d9/de/f66 zero size 2024-02-28T16:31:07.984 INFO:tasks.workunit.client.0.smithi001.stdout:5/416: chown dc/l1c 160085 1 2024-02-28T16:31:07.986 INFO:tasks.workunit.client.0.smithi001.stdout:3/428: write d4/d1e/f33 [3909213,50022] 0 2024-02-28T16:31:07.988 INFO:tasks.workunit.client.0.smithi001.stdout:9/528: mknod d5/d27/d42/d48/cb4 0 2024-02-28T16:31:07.988 INFO:tasks.workunit.client.0.smithi001.stdout:7/436: write d3/d1c/d41/f6d [1816820,28892] 0 2024-02-28T16:31:07.990 INFO:tasks.workunit.client.0.smithi001.stdout:4/418: creat d3/d5/d7/d1f/d3a/d68/d87/f8b x:0 0 0 2024-02-28T16:31:07.992 INFO:tasks.workunit.client.0.smithi001.stdout:5/417: write dc/d1a/d21/d2d/d3c/f4f [3904346,29459] 0 2024-02-28T16:31:07.994 INFO:tasks.workunit.client.0.smithi001.stdout:7/437: read d3/d1c/f51 [316505,52634] 0 2024-02-28T16:31:07.996 INFO:tasks.workunit.client.0.smithi001.stdout:1/562: creat d0/d9/d1a/d2b/d26/d27/d30/d67/fc1 x:0 0 0 2024-02-28T16:31:07.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:07 smithi066 ceph-mon[36236]: pgmap v47: 65 pgs: 65 active+clean; 1.0 GiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 20 MiB/s rd, 110 MiB/s wr, 252 op/s 2024-02-28T16:31:07.998 INFO:tasks.workunit.client.0.smithi001.stdout:4/419: unlink d3/d5/f41 0 2024-02-28T16:31:08.001 INFO:tasks.workunit.client.0.smithi001.stdout:7/438: mkdir d3/d17/d3d/d42/d8e 0 2024-02-28T16:31:08.004 INFO:tasks.workunit.client.0.smithi001.stdout:9/529: symlink d5/d27/db1/lb5 0 2024-02-28T16:31:08.004 INFO:tasks.workunit.client.0.smithi001.stdout:6/434: creat d9/db/d5f/d62/d83/f8f x:0 0 0 2024-02-28T16:31:08.011 INFO:tasks.workunit.client.0.smithi001.stdout:3/429: symlink d4/l8f 0 2024-02-28T16:31:08.011 INFO:tasks.workunit.client.0.smithi001.stdout:9/530: dread - d5/d27/db1/d33/f5d zero size 2024-02-28T16:31:08.011 INFO:tasks.workunit.client.0.smithi001.stdout:1/563: creat d0/d9/d56/fc2 x:0 0 0 2024-02-28T16:31:08.012 INFO:tasks.workunit.client.0.smithi001.stdout:7/439: mknod d3/d17/d52/d55/c8f 0 2024-02-28T16:31:08.013 INFO:tasks.workunit.client.0.smithi001.stdout:5/418: link lb dc/d1a/d21/d2d/l8a 0 2024-02-28T16:31:08.013 INFO:tasks.workunit.client.0.smithi001.stdout:6/435: creat d9/db/d19/f90 x:0 0 0 2024-02-28T16:31:08.014 INFO:tasks.workunit.client.0.smithi001.stdout:0/436: dwrite d18/d22/d2f/d1c/f30 [0,4194304] 0 2024-02-28T16:31:08.015 INFO:tasks.workunit.client.0.smithi001.stdout:6/436: write f8 [619096,47999] 0 2024-02-28T16:31:08.016 INFO:tasks.workunit.client.0.smithi001.stdout:7/440: write d3/fa [2313601,109721] 0 2024-02-28T16:31:08.017 INFO:tasks.workunit.client.0.smithi001.stdout:9/531: dread - d5/d27/db1/d58/f9b zero size 2024-02-28T16:31:08.018 INFO:tasks.workunit.client.0.smithi001.stdout:1/564: chown d0/d9/d1a/d2b/c12 358645 1 2024-02-28T16:31:08.018 INFO:tasks.workunit.client.0.smithi001.stdout:6/437: readlink l4 0 2024-02-28T16:31:08.020 INFO:tasks.workunit.client.0.smithi001.stdout:5/419: symlink dc/d1a/d21/l8b 0 2024-02-28T16:31:08.025 INFO:tasks.workunit.client.0.smithi001.stdout:9/532: fsync d5/d27/db1/f37 0 2024-02-28T16:31:08.026 INFO:tasks.workunit.client.0.smithi001.stdout:1/565: chown d0/d9/d1a/d2b/d26/d27/d30/d52/f5a 23474801 1 2024-02-28T16:31:08.027 INFO:tasks.workunit.client.0.smithi001.stdout:9/533: readlink d5/d27/d42/l66 0 2024-02-28T16:31:08.029 INFO:tasks.workunit.client.0.smithi001.stdout:7/441: getdents d3/d17/d3d/d59 0 2024-02-28T16:31:08.030 INFO:tasks.workunit.client.0.smithi001.stdout:1/566: chown d0/d9/d1a/d2b/fba 1777081010 1 2024-02-28T16:31:08.032 INFO:tasks.workunit.client.0.smithi001.stdout:9/534: creat d5/d27/db1/da6/db2/fb6 x:0 0 0 2024-02-28T16:31:08.035 INFO:tasks.workunit.client.0.smithi001.stdout:7/442: symlink d3/l90 0 2024-02-28T16:31:08.037 INFO:tasks.workunit.client.0.smithi001.stdout:9/535: write d5/d27/db1/f93 [1543334,17704] 0 2024-02-28T16:31:08.037 INFO:tasks.workunit.client.0.smithi001.stdout:1/567: chown d0/d9/d1a/d2b/d26/d27/d30/d57/fa0 3266 1 2024-02-28T16:31:08.038 INFO:tasks.workunit.client.0.smithi001.stdout:1/568: fdatasync d0/d9/fa 0 2024-02-28T16:31:08.041 INFO:tasks.workunit.client.0.smithi001.stdout:1/569: fsync d0/d9/d56/f7f 0 2024-02-28T16:31:08.043 INFO:tasks.workunit.client.0.smithi001.stdout:1/570: dread - d0/d9/d56/fc2 zero size 2024-02-28T16:31:08.044 INFO:tasks.workunit.client.0.smithi001.stdout:7/443: dread d3/d17/d3d/d42/f53 [0,4194304] 0 2024-02-28T16:31:08.046 INFO:tasks.workunit.client.0.smithi001.stdout:4/420: dread d3/d5/d7/d1f/d2e/f39 [0,4194304] 0 2024-02-28T16:31:08.049 INFO:tasks.workunit.client.0.smithi001.stdout:4/421: write d3/d1c/d58/ff [4964015,99722] 0 2024-02-28T16:31:08.049 INFO:tasks.workunit.client.0.smithi001.stdout:9/536: mknod d5/d45/d6f/db3/cb7 0 2024-02-28T16:31:08.057 INFO:tasks.workunit.client.0.smithi001.stdout:4/422: rmdir d3 39 2024-02-28T16:31:08.058 INFO:tasks.workunit.client.0.smithi001.stdout:1/571: symlink d0/d9/d1a/d2b/d26/d27/lc3 0 2024-02-28T16:31:08.062 INFO:tasks.workunit.client.0.smithi001.stdout:5/420: dread dc/d12/f3f [4194304,4194304] 0 2024-02-28T16:31:08.062 INFO:tasks.workunit.client.0.smithi001.stdout:9/537: creat d5/fb8 x:0 0 0 2024-02-28T16:31:08.064 INFO:tasks.workunit.client.0.smithi001.stdout:1/572: write d0/d9/d1a/d2b/d26/d27/d30/d52/d84/f94 [2304746,15036] 0 2024-02-28T16:31:08.064 INFO:tasks.workunit.client.0.smithi001.stdout:9/538: readlink d5/d27/db1/l5b 0 2024-02-28T16:31:08.064 INFO:tasks.workunit.client.0.smithi001.stdout:4/423: rename d3/d5/d4e/l60 to d3/d5/d47/l8c 0 2024-02-28T16:31:08.066 INFO:tasks.workunit.client.0.smithi001.stdout:1/573: dread - d0/d9/d56/d2f/f64 zero size 2024-02-28T16:31:08.068 INFO:tasks.workunit.client.0.smithi001.stdout:9/539: write d5/d27/db1/f91 [1216178,58971] 0 2024-02-28T16:31:08.070 INFO:tasks.workunit.client.0.smithi001.stdout:4/424: creat d3/d5/d7/d1f/f8d x:0 0 0 2024-02-28T16:31:08.071 INFO:tasks.workunit.client.0.smithi001.stdout:1/574: creat d0/d9/d56/d2f/fc4 x:0 0 0 2024-02-28T16:31:08.072 INFO:tasks.workunit.client.0.smithi001.stdout:9/540: creat d5/d27/fb9 x:0 0 0 2024-02-28T16:31:08.076 INFO:tasks.workunit.client.0.smithi001.stdout:4/425: creat d3/f8e x:0 0 0 2024-02-28T16:31:08.076 INFO:tasks.workunit.client.0.smithi001.stdout:9/541: mkdir d5/d9a/dba 0 2024-02-28T16:31:08.077 INFO:tasks.workunit.client.0.smithi001.stdout:7/444: dread d3/f10 [0,4194304] 0 2024-02-28T16:31:08.080 INFO:tasks.workunit.client.0.smithi001.stdout:4/426: write d3/d5/d7/d1f/d2e/f86 [440890,84678] 0 2024-02-28T16:31:08.081 INFO:tasks.workunit.client.0.smithi001.stdout:5/421: dwrite dc/df/f13 [0,4194304] 0 2024-02-28T16:31:08.082 INFO:tasks.workunit.client.0.smithi001.stdout:7/445: mknod d3/d17/d52/d7f/c91 0 2024-02-28T16:31:08.086 INFO:tasks.workunit.client.0.smithi001.stdout:4/427: mknod d3/d5/d7/d1f/d3a/c8f 0 2024-02-28T16:31:08.086 INFO:tasks.workunit.client.0.smithi001.stdout:1/575: link d0/d9/d1a/d2b/d26/d27/d30/d52/l78 d0/d9/d1a/d2b/d26/d27/d30/d67/d87/lc5 0 2024-02-28T16:31:08.088 INFO:tasks.workunit.client.0.smithi001.stdout:7/446: unlink d3/d17/c25 0 2024-02-28T16:31:08.091 INFO:tasks.workunit.client.0.smithi001.stdout:7/447: chown d3/f22 900 1 2024-02-28T16:31:08.095 INFO:tasks.workunit.client.0.smithi001.stdout:7/448: write d3/d17/f2d [4975896,47853] 0 2024-02-28T16:31:08.096 INFO:tasks.workunit.client.0.smithi001.stdout:1/576: rmdir d0/d9/d1a/d2b/d26/d27/d30/d57 39 2024-02-28T16:31:08.108 INFO:tasks.workunit.client.0.smithi001.stdout:4/428: dwrite d3/d5/d7/d1f/d2e/f39 [0,4194304] 0 2024-02-28T16:31:08.113 INFO:tasks.workunit.client.0.smithi001.stdout:1/577: creat d0/d9/d1a/d2b/d26/d27/d30/d57/d8e/fc6 x:0 0 0 2024-02-28T16:31:08.115 INFO:tasks.workunit.client.0.smithi001.stdout:7/449: getdents d3/d1c/d41/d49/d63 0 2024-02-28T16:31:08.116 INFO:tasks.workunit.client.0.smithi001.stdout:5/422: dread dc/d1a/d21/d2d/d3c/f4f [0,4194304] 0 2024-02-28T16:31:08.118 INFO:tasks.workunit.client.0.smithi001.stdout:9/542: dread d5/f1d [0,4194304] 0 2024-02-28T16:31:08.120 INFO:tasks.workunit.client.0.smithi001.stdout:9/543: truncate d5/fb8 367264 0 2024-02-28T16:31:08.120 INFO:tasks.workunit.client.0.smithi001.stdout:7/450: mkdir d3/d17/d52/d92 0 2024-02-28T16:31:08.122 INFO:tasks.workunit.client.0.smithi001.stdout:1/578: symlink d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/lc7 0 2024-02-28T16:31:08.123 INFO:tasks.workunit.client.0.smithi001.stdout:9/544: stat d5/d27/db1/d65/c96 0 2024-02-28T16:31:08.125 INFO:tasks.workunit.client.0.smithi001.stdout:4/429: getdents d3/d5/d7/d1f/d2e 0 2024-02-28T16:31:08.126 INFO:tasks.workunit.client.0.smithi001.stdout:5/423: symlink dc/d1a/d52/l8c 0 2024-02-28T16:31:08.133 INFO:tasks.workunit.client.0.smithi001.stdout:1/579: mknod d0/d9/d56/d2f/cc8 0 2024-02-28T16:31:08.145 INFO:tasks.workunit.client.0.smithi001.stdout:5/424: creat dc/d1a/f8d x:0 0 0 2024-02-28T16:31:08.145 INFO:tasks.workunit.client.0.smithi001.stdout:4/430: dwrite d3/d2d/d4c/f50 [0,4194304] 0 2024-02-28T16:31:08.147 INFO:tasks.workunit.client.0.smithi001.stdout:7/451: dwrite d3/f8 [0,4194304] 0 2024-02-28T16:31:08.149 INFO:tasks.workunit.client.0.smithi001.stdout:9/545: dwrite d5/f89 [0,4194304] 0 2024-02-28T16:31:08.149 INFO:tasks.workunit.client.0.smithi001.stdout:4/431: write d3/d5/d4e/f80 [774631,47822] 0 2024-02-28T16:31:08.152 INFO:tasks.workunit.client.0.smithi001.stdout:5/425: symlink dc/l8e 0 2024-02-28T16:31:08.152 INFO:tasks.workunit.client.0.smithi001.stdout:4/432: stat d3/fa 0 2024-02-28T16:31:08.154 INFO:tasks.workunit.client.0.smithi001.stdout:9/546: mknod d5/d27/db1/d65/cbb 0 2024-02-28T16:31:08.154 INFO:tasks.workunit.client.0.smithi001.stdout:5/426: dread - dc/d12/f6f zero size 2024-02-28T16:31:08.159 INFO:tasks.workunit.client.0.smithi001.stdout:4/433: unlink d3/d5/d4e/l61 0 2024-02-28T16:31:08.159 INFO:tasks.workunit.client.0.smithi001.stdout:8/417: sync 2024-02-28T16:31:08.159 INFO:tasks.workunit.client.0.smithi001.stdout:2/477: sync 2024-02-28T16:31:08.159 INFO:tasks.workunit.client.0.smithi001.stdout:9/547: mkdir d5/d27/d34/dbc 0 2024-02-28T16:31:08.161 INFO:tasks.workunit.client.0.smithi001.stdout:4/434: stat d3/l13 0 2024-02-28T16:31:08.162 INFO:tasks.workunit.client.0.smithi001.stdout:5/427: truncate dc/d12/d41/d54/f71 634538 0 2024-02-28T16:31:08.163 INFO:tasks.workunit.client.0.smithi001.stdout:1/580: dwrite d0/d9/d1a/d2b/d26/d27/d30/d57/d8e/f93 [0,4194304] 0 2024-02-28T16:31:08.166 INFO:tasks.workunit.client.0.smithi001.stdout:9/548: chown d5/d27/db1/f40 0 1 2024-02-28T16:31:08.172 INFO:tasks.workunit.client.0.smithi001.stdout:8/418: chown d4/d7/d22/d27/d4b/c57 1926 1 2024-02-28T16:31:08.174 INFO:tasks.workunit.client.0.smithi001.stdout:9/549: creat d5/d45/d6f/fbd x:0 0 0 2024-02-28T16:31:08.175 INFO:tasks.workunit.client.0.smithi001.stdout:5/428: write dc/d1a/d21/d2d/f66 [125275,61973] 0 2024-02-28T16:31:08.184 INFO:tasks.workunit.client.0.smithi001.stdout:4/435: mkdir d3/d5/d47/d90 0 2024-02-28T16:31:08.188 INFO:tasks.workunit.client.0.smithi001.stdout:5/429: fdatasync dc/d1a/d21/d2d/f42 0 2024-02-28T16:31:08.191 INFO:tasks.workunit.client.0.smithi001.stdout:4/436: fdatasync d3/d1c/f4a 0 2024-02-28T16:31:08.192 INFO:tasks.workunit.client.0.smithi001.stdout:9/550: creat d5/d99/fbe x:0 0 0 2024-02-28T16:31:08.194 INFO:tasks.workunit.client.0.smithi001.stdout:7/452: dwrite d3/d1c/d41/d49/d63/f8a [0,4194304] 0 2024-02-28T16:31:08.195 INFO:tasks.workunit.client.0.smithi001.stdout:5/430: write dc/d1a/d21/d2d/f76 [647512,68413] 0 2024-02-28T16:31:08.196 INFO:tasks.workunit.client.0.smithi001.stdout:1/581: dwrite d0/f4e [0,4194304] 0 2024-02-28T16:31:08.206 INFO:tasks.workunit.client.0.smithi001.stdout:9/551: getdents d5/d9a/dba 0 2024-02-28T16:31:08.214 INFO:tasks.workunit.client.0.smithi001.stdout:4/437: dread d3/d5/d7/d1f/d2e/f86 [0,4194304] 0 2024-02-28T16:31:08.220 INFO:tasks.workunit.client.0.smithi001.stdout:9/552: unlink d5/d27/d34/d50/f52 0 2024-02-28T16:31:08.220 INFO:tasks.workunit.client.0.smithi001.stdout:2/478: dwrite d0/d2/d3/de/d61/d12/f27 [0,4194304] 0 2024-02-28T16:31:08.222 INFO:tasks.workunit.client.0.smithi001.stdout:4/438: chown d3/d5/c5f 66659 1 2024-02-28T16:31:08.224 INFO:tasks.workunit.client.0.smithi001.stdout:8/419: dwrite d4/d7/d22/d27/f4a [4194304,4194304] 0 2024-02-28T16:31:08.231 INFO:tasks.workunit.client.0.smithi001.stdout:1/582: getdents d0/d9/d95 0 2024-02-28T16:31:08.232 INFO:tasks.workunit.client.0.smithi001.stdout:9/553: symlink d5/d27/db1/d65/lbf 0 2024-02-28T16:31:08.239 INFO:tasks.workunit.client.0.smithi001.stdout:8/420: write d4/d7/d22/d27/d4b/d80/f85 [63315,78719] 0 2024-02-28T16:31:08.246 INFO:tasks.workunit.client.0.smithi001.stdout:2/479: mknod d0/d2/d3/de/d61/d12/d46/ca4 0 2024-02-28T16:31:08.246 INFO:tasks.workunit.client.0.smithi001.stdout:4/439: write d3/d1c/d58/d6e/f7a [971925,36286] 0 2024-02-28T16:31:08.247 INFO:tasks.workunit.client.0.smithi001.stdout:4/440: stat d3/d1c/f4a 0 2024-02-28T16:31:08.250 INFO:tasks.workunit.client.0.smithi001.stdout:2/480: write d0/d2/d3/f54 [425083,25348] 0 2024-02-28T16:31:08.251 INFO:tasks.workunit.client.0.smithi001.stdout:8/421: rename d4/d7/c44 to d4/d47/c86 0 2024-02-28T16:31:08.255 INFO:tasks.workunit.client.0.smithi001.stdout:4/441: write d3/d5/d7/d1f/d2e/f86 [1291719,23173] 0 2024-02-28T16:31:08.259 INFO:tasks.workunit.client.0.smithi001.stdout:8/422: creat d4/d7/d22/d27/d58/f87 x:0 0 0 2024-02-28T16:31:08.261 INFO:tasks.workunit.client.0.smithi001.stdout:7/453: dread d3/d17/d3d/f72 [0,4194304] 0 2024-02-28T16:31:08.261 INFO:tasks.workunit.client.0.smithi001.stdout:8/423: chown d4/d7/d22/l4d 342580553 1 2024-02-28T16:31:08.261 INFO:tasks.workunit.client.0.smithi001.stdout:4/442: read d3/d2d/d4c/f51 [809991,85519] 0 2024-02-28T16:31:08.263 INFO:tasks.workunit.client.0.smithi001.stdout:5/431: dread dc/f20 [0,4194304] 0 2024-02-28T16:31:08.263 INFO:tasks.workunit.client.0.smithi001.stdout:1/583: truncate d0/d9/d1a/d2b/d26/d27/d30/f59 7041571 0 2024-02-28T16:31:08.264 INFO:tasks.workunit.client.0.smithi001.stdout:8/424: readlink d4/d47/l76 0 2024-02-28T16:31:08.265 INFO:tasks.workunit.client.0.smithi001.stdout:4/443: unlink d3/f17 0 2024-02-28T16:31:08.267 INFO:tasks.workunit.client.0.smithi001.stdout:7/454: mknod d3/c93 0 2024-02-28T16:31:08.269 INFO:tasks.workunit.client.0.smithi001.stdout:8/425: mknod d4/c88 0 2024-02-28T16:31:08.270 INFO:tasks.workunit.client.0.smithi001.stdout:8/426: chown d4/d7/d22/d27/d4b/d77 3 1 2024-02-28T16:31:08.273 INFO:tasks.workunit.client.0.smithi001.stdout:8/427: chown d4/d32/l34 29098 1 2024-02-28T16:31:08.274 INFO:tasks.workunit.client.0.smithi001.stdout:4/444: link d3/d5/f34 d3/d5/f91 0 2024-02-28T16:31:08.276 INFO:tasks.workunit.client.0.smithi001.stdout:8/428: chown d4/d7/l54 787786 1 2024-02-28T16:31:08.278 INFO:tasks.workunit.client.0.smithi001.stdout:4/445: fsync d3/d2d/d4c/d69/d82/f53 0 2024-02-28T16:31:08.280 INFO:tasks.workunit.client.0.smithi001.stdout:8/429: chown d4/d7/d22/c26 162414308 1 2024-02-28T16:31:08.282 INFO:tasks.workunit.client.0.smithi001.stdout:4/446: read d3/d5/d7/f21 [213313,2452] 0 2024-02-28T16:31:08.282 INFO:tasks.workunit.client.0.smithi001.stdout:1/584: write d0/d9/d56/f60 [358969,127926] 0 2024-02-28T16:31:08.285 INFO:tasks.workunit.client.0.smithi001.stdout:5/432: dwrite dc/d12/f6f [0,4194304] 0 2024-02-28T16:31:08.285 INFO:tasks.workunit.client.0.smithi001.stdout:7/455: truncate d3/f21 3846724 0 2024-02-28T16:31:08.290 INFO:tasks.workunit.client.0.smithi001.stdout:7/456: mkdir d3/d1c/d41/d49/d94 0 2024-02-28T16:31:08.296 INFO:tasks.workunit.client.0.smithi001.stdout:4/447: mkdir d3/d5/d92 0 2024-02-28T16:31:08.300 INFO:tasks.workunit.client.0.smithi001.stdout:5/433: creat dc/d1a/d7d/d87/f8f x:0 0 0 2024-02-28T16:31:08.304 INFO:tasks.workunit.client.0.smithi001.stdout:4/448: unlink d3/d5/d7/d1f/d2e/l75 0 2024-02-28T16:31:08.306 INFO:tasks.workunit.client.0.smithi001.stdout:8/430: dwrite d4/d7/d22/d27/d58/d62/f7e [0,4194304] 0 2024-02-28T16:31:08.307 INFO:tasks.workunit.client.0.smithi001.stdout:4/449: dread - d3/d2d/d4c/d69/d82/f77 zero size 2024-02-28T16:31:08.308 INFO:tasks.workunit.client.0.smithi001.stdout:5/434: creat dc/d12/d56/f90 x:0 0 0 2024-02-28T16:31:08.309 INFO:tasks.workunit.client.0.smithi001.stdout:8/431: chown d4/f52 2 1 2024-02-28T16:31:08.313 INFO:tasks.workunit.client.0.smithi001.stdout:5/435: truncate dc/d12/d41/d54/f6a 2405959 0 2024-02-28T16:31:08.314 INFO:tasks.workunit.client.0.smithi001.stdout:5/436: chown dc 3 1 2024-02-28T16:31:08.314 INFO:tasks.workunit.client.0.smithi001.stdout:7/457: dwrite d3/f36 [0,4194304] 0 2024-02-28T16:31:08.324 INFO:tasks.workunit.client.0.smithi001.stdout:5/437: chown dc/df/f11 158 1 2024-02-28T16:31:08.325 INFO:tasks.workunit.client.0.smithi001.stdout:7/458: symlink d3/d17/d3d/d59/l95 0 2024-02-28T16:31:08.326 INFO:tasks.workunit.client.0.smithi001.stdout:4/450: mkdir d3/d1c/d58/d93 0 2024-02-28T16:31:08.326 INFO:tasks.workunit.client.0.smithi001.stdout:5/438: mkdir dc/d1a/d21/d2d/d91 0 2024-02-28T16:31:08.327 INFO:tasks.workunit.client.0.smithi001.stdout:4/451: chown d3/d5/f34 29114232 1 2024-02-28T16:31:08.333 INFO:tasks.workunit.client.0.smithi001.stdout:5/439: write dc/d1a/d52/f72 [5002639,80861] 0 2024-02-28T16:31:08.333 INFO:tasks.workunit.client.0.smithi001.stdout:4/452: mknod d3/d5/d7/d1f/d3a/d68/d87/c94 0 2024-02-28T16:31:08.336 INFO:tasks.workunit.client.0.smithi001.stdout:5/440: mknod dc/df/c92 0 2024-02-28T16:31:08.337 INFO:tasks.workunit.client.0.smithi001.stdout:8/432: write d4/d7/d22/d27/f2f [2450893,80564] 0 2024-02-28T16:31:08.339 INFO:tasks.workunit.client.0.smithi001.stdout:4/453: creat d3/d5/d7/d1f/d20/f95 x:0 0 0 2024-02-28T16:31:08.341 INFO:tasks.workunit.client.0.smithi001.stdout:4/454: stat d3/d1c/d58/l4b 0 2024-02-28T16:31:08.342 INFO:tasks.workunit.client.0.smithi001.stdout:8/433: unlink d4/d32/c3e 0 2024-02-28T16:31:08.345 INFO:tasks.workunit.client.0.smithi001.stdout:8/434: fsync d4/d32/d69/f72 0 2024-02-28T16:31:08.347 INFO:tasks.workunit.client.0.smithi001.stdout:5/441: getdents dc/d12/d41/d54 0 2024-02-28T16:31:08.350 INFO:tasks.workunit.client.0.smithi001.stdout:8/435: truncate d4/d7/d22/d27/d58/f71 624999 0 2024-02-28T16:31:08.353 INFO:tasks.workunit.client.0.smithi001.stdout:5/442: truncate dc/f64 584330 0 2024-02-28T16:31:08.357 INFO:tasks.workunit.client.0.smithi001.stdout:8/436: rename d4/d6e to d4/d7/d22/d27/d4b/d6b/d89 0 2024-02-28T16:31:08.358 INFO:tasks.workunit.client.0.smithi001.stdout:4/455: dwrite d3/d5/d7/d1f/d2e/f39 [0,4194304] 0 2024-02-28T16:31:08.361 INFO:tasks.workunit.client.0.smithi001.stdout:4/456: chown d3/d5/d7/d1f/d3a/c8f 17 1 2024-02-28T16:31:08.365 INFO:tasks.workunit.client.0.smithi001.stdout:4/457: creat d3/d5/d7/d1f/f96 x:0 0 0 2024-02-28T16:31:08.366 INFO:tasks.workunit.client.0.smithi001.stdout:8/437: link d4/d47/c4e d4/d7/c8a 0 2024-02-28T16:31:08.367 INFO:tasks.workunit.client.0.smithi001.stdout:5/443: truncate dc/d1a/d21/f37 33820 0 2024-02-28T16:31:08.369 INFO:tasks.workunit.client.0.smithi001.stdout:8/438: write d4/d7/d22/d27/f4a [5385148,54429] 0 2024-02-28T16:31:08.372 INFO:tasks.workunit.client.0.smithi001.stdout:4/458: link d3/f3b d3/d5/d4e/f97 0 2024-02-28T16:31:08.385 INFO:tasks.workunit.client.0.smithi001.stdout:4/459: dread d3/d2d/d4c/f51 [0,4194304] 0 2024-02-28T16:31:08.388 INFO:tasks.workunit.client.0.smithi001.stdout:4/460: chown d3/d5/f34 42216124 1 2024-02-28T16:31:08.390 INFO:tasks.workunit.client.0.smithi001.stdout:8/439: dwrite d4/d7/d22/f3f [0,4194304] 0 2024-02-28T16:31:08.394 INFO:tasks.workunit.client.0.smithi001.stdout:5/444: dwrite dc/d1a/d21/d2d/f42 [0,4194304] 0 2024-02-28T16:31:08.394 INFO:tasks.workunit.client.0.smithi001.stdout:4/461: creat d3/d1c/d58/f98 x:0 0 0 2024-02-28T16:31:08.396 INFO:tasks.workunit.client.0.smithi001.stdout:8/440: mkdir d4/d47/d8b 0 2024-02-28T16:31:08.396 INFO:tasks.workunit.client.0.smithi001.stdout:5/445: write dc/d12/f14 [885939,104486] 0 2024-02-28T16:31:08.398 INFO:tasks.workunit.client.0.smithi001.stdout:4/462: symlink d3/d2d/d83/l99 0 2024-02-28T16:31:08.399 INFO:tasks.workunit.client.0.smithi001.stdout:4/463: stat d3/f3b 0 2024-02-28T16:31:08.401 INFO:tasks.workunit.client.0.smithi001.stdout:8/441: truncate d4/d32/f7c 5042855 0 2024-02-28T16:31:08.404 INFO:tasks.workunit.client.0.smithi001.stdout:8/442: fdatasync d4/d7/f23 0 2024-02-28T16:31:08.406 INFO:tasks.workunit.client.0.smithi001.stdout:8/443: chown d4/d32/f5c 505582040 1 2024-02-28T16:31:08.406 INFO:tasks.workunit.client.0.smithi001.stdout:5/446: rename dc/d1a/d52/l8c to dc/d1a/d21/d2d/d3c/d4a/l93 0 2024-02-28T16:31:08.409 INFO:tasks.workunit.client.0.smithi001.stdout:5/447: write dc/d12/d56/f90 [412976,100960] 0 2024-02-28T16:31:08.410 INFO:tasks.workunit.client.0.smithi001.stdout:8/444: link d4/d7/f2b d4/d47/d8b/f8c 0 2024-02-28T16:31:08.413 INFO:tasks.workunit.client.0.smithi001.stdout:5/448: write dc/d1a/d21/d2d/f78 [1882547,34552] 0 2024-02-28T16:31:08.413 INFO:tasks.workunit.client.0.smithi001.stdout:4/464: write d3/d5/d7/f21 [1932775,10442] 0 2024-02-28T16:31:08.415 INFO:tasks.workunit.client.0.smithi001.stdout:8/445: creat d4/d7/d22/d27/d4b/d77/f8d x:0 0 0 2024-02-28T16:31:08.419 INFO:tasks.workunit.client.0.smithi001.stdout:5/449: write dc/d1a/d21/d2d/d3c/d4a/f75 [329262,70134] 0 2024-02-28T16:31:08.419 INFO:tasks.workunit.client.0.smithi001.stdout:8/446: mkdir d4/d7/d22/d27/d8e 0 2024-02-28T16:31:08.420 INFO:tasks.workunit.client.0.smithi001.stdout:5/450: readlink dc/d12/d41/d54/l7a 0 2024-02-28T16:31:08.422 INFO:tasks.workunit.client.0.smithi001.stdout:8/447: unlink d4/d7/l25 0 2024-02-28T16:31:08.425 INFO:tasks.workunit.client.0.smithi001.stdout:8/448: mkdir d4/d32/d69/d8f 0 2024-02-28T16:31:08.425 INFO:tasks.workunit.client.0.smithi001.stdout:5/451: read dc/f5c [497714,92617] 0 2024-02-28T16:31:08.428 INFO:tasks.workunit.client.0.smithi001.stdout:8/449: fsync d4/d7/f23 0 2024-02-28T16:31:08.432 INFO:tasks.workunit.client.0.smithi001.stdout:8/450: chown d4/d47/d8b/f8c 13 1 2024-02-28T16:31:08.433 INFO:tasks.workunit.client.0.smithi001.stdout:8/451: stat d4/d47 0 2024-02-28T16:31:08.433 INFO:tasks.workunit.client.0.smithi001.stdout:4/465: dwrite d3/d5/d4e/f80 [0,4194304] 0 2024-02-28T16:31:08.526 INFO:tasks.workunit.client.0.smithi001.stdout:8/452: mknod d4/d7/d22/d27/d8e/c90 0 2024-02-28T16:31:08.528 INFO:tasks.workunit.client.0.smithi001.stdout:4/466: creat d3/d5/d7/d1f/d20/f9a x:0 0 0 2024-02-28T16:31:08.533 INFO:tasks.workunit.client.0.smithi001.stdout:4/467: mknod d3/d2d/d4c/d69/d82/c9b 0 2024-02-28T16:31:08.534 INFO:tasks.workunit.client.0.smithi001.stdout:5/452: dwrite dc/d1a/d21/d2d/d3c/f5e [0,4194304] 0 2024-02-28T16:31:08.535 INFO:tasks.workunit.client.0.smithi001.stdout:8/453: rename d4/d32/d69/f81 to d4/d7/d22/d27/d4b/d6b/f91 0 2024-02-28T16:31:08.537 INFO:tasks.workunit.client.0.smithi001.stdout:4/468: readlink d3/l11 0 2024-02-28T16:31:08.541 INFO:tasks.workunit.client.0.smithi001.stdout:8/454: symlink d4/d7/l92 0 2024-02-28T16:31:08.544 INFO:tasks.workunit.client.0.smithi001.stdout:4/469: creat d3/d5/d7/d1f/d3a/d68/f9c x:0 0 0 2024-02-28T16:31:08.545 INFO:tasks.workunit.client.0.smithi001.stdout:8/455: mkdir d4/d32/d93 0 2024-02-28T16:31:08.545 INFO:tasks.workunit.client.0.smithi001.stdout:4/470: chown d3/d5/d7 29 1 2024-02-28T16:31:08.547 INFO:tasks.workunit.client.0.smithi001.stdout:4/471: dread - d3/d5/d7/d1f/f96 zero size 2024-02-28T16:31:08.548 INFO:tasks.workunit.client.0.smithi001.stdout:5/453: dread fa [0,4194304] 0 2024-02-28T16:31:08.551 INFO:tasks.workunit.client.0.smithi001.stdout:5/454: fdatasync dc/d1a/d21/d2d/f66 0 2024-02-28T16:31:08.551 INFO:tasks.workunit.client.0.smithi001.stdout:8/456: truncate d4/d7/d22/d27/f30 2916311 0 2024-02-28T16:31:08.554 INFO:tasks.workunit.client.0.smithi001.stdout:5/455: creat dc/d12/d41/d54/f94 x:0 0 0 2024-02-28T16:31:08.555 INFO:tasks.workunit.client.0.smithi001.stdout:8/457: mkdir d4/d7/d22/d27/d58/d94 0 2024-02-28T16:31:08.555 INFO:tasks.workunit.client.0.smithi001.stdout:5/456: chown dc/d1a/d7d/d87/f8f 1789200826 1 2024-02-28T16:31:08.557 INFO:tasks.workunit.client.0.smithi001.stdout:8/458: creat d4/d7/d22/d27/d58/f95 x:0 0 0 2024-02-28T16:31:08.560 INFO:tasks.workunit.client.0.smithi001.stdout:5/457: rename dc/d1a/d21/d2d/f78 to dc/df/d79/f95 0 2024-02-28T16:31:08.563 INFO:tasks.workunit.client.0.smithi001.stdout:4/472: write d3/d2d/f3c [1186142,39992] 0 2024-02-28T16:31:08.564 INFO:tasks.workunit.client.0.smithi001.stdout:5/458: mkdir dc/d12/d56/d96 0 2024-02-28T16:31:08.583 INFO:tasks.workunit.client.0.smithi001.stdout:8/459: dwrite d4/d7/f10 [4194304,4194304] 0 2024-02-28T16:31:08.585 INFO:tasks.workunit.client.0.smithi001.stdout:4/473: dwrite d3/d2d/f45 [4194304,4194304] 0 2024-02-28T16:31:08.585 INFO:tasks.workunit.client.0.smithi001.stdout:8/460: dread - d4/d32/d69/f72 zero size 2024-02-28T16:31:08.591 INFO:tasks.workunit.client.0.smithi001.stdout:4/474: mknod d3/d2d/d4c/d5e/c9d 0 2024-02-28T16:31:08.603 INFO:tasks.workunit.client.0.smithi001.stdout:4/475: link d3/d2d/d4c/d69/d82/l56 d3/d5/d7/d1f/d3a/d68/l9e 0 2024-02-28T16:31:08.604 INFO:tasks.workunit.client.0.smithi001.stdout:4/476: chown d3/d2d/d4c/d69 105 1 2024-02-28T16:31:08.607 INFO:tasks.workunit.client.0.smithi001.stdout:4/477: write d3/d1c/d58/f46 [1047642,125321] 0 2024-02-28T16:31:08.608 INFO:tasks.workunit.client.0.smithi001.stdout:8/461: dwrite d4/d7/d22/d27/d4b/f78 [0,4194304] 0 2024-02-28T16:31:08.609 INFO:tasks.workunit.client.0.smithi001.stdout:4/478: fsync d3/d5/d7/d1f/d3a/f70 0 2024-02-28T16:31:08.611 INFO:tasks.workunit.client.0.smithi001.stdout:8/462: dread - d4/d7/d22/d27/d58/f87 zero size 2024-02-28T16:31:08.611 INFO:tasks.workunit.client.0.smithi001.stdout:4/479: chown d3/d2d/d4c/c73 2222 1 2024-02-28T16:31:08.612 INFO:tasks.workunit.client.0.smithi001.stdout:8/463: readlink d4/d7/l66 0 2024-02-28T16:31:08.614 INFO:tasks.workunit.client.0.smithi001.stdout:4/480: write d3/d5/d7/d1f/d2e/f39 [357836,42048] 0 2024-02-28T16:31:08.615 INFO:tasks.workunit.client.0.smithi001.stdout:4/481: chown d3/d5/d92 149738 1 2024-02-28T16:31:08.618 INFO:tasks.workunit.client.0.smithi001.stdout:4/482: mkdir d3/d1c/d58/d6e/d9f 0 2024-02-28T16:31:08.633 INFO:tasks.workunit.client.0.smithi001.stdout:4/483: truncate d3/d2d/f63 917763 0 2024-02-28T16:31:08.634 INFO:tasks.workunit.client.0.smithi001.stdout:4/484: write d3/d5/d7/d1f/f96 [269052,95591] 0 2024-02-28T16:31:08.636 INFO:tasks.workunit.client.0.smithi001.stdout:4/485: write d3/f78 [150710,3919] 0 2024-02-28T16:31:08.640 INFO:tasks.workunit.client.0.smithi001.stdout:4/486: getdents d3 0 2024-02-28T16:31:08.642 INFO:tasks.workunit.client.0.smithi001.stdout:4/487: stat d3/d5/d7/d1f/d3a/c8f 0 2024-02-28T16:31:08.644 INFO:tasks.workunit.client.0.smithi001.stdout:4/488: mkdir d3/d1c/da0 0 2024-02-28T16:31:08.645 INFO:tasks.workunit.client.0.smithi001.stdout:4/489: fdatasync d3/d5/f38 0 2024-02-28T16:31:08.647 INFO:tasks.workunit.client.0.smithi001.stdout:3/430: sync 2024-02-28T16:31:08.649 INFO:tasks.workunit.client.0.smithi001.stdout:8/464: dread d4/d7/f23 [0,4194304] 0 2024-02-28T16:31:08.650 INFO:tasks.workunit.client.0.smithi001.stdout:4/490: creat d3/d2d/d4c/d5e/fa1 x:0 0 0 2024-02-28T16:31:08.651 INFO:tasks.workunit.client.1.smithi066.stdout:3/0: sync 2024-02-28T16:31:08.652 INFO:tasks.workunit.client.1.smithi066.stdout:3/1: dread - no filename 2024-02-28T16:31:08.652 INFO:tasks.workunit.client.1.smithi066.stdout:3/2: readlink - no filename 2024-02-28T16:31:08.653 INFO:tasks.workunit.client.0.smithi001.stdout:3/431: dread d4/d6/d7/f43 [0,4194304] 0 2024-02-28T16:31:08.654 INFO:tasks.workunit.client.0.smithi001.stdout:8/465: mknod d4/d7/d22/d27/d4b/d80/c96 0 2024-02-28T16:31:08.654 INFO:tasks.workunit.client.0.smithi001.stdout:4/491: creat d3/d5/d47/d90/fa2 x:0 0 0 2024-02-28T16:31:08.655 INFO:tasks.workunit.client.0.smithi001.stdout:3/432: dread - d4/d6/f51 zero size 2024-02-28T16:31:08.656 INFO:tasks.workunit.client.1.smithi066.stdout:3/3: creat f0 x:0 0 0 2024-02-28T16:31:08.656 INFO:tasks.workunit.client.0.smithi001.stdout:3/433: write d4/d18/f6b [653489,9520] 0 2024-02-28T16:31:08.657 INFO:tasks.workunit.client.1.smithi066.stdout:3/4: chown f0 16229072 1 2024-02-28T16:31:08.657 INFO:tasks.workunit.client.0.smithi001.stdout:4/492: chown d3/d2d/d4c/d69/d82/l56 27550 1 2024-02-28T16:31:08.658 INFO:tasks.workunit.client.1.smithi066.stdout:3/5: dread - f0 zero size 2024-02-28T16:31:08.659 INFO:tasks.workunit.client.0.smithi001.stdout:4/493: readlink d3/d5/d4e/l55 0 2024-02-28T16:31:08.660 INFO:tasks.workunit.client.0.smithi001.stdout:4/494: chown d3/d2d/d4c/d69/d82 120 1 2024-02-28T16:31:08.662 INFO:tasks.workunit.client.0.smithi001.stdout:4/495: chown d3/d5/d7/d1f/f7d 32573732 1 2024-02-28T16:31:08.667 INFO:tasks.workunit.client.0.smithi001.stdout:8/466: dread d4/d7/d22/d27/d4b/d77/f83 [0,4194304] 0 2024-02-28T16:31:08.669 INFO:tasks.workunit.client.0.smithi001.stdout:3/434: dread d4/d6/f1c [0,4194304] 0 2024-02-28T16:31:08.671 INFO:tasks.workunit.client.0.smithi001.stdout:8/467: mkdir d4/d32/d69/d8f/d97 0 2024-02-28T16:31:08.672 INFO:tasks.workunit.client.0.smithi001.stdout:3/435: write d4/d18/f6b [302642,121815] 0 2024-02-28T16:31:08.677 INFO:tasks.workunit.client.0.smithi001.stdout:3/436: write d4/d6/fc [1407686,59205] 0 2024-02-28T16:31:08.679 INFO:tasks.workunit.client.1.smithi066.stdout:3/6: dwrite f0 [0,4194304] 0 2024-02-28T16:31:08.681 INFO:tasks.workunit.client.0.smithi001.stdout:4/496: dwrite d3/d5/d47/d90/fa2 [0,4194304] 0 2024-02-28T16:31:08.683 INFO:tasks.workunit.client.0.smithi001.stdout:3/437: mknod d4/d6/d7f/d82/c90 0 2024-02-28T16:31:08.684 INFO:tasks.workunit.client.1.smithi066.stdout:3/7: symlink l1 0 2024-02-28T16:31:08.684 INFO:tasks.workunit.client.0.smithi001.stdout:4/497: rmdir d3/d1c 39 2024-02-28T16:31:08.686 INFO:tasks.workunit.client.1.smithi066.stdout:3/8: write f0 [2493247,83736] 0 2024-02-28T16:31:08.690 INFO:tasks.workunit.client.0.smithi001.stdout:3/438: link d4/d6/f56 d4/d18/f91 0 2024-02-28T16:31:08.692 INFO:tasks.workunit.client.0.smithi001.stdout:4/498: symlink d3/d2d/la3 0 2024-02-28T16:31:08.694 INFO:tasks.workunit.client.0.smithi001.stdout:8/468: dwrite d4/d7/d22/d27/f60 [0,4194304] 0 2024-02-28T16:31:08.700 INFO:tasks.workunit.client.0.smithi001.stdout:4/499: link d3/d5/d7/d1f/d2e/f59 d3/d5/d7/d1f/d3a/fa4 0 2024-02-28T16:31:08.705 INFO:tasks.workunit.client.1.smithi066.stdout:3/9: dread f0 [0,4194304] 0 2024-02-28T16:31:08.710 INFO:tasks.workunit.client.0.smithi001.stdout:3/439: dwrite d4/d6/d7/d23/f2d [0,4194304] 0 2024-02-28T16:31:08.711 INFO:tasks.workunit.client.0.smithi001.stdout:4/500: dread d3/d1c/d58/f76 [0,4194304] 0 2024-02-28T16:31:08.711 INFO:tasks.workunit.client.0.smithi001.stdout:8/469: link d4/d7/d22/c68 d4/d7/d22/d27/d58/d62/c98 0 2024-02-28T16:31:08.714 INFO:tasks.workunit.client.0.smithi001.stdout:3/440: chown d4/d18/l4f 3770 1 2024-02-28T16:31:08.715 INFO:tasks.workunit.client.0.smithi001.stdout:4/501: mknod d3/d1c/ca5 0 2024-02-28T16:31:08.719 INFO:tasks.workunit.client.0.smithi001.stdout:4/502: write d3/d5/d7/d1f/d2e/f86 [1391521,107804] 0 2024-02-28T16:31:08.719 INFO:tasks.workunit.client.0.smithi001.stdout:3/441: dread d4/f35 [0,4194304] 0 2024-02-28T16:31:08.722 INFO:tasks.workunit.client.1.smithi066.stdout:3/10: dread f0 [0,4194304] 0 2024-02-28T16:31:08.722 INFO:tasks.workunit.client.0.smithi001.stdout:4/503: chown d3/d5/d47/l8c 1045614 1 2024-02-28T16:31:08.723 INFO:tasks.workunit.client.0.smithi001.stdout:3/442: creat d4/d6/d7/d78/f92 x:0 0 0 2024-02-28T16:31:08.725 INFO:tasks.workunit.client.1.smithi066.stdout:3/11: mknod c2 0 2024-02-28T16:31:08.737 INFO:tasks.workunit.client.1.smithi066.stdout:3/12: dread f0 [0,4194304] 0 2024-02-28T16:31:08.740 INFO:tasks.workunit.client.1.smithi066.stdout:3/13: rename f0 to f3 0 2024-02-28T16:31:08.746 INFO:tasks.workunit.client.0.smithi001.stdout:4/504: getdents d3/d5/d7/d1f/d3a/d68/d87 0 2024-02-28T16:31:08.747 INFO:tasks.workunit.client.0.smithi001.stdout:3/443: dread d4/d18/f10 [0,4194304] 0 2024-02-28T16:31:08.747 INFO:tasks.workunit.client.1.smithi066.stdout:3/14: truncate f3 64944 0 2024-02-28T16:31:08.748 INFO:tasks.workunit.client.0.smithi001.stdout:4/505: write d3/d1c/d58/f27 [4112528,68395] 0 2024-02-28T16:31:08.748 INFO:tasks.workunit.client.0.smithi001.stdout:3/444: chown d4/d18/d5f/f77 6 1 2024-02-28T16:31:08.751 INFO:tasks.workunit.client.0.smithi001.stdout:3/445: read d4/d18/f19 [1671201,11997] 0 2024-02-28T16:31:08.751 INFO:tasks.workunit.client.0.smithi001.stdout:4/506: unlink d3/d1c/ca5 0 2024-02-28T16:31:08.752 INFO:tasks.workunit.client.1.smithi066.stdout:3/15: dread f3 [0,4194304] 0 2024-02-28T16:31:08.753 INFO:tasks.workunit.client.0.smithi001.stdout:3/446: fdatasync d4/d18/f69 0 2024-02-28T16:31:08.775 INFO:tasks.workunit.client.1.smithi066.stdout:3/16: dwrite f3 [0,4194304] 0 2024-02-28T16:31:08.777 INFO:tasks.workunit.client.1.smithi066.stdout:3/17: read f3 [2789183,122832] 0 2024-02-28T16:31:08.779 INFO:tasks.workunit.client.1.smithi066.stdout:3/18: chown l1 6689 1 2024-02-28T16:31:08.783 INFO:tasks.workunit.client.1.smithi066.stdout:3/19: write f3 [537598,68500] 0 2024-02-28T16:31:08.804 INFO:tasks.workunit.client.1.smithi066.stdout:3/20: fdatasync f3 0 2024-02-28T16:31:08.805 INFO:tasks.workunit.client.1.smithi066.stdout:3/21: readlink l1 0 2024-02-28T16:31:08.807 INFO:tasks.workunit.client.1.smithi066.stdout:3/22: write f3 [3748064,90839] 0 2024-02-28T16:31:08.812 INFO:tasks.workunit.client.1.smithi066.stdout:3/23: symlink l4 0 2024-02-28T16:31:08.817 INFO:tasks.workunit.client.0.smithi001.stdout:3/447: creat d4/d6/d7/f93 x:0 0 0 2024-02-28T16:31:08.821 INFO:tasks.workunit.client.1.smithi066.stdout:3/24: truncate f3 2161479 0 2024-02-28T16:31:08.822 INFO:tasks.workunit.client.0.smithi001.stdout:4/507: unlink d3/d5/f14 0 2024-02-28T16:31:08.825 INFO:tasks.workunit.client.0.smithi001.stdout:4/508: truncate d3/d5/d7/d1f/d2e/f64 901239 0 2024-02-28T16:31:08.832 INFO:tasks.workunit.client.0.smithi001.stdout:3/448: dwrite d4/d3c/f3e [4194304,4194304] 0 2024-02-28T16:31:08.833 INFO:tasks.workunit.client.0.smithi001.stdout:3/449: chown d4/d3c/l60 57671718 1 2024-02-28T16:31:08.836 INFO:tasks.workunit.client.0.smithi001.stdout:6/438: sync 2024-02-28T16:31:08.837 INFO:tasks.workunit.client.0.smithi001.stdout:0/437: sync 2024-02-28T16:31:08.841 INFO:tasks.workunit.client.0.smithi001.stdout:3/450: mknod d4/d6/d7/d78/c94 0 2024-02-28T16:31:08.843 INFO:tasks.workunit.client.0.smithi001.stdout:0/438: write d18/d22/d2b/f37 [2048251,78164] 0 2024-02-28T16:31:08.843 INFO:tasks.workunit.client.0.smithi001.stdout:4/509: dwrite d3/f3e [4194304,4194304] 0 2024-02-28T16:31:08.848 INFO:tasks.workunit.client.0.smithi001.stdout:6/439: rename d9/db/d19/l41 to d9/db/d5f/d89/l91 0 2024-02-28T16:31:08.848 INFO:tasks.workunit.client.0.smithi001.stdout:3/451: unlink d4/d1e/l45 0 2024-02-28T16:31:08.851 INFO:tasks.workunit.client.0.smithi001.stdout:4/510: symlink d3/d5/d7/d1f/d2e/la6 0 2024-02-28T16:31:08.852 INFO:tasks.workunit.client.0.smithi001.stdout:3/452: creat d4/d6/d7/d78/f95 x:0 0 0 2024-02-28T16:31:08.852 INFO:tasks.workunit.client.0.smithi001.stdout:4/511: rename d3 to d3/da7 22 2024-02-28T16:31:08.857 INFO:tasks.workunit.client.0.smithi001.stdout:4/512: symlink d3/d2d/d4c/d5e/la8 0 2024-02-28T16:31:08.859 INFO:tasks.workunit.client.0.smithi001.stdout:3/453: link d4/f44 d4/d6/d7/d78/d81/f96 0 2024-02-28T16:31:08.863 INFO:tasks.workunit.client.0.smithi001.stdout:0/439: dwrite d18/d22/d2f/f24 [4194304,4194304] 0 2024-02-28T16:31:08.873 INFO:tasks.workunit.client.0.smithi001.stdout:6/440: dwrite d9/d39/f3a [0,4194304] 0 2024-02-28T16:31:08.874 INFO:tasks.workunit.client.0.smithi001.stdout:0/440: link d18/d22/d2b/d36/d4d/l80 d18/d22/d3d/l89 0 2024-02-28T16:31:08.874 INFO:tasks.workunit.client.0.smithi001.stdout:3/454: dread d4/d6/f1c [0,4194304] 0 2024-02-28T16:31:08.879 INFO:tasks.workunit.client.0.smithi001.stdout:4/513: truncate d3/d1c/d58/f27 2393717 0 2024-02-28T16:31:08.879 INFO:tasks.workunit.client.0.smithi001.stdout:3/455: chown d4/d1e/d7c 13 1 2024-02-28T16:31:08.882 INFO:tasks.workunit.client.0.smithi001.stdout:0/441: fdatasync d18/d22/d2b/d36/d48/f55 0 2024-02-28T16:31:08.885 INFO:tasks.workunit.client.0.smithi001.stdout:3/456: mkdir d4/d97 0 2024-02-28T16:31:08.888 INFO:tasks.workunit.client.1.smithi066.stdout:7/1: sync 2024-02-28T16:31:08.888 INFO:tasks.workunit.client.1.smithi066.stdout:2/35: sync 2024-02-28T16:31:08.888 INFO:tasks.workunit.client.1.smithi066.stdout:7/2: rename - no filename 2024-02-28T16:31:08.888 INFO:tasks.workunit.client.1.smithi066.stdout:7/3: readlink - no filename 2024-02-28T16:31:08.888 INFO:tasks.workunit.client.1.smithi066.stdout:7/4: write - no filename 2024-02-28T16:31:08.889 INFO:tasks.workunit.client.0.smithi001.stdout:3/457: rename d4/c6a to d4/d38/c98 0 2024-02-28T16:31:08.891 INFO:tasks.workunit.client.1.smithi066.stdout:2/36: symlink d0/l6 0 2024-02-28T16:31:08.894 INFO:tasks.workunit.client.1.smithi066.stdout:2/37: creat d0/f7 x:0 0 0 2024-02-28T16:31:08.894 INFO:tasks.workunit.client.1.smithi066.stdout:2/38: chown d0 3452798 1 2024-02-28T16:31:08.895 INFO:tasks.workunit.client.1.smithi066.stdout:2/39: stat d0/l6 0 2024-02-28T16:31:08.897 INFO:tasks.workunit.client.1.smithi066.stdout:2/40: creat d0/f8 x:0 0 0 2024-02-28T16:31:08.898 INFO:tasks.workunit.client.0.smithi001.stdout:3/458: rename d4/d1e/d7c/l8e to d4/d1e/d7c/d7d/l99 0 2024-02-28T16:31:08.898 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:08 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:31:08.901 INFO:tasks.workunit.client.1.smithi066.stdout:2/41: rename d0/f7 to d0/f9 0 2024-02-28T16:31:08.901 INFO:tasks.workunit.client.1.smithi066.stdout:2/42: stat d0 0 2024-02-28T16:31:08.902 INFO:tasks.workunit.client.0.smithi001.stdout:3/459: chown d4/d6/c1d 1033476794 1 2024-02-28T16:31:08.904 INFO:tasks.workunit.client.1.smithi066.stdout:2/43: rename d0/l6 to d0/la 0 2024-02-28T16:31:08.905 INFO:tasks.workunit.client.0.smithi001.stdout:9/554: sync 2024-02-28T16:31:08.905 INFO:tasks.workunit.client.0.smithi001.stdout:6/441: dwrite d9/d29/d6a/f77 [0,4194304] 0 2024-02-28T16:31:08.910 INFO:tasks.workunit.client.0.smithi001.stdout:0/442: dwrite d18/d22/d2b/d36/d4d/f65 [0,4194304] 0 2024-02-28T16:31:08.911 INFO:tasks.workunit.client.0.smithi001.stdout:9/555: rmdir d5/d27/d42 39 2024-02-28T16:31:08.914 INFO:tasks.workunit.client.0.smithi001.stdout:4/514: dwrite d3/d5/f34 [0,4194304] 0 2024-02-28T16:31:08.914 INFO:tasks.workunit.client.0.smithi001.stdout:0/443: chown f11 122746328 1 2024-02-28T16:31:08.914 INFO:tasks.workunit.client.1.smithi066.stdout:2/44: getdents d0 0 2024-02-28T16:31:08.918 INFO:tasks.workunit.client.0.smithi001.stdout:6/442: symlink d9/d29/d6e/l92 0 2024-02-28T16:31:08.920 INFO:tasks.workunit.client.0.smithi001.stdout:9/556: mknod d5/d45/d6f/cc0 0 2024-02-28T16:31:08.921 INFO:tasks.workunit.client.0.smithi001.stdout:4/515: mknod d3/d1c/d58/d88/ca9 0 2024-02-28T16:31:08.923 INFO:tasks.workunit.client.0.smithi001.stdout:6/443: rmdir d9/db/d10 39 2024-02-28T16:31:08.934 INFO:tasks.workunit.client.0.smithi001.stdout:6/444: write d9/de/f2c [5824090,119206] 0 2024-02-28T16:31:08.935 INFO:tasks.workunit.client.1.smithi066.stdout:2/45: dwrite d0/f9 [0,4194304] 0 2024-02-28T16:31:08.940 INFO:tasks.workunit.client.0.smithi001.stdout:3/460: write d4/d18/f91 [786502,66794] 0 2024-02-28T16:31:08.941 INFO:tasks.workunit.client.0.smithi001.stdout:9/557: truncate d5/d4d/f77 3695121 0 2024-02-28T16:31:08.943 INFO:tasks.workunit.client.0.smithi001.stdout:0/444: dread d18/f33 [0,4194304] 0 2024-02-28T16:31:08.944 INFO:tasks.workunit.client.0.smithi001.stdout:6/445: creat d9/d29/d6e/d4e/f93 x:0 0 0 2024-02-28T16:31:08.945 INFO:tasks.workunit.client.0.smithi001.stdout:9/558: write d5/f18 [5033595,34454] 0 2024-02-28T16:31:08.946 INFO:tasks.workunit.client.0.smithi001.stdout:3/461: mkdir d4/d1e/d7c/d9a 0 2024-02-28T16:31:08.948 INFO:tasks.workunit.client.0.smithi001.stdout:6/446: fsync d9/db/f1b 0 2024-02-28T16:31:08.951 INFO:tasks.workunit.client.0.smithi001.stdout:9/559: read d5/f18 [4033458,99559] 0 2024-02-28T16:31:08.952 INFO:tasks.workunit.client.0.smithi001.stdout:0/445: creat d18/d22/d2b/d36/d48/f8a x:0 0 0 2024-02-28T16:31:08.953 INFO:tasks.workunit.client.0.smithi001.stdout:4/516: dwrite d3/fa [0,4194304] 0 2024-02-28T16:31:08.954 INFO:tasks.workunit.client.0.smithi001.stdout:9/560: chown d5/d27/d34/d50/la5 0 1 2024-02-28T16:31:08.955 INFO:tasks.workunit.client.0.smithi001.stdout:1/585: sync 2024-02-28T16:31:08.955 INFO:tasks.workunit.client.0.smithi001.stdout:8/470: sync 2024-02-28T16:31:08.955 INFO:tasks.workunit.client.0.smithi001.stdout:5/459: sync 2024-02-28T16:31:08.955 INFO:tasks.workunit.client.0.smithi001.stdout:2/481: sync 2024-02-28T16:31:08.955 INFO:tasks.workunit.client.0.smithi001.stdout:7/459: sync 2024-02-28T16:31:08.956 INFO:tasks.workunit.client.0.smithi001.stdout:3/462: creat d4/d6/f9b x:0 0 0 2024-02-28T16:31:08.961 INFO:tasks.workunit.client.0.smithi001.stdout:7/460: mkdir d3/d17/d3d/d59/d96 0 2024-02-28T16:31:08.963 INFO:tasks.workunit.client.0.smithi001.stdout:9/561: fdatasync d5/d27/db1/d58/f9b 0 2024-02-28T16:31:08.964 INFO:tasks.workunit.client.0.smithi001.stdout:6/447: symlink d9/db/d10/l94 0 2024-02-28T16:31:08.969 INFO:tasks.workunit.client.0.smithi001.stdout:5/460: creat dc/d1a/d7d/f97 x:0 0 0 2024-02-28T16:31:08.971 INFO:tasks.workunit.client.0.smithi001.stdout:9/562: mkdir d5/d27/d34/dc1 0 2024-02-28T16:31:08.971 INFO:tasks.workunit.client.0.smithi001.stdout:8/471: symlink d4/d7/d22/d27/d4b/l99 0 2024-02-28T16:31:08.974 INFO:tasks.workunit.client.0.smithi001.stdout:3/463: unlink d4/d1e/d7c/d7d/l99 0 2024-02-28T16:31:08.977 INFO:tasks.workunit.client.0.smithi001.stdout:7/461: creat d3/d1c/d41/d49/d8d/f97 x:0 0 0 2024-02-28T16:31:08.977 INFO:tasks.workunit.client.0.smithi001.stdout:1/586: rename d0/d9/d1a/caf to d0/d9/d1a/d2b/d26/d27/d30/d67/cc9 0 2024-02-28T16:31:08.980 INFO:tasks.workunit.client.0.smithi001.stdout:9/563: symlink d5/d27/db1/d65/lc2 0 2024-02-28T16:31:08.981 INFO:tasks.workunit.client.0.smithi001.stdout:1/587: fsync d0/d9/d1a/f4c 0 2024-02-28T16:31:08.984 INFO:tasks.workunit.client.0.smithi001.stdout:3/464: creat d4/d3c/f9c x:0 0 0 2024-02-28T16:31:08.990 INFO:tasks.workunit.client.0.smithi001.stdout:6/448: rename d9/d29/d6e/l92 to d9/db/d5f/d62/d7e/l95 0 2024-02-28T16:31:08.992 INFO:tasks.workunit.client.0.smithi001.stdout:4/517: dwrite d3/d1c/d58/ff [0,4194304] 0 2024-02-28T16:31:08.993 INFO:tasks.workunit.client.0.smithi001.stdout:0/446: dwrite d18/d31/f6c [0,4194304] 0 2024-02-28T16:31:08.996 INFO:tasks.workunit.client.0.smithi001.stdout:8/472: write d4/d7/d22/d27/d4b/d77/f8d [833127,121822] 0 2024-02-28T16:31:08.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:08 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:31:08.999 INFO:tasks.workunit.client.0.smithi001.stdout:9/564: read d5/d27/d34/f90 [218254,23593] 0 2024-02-28T16:31:09.000 INFO:tasks.workunit.client.0.smithi001.stdout:4/518: readlink d3/d5/d7/d1f/l29 0 2024-02-28T16:31:09.001 INFO:tasks.workunit.client.0.smithi001.stdout:1/588: dread - d0/d9/d1a/d72/d9d/f83 zero size 2024-02-28T16:31:09.006 INFO:tasks.workunit.client.0.smithi001.stdout:6/449: rmdir d9/db/d5f/d8b 39 2024-02-28T16:31:09.009 INFO:tasks.workunit.client.1.smithi066.stdout:2/46: mknod d0/cb 0 2024-02-28T16:31:09.013 INFO:tasks.workunit.client.1.smithi066.stdout:2/47: write d0/f8 [1028669,86522] 0 2024-02-28T16:31:09.020 INFO:tasks.workunit.client.0.smithi001.stdout:6/450: fdatasync d9/db/d5f/d62/d83/f8f 0 2024-02-28T16:31:09.021 INFO:tasks.workunit.client.1.smithi066.stdout:2/48: unlink d0/la 0 2024-02-28T16:31:09.021 INFO:tasks.workunit.client.0.smithi001.stdout:8/473: truncate d4/d32/f46 1614441 0 2024-02-28T16:31:09.027 INFO:tasks.workunit.client.0.smithi001.stdout:6/451: write d9/de/f5a [9317420,17923] 0 2024-02-28T16:31:09.029 INFO:tasks.workunit.client.0.smithi001.stdout:7/462: dwrite d3/d1c/d41/d49/f6c [0,4194304] 0 2024-02-28T16:31:09.030 INFO:tasks.workunit.client.0.smithi001.stdout:0/447: dwrite d18/f6d [4194304,4194304] 0 2024-02-28T16:31:09.033 INFO:tasks.workunit.client.0.smithi001.stdout:2/482: dread d0/d2/d3/de/d22/f48 [0,4194304] 0 2024-02-28T16:31:09.039 INFO:tasks.workunit.client.0.smithi001.stdout:8/474: getdents d4/d7/d22/d27/d4b/d84 0 2024-02-28T16:31:09.043 INFO:tasks.workunit.client.1.smithi066.stdout:2/49: dwrite d0/f9 [4194304,4194304] 0 2024-02-28T16:31:09.044 INFO:tasks.workunit.client.0.smithi001.stdout:0/448: dread - d18/d22/f66 zero size 2024-02-28T16:31:09.046 INFO:tasks.workunit.client.0.smithi001.stdout:3/465: dread d4/d18/d5f/f59 [0,4194304] 0 2024-02-28T16:31:09.048 INFO:tasks.workunit.client.0.smithi001.stdout:7/463: mknod d3/d17/d3d/d5d/c98 0 2024-02-28T16:31:09.050 INFO:tasks.workunit.client.0.smithi001.stdout:8/475: fsync d4/d7/d22/d27/d4b/d80/f85 0 2024-02-28T16:31:09.058 INFO:tasks.workunit.client.0.smithi001.stdout:9/565: dwrite d5/d99/faf [0,4194304] 0 2024-02-28T16:31:09.059 INFO:tasks.workunit.client.0.smithi001.stdout:4/519: dwrite d3/d1c/d58/f76 [0,4194304] 0 2024-02-28T16:31:09.061 INFO:tasks.workunit.client.0.smithi001.stdout:1/589: dwrite d0/d9/f7e [0,4194304] 0 2024-02-28T16:31:09.062 INFO:tasks.workunit.client.0.smithi001.stdout:2/483: unlink d0/d2/d3/de/fa 0 2024-02-28T16:31:09.067 INFO:tasks.workunit.client.0.smithi001.stdout:6/452: getdents d9/d29/d6e 0 2024-02-28T16:31:09.069 INFO:tasks.workunit.client.0.smithi001.stdout:1/590: fdatasync d0/d9/d1a/d72/d9d/f83 0 2024-02-28T16:31:09.070 INFO:tasks.workunit.client.1.smithi066.stdout:2/50: dwrite d0/f8 [0,4194304] 0 2024-02-28T16:31:09.071 INFO:tasks.workunit.client.0.smithi001.stdout:9/566: mkdir d5/d27/db1/d5e/dc3 0 2024-02-28T16:31:09.073 INFO:tasks.workunit.client.0.smithi001.stdout:7/464: fsync d3/d17/f7e 0 2024-02-28T16:31:09.073 INFO:tasks.workunit.client.0.smithi001.stdout:2/484: rmdir d0/d2/d3/de/d8a 39 2024-02-28T16:31:09.078 INFO:tasks.workunit.client.0.smithi001.stdout:3/466: link d4/c31 d4/d6/d7f/c9d 0 2024-02-28T16:31:09.079 INFO:tasks.workunit.client.0.smithi001.stdout:1/591: fsync d0/d9/d1a/d2b/d26/d27/d30/d57/d8e/fc6 0 2024-02-28T16:31:09.081 INFO:tasks.workunit.client.1.smithi066.stdout:2/51: dread d0/f9 [0,4194304] 0 2024-02-28T16:31:09.083 INFO:tasks.workunit.client.1.smithi066.stdout:2/52: symlink d0/lc 0 2024-02-28T16:31:09.084 INFO:tasks.workunit.client.0.smithi001.stdout:4/520: creat d3/d5/faa x:0 0 0 2024-02-28T16:31:09.086 INFO:tasks.workunit.client.1.smithi066.stdout:2/53: symlink d0/ld 0 2024-02-28T16:31:09.086 INFO:tasks.workunit.client.0.smithi001.stdout:8/476: dwrite d4/d7/f10 [0,4194304] 0 2024-02-28T16:31:09.090 INFO:tasks.workunit.client.1.smithi066.stdout:8/16: sync 2024-02-28T16:31:09.090 INFO:tasks.workunit.client.1.smithi066.stdout:9/10: sync 2024-02-28T16:31:09.092 INFO:tasks.workunit.client.0.smithi001.stdout:6/453: dread d9/db/d5f/f65 [0,4194304] 0 2024-02-28T16:31:09.092 INFO:tasks.workunit.client.1.smithi066.stdout:9/11: fdatasync f1 0 2024-02-28T16:31:09.094 INFO:tasks.workunit.client.0.smithi001.stdout:2/485: rmdir d0/d2/d3/de/d61/d12/d55 39 2024-02-28T16:31:09.094 INFO:tasks.workunit.client.0.smithi001.stdout:3/467: creat d4/d1e/f9e x:0 0 0 2024-02-28T16:31:09.097 INFO:tasks.workunit.client.1.smithi066.stdout:8/17: creat d0/d6/f7 x:0 0 0 2024-02-28T16:31:09.098 INFO:tasks.workunit.client.1.smithi066.stdout:9/12: creat f3 x:0 0 0 2024-02-28T16:31:09.098 INFO:tasks.workunit.client.0.smithi001.stdout:8/477: rename d4/d32/d69/d8f/d97 to d4/d32/d69/d9a 0 2024-02-28T16:31:09.099 INFO:tasks.workunit.client.1.smithi066.stdout:8/18: chown d0 0 1 2024-02-28T16:31:09.100 INFO:tasks.workunit.client.1.smithi066.stdout:9/13: truncate f3 864063 0 2024-02-28T16:31:09.101 INFO:tasks.workunit.client.0.smithi001.stdout:6/454: fsync d9/db/d19/f37 0 2024-02-28T16:31:09.101 INFO:tasks.workunit.client.0.smithi001.stdout:2/486: readlink d0/d2/d3/de/d61/d12/d46/l8b 0 2024-02-28T16:31:09.102 INFO:tasks.workunit.client.0.smithi001.stdout:7/465: unlink d3/d17/l18 0 2024-02-28T16:31:09.113 INFO:tasks.workunit.client.0.smithi001.stdout:6/455: mkdir d9/d29/d6e/d4e/d96 0 2024-02-28T16:31:09.116 INFO:tasks.workunit.client.0.smithi001.stdout:8/478: mkdir d4/d9b 0 2024-02-28T16:31:09.116 INFO:tasks.workunit.client.0.smithi001.stdout:9/567: write d5/d45/f71 [606895,19404] 0 2024-02-28T16:31:09.117 INFO:tasks.workunit.client.1.smithi066.stdout:2/54: dwrite d0/f9 [0,4194304] 0 2024-02-28T16:31:09.118 INFO:tasks.workunit.client.0.smithi001.stdout:3/468: rename d4/d6/d7/d23/c28 to d4/d38/c9f 0 2024-02-28T16:31:09.120 INFO:tasks.workunit.client.0.smithi001.stdout:9/568: write d5/f12 [4094187,121451] 0 2024-02-28T16:31:09.128 INFO:tasks.workunit.client.0.smithi001.stdout:8/479: rename d4/d7/d22/c37 to d4/d7/d22/d27/d4b/d77/c9c 0 2024-02-28T16:31:09.128 INFO:tasks.workunit.client.1.smithi066.stdout:2/55: link d0/cb d0/ce 0 2024-02-28T16:31:09.130 INFO:tasks.workunit.client.1.smithi066.stdout:2/56: chown d0 70 1 2024-02-28T16:31:09.131 INFO:tasks.workunit.client.0.smithi001.stdout:4/521: truncate d3/d5/d47/d90/fa2 4024186 0 2024-02-28T16:31:09.131 INFO:tasks.workunit.client.0.smithi001.stdout:8/480: dread - d4/d32/d69/f6f zero size 2024-02-28T16:31:09.132 INFO:tasks.workunit.client.0.smithi001.stdout:6/456: truncate d9/d29/d6e/f31 3074498 0 2024-02-28T16:31:09.136 INFO:tasks.workunit.client.1.smithi066.stdout:8/19: dwrite d0/f3 [0,4194304] 0 2024-02-28T16:31:09.137 INFO:tasks.workunit.client.0.smithi001.stdout:8/481: creat d4/d32/d69/f9d x:0 0 0 2024-02-28T16:31:09.137 INFO:tasks.workunit.client.1.smithi066.stdout:9/14: dwrite f1 [0,4194304] 0 2024-02-28T16:31:09.140 INFO:tasks.workunit.client.0.smithi001.stdout:7/466: truncate d3/d17/d3d/f72 3532748 0 2024-02-28T16:31:09.141 INFO:tasks.workunit.client.0.smithi001.stdout:1/592: dwrite d0/d9/d1a/d2b/d26/f82 [0,4194304] 0 2024-02-28T16:31:09.141 INFO:tasks.workunit.client.0.smithi001.stdout:8/482: fsync d4/d7/d22/d27/d58/d62/f7e 0 2024-02-28T16:31:09.141 INFO:tasks.workunit.client.0.smithi001.stdout:7/467: fdatasync d3/f5f 0 2024-02-28T16:31:09.144 INFO:tasks.workunit.client.1.smithi066.stdout:8/20: mkdir d0/d8 0 2024-02-28T16:31:09.144 INFO:tasks.workunit.client.1.smithi066.stdout:9/15: chown c2 615544 1 2024-02-28T16:31:09.147 INFO:tasks.workunit.client.0.smithi001.stdout:6/457: creat d9/db/d19/f97 x:0 0 0 2024-02-28T16:31:09.147 INFO:tasks.workunit.client.0.smithi001.stdout:7/468: rmdir d3 39 2024-02-28T16:31:09.150 INFO:tasks.workunit.client.0.smithi001.stdout:8/483: mknod d4/d32/c9e 0 2024-02-28T16:31:09.150 INFO:tasks.workunit.client.0.smithi001.stdout:1/593: truncate d0/d9/d56/d2f/f7c 5092374 0 2024-02-28T16:31:09.151 INFO:tasks.workunit.client.0.smithi001.stdout:9/569: dwrite d5/fd [0,4194304] 0 2024-02-28T16:31:09.154 INFO:tasks.workunit.client.0.smithi001.stdout:4/522: creat d3/d5/fab x:0 0 0 2024-02-28T16:31:09.158 INFO:tasks.workunit.client.0.smithi001.stdout:2/487: dread d0/d2/d3/de/d22/f6d [0,4194304] 0 2024-02-28T16:31:09.160 INFO:tasks.workunit.client.1.smithi066.stdout:8/21: rmdir d0 39 2024-02-28T16:31:09.161 INFO:tasks.workunit.client.1.smithi066.stdout:9/16: write f1 [4599477,122627] 0 2024-02-28T16:31:09.170 INFO:tasks.workunit.client.0.smithi001.stdout:3/469: truncate d4/d6/d7/d23/f2d 2137310 0 2024-02-28T16:31:09.170 INFO:tasks.workunit.client.0.smithi001.stdout:9/570: unlink d5/d27/db1/c9c 0 2024-02-28T16:31:09.172 INFO:tasks.workunit.client.0.smithi001.stdout:9/571: chown d5/d99 1452354 1 2024-02-28T16:31:09.173 INFO:tasks.workunit.client.0.smithi001.stdout:1/594: creat d0/d9/d1a/d2b/d26/d27/d30/d67/fca x:0 0 0 2024-02-28T16:31:09.175 INFO:tasks.workunit.client.0.smithi001.stdout:2/488: mknod d0/d2/d3/de/d61/d12/d46/ca5 0 2024-02-28T16:31:09.176 INFO:tasks.workunit.client.1.smithi066.stdout:2/57: dwrite d0/f9 [8388608,4194304] 0 2024-02-28T16:31:09.177 INFO:tasks.workunit.client.0.smithi001.stdout:6/458: rename d9/d39/l42 to d9/d29/l98 0 2024-02-28T16:31:09.179 INFO:tasks.workunit.client.0.smithi001.stdout:2/489: chown d0/d2/l53 0 1 2024-02-28T16:31:09.180 INFO:tasks.workunit.client.1.smithi066.stdout:2/58: chown d0/ld 1 1 2024-02-28T16:31:09.184 INFO:tasks.workunit.client.0.smithi001.stdout:1/595: readlink d0/l3b 0 2024-02-28T16:31:09.184 INFO:tasks.workunit.client.0.smithi001.stdout:7/469: dwrite d3/d17/d52/d7f/f81 [0,4194304] 0 2024-02-28T16:31:09.185 INFO:tasks.workunit.client.1.smithi066.stdout:2/59: unlink d0/c5 0 2024-02-28T16:31:09.186 INFO:tasks.workunit.client.0.smithi001.stdout:7/470: chown d3/d1c/f33 141786 1 2024-02-28T16:31:09.187 INFO:tasks.workunit.client.0.smithi001.stdout:9/572: rename d5/f2e to d5/d27/d34/fc4 0 2024-02-28T16:31:09.189 INFO:tasks.workunit.client.0.smithi001.stdout:2/490: mkdir d0/d2/da6 0 2024-02-28T16:31:09.189 INFO:tasks.workunit.client.0.smithi001.stdout:6/459: creat d9/db/d5f/d62/d83/f99 x:0 0 0 2024-02-28T16:31:09.190 INFO:tasks.workunit.client.1.smithi066.stdout:2/60: creat d0/ff x:0 0 0 2024-02-28T16:31:09.191 INFO:tasks.workunit.client.0.smithi001.stdout:9/573: chown d5/c20 253 1 2024-02-28T16:31:09.191 INFO:tasks.workunit.client.1.smithi066.stdout:8/22: fdatasync d0/f3 0 2024-02-28T16:31:09.192 INFO:tasks.workunit.client.1.smithi066.stdout:9/17: dwrite f1 [0,4194304] 0 2024-02-28T16:31:09.194 INFO:tasks.workunit.client.0.smithi001.stdout:1/596: creat d0/d9/d1a/d2b/d26/fcb x:0 0 0 2024-02-28T16:31:09.197 INFO:tasks.workunit.client.0.smithi001.stdout:9/574: write d5/f6 [4726286,64770] 0 2024-02-28T16:31:09.202 INFO:tasks.workunit.client.1.smithi066.stdout:9/18: rename c0 to c4 0 2024-02-28T16:31:09.205 INFO:tasks.workunit.client.1.smithi066.stdout:9/19: write f1 [3734668,114794] 0 2024-02-28T16:31:09.208 INFO:tasks.workunit.client.1.smithi066.stdout:9/20: creat f5 x:0 0 0 2024-02-28T16:31:09.208 INFO:tasks.workunit.client.1.smithi066.stdout:5/23: sync 2024-02-28T16:31:09.210 INFO:tasks.workunit.client.1.smithi066.stdout:5/24: dread - d3/d4/f5 zero size 2024-02-28T16:31:09.210 INFO:tasks.workunit.client.1.smithi066.stdout:6/14: sync 2024-02-28T16:31:09.211 INFO:tasks.workunit.client.1.smithi066.stdout:2/61: dread d0/f8 [0,4194304] 0 2024-02-28T16:31:09.212 INFO:tasks.workunit.client.1.smithi066.stdout:6/15: chown f1 1119 1 2024-02-28T16:31:09.213 INFO:tasks.workunit.client.0.smithi001.stdout:2/491: creat d0/d2/d3/de/d61/d12/fa7 x:0 0 0 2024-02-28T16:31:09.213 INFO:tasks.workunit.client.1.smithi066.stdout:2/62: readlink d0/ld 0 2024-02-28T16:31:09.213 INFO:tasks.workunit.client.1.smithi066.stdout:5/25: write d3/d4/f5 [402463,37582] 0 2024-02-28T16:31:09.213 INFO:tasks.workunit.client.0.smithi001.stdout:7/471: getdents d3/d17/d52 0 2024-02-28T16:31:09.214 INFO:tasks.workunit.client.0.smithi001.stdout:1/597: creat d0/d9/d1a/d2b/d26/d27/d30/d88/fcc x:0 0 0 2024-02-28T16:31:09.215 INFO:tasks.workunit.client.1.smithi066.stdout:6/16: dread - d2/f3 zero size 2024-02-28T16:31:09.216 INFO:tasks.workunit.client.1.smithi066.stdout:9/21: mknod c6 0 2024-02-28T16:31:09.217 INFO:tasks.workunit.client.1.smithi066.stdout:5/26: write d3/d4/f5 [1368026,15219] 0 2024-02-28T16:31:09.217 INFO:tasks.workunit.client.0.smithi001.stdout:4/523: dread d3/d1c/d58/f27 [0,4194304] 0 2024-02-28T16:31:09.219 INFO:tasks.workunit.client.1.smithi066.stdout:2/63: mknod d0/c10 0 2024-02-28T16:31:09.219 INFO:tasks.workunit.client.0.smithi001.stdout:7/472: chown d3/d1c/d41/d49/d8d 1177878 1 2024-02-28T16:31:09.220 INFO:tasks.workunit.client.0.smithi001.stdout:8/484: dwrite d4/d47/f64 [0,4194304] 0 2024-02-28T16:31:09.221 INFO:tasks.workunit.client.0.smithi001.stdout:4/524: stat d3/d2d/d4c/d5e 0 2024-02-28T16:31:09.225 INFO:tasks.workunit.client.1.smithi066.stdout:6/17: mkdir d2/d4 0 2024-02-28T16:31:09.225 INFO:tasks.workunit.client.1.smithi066.stdout:8/23: write d0/f5 [701056,63880] 0 2024-02-28T16:31:09.230 INFO:tasks.workunit.client.1.smithi066.stdout:6/18: truncate d2/f3 979975 0 2024-02-28T16:31:09.232 INFO:tasks.workunit.client.1.smithi066.stdout:6/19: chown d2/f3 10776592 1 2024-02-28T16:31:09.233 INFO:tasks.workunit.client.1.smithi066.stdout:9/22: dread f1 [0,4194304] 0 2024-02-28T16:31:09.235 INFO:tasks.workunit.client.1.smithi066.stdout:9/23: dread - f5 zero size 2024-02-28T16:31:09.236 INFO:tasks.workunit.client.0.smithi001.stdout:7/473: mkdir d3/d17/d52/d7f/d99 0 2024-02-28T16:31:09.236 INFO:tasks.workunit.client.0.smithi001.stdout:1/598: creat d0/d9/d1a/d2b/d26/d27/d30/d57/d7d/dae/fcd x:0 0 0 2024-02-28T16:31:09.236 INFO:tasks.workunit.client.0.smithi001.stdout:2/492: rename d0/d2/d3/c5a to d0/d2/d3/d36/ca8 0 2024-02-28T16:31:09.239 INFO:tasks.workunit.client.1.smithi066.stdout:9/24: write f3 [1028923,29208] 0 2024-02-28T16:31:09.240 INFO:tasks.workunit.client.0.smithi001.stdout:3/470: dread d4/d1e/f33 [0,4194304] 0 2024-02-28T16:31:09.241 INFO:tasks.workunit.client.1.smithi066.stdout:5/27: write f2 [65514,73040] 0 2024-02-28T16:31:09.242 INFO:tasks.workunit.client.0.smithi001.stdout:6/460: dread d9/db/f13 [0,4194304] 0 2024-02-28T16:31:09.242 INFO:tasks.workunit.client.0.smithi001.stdout:1/599: write d0/d9/d56/d2f/f64 [1026907,44943] 0 2024-02-28T16:31:09.242 INFO:tasks.workunit.client.1.smithi066.stdout:9/25: read f1 [3193140,127363] 0 2024-02-28T16:31:09.245 INFO:tasks.workunit.client.0.smithi001.stdout:9/575: dwrite d5/f12 [0,4194304] 0 2024-02-28T16:31:09.251 INFO:tasks.workunit.client.1.smithi066.stdout:5/28: chown d3/d4/f5 15 1 2024-02-28T16:31:09.251 INFO:tasks.workunit.client.1.smithi066.stdout:9/26: rename c6 to c7 0 2024-02-28T16:31:09.254 INFO:tasks.workunit.client.1.smithi066.stdout:8/24: dwrite d0/f3 [4194304,4194304] 0 2024-02-28T16:31:09.255 INFO:tasks.workunit.client.1.smithi066.stdout:9/27: read f1 [250737,691] 0 2024-02-28T16:31:09.257 INFO:tasks.workunit.client.1.smithi066.stdout:5/29: symlink d3/l6 0 2024-02-28T16:31:09.257 INFO:tasks.workunit.client.0.smithi001.stdout:2/493: mknod d0/d2/d3/d36/ca9 0 2024-02-28T16:31:09.258 INFO:tasks.workunit.client.0.smithi001.stdout:9/576: mknod d5/d4d/cc5 0 2024-02-28T16:31:09.261 INFO:tasks.workunit.client.0.smithi001.stdout:9/577: chown d5/c20 20 1 2024-02-28T16:31:09.261 INFO:tasks.workunit.client.1.smithi066.stdout:6/20: dread f1 [0,4194304] 0 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/5: sync 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/6: read - no filename 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/7: unlink - no file 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/8: link - no file 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/9: unlink - no file 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/10: write - no filename 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/11: write - no filename 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/12: write - no filename 2024-02-28T16:31:09.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/13: write - no filename 2024-02-28T16:31:09.264 INFO:tasks.workunit.client.1.smithi066.stdout:0/110: sync 2024-02-28T16:31:09.264 INFO:tasks.workunit.client.1.smithi066.stdout:3/25: sync 2024-02-28T16:31:09.264 INFO:tasks.workunit.client.1.smithi066.stdout:4/82: sync 2024-02-28T16:31:09.264 INFO:tasks.workunit.client.1.smithi066.stdout:1/86: sync 2024-02-28T16:31:09.264 INFO:tasks.workunit.client.1.smithi066.stdout:9/28: rename f1 to f8 0 2024-02-28T16:31:09.265 INFO:tasks.workunit.client.0.smithi001.stdout:3/471: getdents d4/d1e/d7c/d7d 0 2024-02-28T16:31:09.265 INFO:tasks.workunit.client.0.smithi001.stdout:7/474: getdents d3/d17/d3d/d42 0 2024-02-28T16:31:09.267 INFO:tasks.workunit.client.0.smithi001.stdout:9/578: creat d5/d45/d6f/fc6 x:0 0 0 2024-02-28T16:31:09.269 INFO:tasks.workunit.client.1.smithi066.stdout:8/25: creat d0/d6/f9 x:0 0 0 2024-02-28T16:31:09.270 INFO:tasks.workunit.client.1.smithi066.stdout:2/64: dwrite d0/f8 [0,4194304] 0 2024-02-28T16:31:09.270 INFO:tasks.workunit.client.1.smithi066.stdout:5/30: symlink d3/d4/l7 0 2024-02-28T16:31:09.272 INFO:tasks.workunit.client.0.smithi001.stdout:6/461: dwrite d9/db/d10/f11 [0,4194304] 0 2024-02-28T16:31:09.272 INFO:tasks.workunit.client.0.smithi001.stdout:1/600: dwrite d0/d9/d56/d2f/f7c [0,4194304] 0 2024-02-28T16:31:09.272 INFO:tasks.workunit.client.0.smithi001.stdout:7/475: fsync d3/fa 0 2024-02-28T16:31:09.273 INFO:tasks.workunit.client.0.smithi001.stdout:3/472: mkdir d4/d18/da0 0 2024-02-28T16:31:09.281 INFO:tasks.workunit.client.0.smithi001.stdout:7/476: fsync d3/d1c/d41/d49/f65 0 2024-02-28T16:31:09.281 INFO:tasks.workunit.client.1.smithi066.stdout:3/26: creat f5 x:0 0 0 2024-02-28T16:31:09.281 INFO:tasks.workunit.client.1.smithi066.stdout:8/26: fsync d0/d6/f7 0 2024-02-28T16:31:09.281 INFO:tasks.workunit.client.1.smithi066.stdout:5/31: readlink d3/l6 0 2024-02-28T16:31:09.281 INFO:tasks.workunit.client.1.smithi066.stdout:4/83: write d1/fe [925859,115485] 0 2024-02-28T16:31:09.282 INFO:tasks.workunit.client.1.smithi066.stdout:5/32: chown d3/d4 4475 1 2024-02-28T16:31:09.282 INFO:tasks.workunit.client.0.smithi001.stdout:9/579: chown d5/d27/d42/f7e 93820 1 2024-02-28T16:31:09.286 INFO:tasks.workunit.client.0.smithi001.stdout:3/473: mkdir d4/d6/d7/d78/d81/da1 0 2024-02-28T16:31:09.287 INFO:tasks.workunit.client.1.smithi066.stdout:4/84: chown d1/f13 526611299 1 2024-02-28T16:31:09.287 INFO:tasks.workunit.client.1.smithi066.stdout:7/14: symlink l0 0 2024-02-28T16:31:09.287 INFO:tasks.workunit.client.1.smithi066.stdout:7/15: fsync - no filename 2024-02-28T16:31:09.289 INFO:tasks.workunit.client.0.smithi001.stdout:3/474: dread - d4/d18/d5f/f5b zero size 2024-02-28T16:31:09.290 INFO:tasks.workunit.client.0.smithi001.stdout:1/601: mkdir d0/d9/d1a/d2b/d26/d27/dce 0 2024-02-28T16:31:09.293 INFO:tasks.workunit.client.0.smithi001.stdout:8/485: truncate d4/d7/d22/f3f 2245283 0 2024-02-28T16:31:09.295 INFO:tasks.workunit.client.0.smithi001.stdout:6/462: truncate d9/db/d10/f46 3686469 0 2024-02-28T16:31:09.298 INFO:tasks.workunit.client.0.smithi001.stdout:2/494: dwrite d0/d2/d3/de/d8a/f71 [0,4194304] 0 2024-02-28T16:31:09.302 INFO:tasks.workunit.client.1.smithi066.stdout:4/85: write d1/f12 [132358,75502] 0 2024-02-28T16:31:09.303 INFO:tasks.workunit.client.0.smithi001.stdout:8/486: write d4/d7/d22/d27/d58/f75 [20986,120345] 0 2024-02-28T16:31:09.305 INFO:tasks.workunit.client.1.smithi066.stdout:5/33: creat d3/d4/f8 x:0 0 0 2024-02-28T16:31:09.308 INFO:tasks.workunit.client.0.smithi001.stdout:1/602: mknod d0/d9/ccf 0 2024-02-28T16:31:09.309 INFO:tasks.workunit.client.1.smithi066.stdout:7/16: creat f1 x:0 0 0 2024-02-28T16:31:09.311 INFO:tasks.workunit.client.0.smithi001.stdout:2/495: readlink d0/d2/d3/de/l65 0 2024-02-28T16:31:09.314 INFO:tasks.workunit.client.0.smithi001.stdout:8/487: rename d4/d7/l92 to d4/d32/d69/l9f 0 2024-02-28T16:31:09.324 INFO:tasks.workunit.client.1.smithi066.stdout:9/29: dread f8 [0,4194304] 0 2024-02-28T16:31:09.325 INFO:tasks.workunit.client.0.smithi001.stdout:1/603: creat d0/d9/d1a/d2b/d26/d27/d30/d52/fd0 x:0 0 0 2024-02-28T16:31:09.325 INFO:tasks.workunit.client.0.smithi001.stdout:2/496: fdatasync d0/d2/d3/de/d22/f38 0 2024-02-28T16:31:09.325 INFO:tasks.workunit.client.0.smithi001.stdout:8/488: unlink d4/d7/d22/c42 0 2024-02-28T16:31:09.329 INFO:tasks.workunit.client.1.smithi066.stdout:2/65: getdents d0 0 2024-02-28T16:31:09.330 INFO:tasks.workunit.client.0.smithi001.stdout:7/477: dread d3/d17/d3d/d59/f67 [0,4194304] 0 2024-02-28T16:31:09.333 INFO:tasks.workunit.client.0.smithi001.stdout:9/580: dwrite d5/f79 [0,4194304] 0 2024-02-28T16:31:09.334 INFO:tasks.workunit.client.1.smithi066.stdout:2/66: readlink d0/ld 0 2024-02-28T16:31:09.335 INFO:tasks.workunit.client.0.smithi001.stdout:2/497: write d0/d2/d3/f6c [797244,31409] 0 2024-02-28T16:31:09.335 INFO:tasks.workunit.client.0.smithi001.stdout:1/604: rmdir d0/d9/d56/d2f 39 2024-02-28T16:31:09.336 INFO:tasks.workunit.client.1.smithi066.stdout:8/27: truncate d0/f3 6147431 0 2024-02-28T16:31:09.343 INFO:tasks.workunit.client.0.smithi001.stdout:3/475: dwrite d4/f83 [0,4194304] 0 2024-02-28T16:31:09.343 INFO:tasks.workunit.client.0.smithi001.stdout:6/463: dwrite d9/f4f [0,4194304] 0 2024-02-28T16:31:09.344 INFO:tasks.workunit.client.1.smithi066.stdout:5/34: write d3/d4/f5 [1544449,40011] 0 2024-02-28T16:31:09.345 INFO:tasks.workunit.client.0.smithi001.stdout:6/464: stat d9/d39/f3a 0 2024-02-28T16:31:09.346 INFO:tasks.workunit.client.1.smithi066.stdout:0/111: dwrite d2/fc [0,4194304] 0 2024-02-28T16:31:09.347 INFO:tasks.workunit.client.0.smithi001.stdout:7/478: truncate d3/d17/d3d/d42/f53 619044 0 2024-02-28T16:31:09.349 INFO:tasks.workunit.client.1.smithi066.stdout:8/28: fsync d0/f5 0 2024-02-28T16:31:09.349 INFO:tasks.workunit.client.1.smithi066.stdout:9/30: link c2 c9 0 2024-02-28T16:31:09.349 INFO:tasks.workunit.client.0.smithi001.stdout:1/605: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d57/fa0 0 2024-02-28T16:31:09.352 INFO:tasks.workunit.client.0.smithi001.stdout:2/498: symlink d0/d2/d3/de/d8a/laa 0 2024-02-28T16:31:09.354 INFO:tasks.workunit.client.0.smithi001.stdout:9/581: truncate d5/f64 862066 0 2024-02-28T16:31:09.356 INFO:tasks.workunit.client.0.smithi001.stdout:1/606: dread - d0/d9/d1a/d2b/d26/d27/d30/d67/fca zero size 2024-02-28T16:31:09.357 INFO:tasks.workunit.client.1.smithi066.stdout:2/67: write d0/f8 [3710586,38530] 0 2024-02-28T16:31:09.359 INFO:tasks.workunit.client.1.smithi066.stdout:6/21: dwrite f1 [4194304,4194304] 0 2024-02-28T16:31:09.359 INFO:tasks.workunit.client.0.smithi001.stdout:1/607: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d57/d8e/fc6 0 2024-02-28T16:31:09.359 INFO:tasks.workunit.client.0.smithi001.stdout:6/465: unlink d9/d29/d6e/f40 0 2024-02-28T16:31:09.361 INFO:tasks.workunit.client.0.smithi001.stdout:6/466: chown f8 114675 1 2024-02-28T16:31:09.364 INFO:tasks.workunit.client.1.smithi066.stdout:4/86: dread d1/f16 [0,4194304] 0 2024-02-28T16:31:09.364 INFO:tasks.workunit.client.0.smithi001.stdout:2/499: chown d0/f17 16 1 2024-02-28T16:31:09.368 INFO:tasks.workunit.client.0.smithi001.stdout:9/582: rmdir d5/d27/d34 39 2024-02-28T16:31:09.369 INFO:tasks.workunit.client.1.smithi066.stdout:3/27: dwrite f3 [0,4194304] 0 2024-02-28T16:31:09.369 INFO:tasks.workunit.client.1.smithi066.stdout:5/35: creat d3/d4/f9 x:0 0 0 2024-02-28T16:31:09.371 INFO:tasks.workunit.client.1.smithi066.stdout:3/28: chown f3 43585016 1 2024-02-28T16:31:09.374 INFO:tasks.workunit.client.1.smithi066.stdout:1/87: dwrite d2/d18/d19/f1c [0,4194304] 0 2024-02-28T16:31:09.377 INFO:tasks.workunit.client.1.smithi066.stdout:5/36: mknod d3/d4/ca 0 2024-02-28T16:31:09.379 INFO:tasks.workunit.client.1.smithi066.stdout:5/37: readlink d3/d4/l7 0 2024-02-28T16:31:09.382 INFO:tasks.workunit.client.1.smithi066.stdout:3/29: symlink l6 0 2024-02-28T16:31:09.384 INFO:tasks.workunit.client.1.smithi066.stdout:3/30: truncate f5 280892 0 2024-02-28T16:31:09.384 INFO:tasks.workunit.client.0.smithi001.stdout:1/608: rename d0/d9/d1a/d2b/d26/d27/d30/d57 to d0/d9/d1a/d2b/d26/d27/d30/d88/dd1 0 2024-02-28T16:31:09.386 INFO:tasks.workunit.client.0.smithi001.stdout:2/500: mknod d0/d2/d3/de/d61/cab 0 2024-02-28T16:31:09.388 INFO:tasks.workunit.client.0.smithi001.stdout:9/583: creat d5/d45/d6f/fc7 x:0 0 0 2024-02-28T16:31:09.390 INFO:tasks.workunit.client.1.smithi066.stdout:5/38: creat d3/d4/fb x:0 0 0 2024-02-28T16:31:09.393 INFO:tasks.workunit.client.0.smithi001.stdout:7/479: fsync d3/d17/d3d/d42/f53 0 2024-02-28T16:31:09.394 INFO:tasks.workunit.client.1.smithi066.stdout:3/31: symlink l7 0 2024-02-28T16:31:09.395 INFO:tasks.workunit.client.1.smithi066.stdout:5/39: dread - d3/d4/f9 zero size 2024-02-28T16:31:09.397 INFO:tasks.workunit.client.0.smithi001.stdout:3/476: dread d4/d6/d7/d78/d81/f96 [0,4194304] 0 2024-02-28T16:31:09.397 INFO:tasks.workunit.client.1.smithi066.stdout:2/68: dread d0/f8 [0,4194304] 0 2024-02-28T16:31:09.397 INFO:tasks.workunit.client.1.smithi066.stdout:1/88: mkdir d2/d18/d1f 0 2024-02-28T16:31:09.399 INFO:tasks.workunit.client.0.smithi001.stdout:9/584: truncate d5/d45/d6f/f7f 182615 0 2024-02-28T16:31:09.400 INFO:tasks.workunit.client.0.smithi001.stdout:1/609: mknod d0/d9/d56/cd2 0 2024-02-28T16:31:09.402 INFO:tasks.workunit.client.1.smithi066.stdout:8/29: rename d0/f3 to d0/fa 0 2024-02-28T16:31:09.403 INFO:tasks.workunit.client.0.smithi001.stdout:1/610: fsync d0/f28 0 2024-02-28T16:31:09.404 INFO:tasks.workunit.client.1.smithi066.stdout:1/89: dread - d2/fc zero size 2024-02-28T16:31:09.404 INFO:tasks.workunit.client.1.smithi066.stdout:4/87: link c0 d1/d18/c19 0 2024-02-28T16:31:09.404 INFO:tasks.workunit.client.0.smithi001.stdout:9/585: fdatasync d5/d27/db1/f5f 0 2024-02-28T16:31:09.405 INFO:tasks.workunit.client.1.smithi066.stdout:9/31: getdents . 0 2024-02-28T16:31:09.405 INFO:tasks.workunit.client.1.smithi066.stdout:8/30: chown d0/d6 0 1 2024-02-28T16:31:09.406 INFO:tasks.workunit.client.0.smithi001.stdout:9/586: dread - d5/d99/fbe zero size 2024-02-28T16:31:09.406 INFO:tasks.workunit.client.0.smithi001.stdout:7/480: rename d3/d1c/d41 to d3/d17/d3d/d5d/d9a 0 2024-02-28T16:31:09.408 INFO:tasks.workunit.client.1.smithi066.stdout:0/112: dwrite d2/d9/d10/f1f [0,4194304] 0 2024-02-28T16:31:09.409 INFO:tasks.workunit.client.0.smithi001.stdout:1/611: readlink d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/lc7 0 2024-02-28T16:31:09.409 INFO:tasks.workunit.client.1.smithi066.stdout:5/40: chown d3/d4/f9 2 1 2024-02-28T16:31:09.410 INFO:tasks.workunit.client.0.smithi001.stdout:8/489: dread d4/d7/d22/d27/d4b/d6b/f91 [8388608,4194304] 0 2024-02-28T16:31:09.412 INFO:tasks.workunit.client.1.smithi066.stdout:9/32: write f5 [1002386,47702] 0 2024-02-28T16:31:09.413 INFO:tasks.workunit.client.0.smithi001.stdout:2/501: getdents d0/d2/d3/de/d61/d12/d55 0 2024-02-28T16:31:09.413 INFO:tasks.workunit.client.0.smithi001.stdout:7/481: rmdir d3/d17/d52 39 2024-02-28T16:31:09.414 INFO:tasks.workunit.client.1.smithi066.stdout:1/90: mknod d2/c20 0 2024-02-28T16:31:09.414 INFO:tasks.workunit.client.1.smithi066.stdout:0/113: write d2/f22 [935708,22441] 0 2024-02-28T16:31:09.415 INFO:tasks.workunit.client.1.smithi066.stdout:2/69: unlink d0/ce 0 2024-02-28T16:31:09.416 INFO:tasks.workunit.client.0.smithi001.stdout:3/477: rename d4/d6/d7/f65 to d4/d18/da0/fa2 0 2024-02-28T16:31:09.416 INFO:tasks.workunit.client.1.smithi066.stdout:5/41: mkdir d3/d4/dc 0 2024-02-28T16:31:09.419 INFO:tasks.workunit.client.0.smithi001.stdout:3/478: rmdir d4/d18/da0 39 2024-02-28T16:31:09.419 INFO:tasks.workunit.client.1.smithi066.stdout:0/114: write d2/d9/d10/f1d [208599,42763] 0 2024-02-28T16:31:09.422 INFO:tasks.workunit.client.1.smithi066.stdout:1/91: creat d2/d18/f21 x:0 0 0 2024-02-28T16:31:09.422 INFO:tasks.workunit.client.1.smithi066.stdout:5/42: chown d3/d4/f9 62291440 1 2024-02-28T16:31:09.424 INFO:tasks.workunit.client.1.smithi066.stdout:9/33: unlink c9 0 2024-02-28T16:31:09.427 INFO:tasks.workunit.client.1.smithi066.stdout:9/34: read f3 [903239,87878] 0 2024-02-28T16:31:09.427 INFO:tasks.workunit.client.1.smithi066.stdout:9/35: readlink - no filename 2024-02-28T16:31:09.427 INFO:tasks.workunit.client.0.smithi001.stdout:7/482: dread d3/d17/d52/f77 [0,4194304] 0 2024-02-28T16:31:09.429 INFO:tasks.workunit.client.1.smithi066.stdout:2/70: link d0/cb d0/c11 0 2024-02-28T16:31:09.429 INFO:tasks.workunit.client.0.smithi001.stdout:9/587: dwrite d5/d45/d6f/fbd [0,4194304] 0 2024-02-28T16:31:09.433 INFO:tasks.workunit.client.1.smithi066.stdout:5/43: mkdir d3/d4/dc/dd 0 2024-02-28T16:31:09.433 INFO:tasks.workunit.client.0.smithi001.stdout:7/483: rename d3/d17/f7e to d3/d17/d3d/d42/d8e/f9b 0 2024-02-28T16:31:09.436 INFO:tasks.workunit.client.1.smithi066.stdout:9/36: dread f5 [0,4194304] 0 2024-02-28T16:31:09.436 INFO:tasks.workunit.client.0.smithi001.stdout:9/588: unlink d5/d45/fa0 0 2024-02-28T16:31:09.437 INFO:tasks.workunit.client.0.smithi001.stdout:7/484: readlink d3/d17/d3d/d5d/d9a/d49/l56 0 2024-02-28T16:31:09.438 INFO:tasks.workunit.client.1.smithi066.stdout:5/44: unlink f2 0 2024-02-28T16:31:09.439 INFO:tasks.workunit.client.1.smithi066.stdout:9/37: mkdir da 0 2024-02-28T16:31:09.440 INFO:tasks.workunit.client.0.smithi001.stdout:9/589: rename d5/d99 to d5/d45/d6f/dc8 0 2024-02-28T16:31:09.441 INFO:tasks.workunit.client.1.smithi066.stdout:9/38: chown f5 468 1 2024-02-28T16:31:09.441 INFO:tasks.workunit.client.0.smithi001.stdout:9/590: stat d5/d45/d6f/cc0 0 2024-02-28T16:31:09.442 INFO:tasks.workunit.client.0.smithi001.stdout:7/485: write d3/d17/d61/f71 [630343,54677] 0 2024-02-28T16:31:09.442 INFO:tasks.workunit.client.1.smithi066.stdout:5/45: getdents d3/d4/dc/dd 0 2024-02-28T16:31:09.443 INFO:tasks.workunit.client.1.smithi066.stdout:0/115: dwrite d2/fc [0,4194304] 0 2024-02-28T16:31:09.444 INFO:tasks.workunit.client.0.smithi001.stdout:9/591: creat d5/d27/db1/d33/fc9 x:0 0 0 2024-02-28T16:31:09.445 INFO:tasks.workunit.client.0.smithi001.stdout:7/486: rmdir d3/d17/d3d/d5d/d9a/d49/d63 39 2024-02-28T16:31:09.445 INFO:tasks.workunit.client.1.smithi066.stdout:9/39: creat da/fb x:0 0 0 2024-02-28T16:31:09.445 INFO:tasks.workunit.client.1.smithi066.stdout:0/116: readlink l1 0 2024-02-28T16:31:09.446 INFO:tasks.workunit.client.1.smithi066.stdout:5/46: mknod d3/d4/dc/ce 0 2024-02-28T16:31:09.446 INFO:tasks.workunit.client.0.smithi001.stdout:9/592: chown d5/d27/d42/d48/c4e 260625 1 2024-02-28T16:31:09.447 INFO:tasks.workunit.client.0.smithi001.stdout:9/593: readlink d5/d27/l80 0 2024-02-28T16:31:09.447 INFO:tasks.workunit.client.1.smithi066.stdout:9/40: write f8 [868857,4786] 0 2024-02-28T16:31:09.450 INFO:tasks.workunit.client.0.smithi001.stdout:9/594: symlink d5/d27/d34/lca 0 2024-02-28T16:31:09.453 INFO:tasks.workunit.client.1.smithi066.stdout:0/117: mkdir d2/d27 0 2024-02-28T16:31:09.460 INFO:tasks.workunit.client.1.smithi066.stdout:5/47: rename d3/d4/f8 to d3/ff 0 2024-02-28T16:31:09.462 INFO:tasks.workunit.client.1.smithi066.stdout:0/118: unlink d2/fc 0 2024-02-28T16:31:09.465 INFO:tasks.workunit.client.1.smithi066.stdout:5/48: mknod d3/c10 0 2024-02-28T16:31:09.465 INFO:tasks.workunit.client.1.smithi066.stdout:0/119: write d2/d9/d10/f26 [30710,122169] 0 2024-02-28T16:31:09.467 INFO:tasks.workunit.client.0.smithi001.stdout:9/595: dwrite d5/d27/db1/f91 [0,4194304] 0 2024-02-28T16:31:09.467 INFO:tasks.workunit.client.1.smithi066.stdout:9/41: rename f5 to da/fc 0 2024-02-28T16:31:09.467 INFO:tasks.workunit.client.1.smithi066.stdout:0/120: chown d2/c4 44465055 1 2024-02-28T16:31:09.468 INFO:tasks.workunit.client.1.smithi066.stdout:5/49: rename d3/d4 to d3/d4/d11 22 2024-02-28T16:31:09.470 INFO:tasks.workunit.client.1.smithi066.stdout:9/42: mkdir da/dd 0 2024-02-28T16:31:09.470 INFO:tasks.workunit.client.1.smithi066.stdout:0/121: truncate d2/d9/f21 703776 0 2024-02-28T16:31:09.471 INFO:tasks.workunit.client.1.smithi066.stdout:5/50: creat d3/d4/f12 x:0 0 0 2024-02-28T16:31:09.473 INFO:tasks.workunit.client.1.smithi066.stdout:0/122: chown d2/d9/d10/f1f 11683017 1 2024-02-28T16:31:09.475 INFO:tasks.workunit.client.1.smithi066.stdout:5/51: truncate d3/d4/f9 986230 0 2024-02-28T16:31:09.476 INFO:tasks.workunit.client.1.smithi066.stdout:0/123: fdatasync d2/d9/f1e 0 2024-02-28T16:31:09.480 INFO:tasks.workunit.client.1.smithi066.stdout:9/43: rmdir da/dd 0 2024-02-28T16:31:09.484 INFO:tasks.workunit.client.1.smithi066.stdout:3/32: fsync f3 0 2024-02-28T16:31:09.484 INFO:tasks.workunit.client.1.smithi066.stdout:0/124: creat d2/f28 x:0 0 0 2024-02-28T16:31:09.485 INFO:tasks.workunit.client.1.smithi066.stdout:5/52: link d3/l6 d3/d4/l13 0 2024-02-28T16:31:09.486 INFO:tasks.workunit.client.0.smithi001.stdout:3/479: mknod d4/d6/d7/ca3 0 2024-02-28T16:31:09.487 INFO:tasks.workunit.client.0.smithi001.stdout:3/480: write d4/d6/f52 [40839,48077] 0 2024-02-28T16:31:09.488 INFO:tasks.workunit.client.1.smithi066.stdout:5/53: rmdir d3/d4/dc 39 2024-02-28T16:31:09.488 INFO:tasks.workunit.client.1.smithi066.stdout:5/54: chown d3/d4 2 1 2024-02-28T16:31:09.490 INFO:tasks.workunit.client.0.smithi001.stdout:3/481: symlink d4/d3c/la4 0 2024-02-28T16:31:09.490 INFO:tasks.workunit.client.0.smithi001.stdout:5/461: sync 2024-02-28T16:31:09.490 INFO:tasks.workunit.client.0.smithi001.stdout:0/449: sync 2024-02-28T16:31:09.493 INFO:tasks.workunit.client.0.smithi001.stdout:3/482: chown d4/d6 5545500 1 2024-02-28T16:31:09.493 INFO:tasks.workunit.client.0.smithi001.stdout:9/596: write d5/f64 [307352,111900] 0 2024-02-28T16:31:09.499 INFO:tasks.workunit.client.0.smithi001.stdout:3/483: truncate d4/d3c/f9c 931942 0 2024-02-28T16:31:09.500 INFO:tasks.workunit.client.1.smithi066.stdout:0/125: dread d2/d9/d10/f1d [0,4194304] 0 2024-02-28T16:31:09.501 INFO:tasks.workunit.client.0.smithi001.stdout:9/597: write d5/d27/d34/f90 [2291925,110156] 0 2024-02-28T16:31:09.502 INFO:tasks.workunit.client.0.smithi001.stdout:0/450: rename d18/d22/d2b/d36/f49 to d18/d22/d76/f8b 0 2024-02-28T16:31:09.503 INFO:tasks.workunit.client.1.smithi066.stdout:0/126: dread d2/d9/d10/f26 [0,4194304] 0 2024-02-28T16:31:09.503 INFO:tasks.workunit.client.0.smithi001.stdout:7/487: dwrite d3/d17/d3d/f72 [0,4194304] 0 2024-02-28T16:31:09.508 INFO:tasks.workunit.client.1.smithi066.stdout:0/127: rename d2/d9/c15 to d2/d9/d10/d23/c29 0 2024-02-28T16:31:09.510 INFO:tasks.workunit.client.1.smithi066.stdout:0/128: chown d2/c16 19537 1 2024-02-28T16:31:09.512 INFO:tasks.workunit.client.1.smithi066.stdout:0/129: chown d2/c16 284400479 1 2024-02-28T16:31:09.516 INFO:tasks.workunit.client.0.smithi001.stdout:9/598: unlink d5/d27/f31 0 2024-02-28T16:31:09.516 INFO:tasks.workunit.client.0.smithi001.stdout:5/462: write f7 [1644825,113520] 0 2024-02-28T16:31:09.517 INFO:tasks.workunit.client.1.smithi066.stdout:0/130: getdents d2/d27 0 2024-02-28T16:31:09.518 INFO:tasks.workunit.client.1.smithi066.stdout:0/131: stat d2/f22 0 2024-02-28T16:31:09.518 INFO:tasks.workunit.client.0.smithi001.stdout:7/488: symlink d3/d17/l9c 0 2024-02-28T16:31:09.520 INFO:tasks.workunit.client.0.smithi001.stdout:5/463: fsync dc/d12/f16 0 2024-02-28T16:31:09.521 INFO:tasks.workunit.client.1.smithi066.stdout:0/132: mknod d2/c2a 0 2024-02-28T16:31:09.522 INFO:tasks.workunit.client.0.smithi001.stdout:9/599: chown d5/d4d/f77 6311886 1 2024-02-28T16:31:09.522 INFO:tasks.workunit.client.1.smithi066.stdout:5/55: rmdir d3/d4/dc 39 2024-02-28T16:31:09.524 INFO:tasks.workunit.client.0.smithi001.stdout:3/484: dread d4/d1e/f22 [0,4194304] 0 2024-02-28T16:31:09.524 INFO:tasks.workunit.client.1.smithi066.stdout:4/88: dwrite d1/f16 [0,4194304] 0 2024-02-28T16:31:09.525 INFO:tasks.workunit.client.0.smithi001.stdout:7/489: symlink d3/d17/d3d/d42/d8e/l9d 0 2024-02-28T16:31:09.525 INFO:tasks.workunit.client.1.smithi066.stdout:5/56: write d3/d4/f9 [1974600,44523] 0 2024-02-28T16:31:09.526 INFO:tasks.workunit.client.0.smithi001.stdout:2/502: dread d0/d2/d3/d73/f9b [0,4194304] 0 2024-02-28T16:31:09.526 INFO:tasks.workunit.client.0.smithi001.stdout:5/464: stat dc/d1a/c23 0 2024-02-28T16:31:09.526 INFO:tasks.workunit.client.0.smithi001.stdout:9/600: symlink d5/d45/d6f/dc8/lcb 0 2024-02-28T16:31:09.526 INFO:tasks.workunit.client.1.smithi066.stdout:4/89: chown d1/f11 44897779 1 2024-02-28T16:31:09.527 INFO:tasks.workunit.client.0.smithi001.stdout:3/485: dread - d4/d1e/f9e zero size 2024-02-28T16:31:09.527 INFO:tasks.workunit.client.1.smithi066.stdout:0/133: mknod d2/c2b 0 2024-02-28T16:31:09.530 INFO:tasks.workunit.client.0.smithi001.stdout:9/601: write d5/f64 [1167918,88354] 0 2024-02-28T16:31:09.531 INFO:tasks.workunit.client.0.smithi001.stdout:0/451: dread d18/f6d [4194304,4194304] 0 2024-02-28T16:31:09.531 INFO:tasks.workunit.client.0.smithi001.stdout:3/486: unlink d4/f35 0 2024-02-28T16:31:09.532 INFO:tasks.workunit.client.1.smithi066.stdout:2/71: truncate d0/f8 774606 0 2024-02-28T16:31:09.533 INFO:tasks.workunit.client.0.smithi001.stdout:9/602: chown d5/d27/d34/d50/f92 38 1 2024-02-28T16:31:09.533 INFO:tasks.workunit.client.0.smithi001.stdout:3/487: chown d4/d18/f6d 9514 1 2024-02-28T16:31:09.535 INFO:tasks.workunit.client.0.smithi001.stdout:0/452: mknod d18/d22/d57/d7e/c8c 0 2024-02-28T16:31:09.536 INFO:tasks.workunit.client.1.smithi066.stdout:5/57: stat d3/d4/dc 0 2024-02-28T16:31:09.536 INFO:tasks.workunit.client.0.smithi001.stdout:9/603: mkdir d5/d27/db1/da6/dcc 0 2024-02-28T16:31:09.537 INFO:tasks.workunit.client.1.smithi066.stdout:5/58: fsync d3/d4/f5 0 2024-02-28T16:31:09.538 INFO:tasks.workunit.client.0.smithi001.stdout:0/453: fsync d18/d22/d2b/d36/d4d/f65 0 2024-02-28T16:31:09.540 INFO:tasks.workunit.client.1.smithi066.stdout:2/72: mkdir d0/d12 0 2024-02-28T16:31:09.540 INFO:tasks.workunit.client.0.smithi001.stdout:9/604: rmdir d5/d27/db1/d33 39 2024-02-28T16:31:09.541 INFO:tasks.workunit.client.0.smithi001.stdout:2/503: dread d0/d2/d3/de/d22/f6d [0,4194304] 0 2024-02-28T16:31:09.542 INFO:tasks.workunit.client.0.smithi001.stdout:0/454: symlink d18/d22/d2b/d36/d4d/l8d 0 2024-02-28T16:31:09.543 INFO:tasks.workunit.client.1.smithi066.stdout:5/59: symlink d3/l14 0 2024-02-28T16:31:09.544 INFO:tasks.workunit.client.0.smithi001.stdout:0/455: read - d18/d22/d2b/f87 zero size 2024-02-28T16:31:09.544 INFO:tasks.workunit.client.0.smithi001.stdout:2/504: write d0/d2/d3/f6c [1214617,119558] 0 2024-02-28T16:31:09.547 INFO:tasks.workunit.client.0.smithi001.stdout:2/505: fdatasync d0/d2/d3/d36/f4b 0 2024-02-28T16:31:09.549 INFO:tasks.workunit.client.0.smithi001.stdout:2/506: write d0/d2/d3/de/d22/f38 [4120436,84014] 0 2024-02-28T16:31:09.550 INFO:tasks.workunit.client.1.smithi066.stdout:4/90: symlink d1/l1a 0 2024-02-28T16:31:09.551 INFO:tasks.workunit.client.0.smithi001.stdout:0/456: creat d18/d22/d2f/d1c/d79/f8e x:0 0 0 2024-02-28T16:31:09.554 INFO:tasks.workunit.client.0.smithi001.stdout:2/507: fdatasync d0/d2/d3/de/d22/f24 0 2024-02-28T16:31:09.556 INFO:tasks.workunit.client.1.smithi066.stdout:5/60: truncate d3/d4/f12 48550 0 2024-02-28T16:31:09.557 INFO:tasks.workunit.client.1.smithi066.stdout:0/134: dwrite d2/d9/f1e [0,4194304] 0 2024-02-28T16:31:09.557 INFO:tasks.workunit.client.0.smithi001.stdout:3/488: truncate d4/d6/d7/f2f 2509919 0 2024-02-28T16:31:09.560 INFO:tasks.workunit.client.1.smithi066.stdout:5/61: truncate d3/d4/f12 480901 0 2024-02-28T16:31:09.562 INFO:tasks.workunit.client.1.smithi066.stdout:9/44: truncate f8 3769740 0 2024-02-28T16:31:09.562 INFO:tasks.workunit.client.0.smithi001.stdout:0/457: rmdir d18/d22/d2b/d36/d83 0 2024-02-28T16:31:09.566 INFO:tasks.workunit.client.0.smithi001.stdout:0/458: fsync d18/d22/d3d/f67 0 2024-02-28T16:31:09.566 INFO:tasks.workunit.client.0.smithi001.stdout:2/508: rename d0/c77 to d0/d2/cac 0 2024-02-28T16:31:09.569 INFO:tasks.workunit.client.0.smithi001.stdout:0/459: read d18/d22/d3d/f50 [3120637,55923] 0 2024-02-28T16:31:09.572 INFO:tasks.workunit.client.1.smithi066.stdout:5/62: dread d3/d4/f5 [0,4194304] 0 2024-02-28T16:31:09.573 INFO:tasks.workunit.client.0.smithi001.stdout:0/460: mknod d18/d22/d57/c8f 0 2024-02-28T16:31:09.573 INFO:tasks.workunit.client.1.smithi066.stdout:8/31: dread d0/fa [4194304,4194304] 0 2024-02-28T16:31:09.576 INFO:tasks.workunit.client.0.smithi001.stdout:3/489: dwrite d4/d6/d7/f63 [0,4194304] 0 2024-02-28T16:31:09.584 INFO:tasks.workunit.client.0.smithi001.stdout:3/490: dread d4/d18/d5f/f59 [0,4194304] 0 2024-02-28T16:31:09.592 INFO:tasks.workunit.client.0.smithi001.stdout:0/461: dwrite d18/d22/f66 [0,4194304] 0 2024-02-28T16:31:09.594 INFO:tasks.workunit.client.0.smithi001.stdout:3/491: creat d4/d6/d7/d78/d81/d8a/fa5 x:0 0 0 2024-02-28T16:31:09.596 INFO:tasks.workunit.client.0.smithi001.stdout:3/492: chown d4/d18/d5f/l36 36472 1 2024-02-28T16:31:09.598 INFO:tasks.workunit.client.0.smithi001.stdout:3/493: read d4/d18/f49 [896,77819] 0 2024-02-28T16:31:09.601 INFO:tasks.workunit.client.0.smithi001.stdout:3/494: chown d4/l16 660 1 2024-02-28T16:31:09.608 INFO:tasks.workunit.client.1.smithi066.stdout:2/73: creat d0/d12/f13 x:0 0 0 2024-02-28T16:31:09.610 INFO:tasks.workunit.client.1.smithi066.stdout:0/135: creat d2/d9/d10/f2c x:0 0 0 2024-02-28T16:31:09.611 INFO:tasks.workunit.client.0.smithi001.stdout:0/462: dwrite d18/d22/d2b/d36/f56 [4194304,4194304] 0 2024-02-28T16:31:09.611 INFO:tasks.workunit.client.1.smithi066.stdout:9/45: write da/fc [1432876,35092] 0 2024-02-28T16:31:09.612 INFO:tasks.workunit.client.1.smithi066.stdout:5/63: creat d3/f15 x:0 0 0 2024-02-28T16:31:09.613 INFO:tasks.workunit.client.1.smithi066.stdout:8/32: mkdir d0/db 0 2024-02-28T16:31:09.614 INFO:tasks.workunit.client.1.smithi066.stdout:2/74: creat d0/d12/f14 x:0 0 0 2024-02-28T16:31:09.617 INFO:tasks.workunit.client.1.smithi066.stdout:5/64: symlink d3/l16 0 2024-02-28T16:31:09.618 INFO:tasks.workunit.client.0.smithi001.stdout:2/509: dread d0/f32 [0,4194304] 0 2024-02-28T16:31:09.622 INFO:tasks.workunit.client.1.smithi066.stdout:4/91: creat d1/f1b x:0 0 0 2024-02-28T16:31:09.622 INFO:tasks.workunit.client.1.smithi066.stdout:4/92: chown d1/c14 101 1 2024-02-28T16:31:09.625 INFO:tasks.workunit.client.1.smithi066.stdout:8/33: truncate d0/fa 515123 0 2024-02-28T16:31:09.631 INFO:tasks.workunit.client.1.smithi066.stdout:9/46: getdents da 0 2024-02-28T16:31:09.643 INFO:tasks.workunit.client.1.smithi066.stdout:8/34: truncate d0/f4 104407 0 2024-02-28T16:31:09.645 INFO:tasks.workunit.client.1.smithi066.stdout:2/75: dwrite d0/f9 [0,4194304] 0 2024-02-28T16:31:09.645 INFO:tasks.workunit.client.1.smithi066.stdout:7/17: sync 2024-02-28T16:31:09.647 INFO:tasks.workunit.client.0.smithi001.stdout:4/525: sync 2024-02-28T16:31:09.648 INFO:tasks.workunit.client.0.smithi001.stdout:4/526: chown d3/d2d/d83/l99 45 1 2024-02-28T16:31:09.649 INFO:tasks.workunit.client.0.smithi001.stdout:4/527: chown d3/d1c/d58/f76 20 1 2024-02-28T16:31:09.650 INFO:tasks.workunit.client.1.smithi066.stdout:8/35: mkdir d0/d6/dc 0 2024-02-28T16:31:09.650 INFO:tasks.workunit.client.1.smithi066.stdout:8/36: readlink - no filename 2024-02-28T16:31:09.650 INFO:tasks.workunit.client.1.smithi066.stdout:9/47: link f8 da/fe 0 2024-02-28T16:31:09.651 INFO:tasks.workunit.client.0.smithi001.stdout:4/528: write d3/d2d/d4c/d5e/fa1 [685034,19880] 0 2024-02-28T16:31:09.651 INFO:tasks.workunit.client.1.smithi066.stdout:5/65: dwrite d3/d4/fb [0,4194304] 0 2024-02-28T16:31:09.652 INFO:tasks.workunit.client.1.smithi066.stdout:4/93: dwrite d1/f11 [0,4194304] 0 2024-02-28T16:31:09.652 INFO:tasks.workunit.client.1.smithi066.stdout:4/94: stat d1/f7 0 2024-02-28T16:31:09.653 INFO:tasks.workunit.client.1.smithi066.stdout:4/95: stat d1/f12 0 2024-02-28T16:31:09.667 INFO:tasks.workunit.client.1.smithi066.stdout:4/96: mknod d1/d18/c1c 0 2024-02-28T16:31:09.669 INFO:tasks.workunit.client.1.smithi066.stdout:5/66: mkdir d3/d4/d17 0 2024-02-28T16:31:09.670 INFO:tasks.workunit.client.1.smithi066.stdout:1/92: sync 2024-02-28T16:31:09.670 INFO:tasks.workunit.client.1.smithi066.stdout:6/22: sync 2024-02-28T16:31:09.670 INFO:tasks.workunit.client.1.smithi066.stdout:3/33: sync 2024-02-28T16:31:09.674 INFO:tasks.workunit.client.1.smithi066.stdout:5/67: readlink d3/l14 0 2024-02-28T16:31:09.675 INFO:tasks.workunit.client.0.smithi001.stdout:3/495: link d4/f61 d4/d6/d7/d78/d81/fa6 0 2024-02-28T16:31:09.676 INFO:tasks.workunit.client.0.smithi001.stdout:0/463: mkdir d18/d22/d2f/d1c/d90 0 2024-02-28T16:31:09.677 INFO:tasks.workunit.client.0.smithi001.stdout:2/510: mknod d0/d2/d3/de/d22/cad 0 2024-02-28T16:31:09.680 INFO:tasks.workunit.client.0.smithi001.stdout:4/529: creat d3/d1c/d58/d6c/fac x:0 0 0 2024-02-28T16:31:09.680 INFO:tasks.workunit.client.0.smithi001.stdout:4/530: chown d3 3750 1 2024-02-28T16:31:09.680 INFO:tasks.workunit.client.1.smithi066.stdout:4/97: chown d1/f13 879470 1 2024-02-28T16:31:09.680 INFO:tasks.workunit.client.0.smithi001.stdout:0/464: symlink d18/d22/d2b/d36/d4d/l91 0 2024-02-28T16:31:09.681 INFO:tasks.workunit.client.0.smithi001.stdout:2/511: write d0/d2/d3/f43 [1813994,108884] 0 2024-02-28T16:31:09.683 INFO:tasks.workunit.client.1.smithi066.stdout:4/98: dread - d1/f1b zero size 2024-02-28T16:31:09.687 INFO:tasks.workunit.client.1.smithi066.stdout:0/136: fsync d2/d9/f21 0 2024-02-28T16:31:09.688 INFO:tasks.workunit.client.0.smithi001.stdout:3/496: getdents d4/d38 0 2024-02-28T16:31:09.688 INFO:tasks.workunit.client.0.smithi001.stdout:0/465: mknod d18/d22/c92 0 2024-02-28T16:31:09.688 INFO:tasks.workunit.client.0.smithi001.stdout:2/512: truncate d0/d2/d3/de/f5e 5087039 0 2024-02-28T16:31:09.688 INFO:tasks.workunit.client.0.smithi001.stdout:2/513: chown d0/f59 3764805 1 2024-02-28T16:31:09.689 INFO:tasks.workunit.client.0.smithi001.stdout:3/497: mknod d4/d97/ca7 0 2024-02-28T16:31:09.690 INFO:tasks.workunit.client.0.smithi001.stdout:0/466: write d18/d22/d2b/d36/d48/f8a [521295,29328] 0 2024-02-28T16:31:09.692 INFO:tasks.workunit.client.0.smithi001.stdout:3/498: write d4/d18/d5f/f59 [2427395,92738] 0 2024-02-28T16:31:09.692 INFO:tasks.workunit.client.0.smithi001.stdout:2/514: mknod d0/d2/d3/de/d22/cae 0 2024-02-28T16:31:09.694 INFO:tasks.workunit.client.0.smithi001.stdout:3/499: chown d4/d18/f19 5085704 1 2024-02-28T16:31:09.695 INFO:tasks.workunit.client.0.smithi001.stdout:0/467: mkdir d18/d22/d76/d93 0 2024-02-28T16:31:09.698 INFO:tasks.workunit.client.1.smithi066.stdout:8/37: rename d0/f5 to d0/fd 0 2024-02-28T16:31:09.700 INFO:tasks.workunit.client.0.smithi001.stdout:2/515: rename d0/d2/d3/de/d61/f15 to d0/d2/d3/de/d61/d12/d46/faf 0 2024-02-28T16:31:09.703 INFO:tasks.workunit.client.1.smithi066.stdout:4/99: rename d1/c14 to d1/d18/c1d 0 2024-02-28T16:31:09.704 INFO:tasks.workunit.client.0.smithi001.stdout:0/468: creat d18/d22/d76/d93/f94 x:0 0 0 2024-02-28T16:31:09.705 INFO:tasks.workunit.client.0.smithi001.stdout:2/516: creat d0/d2/da6/fb0 x:0 0 0 2024-02-28T16:31:09.706 INFO:tasks.workunit.client.0.smithi001.stdout:0/469: dread - d18/d5a/f72 zero size 2024-02-28T16:31:09.708 INFO:tasks.workunit.client.1.smithi066.stdout:9/48: dwrite da/fc [0,4194304] 0 2024-02-28T16:31:09.709 INFO:tasks.workunit.client.1.smithi066.stdout:9/49: readlink - no filename 2024-02-28T16:31:09.710 INFO:tasks.workunit.client.1.smithi066.stdout:2/76: dwrite d0/d12/f14 [0,4194304] 0 2024-02-28T16:31:09.712 INFO:tasks.workunit.client.0.smithi001.stdout:4/531: dwrite d3/d1c/d58/d88/f8a [0,4194304] 0 2024-02-28T16:31:09.716 INFO:tasks.workunit.client.1.smithi066.stdout:3/34: dread f3 [0,4194304] 0 2024-02-28T16:31:09.719 INFO:tasks.workunit.client.0.smithi001.stdout:0/470: rename d18/d22/d2f/d1c/d70 to d18/d22/d2b/d95 0 2024-02-28T16:31:09.720 INFO:tasks.workunit.client.0.smithi001.stdout:3/500: write d4/d6/d7/f20 [1978087,111087] 0 2024-02-28T16:31:09.720 INFO:tasks.workunit.client.1.smithi066.stdout:4/100: read d1/f4 [1259188,106020] 0 2024-02-28T16:31:09.725 INFO:tasks.workunit.client.0.smithi001.stdout:4/532: rename d3/d5/f38 to d3/d5/d7/d1f/d3a/fad 0 2024-02-28T16:31:09.725 INFO:tasks.workunit.client.0.smithi001.stdout:2/517: dread d0/f32 [0,4194304] 0 2024-02-28T16:31:09.727 INFO:tasks.workunit.client.0.smithi001.stdout:3/501: mkdir d4/d1e/d7c/da8 0 2024-02-28T16:31:09.727 INFO:tasks.workunit.client.0.smithi001.stdout:4/533: fdatasync d3/d1c/d58/f98 0 2024-02-28T16:31:09.730 INFO:tasks.workunit.client.1.smithi066.stdout:6/23: dwrite d2/f3 [0,4194304] 0 2024-02-28T16:31:09.730 INFO:tasks.workunit.client.0.smithi001.stdout:0/471: rename f15 to d18/d22/d57/d7e/f96 0 2024-02-28T16:31:09.734 INFO:tasks.workunit.client.0.smithi001.stdout:0/472: symlink d18/d31/l97 0 2024-02-28T16:31:09.736 INFO:tasks.workunit.client.0.smithi001.stdout:3/502: getdents d4/d6/d7/d78/d81/d8a 0 2024-02-28T16:31:09.739 INFO:tasks.workunit.client.0.smithi001.stdout:0/473: symlink d18/d22/l98 0 2024-02-28T16:31:09.739 INFO:tasks.workunit.client.0.smithi001.stdout:3/503: rmdir d4/d1e/d7c/d7d 39 2024-02-28T16:31:09.740 INFO:tasks.workunit.client.1.smithi066.stdout:3/35: dread f5 [0,4194304] 0 2024-02-28T16:31:09.740 INFO:tasks.workunit.client.1.smithi066.stdout:4/101: write d1/f12 [262626,77597] 0 2024-02-28T16:31:09.742 INFO:tasks.workunit.client.0.smithi001.stdout:0/474: fdatasync d18/d22/d2b/d36/d46/f4b 0 2024-02-28T16:31:09.750 INFO:tasks.workunit.client.0.smithi001.stdout:3/504: rename d4/d38/f55 to d4/d6/d7/d23/fa9 0 2024-02-28T16:31:09.751 INFO:tasks.workunit.client.1.smithi066.stdout:2/77: truncate d0/f8 243293 0 2024-02-28T16:31:09.753 INFO:tasks.workunit.client.0.smithi001.stdout:3/505: chown d4/d18/d5f/f47 14 1 2024-02-28T16:31:09.754 INFO:tasks.workunit.client.1.smithi066.stdout:3/36: mknod c8 0 2024-02-28T16:31:09.755 INFO:tasks.workunit.client.0.smithi001.stdout:2/518: dread d0/d2/d3/de/d22/f38 [0,4194304] 0 2024-02-28T16:31:09.755 INFO:tasks.workunit.client.1.smithi066.stdout:5/68: dwrite d3/ff [0,4194304] 0 2024-02-28T16:31:09.757 INFO:tasks.workunit.client.1.smithi066.stdout:1/93: dwrite d2/d18/d19/f1c [4194304,4194304] 0 2024-02-28T16:31:09.761 INFO:tasks.workunit.client.1.smithi066.stdout:4/102: fdatasync d1/f5 0 2024-02-28T16:31:09.761 INFO:tasks.workunit.client.1.smithi066.stdout:9/50: getdents da 0 2024-02-28T16:31:09.761 INFO:tasks.workunit.client.0.smithi001.stdout:3/506: link d4/d1e/f33 d4/d1e/d7c/da8/faa 0 2024-02-28T16:31:09.766 INFO:tasks.workunit.client.0.smithi001.stdout:3/507: write d4/d3c/f3e [269431,106979] 0 2024-02-28T16:31:09.767 INFO:tasks.workunit.client.0.smithi001.stdout:4/534: dread d3/d5/d7/d1f/d3a/fad [0,4194304] 0 2024-02-28T16:31:09.770 INFO:tasks.workunit.client.0.smithi001.stdout:3/508: rmdir d4/d3c 39 2024-02-28T16:31:09.774 INFO:tasks.workunit.client.0.smithi001.stdout:0/475: dwrite d18/d22/d2b/d5f/f85 [0,4194304] 0 2024-02-28T16:31:09.776 INFO:tasks.workunit.client.1.smithi066.stdout:3/37: fdatasync f5 0 2024-02-28T16:31:09.781 INFO:tasks.workunit.client.0.smithi001.stdout:3/509: rename d4/d1e/l62 to d4/d3c/lab 0 2024-02-28T16:31:09.781 INFO:tasks.workunit.client.1.smithi066.stdout:9/51: mkdir da/df 0 2024-02-28T16:31:09.782 INFO:tasks.workunit.client.0.smithi001.stdout:3/510: readlink d4/d18/l4f 0 2024-02-28T16:31:09.782 INFO:tasks.workunit.client.0.smithi001.stdout:0/476: mknod d18/d22/d2f/d1c/d90/c99 0 2024-02-28T16:31:09.783 INFO:tasks.workunit.client.1.smithi066.stdout:0/137: dwrite d2/d9/d10/f1f [0,4194304] 0 2024-02-28T16:31:09.783 INFO:tasks.workunit.client.1.smithi066.stdout:5/69: mkdir d3/d4/d17/d18 0 2024-02-28T16:31:09.790 INFO:tasks.workunit.client.0.smithi001.stdout:0/477: chown d18/d22/l44 2791147 1 2024-02-28T16:31:09.791 INFO:tasks.workunit.client.1.smithi066.stdout:9/52: mkdir da/d10 0 2024-02-28T16:31:09.793 INFO:tasks.workunit.client.0.smithi001.stdout:0/478: creat d18/d22/d2b/d36/d48/f9a x:0 0 0 2024-02-28T16:31:09.797 INFO:tasks.workunit.client.0.smithi001.stdout:0/479: link d18/d22/d47/f86 d18/d22/d2b/d36/d46/f9b 0 2024-02-28T16:31:09.798 INFO:tasks.workunit.client.0.smithi001.stdout:2/519: dread d0/f40 [0,4194304] 0 2024-02-28T16:31:09.798 INFO:tasks.workunit.client.1.smithi066.stdout:5/70: dread d3/d4/f12 [0,4194304] 0 2024-02-28T16:31:09.798 INFO:tasks.workunit.client.1.smithi066.stdout:4/103: creat d1/f1e x:0 0 0 2024-02-28T16:31:09.798 INFO:tasks.workunit.client.1.smithi066.stdout:6/24: dwrite f1 [0,4194304] 0 2024-02-28T16:31:09.803 INFO:tasks.workunit.client.1.smithi066.stdout:5/71: write d3/d4/f9 [176412,41054] 0 2024-02-28T16:31:09.803 INFO:tasks.workunit.client.1.smithi066.stdout:0/138: write d2/d9/d10/f1d [1045843,71475] 0 2024-02-28T16:31:09.809 INFO:tasks.workunit.client.0.smithi001.stdout:6/467: sync 2024-02-28T16:31:09.815 INFO:tasks.workunit.client.1.smithi066.stdout:4/104: write d1/f11 [3781639,87320] 0 2024-02-28T16:31:09.816 INFO:tasks.workunit.client.1.smithi066.stdout:6/25: rename c0 to d2/c5 0 2024-02-28T16:31:09.817 INFO:tasks.workunit.client.1.smithi066.stdout:5/72: creat d3/d4/dc/f19 x:0 0 0 2024-02-28T16:31:09.818 INFO:tasks.workunit.client.1.smithi066.stdout:1/94: dwrite d2/ff [0,4194304] 0 2024-02-28T16:31:09.818 INFO:tasks.workunit.client.0.smithi001.stdout:3/511: write d4/d18/f11 [3280718,6081] 0 2024-02-28T16:31:09.822 INFO:tasks.workunit.client.0.smithi001.stdout:6/468: mkdir d9/d29/d6e/d9a 0 2024-02-28T16:31:09.827 INFO:tasks.workunit.client.0.smithi001.stdout:6/469: symlink d9/db/d5f/d89/l9b 0 2024-02-28T16:31:09.829 INFO:tasks.workunit.client.1.smithi066.stdout:2/78: dread d0/d12/f14 [0,4194304] 0 2024-02-28T16:31:09.831 INFO:tasks.workunit.client.1.smithi066.stdout:0/139: getdents d2/d27 0 2024-02-28T16:31:09.831 INFO:tasks.workunit.client.1.smithi066.stdout:7/18: sync 2024-02-28T16:31:09.831 INFO:tasks.workunit.client.1.smithi066.stdout:6/26: write f1 [3299650,71484] 0 2024-02-28T16:31:09.835 INFO:tasks.workunit.client.1.smithi066.stdout:2/79: rename d0 to d0/d12/d15 22 2024-02-28T16:31:09.841 INFO:tasks.workunit.client.1.smithi066.stdout:3/38: dwrite f5 [0,4194304] 0 2024-02-28T16:31:09.841 INFO:tasks.workunit.client.1.smithi066.stdout:7/19: symlink l2 0 2024-02-28T16:31:09.842 INFO:tasks.workunit.client.0.smithi001.stdout:6/470: truncate d9/db/d10/f11 3214448 0 2024-02-28T16:31:09.842 INFO:tasks.workunit.client.0.smithi001.stdout:3/512: dwrite d4/f83 [0,4194304] 0 2024-02-28T16:31:09.846 INFO:tasks.workunit.client.1.smithi066.stdout:5/73: creat d3/d4/d17/d18/f1a x:0 0 0 2024-02-28T16:31:09.852 INFO:tasks.workunit.client.1.smithi066.stdout:3/39: read f5 [1496592,111666] 0 2024-02-28T16:31:09.852 INFO:tasks.workunit.client.1.smithi066.stdout:0/140: creat d2/d9/d10/d23/f2d x:0 0 0 2024-02-28T16:31:09.854 INFO:tasks.workunit.client.0.smithi001.stdout:0/480: dread d18/d22/d2b/d36/f69 [0,4194304] 0 2024-02-28T16:31:09.854 INFO:tasks.workunit.client.1.smithi066.stdout:7/20: creat f3 x:0 0 0 2024-02-28T16:31:09.857 INFO:tasks.workunit.client.0.smithi001.stdout:2/520: dread d0/fd [0,4194304] 0 2024-02-28T16:31:09.859 INFO:tasks.workunit.client.1.smithi066.stdout:5/74: mkdir d3/d1b 0 2024-02-28T16:31:09.859 INFO:tasks.workunit.client.1.smithi066.stdout:2/80: creat d0/d12/f16 x:0 0 0 2024-02-28T16:31:09.859 INFO:tasks.workunit.client.1.smithi066.stdout:9/53: dwrite f8 [4194304,4194304] 0 2024-02-28T16:31:09.861 INFO:tasks.workunit.client.0.smithi001.stdout:6/471: dwrite d9/db/f1b [0,4194304] 0 2024-02-28T16:31:09.861 INFO:tasks.workunit.client.0.smithi001.stdout:3/513: dwrite d4/d6/f51 [0,4194304] 0 2024-02-28T16:31:09.862 INFO:tasks.workunit.client.1.smithi066.stdout:7/21: creat f4 x:0 0 0 2024-02-28T16:31:09.862 INFO:tasks.workunit.client.1.smithi066.stdout:0/141: write d2/d9/f20 [1086774,108633] 0 2024-02-28T16:31:09.868 INFO:tasks.workunit.client.1.smithi066.stdout:7/22: creat f5 x:0 0 0 2024-02-28T16:31:09.868 INFO:tasks.workunit.client.0.smithi001.stdout:3/514: rename d4/d1e to d4/d1e/d7c/da8/dac 22 2024-02-28T16:31:09.868 INFO:tasks.workunit.client.0.smithi001.stdout:6/472: mknod d9/db/d5f/d89/c9c 0 2024-02-28T16:31:09.869 INFO:tasks.workunit.client.1.smithi066.stdout:5/75: symlink d3/d4/l1c 0 2024-02-28T16:31:09.875 INFO:tasks.workunit.client.1.smithi066.stdout:6/27: dwrite f1 [0,4194304] 0 2024-02-28T16:31:09.876 INFO:tasks.workunit.client.1.smithi066.stdout:2/81: write d0/ff [160211,92312] 0 2024-02-28T16:31:09.877 INFO:tasks.workunit.client.1.smithi066.stdout:7/23: unlink f3 0 2024-02-28T16:31:09.882 INFO:tasks.workunit.client.1.smithi066.stdout:4/105: dread d1/f16 [0,4194304] 0 2024-02-28T16:31:09.883 INFO:tasks.workunit.client.1.smithi066.stdout:6/28: chown f1 113138622 1 2024-02-28T16:31:09.884 INFO:tasks.workunit.client.1.smithi066.stdout:0/142: truncate d2/d9/f21 1741689 0 2024-02-28T16:31:09.884 INFO:tasks.workunit.client.0.smithi001.stdout:3/515: link d4/d1e/f22 d4/d18/fad 0 2024-02-28T16:31:09.885 INFO:tasks.workunit.client.0.smithi001.stdout:0/481: dwrite d18/d22/d2f/d1c/d79/f8e [0,4194304] 0 2024-02-28T16:31:09.885 INFO:tasks.workunit.client.1.smithi066.stdout:7/24: truncate f5 723939 0 2024-02-28T16:31:09.886 INFO:tasks.workunit.client.0.smithi001.stdout:6/473: symlink d9/d29/d6e/d4e/d96/l9d 0 2024-02-28T16:31:09.887 INFO:tasks.workunit.client.1.smithi066.stdout:5/76: creat d3/d4/dc/f1d x:0 0 0 2024-02-28T16:31:09.887 INFO:tasks.workunit.client.1.smithi066.stdout:7/25: chown l2 5243384 1 2024-02-28T16:31:09.891 INFO:tasks.workunit.client.0.smithi001.stdout:2/521: dread d0/d2/d3/de/d61/f23 [0,4194304] 0 2024-02-28T16:31:09.891 INFO:tasks.workunit.client.0.smithi001.stdout:6/474: chown d9/db/f1b 4845679 1 2024-02-28T16:31:09.894 INFO:tasks.workunit.client.0.smithi001.stdout:0/482: symlink d18/d22/d2f/d1c/l9c 0 2024-02-28T16:31:09.900 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:09 smithi066 ceph-mon[36236]: pgmap v48: 65 pgs: 65 active+clean; 1.0 GiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 20 MiB/s rd, 110 MiB/s wr, 252 op/s 2024-02-28T16:31:09.901 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:09 smithi066 ceph-mon[36236]: mgrmap e25: smithi066.ztraxe(active, since 90s), standbys: smithi001.rmthtx 2024-02-28T16:31:09.904 INFO:tasks.workunit.client.1.smithi066.stdout:7/26: mknod c6 0 2024-02-28T16:31:09.904 INFO:tasks.workunit.client.1.smithi066.stdout:6/29: symlink d2/l6 0 2024-02-28T16:31:09.904 INFO:tasks.workunit.client.1.smithi066.stdout:9/54: dwrite da/fb [0,4194304] 0 2024-02-28T16:31:09.904 INFO:tasks.workunit.client.0.smithi001.stdout:2/522: rmdir d0/d2/d3/d36 39 2024-02-28T16:31:09.905 INFO:tasks.workunit.client.0.smithi001.stdout:0/483: creat d18/d22/d2b/d36/d46/f9d x:0 0 0 2024-02-28T16:31:09.905 INFO:tasks.workunit.client.1.smithi066.stdout:3/40: dread f3 [0,4194304] 0 2024-02-28T16:31:09.909 INFO:tasks.workunit.client.1.smithi066.stdout:4/106: symlink d1/l1f 0 2024-02-28T16:31:09.912 INFO:tasks.workunit.client.1.smithi066.stdout:7/27: chown f5 1012 1 2024-02-28T16:31:09.913 INFO:tasks.workunit.client.1.smithi066.stdout:0/143: rename d2/d9/d10/f1f to d2/d9/d10/f2e 0 2024-02-28T16:31:09.915 INFO:tasks.workunit.client.1.smithi066.stdout:5/77: dread d3/d4/f12 [0,4194304] 0 2024-02-28T16:31:09.916 INFO:tasks.workunit.client.1.smithi066.stdout:7/28: stat f1 0 2024-02-28T16:31:09.916 INFO:tasks.workunit.client.1.smithi066.stdout:6/30: mknod d2/c7 0 2024-02-28T16:31:09.920 INFO:tasks.workunit.client.1.smithi066.stdout:7/29: write f5 [787865,95026] 0 2024-02-28T16:31:09.921 INFO:tasks.workunit.client.1.smithi066.stdout:8/38: sync 2024-02-28T16:31:09.926 INFO:tasks.workunit.client.1.smithi066.stdout:3/41: symlink l9 0 2024-02-28T16:31:09.930 INFO:tasks.workunit.client.0.smithi001.stdout:0/484: rename d18/d31/f6c to d18/d31/f9e 0 2024-02-28T16:31:09.931 INFO:tasks.workunit.client.1.smithi066.stdout:0/144: truncate d2/d9/f21 2629851 0 2024-02-28T16:31:09.931 INFO:tasks.workunit.client.0.smithi001.stdout:6/475: dwrite d9/db/d5f/d8b/f6d [4194304,4194304] 0 2024-02-28T16:31:09.932 INFO:tasks.workunit.client.0.smithi001.stdout:0/485: chown d18/d22/d2b/f87 6746 1 2024-02-28T16:31:09.934 INFO:tasks.workunit.client.0.smithi001.stdout:6/476: dread - d9/db/d5f/d62/d83/f8f zero size 2024-02-28T16:31:09.935 INFO:tasks.workunit.client.1.smithi066.stdout:5/78: dread - d3/d4/dc/f19 zero size 2024-02-28T16:31:09.938 INFO:tasks.workunit.client.0.smithi001.stdout:0/486: mkdir d18/d22/d2f/d1c/d90/d9f 0 2024-02-28T16:31:09.941 INFO:tasks.workunit.client.1.smithi066.stdout:3/42: write f5 [485577,41348] 0 2024-02-28T16:31:09.943 INFO:tasks.workunit.client.1.smithi066.stdout:7/30: dread f5 [0,4194304] 0 2024-02-28T16:31:09.944 INFO:tasks.workunit.client.1.smithi066.stdout:2/82: dwrite d0/f9 [4194304,4194304] 0 2024-02-28T16:31:09.944 INFO:tasks.workunit.client.0.smithi001.stdout:0/487: rename d18/d22/d3d/c7c to d18/d22/d2b/d5f/ca0 0 2024-02-28T16:31:09.947 INFO:tasks.workunit.client.1.smithi066.stdout:0/145: mknod d2/d9/d10/c2f 0 2024-02-28T16:31:09.949 INFO:tasks.workunit.client.1.smithi066.stdout:6/31: dwrite f1 [0,4194304] 0 2024-02-28T16:31:09.952 INFO:tasks.workunit.client.0.smithi001.stdout:2/523: dread d0/d2/d3/d36/f4b [0,4194304] 0 2024-02-28T16:31:09.953 INFO:tasks.workunit.client.1.smithi066.stdout:0/146: write d2/f28 [1017838,83076] 0 2024-02-28T16:31:09.957 INFO:tasks.workunit.client.1.smithi066.stdout:9/55: dread da/fc [0,4194304] 0 2024-02-28T16:31:09.958 INFO:tasks.workunit.client.0.smithi001.stdout:2/524: creat d0/d2/d3/de/d61/d12/d55/fb1 x:0 0 0 2024-02-28T16:31:09.961 INFO:tasks.workunit.client.0.smithi001.stdout:0/488: dwrite d18/d22/d2b/d36/d46/f4b [0,4194304] 0 2024-02-28T16:31:09.963 INFO:tasks.workunit.client.1.smithi066.stdout:7/31: getdents . 0 2024-02-28T16:31:09.964 INFO:tasks.workunit.client.1.smithi066.stdout:2/83: truncate d0/d12/f16 730196 0 2024-02-28T16:31:09.964 INFO:tasks.workunit.client.0.smithi001.stdout:0/489: write d18/d22/d3d/f67 [650279,104965] 0 2024-02-28T16:31:09.966 INFO:tasks.workunit.client.0.smithi001.stdout:2/525: mkdir d0/d2/d3/de/d61/d12/d55/db2 0 2024-02-28T16:31:09.967 INFO:tasks.workunit.client.0.smithi001.stdout:0/490: write d18/d31/f43 [3268483,40020] 0 2024-02-28T16:31:09.971 INFO:tasks.workunit.client.1.smithi066.stdout:4/107: dwrite d1/f12 [0,4194304] 0 2024-02-28T16:31:09.971 INFO:tasks.workunit.client.1.smithi066.stdout:0/147: creat d2/d9/d10/d23/f30 x:0 0 0 2024-02-28T16:31:09.972 INFO:tasks.workunit.client.0.smithi001.stdout:2/526: fdatasync d0/d2/d3/de/d22/f48 0 2024-02-28T16:31:09.972 INFO:tasks.workunit.client.0.smithi001.stdout:0/491: write d18/d22/d2b/d36/d46/f4b [4839236,113725] 0 2024-02-28T16:31:09.974 INFO:tasks.workunit.client.0.smithi001.stdout:2/527: chown d0/d2/d3/de/d61/c25 4170 1 2024-02-28T16:31:09.975 INFO:tasks.workunit.client.1.smithi066.stdout:8/39: dwrite d0/d6/f7 [0,4194304] 0 2024-02-28T16:31:09.975 INFO:tasks.workunit.client.0.smithi001.stdout:0/492: creat d18/d22/d3d/fa1 x:0 0 0 2024-02-28T16:31:09.976 INFO:tasks.workunit.client.0.smithi001.stdout:0/493: dread - d18/d31/f7f zero size 2024-02-28T16:31:09.977 INFO:tasks.workunit.client.1.smithi066.stdout:4/108: dread - d1/f1e zero size 2024-02-28T16:31:09.978 INFO:tasks.workunit.client.1.smithi066.stdout:2/84: mknod d0/d12/c17 0 2024-02-28T16:31:09.979 INFO:tasks.workunit.client.0.smithi001.stdout:0/494: creat d18/d22/d47/fa2 x:0 0 0 2024-02-28T16:31:09.980 INFO:tasks.workunit.client.1.smithi066.stdout:0/148: write d2/d9/f21 [1858819,73342] 0 2024-02-28T16:31:09.981 INFO:tasks.workunit.client.0.smithi001.stdout:1/612: sync 2024-02-28T16:31:09.981 INFO:tasks.workunit.client.0.smithi001.stdout:8/490: sync 2024-02-28T16:31:09.982 INFO:tasks.workunit.client.1.smithi066.stdout:8/40: truncate d0/d6/f9 885677 0 2024-02-28T16:31:09.984 INFO:tasks.workunit.client.1.smithi066.stdout:9/56: link da/fc da/d10/f11 0 2024-02-28T16:31:09.986 INFO:tasks.workunit.client.0.smithi001.stdout:1/613: read d0/d9/d1a/d2b/d26/f82 [2943976,127626] 0 2024-02-28T16:31:09.987 INFO:tasks.workunit.client.0.smithi001.stdout:0/495: truncate d18/d22/d2b/f88 1488827 0 2024-02-28T16:31:09.987 INFO:tasks.workunit.client.0.smithi001.stdout:8/491: dread - d4/d7/d22/d27/d58/f95 zero size 2024-02-28T16:31:09.989 INFO:tasks.workunit.client.0.smithi001.stdout:8/492: creat d4/d47/fa0 x:0 0 0 2024-02-28T16:31:09.991 INFO:tasks.workunit.client.0.smithi001.stdout:0/496: write d18/d22/d2b/d36/f69 [2907538,126486] 0 2024-02-28T16:31:09.992 INFO:tasks.workunit.client.0.smithi001.stdout:1/614: unlink d0/d9/d1a/d2b/d26/d27/d30/d52/l92 0 2024-02-28T16:31:09.992 INFO:tasks.workunit.client.0.smithi001.stdout:0/497: fdatasync d18/d22/d47/fa2 0 2024-02-28T16:31:09.993 INFO:tasks.workunit.client.1.smithi066.stdout:8/41: truncate d0/fa 1257740 0 2024-02-28T16:31:09.993 INFO:tasks.workunit.client.0.smithi001.stdout:8/493: rmdir d4/d7/d22/d27/d4b/d6b 39 2024-02-28T16:31:09.995 INFO:tasks.workunit.client.0.smithi001.stdout:0/498: mknod d18/d31/d4f/ca3 0 2024-02-28T16:31:09.995 INFO:tasks.workunit.client.1.smithi066.stdout:4/109: creat d1/f20 x:0 0 0 2024-02-28T16:31:09.996 INFO:tasks.workunit.client.1.smithi066.stdout:9/57: symlink da/df/l12 0 2024-02-28T16:31:09.997 INFO:tasks.workunit.client.0.smithi001.stdout:8/494: chown d4/d7/d22/d27/d58/f75 0 1 2024-02-28T16:31:09.999 INFO:tasks.workunit.client.0.smithi001.stdout:1/615: mkdir d0/d9/d1a/d2b/d26/d27/dd3 0 2024-02-28T16:31:09.999 INFO:tasks.workunit.client.1.smithi066.stdout:9/58: write da/fb [3342171,111942] 0 2024-02-28T16:31:10.000 INFO:tasks.workunit.client.0.smithi001.stdout:0/499: creat d18/d22/d2f/fa4 x:0 0 0 2024-02-28T16:31:10.002 INFO:tasks.workunit.client.0.smithi001.stdout:8/495: truncate d4/f13 2709926 0 2024-02-28T16:31:10.003 INFO:tasks.workunit.client.0.smithi001.stdout:0/500: mknod d18/d31/d4f/ca5 0 2024-02-28T16:31:10.004 INFO:tasks.workunit.client.0.smithi001.stdout:8/496: fdatasync d4/f59 0 2024-02-28T16:31:10.004 INFO:tasks.workunit.client.1.smithi066.stdout:2/85: truncate d0/f9 6032337 0 2024-02-28T16:31:10.006 INFO:tasks.workunit.client.1.smithi066.stdout:0/149: dread d2/d9/d10/f2e [0,4194304] 0 2024-02-28T16:31:10.007 INFO:tasks.workunit.client.0.smithi001.stdout:1/616: creat d0/d9/d1a/d2b/d26/d27/d30/d67/d87/fd4 x:0 0 0 2024-02-28T16:31:10.007 INFO:tasks.workunit.client.0.smithi001.stdout:8/497: write d4/d32/d69/f6f [488267,126695] 0 2024-02-28T16:31:10.009 INFO:tasks.workunit.client.1.smithi066.stdout:0/150: stat d2/c2b 0 2024-02-28T16:31:10.009 INFO:tasks.workunit.client.1.smithi066.stdout:2/86: fdatasync d0/f8 0 2024-02-28T16:31:10.010 INFO:tasks.workunit.client.0.smithi001.stdout:0/501: rename d18/d22/d2f/c3a to d18/d22/d2b/d5f/ca6 0 2024-02-28T16:31:10.011 INFO:tasks.workunit.client.0.smithi001.stdout:1/617: dread d0/d9/f85 [0,4194304] 0 2024-02-28T16:31:10.012 INFO:tasks.workunit.client.0.smithi001.stdout:8/498: truncate d4/d7/d22/d27/d4b/f63 352986 0 2024-02-28T16:31:10.015 INFO:tasks.workunit.client.0.smithi001.stdout:0/502: mknod d18/d22/d2b/d36/d46/ca7 0 2024-02-28T16:31:10.015 INFO:tasks.workunit.client.1.smithi066.stdout:2/87: write d0/d12/f14 [2434873,93868] 0 2024-02-28T16:31:10.017 INFO:tasks.workunit.client.0.smithi001.stdout:1/618: mknod d0/d9/d1a/d2b/d26/d27/d30/d67/d87/cd5 0 2024-02-28T16:31:10.017 INFO:tasks.workunit.client.1.smithi066.stdout:9/59: write da/fc [2994468,6688] 0 2024-02-28T16:31:10.018 INFO:tasks.workunit.client.1.smithi066.stdout:4/110: dread d1/f4 [0,4194304] 0 2024-02-28T16:31:10.019 INFO:tasks.workunit.client.1.smithi066.stdout:2/88: chown d0/f8 19669113 1 2024-02-28T16:31:10.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:09 smithi001 ceph-mon[28991]: pgmap v48: 65 pgs: 65 active+clean; 1.0 GiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 20 MiB/s rd, 110 MiB/s wr, 252 op/s 2024-02-28T16:31:10.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:09 smithi001 ceph-mon[28991]: mgrmap e25: smithi066.ztraxe(active, since 90s), standbys: smithi001.rmthtx 2024-02-28T16:31:10.020 INFO:tasks.workunit.client.0.smithi001.stdout:8/499: creat d4/fa1 x:0 0 0 2024-02-28T16:31:10.022 INFO:tasks.workunit.client.0.smithi001.stdout:1/619: write d0/d9/d1a/d2b/d26/d27/d30/d52/fd0 [985690,67827] 0 2024-02-28T16:31:10.022 INFO:tasks.workunit.client.0.smithi001.stdout:0/503: rename d18/d22/c92 to d18/d22/d57/ca8 0 2024-02-28T16:31:10.022 INFO:tasks.workunit.client.1.smithi066.stdout:1/95: sync 2024-02-28T16:31:10.026 INFO:tasks.workunit.client.0.smithi001.stdout:0/504: mkdir d18/da9 0 2024-02-28T16:31:10.027 INFO:tasks.workunit.client.1.smithi066.stdout:2/89: unlink d0/d12/f16 0 2024-02-28T16:31:10.031 INFO:tasks.workunit.client.1.smithi066.stdout:4/111: symlink d1/l21 0 2024-02-28T16:31:10.031 INFO:tasks.workunit.client.1.smithi066.stdout:8/42: dwrite d0/f4 [0,4194304] 0 2024-02-28T16:31:10.032 INFO:tasks.workunit.client.0.smithi001.stdout:1/620: mkdir d0/d9/d1a/d2b/d26/d27/d30/d67/dd6 0 2024-02-28T16:31:10.034 INFO:tasks.workunit.client.1.smithi066.stdout:4/112: readlink d1/l21 0 2024-02-28T16:31:10.036 INFO:tasks.workunit.client.0.smithi001.stdout:1/621: unlink d0/d9/d56/f8f 0 2024-02-28T16:31:10.037 INFO:tasks.workunit.client.0.smithi001.stdout:0/505: getdents d18/d22/d76 0 2024-02-28T16:31:10.038 INFO:tasks.workunit.client.0.smithi001.stdout:8/500: dwrite d4/d47/fa0 [0,4194304] 0 2024-02-28T16:31:10.038 INFO:tasks.workunit.client.0.smithi001.stdout:2/528: dread d0/d2/d3/de/d61/d12/d46/faf [0,4194304] 0 2024-02-28T16:31:10.043 INFO:tasks.workunit.client.1.smithi066.stdout:2/90: rename d0/d12/c17 to d0/d12/c18 0 2024-02-28T16:31:10.045 INFO:tasks.workunit.client.0.smithi001.stdout:8/501: stat d4/d7/d22/l5d 0 2024-02-28T16:31:10.046 INFO:tasks.workunit.client.0.smithi001.stdout:1/622: write d0/d9/d1a/d2b/d26/d27/d30/d52/d84/f94 [2791414,19931] 0 2024-02-28T16:31:10.048 INFO:tasks.workunit.client.0.smithi001.stdout:1/623: chown d0/l41 13349578 1 2024-02-28T16:31:10.049 INFO:tasks.workunit.client.1.smithi066.stdout:2/91: chown d0/c10 23 1 2024-02-28T16:31:10.052 INFO:tasks.workunit.client.1.smithi066.stdout:0/151: dwrite d2/d9/f11 [0,4194304] 0 2024-02-28T16:31:10.057 INFO:tasks.workunit.client.1.smithi066.stdout:2/92: creat d0/f19 x:0 0 0 2024-02-28T16:31:10.058 INFO:tasks.workunit.client.1.smithi066.stdout:4/113: link d1/l15 d1/l22 0 2024-02-28T16:31:10.058 INFO:tasks.workunit.client.0.smithi001.stdout:1/624: symlink d0/d9/d1a/d2b/d26/d27/d30/d88/db6/ld7 0 2024-02-28T16:31:10.061 INFO:tasks.workunit.client.0.smithi001.stdout:8/502: dwrite d4/d32/f7c [0,4194304] 0 2024-02-28T16:31:10.062 INFO:tasks.workunit.client.1.smithi066.stdout:4/114: fdatasync d1/fe 0 2024-02-28T16:31:10.064 INFO:tasks.workunit.client.0.smithi001.stdout:1/625: write d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/fc6 [423295,93749] 0 2024-02-28T16:31:10.065 INFO:tasks.workunit.client.1.smithi066.stdout:4/115: readlink d1/l1f 0 2024-02-28T16:31:10.065 INFO:tasks.workunit.client.1.smithi066.stdout:1/96: dwrite d2/f16 [0,4194304] 0 2024-02-28T16:31:10.066 INFO:tasks.workunit.client.1.smithi066.stdout:9/60: dread da/fe [0,4194304] 0 2024-02-28T16:31:10.066 INFO:tasks.workunit.client.0.smithi001.stdout:0/506: write d18/d5a/f63 [1047147,114513] 0 2024-02-28T16:31:10.066 INFO:tasks.workunit.client.0.smithi001.stdout:8/503: chown d4/d7/d22/d27/d4b/c57 100557 1 2024-02-28T16:31:10.068 INFO:tasks.workunit.client.1.smithi066.stdout:4/116: fsync d1/f1e 0 2024-02-28T16:31:10.068 INFO:tasks.workunit.client.0.smithi001.stdout:8/504: fdatasync d4/d32/f5c 0 2024-02-28T16:31:10.069 INFO:tasks.workunit.client.1.smithi066.stdout:1/97: fdatasync d2/f17 0 2024-02-28T16:31:10.071 INFO:tasks.workunit.client.0.smithi001.stdout:0/507: fsync d18/d22/f66 0 2024-02-28T16:31:10.072 INFO:tasks.workunit.client.1.smithi066.stdout:8/43: dread d0/f4 [0,4194304] 0 2024-02-28T16:31:10.075 INFO:tasks.workunit.client.0.smithi001.stdout:8/505: write d4/f3b [3204317,78369] 0 2024-02-28T16:31:10.075 INFO:tasks.workunit.client.1.smithi066.stdout:0/152: rename d2/d9/d10/l1b to d2/l31 0 2024-02-28T16:31:10.080 INFO:tasks.workunit.client.1.smithi066.stdout:8/44: dread d0/d6/f9 [0,4194304] 0 2024-02-28T16:31:10.082 INFO:tasks.workunit.client.0.smithi001.stdout:8/506: creat d4/d32/d93/fa2 x:0 0 0 2024-02-28T16:31:10.082 INFO:tasks.workunit.client.0.smithi001.stdout:2/529: dwrite d0/d2/d3/de/d61/d12/d6a/f81 [0,4194304] 0 2024-02-28T16:31:10.083 INFO:tasks.workunit.client.1.smithi066.stdout:4/117: rmdir d1/d18 39 2024-02-28T16:31:10.086 INFO:tasks.workunit.client.1.smithi066.stdout:4/118: write d1/f4 [696499,130113] 0 2024-02-28T16:31:10.097 INFO:tasks.workunit.client.1.smithi066.stdout:9/61: dread da/fb [0,4194304] 0 2024-02-28T16:31:10.097 INFO:tasks.workunit.client.1.smithi066.stdout:0/153: creat d2/f32 x:0 0 0 2024-02-28T16:31:10.097 INFO:tasks.workunit.client.1.smithi066.stdout:4/119: mkdir d1/d23 0 2024-02-28T16:31:10.098 INFO:tasks.workunit.client.0.smithi001.stdout:1/626: dwrite d0/d9/d56/d2f/f7c [4194304,4194304] 0 2024-02-28T16:31:10.099 INFO:tasks.workunit.client.0.smithi001.stdout:8/507: creat d4/d7/d22/d27/d4b/fa3 x:0 0 0 2024-02-28T16:31:10.101 INFO:tasks.workunit.client.1.smithi066.stdout:4/120: fdatasync d1/f1b 0 2024-02-28T16:31:10.102 INFO:tasks.workunit.client.0.smithi001.stdout:5/465: sync 2024-02-28T16:31:10.102 INFO:tasks.workunit.client.0.smithi001.stdout:7/490: sync 2024-02-28T16:31:10.103 INFO:tasks.workunit.client.1.smithi066.stdout:2/93: getdents d0/d12 0 2024-02-28T16:31:10.103 INFO:tasks.workunit.client.0.smithi001.stdout:2/530: write d0/d2/d3/de/d8a/f2e [240448,60] 0 2024-02-28T16:31:10.104 INFO:tasks.workunit.client.0.smithi001.stdout:7/491: chown d3/d17/d3d/d42/l5a 166631 1 2024-02-28T16:31:10.106 INFO:tasks.workunit.client.0.smithi001.stdout:7/492: chown d3/d17/d61/c7b 2055484 1 2024-02-28T16:31:10.108 INFO:tasks.workunit.client.1.smithi066.stdout:0/154: write d2/d9/d10/f2e [2175479,57105] 0 2024-02-28T16:31:10.108 INFO:tasks.workunit.client.0.smithi001.stdout:1/627: rename d0/d9/d56/d2f/cc8 to d0/d9/d1a/d2b/d26/d27/dd3/cd8 0 2024-02-28T16:31:10.109 INFO:tasks.workunit.client.1.smithi066.stdout:4/121: fsync d1/f5 0 2024-02-28T16:31:10.109 INFO:tasks.workunit.client.0.smithi001.stdout:7/493: fdatasync d3/d17/d3d/d5d/f73 0 2024-02-28T16:31:10.110 INFO:tasks.workunit.client.0.smithi001.stdout:2/531: creat d0/d2/d3/de/d61/fb3 x:0 0 0 2024-02-28T16:31:10.111 INFO:tasks.workunit.client.1.smithi066.stdout:2/94: mkdir d0/d12/d1a 0 2024-02-28T16:31:10.112 INFO:tasks.workunit.client.0.smithi001.stdout:5/466: mkdir dc/df/d79/d98 0 2024-02-28T16:31:10.113 INFO:tasks.workunit.client.1.smithi066.stdout:1/98: dwrite d2/f4 [0,4194304] 0 2024-02-28T16:31:10.116 INFO:tasks.workunit.client.1.smithi066.stdout:8/45: dwrite d0/d6/f9 [0,4194304] 0 2024-02-28T16:31:10.118 INFO:tasks.workunit.client.1.smithi066.stdout:2/95: chown d0/d12 212 1 2024-02-28T16:31:10.121 INFO:tasks.workunit.client.0.smithi001.stdout:7/494: write d3/f45 [569047,129486] 0 2024-02-28T16:31:10.121 INFO:tasks.workunit.client.0.smithi001.stdout:1/628: mknod d0/d9/d1a/d2b/d26/cd9 0 2024-02-28T16:31:10.121 INFO:tasks.workunit.client.1.smithi066.stdout:4/122: chown c0 1240433 1 2024-02-28T16:31:10.122 INFO:tasks.workunit.client.0.smithi001.stdout:2/532: creat d0/d2/d3/de/d61/fb4 x:0 0 0 2024-02-28T16:31:10.123 INFO:tasks.workunit.client.0.smithi001.stdout:5/467: creat dc/df/f99 x:0 0 0 2024-02-28T16:31:10.125 INFO:tasks.workunit.client.0.smithi001.stdout:0/508: dread f13 [0,4194304] 0 2024-02-28T16:31:10.129 INFO:tasks.workunit.client.1.smithi066.stdout:8/46: unlink d0/fd 0 2024-02-28T16:31:10.131 INFO:tasks.workunit.client.0.smithi001.stdout:5/468: creat dc/d12/d41/d54/f9a x:0 0 0 2024-02-28T16:31:10.131 INFO:tasks.workunit.client.0.smithi001.stdout:7/495: symlink d3/d17/l9e 0 2024-02-28T16:31:10.131 INFO:tasks.workunit.client.0.smithi001.stdout:5/469: stat dc 0 2024-02-28T16:31:10.132 INFO:tasks.workunit.client.0.smithi001.stdout:0/509: rmdir d18/d31/d4f 39 2024-02-28T16:31:10.132 INFO:tasks.workunit.client.0.smithi001.stdout:5/470: write dc/f30 [280940,56281] 0 2024-02-28T16:31:10.134 INFO:tasks.workunit.client.0.smithi001.stdout:1/629: mknod d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/cda 0 2024-02-28T16:31:10.136 INFO:tasks.workunit.client.0.smithi001.stdout:7/496: creat d3/d17/d3d/d59/d6e/f9f x:0 0 0 2024-02-28T16:31:10.139 INFO:tasks.workunit.client.1.smithi066.stdout:0/155: getdents d2/d9/d10 0 2024-02-28T16:31:10.139 INFO:tasks.workunit.client.0.smithi001.stdout:5/471: mknod dc/d1a/d21/d2d/c9b 0 2024-02-28T16:31:10.140 INFO:tasks.workunit.client.1.smithi066.stdout:4/123: truncate d1/f6 791605 0 2024-02-28T16:31:10.141 INFO:tasks.workunit.client.0.smithi001.stdout:1/630: creat d0/d9/d56/d2f/fdb x:0 0 0 2024-02-28T16:31:10.143 INFO:tasks.workunit.client.0.smithi001.stdout:7/497: symlink d3/d17/d3d/d5d/d9a/la0 0 2024-02-28T16:31:10.145 INFO:tasks.workunit.client.0.smithi001.stdout:5/472: creat dc/d12/d41/f9c x:0 0 0 2024-02-28T16:31:10.146 INFO:tasks.workunit.client.1.smithi066.stdout:4/124: fsync d1/f7 0 2024-02-28T16:31:10.146 INFO:tasks.workunit.client.1.smithi066.stdout:9/62: dwrite da/fb [4194304,4194304] 0 2024-02-28T16:31:10.148 INFO:tasks.workunit.client.0.smithi001.stdout:1/631: write d0/d9/d1a/d2b/d26/d27/d30/d52/f5a [1120489,12762] 0 2024-02-28T16:31:10.148 INFO:tasks.workunit.client.1.smithi066.stdout:5/79: sync 2024-02-28T16:31:10.149 INFO:tasks.workunit.client.0.smithi001.stdout:8/508: dread d4/d7/d22/f3f [0,4194304] 0 2024-02-28T16:31:10.149 INFO:tasks.workunit.client.1.smithi066.stdout:9/63: chown da/d10 103303484 1 2024-02-28T16:31:10.150 INFO:tasks.workunit.client.0.smithi001.stdout:4/535: sync 2024-02-28T16:31:10.150 INFO:tasks.workunit.client.0.smithi001.stdout:3/516: sync 2024-02-28T16:31:10.150 INFO:tasks.workunit.client.0.smithi001.stdout:6/477: sync 2024-02-28T16:31:10.150 INFO:tasks.workunit.client.0.smithi001.stdout:9/605: sync 2024-02-28T16:31:10.152 INFO:tasks.workunit.client.1.smithi066.stdout:0/156: chown d2/d9/c13 3718418 1 2024-02-28T16:31:10.153 INFO:tasks.workunit.client.0.smithi001.stdout:4/536: read - d3/d5/d7/d1f/d3a/f70 zero size 2024-02-28T16:31:10.154 INFO:tasks.workunit.client.0.smithi001.stdout:2/533: truncate d0/fd 3977368 0 2024-02-28T16:31:10.156 INFO:tasks.workunit.client.1.smithi066.stdout:1/99: dread d2/f1d [0,4194304] 0 2024-02-28T16:31:10.156 INFO:tasks.workunit.client.0.smithi001.stdout:0/510: dwrite d18/d22/d2b/d36/d48/f8a [0,4194304] 0 2024-02-28T16:31:10.157 INFO:tasks.workunit.client.1.smithi066.stdout:1/100: fdatasync d2/f4 0 2024-02-28T16:31:10.158 INFO:tasks.workunit.client.0.smithi001.stdout:3/517: creat d4/d1e/d7c/da8/fae x:0 0 0 2024-02-28T16:31:10.159 INFO:tasks.workunit.client.0.smithi001.stdout:9/606: fsync d5/d27/d42/f7e 0 2024-02-28T16:31:10.159 INFO:tasks.workunit.client.1.smithi066.stdout:4/125: rename c0 to d1/d23/c24 0 2024-02-28T16:31:10.160 INFO:tasks.workunit.client.0.smithi001.stdout:5/473: rename dc/d1a/d52/f72 to dc/d1a/d21/d2d/f9d 0 2024-02-28T16:31:10.162 INFO:tasks.workunit.client.1.smithi066.stdout:0/157: mkdir d2/d9/d33 0 2024-02-28T16:31:10.163 INFO:tasks.workunit.client.1.smithi066.stdout:2/96: dwrite d0/ff [0,4194304] 0 2024-02-28T16:31:10.164 INFO:tasks.workunit.client.1.smithi066.stdout:8/47: dread d0/d6/f7 [0,4194304] 0 2024-02-28T16:31:10.166 INFO:tasks.workunit.client.1.smithi066.stdout:1/101: write d2/f7 [8475423,46788] 0 2024-02-28T16:31:10.167 INFO:tasks.workunit.client.1.smithi066.stdout:9/64: creat da/d10/f13 x:0 0 0 2024-02-28T16:31:10.170 INFO:tasks.workunit.client.1.smithi066.stdout:5/80: rename d3/f15 to d3/d4/dc/f1e 0 2024-02-28T16:31:10.171 INFO:tasks.workunit.client.1.smithi066.stdout:9/65: fdatasync da/fb 0 2024-02-28T16:31:10.173 INFO:tasks.workunit.client.1.smithi066.stdout:5/81: fdatasync d3/d4/fb 0 2024-02-28T16:31:10.175 INFO:tasks.workunit.client.1.smithi066.stdout:9/66: fdatasync da/fc 0 2024-02-28T16:31:10.177 INFO:tasks.workunit.client.1.smithi066.stdout:8/48: write d0/f4 [3794900,73974] 0 2024-02-28T16:31:10.179 INFO:tasks.workunit.client.1.smithi066.stdout:0/158: unlink d2/d9/d10/d23/l24 0 2024-02-28T16:31:10.181 INFO:tasks.workunit.client.1.smithi066.stdout:5/82: mknod d3/d4/dc/c1f 0 2024-02-28T16:31:10.186 INFO:tasks.workunit.client.1.smithi066.stdout:1/102: getdents d2 0 2024-02-28T16:31:10.186 INFO:tasks.workunit.client.1.smithi066.stdout:0/159: dread d2/d9/d10/f26 [0,4194304] 0 2024-02-28T16:31:10.188 INFO:tasks.workunit.client.1.smithi066.stdout:0/160: readlink d2/d9/lb 0 2024-02-28T16:31:10.189 INFO:tasks.workunit.client.1.smithi066.stdout:1/103: write d2/d18/f21 [595521,49575] 0 2024-02-28T16:31:10.189 INFO:tasks.workunit.client.1.smithi066.stdout:5/83: mkdir d3/d1b/d20 0 2024-02-28T16:31:10.192 INFO:tasks.workunit.client.1.smithi066.stdout:8/49: rename d0/fa to d0/d6/fe 0 2024-02-28T16:31:10.193 INFO:tasks.workunit.client.1.smithi066.stdout:5/84: chown d3/d4/ca 0 1 2024-02-28T16:31:10.195 INFO:tasks.workunit.client.1.smithi066.stdout:5/85: chown d3/d4/dc/f1d 109391 1 2024-02-28T16:31:10.195 INFO:tasks.workunit.client.1.smithi066.stdout:1/104: mknod d2/c22 0 2024-02-28T16:31:10.196 INFO:tasks.workunit.client.1.smithi066.stdout:9/67: dread da/d10/f11 [0,4194304] 0 2024-02-28T16:31:10.197 INFO:tasks.workunit.client.1.smithi066.stdout:0/161: symlink d2/d9/d33/l34 0 2024-02-28T16:31:10.198 INFO:tasks.workunit.client.1.smithi066.stdout:1/105: write d2/d18/d19/f1c [6775876,51084] 0 2024-02-28T16:31:10.199 INFO:tasks.workunit.client.1.smithi066.stdout:4/126: truncate d1/f11 1015291 0 2024-02-28T16:31:10.199 INFO:tasks.workunit.client.1.smithi066.stdout:1/106: stat d2/f4 0 2024-02-28T16:31:10.201 INFO:tasks.workunit.client.1.smithi066.stdout:1/107: fsync d2/ff 0 2024-02-28T16:31:10.201 INFO:tasks.workunit.client.1.smithi066.stdout:9/68: rmdir da/df 39 2024-02-28T16:31:10.204 INFO:tasks.workunit.client.1.smithi066.stdout:1/108: write d2/f10 [156701,98483] 0 2024-02-28T16:31:10.205 INFO:tasks.workunit.client.1.smithi066.stdout:0/162: creat d2/d9/f35 x:0 0 0 2024-02-28T16:31:10.207 INFO:tasks.workunit.client.1.smithi066.stdout:1/109: chown d2/l8 991 1 2024-02-28T16:31:10.209 INFO:tasks.workunit.client.1.smithi066.stdout:4/127: unlink d1/f1b 0 2024-02-28T16:31:10.211 INFO:tasks.workunit.client.1.smithi066.stdout:9/69: write f8 [7648876,115054] 0 2024-02-28T16:31:10.213 INFO:tasks.workunit.client.1.smithi066.stdout:5/86: dwrite d3/ff [0,4194304] 0 2024-02-28T16:31:10.217 INFO:tasks.workunit.client.1.smithi066.stdout:1/110: creat d2/f23 x:0 0 0 2024-02-28T16:31:10.222 INFO:tasks.workunit.client.1.smithi066.stdout:5/87: rmdir d3/d4 39 2024-02-28T16:31:10.225 INFO:tasks.workunit.client.1.smithi066.stdout:1/111: creat d2/d18/d19/f24 x:0 0 0 2024-02-28T16:31:10.229 INFO:tasks.workunit.client.1.smithi066.stdout:9/70: write f3 [1063232,103814] 0 2024-02-28T16:31:10.233 INFO:tasks.workunit.client.1.smithi066.stdout:1/112: write d2/f23 [758430,129368] 0 2024-02-28T16:31:10.238 INFO:tasks.workunit.client.1.smithi066.stdout:9/71: symlink da/l14 0 2024-02-28T16:31:10.239 INFO:tasks.workunit.client.1.smithi066.stdout:0/163: dwrite d2/d9/f21 [0,4194304] 0 2024-02-28T16:31:10.244 INFO:tasks.workunit.client.1.smithi066.stdout:4/128: dwrite d1/fe [0,4194304] 0 2024-02-28T16:31:10.244 INFO:tasks.workunit.client.1.smithi066.stdout:1/113: mknod d2/d18/c25 0 2024-02-28T16:31:10.251 INFO:tasks.workunit.client.1.smithi066.stdout:9/72: unlink da/fe 0 2024-02-28T16:31:10.256 INFO:tasks.workunit.client.1.smithi066.stdout:4/129: mkdir d1/d23/d25 0 2024-02-28T16:31:10.258 INFO:tasks.workunit.client.0.smithi001.stdout:7/498: dread d3/d17/d3d/d59/f6b [0,4194304] 0 2024-02-28T16:31:10.259 INFO:tasks.workunit.client.0.smithi001.stdout:2/534: mknod d0/d2/d3/de/d8a/cb5 0 2024-02-28T16:31:10.260 INFO:tasks.workunit.client.1.smithi066.stdout:1/114: creat d2/d18/f26 x:0 0 0 2024-02-28T16:31:10.261 INFO:tasks.workunit.client.0.smithi001.stdout:9/607: fsync d5/d27/db1/fa4 0 2024-02-28T16:31:10.261 INFO:tasks.workunit.client.0.smithi001.stdout:4/537: creat d3/d2d/d81/fae x:0 0 0 2024-02-28T16:31:10.264 INFO:tasks.workunit.client.0.smithi001.stdout:7/499: fsync d3/d17/d61/f71 0 2024-02-28T16:31:10.265 INFO:tasks.workunit.client.1.smithi066.stdout:4/130: rename d1/cd to d1/d23/d25/c26 0 2024-02-28T16:31:10.266 INFO:tasks.workunit.client.0.smithi001.stdout:4/538: write d3/d5/faa [149369,121046] 0 2024-02-28T16:31:10.267 INFO:tasks.workunit.client.1.smithi066.stdout:1/115: dread - d2/d18/d19/f1e zero size 2024-02-28T16:31:10.268 INFO:tasks.workunit.client.1.smithi066.stdout:5/88: dwrite d3/d4/f5 [0,4194304] 0 2024-02-28T16:31:10.268 INFO:tasks.workunit.client.0.smithi001.stdout:0/511: rename d18/d5a/f72 to d18/d22/d57/d7e/faa 0 2024-02-28T16:31:10.269 INFO:tasks.workunit.client.0.smithi001.stdout:5/474: mknod dc/d1a/d7d/c9e 0 2024-02-28T16:31:10.270 INFO:tasks.workunit.client.1.smithi066.stdout:1/116: mkdir d2/d27 0 2024-02-28T16:31:10.271 INFO:tasks.workunit.client.0.smithi001.stdout:2/535: chown d0/d2/d3/d36/ca8 1495 1 2024-02-28T16:31:10.273 INFO:tasks.workunit.client.1.smithi066.stdout:5/89: rmdir d3/d4 39 2024-02-28T16:31:10.274 INFO:tasks.workunit.client.0.smithi001.stdout:1/632: creat d0/d9/d1a/d2b/d26/d27/d30/d67/dd6/fdc x:0 0 0 2024-02-28T16:31:10.275 INFO:tasks.workunit.client.0.smithi001.stdout:8/509: creat d4/d7/d22/d27/d4b/fa4 x:0 0 0 2024-02-28T16:31:10.275 INFO:tasks.workunit.client.1.smithi066.stdout:1/117: symlink d2/l28 0 2024-02-28T16:31:10.276 INFO:tasks.workunit.client.0.smithi001.stdout:7/500: mknod d3/d17/d61/ca1 0 2024-02-28T16:31:10.277 INFO:tasks.workunit.client.0.smithi001.stdout:3/518: creat d4/d18/faf x:0 0 0 2024-02-28T16:31:10.279 INFO:tasks.workunit.client.0.smithi001.stdout:0/512: write d18/d22/d2b/f41 [2036887,57892] 0 2024-02-28T16:31:10.279 INFO:tasks.workunit.client.1.smithi066.stdout:9/73: dwrite f3 [0,4194304] 0 2024-02-28T16:31:10.280 INFO:tasks.workunit.client.0.smithi001.stdout:5/475: write dc/d12/f32 [1794947,93202] 0 2024-02-28T16:31:10.280 INFO:tasks.workunit.client.0.smithi001.stdout:2/536: symlink d0/d2/d3/de/d22/lb6 0 2024-02-28T16:31:10.281 INFO:tasks.workunit.client.0.smithi001.stdout:1/633: read - d0/d9/f5c zero size 2024-02-28T16:31:10.282 INFO:tasks.workunit.client.1.smithi066.stdout:9/74: chown da/l14 444708467 1 2024-02-28T16:31:10.282 INFO:tasks.workunit.client.1.smithi066.stdout:0/164: truncate d2/fe 4762564 0 2024-02-28T16:31:10.283 INFO:tasks.workunit.client.0.smithi001.stdout:9/608: creat d5/d27/db1/d33/d8a/fcd x:0 0 0 2024-02-28T16:31:10.285 INFO:tasks.workunit.client.0.smithi001.stdout:5/476: write dc/d1a/d21/d2d/d3c/d4a/f82 [511623,117509] 0 2024-02-28T16:31:10.285 INFO:tasks.workunit.client.1.smithi066.stdout:9/75: mkdir da/d10/d15 0 2024-02-28T16:31:10.286 INFO:tasks.workunit.client.1.smithi066.stdout:0/165: mknod d2/d9/d10/d23/c36 0 2024-02-28T16:31:10.287 INFO:tasks.workunit.client.0.smithi001.stdout:3/519: dread - d4/d18/f6d zero size 2024-02-28T16:31:10.287 INFO:tasks.workunit.client.0.smithi001.stdout:5/477: stat dc/df/f99 0 2024-02-28T16:31:10.292 INFO:tasks.workunit.client.1.smithi066.stdout:0/166: symlink d2/d27/l37 0 2024-02-28T16:31:10.293 INFO:tasks.workunit.client.0.smithi001.stdout:4/539: rmdir d3/d1c/d58/d93 0 2024-02-28T16:31:10.294 INFO:tasks.workunit.client.0.smithi001.stdout:8/510: mknod d4/d47/d8b/ca5 0 2024-02-28T16:31:10.294 INFO:tasks.workunit.client.1.smithi066.stdout:1/118: dread d2/f4 [4194304,4194304] 0 2024-02-28T16:31:10.296 INFO:tasks.workunit.client.0.smithi001.stdout:9/609: mkdir d5/d9a/dce 0 2024-02-28T16:31:10.300 INFO:tasks.workunit.client.1.smithi066.stdout:1/119: stat d2/d18/d19/f24 0 2024-02-28T16:31:10.300 INFO:tasks.workunit.client.1.smithi066.stdout:0/167: rename d2/d9/lb to d2/d9/d10/l38 0 2024-02-28T16:31:10.300 INFO:tasks.workunit.client.1.smithi066.stdout:6/32: sync 2024-02-28T16:31:10.300 INFO:tasks.workunit.client.1.smithi066.stdout:7/32: sync 2024-02-28T16:31:10.300 INFO:tasks.workunit.client.1.smithi066.stdout:3/43: sync 2024-02-28T16:31:10.301 INFO:tasks.workunit.client.0.smithi001.stdout:2/537: fsync d0/d2/d3/de/d22/f52 0 2024-02-28T16:31:10.301 INFO:tasks.workunit.client.0.smithi001.stdout:3/520: creat d4/d6/d7f/d82/fb0 x:0 0 0 2024-02-28T16:31:10.304 INFO:tasks.workunit.client.1.smithi066.stdout:3/44: mknod ca 0 2024-02-28T16:31:10.305 INFO:tasks.workunit.client.1.smithi066.stdout:3/45: read f5 [2867668,22417] 0 2024-02-28T16:31:10.306 INFO:tasks.workunit.client.0.smithi001.stdout:0/513: rename d18/d5a/f73 to d18/fab 0 2024-02-28T16:31:10.306 INFO:tasks.workunit.client.1.smithi066.stdout:3/46: fsync f3 0 2024-02-28T16:31:10.310 INFO:tasks.workunit.client.0.smithi001.stdout:8/511: symlink d4/d7/d22/d27/d4b/d77/la6 0 2024-02-28T16:31:10.311 INFO:tasks.workunit.client.0.smithi001.stdout:1/634: dwrite d0/d9/d1a/d2b/d26/d27/d30/d67/d87/fd4 [0,4194304] 0 2024-02-28T16:31:10.311 INFO:tasks.workunit.client.1.smithi066.stdout:7/33: mknod c7 0 2024-02-28T16:31:10.311 INFO:tasks.workunit.client.1.smithi066.stdout:3/47: fdatasync f5 0 2024-02-28T16:31:10.312 INFO:tasks.workunit.client.0.smithi001.stdout:9/610: mkdir d5/d27/d42/d48/dcf 0 2024-02-28T16:31:10.312 INFO:tasks.workunit.client.1.smithi066.stdout:3/48: rmdir - no directory 2024-02-28T16:31:10.314 INFO:tasks.workunit.client.1.smithi066.stdout:9/76: dwrite f3 [0,4194304] 0 2024-02-28T16:31:10.315 INFO:tasks.workunit.client.1.smithi066.stdout:7/34: truncate f4 935753 0 2024-02-28T16:31:10.315 INFO:tasks.workunit.client.0.smithi001.stdout:4/540: dwrite d3/d1c/d58/f98 [0,4194304] 0 2024-02-28T16:31:10.318 INFO:tasks.workunit.client.0.smithi001.stdout:8/512: write d4/fa1 [590632,5578] 0 2024-02-28T16:31:10.320 INFO:tasks.workunit.client.1.smithi066.stdout:1/120: link d2/ff d2/d18/d19/f29 0 2024-02-28T16:31:10.320 INFO:tasks.workunit.client.0.smithi001.stdout:5/478: symlink dc/d1a/d21/d2d/d91/l9f 0 2024-02-28T16:31:10.321 INFO:tasks.workunit.client.1.smithi066.stdout:3/49: mkdir db 0 2024-02-28T16:31:10.322 INFO:tasks.workunit.client.0.smithi001.stdout:1/635: read d0/d9/d1a/f33 [3286042,114496] 0 2024-02-28T16:31:10.324 INFO:tasks.workunit.client.1.smithi066.stdout:7/35: rename c6 to c8 0 2024-02-28T16:31:10.324 INFO:tasks.workunit.client.1.smithi066.stdout:0/168: dread d2/d9/d10/f1d [0,4194304] 0 2024-02-28T16:31:10.325 INFO:tasks.workunit.client.0.smithi001.stdout:2/538: dread d0/d2/f3b [0,4194304] 0 2024-02-28T16:31:10.327 INFO:tasks.workunit.client.0.smithi001.stdout:4/541: write d3/d5/d7/d1f/d2e/f39 [4165588,55151] 0 2024-02-28T16:31:10.327 INFO:tasks.workunit.client.0.smithi001.stdout:1/636: chown d0/d9/f48 43145 1 2024-02-28T16:31:10.329 INFO:tasks.workunit.client.0.smithi001.stdout:5/479: mkdir dc/d1a/d21/d2d/d3c/da0 0 2024-02-28T16:31:10.332 INFO:tasks.workunit.client.1.smithi066.stdout:6/33: dwrite d2/f3 [0,4194304] 0 2024-02-28T16:31:10.332 INFO:tasks.workunit.client.1.smithi066.stdout:3/50: mkdir db/dc 0 2024-02-28T16:31:10.333 INFO:tasks.workunit.client.0.smithi001.stdout:3/521: dread d4/d6/d7/d78/d81/f96 [0,4194304] 0 2024-02-28T16:31:10.334 INFO:tasks.workunit.client.0.smithi001.stdout:4/542: write d3/fa [2796685,20920] 0 2024-02-28T16:31:10.336 INFO:tasks.workunit.client.0.smithi001.stdout:2/539: unlink d0/d2/d3/la0 0 2024-02-28T16:31:10.338 INFO:tasks.workunit.client.1.smithi066.stdout:3/51: rmdir db 39 2024-02-28T16:31:10.338 INFO:tasks.workunit.client.1.smithi066.stdout:6/34: creat d2/f8 x:0 0 0 2024-02-28T16:31:10.339 INFO:tasks.workunit.client.1.smithi066.stdout:1/121: dread d2/f17 [0,4194304] 0 2024-02-28T16:31:10.339 INFO:tasks.workunit.client.0.smithi001.stdout:3/522: stat d4/d1e/d7c/d7d 0 2024-02-28T16:31:10.340 INFO:tasks.workunit.client.1.smithi066.stdout:6/35: read - d2/f8 zero size 2024-02-28T16:31:10.343 INFO:tasks.workunit.client.0.smithi001.stdout:2/540: stat d0/d2/d3/f43 0 2024-02-28T16:31:10.343 INFO:tasks.workunit.client.0.smithi001.stdout:8/513: link d4/d7/d22/d27/d4b/d6b/d89/f79 d4/d9b/fa7 0 2024-02-28T16:31:10.347 INFO:tasks.workunit.client.1.smithi066.stdout:1/122: truncate d2/d18/f26 770613 0 2024-02-28T16:31:10.347 INFO:tasks.workunit.client.0.smithi001.stdout:4/543: creat d3/d5/d7/faf x:0 0 0 2024-02-28T16:31:10.347 INFO:tasks.workunit.client.0.smithi001.stdout:8/514: truncate d4/d7/f10 8620876 0 2024-02-28T16:31:10.347 INFO:tasks.workunit.client.0.smithi001.stdout:7/501: dwrite d3/d1c/f33 [4194304,4194304] 0 2024-02-28T16:31:10.353 INFO:tasks.workunit.client.0.smithi001.stdout:8/515: write d4/d7/f6a [449687,48503] 0 2024-02-28T16:31:10.354 INFO:tasks.workunit.client.0.smithi001.stdout:7/502: chown d3/d17/c20 0 1 2024-02-28T16:31:10.356 INFO:tasks.workunit.client.1.smithi066.stdout:0/169: dwrite d2/d9/d10/d23/f30 [0,4194304] 0 2024-02-28T16:31:10.356 INFO:tasks.workunit.client.0.smithi001.stdout:2/541: chown d0/d2/cac 50 1 2024-02-28T16:31:10.357 INFO:tasks.workunit.client.1.smithi066.stdout:1/123: creat d2/d18/d1f/f2a x:0 0 0 2024-02-28T16:31:10.358 INFO:tasks.workunit.client.1.smithi066.stdout:9/77: dwrite da/fb [4194304,4194304] 0 2024-02-28T16:31:10.358 INFO:tasks.workunit.client.1.smithi066.stdout:0/170: chown d2/d9 6005 1 2024-02-28T16:31:10.364 INFO:tasks.workunit.client.0.smithi001.stdout:5/480: dread dc/d1a/d21/d2d/f66 [0,4194304] 0 2024-02-28T16:31:10.364 INFO:tasks.workunit.client.0.smithi001.stdout:7/503: fdatasync d3/d17/d3d/d5d/d9a/d49/f65 0 2024-02-28T16:31:10.364 INFO:tasks.workunit.client.1.smithi066.stdout:0/171: write d2/d9/f35 [331478,20567] 0 2024-02-28T16:31:10.365 INFO:tasks.workunit.client.0.smithi001.stdout:9/611: dwrite d5/d27/d34/fc4 [0,4194304] 0 2024-02-28T16:31:10.368 INFO:tasks.workunit.client.0.smithi001.stdout:8/516: truncate d4/d7/d22/d27/d58/f87 920640 0 2024-02-28T16:31:10.369 INFO:tasks.workunit.client.1.smithi066.stdout:0/172: fdatasync d2/f32 0 2024-02-28T16:31:10.369 INFO:tasks.workunit.client.1.smithi066.stdout:9/78: unlink da/l14 0 2024-02-28T16:31:10.369 INFO:tasks.workunit.client.0.smithi001.stdout:2/542: rename d0/d2/fb to d0/d2/d3/d36/fb7 0 2024-02-28T16:31:10.371 INFO:tasks.workunit.client.0.smithi001.stdout:3/523: dwrite d4/f26 [0,4194304] 0 2024-02-28T16:31:10.380 INFO:tasks.workunit.client.1.smithi066.stdout:7/36: write f5 [1116312,109451] 0 2024-02-28T16:31:10.380 INFO:tasks.workunit.client.1.smithi066.stdout:3/52: symlink db/dc/ld 0 2024-02-28T16:31:10.381 INFO:tasks.workunit.client.1.smithi066.stdout:7/37: stat f5 0 2024-02-28T16:31:10.382 INFO:tasks.workunit.client.1.smithi066.stdout:7/38: stat c7 0 2024-02-28T16:31:10.382 INFO:tasks.workunit.client.1.smithi066.stdout:0/173: getdents d2/d9/d10/d23 0 2024-02-28T16:31:10.382 INFO:tasks.workunit.client.0.smithi001.stdout:3/524: stat d4/d6/d7/c57 0 2024-02-28T16:31:10.383 INFO:tasks.workunit.client.0.smithi001.stdout:7/504: creat d3/d17/d52/d92/fa2 x:0 0 0 2024-02-28T16:31:10.385 INFO:tasks.workunit.client.0.smithi001.stdout:8/517: fsync d4/d32/f46 0 2024-02-28T16:31:10.385 INFO:tasks.workunit.client.1.smithi066.stdout:3/53: creat db/fe x:0 0 0 2024-02-28T16:31:10.386 INFO:tasks.workunit.client.0.smithi001.stdout:2/543: mkdir d0/d2/d3/de/db8 0 2024-02-28T16:31:10.387 INFO:tasks.workunit.client.1.smithi066.stdout:3/54: write f3 [5172468,78665] 0 2024-02-28T16:31:10.387 INFO:tasks.workunit.client.1.smithi066.stdout:0/174: unlink d2/c4 0 2024-02-28T16:31:10.387 INFO:tasks.workunit.client.1.smithi066.stdout:7/39: link f1 f9 0 2024-02-28T16:31:10.392 INFO:tasks.workunit.client.0.smithi001.stdout:7/505: write d3/d17/d52/d92/fa2 [721814,87695] 0 2024-02-28T16:31:10.399 INFO:tasks.workunit.client.0.smithi001.stdout:8/518: readlink d4/l1a 0 2024-02-28T16:31:10.402 INFO:tasks.workunit.client.0.smithi001.stdout:1/637: dwrite d0/d9/d1a/d2b/f6d [0,4194304] 0 2024-02-28T16:31:10.405 INFO:tasks.workunit.client.0.smithi001.stdout:7/506: mknod d3/ca3 0 2024-02-28T16:31:10.413 INFO:tasks.workunit.client.0.smithi001.stdout:5/481: truncate dc/d12/d41/d54/f71 628616 0 2024-02-28T16:31:10.415 INFO:tasks.workunit.client.0.smithi001.stdout:9/612: dread d5/d27/db1/f41 [0,4194304] 0 2024-02-28T16:31:10.416 INFO:tasks.workunit.client.0.smithi001.stdout:8/519: rename d4/d7/d22/d27/d4b/d6b to d4/d7/da8 0 2024-02-28T16:31:10.417 INFO:tasks.workunit.client.0.smithi001.stdout:4/544: dread d3/d1c/d58/f6b [0,4194304] 0 2024-02-28T16:31:10.417 INFO:tasks.workunit.client.0.smithi001.stdout:7/507: rmdir d3/d17/d3d/d5d/d9a/d49/d63 39 2024-02-28T16:31:10.421 INFO:tasks.workunit.client.0.smithi001.stdout:2/544: write d0/f33 [976003,6894] 0 2024-02-28T16:31:10.422 INFO:tasks.workunit.client.0.smithi001.stdout:9/613: dread - d5/d27/d42/d48/fab zero size 2024-02-28T16:31:10.423 INFO:tasks.workunit.client.0.smithi001.stdout:4/545: fsync d3/d5/d7/f21 0 2024-02-28T16:31:10.424 INFO:tasks.workunit.client.0.smithi001.stdout:7/508: write d3/fa [9473542,85716] 0 2024-02-28T16:31:10.426 INFO:tasks.workunit.client.0.smithi001.stdout:1/638: write d0/d9/d1a/f33 [3670672,113530] 0 2024-02-28T16:31:10.429 INFO:tasks.workunit.client.0.smithi001.stdout:8/520: getdents d4 0 2024-02-28T16:31:10.430 INFO:tasks.workunit.client.0.smithi001.stdout:5/482: truncate dc/d1a/d21/f37 281468 0 2024-02-28T16:31:10.430 INFO:tasks.workunit.client.0.smithi001.stdout:3/525: dwrite d4/d18/f10 [0,4194304] 0 2024-02-28T16:31:10.433 INFO:tasks.workunit.client.0.smithi001.stdout:1/639: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d88/fbc 0 2024-02-28T16:31:10.435 INFO:tasks.workunit.client.0.smithi001.stdout:8/521: write d4/d7/d22/d27/d4b/d77/f8d [1308237,48094] 0 2024-02-28T16:31:10.436 INFO:tasks.workunit.client.0.smithi001.stdout:4/546: mkdir d3/d2d/d4c/d5e/db0 0 2024-02-28T16:31:10.436 INFO:tasks.workunit.client.0.smithi001.stdout:5/483: chown dc/d1a/d21/d2d/d91 100666 1 2024-02-28T16:31:10.437 INFO:tasks.workunit.client.0.smithi001.stdout:3/526: mknod d4/d6/d7/d78/d81/cb1 0 2024-02-28T16:31:10.439 INFO:tasks.workunit.client.0.smithi001.stdout:9/614: mkdir d5/d9a/dce/dd0 0 2024-02-28T16:31:10.443 INFO:tasks.workunit.client.0.smithi001.stdout:2/545: rmdir d0/d2/d3/de/db8 0 2024-02-28T16:31:10.444 INFO:tasks.workunit.client.0.smithi001.stdout:4/547: rmdir d3/d2d/d4c 39 2024-02-28T16:31:10.444 INFO:tasks.workunit.client.0.smithi001.stdout:7/509: link d3/d17/d3d/f70 d3/fa4 0 2024-02-28T16:31:10.445 INFO:tasks.workunit.client.0.smithi001.stdout:5/484: symlink dc/d1a/d21/d2d/d3c/d4a/la1 0 2024-02-28T16:31:10.445 INFO:tasks.workunit.client.0.smithi001.stdout:7/510: readlink d3/d17/d52/l86 0 2024-02-28T16:31:10.449 INFO:tasks.workunit.client.0.smithi001.stdout:1/640: symlink d0/d9/ldd 0 2024-02-28T16:31:10.449 INFO:tasks.workunit.client.0.smithi001.stdout:5/485: readlink dc/l1c 0 2024-02-28T16:31:10.449 INFO:tasks.workunit.client.0.smithi001.stdout:4/548: symlink d3/d5/d7/lb1 0 2024-02-28T16:31:10.452 INFO:tasks.workunit.client.0.smithi001.stdout:5/486: write dc/d1a/d21/d2d/d3c/f53 [5134773,9017] 0 2024-02-28T16:31:10.454 INFO:tasks.workunit.client.0.smithi001.stdout:9/615: mkdir d5/d27/db1/d33/d8a/da8/dd1 0 2024-02-28T16:31:10.456 INFO:tasks.workunit.client.0.smithi001.stdout:5/487: chown dc/d12/f24 424855100 1 2024-02-28T16:31:10.457 INFO:tasks.workunit.client.0.smithi001.stdout:2/546: link d0/d2/d3/f54 d0/d2/d3/de/d22/fb9 0 2024-02-28T16:31:10.458 INFO:tasks.workunit.client.0.smithi001.stdout:9/616: truncate d5/d27/db1/fa3 593226 0 2024-02-28T16:31:10.460 INFO:tasks.workunit.client.0.smithi001.stdout:8/522: link d4/d7/d22/d27/d58/d62/f7e d4/d7/d22/d27/d4b/fa9 0 2024-02-28T16:31:10.461 INFO:tasks.workunit.client.0.smithi001.stdout:7/511: creat d3/d17/fa5 x:0 0 0 2024-02-28T16:31:10.462 INFO:tasks.workunit.client.0.smithi001.stdout:1/641: creat d0/d9/d1a/d2b/d26/d27/dce/fde x:0 0 0 2024-02-28T16:31:10.465 INFO:tasks.workunit.client.0.smithi001.stdout:4/549: creat d3/d1c/da0/fb2 x:0 0 0 2024-02-28T16:31:10.467 INFO:tasks.workunit.client.0.smithi001.stdout:2/547: readlink d0/d2/d3/de/d8a/l31 0 2024-02-28T16:31:10.468 INFO:tasks.workunit.client.0.smithi001.stdout:4/550: read d3/d5/f34 [1252966,83768] 0 2024-02-28T16:31:10.470 INFO:tasks.workunit.client.0.smithi001.stdout:7/512: creat d3/d17/d3d/fa6 x:0 0 0 2024-02-28T16:31:10.472 INFO:tasks.workunit.client.1.smithi066.stdout:0/175: readlink d2/l12 0 2024-02-28T16:31:10.472 INFO:tasks.workunit.client.0.smithi001.stdout:2/548: chown d0/d2/d3/de/d61/d12/d55/l87 718 1 2024-02-28T16:31:10.473 INFO:tasks.workunit.client.0.smithi001.stdout:8/523: creat d4/d7/d22/d27/d58/d94/faa x:0 0 0 2024-02-28T16:31:10.474 INFO:tasks.workunit.client.1.smithi066.stdout:1/124: rename d2/d18/d1f to d2/d27/d2b 0 2024-02-28T16:31:10.474 INFO:tasks.workunit.client.0.smithi001.stdout:4/551: mkdir d3/d5/d7/db3 0 2024-02-28T16:31:10.475 INFO:tasks.workunit.client.1.smithi066.stdout:1/125: truncate d2/f17 798377 0 2024-02-28T16:31:10.476 INFO:tasks.workunit.client.1.smithi066.stdout:7/40: link f5 fa 0 2024-02-28T16:31:10.476 INFO:tasks.workunit.client.0.smithi001.stdout:3/527: dwrite d4/d1e/f25 [0,4194304] 0 2024-02-28T16:31:10.477 INFO:tasks.workunit.client.1.smithi066.stdout:0/176: creat d2/d9/d10/f39 x:0 0 0 2024-02-28T16:31:10.477 INFO:tasks.workunit.client.0.smithi001.stdout:7/513: truncate d3/d17/d3d/d59/d6e/f9f 93796 0 2024-02-28T16:31:10.479 INFO:tasks.workunit.client.0.smithi001.stdout:5/488: creat dc/fa2 x:0 0 0 2024-02-28T16:31:10.479 INFO:tasks.workunit.client.1.smithi066.stdout:1/126: creat d2/d27/d2b/f2c x:0 0 0 2024-02-28T16:31:10.480 INFO:tasks.workunit.client.0.smithi001.stdout:9/617: rmdir d5/d27/db1/d5e/dc3 0 2024-02-28T16:31:10.480 INFO:tasks.workunit.client.0.smithi001.stdout:8/524: truncate d4/d32/d69/f6f 1046112 0 2024-02-28T16:31:10.480 INFO:tasks.workunit.client.1.smithi066.stdout:0/177: fdatasync d2/d9/d10/d23/f2d 0 2024-02-28T16:31:10.481 INFO:tasks.workunit.client.0.smithi001.stdout:8/525: read - d4/d32/d93/fa2 zero size 2024-02-28T16:31:10.482 INFO:tasks.workunit.client.0.smithi001.stdout:5/489: dread - dc/d12/d41/d54/f9a zero size 2024-02-28T16:31:10.482 INFO:tasks.workunit.client.0.smithi001.stdout:1/642: rename d0/d9/f66 to d0/d9/d1a/d2b/d26/d27/d30/d4a/fdf 0 2024-02-28T16:31:10.484 INFO:tasks.workunit.client.0.smithi001.stdout:2/549: read d0/d2/d3/de/d61/d12/f27 [542667,113982] 0 2024-02-28T16:31:10.484 INFO:tasks.workunit.client.1.smithi066.stdout:9/79: rename da/d10/f11 to da/f16 0 2024-02-28T16:31:10.485 INFO:tasks.workunit.client.1.smithi066.stdout:7/41: dread f4 [0,4194304] 0 2024-02-28T16:31:10.486 INFO:tasks.workunit.client.0.smithi001.stdout:3/528: creat d4/d6/d7/d78/d81/fb2 x:0 0 0 2024-02-28T16:31:10.486 INFO:tasks.workunit.client.1.smithi066.stdout:1/127: dread d2/f15 [0,4194304] 0 2024-02-28T16:31:10.487 INFO:tasks.workunit.client.1.smithi066.stdout:9/80: fdatasync da/fb 0 2024-02-28T16:31:10.489 INFO:tasks.workunit.client.1.smithi066.stdout:1/128: read d2/f7 [1900644,105268] 0 2024-02-28T16:31:10.489 INFO:tasks.workunit.client.1.smithi066.stdout:7/42: mknod cb 0 2024-02-28T16:31:10.490 INFO:tasks.workunit.client.1.smithi066.stdout:0/178: write d2/d9/d10/f26 [982963,63480] 0 2024-02-28T16:31:10.491 INFO:tasks.workunit.client.0.smithi001.stdout:7/514: rename d3/d17/d3d/f48 to d3/d17/d3d/fa7 0 2024-02-28T16:31:10.495 INFO:tasks.workunit.client.1.smithi066.stdout:0/179: mknod d2/d27/c3a 0 2024-02-28T16:31:10.496 INFO:tasks.workunit.client.0.smithi001.stdout:5/490: fdatasync dc/d1a/d21/d2d/d3c/f4f 0 2024-02-28T16:31:10.499 INFO:tasks.workunit.client.0.smithi001.stdout:5/491: stat dc/d1a/d52/l70 0 2024-02-28T16:31:10.500 INFO:tasks.workunit.client.1.smithi066.stdout:3/55: dwrite f5 [0,4194304] 0 2024-02-28T16:31:10.500 INFO:tasks.workunit.client.0.smithi001.stdout:1/643: mknod d0/d9/d1a/d72/d9d/d9c/ce0 0 2024-02-28T16:31:10.502 INFO:tasks.workunit.client.0.smithi001.stdout:5/492: chown dc/d1a/d21/l4e 42 1 2024-02-28T16:31:10.502 INFO:tasks.workunit.client.0.smithi001.stdout:6/478: sync 2024-02-28T16:31:10.502 INFO:tasks.workunit.client.0.smithi001.stdout:0/514: sync 2024-02-28T16:31:10.508 INFO:tasks.workunit.client.1.smithi066.stdout:0/180: symlink d2/d9/d33/l3b 0 2024-02-28T16:31:10.508 INFO:tasks.workunit.client.1.smithi066.stdout:3/56: creat db/ff x:0 0 0 2024-02-28T16:31:10.509 INFO:tasks.workunit.client.1.smithi066.stdout:1/129: dread d2/f16 [0,4194304] 0 2024-02-28T16:31:10.509 INFO:tasks.workunit.client.1.smithi066.stdout:9/81: rmdir da/d10 39 2024-02-28T16:31:10.509 INFO:tasks.workunit.client.1.smithi066.stdout:3/57: write f5 [3551740,80007] 0 2024-02-28T16:31:10.509 INFO:tasks.workunit.client.0.smithi001.stdout:2/550: dwrite d0/d2/d3/de/d61/d12/fa7 [0,4194304] 0 2024-02-28T16:31:10.509 INFO:tasks.workunit.client.0.smithi001.stdout:9/618: write d5/f1e [142836,112918] 0 2024-02-28T16:31:10.510 INFO:tasks.workunit.client.1.smithi066.stdout:0/181: truncate d2/d9/d10/f39 325609 0 2024-02-28T16:31:10.512 INFO:tasks.workunit.client.0.smithi001.stdout:5/493: fsync dc/d1a/f7b 0 2024-02-28T16:31:10.512 INFO:tasks.workunit.client.1.smithi066.stdout:9/82: unlink da/f16 0 2024-02-28T16:31:10.513 INFO:tasks.workunit.client.1.smithi066.stdout:1/130: write d2/f1d [3333005,55513] 0 2024-02-28T16:31:10.514 INFO:tasks.workunit.client.1.smithi066.stdout:3/58: creat db/f10 x:0 0 0 2024-02-28T16:31:10.515 INFO:tasks.workunit.client.0.smithi001.stdout:6/479: unlink d9/db/f1b 0 2024-02-28T16:31:10.517 INFO:tasks.workunit.client.1.smithi066.stdout:1/131: fdatasync d2/f15 0 2024-02-28T16:31:10.518 INFO:tasks.workunit.client.0.smithi001.stdout:4/552: getdents d3/d5/d7/d1f 0 2024-02-28T16:31:10.519 INFO:tasks.workunit.client.1.smithi066.stdout:3/59: unlink db/f10 0 2024-02-28T16:31:10.520 INFO:tasks.workunit.client.0.smithi001.stdout:2/551: fsync d0/d2/d3/de/d8a/f6e 0 2024-02-28T16:31:10.521 INFO:tasks.workunit.client.1.smithi066.stdout:0/182: unlink d2/d9/d10/c25 0 2024-02-28T16:31:10.522 INFO:tasks.workunit.client.1.smithi066.stdout:1/132: fsync d2/f5 0 2024-02-28T16:31:10.523 INFO:tasks.workunit.client.0.smithi001.stdout:7/515: creat d3/d17/d3d/d5d/d9a/d49/fa8 x:0 0 0 2024-02-28T16:31:10.524 INFO:tasks.workunit.client.0.smithi001.stdout:4/553: chown d3/d5/d7/lb1 21237 1 2024-02-28T16:31:10.524 INFO:tasks.workunit.client.1.smithi066.stdout:3/60: mkdir db/dc/d11 0 2024-02-28T16:31:10.525 INFO:tasks.workunit.client.0.smithi001.stdout:8/526: dwrite d4/d7/f23 [0,4194304] 0 2024-02-28T16:31:10.527 INFO:tasks.workunit.client.0.smithi001.stdout:3/529: dread d4/d18/da0/fa2 [4194304,4194304] 0 2024-02-28T16:31:10.530 INFO:tasks.workunit.client.1.smithi066.stdout:1/133: symlink d2/d27/l2d 0 2024-02-28T16:31:10.531 INFO:tasks.workunit.client.0.smithi001.stdout:3/530: write d4/d6/f52 [182099,66176] 0 2024-02-28T16:31:10.533 INFO:tasks.workunit.client.0.smithi001.stdout:0/515: creat d18/fac x:0 0 0 2024-02-28T16:31:10.533 INFO:tasks.workunit.client.1.smithi066.stdout:3/61: symlink db/l12 0 2024-02-28T16:31:10.533 INFO:tasks.workunit.client.1.smithi066.stdout:7/43: dwrite f9 [0,4194304] 0 2024-02-28T16:31:10.537 INFO:tasks.workunit.client.0.smithi001.stdout:7/516: symlink d3/d17/d3d/d5d/d9a/la9 0 2024-02-28T16:31:10.538 INFO:tasks.workunit.client.0.smithi001.stdout:7/517: fsync d3/d17/d3d/f72 0 2024-02-28T16:31:10.539 INFO:tasks.workunit.client.0.smithi001.stdout:2/552: rename d0/d2/d3/de/d22/cad to d0/d2/d3/de/d61/d12/d55/db2/cba 0 2024-02-28T16:31:10.542 INFO:tasks.workunit.client.1.smithi066.stdout:1/134: write d2/f5 [759984,39482] 0 2024-02-28T16:31:10.542 INFO:tasks.workunit.client.0.smithi001.stdout:6/480: creat d9/d29/d6e/d53/f9e x:0 0 0 2024-02-28T16:31:10.543 INFO:tasks.workunit.client.1.smithi066.stdout:7/44: fdatasync fa 0 2024-02-28T16:31:10.543 INFO:tasks.workunit.client.1.smithi066.stdout:3/62: mkdir db/d13 0 2024-02-28T16:31:10.550 INFO:tasks.workunit.client.0.smithi001.stdout:9/619: creat d5/d45/fd2 x:0 0 0 2024-02-28T16:31:10.550 INFO:tasks.workunit.client.0.smithi001.stdout:6/481: fsync d9/db/d5f/d62/d83/f8f 0 2024-02-28T16:31:10.550 INFO:tasks.workunit.client.0.smithi001.stdout:3/531: mknod d4/d6/cb3 0 2024-02-28T16:31:10.551 INFO:tasks.workunit.client.0.smithi001.stdout:9/620: chown d5/d27/db1/f93 8773 1 2024-02-28T16:31:10.554 INFO:tasks.workunit.client.0.smithi001.stdout:4/554: creat d3/fb4 x:0 0 0 2024-02-28T16:31:10.554 INFO:tasks.workunit.client.1.smithi066.stdout:7/45: mkdir dc 0 2024-02-28T16:31:10.556 INFO:tasks.workunit.client.0.smithi001.stdout:5/494: dwrite dc/d12/d41/f4c [0,4194304] 0 2024-02-28T16:31:10.557 INFO:tasks.workunit.client.1.smithi066.stdout:8/50: sync 2024-02-28T16:31:10.557 INFO:tasks.workunit.client.1.smithi066.stdout:2/97: sync 2024-02-28T16:31:10.558 INFO:tasks.workunit.client.0.smithi001.stdout:8/527: creat d4/d7/d22/fab x:0 0 0 2024-02-28T16:31:10.558 INFO:tasks.workunit.client.1.smithi066.stdout:1/135: write d2/fc [346783,100425] 0 2024-02-28T16:31:10.560 INFO:tasks.workunit.client.1.smithi066.stdout:3/63: mkdir db/d13/d14 0 2024-02-28T16:31:10.561 INFO:tasks.workunit.client.0.smithi001.stdout:0/516: creat d18/d31/fad x:0 0 0 2024-02-28T16:31:10.562 INFO:tasks.workunit.client.1.smithi066.stdout:7/46: mknod dc/cd 0 2024-02-28T16:31:10.562 INFO:tasks.workunit.client.1.smithi066.stdout:2/98: dread - d0/f19 zero size 2024-02-28T16:31:10.563 INFO:tasks.workunit.client.1.smithi066.stdout:9/83: dwrite da/fc [0,4194304] 0 2024-02-28T16:31:10.563 INFO:tasks.workunit.client.1.smithi066.stdout:8/51: creat d0/d6/ff x:0 0 0 2024-02-28T16:31:10.564 INFO:tasks.workunit.client.1.smithi066.stdout:1/136: unlink d2/f23 0 2024-02-28T16:31:10.565 INFO:tasks.workunit.client.0.smithi001.stdout:6/482: symlink d9/d29/d6e/d4e/l9f 0 2024-02-28T16:31:10.566 INFO:tasks.workunit.client.0.smithi001.stdout:2/553: dwrite d0/d2/d3/de/f8f [0,4194304] 0 2024-02-28T16:31:10.567 INFO:tasks.workunit.client.1.smithi066.stdout:3/64: unlink db/fe 0 2024-02-28T16:31:10.568 INFO:tasks.workunit.client.1.smithi066.stdout:8/52: chown d0/d6/f9 8420977 1 2024-02-28T16:31:10.568 INFO:tasks.workunit.client.1.smithi066.stdout:1/137: chown d2/d27 1356652455 1 2024-02-28T16:31:10.569 INFO:tasks.workunit.client.1.smithi066.stdout:7/47: mkdir dc/de 0 2024-02-28T16:31:10.569 INFO:tasks.workunit.client.1.smithi066.stdout:0/183: dwrite d2/d9/d10/f2c [0,4194304] 0 2024-02-28T16:31:10.569 INFO:tasks.workunit.client.0.smithi001.stdout:3/532: dread d4/d6/d7/d78/d81/f96 [0,4194304] 0 2024-02-28T16:31:10.571 INFO:tasks.workunit.client.1.smithi066.stdout:2/99: symlink d0/d12/l1b 0 2024-02-28T16:31:10.572 INFO:tasks.workunit.client.1.smithi066.stdout:0/184: stat d2/c8 0 2024-02-28T16:31:10.573 INFO:tasks.workunit.client.0.smithi001.stdout:3/533: stat d4/d6/d7/d78/d81/cb1 0 2024-02-28T16:31:10.575 INFO:tasks.workunit.client.1.smithi066.stdout:7/48: fdatasync f5 0 2024-02-28T16:31:10.575 INFO:tasks.workunit.client.1.smithi066.stdout:1/138: chown d2/d18/d19/f29 2 1 2024-02-28T16:31:10.576 INFO:tasks.workunit.client.0.smithi001.stdout:4/555: readlink d3/l19 0 2024-02-28T16:31:10.578 INFO:tasks.workunit.client.0.smithi001.stdout:5/495: symlink dc/d12/d41/d54/la3 0 2024-02-28T16:31:10.582 INFO:tasks.workunit.client.0.smithi001.stdout:0/517: rename d18/d22/d2f/d1c/f38 to d18/d22/d76/d7a/fae 0 2024-02-28T16:31:10.584 INFO:tasks.workunit.client.1.smithi066.stdout:3/65: creat db/d13/d14/f15 x:0 0 0 2024-02-28T16:31:10.584 INFO:tasks.workunit.client.0.smithi001.stdout:4/556: read d3/f3e [740188,122525] 0 2024-02-28T16:31:10.585 INFO:tasks.workunit.client.1.smithi066.stdout:7/49: symlink dc/lf 0 2024-02-28T16:31:10.585 INFO:tasks.workunit.client.1.smithi066.stdout:3/66: fsync f3 0 2024-02-28T16:31:10.586 INFO:tasks.workunit.client.0.smithi001.stdout:3/534: fdatasync d4/f61 0 2024-02-28T16:31:10.587 INFO:tasks.workunit.client.1.smithi066.stdout:8/53: dread d0/d6/fe [0,4194304] 0 2024-02-28T16:31:10.588 INFO:tasks.workunit.client.0.smithi001.stdout:8/528: dwrite d4/d7/d22/d27/d4b/d77/f8d [0,4194304] 0 2024-02-28T16:31:10.589 INFO:tasks.workunit.client.1.smithi066.stdout:3/67: fdatasync db/ff 0 2024-02-28T16:31:10.590 INFO:tasks.workunit.client.1.smithi066.stdout:3/68: chown c8 176 1 2024-02-28T16:31:10.591 INFO:tasks.workunit.client.1.smithi066.stdout:2/100: rename d0/cb to d0/c1c 0 2024-02-28T16:31:10.593 INFO:tasks.workunit.client.0.smithi001.stdout:3/535: write d4/d6/f52 [788570,108739] 0 2024-02-28T16:31:10.594 INFO:tasks.workunit.client.1.smithi066.stdout:9/84: dread f8 [4194304,4194304] 0 2024-02-28T16:31:10.597 INFO:tasks.workunit.client.0.smithi001.stdout:6/483: rmdir d9/d29/d6e/d4e 39 2024-02-28T16:31:10.599 INFO:tasks.workunit.client.0.smithi001.stdout:9/621: write d5/d27/db1/f41 [3245742,46930] 0 2024-02-28T16:31:10.599 INFO:tasks.workunit.client.1.smithi066.stdout:4/131: sync 2024-02-28T16:31:10.599 INFO:tasks.workunit.client.1.smithi066.stdout:5/90: sync 2024-02-28T16:31:10.599 INFO:tasks.workunit.client.1.smithi066.stdout:6/36: sync 2024-02-28T16:31:10.601 INFO:tasks.workunit.client.0.smithi001.stdout:7/518: truncate d3/d17/d3d/d42/f7d 267898 0 2024-02-28T16:31:10.605 INFO:tasks.workunit.client.1.smithi066.stdout:3/69: rename l9 to db/dc/d11/l16 0 2024-02-28T16:31:10.606 INFO:tasks.workunit.client.1.smithi066.stdout:5/91: readlink d3/l16 0 2024-02-28T16:31:10.608 INFO:tasks.workunit.client.0.smithi001.stdout:2/554: dwrite d0/d2/d3/de/d22/f38 [0,4194304] 0 2024-02-28T16:31:10.610 INFO:tasks.workunit.client.1.smithi066.stdout:9/85: unlink da/df/l12 0 2024-02-28T16:31:10.614 INFO:tasks.workunit.client.1.smithi066.stdout:0/185: dwrite d2/d9/f35 [0,4194304] 0 2024-02-28T16:31:10.616 INFO:tasks.workunit.client.0.smithi001.stdout:0/518: dread d18/d22/d2b/d5f/f85 [0,4194304] 0 2024-02-28T16:31:10.619 INFO:tasks.workunit.client.0.smithi001.stdout:3/536: dread d4/d18/f49 [0,4194304] 0 2024-02-28T16:31:10.619 INFO:tasks.workunit.client.1.smithi066.stdout:3/70: write db/ff [438544,47106] 0 2024-02-28T16:31:10.619 INFO:tasks.workunit.client.1.smithi066.stdout:4/132: creat d1/d23/f27 x:0 0 0 2024-02-28T16:31:10.620 INFO:tasks.workunit.client.1.smithi066.stdout:0/186: readlink d2/l12 0 2024-02-28T16:31:10.621 INFO:tasks.workunit.client.1.smithi066.stdout:0/187: stat d2/l12 0 2024-02-28T16:31:10.622 INFO:tasks.workunit.client.0.smithi001.stdout:5/496: symlink dc/df/d79/d98/la4 0 2024-02-28T16:31:10.622 INFO:tasks.workunit.client.0.smithi001.stdout:9/622: mkdir d5/d27/db1/d5e/dd3 0 2024-02-28T16:31:10.625 INFO:tasks.workunit.client.0.smithi001.stdout:4/557: unlink d3/d2d/d4c/d69/d82/l56 0 2024-02-28T16:31:10.626 INFO:tasks.workunit.client.1.smithi066.stdout:0/188: fdatasync d2/d9/f1e 0 2024-02-28T16:31:10.627 INFO:tasks.workunit.client.1.smithi066.stdout:9/86: write da/d10/f13 [1020841,29638] 0 2024-02-28T16:31:10.627 INFO:tasks.workunit.client.1.smithi066.stdout:3/71: write f3 [5729877,104690] 0 2024-02-28T16:31:10.627 INFO:tasks.workunit.client.1.smithi066.stdout:7/50: dread f9 [0,4194304] 0 2024-02-28T16:31:10.629 INFO:tasks.workunit.client.1.smithi066.stdout:1/139: write d2/f13 [380933,70952] 0 2024-02-28T16:31:10.630 INFO:tasks.workunit.client.0.smithi001.stdout:8/529: mknod d4/d7/cac 0 2024-02-28T16:31:10.632 INFO:tasks.workunit.client.0.smithi001.stdout:8/530: chown d4/c88 10461 1 2024-02-28T16:31:10.634 INFO:tasks.workunit.client.0.smithi001.stdout:0/519: fdatasync d18/d22/d76/d7a/fae 0 2024-02-28T16:31:10.634 INFO:tasks.workunit.client.0.smithi001.stdout:3/537: mknod d4/d6/d7/d23/cb4 0 2024-02-28T16:31:10.635 INFO:tasks.workunit.client.1.smithi066.stdout:3/72: unlink l1 0 2024-02-28T16:31:10.635 INFO:tasks.workunit.client.1.smithi066.stdout:0/189: chown d2/f22 230233 1 2024-02-28T16:31:10.636 INFO:tasks.workunit.client.0.smithi001.stdout:8/531: chown d4/d47/d8b/f8c 4504 1 2024-02-28T16:31:10.638 INFO:tasks.workunit.client.1.smithi066.stdout:2/101: dwrite d0/ff [0,4194304] 0 2024-02-28T16:31:10.638 INFO:tasks.workunit.client.1.smithi066.stdout:1/140: rmdir d2/d18 39 2024-02-28T16:31:10.638 INFO:tasks.workunit.client.0.smithi001.stdout:7/519: creat d3/d17/d52/d7f/d99/faa x:0 0 0 2024-02-28T16:31:10.640 INFO:tasks.workunit.client.1.smithi066.stdout:1/141: readlink d2/l6 0 2024-02-28T16:31:10.643 INFO:tasks.workunit.client.0.smithi001.stdout:8/532: write d4/d7/d22/d27/d4b/d80/f85 [494546,54266] 0 2024-02-28T16:31:10.644 INFO:tasks.workunit.client.0.smithi001.stdout:4/558: fdatasync d3/d2d/d4c/f51 0 2024-02-28T16:31:10.645 INFO:tasks.workunit.client.1.smithi066.stdout:9/87: link c7 da/d10/c17 0 2024-02-28T16:31:10.646 INFO:tasks.workunit.client.0.smithi001.stdout:3/538: write d4/d6/d7/f5d [129586,64810] 0 2024-02-28T16:31:10.647 INFO:tasks.workunit.client.1.smithi066.stdout:2/102: write d0/d12/f14 [3122550,71432] 0 2024-02-28T16:31:10.650 INFO:tasks.workunit.client.1.smithi066.stdout:5/92: dread d3/d4/f5 [0,4194304] 0 2024-02-28T16:31:10.650 INFO:tasks.workunit.client.0.smithi001.stdout:6/484: link d9/de/c4a d9/ca0 0 2024-02-28T16:31:10.650 INFO:tasks.workunit.client.0.smithi001.stdout:4/559: fdatasync d3/d5/d7/d1f/d20/f31 0 2024-02-28T16:31:10.653 INFO:tasks.workunit.client.1.smithi066.stdout:8/54: dwrite d0/d6/fe [0,4194304] 0 2024-02-28T16:31:10.654 INFO:tasks.workunit.client.0.smithi001.stdout:6/485: fdatasync d9/d39/f3a 0 2024-02-28T16:31:10.655 INFO:tasks.workunit.client.1.smithi066.stdout:5/93: chown d3/d4/d17/d18 55 1 2024-02-28T16:31:10.657 INFO:tasks.workunit.client.1.smithi066.stdout:4/133: getdents d1/d23 0 2024-02-28T16:31:10.658 INFO:tasks.workunit.client.0.smithi001.stdout:6/486: fdatasync d9/db/d19/f90 0 2024-02-28T16:31:10.660 INFO:tasks.workunit.client.1.smithi066.stdout:2/103: mkdir d0/d12/d1d 0 2024-02-28T16:31:10.661 INFO:tasks.workunit.client.1.smithi066.stdout:0/190: getdents d2/d9/d33 0 2024-02-28T16:31:10.664 INFO:tasks.workunit.client.0.smithi001.stdout:8/533: rename d4/d32/c45 to d4/d7/da8/d89/cad 0 2024-02-28T16:31:10.671 INFO:tasks.workunit.client.1.smithi066.stdout:5/94: creat d3/d4/dc/f21 x:0 0 0 2024-02-28T16:31:10.672 INFO:tasks.workunit.client.1.smithi066.stdout:4/134: creat d1/d23/d25/f28 x:0 0 0 2024-02-28T16:31:10.672 INFO:tasks.workunit.client.0.smithi001.stdout:0/520: dwrite d18/d22/d76/d93/f94 [0,4194304] 0 2024-02-28T16:31:10.673 INFO:tasks.workunit.client.1.smithi066.stdout:0/191: symlink d2/d9/d33/l3c 0 2024-02-28T16:31:10.675 INFO:tasks.workunit.client.1.smithi066.stdout:3/73: dwrite db/ff [0,4194304] 0 2024-02-28T16:31:10.677 INFO:tasks.workunit.client.0.smithi001.stdout:2/555: truncate d0/d2/d3/de/d8a/f2e 2155554 0 2024-02-28T16:31:10.677 INFO:tasks.workunit.client.1.smithi066.stdout:8/55: link d0/d6/f9 d0/db/f10 0 2024-02-28T16:31:10.681 INFO:tasks.workunit.client.0.smithi001.stdout:8/534: mkdir d4/d7/d22/d27/d8e/dae 0 2024-02-28T16:31:10.682 INFO:tasks.workunit.client.1.smithi066.stdout:5/95: fsync d3/d4/d17/d18/f1a 0 2024-02-28T16:31:10.684 INFO:tasks.workunit.client.0.smithi001.stdout:8/535: readlink l0 0 2024-02-28T16:31:10.684 INFO:tasks.workunit.client.1.smithi066.stdout:4/135: unlink d1/l22 0 2024-02-28T16:31:10.687 INFO:tasks.workunit.client.0.smithi001.stdout:8/536: write d4/d7/f1d [33769,21711] 0 2024-02-28T16:31:10.691 INFO:tasks.workunit.client.1.smithi066.stdout:3/74: dread - db/d13/d14/f15 zero size 2024-02-28T16:31:10.693 INFO:tasks.workunit.client.1.smithi066.stdout:1/142: truncate d2/d18/f21 528803 0 2024-02-28T16:31:10.693 INFO:tasks.workunit.client.0.smithi001.stdout:6/487: symlink d9/db/d10/la1 0 2024-02-28T16:31:10.694 INFO:tasks.workunit.client.0.smithi001.stdout:9/623: dwrite d5/d27/db1/f40 [0,4194304] 0 2024-02-28T16:31:10.694 INFO:tasks.workunit.client.0.smithi001.stdout:2/556: mkdir d0/d2/d3/de/d22/dbb 0 2024-02-28T16:31:10.694 INFO:tasks.workunit.client.0.smithi001.stdout:4/560: link d3/d5/f34 d3/d1c/d58/d6c/d79/fb5 0 2024-02-28T16:31:10.697 INFO:tasks.workunit.client.1.smithi066.stdout:0/192: unlink d2/d9/d10/d23/f30 0 2024-02-28T16:31:10.697 INFO:tasks.workunit.client.1.smithi066.stdout:5/96: mknod d3/c22 0 2024-02-28T16:31:10.699 INFO:tasks.workunit.client.0.smithi001.stdout:6/488: creat d9/de/fa2 x:0 0 0 2024-02-28T16:31:10.701 INFO:tasks.workunit.client.0.smithi001.stdout:9/624: fdatasync d5/f1d 0 2024-02-28T16:31:10.701 INFO:tasks.workunit.client.1.smithi066.stdout:4/136: rename d1/cb to d1/d23/c29 0 2024-02-28T16:31:10.702 INFO:tasks.workunit.client.1.smithi066.stdout:1/143: write d2/f10 [823625,45798] 0 2024-02-28T16:31:10.703 INFO:tasks.workunit.client.0.smithi001.stdout:5/497: dread dc/d12/f24 [0,4194304] 0 2024-02-28T16:31:10.707 INFO:tasks.workunit.client.1.smithi066.stdout:2/104: dwrite d0/d12/f13 [0,4194304] 0 2024-02-28T16:31:10.707 INFO:tasks.workunit.client.1.smithi066.stdout:5/97: chown d3/d1b/d20 2124545213 1 2024-02-28T16:31:10.707 INFO:tasks.workunit.client.1.smithi066.stdout:3/75: unlink f3 0 2024-02-28T16:31:10.708 INFO:tasks.workunit.client.0.smithi001.stdout:6/489: rmdir d9/d29/d6e/d53 39 2024-02-28T16:31:10.712 INFO:tasks.workunit.client.0.smithi001.stdout:2/557: rename d0/d2/d3/de/d61/d12/d46/faf to d0/d2/d3/de/d61/d12/d6a/d84/fbc 0 2024-02-28T16:31:10.712 INFO:tasks.workunit.client.0.smithi001.stdout:7/520: dread d3/d17/d52/d7f/f81 [0,4194304] 0 2024-02-28T16:31:10.717 INFO:tasks.workunit.client.1.smithi066.stdout:4/137: write d1/f10 [1070319,102445] 0 2024-02-28T16:31:10.717 INFO:tasks.workunit.client.0.smithi001.stdout:7/521: stat d3/d17/d52/l85 0 2024-02-28T16:31:10.718 INFO:tasks.workunit.client.0.smithi001.stdout:6/490: symlink d9/de/la3 0 2024-02-28T16:31:10.720 INFO:tasks.workunit.client.0.smithi001.stdout:6/491: write d9/db/f13 [2653956,5695] 0 2024-02-28T16:31:10.720 INFO:tasks.workunit.client.1.smithi066.stdout:9/88: dwrite f8 [4194304,4194304] 0 2024-02-28T16:31:10.722 INFO:tasks.workunit.client.1.smithi066.stdout:9/89: fsync da/fc 0 2024-02-28T16:31:10.728 INFO:tasks.workunit.client.0.smithi001.stdout:6/492: write d9/d29/f44 [1204214,105497] 0 2024-02-28T16:31:10.729 INFO:tasks.workunit.client.0.smithi001.stdout:4/561: creat d3/d5/fb6 x:0 0 0 2024-02-28T16:31:10.730 INFO:tasks.workunit.client.1.smithi066.stdout:8/56: dread d0/f4 [0,4194304] 0 2024-02-28T16:31:10.731 INFO:tasks.workunit.client.0.smithi001.stdout:0/521: write d18/d22/d2b/f88 [645408,33739] 0 2024-02-28T16:31:10.731 INFO:tasks.workunit.client.1.smithi066.stdout:9/90: mkdir da/df/d18 0 2024-02-28T16:31:10.735 INFO:tasks.workunit.client.1.smithi066.stdout:0/193: dwrite d2/d9/f21 [0,4194304] 0 2024-02-28T16:31:10.736 INFO:tasks.workunit.client.0.smithi001.stdout:4/562: stat d3/d2d/d4c/d5e/f62 0 2024-02-28T16:31:10.738 INFO:tasks.workunit.client.1.smithi066.stdout:3/76: dread db/ff [0,4194304] 0 2024-02-28T16:31:10.738 INFO:tasks.workunit.client.0.smithi001.stdout:6/493: write d9/d29/d2b/f30 [1457927,69528] 0 2024-02-28T16:31:10.740 INFO:tasks.workunit.client.0.smithi001.stdout:0/522: chown d18/d22/d2b/d36/d48/f55 270 1 2024-02-28T16:31:10.746 INFO:tasks.workunit.client.1.smithi066.stdout:0/194: mknod d2/d9/d33/c3d 0 2024-02-28T16:31:10.747 INFO:tasks.workunit.client.0.smithi001.stdout:9/625: dwrite d5/d4d/f77 [0,4194304] 0 2024-02-28T16:31:10.748 INFO:tasks.workunit.client.0.smithi001.stdout:6/494: creat d9/db/d10/fa4 x:0 0 0 2024-02-28T16:31:10.749 INFO:tasks.workunit.client.1.smithi066.stdout:3/77: mknod db/d13/d14/c17 0 2024-02-28T16:31:10.750 INFO:tasks.workunit.client.0.smithi001.stdout:6/495: fsync d9/db/d19/f78 0 2024-02-28T16:31:10.755 INFO:tasks.workunit.client.1.smithi066.stdout:4/138: dread d1/fe [0,4194304] 0 2024-02-28T16:31:10.755 INFO:tasks.workunit.client.1.smithi066.stdout:3/78: fsync db/ff 0 2024-02-28T16:31:10.757 INFO:tasks.workunit.client.0.smithi001.stdout:6/496: symlink d9/db/la5 0 2024-02-28T16:31:10.758 INFO:tasks.workunit.client.1.smithi066.stdout:1/144: dwrite d2/d18/d19/f1c [4194304,4194304] 0 2024-02-28T16:31:10.763 INFO:tasks.workunit.client.0.smithi001.stdout:8/537: dread d4/f3b [0,4194304] 0 2024-02-28T16:31:10.764 INFO:tasks.workunit.client.1.smithi066.stdout:3/79: creat db/d13/f18 x:0 0 0 2024-02-28T16:31:10.765 INFO:tasks.workunit.client.1.smithi066.stdout:4/139: chown d1/d23/c24 17 1 2024-02-28T16:31:10.766 INFO:tasks.workunit.client.0.smithi001.stdout:0/523: dwrite d18/d22/d2b/d36/f56 [4194304,4194304] 0 2024-02-28T16:31:10.767 INFO:tasks.workunit.client.1.smithi066.stdout:9/91: dread da/fc [0,4194304] 0 2024-02-28T16:31:10.767 INFO:tasks.workunit.client.1.smithi066.stdout:8/57: dread d0/d6/f7 [0,4194304] 0 2024-02-28T16:31:10.768 INFO:tasks.workunit.client.1.smithi066.stdout:3/80: write db/d13/d14/f15 [135900,7531] 0 2024-02-28T16:31:10.770 INFO:tasks.workunit.client.1.smithi066.stdout:9/92: read f3 [3609556,73905] 0 2024-02-28T16:31:10.771 INFO:tasks.workunit.client.0.smithi001.stdout:8/538: unlink d4/c6c 0 2024-02-28T16:31:10.775 INFO:tasks.workunit.client.0.smithi001.stdout:6/497: rmdir d9/d29/d6e/d4e/d96 39 2024-02-28T16:31:10.777 INFO:tasks.workunit.client.0.smithi001.stdout:5/498: dread dc/d12/f16 [0,4194304] 0 2024-02-28T16:31:10.777 INFO:tasks.workunit.client.0.smithi001.stdout:0/524: truncate d18/d22/d2b/d36/d48/f55 5125484 0 2024-02-28T16:31:10.778 INFO:tasks.workunit.client.1.smithi066.stdout:1/145: dwrite d2/f5 [0,4194304] 0 2024-02-28T16:31:10.780 INFO:tasks.workunit.client.0.smithi001.stdout:9/626: dwrite d5/d27/db1/da6/db2/fb6 [0,4194304] 0 2024-02-28T16:31:10.785 INFO:tasks.workunit.client.0.smithi001.stdout:0/525: unlink d18/d22/d2b/d36/d46/f9b 0 2024-02-28T16:31:10.788 INFO:tasks.workunit.client.0.smithi001.stdout:6/498: rename d9/d29/d2b to d9/da6 0 2024-02-28T16:31:10.792 INFO:tasks.workunit.client.0.smithi001.stdout:6/499: fsync d9/db/d10/f5b 0 2024-02-28T16:31:10.793 INFO:tasks.workunit.client.0.smithi001.stdout:0/526: write d18/d22/d57/d7e/f96 [3249881,123155] 0 2024-02-28T16:31:10.797 INFO:tasks.workunit.client.0.smithi001.stdout:6/500: creat d9/d39/fa7 x:0 0 0 2024-02-28T16:31:10.797 INFO:tasks.workunit.client.0.smithi001.stdout:7/522: dread d3/d17/f19 [0,4194304] 0 2024-02-28T16:31:10.803 INFO:tasks.workunit.client.0.smithi001.stdout:7/523: stat d3/d17/d52/d55/l83 0 2024-02-28T16:31:10.803 INFO:tasks.workunit.client.0.smithi001.stdout:6/501: rename d9/de/fa2 to d9/d39/fa8 0 2024-02-28T16:31:10.803 INFO:tasks.workunit.client.0.smithi001.stdout:2/558: dread d0/d2/d3/d36/d94/f96 [0,4194304] 0 2024-02-28T16:31:10.805 INFO:tasks.workunit.client.0.smithi001.stdout:5/499: dread dc/d1a/d21/f48 [0,4194304] 0 2024-02-28T16:31:10.811 INFO:tasks.workunit.client.0.smithi001.stdout:6/502: mkdir d9/d29/d6e/d9a/da9 0 2024-02-28T16:31:10.812 INFO:tasks.workunit.client.0.smithi001.stdout:6/503: dread - d9/de/f66 zero size 2024-02-28T16:31:10.813 INFO:tasks.workunit.client.0.smithi001.stdout:2/559: creat d0/d2/d3/de/d8a/fbd x:0 0 0 2024-02-28T16:31:10.813 INFO:tasks.workunit.client.0.smithi001.stdout:4/563: dread d3/d5/d4e/f97 [0,4194304] 0 2024-02-28T16:31:10.814 INFO:tasks.workunit.client.0.smithi001.stdout:5/500: fsync dc/df/f11 0 2024-02-28T16:31:10.817 INFO:tasks.workunit.client.0.smithi001.stdout:5/501: readlink dc/df/l69 0 2024-02-28T16:31:10.819 INFO:tasks.workunit.client.0.smithi001.stdout:7/524: link d3/f45 d3/d17/d3d/d5d/d9a/d49/d8d/fab 0 2024-02-28T16:31:10.821 INFO:tasks.workunit.client.0.smithi001.stdout:4/564: rename d3/d2d/d4c/f51 to d3/d5/d7/d1f/d2e/fb7 0 2024-02-28T16:31:10.821 INFO:tasks.workunit.client.0.smithi001.stdout:5/502: creat dc/d12/d41/d54/fa5 x:0 0 0 2024-02-28T16:31:10.822 INFO:tasks.workunit.client.0.smithi001.stdout:6/504: creat d9/db/d10/faa x:0 0 0 2024-02-28T16:31:10.823 INFO:tasks.workunit.client.0.smithi001.stdout:7/525: fsync d3/d17/d52/d7f/f81 0 2024-02-28T16:31:10.825 INFO:tasks.workunit.client.1.smithi066.stdout:6/37: sync 2024-02-28T16:31:10.825 INFO:tasks.workunit.client.0.smithi001.stdout:7/526: chown d3/f36 3027 1 2024-02-28T16:31:10.826 INFO:tasks.workunit.client.0.smithi001.stdout:2/560: rename d0/d2/d3/d36/d94/l9e to d0/d2/d3/de/d61/d12/d6a/d84/lbe 0 2024-02-28T16:31:10.826 INFO:tasks.workunit.client.0.smithi001.stdout:6/505: write d9/d29/d6a/f85 [2724216,54560] 0 2024-02-28T16:31:10.828 INFO:tasks.workunit.client.0.smithi001.stdout:2/561: chown d0/d2/d3/de/d22/f38 114 1 2024-02-28T16:31:10.829 INFO:tasks.workunit.client.1.smithi066.stdout:3/81: creat db/d13/f19 x:0 0 0 2024-02-28T16:31:10.830 INFO:tasks.workunit.client.1.smithi066.stdout:9/93: chown da/d10/c17 4 1 2024-02-28T16:31:10.832 INFO:tasks.workunit.client.0.smithi001.stdout:5/503: creat dc/d12/fa6 x:0 0 0 2024-02-28T16:31:10.833 INFO:tasks.workunit.client.1.smithi066.stdout:3/82: write db/d13/d14/f15 [1174087,61663] 0 2024-02-28T16:31:10.833 INFO:tasks.workunit.client.1.smithi066.stdout:1/146: creat d2/d27/d2b/f2e x:0 0 0 2024-02-28T16:31:10.836 INFO:tasks.workunit.client.1.smithi066.stdout:8/58: mknod d0/db/c11 0 2024-02-28T16:31:10.836 INFO:tasks.workunit.client.1.smithi066.stdout:1/147: dread d2/f15 [0,4194304] 0 2024-02-28T16:31:10.836 INFO:tasks.workunit.client.0.smithi001.stdout:5/504: fsync dc/f1d 0 2024-02-28T16:31:10.838 INFO:tasks.workunit.client.0.smithi001.stdout:0/527: read d18/d22/d57/d7e/f96 [3065926,110228] 0 2024-02-28T16:31:10.840 INFO:tasks.workunit.client.1.smithi066.stdout:8/59: write d0/d6/ff [198633,18212] 0 2024-02-28T16:31:10.840 INFO:tasks.workunit.client.0.smithi001.stdout:5/505: rmdir dc/d12/d56 39 2024-02-28T16:31:10.841 INFO:tasks.workunit.client.1.smithi066.stdout:4/140: creat d1/f2a x:0 0 0 2024-02-28T16:31:10.842 INFO:tasks.workunit.client.0.smithi001.stdout:0/528: chown d18/d22/d57/c8f 24266313 1 2024-02-28T16:31:10.843 INFO:tasks.workunit.client.1.smithi066.stdout:8/60: chown d0/d6/f9 11424506 1 2024-02-28T16:31:10.843 INFO:tasks.workunit.client.1.smithi066.stdout:8/61: readlink - no filename 2024-02-28T16:31:10.845 INFO:tasks.workunit.client.0.smithi001.stdout:4/565: dwrite d3/d5/d7/d1f/d3a/f70 [0,4194304] 0 2024-02-28T16:31:10.848 INFO:tasks.workunit.client.0.smithi001.stdout:6/506: dwrite d9/db/d19/f90 [0,4194304] 0 2024-02-28T16:31:10.848 INFO:tasks.workunit.client.1.smithi066.stdout:4/141: dread d1/f11 [0,4194304] 0 2024-02-28T16:31:10.851 INFO:tasks.workunit.client.0.smithi001.stdout:5/506: rename dc/l60 to dc/d1a/d52/la7 0 2024-02-28T16:31:10.853 INFO:tasks.workunit.client.0.smithi001.stdout:9/627: truncate d5/d27/db1/da6/db2/fb6 1339627 0 2024-02-28T16:31:10.855 INFO:tasks.workunit.client.1.smithi066.stdout:9/94: dread f3 [0,4194304] 0 2024-02-28T16:31:10.855 INFO:tasks.workunit.client.1.smithi066.stdout:3/83: truncate f5 1694469 0 2024-02-28T16:31:10.860 INFO:tasks.workunit.client.1.smithi066.stdout:4/142: getdents d1/d23 0 2024-02-28T16:31:10.865 INFO:tasks.workunit.client.1.smithi066.stdout:1/148: dwrite d2/f13 [0,4194304] 0 2024-02-28T16:31:10.870 INFO:tasks.workunit.client.1.smithi066.stdout:1/149: write d2/f5 [28175,41589] 0 2024-02-28T16:31:10.870 INFO:tasks.workunit.client.1.smithi066.stdout:8/62: dread d0/f4 [0,4194304] 0 2024-02-28T16:31:10.870 INFO:tasks.workunit.client.0.smithi001.stdout:1/644: sync 2024-02-28T16:31:10.872 INFO:tasks.workunit.client.1.smithi066.stdout:3/84: mkdir db/d13/d1a 0 2024-02-28T16:31:10.872 INFO:tasks.workunit.client.0.smithi001.stdout:9/628: link d5/d27/d34/c6b d5/d27/db1/cd4 0 2024-02-28T16:31:10.874 INFO:tasks.workunit.client.1.smithi066.stdout:4/143: creat d1/d18/f2b x:0 0 0 2024-02-28T16:31:10.876 INFO:tasks.workunit.client.0.smithi001.stdout:2/562: truncate d0/d2/d3/de/d61/d12/f27 972793 0 2024-02-28T16:31:10.876 INFO:tasks.workunit.client.0.smithi001.stdout:5/507: dwrite dc/f20 [0,4194304] 0 2024-02-28T16:31:10.877 INFO:tasks.workunit.client.1.smithi066.stdout:3/85: dread - db/d13/f18 zero size 2024-02-28T16:31:10.877 INFO:tasks.workunit.client.1.smithi066.stdout:4/144: write d1/f4 [3117204,75692] 0 2024-02-28T16:31:10.878 INFO:tasks.workunit.client.1.smithi066.stdout:1/150: unlink d2/d18/d19/f29 0 2024-02-28T16:31:10.879 INFO:tasks.workunit.client.0.smithi001.stdout:9/629: fdatasync d5/fb 0 2024-02-28T16:31:10.879 INFO:tasks.workunit.client.0.smithi001.stdout:2/563: stat d0/d2/d3/de/f5e 0 2024-02-28T16:31:10.880 INFO:tasks.workunit.client.1.smithi066.stdout:3/86: dread - db/d13/f18 zero size 2024-02-28T16:31:10.883 INFO:tasks.workunit.client.0.smithi001.stdout:0/529: dread fd [0,4194304] 0 2024-02-28T16:31:10.884 INFO:tasks.workunit.client.0.smithi001.stdout:1/645: creat d0/d9/d1a/d2b/d26/d27/d30/d67/db7/fe1 x:0 0 0 2024-02-28T16:31:10.886 INFO:tasks.workunit.client.0.smithi001.stdout:4/566: dread f2 [0,4194304] 0 2024-02-28T16:31:10.887 INFO:tasks.workunit.client.1.smithi066.stdout:4/145: mknod d1/c2c 0 2024-02-28T16:31:10.887 INFO:tasks.workunit.client.0.smithi001.stdout:1/646: readlink d0/l18 0 2024-02-28T16:31:10.888 INFO:tasks.workunit.client.1.smithi066.stdout:1/151: creat d2/d27/f2f x:0 0 0 2024-02-28T16:31:10.889 INFO:tasks.workunit.client.0.smithi001.stdout:7/527: dwrite d3/d1c/f4b [0,4194304] 0 2024-02-28T16:31:10.889 INFO:tasks.workunit.client.1.smithi066.stdout:4/146: read d1/f13 [1281122,59480] 0 2024-02-28T16:31:10.890 INFO:tasks.workunit.client.0.smithi001.stdout:5/508: unlink dc/d12/c84 0 2024-02-28T16:31:10.890 INFO:tasks.workunit.client.0.smithi001.stdout:4/567: read - d3/d5/fab zero size 2024-02-28T16:31:10.890 INFO:tasks.workunit.client.1.smithi066.stdout:9/95: dwrite da/fc [0,4194304] 0 2024-02-28T16:31:10.892 INFO:tasks.workunit.client.0.smithi001.stdout:1/647: stat d0/d9/d1a/d2b/d26/l38 0 2024-02-28T16:31:10.898 INFO:tasks.workunit.client.0.smithi001.stdout:7/528: write d3/d17/d3d/d5d/f73 [1040941,124487] 0 2024-02-28T16:31:10.898 INFO:tasks.workunit.client.0.smithi001.stdout:5/509: creat dc/d1a/d7d/d87/fa8 x:0 0 0 2024-02-28T16:31:10.899 INFO:tasks.workunit.client.1.smithi066.stdout:8/63: dread d0/db/f10 [0,4194304] 0 2024-02-28T16:31:10.902 INFO:tasks.workunit.client.0.smithi001.stdout:8/539: sync 2024-02-28T16:31:10.902 INFO:tasks.workunit.client.0.smithi001.stdout:3/539: sync 2024-02-28T16:31:10.903 INFO:tasks.workunit.client.0.smithi001.stdout:5/510: chown dc/d1a/d21/d2d/d3c/f5e 2 1 2024-02-28T16:31:10.905 INFO:tasks.workunit.client.1.smithi066.stdout:4/147: creat d1/d18/f2d x:0 0 0 2024-02-28T16:31:10.905 INFO:tasks.workunit.client.1.smithi066.stdout:1/152: mknod d2/c30 0 2024-02-28T16:31:10.907 INFO:tasks.workunit.client.1.smithi066.stdout:8/64: mkdir d0/db/d12 0 2024-02-28T16:31:10.912 INFO:tasks.workunit.client.1.smithi066.stdout:8/65: creat d0/f13 x:0 0 0 2024-02-28T16:31:10.915 INFO:tasks.workunit.client.0.smithi001.stdout:1/648: mknod d0/d9/d1a/d2b/d26/d27/d30/d88/db6/ce2 0 2024-02-28T16:31:10.916 INFO:tasks.workunit.client.1.smithi066.stdout:4/148: rename d1/c2c to d1/c2e 0 2024-02-28T16:31:10.917 INFO:tasks.workunit.client.1.smithi066.stdout:8/66: fdatasync d0/d6/f7 0 2024-02-28T16:31:10.917 INFO:tasks.workunit.client.0.smithi001.stdout:7/529: rename d3/f8 to d3/d17/d3d/d59/d6e/fac 0 2024-02-28T16:31:10.918 INFO:tasks.workunit.client.1.smithi066.stdout:4/149: chown d1/d18/c1c 31323 1 2024-02-28T16:31:10.918 INFO:tasks.workunit.client.0.smithi001.stdout:0/530: dwrite d18/d22/d2b/d36/d48/f9a [0,4194304] 0 2024-02-28T16:31:10.919 INFO:tasks.workunit.client.1.smithi066.stdout:1/153: dread d2/f7 [0,4194304] 0 2024-02-28T16:31:10.925 INFO:tasks.workunit.client.0.smithi001.stdout:6/507: write d9/d29/d6e/f18 [623330,113420] 0 2024-02-28T16:31:10.928 INFO:tasks.workunit.client.0.smithi001.stdout:7/530: creat d3/d1c/fad x:0 0 0 2024-02-28T16:31:10.929 INFO:tasks.workunit.client.1.smithi066.stdout:7/51: sync 2024-02-28T16:31:10.934 INFO:tasks.workunit.client.0.smithi001.stdout:9/630: dread d5/d45/d6f/f7d [0,4194304] 0 2024-02-28T16:31:10.937 INFO:tasks.workunit.client.0.smithi001.stdout:1/649: mknod d0/d9/d1a/d2b/d26/d27/d30/d88/db6/ce3 0 2024-02-28T16:31:10.938 INFO:tasks.workunit.client.0.smithi001.stdout:7/531: truncate d3/d17/d3d/f72 4934748 0 2024-02-28T16:31:10.939 INFO:tasks.workunit.client.0.smithi001.stdout:8/540: creat d4/d7/d22/d27/d8e/dae/faf x:0 0 0 2024-02-28T16:31:10.942 INFO:tasks.workunit.client.0.smithi001.stdout:0/531: rmdir d18/d22/d2b/d36/d4d 39 2024-02-28T16:31:10.944 INFO:tasks.workunit.client.1.smithi066.stdout:8/67: truncate d0/d6/f9 282813 0 2024-02-28T16:31:10.945 INFO:tasks.workunit.client.0.smithi001.stdout:5/511: dwrite dc/d12/d41/f9c [0,4194304] 0 2024-02-28T16:31:10.947 INFO:tasks.workunit.client.0.smithi001.stdout:4/568: write d3/d2d/f63 [1914750,2596] 0 2024-02-28T16:31:10.947 INFO:tasks.workunit.client.1.smithi066.stdout:9/96: dwrite f3 [0,4194304] 0 2024-02-28T16:31:10.948 INFO:tasks.workunit.client.1.smithi066.stdout:1/154: dwrite d2/d27/f2f [0,4194304] 0 2024-02-28T16:31:10.950 INFO:tasks.workunit.client.0.smithi001.stdout:8/541: write d4/d7/d22/d27/d4b/d77/f8d [1837036,22395] 0 2024-02-28T16:31:10.950 INFO:tasks.workunit.client.0.smithi001.stdout:3/540: getdents d4/d1e 0 2024-02-28T16:31:10.950 INFO:tasks.workunit.client.0.smithi001.stdout:1/650: chown d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/c98 119485 1 2024-02-28T16:31:10.952 INFO:tasks.workunit.client.0.smithi001.stdout:0/532: fdatasync d18/f6d 0 2024-02-28T16:31:10.952 INFO:tasks.workunit.client.0.smithi001.stdout:9/631: rename d5/c38 to d5/d27/db1/d58/cd5 0 2024-02-28T16:31:10.954 INFO:tasks.workunit.client.0.smithi001.stdout:6/508: rmdir d9/d29/d6e/d4e/d96 39 2024-02-28T16:31:10.956 INFO:tasks.workunit.client.0.smithi001.stdout:1/651: readlink d0/d9/d1a/d2b/d26/d27/d30/d52/d84/l97 0 2024-02-28T16:31:10.958 INFO:tasks.workunit.client.0.smithi001.stdout:4/569: truncate d3/d1c/da0/fb2 525315 0 2024-02-28T16:31:10.958 INFO:tasks.workunit.client.0.smithi001.stdout:8/542: write d4/d32/d93/fa2 [650192,28564] 0 2024-02-28T16:31:10.959 INFO:tasks.workunit.client.0.smithi001.stdout:2/564: dread d0/d2/d3/de/d22/f24 [0,4194304] 0 2024-02-28T16:31:10.959 INFO:tasks.workunit.client.1.smithi066.stdout:4/150: chown d1/c2e 465682143 1 2024-02-28T16:31:10.960 INFO:tasks.workunit.client.1.smithi066.stdout:1/155: creat d2/d18/d19/f31 x:0 0 0 2024-02-28T16:31:10.960 INFO:tasks.workunit.client.0.smithi001.stdout:7/532: write d3/d17/d3d/d59/d6e/fac [2103369,5985] 0 2024-02-28T16:31:10.960 INFO:tasks.workunit.client.0.smithi001.stdout:0/533: mknod d18/d22/d76/caf 0 2024-02-28T16:31:10.961 INFO:tasks.workunit.client.0.smithi001.stdout:8/543: stat d4 0 2024-02-28T16:31:10.962 INFO:tasks.workunit.client.0.smithi001.stdout:6/509: fsync d9/db/d5f/d8b/f72 0 2024-02-28T16:31:10.963 INFO:tasks.workunit.client.1.smithi066.stdout:9/97: rename f8 to da/d10/d15/f19 0 2024-02-28T16:31:10.965 INFO:tasks.workunit.client.0.smithi001.stdout:4/570: unlink d3/d2d/d4c/d69/d82/f53 0 2024-02-28T16:31:10.969 INFO:tasks.workunit.client.0.smithi001.stdout:0/534: write d18/d22/d57/d7e/faa [846001,96775] 0 2024-02-28T16:31:10.970 INFO:tasks.workunit.client.1.smithi066.stdout:4/151: write d1/f12 [2700380,36614] 0 2024-02-28T16:31:10.972 INFO:tasks.workunit.client.0.smithi001.stdout:1/652: creat d0/d9/d1a/d2b/d26/d27/d30/d88/fe4 x:0 0 0 2024-02-28T16:31:10.973 INFO:tasks.workunit.client.0.smithi001.stdout:8/544: creat d4/d7/d22/d27/d8e/dae/fb0 x:0 0 0 2024-02-28T16:31:10.974 INFO:tasks.workunit.client.1.smithi066.stdout:9/98: write da/fb [4481401,113447] 0 2024-02-28T16:31:10.975 INFO:tasks.workunit.client.1.smithi066.stdout:8/68: dread d0/f4 [0,4194304] 0 2024-02-28T16:31:10.975 INFO:tasks.workunit.client.0.smithi001.stdout:3/541: creat d4/d6/fb5 x:0 0 0 2024-02-28T16:31:10.976 INFO:tasks.workunit.client.0.smithi001.stdout:5/512: rename dc/d12/d56/c65 to dc/d12/ca9 0 2024-02-28T16:31:10.976 INFO:tasks.workunit.client.0.smithi001.stdout:3/542: write d4/d18/faf [1008345,6667] 0 2024-02-28T16:31:10.980 INFO:tasks.workunit.client.1.smithi066.stdout:9/99: rename da/d10 to da/d10/d15/d1a 22 2024-02-28T16:31:10.986 INFO:tasks.workunit.client.0.smithi001.stdout:2/565: creat d0/d2/d3/de/d22/dbb/fbf x:0 0 0 2024-02-28T16:31:10.990 INFO:tasks.workunit.client.1.smithi066.stdout:3/87: dwrite f5 [0,4194304] 0 2024-02-28T16:31:10.994 INFO:tasks.workunit.client.0.smithi001.stdout:3/543: rename d4/d3c/c4d to d4/d6/d7f/d82/cb6 0 2024-02-28T16:31:10.995 INFO:tasks.workunit.client.0.smithi001.stdout:8/545: symlink d4/lb1 0 2024-02-28T16:31:10.999 INFO:tasks.workunit.client.0.smithi001.stdout:1/653: mknod d0/d9/d1a/d2b/d26/d27/ce5 0 2024-02-28T16:31:11.000 INFO:tasks.workunit.client.0.smithi001.stdout:2/566: rmdir d0/d2/da6 39 2024-02-28T16:31:11.002 INFO:tasks.workunit.client.1.smithi066.stdout:7/52: dwrite f4 [0,4194304] 0 2024-02-28T16:31:11.003 INFO:tasks.workunit.client.0.smithi001.stdout:2/567: write d0/f59 [1173978,16578] 0 2024-02-28T16:31:11.003 INFO:tasks.workunit.client.1.smithi066.stdout:9/100: symlink da/df/d18/l1b 0 2024-02-28T16:31:11.005 INFO:tasks.workunit.client.0.smithi001.stdout:1/654: chown d0/d9/d1a/d2b/d26/d27/d30/f58 21 1 2024-02-28T16:31:11.007 INFO:tasks.workunit.client.1.smithi066.stdout:5/98: sync 2024-02-28T16:31:11.008 INFO:tasks.workunit.client.1.smithi066.stdout:4/152: truncate d1/f4 1676153 0 2024-02-28T16:31:11.009 INFO:tasks.workunit.client.1.smithi066.stdout:1/156: dwrite d2/fc [0,4194304] 0 2024-02-28T16:31:11.013 INFO:tasks.workunit.client.1.smithi066.stdout:3/88: creat db/d13/d1a/f1b x:0 0 0 2024-02-28T16:31:11.015 INFO:tasks.workunit.client.1.smithi066.stdout:9/101: symlink da/df/d18/l1c 0 2024-02-28T16:31:11.018 INFO:tasks.workunit.client.0.smithi001.stdout:1/655: mknod d0/d9/d1a/d72/d9d/ce6 0 2024-02-28T16:31:11.018 INFO:tasks.workunit.client.1.smithi066.stdout:3/89: stat db/d13/d1a/f1b 0 2024-02-28T16:31:11.022 INFO:tasks.workunit.client.1.smithi066.stdout:8/69: dwrite d0/f13 [0,4194304] 0 2024-02-28T16:31:11.022 INFO:tasks.workunit.client.1.smithi066.stdout:7/53: creat dc/de/f10 x:0 0 0 2024-02-28T16:31:11.024 INFO:tasks.workunit.client.1.smithi066.stdout:7/54: stat dc/de/f10 0 2024-02-28T16:31:11.024 INFO:tasks.workunit.client.1.smithi066.stdout:9/102: creat da/d10/d15/f1d x:0 0 0 2024-02-28T16:31:11.025 INFO:tasks.workunit.client.0.smithi001.stdout:2/568: truncate d0/f7f 921319 0 2024-02-28T16:31:11.025 INFO:tasks.workunit.client.0.smithi001.stdout:5/513: dwrite dc/d1a/d21/d2d/d3c/f46 [0,4194304] 0 2024-02-28T16:31:11.026 INFO:tasks.workunit.client.1.smithi066.stdout:3/90: truncate db/d13/d1a/f1b 652527 0 2024-02-28T16:31:11.026 INFO:tasks.workunit.client.1.smithi066.stdout:7/55: fdatasync dc/de/f10 0 2024-02-28T16:31:11.027 INFO:tasks.workunit.client.1.smithi066.stdout:9/103: readlink da/df/d18/l1c 0 2024-02-28T16:31:11.035 INFO:tasks.workunit.client.1.smithi066.stdout:1/157: dread d2/fc [0,4194304] 0 2024-02-28T16:31:11.035 INFO:tasks.workunit.client.1.smithi066.stdout:4/153: read d1/f5 [1036892,82169] 0 2024-02-28T16:31:11.039 INFO:tasks.workunit.client.0.smithi001.stdout:3/544: dread d4/d6/f40 [0,4194304] 0 2024-02-28T16:31:11.040 INFO:tasks.workunit.client.1.smithi066.stdout:3/91: symlink db/d13/d1a/l1c 0 2024-02-28T16:31:11.041 INFO:tasks.workunit.client.1.smithi066.stdout:3/92: fsync f5 0 2024-02-28T16:31:11.043 INFO:tasks.workunit.client.0.smithi001.stdout:0/535: dwrite d18/d22/d47/f86 [0,4194304] 0 2024-02-28T16:31:11.043 INFO:tasks.workunit.client.1.smithi066.stdout:5/99: getdents d3/d1b 0 2024-02-28T16:31:11.044 INFO:tasks.workunit.client.0.smithi001.stdout:7/533: dread d3/d17/f1b [0,4194304] 0 2024-02-28T16:31:11.044 INFO:tasks.workunit.client.0.smithi001.stdout:4/571: dwrite d3/d2d/d4c/d69/d82/f7c [0,4194304] 0 2024-02-28T16:31:11.047 INFO:tasks.workunit.client.1.smithi066.stdout:4/154: creat d1/d18/f2f x:0 0 0 2024-02-28T16:31:11.049 INFO:tasks.workunit.client.0.smithi001.stdout:9/632: dread d5/d27/f5c [0,4194304] 0 2024-02-28T16:31:11.050 INFO:tasks.workunit.client.1.smithi066.stdout:1/158: creat d2/d18/d19/f32 x:0 0 0 2024-02-28T16:31:11.052 INFO:tasks.workunit.client.0.smithi001.stdout:6/510: dwrite d9/db/d5f/d62/f6b [0,4194304] 0 2024-02-28T16:31:11.052 INFO:tasks.workunit.client.0.smithi001.stdout:2/569: creat d0/fc0 x:0 0 0 2024-02-28T16:31:11.052 INFO:tasks.workunit.client.0.smithi001.stdout:3/545: mkdir d4/d1e/db7 0 2024-02-28T16:31:11.054 INFO:tasks.workunit.client.1.smithi066.stdout:5/100: symlink d3/d4/dc/l23 0 2024-02-28T16:31:11.055 INFO:tasks.workunit.client.1.smithi066.stdout:9/104: creat da/f1e x:0 0 0 2024-02-28T16:31:11.055 INFO:tasks.workunit.client.1.smithi066.stdout:3/93: mknod db/dc/c1d 0 2024-02-28T16:31:11.055 INFO:tasks.workunit.client.0.smithi001.stdout:4/572: mknod d3/d5/d4e/cb8 0 2024-02-28T16:31:11.056 INFO:tasks.workunit.client.1.smithi066.stdout:0/195: sync 2024-02-28T16:31:11.056 INFO:tasks.workunit.client.1.smithi066.stdout:2/105: sync 2024-02-28T16:31:11.056 INFO:tasks.workunit.client.1.smithi066.stdout:6/38: sync 2024-02-28T16:31:11.057 INFO:tasks.workunit.client.0.smithi001.stdout:4/573: readlink d3/d2d/d4c/d69/d82/l52 0 2024-02-28T16:31:11.057 INFO:tasks.workunit.client.1.smithi066.stdout:3/94: fdatasync db/ff 0 2024-02-28T16:31:11.058 INFO:tasks.workunit.client.0.smithi001.stdout:9/633: unlink d5/f79 0 2024-02-28T16:31:11.058 INFO:tasks.workunit.client.0.smithi001.stdout:0/536: fsync f13 0 2024-02-28T16:31:11.058 INFO:tasks.workunit.client.0.smithi001.stdout:3/546: creat d4/d18/d5f/fb8 x:0 0 0 2024-02-28T16:31:11.061 INFO:tasks.workunit.client.1.smithi066.stdout:9/105: dread - da/d10/d15/f1d zero size 2024-02-28T16:31:11.062 INFO:tasks.workunit.client.1.smithi066.stdout:7/56: dwrite f9 [0,4194304] 0 2024-02-28T16:31:11.064 INFO:tasks.workunit.client.0.smithi001.stdout:0/537: truncate d18/d22/d47/fa2 756492 0 2024-02-28T16:31:11.066 INFO:tasks.workunit.client.1.smithi066.stdout:3/95: write db/d13/d14/f15 [349138,13876] 0 2024-02-28T16:31:11.067 INFO:tasks.workunit.client.1.smithi066.stdout:7/57: read f5 [591582,5785] 0 2024-02-28T16:31:11.069 INFO:tasks.workunit.client.1.smithi066.stdout:6/39: dread - d2/f8 zero size 2024-02-28T16:31:11.069 INFO:tasks.workunit.client.1.smithi066.stdout:1/159: link d2/f5 d2/d18/d19/f33 0 2024-02-28T16:31:11.071 INFO:tasks.workunit.client.0.smithi001.stdout:4/574: creat d3/d2d/d83/fb9 x:0 0 0 2024-02-28T16:31:11.071 INFO:tasks.workunit.client.0.smithi001.stdout:1/656: link d0/d9/d1a/d2b/d26/d27/d30/d52/cad d0/d9/d1a/d2b/d26/d27/dd3/ce7 0 2024-02-28T16:31:11.072 INFO:tasks.workunit.client.1.smithi066.stdout:4/155: mknod d1/c30 0 2024-02-28T16:31:11.076 INFO:tasks.workunit.client.0.smithi001.stdout:6/511: creat d9/fab x:0 0 0 2024-02-28T16:31:11.076 INFO:tasks.workunit.client.0.smithi001.stdout:0/538: creat d18/d22/d2b/d95/fb0 x:0 0 0 2024-02-28T16:31:11.076 INFO:tasks.workunit.client.0.smithi001.stdout:1/657: write d0/d9/d1a/d2b/d26/d27/d30/d67/db7/fe1 [341482,43957] 0 2024-02-28T16:31:11.077 INFO:tasks.workunit.client.1.smithi066.stdout:0/196: mknod d2/d27/c3e 0 2024-02-28T16:31:11.078 INFO:tasks.workunit.client.0.smithi001.stdout:9/634: symlink d5/d27/d34/dc1/ld6 0 2024-02-28T16:31:11.080 INFO:tasks.workunit.client.1.smithi066.stdout:3/96: readlink db/dc/d11/l16 0 2024-02-28T16:31:11.081 INFO:tasks.workunit.client.0.smithi001.stdout:9/635: fsync d5/f12 0 2024-02-28T16:31:11.081 INFO:tasks.workunit.client.0.smithi001.stdout:3/547: link d4/d1e/f25 d4/d6/d7/d78/d81/da1/fb9 0 2024-02-28T16:31:11.082 INFO:tasks.workunit.client.1.smithi066.stdout:7/58: mknod dc/de/c11 0 2024-02-28T16:31:11.082 INFO:tasks.workunit.client.0.smithi001.stdout:5/514: write dc/d1a/d21/f48 [630397,92264] 0 2024-02-28T16:31:11.085 INFO:tasks.workunit.client.1.smithi066.stdout:4/156: chown d1/d23/f27 9 1 2024-02-28T16:31:11.086 INFO:tasks.workunit.client.1.smithi066.stdout:7/59: stat c7 0 2024-02-28T16:31:11.087 INFO:tasks.workunit.client.1.smithi066.stdout:7/60: chown dc 4 1 2024-02-28T16:31:11.091 INFO:tasks.workunit.client.1.smithi066.stdout:0/197: mkdir d2/d9/d33/d3f 0 2024-02-28T16:31:11.092 INFO:tasks.workunit.client.0.smithi001.stdout:1/658: readlink d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/lc7 0 2024-02-28T16:31:11.092 INFO:tasks.workunit.client.0.smithi001.stdout:6/512: creat d9/d29/d6e/d53/fac x:0 0 0 2024-02-28T16:31:11.093 INFO:tasks.workunit.client.1.smithi066.stdout:2/106: dread d0/f9 [0,4194304] 0 2024-02-28T16:31:11.095 INFO:tasks.workunit.client.1.smithi066.stdout:1/160: readlink d2/d18/l1b 0 2024-02-28T16:31:11.095 INFO:tasks.workunit.client.1.smithi066.stdout:3/97: chown db/d13/d1a/l1c 16409008 1 2024-02-28T16:31:11.096 INFO:tasks.workunit.client.1.smithi066.stdout:9/106: link da/d10/c17 da/df/c1f 0 2024-02-28T16:31:11.098 INFO:tasks.workunit.client.0.smithi001.stdout:5/515: mkdir dc/d12/daa 0 2024-02-28T16:31:11.098 INFO:tasks.workunit.client.1.smithi066.stdout:5/101: dwrite d3/d4/d17/d18/f1a [0,4194304] 0 2024-02-28T16:31:11.099 INFO:tasks.workunit.client.0.smithi001.stdout:0/539: rename d18/d22/d47/f86 to d18/d22/d2b/d36/d46/fb1 0 2024-02-28T16:31:11.100 INFO:tasks.workunit.client.1.smithi066.stdout:8/70: truncate d0/d6/f9 601580 0 2024-02-28T16:31:11.101 INFO:tasks.workunit.client.1.smithi066.stdout:4/157: fsync d1/f5 0 2024-02-28T16:31:11.101 INFO:tasks.workunit.client.1.smithi066.stdout:0/198: creat d2/d9/f40 x:0 0 0 2024-02-28T16:31:11.105 INFO:tasks.workunit.client.1.smithi066.stdout:5/102: dread - d3/d4/dc/f21 zero size 2024-02-28T16:31:11.106 INFO:tasks.workunit.client.0.smithi001.stdout:7/534: dwrite d3/d17/f19 [0,4194304] 0 2024-02-28T16:31:11.106 INFO:tasks.workunit.client.1.smithi066.stdout:9/107: readlink da/df/d18/l1c 0 2024-02-28T16:31:11.108 INFO:tasks.workunit.client.1.smithi066.stdout:2/107: rename d0/d12/f14 to d0/f1e 0 2024-02-28T16:31:11.109 INFO:tasks.workunit.client.0.smithi001.stdout:5/516: dread - dc/d1a/d7d/f88 zero size 2024-02-28T16:31:11.112 INFO:tasks.workunit.client.1.smithi066.stdout:6/40: dread f1 [0,4194304] 0 2024-02-28T16:31:11.112 INFO:tasks.workunit.client.1.smithi066.stdout:0/199: mkdir d2/d27/d41 0 2024-02-28T16:31:11.113 INFO:tasks.workunit.client.0.smithi001.stdout:3/548: creat d4/d1e/d7c/d9a/fba x:0 0 0 2024-02-28T16:31:11.113 INFO:tasks.workunit.client.0.smithi001.stdout:6/513: write d9/db/d5f/f65 [713443,5990] 0 2024-02-28T16:31:11.115 INFO:tasks.workunit.client.1.smithi066.stdout:4/158: dread d1/f6 [0,4194304] 0 2024-02-28T16:31:11.119 INFO:tasks.workunit.client.0.smithi001.stdout:7/535: readlink d3/d17/d52/l8b 0 2024-02-28T16:31:11.124 INFO:tasks.workunit.client.0.smithi001.stdout:4/575: dwrite d3/f78 [0,4194304] 0 2024-02-28T16:31:11.124 INFO:tasks.workunit.client.0.smithi001.stdout:7/536: chown d3/d17 37 1 2024-02-28T16:31:11.124 INFO:tasks.workunit.client.1.smithi066.stdout:4/159: dread - d1/d18/f2b zero size 2024-02-28T16:31:11.124 INFO:tasks.workunit.client.1.smithi066.stdout:3/98: truncate db/ff 103879 0 2024-02-28T16:31:11.124 INFO:tasks.workunit.client.1.smithi066.stdout:4/160: fdatasync d1/f10 0 2024-02-28T16:31:11.125 INFO:tasks.workunit.client.1.smithi066.stdout:3/99: fdatasync db/d13/d14/f15 0 2024-02-28T16:31:11.130 INFO:tasks.workunit.client.1.smithi066.stdout:3/100: readlink l6 0 2024-02-28T16:31:11.131 INFO:tasks.workunit.client.0.smithi001.stdout:7/537: write d3/d1c/f33 [6059451,13530] 0 2024-02-28T16:31:11.132 INFO:tasks.workunit.client.0.smithi001.stdout:8/546: dread d4/d47/d8b/f8c [4194304,4194304] 0 2024-02-28T16:31:11.133 INFO:tasks.workunit.client.0.smithi001.stdout:7/538: stat d3/d17/d3d/d59/d6e 0 2024-02-28T16:31:11.135 INFO:tasks.workunit.client.0.smithi001.stdout:7/539: stat d3/d17/d3d/fa7 0 2024-02-28T16:31:11.136 INFO:tasks.workunit.client.1.smithi066.stdout:3/101: readlink db/l12 0 2024-02-28T16:31:11.139 INFO:tasks.workunit.client.1.smithi066.stdout:9/108: dread da/d10/f13 [0,4194304] 0 2024-02-28T16:31:11.140 INFO:tasks.workunit.client.0.smithi001.stdout:9/636: dwrite d5/d27/d34/d50/f92 [0,4194304] 0 2024-02-28T16:31:11.142 INFO:tasks.workunit.client.1.smithi066.stdout:8/71: dwrite d0/d6/ff [0,4194304] 0 2024-02-28T16:31:11.143 INFO:tasks.workunit.client.0.smithi001.stdout:0/540: dwrite d18/d31/f9e [0,4194304] 0 2024-02-28T16:31:11.144 INFO:tasks.workunit.client.1.smithi066.stdout:1/161: write d2/d18/f21 [79757,62790] 0 2024-02-28T16:31:11.148 INFO:tasks.workunit.client.1.smithi066.stdout:2/108: write d0/f8 [452152,42618] 0 2024-02-28T16:31:11.151 INFO:tasks.workunit.client.1.smithi066.stdout:3/102: creat db/d13/d1a/f1e x:0 0 0 2024-02-28T16:31:11.151 INFO:tasks.workunit.client.1.smithi066.stdout:6/41: dwrite d2/f3 [0,4194304] 0 2024-02-28T16:31:11.154 INFO:tasks.workunit.client.1.smithi066.stdout:4/161: symlink d1/l31 0 2024-02-28T16:31:11.157 INFO:tasks.workunit.client.1.smithi066.stdout:4/162: readlink d1/l17 0 2024-02-28T16:31:11.158 INFO:tasks.workunit.client.0.smithi001.stdout:5/517: link dc/d1a/d21/d2d/d3c/d4a/f75 dc/d1a/d21/d2d/d3c/da0/fab 0 2024-02-28T16:31:11.159 INFO:tasks.workunit.client.0.smithi001.stdout:9/637: symlink d5/d27/d34/ld7 0 2024-02-28T16:31:11.159 INFO:tasks.workunit.client.0.smithi001.stdout:8/547: creat d4/d7/d22/d27/d58/d62/fb2 x:0 0 0 2024-02-28T16:31:11.159 INFO:tasks.workunit.client.1.smithi066.stdout:9/109: rmdir da 39 2024-02-28T16:31:11.161 INFO:tasks.workunit.client.0.smithi001.stdout:0/541: mknod d18/d22/d76/d93/cb2 0 2024-02-28T16:31:11.163 INFO:tasks.workunit.client.0.smithi001.stdout:3/549: dwrite d4/d1e/f25 [0,4194304] 0 2024-02-28T16:31:11.163 INFO:tasks.workunit.client.0.smithi001.stdout:0/542: fdatasync d18/d22/d3d/f67 0 2024-02-28T16:31:11.167 INFO:tasks.workunit.client.0.smithi001.stdout:0/543: readlink d18/d22/l98 0 2024-02-28T16:31:11.168 INFO:tasks.workunit.client.0.smithi001.stdout:6/514: getdents d9/d29/d6e/d4e/d96 0 2024-02-28T16:31:11.169 INFO:tasks.workunit.client.1.smithi066.stdout:1/162: mkdir d2/d27/d2b/d34 0 2024-02-28T16:31:11.170 INFO:tasks.workunit.client.1.smithi066.stdout:2/109: fdatasync d0/f9 0 2024-02-28T16:31:11.172 INFO:tasks.workunit.client.1.smithi066.stdout:0/200: getdents d2/d27 0 2024-02-28T16:31:11.175 INFO:tasks.workunit.client.1.smithi066.stdout:3/103: mkdir db/dc/d11/d1f 0 2024-02-28T16:31:11.176 INFO:tasks.workunit.client.1.smithi066.stdout:0/201: dread - d2/f32 zero size 2024-02-28T16:31:11.179 INFO:tasks.workunit.client.1.smithi066.stdout:5/103: truncate d3/d4/d17/d18/f1a 3458609 0 2024-02-28T16:31:11.185 INFO:tasks.workunit.client.0.smithi001.stdout:1/659: dread d0/d9/d1a/f4c [4194304,4194304] 0 2024-02-28T16:31:11.186 INFO:tasks.workunit.client.1.smithi066.stdout:8/72: creat d0/d6/dc/f14 x:0 0 0 2024-02-28T16:31:11.186 INFO:tasks.workunit.client.1.smithi066.stdout:8/73: readlink - no filename 2024-02-28T16:31:11.186 INFO:tasks.workunit.client.1.smithi066.stdout:8/74: readlink - no filename 2024-02-28T16:31:11.186 INFO:tasks.workunit.client.0.smithi001.stdout:9/638: dread d5/d45/d6f/dc8/fac [0,4194304] 0 2024-02-28T16:31:11.188 INFO:tasks.workunit.client.1.smithi066.stdout:3/104: mknod db/d13/d1a/c20 0 2024-02-28T16:31:11.188 INFO:tasks.workunit.client.1.smithi066.stdout:8/75: stat d0/db/d12 0 2024-02-28T16:31:11.188 INFO:tasks.workunit.client.1.smithi066.stdout:8/76: readlink - no filename 2024-02-28T16:31:11.190 INFO:tasks.workunit.client.0.smithi001.stdout:7/540: dwrite d3/d1c/f51 [0,4194304] 0 2024-02-28T16:31:11.192 INFO:tasks.workunit.client.1.smithi066.stdout:0/202: mknod d2/c42 0 2024-02-28T16:31:11.192 INFO:tasks.workunit.client.1.smithi066.stdout:2/110: unlink d0/c11 0 2024-02-28T16:31:11.192 INFO:tasks.workunit.client.1.smithi066.stdout:4/163: dwrite d1/f12 [4194304,4194304] 0 2024-02-28T16:31:11.193 INFO:tasks.workunit.client.0.smithi001.stdout:0/544: dwrite d18/f6d [4194304,4194304] 0 2024-02-28T16:31:11.193 INFO:tasks.workunit.client.0.smithi001.stdout:1/660: stat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/lc7 0 2024-02-28T16:31:11.194 INFO:tasks.workunit.client.0.smithi001.stdout:7/541: fdatasync d3/d17/f19 0 2024-02-28T16:31:11.194 INFO:tasks.workunit.client.0.smithi001.stdout:9/639: unlink d5/d27/db1/l5b 0 2024-02-28T16:31:11.195 INFO:tasks.workunit.client.1.smithi066.stdout:3/105: read db/d13/d14/f15 [858980,116022] 0 2024-02-28T16:31:11.195 INFO:tasks.workunit.client.1.smithi066.stdout:4/164: write d1/f2a [127104,93700] 0 2024-02-28T16:31:11.197 INFO:tasks.workunit.client.0.smithi001.stdout:8/548: dwrite d4/d7/d22/d27/d4b/d80/f85 [0,4194304] 0 2024-02-28T16:31:11.198 INFO:tasks.workunit.client.0.smithi001.stdout:7/542: write d3/d1c/fad [336289,117973] 0 2024-02-28T16:31:11.200 INFO:tasks.workunit.client.1.smithi066.stdout:2/111: creat d0/f1f x:0 0 0 2024-02-28T16:31:11.200 INFO:tasks.workunit.client.1.smithi066.stdout:3/106: fsync db/d13/f19 0 2024-02-28T16:31:11.204 INFO:tasks.workunit.client.1.smithi066.stdout:3/107: chown db/d13/d1a 293550156 1 2024-02-28T16:31:11.207 INFO:tasks.workunit.client.0.smithi001.stdout:1/661: mkdir d0/d9/d1a/d2b/d26/d27/dd3/de8 0 2024-02-28T16:31:11.207 INFO:tasks.workunit.client.1.smithi066.stdout:3/108: dread - db/d13/d1a/f1e zero size 2024-02-28T16:31:11.209 INFO:tasks.workunit.client.0.smithi001.stdout:7/543: chown d3/d1c/c82 498051395 1 2024-02-28T16:31:11.210 INFO:tasks.workunit.client.1.smithi066.stdout:6/42: fdatasync d2/f3 0 2024-02-28T16:31:11.215 INFO:tasks.workunit.client.0.smithi001.stdout:9/640: link d5/d27/db1/c7b d5/d45/d6f/db3/cd8 0 2024-02-28T16:31:11.215 INFO:tasks.workunit.client.1.smithi066.stdout:6/43: write d2/f3 [2423663,23530] 0 2024-02-28T16:31:11.217 INFO:tasks.workunit.client.1.smithi066.stdout:6/44: stat d2/f3 0 2024-02-28T16:31:11.217 INFO:tasks.workunit.client.0.smithi001.stdout:7/544: creat d3/d17/d3d/fae x:0 0 0 2024-02-28T16:31:11.219 INFO:tasks.workunit.client.0.smithi001.stdout:4/576: truncate d3/d2d/d4c/d5e/f62 990147 0 2024-02-28T16:31:11.223 INFO:tasks.workunit.client.1.smithi066.stdout:6/45: creat d2/f9 x:0 0 0 2024-02-28T16:31:11.226 INFO:tasks.workunit.client.0.smithi001.stdout:0/545: dwrite d18/d22/d3d/f50 [0,4194304] 0 2024-02-28T16:31:11.226 INFO:tasks.workunit.client.1.smithi066.stdout:4/165: write d1/f13 [2054066,34135] 0 2024-02-28T16:31:11.228 INFO:tasks.workunit.client.1.smithi066.stdout:9/110: dread da/d10/d15/f19 [4194304,4194304] 0 2024-02-28T16:31:11.230 INFO:tasks.workunit.client.0.smithi001.stdout:0/546: write d18/fac [488021,128461] 0 2024-02-28T16:31:11.230 INFO:tasks.workunit.client.0.smithi001.stdout:9/641: unlink d5/d27/db1/d58/l9f 0 2024-02-28T16:31:11.238 INFO:tasks.workunit.client.0.smithi001.stdout:4/577: rename d3/d1c/f4a to d3/d2d/d4c/d5e/db0/fba 0 2024-02-28T16:31:11.241 INFO:tasks.workunit.client.0.smithi001.stdout:9/642: readlink d5/l1c 0 2024-02-28T16:31:11.244 INFO:tasks.workunit.client.1.smithi066.stdout:9/111: creat da/f20 x:0 0 0 2024-02-28T16:31:11.244 INFO:tasks.workunit.client.0.smithi001.stdout:5/518: dwrite fa [0,4194304] 0 2024-02-28T16:31:11.248 INFO:tasks.workunit.client.0.smithi001.stdout:4/578: creat d3/d1c/d58/d6e/fbb x:0 0 0 2024-02-28T16:31:11.250 INFO:tasks.workunit.client.1.smithi066.stdout:0/203: dwrite d2/d9/d10/f2c [0,4194304] 0 2024-02-28T16:31:11.252 INFO:tasks.workunit.client.0.smithi001.stdout:8/549: dwrite d4/d32/f46 [0,4194304] 0 2024-02-28T16:31:11.254 INFO:tasks.workunit.client.1.smithi066.stdout:4/166: link d1/f11 d1/d23/d25/f32 0 2024-02-28T16:31:11.258 INFO:tasks.workunit.client.0.smithi001.stdout:4/579: truncate d3/d2d/d83/fb9 888530 0 2024-02-28T16:31:11.258 INFO:tasks.workunit.client.0.smithi001.stdout:9/643: mkdir d5/d45/dd9 0 2024-02-28T16:31:11.259 INFO:tasks.workunit.client.1.smithi066.stdout:4/167: fdatasync d1/f1e 0 2024-02-28T16:31:11.262 INFO:tasks.workunit.client.1.smithi066.stdout:8/77: dwrite d0/db/f10 [0,4194304] 0 2024-02-28T16:31:11.263 INFO:tasks.workunit.client.1.smithi066.stdout:2/112: dwrite d0/ff [0,4194304] 0 2024-02-28T16:31:11.263 INFO:tasks.workunit.client.1.smithi066.stdout:8/78: chown d0/db/d12 3465614 1 2024-02-28T16:31:11.267 INFO:tasks.workunit.client.1.smithi066.stdout:4/168: write d1/f4 [2616784,50362] 0 2024-02-28T16:31:11.267 INFO:tasks.workunit.client.1.smithi066.stdout:3/109: dwrite db/d13/f18 [0,4194304] 0 2024-02-28T16:31:11.268 INFO:tasks.workunit.client.0.smithi001.stdout:0/547: getdents d18/d22/d2f/d1c 0 2024-02-28T16:31:11.268 INFO:tasks.workunit.client.0.smithi001.stdout:5/519: creat dc/d12/fac x:0 0 0 2024-02-28T16:31:11.270 INFO:tasks.workunit.client.1.smithi066.stdout:6/46: dwrite d2/f3 [0,4194304] 0 2024-02-28T16:31:11.270 INFO:tasks.workunit.client.1.smithi066.stdout:4/169: fsync d1/d23/f27 0 2024-02-28T16:31:11.275 INFO:tasks.workunit.client.1.smithi066.stdout:6/47: chown d2/f3 14523104 1 2024-02-28T16:31:11.275 INFO:tasks.workunit.client.1.smithi066.stdout:0/204: link d2/d9/d10/l17 d2/d9/l43 0 2024-02-28T16:31:11.278 INFO:tasks.workunit.client.1.smithi066.stdout:7/61: sync 2024-02-28T16:31:11.285 INFO:tasks.workunit.client.0.smithi001.stdout:7/545: write d3/d17/d3d/f4d [682758,88920] 0 2024-02-28T16:31:11.288 INFO:tasks.workunit.client.1.smithi066.stdout:3/110: mknod db/dc/d11/d1f/c21 0 2024-02-28T16:31:11.289 INFO:tasks.workunit.client.1.smithi066.stdout:2/113: symlink d0/d12/d1a/l20 0 2024-02-28T16:31:11.294 INFO:tasks.workunit.client.1.smithi066.stdout:6/48: mkdir d2/da 0 2024-02-28T16:31:11.294 INFO:tasks.workunit.client.1.smithi066.stdout:2/114: readlink d0/d12/d1a/l20 0 2024-02-28T16:31:11.296 INFO:tasks.workunit.client.1.smithi066.stdout:6/49: chown d2/f9 6268 1 2024-02-28T16:31:11.297 INFO:tasks.workunit.client.1.smithi066.stdout:6/50: readlink d2/l6 0 2024-02-28T16:31:11.297 INFO:tasks.workunit.client.1.smithi066.stdout:7/62: symlink dc/de/l12 0 2024-02-28T16:31:11.298 INFO:tasks.workunit.client.1.smithi066.stdout:3/111: creat db/dc/d11/f22 x:0 0 0 2024-02-28T16:31:11.301 INFO:tasks.workunit.client.1.smithi066.stdout:4/170: rename d1/f13 to d1/f33 0 2024-02-28T16:31:11.304 INFO:tasks.workunit.client.1.smithi066.stdout:0/205: link d2/d9/d10/d23/f2d d2/d9/f44 0 2024-02-28T16:31:11.307 INFO:tasks.workunit.client.0.smithi001.stdout:7/546: creat d3/d17/d61/faf x:0 0 0 2024-02-28T16:31:11.308 INFO:tasks.workunit.client.1.smithi066.stdout:6/51: unlink d2/l6 0 2024-02-28T16:31:11.308 INFO:tasks.workunit.client.1.smithi066.stdout:7/63: truncate fa 546596 0 2024-02-28T16:31:11.310 INFO:tasks.workunit.client.0.smithi001.stdout:8/550: fsync d4/d9b/fa7 0 2024-02-28T16:31:11.312 INFO:tasks.workunit.client.0.smithi001.stdout:7/547: write d3/d17/d3d/d5d/d9a/d49/fa8 [980715,15290] 0 2024-02-28T16:31:11.312 INFO:tasks.workunit.client.1.smithi066.stdout:3/112: creat db/d13/f23 x:0 0 0 2024-02-28T16:31:11.313 INFO:tasks.workunit.client.0.smithi001.stdout:9/644: mknod d5/d27/d42/cda 0 2024-02-28T16:31:11.314 INFO:tasks.workunit.client.1.smithi066.stdout:2/115: mkdir d0/d12/d1d/d21 0 2024-02-28T16:31:11.316 INFO:tasks.workunit.client.1.smithi066.stdout:0/206: mknod d2/d9/c45 0 2024-02-28T16:31:11.316 INFO:tasks.workunit.client.1.smithi066.stdout:4/171: dread d1/f7 [0,4194304] 0 2024-02-28T16:31:11.316 INFO:tasks.workunit.client.0.smithi001.stdout:0/548: rename d18/d31/l97 to d18/d22/d2b/d36/d4d/lb3 0 2024-02-28T16:31:11.318 INFO:tasks.workunit.client.1.smithi066.stdout:6/52: write f1 [2810921,25231] 0 2024-02-28T16:31:11.320 INFO:tasks.workunit.client.0.smithi001.stdout:5/520: link dc/d1a/d21/d2d/d3c/f5e dc/d12/d56/fad 0 2024-02-28T16:31:11.320 INFO:tasks.workunit.client.1.smithi066.stdout:0/207: truncate d2/d9/f40 318291 0 2024-02-28T16:31:11.320 INFO:tasks.workunit.client.0.smithi001.stdout:4/580: truncate d3/d5/d7/d1f/d3a/d68/d87/f8b 636732 0 2024-02-28T16:31:11.321 INFO:tasks.workunit.client.1.smithi066.stdout:8/79: truncate d0/db/f10 3023901 0 2024-02-28T16:31:11.321 INFO:tasks.workunit.client.0.smithi001.stdout:5/521: stat dc/d12 0 2024-02-28T16:31:11.322 INFO:tasks.workunit.client.0.smithi001.stdout:8/551: mknod d4/d32/d69/d9a/cb3 0 2024-02-28T16:31:11.323 INFO:tasks.workunit.client.1.smithi066.stdout:2/116: mknod d0/c22 0 2024-02-28T16:31:11.323 INFO:tasks.workunit.client.1.smithi066.stdout:0/208: chown d2/c16 1939 1 2024-02-28T16:31:11.323 INFO:tasks.workunit.client.1.smithi066.stdout:3/113: mkdir db/d13/d14/d24 0 2024-02-28T16:31:11.332 INFO:tasks.workunit.client.0.smithi001.stdout:5/522: truncate dc/d12/d41/d54/fa5 762957 0 2024-02-28T16:31:11.332 INFO:tasks.workunit.client.0.smithi001.stdout:5/523: chown dc/l10 193 1 2024-02-28T16:31:11.332 INFO:tasks.workunit.client.0.smithi001.stdout:0/549: rename d18/l6b to d18/d22/d47/lb4 0 2024-02-28T16:31:11.333 INFO:tasks.workunit.client.0.smithi001.stdout:5/524: chown dc/d1a/d21/d2d/c9b 10654102 1 2024-02-28T16:31:11.333 INFO:tasks.workunit.client.1.smithi066.stdout:2/117: mkdir d0/d12/d1a/d23 0 2024-02-28T16:31:11.334 INFO:tasks.workunit.client.0.smithi001.stdout:0/550: dread - d18/d22/d76/f8b zero size 2024-02-28T16:31:11.334 INFO:tasks.workunit.client.0.smithi001.stdout:8/552: mkdir d4/d7/d22/d27/d58/d94/db4 0 2024-02-28T16:31:11.335 INFO:tasks.workunit.client.0.smithi001.stdout:4/581: truncate d3/d5/f91 3135189 0 2024-02-28T16:31:11.337 INFO:tasks.workunit.client.1.smithi066.stdout:6/53: dread f1 [4194304,4194304] 0 2024-02-28T16:31:11.337 INFO:tasks.workunit.client.1.smithi066.stdout:3/114: unlink db/dc/ld 0 2024-02-28T16:31:11.339 INFO:tasks.workunit.client.1.smithi066.stdout:6/54: fdatasync d2/f9 0 2024-02-28T16:31:11.340 INFO:tasks.workunit.client.1.smithi066.stdout:7/64: read fa [100050,32700] 0 2024-02-28T16:31:11.342 INFO:tasks.workunit.client.0.smithi001.stdout:5/525: mknod dc/d1a/d7d/d87/cae 0 2024-02-28T16:31:11.343 INFO:tasks.workunit.client.1.smithi066.stdout:4/172: link d1/d18/c1c d1/d18/c34 0 2024-02-28T16:31:11.343 INFO:tasks.workunit.client.1.smithi066.stdout:3/115: chown db/dc/d11/d1f/c21 574850 1 2024-02-28T16:31:11.343 INFO:tasks.workunit.client.0.smithi001.stdout:7/548: dwrite d3/d17/d3d/d5d/d9a/d49/f6c [0,4194304] 0 2024-02-28T16:31:11.343 INFO:tasks.workunit.client.1.smithi066.stdout:0/209: dread d2/d9/d10/f2c [0,4194304] 0 2024-02-28T16:31:11.345 INFO:tasks.workunit.client.0.smithi001.stdout:7/549: stat d3/l14 0 2024-02-28T16:31:11.345 INFO:tasks.workunit.client.1.smithi066.stdout:7/65: creat dc/de/f13 x:0 0 0 2024-02-28T16:31:11.348 INFO:tasks.workunit.client.1.smithi066.stdout:2/118: rename d0/c1c to d0/d12/d1a/c24 0 2024-02-28T16:31:11.349 INFO:tasks.workunit.client.1.smithi066.stdout:4/173: symlink d1/d23/d25/l35 0 2024-02-28T16:31:11.351 INFO:tasks.workunit.client.0.smithi001.stdout:9/645: dread d5/d27/db1/f41 [0,4194304] 0 2024-02-28T16:31:11.351 INFO:tasks.workunit.client.0.smithi001.stdout:5/526: rename dc/d12/l40 to dc/d1a/d7d/d87/laf 0 2024-02-28T16:31:11.352 INFO:tasks.workunit.client.1.smithi066.stdout:2/119: read d0/f1e [129881,15724] 0 2024-02-28T16:31:11.352 INFO:tasks.workunit.client.0.smithi001.stdout:9/646: chown d5/d27/d34 256475216 1 2024-02-28T16:31:11.353 INFO:tasks.workunit.client.0.smithi001.stdout:0/551: link c1 d18/d22/d57/cb5 0 2024-02-28T16:31:11.355 INFO:tasks.workunit.client.0.smithi001.stdout:7/550: mknod d3/d17/d3d/d5d/d9a/d49/d8d/cb0 0 2024-02-28T16:31:11.356 INFO:tasks.workunit.client.0.smithi001.stdout:8/553: dread d4/d7/f23 [0,4194304] 0 2024-02-28T16:31:11.356 INFO:tasks.workunit.client.1.smithi066.stdout:0/210: rmdir d2 39 2024-02-28T16:31:11.358 INFO:tasks.workunit.client.1.smithi066.stdout:7/66: unlink f9 0 2024-02-28T16:31:11.363 INFO:tasks.workunit.client.0.smithi001.stdout:9/647: rename d5/f3c to d5/d27/db1/d33/fdb 0 2024-02-28T16:31:11.364 INFO:tasks.workunit.client.1.smithi066.stdout:7/67: write f4 [3297679,88699] 0 2024-02-28T16:31:11.364 INFO:tasks.workunit.client.1.smithi066.stdout:2/120: unlink d0/d12/d1a/l20 0 2024-02-28T16:31:11.367 INFO:tasks.workunit.client.0.smithi001.stdout:5/527: symlink dc/d12/daa/lb0 0 2024-02-28T16:31:11.368 INFO:tasks.workunit.client.1.smithi066.stdout:6/55: dwrite d2/f9 [0,4194304] 0 2024-02-28T16:31:11.368 INFO:tasks.workunit.client.1.smithi066.stdout:1/163: sync 2024-02-28T16:31:11.368 INFO:tasks.workunit.client.0.smithi001.stdout:9/648: stat d5/d27/d42 0 2024-02-28T16:31:11.370 INFO:tasks.workunit.client.1.smithi066.stdout:2/121: write d0/d12/f13 [2882300,101970] 0 2024-02-28T16:31:11.374 INFO:tasks.workunit.client.1.smithi066.stdout:4/174: fdatasync d1/f11 0 2024-02-28T16:31:11.374 INFO:tasks.workunit.client.1.smithi066.stdout:0/211: fdatasync d2/f32 0 2024-02-28T16:31:11.374 INFO:tasks.workunit.client.1.smithi066.stdout:8/80: dwrite d0/db/f10 [0,4194304] 0 2024-02-28T16:31:11.376 INFO:tasks.workunit.client.1.smithi066.stdout:3/116: dwrite db/dc/d11/f22 [0,4194304] 0 2024-02-28T16:31:11.377 INFO:tasks.workunit.client.0.smithi001.stdout:5/528: mknod dc/d1a/d21/d2d/d91/cb1 0 2024-02-28T16:31:11.380 INFO:tasks.workunit.client.0.smithi001.stdout:0/552: dwrite d18/d31/f43 [0,4194304] 0 2024-02-28T16:31:11.380 INFO:tasks.workunit.client.0.smithi001.stdout:9/649: mknod d5/d9a/dba/cdc 0 2024-02-28T16:31:11.385 INFO:tasks.workunit.client.0.smithi001.stdout:7/551: getdents d3/d17/d3d/d42/d8e 0 2024-02-28T16:31:11.386 INFO:tasks.workunit.client.0.smithi001.stdout:8/554: getdents d4/d7/d22/d27/d8e 0 2024-02-28T16:31:11.387 INFO:tasks.workunit.client.0.smithi001.stdout:5/529: mkdir dc/d1a/d7d/d87/db2 0 2024-02-28T16:31:11.393 INFO:tasks.workunit.client.1.smithi066.stdout:3/117: truncate db/d13/d1a/f1e 119131 0 2024-02-28T16:31:11.393 INFO:tasks.workunit.client.1.smithi066.stdout:4/175: creat d1/d23/f36 x:0 0 0 2024-02-28T16:31:11.393 INFO:tasks.workunit.client.1.smithi066.stdout:6/56: symlink d2/lb 0 2024-02-28T16:31:11.394 INFO:tasks.workunit.client.0.smithi001.stdout:0/553: symlink d18/d22/d2b/d36/d46/lb6 0 2024-02-28T16:31:11.394 INFO:tasks.workunit.client.0.smithi001.stdout:5/530: rename dc/d12/d41/d54 to dc/d12/d41/d54/db3 22 2024-02-28T16:31:11.394 INFO:tasks.workunit.client.0.smithi001.stdout:4/582: dread d3/d5/d7/d1f/d3a/fa4 [0,4194304] 0 2024-02-28T16:31:11.395 INFO:tasks.workunit.client.0.smithi001.stdout:9/650: link d5/d27/d34/f8d d5/d45/d6f/db3/fdd 0 2024-02-28T16:31:11.395 INFO:tasks.workunit.client.0.smithi001.stdout:8/555: chown d4/d7/d22/d27/d58/f87 9 1 2024-02-28T16:31:11.399 INFO:tasks.workunit.client.0.smithi001.stdout:8/556: write d4/d7/d22/d27/f2f [2828525,65264] 0 2024-02-28T16:31:11.399 INFO:tasks.workunit.client.0.smithi001.stdout:4/583: chown d3/d1c/da0/fb2 36696860 1 2024-02-28T16:31:11.400 INFO:tasks.workunit.client.1.smithi066.stdout:1/164: dread d2/fc [0,4194304] 0 2024-02-28T16:31:11.400 INFO:tasks.workunit.client.1.smithi066.stdout:8/81: write d0/f4 [1012410,58608] 0 2024-02-28T16:31:11.401 INFO:tasks.workunit.client.1.smithi066.stdout:2/122: symlink d0/d12/d1d/d21/l25 0 2024-02-28T16:31:11.404 INFO:tasks.workunit.client.0.smithi001.stdout:9/651: creat d5/d45/d6f/db3/fde x:0 0 0 2024-02-28T16:31:11.404 INFO:tasks.workunit.client.1.smithi066.stdout:7/68: link l0 dc/l14 0 2024-02-28T16:31:11.405 INFO:tasks.workunit.client.0.smithi001.stdout:5/531: dread dc/d1a/d21/f37 [0,4194304] 0 2024-02-28T16:31:11.405 INFO:tasks.workunit.client.1.smithi066.stdout:7/69: stat c7 0 2024-02-28T16:31:11.406 INFO:tasks.workunit.client.1.smithi066.stdout:1/165: write d2/f17 [164015,43992] 0 2024-02-28T16:31:11.407 INFO:tasks.workunit.client.0.smithi001.stdout:8/557: rename d4/d7/d22/d27/d58/f95 to d4/d7/d22/d27/d58/d94/fb5 0 2024-02-28T16:31:11.408 INFO:tasks.workunit.client.1.smithi066.stdout:3/118: mkdir db/d13/d1a/d25 0 2024-02-28T16:31:11.409 INFO:tasks.workunit.client.0.smithi001.stdout:5/532: write dc/d12/d41/f4c [3291180,76720] 0 2024-02-28T16:31:11.409 INFO:tasks.workunit.client.1.smithi066.stdout:7/70: dread fa [0,4194304] 0 2024-02-28T16:31:11.411 INFO:tasks.workunit.client.1.smithi066.stdout:8/82: mkdir d0/db/d15 0 2024-02-28T16:31:11.412 INFO:tasks.workunit.client.1.smithi066.stdout:6/57: rename d2/f9 to d2/da/fc 0 2024-02-28T16:31:11.413 INFO:tasks.workunit.client.0.smithi001.stdout:5/533: write dc/df/f13 [4239795,12685] 0 2024-02-28T16:31:11.413 INFO:tasks.workunit.client.0.smithi001.stdout:4/584: getdents d3/d1c 0 2024-02-28T16:31:11.413 INFO:tasks.workunit.client.1.smithi066.stdout:0/212: dwrite d2/d9/f11 [0,4194304] 0 2024-02-28T16:31:11.415 INFO:tasks.workunit.client.0.smithi001.stdout:8/558: mknod d4/d32/d69/d9a/cb6 0 2024-02-28T16:31:11.418 INFO:tasks.workunit.client.1.smithi066.stdout:3/119: truncate db/d13/f19 678001 0 2024-02-28T16:31:11.420 INFO:tasks.workunit.client.1.smithi066.stdout:1/166: mknod d2/d18/d19/c35 0 2024-02-28T16:31:11.421 INFO:tasks.workunit.client.1.smithi066.stdout:7/71: mknod dc/c15 0 2024-02-28T16:31:11.423 INFO:tasks.workunit.client.1.smithi066.stdout:8/83: mkdir d0/d6/dc/d16 0 2024-02-28T16:31:11.423 INFO:tasks.workunit.client.0.smithi001.stdout:4/585: rename d3/d2d/d4c/d69/d82/f7c to d3/d5/d92/fbc 0 2024-02-28T16:31:11.424 INFO:tasks.workunit.client.0.smithi001.stdout:8/559: symlink d4/lb7 0 2024-02-28T16:31:11.428 INFO:tasks.workunit.client.1.smithi066.stdout:6/58: rename d2/f3 to d2/d4/fd 0 2024-02-28T16:31:11.428 INFO:tasks.workunit.client.1.smithi066.stdout:3/120: creat db/dc/d11/f26 x:0 0 0 2024-02-28T16:31:11.429 INFO:tasks.workunit.client.0.smithi001.stdout:4/586: chown d3/d5/d47/l8c 243 1 2024-02-28T16:31:11.430 INFO:tasks.workunit.client.1.smithi066.stdout:0/213: chown d2/d9/d10/l38 386 1 2024-02-28T16:31:11.431 INFO:tasks.workunit.client.1.smithi066.stdout:7/72: creat dc/f16 x:0 0 0 2024-02-28T16:31:11.432 INFO:tasks.workunit.client.0.smithi001.stdout:5/534: dwrite dc/d12/d41/d54/fa5 [0,4194304] 0 2024-02-28T16:31:11.432 INFO:tasks.workunit.client.1.smithi066.stdout:3/121: dread db/d13/d1a/f1e [0,4194304] 0 2024-02-28T16:31:11.438 INFO:tasks.workunit.client.0.smithi001.stdout:4/587: mknod d3/d5/cbd 0 2024-02-28T16:31:11.440 INFO:tasks.workunit.client.1.smithi066.stdout:4/176: getdents d1/d23 0 2024-02-28T16:31:11.441 INFO:tasks.workunit.client.1.smithi066.stdout:4/177: truncate d1/d18/f2b 158945 0 2024-02-28T16:31:11.441 INFO:tasks.workunit.client.1.smithi066.stdout:8/84: mknod d0/d6/dc/c17 0 2024-02-28T16:31:11.441 INFO:tasks.workunit.client.1.smithi066.stdout:1/167: symlink d2/l36 0 2024-02-28T16:31:11.442 INFO:tasks.workunit.client.1.smithi066.stdout:6/59: creat d2/da/fe x:0 0 0 2024-02-28T16:31:11.442 INFO:tasks.workunit.client.0.smithi001.stdout:8/560: dwrite d4/d32/f46 [4194304,4194304] 0 2024-02-28T16:31:11.444 INFO:tasks.workunit.client.1.smithi066.stdout:7/73: write dc/de/f10 [95078,57336] 0 2024-02-28T16:31:11.446 INFO:tasks.workunit.client.0.smithi001.stdout:4/588: truncate d3/d5/d7/d1f/d20/f95 999941 0 2024-02-28T16:31:11.447 INFO:tasks.workunit.client.1.smithi066.stdout:6/60: readlink d2/lb 0 2024-02-28T16:31:11.447 INFO:tasks.workunit.client.0.smithi001.stdout:4/589: dread - d3/f8e zero size 2024-02-28T16:31:11.449 INFO:tasks.workunit.client.1.smithi066.stdout:8/85: write d0/db/f10 [1582586,34350] 0 2024-02-28T16:31:11.450 INFO:tasks.workunit.client.0.smithi001.stdout:4/590: chown d3/d5/d7/d1f/d20/f95 107029333 1 2024-02-28T16:31:11.450 INFO:tasks.workunit.client.0.smithi001.stdout:9/652: truncate d5/d45/d6f/dc8/faf 1681617 0 2024-02-28T16:31:11.450 INFO:tasks.workunit.client.1.smithi066.stdout:4/178: creat d1/d18/f37 x:0 0 0 2024-02-28T16:31:11.453 INFO:tasks.workunit.client.1.smithi066.stdout:7/74: dread f5 [0,4194304] 0 2024-02-28T16:31:11.455 INFO:tasks.workunit.client.1.smithi066.stdout:5/104: sync 2024-02-28T16:31:11.455 INFO:tasks.workunit.client.1.smithi066.stdout:9/112: sync 2024-02-28T16:31:11.465 INFO:tasks.workunit.client.1.smithi066.stdout:8/86: chown d0/d6/dc/c17 60544 1 2024-02-28T16:31:11.465 INFO:tasks.workunit.client.1.smithi066.stdout:4/179: creat d1/d18/f38 x:0 0 0 2024-02-28T16:31:11.466 INFO:tasks.workunit.client.1.smithi066.stdout:6/61: fsync d2/da/fc 0 2024-02-28T16:31:11.467 INFO:tasks.workunit.client.1.smithi066.stdout:0/214: dwrite d2/d9/d10/f39 [0,4194304] 0 2024-02-28T16:31:11.468 INFO:tasks.workunit.client.0.smithi001.stdout:4/591: dread d3/d5/d7/d1f/d3a/f70 [0,4194304] 0 2024-02-28T16:31:11.469 INFO:tasks.workunit.client.0.smithi001.stdout:9/653: dwrite d5/d45/d6f/f87 [0,4194304] 0 2024-02-28T16:31:11.473 INFO:tasks.workunit.client.1.smithi066.stdout:2/123: dwrite d0/f9 [0,4194304] 0 2024-02-28T16:31:11.476 INFO:tasks.workunit.client.0.smithi001.stdout:4/592: write d3/d2d/d4c/d69/d82/f77 [445782,6449] 0 2024-02-28T16:31:11.477 INFO:tasks.workunit.client.1.smithi066.stdout:2/124: write d0/ff [2023018,85093] 0 2024-02-28T16:31:11.478 INFO:tasks.workunit.client.0.smithi001.stdout:8/561: dwrite d4/d7/d22/f3f [0,4194304] 0 2024-02-28T16:31:11.479 INFO:tasks.workunit.client.0.smithi001.stdout:9/654: truncate d5/d45/f71 781749 0 2024-02-28T16:31:11.479 INFO:tasks.workunit.client.1.smithi066.stdout:6/62: symlink d2/d4/lf 0 2024-02-28T16:31:11.480 INFO:tasks.workunit.client.0.smithi001.stdout:4/593: chown d3/d5/d4e/f80 51946923 1 2024-02-28T16:31:11.483 INFO:tasks.workunit.client.0.smithi001.stdout:8/562: chown d4/d7/d22/d27/f60 744762 1 2024-02-28T16:31:11.484 INFO:tasks.workunit.client.1.smithi066.stdout:4/180: chown d1/d18/c34 17139708 1 2024-02-28T16:31:11.485 INFO:tasks.workunit.client.1.smithi066.stdout:4/181: readlink d1/l8 0 2024-02-28T16:31:11.486 INFO:tasks.workunit.client.1.smithi066.stdout:4/182: readlink d1/l1a 0 2024-02-28T16:31:11.487 INFO:tasks.workunit.client.1.smithi066.stdout:0/215: creat d2/d9/d10/f46 x:0 0 0 2024-02-28T16:31:11.489 INFO:tasks.workunit.client.1.smithi066.stdout:7/75: dread f4 [0,4194304] 0 2024-02-28T16:31:11.489 INFO:tasks.workunit.client.0.smithi001.stdout:8/563: symlink d4/d7/d22/d27/d58/d62/lb8 0 2024-02-28T16:31:11.498 INFO:tasks.workunit.client.1.smithi066.stdout:9/113: getdents da/d10 0 2024-02-28T16:31:11.499 INFO:tasks.workunit.client.0.smithi001.stdout:9/655: dwrite d5/f1e [4194304,4194304] 0 2024-02-28T16:31:11.500 INFO:tasks.workunit.client.1.smithi066.stdout:2/125: creat d0/d12/d1a/f26 x:0 0 0 2024-02-28T16:31:11.505 INFO:tasks.workunit.client.0.smithi001.stdout:8/564: write d4/d32/f50 [6877370,68187] 0 2024-02-28T16:31:11.505 INFO:tasks.workunit.client.1.smithi066.stdout:6/63: rename d2/d4/fd to d2/f10 0 2024-02-28T16:31:11.505 INFO:tasks.workunit.client.1.smithi066.stdout:8/87: link d0/d6/dc/f14 d0/db/d15/f18 0 2024-02-28T16:31:11.505 INFO:tasks.workunit.client.1.smithi066.stdout:8/88: readlink - no filename 2024-02-28T16:31:11.506 INFO:tasks.workunit.client.1.smithi066.stdout:4/183: creat d1/d23/f39 x:0 0 0 2024-02-28T16:31:11.507 INFO:tasks.workunit.client.0.smithi001.stdout:9/656: truncate d5/d27/fb9 352014 0 2024-02-28T16:31:11.508 INFO:tasks.workunit.client.1.smithi066.stdout:4/184: dread - d1/d23/d25/f28 zero size 2024-02-28T16:31:11.510 INFO:tasks.workunit.client.1.smithi066.stdout:7/76: mkdir dc/de/d17 0 2024-02-28T16:31:11.511 INFO:tasks.workunit.client.0.smithi001.stdout:8/565: mknod d4/d47/cb9 0 2024-02-28T16:31:11.515 INFO:tasks.workunit.client.0.smithi001.stdout:9/657: unlink d5/f18 0 2024-02-28T16:31:11.517 INFO:tasks.workunit.client.1.smithi066.stdout:0/216: rename d2/d9/f21 to d2/d9/d10/f47 0 2024-02-28T16:31:11.518 INFO:tasks.workunit.client.1.smithi066.stdout:6/64: write f1 [6356325,116000] 0 2024-02-28T16:31:11.519 INFO:tasks.workunit.client.0.smithi001.stdout:9/658: write d5/d27/db1/d33/f3d [3085474,23323] 0 2024-02-28T16:31:11.521 INFO:tasks.workunit.client.1.smithi066.stdout:8/89: unlink d0/db/d15/f18 0 2024-02-28T16:31:11.522 INFO:tasks.workunit.client.1.smithi066.stdout:1/168: truncate d2/f10 692255 0 2024-02-28T16:31:11.522 INFO:tasks.workunit.client.1.smithi066.stdout:4/185: creat d1/d18/f3a x:0 0 0 2024-02-28T16:31:11.523 INFO:tasks.workunit.client.0.smithi001.stdout:9/659: mkdir d5/d27/db1/d33/d8a/da8/ddf 0 2024-02-28T16:31:11.524 INFO:tasks.workunit.client.1.smithi066.stdout:9/114: mkdir da/d21 0 2024-02-28T16:31:11.524 INFO:tasks.workunit.client.0.smithi001.stdout:9/660: fdatasync d5/d27/db1/fa4 0 2024-02-28T16:31:11.531 INFO:tasks.workunit.client.1.smithi066.stdout:2/126: dwrite d0/f1f [0,4194304] 0 2024-02-28T16:31:11.539 INFO:tasks.workunit.client.1.smithi066.stdout:3/122: dwrite db/ff [0,4194304] 0 2024-02-28T16:31:11.540 INFO:tasks.workunit.client.0.smithi001.stdout:9/661: dwrite d5/d45/d6f/db3/fde [0,4194304] 0 2024-02-28T16:31:11.541 INFO:tasks.workunit.client.1.smithi066.stdout:1/169: rmdir d2 39 2024-02-28T16:31:11.543 INFO:tasks.workunit.client.0.smithi001.stdout:9/662: write d5/d27/d42/d48/fab [940205,118983] 0 2024-02-28T16:31:11.545 INFO:tasks.workunit.client.0.smithi001.stdout:2/570: sync 2024-02-28T16:31:11.545 INFO:tasks.workunit.client.1.smithi066.stdout:9/115: mkdir da/d22 0 2024-02-28T16:31:11.549 INFO:tasks.workunit.client.1.smithi066.stdout:2/127: mknod d0/d12/d1a/c27 0 2024-02-28T16:31:11.551 INFO:tasks.workunit.client.0.smithi001.stdout:2/571: symlink d0/d2/d3/de/d22/lc1 0 2024-02-28T16:31:11.552 INFO:tasks.workunit.client.0.smithi001.stdout:9/663: mkdir d5/d27/db1/d33/d8a/de0 0 2024-02-28T16:31:11.555 INFO:tasks.workunit.client.0.smithi001.stdout:8/566: dread d4/d7/da8/d89/f79 [0,4194304] 0 2024-02-28T16:31:11.556 INFO:tasks.workunit.client.0.smithi001.stdout:9/664: truncate d5/f1d 6351822 0 2024-02-28T16:31:11.560 INFO:tasks.workunit.client.1.smithi066.stdout:2/128: stat d0/f1f 0 2024-02-28T16:31:11.561 INFO:tasks.workunit.client.0.smithi001.stdout:9/665: rename d5/d27/d34/l78 to d5/d27/d42/d48/le1 0 2024-02-28T16:31:11.562 INFO:tasks.workunit.client.1.smithi066.stdout:6/65: truncate f1 5043458 0 2024-02-28T16:31:11.562 INFO:tasks.workunit.client.0.smithi001.stdout:9/666: chown d5/d4d 102432 1 2024-02-28T16:31:11.563 INFO:tasks.workunit.client.1.smithi066.stdout:3/123: dread db/d13/d1a/f1b [0,4194304] 0 2024-02-28T16:31:11.565 INFO:tasks.workunit.client.0.smithi001.stdout:9/667: creat d5/d45/d6f/dc8/fe2 x:0 0 0 2024-02-28T16:31:11.565 INFO:tasks.workunit.client.1.smithi066.stdout:1/170: mkdir d2/d27/d2b/d37 0 2024-02-28T16:31:11.566 INFO:tasks.workunit.client.1.smithi066.stdout:6/66: dread - d2/da/fe zero size 2024-02-28T16:31:11.567 INFO:tasks.workunit.client.1.smithi066.stdout:8/90: dwrite d0/d6/f9 [0,4194304] 0 2024-02-28T16:31:11.569 INFO:tasks.workunit.client.1.smithi066.stdout:2/129: creat d0/d12/d1a/d23/f28 x:0 0 0 2024-02-28T16:31:11.569 INFO:tasks.workunit.client.1.smithi066.stdout:2/130: write d0/f8 [286959,13264] 0 2024-02-28T16:31:11.570 INFO:tasks.workunit.client.0.smithi001.stdout:9/668: creat d5/d27/db1/d5e/dd3/fe3 x:0 0 0 2024-02-28T16:31:11.570 INFO:tasks.workunit.client.1.smithi066.stdout:7/77: dread f1 [0,4194304] 0 2024-02-28T16:31:11.573 INFO:tasks.workunit.client.0.smithi001.stdout:9/669: write d5/f89 [447692,72588] 0 2024-02-28T16:31:11.576 INFO:tasks.workunit.client.0.smithi001.stdout:2/572: dwrite d0/d2/d3/de/d61/d12/d6a/f7e [0,4194304] 0 2024-02-28T16:31:11.577 INFO:tasks.workunit.client.0.smithi001.stdout:8/567: dwrite d4/d7/d22/d27/d8e/dae/faf [0,4194304] 0 2024-02-28T16:31:11.578 INFO:tasks.workunit.client.1.smithi066.stdout:9/116: dread da/fb [4194304,4194304] 0 2024-02-28T16:31:11.578 INFO:tasks.workunit.client.1.smithi066.stdout:1/171: mknod d2/d18/d19/c38 0 2024-02-28T16:31:11.580 INFO:tasks.workunit.client.1.smithi066.stdout:9/117: write da/fc [4506951,91904] 0 2024-02-28T16:31:11.581 INFO:tasks.workunit.client.0.smithi001.stdout:9/670: mkdir d5/d27/db1/da6/db2/de4 0 2024-02-28T16:31:11.584 INFO:tasks.workunit.client.1.smithi066.stdout:3/124: creat db/d13/d1a/f27 x:0 0 0 2024-02-28T16:31:11.585 INFO:tasks.workunit.client.0.smithi001.stdout:8/568: dread d4/d7/d22/d27/d4b/f63 [0,4194304] 0 2024-02-28T16:31:11.585 INFO:tasks.workunit.client.1.smithi066.stdout:3/125: chown db/d13/d14/d24 10 1 2024-02-28T16:31:11.589 INFO:tasks.workunit.client.0.smithi001.stdout:2/573: creat d0/d2/d3/de/fc2 x:0 0 0 2024-02-28T16:31:11.593 INFO:tasks.workunit.client.0.smithi001.stdout:8/569: link d4/d7/f23 d4/d47/d8b/fba 0 2024-02-28T16:31:11.603 INFO:tasks.workunit.client.0.smithi001.stdout:9/671: dwrite d5/d45/d6f/db3/fde [4194304,4194304] 0 2024-02-28T16:31:11.603 INFO:tasks.workunit.client.0.smithi001.stdout:8/570: link d4/d47/d8b/fba d4/d32/d93/fbb 0 2024-02-28T16:31:11.610 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:11 smithi001 ceph-mon[28991]: pgmap v49: 65 pgs: 65 active+clean; 1.4 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 44 MiB/s rd, 184 MiB/s wr, 394 op/s 2024-02-28T16:31:11.610 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:11 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:11.611 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:11 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.rmthtx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:31:11.611 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:11 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.rmthtx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:31:11.611 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:11 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:31:11.611 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:11 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:31:11.621 INFO:tasks.workunit.client.0.smithi001.stdout:8/571: dread d4/d47/d8b/f8c [4194304,4194304] 0 2024-02-28T16:31:11.622 INFO:tasks.workunit.client.1.smithi066.stdout:7/78: truncate f5 1102928 0 2024-02-28T16:31:11.622 INFO:tasks.workunit.client.1.smithi066.stdout:6/67: rename d2/c5 to d2/c11 0 2024-02-28T16:31:11.622 INFO:tasks.workunit.client.1.smithi066.stdout:2/131: rename d0/d12/d1d to d0/d12/d1a/d23/d29 0 2024-02-28T16:31:11.623 INFO:tasks.workunit.client.0.smithi001.stdout:9/672: dwrite d5/d27/db1/f93 [0,4194304] 0 2024-02-28T16:31:11.623 INFO:tasks.workunit.client.1.smithi066.stdout:3/126: fdatasync db/d13/f23 0 2024-02-28T16:31:11.623 INFO:tasks.workunit.client.1.smithi066.stdout:9/118: symlink da/d10/d15/l23 0 2024-02-28T16:31:11.624 INFO:tasks.workunit.client.1.smithi066.stdout:6/68: chown d2/d4 34 1 2024-02-28T16:31:11.625 INFO:tasks.workunit.client.1.smithi066.stdout:7/79: symlink dc/de/l18 0 2024-02-28T16:31:11.629 INFO:tasks.workunit.client.1.smithi066.stdout:9/119: symlink da/d22/l24 0 2024-02-28T16:31:11.630 INFO:tasks.workunit.client.1.smithi066.stdout:7/80: write dc/de/f13 [815604,12298] 0 2024-02-28T16:31:11.631 INFO:tasks.workunit.client.1.smithi066.stdout:2/132: write d0/f1e [4730101,122962] 0 2024-02-28T16:31:11.633 INFO:tasks.workunit.client.0.smithi001.stdout:9/673: mknod d5/d27/db1/d58/ce5 0 2024-02-28T16:31:11.635 INFO:tasks.workunit.client.1.smithi066.stdout:9/120: symlink da/df/l25 0 2024-02-28T16:31:11.635 INFO:tasks.workunit.client.1.smithi066.stdout:2/133: symlink d0/l2a 0 2024-02-28T16:31:11.635 INFO:tasks.workunit.client.1.smithi066.stdout:7/81: creat dc/de/d17/f19 x:0 0 0 2024-02-28T16:31:11.638 INFO:tasks.workunit.client.0.smithi001.stdout:2/574: dread d0/d2/d3/de/d61/d12/d6a/d84/fbc [0,4194304] 0 2024-02-28T16:31:11.639 INFO:tasks.workunit.client.1.smithi066.stdout:2/134: creat d0/d12/d1a/f2b x:0 0 0 2024-02-28T16:31:11.642 INFO:tasks.workunit.client.1.smithi066.stdout:6/69: dread d2/da/fc [0,4194304] 0 2024-02-28T16:31:11.650 INFO:tasks.workunit.client.1.smithi066.stdout:9/121: dread f3 [0,4194304] 0 2024-02-28T16:31:11.650 INFO:tasks.workunit.client.0.smithi001.stdout:2/575: link d0/d2/d3/de/d8a/f2e d0/fc3 0 2024-02-28T16:31:11.650 INFO:tasks.workunit.client.0.smithi001.stdout:8/572: dread d4/d47/f64 [0,4194304] 0 2024-02-28T16:31:11.655 INFO:tasks.workunit.client.0.smithi001.stdout:2/576: creat d0/d2/da6/fc4 x:0 0 0 2024-02-28T16:31:11.656 INFO:tasks.workunit.client.1.smithi066.stdout:7/82: truncate dc/de/f13 301050 0 2024-02-28T16:31:11.656 INFO:tasks.workunit.client.0.smithi001.stdout:9/674: dwrite d5/d27/db1/fa3 [0,4194304] 0 2024-02-28T16:31:11.658 INFO:tasks.workunit.client.0.smithi001.stdout:8/573: truncate d4/f11 2890481 0 2024-02-28T16:31:11.658 INFO:tasks.workunit.client.1.smithi066.stdout:2/135: dwrite d0/d12/f13 [0,4194304] 0 2024-02-28T16:31:11.658 INFO:tasks.workunit.client.1.smithi066.stdout:3/127: dwrite db/d13/d1a/f1e [0,4194304] 0 2024-02-28T16:31:11.663 INFO:tasks.workunit.client.0.smithi001.stdout:9/675: truncate d5/d45/d6f/f7f 1006591 0 2024-02-28T16:31:11.665 INFO:tasks.workunit.client.0.smithi001.stdout:2/577: creat d0/d2/d3/fc5 x:0 0 0 2024-02-28T16:31:11.666 INFO:tasks.workunit.client.1.smithi066.stdout:2/136: mknod d0/d12/d1a/c2c 0 2024-02-28T16:31:11.670 INFO:tasks.workunit.client.1.smithi066.stdout:3/128: symlink db/d13/d14/d24/l28 0 2024-02-28T16:31:11.671 INFO:tasks.workunit.client.0.smithi001.stdout:2/578: creat d0/d2/d3/de/d8a/fc6 x:0 0 0 2024-02-28T16:31:11.675 INFO:tasks.workunit.client.1.smithi066.stdout:2/137: mknod d0/d12/d1a/c2d 0 2024-02-28T16:31:11.675 INFO:tasks.workunit.client.1.smithi066.stdout:7/83: dread f1 [0,4194304] 0 2024-02-28T16:31:11.678 INFO:tasks.workunit.client.1.smithi066.stdout:3/129: rmdir db/d13/d14/d24 39 2024-02-28T16:31:11.680 INFO:tasks.workunit.client.1.smithi066.stdout:3/130: truncate db/d13/f23 817651 0 2024-02-28T16:31:11.681 INFO:tasks.workunit.client.1.smithi066.stdout:2/138: getdents d0/d12/d1a/d23/d29/d21 0 2024-02-28T16:31:11.681 INFO:tasks.workunit.client.0.smithi001.stdout:8/574: dwrite d4/d7/d22/d27/d4b/d77/f7a [0,4194304] 0 2024-02-28T16:31:11.690 INFO:tasks.workunit.client.0.smithi001.stdout:2/579: dwrite d0/d2/d3/de/d61/d12/f41 [0,4194304] 0 2024-02-28T16:31:11.690 INFO:tasks.workunit.client.1.smithi066.stdout:3/131: dread db/ff [0,4194304] 0 2024-02-28T16:31:11.698 INFO:tasks.workunit.client.0.smithi001.stdout:9/676: truncate d5/d45/d6f/f87 545794 0 2024-02-28T16:31:11.700 INFO:tasks.workunit.client.1.smithi066.stdout:3/132: rmdir db 39 2024-02-28T16:31:11.701 INFO:tasks.workunit.client.1.smithi066.stdout:2/139: rmdir d0/d12/d1a 39 2024-02-28T16:31:11.708 INFO:tasks.workunit.client.1.smithi066.stdout:3/133: write db/d13/d1a/f27 [585098,78017] 0 2024-02-28T16:31:11.708 INFO:tasks.workunit.client.1.smithi066.stdout:2/140: mkdir d0/d12/d2e 0 2024-02-28T16:31:11.709 INFO:tasks.workunit.client.0.smithi001.stdout:8/575: dwrite d4/d7/d22/d27/d4b/f6d [0,4194304] 0 2024-02-28T16:31:11.710 INFO:tasks.workunit.client.1.smithi066.stdout:3/134: rmdir db/d13/d14 39 2024-02-28T16:31:11.713 INFO:tasks.workunit.client.0.smithi001.stdout:8/576: read - d4/d7/d22/d27/d58/d94/fb5 zero size 2024-02-28T16:31:11.713 INFO:tasks.workunit.client.0.smithi001.stdout:9/677: dwrite d5/d27/db1/f40 [4194304,4194304] 0 2024-02-28T16:31:11.714 INFO:tasks.workunit.client.1.smithi066.stdout:2/141: mkdir d0/d12/d1a/d23/d29/d21/d2f 0 2024-02-28T16:31:11.723 INFO:tasks.workunit.client.0.smithi001.stdout:3/550: sync 2024-02-28T16:31:11.726 INFO:tasks.workunit.client.0.smithi001.stdout:9/678: stat d5/d27/d34/f90 0 2024-02-28T16:31:11.727 INFO:tasks.workunit.client.0.smithi001.stdout:8/577: unlink d4/d7/da8/d89/f79 0 2024-02-28T16:31:11.728 INFO:tasks.workunit.client.1.smithi066.stdout:3/135: dwrite db/d13/f18 [0,4194304] 0 2024-02-28T16:31:11.728 INFO:tasks.workunit.client.0.smithi001.stdout:3/551: symlink d4/d1e/d7c/da8/lbb 0 2024-02-28T16:31:11.733 INFO:tasks.workunit.client.1.smithi066.stdout:3/136: getdents db/dc/d11 0 2024-02-28T16:31:11.734 INFO:tasks.workunit.client.1.smithi066.stdout:2/142: getdents d0/d12/d1a/d23/d29/d21 0 2024-02-28T16:31:11.740 INFO:tasks.workunit.client.1.smithi066.stdout:2/143: unlink d0/d12/d1a/f26 0 2024-02-28T16:31:11.741 INFO:tasks.workunit.client.0.smithi001.stdout:8/578: getdents d4/d7 0 2024-02-28T16:31:11.742 INFO:tasks.workunit.client.1.smithi066.stdout:3/137: mknod db/d13/d1a/d25/c29 0 2024-02-28T16:31:11.743 INFO:tasks.workunit.client.1.smithi066.stdout:2/144: creat d0/f30 x:0 0 0 2024-02-28T16:31:11.744 INFO:tasks.workunit.client.1.smithi066.stdout:3/138: mkdir db/d2a 0 2024-02-28T16:31:11.745 INFO:tasks.workunit.client.1.smithi066.stdout:3/139: read f5 [3409349,71869] 0 2024-02-28T16:31:11.746 INFO:tasks.workunit.client.0.smithi001.stdout:8/579: mkdir d4/d7/d22/d27/d58/d94/dbc 0 2024-02-28T16:31:11.751 INFO:tasks.workunit.client.1.smithi066.stdout:2/145: dread d0/f8 [0,4194304] 0 2024-02-28T16:31:11.752 INFO:tasks.workunit.client.1.smithi066.stdout:2/146: fdatasync d0/f8 0 2024-02-28T16:31:11.754 INFO:tasks.workunit.client.1.smithi066.stdout:2/147: write d0/f1e [4889630,54739] 0 2024-02-28T16:31:11.755 INFO:tasks.workunit.client.1.smithi066.stdout:3/140: dread f5 [0,4194304] 0 2024-02-28T16:31:11.755 INFO:tasks.workunit.client.0.smithi001.stdout:8/580: creat d4/d7/d22/d27/d58/d94/dbc/fbd x:0 0 0 2024-02-28T16:31:11.759 INFO:tasks.workunit.client.1.smithi066.stdout:2/148: link d0/ff d0/f31 0 2024-02-28T16:31:11.760 INFO:tasks.workunit.client.1.smithi066.stdout:2/149: chown d0/f8 121 1 2024-02-28T16:31:11.761 INFO:tasks.workunit.client.0.smithi001.stdout:3/552: dwrite d4/d6/d7/f88 [0,4194304] 0 2024-02-28T16:31:11.764 INFO:tasks.workunit.client.0.smithi001.stdout:3/553: fdatasync d4/d6/d7/d23/fa9 0 2024-02-28T16:31:11.765 INFO:tasks.workunit.client.1.smithi066.stdout:2/150: rename d0/lc to d0/d12/d1a/d23/d29/d21/l32 0 2024-02-28T16:31:11.765 INFO:tasks.workunit.client.1.smithi066.stdout:3/141: dread db/ff [0,4194304] 0 2024-02-28T16:31:11.766 INFO:tasks.workunit.client.1.smithi066.stdout:3/142: readlink db/l12 0 2024-02-28T16:31:11.768 INFO:tasks.workunit.client.0.smithi001.stdout:3/554: creat d4/d38/fbc x:0 0 0 2024-02-28T16:31:11.773 INFO:tasks.workunit.client.0.smithi001.stdout:8/581: dwrite d4/d47/d8b/f8c [4194304,4194304] 0 2024-02-28T16:31:11.774 INFO:tasks.workunit.client.1.smithi066.stdout:2/151: rename d0/c10 to d0/d12/d1a/d23/d29/c33 0 2024-02-28T16:31:11.778 INFO:tasks.workunit.client.0.smithi001.stdout:8/582: chown d4/d47/c4e 50107368 1 2024-02-28T16:31:11.778 INFO:tasks.workunit.client.0.smithi001.stdout:3/555: write d4/d18/f49 [943939,68667] 0 2024-02-28T16:31:11.782 INFO:tasks.workunit.client.0.smithi001.stdout:3/556: creat d4/d97/fbd x:0 0 0 2024-02-28T16:31:11.782 INFO:tasks.workunit.client.0.smithi001.stdout:8/583: mkdir d4/d7/d22/d27/d8e/dae/dbe 0 2024-02-28T16:31:11.786 INFO:tasks.workunit.client.1.smithi066.stdout:2/152: truncate d0/d12/f13 2507531 0 2024-02-28T16:31:11.787 INFO:tasks.workunit.client.1.smithi066.stdout:2/153: readlink d0/d12/l1b 0 2024-02-28T16:31:11.790 INFO:tasks.workunit.client.1.smithi066.stdout:2/154: truncate d0/d12/d1a/f2b 451892 0 2024-02-28T16:31:11.791 INFO:tasks.workunit.client.0.smithi001.stdout:3/557: rmdir d4/d6/d7/d78/d81 39 2024-02-28T16:31:11.791 INFO:tasks.workunit.client.1.smithi066.stdout:2/155: chown d0/d12/d1a/d23 460968 1 2024-02-28T16:31:11.794 INFO:tasks.workunit.client.1.smithi066.stdout:3/143: dwrite db/d13/d1a/f1b [0,4194304] 0 2024-02-28T16:31:11.797 INFO:tasks.workunit.client.1.smithi066.stdout:2/156: mknod d0/d12/d1a/d23/d29/d21/c34 0 2024-02-28T16:31:11.797 INFO:tasks.workunit.client.1.smithi066.stdout:3/144: chown db/d13/d1a/d25/c29 1 1 2024-02-28T16:31:11.800 INFO:tasks.workunit.client.0.smithi001.stdout:8/584: dwrite d4/d32/f46 [4194304,4194304] 0 2024-02-28T16:31:11.802 INFO:tasks.workunit.client.1.smithi066.stdout:3/145: link f5 db/dc/f2b 0 2024-02-28T16:31:11.803 INFO:tasks.workunit.client.0.smithi001.stdout:8/585: chown d4/d7/d22/d27/f60 3 1 2024-02-28T16:31:11.805 INFO:tasks.workunit.client.1.smithi066.stdout:3/146: write db/d13/f18 [2659326,96635] 0 2024-02-28T16:31:11.808 INFO:tasks.workunit.client.1.smithi066.stdout:2/157: symlink d0/d12/d1a/d23/d29/d21/d2f/l35 0 2024-02-28T16:31:11.808 INFO:tasks.workunit.client.1.smithi066.stdout:3/147: read db/d13/d1a/f1e [2417806,58804] 0 2024-02-28T16:31:11.812 INFO:tasks.workunit.client.1.smithi066.stdout:3/148: symlink db/l2c 0 2024-02-28T16:31:11.815 INFO:tasks.workunit.client.0.smithi001.stdout:8/586: stat d4/d7/d22/d27/d58/f75 0 2024-02-28T16:31:11.816 INFO:tasks.workunit.client.1.smithi066.stdout:3/149: chown db/d13/d1a/f1b 332238763 1 2024-02-28T16:31:11.819 INFO:tasks.workunit.client.1.smithi066.stdout:2/158: unlink d0/d12/f13 0 2024-02-28T16:31:11.820 INFO:tasks.workunit.client.0.smithi001.stdout:3/558: dread d4/f44 [0,4194304] 0 2024-02-28T16:31:11.821 INFO:tasks.workunit.client.0.smithi001.stdout:3/559: chown d4/d6/d7/d78 207215615 1 2024-02-28T16:31:11.822 INFO:tasks.workunit.client.0.smithi001.stdout:3/560: readlink d4/d18/l2e 0 2024-02-28T16:31:11.822 INFO:tasks.workunit.client.1.smithi066.stdout:3/150: mkdir db/d13/d14/d2d 0 2024-02-28T16:31:11.823 INFO:tasks.workunit.client.0.smithi001.stdout:3/561: stat d4/d6/d7/d23/f2d 0 2024-02-28T16:31:11.825 INFO:tasks.workunit.client.1.smithi066.stdout:2/159: link d0/f9 d0/d12/f36 0 2024-02-28T16:31:11.826 INFO:tasks.workunit.client.1.smithi066.stdout:3/151: unlink db/dc/c1d 0 2024-02-28T16:31:11.830 INFO:tasks.workunit.client.1.smithi066.stdout:3/152: link db/ff db/dc/d11/d1f/f2e 0 2024-02-28T16:31:11.833 INFO:tasks.workunit.client.1.smithi066.stdout:3/153: mkdir db/d13/d14/d24/d2f 0 2024-02-28T16:31:11.835 INFO:tasks.workunit.client.0.smithi001.stdout:3/562: dwrite d4/d6/d7f/d82/fb0 [0,4194304] 0 2024-02-28T16:31:11.836 INFO:tasks.workunit.client.0.smithi001.stdout:3/563: chown d4/d1e/d7c 9493 1 2024-02-28T16:31:11.837 INFO:tasks.workunit.client.0.smithi001.stdout:3/564: fdatasync d4/d6/d7/f20 0 2024-02-28T16:31:11.839 INFO:tasks.workunit.client.0.smithi001.stdout:3/565: fsync d4/d6/fc 0 2024-02-28T16:31:11.842 INFO:tasks.workunit.client.1.smithi066.stdout:3/154: dread db/d13/f18 [0,4194304] 0 2024-02-28T16:31:11.844 INFO:tasks.workunit.client.1.smithi066.stdout:3/155: rename l4 to db/d13/d14/d2d/l30 0 2024-02-28T16:31:11.848 INFO:tasks.workunit.client.1.smithi066.stdout:3/156: write db/ff [2477031,12646] 0 2024-02-28T16:31:11.848 INFO:tasks.workunit.client.1.smithi066.stdout:3/157: readlink l6 0 2024-02-28T16:31:11.850 INFO:tasks.workunit.client.1.smithi066.stdout:3/158: symlink db/d2a/l31 0 2024-02-28T16:31:11.852 INFO:tasks.workunit.client.1.smithi066.stdout:4/186: sync 2024-02-28T16:31:11.852 INFO:tasks.workunit.client.1.smithi066.stdout:5/105: sync 2024-02-28T16:31:11.852 INFO:tasks.workunit.client.1.smithi066.stdout:0/217: sync 2024-02-28T16:31:11.855 INFO:tasks.workunit.client.1.smithi066.stdout:4/187: mknod d1/d23/d25/c3b 0 2024-02-28T16:31:11.857 INFO:tasks.workunit.client.1.smithi066.stdout:4/188: dread - d1/d18/f37 zero size 2024-02-28T16:31:11.858 INFO:tasks.workunit.client.1.smithi066.stdout:0/218: symlink d2/d27/d41/l48 0 2024-02-28T16:31:11.859 INFO:tasks.workunit.client.1.smithi066.stdout:5/106: mkdir d3/d4/d17/d18/d24 0 2024-02-28T16:31:11.862 INFO:tasks.workunit.client.1.smithi066.stdout:5/107: write d3/d4/dc/f21 [894031,79559] 0 2024-02-28T16:31:11.863 INFO:tasks.workunit.client.1.smithi066.stdout:0/219: mknod d2/c49 0 2024-02-28T16:31:11.864 INFO:tasks.workunit.client.1.smithi066.stdout:5/108: read - d3/d4/dc/f1e zero size 2024-02-28T16:31:11.866 INFO:tasks.workunit.client.1.smithi066.stdout:4/189: dread d1/f33 [0,4194304] 0 2024-02-28T16:31:11.867 INFO:tasks.workunit.client.1.smithi066.stdout:4/190: chown d1/f11 213273990 1 2024-02-28T16:31:11.869 INFO:tasks.workunit.client.1.smithi066.stdout:4/191: mknod d1/c3c 0 2024-02-28T16:31:11.871 INFO:tasks.workunit.client.1.smithi066.stdout:4/192: write d1/f7 [104014,62939] 0 2024-02-28T16:31:11.872 INFO:tasks.workunit.client.1.smithi066.stdout:0/220: dread d2/f22 [0,4194304] 0 2024-02-28T16:31:11.873 INFO:tasks.workunit.client.0.smithi001.stdout:3/566: dread d4/d6/d7/f63 [0,4194304] 0 2024-02-28T16:31:11.875 INFO:tasks.workunit.client.1.smithi066.stdout:0/221: truncate d2/d9/d10/f2e 5858227 0 2024-02-28T16:31:11.878 INFO:tasks.workunit.client.0.smithi001.stdout:3/567: rename d4/c31 to d4/d1e/d7c/d9a/cbe 0 2024-02-28T16:31:11.881 INFO:tasks.workunit.client.1.smithi066.stdout:0/222: rename d2/c6 to d2/d9/d33/d3f/c4a 0 2024-02-28T16:31:11.882 INFO:tasks.workunit.client.1.smithi066.stdout:5/109: dwrite d3/d4/f9 [0,4194304] 0 2024-02-28T16:31:11.884 INFO:tasks.workunit.client.0.smithi001.stdout:3/568: write d4/d38/f41 [3178840,44957] 0 2024-02-28T16:31:11.887 INFO:tasks.workunit.client.1.smithi066.stdout:0/223: unlink d2/d9/d33/l3c 0 2024-02-28T16:31:11.888 INFO:tasks.workunit.client.1.smithi066.stdout:5/110: mknod d3/d4/dc/c25 0 2024-02-28T16:31:11.889 INFO:tasks.workunit.client.1.smithi066.stdout:0/224: truncate d2/f22 1661445 0 2024-02-28T16:31:11.890 INFO:tasks.workunit.client.0.smithi001.stdout:3/569: link d4/d18/d5f/f77 d4/d18/fbf 0 2024-02-28T16:31:11.891 INFO:tasks.workunit.client.1.smithi066.stdout:4/193: dwrite d1/f12 [4194304,4194304] 0 2024-02-28T16:31:11.891 INFO:tasks.workunit.client.1.smithi066.stdout:5/111: rename d3/d4/d17/d18/d24 to d3/d4/d17/d18/d24/d26 22 2024-02-28T16:31:11.892 INFO:tasks.workunit.client.0.smithi001.stdout:3/570: write d4/d18/f91 [3862391,60501] 0 2024-02-28T16:31:11.898 INFO:tasks.workunit.client.1.smithi066.stdout:4/194: creat d1/d23/f3d x:0 0 0 2024-02-28T16:31:11.903 INFO:tasks.workunit.client.1.smithi066.stdout:4/195: mkdir d1/d23/d25/d3e 0 2024-02-28T16:31:11.903 INFO:tasks.workunit.client.1.smithi066.stdout:0/225: link d2/l12 d2/d9/d33/d3f/l4b 0 2024-02-28T16:31:11.905 INFO:tasks.workunit.client.1.smithi066.stdout:4/196: write d1/f5 [3270813,14379] 0 2024-02-28T16:31:11.911 INFO:tasks.workunit.client.0.smithi001.stdout:3/571: dwrite d4/d6/d7/d78/d81/d8a/fa5 [0,4194304] 0 2024-02-28T16:31:11.912 INFO:tasks.workunit.client.0.smithi001.stdout:3/572: readlink d4/l16 0 2024-02-28T16:31:11.919 INFO:tasks.workunit.client.1.smithi066.stdout:5/112: dwrite d3/ff [4194304,4194304] 0 2024-02-28T16:31:11.928 INFO:tasks.workunit.client.1.smithi066.stdout:5/113: write d3/d4/f5 [20946,116723] 0 2024-02-28T16:31:11.928 INFO:tasks.workunit.client.0.smithi001.stdout:3/573: dwrite d4/d6/f40 [0,4194304] 0 2024-02-28T16:31:11.929 INFO:tasks.workunit.client.1.smithi066.stdout:4/197: dwrite d1/f2a [0,4194304] 0 2024-02-28T16:31:11.930 INFO:tasks.workunit.client.1.smithi066.stdout:0/226: truncate d2/d9/d10/f2e 4198376 0 2024-02-28T16:31:11.930 INFO:tasks.workunit.client.1.smithi066.stdout:5/114: stat d3/d4/d17 0 2024-02-28T16:31:11.931 INFO:tasks.workunit.client.0.smithi001.stdout:3/574: truncate d4/d38/f41 4357662 0 2024-02-28T16:31:11.933 INFO:tasks.workunit.client.1.smithi066.stdout:4/198: rmdir d1/d18 39 2024-02-28T16:31:11.933 INFO:tasks.workunit.client.1.smithi066.stdout:5/115: readlink d3/d4/dc/l23 0 2024-02-28T16:31:11.935 INFO:tasks.workunit.client.1.smithi066.stdout:4/199: write d1/d23/f36 [873333,92249] 0 2024-02-28T16:31:11.936 INFO:tasks.workunit.client.1.smithi066.stdout:0/227: symlink d2/d9/d33/l4c 0 2024-02-28T16:31:11.936 INFO:tasks.workunit.client.0.smithi001.stdout:1/662: sync 2024-02-28T16:31:11.936 INFO:tasks.workunit.client.0.smithi001.stdout:6/515: sync 2024-02-28T16:31:11.938 INFO:tasks.workunit.client.0.smithi001.stdout:3/575: getdents d4/d6/d7/d78/d81/d8a 0 2024-02-28T16:31:11.939 INFO:tasks.workunit.client.0.smithi001.stdout:6/516: readlink d9/db/d19/l22 0 2024-02-28T16:31:11.940 INFO:tasks.workunit.client.1.smithi066.stdout:4/200: dread - d1/d23/f39 zero size 2024-02-28T16:31:11.944 INFO:tasks.workunit.client.0.smithi001.stdout:3/576: write d4/d1e/d7c/da8/fae [83869,27762] 0 2024-02-28T16:31:11.948 INFO:tasks.workunit.client.0.smithi001.stdout:3/577: creat d4/d1e/d7c/d7d/fc0 x:0 0 0 2024-02-28T16:31:11.949 INFO:tasks.workunit.client.1.smithi066.stdout:0/228: symlink d2/d27/l4d 0 2024-02-28T16:31:11.951 INFO:tasks.workunit.client.1.smithi066.stdout:0/229: chown d2/d9/d33 16791 1 2024-02-28T16:31:11.952 INFO:tasks.workunit.client.0.smithi001.stdout:1/663: rename d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/fc0 to d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/db1/fe9 0 2024-02-28T16:31:11.953 INFO:tasks.workunit.client.1.smithi066.stdout:0/230: dread - d2/d9/f44 zero size 2024-02-28T16:31:11.954 INFO:tasks.workunit.client.0.smithi001.stdout:3/578: creat d4/d1e/db7/fc1 x:0 0 0 2024-02-28T16:31:11.955 INFO:tasks.workunit.client.1.smithi066.stdout:5/116: creat d3/d4/f27 x:0 0 0 2024-02-28T16:31:11.955 INFO:tasks.workunit.client.0.smithi001.stdout:3/579: readlink d4/d18/l48 0 2024-02-28T16:31:11.957 INFO:tasks.workunit.client.1.smithi066.stdout:5/117: dread - d3/d4/f27 zero size 2024-02-28T16:31:11.958 INFO:tasks.workunit.client.0.smithi001.stdout:6/517: write f5 [2719059,9055] 0 2024-02-28T16:31:11.958 INFO:tasks.workunit.client.0.smithi001.stdout:3/580: read d4/d6/d7/d78/d81/d8a/fa5 [2347899,12750] 0 2024-02-28T16:31:11.959 INFO:tasks.workunit.client.1.smithi066.stdout:5/118: chown d3/d4/f12 469 1 2024-02-28T16:31:11.961 INFO:tasks.workunit.client.1.smithi066.stdout:0/231: creat d2/d9/d10/f4e x:0 0 0 2024-02-28T16:31:11.966 INFO:tasks.workunit.client.0.smithi001.stdout:6/518: read f1 [1075089,60608] 0 2024-02-28T16:31:11.969 INFO:tasks.workunit.client.0.smithi001.stdout:6/519: unlink d9/d29/l2e 0 2024-02-28T16:31:11.970 INFO:tasks.workunit.client.1.smithi066.stdout:0/232: creat d2/d9/f4f x:0 0 0 2024-02-28T16:31:11.972 INFO:tasks.workunit.client.0.smithi001.stdout:6/520: creat d9/d29/d6e/d9a/fad x:0 0 0 2024-02-28T16:31:11.973 INFO:tasks.workunit.client.1.smithi066.stdout:4/201: dwrite d1/d18/f2d [0,4194304] 0 2024-02-28T16:31:11.975 INFO:tasks.workunit.client.1.smithi066.stdout:5/119: creat d3/d4/d17/d18/d24/f28 x:0 0 0 2024-02-28T16:31:11.976 INFO:tasks.workunit.client.0.smithi001.stdout:1/664: dread d0/d9/d56/f7f [0,4194304] 0 2024-02-28T16:31:11.977 INFO:tasks.workunit.client.1.smithi066.stdout:5/120: read d3/d4/f9 [2530708,7112] 0 2024-02-28T16:31:11.978 INFO:tasks.workunit.client.0.smithi001.stdout:6/521: creat d9/db/d5f/d62/d7e/fae x:0 0 0 2024-02-28T16:31:11.981 INFO:tasks.workunit.client.0.smithi001.stdout:1/665: creat d0/d9/d1a/d2b/d26/d27/d30/d67/dd6/fea x:0 0 0 2024-02-28T16:31:11.982 INFO:tasks.workunit.client.1.smithi066.stdout:5/121: write d3/d4/d17/d18/d24/f28 [576590,36419] 0 2024-02-28T16:31:11.983 INFO:tasks.workunit.client.1.smithi066.stdout:4/202: creat d1/d23/d25/d3e/f3f x:0 0 0 2024-02-28T16:31:11.983 INFO:tasks.workunit.client.0.smithi001.stdout:6/522: symlink d9/d29/d6e/d53/laf 0 2024-02-28T16:31:11.985 INFO:tasks.workunit.client.0.smithi001.stdout:1/666: stat d0/d9/d1a/d2b/f24 0 2024-02-28T16:31:11.987 INFO:tasks.workunit.client.0.smithi001.stdout:6/523: symlink d9/d29/d6a/lb0 0 2024-02-28T16:31:11.988 INFO:tasks.workunit.client.1.smithi066.stdout:4/203: write d1/f16 [2541433,54823] 0 2024-02-28T16:31:11.990 INFO:tasks.workunit.client.0.smithi001.stdout:3/581: dwrite d4/d1e/f22 [0,4194304] 0 2024-02-28T16:31:11.993 INFO:tasks.workunit.client.0.smithi001.stdout:1/667: mknod d0/d9/d1a/d2b/d26/d27/d30/d67/db7/ceb 0 2024-02-28T16:31:11.994 INFO:tasks.workunit.client.1.smithi066.stdout:0/233: dwrite d2/d9/f20 [0,4194304] 0 2024-02-28T16:31:11.995 INFO:tasks.workunit.client.0.smithi001.stdout:3/582: truncate d4/d3c/f71 5311 0 2024-02-28T16:31:11.995 INFO:tasks.workunit.client.1.smithi066.stdout:4/204: mkdir d1/d23/d25/d3e/d40 0 2024-02-28T16:31:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:11 smithi066 ceph-mon[36236]: pgmap v49: 65 pgs: 65 active+clean; 1.4 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 44 MiB/s rd, 184 MiB/s wr, 394 op/s 2024-02-28T16:31:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:11 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:11 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.rmthtx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:31:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:11 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.rmthtx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:31:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:11 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:31:11.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:11 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:31:12.000 INFO:tasks.workunit.client.1.smithi066.stdout:4/205: mkdir d1/d18/d41 0 2024-02-28T16:31:12.001 INFO:tasks.workunit.client.0.smithi001.stdout:3/583: dread d4/d1e/d7c/da8/fae [0,4194304] 0 2024-02-28T16:31:12.003 INFO:tasks.workunit.client.1.smithi066.stdout:4/206: chown d1/l8 128 1 2024-02-28T16:31:12.005 INFO:tasks.workunit.client.0.smithi001.stdout:1/668: creat d0/d9/d1a/fec x:0 0 0 2024-02-28T16:31:12.005 INFO:tasks.workunit.client.1.smithi066.stdout:4/207: write d1/f12 [1667944,73659] 0 2024-02-28T16:31:12.008 INFO:tasks.workunit.client.0.smithi001.stdout:1/669: truncate d0/d9/d56/d2f/f64 1403687 0 2024-02-28T16:31:12.011 INFO:tasks.workunit.client.1.smithi066.stdout:5/122: truncate d3/ff 2483710 0 2024-02-28T16:31:12.012 INFO:tasks.workunit.client.0.smithi001.stdout:1/670: mkdir d0/d9/d1a/d2b/d26/d27/dd3/de8/ded 0 2024-02-28T16:31:12.016 INFO:tasks.workunit.client.1.smithi066.stdout:5/123: getdents d3 0 2024-02-28T16:31:12.016 INFO:tasks.workunit.client.1.smithi066.stdout:1/172: sync 2024-02-28T16:31:12.016 INFO:tasks.workunit.client.1.smithi066.stdout:8/91: sync 2024-02-28T16:31:12.017 INFO:tasks.workunit.client.0.smithi001.stdout:1/671: fsync d0/d9/d1a/d2b/d26/d27/d30/f59 0 2024-02-28T16:31:12.018 INFO:tasks.workunit.client.1.smithi066.stdout:0/234: dwrite d2/d9/d10/f46 [0,4194304] 0 2024-02-28T16:31:12.020 INFO:tasks.workunit.client.0.smithi001.stdout:1/672: read d0/d9/d1a/d2b/d26/d27/d30/f74 [25637,81568] 0 2024-02-28T16:31:12.022 INFO:tasks.workunit.client.1.smithi066.stdout:1/173: rename d2/d27/d2b/f2e to d2/d27/d2b/f39 0 2024-02-28T16:31:12.023 INFO:tasks.workunit.client.1.smithi066.stdout:8/92: creat d0/d6/dc/d16/f19 x:0 0 0 2024-02-28T16:31:12.025 INFO:tasks.workunit.client.1.smithi066.stdout:1/174: read d2/d18/f26 [16037,57742] 0 2024-02-28T16:31:12.026 INFO:tasks.workunit.client.0.smithi001.stdout:1/673: fsync d0/d9/d1a/d2b/d26/d27/d30/d4a/f70 0 2024-02-28T16:31:12.029 INFO:tasks.workunit.client.1.smithi066.stdout:1/175: chown d2/f15 41318241 1 2024-02-28T16:31:12.032 INFO:tasks.workunit.client.1.smithi066.stdout:4/208: getdents d1/d18 0 2024-02-28T16:31:12.032 INFO:tasks.workunit.client.1.smithi066.stdout:8/93: mkdir d0/d6/dc/d16/d1a 0 2024-02-28T16:31:12.036 INFO:tasks.workunit.client.1.smithi066.stdout:0/235: dread d2/d9/d10/f39 [0,4194304] 0 2024-02-28T16:31:12.038 INFO:tasks.workunit.client.1.smithi066.stdout:4/209: mknod d1/d23/d25/c42 0 2024-02-28T16:31:12.042 INFO:tasks.workunit.client.1.smithi066.stdout:1/176: creat d2/d18/f3a x:0 0 0 2024-02-28T16:31:12.045 INFO:tasks.workunit.client.1.smithi066.stdout:4/210: mkdir d1/d23/d25/d43 0 2024-02-28T16:31:12.046 INFO:tasks.workunit.client.0.smithi001.stdout:1/674: dwrite d0/d9/d1a/d2b/d26/d27/d30/d88/fe4 [0,4194304] 0 2024-02-28T16:31:12.049 INFO:tasks.workunit.client.0.smithi001.stdout:1/675: dread d0/d9/d1a/d2b/d26/d27/d30/f74 [0,4194304] 0 2024-02-28T16:31:12.049 INFO:tasks.workunit.client.1.smithi066.stdout:4/211: read d1/f16 [1527877,111470] 0 2024-02-28T16:31:12.050 INFO:tasks.workunit.client.1.smithi066.stdout:1/177: dread d2/d18/f26 [0,4194304] 0 2024-02-28T16:31:12.051 INFO:tasks.workunit.client.1.smithi066.stdout:4/212: read d1/f4 [990945,5004] 0 2024-02-28T16:31:12.052 INFO:tasks.workunit.client.1.smithi066.stdout:8/94: dread d0/f13 [0,4194304] 0 2024-02-28T16:31:12.053 INFO:tasks.workunit.client.1.smithi066.stdout:1/178: chown d2/c22 1 1 2024-02-28T16:31:12.059 INFO:tasks.workunit.client.1.smithi066.stdout:4/213: fdatasync d1/f4 0 2024-02-28T16:31:12.060 INFO:tasks.workunit.client.0.smithi001.stdout:6/524: dread d9/db/fc [0,4194304] 0 2024-02-28T16:31:12.061 INFO:tasks.workunit.client.1.smithi066.stdout:1/179: creat d2/d18/d19/f3b x:0 0 0 2024-02-28T16:31:12.066 INFO:tasks.workunit.client.1.smithi066.stdout:0/236: dwrite d2/f22 [0,4194304] 0 2024-02-28T16:31:12.066 INFO:tasks.workunit.client.1.smithi066.stdout:1/180: creat d2/d27/f3c x:0 0 0 2024-02-28T16:31:12.066 INFO:tasks.workunit.client.0.smithi001.stdout:6/525: getdents d9/db/d5f/d89/d8a 0 2024-02-28T16:31:12.066 INFO:tasks.workunit.client.0.smithi001.stdout:1/676: dwrite d0/d9/d56/d2f/f9f [0,4194304] 0 2024-02-28T16:31:12.069 INFO:tasks.workunit.client.1.smithi066.stdout:4/214: getdents d1 0 2024-02-28T16:31:12.074 INFO:tasks.workunit.client.1.smithi066.stdout:4/215: mknod d1/d23/c44 0 2024-02-28T16:31:12.076 INFO:tasks.workunit.client.0.smithi001.stdout:1/677: link d0/d9/d1a/d2b/d26/f34 d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/fee 0 2024-02-28T16:31:12.076 INFO:tasks.workunit.client.1.smithi066.stdout:0/237: unlink d2/d9/d10/f2e 0 2024-02-28T16:31:12.078 INFO:tasks.workunit.client.1.smithi066.stdout:0/238: write d2/f28 [2006519,81722] 0 2024-02-28T16:31:12.078 INFO:tasks.workunit.client.1.smithi066.stdout:1/181: rename d2/ff to d2/d27/d2b/f3d 0 2024-02-28T16:31:12.081 INFO:tasks.workunit.client.1.smithi066.stdout:4/216: rename d1/d23/c24 to d1/d23/c45 0 2024-02-28T16:31:12.081 INFO:tasks.workunit.client.1.smithi066.stdout:0/239: chown d2/d9/c45 515320894 1 2024-02-28T16:31:12.082 INFO:tasks.workunit.client.1.smithi066.stdout:1/182: rmdir d2 39 2024-02-28T16:31:12.085 INFO:tasks.workunit.client.1.smithi066.stdout:4/217: read d1/d18/f2b [132739,84406] 0 2024-02-28T16:31:12.086 INFO:tasks.workunit.client.1.smithi066.stdout:0/240: mkdir d2/d9/d10/d50 0 2024-02-28T16:31:12.088 INFO:tasks.workunit.client.1.smithi066.stdout:1/183: write d2/d18/d19/f1e [173190,127984] 0 2024-02-28T16:31:12.089 INFO:tasks.workunit.client.0.smithi001.stdout:4/594: sync 2024-02-28T16:31:12.089 INFO:tasks.workunit.client.0.smithi001.stdout:2/580: sync 2024-02-28T16:31:12.090 INFO:tasks.workunit.client.0.smithi001.stdout:5/535: sync 2024-02-28T16:31:12.091 INFO:tasks.workunit.client.0.smithi001.stdout:0/554: sync 2024-02-28T16:31:12.091 INFO:tasks.workunit.client.0.smithi001.stdout:7/552: sync 2024-02-28T16:31:12.091 INFO:tasks.workunit.client.1.smithi066.stdout:8/95: dwrite d0/d6/dc/f14 [0,4194304] 0 2024-02-28T16:31:12.093 INFO:tasks.workunit.client.0.smithi001.stdout:6/526: dwrite d9/db/d19/f1f [0,4194304] 0 2024-02-28T16:31:12.093 INFO:tasks.workunit.client.1.smithi066.stdout:4/218: rename d1/d18/f2f to d1/f46 0 2024-02-28T16:31:12.093 INFO:tasks.workunit.client.1.smithi066.stdout:1/184: rmdir d2/d18/d19 39 2024-02-28T16:31:12.095 INFO:tasks.workunit.client.1.smithi066.stdout:8/96: write d0/d6/dc/d16/f19 [735870,109004] 0 2024-02-28T16:31:12.097 INFO:tasks.workunit.client.0.smithi001.stdout:1/678: truncate d0/f21 1167441 0 2024-02-28T16:31:12.100 INFO:tasks.workunit.client.0.smithi001.stdout:0/555: mknod d18/d22/d76/d7a/cb7 0 2024-02-28T16:31:12.104 INFO:tasks.workunit.client.0.smithi001.stdout:0/556: write d18/d22/d2b/d36/d48/f55 [2553868,36245] 0 2024-02-28T16:31:12.108 INFO:tasks.workunit.client.1.smithi066.stdout:1/185: dread d2/d27/f2f [0,4194304] 0 2024-02-28T16:31:12.108 INFO:tasks.workunit.client.0.smithi001.stdout:7/553: creat d3/d17/d3d/d59/fb1 x:0 0 0 2024-02-28T16:31:12.109 INFO:tasks.workunit.client.0.smithi001.stdout:4/595: rename d3/d5/d7/d1f/d3a/d68 to d3/d5/d7/d1f/d2e/dbe 0 2024-02-28T16:31:12.109 INFO:tasks.workunit.client.0.smithi001.stdout:6/527: fdatasync d9/de/f14 0 2024-02-28T16:31:12.110 INFO:tasks.workunit.client.0.smithi001.stdout:1/679: creat d0/d9/d1a/d2b/d26/d27/d30/d88/fef x:0 0 0 2024-02-28T16:31:12.112 INFO:tasks.workunit.client.1.smithi066.stdout:1/186: unlink d2/f4 0 2024-02-28T16:31:12.114 INFO:tasks.workunit.client.1.smithi066.stdout:4/219: write d1/f5 [136618,35959] 0 2024-02-28T16:31:12.115 INFO:tasks.workunit.client.1.smithi066.stdout:1/187: mknod d2/d27/d2b/c3e 0 2024-02-28T16:31:12.116 INFO:tasks.workunit.client.0.smithi001.stdout:7/554: write d3/d17/d52/d92/fa2 [1450099,100526] 0 2024-02-28T16:31:12.117 INFO:tasks.workunit.client.1.smithi066.stdout:4/220: fdatasync d1/d23/d25/f28 0 2024-02-28T16:31:12.120 INFO:tasks.workunit.client.1.smithi066.stdout:1/188: rename d2/l6 to d2/d27/d2b/l3f 0 2024-02-28T16:31:12.122 INFO:tasks.workunit.client.0.smithi001.stdout:2/581: rename d0/d2/d3/f34 to d0/d2/d3/de/d61/d12/d55/db2/fc7 0 2024-02-28T16:31:12.123 INFO:tasks.workunit.client.1.smithi066.stdout:4/221: creat d1/d18/d41/f47 x:0 0 0 2024-02-28T16:31:12.123 INFO:tasks.workunit.client.1.smithi066.stdout:1/189: fdatasync d2/d18/d19/f33 0 2024-02-28T16:31:12.125 INFO:tasks.workunit.client.1.smithi066.stdout:4/222: read d1/f12 [6460138,29546] 0 2024-02-28T16:31:12.125 INFO:tasks.workunit.client.0.smithi001.stdout:1/680: creat d0/d9/d1a/d2b/d26/d27/d30/d88/db6/ff0 x:0 0 0 2024-02-28T16:31:12.128 INFO:tasks.workunit.client.1.smithi066.stdout:4/223: mknod d1/d18/c48 0 2024-02-28T16:31:12.129 INFO:tasks.workunit.client.1.smithi066.stdout:1/190: symlink d2/d18/l40 0 2024-02-28T16:31:12.132 INFO:tasks.workunit.client.1.smithi066.stdout:4/224: dread d1/d18/f2b [0,4194304] 0 2024-02-28T16:31:12.135 INFO:tasks.workunit.client.0.smithi001.stdout:0/557: getdents d18/d22 0 2024-02-28T16:31:12.137 INFO:tasks.workunit.client.1.smithi066.stdout:1/191: truncate d2/d18/f21 1367334 0 2024-02-28T16:31:12.138 INFO:tasks.workunit.client.1.smithi066.stdout:4/225: creat d1/d18/d41/f49 x:0 0 0 2024-02-28T16:31:12.141 INFO:tasks.workunit.client.1.smithi066.stdout:4/226: creat d1/d18/f4a x:0 0 0 2024-02-28T16:31:12.145 INFO:tasks.workunit.client.1.smithi066.stdout:1/192: fdatasync d2/d18/d19/f3b 0 2024-02-28T16:31:12.145 INFO:tasks.workunit.client.1.smithi066.stdout:4/227: mknod d1/d18/c4b 0 2024-02-28T16:31:12.145 INFO:tasks.workunit.client.0.smithi001.stdout:4/596: dread d3/d1c/d58/f2a [0,4194304] 0 2024-02-28T16:31:12.152 INFO:tasks.workunit.client.0.smithi001.stdout:0/558: rmdir d18/d22/d57 39 2024-02-28T16:31:12.152 INFO:tasks.workunit.client.0.smithi001.stdout:6/528: rename d9/db/d5f/d62/f7d to d9/db/d5f/fb1 0 2024-02-28T16:31:12.152 INFO:tasks.workunit.client.0.smithi001.stdout:2/582: write d0/d2/d3/de/d61/d12/d55/db2/fc7 [748376,40668] 0 2024-02-28T16:31:12.152 INFO:tasks.workunit.client.0.smithi001.stdout:4/597: symlink d3/d2d/d4c/d5e/lbf 0 2024-02-28T16:31:12.153 INFO:tasks.workunit.client.1.smithi066.stdout:4/228: readlink d1/l1f 0 2024-02-28T16:31:12.153 INFO:tasks.workunit.client.1.smithi066.stdout:1/193: rmdir d2/d27/d2b 39 2024-02-28T16:31:12.154 INFO:tasks.workunit.client.1.smithi066.stdout:7/84: sync 2024-02-28T16:31:12.154 INFO:tasks.workunit.client.1.smithi066.stdout:9/122: sync 2024-02-28T16:31:12.154 INFO:tasks.workunit.client.1.smithi066.stdout:6/70: sync 2024-02-28T16:31:12.157 INFO:tasks.workunit.client.0.smithi001.stdout:7/555: dwrite d3/d17/d3d/f70 [0,4194304] 0 2024-02-28T16:31:12.158 INFO:tasks.workunit.client.0.smithi001.stdout:1/681: dread d0/d9/d1a/f33 [0,4194304] 0 2024-02-28T16:31:12.158 INFO:tasks.workunit.client.0.smithi001.stdout:2/583: readlink d0/l60 0 2024-02-28T16:31:12.158 INFO:tasks.workunit.client.0.smithi001.stdout:6/529: mknod d9/cb2 0 2024-02-28T16:31:12.160 INFO:tasks.workunit.client.1.smithi066.stdout:7/85: truncate dc/de/f13 19207 0 2024-02-28T16:31:12.161 INFO:tasks.workunit.client.1.smithi066.stdout:1/194: write d2/d27/d2b/f3d [3347200,3408] 0 2024-02-28T16:31:12.161 INFO:tasks.workunit.client.0.smithi001.stdout:4/598: mknod d3/d5/d7/d1f/d20/cc0 0 2024-02-28T16:31:12.163 INFO:tasks.workunit.client.0.smithi001.stdout:7/556: write d3/d1c/f33 [7460784,129810] 0 2024-02-28T16:31:12.165 INFO:tasks.workunit.client.0.smithi001.stdout:6/530: rmdir d9/db/d19 39 2024-02-28T16:31:12.166 INFO:tasks.workunit.client.0.smithi001.stdout:2/584: mkdir d0/d2/d3/de/d22/dbb/dc8 0 2024-02-28T16:31:12.167 INFO:tasks.workunit.client.0.smithi001.stdout:4/599: chown d3/d5/d7/d1f/d2e/f64 47549 1 2024-02-28T16:31:12.170 INFO:tasks.workunit.client.0.smithi001.stdout:4/600: stat d3/d2d/d83/fb9 0 2024-02-28T16:31:12.171 INFO:tasks.workunit.client.1.smithi066.stdout:9/123: dread f3 [0,4194304] 0 2024-02-28T16:31:12.171 INFO:tasks.workunit.client.1.smithi066.stdout:4/229: dwrite d1/f20 [0,4194304] 0 2024-02-28T16:31:12.172 INFO:tasks.workunit.client.0.smithi001.stdout:2/585: chown d0/d2/d3/de/d61/d12/d55 7556 1 2024-02-28T16:31:12.176 INFO:tasks.workunit.client.1.smithi066.stdout:9/124: write da/f1e [1007268,127064] 0 2024-02-28T16:31:12.177 INFO:tasks.workunit.client.1.smithi066.stdout:7/86: rmdir dc 39 2024-02-28T16:31:12.179 INFO:tasks.workunit.client.0.smithi001.stdout:4/601: fsync d3/d5/d92/fbc 0 2024-02-28T16:31:12.182 INFO:tasks.workunit.client.1.smithi066.stdout:6/71: dwrite d2/f8 [0,4194304] 0 2024-02-28T16:31:12.183 INFO:tasks.workunit.client.0.smithi001.stdout:6/531: creat d9/fb3 x:0 0 0 2024-02-28T16:31:12.186 INFO:tasks.workunit.client.0.smithi001.stdout:4/602: write d3/fb4 [196520,45354] 0 2024-02-28T16:31:12.186 INFO:tasks.workunit.client.1.smithi066.stdout:7/87: chown dc/lf 282264 1 2024-02-28T16:31:12.187 INFO:tasks.workunit.client.1.smithi066.stdout:9/125: symlink da/df/l26 0 2024-02-28T16:31:12.189 INFO:tasks.workunit.client.0.smithi001.stdout:0/559: dwrite d18/d31/d4f/f82 [0,4194304] 0 2024-02-28T16:31:12.192 INFO:tasks.workunit.client.1.smithi066.stdout:6/72: creat d2/f12 x:0 0 0 2024-02-28T16:31:12.192 INFO:tasks.workunit.client.1.smithi066.stdout:7/88: mknod dc/de/d17/c1a 0 2024-02-28T16:31:12.194 INFO:tasks.workunit.client.0.smithi001.stdout:9/679: sync 2024-02-28T16:31:12.194 INFO:tasks.workunit.client.0.smithi001.stdout:8/587: sync 2024-02-28T16:31:12.194 INFO:tasks.workunit.client.0.smithi001.stdout:0/560: fsync d18/f59 0 2024-02-28T16:31:12.196 INFO:tasks.workunit.client.1.smithi066.stdout:1/195: dwrite d2/d27/d2b/f2a [0,4194304] 0 2024-02-28T16:31:12.199 INFO:tasks.workunit.client.0.smithi001.stdout:8/588: chown d4/d7/f21 251702662 1 2024-02-28T16:31:12.199 INFO:tasks.workunit.client.0.smithi001.stdout:6/532: rename d9/db/d10/fa4 to d9/d29/d6e/d53/fb4 0 2024-02-28T16:31:12.200 INFO:tasks.workunit.client.0.smithi001.stdout:4/603: write d3/d5/faa [1287474,26592] 0 2024-02-28T16:31:12.201 INFO:tasks.workunit.client.0.smithi001.stdout:7/557: dwrite d3/d17/d61/f71 [0,4194304] 0 2024-02-28T16:31:12.204 INFO:tasks.workunit.client.0.smithi001.stdout:6/533: chown d9/db/la5 401951 1 2024-02-28T16:31:12.207 INFO:tasks.workunit.client.1.smithi066.stdout:6/73: unlink d2/c11 0 2024-02-28T16:31:12.207 INFO:tasks.workunit.client.1.smithi066.stdout:7/89: creat dc/de/f1b x:0 0 0 2024-02-28T16:31:12.214 INFO:tasks.workunit.client.1.smithi066.stdout:9/126: dread da/f1e [0,4194304] 0 2024-02-28T16:31:12.216 INFO:tasks.workunit.client.1.smithi066.stdout:6/74: write d2/da/fc [4421042,128333] 0 2024-02-28T16:31:12.217 INFO:tasks.workunit.client.1.smithi066.stdout:4/230: dwrite d1/d18/f4a [0,4194304] 0 2024-02-28T16:31:12.220 INFO:tasks.workunit.client.0.smithi001.stdout:9/680: readlink d5/d27/d42/d48/le1 0 2024-02-28T16:31:12.220 INFO:tasks.workunit.client.0.smithi001.stdout:4/604: write d3/f78 [3537934,29309] 0 2024-02-28T16:31:12.220 INFO:tasks.workunit.client.0.smithi001.stdout:2/586: dread d0/d2/d3/de/f8f [0,4194304] 0 2024-02-28T16:31:12.221 INFO:tasks.workunit.client.1.smithi066.stdout:6/75: chown d2/lb 1771 1 2024-02-28T16:31:12.222 INFO:tasks.workunit.client.1.smithi066.stdout:9/127: symlink da/d21/l27 0 2024-02-28T16:31:12.222 INFO:tasks.workunit.client.0.smithi001.stdout:0/561: fsync d18/d22/d3d/f5e 0 2024-02-28T16:31:12.226 INFO:tasks.workunit.client.1.smithi066.stdout:6/76: chown d2/d4/lf 1 1 2024-02-28T16:31:12.227 INFO:tasks.workunit.client.1.smithi066.stdout:9/128: mkdir da/df/d18/d28 0 2024-02-28T16:31:12.228 INFO:tasks.workunit.client.0.smithi001.stdout:7/558: dread d3/d17/d3d/f4d [0,4194304] 0 2024-02-28T16:31:12.228 INFO:tasks.workunit.client.1.smithi066.stdout:9/129: fsync da/fc 0 2024-02-28T16:31:12.229 INFO:tasks.workunit.client.0.smithi001.stdout:2/587: write d0/d2/d3/d36/f4b [4908055,28252] 0 2024-02-28T16:31:12.229 INFO:tasks.workunit.client.0.smithi001.stdout:8/589: rename d4/d7/c38 to d4/d7/d22/d27/d4b/cbf 0 2024-02-28T16:31:12.230 INFO:tasks.workunit.client.0.smithi001.stdout:7/559: chown d3/d17/fa5 71662 1 2024-02-28T16:31:12.231 INFO:tasks.workunit.client.1.smithi066.stdout:6/77: creat d2/da/f13 x:0 0 0 2024-02-28T16:31:12.234 INFO:tasks.workunit.client.1.smithi066.stdout:9/130: creat da/df/f29 x:0 0 0 2024-02-28T16:31:12.235 INFO:tasks.workunit.client.1.smithi066.stdout:6/78: write d2/da/fc [1266778,843] 0 2024-02-28T16:31:12.236 INFO:tasks.workunit.client.1.smithi066.stdout:9/131: stat da/d10 0 2024-02-28T16:31:12.237 INFO:tasks.workunit.client.0.smithi001.stdout:8/590: write d4/d32/d69/f72 [1012623,39172] 0 2024-02-28T16:31:12.237 INFO:tasks.workunit.client.0.smithi001.stdout:6/534: dwrite d9/d29/d6e/f18 [0,4194304] 0 2024-02-28T16:31:12.240 INFO:tasks.workunit.client.0.smithi001.stdout:4/605: rename d3/d2d/d4c/c73 to d3/d2d/d4c/d69/d82/cc1 0 2024-02-28T16:31:12.243 INFO:tasks.workunit.client.0.smithi001.stdout:0/562: fdatasync d18/d22/d2b/d5f/f85 0 2024-02-28T16:31:12.243 INFO:tasks.workunit.client.0.smithi001.stdout:2/588: write d0/d2/d3/de/d61/f67 [572016,74439] 0 2024-02-28T16:31:12.247 INFO:tasks.workunit.client.0.smithi001.stdout:2/589: stat d0/d2/d3/de/d61/d12/d46/l8b 0 2024-02-28T16:31:12.249 INFO:tasks.workunit.client.1.smithi066.stdout:1/196: truncate d2/d27/f2f 1383898 0 2024-02-28T16:31:12.249 INFO:tasks.workunit.client.0.smithi001.stdout:7/560: rename d3/d17/d61/ca1 to d3/d17/d61/cb2 0 2024-02-28T16:31:12.252 INFO:tasks.workunit.client.0.smithi001.stdout:7/561: read - d3/d17/d3d/d59/fb1 zero size 2024-02-28T16:31:12.252 INFO:tasks.workunit.client.1.smithi066.stdout:4/231: dwrite d1/d23/f39 [0,4194304] 0 2024-02-28T16:31:12.252 INFO:tasks.workunit.client.1.smithi066.stdout:9/132: creat da/f2a x:0 0 0 2024-02-28T16:31:12.254 INFO:tasks.workunit.client.0.smithi001.stdout:6/535: mknod d9/db/cb5 0 2024-02-28T16:31:12.254 INFO:tasks.workunit.client.1.smithi066.stdout:9/133: chown da/df/d18 76 1 2024-02-28T16:31:12.255 INFO:tasks.workunit.client.0.smithi001.stdout:7/562: write d3/d1c/f4b [1221479,63878] 0 2024-02-28T16:31:12.256 INFO:tasks.workunit.client.1.smithi066.stdout:1/197: dread d2/f15 [0,4194304] 0 2024-02-28T16:31:12.256 INFO:tasks.workunit.client.1.smithi066.stdout:9/134: dread - da/f20 zero size 2024-02-28T16:31:12.256 INFO:tasks.workunit.client.0.smithi001.stdout:0/563: rename d18/d22/d2b/f88 to d18/d22/d2b/d36/d46/fb8 0 2024-02-28T16:31:12.260 INFO:tasks.workunit.client.1.smithi066.stdout:4/232: mknod d1/d23/d25/d3e/d40/c4c 0 2024-02-28T16:31:12.261 INFO:tasks.workunit.client.0.smithi001.stdout:9/681: link d5/d27/db1/d58/cd5 d5/ce6 0 2024-02-28T16:31:12.261 INFO:tasks.workunit.client.1.smithi066.stdout:1/198: fdatasync d2/d27/f3c 0 2024-02-28T16:31:12.265 INFO:tasks.workunit.client.0.smithi001.stdout:3/584: sync 2024-02-28T16:31:12.265 INFO:tasks.workunit.client.1.smithi066.stdout:7/90: write dc/de/f13 [710152,93999] 0 2024-02-28T16:31:12.270 INFO:tasks.workunit.client.1.smithi066.stdout:9/135: creat da/df/d18/d28/f2b x:0 0 0 2024-02-28T16:31:12.270 INFO:tasks.workunit.client.1.smithi066.stdout:6/79: dwrite d2/f12 [0,4194304] 0 2024-02-28T16:31:12.273 INFO:tasks.workunit.client.1.smithi066.stdout:6/80: read - d2/da/fe zero size 2024-02-28T16:31:12.274 INFO:tasks.workunit.client.1.smithi066.stdout:9/136: write da/d10/d15/f1d [184689,9041] 0 2024-02-28T16:31:12.275 INFO:tasks.workunit.client.1.smithi066.stdout:6/81: chown d2/f10 18706705 1 2024-02-28T16:31:12.278 INFO:tasks.workunit.client.1.smithi066.stdout:6/82: creat d2/f14 x:0 0 0 2024-02-28T16:31:12.280 INFO:tasks.workunit.client.1.smithi066.stdout:6/83: readlink d2/d4/lf 0 2024-02-28T16:31:12.280 INFO:tasks.workunit.client.1.smithi066.stdout:9/137: rename da/d21/l27 to da/df/d18/d28/l2c 0 2024-02-28T16:31:12.281 INFO:tasks.workunit.client.1.smithi066.stdout:6/84: rename d2/d4 to d2/d4/d15 22 2024-02-28T16:31:12.282 INFO:tasks.workunit.client.1.smithi066.stdout:7/91: dread f1 [0,4194304] 0 2024-02-28T16:31:12.282 INFO:tasks.workunit.client.1.smithi066.stdout:9/138: fsync da/d10/d15/f1d 0 2024-02-28T16:31:12.284 INFO:tasks.workunit.client.1.smithi066.stdout:7/92: readlink dc/l14 0 2024-02-28T16:31:12.286 INFO:tasks.workunit.client.1.smithi066.stdout:6/85: symlink d2/l16 0 2024-02-28T16:31:12.287 INFO:tasks.workunit.client.1.smithi066.stdout:1/199: truncate d2/f7 3730827 0 2024-02-28T16:31:12.289 INFO:tasks.workunit.client.1.smithi066.stdout:9/139: write da/d10/d15/f19 [9344224,44299] 0 2024-02-28T16:31:12.292 INFO:tasks.workunit.client.1.smithi066.stdout:7/93: symlink dc/de/d17/l1c 0 2024-02-28T16:31:12.294 INFO:tasks.workunit.client.1.smithi066.stdout:2/160: sync 2024-02-28T16:31:12.295 INFO:tasks.workunit.client.1.smithi066.stdout:1/200: fsync d2/f17 0 2024-02-28T16:31:12.295 INFO:tasks.workunit.client.1.smithi066.stdout:7/94: mknod dc/c1d 0 2024-02-28T16:31:12.296 INFO:tasks.workunit.client.0.smithi001.stdout:6/536: dread d9/db/d5f/d8b/f6d [0,4194304] 0 2024-02-28T16:31:12.297 INFO:tasks.workunit.client.0.smithi001.stdout:6/537: readlink d9/db/la5 0 2024-02-28T16:31:12.300 INFO:tasks.workunit.client.1.smithi066.stdout:1/201: chown d2/d27/l2d 501589 1 2024-02-28T16:31:12.301 INFO:tasks.workunit.client.1.smithi066.stdout:7/95: mkdir dc/de/d17/d1e 0 2024-02-28T16:31:12.302 INFO:tasks.workunit.client.1.smithi066.stdout:9/140: getdents da/df/d18 0 2024-02-28T16:31:12.304 INFO:tasks.workunit.client.1.smithi066.stdout:7/96: dread - dc/de/f1b zero size 2024-02-28T16:31:12.306 INFO:tasks.workunit.client.1.smithi066.stdout:7/97: rmdir dc 39 2024-02-28T16:31:12.308 INFO:tasks.workunit.client.1.smithi066.stdout:6/86: dwrite d2/da/f13 [0,4194304] 0 2024-02-28T16:31:12.310 INFO:tasks.workunit.client.1.smithi066.stdout:7/98: fdatasync fa 0 2024-02-28T16:31:12.311 INFO:tasks.workunit.client.0.smithi001.stdout:2/590: link d0/d2/d3/f5 d0/d2/d3/de/d8a/fc9 0 2024-02-28T16:31:12.314 INFO:tasks.workunit.client.1.smithi066.stdout:2/161: dread d0/f1f [0,4194304] 0 2024-02-28T16:31:12.314 INFO:tasks.workunit.client.0.smithi001.stdout:2/591: write d0/d2/d3/f43 [1186862,61842] 0 2024-02-28T16:31:12.314 INFO:tasks.workunit.client.0.smithi001.stdout:7/563: unlink d3/d17/d3d/d5d/d9a/d49/d63/f8a 0 2024-02-28T16:31:12.316 INFO:tasks.workunit.client.0.smithi001.stdout:9/682: fsync d5/f19 0 2024-02-28T16:31:12.318 INFO:tasks.workunit.client.0.smithi001.stdout:4/606: getdents d3/d5/d47/d90 0 2024-02-28T16:31:12.319 INFO:tasks.workunit.client.1.smithi066.stdout:2/162: rmdir d0 39 2024-02-28T16:31:12.319 INFO:tasks.workunit.client.0.smithi001.stdout:7/564: read d3/d17/d61/f71 [1091650,84758] 0 2024-02-28T16:31:12.320 INFO:tasks.workunit.client.1.smithi066.stdout:7/99: truncate dc/de/f13 853458 0 2024-02-28T16:31:12.320 INFO:tasks.workunit.client.0.smithi001.stdout:8/591: dread - d4/d7/d22/d27/d58/d94/fb5 zero size 2024-02-28T16:31:12.322 INFO:tasks.workunit.client.1.smithi066.stdout:7/100: chown dc/de/l18 231 1 2024-02-28T16:31:12.323 INFO:tasks.workunit.client.0.smithi001.stdout:0/564: dread d18/d22/d2f/f24 [0,4194304] 0 2024-02-28T16:31:12.324 INFO:tasks.workunit.client.0.smithi001.stdout:3/585: symlink d4/d6/lc2 0 2024-02-28T16:31:12.325 INFO:tasks.workunit.client.0.smithi001.stdout:7/565: chown d3/d17/d3d/d59/d6e/f9f 44871 1 2024-02-28T16:31:12.325 INFO:tasks.workunit.client.1.smithi066.stdout:1/202: write d2/d18/f26 [781213,5412] 0 2024-02-28T16:31:12.326 INFO:tasks.workunit.client.1.smithi066.stdout:6/87: dread d2/f12 [0,4194304] 0 2024-02-28T16:31:12.327 INFO:tasks.workunit.client.1.smithi066.stdout:2/163: rename d0/f8 to d0/f37 0 2024-02-28T16:31:12.329 INFO:tasks.workunit.client.1.smithi066.stdout:6/88: write d2/f14 [573953,121117] 0 2024-02-28T16:31:12.330 INFO:tasks.workunit.client.0.smithi001.stdout:4/607: unlink d3/d5/d92/fbc 0 2024-02-28T16:31:12.332 INFO:tasks.workunit.client.0.smithi001.stdout:4/608: stat d3/d5/c5f 0 2024-02-28T16:31:12.332 INFO:tasks.workunit.client.1.smithi066.stdout:6/89: chown f1 13650 1 2024-02-28T16:31:12.333 INFO:tasks.workunit.client.0.smithi001.stdout:3/586: write d4/d6/d7/f88 [4279687,92429] 0 2024-02-28T16:31:12.337 INFO:tasks.workunit.client.1.smithi066.stdout:6/90: creat d2/d4/f17 x:0 0 0 2024-02-28T16:31:12.337 INFO:tasks.workunit.client.0.smithi001.stdout:7/566: write d3/d1c/f4b [2908474,85922] 0 2024-02-28T16:31:12.346 INFO:tasks.workunit.client.0.smithi001.stdout:8/592: dwrite d4/d7/d22/d27/f60 [0,4194304] 0 2024-02-28T16:31:12.349 INFO:tasks.workunit.client.1.smithi066.stdout:2/164: dread d0/f1f [0,4194304] 0 2024-02-28T16:31:12.351 INFO:tasks.workunit.client.0.smithi001.stdout:9/683: rename d5/f15 to d5/d27/d34/fe7 0 2024-02-28T16:31:12.356 INFO:tasks.workunit.client.0.smithi001.stdout:0/565: creat d18/d22/d2f/d1c/fb9 x:0 0 0 2024-02-28T16:31:12.356 INFO:tasks.workunit.client.0.smithi001.stdout:9/684: read d5/d27/db1/f37 [1634562,22436] 0 2024-02-28T16:31:12.357 INFO:tasks.workunit.client.1.smithi066.stdout:2/165: dread d0/f37 [0,4194304] 0 2024-02-28T16:31:12.358 INFO:tasks.workunit.client.1.smithi066.stdout:1/203: dwrite d2/d27/d2b/f2c [0,4194304] 0 2024-02-28T16:31:12.358 INFO:tasks.workunit.client.0.smithi001.stdout:3/587: creat d4/d6/d7/d78/d81/fc3 x:0 0 0 2024-02-28T16:31:12.360 INFO:tasks.workunit.client.0.smithi001.stdout:7/567: mkdir d3/d17/d3d/db3 0 2024-02-28T16:31:12.362 INFO:tasks.workunit.client.0.smithi001.stdout:2/592: creat d0/d2/d3/fca x:0 0 0 2024-02-28T16:31:12.362 INFO:tasks.workunit.client.1.smithi066.stdout:2/166: truncate d0/d12/d1a/f2b 528706 0 2024-02-28T16:31:12.365 INFO:tasks.workunit.client.1.smithi066.stdout:1/204: mknod d2/d18/c41 0 2024-02-28T16:31:12.366 INFO:tasks.workunit.client.0.smithi001.stdout:3/588: stat d4/d18/c12 0 2024-02-28T16:31:12.367 INFO:tasks.workunit.client.1.smithi066.stdout:1/205: readlink d2/le 0 2024-02-28T16:31:12.369 INFO:tasks.workunit.client.1.smithi066.stdout:2/167: mkdir d0/d12/d2e/d38 0 2024-02-28T16:31:12.370 INFO:tasks.workunit.client.0.smithi001.stdout:3/589: read - d4/d6/d7/d78/f95 zero size 2024-02-28T16:31:12.372 INFO:tasks.workunit.client.1.smithi066.stdout:6/91: dwrite f1 [0,4194304] 0 2024-02-28T16:31:12.373 INFO:tasks.workunit.client.0.smithi001.stdout:3/590: dread - d4/d1e/f9e zero size 2024-02-28T16:31:12.375 INFO:tasks.workunit.client.1.smithi066.stdout:2/168: mkdir d0/d12/d1a/d39 0 2024-02-28T16:31:12.379 INFO:tasks.workunit.client.0.smithi001.stdout:9/685: rmdir d5/d9a/dce/dd0 0 2024-02-28T16:31:12.380 INFO:tasks.workunit.client.1.smithi066.stdout:2/169: creat d0/d12/d1a/f3a x:0 0 0 2024-02-28T16:31:12.381 INFO:tasks.workunit.client.0.smithi001.stdout:4/609: getdents d3 0 2024-02-28T16:31:12.383 INFO:tasks.workunit.client.1.smithi066.stdout:2/170: mkdir d0/d12/d2e/d3b 0 2024-02-28T16:31:12.386 INFO:tasks.workunit.client.0.smithi001.stdout:9/686: write d5/d27/fb0 [10861,41401] 0 2024-02-28T16:31:12.386 INFO:tasks.workunit.client.0.smithi001.stdout:3/591: read d4/d18/d5f/f47 [1486267,94867] 0 2024-02-28T16:31:12.388 INFO:tasks.workunit.client.1.smithi066.stdout:2/171: write d0/f37 [1155971,58402] 0 2024-02-28T16:31:12.389 INFO:tasks.workunit.client.1.smithi066.stdout:2/172: dread - d0/f19 zero size 2024-02-28T16:31:12.392 INFO:tasks.workunit.client.0.smithi001.stdout:5/536: sync 2024-02-28T16:31:12.393 INFO:tasks.workunit.client.0.smithi001.stdout:1/682: sync 2024-02-28T16:31:12.394 INFO:tasks.workunit.client.1.smithi066.stdout:6/92: dwrite f1 [0,4194304] 0 2024-02-28T16:31:12.396 INFO:tasks.workunit.client.0.smithi001.stdout:5/537: stat dc/df 0 2024-02-28T16:31:12.400 INFO:tasks.workunit.client.1.smithi066.stdout:6/93: link d2/da/fc d2/da/f18 0 2024-02-28T16:31:12.400 INFO:tasks.workunit.client.0.smithi001.stdout:1/683: mknod d0/d9/d1a/d72/cf1 0 2024-02-28T16:31:12.401 INFO:tasks.workunit.client.0.smithi001.stdout:7/568: dread d3/d17/f19 [0,4194304] 0 2024-02-28T16:31:12.402 INFO:tasks.workunit.client.0.smithi001.stdout:8/593: truncate d4/d47/d8b/f8c 6602163 0 2024-02-28T16:31:12.402 INFO:tasks.workunit.client.0.smithi001.stdout:9/687: symlink d5/d27/d34/d50/le8 0 2024-02-28T16:31:12.403 INFO:tasks.workunit.client.0.smithi001.stdout:2/593: write d0/d2/d3/de/d8a/f51 [3636439,32060] 0 2024-02-28T16:31:12.405 INFO:tasks.workunit.client.0.smithi001.stdout:4/610: dread d3/d5/d7/d1f/d2e/fb7 [0,4194304] 0 2024-02-28T16:31:12.405 INFO:tasks.workunit.client.1.smithi066.stdout:6/94: rename d2/d4 to d2/da/d19 0 2024-02-28T16:31:12.406 INFO:tasks.workunit.client.1.smithi066.stdout:6/95: stat d2/f14 0 2024-02-28T16:31:12.410 INFO:tasks.workunit.client.1.smithi066.stdout:6/96: creat d2/da/f1a x:0 0 0 2024-02-28T16:31:12.410 INFO:tasks.workunit.client.0.smithi001.stdout:7/569: write d3/d17/d3d/d5d/d9a/f6d [3453474,115254] 0 2024-02-28T16:31:12.412 INFO:tasks.workunit.client.0.smithi001.stdout:2/594: mknod d0/d2/d3/de/d8a/ccb 0 2024-02-28T16:31:12.412 INFO:tasks.workunit.client.0.smithi001.stdout:8/594: dread d4/d7/d22/d27/d4b/f63 [0,4194304] 0 2024-02-28T16:31:12.413 INFO:tasks.workunit.client.1.smithi066.stdout:2/173: dwrite d0/f31 [0,4194304] 0 2024-02-28T16:31:12.415 INFO:tasks.workunit.client.1.smithi066.stdout:0/241: sync 2024-02-28T16:31:12.415 INFO:tasks.workunit.client.1.smithi066.stdout:3/159: sync 2024-02-28T16:31:12.415 INFO:tasks.workunit.client.1.smithi066.stdout:5/124: sync 2024-02-28T16:31:12.416 INFO:tasks.workunit.client.1.smithi066.stdout:8/97: sync 2024-02-28T16:31:12.418 INFO:tasks.workunit.client.1.smithi066.stdout:0/242: readlink d2/l31 0 2024-02-28T16:31:12.418 INFO:tasks.workunit.client.1.smithi066.stdout:2/174: symlink d0/d12/d2e/d38/l3c 0 2024-02-28T16:31:12.421 INFO:tasks.workunit.client.1.smithi066.stdout:8/98: unlink d0/d6/dc/d16/f19 0 2024-02-28T16:31:12.424 INFO:tasks.workunit.client.1.smithi066.stdout:8/99: creat d0/d6/f1b x:0 0 0 2024-02-28T16:31:12.428 INFO:tasks.workunit.client.1.smithi066.stdout:0/243: mkdir d2/d9/d10/d50/d51 0 2024-02-28T16:31:12.430 INFO:tasks.workunit.client.0.smithi001.stdout:9/688: creat d5/fe9 x:0 0 0 2024-02-28T16:31:12.433 INFO:tasks.workunit.client.0.smithi001.stdout:7/570: creat d3/d17/d3d/db3/fb4 x:0 0 0 2024-02-28T16:31:12.442 INFO:tasks.workunit.client.0.smithi001.stdout:5/538: dwrite dc/d1a/d52/f89 [0,4194304] 0 2024-02-28T16:31:12.442 INFO:tasks.workunit.client.0.smithi001.stdout:4/611: truncate d3/d5/f34 2098683 0 2024-02-28T16:31:12.443 INFO:tasks.workunit.client.0.smithi001.stdout:8/595: creat d4/d7/d22/d27/d58/d94/db4/fc0 x:0 0 0 2024-02-28T16:31:12.444 INFO:tasks.workunit.client.0.smithi001.stdout:5/539: chown dc 2745416 1 2024-02-28T16:31:12.444 INFO:tasks.workunit.client.0.smithi001.stdout:3/592: dwrite d4/d6/d7/d78/d81/da1/fb9 [0,4194304] 0 2024-02-28T16:31:12.444 INFO:tasks.workunit.client.0.smithi001.stdout:0/566: dread d18/d22/d57/d7e/f96 [0,4194304] 0 2024-02-28T16:31:12.448 INFO:tasks.workunit.client.0.smithi001.stdout:3/593: write d4/d6/fc [4264947,48534] 0 2024-02-28T16:31:12.448 INFO:tasks.workunit.client.0.smithi001.stdout:5/540: chown dc/d1a/d7d/f88 17 1 2024-02-28T16:31:12.453 INFO:tasks.workunit.client.0.smithi001.stdout:9/689: read d5/d27/f2b [7905468,115590] 0 2024-02-28T16:31:12.454 INFO:tasks.workunit.client.0.smithi001.stdout:0/567: chown d18/d22/d2b/d36/d46/ca7 395 1 2024-02-28T16:31:12.460 INFO:tasks.workunit.client.0.smithi001.stdout:5/541: read dc/d12/d41/d54/fa5 [1944761,73058] 0 2024-02-28T16:31:12.460 INFO:tasks.workunit.client.0.smithi001.stdout:3/594: symlink d4/d6/d7/d23/lc4 0 2024-02-28T16:31:12.467 INFO:tasks.workunit.client.0.smithi001.stdout:3/595: fdatasync d4/d6/d7/f88 0 2024-02-28T16:31:12.469 INFO:tasks.workunit.client.0.smithi001.stdout:8/596: creat d4/d7/da8/d67/fc1 x:0 0 0 2024-02-28T16:31:12.469 INFO:tasks.workunit.client.0.smithi001.stdout:1/684: dwrite d0/d9/d1a/d72/fb9 [0,4194304] 0 2024-02-28T16:31:12.473 INFO:tasks.workunit.client.0.smithi001.stdout:6/538: sync 2024-02-28T16:31:12.485 INFO:tasks.workunit.client.0.smithi001.stdout:0/568: dread d18/d22/d2f/f24 [0,4194304] 0 2024-02-28T16:31:12.486 INFO:tasks.workunit.client.1.smithi066.stdout:6/97: getdents d2/da 0 2024-02-28T16:31:12.487 INFO:tasks.workunit.client.0.smithi001.stdout:2/595: dread d0/d2/d3/de/d8a/f6e [0,4194304] 0 2024-02-28T16:31:12.488 INFO:tasks.workunit.client.1.smithi066.stdout:5/125: link d3/c22 d3/d1b/c29 0 2024-02-28T16:31:12.488 INFO:tasks.workunit.client.1.smithi066.stdout:1/206: dread d2/f10 [0,4194304] 0 2024-02-28T16:31:12.490 INFO:tasks.workunit.client.1.smithi066.stdout:6/98: write f1 [4403887,12220] 0 2024-02-28T16:31:12.491 INFO:tasks.workunit.client.0.smithi001.stdout:9/690: dwrite d5/d27/db1/d33/fdb [4194304,4194304] 0 2024-02-28T16:31:12.491 INFO:tasks.workunit.client.0.smithi001.stdout:6/539: creat d9/db/d5f/d62/d83/fb6 x:0 0 0 2024-02-28T16:31:12.494 INFO:tasks.workunit.client.1.smithi066.stdout:1/207: rmdir d2/d18/d19 39 2024-02-28T16:31:12.494 INFO:tasks.workunit.client.1.smithi066.stdout:2/175: dread d0/f1e [0,4194304] 0 2024-02-28T16:31:12.495 INFO:tasks.workunit.client.1.smithi066.stdout:5/126: mknod d3/d4/c2a 0 2024-02-28T16:31:12.496 INFO:tasks.workunit.client.0.smithi001.stdout:1/685: mkdir d0/d9/d1a/d2b/d26/d27/df2 0 2024-02-28T16:31:12.497 INFO:tasks.workunit.client.0.smithi001.stdout:4/612: truncate d3/d5/f91 3115831 0 2024-02-28T16:31:12.498 INFO:tasks.workunit.client.1.smithi066.stdout:6/99: write d2/da/f1a [1009253,29908] 0 2024-02-28T16:31:12.499 INFO:tasks.workunit.client.0.smithi001.stdout:9/691: write d5/d27/fb0 [252201,1702] 0 2024-02-28T16:31:12.500 INFO:tasks.workunit.client.1.smithi066.stdout:1/208: dread - d2/d18/d19/f3b zero size 2024-02-28T16:31:12.500 INFO:tasks.workunit.client.0.smithi001.stdout:5/542: write dc/d1a/d21/d2d/d3c/f49 [564626,8106] 0 2024-02-28T16:31:12.501 INFO:tasks.workunit.client.0.smithi001.stdout:4/613: chown d3/d2d/la3 364846 1 2024-02-28T16:31:12.503 INFO:tasks.workunit.client.1.smithi066.stdout:6/100: creat d2/da/f1b x:0 0 0 2024-02-28T16:31:12.504 INFO:tasks.workunit.client.0.smithi001.stdout:3/596: creat d4/d6/d7/d78/fc5 x:0 0 0 2024-02-28T16:31:12.504 INFO:tasks.workunit.client.0.smithi001.stdout:6/540: mknod d9/db/d5f/cb7 0 2024-02-28T16:31:12.506 INFO:tasks.workunit.client.1.smithi066.stdout:2/176: mknod d0/d12/d1a/c3d 0 2024-02-28T16:31:12.507 INFO:tasks.workunit.client.0.smithi001.stdout:5/543: symlink dc/df/d79/d98/lb4 0 2024-02-28T16:31:12.508 INFO:tasks.workunit.client.0.smithi001.stdout:1/686: write d0/d9/d1a/d2b/d26/d27/d30/d88/db6/ff0 [716861,37] 0 2024-02-28T16:31:12.509 INFO:tasks.workunit.client.1.smithi066.stdout:8/100: dwrite d0/d6/fe [0,4194304] 0 2024-02-28T16:31:12.511 INFO:tasks.workunit.client.1.smithi066.stdout:1/209: rename d2/c30 to d2/d27/d2b/d34/c42 0 2024-02-28T16:31:12.514 INFO:tasks.workunit.client.1.smithi066.stdout:2/177: mkdir d0/d12/d3e 0 2024-02-28T16:31:12.517 INFO:tasks.workunit.client.1.smithi066.stdout:0/244: dwrite d2/d9/f20 [0,4194304] 0 2024-02-28T16:31:12.517 INFO:tasks.workunit.client.1.smithi066.stdout:2/178: stat d0/d12/d1a/d23/d29 0 2024-02-28T16:31:12.517 INFO:tasks.workunit.client.0.smithi001.stdout:6/541: chown d9/db/d5f/d62/d83/f8f 52963 1 2024-02-28T16:31:12.518 INFO:tasks.workunit.client.1.smithi066.stdout:4/233: sync 2024-02-28T16:31:12.524 INFO:tasks.workunit.client.1.smithi066.stdout:2/179: truncate d0/d12/d1a/d23/f28 742724 0 2024-02-28T16:31:12.524 INFO:tasks.workunit.client.1.smithi066.stdout:4/234: write d1/f33 [2425370,99700] 0 2024-02-28T16:31:12.525 INFO:tasks.workunit.client.0.smithi001.stdout:3/597: mkdir d4/d6/d7/dc6 0 2024-02-28T16:31:12.525 INFO:tasks.workunit.client.0.smithi001.stdout:2/596: write d0/d2/d3/de/d8a/fc9 [1934557,104938] 0 2024-02-28T16:31:12.526 INFO:tasks.workunit.client.0.smithi001.stdout:9/692: write d5/d45/d6f/dc8/fac [63209,9468] 0 2024-02-28T16:31:12.527 INFO:tasks.workunit.client.1.smithi066.stdout:6/101: dread d2/f12 [0,4194304] 0 2024-02-28T16:31:12.529 INFO:tasks.workunit.client.0.smithi001.stdout:1/687: write d0/d9/d1a/d2b/d26/d27/d30/d67/db7/fe1 [1049652,80026] 0 2024-02-28T16:31:12.529 INFO:tasks.workunit.client.1.smithi066.stdout:2/180: write d0/f30 [958152,112974] 0 2024-02-28T16:31:12.532 INFO:tasks.workunit.client.1.smithi066.stdout:8/101: truncate d0/d6/fe 1736660 0 2024-02-28T16:31:12.533 INFO:tasks.workunit.client.0.smithi001.stdout:3/598: dread - d4/d38/fbc zero size 2024-02-28T16:31:12.534 INFO:tasks.workunit.client.0.smithi001.stdout:6/542: fsync d9/d29/d6e/f34 0 2024-02-28T16:31:12.535 INFO:tasks.workunit.client.0.smithi001.stdout:9/693: readlink d5/d27/l8c 0 2024-02-28T16:31:12.537 INFO:tasks.workunit.client.1.smithi066.stdout:6/102: creat d2/da/f1c x:0 0 0 2024-02-28T16:31:12.539 INFO:tasks.workunit.client.1.smithi066.stdout:8/102: mkdir d0/d6/dc/d16/d1c 0 2024-02-28T16:31:12.540 INFO:tasks.workunit.client.0.smithi001.stdout:2/597: chown d0/f33 181 1 2024-02-28T16:31:12.540 INFO:tasks.workunit.client.0.smithi001.stdout:0/569: dwrite d18/d31/f9e [4194304,4194304] 0 2024-02-28T16:31:12.541 INFO:tasks.workunit.client.1.smithi066.stdout:6/103: readlink d2/lb 0 2024-02-28T16:31:12.543 INFO:tasks.workunit.client.0.smithi001.stdout:5/544: dread dc/d12/d56/f90 [0,4194304] 0 2024-02-28T16:31:12.544 INFO:tasks.workunit.client.1.smithi066.stdout:8/103: write d0/f4 [3324975,76299] 0 2024-02-28T16:31:12.544 INFO:tasks.workunit.client.1.smithi066.stdout:2/181: creat d0/d12/d1a/d23/d29/d21/d2f/f3f x:0 0 0 2024-02-28T16:31:12.547 INFO:tasks.workunit.client.1.smithi066.stdout:2/182: chown d0/d12/f36 2848862 1 2024-02-28T16:31:12.548 INFO:tasks.workunit.client.0.smithi001.stdout:8/597: dread d4/d7/d22/d27/d58/d62/f7e [0,4194304] 0 2024-02-28T16:31:12.549 INFO:tasks.workunit.client.1.smithi066.stdout:6/104: dread - d2/da/f1b zero size 2024-02-28T16:31:12.549 INFO:tasks.workunit.client.1.smithi066.stdout:0/245: dwrite d2/d9/f40 [0,4194304] 0 2024-02-28T16:31:12.550 INFO:tasks.workunit.client.0.smithi001.stdout:5/545: write dc/d1a/d21/d2d/d3c/d4a/f85 [1616929,50215] 0 2024-02-28T16:31:12.551 INFO:tasks.workunit.client.0.smithi001.stdout:1/688: link d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/cda d0/d9/d1a/d72/d9d/cf3 0 2024-02-28T16:31:12.553 INFO:tasks.workunit.client.0.smithi001.stdout:2/598: truncate d0/d2/d3/d73/f9b 5242027 0 2024-02-28T16:31:12.556 INFO:tasks.workunit.client.0.smithi001.stdout:8/598: symlink d4/d32/d93/lc2 0 2024-02-28T16:31:12.559 INFO:tasks.workunit.client.1.smithi066.stdout:2/183: chown d0/d12/d1a/d23/d29/c33 25868334 1 2024-02-28T16:31:12.559 INFO:tasks.workunit.client.1.smithi066.stdout:4/235: dread d1/f4 [0,4194304] 0 2024-02-28T16:31:12.559 INFO:tasks.workunit.client.0.smithi001.stdout:4/614: truncate d3/d2d/f63 1785866 0 2024-02-28T16:31:12.562 INFO:tasks.workunit.client.1.smithi066.stdout:6/105: rmdir d2/da 39 2024-02-28T16:31:12.563 INFO:tasks.workunit.client.0.smithi001.stdout:9/694: write d5/fb [662942,53281] 0 2024-02-28T16:31:12.563 INFO:tasks.workunit.client.1.smithi066.stdout:0/246: mkdir d2/d9/d10/d50/d52 0 2024-02-28T16:31:12.564 INFO:tasks.workunit.client.1.smithi066.stdout:2/184: write d0/f37 [1388224,93997] 0 2024-02-28T16:31:12.565 INFO:tasks.workunit.client.0.smithi001.stdout:1/689: mknod d0/d9/d1a/d72/d9d/d9c/cf4 0 2024-02-28T16:31:12.565 INFO:tasks.workunit.client.1.smithi066.stdout:6/106: chown d2/f10 6275580 1 2024-02-28T16:31:12.567 INFO:tasks.workunit.client.0.smithi001.stdout:7/571: sync 2024-02-28T16:31:12.567 INFO:tasks.workunit.client.0.smithi001.stdout:4/615: truncate d3/d5/faa 1426042 0 2024-02-28T16:31:12.572 INFO:tasks.workunit.client.0.smithi001.stdout:9/695: creat d5/d27/d34/d50/fea x:0 0 0 2024-02-28T16:31:12.574 INFO:tasks.workunit.client.1.smithi066.stdout:2/185: mkdir d0/d12/d1a/d39/d40 0 2024-02-28T16:31:12.575 INFO:tasks.workunit.client.0.smithi001.stdout:4/616: read d3/d5/d7/d1f/d2e/f39 [4056522,36057] 0 2024-02-28T16:31:12.578 INFO:tasks.workunit.client.0.smithi001.stdout:7/572: mknod d3/cb5 0 2024-02-28T16:31:12.581 INFO:tasks.workunit.client.0.smithi001.stdout:2/599: getdents d0/d2/d3/de/d61/d12/d6a 0 2024-02-28T16:31:12.583 INFO:tasks.workunit.client.0.smithi001.stdout:7/573: stat d3/d1c/f33 0 2024-02-28T16:31:12.585 INFO:tasks.workunit.client.0.smithi001.stdout:4/617: unlink f2 0 2024-02-28T16:31:12.586 INFO:tasks.workunit.client.1.smithi066.stdout:0/247: getdents d2/d9/d10/d50 0 2024-02-28T16:31:12.587 INFO:tasks.workunit.client.0.smithi001.stdout:5/546: dwrite dc/d1a/d21/d2d/d3c/f53 [4194304,4194304] 0 2024-02-28T16:31:12.587 INFO:tasks.workunit.client.1.smithi066.stdout:8/104: truncate d0/d6/dc/f14 3475713 0 2024-02-28T16:31:12.588 INFO:tasks.workunit.client.1.smithi066.stdout:6/107: dwrite f1 [4194304,4194304] 0 2024-02-28T16:31:12.589 INFO:tasks.workunit.client.0.smithi001.stdout:9/696: write d5/d27/db1/d33/d8a/fcd [723271,37036] 0 2024-02-28T16:31:12.594 INFO:tasks.workunit.client.1.smithi066.stdout:4/236: dwrite d1/f5 [0,4194304] 0 2024-02-28T16:31:12.596 INFO:tasks.workunit.client.0.smithi001.stdout:4/618: fdatasync d3/d1c/d58/d6e/f7a 0 2024-02-28T16:31:12.598 INFO:tasks.workunit.client.1.smithi066.stdout:8/105: write d0/f13 [647199,21627] 0 2024-02-28T16:31:12.601 INFO:tasks.workunit.client.0.smithi001.stdout:7/574: mkdir d3/d17/d3d/d5d/d9a/db6 0 2024-02-28T16:31:12.602 INFO:tasks.workunit.client.0.smithi001.stdout:0/570: truncate d18/d22/d47/fa2 458687 0 2024-02-28T16:31:12.602 INFO:tasks.workunit.client.0.smithi001.stdout:4/619: chown d3/d1c/d58/f98 899908 1 2024-02-28T16:31:12.602 INFO:tasks.workunit.client.1.smithi066.stdout:0/248: creat d2/d9/d10/d50/d51/f53 x:0 0 0 2024-02-28T16:31:12.603 INFO:tasks.workunit.client.0.smithi001.stdout:6/543: dread d9/db/f13 [0,4194304] 0 2024-02-28T16:31:12.604 INFO:tasks.workunit.client.0.smithi001.stdout:1/690: write d0/d9/d1a/d2b/fba [1510977,72478] 0 2024-02-28T16:31:12.604 INFO:tasks.workunit.client.0.smithi001.stdout:2/600: unlink d0/d2/d3/de/d8a/f2e 0 2024-02-28T16:31:12.605 INFO:tasks.workunit.client.1.smithi066.stdout:6/108: creat d2/da/d19/f1d x:0 0 0 2024-02-28T16:31:12.605 INFO:tasks.workunit.client.0.smithi001.stdout:5/547: symlink dc/lb5 0 2024-02-28T16:31:12.608 INFO:tasks.workunit.client.0.smithi001.stdout:5/548: fdatasync dc/d1a/d7d/d87/fa8 0 2024-02-28T16:31:12.610 INFO:tasks.workunit.client.1.smithi066.stdout:4/237: creat d1/f4d x:0 0 0 2024-02-28T16:31:12.611 INFO:tasks.workunit.client.1.smithi066.stdout:2/186: dwrite d0/d12/d1a/d23/d29/d21/d2f/f3f [0,4194304] 0 2024-02-28T16:31:12.612 INFO:tasks.workunit.client.0.smithi001.stdout:3/599: dread d4/d1e/f33 [0,4194304] 0 2024-02-28T16:31:12.612 INFO:tasks.workunit.client.0.smithi001.stdout:5/549: stat dc/d1a/d7d/d87/f8f 0 2024-02-28T16:31:12.620 INFO:tasks.workunit.client.1.smithi066.stdout:4/238: mkdir d1/d18/d4e 0 2024-02-28T16:31:12.621 INFO:tasks.workunit.client.0.smithi001.stdout:1/691: dread - d0/d9/f61 zero size 2024-02-28T16:31:12.622 INFO:tasks.workunit.client.1.smithi066.stdout:2/187: creat d0/d12/d1a/d39/d40/f41 x:0 0 0 2024-02-28T16:31:12.624 INFO:tasks.workunit.client.0.smithi001.stdout:6/544: rename d9 to d9/d29/db8 22 2024-02-28T16:31:12.624 INFO:tasks.workunit.client.1.smithi066.stdout:8/106: dread d0/d6/ff [0,4194304] 0 2024-02-28T16:31:12.624 INFO:tasks.workunit.client.1.smithi066.stdout:4/239: write d1/d18/f3a [296420,86419] 0 2024-02-28T16:31:12.625 INFO:tasks.workunit.client.0.smithi001.stdout:3/600: unlink d4/d1e/d7c/d7d/l89 0 2024-02-28T16:31:12.628 INFO:tasks.workunit.client.1.smithi066.stdout:8/107: write d0/d6/f7 [559938,62126] 0 2024-02-28T16:31:12.629 INFO:tasks.workunit.client.1.smithi066.stdout:0/249: dread d2/d9/d10/f46 [0,4194304] 0 2024-02-28T16:31:12.631 INFO:tasks.workunit.client.0.smithi001.stdout:3/601: chown d4/d6/c42 2066547303 1 2024-02-28T16:31:12.631 INFO:tasks.workunit.client.0.smithi001.stdout:1/692: readlink d0/d9/d1a/d2b/d26/d27/lc3 0 2024-02-28T16:31:12.635 INFO:tasks.workunit.client.0.smithi001.stdout:5/550: link dc/f5c dc/d1a/d21/d2d/fb6 0 2024-02-28T16:31:12.635 INFO:tasks.workunit.client.0.smithi001.stdout:8/599: dread d4/d7/f10 [0,4194304] 0 2024-02-28T16:31:12.636 INFO:tasks.workunit.client.0.smithi001.stdout:1/693: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d67/dd6/fea 0 2024-02-28T16:31:12.638 INFO:tasks.workunit.client.0.smithi001.stdout:0/571: rename d18/d31/d4f/f82 to d18/d22/d3d/fba 0 2024-02-28T16:31:12.639 INFO:tasks.workunit.client.0.smithi001.stdout:9/697: write d5/d27/d34/f8d [925764,49733] 0 2024-02-28T16:31:12.640 INFO:tasks.workunit.client.0.smithi001.stdout:1/694: stat d0/l41 0 2024-02-28T16:31:12.644 INFO:tasks.workunit.client.0.smithi001.stdout:8/600: symlink d4/d7/d22/d27/d58/d94/lc3 0 2024-02-28T16:31:12.645 INFO:tasks.workunit.client.0.smithi001.stdout:9/698: chown d5/d45/d6f/fc6 5 1 2024-02-28T16:31:12.646 INFO:tasks.workunit.client.0.smithi001.stdout:5/551: mkdir dc/d12/d41/db7 0 2024-02-28T16:31:12.649 INFO:tasks.workunit.client.1.smithi066.stdout:0/250: mknod d2/d9/d10/d50/c54 0 2024-02-28T16:31:12.652 INFO:tasks.workunit.client.0.smithi001.stdout:6/545: rename f1 to d9/da6/fb9 0 2024-02-28T16:31:12.652 INFO:tasks.workunit.client.1.smithi066.stdout:6/109: write d2/da/fc [4772285,33633] 0 2024-02-28T16:31:12.652 INFO:tasks.workunit.client.0.smithi001.stdout:5/552: chown dc/d1a/d21/d2d/d3c/d4a/f59 3255995 1 2024-02-28T16:31:12.655 INFO:tasks.workunit.client.1.smithi066.stdout:2/188: write d0/d12/f36 [331338,39340] 0 2024-02-28T16:31:12.658 INFO:tasks.workunit.client.0.smithi001.stdout:0/572: mknod d18/d22/d2b/cbb 0 2024-02-28T16:31:12.658 INFO:tasks.workunit.client.0.smithi001.stdout:0/573: chown c8 1936123 1 2024-02-28T16:31:12.661 INFO:tasks.workunit.client.1.smithi066.stdout:7/101: sync 2024-02-28T16:31:12.661 INFO:tasks.workunit.client.0.smithi001.stdout:1/695: mkdir d0/d9/d1a/d72/d9d/df5 0 2024-02-28T16:31:12.662 INFO:tasks.workunit.client.1.smithi066.stdout:9/141: sync 2024-02-28T16:31:12.664 INFO:tasks.workunit.client.1.smithi066.stdout:7/102: dread - dc/f16 zero size 2024-02-28T16:31:12.667 INFO:tasks.workunit.client.1.smithi066.stdout:6/110: getdents d2/da/d19 0 2024-02-28T16:31:12.670 INFO:tasks.workunit.client.1.smithi066.stdout:2/189: rename d0/d12/d1a/d23/d29/d21/d2f/f3f to d0/d12/d1a/d39/f42 0 2024-02-28T16:31:12.671 INFO:tasks.workunit.client.0.smithi001.stdout:7/575: dread d3/f36 [4194304,4194304] 0 2024-02-28T16:31:12.673 INFO:tasks.workunit.client.0.smithi001.stdout:3/602: rename d4/d18/f6d to d4/d6/d7/d78/d81/d8a/fc7 0 2024-02-28T16:31:12.673 INFO:tasks.workunit.client.1.smithi066.stdout:8/108: dwrite d0/d6/ff [4194304,4194304] 0 2024-02-28T16:31:12.675 INFO:tasks.workunit.client.0.smithi001.stdout:1/696: dread - d0/d9/d1a/d2b/d26/d27/d30/d67/dd6/fdc zero size 2024-02-28T16:31:12.677 INFO:tasks.workunit.client.1.smithi066.stdout:2/190: write d0/ff [2825862,57885] 0 2024-02-28T16:31:12.678 INFO:tasks.workunit.client.1.smithi066.stdout:6/111: creat d2/da/f1e x:0 0 0 2024-02-28T16:31:12.680 INFO:tasks.workunit.client.0.smithi001.stdout:9/699: dwrite d5/d27/db1/d33/d8a/fcd [0,4194304] 0 2024-02-28T16:31:12.681 INFO:tasks.workunit.client.0.smithi001.stdout:3/603: write d4/d6/d7/d23/fa9 [420637,124107] 0 2024-02-28T16:31:12.682 INFO:tasks.workunit.client.1.smithi066.stdout:0/251: dwrite d2/d9/d10/f39 [0,4194304] 0 2024-02-28T16:31:12.684 INFO:tasks.workunit.client.1.smithi066.stdout:2/191: mknod d0/d12/c43 0 2024-02-28T16:31:12.688 INFO:tasks.workunit.client.1.smithi066.stdout:2/192: write d0/f9 [2298908,86933] 0 2024-02-28T16:31:12.690 INFO:tasks.workunit.client.0.smithi001.stdout:6/546: symlink d9/lba 0 2024-02-28T16:31:12.691 INFO:tasks.workunit.client.1.smithi066.stdout:9/142: dread f3 [0,4194304] 0 2024-02-28T16:31:12.693 INFO:tasks.workunit.client.1.smithi066.stdout:9/143: read - da/f2a zero size 2024-02-28T16:31:12.694 INFO:tasks.workunit.client.1.smithi066.stdout:2/193: unlink d0/d12/d1a/d23/d29/d21/l25 0 2024-02-28T16:31:12.699 INFO:tasks.workunit.client.1.smithi066.stdout:4/240: truncate d1/f7 1155502 0 2024-02-28T16:31:12.701 INFO:tasks.workunit.client.1.smithi066.stdout:9/144: chown da/d10/f13 2 1 2024-02-28T16:31:12.705 INFO:tasks.workunit.client.1.smithi066.stdout:3/160: sync 2024-02-28T16:31:12.705 INFO:tasks.workunit.client.1.smithi066.stdout:5/127: sync 2024-02-28T16:31:12.707 INFO:tasks.workunit.client.1.smithi066.stdout:6/112: dread d2/f10 [0,4194304] 0 2024-02-28T16:31:12.708 INFO:tasks.workunit.client.1.smithi066.stdout:8/109: dwrite d0/d6/fe [0,4194304] 0 2024-02-28T16:31:12.709 INFO:tasks.workunit.client.1.smithi066.stdout:9/145: mkdir da/df/d18/d2d 0 2024-02-28T16:31:12.712 INFO:tasks.workunit.client.1.smithi066.stdout:2/194: truncate d0/ff 5211847 0 2024-02-28T16:31:12.717 INFO:tasks.workunit.client.1.smithi066.stdout:3/161: mknod db/dc/c32 0 2024-02-28T16:31:12.719 INFO:tasks.workunit.client.1.smithi066.stdout:6/113: fsync d2/da/d19/f1d 0 2024-02-28T16:31:12.736 INFO:tasks.workunit.client.1.smithi066.stdout:3/162: mkdir db/d13/d1a/d33 0 2024-02-28T16:31:12.743 INFO:tasks.workunit.client.0.smithi001.stdout:5/553: mkdir dc/d12/d56/d96/db8 0 2024-02-28T16:31:12.745 INFO:tasks.workunit.client.0.smithi001.stdout:6/547: chown d9/d29/d6e/d9a/fad 16367220 1 2024-02-28T16:31:12.745 INFO:tasks.workunit.client.0.smithi001.stdout:3/604: rename d4/d38/f41 to d4/d1e/fc8 0 2024-02-28T16:31:12.749 INFO:tasks.workunit.client.1.smithi066.stdout:1/210: sync 2024-02-28T16:31:12.749 INFO:tasks.workunit.client.1.smithi066.stdout:3/163: chown db/l2c 15 1 2024-02-28T16:31:12.751 INFO:tasks.workunit.client.0.smithi001.stdout:8/601: link d4/d7/la d4/d32/d69/d9a/lc4 0 2024-02-28T16:31:12.751 INFO:tasks.workunit.client.0.smithi001.stdout:5/554: mkdir dc/d12/d56/d96/db9 0 2024-02-28T16:31:12.753 INFO:tasks.workunit.client.1.smithi066.stdout:8/110: getdents d0 0 2024-02-28T16:31:12.754 INFO:tasks.workunit.client.0.smithi001.stdout:0/574: dread d18/d22/d2b/f37 [0,4194304] 0 2024-02-28T16:31:12.755 INFO:tasks.workunit.client.0.smithi001.stdout:8/602: write d4/d32/f7c [5276557,35804] 0 2024-02-28T16:31:12.755 INFO:tasks.workunit.client.0.smithi001.stdout:3/605: chown d4/l16 433 1 2024-02-28T16:31:12.755 INFO:tasks.workunit.client.0.smithi001.stdout:5/555: dread - dc/d1a/d7d/d87/f8f zero size 2024-02-28T16:31:12.757 INFO:tasks.workunit.client.1.smithi066.stdout:7/103: dwrite f5 [0,4194304] 0 2024-02-28T16:31:12.759 INFO:tasks.workunit.client.0.smithi001.stdout:0/575: chown d18/d22/d76/f8b 1115914831 1 2024-02-28T16:31:12.759 INFO:tasks.workunit.client.0.smithi001.stdout:3/606: write d4/d6/f87 [1879746,117315] 0 2024-02-28T16:31:12.760 INFO:tasks.workunit.client.0.smithi001.stdout:6/548: creat d9/db/d5f/d89/d8a/fbb x:0 0 0 2024-02-28T16:31:12.760 INFO:tasks.workunit.client.1.smithi066.stdout:4/241: dwrite d1/f46 [0,4194304] 0 2024-02-28T16:31:12.763 INFO:tasks.workunit.client.0.smithi001.stdout:3/607: chown d4/d1e/f25 888817 1 2024-02-28T16:31:12.766 INFO:tasks.workunit.client.1.smithi066.stdout:3/164: dread db/d13/f23 [0,4194304] 0 2024-02-28T16:31:12.767 INFO:tasks.workunit.client.0.smithi001.stdout:9/700: truncate d5/d45/d6f/dc8/faf 2261912 0 2024-02-28T16:31:12.770 INFO:tasks.workunit.client.1.smithi066.stdout:7/104: symlink dc/l1f 0 2024-02-28T16:31:12.771 INFO:tasks.workunit.client.1.smithi066.stdout:0/252: dwrite d2/fe [0,4194304] 0 2024-02-28T16:31:12.776 INFO:tasks.workunit.client.0.smithi001.stdout:5/556: mkdir dc/d1a/dba 0 2024-02-28T16:31:12.779 INFO:tasks.workunit.client.0.smithi001.stdout:1/697: dread d0/d9/d1a/d2b/d26/d27/d30/f58 [0,4194304] 0 2024-02-28T16:31:12.780 INFO:tasks.workunit.client.0.smithi001.stdout:6/549: creat d9/fbc x:0 0 0 2024-02-28T16:31:12.784 INFO:tasks.workunit.client.0.smithi001.stdout:0/576: mkdir d18/d31/dbc 0 2024-02-28T16:31:12.786 INFO:tasks.workunit.client.0.smithi001.stdout:4/620: dwrite d3/d2d/f63 [0,4194304] 0 2024-02-28T16:31:12.788 INFO:tasks.workunit.client.1.smithi066.stdout:2/195: dwrite d0/d12/d1a/d23/f28 [0,4194304] 0 2024-02-28T16:31:12.788 INFO:tasks.workunit.client.1.smithi066.stdout:3/165: unlink l7 0 2024-02-28T16:31:12.789 INFO:tasks.workunit.client.1.smithi066.stdout:0/253: dread - d2/d9/f44 zero size 2024-02-28T16:31:12.790 INFO:tasks.workunit.client.0.smithi001.stdout:5/557: mknod dc/d12/d56/cbb 0 2024-02-28T16:31:12.793 INFO:tasks.workunit.client.1.smithi066.stdout:4/242: creat d1/f4f x:0 0 0 2024-02-28T16:31:12.795 INFO:tasks.workunit.client.1.smithi066.stdout:6/114: dwrite d2/da/f1a [0,4194304] 0 2024-02-28T16:31:12.799 INFO:tasks.workunit.client.1.smithi066.stdout:7/105: creat dc/de/d17/d1e/f20 x:0 0 0 2024-02-28T16:31:12.799 INFO:tasks.workunit.client.0.smithi001.stdout:9/701: symlink d5/d27/db1/d33/d8a/de0/leb 0 2024-02-28T16:31:12.803 INFO:tasks.workunit.client.1.smithi066.stdout:9/146: dwrite da/d10/f13 [0,4194304] 0 2024-02-28T16:31:12.805 INFO:tasks.workunit.client.1.smithi066.stdout:2/196: write d0/d12/d1a/d23/f28 [3176048,99866] 0 2024-02-28T16:31:12.805 INFO:tasks.workunit.client.0.smithi001.stdout:1/698: truncate d0/d9/d56/f7f 1047088 0 2024-02-28T16:31:12.807 INFO:tasks.workunit.client.0.smithi001.stdout:1/699: readlink d0/d9/l1c 0 2024-02-28T16:31:12.808 INFO:tasks.workunit.client.1.smithi066.stdout:0/254: write d2/d9/f11 [876976,117797] 0 2024-02-28T16:31:12.809 INFO:tasks.workunit.client.1.smithi066.stdout:3/166: write db/dc/d11/f22 [804343,17568] 0 2024-02-28T16:31:12.810 INFO:tasks.workunit.client.0.smithi001.stdout:8/603: write d4/d7/f2b [6672707,8899] 0 2024-02-28T16:31:12.813 INFO:tasks.workunit.client.0.smithi001.stdout:4/621: write d3/f3b [1512239,5507] 0 2024-02-28T16:31:12.813 INFO:tasks.workunit.client.1.smithi066.stdout:0/255: chown d2/d9/c45 6745764 1 2024-02-28T16:31:12.813 INFO:tasks.workunit.client.1.smithi066.stdout:5/128: dwrite d3/ff [0,4194304] 0 2024-02-28T16:31:12.817 INFO:tasks.workunit.client.1.smithi066.stdout:0/256: read d2/d9/d10/f2c [1836874,120946] 0 2024-02-28T16:31:12.817 INFO:tasks.workunit.client.1.smithi066.stdout:5/129: dread - d3/d4/dc/f19 zero size 2024-02-28T16:31:12.822 INFO:tasks.workunit.client.1.smithi066.stdout:8/111: dread d0/d6/dc/f14 [0,4194304] 0 2024-02-28T16:31:12.824 INFO:tasks.workunit.client.0.smithi001.stdout:1/700: mkdir d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6 0 2024-02-28T16:31:12.825 INFO:tasks.workunit.client.0.smithi001.stdout:6/550: dread d9/f8c [0,4194304] 0 2024-02-28T16:31:12.825 INFO:tasks.workunit.client.0.smithi001.stdout:5/558: mknod dc/cbc 0 2024-02-28T16:31:12.825 INFO:tasks.workunit.client.0.smithi001.stdout:4/622: rmdir d3/d5/d7/d1f 39 2024-02-28T16:31:12.826 INFO:tasks.workunit.client.0.smithi001.stdout:1/701: dread - d0/fa9 zero size 2024-02-28T16:31:12.826 INFO:tasks.workunit.client.0.smithi001.stdout:2/601: sync 2024-02-28T16:31:12.826 INFO:tasks.workunit.client.1.smithi066.stdout:6/115: symlink d2/l1f 0 2024-02-28T16:31:12.826 INFO:tasks.workunit.client.1.smithi066.stdout:2/197: creat d0/f44 x:0 0 0 2024-02-28T16:31:12.835 INFO:tasks.workunit.client.1.smithi066.stdout:0/257: unlink d2/d9/f44 0 2024-02-28T16:31:12.836 INFO:tasks.workunit.client.0.smithi001.stdout:4/623: creat d3/d1c/fc2 x:0 0 0 2024-02-28T16:31:12.838 INFO:tasks.workunit.client.1.smithi066.stdout:5/130: creat d3/d4/d17/d18/f2b x:0 0 0 2024-02-28T16:31:12.839 INFO:tasks.workunit.client.0.smithi001.stdout:5/559: truncate dc/d12/d56/f90 1267335 0 2024-02-28T16:31:12.839 INFO:tasks.workunit.client.0.smithi001.stdout:9/702: dread d5/d27/db1/da6/db2/fb6 [0,4194304] 0 2024-02-28T16:31:12.841 INFO:tasks.workunit.client.1.smithi066.stdout:4/243: dwrite d1/d18/d41/f47 [0,4194304] 0 2024-02-28T16:31:12.842 INFO:tasks.workunit.client.0.smithi001.stdout:6/551: mkdir d9/de/dbd 0 2024-02-28T16:31:12.844 INFO:tasks.workunit.client.0.smithi001.stdout:5/560: write dc/d12/f6f [726148,57258] 0 2024-02-28T16:31:12.845 INFO:tasks.workunit.client.1.smithi066.stdout:5/131: chown d3/d4/dc/f21 1016841786 1 2024-02-28T16:31:12.847 INFO:tasks.workunit.client.1.smithi066.stdout:3/167: dwrite db/d13/d1a/f27 [0,4194304] 0 2024-02-28T16:31:12.848 INFO:tasks.workunit.client.0.smithi001.stdout:9/703: write d5/d27/db1/d33/f5d [60734,15443] 0 2024-02-28T16:31:12.849 INFO:tasks.workunit.client.0.smithi001.stdout:1/702: creat d0/ff7 x:0 0 0 2024-02-28T16:31:12.850 INFO:tasks.workunit.client.0.smithi001.stdout:6/552: read - d9/db/d5f/d89/d8a/fbb zero size 2024-02-28T16:31:12.851 INFO:tasks.workunit.client.1.smithi066.stdout:4/244: creat d1/d18/d4e/f50 x:0 0 0 2024-02-28T16:31:12.852 INFO:tasks.workunit.client.0.smithi001.stdout:8/604: dwrite d4/d7/d22/d27/d58/f75 [0,4194304] 0 2024-02-28T16:31:12.853 INFO:tasks.workunit.client.1.smithi066.stdout:5/132: fsync d3/d4/dc/f19 0 2024-02-28T16:31:12.855 INFO:tasks.workunit.client.0.smithi001.stdout:4/624: dread d3/d5/d7/d1f/d2e/f64 [0,4194304] 0 2024-02-28T16:31:12.856 INFO:tasks.workunit.client.0.smithi001.stdout:9/704: unlink d5/f89 0 2024-02-28T16:31:12.857 INFO:tasks.workunit.client.0.smithi001.stdout:4/625: readlink d3/d5/d4e/l55 0 2024-02-28T16:31:12.860 INFO:tasks.workunit.client.1.smithi066.stdout:5/133: stat d3/d4/d17/d18/f1a 0 2024-02-28T16:31:12.861 INFO:tasks.workunit.client.1.smithi066.stdout:3/168: rename db/d13/d1a/f27 to db/d13/d14/d24/f34 0 2024-02-28T16:31:12.861 INFO:tasks.workunit.client.0.smithi001.stdout:2/602: dread d0/d2/d3/de/d61/f1f [0,4194304] 0 2024-02-28T16:31:12.861 INFO:tasks.workunit.client.0.smithi001.stdout:0/577: dwrite d18/d22/d57/d7e/f96 [0,4194304] 0 2024-02-28T16:31:12.866 INFO:tasks.workunit.client.1.smithi066.stdout:8/112: dwrite d0/d6/ff [4194304,4194304] 0 2024-02-28T16:31:12.866 INFO:tasks.workunit.client.0.smithi001.stdout:5/561: creat dc/d12/d56/d96/db8/fbd x:0 0 0 2024-02-28T16:31:12.867 INFO:tasks.workunit.client.0.smithi001.stdout:6/553: rename d9/de/f66 to d9/d29/d6a/fbe 0 2024-02-28T16:31:12.868 INFO:tasks.workunit.client.1.smithi066.stdout:3/169: stat db/dc/d11/d1f 0 2024-02-28T16:31:12.868 INFO:tasks.workunit.client.1.smithi066.stdout:5/134: fsync d3/d4/dc/f1e 0 2024-02-28T16:31:12.869 INFO:tasks.workunit.client.0.smithi001.stdout:1/703: mkdir d0/d9/d1a/d2b/df8 0 2024-02-28T16:31:12.870 INFO:tasks.workunit.client.0.smithi001.stdout:5/562: write dc/d1a/f7b [1632452,123709] 0 2024-02-28T16:31:12.871 INFO:tasks.workunit.client.1.smithi066.stdout:4/245: dread d1/d18/f3a [0,4194304] 0 2024-02-28T16:31:12.871 INFO:tasks.workunit.client.1.smithi066.stdout:0/258: dwrite d2/d9/f1e [0,4194304] 0 2024-02-28T16:31:12.875 INFO:tasks.workunit.client.0.smithi001.stdout:6/554: write d9/db/d5f/d62/f71 [433174,11901] 0 2024-02-28T16:31:12.875 INFO:tasks.workunit.client.1.smithi066.stdout:0/259: readlink l1 0 2024-02-28T16:31:12.878 INFO:tasks.workunit.client.1.smithi066.stdout:3/170: mkdir db/dc/d35 0 2024-02-28T16:31:12.878 INFO:tasks.workunit.client.0.smithi001.stdout:4/626: rmdir d3/d5/d7/d1f 39 2024-02-28T16:31:12.881 INFO:tasks.workunit.client.0.smithi001.stdout:1/704: write d0/d9/d1a/d2b/d26/d27/d30/d52/fd0 [345601,97161] 0 2024-02-28T16:31:12.882 INFO:tasks.workunit.client.1.smithi066.stdout:5/135: rename d3/ff to d3/d4/d17/d18/f2c 0 2024-02-28T16:31:12.885 INFO:tasks.workunit.client.1.smithi066.stdout:4/246: chown d1/d23/c44 61206122 1 2024-02-28T16:31:12.886 INFO:tasks.workunit.client.0.smithi001.stdout:0/578: creat d18/d22/d2b/d36/d4d/fbd x:0 0 0 2024-02-28T16:31:12.895 INFO:tasks.workunit.client.0.smithi001.stdout:2/603: unlink d0/d2/d3/d36/l5d 0 2024-02-28T16:31:12.896 INFO:tasks.workunit.client.1.smithi066.stdout:5/136: creat d3/d1b/f2d x:0 0 0 2024-02-28T16:31:12.900 INFO:tasks.workunit.client.0.smithi001.stdout:8/605: unlink d4/f3b 0 2024-02-28T16:31:12.902 INFO:tasks.workunit.client.1.smithi066.stdout:4/247: link d1/d23/d25/d3e/d40/c4c d1/c51 0 2024-02-28T16:31:12.904 INFO:tasks.workunit.client.1.smithi066.stdout:4/248: readlink d1/l15 0 2024-02-28T16:31:12.906 INFO:tasks.workunit.client.0.smithi001.stdout:5/563: chown dc/d1a/f5f 342387 1 2024-02-28T16:31:12.913 INFO:tasks.workunit.client.1.smithi066.stdout:2/198: getdents d0 0 2024-02-28T16:31:12.913 INFO:tasks.workunit.client.1.smithi066.stdout:7/106: truncate f4 1523691 0 2024-02-28T16:31:12.915 INFO:tasks.workunit.client.0.smithi001.stdout:9/705: creat d5/d27/db1/d33/d8a/da8/ddf/fec x:0 0 0 2024-02-28T16:31:12.916 INFO:tasks.workunit.client.1.smithi066.stdout:7/107: chown dc/de/d17/d1e/f20 1327 1 2024-02-28T16:31:12.919 INFO:tasks.workunit.client.0.smithi001.stdout:0/579: creat d18/d5a/fbe x:0 0 0 2024-02-28T16:31:12.919 INFO:tasks.workunit.client.1.smithi066.stdout:2/199: creat d0/d12/d1a/d39/d40/f45 x:0 0 0 2024-02-28T16:31:12.919 INFO:tasks.workunit.client.1.smithi066.stdout:6/116: truncate f1 2507865 0 2024-02-28T16:31:12.920 INFO:tasks.workunit.client.1.smithi066.stdout:7/108: rename dc/cd to dc/c21 0 2024-02-28T16:31:12.921 INFO:tasks.workunit.client.0.smithi001.stdout:2/604: mknod d0/d2/d3/de/d61/d12/d6a/d84/ccc 0 2024-02-28T16:31:12.921 INFO:tasks.workunit.client.0.smithi001.stdout:0/580: chown d18/d31/d4f/f60 103611802 1 2024-02-28T16:31:12.922 INFO:tasks.workunit.client.0.smithi001.stdout:4/627: unlink d3/d5/d7/d1f/f8d 0 2024-02-28T16:31:12.922 INFO:tasks.workunit.client.1.smithi066.stdout:4/249: dread d1/d18/f2d [0,4194304] 0 2024-02-28T16:31:12.922 INFO:tasks.workunit.client.1.smithi066.stdout:6/117: truncate d2/da/fe 698057 0 2024-02-28T16:31:12.923 INFO:tasks.workunit.client.1.smithi066.stdout:7/109: rmdir dc/de/d17 39 2024-02-28T16:31:12.923 INFO:tasks.workunit.client.1.smithi066.stdout:6/118: readlink d2/lb 0 2024-02-28T16:31:12.926 INFO:tasks.workunit.client.1.smithi066.stdout:4/250: chown d1/f12 0 1 2024-02-28T16:31:12.926 INFO:tasks.workunit.client.0.smithi001.stdout:0/581: stat d18/d22/d2b/d36/d46/f9d 0 2024-02-28T16:31:12.926 INFO:tasks.workunit.client.1.smithi066.stdout:5/137: rename d3/d4/d17 to d3/d4/dc/dd/d2e 0 2024-02-28T16:31:12.928 INFO:tasks.workunit.client.1.smithi066.stdout:4/251: mknod d1/d23/c52 0 2024-02-28T16:31:12.928 INFO:tasks.workunit.client.0.smithi001.stdout:8/606: dwrite d4/d7/da8/d67/fc1 [0,4194304] 0 2024-02-28T16:31:12.930 INFO:tasks.workunit.client.1.smithi066.stdout:5/138: write d3/d4/dc/dd/d2e/d18/d24/f28 [530349,123488] 0 2024-02-28T16:31:12.930 INFO:tasks.workunit.client.1.smithi066.stdout:4/252: write d1/d23/d25/f28 [84124,63541] 0 2024-02-28T16:31:12.932 INFO:tasks.workunit.client.1.smithi066.stdout:4/253: dread - d1/f4f zero size 2024-02-28T16:31:12.932 INFO:tasks.workunit.client.0.smithi001.stdout:4/628: mkdir d3/d5/d7/d1f/dc3 0 2024-02-28T16:31:12.932 INFO:tasks.workunit.client.0.smithi001.stdout:6/555: getdents d9/db/d5f/d62/d83 0 2024-02-28T16:31:12.933 INFO:tasks.workunit.client.1.smithi066.stdout:0/260: write d2/d9/f35 [4166661,68361] 0 2024-02-28T16:31:12.933 INFO:tasks.workunit.client.1.smithi066.stdout:8/113: truncate d0/d6/ff 8055759 0 2024-02-28T16:31:12.933 INFO:tasks.workunit.client.1.smithi066.stdout:3/171: truncate db/ff 1477829 0 2024-02-28T16:31:12.934 INFO:tasks.workunit.client.0.smithi001.stdout:5/564: link dc/d1a/d21/d2d/c9b dc/d1a/d7d/cbe 0 2024-02-28T16:31:12.936 INFO:tasks.workunit.client.1.smithi066.stdout:5/139: readlink d3/l6 0 2024-02-28T16:31:12.937 INFO:tasks.workunit.client.0.smithi001.stdout:8/607: rmdir d4/d7 39 2024-02-28T16:31:12.938 INFO:tasks.workunit.client.0.smithi001.stdout:5/565: fsync dc/d1a/d21/d2d/f44 0 2024-02-28T16:31:12.939 INFO:tasks.workunit.client.1.smithi066.stdout:4/254: truncate d1/d23/f36 1230223 0 2024-02-28T16:31:12.939 INFO:tasks.workunit.client.1.smithi066.stdout:8/114: chown d0/d6/dc/d16/d1c 3897861 1 2024-02-28T16:31:12.939 INFO:tasks.workunit.client.1.smithi066.stdout:8/115: readlink - no filename 2024-02-28T16:31:12.946 INFO:tasks.workunit.client.0.smithi001.stdout:9/706: dwrite d5/d45/d6f/dc8/fe2 [0,4194304] 0 2024-02-28T16:31:12.947 INFO:tasks.workunit.client.0.smithi001.stdout:0/582: getdents d18/d22/d76/d93 0 2024-02-28T16:31:12.948 INFO:tasks.workunit.client.1.smithi066.stdout:8/116: mkdir d0/db/d12/d1d 0 2024-02-28T16:31:12.948 INFO:tasks.workunit.client.0.smithi001.stdout:4/629: truncate d3/d2d/d4c/d69/d82/f77 1002610 0 2024-02-28T16:31:12.949 INFO:tasks.workunit.client.0.smithi001.stdout:6/556: truncate d9/db/d5f/d89/d8a/fbb 642341 0 2024-02-28T16:31:12.951 INFO:tasks.workunit.client.1.smithi066.stdout:5/140: rename d3/d4/dc/dd/d2e/d18/d24 to d3/d4/dc/d2f 0 2024-02-28T16:31:12.953 INFO:tasks.workunit.client.0.smithi001.stdout:0/583: fdatasync d18/d22/d3d/f5e 0 2024-02-28T16:31:12.953 INFO:tasks.workunit.client.1.smithi066.stdout:6/119: dwrite d2/da/d19/f17 [0,4194304] 0 2024-02-28T16:31:12.955 INFO:tasks.workunit.client.0.smithi001.stdout:2/605: dwrite d0/d2/d3/f5 [4194304,4194304] 0 2024-02-28T16:31:12.956 INFO:tasks.workunit.client.0.smithi001.stdout:5/566: rename dc/d12/ca9 to dc/d12/d56/d96/cbf 0 2024-02-28T16:31:12.957 INFO:tasks.workunit.client.0.smithi001.stdout:0/584: stat d18/d22/d3d/fba 0 2024-02-28T16:31:12.959 INFO:tasks.workunit.client.1.smithi066.stdout:3/172: dread db/d13/d1a/f1e [0,4194304] 0 2024-02-28T16:31:12.959 INFO:tasks.workunit.client.0.smithi001.stdout:4/630: unlink d3/d5/d7/d1f/d20/f5d 0 2024-02-28T16:31:12.959 INFO:tasks.workunit.client.1.smithi066.stdout:0/261: dwrite d2/d9/d10/f47 [4194304,4194304] 0 2024-02-28T16:31:12.962 INFO:tasks.workunit.client.1.smithi066.stdout:0/262: stat d2/d9/d10/f2c 0 2024-02-28T16:31:12.963 INFO:tasks.workunit.client.0.smithi001.stdout:7/576: sync 2024-02-28T16:31:12.964 INFO:tasks.workunit.client.0.smithi001.stdout:5/567: fdatasync dc/d1a/d21/d2d/d3c/f4f 0 2024-02-28T16:31:12.964 INFO:tasks.workunit.client.1.smithi066.stdout:0/263: write d2/d9/d10/f47 [8353658,831] 0 2024-02-28T16:31:12.967 INFO:tasks.workunit.client.1.smithi066.stdout:0/264: dread - d2/f32 zero size 2024-02-28T16:31:12.967 INFO:tasks.workunit.client.0.smithi001.stdout:2/606: write d0/d2/d3/de/d8a/f70 [5234897,116732] 0 2024-02-28T16:31:12.967 INFO:tasks.workunit.client.0.smithi001.stdout:6/557: rename d9/l5c to d9/db/d5f/d89/d8a/lbf 0 2024-02-28T16:31:12.973 INFO:tasks.workunit.client.0.smithi001.stdout:0/585: symlink d18/d31/d4f/lbf 0 2024-02-28T16:31:12.974 INFO:tasks.workunit.client.1.smithi066.stdout:6/120: creat d2/da/d19/f20 x:0 0 0 2024-02-28T16:31:12.976 INFO:tasks.workunit.client.0.smithi001.stdout:0/586: truncate d18/d22/d2b/f87 328130 0 2024-02-28T16:31:12.977 INFO:tasks.workunit.client.1.smithi066.stdout:6/121: fsync d2/da/d19/f1d 0 2024-02-28T16:31:12.979 INFO:tasks.workunit.client.0.smithi001.stdout:6/558: chown d9/db/d5f/cb7 46258 1 2024-02-28T16:31:12.981 INFO:tasks.workunit.client.0.smithi001.stdout:2/607: unlink d0/d2/d3/de/d8a/f88 0 2024-02-28T16:31:12.983 INFO:tasks.workunit.client.0.smithi001.stdout:5/568: creat dc/d1a/dba/fc0 x:0 0 0 2024-02-28T16:31:12.985 INFO:tasks.workunit.client.0.smithi001.stdout:1/705: dwrite d0/d9/d1a/d2b/d26/d27/d30/d4a/f89 [0,4194304] 0 2024-02-28T16:31:12.985 INFO:tasks.workunit.client.1.smithi066.stdout:0/265: dread d2/d9/f40 [0,4194304] 0 2024-02-28T16:31:12.985 INFO:tasks.workunit.client.0.smithi001.stdout:7/577: mkdir d3/db7 0 2024-02-28T16:31:12.986 INFO:tasks.workunit.client.1.smithi066.stdout:3/173: unlink db/dc/c32 0 2024-02-28T16:31:12.988 INFO:tasks.workunit.client.0.smithi001.stdout:6/559: chown d9/db/d19/f37 2 1 2024-02-28T16:31:12.989 INFO:tasks.workunit.client.1.smithi066.stdout:3/174: chown db/dc/d11/l16 38952 1 2024-02-28T16:31:12.991 INFO:tasks.workunit.client.1.smithi066.stdout:4/255: getdents d1/d23/d25/d3e/d40 0 2024-02-28T16:31:12.993 INFO:tasks.workunit.client.1.smithi066.stdout:3/175: write db/d13/f19 [1698993,84400] 0 2024-02-28T16:31:12.994 INFO:tasks.workunit.client.0.smithi001.stdout:1/706: write d0/d9/d1a/d2b/d26/d27/d30/d67/db7/fe1 [581119,20054] 0 2024-02-28T16:31:12.994 INFO:tasks.workunit.client.0.smithi001.stdout:6/560: write d9/db/d5f/d62/f71 [690170,37450] 0 2024-02-28T16:31:12.994 INFO:tasks.workunit.client.1.smithi066.stdout:6/122: dread d2/da/f18 [0,4194304] 0 2024-02-28T16:31:12.996 INFO:tasks.workunit.client.0.smithi001.stdout:5/569: creat dc/d1a/d21/d2d/d3c/d4a/fc1 x:0 0 0 2024-02-28T16:31:12.996 INFO:tasks.workunit.client.1.smithi066.stdout:0/266: rename d2/d9/f35 to d2/d9/d10/d50/d52/f55 0 2024-02-28T16:31:12.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:12 smithi066 ceph-mon[36236]: Upgrade: Updating mgr.smithi001.rmthtx 2024-02-28T16:31:12.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:12 smithi066 ceph-mon[36236]: Deploying daemon mgr.smithi001.rmthtx on smithi001 2024-02-28T16:31:12.999 INFO:tasks.workunit.client.1.smithi066.stdout:6/123: chown d2/da/d19/f1d 1306423213 1 2024-02-28T16:31:13.000 INFO:tasks.workunit.client.1.smithi066.stdout:0/267: chown d2/d9/d10/d50/d52 0 1 2024-02-28T16:31:13.000 INFO:tasks.workunit.client.1.smithi066.stdout:3/176: mknod db/d13/d14/d2d/c36 0 2024-02-28T16:31:13.002 INFO:tasks.workunit.client.0.smithi001.stdout:5/570: read dc/d12/f24 [109752,81456] 0 2024-02-28T16:31:13.002 INFO:tasks.workunit.client.0.smithi001.stdout:4/631: dwrite d3/d1c/d58/f46 [0,4194304] 0 2024-02-28T16:31:13.006 INFO:tasks.workunit.client.0.smithi001.stdout:5/571: fsync dc/d1a/d21/d2d/d3c/f47 0 2024-02-28T16:31:13.007 INFO:tasks.workunit.client.1.smithi066.stdout:0/268: mknod d2/d27/d41/c56 0 2024-02-28T16:31:13.009 INFO:tasks.workunit.client.1.smithi066.stdout:8/117: truncate d0/d6/f9 726729 0 2024-02-28T16:31:13.009 INFO:tasks.workunit.client.1.smithi066.stdout:8/118: readlink - no filename 2024-02-28T16:31:13.011 INFO:tasks.workunit.client.0.smithi001.stdout:5/572: stat dc/df/d79/d98/lb4 0 2024-02-28T16:31:13.014 INFO:tasks.workunit.client.1.smithi066.stdout:0/269: dread - d2/d9/d10/d23/f2d zero size 2024-02-28T16:31:13.014 INFO:tasks.workunit.client.0.smithi001.stdout:5/573: chown dc/d1a/d52/f89 0 1 2024-02-28T16:31:13.014 INFO:tasks.workunit.client.0.smithi001.stdout:6/561: rename d9/d29/d6e/d4e/f93 to d9/d29/d6e/d4e/d96/fc0 0 2024-02-28T16:31:13.015 INFO:tasks.workunit.client.1.smithi066.stdout:3/177: link db/dc/f2b db/d2a/f37 0 2024-02-28T16:31:13.017 INFO:tasks.workunit.client.0.smithi001.stdout:5/574: read - dc/d1a/d7d/f88 zero size 2024-02-28T16:31:13.018 INFO:tasks.workunit.client.0.smithi001.stdout:1/707: mknod d0/d9/d1a/d2b/d26/d27/d30/cf9 0 2024-02-28T16:31:13.019 INFO:tasks.workunit.client.1.smithi066.stdout:8/119: write d0/d6/f7 [2674841,36977] 0 2024-02-28T16:31:13.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:12 smithi001 ceph-mon[28991]: Upgrade: Updating mgr.smithi001.rmthtx 2024-02-28T16:31:13.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:12 smithi001 ceph-mon[28991]: Deploying daemon mgr.smithi001.rmthtx on smithi001 2024-02-28T16:31:13.020 INFO:tasks.workunit.client.1.smithi066.stdout:3/178: rename c2 to db/d13/d1a/d33/c38 0 2024-02-28T16:31:13.021 INFO:tasks.workunit.client.0.smithi001.stdout:0/587: dread f11 [0,4194304] 0 2024-02-28T16:31:13.021 INFO:tasks.workunit.client.0.smithi001.stdout:5/575: write dc/fa2 [966937,51721] 0 2024-02-28T16:31:13.021 INFO:tasks.workunit.client.1.smithi066.stdout:0/270: creat d2/d9/d10/d50/f57 x:0 0 0 2024-02-28T16:31:13.025 INFO:tasks.workunit.client.0.smithi001.stdout:5/576: dread - dc/d12/d56/d96/db8/fbd zero size 2024-02-28T16:31:13.026 INFO:tasks.workunit.client.0.smithi001.stdout:0/588: write d18/d22/d2b/d5f/f85 [1312482,42384] 0 2024-02-28T16:31:13.027 INFO:tasks.workunit.client.0.smithi001.stdout:8/608: dread d4/d7/f23 [0,4194304] 0 2024-02-28T16:31:13.027 INFO:tasks.workunit.client.1.smithi066.stdout:8/120: read d0/f4 [1000046,80582] 0 2024-02-28T16:31:13.028 INFO:tasks.workunit.client.0.smithi001.stdout:0/589: readlink d18/d31/l68 0 2024-02-28T16:31:13.029 INFO:tasks.workunit.client.0.smithi001.stdout:1/708: mknod d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/cfa 0 2024-02-28T16:31:13.029 INFO:tasks.workunit.client.1.smithi066.stdout:8/121: stat d0/db/d12/d1d 0 2024-02-28T16:31:13.030 INFO:tasks.workunit.client.1.smithi066.stdout:3/179: read f5 [3193738,62881] 0 2024-02-28T16:31:13.030 INFO:tasks.workunit.client.0.smithi001.stdout:5/577: rmdir dc/df/d79 39 2024-02-28T16:31:13.033 INFO:tasks.workunit.client.0.smithi001.stdout:0/590: mknod d18/d22/d2b/cc0 0 2024-02-28T16:31:13.033 INFO:tasks.workunit.client.1.smithi066.stdout:3/180: creat db/d13/d1a/d33/f39 x:0 0 0 2024-02-28T16:31:13.036 INFO:tasks.workunit.client.0.smithi001.stdout:7/578: truncate d3/d17/d3d/d59/d6e/fac 1999059 0 2024-02-28T16:31:13.036 INFO:tasks.workunit.client.1.smithi066.stdout:0/271: dread d2/d9/f20 [0,4194304] 0 2024-02-28T16:31:13.038 INFO:tasks.workunit.client.0.smithi001.stdout:8/609: read d4/d7/d22/d27/f2f [1894773,113655] 0 2024-02-28T16:31:13.038 INFO:tasks.workunit.client.0.smithi001.stdout:7/579: chown d3/d17/d52/d7f/d99 6535360 1 2024-02-28T16:31:13.040 INFO:tasks.workunit.client.1.smithi066.stdout:8/122: link d0/db/c11 d0/d6/dc/d16/c1e 0 2024-02-28T16:31:13.042 INFO:tasks.workunit.client.0.smithi001.stdout:7/580: chown d3/d1c/c79 543 1 2024-02-28T16:31:13.043 INFO:tasks.workunit.client.1.smithi066.stdout:6/124: truncate d2/da/fe 307683 0 2024-02-28T16:31:13.045 INFO:tasks.workunit.client.1.smithi066.stdout:8/123: creat d0/d6/f1f x:0 0 0 2024-02-28T16:31:13.045 INFO:tasks.workunit.client.0.smithi001.stdout:5/578: dread dc/d1a/d52/f89 [0,4194304] 0 2024-02-28T16:31:13.047 INFO:tasks.workunit.client.0.smithi001.stdout:7/581: creat d3/d17/d52/fb8 x:0 0 0 2024-02-28T16:31:13.048 INFO:tasks.workunit.client.1.smithi066.stdout:4/256: dwrite d1/f7 [0,4194304] 0 2024-02-28T16:31:13.052 INFO:tasks.workunit.client.0.smithi001.stdout:4/632: dwrite d3/d5/d7/d1f/f7d [0,4194304] 0 2024-02-28T16:31:13.052 INFO:tasks.workunit.client.0.smithi001.stdout:7/582: chown d3/d17/d61/f71 59287 1 2024-02-28T16:31:13.059 INFO:tasks.workunit.client.0.smithi001.stdout:4/633: chown d3/d5/d7/d1f/d2e/f39 116 1 2024-02-28T16:31:13.059 INFO:tasks.workunit.client.1.smithi066.stdout:8/124: symlink d0/d6/dc/d16/d1a/l20 0 2024-02-28T16:31:13.062 INFO:tasks.workunit.client.0.smithi001.stdout:5/579: mkdir dc/d12/d56/d96/db9/dc2 0 2024-02-28T16:31:13.062 INFO:tasks.workunit.client.0.smithi001.stdout:1/709: dwrite d0/d9/f85 [0,4194304] 0 2024-02-28T16:31:13.062 INFO:tasks.workunit.client.0.smithi001.stdout:7/583: mkdir d3/d17/d3d/d5d/d9a/d49/d8d/db9 0 2024-02-28T16:31:13.063 INFO:tasks.workunit.client.1.smithi066.stdout:3/181: dwrite db/dc/f2b [0,4194304] 0 2024-02-28T16:31:13.066 INFO:tasks.workunit.client.1.smithi066.stdout:6/125: getdents d2 0 2024-02-28T16:31:13.067 INFO:tasks.workunit.client.0.smithi001.stdout:5/580: creat dc/d12/d56/d96/db8/fc3 x:0 0 0 2024-02-28T16:31:13.072 INFO:tasks.workunit.client.1.smithi066.stdout:6/126: rename d2/f12 to d2/da/f21 0 2024-02-28T16:31:13.074 INFO:tasks.workunit.client.0.smithi001.stdout:8/610: dread d4/d9b/fa7 [0,4194304] 0 2024-02-28T16:31:13.076 INFO:tasks.workunit.client.0.smithi001.stdout:8/611: chown d4/d7/d22/d27/d4b/d77/f83 13 1 2024-02-28T16:31:13.079 INFO:tasks.workunit.client.1.smithi066.stdout:4/257: dwrite d1/f2a [0,4194304] 0 2024-02-28T16:31:13.085 INFO:tasks.workunit.client.0.smithi001.stdout:4/634: creat d3/d2d/fc4 x:0 0 0 2024-02-28T16:31:13.086 INFO:tasks.workunit.client.1.smithi066.stdout:8/125: dwrite d0/f13 [0,4194304] 0 2024-02-28T16:31:13.087 INFO:tasks.workunit.client.0.smithi001.stdout:5/581: mknod dc/df/d79/d98/cc4 0 2024-02-28T16:31:13.091 INFO:tasks.workunit.client.0.smithi001.stdout:7/584: dwrite d3/d17/d3d/d59/f67 [0,4194304] 0 2024-02-28T16:31:13.092 INFO:tasks.workunit.client.0.smithi001.stdout:8/612: dread d4/d32/d93/fa2 [0,4194304] 0 2024-02-28T16:31:13.094 INFO:tasks.workunit.client.1.smithi066.stdout:4/258: dread d1/f46 [0,4194304] 0 2024-02-28T16:31:13.096 INFO:tasks.workunit.client.1.smithi066.stdout:4/259: chown d1/f12 2 1 2024-02-28T16:31:13.097 INFO:tasks.workunit.client.0.smithi001.stdout:8/613: write d4/d7/d22/d27/d8e/dae/fb0 [547749,126109] 0 2024-02-28T16:31:13.097 INFO:tasks.workunit.client.1.smithi066.stdout:4/260: fdatasync d1/d23/f39 0 2024-02-28T16:31:13.101 INFO:tasks.workunit.client.1.smithi066.stdout:8/126: dread d0/d6/ff [4194304,4194304] 0 2024-02-28T16:31:13.101 INFO:tasks.workunit.client.0.smithi001.stdout:4/635: creat d3/d5/d4e/fc5 x:0 0 0 2024-02-28T16:31:13.101 INFO:tasks.workunit.client.0.smithi001.stdout:1/710: write d0/d9/d56/f4b [556749,86350] 0 2024-02-28T16:31:13.102 INFO:tasks.workunit.client.0.smithi001.stdout:5/582: fdatasync dc/d1a/d21/d2d/d3c/f5e 0 2024-02-28T16:31:13.102 INFO:tasks.workunit.client.0.smithi001.stdout:7/585: stat d3/d17/d3d/d42/f53 0 2024-02-28T16:31:13.103 INFO:tasks.workunit.client.1.smithi066.stdout:8/127: readlink d0/d6/dc/d16/d1a/l20 0 2024-02-28T16:31:13.110 INFO:tasks.workunit.client.0.smithi001.stdout:8/614: rmdir d4/d7/d22/d27/d58/d94 39 2024-02-28T16:31:13.114 INFO:tasks.workunit.client.0.smithi001.stdout:8/615: fsync d4/d7/d22/d27/f4a 0 2024-02-28T16:31:13.118 INFO:tasks.workunit.client.1.smithi066.stdout:9/147: sync 2024-02-28T16:31:13.118 INFO:tasks.workunit.client.1.smithi066.stdout:1/211: sync 2024-02-28T16:31:13.118 INFO:tasks.workunit.client.0.smithi001.stdout:7/586: creat d3/db7/fba x:0 0 0 2024-02-28T16:31:13.119 INFO:tasks.workunit.client.0.smithi001.stdout:1/711: symlink d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/lfb 0 2024-02-28T16:31:13.122 INFO:tasks.workunit.client.0.smithi001.stdout:4/636: dwrite d3/d5/fab [0,4194304] 0 2024-02-28T16:31:13.123 INFO:tasks.workunit.client.0.smithi001.stdout:7/587: write d3/d17/d52/fb8 [573685,87191] 0 2024-02-28T16:31:13.132 INFO:tasks.workunit.client.0.smithi001.stdout:4/637: creat d3/d2d/d4c/d5e/db0/fc6 x:0 0 0 2024-02-28T16:31:13.134 INFO:tasks.workunit.client.0.smithi001.stdout:8/616: getdents d4/d7/d22/d27/d58/d94/db4 0 2024-02-28T16:31:13.141 INFO:tasks.workunit.client.0.smithi001.stdout:8/617: chown d4/d7/d22/d27/d58/f71 212380 1 2024-02-28T16:31:13.144 INFO:tasks.workunit.client.0.smithi001.stdout:1/712: dwrite d0/d9/d56/d2f/fc4 [0,4194304] 0 2024-02-28T16:31:13.145 INFO:tasks.workunit.client.0.smithi001.stdout:4/638: write d3/d5/d4e/f7b [1039868,87927] 0 2024-02-28T16:31:13.147 INFO:tasks.workunit.client.0.smithi001.stdout:8/618: symlink d4/d32/d93/lc5 0 2024-02-28T16:31:13.148 INFO:tasks.workunit.client.1.smithi066.stdout:3/182: link db/dc/d11/f22 db/f3a 0 2024-02-28T16:31:13.149 INFO:tasks.workunit.client.0.smithi001.stdout:4/639: creat d3/d1c/da0/fc7 x:0 0 0 2024-02-28T16:31:13.149 INFO:tasks.workunit.client.1.smithi066.stdout:3/183: write f5 [2069109,29018] 0 2024-02-28T16:31:13.151 INFO:tasks.workunit.client.1.smithi066.stdout:4/261: unlink d1/d23/f39 0 2024-02-28T16:31:13.154 INFO:tasks.workunit.client.0.smithi001.stdout:8/619: mknod d4/d7/d22/d27/d58/d94/db4/cc6 0 2024-02-28T16:31:13.155 INFO:tasks.workunit.client.0.smithi001.stdout:7/588: dread d3/f5f [0,4194304] 0 2024-02-28T16:31:13.157 INFO:tasks.workunit.client.0.smithi001.stdout:5/583: dwrite dc/d12/d41/d54/f71 [0,4194304] 0 2024-02-28T16:31:13.162 INFO:tasks.workunit.client.0.smithi001.stdout:7/589: fsync d3/d17/f1b 0 2024-02-28T16:31:13.166 INFO:tasks.workunit.client.0.smithi001.stdout:4/640: mkdir d3/d1c/d58/d6c/d79/dc8 0 2024-02-28T16:31:13.173 INFO:tasks.workunit.client.0.smithi001.stdout:4/641: creat d3/d5/d92/fc9 x:0 0 0 2024-02-28T16:31:13.175 INFO:tasks.workunit.client.0.smithi001.stdout:3/608: sync 2024-02-28T16:31:13.176 INFO:tasks.workunit.client.0.smithi001.stdout:7/590: read d3/d17/f2a [620463,87418] 0 2024-02-28T16:31:13.178 INFO:tasks.workunit.client.0.smithi001.stdout:4/642: read - d3/d2d/d81/fae zero size 2024-02-28T16:31:13.180 INFO:tasks.workunit.client.0.smithi001.stdout:8/620: dread d4/d7/da8/d67/fc1 [0,4194304] 0 2024-02-28T16:31:13.181 INFO:tasks.workunit.client.0.smithi001.stdout:3/609: write d4/f76 [1667600,108689] 0 2024-02-28T16:31:13.183 INFO:tasks.workunit.client.0.smithi001.stdout:7/591: symlink d3/d17/d3d/d5d/d9a/lbb 0 2024-02-28T16:31:13.188 INFO:tasks.workunit.client.0.smithi001.stdout:7/592: chown d3/f10 0 1 2024-02-28T16:31:13.188 INFO:tasks.workunit.client.0.smithi001.stdout:1/713: dwrite d0/d9/d56/f7f [0,4194304] 0 2024-02-28T16:31:13.188 INFO:tasks.workunit.client.0.smithi001.stdout:5/584: dwrite dc/df/f99 [0,4194304] 0 2024-02-28T16:31:13.191 INFO:tasks.workunit.client.0.smithi001.stdout:3/610: unlink d4/d18/l8d 0 2024-02-28T16:31:13.192 INFO:tasks.workunit.client.0.smithi001.stdout:8/621: mknod d4/d9b/cc7 0 2024-02-28T16:31:13.192 INFO:tasks.workunit.client.0.smithi001.stdout:4/643: mknod d3/d5/d7/cca 0 2024-02-28T16:31:13.195 INFO:tasks.workunit.client.0.smithi001.stdout:1/714: mkdir d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/dfc 0 2024-02-28T16:31:13.196 INFO:tasks.workunit.client.0.smithi001.stdout:3/611: mknod d4/d18/da0/cc9 0 2024-02-28T16:31:13.199 INFO:tasks.workunit.client.0.smithi001.stdout:5/585: truncate dc/df/f13 972276 0 2024-02-28T16:31:13.204 INFO:tasks.workunit.client.0.smithi001.stdout:4/644: mknod d3/d5/ccb 0 2024-02-28T16:31:13.204 INFO:tasks.workunit.client.0.smithi001.stdout:3/612: dread d4/d6/d7/f5d [0,4194304] 0 2024-02-28T16:31:13.209 INFO:tasks.workunit.client.0.smithi001.stdout:3/613: rmdir d4/d1e/d7c/d7d 39 2024-02-28T16:31:13.213 INFO:tasks.workunit.client.1.smithi066.stdout:9/148: symlink da/d22/l2e 0 2024-02-28T16:31:13.214 INFO:tasks.workunit.client.0.smithi001.stdout:8/622: dwrite d4/d7/d22/d27/d58/d94/dbc/fbd [0,4194304] 0 2024-02-28T16:31:13.215 INFO:tasks.workunit.client.0.smithi001.stdout:4/645: rename d3/f3e to d3/d5/d7/d1f/d3a/fcc 0 2024-02-28T16:31:13.216 INFO:tasks.workunit.client.0.smithi001.stdout:3/614: mknod d4/d97/cca 0 2024-02-28T16:31:13.219 INFO:tasks.workunit.client.1.smithi066.stdout:4/262: stat d1/f6 0 2024-02-28T16:31:13.222 INFO:tasks.workunit.client.0.smithi001.stdout:3/615: read d4/d6/d7/f43 [544485,123405] 0 2024-02-28T16:31:13.223 INFO:tasks.workunit.client.0.smithi001.stdout:7/593: dread d3/d17/f1b [0,4194304] 0 2024-02-28T16:31:13.223 INFO:tasks.workunit.client.1.smithi066.stdout:8/128: unlink d0/db/f10 0 2024-02-28T16:31:13.225 INFO:tasks.workunit.client.1.smithi066.stdout:8/129: readlink d0/d6/dc/d16/d1a/l20 0 2024-02-28T16:31:13.226 INFO:tasks.workunit.client.0.smithi001.stdout:8/623: link d4/d7/d22/l5d d4/d7/d22/d27/d4b/d84/lc8 0 2024-02-28T16:31:13.227 INFO:tasks.workunit.client.0.smithi001.stdout:1/715: dwrite d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/fa0 [0,4194304] 0 2024-02-28T16:31:13.228 INFO:tasks.workunit.client.0.smithi001.stdout:3/616: fdatasync d4/d6/d7/d78/d81/fa6 0 2024-02-28T16:31:13.230 INFO:tasks.workunit.client.1.smithi066.stdout:9/149: creat da/df/f2f x:0 0 0 2024-02-28T16:31:13.235 INFO:tasks.workunit.client.1.smithi066.stdout:4/263: read - d1/f1e zero size 2024-02-28T16:31:13.235 INFO:tasks.workunit.client.1.smithi066.stdout:8/130: truncate d0/d6/f1f 181678 0 2024-02-28T16:31:13.236 INFO:tasks.workunit.client.0.smithi001.stdout:8/624: fsync d4/d7/f35 0 2024-02-28T16:31:13.237 INFO:tasks.workunit.client.0.smithi001.stdout:1/716: truncate d0/d9/d1a/d2b/d26/d27/dce/fde 509980 0 2024-02-28T16:31:13.239 INFO:tasks.workunit.client.1.smithi066.stdout:4/264: write d1/d18/f3a [754600,112043] 0 2024-02-28T16:31:13.239 INFO:tasks.workunit.client.1.smithi066.stdout:3/184: link db/l2c db/d13/d14/l3b 0 2024-02-28T16:31:13.240 INFO:tasks.workunit.client.0.smithi001.stdout:1/717: fsync d0/d9/d1a/d2b/d26/d27/d30/d52/d84/f94 0 2024-02-28T16:31:13.242 INFO:tasks.workunit.client.1.smithi066.stdout:9/150: creat da/f30 x:0 0 0 2024-02-28T16:31:13.243 INFO:tasks.workunit.client.1.smithi066.stdout:3/185: symlink db/d13/d1a/l3c 0 2024-02-28T16:31:13.245 INFO:tasks.workunit.client.1.smithi066.stdout:4/265: creat d1/d18/d4e/f53 x:0 0 0 2024-02-28T16:31:13.248 INFO:tasks.workunit.client.1.smithi066.stdout:2/200: sync 2024-02-28T16:31:13.248 INFO:tasks.workunit.client.0.smithi001.stdout:1/718: mkdir d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/dfd 0 2024-02-28T16:31:13.251 INFO:tasks.workunit.client.1.smithi066.stdout:4/266: symlink d1/d23/l54 0 2024-02-28T16:31:13.252 INFO:tasks.workunit.client.0.smithi001.stdout:3/617: dwrite d4/d1e/fc8 [0,4194304] 0 2024-02-28T16:31:13.253 INFO:tasks.workunit.client.1.smithi066.stdout:6/127: dwrite d2/da/f1a [4194304,4194304] 0 2024-02-28T16:31:13.254 INFO:tasks.workunit.client.0.smithi001.stdout:3/618: fdatasync d4/f83 0 2024-02-28T16:31:13.255 INFO:tasks.workunit.client.1.smithi066.stdout:2/201: write d0/d12/d1a/d23/f28 [2260197,95073] 0 2024-02-28T16:31:13.257 INFO:tasks.workunit.client.0.smithi001.stdout:7/594: dwrite d3/d1c/f37 [0,4194304] 0 2024-02-28T16:31:13.258 INFO:tasks.workunit.client.0.smithi001.stdout:8/625: dwrite d4/d7/d22/d27/d4b/d77/f83 [0,4194304] 0 2024-02-28T16:31:13.260 INFO:tasks.workunit.client.1.smithi066.stdout:9/151: rmdir da 39 2024-02-28T16:31:13.265 INFO:tasks.workunit.client.1.smithi066.stdout:6/128: rename d2/da/fc to d2/da/f22 0 2024-02-28T16:31:13.265 INFO:tasks.workunit.client.0.smithi001.stdout:3/619: getdents d4/d1e/d7e 0 2024-02-28T16:31:13.268 INFO:tasks.workunit.client.0.smithi001.stdout:1/719: rename d0/d9/d1a/d2b/d26/d27/d30/d88/db6 to d0/d9/d1a/d2b/dfe 0 2024-02-28T16:31:13.269 INFO:tasks.workunit.client.1.smithi066.stdout:1/212: dwrite d2/d18/d19/f1c [0,4194304] 0 2024-02-28T16:31:13.269 INFO:tasks.workunit.client.1.smithi066.stdout:6/129: chown d2/da/d19/lf 92980 1 2024-02-28T16:31:13.270 INFO:tasks.workunit.client.0.smithi001.stdout:3/620: symlink d4/d1e/db7/lcb 0 2024-02-28T16:31:13.276 INFO:tasks.workunit.client.0.smithi001.stdout:7/595: rename d3/d17/d3d/d59/d6e/f9f to d3/d17/d52/fbc 0 2024-02-28T16:31:13.280 INFO:tasks.workunit.client.0.smithi001.stdout:1/720: creat d0/d9/d1a/d2b/d26/d27/dce/fff x:0 0 0 2024-02-28T16:31:13.284 INFO:tasks.workunit.client.1.smithi066.stdout:4/267: dwrite d1/f16 [0,4194304] 0 2024-02-28T16:31:13.285 INFO:tasks.workunit.client.1.smithi066.stdout:9/152: rename da/df/d18/l1c to da/d21/l31 0 2024-02-28T16:31:13.290 INFO:tasks.workunit.client.0.smithi001.stdout:7/596: dread d3/d17/d3d/d59/f67 [4194304,4194304] 0 2024-02-28T16:31:13.291 INFO:tasks.workunit.client.0.smithi001.stdout:7/597: chown d3/d17/d52/l86 4714227 1 2024-02-28T16:31:13.292 INFO:tasks.workunit.client.1.smithi066.stdout:6/130: dread d2/f14 [0,4194304] 0 2024-02-28T16:31:13.293 INFO:tasks.workunit.client.0.smithi001.stdout:7/598: chown d3/d17/d3d/d42/d8e 0 1 2024-02-28T16:31:13.294 INFO:tasks.workunit.client.1.smithi066.stdout:4/268: mkdir d1/d18/d55 0 2024-02-28T16:31:13.295 INFO:tasks.workunit.client.1.smithi066.stdout:3/186: dwrite db/d13/d14/d24/f34 [0,4194304] 0 2024-02-28T16:31:13.298 INFO:tasks.workunit.client.0.smithi001.stdout:3/621: symlink d4/d97/lcc 0 2024-02-28T16:31:13.300 INFO:tasks.workunit.client.0.smithi001.stdout:3/622: chown d4/d18/da0/fa2 6388 1 2024-02-28T16:31:13.301 INFO:tasks.workunit.client.0.smithi001.stdout:8/626: rename d4/d7/da8/d67 to d4/d7/d22/d27/d58/d94/dc9 0 2024-02-28T16:31:13.304 INFO:tasks.workunit.client.1.smithi066.stdout:4/269: write d1/d18/f38 [113809,117053] 0 2024-02-28T16:31:13.305 INFO:tasks.workunit.client.0.smithi001.stdout:1/721: creat d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f100 x:0 0 0 2024-02-28T16:31:13.310 INFO:tasks.workunit.client.0.smithi001.stdout:8/627: stat d4/d7/c8a 0 2024-02-28T16:31:13.311 INFO:tasks.workunit.client.1.smithi066.stdout:6/131: write d2/da/f18 [1400670,74089] 0 2024-02-28T16:31:13.311 INFO:tasks.workunit.client.1.smithi066.stdout:3/187: mkdir db/dc/d35/d3d 0 2024-02-28T16:31:13.313 INFO:tasks.workunit.client.1.smithi066.stdout:4/270: symlink d1/d23/d25/l56 0 2024-02-28T16:31:13.314 INFO:tasks.workunit.client.0.smithi001.stdout:1/722: mkdir d0/d9/d1a/d2b/d26/d27/dd3/de8/d101 0 2024-02-28T16:31:13.315 INFO:tasks.workunit.client.0.smithi001.stdout:8/628: rmdir d4/d32 39 2024-02-28T16:31:13.319 INFO:tasks.workunit.client.1.smithi066.stdout:6/132: chown d2/f10 1901 1 2024-02-28T16:31:13.320 INFO:tasks.workunit.client.0.smithi001.stdout:1/723: fdatasync d0/d9/d1a/d72/d9d/f6e 0 2024-02-28T16:31:13.320 INFO:tasks.workunit.client.1.smithi066.stdout:8/131: dread d0/d6/f7 [0,4194304] 0 2024-02-28T16:31:13.322 INFO:tasks.workunit.client.1.smithi066.stdout:3/188: creat db/d13/d14/d2d/f3e x:0 0 0 2024-02-28T16:31:13.323 INFO:tasks.workunit.client.0.smithi001.stdout:3/623: link d4/d38/c9f d4/ccd 0 2024-02-28T16:31:13.324 INFO:tasks.workunit.client.0.smithi001.stdout:7/599: getdents d3 0 2024-02-28T16:31:13.325 INFO:tasks.workunit.client.1.smithi066.stdout:1/213: dwrite d2/d18/f21 [0,4194304] 0 2024-02-28T16:31:13.328 INFO:tasks.workunit.client.0.smithi001.stdout:3/624: chown d4/d18/c12 3387344 1 2024-02-28T16:31:13.329 INFO:tasks.workunit.client.1.smithi066.stdout:1/214: stat d2/d18/d19/f24 0 2024-02-28T16:31:13.329 INFO:tasks.workunit.client.1.smithi066.stdout:9/153: dwrite da/f2a [0,4194304] 0 2024-02-28T16:31:13.329 INFO:tasks.workunit.client.0.smithi001.stdout:1/724: write d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f100 [978515,65457] 0 2024-02-28T16:31:13.330 INFO:tasks.workunit.client.1.smithi066.stdout:4/271: rename d1/f7 to d1/d23/d25/d3e/f57 0 2024-02-28T16:31:13.334 INFO:tasks.workunit.client.0.smithi001.stdout:7/600: rename d3/d17/d52/fb8 to d3/d17/d3d/d5d/fbd 0 2024-02-28T16:31:13.335 INFO:tasks.workunit.client.1.smithi066.stdout:1/215: truncate d2/d18/d19/f24 58342 0 2024-02-28T16:31:13.337 INFO:tasks.workunit.client.0.smithi001.stdout:3/625: mknod d4/d6/d7/d78/d81/d8a/cce 0 2024-02-28T16:31:13.339 INFO:tasks.workunit.client.1.smithi066.stdout:8/132: mknod d0/d6/dc/d16/c21 0 2024-02-28T16:31:13.340 INFO:tasks.workunit.client.0.smithi001.stdout:8/629: dwrite d4/d7/d22/d27/d4b/d77/f7a [0,4194304] 0 2024-02-28T16:31:13.344 INFO:tasks.workunit.client.0.smithi001.stdout:3/626: mkdir d4/d1e/dcf 0 2024-02-28T16:31:13.345 INFO:tasks.workunit.client.1.smithi066.stdout:9/154: fsync da/fb 0 2024-02-28T16:31:13.347 INFO:tasks.workunit.client.0.smithi001.stdout:7/601: mknod d3/d17/d3d/d59/d96/cbe 0 2024-02-28T16:31:13.352 INFO:tasks.workunit.client.0.smithi001.stdout:8/630: creat d4/d7/d22/d27/d58/d62/fca x:0 0 0 2024-02-28T16:31:13.355 INFO:tasks.workunit.client.1.smithi066.stdout:8/133: mkdir d0/d22 0 2024-02-28T16:31:13.356 INFO:tasks.workunit.client.1.smithi066.stdout:6/133: dwrite d2/da/f22 [4194304,4194304] 0 2024-02-28T16:31:13.356 INFO:tasks.workunit.client.1.smithi066.stdout:9/155: creat da/d21/f32 x:0 0 0 2024-02-28T16:31:13.357 INFO:tasks.workunit.client.0.smithi001.stdout:8/631: mknod d4/d47/ccb 0 2024-02-28T16:31:13.357 INFO:tasks.workunit.client.0.smithi001.stdout:7/602: symlink d3/d17/d3d/d5d/d9a/d49/d8d/db9/lbf 0 2024-02-28T16:31:13.360 INFO:tasks.workunit.client.1.smithi066.stdout:1/216: unlink d2/cd 0 2024-02-28T16:31:13.362 INFO:tasks.workunit.client.1.smithi066.stdout:4/272: rmdir d1 39 2024-02-28T16:31:13.362 INFO:tasks.workunit.client.1.smithi066.stdout:7/110: sync 2024-02-28T16:31:13.363 INFO:tasks.workunit.client.1.smithi066.stdout:0/272: sync 2024-02-28T16:31:13.363 INFO:tasks.workunit.client.1.smithi066.stdout:5/141: sync 2024-02-28T16:31:13.367 INFO:tasks.workunit.client.1.smithi066.stdout:0/273: readlink d2/d9/d33/l3b 0 2024-02-28T16:31:13.369 INFO:tasks.workunit.client.0.smithi001.stdout:3/627: dwrite d4/d6/f1c [0,4194304] 0 2024-02-28T16:31:13.369 INFO:tasks.workunit.client.1.smithi066.stdout:8/134: rename d0/f4 to d0/f23 0 2024-02-28T16:31:13.370 INFO:tasks.workunit.client.0.smithi001.stdout:1/725: write d0/d9/d1a/d2b/d26/d27/d30/d88/fb3 [981331,127550] 0 2024-02-28T16:31:13.371 INFO:tasks.workunit.client.0.smithi001.stdout:3/628: read d4/d1e/f25 [2730426,69423] 0 2024-02-28T16:31:13.372 INFO:tasks.workunit.client.1.smithi066.stdout:8/135: fdatasync d0/d6/fe 0 2024-02-28T16:31:13.375 INFO:tasks.workunit.client.0.smithi001.stdout:3/629: creat d4/d1e/db7/fd0 x:0 0 0 2024-02-28T16:31:13.377 INFO:tasks.workunit.client.1.smithi066.stdout:9/156: mknod da/c33 0 2024-02-28T16:31:13.381 INFO:tasks.workunit.client.1.smithi066.stdout:7/111: stat dc/de/d17/d1e/f20 0 2024-02-28T16:31:13.381 INFO:tasks.workunit.client.0.smithi001.stdout:1/726: rename d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/db1 to d0/d9/d1a/d2b/d26/d27/dd3/de8/d102 0 2024-02-28T16:31:13.381 INFO:tasks.workunit.client.0.smithi001.stdout:3/630: fdatasync d4/d18/d5f/f5b 0 2024-02-28T16:31:13.384 INFO:tasks.workunit.client.1.smithi066.stdout:7/112: dread - dc/de/f1b zero size 2024-02-28T16:31:13.385 INFO:tasks.workunit.client.0.smithi001.stdout:3/631: rmdir d4/d1e/d7c/da8 39 2024-02-28T16:31:13.388 INFO:tasks.workunit.client.0.smithi001.stdout:1/727: creat d0/d9/d56/f103 x:0 0 0 2024-02-28T16:31:13.388 INFO:tasks.workunit.client.0.smithi001.stdout:3/632: fdatasync d4/d1e/f33 0 2024-02-28T16:31:13.390 INFO:tasks.workunit.client.1.smithi066.stdout:3/189: dwrite db/ff [0,4194304] 0 2024-02-28T16:31:13.390 INFO:tasks.workunit.client.1.smithi066.stdout:1/217: dwrite d2/d18/d19/f32 [0,4194304] 0 2024-02-28T16:31:13.390 INFO:tasks.workunit.client.1.smithi066.stdout:2/202: sync 2024-02-28T16:31:13.391 INFO:tasks.workunit.client.0.smithi001.stdout:3/633: chown d4/d1e/d7c/d9a/fba 58295 1 2024-02-28T16:31:13.391 INFO:tasks.workunit.client.1.smithi066.stdout:0/274: mknod d2/d27/d41/c58 0 2024-02-28T16:31:13.395 INFO:tasks.workunit.client.0.smithi001.stdout:3/634: link d4/d1e/f33 d4/d6/d7/d78/fd1 0 2024-02-28T16:31:13.396 INFO:tasks.workunit.client.1.smithi066.stdout:6/134: creat d2/f23 x:0 0 0 2024-02-28T16:31:13.399 INFO:tasks.workunit.client.0.smithi001.stdout:3/635: rename d4/d18/d5f/l36 to d4/d18/da0/ld2 0 2024-02-28T16:31:13.400 INFO:tasks.workunit.client.1.smithi066.stdout:7/113: mknod dc/de/d17/c22 0 2024-02-28T16:31:13.402 INFO:tasks.workunit.client.1.smithi066.stdout:9/157: dwrite da/f20 [0,4194304] 0 2024-02-28T16:31:13.403 INFO:tasks.workunit.client.1.smithi066.stdout:4/273: write d1/f4 [327232,16290] 0 2024-02-28T16:31:13.403 INFO:tasks.workunit.client.1.smithi066.stdout:2/203: creat d0/d12/d1a/d23/d29/d21/f46 x:0 0 0 2024-02-28T16:31:13.406 INFO:tasks.workunit.client.1.smithi066.stdout:9/158: readlink da/df/l25 0 2024-02-28T16:31:13.409 INFO:tasks.workunit.client.1.smithi066.stdout:4/274: dread d1/d18/f2b [0,4194304] 0 2024-02-28T16:31:13.410 INFO:tasks.workunit.client.1.smithi066.stdout:9/159: write da/d10/f13 [1300741,40132] 0 2024-02-28T16:31:13.411 INFO:tasks.workunit.client.0.smithi001.stdout:8/632: dread d4/d47/d8b/f8c [0,4194304] 0 2024-02-28T16:31:13.412 INFO:tasks.workunit.client.1.smithi066.stdout:9/160: stat da/c33 0 2024-02-28T16:31:13.415 INFO:tasks.workunit.client.0.smithi001.stdout:8/633: write d4/d7/da8/f91 [169376,9721] 0 2024-02-28T16:31:13.416 INFO:tasks.workunit.client.1.smithi066.stdout:5/142: link d3/d4/dc/c1f d3/c30 0 2024-02-28T16:31:13.417 INFO:tasks.workunit.client.1.smithi066.stdout:1/218: dwrite d2/d18/d19/f1e [0,4194304] 0 2024-02-28T16:31:13.426 INFO:tasks.workunit.client.1.smithi066.stdout:3/190: rename db/d13/d1a/d33/c38 to db/d13/d1a/d25/c3f 0 2024-02-28T16:31:13.430 INFO:tasks.workunit.client.1.smithi066.stdout:4/275: creat d1/d18/d41/f58 x:0 0 0 2024-02-28T16:31:13.430 INFO:tasks.workunit.client.0.smithi001.stdout:8/634: dread d4/d47/d8b/fba [0,4194304] 0 2024-02-28T16:31:13.432 INFO:tasks.workunit.client.1.smithi066.stdout:3/191: truncate db/d13/d14/d24/f34 4871114 0 2024-02-28T16:31:13.434 INFO:tasks.workunit.client.1.smithi066.stdout:5/143: creat d3/d1b/f31 x:0 0 0 2024-02-28T16:31:13.435 INFO:tasks.workunit.client.0.smithi001.stdout:1/728: write d0/d9/d1a/d72/f8a [603928,52026] 0 2024-02-28T16:31:13.436 INFO:tasks.workunit.client.0.smithi001.stdout:8/635: symlink d4/d7/d22/d27/d58/d94/db4/lcc 0 2024-02-28T16:31:13.436 INFO:tasks.workunit.client.1.smithi066.stdout:1/219: mkdir d2/d27/d2b/d43 0 2024-02-28T16:31:13.442 INFO:tasks.workunit.client.1.smithi066.stdout:7/114: rename dc/de/f1b to dc/de/d17/d1e/f23 0 2024-02-28T16:31:13.443 INFO:tasks.workunit.client.1.smithi066.stdout:4/276: creat d1/d23/d25/f59 x:0 0 0 2024-02-28T16:31:13.445 INFO:tasks.workunit.client.1.smithi066.stdout:8/136: truncate d0/d6/f1f 130996 0 2024-02-28T16:31:13.445 INFO:tasks.workunit.client.1.smithi066.stdout:4/277: write d1/f4 [2604085,78932] 0 2024-02-28T16:31:13.447 INFO:tasks.workunit.client.1.smithi066.stdout:0/275: link d2/d9/d10/d23/f2d d2/d9/d10/f59 0 2024-02-28T16:31:13.453 INFO:tasks.workunit.client.1.smithi066.stdout:2/204: link d0/f37 d0/d12/f47 0 2024-02-28T16:31:13.456 INFO:tasks.workunit.client.1.smithi066.stdout:5/144: rename d3/d4/f5 to d3/d4/dc/d2f/f32 0 2024-02-28T16:31:13.459 INFO:tasks.workunit.client.1.smithi066.stdout:7/115: dwrite dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:13.463 INFO:tasks.workunit.client.1.smithi066.stdout:1/220: unlink d2/l8 0 2024-02-28T16:31:13.466 INFO:tasks.workunit.client.1.smithi066.stdout:8/137: rmdir d0/db/d12 39 2024-02-28T16:31:13.471 INFO:tasks.workunit.client.1.smithi066.stdout:3/192: truncate db/d13/d1a/f1b 649209 0 2024-02-28T16:31:13.472 INFO:tasks.workunit.client.1.smithi066.stdout:2/205: symlink d0/d12/d1a/d23/d29/l48 0 2024-02-28T16:31:13.474 INFO:tasks.workunit.client.1.smithi066.stdout:7/116: creat dc/de/d17/d1e/f24 x:0 0 0 2024-02-28T16:31:13.478 INFO:tasks.workunit.client.1.smithi066.stdout:4/278: mkdir d1/d5a 0 2024-02-28T16:31:13.480 INFO:tasks.workunit.client.1.smithi066.stdout:6/135: getdents d2/da/d19 0 2024-02-28T16:31:13.481 INFO:tasks.workunit.client.1.smithi066.stdout:0/276: dread d2/d9/f40 [0,4194304] 0 2024-02-28T16:31:13.482 INFO:tasks.workunit.client.1.smithi066.stdout:8/138: creat d0/f24 x:0 0 0 2024-02-28T16:31:13.484 INFO:tasks.workunit.client.1.smithi066.stdout:3/193: mknod db/d2a/c40 0 2024-02-28T16:31:13.485 INFO:tasks.workunit.client.1.smithi066.stdout:2/206: rmdir d0/d12/d1a/d39/d40 39 2024-02-28T16:31:13.490 INFO:tasks.workunit.client.1.smithi066.stdout:3/194: truncate db/d13/d14/d2d/f3e 1021962 0 2024-02-28T16:31:13.491 INFO:tasks.workunit.client.1.smithi066.stdout:4/279: rmdir d1/d18 39 2024-02-28T16:31:13.491 INFO:tasks.workunit.client.1.smithi066.stdout:5/145: write d3/d4/f12 [1485599,44884] 0 2024-02-28T16:31:13.494 INFO:tasks.workunit.client.0.smithi001.stdout:8/636: mkdir d4/d32/d69/d9a/dcd 0 2024-02-28T16:31:13.496 INFO:tasks.workunit.client.0.smithi001.stdout:8/637: fdatasync d4/d7/d22/d27/d8e/dae/fb0 0 2024-02-28T16:31:13.497 INFO:tasks.workunit.client.1.smithi066.stdout:2/207: fsync d0/d12/f47 0 2024-02-28T16:31:13.497 INFO:tasks.workunit.client.0.smithi001.stdout:1/729: rename d0/d9/d1a/d2b/dfe/ce3 to d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/c104 0 2024-02-28T16:31:13.498 INFO:tasks.workunit.client.1.smithi066.stdout:6/136: rename d2/c7 to d2/da/c24 0 2024-02-28T16:31:13.499 INFO:tasks.workunit.client.0.smithi001.stdout:1/730: mkdir d0/d9/d1a/d2b/d26/d105 0 2024-02-28T16:31:13.500 INFO:tasks.workunit.client.1.smithi066.stdout:7/117: dwrite dc/de/f10 [0,4194304] 0 2024-02-28T16:31:13.500 INFO:tasks.workunit.client.1.smithi066.stdout:5/146: chown d3/c22 21 1 2024-02-28T16:31:13.504 INFO:tasks.workunit.client.0.smithi001.stdout:1/731: creat d0/d9/d1a/d2b/d26/d27/dd3/de8/d102/f106 x:0 0 0 2024-02-28T16:31:13.505 INFO:tasks.workunit.client.0.smithi001.stdout:1/732: chown d0/d9/d1a/f4c 13958 1 2024-02-28T16:31:13.506 INFO:tasks.workunit.client.1.smithi066.stdout:3/195: fsync db/dc/d11/f22 0 2024-02-28T16:31:13.509 INFO:tasks.workunit.client.1.smithi066.stdout:1/221: dwrite d2/f10 [0,4194304] 0 2024-02-28T16:31:13.511 INFO:tasks.workunit.client.1.smithi066.stdout:3/196: chown db/dc/d11/l16 124 1 2024-02-28T16:31:13.511 INFO:tasks.workunit.client.1.smithi066.stdout:8/139: mkdir d0/db/d12/d1d/d25 0 2024-02-28T16:31:13.514 INFO:tasks.workunit.client.1.smithi066.stdout:3/197: readlink db/l12 0 2024-02-28T16:31:13.517 INFO:tasks.workunit.client.1.smithi066.stdout:4/280: rmdir d1/d5a 0 2024-02-28T16:31:13.517 INFO:tasks.workunit.client.1.smithi066.stdout:6/137: fdatasync d2/f14 0 2024-02-28T16:31:13.518 INFO:tasks.workunit.client.1.smithi066.stdout:0/277: getdents d2/d9/d33 0 2024-02-28T16:31:13.519 INFO:tasks.workunit.client.1.smithi066.stdout:5/147: link d3/d4/dc/d2f/f32 d3/d4/dc/d2f/f33 0 2024-02-28T16:31:13.522 INFO:tasks.workunit.client.1.smithi066.stdout:3/198: read db/dc/d11/f22 [505201,17225] 0 2024-02-28T16:31:13.524 INFO:tasks.workunit.client.1.smithi066.stdout:8/140: mknod d0/db/d15/c26 0 2024-02-28T16:31:13.526 INFO:tasks.workunit.client.1.smithi066.stdout:0/278: unlink d2/fe 0 2024-02-28T16:31:13.528 INFO:tasks.workunit.client.1.smithi066.stdout:8/141: write d0/f24 [509832,69499] 0 2024-02-28T16:31:13.531 INFO:tasks.workunit.client.1.smithi066.stdout:7/118: dwrite dc/f16 [0,4194304] 0 2024-02-28T16:31:13.531 INFO:tasks.workunit.client.1.smithi066.stdout:1/222: write d2/d18/d19/f3b [12448,35215] 0 2024-02-28T16:31:13.532 INFO:tasks.workunit.client.1.smithi066.stdout:0/279: dread - d2/d9/d10/d50/f57 zero size 2024-02-28T16:31:13.533 INFO:tasks.workunit.client.1.smithi066.stdout:5/148: symlink d3/d4/dc/dd/l34 0 2024-02-28T16:31:13.534 INFO:tasks.workunit.client.1.smithi066.stdout:4/281: dread d1/d18/f3a [0,4194304] 0 2024-02-28T16:31:13.534 INFO:tasks.workunit.client.1.smithi066.stdout:3/199: write db/dc/d11/d1f/f2e [1335145,68362] 0 2024-02-28T16:31:13.537 INFO:tasks.workunit.client.1.smithi066.stdout:0/280: read - d2/d9/d10/f4e zero size 2024-02-28T16:31:13.544 INFO:tasks.workunit.client.1.smithi066.stdout:4/282: readlink d1/l21 0 2024-02-28T16:31:13.545 INFO:tasks.workunit.client.1.smithi066.stdout:6/138: dread d2/da/f1a [0,4194304] 0 2024-02-28T16:31:13.549 INFO:tasks.workunit.client.0.smithi001.stdout:9/707: sync 2024-02-28T16:31:13.552 INFO:tasks.workunit.client.0.smithi001.stdout:8/638: dread d4/d32/d69/f6f [0,4194304] 0 2024-02-28T16:31:13.552 INFO:tasks.workunit.client.1.smithi066.stdout:6/139: mknod d2/da/d19/c25 0 2024-02-28T16:31:13.553 INFO:tasks.workunit.client.1.smithi066.stdout:3/200: creat db/dc/d35/d3d/f41 x:0 0 0 2024-02-28T16:31:13.553 INFO:tasks.workunit.client.0.smithi001.stdout:1/733: chown d0/d9/d1a/d2b/d26/d27/dd3/de8/d102/fe9 653 1 2024-02-28T16:31:13.554 INFO:tasks.workunit.client.1.smithi066.stdout:4/283: truncate d1/d18/f2b 809515 0 2024-02-28T16:31:13.555 INFO:tasks.workunit.client.0.smithi001.stdout:8/639: readlink d4/d47/l53 0 2024-02-28T16:31:13.555 INFO:tasks.workunit.client.1.smithi066.stdout:1/223: rename d2/d18/d19/f31 to d2/d27/d2b/d34/f44 0 2024-02-28T16:31:13.557 INFO:tasks.workunit.client.1.smithi066.stdout:3/201: creat db/d2a/f42 x:0 0 0 2024-02-28T16:31:13.558 INFO:tasks.workunit.client.1.smithi066.stdout:6/140: mkdir d2/d26 0 2024-02-28T16:31:13.559 INFO:tasks.workunit.client.0.smithi001.stdout:8/640: rmdir d4/d7/d22/d27/d4b/d77 39 2024-02-28T16:31:13.559 INFO:tasks.workunit.client.1.smithi066.stdout:4/284: stat d1/f5 0 2024-02-28T16:31:13.564 INFO:tasks.workunit.client.1.smithi066.stdout:3/202: chown ca 62500 1 2024-02-28T16:31:13.570 INFO:tasks.workunit.client.1.smithi066.stdout:0/281: dwrite d2/d9/f11 [0,4194304] 0 2024-02-28T16:31:13.570 INFO:tasks.workunit.client.1.smithi066.stdout:1/224: mknod d2/d27/d2b/d37/c45 0 2024-02-28T16:31:13.572 INFO:tasks.workunit.client.0.smithi001.stdout:1/734: write d0/d9/d1a/d2b/d26/f34 [4394045,79694] 0 2024-02-28T16:31:13.575 INFO:tasks.workunit.client.0.smithi001.stdout:1/735: write d0/d9/d1a/d2b/f6d [544340,14651] 0 2024-02-28T16:31:13.577 INFO:tasks.workunit.client.1.smithi066.stdout:8/142: dwrite d0/d6/dc/f14 [4194304,4194304] 0 2024-02-28T16:31:13.579 INFO:tasks.workunit.client.0.smithi001.stdout:1/736: creat d0/d9/d56/d2f/f107 x:0 0 0 2024-02-28T16:31:13.586 INFO:tasks.workunit.client.0.smithi001.stdout:1/737: creat d0/d9/d1a/d2b/d26/f108 x:0 0 0 2024-02-28T16:31:13.604 INFO:tasks.workunit.client.1.smithi066.stdout:9/161: sync 2024-02-28T16:31:13.611 INFO:tasks.workunit.client.0.smithi001.stdout:1/738: dwrite d0/d9/fa [4194304,4194304] 0 2024-02-28T16:31:13.614 INFO:tasks.workunit.client.0.smithi001.stdout:1/739: chown d0/d9/c9b 11830935 1 2024-02-28T16:31:13.616 INFO:tasks.workunit.client.0.smithi001.stdout:1/740: readlink d0/d9/d1a/d2b/d26/l38 0 2024-02-28T16:31:13.625 INFO:tasks.workunit.client.0.smithi001.stdout:2/608: sync 2024-02-28T16:31:13.625 INFO:tasks.workunit.client.0.smithi001.stdout:6/562: sync 2024-02-28T16:31:13.628 INFO:tasks.workunit.client.1.smithi066.stdout:4/285: mknod d1/d18/d55/c5b 0 2024-02-28T16:31:13.628 INFO:tasks.workunit.client.1.smithi066.stdout:7/119: dwrite dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:13.631 INFO:tasks.workunit.client.0.smithi001.stdout:1/741: link d0/d9/d1a/d2b/fba d0/d9/d1a/d2b/dfe/f109 0 2024-02-28T16:31:13.632 INFO:tasks.workunit.client.0.smithi001.stdout:6/563: mknod d9/db/d5f/d62/cc1 0 2024-02-28T16:31:13.633 INFO:tasks.workunit.client.1.smithi066.stdout:5/149: dwrite d3/d4/dc/d2f/f32 [0,4194304] 0 2024-02-28T16:31:13.634 INFO:tasks.workunit.client.1.smithi066.stdout:8/143: stat d0/d6/dc/d16/d1a 0 2024-02-28T16:31:13.636 INFO:tasks.workunit.client.1.smithi066.stdout:9/162: write da/f30 [1010043,83692] 0 2024-02-28T16:31:13.637 INFO:tasks.workunit.client.1.smithi066.stdout:4/286: write d1/d18/d4e/f50 [571078,45933] 0 2024-02-28T16:31:13.639 INFO:tasks.workunit.client.1.smithi066.stdout:7/120: creat dc/f25 x:0 0 0 2024-02-28T16:31:13.640 INFO:tasks.workunit.client.0.smithi001.stdout:1/742: dread - d0/d9/d1a/d2b/d26/d27/dd3/de8/d102/fe9 zero size 2024-02-28T16:31:13.641 INFO:tasks.workunit.client.1.smithi066.stdout:7/121: stat cb 0 2024-02-28T16:31:13.641 INFO:tasks.workunit.client.1.smithi066.stdout:5/150: mkdir d3/d4/dc/dd/d35 0 2024-02-28T16:31:13.642 INFO:tasks.workunit.client.0.smithi001.stdout:6/564: creat d9/de/dbd/fc2 x:0 0 0 2024-02-28T16:31:13.643 INFO:tasks.workunit.client.1.smithi066.stdout:6/141: dwrite d2/da/d19/f17 [0,4194304] 0 2024-02-28T16:31:13.643 INFO:tasks.workunit.client.1.smithi066.stdout:0/282: link d2/d9/d10/f1d d2/d9/d10/d50/d51/f5a 0 2024-02-28T16:31:13.643 INFO:tasks.workunit.client.1.smithi066.stdout:4/287: write d1/f1e [579868,107414] 0 2024-02-28T16:31:13.647 INFO:tasks.workunit.client.0.smithi001.stdout:1/743: creat d0/d9/d1a/d2b/dfe/f10a x:0 0 0 2024-02-28T16:31:13.648 INFO:tasks.workunit.client.1.smithi066.stdout:9/163: creat da/df/f34 x:0 0 0 2024-02-28T16:31:13.649 INFO:tasks.workunit.client.1.smithi066.stdout:7/122: rmdir dc/de/d17 39 2024-02-28T16:31:13.650 INFO:tasks.workunit.client.1.smithi066.stdout:1/225: truncate d2/f7 3534544 0 2024-02-28T16:31:13.651 INFO:tasks.workunit.client.1.smithi066.stdout:5/151: fsync d3/d4/dc/dd/d2e/d18/f2b 0 2024-02-28T16:31:13.658 INFO:tasks.workunit.client.1.smithi066.stdout:8/144: creat d0/db/f27 x:0 0 0 2024-02-28T16:31:13.658 INFO:tasks.workunit.client.1.smithi066.stdout:5/152: creat d3/d1b/f36 x:0 0 0 2024-02-28T16:31:13.658 INFO:tasks.workunit.client.1.smithi066.stdout:4/288: symlink d1/l5c 0 2024-02-28T16:31:13.658 INFO:tasks.workunit.client.1.smithi066.stdout:8/145: readlink d0/d6/dc/d16/d1a/l20 0 2024-02-28T16:31:13.659 INFO:tasks.workunit.client.1.smithi066.stdout:6/142: creat d2/d26/f27 x:0 0 0 2024-02-28T16:31:13.660 INFO:tasks.workunit.client.1.smithi066.stdout:1/226: mkdir d2/d27/d2b/d43/d46 0 2024-02-28T16:31:13.662 INFO:tasks.workunit.client.1.smithi066.stdout:9/164: creat da/f35 x:0 0 0 2024-02-28T16:31:13.664 INFO:tasks.workunit.client.1.smithi066.stdout:7/123: dread dc/de/f13 [0,4194304] 0 2024-02-28T16:31:13.665 INFO:tasks.workunit.client.1.smithi066.stdout:6/143: mknod d2/da/c28 0 2024-02-28T16:31:13.667 INFO:tasks.workunit.client.1.smithi066.stdout:8/146: symlink d0/d8/l28 0 2024-02-28T16:31:13.668 INFO:tasks.workunit.client.1.smithi066.stdout:9/165: write da/f20 [1012297,24263] 0 2024-02-28T16:31:13.670 INFO:tasks.workunit.client.1.smithi066.stdout:8/147: readlink d0/d8/l28 0 2024-02-28T16:31:13.674 INFO:tasks.workunit.client.0.smithi001.stdout:1/744: truncate d0/d9/d1a/d2b/d26/d27/d30/d88/fb3 219249 0 2024-02-28T16:31:13.675 INFO:tasks.workunit.client.1.smithi066.stdout:5/153: truncate d3/d4/dc/dd/d2e/d18/f2b 960127 0 2024-02-28T16:31:13.677 INFO:tasks.workunit.client.0.smithi001.stdout:1/745: dread - d0/d9/d56/d2f/fdb zero size 2024-02-28T16:31:13.680 INFO:tasks.workunit.client.1.smithi066.stdout:0/283: dwrite d2/f28 [0,4194304] 0 2024-02-28T16:31:13.681 INFO:tasks.workunit.client.1.smithi066.stdout:7/124: write dc/de/d17/d1e/f23 [340770,69341] 0 2024-02-28T16:31:13.681 INFO:tasks.workunit.client.1.smithi066.stdout:6/144: creat d2/f29 x:0 0 0 2024-02-28T16:31:13.683 INFO:tasks.workunit.client.1.smithi066.stdout:4/289: dwrite d1/d18/f4a [0,4194304] 0 2024-02-28T16:31:13.683 INFO:tasks.workunit.client.0.smithi001.stdout:6/565: dwrite d9/d29/d6a/f73 [0,4194304] 0 2024-02-28T16:31:13.685 INFO:tasks.workunit.client.1.smithi066.stdout:5/154: creat d3/d1b/f37 x:0 0 0 2024-02-28T16:31:13.687 INFO:tasks.workunit.client.1.smithi066.stdout:5/155: dread - d3/d4/dc/f1e zero size 2024-02-28T16:31:13.692 INFO:tasks.workunit.client.1.smithi066.stdout:5/156: dread - d3/d4/dc/f1e zero size 2024-02-28T16:31:13.693 INFO:tasks.workunit.client.0.smithi001.stdout:6/566: creat d9/db/fc3 x:0 0 0 2024-02-28T16:31:13.696 INFO:tasks.workunit.client.0.smithi001.stdout:6/567: mkdir d9/d39/dc4 0 2024-02-28T16:31:13.698 INFO:tasks.workunit.client.1.smithi066.stdout:5/157: write d3/d1b/f2d [533821,31693] 0 2024-02-28T16:31:13.699 INFO:tasks.workunit.client.1.smithi066.stdout:1/227: truncate d2/d27/d2b/f2c 29194 0 2024-02-28T16:31:13.703 INFO:tasks.workunit.client.0.smithi001.stdout:6/568: fsync d9/d29/d6e/f2f 0 2024-02-28T16:31:13.705 INFO:tasks.workunit.client.1.smithi066.stdout:8/148: getdents d0 0 2024-02-28T16:31:13.705 INFO:tasks.workunit.client.1.smithi066.stdout:6/145: mkdir d2/d2a 0 2024-02-28T16:31:13.706 INFO:tasks.workunit.client.0.smithi001.stdout:6/569: write d9/db/d5f/d62/f71 [52958,55526] 0 2024-02-28T16:31:13.707 INFO:tasks.workunit.client.1.smithi066.stdout:4/290: creat d1/d23/d25/d43/f5d x:0 0 0 2024-02-28T16:31:13.710 INFO:tasks.workunit.client.1.smithi066.stdout:1/228: mknod d2/d27/d2b/d37/c47 0 2024-02-28T16:31:13.710 INFO:tasks.workunit.client.0.smithi001.stdout:6/570: write d9/de/f2c [6673718,13812] 0 2024-02-28T16:31:13.711 INFO:tasks.workunit.client.1.smithi066.stdout:7/125: dwrite f5 [0,4194304] 0 2024-02-28T16:31:13.712 INFO:tasks.workunit.client.1.smithi066.stdout:0/284: dwrite d2/f28 [0,4194304] 0 2024-02-28T16:31:13.714 INFO:tasks.workunit.client.1.smithi066.stdout:5/158: rename d3/d4/dc/d2f/f28 to d3/d4/dc/dd/d2e/f38 0 2024-02-28T16:31:13.716 INFO:tasks.workunit.client.1.smithi066.stdout:1/229: write d2/f17 [1292995,119127] 0 2024-02-28T16:31:13.718 INFO:tasks.workunit.client.1.smithi066.stdout:8/149: creat d0/db/d12/d1d/f29 x:0 0 0 2024-02-28T16:31:13.720 INFO:tasks.workunit.client.1.smithi066.stdout:0/285: mkdir d2/d27/d41/d5b 0 2024-02-28T16:31:13.720 INFO:tasks.workunit.client.1.smithi066.stdout:6/146: getdents d2/d2a 0 2024-02-28T16:31:13.721 INFO:tasks.workunit.client.1.smithi066.stdout:1/230: mkdir d2/d27/d2b/d37/d48 0 2024-02-28T16:31:13.725 INFO:tasks.workunit.client.1.smithi066.stdout:6/147: chown d2/da/d19/f20 34208820 1 2024-02-28T16:31:13.725 INFO:tasks.workunit.client.1.smithi066.stdout:0/286: write d2/d9/d10/f39 [3890131,66369] 0 2024-02-28T16:31:13.726 INFO:tasks.workunit.client.1.smithi066.stdout:4/291: fsync d1/d18/f4a 0 2024-02-28T16:31:13.729 INFO:tasks.workunit.client.1.smithi066.stdout:0/287: chown d2/f22 3 1 2024-02-28T16:31:13.729 INFO:tasks.workunit.client.1.smithi066.stdout:6/148: mknod d2/d26/c2b 0 2024-02-28T16:31:13.733 INFO:tasks.workunit.client.1.smithi066.stdout:8/150: truncate d0/d6/f1f 753597 0 2024-02-28T16:31:13.736 INFO:tasks.workunit.client.0.smithi001.stdout:6/571: dread d9/d29/d6e/f18 [0,4194304] 0 2024-02-28T16:31:13.737 INFO:tasks.workunit.client.1.smithi066.stdout:8/151: stat d0/db/d15/c26 0 2024-02-28T16:31:13.739 INFO:tasks.workunit.client.1.smithi066.stdout:6/149: mkdir d2/d2c 0 2024-02-28T16:31:13.739 INFO:tasks.workunit.client.1.smithi066.stdout:0/288: link d2/l31 d2/d9/d10/d50/d51/l5c 0 2024-02-28T16:31:13.739 INFO:tasks.workunit.client.0.smithi001.stdout:6/572: creat d9/db/d5f/d8b/fc5 x:0 0 0 2024-02-28T16:31:13.740 INFO:tasks.workunit.client.0.smithi001.stdout:0/591: sync 2024-02-28T16:31:13.742 INFO:tasks.workunit.client.1.smithi066.stdout:5/159: dwrite d3/d4/dc/f1e [0,4194304] 0 2024-02-28T16:31:13.742 INFO:tasks.workunit.client.0.smithi001.stdout:0/592: fsync d18/d22/f3f 0 2024-02-28T16:31:13.745 INFO:tasks.workunit.client.1.smithi066.stdout:5/160: fdatasync d3/d1b/f36 0 2024-02-28T16:31:13.746 INFO:tasks.workunit.client.1.smithi066.stdout:6/150: creat d2/d26/f2d x:0 0 0 2024-02-28T16:31:13.746 INFO:tasks.workunit.client.0.smithi001.stdout:0/593: creat d18/d22/d2b/d36/d48/fc1 x:0 0 0 2024-02-28T16:31:13.749 INFO:tasks.workunit.client.1.smithi066.stdout:1/231: dwrite d2/f1d [0,4194304] 0 2024-02-28T16:31:13.752 INFO:tasks.workunit.client.0.smithi001.stdout:0/594: unlink d18/d22/d57/d7e/faa 0 2024-02-28T16:31:13.756 INFO:tasks.workunit.client.1.smithi066.stdout:4/292: dwrite d1/fe [0,4194304] 0 2024-02-28T16:31:13.758 INFO:tasks.workunit.client.0.smithi001.stdout:6/573: dwrite d9/db/fc [4194304,4194304] 0 2024-02-28T16:31:13.759 INFO:tasks.workunit.client.1.smithi066.stdout:8/152: symlink d0/d6/dc/d16/d1c/l2a 0 2024-02-28T16:31:13.762 INFO:tasks.workunit.client.1.smithi066.stdout:4/293: write d1/d23/f3d [739351,83472] 0 2024-02-28T16:31:13.764 INFO:tasks.workunit.client.1.smithi066.stdout:5/161: mkdir d3/d1b/d39 0 2024-02-28T16:31:13.765 INFO:tasks.workunit.client.1.smithi066.stdout:6/151: symlink d2/da/l2e 0 2024-02-28T16:31:13.765 INFO:tasks.workunit.client.1.smithi066.stdout:1/232: creat d2/d27/d2b/d37/f49 x:0 0 0 2024-02-28T16:31:13.771 INFO:tasks.workunit.client.1.smithi066.stdout:4/294: write d1/d23/d25/d3e/f3f [716879,35312] 0 2024-02-28T16:31:13.771 INFO:tasks.workunit.client.1.smithi066.stdout:8/153: chown d0/d6/f9 3924484 1 2024-02-28T16:31:13.777 INFO:tasks.workunit.client.1.smithi066.stdout:2/208: sync 2024-02-28T16:31:13.779 INFO:tasks.workunit.client.1.smithi066.stdout:6/152: truncate f1 79754 0 2024-02-28T16:31:13.779 INFO:tasks.workunit.client.1.smithi066.stdout:1/233: symlink d2/d27/d2b/l4a 0 2024-02-28T16:31:13.779 INFO:tasks.workunit.client.1.smithi066.stdout:5/162: mknod d3/d4/c3a 0 2024-02-28T16:31:13.782 INFO:tasks.workunit.client.1.smithi066.stdout:4/295: rmdir d1/d18/d4e 39 2024-02-28T16:31:13.784 INFO:tasks.workunit.client.1.smithi066.stdout:0/289: truncate d2/f22 896736 0 2024-02-28T16:31:13.791 INFO:tasks.workunit.client.1.smithi066.stdout:4/296: stat d1/f20 0 2024-02-28T16:31:13.792 INFO:tasks.workunit.client.1.smithi066.stdout:6/153: creat d2/d2a/f2f x:0 0 0 2024-02-28T16:31:13.793 INFO:tasks.workunit.client.1.smithi066.stdout:5/163: symlink d3/d1b/d20/l3b 0 2024-02-28T16:31:13.796 INFO:tasks.workunit.client.0.smithi001.stdout:0/595: readlink d18/d22/l98 0 2024-02-28T16:31:13.797 INFO:tasks.workunit.client.1.smithi066.stdout:2/209: write d0/d12/d1a/d39/d40/f45 [284242,26103] 0 2024-02-28T16:31:13.797 INFO:tasks.workunit.client.1.smithi066.stdout:6/154: fsync d2/da/f1b 0 2024-02-28T16:31:13.798 INFO:tasks.workunit.client.0.smithi001.stdout:0/596: chown d18/d5a/f63 0 1 2024-02-28T16:31:13.801 INFO:tasks.workunit.client.0.smithi001.stdout:6/574: write d9/d29/d6e/f18 [2075524,44469] 0 2024-02-28T16:31:13.805 INFO:tasks.workunit.client.1.smithi066.stdout:4/297: creat d1/d23/d25/d43/f5e x:0 0 0 2024-02-28T16:31:13.806 INFO:tasks.workunit.client.1.smithi066.stdout:0/290: link d2/d9/d33/l4c d2/d9/d33/l5d 0 2024-02-28T16:31:13.806 INFO:tasks.workunit.client.1.smithi066.stdout:8/154: dwrite d0/db/d12/d1d/f29 [0,4194304] 0 2024-02-28T16:31:13.809 INFO:tasks.workunit.client.1.smithi066.stdout:2/210: write d0/d12/d1a/d23/d29/d21/f46 [278924,127464] 0 2024-02-28T16:31:13.815 INFO:tasks.workunit.client.1.smithi066.stdout:1/234: dwrite d2/d27/d2b/f2a [4194304,4194304] 0 2024-02-28T16:31:13.817 INFO:tasks.workunit.client.0.smithi001.stdout:0/597: dwrite d18/d22/d2b/d5f/f85 [0,4194304] 0 2024-02-28T16:31:13.817 INFO:tasks.workunit.client.1.smithi066.stdout:6/155: creat d2/d2c/f30 x:0 0 0 2024-02-28T16:31:13.821 INFO:tasks.workunit.client.1.smithi066.stdout:2/211: rmdir d0/d12/d1a/d39/d40 39 2024-02-28T16:31:13.824 INFO:tasks.workunit.client.0.smithi001.stdout:0/598: read d18/fac [116303,103078] 0 2024-02-28T16:31:13.834 INFO:tasks.workunit.client.1.smithi066.stdout:5/164: dwrite d3/d1b/f31 [0,4194304] 0 2024-02-28T16:31:13.835 INFO:tasks.workunit.client.0.smithi001.stdout:0/599: link d18/d22/d3d/fa1 d18/d22/d2f/d1c/d79/fc2 0 2024-02-28T16:31:13.838 INFO:tasks.workunit.client.1.smithi066.stdout:8/155: dwrite d0/d6/f1b [0,4194304] 0 2024-02-28T16:31:13.841 INFO:tasks.workunit.client.1.smithi066.stdout:8/156: readlink d0/d6/dc/d16/d1a/l20 0 2024-02-28T16:31:13.851 INFO:tasks.workunit.client.0.smithi001.stdout:5/586: sync 2024-02-28T16:31:13.851 INFO:tasks.workunit.client.0.smithi001.stdout:8/641: sync 2024-02-28T16:31:13.851 INFO:tasks.workunit.client.0.smithi001.stdout:3/636: sync 2024-02-28T16:31:13.851 INFO:tasks.workunit.client.0.smithi001.stdout:7/603: sync 2024-02-28T16:31:13.852 INFO:tasks.workunit.client.0.smithi001.stdout:4/646: sync 2024-02-28T16:31:13.853 INFO:tasks.workunit.client.0.smithi001.stdout:1/746: sync 2024-02-28T16:31:13.853 INFO:tasks.workunit.client.0.smithi001.stdout:2/609: sync 2024-02-28T16:31:13.853 INFO:tasks.workunit.client.0.smithi001.stdout:9/708: sync 2024-02-28T16:31:13.856 INFO:tasks.workunit.client.1.smithi066.stdout:5/165: mknod d3/d1b/c3c 0 2024-02-28T16:31:13.858 INFO:tasks.workunit.client.1.smithi066.stdout:1/235: getdents d2/d27/d2b 0 2024-02-28T16:31:13.860 INFO:tasks.workunit.client.0.smithi001.stdout:9/709: write d5/d27/db1/fa4 [179866,109784] 0 2024-02-28T16:31:13.861 INFO:tasks.workunit.client.1.smithi066.stdout:8/157: rmdir d0/d6 39 2024-02-28T16:31:13.862 INFO:tasks.workunit.client.0.smithi001.stdout:2/610: creat d0/d2/d3/de/d61/d12/d55/fcd x:0 0 0 2024-02-28T16:31:13.864 INFO:tasks.workunit.client.0.smithi001.stdout:9/710: stat d5/d9a/la7 0 2024-02-28T16:31:13.867 INFO:tasks.workunit.client.0.smithi001.stdout:3/637: unlink d4/d18/d5f/c4c 0 2024-02-28T16:31:13.869 INFO:tasks.workunit.client.0.smithi001.stdout:7/604: mknod d3/d17/d52/d7f/cc0 0 2024-02-28T16:31:13.870 INFO:tasks.workunit.client.0.smithi001.stdout:5/587: mknod dc/d12/cc5 0 2024-02-28T16:31:13.872 INFO:tasks.workunit.client.0.smithi001.stdout:4/647: symlink d3/d5/d47/lcd 0 2024-02-28T16:31:13.872 INFO:tasks.workunit.client.1.smithi066.stdout:8/158: dread d0/d6/f1f [0,4194304] 0 2024-02-28T16:31:13.875 INFO:tasks.workunit.client.0.smithi001.stdout:4/648: dread - d3/d1c/da0/fc7 zero size 2024-02-28T16:31:13.877 INFO:tasks.workunit.client.0.smithi001.stdout:5/588: unlink dc/d1a/dba/fc0 0 2024-02-28T16:31:13.878 INFO:tasks.workunit.client.1.smithi066.stdout:1/236: dread d2/d18/d19/f1c [0,4194304] 0 2024-02-28T16:31:13.879 INFO:tasks.workunit.client.0.smithi001.stdout:4/649: read - d3/d5/d92/fc9 zero size 2024-02-28T16:31:13.883 INFO:tasks.workunit.client.1.smithi066.stdout:5/166: dwrite d3/d4/dc/f1d [0,4194304] 0 2024-02-28T16:31:13.884 INFO:tasks.workunit.client.1.smithi066.stdout:6/156: truncate d2/f8 3110005 0 2024-02-28T16:31:13.885 INFO:tasks.workunit.client.0.smithi001.stdout:3/638: creat d4/d1e/d7c/fd3 x:0 0 0 2024-02-28T16:31:13.885 INFO:tasks.workunit.client.1.smithi066.stdout:4/298: dread d1/f12 [4194304,4194304] 0 2024-02-28T16:31:13.888 INFO:tasks.workunit.client.0.smithi001.stdout:5/589: creat dc/d12/d41/db7/fc6 x:0 0 0 2024-02-28T16:31:13.891 INFO:tasks.workunit.client.0.smithi001.stdout:3/639: mknod d4/d6/d7f/d82/cd4 0 2024-02-28T16:31:13.894 INFO:tasks.workunit.client.0.smithi001.stdout:3/640: write d4/f26 [1039993,118925] 0 2024-02-28T16:31:13.896 INFO:tasks.workunit.client.0.smithi001.stdout:2/611: dread d0/d2/d3/de/d22/f48 [0,4194304] 0 2024-02-28T16:31:13.896 INFO:tasks.workunit.client.0.smithi001.stdout:5/590: creat dc/d1a/d21/d2d/d91/fc7 x:0 0 0 2024-02-28T16:31:13.896 INFO:tasks.workunit.client.1.smithi066.stdout:1/237: fsync d2/f5 0 2024-02-28T16:31:13.898 INFO:tasks.workunit.client.0.smithi001.stdout:3/641: write d4/d38/fbc [938260,120270] 0 2024-02-28T16:31:13.902 INFO:tasks.workunit.client.1.smithi066.stdout:1/238: read - d2/d18/f3a zero size 2024-02-28T16:31:13.902 INFO:tasks.workunit.client.0.smithi001.stdout:4/650: dwrite d3/d5/d7/d1f/d20/f9a [0,4194304] 0 2024-02-28T16:31:13.902 INFO:tasks.workunit.client.0.smithi001.stdout:5/591: write dc/d12/d41/d54/f94 [95743,79369] 0 2024-02-28T16:31:13.904 INFO:tasks.workunit.client.0.smithi001.stdout:5/592: stat dc/d12/f3f 0 2024-02-28T16:31:13.904 INFO:tasks.workunit.client.0.smithi001.stdout:3/642: truncate d4/d38/fbc 1890792 0 2024-02-28T16:31:13.905 INFO:tasks.workunit.client.0.smithi001.stdout:4/651: write d3/d5/d4e/f7b [495424,49675] 0 2024-02-28T16:31:13.906 INFO:tasks.workunit.client.1.smithi066.stdout:8/159: dwrite d0/d6/ff [4194304,4194304] 0 2024-02-28T16:31:13.909 INFO:tasks.workunit.client.1.smithi066.stdout:1/239: truncate d2/d27/f3c 688727 0 2024-02-28T16:31:13.909 INFO:tasks.workunit.client.0.smithi001.stdout:2/612: symlink d0/d2/d3/d36/lce 0 2024-02-28T16:31:13.911 INFO:tasks.workunit.client.1.smithi066.stdout:5/167: dwrite d3/d4/dc/f1e [0,4194304] 0 2024-02-28T16:31:13.912 INFO:tasks.workunit.client.1.smithi066.stdout:6/157: link d2/da/f21 d2/d26/f31 0 2024-02-28T16:31:13.914 INFO:tasks.workunit.client.1.smithi066.stdout:6/158: dread - d2/da/f1e zero size 2024-02-28T16:31:13.914 INFO:tasks.workunit.client.0.smithi001.stdout:5/593: link dc/l8e dc/d12/d41/lc8 0 2024-02-28T16:31:13.915 INFO:tasks.workunit.client.1.smithi066.stdout:2/212: dread d0/d12/d1a/d39/f42 [0,4194304] 0 2024-02-28T16:31:13.916 INFO:tasks.workunit.client.1.smithi066.stdout:4/299: creat d1/f5f x:0 0 0 2024-02-28T16:31:13.917 INFO:tasks.workunit.client.1.smithi066.stdout:3/203: sync 2024-02-28T16:31:13.920 INFO:tasks.workunit.client.1.smithi066.stdout:5/168: write d3/d4/f27 [784573,43377] 0 2024-02-28T16:31:13.920 INFO:tasks.workunit.client.1.smithi066.stdout:1/240: mknod d2/d27/d2b/d37/c4b 0 2024-02-28T16:31:13.920 INFO:tasks.workunit.client.0.smithi001.stdout:4/652: rmdir d3/d2d/d4c/d5e/db0 39 2024-02-28T16:31:13.921 INFO:tasks.workunit.client.1.smithi066.stdout:3/204: fdatasync db/dc/f2b 0 2024-02-28T16:31:13.921 INFO:tasks.workunit.client.0.smithi001.stdout:4/653: chown d3/d1c/fc2 83 1 2024-02-28T16:31:13.924 INFO:tasks.workunit.client.1.smithi066.stdout:8/160: mknod d0/d22/c2b 0 2024-02-28T16:31:13.926 INFO:tasks.workunit.client.1.smithi066.stdout:2/213: symlink d0/d12/d1a/d23/d29/l49 0 2024-02-28T16:31:13.928 INFO:tasks.workunit.client.0.smithi001.stdout:2/613: creat d0/d2/d3/de/d61/d12/d6a/d84/fcf x:0 0 0 2024-02-28T16:31:13.931 INFO:tasks.workunit.client.0.smithi001.stdout:1/747: unlink d0/d9/l1c 0 2024-02-28T16:31:13.932 INFO:tasks.workunit.client.0.smithi001.stdout:5/594: truncate dc/d1a/d21/d2d/d3c/f5e 3125387 0 2024-02-28T16:31:13.932 INFO:tasks.workunit.client.0.smithi001.stdout:1/748: fdatasync d0/fa9 0 2024-02-28T16:31:13.933 INFO:tasks.workunit.client.1.smithi066.stdout:3/205: symlink db/dc/d35/l43 0 2024-02-28T16:31:13.934 INFO:tasks.workunit.client.1.smithi066.stdout:5/169: symlink d3/d1b/d39/l3d 0 2024-02-28T16:31:13.934 INFO:tasks.workunit.client.1.smithi066.stdout:8/161: truncate d0/d6/f9 1773005 0 2024-02-28T16:31:13.935 INFO:tasks.workunit.client.0.smithi001.stdout:5/595: readlink dc/d1a/d21/d2d/d3c/l58 0 2024-02-28T16:31:13.936 INFO:tasks.workunit.client.1.smithi066.stdout:2/214: dread - d0/d12/d1a/d39/d40/f41 zero size 2024-02-28T16:31:13.937 INFO:tasks.workunit.client.0.smithi001.stdout:2/614: symlink d0/d2/d3/de/d22/dbb/ld0 0 2024-02-28T16:31:13.938 INFO:tasks.workunit.client.1.smithi066.stdout:8/162: read d0/d6/ff [7877400,93337] 0 2024-02-28T16:31:13.941 INFO:tasks.workunit.client.1.smithi066.stdout:4/300: rename d1/l1a to d1/d18/d4e/l60 0 2024-02-28T16:31:13.943 INFO:tasks.workunit.client.0.smithi001.stdout:2/615: chown d0/d2/d3/de/d61/d12/d55/fb1 104405923 1 2024-02-28T16:31:13.943 INFO:tasks.workunit.client.0.smithi001.stdout:4/654: dwrite d3/d2d/f3c [0,4194304] 0 2024-02-28T16:31:13.946 INFO:tasks.workunit.client.1.smithi066.stdout:1/241: dwrite d2/d27/d2b/d37/f49 [0,4194304] 0 2024-02-28T16:31:13.946 INFO:tasks.workunit.client.1.smithi066.stdout:3/206: mkdir db/d2a/d44 0 2024-02-28T16:31:13.948 INFO:tasks.workunit.client.1.smithi066.stdout:1/242: chown d2/d18/f3a 6 1 2024-02-28T16:31:13.949 INFO:tasks.workunit.client.1.smithi066.stdout:7/126: sync 2024-02-28T16:31:13.949 INFO:tasks.workunit.client.1.smithi066.stdout:0/291: sync 2024-02-28T16:31:13.949 INFO:tasks.workunit.client.1.smithi066.stdout:9/166: sync 2024-02-28T16:31:13.951 INFO:tasks.workunit.client.0.smithi001.stdout:1/749: truncate d0/d9/d1a/d2b/d26/d27/d30/d52/d84/f94 1585786 0 2024-02-28T16:31:13.954 INFO:tasks.workunit.client.0.smithi001.stdout:9/711: truncate d5/d27/db1/f91 2834841 0 2024-02-28T16:31:13.956 INFO:tasks.workunit.client.0.smithi001.stdout:4/655: mknod d3/d2d/d4c/d5e/db0/cce 0 2024-02-28T16:31:13.956 INFO:tasks.workunit.client.1.smithi066.stdout:1/243: read d2/d27/d2b/f3d [463785,87349] 0 2024-02-28T16:31:13.957 INFO:tasks.workunit.client.0.smithi001.stdout:9/712: readlink d5/l1c 0 2024-02-28T16:31:13.959 INFO:tasks.workunit.client.1.smithi066.stdout:8/163: mknod d0/d6/dc/d16/d1c/c2c 0 2024-02-28T16:31:13.959 INFO:tasks.workunit.client.1.smithi066.stdout:5/170: creat d3/d4/dc/dd/f3e x:0 0 0 2024-02-28T16:31:13.960 INFO:tasks.workunit.client.0.smithi001.stdout:2/616: rmdir d0/d2/d3/de/d22 39 2024-02-28T16:31:13.963 INFO:tasks.workunit.client.0.smithi001.stdout:1/750: symlink d0/d9/d1a/d2b/d26/d27/d30/d67/db7/l10b 0 2024-02-28T16:31:13.963 INFO:tasks.workunit.client.1.smithi066.stdout:5/171: write d3/d1b/f31 [1706424,101142] 0 2024-02-28T16:31:13.964 INFO:tasks.workunit.client.0.smithi001.stdout:6/575: rmdir d9/d29 39 2024-02-28T16:31:13.966 INFO:tasks.workunit.client.1.smithi066.stdout:4/301: symlink d1/d23/d25/d3e/l61 0 2024-02-28T16:31:13.967 INFO:tasks.workunit.client.0.smithi001.stdout:9/713: truncate d5/d45/d6f/f7d 4330190 0 2024-02-28T16:31:13.969 INFO:tasks.workunit.client.1.smithi066.stdout:7/127: dread fa [0,4194304] 0 2024-02-28T16:31:13.972 INFO:tasks.workunit.client.1.smithi066.stdout:9/167: rmdir da/df 39 2024-02-28T16:31:13.974 INFO:tasks.workunit.client.0.smithi001.stdout:4/656: creat d3/d5/fcf x:0 0 0 2024-02-28T16:31:13.976 INFO:tasks.workunit.client.0.smithi001.stdout:4/657: truncate d3/d1c/d58/f76 5197670 0 2024-02-28T16:31:13.978 INFO:tasks.workunit.client.0.smithi001.stdout:3/643: truncate d4/d18/d5f/f59 1733066 0 2024-02-28T16:31:13.978 INFO:tasks.workunit.client.0.smithi001.stdout:6/576: creat d9/d29/d6e/d4e/d96/fc6 x:0 0 0 2024-02-28T16:31:13.979 INFO:tasks.workunit.client.0.smithi001.stdout:9/714: creat d5/d27/d42/d48/dcf/fed x:0 0 0 2024-02-28T16:31:13.981 INFO:tasks.workunit.client.0.smithi001.stdout:4/658: creat d3/d1c/da0/fd0 x:0 0 0 2024-02-28T16:31:13.981 INFO:tasks.workunit.client.1.smithi066.stdout:6/159: chown d2/f8 499 1 2024-02-28T16:31:13.982 INFO:tasks.workunit.client.0.smithi001.stdout:4/659: chown d3/d1c/da0 114488 1 2024-02-28T16:31:13.982 INFO:tasks.workunit.client.0.smithi001.stdout:3/644: mkdir d4/d6/d7f/d82/dd5 0 2024-02-28T16:31:13.984 INFO:tasks.workunit.client.0.smithi001.stdout:9/715: mknod d5/d27/db1/d33/d8a/cee 0 2024-02-28T16:31:13.984 INFO:tasks.workunit.client.1.smithi066.stdout:5/172: creat d3/d4/dc/f3f x:0 0 0 2024-02-28T16:31:13.985 INFO:tasks.workunit.client.0.smithi001.stdout:4/660: chown d3/d5/d7/d1f/d2e/fb7 229953580 1 2024-02-28T16:31:13.985 INFO:tasks.workunit.client.1.smithi066.stdout:8/164: rename d0/d6/dc/d16/c21 to d0/db/d15/c2d 0 2024-02-28T16:31:13.986 INFO:tasks.workunit.client.0.smithi001.stdout:9/716: fsync d5/d27/db1/d33/f5d 0 2024-02-28T16:31:13.987 INFO:tasks.workunit.client.1.smithi066.stdout:5/173: fdatasync d3/d4/dc/f3f 0 2024-02-28T16:31:13.989 INFO:tasks.workunit.client.0.smithi001.stdout:8/642: rename d4/f52 to d4/d7/da8/d89/fce 0 2024-02-28T16:31:13.990 INFO:tasks.workunit.client.1.smithi066.stdout:9/168: mknod da/d10/c36 0 2024-02-28T16:31:13.992 INFO:tasks.workunit.client.0.smithi001.stdout:4/661: creat d3/d5/d7/d1f/d2e/dbe/d87/fd1 x:0 0 0 2024-02-28T16:31:13.996 INFO:tasks.workunit.client.1.smithi066.stdout:1/244: mkdir d2/d27/d4c 0 2024-02-28T16:31:13.996 INFO:tasks.workunit.client.0.smithi001.stdout:7/605: rename d3/d17/d3d/d5d/d9a to d3/d17/d52/d7f/d99/dc1 0 2024-02-28T16:31:13.997 INFO:tasks.workunit.client.0.smithi001.stdout:6/577: dwrite d9/db/d10/f5b [0,4194304] 0 2024-02-28T16:31:13.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:13 smithi066 ceph-mon[36236]: pgmap v50: 65 pgs: 65 active+clean; 1.4 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 43 MiB/s rd, 147 MiB/s wr, 314 op/s 2024-02-28T16:31:13.998 INFO:tasks.workunit.client.1.smithi066.stdout:2/215: truncate d0/d12/d1a/d39/f42 831410 0 2024-02-28T16:31:14.001 INFO:tasks.workunit.client.0.smithi001.stdout:4/662: creat d3/d1c/d58/d6c/fd2 x:0 0 0 2024-02-28T16:31:14.001 INFO:tasks.workunit.client.1.smithi066.stdout:6/160: mkdir d2/d26/d32 0 2024-02-28T16:31:14.003 INFO:tasks.workunit.client.0.smithi001.stdout:7/606: write d3/d17/d52/d7f/d99/dc1/d49/d8d/f97 [155266,34730] 0 2024-02-28T16:31:14.003 INFO:tasks.workunit.client.1.smithi066.stdout:8/165: symlink d0/d6/dc/l2e 0 2024-02-28T16:31:14.004 INFO:tasks.workunit.client.1.smithi066.stdout:0/292: creat d2/f5e x:0 0 0 2024-02-28T16:31:14.010 INFO:tasks.workunit.client.1.smithi066.stdout:5/174: dwrite d3/d4/dc/d2f/f33 [0,4194304] 0 2024-02-28T16:31:14.012 INFO:tasks.workunit.client.1.smithi066.stdout:4/302: rename d1/d18/c1c to d1/d18/c62 0 2024-02-28T16:31:14.016 INFO:tasks.workunit.client.0.smithi001.stdout:2/617: rename d0/d2/d3/d73/f9b to d0/d2/d3/de/d22/fd1 0 2024-02-28T16:31:14.016 INFO:tasks.workunit.client.1.smithi066.stdout:7/128: rename dc/de/f10 to dc/de/f26 0 2024-02-28T16:31:14.017 INFO:tasks.workunit.client.1.smithi066.stdout:6/161: mknod d2/d2a/c33 0 2024-02-28T16:31:14.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:13 smithi001 ceph-mon[28991]: pgmap v50: 65 pgs: 65 active+clean; 1.4 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 43 MiB/s rd, 147 MiB/s wr, 314 op/s 2024-02-28T16:31:14.019 INFO:tasks.workunit.client.1.smithi066.stdout:4/303: dread - d1/d23/d25/d43/f5e zero size 2024-02-28T16:31:14.020 INFO:tasks.workunit.client.0.smithi001.stdout:9/717: getdents d5/d45/d6f 0 2024-02-28T16:31:14.022 INFO:tasks.workunit.client.1.smithi066.stdout:3/207: dwrite db/d13/d1a/f1b [0,4194304] 0 2024-02-28T16:31:14.025 INFO:tasks.workunit.client.0.smithi001.stdout:1/751: dwrite d0/d9/d56/f8d [0,4194304] 0 2024-02-28T16:31:14.025 INFO:tasks.workunit.client.1.smithi066.stdout:9/169: creat da/df/d18/f37 x:0 0 0 2024-02-28T16:31:14.026 INFO:tasks.workunit.client.0.smithi001.stdout:8/643: dwrite d4/d47/d8b/fba [0,4194304] 0 2024-02-28T16:31:14.027 INFO:tasks.workunit.client.0.smithi001.stdout:9/718: write d5/f12 [585160,107725] 0 2024-02-28T16:31:14.028 INFO:tasks.workunit.client.0.smithi001.stdout:6/578: dwrite d9/d29/d6e/f34 [0,4194304] 0 2024-02-28T16:31:14.030 INFO:tasks.workunit.client.0.smithi001.stdout:3/645: rename d4/d1e/d7e to d4/d6/d7/d78/d81/dd6 0 2024-02-28T16:31:14.032 INFO:tasks.workunit.client.1.smithi066.stdout:2/216: mkdir d0/d12/d4a 0 2024-02-28T16:31:14.032 INFO:tasks.workunit.client.1.smithi066.stdout:1/245: rmdir d2/d27 39 2024-02-28T16:31:14.033 INFO:tasks.workunit.client.0.smithi001.stdout:9/719: write d5/f64 [2214659,96282] 0 2024-02-28T16:31:14.035 INFO:tasks.workunit.client.1.smithi066.stdout:5/175: unlink d3/d4/c2a 0 2024-02-28T16:31:14.035 INFO:tasks.workunit.client.1.smithi066.stdout:7/129: mknod dc/de/d17/c27 0 2024-02-28T16:31:14.036 INFO:tasks.workunit.client.1.smithi066.stdout:4/304: symlink d1/d23/d25/d3e/d40/l63 0 2024-02-28T16:31:14.036 INFO:tasks.workunit.client.0.smithi001.stdout:6/579: truncate d9/db/d10/f16 3512268 0 2024-02-28T16:31:14.037 INFO:tasks.workunit.client.1.smithi066.stdout:2/217: read - d0/d12/d1a/f3a zero size 2024-02-28T16:31:14.040 INFO:tasks.workunit.client.1.smithi066.stdout:3/208: symlink db/d13/d1a/d33/l45 0 2024-02-28T16:31:14.040 INFO:tasks.workunit.client.0.smithi001.stdout:1/752: unlink d0/d9/d1a/d2b/d26/d27/d30/d67/fca 0 2024-02-28T16:31:14.045 INFO:tasks.workunit.client.0.smithi001.stdout:1/753: write d0/d9/d1a/d2b/d26/d27/d30/d52/f5a [1488717,7981] 0 2024-02-28T16:31:14.046 INFO:tasks.workunit.client.1.smithi066.stdout:7/130: creat dc/de/d17/d1e/f28 x:0 0 0 2024-02-28T16:31:14.049 INFO:tasks.workunit.client.1.smithi066.stdout:7/131: chown dc/de/d17/d1e/f24 214 1 2024-02-28T16:31:14.049 INFO:tasks.workunit.client.1.smithi066.stdout:4/305: creat d1/d23/d25/d3e/d40/f64 x:0 0 0 2024-02-28T16:31:14.050 INFO:tasks.workunit.client.0.smithi001.stdout:9/720: mkdir d5/d27/d42/def 0 2024-02-28T16:31:14.050 INFO:tasks.workunit.client.1.smithi066.stdout:0/293: dwrite d2/d9/d10/f2c [4194304,4194304] 0 2024-02-28T16:31:14.053 INFO:tasks.workunit.client.0.smithi001.stdout:9/721: fsync d5/d27/d42/d48/fab 0 2024-02-28T16:31:14.054 INFO:tasks.workunit.client.0.smithi001.stdout:6/580: rename d9/de/c4a to d9/de/dbd/cc7 0 2024-02-28T16:31:14.056 INFO:tasks.workunit.client.0.smithi001.stdout:8/644: dread d4/d7/d22/d27/d58/f7b [0,4194304] 0 2024-02-28T16:31:14.056 INFO:tasks.workunit.client.1.smithi066.stdout:0/294: write d2/d9/d10/f47 [1076655,126453] 0 2024-02-28T16:31:14.058 INFO:tasks.workunit.client.0.smithi001.stdout:0/600: sync 2024-02-28T16:31:14.058 INFO:tasks.workunit.client.1.smithi066.stdout:2/218: mknod d0/d12/d1a/d39/d40/c4b 0 2024-02-28T16:31:14.059 INFO:tasks.workunit.client.0.smithi001.stdout:6/581: chown d9/db/d19/l22 268926 1 2024-02-28T16:31:14.060 INFO:tasks.workunit.client.1.smithi066.stdout:5/176: creat d3/d4/dc/dd/d2e/f40 x:0 0 0 2024-02-28T16:31:14.062 INFO:tasks.workunit.client.1.smithi066.stdout:8/166: getdents d0/db/d15 0 2024-02-28T16:31:14.063 INFO:tasks.workunit.client.0.smithi001.stdout:1/754: rename d0/d9/d1a/d2b/d26/d27/d30/d67/d87/cd5 to d0/d9/d1a/d2b/d26/d27/dd3/de8/d102/c10c 0 2024-02-28T16:31:14.067 INFO:tasks.workunit.client.1.smithi066.stdout:7/132: mkdir dc/d29 0 2024-02-28T16:31:14.068 INFO:tasks.workunit.client.0.smithi001.stdout:9/722: rename d5/d27/db1/d58/ce5 to d5/d27/db1/d58/cf0 0 2024-02-28T16:31:14.068 INFO:tasks.workunit.client.1.smithi066.stdout:9/170: dwrite da/fb [0,4194304] 0 2024-02-28T16:31:14.068 INFO:tasks.workunit.client.1.smithi066.stdout:1/246: chown d2/d27/d2b/d34/c42 429265 1 2024-02-28T16:31:14.070 INFO:tasks.workunit.client.0.smithi001.stdout:0/601: symlink d18/d22/d76/d7a/lc3 0 2024-02-28T16:31:14.071 INFO:tasks.workunit.client.0.smithi001.stdout:4/663: write d3/f7e [142851,94626] 0 2024-02-28T16:31:14.072 INFO:tasks.workunit.client.0.smithi001.stdout:9/723: chown d5/d27/db1/d5e/dd3 58003 1 2024-02-28T16:31:14.074 INFO:tasks.workunit.client.0.smithi001.stdout:6/582: creat d9/db/d5f/d69/fc8 x:0 0 0 2024-02-28T16:31:14.075 INFO:tasks.workunit.client.0.smithi001.stdout:9/724: readlink d5/l1c 0 2024-02-28T16:31:14.078 INFO:tasks.workunit.client.0.smithi001.stdout:3/646: dread d4/d6/f56 [0,4194304] 0 2024-02-28T16:31:14.079 INFO:tasks.workunit.client.0.smithi001.stdout:4/664: write d3/d1c/da0/fb2 [691134,90945] 0 2024-02-28T16:31:14.080 INFO:tasks.workunit.client.0.smithi001.stdout:1/755: creat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f10d x:0 0 0 2024-02-28T16:31:14.081 INFO:tasks.workunit.client.0.smithi001.stdout:6/583: creat d9/d29/d6e/d53/fc9 x:0 0 0 2024-02-28T16:31:14.083 INFO:tasks.workunit.client.1.smithi066.stdout:4/306: creat d1/d23/d25/d43/f65 x:0 0 0 2024-02-28T16:31:14.083 INFO:tasks.workunit.client.1.smithi066.stdout:0/295: rename d2/d27/d41/c56 to d2/d9/c5f 0 2024-02-28T16:31:14.084 INFO:tasks.workunit.client.0.smithi001.stdout:4/665: dread - d3/d2d/d81/fae zero size 2024-02-28T16:31:14.085 INFO:tasks.workunit.client.0.smithi001.stdout:7/607: dwrite d3/d17/f3b [0,4194304] 0 2024-02-28T16:31:14.085 INFO:tasks.workunit.client.1.smithi066.stdout:9/171: write da/fc [1013337,110790] 0 2024-02-28T16:31:14.086 INFO:tasks.workunit.client.0.smithi001.stdout:0/602: link fd d18/d22/d76/d7a/fc4 0 2024-02-28T16:31:14.088 INFO:tasks.workunit.client.1.smithi066.stdout:0/296: fsync d2/f28 0 2024-02-28T16:31:14.088 INFO:tasks.workunit.client.1.smithi066.stdout:3/209: symlink db/d13/d14/l46 0 2024-02-28T16:31:14.092 INFO:tasks.workunit.client.0.smithi001.stdout:6/584: mknod d9/d29/d6e/d9a/cca 0 2024-02-28T16:31:14.092 INFO:tasks.workunit.client.0.smithi001.stdout:1/756: chown d0/d9/d1a/d2b/d26/d27/dd3/de8/d102 1173813681 1 2024-02-28T16:31:14.094 INFO:tasks.workunit.client.0.smithi001.stdout:7/608: mknod d3/d17/d52/d7f/d99/dc1/cc2 0 2024-02-28T16:31:14.096 INFO:tasks.workunit.client.0.smithi001.stdout:4/666: creat d3/d1c/d58/d6c/d79/dc8/fd3 x:0 0 0 2024-02-28T16:31:14.096 INFO:tasks.workunit.client.0.smithi001.stdout:9/725: getdents d5/d27/db1/d5e 0 2024-02-28T16:31:14.097 INFO:tasks.workunit.client.0.smithi001.stdout:6/585: chown d9/db/d19/c8d 468 1 2024-02-28T16:31:14.099 INFO:tasks.workunit.client.0.smithi001.stdout:3/647: dread d4/d1e/d7c/da8/fae [0,4194304] 0 2024-02-28T16:31:14.099 INFO:tasks.workunit.client.0.smithi001.stdout:7/609: fdatasync d3/d17/d3d/f70 0 2024-02-28T16:31:14.099 INFO:tasks.workunit.client.0.smithi001.stdout:9/726: chown d5/f12 1670338504 1 2024-02-28T16:31:14.100 INFO:tasks.workunit.client.1.smithi066.stdout:2/219: dwrite d0/d12/d1a/f2b [0,4194304] 0 2024-02-28T16:31:14.101 INFO:tasks.workunit.client.0.smithi001.stdout:7/610: dread - d3/d17/fa5 zero size 2024-02-28T16:31:14.103 INFO:tasks.workunit.client.1.smithi066.stdout:0/297: mkdir d2/d9/d10/d50/d60 0 2024-02-28T16:31:14.107 INFO:tasks.workunit.client.1.smithi066.stdout:8/167: dwrite d0/db/d12/d1d/f29 [0,4194304] 0 2024-02-28T16:31:14.107 INFO:tasks.workunit.client.1.smithi066.stdout:9/172: mknod da/df/d18/d28/c38 0 2024-02-28T16:31:14.108 INFO:tasks.workunit.client.1.smithi066.stdout:0/298: chown d2/d9/d10 57 1 2024-02-28T16:31:14.112 INFO:tasks.workunit.client.0.smithi001.stdout:6/586: creat d9/db/d5f/d89/fcb x:0 0 0 2024-02-28T16:31:14.113 INFO:tasks.workunit.client.1.smithi066.stdout:3/210: mknod db/dc/d35/c47 0 2024-02-28T16:31:14.113 INFO:tasks.workunit.client.0.smithi001.stdout:7/611: truncate d3/f8c 47530 0 2024-02-28T16:31:14.115 INFO:tasks.workunit.client.0.smithi001.stdout:9/727: mkdir d5/d45/df1 0 2024-02-28T16:31:14.122 INFO:tasks.workunit.client.1.smithi066.stdout:0/299: chown d2/d9/f4f 71 1 2024-02-28T16:31:14.129 INFO:tasks.workunit.client.1.smithi066.stdout:7/133: dwrite dc/f16 [0,4194304] 0 2024-02-28T16:31:14.131 INFO:tasks.workunit.client.1.smithi066.stdout:3/211: mkdir db/d2a/d44/d48 0 2024-02-28T16:31:14.131 INFO:tasks.workunit.client.0.smithi001.stdout:8/645: write d4/d7/d22/d27/d4b/f78 [4995402,59811] 0 2024-02-28T16:31:14.135 INFO:tasks.workunit.client.0.smithi001.stdout:0/603: dwrite d18/d22/d2f/fa4 [0,4194304] 0 2024-02-28T16:31:14.135 INFO:tasks.workunit.client.1.smithi066.stdout:1/247: dwrite d2/d18/d19/f3b [0,4194304] 0 2024-02-28T16:31:14.136 INFO:tasks.workunit.client.1.smithi066.stdout:4/307: dread d1/d23/f36 [0,4194304] 0 2024-02-28T16:31:14.139 INFO:tasks.workunit.client.0.smithi001.stdout:6/587: creat d9/d29/fcc x:0 0 0 2024-02-28T16:31:14.141 INFO:tasks.workunit.client.0.smithi001.stdout:7/612: fdatasync d3/d17/d3d/d59/d6e/fac 0 2024-02-28T16:31:14.145 INFO:tasks.workunit.client.0.smithi001.stdout:2/618: dwrite d0/f63 [4194304,4194304] 0 2024-02-28T16:31:14.146 INFO:tasks.workunit.client.0.smithi001.stdout:4/667: dwrite d3/d5/d92/fc9 [0,4194304] 0 2024-02-28T16:31:14.146 INFO:tasks.workunit.client.0.smithi001.stdout:5/596: sync 2024-02-28T16:31:14.149 INFO:tasks.workunit.client.1.smithi066.stdout:3/212: mknod db/dc/d11/c49 0 2024-02-28T16:31:14.155 INFO:tasks.workunit.client.1.smithi066.stdout:5/177: truncate d3/d4/dc/f1d 3712883 0 2024-02-28T16:31:14.157 INFO:tasks.workunit.client.0.smithi001.stdout:4/668: chown d3/d1c 25 1 2024-02-28T16:31:14.157 INFO:tasks.workunit.client.0.smithi001.stdout:8/646: creat d4/d32/d69/d9a/fcf x:0 0 0 2024-02-28T16:31:14.159 INFO:tasks.workunit.client.0.smithi001.stdout:0/604: chown d18/d22/d76/d7a/cb7 2 1 2024-02-28T16:31:14.165 INFO:tasks.workunit.client.0.smithi001.stdout:6/588: rename d9/de/dbd/cc7 to d9/d39/ccd 0 2024-02-28T16:31:14.165 INFO:tasks.workunit.client.1.smithi066.stdout:9/173: dread da/f30 [0,4194304] 0 2024-02-28T16:31:14.165 INFO:tasks.workunit.client.1.smithi066.stdout:7/134: creat dc/de/d17/d1e/f2a x:0 0 0 2024-02-28T16:31:14.169 INFO:tasks.workunit.client.1.smithi066.stdout:4/308: readlink d1/l1f 0 2024-02-28T16:31:14.171 INFO:tasks.workunit.client.1.smithi066.stdout:5/178: mkdir d3/d4/dc/dd/d41 0 2024-02-28T16:31:14.172 INFO:tasks.workunit.client.1.smithi066.stdout:2/220: dwrite d0/d12/d1a/d39/d40/f45 [0,4194304] 0 2024-02-28T16:31:14.172 INFO:tasks.workunit.client.0.smithi001.stdout:3/648: dwrite d4/d6/d7/d78/d81/d8a/fa5 [0,4194304] 0 2024-02-28T16:31:14.174 INFO:tasks.workunit.client.0.smithi001.stdout:5/597: truncate dc/d12/d56/f61 970878 0 2024-02-28T16:31:14.176 INFO:tasks.workunit.client.1.smithi066.stdout:7/135: symlink dc/de/d17/d1e/l2b 0 2024-02-28T16:31:14.177 INFO:tasks.workunit.client.1.smithi066.stdout:7/136: stat dc/c1d 0 2024-02-28T16:31:14.178 INFO:tasks.workunit.client.0.smithi001.stdout:4/669: rename d3/d5/c25 to d3/d1c/d58/d6c/cd4 0 2024-02-28T16:31:14.179 INFO:tasks.workunit.client.0.smithi001.stdout:6/589: chown d9/d29/d6a/f85 33553 1 2024-02-28T16:31:14.179 INFO:tasks.workunit.client.0.smithi001.stdout:0/605: dread - d18/d22/d2f/d1c/d79/fc2 zero size 2024-02-28T16:31:14.182 INFO:tasks.workunit.client.1.smithi066.stdout:8/168: dwrite d0/f23 [0,4194304] 0 2024-02-28T16:31:14.184 INFO:tasks.workunit.client.0.smithi001.stdout:7/613: dwrite d3/d1c/f33 [0,4194304] 0 2024-02-28T16:31:14.189 INFO:tasks.workunit.client.0.smithi001.stdout:8/647: creat d4/d7/d22/d27/d8e/dae/fd0 x:0 0 0 2024-02-28T16:31:14.190 INFO:tasks.workunit.client.1.smithi066.stdout:3/213: link db/dc/f2b db/d13/d14/d24/d2f/f4a 0 2024-02-28T16:31:14.190 INFO:tasks.workunit.client.0.smithi001.stdout:3/649: write d4/d6/d7/f63 [3642020,688] 0 2024-02-28T16:31:14.192 INFO:tasks.workunit.client.1.smithi066.stdout:9/174: link da/fb da/d22/f39 0 2024-02-28T16:31:14.192 INFO:tasks.workunit.client.1.smithi066.stdout:8/169: mknod d0/d6/c2f 0 2024-02-28T16:31:14.193 INFO:tasks.workunit.client.0.smithi001.stdout:7/614: chown d3/c15 173 1 2024-02-28T16:31:14.198 INFO:tasks.workunit.client.0.smithi001.stdout:1/757: dwrite d0/d9/d1a/fbe [0,4194304] 0 2024-02-28T16:31:14.198 INFO:tasks.workunit.client.0.smithi001.stdout:9/728: dread d5/d45/d6f/fbd [0,4194304] 0 2024-02-28T16:31:14.198 INFO:tasks.workunit.client.0.smithi001.stdout:6/590: truncate d9/d29/f3b 28855 0 2024-02-28T16:31:14.199 INFO:tasks.workunit.client.0.smithi001.stdout:7/615: write d3/d1c/f33 [871275,116260] 0 2024-02-28T16:31:14.199 INFO:tasks.workunit.client.1.smithi066.stdout:3/214: write db/dc/d11/f26 [1012979,18517] 0 2024-02-28T16:31:14.199 INFO:tasks.workunit.client.1.smithi066.stdout:2/221: getdents d0 0 2024-02-28T16:31:14.201 INFO:tasks.workunit.client.1.smithi066.stdout:8/170: symlink d0/d6/dc/d16/l30 0 2024-02-28T16:31:14.201 INFO:tasks.workunit.client.0.smithi001.stdout:7/616: chown d3 8 1 2024-02-28T16:31:14.202 INFO:tasks.workunit.client.0.smithi001.stdout:6/591: chown d9/db/d19/f97 12899 1 2024-02-28T16:31:14.202 INFO:tasks.workunit.client.0.smithi001.stdout:8/648: write d4/d7/d22/d27/d4b/d80/f85 [3424138,64603] 0 2024-02-28T16:31:14.209 INFO:tasks.workunit.client.1.smithi066.stdout:7/137: dwrite dc/f16 [0,4194304] 0 2024-02-28T16:31:14.210 INFO:tasks.workunit.client.0.smithi001.stdout:5/598: getdents dc/df 0 2024-02-28T16:31:14.210 INFO:tasks.workunit.client.0.smithi001.stdout:6/592: readlink d9/db/d10/la1 0 2024-02-28T16:31:14.211 INFO:tasks.workunit.client.0.smithi001.stdout:1/758: read - d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/fcd zero size 2024-02-28T16:31:14.213 INFO:tasks.workunit.client.1.smithi066.stdout:4/309: dwrite d1/d18/f38 [0,4194304] 0 2024-02-28T16:31:14.215 INFO:tasks.workunit.client.1.smithi066.stdout:2/222: write d0/f1e [664218,11205] 0 2024-02-28T16:31:14.215 INFO:tasks.workunit.client.1.smithi066.stdout:9/175: rename da/d10 to da/d3a 0 2024-02-28T16:31:14.218 INFO:tasks.workunit.client.1.smithi066.stdout:8/171: creat d0/db/d15/f31 x:0 0 0 2024-02-28T16:31:14.218 INFO:tasks.workunit.client.1.smithi066.stdout:2/223: chown d0/d12/d4a 183037717 1 2024-02-28T16:31:14.219 INFO:tasks.workunit.client.0.smithi001.stdout:9/729: unlink d5/d27/d42/d48/fab 0 2024-02-28T16:31:14.219 INFO:tasks.workunit.client.1.smithi066.stdout:2/224: fsync d0/f44 0 2024-02-28T16:31:14.224 INFO:tasks.workunit.client.0.smithi001.stdout:8/649: readlink d4/l9 0 2024-02-28T16:31:14.230 INFO:tasks.workunit.client.1.smithi066.stdout:9/176: creat da/df/d18/f3b x:0 0 0 2024-02-28T16:31:14.230 INFO:tasks.workunit.client.1.smithi066.stdout:8/172: mkdir d0/db/d15/d32 0 2024-02-28T16:31:14.231 INFO:tasks.workunit.client.0.smithi001.stdout:8/650: creat d4/d32/fd1 x:0 0 0 2024-02-28T16:31:14.232 INFO:tasks.workunit.client.0.smithi001.stdout:3/650: dwrite d4/d6/d7/d78/d81/f96 [0,4194304] 0 2024-02-28T16:31:14.237 INFO:tasks.workunit.client.1.smithi066.stdout:8/173: rmdir d0/d8 39 2024-02-28T16:31:14.239 INFO:tasks.workunit.client.1.smithi066.stdout:7/138: link dc/de/d17/d1e/l2b dc/l2c 0 2024-02-28T16:31:14.242 INFO:tasks.workunit.client.1.smithi066.stdout:3/215: dread db/dc/f2b [0,4194304] 0 2024-02-28T16:31:14.244 INFO:tasks.workunit.client.1.smithi066.stdout:7/139: write dc/f25 [1008955,3577] 0 2024-02-28T16:31:14.245 INFO:tasks.workunit.client.1.smithi066.stdout:9/177: mknod da/df/d18/c3c 0 2024-02-28T16:31:14.247 INFO:tasks.workunit.client.0.smithi001.stdout:8/651: truncate d4/d7/f35 2295870 0 2024-02-28T16:31:14.248 INFO:tasks.workunit.client.1.smithi066.stdout:4/310: dwrite d1/d23/d25/d43/f5d [0,4194304] 0 2024-02-28T16:31:14.251 INFO:tasks.workunit.client.1.smithi066.stdout:8/174: symlink d0/d6/dc/d16/d1a/l33 0 2024-02-28T16:31:14.254 INFO:tasks.workunit.client.1.smithi066.stdout:2/225: dwrite d0/ff [0,4194304] 0 2024-02-28T16:31:14.254 INFO:tasks.workunit.client.1.smithi066.stdout:3/216: symlink db/d13/d14/d2d/l4b 0 2024-02-28T16:31:14.255 INFO:tasks.workunit.client.1.smithi066.stdout:9/178: rmdir da/df/d18/d28 39 2024-02-28T16:31:14.256 INFO:tasks.workunit.client.1.smithi066.stdout:7/140: truncate f5 862275 0 2024-02-28T16:31:14.257 INFO:tasks.workunit.client.1.smithi066.stdout:4/311: creat d1/d23/d25/d3e/f66 x:0 0 0 2024-02-28T16:31:14.258 INFO:tasks.workunit.client.0.smithi001.stdout:8/652: stat d4/d7/d22/d27/f4a 0 2024-02-28T16:31:14.259 INFO:tasks.workunit.client.0.smithi001.stdout:6/593: dwrite d9/d29/d6a/f77 [0,4194304] 0 2024-02-28T16:31:14.260 INFO:tasks.workunit.client.1.smithi066.stdout:4/312: chown d1/d23/f3d 1784315 1 2024-02-28T16:31:14.261 INFO:tasks.workunit.client.0.smithi001.stdout:1/759: dread d0/d9/d1a/d2b/d26/d27/d30/d52/fd0 [0,4194304] 0 2024-02-28T16:31:14.262 INFO:tasks.workunit.client.0.smithi001.stdout:5/599: dwrite dc/d12/d41/f9c [4194304,4194304] 0 2024-02-28T16:31:14.264 INFO:tasks.workunit.client.0.smithi001.stdout:1/760: chown d0 41618 1 2024-02-28T16:31:14.264 INFO:tasks.workunit.client.1.smithi066.stdout:7/141: truncate dc/de/d17/d1e/f28 677660 0 2024-02-28T16:31:14.269 INFO:tasks.workunit.client.1.smithi066.stdout:9/179: creat da/df/f3d x:0 0 0 2024-02-28T16:31:14.272 INFO:tasks.workunit.client.0.smithi001.stdout:3/651: link d4/d18/c4a d4/d1e/d7c/d9a/cd7 0 2024-02-28T16:31:14.272 INFO:tasks.workunit.client.0.smithi001.stdout:4/670: truncate d3/d5/d7/d1f/d2e/f86 4213 0 2024-02-28T16:31:14.277 INFO:tasks.workunit.client.0.smithi001.stdout:6/594: creat d9/d29/d6e/d4e/d96/fce x:0 0 0 2024-02-28T16:31:14.277 INFO:tasks.workunit.client.1.smithi066.stdout:4/313: symlink d1/d23/l67 0 2024-02-28T16:31:14.277 INFO:tasks.workunit.client.1.smithi066.stdout:6/162: sync 2024-02-28T16:31:14.278 INFO:tasks.workunit.client.0.smithi001.stdout:0/606: dwrite d18/d22/d2f/d1c/f30 [0,4194304] 0 2024-02-28T16:31:14.280 INFO:tasks.workunit.client.1.smithi066.stdout:4/314: fsync d1/d18/d4e/f50 0 2024-02-28T16:31:14.281 INFO:tasks.workunit.client.1.smithi066.stdout:3/217: mknod db/d13/d14/c4c 0 2024-02-28T16:31:14.281 INFO:tasks.workunit.client.1.smithi066.stdout:8/175: dread d0/f13 [0,4194304] 0 2024-02-28T16:31:14.289 INFO:tasks.workunit.client.0.smithi001.stdout:8/653: dread d4/fa1 [0,4194304] 0 2024-02-28T16:31:14.289 INFO:tasks.workunit.client.0.smithi001.stdout:0/607: write d18/d22/d57/d7e/f96 [3361502,67339] 0 2024-02-28T16:31:14.290 INFO:tasks.workunit.client.1.smithi066.stdout:9/180: unlink da/df/d18/l1b 0 2024-02-28T16:31:14.290 INFO:tasks.workunit.client.1.smithi066.stdout:3/218: truncate db/dc/d11/d1f/f2e 4866295 0 2024-02-28T16:31:14.290 INFO:tasks.workunit.client.0.smithi001.stdout:9/730: truncate d5/d27/db1/f41 1066958 0 2024-02-28T16:31:14.291 INFO:tasks.workunit.client.1.smithi066.stdout:0/300: truncate d2/d9/f11 3626620 0 2024-02-28T16:31:14.294 INFO:tasks.workunit.client.1.smithi066.stdout:4/315: creat d1/d18/d41/f68 x:0 0 0 2024-02-28T16:31:14.296 INFO:tasks.workunit.client.1.smithi066.stdout:9/181: symlink da/d3a/l3e 0 2024-02-28T16:31:14.297 INFO:tasks.workunit.client.1.smithi066.stdout:0/301: write d2/d9/d10/d50/d51/f53 [990121,51548] 0 2024-02-28T16:31:14.297 INFO:tasks.workunit.client.1.smithi066.stdout:6/163: mknod d2/d26/d32/c34 0 2024-02-28T16:31:14.298 INFO:tasks.workunit.client.1.smithi066.stdout:3/219: truncate db/dc/d35/d3d/f41 115663 0 2024-02-28T16:31:14.299 INFO:tasks.workunit.client.0.smithi001.stdout:5/600: dwrite dc/d12/d41/d54/f94 [0,4194304] 0 2024-02-28T16:31:14.300 INFO:tasks.workunit.client.0.smithi001.stdout:7/617: dwrite d3/d17/d3d/d42/f53 [0,4194304] 0 2024-02-28T16:31:14.303 INFO:tasks.workunit.client.1.smithi066.stdout:4/316: chown d1/d23/d25/f59 2070159 1 2024-02-28T16:31:14.305 INFO:tasks.workunit.client.1.smithi066.stdout:0/302: write d2/d9/d10/f2c [8065216,70726] 0 2024-02-28T16:31:14.308 INFO:tasks.workunit.client.1.smithi066.stdout:8/176: dread d0/f13 [0,4194304] 0 2024-02-28T16:31:14.309 INFO:tasks.workunit.client.1.smithi066.stdout:6/164: write d2/da/f1a [6798083,123636] 0 2024-02-28T16:31:14.314 INFO:tasks.workunit.client.0.smithi001.stdout:1/761: write d0/d9/d1a/d2b/dfe/f109 [1376321,32880] 0 2024-02-28T16:31:14.316 INFO:tasks.workunit.client.1.smithi066.stdout:3/220: symlink db/dc/d11/l4d 0 2024-02-28T16:31:14.318 INFO:tasks.workunit.client.1.smithi066.stdout:3/221: chown db/d13/f18 3705592 1 2024-02-28T16:31:14.319 INFO:tasks.workunit.client.1.smithi066.stdout:7/142: rmdir dc/de 39 2024-02-28T16:31:14.321 INFO:tasks.workunit.client.1.smithi066.stdout:7/143: fsync dc/f16 0 2024-02-28T16:31:14.323 INFO:tasks.workunit.client.1.smithi066.stdout:3/222: write db/dc/d11/d1f/f2e [1014428,56763] 0 2024-02-28T16:31:14.325 INFO:tasks.workunit.client.1.smithi066.stdout:0/303: rename d2/d27/d41/d5b to d2/d9/d10/d50/d52/d61 0 2024-02-28T16:31:14.327 INFO:tasks.workunit.client.1.smithi066.stdout:6/165: rmdir d2/da/d19 39 2024-02-28T16:31:14.327 INFO:tasks.workunit.client.1.smithi066.stdout:0/304: chown d2/c2a 311 1 2024-02-28T16:31:14.331 INFO:tasks.workunit.client.1.smithi066.stdout:4/317: dread d1/d23/f36 [0,4194304] 0 2024-02-28T16:31:14.331 INFO:tasks.workunit.client.0.smithi001.stdout:3/652: unlink d4/d6/d7/d78/d81/f96 0 2024-02-28T16:31:14.334 INFO:tasks.workunit.client.0.smithi001.stdout:4/671: rename d3/d2d/d83/l99 to d3/d5/d92/ld5 0 2024-02-28T16:31:14.335 INFO:tasks.workunit.client.1.smithi066.stdout:8/177: dwrite d0/d6/dc/f14 [0,4194304] 0 2024-02-28T16:31:14.335 INFO:tasks.workunit.client.0.smithi001.stdout:7/618: rename d3/d17/d52/d7f/d99/dc1/d49/d8d/db9 to d3/d17/d52/d7f/d99/dc1/d49/d8d/db9/dc3 22 2024-02-28T16:31:14.335 INFO:tasks.workunit.client.0.smithi001.stdout:3/653: chown d4/d6/d7/d78/fc5 4 1 2024-02-28T16:31:14.339 INFO:tasks.workunit.client.0.smithi001.stdout:0/608: rmdir d18/d22/d2f/d1c 39 2024-02-28T16:31:14.342 INFO:tasks.workunit.client.1.smithi066.stdout:6/166: creat d2/d2a/f35 x:0 0 0 2024-02-28T16:31:14.343 INFO:tasks.workunit.client.0.smithi001.stdout:0/609: write d18/d22/d76/d93/f94 [3107438,2816] 0 2024-02-28T16:31:14.347 INFO:tasks.workunit.client.1.smithi066.stdout:0/305: dread - d2/d9/d10/f59 zero size 2024-02-28T16:31:14.347 INFO:tasks.workunit.client.0.smithi001.stdout:1/762: rmdir d0/d9/d1a 39 2024-02-28T16:31:14.348 INFO:tasks.workunit.client.1.smithi066.stdout:4/318: creat d1/d18/d41/f69 x:0 0 0 2024-02-28T16:31:14.348 INFO:tasks.workunit.client.1.smithi066.stdout:7/144: creat dc/de/d17/d1e/f2d x:0 0 0 2024-02-28T16:31:14.349 INFO:tasks.workunit.client.0.smithi001.stdout:6/595: symlink d9/d29/d6e/lcf 0 2024-02-28T16:31:14.352 INFO:tasks.workunit.client.0.smithi001.stdout:6/596: read - d9/d29/d6e/d53/f9e zero size 2024-02-28T16:31:14.353 INFO:tasks.workunit.client.0.smithi001.stdout:7/619: mknod d3/d17/d52/cc4 0 2024-02-28T16:31:14.353 INFO:tasks.workunit.client.0.smithi001.stdout:2/619: sync 2024-02-28T16:31:14.354 INFO:tasks.workunit.client.1.smithi066.stdout:8/178: mknod d0/d6/dc/d16/d1c/c34 0 2024-02-28T16:31:14.355 INFO:tasks.workunit.client.1.smithi066.stdout:7/145: chown dc/de/d17/f19 17309 1 2024-02-28T16:31:14.358 INFO:tasks.workunit.client.1.smithi066.stdout:8/179: write d0/f24 [1258084,6567] 0 2024-02-28T16:31:14.358 INFO:tasks.workunit.client.1.smithi066.stdout:9/182: getdents da/df/d18/d28 0 2024-02-28T16:31:14.359 INFO:tasks.workunit.client.0.smithi001.stdout:8/654: creat d4/d7/d22/d27/d8e/dae/dbe/fd2 x:0 0 0 2024-02-28T16:31:14.367 INFO:tasks.workunit.client.1.smithi066.stdout:9/183: chown da/df/f2f 6 1 2024-02-28T16:31:14.367 INFO:tasks.workunit.client.0.smithi001.stdout:7/620: fdatasync d3/f8c 0 2024-02-28T16:31:14.368 INFO:tasks.workunit.client.1.smithi066.stdout:2/226: dwrite d0/d12/d1a/d39/d40/f45 [4194304,4194304] 0 2024-02-28T16:31:14.371 INFO:tasks.workunit.client.0.smithi001.stdout:7/621: truncate d3/d17/d3d/fa6 887040 0 2024-02-28T16:31:14.375 INFO:tasks.workunit.client.1.smithi066.stdout:8/180: creat d0/d6/dc/d16/d1a/f35 x:0 0 0 2024-02-28T16:31:14.377 INFO:tasks.workunit.client.1.smithi066.stdout:6/167: dwrite d2/f29 [0,4194304] 0 2024-02-28T16:31:14.377 INFO:tasks.workunit.client.1.smithi066.stdout:3/223: write db/dc/d11/f22 [1028544,41462] 0 2024-02-28T16:31:14.384 INFO:tasks.workunit.client.0.smithi001.stdout:2/620: dwrite d0/d2/d3/de/d8a/fbd [0,4194304] 0 2024-02-28T16:31:14.386 INFO:tasks.workunit.client.0.smithi001.stdout:8/655: link d4/d7/l66 d4/d9b/ld3 0 2024-02-28T16:31:14.389 INFO:tasks.workunit.client.1.smithi066.stdout:8/181: chown d0/db/d15 1980 1 2024-02-28T16:31:14.391 INFO:tasks.workunit.client.0.smithi001.stdout:1/763: getdents d0/d9/d1a/d2b/d26/d27/d30/d67/db7 0 2024-02-28T16:31:14.394 INFO:tasks.workunit.client.1.smithi066.stdout:4/319: dwrite d1/f20 [0,4194304] 0 2024-02-28T16:31:14.395 INFO:tasks.workunit.client.0.smithi001.stdout:7/622: dwrite d3/d17/f1b [0,4194304] 0 2024-02-28T16:31:14.396 INFO:tasks.workunit.client.1.smithi066.stdout:2/227: link d0/f31 d0/d12/d1a/f4c 0 2024-02-28T16:31:14.397 INFO:tasks.workunit.client.1.smithi066.stdout:0/306: dwrite d2/f5e [0,4194304] 0 2024-02-28T16:31:14.399 INFO:tasks.workunit.client.1.smithi066.stdout:4/320: chown d1/d18/d41/f47 0 1 2024-02-28T16:31:14.399 INFO:tasks.workunit.client.1.smithi066.stdout:7/146: getdents dc/de 0 2024-02-28T16:31:14.401 INFO:tasks.workunit.client.1.smithi066.stdout:6/168: readlink d2/l16 0 2024-02-28T16:31:14.403 INFO:tasks.workunit.client.0.smithi001.stdout:1/764: write d0/d9/d1a/d2b/d26/d27/d30/d88/fe4 [3031225,93066] 0 2024-02-28T16:31:14.407 INFO:tasks.workunit.client.0.smithi001.stdout:4/672: write d3/d5/d7/d1f/d2e/f59 [4270735,76339] 0 2024-02-28T16:31:14.409 INFO:tasks.workunit.client.1.smithi066.stdout:8/182: symlink d0/d6/dc/l36 0 2024-02-28T16:31:14.411 INFO:tasks.workunit.client.1.smithi066.stdout:0/307: creat d2/d9/d33/f62 x:0 0 0 2024-02-28T16:31:14.412 INFO:tasks.workunit.client.0.smithi001.stdout:2/621: getdents d0/d2/d3/de/d22/dbb/dc8 0 2024-02-28T16:31:14.415 INFO:tasks.workunit.client.0.smithi001.stdout:6/597: write d9/f86 [1023884,62422] 0 2024-02-28T16:31:14.417 INFO:tasks.workunit.client.0.smithi001.stdout:1/765: chown d0/d9/d1a/d2b/d26/d27/d30/d67/d87/f9a 78848167 1 2024-02-28T16:31:14.417 INFO:tasks.workunit.client.1.smithi066.stdout:8/183: chown d0/d6/dc/f14 196 1 2024-02-28T16:31:14.418 INFO:tasks.workunit.client.0.smithi001.stdout:3/654: truncate d4/d18/f11 1473960 0 2024-02-28T16:31:14.418 INFO:tasks.workunit.client.1.smithi066.stdout:7/147: rmdir dc 39 2024-02-28T16:31:14.419 INFO:tasks.workunit.client.1.smithi066.stdout:7/148: readlink l0 0 2024-02-28T16:31:14.420 INFO:tasks.workunit.client.0.smithi001.stdout:5/601: dwrite dc/df/f11 [0,4194304] 0 2024-02-28T16:31:14.420 INFO:tasks.workunit.client.0.smithi001.stdout:1/766: stat d0/d9/d56/d2f/f64 0 2024-02-28T16:31:14.425 INFO:tasks.workunit.client.1.smithi066.stdout:4/321: dwrite d1/d23/f27 [0,4194304] 0 2024-02-28T16:31:14.426 INFO:tasks.workunit.client.0.smithi001.stdout:3/655: stat d4/d6/d7/f88 0 2024-02-28T16:31:14.428 INFO:tasks.workunit.client.0.smithi001.stdout:6/598: truncate d9/d39/fa8 590029 0 2024-02-28T16:31:14.431 INFO:tasks.workunit.client.1.smithi066.stdout:4/322: write d1/d18/d4e/f50 [918057,33550] 0 2024-02-28T16:31:14.433 INFO:tasks.workunit.client.0.smithi001.stdout:9/731: dread d5/f6 [0,4194304] 0 2024-02-28T16:31:14.434 INFO:tasks.workunit.client.1.smithi066.stdout:8/184: getdents d0/db/d15 0 2024-02-28T16:31:14.435 INFO:tasks.workunit.client.0.smithi001.stdout:7/623: dread d3/d17/f1b [0,4194304] 0 2024-02-28T16:31:14.435 INFO:tasks.workunit.client.0.smithi001.stdout:5/602: write dc/d1a/d21/d2d/d3c/d4a/fc1 [897640,59345] 0 2024-02-28T16:31:14.436 INFO:tasks.workunit.client.1.smithi066.stdout:0/308: link d2/d9/d10/d50/d51/l5c d2/d9/d10/d50/l63 0 2024-02-28T16:31:14.437 INFO:tasks.workunit.client.0.smithi001.stdout:4/673: rmdir d3/d5/d7/db3 0 2024-02-28T16:31:14.439 INFO:tasks.workunit.client.1.smithi066.stdout:0/309: mkdir d2/d9/d10/d50/d64 0 2024-02-28T16:31:14.443 INFO:tasks.workunit.client.0.smithi001.stdout:2/622: getdents d0/d2/d3/de/d22/dbb 0 2024-02-28T16:31:14.445 INFO:tasks.workunit.client.1.smithi066.stdout:0/310: mkdir d2/d9/d10/d50/d52/d61/d65 0 2024-02-28T16:31:14.450 INFO:tasks.workunit.client.0.smithi001.stdout:0/610: dwrite d18/d22/d2f/d1c/f21 [4194304,4194304] 0 2024-02-28T16:31:14.455 INFO:tasks.workunit.client.0.smithi001.stdout:5/603: write dc/d1a/d21/d2d/d3c/d4a/f82 [978735,108102] 0 2024-02-28T16:31:14.456 INFO:tasks.workunit.client.0.smithi001.stdout:7/624: chown d3/d17/d3d/db3/fb4 182 1 2024-02-28T16:31:14.457 INFO:tasks.workunit.client.1.smithi066.stdout:5/179: sync 2024-02-28T16:31:14.457 INFO:tasks.workunit.client.1.smithi066.stdout:1/248: sync 2024-02-28T16:31:14.458 INFO:tasks.workunit.client.0.smithi001.stdout:4/674: truncate d3/d1c/d58/f28 1856064 0 2024-02-28T16:31:14.458 INFO:tasks.workunit.client.1.smithi066.stdout:8/185: dwrite d0/d6/f7 [0,4194304] 0 2024-02-28T16:31:14.463 INFO:tasks.workunit.client.1.smithi066.stdout:1/249: read d2/f17 [549646,13922] 0 2024-02-28T16:31:14.463 INFO:tasks.workunit.client.0.smithi001.stdout:4/675: read - d3/f8e zero size 2024-02-28T16:31:14.470 INFO:tasks.workunit.client.1.smithi066.stdout:8/186: creat d0/db/d15/f37 x:0 0 0 2024-02-28T16:31:14.471 INFO:tasks.workunit.client.0.smithi001.stdout:8/656: dwrite d4/d9b/fa7 [4194304,4194304] 0 2024-02-28T16:31:14.471 INFO:tasks.workunit.client.0.smithi001.stdout:6/599: dwrite d9/d29/d6e/d4e/d96/fce [0,4194304] 0 2024-02-28T16:31:14.471 INFO:tasks.workunit.client.0.smithi001.stdout:5/604: chown dc/d1a/d21/d2d/d3c/d4a/fc1 38525 1 2024-02-28T16:31:14.472 INFO:tasks.workunit.client.1.smithi066.stdout:5/180: dread d3/d4/f27 [0,4194304] 0 2024-02-28T16:31:14.476 INFO:tasks.workunit.client.0.smithi001.stdout:6/600: readlink d9/d29/d6e/d4e/l9f 0 2024-02-28T16:31:14.476 INFO:tasks.workunit.client.1.smithi066.stdout:2/228: truncate d0/d12/f36 3514220 0 2024-02-28T16:31:14.479 INFO:tasks.workunit.client.1.smithi066.stdout:1/250: mknod d2/d27/d2b/d43/d46/c4d 0 2024-02-28T16:31:14.480 INFO:tasks.workunit.client.0.smithi001.stdout:4/676: stat d3/d2d/d4c 0 2024-02-28T16:31:14.481 INFO:tasks.workunit.client.1.smithi066.stdout:2/229: chown d0/d12/l1b 9915 1 2024-02-28T16:31:14.481 INFO:tasks.workunit.client.1.smithi066.stdout:1/251: read d2/d27/f3c [162089,94684] 0 2024-02-28T16:31:14.484 INFO:tasks.workunit.client.1.smithi066.stdout:5/181: mkdir d3/d4/dc/d42 0 2024-02-28T16:31:14.485 INFO:tasks.workunit.client.0.smithi001.stdout:5/605: rename dc/d12/d56 to dc/d12/d56/d96/dc9 22 2024-02-28T16:31:14.486 INFO:tasks.workunit.client.1.smithi066.stdout:0/311: write d2/d9/d10/f59 [912243,96515] 0 2024-02-28T16:31:14.486 INFO:tasks.workunit.client.1.smithi066.stdout:2/230: write d0/d12/d1a/f3a [1012275,81641] 0 2024-02-28T16:31:14.487 INFO:tasks.workunit.client.0.smithi001.stdout:8/657: creat d4/d7/d22/d27/fd4 x:0 0 0 2024-02-28T16:31:14.488 INFO:tasks.workunit.client.0.smithi001.stdout:1/767: dread d0/d9/d1a/d2b/dfe/ff0 [0,4194304] 0 2024-02-28T16:31:14.490 INFO:tasks.workunit.client.1.smithi066.stdout:5/182: write d3/d4/fb [4372315,113073] 0 2024-02-28T16:31:14.490 INFO:tasks.workunit.client.1.smithi066.stdout:0/312: stat d2/d9/f40 0 2024-02-28T16:31:14.491 INFO:tasks.workunit.client.1.smithi066.stdout:1/252: chown d2/l36 5145988 1 2024-02-28T16:31:14.494 INFO:tasks.workunit.client.0.smithi001.stdout:9/732: dwrite d5/d27/fb9 [0,4194304] 0 2024-02-28T16:31:14.497 INFO:tasks.workunit.client.1.smithi066.stdout:2/231: creat d0/d12/d2e/d38/f4d x:0 0 0 2024-02-28T16:31:14.498 INFO:tasks.workunit.client.1.smithi066.stdout:0/313: fdatasync d2/d9/f11 0 2024-02-28T16:31:14.503 INFO:tasks.workunit.client.0.smithi001.stdout:1/768: chown d0/d9/d1a/d2b/d26/d27/dd3/de8/d102 17 1 2024-02-28T16:31:14.506 INFO:tasks.workunit.client.1.smithi066.stdout:1/253: rename d2/f15 to d2/d27/d2b/d43/f4e 0 2024-02-28T16:31:14.506 INFO:tasks.workunit.client.0.smithi001.stdout:4/677: creat d3/fd6 x:0 0 0 2024-02-28T16:31:14.506 INFO:tasks.workunit.client.0.smithi001.stdout:8/658: write d4/d7/d22/d27/d58/f75 [387385,82041] 0 2024-02-28T16:31:14.509 INFO:tasks.workunit.client.0.smithi001.stdout:5/606: rename dc/d1a/d21/d2d to dc/d12/d56/d96/db9/dc2/dca 0 2024-02-28T16:31:14.512 INFO:tasks.workunit.client.1.smithi066.stdout:2/232: creat d0/d12/d3e/f4e x:0 0 0 2024-02-28T16:31:14.512 INFO:tasks.workunit.client.1.smithi066.stdout:0/314: symlink d2/l66 0 2024-02-28T16:31:14.514 INFO:tasks.workunit.client.0.smithi001.stdout:7/625: dwrite d3/d17/d52/fbc [0,4194304] 0 2024-02-28T16:31:14.515 INFO:tasks.workunit.client.1.smithi066.stdout:8/187: getdents d0/db/d15 0 2024-02-28T16:31:14.520 INFO:tasks.workunit.client.0.smithi001.stdout:1/769: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d88/fef 0 2024-02-28T16:31:14.522 INFO:tasks.workunit.client.1.smithi066.stdout:0/315: mkdir d2/d9/d10/d50/d67 0 2024-02-28T16:31:14.526 INFO:tasks.workunit.client.1.smithi066.stdout:5/183: dwrite d3/d4/f27 [0,4194304] 0 2024-02-28T16:31:14.527 INFO:tasks.workunit.client.0.smithi001.stdout:7/626: mkdir d3/dc5 0 2024-02-28T16:31:14.527 INFO:tasks.workunit.client.1.smithi066.stdout:6/169: dread d2/da/f21 [0,4194304] 0 2024-02-28T16:31:14.529 INFO:tasks.workunit.client.0.smithi001.stdout:6/601: dread d9/d29/d6a/f77 [0,4194304] 0 2024-02-28T16:31:14.529 INFO:tasks.workunit.client.0.smithi001.stdout:1/770: chown d0/d9/d1a/d2b/d26/d27/d30/d67/dd6/fea 393437 1 2024-02-28T16:31:14.529 INFO:tasks.workunit.client.0.smithi001.stdout:3/656: dread d4/d6/f51 [0,4194304] 0 2024-02-28T16:31:14.534 INFO:tasks.workunit.client.0.smithi001.stdout:4/678: creat d3/d2d/fd7 x:0 0 0 2024-02-28T16:31:14.534 INFO:tasks.workunit.client.1.smithi066.stdout:2/233: unlink d0/d12/d1a/d23/d29/d21/f46 0 2024-02-28T16:31:14.537 INFO:tasks.workunit.client.1.smithi066.stdout:0/316: symlink d2/d9/d33/d3f/l68 0 2024-02-28T16:31:14.539 INFO:tasks.workunit.client.0.smithi001.stdout:8/659: truncate d4/d7/f20 1278563 0 2024-02-28T16:31:14.540 INFO:tasks.workunit.client.1.smithi066.stdout:0/317: chown d2/d9/f1e 3714927 1 2024-02-28T16:31:14.545 INFO:tasks.workunit.client.1.smithi066.stdout:1/254: dread d2/d18/d19/f3b [0,4194304] 0 2024-02-28T16:31:14.549 INFO:tasks.workunit.client.0.smithi001.stdout:1/771: rename d0/d9/d1a/d72/d9d to d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e 0 2024-02-28T16:31:14.550 INFO:tasks.workunit.client.1.smithi066.stdout:2/234: unlink d0/d12/d1a/d23/d29/l48 0 2024-02-28T16:31:14.552 INFO:tasks.workunit.client.0.smithi001.stdout:8/660: mkdir d4/d7/d22/d27/d58/d94/db4/dd5 0 2024-02-28T16:31:14.553 INFO:tasks.workunit.client.1.smithi066.stdout:5/184: mknod d3/d4/dc/d42/c43 0 2024-02-28T16:31:14.553 INFO:tasks.workunit.client.0.smithi001.stdout:0/611: dread d18/d22/d2b/d36/d48/f55 [0,4194304] 0 2024-02-28T16:31:14.554 INFO:tasks.workunit.client.0.smithi001.stdout:4/679: rename d3/d5/d4e/l55 to d3/d5/d7/d1f/d2e/dbe/d87/ld8 0 2024-02-28T16:31:14.555 INFO:tasks.workunit.client.1.smithi066.stdout:6/170: dwrite d2/da/f1a [0,4194304] 0 2024-02-28T16:31:14.556 INFO:tasks.workunit.client.1.smithi066.stdout:0/318: creat d2/d9/d10/d50/d52/f69 x:0 0 0 2024-02-28T16:31:14.558 INFO:tasks.workunit.client.1.smithi066.stdout:0/319: chown d2/d9/f40 28009 1 2024-02-28T16:31:14.559 INFO:tasks.workunit.client.0.smithi001.stdout:5/607: write dc/df/f5d [452732,130145] 0 2024-02-28T16:31:14.559 INFO:tasks.workunit.client.0.smithi001.stdout:2/623: dread d0/d2/d3/de/d8a/f71 [0,4194304] 0 2024-02-28T16:31:14.560 INFO:tasks.workunit.client.0.smithi001.stdout:8/661: read d4/d7/d22/d27/d4b/d77/f83 [447914,22977] 0 2024-02-28T16:31:14.562 INFO:tasks.workunit.client.1.smithi066.stdout:0/320: truncate d2/d9/d10/f4e 185582 0 2024-02-28T16:31:14.563 INFO:tasks.workunit.client.1.smithi066.stdout:5/185: creat d3/d1b/d39/f44 x:0 0 0 2024-02-28T16:31:14.567 INFO:tasks.workunit.client.0.smithi001.stdout:6/602: dwrite d9/db/d5f/d8b/f6d [0,4194304] 0 2024-02-28T16:31:14.567 INFO:tasks.workunit.client.1.smithi066.stdout:8/188: truncate d0/d6/dc/f14 4591302 0 2024-02-28T16:31:14.568 INFO:tasks.workunit.client.1.smithi066.stdout:2/235: creat d0/d12/d4a/f4f x:0 0 0 2024-02-28T16:31:14.569 INFO:tasks.workunit.client.1.smithi066.stdout:8/189: readlink d0/d6/dc/l2e 0 2024-02-28T16:31:14.570 INFO:tasks.workunit.client.0.smithi001.stdout:6/603: readlink d9/da6/l81 0 2024-02-28T16:31:14.571 INFO:tasks.workunit.client.1.smithi066.stdout:0/321: readlink d2/l12 0 2024-02-28T16:31:14.572 INFO:tasks.workunit.client.0.smithi001.stdout:4/680: creat d3/d2d/d4c/d5e/db0/fd9 x:0 0 0 2024-02-28T16:31:14.572 INFO:tasks.workunit.client.0.smithi001.stdout:2/624: rmdir d0/d2/d3/de/d8a 39 2024-02-28T16:31:14.573 INFO:tasks.workunit.client.1.smithi066.stdout:8/190: write d0/db/d15/f31 [373418,43578] 0 2024-02-28T16:31:14.574 INFO:tasks.workunit.client.1.smithi066.stdout:5/186: dread d3/d4/dc/f21 [0,4194304] 0 2024-02-28T16:31:14.575 INFO:tasks.workunit.client.1.smithi066.stdout:1/255: write d2/d18/d19/f1c [1188444,48038] 0 2024-02-28T16:31:14.575 INFO:tasks.workunit.client.1.smithi066.stdout:8/191: chown d0/db/d12/d1d/d25 32 1 2024-02-28T16:31:14.575 INFO:tasks.workunit.client.1.smithi066.stdout:5/187: readlink d3/l14 0 2024-02-28T16:31:14.576 INFO:tasks.workunit.client.1.smithi066.stdout:6/171: truncate f1 858995 0 2024-02-28T16:31:14.577 INFO:tasks.workunit.client.1.smithi066.stdout:2/236: symlink d0/d12/d1a/d23/d29/d21/l50 0 2024-02-28T16:31:14.577 INFO:tasks.workunit.client.1.smithi066.stdout:0/322: fsync d2/d9/d10/d50/d52/f55 0 2024-02-28T16:31:14.579 INFO:tasks.workunit.client.1.smithi066.stdout:8/192: mkdir d0/db/d12/d38 0 2024-02-28T16:31:14.580 INFO:tasks.workunit.client.0.smithi001.stdout:0/612: creat d18/d31/fc5 x:0 0 0 2024-02-28T16:31:14.581 INFO:tasks.workunit.client.1.smithi066.stdout:5/188: symlink d3/d4/dc/dd/l45 0 2024-02-28T16:31:14.583 INFO:tasks.workunit.client.1.smithi066.stdout:8/193: chown d0/db/c11 24463 1 2024-02-28T16:31:14.583 INFO:tasks.workunit.client.0.smithi001.stdout:9/733: dwrite d5/d27/db1/f41 [0,4194304] 0 2024-02-28T16:31:14.584 INFO:tasks.workunit.client.0.smithi001.stdout:5/608: dread dc/df/f3a [0,4194304] 0 2024-02-28T16:31:14.585 INFO:tasks.workunit.client.0.smithi001.stdout:7/627: symlink d3/d17/d52/lc6 0 2024-02-28T16:31:14.585 INFO:tasks.workunit.client.1.smithi066.stdout:2/237: creat d0/d12/d1a/d23/d29/d21/d2f/f51 x:0 0 0 2024-02-28T16:31:14.585 INFO:tasks.workunit.client.1.smithi066.stdout:6/172: rmdir d2/da/d19 39 2024-02-28T16:31:14.587 INFO:tasks.workunit.client.1.smithi066.stdout:2/238: chown d0/d12/d1a/d23/d29/c33 8 1 2024-02-28T16:31:14.589 INFO:tasks.workunit.client.1.smithi066.stdout:2/239: chown d0/f19 8320529 1 2024-02-28T16:31:14.590 INFO:tasks.workunit.client.0.smithi001.stdout:3/657: dread d4/d6/d7/f20 [0,4194304] 0 2024-02-28T16:31:14.590 INFO:tasks.workunit.client.1.smithi066.stdout:0/323: rename d2/d9/d10/d50/d52/d61/d65 to d2/d9/d10/d50/d64/d6a 0 2024-02-28T16:31:14.595 INFO:tasks.workunit.client.0.smithi001.stdout:8/662: rename d4/d7/d22/d27/d8e/dae/dbe to d4/d32/dd6 0 2024-02-28T16:31:14.597 INFO:tasks.workunit.client.1.smithi066.stdout:2/240: truncate d0/f31 3082 0 2024-02-28T16:31:14.598 INFO:tasks.workunit.client.1.smithi066.stdout:0/324: chown d2/d9/d33/l4c 2059780 1 2024-02-28T16:31:14.599 INFO:tasks.workunit.client.1.smithi066.stdout:6/173: creat d2/da/f36 x:0 0 0 2024-02-28T16:31:14.600 INFO:tasks.workunit.client.0.smithi001.stdout:2/625: dwrite d0/d2/d3/fca [0,4194304] 0 2024-02-28T16:31:14.600 INFO:tasks.workunit.client.1.smithi066.stdout:8/194: link d0/d6/f1f d0/db/d15/d32/f39 0 2024-02-28T16:31:14.601 INFO:tasks.workunit.client.1.smithi066.stdout:5/189: rmdir d3/d4/dc/dd 39 2024-02-28T16:31:14.602 INFO:tasks.workunit.client.1.smithi066.stdout:1/256: truncate d2/f16 3033549 0 2024-02-28T16:31:14.603 INFO:tasks.workunit.client.0.smithi001.stdout:6/604: dread d9/d29/d6e/d4e/d96/fce [0,4194304] 0 2024-02-28T16:31:14.603 INFO:tasks.workunit.client.0.smithi001.stdout:1/772: truncate d0/d9/d56/d2f/f7c 3788047 0 2024-02-28T16:31:14.604 INFO:tasks.workunit.client.1.smithi066.stdout:5/190: write d3/d1b/f2d [828009,80326] 0 2024-02-28T16:31:14.604 INFO:tasks.workunit.client.1.smithi066.stdout:0/325: creat d2/d9/d33/f6b x:0 0 0 2024-02-28T16:31:14.608 INFO:tasks.workunit.client.0.smithi001.stdout:0/613: mknod d18/d22/d2b/cc6 0 2024-02-28T16:31:14.611 INFO:tasks.workunit.client.1.smithi066.stdout:5/191: chown d3/c30 5858688 1 2024-02-28T16:31:14.612 INFO:tasks.workunit.client.1.smithi066.stdout:6/174: mknod d2/da/c37 0 2024-02-28T16:31:14.612 INFO:tasks.workunit.client.0.smithi001.stdout:7/628: read d3/f8c [17187,25974] 0 2024-02-28T16:31:14.613 INFO:tasks.workunit.client.0.smithi001.stdout:7/629: dread - d3/d17/d3d/d59/fb1 zero size 2024-02-28T16:31:14.613 INFO:tasks.workunit.client.1.smithi066.stdout:0/326: write d2/d9/d10/d50/d51/f53 [2040298,16218] 0 2024-02-28T16:31:14.614 INFO:tasks.workunit.client.1.smithi066.stdout:5/192: chown d3/d1b/d39 3845478 1 2024-02-28T16:31:14.615 INFO:tasks.workunit.client.0.smithi001.stdout:9/734: symlink d5/d45/d6f/dc8/lf2 0 2024-02-28T16:31:14.616 INFO:tasks.workunit.client.1.smithi066.stdout:6/175: truncate d2/f29 5063655 0 2024-02-28T16:31:14.617 INFO:tasks.workunit.client.1.smithi066.stdout:9/184: sync 2024-02-28T16:31:14.619 INFO:tasks.workunit.client.1.smithi066.stdout:9/185: chown da/df/d18/c3c 15 1 2024-02-28T16:31:14.620 INFO:tasks.workunit.client.1.smithi066.stdout:0/327: write d2/d9/d10/f47 [3070251,114278] 0 2024-02-28T16:31:14.622 INFO:tasks.workunit.client.0.smithi001.stdout:1/773: dwrite d0/d9/d1a/f4c [4194304,4194304] 0 2024-02-28T16:31:14.624 INFO:tasks.workunit.client.1.smithi066.stdout:5/193: write d3/d4/dc/dd/d2e/d18/f2b [1805458,120540] 0 2024-02-28T16:31:14.624 INFO:tasks.workunit.client.0.smithi001.stdout:3/658: rename d4/d6/d7/d23/f2d to d4/d1e/d7c/da8/fd8 0 2024-02-28T16:31:14.625 INFO:tasks.workunit.client.1.smithi066.stdout:9/186: symlink da/d3a/l3f 0 2024-02-28T16:31:14.627 INFO:tasks.workunit.client.0.smithi001.stdout:4/681: write d3/d1c/d58/f6b [2131714,18771] 0 2024-02-28T16:31:14.629 INFO:tasks.workunit.client.0.smithi001.stdout:2/626: truncate d0/d2/d3/de/d61/f1f 632237 0 2024-02-28T16:31:14.631 INFO:tasks.workunit.client.0.smithi001.stdout:4/682: truncate d3/f7e 285135 0 2024-02-28T16:31:14.632 INFO:tasks.workunit.client.0.smithi001.stdout:2/627: fdatasync d0/d2/d3/d36/f4b 0 2024-02-28T16:31:14.633 INFO:tasks.workunit.client.1.smithi066.stdout:5/194: truncate d3/d4/dc/f3f 234424 0 2024-02-28T16:31:14.634 INFO:tasks.workunit.client.1.smithi066.stdout:1/257: dwrite d2/d27/d2b/d37/f49 [4194304,4194304] 0 2024-02-28T16:31:14.635 INFO:tasks.workunit.client.0.smithi001.stdout:4/683: dread - d3/d2d/d81/fae zero size 2024-02-28T16:31:14.636 INFO:tasks.workunit.client.1.smithi066.stdout:5/195: write d3/d4/fb [243520,114876] 0 2024-02-28T16:31:14.636 INFO:tasks.workunit.client.0.smithi001.stdout:2/628: chown d0/d2/d3/f43 282018777 1 2024-02-28T16:31:14.637 INFO:tasks.workunit.client.1.smithi066.stdout:2/241: dread d0/f37 [0,4194304] 0 2024-02-28T16:31:14.638 INFO:tasks.workunit.client.0.smithi001.stdout:0/614: creat d18/d22/d2f/d1c/fc7 x:0 0 0 2024-02-28T16:31:14.638 INFO:tasks.workunit.client.0.smithi001.stdout:5/609: truncate dc/d12/f16 1792696 0 2024-02-28T16:31:14.641 INFO:tasks.workunit.client.1.smithi066.stdout:2/242: chown d0/f1f 10643291 1 2024-02-28T16:31:14.642 INFO:tasks.workunit.client.0.smithi001.stdout:5/610: write dc/df/f11 [565767,656] 0 2024-02-28T16:31:14.645 INFO:tasks.workunit.client.0.smithi001.stdout:1/774: creat d0/d9/d1a/d2b/d26/d27/d30/d67/db7/f10f x:0 0 0 2024-02-28T16:31:14.649 INFO:tasks.workunit.client.0.smithi001.stdout:3/659: mkdir d4/d6/d7/d78/d81/dd6/dd9 0 2024-02-28T16:31:14.653 INFO:tasks.workunit.client.0.smithi001.stdout:4/684: mkdir d3/d1c/dda 0 2024-02-28T16:31:14.655 INFO:tasks.workunit.client.0.smithi001.stdout:5/611: creat dc/d1a/d21/fcb x:0 0 0 2024-02-28T16:31:14.658 INFO:tasks.workunit.client.1.smithi066.stdout:9/187: getdents da/d3a/d15 0 2024-02-28T16:31:14.661 INFO:tasks.workunit.client.1.smithi066.stdout:9/188: mknod da/d21/c40 0 2024-02-28T16:31:14.662 INFO:tasks.workunit.client.1.smithi066.stdout:5/196: link d3/d4/l13 d3/d4/l46 0 2024-02-28T16:31:14.663 INFO:tasks.workunit.client.1.smithi066.stdout:1/258: dread d2/d18/f26 [0,4194304] 0 2024-02-28T16:31:14.663 INFO:tasks.workunit.client.1.smithi066.stdout:9/189: truncate da/f1e 2080090 0 2024-02-28T16:31:14.664 INFO:tasks.workunit.client.0.smithi001.stdout:6/605: dread d9/f4f [0,4194304] 0 2024-02-28T16:31:14.665 INFO:tasks.workunit.client.1.smithi066.stdout:5/197: read d3/d4/dc/f3f [233694,17931] 0 2024-02-28T16:31:14.665 INFO:tasks.workunit.client.1.smithi066.stdout:7/149: sync 2024-02-28T16:31:14.665 INFO:tasks.workunit.client.1.smithi066.stdout:3/224: sync 2024-02-28T16:31:14.665 INFO:tasks.workunit.client.1.smithi066.stdout:4/323: sync 2024-02-28T16:31:14.671 INFO:tasks.workunit.client.1.smithi066.stdout:1/259: chown d2/d27/d2b/d37/c4b 3096 1 2024-02-28T16:31:14.672 INFO:tasks.workunit.client.1.smithi066.stdout:3/225: creat db/d13/d1a/f4e x:0 0 0 2024-02-28T16:31:14.673 INFO:tasks.workunit.client.0.smithi001.stdout:6/606: chown d9/d29/l98 63005 1 2024-02-28T16:31:14.676 INFO:tasks.workunit.client.0.smithi001.stdout:6/607: fdatasync f5 0 2024-02-28T16:31:14.676 INFO:tasks.workunit.client.1.smithi066.stdout:4/324: write d1/d18/d41/f58 [42361,122794] 0 2024-02-28T16:31:14.676 INFO:tasks.workunit.client.1.smithi066.stdout:5/198: rename d3/d4/f27 to d3/d1b/d39/f47 0 2024-02-28T16:31:14.678 INFO:tasks.workunit.client.1.smithi066.stdout:9/190: link da/df/f34 da/d3a/f41 0 2024-02-28T16:31:14.679 INFO:tasks.workunit.client.1.smithi066.stdout:2/243: write d0/d12/f47 [1082147,67399] 0 2024-02-28T16:31:14.681 INFO:tasks.workunit.client.0.smithi001.stdout:6/608: chown d9/db/d5f/d62/d83/f8f 0 1 2024-02-28T16:31:14.682 INFO:tasks.workunit.client.0.smithi001.stdout:2/629: rename d0/d2/d3/d73/l90 to d0/d2/d3/de/ld2 0 2024-02-28T16:31:14.682 INFO:tasks.workunit.client.0.smithi001.stdout:9/735: dwrite d5/f30 [0,4194304] 0 2024-02-28T16:31:14.683 INFO:tasks.workunit.client.1.smithi066.stdout:5/199: chown d3/d4/dc/f1e 1494 1 2024-02-28T16:31:14.687 INFO:tasks.workunit.client.1.smithi066.stdout:8/195: dwrite d0/d6/f1f [0,4194304] 0 2024-02-28T16:31:14.689 INFO:tasks.workunit.client.0.smithi001.stdout:2/630: dread - d0/f91 zero size 2024-02-28T16:31:14.690 INFO:tasks.workunit.client.0.smithi001.stdout:0/615: rename d18/d22/d2b/l51 to d18/d22/d2f/d1c/d90/lc8 0 2024-02-28T16:31:14.690 INFO:tasks.workunit.client.0.smithi001.stdout:3/660: write d4/d1e/d7c/da8/faa [5232023,106636] 0 2024-02-28T16:31:14.691 INFO:tasks.workunit.client.1.smithi066.stdout:5/200: unlink d3/c22 0 2024-02-28T16:31:14.695 INFO:tasks.workunit.client.0.smithi001.stdout:6/609: write d9/d29/d6e/d4e/d96/fc0 [826266,72907] 0 2024-02-28T16:31:14.699 INFO:tasks.workunit.client.0.smithi001.stdout:2/631: stat d0/d2/d3/de/d61/d12/d6a/f81 0 2024-02-28T16:31:14.701 INFO:tasks.workunit.client.1.smithi066.stdout:5/201: fdatasync d3/d4/dc/d2f/f33 0 2024-02-28T16:31:14.702 INFO:tasks.workunit.client.1.smithi066.stdout:8/196: truncate d0/db/d15/f31 572966 0 2024-02-28T16:31:14.703 INFO:tasks.workunit.client.0.smithi001.stdout:4/685: dread d3/d1c/d58/f46 [0,4194304] 0 2024-02-28T16:31:14.704 INFO:tasks.workunit.client.0.smithi001.stdout:9/736: rename d5/l1c to d5/d4d/lf3 0 2024-02-28T16:31:14.704 INFO:tasks.workunit.client.0.smithi001.stdout:7/630: dwrite d3/d17/f32 [0,4194304] 0 2024-02-28T16:31:14.707 INFO:tasks.workunit.client.1.smithi066.stdout:0/328: dwrite d2/f22 [0,4194304] 0 2024-02-28T16:31:14.708 INFO:tasks.workunit.client.1.smithi066.stdout:8/197: readlink d0/d8/l28 0 2024-02-28T16:31:14.713 INFO:tasks.workunit.client.1.smithi066.stdout:5/202: mknod d3/d4/dc/d42/c48 0 2024-02-28T16:31:14.715 INFO:tasks.workunit.client.0.smithi001.stdout:6/610: write d9/db/d19/f1f [4953487,74460] 0 2024-02-28T16:31:14.716 INFO:tasks.workunit.client.1.smithi066.stdout:1/260: link d2/f16 d2/d18/f4f 0 2024-02-28T16:31:14.717 INFO:tasks.workunit.client.0.smithi001.stdout:3/661: rmdir d4/d6/d7/dc6 0 2024-02-28T16:31:14.719 INFO:tasks.workunit.client.0.smithi001.stdout:6/611: chown d9/db/d19/f90 245 1 2024-02-28T16:31:14.719 INFO:tasks.workunit.client.0.smithi001.stdout:5/612: dwrite dc/d12/d56/d96/db9/dc2/dca/f44 [0,4194304] 0 2024-02-28T16:31:14.722 INFO:tasks.workunit.client.0.smithi001.stdout:0/616: rename d18/d22/d47 to d18/da9/dc9 0 2024-02-28T16:31:14.723 INFO:tasks.workunit.client.1.smithi066.stdout:3/226: dwrite db/d13/f18 [0,4194304] 0 2024-02-28T16:31:14.725 INFO:tasks.workunit.client.1.smithi066.stdout:0/329: rename d2/d9/d10/d50/c54 to d2/d9/d10/d23/c6c 0 2024-02-28T16:31:14.729 INFO:tasks.workunit.client.1.smithi066.stdout:4/325: dwrite d1/f4f [0,4194304] 0 2024-02-28T16:31:14.729 INFO:tasks.workunit.client.1.smithi066.stdout:0/330: chown d2/d9/d33/d3f/l68 9 1 2024-02-28T16:31:14.730 INFO:tasks.workunit.client.1.smithi066.stdout:9/191: dwrite da/df/d18/f37 [0,4194304] 0 2024-02-28T16:31:14.730 INFO:tasks.workunit.client.0.smithi001.stdout:7/631: creat d3/d17/d3d/d5d/fc7 x:0 0 0 2024-02-28T16:31:14.735 INFO:tasks.workunit.client.0.smithi001.stdout:4/686: mkdir d3/d5/d7/ddb 0 2024-02-28T16:31:14.738 INFO:tasks.workunit.client.0.smithi001.stdout:9/737: creat d5/d27/db1/d33/d8a/da8/ff4 x:0 0 0 2024-02-28T16:31:14.740 INFO:tasks.workunit.client.0.smithi001.stdout:2/632: creat d0/d2/d3/de/d61/d12/d55/db2/fd3 x:0 0 0 2024-02-28T16:31:14.741 INFO:tasks.workunit.client.1.smithi066.stdout:3/227: creat db/d2a/d44/f4f x:0 0 0 2024-02-28T16:31:14.746 INFO:tasks.workunit.client.1.smithi066.stdout:0/331: mkdir d2/d9/d10/d23/d6d 0 2024-02-28T16:31:14.749 INFO:tasks.workunit.client.1.smithi066.stdout:5/203: dread d3/d4/dc/d2f/f33 [0,4194304] 0 2024-02-28T16:31:14.753 INFO:tasks.workunit.client.1.smithi066.stdout:5/204: truncate d3/d4/dc/dd/f3e 111911 0 2024-02-28T16:31:14.754 INFO:tasks.workunit.client.1.smithi066.stdout:4/326: mknod d1/d18/c6a 0 2024-02-28T16:31:14.754 INFO:tasks.workunit.client.0.smithi001.stdout:0/617: mknod d18/d22/d2f/d1c/cca 0 2024-02-28T16:31:14.754 INFO:tasks.workunit.client.0.smithi001.stdout:1/775: dwrite d0/d9/d56/d2f/f7c [0,4194304] 0 2024-02-28T16:31:14.754 INFO:tasks.workunit.client.1.smithi066.stdout:2/244: truncate d0/d12/d1a/d39/d40/f45 627607 0 2024-02-28T16:31:14.756 INFO:tasks.workunit.client.0.smithi001.stdout:3/662: rename d4/d6/c53 to d4/d6/d7/d23/cda 0 2024-02-28T16:31:14.756 INFO:tasks.workunit.client.0.smithi001.stdout:0/618: write f13 [228026,53820] 0 2024-02-28T16:31:14.758 INFO:tasks.workunit.client.0.smithi001.stdout:2/633: write d0/d2/d3/d36/f4b [1185232,129093] 0 2024-02-28T16:31:14.760 INFO:tasks.workunit.client.0.smithi001.stdout:0/619: chown d18/d31/d4f/lbf 39929420 1 2024-02-28T16:31:14.762 INFO:tasks.workunit.client.0.smithi001.stdout:4/687: mkdir d3/d1c/d58/ddc 0 2024-02-28T16:31:14.762 INFO:tasks.workunit.client.0.smithi001.stdout:6/612: dwrite d9/d29/d6a/f85 [0,4194304] 0 2024-02-28T16:31:14.763 INFO:tasks.workunit.client.1.smithi066.stdout:0/332: symlink d2/d9/l6e 0 2024-02-28T16:31:14.765 INFO:tasks.workunit.client.0.smithi001.stdout:0/620: write d18/f6d [7274282,15464] 0 2024-02-28T16:31:14.766 INFO:tasks.workunit.client.1.smithi066.stdout:4/327: link d1/d18/f37 d1/d23/d25/f6b 0 2024-02-28T16:31:14.769 INFO:tasks.workunit.client.0.smithi001.stdout:9/738: rename d5/d27/d42/l66 to d5/d27/db1/d65/lf5 0 2024-02-28T16:31:14.770 INFO:tasks.workunit.client.0.smithi001.stdout:2/634: creat d0/d2/d3/d36/fd4 x:0 0 0 2024-02-28T16:31:14.771 INFO:tasks.workunit.client.0.smithi001.stdout:5/613: getdents dc/d12/d41 0 2024-02-28T16:31:14.771 INFO:tasks.workunit.client.1.smithi066.stdout:2/245: read d0/d12/d1a/d39/f42 [819300,55320] 0 2024-02-28T16:31:14.771 INFO:tasks.workunit.client.1.smithi066.stdout:4/328: dread - d1/d23/d25/f6b zero size 2024-02-28T16:31:14.772 INFO:tasks.workunit.client.0.smithi001.stdout:0/621: write d18/d31/fad [809485,107471] 0 2024-02-28T16:31:14.776 INFO:tasks.workunit.client.1.smithi066.stdout:2/246: symlink d0/d12/d2e/d38/l52 0 2024-02-28T16:31:14.781 INFO:tasks.workunit.client.1.smithi066.stdout:0/333: read d2/d9/d10/d23/f2d [55538,43995] 0 2024-02-28T16:31:14.782 INFO:tasks.workunit.client.0.smithi001.stdout:3/663: mkdir d4/d18/ddb 0 2024-02-28T16:31:14.783 INFO:tasks.workunit.client.0.smithi001.stdout:1/776: rmdir d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6 39 2024-02-28T16:31:14.785 INFO:tasks.workunit.client.1.smithi066.stdout:0/334: write d2/d9/f4f [561307,40443] 0 2024-02-28T16:31:14.789 INFO:tasks.workunit.client.0.smithi001.stdout:4/688: dwrite d3/d1c/d58/f76 [4194304,4194304] 0 2024-02-28T16:31:14.790 INFO:tasks.workunit.client.1.smithi066.stdout:1/261: dwrite d2/d27/f3c [0,4194304] 0 2024-02-28T16:31:14.792 INFO:tasks.workunit.client.1.smithi066.stdout:0/335: creat d2/d9/f6f x:0 0 0 2024-02-28T16:31:14.792 INFO:tasks.workunit.client.1.smithi066.stdout:8/198: truncate d0/d6/f7 2947039 0 2024-02-28T16:31:14.793 INFO:tasks.workunit.client.0.smithi001.stdout:7/632: rename d3/c5e to d3/cc8 0 2024-02-28T16:31:14.793 INFO:tasks.workunit.client.1.smithi066.stdout:1/262: mknod d2/d27/d2b/d43/c50 0 2024-02-28T16:31:14.794 INFO:tasks.workunit.client.1.smithi066.stdout:8/199: stat d0/db/f27 0 2024-02-28T16:31:14.797 INFO:tasks.workunit.client.1.smithi066.stdout:0/336: write d2/d9/d10/d50/d52/f55 [4923277,88725] 0 2024-02-28T16:31:14.802 INFO:tasks.workunit.client.1.smithi066.stdout:8/200: mkdir d0/d8/d3a 0 2024-02-28T16:31:14.803 INFO:tasks.workunit.client.0.smithi001.stdout:2/635: dwrite d0/f63 [0,4194304] 0 2024-02-28T16:31:14.805 INFO:tasks.workunit.client.0.smithi001.stdout:9/739: unlink d5/fe 0 2024-02-28T16:31:14.809 INFO:tasks.workunit.client.0.smithi001.stdout:5/614: dwrite dc/d12/d56/d96/db9/dc2/dca/d91/fc7 [0,4194304] 0 2024-02-28T16:31:14.809 INFO:tasks.workunit.client.0.smithi001.stdout:3/664: dwrite d4/d6/f1c [0,4194304] 0 2024-02-28T16:31:14.813 INFO:tasks.workunit.client.1.smithi066.stdout:8/201: symlink d0/d6/dc/d16/d1c/l3b 0 2024-02-28T16:31:14.820 INFO:tasks.workunit.client.1.smithi066.stdout:9/192: dwrite da/fb [4194304,4194304] 0 2024-02-28T16:31:14.823 INFO:tasks.workunit.client.1.smithi066.stdout:9/193: stat da/f30 0 2024-02-28T16:31:14.832 INFO:tasks.workunit.client.1.smithi066.stdout:9/194: truncate da/d21/f32 253299 0 2024-02-28T16:31:14.834 INFO:tasks.workunit.client.1.smithi066.stdout:3/228: dwrite db/d2a/f37 [0,4194304] 0 2024-02-28T16:31:14.834 INFO:tasks.workunit.client.1.smithi066.stdout:1/263: dwrite d2/f1d [0,4194304] 0 2024-02-28T16:31:14.838 INFO:tasks.workunit.client.1.smithi066.stdout:4/329: rmdir d1/d23/d25 39 2024-02-28T16:31:14.838 INFO:tasks.workunit.client.1.smithi066.stdout:3/229: readlink db/dc/d11/l16 0 2024-02-28T16:31:14.838 INFO:tasks.workunit.client.1.smithi066.stdout:4/330: write d1/f4 [3232002,40153] 0 2024-02-28T16:31:14.839 INFO:tasks.workunit.client.0.smithi001.stdout:2/636: mkdir d0/d2/d3/de/d61/dd5 0 2024-02-28T16:31:14.839 INFO:tasks.workunit.client.0.smithi001.stdout:2/637: readlink d0/l60 0 2024-02-28T16:31:14.841 INFO:tasks.workunit.client.0.smithi001.stdout:3/665: dread d4/d6/f87 [0,4194304] 0 2024-02-28T16:31:14.843 INFO:tasks.workunit.client.1.smithi066.stdout:3/230: dread - db/d13/d1a/d33/f39 zero size 2024-02-28T16:31:14.848 INFO:tasks.workunit.client.0.smithi001.stdout:9/740: truncate d5/d27/f2b 8273824 0 2024-02-28T16:31:14.858 INFO:tasks.workunit.client.1.smithi066.stdout:1/264: rename d2/d27/d4c to d2/d27/d2b/d37/d51 0 2024-02-28T16:31:14.858 INFO:tasks.workunit.client.1.smithi066.stdout:4/331: write d1/d18/d41/f47 [1988057,33402] 0 2024-02-28T16:31:14.860 INFO:tasks.workunit.client.0.smithi001.stdout:6/613: creat d9/d29/d6e/d4e/fd0 x:0 0 0 2024-02-28T16:31:14.861 INFO:tasks.workunit.client.1.smithi066.stdout:5/205: dwrite d3/d4/dc/f1d [0,4194304] 0 2024-02-28T16:31:14.867 INFO:tasks.workunit.client.1.smithi066.stdout:3/231: write db/d13/d14/d2d/f3e [1310068,111169] 0 2024-02-28T16:31:14.873 INFO:tasks.workunit.client.0.smithi001.stdout:5/615: symlink dc/d1a/d7d/d87/db2/lcc 0 2024-02-28T16:31:14.874 INFO:tasks.workunit.client.1.smithi066.stdout:5/206: write d3/d4/dc/d2f/f32 [1867414,45310] 0 2024-02-28T16:31:14.875 INFO:tasks.workunit.client.0.smithi001.stdout:5/616: chown dc/d1a/d21/l8b 11805 1 2024-02-28T16:31:14.876 INFO:tasks.workunit.client.1.smithi066.stdout:1/265: unlink d2/d18/d19/c1a 0 2024-02-28T16:31:14.877 INFO:tasks.workunit.client.0.smithi001.stdout:6/614: unlink d9/d29/d6e/d53/fb4 0 2024-02-28T16:31:14.877 INFO:tasks.workunit.client.0.smithi001.stdout:5/617: chown dc/d1a/d7d/d87/f8f 59158636 1 2024-02-28T16:31:14.880 INFO:tasks.workunit.client.1.smithi066.stdout:8/202: dwrite d0/db/d15/d32/f39 [4194304,4194304] 0 2024-02-28T16:31:14.882 INFO:tasks.workunit.client.0.smithi001.stdout:5/618: write dc/d12/d56/d96/db9/dc2/dca/d3c/f4f [1699536,109627] 0 2024-02-28T16:31:14.883 INFO:tasks.workunit.client.1.smithi066.stdout:4/332: rename d1/l15 to d1/d18/d4e/l6c 0 2024-02-28T16:31:14.883 INFO:tasks.workunit.client.0.smithi001.stdout:0/622: truncate d18/d22/d2f/fa4 3269460 0 2024-02-28T16:31:14.884 INFO:tasks.workunit.client.0.smithi001.stdout:4/689: creat d3/d1c/d58/fdd x:0 0 0 2024-02-28T16:31:14.884 INFO:tasks.workunit.client.1.smithi066.stdout:2/247: dwrite d0/ff [0,4194304] 0 2024-02-28T16:31:14.888 INFO:tasks.workunit.client.0.smithi001.stdout:9/741: creat d5/d27/d42/def/ff6 x:0 0 0 2024-02-28T16:31:14.888 INFO:tasks.workunit.client.1.smithi066.stdout:9/195: truncate da/fc 1643704 0 2024-02-28T16:31:14.889 INFO:tasks.workunit.client.0.smithi001.stdout:4/690: write d3/d2d/d4c/d5e/db0/fc6 [511792,97594] 0 2024-02-28T16:31:14.889 INFO:tasks.workunit.client.0.smithi001.stdout:7/633: rmdir d3/d17/d52/d92 39 2024-02-28T16:31:14.890 INFO:tasks.workunit.client.1.smithi066.stdout:1/266: chown d2/c20 788 1 2024-02-28T16:31:14.890 INFO:tasks.workunit.client.0.smithi001.stdout:9/742: chown d5/d27/db1/d65/lbf 0 1 2024-02-28T16:31:14.891 INFO:tasks.workunit.client.1.smithi066.stdout:6/176: sync 2024-02-28T16:31:14.892 INFO:tasks.workunit.client.0.smithi001.stdout:1/777: write d0/f21 [269052,35247] 0 2024-02-28T16:31:14.892 INFO:tasks.workunit.client.0.smithi001.stdout:2/638: dread d0/d2/d3/de/d61/d12/d6a/f81 [0,4194304] 0 2024-02-28T16:31:14.893 INFO:tasks.workunit.client.0.smithi001.stdout:8/663: sync 2024-02-28T16:31:14.897 INFO:tasks.workunit.client.1.smithi066.stdout:9/196: mkdir da/df/d18/d28/d42 0 2024-02-28T16:31:14.898 INFO:tasks.workunit.client.1.smithi066.stdout:4/333: unlink d1/d18/c4b 0 2024-02-28T16:31:14.899 INFO:tasks.workunit.client.0.smithi001.stdout:0/623: mknod d18/d31/d4f/ccb 0 2024-02-28T16:31:14.899 INFO:tasks.workunit.client.0.smithi001.stdout:3/666: getdents d4/d6/d7f 0 2024-02-28T16:31:14.901 INFO:tasks.workunit.client.0.smithi001.stdout:4/691: mkdir d3/d5/d7/d1f/d2e/dde 0 2024-02-28T16:31:14.902 INFO:tasks.workunit.client.1.smithi066.stdout:1/267: dread d2/d18/d19/f24 [0,4194304] 0 2024-02-28T16:31:14.903 INFO:tasks.workunit.client.1.smithi066.stdout:8/203: dread d0/f13 [0,4194304] 0 2024-02-28T16:31:14.903 INFO:tasks.workunit.client.0.smithi001.stdout:5/619: chown dc/d1a/d7d/d87/laf 521350 1 2024-02-28T16:31:14.904 INFO:tasks.workunit.client.0.smithi001.stdout:7/634: mkdir d3/d17/d3d/d59/d96/dc9 0 2024-02-28T16:31:14.904 INFO:tasks.workunit.client.1.smithi066.stdout:5/207: link d3/d4/dc/dd/d2e/d18/f2b d3/d4/dc/d42/f49 0 2024-02-28T16:31:14.906 INFO:tasks.workunit.client.1.smithi066.stdout:9/197: mknod da/d21/c43 0 2024-02-28T16:31:14.907 INFO:tasks.workunit.client.0.smithi001.stdout:6/615: dwrite d9/fab [0,4194304] 0 2024-02-28T16:31:14.910 INFO:tasks.workunit.client.1.smithi066.stdout:2/248: dwrite d0/ff [0,4194304] 0 2024-02-28T16:31:14.910 INFO:tasks.workunit.client.1.smithi066.stdout:3/232: link db/d13/d1a/l1c db/d13/d1a/l50 0 2024-02-28T16:31:14.910 INFO:tasks.workunit.client.1.smithi066.stdout:2/249: stat d0/d12/d1a/d23/d29/d21 0 2024-02-28T16:31:14.910 INFO:tasks.workunit.client.0.smithi001.stdout:1/778: mkdir d0/d9/d110 0 2024-02-28T16:31:14.911 INFO:tasks.workunit.client.0.smithi001.stdout:4/692: write d3/d1c/d58/d6c/fd2 [398250,52477] 0 2024-02-28T16:31:14.912 INFO:tasks.workunit.client.0.smithi001.stdout:8/664: creat d4/d7/d22/d27/d58/d94/db4/dd5/fd7 x:0 0 0 2024-02-28T16:31:14.913 INFO:tasks.workunit.client.1.smithi066.stdout:3/233: write db/d13/d14/d24/f34 [4295032,87358] 0 2024-02-28T16:31:14.913 INFO:tasks.workunit.client.0.smithi001.stdout:5/620: chown dc/df/f3a 3 1 2024-02-28T16:31:14.914 INFO:tasks.workunit.client.0.smithi001.stdout:0/624: mkdir d18/d22/d2f/d1c/d90/dcc 0 2024-02-28T16:31:14.914 INFO:tasks.workunit.client.1.smithi066.stdout:6/177: mkdir d2/da/d38 0 2024-02-28T16:31:14.914 INFO:tasks.workunit.client.1.smithi066.stdout:4/334: symlink d1/d18/d41/l6d 0 2024-02-28T16:31:14.917 INFO:tasks.workunit.client.0.smithi001.stdout:3/667: link d4/d6/f40 d4/d97/fdc 0 2024-02-28T16:31:14.926 INFO:tasks.workunit.client.0.smithi001.stdout:0/625: write d18/d31/f43 [4553501,47384] 0 2024-02-28T16:31:14.928 INFO:tasks.workunit.client.0.smithi001.stdout:7/635: truncate d3/d17/d3d/d42/d8e/f9b 946904 0 2024-02-28T16:31:14.929 INFO:tasks.workunit.client.1.smithi066.stdout:5/208: rename d3/d4/dc/dd/f3e to d3/d4/dc/dd/d2e/d18/f4a 0 2024-02-28T16:31:14.930 INFO:tasks.workunit.client.0.smithi001.stdout:2/639: dwrite d0/d2/d3/de/d61/d12/d55/db2/fc7 [0,4194304] 0 2024-02-28T16:31:14.931 INFO:tasks.workunit.client.0.smithi001.stdout:5/621: creat dc/d1a/d7d/fcd x:0 0 0 2024-02-28T16:31:14.932 INFO:tasks.workunit.client.1.smithi066.stdout:2/250: dwrite d0/d12/d4a/f4f [0,4194304] 0 2024-02-28T16:31:14.932 INFO:tasks.workunit.client.1.smithi066.stdout:1/268: mknod d2/d18/d19/c52 0 2024-02-28T16:31:14.935 INFO:tasks.workunit.client.0.smithi001.stdout:8/665: write d4/d7/d22/d27/f4a [4108127,125294] 0 2024-02-28T16:31:14.935 INFO:tasks.workunit.client.0.smithi001.stdout:8/666: write d4/d47/d8b/fba [681689,111349] 0 2024-02-28T16:31:14.937 INFO:tasks.workunit.client.0.smithi001.stdout:6/616: creat d9/d29/d6e/d4e/fd1 x:0 0 0 2024-02-28T16:31:14.937 INFO:tasks.workunit.client.0.smithi001.stdout:4/693: creat d3/d1c/d58/fdf x:0 0 0 2024-02-28T16:31:14.939 INFO:tasks.workunit.client.0.smithi001.stdout:6/617: readlink d9/db/d5f/d8b/l74 0 2024-02-28T16:31:14.941 INFO:tasks.workunit.client.1.smithi066.stdout:8/204: symlink d0/db/d12/d1d/d25/l3c 0 2024-02-28T16:31:14.941 INFO:tasks.workunit.client.0.smithi001.stdout:0/626: creat d18/d31/d4f/fcd x:0 0 0 2024-02-28T16:31:14.947 INFO:tasks.workunit.client.0.smithi001.stdout:3/668: rename d4/d6/d7/d78/d81/d8a to d4/d1e/ddd 0 2024-02-28T16:31:14.949 INFO:tasks.workunit.client.0.smithi001.stdout:3/669: truncate d4/d18/f49 1091916 0 2024-02-28T16:31:14.952 INFO:tasks.workunit.client.0.smithi001.stdout:7/636: mkdir d3/d17/d52/d55/dca 0 2024-02-28T16:31:14.954 INFO:tasks.workunit.client.0.smithi001.stdout:7/637: chown d3/d17/d3d/fa7 237514939 1 2024-02-28T16:31:14.957 INFO:tasks.workunit.client.0.smithi001.stdout:5/622: symlink dc/d1a/d7d/d87/db2/lce 0 2024-02-28T16:31:14.957 INFO:tasks.workunit.client.0.smithi001.stdout:7/638: read d3/d17/f2a [463719,120361] 0 2024-02-28T16:31:14.960 INFO:tasks.workunit.client.1.smithi066.stdout:8/205: dwrite d0/db/d12/d1d/f29 [0,4194304] 0 2024-02-28T16:31:14.961 INFO:tasks.workunit.client.0.smithi001.stdout:9/743: truncate d5/d27/db1/fa3 2152381 0 2024-02-28T16:31:14.962 INFO:tasks.workunit.client.1.smithi066.stdout:2/251: rename d0/ld to d0/d12/d3e/l53 0 2024-02-28T16:31:14.963 INFO:tasks.workunit.client.1.smithi066.stdout:4/335: symlink d1/d23/l6e 0 2024-02-28T16:31:14.965 INFO:tasks.workunit.client.1.smithi066.stdout:3/234: getdents db/d2a/d44/d48 0 2024-02-28T16:31:14.965 INFO:tasks.workunit.client.1.smithi066.stdout:6/178: link d2/f29 d2/da/d38/f39 0 2024-02-28T16:31:14.966 INFO:tasks.workunit.client.0.smithi001.stdout:2/640: write d0/d2/d3/de/d8a/fc6 [901066,6069] 0 2024-02-28T16:31:14.967 INFO:tasks.workunit.client.0.smithi001.stdout:6/618: symlink d9/db/d5f/d89/ld2 0 2024-02-28T16:31:14.968 INFO:tasks.workunit.client.1.smithi066.stdout:6/179: fsync d2/da/f1c 0 2024-02-28T16:31:14.970 INFO:tasks.workunit.client.1.smithi066.stdout:6/180: dread - d2/da/f1e zero size 2024-02-28T16:31:14.971 INFO:tasks.workunit.client.0.smithi001.stdout:4/694: dread d3/d5/fab [0,4194304] 0 2024-02-28T16:31:14.974 INFO:tasks.workunit.client.1.smithi066.stdout:5/209: dread - d3/d1b/d39/f44 zero size 2024-02-28T16:31:14.974 INFO:tasks.workunit.client.0.smithi001.stdout:3/670: read d4/d6/d7/d78/d81/da1/fb9 [3394857,6458] 0 2024-02-28T16:31:14.975 INFO:tasks.workunit.client.0.smithi001.stdout:3/671: stat d4/d6/f56 0 2024-02-28T16:31:14.978 INFO:tasks.workunit.client.0.smithi001.stdout:1/779: write d0/d9/d1a/d2b/d26/d27/d30/f76 [104920,98306] 0 2024-02-28T16:31:14.978 INFO:tasks.workunit.client.0.smithi001.stdout:3/672: write d4/f83 [156768,3602] 0 2024-02-28T16:31:14.979 INFO:tasks.workunit.client.0.smithi001.stdout:9/744: symlink d5/d27/d34/d50/lf7 0 2024-02-28T16:31:14.980 INFO:tasks.workunit.client.0.smithi001.stdout:3/673: chown d4/d6/d7/d23/lc4 59391850 1 2024-02-28T16:31:14.980 INFO:tasks.workunit.client.1.smithi066.stdout:8/206: mkdir d0/d6/dc/d16/d1c/d3d 0 2024-02-28T16:31:14.982 INFO:tasks.workunit.client.0.smithi001.stdout:3/674: chown d4/d18/f69 184526856 1 2024-02-28T16:31:14.983 INFO:tasks.workunit.client.1.smithi066.stdout:6/181: mknod d2/d26/c3a 0 2024-02-28T16:31:14.983 INFO:tasks.workunit.client.0.smithi001.stdout:2/641: symlink d0/d2/d3/de/d61/d12/d6a/ld6 0 2024-02-28T16:31:14.987 INFO:tasks.workunit.client.0.smithi001.stdout:6/619: rename d9/d29/d6a/f73 to d9/db/d19/fd3 0 2024-02-28T16:31:14.988 INFO:tasks.workunit.client.1.smithi066.stdout:2/252: unlink d0/d12/f36 0 2024-02-28T16:31:14.991 INFO:tasks.workunit.client.0.smithi001.stdout:7/639: creat d3/d1c/fcb x:0 0 0 2024-02-28T16:31:14.992 INFO:tasks.workunit.client.1.smithi066.stdout:4/336: creat d1/d23/d25/f6f x:0 0 0 2024-02-28T16:31:14.992 INFO:tasks.workunit.client.1.smithi066.stdout:3/235: dwrite db/d2a/d44/f4f [0,4194304] 0 2024-02-28T16:31:14.993 INFO:tasks.workunit.client.0.smithi001.stdout:8/667: creat d4/d7/d22/d27/d4b/fd8 x:0 0 0 2024-02-28T16:31:14.994 INFO:tasks.workunit.client.1.smithi066.stdout:2/253: write d0/f30 [871866,30181] 0 2024-02-28T16:31:14.995 INFO:tasks.workunit.client.1.smithi066.stdout:5/210: dread d3/d4/dc/d2f/f33 [0,4194304] 0 2024-02-28T16:31:14.995 INFO:tasks.workunit.client.0.smithi001.stdout:9/745: chown d5/d4d/lf3 1 1 2024-02-28T16:31:14.999 INFO:tasks.workunit.client.1.smithi066.stdout:8/207: link d0/d6/dc/d16/l30 d0/db/d12/d38/l3e 0 2024-02-28T16:31:14.999 INFO:tasks.workunit.client.0.smithi001.stdout:9/746: write d5/d27/fb0 [289991,91903] 0 2024-02-28T16:31:15.000 INFO:tasks.workunit.client.0.smithi001.stdout:9/747: fsync d5/f12 0 2024-02-28T16:31:15.001 INFO:tasks.workunit.client.1.smithi066.stdout:3/236: mknod db/dc/d35/d3d/c51 0 2024-02-28T16:31:15.001 INFO:tasks.workunit.client.1.smithi066.stdout:1/269: dwrite d2/d27/d2b/f3d [0,4194304] 0 2024-02-28T16:31:15.004 INFO:tasks.workunit.client.0.smithi001.stdout:5/623: link dc/d1a/d21/l4e dc/df/lcf 0 2024-02-28T16:31:15.004 INFO:tasks.workunit.client.0.smithi001.stdout:0/627: dwrite d18/d31/d4f/f60 [0,4194304] 0 2024-02-28T16:31:15.005 INFO:tasks.workunit.client.1.smithi066.stdout:2/254: chown d0/d12/d1a/d23/d29/d21 6562 1 2024-02-28T16:31:15.005 INFO:tasks.workunit.client.0.smithi001.stdout:7/640: truncate d3/d17/d61/f71 4806871 0 2024-02-28T16:31:15.008 INFO:tasks.workunit.client.0.smithi001.stdout:2/642: link d0/d2/d3/de/d22/f38 d0/d2/d3/de/d61/d12/d6a/d84/fd7 0 2024-02-28T16:31:15.011 INFO:tasks.workunit.client.1.smithi066.stdout:4/337: truncate d1/d18/f4a 2147700 0 2024-02-28T16:31:15.011 INFO:tasks.workunit.client.1.smithi066.stdout:5/211: fsync d3/d4/dc/dd/d2e/f38 0 2024-02-28T16:31:15.012 INFO:tasks.workunit.client.1.smithi066.stdout:7/150: sync 2024-02-28T16:31:15.013 INFO:tasks.workunit.client.1.smithi066.stdout:2/255: stat d0/d12/d1a/d39/d40 0 2024-02-28T16:31:15.017 INFO:tasks.workunit.client.1.smithi066.stdout:5/212: fsync d3/d4/dc/f1e 0 2024-02-28T16:31:15.018 INFO:tasks.workunit.client.0.smithi001.stdout:8/668: symlink d4/d7/d22/d27/d4b/ld9 0 2024-02-28T16:31:15.021 INFO:tasks.workunit.client.1.smithi066.stdout:6/182: dwrite d2/da/f21 [0,4194304] 0 2024-02-28T16:31:15.021 INFO:tasks.workunit.client.0.smithi001.stdout:8/669: write d4/d7/d22/d27/d58/d94/db4/fc0 [523297,76105] 0 2024-02-28T16:31:15.023 INFO:tasks.workunit.client.0.smithi001.stdout:1/780: creat d0/d9/d1a/d2b/d26/d27/d30/f111 x:0 0 0 2024-02-28T16:31:15.024 INFO:tasks.workunit.client.1.smithi066.stdout:6/183: write d2/da/f21 [3738186,71767] 0 2024-02-28T16:31:15.025 INFO:tasks.workunit.client.1.smithi066.stdout:8/208: link d0/d6/dc/d16/d1c/l3b d0/db/d15/l3f 0 2024-02-28T16:31:15.027 INFO:tasks.workunit.client.0.smithi001.stdout:5/624: chown dc/d1a/c31 5594173 1 2024-02-28T16:31:15.029 INFO:tasks.workunit.client.0.smithi001.stdout:1/781: truncate d0/d9/d56/d2f/f107 845097 0 2024-02-28T16:31:15.030 INFO:tasks.workunit.client.0.smithi001.stdout:6/620: rmdir d9/d29/d6e/d9a/da9 0 2024-02-28T16:31:15.031 INFO:tasks.workunit.client.0.smithi001.stdout:6/621: readlink d9/lba 0 2024-02-28T16:31:15.032 INFO:tasks.workunit.client.1.smithi066.stdout:1/270: link d2/d27/f3c d2/d27/d2b/d43/f53 0 2024-02-28T16:31:15.032 INFO:tasks.workunit.client.1.smithi066.stdout:4/338: mkdir d1/d23/d70 0 2024-02-28T16:31:15.033 INFO:tasks.workunit.client.0.smithi001.stdout:4/695: getdents d3/d2d/d4c/d69 0 2024-02-28T16:31:15.034 INFO:tasks.workunit.client.1.smithi066.stdout:7/151: rename dc/de/f26 to dc/d29/f2e 0 2024-02-28T16:31:15.034 INFO:tasks.workunit.client.0.smithi001.stdout:1/782: chown d0/d9/d1a/d2b/d26/d27/d30/d52/d84 89601811 1 2024-02-28T16:31:15.034 INFO:tasks.workunit.client.0.smithi001.stdout:3/675: write d4/d6/d7/f2f [1003646,33318] 0 2024-02-28T16:31:15.036 INFO:tasks.workunit.client.1.smithi066.stdout:1/271: stat d2/d27/d2b/d34/f44 0 2024-02-28T16:31:15.036 INFO:tasks.workunit.client.1.smithi066.stdout:8/209: rmdir d0/d6 39 2024-02-28T16:31:15.037 INFO:tasks.workunit.client.0.smithi001.stdout:4/696: read - d3/fd6 zero size 2024-02-28T16:31:15.037 INFO:tasks.workunit.client.0.smithi001.stdout:9/748: symlink d5/d27/db1/da6/db2/de4/lf8 0 2024-02-28T16:31:15.038 INFO:tasks.workunit.client.1.smithi066.stdout:4/339: creat d1/d18/d4e/f71 x:0 0 0 2024-02-28T16:31:15.038 INFO:tasks.workunit.client.1.smithi066.stdout:6/184: write d2/da/d19/f20 [998849,79666] 0 2024-02-28T16:31:15.039 INFO:tasks.workunit.client.0.smithi001.stdout:8/670: creat d4/d7/d22/d27/d58/d94/db4/dd5/fda x:0 0 0 2024-02-28T16:31:15.041 INFO:tasks.workunit.client.0.smithi001.stdout:9/749: read d5/f6 [538335,35950] 0 2024-02-28T16:31:15.041 INFO:tasks.workunit.client.0.smithi001.stdout:6/622: stat d9/de/c6f 0 2024-02-28T16:31:15.042 INFO:tasks.workunit.client.1.smithi066.stdout:5/213: dread d3/d4/dc/d2f/f33 [0,4194304] 0 2024-02-28T16:31:15.044 INFO:tasks.workunit.client.1.smithi066.stdout:4/340: creat d1/d23/d25/d3e/d40/f72 x:0 0 0 2024-02-28T16:31:15.049 INFO:tasks.workunit.client.1.smithi066.stdout:2/256: dread d0/d12/d1a/f2b [0,4194304] 0 2024-02-28T16:31:15.050 INFO:tasks.workunit.client.1.smithi066.stdout:5/214: write d3/d4/dc/dd/d2e/f38 [724870,108760] 0 2024-02-28T16:31:15.050 INFO:tasks.workunit.client.1.smithi066.stdout:3/237: truncate db/d13/d14/d24/d2f/f4a 2050593 0 2024-02-28T16:31:15.053 INFO:tasks.workunit.client.0.smithi001.stdout:0/628: link d18/d22/d76/caf d18/d31/cce 0 2024-02-28T16:31:15.054 INFO:tasks.workunit.client.1.smithi066.stdout:2/257: read - d0/f44 zero size 2024-02-28T16:31:15.058 INFO:tasks.workunit.client.1.smithi066.stdout:5/215: write d3/d4/dc/dd/d2e/f40 [620716,87111] 0 2024-02-28T16:31:15.058 INFO:tasks.workunit.client.1.smithi066.stdout:4/341: rename d1/cc to d1/d23/d25/d43/c73 0 2024-02-28T16:31:15.058 INFO:tasks.workunit.client.0.smithi001.stdout:6/623: write d9/d29/d6e/d4e/d96/fc0 [1543462,122951] 0 2024-02-28T16:31:15.059 INFO:tasks.workunit.client.0.smithi001.stdout:7/641: truncate d3/d17/d61/f71 1715651 0 2024-02-28T16:31:15.059 INFO:tasks.workunit.client.0.smithi001.stdout:9/750: mkdir d5/d27/db1/d5e/df9 0 2024-02-28T16:31:15.060 INFO:tasks.workunit.client.1.smithi066.stdout:3/238: mkdir db/d2a/d44/d48/d52 0 2024-02-28T16:31:15.060 INFO:tasks.workunit.client.0.smithi001.stdout:5/625: getdents dc/d12/d56/d96/db9/dc2/dca/d91 0 2024-02-28T16:31:15.066 INFO:tasks.workunit.client.1.smithi066.stdout:2/258: rename d0/d12/d4a/f4f to d0/d12/d1a/f54 0 2024-02-28T16:31:15.066 INFO:tasks.workunit.client.1.smithi066.stdout:4/342: readlink d1/d23/d25/l56 0 2024-02-28T16:31:15.066 INFO:tasks.workunit.client.0.smithi001.stdout:9/751: chown d5/d45/d6f/c8f 524096 1 2024-02-28T16:31:15.069 INFO:tasks.workunit.client.0.smithi001.stdout:7/642: creat d3/d17/d3d/fcc x:0 0 0 2024-02-28T16:31:15.069 INFO:tasks.workunit.client.1.smithi066.stdout:6/185: dwrite d2/d2a/f2f [0,4194304] 0 2024-02-28T16:31:15.070 INFO:tasks.workunit.client.1.smithi066.stdout:2/259: write d0/f19 [400083,8937] 0 2024-02-28T16:31:15.070 INFO:tasks.workunit.client.0.smithi001.stdout:0/629: truncate d18/d22/d3d/f52 374627 0 2024-02-28T16:31:15.071 INFO:tasks.workunit.client.1.smithi066.stdout:5/216: fsync d3/d4/dc/dd/d2e/d18/f1a 0 2024-02-28T16:31:15.071 INFO:tasks.workunit.client.0.smithi001.stdout:3/676: dread d4/d18/f91 [0,4194304] 0 2024-02-28T16:31:15.072 INFO:tasks.workunit.client.1.smithi066.stdout:7/152: dread - dc/de/d17/d1e/f24 zero size 2024-02-28T16:31:15.072 INFO:tasks.workunit.client.0.smithi001.stdout:9/752: fsync d5/fd 0 2024-02-28T16:31:15.073 INFO:tasks.workunit.client.1.smithi066.stdout:1/272: truncate d2/fc 1681895 0 2024-02-28T16:31:15.073 INFO:tasks.workunit.client.0.smithi001.stdout:2/643: dread d0/f32 [0,4194304] 0 2024-02-28T16:31:15.073 INFO:tasks.workunit.client.1.smithi066.stdout:7/153: chown dc/de/d17 15 1 2024-02-28T16:31:15.075 INFO:tasks.workunit.client.0.smithi001.stdout:3/677: write d4/d6/fc [2933428,7649] 0 2024-02-28T16:31:15.075 INFO:tasks.workunit.client.0.smithi001.stdout:5/626: fsync dc/d12/d56/d96/db9/dc2/dca/f42 0 2024-02-28T16:31:15.078 INFO:tasks.workunit.client.1.smithi066.stdout:6/186: truncate d2/d26/f27 523979 0 2024-02-28T16:31:15.079 INFO:tasks.workunit.client.0.smithi001.stdout:6/624: mkdir d9/db/d5f/dd4 0 2024-02-28T16:31:15.079 INFO:tasks.workunit.client.1.smithi066.stdout:1/273: rmdir d2/d27 39 2024-02-28T16:31:15.082 INFO:tasks.workunit.client.1.smithi066.stdout:1/274: stat d2/d18/d19 0 2024-02-28T16:31:15.084 INFO:tasks.workunit.client.1.smithi066.stdout:7/154: fsync f4 0 2024-02-28T16:31:15.085 INFO:tasks.workunit.client.0.smithi001.stdout:3/678: read - d4/d1e/ddd/fc7 zero size 2024-02-28T16:31:15.086 INFO:tasks.workunit.client.1.smithi066.stdout:6/187: rmdir d2/da 39 2024-02-28T16:31:15.086 INFO:tasks.workunit.client.1.smithi066.stdout:2/260: creat d0/d12/d1a/d39/d40/f55 x:0 0 0 2024-02-28T16:31:15.087 INFO:tasks.workunit.client.0.smithi001.stdout:6/625: write d9/da6/f30 [3659488,5008] 0 2024-02-28T16:31:15.090 INFO:tasks.workunit.client.1.smithi066.stdout:5/217: creat d3/d4/dc/dd/d2e/d18/f4b x:0 0 0 2024-02-28T16:31:15.091 INFO:tasks.workunit.client.0.smithi001.stdout:5/627: chown dc/d1a/d7d/f88 416 1 2024-02-28T16:31:15.093 INFO:tasks.workunit.client.0.smithi001.stdout:1/783: truncate d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f100 508669 0 2024-02-28T16:31:15.093 INFO:tasks.workunit.client.0.smithi001.stdout:4/697: dwrite d3/d1c/d58/d6c/d79/fb5 [0,4194304] 0 2024-02-28T16:31:15.093 INFO:tasks.workunit.client.0.smithi001.stdout:6/626: write d9/d29/d6e/d9a/fad [970890,22294] 0 2024-02-28T16:31:15.097 INFO:tasks.workunit.client.0.smithi001.stdout:5/628: chown dc/d12/d41/d54/la3 36 1 2024-02-28T16:31:15.100 INFO:tasks.workunit.client.0.smithi001.stdout:7/643: write d3/d17/d52/f77 [509697,17537] 0 2024-02-28T16:31:15.100 INFO:tasks.workunit.client.0.smithi001.stdout:1/784: fsync d0/d9/d1a/d2b/d26/f108 0 2024-02-28T16:31:15.101 INFO:tasks.workunit.client.0.smithi001.stdout:6/627: chown d9/d29/d6e/d53/c79 15 1 2024-02-28T16:31:15.103 INFO:tasks.workunit.client.1.smithi066.stdout:4/343: dwrite d1/d23/d25/d43/f5d [0,4194304] 0 2024-02-28T16:31:15.103 INFO:tasks.workunit.client.0.smithi001.stdout:3/679: creat d4/d1e/d7c/d7d/fde x:0 0 0 2024-02-28T16:31:15.104 INFO:tasks.workunit.client.0.smithi001.stdout:4/698: rmdir d3/d5/d7/d1f 39 2024-02-28T16:31:15.106 INFO:tasks.workunit.client.1.smithi066.stdout:3/239: truncate db/d13/d14/d24/f34 505618 0 2024-02-28T16:31:15.107 INFO:tasks.workunit.client.0.smithi001.stdout:7/644: chown d3/d17/d52/d55/l83 88 1 2024-02-28T16:31:15.108 INFO:tasks.workunit.client.0.smithi001.stdout:5/629: mkdir dc/d12/d56/d96/dd0 0 2024-02-28T16:31:15.108 INFO:tasks.workunit.client.1.smithi066.stdout:7/155: dread fa [0,4194304] 0 2024-02-28T16:31:15.109 INFO:tasks.workunit.client.1.smithi066.stdout:3/240: fdatasync db/dc/d35/d3d/f41 0 2024-02-28T16:31:15.110 INFO:tasks.workunit.client.1.smithi066.stdout:2/261: read d0/f19 [386441,31769] 0 2024-02-28T16:31:15.110 INFO:tasks.workunit.client.0.smithi001.stdout:1/785: chown d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/d9c/ce0 6523 1 2024-02-28T16:31:15.112 INFO:tasks.workunit.client.1.smithi066.stdout:3/241: chown db/dc/d11 22379939 1 2024-02-28T16:31:15.114 INFO:tasks.workunit.client.1.smithi066.stdout:6/188: rename d2/da/f36 to d2/da/f3b 0 2024-02-28T16:31:15.116 INFO:tasks.workunit.client.0.smithi001.stdout:6/628: fdatasync d9/da6/fb9 0 2024-02-28T16:31:15.117 INFO:tasks.workunit.client.1.smithi066.stdout:4/344: unlink d1/d18/d41/f58 0 2024-02-28T16:31:15.118 INFO:tasks.workunit.client.1.smithi066.stdout:3/242: readlink db/d2a/l31 0 2024-02-28T16:31:15.118 INFO:tasks.workunit.client.0.smithi001.stdout:9/753: dread d5/d45/d6f/db3/fdd [0,4194304] 0 2024-02-28T16:31:15.118 INFO:tasks.workunit.client.0.smithi001.stdout:2/644: write d0/d2/d3/de/d61/f1f [782750,21860] 0 2024-02-28T16:31:15.122 INFO:tasks.workunit.client.1.smithi066.stdout:6/189: write d2/da/f21 [1720343,24235] 0 2024-02-28T16:31:15.123 INFO:tasks.workunit.client.0.smithi001.stdout:4/699: write d3/fb4 [989276,51588] 0 2024-02-28T16:31:15.125 INFO:tasks.workunit.client.0.smithi001.stdout:4/700: chown d3/d1c/d58/ff 503898629 1 2024-02-28T16:31:15.128 INFO:tasks.workunit.client.1.smithi066.stdout:7/156: symlink dc/de/l2f 0 2024-02-28T16:31:15.129 INFO:tasks.workunit.client.0.smithi001.stdout:3/680: dwrite d4/d1e/d7c/da8/faa [0,4194304] 0 2024-02-28T16:31:15.130 INFO:tasks.workunit.client.1.smithi066.stdout:4/345: rename d1/ca to d1/d18/d41/c74 0 2024-02-28T16:31:15.132 INFO:tasks.workunit.client.1.smithi066.stdout:5/218: dwrite d3/d4/f12 [0,4194304] 0 2024-02-28T16:31:15.132 INFO:tasks.workunit.client.0.smithi001.stdout:6/629: creat d9/db/d19/fd5 x:0 0 0 2024-02-28T16:31:15.135 INFO:tasks.workunit.client.0.smithi001.stdout:1/786: unlink d0/l3d 0 2024-02-28T16:31:15.137 INFO:tasks.workunit.client.1.smithi066.stdout:4/346: truncate d1/f1e 1470112 0 2024-02-28T16:31:15.138 INFO:tasks.workunit.client.0.smithi001.stdout:2/645: unlink d0/d2/d3/de/d61/d12/d6a/d84/fbc 0 2024-02-28T16:31:15.139 INFO:tasks.workunit.client.1.smithi066.stdout:5/219: truncate d3/d4/f12 4203840 0 2024-02-28T16:31:15.139 INFO:tasks.workunit.client.1.smithi066.stdout:6/190: chown d2/da/f22 24752 1 2024-02-28T16:31:15.140 INFO:tasks.workunit.client.0.smithi001.stdout:3/681: mknod d4/d1e/db7/cdf 0 2024-02-28T16:31:15.140 INFO:tasks.workunit.client.1.smithi066.stdout:0/337: sync 2024-02-28T16:31:15.140 INFO:tasks.workunit.client.1.smithi066.stdout:9/198: sync 2024-02-28T16:31:15.141 INFO:tasks.workunit.client.0.smithi001.stdout:9/754: mknod d5/d27/db1/cfa 0 2024-02-28T16:31:15.142 INFO:tasks.workunit.client.0.smithi001.stdout:2/646: fdatasync d0/d2/d3/de/d8a/f51 0 2024-02-28T16:31:15.143 INFO:tasks.workunit.client.0.smithi001.stdout:4/701: mkdir d3/d5/d7/d1f/d20/de0 0 2024-02-28T16:31:15.144 INFO:tasks.workunit.client.0.smithi001.stdout:0/630: dread d18/d22/d2b/d36/d48/f8a [0,4194304] 0 2024-02-28T16:31:15.145 INFO:tasks.workunit.client.0.smithi001.stdout:3/682: chown d4/d1e/l7b 411457 1 2024-02-28T16:31:15.145 INFO:tasks.workunit.client.0.smithi001.stdout:5/630: write dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/f59 [920623,56062] 0 2024-02-28T16:31:15.147 INFO:tasks.workunit.client.1.smithi066.stdout:1/275: dwrite d2/d18/d19/f24 [0,4194304] 0 2024-02-28T16:31:15.147 INFO:tasks.workunit.client.0.smithi001.stdout:9/755: chown d5/d45/c57 1556154 1 2024-02-28T16:31:15.148 INFO:tasks.workunit.client.1.smithi066.stdout:4/347: dread - d1/d23/d25/d43/f5e zero size 2024-02-28T16:31:15.149 INFO:tasks.workunit.client.1.smithi066.stdout:2/262: dwrite d0/d12/d1a/f4c [4194304,4194304] 0 2024-02-28T16:31:15.149 INFO:tasks.workunit.client.0.smithi001.stdout:3/683: stat d4/d6/d7/d78 0 2024-02-28T16:31:15.151 INFO:tasks.workunit.client.1.smithi066.stdout:9/199: fdatasync da/df/d18/f37 0 2024-02-28T16:31:15.153 INFO:tasks.workunit.client.1.smithi066.stdout:5/220: chown d3/d4/dc/ce 24 1 2024-02-28T16:31:15.156 INFO:tasks.workunit.client.0.smithi001.stdout:5/631: rmdir dc/d12/d56/d96/db9/dc2 39 2024-02-28T16:31:15.158 INFO:tasks.workunit.client.0.smithi001.stdout:0/631: read d18/d22/d2f/f4e [1649180,120682] 0 2024-02-28T16:31:15.168 INFO:tasks.workunit.client.0.smithi001.stdout:6/630: dwrite d9/db/d5f/d8b/fc5 [0,4194304] 0 2024-02-28T16:31:15.169 INFO:tasks.workunit.client.1.smithi066.stdout:9/200: read f3 [3074297,10595] 0 2024-02-28T16:31:15.171 INFO:tasks.workunit.client.1.smithi066.stdout:4/348: creat d1/d23/f75 x:0 0 0 2024-02-28T16:31:15.178 INFO:tasks.workunit.client.0.smithi001.stdout:6/631: unlink d9/l20 0 2024-02-28T16:31:15.178 INFO:tasks.workunit.client.0.smithi001.stdout:7/645: dread d3/d17/f19 [0,4194304] 0 2024-02-28T16:31:15.181 INFO:tasks.workunit.client.0.smithi001.stdout:6/632: fsync d9/db/d5f/d8b/f6d 0 2024-02-28T16:31:15.185 INFO:tasks.workunit.client.0.smithi001.stdout:4/702: dwrite d3/d5/d7/d1f/d20/f9a [0,4194304] 0 2024-02-28T16:31:15.186 INFO:tasks.workunit.client.1.smithi066.stdout:8/210: sync 2024-02-28T16:31:15.187 INFO:tasks.workunit.client.1.smithi066.stdout:6/191: dwrite d2/d26/f2d [0,4194304] 0 2024-02-28T16:31:15.190 INFO:tasks.workunit.client.1.smithi066.stdout:6/192: fsync d2/da/f1c 0 2024-02-28T16:31:15.191 INFO:tasks.workunit.client.1.smithi066.stdout:1/276: dread d2/d18/d19/f1c [0,4194304] 0 2024-02-28T16:31:15.195 INFO:tasks.workunit.client.0.smithi001.stdout:0/632: getdents d18/d22/d2f 0 2024-02-28T16:31:15.196 INFO:tasks.workunit.client.1.smithi066.stdout:2/263: dwrite d0/d12/f47 [0,4194304] 0 2024-02-28T16:31:15.196 INFO:tasks.workunit.client.0.smithi001.stdout:7/646: creat d3/d17/d3d/fcd x:0 0 0 2024-02-28T16:31:15.197 INFO:tasks.workunit.client.0.smithi001.stdout:1/787: write d0/d9/d1a/d2b/d26/d27/d30/d88/fcc [889772,80917] 0 2024-02-28T16:31:15.197 INFO:tasks.workunit.client.0.smithi001.stdout:2/647: write d0/d2/d3/de/d22/fb9 [523710,5668] 0 2024-02-28T16:31:15.198 INFO:tasks.workunit.client.1.smithi066.stdout:9/201: creat da/df/d18/d2d/f44 x:0 0 0 2024-02-28T16:31:15.199 INFO:tasks.workunit.client.0.smithi001.stdout:6/633: symlink d9/db/d5f/ld6 0 2024-02-28T16:31:15.199 INFO:tasks.workunit.client.1.smithi066.stdout:4/349: rename d1/f5f to d1/d23/d25/d43/f76 0 2024-02-28T16:31:15.204 INFO:tasks.workunit.client.0.smithi001.stdout:4/703: link d3/d5/d7/c74 d3/d5/d7/d1f/d2e/dbe/d87/ce1 0 2024-02-28T16:31:15.204 INFO:tasks.workunit.client.0.smithi001.stdout:0/633: mkdir d18/d22/d76/dcf 0 2024-02-28T16:31:15.206 INFO:tasks.workunit.client.0.smithi001.stdout:7/647: stat d3/d17/d52/d7f/d99/dc1/d49/d63 0 2024-02-28T16:31:15.210 INFO:tasks.workunit.client.0.smithi001.stdout:0/634: read d18/d22/d2f/d1c/f30 [1430189,55645] 0 2024-02-28T16:31:15.213 INFO:tasks.workunit.client.1.smithi066.stdout:0/338: dread d2/d9/d10/f39 [0,4194304] 0 2024-02-28T16:31:15.215 INFO:tasks.workunit.client.1.smithi066.stdout:6/193: mknod d2/da/c3c 0 2024-02-28T16:31:15.215 INFO:tasks.workunit.client.1.smithi066.stdout:1/277: mkdir d2/d27/d2b/d37/d54 0 2024-02-28T16:31:15.216 INFO:tasks.workunit.client.0.smithi001.stdout:6/634: rename d9/d39/c5e to d9/cd7 0 2024-02-28T16:31:15.216 INFO:tasks.workunit.client.0.smithi001.stdout:3/684: dwrite d4/d6/d7/f43 [0,4194304] 0 2024-02-28T16:31:15.218 INFO:tasks.workunit.client.1.smithi066.stdout:0/339: chown d2/d9/d10/d50/d52 133 1 2024-02-28T16:31:15.223 INFO:tasks.workunit.client.1.smithi066.stdout:5/221: write d3/d4/dc/d42/f49 [1433268,52693] 0 2024-02-28T16:31:15.226 INFO:tasks.workunit.client.1.smithi066.stdout:6/194: mkdir d2/d2a/d3d 0 2024-02-28T16:31:15.226 INFO:tasks.workunit.client.1.smithi066.stdout:1/278: fsync d2/d27/d2b/f2a 0 2024-02-28T16:31:15.227 INFO:tasks.workunit.client.0.smithi001.stdout:9/756: dwrite d5/f1a [0,4194304] 0 2024-02-28T16:31:15.231 INFO:tasks.workunit.client.1.smithi066.stdout:8/211: getdents d0/d6/dc/d16 0 2024-02-28T16:31:15.234 INFO:tasks.workunit.client.1.smithi066.stdout:0/340: creat d2/d9/d10/d50/d51/f70 x:0 0 0 2024-02-28T16:31:15.239 INFO:tasks.workunit.client.1.smithi066.stdout:8/212: mknod d0/db/d12/c40 0 2024-02-28T16:31:15.239 INFO:tasks.workunit.client.0.smithi001.stdout:4/704: truncate d3/d5/d7/d1f/f96 323688 0 2024-02-28T16:31:15.240 INFO:tasks.workunit.client.1.smithi066.stdout:6/195: mkdir d2/da/d3e 0 2024-02-28T16:31:15.241 INFO:tasks.workunit.client.0.smithi001.stdout:3/685: mkdir d4/d1e/d7c/d7d/de0 0 2024-02-28T16:31:15.246 INFO:tasks.workunit.client.0.smithi001.stdout:1/788: dwrite d0/d9/d1a/d2b/d26/d27/d30/d4a/f89 [0,4194304] 0 2024-02-28T16:31:15.246 INFO:tasks.workunit.client.0.smithi001.stdout:7/648: dread d3/d17/d52/fbc [0,4194304] 0 2024-02-28T16:31:15.248 INFO:tasks.workunit.client.0.smithi001.stdout:0/635: creat d18/fd0 x:0 0 0 2024-02-28T16:31:15.251 INFO:tasks.workunit.client.1.smithi066.stdout:4/350: truncate d1/d23/d25/d3e/f57 142847 0 2024-02-28T16:31:15.253 INFO:tasks.workunit.client.1.smithi066.stdout:7/157: dwrite f5 [0,4194304] 0 2024-02-28T16:31:15.255 INFO:tasks.workunit.client.1.smithi066.stdout:3/243: dwrite db/d13/d14/d24/d2f/f4a [0,4194304] 0 2024-02-28T16:31:15.255 INFO:tasks.workunit.client.0.smithi001.stdout:2/648: dwrite d0/d2/d3/d36/fd4 [0,4194304] 0 2024-02-28T16:31:15.256 INFO:tasks.workunit.client.1.smithi066.stdout:9/202: dwrite da/df/f2f [0,4194304] 0 2024-02-28T16:31:15.256 INFO:tasks.workunit.client.1.smithi066.stdout:2/264: dwrite d0/f1f [0,4194304] 0 2024-02-28T16:31:15.257 INFO:tasks.workunit.client.0.smithi001.stdout:5/632: dwrite dc/df/f81 [0,4194304] 0 2024-02-28T16:31:15.257 INFO:tasks.workunit.client.0.smithi001.stdout:4/705: creat d3/d5/fe2 x:0 0 0 2024-02-28T16:31:15.258 INFO:tasks.workunit.client.1.smithi066.stdout:5/222: rename d3/l16 to d3/d4/dc/d42/l4c 0 2024-02-28T16:31:15.258 INFO:tasks.workunit.client.1.smithi066.stdout:6/196: creat d2/d2a/f3f x:0 0 0 2024-02-28T16:31:15.259 INFO:tasks.workunit.client.0.smithi001.stdout:7/649: rmdir d3/d17/d3d/d42 39 2024-02-28T16:31:15.259 INFO:tasks.workunit.client.1.smithi066.stdout:9/203: stat da/f2a 0 2024-02-28T16:31:15.261 INFO:tasks.workunit.client.1.smithi066.stdout:3/244: chown db/d13/d1a/d25/c29 60473849 1 2024-02-28T16:31:15.264 INFO:tasks.workunit.client.0.smithi001.stdout:0/636: chown d18/d22/d57/d7e/f96 3193847 1 2024-02-28T16:31:15.265 INFO:tasks.workunit.client.1.smithi066.stdout:5/223: write d3/d4/dc/dd/d2e/f40 [1050981,118113] 0 2024-02-28T16:31:15.266 INFO:tasks.workunit.client.1.smithi066.stdout:3/245: write db/dc/f2b [3742260,74842] 0 2024-02-28T16:31:15.267 INFO:tasks.workunit.client.0.smithi001.stdout:0/637: stat d18/d31/f7f 0 2024-02-28T16:31:15.267 INFO:tasks.workunit.client.1.smithi066.stdout:5/224: fsync d3/d4/dc/dd/d2e/d18/f4b 0 2024-02-28T16:31:15.270 INFO:tasks.workunit.client.1.smithi066.stdout:8/213: truncate d0/d6/f7 1446295 0 2024-02-28T16:31:15.271 INFO:tasks.workunit.client.0.smithi001.stdout:1/789: mkdir d0/d9/d1a/d2b/d26/d27/d112 0 2024-02-28T16:31:15.272 INFO:tasks.workunit.client.0.smithi001.stdout:2/649: rmdir d0/d2/d3/de/d22/dbb 39 2024-02-28T16:31:15.273 INFO:tasks.workunit.client.0.smithi001.stdout:5/633: chown dc/d12/l5a 25419579 1 2024-02-28T16:31:15.274 INFO:tasks.workunit.client.1.smithi066.stdout:5/225: stat d3/d4/dc/ce 0 2024-02-28T16:31:15.276 INFO:tasks.workunit.client.1.smithi066.stdout:2/265: rmdir d0/d12/d1a 39 2024-02-28T16:31:15.276 INFO:tasks.workunit.client.0.smithi001.stdout:6/635: dread d9/da6/f30 [0,4194304] 0 2024-02-28T16:31:15.280 INFO:tasks.workunit.client.1.smithi066.stdout:2/266: write d0/f37 [758172,69726] 0 2024-02-28T16:31:15.283 INFO:tasks.workunit.client.0.smithi001.stdout:7/650: creat d3/d17/d52/d55/dca/fce x:0 0 0 2024-02-28T16:31:15.284 INFO:tasks.workunit.client.1.smithi066.stdout:2/267: write d0/d12/d3e/f4e [500394,73180] 0 2024-02-28T16:31:15.290 INFO:tasks.workunit.client.1.smithi066.stdout:4/351: mkdir d1/d23/d25/d77 0 2024-02-28T16:31:15.291 INFO:tasks.workunit.client.1.smithi066.stdout:7/158: symlink dc/de/l30 0 2024-02-28T16:31:15.292 INFO:tasks.workunit.client.1.smithi066.stdout:9/204: dread da/fb [4194304,4194304] 0 2024-02-28T16:31:15.293 INFO:tasks.workunit.client.0.smithi001.stdout:1/790: read - d0/d9/d1a/d2b/d26/d27/d30/d67/d87/f9a zero size 2024-02-28T16:31:15.295 INFO:tasks.workunit.client.1.smithi066.stdout:3/246: symlink db/d2a/l53 0 2024-02-28T16:31:15.297 INFO:tasks.workunit.client.1.smithi066.stdout:8/214: creat d0/d6/dc/f41 x:0 0 0 2024-02-28T16:31:15.298 INFO:tasks.workunit.client.1.smithi066.stdout:6/197: dwrite d2/da/d19/f17 [0,4194304] 0 2024-02-28T16:31:15.300 INFO:tasks.workunit.client.1.smithi066.stdout:5/226: readlink d3/d1b/d20/l3b 0 2024-02-28T16:31:15.302 INFO:tasks.workunit.client.1.smithi066.stdout:9/205: dread da/f30 [0,4194304] 0 2024-02-28T16:31:15.306 INFO:tasks.workunit.client.0.smithi001.stdout:9/757: dread d5/d45/d6f/fbd [0,4194304] 0 2024-02-28T16:31:15.308 INFO:tasks.workunit.client.1.smithi066.stdout:0/341: link d2/c42 d2/d9/d10/d50/d64/c71 0 2024-02-28T16:31:15.313 INFO:tasks.workunit.client.1.smithi066.stdout:7/159: mknod dc/d29/c31 0 2024-02-28T16:31:15.323 INFO:tasks.workunit.client.0.smithi001.stdout:7/651: dwrite d3/d1c/fad [0,4194304] 0 2024-02-28T16:31:15.326 INFO:tasks.workunit.client.1.smithi066.stdout:5/227: mknod d3/d4/dc/d2f/c4d 0 2024-02-28T16:31:15.329 INFO:tasks.workunit.client.0.smithi001.stdout:6/636: dwrite d9/d29/d6e/d53/f9e [0,4194304] 0 2024-02-28T16:31:15.330 INFO:tasks.workunit.client.1.smithi066.stdout:4/352: rmdir d1/d23/d70 0 2024-02-28T16:31:15.333 INFO:tasks.workunit.client.1.smithi066.stdout:7/160: dread - dc/de/d17/d1e/f2d zero size 2024-02-28T16:31:15.333 INFO:tasks.workunit.client.0.smithi001.stdout:0/638: dwrite d18/f1e [0,4194304] 0 2024-02-28T16:31:15.334 INFO:tasks.workunit.client.1.smithi066.stdout:8/215: dwrite d0/db/d15/d32/f39 [0,4194304] 0 2024-02-28T16:31:15.335 INFO:tasks.workunit.client.0.smithi001.stdout:0/639: stat d18/d22/d57/l5d 0 2024-02-28T16:31:15.335 INFO:tasks.workunit.client.1.smithi066.stdout:4/353: chown d1/d23/f36 49529328 1 2024-02-28T16:31:15.336 INFO:tasks.workunit.client.0.smithi001.stdout:0/640: fsync d18/d22/d76/d7a/fae 0 2024-02-28T16:31:15.337 INFO:tasks.workunit.client.1.smithi066.stdout:2/268: rename d0/d12/d2e/d38/l3c to d0/d12/d1a/d23/d29/d21/d2f/l56 0 2024-02-28T16:31:15.339 INFO:tasks.workunit.client.0.smithi001.stdout:5/634: rename dc/d12/d41/l45 to dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/ld1 0 2024-02-28T16:31:15.339 INFO:tasks.workunit.client.0.smithi001.stdout:1/791: readlink d0/d9/d1a/d72/la3 0 2024-02-28T16:31:15.340 INFO:tasks.workunit.client.1.smithi066.stdout:3/247: dwrite db/d13/f19 [0,4194304] 0 2024-02-28T16:31:15.346 INFO:tasks.workunit.client.0.smithi001.stdout:1/792: truncate d0/d9/d1a/d2b/d26/d27/d30/d88/fef 318409 0 2024-02-28T16:31:15.349 INFO:tasks.workunit.client.1.smithi066.stdout:2/269: truncate d0/d12/d1a/d39/d40/f41 748839 0 2024-02-28T16:31:15.349 INFO:tasks.workunit.client.0.smithi001.stdout:2/650: creat d0/d2/d3/de/d22/fd8 x:0 0 0 2024-02-28T16:31:15.351 INFO:tasks.workunit.client.0.smithi001.stdout:0/641: creat d18/d22/d76/fd1 x:0 0 0 2024-02-28T16:31:15.355 INFO:tasks.workunit.client.1.smithi066.stdout:2/270: fdatasync d0/d12/d3e/f4e 0 2024-02-28T16:31:15.356 INFO:tasks.workunit.client.1.smithi066.stdout:8/216: stat d0/d6/dc/d16/l30 0 2024-02-28T16:31:15.361 INFO:tasks.workunit.client.1.smithi066.stdout:3/248: mknod db/d2a/c54 0 2024-02-28T16:31:15.363 INFO:tasks.workunit.client.0.smithi001.stdout:9/758: rename d5/c8 to d5/d27/d34/dc1/cfb 0 2024-02-28T16:31:15.364 INFO:tasks.workunit.client.1.smithi066.stdout:6/198: getdents d2/d2c 0 2024-02-28T16:31:15.366 INFO:tasks.workunit.client.0.smithi001.stdout:5/635: truncate dc/d12/d56/d96/db9/dc2/dca/d3c/f5e 2380328 0 2024-02-28T16:31:15.367 INFO:tasks.workunit.client.0.smithi001.stdout:6/637: creat d9/db/fd8 x:0 0 0 2024-02-28T16:31:15.367 INFO:tasks.workunit.client.1.smithi066.stdout:7/161: mkdir dc/de/d32 0 2024-02-28T16:31:15.371 INFO:tasks.workunit.client.1.smithi066.stdout:8/217: mkdir d0/d22/d42 0 2024-02-28T16:31:15.372 INFO:tasks.workunit.client.1.smithi066.stdout:5/228: creat d3/d4/f4e x:0 0 0 2024-02-28T16:31:15.373 INFO:tasks.workunit.client.0.smithi001.stdout:4/706: link d3/d5/d7/d1f/f96 d3/d1c/d58/d6e/d9f/fe3 0 2024-02-28T16:31:15.374 INFO:tasks.workunit.client.0.smithi001.stdout:9/759: write d5/d27/d34/d50/f92 [3805566,64446] 0 2024-02-28T16:31:15.374 INFO:tasks.workunit.client.1.smithi066.stdout:3/249: creat db/dc/f55 x:0 0 0 2024-02-28T16:31:15.374 INFO:tasks.workunit.client.1.smithi066.stdout:7/162: rmdir dc/d29 39 2024-02-28T16:31:15.376 INFO:tasks.workunit.client.1.smithi066.stdout:4/354: creat d1/d18/f78 x:0 0 0 2024-02-28T16:31:15.376 INFO:tasks.workunit.client.1.smithi066.stdout:5/229: stat d3/d4/dc/d2f/f33 0 2024-02-28T16:31:15.377 INFO:tasks.workunit.client.0.smithi001.stdout:9/760: chown d5/d45/l85 987374 1 2024-02-28T16:31:15.378 INFO:tasks.workunit.client.0.smithi001.stdout:7/652: rename d3/f1e to d3/d17/d3d/d5d/fcf 0 2024-02-28T16:31:15.381 INFO:tasks.workunit.client.1.smithi066.stdout:7/163: fsync dc/de/d17/d1e/f2d 0 2024-02-28T16:31:15.382 INFO:tasks.workunit.client.0.smithi001.stdout:5/636: unlink dc/l8e 0 2024-02-28T16:31:15.382 INFO:tasks.workunit.client.0.smithi001.stdout:4/707: mknod d3/d5/d4e/ce4 0 2024-02-28T16:31:15.384 INFO:tasks.workunit.client.1.smithi066.stdout:2/271: getdents d0 0 2024-02-28T16:31:15.385 INFO:tasks.workunit.client.0.smithi001.stdout:4/708: write d3/d2d/d81/fae [569922,69488] 0 2024-02-28T16:31:15.391 INFO:tasks.workunit.client.1.smithi066.stdout:9/206: truncate da/f30 449604 0 2024-02-28T16:31:15.391 INFO:tasks.workunit.client.0.smithi001.stdout:5/637: creat dc/df/fd2 x:0 0 0 2024-02-28T16:31:15.393 INFO:tasks.workunit.client.1.smithi066.stdout:4/355: rename d1/d23/d25/d3e/d40/l63 to d1/d18/d41/l79 0 2024-02-28T16:31:15.396 INFO:tasks.workunit.client.0.smithi001.stdout:8/671: sync 2024-02-28T16:31:15.399 INFO:tasks.workunit.client.1.smithi066.stdout:8/218: creat d0/f43 x:0 0 0 2024-02-28T16:31:15.400 INFO:tasks.workunit.client.0.smithi001.stdout:8/672: dread d4/d7/d22/d27/d4b/f63 [0,4194304] 0 2024-02-28T16:31:15.403 INFO:tasks.workunit.client.1.smithi066.stdout:3/250: link db/d13/d14/l46 db/d13/d1a/d33/l56 0 2024-02-28T16:31:15.404 INFO:tasks.workunit.client.1.smithi066.stdout:2/272: dwrite d0/f31 [4194304,4194304] 0 2024-02-28T16:31:15.404 INFO:tasks.workunit.client.1.smithi066.stdout:9/207: mknod da/df/d18/c45 0 2024-02-28T16:31:15.405 INFO:tasks.workunit.client.1.smithi066.stdout:8/219: truncate d0/db/f27 182933 0 2024-02-28T16:31:15.407 INFO:tasks.workunit.client.0.smithi001.stdout:5/638: getdents dc/d12/d56/d96 0 2024-02-28T16:31:15.407 INFO:tasks.workunit.client.0.smithi001.stdout:5/639: readlink dc/l1c 0 2024-02-28T16:31:15.409 INFO:tasks.workunit.client.1.smithi066.stdout:7/164: creat dc/f33 x:0 0 0 2024-02-28T16:31:15.411 INFO:tasks.workunit.client.0.smithi001.stdout:2/651: truncate d0/d2/d3/f54 427642 0 2024-02-28T16:31:15.412 INFO:tasks.workunit.client.0.smithi001.stdout:6/638: write d9/d29/f87 [1459624,11348] 0 2024-02-28T16:31:15.414 INFO:tasks.workunit.client.0.smithi001.stdout:5/640: mknod dc/d1a/d52/cd3 0 2024-02-28T16:31:15.415 INFO:tasks.workunit.client.0.smithi001.stdout:6/639: readlink d9/db/la5 0 2024-02-28T16:31:15.418 INFO:tasks.workunit.client.0.smithi001.stdout:1/793: dwrite d0/d9/d1a/d2b/d26/f82 [0,4194304] 0 2024-02-28T16:31:15.419 INFO:tasks.workunit.client.0.smithi001.stdout:5/641: chown dc/d12/d56 0 1 2024-02-28T16:31:15.419 INFO:tasks.workunit.client.1.smithi066.stdout:4/356: dwrite d1/d23/f75 [0,4194304] 0 2024-02-28T16:31:15.420 INFO:tasks.workunit.client.0.smithi001.stdout:9/761: write d5/d27/db1/d58/f9b [507588,20310] 0 2024-02-28T16:31:15.430 INFO:tasks.workunit.client.0.smithi001.stdout:0/642: dwrite d18/fab [0,4194304] 0 2024-02-28T16:31:15.432 INFO:tasks.workunit.client.0.smithi001.stdout:6/640: mkdir d9/db/dd9 0 2024-02-28T16:31:15.432 INFO:tasks.workunit.client.0.smithi001.stdout:4/709: dread d3/f12 [0,4194304] 0 2024-02-28T16:31:15.435 INFO:tasks.workunit.client.0.smithi001.stdout:5/642: unlink dc/d1a/d21/f37 0 2024-02-28T16:31:15.436 INFO:tasks.workunit.client.0.smithi001.stdout:1/794: mkdir d0/d9/d1a/d2b/d26/d27/d113 0 2024-02-28T16:31:15.438 INFO:tasks.workunit.client.0.smithi001.stdout:6/641: chown d9/db/d19/fd5 3751 1 2024-02-28T16:31:15.439 INFO:tasks.workunit.client.0.smithi001.stdout:1/795: chown d0/d9/d1a/d2b/d26 111 1 2024-02-28T16:31:15.450 INFO:tasks.workunit.client.0.smithi001.stdout:4/710: unlink d3/d1c/d58/d6c/d79/fb5 0 2024-02-28T16:31:15.452 INFO:tasks.workunit.client.0.smithi001.stdout:6/642: symlink d9/de/lda 0 2024-02-28T16:31:15.454 INFO:tasks.workunit.client.0.smithi001.stdout:2/652: write d0/d2/d3/de/d22/f6f [454647,116130] 0 2024-02-28T16:31:15.454 INFO:tasks.workunit.client.0.smithi001.stdout:5/643: rmdir dc/d12/d56/d96/db9/dc2/dca 39 2024-02-28T16:31:15.456 INFO:tasks.workunit.client.0.smithi001.stdout:8/673: dread d4/d7/da8/f91 [8388608,4194304] 0 2024-02-28T16:31:15.457 INFO:tasks.workunit.client.1.smithi066.stdout:1/279: sync 2024-02-28T16:31:15.458 INFO:tasks.workunit.client.0.smithi001.stdout:7/653: symlink d3/d17/d52/d7f/d99/dc1/d49/d94/ld0 0 2024-02-28T16:31:15.463 INFO:tasks.workunit.client.0.smithi001.stdout:4/711: symlink d3/d2d/d4c/d69/d82/le5 0 2024-02-28T16:31:15.466 INFO:tasks.workunit.client.1.smithi066.stdout:8/220: creat d0/db/d15/f44 x:0 0 0 2024-02-28T16:31:15.467 INFO:tasks.workunit.client.0.smithi001.stdout:1/796: creat d0/d9/d1a/d2b/d26/d105/f114 x:0 0 0 2024-02-28T16:31:15.468 INFO:tasks.workunit.client.0.smithi001.stdout:2/653: rmdir d0/d2/d3/de/d61/d12/d6a 39 2024-02-28T16:31:15.468 INFO:tasks.workunit.client.0.smithi001.stdout:3/686: sync 2024-02-28T16:31:15.470 INFO:tasks.workunit.client.0.smithi001.stdout:7/654: creat d3/d17/d3d/db3/fd1 x:0 0 0 2024-02-28T16:31:15.470 INFO:tasks.workunit.client.1.smithi066.stdout:5/230: truncate d3/d4/dc/d2f/f33 2992642 0 2024-02-28T16:31:15.472 INFO:tasks.workunit.client.0.smithi001.stdout:0/643: getdents d18/d22/d57 0 2024-02-28T16:31:15.473 INFO:tasks.workunit.client.0.smithi001.stdout:9/762: dwrite d5/d27/db1/d33/fc9 [0,4194304] 0 2024-02-28T16:31:15.475 INFO:tasks.workunit.client.0.smithi001.stdout:8/674: fdatasync d4/f59 0 2024-02-28T16:31:15.475 INFO:tasks.workunit.client.1.smithi066.stdout:4/357: write d1/d23/d25/d3e/f3f [518800,113013] 0 2024-02-28T16:31:15.477 INFO:tasks.workunit.client.0.smithi001.stdout:9/763: write d5/d27/db1/f93 [5029033,66142] 0 2024-02-28T16:31:15.481 INFO:tasks.workunit.client.0.smithi001.stdout:6/643: creat d9/fdb x:0 0 0 2024-02-28T16:31:15.481 INFO:tasks.workunit.client.0.smithi001.stdout:1/797: read - d0/d9/d1a/d2b/d26/fcb zero size 2024-02-28T16:31:15.481 INFO:tasks.workunit.client.0.smithi001.stdout:3/687: rmdir d4/d18/da0 39 2024-02-28T16:31:15.482 INFO:tasks.workunit.client.0.smithi001.stdout:0/644: rmdir d18/d22/d2b/d36/d4d 39 2024-02-28T16:31:15.487 INFO:tasks.workunit.client.0.smithi001.stdout:6/644: read - d9/db/d5f/d62/d83/f8f zero size 2024-02-28T16:31:15.488 INFO:tasks.workunit.client.1.smithi066.stdout:5/231: read - d3/d4/dc/dd/d2e/d18/f4b zero size 2024-02-28T16:31:15.488 INFO:tasks.workunit.client.1.smithi066.stdout:2/273: dread d0/f31 [4194304,4194304] 0 2024-02-28T16:31:15.490 INFO:tasks.workunit.client.0.smithi001.stdout:8/675: rename d4/d7/c8a to d4/d7/d22/d27/d58/d94/dbc/cdb 0 2024-02-28T16:31:15.493 INFO:tasks.workunit.client.0.smithi001.stdout:6/645: write d9/db/d5f/d62/d83/fb6 [329686,87396] 0 2024-02-28T16:31:15.495 INFO:tasks.workunit.client.0.smithi001.stdout:0/645: creat d18/d22/d2f/fd2 x:0 0 0 2024-02-28T16:31:15.496 INFO:tasks.workunit.client.0.smithi001.stdout:8/676: chown d4/d7/d22/d27/d58/d94/dbc 9 1 2024-02-28T16:31:15.496 INFO:tasks.workunit.client.1.smithi066.stdout:8/221: getdents d0/db/d12/d1d 0 2024-02-28T16:31:15.496 INFO:tasks.workunit.client.1.smithi066.stdout:6/199: dwrite f1 [0,4194304] 0 2024-02-28T16:31:15.497 INFO:tasks.workunit.client.0.smithi001.stdout:6/646: dread - d9/db/d19/f78 zero size 2024-02-28T16:31:15.499 INFO:tasks.workunit.client.1.smithi066.stdout:9/208: dwrite da/df/f2f [4194304,4194304] 0 2024-02-28T16:31:15.499 INFO:tasks.workunit.client.0.smithi001.stdout:1/798: readlink d0/d9/d1a/d2b/d26/d27/d30/d52/l78 0 2024-02-28T16:31:15.500 INFO:tasks.workunit.client.0.smithi001.stdout:4/712: getdents d3/d5/d92 0 2024-02-28T16:31:15.500 INFO:tasks.workunit.client.0.smithi001.stdout:2/654: dwrite d0/d2/d3/de/d22/f48 [0,4194304] 0 2024-02-28T16:31:15.504 INFO:tasks.workunit.client.1.smithi066.stdout:2/274: symlink d0/d12/d1a/d23/l57 0 2024-02-28T16:31:15.504 INFO:tasks.workunit.client.1.smithi066.stdout:6/200: rename d2/d2c to d2/d2c/d40 22 2024-02-28T16:31:15.504 INFO:tasks.workunit.client.1.smithi066.stdout:7/165: dwrite fa [0,4194304] 0 2024-02-28T16:31:15.504 INFO:tasks.workunit.client.1.smithi066.stdout:9/209: readlink da/d22/l24 0 2024-02-28T16:31:15.506 INFO:tasks.workunit.client.0.smithi001.stdout:5/644: truncate dc/df/f13 944280 0 2024-02-28T16:31:15.513 INFO:tasks.workunit.client.1.smithi066.stdout:5/232: symlink d3/d4/dc/l4f 0 2024-02-28T16:31:15.514 INFO:tasks.workunit.client.0.smithi001.stdout:0/646: creat d18/d22/d2b/d95/fd3 x:0 0 0 2024-02-28T16:31:15.516 INFO:tasks.workunit.client.1.smithi066.stdout:4/358: dwrite d1/d23/d25/d43/f65 [0,4194304] 0 2024-02-28T16:31:15.517 INFO:tasks.workunit.client.1.smithi066.stdout:8/222: mknod d0/d6/dc/d16/d1c/c45 0 2024-02-28T16:31:15.518 INFO:tasks.workunit.client.0.smithi001.stdout:8/677: mknod d4/d7/d22/d27/d8e/dae/cdc 0 2024-02-28T16:31:15.518 INFO:tasks.workunit.client.1.smithi066.stdout:7/166: creat dc/d29/f34 x:0 0 0 2024-02-28T16:31:15.519 INFO:tasks.workunit.client.0.smithi001.stdout:5/645: rmdir dc/d12/d56/d96 39 2024-02-28T16:31:15.520 INFO:tasks.workunit.client.1.smithi066.stdout:5/233: mknod d3/d4/dc/dd/d2e/d18/c50 0 2024-02-28T16:31:15.522 INFO:tasks.workunit.client.1.smithi066.stdout:6/201: link d2/f14 d2/da/d19/f41 0 2024-02-28T16:31:15.523 INFO:tasks.workunit.client.0.smithi001.stdout:2/655: rmdir d0/d2/d3/de/d61/d12/d6a/d84 39 2024-02-28T16:31:15.524 INFO:tasks.workunit.client.1.smithi066.stdout:4/359: chown d1/d23/d25/d3e/d40/f64 763 1 2024-02-28T16:31:15.527 INFO:tasks.workunit.client.0.smithi001.stdout:9/764: dread d5/d27/db1/fa3 [0,4194304] 0 2024-02-28T16:31:15.530 INFO:tasks.workunit.client.1.smithi066.stdout:6/202: creat d2/d2a/f42 x:0 0 0 2024-02-28T16:31:15.533 INFO:tasks.workunit.client.0.smithi001.stdout:8/678: mknod d4/cdd 0 2024-02-28T16:31:15.536 INFO:tasks.workunit.client.1.smithi066.stdout:1/280: dread d2/f10 [0,4194304] 0 2024-02-28T16:31:15.536 INFO:tasks.workunit.client.0.smithi001.stdout:7/655: dwrite d3/d1c/fcb [0,4194304] 0 2024-02-28T16:31:15.537 INFO:tasks.workunit.client.1.smithi066.stdout:5/234: unlink d3/d1b/c3c 0 2024-02-28T16:31:15.540 INFO:tasks.workunit.client.0.smithi001.stdout:9/765: chown d5/d27/db1/d33/d8a/da8 1859291127 1 2024-02-28T16:31:15.540 INFO:tasks.workunit.client.0.smithi001.stdout:3/688: dwrite d4/d18/fbf [0,4194304] 0 2024-02-28T16:31:15.542 INFO:tasks.workunit.client.1.smithi066.stdout:4/360: mkdir d1/d7a 0 2024-02-28T16:31:15.543 INFO:tasks.workunit.client.1.smithi066.stdout:5/235: truncate d3/d4/f9 5237202 0 2024-02-28T16:31:15.543 INFO:tasks.workunit.client.1.smithi066.stdout:6/203: fdatasync d2/da/fe 0 2024-02-28T16:31:15.544 INFO:tasks.workunit.client.0.smithi001.stdout:3/689: chown d4/d18/d5f 810189220 1 2024-02-28T16:31:15.545 INFO:tasks.workunit.client.0.smithi001.stdout:0/647: rename d18/d22/d2b/f37 to d18/d31/dbc/fd4 0 2024-02-28T16:31:15.548 INFO:tasks.workunit.client.0.smithi001.stdout:4/713: dwrite d3/fd6 [0,4194304] 0 2024-02-28T16:31:15.549 INFO:tasks.workunit.client.1.smithi066.stdout:6/204: write d2/d26/f31 [4423385,128457] 0 2024-02-28T16:31:15.550 INFO:tasks.workunit.client.0.smithi001.stdout:9/766: chown d5/d27/d34/c74 14332 1 2024-02-28T16:31:15.550 INFO:tasks.workunit.client.0.smithi001.stdout:6/647: dwrite d9/da6/f30 [0,4194304] 0 2024-02-28T16:31:15.551 INFO:tasks.workunit.client.0.smithi001.stdout:3/690: fsync d4/d6/d7/f2f 0 2024-02-28T16:31:15.551 INFO:tasks.workunit.client.0.smithi001.stdout:2/656: mkdir d0/d2/dd9 0 2024-02-28T16:31:15.555 INFO:tasks.workunit.client.0.smithi001.stdout:1/799: truncate d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/f8c 1438812 0 2024-02-28T16:31:15.556 INFO:tasks.workunit.client.0.smithi001.stdout:2/657: stat d0/d2/d3/de/d8a/ccb 0 2024-02-28T16:31:15.558 INFO:tasks.workunit.client.1.smithi066.stdout:9/210: truncate da/f20 3681931 0 2024-02-28T16:31:15.561 INFO:tasks.workunit.client.1.smithi066.stdout:9/211: chown da/df/d18/f3b 324674 1 2024-02-28T16:31:15.563 INFO:tasks.workunit.client.1.smithi066.stdout:1/281: dread d2/d18/d19/f24 [0,4194304] 0 2024-02-28T16:31:15.564 INFO:tasks.workunit.client.0.smithi001.stdout:8/679: symlink d4/d7/d22/d27/d4b/d80/lde 0 2024-02-28T16:31:15.570 INFO:tasks.workunit.client.0.smithi001.stdout:0/648: creat d18/d22/d57/fd5 x:0 0 0 2024-02-28T16:31:15.571 INFO:tasks.workunit.client.1.smithi066.stdout:6/205: getdents d2/da/d38 0 2024-02-28T16:31:15.572 INFO:tasks.workunit.client.0.smithi001.stdout:0/649: chown d18/d22/d2b/f41 20401 1 2024-02-28T16:31:15.577 INFO:tasks.workunit.client.0.smithi001.stdout:7/656: symlink d3/d17/d52/d7f/d99/dc1/d49/d8d/db9/ld2 0 2024-02-28T16:31:15.577 INFO:tasks.workunit.client.0.smithi001.stdout:7/657: chown d3/d17 777434128 1 2024-02-28T16:31:15.578 INFO:tasks.workunit.client.1.smithi066.stdout:7/167: dwrite dc/de/f13 [0,4194304] 0 2024-02-28T16:31:15.579 INFO:tasks.workunit.client.0.smithi001.stdout:6/648: creat d9/de/dbd/fdc x:0 0 0 2024-02-28T16:31:15.580 INFO:tasks.workunit.client.1.smithi066.stdout:5/236: truncate d3/d4/dc/f1d 1633136 0 2024-02-28T16:31:15.581 INFO:tasks.workunit.client.0.smithi001.stdout:6/649: fsync d9/de/f2c 0 2024-02-28T16:31:15.581 INFO:tasks.workunit.client.1.smithi066.stdout:1/282: getdents d2 0 2024-02-28T16:31:15.585 INFO:tasks.workunit.client.1.smithi066.stdout:5/237: write d3/d4/dc/dd/d2e/f40 [1386008,110414] 0 2024-02-28T16:31:15.585 INFO:tasks.workunit.client.0.smithi001.stdout:4/714: rename d3/d5/d4e to d3/d5/d7/d1f/d20/de0/de6 0 2024-02-28T16:31:15.592 INFO:tasks.workunit.client.0.smithi001.stdout:1/800: fsync d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/f6e 0 2024-02-28T16:31:15.593 INFO:tasks.workunit.client.1.smithi066.stdout:1/283: rename d2/f17 to d2/d27/d2b/d34/f55 0 2024-02-28T16:31:15.595 INFO:tasks.workunit.client.1.smithi066.stdout:7/168: dread dc/f25 [0,4194304] 0 2024-02-28T16:31:15.596 INFO:tasks.workunit.client.1.smithi066.stdout:6/206: dread d2/d26/f31 [0,4194304] 0 2024-02-28T16:31:15.596 INFO:tasks.workunit.client.1.smithi066.stdout:5/238: mkdir d3/d4/dc/dd/d41/d51 0 2024-02-28T16:31:15.598 INFO:tasks.workunit.client.1.smithi066.stdout:7/169: chown dc/de/d32 3657 1 2024-02-28T16:31:15.601 INFO:tasks.workunit.client.1.smithi066.stdout:7/170: fdatasync f5 0 2024-02-28T16:31:15.603 INFO:tasks.workunit.client.0.smithi001.stdout:0/650: rmdir d18/d22/d57 39 2024-02-28T16:31:15.603 INFO:tasks.workunit.client.1.smithi066.stdout:7/171: fsync f1 0 2024-02-28T16:31:15.604 INFO:tasks.workunit.client.0.smithi001.stdout:7/658: creat d3/d17/d61/fd3 x:0 0 0 2024-02-28T16:31:15.604 INFO:tasks.workunit.client.1.smithi066.stdout:5/239: symlink d3/d4/dc/d42/l52 0 2024-02-28T16:31:15.605 INFO:tasks.workunit.client.0.smithi001.stdout:4/715: symlink d3/d5/d7/d1f/d20/de0/de6/le7 0 2024-02-28T16:31:15.605 INFO:tasks.workunit.client.1.smithi066.stdout:6/207: creat d2/f43 x:0 0 0 2024-02-28T16:31:15.606 INFO:tasks.workunit.client.1.smithi066.stdout:4/361: dread d1/d18/f4a [0,4194304] 0 2024-02-28T16:31:15.607 INFO:tasks.workunit.client.1.smithi066.stdout:9/212: dwrite da/f30 [0,4194304] 0 2024-02-28T16:31:15.608 INFO:tasks.workunit.client.1.smithi066.stdout:7/172: write dc/f16 [4990511,21106] 0 2024-02-28T16:31:15.609 INFO:tasks.workunit.client.1.smithi066.stdout:4/362: dread - d1/f4d zero size 2024-02-28T16:31:15.609 INFO:tasks.workunit.client.1.smithi066.stdout:5/240: readlink d3/l6 0 2024-02-28T16:31:15.609 INFO:tasks.workunit.client.0.smithi001.stdout:8/680: symlink d4/d32/d69/d9a/dcd/ldf 0 2024-02-28T16:31:15.611 INFO:tasks.workunit.client.1.smithi066.stdout:6/208: write d2/da/d19/f1d [401045,76140] 0 2024-02-28T16:31:15.611 INFO:tasks.workunit.client.0.smithi001.stdout:9/767: dread d5/d27/d34/fe7 [0,4194304] 0 2024-02-28T16:31:15.612 INFO:tasks.workunit.client.0.smithi001.stdout:2/658: write d0/d2/f8c [801150,37345] 0 2024-02-28T16:31:15.614 INFO:tasks.workunit.client.0.smithi001.stdout:6/650: mknod d9/db/d5f/cdd 0 2024-02-28T16:31:15.618 INFO:tasks.workunit.client.1.smithi066.stdout:9/213: mkdir da/d22/d46 0 2024-02-28T16:31:15.618 INFO:tasks.workunit.client.1.smithi066.stdout:5/241: read d3/d4/dc/dd/d2e/d18/f4a [97862,25569] 0 2024-02-28T16:31:15.618 INFO:tasks.workunit.client.1.smithi066.stdout:4/363: chown d1/l17 182 1 2024-02-28T16:31:15.618 INFO:tasks.workunit.client.0.smithi001.stdout:1/801: symlink d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/df5/l115 0 2024-02-28T16:31:15.619 INFO:tasks.workunit.client.1.smithi066.stdout:0/342: sync 2024-02-28T16:31:15.619 INFO:tasks.workunit.client.0.smithi001.stdout:2/659: chown d0/d2/d3/de/d61/d12/d46/ca4 74 1 2024-02-28T16:31:15.619 INFO:tasks.workunit.client.1.smithi066.stdout:4/364: chown d1/f16 48 1 2024-02-28T16:31:15.621 INFO:tasks.workunit.client.1.smithi066.stdout:4/365: readlink d1/d18/d41/l6d 0 2024-02-28T16:31:15.623 INFO:tasks.workunit.client.0.smithi001.stdout:7/659: rename d3/d17/d3d/d59/l76 to d3/d17/d52/d55/dca/ld4 0 2024-02-28T16:31:15.623 INFO:tasks.workunit.client.1.smithi066.stdout:9/214: rename da/d22/f39 to da/d3a/f47 0 2024-02-28T16:31:15.625 INFO:tasks.workunit.client.1.smithi066.stdout:4/366: truncate d1/fe 4664527 0 2024-02-28T16:31:15.625 INFO:tasks.workunit.client.1.smithi066.stdout:9/215: dread - da/d3a/f41 zero size 2024-02-28T16:31:15.626 INFO:tasks.workunit.client.1.smithi066.stdout:4/367: fdatasync d1/fe 0 2024-02-28T16:31:15.629 INFO:tasks.workunit.client.0.smithi001.stdout:0/651: unlink d18/da9/dc9/lb4 0 2024-02-28T16:31:15.629 INFO:tasks.workunit.client.0.smithi001.stdout:3/691: getdents d4/d6 0 2024-02-28T16:31:15.629 INFO:tasks.workunit.client.1.smithi066.stdout:0/343: write d2/d9/d10/f46 [2250984,30984] 0 2024-02-28T16:31:15.631 INFO:tasks.workunit.client.0.smithi001.stdout:4/716: dread d3/d5/d7/f21 [0,4194304] 0 2024-02-28T16:31:15.633 INFO:tasks.workunit.client.0.smithi001.stdout:9/768: rename d5/d27/db1/f3a to d5/d27/d34/ffc 0 2024-02-28T16:31:15.633 INFO:tasks.workunit.client.0.smithi001.stdout:2/660: symlink d0/d2/lda 0 2024-02-28T16:31:15.635 INFO:tasks.workunit.client.1.smithi066.stdout:7/173: dread f5 [0,4194304] 0 2024-02-28T16:31:15.635 INFO:tasks.workunit.client.1.smithi066.stdout:6/209: dwrite d2/da/d19/f17 [0,4194304] 0 2024-02-28T16:31:15.635 INFO:tasks.workunit.client.1.smithi066.stdout:9/216: mkdir da/d3a/d15/d48 0 2024-02-28T16:31:15.637 INFO:tasks.workunit.client.1.smithi066.stdout:0/344: rmdir d2/d9/d33/d3f 39 2024-02-28T16:31:15.638 INFO:tasks.workunit.client.1.smithi066.stdout:7/174: chown dc/de/d17/d1e 3247138 1 2024-02-28T16:31:15.639 INFO:tasks.workunit.client.0.smithi001.stdout:4/717: rmdir d3/d5/d47/d90 39 2024-02-28T16:31:15.640 INFO:tasks.workunit.client.0.smithi001.stdout:2/661: mknod d0/d2/d3/d36/d94/cdb 0 2024-02-28T16:31:15.641 INFO:tasks.workunit.client.1.smithi066.stdout:4/368: chown d1/f4 5414 1 2024-02-28T16:31:15.641 INFO:tasks.workunit.client.0.smithi001.stdout:4/718: chown d3/d1c/d58/d88/ca9 62316 1 2024-02-28T16:31:15.642 INFO:tasks.workunit.client.1.smithi066.stdout:6/210: truncate d2/da/f1c 732406 0 2024-02-28T16:31:15.642 INFO:tasks.workunit.client.1.smithi066.stdout:7/175: read - dc/d29/f34 zero size 2024-02-28T16:31:15.644 INFO:tasks.workunit.client.1.smithi066.stdout:0/345: read d2/d9/d10/f46 [203032,107920] 0 2024-02-28T16:31:15.645 INFO:tasks.workunit.client.0.smithi001.stdout:2/662: creat d0/d2/da6/fdc x:0 0 0 2024-02-28T16:31:15.645 INFO:tasks.workunit.client.0.smithi001.stdout:4/719: chown d3/d2d/d4c/d69/d82/c67 184338371 1 2024-02-28T16:31:15.646 INFO:tasks.workunit.client.1.smithi066.stdout:4/369: write d1/d23/d25/d3e/d40/f72 [795567,101158] 0 2024-02-28T16:31:15.650 INFO:tasks.workunit.client.0.smithi001.stdout:4/720: mkdir d3/d1c/de8 0 2024-02-28T16:31:15.651 INFO:tasks.workunit.client.1.smithi066.stdout:0/346: write d2/d9/d33/f62 [220509,1365] 0 2024-02-28T16:31:15.651 INFO:tasks.workunit.client.1.smithi066.stdout:1/284: write d2/f16 [2266998,47713] 0 2024-02-28T16:31:15.652 INFO:tasks.workunit.client.1.smithi066.stdout:9/217: rename da/df/f34 to da/df/d18/d28/d42/f49 0 2024-02-28T16:31:15.652 INFO:tasks.workunit.client.1.smithi066.stdout:0/347: dread - d2/f32 zero size 2024-02-28T16:31:15.653 INFO:tasks.workunit.client.0.smithi001.stdout:6/651: write d9/db/d5f/d62/d83/f99 [217766,38924] 0 2024-02-28T16:31:15.653 INFO:tasks.workunit.client.0.smithi001.stdout:8/681: dwrite d4/d47/f64 [0,4194304] 0 2024-02-28T16:31:15.654 INFO:tasks.workunit.client.0.smithi001.stdout:2/663: creat d0/d2/dd9/fdd x:0 0 0 2024-02-28T16:31:15.655 INFO:tasks.workunit.client.0.smithi001.stdout:0/652: link d18/d22/d2b/d36/d4d/f65 d18/d22/d3d/fd6 0 2024-02-28T16:31:15.657 INFO:tasks.workunit.client.0.smithi001.stdout:4/721: rmdir d3/d5/d92 39 2024-02-28T16:31:15.658 INFO:tasks.workunit.client.0.smithi001.stdout:9/769: write d5/d27/db1/f5f [170918,7829] 0 2024-02-28T16:31:15.659 INFO:tasks.workunit.client.0.smithi001.stdout:7/660: dwrite d3/d17/d3d/d42/d8e/f9b [0,4194304] 0 2024-02-28T16:31:15.660 INFO:tasks.workunit.client.0.smithi001.stdout:4/722: readlink d3/d5/d7/l10 0 2024-02-28T16:31:15.661 INFO:tasks.workunit.client.1.smithi066.stdout:9/218: mknod da/df/d18/d28/d42/c4a 0 2024-02-28T16:31:15.661 INFO:tasks.workunit.client.1.smithi066.stdout:3/251: sync 2024-02-28T16:31:15.661 INFO:tasks.workunit.client.0.smithi001.stdout:6/652: rename d9/l17 to d9/db/d5f/d62/d83/lde 0 2024-02-28T16:31:15.664 INFO:tasks.workunit.client.0.smithi001.stdout:8/682: symlink d4/d7/d22/d27/d58/d94/dbc/le0 0 2024-02-28T16:31:15.664 INFO:tasks.workunit.client.1.smithi066.stdout:0/348: creat d2/d9/d10/d50/d64/d6a/f72 x:0 0 0 2024-02-28T16:31:15.664 INFO:tasks.workunit.client.1.smithi066.stdout:6/211: link d2/da/c28 d2/d26/d32/c44 0 2024-02-28T16:31:15.666 INFO:tasks.workunit.client.1.smithi066.stdout:4/370: creat d1/d23/f7b x:0 0 0 2024-02-28T16:31:15.667 INFO:tasks.workunit.client.1.smithi066.stdout:5/242: dwrite d3/d4/dc/d2f/f32 [0,4194304] 0 2024-02-28T16:31:15.667 INFO:tasks.workunit.client.0.smithi001.stdout:4/723: symlink d3/d1c/de8/le9 0 2024-02-28T16:31:15.668 INFO:tasks.workunit.client.0.smithi001.stdout:7/661: chown d3/d17/l9e 2638969 1 2024-02-28T16:31:15.670 INFO:tasks.workunit.client.1.smithi066.stdout:1/285: link d2/d18/f21 d2/d27/d2b/f56 0 2024-02-28T16:31:15.672 INFO:tasks.workunit.client.1.smithi066.stdout:6/212: creat d2/d2a/f45 x:0 0 0 2024-02-28T16:31:15.672 INFO:tasks.workunit.client.0.smithi001.stdout:4/724: creat d3/d2d/d4c/fea x:0 0 0 2024-02-28T16:31:15.675 INFO:tasks.workunit.client.1.smithi066.stdout:3/252: mkdir db/d13/d14/d2d/d57 0 2024-02-28T16:31:15.675 INFO:tasks.workunit.client.1.smithi066.stdout:9/219: unlink da/d21/c43 0 2024-02-28T16:31:15.675 INFO:tasks.workunit.client.0.smithi001.stdout:1/802: dwrite d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f100 [0,4194304] 0 2024-02-28T16:31:15.675 INFO:tasks.workunit.client.0.smithi001.stdout:9/770: truncate d5/d27/db1/f7c 1694202 0 2024-02-28T16:31:15.675 INFO:tasks.workunit.client.0.smithi001.stdout:7/662: read d3/d1c/f4b [1575633,123309] 0 2024-02-28T16:31:15.678 INFO:tasks.workunit.client.0.smithi001.stdout:4/725: mknod d3/d2d/d4c/d69/ceb 0 2024-02-28T16:31:15.679 INFO:tasks.workunit.client.1.smithi066.stdout:7/176: dwrite dc/de/d17/d1e/f28 [0,4194304] 0 2024-02-28T16:31:15.679 INFO:tasks.workunit.client.0.smithi001.stdout:9/771: fdatasync d5/d45/d6f/dc8/fbe 0 2024-02-28T16:31:15.680 INFO:tasks.workunit.client.1.smithi066.stdout:0/349: mkdir d2/d9/d10/d50/d73 0 2024-02-28T16:31:15.680 INFO:tasks.workunit.client.1.smithi066.stdout:1/286: symlink d2/d27/d2b/l57 0 2024-02-28T16:31:15.680 INFO:tasks.workunit.client.1.smithi066.stdout:9/220: fsync da/df/f2f 0 2024-02-28T16:31:15.681 INFO:tasks.workunit.client.1.smithi066.stdout:4/371: chown d1/d18/d41/l79 604 1 2024-02-28T16:31:15.684 INFO:tasks.workunit.client.0.smithi001.stdout:7/663: creat d3/d17/d52/d7f/d99/fd5 x:0 0 0 2024-02-28T16:31:15.685 INFO:tasks.workunit.client.1.smithi066.stdout:3/253: write db/dc/d11/d1f/f2e [2124965,39794] 0 2024-02-28T16:31:15.687 INFO:tasks.workunit.client.0.smithi001.stdout:4/726: chown d3/d5/d7/d1f/d2e/dbe/d87/ld8 8709 1 2024-02-28T16:31:15.687 INFO:tasks.workunit.client.0.smithi001.stdout:0/653: dwrite d18/d22/d3d/fba [0,4194304] 0 2024-02-28T16:31:15.687 INFO:tasks.workunit.client.1.smithi066.stdout:5/243: write d3/d4/dc/dd/d2e/d18/f4a [827256,32512] 0 2024-02-28T16:31:15.688 INFO:tasks.workunit.client.1.smithi066.stdout:9/221: write da/df/f2f [2023676,40860] 0 2024-02-28T16:31:15.689 INFO:tasks.workunit.client.1.smithi066.stdout:1/287: mknod d2/d27/d2b/d43/d46/c58 0 2024-02-28T16:31:15.690 INFO:tasks.workunit.client.0.smithi001.stdout:4/727: truncate d3/d1c/d58/fdf 1009494 0 2024-02-28T16:31:15.692 INFO:tasks.workunit.client.0.smithi001.stdout:4/728: readlink d3/d2d/d83/l89 0 2024-02-28T16:31:15.692 INFO:tasks.workunit.client.1.smithi066.stdout:7/177: rename dc/de/l30 to dc/de/d17/d1e/l35 0 2024-02-28T16:31:15.692 INFO:tasks.workunit.client.1.smithi066.stdout:0/350: read d2/d9/f11 [692525,27949] 0 2024-02-28T16:31:15.693 INFO:tasks.workunit.client.0.smithi001.stdout:1/803: dread d0/d9/d1a/d72/f8a [0,4194304] 0 2024-02-28T16:31:15.694 INFO:tasks.workunit.client.0.smithi001.stdout:9/772: getdents d5/d27/db1/d58 0 2024-02-28T16:31:15.696 INFO:tasks.workunit.client.1.smithi066.stdout:7/178: truncate dc/d29/f34 301494 0 2024-02-28T16:31:15.698 INFO:tasks.workunit.client.1.smithi066.stdout:1/288: rmdir d2/d27/d2b/d37 39 2024-02-28T16:31:15.702 INFO:tasks.workunit.client.0.smithi001.stdout:0/654: rmdir d18/d22/d76/dcf 0 2024-02-28T16:31:15.702 INFO:tasks.workunit.client.0.smithi001.stdout:1/804: read d0/d9/d1a/d2b/d26/d27/d30/d4a/f70 [532741,109249] 0 2024-02-28T16:31:15.704 INFO:tasks.workunit.client.0.smithi001.stdout:9/773: symlink d5/d9a/dba/lfd 0 2024-02-28T16:31:15.706 INFO:tasks.workunit.client.0.smithi001.stdout:8/683: truncate f1 3589579 0 2024-02-28T16:31:15.708 INFO:tasks.workunit.client.1.smithi066.stdout:7/179: symlink dc/d29/l36 0 2024-02-28T16:31:15.711 INFO:tasks.workunit.client.1.smithi066.stdout:5/244: link d3/l14 d3/d4/dc/d2f/l53 0 2024-02-28T16:31:15.711 INFO:tasks.workunit.client.1.smithi066.stdout:1/289: mknod d2/d27/d2b/d43/c59 0 2024-02-28T16:31:15.712 INFO:tasks.workunit.client.0.smithi001.stdout:0/655: fdatasync d18/d22/d2f/d1c/d79/fc2 0 2024-02-28T16:31:15.713 INFO:tasks.workunit.client.1.smithi066.stdout:3/254: getdents db/d13/d1a 0 2024-02-28T16:31:15.714 INFO:tasks.workunit.client.0.smithi001.stdout:4/729: dwrite d3/d5/d7/d1f/d2e/dbe/d87/fd1 [0,4194304] 0 2024-02-28T16:31:15.716 INFO:tasks.workunit.client.1.smithi066.stdout:9/222: dwrite da/fb [4194304,4194304] 0 2024-02-28T16:31:15.717 INFO:tasks.workunit.client.0.smithi001.stdout:9/774: rename d5/d27/db1/d58/f9b to d5/d45/ffe 0 2024-02-28T16:31:15.723 INFO:tasks.workunit.client.0.smithi001.stdout:0/656: mkdir d18/da9/dc9/dd7 0 2024-02-28T16:31:15.728 INFO:tasks.workunit.client.1.smithi066.stdout:5/245: link d3/d4/dc/dd/l34 d3/d4/dc/dd/d35/l54 0 2024-02-28T16:31:15.731 INFO:tasks.workunit.client.0.smithi001.stdout:8/684: link d4/c29 d4/d32/d93/ce1 0 2024-02-28T16:31:15.734 INFO:tasks.workunit.client.1.smithi066.stdout:1/290: link d2/d18/c41 d2/c5a 0 2024-02-28T16:31:15.743 INFO:tasks.workunit.client.0.smithi001.stdout:8/685: fdatasync d4/d7/d22/d27/d4b/d77/f8d 0 2024-02-28T16:31:15.744 INFO:tasks.workunit.client.1.smithi066.stdout:3/255: dwrite db/dc/d11/f22 [0,4194304] 0 2024-02-28T16:31:15.744 INFO:tasks.workunit.client.1.smithi066.stdout:9/223: dwrite da/d3a/d15/f19 [0,4194304] 0 2024-02-28T16:31:15.745 INFO:tasks.workunit.client.0.smithi001.stdout:4/730: dwrite d3/d1c/da0/fb2 [0,4194304] 0 2024-02-28T16:31:15.749 INFO:tasks.workunit.client.1.smithi066.stdout:3/256: write db/d13/d14/d2d/f3e [1160260,129666] 0 2024-02-28T16:31:15.766 INFO:tasks.workunit.client.1.smithi066.stdout:3/257: creat db/dc/d35/f58 x:0 0 0 2024-02-28T16:31:15.767 INFO:tasks.workunit.client.1.smithi066.stdout:3/258: dread - db/d2a/f42 zero size 2024-02-28T16:31:15.768 INFO:tasks.workunit.client.1.smithi066.stdout:9/224: mkdir da/d4b 0 2024-02-28T16:31:15.770 INFO:tasks.workunit.client.1.smithi066.stdout:3/259: read db/d13/d14/d24/d2f/f4a [1130328,86921] 0 2024-02-28T16:31:15.773 INFO:tasks.workunit.client.1.smithi066.stdout:0/351: write d2/d9/d10/d50/d51/f5a [406169,124573] 0 2024-02-28T16:31:15.775 INFO:tasks.workunit.client.1.smithi066.stdout:1/291: rename d2/d18/d19/f32 to d2/d27/f5b 0 2024-02-28T16:31:15.775 INFO:tasks.workunit.client.1.smithi066.stdout:3/260: truncate db/dc/d35/f58 801848 0 2024-02-28T16:31:15.779 INFO:tasks.workunit.client.1.smithi066.stdout:7/180: fdatasync dc/de/d17/d1e/f28 0 2024-02-28T16:31:15.779 INFO:tasks.workunit.client.1.smithi066.stdout:0/352: symlink d2/d9/l74 0 2024-02-28T16:31:15.780 INFO:tasks.workunit.client.1.smithi066.stdout:7/181: read f5 [1979788,102781] 0 2024-02-28T16:31:15.781 INFO:tasks.workunit.client.1.smithi066.stdout:9/225: rename da/df/d18/f37 to da/df/d18/d2d/f4c 0 2024-02-28T16:31:15.782 INFO:tasks.workunit.client.1.smithi066.stdout:0/353: dread - d2/d9/d10/d50/f57 zero size 2024-02-28T16:31:15.783 INFO:tasks.workunit.client.1.smithi066.stdout:7/182: chown dc/de/d17/d1e/f24 288895 1 2024-02-28T16:31:15.786 INFO:tasks.workunit.client.1.smithi066.stdout:4/372: dread d1/f46 [0,4194304] 0 2024-02-28T16:31:15.786 INFO:tasks.workunit.client.1.smithi066.stdout:7/183: write fa [500638,104898] 0 2024-02-28T16:31:15.790 INFO:tasks.workunit.client.1.smithi066.stdout:0/354: stat d2/d9/d10/d50/d64/c71 0 2024-02-28T16:31:15.793 INFO:tasks.workunit.client.1.smithi066.stdout:7/184: truncate dc/de/d17/d1e/f2a 64092 0 2024-02-28T16:31:15.793 INFO:tasks.workunit.client.1.smithi066.stdout:0/355: fdatasync d2/d9/d10/d50/f57 0 2024-02-28T16:31:15.795 INFO:tasks.workunit.client.1.smithi066.stdout:4/373: mknod d1/d23/d25/d3e/c7c 0 2024-02-28T16:31:15.797 INFO:tasks.workunit.client.1.smithi066.stdout:1/292: getdents d2/d27 0 2024-02-28T16:31:15.798 INFO:tasks.workunit.client.1.smithi066.stdout:7/185: dread - dc/de/d17/d1e/f20 zero size 2024-02-28T16:31:15.802 INFO:tasks.workunit.client.1.smithi066.stdout:3/261: dwrite db/d2a/f37 [0,4194304] 0 2024-02-28T16:31:15.808 INFO:tasks.workunit.client.1.smithi066.stdout:7/186: rename dc/de/d32 to dc/de/d17/d37 0 2024-02-28T16:31:15.809 INFO:tasks.workunit.client.1.smithi066.stdout:3/262: mkdir db/d2a/d44/d59 0 2024-02-28T16:31:15.815 INFO:tasks.workunit.client.1.smithi066.stdout:9/226: dread da/f20 [0,4194304] 0 2024-02-28T16:31:15.816 INFO:tasks.workunit.client.1.smithi066.stdout:7/187: unlink dc/de/d17/c27 0 2024-02-28T16:31:15.817 INFO:tasks.workunit.client.0.smithi001.stdout:8/686: rmdir d4/d7 39 2024-02-28T16:31:15.819 INFO:tasks.workunit.client.0.smithi001.stdout:7/664: dwrite d3/d1c/f40 [0,4194304] 0 2024-02-28T16:31:15.820 INFO:tasks.workunit.client.1.smithi066.stdout:3/263: mknod db/d2a/c5a 0 2024-02-28T16:31:15.820 INFO:tasks.workunit.client.0.smithi001.stdout:9/775: dwrite d5/d45/d6f/db3/fde [4194304,4194304] 0 2024-02-28T16:31:15.821 INFO:tasks.workunit.client.1.smithi066.stdout:0/356: dwrite d2/d9/d10/f59 [0,4194304] 0 2024-02-28T16:31:15.823 INFO:tasks.workunit.client.1.smithi066.stdout:2/275: sync 2024-02-28T16:31:15.823 INFO:tasks.workunit.client.1.smithi066.stdout:8/223: sync 2024-02-28T16:31:15.823 INFO:tasks.workunit.client.1.smithi066.stdout:4/374: dwrite d1/d23/d25/f6f [0,4194304] 0 2024-02-28T16:31:15.823 INFO:tasks.workunit.client.1.smithi066.stdout:7/188: chown dc/l14 501556115 1 2024-02-28T16:31:15.824 INFO:tasks.workunit.client.1.smithi066.stdout:7/189: stat f1 0 2024-02-28T16:31:15.824 INFO:tasks.workunit.client.1.smithi066.stdout:1/293: dwrite d2/d18/d19/f3b [4194304,4194304] 0 2024-02-28T16:31:15.827 INFO:tasks.workunit.client.0.smithi001.stdout:9/776: readlink d5/d9a/la9 0 2024-02-28T16:31:15.827 INFO:tasks.workunit.client.0.smithi001.stdout:7/665: write d3/d17/f1b [2363630,31013] 0 2024-02-28T16:31:15.828 INFO:tasks.workunit.client.0.smithi001.stdout:9/777: chown d5/l13 992117 1 2024-02-28T16:31:15.828 INFO:tasks.workunit.client.1.smithi066.stdout:2/276: dread - d0/d12/d2e/d38/f4d zero size 2024-02-28T16:31:15.828 INFO:tasks.workunit.client.1.smithi066.stdout:4/375: dread - d1/d18/d4e/f71 zero size 2024-02-28T16:31:15.831 INFO:tasks.workunit.client.0.smithi001.stdout:9/778: chown d5/d27/db1/d58 100315 1 2024-02-28T16:31:15.836 INFO:tasks.workunit.client.1.smithi066.stdout:9/227: rename da/f1e to da/df/d18/d28/f4d 0 2024-02-28T16:31:15.837 INFO:tasks.workunit.client.1.smithi066.stdout:0/357: fdatasync d2/d9/d10/f39 0 2024-02-28T16:31:15.840 INFO:tasks.workunit.client.0.smithi001.stdout:7/666: creat d3/d17/d3d/d59/d6e/fd6 x:0 0 0 2024-02-28T16:31:15.840 INFO:tasks.workunit.client.0.smithi001.stdout:8/687: getdents d4/d7/d22/d27 0 2024-02-28T16:31:15.841 INFO:tasks.workunit.client.0.smithi001.stdout:9/779: mkdir d5/d27/db1/da6/db2/dff 0 2024-02-28T16:31:15.843 INFO:tasks.workunit.client.0.smithi001.stdout:7/667: dread - d3/d17/d52/d7f/d99/faa zero size 2024-02-28T16:31:15.847 INFO:tasks.workunit.client.1.smithi066.stdout:4/376: chown d1/d18/c62 979814 1 2024-02-28T16:31:15.848 INFO:tasks.workunit.client.0.smithi001.stdout:8/688: write d4/d32/d69/f72 [1218322,91485] 0 2024-02-28T16:31:15.849 INFO:tasks.workunit.client.1.smithi066.stdout:0/358: creat d2/d9/f75 x:0 0 0 2024-02-28T16:31:15.849 INFO:tasks.workunit.client.0.smithi001.stdout:9/780: symlink d5/d45/df1/l100 0 2024-02-28T16:31:15.850 INFO:tasks.workunit.client.1.smithi066.stdout:7/190: unlink dc/de/d17/d1e/l2b 0 2024-02-28T16:31:15.852 INFO:tasks.workunit.client.1.smithi066.stdout:0/359: mkdir d2/d27/d41/d76 0 2024-02-28T16:31:15.854 INFO:tasks.workunit.client.0.smithi001.stdout:7/668: rmdir d3/d17/d52/d7f/d99 39 2024-02-28T16:31:15.856 INFO:tasks.workunit.client.1.smithi066.stdout:7/191: unlink dc/f33 0 2024-02-28T16:31:15.856 INFO:tasks.workunit.client.0.smithi001.stdout:9/781: chown d5/d27/d34/c6b 45217 1 2024-02-28T16:31:15.857 INFO:tasks.workunit.client.0.smithi001.stdout:7/669: fdatasync d3/d17/d3d/db3/fb4 0 2024-02-28T16:31:15.857 INFO:tasks.workunit.client.0.smithi001.stdout:8/689: mknod d4/d32/d69/d8f/ce2 0 2024-02-28T16:31:15.859 INFO:tasks.workunit.client.1.smithi066.stdout:8/224: rename d0/db/d12/d1d/f29 to d0/f46 0 2024-02-28T16:31:15.859 INFO:tasks.workunit.client.0.smithi001.stdout:0/657: dread d18/d22/d2b/d36/d46/fb1 [0,4194304] 0 2024-02-28T16:31:15.860 INFO:tasks.workunit.client.1.smithi066.stdout:4/377: mknod d1/d18/c7d 0 2024-02-28T16:31:15.861 INFO:tasks.workunit.client.1.smithi066.stdout:8/225: readlink d0/d6/dc/l2e 0 2024-02-28T16:31:15.861 INFO:tasks.workunit.client.0.smithi001.stdout:9/782: unlink d5/d27/db1/d5e/c83 0 2024-02-28T16:31:15.862 INFO:tasks.workunit.client.1.smithi066.stdout:8/226: write d0/f43 [151429,35299] 0 2024-02-28T16:31:15.863 INFO:tasks.workunit.client.1.smithi066.stdout:4/378: chown d1/d23/d25/f6b 994 1 2024-02-28T16:31:15.863 INFO:tasks.workunit.client.0.smithi001.stdout:8/690: symlink d4/d32/d69/le3 0 2024-02-28T16:31:15.863 INFO:tasks.workunit.client.0.smithi001.stdout:7/670: stat d3/d17/d52/d7f/d99/dc1/d49/d63/l74 0 2024-02-28T16:31:15.865 INFO:tasks.workunit.client.0.smithi001.stdout:0/658: truncate d18/d22/d2f/d1c/f21 9382770 0 2024-02-28T16:31:15.866 INFO:tasks.workunit.client.1.smithi066.stdout:0/360: unlink d2/d9/d10/f1d 0 2024-02-28T16:31:15.870 INFO:tasks.workunit.client.1.smithi066.stdout:7/192: mknod dc/c38 0 2024-02-28T16:31:15.870 INFO:tasks.workunit.client.1.smithi066.stdout:0/361: readlink d2/d27/l37 0 2024-02-28T16:31:15.871 INFO:tasks.workunit.client.0.smithi001.stdout:7/671: link d3/d17/d52/fbc d3/d17/d52/d7f/d99/dc1/fd7 0 2024-02-28T16:31:15.873 INFO:tasks.workunit.client.0.smithi001.stdout:0/659: rename d18/d31/d4f/ccb to d18/d22/d2f/cd8 0 2024-02-28T16:31:15.874 INFO:tasks.workunit.client.1.smithi066.stdout:1/294: rename d2/le to d2/d27/d2b/d37/d54/l5c 0 2024-02-28T16:31:15.876 INFO:tasks.workunit.client.1.smithi066.stdout:3/264: dwrite db/d13/f23 [0,4194304] 0 2024-02-28T16:31:15.877 INFO:tasks.workunit.client.0.smithi001.stdout:0/660: rmdir d18/d22/d57/d7e 39 2024-02-28T16:31:15.880 INFO:tasks.workunit.client.0.smithi001.stdout:7/672: creat d3/d17/d52/d7f/d99/dc1/d49/fd8 x:0 0 0 2024-02-28T16:31:15.881 INFO:tasks.workunit.client.1.smithi066.stdout:0/362: unlink d2/l12 0 2024-02-28T16:31:15.881 INFO:tasks.workunit.client.1.smithi066.stdout:3/265: write db/dc/d35/f58 [1263932,86139] 0 2024-02-28T16:31:15.882 INFO:tasks.workunit.client.0.smithi001.stdout:9/783: dread d5/d27/db1/f7c [0,4194304] 0 2024-02-28T16:31:15.883 INFO:tasks.workunit.client.1.smithi066.stdout:7/193: rmdir dc/d29 39 2024-02-28T16:31:15.883 INFO:tasks.workunit.client.0.smithi001.stdout:7/673: write d3/d1c/f33 [8109263,58593] 0 2024-02-28T16:31:15.884 INFO:tasks.workunit.client.0.smithi001.stdout:0/661: mknod d18/d22/d2b/d36/cd9 0 2024-02-28T16:31:15.885 INFO:tasks.workunit.client.1.smithi066.stdout:4/379: getdents d1/d18/d4e 0 2024-02-28T16:31:15.885 INFO:tasks.workunit.client.1.smithi066.stdout:3/266: dread - db/dc/f55 zero size 2024-02-28T16:31:15.886 INFO:tasks.workunit.client.0.smithi001.stdout:9/784: write d5/d27/db1/f5f [434716,30679] 0 2024-02-28T16:31:15.887 INFO:tasks.workunit.client.1.smithi066.stdout:4/380: fsync d1/f20 0 2024-02-28T16:31:15.887 INFO:tasks.workunit.client.1.smithi066.stdout:7/194: stat dc/de/d17/d1e/f20 0 2024-02-28T16:31:15.887 INFO:tasks.workunit.client.1.smithi066.stdout:2/277: dread d0/d12/d1a/d23/f28 [0,4194304] 0 2024-02-28T16:31:15.889 INFO:tasks.workunit.client.0.smithi001.stdout:0/662: creat d18/d5a/fda x:0 0 0 2024-02-28T16:31:15.889 INFO:tasks.workunit.client.1.smithi066.stdout:1/295: unlink d2/d27/d2b/f2c 0 2024-02-28T16:31:15.892 INFO:tasks.workunit.client.1.smithi066.stdout:1/296: chown d2/d18/l40 213 1 2024-02-28T16:31:15.893 INFO:tasks.workunit.client.0.smithi001.stdout:9/785: getdents d5/d45/d6f/dc8 0 2024-02-28T16:31:15.899 INFO:tasks.workunit.client.1.smithi066.stdout:0/363: mkdir d2/d9/d10/d50/d64/d6a/d77 0 2024-02-28T16:31:15.907 INFO:tasks.workunit.client.0.smithi001.stdout:7/674: dwrite d3/ff [0,4194304] 0 2024-02-28T16:31:15.907 INFO:tasks.workunit.client.1.smithi066.stdout:3/267: symlink db/d13/d1a/l5b 0 2024-02-28T16:31:15.907 INFO:tasks.workunit.client.1.smithi066.stdout:0/364: write d2/d9/d10/f4e [341792,66816] 0 2024-02-28T16:31:15.909 INFO:tasks.workunit.client.1.smithi066.stdout:0/365: chown d2/d9/d10/d50/d64 0 1 2024-02-28T16:31:15.910 INFO:tasks.workunit.client.0.smithi001.stdout:0/663: dwrite d18/d22/d3d/f50 [0,4194304] 0 2024-02-28T16:31:15.911 INFO:tasks.workunit.client.0.smithi001.stdout:9/786: dwrite d5/d27/db1/f93 [0,4194304] 0 2024-02-28T16:31:15.911 INFO:tasks.workunit.client.1.smithi066.stdout:4/381: mkdir d1/d18/d7e 0 2024-02-28T16:31:15.912 INFO:tasks.workunit.client.0.smithi001.stdout:9/787: chown d5/c1b 1 1 2024-02-28T16:31:15.913 INFO:tasks.workunit.client.0.smithi001.stdout:7/675: symlink d3/d17/d61/ld9 0 2024-02-28T16:31:15.918 INFO:tasks.workunit.client.0.smithi001.stdout:0/664: chown d18/d22/d2b/d95/fb0 2 1 2024-02-28T16:31:15.918 INFO:tasks.workunit.client.0.smithi001.stdout:0/665: rmdir d18/d22/d2b/d36 39 2024-02-28T16:31:15.920 INFO:tasks.workunit.client.0.smithi001.stdout:7/676: symlink d3/d1c/lda 0 2024-02-28T16:31:15.925 INFO:tasks.workunit.client.0.smithi001.stdout:9/788: read d5/d27/db1/d33/fdb [5511772,8229] 0 2024-02-28T16:31:15.925 INFO:tasks.workunit.client.0.smithi001.stdout:0/666: mknod d18/d22/d76/d7a/cdb 0 2024-02-28T16:31:15.927 INFO:tasks.workunit.client.0.smithi001.stdout:0/667: readlink d18/d31/l74 0 2024-02-28T16:31:15.927 INFO:tasks.workunit.client.0.smithi001.stdout:9/789: chown d5/d4d/f70 732854 1 2024-02-28T16:31:15.929 INFO:tasks.workunit.client.0.smithi001.stdout:9/790: chown d5/d27/db1/d65/l69 99372931 1 2024-02-28T16:31:15.931 INFO:tasks.workunit.client.0.smithi001.stdout:0/668: symlink d18/d22/d2b/d36/ldc 0 2024-02-28T16:31:15.933 INFO:tasks.workunit.client.0.smithi001.stdout:0/669: read d18/d22/d2b/d36/d48/f55 [3843884,47502] 0 2024-02-28T16:31:15.938 INFO:tasks.workunit.client.0.smithi001.stdout:0/670: getdents d18/d31/d4f 0 2024-02-28T16:31:15.940 INFO:tasks.workunit.client.0.smithi001.stdout:0/671: write d18/d31/d4f/fcd [97531,26818] 0 2024-02-28T16:31:15.977 INFO:tasks.workunit.client.0.smithi001.stdout:0/672: dread d18/f33 [0,4194304] 0 2024-02-28T16:31:15.978 INFO:tasks.workunit.client.0.smithi001.stdout:7/677: write d3/d17/d52/d7f/d99/dc1/fd7 [4883517,91733] 0 2024-02-28T16:31:15.980 INFO:tasks.workunit.client.1.smithi066.stdout:5/246: sync 2024-02-28T16:31:15.980 INFO:tasks.workunit.client.1.smithi066.stdout:6/213: sync 2024-02-28T16:31:15.981 INFO:tasks.workunit.client.0.smithi001.stdout:0/673: symlink d18/ldd 0 2024-02-28T16:31:15.984 INFO:tasks.workunit.client.0.smithi001.stdout:0/674: creat d18/d22/d2b/d5f/fde x:0 0 0 2024-02-28T16:31:15.986 INFO:tasks.workunit.client.0.smithi001.stdout:0/675: write d18/d22/d3d/f50 [3300513,31177] 0 2024-02-28T16:31:15.986 INFO:tasks.workunit.client.0.smithi001.stdout:9/791: dwrite d5/d27/d42/f7e [0,4194304] 0 2024-02-28T16:31:15.990 INFO:tasks.workunit.client.0.smithi001.stdout:9/792: write d5/d45/d6f/dc8/fe2 [4628318,38631] 0 2024-02-28T16:31:15.993 INFO:tasks.workunit.client.0.smithi001.stdout:0/676: chown d18/d22/d2f/d1c/d90/lc8 15711898 1 2024-02-28T16:31:15.997 INFO:tasks.workunit.client.0.smithi001.stdout:0/677: symlink d18/d22/d2f/d1c/ldf 0 2024-02-28T16:31:15.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:15 smithi066 ceph-mon[36236]: pgmap v51: 65 pgs: 65 active+clean; 1.9 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 62 MiB/s rd, 224 MiB/s wr, 437 op/s 2024-02-28T16:31:15.998 INFO:tasks.workunit.client.0.smithi001.stdout:7/678: dwrite d3/d17/d3d/f70 [0,4194304] 0 2024-02-28T16:31:15.999 INFO:tasks.workunit.client.0.smithi001.stdout:9/793: rename d5/d27/d42/d48/cb4 to d5/d27/db1/da6/db2/dff/c101 0 2024-02-28T16:31:16.000 INFO:tasks.workunit.client.1.smithi066.stdout:2/278: getdents d0/d12/d2e/d3b 0 2024-02-28T16:31:16.001 INFO:tasks.workunit.client.0.smithi001.stdout:9/794: chown d5/d27/d42 736 1 2024-02-28T16:31:16.003 INFO:tasks.workunit.client.0.smithi001.stdout:7/679: mknod d3/d17/d3d/d59/cdb 0 2024-02-28T16:31:16.003 INFO:tasks.workunit.client.1.smithi066.stdout:1/297: truncate d2/d18/d19/f33 2092311 0 2024-02-28T16:31:16.005 INFO:tasks.workunit.client.1.smithi066.stdout:2/279: write d0/d12/d1a/d39/d40/f55 [266324,103500] 0 2024-02-28T16:31:16.005 INFO:tasks.workunit.client.1.smithi066.stdout:1/298: read - d2/d18/f3a zero size 2024-02-28T16:31:16.007 INFO:tasks.workunit.client.1.smithi066.stdout:4/382: readlink d1/l31 0 2024-02-28T16:31:16.007 INFO:tasks.workunit.client.1.smithi066.stdout:6/214: mkdir d2/d26/d32/d46 0 2024-02-28T16:31:16.008 INFO:tasks.workunit.client.0.smithi001.stdout:9/795: mknod d5/d45/d6f/db3/c102 0 2024-02-28T16:31:16.008 INFO:tasks.workunit.client.0.smithi001.stdout:7/680: symlink d3/d17/d52/d55/dca/ldc 0 2024-02-28T16:31:16.008 INFO:tasks.workunit.client.1.smithi066.stdout:5/247: mkdir d3/d4/dc/d2f/d55 0 2024-02-28T16:31:16.010 INFO:tasks.workunit.client.0.smithi001.stdout:9/796: chown d5/d27/d34/c6b 1995671 1 2024-02-28T16:31:16.015 INFO:tasks.workunit.client.1.smithi066.stdout:2/280: write d0/f19 [526403,128768] 0 2024-02-28T16:31:16.017 INFO:tasks.workunit.client.0.smithi001.stdout:0/678: dwrite d18/d22/d2b/d5f/f85 [0,4194304] 0 2024-02-28T16:31:16.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:15 smithi001 ceph-mon[28991]: pgmap v51: 65 pgs: 65 active+clean; 1.9 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 62 MiB/s rd, 224 MiB/s wr, 437 op/s 2024-02-28T16:31:16.019 INFO:tasks.workunit.client.0.smithi001.stdout:7/681: rename d3/d17/c20 to d3/d17/d52/d7f/cdd 0 2024-02-28T16:31:16.020 INFO:tasks.workunit.client.1.smithi066.stdout:6/215: creat d2/d26/f47 x:0 0 0 2024-02-28T16:31:16.021 INFO:tasks.workunit.client.0.smithi001.stdout:0/679: creat d18/d22/d76/d93/fe0 x:0 0 0 2024-02-28T16:31:16.021 INFO:tasks.workunit.client.1.smithi066.stdout:6/216: stat d2/da/d19 0 2024-02-28T16:31:16.026 INFO:tasks.workunit.client.1.smithi066.stdout:7/195: symlink dc/d29/l39 0 2024-02-28T16:31:16.026 INFO:tasks.workunit.client.1.smithi066.stdout:3/268: link db/d13/d14/c17 db/dc/d11/c5c 0 2024-02-28T16:31:16.027 INFO:tasks.workunit.client.1.smithi066.stdout:3/269: chown db 147 1 2024-02-28T16:31:16.028 INFO:tasks.workunit.client.0.smithi001.stdout:7/682: rename d3/d17/c43 to d3/d17/d52/d7f/d99/dc1/d49/d94/cde 0 2024-02-28T16:31:16.029 INFO:tasks.workunit.client.1.smithi066.stdout:1/299: link l1 d2/d27/l5d 0 2024-02-28T16:31:16.030 INFO:tasks.workunit.client.0.smithi001.stdout:7/683: write d3/d17/fa5 [437869,71117] 0 2024-02-28T16:31:16.032 INFO:tasks.workunit.client.1.smithi066.stdout:0/366: creat d2/d9/d10/d50/d51/f78 x:0 0 0 2024-02-28T16:31:16.033 INFO:tasks.workunit.client.0.smithi001.stdout:7/684: chown d3/d17/d3d/db3/fb4 0 1 2024-02-28T16:31:16.035 INFO:tasks.workunit.client.1.smithi066.stdout:6/217: creat d2/d2a/d3d/f48 x:0 0 0 2024-02-28T16:31:16.038 INFO:tasks.workunit.client.0.smithi001.stdout:7/685: mknod d3/d17/d52/d55/dca/cdf 0 2024-02-28T16:31:16.038 INFO:tasks.workunit.client.0.smithi001.stdout:9/797: dwrite d5/f19 [0,4194304] 0 2024-02-28T16:31:16.038 INFO:tasks.workunit.client.1.smithi066.stdout:3/270: readlink db/d13/d14/l3b 0 2024-02-28T16:31:16.041 INFO:tasks.workunit.client.0.smithi001.stdout:9/798: creat d5/d27/d42/d48/f103 x:0 0 0 2024-02-28T16:31:16.042 INFO:tasks.workunit.client.0.smithi001.stdout:7/686: rename d3/d17/d52/d7f/d99/dc1 to d3/d17/d52/d7f/d99/dc1/d49/d8d/db9/de0 22 2024-02-28T16:31:16.043 INFO:tasks.workunit.client.1.smithi066.stdout:2/281: link d0/d12/c18 d0/d12/d2e/d3b/c58 0 2024-02-28T16:31:16.043 INFO:tasks.workunit.client.1.smithi066.stdout:7/196: write dc/de/d17/d1e/f23 [1144338,26566] 0 2024-02-28T16:31:16.044 INFO:tasks.workunit.client.1.smithi066.stdout:2/282: readlink d0/d12/d1a/d23/d29/l49 0 2024-02-28T16:31:16.044 INFO:tasks.workunit.client.0.smithi001.stdout:0/680: dread d18/d22/d2f/f24 [4194304,4194304] 0 2024-02-28T16:31:16.045 INFO:tasks.workunit.client.0.smithi001.stdout:9/799: creat d5/d27/db1/d5e/dd3/f104 x:0 0 0 2024-02-28T16:31:16.046 INFO:tasks.workunit.client.1.smithi066.stdout:4/383: dwrite d1/f10 [0,4194304] 0 2024-02-28T16:31:16.046 INFO:tasks.workunit.client.0.smithi001.stdout:7/687: creat d3/d17/d3d/d42/fe1 x:0 0 0 2024-02-28T16:31:16.049 INFO:tasks.workunit.client.0.smithi001.stdout:9/800: symlink d5/l105 0 2024-02-28T16:31:16.049 INFO:tasks.workunit.client.1.smithi066.stdout:7/197: fsync dc/d29/f2e 0 2024-02-28T16:31:16.051 INFO:tasks.workunit.client.0.smithi001.stdout:9/801: fdatasync d5/d27/db1/d33/f5d 0 2024-02-28T16:31:16.051 INFO:tasks.workunit.client.1.smithi066.stdout:5/248: dwrite d3/d1b/f36 [0,4194304] 0 2024-02-28T16:31:16.052 INFO:tasks.workunit.client.0.smithi001.stdout:0/681: getdents d18/d5a 0 2024-02-28T16:31:16.052 INFO:tasks.workunit.client.1.smithi066.stdout:0/367: rmdir d2/d9/d10/d50/d52/d61 0 2024-02-28T16:31:16.057 INFO:tasks.workunit.client.1.smithi066.stdout:2/283: fdatasync d0/d12/d1a/f4c 0 2024-02-28T16:31:16.060 INFO:tasks.workunit.client.0.smithi001.stdout:9/802: read d5/d27/db1/da6/db2/fb6 [310926,48785] 0 2024-02-28T16:31:16.060 INFO:tasks.workunit.client.1.smithi066.stdout:0/368: read d2/d9/f40 [1682546,23285] 0 2024-02-28T16:31:16.062 INFO:tasks.workunit.client.1.smithi066.stdout:2/284: chown d0/d12/d1a/d39/d40/c4b 4096076 1 2024-02-28T16:31:16.063 INFO:tasks.workunit.client.1.smithi066.stdout:0/369: dread - d2/d9/d10/d50/d64/d6a/f72 zero size 2024-02-28T16:31:16.064 INFO:tasks.workunit.client.1.smithi066.stdout:5/249: read d3/d4/dc/f1d [1427503,40591] 0 2024-02-28T16:31:16.066 INFO:tasks.workunit.client.1.smithi066.stdout:5/250: read - d3/d4/dc/f19 zero size 2024-02-28T16:31:16.067 INFO:tasks.workunit.client.1.smithi066.stdout:2/285: write d0/f31 [7377036,32942] 0 2024-02-28T16:31:16.068 INFO:tasks.workunit.client.1.smithi066.stdout:7/198: unlink dc/de/l12 0 2024-02-28T16:31:16.070 INFO:tasks.workunit.client.0.smithi001.stdout:7/688: getdents d3/d17/d3d/d42 0 2024-02-28T16:31:16.071 INFO:tasks.workunit.client.1.smithi066.stdout:0/370: rename d2/d9/d10/f59 to d2/d9/d10/d23/d6d/f79 0 2024-02-28T16:31:16.073 INFO:tasks.workunit.client.0.smithi001.stdout:7/689: fdatasync d3/d17/d3d/fcc 0 2024-02-28T16:31:16.074 INFO:tasks.workunit.client.0.smithi001.stdout:9/803: link d5/d27/d34/f8d d5/d45/d6f/dc8/f106 0 2024-02-28T16:31:16.075 INFO:tasks.workunit.client.1.smithi066.stdout:2/286: unlink d0/f30 0 2024-02-28T16:31:16.078 INFO:tasks.workunit.client.0.smithi001.stdout:9/804: mkdir d5/d27/db1/d58/d107 0 2024-02-28T16:31:16.078 INFO:tasks.workunit.client.0.smithi001.stdout:7/690: mkdir d3/d17/d3d/d59/d6e/de2 0 2024-02-28T16:31:16.078 INFO:tasks.workunit.client.1.smithi066.stdout:4/384: fdatasync d1/f10 0 2024-02-28T16:31:16.080 INFO:tasks.workunit.client.0.smithi001.stdout:7/691: chown d3/d17/d52/d7f/c91 114016 1 2024-02-28T16:31:16.081 INFO:tasks.workunit.client.0.smithi001.stdout:9/805: creat d5/d27/db1/d33/d8a/da8/ddf/f108 x:0 0 0 2024-02-28T16:31:16.082 INFO:tasks.workunit.client.0.smithi001.stdout:7/692: read - d3/d17/d52/d55/dca/fce zero size 2024-02-28T16:31:16.084 INFO:tasks.workunit.client.0.smithi001.stdout:7/693: chown d3/d17/c2e 5 1 2024-02-28T16:31:16.085 INFO:tasks.workunit.client.1.smithi066.stdout:0/371: dread - d2/d9/d10/d50/d51/f78 zero size 2024-02-28T16:31:16.085 INFO:tasks.workunit.client.1.smithi066.stdout:7/199: dread fa [0,4194304] 0 2024-02-28T16:31:16.086 INFO:tasks.workunit.client.0.smithi001.stdout:7/694: stat d3/d17/d52/d7f/d99/dc1/d49/l58 0 2024-02-28T16:31:16.087 INFO:tasks.workunit.client.1.smithi066.stdout:2/287: creat d0/d12/d2e/f59 x:0 0 0 2024-02-28T16:31:16.088 INFO:tasks.workunit.client.1.smithi066.stdout:1/300: write d2/d18/d19/f33 [2814110,21064] 0 2024-02-28T16:31:16.090 INFO:tasks.workunit.client.1.smithi066.stdout:2/288: read d0/f37 [470745,125455] 0 2024-02-28T16:31:16.095 INFO:tasks.workunit.client.0.smithi001.stdout:9/806: dwrite d5/d45/d6f/f7f [0,4194304] 0 2024-02-28T16:31:16.096 INFO:tasks.workunit.client.1.smithi066.stdout:2/289: chown d0/d12/d1a/d23/d29/d21/l50 95900904 1 2024-02-28T16:31:16.100 INFO:tasks.workunit.client.0.smithi001.stdout:7/695: dwrite d3/d17/d3d/d59/fb1 [0,4194304] 0 2024-02-28T16:31:16.101 INFO:tasks.workunit.client.0.smithi001.stdout:7/696: readlink d3/l90 0 2024-02-28T16:31:16.102 INFO:tasks.workunit.client.1.smithi066.stdout:0/372: readlink d2/d9/d33/d3f/l4b 0 2024-02-28T16:31:16.114 INFO:tasks.workunit.client.1.smithi066.stdout:7/200: dwrite dc/de/d17/d1e/f23 [0,4194304] 0 2024-02-28T16:31:16.114 INFO:tasks.workunit.client.1.smithi066.stdout:3/271: dread db/dc/d11/d1f/f2e [0,4194304] 0 2024-02-28T16:31:16.116 INFO:tasks.workunit.client.1.smithi066.stdout:5/251: dwrite d3/d4/dc/f3f [0,4194304] 0 2024-02-28T16:31:16.118 INFO:tasks.workunit.client.1.smithi066.stdout:2/290: link d0/d12/d1a/d23/l57 d0/d12/d1a/d39/d40/l5a 0 2024-02-28T16:31:16.120 INFO:tasks.workunit.client.1.smithi066.stdout:4/385: dwrite d1/f12 [0,4194304] 0 2024-02-28T16:31:16.121 INFO:tasks.workunit.client.1.smithi066.stdout:5/252: chown d3/d4/dc/l23 33637 1 2024-02-28T16:31:16.125 INFO:tasks.workunit.client.1.smithi066.stdout:4/386: chown d1/f16 1333306 1 2024-02-28T16:31:16.128 INFO:tasks.workunit.client.1.smithi066.stdout:2/291: unlink d0/d12/d1a/d23/d29/d21/d2f/l35 0 2024-02-28T16:31:16.128 INFO:tasks.workunit.client.1.smithi066.stdout:4/387: stat d1/f5 0 2024-02-28T16:31:16.129 INFO:tasks.workunit.client.1.smithi066.stdout:3/272: truncate db/d13/d14/f15 654335 0 2024-02-28T16:31:16.135 INFO:tasks.workunit.client.1.smithi066.stdout:5/253: mkdir d3/d4/dc/d56 0 2024-02-28T16:31:16.136 INFO:tasks.workunit.client.1.smithi066.stdout:3/273: write f5 [918517,78681] 0 2024-02-28T16:31:16.136 INFO:tasks.workunit.client.1.smithi066.stdout:2/292: chown d0/d12/d2e/d3b/c58 4068 1 2024-02-28T16:31:16.139 INFO:tasks.workunit.client.1.smithi066.stdout:2/293: write d0/f1f [3004098,128293] 0 2024-02-28T16:31:16.143 INFO:tasks.workunit.client.1.smithi066.stdout:7/201: dread dc/de/d17/d1e/f23 [0,4194304] 0 2024-02-28T16:31:16.145 INFO:tasks.workunit.client.1.smithi066.stdout:2/294: write d0/f31 [2117133,38391] 0 2024-02-28T16:31:16.145 INFO:tasks.workunit.client.1.smithi066.stdout:5/254: mkdir d3/d4/dc/d42/d57 0 2024-02-28T16:31:16.149 INFO:tasks.workunit.client.1.smithi066.stdout:7/202: rename cb to dc/d29/c3a 0 2024-02-28T16:31:16.150 INFO:tasks.workunit.client.1.smithi066.stdout:4/388: dread d1/d23/f36 [0,4194304] 0 2024-02-28T16:31:16.151 INFO:tasks.workunit.client.1.smithi066.stdout:9/228: sync 2024-02-28T16:31:16.153 INFO:tasks.workunit.client.1.smithi066.stdout:5/255: truncate d3/d1b/f2d 1471287 0 2024-02-28T16:31:16.154 INFO:tasks.workunit.client.1.smithi066.stdout:7/203: symlink dc/l3b 0 2024-02-28T16:31:16.162 INFO:tasks.workunit.client.1.smithi066.stdout:3/274: dread f5 [0,4194304] 0 2024-02-28T16:31:16.164 INFO:tasks.workunit.client.0.smithi001.stdout:9/807: dwrite d5/d45/d6f/dc8/f106 [0,4194304] 0 2024-02-28T16:31:16.165 INFO:tasks.workunit.client.1.smithi066.stdout:7/204: creat dc/de/d17/d1e/f3c x:0 0 0 2024-02-28T16:31:16.165 INFO:tasks.workunit.client.1.smithi066.stdout:9/229: link da/d22/l24 da/d22/d46/l4e 0 2024-02-28T16:31:16.170 INFO:tasks.workunit.client.1.smithi066.stdout:7/205: mknod dc/de/d17/d1e/c3d 0 2024-02-28T16:31:16.172 INFO:tasks.workunit.client.0.smithi001.stdout:9/808: rmdir d5 39 2024-02-28T16:31:16.173 INFO:tasks.workunit.client.1.smithi066.stdout:3/275: truncate db/d13/d14/d24/f34 159229 0 2024-02-28T16:31:16.177 INFO:tasks.workunit.client.1.smithi066.stdout:4/389: dwrite d1/f20 [0,4194304] 0 2024-02-28T16:31:16.177 INFO:tasks.workunit.client.1.smithi066.stdout:7/206: write f5 [5070766,119692] 0 2024-02-28T16:31:16.178 INFO:tasks.workunit.client.0.smithi001.stdout:9/809: rename d5/d45 to d5/d27/db1/d5e/dd3/d109 0 2024-02-28T16:31:16.181 INFO:tasks.workunit.client.0.smithi001.stdout:5/646: sync 2024-02-28T16:31:16.182 INFO:tasks.workunit.client.0.smithi001.stdout:3/692: sync 2024-02-28T16:31:16.184 INFO:tasks.workunit.client.1.smithi066.stdout:4/390: rmdir d1/d18/d41 39 2024-02-28T16:31:16.184 INFO:tasks.workunit.client.1.smithi066.stdout:7/207: creat dc/de/d17/d1e/f3e x:0 0 0 2024-02-28T16:31:16.185 INFO:tasks.workunit.client.0.smithi001.stdout:9/810: fsync d5/d27/db1/d5e/dd3/d109/fd2 0 2024-02-28T16:31:16.185 INFO:tasks.workunit.client.0.smithi001.stdout:3/693: chown d4/d1e/f27 175 1 2024-02-28T16:31:16.188 INFO:tasks.workunit.client.0.smithi001.stdout:5/647: symlink dc/d1a/d52/ld4 0 2024-02-28T16:31:16.191 INFO:tasks.workunit.client.0.smithi001.stdout:9/811: creat d5/d27/db1/d58/d107/f10a x:0 0 0 2024-02-28T16:31:16.195 INFO:tasks.workunit.client.0.smithi001.stdout:5/648: getdents dc/df 0 2024-02-28T16:31:16.196 INFO:tasks.workunit.client.1.smithi066.stdout:7/208: creat dc/de/f3f x:0 0 0 2024-02-28T16:31:16.197 INFO:tasks.workunit.client.0.smithi001.stdout:5/649: symlink dc/d1a/d52/ld5 0 2024-02-28T16:31:16.199 INFO:tasks.workunit.client.0.smithi001.stdout:3/694: write d4/d6/f40 [1205301,42876] 0 2024-02-28T16:31:16.203 INFO:tasks.workunit.client.0.smithi001.stdout:5/650: unlink dc/d12/d56/d96/db8/fc3 0 2024-02-28T16:31:16.205 INFO:tasks.workunit.client.1.smithi066.stdout:5/256: truncate d3/d4/dc/dd/d2e/d18/f2c 709007 0 2024-02-28T16:31:16.205 INFO:tasks.workunit.client.1.smithi066.stdout:4/391: link d1/d18/c19 d1/c7f 0 2024-02-28T16:31:16.206 INFO:tasks.workunit.client.0.smithi001.stdout:5/651: mkdir dc/d12/d41/dd6 0 2024-02-28T16:31:16.207 INFO:tasks.workunit.client.1.smithi066.stdout:9/230: fsync da/df/d18/d28/f4d 0 2024-02-28T16:31:16.208 INFO:tasks.workunit.client.1.smithi066.stdout:4/392: write d1/f2a [2760526,108981] 0 2024-02-28T16:31:16.209 INFO:tasks.workunit.client.0.smithi001.stdout:5/652: mkdir dc/d12/d56/d96/db8/dd7 0 2024-02-28T16:31:16.212 INFO:tasks.workunit.client.1.smithi066.stdout:7/209: getdents dc 0 2024-02-28T16:31:16.214 INFO:tasks.workunit.client.1.smithi066.stdout:5/257: dread d3/d4/dc/dd/d2e/f38 [0,4194304] 0 2024-02-28T16:31:16.216 INFO:tasks.workunit.client.0.smithi001.stdout:3/695: dwrite d4/d1e/d7c/da8/faa [0,4194304] 0 2024-02-28T16:31:16.216 INFO:tasks.workunit.client.1.smithi066.stdout:9/231: creat da/d3a/d15/f4f x:0 0 0 2024-02-28T16:31:16.217 INFO:tasks.workunit.client.1.smithi066.stdout:5/258: fdatasync d3/d4/f12 0 2024-02-28T16:31:16.220 INFO:tasks.workunit.client.1.smithi066.stdout:9/232: fsync da/d3a/d15/f19 0 2024-02-28T16:31:16.221 INFO:tasks.workunit.client.1.smithi066.stdout:4/393: chown d1/d23/d25/c26 8 1 2024-02-28T16:31:16.224 INFO:tasks.workunit.client.1.smithi066.stdout:4/394: read d1/f46 [506612,117296] 0 2024-02-28T16:31:16.226 INFO:tasks.workunit.client.1.smithi066.stdout:9/233: stat da/d22/d46 0 2024-02-28T16:31:16.227 INFO:tasks.workunit.client.1.smithi066.stdout:5/259: symlink d3/d4/dc/dd/d2e/l58 0 2024-02-28T16:31:16.228 INFO:tasks.workunit.client.1.smithi066.stdout:7/210: truncate dc/d29/f2e 1528271 0 2024-02-28T16:31:16.228 INFO:tasks.workunit.client.1.smithi066.stdout:4/395: chown d1/d23/c52 1 1 2024-02-28T16:31:16.230 INFO:tasks.workunit.client.1.smithi066.stdout:9/234: write da/df/f2f [8046909,1855] 0 2024-02-28T16:31:16.230 INFO:tasks.workunit.client.0.smithi001.stdout:5/653: dwrite dc/d12/d56/fad [0,4194304] 0 2024-02-28T16:31:16.231 INFO:tasks.workunit.client.1.smithi066.stdout:4/396: fsync d1/f16 0 2024-02-28T16:31:16.232 INFO:tasks.workunit.client.0.smithi001.stdout:5/654: fdatasync dc/fa2 0 2024-02-28T16:31:16.234 INFO:tasks.workunit.client.0.smithi001.stdout:3/696: dwrite d4/d6/f1c [4194304,4194304] 0 2024-02-28T16:31:16.236 INFO:tasks.workunit.client.1.smithi066.stdout:5/260: symlink d3/d4/dc/d2f/l59 0 2024-02-28T16:31:16.237 INFO:tasks.workunit.client.1.smithi066.stdout:7/211: read dc/de/d17/d1e/f23 [2453404,121975] 0 2024-02-28T16:31:16.239 INFO:tasks.workunit.client.1.smithi066.stdout:9/235: creat da/df/d18/d2d/f50 x:0 0 0 2024-02-28T16:31:16.243 INFO:tasks.workunit.client.1.smithi066.stdout:5/261: write d3/d4/dc/f3f [3603867,27300] 0 2024-02-28T16:31:16.244 INFO:tasks.workunit.client.1.smithi066.stdout:3/276: dwrite db/d13/d14/d24/f34 [0,4194304] 0 2024-02-28T16:31:16.247 INFO:tasks.workunit.client.1.smithi066.stdout:3/277: fsync db/d2a/d44/f4f 0 2024-02-28T16:31:16.247 INFO:tasks.workunit.client.0.smithi001.stdout:9/812: dread d5/d27/d34/f90 [0,4194304] 0 2024-02-28T16:31:16.248 INFO:tasks.workunit.client.0.smithi001.stdout:3/697: dread d4/d18/f49 [0,4194304] 0 2024-02-28T16:31:16.250 INFO:tasks.workunit.client.1.smithi066.stdout:9/236: creat da/d22/f51 x:0 0 0 2024-02-28T16:31:16.250 INFO:tasks.workunit.client.1.smithi066.stdout:7/212: mknod dc/d29/c40 0 2024-02-28T16:31:16.253 INFO:tasks.workunit.client.0.smithi001.stdout:3/698: mknod d4/d1e/d7c/d9a/ce1 0 2024-02-28T16:31:16.254 INFO:tasks.workunit.client.1.smithi066.stdout:9/237: truncate da/df/d18/d2d/f44 307472 0 2024-02-28T16:31:16.255 INFO:tasks.workunit.client.0.smithi001.stdout:9/813: rmdir d5/d27/db1/d5e/df9 0 2024-02-28T16:31:16.257 INFO:tasks.workunit.client.1.smithi066.stdout:3/278: mkdir db/d2a/d44/d48/d5d 0 2024-02-28T16:31:16.258 INFO:tasks.workunit.client.0.smithi001.stdout:3/699: unlink d4/d97/ca7 0 2024-02-28T16:31:16.258 INFO:tasks.workunit.client.1.smithi066.stdout:7/213: creat dc/d29/f41 x:0 0 0 2024-02-28T16:31:16.261 INFO:tasks.workunit.client.0.smithi001.stdout:3/700: chown d4/d3c/f3e 390612 1 2024-02-28T16:31:16.261 INFO:tasks.workunit.client.1.smithi066.stdout:9/238: chown da/d22/l24 6579970 1 2024-02-28T16:31:16.262 INFO:tasks.workunit.client.1.smithi066.stdout:7/214: chown dc/de/d17/d1e/f2d 45608 1 2024-02-28T16:31:16.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/215: fdatasync dc/d29/f34 0 2024-02-28T16:31:16.264 INFO:tasks.workunit.client.0.smithi001.stdout:5/655: dread dc/f27 [0,4194304] 0 2024-02-28T16:31:16.264 INFO:tasks.workunit.client.0.smithi001.stdout:9/814: getdents d5/d9a/dba 0 2024-02-28T16:31:16.265 INFO:tasks.workunit.client.1.smithi066.stdout:3/279: rename db/f3a to db/dc/d11/d1f/f5e 0 2024-02-28T16:31:16.265 INFO:tasks.workunit.client.0.smithi001.stdout:3/701: creat d4/d6/d7/d78/d81/dd6/fe2 x:0 0 0 2024-02-28T16:31:16.271 INFO:tasks.workunit.client.1.smithi066.stdout:9/239: getdents da/d22 0 2024-02-28T16:31:16.273 INFO:tasks.workunit.client.0.smithi001.stdout:9/815: dread d5/d27/db1/d5e/dd3/d109/ffe [0,4194304] 0 2024-02-28T16:31:16.273 INFO:tasks.workunit.client.0.smithi001.stdout:5/656: link lb dc/d12/d56/d96/db8/dd7/ld8 0 2024-02-28T16:31:16.274 INFO:tasks.workunit.client.1.smithi066.stdout:9/240: chown da/df/d18/d28/f2b 7625520 1 2024-02-28T16:31:16.275 INFO:tasks.workunit.client.1.smithi066.stdout:7/216: write f4 [1577996,19450] 0 2024-02-28T16:31:16.275 INFO:tasks.workunit.client.1.smithi066.stdout:9/241: chown da/df/d18 184 1 2024-02-28T16:31:16.278 INFO:tasks.workunit.client.1.smithi066.stdout:7/217: symlink dc/l42 0 2024-02-28T16:31:16.282 INFO:tasks.workunit.client.1.smithi066.stdout:9/242: getdents da/d22/d46 0 2024-02-28T16:31:16.282 INFO:tasks.workunit.client.1.smithi066.stdout:8/227: sync 2024-02-28T16:31:16.282 INFO:tasks.workunit.client.1.smithi066.stdout:6/218: sync 2024-02-28T16:31:16.283 INFO:tasks.workunit.client.0.smithi001.stdout:1/805: sync 2024-02-28T16:31:16.283 INFO:tasks.workunit.client.0.smithi001.stdout:6/653: sync 2024-02-28T16:31:16.283 INFO:tasks.workunit.client.0.smithi001.stdout:2/664: sync 2024-02-28T16:31:16.283 INFO:tasks.workunit.client.0.smithi001.stdout:4/731: sync 2024-02-28T16:31:16.283 INFO:tasks.workunit.client.0.smithi001.stdout:0/682: sync 2024-02-28T16:31:16.283 INFO:tasks.workunit.client.0.smithi001.stdout:8/691: sync 2024-02-28T16:31:16.285 INFO:tasks.workunit.client.0.smithi001.stdout:0/683: readlink d18/d31/l68 0 2024-02-28T16:31:16.287 INFO:tasks.workunit.client.0.smithi001.stdout:4/732: dread - d3/d2d/d4c/fea zero size 2024-02-28T16:31:16.289 INFO:tasks.workunit.client.1.smithi066.stdout:5/262: dread d3/d4/f9 [0,4194304] 0 2024-02-28T16:31:16.289 INFO:tasks.workunit.client.0.smithi001.stdout:4/733: stat d3/d2d/d4c/d69 0 2024-02-28T16:31:16.289 INFO:tasks.workunit.client.1.smithi066.stdout:3/280: dwrite f5 [0,4194304] 0 2024-02-28T16:31:16.291 INFO:tasks.workunit.client.1.smithi066.stdout:6/219: creat d2/da/d38/f49 x:0 0 0 2024-02-28T16:31:16.292 INFO:tasks.workunit.client.1.smithi066.stdout:6/220: readlink d2/da/l2e 0 2024-02-28T16:31:16.294 INFO:tasks.workunit.client.1.smithi066.stdout:5/263: stat d3/d4/dc/d2f/l59 0 2024-02-28T16:31:16.295 INFO:tasks.workunit.client.1.smithi066.stdout:7/218: unlink dc/de/d17/d1e/f2a 0 2024-02-28T16:31:16.295 INFO:tasks.workunit.client.1.smithi066.stdout:8/228: creat d0/db/d12/d1d/d25/f47 x:0 0 0 2024-02-28T16:31:16.296 INFO:tasks.workunit.client.0.smithi001.stdout:9/816: truncate d5/f1d 5809513 0 2024-02-28T16:31:16.296 INFO:tasks.workunit.client.0.smithi001.stdout:8/692: read d4/d7/f23 [3828813,112322] 0 2024-02-28T16:31:16.296 INFO:tasks.workunit.client.1.smithi066.stdout:3/281: creat db/d13/d1a/d25/f5f x:0 0 0 2024-02-28T16:31:16.297 INFO:tasks.workunit.client.0.smithi001.stdout:6/654: creat d9/db/d5f/d62/d83/fdf x:0 0 0 2024-02-28T16:31:16.297 INFO:tasks.workunit.client.0.smithi001.stdout:2/665: mknod d0/d2/d3/de/d8a/cde 0 2024-02-28T16:31:16.298 INFO:tasks.workunit.client.1.smithi066.stdout:5/264: symlink d3/l5a 0 2024-02-28T16:31:16.298 INFO:tasks.workunit.client.0.smithi001.stdout:0/684: mknod d18/d22/d2f/d1c/d79/ce1 0 2024-02-28T16:31:16.299 INFO:tasks.workunit.client.1.smithi066.stdout:8/229: fsync d0/d6/fe 0 2024-02-28T16:31:16.299 INFO:tasks.workunit.client.1.smithi066.stdout:6/221: unlink d2/da/d19/f17 0 2024-02-28T16:31:16.303 INFO:tasks.workunit.client.1.smithi066.stdout:8/230: write d0/db/d15/d32/f39 [1193974,78211] 0 2024-02-28T16:31:16.304 INFO:tasks.workunit.client.1.smithi066.stdout:5/265: write d3/d4/dc/f19 [740440,60164] 0 2024-02-28T16:31:16.306 INFO:tasks.workunit.client.1.smithi066.stdout:6/222: creat d2/d2c/f4a x:0 0 0 2024-02-28T16:31:16.308 INFO:tasks.workunit.client.1.smithi066.stdout:8/231: readlink d0/d6/dc/d16/l30 0 2024-02-28T16:31:16.308 INFO:tasks.workunit.client.0.smithi001.stdout:6/655: read - d9/db/fc3 zero size 2024-02-28T16:31:16.309 INFO:tasks.workunit.client.1.smithi066.stdout:3/282: link db/dc/d35/f58 db/dc/d35/f60 0 2024-02-28T16:31:16.310 INFO:tasks.workunit.client.1.smithi066.stdout:6/223: dread - d2/d2a/f45 zero size 2024-02-28T16:31:16.312 INFO:tasks.workunit.client.1.smithi066.stdout:3/283: fdatasync db/d13/d14/d24/f34 0 2024-02-28T16:31:16.313 INFO:tasks.workunit.client.0.smithi001.stdout:2/666: chown d0/d2/d3/de/d22/f38 5 1 2024-02-28T16:31:16.313 INFO:tasks.workunit.client.0.smithi001.stdout:3/702: dread d4/d18/da0/fa2 [0,4194304] 0 2024-02-28T16:31:16.317 INFO:tasks.workunit.client.1.smithi066.stdout:3/284: creat db/d13/d14/d2d/f61 x:0 0 0 2024-02-28T16:31:16.318 INFO:tasks.workunit.client.1.smithi066.stdout:8/232: truncate d0/d6/f7 54454 0 2024-02-28T16:31:16.324 INFO:tasks.workunit.client.0.smithi001.stdout:1/806: dwrite d0/d9/d56/d2f/fdb [0,4194304] 0 2024-02-28T16:31:16.330 INFO:tasks.workunit.client.0.smithi001.stdout:5/657: dread dc/d12/d56/d96/db9/dc2/dca/f9d [0,4194304] 0 2024-02-28T16:31:16.330 INFO:tasks.workunit.client.1.smithi066.stdout:8/233: rename d0/d6/dc/f14 to d0/d8/d3a/f48 0 2024-02-28T16:31:16.330 INFO:tasks.workunit.client.0.smithi001.stdout:0/685: creat d18/d22/d76/d7a/fe2 x:0 0 0 2024-02-28T16:31:16.331 INFO:tasks.workunit.client.1.smithi066.stdout:9/243: dread da/d3a/f47 [0,4194304] 0 2024-02-28T16:31:16.331 INFO:tasks.workunit.client.0.smithi001.stdout:4/734: dwrite d3/d2d/d81/fae [0,4194304] 0 2024-02-28T16:31:16.332 INFO:tasks.workunit.client.0.smithi001.stdout:3/703: creat d4/d3c/fe3 x:0 0 0 2024-02-28T16:31:16.334 INFO:tasks.workunit.client.1.smithi066.stdout:8/234: symlink d0/l49 0 2024-02-28T16:31:16.334 INFO:tasks.workunit.client.0.smithi001.stdout:0/686: write d18/d22/d2b/d95/fd3 [29790,69370] 0 2024-02-28T16:31:16.336 INFO:tasks.workunit.client.0.smithi001.stdout:2/667: creat d0/d2/d3/de/d61/d12/d55/fdf x:0 0 0 2024-02-28T16:31:16.337 INFO:tasks.workunit.client.1.smithi066.stdout:7/219: dread dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:16.338 INFO:tasks.workunit.client.1.smithi066.stdout:5/266: truncate d3/d1b/f2d 552808 0 2024-02-28T16:31:16.339 INFO:tasks.workunit.client.0.smithi001.stdout:1/807: creat d0/d9/d1a/d72/f116 x:0 0 0 2024-02-28T16:31:16.341 INFO:tasks.workunit.client.0.smithi001.stdout:0/687: chown d18/d22/d2f/d1c/ldf 997867 1 2024-02-28T16:31:16.343 INFO:tasks.workunit.client.1.smithi066.stdout:8/235: getdents d0/d8 0 2024-02-28T16:31:16.344 INFO:tasks.workunit.client.1.smithi066.stdout:5/267: creat d3/d4/dc/d2f/d55/f5b x:0 0 0 2024-02-28T16:31:16.344 INFO:tasks.workunit.client.0.smithi001.stdout:1/808: dread - d0/d9/d1a/d2b/dfe/f10a zero size 2024-02-28T16:31:16.346 INFO:tasks.workunit.client.1.smithi066.stdout:5/268: readlink d3/l5a 0 2024-02-28T16:31:16.348 INFO:tasks.workunit.client.1.smithi066.stdout:7/220: write dc/de/d17/d1e/f28 [4061412,18780] 0 2024-02-28T16:31:16.350 INFO:tasks.workunit.client.1.smithi066.stdout:7/221: chown l0 433342123 1 2024-02-28T16:31:16.350 INFO:tasks.workunit.client.1.smithi066.stdout:5/269: stat d3/d1b/f37 0 2024-02-28T16:31:16.352 INFO:tasks.workunit.client.1.smithi066.stdout:8/236: rename d0/d6/dc/d16/d1a/f35 to d0/d8/d3a/f4a 0 2024-02-28T16:31:16.353 INFO:tasks.workunit.client.0.smithi001.stdout:6/656: dwrite d9/f4f [0,4194304] 0 2024-02-28T16:31:16.355 INFO:tasks.workunit.client.1.smithi066.stdout:7/222: symlink dc/de/d17/l43 0 2024-02-28T16:31:16.358 INFO:tasks.workunit.client.1.smithi066.stdout:8/237: creat d0/db/d12/f4b x:0 0 0 2024-02-28T16:31:16.359 INFO:tasks.workunit.client.1.smithi066.stdout:5/270: creat d3/d4/f5c x:0 0 0 2024-02-28T16:31:16.359 INFO:tasks.workunit.client.0.smithi001.stdout:6/657: chown d9/db/d10/f7b 403221410 1 2024-02-28T16:31:16.360 INFO:tasks.workunit.client.0.smithi001.stdout:1/809: symlink d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/l117 0 2024-02-28T16:31:16.363 INFO:tasks.workunit.client.1.smithi066.stdout:7/223: creat dc/d29/f44 x:0 0 0 2024-02-28T16:31:16.363 INFO:tasks.workunit.client.1.smithi066.stdout:5/271: read - d3/d1b/d39/f44 zero size 2024-02-28T16:31:16.364 INFO:tasks.workunit.client.0.smithi001.stdout:1/810: chown d0/d9/d56 543 1 2024-02-28T16:31:16.364 INFO:tasks.workunit.client.1.smithi066.stdout:9/244: dwrite da/f30 [0,4194304] 0 2024-02-28T16:31:16.368 INFO:tasks.workunit.client.1.smithi066.stdout:7/224: stat dc/l1f 0 2024-02-28T16:31:16.368 INFO:tasks.workunit.client.1.smithi066.stdout:5/272: write d3/d4/dc/f3f [1121480,55814] 0 2024-02-28T16:31:16.369 INFO:tasks.workunit.client.0.smithi001.stdout:5/658: link dc/d12/d41/db7/fc6 dc/d12/d56/d96/fd9 0 2024-02-28T16:31:16.374 INFO:tasks.workunit.client.0.smithi001.stdout:0/688: rename d18/da9/dc9/fa2 to d18/d22/d2f/d1c/d90/d9f/fe3 0 2024-02-28T16:31:16.375 INFO:tasks.workunit.client.0.smithi001.stdout:4/735: dwrite d3/d1c/d58/d6e/fbb [0,4194304] 0 2024-02-28T16:31:16.375 INFO:tasks.workunit.client.1.smithi066.stdout:7/225: chown dc/de/l18 575 1 2024-02-28T16:31:16.375 INFO:tasks.workunit.client.0.smithi001.stdout:6/658: mkdir d9/d39/de0 0 2024-02-28T16:31:16.377 INFO:tasks.workunit.client.1.smithi066.stdout:9/245: creat da/d3a/d15/d48/f52 x:0 0 0 2024-02-28T16:31:16.379 INFO:tasks.workunit.client.0.smithi001.stdout:1/811: chown d0/d9/d1a/d2b/d26/d27/d30/d67/d87/fd4 0 1 2024-02-28T16:31:16.382 INFO:tasks.workunit.client.0.smithi001.stdout:2/668: write d0/d2/d3/de/f5e [5832522,55042] 0 2024-02-28T16:31:16.382 INFO:tasks.workunit.client.0.smithi001.stdout:4/736: chown d3/d5/c5f 7621051 1 2024-02-28T16:31:16.383 INFO:tasks.workunit.client.0.smithi001.stdout:8/693: dread d4/f11 [0,4194304] 0 2024-02-28T16:31:16.384 INFO:tasks.workunit.client.1.smithi066.stdout:8/238: dwrite d0/db/d15/f44 [0,4194304] 0 2024-02-28T16:31:16.386 INFO:tasks.workunit.client.1.smithi066.stdout:8/239: chown d0/d6/dc/l36 83959261 1 2024-02-28T16:31:16.386 INFO:tasks.workunit.client.0.smithi001.stdout:5/659: mknod dc/d12/d56/d96/db8/dd7/cda 0 2024-02-28T16:31:16.387 INFO:tasks.workunit.client.1.smithi066.stdout:5/273: getdents d3/d4/dc/d42 0 2024-02-28T16:31:16.389 INFO:tasks.workunit.client.1.smithi066.stdout:8/240: readlink d0/d6/dc/l2e 0 2024-02-28T16:31:16.390 INFO:tasks.workunit.client.0.smithi001.stdout:1/812: write d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/fa0 [3951423,113799] 0 2024-02-28T16:31:16.390 INFO:tasks.workunit.client.0.smithi001.stdout:3/704: truncate d4/d18/d5f/f77 3166290 0 2024-02-28T16:31:16.391 INFO:tasks.workunit.client.0.smithi001.stdout:0/689: mknod d18/da9/dc9/dd7/ce4 0 2024-02-28T16:31:16.391 INFO:tasks.workunit.client.1.smithi066.stdout:9/246: mkdir da/d4b/d53 0 2024-02-28T16:31:16.395 INFO:tasks.workunit.client.0.smithi001.stdout:1/813: write d0/d9/d1a/d72/fb9 [1357362,33316] 0 2024-02-28T16:31:16.396 INFO:tasks.workunit.client.0.smithi001.stdout:0/690: rename d18/da9 to d18/da9/dc9/dd7/de5 22 2024-02-28T16:31:16.399 INFO:tasks.workunit.client.0.smithi001.stdout:6/659: link d9/db/fd8 d9/db/d5f/d89/d8a/fe1 0 2024-02-28T16:31:16.401 INFO:tasks.workunit.client.0.smithi001.stdout:4/737: creat d3/d1c/d58/d6e/d9f/fec x:0 0 0 2024-02-28T16:31:16.402 INFO:tasks.workunit.client.1.smithi066.stdout:3/285: dread db/d13/d1a/f1b [0,4194304] 0 2024-02-28T16:31:16.403 INFO:tasks.workunit.client.1.smithi066.stdout:5/274: truncate d3/d4/dc/dd/d2e/d18/f2c 285729 0 2024-02-28T16:31:16.409 INFO:tasks.workunit.client.1.smithi066.stdout:3/286: mknod db/d2a/d44/d48/d5d/c62 0 2024-02-28T16:31:16.413 INFO:tasks.workunit.client.1.smithi066.stdout:9/247: dread da/f20 [0,4194304] 0 2024-02-28T16:31:16.414 INFO:tasks.workunit.client.1.smithi066.stdout:5/275: truncate d3/d4/dc/dd/d2e/f38 1156902 0 2024-02-28T16:31:16.414 INFO:tasks.workunit.client.1.smithi066.stdout:7/226: dwrite dc/f25 [0,4194304] 0 2024-02-28T16:31:16.414 INFO:tasks.workunit.client.1.smithi066.stdout:8/241: dread d0/d6/f9 [0,4194304] 0 2024-02-28T16:31:16.416 INFO:tasks.workunit.client.1.smithi066.stdout:9/248: chown da/f35 127 1 2024-02-28T16:31:16.417 INFO:tasks.workunit.client.1.smithi066.stdout:3/287: rename f5 to db/d2a/d44/d48/f63 0 2024-02-28T16:31:16.418 INFO:tasks.workunit.client.0.smithi001.stdout:3/705: getdents d4/d1e/dcf 0 2024-02-28T16:31:16.418 INFO:tasks.workunit.client.0.smithi001.stdout:5/660: creat dc/d12/d56/fdb x:0 0 0 2024-02-28T16:31:16.423 INFO:tasks.workunit.client.0.smithi001.stdout:3/706: stat d4/d18/d5f/f47 0 2024-02-28T16:31:16.423 INFO:tasks.workunit.client.1.smithi066.stdout:7/227: unlink dc/de/d17/l43 0 2024-02-28T16:31:16.425 INFO:tasks.workunit.client.0.smithi001.stdout:4/738: write d3/f78 [4893621,27565] 0 2024-02-28T16:31:16.425 INFO:tasks.workunit.client.1.smithi066.stdout:9/249: creat da/d4b/f54 x:0 0 0 2024-02-28T16:31:16.427 INFO:tasks.workunit.client.1.smithi066.stdout:5/276: dread d3/d4/dc/f1d [0,4194304] 0 2024-02-28T16:31:16.429 INFO:tasks.workunit.client.0.smithi001.stdout:8/694: dwrite d4/d32/d93/fa2 [0,4194304] 0 2024-02-28T16:31:16.430 INFO:tasks.workunit.client.1.smithi066.stdout:3/288: truncate db/dc/d35/f58 1183032 0 2024-02-28T16:31:16.430 INFO:tasks.workunit.client.0.smithi001.stdout:0/691: dwrite d18/d31/fad [0,4194304] 0 2024-02-28T16:31:16.432 INFO:tasks.workunit.client.0.smithi001.stdout:1/814: dwrite d0/d9/d1a/d2b/dfe/f109 [0,4194304] 0 2024-02-28T16:31:16.433 INFO:tasks.workunit.client.1.smithi066.stdout:5/277: stat d3/d1b/c29 0 2024-02-28T16:31:16.434 INFO:tasks.workunit.client.0.smithi001.stdout:5/661: fdatasync dc/d12/d56/d96/db9/dc2/dca/d3c/f53 0 2024-02-28T16:31:16.434 INFO:tasks.workunit.client.1.smithi066.stdout:8/242: getdents d0 0 2024-02-28T16:31:16.437 INFO:tasks.workunit.client.1.smithi066.stdout:8/243: stat d0/db/f27 0 2024-02-28T16:31:16.438 INFO:tasks.workunit.client.0.smithi001.stdout:3/707: symlink d4/d6/d7/d78/d81/dd6/le4 0 2024-02-28T16:31:16.438 INFO:tasks.workunit.client.1.smithi066.stdout:8/244: chown d0/db/d15/f37 1557 1 2024-02-28T16:31:16.440 INFO:tasks.workunit.client.1.smithi066.stdout:5/278: mkdir d3/d4/dc/dd/d41/d5d 0 2024-02-28T16:31:16.440 INFO:tasks.workunit.client.1.smithi066.stdout:1/301: sync 2024-02-28T16:31:16.446 INFO:tasks.workunit.client.1.smithi066.stdout:8/245: rename d0/d22 to d0/d6/dc/d16/d4c 0 2024-02-28T16:31:16.446 INFO:tasks.workunit.client.0.smithi001.stdout:0/692: mknod d18/d22/d2b/d95/ce6 0 2024-02-28T16:31:16.446 INFO:tasks.workunit.client.0.smithi001.stdout:5/662: chown dc/d12/d56/d96/db8/dd7/ld8 0 1 2024-02-28T16:31:16.447 INFO:tasks.workunit.client.0.smithi001.stdout:3/708: symlink d4/d6/d7/d78/d81/le5 0 2024-02-28T16:31:16.448 INFO:tasks.workunit.client.0.smithi001.stdout:7/697: sync 2024-02-28T16:31:16.449 INFO:tasks.workunit.client.0.smithi001.stdout:6/660: write d9/db/d19/f8e [423596,42304] 0 2024-02-28T16:31:16.453 INFO:tasks.workunit.client.0.smithi001.stdout:6/661: chown d9/d29/l98 10 1 2024-02-28T16:31:16.453 INFO:tasks.workunit.client.1.smithi066.stdout:1/302: link d2/d18/l40 d2/l5e 0 2024-02-28T16:31:16.454 INFO:tasks.workunit.client.0.smithi001.stdout:3/709: chown d4/d6/d7/d78/d81/dd6/dd9 22 1 2024-02-28T16:31:16.457 INFO:tasks.workunit.client.0.smithi001.stdout:6/662: stat d9/fdb 0 2024-02-28T16:31:16.457 INFO:tasks.workunit.client.0.smithi001.stdout:5/663: fsync dc/d12/f24 0 2024-02-28T16:31:16.459 INFO:tasks.workunit.client.0.smithi001.stdout:7/698: write d3/d17/d52/d7f/d99/dc1/d49/fd8 [1021645,52741] 0 2024-02-28T16:31:16.460 INFO:tasks.workunit.client.0.smithi001.stdout:3/710: mknod d4/d6/ce6 0 2024-02-28T16:31:16.461 INFO:tasks.workunit.client.1.smithi066.stdout:9/250: rmdir da/d3a/d15 39 2024-02-28T16:31:16.461 INFO:tasks.workunit.client.1.smithi066.stdout:1/303: symlink d2/d27/d2b/d37/l5f 0 2024-02-28T16:31:16.464 INFO:tasks.workunit.client.0.smithi001.stdout:6/663: dread - d9/d39/fa7 zero size 2024-02-28T16:31:16.466 INFO:tasks.workunit.client.1.smithi066.stdout:5/279: dread d3/d4/dc/d42/f49 [0,4194304] 0 2024-02-28T16:31:16.467 INFO:tasks.workunit.client.1.smithi066.stdout:9/251: fsync da/d3a/d15/d48/f52 0 2024-02-28T16:31:16.468 INFO:tasks.workunit.client.1.smithi066.stdout:1/304: creat d2/d27/d2b/d37/d51/f60 x:0 0 0 2024-02-28T16:31:16.468 INFO:tasks.workunit.client.1.smithi066.stdout:5/280: stat d3/d4/f12 0 2024-02-28T16:31:16.470 INFO:tasks.workunit.client.0.smithi001.stdout:8/695: dwrite d4/d7/f10 [8388608,4194304] 0 2024-02-28T16:31:16.477 INFO:tasks.workunit.client.0.smithi001.stdout:6/664: write d9/db/fc3 [2503,71884] 0 2024-02-28T16:31:16.479 INFO:tasks.workunit.client.0.smithi001.stdout:1/815: truncate d0/d9/d56/f4b 1550049 0 2024-02-28T16:31:16.480 INFO:tasks.workunit.client.0.smithi001.stdout:4/739: dwrite d3/d1c/da0/fb2 [4194304,4194304] 0 2024-02-28T16:31:16.481 INFO:tasks.workunit.client.0.smithi001.stdout:4/740: chown d3/d2d/d4c 1224468 1 2024-02-28T16:31:16.483 INFO:tasks.workunit.client.0.smithi001.stdout:6/665: truncate d9/db/d19/f1f 5467264 0 2024-02-28T16:31:16.486 INFO:tasks.workunit.client.1.smithi066.stdout:3/289: truncate db/dc/d11/d1f/f5e 3137823 0 2024-02-28T16:31:16.486 INFO:tasks.workunit.client.1.smithi066.stdout:7/228: write f1 [139685,44845] 0 2024-02-28T16:31:16.487 INFO:tasks.workunit.client.0.smithi001.stdout:0/693: getdents d18/d22/d57/d7e 0 2024-02-28T16:31:16.489 INFO:tasks.workunit.client.0.smithi001.stdout:0/694: chown d18/d22/d2f/d1c/d79 11 1 2024-02-28T16:31:16.491 INFO:tasks.workunit.client.0.smithi001.stdout:6/666: creat d9/de/fe2 x:0 0 0 2024-02-28T16:31:16.492 INFO:tasks.workunit.client.0.smithi001.stdout:1/816: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/f86 0 2024-02-28T16:31:16.493 INFO:tasks.workunit.client.0.smithi001.stdout:5/664: link dc/d12/d41/f4c dc/d12/d56/d96/db9/fdc 0 2024-02-28T16:31:16.496 INFO:tasks.workunit.client.1.smithi066.stdout:7/229: read dc/d29/f2e [155318,103550] 0 2024-02-28T16:31:16.497 INFO:tasks.workunit.client.0.smithi001.stdout:4/741: mknod d3/d5/d47/d90/ced 0 2024-02-28T16:31:16.498 INFO:tasks.workunit.client.1.smithi066.stdout:3/290: mknod db/d13/d1a/d33/c64 0 2024-02-28T16:31:16.500 INFO:tasks.workunit.client.0.smithi001.stdout:7/699: write d3/d17/d3d/f72 [3301656,117363] 0 2024-02-28T16:31:16.502 INFO:tasks.workunit.client.1.smithi066.stdout:5/281: write d3/d4/f9 [33132,39868] 0 2024-02-28T16:31:16.506 INFO:tasks.workunit.client.1.smithi066.stdout:5/282: fsync d3/d4/dc/d2f/f32 0 2024-02-28T16:31:16.508 INFO:tasks.workunit.client.0.smithi001.stdout:6/667: creat d9/db/dd9/fe3 x:0 0 0 2024-02-28T16:31:16.508 INFO:tasks.workunit.client.0.smithi001.stdout:1/817: creat d0/d9/d110/f118 x:0 0 0 2024-02-28T16:31:16.509 INFO:tasks.workunit.client.1.smithi066.stdout:9/252: dwrite da/df/d18/d28/f2b [0,4194304] 0 2024-02-28T16:31:16.509 INFO:tasks.workunit.client.0.smithi001.stdout:8/696: link d4/d7/d22/c26 d4/d7/d22/d27/d4b/ce4 0 2024-02-28T16:31:16.512 INFO:tasks.workunit.client.0.smithi001.stdout:1/818: write d0/d9/f85 [2946189,36545] 0 2024-02-28T16:31:16.512 INFO:tasks.workunit.client.1.smithi066.stdout:1/305: dwrite d2/f16 [0,4194304] 0 2024-02-28T16:31:16.515 INFO:tasks.workunit.client.1.smithi066.stdout:9/253: write da/d3a/d15/f1d [961989,80562] 0 2024-02-28T16:31:16.516 INFO:tasks.workunit.client.0.smithi001.stdout:4/742: dread d3/d1c/da0/fb2 [0,4194304] 0 2024-02-28T16:31:16.521 INFO:tasks.workunit.client.0.smithi001.stdout:3/711: unlink d4/d6/d7/d78/d81/fa6 0 2024-02-28T16:31:16.522 INFO:tasks.workunit.client.0.smithi001.stdout:6/668: creat d9/de/dbd/fe4 x:0 0 0 2024-02-28T16:31:16.522 INFO:tasks.workunit.client.1.smithi066.stdout:7/230: dwrite f5 [4194304,4194304] 0 2024-02-28T16:31:16.523 INFO:tasks.workunit.client.1.smithi066.stdout:9/254: fsync da/df/f29 0 2024-02-28T16:31:16.524 INFO:tasks.workunit.client.1.smithi066.stdout:5/283: truncate d3/d4/f12 1082036 0 2024-02-28T16:31:16.525 INFO:tasks.workunit.client.0.smithi001.stdout:5/665: rename dc/d1a/d7d/d87/db2 to dc/d12/d56/d96/db9/ddd 0 2024-02-28T16:31:16.525 INFO:tasks.workunit.client.1.smithi066.stdout:5/284: dread - d3/d1b/f37 zero size 2024-02-28T16:31:16.528 INFO:tasks.workunit.client.0.smithi001.stdout:0/695: dwrite d18/d22/d2b/f87 [0,4194304] 0 2024-02-28T16:31:16.528 INFO:tasks.workunit.client.0.smithi001.stdout:1/819: creat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f119 x:0 0 0 2024-02-28T16:31:16.528 INFO:tasks.workunit.client.1.smithi066.stdout:7/231: mkdir dc/de/d17/d1e/d45 0 2024-02-28T16:31:16.530 INFO:tasks.workunit.client.1.smithi066.stdout:9/255: rmdir da/df/d18/d2d 39 2024-02-28T16:31:16.530 INFO:tasks.workunit.client.1.smithi066.stdout:5/285: write d3/d1b/f31 [3403053,57907] 0 2024-02-28T16:31:16.532 INFO:tasks.workunit.client.0.smithi001.stdout:5/666: truncate dc/d1a/d7d/d87/f8f 1015843 0 2024-02-28T16:31:16.532 INFO:tasks.workunit.client.1.smithi066.stdout:9/256: readlink da/df/l25 0 2024-02-28T16:31:16.534 INFO:tasks.workunit.client.1.smithi066.stdout:1/306: dread d2/f16 [0,4194304] 0 2024-02-28T16:31:16.536 INFO:tasks.workunit.client.1.smithi066.stdout:9/257: fsync da/df/f3d 0 2024-02-28T16:31:16.537 INFO:tasks.workunit.client.0.smithi001.stdout:6/669: read d9/db/d5f/d62/f71 [366738,118210] 0 2024-02-28T16:31:16.537 INFO:tasks.workunit.client.0.smithi001.stdout:8/697: link d4/d7/d22/d27/d58/f7b d4/d47/d8b/fe5 0 2024-02-28T16:31:16.538 INFO:tasks.workunit.client.1.smithi066.stdout:5/286: creat d3/d4/dc/d42/d57/f5e x:0 0 0 2024-02-28T16:31:16.539 INFO:tasks.workunit.client.1.smithi066.stdout:1/307: chown d2/d27/d2b/d34/c42 7807 1 2024-02-28T16:31:16.540 INFO:tasks.workunit.client.1.smithi066.stdout:7/232: rename dc/l2c to dc/de/d17/l46 0 2024-02-28T16:31:16.542 INFO:tasks.workunit.client.1.smithi066.stdout:7/233: fsync f4 0 2024-02-28T16:31:16.542 INFO:tasks.workunit.client.1.smithi066.stdout:5/287: stat d3/d4/dc/dd/d41/d51 0 2024-02-28T16:31:16.543 INFO:tasks.workunit.client.0.smithi001.stdout:8/698: dread - d4/d7/d22/d27/f56 zero size 2024-02-28T16:31:16.545 INFO:tasks.workunit.client.0.smithi001.stdout:4/743: creat d3/d5/d7/d1f/dc3/fee x:0 0 0 2024-02-28T16:31:16.547 INFO:tasks.workunit.client.0.smithi001.stdout:7/700: rename d3/d17/d3d/fae to d3/d17/d3d/d42/d8e/fe3 0 2024-02-28T16:31:16.550 INFO:tasks.workunit.client.0.smithi001.stdout:6/670: write d9/d29/d6a/f77 [4136629,25529] 0 2024-02-28T16:31:16.553 INFO:tasks.workunit.client.1.smithi066.stdout:9/258: rename da/d21/l31 to da/df/l55 0 2024-02-28T16:31:16.555 INFO:tasks.workunit.client.0.smithi001.stdout:1/820: rename d0/d9/d1a/d2b/d26/d27/d30/l91 to d0/d9/d110/l11a 0 2024-02-28T16:31:16.555 INFO:tasks.workunit.client.0.smithi001.stdout:4/744: write d3/d2d/d4c/d5e/db0/fba [29521,60847] 0 2024-02-28T16:31:16.557 INFO:tasks.workunit.client.1.smithi066.stdout:3/291: write db/dc/d11/f22 [3111567,75037] 0 2024-02-28T16:31:16.559 INFO:tasks.workunit.client.1.smithi066.stdout:1/308: link d2/d18/d19/f1c d2/d27/d2b/d43/d46/f61 0 2024-02-28T16:31:16.559 INFO:tasks.workunit.client.0.smithi001.stdout:7/701: creat d3/d17/d3d/db3/fe4 x:0 0 0 2024-02-28T16:31:16.560 INFO:tasks.workunit.client.0.smithi001.stdout:5/667: fsync dc/d12/d41/d54/f9a 0 2024-02-28T16:31:16.561 INFO:tasks.workunit.client.1.smithi066.stdout:9/259: fdatasync da/d3a/f47 0 2024-02-28T16:31:16.562 INFO:tasks.workunit.client.1.smithi066.stdout:5/288: creat d3/d1b/f5f x:0 0 0 2024-02-28T16:31:16.562 INFO:tasks.workunit.client.0.smithi001.stdout:9/817: sync 2024-02-28T16:31:16.565 INFO:tasks.workunit.client.1.smithi066.stdout:1/309: truncate d2/d27/d2b/d43/d46/f61 3829589 0 2024-02-28T16:31:16.565 INFO:tasks.workunit.client.0.smithi001.stdout:4/745: write d3/f3b [4454705,108050] 0 2024-02-28T16:31:16.566 INFO:tasks.workunit.client.1.smithi066.stdout:4/397: sync 2024-02-28T16:31:16.566 INFO:tasks.workunit.client.1.smithi066.stdout:0/373: sync 2024-02-28T16:31:16.566 INFO:tasks.workunit.client.1.smithi066.stdout:2/295: sync 2024-02-28T16:31:16.566 INFO:tasks.workunit.client.1.smithi066.stdout:6/224: sync 2024-02-28T16:31:16.567 INFO:tasks.workunit.client.0.smithi001.stdout:1/821: symlink d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/l11b 0 2024-02-28T16:31:16.567 INFO:tasks.workunit.client.1.smithi066.stdout:5/289: mkdir d3/d4/dc/d42/d60 0 2024-02-28T16:31:16.568 INFO:tasks.workunit.client.0.smithi001.stdout:7/702: truncate d3/d17/d3d/d42/f7d 543935 0 2024-02-28T16:31:16.572 INFO:tasks.workunit.client.0.smithi001.stdout:4/746: dread d3/d5/d7/d1f/d2e/dbe/d87/f8b [0,4194304] 0 2024-02-28T16:31:16.574 INFO:tasks.workunit.client.0.smithi001.stdout:1/822: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/fc6 0 2024-02-28T16:31:16.575 INFO:tasks.workunit.client.1.smithi066.stdout:1/310: creat d2/d27/d2b/d37/d51/f62 x:0 0 0 2024-02-28T16:31:16.576 INFO:tasks.workunit.client.1.smithi066.stdout:0/374: chown d2/d9/f75 23670 1 2024-02-28T16:31:16.577 INFO:tasks.workunit.client.1.smithi066.stdout:5/290: write d3/d4/f9 [1754072,40757] 0 2024-02-28T16:31:16.578 INFO:tasks.workunit.client.0.smithi001.stdout:3/712: write d4/d6/d7/d23/f7a [662133,118533] 0 2024-02-28T16:31:16.578 INFO:tasks.workunit.client.0.smithi001.stdout:9/818: dread d5/d27/d34/f8d [0,4194304] 0 2024-02-28T16:31:16.580 INFO:tasks.workunit.client.0.smithi001.stdout:1/823: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f119 0 2024-02-28T16:31:16.581 INFO:tasks.workunit.client.1.smithi066.stdout:6/225: symlink d2/da/d19/l4b 0 2024-02-28T16:31:16.583 INFO:tasks.workunit.client.0.smithi001.stdout:3/713: dread - d4/d6/f9b zero size 2024-02-28T16:31:16.584 INFO:tasks.workunit.client.1.smithi066.stdout:0/375: write d2/d9/f75 [265549,94054] 0 2024-02-28T16:31:16.585 INFO:tasks.workunit.client.0.smithi001.stdout:9/819: fdatasync d5/fd 0 2024-02-28T16:31:16.585 INFO:tasks.workunit.client.1.smithi066.stdout:3/292: truncate db/ff 783479 0 2024-02-28T16:31:16.586 INFO:tasks.workunit.client.0.smithi001.stdout:3/714: chown d4/d18/d5f/fb8 33514 1 2024-02-28T16:31:16.586 INFO:tasks.workunit.client.0.smithi001.stdout:0/696: write d18/d31/fad [5108232,6808] 0 2024-02-28T16:31:16.588 INFO:tasks.workunit.client.0.smithi001.stdout:8/699: write d4/f11 [473514,76881] 0 2024-02-28T16:31:16.588 INFO:tasks.workunit.client.1.smithi066.stdout:8/246: sync 2024-02-28T16:31:16.589 INFO:tasks.workunit.client.1.smithi066.stdout:9/260: write da/d3a/f41 [987960,70396] 0 2024-02-28T16:31:16.590 INFO:tasks.workunit.client.0.smithi001.stdout:8/700: chown d4/d32/d69/f9d 1 1 2024-02-28T16:31:16.590 INFO:tasks.workunit.client.1.smithi066.stdout:6/226: unlink d2/d2a/f35 0 2024-02-28T16:31:16.591 INFO:tasks.workunit.client.1.smithi066.stdout:1/311: creat d2/d27/d2b/d43/d46/f63 x:0 0 0 2024-02-28T16:31:16.593 INFO:tasks.workunit.client.1.smithi066.stdout:6/227: fdatasync f1 0 2024-02-28T16:31:16.594 INFO:tasks.workunit.client.0.smithi001.stdout:0/697: dread - d18/d22/d3d/fa1 zero size 2024-02-28T16:31:16.596 INFO:tasks.workunit.client.1.smithi066.stdout:2/296: dwrite d0/f1e [0,4194304] 0 2024-02-28T16:31:16.597 INFO:tasks.workunit.client.1.smithi066.stdout:5/291: write d3/d4/dc/dd/d2e/d18/f2b [2729311,54583] 0 2024-02-28T16:31:16.598 INFO:tasks.workunit.client.0.smithi001.stdout:4/747: dread d3/d5/d7/d1f/d20/de0/de6/f80 [0,4194304] 0 2024-02-28T16:31:16.599 INFO:tasks.workunit.client.1.smithi066.stdout:4/398: getdents d1/d23/d25/d3e/d40 0 2024-02-28T16:31:16.600 INFO:tasks.workunit.client.1.smithi066.stdout:9/261: mkdir da/d22/d46/d56 0 2024-02-28T16:31:16.600 INFO:tasks.workunit.client.1.smithi066.stdout:0/376: unlink d2/d9/d10/d23/c6c 0 2024-02-28T16:31:16.601 INFO:tasks.workunit.client.1.smithi066.stdout:2/297: readlink d0/d12/l1b 0 2024-02-28T16:31:16.603 INFO:tasks.workunit.client.0.smithi001.stdout:4/748: write d3/d5/fe2 [863690,76584] 0 2024-02-28T16:31:16.604 INFO:tasks.workunit.client.1.smithi066.stdout:5/292: fsync d3/d1b/f31 0 2024-02-28T16:31:16.604 INFO:tasks.workunit.client.1.smithi066.stdout:3/293: dread db/dc/d11/f26 [0,4194304] 0 2024-02-28T16:31:16.604 INFO:tasks.workunit.client.1.smithi066.stdout:6/228: mknod d2/da/d3e/c4c 0 2024-02-28T16:31:16.605 INFO:tasks.workunit.client.0.smithi001.stdout:2/669: sync 2024-02-28T16:31:16.609 INFO:tasks.workunit.client.0.smithi001.stdout:3/715: mknod d4/d6/d7f/ce7 0 2024-02-28T16:31:16.609 INFO:tasks.workunit.client.1.smithi066.stdout:6/229: readlink d2/da/d19/l4b 0 2024-02-28T16:31:16.610 INFO:tasks.workunit.client.1.smithi066.stdout:2/298: chown d0/d12/d1a/c24 9308638 1 2024-02-28T16:31:16.612 INFO:tasks.workunit.client.0.smithi001.stdout:6/671: dwrite d9/db/d5f/d89/d8a/fe1 [0,4194304] 0 2024-02-28T16:31:16.614 INFO:tasks.workunit.client.1.smithi066.stdout:4/399: rename d1/d18/f3a to d1/d18/d4e/f80 0 2024-02-28T16:31:16.616 INFO:tasks.workunit.client.0.smithi001.stdout:3/716: read - d4/d6/d7/d78/d81/dd6/fe2 zero size 2024-02-28T16:31:16.621 INFO:tasks.workunit.client.0.smithi001.stdout:9/820: creat d5/d27/d42/f10b x:0 0 0 2024-02-28T16:31:16.622 INFO:tasks.workunit.client.0.smithi001.stdout:7/703: truncate d3/d17/d52/d7f/d99/dc1/f6d 1442016 0 2024-02-28T16:31:16.625 INFO:tasks.workunit.client.1.smithi066.stdout:8/247: mkdir d0/db/d4d 0 2024-02-28T16:31:16.625 INFO:tasks.workunit.client.1.smithi066.stdout:9/262: dwrite da/d4b/f54 [0,4194304] 0 2024-02-28T16:31:16.626 INFO:tasks.workunit.client.1.smithi066.stdout:7/234: write dc/d29/f2e [2464829,94545] 0 2024-02-28T16:31:16.627 INFO:tasks.workunit.client.0.smithi001.stdout:5/668: dwrite dc/d1a/f8d [0,4194304] 0 2024-02-28T16:31:16.632 INFO:tasks.workunit.client.1.smithi066.stdout:2/299: creat d0/d12/d1a/d39/d40/f5b x:0 0 0 2024-02-28T16:31:16.637 INFO:tasks.workunit.client.0.smithi001.stdout:9/821: mknod d5/d27/d42/def/c10c 0 2024-02-28T16:31:16.639 INFO:tasks.workunit.client.0.smithi001.stdout:7/704: truncate d3/d17/d52/d92/fa2 2354025 0 2024-02-28T16:31:16.640 INFO:tasks.workunit.client.1.smithi066.stdout:4/400: mkdir d1/d18/d55/d81 0 2024-02-28T16:31:16.641 INFO:tasks.workunit.client.0.smithi001.stdout:0/698: getdents d18/da9 0 2024-02-28T16:31:16.643 INFO:tasks.workunit.client.1.smithi066.stdout:3/294: fsync db/dc/d35/f60 0 2024-02-28T16:31:16.644 INFO:tasks.workunit.client.1.smithi066.stdout:5/293: truncate d3/d4/f12 2020209 0 2024-02-28T16:31:16.645 INFO:tasks.workunit.client.0.smithi001.stdout:6/672: dwrite d9/db/d5f/d62/d83/f8f [0,4194304] 0 2024-02-28T16:31:16.646 INFO:tasks.workunit.client.1.smithi066.stdout:1/312: getdents d2/d27/d2b/d37/d54 0 2024-02-28T16:31:16.651 INFO:tasks.workunit.client.0.smithi001.stdout:1/824: dwrite d0/d9/f5c [0,4194304] 0 2024-02-28T16:31:16.655 INFO:tasks.workunit.client.0.smithi001.stdout:6/673: write d9/db/d5f/d8b/f6d [7728166,33724] 0 2024-02-28T16:31:16.655 INFO:tasks.workunit.client.1.smithi066.stdout:7/235: write dc/de/d17/d1e/f20 [502319,112611] 0 2024-02-28T16:31:16.657 INFO:tasks.workunit.client.0.smithi001.stdout:2/670: dread d0/d2/d3/de/d8a/f71 [0,4194304] 0 2024-02-28T16:31:16.659 INFO:tasks.workunit.client.0.smithi001.stdout:9/822: creat d5/d27/db1/d5e/dd3/d109/d6f/f10d x:0 0 0 2024-02-28T16:31:16.661 INFO:tasks.workunit.client.1.smithi066.stdout:3/295: creat db/d2a/f65 x:0 0 0 2024-02-28T16:31:16.664 INFO:tasks.workunit.client.1.smithi066.stdout:4/401: rename d1/f5 to d1/d23/d25/d43/f82 0 2024-02-28T16:31:16.665 INFO:tasks.workunit.client.0.smithi001.stdout:7/705: write d3/f45 [547840,110431] 0 2024-02-28T16:31:16.665 INFO:tasks.workunit.client.0.smithi001.stdout:3/717: dwrite d4/d6/f1c [4194304,4194304] 0 2024-02-28T16:31:16.665 INFO:tasks.workunit.client.1.smithi066.stdout:9/263: creat da/df/d18/d2d/f57 x:0 0 0 2024-02-28T16:31:16.666 INFO:tasks.workunit.client.1.smithi066.stdout:1/313: fdatasync d2/d27/d2b/f39 0 2024-02-28T16:31:16.667 INFO:tasks.workunit.client.0.smithi001.stdout:6/674: creat d9/d29/d6e/d4e/d96/fe5 x:0 0 0 2024-02-28T16:31:16.671 INFO:tasks.workunit.client.0.smithi001.stdout:4/749: write d3/d1c/d58/f2a [1464421,11537] 0 2024-02-28T16:31:16.672 INFO:tasks.workunit.client.0.smithi001.stdout:5/669: dwrite dc/df/f81 [0,4194304] 0 2024-02-28T16:31:16.675 INFO:tasks.workunit.client.0.smithi001.stdout:4/750: fdatasync d3/fd6 0 2024-02-28T16:31:16.675 INFO:tasks.workunit.client.0.smithi001.stdout:3/718: read d4/d18/f91 [3064682,43498] 0 2024-02-28T16:31:16.676 INFO:tasks.workunit.client.1.smithi066.stdout:0/377: dwrite d2/d9/f11 [0,4194304] 0 2024-02-28T16:31:16.676 INFO:tasks.workunit.client.0.smithi001.stdout:9/823: fdatasync d5/d27/db1/d5e/dd3/d109/d6f/fc7 0 2024-02-28T16:31:16.677 INFO:tasks.workunit.client.0.smithi001.stdout:8/701: write d4/d7/da8/d89/fce [147394,46331] 0 2024-02-28T16:31:16.678 INFO:tasks.workunit.client.1.smithi066.stdout:1/314: stat d2/f13 0 2024-02-28T16:31:16.681 INFO:tasks.workunit.client.1.smithi066.stdout:2/300: creat d0/d12/f5c x:0 0 0 2024-02-28T16:31:16.683 INFO:tasks.workunit.client.0.smithi001.stdout:1/825: creat d0/d9/d56/f11c x:0 0 0 2024-02-28T16:31:16.684 INFO:tasks.workunit.client.1.smithi066.stdout:7/236: dread dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:16.684 INFO:tasks.workunit.client.0.smithi001.stdout:5/670: symlink dc/d12/d56/d96/db8/dd7/lde 0 2024-02-28T16:31:16.685 INFO:tasks.workunit.client.1.smithi066.stdout:4/402: creat d1/d23/d25/f83 x:0 0 0 2024-02-28T16:31:16.688 INFO:tasks.workunit.client.0.smithi001.stdout:4/751: creat d3/d2d/d4c/d69/d82/fef x:0 0 0 2024-02-28T16:31:16.690 INFO:tasks.workunit.client.1.smithi066.stdout:9/264: mkdir da/d3a/d58 0 2024-02-28T16:31:16.691 INFO:tasks.workunit.client.0.smithi001.stdout:3/719: fsync d4/d18/d5f/f59 0 2024-02-28T16:31:16.695 INFO:tasks.workunit.client.1.smithi066.stdout:5/294: rmdir d3 39 2024-02-28T16:31:16.696 INFO:tasks.workunit.client.0.smithi001.stdout:9/824: dread d5/d27/db1/d33/f5d [0,4194304] 0 2024-02-28T16:31:16.696 INFO:tasks.workunit.client.0.smithi001.stdout:2/671: link d0/d2/d3/de/d8a/f6e d0/d2/d3/de/d61/d12/fe0 0 2024-02-28T16:31:16.698 INFO:tasks.workunit.client.1.smithi066.stdout:0/378: creat d2/d9/d10/d23/f7a x:0 0 0 2024-02-28T16:31:16.698 INFO:tasks.workunit.client.0.smithi001.stdout:3/720: mkdir d4/d18/ddb/de8 0 2024-02-28T16:31:16.699 INFO:tasks.workunit.client.1.smithi066.stdout:2/301: rename d0/d12/d2e/d38 to d0/d12/d1a/d23/d5d 0 2024-02-28T16:31:16.699 INFO:tasks.workunit.client.1.smithi066.stdout:8/248: dwrite d0/d6/dc/f41 [0,4194304] 0 2024-02-28T16:31:16.699 INFO:tasks.workunit.client.1.smithi066.stdout:1/315: symlink d2/d27/d2b/d34/l64 0 2024-02-28T16:31:16.702 INFO:tasks.workunit.client.1.smithi066.stdout:4/403: write d1/d23/d25/d43/f5e [666150,69974] 0 2024-02-28T16:31:16.702 INFO:tasks.workunit.client.0.smithi001.stdout:4/752: chown d3/d5/d7/c2f 1943 1 2024-02-28T16:31:16.705 INFO:tasks.workunit.client.1.smithi066.stdout:4/404: fdatasync d1/f2a 0 2024-02-28T16:31:16.706 INFO:tasks.workunit.client.0.smithi001.stdout:1/826: creat d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/f11d x:0 0 0 2024-02-28T16:31:16.707 INFO:tasks.workunit.client.1.smithi066.stdout:8/249: readlink d0/d6/dc/l2e 0 2024-02-28T16:31:16.707 INFO:tasks.workunit.client.1.smithi066.stdout:3/296: dread db/dc/f2b [0,4194304] 0 2024-02-28T16:31:16.707 INFO:tasks.workunit.client.1.smithi066.stdout:0/379: mknod d2/d27/c7b 0 2024-02-28T16:31:16.709 INFO:tasks.workunit.client.0.smithi001.stdout:9/825: truncate d5/d27/d34/f90 5027884 0 2024-02-28T16:31:16.709 INFO:tasks.workunit.client.1.smithi066.stdout:2/302: write d0/f19 [373890,78647] 0 2024-02-28T16:31:16.709 INFO:tasks.workunit.client.1.smithi066.stdout:9/265: creat da/df/d18/d28/f59 x:0 0 0 2024-02-28T16:31:16.711 INFO:tasks.workunit.client.0.smithi001.stdout:2/672: symlink d0/d2/d3/de/d22/le1 0 2024-02-28T16:31:16.711 INFO:tasks.workunit.client.0.smithi001.stdout:9/826: stat d5/d27/db1/d65/lbf 0 2024-02-28T16:31:16.713 INFO:tasks.workunit.client.1.smithi066.stdout:8/250: fsync d0/db/d15/d32/f39 0 2024-02-28T16:31:16.713 INFO:tasks.workunit.client.1.smithi066.stdout:9/266: dread - da/df/d18/d28/f59 zero size 2024-02-28T16:31:16.715 INFO:tasks.workunit.client.1.smithi066.stdout:3/297: creat db/f66 x:0 0 0 2024-02-28T16:31:16.716 INFO:tasks.workunit.client.0.smithi001.stdout:4/753: mknod d3/d5/d7/d1f/d3a/cf0 0 2024-02-28T16:31:16.718 INFO:tasks.workunit.client.1.smithi066.stdout:4/405: write d1/d18/f2d [4489789,122099] 0 2024-02-28T16:31:16.718 INFO:tasks.workunit.client.0.smithi001.stdout:0/699: write d18/d22/d3d/fd6 [2432897,24764] 0 2024-02-28T16:31:16.719 INFO:tasks.workunit.client.1.smithi066.stdout:0/380: rename d2/d27/l37 to d2/d9/d10/l7c 0 2024-02-28T16:31:16.719 INFO:tasks.workunit.client.1.smithi066.stdout:3/298: chown db/dc/d35/f58 208 1 2024-02-28T16:31:16.720 INFO:tasks.workunit.client.1.smithi066.stdout:2/303: mknod d0/d12/d1a/d23/c5e 0 2024-02-28T16:31:16.721 INFO:tasks.workunit.client.0.smithi001.stdout:9/827: readlink d5/d27/d34/dc1/ld6 0 2024-02-28T16:31:16.722 INFO:tasks.workunit.client.0.smithi001.stdout:5/671: link dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/la1 dc/d1a/dba/ldf 0 2024-02-28T16:31:16.722 INFO:tasks.workunit.client.1.smithi066.stdout:5/295: creat d3/d4/f61 x:0 0 0 2024-02-28T16:31:16.722 INFO:tasks.workunit.client.1.smithi066.stdout:0/381: chown d2/d9/d10/f4e 183840 1 2024-02-28T16:31:16.724 INFO:tasks.workunit.client.1.smithi066.stdout:4/406: write d1/d23/d25/f59 [128611,114338] 0 2024-02-28T16:31:16.725 INFO:tasks.workunit.client.0.smithi001.stdout:0/700: read d18/d22/d2b/d36/d46/fb1 [997014,22097] 0 2024-02-28T16:31:16.726 INFO:tasks.workunit.client.0.smithi001.stdout:8/702: dwrite d4/d7/d22/d27/fd4 [0,4194304] 0 2024-02-28T16:31:16.727 INFO:tasks.workunit.client.0.smithi001.stdout:2/673: mknod d0/d2/d3/d36/ce2 0 2024-02-28T16:31:16.728 INFO:tasks.workunit.client.1.smithi066.stdout:9/267: mknod da/df/c5a 0 2024-02-28T16:31:16.730 INFO:tasks.workunit.client.1.smithi066.stdout:7/237: dread dc/de/f13 [0,4194304] 0 2024-02-28T16:31:16.730 INFO:tasks.workunit.client.1.smithi066.stdout:5/296: chown d3/d4/dc/f19 1720 1 2024-02-28T16:31:16.732 INFO:tasks.workunit.client.0.smithi001.stdout:9/828: write d5/d27/db1/d5e/dd3/d109/fd2 [427458,15206] 0 2024-02-28T16:31:16.732 INFO:tasks.workunit.client.0.smithi001.stdout:1/827: truncate d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/f11d 922602 0 2024-02-28T16:31:16.732 INFO:tasks.workunit.client.1.smithi066.stdout:8/251: truncate d0/d6/f7 221441 0 2024-02-28T16:31:16.732 INFO:tasks.workunit.client.1.smithi066.stdout:2/304: rmdir d0/d12/d2e/d3b 39 2024-02-28T16:31:16.736 INFO:tasks.workunit.client.1.smithi066.stdout:0/382: mknod d2/d9/d10/d50/d60/c7d 0 2024-02-28T16:31:16.739 INFO:tasks.workunit.client.0.smithi001.stdout:6/675: dwrite d9/db/d5f/f65 [0,4194304] 0 2024-02-28T16:31:16.739 INFO:tasks.workunit.client.0.smithi001.stdout:0/701: mknod d18/ce7 0 2024-02-28T16:31:16.741 INFO:tasks.workunit.client.1.smithi066.stdout:4/407: mknod d1/d23/d25/c84 0 2024-02-28T16:31:16.741 INFO:tasks.workunit.client.1.smithi066.stdout:0/383: chown d2/d9/d10/d50/d60 7 1 2024-02-28T16:31:16.742 INFO:tasks.workunit.client.1.smithi066.stdout:9/268: mknod da/df/d18/d28/c5b 0 2024-02-28T16:31:16.743 INFO:tasks.workunit.client.1.smithi066.stdout:8/252: write d0/db/d12/d1d/d25/f47 [871263,80707] 0 2024-02-28T16:31:16.743 INFO:tasks.workunit.client.0.smithi001.stdout:4/754: creat d3/d2d/ff1 x:0 0 0 2024-02-28T16:31:16.744 INFO:tasks.workunit.client.1.smithi066.stdout:7/238: mkdir dc/de/d17/d37/d47 0 2024-02-28T16:31:16.744 INFO:tasks.workunit.client.0.smithi001.stdout:7/706: dread d3/d17/f19 [0,4194304] 0 2024-02-28T16:31:16.745 INFO:tasks.workunit.client.0.smithi001.stdout:9/829: mknod d5/d9a/c10e 0 2024-02-28T16:31:16.745 INFO:tasks.workunit.client.0.smithi001.stdout:4/755: dread - d3/d2d/d4c/fea zero size 2024-02-28T16:31:16.748 INFO:tasks.workunit.client.1.smithi066.stdout:3/299: dwrite db/d13/f23 [0,4194304] 0 2024-02-28T16:31:16.751 INFO:tasks.workunit.client.1.smithi066.stdout:8/253: read d0/d6/f9 [1270130,336] 0 2024-02-28T16:31:16.751 INFO:tasks.workunit.client.1.smithi066.stdout:2/305: creat d0/d12/d1a/d23/d29/d21/d2f/f5f x:0 0 0 2024-02-28T16:31:16.753 INFO:tasks.workunit.client.0.smithi001.stdout:5/672: dread dc/d12/d56/fad [0,4194304] 0 2024-02-28T16:31:16.754 INFO:tasks.workunit.client.1.smithi066.stdout:7/239: mknod dc/d29/c48 0 2024-02-28T16:31:16.755 INFO:tasks.workunit.client.1.smithi066.stdout:6/230: sync 2024-02-28T16:31:16.756 INFO:tasks.workunit.client.1.smithi066.stdout:1/316: truncate d2/d27/d2b/d43/f4e 61963 0 2024-02-28T16:31:16.756 INFO:tasks.workunit.client.0.smithi001.stdout:1/828: creat d0/d9/d1a/d2b/d26/f11e x:0 0 0 2024-02-28T16:31:16.757 INFO:tasks.workunit.client.0.smithi001.stdout:3/721: write d4/d18/f69 [911369,124490] 0 2024-02-28T16:31:16.758 INFO:tasks.workunit.client.1.smithi066.stdout:3/300: write db/d13/d14/d24/f34 [4733033,103742] 0 2024-02-28T16:31:16.759 INFO:tasks.workunit.client.0.smithi001.stdout:8/703: write d4/d7/d22/d27/d58/d94/fb5 [409474,37042] 0 2024-02-28T16:31:16.759 INFO:tasks.workunit.client.0.smithi001.stdout:6/676: dread - d9/db/d10/f7b zero size 2024-02-28T16:31:16.763 INFO:tasks.workunit.client.1.smithi066.stdout:0/384: write d2/d9/d10/d23/d6d/f79 [4518362,21486] 0 2024-02-28T16:31:16.763 INFO:tasks.workunit.client.1.smithi066.stdout:6/231: write d2/d2a/f42 [234189,38339] 0 2024-02-28T16:31:16.763 INFO:tasks.workunit.client.1.smithi066.stdout:4/408: rename d1/d18/d4e/f80 to d1/d18/d41/f85 0 2024-02-28T16:31:16.765 INFO:tasks.workunit.client.0.smithi001.stdout:4/756: rename d3/d1c/l65 to d3/d1c/d58/d6e/d9f/lf2 0 2024-02-28T16:31:16.765 INFO:tasks.workunit.client.1.smithi066.stdout:5/297: link d3/d4/dc/dd/d2e/f38 d3/d4/dc/dd/d35/f62 0 2024-02-28T16:31:16.765 INFO:tasks.workunit.client.1.smithi066.stdout:7/240: dread dc/d29/f34 [0,4194304] 0 2024-02-28T16:31:16.770 INFO:tasks.workunit.client.0.smithi001.stdout:1/829: unlink d0/fa9 0 2024-02-28T16:31:16.773 INFO:tasks.workunit.client.1.smithi066.stdout:0/385: chown d2/d9/f75 3 1 2024-02-28T16:31:16.773 INFO:tasks.workunit.client.1.smithi066.stdout:6/232: chown d2/da/d38/f49 88 1 2024-02-28T16:31:16.774 INFO:tasks.workunit.client.1.smithi066.stdout:9/269: truncate da/d3a/f47 2175297 0 2024-02-28T16:31:16.774 INFO:tasks.workunit.client.0.smithi001.stdout:2/674: truncate d0/f63 6957450 0 2024-02-28T16:31:16.774 INFO:tasks.workunit.client.1.smithi066.stdout:1/317: truncate d2/d27/d2b/d37/d51/f60 378092 0 2024-02-28T16:31:16.774 INFO:tasks.workunit.client.1.smithi066.stdout:8/254: rmdir d0/db/d12/d1d/d25 39 2024-02-28T16:31:16.775 INFO:tasks.workunit.client.1.smithi066.stdout:5/298: dread - d3/d4/f61 zero size 2024-02-28T16:31:16.776 INFO:tasks.workunit.client.1.smithi066.stdout:3/301: creat db/d13/d1a/d25/f67 x:0 0 0 2024-02-28T16:31:16.777 INFO:tasks.workunit.client.1.smithi066.stdout:2/306: symlink d0/d12/d1a/d39/l60 0 2024-02-28T16:31:16.777 INFO:tasks.workunit.client.0.smithi001.stdout:2/675: read d0/d2/d3/de/d8a/f70 [2769690,84118] 0 2024-02-28T16:31:16.779 INFO:tasks.workunit.client.0.smithi001.stdout:8/704: creat d4/d32/dd6/fe6 x:0 0 0 2024-02-28T16:31:16.781 INFO:tasks.workunit.client.1.smithi066.stdout:6/233: symlink d2/da/d38/l4d 0 2024-02-28T16:31:16.781 INFO:tasks.workunit.client.1.smithi066.stdout:6/234: readlink d2/da/d19/lf 0 2024-02-28T16:31:16.782 INFO:tasks.workunit.client.0.smithi001.stdout:2/676: read d0/f32 [3088086,106942] 0 2024-02-28T16:31:16.782 INFO:tasks.workunit.client.1.smithi066.stdout:0/386: rename d2/c8 to d2/d9/d10/c7e 0 2024-02-28T16:31:16.782 INFO:tasks.workunit.client.0.smithi001.stdout:9/830: unlink d5/d27/db1/f91 0 2024-02-28T16:31:16.783 INFO:tasks.workunit.client.0.smithi001.stdout:5/673: mknod dc/d12/d41/dd6/ce0 0 2024-02-28T16:31:16.784 INFO:tasks.workunit.client.0.smithi001.stdout:7/707: write d3/f36 [9549561,28169] 0 2024-02-28T16:31:16.785 INFO:tasks.workunit.client.1.smithi066.stdout:4/409: creat d1/d18/f86 x:0 0 0 2024-02-28T16:31:16.786 INFO:tasks.workunit.client.0.smithi001.stdout:1/830: mknod d0/d9/d1a/d2b/d26/d27/dd3/de8/d102/c11f 0 2024-02-28T16:31:16.786 INFO:tasks.workunit.client.0.smithi001.stdout:4/757: fdatasync d3/d1c/d58/f46 0 2024-02-28T16:31:16.788 INFO:tasks.workunit.client.1.smithi066.stdout:2/307: unlink d0/f19 0 2024-02-28T16:31:16.788 INFO:tasks.workunit.client.1.smithi066.stdout:3/302: dread db/dc/d11/d1f/f2e [0,4194304] 0 2024-02-28T16:31:16.790 INFO:tasks.workunit.client.0.smithi001.stdout:6/677: unlink d9/db/d5f/d62/d7e/l95 0 2024-02-28T16:31:16.790 INFO:tasks.workunit.client.1.smithi066.stdout:2/308: chown d0/d12/d1a/d23 10 1 2024-02-28T16:31:16.793 INFO:tasks.workunit.client.1.smithi066.stdout:0/387: creat d2/d9/d10/f7f x:0 0 0 2024-02-28T16:31:16.794 INFO:tasks.workunit.client.0.smithi001.stdout:4/758: rmdir d3/d1c/d58/d88 39 2024-02-28T16:31:16.794 INFO:tasks.workunit.client.0.smithi001.stdout:0/702: getdents d18/d22/d2f 0 2024-02-28T16:31:16.797 INFO:tasks.workunit.client.1.smithi066.stdout:1/318: rename d2/c22 to d2/d27/d2b/d34/c65 0 2024-02-28T16:31:16.799 INFO:tasks.workunit.client.0.smithi001.stdout:4/759: write d3/d1c/d58/d6e/fbb [4227432,126173] 0 2024-02-28T16:31:16.800 INFO:tasks.workunit.client.1.smithi066.stdout:3/303: symlink db/d2a/d44/l68 0 2024-02-28T16:31:16.803 INFO:tasks.workunit.client.1.smithi066.stdout:2/309: creat d0/d12/d2e/d3b/f61 x:0 0 0 2024-02-28T16:31:16.805 INFO:tasks.workunit.client.0.smithi001.stdout:1/831: rename d0/d9/d1a/d2b/d26/d27/d30/d4a/fa5 to d0/d9/d1a/d2b/d26/f120 0 2024-02-28T16:31:16.806 INFO:tasks.workunit.client.1.smithi066.stdout:3/304: write db/d13/d14/d2d/f61 [476234,55195] 0 2024-02-28T16:31:16.806 INFO:tasks.workunit.client.1.smithi066.stdout:2/310: chown d0/d12/d1a/d23 1934800 1 2024-02-28T16:31:16.809 INFO:tasks.workunit.client.1.smithi066.stdout:2/311: dread - d0/d12/d2e/d3b/f61 zero size 2024-02-28T16:31:16.810 INFO:tasks.workunit.client.1.smithi066.stdout:5/299: rename d3/d4/dc/d42/d57/f5e to d3/d1b/d20/f63 0 2024-02-28T16:31:16.812 INFO:tasks.workunit.client.0.smithi001.stdout:9/831: getdents d5/d27/db1/d65 0 2024-02-28T16:31:16.814 INFO:tasks.workunit.client.1.smithi066.stdout:3/305: truncate db/d13/d1a/f4e 10914 0 2024-02-28T16:31:16.815 INFO:tasks.workunit.client.1.smithi066.stdout:1/319: creat d2/d27/d2b/d37/d48/f66 x:0 0 0 2024-02-28T16:31:16.815 INFO:tasks.workunit.client.1.smithi066.stdout:5/300: write d3/d4/dc/d2f/f33 [3405025,48689] 0 2024-02-28T16:31:16.815 INFO:tasks.workunit.client.0.smithi001.stdout:8/705: link d4/d32/d93/ce1 d4/d47/d8b/ce7 0 2024-02-28T16:31:16.815 INFO:tasks.workunit.client.0.smithi001.stdout:5/674: dwrite f5 [0,4194304] 0 2024-02-28T16:31:16.816 INFO:tasks.workunit.client.0.smithi001.stdout:9/832: dread - d5/d27/d42/d48/f103 zero size 2024-02-28T16:31:16.816 INFO:tasks.workunit.client.0.smithi001.stdout:3/722: dwrite d4/d18/faf [0,4194304] 0 2024-02-28T16:31:16.817 INFO:tasks.workunit.client.1.smithi066.stdout:0/388: dwrite d2/d9/f11 [0,4194304] 0 2024-02-28T16:31:16.819 INFO:tasks.workunit.client.0.smithi001.stdout:3/723: chown d4 228 1 2024-02-28T16:31:16.821 INFO:tasks.workunit.client.0.smithi001.stdout:1/832: mkdir d0/d9/d1a/d2b/d26/d105/d121 0 2024-02-28T16:31:16.822 INFO:tasks.workunit.client.0.smithi001.stdout:4/760: truncate d3/d5/f91 3442970 0 2024-02-28T16:31:16.823 INFO:tasks.workunit.client.0.smithi001.stdout:2/677: write d0/d2/d3/de/d61/d12/d6a/d84/fd7 [1283273,89787] 0 2024-02-28T16:31:16.824 INFO:tasks.workunit.client.1.smithi066.stdout:8/255: rename d0/d6/dc/f41 to d0/d6/f4e 0 2024-02-28T16:31:16.825 INFO:tasks.workunit.client.0.smithi001.stdout:9/833: truncate d5/d27/d34/d50/fea 743935 0 2024-02-28T16:31:16.827 INFO:tasks.workunit.client.1.smithi066.stdout:6/235: dread d2/f29 [0,4194304] 0 2024-02-28T16:31:16.828 INFO:tasks.workunit.client.0.smithi001.stdout:9/834: chown d5/d27/db1/d58/d107 30 1 2024-02-28T16:31:16.829 INFO:tasks.workunit.client.0.smithi001.stdout:3/724: creat d4/d6/d7f/d82/dd5/fe9 x:0 0 0 2024-02-28T16:31:16.831 INFO:tasks.workunit.client.1.smithi066.stdout:1/320: mknod d2/d27/d2b/d34/c67 0 2024-02-28T16:31:16.832 INFO:tasks.workunit.client.0.smithi001.stdout:1/833: fdatasync d0/d9/d1a/d2b/d26/d27/d30/f59 0 2024-02-28T16:31:16.835 INFO:tasks.workunit.client.0.smithi001.stdout:4/761: rmdir d3/d5/d7/d1f/d2e 39 2024-02-28T16:31:16.835 INFO:tasks.workunit.client.0.smithi001.stdout:8/706: getdents d4/d7/d22/d27/d58/d94/dbc 0 2024-02-28T16:31:16.837 INFO:tasks.workunit.client.1.smithi066.stdout:3/306: dread db/dc/d11/d1f/f5e [0,4194304] 0 2024-02-28T16:31:16.837 INFO:tasks.workunit.client.1.smithi066.stdout:3/307: read - db/d2a/f42 zero size 2024-02-28T16:31:16.838 INFO:tasks.workunit.client.0.smithi001.stdout:6/678: dwrite d9/d29/f44 [0,4194304] 0 2024-02-28T16:31:16.838 INFO:tasks.workunit.client.0.smithi001.stdout:9/835: chown d5/d27/db1/d5e/dd3/d109/d6f/c8f 46262739 1 2024-02-28T16:31:16.841 INFO:tasks.workunit.client.1.smithi066.stdout:0/389: fdatasync d2/d9/d10/f26 0 2024-02-28T16:31:16.842 INFO:tasks.workunit.client.0.smithi001.stdout:2/678: creat d0/d2/d3/de/d61/d12/fe3 x:0 0 0 2024-02-28T16:31:16.843 INFO:tasks.workunit.client.0.smithi001.stdout:6/679: readlink l4 0 2024-02-28T16:31:16.844 INFO:tasks.workunit.client.1.smithi066.stdout:0/390: chown d2/d27/c7b 6140 1 2024-02-28T16:31:16.845 INFO:tasks.workunit.client.0.smithi001.stdout:8/707: write d4/d32/fd1 [37768,7952] 0 2024-02-28T16:31:16.845 INFO:tasks.workunit.client.0.smithi001.stdout:2/679: chown d0/f17 2943 1 2024-02-28T16:31:16.851 INFO:tasks.workunit.client.1.smithi066.stdout:0/391: dread d2/d9/f75 [0,4194304] 0 2024-02-28T16:31:16.851 INFO:tasks.workunit.client.1.smithi066.stdout:3/308: mknod db/d2a/d44/d48/d5d/c69 0 2024-02-28T16:31:16.854 INFO:tasks.workunit.client.0.smithi001.stdout:4/762: unlink d3/d5/d7/c74 0 2024-02-28T16:31:16.854 INFO:tasks.workunit.client.1.smithi066.stdout:3/309: chown db/d13/d14/d24/l28 0 1 2024-02-28T16:31:16.855 INFO:tasks.workunit.client.0.smithi001.stdout:9/836: rmdir d5/d27/db1/d5e/dd3/d109/d6f/dc8 39 2024-02-28T16:31:16.855 INFO:tasks.workunit.client.1.smithi066.stdout:0/392: stat d2/d9/d10/d50/d67 0 2024-02-28T16:31:16.858 INFO:tasks.workunit.client.1.smithi066.stdout:8/256: fdatasync d0/d8/d3a/f4a 0 2024-02-28T16:31:16.858 INFO:tasks.workunit.client.0.smithi001.stdout:8/708: unlink d4/d7/d22/d27/l2e 0 2024-02-28T16:31:16.862 INFO:tasks.workunit.client.0.smithi001.stdout:4/763: mknod d3/d2d/d4c/d69/cf3 0 2024-02-28T16:31:16.862 INFO:tasks.workunit.client.0.smithi001.stdout:5/675: fdatasync f5 0 2024-02-28T16:31:16.862 INFO:tasks.workunit.client.1.smithi066.stdout:6/236: link d2/d26/f27 d2/d26/f4e 0 2024-02-28T16:31:16.864 INFO:tasks.workunit.client.1.smithi066.stdout:4/410: rename d1/f16 to d1/d18/f87 0 2024-02-28T16:31:16.864 INFO:tasks.workunit.client.1.smithi066.stdout:6/237: dread - d2/f23 zero size 2024-02-28T16:31:16.865 INFO:tasks.workunit.client.0.smithi001.stdout:6/680: creat d9/d29/fe6 x:0 0 0 2024-02-28T16:31:16.866 INFO:tasks.workunit.client.0.smithi001.stdout:8/709: symlink d4/d7/d22/d27/d58/d94/dc9/le8 0 2024-02-28T16:31:16.870 INFO:tasks.workunit.client.1.smithi066.stdout:2/312: dwrite d0/d12/d1a/d39/d40/f45 [0,4194304] 0 2024-02-28T16:31:16.870 INFO:tasks.workunit.client.0.smithi001.stdout:3/725: dwrite d4/d18/f91 [0,4194304] 0 2024-02-28T16:31:16.874 INFO:tasks.workunit.client.0.smithi001.stdout:4/764: fdatasync d3/d5/f34 0 2024-02-28T16:31:16.875 INFO:tasks.workunit.client.1.smithi066.stdout:5/301: write d3/d4/dc/dd/d2e/d18/f2c [1279081,120617] 0 2024-02-28T16:31:16.880 INFO:tasks.workunit.client.1.smithi066.stdout:8/257: rename d0/db/d4d to d0/db/d12/d1d/d25/d4f 0 2024-02-28T16:31:16.882 INFO:tasks.workunit.client.0.smithi001.stdout:4/765: dread - d3/d1c/d58/d6c/d79/dc8/fd3 zero size 2024-02-28T16:31:16.883 INFO:tasks.workunit.client.1.smithi066.stdout:3/310: link db/d2a/l31 db/d2a/d44/l6a 0 2024-02-28T16:31:16.883 INFO:tasks.workunit.client.1.smithi066.stdout:1/321: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:16.885 INFO:tasks.workunit.client.0.smithi001.stdout:9/837: dwrite d5/fb [0,4194304] 0 2024-02-28T16:31:16.886 INFO:tasks.workunit.client.1.smithi066.stdout:3/311: fsync db/d13/f23 0 2024-02-28T16:31:16.887 INFO:tasks.workunit.client.0.smithi001.stdout:8/710: dread d4/d7/d22/d27/d58/d94/db4/fc0 [0,4194304] 0 2024-02-28T16:31:16.889 INFO:tasks.workunit.client.1.smithi066.stdout:6/238: link d2/d26/f2d d2/da/d19/f4f 0 2024-02-28T16:31:16.893 INFO:tasks.workunit.client.1.smithi066.stdout:5/302: creat d3/d4/dc/dd/d35/f64 x:0 0 0 2024-02-28T16:31:16.895 INFO:tasks.workunit.client.0.smithi001.stdout:2/680: dread d0/d2/d3/de/d22/f24 [0,4194304] 0 2024-02-28T16:31:16.896 INFO:tasks.workunit.client.1.smithi066.stdout:2/313: creat d0/d12/f62 x:0 0 0 2024-02-28T16:31:16.898 INFO:tasks.workunit.client.1.smithi066.stdout:0/393: dwrite d2/d9/d10/f26 [0,4194304] 0 2024-02-28T16:31:16.898 INFO:tasks.workunit.client.1.smithi066.stdout:3/312: mknod db/dc/d11/d1f/c6b 0 2024-02-28T16:31:16.898 INFO:tasks.workunit.client.0.smithi001.stdout:9/838: symlink d5/d27/db1/da6/db2/l10f 0 2024-02-28T16:31:16.899 INFO:tasks.workunit.client.0.smithi001.stdout:8/711: mknod d4/d7/da8/ce9 0 2024-02-28T16:31:16.902 INFO:tasks.workunit.client.0.smithi001.stdout:3/726: dwrite d4/d6/f56 [0,4194304] 0 2024-02-28T16:31:16.903 INFO:tasks.workunit.client.1.smithi066.stdout:4/411: write d1/f6 [1542142,14185] 0 2024-02-28T16:31:16.903 INFO:tasks.workunit.client.1.smithi066.stdout:6/239: creat d2/da/d19/f50 x:0 0 0 2024-02-28T16:31:16.905 INFO:tasks.workunit.client.1.smithi066.stdout:3/313: chown db/d13/d14/d2d/f3e 71 1 2024-02-28T16:31:16.905 INFO:tasks.workunit.client.1.smithi066.stdout:2/314: chown d0/d12/d2e/f59 129648 1 2024-02-28T16:31:16.905 INFO:tasks.workunit.client.0.smithi001.stdout:2/681: chown d0/d2/d3/de/d22/f38 1 1 2024-02-28T16:31:16.905 INFO:tasks.workunit.client.0.smithi001.stdout:9/839: write d5/d27/d34/d50/f92 [3163101,53579] 0 2024-02-28T16:31:16.908 INFO:tasks.workunit.client.0.smithi001.stdout:6/681: dwrite d9/db/d19/fd3 [0,4194304] 0 2024-02-28T16:31:16.908 INFO:tasks.workunit.client.0.smithi001.stdout:8/712: unlink d4/d7/cac 0 2024-02-28T16:31:16.909 INFO:tasks.workunit.client.0.smithi001.stdout:4/766: write d3/d5/d7/d1f/d3a/fad [2726270,59618] 0 2024-02-28T16:31:16.911 INFO:tasks.workunit.client.1.smithi066.stdout:2/315: read - d0/d12/d1a/d39/d40/f5b zero size 2024-02-28T16:31:16.914 INFO:tasks.workunit.client.0.smithi001.stdout:4/767: chown d3/d5/d7/d1f/d3a 3464396 1 2024-02-28T16:31:16.917 INFO:tasks.workunit.client.0.smithi001.stdout:8/713: symlink d4/d7/d22/d27/d58/d94/db4/lea 0 2024-02-28T16:31:16.917 INFO:tasks.workunit.client.0.smithi001.stdout:6/682: dread - d9/db/d19/f78 zero size 2024-02-28T16:31:16.921 INFO:tasks.workunit.client.0.smithi001.stdout:8/714: write d4/d7/d22/d27/d4b/f78 [3600546,85356] 0 2024-02-28T16:31:16.924 INFO:tasks.workunit.client.0.smithi001.stdout:4/768: rename d3/d5/f91 to d3/d2d/d83/ff4 0 2024-02-28T16:31:16.925 INFO:tasks.workunit.client.0.smithi001.stdout:6/683: creat d9/db/d5f/d89/fe7 x:0 0 0 2024-02-28T16:31:16.926 INFO:tasks.workunit.client.1.smithi066.stdout:6/240: creat d2/d2c/f51 x:0 0 0 2024-02-28T16:31:16.927 INFO:tasks.workunit.client.1.smithi066.stdout:3/314: creat db/d2a/d44/d48/f6c x:0 0 0 2024-02-28T16:31:16.930 INFO:tasks.workunit.client.1.smithi066.stdout:4/412: symlink d1/d23/d25/d3e/l88 0 2024-02-28T16:31:16.931 INFO:tasks.workunit.client.0.smithi001.stdout:4/769: mknod d3/d5/d7/d1f/cf5 0 2024-02-28T16:31:16.932 INFO:tasks.workunit.client.0.smithi001.stdout:5/676: dread dc/d12/f14 [0,4194304] 0 2024-02-28T16:31:16.932 INFO:tasks.workunit.client.0.smithi001.stdout:4/770: write d3/fd6 [1150511,48817] 0 2024-02-28T16:31:16.934 INFO:tasks.workunit.client.1.smithi066.stdout:8/258: dread d0/d6/f1f [4194304,4194304] 0 2024-02-28T16:31:16.937 INFO:tasks.workunit.client.1.smithi066.stdout:6/241: rename d2/d2a/f42 to d2/da/d38/f52 0 2024-02-28T16:31:16.939 INFO:tasks.workunit.client.0.smithi001.stdout:3/727: dread d4/d6/d7/d78/fd1 [0,4194304] 0 2024-02-28T16:31:16.942 INFO:tasks.workunit.client.1.smithi066.stdout:5/303: dwrite d3/d4/dc/dd/d2e/d18/f4a [0,4194304] 0 2024-02-28T16:31:16.944 INFO:tasks.workunit.client.0.smithi001.stdout:5/677: creat dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/fe1 x:0 0 0 2024-02-28T16:31:16.946 INFO:tasks.workunit.client.0.smithi001.stdout:9/840: dwrite d5/d27/db1/f40 [0,4194304] 0 2024-02-28T16:31:16.948 INFO:tasks.workunit.client.0.smithi001.stdout:3/728: rmdir d4/d6/d7/d78/d81 39 2024-02-28T16:31:16.949 INFO:tasks.workunit.client.1.smithi066.stdout:3/315: rename db/d13/d14/d2d/f61 to db/d2a/d44/f6d 0 2024-02-28T16:31:16.952 INFO:tasks.workunit.client.1.smithi066.stdout:6/242: dread d2/da/d38/f39 [0,4194304] 0 2024-02-28T16:31:16.952 INFO:tasks.workunit.client.1.smithi066.stdout:3/316: fsync db/d2a/f42 0 2024-02-28T16:31:16.952 INFO:tasks.workunit.client.0.smithi001.stdout:4/771: dread d3/d5/d7/f21 [0,4194304] 0 2024-02-28T16:31:16.954 INFO:tasks.workunit.client.0.smithi001.stdout:8/715: dwrite d4/d32/fd1 [0,4194304] 0 2024-02-28T16:31:16.955 INFO:tasks.workunit.client.0.smithi001.stdout:6/684: write f8 [4768944,53743] 0 2024-02-28T16:31:16.956 INFO:tasks.workunit.client.0.smithi001.stdout:3/729: write d4/d6/d7f/d82/fb0 [4668040,69373] 0 2024-02-28T16:31:16.956 INFO:tasks.workunit.client.1.smithi066.stdout:6/243: write d2/da/f22 [321785,116384] 0 2024-02-28T16:31:16.956 INFO:tasks.workunit.client.1.smithi066.stdout:2/316: link d0/f9 d0/d12/d3e/f63 0 2024-02-28T16:31:16.957 INFO:tasks.workunit.client.0.smithi001.stdout:4/772: chown d3/d2d/f63 1188307 1 2024-02-28T16:31:16.959 INFO:tasks.workunit.client.0.smithi001.stdout:9/841: truncate d5/d27/d34/f90 1715320 0 2024-02-28T16:31:16.959 INFO:tasks.workunit.client.1.smithi066.stdout:3/317: readlink db/d13/d14/l46 0 2024-02-28T16:31:16.960 INFO:tasks.workunit.client.0.smithi001.stdout:4/773: dread - d3/d2d/d4c/fea zero size 2024-02-28T16:31:16.960 INFO:tasks.workunit.client.1.smithi066.stdout:8/259: getdents d0/db/d12/d38 0 2024-02-28T16:31:16.963 INFO:tasks.workunit.client.0.smithi001.stdout:3/730: mkdir d4/d6/d7f/d82/dea 0 2024-02-28T16:31:16.967 INFO:tasks.workunit.client.1.smithi066.stdout:3/318: mknod db/d13/d1a/d25/c6e 0 2024-02-28T16:31:16.968 INFO:tasks.workunit.client.0.smithi001.stdout:6/685: creat d9/db/d10/fe8 x:0 0 0 2024-02-28T16:31:16.969 INFO:tasks.workunit.client.0.smithi001.stdout:8/716: stat d4/d7/d22/d27/d58/d94/faa 0 2024-02-28T16:31:16.970 INFO:tasks.workunit.client.0.smithi001.stdout:3/731: chown d4/d1e/fc8 499318 1 2024-02-28T16:31:16.972 INFO:tasks.workunit.client.1.smithi066.stdout:0/394: dread d2/d9/d10/f2c [4194304,4194304] 0 2024-02-28T16:31:16.972 INFO:tasks.workunit.client.1.smithi066.stdout:8/260: creat d0/d6/dc/d16/f50 x:0 0 0 2024-02-28T16:31:16.973 INFO:tasks.workunit.client.1.smithi066.stdout:6/244: link d2/da/d38/f49 d2/d26/f53 0 2024-02-28T16:31:16.974 INFO:tasks.workunit.client.1.smithi066.stdout:0/395: stat d2/d9/d10/d23/d6d/f79 0 2024-02-28T16:31:16.976 INFO:tasks.workunit.client.1.smithi066.stdout:8/261: fdatasync d0/db/d15/f31 0 2024-02-28T16:31:16.980 INFO:tasks.workunit.client.0.smithi001.stdout:8/717: truncate d4/d7/d22/d27/f3a 7799675 0 2024-02-28T16:31:16.980 INFO:tasks.workunit.client.1.smithi066.stdout:0/396: truncate d2/d9/d10/f4e 1370292 0 2024-02-28T16:31:16.982 INFO:tasks.workunit.client.1.smithi066.stdout:0/397: fdatasync d2/d9/f4f 0 2024-02-28T16:31:16.985 INFO:tasks.workunit.client.0.smithi001.stdout:8/718: readlink d4/d7/d22/d27/d58/d94/dbc/le0 0 2024-02-28T16:31:16.987 INFO:tasks.workunit.client.0.smithi001.stdout:5/678: truncate dc/d12/d41/f9c 6185591 0 2024-02-28T16:31:16.987 INFO:tasks.workunit.client.0.smithi001.stdout:6/686: fsync d9/d29/d6e/d53/f7a 0 2024-02-28T16:31:16.987 INFO:tasks.workunit.client.1.smithi066.stdout:0/398: fdatasync d2/d9/d10/d50/d52/f55 0 2024-02-28T16:31:16.988 INFO:tasks.workunit.client.1.smithi066.stdout:3/319: creat db/d13/d14/f6f x:0 0 0 2024-02-28T16:31:16.988 INFO:tasks.workunit.client.1.smithi066.stdout:2/317: dread d0/f37 [0,4194304] 0 2024-02-28T16:31:16.990 INFO:tasks.workunit.client.1.smithi066.stdout:6/245: mkdir d2/da/d19/d54 0 2024-02-28T16:31:16.991 INFO:tasks.workunit.client.1.smithi066.stdout:6/246: chown d2/d2a/d3d 0 1 2024-02-28T16:31:16.992 INFO:tasks.workunit.client.0.smithi001.stdout:4/774: dwrite d3/d2d/d4c/d5e/db0/fba [0,4194304] 0 2024-02-28T16:31:16.993 INFO:tasks.workunit.client.0.smithi001.stdout:6/687: creat d9/d29/fe9 x:0 0 0 2024-02-28T16:31:16.994 INFO:tasks.workunit.client.1.smithi066.stdout:8/262: mknod d0/db/d12/d1d/d25/d4f/c51 0 2024-02-28T16:31:16.995 INFO:tasks.workunit.client.0.smithi001.stdout:9/842: dwrite d5/d4d/f70 [0,4194304] 0 2024-02-28T16:31:16.995 INFO:tasks.workunit.client.0.smithi001.stdout:3/732: dread d4/d18/fbf [0,4194304] 0 2024-02-28T16:31:16.997 INFO:tasks.workunit.client.1.smithi066.stdout:0/399: unlink d2/d27/c7b 0 2024-02-28T16:31:17.002 INFO:tasks.workunit.client.1.smithi066.stdout:6/247: readlink d2/l1f 0 2024-02-28T16:31:17.004 INFO:tasks.workunit.client.0.smithi001.stdout:6/688: rename d9/d29/d6e/d53/c79 to d9/da6/cea 0 2024-02-28T16:31:17.005 INFO:tasks.workunit.client.1.smithi066.stdout:2/318: fdatasync d0/f9 0 2024-02-28T16:31:17.005 INFO:tasks.workunit.client.0.smithi001.stdout:9/843: truncate d5/d27/db1/d5e/dd3/d109/ffe 1330585 0 2024-02-28T16:31:17.006 INFO:tasks.workunit.client.1.smithi066.stdout:8/263: mkdir d0/d52 0 2024-02-28T16:31:17.008 INFO:tasks.workunit.client.0.smithi001.stdout:3/733: mknod d4/d6/ceb 0 2024-02-28T16:31:17.010 INFO:tasks.workunit.client.1.smithi066.stdout:6/248: mknod d2/da/c55 0 2024-02-28T16:31:17.010 INFO:tasks.workunit.client.0.smithi001.stdout:9/844: symlink d5/d27/d34/d50/d9d/l110 0 2024-02-28T16:31:17.011 INFO:tasks.workunit.client.1.smithi066.stdout:0/400: link d2/d27/l4d d2/d9/d10/d50/d52/l80 0 2024-02-28T16:31:17.012 INFO:tasks.workunit.client.0.smithi001.stdout:9/845: fdatasync d5/f30 0 2024-02-28T16:31:17.013 INFO:tasks.workunit.client.1.smithi066.stdout:0/401: chown d2/d9/d10/d23/d6d 19 1 2024-02-28T16:31:17.014 INFO:tasks.workunit.client.1.smithi066.stdout:2/319: readlink d0/d12/d1a/d23/d29/d21/d2f/l56 0 2024-02-28T16:31:17.015 INFO:tasks.workunit.client.1.smithi066.stdout:0/402: chown d2/c42 595 1 2024-02-28T16:31:17.015 INFO:tasks.workunit.client.0.smithi001.stdout:3/734: truncate d4/d18/d5f/fb8 801419 0 2024-02-28T16:31:17.015 INFO:tasks.workunit.client.1.smithi066.stdout:8/264: symlink d0/db/d15/d32/l53 0 2024-02-28T16:31:17.017 INFO:tasks.workunit.client.1.smithi066.stdout:0/403: stat d2/d27/d41/l48 0 2024-02-28T16:31:17.017 INFO:tasks.workunit.client.0.smithi001.stdout:9/846: dread d5/fb8 [0,4194304] 0 2024-02-28T16:31:17.019 INFO:tasks.workunit.client.0.smithi001.stdout:9/847: readlink d5/l13 0 2024-02-28T16:31:17.022 INFO:tasks.workunit.client.0.smithi001.stdout:3/735: rmdir d4/d6/d7f 39 2024-02-28T16:31:17.023 INFO:tasks.workunit.client.0.smithi001.stdout:9/848: dread d5/fb8 [0,4194304] 0 2024-02-28T16:31:17.025 INFO:tasks.workunit.client.1.smithi066.stdout:0/404: rename d2/d9/f40 to d2/d9/d10/d50/d52/f81 0 2024-02-28T16:31:17.027 INFO:tasks.workunit.client.0.smithi001.stdout:3/736: chown d4/d6/cb3 195000 1 2024-02-28T16:31:17.027 INFO:tasks.workunit.client.0.smithi001.stdout:6/689: dread d9/db/d19/f1f [0,4194304] 0 2024-02-28T16:31:17.028 INFO:tasks.workunit.client.0.smithi001.stdout:8/719: dread d4/d7/da8/f91 [0,4194304] 0 2024-02-28T16:31:17.029 INFO:tasks.workunit.client.0.smithi001.stdout:6/690: stat d9/db/d5f/d8b 0 2024-02-28T16:31:17.033 INFO:tasks.workunit.client.0.smithi001.stdout:8/720: chown d4/d7/d22/d27/d4b/f6d 0 1 2024-02-28T16:31:17.033 INFO:tasks.workunit.client.0.smithi001.stdout:3/737: symlink d4/d6/d7/d78/lec 0 2024-02-28T16:31:17.035 INFO:tasks.workunit.client.0.smithi001.stdout:6/691: unlink d9/db/f68 0 2024-02-28T16:31:17.037 INFO:tasks.workunit.client.0.smithi001.stdout:6/692: dread - d9/db/d19/f78 zero size 2024-02-28T16:31:17.039 INFO:tasks.workunit.client.0.smithi001.stdout:3/738: link d4/d6/d7/f43 d4/d97/fed 0 2024-02-28T16:31:17.039 INFO:tasks.workunit.client.1.smithi066.stdout:7/241: sync 2024-02-28T16:31:17.039 INFO:tasks.workunit.client.1.smithi066.stdout:0/405: creat d2/d9/d10/d50/d51/f82 x:0 0 0 2024-02-28T16:31:17.039 INFO:tasks.workunit.client.1.smithi066.stdout:8/265: getdents d0/d6 0 2024-02-28T16:31:17.040 INFO:tasks.workunit.client.1.smithi066.stdout:7/242: chown dc/d29/l39 3907 1 2024-02-28T16:31:17.040 INFO:tasks.workunit.client.0.smithi001.stdout:6/693: creat d9/de/feb x:0 0 0 2024-02-28T16:31:17.043 INFO:tasks.workunit.client.0.smithi001.stdout:4/775: truncate d3/d2d/d83/ff4 3404755 0 2024-02-28T16:31:17.043 INFO:tasks.workunit.client.1.smithi066.stdout:0/406: mknod d2/d9/d10/d23/c83 0 2024-02-28T16:31:17.047 INFO:tasks.workunit.client.1.smithi066.stdout:0/407: readlink d2/d9/d33/l5d 0 2024-02-28T16:31:17.047 INFO:tasks.workunit.client.1.smithi066.stdout:3/320: dwrite db/d13/f19 [4194304,4194304] 0 2024-02-28T16:31:17.047 INFO:tasks.workunit.client.0.smithi001.stdout:3/739: mkdir d4/d6/d7f/dee 0 2024-02-28T16:31:17.050 INFO:tasks.workunit.client.1.smithi066.stdout:2/320: truncate d0/d12/d1a/d39/d40/f55 223491 0 2024-02-28T16:31:17.053 INFO:tasks.workunit.client.1.smithi066.stdout:7/243: mknod dc/c49 0 2024-02-28T16:31:17.054 INFO:tasks.workunit.client.1.smithi066.stdout:3/321: symlink db/d2a/d44/d48/d5d/l70 0 2024-02-28T16:31:17.056 INFO:tasks.workunit.client.0.smithi001.stdout:8/721: truncate d4/fa1 560353 0 2024-02-28T16:31:17.056 INFO:tasks.workunit.client.0.smithi001.stdout:6/694: dwrite d9/db/d10/fe8 [0,4194304] 0 2024-02-28T16:31:17.059 INFO:tasks.workunit.client.0.smithi001.stdout:8/722: chown d4/d7/d22/d27/d8e/dae/faf 4 1 2024-02-28T16:31:17.063 INFO:tasks.workunit.client.1.smithi066.stdout:6/249: dwrite d2/d26/f2d [4194304,4194304] 0 2024-02-28T16:31:17.066 INFO:tasks.workunit.client.1.smithi066.stdout:2/321: read - d0/d12/d1a/d23/d29/d21/d2f/f51 zero size 2024-02-28T16:31:17.066 INFO:tasks.workunit.client.1.smithi066.stdout:7/244: mkdir dc/de/d17/d4a 0 2024-02-28T16:31:17.067 INFO:tasks.workunit.client.0.smithi001.stdout:9/849: rename d5/d27/d34/f90 to d5/d27/db1/d5e/dd3/d109/d6f/f111 0 2024-02-28T16:31:17.072 INFO:tasks.workunit.client.1.smithi066.stdout:8/266: dwrite d0/d6/f9 [0,4194304] 0 2024-02-28T16:31:17.077 INFO:tasks.workunit.client.0.smithi001.stdout:9/850: getdents d5 0 2024-02-28T16:31:17.080 INFO:tasks.workunit.client.0.smithi001.stdout:9/851: mknod d5/d27/d42/def/c112 0 2024-02-28T16:31:17.082 INFO:tasks.workunit.client.1.smithi066.stdout:0/408: dwrite d2/d9/d10/d50/d51/f78 [0,4194304] 0 2024-02-28T16:31:17.082 INFO:tasks.workunit.client.1.smithi066.stdout:3/322: dread db/dc/d11/f22 [0,4194304] 0 2024-02-28T16:31:17.087 INFO:tasks.workunit.client.0.smithi001.stdout:4/776: dwrite d3/fa [0,4194304] 0 2024-02-28T16:31:17.088 INFO:tasks.workunit.client.0.smithi001.stdout:9/852: rmdir d5/d27/db1/d5e/dd3/d109/dd9 0 2024-02-28T16:31:17.089 INFO:tasks.workunit.client.1.smithi066.stdout:6/250: dread d2/f14 [0,4194304] 0 2024-02-28T16:31:17.089 INFO:tasks.workunit.client.1.smithi066.stdout:7/245: creat dc/f4b x:0 0 0 2024-02-28T16:31:17.091 INFO:tasks.workunit.client.1.smithi066.stdout:7/246: chown f4 460642 1 2024-02-28T16:31:17.091 INFO:tasks.workunit.client.1.smithi066.stdout:3/323: mknod db/c71 0 2024-02-28T16:31:17.091 INFO:tasks.workunit.client.1.smithi066.stdout:6/251: chown d2/da/d19/d54 2186 1 2024-02-28T16:31:17.092 INFO:tasks.workunit.client.0.smithi001.stdout:8/723: dread d4/d32/f46 [0,4194304] 0 2024-02-28T16:31:17.092 INFO:tasks.workunit.client.0.smithi001.stdout:6/695: write d9/da6/fb9 [2117681,81900] 0 2024-02-28T16:31:17.093 INFO:tasks.workunit.client.1.smithi066.stdout:6/252: chown d2/da/c3c 76327 1 2024-02-28T16:31:17.094 INFO:tasks.workunit.client.1.smithi066.stdout:2/322: read d0/f31 [7370985,22203] 0 2024-02-28T16:31:17.095 INFO:tasks.workunit.client.1.smithi066.stdout:6/253: dread - d2/d26/f47 zero size 2024-02-28T16:31:17.097 INFO:tasks.workunit.client.0.smithi001.stdout:9/853: creat d5/d27/db1/da6/dcc/f113 x:0 0 0 2024-02-28T16:31:17.097 INFO:tasks.workunit.client.0.smithi001.stdout:9/854: write d5/f12 [3291744,40313] 0 2024-02-28T16:31:17.097 INFO:tasks.workunit.client.1.smithi066.stdout:8/267: dread d0/d6/f1f [4194304,4194304] 0 2024-02-28T16:31:17.097 INFO:tasks.workunit.client.1.smithi066.stdout:0/409: symlink d2/d27/d41/d76/l84 0 2024-02-28T16:31:17.098 INFO:tasks.workunit.client.1.smithi066.stdout:3/324: creat db/dc/d35/f72 x:0 0 0 2024-02-28T16:31:17.099 INFO:tasks.workunit.client.1.smithi066.stdout:7/247: truncate dc/de/d17/d1e/f3c 222806 0 2024-02-28T16:31:17.101 INFO:tasks.workunit.client.1.smithi066.stdout:2/323: chown d0/d12/d1a/d23/l57 455 1 2024-02-28T16:31:17.102 INFO:tasks.workunit.client.0.smithi001.stdout:8/724: creat d4/d32/dd6/feb x:0 0 0 2024-02-28T16:31:17.102 INFO:tasks.workunit.client.0.smithi001.stdout:9/855: truncate d5/d27/db1/d5e/dd3/d109/f8e 389001 0 2024-02-28T16:31:17.106 INFO:tasks.workunit.client.1.smithi066.stdout:0/410: mkdir d2/d85 0 2024-02-28T16:31:17.106 INFO:tasks.workunit.client.0.smithi001.stdout:8/725: chown d4/f43 32216 1 2024-02-28T16:31:17.108 INFO:tasks.workunit.client.1.smithi066.stdout:8/268: rename d0/d6/f4e to d0/db/d12/d38/f54 0 2024-02-28T16:31:17.108 INFO:tasks.workunit.client.1.smithi066.stdout:7/248: dread dc/d29/f34 [0,4194304] 0 2024-02-28T16:31:17.113 INFO:tasks.workunit.client.0.smithi001.stdout:6/696: dwrite d9/d29/d6e/f34 [4194304,4194304] 0 2024-02-28T16:31:17.116 INFO:tasks.workunit.client.0.smithi001.stdout:8/726: link d4/d7/d22/l7f d4/lec 0 2024-02-28T16:31:17.118 INFO:tasks.workunit.client.1.smithi066.stdout:3/325: rename db/d13/d14/l3b to db/d13/d1a/d25/l73 0 2024-02-28T16:31:17.119 INFO:tasks.workunit.client.0.smithi001.stdout:6/697: write d9/d29/d6e/d53/f9e [2113903,89486] 0 2024-02-28T16:31:17.126 INFO:tasks.workunit.client.0.smithi001.stdout:9/856: dwrite d5/d27/db1/fa4 [0,4194304] 0 2024-02-28T16:31:17.130 INFO:tasks.workunit.client.0.smithi001.stdout:4/777: dread d3/d1c/d58/ff [0,4194304] 0 2024-02-28T16:31:17.130 INFO:tasks.workunit.client.1.smithi066.stdout:2/324: dwrite d0/d12/f62 [0,4194304] 0 2024-02-28T16:31:17.131 INFO:tasks.workunit.client.0.smithi001.stdout:6/698: read d9/d29/d6a/f85 [4010043,29910] 0 2024-02-28T16:31:17.131 INFO:tasks.workunit.client.0.smithi001.stdout:9/857: dread - d5/d27/db1/d33/d8a/da8/ddf/fec zero size 2024-02-28T16:31:17.133 INFO:tasks.workunit.client.1.smithi066.stdout:6/254: write d2/d26/f53 [66341,101112] 0 2024-02-28T16:31:17.134 INFO:tasks.workunit.client.0.smithi001.stdout:6/699: chown d9/db/d19/fd3 1 1 2024-02-28T16:31:17.135 INFO:tasks.workunit.client.1.smithi066.stdout:8/269: rmdir d0/d52 0 2024-02-28T16:31:17.137 INFO:tasks.workunit.client.0.smithi001.stdout:6/700: chown l4 58786 1 2024-02-28T16:31:17.137 INFO:tasks.workunit.client.0.smithi001.stdout:9/858: unlink d5/l26 0 2024-02-28T16:31:17.138 INFO:tasks.workunit.client.0.smithi001.stdout:4/778: truncate d3/d5/d7/d1f/dc3/fee 762223 0 2024-02-28T16:31:17.143 INFO:tasks.workunit.client.1.smithi066.stdout:0/411: link d2/d9/d10/d50/d51/l5c d2/d9/l86 0 2024-02-28T16:31:17.143 INFO:tasks.workunit.client.1.smithi066.stdout:7/249: dwrite dc/de/d17/d1e/f2d [0,4194304] 0 2024-02-28T16:31:17.143 INFO:tasks.workunit.client.0.smithi001.stdout:6/701: unlink d9/db/d10/l94 0 2024-02-28T16:31:17.144 INFO:tasks.workunit.client.0.smithi001.stdout:4/779: rename d3/d2d/d4c/d69/ceb to d3/d1c/de8/cf6 0 2024-02-28T16:31:17.159 INFO:tasks.workunit.client.0.smithi001.stdout:8/727: rmdir d4/d7/d22 39 2024-02-28T16:31:17.175 INFO:tasks.workunit.client.0.smithi001.stdout:4/780: dwrite d3/d2d/d83/fb9 [0,4194304] 0 2024-02-28T16:31:17.189 INFO:tasks.workunit.client.1.smithi066.stdout:9/270: sync 2024-02-28T16:31:17.191 INFO:tasks.workunit.client.1.smithi066.stdout:6/255: rename d2/da/c37 to d2/da/d3e/c56 0 2024-02-28T16:31:17.192 INFO:tasks.workunit.client.1.smithi066.stdout:0/412: chown d2/d27/d41/c58 12 1 2024-02-28T16:31:17.192 INFO:tasks.workunit.client.1.smithi066.stdout:8/270: truncate d0/db/d15/f37 428121 0 2024-02-28T16:31:17.193 INFO:tasks.workunit.client.1.smithi066.stdout:3/326: dwrite db/d13/d14/d2d/f3e [0,4194304] 0 2024-02-28T16:31:17.197 INFO:tasks.workunit.client.1.smithi066.stdout:8/271: symlink d0/d6/dc/d16/l55 0 2024-02-28T16:31:17.199 INFO:tasks.workunit.client.1.smithi066.stdout:6/256: rename d2/f10 to d2/d26/f57 0 2024-02-28T16:31:17.202 INFO:tasks.workunit.client.1.smithi066.stdout:6/257: write d2/d2c/f51 [372517,101331] 0 2024-02-28T16:31:17.203 INFO:tasks.workunit.client.1.smithi066.stdout:0/413: rename d2/d27/d41/c58 to d2/d27/c87 0 2024-02-28T16:31:17.209 INFO:tasks.workunit.client.1.smithi066.stdout:0/414: creat d2/d85/f88 x:0 0 0 2024-02-28T16:31:17.211 INFO:tasks.workunit.client.1.smithi066.stdout:6/258: rename d2/lb to d2/d26/l58 0 2024-02-28T16:31:17.213 INFO:tasks.workunit.client.1.smithi066.stdout:7/250: dwrite dc/de/d17/d1e/f20 [0,4194304] 0 2024-02-28T16:31:17.219 INFO:tasks.workunit.client.1.smithi066.stdout:6/259: creat d2/da/d38/f59 x:0 0 0 2024-02-28T16:31:17.220 INFO:tasks.workunit.client.1.smithi066.stdout:6/260: chown d2/d26 469 1 2024-02-28T16:31:17.224 INFO:tasks.workunit.client.1.smithi066.stdout:3/327: dwrite db/d2a/f42 [0,4194304] 0 2024-02-28T16:31:17.225 INFO:tasks.workunit.client.1.smithi066.stdout:6/261: creat d2/d26/f5a x:0 0 0 2024-02-28T16:31:17.226 INFO:tasks.workunit.client.1.smithi066.stdout:8/272: dwrite d0/f13 [0,4194304] 0 2024-02-28T16:31:17.228 INFO:tasks.workunit.client.1.smithi066.stdout:3/328: write db/dc/d35/d3d/f41 [68529,42586] 0 2024-02-28T16:31:17.235 INFO:tasks.workunit.client.1.smithi066.stdout:0/415: dread d2/d9/d10/d23/f2d [0,4194304] 0 2024-02-28T16:31:17.238 INFO:tasks.workunit.client.1.smithi066.stdout:6/262: rename d2/f14 to d2/d2a/f5b 0 2024-02-28T16:31:17.240 INFO:tasks.workunit.client.0.smithi001.stdout:8/728: dread d4/d7/d22/d27/d4b/d77/f7a [0,4194304] 0 2024-02-28T16:31:17.241 INFO:tasks.workunit.client.0.smithi001.stdout:9/859: dwrite d5/d27/db1/d5e/dd3/d109/d6f/fc6 [0,4194304] 0 2024-02-28T16:31:17.242 INFO:tasks.workunit.client.1.smithi066.stdout:0/416: symlink d2/d9/d10/d50/d64/d6a/l89 0 2024-02-28T16:31:17.243 INFO:tasks.workunit.client.0.smithi001.stdout:8/729: readlink d4/d7/d22/l7f 0 2024-02-28T16:31:17.244 INFO:tasks.workunit.client.0.smithi001.stdout:4/781: dwrite d3/d1c/da0/fc7 [0,4194304] 0 2024-02-28T16:31:17.244 INFO:tasks.workunit.client.1.smithi066.stdout:6/263: truncate d2/da/d38/f59 669226 0 2024-02-28T16:31:17.246 INFO:tasks.workunit.client.0.smithi001.stdout:8/730: chown d4/d32/dd6/feb 543774730 1 2024-02-28T16:31:17.249 INFO:tasks.workunit.client.0.smithi001.stdout:9/860: fsync d5/d27/d34/fc4 0 2024-02-28T16:31:17.249 INFO:tasks.workunit.client.1.smithi066.stdout:0/417: unlink d2/d9/c5f 0 2024-02-28T16:31:17.251 INFO:tasks.workunit.client.1.smithi066.stdout:8/273: dwrite d0/d8/d3a/f4a [0,4194304] 0 2024-02-28T16:31:17.252 INFO:tasks.workunit.client.0.smithi001.stdout:8/731: chown d4/d7/lb 12785170 1 2024-02-28T16:31:17.254 INFO:tasks.workunit.client.1.smithi066.stdout:6/264: symlink d2/d26/d32/d46/l5c 0 2024-02-28T16:31:17.255 INFO:tasks.workunit.client.1.smithi066.stdout:8/274: truncate d0/d6/f1b 4554540 0 2024-02-28T16:31:17.260 INFO:tasks.workunit.client.1.smithi066.stdout:8/275: write d0/db/d12/d1d/d25/f47 [996916,17108] 0 2024-02-28T16:31:17.265 INFO:tasks.workunit.client.1.smithi066.stdout:3/329: write db/d13/d14/f15 [258803,16664] 0 2024-02-28T16:31:17.268 INFO:tasks.workunit.client.0.smithi001.stdout:4/782: dwrite d3/d5/fe2 [0,4194304] 0 2024-02-28T16:31:17.268 INFO:tasks.workunit.client.1.smithi066.stdout:0/418: creat d2/d9/d10/d50/d51/f8a x:0 0 0 2024-02-28T16:31:17.268 INFO:tasks.workunit.client.1.smithi066.stdout:3/330: dread - db/d13/d1a/d25/f67 zero size 2024-02-28T16:31:17.269 INFO:tasks.workunit.client.1.smithi066.stdout:6/265: mkdir d2/da/d19/d54/d5d 0 2024-02-28T16:31:17.270 INFO:tasks.workunit.client.0.smithi001.stdout:4/783: chown d3/fd6 1 1 2024-02-28T16:31:17.271 INFO:tasks.workunit.client.1.smithi066.stdout:8/276: unlink d0/f46 0 2024-02-28T16:31:17.271 INFO:tasks.workunit.client.1.smithi066.stdout:0/419: mknod d2/d9/c8b 0 2024-02-28T16:31:17.274 INFO:tasks.workunit.client.1.smithi066.stdout:8/277: chown d0/f24 5 1 2024-02-28T16:31:17.274 INFO:tasks.workunit.client.0.smithi001.stdout:8/732: dwrite d4/d7/d22/d27/d8e/dae/fb0 [0,4194304] 0 2024-02-28T16:31:17.275 INFO:tasks.workunit.client.1.smithi066.stdout:8/278: fsync d0/d6/fe 0 2024-02-28T16:31:17.277 INFO:tasks.workunit.client.1.smithi066.stdout:3/331: rename db/dc/d35/f72 to db/d13/d14/d2d/d57/f74 0 2024-02-28T16:31:17.280 INFO:tasks.workunit.client.1.smithi066.stdout:8/279: read d0/db/d12/d38/f54 [967997,49762] 0 2024-02-28T16:31:17.285 INFO:tasks.workunit.client.1.smithi066.stdout:8/280: chown d0/d6/f1b 1977372 1 2024-02-28T16:31:17.286 INFO:tasks.workunit.client.1.smithi066.stdout:6/266: link d2/da/d19/f20 d2/da/d38/f5e 0 2024-02-28T16:31:17.286 INFO:tasks.workunit.client.1.smithi066.stdout:0/420: symlink d2/d9/d10/d50/d51/l8c 0 2024-02-28T16:31:17.286 INFO:tasks.workunit.client.1.smithi066.stdout:3/332: symlink db/dc/d11/l75 0 2024-02-28T16:31:17.290 INFO:tasks.workunit.client.0.smithi001.stdout:9/861: dread d5/d27/db1/d33/fdb [4194304,4194304] 0 2024-02-28T16:31:17.291 INFO:tasks.workunit.client.0.smithi001.stdout:4/784: dwrite d3/d5/d7/d1f/d20/f95 [0,4194304] 0 2024-02-28T16:31:17.293 INFO:tasks.workunit.client.0.smithi001.stdout:9/862: write d5/d4d/f70 [2334035,49804] 0 2024-02-28T16:31:17.299 INFO:tasks.workunit.client.0.smithi001.stdout:4/785: creat d3/d5/d7/d1f/d20/ff7 x:0 0 0 2024-02-28T16:31:17.310 INFO:tasks.workunit.client.0.smithi001.stdout:9/863: creat d5/d27/db1/d33/d8a/f114 x:0 0 0 2024-02-28T16:31:17.312 INFO:tasks.workunit.client.0.smithi001.stdout:9/864: stat d5/d27/db1/da6/db2 0 2024-02-28T16:31:17.312 INFO:tasks.workunit.client.0.smithi001.stdout:4/786: dread - d3/d5/d7/d1f/d2e/dbe/f9c zero size 2024-02-28T16:31:17.315 INFO:tasks.workunit.client.0.smithi001.stdout:9/865: write d5/d27/fb9 [4377803,80123] 0 2024-02-28T16:31:17.316 INFO:tasks.workunit.client.1.smithi066.stdout:4/413: sync 2024-02-28T16:31:17.316 INFO:tasks.workunit.client.1.smithi066.stdout:1/322: sync 2024-02-28T16:31:17.316 INFO:tasks.workunit.client.1.smithi066.stdout:5/304: sync 2024-02-28T16:31:17.316 INFO:tasks.workunit.client.0.smithi001.stdout:4/787: mkdir d3/df8 0 2024-02-28T16:31:17.320 INFO:tasks.workunit.client.0.smithi001.stdout:9/866: mkdir d5/d27/db1/d33/d115 0 2024-02-28T16:31:17.323 INFO:tasks.workunit.client.0.smithi001.stdout:9/867: chown d5/l13 337294095 1 2024-02-28T16:31:17.325 INFO:tasks.workunit.client.0.smithi001.stdout:4/788: rename d3/d5/d7/d1f/d3a/fcc to d3/d2d/d4c/d69/ff9 0 2024-02-28T16:31:17.328 INFO:tasks.workunit.client.0.smithi001.stdout:9/868: creat d5/d27/db1/d33/d8a/da8/dd1/f116 x:0 0 0 2024-02-28T16:31:17.329 INFO:tasks.workunit.client.1.smithi066.stdout:6/267: creat d2/d26/f5f x:0 0 0 2024-02-28T16:31:17.329 INFO:tasks.workunit.client.1.smithi066.stdout:3/333: fsync db/d13/f23 0 2024-02-28T16:31:17.331 INFO:tasks.workunit.client.1.smithi066.stdout:5/305: read d3/d4/dc/f21 [615257,53976] 0 2024-02-28T16:31:17.333 INFO:tasks.workunit.client.1.smithi066.stdout:4/414: write d1/d23/d25/f6f [4011588,121777] 0 2024-02-28T16:31:17.335 INFO:tasks.workunit.client.1.smithi066.stdout:1/323: write d2/d27/d2b/d43/d46/f63 [477540,122954] 0 2024-02-28T16:31:17.336 INFO:tasks.workunit.client.1.smithi066.stdout:6/268: rmdir d2/d2a/d3d 39 2024-02-28T16:31:17.337 INFO:tasks.workunit.client.1.smithi066.stdout:6/269: readlink d2/da/l2e 0 2024-02-28T16:31:17.339 INFO:tasks.workunit.client.1.smithi066.stdout:0/421: dread d2/d9/f4f [0,4194304] 0 2024-02-28T16:31:17.341 INFO:tasks.workunit.client.1.smithi066.stdout:5/306: unlink d3/c30 0 2024-02-28T16:31:17.343 INFO:tasks.workunit.client.1.smithi066.stdout:4/415: unlink d1/d18/d4e/l60 0 2024-02-28T16:31:17.345 INFO:tasks.workunit.client.0.smithi001.stdout:4/789: truncate d3/f3b 3803514 0 2024-02-28T16:31:17.345 INFO:tasks.workunit.client.0.smithi001.stdout:9/869: write d5/d27/db1/d5e/dd3/d109/d6f/f7d [2644493,91295] 0 2024-02-28T16:31:17.346 INFO:tasks.workunit.client.1.smithi066.stdout:1/324: mkdir d2/d27/d2b/d43/d46/d68 0 2024-02-28T16:31:17.347 INFO:tasks.workunit.client.1.smithi066.stdout:0/422: mkdir d2/d85/d8d 0 2024-02-28T16:31:17.348 INFO:tasks.workunit.client.0.smithi001.stdout:4/790: symlink d3/d1c/d58/d6e/d9f/lfa 0 2024-02-28T16:31:17.350 INFO:tasks.workunit.client.1.smithi066.stdout:5/307: creat d3/d4/dc/dd/d41/f65 x:0 0 0 2024-02-28T16:31:17.350 INFO:tasks.workunit.client.0.smithi001.stdout:9/870: truncate d5/d27/db1/d5e/dd3/d109/d6f/f10d 153595 0 2024-02-28T16:31:17.351 INFO:tasks.workunit.client.1.smithi066.stdout:4/416: unlink d1/d23/l67 0 2024-02-28T16:31:17.352 INFO:tasks.workunit.client.0.smithi001.stdout:4/791: dread d3/d1c/d58/fdf [0,4194304] 0 2024-02-28T16:31:17.355 INFO:tasks.workunit.client.1.smithi066.stdout:0/423: unlink d2/l66 0 2024-02-28T16:31:17.355 INFO:tasks.workunit.client.1.smithi066.stdout:5/308: write d3/d1b/d39/f44 [50107,57658] 0 2024-02-28T16:31:17.357 INFO:tasks.workunit.client.1.smithi066.stdout:3/334: dwrite db/d13/d1a/f4e [0,4194304] 0 2024-02-28T16:31:17.364 INFO:tasks.workunit.client.0.smithi001.stdout:9/871: dwrite d5/d27/db1/d5e/dd3/d109/d6f/f7d [0,4194304] 0 2024-02-28T16:31:17.364 INFO:tasks.workunit.client.1.smithi066.stdout:6/270: dwrite d2/da/f22 [4194304,4194304] 0 2024-02-28T16:31:17.367 INFO:tasks.workunit.client.1.smithi066.stdout:5/309: chown d3/d4/dc/dd/d35/f62 8041 1 2024-02-28T16:31:17.368 INFO:tasks.workunit.client.0.smithi001.stdout:4/792: dwrite d3/d5/d7/d1f/d20/de0/de6/f80 [0,4194304] 0 2024-02-28T16:31:17.371 INFO:tasks.workunit.client.0.smithi001.stdout:9/872: creat d5/d27/db1/d33/f117 x:0 0 0 2024-02-28T16:31:17.372 INFO:tasks.workunit.client.0.smithi001.stdout:4/793: fsync d3/d1c/da0/fb2 0 2024-02-28T16:31:17.377 INFO:tasks.workunit.client.1.smithi066.stdout:0/424: rename d2/d9/d10/d50/d64 to d2/d9/d33/d3f/d8e 0 2024-02-28T16:31:17.377 INFO:tasks.workunit.client.1.smithi066.stdout:1/325: dwrite d2/d18/d19/f24 [4194304,4194304] 0 2024-02-28T16:31:17.381 INFO:tasks.workunit.client.0.smithi001.stdout:4/794: creat d3/d1c/de8/ffb x:0 0 0 2024-02-28T16:31:17.385 INFO:tasks.workunit.client.1.smithi066.stdout:3/335: rename db/d13/d14/d24/d2f to db/dc/d35/d3d/d76 0 2024-02-28T16:31:17.385 INFO:tasks.workunit.client.1.smithi066.stdout:1/326: fsync d2/d18/f4f 0 2024-02-28T16:31:17.387 INFO:tasks.workunit.client.1.smithi066.stdout:0/425: symlink d2/d9/d10/d23/l8f 0 2024-02-28T16:31:17.387 INFO:tasks.workunit.client.0.smithi001.stdout:4/795: mknod d3/d5/d7/d1f/d2e/cfc 0 2024-02-28T16:31:17.388 INFO:tasks.workunit.client.0.smithi001.stdout:9/873: getdents d5/d27/d42/def 0 2024-02-28T16:31:17.390 INFO:tasks.workunit.client.1.smithi066.stdout:1/327: write d2/f5 [940591,97715] 0 2024-02-28T16:31:17.390 INFO:tasks.workunit.client.1.smithi066.stdout:0/426: chown d2/d9/d33/f6b 0 1 2024-02-28T16:31:17.391 INFO:tasks.workunit.client.0.smithi001.stdout:9/874: fsync d5/d27/db1/d5e/dd3/f104 0 2024-02-28T16:31:17.392 INFO:tasks.workunit.client.1.smithi066.stdout:3/336: fsync db/dc/d35/f58 0 2024-02-28T16:31:17.395 INFO:tasks.workunit.client.0.smithi001.stdout:4/796: rename d3/d5/d47/d90/ced to d3/d2d/d81/cfd 0 2024-02-28T16:31:17.396 INFO:tasks.workunit.client.0.smithi001.stdout:9/875: symlink d5/d27/db1/d33/d8a/l118 0 2024-02-28T16:31:17.398 INFO:tasks.workunit.client.0.smithi001.stdout:4/797: unlink d3/d5/faa 0 2024-02-28T16:31:17.399 INFO:tasks.workunit.client.1.smithi066.stdout:5/310: dwrite d3/d1b/d20/f63 [0,4194304] 0 2024-02-28T16:31:17.399 INFO:tasks.workunit.client.1.smithi066.stdout:0/427: rename d2/f22 to d2/d9/d10/d50/d73/f90 0 2024-02-28T16:31:17.403 INFO:tasks.workunit.client.1.smithi066.stdout:4/417: truncate d1/d23/d25/d43/f65 4180974 0 2024-02-28T16:31:17.404 INFO:tasks.workunit.client.0.smithi001.stdout:4/798: mkdir d3/d1c/d58/d6e/dfe 0 2024-02-28T16:31:17.405 INFO:tasks.workunit.client.1.smithi066.stdout:2/325: sync 2024-02-28T16:31:17.406 INFO:tasks.workunit.client.1.smithi066.stdout:8/281: sync 2024-02-28T16:31:17.406 INFO:tasks.workunit.client.1.smithi066.stdout:9/271: sync 2024-02-28T16:31:17.406 INFO:tasks.workunit.client.1.smithi066.stdout:7/251: sync 2024-02-28T16:31:17.407 INFO:tasks.workunit.client.0.smithi001.stdout:4/799: symlink d3/d5/d7/lff 0 2024-02-28T16:31:17.408 INFO:tasks.workunit.client.1.smithi066.stdout:2/326: fsync d0/f44 0 2024-02-28T16:31:17.410 INFO:tasks.workunit.client.1.smithi066.stdout:0/428: mknod d2/d9/d33/c91 0 2024-02-28T16:31:17.410 INFO:tasks.workunit.client.1.smithi066.stdout:8/282: readlink d0/d6/dc/d16/l55 0 2024-02-28T16:31:17.411 INFO:tasks.workunit.client.1.smithi066.stdout:5/311: rename d3/d4/dc/dd/d41/d51 to d3/d4/dc/d42/d57/d66 0 2024-02-28T16:31:17.413 INFO:tasks.workunit.client.0.smithi001.stdout:9/876: dwrite d5/d27/d42/f7e [0,4194304] 0 2024-02-28T16:31:17.413 INFO:tasks.workunit.client.0.smithi001.stdout:4/800: truncate d3/d2d/d4c/f50 1467811 0 2024-02-28T16:31:17.415 INFO:tasks.workunit.client.1.smithi066.stdout:0/429: write d2/d9/d10/f39 [2650626,69569] 0 2024-02-28T16:31:17.415 INFO:tasks.workunit.client.1.smithi066.stdout:1/328: link d2/d18/c41 d2/d27/d2b/d37/c69 0 2024-02-28T16:31:17.417 INFO:tasks.workunit.client.1.smithi066.stdout:7/252: readlink dc/l3b 0 2024-02-28T16:31:17.420 INFO:tasks.workunit.client.1.smithi066.stdout:9/272: creat da/d21/f5c x:0 0 0 2024-02-28T16:31:17.421 INFO:tasks.workunit.client.1.smithi066.stdout:4/418: symlink d1/d7a/l89 0 2024-02-28T16:31:17.421 INFO:tasks.workunit.client.0.smithi001.stdout:9/877: write d5/d27/db1/d5e/dd3/d109/d6f/f10d [129703,95292] 0 2024-02-28T16:31:17.421 INFO:tasks.workunit.client.0.smithi001.stdout:4/801: mkdir d3/d2d/d4c/d69/d100 0 2024-02-28T16:31:17.427 INFO:tasks.workunit.client.1.smithi066.stdout:7/253: write dc/d29/f2e [1499937,95631] 0 2024-02-28T16:31:17.430 INFO:tasks.workunit.client.1.smithi066.stdout:9/273: mknod da/df/c5d 0 2024-02-28T16:31:17.430 INFO:tasks.workunit.client.1.smithi066.stdout:6/271: dwrite d2/f8 [0,4194304] 0 2024-02-28T16:31:17.431 INFO:tasks.workunit.client.0.smithi001.stdout:9/878: unlink d5/ce6 0 2024-02-28T16:31:17.431 INFO:tasks.workunit.client.0.smithi001.stdout:4/802: mknod d3/d1c/d58/c101 0 2024-02-28T16:31:17.432 INFO:tasks.workunit.client.1.smithi066.stdout:1/329: symlink d2/d27/d2b/l6a 0 2024-02-28T16:31:17.434 INFO:tasks.workunit.client.0.smithi001.stdout:9/879: mknod d5/d27/db1/d65/c119 0 2024-02-28T16:31:17.435 INFO:tasks.workunit.client.0.smithi001.stdout:4/803: creat d3/d2d/d4c/d69/d82/f102 x:0 0 0 2024-02-28T16:31:17.438 INFO:tasks.workunit.client.1.smithi066.stdout:4/419: truncate d1/d18/f78 500690 0 2024-02-28T16:31:17.441 INFO:tasks.workunit.client.1.smithi066.stdout:4/420: stat d1/d18/c48 0 2024-02-28T16:31:17.442 INFO:tasks.workunit.client.0.smithi001.stdout:9/880: unlink d5/d27/db1/d5e/dd3/d109/d6f/dc8/lf2 0 2024-02-28T16:31:17.442 INFO:tasks.workunit.client.1.smithi066.stdout:2/327: rename d0/d12/d1a/d23/d29/d21/d2f/f51 to d0/d12/d2e/d3b/f64 0 2024-02-28T16:31:17.442 INFO:tasks.workunit.client.1.smithi066.stdout:0/430: mknod d2/d9/d33/d3f/d8e/d6a/c92 0 2024-02-28T16:31:17.443 INFO:tasks.workunit.client.1.smithi066.stdout:3/337: dwrite db/d2a/d44/f6d [0,4194304] 0 2024-02-28T16:31:17.443 INFO:tasks.workunit.client.0.smithi001.stdout:9/881: write d5/d27/db1/da6/dcc/f113 [638591,58463] 0 2024-02-28T16:31:17.448 INFO:tasks.workunit.client.0.smithi001.stdout:9/882: write d5/d4d/f70 [2702142,29167] 0 2024-02-28T16:31:17.457 INFO:tasks.workunit.client.1.smithi066.stdout:0/431: truncate d2/d9/d10/d50/d52/f69 616001 0 2024-02-28T16:31:17.458 INFO:tasks.workunit.client.1.smithi066.stdout:7/254: write dc/de/d17/d1e/f23 [45752,76903] 0 2024-02-28T16:31:17.460 INFO:tasks.workunit.client.1.smithi066.stdout:2/328: symlink d0/d12/d4a/l65 0 2024-02-28T16:31:17.463 INFO:tasks.workunit.client.1.smithi066.stdout:8/283: dwrite d0/f13 [0,4194304] 0 2024-02-28T16:31:17.464 INFO:tasks.workunit.client.0.smithi001.stdout:7/708: sync 2024-02-28T16:31:17.466 INFO:tasks.workunit.client.0.smithi001.stdout:7/709: dread - d3/d17/d52/d7f/d99/faa zero size 2024-02-28T16:31:17.466 INFO:tasks.workunit.client.0.smithi001.stdout:9/883: dread d5/d27/db1/d5e/dd3/d109/d6f/f111 [0,4194304] 0 2024-02-28T16:31:17.467 INFO:tasks.workunit.client.1.smithi066.stdout:5/312: dwrite d3/d1b/d39/f47 [0,4194304] 0 2024-02-28T16:31:17.469 INFO:tasks.workunit.client.0.smithi001.stdout:7/710: symlink d3/d1c/le5 0 2024-02-28T16:31:17.469 INFO:tasks.workunit.client.0.smithi001.stdout:9/884: chown d5/d27/db1/d5e/dd3/d109/d6f/c82 7866 1 2024-02-28T16:31:17.470 INFO:tasks.workunit.client.0.smithi001.stdout:7/711: fdatasync d3/ff 0 2024-02-28T16:31:17.471 INFO:tasks.workunit.client.0.smithi001.stdout:9/885: chown d5/d27/db1/d65/lf5 726 1 2024-02-28T16:31:17.472 INFO:tasks.workunit.client.0.smithi001.stdout:7/712: write d3/fa [11195748,90163] 0 2024-02-28T16:31:17.473 INFO:tasks.workunit.client.0.smithi001.stdout:9/886: write d5/d27/d34/d50/f92 [2733128,73709] 0 2024-02-28T16:31:17.474 INFO:tasks.workunit.client.1.smithi066.stdout:8/284: truncate d0/db/d12/f4b 102061 0 2024-02-28T16:31:17.474 INFO:tasks.workunit.client.1.smithi066.stdout:9/274: dwrite da/f35 [0,4194304] 0 2024-02-28T16:31:17.474 INFO:tasks.workunit.client.1.smithi066.stdout:2/329: write d0/d12/d2e/f59 [232327,52636] 0 2024-02-28T16:31:17.478 INFO:tasks.workunit.client.0.smithi001.stdout:9/887: mkdir d5/d27/d42/def/d11a 0 2024-02-28T16:31:17.479 INFO:tasks.workunit.client.1.smithi066.stdout:1/330: write d2/d27/f2f [976008,110662] 0 2024-02-28T16:31:17.480 INFO:tasks.workunit.client.0.smithi001.stdout:7/713: dread d3/d17/d52/f77 [0,4194304] 0 2024-02-28T16:31:17.483 INFO:tasks.workunit.client.0.smithi001.stdout:9/888: getdents d5/d27/db1/da6 0 2024-02-28T16:31:17.484 INFO:tasks.workunit.client.1.smithi066.stdout:4/421: truncate d1/d23/d25/d43/f82 583784 0 2024-02-28T16:31:17.486 INFO:tasks.workunit.client.1.smithi066.stdout:3/338: dwrite db/d2a/d44/f4f [0,4194304] 0 2024-02-28T16:31:17.486 INFO:tasks.workunit.client.0.smithi001.stdout:7/714: symlink d3/d17/d52/d7f/d99/dc1/db6/le6 0 2024-02-28T16:31:17.491 INFO:tasks.workunit.client.1.smithi066.stdout:3/339: readlink db/d13/d1a/d33/l56 0 2024-02-28T16:31:17.491 INFO:tasks.workunit.client.1.smithi066.stdout:3/340: stat db/d13/d14/d2d/c36 0 2024-02-28T16:31:17.491 INFO:tasks.workunit.client.0.smithi001.stdout:7/715: mknod d3/d17/d52/d7f/d99/dc1/db6/ce7 0 2024-02-28T16:31:17.493 INFO:tasks.workunit.client.0.smithi001.stdout:7/716: readlink d3/d17/d52/d7f/d99/dc1/l84 0 2024-02-28T16:31:17.538 INFO:tasks.workunit.client.0.smithi001.stdout:9/889: dwrite d5/d27/db1/d33/f3d [0,4194304] 0 2024-02-28T16:31:17.540 INFO:tasks.workunit.client.0.smithi001.stdout:7/717: dwrite d3/f8c [0,4194304] 0 2024-02-28T16:31:17.540 INFO:tasks.workunit.client.1.smithi066.stdout:0/432: rename d2/d9 to d2/d27/d41/d76/d93 0 2024-02-28T16:31:17.542 INFO:tasks.workunit.client.0.smithi001.stdout:9/890: creat d5/d27/d42/d48/dcf/f11b x:0 0 0 2024-02-28T16:31:17.544 INFO:tasks.workunit.client.1.smithi066.stdout:7/255: truncate fa 5905155 0 2024-02-28T16:31:17.545 INFO:tasks.workunit.client.1.smithi066.stdout:4/422: write d1/d23/d25/f6b [802104,92964] 0 2024-02-28T16:31:17.546 INFO:tasks.workunit.client.1.smithi066.stdout:2/330: read d0/d12/d1a/d39/d40/f55 [177190,58051] 0 2024-02-28T16:31:17.546 INFO:tasks.workunit.client.1.smithi066.stdout:9/275: dread da/f35 [0,4194304] 0 2024-02-28T16:31:17.546 INFO:tasks.workunit.client.1.smithi066.stdout:3/341: rename db/d13/d14/d2d/d57 to db/d2a/d44/d48/d52/d77 0 2024-02-28T16:31:17.546 INFO:tasks.workunit.client.0.smithi001.stdout:7/718: symlink d3/d17/d52/d7f/le8 0 2024-02-28T16:31:17.547 INFO:tasks.workunit.client.1.smithi066.stdout:4/423: stat d1/d23/d25/d3e/l61 0 2024-02-28T16:31:17.548 INFO:tasks.workunit.client.1.smithi066.stdout:5/313: link d3/d4/dc/dd/d2e/d18/f1a d3/d4/dc/d56/f67 0 2024-02-28T16:31:17.549 INFO:tasks.workunit.client.0.smithi001.stdout:7/719: chown d3/d17/d52/d7f/d99/dc1/db6 318829 1 2024-02-28T16:31:17.550 INFO:tasks.workunit.client.0.smithi001.stdout:9/891: getdents d5/d27/db1/d33/d8a 0 2024-02-28T16:31:17.550 INFO:tasks.workunit.client.1.smithi066.stdout:9/276: write da/d3a/f13 [5241631,86597] 0 2024-02-28T16:31:17.550 INFO:tasks.workunit.client.1.smithi066.stdout:3/342: chown db/dc/d11/f26 12006444 1 2024-02-28T16:31:17.553 INFO:tasks.workunit.client.1.smithi066.stdout:1/331: mkdir d2/d18/d6b 0 2024-02-28T16:31:17.553 INFO:tasks.workunit.client.1.smithi066.stdout:9/277: read da/f30 [3266171,2779] 0 2024-02-28T16:31:17.553 INFO:tasks.workunit.client.1.smithi066.stdout:4/424: stat d1/d18/d4e/f50 0 2024-02-28T16:31:17.555 INFO:tasks.workunit.client.0.smithi001.stdout:7/720: creat d3/d17/d52/d7f/d99/dc1/fe9 x:0 0 0 2024-02-28T16:31:17.557 INFO:tasks.workunit.client.1.smithi066.stdout:7/256: chown dc/f16 2 1 2024-02-28T16:31:17.558 INFO:tasks.workunit.client.0.smithi001.stdout:9/892: rename d5/d27/d34/f8d to d5/d27/d42/f11c 0 2024-02-28T16:31:17.559 INFO:tasks.workunit.client.1.smithi066.stdout:7/257: fsync dc/d29/f2e 0 2024-02-28T16:31:17.560 INFO:tasks.workunit.client.1.smithi066.stdout:2/331: rename d0/d12/d4a to d0/d12/d2e/d3b/d66 0 2024-02-28T16:31:17.561 INFO:tasks.workunit.client.1.smithi066.stdout:7/258: chown dc/de/d17/l1c 3995974 1 2024-02-28T16:31:17.562 INFO:tasks.workunit.client.0.smithi001.stdout:7/721: link d3/d1c/f37 d3/d17/d52/d7f/d99/dc1/d49/d94/fea 0 2024-02-28T16:31:17.562 INFO:tasks.workunit.client.1.smithi066.stdout:0/433: mkdir d2/d27/d41/d76/d93/d10/d50/d67/d94 0 2024-02-28T16:31:17.563 INFO:tasks.workunit.client.0.smithi001.stdout:9/893: mknod d5/d27/d34/dc1/c11d 0 2024-02-28T16:31:17.565 INFO:tasks.workunit.client.1.smithi066.stdout:2/332: write d0/d12/d2e/f59 [484675,78726] 0 2024-02-28T16:31:17.565 INFO:tasks.workunit.client.1.smithi066.stdout:4/425: chown d1/c30 19 1 2024-02-28T16:31:17.566 INFO:tasks.workunit.client.1.smithi066.stdout:3/343: mkdir db/dc/d11/d1f/d78 0 2024-02-28T16:31:17.566 INFO:tasks.workunit.client.0.smithi001.stdout:7/722: write d3/d17/d52/d7f/d99/fd5 [730562,26892] 0 2024-02-28T16:31:17.568 INFO:tasks.workunit.client.0.smithi001.stdout:9/894: creat d5/d27/db1/d33/d8a/de0/f11e x:0 0 0 2024-02-28T16:31:17.569 INFO:tasks.workunit.client.1.smithi066.stdout:9/278: mkdir da/d4b/d5e 0 2024-02-28T16:31:17.569 INFO:tasks.workunit.client.0.smithi001.stdout:7/723: fdatasync d3/d17/d3d/db3/fd1 0 2024-02-28T16:31:17.569 INFO:tasks.workunit.client.1.smithi066.stdout:1/332: creat d2/d27/d2b/d43/d46/f6c x:0 0 0 2024-02-28T16:31:17.571 INFO:tasks.workunit.client.1.smithi066.stdout:4/426: write d1/d23/d25/d3e/d40/f72 [1794880,14709] 0 2024-02-28T16:31:17.571 INFO:tasks.workunit.client.0.smithi001.stdout:7/724: mknod d3/d17/d3d/d59/d96/ceb 0 2024-02-28T16:31:17.574 INFO:tasks.workunit.client.1.smithi066.stdout:5/314: rename d3/d4/c3a to d3/d1b/d20/c68 0 2024-02-28T16:31:17.575 INFO:tasks.workunit.client.0.smithi001.stdout:7/725: truncate d3/d17/d3d/d59/d6e/fac 2209238 0 2024-02-28T16:31:17.577 INFO:tasks.workunit.client.0.smithi001.stdout:7/726: creat d3/d17/d3d/d59/d6e/fec x:0 0 0 2024-02-28T16:31:17.578 INFO:tasks.workunit.client.1.smithi066.stdout:3/344: mknod db/d13/d1a/d25/c79 0 2024-02-28T16:31:17.578 INFO:tasks.workunit.client.1.smithi066.stdout:5/315: write d3/d1b/f31 [3805280,11761] 0 2024-02-28T16:31:17.579 INFO:tasks.workunit.client.1.smithi066.stdout:2/333: mkdir d0/d12/d1a/d23/d29/d21/d2f/d67 0 2024-02-28T16:31:17.579 INFO:tasks.workunit.client.0.smithi001.stdout:7/727: symlink d3/d17/d61/led 0 2024-02-28T16:31:17.581 INFO:tasks.workunit.client.1.smithi066.stdout:3/345: write db/d13/d14/d24/f34 [1340451,71905] 0 2024-02-28T16:31:17.581 INFO:tasks.workunit.client.0.smithi001.stdout:7/728: write d3/d17/d3d/d5d/fc7 [653028,105005] 0 2024-02-28T16:31:17.583 INFO:tasks.workunit.client.0.smithi001.stdout:7/729: readlink d3/d17/d3d/d59/l95 0 2024-02-28T16:31:17.584 INFO:tasks.workunit.client.1.smithi066.stdout:4/427: getdents d1/d18/d55/d81 0 2024-02-28T16:31:17.585 INFO:tasks.workunit.client.0.smithi001.stdout:7/730: chown d3/d17/d52/d55/c78 0 1 2024-02-28T16:31:17.590 INFO:tasks.workunit.client.1.smithi066.stdout:9/279: link da/f2a da/d3a/f5f 0 2024-02-28T16:31:17.593 INFO:tasks.workunit.client.1.smithi066.stdout:2/334: mkdir d0/d12/d1a/d39/d68 0 2024-02-28T16:31:17.594 INFO:tasks.workunit.client.1.smithi066.stdout:3/346: dread db/d13/d1a/f4e [0,4194304] 0 2024-02-28T16:31:17.595 INFO:tasks.workunit.client.1.smithi066.stdout:5/316: creat d3/d4/dc/dd/d41/d5d/f69 x:0 0 0 2024-02-28T16:31:17.597 INFO:tasks.workunit.client.0.smithi001.stdout:7/731: dwrite d3/d17/fa5 [0,4194304] 0 2024-02-28T16:31:17.597 INFO:tasks.workunit.client.1.smithi066.stdout:6/272: sync 2024-02-28T16:31:17.598 INFO:tasks.workunit.client.1.smithi066.stdout:3/347: read db/dc/f2b [780457,76638] 0 2024-02-28T16:31:17.598 INFO:tasks.workunit.client.1.smithi066.stdout:1/333: getdents d2/d27/d2b/d37 0 2024-02-28T16:31:17.602 INFO:tasks.workunit.client.1.smithi066.stdout:7/259: truncate dc/de/d17/d1e/f28 2583828 0 2024-02-28T16:31:17.608 INFO:tasks.workunit.client.1.smithi066.stdout:4/428: link d1/c3c d1/d23/d25/d77/c8a 0 2024-02-28T16:31:17.609 INFO:tasks.workunit.client.1.smithi066.stdout:3/348: write db/d13/d14/d24/f34 [2224925,16735] 0 2024-02-28T16:31:17.609 INFO:tasks.workunit.client.0.smithi001.stdout:0/703: sync 2024-02-28T16:31:17.611 INFO:tasks.workunit.client.1.smithi066.stdout:1/334: symlink d2/d27/d2b/d37/d51/l6d 0 2024-02-28T16:31:17.613 INFO:tasks.workunit.client.0.smithi001.stdout:0/704: creat d18/d31/d4f/fe8 x:0 0 0 2024-02-28T16:31:17.613 INFO:tasks.workunit.client.0.smithi001.stdout:7/732: write d3/d17/d52/d7f/d99/dc1/f6d [2119618,9409] 0 2024-02-28T16:31:17.618 INFO:tasks.workunit.client.1.smithi066.stdout:3/349: fdatasync db/d13/d14/f15 0 2024-02-28T16:31:17.618 INFO:tasks.workunit.client.0.smithi001.stdout:0/705: write d18/d22/d2b/d36/d48/f55 [6017903,120253] 0 2024-02-28T16:31:17.620 INFO:tasks.workunit.client.1.smithi066.stdout:4/429: creat d1/d23/d25/d43/f8b x:0 0 0 2024-02-28T16:31:17.622 INFO:tasks.workunit.client.0.smithi001.stdout:0/706: read d18/d31/d4f/f60 [1288189,121486] 0 2024-02-28T16:31:17.623 INFO:tasks.workunit.client.0.smithi001.stdout:7/733: dread d3/f36 [8388608,4194304] 0 2024-02-28T16:31:17.626 INFO:tasks.workunit.client.0.smithi001.stdout:0/707: creat d18/d22/d2f/fe9 x:0 0 0 2024-02-28T16:31:17.627 INFO:tasks.workunit.client.1.smithi066.stdout:9/280: dwrite da/df/d18/f3b [0,4194304] 0 2024-02-28T16:31:17.632 INFO:tasks.workunit.client.1.smithi066.stdout:5/317: dwrite d3/d4/dc/f3f [4194304,4194304] 0 2024-02-28T16:31:17.635 INFO:tasks.workunit.client.1.smithi066.stdout:3/350: write db/d2a/d44/d48/f63 [2909382,124517] 0 2024-02-28T16:31:17.639 INFO:tasks.workunit.client.1.smithi066.stdout:6/273: dwrite d2/d26/f53 [0,4194304] 0 2024-02-28T16:31:17.640 INFO:tasks.workunit.client.1.smithi066.stdout:7/260: dwrite dc/f16 [0,4194304] 0 2024-02-28T16:31:17.641 INFO:tasks.workunit.client.1.smithi066.stdout:2/335: dread d0/f31 [4194304,4194304] 0 2024-02-28T16:31:17.646 INFO:tasks.workunit.client.1.smithi066.stdout:6/274: chown d2/f8 919912 1 2024-02-28T16:31:17.647 INFO:tasks.workunit.client.1.smithi066.stdout:2/336: fsync d0/d12/d1a/d39/d40/f5b 0 2024-02-28T16:31:17.647 INFO:tasks.workunit.client.1.smithi066.stdout:7/261: chown dc/de/d17/d1e/d45 2056520761 1 2024-02-28T16:31:17.652 INFO:tasks.workunit.client.0.smithi001.stdout:0/708: truncate d18/d22/f3f 1954449 0 2024-02-28T16:31:17.654 INFO:tasks.workunit.client.1.smithi066.stdout:2/337: truncate d0/d12/d1a/f54 4544799 0 2024-02-28T16:31:17.665 INFO:tasks.workunit.client.0.smithi001.stdout:7/734: dwrite d3/d1c/f51 [0,4194304] 0 2024-02-28T16:31:17.673 INFO:tasks.workunit.client.1.smithi066.stdout:7/262: dwrite dc/d29/f34 [0,4194304] 0 2024-02-28T16:31:17.678 INFO:tasks.workunit.client.0.smithi001.stdout:0/709: dwrite d18/d22/d2f/d1c/fc7 [0,4194304] 0 2024-02-28T16:31:17.682 INFO:tasks.workunit.client.1.smithi066.stdout:8/285: sync 2024-02-28T16:31:17.682 INFO:tasks.workunit.client.1.smithi066.stdout:6/275: dwrite d2/f8 [0,4194304] 0 2024-02-28T16:31:17.683 INFO:tasks.workunit.client.0.smithi001.stdout:0/710: write d18/d22/d2b/d36/d48/f8a [3088819,62560] 0 2024-02-28T16:31:17.684 INFO:tasks.workunit.client.0.smithi001.stdout:7/735: dwrite d3/d17/d61/fd3 [0,4194304] 0 2024-02-28T16:31:17.686 INFO:tasks.workunit.client.0.smithi001.stdout:0/711: dread - d18/d22/d2f/d1c/d79/fc2 zero size 2024-02-28T16:31:17.687 INFO:tasks.workunit.client.1.smithi066.stdout:6/276: chown d2/da/d38/f49 11 1 2024-02-28T16:31:17.689 INFO:tasks.workunit.client.0.smithi001.stdout:0/712: chown d18/d22/l35 25951653 1 2024-02-28T16:31:17.691 INFO:tasks.workunit.client.1.smithi066.stdout:5/318: unlink d3/d4/dc/f1e 0 2024-02-28T16:31:17.692 INFO:tasks.workunit.client.0.smithi001.stdout:0/713: mkdir d18/da9/dea 0 2024-02-28T16:31:17.695 INFO:tasks.workunit.client.1.smithi066.stdout:5/319: chown d3/d4/dc/dd/d2e/l58 467556 1 2024-02-28T16:31:17.706 INFO:tasks.workunit.client.1.smithi066.stdout:7/263: dwrite dc/d29/f44 [0,4194304] 0 2024-02-28T16:31:17.714 INFO:tasks.workunit.client.1.smithi066.stdout:8/286: unlink d0/d8/d3a/f4a 0 2024-02-28T16:31:17.716 INFO:tasks.workunit.client.1.smithi066.stdout:6/277: dwrite d2/d26/f2d [4194304,4194304] 0 2024-02-28T16:31:17.723 INFO:tasks.workunit.client.1.smithi066.stdout:8/287: chown d0/d6/dc/d16/d1a/l33 1774 1 2024-02-28T16:31:17.723 INFO:tasks.workunit.client.1.smithi066.stdout:0/434: sync 2024-02-28T16:31:17.731 INFO:tasks.workunit.client.1.smithi066.stdout:4/430: getdents d1/d18 0 2024-02-28T16:31:17.733 INFO:tasks.workunit.client.0.smithi001.stdout:7/736: write d3/d17/d3d/f4d [1712318,71714] 0 2024-02-28T16:31:17.734 INFO:tasks.workunit.client.0.smithi001.stdout:3/740: sync 2024-02-28T16:31:17.734 INFO:tasks.workunit.client.0.smithi001.stdout:2/682: sync 2024-02-28T16:31:17.735 INFO:tasks.workunit.client.0.smithi001.stdout:5/679: sync 2024-02-28T16:31:17.735 INFO:tasks.workunit.client.0.smithi001.stdout:4/804: sync 2024-02-28T16:31:17.735 INFO:tasks.workunit.client.0.smithi001.stdout:8/733: sync 2024-02-28T16:31:17.735 INFO:tasks.workunit.client.0.smithi001.stdout:1/834: sync 2024-02-28T16:31:17.735 INFO:tasks.workunit.client.0.smithi001.stdout:6/702: sync 2024-02-28T16:31:17.737 INFO:tasks.workunit.client.1.smithi066.stdout:6/278: creat d2/d26/d32/d46/f60 x:0 0 0 2024-02-28T16:31:17.740 INFO:tasks.workunit.client.0.smithi001.stdout:6/703: stat d9/db/fd8 0 2024-02-28T16:31:17.741 INFO:tasks.workunit.client.0.smithi001.stdout:0/714: truncate d18/d22/d2b/d36/d4d/f65 620602 0 2024-02-28T16:31:17.741 INFO:tasks.workunit.client.0.smithi001.stdout:5/680: unlink dc/df/f34 0 2024-02-28T16:31:17.741 INFO:tasks.workunit.client.1.smithi066.stdout:8/288: dwrite d0/db/d15/f31 [0,4194304] 0 2024-02-28T16:31:17.744 INFO:tasks.workunit.client.0.smithi001.stdout:3/741: fdatasync d4/d1e/f27 0 2024-02-28T16:31:17.746 INFO:tasks.workunit.client.1.smithi066.stdout:0/435: creat d2/d27/d41/d76/d93/d10/d50/d60/f95 x:0 0 0 2024-02-28T16:31:17.748 INFO:tasks.workunit.client.0.smithi001.stdout:4/805: chown d3/d5/d7/d1f/d20/l2b 942903331 1 2024-02-28T16:31:17.752 INFO:tasks.workunit.client.1.smithi066.stdout:6/279: write d2/d26/f57 [4835231,90535] 0 2024-02-28T16:31:17.752 INFO:tasks.workunit.client.0.smithi001.stdout:6/704: unlink d9/db/d5f/d89/fe7 0 2024-02-28T16:31:17.753 INFO:tasks.workunit.client.1.smithi066.stdout:6/280: fdatasync d2/d2c/f30 0 2024-02-28T16:31:17.755 INFO:tasks.workunit.client.1.smithi066.stdout:8/289: dread d0/d6/f9 [0,4194304] 0 2024-02-28T16:31:17.757 INFO:tasks.workunit.client.0.smithi001.stdout:1/835: symlink d0/d9/d1a/d2b/d26/d27/d30/d88/l122 0 2024-02-28T16:31:17.758 INFO:tasks.workunit.client.0.smithi001.stdout:2/683: symlink d0/d2/d3/de/le4 0 2024-02-28T16:31:17.762 INFO:tasks.workunit.client.1.smithi066.stdout:4/431: creat d1/d18/f8c x:0 0 0 2024-02-28T16:31:17.763 INFO:tasks.workunit.client.1.smithi066.stdout:4/432: readlink d1/d23/l6e 0 2024-02-28T16:31:17.763 INFO:tasks.workunit.client.1.smithi066.stdout:5/320: getdents d3/d4/dc/dd/d41 0 2024-02-28T16:31:17.769 INFO:tasks.workunit.client.1.smithi066.stdout:8/290: rename d0/d6/dc/d16/d1c/c2c to d0/d8/c56 0 2024-02-28T16:31:17.771 INFO:tasks.workunit.client.1.smithi066.stdout:8/291: read d0/db/d15/f31 [391799,34711] 0 2024-02-28T16:31:17.774 INFO:tasks.workunit.client.0.smithi001.stdout:9/895: sync 2024-02-28T16:31:17.775 INFO:tasks.workunit.client.1.smithi066.stdout:0/436: creat d2/f96 x:0 0 0 2024-02-28T16:31:17.776 INFO:tasks.workunit.client.1.smithi066.stdout:9/281: write da/d3a/f5f [4608124,23713] 0 2024-02-28T16:31:17.781 INFO:tasks.workunit.client.1.smithi066.stdout:9/282: creat da/d21/f60 x:0 0 0 2024-02-28T16:31:17.782 INFO:tasks.workunit.client.1.smithi066.stdout:6/281: getdents d2 0 2024-02-28T16:31:17.782 INFO:tasks.workunit.client.1.smithi066.stdout:5/321: symlink d3/d4/dc/dd/d2e/l6a 0 2024-02-28T16:31:17.786 INFO:tasks.workunit.client.1.smithi066.stdout:9/283: mknod da/d21/c61 0 2024-02-28T16:31:17.787 INFO:tasks.workunit.client.1.smithi066.stdout:9/284: readlink da/d22/l2e 0 2024-02-28T16:31:17.788 INFO:tasks.workunit.client.1.smithi066.stdout:1/335: truncate d2/d27/d2b/d37/d51/f60 132080 0 2024-02-28T16:31:17.796 INFO:tasks.workunit.client.1.smithi066.stdout:9/285: read da/df/d18/d28/f4d [96303,86203] 0 2024-02-28T16:31:17.798 INFO:tasks.workunit.client.1.smithi066.stdout:9/286: unlink c4 0 2024-02-28T16:31:17.799 INFO:tasks.workunit.client.0.smithi001.stdout:8/734: dwrite d4/d7/d22/d27/d4b/d77/f8d [0,4194304] 0 2024-02-28T16:31:17.800 INFO:tasks.workunit.client.1.smithi066.stdout:1/336: creat d2/d18/f6e x:0 0 0 2024-02-28T16:31:17.801 INFO:tasks.workunit.client.1.smithi066.stdout:6/282: rename d2/da/l2e to d2/da/l61 0 2024-02-28T16:31:17.802 INFO:tasks.workunit.client.1.smithi066.stdout:5/322: dwrite d3/d4/f4e [0,4194304] 0 2024-02-28T16:31:17.806 INFO:tasks.workunit.client.1.smithi066.stdout:9/287: read da/fc [725206,70848] 0 2024-02-28T16:31:17.808 INFO:tasks.workunit.client.1.smithi066.stdout:5/323: fsync d3/d1b/d39/f44 0 2024-02-28T16:31:17.809 INFO:tasks.workunit.client.0.smithi001.stdout:3/742: write d4/d18/d5f/fb8 [301180,41944] 0 2024-02-28T16:31:17.810 INFO:tasks.workunit.client.0.smithi001.stdout:0/715: getdents d18/d22/d2b/d95 0 2024-02-28T16:31:17.810 INFO:tasks.workunit.client.0.smithi001.stdout:8/735: chown d4/d7/d22/d27/d58/d94/db4/dd5/fda 7016224 1 2024-02-28T16:31:17.811 INFO:tasks.workunit.client.1.smithi066.stdout:6/283: symlink d2/da/d19/d54/l62 0 2024-02-28T16:31:17.814 INFO:tasks.workunit.client.1.smithi066.stdout:5/324: dread - d3/d4/dc/dd/d2e/d18/f4b zero size 2024-02-28T16:31:17.814 INFO:tasks.workunit.client.1.smithi066.stdout:6/284: chown d2/d2a/c33 0 1 2024-02-28T16:31:17.815 INFO:tasks.workunit.client.0.smithi001.stdout:9/896: rename d5/d27/db1/d33/d8a to d5/d27/db1/d33/d115/d11f 0 2024-02-28T16:31:17.816 INFO:tasks.workunit.client.1.smithi066.stdout:3/351: mkdir db/d7a 0 2024-02-28T16:31:17.817 INFO:tasks.workunit.client.0.smithi001.stdout:1/836: creat d0/d9/d1a/d2b/d26/d27/f123 x:0 0 0 2024-02-28T16:31:17.822 INFO:tasks.workunit.client.0.smithi001.stdout:7/737: dwrite d3/f22 [0,4194304] 0 2024-02-28T16:31:17.822 INFO:tasks.workunit.client.0.smithi001.stdout:3/743: rename d4/d3c to d4/d1e/def 0 2024-02-28T16:31:17.823 INFO:tasks.workunit.client.1.smithi066.stdout:9/288: dread da/d3a/f41 [0,4194304] 0 2024-02-28T16:31:17.825 INFO:tasks.workunit.client.0.smithi001.stdout:6/705: dwrite d9/d29/d6e/d4e/d96/fc0 [0,4194304] 0 2024-02-28T16:31:17.829 INFO:tasks.workunit.client.1.smithi066.stdout:6/285: dread - d2/d2a/d3d/f48 zero size 2024-02-28T16:31:17.830 INFO:tasks.workunit.client.0.smithi001.stdout:2/684: getdents d0/d2/d3/de/d61/d12/d55/db2 0 2024-02-28T16:31:17.835 INFO:tasks.workunit.client.0.smithi001.stdout:9/897: mknod d5/d27/db1/d5e/dd3/d109/df1/c120 0 2024-02-28T16:31:17.835 INFO:tasks.workunit.client.1.smithi066.stdout:9/289: write da/d3a/f41 [366717,63469] 0 2024-02-28T16:31:17.836 INFO:tasks.workunit.client.0.smithi001.stdout:6/706: rmdir d9/db/d19 39 2024-02-28T16:31:17.837 INFO:tasks.workunit.client.1.smithi066.stdout:1/337: dwrite d2/d27/d2b/d37/f49 [4194304,4194304] 0 2024-02-28T16:31:17.838 INFO:tasks.workunit.client.1.smithi066.stdout:8/292: truncate d0/d6/ff 7094657 0 2024-02-28T16:31:17.845 INFO:tasks.workunit.client.1.smithi066.stdout:3/352: dwrite db/d13/d1a/d25/f5f [0,4194304] 0 2024-02-28T16:31:17.846 INFO:tasks.workunit.client.0.smithi001.stdout:6/707: chown d9/d29/d6e/f31 301510 1 2024-02-28T16:31:17.849 INFO:tasks.workunit.client.0.smithi001.stdout:2/685: rename d0/d2/d3/ca2 to d0/d2/d3/de/d61/d12/d55/ce5 0 2024-02-28T16:31:17.850 INFO:tasks.workunit.client.0.smithi001.stdout:3/744: link d4/d1e/d7c/da8/fd8 d4/d6/d7f/ff0 0 2024-02-28T16:31:17.850 INFO:tasks.workunit.client.0.smithi001.stdout:4/806: dread d3/fd6 [0,4194304] 0 2024-02-28T16:31:17.851 INFO:tasks.workunit.client.0.smithi001.stdout:9/898: write d5/f12 [2054808,122037] 0 2024-02-28T16:31:17.852 INFO:tasks.workunit.client.1.smithi066.stdout:1/338: fdatasync d2/d18/d19/f24 0 2024-02-28T16:31:17.853 INFO:tasks.workunit.client.0.smithi001.stdout:7/738: truncate d3/d17/d52/d7f/d99/dc1/d49/d94/fea 3767064 0 2024-02-28T16:31:17.853 INFO:tasks.workunit.client.0.smithi001.stdout:3/745: chown d4/d97/lcc 6 1 2024-02-28T16:31:17.854 INFO:tasks.workunit.client.1.smithi066.stdout:2/338: rmdir d0/d12/d1a/d23 39 2024-02-28T16:31:17.859 INFO:tasks.workunit.client.1.smithi066.stdout:9/290: unlink da/df/d18/d2d/f44 0 2024-02-28T16:31:17.860 INFO:tasks.workunit.client.0.smithi001.stdout:8/736: dread d4/d47/d8b/f8c [4194304,4194304] 0 2024-02-28T16:31:17.861 INFO:tasks.workunit.client.0.smithi001.stdout:2/686: rmdir d0/d2/d3/de/d61/d12/d46 39 2024-02-28T16:31:17.864 INFO:tasks.workunit.client.0.smithi001.stdout:7/739: creat d3/d17/d3d/d42/fee x:0 0 0 2024-02-28T16:31:17.877 INFO:tasks.workunit.client.1.smithi066.stdout:5/325: truncate d3/d4/dc/dd/d2e/d18/f2b 2501279 0 2024-02-28T16:31:17.879 INFO:tasks.workunit.client.1.smithi066.stdout:0/437: dwrite d2/d27/d41/d76/d93/f75 [0,4194304] 0 2024-02-28T16:31:17.881 INFO:tasks.workunit.client.1.smithi066.stdout:4/433: dwrite d1/d18/d4e/f53 [0,4194304] 0 2024-02-28T16:31:17.882 INFO:tasks.workunit.client.1.smithi066.stdout:9/291: creat da/df/d18/d2d/f62 x:0 0 0 2024-02-28T16:31:17.884 INFO:tasks.workunit.client.1.smithi066.stdout:7/264: dwrite dc/de/d17/d1e/f28 [0,4194304] 0 2024-02-28T16:31:17.885 INFO:tasks.workunit.client.0.smithi001.stdout:5/681: sync 2024-02-28T16:31:17.886 INFO:tasks.workunit.client.0.smithi001.stdout:7/740: symlink d3/d17/d52/d7f/d99/dc1/d49/d94/lef 0 2024-02-28T16:31:17.892 INFO:tasks.workunit.client.1.smithi066.stdout:8/293: dread d0/d6/f9 [0,4194304] 0 2024-02-28T16:31:17.894 INFO:tasks.workunit.client.1.smithi066.stdout:1/339: rename d2/d18/d19/c52 to d2/d18/d19/c6f 0 2024-02-28T16:31:17.895 INFO:tasks.workunit.client.1.smithi066.stdout:5/326: write d3/d4/dc/dd/d35/f64 [67985,1253] 0 2024-02-28T16:31:17.896 INFO:tasks.workunit.client.1.smithi066.stdout:3/353: dread db/dc/d11/f22 [0,4194304] 0 2024-02-28T16:31:17.897 INFO:tasks.workunit.client.1.smithi066.stdout:7/265: write dc/de/d17/d1e/f20 [3564325,68806] 0 2024-02-28T16:31:17.899 INFO:tasks.workunit.client.0.smithi001.stdout:9/899: dread d5/d4d/f70 [0,4194304] 0 2024-02-28T16:31:17.901 INFO:tasks.workunit.client.0.smithi001.stdout:3/746: dwrite d4/d1e/d7c/d7d/fde [0,4194304] 0 2024-02-28T16:31:17.902 INFO:tasks.workunit.client.0.smithi001.stdout:1/837: dread d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f93 [0,4194304] 0 2024-02-28T16:31:17.903 INFO:tasks.workunit.client.0.smithi001.stdout:6/708: dwrite d9/db/d5f/d62/d83/f8f [0,4194304] 0 2024-02-28T16:31:17.903 INFO:tasks.workunit.client.0.smithi001.stdout:5/682: dread - dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/fe1 zero size 2024-02-28T16:31:17.904 INFO:tasks.workunit.client.0.smithi001.stdout:7/741: unlink d3/f8c 0 2024-02-28T16:31:17.905 INFO:tasks.workunit.client.1.smithi066.stdout:9/292: rename da/d21 to da/df/d18/d28/d63 0 2024-02-28T16:31:17.906 INFO:tasks.workunit.client.1.smithi066.stdout:8/294: dread d0/db/f27 [0,4194304] 0 2024-02-28T16:31:17.908 INFO:tasks.workunit.client.1.smithi066.stdout:2/339: dread d0/f1e [0,4194304] 0 2024-02-28T16:31:17.909 INFO:tasks.workunit.client.1.smithi066.stdout:0/438: truncate d2/d27/d41/d76/d93/d10/f7f 874333 0 2024-02-28T16:31:17.910 INFO:tasks.workunit.client.0.smithi001.stdout:4/807: dwrite d3/d1c/da0/fd0 [0,4194304] 0 2024-02-28T16:31:17.914 INFO:tasks.workunit.client.1.smithi066.stdout:3/354: rmdir db 39 2024-02-28T16:31:17.915 INFO:tasks.workunit.client.0.smithi001.stdout:0/716: dwrite d18/d22/d2f/d1c/d90/d9f/fe3 [0,4194304] 0 2024-02-28T16:31:17.918 INFO:tasks.workunit.client.1.smithi066.stdout:9/293: write f3 [534688,20277] 0 2024-02-28T16:31:17.919 INFO:tasks.workunit.client.1.smithi066.stdout:1/340: symlink d2/d27/d2b/d37/d48/l70 0 2024-02-28T16:31:17.920 INFO:tasks.workunit.client.0.smithi001.stdout:5/683: read fa [2750257,108437] 0 2024-02-28T16:31:17.922 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:17 smithi001 ceph-mon[28991]: pgmap v52: 65 pgs: 65 active+clean; 2.0 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 50 MiB/s rd, 183 MiB/s wr, 331 op/s 2024-02-28T16:31:17.924 INFO:tasks.workunit.client.0.smithi001.stdout:1/838: mknod d0/d9/d1a/d2b/d26/d105/c124 0 2024-02-28T16:31:17.927 INFO:tasks.workunit.client.0.smithi001.stdout:2/687: write d0/d2/d3/de/d22/f24 [222745,87234] 0 2024-02-28T16:31:17.927 INFO:tasks.workunit.client.0.smithi001.stdout:8/737: write d4/d7/d22/fab [105741,19692] 0 2024-02-28T16:31:17.928 INFO:tasks.workunit.client.1.smithi066.stdout:4/434: getdents d1/d18 0 2024-02-28T16:31:17.931 INFO:tasks.workunit.client.0.smithi001.stdout:7/742: mkdir d3/d17/df0 0 2024-02-28T16:31:17.932 INFO:tasks.workunit.client.1.smithi066.stdout:7/266: dwrite dc/d29/f44 [0,4194304] 0 2024-02-28T16:31:17.934 INFO:tasks.workunit.client.0.smithi001.stdout:7/743: fsync d3/d17/d3d/d42/d8e/f9b 0 2024-02-28T16:31:17.939 INFO:tasks.workunit.client.0.smithi001.stdout:5/684: unlink dc/d12/d56/d96/db9/dc2/dca/d91/cb1 0 2024-02-28T16:31:17.943 INFO:tasks.workunit.client.0.smithi001.stdout:0/717: rmdir d18/d22/d2f/d1c/d90/dcc 0 2024-02-28T16:31:17.945 INFO:tasks.workunit.client.0.smithi001.stdout:4/808: dwrite d3/d2d/d4c/d5e/db0/fd9 [0,4194304] 0 2024-02-28T16:31:17.947 INFO:tasks.workunit.client.0.smithi001.stdout:0/718: write d18/d22/d3d/f50 [1761460,114514] 0 2024-02-28T16:31:17.950 INFO:tasks.workunit.client.1.smithi066.stdout:5/327: mknod d3/d4/dc/d56/c6b 0 2024-02-28T16:31:17.950 INFO:tasks.workunit.client.0.smithi001.stdout:5/685: mkdir dc/d1a/d21/de2 0 2024-02-28T16:31:17.951 INFO:tasks.workunit.client.0.smithi001.stdout:1/839: creat d0/d9/d1a/d2b/d26/d27/d30/d4a/f125 x:0 0 0 2024-02-28T16:31:17.951 INFO:tasks.workunit.client.0.smithi001.stdout:0/719: write d18/d31/d4f/fe8 [861273,83894] 0 2024-02-28T16:31:17.952 INFO:tasks.workunit.client.0.smithi001.stdout:4/809: stat d3/d2d/d4c/d5e/f62 0 2024-02-28T16:31:17.952 INFO:tasks.workunit.client.1.smithi066.stdout:3/355: write db/d13/d14/d24/f34 [5534598,23011] 0 2024-02-28T16:31:17.955 INFO:tasks.workunit.client.1.smithi066.stdout:1/341: readlink l1 0 2024-02-28T16:31:17.957 INFO:tasks.workunit.client.0.smithi001.stdout:4/810: mknod d3/d5/d7/d1f/d20/de0/c103 0 2024-02-28T16:31:17.961 INFO:tasks.workunit.client.0.smithi001.stdout:1/840: creat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/f126 x:0 0 0 2024-02-28T16:31:17.962 INFO:tasks.workunit.client.1.smithi066.stdout:4/435: fsync d1/d18/f87 0 2024-02-28T16:31:17.963 INFO:tasks.workunit.client.0.smithi001.stdout:4/811: mknod d3/d1c/d58/d6e/c104 0 2024-02-28T16:31:17.964 INFO:tasks.workunit.client.1.smithi066.stdout:0/439: fdatasync d2/d27/d41/d76/d93/d10/d50/d51/f5a 0 2024-02-28T16:31:17.964 INFO:tasks.workunit.client.0.smithi001.stdout:4/812: fsync d3/f78 0 2024-02-28T16:31:17.964 INFO:tasks.workunit.client.1.smithi066.stdout:3/356: truncate db/d2a/f65 755183 0 2024-02-28T16:31:17.965 INFO:tasks.workunit.client.0.smithi001.stdout:1/841: write d0/d9/d1a/d2b/dfe/f109 [3789451,107167] 0 2024-02-28T16:31:17.967 INFO:tasks.workunit.client.1.smithi066.stdout:9/294: dread da/d4b/f54 [0,4194304] 0 2024-02-28T16:31:17.968 INFO:tasks.workunit.client.1.smithi066.stdout:6/286: dread d2/d2a/f2f [0,4194304] 0 2024-02-28T16:31:17.969 INFO:tasks.workunit.client.1.smithi066.stdout:8/295: dwrite d0/d6/f9 [4194304,4194304] 0 2024-02-28T16:31:17.970 INFO:tasks.workunit.client.0.smithi001.stdout:7/744: getdents d3 0 2024-02-28T16:31:17.974 INFO:tasks.workunit.client.1.smithi066.stdout:1/342: fsync d2/d27/f5b 0 2024-02-28T16:31:17.976 INFO:tasks.workunit.client.0.smithi001.stdout:1/842: rename d0/d9/d1a/fbe to d0/d9/d1a/d2b/d26/d27/dd3/de8/f127 0 2024-02-28T16:31:17.977 INFO:tasks.workunit.client.1.smithi066.stdout:2/340: link d0/d12/c18 d0/d12/d1a/d23/d29/d21/d2f/c69 0 2024-02-28T16:31:17.978 INFO:tasks.workunit.client.0.smithi001.stdout:4/813: truncate d3/d1c/d58/d6e/d9f/fe3 1332768 0 2024-02-28T16:31:17.979 INFO:tasks.workunit.client.0.smithi001.stdout:9/900: write d5/d27/db1/d5e/dd3/d109/d6f/f87 [381337,49188] 0 2024-02-28T16:31:17.980 INFO:tasks.workunit.client.1.smithi066.stdout:9/295: readlink da/d22/l24 0 2024-02-28T16:31:17.980 INFO:tasks.workunit.client.1.smithi066.stdout:0/440: chown d2/d27/d41/d76/d93/d10/d50/d52 300957 1 2024-02-28T16:31:17.985 INFO:tasks.workunit.client.1.smithi066.stdout:4/436: unlink d1/d23/d25/d3e/d40/f64 0 2024-02-28T16:31:17.991 INFO:tasks.workunit.client.1.smithi066.stdout:3/357: symlink db/d2a/d44/d48/l7b 0 2024-02-28T16:31:17.992 INFO:tasks.workunit.client.1.smithi066.stdout:8/296: rmdir d0/d8/d3a 39 2024-02-28T16:31:17.993 INFO:tasks.workunit.client.1.smithi066.stdout:0/441: chown d2/d85/f88 8269427 1 2024-02-28T16:31:17.993 INFO:tasks.workunit.client.0.smithi001.stdout:4/814: fsync d3/d2d/d4c/f50 0 2024-02-28T16:31:17.996 INFO:tasks.workunit.client.1.smithi066.stdout:1/343: read d2/d18/f21 [1810886,53182] 0 2024-02-28T16:31:17.996 INFO:tasks.workunit.client.0.smithi001.stdout:0/720: dread d18/d22/d2f/d1c/d79/f8e [0,4194304] 0 2024-02-28T16:31:17.996 INFO:tasks.workunit.client.1.smithi066.stdout:9/296: truncate da/df/d18/d28/d63/f5c 765430 0 2024-02-28T16:31:17.998 INFO:tasks.workunit.client.0.smithi001.stdout:1/843: dwrite d0/d9/f5c [0,4194304] 0 2024-02-28T16:31:18.001 INFO:tasks.workunit.client.0.smithi001.stdout:2/688: truncate d0/d2/d3/de/d22/f48 2520085 0 2024-02-28T16:31:18.001 INFO:tasks.workunit.client.0.smithi001.stdout:9/901: dwrite d5/d27/db1/f5f [0,4194304] 0 2024-02-28T16:31:18.003 INFO:tasks.workunit.client.0.smithi001.stdout:4/815: symlink d3/d5/d7/d1f/d20/de0/l105 0 2024-02-28T16:31:18.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:17 smithi066 ceph-mon[36236]: pgmap v52: 65 pgs: 65 active+clean; 2.0 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 50 MiB/s rd, 183 MiB/s wr, 331 op/s 2024-02-28T16:31:18.004 INFO:tasks.workunit.client.1.smithi066.stdout:5/328: dwrite d3/d4/f4e [0,4194304] 0 2024-02-28T16:31:18.004 INFO:tasks.workunit.client.1.smithi066.stdout:6/287: dread d2/da/d19/f1d [0,4194304] 0 2024-02-28T16:31:18.005 INFO:tasks.workunit.client.0.smithi001.stdout:1/844: read d0/d9/d1a/d2b/f4d [1498054,52962] 0 2024-02-28T16:31:18.006 INFO:tasks.workunit.client.1.smithi066.stdout:7/267: dwrite dc/f16 [4194304,4194304] 0 2024-02-28T16:31:18.006 INFO:tasks.workunit.client.1.smithi066.stdout:4/437: creat d1/d23/d25/d77/f8d x:0 0 0 2024-02-28T16:31:18.006 INFO:tasks.workunit.client.0.smithi001.stdout:5/686: write dc/f1d [4573986,99393] 0 2024-02-28T16:31:18.008 INFO:tasks.workunit.client.1.smithi066.stdout:3/358: mkdir db/d7a/d7c 0 2024-02-28T16:31:18.008 INFO:tasks.workunit.client.0.smithi001.stdout:0/721: creat d18/d31/feb x:0 0 0 2024-02-28T16:31:18.009 INFO:tasks.workunit.client.1.smithi066.stdout:8/297: write d0/db/d15/f37 [822604,49673] 0 2024-02-28T16:31:18.009 INFO:tasks.workunit.client.0.smithi001.stdout:9/902: mknod d5/d27/d42/c121 0 2024-02-28T16:31:18.011 INFO:tasks.workunit.client.0.smithi001.stdout:5/687: fdatasync dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/f82 0 2024-02-28T16:31:18.012 INFO:tasks.workunit.client.0.smithi001.stdout:4/816: creat d3/f106 x:0 0 0 2024-02-28T16:31:18.015 INFO:tasks.workunit.client.0.smithi001.stdout:0/722: creat d18/d22/d76/fec x:0 0 0 2024-02-28T16:31:18.016 INFO:tasks.workunit.client.1.smithi066.stdout:5/329: creat d3/d4/dc/d42/d57/d66/f6c x:0 0 0 2024-02-28T16:31:18.016 INFO:tasks.workunit.client.1.smithi066.stdout:8/298: readlink d0/d6/dc/l36 0 2024-02-28T16:31:18.017 INFO:tasks.workunit.client.1.smithi066.stdout:4/438: rmdir d1/d18/d4e 39 2024-02-28T16:31:18.017 INFO:tasks.workunit.client.0.smithi001.stdout:9/903: unlink d5/d27/db1/d5e/dd3/d109/l62 0 2024-02-28T16:31:18.018 INFO:tasks.workunit.client.1.smithi066.stdout:3/359: chown db/d13/d1a/d25/l73 1 1 2024-02-28T16:31:18.019 INFO:tasks.workunit.client.1.smithi066.stdout:0/442: symlink d2/d27/d41/d76/d93/d33/d3f/d8e/l97 0 2024-02-28T16:31:18.029 INFO:tasks.workunit.client.0.smithi001.stdout:9/904: write d5/d27/db1/d5e/dd3/d109/d6f/dc8/f106 [903607,18676] 0 2024-02-28T16:31:18.030 INFO:tasks.workunit.client.1.smithi066.stdout:5/330: write d3/d4/dc/f19 [612962,39670] 0 2024-02-28T16:31:18.030 INFO:tasks.workunit.client.0.smithi001.stdout:1/845: getdents d0/d9/d1a/d2b/d26/d27/d30 0 2024-02-28T16:31:18.031 INFO:tasks.workunit.client.0.smithi001.stdout:1/846: fsync d0/d9/f85 0 2024-02-28T16:31:18.032 INFO:tasks.workunit.client.0.smithi001.stdout:9/905: chown d5/d27/d42 360779 1 2024-02-28T16:31:18.032 INFO:tasks.workunit.client.1.smithi066.stdout:2/341: dwrite d0/f1f [0,4194304] 0 2024-02-28T16:31:18.033 INFO:tasks.workunit.client.0.smithi001.stdout:7/745: dread d3/f10 [0,4194304] 0 2024-02-28T16:31:18.034 INFO:tasks.workunit.client.0.smithi001.stdout:5/688: truncate dc/d12/d56/d96/db9/dc2/dca/f77 415129 0 2024-02-28T16:31:18.034 INFO:tasks.workunit.client.0.smithi001.stdout:0/723: creat d18/d22/fed x:0 0 0 2024-02-28T16:31:18.034 INFO:tasks.workunit.client.1.smithi066.stdout:0/443: write d2/d27/d41/d76/d93/d10/d50/d51/f8a [197270,126651] 0 2024-02-28T16:31:18.038 INFO:tasks.workunit.client.0.smithi001.stdout:0/724: rename d18/d22/d2b to d18/d22/d2b/d5f/dee 22 2024-02-28T16:31:18.039 INFO:tasks.workunit.client.1.smithi066.stdout:2/342: readlink d0/d12/d1a/d23/d5d/l52 0 2024-02-28T16:31:18.040 INFO:tasks.workunit.client.0.smithi001.stdout:0/725: write d18/d22/d3d/f5e [8020669,75077] 0 2024-02-28T16:31:18.040 INFO:tasks.workunit.client.0.smithi001.stdout:7/746: symlink d3/d17/d52/d7f/lf1 0 2024-02-28T16:31:18.044 INFO:tasks.workunit.client.0.smithi001.stdout:7/747: write d3/d17/d3d/d42/fe1 [379400,23420] 0 2024-02-28T16:31:18.044 INFO:tasks.workunit.client.0.smithi001.stdout:0/726: readlink d18/d22/l53 0 2024-02-28T16:31:18.045 INFO:tasks.workunit.client.1.smithi066.stdout:6/288: mkdir d2/da/d19/d54/d5d/d63 0 2024-02-28T16:31:18.045 INFO:tasks.workunit.client.1.smithi066.stdout:3/360: mkdir db/d2a/d44/d48/d7d 0 2024-02-28T16:31:18.045 INFO:tasks.workunit.client.1.smithi066.stdout:9/297: creat da/f64 x:0 0 0 2024-02-28T16:31:18.045 INFO:tasks.workunit.client.1.smithi066.stdout:5/331: creat d3/d4/dc/d42/f6d x:0 0 0 2024-02-28T16:31:18.046 INFO:tasks.workunit.client.1.smithi066.stdout:2/343: symlink d0/d12/d2e/d3b/l6a 0 2024-02-28T16:31:18.048 INFO:tasks.workunit.client.1.smithi066.stdout:9/298: chown da 1652789 1 2024-02-28T16:31:18.048 INFO:tasks.workunit.client.0.smithi001.stdout:7/748: dread - d3/d17/d3d/d42/fee zero size 2024-02-28T16:31:18.050 INFO:tasks.workunit.client.1.smithi066.stdout:8/299: dwrite d0/f13 [0,4194304] 0 2024-02-28T16:31:18.052 INFO:tasks.workunit.client.1.smithi066.stdout:2/344: write d0/d12/d2e/d3b/f61 [735335,46178] 0 2024-02-28T16:31:18.053 INFO:tasks.workunit.client.0.smithi001.stdout:0/727: rename d18/d22/d57/c84 to d18/da9/dc9/cef 0 2024-02-28T16:31:18.054 INFO:tasks.workunit.client.1.smithi066.stdout:6/289: creat d2/da/d19/d54/d5d/f64 x:0 0 0 2024-02-28T16:31:18.056 INFO:tasks.workunit.client.0.smithi001.stdout:5/689: dwrite dc/d1a/d7d/d87/f8f [0,4194304] 0 2024-02-28T16:31:18.058 INFO:tasks.workunit.client.1.smithi066.stdout:6/290: stat d2/f8 0 2024-02-28T16:31:18.059 INFO:tasks.workunit.client.0.smithi001.stdout:5/690: rename dc to dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/de3 22 2024-02-28T16:31:18.059 INFO:tasks.workunit.client.1.smithi066.stdout:0/444: getdents d2/d27/d41/d76/d93/d10/d23/d6d 0 2024-02-28T16:31:18.062 INFO:tasks.workunit.client.1.smithi066.stdout:8/300: unlink d0/d6/f9 0 2024-02-28T16:31:18.064 INFO:tasks.workunit.client.1.smithi066.stdout:6/291: chown d2/d2c/f4a 61 1 2024-02-28T16:31:18.065 INFO:tasks.workunit.client.1.smithi066.stdout:2/345: mknod d0/d12/d1a/d23/d29/d21/d2f/c6b 0 2024-02-28T16:31:18.068 INFO:tasks.workunit.client.0.smithi001.stdout:0/728: dwrite d18/d5a/fbe [0,4194304] 0 2024-02-28T16:31:18.071 INFO:tasks.workunit.client.1.smithi066.stdout:2/346: chown d0/d12/d1a/d23/d29/d21/d2f/c6b 211 1 2024-02-28T16:31:18.071 INFO:tasks.workunit.client.0.smithi001.stdout:0/729: creat d18/d22/d2f/d1c/d79/ff0 x:0 0 0 2024-02-28T16:31:18.073 INFO:tasks.workunit.client.0.smithi001.stdout:0/730: dread - d18/d31/feb zero size 2024-02-28T16:31:18.073 INFO:tasks.workunit.client.1.smithi066.stdout:4/439: fsync d1/d23/d25/d77/f8d 0 2024-02-28T16:31:18.074 INFO:tasks.workunit.client.1.smithi066.stdout:2/347: write d0/f44 [74301,41754] 0 2024-02-28T16:31:18.074 INFO:tasks.workunit.client.1.smithi066.stdout:5/332: link d3/l5a d3/d4/dc/dd/d2e/d18/l6e 0 2024-02-28T16:31:18.074 INFO:tasks.workunit.client.1.smithi066.stdout:3/361: dwrite db/d13/f23 [0,4194304] 0 2024-02-28T16:31:18.079 INFO:tasks.workunit.client.1.smithi066.stdout:2/348: fsync d0/d12/d1a/d39/d40/f5b 0 2024-02-28T16:31:18.079 INFO:tasks.workunit.client.1.smithi066.stdout:8/301: symlink d0/d6/dc/d16/d4c/d42/l57 0 2024-02-28T16:31:18.080 INFO:tasks.workunit.client.1.smithi066.stdout:3/362: fdatasync db/d13/d14/f6f 0 2024-02-28T16:31:18.081 INFO:tasks.workunit.client.1.smithi066.stdout:6/292: truncate d2/da/d38/f59 759026 0 2024-02-28T16:31:18.085 INFO:tasks.workunit.client.1.smithi066.stdout:6/293: chown d2/da/c3c 253346968 1 2024-02-28T16:31:18.088 INFO:tasks.workunit.client.1.smithi066.stdout:4/440: fsync d1/d18/f78 0 2024-02-28T16:31:18.096 INFO:tasks.workunit.client.1.smithi066.stdout:6/294: dread d2/da/d38/f59 [0,4194304] 0 2024-02-28T16:31:18.098 INFO:tasks.workunit.client.1.smithi066.stdout:2/349: symlink d0/d12/d1a/d39/d40/l6c 0 2024-02-28T16:31:18.099 INFO:tasks.workunit.client.1.smithi066.stdout:5/333: read d3/d4/dc/d2f/f33 [856278,96243] 0 2024-02-28T16:31:18.104 INFO:tasks.workunit.client.1.smithi066.stdout:8/302: truncate d0/d6/ff 654162 0 2024-02-28T16:31:18.104 INFO:tasks.workunit.client.1.smithi066.stdout:7/268: dread dc/de/d17/d1e/f23 [0,4194304] 0 2024-02-28T16:31:18.104 INFO:tasks.workunit.client.1.smithi066.stdout:2/350: chown d0/d12/d3e/f63 219081307 1 2024-02-28T16:31:18.104 INFO:tasks.workunit.client.1.smithi066.stdout:0/445: dwrite d2/d27/d41/d76/d93/d10/d50/d51/f78 [0,4194304] 0 2024-02-28T16:31:18.108 INFO:tasks.workunit.client.1.smithi066.stdout:4/441: mknod d1/d7a/c8e 0 2024-02-28T16:31:18.108 INFO:tasks.workunit.client.1.smithi066.stdout:7/269: chown dc/d29/l39 11058383 1 2024-02-28T16:31:18.108 INFO:tasks.workunit.client.1.smithi066.stdout:6/295: mknod d2/d26/d32/c65 0 2024-02-28T16:31:18.108 INFO:tasks.workunit.client.1.smithi066.stdout:5/334: truncate d3/d4/dc/f19 1708110 0 2024-02-28T16:31:18.110 INFO:tasks.workunit.client.1.smithi066.stdout:6/296: write f1 [2306005,84053] 0 2024-02-28T16:31:18.113 INFO:tasks.workunit.client.1.smithi066.stdout:6/297: fsync d2/d26/f53 0 2024-02-28T16:31:18.120 INFO:tasks.workunit.client.1.smithi066.stdout:8/303: rename d0/d8/d3a to d0/d6/dc/d16/d1c/d58 0 2024-02-28T16:31:18.125 INFO:tasks.workunit.client.1.smithi066.stdout:2/351: mknod d0/d12/d1a/c6d 0 2024-02-28T16:31:18.125 INFO:tasks.workunit.client.1.smithi066.stdout:5/335: link d3/d4/f61 d3/d4/dc/dd/d2e/f6f 0 2024-02-28T16:31:18.126 INFO:tasks.workunit.client.1.smithi066.stdout:8/304: write d0/db/d15/d32/f39 [7251480,129039] 0 2024-02-28T16:31:18.127 INFO:tasks.workunit.client.1.smithi066.stdout:6/298: symlink d2/d2a/l66 0 2024-02-28T16:31:18.128 INFO:tasks.workunit.client.1.smithi066.stdout:5/336: stat d3/d4/dc/f19 0 2024-02-28T16:31:18.130 INFO:tasks.workunit.client.1.smithi066.stdout:7/270: creat dc/f4c x:0 0 0 2024-02-28T16:31:18.132 INFO:tasks.workunit.client.0.smithi001.stdout:2/689: write d0/f33 [1693435,100794] 0 2024-02-28T16:31:18.133 INFO:tasks.workunit.client.1.smithi066.stdout:6/299: dread - d2/d26/f47 zero size 2024-02-28T16:31:18.133 INFO:tasks.workunit.client.0.smithi001.stdout:4/817: write d3/d2d/d4c/d5e/fa1 [1004484,31755] 0 2024-02-28T16:31:18.134 INFO:tasks.workunit.client.0.smithi001.stdout:0/731: read d18/f6d [1571543,36433] 0 2024-02-28T16:31:18.135 INFO:tasks.workunit.client.1.smithi066.stdout:5/337: rmdir d3/d1b/d20 39 2024-02-28T16:31:18.139 INFO:tasks.workunit.client.0.smithi001.stdout:2/690: chown d0/d2/d3/de/d61/d12/f27 21 1 2024-02-28T16:31:18.140 INFO:tasks.workunit.client.1.smithi066.stdout:8/305: unlink d0/d6/dc/d16/l55 0 2024-02-28T16:31:18.140 INFO:tasks.workunit.client.1.smithi066.stdout:5/338: write d3/d1b/d39/f44 [1004647,89919] 0 2024-02-28T16:31:18.143 INFO:tasks.workunit.client.0.smithi001.stdout:1/847: dread d0/d9/d1a/d2b/d26/f82 [0,4194304] 0 2024-02-28T16:31:18.143 INFO:tasks.workunit.client.0.smithi001.stdout:4/818: creat d3/d5/d7/d1f/f107 x:0 0 0 2024-02-28T16:31:18.143 INFO:tasks.workunit.client.0.smithi001.stdout:2/691: fsync d0/d2/d3/de/d61/d12/d55/fcd 0 2024-02-28T16:31:18.143 INFO:tasks.workunit.client.1.smithi066.stdout:6/300: rename d2/da/d19/d54/l62 to d2/da/d19/d54/d5d/l67 0 2024-02-28T16:31:18.144 INFO:tasks.workunit.client.1.smithi066.stdout:7/271: symlink dc/l4d 0 2024-02-28T16:31:18.145 INFO:tasks.workunit.client.1.smithi066.stdout:6/301: chown d2/f29 2017722626 1 2024-02-28T16:31:18.146 INFO:tasks.workunit.client.0.smithi001.stdout:9/906: dwrite d5/d27/db1/d5e/dd3/d109/d6f/dc8/fbe [0,4194304] 0 2024-02-28T16:31:18.148 INFO:tasks.workunit.client.1.smithi066.stdout:2/352: dread d0/f1f [0,4194304] 0 2024-02-28T16:31:18.148 INFO:tasks.workunit.client.1.smithi066.stdout:6/302: stat d2/da/d19/d54/d5d/f64 0 2024-02-28T16:31:18.151 INFO:tasks.workunit.client.0.smithi001.stdout:1/848: write d0/d9/d1a/d2b/d26/d27/f123 [88752,27448] 0 2024-02-28T16:31:18.151 INFO:tasks.workunit.client.1.smithi066.stdout:3/363: truncate db/dc/d35/d3d/f41 94614 0 2024-02-28T16:31:18.153 INFO:tasks.workunit.client.0.smithi001.stdout:7/749: dread d3/d1c/fcb [0,4194304] 0 2024-02-28T16:31:18.154 INFO:tasks.workunit.client.1.smithi066.stdout:5/339: readlink d3/d4/dc/dd/d35/l54 0 2024-02-28T16:31:18.154 INFO:tasks.workunit.client.1.smithi066.stdout:3/364: fsync db/d13/d1a/d33/f39 0 2024-02-28T16:31:18.155 INFO:tasks.workunit.client.0.smithi001.stdout:7/750: chown d3/c15 20 1 2024-02-28T16:31:18.157 INFO:tasks.workunit.client.0.smithi001.stdout:1/849: fsync d0/d9/f7e 0 2024-02-28T16:31:18.157 INFO:tasks.workunit.client.0.smithi001.stdout:9/907: rmdir d5/d27/db1/d5e/dd3/d109/d6f/db3 39 2024-02-28T16:31:18.158 INFO:tasks.workunit.client.1.smithi066.stdout:5/340: dread - d3/d4/dc/d42/d57/d66/f6c zero size 2024-02-28T16:31:18.158 INFO:tasks.workunit.client.1.smithi066.stdout:3/365: write db/d2a/d44/f4f [2319649,129291] 0 2024-02-28T16:31:18.159 INFO:tasks.workunit.client.0.smithi001.stdout:7/751: read d3/d17/d3d/d59/fb1 [3503646,54081] 0 2024-02-28T16:31:18.160 INFO:tasks.workunit.client.1.smithi066.stdout:0/446: dread d2/d27/d41/d76/d93/d10/d50/d52/f55 [0,4194304] 0 2024-02-28T16:31:18.161 INFO:tasks.workunit.client.1.smithi066.stdout:5/341: readlink d3/d1b/d39/l3d 0 2024-02-28T16:31:18.162 INFO:tasks.workunit.client.0.smithi001.stdout:1/850: chown d0/d9/d1a/d2b/d26/d27/d30/d88/fe4 0 1 2024-02-28T16:31:18.164 INFO:tasks.workunit.client.1.smithi066.stdout:3/366: fdatasync db/dc/d35/d3d/d76/f4a 0 2024-02-28T16:31:18.166 INFO:tasks.workunit.client.0.smithi001.stdout:7/752: symlink d3/d17/d52/d7f/d99/dc1/d49/lf2 0 2024-02-28T16:31:18.167 INFO:tasks.workunit.client.0.smithi001.stdout:9/908: rename d5/d27/db1/d5e/dd3/d109/d6f/db3/c102 to d5/d27/d34/d50/d9d/c122 0 2024-02-28T16:31:18.169 INFO:tasks.workunit.client.0.smithi001.stdout:9/909: dread - d5/d27/db1/d65/fae zero size 2024-02-28T16:31:18.169 INFO:tasks.workunit.client.1.smithi066.stdout:4/442: truncate d1/d18/d4e/f50 473179 0 2024-02-28T16:31:18.171 INFO:tasks.workunit.client.0.smithi001.stdout:9/910: chown d5/d27/d34/d50/fea 0 1 2024-02-28T16:31:18.172 INFO:tasks.workunit.client.1.smithi066.stdout:5/342: creat d3/d4/dc/d2f/d55/f70 x:0 0 0 2024-02-28T16:31:18.174 INFO:tasks.workunit.client.0.smithi001.stdout:9/911: creat d5/d27/db1/d58/f123 x:0 0 0 2024-02-28T16:31:18.175 INFO:tasks.workunit.client.1.smithi066.stdout:2/353: dwrite d0/d12/d1a/d39/d40/f5b [0,4194304] 0 2024-02-28T16:31:18.176 INFO:tasks.workunit.client.1.smithi066.stdout:0/447: creat d2/d27/d41/d76/d93/d10/f98 x:0 0 0 2024-02-28T16:31:18.176 INFO:tasks.workunit.client.0.smithi001.stdout:2/692: truncate d0/d2/d3/fca 3640253 0 2024-02-28T16:31:18.178 INFO:tasks.workunit.client.0.smithi001.stdout:4/819: truncate d3/d1c/d58/f6b 2593034 0 2024-02-28T16:31:18.179 INFO:tasks.workunit.client.0.smithi001.stdout:4/820: readlink d3/d2d/l36 0 2024-02-28T16:31:18.181 INFO:tasks.workunit.client.1.smithi066.stdout:8/306: dwrite d0/f23 [0,4194304] 0 2024-02-28T16:31:18.182 INFO:tasks.workunit.client.1.smithi066.stdout:3/367: symlink db/d2a/d44/d48/d5d/l7e 0 2024-02-28T16:31:18.183 INFO:tasks.workunit.client.1.smithi066.stdout:0/448: write d2/f32 [269058,11840] 0 2024-02-28T16:31:18.189 INFO:tasks.workunit.client.0.smithi001.stdout:9/912: dwrite d5/d27/db1/d33/fdb [4194304,4194304] 0 2024-02-28T16:31:18.190 INFO:tasks.workunit.client.1.smithi066.stdout:7/272: dwrite dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:18.192 INFO:tasks.workunit.client.0.smithi001.stdout:9/913: chown d5/d27/db1/d33/c98 14 1 2024-02-28T16:31:18.197 INFO:tasks.workunit.client.1.smithi066.stdout:0/449: stat d2/d27/d41/d76/d93/d10/d50/l63 0 2024-02-28T16:31:18.201 INFO:tasks.workunit.client.1.smithi066.stdout:0/450: chown d2/d27/d41/d76/d93/d10/d23/d6d/f79 47 1 2024-02-28T16:31:18.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/354: dread d0/d12/d1a/d39/d40/f5b [0,4194304] 0 2024-02-28T16:31:18.205 INFO:tasks.workunit.client.1.smithi066.stdout:7/273: rename l2 to dc/l4e 0 2024-02-28T16:31:18.214 INFO:tasks.workunit.client.1.smithi066.stdout:2/355: rename d0/d12/d1a/f4c to d0/d12/d1a/d23/d29/f6e 0 2024-02-28T16:31:18.216 INFO:tasks.workunit.client.0.smithi001.stdout:1/851: write d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/f6e [795224,73208] 0 2024-02-28T16:31:18.217 INFO:tasks.workunit.client.1.smithi066.stdout:7/274: link dc/de/d17/d1e/c3d dc/de/c4f 0 2024-02-28T16:31:18.218 INFO:tasks.workunit.client.1.smithi066.stdout:2/356: readlink d0/d12/d1a/d23/d5d/l52 0 2024-02-28T16:31:18.224 INFO:tasks.workunit.client.1.smithi066.stdout:7/275: mknod dc/de/d17/d1e/c50 0 2024-02-28T16:31:18.224 INFO:tasks.workunit.client.1.smithi066.stdout:0/451: dwrite d2/d27/d41/d76/d93/d10/f47 [0,4194304] 0 2024-02-28T16:31:18.225 INFO:tasks.workunit.client.1.smithi066.stdout:2/357: mknod d0/d12/d1a/d23/d29/d21/c6f 0 2024-02-28T16:31:18.227 INFO:tasks.workunit.client.0.smithi001.stdout:2/693: dwrite d0/d2/da6/fc4 [0,4194304] 0 2024-02-28T16:31:18.228 INFO:tasks.workunit.client.0.smithi001.stdout:4/821: dwrite d3/d2d/d4c/d69/ff9 [0,4194304] 0 2024-02-28T16:31:18.228 INFO:tasks.workunit.client.0.smithi001.stdout:1/852: dread d0/d9/d56/f6c [0,4194304] 0 2024-02-28T16:31:18.229 INFO:tasks.workunit.client.1.smithi066.stdout:2/358: read d0/f9 [2969551,1007] 0 2024-02-28T16:31:18.233 INFO:tasks.workunit.client.1.smithi066.stdout:2/359: read - d0/d12/d1a/d23/d5d/f4d zero size 2024-02-28T16:31:18.235 INFO:tasks.workunit.client.0.smithi001.stdout:7/753: dread d3/d17/d61/f71 [0,4194304] 0 2024-02-28T16:31:18.239 INFO:tasks.workunit.client.0.smithi001.stdout:7/754: fdatasync d3/d17/f1b 0 2024-02-28T16:31:18.240 INFO:tasks.workunit.client.0.smithi001.stdout:2/694: creat d0/d2/d3/de/d61/d12/d55/db2/fe6 x:0 0 0 2024-02-28T16:31:18.249 INFO:tasks.workunit.client.1.smithi066.stdout:5/343: write d3/d4/f61 [781919,57460] 0 2024-02-28T16:31:18.249 INFO:tasks.workunit.client.0.smithi001.stdout:7/755: symlink d3/d17/d52/d7f/d99/dc1/d49/lf3 0 2024-02-28T16:31:18.249 INFO:tasks.workunit.client.1.smithi066.stdout:4/443: chown d1/d18/d4e/f50 922 1 2024-02-28T16:31:18.251 INFO:tasks.workunit.client.0.smithi001.stdout:2/695: link d0/d2/d3/de/d8a/ccb d0/d2/d3/de/d61/ce7 0 2024-02-28T16:31:18.252 INFO:tasks.workunit.client.1.smithi066.stdout:7/276: dwrite dc/f4b [0,4194304] 0 2024-02-28T16:31:18.254 INFO:tasks.workunit.client.0.smithi001.stdout:7/756: rename d3/c34 to d3/d17/d3d/d59/cf4 0 2024-02-28T16:31:18.254 INFO:tasks.workunit.client.1.smithi066.stdout:7/277: fdatasync dc/d29/f2e 0 2024-02-28T16:31:18.258 INFO:tasks.workunit.client.1.smithi066.stdout:8/307: write d0/d6/f7 [1045145,124772] 0 2024-02-28T16:31:18.261 INFO:tasks.workunit.client.1.smithi066.stdout:3/368: dread db/dc/d35/d3d/f41 [0,4194304] 0 2024-02-28T16:31:18.261 INFO:tasks.workunit.client.0.smithi001.stdout:4/822: write d3/d5/fb6 [959991,67776] 0 2024-02-28T16:31:18.263 INFO:tasks.workunit.client.1.smithi066.stdout:3/369: dread - db/f66 zero size 2024-02-28T16:31:18.264 INFO:tasks.workunit.client.1.smithi066.stdout:0/452: rename d2/d27/c3a to d2/d27/d41/d76/d93/d10/d50/d51/c99 0 2024-02-28T16:31:18.266 INFO:tasks.workunit.client.0.smithi001.stdout:4/823: getdents d3/d2d/d4c/d5e/db0 0 2024-02-28T16:31:18.270 INFO:tasks.workunit.client.1.smithi066.stdout:0/453: write d2/d27/d41/d76/d93/d10/d23/f7a [638068,27459] 0 2024-02-28T16:31:18.272 INFO:tasks.workunit.client.1.smithi066.stdout:5/344: dread d3/d4/f4e [0,4194304] 0 2024-02-28T16:31:18.272 INFO:tasks.workunit.client.0.smithi001.stdout:4/824: link d3/d1c/d58/d6e/d9f/lfa d3/d5/d7/ddb/l108 0 2024-02-28T16:31:18.273 INFO:tasks.workunit.client.0.smithi001.stdout:4/825: readlink d3/d5/d47/l8c 0 2024-02-28T16:31:18.275 INFO:tasks.workunit.client.0.smithi001.stdout:4/826: write d3/f78 [1112094,17361] 0 2024-02-28T16:31:18.277 INFO:tasks.workunit.client.1.smithi066.stdout:5/345: fdatasync d3/d4/dc/d42/d57/d66/f6c 0 2024-02-28T16:31:18.277 INFO:tasks.workunit.client.1.smithi066.stdout:2/360: getdents d0/d12/d1a/d39/d40 0 2024-02-28T16:31:18.278 INFO:tasks.workunit.client.1.smithi066.stdout:4/444: dwrite d1/d23/d25/d43/f5d [4194304,4194304] 0 2024-02-28T16:31:18.280 INFO:tasks.workunit.client.0.smithi001.stdout:4/827: rename d3/d5/d47/l8c to d3/df8/l109 0 2024-02-28T16:31:18.283 INFO:tasks.workunit.client.0.smithi001.stdout:7/757: creat d3/d17/d52/d7f/d99/ff5 x:0 0 0 2024-02-28T16:31:18.284 INFO:tasks.workunit.client.1.smithi066.stdout:3/370: rename db/dc/d11/l16 to db/dc/d35/d3d/l7f 0 2024-02-28T16:31:18.284 INFO:tasks.workunit.client.0.smithi001.stdout:1/853: dwrite d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/f8c [0,4194304] 0 2024-02-28T16:31:18.285 INFO:tasks.workunit.client.0.smithi001.stdout:7/758: stat d3/c4a 0 2024-02-28T16:31:18.286 INFO:tasks.workunit.client.1.smithi066.stdout:4/445: write d1/d23/d25/f83 [632859,36618] 0 2024-02-28T16:31:18.286 INFO:tasks.workunit.client.0.smithi001.stdout:1/854: fsync d0/d9/d1a/d2b/f6d 0 2024-02-28T16:31:18.287 INFO:tasks.workunit.client.1.smithi066.stdout:2/361: fsync d0/ff 0 2024-02-28T16:31:18.289 INFO:tasks.workunit.client.1.smithi066.stdout:4/446: stat d1/d23/d25/l35 0 2024-02-28T16:31:18.291 INFO:tasks.workunit.client.0.smithi001.stdout:2/696: dread d0/d2/d3/de/d22/f24 [0,4194304] 0 2024-02-28T16:31:18.292 INFO:tasks.workunit.client.1.smithi066.stdout:7/278: dwrite dc/de/d17/d1e/f24 [0,4194304] 0 2024-02-28T16:31:18.293 INFO:tasks.workunit.client.1.smithi066.stdout:5/346: unlink d3/d4/dc/f19 0 2024-02-28T16:31:18.297 INFO:tasks.workunit.client.0.smithi001.stdout:1/855: link d0/d9/d1a/d2b/d26/f108 d0/d9/d1a/d2b/d26/d105/f128 0 2024-02-28T16:31:18.298 INFO:tasks.workunit.client.1.smithi066.stdout:3/371: fdatasync db/dc/d11/d1f/f2e 0 2024-02-28T16:31:18.299 INFO:tasks.workunit.client.1.smithi066.stdout:5/347: write d3/d4/dc/dd/d35/f64 [269187,80713] 0 2024-02-28T16:31:18.302 INFO:tasks.workunit.client.1.smithi066.stdout:8/308: dread d0/db/d15/f31 [0,4194304] 0 2024-02-28T16:31:18.304 INFO:tasks.workunit.client.0.smithi001.stdout:2/697: mkdir d0/d2/d3/de/d61/d12/de8 0 2024-02-28T16:31:18.305 INFO:tasks.workunit.client.0.smithi001.stdout:7/759: link d3/d17/d52/d7f/d99/dc1/d49/d94/fea d3/d17/d3d/d59/ff6 0 2024-02-28T16:31:18.305 INFO:tasks.workunit.client.1.smithi066.stdout:2/362: mknod d0/d12/d1a/d39/d40/c70 0 2024-02-28T16:31:18.308 INFO:tasks.workunit.client.1.smithi066.stdout:3/372: chown db/d13/d1a/d33/c64 32945492 1 2024-02-28T16:31:18.308 INFO:tasks.workunit.client.1.smithi066.stdout:5/348: chown d3/d4/dc/dd/d41/f65 146 1 2024-02-28T16:31:18.309 INFO:tasks.workunit.client.0.smithi001.stdout:7/760: chown d3/d17/d52/d55/c78 32251 1 2024-02-28T16:31:18.311 INFO:tasks.workunit.client.1.smithi066.stdout:0/454: dwrite d2/d27/d41/d76/d93/f1e [4194304,4194304] 0 2024-02-28T16:31:18.314 INFO:tasks.workunit.client.1.smithi066.stdout:4/447: rename d1/d18/d55/c5b to d1/d18/d4e/c8f 0 2024-02-28T16:31:18.322 INFO:tasks.workunit.client.1.smithi066.stdout:3/373: dread db/d13/f23 [0,4194304] 0 2024-02-28T16:31:18.325 INFO:tasks.workunit.client.1.smithi066.stdout:0/455: creat d2/d27/d41/d76/d93/d10/d50/d52/f9a x:0 0 0 2024-02-28T16:31:18.325 INFO:tasks.workunit.client.1.smithi066.stdout:4/448: creat d1/d7a/f90 x:0 0 0 2024-02-28T16:31:18.331 INFO:tasks.workunit.client.1.smithi066.stdout:4/449: read d1/f20 [3521924,51363] 0 2024-02-28T16:31:18.332 INFO:tasks.workunit.client.1.smithi066.stdout:0/456: chown d2/d27/d41/d76/d93/d10/d50/d60/f95 15 1 2024-02-28T16:31:18.338 INFO:tasks.workunit.client.1.smithi066.stdout:3/374: rmdir db/dc/d35 39 2024-02-28T16:31:18.338 INFO:tasks.workunit.client.0.smithi001.stdout:4/828: dread d3/d5/d7/d1f/f7d [0,4194304] 0 2024-02-28T16:31:18.339 INFO:tasks.workunit.client.1.smithi066.stdout:0/457: write d2/d27/d41/d76/d93/d10/d50/d52/f9a [966359,38639] 0 2024-02-28T16:31:18.341 INFO:tasks.workunit.client.1.smithi066.stdout:8/309: dwrite d0/f43 [0,4194304] 0 2024-02-28T16:31:18.344 INFO:tasks.workunit.client.0.smithi001.stdout:3/747: sync 2024-02-28T16:31:18.344 INFO:tasks.workunit.client.0.smithi001.stdout:8/738: sync 2024-02-28T16:31:18.344 INFO:tasks.workunit.client.0.smithi001.stdout:6/709: sync 2024-02-28T16:31:18.344 INFO:tasks.workunit.client.1.smithi066.stdout:0/458: fdatasync d2/d27/d41/d76/d93/d10/d50/d52/f9a 0 2024-02-28T16:31:18.345 INFO:tasks.workunit.client.0.smithi001.stdout:1/856: fsync d0/d9/d1a/d2b/d26/d27/d30/d4a/f70 0 2024-02-28T16:31:18.346 INFO:tasks.workunit.client.1.smithi066.stdout:0/459: read - d2/d85/f88 zero size 2024-02-28T16:31:18.346 INFO:tasks.workunit.client.1.smithi066.stdout:2/363: getdents d0/d12/d1a/d23/d29/d21/d2f 0 2024-02-28T16:31:18.349 INFO:tasks.workunit.client.1.smithi066.stdout:0/460: write d2/d85/f88 [103076,34241] 0 2024-02-28T16:31:18.350 INFO:tasks.workunit.client.1.smithi066.stdout:8/310: mknod d0/db/d15/c59 0 2024-02-28T16:31:18.351 INFO:tasks.workunit.client.0.smithi001.stdout:3/748: truncate d4/d1e/db7/fd0 724919 0 2024-02-28T16:31:18.351 INFO:tasks.workunit.client.0.smithi001.stdout:2/698: link d0/d2/d3/de/d8a/f51 d0/d2/da6/fe9 0 2024-02-28T16:31:18.353 INFO:tasks.workunit.client.1.smithi066.stdout:3/375: symlink db/d7a/d7c/l80 0 2024-02-28T16:31:18.355 INFO:tasks.workunit.client.0.smithi001.stdout:8/739: symlink d4/d32/d69/d8f/led 0 2024-02-28T16:31:18.355 INFO:tasks.workunit.client.1.smithi066.stdout:2/364: creat d0/f71 x:0 0 0 2024-02-28T16:31:18.357 INFO:tasks.workunit.client.1.smithi066.stdout:2/365: readlink d0/d12/l1b 0 2024-02-28T16:31:18.358 INFO:tasks.workunit.client.1.smithi066.stdout:8/311: mkdir d0/d6/dc/d16/d1c/d58/d5a 0 2024-02-28T16:31:18.359 INFO:tasks.workunit.client.1.smithi066.stdout:3/376: creat db/d2a/d44/d48/d5d/f81 x:0 0 0 2024-02-28T16:31:18.361 INFO:tasks.workunit.client.1.smithi066.stdout:2/366: mknod d0/d12/c72 0 2024-02-28T16:31:18.365 INFO:tasks.workunit.client.0.smithi001.stdout:2/699: creat d0/d2/d3/de/d61/fea x:0 0 0 2024-02-28T16:31:18.366 INFO:tasks.workunit.client.0.smithi001.stdout:7/761: rename d3/d17/d3d/d59/l95 to d3/d17/d52/lf7 0 2024-02-28T16:31:18.368 INFO:tasks.workunit.client.0.smithi001.stdout:8/740: creat d4/d32/d93/fee x:0 0 0 2024-02-28T16:31:18.369 INFO:tasks.workunit.client.1.smithi066.stdout:8/312: creat d0/d6/dc/d16/d1c/d3d/f5b x:0 0 0 2024-02-28T16:31:18.372 INFO:tasks.workunit.client.1.smithi066.stdout:2/367: mkdir d0/d12/d1a/d39/d73 0 2024-02-28T16:31:18.373 INFO:tasks.workunit.client.0.smithi001.stdout:1/857: dwrite d0/d9/d56/f8d [0,4194304] 0 2024-02-28T16:31:18.374 INFO:tasks.workunit.client.1.smithi066.stdout:8/313: mknod d0/db/d12/d38/c5c 0 2024-02-28T16:31:18.375 INFO:tasks.workunit.client.0.smithi001.stdout:2/700: readlink d0/l79 0 2024-02-28T16:31:18.375 INFO:tasks.workunit.client.1.smithi066.stdout:4/450: dwrite d1/d23/f36 [0,4194304] 0 2024-02-28T16:31:18.378 INFO:tasks.workunit.client.0.smithi001.stdout:7/762: fsync d3/d17/d52/d7f/d99/dc1/f62 0 2024-02-28T16:31:18.380 INFO:tasks.workunit.client.1.smithi066.stdout:8/314: creat d0/d6/f5d x:0 0 0 2024-02-28T16:31:18.382 INFO:tasks.workunit.client.0.smithi001.stdout:4/829: dread d3/d5/d7/d1f/d3a/fad [0,4194304] 0 2024-02-28T16:31:18.382 INFO:tasks.workunit.client.0.smithi001.stdout:3/749: dwrite d4/d1e/d7c/fd3 [0,4194304] 0 2024-02-28T16:31:18.382 INFO:tasks.workunit.client.0.smithi001.stdout:2/701: rmdir d0/d2/d3 39 2024-02-28T16:31:18.383 INFO:tasks.workunit.client.1.smithi066.stdout:8/315: write d0/f23 [2580207,102154] 0 2024-02-28T16:31:18.386 INFO:tasks.workunit.client.1.smithi066.stdout:4/451: stat d1/d18/d41/f69 0 2024-02-28T16:31:18.387 INFO:tasks.workunit.client.0.smithi001.stdout:3/750: chown d4/d6/d7/f20 58498635 1 2024-02-28T16:31:18.387 INFO:tasks.workunit.client.0.smithi001.stdout:5/691: sync 2024-02-28T16:31:18.389 INFO:tasks.workunit.client.0.smithi001.stdout:0/732: sync 2024-02-28T16:31:18.390 INFO:tasks.workunit.client.0.smithi001.stdout:3/751: write d4/d18/d5f/f5b [788821,23623] 0 2024-02-28T16:31:18.391 INFO:tasks.workunit.client.1.smithi066.stdout:4/452: mknod d1/d7a/c91 0 2024-02-28T16:31:18.393 INFO:tasks.workunit.client.1.smithi066.stdout:0/461: truncate d2/d27/d41/d76/d93/d10/d50/d51/f53 1916466 0 2024-02-28T16:31:18.397 INFO:tasks.workunit.client.1.smithi066.stdout:4/453: mknod d1/d23/d25/d77/c92 0 2024-02-28T16:31:18.399 INFO:tasks.workunit.client.0.smithi001.stdout:1/858: rename d0/d9/d1a/d72/cf1 to d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/c129 0 2024-02-28T16:31:18.403 INFO:tasks.workunit.client.0.smithi001.stdout:4/830: mknod d3/d5/d7/d1f/d2e/dbe/c10a 0 2024-02-28T16:31:18.404 INFO:tasks.workunit.client.0.smithi001.stdout:8/741: dread d4/d7/d22/d27/d4b/d77/f7a [0,4194304] 0 2024-02-28T16:31:18.408 INFO:tasks.workunit.client.0.smithi001.stdout:7/763: dwrite d3/d17/d3d/db3/fb4 [0,4194304] 0 2024-02-28T16:31:18.409 INFO:tasks.workunit.client.1.smithi066.stdout:0/462: dread d2/d27/d41/d76/d93/d10/d50/d51/f78 [0,4194304] 0 2024-02-28T16:31:18.411 INFO:tasks.workunit.client.1.smithi066.stdout:2/368: dwrite d0/d12/d1a/d39/f42 [0,4194304] 0 2024-02-28T16:31:18.412 INFO:tasks.workunit.client.0.smithi001.stdout:1/859: readlink d0/d9/d1a/d2b/d26/d27/d30/d67/d87/l71 0 2024-02-28T16:31:18.415 INFO:tasks.workunit.client.0.smithi001.stdout:3/752: symlink d4/d6/d7/lf1 0 2024-02-28T16:31:18.417 INFO:tasks.workunit.client.0.smithi001.stdout:3/753: chown d4/d1e 32754 1 2024-02-28T16:31:18.419 INFO:tasks.workunit.client.1.smithi066.stdout:0/463: rename d2/c42 to d2/d27/d41/d76/c9b 0 2024-02-28T16:31:18.421 INFO:tasks.workunit.client.1.smithi066.stdout:2/369: symlink d0/d12/d3e/l74 0 2024-02-28T16:31:18.422 INFO:tasks.workunit.client.0.smithi001.stdout:5/692: fdatasync dc/df/f3a 0 2024-02-28T16:31:18.422 INFO:tasks.workunit.client.0.smithi001.stdout:3/754: fsync d4/d18/f91 0 2024-02-28T16:31:18.425 INFO:tasks.workunit.client.1.smithi066.stdout:8/316: truncate d0/db/d15/f37 437668 0 2024-02-28T16:31:18.430 INFO:tasks.workunit.client.1.smithi066.stdout:0/464: truncate d2/d27/d41/d76/d93/f6f 892847 0 2024-02-28T16:31:18.431 INFO:tasks.workunit.client.1.smithi066.stdout:1/344: sync 2024-02-28T16:31:18.431 INFO:tasks.workunit.client.0.smithi001.stdout:3/755: mkdir d4/d1e/d7c/d7d/de0/df2 0 2024-02-28T16:31:18.432 INFO:tasks.workunit.client.0.smithi001.stdout:4/831: dread d3/d5/d7/d1f/d20/f95 [0,4194304] 0 2024-02-28T16:31:18.433 INFO:tasks.workunit.client.1.smithi066.stdout:1/345: fdatasync d2/d27/f2f 0 2024-02-28T16:31:18.435 INFO:tasks.workunit.client.0.smithi001.stdout:3/756: mknod d4/d1e/d7c/d9a/cf3 0 2024-02-28T16:31:18.437 INFO:tasks.workunit.client.1.smithi066.stdout:1/346: chown d2/d27/d2b/d37/c45 3110 1 2024-02-28T16:31:18.438 INFO:tasks.workunit.client.1.smithi066.stdout:0/465: mknod d2/d27/d41/d76/d93/d10/d50/d60/c9c 0 2024-02-28T16:31:18.440 INFO:tasks.workunit.client.0.smithi001.stdout:6/710: truncate d9/db/d5f/d89/d8a/fe1 216655 0 2024-02-28T16:31:18.440 INFO:tasks.workunit.client.0.smithi001.stdout:2/702: write d0/fc0 [239825,57916] 0 2024-02-28T16:31:18.442 INFO:tasks.workunit.client.1.smithi066.stdout:8/317: dread d0/f24 [0,4194304] 0 2024-02-28T16:31:18.444 INFO:tasks.workunit.client.0.smithi001.stdout:3/757: rename d4/d6/d7f/d82/cd4 to d4/d1e/dcf/cf4 0 2024-02-28T16:31:18.445 INFO:tasks.workunit.client.1.smithi066.stdout:8/318: readlink d0/db/d12/d38/l3e 0 2024-02-28T16:31:18.446 INFO:tasks.workunit.client.0.smithi001.stdout:3/758: stat d4/d1e/f27 0 2024-02-28T16:31:18.450 INFO:tasks.workunit.client.0.smithi001.stdout:0/733: write d18/d22/d2b/d36/d4d/f65 [954748,30278] 0 2024-02-28T16:31:18.450 INFO:tasks.workunit.client.1.smithi066.stdout:2/370: dwrite d0/d12/f5c [0,4194304] 0 2024-02-28T16:31:18.453 INFO:tasks.workunit.client.1.smithi066.stdout:8/319: symlink d0/d6/dc/d16/l5e 0 2024-02-28T16:31:18.455 INFO:tasks.workunit.client.0.smithi001.stdout:7/764: write d3/d17/d61/faf [803706,64177] 0 2024-02-28T16:31:18.457 INFO:tasks.workunit.client.0.smithi001.stdout:4/832: dread d3/f12 [0,4194304] 0 2024-02-28T16:31:18.459 INFO:tasks.workunit.client.0.smithi001.stdout:8/742: truncate d4/d7/d22/d27/d58/d94/db4/fc0 317618 0 2024-02-28T16:31:18.459 INFO:tasks.workunit.client.1.smithi066.stdout:4/454: dread d1/f4f [0,4194304] 0 2024-02-28T16:31:18.461 INFO:tasks.workunit.client.0.smithi001.stdout:7/765: fsync d3/d1c/f51 0 2024-02-28T16:31:18.461 INFO:tasks.workunit.client.0.smithi001.stdout:4/833: read - d3/d5/d7/d1f/d2e/dbe/f9c zero size 2024-02-28T16:31:18.464 INFO:tasks.workunit.client.0.smithi001.stdout:8/743: fsync d4/d32/dd6/fe6 0 2024-02-28T16:31:18.467 INFO:tasks.workunit.client.0.smithi001.stdout:4/834: dread - d3/d2d/d4c/d69/d82/fef zero size 2024-02-28T16:31:18.467 INFO:tasks.workunit.client.0.smithi001.stdout:6/711: write d9/db/d5f/d62/d7e/fae [173608,8171] 0 2024-02-28T16:31:18.467 INFO:tasks.workunit.client.0.smithi001.stdout:2/703: write d0/d2/d3/de/d22/f52 [2757203,55278] 0 2024-02-28T16:31:18.470 INFO:tasks.workunit.client.1.smithi066.stdout:0/466: write d2/d27/d41/d76/d93/d10/d50/d52/f81 [597912,76803] 0 2024-02-28T16:31:18.473 INFO:tasks.workunit.client.0.smithi001.stdout:5/693: dwrite dc/d12/f14 [0,4194304] 0 2024-02-28T16:31:18.475 INFO:tasks.workunit.client.0.smithi001.stdout:6/712: mknod d9/cec 0 2024-02-28T16:31:18.477 INFO:tasks.workunit.client.1.smithi066.stdout:2/371: dwrite d0/f71 [0,4194304] 0 2024-02-28T16:31:18.477 INFO:tasks.workunit.client.1.smithi066.stdout:1/347: dread d2/d18/d19/f1c [0,4194304] 0 2024-02-28T16:31:18.477 INFO:tasks.workunit.client.0.smithi001.stdout:2/704: fsync d0/d2/d3/de/d22/dbb/fbf 0 2024-02-28T16:31:18.478 INFO:tasks.workunit.client.1.smithi066.stdout:4/455: dread d1/f11 [0,4194304] 0 2024-02-28T16:31:18.478 INFO:tasks.workunit.client.0.smithi001.stdout:3/759: write d4/d6/f52 [1223998,77142] 0 2024-02-28T16:31:18.479 INFO:tasks.workunit.client.1.smithi066.stdout:0/467: creat d2/d27/d41/d76/d93/d10/d23/f9d x:0 0 0 2024-02-28T16:31:18.480 INFO:tasks.workunit.client.0.smithi001.stdout:4/835: creat d3/d5/d47/f10b x:0 0 0 2024-02-28T16:31:18.482 INFO:tasks.workunit.client.0.smithi001.stdout:8/744: getdents d4/d7/d22/d27/d58/d62 0 2024-02-28T16:31:18.483 INFO:tasks.workunit.client.1.smithi066.stdout:2/372: readlink d0/d12/d1a/d39/d40/l6c 0 2024-02-28T16:31:18.485 INFO:tasks.workunit.client.1.smithi066.stdout:1/348: fdatasync d2/d27/d2b/d43/d46/f6c 0 2024-02-28T16:31:18.487 INFO:tasks.workunit.client.0.smithi001.stdout:3/760: readlink d4/d6/d7/d78/d81/dd6/le4 0 2024-02-28T16:31:18.488 INFO:tasks.workunit.client.0.smithi001.stdout:5/694: rename dc/d12/f6f to dc/d12/d56/d96/db9/dc2/dca/fe4 0 2024-02-28T16:31:18.491 INFO:tasks.workunit.client.0.smithi001.stdout:4/836: creat d3/d2d/d83/f10c x:0 0 0 2024-02-28T16:31:18.492 INFO:tasks.workunit.client.1.smithi066.stdout:2/373: chown d0/ff 10 1 2024-02-28T16:31:18.493 INFO:tasks.workunit.client.0.smithi001.stdout:8/745: symlink d4/d7/d22/d27/d58/d62/lef 0 2024-02-28T16:31:18.495 INFO:tasks.workunit.client.1.smithi066.stdout:1/349: chown d2/d27/d2b/d37/d51/l6d 1318 1 2024-02-28T16:31:18.495 INFO:tasks.workunit.client.0.smithi001.stdout:2/705: mknod d0/d2/d3/de/d61/dd5/ceb 0 2024-02-28T16:31:18.496 INFO:tasks.workunit.client.1.smithi066.stdout:8/320: dwrite d0/db/d15/f31 [4194304,4194304] 0 2024-02-28T16:31:18.497 INFO:tasks.workunit.client.0.smithi001.stdout:3/761: mknod d4/d18/cf5 0 2024-02-28T16:31:18.498 INFO:tasks.workunit.client.0.smithi001.stdout:5/695: mknod dc/d12/d56/d96/db9/dc2/ce5 0 2024-02-28T16:31:18.499 INFO:tasks.workunit.client.1.smithi066.stdout:2/374: creat d0/d12/d1a/d23/d29/d21/f75 x:0 0 0 2024-02-28T16:31:18.500 INFO:tasks.workunit.client.0.smithi001.stdout:6/713: dwrite d9/de/f2c [0,4194304] 0 2024-02-28T16:31:18.502 INFO:tasks.workunit.client.0.smithi001.stdout:2/706: creat d0/d2/da6/fec x:0 0 0 2024-02-28T16:31:18.503 INFO:tasks.workunit.client.0.smithi001.stdout:8/746: symlink d4/d32/d69/d9a/dcd/lf0 0 2024-02-28T16:31:18.506 INFO:tasks.workunit.client.1.smithi066.stdout:8/321: creat d0/db/f5f x:0 0 0 2024-02-28T16:31:18.508 INFO:tasks.workunit.client.1.smithi066.stdout:0/468: dwrite d2/d27/d41/d76/d93/d10/d50/d52/f81 [0,4194304] 0 2024-02-28T16:31:18.511 INFO:tasks.workunit.client.1.smithi066.stdout:2/375: rename d0/d12/d1a/d23/d29/d21/c6f to d0/d12/d2e/d3b/d66/c76 0 2024-02-28T16:31:18.515 INFO:tasks.workunit.client.1.smithi066.stdout:0/469: stat d2/d27/d41/d76/d93/d10/d23/d6d 0 2024-02-28T16:31:18.520 INFO:tasks.workunit.client.1.smithi066.stdout:4/456: dwrite d1/d23/d25/d43/f82 [0,4194304] 0 2024-02-28T16:31:18.520 INFO:tasks.workunit.client.1.smithi066.stdout:2/376: dread d0/d12/d1a/d39/d40/f55 [0,4194304] 0 2024-02-28T16:31:18.522 INFO:tasks.workunit.client.1.smithi066.stdout:4/457: dread - d1/d18/f8c zero size 2024-02-28T16:31:18.527 INFO:tasks.workunit.client.1.smithi066.stdout:4/458: mknod d1/d18/d55/d81/c93 0 2024-02-28T16:31:18.530 INFO:tasks.workunit.client.0.smithi001.stdout:4/837: dwrite d3/d1c/d58/d6e/d9f/fec [0,4194304] 0 2024-02-28T16:31:18.532 INFO:tasks.workunit.client.1.smithi066.stdout:4/459: write d1/d23/d25/f6b [611118,44360] 0 2024-02-28T16:31:18.536 INFO:tasks.workunit.client.1.smithi066.stdout:8/322: truncate d0/f24 783139 0 2024-02-28T16:31:18.541 INFO:tasks.workunit.client.1.smithi066.stdout:0/470: dwrite d2/d27/d41/d76/d93/f75 [4194304,4194304] 0 2024-02-28T16:31:18.542 INFO:tasks.workunit.client.1.smithi066.stdout:8/323: symlink d0/d6/dc/d16/l60 0 2024-02-28T16:31:18.545 INFO:tasks.workunit.client.1.smithi066.stdout:1/350: dread d2/d27/d2b/f2a [4194304,4194304] 0 2024-02-28T16:31:18.554 INFO:tasks.workunit.client.1.smithi066.stdout:1/351: mkdir d2/d27/d2b/d37/d71 0 2024-02-28T16:31:18.556 INFO:tasks.workunit.client.1.smithi066.stdout:8/324: mknod d0/db/d12/d1d/c61 0 2024-02-28T16:31:18.558 INFO:tasks.workunit.client.1.smithi066.stdout:8/325: readlink d0/d6/dc/d16/d1a/l33 0 2024-02-28T16:31:18.559 INFO:tasks.workunit.client.1.smithi066.stdout:0/471: getdents d2/d27/d41/d76 0 2024-02-28T16:31:18.559 INFO:tasks.workunit.client.1.smithi066.stdout:1/352: mkdir d2/d18/d6b/d72 0 2024-02-28T16:31:18.562 INFO:tasks.workunit.client.1.smithi066.stdout:8/326: creat d0/d6/dc/d16/d1c/d3d/f62 x:0 0 0 2024-02-28T16:31:18.564 INFO:tasks.workunit.client.1.smithi066.stdout:8/327: stat d0/d6/f7 0 2024-02-28T16:31:18.566 INFO:tasks.workunit.client.1.smithi066.stdout:1/353: link d2/f1d d2/d27/d2b/d37/d71/f73 0 2024-02-28T16:31:18.567 INFO:tasks.workunit.client.1.smithi066.stdout:4/460: getdents d1/d18/d55/d81 0 2024-02-28T16:31:18.569 INFO:tasks.workunit.client.1.smithi066.stdout:4/461: fdatasync d1/f6 0 2024-02-28T16:31:18.569 INFO:tasks.workunit.client.1.smithi066.stdout:1/354: write d2/f16 [905476,111241] 0 2024-02-28T16:31:18.571 INFO:tasks.workunit.client.1.smithi066.stdout:1/355: dread - d2/d27/d2b/f39 zero size 2024-02-28T16:31:18.573 INFO:tasks.workunit.client.1.smithi066.stdout:4/462: creat d1/d18/d4e/f94 x:0 0 0 2024-02-28T16:31:18.574 INFO:tasks.workunit.client.1.smithi066.stdout:8/328: read d0/d6/dc/d16/d1c/d58/f48 [138961,4854] 0 2024-02-28T16:31:18.577 INFO:tasks.workunit.client.1.smithi066.stdout:4/463: creat d1/d18/d41/f95 x:0 0 0 2024-02-28T16:31:18.581 INFO:tasks.workunit.client.1.smithi066.stdout:0/472: truncate d2/d27/d41/d76/d93/d10/d50/d73/f90 759620 0 2024-02-28T16:31:18.586 INFO:tasks.workunit.client.1.smithi066.stdout:8/329: rename d0/d6/dc/d16/d1c to d0/d6/dc/d16/d1a/d63 0 2024-02-28T16:31:18.586 INFO:tasks.workunit.client.0.smithi001.stdout:9/914: sync 2024-02-28T16:31:18.590 INFO:tasks.workunit.client.1.smithi066.stdout:0/473: rename d2/d27/d41/d76/d93/d10/f46 to d2/d27/d41/d76/f9e 0 2024-02-28T16:31:18.593 INFO:tasks.workunit.client.1.smithi066.stdout:1/356: dwrite d2/d18/f26 [0,4194304] 0 2024-02-28T16:31:18.596 INFO:tasks.workunit.client.1.smithi066.stdout:1/357: fdatasync d2/d27/d2b/d37/d51/f62 0 2024-02-28T16:31:18.597 INFO:tasks.workunit.client.1.smithi066.stdout:8/330: creat d0/d6/dc/d16/d1a/d63/d58/d5a/f64 x:0 0 0 2024-02-28T16:31:18.599 INFO:tasks.workunit.client.1.smithi066.stdout:0/474: creat d2/d85/f9f x:0 0 0 2024-02-28T16:31:18.599 INFO:tasks.workunit.client.0.smithi001.stdout:5/696: write f2 [2701626,86805] 0 2024-02-28T16:31:18.599 INFO:tasks.workunit.client.0.smithi001.stdout:3/762: creat d4/d1e/d7c/ff6 x:0 0 0 2024-02-28T16:31:18.600 INFO:tasks.workunit.client.0.smithi001.stdout:6/714: write d9/db/d10/f7b [597303,654] 0 2024-02-28T16:31:18.601 INFO:tasks.workunit.client.1.smithi066.stdout:1/358: mknod d2/d27/d2b/d34/c74 0 2024-02-28T16:31:18.601 INFO:tasks.workunit.client.0.smithi001.stdout:9/915: fsync d5/d27/db1/f93 0 2024-02-28T16:31:18.603 INFO:tasks.workunit.client.0.smithi001.stdout:4/838: rename d3/d5/d7/d1f/d2e/f59 to d3/d5/d7/d1f/d3a/f10d 0 2024-02-28T16:31:18.603 INFO:tasks.workunit.client.1.smithi066.stdout:1/359: chown d2/d18/f6e 3268032 1 2024-02-28T16:31:18.603 INFO:tasks.workunit.client.1.smithi066.stdout:4/464: dwrite d1/d18/d4e/f94 [0,4194304] 0 2024-02-28T16:31:18.606 INFO:tasks.workunit.client.0.smithi001.stdout:6/715: chown d9/d29/d6e/d53/fc9 38497 1 2024-02-28T16:31:18.608 INFO:tasks.workunit.client.0.smithi001.stdout:5/697: mknod dc/d1a/ce6 0 2024-02-28T16:31:18.608 INFO:tasks.workunit.client.1.smithi066.stdout:0/475: unlink d2/d27/d41/d76/d93/d10/d50/d51/f78 0 2024-02-28T16:31:18.608 INFO:tasks.workunit.client.0.smithi001.stdout:8/747: dwrite d4/d7/d22/d27/f4a [8388608,4194304] 0 2024-02-28T16:31:18.608 INFO:tasks.workunit.client.1.smithi066.stdout:1/360: fdatasync d2/d18/f21 0 2024-02-28T16:31:18.613 INFO:tasks.workunit.client.0.smithi001.stdout:4/839: readlink d3/d5/d7/d1f/d20/l2b 0 2024-02-28T16:31:18.614 INFO:tasks.workunit.client.1.smithi066.stdout:1/361: unlink d2/d27/d2b/c3e 0 2024-02-28T16:31:18.614 INFO:tasks.workunit.client.0.smithi001.stdout:6/716: chown d9/db/d5f/d69/f75 3 1 2024-02-28T16:31:18.615 INFO:tasks.workunit.client.1.smithi066.stdout:4/465: creat d1/f96 x:0 0 0 2024-02-28T16:31:18.616 INFO:tasks.workunit.client.1.smithi066.stdout:1/362: chown d2/d18/d19/f33 0 1 2024-02-28T16:31:18.621 INFO:tasks.workunit.client.0.smithi001.stdout:4/840: creat d3/d2d/d4c/f10e x:0 0 0 2024-02-28T16:31:18.622 INFO:tasks.workunit.client.1.smithi066.stdout:4/466: write d1/d18/d41/f69 [654634,11865] 0 2024-02-28T16:31:18.623 INFO:tasks.workunit.client.0.smithi001.stdout:4/841: write d3/d2d/d81/fae [1237235,104208] 0 2024-02-28T16:31:18.628 INFO:tasks.workunit.client.1.smithi066.stdout:4/467: symlink d1/d23/d25/d3e/l97 0 2024-02-28T16:31:18.629 INFO:tasks.workunit.client.0.smithi001.stdout:8/748: symlink d4/d7/d22/d27/d58/d94/db4/dd5/lf1 0 2024-02-28T16:31:18.630 INFO:tasks.workunit.client.0.smithi001.stdout:3/763: link d4/d18/d5f/f59 d4/d1e/dcf/ff7 0 2024-02-28T16:31:18.630 INFO:tasks.workunit.client.1.smithi066.stdout:8/331: write d0/f24 [1601436,31726] 0 2024-02-28T16:31:18.631 INFO:tasks.workunit.client.1.smithi066.stdout:9/299: sync 2024-02-28T16:31:18.631 INFO:tasks.workunit.client.1.smithi066.stdout:6/303: sync 2024-02-28T16:31:18.632 INFO:tasks.workunit.client.0.smithi001.stdout:5/698: getdents dc/d1a/d52 0 2024-02-28T16:31:18.632 INFO:tasks.workunit.client.0.smithi001.stdout:2/707: dwrite d0/d2/d3/d36/f68 [0,4194304] 0 2024-02-28T16:31:18.634 INFO:tasks.workunit.client.1.smithi066.stdout:0/476: dwrite d2/d27/d41/d76/d93/d33/f62 [0,4194304] 0 2024-02-28T16:31:18.635 INFO:tasks.workunit.client.1.smithi066.stdout:6/304: read - d2/d2c/f4a zero size 2024-02-28T16:31:18.635 INFO:tasks.workunit.client.1.smithi066.stdout:8/332: dread - d0/d6/dc/d16/f50 zero size 2024-02-28T16:31:18.635 INFO:tasks.workunit.client.0.smithi001.stdout:8/749: write d4/d32/dd6/fd2 [485125,86496] 0 2024-02-28T16:31:18.637 INFO:tasks.workunit.client.0.smithi001.stdout:9/916: dwrite d5/d27/db1/f93 [0,4194304] 0 2024-02-28T16:31:18.638 INFO:tasks.workunit.client.0.smithi001.stdout:8/750: chown d4/d9b/cc7 28404789 1 2024-02-28T16:31:18.638 INFO:tasks.workunit.client.1.smithi066.stdout:9/300: chown da/df/d18/d28/d63/c61 30508205 1 2024-02-28T16:31:18.639 INFO:tasks.workunit.client.1.smithi066.stdout:6/305: chown d2/d26/c3a 305534 1 2024-02-28T16:31:18.642 INFO:tasks.workunit.client.1.smithi066.stdout:6/306: dread - d2/da/d19/d54/d5d/f64 zero size 2024-02-28T16:31:18.645 INFO:tasks.workunit.client.1.smithi066.stdout:9/301: write da/df/d18/d2d/f62 [782801,17563] 0 2024-02-28T16:31:18.645 INFO:tasks.workunit.client.1.smithi066.stdout:0/477: write d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f72 [28360,110824] 0 2024-02-28T16:31:18.645 INFO:tasks.workunit.client.1.smithi066.stdout:8/333: mkdir d0/db/d12/d1d/d25/d65 0 2024-02-28T16:31:18.646 INFO:tasks.workunit.client.1.smithi066.stdout:0/478: dread - d2/f96 zero size 2024-02-28T16:31:18.653 INFO:tasks.workunit.client.0.smithi001.stdout:9/917: mkdir d5/d27/db1/d33/d115/d11f/da8/ddf/d124 0 2024-02-28T16:31:18.653 INFO:tasks.workunit.client.1.smithi066.stdout:8/334: mknod d0/d6/dc/d16/d4c/c66 0 2024-02-28T16:31:18.654 INFO:tasks.workunit.client.0.smithi001.stdout:3/764: write d4/d6/d7/d78/fc5 [159161,79576] 0 2024-02-28T16:31:18.655 INFO:tasks.workunit.client.0.smithi001.stdout:8/751: dread d4/d32/dd6/fd2 [0,4194304] 0 2024-02-28T16:31:18.656 INFO:tasks.workunit.client.1.smithi066.stdout:6/307: link d2/da/f18 d2/f68 0 2024-02-28T16:31:18.660 INFO:tasks.workunit.client.0.smithi001.stdout:3/765: stat d4/d1e/d7c/d9a/ce1 0 2024-02-28T16:31:18.660 INFO:tasks.workunit.client.1.smithi066.stdout:8/335: readlink d0/d6/dc/d16/d1a/d63/l3b 0 2024-02-28T16:31:18.661 INFO:tasks.workunit.client.1.smithi066.stdout:4/468: dread d1/d23/d25/f28 [0,4194304] 0 2024-02-28T16:31:18.663 INFO:tasks.workunit.client.0.smithi001.stdout:9/918: creat d5/d27/db1/da6/db2/dff/f125 x:0 0 0 2024-02-28T16:31:18.664 INFO:tasks.workunit.client.1.smithi066.stdout:4/469: creat d1/d7a/f98 x:0 0 0 2024-02-28T16:31:18.665 INFO:tasks.workunit.client.0.smithi001.stdout:6/717: dwrite d9/da6/f33 [0,4194304] 0 2024-02-28T16:31:18.666 INFO:tasks.workunit.client.1.smithi066.stdout:8/336: dread d0/db/f27 [0,4194304] 0 2024-02-28T16:31:18.668 INFO:tasks.workunit.client.0.smithi001.stdout:4/842: dwrite d3/d1c/d58/f98 [0,4194304] 0 2024-02-28T16:31:18.669 INFO:tasks.workunit.client.1.smithi066.stdout:1/363: dread d2/d18/f4f [0,4194304] 0 2024-02-28T16:31:18.669 INFO:tasks.workunit.client.1.smithi066.stdout:4/470: truncate d1/d23/d25/d3e/d40/f72 2029772 0 2024-02-28T16:31:18.674 INFO:tasks.workunit.client.0.smithi001.stdout:3/766: rename d4/d6/d7f to d4/d1e/d7c/d7d/df8 0 2024-02-28T16:31:18.677 INFO:tasks.workunit.client.1.smithi066.stdout:6/308: write d2/d2a/d3d/f48 [419185,21803] 0 2024-02-28T16:31:18.679 INFO:tasks.workunit.client.1.smithi066.stdout:4/471: read d1/d23/d25/f6f [3098276,25407] 0 2024-02-28T16:31:18.679 INFO:tasks.workunit.client.0.smithi001.stdout:4/843: mkdir d3/d2d/d4c/d10f 0 2024-02-28T16:31:18.679 INFO:tasks.workunit.client.1.smithi066.stdout:8/337: rename d0/db/d15/d32/l53 to d0/d6/dc/d16/d1a/d63/d58/l67 0 2024-02-28T16:31:18.680 INFO:tasks.workunit.client.1.smithi066.stdout:4/472: readlink d1/l21 0 2024-02-28T16:31:18.682 INFO:tasks.workunit.client.1.smithi066.stdout:4/473: stat d1/d23/d25/d3e/d40/f72 0 2024-02-28T16:31:18.687 INFO:tasks.workunit.client.1.smithi066.stdout:1/364: getdents d2 0 2024-02-28T16:31:18.691 INFO:tasks.workunit.client.0.smithi001.stdout:6/718: write d9/d39/f3a [2979028,14538] 0 2024-02-28T16:31:18.693 INFO:tasks.workunit.client.1.smithi066.stdout:1/365: rmdir d2/d27/d2b/d34 39 2024-02-28T16:31:18.695 INFO:tasks.workunit.client.1.smithi066.stdout:0/479: dread d2/d27/d41/d76/d93/d10/f2c [4194304,4194304] 0 2024-02-28T16:31:18.696 INFO:tasks.workunit.client.1.smithi066.stdout:6/309: mknod d2/da/d19/d54/d5d/d63/c69 0 2024-02-28T16:31:18.696 INFO:tasks.workunit.client.0.smithi001.stdout:6/719: chown d9/d29/d6a/f77 54017 1 2024-02-28T16:31:18.697 INFO:tasks.workunit.client.0.smithi001.stdout:2/708: dread d0/f40 [0,4194304] 0 2024-02-28T16:31:18.698 INFO:tasks.workunit.client.0.smithi001.stdout:4/844: rename d3/d2d/d4c/f10e to d3/d1c/d58/f110 0 2024-02-28T16:31:18.701 INFO:tasks.workunit.client.1.smithi066.stdout:1/366: truncate d2/d27/d2b/f56 4827025 0 2024-02-28T16:31:18.704 INFO:tasks.workunit.client.1.smithi066.stdout:6/310: readlink d2/da/d19/d54/d5d/l67 0 2024-02-28T16:31:18.705 INFO:tasks.workunit.client.0.smithi001.stdout:4/845: mknod d3/d5/d7/d1f/dc3/c111 0 2024-02-28T16:31:18.705 INFO:tasks.workunit.client.0.smithi001.stdout:5/699: dread dc/df/d79/f95 [0,4194304] 0 2024-02-28T16:31:18.707 INFO:tasks.workunit.client.0.smithi001.stdout:2/709: getdents d0/d2/d3/de/d61/d12 0 2024-02-28T16:31:18.710 INFO:tasks.workunit.client.0.smithi001.stdout:4/846: mkdir d3/d2d/d83/d112 0 2024-02-28T16:31:18.713 INFO:tasks.workunit.client.1.smithi066.stdout:0/480: link d2/d27/d41/d76/d93/d10/d50/d51/f8a d2/d27/d41/d76/d93/d10/d23/fa0 0 2024-02-28T16:31:18.713 INFO:tasks.workunit.client.1.smithi066.stdout:4/474: dwrite d1/f33 [0,4194304] 0 2024-02-28T16:31:18.714 INFO:tasks.workunit.client.0.smithi001.stdout:3/767: dread d4/d18/f91 [0,4194304] 0 2024-02-28T16:31:18.716 INFO:tasks.workunit.client.1.smithi066.stdout:8/338: dread d0/d6/f1f [4194304,4194304] 0 2024-02-28T16:31:18.717 INFO:tasks.workunit.client.0.smithi001.stdout:5/700: rmdir dc/d12/d56/d96/db9/ddd 39 2024-02-28T16:31:18.718 INFO:tasks.workunit.client.0.smithi001.stdout:4/847: mkdir d3/d2d/d81/d113 0 2024-02-28T16:31:18.719 INFO:tasks.workunit.client.0.smithi001.stdout:9/919: dread d5/d27/db1/d5e/dd3/d109/d6f/dc8/fe2 [0,4194304] 0 2024-02-28T16:31:18.719 INFO:tasks.workunit.client.0.smithi001.stdout:5/701: chown dc/d1a/d52/l70 14401028 1 2024-02-28T16:31:18.720 INFO:tasks.workunit.client.0.smithi001.stdout:3/768: mknod d4/d1e/ddd/cf9 0 2024-02-28T16:31:18.721 INFO:tasks.workunit.client.1.smithi066.stdout:8/339: read d0/d6/dc/d16/d1a/d63/d58/f48 [710252,118895] 0 2024-02-28T16:31:18.726 INFO:tasks.workunit.client.1.smithi066.stdout:6/311: dwrite d2/da/d19/d54/d5d/f64 [0,4194304] 0 2024-02-28T16:31:18.727 INFO:tasks.workunit.client.0.smithi001.stdout:4/848: mkdir d3/d1c/d58/d6e/d114 0 2024-02-28T16:31:18.727 INFO:tasks.workunit.client.0.smithi001.stdout:5/702: readlink dc/d12/d56/d96/db9/dc2/dca/d3c/l58 0 2024-02-28T16:31:18.728 INFO:tasks.workunit.client.0.smithi001.stdout:3/769: truncate d4/d1e/f27 4864399 0 2024-02-28T16:31:18.729 INFO:tasks.workunit.client.0.smithi001.stdout:8/752: dread d4/d7/d22/f3f [0,4194304] 0 2024-02-28T16:31:18.729 INFO:tasks.workunit.client.0.smithi001.stdout:9/920: read d5/d27/db1/d5e/dd3/d109/d6f/db3/fde [5819039,67737] 0 2024-02-28T16:31:18.730 INFO:tasks.workunit.client.1.smithi066.stdout:1/367: truncate d2/d18/d19/f3b 4957788 0 2024-02-28T16:31:18.730 INFO:tasks.workunit.client.1.smithi066.stdout:8/340: mknod d0/db/d15/d32/c68 0 2024-02-28T16:31:18.731 INFO:tasks.workunit.client.0.smithi001.stdout:8/753: stat d4/d7/d22/d27/d4b/d77/f8d 0 2024-02-28T16:31:18.732 INFO:tasks.workunit.client.1.smithi066.stdout:6/312: chown d2/d26/c2b 1996 1 2024-02-28T16:31:18.733 INFO:tasks.workunit.client.0.smithi001.stdout:8/754: dread - d4/d32/dd6/feb zero size 2024-02-28T16:31:18.734 INFO:tasks.workunit.client.0.smithi001.stdout:4/849: chown d3/d2d/d4c/d69/d82/cc1 251 1 2024-02-28T16:31:18.736 INFO:tasks.workunit.client.0.smithi001.stdout:9/921: creat d5/d27/db1/d33/f126 x:0 0 0 2024-02-28T16:31:18.736 INFO:tasks.workunit.client.0.smithi001.stdout:2/710: truncate d0/d2/d3/de/d8a/f71 2665494 0 2024-02-28T16:31:18.737 INFO:tasks.workunit.client.0.smithi001.stdout:9/922: fsync d5/d27/d42/f7e 0 2024-02-28T16:31:18.738 INFO:tasks.workunit.client.1.smithi066.stdout:8/341: symlink d0/db/d15/l69 0 2024-02-28T16:31:18.740 INFO:tasks.workunit.client.0.smithi001.stdout:3/770: mknod d4/d1e/d7c/cfa 0 2024-02-28T16:31:18.740 INFO:tasks.workunit.client.1.smithi066.stdout:4/475: dwrite d1/f6 [0,4194304] 0 2024-02-28T16:31:18.741 INFO:tasks.workunit.client.1.smithi066.stdout:6/313: mkdir d2/da/d19/d54/d5d/d63/d6a 0 2024-02-28T16:31:18.742 INFO:tasks.workunit.client.0.smithi001.stdout:8/755: mknod d4/d7/cf2 0 2024-02-28T16:31:18.742 INFO:tasks.workunit.client.0.smithi001.stdout:5/703: creat dc/d12/d56/d96/db9/dc2/dca/d3c/da0/fe7 x:0 0 0 2024-02-28T16:31:18.742 INFO:tasks.workunit.client.0.smithi001.stdout:4/850: mkdir d3/d5/d7/d1f/d2e/dbe/d115 0 2024-02-28T16:31:18.743 INFO:tasks.workunit.client.1.smithi066.stdout:8/342: stat d0/d6/dc/d16/d1a/d63/c45 0 2024-02-28T16:31:18.744 INFO:tasks.workunit.client.0.smithi001.stdout:5/704: stat dc/d1a/d7d/fcd 0 2024-02-28T16:31:18.745 INFO:tasks.workunit.client.1.smithi066.stdout:6/314: rmdir d2 39 2024-02-28T16:31:18.749 INFO:tasks.workunit.client.0.smithi001.stdout:2/711: fdatasync d0/d2/d3/de/d61/fb3 0 2024-02-28T16:31:18.750 INFO:tasks.workunit.client.0.smithi001.stdout:8/756: creat d4/d47/d8b/ff3 x:0 0 0 2024-02-28T16:31:18.751 INFO:tasks.workunit.client.1.smithi066.stdout:0/481: dwrite d2/d27/d41/d76/d93/d10/d50/d51/f53 [0,4194304] 0 2024-02-28T16:31:18.754 INFO:tasks.workunit.client.1.smithi066.stdout:6/315: readlink d2/da/d38/l4d 0 2024-02-28T16:31:18.756 INFO:tasks.workunit.client.1.smithi066.stdout:1/368: dread d2/d27/d2b/f2a [4194304,4194304] 0 2024-02-28T16:31:18.759 INFO:tasks.workunit.client.1.smithi066.stdout:8/343: mknod d0/d6/dc/d16/d4c/c6a 0 2024-02-28T16:31:18.763 INFO:tasks.workunit.client.1.smithi066.stdout:0/482: unlink d2/d27/d41/d76/d93/l6e 0 2024-02-28T16:31:18.767 INFO:tasks.workunit.client.0.smithi001.stdout:9/923: dread d5/d27/db1/f41 [0,4194304] 0 2024-02-28T16:31:18.768 INFO:tasks.workunit.client.1.smithi066.stdout:5/349: sync 2024-02-28T16:31:18.768 INFO:tasks.workunit.client.1.smithi066.stdout:7/279: sync 2024-02-28T16:31:18.769 INFO:tasks.workunit.client.1.smithi066.stdout:6/316: unlink d2/f29 0 2024-02-28T16:31:18.770 INFO:tasks.workunit.client.1.smithi066.stdout:0/483: read d2/d27/d41/d76/d93/d10/d23/d6d/f79 [3329199,87386] 0 2024-02-28T16:31:18.776 INFO:tasks.workunit.client.1.smithi066.stdout:1/369: dread d2/f16 [0,4194304] 0 2024-02-28T16:31:18.778 INFO:tasks.workunit.client.1.smithi066.stdout:8/344: getdents d0/d6/dc 0 2024-02-28T16:31:18.779 INFO:tasks.workunit.client.1.smithi066.stdout:0/484: readlink d2/d27/d41/d76/d93/d10/d50/d51/l8c 0 2024-02-28T16:31:18.791 INFO:tasks.workunit.client.1.smithi066.stdout:7/280: write dc/de/d17/d1e/f23 [2508884,17155] 0 2024-02-28T16:31:18.792 INFO:tasks.workunit.client.1.smithi066.stdout:4/476: truncate d1/d18/d41/f47 3797785 0 2024-02-28T16:31:18.795 INFO:tasks.workunit.client.1.smithi066.stdout:8/345: creat d0/d6/dc/f6b x:0 0 0 2024-02-28T16:31:18.798 INFO:tasks.workunit.client.1.smithi066.stdout:7/281: readlink dc/l3b 0 2024-02-28T16:31:18.807 INFO:tasks.workunit.client.1.smithi066.stdout:1/370: getdents d2/d27/d2b 0 2024-02-28T16:31:18.809 INFO:tasks.workunit.client.0.smithi001.stdout:2/712: mkdir d0/d2/d3/de/d22/ded 0 2024-02-28T16:31:18.809 INFO:tasks.workunit.client.1.smithi066.stdout:5/350: dwrite d3/d4/dc/dd/d35/f64 [0,4194304] 0 2024-02-28T16:31:18.810 INFO:tasks.workunit.client.0.smithi001.stdout:4/851: link d3/d2d/d4c/d69/d82/l52 d3/d5/d7/d1f/d2e/dbe/l116 0 2024-02-28T16:31:18.812 INFO:tasks.workunit.client.0.smithi001.stdout:3/771: write d4/d6/d7/d78/d81/da1/fb9 [4995391,73475] 0 2024-02-28T16:31:18.813 INFO:tasks.workunit.client.1.smithi066.stdout:6/317: dwrite d2/f68 [4194304,4194304] 0 2024-02-28T16:31:18.816 INFO:tasks.workunit.client.1.smithi066.stdout:5/351: dread - d3/d4/dc/dd/d41/d5d/f69 zero size 2024-02-28T16:31:18.817 INFO:tasks.workunit.client.1.smithi066.stdout:8/346: mkdir d0/d6/dc/d16/d4c/d6c 0 2024-02-28T16:31:18.818 INFO:tasks.workunit.client.0.smithi001.stdout:4/852: mkdir d3/d1c/da0/d117 0 2024-02-28T16:31:18.820 INFO:tasks.workunit.client.1.smithi066.stdout:1/371: creat d2/f75 x:0 0 0 2024-02-28T16:31:18.820 INFO:tasks.workunit.client.0.smithi001.stdout:3/772: mkdir d4/d18/ddb/dfb 0 2024-02-28T16:31:18.821 INFO:tasks.workunit.client.1.smithi066.stdout:4/477: write d1/d18/d41/f49 [72014,128766] 0 2024-02-28T16:31:18.822 INFO:tasks.workunit.client.0.smithi001.stdout:4/853: read d3/d1c/d58/d6c/fd2 [170161,108416] 0 2024-02-28T16:31:18.822 INFO:tasks.workunit.client.0.smithi001.stdout:5/705: dwrite dc/d12/f24 [0,4194304] 0 2024-02-28T16:31:18.826 INFO:tasks.workunit.client.1.smithi066.stdout:4/478: readlink d1/l31 0 2024-02-28T16:31:18.829 INFO:tasks.workunit.client.1.smithi066.stdout:8/347: symlink d0/db/d12/d1d/d25/d65/l6d 0 2024-02-28T16:31:18.832 INFO:tasks.workunit.client.1.smithi066.stdout:4/479: fsync d1/d23/d25/d43/f8b 0 2024-02-28T16:31:18.833 INFO:tasks.workunit.client.0.smithi001.stdout:2/713: link d0/d2/d3/de/d61/d12/c64 d0/cee 0 2024-02-28T16:31:18.837 INFO:tasks.workunit.client.0.smithi001.stdout:3/773: rename d4/d38/l79 to d4/d18/ddb/de8/lfc 0 2024-02-28T16:31:18.838 INFO:tasks.workunit.client.1.smithi066.stdout:6/318: dwrite d2/d2a/d3d/f48 [0,4194304] 0 2024-02-28T16:31:18.839 INFO:tasks.workunit.client.0.smithi001.stdout:4/854: creat d3/d1c/d58/d6e/dfe/f118 x:0 0 0 2024-02-28T16:31:18.840 INFO:tasks.workunit.client.1.smithi066.stdout:6/319: chown d2/d2a/d3d/f48 16 1 2024-02-28T16:31:18.841 INFO:tasks.workunit.client.1.smithi066.stdout:6/320: chown d2/d2a/f45 112090544 1 2024-02-28T16:31:18.852 INFO:tasks.workunit.client.0.smithi001.stdout:3/774: creat d4/d38/ffd x:0 0 0 2024-02-28T16:31:18.853 INFO:tasks.workunit.client.1.smithi066.stdout:5/352: rename d3/d4/dc/d42/d57/d66/f6c to d3/d4/f71 0 2024-02-28T16:31:18.853 INFO:tasks.workunit.client.1.smithi066.stdout:8/348: fsync d0/db/d12/d38/f54 0 2024-02-28T16:31:18.853 INFO:tasks.workunit.client.1.smithi066.stdout:3/377: sync 2024-02-28T16:31:18.853 INFO:tasks.workunit.client.1.smithi066.stdout:2/377: sync 2024-02-28T16:31:18.854 INFO:tasks.workunit.client.0.smithi001.stdout:3/775: rmdir d4/d97 39 2024-02-28T16:31:18.855 INFO:tasks.workunit.client.1.smithi066.stdout:0/485: dread d2/d27/d41/d76/d93/d10/f26 [0,4194304] 0 2024-02-28T16:31:18.856 INFO:tasks.workunit.client.0.smithi001.stdout:3/776: readlink d4/d1e/db7/lcb 0 2024-02-28T16:31:18.857 INFO:tasks.workunit.client.1.smithi066.stdout:4/480: link d1/f6 d1/d23/d25/f99 0 2024-02-28T16:31:18.858 INFO:tasks.workunit.client.0.smithi001.stdout:4/855: link d3/d5/d7/d1f/d3a/l71 d3/d5/d47/d90/l119 0 2024-02-28T16:31:18.859 INFO:tasks.workunit.client.0.smithi001.stdout:5/706: dread dc/d12/d56/d96/db9/dc2/dca/d3c/f47 [0,4194304] 0 2024-02-28T16:31:18.859 INFO:tasks.workunit.client.1.smithi066.stdout:1/372: rmdir d2/d27/d2b/d37/d54 39 2024-02-28T16:31:18.862 INFO:tasks.workunit.client.1.smithi066.stdout:3/378: stat db/d13/d1a/d25/c3f 0 2024-02-28T16:31:18.863 INFO:tasks.workunit.client.1.smithi066.stdout:2/378: fdatasync d0/d12/d1a/f2b 0 2024-02-28T16:31:18.866 INFO:tasks.workunit.client.1.smithi066.stdout:0/486: write d2/d27/d41/d76/d93/d10/d23/f9d [650718,6344] 0 2024-02-28T16:31:18.866 INFO:tasks.workunit.client.1.smithi066.stdout:3/379: truncate db/f66 633604 0 2024-02-28T16:31:18.867 INFO:tasks.workunit.client.0.smithi001.stdout:5/707: creat dc/df/d79/d98/fe8 x:0 0 0 2024-02-28T16:31:18.869 INFO:tasks.workunit.client.0.smithi001.stdout:4/856: unlink d3/d2d/d81/cfd 0 2024-02-28T16:31:18.870 INFO:tasks.workunit.client.1.smithi066.stdout:4/481: truncate d1/d18/f2d 5287163 0 2024-02-28T16:31:18.871 INFO:tasks.workunit.client.1.smithi066.stdout:5/353: rename d3/d1b/d20/l3b to d3/d4/dc/dd/l72 0 2024-02-28T16:31:18.871 INFO:tasks.workunit.client.0.smithi001.stdout:3/777: link d4/d1e/d7c/ff6 d4/d6/ffe 0 2024-02-28T16:31:18.873 INFO:tasks.workunit.client.1.smithi066.stdout:4/482: stat d1/d23/f36 0 2024-02-28T16:31:18.873 INFO:tasks.workunit.client.0.smithi001.stdout:3/778: readlink d4/d1e/l7b 0 2024-02-28T16:31:18.874 INFO:tasks.workunit.client.0.smithi001.stdout:4/857: symlink d3/d1c/da0/l11a 0 2024-02-28T16:31:18.876 INFO:tasks.workunit.client.1.smithi066.stdout:5/354: write d3/d4/dc/f3f [891099,19629] 0 2024-02-28T16:31:18.878 INFO:tasks.workunit.client.1.smithi066.stdout:8/349: creat d0/db/f6e x:0 0 0 2024-02-28T16:31:18.878 INFO:tasks.workunit.client.1.smithi066.stdout:3/380: stat db/d13/d1a/l1c 0 2024-02-28T16:31:18.879 INFO:tasks.workunit.client.0.smithi001.stdout:7/766: sync 2024-02-28T16:31:18.879 INFO:tasks.workunit.client.0.smithi001.stdout:6/720: sync 2024-02-28T16:31:18.880 INFO:tasks.workunit.client.0.smithi001.stdout:1/860: sync 2024-02-28T16:31:18.880 INFO:tasks.workunit.client.0.smithi001.stdout:0/734: sync 2024-02-28T16:31:18.883 INFO:tasks.workunit.client.0.smithi001.stdout:5/708: link dc/df/d79/f95 dc/d1a/dba/fe9 0 2024-02-28T16:31:18.883 INFO:tasks.workunit.client.0.smithi001.stdout:1/861: chown d0/d9/ccf 11702 1 2024-02-28T16:31:18.884 INFO:tasks.workunit.client.0.smithi001.stdout:7/767: write d3/d17/d52/d7f/d99/ff5 [101974,100497] 0 2024-02-28T16:31:18.885 INFO:tasks.workunit.client.1.smithi066.stdout:8/350: chown d0/l49 59425700 1 2024-02-28T16:31:18.885 INFO:tasks.workunit.client.0.smithi001.stdout:0/735: read d18/d22/d3d/fba [5516022,30700] 0 2024-02-28T16:31:18.887 INFO:tasks.workunit.client.0.smithi001.stdout:5/709: symlink dc/d12/d56/d96/db8/lea 0 2024-02-28T16:31:18.889 INFO:tasks.workunit.client.0.smithi001.stdout:0/736: chown d18/d22/d2f/d1c/l9c 14673 1 2024-02-28T16:31:18.890 INFO:tasks.workunit.client.0.smithi001.stdout:5/710: write dc/f1d [669748,114814] 0 2024-02-28T16:31:18.893 INFO:tasks.workunit.client.0.smithi001.stdout:0/737: write d18/d22/d3d/fd6 [1703478,124355] 0 2024-02-28T16:31:18.895 INFO:tasks.workunit.client.1.smithi066.stdout:6/321: dread d2/d26/f31 [0,4194304] 0 2024-02-28T16:31:18.907 INFO:tasks.workunit.client.1.smithi066.stdout:1/373: write d2/d27/d2b/d43/f53 [780823,7506] 0 2024-02-28T16:31:18.912 INFO:tasks.workunit.client.1.smithi066.stdout:3/381: symlink db/dc/d35/d3d/l82 0 2024-02-28T16:31:18.913 INFO:tasks.workunit.client.1.smithi066.stdout:0/487: getdents d2/d27/d41/d76/d93/d10/d50/d52 0 2024-02-28T16:31:18.915 INFO:tasks.workunit.client.1.smithi066.stdout:8/351: rename d0/db/d15/d32 to d0/d6/dc/d16/d4c/d6c/d6f 0 2024-02-28T16:31:18.918 INFO:tasks.workunit.client.1.smithi066.stdout:2/379: truncate d0/d12/d1a/d39/d40/f41 723132 0 2024-02-28T16:31:18.920 INFO:tasks.workunit.client.1.smithi066.stdout:4/483: getdents d1/d18/d4e 0 2024-02-28T16:31:18.920 INFO:tasks.workunit.client.1.smithi066.stdout:5/355: dread d3/d4/dc/d56/f67 [0,4194304] 0 2024-02-28T16:31:18.921 INFO:tasks.workunit.client.1.smithi066.stdout:2/380: fsync d0/ff 0 2024-02-28T16:31:18.921 INFO:tasks.workunit.client.1.smithi066.stdout:6/322: write d2/f43 [601510,127499] 0 2024-02-28T16:31:18.922 INFO:tasks.workunit.client.1.smithi066.stdout:3/382: unlink db/d13/d1a/d33/l56 0 2024-02-28T16:31:18.924 INFO:tasks.workunit.client.1.smithi066.stdout:1/374: creat d2/d27/d2b/d43/d46/d68/f76 x:0 0 0 2024-02-28T16:31:18.926 INFO:tasks.workunit.client.1.smithi066.stdout:4/484: chown d1/d18/d7e 47 1 2024-02-28T16:31:18.927 INFO:tasks.workunit.client.1.smithi066.stdout:9/302: sync 2024-02-28T16:31:18.928 INFO:tasks.workunit.client.1.smithi066.stdout:0/488: mkdir d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/da1 0 2024-02-28T16:31:18.933 INFO:tasks.workunit.client.1.smithi066.stdout:8/352: creat d0/d6/dc/d16/d1a/d63/d58/f70 x:0 0 0 2024-02-28T16:31:18.935 INFO:tasks.workunit.client.1.smithi066.stdout:5/356: creat d3/d4/dc/d42/f73 x:0 0 0 2024-02-28T16:31:18.937 INFO:tasks.workunit.client.1.smithi066.stdout:3/383: creat db/dc/d35/d3d/d76/f83 x:0 0 0 2024-02-28T16:31:18.938 INFO:tasks.workunit.client.1.smithi066.stdout:2/381: rename d0/d12/c72 to d0/d12/d1a/d39/d40/c77 0 2024-02-28T16:31:18.939 INFO:tasks.workunit.client.1.smithi066.stdout:8/353: dread - d0/d6/dc/d16/d1a/d63/d58/d5a/f64 zero size 2024-02-28T16:31:18.940 INFO:tasks.workunit.client.1.smithi066.stdout:1/375: rmdir d2/d27/d2b/d37/d51 39 2024-02-28T16:31:18.944 INFO:tasks.workunit.client.1.smithi066.stdout:3/384: chown db/dc/d35/d3d/d76/f83 2239706 1 2024-02-28T16:31:18.944 INFO:tasks.workunit.client.1.smithi066.stdout:8/354: truncate d0/db/f6e 334039 0 2024-02-28T16:31:18.946 INFO:tasks.workunit.client.1.smithi066.stdout:7/282: dread dc/de/d17/d1e/f2d [0,4194304] 0 2024-02-28T16:31:18.947 INFO:tasks.workunit.client.1.smithi066.stdout:4/485: dread d1/f33 [0,4194304] 0 2024-02-28T16:31:18.948 INFO:tasks.workunit.client.1.smithi066.stdout:5/357: mkdir d3/d4/dc/d2f/d55/d74 0 2024-02-28T16:31:18.951 INFO:tasks.workunit.client.1.smithi066.stdout:5/358: dread - d3/d4/dc/d42/f6d zero size 2024-02-28T16:31:18.957 INFO:tasks.workunit.client.1.smithi066.stdout:9/303: dwrite da/df/f2f [0,4194304] 0 2024-02-28T16:31:18.960 INFO:tasks.workunit.client.1.smithi066.stdout:8/355: dread d0/db/d12/d1d/d25/f47 [0,4194304] 0 2024-02-28T16:31:18.963 INFO:tasks.workunit.client.1.smithi066.stdout:1/376: creat d2/d18/d6b/f77 x:0 0 0 2024-02-28T16:31:18.966 INFO:tasks.workunit.client.1.smithi066.stdout:4/486: creat d1/d18/d41/f9a x:0 0 0 2024-02-28T16:31:18.969 INFO:tasks.workunit.client.1.smithi066.stdout:5/359: creat d3/d4/dc/d56/f75 x:0 0 0 2024-02-28T16:31:18.973 INFO:tasks.workunit.client.1.smithi066.stdout:3/385: mkdir db/d13/d14/d84 0 2024-02-28T16:31:18.980 INFO:tasks.workunit.client.1.smithi066.stdout:0/489: truncate d2/f5e 3203395 0 2024-02-28T16:31:18.980 INFO:tasks.workunit.client.1.smithi066.stdout:5/360: unlink d3/d1b/f36 0 2024-02-28T16:31:18.982 INFO:tasks.workunit.client.1.smithi066.stdout:9/304: truncate da/d3a/f47 702977 0 2024-02-28T16:31:18.986 INFO:tasks.workunit.client.1.smithi066.stdout:7/283: rmdir dc/de/d17/d4a 0 2024-02-28T16:31:18.991 INFO:tasks.workunit.client.1.smithi066.stdout:5/361: readlink d3/d4/dc/dd/l72 0 2024-02-28T16:31:18.991 INFO:tasks.workunit.client.1.smithi066.stdout:4/487: rename d1/d23/d25/f28 to d1/f9b 0 2024-02-28T16:31:18.991 INFO:tasks.workunit.client.1.smithi066.stdout:0/490: readlink d2/d27/d41/d76/d93/d10/d23/l8f 0 2024-02-28T16:31:18.992 INFO:tasks.workunit.client.1.smithi066.stdout:6/323: dwrite d2/da/d38/f59 [0,4194304] 0 2024-02-28T16:31:19.000 INFO:tasks.workunit.client.1.smithi066.stdout:2/382: dwrite d0/d12/f47 [4194304,4194304] 0 2024-02-28T16:31:19.003 INFO:tasks.workunit.client.1.smithi066.stdout:7/284: mknod dc/de/d17/c51 0 2024-02-28T16:31:19.006 INFO:tasks.workunit.client.1.smithi066.stdout:8/356: dwrite d0/d6/f1f [4194304,4194304] 0 2024-02-28T16:31:19.006 INFO:tasks.workunit.client.1.smithi066.stdout:4/488: chown d1/d18/d41/f47 532141752 1 2024-02-28T16:31:19.007 INFO:tasks.workunit.client.1.smithi066.stdout:9/305: write da/d4b/f54 [280381,110049] 0 2024-02-28T16:31:19.008 INFO:tasks.workunit.client.1.smithi066.stdout:6/324: truncate d2/d26/f47 731387 0 2024-02-28T16:31:19.011 INFO:tasks.workunit.client.1.smithi066.stdout:0/491: stat d2/d27/d41/d76/d93/d10/d23 0 2024-02-28T16:31:19.014 INFO:tasks.workunit.client.1.smithi066.stdout:7/285: fdatasync dc/d29/f44 0 2024-02-28T16:31:19.017 INFO:tasks.workunit.client.1.smithi066.stdout:1/377: dwrite d2/d27/d2b/d34/f55 [0,4194304] 0 2024-02-28T16:31:19.018 INFO:tasks.workunit.client.1.smithi066.stdout:3/386: dwrite db/d2a/f42 [4194304,4194304] 0 2024-02-28T16:31:19.021 INFO:tasks.workunit.client.1.smithi066.stdout:6/325: mkdir d2/d26/d6b 0 2024-02-28T16:31:19.027 INFO:tasks.workunit.client.1.smithi066.stdout:6/326: dread - d2/d2a/f45 zero size 2024-02-28T16:31:19.027 INFO:tasks.workunit.client.1.smithi066.stdout:2/383: fdatasync d0/d12/f47 0 2024-02-28T16:31:19.029 INFO:tasks.workunit.client.1.smithi066.stdout:9/306: creat da/df/d18/f65 x:0 0 0 2024-02-28T16:31:19.035 INFO:tasks.workunit.client.1.smithi066.stdout:7/286: symlink dc/de/l52 0 2024-02-28T16:31:19.036 INFO:tasks.workunit.client.1.smithi066.stdout:3/387: creat db/d2a/d44/d48/f85 x:0 0 0 2024-02-28T16:31:19.038 INFO:tasks.workunit.client.1.smithi066.stdout:1/378: rmdir d2/d27/d2b/d43 39 2024-02-28T16:31:19.039 INFO:tasks.workunit.client.1.smithi066.stdout:3/388: chown db/d2a/f42 2739 1 2024-02-28T16:31:19.042 INFO:tasks.workunit.client.1.smithi066.stdout:7/287: fsync dc/de/d17/d1e/f3c 0 2024-02-28T16:31:19.044 INFO:tasks.workunit.client.1.smithi066.stdout:0/492: truncate d2/f28 1766816 0 2024-02-28T16:31:19.046 INFO:tasks.workunit.client.1.smithi066.stdout:9/307: creat da/d3a/d15/d48/f66 x:0 0 0 2024-02-28T16:31:19.047 INFO:tasks.workunit.client.1.smithi066.stdout:3/389: fsync db/d13/d1a/d25/f5f 0 2024-02-28T16:31:19.053 INFO:tasks.workunit.client.1.smithi066.stdout:0/493: mkdir d2/d27/d41/d76/d93/d10/d50/d52/da2 0 2024-02-28T16:31:19.055 INFO:tasks.workunit.client.1.smithi066.stdout:9/308: dread - da/df/d18/d2d/f50 zero size 2024-02-28T16:31:19.056 INFO:tasks.workunit.client.1.smithi066.stdout:7/288: truncate dc/de/d17/d1e/f3c 675159 0 2024-02-28T16:31:19.056 INFO:tasks.workunit.client.1.smithi066.stdout:3/390: dread db/f66 [0,4194304] 0 2024-02-28T16:31:19.061 INFO:tasks.workunit.client.1.smithi066.stdout:1/379: dread d2/d27/d2b/d43/f4e [0,4194304] 0 2024-02-28T16:31:19.061 INFO:tasks.workunit.client.1.smithi066.stdout:8/357: dwrite d0/d6/f7 [0,4194304] 0 2024-02-28T16:31:19.062 INFO:tasks.workunit.client.0.smithi001.stdout:9/924: write d5/d27/db1/d5e/dd3/d109/d6f/fc7 [706975,112009] 0 2024-02-28T16:31:19.065 INFO:tasks.workunit.client.1.smithi066.stdout:0/494: creat d2/d27/fa3 x:0 0 0 2024-02-28T16:31:19.066 INFO:tasks.workunit.client.0.smithi001.stdout:7/768: unlink d3/d17/c35 0 2024-02-28T16:31:19.067 INFO:tasks.workunit.client.0.smithi001.stdout:4/858: write d3/d1c/d58/f6b [1871781,33845] 0 2024-02-28T16:31:19.069 INFO:tasks.workunit.client.0.smithi001.stdout:5/711: fdatasync dc/d12/d41/d54/fa5 0 2024-02-28T16:31:19.072 INFO:tasks.workunit.client.1.smithi066.stdout:8/358: chown d0/d6/dc/d16/d4c/d42/l57 3135179 1 2024-02-28T16:31:19.072 INFO:tasks.workunit.client.1.smithi066.stdout:2/384: dread d0/f9 [0,4194304] 0 2024-02-28T16:31:19.072 INFO:tasks.workunit.client.1.smithi066.stdout:0/495: readlink d2/d27/d41/d76/d93/d10/d50/d51/l5c 0 2024-02-28T16:31:19.072 INFO:tasks.workunit.client.1.smithi066.stdout:8/359: stat d0/db/d12/d38 0 2024-02-28T16:31:19.073 INFO:tasks.workunit.client.0.smithi001.stdout:4/859: write d3/d5/fcf [215266,110982] 0 2024-02-28T16:31:19.075 INFO:tasks.workunit.client.0.smithi001.stdout:0/738: creat d18/d22/d2f/ff1 x:0 0 0 2024-02-28T16:31:19.076 INFO:tasks.workunit.client.0.smithi001.stdout:7/769: chown d3/d17/d52/d7f/d99/ff5 0 1 2024-02-28T16:31:19.077 INFO:tasks.workunit.client.1.smithi066.stdout:8/360: read d0/d6/fe [2656664,11573] 0 2024-02-28T16:31:19.077 INFO:tasks.workunit.client.0.smithi001.stdout:8/757: dwrite d4/d32/f50 [0,4194304] 0 2024-02-28T16:31:19.079 INFO:tasks.workunit.client.1.smithi066.stdout:1/380: symlink d2/d18/d6b/l78 0 2024-02-28T16:31:19.079 INFO:tasks.workunit.client.1.smithi066.stdout:8/361: stat d0/d6/dc/d16/f50 0 2024-02-28T16:31:19.080 INFO:tasks.workunit.client.0.smithi001.stdout:2/714: dwrite d0/d2/d3/de/d8a/f70 [0,4194304] 0 2024-02-28T16:31:19.081 INFO:tasks.workunit.client.0.smithi001.stdout:9/925: rename d5/d27/db1/d58/d107 to d5/d27/db1/d127 0 2024-02-28T16:31:19.082 INFO:tasks.workunit.client.1.smithi066.stdout:2/385: stat d0/d12/d1a/d23/d29/d21/d2f/c69 0 2024-02-28T16:31:19.084 INFO:tasks.workunit.client.0.smithi001.stdout:5/712: fdatasync dc/d12/d56/f61 0 2024-02-28T16:31:19.085 INFO:tasks.workunit.client.1.smithi066.stdout:8/362: dread d0/db/d12/f4b [0,4194304] 0 2024-02-28T16:31:19.087 INFO:tasks.workunit.client.1.smithi066.stdout:4/489: truncate d1/d23/f27 637833 0 2024-02-28T16:31:19.088 INFO:tasks.workunit.client.0.smithi001.stdout:0/739: symlink d18/d22/lf2 0 2024-02-28T16:31:19.090 INFO:tasks.workunit.client.1.smithi066.stdout:7/289: dwrite dc/f4b [0,4194304] 0 2024-02-28T16:31:19.091 INFO:tasks.workunit.client.0.smithi001.stdout:2/715: creat d0/d2/d3/de/d22/dbb/dc8/fef x:0 0 0 2024-02-28T16:31:19.096 INFO:tasks.workunit.client.1.smithi066.stdout:5/362: truncate d3/d4/dc/d2f/f32 1365114 0 2024-02-28T16:31:19.098 INFO:tasks.workunit.client.1.smithi066.stdout:7/290: chown dc/de/l18 39128 1 2024-02-28T16:31:19.100 INFO:tasks.workunit.client.0.smithi001.stdout:9/926: creat d5/d27/db1/d33/d115/d11f/f128 x:0 0 0 2024-02-28T16:31:19.102 INFO:tasks.workunit.client.1.smithi066.stdout:6/327: dwrite d2/d26/f4e [0,4194304] 0 2024-02-28T16:31:19.104 INFO:tasks.workunit.client.0.smithi001.stdout:5/713: creat dc/d12/d56/feb x:0 0 0 2024-02-28T16:31:19.107 INFO:tasks.workunit.client.1.smithi066.stdout:0/496: write d2/d27/d41/d76/d93/f20 [2140819,120389] 0 2024-02-28T16:31:19.109 INFO:tasks.workunit.client.1.smithi066.stdout:9/309: dwrite da/fc [0,4194304] 0 2024-02-28T16:31:19.112 INFO:tasks.workunit.client.0.smithi001.stdout:1/862: dread d0/d9/d1a/d2b/d26/d27/d30/f76 [0,4194304] 0 2024-02-28T16:31:19.116 INFO:tasks.workunit.client.0.smithi001.stdout:8/758: dwrite d4/d32/dd6/fe6 [0,4194304] 0 2024-02-28T16:31:19.118 INFO:tasks.workunit.client.0.smithi001.stdout:0/740: rename d18/d22/d2f/d1c/fc7 to d18/d22/d2f/ff3 0 2024-02-28T16:31:19.118 INFO:tasks.workunit.client.1.smithi066.stdout:8/363: write d0/d6/f7 [2932983,39947] 0 2024-02-28T16:31:19.119 INFO:tasks.workunit.client.1.smithi066.stdout:2/386: rename d0/d12/d1a/d23/d29/c33 to d0/d12/d1a/d23/c78 0 2024-02-28T16:31:19.119 INFO:tasks.workunit.client.1.smithi066.stdout:4/490: dread d1/d23/d25/d43/f5e [0,4194304] 0 2024-02-28T16:31:19.119 INFO:tasks.workunit.client.0.smithi001.stdout:2/716: link d0/d2/d3/c3e d0/d2/dd9/cf0 0 2024-02-28T16:31:19.119 INFO:tasks.workunit.client.1.smithi066.stdout:3/391: truncate db/d2a/f42 5538221 0 2024-02-28T16:31:19.123 INFO:tasks.workunit.client.0.smithi001.stdout:9/927: truncate d5/d27/db1/d33/d115/d11f/de0/f11e 1042447 0 2024-02-28T16:31:19.124 INFO:tasks.workunit.client.0.smithi001.stdout:4/860: dwrite d3/d2d/d4c/d69/d82/f77 [0,4194304] 0 2024-02-28T16:31:19.125 INFO:tasks.workunit.client.0.smithi001.stdout:7/770: dwrite d3/d17/d52/d7f/d99/dc1/d49/fd8 [0,4194304] 0 2024-02-28T16:31:19.127 INFO:tasks.workunit.client.1.smithi066.stdout:5/363: mknod d3/d4/dc/dd/c76 0 2024-02-28T16:31:19.133 INFO:tasks.workunit.client.1.smithi066.stdout:6/328: dread d2/da/f22 [4194304,4194304] 0 2024-02-28T16:31:19.133 INFO:tasks.workunit.client.0.smithi001.stdout:5/714: rename dc/c86 to dc/df/cec 0 2024-02-28T16:31:19.134 INFO:tasks.workunit.client.0.smithi001.stdout:8/759: truncate d4/d32/f5c 162728 0 2024-02-28T16:31:19.136 INFO:tasks.workunit.client.1.smithi066.stdout:6/329: rename d2/da to d2/da/d19/d54/d6c 22 2024-02-28T16:31:19.139 INFO:tasks.workunit.client.1.smithi066.stdout:1/381: dwrite d2/d27/d2b/d37/d48/f66 [0,4194304] 0 2024-02-28T16:31:19.141 INFO:tasks.workunit.client.1.smithi066.stdout:0/497: unlink d2/d27/d41/d76/d93/d10/f4e 0 2024-02-28T16:31:19.142 INFO:tasks.workunit.client.0.smithi001.stdout:2/717: creat d0/d2/d3/d36/d94/ff1 x:0 0 0 2024-02-28T16:31:19.142 INFO:tasks.workunit.client.0.smithi001.stdout:1/863: symlink d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/d9c/l12a 0 2024-02-28T16:31:19.144 INFO:tasks.workunit.client.0.smithi001.stdout:8/760: fsync d4/d7/d22/d27/d58/f7b 0 2024-02-28T16:31:19.147 INFO:tasks.workunit.client.1.smithi066.stdout:6/330: read d2/d26/f2d [812580,105066] 0 2024-02-28T16:31:19.149 INFO:tasks.workunit.client.1.smithi066.stdout:4/491: rmdir d1/d23/d25/d43 39 2024-02-28T16:31:19.153 INFO:tasks.workunit.client.0.smithi001.stdout:7/771: mknod d3/d17/d52/d7f/d99/dc1/d49/cf8 0 2024-02-28T16:31:19.156 INFO:tasks.workunit.client.0.smithi001.stdout:2/718: fdatasync d0/d2/d3/de/d61/d12/fe0 0 2024-02-28T16:31:19.159 INFO:tasks.workunit.client.0.smithi001.stdout:8/761: mkdir d4/d47/df4 0 2024-02-28T16:31:19.160 INFO:tasks.workunit.client.0.smithi001.stdout:9/928: dwrite d5/d27/db1/d5e/dd3/d109/d6f/f7f [4194304,4194304] 0 2024-02-28T16:31:19.168 INFO:tasks.workunit.client.1.smithi066.stdout:3/392: unlink db/d13/d1a/d33/c64 0 2024-02-28T16:31:19.172 INFO:tasks.workunit.client.1.smithi066.stdout:5/364: mknod d3/d4/dc/dd/c77 0 2024-02-28T16:31:19.177 INFO:tasks.workunit.client.0.smithi001.stdout:4/861: write d3/d1c/fc2 [154097,34351] 0 2024-02-28T16:31:19.177 INFO:tasks.workunit.client.0.smithi001.stdout:9/929: mkdir d5/d27/db1/d65/d129 0 2024-02-28T16:31:19.182 INFO:tasks.workunit.client.1.smithi066.stdout:2/387: dwrite d0/d12/d2e/d3b/f64 [0,4194304] 0 2024-02-28T16:31:19.183 INFO:tasks.workunit.client.0.smithi001.stdout:8/762: mkdir d4/d47/df4/df5 0 2024-02-28T16:31:19.183 INFO:tasks.workunit.client.0.smithi001.stdout:5/715: write dc/d1a/f7b [693703,18240] 0 2024-02-28T16:31:19.184 INFO:tasks.workunit.client.1.smithi066.stdout:9/310: mknod da/d3a/c67 0 2024-02-28T16:31:19.188 INFO:tasks.workunit.client.1.smithi066.stdout:1/382: unlink d2/d18/f4f 0 2024-02-28T16:31:19.188 INFO:tasks.workunit.client.0.smithi001.stdout:9/930: creat d5/d27/db1/da6/db2/de4/f12a x:0 0 0 2024-02-28T16:31:19.189 INFO:tasks.workunit.client.1.smithi066.stdout:2/388: chown d0/d12/d1a/d39/d40/f45 1 1 2024-02-28T16:31:19.190 INFO:tasks.workunit.client.0.smithi001.stdout:5/716: stat dc/d12/daa/lb0 0 2024-02-28T16:31:19.196 INFO:tasks.workunit.client.0.smithi001.stdout:9/931: fdatasync d5/d27/db1/d5e/dd3/d109/d6f/fc7 0 2024-02-28T16:31:19.198 INFO:tasks.workunit.client.1.smithi066.stdout:6/331: symlink d2/d26/d32/d46/l6d 0 2024-02-28T16:31:19.199 INFO:tasks.workunit.client.0.smithi001.stdout:7/772: dwrite d3/fa4 [0,4194304] 0 2024-02-28T16:31:19.199 INFO:tasks.workunit.client.0.smithi001.stdout:1/864: dread d0/d9/d1a/d2b/f4d [0,4194304] 0 2024-02-28T16:31:19.201 INFO:tasks.workunit.client.1.smithi066.stdout:0/498: rename d2/d27/d41/d76/d93/d33/l5d to d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/la4 0 2024-02-28T16:31:19.202 INFO:tasks.workunit.client.0.smithi001.stdout:5/717: chown dc/d12/d56/d96/db9/ddd/lcc 4016 1 2024-02-28T16:31:19.203 INFO:tasks.workunit.client.1.smithi066.stdout:6/332: fsync d2/d26/d32/d46/f60 0 2024-02-28T16:31:19.209 INFO:tasks.workunit.client.0.smithi001.stdout:6/721: sync 2024-02-28T16:31:19.209 INFO:tasks.workunit.client.0.smithi001.stdout:7/773: mknod d3/d17/d61/cf9 0 2024-02-28T16:31:19.210 INFO:tasks.workunit.client.0.smithi001.stdout:3/779: sync 2024-02-28T16:31:19.212 INFO:tasks.workunit.client.0.smithi001.stdout:9/932: creat d5/d27/db1/d5e/dd3/d109/df1/f12b x:0 0 0 2024-02-28T16:31:19.213 INFO:tasks.workunit.client.1.smithi066.stdout:9/311: dwrite da/d3a/f13 [4194304,4194304] 0 2024-02-28T16:31:19.216 INFO:tasks.workunit.client.0.smithi001.stdout:4/862: dwrite d3/f78 [0,4194304] 0 2024-02-28T16:31:19.219 INFO:tasks.workunit.client.0.smithi001.stdout:2/719: dwrite d0/d2/d3/de/d22/f24 [0,4194304] 0 2024-02-28T16:31:19.222 INFO:tasks.workunit.client.0.smithi001.stdout:2/720: readlink d0/d2/l2d 0 2024-02-28T16:31:19.233 INFO:tasks.workunit.client.1.smithi066.stdout:7/291: dread f5 [0,4194304] 0 2024-02-28T16:31:19.233 INFO:tasks.workunit.client.1.smithi066.stdout:1/383: dwrite d2/d18/d19/f24 [4194304,4194304] 0 2024-02-28T16:31:19.233 INFO:tasks.workunit.client.1.smithi066.stdout:5/365: rename d3/l14 to d3/d4/dc/dd/d41/l78 0 2024-02-28T16:31:19.234 INFO:tasks.workunit.client.0.smithi001.stdout:1/865: symlink d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/l12b 0 2024-02-28T16:31:19.235 INFO:tasks.workunit.client.0.smithi001.stdout:3/780: creat d4/d1e/d7c/d7d/df8/fff x:0 0 0 2024-02-28T16:31:19.235 INFO:tasks.workunit.client.0.smithi001.stdout:0/741: dread d18/d22/d2b/d36/d48/f9a [0,4194304] 0 2024-02-28T16:31:19.235 INFO:tasks.workunit.client.0.smithi001.stdout:9/933: creat d5/d27/db1/d33/d115/d11f/da8/dd1/f12c x:0 0 0 2024-02-28T16:31:19.237 INFO:tasks.workunit.client.0.smithi001.stdout:7/774: creat d3/db7/ffa x:0 0 0 2024-02-28T16:31:19.238 INFO:tasks.workunit.client.1.smithi066.stdout:5/366: chown d3/d4/dc/d56/c6b 13476324 1 2024-02-28T16:31:19.239 INFO:tasks.workunit.client.0.smithi001.stdout:0/742: chown d18/d22/d2f/d1c 4542196 1 2024-02-28T16:31:19.240 INFO:tasks.workunit.client.1.smithi066.stdout:5/367: readlink d3/d4/dc/dd/d35/l54 0 2024-02-28T16:31:19.244 INFO:tasks.workunit.client.1.smithi066.stdout:8/364: truncate d0/f43 1152621 0 2024-02-28T16:31:19.244 INFO:tasks.workunit.client.0.smithi001.stdout:5/718: write dc/d12/d41/d54/fa5 [1404137,118287] 0 2024-02-28T16:31:19.245 INFO:tasks.workunit.client.1.smithi066.stdout:6/333: mkdir d2/d2c/d6e 0 2024-02-28T16:31:19.245 INFO:tasks.workunit.client.1.smithi066.stdout:0/499: dwrite d2/d85/f88 [0,4194304] 0 2024-02-28T16:31:19.247 INFO:tasks.workunit.client.0.smithi001.stdout:9/934: creat d5/d27/db1/d127/f12d x:0 0 0 2024-02-28T16:31:19.248 INFO:tasks.workunit.client.0.smithi001.stdout:8/763: write d4/d7/f1d [2194408,18535] 0 2024-02-28T16:31:19.250 INFO:tasks.workunit.client.1.smithi066.stdout:6/334: truncate d2/d2c/f51 997174 0 2024-02-28T16:31:19.252 INFO:tasks.workunit.client.0.smithi001.stdout:3/781: unlink d4/d6/c74 0 2024-02-28T16:31:19.253 INFO:tasks.workunit.client.1.smithi066.stdout:6/335: read - d2/da/d19/f50 zero size 2024-02-28T16:31:19.255 INFO:tasks.workunit.client.0.smithi001.stdout:2/721: dwrite d0/d2/da6/fc4 [4194304,4194304] 0 2024-02-28T16:31:19.256 INFO:tasks.workunit.client.0.smithi001.stdout:5/719: rmdir dc/d12/d56/d96/db9/dc2/dca/d3c/d4a 39 2024-02-28T16:31:19.258 INFO:tasks.workunit.client.1.smithi066.stdout:8/365: rename d0/d6/dc/d16/d4c/d6c/d6f/c68 to d0/db/d12/d38/c71 0 2024-02-28T16:31:19.261 INFO:tasks.workunit.client.1.smithi066.stdout:8/366: read - d0/d6/f5d zero size 2024-02-28T16:31:19.261 INFO:tasks.workunit.client.1.smithi066.stdout:1/384: unlink d2/d18/d19/c6f 0 2024-02-28T16:31:19.262 INFO:tasks.workunit.client.0.smithi001.stdout:7/775: dread d3/d17/f2a [0,4194304] 0 2024-02-28T16:31:19.262 INFO:tasks.workunit.client.0.smithi001.stdout:5/720: chown dc/d12/f24 905680 1 2024-02-28T16:31:19.264 INFO:tasks.workunit.client.1.smithi066.stdout:4/492: dwrite d1/f6 [0,4194304] 0 2024-02-28T16:31:19.265 INFO:tasks.workunit.client.1.smithi066.stdout:8/367: write d0/d6/fe [3959178,38378] 0 2024-02-28T16:31:19.265 INFO:tasks.workunit.client.1.smithi066.stdout:1/385: write d2/d18/f26 [432165,81577] 0 2024-02-28T16:31:19.267 INFO:tasks.workunit.client.0.smithi001.stdout:8/764: mkdir d4/d7/d22/df6 0 2024-02-28T16:31:19.269 INFO:tasks.workunit.client.1.smithi066.stdout:0/500: rename d2/c49 to d2/d27/d41/d76/d93/d10/d23/ca5 0 2024-02-28T16:31:19.275 INFO:tasks.workunit.client.1.smithi066.stdout:9/312: truncate da/df/d18/d28/f2b 1571057 0 2024-02-28T16:31:19.275 INFO:tasks.workunit.client.1.smithi066.stdout:7/292: getdents dc/de/d17/d37 0 2024-02-28T16:31:19.277 INFO:tasks.workunit.client.1.smithi066.stdout:8/368: mkdir d0/db/d12/d1d/d25/d72 0 2024-02-28T16:31:19.278 INFO:tasks.workunit.client.1.smithi066.stdout:4/493: mkdir d1/d23/d25/d3e/d40/d9c 0 2024-02-28T16:31:19.279 INFO:tasks.workunit.client.0.smithi001.stdout:5/721: creat dc/fed x:0 0 0 2024-02-28T16:31:19.279 INFO:tasks.workunit.client.0.smithi001.stdout:4/863: dwrite d3/d1c/d58/d6e/f7a [0,4194304] 0 2024-02-28T16:31:19.280 INFO:tasks.workunit.client.0.smithi001.stdout:7/776: creat d3/d17/d3d/d42/ffb x:0 0 0 2024-02-28T16:31:19.281 INFO:tasks.workunit.client.1.smithi066.stdout:6/336: creat d2/f6f x:0 0 0 2024-02-28T16:31:19.282 INFO:tasks.workunit.client.1.smithi066.stdout:2/389: dwrite d0/d12/d1a/d39/d40/f55 [0,4194304] 0 2024-02-28T16:31:19.283 INFO:tasks.workunit.client.1.smithi066.stdout:8/369: write d0/d6/dc/d16/f50 [399915,72813] 0 2024-02-28T16:31:19.284 INFO:tasks.workunit.client.0.smithi001.stdout:1/866: dwrite d0/d9/d56/d2f/f107 [0,4194304] 0 2024-02-28T16:31:19.285 INFO:tasks.workunit.client.1.smithi066.stdout:1/386: mknod d2/d27/c79 0 2024-02-28T16:31:19.285 INFO:tasks.workunit.client.1.smithi066.stdout:7/293: creat dc/de/d17/d1e/f53 x:0 0 0 2024-02-28T16:31:19.288 INFO:tasks.workunit.client.1.smithi066.stdout:9/313: mkdir da/df/d18/d28/d42/d68 0 2024-02-28T16:31:19.290 INFO:tasks.workunit.client.1.smithi066.stdout:9/314: read - da/df/f29 zero size 2024-02-28T16:31:19.291 INFO:tasks.workunit.client.1.smithi066.stdout:1/387: truncate d2/d27/d2b/d43/d46/f61 4160989 0 2024-02-28T16:31:19.291 INFO:tasks.workunit.client.0.smithi001.stdout:4/864: chown d3/d5/d7/d1f/d2e/dbe/c6d 18355764 1 2024-02-28T16:31:19.292 INFO:tasks.workunit.client.1.smithi066.stdout:2/390: chown d0/d12/d1a/d23/d29/d21/d2f/c69 0 1 2024-02-28T16:31:19.294 INFO:tasks.workunit.client.1.smithi066.stdout:0/501: rename d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/la4 to d2/d85/d8d/la6 0 2024-02-28T16:31:19.295 INFO:tasks.workunit.client.0.smithi001.stdout:6/722: dread d9/db/d10/f46 [0,4194304] 0 2024-02-28T16:31:19.295 INFO:tasks.workunit.client.0.smithi001.stdout:7/777: unlink d3/d17/d3d/d42/l4c 0 2024-02-28T16:31:19.295 INFO:tasks.workunit.client.0.smithi001.stdout:1/867: rename d0/d9/d1a/d2b/d26/d27/d30/d67/d87/lc5 to d0/d9/d1a/d72/l12c 0 2024-02-28T16:31:19.296 INFO:tasks.workunit.client.1.smithi066.stdout:6/337: creat d2/d26/d6b/f70 x:0 0 0 2024-02-28T16:31:19.297 INFO:tasks.workunit.client.0.smithi001.stdout:6/723: readlink d9/de/la3 0 2024-02-28T16:31:19.299 INFO:tasks.workunit.client.1.smithi066.stdout:9/315: creat da/df/d18/d2d/f69 x:0 0 0 2024-02-28T16:31:19.301 INFO:tasks.workunit.client.0.smithi001.stdout:4/865: mknod d3/d5/d7/d1f/d3a/c11b 0 2024-02-28T16:31:19.301 INFO:tasks.workunit.client.0.smithi001.stdout:6/724: chown d9/db/d5f/cb7 0 1 2024-02-28T16:31:19.301 INFO:tasks.workunit.client.0.smithi001.stdout:7/778: mknod d3/d17/d61/cfc 0 2024-02-28T16:31:19.303 INFO:tasks.workunit.client.1.smithi066.stdout:7/294: rename dc/f16 to dc/de/d17/d37/f54 0 2024-02-28T16:31:19.307 INFO:tasks.workunit.client.1.smithi066.stdout:9/316: chown da/df/d18/d2d/f50 1750 1 2024-02-28T16:31:19.307 INFO:tasks.workunit.client.1.smithi066.stdout:8/370: truncate d0/d6/dc/d16/d1a/d63/d58/d5a/f64 445849 0 2024-02-28T16:31:19.308 INFO:tasks.workunit.client.1.smithi066.stdout:4/494: dread d1/d23/d25/d3e/f57 [0,4194304] 0 2024-02-28T16:31:19.310 INFO:tasks.workunit.client.0.smithi001.stdout:6/725: mkdir d9/d29/d6a/ded 0 2024-02-28T16:31:19.311 INFO:tasks.workunit.client.0.smithi001.stdout:4/866: read d3/d1c/d58/ff [142500,118369] 0 2024-02-28T16:31:19.313 INFO:tasks.workunit.client.0.smithi001.stdout:6/726: readlink d9/d29/d6e/d4e/l51 0 2024-02-28T16:31:19.314 INFO:tasks.workunit.client.1.smithi066.stdout:4/495: fdatasync d1/d7a/f90 0 2024-02-28T16:31:19.315 INFO:tasks.workunit.client.1.smithi066.stdout:8/371: fdatasync d0/d6/dc/d16/d1a/d63/d58/f70 0 2024-02-28T16:31:19.316 INFO:tasks.workunit.client.0.smithi001.stdout:6/727: fsync d9/fab 0 2024-02-28T16:31:19.322 INFO:tasks.workunit.client.0.smithi001.stdout:5/722: dread dc/d1a/f7b [0,4194304] 0 2024-02-28T16:31:19.324 INFO:tasks.workunit.client.1.smithi066.stdout:6/338: unlink d2/f8 0 2024-02-28T16:31:19.329 INFO:tasks.workunit.client.1.smithi066.stdout:6/339: write d2/d26/f5a [217292,90554] 0 2024-02-28T16:31:19.332 INFO:tasks.workunit.client.0.smithi001.stdout:4/867: dwrite d3/d2d/d4c/d69/ff9 [0,4194304] 0 2024-02-28T16:31:19.336 INFO:tasks.workunit.client.1.smithi066.stdout:2/391: creat d0/d12/d1a/f79 x:0 0 0 2024-02-28T16:31:19.336 INFO:tasks.workunit.client.1.smithi066.stdout:9/317: creat da/d3a/d15/f6a x:0 0 0 2024-02-28T16:31:19.341 INFO:tasks.workunit.client.0.smithi001.stdout:7/779: unlink d3/d17/d52/d7f/le8 0 2024-02-28T16:31:19.341 INFO:tasks.workunit.client.0.smithi001.stdout:1/868: rename d0/d9/d1a/d2b/d26/d27/d30/d67/d87/f9a to d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/f12d 0 2024-02-28T16:31:19.342 INFO:tasks.workunit.client.1.smithi066.stdout:8/372: dread d0/db/d12/f4b [0,4194304] 0 2024-02-28T16:31:19.342 INFO:tasks.workunit.client.1.smithi066.stdout:4/496: dwrite d1/f10 [0,4194304] 0 2024-02-28T16:31:19.350 INFO:tasks.workunit.client.0.smithi001.stdout:6/728: symlink d9/db/d5f/d62/d83/lee 0 2024-02-28T16:31:19.350 INFO:tasks.workunit.client.1.smithi066.stdout:1/388: rename d2/d27/d2b/d37/d71 to d2/d18/d7a 0 2024-02-28T16:31:19.351 INFO:tasks.workunit.client.0.smithi001.stdout:5/723: rmdir dc/d12/d56/d96/db8/dd7 39 2024-02-28T16:31:19.352 INFO:tasks.workunit.client.1.smithi066.stdout:6/340: symlink d2/d2c/d6e/l71 0 2024-02-28T16:31:19.353 INFO:tasks.workunit.client.0.smithi001.stdout:3/782: write d4/f14 [4454042,94275] 0 2024-02-28T16:31:19.354 INFO:tasks.workunit.client.1.smithi066.stdout:8/373: chown d0/d6/dc/d16/d1a/l33 7 1 2024-02-28T16:31:19.356 INFO:tasks.workunit.client.1.smithi066.stdout:6/341: write d2/d2a/d3d/f48 [2167101,42671] 0 2024-02-28T16:31:19.357 INFO:tasks.workunit.client.1.smithi066.stdout:9/318: mkdir da/d6b 0 2024-02-28T16:31:19.358 INFO:tasks.workunit.client.0.smithi001.stdout:4/868: unlink d3/d5/d7/d1f/d3a/fad 0 2024-02-28T16:31:19.358 INFO:tasks.workunit.client.0.smithi001.stdout:5/724: write dc/d12/d41/d54/fa5 [3786197,93695] 0 2024-02-28T16:31:19.360 INFO:tasks.workunit.client.0.smithi001.stdout:4/869: fdatasync d3/d1c/fc2 0 2024-02-28T16:31:19.366 INFO:tasks.workunit.client.1.smithi066.stdout:7/295: dread dc/de/d17/d1e/f28 [0,4194304] 0 2024-02-28T16:31:19.366 INFO:tasks.workunit.client.0.smithi001.stdout:9/935: dwrite d5/d27/d34/fc4 [0,4194304] 0 2024-02-28T16:31:19.369 INFO:tasks.workunit.client.1.smithi066.stdout:4/497: dwrite d1/d23/d25/d3e/d40/f72 [0,4194304] 0 2024-02-28T16:31:19.369 INFO:tasks.workunit.client.0.smithi001.stdout:9/936: write d5/f30 [4020545,79998] 0 2024-02-28T16:31:19.369 INFO:tasks.workunit.client.0.smithi001.stdout:7/780: symlink d3/d17/d3d/d59/d6e/lfd 0 2024-02-28T16:31:19.371 INFO:tasks.workunit.client.1.smithi066.stdout:7/296: write dc/f4b [5024425,498] 0 2024-02-28T16:31:19.372 INFO:tasks.workunit.client.1.smithi066.stdout:8/374: mkdir d0/d6/dc/d16/d4c/d42/d73 0 2024-02-28T16:31:19.374 INFO:tasks.workunit.client.0.smithi001.stdout:9/937: chown d5/d27/db1/d5e/dd3/d109/d6f/fc6 4185347 1 2024-02-28T16:31:19.376 INFO:tasks.workunit.client.1.smithi066.stdout:8/375: chown d0/d6/dc 1748760478 1 2024-02-28T16:31:19.377 INFO:tasks.workunit.client.0.smithi001.stdout:2/722: dwrite d0/d2/d3/de/d61/f67 [0,4194304] 0 2024-02-28T16:31:19.378 INFO:tasks.workunit.client.1.smithi066.stdout:2/392: dwrite d0/d12/d1a/d23/d29/d21/f75 [0,4194304] 0 2024-02-28T16:31:19.380 INFO:tasks.workunit.client.0.smithi001.stdout:9/938: dread - d5/d27/db1/d33/d115/d11f/da8/ddf/f108 zero size 2024-02-28T16:31:19.381 INFO:tasks.workunit.client.1.smithi066.stdout:8/376: write d0/db/f5f [240048,100488] 0 2024-02-28T16:31:19.382 INFO:tasks.workunit.client.0.smithi001.stdout:1/869: getdents d0/d9/d1a/d2b/d26/d105/d121 0 2024-02-28T16:31:19.383 INFO:tasks.workunit.client.0.smithi001.stdout:4/870: symlink d3/d5/d7/d1f/d3a/l11c 0 2024-02-28T16:31:19.383 INFO:tasks.workunit.client.0.smithi001.stdout:2/723: fdatasync d0/d2/d3/de/d22/dbb/dc8/fef 0 2024-02-28T16:31:19.384 INFO:tasks.workunit.client.1.smithi066.stdout:2/393: chown d0/d12/d1a/d23/d29/d21/f75 0 1 2024-02-28T16:31:19.388 INFO:tasks.workunit.client.0.smithi001.stdout:7/781: mkdir d3/d17/d52/d55/dca/dfe 0 2024-02-28T16:31:19.390 INFO:tasks.workunit.client.0.smithi001.stdout:2/724: fdatasync d0/d2/d3/de/d22/fd8 0 2024-02-28T16:31:19.391 INFO:tasks.workunit.client.0.smithi001.stdout:6/729: dwrite d9/d29/d6e/d4e/fd0 [0,4194304] 0 2024-02-28T16:31:19.392 INFO:tasks.workunit.client.1.smithi066.stdout:7/297: symlink dc/d29/l55 0 2024-02-28T16:31:19.392 INFO:tasks.workunit.client.1.smithi066.stdout:1/389: creat d2/d27/f7b x:0 0 0 2024-02-28T16:31:19.395 INFO:tasks.workunit.client.0.smithi001.stdout:8/765: write d4/d7/f20 [1775498,72212] 0 2024-02-28T16:31:19.395 INFO:tasks.workunit.client.0.smithi001.stdout:9/939: chown d5/d27/d34/dc1/c11d 10067905 1 2024-02-28T16:31:19.396 INFO:tasks.workunit.client.0.smithi001.stdout:4/871: mknod d3/d1c/d58/d6c/c11d 0 2024-02-28T16:31:19.396 INFO:tasks.workunit.client.1.smithi066.stdout:7/298: mkdir dc/d56 0 2024-02-28T16:31:19.400 INFO:tasks.workunit.client.0.smithi001.stdout:5/725: creat dc/d12/d56/d96/db9/dc2/dca/d3c/fee x:0 0 0 2024-02-28T16:31:19.405 INFO:tasks.workunit.client.1.smithi066.stdout:7/299: link dc/l42 dc/de/d17/d1e/d45/l57 0 2024-02-28T16:31:19.406 INFO:tasks.workunit.client.0.smithi001.stdout:5/726: stat dc/d12/d56/fdb 0 2024-02-28T16:31:19.407 INFO:tasks.workunit.client.0.smithi001.stdout:6/730: mkdir d9/db/d5f/d62/def 0 2024-02-28T16:31:19.407 INFO:tasks.workunit.client.0.smithi001.stdout:2/725: creat d0/d2/d3/d36/ff2 x:0 0 0 2024-02-28T16:31:19.408 INFO:tasks.workunit.client.0.smithi001.stdout:5/727: stat dc 0 2024-02-28T16:31:19.409 INFO:tasks.workunit.client.1.smithi066.stdout:7/300: write f1 [4224092,4741] 0 2024-02-28T16:31:19.409 INFO:tasks.workunit.client.0.smithi001.stdout:8/766: creat d4/d7/d22/d27/d4b/d77/ff7 x:0 0 0 2024-02-28T16:31:19.411 INFO:tasks.workunit.client.1.smithi066.stdout:7/301: write f1 [4966885,104198] 0 2024-02-28T16:31:19.413 INFO:tasks.workunit.client.0.smithi001.stdout:4/872: rmdir d3/d5/d7/d1f/dc3 39 2024-02-28T16:31:19.414 INFO:tasks.workunit.client.0.smithi001.stdout:1/870: creat d0/d9/d1a/d2b/d26/d27/f12e x:0 0 0 2024-02-28T16:31:19.417 INFO:tasks.workunit.client.0.smithi001.stdout:2/726: chown d0/l26 46409339 1 2024-02-28T16:31:19.417 INFO:tasks.workunit.client.0.smithi001.stdout:1/871: chown d0/d9/l5e 241 1 2024-02-28T16:31:19.419 INFO:tasks.workunit.client.1.smithi066.stdout:7/302: write f1 [5100193,88887] 0 2024-02-28T16:31:19.419 INFO:tasks.workunit.client.1.smithi066.stdout:1/390: dwrite d2/d18/d19/f33 [0,4194304] 0 2024-02-28T16:31:19.423 INFO:tasks.workunit.client.1.smithi066.stdout:7/303: write f1 [1924540,48896] 0 2024-02-28T16:31:19.423 INFO:tasks.workunit.client.1.smithi066.stdout:6/342: rmdir d2/d2c 39 2024-02-28T16:31:19.431 INFO:tasks.workunit.client.0.smithi001.stdout:0/743: sync 2024-02-28T16:31:19.431 INFO:tasks.workunit.client.1.smithi066.stdout:1/391: mkdir d2/d27/d2b/d43/d46/d7c 0 2024-02-28T16:31:19.432 INFO:tasks.workunit.client.0.smithi001.stdout:5/728: read dc/d12/d56/d96/db9/dc2/dca/f76 [215595,4421] 0 2024-02-28T16:31:19.432 INFO:tasks.workunit.client.1.smithi066.stdout:2/394: dwrite d0/d12/d1a/d23/d5d/f4d [0,4194304] 0 2024-02-28T16:31:19.435 INFO:tasks.workunit.client.1.smithi066.stdout:6/343: dread - d2/f23 zero size 2024-02-28T16:31:19.438 INFO:tasks.workunit.client.0.smithi001.stdout:0/744: fsync d18/d22/d2b/d36/d48/f55 0 2024-02-28T16:31:19.438 INFO:tasks.workunit.client.1.smithi066.stdout:7/304: mknod dc/de/d17/d1e/d45/c58 0 2024-02-28T16:31:19.441 INFO:tasks.workunit.client.0.smithi001.stdout:7/782: dwrite d3/d17/d3d/f72 [4194304,4194304] 0 2024-02-28T16:31:19.445 INFO:tasks.workunit.client.0.smithi001.stdout:7/783: fsync d3/d1c/f33 0 2024-02-28T16:31:19.446 INFO:tasks.workunit.client.0.smithi001.stdout:0/745: rmdir d18/d22/d76/d7a 39 2024-02-28T16:31:19.448 INFO:tasks.workunit.client.0.smithi001.stdout:2/727: getdents d0/d2/d3/de/d61/d12/de8 0 2024-02-28T16:31:19.449 INFO:tasks.workunit.client.1.smithi066.stdout:2/395: fsync d0/d12/d1a/d39/d40/f41 0 2024-02-28T16:31:19.449 INFO:tasks.workunit.client.0.smithi001.stdout:7/784: write d3/d17/d3d/fa6 [1422216,19503] 0 2024-02-28T16:31:19.451 INFO:tasks.workunit.client.1.smithi066.stdout:2/396: chown d0/d12/d2e/d3b/d66 2439 1 2024-02-28T16:31:19.452 INFO:tasks.workunit.client.0.smithi001.stdout:1/872: symlink d0/d9/d1a/d2b/df8/l12f 0 2024-02-28T16:31:19.453 INFO:tasks.workunit.client.1.smithi066.stdout:4/498: truncate d1/d23/d25/d43/f82 688447 0 2024-02-28T16:31:19.454 INFO:tasks.workunit.client.1.smithi066.stdout:1/392: link d2/d18/c25 d2/d18/d19/c7d 0 2024-02-28T16:31:19.457 INFO:tasks.workunit.client.1.smithi066.stdout:2/397: truncate d0/d12/d3e/f4e 1253482 0 2024-02-28T16:31:19.459 INFO:tasks.workunit.client.1.smithi066.stdout:0/502: dread d2/d27/d41/d76/d93/f20 [0,4194304] 0 2024-02-28T16:31:19.461 INFO:tasks.workunit.client.1.smithi066.stdout:4/499: chown d1/d23/f27 24481075 1 2024-02-28T16:31:19.462 INFO:tasks.workunit.client.1.smithi066.stdout:2/398: readlink d0/d12/d3e/l74 0 2024-02-28T16:31:19.467 INFO:tasks.workunit.client.1.smithi066.stdout:1/393: rename d2/f10 to d2/d27/d2b/d43/d46/d68/f7e 0 2024-02-28T16:31:19.472 INFO:tasks.workunit.client.1.smithi066.stdout:4/500: creat d1/d18/d55/d81/f9d x:0 0 0 2024-02-28T16:31:19.473 INFO:tasks.workunit.client.0.smithi001.stdout:5/729: mknod dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/cef 0 2024-02-28T16:31:19.474 INFO:tasks.workunit.client.0.smithi001.stdout:2/728: dread d0/d2/d3/de/d8a/f70 [0,4194304] 0 2024-02-28T16:31:19.476 INFO:tasks.workunit.client.0.smithi001.stdout:1/873: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/fc6 0 2024-02-28T16:31:19.479 INFO:tasks.workunit.client.0.smithi001.stdout:7/785: dwrite d3/d17/d52/d7f/d99/dc1/f6d [0,4194304] 0 2024-02-28T16:31:19.480 INFO:tasks.workunit.client.1.smithi066.stdout:2/399: rename d0/d12/d1a/d39/d40 to d0/d12/d2e/d7a 0 2024-02-28T16:31:19.482 INFO:tasks.workunit.client.0.smithi001.stdout:3/783: write d4/d18/d5f/f47 [1947395,72853] 0 2024-02-28T16:31:19.482 INFO:tasks.workunit.client.0.smithi001.stdout:7/786: chown d3/d1c/l38 5 1 2024-02-28T16:31:19.484 INFO:tasks.workunit.client.0.smithi001.stdout:2/729: creat d0/d2/da6/ff3 x:0 0 0 2024-02-28T16:31:19.486 INFO:tasks.workunit.client.0.smithi001.stdout:3/784: stat d4/d6/d7/d78/d81/cb1 0 2024-02-28T16:31:19.488 INFO:tasks.workunit.client.0.smithi001.stdout:3/785: rename d4 to d4/d1e/dcf/d100 22 2024-02-28T16:31:19.489 INFO:tasks.workunit.client.0.smithi001.stdout:6/731: dread d9/db/d19/f37 [0,4194304] 0 2024-02-28T16:31:19.489 INFO:tasks.workunit.client.0.smithi001.stdout:7/787: mkdir d3/d17/d3d/db3/dff 0 2024-02-28T16:31:19.490 INFO:tasks.workunit.client.1.smithi066.stdout:2/400: symlink d0/d12/d2e/l7b 0 2024-02-28T16:31:19.491 INFO:tasks.workunit.client.1.smithi066.stdout:9/319: dread da/df/d18/d28/f2b [0,4194304] 0 2024-02-28T16:31:19.492 INFO:tasks.workunit.client.0.smithi001.stdout:1/874: symlink d0/d9/d1a/d2b/d26/d27/df2/l130 0 2024-02-28T16:31:19.493 INFO:tasks.workunit.client.0.smithi001.stdout:9/940: write d5/f1e [8451624,57527] 0 2024-02-28T16:31:19.494 INFO:tasks.workunit.client.1.smithi066.stdout:7/305: dread dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:19.494 INFO:tasks.workunit.client.0.smithi001.stdout:6/732: write d9/d39/f3a [1504299,127977] 0 2024-02-28T16:31:19.496 INFO:tasks.workunit.client.1.smithi066.stdout:7/306: write dc/de/f3f [881787,78230] 0 2024-02-28T16:31:19.501 INFO:tasks.workunit.client.1.smithi066.stdout:7/307: truncate dc/de/d17/d1e/f23 4915600 0 2024-02-28T16:31:19.506 INFO:tasks.workunit.client.1.smithi066.stdout:7/308: creat dc/de/d17/d1e/f59 x:0 0 0 2024-02-28T16:31:19.506 INFO:tasks.workunit.client.0.smithi001.stdout:1/875: creat d0/d9/d1a/d2b/df8/f131 x:0 0 0 2024-02-28T16:31:19.509 INFO:tasks.workunit.client.0.smithi001.stdout:8/767: dwrite d4/d7/d22/d27/f3a [4194304,4194304] 0 2024-02-28T16:31:19.509 INFO:tasks.workunit.client.1.smithi066.stdout:7/309: chown dc/c1d 40 1 2024-02-28T16:31:19.511 INFO:tasks.workunit.client.1.smithi066.stdout:7/310: readlink dc/l14 0 2024-02-28T16:31:19.513 INFO:tasks.workunit.client.1.smithi066.stdout:6/344: dread d2/d26/f2d [4194304,4194304] 0 2024-02-28T16:31:19.514 INFO:tasks.workunit.client.0.smithi001.stdout:9/941: creat d5/d27/db1/da6/db2/f12e x:0 0 0 2024-02-28T16:31:19.515 INFO:tasks.workunit.client.1.smithi066.stdout:6/345: chown d2/da/fe 0 1 2024-02-28T16:31:19.515 INFO:tasks.workunit.client.0.smithi001.stdout:2/730: getdents d0 0 2024-02-28T16:31:19.516 INFO:tasks.workunit.client.1.smithi066.stdout:7/311: mknod dc/de/d17/d37/d47/c5a 0 2024-02-28T16:31:19.519 INFO:tasks.workunit.client.1.smithi066.stdout:9/320: dread da/df/d18/d2d/f4c [0,4194304] 0 2024-02-28T16:31:19.519 INFO:tasks.workunit.client.1.smithi066.stdout:6/346: fsync d2/da/f13 0 2024-02-28T16:31:19.521 INFO:tasks.workunit.client.0.smithi001.stdout:5/730: dwrite dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/fc1 [0,4194304] 0 2024-02-28T16:31:19.524 INFO:tasks.workunit.client.0.smithi001.stdout:9/942: read d5/d27/db1/d33/d115/d11f/de0/f11e [114478,27714] 0 2024-02-28T16:31:19.526 INFO:tasks.workunit.client.0.smithi001.stdout:4/873: dwrite d3/d5/fe2 [4194304,4194304] 0 2024-02-28T16:31:19.527 INFO:tasks.workunit.client.0.smithi001.stdout:0/746: dwrite d18/d22/d2b/f41 [0,4194304] 0 2024-02-28T16:31:19.528 INFO:tasks.workunit.client.1.smithi066.stdout:9/321: mknod da/c6c 0 2024-02-28T16:31:19.529 INFO:tasks.workunit.client.1.smithi066.stdout:7/312: getdents dc 0 2024-02-28T16:31:19.531 INFO:tasks.workunit.client.1.smithi066.stdout:9/322: mknod da/d4b/c6d 0 2024-02-28T16:31:19.536 INFO:tasks.workunit.client.1.smithi066.stdout:1/394: dread d2/d27/d2b/d43/f53 [0,4194304] 0 2024-02-28T16:31:19.536 INFO:tasks.workunit.client.1.smithi066.stdout:9/323: rename da/d3a/l3f to da/df/d18/d28/d42/l6e 0 2024-02-28T16:31:19.539 INFO:tasks.workunit.client.1.smithi066.stdout:1/395: stat d2/d18/d19/c38 0 2024-02-28T16:31:19.543 INFO:tasks.workunit.client.0.smithi001.stdout:2/731: mknod d0/d2/d3/de/d61/dd5/cf4 0 2024-02-28T16:31:19.543 INFO:tasks.workunit.client.0.smithi001.stdout:9/943: read d5/d27/db1/d5e/dd3/d109/d6f/fc7 [346673,74231] 0 2024-02-28T16:31:19.546 INFO:tasks.workunit.client.1.smithi066.stdout:4/501: dread d1/d23/d25/d43/f5d [0,4194304] 0 2024-02-28T16:31:19.549 INFO:tasks.workunit.client.1.smithi066.stdout:4/502: chown d1/d23/d25/d3e/c7c 3422711 1 2024-02-28T16:31:19.550 INFO:tasks.workunit.client.0.smithi001.stdout:5/731: symlink dc/d12/daa/lf0 0 2024-02-28T16:31:19.550 INFO:tasks.workunit.client.0.smithi001.stdout:1/876: symlink d0/d9/d1a/d2b/d26/d105/d121/l132 0 2024-02-28T16:31:19.553 INFO:tasks.workunit.client.1.smithi066.stdout:7/313: dwrite dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:19.556 INFO:tasks.workunit.client.0.smithi001.stdout:4/874: mkdir d3/d5/d7/d1f/d2e/dbe/d115/d11e 0 2024-02-28T16:31:19.561 INFO:tasks.workunit.client.1.smithi066.stdout:4/503: mknod d1/d23/d25/d77/c9e 0 2024-02-28T16:31:19.564 INFO:tasks.workunit.client.1.smithi066.stdout:7/314: creat dc/de/d17/f5b x:0 0 0 2024-02-28T16:31:19.567 INFO:tasks.workunit.client.0.smithi001.stdout:5/732: rmdir dc/d1a/d52 39 2024-02-28T16:31:19.568 INFO:tasks.workunit.client.0.smithi001.stdout:3/786: dwrite d4/d6/d7/d78/f95 [0,4194304] 0 2024-02-28T16:31:19.570 INFO:tasks.workunit.client.0.smithi001.stdout:8/768: dwrite d4/d7/da8/f91 [8388608,4194304] 0 2024-02-28T16:31:19.575 INFO:tasks.workunit.client.0.smithi001.stdout:7/788: dwrite d3/d17/d3d/d5d/fcf [0,4194304] 0 2024-02-28T16:31:19.580 INFO:tasks.workunit.client.0.smithi001.stdout:9/944: write d5/d27/db1/d33/d115/d11f/da8/ddf/fec [243616,30248] 0 2024-02-28T16:31:19.581 INFO:tasks.workunit.client.0.smithi001.stdout:5/733: fsync dc/f64 0 2024-02-28T16:31:19.582 INFO:tasks.workunit.client.0.smithi001.stdout:6/733: dwrite d9/fbc [0,4194304] 0 2024-02-28T16:31:19.584 INFO:tasks.workunit.client.0.smithi001.stdout:8/769: dread - d4/d32/d93/fee zero size 2024-02-28T16:31:19.587 INFO:tasks.workunit.client.1.smithi066.stdout:3/393: sync 2024-02-28T16:31:19.587 INFO:tasks.workunit.client.1.smithi066.stdout:5/368: sync 2024-02-28T16:31:19.591 INFO:tasks.workunit.client.0.smithi001.stdout:4/875: rename d3/d1c/d58/d6e/fbb to d3/d1c/d58/d88/f11f 0 2024-02-28T16:31:19.592 INFO:tasks.workunit.client.0.smithi001.stdout:0/747: write d18/d22/d3d/f52 [391838,116167] 0 2024-02-28T16:31:19.595 INFO:tasks.workunit.client.1.smithi066.stdout:3/394: symlink db/d2a/d44/d59/l86 0 2024-02-28T16:31:19.600 INFO:tasks.workunit.client.1.smithi066.stdout:9/324: dread da/f30 [0,4194304] 0 2024-02-28T16:31:19.601 INFO:tasks.workunit.client.0.smithi001.stdout:6/734: symlink d9/de/dbd/lf0 0 2024-02-28T16:31:19.601 INFO:tasks.workunit.client.0.smithi001.stdout:5/734: creat dc/d12/daa/ff1 x:0 0 0 2024-02-28T16:31:19.602 INFO:tasks.workunit.client.0.smithi001.stdout:1/877: link d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/l117 d0/d9/d1a/d2b/d26/d105/l133 0 2024-02-28T16:31:19.605 INFO:tasks.workunit.client.0.smithi001.stdout:3/787: creat d4/d1e/d7c/d7d/df8/f101 x:0 0 0 2024-02-28T16:31:19.607 INFO:tasks.workunit.client.0.smithi001.stdout:0/748: mknod d18/d22/d2f/d1c/cf4 0 2024-02-28T16:31:19.610 INFO:tasks.workunit.client.0.smithi001.stdout:8/770: symlink d4/d7/d22/lf8 0 2024-02-28T16:31:19.611 INFO:tasks.workunit.client.1.smithi066.stdout:3/395: write db/dc/d35/d3d/f41 [253840,43480] 0 2024-02-28T16:31:19.611 INFO:tasks.workunit.client.0.smithi001.stdout:6/735: fsync d9/f8c 0 2024-02-28T16:31:19.612 INFO:tasks.workunit.client.1.smithi066.stdout:5/369: dwrite d3/d4/dc/d56/f67 [0,4194304] 0 2024-02-28T16:31:19.612 INFO:tasks.workunit.client.1.smithi066.stdout:9/325: creat da/df/d18/f6f x:0 0 0 2024-02-28T16:31:19.616 INFO:tasks.workunit.client.0.smithi001.stdout:7/789: dwrite d3/d17/d52/d7f/d99/dc1/fd7 [0,4194304] 0 2024-02-28T16:31:19.618 INFO:tasks.workunit.client.0.smithi001.stdout:2/732: dwrite d0/d2/d3/de/d61/fb3 [0,4194304] 0 2024-02-28T16:31:19.621 INFO:tasks.workunit.client.1.smithi066.stdout:9/326: write da/d3a/f13 [4571270,126710] 0 2024-02-28T16:31:19.621 INFO:tasks.workunit.client.1.smithi066.stdout:3/396: write db/dc/d35/d3d/f41 [287401,121878] 0 2024-02-28T16:31:19.622 INFO:tasks.workunit.client.0.smithi001.stdout:9/945: getdents d5/d9a 0 2024-02-28T16:31:19.623 INFO:tasks.workunit.client.1.smithi066.stdout:4/504: dwrite d1/f9b [0,4194304] 0 2024-02-28T16:31:19.625 INFO:tasks.workunit.client.1.smithi066.stdout:9/327: chown da/d22/d46/l4e 172568 1 2024-02-28T16:31:19.626 INFO:tasks.workunit.client.1.smithi066.stdout:7/315: dread dc/de/d17/d1e/f23 [0,4194304] 0 2024-02-28T16:31:19.626 INFO:tasks.workunit.client.0.smithi001.stdout:7/790: readlink d3/d17/d61/led 0 2024-02-28T16:31:19.629 INFO:tasks.workunit.client.1.smithi066.stdout:4/505: write d1/d18/d41/f49 [404192,40152] 0 2024-02-28T16:31:19.630 INFO:tasks.workunit.client.1.smithi066.stdout:9/328: write da/df/d18/d28/d63/f5c [1565027,130628] 0 2024-02-28T16:31:19.631 INFO:tasks.workunit.client.0.smithi001.stdout:1/878: truncate d0/d9/d1a/d2b/d26/d27/d30/d67/db7/f10f 162216 0 2024-02-28T16:31:19.634 INFO:tasks.workunit.client.1.smithi066.stdout:9/329: truncate da/d22/f51 311783 0 2024-02-28T16:31:19.636 INFO:tasks.workunit.client.1.smithi066.stdout:5/370: rename d3/d4/dc/f3f to d3/d4/dc/dd/d2e/f79 0 2024-02-28T16:31:19.638 INFO:tasks.workunit.client.1.smithi066.stdout:7/316: write dc/de/d17/d1e/f23 [246409,98622] 0 2024-02-28T16:31:19.639 INFO:tasks.workunit.client.1.smithi066.stdout:5/371: dread - d3/d1b/f5f zero size 2024-02-28T16:31:19.643 INFO:tasks.workunit.client.1.smithi066.stdout:5/372: readlink d3/d4/dc/l23 0 2024-02-28T16:31:19.644 INFO:tasks.workunit.client.1.smithi066.stdout:9/330: getdents da/df/d18/d28 0 2024-02-28T16:31:19.646 INFO:tasks.workunit.client.0.smithi001.stdout:8/771: write d4/d47/d8b/fba [5033507,3566] 0 2024-02-28T16:31:19.647 INFO:tasks.workunit.client.1.smithi066.stdout:9/331: mkdir da/df/d70 0 2024-02-28T16:31:19.648 INFO:tasks.workunit.client.0.smithi001.stdout:3/788: rename d4/d1e/def/lab to d4/d6/d7/d78/d81/l102 0 2024-02-28T16:31:19.653 INFO:tasks.workunit.client.0.smithi001.stdout:6/736: dwrite d9/de/fe2 [0,4194304] 0 2024-02-28T16:31:19.655 INFO:tasks.workunit.client.1.smithi066.stdout:3/397: dwrite db/d2a/d44/f4f [0,4194304] 0 2024-02-28T16:31:19.656 INFO:tasks.workunit.client.1.smithi066.stdout:9/332: mknod da/d6b/c71 0 2024-02-28T16:31:19.657 INFO:tasks.workunit.client.0.smithi001.stdout:5/735: dwrite dc/d12/d41/d54/fa5 [0,4194304] 0 2024-02-28T16:31:19.658 INFO:tasks.workunit.client.1.smithi066.stdout:9/333: stat da/d6b 0 2024-02-28T16:31:19.658 INFO:tasks.workunit.client.1.smithi066.stdout:7/317: dwrite dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:19.659 INFO:tasks.workunit.client.0.smithi001.stdout:1/879: creat d0/d9/d1a/d2b/d26/d27/dce/f134 x:0 0 0 2024-02-28T16:31:19.661 INFO:tasks.workunit.client.0.smithi001.stdout:0/749: rename d18/d31 to d18/d22/d2b/d36/d46/df5 0 2024-02-28T16:31:19.663 INFO:tasks.workunit.client.0.smithi001.stdout:1/880: stat d0/d9/d1a/d72/la4 0 2024-02-28T16:31:19.664 INFO:tasks.workunit.client.1.smithi066.stdout:9/334: chown da/df/d18/d28/d42/c4a 95773 1 2024-02-28T16:31:19.665 INFO:tasks.workunit.client.0.smithi001.stdout:6/737: symlink d9/d29/d6e/d4e/d96/lf1 0 2024-02-28T16:31:19.669 INFO:tasks.workunit.client.0.smithi001.stdout:8/772: symlink d4/d7/d22/d27/d58/d94/db4/dd5/lf9 0 2024-02-28T16:31:19.669 INFO:tasks.workunit.client.1.smithi066.stdout:5/373: dwrite d3/d4/dc/dd/d41/d5d/f69 [0,4194304] 0 2024-02-28T16:31:19.680 INFO:tasks.workunit.client.1.smithi066.stdout:7/318: dwrite dc/de/d17/d1e/f20 [0,4194304] 0 2024-02-28T16:31:19.683 INFO:tasks.workunit.client.1.smithi066.stdout:9/335: creat da/d22/d46/d56/f72 x:0 0 0 2024-02-28T16:31:19.687 INFO:tasks.workunit.client.1.smithi066.stdout:4/506: write d1/d23/d25/d3e/f66 [145048,10951] 0 2024-02-28T16:31:19.687 INFO:tasks.workunit.client.1.smithi066.stdout:4/507: readlink d1/d7a/l89 0 2024-02-28T16:31:19.688 INFO:tasks.workunit.client.0.smithi001.stdout:6/738: dread - d9/d29/d6e/d53/fac zero size 2024-02-28T16:31:19.690 INFO:tasks.workunit.client.1.smithi066.stdout:3/398: getdents db/d2a/d44/d48/d5d 0 2024-02-28T16:31:19.690 INFO:tasks.workunit.client.1.smithi066.stdout:4/508: readlink d1/d23/d25/d3e/l88 0 2024-02-28T16:31:19.691 INFO:tasks.workunit.client.0.smithi001.stdout:1/881: dread d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/f11d [0,4194304] 0 2024-02-28T16:31:19.691 INFO:tasks.workunit.client.0.smithi001.stdout:0/750: mknod d18/d22/d3d/cf6 0 2024-02-28T16:31:19.692 INFO:tasks.workunit.client.1.smithi066.stdout:7/319: mkdir dc/d29/d5c 0 2024-02-28T16:31:19.693 INFO:tasks.workunit.client.0.smithi001.stdout:4/876: dread d3/d2d/f3c [0,4194304] 0 2024-02-28T16:31:19.695 INFO:tasks.workunit.client.1.smithi066.stdout:7/320: write dc/d29/f44 [3393657,44905] 0 2024-02-28T16:31:19.700 INFO:tasks.workunit.client.0.smithi001.stdout:0/751: dread - d18/d22/d3d/fa1 zero size 2024-02-28T16:31:19.702 INFO:tasks.workunit.client.1.smithi066.stdout:3/399: symlink db/d13/d1a/l87 0 2024-02-28T16:31:19.703 INFO:tasks.workunit.client.1.smithi066.stdout:7/321: creat dc/f5d x:0 0 0 2024-02-28T16:31:19.703 INFO:tasks.workunit.client.1.smithi066.stdout:3/400: readlink db/d13/d1a/l1c 0 2024-02-28T16:31:19.705 INFO:tasks.workunit.client.1.smithi066.stdout:4/509: fsync d1/d23/d25/d43/f65 0 2024-02-28T16:31:19.706 INFO:tasks.workunit.client.0.smithi001.stdout:6/739: symlink d9/d29/lf2 0 2024-02-28T16:31:19.707 INFO:tasks.workunit.client.0.smithi001.stdout:8/773: creat d4/d7/d22/ffa x:0 0 0 2024-02-28T16:31:19.708 INFO:tasks.workunit.client.1.smithi066.stdout:4/510: write d1/d18/d4e/f94 [191139,112213] 0 2024-02-28T16:31:19.711 INFO:tasks.workunit.client.1.smithi066.stdout:4/511: write d1/f9b [3531989,70344] 0 2024-02-28T16:31:19.712 INFO:tasks.workunit.client.0.smithi001.stdout:2/733: dwrite d0/f62 [0,4194304] 0 2024-02-28T16:31:19.713 INFO:tasks.workunit.client.0.smithi001.stdout:1/882: read - d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/fcd zero size 2024-02-28T16:31:19.713 INFO:tasks.workunit.client.1.smithi066.stdout:4/512: dread - d1/f96 zero size 2024-02-28T16:31:19.714 INFO:tasks.workunit.client.1.smithi066.stdout:4/513: fsync d1/f96 0 2024-02-28T16:31:19.717 INFO:tasks.workunit.client.1.smithi066.stdout:3/401: dread db/dc/d35/f60 [0,4194304] 0 2024-02-28T16:31:19.717 INFO:tasks.workunit.client.0.smithi001.stdout:4/877: symlink d3/d1c/d58/d6c/l120 0 2024-02-28T16:31:19.721 INFO:tasks.workunit.client.0.smithi001.stdout:4/878: dread - d3/d2d/d83/f10c zero size 2024-02-28T16:31:19.722 INFO:tasks.workunit.client.1.smithi066.stdout:3/402: chown db/dc/d35/d3d/l7f 32051346 1 2024-02-28T16:31:19.724 INFO:tasks.workunit.client.1.smithi066.stdout:3/403: write db/d13/f19 [9391517,19744] 0 2024-02-28T16:31:19.727 INFO:tasks.workunit.client.0.smithi001.stdout:5/736: dwrite dc/d12/d56/d96/db9/dc2/dca/d3c/f4f [0,4194304] 0 2024-02-28T16:31:19.728 INFO:tasks.workunit.client.1.smithi066.stdout:3/404: creat db/d7a/f88 x:0 0 0 2024-02-28T16:31:19.730 INFO:tasks.workunit.client.0.smithi001.stdout:6/740: symlink d9/da6/lf3 0 2024-02-28T16:31:19.733 INFO:tasks.workunit.client.1.smithi066.stdout:3/405: chown db/d2a/d44/l68 30156535 1 2024-02-28T16:31:19.733 INFO:tasks.workunit.client.1.smithi066.stdout:4/514: dwrite d1/d18/d41/f95 [0,4194304] 0 2024-02-28T16:31:19.734 INFO:tasks.workunit.client.1.smithi066.stdout:3/406: stat db/d13/d14 0 2024-02-28T16:31:19.735 INFO:tasks.workunit.client.1.smithi066.stdout:4/515: read d1/f46 [1670853,9242] 0 2024-02-28T16:31:19.737 INFO:tasks.workunit.client.0.smithi001.stdout:9/946: dread d5/d27/f73 [0,4194304] 0 2024-02-28T16:31:19.739 INFO:tasks.workunit.client.1.smithi066.stdout:4/516: read d1/f11 [236470,12476] 0 2024-02-28T16:31:19.743 INFO:tasks.workunit.client.1.smithi066.stdout:4/517: write d1/d23/d25/d3e/f3f [1472812,5732] 0 2024-02-28T16:31:19.745 INFO:tasks.workunit.client.0.smithi001.stdout:0/752: rename d18/d22/d2f/d1c/l9c to d18/d22/d2b/d36/d4d/lf7 0 2024-02-28T16:31:19.746 INFO:tasks.workunit.client.1.smithi066.stdout:3/407: mkdir db/dc/d35/d3d/d76/d89 0 2024-02-28T16:31:19.748 INFO:tasks.workunit.client.0.smithi001.stdout:5/737: write dc/d12/d56/d96/db9/dc2/dca/f9d [899777,16320] 0 2024-02-28T16:31:19.750 INFO:tasks.workunit.client.1.smithi066.stdout:7/322: link f1 dc/de/d17/d37/d47/f5e 0 2024-02-28T16:31:19.750 INFO:tasks.workunit.client.1.smithi066.stdout:5/374: dread d3/d4/dc/dd/d2e/f79 [4194304,4194304] 0 2024-02-28T16:31:19.752 INFO:tasks.workunit.client.1.smithi066.stdout:3/408: symlink db/d2a/d44/d48/l8a 0 2024-02-28T16:31:19.753 INFO:tasks.workunit.client.0.smithi001.stdout:6/741: mkdir d9/d39/dc4/df4 0 2024-02-28T16:31:19.754 INFO:tasks.workunit.client.1.smithi066.stdout:7/323: write dc/f4c [175966,109951] 0 2024-02-28T16:31:19.755 INFO:tasks.workunit.client.1.smithi066.stdout:3/409: read - db/d2a/d44/d48/f6c zero size 2024-02-28T16:31:19.755 INFO:tasks.workunit.client.1.smithi066.stdout:5/375: dread - d3/d4/dc/d2f/d55/f70 zero size 2024-02-28T16:31:19.757 INFO:tasks.workunit.client.0.smithi001.stdout:1/883: rename d0/d9/d1a/d72/f116 to d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/f135 0 2024-02-28T16:31:19.758 INFO:tasks.workunit.client.1.smithi066.stdout:4/518: rename d1/c51 to d1/d18/c9f 0 2024-02-28T16:31:19.758 INFO:tasks.workunit.client.1.smithi066.stdout:5/376: write d3/d4/f5c [13912,70851] 0 2024-02-28T16:31:19.759 INFO:tasks.workunit.client.1.smithi066.stdout:7/324: symlink dc/de/d17/d1e/d45/l5f 0 2024-02-28T16:31:19.761 INFO:tasks.workunit.client.1.smithi066.stdout:5/377: write d3/d4/f61 [1343956,112492] 0 2024-02-28T16:31:19.762 INFO:tasks.workunit.client.0.smithi001.stdout:5/738: truncate fa 3180845 0 2024-02-28T16:31:19.763 INFO:tasks.workunit.client.1.smithi066.stdout:7/325: creat dc/de/d17/f60 x:0 0 0 2024-02-28T16:31:19.763 INFO:tasks.workunit.client.1.smithi066.stdout:5/378: chown d3/d4/dc/d2f/d55/f70 122626 1 2024-02-28T16:31:19.764 INFO:tasks.workunit.client.0.smithi001.stdout:6/742: write d9/db/d19/fd5 [309026,83799] 0 2024-02-28T16:31:19.765 INFO:tasks.workunit.client.1.smithi066.stdout:7/326: unlink dc/de/d17/c22 0 2024-02-28T16:31:19.765 INFO:tasks.workunit.client.1.smithi066.stdout:5/379: chown d3/d4/dc/dd/l45 17526786 1 2024-02-28T16:31:19.767 INFO:tasks.workunit.client.1.smithi066.stdout:5/380: read - d3/d4/dc/d2f/d55/f5b zero size 2024-02-28T16:31:19.768 INFO:tasks.workunit.client.0.smithi001.stdout:1/884: chown d0/d9/d1a/d2b/dfe/f109 743773 1 2024-02-28T16:31:19.768 INFO:tasks.workunit.client.0.smithi001.stdout:0/753: truncate d18/d22/d2b/d36/d48/f9a 2954296 0 2024-02-28T16:31:19.771 INFO:tasks.workunit.client.0.smithi001.stdout:7/791: truncate d3/f45 644911 0 2024-02-28T16:31:19.772 INFO:tasks.workunit.client.1.smithi066.stdout:5/381: mkdir d3/d4/dc/dd/d41/d7a 0 2024-02-28T16:31:19.773 INFO:tasks.workunit.client.1.smithi066.stdout:5/382: readlink d3/d4/dc/dd/l72 0 2024-02-28T16:31:19.775 INFO:tasks.workunit.client.0.smithi001.stdout:5/739: chown dc/d12/d56/d96/db9/dc2/dca/d91 3652 1 2024-02-28T16:31:19.775 INFO:tasks.workunit.client.1.smithi066.stdout:5/383: mknod d3/c7b 0 2024-02-28T16:31:19.775 INFO:tasks.workunit.client.0.smithi001.stdout:1/885: fsync d0/d9/d56/d2f/f7c 0 2024-02-28T16:31:19.780 INFO:tasks.workunit.client.0.smithi001.stdout:3/789: dread d4/d6/d7/d78/d81/da1/fb9 [0,4194304] 0 2024-02-28T16:31:19.780 INFO:tasks.workunit.client.0.smithi001.stdout:2/734: dwrite d0/d2/d3/de/d8a/f70 [0,4194304] 0 2024-02-28T16:31:19.781 INFO:tasks.workunit.client.0.smithi001.stdout:6/743: creat d9/db/d5f/d89/d8a/ff5 x:0 0 0 2024-02-28T16:31:19.781 INFO:tasks.workunit.client.0.smithi001.stdout:1/886: stat d0/d9/d1a/d72/fb2 0 2024-02-28T16:31:19.783 INFO:tasks.workunit.client.1.smithi066.stdout:7/327: dwrite dc/de/d17/d37/d47/f5e [0,4194304] 0 2024-02-28T16:31:19.786 INFO:tasks.workunit.client.1.smithi066.stdout:7/328: write dc/de/d17/d1e/f24 [832174,102302] 0 2024-02-28T16:31:19.787 INFO:tasks.workunit.client.0.smithi001.stdout:5/740: fsync dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/fc1 0 2024-02-28T16:31:19.789 INFO:tasks.workunit.client.1.smithi066.stdout:7/329: chown dc/de/l2f 88086 1 2024-02-28T16:31:19.789 INFO:tasks.workunit.client.1.smithi066.stdout:2/401: sync 2024-02-28T16:31:19.790 INFO:tasks.workunit.client.1.smithi066.stdout:8/377: sync 2024-02-28T16:31:19.790 INFO:tasks.workunit.client.1.smithi066.stdout:0/503: sync 2024-02-28T16:31:19.791 INFO:tasks.workunit.client.1.smithi066.stdout:5/384: dwrite d3/d1b/d20/f63 [0,4194304] 0 2024-02-28T16:31:19.797 INFO:tasks.workunit.client.1.smithi066.stdout:8/378: creat d0/db/d12/d1d/d25/f74 x:0 0 0 2024-02-28T16:31:19.798 INFO:tasks.workunit.client.1.smithi066.stdout:2/402: write d0/d12/d1a/d23/d29/f6e [7176393,100147] 0 2024-02-28T16:31:19.800 INFO:tasks.workunit.client.1.smithi066.stdout:0/504: symlink d2/d27/d41/d76/d93/d10/d23/d6d/la7 0 2024-02-28T16:31:19.804 INFO:tasks.workunit.client.0.smithi001.stdout:4/879: dread d3/f12 [0,4194304] 0 2024-02-28T16:31:19.807 INFO:tasks.workunit.client.0.smithi001.stdout:9/947: write d5/d27/f73 [3505280,17211] 0 2024-02-28T16:31:19.810 INFO:tasks.workunit.client.1.smithi066.stdout:2/403: mkdir d0/d12/d1a/d39/d7c 0 2024-02-28T16:31:19.811 INFO:tasks.workunit.client.1.smithi066.stdout:7/330: dread dc/de/d17/d1e/f23 [0,4194304] 0 2024-02-28T16:31:19.815 INFO:tasks.workunit.client.0.smithi001.stdout:4/880: chown d3/d5/c5b 161841144 1 2024-02-28T16:31:19.815 INFO:tasks.workunit.client.0.smithi001.stdout:2/735: creat d0/d2/dd9/ff5 x:0 0 0 2024-02-28T16:31:19.815 INFO:tasks.workunit.client.0.smithi001.stdout:3/790: chown d4/d6/c66 41 1 2024-02-28T16:31:19.815 INFO:tasks.workunit.client.1.smithi066.stdout:0/505: rename d2/d27/d41/d76/d93/d10/d23/f2d to d2/d27/d41/d76/d93/d10/d50/d73/fa8 0 2024-02-28T16:31:19.816 INFO:tasks.workunit.client.1.smithi066.stdout:5/385: getdents d3/d1b 0 2024-02-28T16:31:19.818 INFO:tasks.workunit.client.1.smithi066.stdout:8/379: getdents d0/db 0 2024-02-28T16:31:19.818 INFO:tasks.workunit.client.1.smithi066.stdout:5/386: readlink d3/d4/dc/dd/l72 0 2024-02-28T16:31:19.820 INFO:tasks.workunit.client.1.smithi066.stdout:7/331: chown dc/l1f 806713 1 2024-02-28T16:31:19.820 INFO:tasks.workunit.client.1.smithi066.stdout:0/506: mkdir d2/d85/da9 0 2024-02-28T16:31:19.821 INFO:tasks.workunit.client.1.smithi066.stdout:5/387: dread - d3/d4/dc/d2f/d55/f5b zero size 2024-02-28T16:31:19.822 INFO:tasks.workunit.client.1.smithi066.stdout:8/380: truncate d0/d6/f5d 58290 0 2024-02-28T16:31:19.823 INFO:tasks.workunit.client.0.smithi001.stdout:1/887: creat d0/d9/d1a/d2b/d26/f136 x:0 0 0 2024-02-28T16:31:19.824 INFO:tasks.workunit.client.1.smithi066.stdout:0/507: write d2/d27/d41/d76/d93/d10/f7f [482545,38117] 0 2024-02-28T16:31:19.826 INFO:tasks.workunit.client.1.smithi066.stdout:7/332: rename dc/de/l52 to dc/de/d17/d1e/l61 0 2024-02-28T16:31:19.827 INFO:tasks.workunit.client.0.smithi001.stdout:0/754: mknod d18/d22/d2f/cf8 0 2024-02-28T16:31:19.828 INFO:tasks.workunit.client.1.smithi066.stdout:5/388: creat d3/d4/dc/dd/d41/d5d/f7c x:0 0 0 2024-02-28T16:31:19.829 INFO:tasks.workunit.client.1.smithi066.stdout:8/381: creat d0/db/d12/d1d/d25/f75 x:0 0 0 2024-02-28T16:31:19.830 INFO:tasks.workunit.client.1.smithi066.stdout:5/389: chown d3/d4/fb 17 1 2024-02-28T16:31:19.831 INFO:tasks.workunit.client.1.smithi066.stdout:3/410: dread db/d13/d1a/d25/f5f [0,4194304] 0 2024-02-28T16:31:19.835 INFO:tasks.workunit.client.1.smithi066.stdout:3/411: readlink db/d13/d14/d2d/l4b 0 2024-02-28T16:31:19.835 INFO:tasks.workunit.client.1.smithi066.stdout:5/390: mkdir d3/d1b/d7d 0 2024-02-28T16:31:19.837 INFO:tasks.workunit.client.1.smithi066.stdout:2/404: dwrite d0/f1e [0,4194304] 0 2024-02-28T16:31:19.838 INFO:tasks.workunit.client.1.smithi066.stdout:8/382: symlink d0/db/d12/d1d/l76 0 2024-02-28T16:31:19.841 INFO:tasks.workunit.client.0.smithi001.stdout:9/948: rmdir d5/d27/d34/d50/d9d 39 2024-02-28T16:31:19.842 INFO:tasks.workunit.client.1.smithi066.stdout:5/391: truncate d3/d4/f71 859238 0 2024-02-28T16:31:19.845 INFO:tasks.workunit.client.1.smithi066.stdout:2/405: stat d0/d12/d1a/d39/f42 0 2024-02-28T16:31:19.848 INFO:tasks.workunit.client.1.smithi066.stdout:8/383: creat d0/db/d12/d1d/d25/d72/f77 x:0 0 0 2024-02-28T16:31:19.854 INFO:tasks.workunit.client.1.smithi066.stdout:8/384: write d0/d6/f5d [660654,73761] 0 2024-02-28T16:31:19.856 INFO:tasks.workunit.client.1.smithi066.stdout:2/406: write d0/d12/d1a/d23/d29/d21/f75 [4737700,25743] 0 2024-02-28T16:31:19.857 INFO:tasks.workunit.client.1.smithi066.stdout:0/508: dwrite d2/d27/d41/d76/d93/d10/d50/d52/f69 [0,4194304] 0 2024-02-28T16:31:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.869+0000 7f0d42c00700 1 -- 172.21.15.1:0/1832378057 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d3c0708c0 msgr2=0x7f0d3c070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.869+0000 7f0d42c00700 1 --2- 172.21.15.1:0/1832378057 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d3c0708c0 0x7f0d3c070d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009ce0 tx=0x7f0d2c004d10 comp rx=0 tx=0).stop 2024-02-28T16:31:19.860 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:19 smithi001 ceph-mon[28991]: pgmap v53: 65 pgs: 65 active+clean; 2.0 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 50 MiB/s rd, 183 MiB/s wr, 331 op/s 2024-02-28T16:31:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.871+0000 7f0d42c00700 1 -- 172.21.15.1:0/1832378057 shutdown_connections 2024-02-28T16:31:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.871+0000 7f0d42c00700 1 --2- 172.21.15.1:0/1832378057 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d3c0708c0 0x7f0d3c070d10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.871+0000 7f0d42c00700 1 --2- 172.21.15.1:0/1832378057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3c071f10 0x7f0d3c070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.871+0000 7f0d42c00700 1 -- 172.21.15.1:0/1832378057 >> 172.21.15.1:0/1832378057 conn(0x7f0d3c0f8620 msgr2=0x7f0d3c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:19.863 INFO:tasks.workunit.client.1.smithi066.stdout:7/333: dwrite dc/de/d17/d1e/f53 [0,4194304] 0 2024-02-28T16:31:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.872+0000 7f0d42c00700 1 -- 172.21.15.1:0/1832378057 shutdown_connections 2024-02-28T16:31:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.872+0000 7f0d42c00700 1 -- 172.21.15.1:0/1832378057 wait complete. 2024-02-28T16:31:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.873+0000 7f0d42c00700 1 Processor -- start 2024-02-28T16:31:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.873+0000 7f0d42c00700 1 -- start start 2024-02-28T16:31:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.874+0000 7f0d42c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3c0708c0 0x7f0d3c114150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.874+0000 7f0d42c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d3c071f10 0x7f0d3c114690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.874+0000 7f0d42c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c114c90 con 0x7f0d3c0708c0 2024-02-28T16:31:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.874+0000 7f0d42c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c115530 con 0x7f0d3c071f10 2024-02-28T16:31:19.864 INFO:tasks.workunit.client.0.smithi001.stdout:7/792: link d3/d1c/fcb d3/d17/d3d/f100 0 2024-02-28T16:31:19.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.875+0000 7f0d3be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3c0708c0 0x7f0d3c114150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:19.865 INFO:tasks.workunit.client.1.smithi066.stdout:2/407: fdatasync d0/f1f 0 2024-02-28T16:31:19.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.875+0000 7f0d3b400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d3c071f10 0x7f0d3c114690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:19.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.876+0000 7f0d3be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3c0708c0 0x7f0d3c114150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40804/0 (socket says 172.21.15.1:40804) 2024-02-28T16:31:19.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.876+0000 7f0d3be00700 1 -- 172.21.15.1:0/3883754527 learned_addr learned my addr 172.21.15.1:0/3883754527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:19.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.876+0000 7f0d3be00700 1 -- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d3c071f10 msgr2=0x7f0d3c114690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:19.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.876+0000 7f0d3be00700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d3c071f10 0x7f0d3c114690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:19.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.876+0000 7f0d3be00700 1 -- 172.21.15.1:0/3883754527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c009940 con 0x7f0d3c0708c0 2024-02-28T16:31:19.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.877+0000 7f0d3be00700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3c0708c0 0x7f0d3c114150 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400c830 tx=0x7f0d34004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:19.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.877+0000 7f0d33e00700 1 -- 172.21.15.1:0/3883754527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3400a760 con 0x7f0d3c0708c0 2024-02-28T16:31:19.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.877+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c115810 con 0x7f0d3c0708c0 2024-02-28T16:31:19.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.878+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c115d50 con 0x7f0d3c0708c0 2024-02-28T16:31:19.868 INFO:tasks.workunit.client.0.smithi001.stdout:2/736: mknod d0/d2/cf6 0 2024-02-28T16:31:19.869 INFO:tasks.workunit.client.1.smithi066.stdout:2/408: readlink d0/d12/d1a/d23/d29/l49 0 2024-02-28T16:31:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.879+0000 7f0d33e00700 1 -- 172.21.15.1:0/3883754527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d3400ad90 con 0x7f0d3c0708c0 2024-02-28T16:31:19.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.879+0000 7f0d33e00700 1 -- 172.21.15.1:0/3883754527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3400a760 con 0x7f0d3c0708c0 2024-02-28T16:31:19.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.881+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c0052e0 con 0x7f0d3c0708c0 2024-02-28T16:31:19.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.883+0000 7f0d33e00700 1 -- 172.21.15.1:0/3883754527 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 25) v1 ==== 88910+0+0 (secure 0 0 0) 0x7f0d34012400 con 0x7f0d3c0708c0 2024-02-28T16:31:19.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.884+0000 7f0d33e00700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0d2006b280 0x7f0d2006d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:19.874 INFO:tasks.workunit.client.1.smithi066.stdout:2/409: chown d0/d12/d1a/d39/d7c 144492294 1 2024-02-28T16:31:19.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.889+0000 7f0d33e00700 1 -- 172.21.15.1:0/3883754527 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f0d34057eb0 con 0x7f0d3c0708c0 2024-02-28T16:31:19.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.890+0000 7f0d3b400700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0d2006b280 0x7f0d2006d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:19.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.891+0000 7f0d3b400700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0d2006b280 0x7f0d2006d730 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c005010 tx=0x7f0d2c004d10 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:19.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:19.894+0000 7f0d33e00700 1 -- 172.21.15.1:0/3883754527 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f0d34057580 con 0x7f0d3c0708c0 2024-02-28T16:31:19.884 INFO:tasks.workunit.client.0.smithi001.stdout:4/881: unlink d3/d5/d7/d1f/d2e/dbe/d87/fd1 0 2024-02-28T16:31:19.890 INFO:tasks.workunit.client.1.smithi066.stdout:3/412: dwrite db/d13/d1a/f1e [0,4194304] 0 2024-02-28T16:31:19.892 INFO:tasks.workunit.client.0.smithi001.stdout:9/949: dread d5/d27/db1/d5e/dd3/d109/d6f/db3/fdd [0,4194304] 0 2024-02-28T16:31:19.893 INFO:tasks.workunit.client.1.smithi066.stdout:3/413: rename db/dc/d11 to db/dc/d11/d1f/d8b 22 2024-02-28T16:31:19.893 INFO:tasks.workunit.client.0.smithi001.stdout:5/741: write dc/df/d79/f95 [3210620,97440] 0 2024-02-28T16:31:19.894 INFO:tasks.workunit.client.1.smithi066.stdout:2/410: fdatasync d0/f1e 0 2024-02-28T16:31:19.898 INFO:tasks.workunit.client.0.smithi001.stdout:0/755: creat d18/d22/ff9 x:0 0 0 2024-02-28T16:31:19.899 INFO:tasks.workunit.client.1.smithi066.stdout:3/414: symlink db/d13/d14/l8c 0 2024-02-28T16:31:19.899 INFO:tasks.workunit.client.0.smithi001.stdout:6/744: link d9/db/d10/f11 d9/db/d10/ff6 0 2024-02-28T16:31:19.900 INFO:tasks.workunit.client.0.smithi001.stdout:1/888: dwrite d0/d9/f7e [0,4194304] 0 2024-02-28T16:31:19.905 INFO:tasks.workunit.client.1.smithi066.stdout:3/415: symlink db/dc/d35/d3d/d76/l8d 0 2024-02-28T16:31:19.905 INFO:tasks.workunit.client.1.smithi066.stdout:1/396: sync 2024-02-28T16:31:19.906 INFO:tasks.workunit.client.1.smithi066.stdout:6/347: sync 2024-02-28T16:31:19.906 INFO:tasks.workunit.client.1.smithi066.stdout:9/336: sync 2024-02-28T16:31:19.909 INFO:tasks.workunit.client.1.smithi066.stdout:5/392: dread d3/d4/dc/dd/d35/f64 [0,4194304] 0 2024-02-28T16:31:19.909 INFO:tasks.workunit.client.1.smithi066.stdout:8/385: dwrite d0/db/d12/d1d/d25/f47 [0,4194304] 0 2024-02-28T16:31:19.910 INFO:tasks.workunit.client.1.smithi066.stdout:0/509: dread d2/f5e [0,4194304] 0 2024-02-28T16:31:19.912 INFO:tasks.workunit.client.0.smithi001.stdout:7/793: chown d3/d17/d52/d7f/d99/dc1/d49/d8d/fab 73229 1 2024-02-28T16:31:19.914 INFO:tasks.workunit.client.1.smithi066.stdout:9/337: write da/df/d18/d28/d42/f49 [1758108,45261] 0 2024-02-28T16:31:19.915 INFO:tasks.workunit.client.0.smithi001.stdout:0/756: dread d18/d22/d3d/f52 [0,4194304] 0 2024-02-28T16:31:19.915 INFO:tasks.workunit.client.1.smithi066.stdout:3/416: mknod db/dc/c8e 0 2024-02-28T16:31:19.919 INFO:tasks.workunit.client.1.smithi066.stdout:3/417: write db/dc/f55 [830508,34180] 0 2024-02-28T16:31:19.922 INFO:tasks.workunit.client.1.smithi066.stdout:6/348: mkdir d2/d2c/d72 0 2024-02-28T16:31:19.925 INFO:tasks.workunit.client.0.smithi001.stdout:2/737: write d0/d2/d3/de/d61/d12/d6a/d84/fcf [244468,113474] 0 2024-02-28T16:31:19.925 INFO:tasks.workunit.client.1.smithi066.stdout:5/393: truncate d3/d4/dc/d56/f75 240760 0 2024-02-28T16:31:19.927 INFO:tasks.workunit.client.1.smithi066.stdout:3/418: mknod db/c8f 0 2024-02-28T16:31:19.928 INFO:tasks.workunit.client.1.smithi066.stdout:7/334: truncate dc/f4b 2026676 0 2024-02-28T16:31:19.932 INFO:tasks.workunit.client.0.smithi001.stdout:6/745: dwrite d9/de/feb [0,4194304] 0 2024-02-28T16:31:19.933 INFO:tasks.workunit.client.1.smithi066.stdout:8/386: truncate d0/d6/ff 807506 0 2024-02-28T16:31:19.935 INFO:tasks.workunit.client.1.smithi066.stdout:2/411: dwrite d0/f71 [4194304,4194304] 0 2024-02-28T16:31:19.935 INFO:tasks.workunit.client.0.smithi001.stdout:2/738: rmdir d0/d2/d3/de/d61/d12/d55 39 2024-02-28T16:31:19.936 INFO:tasks.workunit.client.0.smithi001.stdout:6/746: chown d9/db/d5f/dd4 3010861 1 2024-02-28T16:31:19.937 INFO:tasks.workunit.client.1.smithi066.stdout:7/335: chown dc/d29/c48 14564613 1 2024-02-28T16:31:19.940 INFO:tasks.workunit.client.0.smithi001.stdout:6/747: write d9/db/d10/f7b [48738,67824] 0 2024-02-28T16:31:19.943 INFO:tasks.workunit.client.0.smithi001.stdout:9/950: getdents d5/d27/db1/d33/d115/d11f 0 2024-02-28T16:31:19.943 INFO:tasks.workunit.client.0.smithi001.stdout:5/742: dread dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/f85 [0,4194304] 0 2024-02-28T16:31:19.944 INFO:tasks.workunit.client.1.smithi066.stdout:1/397: dwrite d2/d27/f7b [0,4194304] 0 2024-02-28T16:31:19.946 INFO:tasks.workunit.client.0.smithi001.stdout:2/739: readlink d0/d2/d3/de/d8a/l13 0 2024-02-28T16:31:19.947 INFO:tasks.workunit.client.0.smithi001.stdout:4/882: dwrite d3/d5/d7/d1f/d3a/f70 [0,4194304] 0 2024-02-28T16:31:19.948 INFO:tasks.workunit.client.1.smithi066.stdout:2/412: unlink d0/d12/d1a/c6d 0 2024-02-28T16:31:19.949 INFO:tasks.workunit.client.0.smithi001.stdout:1/889: write d0/d9/d1a/d2b/d26/d27/d30/d88/fb3 [1243414,112589] 0 2024-02-28T16:31:19.952 INFO:tasks.workunit.client.0.smithi001.stdout:7/794: link d3/d17/d3d/d42/f7d d3/d17/d3d/db3/f101 0 2024-02-28T16:31:19.954 INFO:tasks.workunit.client.0.smithi001.stdout:9/951: creat d5/d9a/dba/f12f x:0 0 0 2024-02-28T16:31:19.954 INFO:tasks.workunit.client.1.smithi066.stdout:0/510: write d2/d27/d41/d76/d93/d10/d23/fa0 [79736,67534] 0 2024-02-28T16:31:19.955 INFO:tasks.workunit.client.0.smithi001.stdout:6/748: mknod d9/db/d5f/dd4/cf7 0 2024-02-28T16:31:19.957 INFO:tasks.workunit.client.0.smithi001.stdout:2/740: fsync d0/d2/d3/de/d61/fea 0 2024-02-28T16:31:19.959 INFO:tasks.workunit.client.1.smithi066.stdout:1/398: chown d2/d27/d2b/d34/l64 27738237 1 2024-02-28T16:31:19.959 INFO:tasks.workunit.client.0.smithi001.stdout:5/743: creat dc/d12/d56/d96/db9/dc2/dca/d91/ff2 x:0 0 0 2024-02-28T16:31:19.961 INFO:tasks.workunit.client.1.smithi066.stdout:9/338: dwrite da/d3a/d15/d48/f52 [0,4194304] 0 2024-02-28T16:31:19.963 INFO:tasks.workunit.client.0.smithi001.stdout:7/795: truncate d3/d17/d3d/d42/ffb 299267 0 2024-02-28T16:31:19.963 INFO:tasks.workunit.client.0.smithi001.stdout:2/741: stat d0/d2/d3/de/d8a 0 2024-02-28T16:31:19.964 INFO:tasks.workunit.client.0.smithi001.stdout:1/890: readlink d0/d9/d1a/d2b/dfe/ld7 0 2024-02-28T16:31:19.967 INFO:tasks.workunit.client.0.smithi001.stdout:9/952: mkdir d5/d27/db1/d58/d130 0 2024-02-28T16:31:19.967 INFO:tasks.workunit.client.1.smithi066.stdout:0/511: readlink d2/d27/d41/d76/d93/d33/l34 0 2024-02-28T16:31:19.969 INFO:tasks.workunit.client.1.smithi066.stdout:4/519: sync 2024-02-28T16:31:19.973 INFO:tasks.workunit.client.0.smithi001.stdout:2/742: write d0/d2/d3/de/d61/d12/d6a/d84/fd7 [791027,109294] 0 2024-02-28T16:31:19.974 INFO:tasks.workunit.client.1.smithi066.stdout:3/419: dwrite db/d13/d14/f15 [0,4194304] 0 2024-02-28T16:31:19.981 INFO:tasks.workunit.client.0.smithi001.stdout:0/757: dwrite d18/d22/d2f/ff3 [0,4194304] 0 2024-02-28T16:31:19.982 INFO:tasks.workunit.client.0.smithi001.stdout:5/744: mkdir dc/d1a/d7d/d87/df3 0 2024-02-28T16:31:19.983 INFO:tasks.workunit.client.1.smithi066.stdout:5/394: dwrite d3/d4/dc/dd/d2e/f38 [0,4194304] 0 2024-02-28T16:31:19.983 INFO:tasks.workunit.client.0.smithi001.stdout:1/891: unlink d0/d9/laa 0 2024-02-28T16:31:19.984 INFO:tasks.workunit.client.1.smithi066.stdout:1/399: write d2/d27/d2b/d37/d48/f66 [2552129,39473] 0 2024-02-28T16:31:19.988 INFO:tasks.workunit.client.1.smithi066.stdout:3/420: dread - db/d2a/d44/d48/d5d/f81 zero size 2024-02-28T16:31:19.989 INFO:tasks.workunit.client.1.smithi066.stdout:9/339: mknod da/d22/d46/c73 0 2024-02-28T16:31:19.990 INFO:tasks.workunit.client.1.smithi066.stdout:6/349: dwrite d2/da/d19/f41 [0,4194304] 0 2024-02-28T16:31:19.990 INFO:tasks.workunit.client.1.smithi066.stdout:7/336: dread f4 [0,4194304] 0 2024-02-28T16:31:19.993 INFO:tasks.workunit.client.0.smithi001.stdout:9/953: mkdir d5/d27/db1/d33/d115/d11f/d131 0 2024-02-28T16:31:19.997 INFO:tasks.workunit.client.0.smithi001.stdout:6/749: dwrite d9/f86 [0,4194304] 0 2024-02-28T16:31:19.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:19 smithi066 ceph-mon[36236]: pgmap v53: 65 pgs: 65 active+clean; 2.0 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 50 MiB/s rd, 183 MiB/s wr, 331 op/s 2024-02-28T16:31:19.998 INFO:tasks.workunit.client.1.smithi066.stdout:9/340: write da/df/d18/d28/d63/f32 [215077,93392] 0 2024-02-28T16:31:19.999 INFO:tasks.workunit.client.0.smithi001.stdout:6/750: dread - d9/d29/fe9 zero size 2024-02-28T16:31:20.002 INFO:tasks.workunit.client.1.smithi066.stdout:9/341: read - da/df/d18/d2d/f69 zero size 2024-02-28T16:31:20.003 INFO:tasks.workunit.client.0.smithi001.stdout:1/892: rmdir d0/d9/d1a/d2b/d26/d27/dd3/de8/ded 39 2024-02-28T16:31:20.007 INFO:tasks.workunit.client.0.smithi001.stdout:6/751: truncate d9/db/dd9/fe3 493111 0 2024-02-28T16:31:20.008 INFO:tasks.workunit.client.1.smithi066.stdout:9/342: write da/df/d18/d28/f4d [2147663,90673] 0 2024-02-28T16:31:20.008 INFO:tasks.workunit.client.0.smithi001.stdout:6/752: chown d9/db/d5f/d8b/f6d 344289910 1 2024-02-28T16:31:20.010 INFO:tasks.workunit.client.1.smithi066.stdout:5/395: symlink d3/d4/dc/d42/d60/l7e 0 2024-02-28T16:31:20.011 INFO:tasks.workunit.client.1.smithi066.stdout:1/400: mknod d2/d18/c7f 0 2024-02-28T16:31:20.012 INFO:tasks.workunit.client.1.smithi066.stdout:9/343: write da/df/d18/d2d/f69 [775031,97202] 0 2024-02-28T16:31:20.015 INFO:tasks.workunit.client.0.smithi001.stdout:6/753: rmdir d9/db/d19 39 2024-02-28T16:31:20.016 INFO:tasks.workunit.client.1.smithi066.stdout:5/396: write d3/d4/dc/d56/f75 [274425,14002] 0 2024-02-28T16:31:20.016 INFO:tasks.workunit.client.1.smithi066.stdout:7/337: rmdir dc/de/d17/d1e/d45 39 2024-02-28T16:31:20.022 INFO:tasks.workunit.client.1.smithi066.stdout:1/401: unlink d2/d27/c79 0 2024-02-28T16:31:20.023 INFO:tasks.workunit.client.1.smithi066.stdout:7/338: stat dc/de/d17/d1e/c50 0 2024-02-28T16:31:20.023 INFO:tasks.workunit.client.1.smithi066.stdout:1/402: readlink l1 0 2024-02-28T16:31:20.027 INFO:tasks.workunit.client.1.smithi066.stdout:6/350: link d2/d2c/f51 d2/da/d3e/f73 0 2024-02-28T16:31:20.028 INFO:tasks.workunit.client.1.smithi066.stdout:3/421: creat db/dc/d11/d1f/d78/f90 x:0 0 0 2024-02-28T16:31:20.030 INFO:tasks.workunit.client.0.smithi001.stdout:1/893: rename d0/d9/d1a/d2b/dfe/f109 to d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f137 0 2024-02-28T16:31:20.034 INFO:tasks.workunit.client.1.smithi066.stdout:5/397: unlink d3/d4/dc/dd/l72 0 2024-02-28T16:31:20.036 INFO:tasks.workunit.client.1.smithi066.stdout:2/413: dread d0/d12/d1a/f3a [0,4194304] 0 2024-02-28T16:31:20.037 INFO:tasks.workunit.client.0.smithi001.stdout:1/894: readlink d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/l11b 0 2024-02-28T16:31:20.040 INFO:tasks.workunit.client.1.smithi066.stdout:8/387: dread d0/d6/dc/d16/d1a/d63/d58/f48 [0,4194304] 0 2024-02-28T16:31:20.040 INFO:tasks.workunit.client.1.smithi066.stdout:9/344: rename da/d3a/d15/d48/f52 to da/d22/d46/f74 0 2024-02-28T16:31:20.047 INFO:tasks.workunit.client.1.smithi066.stdout:1/403: fsync d2/d27/d2b/f3d 0 2024-02-28T16:31:20.047 INFO:tasks.workunit.client.1.smithi066.stdout:3/422: creat db/d2a/d44/d48/d5d/f91 x:0 0 0 2024-02-28T16:31:20.047 INFO:tasks.workunit.client.1.smithi066.stdout:7/339: dread dc/de/d17/d1e/f20 [0,4194304] 0 2024-02-28T16:31:20.048 INFO:tasks.workunit.client.1.smithi066.stdout:3/423: readlink db/dc/d11/l75 0 2024-02-28T16:31:20.048 INFO:tasks.workunit.client.1.smithi066.stdout:0/512: dwrite d2/f5e [0,4194304] 0 2024-02-28T16:31:20.049 INFO:tasks.workunit.client.0.smithi001.stdout:6/754: rename d9/d29/fe6 to d9/db/d5f/d89/ff8 0 2024-02-28T16:31:20.052 INFO:tasks.workunit.client.0.smithi001.stdout:7/796: dread d3/d17/d52/d7f/d99/dc1/d49/d94/fea [0,4194304] 0 2024-02-28T16:31:20.056 INFO:tasks.workunit.client.0.smithi001.stdout:7/797: truncate d3/d17/d3d/d5d/fbd 1340282 0 2024-02-28T16:31:20.078 INFO:tasks.workunit.client.0.smithi001.stdout:1/895: getdents d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/d9c 0 2024-02-28T16:31:20.079 INFO:tasks.workunit.client.0.smithi001.stdout:6/755: getdents d9/de 0 2024-02-28T16:31:20.082 INFO:tasks.workunit.client.0.smithi001.stdout:5/745: dwrite dc/d12/f16 [0,4194304] 0 2024-02-28T16:31:20.082 INFO:tasks.workunit.client.0.smithi001.stdout:7/798: truncate d3/d17/d3d/f72 370542 0 2024-02-28T16:31:20.086 INFO:tasks.workunit.client.0.smithi001.stdout:9/954: dwrite d5/fd [4194304,4194304] 0 2024-02-28T16:31:20.087 INFO:tasks.workunit.client.0.smithi001.stdout:7/799: fsync d3/d17/d3d/d5d/fcf 0 2024-02-28T16:31:20.103 INFO:tasks.workunit.client.0.smithi001.stdout:9/955: dread - d5/d27/db1/d5e/dd3/d109/f4a zero size 2024-02-28T16:31:20.104 INFO:tasks.workunit.client.0.smithi001.stdout:6/756: creat d9/db/d19/ff9 x:0 0 0 2024-02-28T16:31:20.110 INFO:tasks.workunit.client.0.smithi001.stdout:1/896: dwrite d0/d9/d1a/d2b/d26/d27/f12e [0,4194304] 0 2024-02-28T16:31:20.111 INFO:tasks.workunit.client.1.smithi066.stdout:2/414: creat d0/d12/d1a/d23/f7d x:0 0 0 2024-02-28T16:31:20.112 INFO:tasks.workunit.client.1.smithi066.stdout:8/388: creat d0/d6/dc/d16/d4c/d6c/f78 x:0 0 0 2024-02-28T16:31:20.113 INFO:tasks.workunit.client.1.smithi066.stdout:6/351: link d2/d26/f53 d2/d26/d32/d46/f74 0 2024-02-28T16:31:20.113 INFO:tasks.workunit.client.1.smithi066.stdout:1/404: mkdir d2/d27/d2b/d43/d46/d80 0 2024-02-28T16:31:20.114 INFO:tasks.workunit.client.1.smithi066.stdout:3/424: mkdir db/d13/d14/d2d/d92 0 2024-02-28T16:31:20.115 INFO:tasks.workunit.client.0.smithi001.stdout:9/956: chown d5/lf 59235 1 2024-02-28T16:31:20.119 INFO:tasks.workunit.client.0.smithi001.stdout:1/897: fsync d0/d9/d1a/d2b/d26/d105/f114 0 2024-02-28T16:31:20.120 INFO:tasks.workunit.client.1.smithi066.stdout:2/415: symlink d0/d12/d2e/d7a/l7e 0 2024-02-28T16:31:20.120 INFO:tasks.workunit.client.1.smithi066.stdout:5/398: write d3/d4/dc/d2f/f33 [850393,81107] 0 2024-02-28T16:31:20.120 INFO:tasks.workunit.client.1.smithi066.stdout:0/513: fsync d2/d27/d41/d76/d93/f20 0 2024-02-28T16:31:20.120 INFO:tasks.workunit.client.1.smithi066.stdout:3/425: fsync db/dc/f2b 0 2024-02-28T16:31:20.120 INFO:tasks.workunit.client.1.smithi066.stdout:6/352: mkdir d2/da/d19/d75 0 2024-02-28T16:31:20.121 INFO:tasks.workunit.client.1.smithi066.stdout:1/405: fdatasync d2/d27/d2b/d43/d46/f63 0 2024-02-28T16:31:20.121 INFO:tasks.workunit.client.0.smithi001.stdout:6/757: rename d9/d29/d6e/d4e/l9f to d9/de/lfa 0 2024-02-28T16:31:20.122 INFO:tasks.workunit.client.0.smithi001.stdout:5/746: dwrite dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/fe1 [0,4194304] 0 2024-02-28T16:31:20.124 INFO:tasks.workunit.client.1.smithi066.stdout:8/389: dread d0/db/f5f [0,4194304] 0 2024-02-28T16:31:20.124 INFO:tasks.workunit.client.0.smithi001.stdout:7/800: write d3/d17/d3d/f72 [68402,68479] 0 2024-02-28T16:31:20.125 INFO:tasks.workunit.client.0.smithi001.stdout:6/758: dread - d9/db/d19/f78 zero size 2024-02-28T16:31:20.128 INFO:tasks.workunit.client.1.smithi066.stdout:5/399: mkdir d3/d4/dc/dd/d41/d7f 0 2024-02-28T16:31:20.128 INFO:tasks.workunit.client.1.smithi066.stdout:2/416: read d0/d12/d1a/f2b [3960077,95085] 0 2024-02-28T16:31:20.129 INFO:tasks.workunit.client.0.smithi001.stdout:5/747: chown dc/d12/d41/d54/fa5 132505385 1 2024-02-28T16:31:20.129 INFO:tasks.workunit.client.0.smithi001.stdout:6/759: stat d9/db/d10 0 2024-02-28T16:31:20.129 INFO:tasks.workunit.client.1.smithi066.stdout:7/340: dwrite dc/d29/f41 [0,4194304] 0 2024-02-28T16:31:20.130 INFO:tasks.workunit.client.0.smithi001.stdout:9/957: creat d5/d27/db1/d5e/dd3/d109/d6f/dc8/f132 x:0 0 0 2024-02-28T16:31:20.132 INFO:tasks.workunit.client.0.smithi001.stdout:5/748: chown dc/d12/d56/c74 163963 1 2024-02-28T16:31:20.135 INFO:tasks.workunit.client.1.smithi066.stdout:3/426: creat db/dc/d35/d3d/f93 x:0 0 0 2024-02-28T16:31:20.135 INFO:tasks.workunit.client.1.smithi066.stdout:7/341: fdatasync dc/de/d17/d1e/f53 0 2024-02-28T16:31:20.135 INFO:tasks.workunit.client.1.smithi066.stdout:0/514: write d2/d27/d41/d76/d93/d10/d50/d51/f82 [13953,74990] 0 2024-02-28T16:31:20.135 INFO:tasks.workunit.client.0.smithi001.stdout:7/801: symlink d3/d1c/l102 0 2024-02-28T16:31:20.136 INFO:tasks.workunit.client.0.smithi001.stdout:5/749: chown f7 46318 1 2024-02-28T16:31:20.136 INFO:tasks.workunit.client.0.smithi001.stdout:9/958: truncate d5/d27/f73 4893867 0 2024-02-28T16:31:20.137 INFO:tasks.workunit.client.1.smithi066.stdout:8/390: symlink d0/d6/dc/d16/d1a/d63/d58/l79 0 2024-02-28T16:31:20.137 INFO:tasks.workunit.client.1.smithi066.stdout:6/353: creat d2/da/f76 x:0 0 0 2024-02-28T16:31:20.139 INFO:tasks.workunit.client.0.smithi001.stdout:6/760: mkdir d9/db/d5f/dd4/dfb 0 2024-02-28T16:31:20.139 INFO:tasks.workunit.client.1.smithi066.stdout:8/391: readlink d0/d6/dc/d16/l60 0 2024-02-28T16:31:20.140 INFO:tasks.workunit.client.1.smithi066.stdout:7/342: write dc/f25 [2200271,79779] 0 2024-02-28T16:31:20.143 INFO:tasks.workunit.client.0.smithi001.stdout:9/959: unlink d5/d27/db1/fa4 0 2024-02-28T16:31:20.144 INFO:tasks.workunit.client.1.smithi066.stdout:5/400: chown d3/d1b/d20/f63 24 1 2024-02-28T16:31:20.144 INFO:tasks.workunit.client.0.smithi001.stdout:7/802: mknod d3/d17/d3d/d59/d96/c103 0 2024-02-28T16:31:20.145 INFO:tasks.workunit.client.1.smithi066.stdout:0/515: dread d2/f32 [0,4194304] 0 2024-02-28T16:31:20.145 INFO:tasks.workunit.client.1.smithi066.stdout:1/406: rename d2/d18/f21 to d2/d27/d2b/d37/d48/f81 0 2024-02-28T16:31:20.146 INFO:tasks.workunit.client.0.smithi001.stdout:7/803: write d3/fa4 [2779421,87263] 0 2024-02-28T16:31:20.148 INFO:tasks.workunit.client.1.smithi066.stdout:6/354: mknod d2/d26/c77 0 2024-02-28T16:31:20.149 INFO:tasks.workunit.client.0.smithi001.stdout:7/804: chown d3/d17/d3d/d42/d8e/l9d 814268 1 2024-02-28T16:31:20.152 INFO:tasks.workunit.client.0.smithi001.stdout:1/898: creat d0/d9/d1a/d2b/d26/d27/d30/d67/db7/f138 x:0 0 0 2024-02-28T16:31:20.156 INFO:tasks.workunit.client.0.smithi001.stdout:7/805: truncate d3/d17/d3d/d42/fee 822440 0 2024-02-28T16:31:20.156 INFO:tasks.workunit.client.1.smithi066.stdout:7/343: write dc/de/d17/d37/f54 [637557,38880] 0 2024-02-28T16:31:20.157 INFO:tasks.workunit.client.0.smithi001.stdout:9/960: mknod d5/d27/db1/c133 0 2024-02-28T16:31:20.158 INFO:tasks.workunit.client.0.smithi001.stdout:5/750: link dc/d1a/d7d/d87/cae dc/d12/d41/dd6/cf4 0 2024-02-28T16:31:20.159 INFO:tasks.workunit.client.1.smithi066.stdout:6/355: mknod d2/da/d19/d54/d5d/c78 0 2024-02-28T16:31:20.159 INFO:tasks.workunit.client.0.smithi001.stdout:8/774: sync 2024-02-28T16:31:20.160 INFO:tasks.workunit.client.1.smithi066.stdout:5/401: mkdir d3/d4/dc/dd/d41/d7f/d80 0 2024-02-28T16:31:20.160 INFO:tasks.workunit.client.1.smithi066.stdout:7/344: write dc/d29/f44 [2206880,43985] 0 2024-02-28T16:31:20.160 INFO:tasks.workunit.client.0.smithi001.stdout:3/791: sync 2024-02-28T16:31:20.161 INFO:tasks.workunit.client.0.smithi001.stdout:4/883: sync 2024-02-28T16:31:20.161 INFO:tasks.workunit.client.0.smithi001.stdout:2/743: sync 2024-02-28T16:31:20.162 INFO:tasks.workunit.client.0.smithi001.stdout:0/758: sync 2024-02-28T16:31:20.163 INFO:tasks.workunit.client.1.smithi066.stdout:2/417: dread d0/f1f [0,4194304] 0 2024-02-28T16:31:20.163 INFO:tasks.workunit.client.1.smithi066.stdout:1/407: creat d2/d27/d2b/d37/f82 x:0 0 0 2024-02-28T16:31:20.164 INFO:tasks.workunit.client.1.smithi066.stdout:4/520: sync 2024-02-28T16:31:20.165 INFO:tasks.workunit.client.1.smithi066.stdout:5/402: chown d3/d1b/d39 58495407 1 2024-02-28T16:31:20.166 INFO:tasks.workunit.client.0.smithi001.stdout:5/751: write dc/d12/d56/fdb [134489,100830] 0 2024-02-28T16:31:20.167 INFO:tasks.workunit.client.1.smithi066.stdout:2/418: chown d0/d12/d2e/d7a/c4b 1262449 1 2024-02-28T16:31:20.168 INFO:tasks.workunit.client.1.smithi066.stdout:7/345: truncate dc/de/d17/f5b 873138 0 2024-02-28T16:31:20.169 INFO:tasks.workunit.client.1.smithi066.stdout:4/521: write d1/d18/d4e/f53 [2077719,64202] 0 2024-02-28T16:31:20.169 INFO:tasks.workunit.client.0.smithi001.stdout:1/899: creat d0/d9/d1a/d2b/d26/d27/df2/f139 x:0 0 0 2024-02-28T16:31:20.169 INFO:tasks.workunit.client.0.smithi001.stdout:6/761: link d9/c61 d9/db/d19/cfc 0 2024-02-28T16:31:20.170 INFO:tasks.workunit.client.0.smithi001.stdout:0/759: chown d18/da9/dc9/dd7/ce4 1 1 2024-02-28T16:31:20.171 INFO:tasks.workunit.client.1.smithi066.stdout:1/408: unlink d2/d27/d2b/d37/l5f 0 2024-02-28T16:31:20.173 INFO:tasks.workunit.client.1.smithi066.stdout:6/356: rename d2/d26/f5a to d2/da/f79 0 2024-02-28T16:31:20.174 INFO:tasks.workunit.client.0.smithi001.stdout:5/752: stat dc/d12/d41/dd6/ce0 0 2024-02-28T16:31:20.174 INFO:tasks.workunit.client.0.smithi001.stdout:3/792: chown d4/d6/f1c 14 1 2024-02-28T16:31:20.175 INFO:tasks.workunit.client.0.smithi001.stdout:8/775: creat d4/d7/d22/d27/d58/d94/dbc/ffb x:0 0 0 2024-02-28T16:31:20.176 INFO:tasks.workunit.client.1.smithi066.stdout:1/409: truncate d2/d18/f3a 512294 0 2024-02-28T16:31:20.178 INFO:tasks.workunit.client.0.smithi001.stdout:6/762: fdatasync d9/db/f88 0 2024-02-28T16:31:20.181 INFO:tasks.workunit.client.1.smithi066.stdout:4/522: truncate d1/d23/d25/d43/f76 148913 0 2024-02-28T16:31:20.181 INFO:tasks.workunit.client.1.smithi066.stdout:1/410: chown d2/d27/d2b/d34/f55 0 1 2024-02-28T16:31:20.181 INFO:tasks.workunit.client.1.smithi066.stdout:5/403: mknod d3/d4/dc/dd/d2e/c81 0 2024-02-28T16:31:20.182 INFO:tasks.workunit.client.1.smithi066.stdout:6/357: mknod d2/da/d19/d54/c7a 0 2024-02-28T16:31:20.183 INFO:tasks.workunit.client.1.smithi066.stdout:2/419: creat d0/d12/d2e/d3b/d66/f7f x:0 0 0 2024-02-28T16:31:20.184 INFO:tasks.workunit.client.0.smithi001.stdout:2/744: rename d0/fc3 to d0/d2/d3/de/d8a/ff7 0 2024-02-28T16:31:20.184 INFO:tasks.workunit.client.1.smithi066.stdout:1/411: chown d2/d27/d2b/d43/d46/d80 0 1 2024-02-28T16:31:20.184 INFO:tasks.workunit.client.1.smithi066.stdout:4/523: mkdir d1/d18/d4e/da0 0 2024-02-28T16:31:20.185 INFO:tasks.workunit.client.0.smithi001.stdout:9/961: dread d5/d27/db1/da6/db2/fb6 [0,4194304] 0 2024-02-28T16:31:20.185 INFO:tasks.workunit.client.0.smithi001.stdout:1/900: fsync d0/d9/d1a/d2b/d26/d27/d30/d88/fbc 0 2024-02-28T16:31:20.185 INFO:tasks.workunit.client.0.smithi001.stdout:0/760: symlink d18/d22/d76/d93/lfa 0 2024-02-28T16:31:20.188 INFO:tasks.workunit.client.1.smithi066.stdout:0/516: truncate d2/f5e 157288 0 2024-02-28T16:31:20.190 INFO:tasks.workunit.client.0.smithi001.stdout:0/761: write d18/d22/d2b/d36/d48/f55 [5530563,98214] 0 2024-02-28T16:31:20.190 INFO:tasks.workunit.client.1.smithi066.stdout:4/524: readlink d1/d23/d25/l56 0 2024-02-28T16:31:20.193 INFO:tasks.workunit.client.1.smithi066.stdout:6/358: fsync d2/da/f18 0 2024-02-28T16:31:20.193 INFO:tasks.workunit.client.1.smithi066.stdout:1/412: fdatasync d2/d27/d2b/d43/d46/d68/f76 0 2024-02-28T16:31:20.196 INFO:tasks.workunit.client.1.smithi066.stdout:5/404: truncate d3/d4/dc/d2f/d55/f70 943236 0 2024-02-28T16:31:20.197 INFO:tasks.workunit.client.1.smithi066.stdout:8/392: dwrite d0/db/d15/f37 [0,4194304] 0 2024-02-28T16:31:20.198 INFO:tasks.workunit.client.0.smithi001.stdout:5/753: truncate dc/d12/d56/d96/db9/dc2/dca/f76 188298 0 2024-02-28T16:31:20.200 INFO:tasks.workunit.client.0.smithi001.stdout:9/962: dread d5/d27/db1/d5e/dd3/d109/d6f/fc7 [0,4194304] 0 2024-02-28T16:31:20.201 INFO:tasks.workunit.client.1.smithi066.stdout:4/525: mknod d1/d23/d25/ca1 0 2024-02-28T16:31:20.202 INFO:tasks.workunit.client.0.smithi001.stdout:7/806: getdents d3/d17/d3d/d59/d6e 0 2024-02-28T16:31:20.202 INFO:tasks.workunit.client.1.smithi066.stdout:2/420: rename d0/d12/d1a/d23/c5e to d0/d12/d1a/d23/d29/d21/d2f/c80 0 2024-02-28T16:31:20.206 INFO:tasks.workunit.client.1.smithi066.stdout:5/405: unlink d3/d4/dc/l23 0 2024-02-28T16:31:20.207 INFO:tasks.workunit.client.1.smithi066.stdout:8/393: creat d0/db/d15/f7a x:0 0 0 2024-02-28T16:31:20.208 INFO:tasks.workunit.client.1.smithi066.stdout:7/346: dwrite f5 [0,4194304] 0 2024-02-28T16:31:20.209 INFO:tasks.workunit.client.0.smithi001.stdout:9/963: write d5/d27/db1/da6/db2/de4/f12a [489162,57605] 0 2024-02-28T16:31:20.212 INFO:tasks.workunit.client.1.smithi066.stdout:0/517: creat d2/d27/d41/d76/d93/d33/d3f/faa x:0 0 0 2024-02-28T16:31:20.212 INFO:tasks.workunit.client.1.smithi066.stdout:6/359: creat d2/d2c/d72/f7b x:0 0 0 2024-02-28T16:31:20.213 INFO:tasks.workunit.client.1.smithi066.stdout:7/347: write dc/de/d17/f5b [1714922,21709] 0 2024-02-28T16:31:20.213 INFO:tasks.workunit.client.0.smithi001.stdout:0/762: creat d18/d22/d2f/d1c/d90/ffb x:0 0 0 2024-02-28T16:31:20.218 INFO:tasks.workunit.client.1.smithi066.stdout:8/394: write d0/f13 [2378691,95853] 0 2024-02-28T16:31:20.218 INFO:tasks.workunit.client.1.smithi066.stdout:5/406: creat d3/d4/dc/dd/d2e/d18/f82 x:0 0 0 2024-02-28T16:31:20.219 INFO:tasks.workunit.client.0.smithi001.stdout:8/776: dwrite d4/d32/d93/fee [0,4194304] 0 2024-02-28T16:31:20.220 INFO:tasks.workunit.client.1.smithi066.stdout:4/526: creat d1/d18/d4e/da0/fa2 x:0 0 0 2024-02-28T16:31:20.220 INFO:tasks.workunit.client.1.smithi066.stdout:6/360: stat d2/d26 0 2024-02-28T16:31:20.223 INFO:tasks.workunit.client.1.smithi066.stdout:5/407: readlink d3/d4/l46 0 2024-02-28T16:31:20.224 INFO:tasks.workunit.client.1.smithi066.stdout:8/395: mknod d0/d6/c7b 0 2024-02-28T16:31:20.226 INFO:tasks.workunit.client.0.smithi001.stdout:9/964: creat d5/d27/db1/d127/f134 x:0 0 0 2024-02-28T16:31:20.229 INFO:tasks.workunit.client.1.smithi066.stdout:5/408: readlink d3/d4/dc/d2f/l59 0 2024-02-28T16:31:20.233 INFO:tasks.workunit.client.1.smithi066.stdout:4/527: creat d1/d23/d25/d77/fa3 x:0 0 0 2024-02-28T16:31:20.234 INFO:tasks.workunit.client.0.smithi001.stdout:9/965: write d5/d27/db1/d5e/dd3/d109/df1/f12b [158429,64884] 0 2024-02-28T16:31:20.237 INFO:tasks.workunit.client.1.smithi066.stdout:1/413: dwrite d2/d18/d19/f1c [0,4194304] 0 2024-02-28T16:31:20.240 INFO:tasks.workunit.client.1.smithi066.stdout:0/518: rename d2/d27/d41/d76/d93/l74 to d2/lab 0 2024-02-28T16:31:20.243 INFO:tasks.workunit.client.1.smithi066.stdout:8/396: fdatasync d0/f13 0 2024-02-28T16:31:20.243 INFO:tasks.workunit.client.1.smithi066.stdout:1/414: read d2/f7 [2074323,26277] 0 2024-02-28T16:31:20.244 INFO:tasks.workunit.client.0.smithi001.stdout:4/884: truncate d3/d5/fcf 189406 0 2024-02-28T16:31:20.244 INFO:tasks.workunit.client.0.smithi001.stdout:0/763: mknod d18/d22/d57/cfc 0 2024-02-28T16:31:20.249 INFO:tasks.workunit.client.0.smithi001.stdout:1/901: dread d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f137 [0,4194304] 0 2024-02-28T16:31:20.249 INFO:tasks.workunit.client.0.smithi001.stdout:2/745: stat d0/d2/c47 0 2024-02-28T16:31:20.251 INFO:tasks.workunit.client.1.smithi066.stdout:6/361: rename d2/da/d19/f1d to d2/d26/d32/d46/f7c 0 2024-02-28T16:31:20.252 INFO:tasks.workunit.client.1.smithi066.stdout:8/397: fsync d0/d6/dc/d16/d4c/d6c/f78 0 2024-02-28T16:31:20.253 INFO:tasks.workunit.client.0.smithi001.stdout:3/793: write d4/d18/da0/fa2 [1278002,97019] 0 2024-02-28T16:31:20.254 INFO:tasks.workunit.client.1.smithi066.stdout:0/519: fsync d2/d27/d41/d76/d93/d10/d50/d52/f69 0 2024-02-28T16:31:20.258 INFO:tasks.workunit.client.1.smithi066.stdout:1/415: creat d2/d27/d2b/d37/d51/f83 x:0 0 0 2024-02-28T16:31:20.259 INFO:tasks.workunit.client.0.smithi001.stdout:1/902: unlink d0/d9/d1a/d2b/d26/l38 0 2024-02-28T16:31:20.260 INFO:tasks.workunit.client.1.smithi066.stdout:7/348: dwrite dc/de/d17/d1e/f53 [0,4194304] 0 2024-02-28T16:31:20.264 INFO:tasks.workunit.client.0.smithi001.stdout:2/746: creat d0/d2/d3/d36/d94/ff8 x:0 0 0 2024-02-28T16:31:20.270 INFO:tasks.workunit.client.1.smithi066.stdout:4/528: dwrite d1/d23/d25/d43/f5d [0,4194304] 0 2024-02-28T16:31:20.272 INFO:tasks.workunit.client.1.smithi066.stdout:1/416: write d2/d27/d2b/f2a [1136870,28864] 0 2024-02-28T16:31:20.275 INFO:tasks.workunit.client.0.smithi001.stdout:4/885: getdents d3/d2d/d4c/d69/d100 0 2024-02-28T16:31:20.277 INFO:tasks.workunit.client.1.smithi066.stdout:5/409: write d3/d1b/f2d [531790,130913] 0 2024-02-28T16:31:20.279 INFO:tasks.workunit.client.1.smithi066.stdout:7/349: creat dc/de/d17/d1e/d45/f62 x:0 0 0 2024-02-28T16:31:20.280 INFO:tasks.workunit.client.1.smithi066.stdout:1/417: dread - d2/d27/d2b/d34/f44 zero size 2024-02-28T16:31:20.286 INFO:tasks.workunit.client.1.smithi066.stdout:4/529: rename d1/c2e to d1/d23/d25/d43/ca4 0 2024-02-28T16:31:20.287 INFO:tasks.workunit.client.1.smithi066.stdout:7/350: write dc/f25 [3861770,44116] 0 2024-02-28T16:31:20.288 INFO:tasks.workunit.client.0.smithi001.stdout:6/763: dwrite d9/db/fd8 [0,4194304] 0 2024-02-28T16:31:20.289 INFO:tasks.workunit.client.0.smithi001.stdout:1/903: mkdir d0/d9/d1a/d2b/d26/d27/dd3/de8/d102/d13a 0 2024-02-28T16:31:20.292 INFO:tasks.workunit.client.1.smithi066.stdout:6/362: dwrite d2/d26/d6b/f70 [0,4194304] 0 2024-02-28T16:31:20.294 INFO:tasks.workunit.client.0.smithi001.stdout:0/764: dread d18/d22/d3d/fd6 [0,4194304] 0 2024-02-28T16:31:20.297 INFO:tasks.workunit.client.0.smithi001.stdout:5/754: dread dc/d12/f32 [0,4194304] 0 2024-02-28T16:31:20.297 INFO:tasks.workunit.client.0.smithi001.stdout:9/966: dwrite d5/d27/db1/da6/db2/dff/f125 [0,4194304] 0 2024-02-28T16:31:20.298 INFO:tasks.workunit.client.0.smithi001.stdout:8/777: dwrite d4/d32/d69/f6f [0,4194304] 0 2024-02-28T16:31:20.303 INFO:tasks.workunit.client.1.smithi066.stdout:3/427: sync 2024-02-28T16:31:20.304 INFO:tasks.workunit.client.1.smithi066.stdout:9/345: sync 2024-02-28T16:31:20.305 INFO:tasks.workunit.client.0.smithi001.stdout:3/794: link d4/d6/c66 d4/d1e/dcf/c103 0 2024-02-28T16:31:20.310 INFO:tasks.workunit.client.1.smithi066.stdout:1/418: rename d2/d18/d6b/d72 to d2/d27/d2b/d37/d54/d84 0 2024-02-28T16:31:20.310 INFO:tasks.workunit.client.1.smithi066.stdout:0/520: link d2/d27/d41/d76/d93/d33/d3f/d8e/c71 d2/d27/d41/d76/d93/d10/d23/d6d/cac 0 2024-02-28T16:31:20.314 INFO:tasks.workunit.client.1.smithi066.stdout:9/346: dread - da/df/d18/d28/f59 zero size 2024-02-28T16:31:20.315 INFO:tasks.workunit.client.1.smithi066.stdout:2/421: dread d0/d12/f47 [4194304,4194304] 0 2024-02-28T16:31:20.316 INFO:tasks.workunit.client.0.smithi001.stdout:1/904: mknod d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/c13b 0 2024-02-28T16:31:20.317 INFO:tasks.workunit.client.0.smithi001.stdout:0/765: mkdir d18/da9/dc9/dd7/dfd 0 2024-02-28T16:31:20.319 INFO:tasks.workunit.client.0.smithi001.stdout:2/747: link d0/d2/d3/de/d61/f67 d0/d2/d3/de/d61/d12/d6a/d84/ff9 0 2024-02-28T16:31:20.319 INFO:tasks.workunit.client.1.smithi066.stdout:4/530: getdents d1/d18/d4e/da0 0 2024-02-28T16:31:20.319 INFO:tasks.workunit.client.1.smithi066.stdout:7/351: dread dc/de/d17/d1e/f28 [0,4194304] 0 2024-02-28T16:31:20.321 INFO:tasks.workunit.client.1.smithi066.stdout:4/531: fdatasync d1/d18/d41/f95 0 2024-02-28T16:31:20.322 INFO:tasks.workunit.client.1.smithi066.stdout:6/363: dread d2/d26/d6b/f70 [0,4194304] 0 2024-02-28T16:31:20.324 INFO:tasks.workunit.client.1.smithi066.stdout:1/419: unlink d2/d27/d2b/d43/f53 0 2024-02-28T16:31:20.325 INFO:tasks.workunit.client.0.smithi001.stdout:8/778: mknod d4/d7/da8/d89/cfc 0 2024-02-28T16:31:20.328 INFO:tasks.workunit.client.1.smithi066.stdout:9/347: symlink da/df/d18/d28/l75 0 2024-02-28T16:31:20.330 INFO:tasks.workunit.client.1.smithi066.stdout:0/521: mkdir d2/d27/d41/d76/d93/d10/d23/d6d/dad 0 2024-02-28T16:31:20.336 INFO:tasks.workunit.client.0.smithi001.stdout:6/764: read d9/db/d10/f16 [13097,31846] 0 2024-02-28T16:31:20.336 INFO:tasks.workunit.client.0.smithi001.stdout:4/886: mknod d3/c121 0 2024-02-28T16:31:20.341 INFO:tasks.workunit.client.1.smithi066.stdout:4/532: dread d1/d23/d25/f83 [0,4194304] 0 2024-02-28T16:31:20.345 INFO:tasks.workunit.client.1.smithi066.stdout:3/428: truncate db/d13/f19 9362233 0 2024-02-28T16:31:20.345 INFO:tasks.workunit.client.1.smithi066.stdout:1/420: readlink d2/d27/d2b/l3f 0 2024-02-28T16:31:20.345 INFO:tasks.workunit.client.1.smithi066.stdout:6/364: mknod d2/d2c/d72/c7d 0 2024-02-28T16:31:20.345 INFO:tasks.workunit.client.1.smithi066.stdout:9/348: mkdir da/d3a/d15/d48/d76 0 2024-02-28T16:31:20.346 INFO:tasks.workunit.client.1.smithi066.stdout:4/533: write d1/d18/d4e/f94 [481204,28007] 0 2024-02-28T16:31:20.348 INFO:tasks.workunit.client.0.smithi001.stdout:5/755: write dc/d12/f32 [1949028,123490] 0 2024-02-28T16:31:20.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.357+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d1c000bf0 con 0x7f0d2006b280 2024-02-28T16:31:20.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.361+0000 7f0d33e00700 1 -- 172.21.15.1:0/3883754527 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f0d1c000bf0 con 0x7f0d2006b280 2024-02-28T16:31:20.352 INFO:tasks.workunit.client.1.smithi066.stdout:7/352: mkdir dc/d63 0 2024-02-28T16:31:20.354 INFO:tasks.workunit.client.1.smithi066.stdout:2/422: rename d0/d12/d1a/d23/c78 to d0/d12/d1a/d23/d29/d21/d2f/d67/c81 0 2024-02-28T16:31:20.356 INFO:tasks.workunit.client.0.smithi001.stdout:5/756: chown dc/d12/d56/d96/db8/lea 4 1 2024-02-28T16:31:20.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.366+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0d2006b280 msgr2=0x7f0d2006d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:20.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.366+0000 7f0d42c00700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0d2006b280 0x7f0d2006d730 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c005010 tx=0x7f0d2c004d10 comp rx=0 tx=0).stop 2024-02-28T16:31:20.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.366+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3c0708c0 msgr2=0x7f0d3c114150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:20.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.366+0000 7f0d42c00700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3c0708c0 0x7f0d3c114150 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400c830 tx=0x7f0d34004470 comp rx=0 tx=0).stop 2024-02-28T16:31:20.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.367+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 shutdown_connections 2024-02-28T16:31:20.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.367+0000 7f0d42c00700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f0d2006b280 0x7f0d2006d730 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c005010 tx=0x7f0d2c004d10 comp rx=0 tx=0).stop 2024-02-28T16:31:20.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.367+0000 7f0d42c00700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3c0708c0 0x7f0d3c114150 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:20.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.367+0000 7f0d42c00700 1 --2- 172.21.15.1:0/3883754527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d3c071f10 0x7f0d3c114690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:20.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.367+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 >> 172.21.15.1:0/3883754527 conn(0x7f0d3c0f8620 msgr2=0x7f0d3c0fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:20.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.367+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 shutdown_connections 2024-02-28T16:31:20.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.367+0000 7f0d42c00700 1 -- 172.21.15.1:0/3883754527 wait complete. 2024-02-28T16:31:20.360 INFO:tasks.workunit.client.1.smithi066.stdout:3/429: creat db/d2a/d44/d59/f94 x:0 0 0 2024-02-28T16:31:20.362 INFO:tasks.workunit.client.1.smithi066.stdout:1/421: creat d2/d27/d2b/d43/d46/f85 x:0 0 0 2024-02-28T16:31:20.367 INFO:tasks.workunit.client.1.smithi066.stdout:8/398: dread d0/db/d15/f31 [0,4194304] 0 2024-02-28T16:31:20.367 INFO:tasks.workunit.client.1.smithi066.stdout:3/430: write db/d2a/d44/d48/d5d/f91 [95732,99355] 0 2024-02-28T16:31:20.368 INFO:tasks.workunit.client.1.smithi066.stdout:5/410: dread d3/d4/f4e [0,4194304] 0 2024-02-28T16:31:20.375 INFO:tasks.workunit.client.1.smithi066.stdout:7/353: symlink dc/l64 0 2024-02-28T16:31:20.375 INFO:tasks.workunit.client.1.smithi066.stdout:0/522: dwrite d2/d27/d41/d76/d93/d10/d50/d60/f95 [0,4194304] 0 2024-02-28T16:31:20.376 INFO:tasks.workunit.client.0.smithi001.stdout:0/766: dread d18/d22/d2b/d36/d48/f55 [0,4194304] 0 2024-02-28T16:31:20.377 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:31:20.378 INFO:tasks.workunit.client.1.smithi066.stdout:3/431: truncate db/d7a/f88 159104 0 2024-02-28T16:31:20.379 INFO:tasks.workunit.client.0.smithi001.stdout:0/767: write d18/f1e [4485973,63512] 0 2024-02-28T16:31:20.379 INFO:tasks.workunit.client.1.smithi066.stdout:6/365: rename d2/da/d3e/c56 to d2/d2c/d72/c7e 0 2024-02-28T16:31:20.387 INFO:tasks.workunit.client.1.smithi066.stdout:8/399: rmdir d0/db/d12/d1d/d25 39 2024-02-28T16:31:20.390 INFO:tasks.workunit.client.0.smithi001.stdout:9/967: truncate d5/f1d 6650113 0 2024-02-28T16:31:20.391 INFO:tasks.workunit.client.1.smithi066.stdout:9/349: link da/df/d18/c3c da/df/d18/d28/d63/c77 0 2024-02-28T16:31:20.391 INFO:tasks.workunit.client.1.smithi066.stdout:7/354: fdatasync dc/d29/f41 0 2024-02-28T16:31:20.393 INFO:tasks.workunit.client.0.smithi001.stdout:2/748: symlink d0/d2/d3/de/d61/dd5/lfa 0 2024-02-28T16:31:20.397 INFO:tasks.workunit.client.1.smithi066.stdout:1/422: creat d2/d27/d2b/f86 x:0 0 0 2024-02-28T16:31:20.397 INFO:tasks.workunit.client.1.smithi066.stdout:2/423: creat d0/d12/f82 x:0 0 0 2024-02-28T16:31:20.399 INFO:tasks.workunit.client.0.smithi001.stdout:8/779: creat d4/d32/d69/ffd x:0 0 0 2024-02-28T16:31:20.401 INFO:tasks.workunit.client.0.smithi001.stdout:6/765: mknod d9/db/d5f/d8b/cfd 0 2024-02-28T16:31:20.405 INFO:tasks.workunit.client.0.smithi001.stdout:4/887: creat d3/d2d/d4c/d69/d82/f122 x:0 0 0 2024-02-28T16:31:20.406 INFO:tasks.workunit.client.1.smithi066.stdout:3/432: dwrite db/d2a/d44/d48/f63 [0,4194304] 0 2024-02-28T16:31:20.406 INFO:tasks.workunit.client.1.smithi066.stdout:0/523: mknod d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/da1/cae 0 2024-02-28T16:31:20.408 INFO:tasks.workunit.client.0.smithi001.stdout:6/766: write d9/d29/d6a/f77 [4906255,74029] 0 2024-02-28T16:31:20.411 INFO:tasks.workunit.client.1.smithi066.stdout:5/411: creat d3/d4/f83 x:0 0 0 2024-02-28T16:31:20.412 INFO:tasks.workunit.client.1.smithi066.stdout:7/355: dread dc/d29/f41 [0,4194304] 0 2024-02-28T16:31:20.414 INFO:tasks.workunit.client.1.smithi066.stdout:2/424: symlink d0/d12/d3e/l83 0 2024-02-28T16:31:20.414 INFO:tasks.workunit.client.0.smithi001.stdout:7/807: sync 2024-02-28T16:31:20.414 INFO:tasks.workunit.client.0.smithi001.stdout:5/757: dread dc/df/d79/f95 [0,4194304] 0 2024-02-28T16:31:20.419 INFO:tasks.workunit.client.1.smithi066.stdout:5/412: truncate d3/d4/dc/dd/d2e/d18/f4b 467911 0 2024-02-28T16:31:20.420 INFO:tasks.workunit.client.1.smithi066.stdout:0/524: creat d2/d27/d41/d76/d93/d10/d23/d6d/faf x:0 0 0 2024-02-28T16:31:20.421 INFO:tasks.workunit.client.0.smithi001.stdout:7/808: truncate d3/d17/d3d/f72 815527 0 2024-02-28T16:31:20.423 INFO:tasks.workunit.client.0.smithi001.stdout:6/767: symlink d9/d29/d6e/lfe 0 2024-02-28T16:31:20.426 INFO:tasks.workunit.client.1.smithi066.stdout:8/400: creat d0/f7c x:0 0 0 2024-02-28T16:31:20.427 INFO:tasks.workunit.client.1.smithi066.stdout:1/423: creat d2/d27/d2b/d43/d46/d80/f87 x:0 0 0 2024-02-28T16:31:20.428 INFO:tasks.workunit.client.0.smithi001.stdout:5/758: creat dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/ff5 x:0 0 0 2024-02-28T16:31:20.429 INFO:tasks.workunit.client.1.smithi066.stdout:3/433: symlink db/d13/d14/d2d/d92/l95 0 2024-02-28T16:31:20.431 INFO:tasks.workunit.client.0.smithi001.stdout:4/888: link d3/d1c/da0/fd0 d3/d5/d7/d1f/d2e/dbe/d115/f123 0 2024-02-28T16:31:20.432 INFO:tasks.workunit.client.1.smithi066.stdout:8/401: creat d0/db/f7d x:0 0 0 2024-02-28T16:31:20.432 INFO:tasks.workunit.client.1.smithi066.stdout:4/534: dwrite d1/f4 [0,4194304] 0 2024-02-28T16:31:20.432 INFO:tasks.workunit.client.1.smithi066.stdout:1/424: stat d2/f5 0 2024-02-28T16:31:20.434 INFO:tasks.workunit.client.1.smithi066.stdout:0/525: symlink d2/d27/d41/d76/d93/d10/d50/d73/lb0 0 2024-02-28T16:31:20.436 INFO:tasks.workunit.client.1.smithi066.stdout:2/425: creat d0/d12/d2e/f84 x:0 0 0 2024-02-28T16:31:20.438 INFO:tasks.workunit.client.1.smithi066.stdout:7/356: creat dc/de/f65 x:0 0 0 2024-02-28T16:31:20.439 INFO:tasks.workunit.client.0.smithi001.stdout:8/780: dwrite d4/d7/d22/f48 [0,4194304] 0 2024-02-28T16:31:20.439 INFO:tasks.workunit.client.0.smithi001.stdout:7/809: mkdir d3/d17/d3d/d59/d104 0 2024-02-28T16:31:20.440 INFO:tasks.workunit.client.1.smithi066.stdout:4/535: creat d1/d18/d4e/fa5 x:0 0 0 2024-02-28T16:31:20.442 INFO:tasks.workunit.client.0.smithi001.stdout:0/768: dread d18/fab [0,4194304] 0 2024-02-28T16:31:20.446 INFO:tasks.workunit.client.1.smithi066.stdout:8/402: symlink d0/d6/dc/d16/d4c/d42/d73/l7e 0 2024-02-28T16:31:20.448 INFO:tasks.workunit.client.1.smithi066.stdout:4/536: read d1/fe [3527358,63847] 0 2024-02-28T16:31:20.448 INFO:tasks.workunit.client.0.smithi001.stdout:2/749: dwrite d0/d2/d3/de/d8a/fbd [0,4194304] 0 2024-02-28T16:31:20.452 INFO:tasks.workunit.client.1.smithi066.stdout:6/366: write d2/da/f22 [5826450,40254] 0 2024-02-28T16:31:20.454 INFO:tasks.workunit.client.0.smithi001.stdout:4/889: rmdir d3/d1c/d58/d6c 39 2024-02-28T16:31:20.456 INFO:tasks.workunit.client.1.smithi066.stdout:6/367: stat d2/d2a/f2f 0 2024-02-28T16:31:20.460 INFO:tasks.workunit.client.1.smithi066.stdout:4/537: creat d1/d23/d25/d43/fa6 x:0 0 0 2024-02-28T16:31:20.463 INFO:tasks.workunit.client.1.smithi066.stdout:6/368: chown d2/da/d19/d54/d5d/c78 242494428 1 2024-02-28T16:31:20.463 INFO:tasks.workunit.client.1.smithi066.stdout:3/434: dwrite db/d2a/f65 [0,4194304] 0 2024-02-28T16:31:20.464 INFO:tasks.workunit.client.0.smithi001.stdout:0/769: truncate d18/fac 637705 0 2024-02-28T16:31:20.466 INFO:tasks.workunit.client.1.smithi066.stdout:3/435: chown db/c71 14687793 1 2024-02-28T16:31:20.467 INFO:tasks.workunit.client.0.smithi001.stdout:2/750: mknod d0/d2/d3/d36/cfb 0 2024-02-28T16:31:20.468 INFO:tasks.workunit.client.0.smithi001.stdout:2/751: stat d0/d2/da6 0 2024-02-28T16:31:20.469 INFO:tasks.workunit.client.1.smithi066.stdout:0/526: rename d2/d27/d41/d76/d93/d10/d50/d51/c99 to d2/d27/d41/d76/d93/d10/d50/d51/cb1 0 2024-02-28T16:31:20.469 INFO:tasks.workunit.client.1.smithi066.stdout:2/426: dwrite d0/d12/d2e/d3b/f64 [4194304,4194304] 0 2024-02-28T16:31:20.469 INFO:tasks.workunit.client.1.smithi066.stdout:6/369: read d2/da/d19/f4f [5728630,27772] 0 2024-02-28T16:31:20.472 INFO:tasks.workunit.client.1.smithi066.stdout:3/436: creat db/d13/d1a/d25/f96 x:0 0 0 2024-02-28T16:31:20.472 INFO:tasks.workunit.client.0.smithi001.stdout:2/752: readlink d0/d2/d3/d36/d94/l9a 0 2024-02-28T16:31:20.473 INFO:tasks.workunit.client.1.smithi066.stdout:0/527: mkdir d2/d27/d41/d76/d93/db2 0 2024-02-28T16:31:20.474 INFO:tasks.workunit.client.0.smithi001.stdout:4/890: unlink d3/d1c/fc2 0 2024-02-28T16:31:20.475 INFO:tasks.workunit.client.1.smithi066.stdout:3/437: fdatasync db/d13/d14/f15 0 2024-02-28T16:31:20.477 INFO:tasks.workunit.client.1.smithi066.stdout:2/427: rename d0/l2a to d0/d12/d1a/d23/d29/d21/d2f/l85 0 2024-02-28T16:31:20.479 INFO:tasks.workunit.client.1.smithi066.stdout:3/438: dread db/d7a/f88 [0,4194304] 0 2024-02-28T16:31:20.480 INFO:tasks.workunit.client.0.smithi001.stdout:9/968: dread d5/d27/db1/d5e/dd3/d109/d6f/dc8/fbe [0,4194304] 0 2024-02-28T16:31:20.484 INFO:tasks.workunit.client.1.smithi066.stdout:6/370: rename d2/da/d3e/f73 to d2/d2a/d3d/f7f 0 2024-02-28T16:31:20.487 INFO:tasks.workunit.client.1.smithi066.stdout:3/439: creat db/dc/d35/d3d/f97 x:0 0 0 2024-02-28T16:31:20.489 INFO:tasks.workunit.client.0.smithi001.stdout:2/753: creat d0/d2/d3/d36/ffc x:0 0 0 2024-02-28T16:31:20.490 INFO:tasks.workunit.client.1.smithi066.stdout:6/371: unlink d2/d2a/f45 0 2024-02-28T16:31:20.491 INFO:tasks.workunit.client.0.smithi001.stdout:5/759: link dc/df/cec dc/d12/d41/dd6/cf6 0 2024-02-28T16:31:20.492 INFO:tasks.workunit.client.0.smithi001.stdout:2/754: readlink d0/d2/d3/de/d8a/l31 0 2024-02-28T16:31:20.494 INFO:tasks.workunit.client.1.smithi066.stdout:3/440: readlink db/d13/d14/l46 0 2024-02-28T16:31:20.497 INFO:tasks.workunit.client.1.smithi066.stdout:0/528: dwrite d2/d27/d41/d76/d93/d10/d50/f57 [0,4194304] 0 2024-02-28T16:31:20.498 INFO:tasks.workunit.client.0.smithi001.stdout:4/891: mknod d3/c124 0 2024-02-28T16:31:20.499 INFO:tasks.workunit.client.1.smithi066.stdout:6/372: truncate d2/da/f76 801920 0 2024-02-28T16:31:20.503 INFO:tasks.workunit.client.1.smithi066.stdout:5/413: truncate d3/d4/dc/dd/d2e/f79 5589021 0 2024-02-28T16:31:20.505 INFO:tasks.workunit.client.0.smithi001.stdout:0/770: link d18/d22/d2b/d36/d46/df5/l68 d18/d22/d76/d93/lfe 0 2024-02-28T16:31:20.506 INFO:tasks.workunit.client.1.smithi066.stdout:4/538: dwrite d1/d23/d25/d3e/f66 [0,4194304] 0 2024-02-28T16:31:20.506 INFO:tasks.workunit.client.1.smithi066.stdout:5/414: chown d3/c10 162 1 2024-02-28T16:31:20.506 INFO:tasks.workunit.client.0.smithi001.stdout:0/771: chown f13 6716400 1 2024-02-28T16:31:20.507 INFO:tasks.workunit.client.1.smithi066.stdout:2/428: getdents d0/d12/d1a/d23/d29 0 2024-02-28T16:31:20.508 INFO:tasks.workunit.client.0.smithi001.stdout:8/781: dread d4/d7/f10 [8388608,4194304] 0 2024-02-28T16:31:20.509 INFO:tasks.workunit.client.1.smithi066.stdout:1/425: truncate d2/d27/d2b/d43/d46/f61 3660014 0 2024-02-28T16:31:20.509 INFO:tasks.workunit.client.1.smithi066.stdout:3/441: unlink db/d13/d1a/d33/l45 0 2024-02-28T16:31:20.518 INFO:tasks.workunit.client.1.smithi066.stdout:0/529: fsync d2/d27/d41/d76/d93/d10/d50/d51/f5a 0 2024-02-28T16:31:20.524 INFO:tasks.workunit.client.1.smithi066.stdout:2/429: truncate d0/d12/d1a/f3a 916734 0 2024-02-28T16:31:20.530 INFO:tasks.workunit.client.1.smithi066.stdout:3/442: rmdir db/d13/d14/d24 39 2024-02-28T16:31:20.536 INFO:tasks.workunit.client.1.smithi066.stdout:0/530: truncate d2/d27/d41/d76/d93/d10/d50/d52/f9a 1897754 0 2024-02-28T16:31:20.538 INFO:tasks.workunit.client.1.smithi066.stdout:8/403: dread d0/d6/dc/d16/d4c/d6c/d6f/f39 [4194304,4194304] 0 2024-02-28T16:31:20.541 INFO:tasks.workunit.client.0.smithi001.stdout:9/969: dread d5/f64 [0,4194304] 0 2024-02-28T16:31:20.541 INFO:tasks.workunit.client.0.smithi001.stdout:4/892: truncate d3/fb4 799316 0 2024-02-28T16:31:20.542 INFO:tasks.workunit.client.1.smithi066.stdout:7/357: dread dc/d29/f44 [0,4194304] 0 2024-02-28T16:31:20.543 INFO:tasks.workunit.client.0.smithi001.stdout:9/970: stat d5/d27/f73 0 2024-02-28T16:31:20.544 INFO:tasks.workunit.client.1.smithi066.stdout:4/539: dwrite d1/d23/d25/d43/f76 [0,4194304] 0 2024-02-28T16:31:20.550 INFO:tasks.workunit.client.0.smithi001.stdout:9/971: fsync d5/d27/d42/def/ff6 0 2024-02-28T16:31:20.550 INFO:tasks.workunit.client.1.smithi066.stdout:2/430: mknod d0/d12/d2e/c86 0 2024-02-28T16:31:20.551 INFO:tasks.workunit.client.1.smithi066.stdout:3/443: symlink db/d13/d14/l98 0 2024-02-28T16:31:20.552 INFO:tasks.workunit.client.1.smithi066.stdout:8/404: creat d0/d6/dc/d16/d4c/d42/d73/f7f x:0 0 0 2024-02-28T16:31:20.556 INFO:tasks.workunit.client.1.smithi066.stdout:4/540: chown d1/d23/d25/d43/c73 3811457 1 2024-02-28T16:31:20.557 INFO:tasks.workunit.client.1.smithi066.stdout:1/426: write d2/d27/d2b/d37/d51/f60 [1094016,66998] 0 2024-02-28T16:31:20.557 INFO:tasks.workunit.client.1.smithi066.stdout:5/415: truncate d3/d4/dc/dd/d2e/d18/f4a 3167889 0 2024-02-28T16:31:20.560 INFO:tasks.workunit.client.1.smithi066.stdout:4/541: read d1/d18/d4e/f94 [3545853,36249] 0 2024-02-28T16:31:20.561 INFO:tasks.workunit.client.0.smithi001.stdout:1/905: dwrite d0/d9/d1a/d2b/d26/d27/d30/f59 [0,4194304] 0 2024-02-28T16:31:20.562 INFO:tasks.workunit.client.1.smithi066.stdout:8/405: chown d0/db/f7d 11522 1 2024-02-28T16:31:20.563 INFO:tasks.workunit.client.1.smithi066.stdout:0/531: mknod d2/d27/d41/d76/d93/d10/d50/d52/da2/cb3 0 2024-02-28T16:31:20.564 INFO:tasks.workunit.client.1.smithi066.stdout:2/431: mkdir d0/d12/d1a/d39/d7c/d87 0 2024-02-28T16:31:20.568 INFO:tasks.workunit.client.1.smithi066.stdout:4/542: mkdir d1/d18/d41/da7 0 2024-02-28T16:31:20.569 INFO:tasks.workunit.client.0.smithi001.stdout:6/768: write d9/db/d5f/d89/d8a/fbb [549090,119693] 0 2024-02-28T16:31:20.570 INFO:tasks.workunit.client.1.smithi066.stdout:7/358: rmdir dc/d63 0 2024-02-28T16:31:20.572 INFO:tasks.workunit.client.1.smithi066.stdout:7/359: fdatasync dc/d29/f41 0 2024-02-28T16:31:20.572 INFO:tasks.workunit.client.0.smithi001.stdout:1/906: mknod d0/d9/d1a/d2b/d26/d27/dce/c13c 0 2024-02-28T16:31:20.576 INFO:tasks.workunit.client.0.smithi001.stdout:9/972: dwrite d5/d27/d34/fc4 [0,4194304] 0 2024-02-28T16:31:20.579 INFO:tasks.workunit.client.1.smithi066.stdout:3/444: write db/dc/d35/f60 [2076305,109321] 0 2024-02-28T16:31:20.579 INFO:tasks.workunit.client.1.smithi066.stdout:1/427: write d2/d27/f3c [2365437,123666] 0 2024-02-28T16:31:20.580 INFO:tasks.workunit.client.0.smithi001.stdout:0/772: write d18/d22/d76/d93/f94 [618256,80207] 0 2024-02-28T16:31:20.580 INFO:tasks.workunit.client.0.smithi001.stdout:8/782: write d4/d7/d22/f3f [4910785,106154] 0 2024-02-28T16:31:20.580 INFO:tasks.workunit.client.0.smithi001.stdout:4/893: write d3/d5/d7/d1f/d2e/f64 [1756662,47690] 0 2024-02-28T16:31:20.581 INFO:tasks.workunit.client.0.smithi001.stdout:1/907: dread - d0/d9/d1a/d2b/d26/d27/dce/f134 zero size 2024-02-28T16:31:20.582 INFO:tasks.workunit.client.1.smithi066.stdout:0/532: creat d2/d27/d41/d76/d93/d33/d3f/d8e/fb4 x:0 0 0 2024-02-28T16:31:20.585 INFO:tasks.workunit.client.0.smithi001.stdout:6/769: rename d9/da6/f30 to d9/d29/d6e/d9a/fff 0 2024-02-28T16:31:20.588 INFO:tasks.workunit.client.0.smithi001.stdout:1/908: readlink d0/d9/d1a/d2b/dfe/ld7 0 2024-02-28T16:31:20.592 INFO:tasks.workunit.client.1.smithi066.stdout:7/360: unlink dc/de/d17/d1e/f23 0 2024-02-28T16:31:20.592 INFO:tasks.workunit.client.1.smithi066.stdout:8/406: write d0/db/f5f [725257,42478] 0 2024-02-28T16:31:20.594 INFO:tasks.workunit.client.1.smithi066.stdout:4/543: mkdir d1/d18/da8 0 2024-02-28T16:31:20.595 INFO:tasks.workunit.client.0.smithi001.stdout:0/773: creat d18/d22/d2f/d1c/fff x:0 0 0 2024-02-28T16:31:20.596 INFO:tasks.workunit.client.0.smithi001.stdout:5/760: dwrite dc/df/f3a [0,4194304] 0 2024-02-28T16:31:20.596 INFO:tasks.workunit.client.1.smithi066.stdout:1/428: creat d2/d27/d2b/d43/d46/f88 x:0 0 0 2024-02-28T16:31:20.600 INFO:tasks.workunit.client.1.smithi066.stdout:0/533: write d2/d27/d41/d76/d93/d33/d3f/d8e/fb4 [688302,67723] 0 2024-02-28T16:31:20.600 INFO:tasks.workunit.client.0.smithi001.stdout:8/783: rename d4/d7/cf2 to d4/d7/da8/cfe 0 2024-02-28T16:31:20.603 INFO:tasks.workunit.client.1.smithi066.stdout:3/445: unlink db/d13/d1a/f1b 0 2024-02-28T16:31:20.603 INFO:tasks.workunit.client.1.smithi066.stdout:5/416: dwrite d3/d1b/f5f [0,4194304] 0 2024-02-28T16:31:20.608 INFO:tasks.workunit.client.1.smithi066.stdout:8/407: truncate d0/d6/dc/d16/d4c/d42/d73/f7f 170392 0 2024-02-28T16:31:20.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.619+0000 7fe76ae00700 1 -- 172.21.15.1:0/1273029420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 msgr2=0x7fe7640fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:20.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.619+0000 7fe76ae00700 1 --2- 172.21.15.1:0/1273029420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 0x7fe7640fd320 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe754004940 tx=0x7fe754009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:20.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.620+0000 7fe76ae00700 1 -- 172.21.15.1:0/1273029420 shutdown_connections 2024-02-28T16:31:20.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.620+0000 7fe76ae00700 1 --2- 172.21.15.1:0/1273029420 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7640fe0e0 0x7fe7640fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:20.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.620+0000 7fe76ae00700 1 --2- 172.21.15.1:0/1273029420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 0x7fe7640fd320 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:20.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.620+0000 7fe76ae00700 1 -- 172.21.15.1:0/1273029420 >> 172.21.15.1:0/1273029420 conn(0x7fe7640f8620 msgr2=0x7fe7640faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:20.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.620+0000 7fe76ae00700 1 -- 172.21.15.1:0/1273029420 shutdown_connections 2024-02-28T16:31:20.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.620+0000 7fe76ae00700 1 -- 172.21.15.1:0/1273029420 wait complete. 2024-02-28T16:31:20.610 INFO:tasks.workunit.client.1.smithi066.stdout:7/361: creat dc/d29/d5c/f66 x:0 0 0 2024-02-28T16:31:20.611 INFO:tasks.workunit.client.0.smithi001.stdout:6/770: creat d9/db/d5f/d69/f100 x:0 0 0 2024-02-28T16:31:20.612 INFO:tasks.workunit.client.1.smithi066.stdout:3/446: chown db/d13/d14/f6f 348 1 2024-02-28T16:31:20.613 INFO:tasks.workunit.client.1.smithi066.stdout:2/432: dread d0/d12/d2e/d7a/f5b [0,4194304] 0 2024-02-28T16:31:20.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.624+0000 7fe76ae00700 1 Processor -- start 2024-02-28T16:31:20.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.624+0000 7fe76ae00700 1 -- start start 2024-02-28T16:31:20.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.625+0000 7fe76ae00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 0x7fe76410d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:20.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.625+0000 7fe76ae00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7640fe0e0 0x7fe76410dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:20.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.625+0000 7fe76ae00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76410e260 con 0x7fe7640fcef0 2024-02-28T16:31:20.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.625+0000 7fe76ae00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76410eb00 con 0x7fe7640fe0e0 2024-02-28T16:31:20.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.625+0000 7fe763e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 0x7fe76410d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:20.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.625+0000 7fe763e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 0x7fe76410d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40820/0 (socket says 172.21.15.1:40820) 2024-02-28T16:31:20.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.625+0000 7fe763e00700 1 -- 172.21.15.1:0/1088351272 learned_addr learned my addr 172.21.15.1:0/1088351272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:20.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.625+0000 7fe763400700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7640fe0e0 0x7fe76410dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:20.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.626+0000 7fe763e00700 1 -- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7640fe0e0 msgr2=0x7fe76410dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:20.617 INFO:tasks.workunit.client.1.smithi066.stdout:2/433: fdatasync d0/f1f 0 2024-02-28T16:31:20.618 INFO:tasks.workunit.client.1.smithi066.stdout:4/544: creat d1/fa9 x:0 0 0 2024-02-28T16:31:20.618 INFO:tasks.workunit.client.1.smithi066.stdout:0/534: write d2/d27/d41/d76/d93/d10/f26 [2149539,67940] 0 2024-02-28T16:31:20.620 INFO:tasks.workunit.client.0.smithi001.stdout:5/761: symlink dc/df/d79/d98/lf7 0 2024-02-28T16:31:20.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.626+0000 7fe763e00700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7640fe0e0 0x7fe76410dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:20.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.626+0000 7fe763e00700 1 -- 172.21.15.1:0/1088351272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe754005040 con 0x7fe7640fcef0 2024-02-28T16:31:20.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.626+0000 7fe763e00700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 0x7fe76410d720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe7540043d0 tx=0x7fe754009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:20.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.627+0000 7fe75be00700 1 -- 172.21.15.1:0/1088351272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe75400e040 con 0x7fe7640fcef0 2024-02-28T16:31:20.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.627+0000 7fe75be00700 1 -- 172.21.15.1:0/1088351272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7540138f0 con 0x7fe7640fcef0 2024-02-28T16:31:20.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.627+0000 7fe75be00700 1 -- 172.21.15.1:0/1088351272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe75401c870 con 0x7fe7640fcef0 2024-02-28T16:31:20.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.627+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76410ed80 con 0x7fe7640fcef0 2024-02-28T16:31:20.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.627+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76410f2c0 con 0x7fe7640fcef0 2024-02-28T16:31:20.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.629+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76404bed0 con 0x7fe7640fcef0 2024-02-28T16:31:20.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.631+0000 7fe75be00700 1 -- 172.21.15.1:0/1088351272 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 25) v1 ==== 88910+0+0 (secure 0 0 0) 0x7fe754013420 con 0x7fe7640fcef0 2024-02-28T16:31:20.622 INFO:tasks.workunit.client.1.smithi066.stdout:2/434: chown d0/d12/d1a/d39 20 1 2024-02-28T16:31:20.622 INFO:tasks.workunit.client.1.smithi066.stdout:1/429: link d2/d27/d2b/d34/f44 d2/d18/d19/f89 0 2024-02-28T16:31:20.622 INFO:tasks.workunit.client.1.smithi066.stdout:5/417: mkdir d3/d4/dc/d2f/d55/d74/d84 0 2024-02-28T16:31:20.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.633+0000 7fe75be00700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fe74806b1d0 0x7fe74806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:20.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.633+0000 7fe75be00700 1 -- 172.21.15.1:0/1088351272 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fe75405a3e0 con 0x7fe7640fcef0 2024-02-28T16:31:20.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.634+0000 7fe763400700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fe74806b1d0 0x7fe74806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:20.623 INFO:tasks.workunit.client.0.smithi001.stdout:0/774: truncate d18/d22/d3d/fa1 268716 0 2024-02-28T16:31:20.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.635+0000 7fe763400700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fe74806b1d0 0x7fe74806d680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe750004990 tx=0x7fe750009ee0 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:20.624 INFO:tasks.workunit.client.0.smithi001.stdout:8/784: dread - d4/d7/d22/d27/d4b/fa4 zero size 2024-02-28T16:31:20.625 INFO:tasks.workunit.client.1.smithi066.stdout:2/435: chown d0/d12/d1a/f2b 502797 1 2024-02-28T16:31:20.626 INFO:tasks.workunit.client.1.smithi066.stdout:4/545: symlink d1/d23/d25/d3e/laa 0 2024-02-28T16:31:20.628 INFO:tasks.workunit.client.1.smithi066.stdout:3/447: creat db/d13/d14/f99 x:0 0 0 2024-02-28T16:31:20.628 INFO:tasks.workunit.client.1.smithi066.stdout:2/436: stat d0/d12/d2e/d7a/l6c 0 2024-02-28T16:31:20.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:20.639+0000 7fe75be00700 1 -- 172.21.15.1:0/1088351272 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fe754059ab0 con 0x7fe7640fcef0 2024-02-28T16:31:20.630 INFO:tasks.workunit.client.1.smithi066.stdout:2/437: chown d0/ff 32526 1 2024-02-28T16:31:20.630 INFO:tasks.workunit.client.1.smithi066.stdout:1/430: chown d2/c5a 868 1 2024-02-28T16:31:20.631 INFO:tasks.workunit.client.1.smithi066.stdout:5/418: write d3/d4/dc/dd/d35/f64 [4941176,21408] 0 2024-02-28T16:31:20.631 INFO:tasks.workunit.client.0.smithi001.stdout:4/894: getdents d3/d5/d7/d1f/d2e 0 2024-02-28T16:31:20.632 INFO:tasks.workunit.client.0.smithi001.stdout:6/771: mknod d9/db/d5f/c101 0 2024-02-28T16:31:20.634 INFO:tasks.workunit.client.0.smithi001.stdout:0/775: write d18/d22/d3d/f5e [9020707,82407] 0 2024-02-28T16:31:20.634 INFO:tasks.workunit.client.1.smithi066.stdout:5/419: readlink d3/d4/dc/dd/l34 0 2024-02-28T16:31:20.635 INFO:tasks.workunit.client.1.smithi066.stdout:0/535: getdents d2/d27/d41/d76/d93/db2 0 2024-02-28T16:31:20.635 INFO:tasks.workunit.client.1.smithi066.stdout:3/448: chown db/d2a/l31 5019 1 2024-02-28T16:31:20.636 INFO:tasks.workunit.client.0.smithi001.stdout:4/895: chown d3/f12 1336370 1 2024-02-28T16:31:20.636 INFO:tasks.workunit.client.1.smithi066.stdout:7/362: dread dc/de/d17/d1e/f20 [0,4194304] 0 2024-02-28T16:31:20.637 INFO:tasks.workunit.client.1.smithi066.stdout:3/449: dread - db/d2a/d44/d48/f6c zero size 2024-02-28T16:31:20.638 INFO:tasks.workunit.client.1.smithi066.stdout:8/408: rename d0/db/d12/d1d/d25/f47 to d0/db/f80 0 2024-02-28T16:31:20.639 INFO:tasks.workunit.client.0.smithi001.stdout:8/785: link d4/d32/d69/d9a/dcd/lf0 d4/d47/d8b/lff 0 2024-02-28T16:31:20.639 INFO:tasks.workunit.client.1.smithi066.stdout:0/536: dread - d2/d27/d41/d76/d93/d33/d3f/faa zero size 2024-02-28T16:31:20.640 INFO:tasks.workunit.client.1.smithi066.stdout:8/409: dread - d0/db/f7d zero size 2024-02-28T16:31:20.641 INFO:tasks.workunit.client.0.smithi001.stdout:1/909: dwrite d0/d9/d1a/d2b/d26/f120 [8388608,4194304] 0 2024-02-28T16:31:20.641 INFO:tasks.workunit.client.0.smithi001.stdout:4/896: chown d3/d5/d7/d1f/l29 738112 1 2024-02-28T16:31:20.642 INFO:tasks.workunit.client.1.smithi066.stdout:9/350: sync 2024-02-28T16:31:20.642 INFO:tasks.workunit.client.1.smithi066.stdout:1/431: fdatasync d2/fc 0 2024-02-28T16:31:20.644 INFO:tasks.workunit.client.0.smithi001.stdout:8/786: chown d4/d7/d22/d27/d4b/fa3 1369431 1 2024-02-28T16:31:20.645 INFO:tasks.workunit.client.1.smithi066.stdout:8/410: fsync d0/d6/dc/d16/d4c/d6c/f78 0 2024-02-28T16:31:20.648 INFO:tasks.workunit.client.1.smithi066.stdout:7/363: mknod dc/d29/d5c/c67 0 2024-02-28T16:31:20.650 INFO:tasks.workunit.client.0.smithi001.stdout:0/776: rmdir d18/d22/d2b 39 2024-02-28T16:31:20.651 INFO:tasks.workunit.client.0.smithi001.stdout:1/910: chown d0/d9/d56/d2f/f7c 19 1 2024-02-28T16:31:20.651 INFO:tasks.workunit.client.0.smithi001.stdout:6/772: truncate d9/db/d10/faa 838802 0 2024-02-28T16:31:20.653 INFO:tasks.workunit.client.1.smithi066.stdout:9/351: truncate da/df/d18/d2d/f4c 5099028 0 2024-02-28T16:31:20.653 INFO:tasks.workunit.client.1.smithi066.stdout:4/546: link d1/d18/f4a d1/d18/d7e/fab 0 2024-02-28T16:31:20.653 INFO:tasks.workunit.client.1.smithi066.stdout:5/420: dread d3/d4/dc/f1d [0,4194304] 0 2024-02-28T16:31:20.658 INFO:tasks.workunit.client.0.smithi001.stdout:8/787: truncate d4/d7/d22/d27/d58/d62/fb2 536143 0 2024-02-28T16:31:20.659 INFO:tasks.workunit.client.1.smithi066.stdout:5/421: read - d3/d4/dc/dd/d2e/d18/f82 zero size 2024-02-28T16:31:20.661 INFO:tasks.workunit.client.1.smithi066.stdout:0/537: rename d2/d27/d41/d76/d93/d10/d50 to d2/d27/d41/d76/d93/db2/db5 0 2024-02-28T16:31:20.662 INFO:tasks.workunit.client.1.smithi066.stdout:0/538: fdatasync d2/f96 0 2024-02-28T16:31:20.663 INFO:tasks.workunit.client.1.smithi066.stdout:5/422: write d3/d4/dc/dd/d2e/d18/f82 [360587,108934] 0 2024-02-28T16:31:20.664 INFO:tasks.workunit.client.0.smithi001.stdout:1/911: symlink d0/d9/d1a/d2b/l13d 0 2024-02-28T16:31:20.668 INFO:tasks.workunit.client.1.smithi066.stdout:0/539: dread d2/f32 [0,4194304] 0 2024-02-28T16:31:20.669 INFO:tasks.workunit.client.1.smithi066.stdout:3/450: dwrite db/d13/f23 [4194304,4194304] 0 2024-02-28T16:31:20.672 INFO:tasks.workunit.client.1.smithi066.stdout:4/547: symlink d1/d23/d25/d3e/d40/lac 0 2024-02-28T16:31:20.675 INFO:tasks.workunit.client.1.smithi066.stdout:3/451: write db/d2a/d44/d48/d5d/f81 [26878,109452] 0 2024-02-28T16:31:20.677 INFO:tasks.workunit.client.0.smithi001.stdout:4/897: dwrite d3/d2d/d4c/d69/d82/f122 [0,4194304] 0 2024-02-28T16:31:20.678 INFO:tasks.workunit.client.1.smithi066.stdout:3/452: chown db/d2a/d44/d48/d5d 418525811 1 2024-02-28T16:31:20.679 INFO:tasks.workunit.client.0.smithi001.stdout:1/912: creat d0/d9/d1a/d2b/d26/f13e x:0 0 0 2024-02-28T16:31:20.680 INFO:tasks.workunit.client.1.smithi066.stdout:7/364: mknod dc/de/c68 0 2024-02-28T16:31:20.682 INFO:tasks.workunit.client.1.smithi066.stdout:5/423: creat d3/d4/dc/d42/d57/d66/f85 x:0 0 0 2024-02-28T16:31:20.682 INFO:tasks.workunit.client.1.smithi066.stdout:7/365: write dc/de/f3f [710173,118199] 0 2024-02-28T16:31:20.682 INFO:tasks.workunit.client.0.smithi001.stdout:3/795: sync 2024-02-28T16:31:20.685 INFO:tasks.workunit.client.1.smithi066.stdout:2/438: dread d0/d12/d1a/d23/d5d/f4d [0,4194304] 0 2024-02-28T16:31:20.685 INFO:tasks.workunit.client.1.smithi066.stdout:0/540: write d2/d27/d41/d76/d93/d10/f2c [5953969,47681] 0 2024-02-28T16:31:20.687 INFO:tasks.workunit.client.1.smithi066.stdout:8/411: link d0/db/d15/f7a d0/d6/dc/d16/d1a/d63/d58/d5a/f81 0 2024-02-28T16:31:20.687 INFO:tasks.workunit.client.1.smithi066.stdout:5/424: fdatasync d3/d4/dc/d2f/d55/f70 0 2024-02-28T16:31:20.688 INFO:tasks.workunit.client.0.smithi001.stdout:3/796: chown d4/d6/f1c 2 1 2024-02-28T16:31:20.692 INFO:tasks.workunit.client.0.smithi001.stdout:5/762: dread dc/d1a/d52/f89 [0,4194304] 0 2024-02-28T16:31:20.696 INFO:tasks.workunit.client.1.smithi066.stdout:3/453: creat db/f9a x:0 0 0 2024-02-28T16:31:20.697 INFO:tasks.workunit.client.0.smithi001.stdout:6/773: dwrite d9/de/f2c [4194304,4194304] 0 2024-02-28T16:31:20.697 INFO:tasks.workunit.client.1.smithi066.stdout:9/352: creat da/df/d18/f78 x:0 0 0 2024-02-28T16:31:20.707 INFO:tasks.workunit.client.1.smithi066.stdout:0/541: mkdir d2/d27/d41/d76/d93/d10/d23/db6 0 2024-02-28T16:31:20.708 INFO:tasks.workunit.client.0.smithi001.stdout:5/763: creat dc/d1a/d21/de2/ff8 x:0 0 0 2024-02-28T16:31:20.710 INFO:tasks.workunit.client.0.smithi001.stdout:8/788: write d4/d7/d22/d27/d58/d94/dbc/fbd [5141535,35584] 0 2024-02-28T16:31:20.710 INFO:tasks.workunit.client.1.smithi066.stdout:6/373: sync 2024-02-28T16:31:20.710 INFO:tasks.workunit.client.1.smithi066.stdout:0/542: fsync d2/d27/d41/d76/d93/d10/f98 0 2024-02-28T16:31:20.712 INFO:tasks.workunit.client.1.smithi066.stdout:8/412: creat d0/db/d12/d1d/d25/d4f/f82 x:0 0 0 2024-02-28T16:31:20.715 INFO:tasks.workunit.client.1.smithi066.stdout:1/432: dwrite d2/d18/d19/f3b [0,4194304] 0 2024-02-28T16:31:20.715 INFO:tasks.workunit.client.0.smithi001.stdout:6/774: getdents d9/db/d5f/d62/def 0 2024-02-28T16:31:20.715 INFO:tasks.workunit.client.0.smithi001.stdout:0/777: dread d18/d22/d76/d93/f94 [0,4194304] 0 2024-02-28T16:31:20.718 INFO:tasks.workunit.client.1.smithi066.stdout:1/433: stat d2/d27/d2b/d37/c4b 0 2024-02-28T16:31:20.722 INFO:tasks.workunit.client.0.smithi001.stdout:5/764: symlink dc/lf9 0 2024-02-28T16:31:20.729 INFO:tasks.workunit.client.0.smithi001.stdout:6/775: creat d9/db/d5f/dd4/dfb/f102 x:0 0 0 2024-02-28T16:31:20.733 INFO:tasks.workunit.client.0.smithi001.stdout:1/913: dwrite d0/d9/d1a/d2b/d26/d27/df2/f139 [0,4194304] 0 2024-02-28T16:31:20.735 INFO:tasks.workunit.client.0.smithi001.stdout:4/898: dwrite d3/d5/d7/d1f/d2e/f39 [0,4194304] 0 2024-02-28T16:31:20.737 INFO:tasks.workunit.client.0.smithi001.stdout:3/797: dwrite d4/d18/f69 [0,4194304] 0 2024-02-28T16:31:20.745 INFO:tasks.workunit.client.0.smithi001.stdout:0/778: rename d18/fab to d18/d22/d2f/d1c/d90/f100 0 2024-02-28T16:31:20.751 INFO:tasks.workunit.client.0.smithi001.stdout:5/765: mkdir dc/d1a/d52/dfa 0 2024-02-28T16:31:20.751 INFO:tasks.workunit.client.0.smithi001.stdout:1/914: mknod d0/d9/d1a/d2b/d26/d105/d121/c13f 0 2024-02-28T16:31:20.751 INFO:tasks.workunit.client.0.smithi001.stdout:8/789: write d4/d32/f7c [4023389,33514] 0 2024-02-28T16:31:20.752 INFO:tasks.workunit.client.0.smithi001.stdout:3/798: symlink d4/d18/ddb/l104 0 2024-02-28T16:31:20.756 INFO:tasks.workunit.client.1.smithi066.stdout:7/366: symlink dc/d56/l69 0 2024-02-28T16:31:20.756 INFO:tasks.workunit.client.1.smithi066.stdout:2/439: rename d0/d12/d1a/c27 to d0/d12/d2e/c88 0 2024-02-28T16:31:20.756 INFO:tasks.workunit.client.1.smithi066.stdout:4/548: write d1/d18/d41/f68 [193884,82793] 0 2024-02-28T16:31:20.757 INFO:tasks.workunit.client.1.smithi066.stdout:0/543: mkdir d2/d27/d41/d76/d93/d33/d3f/db7 0 2024-02-28T16:31:20.759 INFO:tasks.workunit.client.1.smithi066.stdout:5/425: truncate d3/d4/dc/dd/d2e/d18/f2c 682480 0 2024-02-28T16:31:20.760 INFO:tasks.workunit.client.0.smithi001.stdout:0/779: dread - d18/d22/d2b/d95/fb0 zero size 2024-02-28T16:31:20.760 INFO:tasks.workunit.client.1.smithi066.stdout:3/454: rename db/d13/d14/l46 to db/d7a/d7c/l9b 0 2024-02-28T16:31:20.761 INFO:tasks.workunit.client.0.smithi001.stdout:2/755: sync 2024-02-28T16:31:20.761 INFO:tasks.workunit.client.0.smithi001.stdout:9/973: sync 2024-02-28T16:31:20.761 INFO:tasks.workunit.client.0.smithi001.stdout:7/810: sync 2024-02-28T16:31:20.761 INFO:tasks.workunit.client.0.smithi001.stdout:4/899: mkdir d3/d5/d7/ddb/d125 0 2024-02-28T16:31:20.762 INFO:tasks.workunit.client.1.smithi066.stdout:4/549: fsync d1/d23/d25/d3e/d40/f72 0 2024-02-28T16:31:20.762 INFO:tasks.workunit.client.1.smithi066.stdout:1/434: unlink d2/f16 0 2024-02-28T16:31:20.764 INFO:tasks.workunit.client.1.smithi066.stdout:7/367: dread dc/f4c [0,4194304] 0 2024-02-28T16:31:20.764 INFO:tasks.workunit.client.1.smithi066.stdout:3/455: fsync db/dc/f55 0 2024-02-28T16:31:20.765 INFO:tasks.workunit.client.1.smithi066.stdout:2/440: creat d0/d12/d1a/d23/d29/d21/f89 x:0 0 0 2024-02-28T16:31:20.765 INFO:tasks.workunit.client.0.smithi001.stdout:1/915: chown d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/d9c/cf4 216135 1 2024-02-28T16:31:20.765 INFO:tasks.workunit.client.0.smithi001.stdout:3/799: creat d4/d1e/dcf/f105 x:0 0 0 2024-02-28T16:31:20.767 INFO:tasks.workunit.client.1.smithi066.stdout:6/374: rename d2/da/d19/d54 to d2/d26/d32/d46/d80 0 2024-02-28T16:31:20.767 INFO:tasks.workunit.client.1.smithi066.stdout:7/368: write dc/f5d [252700,2106] 0 2024-02-28T16:31:20.770 INFO:tasks.workunit.client.0.smithi001.stdout:8/790: creat d4/d7/d22/df6/f100 x:0 0 0 2024-02-28T16:31:20.773 INFO:tasks.workunit.client.1.smithi066.stdout:3/456: rmdir db/dc/d11/d1f/d78 39 2024-02-28T16:31:20.775 INFO:tasks.workunit.client.1.smithi066.stdout:2/441: unlink d0/ff 0 2024-02-28T16:31:20.775 INFO:tasks.workunit.client.0.smithi001.stdout:9/974: mknod d5/d27/db1/da6/c135 0 2024-02-28T16:31:20.776 INFO:tasks.workunit.client.0.smithi001.stdout:4/900: mkdir d3/d2d/d4c/d69/d126 0 2024-02-28T16:31:20.776 INFO:tasks.workunit.client.1.smithi066.stdout:8/413: rename d0/d6/dc/d16/d1a/d63/c34 to d0/d6/dc/d16/d1a/d63/d58/d5a/c83 0 2024-02-28T16:31:20.777 INFO:tasks.workunit.client.1.smithi066.stdout:4/550: mknod d1/d18/cad 0 2024-02-28T16:31:20.777 INFO:tasks.workunit.client.1.smithi066.stdout:2/442: dread - d0/d12/d1a/f79 zero size 2024-02-28T16:31:20.779 INFO:tasks.workunit.client.1.smithi066.stdout:7/369: symlink dc/de/d17/d37/l6a 0 2024-02-28T16:31:20.779 INFO:tasks.workunit.client.1.smithi066.stdout:8/414: chown d0/db/d12/d1d/d25/d72 1296 1 2024-02-28T16:31:20.779 INFO:tasks.workunit.client.0.smithi001.stdout:6/776: rename d9/d29/d6e/d53/fc9 to d9/f103 0 2024-02-28T16:31:20.780 INFO:tasks.workunit.client.1.smithi066.stdout:6/375: creat d2/da/d3e/f81 x:0 0 0 2024-02-28T16:31:20.783 INFO:tasks.workunit.client.0.smithi001.stdout:9/975: read d5/d27/db1/d5e/dd3/d109/d6f/dc8/f106 [436009,57740] 0 2024-02-28T16:31:20.784 INFO:tasks.workunit.client.0.smithi001.stdout:5/766: dwrite dc/d12/d56/feb [0,4194304] 0 2024-02-28T16:31:20.789 INFO:tasks.workunit.client.0.smithi001.stdout:7/811: symlink d3/d17/d52/d7f/d99/dc1/l105 0 2024-02-28T16:31:20.793 INFO:tasks.workunit.client.0.smithi001.stdout:4/901: creat d3/d1c/da0/f127 x:0 0 0 2024-02-28T16:31:20.798 INFO:tasks.workunit.client.0.smithi001.stdout:4/902: chown d3/d2d/d81 32668729 1 2024-02-28T16:31:20.800 INFO:tasks.workunit.client.0.smithi001.stdout:3/800: dwrite d4/d6/f52 [0,4194304] 0 2024-02-28T16:31:20.805 INFO:tasks.workunit.client.0.smithi001.stdout:9/976: creat d5/d27/db1/d58/d130/f136 x:0 0 0 2024-02-28T16:31:20.806 INFO:tasks.workunit.client.0.smithi001.stdout:5/767: dread - dc/d12/d56/d96/db9/dc2/dca/d91/ff2 zero size 2024-02-28T16:31:20.811 INFO:tasks.workunit.client.0.smithi001.stdout:4/903: chown d3/d2d/d4c/d5e/db0/fc6 0 1 2024-02-28T16:31:20.813 INFO:tasks.workunit.client.0.smithi001.stdout:7/812: mkdir d3/d17/d52/d55/dca/d106 0 2024-02-28T16:31:20.814 INFO:tasks.workunit.client.0.smithi001.stdout:1/916: dwrite d0/d9/d1a/d2b/d26/f13e [0,4194304] 0 2024-02-28T16:31:20.814 INFO:tasks.workunit.client.0.smithi001.stdout:0/780: link d18/d22/d2f/ff3 d18/d22/d2f/f101 0 2024-02-28T16:31:20.825 INFO:tasks.workunit.client.0.smithi001.stdout:5/768: write dc/d12/d56/d96/db9/dc2/dca/d3c/da0/fe7 [961503,71931] 0 2024-02-28T16:31:20.829 INFO:tasks.workunit.client.0.smithi001.stdout:0/781: creat d18/da9/f102 x:0 0 0 2024-02-28T16:31:20.834 INFO:tasks.workunit.client.0.smithi001.stdout:8/791: dwrite d4/d47/d8b/f8c [4194304,4194304] 0 2024-02-28T16:31:20.836 INFO:tasks.workunit.client.1.smithi066.stdout:5/426: rename d3/d4/dc/d42/d57/d66 to d3/d4/dc/dd/d35/d86 0 2024-02-28T16:31:20.837 INFO:tasks.workunit.client.1.smithi066.stdout:1/435: link d2/d27/d2b/d43/d46/f63 d2/d27/d2b/d37/d54/f8a 0 2024-02-28T16:31:20.839 INFO:tasks.workunit.client.0.smithi001.stdout:7/813: creat d3/d17/d52/d55/dca/f107 x:0 0 0 2024-02-28T16:31:20.839 INFO:tasks.workunit.client.1.smithi066.stdout:4/551: truncate d1/d18/d4e/f50 1197384 0 2024-02-28T16:31:20.841 INFO:tasks.workunit.client.0.smithi001.stdout:3/801: truncate d4/d6/d7/d78/d81/fb2 567183 0 2024-02-28T16:31:20.841 INFO:tasks.workunit.client.1.smithi066.stdout:8/415: fsync d0/d6/dc/d16/d4c/d6c/d6f/f39 0 2024-02-28T16:31:20.844 INFO:tasks.workunit.client.0.smithi001.stdout:6/777: getdents d9/db/d5f/d89/d8a 0 2024-02-28T16:31:20.845 INFO:tasks.workunit.client.1.smithi066.stdout:1/436: dread d2/d18/f3a [0,4194304] 0 2024-02-28T16:31:20.845 INFO:tasks.workunit.client.1.smithi066.stdout:6/376: rename d2/da/d3e to d2/d2c/d82 0 2024-02-28T16:31:20.845 INFO:tasks.workunit.client.0.smithi001.stdout:4/904: fdatasync d3/d5/d7/d1f/d20/de0/de6/f7b 0 2024-02-28T16:31:20.847 INFO:tasks.workunit.client.1.smithi066.stdout:0/544: dread d2/d27/d41/d76/d93/f20 [0,4194304] 0 2024-02-28T16:31:20.848 INFO:tasks.workunit.client.0.smithi001.stdout:2/756: dwrite d0/d2/d3/f43 [0,4194304] 0 2024-02-28T16:31:20.848 INFO:tasks.workunit.client.1.smithi066.stdout:2/443: truncate d0/d12/d2e/d7a/f41 951400 0 2024-02-28T16:31:20.853 INFO:tasks.workunit.client.1.smithi066.stdout:8/416: write d0/db/f80 [3390991,59788] 0 2024-02-28T16:31:20.854 INFO:tasks.workunit.client.0.smithi001.stdout:9/977: write d5/d27/db1/d5e/dd3/d109/f4a [824335,95219] 0 2024-02-28T16:31:20.855 INFO:tasks.workunit.client.0.smithi001.stdout:2/757: fsync d0/d2/d3/d36/ff2 0 2024-02-28T16:31:20.856 INFO:tasks.workunit.client.0.smithi001.stdout:0/782: rename d18/d22/d2b/d36/d46/df5/l74 to d18/d22/d57/d7e/l103 0 2024-02-28T16:31:20.861 INFO:tasks.workunit.client.0.smithi001.stdout:7/814: write d3/d17/d3d/d5d/fbd [1516143,114499] 0 2024-02-28T16:31:20.861 INFO:tasks.workunit.client.1.smithi066.stdout:4/552: getdents d1/d18/d55/d81 0 2024-02-28T16:31:20.863 INFO:tasks.workunit.client.0.smithi001.stdout:1/917: write d0/d9/d1a/d2b/d26/d27/d30/d88/fe4 [4225490,106758] 0 2024-02-28T16:31:20.864 INFO:tasks.workunit.client.1.smithi066.stdout:8/417: creat d0/db/d12/d1d/d25/d65/f84 x:0 0 0 2024-02-28T16:31:20.866 INFO:tasks.workunit.client.1.smithi066.stdout:5/427: dwrite d3/d1b/f2d [0,4194304] 0 2024-02-28T16:31:20.866 INFO:tasks.workunit.client.0.smithi001.stdout:7/815: write d3/d17/d3d/f72 [397893,112326] 0 2024-02-28T16:31:20.872 INFO:tasks.workunit.client.1.smithi066.stdout:5/428: fsync d3/d4/dc/dd/d2e/f40 0 2024-02-28T16:31:20.872 INFO:tasks.workunit.client.0.smithi001.stdout:2/758: creat d0/d2/dd9/ffd x:0 0 0 2024-02-28T16:31:20.873 INFO:tasks.workunit.client.1.smithi066.stdout:6/377: dwrite d2/d26/f5f [0,4194304] 0 2024-02-28T16:31:20.874 INFO:tasks.workunit.client.0.smithi001.stdout:7/816: write d3/d17/d52/d92/fa2 [1697242,87898] 0 2024-02-28T16:31:20.878 INFO:tasks.workunit.client.1.smithi066.stdout:5/429: dread - d3/d4/f83 zero size 2024-02-28T16:31:20.878 INFO:tasks.workunit.client.1.smithi066.stdout:1/437: dwrite d2/d27/d2b/d43/d46/f88 [0,4194304] 0 2024-02-28T16:31:20.881 INFO:tasks.workunit.client.0.smithi001.stdout:0/783: mknod d18/da9/dc9/dd7/c104 0 2024-02-28T16:31:20.883 INFO:tasks.workunit.client.1.smithi066.stdout:8/418: dread d0/d6/dc/d16/d1a/d63/d58/d5a/f64 [0,4194304] 0 2024-02-28T16:31:20.885 INFO:tasks.workunit.client.1.smithi066.stdout:5/430: readlink d3/d4/dc/dd/d35/l54 0 2024-02-28T16:31:20.886 INFO:tasks.workunit.client.0.smithi001.stdout:1/918: symlink d0/d9/d1a/d72/l140 0 2024-02-28T16:31:20.888 INFO:tasks.workunit.client.1.smithi066.stdout:0/545: dwrite d2/d27/d41/d76/d93/db2/db5/d73/fa8 [0,4194304] 0 2024-02-28T16:31:20.888 INFO:tasks.workunit.client.0.smithi001.stdout:5/769: link dc/d12/l67 dc/d12/d41/d54/lfb 0 2024-02-28T16:31:20.893 INFO:tasks.workunit.client.1.smithi066.stdout:7/370: truncate dc/de/d17/f19 4064300 0 2024-02-28T16:31:20.898 INFO:tasks.workunit.client.0.smithi001.stdout:3/802: link d4/d6/f1c d4/d1e/d7c/f106 0 2024-02-28T16:31:20.899 INFO:tasks.workunit.client.1.smithi066.stdout:6/378: rename d2/d2a/c33 to d2/d2c/d82/c83 0 2024-02-28T16:31:20.902 INFO:tasks.workunit.client.1.smithi066.stdout:1/438: dwrite d2/d27/d2b/d43/d46/f88 [0,4194304] 0 2024-02-28T16:31:20.904 INFO:tasks.workunit.client.1.smithi066.stdout:0/546: stat d2/d27/d41/d76/d93/d33/l4c 0 2024-02-28T16:31:20.905 INFO:tasks.workunit.client.1.smithi066.stdout:9/353: sync 2024-02-28T16:31:20.908 INFO:tasks.workunit.client.0.smithi001.stdout:7/817: write d3/d17/d3d/d42/f53 [1765956,49639] 0 2024-02-28T16:31:20.909 INFO:tasks.workunit.client.1.smithi066.stdout:0/547: chown d2/c16 55 1 2024-02-28T16:31:20.911 INFO:tasks.workunit.client.1.smithi066.stdout:6/379: chown d2/d26/d32/d46/d80/d5d/c78 0 1 2024-02-28T16:31:20.913 INFO:tasks.workunit.client.1.smithi066.stdout:8/419: rename d0/d8/c56 to d0/d6/dc/d16/d4c/d42/d73/c85 0 2024-02-28T16:31:20.915 INFO:tasks.workunit.client.0.smithi001.stdout:1/919: getdents d0/d9/d1a/d2b/d26/d27/d30/d67 0 2024-02-28T16:31:20.918 INFO:tasks.workunit.client.1.smithi066.stdout:5/431: dread d3/d4/dc/dd/d2e/d18/f2c [0,4194304] 0 2024-02-28T16:31:20.920 INFO:tasks.workunit.client.1.smithi066.stdout:2/444: getdents d0/d12/d2e 0 2024-02-28T16:31:20.921 INFO:tasks.workunit.client.0.smithi001.stdout:0/784: dwrite d18/d22/d2f/d1c/f21 [0,4194304] 0 2024-02-28T16:31:20.921 INFO:tasks.workunit.client.1.smithi066.stdout:6/380: creat d2/d26/d32/d46/d80/d5d/d63/f84 x:0 0 0 2024-02-28T16:31:20.922 INFO:tasks.workunit.client.1.smithi066.stdout:8/420: chown d0/d6/dc/d16/d1a/d63/d3d/f62 0 1 2024-02-28T16:31:20.927 INFO:tasks.workunit.client.0.smithi001.stdout:4/905: dwrite d3/d5/d92/fc9 [4194304,4194304] 0 2024-02-28T16:31:20.929 INFO:tasks.workunit.client.1.smithi066.stdout:0/548: rename d2/d27/d41/d76/d93/db2/db5/d52/f69 to d2/d27/d41/d76/d93/db2/db5/d67/d94/fb8 0 2024-02-28T16:31:20.930 INFO:tasks.workunit.client.1.smithi066.stdout:2/445: symlink d0/d12/d2e/d3b/d66/l8a 0 2024-02-28T16:31:20.933 INFO:tasks.workunit.client.1.smithi066.stdout:5/432: mknod d3/d4/dc/c87 0 2024-02-28T16:31:20.936 INFO:tasks.workunit.client.1.smithi066.stdout:2/446: write d0/f37 [7955472,62214] 0 2024-02-28T16:31:20.938 INFO:tasks.workunit.client.0.smithi001.stdout:5/770: dwrite dc/df/f81 [0,4194304] 0 2024-02-28T16:31:20.947 INFO:tasks.workunit.client.0.smithi001.stdout:1/920: creat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/f141 x:0 0 0 2024-02-28T16:31:20.947 INFO:tasks.workunit.client.0.smithi001.stdout:3/803: rename d4/d1e/dcf/cf4 to d4/d6/d7/d78/c107 0 2024-02-28T16:31:20.948 INFO:tasks.workunit.client.0.smithi001.stdout:7/818: dwrite d3/d17/d3d/f72 [0,4194304] 0 2024-02-28T16:31:20.948 INFO:tasks.workunit.client.1.smithi066.stdout:6/381: link d2/d26/f31 d2/da/d38/f85 0 2024-02-28T16:31:20.950 INFO:tasks.workunit.client.1.smithi066.stdout:6/382: chown d2/d26/d32/d46 1478 1 2024-02-28T16:31:20.955 INFO:tasks.workunit.client.0.smithi001.stdout:7/819: stat d3/d17/d3d/fcd 0 2024-02-28T16:31:20.955 INFO:tasks.workunit.client.1.smithi066.stdout:2/447: symlink d0/d12/d1a/d23/d29/d21/l8b 0 2024-02-28T16:31:20.955 INFO:tasks.workunit.client.1.smithi066.stdout:8/421: truncate d0/d6/dc/d16/d1a/d63/d58/d5a/f64 1098654 0 2024-02-28T16:31:20.957 INFO:tasks.workunit.client.1.smithi066.stdout:9/354: write da/df/d18/d28/f2b [2330815,53322] 0 2024-02-28T16:31:20.958 INFO:tasks.workunit.client.0.smithi001.stdout:7/820: chown d3/d17/d52/lc6 32721344 1 2024-02-28T16:31:20.958 INFO:tasks.workunit.client.1.smithi066.stdout:2/448: chown d0/d12/d1a/d23/d29 580805 1 2024-02-28T16:31:20.958 INFO:tasks.workunit.client.1.smithi066.stdout:6/383: creat d2/d2c/f86 x:0 0 0 2024-02-28T16:31:20.959 INFO:tasks.workunit.client.1.smithi066.stdout:9/355: dread - da/df/d18/f78 zero size 2024-02-28T16:31:20.960 INFO:tasks.workunit.client.1.smithi066.stdout:0/549: rmdir d2/d27/d41/d76/d93/d10/d23/d6d/dad 0 2024-02-28T16:31:20.960 INFO:tasks.workunit.client.1.smithi066.stdout:1/439: write d2/d27/d2b/d43/d46/f61 [3282860,88850] 0 2024-02-28T16:31:20.962 INFO:tasks.workunit.client.1.smithi066.stdout:8/422: creat d0/db/d12/d1d/d25/d4f/f86 x:0 0 0 2024-02-28T16:31:20.964 INFO:tasks.workunit.client.1.smithi066.stdout:6/384: read - d2/da/d19/f50 zero size 2024-02-28T16:31:20.964 INFO:tasks.workunit.client.0.smithi001.stdout:1/921: creat d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f142 x:0 0 0 2024-02-28T16:31:20.970 INFO:tasks.workunit.client.1.smithi066.stdout:2/449: symlink d0/d12/d1a/d39/d68/l8c 0 2024-02-28T16:31:20.971 INFO:tasks.workunit.client.0.smithi001.stdout:3/804: link d4/d1e/d7c/fd3 d4/d6/d7/d78/d81/dd6/dd9/f108 0 2024-02-28T16:31:20.974 INFO:tasks.workunit.client.1.smithi066.stdout:9/356: write da/df/d18/d28/d63/f5c [1070497,100592] 0 2024-02-28T16:31:20.974 INFO:tasks.workunit.client.1.smithi066.stdout:8/423: dread d0/d6/dc/d16/d4c/d42/d73/f7f [0,4194304] 0 2024-02-28T16:31:20.974 INFO:tasks.workunit.client.1.smithi066.stdout:2/450: fsync d0/d12/d2e/f59 0 2024-02-28T16:31:20.975 INFO:tasks.workunit.client.1.smithi066.stdout:1/440: read d2/d27/d2b/d34/f55 [2074809,115617] 0 2024-02-28T16:31:20.975 INFO:tasks.workunit.client.1.smithi066.stdout:0/550: mknod d2/d27/d41/d76/d93/db2/cb9 0 2024-02-28T16:31:20.975 INFO:tasks.workunit.client.1.smithi066.stdout:2/451: read d0/f37 [7625718,115108] 0 2024-02-28T16:31:20.975 INFO:tasks.workunit.client.1.smithi066.stdout:9/357: chown da/df/c5d 6754 1 2024-02-28T16:31:20.976 INFO:tasks.workunit.client.1.smithi066.stdout:0/551: readlink l1 0 2024-02-28T16:31:20.976 INFO:tasks.workunit.client.0.smithi001.stdout:0/785: dwrite d18/d22/d76/d93/f94 [0,4194304] 0 2024-02-28T16:31:20.979 INFO:tasks.workunit.client.0.smithi001.stdout:0/786: fdatasync d18/d5a/f63 0 2024-02-28T16:31:20.981 INFO:tasks.workunit.client.0.smithi001.stdout:5/771: getdents dc/d12/daa 0 2024-02-28T16:31:20.982 INFO:tasks.workunit.client.1.smithi066.stdout:8/424: rename d0/d6/f7 to d0/db/d12/f87 0 2024-02-28T16:31:20.982 INFO:tasks.workunit.client.1.smithi066.stdout:2/452: chown d0/d12/d1a/d23/d29/d21/d2f/c69 1671 1 2024-02-28T16:31:20.993 INFO:tasks.workunit.client.0.smithi001.stdout:4/906: dwrite d3/d1c/d58/d6c/d79/dc8/fd3 [0,4194304] 0 2024-02-28T16:31:20.993 INFO:tasks.workunit.client.0.smithi001.stdout:7/821: dwrite d3/d17/d52/d7f/d99/dc1/fe9 [0,4194304] 0 2024-02-28T16:31:20.993 INFO:tasks.workunit.client.1.smithi066.stdout:1/441: fsync d2/d18/d19/f1e 0 2024-02-28T16:31:20.996 INFO:tasks.workunit.client.1.smithi066.stdout:2/453: mkdir d0/d12/d1a/d23/d29/d21/d2f/d67/d8d 0 2024-02-28T16:31:21.001 INFO:tasks.workunit.client.1.smithi066.stdout:6/385: dwrite d2/d26/f5f [0,4194304] 0 2024-02-28T16:31:21.001 INFO:tasks.workunit.client.0.smithi001.stdout:5/772: chown dc/d12/d56/d96/db9/dc2/dca/d3c/fee 2760098 1 2024-02-28T16:31:21.001 INFO:tasks.workunit.client.0.smithi001.stdout:7/822: chown d3/d17/d52/d7f/d99/faa 1 1 2024-02-28T16:31:21.005 INFO:tasks.workunit.client.1.smithi066.stdout:2/454: fsync d0/d12/f47 0 2024-02-28T16:31:21.007 INFO:tasks.workunit.client.1.smithi066.stdout:5/433: dwrite d3/d4/dc/dd/d2e/d18/f2c [0,4194304] 0 2024-02-28T16:31:21.007 INFO:tasks.workunit.client.0.smithi001.stdout:1/922: link d0/d9/d1a/d2b/d26/d27/d30/d4a/c69 d0/d9/d1a/d2b/d26/d27/d112/c143 0 2024-02-28T16:31:21.007 INFO:tasks.workunit.client.0.smithi001.stdout:4/907: chown d3/d2d/d4c/d69/d82/c67 7555490 1 2024-02-28T16:31:21.008 INFO:tasks.workunit.client.0.smithi001.stdout:7/823: chown d3/d17/d52/d55/dca/fce 26 1 2024-02-28T16:31:21.010 INFO:tasks.workunit.client.1.smithi066.stdout:1/442: dread - d2/d27/d2b/d43/d46/d80/f87 zero size 2024-02-28T16:31:21.011 INFO:tasks.workunit.client.1.smithi066.stdout:1/443: stat d2/d27/f2f 0 2024-02-28T16:31:21.013 INFO:tasks.workunit.client.0.smithi001.stdout:2/759: dread d0/d2/d3/de/d61/d12/f41 [0,4194304] 0 2024-02-28T16:31:21.014 INFO:tasks.workunit.client.1.smithi066.stdout:7/371: dwrite dc/de/d17/f19 [0,4194304] 0 2024-02-28T16:31:21.017 INFO:tasks.workunit.client.1.smithi066.stdout:6/386: dread d2/da/f1c [0,4194304] 0 2024-02-28T16:31:21.017 INFO:tasks.workunit.client.0.smithi001.stdout:1/923: chown d0/d9/d1a/d2b/c13 1621618 1 2024-02-28T16:31:21.018 INFO:tasks.workunit.client.0.smithi001.stdout:5/773: link dc/d12/d41/d54/l7a dc/d1a/d7d/d87/df3/lfc 0 2024-02-28T16:31:21.020 INFO:tasks.workunit.client.0.smithi001.stdout:8/792: sync 2024-02-28T16:31:21.022 INFO:tasks.workunit.client.1.smithi066.stdout:9/358: dwrite da/d22/d46/d56/f72 [0,4194304] 0 2024-02-28T16:31:21.025 INFO:tasks.workunit.client.1.smithi066.stdout:6/387: chown d2/d2c/d72 69315 1 2024-02-28T16:31:21.027 INFO:tasks.workunit.client.0.smithi001.stdout:1/924: dread - d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/f141 zero size 2024-02-28T16:31:21.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.042+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7641027f0 con 0x7fe74806b1d0 2024-02-28T16:31:21.032 INFO:tasks.workunit.client.1.smithi066.stdout:6/388: chown d2/da/d19/l4b 47918712 1 2024-02-28T16:31:21.032 INFO:tasks.workunit.client.0.smithi001.stdout:1/925: chown d0/d9/d1a/d72/l96 50863 1 2024-02-28T16:31:21.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.045+0000 7fe75be00700 1 -- 172.21.15.1:0/1088351272 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fe7641027f0 con 0x7fe74806b1d0 2024-02-28T16:31:21.034 INFO:tasks.workunit.client.1.smithi066.stdout:9/359: write da/d22/d46/d56/f72 [3265807,91490] 0 2024-02-28T16:31:21.037 INFO:tasks.workunit.client.1.smithi066.stdout:7/372: creat dc/de/d17/d37/d47/f6b x:0 0 0 2024-02-28T16:31:21.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.049+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fe74806b1d0 msgr2=0x7fe74806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.049+0000 7fe76ae00700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fe74806b1d0 0x7fe74806d680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe750004990 tx=0x7fe750009ee0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.049+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 msgr2=0x7fe76410d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.049+0000 7fe76ae00700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 0x7fe76410d720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe7540043d0 tx=0x7fe754009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.050+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 shutdown_connections 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.050+0000 7fe76ae00700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7fe74806b1d0 0x7fe74806d680 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.050+0000 7fe76ae00700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7640fcef0 0x7fe76410d720 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.050+0000 7fe76ae00700 1 --2- 172.21.15.1:0/1088351272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7640fe0e0 0x7fe76410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.050+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 >> 172.21.15.1:0/1088351272 conn(0x7fe7640f8620 msgr2=0x7fe7640fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.050+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 shutdown_connections 2024-02-28T16:31:21.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.050+0000 7fe76ae00700 1 -- 172.21.15.1:0/1088351272 wait complete. 2024-02-28T16:31:21.040 INFO:tasks.workunit.client.0.smithi001.stdout:1/926: read d0/d9/d1a/d2b/d26/d27/d30/d88/fef [134798,19729] 0 2024-02-28T16:31:21.042 INFO:tasks.workunit.client.0.smithi001.stdout:2/760: link d0/d2/d3/de/d61/c25 d0/d2/d3/de/d61/d12/d46/cfe 0 2024-02-28T16:31:21.047 INFO:tasks.workunit.client.1.smithi066.stdout:6/389: rename d2/d26/d32/d46/l5c to d2/d26/d32/l87 0 2024-02-28T16:31:21.048 INFO:tasks.workunit.client.1.smithi066.stdout:0/552: truncate d2/d27/d41/d76/d93/db2/db5/d52/f81 3397858 0 2024-02-28T16:31:21.049 INFO:tasks.workunit.client.1.smithi066.stdout:2/455: dwrite d0/d12/d3e/f4e [0,4194304] 0 2024-02-28T16:31:21.053 INFO:tasks.workunit.client.1.smithi066.stdout:0/553: write d2/d27/d41/d76/d93/f1e [4044590,22979] 0 2024-02-28T16:31:21.053 INFO:tasks.workunit.client.0.smithi001.stdout:3/805: dwrite d4/d1e/d7c/d7d/df8/ff0 [0,4194304] 0 2024-02-28T16:31:21.055 INFO:tasks.workunit.client.1.smithi066.stdout:5/434: dread d3/d4/dc/dd/d2e/f40 [0,4194304] 0 2024-02-28T16:31:21.058 INFO:tasks.workunit.client.0.smithi001.stdout:8/793: dwrite d4/d7/d22/d27/d58/d94/fb5 [0,4194304] 0 2024-02-28T16:31:21.065 INFO:tasks.workunit.client.1.smithi066.stdout:2/456: chown d0/d12/d3e/l53 344 1 2024-02-28T16:31:21.069 INFO:tasks.workunit.client.1.smithi066.stdout:5/435: fdatasync d3/d4/dc/dd/d2e/d18/f2c 0 2024-02-28T16:31:21.070 INFO:tasks.workunit.client.1.smithi066.stdout:9/360: rename da/df/d18/c3c to da/df/d18/d28/d42/c79 0 2024-02-28T16:31:21.071 INFO:tasks.workunit.client.0.smithi001.stdout:5/774: write dc/d1a/d7d/d87/fa8 [847318,108703] 0 2024-02-28T16:31:21.071 INFO:tasks.workunit.client.0.smithi001.stdout:0/787: dwrite d18/d22/d2b/d36/d48/f55 [0,4194304] 0 2024-02-28T16:31:21.073 INFO:tasks.workunit.client.0.smithi001.stdout:8/794: creat d4/d32/d69/d8f/f101 x:0 0 0 2024-02-28T16:31:21.074 INFO:tasks.workunit.client.1.smithi066.stdout:7/373: creat dc/de/d17/d1e/f6c x:0 0 0 2024-02-28T16:31:21.075 INFO:tasks.workunit.client.0.smithi001.stdout:3/806: link d4/f76 d4/d18/f109 0 2024-02-28T16:31:21.077 INFO:tasks.workunit.client.1.smithi066.stdout:0/554: getdents d2/d85/da9 0 2024-02-28T16:31:21.079 INFO:tasks.workunit.client.0.smithi001.stdout:9/978: sync 2024-02-28T16:31:21.079 INFO:tasks.workunit.client.0.smithi001.stdout:3/807: dread - d4/d1e/def/fe3 zero size 2024-02-28T16:31:21.079 INFO:tasks.workunit.client.0.smithi001.stdout:5/775: creat dc/d12/d41/dd6/ffd x:0 0 0 2024-02-28T16:31:21.079 INFO:tasks.workunit.client.0.smithi001.stdout:6/778: sync 2024-02-28T16:31:21.080 INFO:tasks.workunit.client.1.smithi066.stdout:5/436: creat d3/d4/dc/dd/d35/d86/f88 x:0 0 0 2024-02-28T16:31:21.083 INFO:tasks.workunit.client.0.smithi001.stdout:8/795: chown d4/d7/d22/d27/d4b/d77/f7a 1855929 1 2024-02-28T16:31:21.085 INFO:tasks.workunit.client.1.smithi066.stdout:9/361: creat da/df/d18/d28/d42/d68/f7a x:0 0 0 2024-02-28T16:31:21.090 INFO:tasks.workunit.client.0.smithi001.stdout:6/779: write d9/d29/f44 [530948,93711] 0 2024-02-28T16:31:21.090 INFO:tasks.workunit.client.0.smithi001.stdout:4/908: dread d3/d5/d7/d1f/d3a/fa4 [0,4194304] 0 2024-02-28T16:31:21.090 INFO:tasks.workunit.client.0.smithi001.stdout:1/927: dread d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f137 [0,4194304] 0 2024-02-28T16:31:21.091 INFO:tasks.workunit.client.0.smithi001.stdout:5/776: fsync dc/d12/daa/ff1 0 2024-02-28T16:31:21.091 INFO:tasks.workunit.client.1.smithi066.stdout:0/555: dread d2/d27/d41/d76/d93/f6f [0,4194304] 0 2024-02-28T16:31:21.092 INFO:tasks.workunit.client.0.smithi001.stdout:9/979: creat d5/d27/db1/d33/d115/f137 x:0 0 0 2024-02-28T16:31:21.094 INFO:tasks.workunit.client.1.smithi066.stdout:2/457: dread d0/d12/d2e/d3b/f64 [4194304,4194304] 0 2024-02-28T16:31:21.097 INFO:tasks.workunit.client.0.smithi001.stdout:3/808: creat d4/d1e/d7c/d7d/d80/f10a x:0 0 0 2024-02-28T16:31:21.098 INFO:tasks.workunit.client.0.smithi001.stdout:0/788: truncate d18/d22/d76/d7a/fae 1703515 0 2024-02-28T16:31:21.099 INFO:tasks.workunit.client.1.smithi066.stdout:5/437: dread d3/d4/fb [4194304,4194304] 0 2024-02-28T16:31:21.099 INFO:tasks.workunit.client.1.smithi066.stdout:9/362: rename da/df/l25 to da/d3a/d15/d48/d76/l7b 0 2024-02-28T16:31:21.100 INFO:tasks.workunit.client.1.smithi066.stdout:7/374: dread dc/f5d [0,4194304] 0 2024-02-28T16:31:21.101 INFO:tasks.workunit.client.0.smithi001.stdout:8/796: write d4/d7/d22/d27/f3a [4538719,101372] 0 2024-02-28T16:31:21.103 INFO:tasks.workunit.client.1.smithi066.stdout:7/375: write dc/de/f65 [416380,90729] 0 2024-02-28T16:31:21.103 INFO:tasks.workunit.client.0.smithi001.stdout:4/909: rmdir d3/d5/d7/d1f/d20 39 2024-02-28T16:31:21.106 INFO:tasks.workunit.client.0.smithi001.stdout:0/789: write d18/d22/d2f/ff1 [243049,28215] 0 2024-02-28T16:31:21.106 INFO:tasks.workunit.client.1.smithi066.stdout:1/444: truncate d2/d27/d2b/d43/d46/f61 2143079 0 2024-02-28T16:31:21.107 INFO:tasks.workunit.client.1.smithi066.stdout:2/458: symlink d0/d12/d1a/d23/d29/l8e 0 2024-02-28T16:31:21.107 INFO:tasks.workunit.client.0.smithi001.stdout:6/780: creat d9/db/d5f/d89/d8a/f104 x:0 0 0 2024-02-28T16:31:21.108 INFO:tasks.workunit.client.0.smithi001.stdout:3/809: read d4/d6/d7/f88 [1516836,7099] 0 2024-02-28T16:31:21.108 INFO:tasks.workunit.client.0.smithi001.stdout:5/777: unlink dc/d12/d56/d96/db8/dd7/cda 0 2024-02-28T16:31:21.108 INFO:tasks.workunit.client.1.smithi066.stdout:3/457: sync 2024-02-28T16:31:21.109 INFO:tasks.workunit.client.0.smithi001.stdout:9/980: symlink d5/d27/db1/d5e/dd3/d109/df1/l138 0 2024-02-28T16:31:21.111 INFO:tasks.workunit.client.1.smithi066.stdout:5/438: creat d3/d4/dc/dd/f89 x:0 0 0 2024-02-28T16:31:21.111 INFO:tasks.workunit.client.1.smithi066.stdout:1/445: write d2/d18/d6b/f77 [95728,96330] 0 2024-02-28T16:31:21.115 INFO:tasks.workunit.client.0.smithi001.stdout:1/928: mknod d0/d9/d1a/d2b/d26/d27/d30/c144 0 2024-02-28T16:31:21.116 INFO:tasks.workunit.client.0.smithi001.stdout:9/981: mkdir d5/d27/db1/d58/d139 0 2024-02-28T16:31:21.116 INFO:tasks.workunit.client.0.smithi001.stdout:8/797: write d4/d32/dd6/fd2 [997362,23537] 0 2024-02-28T16:31:21.118 INFO:tasks.workunit.client.1.smithi066.stdout:9/363: mkdir da/d4b/d7c 0 2024-02-28T16:31:21.118 INFO:tasks.workunit.client.0.smithi001.stdout:4/910: creat d3/d2d/d4c/d69/f128 x:0 0 0 2024-02-28T16:31:21.118 INFO:tasks.workunit.client.0.smithi001.stdout:7/824: dread d3/d17/d3d/d59/fb1 [0,4194304] 0 2024-02-28T16:31:21.121 INFO:tasks.workunit.client.1.smithi066.stdout:7/376: mknod dc/de/d17/d37/d47/c6d 0 2024-02-28T16:31:21.121 INFO:tasks.workunit.client.1.smithi066.stdout:1/446: write d2/d27/d2b/d37/d51/f62 [349185,83189] 0 2024-02-28T16:31:21.123 INFO:tasks.workunit.client.1.smithi066.stdout:9/364: write da/d3a/d15/f4f [246991,218] 0 2024-02-28T16:31:21.125 INFO:tasks.workunit.client.1.smithi066.stdout:5/439: dread d3/d4/dc/d2f/d55/f70 [0,4194304] 0 2024-02-28T16:31:21.125 INFO:tasks.workunit.client.0.smithi001.stdout:6/781: mkdir d9/d29/d6a/ded/d105 0 2024-02-28T16:31:21.127 INFO:tasks.workunit.client.1.smithi066.stdout:6/390: dwrite d2/d26/f31 [0,4194304] 0 2024-02-28T16:31:21.129 INFO:tasks.workunit.client.0.smithi001.stdout:1/929: mknod d0/d9/d1a/d2b/df8/c145 0 2024-02-28T16:31:21.131 INFO:tasks.workunit.client.1.smithi066.stdout:3/458: creat db/d13/d14/d84/f9c x:0 0 0 2024-02-28T16:31:21.131 INFO:tasks.workunit.client.0.smithi001.stdout:8/798: write d4/d7/d22/d27/d58/d94/dbc/fbd [5264601,19884] 0 2024-02-28T16:31:21.133 INFO:tasks.workunit.client.1.smithi066.stdout:2/459: rename d0/d12/d1a/d23/d29/d21/d2f/l85 to d0/d12/d3e/l8f 0 2024-02-28T16:31:21.134 INFO:tasks.workunit.client.0.smithi001.stdout:2/761: dread d0/d2/d3/de/d61/d12/fe0 [0,4194304] 0 2024-02-28T16:31:21.135 INFO:tasks.workunit.client.1.smithi066.stdout:1/447: creat d2/d18/d7a/f8b x:0 0 0 2024-02-28T16:31:21.136 INFO:tasks.workunit.client.0.smithi001.stdout:7/825: mknod d3/d17/d52/d7f/d99/c108 0 2024-02-28T16:31:21.138 INFO:tasks.workunit.client.0.smithi001.stdout:0/790: dwrite d18/d22/d3d/f50 [0,4194304] 0 2024-02-28T16:31:21.140 INFO:tasks.workunit.client.1.smithi066.stdout:7/377: mkdir dc/d6e 0 2024-02-28T16:31:21.145 INFO:tasks.workunit.client.0.smithi001.stdout:7/826: fdatasync d3/d17/d52/d7f/d99/fd5 0 2024-02-28T16:31:21.146 INFO:tasks.workunit.client.0.smithi001.stdout:8/799: truncate d4/f43 2145655 0 2024-02-28T16:31:21.146 INFO:tasks.workunit.client.1.smithi066.stdout:5/440: dwrite d3/d1b/f2d [0,4194304] 0 2024-02-28T16:31:21.151 INFO:tasks.workunit.client.1.smithi066.stdout:0/556: truncate d2/d27/d41/d76/d93/db2/db5/f57 636984 0 2024-02-28T16:31:21.153 INFO:tasks.workunit.client.1.smithi066.stdout:6/391: mkdir d2/da/d19/d75/d88 0 2024-02-28T16:31:21.154 INFO:tasks.workunit.client.0.smithi001.stdout:6/782: mkdir d9/d29/d6a/ded/d105/d106 0 2024-02-28T16:31:21.156 INFO:tasks.workunit.client.1.smithi066.stdout:3/459: symlink db/dc/d11/d1f/l9d 0 2024-02-28T16:31:21.159 INFO:tasks.workunit.client.1.smithi066.stdout:2/460: truncate d0/f9 1804898 0 2024-02-28T16:31:21.159 INFO:tasks.workunit.client.1.smithi066.stdout:3/460: chown db/d13/d14/l8c 24614829 1 2024-02-28T16:31:21.166 INFO:tasks.workunit.client.1.smithi066.stdout:3/461: dread - db/d2a/d44/d48/d52/d77/f74 zero size 2024-02-28T16:31:21.167 INFO:tasks.workunit.client.1.smithi066.stdout:2/461: unlink d0/d12/d1a/c2d 0 2024-02-28T16:31:21.167 INFO:tasks.workunit.client.0.smithi001.stdout:7/827: symlink d3/d17/d52/d92/l109 0 2024-02-28T16:31:21.168 INFO:tasks.workunit.client.0.smithi001.stdout:3/810: write d4/d6/d7/f43 [2584181,106717] 0 2024-02-28T16:31:21.170 INFO:tasks.workunit.client.0.smithi001.stdout:2/762: truncate d0/d2/d3/de/d8a/f6e 3747026 0 2024-02-28T16:31:21.175 INFO:tasks.workunit.client.0.smithi001.stdout:2/763: chown d0/d2/d3/de/l65 8389248 1 2024-02-28T16:31:21.176 INFO:tasks.workunit.client.1.smithi066.stdout:5/441: truncate d3/d4/dc/dd/d2e/d18/f2b 1108691 0 2024-02-28T16:31:21.180 INFO:tasks.workunit.client.0.smithi001.stdout:9/982: getdents d5/d27/db1/d5e/dd3/d109/df1 0 2024-02-28T16:31:21.182 INFO:tasks.workunit.client.1.smithi066.stdout:3/462: unlink db/d13/d14/d2d/l30 0 2024-02-28T16:31:21.184 INFO:tasks.workunit.client.0.smithi001.stdout:9/983: chown d5/d27/d42 219 1 2024-02-28T16:31:21.184 INFO:tasks.workunit.client.1.smithi066.stdout:5/442: read d3/d4/dc/dd/d41/d5d/f69 [2580633,59195] 0 2024-02-28T16:31:21.189 INFO:tasks.workunit.client.1.smithi066.stdout:7/378: dread dc/de/d17/d1e/f2d [0,4194304] 0 2024-02-28T16:31:21.190 INFO:tasks.workunit.client.0.smithi001.stdout:5/778: dwrite dc/d12/d56/d96/db9/dc2/dca/d91/fc7 [0,4194304] 0 2024-02-28T16:31:21.192 INFO:tasks.workunit.client.1.smithi066.stdout:3/463: getdents db/d7a/d7c 0 2024-02-28T16:31:21.193 INFO:tasks.workunit.client.0.smithi001.stdout:1/930: dread d0/d9/d1a/d2b/d26/d27/d30/d4a/f89 [0,4194304] 0 2024-02-28T16:31:21.193 INFO:tasks.workunit.client.1.smithi066.stdout:7/379: chown dc/de/l18 1923668 1 2024-02-28T16:31:21.195 INFO:tasks.workunit.client.1.smithi066.stdout:2/462: dwrite d0/d12/d2e/d7a/f5b [4194304,4194304] 0 2024-02-28T16:31:21.196 INFO:tasks.workunit.client.0.smithi001.stdout:5/779: write dc/d12/d41/d54/fa5 [2030088,61799] 0 2024-02-28T16:31:21.196 INFO:tasks.workunit.client.0.smithi001.stdout:3/811: symlink d4/d18/da0/l10b 0 2024-02-28T16:31:21.199 INFO:tasks.workunit.client.0.smithi001.stdout:9/984: creat d5/d27/db1/d33/d115/d11f/de0/f13a x:0 0 0 2024-02-28T16:31:21.200 INFO:tasks.workunit.client.1.smithi066.stdout:7/380: rmdir dc/d56 39 2024-02-28T16:31:21.202 INFO:tasks.workunit.client.1.smithi066.stdout:0/557: dread d2/d27/d41/d76/d93/db2/db5/d67/d94/fb8 [0,4194304] 0 2024-02-28T16:31:21.202 INFO:tasks.workunit.client.0.smithi001.stdout:0/791: write d18/f59 [2614212,60652] 0 2024-02-28T16:31:21.206 INFO:tasks.workunit.client.1.smithi066.stdout:6/392: write d2/d2a/d3d/f7f [632179,104652] 0 2024-02-28T16:31:21.208 INFO:tasks.workunit.client.0.smithi001.stdout:4/911: dwrite d3/d1c/d58/fdf [0,4194304] 0 2024-02-28T16:31:21.208 INFO:tasks.workunit.client.0.smithi001.stdout:5/780: rmdir dc/d12/d41/d54 39 2024-02-28T16:31:21.209 INFO:tasks.workunit.client.1.smithi066.stdout:6/393: stat d2/d2c/f30 0 2024-02-28T16:31:21.212 INFO:tasks.workunit.client.1.smithi066.stdout:7/381: rename dc/de/d17/d37/d47/f6b to dc/d6e/f6f 0 2024-02-28T16:31:21.213 INFO:tasks.workunit.client.1.smithi066.stdout:4/553: sync 2024-02-28T16:31:21.214 INFO:tasks.workunit.client.0.smithi001.stdout:6/783: getdents d9/db/d5f/d8b 0 2024-02-28T16:31:21.214 INFO:tasks.workunit.client.1.smithi066.stdout:0/558: symlink d2/d27/d41/d76/d93/db2/db5/d52/da2/lba 0 2024-02-28T16:31:21.214 INFO:tasks.workunit.client.0.smithi001.stdout:9/985: symlink d5/d27/d42/def/l13b 0 2024-02-28T16:31:21.219 INFO:tasks.workunit.client.1.smithi066.stdout:1/448: dwrite d2/f1d [4194304,4194304] 0 2024-02-28T16:31:21.219 INFO:tasks.workunit.client.0.smithi001.stdout:9/986: chown d5/d27/db1/d5e/dd3/d109/f4a 46 1 2024-02-28T16:31:21.224 INFO:tasks.workunit.client.1.smithi066.stdout:7/382: write dc/d29/f44 [2534972,48237] 0 2024-02-28T16:31:21.224 INFO:tasks.workunit.client.1.smithi066.stdout:4/554: read d1/d18/f37 [199292,19217] 0 2024-02-28T16:31:21.225 INFO:tasks.workunit.client.0.smithi001.stdout:8/800: link d4/d7/d22/d27/d58/d62/fb2 d4/d32/d69/d9a/f102 0 2024-02-28T16:31:21.228 INFO:tasks.workunit.client.1.smithi066.stdout:0/559: creat d2/d27/d41/d76/d93/d33/d3f/fbb x:0 0 0 2024-02-28T16:31:21.232 INFO:tasks.workunit.client.1.smithi066.stdout:7/383: symlink dc/de/d17/d1e/l70 0 2024-02-28T16:31:21.233 INFO:tasks.workunit.client.1.smithi066.stdout:7/384: stat dc 0 2024-02-28T16:31:21.236 INFO:tasks.workunit.client.1.smithi066.stdout:2/463: dwrite d0/d12/d2e/d3b/f64 [8388608,4194304] 0 2024-02-28T16:31:21.240 INFO:tasks.workunit.client.1.smithi066.stdout:4/555: mkdir d1/d18/d41/da7/dae 0 2024-02-28T16:31:21.240 INFO:tasks.workunit.client.0.smithi001.stdout:9/987: read d5/d27/db1/d5e/dd3/d109/d6f/f10d [197006,112646] 0 2024-02-28T16:31:21.240 INFO:tasks.workunit.client.1.smithi066.stdout:0/560: symlink d2/d27/d41/d76/d93/d33/d3f/d8e/lbc 0 2024-02-28T16:31:21.242 INFO:tasks.workunit.client.1.smithi066.stdout:7/385: mknod dc/de/d17/d37/d47/c71 0 2024-02-28T16:31:21.244 INFO:tasks.workunit.client.0.smithi001.stdout:4/912: unlink d3/c6f 0 2024-02-28T16:31:21.244 INFO:tasks.workunit.client.0.smithi001.stdout:2/764: write d0/f32 [4407655,61812] 0 2024-02-28T16:31:21.247 INFO:tasks.workunit.client.1.smithi066.stdout:2/464: link d0/d12/d1a/d39/l60 d0/d12/d1a/d23/l90 0 2024-02-28T16:31:21.248 INFO:tasks.workunit.client.0.smithi001.stdout:0/792: write d18/d22/d2f/d1c/fb9 [220223,5659] 0 2024-02-28T16:31:21.249 INFO:tasks.workunit.client.0.smithi001.stdout:8/801: chown d4/d7/d22/d27/d4b/d77/c9c 5864692 1 2024-02-28T16:31:21.253 INFO:tasks.workunit.client.0.smithi001.stdout:6/784: dwrite d9/db/d5f/d89/d8a/ff5 [0,4194304] 0 2024-02-28T16:31:21.254 INFO:tasks.workunit.client.1.smithi066.stdout:0/561: rename d2/d85/d8d/la6 to d2/d27/d41/d76/d93/d33/d3f/d8e/lbd 0 2024-02-28T16:31:21.254 INFO:tasks.workunit.client.1.smithi066.stdout:8/425: sync 2024-02-28T16:31:21.254 INFO:tasks.workunit.client.1.smithi066.stdout:4/556: link d1/d23/d25/c42 d1/d18/d4e/caf 0 2024-02-28T16:31:21.255 INFO:tasks.workunit.client.0.smithi001.stdout:0/793: chown d18/d22/d76/d93/lfa 1604 1 2024-02-28T16:31:21.255 INFO:tasks.workunit.client.1.smithi066.stdout:5/443: dread d3/d4/dc/dd/d2e/d18/f2b [0,4194304] 0 2024-02-28T16:31:21.256 INFO:tasks.workunit.client.0.smithi001.stdout:7/828: dread d3/d1c/f40 [0,4194304] 0 2024-02-28T16:31:21.259 INFO:tasks.workunit.client.0.smithi001.stdout:3/812: getdents d4/d6 0 2024-02-28T16:31:21.264 INFO:tasks.workunit.client.0.smithi001.stdout:2/765: rmdir d0/d2/d3/de/d22 39 2024-02-28T16:31:21.265 INFO:tasks.workunit.client.1.smithi066.stdout:1/449: dwrite d2/d18/d6b/f77 [0,4194304] 0 2024-02-28T16:31:21.265 INFO:tasks.workunit.client.1.smithi066.stdout:4/557: write d1/d18/d41/f49 [4823,79943] 0 2024-02-28T16:31:21.265 INFO:tasks.workunit.client.0.smithi001.stdout:4/913: creat d3/d5/d7/d1f/d2e/dbe/d115/f129 x:0 0 0 2024-02-28T16:31:21.265 INFO:tasks.workunit.client.0.smithi001.stdout:6/785: dread - d9/d29/d6e/d53/fac zero size 2024-02-28T16:31:21.268 INFO:tasks.workunit.client.0.smithi001.stdout:1/931: dread d0/d9/f5c [0,4194304] 0 2024-02-28T16:31:21.269 INFO:tasks.workunit.client.0.smithi001.stdout:5/781: getdents dc/d12/d56/d96/db9/dc2/dca/d91 0 2024-02-28T16:31:21.269 INFO:tasks.workunit.client.0.smithi001.stdout:3/813: write d4/d18/d5f/fb8 [148022,106615] 0 2024-02-28T16:31:21.270 INFO:tasks.workunit.client.1.smithi066.stdout:3/464: write db/ff [928617,48320] 0 2024-02-28T16:31:21.274 INFO:tasks.workunit.client.0.smithi001.stdout:7/829: unlink d3/d17/d3d/fa7 0 2024-02-28T16:31:21.274 INFO:tasks.workunit.client.1.smithi066.stdout:1/450: chown d2/d27/d2b/d43/c50 1639954730 1 2024-02-28T16:31:21.279 INFO:tasks.workunit.client.1.smithi066.stdout:1/451: chown d2/d27/d2b/d37/d51/f60 4028 1 2024-02-28T16:31:21.279 INFO:tasks.workunit.client.0.smithi001.stdout:4/914: chown d3/d1c/d58/d88/ca9 1786542 1 2024-02-28T16:31:21.281 INFO:tasks.workunit.client.1.smithi066.stdout:8/426: getdents d0/db/d12/d1d/d25/d72 0 2024-02-28T16:31:21.283 INFO:tasks.workunit.client.0.smithi001.stdout:0/794: unlink d18/d22/d2b/d36/d48/fc1 0 2024-02-28T16:31:21.283 INFO:tasks.workunit.client.1.smithi066.stdout:2/465: dread d0/d12/d2e/d3b/f64 [4194304,4194304] 0 2024-02-28T16:31:21.284 INFO:tasks.workunit.client.0.smithi001.stdout:1/932: chown d0/d9/d1a/d2b/d26/d27/d30/d4a/f89 1488454377 1 2024-02-28T16:31:21.285 INFO:tasks.workunit.client.1.smithi066.stdout:3/465: rename db/dc/d35/f60 to db/d13/d14/d2d/f9e 0 2024-02-28T16:31:21.286 INFO:tasks.workunit.client.1.smithi066.stdout:0/562: getdents d2/d27/d41/d76/d93/d10/d23 0 2024-02-28T16:31:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.300+0000 7f4f26a00700 1 -- 172.21.15.1:0/3144648077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 msgr2=0x7f4f20102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.300+0000 7f4f26a00700 1 --2- 172.21.15.1:0/3144648077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 0x7f4f20102780 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c004880 tx=0x7f4f0c009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.302+0000 7f4f26a00700 1 -- 172.21.15.1:0/3144648077 shutdown_connections 2024-02-28T16:31:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.302+0000 7f4f26a00700 1 --2- 172.21.15.1:0/3144648077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 0x7f4f20102780 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.302+0000 7f4f26a00700 1 --2- 172.21.15.1:0/3144648077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f200fda80 0x7f4f200ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.302+0000 7f4f26a00700 1 -- 172.21.15.1:0/3144648077 >> 172.21.15.1:0/3144648077 conn(0x7f4f200f74e0 msgr2=0x7f4f200f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.302+0000 7f4f26a00700 1 -- 172.21.15.1:0/3144648077 shutdown_connections 2024-02-28T16:31:21.292 INFO:tasks.workunit.client.0.smithi001.stdout:7/830: stat d3/d17/d52/f77 0 2024-02-28T16:31:21.292 INFO:tasks.workunit.client.0.smithi001.stdout:1/933: chown d0/d9/d1a/d2b 57306191 1 2024-02-28T16:31:21.292 INFO:tasks.workunit.client.0.smithi001.stdout:3/814: creat d4/d6/d7/d78/d81/da1/f10c x:0 0 0 2024-02-28T16:31:21.292 INFO:tasks.workunit.client.0.smithi001.stdout:6/786: truncate d9/db/d5f/d62/d83/fb6 580212 0 2024-02-28T16:31:21.293 INFO:tasks.workunit.client.1.smithi066.stdout:0/563: truncate d2/d27/fa3 96755 0 2024-02-28T16:31:21.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.302+0000 7f4f26a00700 1 -- 172.21.15.1:0/3144648077 wait complete. 2024-02-28T16:31:21.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.304+0000 7f4f26a00700 1 Processor -- start 2024-02-28T16:31:21.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.305+0000 7f4f26a00700 1 -- start start 2024-02-28T16:31:21.294 INFO:tasks.workunit.client.1.smithi066.stdout:3/466: readlink db/d2a/d44/d48/d5d/l7e 0 2024-02-28T16:31:21.295 INFO:tasks.workunit.client.1.smithi066.stdout:5/444: dwrite d3/d4/dc/d2f/d55/f70 [0,4194304] 0 2024-02-28T16:31:21.295 INFO:tasks.workunit.client.1.smithi066.stdout:8/427: creat d0/d6/dc/d16/d1a/f88 x:0 0 0 2024-02-28T16:31:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.306+0000 7f4f26a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f200fda80 0x7f4f2006ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.307+0000 7f4f26a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 0x7f4f2006f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.307+0000 7f4f1fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f200fda80 0x7f4f2006ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:21.296 INFO:tasks.workunit.client.0.smithi001.stdout:0/795: mkdir d18/d22/d105 0 2024-02-28T16:31:21.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.307+0000 7f4f1fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f200fda80 0x7f4f2006ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40834/0 (socket says 172.21.15.1:40834) 2024-02-28T16:31:21.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.307+0000 7f4f1fe00700 1 -- 172.21.15.1:0/443278372 learned_addr learned my addr 172.21.15.1:0/443278372 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:21.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.307+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2006faf0 con 0x7f4f200fda80 2024-02-28T16:31:21.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.308+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f201174e0 con 0x7f4f201003a0 2024-02-28T16:31:21.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.308+0000 7f4f1f400700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 0x7f4f2006f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.309+0000 7f4f1f400700 1 -- 172.21.15.1:0/443278372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f200fda80 msgr2=0x7f4f2006ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:21.297 INFO:tasks.workunit.client.1.smithi066.stdout:2/466: write d0/d12/d2e/d3b/f64 [11583610,30034] 0 2024-02-28T16:31:21.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.309+0000 7f4f1f400700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f200fda80 0x7f4f2006ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.309+0000 7f4f1f400700 1 -- 172.21.15.1:0/443278372 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c005040 con 0x7f4f201003a0 2024-02-28T16:31:21.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.309+0000 7f4f1f400700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 0x7f4f2006f470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c005010 tx=0x7f4f0c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:21.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.310+0000 7f4f17e00700 1 -- 172.21.15.1:0/443278372 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f0c00e040 con 0x7f4f201003a0 2024-02-28T16:31:21.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.310+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f20117710 con 0x7f4f201003a0 2024-02-28T16:31:21.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.310+0000 7f4f17e00700 1 -- 172.21.15.1:0/443278372 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f0c014c30 con 0x7f4f201003a0 2024-02-28T16:31:21.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.310+0000 7f4f17e00700 1 -- 172.21.15.1:0/443278372 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f0c012600 con 0x7f4f201003a0 2024-02-28T16:31:21.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.311+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f20117c50 con 0x7f4f201003a0 2024-02-28T16:31:21.299 INFO:tasks.workunit.client.1.smithi066.stdout:3/467: chown db/d2a/c5a 1022798803 1 2024-02-28T16:31:21.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.312+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2004bed0 con 0x7f4f201003a0 2024-02-28T16:31:21.301 INFO:tasks.workunit.client.1.smithi066.stdout:5/445: chown d3/d4/dc/dd/d41/d7f/d80 224590785 1 2024-02-28T16:31:21.303 INFO:tasks.workunit.client.1.smithi066.stdout:2/467: stat d0/d12/d2e/d7a/l5a 0 2024-02-28T16:31:21.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.316+0000 7f4f17e00700 1 -- 172.21.15.1:0/443278372 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 25) v1 ==== 88910+0+0 (secure 0 0 0) 0x7f4f0c01c3f0 con 0x7f4f201003a0 2024-02-28T16:31:21.306 INFO:tasks.workunit.client.0.smithi001.stdout:7/831: creat d3/d17/d3d/d42/f10a x:0 0 0 2024-02-28T16:31:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.319+0000 7f4f17e00700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f4f0406b2a0 0x7f4f0406d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:21.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.319+0000 7f4f17e00700 1 -- 172.21.15.1:0/443278372 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f4f0c08b330 con 0x7f4f201003a0 2024-02-28T16:31:21.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.320+0000 7f4f1fe00700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f4f0406b2a0 0x7f4f0406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:21.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.322+0000 7f4f1fe00700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f4f0406b2a0 0x7f4f0406d750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f2006fd90 tx=0x7f4f1000b040 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:21.312 INFO:tasks.workunit.client.1.smithi066.stdout:2/468: write d0/d12/f82 [635477,29435] 0 2024-02-28T16:31:21.312 INFO:tasks.workunit.client.1.smithi066.stdout:1/452: rename d2/d27/d2b/d37/d51/f60 to d2/d27/d2b/d37/d54/d84/f8c 0 2024-02-28T16:31:21.312 INFO:tasks.workunit.client.1.smithi066.stdout:4/558: rmdir d1/d18/d41 39 2024-02-28T16:31:21.314 INFO:tasks.workunit.client.0.smithi001.stdout:7/832: write d3/d17/d3d/d5d/fbd [418516,73785] 0 2024-02-28T16:31:21.315 INFO:tasks.workunit.client.1.smithi066.stdout:0/564: unlink d2/d27/d41/d76/d93/db2/db5/d51/cb1 0 2024-02-28T16:31:21.315 INFO:tasks.workunit.client.0.smithi001.stdout:4/915: link d3/d2d/d4c/d69/d82/fef d3/d2d/d4c/d5e/f12a 0 2024-02-28T16:31:21.315 INFO:tasks.workunit.client.0.smithi001.stdout:5/782: dwrite dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/fe1 [0,4194304] 0 2024-02-28T16:31:21.319 INFO:tasks.workunit.client.1.smithi066.stdout:0/565: chown d2/d27/d41/d76/d93/db2 7981 1 2024-02-28T16:31:21.319 INFO:tasks.workunit.client.1.smithi066.stdout:6/394: dwrite d2/da/d38/f39 [4194304,4194304] 0 2024-02-28T16:31:21.323 INFO:tasks.workunit.client.1.smithi066.stdout:5/446: rmdir d3/d4/dc/d42/d60 39 2024-02-28T16:31:21.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.336+0000 7f4f17e00700 1 -- 172.21.15.1:0/443278372 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f4f0c0556b0 con 0x7f4f201003a0 2024-02-28T16:31:21.325 INFO:tasks.workunit.client.1.smithi066.stdout:5/447: stat d3/d4/dc/d2f/f33 0 2024-02-28T16:31:21.326 INFO:tasks.workunit.client.1.smithi066.stdout:3/468: creat db/dc/d35/d3d/d76/d89/f9f x:0 0 0 2024-02-28T16:31:21.331 INFO:tasks.workunit.client.0.smithi001.stdout:9/988: dread d5/d27/db1/d5e/dd3/d109/d6f/dc8/faf [0,4194304] 0 2024-02-28T16:31:21.331 INFO:tasks.workunit.client.0.smithi001.stdout:8/802: truncate d4/d7/f20 1462368 0 2024-02-28T16:31:21.332 INFO:tasks.workunit.client.0.smithi001.stdout:3/815: fdatasync d4/d18/f11 0 2024-02-28T16:31:21.333 INFO:tasks.workunit.client.0.smithi001.stdout:2/766: write d0/f17 [1123994,91740] 0 2024-02-28T16:31:21.334 INFO:tasks.workunit.client.1.smithi066.stdout:6/395: rmdir d2/d26/d32 39 2024-02-28T16:31:21.334 INFO:tasks.workunit.client.0.smithi001.stdout:1/934: truncate d0/d9/d1a/d2b/d26/d27/d30/d4a/fdf 1081851 0 2024-02-28T16:31:21.334 INFO:tasks.workunit.client.0.smithi001.stdout:0/796: creat d18/d22/d2b/d36/d46/df5/dbc/f106 x:0 0 0 2024-02-28T16:31:21.337 INFO:tasks.workunit.client.1.smithi066.stdout:1/453: dwrite d2/d27/d2b/d37/d51/f62 [0,4194304] 0 2024-02-28T16:31:21.337 INFO:tasks.workunit.client.1.smithi066.stdout:0/566: unlink d2/d27/d41/d76/d93/f11 0 2024-02-28T16:31:21.337 INFO:tasks.workunit.client.0.smithi001.stdout:3/816: stat d4/d1e/d7c/d7d/df8/dee 0 2024-02-28T16:31:21.341 INFO:tasks.workunit.client.1.smithi066.stdout:5/448: truncate d3/d4/fb 3213913 0 2024-02-28T16:31:21.341 INFO:tasks.workunit.client.1.smithi066.stdout:0/567: dread - d2/d27/d41/d76/d93/d10/f98 zero size 2024-02-28T16:31:21.342 INFO:tasks.workunit.client.1.smithi066.stdout:2/469: symlink d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/l91 0 2024-02-28T16:31:21.350 INFO:tasks.workunit.client.1.smithi066.stdout:2/470: truncate d0/d12/d2e/f84 337876 0 2024-02-28T16:31:21.354 INFO:tasks.workunit.client.1.smithi066.stdout:4/559: dread d1/d18/d4e/f94 [0,4194304] 0 2024-02-28T16:31:21.354 INFO:tasks.workunit.client.1.smithi066.stdout:8/428: write d0/db/f27 [944032,129812] 0 2024-02-28T16:31:21.354 INFO:tasks.workunit.client.0.smithi001.stdout:7/833: dwrite d3/d17/d3d/f4d [0,4194304] 0 2024-02-28T16:31:21.355 INFO:tasks.workunit.client.1.smithi066.stdout:0/568: mknod d2/d27/d41/d76/d93/db2/db5/cbe 0 2024-02-28T16:31:21.361 INFO:tasks.workunit.client.0.smithi001.stdout:8/803: dwrite d4/d32/dd6/fe6 [0,4194304] 0 2024-02-28T16:31:21.364 INFO:tasks.workunit.client.1.smithi066.stdout:9/365: sync 2024-02-28T16:31:21.367 INFO:tasks.workunit.client.1.smithi066.stdout:5/449: symlink d3/d1b/d7d/l8a 0 2024-02-28T16:31:21.368 INFO:tasks.workunit.client.1.smithi066.stdout:3/469: write db/d7a/f88 [966803,33289] 0 2024-02-28T16:31:21.368 INFO:tasks.workunit.client.1.smithi066.stdout:8/429: dread d0/d6/f5d [0,4194304] 0 2024-02-28T16:31:21.373 INFO:tasks.workunit.client.1.smithi066.stdout:3/470: readlink db/dc/d35/d3d/l7f 0 2024-02-28T16:31:21.373 INFO:tasks.workunit.client.1.smithi066.stdout:4/560: symlink d1/d18/d4e/lb0 0 2024-02-28T16:31:21.375 INFO:tasks.workunit.client.1.smithi066.stdout:9/366: rmdir da/df/d18/d28 39 2024-02-28T16:31:21.383 INFO:tasks.workunit.client.1.smithi066.stdout:2/471: dread d0/d12/d2e/d7a/f41 [0,4194304] 0 2024-02-28T16:31:21.388 INFO:tasks.workunit.client.1.smithi066.stdout:3/471: creat db/d7a/fa0 x:0 0 0 2024-02-28T16:31:21.388 INFO:tasks.workunit.client.1.smithi066.stdout:0/569: dread d2/d27/d41/d76/d93/d10/d23/d6d/f79 [0,4194304] 0 2024-02-28T16:31:21.395 INFO:tasks.workunit.client.0.smithi001.stdout:9/989: mkdir d5/d27/db1/da6/db2/de4/d13c 0 2024-02-28T16:31:21.396 INFO:tasks.workunit.client.0.smithi001.stdout:3/817: rmdir d4/d1e/db7 39 2024-02-28T16:31:21.396 INFO:tasks.workunit.client.1.smithi066.stdout:8/430: dread d0/db/d15/f31 [4194304,4194304] 0 2024-02-28T16:31:21.400 INFO:tasks.workunit.client.0.smithi001.stdout:1/935: creat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f146 x:0 0 0 2024-02-28T16:31:21.402 INFO:tasks.workunit.client.0.smithi001.stdout:0/797: read d18/d22/d2f/fa4 [186298,39961] 0 2024-02-28T16:31:21.403 INFO:tasks.workunit.client.1.smithi066.stdout:4/561: mknod d1/d18/d41/da7/dae/cb1 0 2024-02-28T16:31:21.404 INFO:tasks.workunit.client.0.smithi001.stdout:8/804: creat d4/d47/d8b/f103 x:0 0 0 2024-02-28T16:31:21.405 INFO:tasks.workunit.client.0.smithi001.stdout:6/787: link d9/d29/d6e/d9a/fad d9/d39/dc4/f107 0 2024-02-28T16:31:21.406 INFO:tasks.workunit.client.0.smithi001.stdout:4/916: creat d3/d2d/d83/d112/f12b x:0 0 0 2024-02-28T16:31:21.407 INFO:tasks.workunit.client.1.smithi066.stdout:6/396: dread d2/d26/d32/d46/d80/d5d/f64 [0,4194304] 0 2024-02-28T16:31:21.408 INFO:tasks.workunit.client.1.smithi066.stdout:5/450: dwrite d3/d4/dc/d2f/f32 [0,4194304] 0 2024-02-28T16:31:21.408 INFO:tasks.workunit.client.1.smithi066.stdout:8/431: chown d0/d6/ff 0 1 2024-02-28T16:31:21.408 INFO:tasks.workunit.client.0.smithi001.stdout:7/834: read d3/d17/d52/d7f/d99/dc1/d49/f65 [1361860,71056] 0 2024-02-28T16:31:21.408 INFO:tasks.workunit.client.1.smithi066.stdout:4/562: readlink d1/d23/d25/d3e/d40/lac 0 2024-02-28T16:31:21.410 INFO:tasks.workunit.client.1.smithi066.stdout:3/472: creat db/dc/d11/d1f/d78/fa1 x:0 0 0 2024-02-28T16:31:21.410 INFO:tasks.workunit.client.0.smithi001.stdout:6/788: readlink d9/da6/l81 0 2024-02-28T16:31:21.410 INFO:tasks.workunit.client.0.smithi001.stdout:9/990: creat d5/d27/db1/d58/f13d x:0 0 0 2024-02-28T16:31:21.413 INFO:tasks.workunit.client.0.smithi001.stdout:3/818: read d4/d1e/db7/fd0 [385821,78343] 0 2024-02-28T16:31:21.413 INFO:tasks.workunit.client.0.smithi001.stdout:2/767: truncate d0/d2/d3/de/d22/f6d 965526 0 2024-02-28T16:31:21.414 INFO:tasks.workunit.client.1.smithi066.stdout:6/397: chown d2/d2a/f5b 211463570 1 2024-02-28T16:31:21.414 INFO:tasks.workunit.client.0.smithi001.stdout:8/805: mkdir d4/d47/df4/d104 0 2024-02-28T16:31:21.420 INFO:tasks.workunit.client.1.smithi066.stdout:3/473: fdatasync db/dc/d35/d3d/d76/f4a 0 2024-02-28T16:31:21.420 INFO:tasks.workunit.client.1.smithi066.stdout:1/454: dwrite d2/d18/d19/f1c [0,4194304] 0 2024-02-28T16:31:21.421 INFO:tasks.workunit.client.0.smithi001.stdout:6/789: symlink d9/db/dd9/l108 0 2024-02-28T16:31:21.425 INFO:tasks.workunit.client.0.smithi001.stdout:8/806: chown d4/d7/l82 32 1 2024-02-28T16:31:21.426 INFO:tasks.workunit.client.1.smithi066.stdout:3/474: truncate db/dc/d35/d3d/d76/f83 251920 0 2024-02-28T16:31:21.428 INFO:tasks.workunit.client.1.smithi066.stdout:2/472: dwrite d0/d12/d1a/f54 [0,4194304] 0 2024-02-28T16:31:21.431 INFO:tasks.workunit.client.1.smithi066.stdout:6/398: readlink d2/l16 0 2024-02-28T16:31:21.438 INFO:tasks.workunit.client.0.smithi001.stdout:3/819: symlink d4/d1e/d7c/l10d 0 2024-02-28T16:31:21.438 INFO:tasks.workunit.client.1.smithi066.stdout:3/475: symlink db/d13/d14/d84/la2 0 2024-02-28T16:31:21.441 INFO:tasks.workunit.client.0.smithi001.stdout:6/790: truncate d9/de/f14 3076941 0 2024-02-28T16:31:21.442 INFO:tasks.workunit.client.1.smithi066.stdout:5/451: creat d3/d4/dc/f8b x:0 0 0 2024-02-28T16:31:21.444 INFO:tasks.workunit.client.1.smithi066.stdout:2/473: creat d0/d12/d2e/d3b/d66/f92 x:0 0 0 2024-02-28T16:31:21.446 INFO:tasks.workunit.client.1.smithi066.stdout:6/399: creat d2/d2a/d3d/f89 x:0 0 0 2024-02-28T16:31:21.447 INFO:tasks.workunit.client.0.smithi001.stdout:7/835: dread d3/d17/d3d/f72 [0,4194304] 0 2024-02-28T16:31:21.447 INFO:tasks.workunit.client.1.smithi066.stdout:0/570: truncate d2/d27/d41/d76/d93/d33/f62 2433202 0 2024-02-28T16:31:21.449 INFO:tasks.workunit.client.1.smithi066.stdout:4/563: link d1/d23/c45 d1/d7a/cb2 0 2024-02-28T16:31:21.450 INFO:tasks.workunit.client.0.smithi001.stdout:8/807: creat d4/d32/d69/d9a/f105 x:0 0 0 2024-02-28T16:31:21.453 INFO:tasks.workunit.client.1.smithi066.stdout:2/474: write d0/d12/f82 [96342,82294] 0 2024-02-28T16:31:21.453 INFO:tasks.workunit.client.1.smithi066.stdout:8/432: dread d0/f24 [0,4194304] 0 2024-02-28T16:31:21.455 INFO:tasks.workunit.client.0.smithi001.stdout:8/808: chown d4/d7/d22/f3f 7064 1 2024-02-28T16:31:21.455 INFO:tasks.workunit.client.1.smithi066.stdout:8/433: readlink d0/db/d15/l3f 0 2024-02-28T16:31:21.456 INFO:tasks.workunit.client.1.smithi066.stdout:8/434: chown d0/d6/c7b 24335 1 2024-02-28T16:31:21.458 INFO:tasks.workunit.client.0.smithi001.stdout:0/798: dread d18/f1e [0,4194304] 0 2024-02-28T16:31:21.459 INFO:tasks.workunit.client.1.smithi066.stdout:0/571: creat d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/fbf x:0 0 0 2024-02-28T16:31:21.459 INFO:tasks.workunit.client.1.smithi066.stdout:4/564: symlink d1/d18/da8/lb3 0 2024-02-28T16:31:21.459 INFO:tasks.workunit.client.1.smithi066.stdout:8/435: fsync d0/d6/dc/d16/d1a/d63/d58/f70 0 2024-02-28T16:31:21.460 INFO:tasks.workunit.client.0.smithi001.stdout:8/809: chown d4/d32/d93/fbb 927418100 1 2024-02-28T16:31:21.461 INFO:tasks.workunit.client.0.smithi001.stdout:3/820: rename d4/d1e/d7c/d7d/de0/df2 to d4/d1e/d7c/d7d/df8/d82/dd5/d10e 0 2024-02-28T16:31:21.465 INFO:tasks.workunit.client.0.smithi001.stdout:7/836: mknod d3/d17/d52/d92/c10b 0 2024-02-28T16:31:21.466 INFO:tasks.workunit.client.0.smithi001.stdout:8/810: chown d4/d32/d69/d8f/ce2 82132755 1 2024-02-28T16:31:21.467 INFO:tasks.workunit.client.0.smithi001.stdout:3/821: dread - d4/d6/d7/d78/d81/f86 zero size 2024-02-28T16:31:21.471 INFO:tasks.workunit.client.0.smithi001.stdout:3/822: stat d4/d18/l4f 0 2024-02-28T16:31:21.475 INFO:tasks.workunit.client.0.smithi001.stdout:5/783: truncate dc/f1d 1657925 0 2024-02-28T16:31:21.476 INFO:tasks.workunit.client.0.smithi001.stdout:7/837: fdatasync d3/d17/d3d/db3/fb4 0 2024-02-28T16:31:21.482 INFO:tasks.workunit.client.0.smithi001.stdout:4/917: dread d3/d2d/d4c/d5e/db0/fba [0,4194304] 0 2024-02-28T16:31:21.482 INFO:tasks.workunit.client.0.smithi001.stdout:6/791: dwrite d9/d29/d6a/fbe [0,4194304] 0 2024-02-28T16:31:21.483 INFO:tasks.workunit.client.0.smithi001.stdout:8/811: symlink d4/d7/d22/d27/d58/l106 0 2024-02-28T16:31:21.485 INFO:tasks.workunit.client.0.smithi001.stdout:1/936: dread d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/fee [0,4194304] 0 2024-02-28T16:31:21.489 INFO:tasks.workunit.client.0.smithi001.stdout:7/838: rename d3/d17/d52/d55/l83 to d3/d17/d3d/db3/dff/l10c 0 2024-02-28T16:31:21.492 INFO:tasks.workunit.client.0.smithi001.stdout:6/792: creat d9/db/d5f/d62/d83/f109 x:0 0 0 2024-02-28T16:31:21.493 INFO:tasks.workunit.client.0.smithi001.stdout:5/784: write dc/d12/d56/d96/db9/dc2/dca/d3c/f5e [2410861,17875] 0 2024-02-28T16:31:21.494 INFO:tasks.workunit.client.0.smithi001.stdout:4/918: fdatasync d3/d2d/f3c 0 2024-02-28T16:31:21.495 INFO:tasks.workunit.client.0.smithi001.stdout:3/823: mknod d4/d1e/d7c/d7d/df8/dee/c10f 0 2024-02-28T16:31:21.499 INFO:tasks.workunit.client.1.smithi066.stdout:7/386: sync 2024-02-28T16:31:21.505 INFO:tasks.workunit.client.0.smithi001.stdout:5/785: unlink dc/d12/d56/d96/db9/dc2/dca/d91/fc7 0 2024-02-28T16:31:21.508 INFO:tasks.workunit.client.0.smithi001.stdout:4/919: rename d3 to d3/d2d/d4c/d5e/d12c 22 2024-02-28T16:31:21.509 INFO:tasks.workunit.client.0.smithi001.stdout:3/824: symlink d4/d1e/l110 0 2024-02-28T16:31:21.509 INFO:tasks.workunit.client.1.smithi066.stdout:3/476: getdents db/dc/d11 0 2024-02-28T16:31:21.514 INFO:tasks.workunit.client.0.smithi001.stdout:4/920: chown d3/d5/d7/d1f/d2e/la6 79898259 1 2024-02-28T16:31:21.514 INFO:tasks.workunit.client.1.smithi066.stdout:5/452: rmdir d3/d4/dc 39 2024-02-28T16:31:21.514 INFO:tasks.workunit.client.1.smithi066.stdout:1/455: dread d2/fc [0,4194304] 0 2024-02-28T16:31:21.515 INFO:tasks.workunit.client.1.smithi066.stdout:3/477: unlink db/f9a 0 2024-02-28T16:31:21.516 INFO:tasks.workunit.client.1.smithi066.stdout:0/572: fdatasync d2/d27/d41/d76/d93/d33/d3f/d8e/fb4 0 2024-02-28T16:31:21.517 INFO:tasks.workunit.client.1.smithi066.stdout:7/387: write dc/de/d17/d1e/f3e [573209,103454] 0 2024-02-28T16:31:21.518 INFO:tasks.workunit.client.0.smithi001.stdout:7/839: dwrite d3/d17/d3d/d5d/fbd [0,4194304] 0 2024-02-28T16:31:21.518 INFO:tasks.workunit.client.0.smithi001.stdout:4/921: read d3/d2d/d4c/d5e/f62 [870238,34489] 0 2024-02-28T16:31:21.519 INFO:tasks.workunit.client.1.smithi066.stdout:8/436: symlink d0/d6/dc/d16/d4c/d6c/l89 0 2024-02-28T16:31:21.522 INFO:tasks.workunit.client.1.smithi066.stdout:3/478: unlink db/d2a/l31 0 2024-02-28T16:31:21.525 INFO:tasks.workunit.client.0.smithi001.stdout:9/991: dread d5/d27/db1/da6/db2/dff/f125 [0,4194304] 0 2024-02-28T16:31:21.530 INFO:tasks.workunit.client.0.smithi001.stdout:3/825: creat d4/d18/ddb/de8/f111 x:0 0 0 2024-02-28T16:31:21.531 INFO:tasks.workunit.client.1.smithi066.stdout:3/479: creat db/dc/d11/d1f/d78/fa3 x:0 0 0 2024-02-28T16:31:21.532 INFO:tasks.workunit.client.0.smithi001.stdout:2/768: dwrite d0/d2/d3/de/d61/d12/d55/db2/fd3 [0,4194304] 0 2024-02-28T16:31:21.533 INFO:tasks.workunit.client.0.smithi001.stdout:0/799: write d18/d22/d2f/d1c/d79/fc2 [536464,82276] 0 2024-02-28T16:31:21.534 INFO:tasks.workunit.client.1.smithi066.stdout:3/480: fsync db/dc/d35/d3d/d76/f83 0 2024-02-28T16:31:21.536 INFO:tasks.workunit.client.0.smithi001.stdout:0/800: chown d18/d5a 374824150 1 2024-02-28T16:31:21.539 INFO:tasks.workunit.client.1.smithi066.stdout:3/481: chown db/c71 29590 1 2024-02-28T16:31:21.543 INFO:tasks.workunit.client.1.smithi066.stdout:4/565: truncate d1/f10 1842881 0 2024-02-28T16:31:21.543 INFO:tasks.workunit.client.1.smithi066.stdout:8/437: write d0/d6/dc/d16/d4c/d42/d73/f7f [1092998,68606] 0 2024-02-28T16:31:21.545 INFO:tasks.workunit.client.1.smithi066.stdout:3/482: truncate db/d13/d1a/d33/f39 886348 0 2024-02-28T16:31:21.545 INFO:tasks.workunit.client.0.smithi001.stdout:5/786: getdents dc/d12/d56 0 2024-02-28T16:31:21.546 INFO:tasks.workunit.client.0.smithi001.stdout:9/992: symlink d5/d27/db1/d33/d115/d11f/da8/dd1/l13e 0 2024-02-28T16:31:21.546 INFO:tasks.workunit.client.0.smithi001.stdout:6/793: write d9/db/d5f/d69/fc8 [783385,96552] 0 2024-02-28T16:31:21.546 INFO:tasks.workunit.client.0.smithi001.stdout:0/801: dread d18/d22/d2f/ff1 [0,4194304] 0 2024-02-28T16:31:21.547 INFO:tasks.workunit.client.1.smithi066.stdout:4/566: dread - d1/d23/d25/d43/f8b zero size 2024-02-28T16:31:21.550 INFO:tasks.workunit.client.1.smithi066.stdout:5/453: write d3/d4/dc/dd/d2e/d18/f4a [2477491,84824] 0 2024-02-28T16:31:21.551 INFO:tasks.workunit.client.1.smithi066.stdout:8/438: mkdir d0/db/d12/d8a 0 2024-02-28T16:31:21.552 INFO:tasks.workunit.client.0.smithi001.stdout:2/769: creat d0/d2/da6/fff x:0 0 0 2024-02-28T16:31:21.556 INFO:tasks.workunit.client.1.smithi066.stdout:3/483: write db/dc/d11/f26 [740517,95209] 0 2024-02-28T16:31:21.556 INFO:tasks.workunit.client.1.smithi066.stdout:4/567: write d1/d23/d25/d3e/f66 [2654094,38094] 0 2024-02-28T16:31:21.558 INFO:tasks.workunit.client.1.smithi066.stdout:8/439: unlink d0/db/d15/l3f 0 2024-02-28T16:31:21.558 INFO:tasks.workunit.client.0.smithi001.stdout:6/794: fsync f5 0 2024-02-28T16:31:21.561 INFO:tasks.workunit.client.0.smithi001.stdout:0/802: truncate d18/d22/d57/fd5 223404 0 2024-02-28T16:31:21.561 INFO:tasks.workunit.client.1.smithi066.stdout:5/454: chown d3/d4/dc/d2f/f33 400690 1 2024-02-28T16:31:21.562 INFO:tasks.workunit.client.0.smithi001.stdout:7/840: write d3/d17/d52/d7f/d99/dc1/d49/f65 [1309188,90237] 0 2024-02-28T16:31:21.562 INFO:tasks.workunit.client.1.smithi066.stdout:3/484: dread - db/d13/d1a/d25/f67 zero size 2024-02-28T16:31:21.564 INFO:tasks.workunit.client.1.smithi066.stdout:7/388: dwrite dc/f4b [0,4194304] 0 2024-02-28T16:31:21.565 INFO:tasks.workunit.client.1.smithi066.stdout:0/573: dwrite d2/f5e [0,4194304] 0 2024-02-28T16:31:21.566 INFO:tasks.workunit.client.0.smithi001.stdout:0/803: read d18/d22/d76/d7a/fae [472382,102314] 0 2024-02-28T16:31:21.566 INFO:tasks.workunit.client.0.smithi001.stdout:2/770: chown d0/d2/d3/de/d22/fd1 17379908 1 2024-02-28T16:31:21.567 INFO:tasks.workunit.client.1.smithi066.stdout:3/485: chown db/d2a/c40 310435566 1 2024-02-28T16:31:21.567 INFO:tasks.workunit.client.1.smithi066.stdout:5/455: symlink d3/d1b/d20/l8c 0 2024-02-28T16:31:21.571 INFO:tasks.workunit.client.1.smithi066.stdout:0/574: chown d2/d27/d41/d76/d93/db2 2080036538 1 2024-02-28T16:31:21.571 INFO:tasks.workunit.client.0.smithi001.stdout:4/922: rename d3/d5/d7/d1f/d20/f95 to d3/d5/d7/d1f/dc3/f12d 0 2024-02-28T16:31:21.572 INFO:tasks.workunit.client.0.smithi001.stdout:9/993: mknod d5/d27/db1/d33/d115/d11f/d131/c13f 0 2024-02-28T16:31:21.572 INFO:tasks.workunit.client.0.smithi001.stdout:4/923: fdatasync d3/f78 0 2024-02-28T16:31:21.574 INFO:tasks.workunit.client.0.smithi001.stdout:3/826: truncate d4/d6/d7/d23/f7a 3821311 0 2024-02-28T16:31:21.574 INFO:tasks.workunit.client.1.smithi066.stdout:3/486: creat db/dc/d11/d1f/d78/fa4 x:0 0 0 2024-02-28T16:31:21.575 INFO:tasks.workunit.client.0.smithi001.stdout:0/804: mknod d18/d22/d2f/d1c/d90/d9f/c107 0 2024-02-28T16:31:21.577 INFO:tasks.workunit.client.1.smithi066.stdout:0/575: write d2/d27/d41/d76/d93/d10/f7f [122842,46798] 0 2024-02-28T16:31:21.578 INFO:tasks.workunit.client.0.smithi001.stdout:2/771: rmdir d0/d2/d3/de/d22/dbb 39 2024-02-28T16:31:21.579 INFO:tasks.workunit.client.1.smithi066.stdout:5/456: link d3/d4/dc/dd/f89 d3/d4/dc/d2f/d55/d74/d84/f8d 0 2024-02-28T16:31:21.580 INFO:tasks.workunit.client.0.smithi001.stdout:0/805: readlink d18/d22/d2f/d1c/l27 0 2024-02-28T16:31:21.580 INFO:tasks.workunit.client.0.smithi001.stdout:6/795: rename d9/d29/d6a/f77 to d9/d29/d6a/ded/f10a 0 2024-02-28T16:31:21.581 INFO:tasks.workunit.client.1.smithi066.stdout:3/487: rename db/d2a/c5a to db/d7a/d7c/ca5 0 2024-02-28T16:31:21.584 INFO:tasks.workunit.client.1.smithi066.stdout:0/576: write d2/f32 [354149,53309] 0 2024-02-28T16:31:21.585 INFO:tasks.workunit.client.0.smithi001.stdout:7/841: creat d3/dc5/f10d x:0 0 0 2024-02-28T16:31:21.586 INFO:tasks.workunit.client.1.smithi066.stdout:7/389: dread dc/d29/f41 [0,4194304] 0 2024-02-28T16:31:21.587 INFO:tasks.workunit.client.0.smithi001.stdout:4/924: mknod d3/d1c/d58/d6e/c12e 0 2024-02-28T16:31:21.587 INFO:tasks.workunit.client.0.smithi001.stdout:5/787: link dc/d1a/d52/ld4 dc/d12/d41/d54/lfe 0 2024-02-28T16:31:21.591 INFO:tasks.workunit.client.0.smithi001.stdout:0/806: fdatasync d18/d5a/fda 0 2024-02-28T16:31:21.592 INFO:tasks.workunit.client.1.smithi066.stdout:4/568: dread d1/f4f [0,4194304] 0 2024-02-28T16:31:21.595 INFO:tasks.workunit.client.0.smithi001.stdout:6/796: write d9/db/d5f/fb1 [921496,19510] 0 2024-02-28T16:31:21.596 INFO:tasks.workunit.client.0.smithi001.stdout:4/925: unlink d3/d2d/l36 0 2024-02-28T16:31:21.597 INFO:tasks.workunit.client.1.smithi066.stdout:0/577: creat d2/d27/d41/d76/d93/db2/db5/d67/fc0 x:0 0 0 2024-02-28T16:31:21.600 INFO:tasks.workunit.client.0.smithi001.stdout:5/788: rmdir dc/d1a 39 2024-02-28T16:31:21.601 INFO:tasks.workunit.client.1.smithi066.stdout:7/390: creat dc/de/d17/d37/f72 x:0 0 0 2024-02-28T16:31:21.602 INFO:tasks.workunit.client.0.smithi001.stdout:9/994: rename d5/d27/db1/d33/d115/f137 to d5/f140 0 2024-02-28T16:31:21.604 INFO:tasks.workunit.client.0.smithi001.stdout:5/789: fsync dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/f82 0 2024-02-28T16:31:21.605 INFO:tasks.workunit.client.0.smithi001.stdout:7/842: creat d3/d17/f10e x:0 0 0 2024-02-28T16:31:21.607 INFO:tasks.workunit.client.1.smithi066.stdout:0/578: fdatasync d2/d27/d41/d76/d93/db2/db5/d52/f55 0 2024-02-28T16:31:21.608 INFO:tasks.workunit.client.0.smithi001.stdout:9/995: stat d5/d27/d34/d50/la2 0 2024-02-28T16:31:21.608 INFO:tasks.workunit.client.1.smithi066.stdout:3/488: link db/d2a/d44/l6a db/d13/d1a/la6 0 2024-02-28T16:31:21.608 INFO:tasks.workunit.client.1.smithi066.stdout:4/569: creat d1/d18/d4e/da0/fb4 x:0 0 0 2024-02-28T16:31:21.610 INFO:tasks.workunit.client.1.smithi066.stdout:3/489: dread - db/dc/d11/d1f/d78/fa4 zero size 2024-02-28T16:31:21.610 INFO:tasks.workunit.client.1.smithi066.stdout:5/457: dwrite d3/d1b/f31 [0,4194304] 0 2024-02-28T16:31:21.614 INFO:tasks.workunit.client.1.smithi066.stdout:3/490: write db/dc/d11/f26 [789510,61230] 0 2024-02-28T16:31:21.618 INFO:tasks.workunit.client.0.smithi001.stdout:9/996: unlink d5/d27/c32 0 2024-02-28T16:31:21.621 INFO:tasks.workunit.client.1.smithi066.stdout:4/570: symlink d1/d23/d25/d3e/d40/d9c/lb5 0 2024-02-28T16:31:21.621 INFO:tasks.workunit.client.1.smithi066.stdout:3/491: stat db/d2a/d44/d48/d52/d77 0 2024-02-28T16:31:21.623 INFO:tasks.workunit.client.0.smithi001.stdout:3/827: write d4/d6/d7/f2f [1508102,68701] 0 2024-02-28T16:31:21.626 INFO:tasks.workunit.client.1.smithi066.stdout:3/492: unlink db/d13/d1a/l50 0 2024-02-28T16:31:21.626 INFO:tasks.workunit.client.1.smithi066.stdout:4/571: creat d1/d23/d25/d77/fb6 x:0 0 0 2024-02-28T16:31:21.627 INFO:tasks.workunit.client.0.smithi001.stdout:9/997: creat d5/d27/db1/d33/d115/f141 x:0 0 0 2024-02-28T16:31:21.627 INFO:tasks.workunit.client.1.smithi066.stdout:7/391: dread dc/d29/f2e [0,4194304] 0 2024-02-28T16:31:21.630 INFO:tasks.workunit.client.0.smithi001.stdout:0/807: dread d18/d22/d76/d7a/fc4 [0,4194304] 0 2024-02-28T16:31:21.630 INFO:tasks.workunit.client.0.smithi001.stdout:3/828: symlink d4/d18/ddb/l112 0 2024-02-28T16:31:21.631 INFO:tasks.workunit.client.1.smithi066.stdout:7/392: mknod dc/d6e/c73 0 2024-02-28T16:31:21.631 INFO:tasks.workunit.client.1.smithi066.stdout:3/493: mknod db/d13/d14/ca7 0 2024-02-28T16:31:21.632 INFO:tasks.workunit.client.0.smithi001.stdout:9/998: write d5/d27/db1/d33/fdb [8199078,74533] 0 2024-02-28T16:31:21.632 INFO:tasks.workunit.client.1.smithi066.stdout:4/572: symlink d1/d23/d25/lb7 0 2024-02-28T16:31:21.634 INFO:tasks.workunit.client.1.smithi066.stdout:4/573: chown d1/d23/d25/lb7 733 1 2024-02-28T16:31:21.634 INFO:tasks.workunit.client.1.smithi066.stdout:5/458: dwrite d3/d4/f5c [0,4194304] 0 2024-02-28T16:31:21.636 INFO:tasks.workunit.client.0.smithi001.stdout:5/790: write dc/df/f99 [3633877,96928] 0 2024-02-28T16:31:21.637 INFO:tasks.workunit.client.1.smithi066.stdout:4/574: write d1/d23/f36 [2839375,98005] 0 2024-02-28T16:31:21.640 INFO:tasks.workunit.client.1.smithi066.stdout:5/459: mknod d3/d4/dc/d42/d57/c8e 0 2024-02-28T16:31:21.642 INFO:tasks.workunit.client.0.smithi001.stdout:9/999: mknod d5/d27/d42/c142 0 2024-02-28T16:31:21.643 INFO:tasks.workunit.client.1.smithi066.stdout:4/575: mknod d1/d23/d25/cb8 0 2024-02-28T16:31:21.645 INFO:tasks.workunit.client.1.smithi066.stdout:5/460: stat d3/d4/dc/d2f/d55/d74/d84/f8d 0 2024-02-28T16:31:21.646 INFO:tasks.workunit.client.0.smithi001.stdout:0/808: write d18/d22/d2b/d36/d46/df5/dbc/f106 [413629,111703] 0 2024-02-28T16:31:21.646 INFO:tasks.workunit.client.0.smithi001.stdout:2/772: dwrite d0/d2/f3b [0,4194304] 0 2024-02-28T16:31:21.650 INFO:tasks.workunit.client.1.smithi066.stdout:4/576: creat d1/d23/d25/d3e/d40/d9c/fb9 x:0 0 0 2024-02-28T16:31:21.651 INFO:tasks.workunit.client.0.smithi001.stdout:6/797: dread d9/de/fe2 [0,4194304] 0 2024-02-28T16:31:21.651 INFO:tasks.workunit.client.1.smithi066.stdout:5/461: mkdir d3/d4/dc/dd/d2e/d8f 0 2024-02-28T16:31:21.652 INFO:tasks.workunit.client.1.smithi066.stdout:3/494: dwrite db/d13/d1a/d33/f39 [0,4194304] 0 2024-02-28T16:31:21.653 INFO:tasks.workunit.client.0.smithi001.stdout:3/829: rename d4/d1e/d7c/d9a/cbe to d4/d18/ddb/dfb/c113 0 2024-02-28T16:31:21.655 INFO:tasks.workunit.client.0.smithi001.stdout:3/830: chown d4/d1e/def 6 1 2024-02-28T16:31:21.656 INFO:tasks.workunit.client.1.smithi066.stdout:7/393: dread f4 [0,4194304] 0 2024-02-28T16:31:21.656 INFO:tasks.workunit.client.1.smithi066.stdout:3/495: truncate db/d7a/fa0 837752 0 2024-02-28T16:31:21.656 INFO:tasks.workunit.client.1.smithi066.stdout:4/577: rename d1/d18/d55/d81 to d1/d18/d41/dba 0 2024-02-28T16:31:21.657 INFO:tasks.workunit.client.0.smithi001.stdout:6/798: write d9/d39/f3a [694599,64306] 0 2024-02-28T16:31:21.658 INFO:tasks.workunit.client.0.smithi001.stdout:2/773: link d0/d2/da6/ff3 d0/d2/d3/de/d61/d12/d55/db2/f100 0 2024-02-28T16:31:21.661 INFO:tasks.workunit.client.0.smithi001.stdout:6/799: creat d9/d29/d6a/f10b x:0 0 0 2024-02-28T16:31:21.661 INFO:tasks.workunit.client.1.smithi066.stdout:7/394: chown dc/l1f 12792 1 2024-02-28T16:31:21.662 INFO:tasks.workunit.client.0.smithi001.stdout:2/774: readlink d0/d2/d3/de/d22/lb6 0 2024-02-28T16:31:21.663 INFO:tasks.workunit.client.1.smithi066.stdout:4/578: unlink d1/d18/d41/f9a 0 2024-02-28T16:31:21.664 INFO:tasks.workunit.client.0.smithi001.stdout:5/791: truncate dc/d1a/d7d/d87/f8f 2942827 0 2024-02-28T16:31:21.664 INFO:tasks.workunit.client.1.smithi066.stdout:4/579: write d1/f96 [399100,89337] 0 2024-02-28T16:31:21.665 INFO:tasks.workunit.client.1.smithi066.stdout:7/395: symlink dc/de/d17/l74 0 2024-02-28T16:31:21.666 INFO:tasks.workunit.client.0.smithi001.stdout:2/775: unlink d0/d2/cf6 0 2024-02-28T16:31:21.666 INFO:tasks.workunit.client.0.smithi001.stdout:7/843: dread d3/d17/d3d/f70 [0,4194304] 0 2024-02-28T16:31:21.668 INFO:tasks.workunit.client.1.smithi066.stdout:9/367: sync 2024-02-28T16:31:21.668 INFO:tasks.workunit.client.1.smithi066.stdout:4/580: mkdir d1/d18/d7e/dbb 0 2024-02-28T16:31:21.669 INFO:tasks.workunit.client.1.smithi066.stdout:7/396: symlink dc/de/d17/d37/l75 0 2024-02-28T16:31:21.669 INFO:tasks.workunit.client.0.smithi001.stdout:2/776: fsync d0/d2/d3/de/d22/f24 0 2024-02-28T16:31:21.674 INFO:tasks.workunit.client.1.smithi066.stdout:4/581: mkdir d1/d18/d4e/dbc 0 2024-02-28T16:31:21.674 INFO:tasks.workunit.client.1.smithi066.stdout:7/397: truncate dc/de/d17/d1e/f6c 898322 0 2024-02-28T16:31:21.674 INFO:tasks.workunit.client.0.smithi001.stdout:2/777: write d0/d2/d3/de/d61/d12/d55/db2/fe6 [549555,59407] 0 2024-02-28T16:31:21.678 INFO:tasks.workunit.client.1.smithi066.stdout:0/579: truncate d2/d27/d41/d76/d93/d10/f2c 6408469 0 2024-02-28T16:31:21.680 INFO:tasks.workunit.client.1.smithi066.stdout:4/582: mknod d1/d18/d41/cbd 0 2024-02-28T16:31:21.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:21 smithi001 ceph-mon[28991]: pgmap v54: 65 pgs: 65 active+clean; 2.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 83 MiB/s rd, 294 MiB/s wr, 552 op/s 2024-02-28T16:31:21.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:21 smithi001 ceph-mon[28991]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:21.682 INFO:tasks.workunit.client.1.smithi066.stdout:0/580: symlink d2/d85/d8d/lc1 0 2024-02-28T16:31:21.683 INFO:tasks.workunit.client.1.smithi066.stdout:7/398: dread dc/de/d17/d1e/f6c [0,4194304] 0 2024-02-28T16:31:21.685 INFO:tasks.workunit.client.1.smithi066.stdout:4/583: creat d1/d18/da8/fbe x:0 0 0 2024-02-28T16:31:21.687 INFO:tasks.workunit.client.0.smithi001.stdout:5/792: dread dc/d12/d56/d96/db9/fdc [0,4194304] 0 2024-02-28T16:31:21.688 INFO:tasks.workunit.client.1.smithi066.stdout:4/584: chown d1/f4f 32894 1 2024-02-28T16:31:21.689 INFO:tasks.workunit.client.0.smithi001.stdout:6/800: dwrite d9/db/d19/f78 [0,4194304] 0 2024-02-28T16:31:21.690 INFO:tasks.workunit.client.0.smithi001.stdout:5/793: creat dc/d12/d56/fff x:0 0 0 2024-02-28T16:31:21.692 INFO:tasks.workunit.client.1.smithi066.stdout:9/368: dwrite da/df/d18/d2d/f50 [0,4194304] 0 2024-02-28T16:31:21.692 INFO:tasks.workunit.client.1.smithi066.stdout:7/399: unlink dc/de/d17/d37/l75 0 2024-02-28T16:31:21.693 INFO:tasks.workunit.client.0.smithi001.stdout:6/801: chown d9/d29/d6e/d4e/fd0 13059224 1 2024-02-28T16:31:21.695 INFO:tasks.workunit.client.0.smithi001.stdout:0/809: fdatasync d18/d22/d2b/d36/d46/df5/dbc/f106 0 2024-02-28T16:31:21.696 INFO:tasks.workunit.client.1.smithi066.stdout:9/369: read da/d3a/f41 [1576591,80884] 0 2024-02-28T16:31:21.698 INFO:tasks.workunit.client.1.smithi066.stdout:4/585: creat d1/d18/d41/fbf x:0 0 0 2024-02-28T16:31:21.706 INFO:tasks.workunit.client.1.smithi066.stdout:4/586: truncate d1/d23/f75 3770745 0 2024-02-28T16:31:21.707 INFO:tasks.workunit.client.1.smithi066.stdout:4/587: chown d1/d18/d41/da7 1357 1 2024-02-28T16:31:21.708 INFO:tasks.workunit.client.1.smithi066.stdout:9/370: stat da/df/d18/d28/d42/c79 0 2024-02-28T16:31:21.708 INFO:tasks.workunit.client.1.smithi066.stdout:0/581: dwrite d2/f32 [0,4194304] 0 2024-02-28T16:31:21.715 INFO:tasks.workunit.client.1.smithi066.stdout:4/588: mknod d1/d18/d4e/da0/cc0 0 2024-02-28T16:31:21.715 INFO:tasks.workunit.client.1.smithi066.stdout:9/371: symlink da/d22/d46/d56/l7d 0 2024-02-28T16:31:21.719 INFO:tasks.workunit.client.1.smithi066.stdout:4/589: chown d1/c3c 256 1 2024-02-28T16:31:21.720 INFO:tasks.workunit.client.1.smithi066.stdout:0/582: link d2/l3 d2/d27/d41/d76/d93/db2/db5/d52/lc2 0 2024-02-28T16:31:21.722 INFO:tasks.workunit.client.1.smithi066.stdout:4/590: chown d1/d18/d41/dba/f9d 21810950 1 2024-02-28T16:31:21.724 INFO:tasks.workunit.client.1.smithi066.stdout:9/372: getdents da/df/d18/d2d 0 2024-02-28T16:31:21.725 INFO:tasks.workunit.client.1.smithi066.stdout:0/583: rename d2/f32 to d2/d27/d41/d76/d93/db2/db5/d73/fc3 0 2024-02-28T16:31:21.726 INFO:tasks.workunit.client.1.smithi066.stdout:4/591: mkdir d1/d18/d41/da7/dae/dc1 0 2024-02-28T16:31:21.728 INFO:tasks.workunit.client.1.smithi066.stdout:4/592: readlink d1/l8 0 2024-02-28T16:31:21.729 INFO:tasks.workunit.client.1.smithi066.stdout:0/584: write d2/d27/d41/d76/d93/db2/db5/d51/f82 [357186,90391] 0 2024-02-28T16:31:21.731 INFO:tasks.workunit.client.1.smithi066.stdout:4/593: read d1/d18/f2b [474362,3286] 0 2024-02-28T16:31:21.732 INFO:tasks.workunit.client.1.smithi066.stdout:0/585: creat d2/d27/d41/d76/d93/d10/d23/fc4 x:0 0 0 2024-02-28T16:31:21.734 INFO:tasks.workunit.client.1.smithi066.stdout:0/586: stat d2/d27/d41/d76/d93/l86 0 2024-02-28T16:31:21.735 INFO:tasks.workunit.client.1.smithi066.stdout:0/587: stat d2/d27/d41/d76/d93/d33/d3f/faa 0 2024-02-28T16:31:21.737 INFO:tasks.workunit.client.1.smithi066.stdout:0/588: mkdir d2/d27/d41/dc5 0 2024-02-28T16:31:21.740 INFO:tasks.workunit.client.1.smithi066.stdout:0/589: mknod d2/d27/d41/d76/d93/db2/db5/d67/d94/cc6 0 2024-02-28T16:31:21.745 INFO:tasks.workunit.client.1.smithi066.stdout:4/594: dwrite d1/d23/d25/d77/fa3 [0,4194304] 0 2024-02-28T16:31:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.757+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f20101000 con 0x7f4f0406b2a0 2024-02-28T16:31:21.752 INFO:tasks.workunit.client.1.smithi066.stdout:4/595: rmdir d1/d18/d4e/dbc 0 2024-02-28T16:31:21.754 INFO:tasks.workunit.client.1.smithi066.stdout:4/596: chown d1/d23/d25/d43/f82 6614803 1 2024-02-28T16:31:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.771+0000 7f4f17e00700 1 -- 172.21.15.1:0/443278372 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f4f20101000 con 0x7f4f0406b2a0 2024-02-28T16:31:21.759 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:31:21.759 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 starting - - - - 2024-02-28T16:31:21.759 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (12m) 89s ago 12m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:31:21.759 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (10m) 88s ago 10m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 starting - - - - 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (4m) 89s ago 4m 67.7M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (4m) 89s ago 4m 11.8M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (4m) 88s ago 4m 11.9M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (4m) 88s ago 4m 68.6M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:9283 running (13m) 89s ago 13m 336M - 17.2.7-535-gc0f02e60 1c37ed371348 8eb0fd290742 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283 running (119s) 88s ago 10m 492M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 30a2de341a4e 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (13m) 89s ago 13m 47.6M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (10m) 88s ago 10m 36.1M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 starting - - - - 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 starting - - - - 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (9m) 89s ago 9m 62.2M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:31:21.760 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (8m) 89s ago 8m 66.5M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:31:21.761 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (8m) 89s ago 8m 59.1M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:31:21.761 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (7m) 88s ago 7m 69.1M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:31:21.761 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (7m) 88s ago 7m 65.0M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:31:21.761 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (6m) 88s ago 6m 60.9M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:31:21.761 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 starting - - - - 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f4f0406b2a0 msgr2=0x7f4f0406d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f4f0406b2a0 0x7f4f0406d750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f2006fd90 tx=0x7f4f1000b040 comp rx=0 tx=0).stop 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 msgr2=0x7f4f2006f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 0x7f4f2006f470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c005010 tx=0x7f4f0c009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 shutdown_connections 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f4f0406b2a0 0x7f4f0406d750 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f200fda80 0x7f4f2006ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 --2- 172.21.15.1:0/443278372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4f201003a0 0x7f4f2006f470 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.773+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 >> 172.21.15.1:0/443278372 conn(0x7f4f200f74e0 msgr2=0x7f4f200f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.774+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 shutdown_connections 2024-02-28T16:31:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:21.774+0000 7f4f26a00700 1 -- 172.21.15.1:0/443278372 wait complete. 2024-02-28T16:31:21.771 INFO:tasks.workunit.client.1.smithi066.stdout:4/597: dread d1/d18/d41/f47 [0,4194304] 0 2024-02-28T16:31:21.772 INFO:tasks.workunit.client.1.smithi066.stdout:4/598: read - d1/d18/d4e/da0/fb4 zero size 2024-02-28T16:31:21.775 INFO:tasks.workunit.client.0.smithi001.stdout:6/802: creat d9/d29/d6a/f10c x:0 0 0 2024-02-28T16:31:21.785 INFO:tasks.workunit.client.0.smithi001.stdout:5/794: dread dc/d1a/f7b [0,4194304] 0 2024-02-28T16:31:21.788 INFO:tasks.workunit.client.0.smithi001.stdout:6/803: truncate d9/db/d5f/d8b/f72 772035 0 2024-02-28T16:31:21.793 INFO:tasks.workunit.client.0.smithi001.stdout:2/778: dwrite d0/d2/d3/de/d22/f38 [0,4194304] 0 2024-02-28T16:31:21.794 INFO:tasks.workunit.client.0.smithi001.stdout:7/844: dwrite d3/d17/d3d/d5d/fcf [0,4194304] 0 2024-02-28T16:31:21.803 INFO:tasks.workunit.client.0.smithi001.stdout:2/779: write d0/d2/d3/de/d8a/f70 [6169886,119290] 0 2024-02-28T16:31:21.803 INFO:tasks.workunit.client.0.smithi001.stdout:5/795: creat dc/d12/d41/dd6/f100 x:0 0 0 2024-02-28T16:31:21.807 INFO:tasks.workunit.client.0.smithi001.stdout:7/845: chown d3/d17/d52/d7f/d99/dc1/d49/f65 101 1 2024-02-28T16:31:21.807 INFO:tasks.workunit.client.0.smithi001.stdout:6/804: rmdir d9 39 2024-02-28T16:31:21.811 INFO:tasks.workunit.client.1.smithi066.stdout:0/590: write d2/d27/d41/d76/d93/d10/f2c [6034384,105457] 0 2024-02-28T16:31:21.811 INFO:tasks.workunit.client.1.smithi066.stdout:4/599: write d1/d18/f4a [2606828,24287] 0 2024-02-28T16:31:21.812 INFO:tasks.workunit.client.0.smithi001.stdout:0/810: dwrite d18/d5a/fbe [0,4194304] 0 2024-02-28T16:31:21.814 INFO:tasks.workunit.client.1.smithi066.stdout:7/400: dread dc/de/d17/d37/f54 [0,4194304] 0 2024-02-28T16:31:21.819 INFO:tasks.workunit.client.1.smithi066.stdout:7/401: write dc/de/d17/d1e/f59 [362314,125387] 0 2024-02-28T16:31:21.822 INFO:tasks.workunit.client.1.smithi066.stdout:0/591: rmdir d2/d27/d41/d76/d93/db2/db5 39 2024-02-28T16:31:21.825 INFO:tasks.workunit.client.0.smithi001.stdout:3/831: dread d4/d18/f91 [0,4194304] 0 2024-02-28T16:31:21.825 INFO:tasks.workunit.client.1.smithi066.stdout:0/592: dread - d2/d27/d41/d76/d93/d10/d23/fc4 zero size 2024-02-28T16:31:21.826 INFO:tasks.workunit.client.1.smithi066.stdout:4/600: link d1/d18/c34 d1/d18/d55/cc2 0 2024-02-28T16:31:21.826 INFO:tasks.workunit.client.0.smithi001.stdout:0/811: rename d18/d22/d2b/l39 to d18/d22/d57/d7e/l108 0 2024-02-28T16:31:21.830 INFO:tasks.workunit.client.0.smithi001.stdout:2/780: read d0/f62 [3756528,82511] 0 2024-02-28T16:31:21.832 INFO:tasks.workunit.client.0.smithi001.stdout:6/805: fdatasync d9/db/d10/f46 0 2024-02-28T16:31:21.835 INFO:tasks.workunit.client.0.smithi001.stdout:7/846: creat d3/d17/d3d/f10f x:0 0 0 2024-02-28T16:31:21.838 INFO:tasks.workunit.client.0.smithi001.stdout:2/781: stat d0/d2/l97 0 2024-02-28T16:31:21.839 INFO:tasks.workunit.client.0.smithi001.stdout:3/832: link d4/d6/f9b d4/d18/d5f/f114 0 2024-02-28T16:31:21.842 INFO:tasks.workunit.client.1.smithi066.stdout:0/593: write d2/d27/d41/d76/d93/db2/db5/d52/f81 [1443015,60568] 0 2024-02-28T16:31:21.845 INFO:tasks.workunit.client.0.smithi001.stdout:0/812: dread d18/d22/d76/d7a/fae [0,4194304] 0 2024-02-28T16:31:21.845 INFO:tasks.workunit.client.0.smithi001.stdout:6/806: rename d9/db/d5f/d89/d8a/ff5 to d9/db/f10d 0 2024-02-28T16:31:21.847 INFO:tasks.workunit.client.0.smithi001.stdout:3/833: stat d4/d6/d7/d78/c107 0 2024-02-28T16:31:21.855 INFO:tasks.workunit.client.0.smithi001.stdout:2/782: creat d0/d2/d3/de/d22/f101 x:0 0 0 2024-02-28T16:31:21.858 INFO:tasks.workunit.client.1.smithi066.stdout:0/594: dwrite d2/d27/d41/d76/d93/db2/db5/d52/f55 [4194304,4194304] 0 2024-02-28T16:31:21.859 INFO:tasks.workunit.client.0.smithi001.stdout:3/834: truncate d4/d6/f87 2529703 0 2024-02-28T16:31:21.862 INFO:tasks.workunit.client.1.smithi066.stdout:7/402: dread dc/de/f13 [0,4194304] 0 2024-02-28T16:31:21.864 INFO:tasks.workunit.client.1.smithi066.stdout:7/403: chown dc/f5d 20147192 1 2024-02-28T16:31:21.865 INFO:tasks.workunit.client.0.smithi001.stdout:0/813: truncate d18/d22/d2f/f24 1766296 0 2024-02-28T16:31:21.869 INFO:tasks.workunit.client.1.smithi066.stdout:0/595: link d2/d27/d41/d76/d93/d10/d23/d6d/cac d2/d85/d8d/cc7 0 2024-02-28T16:31:21.870 INFO:tasks.workunit.client.0.smithi001.stdout:3/835: read d4/d18/fad [4127688,103191] 0 2024-02-28T16:31:21.872 INFO:tasks.workunit.client.1.smithi066.stdout:0/596: chown d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f72 217 1 2024-02-28T16:31:21.873 INFO:tasks.workunit.client.0.smithi001.stdout:5/796: dwrite dc/d12/d56/d96/db9/dc2/dca/f77 [0,4194304] 0 2024-02-28T16:31:21.877 INFO:tasks.workunit.client.1.smithi066.stdout:0/597: mknod d2/d27/d41/d76/d93/db2/db5/d52/da2/cc8 0 2024-02-28T16:31:21.878 INFO:tasks.workunit.client.0.smithi001.stdout:7/847: dread d3/d17/d3d/d42/f53 [0,4194304] 0 2024-02-28T16:31:21.880 INFO:tasks.workunit.client.1.smithi066.stdout:0/598: mknod d2/d27/d41/d76/d93/db2/db5/d52/cc9 0 2024-02-28T16:31:21.884 INFO:tasks.workunit.client.0.smithi001.stdout:6/807: dwrite d9/de/feb [0,4194304] 0 2024-02-28T16:31:21.887 INFO:tasks.workunit.client.1.smithi066.stdout:0/599: rename d2/d27/d41/d76/d93/d33/d3f/d8e/lbc to d2/d27/d41/d76/d93/db2/db5/d67/lca 0 2024-02-28T16:31:21.888 INFO:tasks.workunit.client.1.smithi066.stdout:0/600: chown d2/d85/d8d/lc1 403 1 2024-02-28T16:31:21.892 INFO:tasks.workunit.client.1.smithi066.stdout:0/601: symlink d2/d85/d8d/lcb 0 2024-02-28T16:31:21.896 INFO:tasks.workunit.client.1.smithi066.stdout:0/602: fsync d2/d27/d41/d76/d93/d33/d3f/d8e/fb4 0 2024-02-28T16:31:21.897 INFO:tasks.workunit.client.1.smithi066.stdout:7/404: dread f5 [0,4194304] 0 2024-02-28T16:31:21.902 INFO:tasks.workunit.client.1.smithi066.stdout:6/400: sync 2024-02-28T16:31:21.903 INFO:tasks.workunit.client.1.smithi066.stdout:6/401: fsync d2/da/f18 0 2024-02-28T16:31:21.906 INFO:tasks.workunit.client.1.smithi066.stdout:6/402: chown d2/d26/d32/d46/d80/d5d/c78 2 1 2024-02-28T16:31:21.910 INFO:tasks.workunit.client.1.smithi066.stdout:7/405: dread dc/f4b [0,4194304] 0 2024-02-28T16:31:21.912 INFO:tasks.workunit.client.1.smithi066.stdout:7/406: mknod dc/d29/d5c/c76 0 2024-02-28T16:31:21.915 INFO:tasks.workunit.client.0.smithi001.stdout:5/797: unlink dc/d12/d56/d96/db9/dc2/dca/d3c/da0/fab 0 2024-02-28T16:31:21.921 INFO:tasks.workunit.client.1.smithi066.stdout:0/603: dwrite d2/d27/d41/d76/d93/db2/db5/d73/fa8 [0,4194304] 0 2024-02-28T16:31:21.921 INFO:tasks.workunit.client.0.smithi001.stdout:2/783: dwrite d0/d2/d3/d36/f4b [4194304,4194304] 0 2024-02-28T16:31:21.927 INFO:tasks.workunit.client.1.smithi066.stdout:0/604: stat d2/d27/d41/d76/d93/l43 0 2024-02-28T16:31:21.927 INFO:tasks.workunit.client.0.smithi001.stdout:7/848: rename d3/d17/d52/d7f/c91 to d3/d17/d52/d7f/d99/c110 0 2024-02-28T16:31:21.930 INFO:tasks.workunit.client.0.smithi001.stdout:0/814: dwrite d18/d22/d2b/d36/d46/df5/f7f [0,4194304] 0 2024-02-28T16:31:21.930 INFO:tasks.workunit.client.0.smithi001.stdout:6/808: link d9/db/d5f/d8b/l74 d9/d29/d6e/d9a/l10e 0 2024-02-28T16:31:21.932 INFO:tasks.workunit.client.1.smithi066.stdout:0/605: chown d2/d27/d41/d76/d93/d10/c7e 14732 1 2024-02-28T16:31:21.938 INFO:tasks.workunit.client.0.smithi001.stdout:2/784: read d0/d2/d3/de/d61/d12/d6a/f7e [48556,51569] 0 2024-02-28T16:31:21.943 INFO:tasks.workunit.client.1.smithi066.stdout:0/606: rename d2/d27/d41/d76/l84 to d2/d27/d41/dc5/lcc 0 2024-02-28T16:31:21.945 INFO:tasks.workunit.client.0.smithi001.stdout:7/849: truncate d3/f21 3978229 0 2024-02-28T16:31:21.945 INFO:tasks.workunit.client.1.smithi066.stdout:7/407: dread dc/de/d17/d37/d47/f5e [0,4194304] 0 2024-02-28T16:31:21.946 INFO:tasks.workunit.client.1.smithi066.stdout:7/408: stat dc/lf 0 2024-02-28T16:31:21.947 INFO:tasks.workunit.client.0.smithi001.stdout:7/850: readlink d3/d17/d52/d7f/d99/dc1/l105 0 2024-02-28T16:31:21.948 INFO:tasks.workunit.client.1.smithi066.stdout:0/607: creat d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/fcd x:0 0 0 2024-02-28T16:31:21.949 INFO:tasks.workunit.client.1.smithi066.stdout:7/409: write dc/d29/f2e [667232,43965] 0 2024-02-28T16:31:21.950 INFO:tasks.workunit.client.0.smithi001.stdout:5/798: rename dc/d12/f51 to dc/d12/d56/d96/db9/dc2/dca/d3c/f101 0 2024-02-28T16:31:21.950 INFO:tasks.workunit.client.0.smithi001.stdout:7/851: write d3/d17/d3d/d42/fee [574122,23570] 0 2024-02-28T16:31:21.952 INFO:tasks.workunit.client.1.smithi066.stdout:7/410: write dc/de/d17/d1e/f28 [3833507,20886] 0 2024-02-28T16:31:21.952 INFO:tasks.workunit.client.0.smithi001.stdout:6/809: write d9/db/d19/f1f [218067,107877] 0 2024-02-28T16:31:21.954 INFO:tasks.workunit.client.0.smithi001.stdout:0/815: rename d18/d22/d2b/f87 to d18/d5a/f109 0 2024-02-28T16:31:21.958 INFO:tasks.workunit.client.1.smithi066.stdout:7/411: rename dc/de/f65 to dc/d56/f77 0 2024-02-28T16:31:21.958 INFO:tasks.workunit.client.0.smithi001.stdout:5/799: symlink dc/d12/d56/d96/db9/l102 0 2024-02-28T16:31:21.960 INFO:tasks.workunit.client.0.smithi001.stdout:0/816: symlink d18/d22/d2f/d1c/d90/d9f/l10a 0 2024-02-28T16:31:21.966 INFO:tasks.workunit.client.0.smithi001.stdout:5/800: read dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/fe1 [1431763,22978] 0 2024-02-28T16:31:21.966 INFO:tasks.workunit.client.1.smithi066.stdout:0/608: dwrite d2/d27/d41/d76/d93/d10/f98 [0,4194304] 0 2024-02-28T16:31:21.970 INFO:tasks.workunit.client.1.smithi066.stdout:7/412: dread f5 [0,4194304] 0 2024-02-28T16:31:21.970 INFO:tasks.workunit.client.1.smithi066.stdout:0/609: write d2/d27/d41/d76/d93/d10/d23/fa0 [741713,16080] 0 2024-02-28T16:31:21.973 INFO:tasks.workunit.client.0.smithi001.stdout:5/801: readlink dc/d1a/d52/ld4 0 2024-02-28T16:31:21.974 INFO:tasks.workunit.client.1.smithi066.stdout:0/610: mknod d2/d85/da9/cce 0 2024-02-28T16:31:21.977 INFO:tasks.workunit.client.0.smithi001.stdout:0/817: link d18/d22/d2b/d36/d46/df5/d4f/ca3 d18/da9/c10b 0 2024-02-28T16:31:21.978 INFO:tasks.workunit.client.1.smithi066.stdout:0/611: mkdir d2/d27/d41/d76/dcf 0 2024-02-28T16:31:21.978 INFO:tasks.workunit.client.0.smithi001.stdout:5/802: write dc/d12/f14 [3975910,109482] 0 2024-02-28T16:31:21.980 INFO:tasks.workunit.client.1.smithi066.stdout:0/612: chown d2/d27/d41/d76/d93/db2/db5/d73/fc3 329316 1 2024-02-28T16:31:21.984 INFO:tasks.workunit.client.0.smithi001.stdout:7/852: dwrite d3/d17/d52/d7f/d99/faa [0,4194304] 0 2024-02-28T16:31:21.985 INFO:tasks.workunit.client.0.smithi001.stdout:0/818: mkdir d18/da9/dea/d10c 0 2024-02-28T16:31:21.987 INFO:tasks.workunit.client.1.smithi066.stdout:7/413: dwrite dc/de/d17/f5b [0,4194304] 0 2024-02-28T16:31:21.988 INFO:tasks.workunit.client.1.smithi066.stdout:0/613: rename d2/d27/d41/d76/d93/db2/db5/d52/f55 to d2/d85/d8d/fd0 0 2024-02-28T16:31:21.989 INFO:tasks.workunit.client.0.smithi001.stdout:2/785: dread d0/d2/d3/d36/d94/f96 [0,4194304] 0 2024-02-28T16:31:21.992 INFO:tasks.workunit.client.1.smithi066.stdout:7/414: chown dc/d29/c48 814333 1 2024-02-28T16:31:21.992 INFO:tasks.workunit.client.1.smithi066.stdout:7/415: chown f4 4799878 1 2024-02-28T16:31:21.994 INFO:tasks.workunit.client.0.smithi001.stdout:0/819: chown d18/d22/d2b/d36/d4d/lf7 64824144 1 2024-02-28T16:31:21.994 INFO:tasks.workunit.client.0.smithi001.stdout:6/810: dwrite d9/d29/d6e/d4e/fd1 [0,4194304] 0 2024-02-28T16:31:21.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:21 smithi066 ceph-mon[36236]: pgmap v54: 65 pgs: 65 active+clean; 2.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 83 MiB/s rd, 294 MiB/s wr, 552 op/s 2024-02-28T16:31:21.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:21 smithi066 ceph-mon[36236]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:21.999 INFO:tasks.workunit.client.1.smithi066.stdout:7/416: dread dc/de/d17/d1e/f3c [0,4194304] 0 2024-02-28T16:31:22.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.013+0000 7f129c800700 1 -- 172.21.15.1:0/3579774206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12940fbe40 msgr2=0x7f12940fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:22.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.013+0000 7f129c800700 1 --2- 172.21.15.1:0/3579774206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12940fbe40 0x7f12940fc250 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1290004c20 tx=0x7f1290009ca0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.002 INFO:tasks.workunit.client.0.smithi001.stdout:7/853: link d3/d17/d3d/d42/fee d3/d17/d52/d7f/d99/dc1/d49/d8d/f111 0 2024-02-28T16:31:22.003 INFO:tasks.workunit.client.0.smithi001.stdout:0/820: write d18/d22/d3d/fd6 [2162833,109135] 0 2024-02-28T16:31:22.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.015+0000 7f129c800700 1 -- 172.21.15.1:0/3579774206 shutdown_connections 2024-02-28T16:31:22.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.015+0000 7f129c800700 1 --2- 172.21.15.1:0/3579774206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12940fcfb0 0x7f12940fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.015+0000 7f129c800700 1 --2- 172.21.15.1:0/3579774206 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12940fbe40 0x7f12940fc250 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.004 INFO:tasks.workunit.client.0.smithi001.stdout:5/803: truncate dc/df/d79/f95 3173880 0 2024-02-28T16:31:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.015+0000 7f129c800700 1 -- 172.21.15.1:0/3579774206 >> 172.21.15.1:0/3579774206 conn(0x7f12940f74e0 msgr2=0x7f12940f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.016+0000 7f129c800700 1 -- 172.21.15.1:0/3579774206 shutdown_connections 2024-02-28T16:31:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.016+0000 7f129c800700 1 -- 172.21.15.1:0/3579774206 wait complete. 2024-02-28T16:31:22.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.017+0000 7f129c800700 1 Processor -- start 2024-02-28T16:31:22.006 INFO:tasks.workunit.client.1.smithi066.stdout:7/417: chown dc/d29/l55 31893220 1 2024-02-28T16:31:22.006 INFO:tasks.workunit.client.1.smithi066.stdout:7/418: write dc/de/d17/f5b [740908,26591] 0 2024-02-28T16:31:22.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.017+0000 7f129c800700 1 -- start start 2024-02-28T16:31:22.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.017+0000 7f129c800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12940fbe40 0x7f12940fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:22.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.017+0000 7f129c800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12940fcfb0 0x7f12940fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:22.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.017+0000 7f129c800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1294111450 con 0x7f12940fcfb0 2024-02-28T16:31:22.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.017+0000 7f129c800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941115c0 con 0x7f12940fbe40 2024-02-28T16:31:22.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.018+0000 7f1299400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12940fcfb0 0x7f12940fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:22.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.018+0000 7f1299400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12940fcfb0 0x7f12940fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56272/0 (socket says 172.21.15.1:56272) 2024-02-28T16:31:22.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.018+0000 7f1299400700 1 -- 172.21.15.1:0/4291618735 learned_addr learned my addr 172.21.15.1:0/4291618735 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:22.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.018+0000 7f1299e00700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12940fbe40 0x7f12940fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:22.007 INFO:tasks.workunit.client.0.smithi001.stdout:6/811: creat d9/db/f10f x:0 0 0 2024-02-28T16:31:22.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.018+0000 7f1299400700 1 -- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12940fbe40 msgr2=0x7f12940fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:22.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.018+0000 7f1299400700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12940fbe40 0x7f12940fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.018+0000 7f1299400700 1 -- 172.21.15.1:0/4291618735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12900048f0 con 0x7f12940fcfb0 2024-02-28T16:31:22.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.019+0000 7f1299400700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12940fcfb0 0x7f12940fe380 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1284002a50 tx=0x7f1284004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:22.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.019+0000 7f128e000700 1 -- 172.21.15.1:0/4291618735 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1284007cc0 con 0x7f12940fcfb0 2024-02-28T16:31:22.009 INFO:tasks.workunit.client.0.smithi001.stdout:6/812: chown d9/db/dd9/l108 13295 1 2024-02-28T16:31:22.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.019+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12940fe9e0 con 0x7f12940fcfb0 2024-02-28T16:31:22.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.019+0000 7f128e000700 1 -- 172.21.15.1:0/4291618735 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f128400f040 con 0x7f12940fcfb0 2024-02-28T16:31:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.019+0000 7f128e000700 1 -- 172.21.15.1:0/4291618735 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1284007cc0 con 0x7f12940fcfb0 2024-02-28T16:31:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.020+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12940fee60 con 0x7f12940fcfb0 2024-02-28T16:31:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.021+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129404bed0 con 0x7f12940fcfb0 2024-02-28T16:31:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.022+0000 7f128e000700 1 -- 172.21.15.1:0/4291618735 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 25) v1 ==== 88910+0+0 (secure 0 0 0) 0x7f12840097c0 con 0x7f12940fcfb0 2024-02-28T16:31:22.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.023+0000 7f128e000700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f127c06b220 0x7f127c06d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.024+0000 7f128e000700 1 -- 172.21.15.1:0/4291618735 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f1284007e20 con 0x7f12940fcfb0 2024-02-28T16:31:22.012 INFO:tasks.workunit.client.1.smithi066.stdout:0/614: dwrite d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/fcd [0,4194304] 0 2024-02-28T16:31:22.012 INFO:tasks.workunit.client.0.smithi001.stdout:5/804: rename dc/df/d79/d98/fe8 to dc/d12/d56/d96/db9/dc2/dca/d91/f103 0 2024-02-28T16:31:22.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.025+0000 7f1299e00700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f127c06b220 0x7f127c06d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:22.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.027+0000 7f1299e00700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f127c06b220 0x7f127c06d6d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f12940ffc90 tx=0x7f1290009ca0 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:22.015 INFO:tasks.workunit.client.1.smithi066.stdout:7/419: dread dc/d29/f2e [0,4194304] 0 2024-02-28T16:31:22.016 INFO:tasks.workunit.client.0.smithi001.stdout:5/805: write dc/d12/f32 [1071008,44370] 0 2024-02-28T16:31:22.018 INFO:tasks.workunit.client.1.smithi066.stdout:0/615: mkdir d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/dd1 0 2024-02-28T16:31:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.031+0000 7f128e000700 1 -- 172.21.15.1:0/4291618735 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f1284053780 con 0x7f12940fcfb0 2024-02-28T16:31:22.020 INFO:tasks.workunit.client.0.smithi001.stdout:1/937: sync 2024-02-28T16:31:22.020 INFO:tasks.workunit.client.0.smithi001.stdout:6/813: unlink d9/d29/d6e/d4e/d96/l9d 0 2024-02-28T16:31:22.020 INFO:tasks.workunit.client.1.smithi066.stdout:7/420: creat dc/de/d17/d37/d47/f78 x:0 0 0 2024-02-28T16:31:22.023 INFO:tasks.workunit.client.1.smithi066.stdout:7/421: write dc/d6e/f6f [776254,41508] 0 2024-02-28T16:31:22.025 INFO:tasks.workunit.client.1.smithi066.stdout:7/422: stat dc/d6e/c73 0 2024-02-28T16:31:22.025 INFO:tasks.workunit.client.0.smithi001.stdout:0/821: link d18/d22/d57/d7e/f96 d18/d22/d2b/f10d 0 2024-02-28T16:31:22.027 INFO:tasks.workunit.client.0.smithi001.stdout:2/786: dwrite d0/f33 [0,4194304] 0 2024-02-28T16:31:22.029 INFO:tasks.workunit.client.1.smithi066.stdout:0/616: rename d2/d27/d41/d76/d93/d10/d23/fa0 to d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/da1/fd2 0 2024-02-28T16:31:22.029 INFO:tasks.workunit.client.1.smithi066.stdout:7/423: truncate dc/de/d17/d1e/f59 1469546 0 2024-02-28T16:31:22.030 INFO:tasks.workunit.client.0.smithi001.stdout:5/806: mknod dc/d12/d56/d96/db9/dc2/dca/d91/c104 0 2024-02-28T16:31:22.031 INFO:tasks.workunit.client.1.smithi066.stdout:7/424: stat dc/lf 0 2024-02-28T16:31:22.034 INFO:tasks.workunit.client.1.smithi066.stdout:0/617: symlink d2/d27/d41/d76/d93/d10/ld3 0 2024-02-28T16:31:22.037 INFO:tasks.workunit.client.0.smithi001.stdout:6/814: rename d9/db/d19/f97 to d9/db/d5f/d8b/f110 0 2024-02-28T16:31:22.038 INFO:tasks.workunit.client.0.smithi001.stdout:0/822: creat d18/da9/f10e x:0 0 0 2024-02-28T16:31:22.038 INFO:tasks.workunit.client.1.smithi066.stdout:7/425: creat dc/de/d17/d1e/d45/f79 x:0 0 0 2024-02-28T16:31:22.040 INFO:tasks.workunit.client.1.smithi066.stdout:7/426: fdatasync dc/de/d17/f19 0 2024-02-28T16:31:22.040 INFO:tasks.workunit.client.0.smithi001.stdout:2/787: mknod d0/d2/d3/d73/c102 0 2024-02-28T16:31:22.043 INFO:tasks.workunit.client.0.smithi001.stdout:0/823: stat d18/d22/d57/d7e 0 2024-02-28T16:31:22.045 INFO:tasks.workunit.client.0.smithi001.stdout:6/815: truncate d9/d29/d6a/f10b 546080 0 2024-02-28T16:31:22.049 INFO:tasks.workunit.client.0.smithi001.stdout:7/854: dwrite d3/d17/d3d/db3/fd1 [0,4194304] 0 2024-02-28T16:31:22.051 INFO:tasks.workunit.client.0.smithi001.stdout:5/807: creat dc/d12/d56/d96/db9/f105 x:0 0 0 2024-02-28T16:31:22.052 INFO:tasks.workunit.client.0.smithi001.stdout:5/808: chown dc/c1b 186219837 1 2024-02-28T16:31:22.052 INFO:tasks.workunit.client.1.smithi066.stdout:7/427: truncate dc/f4b 1101814 0 2024-02-28T16:31:22.054 INFO:tasks.workunit.client.0.smithi001.stdout:6/816: creat d9/d29/d6e/d9a/f111 x:0 0 0 2024-02-28T16:31:22.066 INFO:tasks.workunit.client.0.smithi001.stdout:7/855: creat d3/d17/d3d/d59/d96/dc9/f112 x:0 0 0 2024-02-28T16:31:22.066 INFO:tasks.workunit.client.0.smithi001.stdout:5/809: mknod dc/d1a/d7d/d87/df3/c106 0 2024-02-28T16:31:22.068 INFO:tasks.workunit.client.1.smithi066.stdout:2/475: sync 2024-02-28T16:31:22.070 INFO:tasks.workunit.client.1.smithi066.stdout:7/428: dwrite dc/de/d17/d1e/f3c [0,4194304] 0 2024-02-28T16:31:22.070 INFO:tasks.workunit.client.0.smithi001.stdout:2/788: dread d0/d2/d3/f6c [0,4194304] 0 2024-02-28T16:31:22.072 INFO:tasks.workunit.client.1.smithi066.stdout:7/429: truncate dc/f5d 639168 0 2024-02-28T16:31:22.074 INFO:tasks.workunit.client.0.smithi001.stdout:3/836: sync 2024-02-28T16:31:22.075 INFO:tasks.workunit.client.0.smithi001.stdout:8/812: sync 2024-02-28T16:31:22.076 INFO:tasks.workunit.client.1.smithi066.stdout:2/476: mknod d0/d12/d1a/d39/d7c/d87/c93 0 2024-02-28T16:31:22.077 INFO:tasks.workunit.client.0.smithi001.stdout:4/926: sync 2024-02-28T16:31:22.079 INFO:tasks.workunit.client.0.smithi001.stdout:8/813: chown d4/d7/da8/ce9 2179 1 2024-02-28T16:31:22.080 INFO:tasks.workunit.client.1.smithi066.stdout:2/477: creat d0/d12/d1a/d39/d7c/d87/f94 x:0 0 0 2024-02-28T16:31:22.082 INFO:tasks.workunit.client.0.smithi001.stdout:0/824: write d18/f33 [534725,93140] 0 2024-02-28T16:31:22.084 INFO:tasks.workunit.client.1.smithi066.stdout:7/430: getdents dc/de/d17/d37 0 2024-02-28T16:31:22.084 INFO:tasks.workunit.client.1.smithi066.stdout:2/478: rmdir d0/d12/d1a/d23/d29/d21 39 2024-02-28T16:31:22.086 INFO:tasks.workunit.client.1.smithi066.stdout:2/479: fdatasync d0/d12/d1a/f79 0 2024-02-28T16:31:22.088 INFO:tasks.workunit.client.1.smithi066.stdout:2/480: fdatasync d0/d12/d2e/d7a/f5b 0 2024-02-28T16:31:22.089 INFO:tasks.workunit.client.0.smithi001.stdout:0/825: chown d18/d22/d2b/d36/d48/c64 513901959 1 2024-02-28T16:31:22.089 INFO:tasks.workunit.client.0.smithi001.stdout:4/927: creat d3/d5/d7/ddb/f12f x:0 0 0 2024-02-28T16:31:22.093 INFO:tasks.workunit.client.0.smithi001.stdout:8/814: mknod d4/d7/d22/d27/d4b/d77/c107 0 2024-02-28T16:31:22.094 INFO:tasks.workunit.client.1.smithi066.stdout:2/481: mknod d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/c95 0 2024-02-28T16:31:22.096 INFO:tasks.workunit.client.0.smithi001.stdout:0/826: creat d18/d22/d2f/d1c/f10f x:0 0 0 2024-02-28T16:31:22.097 INFO:tasks.workunit.client.0.smithi001.stdout:6/817: dread d9/d29/d6a/ded/f10a [0,4194304] 0 2024-02-28T16:31:22.097 INFO:tasks.workunit.client.0.smithi001.stdout:5/810: dread f2 [0,4194304] 0 2024-02-28T16:31:22.099 INFO:tasks.workunit.client.1.smithi066.stdout:7/431: dwrite f4 [0,4194304] 0 2024-02-28T16:31:22.102 INFO:tasks.workunit.client.1.smithi066.stdout:7/432: symlink dc/de/d17/d37/l7a 0 2024-02-28T16:31:22.103 INFO:tasks.workunit.client.0.smithi001.stdout:4/928: rename d3/d5/ccb to d3/d5/d92/c130 0 2024-02-28T16:31:22.104 INFO:tasks.workunit.client.0.smithi001.stdout:2/789: dwrite d0/d2/d3/d36/f4b [0,4194304] 0 2024-02-28T16:31:22.104 INFO:tasks.workunit.client.0.smithi001.stdout:8/815: rmdir d4/d7/d22/d27/d58/d94/dc9 39 2024-02-28T16:31:22.108 INFO:tasks.workunit.client.0.smithi001.stdout:5/811: write dc/d12/f32 [4741353,60895] 0 2024-02-28T16:31:22.108 INFO:tasks.workunit.client.0.smithi001.stdout:1/938: dread d0/d9/ff [0,4194304] 0 2024-02-28T16:31:22.110 INFO:tasks.workunit.client.1.smithi066.stdout:2/482: dwrite d0/d12/d2e/f59 [0,4194304] 0 2024-02-28T16:31:22.113 INFO:tasks.workunit.client.0.smithi001.stdout:4/929: mknod d3/d1c/d58/d6e/c131 0 2024-02-28T16:31:22.114 INFO:tasks.workunit.client.1.smithi066.stdout:7/433: dread fa [0,4194304] 0 2024-02-28T16:31:22.118 INFO:tasks.workunit.client.1.smithi066.stdout:2/483: rename d0/d12/d2e/d3b/l6a to d0/d12/d1a/d39/l96 0 2024-02-28T16:31:22.119 INFO:tasks.workunit.client.0.smithi001.stdout:1/939: write d0/d9/d1a/d2b/d26/d27/d30/d67/db7/f10f [310041,16091] 0 2024-02-28T16:31:22.122 INFO:tasks.workunit.client.1.smithi066.stdout:7/434: unlink dc/de/d17/d1e/l61 0 2024-02-28T16:31:22.123 INFO:tasks.workunit.client.1.smithi066.stdout:7/435: readlink dc/l64 0 2024-02-28T16:31:22.124 INFO:tasks.workunit.client.1.smithi066.stdout:2/484: creat d0/d12/d1a/d23/d29/d21/d2f/d67/f97 x:0 0 0 2024-02-28T16:31:22.124 INFO:tasks.workunit.client.1.smithi066.stdout:7/436: chown dc/de/l2f 256555657 1 2024-02-28T16:31:22.126 INFO:tasks.workunit.client.0.smithi001.stdout:5/812: symlink dc/d12/d41/db7/l107 0 2024-02-28T16:31:22.127 INFO:tasks.workunit.client.1.smithi066.stdout:7/437: mknod dc/de/d17/d1e/d45/c7b 0 2024-02-28T16:31:22.128 INFO:tasks.workunit.client.0.smithi001.stdout:7/856: dread d3/d17/d3d/db3/fb4 [0,4194304] 0 2024-02-28T16:31:22.134 INFO:tasks.workunit.client.0.smithi001.stdout:6/818: rename d9/fdb to d9/db/d10/f112 0 2024-02-28T16:31:22.135 INFO:tasks.workunit.client.0.smithi001.stdout:7/857: write d3/d17/d3d/db3/fb4 [3894469,57053] 0 2024-02-28T16:31:22.136 INFO:tasks.workunit.client.0.smithi001.stdout:2/790: creat d0/d2/f103 x:0 0 0 2024-02-28T16:31:22.137 INFO:tasks.workunit.client.0.smithi001.stdout:7/858: stat d3/d17/d61/faf 0 2024-02-28T16:31:22.140 INFO:tasks.workunit.client.0.smithi001.stdout:4/930: mknod d3/d2d/d81/d113/c132 0 2024-02-28T16:31:22.142 INFO:tasks.workunit.client.0.smithi001.stdout:4/931: chown d3/d5/d7/d1f 592169277 1 2024-02-28T16:31:22.143 INFO:tasks.workunit.client.1.smithi066.stdout:2/485: dwrite d0/d12/d1a/d39/d7c/d87/f94 [0,4194304] 0 2024-02-28T16:31:22.145 INFO:tasks.workunit.client.1.smithi066.stdout:7/438: dwrite dc/d56/f77 [0,4194304] 0 2024-02-28T16:31:22.146 INFO:tasks.workunit.client.0.smithi001.stdout:5/813: fsync dc/d12/d56/d96/db9/dc2/dca/d91/f103 0 2024-02-28T16:31:22.148 INFO:tasks.workunit.client.0.smithi001.stdout:6/819: creat d9/de/dbd/f113 x:0 0 0 2024-02-28T16:31:22.152 INFO:tasks.workunit.client.0.smithi001.stdout:4/932: creat d3/d5/d7/d1f/d2e/dbe/d115/f133 x:0 0 0 2024-02-28T16:31:22.153 INFO:tasks.workunit.client.0.smithi001.stdout:5/814: write dc/d12/d56/d96/db9/dc2/dca/f9d [2151822,119135] 0 2024-02-28T16:31:22.156 INFO:tasks.workunit.client.0.smithi001.stdout:1/940: link d0/d9/d1a/d2b/d26/f11e d0/d9/d1a/d2b/d26/d27/dce/f147 0 2024-02-28T16:31:22.156 INFO:tasks.workunit.client.0.smithi001.stdout:6/820: write d9/d29/d6a/f10b [980788,41143] 0 2024-02-28T16:31:22.163 INFO:tasks.workunit.client.0.smithi001.stdout:4/933: chown d3/d5/d7/d1f/d2e/dbe/d87 950496 1 2024-02-28T16:31:22.164 INFO:tasks.workunit.client.0.smithi001.stdout:4/934: read - d3/d2d/d4c/d69/f128 zero size 2024-02-28T16:31:22.166 INFO:tasks.workunit.client.1.smithi066.stdout:2/486: dwrite d0/d12/d2e/d3b/d66/f7f [0,4194304] 0 2024-02-28T16:31:22.168 INFO:tasks.workunit.client.0.smithi001.stdout:8/816: dwrite d4/d7/d22/d27/d4b/fa3 [0,4194304] 0 2024-02-28T16:31:22.171 INFO:tasks.workunit.client.1.smithi066.stdout:2/487: symlink d0/d12/d1a/d23/l98 0 2024-02-28T16:31:22.171 INFO:tasks.workunit.client.1.smithi066.stdout:7/439: dwrite dc/f25 [0,4194304] 0 2024-02-28T16:31:22.174 INFO:tasks.workunit.client.0.smithi001.stdout:0/827: dwrite d18/d22/d2b/d36/d46/df5/f43 [0,4194304] 0 2024-02-28T16:31:22.174 INFO:tasks.workunit.client.1.smithi066.stdout:7/440: creat dc/de/d17/f7c x:0 0 0 2024-02-28T16:31:22.179 INFO:tasks.workunit.client.0.smithi001.stdout:7/859: write d3/d17/f3b [4139925,2291] 0 2024-02-28T16:31:22.181 INFO:tasks.workunit.client.1.smithi066.stdout:2/488: link d0/d12/d3e/f4e d0/d12/d2e/d7a/f99 0 2024-02-28T16:31:22.185 INFO:tasks.workunit.client.0.smithi001.stdout:7/860: write d3/d17/d3d/d5d/fc7 [1310232,55151] 0 2024-02-28T16:31:22.187 INFO:tasks.workunit.client.0.smithi001.stdout:4/935: symlink d3/d1c/d58/d6e/l134 0 2024-02-28T16:31:22.191 INFO:tasks.workunit.client.0.smithi001.stdout:6/821: dwrite d9/db/d19/ff9 [0,4194304] 0 2024-02-28T16:31:22.197 INFO:tasks.workunit.client.1.smithi066.stdout:2/489: rename d0/d12/d1a/d23/d29/d21/f89 to d0/d12/d1a/f9a 0 2024-02-28T16:31:22.198 INFO:tasks.workunit.client.0.smithi001.stdout:0/828: rmdir d18/d22/d2f/d1c/d79 39 2024-02-28T16:31:22.200 INFO:tasks.workunit.client.1.smithi066.stdout:2/490: write d0/d12/f5c [51076,59039] 0 2024-02-28T16:31:22.200 INFO:tasks.workunit.client.0.smithi001.stdout:8/817: truncate d4/d7/f21 564677 0 2024-02-28T16:31:22.201 INFO:tasks.workunit.client.0.smithi001.stdout:7/861: creat d3/d17/d61/f113 x:0 0 0 2024-02-28T16:31:22.202 INFO:tasks.workunit.client.0.smithi001.stdout:1/941: dread d0/d9/d1a/d2b/dfe/ff0 [0,4194304] 0 2024-02-28T16:31:22.203 INFO:tasks.workunit.client.0.smithi001.stdout:4/936: creat d3/d1c/da0/f135 x:0 0 0 2024-02-28T16:31:22.206 INFO:tasks.workunit.client.0.smithi001.stdout:6/822: mknod d9/db/d5f/d62/d83/c114 0 2024-02-28T16:31:22.207 INFO:tasks.workunit.client.1.smithi066.stdout:2/491: stat d0/f71 0 2024-02-28T16:31:22.215 INFO:tasks.workunit.client.0.smithi001.stdout:2/791: dwrite d0/f7f [0,4194304] 0 2024-02-28T16:31:22.219 INFO:tasks.workunit.client.0.smithi001.stdout:5/815: getdents dc/d12/d56/d96/db9/dc2/dca/d3c 0 2024-02-28T16:31:22.220 INFO:tasks.workunit.client.0.smithi001.stdout:7/862: dread - d3/d17/d52/d55/dca/fce zero size 2024-02-28T16:31:22.220 INFO:tasks.workunit.client.0.smithi001.stdout:8/818: truncate d4/d32/d69/d9a/f105 826020 0 2024-02-28T16:31:22.224 INFO:tasks.workunit.client.1.smithi066.stdout:7/441: dread dc/de/d17/d1e/f28 [0,4194304] 0 2024-02-28T16:31:22.226 INFO:tasks.workunit.client.0.smithi001.stdout:0/829: truncate d18/d22/d2b/d36/d46/df5/f9e 247468 0 2024-02-28T16:31:22.226 INFO:tasks.workunit.client.1.smithi066.stdout:2/492: dwrite d0/d12/f5c [0,4194304] 0 2024-02-28T16:31:22.227 INFO:tasks.workunit.client.1.smithi066.stdout:7/442: chown dc/c15 0 1 2024-02-28T16:31:22.228 INFO:tasks.workunit.client.0.smithi001.stdout:8/819: mkdir d4/d47/d8b/d108 0 2024-02-28T16:31:22.229 INFO:tasks.workunit.client.1.smithi066.stdout:2/493: fsync d0/d12/d2e/d3b/f64 0 2024-02-28T16:31:22.229 INFO:tasks.workunit.client.1.smithi066.stdout:1/456: sync 2024-02-28T16:31:22.232 INFO:tasks.workunit.client.0.smithi001.stdout:1/942: rename d0/d9/d1a/d2b/f4d to d0/d9/f148 0 2024-02-28T16:31:22.233 INFO:tasks.workunit.client.0.smithi001.stdout:0/830: dread - d18/d22/d76/fd1 zero size 2024-02-28T16:31:22.233 INFO:tasks.workunit.client.1.smithi066.stdout:7/443: symlink dc/de/l7d 0 2024-02-28T16:31:22.235 INFO:tasks.workunit.client.1.smithi066.stdout:2/494: chown d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/c95 25293643 1 2024-02-28T16:31:22.239 INFO:tasks.workunit.client.0.smithi001.stdout:7/863: link d3/d17/d52/f77 d3/d17/d52/d92/f114 0 2024-02-28T16:31:22.239 INFO:tasks.workunit.client.0.smithi001.stdout:8/820: chown d4/d32/d69/d9a/dcd/ldf 0 1 2024-02-28T16:31:22.248 INFO:tasks.workunit.client.1.smithi066.stdout:7/444: dread dc/de/d17/d1e/f3e [0,4194304] 0 2024-02-28T16:31:22.249 INFO:tasks.workunit.client.0.smithi001.stdout:4/937: dwrite d3/d5/d7/d1f/d2e/dbe/d115/f129 [0,4194304] 0 2024-02-28T16:31:22.251 INFO:tasks.workunit.client.1.smithi066.stdout:7/445: mknod dc/c7e 0 2024-02-28T16:31:22.251 INFO:tasks.workunit.client.0.smithi001.stdout:6/823: dwrite d9/d29/d6e/d53/f7a [0,4194304] 0 2024-02-28T16:31:22.253 INFO:tasks.workunit.client.1.smithi066.stdout:7/446: readlink dc/de/d17/d1e/d45/l57 0 2024-02-28T16:31:22.254 INFO:tasks.workunit.client.1.smithi066.stdout:7/447: readlink dc/d56/l69 0 2024-02-28T16:31:22.257 INFO:tasks.workunit.client.1.smithi066.stdout:7/448: unlink dc/d29/d5c/c67 0 2024-02-28T16:31:22.261 INFO:tasks.workunit.client.1.smithi066.stdout:1/457: truncate d2/d27/d2b/d37/d51/f62 1024269 0 2024-02-28T16:31:22.265 INFO:tasks.workunit.client.1.smithi066.stdout:7/449: unlink dc/l14 0 2024-02-28T16:31:22.266 INFO:tasks.workunit.client.1.smithi066.stdout:1/458: mkdir d2/d18/d7a/d8d 0 2024-02-28T16:31:22.266 INFO:tasks.workunit.client.1.smithi066.stdout:7/450: truncate f4 4348090 0 2024-02-28T16:31:22.267 INFO:tasks.workunit.client.0.smithi001.stdout:5/816: dread dc/df/f13 [0,4194304] 0 2024-02-28T16:31:22.267 INFO:tasks.workunit.client.0.smithi001.stdout:2/792: dwrite d0/d2/d3/d73/f85 [0,4194304] 0 2024-02-28T16:31:22.268 INFO:tasks.workunit.client.1.smithi066.stdout:7/451: dread - dc/de/d17/d37/d47/f78 zero size 2024-02-28T16:31:22.269 INFO:tasks.workunit.client.0.smithi001.stdout:7/864: dwrite d3/d17/d3d/d42/d8e/f9b [0,4194304] 0 2024-02-28T16:31:22.271 INFO:tasks.workunit.client.0.smithi001.stdout:8/821: creat d4/d7/d22/d27/d4b/d84/f109 x:0 0 0 2024-02-28T16:31:22.273 INFO:tasks.workunit.client.1.smithi066.stdout:7/452: creat dc/f7f x:0 0 0 2024-02-28T16:31:22.277 INFO:tasks.workunit.client.0.smithi001.stdout:0/831: dwrite d18/d22/d76/d7a/fc4 [0,4194304] 0 2024-02-28T16:31:22.277 INFO:tasks.workunit.client.0.smithi001.stdout:5/817: dread dc/df/f13 [0,4194304] 0 2024-02-28T16:31:22.278 INFO:tasks.workunit.client.1.smithi066.stdout:7/453: rename dc/de/d17/f60 to dc/d56/f80 0 2024-02-28T16:31:22.278 INFO:tasks.workunit.client.0.smithi001.stdout:7/865: rmdir d3/d1c 39 2024-02-28T16:31:22.279 INFO:tasks.workunit.client.1.smithi066.stdout:7/454: fdatasync dc/d29/d5c/f66 0 2024-02-28T16:31:22.281 INFO:tasks.workunit.client.1.smithi066.stdout:7/455: write f5 [180159,56420] 0 2024-02-28T16:31:22.282 INFO:tasks.workunit.client.0.smithi001.stdout:2/793: chown d0/d2/d3/de/c95 254 1 2024-02-28T16:31:22.284 INFO:tasks.workunit.client.0.smithi001.stdout:8/822: chown d4/c1b 40492 1 2024-02-28T16:31:22.287 INFO:tasks.workunit.client.0.smithi001.stdout:0/832: fsync d18/d22/d2f/d1c/d90/f100 0 2024-02-28T16:31:22.289 INFO:tasks.workunit.client.0.smithi001.stdout:0/833: stat d18/d22/d2b/d36/d46/df5/f71 0 2024-02-28T16:31:22.296 INFO:tasks.workunit.client.0.smithi001.stdout:0/834: read - d18/da9/f102 zero size 2024-02-28T16:31:22.300 INFO:tasks.workunit.client.0.smithi001.stdout:5/818: creat dc/f108 x:0 0 0 2024-02-28T16:31:22.302 INFO:tasks.workunit.client.0.smithi001.stdout:7/866: mkdir d3/d17/d52/d7f/d99/d115 0 2024-02-28T16:31:22.308 INFO:tasks.workunit.client.0.smithi001.stdout:7/867: read d3/d17/d52/f77 [459709,64753] 0 2024-02-28T16:31:22.308 INFO:tasks.workunit.client.1.smithi066.stdout:7/456: dwrite dc/de/d17/d1e/d45/f79 [0,4194304] 0 2024-02-28T16:31:22.313 INFO:tasks.workunit.client.0.smithi001.stdout:7/868: creat d3/d17/d3d/d59/d6e/de2/f116 x:0 0 0 2024-02-28T16:31:22.313 INFO:tasks.workunit.client.1.smithi066.stdout:1/459: dwrite d2/d18/f3a [0,4194304] 0 2024-02-28T16:31:22.314 INFO:tasks.workunit.client.0.smithi001.stdout:6/824: truncate d9/d39/f3a 1989775 0 2024-02-28T16:31:22.315 INFO:tasks.workunit.client.0.smithi001.stdout:7/869: fdatasync d3/d17/d52/d7f/d99/dc1/f6d 0 2024-02-28T16:31:22.319 INFO:tasks.workunit.client.1.smithi066.stdout:1/460: readlink d2/d27/d2b/d37/d54/l5c 0 2024-02-28T16:31:22.319 INFO:tasks.workunit.client.0.smithi001.stdout:6/825: symlink d9/d29/d6e/d53/l115 0 2024-02-28T16:31:22.320 INFO:tasks.workunit.client.1.smithi066.stdout:7/457: link dc/de/d17/d1e/f53 dc/de/d17/d37/d47/f81 0 2024-02-28T16:31:22.322 INFO:tasks.workunit.client.1.smithi066.stdout:1/461: creat d2/d27/d2b/d43/f8e x:0 0 0 2024-02-28T16:31:22.324 INFO:tasks.workunit.client.0.smithi001.stdout:8/823: dwrite d4/d7/d22/d27/f60 [0,4194304] 0 2024-02-28T16:31:22.337 INFO:tasks.workunit.client.1.smithi066.stdout:7/458: dwrite dc/de/d17/d37/f72 [0,4194304] 0 2024-02-28T16:31:22.345 INFO:tasks.workunit.client.0.smithi001.stdout:7/870: dwrite d3/d17/f10e [0,4194304] 0 2024-02-28T16:31:22.347 INFO:tasks.workunit.client.1.smithi066.stdout:4/601: sync 2024-02-28T16:31:22.347 INFO:tasks.workunit.client.1.smithi066.stdout:3/496: sync 2024-02-28T16:31:22.347 INFO:tasks.workunit.client.1.smithi066.stdout:0/618: sync 2024-02-28T16:31:22.347 INFO:tasks.workunit.client.1.smithi066.stdout:9/373: sync 2024-02-28T16:31:22.347 INFO:tasks.workunit.client.1.smithi066.stdout:8/440: sync 2024-02-28T16:31:22.347 INFO:tasks.workunit.client.1.smithi066.stdout:5/462: sync 2024-02-28T16:31:22.347 INFO:tasks.workunit.client.1.smithi066.stdout:6/403: sync 2024-02-28T16:31:22.350 INFO:tasks.workunit.client.1.smithi066.stdout:9/374: write da/f64 [949763,42778] 0 2024-02-28T16:31:22.353 INFO:tasks.workunit.client.1.smithi066.stdout:3/497: symlink db/la8 0 2024-02-28T16:31:22.353 INFO:tasks.workunit.client.1.smithi066.stdout:6/404: rmdir d2/d2c 39 2024-02-28T16:31:22.355 INFO:tasks.workunit.client.1.smithi066.stdout:4/602: mknod d1/d23/d25/d3e/cc3 0 2024-02-28T16:31:22.357 INFO:tasks.workunit.client.1.smithi066.stdout:8/441: creat d0/d6/dc/d16/d4c/d6c/f8b x:0 0 0 2024-02-28T16:31:22.357 INFO:tasks.workunit.client.1.smithi066.stdout:0/619: write d2/d27/d41/d76/d93/d10/d23/d6d/faf [265379,21249] 0 2024-02-28T16:31:22.359 INFO:tasks.workunit.client.0.smithi001.stdout:0/835: dread f13 [0,4194304] 0 2024-02-28T16:31:22.364 INFO:tasks.workunit.client.1.smithi066.stdout:3/498: rename db/d13/d1a/f1e to db/d2a/d44/d48/d5d/fa9 0 2024-02-28T16:31:22.365 INFO:tasks.workunit.client.0.smithi001.stdout:6/826: dread d9/d29/d6e/d53/f9e [0,4194304] 0 2024-02-28T16:31:22.367 INFO:tasks.workunit.client.0.smithi001.stdout:6/827: chown d9/db/f88 1592 1 2024-02-28T16:31:22.368 INFO:tasks.workunit.client.1.smithi066.stdout:5/463: link d3/d4/dc/d2f/f32 d3/d4/dc/dd/d2e/f90 0 2024-02-28T16:31:22.369 INFO:tasks.workunit.client.0.smithi001.stdout:6/828: write d9/fab [4374428,80525] 0 2024-02-28T16:31:22.370 INFO:tasks.workunit.client.1.smithi066.stdout:4/603: rmdir d1/d18/d41/dba 39 2024-02-28T16:31:22.371 INFO:tasks.workunit.client.0.smithi001.stdout:6/829: readlink d9/db/d5f/d89/ld2 0 2024-02-28T16:31:22.372 INFO:tasks.workunit.client.1.smithi066.stdout:5/464: dread - d3/d4/dc/d42/f73 zero size 2024-02-28T16:31:22.373 INFO:tasks.workunit.client.0.smithi001.stdout:8/824: read - d4/d7/d22/d27/d58/d94/db4/dd5/fd7 zero size 2024-02-28T16:31:22.374 INFO:tasks.workunit.client.1.smithi066.stdout:3/499: mkdir db/dc/d11/daa 0 2024-02-28T16:31:22.374 INFO:tasks.workunit.client.0.smithi001.stdout:0/836: dread d18/d22/f3f [0,4194304] 0 2024-02-28T16:31:22.376 INFO:tasks.workunit.client.1.smithi066.stdout:4/604: write d1/d18/d41/f68 [42704,116796] 0 2024-02-28T16:31:22.381 INFO:tasks.workunit.client.0.smithi001.stdout:8/825: read d4/d7/f10 [515293,51555] 0 2024-02-28T16:31:22.381 INFO:tasks.workunit.client.1.smithi066.stdout:5/465: fdatasync d3/d4/dc/dd/f89 0 2024-02-28T16:31:22.384 INFO:tasks.workunit.client.1.smithi066.stdout:6/405: dwrite d2/d2c/f51 [0,4194304] 0 2024-02-28T16:31:22.386 INFO:tasks.workunit.client.1.smithi066.stdout:9/375: dwrite da/df/d18/d28/d63/f5c [0,4194304] 0 2024-02-28T16:31:22.388 INFO:tasks.workunit.client.1.smithi066.stdout:9/376: write da/f2a [3407550,59096] 0 2024-02-28T16:31:22.391 INFO:tasks.workunit.client.1.smithi066.stdout:5/466: dread d3/d4/dc/d56/f75 [0,4194304] 0 2024-02-28T16:31:22.391 INFO:tasks.workunit.client.1.smithi066.stdout:4/605: mkdir d1/d18/d55/dc4 0 2024-02-28T16:31:22.391 INFO:tasks.workunit.client.0.smithi001.stdout:8/826: creat d4/d47/f10a x:0 0 0 2024-02-28T16:31:22.392 INFO:tasks.workunit.client.0.smithi001.stdout:8/827: readlink d4/d7/d22/l3c 0 2024-02-28T16:31:22.393 INFO:tasks.workunit.client.1.smithi066.stdout:4/606: fdatasync d1/d23/d25/f59 0 2024-02-28T16:31:22.394 INFO:tasks.workunit.client.1.smithi066.stdout:6/406: creat d2/d2c/d82/f8a x:0 0 0 2024-02-28T16:31:22.400 INFO:tasks.workunit.client.1.smithi066.stdout:3/500: rename db/d2a/d44/f6d to db/d13/d1a/fab 0 2024-02-28T16:31:22.402 INFO:tasks.workunit.client.0.smithi001.stdout:0/837: dwrite d18/d5a/f109 [0,4194304] 0 2024-02-28T16:31:22.403 INFO:tasks.workunit.client.1.smithi066.stdout:5/467: fsync d3/d4/dc/f1d 0 2024-02-28T16:31:22.406 INFO:tasks.workunit.client.1.smithi066.stdout:3/501: truncate db/dc/d11/d1f/d78/fa4 493771 0 2024-02-28T16:31:22.407 INFO:tasks.workunit.client.1.smithi066.stdout:6/407: unlink d2/d26/d32/d46/l6d 0 2024-02-28T16:31:22.408 INFO:tasks.workunit.client.1.smithi066.stdout:4/607: creat d1/d18/d4e/da0/fc5 x:0 0 0 2024-02-28T16:31:22.409 INFO:tasks.workunit.client.0.smithi001.stdout:6/830: write d9/d39/f3a [1329777,41730] 0 2024-02-28T16:31:22.413 INFO:tasks.workunit.client.1.smithi066.stdout:3/502: write db/dc/d11/d1f/d78/fa1 [371162,35483] 0 2024-02-28T16:31:22.414 INFO:tasks.workunit.client.1.smithi066.stdout:5/468: getdents d3/d4/dc/dd/d41/d7f/d80 0 2024-02-28T16:31:22.416 INFO:tasks.workunit.client.1.smithi066.stdout:6/408: symlink d2/d26/l8b 0 2024-02-28T16:31:22.418 INFO:tasks.workunit.client.1.smithi066.stdout:5/469: mknod d3/c91 0 2024-02-28T16:31:22.420 INFO:tasks.workunit.client.0.smithi001.stdout:7/871: dwrite d3/f5f [0,4194304] 0 2024-02-28T16:31:22.421 INFO:tasks.workunit.client.0.smithi001.stdout:8/828: dwrite d4/d7/d22/d27/d4b/fa4 [0,4194304] 0 2024-02-28T16:31:22.431 INFO:tasks.workunit.client.0.smithi001.stdout:7/872: creat d3/d1c/f117 x:0 0 0 2024-02-28T16:31:22.431 INFO:tasks.workunit.client.0.smithi001.stdout:6/831: dwrite d9/d39/f3a [0,4194304] 0 2024-02-28T16:31:22.436 INFO:tasks.workunit.client.0.smithi001.stdout:8/829: rename d4/d32/d93/lc5 to d4/d7/da8/d89/l10b 0 2024-02-28T16:31:22.440 INFO:tasks.workunit.client.0.smithi001.stdout:3/837: sync 2024-02-28T16:31:22.441 INFO:tasks.workunit.client.0.smithi001.stdout:6/832: read d9/db/d5f/d89/d8a/fe1 [3357380,91296] 0 2024-02-28T16:31:22.443 INFO:tasks.workunit.client.1.smithi066.stdout:4/608: dwrite d1/d23/d25/f6b [0,4194304] 0 2024-02-28T16:31:22.443 INFO:tasks.workunit.client.1.smithi066.stdout:6/409: dwrite d2/da/d19/f50 [0,4194304] 0 2024-02-28T16:31:22.449 INFO:tasks.workunit.client.0.smithi001.stdout:6/833: unlink d9/de/dbd/f113 0 2024-02-28T16:31:22.453 INFO:tasks.workunit.client.0.smithi001.stdout:3/838: symlink d4/d6/d7/d23/l115 0 2024-02-28T16:31:22.453 INFO:tasks.workunit.client.1.smithi066.stdout:6/410: creat d2/d2c/f8c x:0 0 0 2024-02-28T16:31:22.455 INFO:tasks.workunit.client.1.smithi066.stdout:6/411: chown d2/d2c/d72 0 1 2024-02-28T16:31:22.456 INFO:tasks.workunit.client.1.smithi066.stdout:5/470: write d3/d4/f12 [2649472,40128] 0 2024-02-28T16:31:22.460 INFO:tasks.workunit.client.1.smithi066.stdout:6/412: write d2/d26/f4e [4479653,60344] 0 2024-02-28T16:31:22.462 INFO:tasks.workunit.client.0.smithi001.stdout:7/873: dwrite d3/d1c/f51 [0,4194304] 0 2024-02-28T16:31:22.462 INFO:tasks.workunit.client.1.smithi066.stdout:6/413: readlink d2/da/d19/l4b 0 2024-02-28T16:31:22.464 INFO:tasks.workunit.client.1.smithi066.stdout:9/377: dread da/d3a/d15/f19 [0,4194304] 0 2024-02-28T16:31:22.473 INFO:tasks.workunit.client.1.smithi066.stdout:6/414: creat d2/da/d19/d75/d88/f8d x:0 0 0 2024-02-28T16:31:22.473 INFO:tasks.workunit.client.0.smithi001.stdout:7/874: truncate d3/d17/d52/d7f/d99/faa 4606755 0 2024-02-28T16:31:22.474 INFO:tasks.workunit.client.1.smithi066.stdout:5/471: getdents d3/d4/dc/dd/d41 0 2024-02-28T16:31:22.474 INFO:tasks.workunit.client.0.smithi001.stdout:8/830: write d4/d47/d8b/fe5 [9218558,62347] 0 2024-02-28T16:31:22.477 INFO:tasks.workunit.client.1.smithi066.stdout:9/378: symlink da/d3a/l7e 0 2024-02-28T16:31:22.477 INFO:tasks.workunit.client.1.smithi066.stdout:6/415: creat d2/d2c/f8e x:0 0 0 2024-02-28T16:31:22.478 INFO:tasks.workunit.client.1.smithi066.stdout:9/379: stat da/df/f2f 0 2024-02-28T16:31:22.481 INFO:tasks.workunit.client.0.smithi001.stdout:3/839: dread d4/d6/f56 [0,4194304] 0 2024-02-28T16:31:22.482 INFO:tasks.workunit.client.0.smithi001.stdout:6/834: dread d9/db/d19/f78 [0,4194304] 0 2024-02-28T16:31:22.485 INFO:tasks.workunit.client.0.smithi001.stdout:8/831: readlink d4/d32/l34 0 2024-02-28T16:31:22.486 INFO:tasks.workunit.client.1.smithi066.stdout:2/495: sync 2024-02-28T16:31:22.490 INFO:tasks.workunit.client.1.smithi066.stdout:2/496: stat d0/d12/d3e/l83 0 2024-02-28T16:31:22.492 INFO:tasks.workunit.client.0.smithi001.stdout:6/835: read - d9/d29/f48 zero size 2024-02-28T16:31:22.492 INFO:tasks.workunit.client.1.smithi066.stdout:4/609: write d1/d23/d25/d43/f5e [631531,64387] 0 2024-02-28T16:31:22.497 INFO:tasks.workunit.client.0.smithi001.stdout:3/840: creat d4/d1e/d7c/d7d/df8/f116 x:0 0 0 2024-02-28T16:31:22.501 INFO:tasks.workunit.client.1.smithi066.stdout:2/497: read d0/f1e [376531,87090] 0 2024-02-28T16:31:22.501 INFO:tasks.workunit.client.1.smithi066.stdout:4/610: write d1/d7a/f90 [136727,92429] 0 2024-02-28T16:31:22.501 INFO:tasks.workunit.client.0.smithi001.stdout:6/836: creat d9/db/dd9/f116 x:0 0 0 2024-02-28T16:31:22.505 INFO:tasks.workunit.client.1.smithi066.stdout:3/503: dread db/d13/f19 [0,4194304] 0 2024-02-28T16:31:22.509 INFO:tasks.workunit.client.0.smithi001.stdout:3/841: creat d4/d1e/d7c/d7d/df8/d82/f117 x:0 0 0 2024-02-28T16:31:22.510 INFO:tasks.workunit.client.1.smithi066.stdout:6/416: dwrite d2/d2a/f5b [0,4194304] 0 2024-02-28T16:31:22.511 INFO:tasks.workunit.client.0.smithi001.stdout:8/832: rename d4/d7/d22/d27/d58/l70 to d4/d7/d22/df6/l10c 0 2024-02-28T16:31:22.511 INFO:tasks.workunit.client.0.smithi001.stdout:6/837: mknod d9/d39/dc4/c117 0 2024-02-28T16:31:22.512 INFO:tasks.workunit.client.1.smithi066.stdout:9/380: dwrite da/d3a/d15/f6a [0,4194304] 0 2024-02-28T16:31:22.512 INFO:tasks.workunit.client.1.smithi066.stdout:5/472: dwrite d3/d4/dc/dd/d41/f65 [0,4194304] 0 2024-02-28T16:31:22.513 INFO:tasks.workunit.client.1.smithi066.stdout:3/504: chown db/dc/d11/l75 155 1 2024-02-28T16:31:22.515 INFO:tasks.workunit.client.1.smithi066.stdout:3/505: dread - db/d13/d1a/d25/f67 zero size 2024-02-28T16:31:22.516 INFO:tasks.workunit.client.0.smithi001.stdout:8/833: symlink d4/d32/d69/d9a/l10d 0 2024-02-28T16:31:22.517 INFO:tasks.workunit.client.1.smithi066.stdout:5/473: fsync d3/d4/f5c 0 2024-02-28T16:31:22.518 INFO:tasks.workunit.client.0.smithi001.stdout:6/838: creat d9/db/d5f/d89/f118 x:0 0 0 2024-02-28T16:31:22.519 INFO:tasks.workunit.client.1.smithi066.stdout:5/474: stat d3/d4/dc/dd/d41/f65 0 2024-02-28T16:31:22.524 INFO:tasks.workunit.client.0.smithi001.stdout:6/839: creat d9/f119 x:0 0 0 2024-02-28T16:31:22.527 INFO:tasks.workunit.client.1.smithi066.stdout:6/417: unlink d2/l1f 0 2024-02-28T16:31:22.530 INFO:tasks.workunit.client.0.smithi001.stdout:6/840: symlink d9/db/d5f/dd4/dfb/l11a 0 2024-02-28T16:31:22.531 INFO:tasks.workunit.client.1.smithi066.stdout:4/611: creat d1/d18/d41/dba/fc6 x:0 0 0 2024-02-28T16:31:22.531 INFO:tasks.workunit.client.1.smithi066.stdout:9/381: mknod da/df/d18/c7f 0 2024-02-28T16:31:22.532 INFO:tasks.workunit.client.1.smithi066.stdout:2/498: rename d0/d12/d2e/d7a/l5a to d0/l9b 0 2024-02-28T16:31:22.535 INFO:tasks.workunit.client.1.smithi066.stdout:9/382: chown da/df/d18/d2d 1342 1 2024-02-28T16:31:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.546+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f12940ff130 con 0x7f12940fcfb0 2024-02-28T16:31:22.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.547+0000 7f128e000700 1 -- 172.21.15.1:0/4291618735 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f1284017070 con 0x7f12940fcfb0 2024-02-28T16:31:22.537 INFO:tasks.workunit.client.1.smithi066.stdout:6/418: symlink d2/d2c/l8f 0 2024-02-28T16:31:22.537 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:31:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:31:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:31:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:31:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:31:22.537 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 13, 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:31:22.538 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:31:22.539 INFO:tasks.workunit.client.0.smithi001.stdout:6/841: truncate d9/d29/f3b 932282 0 2024-02-28T16:31:22.539 INFO:tasks.workunit.client.1.smithi066.stdout:9/383: write da/f64 [445552,111325] 0 2024-02-28T16:31:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f127c06b220 msgr2=0x7f127c06d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f127c06b220 0x7f127c06d6d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f12940ffc90 tx=0x7f1290009ca0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12940fcfb0 msgr2=0x7f12940fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12940fcfb0 0x7f12940fe380 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1284002a50 tx=0x7f1284004630 comp rx=0 tx=0).stop 2024-02-28T16:31:22.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 shutdown_connections 2024-02-28T16:31:22.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7f127c06b220 0x7f127c06d6d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12940fbe40 0x7f12940fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 --2- 172.21.15.1:0/4291618735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12940fcfb0 0x7f12940fe380 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.553+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 >> 172.21.15.1:0/4291618735 conn(0x7f12940f74e0 msgr2=0x7f12940f8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:22.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.554+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 shutdown_connections 2024-02-28T16:31:22.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.554+0000 7f129c800700 1 -- 172.21.15.1:0/4291618735 wait complete. 2024-02-28T16:31:22.544 INFO:tasks.workunit.client.1.smithi066.stdout:6/419: truncate d2/da/f3b 218098 0 2024-02-28T16:31:22.545 INFO:tasks.workunit.client.1.smithi066.stdout:5/475: dwrite d3/d1b/d39/f44 [0,4194304] 0 2024-02-28T16:31:22.545 INFO:tasks.workunit.client.1.smithi066.stdout:3/506: rename db/d2a/f42 to db/d13/d14/d2d/d92/fac 0 2024-02-28T16:31:22.545 INFO:tasks.workunit.client.1.smithi066.stdout:9/384: truncate da/d22/f51 380169 0 2024-02-28T16:31:22.546 INFO:tasks.workunit.client.0.smithi001.stdout:6/842: mknod d9/d29/d6a/ded/d105/d106/c11b 0 2024-02-28T16:31:22.548 INFO:tasks.workunit.client.0.smithi001.stdout:6/843: write d9/f86 [2649077,26752] 0 2024-02-28T16:31:22.552 INFO:tasks.workunit.client.0.smithi001.stdout:6/844: dread - d9/db/d5f/d89/d8a/f104 zero size 2024-02-28T16:31:22.554 INFO:tasks.workunit.client.0.smithi001.stdout:3/842: dwrite d4/f26 [0,4194304] 0 2024-02-28T16:31:22.556 INFO:tasks.workunit.client.1.smithi066.stdout:5/476: mkdir d3/d4/dc/dd/d35/d92 0 2024-02-28T16:31:22.556 INFO:tasks.workunit.client.1.smithi066.stdout:9/385: creat da/df/d18/d28/d42/f80 x:0 0 0 2024-02-28T16:31:22.557 INFO:tasks.workunit.client.0.smithi001.stdout:6/845: write d9/d29/d6e/d9a/fff [2188016,26598] 0 2024-02-28T16:31:22.557 INFO:tasks.workunit.client.0.smithi001.stdout:3/843: symlink d4/d6/d7/d23/l118 0 2024-02-28T16:31:22.559 INFO:tasks.workunit.client.0.smithi001.stdout:6/846: fdatasync d9/fb3 0 2024-02-28T16:31:22.559 INFO:tasks.workunit.client.0.smithi001.stdout:3/844: stat d4/d6/f52 0 2024-02-28T16:31:22.560 INFO:tasks.workunit.client.1.smithi066.stdout:6/420: creat d2/da/d38/f90 x:0 0 0 2024-02-28T16:31:22.561 INFO:tasks.workunit.client.1.smithi066.stdout:4/612: dwrite d1/d18/d41/f68 [0,4194304] 0 2024-02-28T16:31:22.566 INFO:tasks.workunit.client.0.smithi001.stdout:6/847: symlink d9/d39/de0/l11c 0 2024-02-28T16:31:22.567 INFO:tasks.workunit.client.1.smithi066.stdout:1/462: sync 2024-02-28T16:31:22.568 INFO:tasks.workunit.client.1.smithi066.stdout:9/386: mkdir da/df/d18/d2d/d81 0 2024-02-28T16:31:22.569 INFO:tasks.workunit.client.1.smithi066.stdout:0/620: sync 2024-02-28T16:31:22.569 INFO:tasks.workunit.client.1.smithi066.stdout:8/442: sync 2024-02-28T16:31:22.569 INFO:tasks.workunit.client.1.smithi066.stdout:9/387: readlink da/df/l26 0 2024-02-28T16:31:22.571 INFO:tasks.workunit.client.0.smithi001.stdout:6/848: mknod d9/db/d5f/d8b/c11d 0 2024-02-28T16:31:22.571 INFO:tasks.workunit.client.1.smithi066.stdout:7/459: sync 2024-02-28T16:31:22.572 INFO:tasks.workunit.client.1.smithi066.stdout:5/477: mkdir d3/d4/dc/d56/d93 0 2024-02-28T16:31:22.578 INFO:tasks.workunit.client.1.smithi066.stdout:6/421: creat d2/d26/d32/d46/d80/f91 x:0 0 0 2024-02-28T16:31:22.581 INFO:tasks.workunit.client.0.smithi001.stdout:6/849: dread d9/db/d5f/d8b/f72 [0,4194304] 0 2024-02-28T16:31:22.583 INFO:tasks.workunit.client.1.smithi066.stdout:1/463: write d2/d27/d2b/d43/d46/d80/f87 [992904,12571] 0 2024-02-28T16:31:22.587 INFO:tasks.workunit.client.0.smithi001.stdout:8/834: dread d4/d7/d22/d27/d58/d94/dbc/fbd [0,4194304] 0 2024-02-28T16:31:22.588 INFO:tasks.workunit.client.1.smithi066.stdout:0/621: write d2/d85/f9f [731627,61782] 0 2024-02-28T16:31:22.589 INFO:tasks.workunit.client.1.smithi066.stdout:3/507: dwrite db/d13/d14/d24/f34 [0,4194304] 0 2024-02-28T16:31:22.590 INFO:tasks.workunit.client.1.smithi066.stdout:5/478: fdatasync d3/d4/dc/dd/d35/d86/f88 0 2024-02-28T16:31:22.590 INFO:tasks.workunit.client.0.smithi001.stdout:6/850: link d9/db/d19/ff9 d9/db/d5f/d8b/f11e 0 2024-02-28T16:31:22.592 INFO:tasks.workunit.client.0.smithi001.stdout:8/835: write d4/d7/d22/d27/d4b/d77/f7a [683884,31260] 0 2024-02-28T16:31:22.594 INFO:tasks.workunit.client.1.smithi066.stdout:5/479: readlink d3/d4/l7 0 2024-02-28T16:31:22.601 INFO:tasks.workunit.client.0.smithi001.stdout:8/836: truncate d4/d7/d22/d27/d8e/dae/faf 4025173 0 2024-02-28T16:31:22.605 INFO:tasks.workunit.client.0.smithi001.stdout:3/845: dwrite d4/d1e/f27 [0,4194304] 0 2024-02-28T16:31:22.606 INFO:tasks.workunit.client.1.smithi066.stdout:3/508: chown db/d2a/d44/f4f 3 1 2024-02-28T16:31:22.609 INFO:tasks.workunit.client.0.smithi001.stdout:6/851: dwrite d9/f119 [0,4194304] 0 2024-02-28T16:31:22.616 INFO:tasks.workunit.client.1.smithi066.stdout:9/388: dwrite da/df/d18/f78 [0,4194304] 0 2024-02-28T16:31:22.617 INFO:tasks.workunit.client.0.smithi001.stdout:1/943: sync 2024-02-28T16:31:22.618 INFO:tasks.workunit.client.1.smithi066.stdout:8/443: dwrite d0/d6/dc/d16/d1a/d63/d58/f70 [0,4194304] 0 2024-02-28T16:31:22.622 INFO:tasks.workunit.client.0.smithi001.stdout:6/852: link d9/db/d19/fd3 d9/db/dd9/f11f 0 2024-02-28T16:31:22.622 INFO:tasks.workunit.client.1.smithi066.stdout:9/389: dread - da/d3a/d15/d48/f66 zero size 2024-02-28T16:31:22.623 INFO:tasks.workunit.client.1.smithi066.stdout:7/460: dwrite dc/de/d17/d1e/f28 [0,4194304] 0 2024-02-28T16:31:22.636 INFO:tasks.workunit.client.1.smithi066.stdout:4/613: dread d1/f6 [0,4194304] 0 2024-02-28T16:31:22.638 INFO:tasks.workunit.client.0.smithi001.stdout:3/846: dwrite d4/d18/f6b [0,4194304] 0 2024-02-28T16:31:22.638 INFO:tasks.workunit.client.1.smithi066.stdout:3/509: mkdir db/d13/d14/dad 0 2024-02-28T16:31:22.638 INFO:tasks.workunit.client.1.smithi066.stdout:6/422: getdents d2/d26/d32/d46/d80/d5d/d63 0 2024-02-28T16:31:22.656 INFO:tasks.workunit.client.0.smithi001.stdout:1/944: dwrite d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f146 [0,4194304] 0 2024-02-28T16:31:22.656 INFO:tasks.workunit.client.0.smithi001.stdout:3/847: rename d4/d6/ce6 to d4/d1e/d7c/da8/c119 0 2024-02-28T16:31:22.656 INFO:tasks.workunit.client.1.smithi066.stdout:9/390: dwrite da/df/d18/f6f [0,4194304] 0 2024-02-28T16:31:22.659 INFO:tasks.workunit.client.0.smithi001.stdout:1/945: stat d0/d9/d1a/d2b/d26/d27/d30/d88/fbc 0 2024-02-28T16:31:22.663 INFO:tasks.workunit.client.0.smithi001.stdout:3/848: truncate d4/d97/fdc 4417286 0 2024-02-28T16:31:22.678 INFO:tasks.workunit.client.1.smithi066.stdout:0/622: getdents d2/d27/d41/d76/d93/db2/db5/d67 0 2024-02-28T16:31:22.680 INFO:tasks.workunit.client.1.smithi066.stdout:3/510: creat db/d7a/d7c/fae x:0 0 0 2024-02-28T16:31:22.684 INFO:tasks.workunit.client.1.smithi066.stdout:3/511: dread - db/d13/d1a/d25/f96 zero size 2024-02-28T16:31:22.684 INFO:tasks.workunit.client.1.smithi066.stdout:6/423: rename d2/f68 to d2/da/d19/d75/f92 0 2024-02-28T16:31:22.686 INFO:tasks.workunit.client.1.smithi066.stdout:3/512: fsync db/d2a/f37 0 2024-02-28T16:31:22.688 INFO:tasks.workunit.client.1.smithi066.stdout:6/424: chown d2/d2c/d72/f7b 1021515049 1 2024-02-28T16:31:22.690 INFO:tasks.workunit.client.0.smithi001.stdout:1/946: dread d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f93 [0,4194304] 0 2024-02-28T16:31:22.690 INFO:tasks.workunit.client.1.smithi066.stdout:3/513: chown db/d13/d14/d2d 250341842 1 2024-02-28T16:31:22.694 INFO:tasks.workunit.client.1.smithi066.stdout:3/514: chown db/d13/d1a/d33/f39 5169 1 2024-02-28T16:31:22.696 INFO:tasks.workunit.client.1.smithi066.stdout:5/480: dread d3/d4/dc/dd/d41/d5d/f69 [0,4194304] 0 2024-02-28T16:31:22.697 INFO:tasks.workunit.client.1.smithi066.stdout:6/425: dread - d2/d26/d32/d46/d80/d5d/d63/f84 zero size 2024-02-28T16:31:22.699 INFO:tasks.workunit.client.1.smithi066.stdout:2/499: sync 2024-02-28T16:31:22.701 INFO:tasks.workunit.client.1.smithi066.stdout:3/515: symlink db/d13/d14/d2d/d92/laf 0 2024-02-28T16:31:22.701 INFO:tasks.workunit.client.0.smithi001.stdout:1/947: creat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/dfd/f149 x:0 0 0 2024-02-28T16:31:22.703 INFO:tasks.workunit.client.1.smithi066.stdout:9/391: creat da/df/d18/f82 x:0 0 0 2024-02-28T16:31:22.704 INFO:tasks.workunit.client.1.smithi066.stdout:2/500: write d0/d12/d1a/f2b [388809,86411] 0 2024-02-28T16:31:22.708 INFO:tasks.workunit.client.0.smithi001.stdout:6/853: dread d9/db/d19/f90 [0,4194304] 0 2024-02-28T16:31:22.708 INFO:tasks.workunit.client.0.smithi001.stdout:3/849: dwrite d4/d6/f9b [0,4194304] 0 2024-02-28T16:31:22.709 INFO:tasks.workunit.client.0.smithi001.stdout:1/948: mknod d0/d9/d1a/d2b/d26/d27/dd3/de8/d101/c14a 0 2024-02-28T16:31:22.709 INFO:tasks.workunit.client.0.smithi001.stdout:6/854: chown d9/f52 108087828 1 2024-02-28T16:31:22.711 INFO:tasks.workunit.client.1.smithi066.stdout:0/623: dwrite d2/d27/d41/d76/d93/db2/db5/d52/f9a [0,4194304] 0 2024-02-28T16:31:22.713 INFO:tasks.workunit.client.1.smithi066.stdout:3/516: creat db/dc/d35/d3d/d76/d89/fb0 x:0 0 0 2024-02-28T16:31:22.713 INFO:tasks.workunit.client.1.smithi066.stdout:3/517: chown db/dc 3 1 2024-02-28T16:31:22.714 INFO:tasks.workunit.client.0.smithi001.stdout:3/850: unlink d4/d1e/d7c/l10d 0 2024-02-28T16:31:22.716 INFO:tasks.workunit.client.0.smithi001.stdout:1/949: creat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f14b x:0 0 0 2024-02-28T16:31:22.716 INFO:tasks.workunit.client.1.smithi066.stdout:0/624: fdatasync d2/d27/d41/d76/d93/f1e 0 2024-02-28T16:31:22.717 INFO:tasks.workunit.client.1.smithi066.stdout:5/481: mknod d3/d4/dc/d42/d60/c94 0 2024-02-28T16:31:22.718 INFO:tasks.workunit.client.0.smithi001.stdout:1/950: chown d0/d9/d1a/d2b/d26/d105/d121 37866 1 2024-02-28T16:31:22.721 INFO:tasks.workunit.client.1.smithi066.stdout:2/501: readlink d0/d12/d1a/d23/d29/d21/l32 0 2024-02-28T16:31:22.721 INFO:tasks.workunit.client.0.smithi001.stdout:3/851: write d4/d6/d7/d78/fd1 [3586555,97328] 0 2024-02-28T16:31:22.724 INFO:tasks.workunit.client.0.smithi001.stdout:6/855: rename d9/d29/d6e/d9a/fad to d9/db/d5f/d62/f120 0 2024-02-28T16:31:22.725 INFO:tasks.workunit.client.1.smithi066.stdout:0/625: write d2/d27/d41/d76/d93/d10/f2c [3127472,44890] 0 2024-02-28T16:31:22.726 INFO:tasks.workunit.client.1.smithi066.stdout:1/464: sync 2024-02-28T16:31:22.728 INFO:tasks.workunit.client.1.smithi066.stdout:2/502: write d0/d12/f47 [2778203,69555] 0 2024-02-28T16:31:22.729 INFO:tasks.workunit.client.0.smithi001.stdout:6/856: chown d9/d29/lf2 0 1 2024-02-28T16:31:22.730 INFO:tasks.workunit.client.1.smithi066.stdout:1/465: chown d2/d27/d2b/d37/d48/l70 171255899 1 2024-02-28T16:31:22.733 INFO:tasks.workunit.client.0.smithi001.stdout:4/938: sync 2024-02-28T16:31:22.735 INFO:tasks.workunit.client.0.smithi001.stdout:3/852: creat d4/d6/d7/d78/d81/dd6/dd9/f11a x:0 0 0 2024-02-28T16:31:22.739 INFO:tasks.workunit.client.0.smithi001.stdout:6/857: rename d9/da6 to d9/db/d5f/dd4/dfb/d121 0 2024-02-28T16:31:22.740 INFO:tasks.workunit.client.0.smithi001.stdout:4/939: creat d3/d2d/d81/d113/f136 x:0 0 0 2024-02-28T16:31:22.740 INFO:tasks.workunit.client.1.smithi066.stdout:2/503: creat d0/d12/d1a/d23/d29/d21/d2f/f9c x:0 0 0 2024-02-28T16:31:22.740 INFO:tasks.workunit.client.1.smithi066.stdout:0/626: dread d2/d85/f9f [0,4194304] 0 2024-02-28T16:31:22.741 INFO:tasks.workunit.client.1.smithi066.stdout:3/518: rename db/dc/d35/d3d/f93 to db/d13/fb1 0 2024-02-28T16:31:22.741 INFO:tasks.workunit.client.0.smithi001.stdout:3/853: fsync d4/f61 0 2024-02-28T16:31:22.744 INFO:tasks.workunit.client.0.smithi001.stdout:6/858: fsync d9/d29/d6a/f85 0 2024-02-28T16:31:22.747 INFO:tasks.workunit.client.0.smithi001.stdout:4/940: fdatasync d3/d5/fab 0 2024-02-28T16:31:22.747 INFO:tasks.workunit.client.0.smithi001.stdout:3/854: chown d4/d1e/d7c/d7d/df8/d82/f117 4235372 1 2024-02-28T16:31:22.747 INFO:tasks.workunit.client.1.smithi066.stdout:0/627: unlink d2/d27/fa3 0 2024-02-28T16:31:22.750 INFO:tasks.workunit.client.0.smithi001.stdout:6/859: fdatasync d9/d29/d6e/f34 0 2024-02-28T16:31:22.753 INFO:tasks.workunit.client.1.smithi066.stdout:3/519: creat db/d13/d14/d24/fb2 x:0 0 0 2024-02-28T16:31:22.754 INFO:tasks.workunit.client.0.smithi001.stdout:1/951: dwrite d0/d9/d1a/d2b/d26/d27/dce/f134 [0,4194304] 0 2024-02-28T16:31:22.756 INFO:tasks.workunit.client.1.smithi066.stdout:9/392: dread da/df/d18/d2d/f4c [0,4194304] 0 2024-02-28T16:31:22.757 INFO:tasks.workunit.client.1.smithi066.stdout:3/520: chown db/d2a/d44/d48/l7b 3 1 2024-02-28T16:31:22.758 INFO:tasks.workunit.client.1.smithi066.stdout:2/504: truncate d0/d12/d2e/d7a/f5b 6438310 0 2024-02-28T16:31:22.758 INFO:tasks.workunit.client.0.smithi001.stdout:4/941: getdents d3/d2d/d4c/d10f 0 2024-02-28T16:31:22.760 INFO:tasks.workunit.client.1.smithi066.stdout:9/393: write da/df/d18/f6f [2828996,120936] 0 2024-02-28T16:31:22.762 INFO:tasks.workunit.client.1.smithi066.stdout:0/628: mknod d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/cd4 0 2024-02-28T16:31:22.765 INFO:tasks.workunit.client.0.smithi001.stdout:6/860: truncate d9/db/d5f/d62/f6b 705855 0 2024-02-28T16:31:22.765 INFO:tasks.workunit.client.0.smithi001.stdout:1/952: rename d0/d9/d1a/d2b/d26/d27/d30/l31 to d0/d9/d110/l14c 0 2024-02-28T16:31:22.767 INFO:tasks.workunit.client.0.smithi001.stdout:6/861: chown d9/d29/d6a/ded/d105/d106 7571750 1 2024-02-28T16:31:22.767 INFO:tasks.workunit.client.1.smithi066.stdout:9/394: chown da/df/d18/d28/d42/f49 44 1 2024-02-28T16:31:22.768 INFO:tasks.workunit.client.0.smithi001.stdout:2/794: sync 2024-02-28T16:31:22.768 INFO:tasks.workunit.client.0.smithi001.stdout:5/819: sync 2024-02-28T16:31:22.768 INFO:tasks.workunit.client.0.smithi001.stdout:7/875: sync 2024-02-28T16:31:22.768 INFO:tasks.workunit.client.0.smithi001.stdout:0/838: sync 2024-02-28T16:31:22.768 INFO:tasks.workunit.client.0.smithi001.stdout:8/837: sync 2024-02-28T16:31:22.771 INFO:tasks.workunit.client.1.smithi066.stdout:2/505: rename d0/d12/d1a/f79 to d0/d12/d2e/f9d 0 2024-02-28T16:31:22.773 INFO:tasks.workunit.client.1.smithi066.stdout:1/466: dread d2/d18/d19/f3b [0,4194304] 0 2024-02-28T16:31:22.775 INFO:tasks.workunit.client.1.smithi066.stdout:3/521: mkdir db/dc/db3 0 2024-02-28T16:31:22.778 INFO:tasks.workunit.client.0.smithi001.stdout:3/855: getdents d4/d1e/d7c/d7d/df8 0 2024-02-28T16:31:22.780 INFO:tasks.workunit.client.1.smithi066.stdout:2/506: write d0/d12/d1a/d23/d29/d21/d2f/f9c [204120,59004] 0 2024-02-28T16:31:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.791+0000 7efd43200700 1 -- 172.21.15.1:0/2893993995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd3c1035a0 msgr2=0x7efd3c1039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.791+0000 7efd43200700 1 --2- 172.21.15.1:0/2893993995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd3c1035a0 0x7efd3c1039b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efd2c004940 tx=0x7efd2c009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:22.781 INFO:tasks.workunit.client.0.smithi001.stdout:8/838: write d4/d32/f7c [5623247,118966] 0 2024-02-28T16:31:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.792+0000 7efd43200700 1 -- 172.21.15.1:0/2893993995 shutdown_connections 2024-02-28T16:31:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.792+0000 7efd43200700 1 --2- 172.21.15.1:0/2893993995 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd3c104760 0x7efd3c104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.792+0000 7efd43200700 1 --2- 172.21.15.1:0/2893993995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd3c1035a0 0x7efd3c1039b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.792+0000 7efd43200700 1 -- 172.21.15.1:0/2893993995 >> 172.21.15.1:0/2893993995 conn(0x7efd3c0668f0 msgr2=0x7efd3c066d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.792+0000 7efd43200700 1 -- 172.21.15.1:0/2893993995 shutdown_connections 2024-02-28T16:31:22.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.793+0000 7efd43200700 1 -- 172.21.15.1:0/2893993995 wait complete. 2024-02-28T16:31:22.782 INFO:tasks.workunit.client.0.smithi001.stdout:0/839: chown d18/d22/d76/d7a/fe2 0 1 2024-02-28T16:31:22.782 INFO:tasks.workunit.client.0.smithi001.stdout:8/839: dread - d4/d47/f10a zero size 2024-02-28T16:31:22.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.794+0000 7efd43200700 1 Processor -- start 2024-02-28T16:31:22.783 INFO:tasks.workunit.client.0.smithi001.stdout:4/942: read d3/d1c/d58/d6e/d9f/fe3 [972935,121081] 0 2024-02-28T16:31:22.783 INFO:tasks.workunit.client.1.smithi066.stdout:0/629: link d2/d27/d41/d76/d93/d10/d23/d6d/f79 d2/d27/d41/d76/d93/db2/db5/d73/fd5 0 2024-02-28T16:31:22.783 INFO:tasks.workunit.client.1.smithi066.stdout:7/461: link dc/de/c68 dc/de/d17/c82 0 2024-02-28T16:31:22.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.795+0000 7efd43200700 1 -- start start 2024-02-28T16:31:22.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.796+0000 7efd43200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd3c1035a0 0x7efd3c113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:22.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.796+0000 7efd43200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd3c104760 0x7efd3c114450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:22.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:22 smithi001 ceph-mon[28991]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:22.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:22 smithi001 ceph-mon[28991]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:22.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:22 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/4291618735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:31:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.796+0000 7efd43200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c114a50 con 0x7efd3c1035a0 2024-02-28T16:31:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.796+0000 7efd43200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c1152f0 con 0x7efd3c104760 2024-02-28T16:31:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.796+0000 7efd3b400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd3c104760 0x7efd3c114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.796+0000 7efd3b400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd3c104760 0x7efd3c114450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:42788/0 (socket says 172.21.15.1:42788) 2024-02-28T16:31:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.796+0000 7efd3be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd3c1035a0 0x7efd3c113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.796+0000 7efd3b400700 1 -- 172.21.15.1:0/3100948937 learned_addr learned my addr 172.21.15.1:0/3100948937 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:22.786 INFO:tasks.workunit.client.0.smithi001.stdout:5/820: fdatasync dc/d1a/d7d/f88 0 2024-02-28T16:31:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.797+0000 7efd3b400700 1 -- 172.21.15.1:0/3100948937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd3c1035a0 msgr2=0x7efd3c113f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.797+0000 7efd3b400700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd3c1035a0 0x7efd3c113f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.798+0000 7efd3b400700 1 -- 172.21.15.1:0/3100948937 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c005040 con 0x7efd3c104760 2024-02-28T16:31:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.800+0000 7efd3b400700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd3c104760 0x7efd3c114450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd2800eb40 tx=0x7efd2800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.801+0000 7efd33e00700 1 -- 172.21.15.1:0/3100948937 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd28007cc0 con 0x7efd3c104760 2024-02-28T16:31:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.801+0000 7efd33e00700 1 -- 172.21.15.1:0/3100948937 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd2800f040 con 0x7efd3c104760 2024-02-28T16:31:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.801+0000 7efd33e00700 1 -- 172.21.15.1:0/3100948937 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd2800ca10 con 0x7efd3c104760 2024-02-28T16:31:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.801+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c115570 con 0x7efd3c104760 2024-02-28T16:31:22.790 INFO:tasks.workunit.client.0.smithi001.stdout:1/953: mkdir d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/dfd/d14d 0 2024-02-28T16:31:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.801+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c115ab0 con 0x7efd3c104760 2024-02-28T16:31:22.791 INFO:tasks.workunit.client.1.smithi066.stdout:3/522: rename db/d7a/d7c to db/d2a/d44/d48/d7d/db4 0 2024-02-28T16:31:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.802+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c04bed0 con 0x7efd3c104760 2024-02-28T16:31:22.794 INFO:tasks.workunit.client.0.smithi001.stdout:2/795: symlink d0/d2/d3/de/d61/dd5/l104 0 2024-02-28T16:31:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.806+0000 7efd33e00700 1 -- 172.21.15.1:0/3100948937 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 25) v1 ==== 88910+0+0 (secure 0 0 0) 0x7efd2800cb70 con 0x7efd3c104760 2024-02-28T16:31:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.807+0000 7efd33e00700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7efd2006b1d0 0x7efd2006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:22.796 INFO:tasks.workunit.client.0.smithi001.stdout:2/796: stat d0/d2/d3/de/d61/f67 0 2024-02-28T16:31:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.808+0000 7efd33e00700 1 -- 172.21.15.1:0/3100948937 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5722+0+0 (secure 0 0 0) 0x7efd280898f0 con 0x7efd3c104760 2024-02-28T16:31:22.798 INFO:tasks.workunit.client.0.smithi001.stdout:8/840: stat d4/f13 0 2024-02-28T16:31:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.810+0000 7efd3be00700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7efd2006b1d0 0x7efd2006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:22.800 INFO:tasks.workunit.client.1.smithi066.stdout:0/630: link d2/d27/d41/d76/d93/db2/db5/d73/fc3 d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/dd1/fd6 0 2024-02-28T16:31:22.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.813+0000 7efd3be00700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7efd2006b1d0 0x7efd2006d680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efd2c013740 tx=0x7efd2c009e30 comp rx=0 tx=0).ready entity=mgr.24347 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:22.802 INFO:tasks.workunit.client.0.smithi001.stdout:4/943: rename d3/d1c/d58/d88/ca9 to d3/d5/d7/d1f/d2e/dbe/d115/c137 0 2024-02-28T16:31:22.802 INFO:tasks.workunit.client.1.smithi066.stdout:4/614: truncate d1/d23/d25/d3e/f66 2520173 0 2024-02-28T16:31:22.804 INFO:tasks.workunit.client.1.smithi066.stdout:9/395: dread da/df/d18/d28/f4d [0,4194304] 0 2024-02-28T16:31:22.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:22.818+0000 7efd33e00700 1 -- 172.21.15.1:0/3100948937 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7efd28057680 con 0x7efd3c104760 2024-02-28T16:31:22.807 INFO:tasks.workunit.client.1.smithi066.stdout:3/523: fsync db/dc/d35/d3d/d76/d89/fb0 0 2024-02-28T16:31:22.808 INFO:tasks.workunit.client.1.smithi066.stdout:4/615: fdatasync d1/d23/d25/d3e/d40/d9c/fb9 0 2024-02-28T16:31:22.811 INFO:tasks.workunit.client.1.smithi066.stdout:2/507: dwrite d0/d12/d1a/f2b [0,4194304] 0 2024-02-28T16:31:22.815 INFO:tasks.workunit.client.1.smithi066.stdout:3/524: chown db/d13/d1a/d33/f39 514605 1 2024-02-28T16:31:22.815 INFO:tasks.workunit.client.0.smithi001.stdout:3/856: write d4/d6/d7/f93 [269495,100585] 0 2024-02-28T16:31:22.817 INFO:tasks.workunit.client.1.smithi066.stdout:4/616: creat d1/d23/d25/d3e/d40/d9c/fc7 x:0 0 0 2024-02-28T16:31:22.821 INFO:tasks.workunit.client.1.smithi066.stdout:4/617: readlink d1/d23/d25/d3e/l61 0 2024-02-28T16:31:22.824 INFO:tasks.workunit.client.0.smithi001.stdout:1/954: write d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/f83 [629121,77085] 0 2024-02-28T16:31:22.825 INFO:tasks.workunit.client.0.smithi001.stdout:8/841: dread - d4/d32/d69/d9a/fcf zero size 2024-02-28T16:31:22.826 INFO:tasks.workunit.client.1.smithi066.stdout:0/631: getdents d2/d27/d41/d76/d93/db2/db5/d60 0 2024-02-28T16:31:22.830 INFO:tasks.workunit.client.0.smithi001.stdout:7/876: dread d3/d17/d52/d7f/d99/dc1/fe9 [0,4194304] 0 2024-02-28T16:31:22.831 INFO:tasks.workunit.client.0.smithi001.stdout:4/944: rename d3/d1c/d58/c44 to d3/d1c/d58/d6c/d79/dc8/c138 0 2024-02-28T16:31:22.832 INFO:tasks.workunit.client.1.smithi066.stdout:3/525: link db/dc/f55 db/d2a/d44/d48/d52/d77/fb5 0 2024-02-28T16:31:22.833 INFO:tasks.workunit.client.0.smithi001.stdout:6/862: getdents d9/de 0 2024-02-28T16:31:22.833 INFO:tasks.workunit.client.1.smithi066.stdout:8/444: mkdir d0/d6/dc/d16/d8c 0 2024-02-28T16:31:22.834 INFO:tasks.workunit.client.0.smithi001.stdout:2/797: symlink d0/d2/d3/de/l105 0 2024-02-28T16:31:22.836 INFO:tasks.workunit.client.1.smithi066.stdout:4/618: mknod d1/d18/cc8 0 2024-02-28T16:31:22.836 INFO:tasks.workunit.client.1.smithi066.stdout:5/482: dwrite d3/d1b/f37 [0,4194304] 0 2024-02-28T16:31:22.837 INFO:tasks.workunit.client.0.smithi001.stdout:6/863: dread - d9/d29/f48 zero size 2024-02-28T16:31:22.837 INFO:tasks.workunit.client.0.smithi001.stdout:1/955: rename d0/d9/ldd to d0/d9/d1a/d72/l14e 0 2024-02-28T16:31:22.838 INFO:tasks.workunit.client.1.smithi066.stdout:9/396: dwrite da/d22/d46/f74 [0,4194304] 0 2024-02-28T16:31:22.838 INFO:tasks.workunit.client.1.smithi066.stdout:2/508: truncate d0/f31 275849 0 2024-02-28T16:31:22.840 INFO:tasks.workunit.client.0.smithi001.stdout:4/945: dread - d3/d5/d7/d1f/d2e/dbe/f9c zero size 2024-02-28T16:31:22.843 INFO:tasks.workunit.client.1.smithi066.stdout:0/632: dread d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f72 [0,4194304] 0 2024-02-28T16:31:22.845 INFO:tasks.workunit.client.1.smithi066.stdout:4/619: dread - d1/f4d zero size 2024-02-28T16:31:22.847 INFO:tasks.workunit.client.1.smithi066.stdout:0/633: fsync d2/d27/d41/d76/d93/d10/f47 0 2024-02-28T16:31:22.851 INFO:tasks.workunit.client.1.smithi066.stdout:2/509: mkdir d0/d12/d3e/d9e 0 2024-02-28T16:31:22.851 INFO:tasks.workunit.client.0.smithi001.stdout:2/798: creat d0/d2/d3/de/d61/d12/d55/db2/f106 x:0 0 0 2024-02-28T16:31:22.851 INFO:tasks.workunit.client.0.smithi001.stdout:4/946: read d3/d2d/d4c/d5e/fa1 [55459,34297] 0 2024-02-28T16:31:22.855 INFO:tasks.workunit.client.1.smithi066.stdout:3/526: creat db/dc/d11/daa/fb6 x:0 0 0 2024-02-28T16:31:22.857 INFO:tasks.workunit.client.1.smithi066.stdout:3/527: fsync db/dc/d11/d1f/d78/fa4 0 2024-02-28T16:31:22.859 INFO:tasks.workunit.client.0.smithi001.stdout:6/864: unlink d9/db/d19/ff9 0 2024-02-28T16:31:22.862 INFO:tasks.workunit.client.0.smithi001.stdout:0/840: dwrite d18/d22/d2b/f10d [4194304,4194304] 0 2024-02-28T16:31:22.863 INFO:tasks.workunit.client.1.smithi066.stdout:5/483: symlink d3/d4/dc/dd/d2e/d8f/l95 0 2024-02-28T16:31:22.863 INFO:tasks.workunit.client.0.smithi001.stdout:5/821: dwrite dc/d12/d56/d96/db9/dc2/dca/f76 [0,4194304] 0 2024-02-28T16:31:22.863 INFO:tasks.workunit.client.0.smithi001.stdout:6/865: dread - d9/fb3 zero size 2024-02-28T16:31:22.865 INFO:tasks.workunit.client.1.smithi066.stdout:3/528: mknod db/dc/d35/d3d/d76/d89/cb7 0 2024-02-28T16:31:22.867 INFO:tasks.workunit.client.1.smithi066.stdout:8/445: getdents d0/d6/dc/d16/d1a/d63/d58/d5a 0 2024-02-28T16:31:22.867 INFO:tasks.workunit.client.1.smithi066.stdout:5/484: write d3/d4/dc/dd/d2e/d18/f4b [804762,10024] 0 2024-02-28T16:31:22.868 INFO:tasks.workunit.client.1.smithi066.stdout:3/529: dread - db/d13/d14/f6f zero size 2024-02-28T16:31:22.870 INFO:tasks.workunit.client.1.smithi066.stdout:5/485: fsync d3/d4/dc/dd/d2e/d18/f4a 0 2024-02-28T16:31:22.870 INFO:tasks.workunit.client.1.smithi066.stdout:1/467: rename d2/d18/f26 to d2/d18/d7a/d8d/f8f 0 2024-02-28T16:31:22.873 INFO:tasks.workunit.client.0.smithi001.stdout:3/857: write d4/d6/d7/d23/f7a [2180943,108086] 0 2024-02-28T16:31:22.878 INFO:tasks.workunit.client.1.smithi066.stdout:8/446: creat d0/db/d12/d8a/f8d x:0 0 0 2024-02-28T16:31:22.881 INFO:tasks.workunit.client.0.smithi001.stdout:8/842: dwrite d4/d32/f5c [0,4194304] 0 2024-02-28T16:31:22.883 INFO:tasks.workunit.client.1.smithi066.stdout:8/447: stat d0/d6/dc/d16/d4c/d6c/f8b 0 2024-02-28T16:31:22.884 INFO:tasks.workunit.client.1.smithi066.stdout:2/510: dwrite d0/f1f [4194304,4194304] 0 2024-02-28T16:31:22.885 INFO:tasks.workunit.client.0.smithi001.stdout:2/799: rmdir d0/d2/d3/de/d61/d12/d46 39 2024-02-28T16:31:22.885 INFO:tasks.workunit.client.1.smithi066.stdout:1/468: symlink d2/d27/d2b/d37/d51/l90 0 2024-02-28T16:31:22.886 INFO:tasks.workunit.client.0.smithi001.stdout:4/947: mknod d3/d5/d7/d1f/d2e/c139 0 2024-02-28T16:31:22.889 INFO:tasks.workunit.client.1.smithi066.stdout:6/426: creat d2/da/d19/f93 x:0 0 0 2024-02-28T16:31:22.889 INFO:tasks.workunit.client.1.smithi066.stdout:2/511: read - d0/d12/d1a/d23/d29/d21/d2f/d67/f97 zero size 2024-02-28T16:31:22.889 INFO:tasks.workunit.client.1.smithi066.stdout:7/462: rename dc/c1d to dc/d29/c83 0 2024-02-28T16:31:22.892 INFO:tasks.workunit.client.0.smithi001.stdout:5/822: creat dc/d12/daa/f109 x:0 0 0 2024-02-28T16:31:22.894 INFO:tasks.workunit.client.0.smithi001.stdout:7/877: truncate d3/d17/d52/d7f/d99/dc1/f6d 2907858 0 2024-02-28T16:31:22.895 INFO:tasks.workunit.client.1.smithi066.stdout:6/427: write d2/f43 [997482,12428] 0 2024-02-28T16:31:22.897 INFO:tasks.workunit.client.0.smithi001.stdout:5/823: write dc/d12/d41/dd6/f100 [164689,119747] 0 2024-02-28T16:31:22.899 INFO:tasks.workunit.client.1.smithi066.stdout:3/530: dread db/dc/d11/d1f/f2e [0,4194304] 0 2024-02-28T16:31:22.900 INFO:tasks.workunit.client.0.smithi001.stdout:3/858: creat d4/d6/d7/d78/d81/dd6/dd9/f11b x:0 0 0 2024-02-28T16:31:22.901 INFO:tasks.workunit.client.1.smithi066.stdout:8/448: creat d0/d6/dc/d16/d1a/f8e x:0 0 0 2024-02-28T16:31:22.901 INFO:tasks.workunit.client.1.smithi066.stdout:1/469: rmdir d2/d18/d6b 39 2024-02-28T16:31:22.901 INFO:tasks.workunit.client.0.smithi001.stdout:2/800: symlink d0/d2/d3/d36/l107 0 2024-02-28T16:31:22.904 INFO:tasks.workunit.client.0.smithi001.stdout:3/859: chown d4/d1e/d7c/d7d/df8/dee 346811803 1 2024-02-28T16:31:22.905 INFO:tasks.workunit.client.1.smithi066.stdout:1/470: dread - d2/d18/d19/f89 zero size 2024-02-28T16:31:22.906 INFO:tasks.workunit.client.1.smithi066.stdout:0/634: unlink d2/d27/d41/d76/d93/l43 0 2024-02-28T16:31:22.908 INFO:tasks.workunit.client.0.smithi001.stdout:3/860: write d4/d6/f9b [2873827,13780] 0 2024-02-28T16:31:22.908 INFO:tasks.workunit.client.1.smithi066.stdout:5/486: mknod d3/c96 0 2024-02-28T16:31:22.911 INFO:tasks.workunit.client.0.smithi001.stdout:1/956: rmdir d0/d9/d1a/d2b/d26/d27/dd3/de8/d102/d13a 0 2024-02-28T16:31:22.912 INFO:tasks.workunit.client.1.smithi066.stdout:6/428: fdatasync d2/d26/d32/d46/d80/d5d/f64 0 2024-02-28T16:31:22.913 INFO:tasks.workunit.client.1.smithi066.stdout:3/531: unlink db/d13/d1a/d25/c3f 0 2024-02-28T16:31:22.916 INFO:tasks.workunit.client.0.smithi001.stdout:2/801: symlink d0/d2/d3/d36/l108 0 2024-02-28T16:31:22.916 INFO:tasks.workunit.client.1.smithi066.stdout:8/449: creat d0/d6/dc/d16/d4c/f8f x:0 0 0 2024-02-28T16:31:22.918 INFO:tasks.workunit.client.0.smithi001.stdout:3/861: symlink d4/d1e/ddd/l11c 0 2024-02-28T16:31:22.918 INFO:tasks.workunit.client.0.smithi001.stdout:0/841: getdents d18/d22/d57/d7e 0 2024-02-28T16:31:22.919 INFO:tasks.workunit.client.1.smithi066.stdout:7/463: dwrite dc/de/f3f [0,4194304] 0 2024-02-28T16:31:22.924 INFO:tasks.workunit.client.1.smithi066.stdout:5/487: mknod d3/d4/dc/d42/d57/c97 0 2024-02-28T16:31:22.926 INFO:tasks.workunit.client.1.smithi066.stdout:4/620: rename d1/d23/d25/l56 to d1/d18/d41/lc9 0 2024-02-28T16:31:22.930 INFO:tasks.workunit.client.0.smithi001.stdout:0/842: read d18/d22/d2b/d36/d48/f9a [1786220,69300] 0 2024-02-28T16:31:22.932 INFO:tasks.workunit.client.1.smithi066.stdout:2/512: symlink d0/d12/d2e/l9f 0 2024-02-28T16:31:22.934 INFO:tasks.workunit.client.0.smithi001.stdout:5/824: write dc/d1a/d52/f89 [2282812,95060] 0 2024-02-28T16:31:22.934 INFO:tasks.workunit.client.1.smithi066.stdout:0/635: fsync d2/f28 0 2024-02-28T16:31:22.934 INFO:tasks.workunit.client.1.smithi066.stdout:7/464: dread dc/de/f13 [0,4194304] 0 2024-02-28T16:31:22.936 INFO:tasks.workunit.client.1.smithi066.stdout:3/532: stat db/d13/d14/d2d/d92/fac 0 2024-02-28T16:31:22.940 INFO:tasks.workunit.client.1.smithi066.stdout:5/488: creat d3/f98 x:0 0 0 2024-02-28T16:31:22.941 INFO:tasks.workunit.client.1.smithi066.stdout:8/450: readlink d0/d6/dc/d16/d1a/d63/d58/l67 0 2024-02-28T16:31:22.941 INFO:tasks.workunit.client.1.smithi066.stdout:1/471: rename d2/d18/d19/f33 to d2/d27/d2b/d37/d48/f91 0 2024-02-28T16:31:22.941 INFO:tasks.workunit.client.1.smithi066.stdout:9/397: dwrite da/f20 [0,4194304] 0 2024-02-28T16:31:22.943 INFO:tasks.workunit.client.0.smithi001.stdout:6/866: dwrite d9/db/d5f/d62/f120 [0,4194304] 0 2024-02-28T16:31:22.943 INFO:tasks.workunit.client.0.smithi001.stdout:7/878: truncate d3/d17/d52/d7f/d99/dc1/fe9 2752721 0 2024-02-28T16:31:22.944 INFO:tasks.workunit.client.0.smithi001.stdout:5/825: stat dc/d12/d41/dd6/ce0 0 2024-02-28T16:31:22.944 INFO:tasks.workunit.client.1.smithi066.stdout:4/621: mknod d1/d18/d4e/da0/cca 0 2024-02-28T16:31:22.947 INFO:tasks.workunit.client.1.smithi066.stdout:3/533: unlink db/dc/c8e 0 2024-02-28T16:31:22.949 INFO:tasks.workunit.client.1.smithi066.stdout:0/636: mknod d2/d27/d41/d76/d93/d10/cd7 0 2024-02-28T16:31:22.951 INFO:tasks.workunit.client.1.smithi066.stdout:7/465: creat dc/de/d17/d1e/d45/f84 x:0 0 0 2024-02-28T16:31:22.952 INFO:tasks.workunit.client.1.smithi066.stdout:8/451: symlink d0/db/d12/d38/l90 0 2024-02-28T16:31:22.957 INFO:tasks.workunit.client.0.smithi001.stdout:7/879: mkdir d3/d17/d3d/d59/d104/d118 0 2024-02-28T16:31:22.961 INFO:tasks.workunit.client.0.smithi001.stdout:5/826: mknod dc/c10a 0 2024-02-28T16:31:22.961 INFO:tasks.workunit.client.1.smithi066.stdout:2/513: rename d0/d12/d1a/d23/d29/d21/l8b to d0/d12/d3e/d9e/la0 0 2024-02-28T16:31:22.962 INFO:tasks.workunit.client.1.smithi066.stdout:4/622: rmdir d1/d7a 39 2024-02-28T16:31:22.962 INFO:tasks.workunit.client.0.smithi001.stdout:0/843: dread d18/d22/d2b/d36/d46/df5/d4f/fe8 [0,4194304] 0 2024-02-28T16:31:22.963 INFO:tasks.workunit.client.1.smithi066.stdout:3/534: unlink db/d13/d14/d2d/l4b 0 2024-02-28T16:31:22.965 INFO:tasks.workunit.client.1.smithi066.stdout:4/623: fsync d1/d18/d4e/fa5 0 2024-02-28T16:31:22.965 INFO:tasks.workunit.client.1.smithi066.stdout:3/535: read - db/d13/d1a/d25/f96 zero size 2024-02-28T16:31:22.967 INFO:tasks.workunit.client.0.smithi001.stdout:7/880: mknod d3/d17/d52/d7f/d99/dc1/d49/d8d/c119 0 2024-02-28T16:31:22.967 INFO:tasks.workunit.client.0.smithi001.stdout:3/862: dread d4/d6/f51 [0,4194304] 0 2024-02-28T16:31:22.968 INFO:tasks.workunit.client.0.smithi001.stdout:2/802: dwrite d0/d2/d3/fca [0,4194304] 0 2024-02-28T16:31:22.969 INFO:tasks.workunit.client.1.smithi066.stdout:4/624: mkdir d1/d18/da8/dcb 0 2024-02-28T16:31:22.972 INFO:tasks.workunit.client.0.smithi001.stdout:0/844: rmdir d18/d22/d57 39 2024-02-28T16:31:22.973 INFO:tasks.workunit.client.1.smithi066.stdout:1/472: getdents d2/d27 0 2024-02-28T16:31:22.975 INFO:tasks.workunit.client.0.smithi001.stdout:3/863: creat d4/d1e/d7c/d9a/f11d x:0 0 0 2024-02-28T16:31:22.977 INFO:tasks.workunit.client.0.smithi001.stdout:5/827: fdatasync dc/d1a/dba/fe9 0 2024-02-28T16:31:22.978 INFO:tasks.workunit.client.1.smithi066.stdout:7/466: getdents dc/d56 0 2024-02-28T16:31:22.978 INFO:tasks.workunit.client.1.smithi066.stdout:5/489: rename d3/d4/dc/d42/d57/c8e to d3/d4/c99 0 2024-02-28T16:31:22.978 INFO:tasks.workunit.client.1.smithi066.stdout:3/536: dread db/d2a/d44/d48/d5d/f81 [0,4194304] 0 2024-02-28T16:31:22.980 INFO:tasks.workunit.client.1.smithi066.stdout:7/467: readlink dc/l64 0 2024-02-28T16:31:22.983 INFO:tasks.workunit.client.1.smithi066.stdout:8/452: getdents d0/db/d12/d1d 0 2024-02-28T16:31:22.983 INFO:tasks.workunit.client.1.smithi066.stdout:5/490: read - d3/d4/dc/dd/d35/d86/f88 zero size 2024-02-28T16:31:22.986 INFO:tasks.workunit.client.1.smithi066.stdout:8/453: chown d0/db/d12/d38/l3e 682935 1 2024-02-28T16:31:22.988 INFO:tasks.workunit.client.1.smithi066.stdout:4/625: symlink d1/d18/d41/lcc 0 2024-02-28T16:31:22.990 INFO:tasks.workunit.client.0.smithi001.stdout:0/845: dread d18/d22/d2b/d36/d46/df5/dbc/f106 [0,4194304] 0 2024-02-28T16:31:22.994 INFO:tasks.workunit.client.1.smithi066.stdout:5/491: stat d3/d4/dc/c1f 0 2024-02-28T16:31:22.994 INFO:tasks.workunit.client.1.smithi066.stdout:8/454: creat d0/d6/dc/d16/d4c/d6c/f91 x:0 0 0 2024-02-28T16:31:22.994 INFO:tasks.workunit.client.1.smithi066.stdout:9/398: dread da/df/f2f [4194304,4194304] 0 2024-02-28T16:31:22.997 INFO:tasks.workunit.client.1.smithi066.stdout:7/468: dread dc/de/d17/d1e/f6c [0,4194304] 0 2024-02-28T16:31:22.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:22 smithi066 ceph-mon[36236]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:22.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:22 smithi066 ceph-mon[36236]: from='client.24429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:22.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:22 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/4291618735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:31:22.999 INFO:tasks.workunit.client.1.smithi066.stdout:9/399: readlink da/d22/l24 0 2024-02-28T16:31:22.999 INFO:tasks.workunit.client.1.smithi066.stdout:3/537: link db/d13/d14/d2d/d92/laf db/dc/d35/lb8 0 2024-02-28T16:31:23.000 INFO:tasks.workunit.client.1.smithi066.stdout:5/492: read d3/d4/dc/dd/d2e/f40 [745813,102167] 0 2024-02-28T16:31:23.002 INFO:tasks.workunit.client.1.smithi066.stdout:3/538: readlink db/d13/d14/d24/l28 0 2024-02-28T16:31:23.008 INFO:tasks.workunit.client.1.smithi066.stdout:0/637: dread d2/d27/d41/d76/d93/db2/db5/d51/f53 [0,4194304] 0 2024-02-28T16:31:23.008 INFO:tasks.workunit.client.1.smithi066.stdout:8/455: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92 0 2024-02-28T16:31:23.009 INFO:tasks.workunit.client.1.smithi066.stdout:3/539: truncate db/dc/d11/d1f/d78/f90 299754 0 2024-02-28T16:31:23.010 INFO:tasks.workunit.client.1.smithi066.stdout:7/469: rename dc/de/d17/d1e/l70 to dc/d29/l85 0 2024-02-28T16:31:23.011 INFO:tasks.workunit.client.1.smithi066.stdout:3/540: readlink db/dc/d35/l43 0 2024-02-28T16:31:23.013 INFO:tasks.workunit.client.1.smithi066.stdout:9/400: symlink da/df/d18/d28/d42/d68/l83 0 2024-02-28T16:31:23.014 INFO:tasks.workunit.client.1.smithi066.stdout:5/493: creat d3/d4/dc/dd/d2e/d8f/f9a x:0 0 0 2024-02-28T16:31:23.014 INFO:tasks.workunit.client.1.smithi066.stdout:3/541: fdatasync db/dc/d35/d3d/f97 0 2024-02-28T16:31:23.019 INFO:tasks.workunit.client.1.smithi066.stdout:4/626: dread d1/f1e [0,4194304] 0 2024-02-28T16:31:23.023 INFO:tasks.workunit.client.1.smithi066.stdout:8/456: mkdir d0/db/d12/d1d/d25/d65/d93 0 2024-02-28T16:31:23.024 INFO:tasks.workunit.client.1.smithi066.stdout:0/638: mknod d2/d27/d41/d76/d93/db2/db5/d52/da2/cd8 0 2024-02-28T16:31:23.036 INFO:tasks.workunit.client.1.smithi066.stdout:2/514: dread d0/f31 [0,4194304] 0 2024-02-28T16:31:23.042 INFO:tasks.workunit.client.1.smithi066.stdout:3/542: dwrite db/d7a/fa0 [0,4194304] 0 2024-02-28T16:31:23.042 INFO:tasks.workunit.client.1.smithi066.stdout:8/457: link d0/d6/dc/d16/d4c/d6c/l89 d0/d6/dc/l94 0 2024-02-28T16:31:23.049 INFO:tasks.workunit.client.1.smithi066.stdout:8/458: chown d0/f24 6532742 1 2024-02-28T16:31:23.049 INFO:tasks.workunit.client.1.smithi066.stdout:7/470: dread dc/de/d17/d1e/f20 [0,4194304] 0 2024-02-28T16:31:23.050 INFO:tasks.workunit.client.1.smithi066.stdout:6/429: dwrite d2/da/fe [0,4194304] 0 2024-02-28T16:31:23.054 INFO:tasks.workunit.client.1.smithi066.stdout:8/459: stat d0/d6/dc/d16/d4c/d6c/f91 0 2024-02-28T16:31:23.070 INFO:tasks.workunit.client.1.smithi066.stdout:2/515: rename d0/d12/f47 to d0/d12/d1a/d39/d7c/d87/fa1 0 2024-02-28T16:31:23.071 INFO:tasks.workunit.client.1.smithi066.stdout:7/471: creat dc/de/d17/d37/d47/f86 x:0 0 0 2024-02-28T16:31:23.074 INFO:tasks.workunit.client.1.smithi066.stdout:3/543: dwrite db/dc/d11/f26 [0,4194304] 0 2024-02-28T16:31:23.074 INFO:tasks.workunit.client.1.smithi066.stdout:8/460: mkdir d0/db/d15/d95 0 2024-02-28T16:31:23.074 INFO:tasks.workunit.client.1.smithi066.stdout:1/473: dwrite d2/d18/d19/f1e [0,4194304] 0 2024-02-28T16:31:23.079 INFO:tasks.workunit.client.1.smithi066.stdout:7/472: write f4 [1353712,99609] 0 2024-02-28T16:31:23.082 INFO:tasks.workunit.client.1.smithi066.stdout:0/639: dread d2/d27/d41/d76/d93/db2/db5/d67/d94/fb8 [0,4194304] 0 2024-02-28T16:31:23.082 INFO:tasks.workunit.client.1.smithi066.stdout:4/627: truncate d1/d23/d25/f6f 642868 0 2024-02-28T16:31:23.095 INFO:tasks.workunit.client.1.smithi066.stdout:5/494: dwrite d3/d4/dc/dd/d2e/d18/f2b [0,4194304] 0 2024-02-28T16:31:23.096 INFO:tasks.workunit.client.1.smithi066.stdout:1/474: truncate d2/d27/d2b/d43/d46/f61 5111220 0 2024-02-28T16:31:23.097 INFO:tasks.workunit.client.1.smithi066.stdout:5/495: readlink d3/d4/dc/d42/l52 0 2024-02-28T16:31:23.099 INFO:tasks.workunit.client.1.smithi066.stdout:3/544: rename db/dc/d11/f26 to db/dc/d35/d3d/d76/fb9 0 2024-02-28T16:31:23.104 INFO:tasks.workunit.client.1.smithi066.stdout:8/461: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92/d96 0 2024-02-28T16:31:23.105 INFO:tasks.workunit.client.1.smithi066.stdout:1/475: creat d2/d27/d2b/d37/d54/f92 x:0 0 0 2024-02-28T16:31:23.107 INFO:tasks.workunit.client.1.smithi066.stdout:6/430: truncate d2/da/d38/f39 8147186 0 2024-02-28T16:31:23.110 INFO:tasks.workunit.client.1.smithi066.stdout:4/628: rename d1/f46 to d1/d7a/fcd 0 2024-02-28T16:31:23.111 INFO:tasks.workunit.client.1.smithi066.stdout:2/516: truncate d0/d12/d1a/f54 1214734 0 2024-02-28T16:31:23.114 INFO:tasks.workunit.client.1.smithi066.stdout:7/473: dwrite dc/de/d17/f7c [0,4194304] 0 2024-02-28T16:31:23.115 INFO:tasks.workunit.client.1.smithi066.stdout:3/545: unlink db/d13/d1a/d25/f67 0 2024-02-28T16:31:23.124 INFO:tasks.workunit.client.1.smithi066.stdout:6/431: rmdir d2/d2c 39 2024-02-28T16:31:23.130 INFO:tasks.workunit.client.1.smithi066.stdout:4/629: creat d1/d18/d41/da7/dae/fce x:0 0 0 2024-02-28T16:31:23.131 INFO:tasks.workunit.client.0.smithi001.stdout:3/864: creat d4/d38/f11e x:0 0 0 2024-02-28T16:31:23.132 INFO:tasks.workunit.client.1.smithi066.stdout:0/640: creat d2/d27/d41/d76/d93/db2/db5/d51/fd9 x:0 0 0 2024-02-28T16:31:23.132 INFO:tasks.workunit.client.0.smithi001.stdout:5/828: mknod dc/d12/d56/d96/db9/dc2/dca/d3c/c10b 0 2024-02-28T16:31:23.134 INFO:tasks.workunit.client.1.smithi066.stdout:4/630: fsync d1/d23/d25/d43/f8b 0 2024-02-28T16:31:23.135 INFO:tasks.workunit.client.1.smithi066.stdout:7/474: dwrite dc/de/d17/d1e/f28 [0,4194304] 0 2024-02-28T16:31:23.136 INFO:tasks.workunit.client.1.smithi066.stdout:3/546: rename db/d2a/d44/d48/d5d/l70 to db/d7a/lba 0 2024-02-28T16:31:23.137 INFO:tasks.workunit.client.0.smithi001.stdout:0/846: mkdir d18/d22/d57/d7e/d110 0 2024-02-28T16:31:23.137 INFO:tasks.workunit.client.1.smithi066.stdout:1/476: unlink d2/d27/d2b/l4a 0 2024-02-28T16:31:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.150+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 --> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd3c115fd0 con 0x7efd2006b1d0 2024-02-28T16:31:23.138 INFO:tasks.workunit.client.0.smithi001.stdout:7/881: rmdir d3/d17/d52/d7f/d99 39 2024-02-28T16:31:23.138 INFO:tasks.workunit.client.1.smithi066.stdout:7/475: stat dc/l1f 0 2024-02-28T16:31:23.139 INFO:tasks.workunit.client.0.smithi001.stdout:5/829: mkdir dc/d12/d56/d96/db9/dc2/dca/d10c 0 2024-02-28T16:31:23.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.153+0000 7efd33e00700 1 -- 172.21.15.1:0/3100948937 <== mgr.24347 v2:172.21.15.66:6828/2196519723 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7efd3c115fd0 con 0x7efd2006b1d0 2024-02-28T16:31:23.141 INFO:tasks.workunit.client.1.smithi066.stdout:0/641: chown d2/d27/d41/d76/d93/db2/db5/d51/l5c 60382 1 2024-02-28T16:31:23.144 INFO:tasks.workunit.client.1.smithi066.stdout:6/432: symlink d2/da/d19/l94 0 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [], 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "1/2 daemons upgraded", 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mgr daemons", 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:31:23.145 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:31:23.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7efd2006b1d0 msgr2=0x7efd2006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:23.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7efd2006b1d0 0x7efd2006d680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efd2c013740 tx=0x7efd2c009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:23.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd3c104760 msgr2=0x7efd3c114450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:23.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd3c104760 0x7efd3c114450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd2800eb40 tx=0x7efd2800b040 comp rx=0 tx=0).stop 2024-02-28T16:31:23.147 INFO:tasks.workunit.client.0.smithi001.stdout:3/865: dread d4/d6/f9b [0,4194304] 0 2024-02-28T16:31:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 shutdown_connections 2024-02-28T16:31:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:6828/2196519723,v1:172.21.15.66:6829/2196519723] conn(0x7efd2006b1d0 0x7efd2006d680 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd3c1035a0 0x7efd3c113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 --2- 172.21.15.1:0/3100948937 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd3c104760 0x7efd3c114450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.158+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 >> 172.21.15.1:0/3100948937 conn(0x7efd3c0668f0 msgr2=0x7efd3c107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:23.148 INFO:tasks.workunit.client.1.smithi066.stdout:8/462: dwrite d0/db/d15/f31 [4194304,4194304] 0 2024-02-28T16:31:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.159+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 shutdown_connections 2024-02-28T16:31:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:23.160+0000 7efd43200700 1 -- 172.21.15.1:0/3100948937 wait complete. 2024-02-28T16:31:23.156 INFO:tasks.workunit.client.1.smithi066.stdout:0/642: creat d2/d27/d41/d76/d93/d33/d3f/db7/fda x:0 0 0 2024-02-28T16:31:23.156 INFO:tasks.workunit.client.1.smithi066.stdout:6/433: creat d2/d2c/f95 x:0 0 0 2024-02-28T16:31:23.159 INFO:tasks.workunit.client.0.smithi001.stdout:3/866: getdents d4/d1e/d7c/d7d/df8/d82/dea 0 2024-02-28T16:31:23.160 INFO:tasks.workunit.client.1.smithi066.stdout:4/631: getdents d1/d23/d25/d43 0 2024-02-28T16:31:23.164 INFO:tasks.workunit.client.0.smithi001.stdout:7/882: dread d3/d17/d52/d7f/d99/dc1/d49/d8d/fab [0,4194304] 0 2024-02-28T16:31:23.164 INFO:tasks.workunit.client.1.smithi066.stdout:5/496: truncate d3/d4/f61 1262771 0 2024-02-28T16:31:23.165 INFO:tasks.workunit.client.0.smithi001.stdout:3/867: mkdir d4/d6/d7/d78/d81/dd6/dd9/d11f 0 2024-02-28T16:31:23.167 INFO:tasks.workunit.client.0.smithi001.stdout:3/868: readlink d4/d18/l4f 0 2024-02-28T16:31:23.167 INFO:tasks.workunit.client.1.smithi066.stdout:1/477: dwrite d2/d27/f3c [0,4194304] 0 2024-02-28T16:31:23.171 INFO:tasks.workunit.client.1.smithi066.stdout:4/632: rmdir d1/d18/d41/dba 39 2024-02-28T16:31:23.171 INFO:tasks.workunit.client.0.smithi001.stdout:3/869: mknod d4/d1e/d7c/d7d/de0/c120 0 2024-02-28T16:31:23.172 INFO:tasks.workunit.client.1.smithi066.stdout:4/633: readlink d1/d18/d41/l79 0 2024-02-28T16:31:23.173 INFO:tasks.workunit.client.0.smithi001.stdout:0/847: dwrite d18/d22/d2f/ff1 [0,4194304] 0 2024-02-28T16:31:23.179 INFO:tasks.workunit.client.0.smithi001.stdout:3/870: getdents d4/d6/d7 0 2024-02-28T16:31:23.179 INFO:tasks.workunit.client.0.smithi001.stdout:0/848: stat d18/d22/d2b/d36/d46/df5/feb 0 2024-02-28T16:31:23.180 INFO:tasks.workunit.client.0.smithi001.stdout:0/849: chown d18/d22/l35 45071 1 2024-02-28T16:31:23.188 INFO:tasks.workunit.client.1.smithi066.stdout:6/434: dwrite d2/da/f22 [0,4194304] 0 2024-02-28T16:31:23.188 INFO:tasks.workunit.client.1.smithi066.stdout:4/634: fsync d1/d23/d25/f32 0 2024-02-28T16:31:23.190 INFO:tasks.workunit.client.1.smithi066.stdout:4/635: write d1/d18/d4e/da0/fa2 [964634,110516] 0 2024-02-28T16:31:23.191 INFO:tasks.workunit.client.0.smithi001.stdout:5/830: truncate dc/d12/d41/f4c 2991542 0 2024-02-28T16:31:23.192 INFO:tasks.workunit.client.0.smithi001.stdout:0/850: symlink d18/d22/d2f/d1c/l111 0 2024-02-28T16:31:23.193 INFO:tasks.workunit.client.0.smithi001.stdout:3/871: dread d4/d6/d7/d78/fc5 [0,4194304] 0 2024-02-28T16:31:23.197 INFO:tasks.workunit.client.0.smithi001.stdout:7/883: write d3/d17/d3d/d42/d8e/fe3 [380158,25831] 0 2024-02-28T16:31:23.197 INFO:tasks.workunit.client.1.smithi066.stdout:3/547: truncate db/d2a/f65 4051637 0 2024-02-28T16:31:23.199 INFO:tasks.workunit.client.1.smithi066.stdout:4/636: rmdir d1/d18 39 2024-02-28T16:31:23.200 INFO:tasks.workunit.client.0.smithi001.stdout:0/851: read d18/f6d [8137297,17637] 0 2024-02-28T16:31:23.201 INFO:tasks.workunit.client.1.smithi066.stdout:6/435: rename d2/da/d19/l94 to d2/da/d19/l96 0 2024-02-28T16:31:23.203 INFO:tasks.workunit.client.0.smithi001.stdout:0/852: chown d18/d22/d3d/fba 1963 1 2024-02-28T16:31:23.205 INFO:tasks.workunit.client.1.smithi066.stdout:8/463: truncate d0/f23 1330446 0 2024-02-28T16:31:23.207 INFO:tasks.workunit.client.0.smithi001.stdout:7/884: creat d3/d17/df0/f11a x:0 0 0 2024-02-28T16:31:23.208 INFO:tasks.workunit.client.0.smithi001.stdout:3/872: getdents d4/d1e 0 2024-02-28T16:31:23.211 INFO:tasks.workunit.client.1.smithi066.stdout:4/637: creat d1/d7a/fcf x:0 0 0 2024-02-28T16:31:23.215 INFO:tasks.workunit.client.0.smithi001.stdout:7/885: readlink d3/d17/d52/d7f/d99/dc1/d49/l58 0 2024-02-28T16:31:23.215 INFO:tasks.workunit.client.1.smithi066.stdout:3/548: chown db/d2a/d44/l68 10 1 2024-02-28T16:31:23.215 INFO:tasks.workunit.client.1.smithi066.stdout:2/517: dwrite d0/d12/d2e/d7a/f41 [0,4194304] 0 2024-02-28T16:31:23.218 INFO:tasks.workunit.client.0.smithi001.stdout:3/873: symlink d4/d1e/d7c/d9a/l121 0 2024-02-28T16:31:23.221 INFO:tasks.workunit.client.1.smithi066.stdout:4/638: mknod d1/d7a/cd0 0 2024-02-28T16:31:23.226 INFO:tasks.workunit.client.1.smithi066.stdout:1/478: dread d2/d18/d7a/f73 [0,4194304] 0 2024-02-28T16:31:23.226 INFO:tasks.workunit.client.0.smithi001.stdout:5/831: dread dc/df/d79/f95 [0,4194304] 0 2024-02-28T16:31:23.226 INFO:tasks.workunit.client.1.smithi066.stdout:3/549: rename db/d13/d14/d24 to db/dc/d11/dbb 0 2024-02-28T16:31:23.228 INFO:tasks.workunit.client.0.smithi001.stdout:5/832: chown dc/df 1106428 1 2024-02-28T16:31:23.229 INFO:tasks.workunit.client.0.smithi001.stdout:5/833: chown dc/d1a/d52/dfa 9434809 1 2024-02-28T16:31:23.230 INFO:tasks.workunit.client.0.smithi001.stdout:3/874: dread d4/d6/d7/d78/d81/fb2 [0,4194304] 0 2024-02-28T16:31:23.232 INFO:tasks.workunit.client.1.smithi066.stdout:2/518: creat d0/d12/d1a/d23/d29/d21/d2f/fa2 x:0 0 0 2024-02-28T16:31:23.234 INFO:tasks.workunit.client.0.smithi001.stdout:0/853: dread d18/d22/d2b/d36/d4d/f65 [0,4194304] 0 2024-02-28T16:31:23.234 INFO:tasks.workunit.client.1.smithi066.stdout:5/497: dwrite d3/d4/dc/d56/f75 [0,4194304] 0 2024-02-28T16:31:23.235 INFO:tasks.workunit.client.1.smithi066.stdout:8/464: link d0/d6/dc/d16/d4c/d6c/d6f/f39 d0/db/d12/d8a/f97 0 2024-02-28T16:31:23.237 INFO:tasks.workunit.client.1.smithi066.stdout:0/643: dwrite d2/d27/d41/d76/d93/d33/f62 [0,4194304] 0 2024-02-28T16:31:23.237 INFO:tasks.workunit.client.1.smithi066.stdout:2/519: write d0/d12/f5c [4193575,48590] 0 2024-02-28T16:31:23.237 INFO:tasks.workunit.client.1.smithi066.stdout:4/639: rmdir d1/d23 39 2024-02-28T16:31:23.238 INFO:tasks.workunit.client.0.smithi001.stdout:0/854: truncate d18/d22/d2f/ff3 4840988 0 2024-02-28T16:31:23.240 INFO:tasks.workunit.client.0.smithi001.stdout:0/855: stat d18/d22/d3d/fa1 0 2024-02-28T16:31:23.240 INFO:tasks.workunit.client.0.smithi001.stdout:5/834: rename dc/d12/d56/d96/db9/dc2/dca/d3c/f101 to dc/d1a/d7d/d87/df3/f10d 0 2024-02-28T16:31:23.247 INFO:tasks.workunit.client.1.smithi066.stdout:8/465: readlink d0/db/d12/d38/l90 0 2024-02-28T16:31:23.248 INFO:tasks.workunit.client.1.smithi066.stdout:6/436: dwrite d2/d2c/f4a [0,4194304] 0 2024-02-28T16:31:23.248 INFO:tasks.workunit.client.1.smithi066.stdout:3/550: symlink db/dc/d11/daa/lbc 0 2024-02-28T16:31:23.249 INFO:tasks.workunit.client.0.smithi001.stdout:7/886: dwrite d3/d17/d3d/d59/d6e/fac [0,4194304] 0 2024-02-28T16:31:23.253 INFO:tasks.workunit.client.0.smithi001.stdout:7/887: symlink d3/d17/d3d/d59/l11b 0 2024-02-28T16:31:23.255 INFO:tasks.workunit.client.0.smithi001.stdout:7/888: readlink d3/d17/d61/led 0 2024-02-28T16:31:23.255 INFO:tasks.workunit.client.1.smithi066.stdout:3/551: write db/d13/d14/f6f [460931,108802] 0 2024-02-28T16:31:23.262 INFO:tasks.workunit.client.1.smithi066.stdout:1/479: dwrite d2/f75 [0,4194304] 0 2024-02-28T16:31:23.262 INFO:tasks.workunit.client.1.smithi066.stdout:2/520: creat d0/d12/d3e/d9e/fa3 x:0 0 0 2024-02-28T16:31:23.269 INFO:tasks.workunit.client.1.smithi066.stdout:4/640: creat d1/d7a/fd1 x:0 0 0 2024-02-28T16:31:23.274 INFO:tasks.workunit.client.1.smithi066.stdout:0/644: rmdir d2/d27/d41/d76/d93/db2/db5/d73 39 2024-02-28T16:31:23.281 INFO:tasks.workunit.client.1.smithi066.stdout:6/437: creat d2/d2c/f97 x:0 0 0 2024-02-28T16:31:23.288 INFO:tasks.workunit.client.1.smithi066.stdout:5/498: fsync d3/d4/dc/dd/d2e/f6f 0 2024-02-28T16:31:23.289 INFO:tasks.workunit.client.1.smithi066.stdout:2/521: fsync d0/d12/d3e/f4e 0 2024-02-28T16:31:23.291 INFO:tasks.workunit.client.1.smithi066.stdout:1/480: unlink d2/d27/d2b/d43/d46/d68/f76 0 2024-02-28T16:31:23.299 INFO:tasks.workunit.client.1.smithi066.stdout:6/438: mknod d2/d26/d32/d46/d80/d5d/d63/c98 0 2024-02-28T16:31:23.301 INFO:tasks.workunit.client.0.smithi001.stdout:3/875: write d4/d6/ffe [27929,129308] 0 2024-02-28T16:31:23.302 INFO:tasks.workunit.client.1.smithi066.stdout:3/552: link db/dc/d11/f22 db/fbd 0 2024-02-28T16:31:23.304 INFO:tasks.workunit.client.0.smithi001.stdout:3/876: fdatasync d4/d6/f56 0 2024-02-28T16:31:23.306 INFO:tasks.workunit.client.0.smithi001.stdout:7/889: write d3/d17/d52/f77 [636303,84245] 0 2024-02-28T16:31:23.310 INFO:tasks.workunit.client.1.smithi066.stdout:5/499: fdatasync d3/d4/dc/dd/f89 0 2024-02-28T16:31:23.312 INFO:tasks.workunit.client.0.smithi001.stdout:7/890: creat d3/d17/d3d/d59/d96/f11c x:0 0 0 2024-02-28T16:31:23.312 INFO:tasks.workunit.client.0.smithi001.stdout:5/835: dwrite dc/d12/d56/d96/db9/dc2/dca/f42 [0,4194304] 0 2024-02-28T16:31:23.313 INFO:tasks.workunit.client.1.smithi066.stdout:4/641: dread d1/d23/d25/f99 [0,4194304] 0 2024-02-28T16:31:23.313 INFO:tasks.workunit.client.1.smithi066.stdout:0/645: truncate d2/d27/d41/d76/d93/db2/db5/d51/f70 731185 0 2024-02-28T16:31:23.314 INFO:tasks.workunit.client.1.smithi066.stdout:5/500: chown d3/d4/l13 0 1 2024-02-28T16:31:23.315 INFO:tasks.workunit.client.0.smithi001.stdout:0/856: dread d18/f59 [0,4194304] 0 2024-02-28T16:31:23.315 INFO:tasks.workunit.client.0.smithi001.stdout:7/891: truncate d3/d17/d3d/d42/fe1 1403705 0 2024-02-28T16:31:23.316 INFO:tasks.workunit.client.0.smithi001.stdout:0/857: chown d18/d22/d2f/d1c/d90/ffb 1 1 2024-02-28T16:31:23.317 INFO:tasks.workunit.client.1.smithi066.stdout:3/553: creat db/d2a/d44/d48/d5d/fbe x:0 0 0 2024-02-28T16:31:23.317 INFO:tasks.workunit.client.1.smithi066.stdout:8/466: write d0/f43 [150193,46114] 0 2024-02-28T16:31:23.318 INFO:tasks.workunit.client.1.smithi066.stdout:1/481: dwrite d2/d27/f2f [0,4194304] 0 2024-02-28T16:31:23.321 INFO:tasks.workunit.client.0.smithi001.stdout:0/858: symlink d18/d22/d2b/d36/d4d/l112 0 2024-02-28T16:31:23.321 INFO:tasks.workunit.client.0.smithi001.stdout:8/843: sync 2024-02-28T16:31:23.322 INFO:tasks.workunit.client.0.smithi001.stdout:4/948: sync 2024-02-28T16:31:23.323 INFO:tasks.workunit.client.0.smithi001.stdout:0/859: chown d18/d22/d2f 54774369 1 2024-02-28T16:31:23.324 INFO:tasks.workunit.client.1.smithi066.stdout:2/522: mknod d0/d12/d2e/d3b/ca4 0 2024-02-28T16:31:23.324 INFO:tasks.workunit.client.1.smithi066.stdout:4/642: unlink d1/d23/d25/f6b 0 2024-02-28T16:31:23.325 INFO:tasks.workunit.client.1.smithi066.stdout:8/467: readlink d0/d6/dc/d16/d1a/d63/d58/l67 0 2024-02-28T16:31:23.326 INFO:tasks.workunit.client.1.smithi066.stdout:8/468: fsync d0/f7c 0 2024-02-28T16:31:23.327 INFO:tasks.workunit.client.0.smithi001.stdout:8/844: creat d4/d47/df4/f10e x:0 0 0 2024-02-28T16:31:23.327 INFO:tasks.workunit.client.0.smithi001.stdout:3/877: dwrite d4/d6/f9b [4194304,4194304] 0 2024-02-28T16:31:23.330 INFO:tasks.workunit.client.0.smithi001.stdout:0/860: write d18/d22/d2f/fe9 [679651,92113] 0 2024-02-28T16:31:23.331 INFO:tasks.workunit.client.0.smithi001.stdout:3/878: fsync d4/d6/d7/d78/d81/dd6/dd9/f11a 0 2024-02-28T16:31:23.335 INFO:tasks.workunit.client.1.smithi066.stdout:5/501: creat d3/d4/dc/d2f/d55/d74/d84/f9b x:0 0 0 2024-02-28T16:31:23.336 INFO:tasks.workunit.client.0.smithi001.stdout:7/892: link d3/d17/d52/d7f/d99/dc1/d49/fa8 d3/d17/d52/d7f/d99/dc1/f11d 0 2024-02-28T16:31:23.338 INFO:tasks.workunit.client.0.smithi001.stdout:5/836: dread dc/d12/d56/d96/db9/dc2/dca/d3c/f3d [0,4194304] 0 2024-02-28T16:31:23.339 INFO:tasks.workunit.client.0.smithi001.stdout:3/879: chown d4/d1e/c6c 189203 1 2024-02-28T16:31:23.339 INFO:tasks.workunit.client.1.smithi066.stdout:4/643: rmdir d1/d23/d25/d77 39 2024-02-28T16:31:23.346 INFO:tasks.workunit.client.1.smithi066.stdout:5/502: readlink d3/d4/dc/dd/d41/l78 0 2024-02-28T16:31:23.348 INFO:tasks.workunit.client.1.smithi066.stdout:3/554: dwrite db/d13/d1a/d33/f39 [4194304,4194304] 0 2024-02-28T16:31:23.353 INFO:tasks.workunit.client.1.smithi066.stdout:2/523: link d0/d12/d1a/c24 d0/d12/d1a/d23/d29/d21/d2f/d67/ca5 0 2024-02-28T16:31:23.356 INFO:tasks.workunit.client.1.smithi066.stdout:5/503: mkdir d3/d1b/d7d/d9c 0 2024-02-28T16:31:23.357 INFO:tasks.workunit.client.1.smithi066.stdout:4/644: symlink d1/d18/d7e/dbb/ld2 0 2024-02-28T16:31:23.357 INFO:tasks.workunit.client.1.smithi066.stdout:2/524: creat d0/d12/d1a/d23/fa6 x:0 0 0 2024-02-28T16:31:23.367 INFO:tasks.workunit.client.0.smithi001.stdout:8/845: unlink d4/c88 0 2024-02-28T16:31:23.369 INFO:tasks.workunit.client.1.smithi066.stdout:3/555: link db/dc/d35/d3d/d76/fb9 db/dc/d11/d1f/d78/fbf 0 2024-02-28T16:31:23.369 INFO:tasks.workunit.client.1.smithi066.stdout:5/504: symlink d3/d4/dc/d2f/d55/d74/l9d 0 2024-02-28T16:31:23.370 INFO:tasks.workunit.client.0.smithi001.stdout:7/893: symlink d3/d17/d52/d92/l11e 0 2024-02-28T16:31:23.370 INFO:tasks.workunit.client.0.smithi001.stdout:1/957: sync 2024-02-28T16:31:23.370 INFO:tasks.workunit.client.0.smithi001.stdout:2/803: sync 2024-02-28T16:31:23.370 INFO:tasks.workunit.client.0.smithi001.stdout:6/867: sync 2024-02-28T16:31:23.370 INFO:tasks.workunit.client.1.smithi066.stdout:8/469: dwrite d0/d6/f1f [4194304,4194304] 0 2024-02-28T16:31:23.371 INFO:tasks.workunit.client.0.smithi001.stdout:3/880: unlink d4/f61 0 2024-02-28T16:31:23.372 INFO:tasks.workunit.client.1.smithi066.stdout:4/645: creat d1/d18/d55/dc4/fd3 x:0 0 0 2024-02-28T16:31:23.372 INFO:tasks.workunit.client.0.smithi001.stdout:5/837: fsync dc/d12/d41/db7/fc6 0 2024-02-28T16:31:23.374 INFO:tasks.workunit.client.0.smithi001.stdout:6/868: rmdir d9/db/d5f/dd4 39 2024-02-28T16:31:23.374 INFO:tasks.workunit.client.0.smithi001.stdout:5/838: stat dc/df/f81 0 2024-02-28T16:31:23.375 INFO:tasks.workunit.client.0.smithi001.stdout:2/804: creat d0/d2/d3/de/d61/dd5/f109 x:0 0 0 2024-02-28T16:31:23.377 INFO:tasks.workunit.client.1.smithi066.stdout:1/482: dread d2/d27/d2b/d43/d46/d68/f7e [0,4194304] 0 2024-02-28T16:31:23.377 INFO:tasks.workunit.client.0.smithi001.stdout:0/861: creat d18/d22/d76/f113 x:0 0 0 2024-02-28T16:31:23.378 INFO:tasks.workunit.client.1.smithi066.stdout:3/556: mknod db/dc/d35/d3d/cc0 0 2024-02-28T16:31:23.378 INFO:tasks.workunit.client.0.smithi001.stdout:7/894: mknod d3/d17/d3d/d59/d96/dc9/c11f 0 2024-02-28T16:31:23.380 INFO:tasks.workunit.client.0.smithi001.stdout:1/958: mknod d0/d9/d1a/d2b/d26/d27/d30/d67/c14f 0 2024-02-28T16:31:23.381 INFO:tasks.workunit.client.1.smithi066.stdout:2/525: getdents d0/d12/d2e/d3b/d66 0 2024-02-28T16:31:23.381 INFO:tasks.workunit.client.0.smithi001.stdout:8/846: mkdir d4/d7/d22/d27/d4b/d10f 0 2024-02-28T16:31:23.383 INFO:tasks.workunit.client.0.smithi001.stdout:8/847: chown d4/d32/d69/d9a/lc4 12189675 1 2024-02-28T16:31:23.384 INFO:tasks.workunit.client.1.smithi066.stdout:1/483: rmdir d2/d27/d2b/d37 39 2024-02-28T16:31:23.386 INFO:tasks.workunit.client.0.smithi001.stdout:5/839: chown dc/d1a/d7d/d87/f8f 209624 1 2024-02-28T16:31:23.386 INFO:tasks.workunit.client.1.smithi066.stdout:5/505: mknod d3/d4/dc/dd/d41/d7a/c9e 0 2024-02-28T16:31:23.387 INFO:tasks.workunit.client.0.smithi001.stdout:5/840: readlink dc/d1a/d21/l6e 0 2024-02-28T16:31:23.388 INFO:tasks.workunit.client.0.smithi001.stdout:2/805: rmdir d0/d2/d3/de/d61/d12/d6a/d84 39 2024-02-28T16:31:23.389 INFO:tasks.workunit.client.1.smithi066.stdout:8/470: rename d0/db/d12/d38/c71 to d0/d6/dc/d16/d8c/c98 0 2024-02-28T16:31:23.391 INFO:tasks.workunit.client.1.smithi066.stdout:5/506: dread - d3/d4/dc/dd/f89 zero size 2024-02-28T16:31:23.392 INFO:tasks.workunit.client.0.smithi001.stdout:2/806: chown d0/d2/d3/f6c 13 1 2024-02-28T16:31:23.392 INFO:tasks.workunit.client.0.smithi001.stdout:7/895: chown d3/d17/d52/d7f/d99/c110 3172805 1 2024-02-28T16:31:23.392 INFO:tasks.workunit.client.0.smithi001.stdout:4/949: write d3/d5/d7/d1f/d20/de0/de6/fc5 [344690,105930] 0 2024-02-28T16:31:23.395 INFO:tasks.workunit.client.1.smithi066.stdout:1/484: symlink d2/d27/d2b/d43/d46/d80/l93 0 2024-02-28T16:31:23.396 INFO:tasks.workunit.client.1.smithi066.stdout:2/526: mknod d0/d12/d1a/d39/d7c/d87/ca7 0 2024-02-28T16:31:23.396 INFO:tasks.workunit.client.0.smithi001.stdout:1/959: creat d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/f150 x:0 0 0 2024-02-28T16:31:23.396 INFO:tasks.workunit.client.0.smithi001.stdout:6/869: symlink d9/db/d5f/dd4/dfb/d121/l122 0 2024-02-28T16:31:23.398 INFO:tasks.workunit.client.0.smithi001.stdout:0/862: symlink d18/l114 0 2024-02-28T16:31:23.400 INFO:tasks.workunit.client.1.smithi066.stdout:4/646: getdents d1/d23 0 2024-02-28T16:31:23.401 INFO:tasks.workunit.client.0.smithi001.stdout:5/841: chown dc/f1d 639702 1 2024-02-28T16:31:23.401 INFO:tasks.workunit.client.1.smithi066.stdout:2/527: dread - d0/d12/d3e/d9e/fa3 zero size 2024-02-28T16:31:23.402 INFO:tasks.workunit.client.0.smithi001.stdout:8/848: creat d4/d47/df4/df5/f110 x:0 0 0 2024-02-28T16:31:23.407 INFO:tasks.workunit.client.1.smithi066.stdout:3/557: dwrite db/d13/d1a/d25/f96 [0,4194304] 0 2024-02-28T16:31:23.407 INFO:tasks.workunit.client.0.smithi001.stdout:4/950: dread d3/f7e [0,4194304] 0 2024-02-28T16:31:23.408 INFO:tasks.workunit.client.0.smithi001.stdout:8/849: dread - d4/d7/d22/d27/d58/d94/db4/dd5/fd7 zero size 2024-02-28T16:31:23.408 INFO:tasks.workunit.client.1.smithi066.stdout:8/471: write d0/d6/ff [1285009,22005] 0 2024-02-28T16:31:23.410 INFO:tasks.workunit.client.0.smithi001.stdout:8/850: chown d4/d7/d22/d27/d58 141 1 2024-02-28T16:31:23.411 INFO:tasks.workunit.client.1.smithi066.stdout:4/647: mknod d1/d23/d25/d3e/d40/d9c/cd4 0 2024-02-28T16:31:23.412 INFO:tasks.workunit.client.1.smithi066.stdout:3/558: rmdir db/d13 39 2024-02-28T16:31:23.415 INFO:tasks.workunit.client.0.smithi001.stdout:7/896: getdents d3/d17/d52/d7f/d99 0 2024-02-28T16:31:23.416 INFO:tasks.workunit.client.0.smithi001.stdout:4/951: read - d3/d1c/de8/ffb zero size 2024-02-28T16:31:23.416 INFO:tasks.workunit.client.1.smithi066.stdout:8/472: truncate d0/d6/dc/d16/d4c/d6c/f91 464742 0 2024-02-28T16:31:23.419 INFO:tasks.workunit.client.0.smithi001.stdout:7/897: write d3/d1c/f33 [1870789,14251] 0 2024-02-28T16:31:23.419 INFO:tasks.workunit.client.0.smithi001.stdout:4/952: chown d3/d5/d47/lcd 27318150 1 2024-02-28T16:31:23.422 INFO:tasks.workunit.client.1.smithi066.stdout:4/648: creat d1/d18/d4e/fd5 x:0 0 0 2024-02-28T16:31:23.424 INFO:tasks.workunit.client.0.smithi001.stdout:4/953: write d3/d2d/d4c/d69/f128 [641699,26467] 0 2024-02-28T16:31:23.425 INFO:tasks.workunit.client.0.smithi001.stdout:1/960: rename d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/cbb to d0/d9/d1a/d2b/d26/d27/c151 0 2024-02-28T16:31:23.426 INFO:tasks.workunit.client.1.smithi066.stdout:3/559: write db/d13/d1a/d25/f5f [1876737,105933] 0 2024-02-28T16:31:23.427 INFO:tasks.workunit.client.1.smithi066.stdout:4/649: chown d1/d18/d41/l79 0 1 2024-02-28T16:31:23.429 INFO:tasks.workunit.client.1.smithi066.stdout:2/528: dwrite d0/d12/d1a/d23/d29/d21/d2f/fa2 [0,4194304] 0 2024-02-28T16:31:23.430 INFO:tasks.workunit.client.1.smithi066.stdout:1/485: dwrite d2/d18/d7a/d8d/f8f [0,4194304] 0 2024-02-28T16:31:23.430 INFO:tasks.workunit.client.1.smithi066.stdout:8/473: creat d0/db/d12/d1d/d25/d65/d93/f99 x:0 0 0 2024-02-28T16:31:23.435 INFO:tasks.workunit.client.0.smithi001.stdout:7/898: symlink d3/db7/l120 0 2024-02-28T16:31:23.436 INFO:tasks.workunit.client.0.smithi001.stdout:0/863: dread d18/d22/d2b/d36/d46/df5/d4f/f60 [0,4194304] 0 2024-02-28T16:31:23.439 INFO:tasks.workunit.client.1.smithi066.stdout:3/560: creat db/d2a/d44/d48/d5d/fc1 x:0 0 0 2024-02-28T16:31:23.439 INFO:tasks.workunit.client.1.smithi066.stdout:5/507: dread d3/d1b/d39/f47 [0,4194304] 0 2024-02-28T16:31:23.440 INFO:tasks.workunit.client.1.smithi066.stdout:1/486: truncate d2/d27/d2b/d43/d46/f6c 620866 0 2024-02-28T16:31:23.440 INFO:tasks.workunit.client.0.smithi001.stdout:0/864: mknod d18/d22/d76/d93/c115 0 2024-02-28T16:31:23.442 INFO:tasks.workunit.client.0.smithi001.stdout:1/961: creat d0/d9/d1a/d2b/d26/d27/d30/d67/db7/f152 x:0 0 0 2024-02-28T16:31:23.443 INFO:tasks.workunit.client.0.smithi001.stdout:0/865: write d18/d22/d2b/d36/d48/f55 [1875376,101652] 0 2024-02-28T16:31:23.448 INFO:tasks.workunit.client.0.smithi001.stdout:1/962: fsync d0/d9/d1a/d2b/df8/f131 0 2024-02-28T16:31:23.448 INFO:tasks.workunit.client.0.smithi001.stdout:7/899: dread - d3/d17/d3d/fcc zero size 2024-02-28T16:31:23.450 INFO:tasks.workunit.client.0.smithi001.stdout:5/842: dread dc/d1a/d7d/d87/df3/f10d [0,4194304] 0 2024-02-28T16:31:23.451 INFO:tasks.workunit.client.1.smithi066.stdout:1/487: symlink d2/d27/d2b/d37/d48/l94 0 2024-02-28T16:31:23.454 INFO:tasks.workunit.client.1.smithi066.stdout:3/561: mknod db/dc/db3/cc2 0 2024-02-28T16:31:23.455 INFO:tasks.workunit.client.1.smithi066.stdout:2/529: link d0/d12/d2e/f59 d0/d12/d1a/d23/d29/fa8 0 2024-02-28T16:31:23.455 INFO:tasks.workunit.client.0.smithi001.stdout:8/851: write d4/d7/d22/d27/f30 [1454473,50086] 0 2024-02-28T16:31:23.456 INFO:tasks.workunit.client.0.smithi001.stdout:7/900: symlink d3/d17/df0/l121 0 2024-02-28T16:31:23.459 INFO:tasks.workunit.client.0.smithi001.stdout:8/852: stat d4/d9b/fa7 0 2024-02-28T16:31:23.462 INFO:tasks.workunit.client.1.smithi066.stdout:8/474: truncate d0/db/f80 78269 0 2024-02-28T16:31:23.462 INFO:tasks.workunit.client.1.smithi066.stdout:2/530: truncate d0/d12/d2e/d7a/f99 2960262 0 2024-02-28T16:31:23.463 INFO:tasks.workunit.client.1.smithi066.stdout:2/531: chown d0/d12/d2e/d3b 3 1 2024-02-28T16:31:23.466 INFO:tasks.workunit.client.1.smithi066.stdout:9/401: sync 2024-02-28T16:31:23.466 INFO:tasks.workunit.client.1.smithi066.stdout:7/476: sync 2024-02-28T16:31:23.475 INFO:tasks.workunit.client.0.smithi001.stdout:6/870: dwrite d9/d29/fcc [0,4194304] 0 2024-02-28T16:31:23.476 INFO:tasks.workunit.client.1.smithi066.stdout:1/488: rmdir d2/d27/d2b 39 2024-02-28T16:31:23.477 INFO:tasks.workunit.client.0.smithi001.stdout:3/881: dwrite d4/d1e/dcf/ff7 [0,4194304] 0 2024-02-28T16:31:23.480 INFO:tasks.workunit.client.0.smithi001.stdout:4/954: dwrite d3/d2d/d4c/d69/d82/f102 [0,4194304] 0 2024-02-28T16:31:23.481 INFO:tasks.workunit.client.0.smithi001.stdout:6/871: fsync d9/db/d10/f5b 0 2024-02-28T16:31:23.486 INFO:tasks.workunit.client.1.smithi066.stdout:2/532: truncate d0/f9 2586113 0 2024-02-28T16:31:23.488 INFO:tasks.workunit.client.0.smithi001.stdout:0/866: dwrite d18/d22/d2b/d36/f56 [4194304,4194304] 0 2024-02-28T16:31:23.493 INFO:tasks.workunit.client.1.smithi066.stdout:5/508: dread d3/d4/dc/dd/d2e/d18/f4a [0,4194304] 0 2024-02-28T16:31:23.494 INFO:tasks.workunit.client.1.smithi066.stdout:7/477: rename dc/d29/d5c to dc/de/d87 0 2024-02-28T16:31:23.502 INFO:tasks.workunit.client.0.smithi001.stdout:8/853: dwrite d4/d7/d22/df6/f100 [0,4194304] 0 2024-02-28T16:31:23.502 INFO:tasks.workunit.client.0.smithi001.stdout:4/955: write d3/d5/d7/d1f/d3a/fa4 [2368766,118985] 0 2024-02-28T16:31:23.504 INFO:tasks.workunit.client.1.smithi066.stdout:3/562: dwrite db/d2a/f65 [4194304,4194304] 0 2024-02-28T16:31:23.504 INFO:tasks.workunit.client.1.smithi066.stdout:8/475: dwrite d0/f7c [0,4194304] 0 2024-02-28T16:31:23.505 INFO:tasks.workunit.client.0.smithi001.stdout:3/882: getdents d4/d1e/d7c/d7d/d80 0 2024-02-28T16:31:23.509 INFO:tasks.workunit.client.0.smithi001.stdout:0/867: symlink d18/d22/d2b/d36/d4d/l116 0 2024-02-28T16:31:23.510 INFO:tasks.workunit.client.1.smithi066.stdout:4/650: dread d1/f12 [4194304,4194304] 0 2024-02-28T16:31:23.510 INFO:tasks.workunit.client.0.smithi001.stdout:1/963: dwrite d0/f28 [4194304,4194304] 0 2024-02-28T16:31:23.511 INFO:tasks.workunit.client.1.smithi066.stdout:7/478: fsync dc/de/d17/d37/f72 0 2024-02-28T16:31:23.514 INFO:tasks.workunit.client.0.smithi001.stdout:4/956: chown d3/d2d/d4c/f50 5582 1 2024-02-28T16:31:23.515 INFO:tasks.workunit.client.1.smithi066.stdout:5/509: creat d3/d4/dc/dd/f9f x:0 0 0 2024-02-28T16:31:23.517 INFO:tasks.workunit.client.0.smithi001.stdout:7/901: dwrite d3/d17/d3d/d42/f7d [0,4194304] 0 2024-02-28T16:31:23.517 INFO:tasks.workunit.client.0.smithi001.stdout:5/843: dwrite dc/d12/d56/d96/db9/dc2/dca/d3c/f53 [4194304,4194304] 0 2024-02-28T16:31:23.518 INFO:tasks.workunit.client.1.smithi066.stdout:5/510: fdatasync d3/d1b/f31 0 2024-02-28T16:31:23.520 INFO:tasks.workunit.client.1.smithi066.stdout:8/476: write d0/db/d12/d1d/d25/d4f/f86 [1024887,116495] 0 2024-02-28T16:31:23.522 INFO:tasks.workunit.client.1.smithi066.stdout:2/533: mknod d0/d12/d1a/ca9 0 2024-02-28T16:31:23.527 INFO:tasks.workunit.client.0.smithi001.stdout:5/844: mknod dc/d12/d41/dd6/c10e 0 2024-02-28T16:31:23.528 INFO:tasks.workunit.client.1.smithi066.stdout:7/479: mkdir dc/de/d17/d37/d47/d88 0 2024-02-28T16:31:23.529 INFO:tasks.workunit.client.0.smithi001.stdout:0/868: rename d18/d22/d2b/d36/d46/df5/d4f/f60 to d18/d5a/f117 0 2024-02-28T16:31:23.532 INFO:tasks.workunit.client.0.smithi001.stdout:1/964: creat d0/d9/d1a/f153 x:0 0 0 2024-02-28T16:31:23.534 INFO:tasks.workunit.client.1.smithi066.stdout:9/402: dread da/f35 [0,4194304] 0 2024-02-28T16:31:23.539 INFO:tasks.workunit.client.0.smithi001.stdout:6/872: write d9/db/d5f/d69/f75 [1547834,90300] 0 2024-02-28T16:31:23.540 INFO:tasks.workunit.client.1.smithi066.stdout:4/651: fsync d1/d23/d25/d43/f65 0 2024-02-28T16:31:23.540 INFO:tasks.workunit.client.0.smithi001.stdout:7/902: truncate d3/d17/d52/d7f/d99/dc1/f11d 931365 0 2024-02-28T16:31:23.545 INFO:tasks.workunit.client.1.smithi066.stdout:4/652: read - d1/d7a/fd1 zero size 2024-02-28T16:31:23.546 INFO:tasks.workunit.client.0.smithi001.stdout:6/873: creat d9/d39/dc4/f123 x:0 0 0 2024-02-28T16:31:23.549 INFO:tasks.workunit.client.1.smithi066.stdout:1/489: rename d2/d27/l2d to d2/d27/d2b/d37/d54/d84/l95 0 2024-02-28T16:31:23.551 INFO:tasks.workunit.client.1.smithi066.stdout:7/480: mknod dc/d29/c89 0 2024-02-28T16:31:23.552 INFO:tasks.workunit.client.1.smithi066.stdout:2/534: dwrite d0/d12/d1a/d23/d29/d21/f75 [4194304,4194304] 0 2024-02-28T16:31:23.553 INFO:tasks.workunit.client.1.smithi066.stdout:8/477: link d0/db/d12/d8a/f97 d0/d6/f9a 0 2024-02-28T16:31:23.555 INFO:tasks.workunit.client.1.smithi066.stdout:1/490: readlink d2/d27/d2b/d34/l64 0 2024-02-28T16:31:23.555 INFO:tasks.workunit.client.1.smithi066.stdout:3/563: dwrite db/d2a/d44/d48/d5d/fbe [0,4194304] 0 2024-02-28T16:31:23.561 INFO:tasks.workunit.client.1.smithi066.stdout:4/653: rename d1/f20 to d1/d18/da8/fd6 0 2024-02-28T16:31:23.564 INFO:tasks.workunit.client.0.smithi001.stdout:7/903: creat d3/d17/f122 x:0 0 0 2024-02-28T16:31:23.568 INFO:tasks.workunit.client.0.smithi001.stdout:1/965: link d0/d9/d1a/d2b/d26/d27/f123 d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/f154 0 2024-02-28T16:31:23.569 INFO:tasks.workunit.client.1.smithi066.stdout:4/654: dread - d1/d23/d25/d43/f8b zero size 2024-02-28T16:31:23.573 INFO:tasks.workunit.client.1.smithi066.stdout:8/478: creat d0/db/d12/d1d/d25/d65/f9b x:0 0 0 2024-02-28T16:31:23.574 INFO:tasks.workunit.client.0.smithi001.stdout:6/874: getdents d9/db/dd9 0 2024-02-28T16:31:23.578 INFO:tasks.workunit.client.1.smithi066.stdout:5/511: dwrite d3/d4/dc/f8b [0,4194304] 0 2024-02-28T16:31:23.579 INFO:tasks.workunit.client.0.smithi001.stdout:7/904: symlink d3/d1c/l123 0 2024-02-28T16:31:23.580 INFO:tasks.workunit.client.1.smithi066.stdout:8/479: readlink d0/d6/dc/d16/d1a/d63/d58/l79 0 2024-02-28T16:31:23.581 INFO:tasks.workunit.client.0.smithi001.stdout:3/883: dread d4/d1e/d7c/fd3 [0,4194304] 0 2024-02-28T16:31:23.582 INFO:tasks.workunit.client.0.smithi001.stdout:1/966: rmdir d0/d9/d1a/d2b/d26/d27/dd3/de8/d102 39 2024-02-28T16:31:23.582 INFO:tasks.workunit.client.1.smithi066.stdout:2/535: mknod d0/d12/d1a/d39/d73/caa 0 2024-02-28T16:31:23.583 INFO:tasks.workunit.client.1.smithi066.stdout:3/564: write db/d2a/d44/d48/d5d/fa9 [3060626,21118] 0 2024-02-28T16:31:23.586 INFO:tasks.workunit.client.0.smithi001.stdout:6/875: getdents d9/db/dd9 0 2024-02-28T16:31:23.587 INFO:tasks.workunit.client.1.smithi066.stdout:1/491: dread d2/f75 [0,4194304] 0 2024-02-28T16:31:23.589 INFO:tasks.workunit.client.0.smithi001.stdout:8/854: dread d4/d7/d22/d27/d58/f61 [0,4194304] 0 2024-02-28T16:31:23.593 INFO:tasks.workunit.client.1.smithi066.stdout:6/439: sync 2024-02-28T16:31:23.594 INFO:tasks.workunit.client.1.smithi066.stdout:4/655: rename d1/d23/f75 to d1/d7a/fd7 0 2024-02-28T16:31:23.599 INFO:tasks.workunit.client.1.smithi066.stdout:5/512: mknod d3/d4/dc/dd/d41/d7f/ca0 0 2024-02-28T16:31:23.600 INFO:tasks.workunit.client.0.smithi001.stdout:3/884: mknod d4/d1e/d7c/d7d/df8/d82/c122 0 2024-02-28T16:31:23.603 INFO:tasks.workunit.client.0.smithi001.stdout:6/876: unlink d9/c61 0 2024-02-28T16:31:23.603 INFO:tasks.workunit.client.0.smithi001.stdout:5/845: dread dc/df/f81 [0,4194304] 0 2024-02-28T16:31:23.604 INFO:tasks.workunit.client.1.smithi066.stdout:3/565: mknod db/d2a/d44/d48/d5d/cc3 0 2024-02-28T16:31:23.605 INFO:tasks.workunit.client.1.smithi066.stdout:1/492: creat d2/d27/d2b/d43/f96 x:0 0 0 2024-02-28T16:31:23.606 INFO:tasks.workunit.client.0.smithi001.stdout:8/855: mknod d4/d7/d22/d27/d58/d94/dbc/c111 0 2024-02-28T16:31:23.611 INFO:tasks.workunit.client.1.smithi066.stdout:6/440: rename d2/da/d38 to d2/d26/d32/d46/d80/d5d/d99 0 2024-02-28T16:31:23.612 INFO:tasks.workunit.client.1.smithi066.stdout:9/403: dread da/fc [0,4194304] 0 2024-02-28T16:31:23.612 INFO:tasks.workunit.client.0.smithi001.stdout:2/807: sync 2024-02-28T16:31:23.613 INFO:tasks.workunit.client.1.smithi066.stdout:5/513: mknod d3/d4/dc/d2f/ca1 0 2024-02-28T16:31:23.614 INFO:tasks.workunit.client.0.smithi001.stdout:1/967: truncate d0/d9/d1a/d2b/d26/d27/dd3/de8/d102/fe9 17425 0 2024-02-28T16:31:23.615 INFO:tasks.workunit.client.1.smithi066.stdout:6/441: truncate d2/da/d19/f93 571108 0 2024-02-28T16:31:23.616 INFO:tasks.workunit.client.1.smithi066.stdout:9/404: fsync da/df/d18/d28/d42/d68/f7a 0 2024-02-28T16:31:23.616 INFO:tasks.workunit.client.1.smithi066.stdout:5/514: chown d3/d4/dc/dd/d2e/d18 853 1 2024-02-28T16:31:23.617 INFO:tasks.workunit.client.0.smithi001.stdout:6/877: mkdir d9/db/d5f/d62/d7e/d124 0 2024-02-28T16:31:23.619 INFO:tasks.workunit.client.1.smithi066.stdout:5/515: readlink d3/d4/dc/dd/d2e/l58 0 2024-02-28T16:31:23.619 INFO:tasks.workunit.client.0.smithi001.stdout:0/869: dread d18/d22/d2f/d1c/d90/d9f/fe3 [0,4194304] 0 2024-02-28T16:31:23.622 INFO:tasks.workunit.client.0.smithi001.stdout:2/808: chown d0/d2/d3/de/d61/d12/d55/fb1 24205 1 2024-02-28T16:31:23.625 INFO:tasks.workunit.client.0.smithi001.stdout:1/968: symlink d0/d9/d1a/d2b/d26/d27/d30/d52/l155 0 2024-02-28T16:31:23.627 INFO:tasks.workunit.client.0.smithi001.stdout:5/846: symlink dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/l10f 0 2024-02-28T16:31:23.629 INFO:tasks.workunit.client.0.smithi001.stdout:6/878: mknod d9/d29/d6e/d9a/c125 0 2024-02-28T16:31:23.630 INFO:tasks.workunit.client.0.smithi001.stdout:4/957: dwrite d3/d5/d7/d1f/d2e/dbe/d115/f123 [0,4194304] 0 2024-02-28T16:31:23.632 INFO:tasks.workunit.client.0.smithi001.stdout:6/879: write d9/d29/fcc [4288168,17773] 0 2024-02-28T16:31:23.637 INFO:tasks.workunit.client.1.smithi066.stdout:7/481: dread fa [0,4194304] 0 2024-02-28T16:31:23.640 INFO:tasks.workunit.client.1.smithi066.stdout:4/656: symlink d1/d18/d41/ld8 0 2024-02-28T16:31:23.640 INFO:tasks.workunit.client.1.smithi066.stdout:6/442: symlink d2/d2c/d6e/l9a 0 2024-02-28T16:31:23.640 INFO:tasks.workunit.client.1.smithi066.stdout:9/405: creat da/df/d18/d28/d42/d68/f84 x:0 0 0 2024-02-28T16:31:23.641 INFO:tasks.workunit.client.0.smithi001.stdout:3/885: getdents d4/d18/ddb/dfb 0 2024-02-28T16:31:23.642 INFO:tasks.workunit.client.1.smithi066.stdout:0/646: sync 2024-02-28T16:31:23.643 INFO:tasks.workunit.client.0.smithi001.stdout:7/905: dwrite d3/d17/d52/d7f/d99/dc1/d49/d8d/f97 [0,4194304] 0 2024-02-28T16:31:23.644 INFO:tasks.workunit.client.1.smithi066.stdout:5/516: rename d3/d4/dc/d2f/d55/d74/d84 to d3/d4/dc/dd/d41/da2 0 2024-02-28T16:31:23.644 INFO:tasks.workunit.client.1.smithi066.stdout:6/443: readlink d2/da/d19/lf 0 2024-02-28T16:31:23.645 INFO:tasks.workunit.client.1.smithi066.stdout:3/566: mknod db/dc/cc4 0 2024-02-28T16:31:23.647 INFO:tasks.workunit.client.1.smithi066.stdout:1/493: mknod d2/c97 0 2024-02-28T16:31:23.648 INFO:tasks.workunit.client.0.smithi001.stdout:8/856: dread d4/d47/fa0 [0,4194304] 0 2024-02-28T16:31:23.651 INFO:tasks.workunit.client.1.smithi066.stdout:4/657: fsync d1/d18/d41/dba/fc6 0 2024-02-28T16:31:23.651 INFO:tasks.workunit.client.1.smithi066.stdout:1/494: chown d2/d18/f6e 1201 1 2024-02-28T16:31:23.653 INFO:tasks.workunit.client.1.smithi066.stdout:0/647: mknod d2/d27/d41/d76/cdb 0 2024-02-28T16:31:23.656 INFO:tasks.workunit.client.1.smithi066.stdout:0/648: read d2/f28 [440423,73826] 0 2024-02-28T16:31:23.659 INFO:tasks.workunit.client.0.smithi001.stdout:0/870: rename d18/d22/d2b/d36/d46/ca7 to d18/d22/d2b/c118 0 2024-02-28T16:31:23.660 INFO:tasks.workunit.client.1.smithi066.stdout:3/567: symlink db/dc/d11/dbb/lc5 0 2024-02-28T16:31:23.660 INFO:tasks.workunit.client.0.smithi001.stdout:7/906: read d3/d17/f19 [602672,21355] 0 2024-02-28T16:31:23.661 INFO:tasks.workunit.client.0.smithi001.stdout:6/880: creat d9/db/d5f/d62/def/f126 x:0 0 0 2024-02-28T16:31:23.663 INFO:tasks.workunit.client.0.smithi001.stdout:0/871: write d18/d5a/f109 [622791,96818] 0 2024-02-28T16:31:23.665 INFO:tasks.workunit.client.1.smithi066.stdout:1/495: readlink d2/d27/d2b/l6a 0 2024-02-28T16:31:23.667 INFO:tasks.workunit.client.1.smithi066.stdout:0/649: mknod d2/d27/d41/d76/cdc 0 2024-02-28T16:31:23.668 INFO:tasks.workunit.client.0.smithi001.stdout:6/881: mkdir d9/db/d5f/d62/d127 0 2024-02-28T16:31:23.671 INFO:tasks.workunit.client.1.smithi066.stdout:5/517: link d3/d1b/f37 d3/d4/dc/d2f/d55/fa3 0 2024-02-28T16:31:23.672 INFO:tasks.workunit.client.0.smithi001.stdout:6/882: stat d9/d29/d6e/f2f 0 2024-02-28T16:31:23.674 INFO:tasks.workunit.client.0.smithi001.stdout:0/872: fsync d18/d22/d76/d7a/fe2 0 2024-02-28T16:31:23.674 INFO:tasks.workunit.client.1.smithi066.stdout:3/568: fsync db/fbd 0 2024-02-28T16:31:23.676 INFO:tasks.workunit.client.1.smithi066.stdout:7/482: getdents dc/de 0 2024-02-28T16:31:23.677 INFO:tasks.workunit.client.0.smithi001.stdout:7/907: getdents d3/d17/d61 0 2024-02-28T16:31:23.679 INFO:tasks.workunit.client.1.smithi066.stdout:0/650: chown d2/d27/c87 1191422 1 2024-02-28T16:31:23.681 INFO:tasks.workunit.client.0.smithi001.stdout:6/883: unlink d9/de/feb 0 2024-02-28T16:31:23.682 INFO:tasks.workunit.client.1.smithi066.stdout:6/444: dwrite d2/da/f3b [0,4194304] 0 2024-02-28T16:31:23.683 INFO:tasks.workunit.client.0.smithi001.stdout:0/873: rmdir d18/d22/d76/d7a 39 2024-02-28T16:31:23.686 INFO:tasks.workunit.client.1.smithi066.stdout:9/406: dread da/d22/d46/d56/f72 [0,4194304] 0 2024-02-28T16:31:23.686 INFO:tasks.workunit.client.1.smithi066.stdout:4/658: mkdir d1/d23/d25/dd9 0 2024-02-28T16:31:23.688 INFO:tasks.workunit.client.0.smithi001.stdout:8/857: dread d4/d7/f35 [0,4194304] 0 2024-02-28T16:31:23.688 INFO:tasks.workunit.client.1.smithi066.stdout:7/483: creat dc/d56/f8a x:0 0 0 2024-02-28T16:31:23.688 INFO:tasks.workunit.client.1.smithi066.stdout:8/480: truncate d0/d6/dc/d16/d4c/d6c/d6f/f39 1362386 0 2024-02-28T16:31:23.693 INFO:tasks.workunit.client.1.smithi066.stdout:9/407: write da/df/f29 [843041,100440] 0 2024-02-28T16:31:23.694 INFO:tasks.workunit.client.0.smithi001.stdout:7/908: symlink d3/d17/d52/l124 0 2024-02-28T16:31:23.694 INFO:tasks.workunit.client.0.smithi001.stdout:4/958: dread d3/d2d/d83/ff4 [0,4194304] 0 2024-02-28T16:31:23.696 INFO:tasks.workunit.client.0.smithi001.stdout:2/809: dwrite d0/d2/d3/de/d8a/f51 [0,4194304] 0 2024-02-28T16:31:23.696 INFO:tasks.workunit.client.1.smithi066.stdout:9/408: fdatasync da/d3a/f13 0 2024-02-28T16:31:23.700 INFO:tasks.workunit.client.0.smithi001.stdout:6/884: symlink d9/d29/d6e/d53/l128 0 2024-02-28T16:31:23.700 INFO:tasks.workunit.client.0.smithi001.stdout:1/969: dwrite d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/f8c [4194304,4194304] 0 2024-02-28T16:31:23.702 INFO:tasks.workunit.client.1.smithi066.stdout:8/481: dread - d0/d6/dc/d16/d1a/d63/d3d/f62 zero size 2024-02-28T16:31:23.703 INFO:tasks.workunit.client.1.smithi066.stdout:4/659: mknod d1/d18/d55/cda 0 2024-02-28T16:31:23.706 INFO:tasks.workunit.client.1.smithi066.stdout:2/536: dwrite d0/d12/d2e/d7a/f5b [0,4194304] 0 2024-02-28T16:31:23.708 INFO:tasks.workunit.client.0.smithi001.stdout:3/886: dwrite d4/d18/f10 [0,4194304] 0 2024-02-28T16:31:23.712 INFO:tasks.workunit.client.0.smithi001.stdout:1/970: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d67/db7/f138 0 2024-02-28T16:31:23.712 INFO:tasks.workunit.client.0.smithi001.stdout:5/847: dwrite dc/d12/d41/f9c [0,4194304] 0 2024-02-28T16:31:23.715 INFO:tasks.workunit.client.0.smithi001.stdout:7/909: truncate d3/d17/df0/f11a 842894 0 2024-02-28T16:31:23.716 INFO:tasks.workunit.client.1.smithi066.stdout:3/569: getdents db/dc/d11/d1f 0 2024-02-28T16:31:23.716 INFO:tasks.workunit.client.1.smithi066.stdout:5/518: getdents d3/d4/dc/d42 0 2024-02-28T16:31:23.717 INFO:tasks.workunit.client.0.smithi001.stdout:5/848: chown dc/d12/d56/d96 11489463 1 2024-02-28T16:31:23.719 INFO:tasks.workunit.client.0.smithi001.stdout:0/874: rename d18/d22/d2f/f24 to d18/d22/d2b/d36/f119 0 2024-02-28T16:31:23.723 INFO:tasks.workunit.client.0.smithi001.stdout:5/849: unlink dc/d1a/d52/ld5 0 2024-02-28T16:31:23.723 INFO:tasks.workunit.client.1.smithi066.stdout:4/660: symlink d1/d23/d25/d3e/d40/ldb 0 2024-02-28T16:31:23.723 INFO:tasks.workunit.client.1.smithi066.stdout:8/482: creat d0/d6/dc/d16/d4c/d42/f9c x:0 0 0 2024-02-28T16:31:23.726 INFO:tasks.workunit.client.0.smithi001.stdout:7/910: dread d3/d17/d52/d7f/d99/dc1/d49/d8d/fab [0,4194304] 0 2024-02-28T16:31:23.727 INFO:tasks.workunit.client.1.smithi066.stdout:2/537: symlink d0/d12/d1a/d39/lab 0 2024-02-28T16:31:23.727 INFO:tasks.workunit.client.0.smithi001.stdout:5/850: fsync dc/d12/d41/d54/fa5 0 2024-02-28T16:31:23.729 INFO:tasks.workunit.client.1.smithi066.stdout:0/651: dread d2/d27/d41/d76/d93/d10/f26 [0,4194304] 0 2024-02-28T16:31:23.733 INFO:tasks.workunit.client.1.smithi066.stdout:0/652: stat d2/d27/c3e 0 2024-02-28T16:31:23.733 INFO:tasks.workunit.client.1.smithi066.stdout:8/483: write d0/d6/dc/d16/d1a/d63/d58/f70 [3680817,88468] 0 2024-02-28T16:31:23.733 INFO:tasks.workunit.client.0.smithi001.stdout:1/971: read d0/f21 [170437,81372] 0 2024-02-28T16:31:23.733 INFO:tasks.workunit.client.0.smithi001.stdout:7/911: chown d3/d17/d3d/d59/d6e/fec 25 1 2024-02-28T16:31:23.736 INFO:tasks.workunit.client.0.smithi001.stdout:5/851: rmdir dc/d12/d41/db7 39 2024-02-28T16:31:23.737 INFO:tasks.workunit.client.0.smithi001.stdout:7/912: stat d3/dc5 0 2024-02-28T16:31:23.737 INFO:tasks.workunit.client.1.smithi066.stdout:1/496: dwrite d2/d18/f6e [0,4194304] 0 2024-02-28T16:31:23.737 INFO:tasks.workunit.client.1.smithi066.stdout:2/538: rmdir d0/d12/d1a/d39/d73 39 2024-02-28T16:31:23.738 INFO:tasks.workunit.client.0.smithi001.stdout:0/875: rename d18/d22/d57 to d18/d22/d2b/d36/d48/d11a 0 2024-02-28T16:31:23.742 INFO:tasks.workunit.client.1.smithi066.stdout:0/653: chown d2/d27/d41/d76/d93/d10/d23/f7a 50 1 2024-02-28T16:31:23.744 INFO:tasks.workunit.client.0.smithi001.stdout:5/852: unlink fa 0 2024-02-28T16:31:23.746 INFO:tasks.workunit.client.1.smithi066.stdout:0/654: dread - d2/d27/d41/d76/d93/d33/d3f/fbb zero size 2024-02-28T16:31:23.747 INFO:tasks.workunit.client.0.smithi001.stdout:6/885: dwrite d9/d29/fcc [0,4194304] 0 2024-02-28T16:31:23.747 INFO:tasks.workunit.client.1.smithi066.stdout:4/661: getdents d1/d18/d55/dc4 0 2024-02-28T16:31:23.751 INFO:tasks.workunit.client.0.smithi001.stdout:7/913: fsync d3/d17/d52/d7f/d99/dc1/fd7 0 2024-02-28T16:31:23.752 INFO:tasks.workunit.client.1.smithi066.stdout:1/497: creat d2/d27/d2b/d43/d46/d80/f98 x:0 0 0 2024-02-28T16:31:23.752 INFO:tasks.workunit.client.1.smithi066.stdout:8/484: chown d0/d6/f9a 240 1 2024-02-28T16:31:23.755 INFO:tasks.workunit.client.1.smithi066.stdout:3/570: dread db/dc/d35/f58 [0,4194304] 0 2024-02-28T16:31:23.756 INFO:tasks.workunit.client.1.smithi066.stdout:4/662: write d1/d23/d25/d3e/f57 [3220125,41930] 0 2024-02-28T16:31:23.759 INFO:tasks.workunit.client.0.smithi001.stdout:5/853: symlink dc/d1a/d21/de2/l110 0 2024-02-28T16:31:23.761 INFO:tasks.workunit.client.1.smithi066.stdout:1/498: creat d2/d27/d2b/d37/d51/f99 x:0 0 0 2024-02-28T16:31:23.761 INFO:tasks.workunit.client.1.smithi066.stdout:7/484: dread dc/de/d17/f5b [0,4194304] 0 2024-02-28T16:31:23.763 INFO:tasks.workunit.client.1.smithi066.stdout:1/499: chown d2/d27/d2b/d43/c50 132 1 2024-02-28T16:31:23.764 INFO:tasks.workunit.client.1.smithi066.stdout:4/663: truncate d1/f12 173928 0 2024-02-28T16:31:23.767 INFO:tasks.workunit.client.1.smithi066.stdout:8/485: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d 0 2024-02-28T16:31:23.769 INFO:tasks.workunit.client.1.smithi066.stdout:2/539: dwrite d0/f71 [0,4194304] 0 2024-02-28T16:31:23.770 INFO:tasks.workunit.client.1.smithi066.stdout:7/485: symlink dc/de/d87/l8b 0 2024-02-28T16:31:23.770 INFO:tasks.workunit.client.1.smithi066.stdout:3/571: write db/d2a/d44/d48/d5d/fc1 [13206,30480] 0 2024-02-28T16:31:23.773 INFO:tasks.workunit.client.1.smithi066.stdout:4/664: symlink d1/d18/d55/dc4/ldc 0 2024-02-28T16:31:23.773 INFO:tasks.workunit.client.0.smithi001.stdout:8/858: dwrite d4/fa1 [0,4194304] 0 2024-02-28T16:31:23.773 INFO:tasks.workunit.client.0.smithi001.stdout:2/810: dwrite d0/d2/d3/d36/f39 [0,4194304] 0 2024-02-28T16:31:23.776 INFO:tasks.workunit.client.1.smithi066.stdout:0/655: dread d2/d27/d41/d76/d93/d10/d23/f9d [0,4194304] 0 2024-02-28T16:31:23.776 INFO:tasks.workunit.client.1.smithi066.stdout:8/486: mknod d0/db/d12/d1d/c9e 0 2024-02-28T16:31:23.779 INFO:tasks.workunit.client.1.smithi066.stdout:4/665: chown d1/d23/d25/d3e/f3f 1687329621 1 2024-02-28T16:31:23.780 INFO:tasks.workunit.client.1.smithi066.stdout:6/445: dread d2/d26/d32/d46/d80/d5d/d99/f85 [0,4194304] 0 2024-02-28T16:31:23.782 INFO:tasks.workunit.client.1.smithi066.stdout:4/666: fsync d1/d23/d25/d43/fa6 0 2024-02-28T16:31:23.783 INFO:tasks.workunit.client.1.smithi066.stdout:6/446: readlink d2/da/d19/lf 0 2024-02-28T16:31:23.783 INFO:tasks.workunit.client.1.smithi066.stdout:3/572: creat db/d2a/d44/fc6 x:0 0 0 2024-02-28T16:31:23.783 INFO:tasks.workunit.client.1.smithi066.stdout:1/500: read d2/d27/f5b [2435075,39826] 0 2024-02-28T16:31:23.783 INFO:tasks.workunit.client.0.smithi001.stdout:4/959: dread d3/d1c/da0/fb2 [0,4194304] 0 2024-02-28T16:31:23.786 INFO:tasks.workunit.client.1.smithi066.stdout:4/667: read d1/f1e [1369459,103424] 0 2024-02-28T16:31:23.786 INFO:tasks.workunit.client.1.smithi066.stdout:0/656: readlink d2/d27/d41/d76/d93/db2/db5/d73/lb0 0 2024-02-28T16:31:23.789 INFO:tasks.workunit.client.1.smithi066.stdout:1/501: creat d2/d18/d7a/f9a x:0 0 0 2024-02-28T16:31:23.789 INFO:tasks.workunit.client.1.smithi066.stdout:4/668: fdatasync d1/d18/d4e/fd5 0 2024-02-28T16:31:23.790 INFO:tasks.workunit.client.1.smithi066.stdout:6/447: write d2/d2c/d72/f7b [463642,79523] 0 2024-02-28T16:31:23.793 INFO:tasks.workunit.client.1.smithi066.stdout:4/669: readlink d1/d18/da8/lb3 0 2024-02-28T16:31:23.793 INFO:tasks.workunit.client.1.smithi066.stdout:3/573: creat db/d13/d14/dad/fc7 x:0 0 0 2024-02-28T16:31:23.795 INFO:tasks.workunit.client.1.smithi066.stdout:6/448: mknod d2/d2c/d72/c9b 0 2024-02-28T16:31:23.796 INFO:tasks.workunit.client.0.smithi001.stdout:0/876: dread d18/d5a/f63 [0,4194304] 0 2024-02-28T16:31:23.797 INFO:tasks.workunit.client.1.smithi066.stdout:1/502: creat d2/d27/d2b/d43/f9b x:0 0 0 2024-02-28T16:31:23.799 INFO:tasks.workunit.client.1.smithi066.stdout:6/449: fsync d2/da/fe 0 2024-02-28T16:31:23.800 INFO:tasks.workunit.client.1.smithi066.stdout:3/574: mknod db/dc/d11/daa/cc8 0 2024-02-28T16:31:23.800 INFO:tasks.workunit.client.1.smithi066.stdout:0/657: truncate d2/d27/d41/d76/d93/d10/d23/f9d 1536129 0 2024-02-28T16:31:23.801 INFO:tasks.workunit.client.0.smithi001.stdout:6/886: chown d9/d29/f3b 405099401 1 2024-02-28T16:31:23.807 INFO:tasks.workunit.client.1.smithi066.stdout:3/575: chown db/d2a/d44/d59 9981247 1 2024-02-28T16:31:23.809 INFO:tasks.workunit.client.1.smithi066.stdout:3/576: chown db/dc/d11/dbb 10919 1 2024-02-28T16:31:23.813 INFO:tasks.workunit.client.1.smithi066.stdout:3/577: dread - db/dc/d35/d3d/d76/d89/f9f zero size 2024-02-28T16:31:23.816 INFO:tasks.workunit.client.1.smithi066.stdout:6/450: rename d2/d2a to d2/da/d19/d9c 0 2024-02-28T16:31:23.816 INFO:tasks.workunit.client.1.smithi066.stdout:3/578: readlink db/dc/d11/l4d 0 2024-02-28T16:31:23.819 INFO:tasks.workunit.client.1.smithi066.stdout:3/579: write db/d2a/d44/d59/f94 [917585,118919] 0 2024-02-28T16:31:23.828 INFO:tasks.workunit.client.1.smithi066.stdout:1/503: link d2/d27/d2b/d37/f82 d2/d27/d2b/d43/f9c 0 2024-02-28T16:31:23.828 INFO:tasks.workunit.client.1.smithi066.stdout:6/451: unlink d2/da/d19/c25 0 2024-02-28T16:31:23.832 INFO:tasks.workunit.client.0.smithi001.stdout:3/887: write d4/f76 [3336598,64654] 0 2024-02-28T16:31:23.833 INFO:tasks.workunit.client.1.smithi066.stdout:3/580: write db/d13/d14/d2d/f9e [3127226,125179] 0 2024-02-28T16:31:23.836 INFO:tasks.workunit.client.0.smithi001.stdout:7/914: rmdir d3/d17/d52/d7f/d99/dc1/db6 39 2024-02-28T16:31:23.836 INFO:tasks.workunit.client.1.smithi066.stdout:0/658: dwrite d2/d27/d41/d76/d93/d10/d23/d6d/faf [0,4194304] 0 2024-02-28T16:31:23.838 INFO:tasks.workunit.client.0.smithi001.stdout:7/915: chown d3/d17/d61/cb2 228543 1 2024-02-28T16:31:23.843 INFO:tasks.workunit.client.1.smithi066.stdout:3/581: mkdir db/dc/db3/dc9 0 2024-02-28T16:31:23.844 INFO:tasks.workunit.client.1.smithi066.stdout:3/582: write db/d2a/f37 [1512980,27358] 0 2024-02-28T16:31:23.847 INFO:tasks.workunit.client.0.smithi001.stdout:2/811: fsync d0/d2/d3/d36/fd4 0 2024-02-28T16:31:23.851 INFO:tasks.workunit.client.1.smithi066.stdout:0/659: unlink d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/fcd 0 2024-02-28T16:31:23.852 INFO:tasks.workunit.client.0.smithi001.stdout:3/888: dwrite d4/d18/f69 [0,4194304] 0 2024-02-28T16:31:23.854 INFO:tasks.workunit.client.1.smithi066.stdout:1/504: dread d2/d18/d7a/f73 [0,4194304] 0 2024-02-28T16:31:23.854 INFO:tasks.workunit.client.0.smithi001.stdout:0/877: symlink d18/d22/d3d/l11b 0 2024-02-28T16:31:23.854 INFO:tasks.workunit.client.1.smithi066.stdout:6/452: dwrite d2/da/d19/f41 [0,4194304] 0 2024-02-28T16:31:23.855 INFO:tasks.workunit.client.0.smithi001.stdout:6/887: creat d9/d39/f129 x:0 0 0 2024-02-28T16:31:23.865 INFO:tasks.workunit.client.0.smithi001.stdout:7/916: symlink d3/d17/d3d/d59/d96/l125 0 2024-02-28T16:31:23.868 INFO:tasks.workunit.client.1.smithi066.stdout:6/453: dread - d2/da/f1b zero size 2024-02-28T16:31:23.869 INFO:tasks.workunit.client.0.smithi001.stdout:5/854: symlink dc/d12/d56/d96/db9/dc2/dca/d3c/l111 0 2024-02-28T16:31:23.873 INFO:tasks.workunit.client.1.smithi066.stdout:0/660: creat d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/fdd x:0 0 0 2024-02-28T16:31:23.874 INFO:tasks.workunit.client.0.smithi001.stdout:8/859: getdents d4/d47/df4/d104 0 2024-02-28T16:31:23.876 INFO:tasks.workunit.client.1.smithi066.stdout:3/583: creat db/dc/fca x:0 0 0 2024-02-28T16:31:23.877 INFO:tasks.workunit.client.0.smithi001.stdout:4/960: fsync d3/d1c/d58/f28 0 2024-02-28T16:31:23.884 INFO:tasks.workunit.client.0.smithi001.stdout:1/972: getdents d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d 0 2024-02-28T16:31:23.885 INFO:tasks.workunit.client.1.smithi066.stdout:6/454: creat d2/d2c/d6e/f9d x:0 0 0 2024-02-28T16:31:23.887 INFO:tasks.workunit.client.0.smithi001.stdout:7/917: dread - d3/d17/d3d/d59/d6e/fec zero size 2024-02-28T16:31:23.887 INFO:tasks.workunit.client.1.smithi066.stdout:6/455: stat d2/d26/d32/c65 0 2024-02-28T16:31:23.889 INFO:tasks.workunit.client.0.smithi001.stdout:5/855: rmdir dc/d12/d56/d96/db9/dc2/dca/d91 39 2024-02-28T16:31:23.890 INFO:tasks.workunit.client.1.smithi066.stdout:3/584: readlink db/dc/d35/lb8 0 2024-02-28T16:31:23.895 INFO:tasks.workunit.client.0.smithi001.stdout:1/973: dread d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/f83 [0,4194304] 0 2024-02-28T16:31:23.896 INFO:tasks.workunit.client.1.smithi066.stdout:6/456: chown d2/d26/d32/d46/d80/d5d/d63/c98 1004275543 1 2024-02-28T16:31:23.896 INFO:tasks.workunit.client.1.smithi066.stdout:9/409: dwrite da/df/f3d [0,4194304] 0 2024-02-28T16:31:23.896 INFO:tasks.workunit.client.0.smithi001.stdout:3/889: fsync d4/f44 0 2024-02-28T16:31:23.899 INFO:tasks.workunit.client.0.smithi001.stdout:3/890: chown d4/d1e/dcf/f105 92777570 1 2024-02-28T16:31:23.902 INFO:tasks.workunit.client.0.smithi001.stdout:3/891: chown d4/d6/d7/d23/cb4 12 1 2024-02-28T16:31:23.904 INFO:tasks.workunit.client.1.smithi066.stdout:3/585: mkdir db/d2a/d44/d48/d52/dcb 0 2024-02-28T16:31:23.907 INFO:tasks.workunit.client.0.smithi001.stdout:1/974: mkdir d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/d156 0 2024-02-28T16:31:23.907 INFO:tasks.workunit.client.1.smithi066.stdout:6/457: dread d2/d2c/d72/f7b [0,4194304] 0 2024-02-28T16:31:23.911 INFO:tasks.workunit.client.0.smithi001.stdout:7/918: dwrite d3/d17/d52/d55/dca/f107 [0,4194304] 0 2024-02-28T16:31:23.915 INFO:tasks.workunit.client.1.smithi066.stdout:4/670: write d1/d7a/fcd [1993273,94391] 0 2024-02-28T16:31:23.916 INFO:tasks.workunit.client.0.smithi001.stdout:1/975: creat d0/d9/d1a/d2b/d26/d27/d30/d67/dd6/f157 x:0 0 0 2024-02-28T16:31:23.918 INFO:tasks.workunit.client.0.smithi001.stdout:3/892: mknod d4/d1e/d7c/d7d/df8/d82/dea/c123 0 2024-02-28T16:31:23.918 INFO:tasks.workunit.client.1.smithi066.stdout:6/458: truncate d2/da/d19/d75/d88/f8d 230247 0 2024-02-28T16:31:23.921 INFO:tasks.workunit.client.1.smithi066.stdout:3/586: rename db/dc/d11/dbb to db/dc/d35/d3d/d76/d89/dcc 0 2024-02-28T16:31:23.923 INFO:tasks.workunit.client.1.smithi066.stdout:4/671: write d1/f33 [1510983,89355] 0 2024-02-28T16:31:23.923 INFO:tasks.workunit.client.1.smithi066.stdout:6/459: readlink d2/da/d19/lf 0 2024-02-28T16:31:23.927 INFO:tasks.workunit.client.1.smithi066.stdout:9/410: dwrite da/df/f3d [0,4194304] 0 2024-02-28T16:31:23.928 INFO:tasks.workunit.client.0.smithi001.stdout:3/893: symlink d4/d6/d7/l124 0 2024-02-28T16:31:23.930 INFO:tasks.workunit.client.0.smithi001.stdout:3/894: chown d4/d1e/f68 0 1 2024-02-28T16:31:23.930 INFO:tasks.workunit.client.1.smithi066.stdout:8/487: write d0/db/d12/d1d/d25/d65/f9b [919063,130371] 0 2024-02-28T16:31:23.932 INFO:tasks.workunit.client.0.smithi001.stdout:7/919: creat d3/d17/d3d/f126 x:0 0 0 2024-02-28T16:31:23.933 INFO:tasks.workunit.client.1.smithi066.stdout:4/672: rmdir d1/d18/da8 39 2024-02-28T16:31:23.933 INFO:tasks.workunit.client.1.smithi066.stdout:3/587: creat db/dc/db3/fcd x:0 0 0 2024-02-28T16:31:23.934 INFO:tasks.workunit.client.1.smithi066.stdout:2/540: dwrite d0/d12/d1a/d39/f42 [0,4194304] 0 2024-02-28T16:31:23.938 INFO:tasks.workunit.client.1.smithi066.stdout:2/541: readlink d0/d12/d2e/d3b/d66/l8a 0 2024-02-28T16:31:23.940 INFO:tasks.workunit.client.0.smithi001.stdout:3/895: write d4/d18/d5f/f59 [2358890,89835] 0 2024-02-28T16:31:23.940 INFO:tasks.workunit.client.0.smithi001.stdout:7/920: creat d3/d17/d52/d55/dca/f127 x:0 0 0 2024-02-28T16:31:23.941 INFO:tasks.workunit.client.0.smithi001.stdout:1/976: dwrite d0/d9/d1a/f153 [0,4194304] 0 2024-02-28T16:31:23.941 INFO:tasks.workunit.client.1.smithi066.stdout:9/411: unlink da/d22/l24 0 2024-02-28T16:31:23.942 INFO:tasks.workunit.client.1.smithi066.stdout:8/488: chown d0/d6/dc/d16/d4c/d42/f9c 0 1 2024-02-28T16:31:23.944 INFO:tasks.workunit.client.1.smithi066.stdout:6/460: creat d2/d26/f9e x:0 0 0 2024-02-28T16:31:23.945 INFO:tasks.workunit.client.1.smithi066.stdout:3/588: symlink db/d13/d1a/lce 0 2024-02-28T16:31:23.946 INFO:tasks.workunit.client.0.smithi001.stdout:3/896: write d4/d1e/d7c/d7d/d80/f10a [331610,27641] 0 2024-02-28T16:31:23.949 INFO:tasks.workunit.client.1.smithi066.stdout:4/673: creat d1/d23/d25/d3e/d40/fdd x:0 0 0 2024-02-28T16:31:23.950 INFO:tasks.workunit.client.0.smithi001.stdout:5/856: dread dc/d12/f32 [0,4194304] 0 2024-02-28T16:31:23.952 INFO:tasks.workunit.client.1.smithi066.stdout:6/461: stat d2/da/d19/d75/f92 0 2024-02-28T16:31:23.953 INFO:tasks.workunit.client.0.smithi001.stdout:8/860: mknod d4/d47/c112 0 2024-02-28T16:31:23.955 INFO:tasks.workunit.client.1.smithi066.stdout:6/462: fdatasync d2/da/fe 0 2024-02-28T16:31:23.956 INFO:tasks.workunit.client.0.smithi001.stdout:8/861: truncate d4/d7/d22/ffa 443320 0 2024-02-28T16:31:23.958 INFO:tasks.workunit.client.0.smithi001.stdout:7/921: readlink d3/d17/d52/d55/dca/ld4 0 2024-02-28T16:31:23.958 INFO:tasks.workunit.client.0.smithi001.stdout:1/977: symlink d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/l158 0 2024-02-28T16:31:23.960 INFO:tasks.workunit.client.0.smithi001.stdout:3/897: rename d4/d1e/d7c/d9a/f11d to d4/d18/d5f/f125 0 2024-02-28T16:31:23.962 INFO:tasks.workunit.client.1.smithi066.stdout:2/542: rename d0/d12/d1a/d39/d7c/d87/ca7 to d0/d12/d3e/cac 0 2024-02-28T16:31:23.963 INFO:tasks.workunit.client.0.smithi001.stdout:5/857: mkdir dc/d1a/d7d/d87/df3/d112 0 2024-02-28T16:31:23.963 INFO:tasks.workunit.client.1.smithi066.stdout:0/661: dwrite d2/d27/d41/d76/d93/d10/f39 [4194304,4194304] 0 2024-02-28T16:31:23.964 INFO:tasks.workunit.client.1.smithi066.stdout:8/489: creat d0/db/d12/d1d/d25/d4f/f9f x:0 0 0 2024-02-28T16:31:23.969 INFO:tasks.workunit.client.1.smithi066.stdout:9/412: truncate da/d3a/d15/f19 487489 0 2024-02-28T16:31:23.970 INFO:tasks.workunit.client.0.smithi001.stdout:1/978: truncate d0/d9/d1a/d2b/d26/fcb 689719 0 2024-02-28T16:31:23.976 INFO:tasks.workunit.client.0.smithi001.stdout:8/862: dwrite d4/d7/d22/d27/d4b/d77/ff7 [0,4194304] 0 2024-02-28T16:31:23.976 INFO:tasks.workunit.client.0.smithi001.stdout:7/922: symlink d3/d17/d52/d7f/d99/dc1/d49/l128 0 2024-02-28T16:31:23.977 INFO:tasks.workunit.client.1.smithi066.stdout:6/463: mknod d2/d26/c9f 0 2024-02-28T16:31:23.977 INFO:tasks.workunit.client.1.smithi066.stdout:8/490: chown d0/d6/dc/d16/d1a/d63/d58/d5a/f64 375504432 1 2024-02-28T16:31:23.980 INFO:tasks.workunit.client.1.smithi066.stdout:4/674: truncate d1/d23/d25/d77/fa3 4735936 0 2024-02-28T16:31:23.981 INFO:tasks.workunit.client.1.smithi066.stdout:9/413: getdents da/df/d18/d28/d42 0 2024-02-28T16:31:23.981 INFO:tasks.workunit.client.1.smithi066.stdout:6/464: chown d2/da/f3b 1107865958 1 2024-02-28T16:31:23.985 INFO:tasks.workunit.client.1.smithi066.stdout:8/491: creat d0/db/d12/d8a/fa0 x:0 0 0 2024-02-28T16:31:23.987 INFO:tasks.workunit.client.0.smithi001.stdout:1/979: fdatasync d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d8e/f119 0 2024-02-28T16:31:23.988 INFO:tasks.workunit.client.1.smithi066.stdout:9/414: symlink da/df/d18/d2d/d81/l85 0 2024-02-28T16:31:23.988 INFO:tasks.workunit.client.1.smithi066.stdout:0/662: truncate d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/fbf 494244 0 2024-02-28T16:31:23.989 INFO:tasks.workunit.client.1.smithi066.stdout:6/465: creat d2/da/d19/d75/d88/fa0 x:0 0 0 2024-02-28T16:31:23.989 INFO:tasks.workunit.client.0.smithi001.stdout:8/863: mknod d4/d32/d69/d9a/c113 0 2024-02-28T16:31:23.994 INFO:tasks.workunit.client.1.smithi066.stdout:0/663: mknod d2/d27/d41/dc5/cde 0 2024-02-28T16:31:23.996 INFO:tasks.workunit.client.0.smithi001.stdout:5/858: dwrite dc/d12/d56/d96/db9/dc2/dca/d91/f103 [0,4194304] 0 2024-02-28T16:31:23.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:23 smithi066 ceph-mon[36236]: pgmap v55: 65 pgs: 65 active+clean; 2.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 59 MiB/s rd, 221 MiB/s wr, 410 op/s 2024-02-28T16:31:23.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:23 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:31:24.000 INFO:tasks.workunit.client.0.smithi001.stdout:1/980: dread - d0/d9/d1a/d2b/d26/d27/d30/d4a/f125 zero size 2024-02-28T16:31:24.003 INFO:tasks.workunit.client.0.smithi001.stdout:8/864: mknod d4/d7/d22/d27/d58/d62/c114 0 2024-02-28T16:31:24.004 INFO:tasks.workunit.client.1.smithi066.stdout:6/466: rename d2/d26/d32/d46/d80 to d2/d2c/da1 0 2024-02-28T16:31:24.006 INFO:tasks.workunit.client.1.smithi066.stdout:6/467: dread - d2/f6f zero size 2024-02-28T16:31:24.012 INFO:tasks.workunit.client.1.smithi066.stdout:5/519: sync 2024-02-28T16:31:24.015 INFO:tasks.workunit.client.1.smithi066.stdout:2/543: write d0/d12/d1a/d23/d29/d21/d2f/f5f [873519,52210] 0 2024-02-28T16:31:24.016 INFO:tasks.workunit.client.0.smithi001.stdout:6/888: write d9/d29/d6e/f18 [3209231,38040] 0 2024-02-28T16:31:24.017 INFO:tasks.workunit.client.1.smithi066.stdout:8/492: rename d0/d6/dc/d16/d4c/d42/d73/l7e to d0/db/d12/d1d/d25/d4f/la1 0 2024-02-28T16:31:24.017 INFO:tasks.workunit.client.1.smithi066.stdout:4/675: dwrite d1/d7a/f98 [0,4194304] 0 2024-02-28T16:31:24.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:23 smithi001 ceph-mon[28991]: pgmap v55: 65 pgs: 65 active+clean; 2.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 59 MiB/s rd, 221 MiB/s wr, 410 op/s 2024-02-28T16:31:24.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:23 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:31:24.020 INFO:tasks.workunit.client.0.smithi001.stdout:4/961: truncate d3/d5/fe2 2301278 0 2024-02-28T16:31:24.021 INFO:tasks.workunit.client.1.smithi066.stdout:9/415: getdents da/d3a/d15/d48 0 2024-02-28T16:31:24.023 INFO:tasks.workunit.client.0.smithi001.stdout:5/859: dread dc/df/d79/f95 [0,4194304] 0 2024-02-28T16:31:24.026 INFO:tasks.workunit.client.1.smithi066.stdout:2/544: fsync d0/d12/d3e/d9e/fa3 0 2024-02-28T16:31:24.026 INFO:tasks.workunit.client.1.smithi066.stdout:4/676: write d1/d7a/f90 [861764,65563] 0 2024-02-28T16:31:24.031 INFO:tasks.workunit.client.1.smithi066.stdout:5/520: mknod d3/d4/dc/dd/d41/d5d/ca4 0 2024-02-28T16:31:24.036 INFO:tasks.workunit.client.1.smithi066.stdout:3/589: dwrite db/fbd [0,4194304] 0 2024-02-28T16:31:24.037 INFO:tasks.workunit.client.1.smithi066.stdout:6/468: dwrite d2/d2c/f8e [0,4194304] 0 2024-02-28T16:31:24.045 INFO:tasks.workunit.client.1.smithi066.stdout:1/505: sync 2024-02-28T16:31:24.045 INFO:tasks.workunit.client.1.smithi066.stdout:7/486: sync 2024-02-28T16:31:24.045 INFO:tasks.workunit.client.1.smithi066.stdout:9/416: rename da/df/d18/d28/c38 to da/df/d18/d28/d42/c86 0 2024-02-28T16:31:24.049 INFO:tasks.workunit.client.1.smithi066.stdout:9/417: stat da/f20 0 2024-02-28T16:31:24.050 INFO:tasks.workunit.client.1.smithi066.stdout:2/545: symlink d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/lad 0 2024-02-28T16:31:24.050 INFO:tasks.workunit.client.1.smithi066.stdout:1/506: write d2/d18/f3a [2310011,73184] 0 2024-02-28T16:31:24.051 INFO:tasks.workunit.client.0.smithi001.stdout:0/878: rmdir d18/d5a 39 2024-02-28T16:31:24.054 INFO:tasks.workunit.client.0.smithi001.stdout:0/879: chown d18/d22/d2b/d36/d48/f55 20212742 1 2024-02-28T16:31:24.056 INFO:tasks.workunit.client.0.smithi001.stdout:8/865: dread d4/d7/d22/d27/d58/f7b [0,4194304] 0 2024-02-28T16:31:24.062 INFO:tasks.workunit.client.0.smithi001.stdout:3/898: rename d4/d6/cb3 to d4/d1e/d7c/d7d/df8/c126 0 2024-02-28T16:31:24.064 INFO:tasks.workunit.client.1.smithi066.stdout:5/521: symlink d3/d1b/la5 0 2024-02-28T16:31:24.064 INFO:tasks.workunit.client.0.smithi001.stdout:7/923: dwrite d3/d17/d61/fd3 [0,4194304] 0 2024-02-28T16:31:24.064 INFO:tasks.workunit.client.1.smithi066.stdout:0/664: dread d2/d27/d41/d76/d93/db2/db5/d73/fa8 [0,4194304] 0 2024-02-28T16:31:24.068 INFO:tasks.workunit.client.1.smithi066.stdout:2/546: chown d0/d12/d3e/d9e/la0 205499132 1 2024-02-28T16:31:24.068 INFO:tasks.workunit.client.0.smithi001.stdout:8/866: symlink d4/d32/d69/d9a/dcd/l115 0 2024-02-28T16:31:24.069 INFO:tasks.workunit.client.1.smithi066.stdout:1/507: creat d2/d27/d2b/d43/f9d x:0 0 0 2024-02-28T16:31:24.070 INFO:tasks.workunit.client.1.smithi066.stdout:2/547: chown d0/d12/d1a/d23/d29/l49 1 1 2024-02-28T16:31:24.070 INFO:tasks.workunit.client.0.smithi001.stdout:5/860: write dc/d12/d41/d54/f71 [127058,11652] 0 2024-02-28T16:31:24.071 INFO:tasks.workunit.client.1.smithi066.stdout:8/493: write d0/db/f80 [1117669,85644] 0 2024-02-28T16:31:24.073 INFO:tasks.workunit.client.1.smithi066.stdout:7/487: symlink dc/de/d17/d1e/l8c 0 2024-02-28T16:31:24.074 INFO:tasks.workunit.client.0.smithi001.stdout:1/981: dread d0/d9/d1a/f33 [0,4194304] 0 2024-02-28T16:31:24.076 INFO:tasks.workunit.client.1.smithi066.stdout:6/469: dwrite d2/da/d19/f93 [0,4194304] 0 2024-02-28T16:31:24.077 INFO:tasks.workunit.client.0.smithi001.stdout:6/889: rename d9/cd7 to d9/de/dbd/c12a 0 2024-02-28T16:31:24.078 INFO:tasks.workunit.client.0.smithi001.stdout:7/924: read d3/d17/d3d/d5d/f73 [886514,46986] 0 2024-02-28T16:31:24.083 INFO:tasks.workunit.client.0.smithi001.stdout:4/962: dwrite d3/d2d/ff1 [0,4194304] 0 2024-02-28T16:31:24.086 INFO:tasks.workunit.client.0.smithi001.stdout:7/925: mknod d3/d17/d3d/db3/dff/c129 0 2024-02-28T16:31:24.086 INFO:tasks.workunit.client.0.smithi001.stdout:1/982: dread d0/d9/d1a/d2b/d26/d27/d30/d88/fef [0,4194304] 0 2024-02-28T16:31:24.091 INFO:tasks.workunit.client.1.smithi066.stdout:5/522: creat d3/d1b/d39/fa6 x:0 0 0 2024-02-28T16:31:24.094 INFO:tasks.workunit.client.1.smithi066.stdout:9/418: dwrite da/df/d18/d28/d42/f80 [0,4194304] 0 2024-02-28T16:31:24.095 INFO:tasks.workunit.client.0.smithi001.stdout:1/983: fdatasync d0/d9/d1a/d2b/d26/d27/d30/f111 0 2024-02-28T16:31:24.095 INFO:tasks.workunit.client.0.smithi001.stdout:7/926: truncate d3/d17/d61/f71 2066937 0 2024-02-28T16:31:24.096 INFO:tasks.workunit.client.0.smithi001.stdout:0/880: truncate d18/d22/d2b/f41 1533623 0 2024-02-28T16:31:24.098 INFO:tasks.workunit.client.1.smithi066.stdout:9/419: write da/df/f29 [158422,44168] 0 2024-02-28T16:31:24.100 INFO:tasks.workunit.client.0.smithi001.stdout:1/984: chown d0/d9/d1a/d2b/d26/d27/d30/d67/db7 18058 1 2024-02-28T16:31:24.103 INFO:tasks.workunit.client.1.smithi066.stdout:4/677: getdents d1/d18/d55 0 2024-02-28T16:31:24.104 INFO:tasks.workunit.client.1.smithi066.stdout:9/420: write da/df/d18/d28/f2b [1707369,113462] 0 2024-02-28T16:31:24.105 INFO:tasks.workunit.client.0.smithi001.stdout:5/861: dread dc/d12/d56/d96/db9/dc2/dca/f44 [0,4194304] 0 2024-02-28T16:31:24.105 INFO:tasks.workunit.client.0.smithi001.stdout:0/881: creat d18/d22/d2b/d36/d48/f11c x:0 0 0 2024-02-28T16:31:24.106 INFO:tasks.workunit.client.1.smithi066.stdout:1/508: truncate d2/d27/d2b/d43/d46/d68/f7e 4252935 0 2024-02-28T16:31:24.107 INFO:tasks.workunit.client.1.smithi066.stdout:4/678: read - d1/d7a/fd1 zero size 2024-02-28T16:31:24.108 INFO:tasks.workunit.client.0.smithi001.stdout:7/927: creat d3/d17/d52/d55/dca/dfe/f12a x:0 0 0 2024-02-28T16:31:24.109 INFO:tasks.workunit.client.1.smithi066.stdout:2/548: rename d0/d12/d1a/f9a to d0/d12/d1a/d39/d7c/fae 0 2024-02-28T16:31:24.110 INFO:tasks.workunit.client.1.smithi066.stdout:6/470: dwrite d2/d26/f5f [4194304,4194304] 0 2024-02-28T16:31:24.111 INFO:tasks.workunit.client.0.smithi001.stdout:4/963: write d3/d2d/d83/fb9 [866971,115428] 0 2024-02-28T16:31:24.112 INFO:tasks.workunit.client.1.smithi066.stdout:3/590: truncate db/d13/f23 587616 0 2024-02-28T16:31:24.113 INFO:tasks.workunit.client.0.smithi001.stdout:3/899: dwrite d4/d1e/db7/fc1 [0,4194304] 0 2024-02-28T16:31:24.115 INFO:tasks.workunit.client.1.smithi066.stdout:2/549: write d0/d12/d2e/d3b/d66/f7f [1936389,40306] 0 2024-02-28T16:31:24.118 INFO:tasks.workunit.client.0.smithi001.stdout:3/900: fsync d4/d1e/dcf/ff7 0 2024-02-28T16:31:24.120 INFO:tasks.workunit.client.1.smithi066.stdout:8/494: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2 0 2024-02-28T16:31:24.123 INFO:tasks.workunit.client.1.smithi066.stdout:4/679: mknod d1/d7a/cde 0 2024-02-28T16:31:24.124 INFO:tasks.workunit.client.1.smithi066.stdout:1/509: rmdir d2/d27/d2b/d37/d54/d84 39 2024-02-28T16:31:24.124 INFO:tasks.workunit.client.1.smithi066.stdout:6/471: dread - d2/da/f1e zero size 2024-02-28T16:31:24.135 INFO:tasks.workunit.client.1.smithi066.stdout:3/591: creat db/d13/d1a/d25/fcf x:0 0 0 2024-02-28T16:31:24.135 INFO:tasks.workunit.client.1.smithi066.stdout:2/550: mkdir d0/d12/d1a/d39/d7c/d87/daf 0 2024-02-28T16:31:24.136 INFO:tasks.workunit.client.1.smithi066.stdout:8/495: dread d0/f7c [0,4194304] 0 2024-02-28T16:31:24.136 INFO:tasks.workunit.client.1.smithi066.stdout:6/472: mknod d2/d26/d32/d46/ca2 0 2024-02-28T16:31:24.138 INFO:tasks.workunit.client.1.smithi066.stdout:7/488: getdents dc 0 2024-02-28T16:31:24.139 INFO:tasks.workunit.client.1.smithi066.stdout:2/551: stat d0/d12/d3e/l83 0 2024-02-28T16:31:24.139 INFO:tasks.workunit.client.1.smithi066.stdout:3/592: dread - db/dc/fca zero size 2024-02-28T16:31:24.142 INFO:tasks.workunit.client.1.smithi066.stdout:6/473: read - d2/da/d19/d9c/f3f zero size 2024-02-28T16:31:24.144 INFO:tasks.workunit.client.1.smithi066.stdout:3/593: write db/d13/d14/d2d/f9e [3321593,126013] 0 2024-02-28T16:31:24.144 INFO:tasks.workunit.client.1.smithi066.stdout:8/496: mknod d0/db/ca3 0 2024-02-28T16:31:24.145 INFO:tasks.workunit.client.1.smithi066.stdout:6/474: stat d2/d2c/da1/d5d/f64 0 2024-02-28T16:31:24.146 INFO:tasks.workunit.client.1.smithi066.stdout:8/497: chown d0/db/d12/f4b 58 1 2024-02-28T16:31:24.148 INFO:tasks.workunit.client.1.smithi066.stdout:6/475: dread - d2/d2c/da1/f91 zero size 2024-02-28T16:31:24.150 INFO:tasks.workunit.client.1.smithi066.stdout:3/594: creat db/dc/d11/d1f/fd0 x:0 0 0 2024-02-28T16:31:24.155 INFO:tasks.workunit.client.1.smithi066.stdout:3/595: mkdir db/d13/d1a/d25/dd1 0 2024-02-28T16:31:24.156 INFO:tasks.workunit.client.1.smithi066.stdout:6/476: read d2/d26/f57 [4344589,29726] 0 2024-02-28T16:31:24.160 INFO:tasks.workunit.client.1.smithi066.stdout:3/596: chown db/d13/d1a/f4e 4172 1 2024-02-28T16:31:24.162 INFO:tasks.workunit.client.1.smithi066.stdout:7/489: dwrite f5 [4194304,4194304] 0 2024-02-28T16:31:24.164 INFO:tasks.workunit.client.1.smithi066.stdout:2/552: dwrite d0/d12/d2e/d3b/d66/f92 [0,4194304] 0 2024-02-28T16:31:24.167 INFO:tasks.workunit.client.1.smithi066.stdout:6/477: creat d2/d26/fa3 x:0 0 0 2024-02-28T16:31:24.168 INFO:tasks.workunit.client.1.smithi066.stdout:3/597: write db/d13/d14/f6f [280319,129562] 0 2024-02-28T16:31:24.173 INFO:tasks.workunit.client.1.smithi066.stdout:3/598: read - db/dc/d35/d3d/d76/d89/f9f zero size 2024-02-28T16:31:24.175 INFO:tasks.workunit.client.1.smithi066.stdout:2/553: chown d0/d12/d3e/f4e 920 1 2024-02-28T16:31:24.175 INFO:tasks.workunit.client.1.smithi066.stdout:6/478: rename d2/da/d19/d9c/l66 to d2/da/d19/d75/d88/la4 0 2024-02-28T16:31:24.176 INFO:tasks.workunit.client.1.smithi066.stdout:7/490: truncate dc/d29/f34 2720130 0 2024-02-28T16:31:24.179 INFO:tasks.workunit.client.1.smithi066.stdout:8/498: dwrite d0/d6/ff [0,4194304] 0 2024-02-28T16:31:24.183 INFO:tasks.workunit.client.1.smithi066.stdout:1/510: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:24.184 INFO:tasks.workunit.client.1.smithi066.stdout:2/554: creat d0/d12/d1a/d23/d29/d21/fb0 x:0 0 0 2024-02-28T16:31:24.186 INFO:tasks.workunit.client.1.smithi066.stdout:7/491: creat dc/de/d17/d37/f8d x:0 0 0 2024-02-28T16:31:24.187 INFO:tasks.workunit.client.1.smithi066.stdout:8/499: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/fa4 x:0 0 0 2024-02-28T16:31:24.191 INFO:tasks.workunit.client.1.smithi066.stdout:6/479: dread d2/da/f21 [0,4194304] 0 2024-02-28T16:31:24.192 INFO:tasks.workunit.client.0.smithi001.stdout:2/812: sync 2024-02-28T16:31:24.195 INFO:tasks.workunit.client.1.smithi066.stdout:8/500: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/fa5 x:0 0 0 2024-02-28T16:31:24.199 INFO:tasks.workunit.client.1.smithi066.stdout:6/480: chown d2/d26/d32/l87 156 1 2024-02-28T16:31:24.201 INFO:tasks.workunit.client.0.smithi001.stdout:7/928: read d3/d17/d52/d7f/d99/faa [2411277,60826] 0 2024-02-28T16:31:24.204 INFO:tasks.workunit.client.0.smithi001.stdout:8/867: dwrite d4/d7/d22/d27/d58/f75 [4194304,4194304] 0 2024-02-28T16:31:24.215 INFO:tasks.workunit.client.0.smithi001.stdout:6/890: truncate d9/db/d19/f78 1829821 0 2024-02-28T16:31:24.216 INFO:tasks.workunit.client.1.smithi066.stdout:2/555: dwrite d0/d12/d2e/f9d [0,4194304] 0 2024-02-28T16:31:24.225 INFO:tasks.workunit.client.1.smithi066.stdout:5/523: truncate d3/d1b/f2d 1528331 0 2024-02-28T16:31:24.228 INFO:tasks.workunit.client.1.smithi066.stdout:5/524: write d3/d4/dc/dd/d41/f65 [4299309,97828] 0 2024-02-28T16:31:24.233 INFO:tasks.workunit.client.1.smithi066.stdout:2/556: link d0/d12/d2e/f84 d0/d12/d2e/d3b/fb1 0 2024-02-28T16:31:24.236 INFO:tasks.workunit.client.0.smithi001.stdout:3/901: fdatasync d4/d6/d7/f43 0 2024-02-28T16:31:24.237 INFO:tasks.workunit.client.1.smithi066.stdout:3/599: rmdir db 39 2024-02-28T16:31:24.239 INFO:tasks.workunit.client.0.smithi001.stdout:1/985: dwrite d0/d9/d1a/d2b/d26/f34 [8388608,4194304] 0 2024-02-28T16:31:24.240 INFO:tasks.workunit.client.1.smithi066.stdout:2/557: creat d0/d12/d3e/d9e/fb2 x:0 0 0 2024-02-28T16:31:24.241 INFO:tasks.workunit.client.0.smithi001.stdout:0/882: dwrite d18/d22/d2b/d36/d48/f55 [4194304,4194304] 0 2024-02-28T16:31:24.243 INFO:tasks.workunit.client.0.smithi001.stdout:4/964: dwrite d3/fd6 [4194304,4194304] 0 2024-02-28T16:31:24.246 INFO:tasks.workunit.client.0.smithi001.stdout:7/929: creat d3/d17/d52/d7f/d99/dc1/f12b x:0 0 0 2024-02-28T16:31:24.249 INFO:tasks.workunit.client.1.smithi066.stdout:1/511: truncate d2/d18/d19/f1c 3869120 0 2024-02-28T16:31:24.250 INFO:tasks.workunit.client.1.smithi066.stdout:3/600: truncate db/d13/d14/d2d/f3e 64572 0 2024-02-28T16:31:24.250 INFO:tasks.workunit.client.0.smithi001.stdout:7/930: dread - d3/d17/d3d/d59/d96/f11c zero size 2024-02-28T16:31:24.252 INFO:tasks.workunit.client.0.smithi001.stdout:4/965: fsync d3/d2d/d4c/d5e/f62 0 2024-02-28T16:31:24.253 INFO:tasks.workunit.client.0.smithi001.stdout:3/902: mkdir d4/d6/d7/d78/d81/dd6/d127 0 2024-02-28T16:31:24.255 INFO:tasks.workunit.client.1.smithi066.stdout:1/512: chown d2/d18/l40 32391522 1 2024-02-28T16:31:24.260 INFO:tasks.workunit.client.0.smithi001.stdout:2/813: dwrite d0/d2/d3/de/d61/d12/fa7 [0,4194304] 0 2024-02-28T16:31:24.266 INFO:tasks.workunit.client.0.smithi001.stdout:8/868: truncate d4/d32/dd6/fe6 3490305 0 2024-02-28T16:31:24.269 INFO:tasks.workunit.client.0.smithi001.stdout:0/883: write d18/d22/d76/f8b [1045708,58276] 0 2024-02-28T16:31:24.269 INFO:tasks.workunit.client.1.smithi066.stdout:3/601: symlink db/dc/d11/d1f/ld2 0 2024-02-28T16:31:24.271 INFO:tasks.workunit.client.0.smithi001.stdout:7/931: truncate d3/d17/f10e 1280207 0 2024-02-28T16:31:24.273 INFO:tasks.workunit.client.0.smithi001.stdout:8/869: rmdir d4/d32/d69 39 2024-02-28T16:31:24.274 INFO:tasks.workunit.client.0.smithi001.stdout:6/891: dread d9/db/f13 [0,4194304] 0 2024-02-28T16:31:24.274 INFO:tasks.workunit.client.1.smithi066.stdout:7/492: dwrite dc/f4b [0,4194304] 0 2024-02-28T16:31:24.275 INFO:tasks.workunit.client.1.smithi066.stdout:3/602: chown db/dc/d11/daa/cc8 5879108 1 2024-02-28T16:31:24.278 INFO:tasks.workunit.client.1.smithi066.stdout:8/501: dwrite d0/d6/f1f [0,4194304] 0 2024-02-28T16:31:24.280 INFO:tasks.workunit.client.1.smithi066.stdout:3/603: write db/d2a/f37 [1527294,106743] 0 2024-02-28T16:31:24.281 INFO:tasks.workunit.client.0.smithi001.stdout:4/966: dread d3/d1c/da0/fd0 [0,4194304] 0 2024-02-28T16:31:24.283 INFO:tasks.workunit.client.0.smithi001.stdout:8/870: creat d4/d47/d8b/d108/f116 x:0 0 0 2024-02-28T16:31:24.285 INFO:tasks.workunit.client.1.smithi066.stdout:7/493: dread - dc/d56/f8a zero size 2024-02-28T16:31:24.286 INFO:tasks.workunit.client.0.smithi001.stdout:6/892: creat d9/db/d5f/dd4/dfb/f12b x:0 0 0 2024-02-28T16:31:24.287 INFO:tasks.workunit.client.1.smithi066.stdout:9/421: sync 2024-02-28T16:31:24.294 INFO:tasks.workunit.client.1.smithi066.stdout:6/481: dwrite d2/d26/f2d [0,4194304] 0 2024-02-28T16:31:24.294 INFO:tasks.workunit.client.1.smithi066.stdout:2/558: dread d0/d12/d3e/f63 [0,4194304] 0 2024-02-28T16:31:24.296 INFO:tasks.workunit.client.0.smithi001.stdout:6/893: creat d9/d29/d6e/d53/f12c x:0 0 0 2024-02-28T16:31:24.299 INFO:tasks.workunit.client.1.smithi066.stdout:1/513: dwrite d2/d27/d2b/d43/d46/d80/f87 [0,4194304] 0 2024-02-28T16:31:24.302 INFO:tasks.workunit.client.0.smithi001.stdout:4/967: creat d3/d5/d7/d1f/d2e/dde/f13a x:0 0 0 2024-02-28T16:31:24.303 INFO:tasks.workunit.client.0.smithi001.stdout:6/894: mknod d9/db/d5f/d62/d7e/c12d 0 2024-02-28T16:31:24.309 INFO:tasks.workunit.client.0.smithi001.stdout:7/932: dwrite d3/d17/d3d/d59/d96/dc9/f112 [0,4194304] 0 2024-02-28T16:31:24.310 INFO:tasks.workunit.client.1.smithi066.stdout:2/559: mknod d0/d12/d1a/d23/cb3 0 2024-02-28T16:31:24.312 INFO:tasks.workunit.client.0.smithi001.stdout:8/871: dwrite d4/d7/d22/d27/f30 [0,4194304] 0 2024-02-28T16:31:24.312 INFO:tasks.workunit.client.1.smithi066.stdout:1/514: mkdir d2/d27/d2b/d34/d9e 0 2024-02-28T16:31:24.313 INFO:tasks.workunit.client.1.smithi066.stdout:2/560: chown d0/d12/d1a/d39 334438 1 2024-02-28T16:31:24.318 INFO:tasks.workunit.client.0.smithi001.stdout:4/968: symlink d3/d5/d7/d1f/d20/de0/de6/l13b 0 2024-02-28T16:31:24.324 INFO:tasks.workunit.client.0.smithi001.stdout:7/933: symlink d3/d17/d3d/db3/dff/l12c 0 2024-02-28T16:31:24.324 INFO:tasks.workunit.client.1.smithi066.stdout:3/604: dread db/d13/d14/d2d/f3e [0,4194304] 0 2024-02-28T16:31:24.325 INFO:tasks.workunit.client.1.smithi066.stdout:5/525: dread d3/d4/dc/dd/d35/f64 [0,4194304] 0 2024-02-28T16:31:24.325 INFO:tasks.workunit.client.0.smithi001.stdout:4/969: read - d3/d2d/d4c/d69/d82/fef zero size 2024-02-28T16:31:24.327 INFO:tasks.workunit.client.0.smithi001.stdout:4/970: stat d3/d1c/da0/fc7 0 2024-02-28T16:31:24.328 INFO:tasks.workunit.client.0.smithi001.stdout:3/903: dread d4/d18/f19 [0,4194304] 0 2024-02-28T16:31:24.329 INFO:tasks.workunit.client.0.smithi001.stdout:6/895: link d9/db/d5f/d62/d7e/c12d d9/db/d5f/d62/d7e/c12e 0 2024-02-28T16:31:24.331 INFO:tasks.workunit.client.1.smithi066.stdout:2/561: creat d0/d12/d2e/d3b/fb4 x:0 0 0 2024-02-28T16:31:24.332 INFO:tasks.workunit.client.1.smithi066.stdout:0/665: sync 2024-02-28T16:31:24.332 INFO:tasks.workunit.client.1.smithi066.stdout:4/680: sync 2024-02-28T16:31:24.333 INFO:tasks.workunit.client.0.smithi001.stdout:4/971: truncate d3/d5/d7/d1f/d2e/dbe/d115/f123 4226212 0 2024-02-28T16:31:24.336 INFO:tasks.workunit.client.0.smithi001.stdout:6/896: symlink d9/db/d5f/dd4/dfb/d121/l12f 0 2024-02-28T16:31:24.337 INFO:tasks.workunit.client.1.smithi066.stdout:5/526: write d3/d4/dc/d56/f67 [2530905,105740] 0 2024-02-28T16:31:24.342 INFO:tasks.workunit.client.0.smithi001.stdout:3/904: creat d4/d6/d7/d78/d81/da1/f128 x:0 0 0 2024-02-28T16:31:24.342 INFO:tasks.workunit.client.1.smithi066.stdout:3/605: write db/d2a/d44/d48/d5d/fbe [2976671,71683] 0 2024-02-28T16:31:24.343 INFO:tasks.workunit.client.1.smithi066.stdout:6/482: dwrite d2/d2c/da1/d5d/d63/f84 [0,4194304] 0 2024-02-28T16:31:24.348 INFO:tasks.workunit.client.0.smithi001.stdout:3/905: dread d4/d6/d7/d78/d81/fb2 [0,4194304] 0 2024-02-28T16:31:24.353 INFO:tasks.workunit.client.1.smithi066.stdout:4/681: read - d1/d18/f8c zero size 2024-02-28T16:31:24.355 INFO:tasks.workunit.client.1.smithi066.stdout:8/502: dwrite d0/db/d12/d8a/f97 [0,4194304] 0 2024-02-28T16:31:24.363 INFO:tasks.workunit.client.1.smithi066.stdout:6/483: unlink d2/l16 0 2024-02-28T16:31:24.363 INFO:tasks.workunit.client.1.smithi066.stdout:9/422: dread da/f30 [0,4194304] 0 2024-02-28T16:31:24.364 INFO:tasks.workunit.client.1.smithi066.stdout:3/606: creat db/dc/d11/daa/fd3 x:0 0 0 2024-02-28T16:31:24.369 INFO:tasks.workunit.client.1.smithi066.stdout:5/527: creat d3/d4/dc/d56/d93/fa7 x:0 0 0 2024-02-28T16:31:24.369 INFO:tasks.workunit.client.1.smithi066.stdout:9/423: readlink da/d22/l2e 0 2024-02-28T16:31:24.377 INFO:tasks.workunit.client.1.smithi066.stdout:6/484: creat d2/d2c/d72/fa5 x:0 0 0 2024-02-28T16:31:24.378 INFO:tasks.workunit.client.0.smithi001.stdout:0/884: dwrite d18/d22/d2b/d95/fb0 [0,4194304] 0 2024-02-28T16:31:24.378 INFO:tasks.workunit.client.1.smithi066.stdout:1/515: write d2/d27/d2b/d37/d48/f81 [1376091,24632] 0 2024-02-28T16:31:24.381 INFO:tasks.workunit.client.0.smithi001.stdout:0/885: readlink d18/d22/d2f/d1c/l111 0 2024-02-28T16:31:24.381 INFO:tasks.workunit.client.0.smithi001.stdout:1/986: dread d0/d9/fa [4194304,4194304] 0 2024-02-28T16:31:24.383 INFO:tasks.workunit.client.1.smithi066.stdout:8/503: symlink d0/db/d12/d1d/la6 0 2024-02-28T16:31:24.384 INFO:tasks.workunit.client.1.smithi066.stdout:5/528: symlink d3/d4/dc/d56/la8 0 2024-02-28T16:31:24.386 INFO:tasks.workunit.client.0.smithi001.stdout:1/987: chown d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/d7d/dae/d10e/d9c 6 1 2024-02-28T16:31:24.390 INFO:tasks.workunit.client.1.smithi066.stdout:8/504: read d0/d6/f9a [3654993,81047] 0 2024-02-28T16:31:24.392 INFO:tasks.workunit.client.1.smithi066.stdout:6/485: mkdir d2/da/d19/d75/da6 0 2024-02-28T16:31:24.393 INFO:tasks.workunit.client.1.smithi066.stdout:9/424: fsync da/d3a/f47 0 2024-02-28T16:31:24.393 INFO:tasks.workunit.client.0.smithi001.stdout:3/906: dread d4/f76 [4194304,4194304] 0 2024-02-28T16:31:24.396 INFO:tasks.workunit.client.1.smithi066.stdout:3/607: dwrite db/d2a/d44/d48/d5d/fc1 [0,4194304] 0 2024-02-28T16:31:24.398 INFO:tasks.workunit.client.0.smithi001.stdout:8/872: dwrite d4/d7/d22/d27/d58/d94/fb5 [4194304,4194304] 0 2024-02-28T16:31:24.399 INFO:tasks.workunit.client.1.smithi066.stdout:9/425: chown da/d22/d46 80 1 2024-02-28T16:31:24.402 INFO:tasks.workunit.client.1.smithi066.stdout:9/426: chown da/d22/f51 10713 1 2024-02-28T16:31:24.403 INFO:tasks.workunit.client.0.smithi001.stdout:4/972: dwrite d3/d2d/d4c/d5e/f12a [0,4194304] 0 2024-02-28T16:31:24.405 INFO:tasks.workunit.client.0.smithi001.stdout:4/973: chown d3/d5/d7/d1f/cf5 2296602 1 2024-02-28T16:31:24.407 INFO:tasks.workunit.client.1.smithi066.stdout:6/486: symlink d2/d2c/la7 0 2024-02-28T16:31:24.415 INFO:tasks.workunit.client.0.smithi001.stdout:6/897: rename d9/d29/d6e/d53/laf to d9/d29/d6e/d53/l130 0 2024-02-28T16:31:24.419 INFO:tasks.workunit.client.0.smithi001.stdout:0/886: dread - d18/d22/d76/d93/fe0 zero size 2024-02-28T16:31:24.420 INFO:tasks.workunit.client.1.smithi066.stdout:1/516: dwrite d2/d27/d2b/f2a [4194304,4194304] 0 2024-02-28T16:31:24.422 INFO:tasks.workunit.client.1.smithi066.stdout:6/487: getdents d2/d2c/da1/d5d 0 2024-02-28T16:31:24.424 INFO:tasks.workunit.client.0.smithi001.stdout:8/873: creat d4/d7/d22/d27/d58/d62/f117 x:0 0 0 2024-02-28T16:31:24.430 INFO:tasks.workunit.client.1.smithi066.stdout:3/608: dwrite db/dc/d35/d3d/f97 [0,4194304] 0 2024-02-28T16:31:24.433 INFO:tasks.workunit.client.0.smithi001.stdout:8/874: creat d4/d47/df4/df5/f118 x:0 0 0 2024-02-28T16:31:24.433 INFO:tasks.workunit.client.0.smithi001.stdout:8/875: chown d4/d9b 718 1 2024-02-28T16:31:24.435 INFO:tasks.workunit.client.0.smithi001.stdout:6/898: rename d9/d29/d6a/f10b to d9/d29/d6e/d4e/f131 0 2024-02-28T16:31:24.437 INFO:tasks.workunit.client.1.smithi066.stdout:9/427: dwrite da/d22/d46/d56/f72 [0,4194304] 0 2024-02-28T16:31:24.441 INFO:tasks.workunit.client.0.smithi001.stdout:8/876: chown d4/d32/d69/ffd 25 1 2024-02-28T16:31:24.445 INFO:tasks.workunit.client.1.smithi066.stdout:9/428: read da/df/d18/d28/f2b [1832710,33891] 0 2024-02-28T16:31:24.451 INFO:tasks.workunit.client.1.smithi066.stdout:9/429: dread - da/df/d18/d28/d63/f60 zero size 2024-02-28T16:31:24.452 INFO:tasks.workunit.client.0.smithi001.stdout:6/899: read d9/db/d19/fd3 [644460,11187] 0 2024-02-28T16:31:24.454 INFO:tasks.workunit.client.1.smithi066.stdout:3/609: dwrite db/d2a/d44/fc6 [0,4194304] 0 2024-02-28T16:31:24.455 INFO:tasks.workunit.client.0.smithi001.stdout:6/900: chown d9/db/d5f/dd4/cf7 6000 1 2024-02-28T16:31:24.462 INFO:tasks.workunit.client.1.smithi066.stdout:0/666: truncate d2/d27/d41/d76/d93/d10/f7f 502616 0 2024-02-28T16:31:24.464 INFO:tasks.workunit.client.1.smithi066.stdout:9/430: mknod da/df/d18/d2d/c87 0 2024-02-28T16:31:24.464 INFO:tasks.workunit.client.1.smithi066.stdout:3/610: mkdir db/d2a/dd4 0 2024-02-28T16:31:24.466 INFO:tasks.workunit.client.0.smithi001.stdout:6/901: rename d9/c49 to d9/db/d5f/d62/d7e/c132 0 2024-02-28T16:31:24.469 INFO:tasks.workunit.client.1.smithi066.stdout:6/488: write d2/da/f1c [851182,119864] 0 2024-02-28T16:31:24.470 INFO:tasks.workunit.client.1.smithi066.stdout:1/517: truncate d2/d18/d7a/f73 5654334 0 2024-02-28T16:31:24.471 INFO:tasks.workunit.client.0.smithi001.stdout:6/902: chown d9/db/d10/faa 1 1 2024-02-28T16:31:24.474 INFO:tasks.workunit.client.0.smithi001.stdout:1/988: write d0/d9/d1a/d2b/d26/d27/d30/d88/fef [918268,6812] 0 2024-02-28T16:31:24.476 INFO:tasks.workunit.client.1.smithi066.stdout:9/431: mknod da/df/d18/d28/d42/d68/c88 0 2024-02-28T16:31:24.480 INFO:tasks.workunit.client.1.smithi066.stdout:3/611: rename db/dc/d35/d3d/d76/d89/dcc/f34 to db/dc/d35/fd5 0 2024-02-28T16:31:24.483 INFO:tasks.workunit.client.1.smithi066.stdout:6/489: symlink d2/d26/la8 0 2024-02-28T16:31:24.483 INFO:tasks.workunit.client.1.smithi066.stdout:5/529: dwrite d3/d4/dc/dd/d2e/f90 [0,4194304] 0 2024-02-28T16:31:24.485 INFO:tasks.workunit.client.1.smithi066.stdout:9/432: mknod da/d3a/d15/d48/d76/c89 0 2024-02-28T16:31:24.487 INFO:tasks.workunit.client.1.smithi066.stdout:3/612: creat db/d2a/d44/d48/fd6 x:0 0 0 2024-02-28T16:31:24.488 INFO:tasks.workunit.client.1.smithi066.stdout:8/505: dwrite d0/d6/ff [4194304,4194304] 0 2024-02-28T16:31:24.490 INFO:tasks.workunit.client.0.smithi001.stdout:0/887: dwrite d18/d22/d2b/d36/d46/df5/f71 [0,4194304] 0 2024-02-28T16:31:24.491 INFO:tasks.workunit.client.1.smithi066.stdout:9/433: chown da/df/d18/d28/f59 103330 1 2024-02-28T16:31:24.492 INFO:tasks.workunit.client.0.smithi001.stdout:3/907: dwrite d4/d97/fdc [0,4194304] 0 2024-02-28T16:31:24.494 INFO:tasks.workunit.client.0.smithi001.stdout:6/903: dwrite d9/d29/d6e/d53/f9e [0,4194304] 0 2024-02-28T16:31:24.500 INFO:tasks.workunit.client.1.smithi066.stdout:5/530: creat d3/d4/dc/dd/d41/d7f/fa9 x:0 0 0 2024-02-28T16:31:24.503 INFO:tasks.workunit.client.1.smithi066.stdout:3/613: rename db/d2a/f37 to db/d2a/d44/d48/d52/dcb/fd7 0 2024-02-28T16:31:24.505 INFO:tasks.workunit.client.1.smithi066.stdout:1/518: write d2/d18/d19/f89 [577614,29599] 0 2024-02-28T16:31:24.506 INFO:tasks.workunit.client.1.smithi066.stdout:3/614: stat db/dc/d11/daa 0 2024-02-28T16:31:24.513 INFO:tasks.workunit.client.0.smithi001.stdout:1/989: write d0/d9/d1a/d2b/d26/d27/d30/d4a/f70 [1273774,20086] 0 2024-02-28T16:31:24.513 INFO:tasks.workunit.client.0.smithi001.stdout:3/908: truncate d4/d1e/f22 4169309 0 2024-02-28T16:31:24.517 INFO:tasks.workunit.client.0.smithi001.stdout:3/909: stat d4/d18/d5f 0 2024-02-28T16:31:24.517 INFO:tasks.workunit.client.0.smithi001.stdout:0/888: mkdir d18/da9/dea/d11d 0 2024-02-28T16:31:24.519 INFO:tasks.workunit.client.1.smithi066.stdout:5/531: unlink d3/d1b/f2d 0 2024-02-28T16:31:24.519 INFO:tasks.workunit.client.0.smithi001.stdout:6/904: mknod d9/db/d5f/dd4/dfb/c133 0 2024-02-28T16:31:24.519 INFO:tasks.workunit.client.0.smithi001.stdout:8/877: dread d4/d32/f50 [0,4194304] 0 2024-02-28T16:31:24.520 INFO:tasks.workunit.client.1.smithi066.stdout:8/506: dread d0/db/d12/d1d/d25/d4f/f86 [0,4194304] 0 2024-02-28T16:31:24.521 INFO:tasks.workunit.client.0.smithi001.stdout:1/990: mkdir d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/dfc/d159 0 2024-02-28T16:31:24.525 INFO:tasks.workunit.client.0.smithi001.stdout:3/910: creat d4/d1e/d7c/d7d/df8/dee/f129 x:0 0 0 2024-02-28T16:31:24.527 INFO:tasks.workunit.client.0.smithi001.stdout:0/889: rename d18/d22/d2f/d1c/fff to d18/d22/d2f/d1c/d90/f11e 0 2024-02-28T16:31:24.527 INFO:tasks.workunit.client.0.smithi001.stdout:6/905: creat d9/d39/dc4/f134 x:0 0 0 2024-02-28T16:31:24.528 INFO:tasks.workunit.client.1.smithi066.stdout:1/519: creat d2/d27/d2b/d43/d46/d80/f9f x:0 0 0 2024-02-28T16:31:24.528 INFO:tasks.workunit.client.1.smithi066.stdout:9/434: link da/df/d18/d28/d42/d68/c88 da/d3a/d15/c8a 0 2024-02-28T16:31:24.529 INFO:tasks.workunit.client.1.smithi066.stdout:3/615: mkdir db/d2a/d44/dd8 0 2024-02-28T16:31:24.529 INFO:tasks.workunit.client.1.smithi066.stdout:1/520: fdatasync d2/d27/d2b/d43/d46/f88 0 2024-02-28T16:31:24.529 INFO:tasks.workunit.client.0.smithi001.stdout:6/906: read - d9/fb3 zero size 2024-02-28T16:31:24.530 INFO:tasks.workunit.client.0.smithi001.stdout:1/991: stat d0/d9/d1a/d2b/d26/d27/dd3/de8/ded 0 2024-02-28T16:31:24.531 INFO:tasks.workunit.client.1.smithi066.stdout:5/532: creat d3/d4/dc/dd/d2e/d8f/faa x:0 0 0 2024-02-28T16:31:24.531 INFO:tasks.workunit.client.0.smithi001.stdout:3/911: stat d4/d18/da0/ld2 0 2024-02-28T16:31:24.532 INFO:tasks.workunit.client.1.smithi066.stdout:6/490: dread d2/d26/d6b/f70 [0,4194304] 0 2024-02-28T16:31:24.532 INFO:tasks.workunit.client.0.smithi001.stdout:0/890: mknod d18/d22/d3d/c11f 0 2024-02-28T16:31:24.533 INFO:tasks.workunit.client.0.smithi001.stdout:1/992: dread - d0/d9/f61 zero size 2024-02-28T16:31:24.535 INFO:tasks.workunit.client.1.smithi066.stdout:1/521: mkdir d2/da0 0 2024-02-28T16:31:24.536 INFO:tasks.workunit.client.1.smithi066.stdout:3/616: mknod db/d13/d14/d2d/d92/cd9 0 2024-02-28T16:31:24.537 INFO:tasks.workunit.client.0.smithi001.stdout:0/891: readlink d18/d22/d2f/d1c/l20 0 2024-02-28T16:31:24.537 INFO:tasks.workunit.client.1.smithi066.stdout:1/522: chown d2/d18/d7a 13216221 1 2024-02-28T16:31:24.538 INFO:tasks.workunit.client.0.smithi001.stdout:3/912: creat d4/d1e/d7c/d7d/df8/dee/f12a x:0 0 0 2024-02-28T16:31:24.538 INFO:tasks.workunit.client.0.smithi001.stdout:8/878: link d4/d32/d93/ce1 d4/d7/da8/d89/c119 0 2024-02-28T16:31:24.538 INFO:tasks.workunit.client.0.smithi001.stdout:1/993: chown d0/d9/d1a/d2b/d26/d27/d30/d67/db7/df6/f137 58027 1 2024-02-28T16:31:24.539 INFO:tasks.workunit.client.0.smithi001.stdout:6/907: creat d9/db/d5f/d62/d7e/d124/f135 x:0 0 0 2024-02-28T16:31:24.540 INFO:tasks.workunit.client.1.smithi066.stdout:9/435: link da/df/d18/f65 da/d22/f8b 0 2024-02-28T16:31:24.540 INFO:tasks.workunit.client.1.smithi066.stdout:6/491: symlink d2/d2c/da1/d5d/la9 0 2024-02-28T16:31:24.542 INFO:tasks.workunit.client.0.smithi001.stdout:1/994: stat d0/d9/d1a/d2b 0 2024-02-28T16:31:24.545 INFO:tasks.workunit.client.1.smithi066.stdout:9/436: stat da/df/d18/d28/d63/c77 0 2024-02-28T16:31:24.546 INFO:tasks.workunit.client.1.smithi066.stdout:0/667: dwrite d2/d27/d41/d76/d93/d10/f7f [0,4194304] 0 2024-02-28T16:31:24.547 INFO:tasks.workunit.client.0.smithi001.stdout:3/913: rename d4/d6/f87 to d4/d18/ddb/de8/f12b 0 2024-02-28T16:31:24.548 INFO:tasks.workunit.client.1.smithi066.stdout:6/492: mkdir d2/da/d19/d75/da6/daa 0 2024-02-28T16:31:24.550 INFO:tasks.workunit.client.0.smithi001.stdout:1/995: mknod d0/d9/d1a/d2b/d26/d27/d30/d88/dd1/dfc/d159/c15a 0 2024-02-28T16:31:24.552 INFO:tasks.workunit.client.1.smithi066.stdout:9/437: creat da/df/d18/d28/d42/d68/f8c x:0 0 0 2024-02-28T16:31:24.553 INFO:tasks.workunit.client.1.smithi066.stdout:6/493: symlink d2/d2c/da1/d5d/d63/lab 0 2024-02-28T16:31:24.556 INFO:tasks.workunit.client.1.smithi066.stdout:9/438: readlink da/d22/d46/l4e 0 2024-02-28T16:31:24.556 INFO:tasks.workunit.client.0.smithi001.stdout:3/914: truncate d4/d1e/db7/fd0 191266 0 2024-02-28T16:31:24.558 INFO:tasks.workunit.client.0.smithi001.stdout:1/996: creat d0/d9/d1a/d2b/d26/d27/d30/f15b x:0 0 0 2024-02-28T16:31:24.562 INFO:tasks.workunit.client.0.smithi001.stdout:8/879: rename d4/d7/d22/d27/d8e/dae to d4/d32/d11a 0 2024-02-28T16:31:24.562 INFO:tasks.workunit.client.0.smithi001.stdout:3/915: symlink d4/d1e/d7c/d7d/df8/d82/l12c 0 2024-02-28T16:31:24.563 INFO:tasks.workunit.client.1.smithi066.stdout:7/494: sync 2024-02-28T16:31:24.564 INFO:tasks.workunit.client.1.smithi066.stdout:2/562: sync 2024-02-28T16:31:24.564 INFO:tasks.workunit.client.1.smithi066.stdout:0/668: rename d2/d27/d41/d76/d93/db2/db5/d60/c7d to d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/cdf 0 2024-02-28T16:31:24.567 INFO:tasks.workunit.client.1.smithi066.stdout:9/439: dread da/fb [0,4194304] 0 2024-02-28T16:31:24.567 INFO:tasks.workunit.client.1.smithi066.stdout:7/495: write dc/de/d17/d1e/f24 [937024,103420] 0 2024-02-28T16:31:24.569 INFO:tasks.workunit.client.0.smithi001.stdout:1/997: unlink d0/d9/c9b 0 2024-02-28T16:31:24.569 INFO:tasks.workunit.client.1.smithi066.stdout:2/563: read - d0/d12/d2e/d3b/fb4 zero size 2024-02-28T16:31:24.570 INFO:tasks.workunit.client.0.smithi001.stdout:8/880: creat d4/d47/d8b/f11b x:0 0 0 2024-02-28T16:31:24.570 INFO:tasks.workunit.client.0.smithi001.stdout:3/916: mknod d4/d1e/db7/c12d 0 2024-02-28T16:31:24.574 INFO:tasks.workunit.client.1.smithi066.stdout:8/507: dread d0/db/d12/f87 [0,4194304] 0 2024-02-28T16:31:24.575 INFO:tasks.workunit.client.1.smithi066.stdout:1/523: write d2/d27/d2b/d37/f49 [5738844,93175] 0 2024-02-28T16:31:24.575 INFO:tasks.workunit.client.1.smithi066.stdout:0/669: write d2/d27/d41/d76/d93/f20 [3994804,77658] 0 2024-02-28T16:31:24.577 INFO:tasks.workunit.client.1.smithi066.stdout:9/440: creat da/df/d18/d2d/f8d x:0 0 0 2024-02-28T16:31:24.579 INFO:tasks.workunit.client.0.smithi001.stdout:0/892: dwrite d18/d22/d2f/f4e [0,4194304] 0 2024-02-28T16:31:24.580 INFO:tasks.workunit.client.1.smithi066.stdout:2/564: write d0/f1e [1233519,125278] 0 2024-02-28T16:31:24.583 INFO:tasks.workunit.client.1.smithi066.stdout:3/617: dread db/d13/f19 [4194304,4194304] 0 2024-02-28T16:31:24.583 INFO:tasks.workunit.client.1.smithi066.stdout:7/496: write dc/f7f [752986,62469] 0 2024-02-28T16:31:24.586 INFO:tasks.workunit.client.0.smithi001.stdout:0/893: mknod d18/d22/d2b/d36/c120 0 2024-02-28T16:31:24.587 INFO:tasks.workunit.client.1.smithi066.stdout:9/441: dread - da/df/d18/f65 zero size 2024-02-28T16:31:24.592 INFO:tasks.workunit.client.0.smithi001.stdout:1/998: write d0/d9/d1a/d2b/d26/d27/dd3/de8/ded/f11d [599479,32224] 0 2024-02-28T16:31:24.593 INFO:tasks.workunit.client.1.smithi066.stdout:9/442: dread - da/df/d18/f82 zero size 2024-02-28T16:31:24.595 INFO:tasks.workunit.client.0.smithi001.stdout:0/894: dread d18/d22/d2b/d36/d46/df5/d4f/fcd [0,4194304] 0 2024-02-28T16:31:24.598 INFO:tasks.workunit.client.1.smithi066.stdout:8/508: fdatasync d0/d6/dc/d16/d1a/d63/d3d/f62 0 2024-02-28T16:31:24.598 INFO:tasks.workunit.client.0.smithi001.stdout:1/999: dread - d0/d9/d1a/d2b/d26/d27/d30/d67/dd6/fdc zero size 2024-02-28T16:31:24.599 INFO:tasks.workunit.client.1.smithi066.stdout:5/533: dread d3/d4/dc/d56/f75 [0,4194304] 0 2024-02-28T16:31:24.600 INFO:tasks.workunit.client.1.smithi066.stdout:3/618: write db/d13/d14/d2d/f9e [4209503,78874] 0 2024-02-28T16:31:24.602 INFO:tasks.workunit.client.0.smithi001.stdout:3/917: dwrite d4/d6/fb5 [0,4194304] 0 2024-02-28T16:31:24.602 INFO:tasks.workunit.client.1.smithi066.stdout:6/494: dwrite d2/d26/fa3 [0,4194304] 0 2024-02-28T16:31:24.603 INFO:tasks.workunit.client.0.smithi001.stdout:0/895: rename f11 to d18/d22/d2b/f121 0 2024-02-28T16:31:24.604 INFO:tasks.workunit.client.0.smithi001.stdout:3/918: chown d4/d6/d7/d78/d81/dd6/dd9 3037 1 2024-02-28T16:31:24.608 INFO:tasks.workunit.client.0.smithi001.stdout:3/919: getdents d4/d6/d7 0 2024-02-28T16:31:24.609 INFO:tasks.workunit.client.0.smithi001.stdout:3/920: readlink d4/d1e/def/l50 0 2024-02-28T16:31:24.610 INFO:tasks.workunit.client.0.smithi001.stdout:0/896: rename d18/d22/d2f/d1c/d90/f100 to d18/d22/d2b/d36/d48/d11a/f122 0 2024-02-28T16:31:24.610 INFO:tasks.workunit.client.1.smithi066.stdout:2/565: read d0/d12/d1a/d23/d29/d21/d2f/f9c [5520,58137] 0 2024-02-28T16:31:24.613 INFO:tasks.workunit.client.1.smithi066.stdout:5/534: dread - d3/d4/dc/dd/d41/d7f/fa9 zero size 2024-02-28T16:31:24.615 INFO:tasks.workunit.client.0.smithi001.stdout:0/897: link d18/d22/c28 d18/d22/d2f/d1c/d90/c123 0 2024-02-28T16:31:24.619 INFO:tasks.workunit.client.1.smithi066.stdout:4/682: sync 2024-02-28T16:31:24.630 INFO:tasks.workunit.client.1.smithi066.stdout:0/670: write d2/d27/d41/d76/d93/db2/db5/d67/d94/fb8 [736787,121815] 0 2024-02-28T16:31:24.636 INFO:tasks.workunit.client.1.smithi066.stdout:1/524: truncate d2/d27/d2b/d43/d46/d68/f7e 1072955 0 2024-02-28T16:31:24.637 INFO:tasks.workunit.client.1.smithi066.stdout:6/495: mkdir d2/dac 0 2024-02-28T16:31:24.639 INFO:tasks.workunit.client.1.smithi066.stdout:3/619: mkdir db/d13/d14/dda 0 2024-02-28T16:31:24.639 INFO:tasks.workunit.client.1.smithi066.stdout:8/509: dwrite d0/db/f7d [0,4194304] 0 2024-02-28T16:31:24.639 INFO:tasks.workunit.client.0.smithi001.stdout:3/921: dwrite d4/d6/d7/f88 [0,4194304] 0 2024-02-28T16:31:24.642 INFO:tasks.workunit.client.0.smithi001.stdout:3/922: truncate d4/d18/ddb/de8/f111 578852 0 2024-02-28T16:31:24.644 INFO:tasks.workunit.client.1.smithi066.stdout:8/510: read d0/db/d12/f4b [73006,46896] 0 2024-02-28T16:31:24.645 INFO:tasks.workunit.client.0.smithi001.stdout:3/923: read - d4/d1e/d7c/d7d/df8/dee/f129 zero size 2024-02-28T16:31:24.646 INFO:tasks.workunit.client.1.smithi066.stdout:6/496: fsync d2/da/d19/d9c/f5b 0 2024-02-28T16:31:24.646 INFO:tasks.workunit.client.1.smithi066.stdout:9/443: getdents da/d22/d46 0 2024-02-28T16:31:24.648 INFO:tasks.workunit.client.1.smithi066.stdout:7/497: dwrite dc/d56/f8a [0,4194304] 0 2024-02-28T16:31:24.652 INFO:tasks.workunit.client.1.smithi066.stdout:4/683: rename d1/d18/d4e/lb0 to d1/d18/d4e/ldf 0 2024-02-28T16:31:24.652 INFO:tasks.workunit.client.1.smithi066.stdout:9/444: readlink da/d22/l2e 0 2024-02-28T16:31:24.653 INFO:tasks.workunit.client.1.smithi066.stdout:5/535: link d3/d4/dc/dd/d41/d5d/f69 d3/d4/dc/dd/d35/d92/fab 0 2024-02-28T16:31:24.657 INFO:tasks.workunit.client.0.smithi001.stdout:3/924: dread d4/d97/fdc [0,4194304] 0 2024-02-28T16:31:24.665 INFO:tasks.workunit.client.1.smithi066.stdout:6/497: mkdir d2/d2c/da1/dad 0 2024-02-28T16:31:24.666 INFO:tasks.workunit.client.1.smithi066.stdout:9/445: chown da/df/d18/d28/d63/f60 13981691 1 2024-02-28T16:31:24.667 INFO:tasks.workunit.client.1.smithi066.stdout:9/446: stat da/c6c 0 2024-02-28T16:31:24.672 INFO:tasks.workunit.client.1.smithi066.stdout:1/525: rename d2/d18/d7a/f9a to d2/d18/d6b/fa1 0 2024-02-28T16:31:24.675 INFO:tasks.workunit.client.1.smithi066.stdout:6/498: chown d2/d26/d32/c44 37 1 2024-02-28T16:31:24.675 INFO:tasks.workunit.client.1.smithi066.stdout:3/620: dwrite db/ff [0,4194304] 0 2024-02-28T16:31:24.676 INFO:tasks.workunit.client.0.smithi001.stdout:3/925: dread d4/d18/fad [0,4194304] 0 2024-02-28T16:31:24.677 INFO:tasks.workunit.client.1.smithi066.stdout:8/511: dread d0/db/f27 [0,4194304] 0 2024-02-28T16:31:24.679 INFO:tasks.workunit.client.1.smithi066.stdout:6/499: readlink d2/d26/d32/l87 0 2024-02-28T16:31:24.680 INFO:tasks.workunit.client.1.smithi066.stdout:1/526: dread - d2/d27/d2b/d43/d46/d80/f9f zero size 2024-02-28T16:31:24.680 INFO:tasks.workunit.client.1.smithi066.stdout:2/566: write d0/d12/d2e/d7a/f45 [2287955,120605] 0 2024-02-28T16:31:24.683 INFO:tasks.workunit.client.0.smithi001.stdout:3/926: getdents d4/d1e/d7c/d7d 0 2024-02-28T16:31:24.683 INFO:tasks.workunit.client.1.smithi066.stdout:0/671: getdents d2/d27/d41/d76/d93/db2/db5/d60 0 2024-02-28T16:31:24.683 INFO:tasks.workunit.client.1.smithi066.stdout:6/500: fsync d2/d2c/f86 0 2024-02-28T16:31:24.684 INFO:tasks.workunit.client.1.smithi066.stdout:5/536: getdents d3/d4/dc/d42/d60 0 2024-02-28T16:31:24.686 INFO:tasks.workunit.client.0.smithi001.stdout:3/927: mkdir d4/d18/ddb/de8/d12e 0 2024-02-28T16:31:24.686 INFO:tasks.workunit.client.1.smithi066.stdout:7/498: dwrite dc/de/d17/d37/f72 [0,4194304] 0 2024-02-28T16:31:24.687 INFO:tasks.workunit.client.0.smithi001.stdout:3/928: fdatasync d4/d1e/d7c/ff6 0 2024-02-28T16:31:24.688 INFO:tasks.workunit.client.1.smithi066.stdout:8/512: stat d0/d6/dc/d16/d4c/d42 0 2024-02-28T16:31:24.688 INFO:tasks.workunit.client.1.smithi066.stdout:6/501: stat d2/d26/la8 0 2024-02-28T16:31:24.689 INFO:tasks.workunit.client.0.smithi001.stdout:3/929: read - d4/d6/d7/d78/d81/da1/f128 zero size 2024-02-28T16:31:24.690 INFO:tasks.workunit.client.1.smithi066.stdout:1/527: chown d2/d27/d2b/d34/c42 219944 1 2024-02-28T16:31:24.693 INFO:tasks.workunit.client.1.smithi066.stdout:7/499: chown dc/f4c 0 1 2024-02-28T16:31:24.698 INFO:tasks.workunit.client.1.smithi066.stdout:5/537: truncate d3/d4/dc/dd/d41/d5d/f7c 886789 0 2024-02-28T16:31:24.698 INFO:tasks.workunit.client.1.smithi066.stdout:2/567: creat d0/d12/d1a/d23/d5d/fb5 x:0 0 0 2024-02-28T16:31:24.702 INFO:tasks.workunit.client.1.smithi066.stdout:5/538: write d3/d4/dc/dd/d35/d86/f85 [618217,122272] 0 2024-02-28T16:31:24.703 INFO:tasks.workunit.client.1.smithi066.stdout:6/502: mknod d2/da/cae 0 2024-02-28T16:31:24.705 INFO:tasks.workunit.client.1.smithi066.stdout:8/513: mkdir d0/d6/dc/d16/d4c/d6c/d6f/da7 0 2024-02-28T16:31:24.708 INFO:tasks.workunit.client.1.smithi066.stdout:0/672: dread d2/d27/d41/d76/d93/db2/db5/d73/f90 [0,4194304] 0 2024-02-28T16:31:24.710 INFO:tasks.workunit.client.0.smithi001.stdout:3/930: dwrite d4/d1e/d7c/da8/faa [0,4194304] 0 2024-02-28T16:31:24.712 INFO:tasks.workunit.client.1.smithi066.stdout:2/568: write d0/f37 [2336350,104440] 0 2024-02-28T16:31:24.713 INFO:tasks.workunit.client.1.smithi066.stdout:6/503: mknod d2/d2c/d72/caf 0 2024-02-28T16:31:24.714 INFO:tasks.workunit.client.1.smithi066.stdout:5/539: rename d3/d4/dc/d42 to d3/d4/dc/dd/d41/d7a/dac 0 2024-02-28T16:31:24.714 INFO:tasks.workunit.client.0.smithi001.stdout:3/931: rename d4/d18/cf5 to d4/d1e/def/c12f 0 2024-02-28T16:31:24.717 INFO:tasks.workunit.client.1.smithi066.stdout:2/569: rename d0 to d0/d12/d1a/d23/db6 22 2024-02-28T16:31:24.717 INFO:tasks.workunit.client.1.smithi066.stdout:6/504: readlink d2/d2c/da1/d5d/la9 0 2024-02-28T16:31:24.720 INFO:tasks.workunit.client.0.smithi001.stdout:3/932: creat d4/d1e/d7c/f130 x:0 0 0 2024-02-28T16:31:24.723 INFO:tasks.workunit.client.1.smithi066.stdout:3/621: dwrite db/d2a/d44/d48/f85 [0,4194304] 0 2024-02-28T16:31:24.723 INFO:tasks.workunit.client.0.smithi001.stdout:3/933: truncate d4/f83 3633177 0 2024-02-28T16:31:24.724 INFO:tasks.workunit.client.0.smithi001.stdout:3/934: chown d4/d6/d7/d78/d81/da1 1183174 1 2024-02-28T16:31:24.727 INFO:tasks.workunit.client.1.smithi066.stdout:2/570: symlink d0/d12/d2e/d3b/d66/lb7 0 2024-02-28T16:31:24.727 INFO:tasks.workunit.client.1.smithi066.stdout:6/505: symlink d2/da/lb0 0 2024-02-28T16:31:24.727 INFO:tasks.workunit.client.0.smithi001.stdout:3/935: rmdir d4/d1e/d7c/d7d/df8/dee 39 2024-02-28T16:31:24.728 INFO:tasks.workunit.client.1.smithi066.stdout:5/540: getdents d3/d1b 0 2024-02-28T16:31:24.731 INFO:tasks.workunit.client.0.smithi001.stdout:3/936: creat d4/d18/ddb/de8/d12e/f131 x:0 0 0 2024-02-28T16:31:24.731 INFO:tasks.workunit.client.1.smithi066.stdout:7/500: dwrite dc/f5d [0,4194304] 0 2024-02-28T16:31:24.733 INFO:tasks.workunit.client.1.smithi066.stdout:6/506: chown d2/d26/d32/d46/f74 2 1 2024-02-28T16:31:24.734 INFO:tasks.workunit.client.1.smithi066.stdout:3/622: dread - db/dc/d11/d1f/d78/fa3 zero size 2024-02-28T16:31:24.735 INFO:tasks.workunit.client.1.smithi066.stdout:1/528: truncate d2/d18/d19/f1e 886437 0 2024-02-28T16:31:24.737 INFO:tasks.workunit.client.1.smithi066.stdout:1/529: write d2/d18/f3a [2124623,100261] 0 2024-02-28T16:31:24.739 INFO:tasks.workunit.client.1.smithi066.stdout:2/571: symlink d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/lb8 0 2024-02-28T16:31:24.745 INFO:tasks.workunit.client.1.smithi066.stdout:7/501: symlink dc/d6e/l8e 0 2024-02-28T16:31:24.749 INFO:tasks.workunit.client.1.smithi066.stdout:3/623: symlink db/d2a/d44/d48/d52/d77/ldb 0 2024-02-28T16:31:24.751 INFO:tasks.workunit.client.1.smithi066.stdout:3/624: write db/d13/d14/d84/f9c [619178,25096] 0 2024-02-28T16:31:24.753 INFO:tasks.workunit.client.0.smithi001.stdout:5/862: sync 2024-02-28T16:31:24.754 INFO:tasks.workunit.client.0.smithi001.stdout:3/937: dread d4/d1e/d7c/fd3 [0,4194304] 0 2024-02-28T16:31:24.756 INFO:tasks.workunit.client.1.smithi066.stdout:1/530: write d2/d27/d2b/d34/f55 [2886610,39730] 0 2024-02-28T16:31:24.756 INFO:tasks.workunit.client.0.smithi001.stdout:5/863: write dc/d12/d41/f9c [1234990,61786] 0 2024-02-28T16:31:24.758 INFO:tasks.workunit.client.0.smithi001.stdout:5/864: chown dc/d12/d56/d96/db9/ddd 0 1 2024-02-28T16:31:24.759 INFO:tasks.workunit.client.1.smithi066.stdout:5/541: creat d3/d4/dc/dd/d35/d92/fad x:0 0 0 2024-02-28T16:31:24.760 INFO:tasks.workunit.client.1.smithi066.stdout:6/507: symlink d2/da/d19/d9c/lb1 0 2024-02-28T16:31:24.762 INFO:tasks.workunit.client.0.smithi001.stdout:5/865: getdents dc/d1a/d21/de2 0 2024-02-28T16:31:24.764 INFO:tasks.workunit.client.1.smithi066.stdout:0/673: dwrite d2/d27/d41/d76/d93/db2/db5/f57 [0,4194304] 0 2024-02-28T16:31:24.764 INFO:tasks.workunit.client.1.smithi066.stdout:3/625: creat db/dc/d35/d3d/d76/fdc x:0 0 0 2024-02-28T16:31:24.764 INFO:tasks.workunit.client.1.smithi066.stdout:6/508: fsync d2/d26/f2d 0 2024-02-28T16:31:24.764 INFO:tasks.workunit.client.1.smithi066.stdout:1/531: stat d2/l28 0 2024-02-28T16:31:24.765 INFO:tasks.workunit.client.0.smithi001.stdout:5/866: symlink dc/d12/d56/l113 0 2024-02-28T16:31:24.767 INFO:tasks.workunit.client.0.smithi001.stdout:5/867: symlink dc/d1a/d52/l114 0 2024-02-28T16:31:24.769 INFO:tasks.workunit.client.1.smithi066.stdout:3/626: stat db/d13/d1a/d33 0 2024-02-28T16:31:24.771 INFO:tasks.workunit.client.1.smithi066.stdout:7/502: dwrite dc/d29/f44 [0,4194304] 0 2024-02-28T16:31:24.774 INFO:tasks.workunit.client.1.smithi066.stdout:1/532: mknod d2/d27/d2b/ca2 0 2024-02-28T16:31:24.775 INFO:tasks.workunit.client.1.smithi066.stdout:0/674: dread d2/d27/d41/d76/d93/db2/db5/d51/f70 [0,4194304] 0 2024-02-28T16:31:24.775 INFO:tasks.workunit.client.1.smithi066.stdout:6/509: rename d2/d2c/d72/f7b to d2/d2c/da1/d5d/d63/d6a/fb2 0 2024-02-28T16:31:24.781 INFO:tasks.workunit.client.1.smithi066.stdout:2/572: rmdir d0/d12/d1a/d23/d29/d21/d2f 39 2024-02-28T16:31:24.784 INFO:tasks.workunit.client.0.smithi001.stdout:3/938: write d4/d18/d5f/f77 [1827885,3071] 0 2024-02-28T16:31:24.785 INFO:tasks.workunit.client.1.smithi066.stdout:0/675: truncate d2/d27/d41/d76/d93/d10/d23/d6d/f79 4950743 0 2024-02-28T16:31:24.787 INFO:tasks.workunit.client.0.smithi001.stdout:3/939: read d4/d6/d7/f88 [2213281,13470] 0 2024-02-28T16:31:24.788 INFO:tasks.workunit.client.1.smithi066.stdout:5/542: getdents d3/d4/dc/dd/d35 0 2024-02-28T16:31:24.790 INFO:tasks.workunit.client.0.smithi001.stdout:5/868: dwrite dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/fc1 [0,4194304] 0 2024-02-28T16:31:24.793 INFO:tasks.workunit.client.1.smithi066.stdout:6/510: getdents d2/da/d19/d75/da6/daa 0 2024-02-28T16:31:24.794 INFO:tasks.workunit.client.1.smithi066.stdout:2/573: getdents d0/d12/d1a/d39/d7c/d87/daf 0 2024-02-28T16:31:24.794 INFO:tasks.workunit.client.0.smithi001.stdout:5/869: chown dc/d12/f14 128891 1 2024-02-28T16:31:24.797 INFO:tasks.workunit.client.1.smithi066.stdout:7/503: dwrite f5 [0,4194304] 0 2024-02-28T16:31:24.800 INFO:tasks.workunit.client.1.smithi066.stdout:1/533: rename d2/d27/f7b to d2/d27/fa3 0 2024-02-28T16:31:24.800 INFO:tasks.workunit.client.0.smithi001.stdout:5/870: truncate dc/d12/d41/db7/fc6 321330 0 2024-02-28T16:31:24.806 INFO:tasks.workunit.client.1.smithi066.stdout:0/676: rename d2/d27/d41/d76/d93/d33/d3f/db7 to d2/d27/d41/d76/d93/d10/d23/db6/de0 0 2024-02-28T16:31:24.808 INFO:tasks.workunit.client.1.smithi066.stdout:3/627: dread db/d13/d1a/d33/f39 [0,4194304] 0 2024-02-28T16:31:24.808 INFO:tasks.workunit.client.0.smithi001.stdout:5/871: write dc/d12/d56/d96/db9/dc2/dca/f42 [1162162,56674] 0 2024-02-28T16:31:24.813 INFO:tasks.workunit.client.0.smithi001.stdout:5/872: dread - dc/d12/daa/f109 zero size 2024-02-28T16:31:24.813 INFO:tasks.workunit.client.0.smithi001.stdout:2/814: sync 2024-02-28T16:31:24.813 INFO:tasks.workunit.client.0.smithi001.stdout:8/881: sync 2024-02-28T16:31:24.813 INFO:tasks.workunit.client.0.smithi001.stdout:4/974: sync 2024-02-28T16:31:24.813 INFO:tasks.workunit.client.0.smithi001.stdout:0/898: sync 2024-02-28T16:31:24.813 INFO:tasks.workunit.client.0.smithi001.stdout:7/934: sync 2024-02-28T16:31:24.813 INFO:tasks.workunit.client.0.smithi001.stdout:6/908: sync 2024-02-28T16:31:24.814 INFO:tasks.workunit.client.1.smithi066.stdout:6/511: truncate d2/da/f1b 120643 0 2024-02-28T16:31:24.817 INFO:tasks.workunit.client.1.smithi066.stdout:2/574: symlink d0/lb9 0 2024-02-28T16:31:24.819 INFO:tasks.workunit.client.1.smithi066.stdout:0/677: creat d2/d27/d41/d76/d93/db2/db5/d67/fe1 x:0 0 0 2024-02-28T16:31:24.820 INFO:tasks.workunit.client.1.smithi066.stdout:6/512: write d2/da/f1c [1825582,42430] 0 2024-02-28T16:31:24.823 INFO:tasks.workunit.client.0.smithi001.stdout:5/873: creat dc/df/d79/f115 x:0 0 0 2024-02-28T16:31:24.825 INFO:tasks.workunit.client.0.smithi001.stdout:8/882: mknod d4/d7/da8/c11c 0 2024-02-28T16:31:24.826 INFO:tasks.workunit.client.0.smithi001.stdout:6/909: truncate d9/db/f13 413434 0 2024-02-28T16:31:24.827 INFO:tasks.workunit.client.1.smithi066.stdout:2/575: fdatasync d0/d12/d3e/f4e 0 2024-02-28T16:31:24.830 INFO:tasks.workunit.client.0.smithi001.stdout:0/899: getdents d18/da9/dea/d10c 0 2024-02-28T16:31:24.830 INFO:tasks.workunit.client.0.smithi001.stdout:3/940: dread d4/d1e/f25 [0,4194304] 0 2024-02-28T16:31:24.831 INFO:tasks.workunit.client.0.smithi001.stdout:8/883: stat d4/d7/d22/d27/d4b/d84/f109 0 2024-02-28T16:31:24.832 INFO:tasks.workunit.client.1.smithi066.stdout:6/513: rename d2/d2c/da1/d5d/d63/lab to d2/d26/d32/lb3 0 2024-02-28T16:31:24.833 INFO:tasks.workunit.client.0.smithi001.stdout:6/910: dread - d9/d29/d6e/d53/fac zero size 2024-02-28T16:31:24.834 INFO:tasks.workunit.client.0.smithi001.stdout:5/874: mkdir dc/d12/d116 0 2024-02-28T16:31:24.835 INFO:tasks.workunit.client.1.smithi066.stdout:6/514: write d2/da/f76 [1293698,52082] 0 2024-02-28T16:31:24.835 INFO:tasks.workunit.client.1.smithi066.stdout:0/678: getdents d2/d27/d41/d76/dcf 0 2024-02-28T16:31:24.837 INFO:tasks.workunit.client.0.smithi001.stdout:0/900: stat d18/d22/d2f/d1c/f10f 0 2024-02-28T16:31:24.838 INFO:tasks.workunit.client.1.smithi066.stdout:2/576: creat d0/d12/d2e/d7a/fba x:0 0 0 2024-02-28T16:31:24.839 INFO:tasks.workunit.client.1.smithi066.stdout:1/534: write d2/d27/d2b/d43/d46/f63 [153560,110337] 0 2024-02-28T16:31:24.840 INFO:tasks.workunit.client.0.smithi001.stdout:5/875: read - dc/d12/d56/d96/db9/f105 zero size 2024-02-28T16:31:24.841 INFO:tasks.workunit.client.1.smithi066.stdout:3/628: getdents db/d2a/d44/d48/d52/dcb 0 2024-02-28T16:31:24.842 INFO:tasks.workunit.client.1.smithi066.stdout:5/543: dread d3/d4/dc/dd/d2e/f38 [0,4194304] 0 2024-02-28T16:31:24.843 INFO:tasks.workunit.client.1.smithi066.stdout:6/515: mknod d2/d2c/d6e/cb4 0 2024-02-28T16:31:24.848 INFO:tasks.workunit.client.1.smithi066.stdout:1/535: chown d2/d27/d2b/d43/f4e 782309 1 2024-02-28T16:31:24.848 INFO:tasks.workunit.client.0.smithi001.stdout:0/901: symlink d18/d22/d2b/d36/d46/l124 0 2024-02-28T16:31:24.849 INFO:tasks.workunit.client.1.smithi066.stdout:2/577: creat d0/d12/d1a/d23/d5d/fbb x:0 0 0 2024-02-28T16:31:24.850 INFO:tasks.workunit.client.1.smithi066.stdout:3/629: dread db/dc/d11/d1f/d78/f90 [0,4194304] 0 2024-02-28T16:31:24.852 INFO:tasks.workunit.client.1.smithi066.stdout:7/504: dwrite dc/de/d17/d1e/f6c [0,4194304] 0 2024-02-28T16:31:24.854 INFO:tasks.workunit.client.1.smithi066.stdout:6/516: creat d2/fb5 x:0 0 0 2024-02-28T16:31:24.854 INFO:tasks.workunit.client.1.smithi066.stdout:4/684: sync 2024-02-28T16:31:24.857 INFO:tasks.workunit.client.0.smithi001.stdout:3/941: truncate d4/d18/ddb/de8/f12b 1997402 0 2024-02-28T16:31:24.858 INFO:tasks.workunit.client.1.smithi066.stdout:3/630: dread - db/dc/d11/daa/fd3 zero size 2024-02-28T16:31:24.862 INFO:tasks.workunit.client.1.smithi066.stdout:6/517: symlink d2/d2c/da1/d5d/d63/lb6 0 2024-02-28T16:31:24.866 INFO:tasks.workunit.client.0.smithi001.stdout:0/902: read d18/d22/f66 [3224224,122960] 0 2024-02-28T16:31:24.867 INFO:tasks.workunit.client.0.smithi001.stdout:2/815: dwrite d0/d2/d3/de/d61/d12/d55/fcd [0,4194304] 0 2024-02-28T16:31:24.867 INFO:tasks.workunit.client.0.smithi001.stdout:4/975: dwrite d3/d2d/d4c/d5e/fa1 [0,4194304] 0 2024-02-28T16:31:24.867 INFO:tasks.workunit.client.0.smithi001.stdout:5/876: rename dc/d12/d41/db7/fc6 to dc/d1a/f117 0 2024-02-28T16:31:24.868 INFO:tasks.workunit.client.1.smithi066.stdout:4/685: creat d1/d18/d41/dba/fe0 x:0 0 0 2024-02-28T16:31:24.868 INFO:tasks.workunit.client.1.smithi066.stdout:0/679: write d2/d27/d41/d76/d93/f75 [6949900,100888] 0 2024-02-28T16:31:24.869 INFO:tasks.workunit.client.0.smithi001.stdout:3/942: symlink d4/d6/d7/d78/d81/l132 0 2024-02-28T16:31:24.871 INFO:tasks.workunit.client.1.smithi066.stdout:5/544: dwrite d3/d1b/f31 [0,4194304] 0 2024-02-28T16:31:24.879 INFO:tasks.workunit.client.0.smithi001.stdout:7/935: dread d3/fa4 [0,4194304] 0 2024-02-28T16:31:24.879 INFO:tasks.workunit.client.1.smithi066.stdout:6/518: creat d2/d2c/da1/fb7 x:0 0 0 2024-02-28T16:31:24.880 INFO:tasks.workunit.client.1.smithi066.stdout:3/631: truncate db/dc/d11/d1f/f2e 4502491 0 2024-02-28T16:31:24.880 INFO:tasks.workunit.client.0.smithi001.stdout:6/911: dwrite d9/d29/f3b [0,4194304] 0 2024-02-28T16:31:24.880 INFO:tasks.workunit.client.0.smithi001.stdout:8/884: dwrite d4/d7/f10 [4194304,4194304] 0 2024-02-28T16:31:24.881 INFO:tasks.workunit.client.1.smithi066.stdout:0/680: fsync d2/d27/d41/d76/d93/d33/d3f/faa 0 2024-02-28T16:31:24.884 INFO:tasks.workunit.client.0.smithi001.stdout:2/816: creat d0/d2/d3/d36/d94/f10a x:0 0 0 2024-02-28T16:31:24.888 INFO:tasks.workunit.client.1.smithi066.stdout:5/545: unlink d3/d4/dc/dd/d35/d86/f85 0 2024-02-28T16:31:24.889 INFO:tasks.workunit.client.0.smithi001.stdout:4/976: rename d3/c124 to d3/d1c/d58/d88/c13c 0 2024-02-28T16:31:24.890 INFO:tasks.workunit.client.1.smithi066.stdout:7/505: dwrite dc/de/d17/d37/d47/f5e [0,4194304] 0 2024-02-28T16:31:24.894 INFO:tasks.workunit.client.1.smithi066.stdout:6/519: rmdir d2/d26 39 2024-02-28T16:31:24.895 INFO:tasks.workunit.client.0.smithi001.stdout:4/977: read d3/d5/d7/d1f/d2e/f64 [1058245,3499] 0 2024-02-28T16:31:24.895 INFO:tasks.workunit.client.0.smithi001.stdout:8/885: write d4/d7/d22/d27/d4b/d77/ff7 [4265548,106397] 0 2024-02-28T16:31:24.899 INFO:tasks.workunit.client.1.smithi066.stdout:3/632: rmdir db/dc/d11/d1f/d78 39 2024-02-28T16:31:24.900 INFO:tasks.workunit.client.1.smithi066.stdout:5/546: symlink d3/d4/dc/dd/d2e/lae 0 2024-02-28T16:31:24.901 INFO:tasks.workunit.client.0.smithi001.stdout:4/978: stat d3/d1c/da0/f135 0 2024-02-28T16:31:24.903 INFO:tasks.workunit.client.1.smithi066.stdout:3/633: fsync db/d2a/d44/d48/f6c 0 2024-02-28T16:31:24.906 INFO:tasks.workunit.client.1.smithi066.stdout:0/681: write d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/da1/fd2 [1631433,28049] 0 2024-02-28T16:31:24.907 INFO:tasks.workunit.client.0.smithi001.stdout:4/979: write d3/d5/d92/fc9 [7035073,38510] 0 2024-02-28T16:31:24.907 INFO:tasks.workunit.client.0.smithi001.stdout:6/912: creat d9/db/d5f/d62/d127/f136 x:0 0 0 2024-02-28T16:31:24.907 INFO:tasks.workunit.client.1.smithi066.stdout:8/514: sync 2024-02-28T16:31:24.908 INFO:tasks.workunit.client.1.smithi066.stdout:9/447: sync 2024-02-28T16:31:24.908 INFO:tasks.workunit.client.1.smithi066.stdout:6/520: mknod d2/d2c/d72/cb8 0 2024-02-28T16:31:24.909 INFO:tasks.workunit.client.1.smithi066.stdout:0/682: chown d2/d27/c3e 1 1 2024-02-28T16:31:24.910 INFO:tasks.workunit.client.1.smithi066.stdout:6/521: dread - d2/d2c/d72/fa5 zero size 2024-02-28T16:31:24.912 INFO:tasks.workunit.client.1.smithi066.stdout:7/506: rename dc/d29/f34 to dc/de/d17/d1e/d45/f8f 0 2024-02-28T16:31:24.914 INFO:tasks.workunit.client.0.smithi001.stdout:2/817: read d0/d2/d3/de/d22/f48 [1252756,39427] 0 2024-02-28T16:31:24.914 INFO:tasks.workunit.client.1.smithi066.stdout:1/536: dread d2/d18/d6b/f77 [0,4194304] 0 2024-02-28T16:31:24.915 INFO:tasks.workunit.client.0.smithi001.stdout:5/877: dwrite dc/d12/d56/fff [0,4194304] 0 2024-02-28T16:31:24.920 INFO:tasks.workunit.client.1.smithi066.stdout:7/507: stat dc/d29/f2e 0 2024-02-28T16:31:24.920 INFO:tasks.workunit.client.1.smithi066.stdout:5/547: mknod d3/d1b/d7d/caf 0 2024-02-28T16:31:24.923 INFO:tasks.workunit.client.0.smithi001.stdout:8/886: creat d4/d32/d93/f11d x:0 0 0 2024-02-28T16:31:24.924 INFO:tasks.workunit.client.1.smithi066.stdout:0/683: mknod d2/d27/d41/dc5/ce2 0 2024-02-28T16:31:24.924 INFO:tasks.workunit.client.1.smithi066.stdout:8/515: chown d0/db/c11 4150529 1 2024-02-28T16:31:24.925 INFO:tasks.workunit.client.0.smithi001.stdout:4/980: truncate d3/d5/d7/d1f/d2e/dbe/d115/f133 809243 0 2024-02-28T16:31:24.928 INFO:tasks.workunit.client.1.smithi066.stdout:8/516: readlink d0/db/d15/l69 0 2024-02-28T16:31:24.928 INFO:tasks.workunit.client.1.smithi066.stdout:7/508: readlink dc/de/d17/d37/l6a 0 2024-02-28T16:31:24.928 INFO:tasks.workunit.client.0.smithi001.stdout:8/887: stat d4/d7/d22/d27/d58/f71 0 2024-02-28T16:31:24.932 INFO:tasks.workunit.client.1.smithi066.stdout:0/684: chown d2/d27/d41/dc5/ce2 340484 1 2024-02-28T16:31:24.935 INFO:tasks.workunit.client.0.smithi001.stdout:3/943: link d4/d1e/def/f9c d4/d6/d7/d78/d81/dd6/dd9/f133 0 2024-02-28T16:31:24.936 INFO:tasks.workunit.client.0.smithi001.stdout:8/888: read d4/d7/f2b [5535139,21891] 0 2024-02-28T16:31:24.938 INFO:tasks.workunit.client.1.smithi066.stdout:4/686: truncate d1/d23/d25/f99 4037657 0 2024-02-28T16:31:24.940 INFO:tasks.workunit.client.0.smithi001.stdout:4/981: dread - d3/d1c/d58/d6c/fac zero size 2024-02-28T16:31:24.940 INFO:tasks.workunit.client.1.smithi066.stdout:1/537: dread d2/d27/d2b/d43/d46/f63 [0,4194304] 0 2024-02-28T16:31:24.941 INFO:tasks.workunit.client.1.smithi066.stdout:0/685: truncate d2/f5e 5073197 0 2024-02-28T16:31:24.943 INFO:tasks.workunit.client.1.smithi066.stdout:8/517: mkdir d0/d6/dc/d16/d4c/d42/da8 0 2024-02-28T16:31:24.945 INFO:tasks.workunit.client.0.smithi001.stdout:7/936: write d3/d17/d52/d7f/d99/dc1/fe9 [2149243,64132] 0 2024-02-28T16:31:24.949 INFO:tasks.workunit.client.1.smithi066.stdout:3/634: dwrite db/d7a/fa0 [0,4194304] 0 2024-02-28T16:31:24.949 INFO:tasks.workunit.client.0.smithi001.stdout:7/937: dread - d3/d17/d52/d7f/d99/dc1/f12b zero size 2024-02-28T16:31:24.953 INFO:tasks.workunit.client.0.smithi001.stdout:0/903: dread d18/d22/d76/d7a/fc4 [0,4194304] 0 2024-02-28T16:31:24.956 INFO:tasks.workunit.client.1.smithi066.stdout:1/538: symlink d2/d27/d2b/d37/d48/la4 0 2024-02-28T16:31:24.956 INFO:tasks.workunit.client.1.smithi066.stdout:2/578: dread d0/d12/d2e/d3b/f64 [8388608,4194304] 0 2024-02-28T16:31:24.958 INFO:tasks.workunit.client.0.smithi001.stdout:5/878: write dc/d1a/d7d/d87/f8f [1947558,51925] 0 2024-02-28T16:31:24.961 INFO:tasks.workunit.client.1.smithi066.stdout:0/686: chown d2/d27/d41/d76/d93/d10/l38 29614335 1 2024-02-28T16:31:24.963 INFO:tasks.workunit.client.1.smithi066.stdout:6/522: dwrite d2/d2c/f95 [0,4194304] 0 2024-02-28T16:31:24.964 INFO:tasks.workunit.client.0.smithi001.stdout:0/904: creat d18/d22/d2b/d36/d48/d11a/d7e/f125 x:0 0 0 2024-02-28T16:31:24.964 INFO:tasks.workunit.client.1.smithi066.stdout:3/635: write db/d13/d14/f6f [1059712,82485] 0 2024-02-28T16:31:24.965 INFO:tasks.workunit.client.0.smithi001.stdout:3/944: creat d4/d1e/d7c/f134 x:0 0 0 2024-02-28T16:31:24.965 INFO:tasks.workunit.client.1.smithi066.stdout:2/579: fdatasync d0/d12/d1a/d23/fa6 0 2024-02-28T16:31:24.968 INFO:tasks.workunit.client.1.smithi066.stdout:8/518: symlink d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/la9 0 2024-02-28T16:31:24.968 INFO:tasks.workunit.client.0.smithi001.stdout:2/818: dwrite d0/d2/d3/de/f5e [4194304,4194304] 0 2024-02-28T16:31:24.970 INFO:tasks.workunit.client.0.smithi001.stdout:0/905: chown d18/d22/d2f/d1c/c61 4240546 1 2024-02-28T16:31:24.973 INFO:tasks.workunit.client.0.smithi001.stdout:8/889: getdents d4/d7/da8/d89 0 2024-02-28T16:31:24.973 INFO:tasks.workunit.client.0.smithi001.stdout:2/819: chown d0/d2/d3/d73/c86 206977232 1 2024-02-28T16:31:24.974 INFO:tasks.workunit.client.1.smithi066.stdout:0/687: creat d2/d27/d41/d76/d93/d33/d3f/d8e/fe3 x:0 0 0 2024-02-28T16:31:24.975 INFO:tasks.workunit.client.0.smithi001.stdout:7/938: creat d3/d17/d52/f12d x:0 0 0 2024-02-28T16:31:24.976 INFO:tasks.workunit.client.0.smithi001.stdout:8/890: chown d4/d47/d8b/f11b 197 1 2024-02-28T16:31:24.976 INFO:tasks.workunit.client.1.smithi066.stdout:8/519: dread - d0/d6/dc/d16/d4c/f8f zero size 2024-02-28T16:31:24.979 INFO:tasks.workunit.client.1.smithi066.stdout:6/523: mkdir d2/d26/d32/db9 0 2024-02-28T16:31:24.980 INFO:tasks.workunit.client.1.smithi066.stdout:0/688: chown d2/d27/d41/d76/d93/db2/db5/d51/fd9 2633 1 2024-02-28T16:31:24.981 INFO:tasks.workunit.client.0.smithi001.stdout:0/906: creat d18/d22/d2b/d36/d46/df5/dbc/f126 x:0 0 0 2024-02-28T16:31:24.982 INFO:tasks.workunit.client.1.smithi066.stdout:4/687: write d1/f6 [553132,128694] 0 2024-02-28T16:31:24.985 INFO:tasks.workunit.client.1.smithi066.stdout:9/448: dread da/d22/d46/f74 [0,4194304] 0 2024-02-28T16:31:24.986 INFO:tasks.workunit.client.0.smithi001.stdout:4/982: dread d3/d2d/d4c/d69/ff9 [0,4194304] 0 2024-02-28T16:31:24.991 INFO:tasks.workunit.client.1.smithi066.stdout:8/520: symlink d0/d6/dc/d16/d1a/d63/d58/d5a/laa 0 2024-02-28T16:31:24.992 INFO:tasks.workunit.client.1.smithi066.stdout:7/509: dread dc/de/d17/d1e/f53 [0,4194304] 0 2024-02-28T16:31:24.992 INFO:tasks.workunit.client.0.smithi001.stdout:7/939: getdents d3/d17/d52/d7f/d99/d115 0 2024-02-28T16:31:24.993 INFO:tasks.workunit.client.1.smithi066.stdout:4/688: mknod d1/d7a/ce1 0 2024-02-28T16:31:24.994 INFO:tasks.workunit.client.0.smithi001.stdout:8/891: getdents d4/d7/d22/d27/d58/d94/db4/dd5 0 2024-02-28T16:31:24.994 INFO:tasks.workunit.client.1.smithi066.stdout:3/636: creat db/dc/d11/d1f/fdd x:0 0 0 2024-02-28T16:31:24.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:24 smithi066 ceph-mon[36236]: from='client.24437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:24.999 INFO:tasks.workunit.client.1.smithi066.stdout:9/449: rename da/d3a/d15/d48/d76/c89 to da/d22/d46/d56/c8e 0 2024-02-28T16:31:24.999 INFO:tasks.workunit.client.1.smithi066.stdout:1/539: truncate d2/d27/f3c 1707333 0 2024-02-28T16:31:25.001 INFO:tasks.workunit.client.0.smithi001.stdout:3/945: dwrite d4/d1e/f25 [0,4194304] 0 2024-02-28T16:31:25.001 INFO:tasks.workunit.client.1.smithi066.stdout:2/580: link d0/f1f d0/d12/d1a/d39/d73/fbc 0 2024-02-28T16:31:25.001 INFO:tasks.workunit.client.0.smithi001.stdout:7/940: truncate d3/d17/fa5 161547 0 2024-02-28T16:31:25.005 INFO:tasks.workunit.client.0.smithi001.stdout:2/820: dwrite d0/d2/d3/de/d22/fd8 [0,4194304] 0 2024-02-28T16:31:25.005 INFO:tasks.workunit.client.1.smithi066.stdout:8/521: creat d0/d6/dc/d16/d1a/d63/d3d/fab x:0 0 0 2024-02-28T16:31:25.006 INFO:tasks.workunit.client.0.smithi001.stdout:7/941: write d3/d17/d3d/d59/d6e/de2/f116 [627080,61876] 0 2024-02-28T16:31:25.007 INFO:tasks.workunit.client.0.smithi001.stdout:2/821: dread - d0/d2/dd9/ffd zero size 2024-02-28T16:31:25.009 INFO:tasks.workunit.client.1.smithi066.stdout:4/689: rename d1/d18/d41/lcc to d1/d18/d41/da7/dae/dc1/le2 0 2024-02-28T16:31:25.014 INFO:tasks.workunit.client.1.smithi066.stdout:2/581: truncate d0/d12/d3e/f63 1336510 0 2024-02-28T16:31:25.019 INFO:tasks.workunit.client.1.smithi066.stdout:3/637: dread db/d7a/fa0 [0,4194304] 0 2024-02-28T16:31:25.019 INFO:tasks.workunit.client.0.smithi001.stdout:7/942: symlink d3/d17/d3d/db3/dff/l12e 0 2024-02-28T16:31:25.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:24 smithi001 ceph-mon[28991]: from='client.24437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:25.022 INFO:tasks.workunit.client.0.smithi001.stdout:2/822: creat d0/d2/d3/de/d61/dd5/f10b x:0 0 0 2024-02-28T16:31:25.023 INFO:tasks.workunit.client.1.smithi066.stdout:6/524: dread d2/da/d19/f41 [0,4194304] 0 2024-02-28T16:31:25.025 INFO:tasks.workunit.client.1.smithi066.stdout:1/540: mkdir d2/d18/da5 0 2024-02-28T16:31:25.026 INFO:tasks.workunit.client.1.smithi066.stdout:9/450: rename da/df/d18 to da/d22/d46/d56/d8f 0 2024-02-28T16:31:25.028 INFO:tasks.workunit.client.1.smithi066.stdout:0/689: getdents d2/d27/d41/d76/d93/db2/db5/d67 0 2024-02-28T16:31:25.029 INFO:tasks.workunit.client.0.smithi001.stdout:0/907: dwrite d18/d22/d2b/d95/fd3 [0,4194304] 0 2024-02-28T16:31:25.030 INFO:tasks.workunit.client.1.smithi066.stdout:3/638: chown db/d13/d14/d2d/d92/l95 998333 1 2024-02-28T16:31:25.032 INFO:tasks.workunit.client.0.smithi001.stdout:2/823: read - d0/f83 zero size 2024-02-28T16:31:25.033 INFO:tasks.workunit.client.0.smithi001.stdout:4/983: dwrite d3/f106 [0,4194304] 0 2024-02-28T16:31:25.033 INFO:tasks.workunit.client.1.smithi066.stdout:7/510: link dc/de/d87/l8b dc/d56/l90 0 2024-02-28T16:31:25.035 INFO:tasks.workunit.client.0.smithi001.stdout:7/943: mkdir d3/d17/d52/d7f/d99/dc1/d49/d8d/db9/d12f 0 2024-02-28T16:31:25.035 INFO:tasks.workunit.client.1.smithi066.stdout:2/582: creat d0/d12/d1a/d23/d5d/fbd x:0 0 0 2024-02-28T16:31:25.036 INFO:tasks.workunit.client.1.smithi066.stdout:7/511: chown dc/l1f 1914817 1 2024-02-28T16:31:25.038 INFO:tasks.workunit.client.0.smithi001.stdout:4/984: write d3/d5/d7/d1f/d2e/f39 [3864294,8797] 0 2024-02-28T16:31:25.040 INFO:tasks.workunit.client.0.smithi001.stdout:2/824: dread - d0/d2/da6/fb0 zero size 2024-02-28T16:31:25.047 INFO:tasks.workunit.client.0.smithi001.stdout:8/892: truncate d4/d7/f10 4894659 0 2024-02-28T16:31:25.047 INFO:tasks.workunit.client.0.smithi001.stdout:4/985: mkdir d3/d1c/d58/d6e/d13d 0 2024-02-28T16:31:25.047 INFO:tasks.workunit.client.1.smithi066.stdout:8/522: dwrite d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/fa4 [0,4194304] 0 2024-02-28T16:31:25.051 INFO:tasks.workunit.client.0.smithi001.stdout:2/825: mkdir d0/d2/d3/de/d61/d12/d6a/d10c 0 2024-02-28T16:31:25.051 INFO:tasks.workunit.client.1.smithi066.stdout:0/690: write d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77/fdd [587393,37167] 0 2024-02-28T16:31:25.052 INFO:tasks.workunit.client.1.smithi066.stdout:9/451: mkdir da/df/d70/d90 0 2024-02-28T16:31:25.053 INFO:tasks.workunit.client.0.smithi001.stdout:4/986: rename d3/d1c/da0/fd0 to d3/d2d/d4c/d69/d100/f13e 0 2024-02-28T16:31:25.054 INFO:tasks.workunit.client.1.smithi066.stdout:0/691: chown d2/f96 2062885 1 2024-02-28T16:31:25.055 INFO:tasks.workunit.client.1.smithi066.stdout:5/548: sync 2024-02-28T16:31:25.056 INFO:tasks.workunit.client.1.smithi066.stdout:3/639: creat db/d2a/d44/d48/d52/fde x:0 0 0 2024-02-28T16:31:25.058 INFO:tasks.workunit.client.1.smithi066.stdout:9/452: write da/d22/d46/d56/d8f/d28/f59 [697463,125142] 0 2024-02-28T16:31:25.063 INFO:tasks.workunit.client.1.smithi066.stdout:1/541: dwrite d2/d27/d2b/d43/d46/d80/f87 [0,4194304] 0 2024-02-28T16:31:25.067 INFO:tasks.workunit.client.1.smithi066.stdout:6/525: dread d2/d2c/da1/d5d/d99/f5e [0,4194304] 0 2024-02-28T16:31:25.070 INFO:tasks.workunit.client.1.smithi066.stdout:7/512: fsync dc/de/f13 0 2024-02-28T16:31:25.072 INFO:tasks.workunit.client.1.smithi066.stdout:2/583: write d0/d12/d1a/d23/d29/f6e [1237056,130197] 0 2024-02-28T16:31:25.075 INFO:tasks.workunit.client.1.smithi066.stdout:3/640: creat db/d13/d14/dad/fdf x:0 0 0 2024-02-28T16:31:25.076 INFO:tasks.workunit.client.0.smithi001.stdout:4/987: dwrite d3/d5/d7/d1f/d2e/f39 [0,4194304] 0 2024-02-28T16:31:25.079 INFO:tasks.workunit.client.1.smithi066.stdout:9/453: creat da/d22/d46/d56/f91 x:0 0 0 2024-02-28T16:31:25.080 INFO:tasks.workunit.client.0.smithi001.stdout:2/826: dread d0/d2/d3/de/d22/fd1 [0,4194304] 0 2024-02-28T16:31:25.087 INFO:tasks.workunit.client.0.smithi001.stdout:2/827: mknod d0/d2/d3/de/d61/d12/d6a/d10c/c10d 0 2024-02-28T16:31:25.088 INFO:tasks.workunit.client.1.smithi066.stdout:5/549: mknod d3/d4/dc/dd/d41/d7a/dac/d60/cb0 0 2024-02-28T16:31:25.090 INFO:tasks.workunit.client.1.smithi066.stdout:8/523: dwrite d0/d6/dc/d16/d4c/d6c/f91 [0,4194304] 0 2024-02-28T16:31:25.093 INFO:tasks.workunit.client.1.smithi066.stdout:1/542: read d2/d27/d2b/d43/d46/d68/f7e [602673,108156] 0 2024-02-28T16:31:25.096 INFO:tasks.workunit.client.0.smithi001.stdout:8/893: dread d4/d32/dd6/fd2 [0,4194304] 0 2024-02-28T16:31:25.098 INFO:tasks.workunit.client.0.smithi001.stdout:3/946: write d4/d1e/def/f71 [955464,74612] 0 2024-02-28T16:31:25.099 INFO:tasks.workunit.client.1.smithi066.stdout:2/584: creat d0/d12/d1a/d39/d7c/fbe x:0 0 0 2024-02-28T16:31:25.099 INFO:tasks.workunit.client.1.smithi066.stdout:8/524: chown d0/d6/dc/d16/d4c/d6c/d6f 1736 1 2024-02-28T16:31:25.100 INFO:tasks.workunit.client.0.smithi001.stdout:8/894: chown d4/c1b 17 1 2024-02-28T16:31:25.101 INFO:tasks.workunit.client.0.smithi001.stdout:4/988: dwrite d3/d5/d7/d1f/d2e/dde/f13a [0,4194304] 0 2024-02-28T16:31:25.104 INFO:tasks.workunit.client.1.smithi066.stdout:8/525: chown d0/db/d15/l69 60673 1 2024-02-28T16:31:25.104 INFO:tasks.workunit.client.1.smithi066.stdout:4/690: write d1/d18/f86 [566884,93636] 0 2024-02-28T16:31:25.105 INFO:tasks.workunit.client.1.smithi066.stdout:8/526: chown d0/db/d12/d38/f54 22234658 1 2024-02-28T16:31:25.110 INFO:tasks.workunit.client.1.smithi066.stdout:3/641: mknod db/dc/d35/d3d/d76/d89/ce0 0 2024-02-28T16:31:25.110 INFO:tasks.workunit.client.0.smithi001.stdout:6/913: sync 2024-02-28T16:31:25.111 INFO:tasks.workunit.client.0.smithi001.stdout:6/914: chown d9/db/d5f/d89 601 1 2024-02-28T16:31:25.115 INFO:tasks.workunit.client.1.smithi066.stdout:5/550: fdatasync d3/d4/dc/d56/f75 0 2024-02-28T16:31:25.116 INFO:tasks.workunit.client.0.smithi001.stdout:7/944: dwrite d3/d17/d61/faf [0,4194304] 0 2024-02-28T16:31:25.121 INFO:tasks.workunit.client.0.smithi001.stdout:3/947: mknod d4/d1e/c135 0 2024-02-28T16:31:25.122 INFO:tasks.workunit.client.1.smithi066.stdout:5/551: truncate d3/d4/dc/dd/d35/d92/fab 4664205 0 2024-02-28T16:31:25.122 INFO:tasks.workunit.client.0.smithi001.stdout:6/915: read d9/d39/dc4/f107 [1901886,97742] 0 2024-02-28T16:31:25.126 INFO:tasks.workunit.client.0.smithi001.stdout:3/948: write d4/d6/d7/d78/d81/da1/f128 [102170,20002] 0 2024-02-28T16:31:25.129 INFO:tasks.workunit.client.1.smithi066.stdout:7/513: getdents dc/de/d17/d37/d47/d88 0 2024-02-28T16:31:25.130 INFO:tasks.workunit.client.0.smithi001.stdout:3/949: read d4/d6/d7/d78/d81/fb2 [249909,10960] 0 2024-02-28T16:31:25.137 INFO:tasks.workunit.client.1.smithi066.stdout:4/691: creat d1/d18/d41/da7/dae/fe3 x:0 0 0 2024-02-28T16:31:25.139 INFO:tasks.workunit.client.0.smithi001.stdout:7/945: dwrite d3/d17/d3d/d5d/fbd [0,4194304] 0 2024-02-28T16:31:25.140 INFO:tasks.workunit.client.1.smithi066.stdout:2/585: mknod d0/d12/d1a/d23/d29/d21/d2f/cbf 0 2024-02-28T16:31:25.140 INFO:tasks.workunit.client.1.smithi066.stdout:3/642: write db/dc/d35/d3d/d76/f4a [793116,90954] 0 2024-02-28T16:31:25.140 INFO:tasks.workunit.client.0.smithi001.stdout:4/989: link d3/d5/d7/d1f/d3a/f70 d3/d1c/d58/d6e/d9f/f13f 0 2024-02-28T16:31:25.142 INFO:tasks.workunit.client.1.smithi066.stdout:7/514: mkdir dc/de/d17/d91 0 2024-02-28T16:31:25.143 INFO:tasks.workunit.client.1.smithi066.stdout:0/692: dwrite d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/dd1/fd6 [4194304,4194304] 0 2024-02-28T16:31:25.143 INFO:tasks.workunit.client.1.smithi066.stdout:8/527: read d0/d6/dc/d16/f50 [321287,37889] 0 2024-02-28T16:31:25.145 INFO:tasks.workunit.client.1.smithi066.stdout:1/543: rename d2/d27/d2b/d43/d46/d68 to d2/d27/d2b/d37/da6 0 2024-02-28T16:31:25.148 INFO:tasks.workunit.client.1.smithi066.stdout:3/643: stat db/d13/d1a/d25/c79 0 2024-02-28T16:31:25.148 INFO:tasks.workunit.client.1.smithi066.stdout:9/454: dread da/d22/d46/d56/d8f/f78 [0,4194304] 0 2024-02-28T16:31:25.150 INFO:tasks.workunit.client.0.smithi001.stdout:2/828: link d0/d2/d3/de/d61/d12/d55/fdf d0/d2/d3/de/d61/d12/d6a/d84/f10e 0 2024-02-28T16:31:25.150 INFO:tasks.workunit.client.1.smithi066.stdout:5/552: truncate d3/d4/dc/dd/d2e/d18/f4a 2634142 0 2024-02-28T16:31:25.152 INFO:tasks.workunit.client.1.smithi066.stdout:4/692: mknod d1/ce4 0 2024-02-28T16:31:25.153 INFO:tasks.workunit.client.0.smithi001.stdout:4/990: fsync d3/d2d/fc4 0 2024-02-28T16:31:25.154 INFO:tasks.workunit.client.1.smithi066.stdout:2/586: creat d0/d12/d2e/d7a/fc0 x:0 0 0 2024-02-28T16:31:25.157 INFO:tasks.workunit.client.1.smithi066.stdout:1/544: truncate d2/d27/d2b/d43/d46/f85 1011002 0 2024-02-28T16:31:25.157 INFO:tasks.workunit.client.0.smithi001.stdout:8/895: link d4/d32/dd6/fe6 d4/d9b/f11e 0 2024-02-28T16:31:25.159 INFO:tasks.workunit.client.1.smithi066.stdout:2/587: truncate d0/d12/d1a/d23/f7d 842961 0 2024-02-28T16:31:25.162 INFO:tasks.workunit.client.0.smithi001.stdout:6/916: link d9/d29/d6e/d4e/l51 d9/db/d5f/d89/l137 0 2024-02-28T16:31:25.162 INFO:tasks.workunit.client.1.smithi066.stdout:7/515: mkdir dc/d29/d92 0 2024-02-28T16:31:25.162 INFO:tasks.workunit.client.1.smithi066.stdout:8/528: mkdir d0/d6/dc/d16/d1a/d63/d58/dac 0 2024-02-28T16:31:25.162 INFO:tasks.workunit.client.1.smithi066.stdout:6/526: write d2/d2c/da1/d5d/d99/f39 [4602845,52713] 0 2024-02-28T16:31:25.165 INFO:tasks.workunit.client.0.smithi001.stdout:4/991: mkdir d3/d5/d7/ddb/d140 0 2024-02-28T16:31:25.168 INFO:tasks.workunit.client.1.smithi066.stdout:9/455: symlink da/d22/d46/d56/d8f/d2d/d81/l92 0 2024-02-28T16:31:25.168 INFO:tasks.workunit.client.0.smithi001.stdout:3/950: mknod d4/d1e/def/c136 0 2024-02-28T16:31:25.170 INFO:tasks.workunit.client.0.smithi001.stdout:8/896: dread - d4/d32/d11a/fd0 zero size 2024-02-28T16:31:25.171 INFO:tasks.workunit.client.0.smithi001.stdout:2/829: mkdir d0/d2/d3/de/d10f 0 2024-02-28T16:31:25.172 INFO:tasks.workunit.client.0.smithi001.stdout:6/917: mknod d9/db/d5f/d62/c138 0 2024-02-28T16:31:25.173 INFO:tasks.workunit.client.1.smithi066.stdout:9/456: truncate da/d3a/f13 9246556 0 2024-02-28T16:31:25.174 INFO:tasks.workunit.client.1.smithi066.stdout:0/693: read d2/d27/d41/d76/d93/db2/db5/d51/f82 [258703,121329] 0 2024-02-28T16:31:25.177 INFO:tasks.workunit.client.0.smithi001.stdout:4/992: rename d3/d5/d7/d1f/d2e/cfc to d3/d2d/d4c/d69/c141 0 2024-02-28T16:31:25.177 INFO:tasks.workunit.client.0.smithi001.stdout:7/946: dwrite d3/d17/d52/d92/f114 [0,4194304] 0 2024-02-28T16:31:25.178 INFO:tasks.workunit.client.1.smithi066.stdout:6/527: mkdir d2/d2c/dba 0 2024-02-28T16:31:25.178 INFO:tasks.workunit.client.1.smithi066.stdout:7/516: rename dc/de/d17/d37/d47/f5e to dc/de/d17/d37/d47/d88/f93 0 2024-02-28T16:31:25.178 INFO:tasks.workunit.client.0.smithi001.stdout:3/951: write d4/d1e/f68 [2472014,49684] 0 2024-02-28T16:31:25.181 INFO:tasks.workunit.client.1.smithi066.stdout:7/517: chown dc/c38 1222098 1 2024-02-28T16:31:25.182 INFO:tasks.workunit.client.0.smithi001.stdout:8/897: creat d4/d7/d22/d27/d4b/d77/f11f x:0 0 0 2024-02-28T16:31:25.182 INFO:tasks.workunit.client.0.smithi001.stdout:2/830: symlink d0/d2/d3/de/d61/d12/d55/db2/l110 0 2024-02-28T16:31:25.183 INFO:tasks.workunit.client.0.smithi001.stdout:8/898: write d4/d47/df4/f10e [415365,19456] 0 2024-02-28T16:31:25.184 INFO:tasks.workunit.client.1.smithi066.stdout:8/529: dread d0/f43 [0,4194304] 0 2024-02-28T16:31:25.185 INFO:tasks.workunit.client.1.smithi066.stdout:9/457: chown da/d22/d46/d56/d8f/d2d/f8d 765 1 2024-02-28T16:31:25.187 INFO:tasks.workunit.client.1.smithi066.stdout:6/528: fsync d2/d26/fa3 0 2024-02-28T16:31:25.188 INFO:tasks.workunit.client.0.smithi001.stdout:6/918: mkdir d9/d29/d6e/d139 0 2024-02-28T16:31:25.188 INFO:tasks.workunit.client.1.smithi066.stdout:1/545: rename d2/d27/d2b/d37/d54/d84/l95 to d2/da0/la7 0 2024-02-28T16:31:25.190 INFO:tasks.workunit.client.0.smithi001.stdout:7/947: dread - d3/d17/d3d/d59/d6e/fd6 zero size 2024-02-28T16:31:25.191 INFO:tasks.workunit.client.1.smithi066.stdout:0/694: fdatasync d2/d27/d41/d76/d93/d33/d3f/fbb 0 2024-02-28T16:31:25.194 INFO:tasks.workunit.client.0.smithi001.stdout:6/919: rmdir d9/db/d5f/d62/d7e/d124 39 2024-02-28T16:31:25.195 INFO:tasks.workunit.client.1.smithi066.stdout:8/530: write d0/d6/dc/d16/d1a/f88 [445914,18379] 0 2024-02-28T16:31:25.197 INFO:tasks.workunit.client.1.smithi066.stdout:7/518: write dc/de/d17/d1e/f53 [1326899,41252] 0 2024-02-28T16:31:25.197 INFO:tasks.workunit.client.0.smithi001.stdout:6/920: chown d9/d29/d6e/d4e/f131 122 1 2024-02-28T16:31:25.197 INFO:tasks.workunit.client.0.smithi001.stdout:8/899: rename d4/d7/d22/d27/d58/d94/dbc/fbd to d4/d7/d22/d27/d58/d94/db4/f120 0 2024-02-28T16:31:25.198 INFO:tasks.workunit.client.1.smithi066.stdout:2/588: dwrite d0/d12/f82 [0,4194304] 0 2024-02-28T16:31:25.200 INFO:tasks.workunit.client.0.smithi001.stdout:7/948: creat d3/d17/d3d/d59/d6e/f130 x:0 0 0 2024-02-28T16:31:25.202 INFO:tasks.workunit.client.0.smithi001.stdout:6/921: unlink d9/db/d5f/d89/l9b 0 2024-02-28T16:31:25.203 INFO:tasks.workunit.client.0.smithi001.stdout:8/900: write d4/d7/d22/d27/d4b/fa3 [4248739,76094] 0 2024-02-28T16:31:25.205 INFO:tasks.workunit.client.1.smithi066.stdout:6/529: chown d2/d26/d32/lb3 29 1 2024-02-28T16:31:25.206 INFO:tasks.workunit.client.0.smithi001.stdout:8/901: write d4/d7/da8/f91 [8257270,131] 0 2024-02-28T16:31:25.207 INFO:tasks.workunit.client.1.smithi066.stdout:8/531: dread - d0/db/d12/d1d/d25/d4f/f9f zero size 2024-02-28T16:31:25.208 INFO:tasks.workunit.client.0.smithi001.stdout:8/902: dread - d4/d47/d8b/f103 zero size 2024-02-28T16:31:25.208 INFO:tasks.workunit.client.1.smithi066.stdout:5/553: dwrite d3/d4/dc/d2f/f32 [0,4194304] 0 2024-02-28T16:31:25.211 INFO:tasks.workunit.client.1.smithi066.stdout:0/695: chown d2/d27/d41/d76/d93/d10/d23/ca5 745 1 2024-02-28T16:31:25.216 INFO:tasks.workunit.client.0.smithi001.stdout:2/831: write d0/d2/d3/de/d61/d12/d55/fb1 [1037630,62536] 0 2024-02-28T16:31:25.216 INFO:tasks.workunit.client.0.smithi001.stdout:3/952: write d4/d6/f51 [1355356,100055] 0 2024-02-28T16:31:25.217 INFO:tasks.workunit.client.1.smithi066.stdout:7/519: rename dc/d29/c89 to dc/de/d17/d37/d47/d88/c94 0 2024-02-28T16:31:25.218 INFO:tasks.workunit.client.1.smithi066.stdout:6/530: creat d2/d2c/fbb x:0 0 0 2024-02-28T16:31:25.220 INFO:tasks.workunit.client.1.smithi066.stdout:4/693: dwrite d1/d18/f2b [0,4194304] 0 2024-02-28T16:31:25.220 INFO:tasks.workunit.client.1.smithi066.stdout:3/644: dread db/dc/d35/d3d/d76/f4a [0,4194304] 0 2024-02-28T16:31:25.225 INFO:tasks.workunit.client.1.smithi066.stdout:1/546: creat d2/d27/d2b/d34/d9e/fa8 x:0 0 0 2024-02-28T16:31:25.225 INFO:tasks.workunit.client.1.smithi066.stdout:8/532: dread d0/db/d12/d1d/d25/d4f/f86 [0,4194304] 0 2024-02-28T16:31:25.226 INFO:tasks.workunit.client.0.smithi001.stdout:6/922: fdatasync d9/db/d5f/d8b/f6d 0 2024-02-28T16:31:25.230 INFO:tasks.workunit.client.1.smithi066.stdout:5/554: mkdir d3/d4/dc/dd/d41/d7a/dac/d60/db1 0 2024-02-28T16:31:25.230 INFO:tasks.workunit.client.1.smithi066.stdout:7/520: creat dc/de/d17/d1e/d45/f95 x:0 0 0 2024-02-28T16:31:25.231 INFO:tasks.workunit.client.0.smithi001.stdout:7/949: dwrite d3/d17/d3d/d59/f67 [0,4194304] 0 2024-02-28T16:31:25.231 INFO:tasks.workunit.client.0.smithi001.stdout:2/832: symlink d0/l111 0 2024-02-28T16:31:25.232 INFO:tasks.workunit.client.1.smithi066.stdout:5/555: readlink d3/l6 0 2024-02-28T16:31:25.232 INFO:tasks.workunit.client.0.smithi001.stdout:4/993: dwrite d3/d2d/d4c/d5e/db0/fd9 [4194304,4194304] 0 2024-02-28T16:31:25.233 INFO:tasks.workunit.client.0.smithi001.stdout:6/923: rmdir d9/d29/d6a/ded/d105 39 2024-02-28T16:31:25.235 INFO:tasks.workunit.client.1.smithi066.stdout:5/556: readlink d3/d4/l1c 0 2024-02-28T16:31:25.236 INFO:tasks.workunit.client.1.smithi066.stdout:3/645: creat db/d13/d14/dad/fe1 x:0 0 0 2024-02-28T16:31:25.240 INFO:tasks.workunit.client.1.smithi066.stdout:5/557: dread - d3/d4/dc/dd/f89 zero size 2024-02-28T16:31:25.245 INFO:tasks.workunit.client.1.smithi066.stdout:9/458: truncate da/d22/d46/d56/f72 2001204 0 2024-02-28T16:31:25.245 INFO:tasks.workunit.client.1.smithi066.stdout:5/558: readlink d3/d4/dc/d2f/l53 0 2024-02-28T16:31:25.248 INFO:tasks.workunit.client.1.smithi066.stdout:8/533: unlink d0/d6/dc/d16/d4c/d42/f9c 0 2024-02-28T16:31:25.255 INFO:tasks.workunit.client.1.smithi066.stdout:3/646: symlink db/d13/le2 0 2024-02-28T16:31:25.255 INFO:tasks.workunit.client.1.smithi066.stdout:1/547: dwrite d2/d27/d2b/d43/d46/f6c [0,4194304] 0 2024-02-28T16:31:25.255 INFO:tasks.workunit.client.0.smithi001.stdout:5/879: sync 2024-02-28T16:31:25.255 INFO:tasks.workunit.client.0.smithi001.stdout:0/908: sync 2024-02-28T16:31:25.257 INFO:tasks.workunit.client.0.smithi001.stdout:4/994: read - d3/d2d/d83/d112/f12b zero size 2024-02-28T16:31:25.259 INFO:tasks.workunit.client.0.smithi001.stdout:7/950: write d3/d17/d3d/d42/f7d [2728166,74125] 0 2024-02-28T16:31:25.263 INFO:tasks.workunit.client.1.smithi066.stdout:0/696: dwrite d2/d27/d41/d76/d93/d33/d3f/faa [0,4194304] 0 2024-02-28T16:31:25.264 INFO:tasks.workunit.client.1.smithi066.stdout:9/459: creat da/df/d70/d90/f93 x:0 0 0 2024-02-28T16:31:25.267 INFO:tasks.workunit.client.1.smithi066.stdout:3/647: mkdir db/d13/d14/d2d/de3 0 2024-02-28T16:31:25.267 INFO:tasks.workunit.client.1.smithi066.stdout:8/534: write d0/d6/f9a [2941236,50503] 0 2024-02-28T16:31:25.267 INFO:tasks.workunit.client.0.smithi001.stdout:8/903: dwrite d4/d9b/fa7 [0,4194304] 0 2024-02-28T16:31:25.269 INFO:tasks.workunit.client.0.smithi001.stdout:0/909: mknod d18/d22/c127 0 2024-02-28T16:31:25.269 INFO:tasks.workunit.client.1.smithi066.stdout:9/460: stat da/d3a/f5f 0 2024-02-28T16:31:25.273 INFO:tasks.workunit.client.0.smithi001.stdout:2/833: read d0/d2/d3/de/d22/fb9 [39144,95227] 0 2024-02-28T16:31:25.276 INFO:tasks.workunit.client.1.smithi066.stdout:3/648: readlink db/dc/d35/d3d/d76/d89/dcc/lc5 0 2024-02-28T16:31:25.281 INFO:tasks.workunit.client.1.smithi066.stdout:6/531: dwrite d2/d2c/da1/d5d/d63/d6a/fb2 [0,4194304] 0 2024-02-28T16:31:25.285 INFO:tasks.workunit.client.1.smithi066.stdout:5/559: rename d3/d4/dc/dd/d41/d7a/dac/f6d to d3/d4/dc/dd/d41/d7a/dac/d60/fb2 0 2024-02-28T16:31:25.286 INFO:tasks.workunit.client.1.smithi066.stdout:8/535: write d0/db/d12/d1d/d25/d72/f77 [970126,127698] 0 2024-02-28T16:31:25.286 INFO:tasks.workunit.client.0.smithi001.stdout:8/904: fsync d4/d7/d22/d27/d58/d94/faa 0 2024-02-28T16:31:25.288 INFO:tasks.workunit.client.1.smithi066.stdout:7/521: dwrite dc/de/d17/d1e/f59 [0,4194304] 0 2024-02-28T16:31:25.290 INFO:tasks.workunit.client.1.smithi066.stdout:9/461: unlink da/d22/d46/d56/d8f/d2d/f50 0 2024-02-28T16:31:25.292 INFO:tasks.workunit.client.0.smithi001.stdout:5/880: link f5 dc/d12/d41/d54/f118 0 2024-02-28T16:31:25.292 INFO:tasks.workunit.client.0.smithi001.stdout:7/951: creat d3/d17/d52/d7f/d99/d115/f131 x:0 0 0 2024-02-28T16:31:25.294 INFO:tasks.workunit.client.1.smithi066.stdout:7/522: readlink dc/de/d17/d37/l7a 0 2024-02-28T16:31:25.295 INFO:tasks.workunit.client.0.smithi001.stdout:8/905: readlink d4/d7/l54 0 2024-02-28T16:31:25.296 INFO:tasks.workunit.client.0.smithi001.stdout:2/834: dread d0/fc0 [0,4194304] 0 2024-02-28T16:31:25.299 INFO:tasks.workunit.client.1.smithi066.stdout:5/560: mkdir d3/d1b/db3 0 2024-02-28T16:31:25.300 INFO:tasks.workunit.client.1.smithi066.stdout:5/561: chown d3/d4/dc/dd/d2e/l6a 69990 1 2024-02-28T16:31:25.304 INFO:tasks.workunit.client.1.smithi066.stdout:6/532: mkdir d2/da/d19/d75/d88/dbc 0 2024-02-28T16:31:25.305 INFO:tasks.workunit.client.0.smithi001.stdout:0/910: getdents d18 0 2024-02-28T16:31:25.306 INFO:tasks.workunit.client.0.smithi001.stdout:7/952: rename d3/d17/d3d/d59/d6e to d3/d17/d3d/d59/d6e/de2/d132 22 2024-02-28T16:31:25.310 INFO:tasks.workunit.client.1.smithi066.stdout:0/697: rename d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/d77 to d2/d27/d41/d76/d93/db2/db5/d67/d94/de4 0 2024-02-28T16:31:25.311 INFO:tasks.workunit.client.0.smithi001.stdout:0/911: fdatasync d18/d22/d2f/d1c/f21 0 2024-02-28T16:31:25.312 INFO:tasks.workunit.client.0.smithi001.stdout:8/906: creat d4/d32/dd6/f121 x:0 0 0 2024-02-28T16:31:25.312 INFO:tasks.workunit.client.0.smithi001.stdout:7/953: truncate d3/d1c/f117 474657 0 2024-02-28T16:31:25.313 INFO:tasks.workunit.client.1.smithi066.stdout:8/536: mkdir d0/d6/dc/d16/d8c/dad 0 2024-02-28T16:31:25.316 INFO:tasks.workunit.client.1.smithi066.stdout:0/698: write d2/d27/d41/d76/d93/db2/db5/d67/fc0 [1004731,56077] 0 2024-02-28T16:31:25.318 INFO:tasks.workunit.client.0.smithi001.stdout:0/912: readlink d18/d22/d2f/d1c/l20 0 2024-02-28T16:31:25.319 INFO:tasks.workunit.client.1.smithi066.stdout:3/649: mkdir db/dc/db3/dc9/de4 0 2024-02-28T16:31:25.320 INFO:tasks.workunit.client.1.smithi066.stdout:7/523: dwrite dc/d29/f41 [0,4194304] 0 2024-02-28T16:31:25.322 INFO:tasks.workunit.client.1.smithi066.stdout:2/589: dread d0/d12/f62 [0,4194304] 0 2024-02-28T16:31:25.326 INFO:tasks.workunit.client.1.smithi066.stdout:6/533: rename d2/dac to d2/da/d19/d75/dbd 0 2024-02-28T16:31:25.328 INFO:tasks.workunit.client.0.smithi001.stdout:0/913: read d18/fac [366944,106120] 0 2024-02-28T16:31:25.330 INFO:tasks.workunit.client.0.smithi001.stdout:7/954: creat d3/d17/d3d/d59/d104/f133 x:0 0 0 2024-02-28T16:31:25.330 INFO:tasks.workunit.client.0.smithi001.stdout:8/907: creat d4/d7/d22/d27/d4b/d84/f122 x:0 0 0 2024-02-28T16:31:25.332 INFO:tasks.workunit.client.1.smithi066.stdout:7/524: rename dc/de/d17/d1e/d45/c7b to dc/de/d17/d37/c96 0 2024-02-28T16:31:25.332 INFO:tasks.workunit.client.0.smithi001.stdout:0/914: readlink d18/d22/d2f/d1c/ldf 0 2024-02-28T16:31:25.333 INFO:tasks.workunit.client.0.smithi001.stdout:4/995: dread d3/d5/d7/d1f/d2e/f64 [0,4194304] 0 2024-02-28T16:31:25.334 INFO:tasks.workunit.client.1.smithi066.stdout:8/537: symlink d0/lae 0 2024-02-28T16:31:25.334 INFO:tasks.workunit.client.0.smithi001.stdout:5/881: link dc/d1a/d21/l4e dc/d1a/dba/l119 0 2024-02-28T16:31:25.335 INFO:tasks.workunit.client.1.smithi066.stdout:7/525: dread - dc/de/d17/d37/f8d zero size 2024-02-28T16:31:25.336 INFO:tasks.workunit.client.1.smithi066.stdout:6/534: mkdir d2/d26/dbe 0 2024-02-28T16:31:25.339 INFO:tasks.workunit.client.1.smithi066.stdout:2/590: creat d0/d12/d2e/fc1 x:0 0 0 2024-02-28T16:31:25.340 INFO:tasks.workunit.client.1.smithi066.stdout:7/526: mkdir dc/d56/d97 0 2024-02-28T16:31:25.341 INFO:tasks.workunit.client.1.smithi066.stdout:8/538: mkdir d0/d6/dc/d16/d4c/d6c/daf 0 2024-02-28T16:31:25.342 INFO:tasks.workunit.client.1.smithi066.stdout:3/650: link db/d7a/lba db/d2a/d44/d48/d52/d77/le5 0 2024-02-28T16:31:25.345 INFO:tasks.workunit.client.1.smithi066.stdout:8/539: read - d0/db/d12/d1d/d25/d4f/f9f zero size 2024-02-28T16:31:25.346 INFO:tasks.workunit.client.1.smithi066.stdout:6/535: mkdir d2/da/d19/d75/da6/daa/dbf 0 2024-02-28T16:31:25.350 INFO:tasks.workunit.client.1.smithi066.stdout:2/591: symlink d0/d12/d1a/d23/d29/d21/lc2 0 2024-02-28T16:31:25.351 INFO:tasks.workunit.client.0.smithi001.stdout:5/882: mkdir dc/d12/d41/d54/d11a 0 2024-02-28T16:31:25.354 INFO:tasks.workunit.client.1.smithi066.stdout:3/651: creat db/d13/d14/dda/fe6 x:0 0 0 2024-02-28T16:31:25.355 INFO:tasks.workunit.client.0.smithi001.stdout:6/924: dwrite d9/db/d5f/d62/d83/f8f [0,4194304] 0 2024-02-28T16:31:25.357 INFO:tasks.workunit.client.1.smithi066.stdout:8/540: mknod d0/d6/dc/d16/cb0 0 2024-02-28T16:31:25.359 INFO:tasks.workunit.client.1.smithi066.stdout:7/527: link dc/de/d17/f19 dc/de/d17/d1e/f98 0 2024-02-28T16:31:25.362 INFO:tasks.workunit.client.1.smithi066.stdout:8/541: chown d0/db/d12/d1d/d25/d72 0 1 2024-02-28T16:31:25.362 INFO:tasks.workunit.client.1.smithi066.stdout:2/592: mknod d0/d12/d1a/d23/d29/d21/d2f/cc3 0 2024-02-28T16:31:25.362 INFO:tasks.workunit.client.0.smithi001.stdout:6/925: write d9/db/d5f/d89/d8a/f104 [52661,22339] 0 2024-02-28T16:31:25.363 INFO:tasks.workunit.client.1.smithi066.stdout:2/593: chown d0/f31 515302 1 2024-02-28T16:31:25.364 INFO:tasks.workunit.client.1.smithi066.stdout:3/652: symlink db/d13/d14/d2d/d92/le7 0 2024-02-28T16:31:25.365 INFO:tasks.workunit.client.1.smithi066.stdout:1/548: dwrite d2/d27/d2b/d37/da6/f7e [0,4194304] 0 2024-02-28T16:31:25.365 INFO:tasks.workunit.client.1.smithi066.stdout:8/542: chown d0/db/d12/d1d/la6 14627724 1 2024-02-28T16:31:25.368 INFO:tasks.workunit.client.0.smithi001.stdout:3/953: sync 2024-02-28T16:31:25.369 INFO:tasks.workunit.client.0.smithi001.stdout:2/835: dwrite d0/f59 [0,4194304] 0 2024-02-28T16:31:25.369 INFO:tasks.workunit.client.1.smithi066.stdout:2/594: write d0/f1e [3334068,85201] 0 2024-02-28T16:31:25.378 INFO:tasks.workunit.client.1.smithi066.stdout:3/653: rename db/d13/d1a/c20 to db/dc/db3/dc9/ce8 0 2024-02-28T16:31:25.379 INFO:tasks.workunit.client.1.smithi066.stdout:6/536: dwrite d2/da/d19/f50 [0,4194304] 0 2024-02-28T16:31:25.382 INFO:tasks.workunit.client.0.smithi001.stdout:7/955: dwrite d3/d17/d52/d7f/d99/dc1/f11d [0,4194304] 0 2024-02-28T16:31:25.388 INFO:tasks.workunit.client.1.smithi066.stdout:7/528: fdatasync dc/de/d17/d1e/d45/f8f 0 2024-02-28T16:31:25.395 INFO:tasks.workunit.client.1.smithi066.stdout:3/654: fdatasync db/dc/f55 0 2024-02-28T16:31:25.397 INFO:tasks.workunit.client.0.smithi001.stdout:4/996: mkdir d3/d2d/d4c/d5e/d142 0 2024-02-28T16:31:25.400 INFO:tasks.workunit.client.1.smithi066.stdout:7/529: write dc/de/d17/d1e/f28 [4791309,52423] 0 2024-02-28T16:31:25.403 INFO:tasks.workunit.client.0.smithi001.stdout:5/883: mkdir dc/d1a/d21/d11b 0 2024-02-28T16:31:25.405 INFO:tasks.workunit.client.1.smithi066.stdout:6/537: mkdir d2/da/d19/d9c/dc0 0 2024-02-28T16:31:25.410 INFO:tasks.workunit.client.1.smithi066.stdout:9/462: dwrite da/f35 [0,4194304] 0 2024-02-28T16:31:25.410 INFO:tasks.workunit.client.1.smithi066.stdout:3/655: mkdir db/d2a/de9 0 2024-02-28T16:31:25.412 INFO:tasks.workunit.client.1.smithi066.stdout:9/463: chown da/c33 4 1 2024-02-28T16:31:25.413 INFO:tasks.workunit.client.1.smithi066.stdout:5/562: dwrite d3/d4/dc/dd/d41/d5d/f69 [0,4194304] 0 2024-02-28T16:31:25.415 INFO:tasks.workunit.client.0.smithi001.stdout:8/908: rename d4/d7/d22/d27/d4b/d10f to d4/d7/d22/d123 0 2024-02-28T16:31:25.417 INFO:tasks.workunit.client.0.smithi001.stdout:5/884: stat dc/d1a/l26 0 2024-02-28T16:31:25.417 INFO:tasks.workunit.client.0.smithi001.stdout:2/836: mknod d0/c112 0 2024-02-28T16:31:25.419 INFO:tasks.workunit.client.0.smithi001.stdout:4/997: read d3/d1c/d58/f6b [2033987,103255] 0 2024-02-28T16:31:25.420 INFO:tasks.workunit.client.1.smithi066.stdout:7/530: write dc/de/d17/d37/d47/d88/f93 [3874989,16328] 0 2024-02-28T16:31:25.423 INFO:tasks.workunit.client.0.smithi001.stdout:2/837: write d0/d2/d3/de/d61/d12/d55/db2/fe6 [1114884,62077] 0 2024-02-28T16:31:25.424 INFO:tasks.workunit.client.0.smithi001.stdout:6/926: dread d9/db/d5f/d69/f75 [0,4194304] 0 2024-02-28T16:31:25.426 INFO:tasks.workunit.client.0.smithi001.stdout:4/998: creat d3/d1c/d58/d6c/f143 x:0 0 0 2024-02-28T16:31:25.426 INFO:tasks.workunit.client.1.smithi066.stdout:3/656: symlink db/dc/d11/d1f/lea 0 2024-02-28T16:31:25.427 INFO:tasks.workunit.client.0.smithi001.stdout:5/885: creat dc/d12/d41/dd6/f11c x:0 0 0 2024-02-28T16:31:25.428 INFO:tasks.workunit.client.0.smithi001.stdout:7/956: getdents d3/d17/d52/d55/dca/dfe 0 2024-02-28T16:31:25.430 INFO:tasks.workunit.client.1.smithi066.stdout:3/657: fdatasync db/d2a/d44/fc6 0 2024-02-28T16:31:25.432 INFO:tasks.workunit.client.0.smithi001.stdout:3/954: link d4/d1e/d7c/d7d/df8/dee/c10f d4/d18/ddb/dfb/c137 0 2024-02-28T16:31:25.434 INFO:tasks.workunit.client.0.smithi001.stdout:6/927: mkdir d9/db/d5f/d62/def/d13a 0 2024-02-28T16:31:25.435 INFO:tasks.workunit.client.1.smithi066.stdout:9/464: mkdir da/d22/d46/d56/d8f/d94 0 2024-02-28T16:31:25.438 INFO:tasks.workunit.client.0.smithi001.stdout:4/999: creat d3/d1c/d58/d88/f144 x:0 0 0 2024-02-28T16:31:25.439 INFO:tasks.workunit.client.1.smithi066.stdout:9/465: fdatasync da/d3a/d15/d48/f66 0 2024-02-28T16:31:25.441 INFO:tasks.workunit.client.0.smithi001.stdout:3/955: mkdir d4/d6/d7/d78/d81/d138 0 2024-02-28T16:31:25.441 INFO:tasks.workunit.client.0.smithi001.stdout:8/909: dwrite d4/d32/f7c [4194304,4194304] 0 2024-02-28T16:31:25.441 INFO:tasks.workunit.client.0.smithi001.stdout:3/956: stat d4/d6/fb5 0 2024-02-28T16:31:25.442 INFO:tasks.workunit.client.1.smithi066.stdout:5/563: dwrite d3/d4/dc/dd/d2e/d18/f1a [4194304,4194304] 0 2024-02-28T16:31:25.445 INFO:tasks.workunit.client.0.smithi001.stdout:6/928: symlink d9/d29/d6a/l13b 0 2024-02-28T16:31:25.446 INFO:tasks.workunit.client.0.smithi001.stdout:2/838: link d0/d2/d3/d36/ce2 d0/d2/d3/de/d22/dbb/dc8/c113 0 2024-02-28T16:31:25.449 INFO:tasks.workunit.client.0.smithi001.stdout:2/839: chown d0/d2/d3/de/d8a/fbd 62 1 2024-02-28T16:31:25.454 INFO:tasks.workunit.client.0.smithi001.stdout:2/840: fdatasync d0/d2/d3/de/d61/d12/f27 0 2024-02-28T16:31:25.455 INFO:tasks.workunit.client.1.smithi066.stdout:7/531: dwrite dc/de/d17/d37/f54 [0,4194304] 0 2024-02-28T16:31:25.456 INFO:tasks.workunit.client.0.smithi001.stdout:8/910: truncate d4/f43 1355513 0 2024-02-28T16:31:25.456 INFO:tasks.workunit.client.1.smithi066.stdout:8/543: truncate d0/d6/f1f 2354383 0 2024-02-28T16:31:25.457 INFO:tasks.workunit.client.1.smithi066.stdout:5/564: rmdir d3/d4/dc/d56 39 2024-02-28T16:31:25.458 INFO:tasks.workunit.client.1.smithi066.stdout:8/544: chown d0/db/d15 45589 1 2024-02-28T16:31:25.461 INFO:tasks.workunit.client.0.smithi001.stdout:8/911: rmdir d4/d7/d22/d27/d58 39 2024-02-28T16:31:25.465 INFO:tasks.workunit.client.1.smithi066.stdout:2/595: truncate d0/d12/f5c 1800160 0 2024-02-28T16:31:25.466 INFO:tasks.workunit.client.1.smithi066.stdout:3/658: creat db/d2a/d44/dd8/feb x:0 0 0 2024-02-28T16:31:25.466 INFO:tasks.workunit.client.0.smithi001.stdout:2/841: creat d0/d2/d3/de/d61/d12/d55/f114 x:0 0 0 2024-02-28T16:31:25.467 INFO:tasks.workunit.client.1.smithi066.stdout:9/466: mknod da/d4b/d7c/c95 0 2024-02-28T16:31:25.467 INFO:tasks.workunit.client.0.smithi001.stdout:5/886: dread dc/f27 [0,4194304] 0 2024-02-28T16:31:25.471 INFO:tasks.workunit.client.0.smithi001.stdout:5/887: stat dc/c1b 0 2024-02-28T16:31:25.472 INFO:tasks.workunit.client.0.smithi001.stdout:6/929: dwrite d9/f119 [0,4194304] 0 2024-02-28T16:31:25.476 INFO:tasks.workunit.client.1.smithi066.stdout:9/467: getdents da/d4b/d53 0 2024-02-28T16:31:25.477 INFO:tasks.workunit.client.0.smithi001.stdout:5/888: rmdir dc/d1a 39 2024-02-28T16:31:25.481 INFO:tasks.workunit.client.1.smithi066.stdout:3/659: dread db/d2a/d44/d48/d5d/f81 [0,4194304] 0 2024-02-28T16:31:25.482 INFO:tasks.workunit.client.1.smithi066.stdout:9/468: chown da/f20 230 1 2024-02-28T16:31:25.483 INFO:tasks.workunit.client.0.smithi001.stdout:6/930: symlink d9/db/d5f/d69/l13c 0 2024-02-28T16:31:25.485 INFO:tasks.workunit.client.1.smithi066.stdout:2/596: truncate d0/d12/d1a/d23/d5d/fbd 824308 0 2024-02-28T16:31:25.485 INFO:tasks.workunit.client.0.smithi001.stdout:2/842: rename d0/d2/d3/de/d61/d12/d55/fdf to d0/d2/dd9/f115 0 2024-02-28T16:31:25.486 INFO:tasks.workunit.client.1.smithi066.stdout:1/549: dread d2/d18/d19/f24 [4194304,4194304] 0 2024-02-28T16:31:25.492 INFO:tasks.workunit.client.1.smithi066.stdout:6/538: dwrite d2/da/f1e [0,4194304] 0 2024-02-28T16:31:25.494 INFO:tasks.workunit.client.0.smithi001.stdout:2/843: unlink d0/d2/d3/de/f8f 0 2024-02-28T16:31:25.496 INFO:tasks.workunit.client.1.smithi066.stdout:5/565: dwrite d3/d4/dc/dd/d35/d86/f88 [0,4194304] 0 2024-02-28T16:31:25.496 INFO:tasks.workunit.client.1.smithi066.stdout:2/597: write d0/d12/d1a/f2b [1006048,95700] 0 2024-02-28T16:31:25.496 INFO:tasks.workunit.client.1.smithi066.stdout:3/660: mknod db/d7a/cec 0 2024-02-28T16:31:25.496 INFO:tasks.workunit.client.0.smithi001.stdout:5/889: rename dc/d1a/l26 to dc/d12/d41/d54/l11d 0 2024-02-28T16:31:25.499 INFO:tasks.workunit.client.0.smithi001.stdout:5/890: write dc/d12/d56/d96/db9/dc2/dca/d3c/f53 [2149687,87904] 0 2024-02-28T16:31:25.501 INFO:tasks.workunit.client.0.smithi001.stdout:2/844: creat d0/d2/d3/de/d22/dbb/dc8/f116 x:0 0 0 2024-02-28T16:31:25.501 INFO:tasks.workunit.client.0.smithi001.stdout:5/891: dread - dc/df/d79/f115 zero size 2024-02-28T16:31:25.502 INFO:tasks.workunit.client.1.smithi066.stdout:8/545: dwrite d0/d6/dc/d16/d4c/d6c/f91 [4194304,4194304] 0 2024-02-28T16:31:25.506 INFO:tasks.workunit.client.1.smithi066.stdout:1/550: mkdir d2/d27/d2b/d34/da9 0 2024-02-28T16:31:25.507 INFO:tasks.workunit.client.0.smithi001.stdout:5/892: symlink dc/d12/d41/db7/l11e 0 2024-02-28T16:31:25.508 INFO:tasks.workunit.client.1.smithi066.stdout:3/661: mkdir db/dc/d11/daa/ded 0 2024-02-28T16:31:25.508 INFO:tasks.workunit.client.1.smithi066.stdout:9/469: link da/d22/d46/d56/d8f/d28/f2b da/d3a/d58/f96 0 2024-02-28T16:31:25.508 INFO:tasks.workunit.client.0.smithi001.stdout:2/845: mkdir d0/d2/d3/de/d22/dbb/dc8/d117 0 2024-02-28T16:31:25.515 INFO:tasks.workunit.client.0.smithi001.stdout:0/915: write d18/d22/d2b/d36/d46/fb8 [1465359,49998] 0 2024-02-28T16:31:25.518 INFO:tasks.workunit.client.1.smithi066.stdout:6/539: creat d2/fc1 x:0 0 0 2024-02-28T16:31:25.519 INFO:tasks.workunit.client.1.smithi066.stdout:8/546: rename d0/d6/dc/d16/d8c to d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1 0 2024-02-28T16:31:25.520 INFO:tasks.workunit.client.0.smithi001.stdout:2/846: symlink d0/d2/d3/de/d61/d12/d6a/d10c/l118 0 2024-02-28T16:31:25.522 INFO:tasks.workunit.client.0.smithi001.stdout:5/893: chown dc/d12/d56/fdb 881304 1 2024-02-28T16:31:25.523 INFO:tasks.workunit.client.1.smithi066.stdout:1/551: creat d2/d27/d2b/d37/da6/faa x:0 0 0 2024-02-28T16:31:25.523 INFO:tasks.workunit.client.1.smithi066.stdout:3/662: dread - db/dc/d11/d1f/d78/fa3 zero size 2024-02-28T16:31:25.527 INFO:tasks.workunit.client.1.smithi066.stdout:7/532: dread dc/de/d17/d1e/d45/f79 [0,4194304] 0 2024-02-28T16:31:25.528 INFO:tasks.workunit.client.0.smithi001.stdout:5/894: read - dc/d1a/d7d/fcd zero size 2024-02-28T16:31:25.530 INFO:tasks.workunit.client.1.smithi066.stdout:5/566: read d3/d1b/d20/f63 [2543339,20298] 0 2024-02-28T16:31:25.531 INFO:tasks.workunit.client.1.smithi066.stdout:6/540: creat d2/d2c/da1/d5d/d63/fc2 x:0 0 0 2024-02-28T16:31:25.531 INFO:tasks.workunit.client.0.smithi001.stdout:7/957: write d3/db7/fba [788448,117861] 0 2024-02-28T16:31:25.532 INFO:tasks.workunit.client.1.smithi066.stdout:9/470: getdents da/d3a/d15 0 2024-02-28T16:31:25.533 INFO:tasks.workunit.client.0.smithi001.stdout:2/847: chown d0/d2/d3/d36/ce2 0 1 2024-02-28T16:31:25.535 INFO:tasks.workunit.client.1.smithi066.stdout:7/533: mknod dc/c99 0 2024-02-28T16:31:25.536 INFO:tasks.workunit.client.0.smithi001.stdout:7/958: write d3/d17/d3d/d42/f10a [680168,2057] 0 2024-02-28T16:31:25.536 INFO:tasks.workunit.client.1.smithi066.stdout:1/552: read d2/d27/d2b/d43/d46/f61 [3145417,60853] 0 2024-02-28T16:31:25.538 INFO:tasks.workunit.client.0.smithi001.stdout:0/916: dread d18/d22/d2b/d36/d46/df5/d4f/fe8 [0,4194304] 0 2024-02-28T16:31:25.540 INFO:tasks.workunit.client.1.smithi066.stdout:3/663: mkdir db/dc/d11/daa/ded/dee 0 2024-02-28T16:31:25.544 INFO:tasks.workunit.client.1.smithi066.stdout:8/547: dread d0/db/d12/d1d/d25/d72/f77 [0,4194304] 0 2024-02-28T16:31:25.545 INFO:tasks.workunit.client.1.smithi066.stdout:3/664: write db/fbd [4779337,2659] 0 2024-02-28T16:31:25.546 INFO:tasks.workunit.client.1.smithi066.stdout:8/548: write d0/db/d15/f37 [947982,34657] 0 2024-02-28T16:31:25.547 INFO:tasks.workunit.client.1.smithi066.stdout:9/471: rename da/d4b/d7c to da/d3a/d58/d97 0 2024-02-28T16:31:25.549 INFO:tasks.workunit.client.0.smithi001.stdout:2/848: unlink d0/d2/d3/de/d8a/ccb 0 2024-02-28T16:31:25.554 INFO:tasks.workunit.client.0.smithi001.stdout:0/917: truncate d18/d22/d2b/d36/d46/f9d 340539 0 2024-02-28T16:31:25.554 INFO:tasks.workunit.client.1.smithi066.stdout:3/665: creat db/d13/d14/dad/fef x:0 0 0 2024-02-28T16:31:25.555 INFO:tasks.workunit.client.0.smithi001.stdout:6/931: dwrite d9/db/d5f/f65 [4194304,4194304] 0 2024-02-28T16:31:25.556 INFO:tasks.workunit.client.1.smithi066.stdout:4/694: sync 2024-02-28T16:31:25.556 INFO:tasks.workunit.client.1.smithi066.stdout:0/699: sync 2024-02-28T16:31:25.556 INFO:tasks.workunit.client.0.smithi001.stdout:7/959: getdents d3/d17/d52/d7f/d99 0 2024-02-28T16:31:25.557 INFO:tasks.workunit.client.1.smithi066.stdout:3/666: chown db/d13/d1a/d25/dd1 15 1 2024-02-28T16:31:25.558 INFO:tasks.workunit.client.1.smithi066.stdout:5/567: link d3/d4/dc/dd/d41/d5d/f7c d3/d4/dc/dd/d41/d7a/dac/fb4 0 2024-02-28T16:31:25.558 INFO:tasks.workunit.client.0.smithi001.stdout:2/849: mkdir d0/d2/d3/de/d22/dbb/dc8/d117/d119 0 2024-02-28T16:31:25.560 INFO:tasks.workunit.client.1.smithi066.stdout:9/472: mkdir da/df/d70/d90/d98 0 2024-02-28T16:31:25.560 INFO:tasks.workunit.client.1.smithi066.stdout:0/700: stat d2/d27/d41/dc5/cde 0 2024-02-28T16:31:25.561 INFO:tasks.workunit.client.0.smithi001.stdout:2/850: chown d0/d2/d3/de/d8a/l13 75912 1 2024-02-28T16:31:25.563 INFO:tasks.workunit.client.1.smithi066.stdout:4/695: dread - d1/d18/d55/dc4/fd3 zero size 2024-02-28T16:31:25.563 INFO:tasks.workunit.client.0.smithi001.stdout:0/918: mkdir d18/d22/d2f/d1c/d90/d9f/d128 0 2024-02-28T16:31:25.564 INFO:tasks.workunit.client.1.smithi066.stdout:6/541: creat d2/d26/fc3 x:0 0 0 2024-02-28T16:31:25.565 INFO:tasks.workunit.client.0.smithi001.stdout:2/851: symlink d0/d2/da6/l11a 0 2024-02-28T16:31:25.565 INFO:tasks.workunit.client.1.smithi066.stdout:8/549: creat d0/d6/dc/d16/d4c/d42/fb2 x:0 0 0 2024-02-28T16:31:25.568 INFO:tasks.workunit.client.0.smithi001.stdout:7/960: mkdir d3/d17/d52/d7f/d99/dc1/db6/d134 0 2024-02-28T16:31:25.568 INFO:tasks.workunit.client.0.smithi001.stdout:6/932: creat d9/d29/f13d x:0 0 0 2024-02-28T16:31:25.570 INFO:tasks.workunit.client.1.smithi066.stdout:9/473: write da/d22/d46/d56/d8f/d28/d42/f80 [4418173,113454] 0 2024-02-28T16:31:25.570 INFO:tasks.workunit.client.1.smithi066.stdout:6/542: dread - d2/da/d19/d75/d88/fa0 zero size 2024-02-28T16:31:25.571 INFO:tasks.workunit.client.1.smithi066.stdout:8/550: read - d0/d6/dc/d16/d1a/f8e zero size 2024-02-28T16:31:25.571 INFO:tasks.workunit.client.1.smithi066.stdout:4/696: rmdir d1/d23/d25 39 2024-02-28T16:31:25.572 INFO:tasks.workunit.client.0.smithi001.stdout:2/852: creat d0/d2/d3/de/d22/dbb/f11b x:0 0 0 2024-02-28T16:31:25.572 INFO:tasks.workunit.client.0.smithi001.stdout:7/961: readlink d3/d17/d52/d7f/d99/dc1/d49/l58 0 2024-02-28T16:31:25.573 INFO:tasks.workunit.client.1.smithi066.stdout:3/667: creat db/d2a/d44/d48/d52/dcb/ff0 x:0 0 0 2024-02-28T16:31:25.575 INFO:tasks.workunit.client.0.smithi001.stdout:0/919: mkdir d18/d22/d2f/d1c/d79/d129 0 2024-02-28T16:31:25.577 INFO:tasks.workunit.client.1.smithi066.stdout:3/668: chown db/d13/d1a/d25/c6e 237328 1 2024-02-28T16:31:25.577 INFO:tasks.workunit.client.1.smithi066.stdout:0/701: write d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f72 [78613,33071] 0 2024-02-28T16:31:25.578 INFO:tasks.workunit.client.1.smithi066.stdout:1/553: dread d2/d18/d19/f1e [0,4194304] 0 2024-02-28T16:31:25.578 INFO:tasks.workunit.client.0.smithi001.stdout:6/933: rename d9/de/dbd/lf0 to d9/db/d5f/d62/def/l13e 0 2024-02-28T16:31:25.580 INFO:tasks.workunit.client.0.smithi001.stdout:0/920: chown d18/d22/d2b/d36/d4d/fbd 879937324 1 2024-02-28T16:31:25.583 INFO:tasks.workunit.client.1.smithi066.stdout:6/543: symlink d2/da/d19/d75/lc4 0 2024-02-28T16:31:25.584 INFO:tasks.workunit.client.0.smithi001.stdout:7/962: creat d3/d17/d52/d7f/d99/dc1/d49/d8d/db9/d12f/f135 x:0 0 0 2024-02-28T16:31:25.584 INFO:tasks.workunit.client.0.smithi001.stdout:0/921: creat d18/da9/dea/f12a x:0 0 0 2024-02-28T16:31:25.584 INFO:tasks.workunit.client.1.smithi066.stdout:5/568: rename d3/d4/dc/dd/d41/d7a/c9e to d3/d4/dc/dd/d41/d7a/dac/cb5 0 2024-02-28T16:31:25.585 INFO:tasks.workunit.client.1.smithi066.stdout:9/474: mknod da/d22/d46/d56/c99 0 2024-02-28T16:31:25.585 INFO:tasks.workunit.client.1.smithi066.stdout:8/551: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db3 0 2024-02-28T16:31:25.586 INFO:tasks.workunit.client.1.smithi066.stdout:0/702: mknod d2/d27/ce5 0 2024-02-28T16:31:25.586 INFO:tasks.workunit.client.0.smithi001.stdout:7/963: symlink d3/db7/l136 0 2024-02-28T16:31:25.587 INFO:tasks.workunit.client.1.smithi066.stdout:9/475: readlink da/d22/d46/d56/l7d 0 2024-02-28T16:31:25.588 INFO:tasks.workunit.client.1.smithi066.stdout:3/669: mknod db/d13/d14/dad/cf1 0 2024-02-28T16:31:25.588 INFO:tasks.workunit.client.0.smithi001.stdout:7/964: rmdir d3/d17/d52/d55 39 2024-02-28T16:31:25.589 INFO:tasks.workunit.client.1.smithi066.stdout:1/554: creat d2/d27/d2b/d43/fab x:0 0 0 2024-02-28T16:31:25.590 INFO:tasks.workunit.client.1.smithi066.stdout:8/552: fdatasync d0/d6/dc/d16/d1a/f8e 0 2024-02-28T16:31:25.593 INFO:tasks.workunit.client.1.smithi066.stdout:8/553: readlink d0/d6/dc/d16/d1a/d63/l2a 0 2024-02-28T16:31:25.593 INFO:tasks.workunit.client.0.smithi001.stdout:7/965: creat d3/d17/d52/f137 x:0 0 0 2024-02-28T16:31:25.597 INFO:tasks.workunit.client.1.smithi066.stdout:5/569: write d3/d4/dc/dd/d35/f62 [3198969,104677] 0 2024-02-28T16:31:25.602 INFO:tasks.workunit.client.1.smithi066.stdout:9/476: write da/d3a/f41 [1985047,110694] 0 2024-02-28T16:31:25.602 INFO:tasks.workunit.client.1.smithi066.stdout:8/554: write d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/fa5 [925717,15786] 0 2024-02-28T16:31:25.604 INFO:tasks.workunit.client.1.smithi066.stdout:1/555: creat d2/d27/d2b/d34/fac x:0 0 0 2024-02-28T16:31:25.607 INFO:tasks.workunit.client.0.smithi001.stdout:0/922: truncate d18/f6d 5535234 0 2024-02-28T16:31:25.607 INFO:tasks.workunit.client.0.smithi001.stdout:2/853: rename d0/d2/d3/de/d61/d12/f41 to d0/d2/d3/d36/d94/f11c 0 2024-02-28T16:31:25.608 INFO:tasks.workunit.client.1.smithi066.stdout:6/544: dwrite d2/d26/fa3 [0,4194304] 0 2024-02-28T16:31:25.608 INFO:tasks.workunit.client.0.smithi001.stdout:6/934: dwrite d9/db/d5f/d62/d83/fdf [0,4194304] 0 2024-02-28T16:31:25.612 INFO:tasks.workunit.client.0.smithi001.stdout:0/923: mknod d18/d22/d76/d7a/c12b 0 2024-02-28T16:31:25.612 INFO:tasks.workunit.client.0.smithi001.stdout:2/854: symlink d0/d2/l11d 0 2024-02-28T16:31:25.617 INFO:tasks.workunit.client.0.smithi001.stdout:8/912: sync 2024-02-28T16:31:25.617 INFO:tasks.workunit.client.0.smithi001.stdout:3/957: sync 2024-02-28T16:31:25.618 INFO:tasks.workunit.client.0.smithi001.stdout:0/924: chown d18/d22/d2b/d36/d48/f8a 16 1 2024-02-28T16:31:25.620 INFO:tasks.workunit.client.0.smithi001.stdout:6/935: read d9/fbc [1268974,76381] 0 2024-02-28T16:31:25.621 INFO:tasks.workunit.client.1.smithi066.stdout:7/534: write dc/de/d17/d1e/d45/f8f [819372,102560] 0 2024-02-28T16:31:25.621 INFO:tasks.workunit.client.0.smithi001.stdout:7/966: dwrite d3/f21 [4194304,4194304] 0 2024-02-28T16:31:25.622 INFO:tasks.workunit.client.1.smithi066.stdout:1/556: write d2/f75 [3497066,61103] 0 2024-02-28T16:31:25.624 INFO:tasks.workunit.client.0.smithi001.stdout:8/913: read - d4/d47/d8b/ff3 zero size 2024-02-28T16:31:25.630 INFO:tasks.workunit.client.1.smithi066.stdout:3/670: dwrite db/d13/d1a/d25/f5f [4194304,4194304] 0 2024-02-28T16:31:25.631 INFO:tasks.workunit.client.0.smithi001.stdout:0/925: unlink d18/d22/d3d/c11f 0 2024-02-28T16:31:25.631 INFO:tasks.workunit.client.0.smithi001.stdout:3/958: truncate d4/d97/fbd 78232 0 2024-02-28T16:31:25.635 INFO:tasks.workunit.client.0.smithi001.stdout:5/895: sync 2024-02-28T16:31:25.636 INFO:tasks.workunit.client.1.smithi066.stdout:1/557: read d2/d27/d2b/d37/da6/f7e [2334572,102990] 0 2024-02-28T16:31:25.636 INFO:tasks.workunit.client.0.smithi001.stdout:7/967: creat d3/d17/d3d/d5d/f138 x:0 0 0 2024-02-28T16:31:25.637 INFO:tasks.workunit.client.0.smithi001.stdout:8/914: fdatasync d4/d32/d69/f9d 0 2024-02-28T16:31:25.642 INFO:tasks.workunit.client.0.smithi001.stdout:3/959: dread d4/d6/d7/d78/fc5 [0,4194304] 0 2024-02-28T16:31:25.642 INFO:tasks.workunit.client.0.smithi001.stdout:5/896: rmdir dc 39 2024-02-28T16:31:25.642 INFO:tasks.workunit.client.1.smithi066.stdout:8/555: dwrite d0/d6/dc/d16/d4c/d6c/f8b [0,4194304] 0 2024-02-28T16:31:25.642 INFO:tasks.workunit.client.1.smithi066.stdout:4/697: dwrite d1/d23/d25/f6f [0,4194304] 0 2024-02-28T16:31:25.644 INFO:tasks.workunit.client.0.smithi001.stdout:3/960: chown d4/d6/d7/d78/d81/dd6/le4 2983 1 2024-02-28T16:31:25.645 INFO:tasks.workunit.client.0.smithi001.stdout:7/968: rmdir d3/d17/d52/d7f 39 2024-02-28T16:31:25.646 INFO:tasks.workunit.client.1.smithi066.stdout:1/558: dread d2/d27/d2b/d37/d54/f8a [0,4194304] 0 2024-02-28T16:31:25.646 INFO:tasks.workunit.client.0.smithi001.stdout:2/855: truncate d0/d2/d3/de/d61/f67 444973 0 2024-02-28T16:31:25.650 INFO:tasks.workunit.client.1.smithi066.stdout:4/698: truncate d1/d18/d41/dba/fe0 256320 0 2024-02-28T16:31:25.654 INFO:tasks.workunit.client.0.smithi001.stdout:7/969: creat d3/d17/d3d/d42/f139 x:0 0 0 2024-02-28T16:31:25.656 INFO:tasks.workunit.client.1.smithi066.stdout:0/703: creat d2/d27/d41/d76/d93/db2/fe6 x:0 0 0 2024-02-28T16:31:25.657 INFO:tasks.workunit.client.1.smithi066.stdout:3/671: dwrite db/d13/d1a/d25/fcf [0,4194304] 0 2024-02-28T16:31:25.657 INFO:tasks.workunit.client.0.smithi001.stdout:5/897: mknod dc/d12/d56/d96/c11f 0 2024-02-28T16:31:25.657 INFO:tasks.workunit.client.0.smithi001.stdout:8/915: symlink d4/d7/d22/d27/d58/l124 0 2024-02-28T16:31:25.659 INFO:tasks.workunit.client.0.smithi001.stdout:7/970: symlink d3/d17/d3d/d59/d96/dc9/l13a 0 2024-02-28T16:31:25.660 INFO:tasks.workunit.client.1.smithi066.stdout:9/477: rename da/d22/f8b to da/d22/d46/d56/f9a 0 2024-02-28T16:31:25.660 INFO:tasks.workunit.client.0.smithi001.stdout:5/898: chown dc/d12/fa6 1 1 2024-02-28T16:31:25.662 INFO:tasks.workunit.client.0.smithi001.stdout:2/856: truncate d0/d2/d3/de/d61/d12/d46/f57 528515 0 2024-02-28T16:31:25.662 INFO:tasks.workunit.client.1.smithi066.stdout:6/545: chown d2/d2c/d72/c7e 171569 1 2024-02-28T16:31:25.664 INFO:tasks.workunit.client.1.smithi066.stdout:9/478: stat da/d22/d46/d56/d8f/d28/d42/d68/f7a 0 2024-02-28T16:31:25.669 INFO:tasks.workunit.client.0.smithi001.stdout:5/899: mknod dc/df/d79/c120 0 2024-02-28T16:31:25.671 INFO:tasks.workunit.client.0.smithi001.stdout:2/857: creat d0/d2/d3/de/d61/d12/d55/db2/f11e x:0 0 0 2024-02-28T16:31:25.675 INFO:tasks.workunit.client.1.smithi066.stdout:4/699: symlink d1/d23/d25/d3e/d40/le5 0 2024-02-28T16:31:25.676 INFO:tasks.workunit.client.0.smithi001.stdout:5/900: truncate dc/d12/daa/ff1 748723 0 2024-02-28T16:31:25.676 INFO:tasks.workunit.client.1.smithi066.stdout:6/546: fdatasync d2/da/f13 0 2024-02-28T16:31:25.682 INFO:tasks.workunit.client.1.smithi066.stdout:9/479: fsync da/f30 0 2024-02-28T16:31:25.685 INFO:tasks.workunit.client.1.smithi066.stdout:8/556: creat d0/d6/dc/d16/d1a/d63/d58/dac/fb4 x:0 0 0 2024-02-28T16:31:25.687 INFO:tasks.workunit.client.1.smithi066.stdout:1/559: mknod d2/d18/cad 0 2024-02-28T16:31:25.687 INFO:tasks.workunit.client.1.smithi066.stdout:9/480: chown da/d22/d46/d56/d8f/d2d/f57 3586 1 2024-02-28T16:31:25.689 INFO:tasks.workunit.client.1.smithi066.stdout:3/672: dwrite db/dc/db3/fcd [0,4194304] 0 2024-02-28T16:31:25.690 INFO:tasks.workunit.client.0.smithi001.stdout:7/971: dwrite d3/d17/d3d/db3/fb4 [0,4194304] 0 2024-02-28T16:31:25.690 INFO:tasks.workunit.client.1.smithi066.stdout:4/700: mknod d1/d23/d25/d3e/d40/ce6 0 2024-02-28T16:31:25.692 INFO:tasks.workunit.client.1.smithi066.stdout:5/570: dread d3/d4/fb [0,4194304] 0 2024-02-28T16:31:25.697 INFO:tasks.workunit.client.1.smithi066.stdout:2/598: sync 2024-02-28T16:31:25.700 INFO:tasks.workunit.client.0.smithi001.stdout:7/972: creat d3/d17/d52/d7f/d99/f13b x:0 0 0 2024-02-28T16:31:25.704 INFO:tasks.workunit.client.0.smithi001.stdout:2/858: dwrite d0/f17 [0,4194304] 0 2024-02-28T16:31:25.705 INFO:tasks.workunit.client.1.smithi066.stdout:3/673: symlink db/d2a/d44/d48/d52/lf2 0 2024-02-28T16:31:25.705 INFO:tasks.workunit.client.0.smithi001.stdout:6/936: dwrite d9/fbc [4194304,4194304] 0 2024-02-28T16:31:25.708 INFO:tasks.workunit.client.1.smithi066.stdout:3/674: chown db/d2a/d44/fc6 107 1 2024-02-28T16:31:25.711 INFO:tasks.workunit.client.0.smithi001.stdout:7/973: getdents d3/d17/d3d/d59/d104/d118 0 2024-02-28T16:31:25.714 INFO:tasks.workunit.client.1.smithi066.stdout:7/535: link dc/d29/c48 dc/de/d17/d1e/d45/c9a 0 2024-02-28T16:31:25.716 INFO:tasks.workunit.client.0.smithi001.stdout:2/859: stat d0/d2/dd9/f115 0 2024-02-28T16:31:25.718 INFO:tasks.workunit.client.1.smithi066.stdout:2/599: unlink d0/d12/d2e/d3b/c58 0 2024-02-28T16:31:25.720 INFO:tasks.workunit.client.0.smithi001.stdout:6/937: creat d9/d39/f13f x:0 0 0 2024-02-28T16:31:25.722 INFO:tasks.workunit.client.1.smithi066.stdout:2/600: fdatasync d0/d12/d1a/d39/d7c/d87/f94 0 2024-02-28T16:31:25.724 INFO:tasks.workunit.client.0.smithi001.stdout:0/926: dwrite f13 [0,4194304] 0 2024-02-28T16:31:25.724 INFO:tasks.workunit.client.0.smithi001.stdout:2/860: readlink d0/d2/d3/de/d61/d12/d46/l8d 0 2024-02-28T16:31:25.731 INFO:tasks.workunit.client.0.smithi001.stdout:6/938: symlink d9/d29/d6a/l140 0 2024-02-28T16:31:25.731 INFO:tasks.workunit.client.1.smithi066.stdout:0/704: link d2/d27/d41/d76/d93/d10/d23/d6d/cac d2/d27/d41/d76/d93/db2/db5/d67/d94/de4/da1/ce7 0 2024-02-28T16:31:25.734 INFO:tasks.workunit.client.0.smithi001.stdout:0/927: fsync d18/d22/d2f/d1c/d90/d9f/fe3 0 2024-02-28T16:31:25.735 INFO:tasks.workunit.client.0.smithi001.stdout:6/939: dread - d9/db/d5f/d62/d127/f136 zero size 2024-02-28T16:31:25.736 INFO:tasks.workunit.client.1.smithi066.stdout:9/481: creat da/d3a/f9b x:0 0 0 2024-02-28T16:31:25.738 INFO:tasks.workunit.client.0.smithi001.stdout:3/961: dread d4/d6/d7/f43 [0,4194304] 0 2024-02-28T16:31:25.738 INFO:tasks.workunit.client.1.smithi066.stdout:6/547: getdents d2/d2c/d72 0 2024-02-28T16:31:25.741 INFO:tasks.workunit.client.1.smithi066.stdout:4/701: link d1/d23/d25/d3e/d40/d9c/cd4 d1/d23/d25/d43/ce7 0 2024-02-28T16:31:25.743 INFO:tasks.workunit.client.1.smithi066.stdout:9/482: fdatasync da/d22/d46/d56/d8f/d28/d63/f5c 0 2024-02-28T16:31:25.744 INFO:tasks.workunit.client.0.smithi001.stdout:0/928: dread - d18/d22/ff9 zero size 2024-02-28T16:31:25.745 INFO:tasks.workunit.client.0.smithi001.stdout:8/916: dread d4/d7/d22/d27/d4b/d80/f85 [0,4194304] 0 2024-02-28T16:31:25.745 INFO:tasks.workunit.client.1.smithi066.stdout:2/601: dwrite d0/d12/d3e/d9e/fa3 [0,4194304] 0 2024-02-28T16:31:25.746 INFO:tasks.workunit.client.0.smithi001.stdout:6/940: rename d9/db/d10/f7b to d9/d29/d6e/d4e/f141 0 2024-02-28T16:31:25.747 INFO:tasks.workunit.client.1.smithi066.stdout:5/571: link d3/d4/dc/dd/d41/d7f/fa9 d3/d1b/d39/fb6 0 2024-02-28T16:31:25.749 INFO:tasks.workunit.client.1.smithi066.stdout:7/536: readlink dc/d56/l90 0 2024-02-28T16:31:25.749 INFO:tasks.workunit.client.0.smithi001.stdout:6/941: write d9/de/f5a [368033,20767] 0 2024-02-28T16:31:25.749 INFO:tasks.workunit.client.0.smithi001.stdout:0/929: write d18/d22/d3d/fd6 [1798415,18298] 0 2024-02-28T16:31:25.752 INFO:tasks.workunit.client.1.smithi066.stdout:0/705: creat d2/d27/d41/d76/d93/db2/db5/d73/fe8 x:0 0 0 2024-02-28T16:31:25.754 INFO:tasks.workunit.client.1.smithi066.stdout:6/548: symlink d2/da/d19/d9c/d3d/lc5 0 2024-02-28T16:31:25.756 INFO:tasks.workunit.client.0.smithi001.stdout:8/917: creat d4/d7/d22/d27/d4b/f125 x:0 0 0 2024-02-28T16:31:25.757 INFO:tasks.workunit.client.0.smithi001.stdout:0/930: chown d18/d22/d2b/d36/f56 3795064 1 2024-02-28T16:31:25.759 INFO:tasks.workunit.client.1.smithi066.stdout:3/675: rename db/dc/d35/d3d/f41 to db/d13/d14/ff3 0 2024-02-28T16:31:25.760 INFO:tasks.workunit.client.0.smithi001.stdout:0/931: stat d18/d22/d2b/d36/d46/l6a 0 2024-02-28T16:31:25.760 INFO:tasks.workunit.client.1.smithi066.stdout:9/483: creat da/d3a/d15/f9c x:0 0 0 2024-02-28T16:31:25.762 INFO:tasks.workunit.client.0.smithi001.stdout:6/942: creat d9/d29/d6a/ded/d105/f142 x:0 0 0 2024-02-28T16:31:25.764 INFO:tasks.workunit.client.1.smithi066.stdout:5/572: unlink d3/d4/dc/c1f 0 2024-02-28T16:31:25.765 INFO:tasks.workunit.client.1.smithi066.stdout:2/602: mkdir d0/d12/d1a/d39/d7c/d87/dc4 0 2024-02-28T16:31:25.767 INFO:tasks.workunit.client.0.smithi001.stdout:8/918: rename d4/d47/l76 to d4/d7/d22/d27/d58/d94/dbc/l126 0 2024-02-28T16:31:25.767 INFO:tasks.workunit.client.1.smithi066.stdout:6/549: mkdir d2/da/d19/d75/dbd/dc6 0 2024-02-28T16:31:25.768 INFO:tasks.workunit.client.0.smithi001.stdout:2/861: dread d0/d2/d3/de/d8a/fc9 [0,4194304] 0 2024-02-28T16:31:25.769 INFO:tasks.workunit.client.1.smithi066.stdout:5/573: truncate d3/d1b/d39/fa6 250440 0 2024-02-28T16:31:25.770 INFO:tasks.workunit.client.1.smithi066.stdout:4/702: rename d1/d7a/f90 to d1/d18/d4e/fe8 0 2024-02-28T16:31:25.773 INFO:tasks.workunit.client.1.smithi066.stdout:0/706: truncate d2/d27/d41/d76/d93/d10/d23/f7a 167059 0 2024-02-28T16:31:25.774 INFO:tasks.workunit.client.0.smithi001.stdout:8/919: fsync d4/d7/da8/d89/fce 0 2024-02-28T16:31:25.775 INFO:tasks.workunit.client.1.smithi066.stdout:2/603: creat d0/d12/d2e/d7a/fc5 x:0 0 0 2024-02-28T16:31:25.778 INFO:tasks.workunit.client.1.smithi066.stdout:0/707: fdatasync d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f72 0 2024-02-28T16:31:25.778 INFO:tasks.workunit.client.1.smithi066.stdout:7/537: rename dc/de/d17/d37/d47/d88/f93 to dc/de/d17/d91/f9b 0 2024-02-28T16:31:25.779 INFO:tasks.workunit.client.1.smithi066.stdout:7/538: fsync dc/de/f3f 0 2024-02-28T16:31:25.782 INFO:tasks.workunit.client.0.smithi001.stdout:7/974: dread d3/d17/d3d/d59/d6e/fac [0,4194304] 0 2024-02-28T16:31:25.782 INFO:tasks.workunit.client.0.smithi001.stdout:2/862: link d0/d2/d3/d36/d94/l9a d0/d2/d3/d36/l11f 0 2024-02-28T16:31:25.784 INFO:tasks.workunit.client.1.smithi066.stdout:5/574: symlink d3/d4/dc/dd/d41/d7a/dac/d60/lb7 0 2024-02-28T16:31:25.786 INFO:tasks.workunit.client.1.smithi066.stdout:4/703: rename d1/d23/d25/d77/f8d to d1/d23/d25/d77/fe9 0 2024-02-28T16:31:25.789 INFO:tasks.workunit.client.1.smithi066.stdout:1/560: dwrite d2/d27/d2b/d37/d48/f66 [0,4194304] 0 2024-02-28T16:31:25.791 INFO:tasks.workunit.client.0.smithi001.stdout:7/975: mknod d3/d17/d52/d7f/d99/dc1/d49/d8d/db9/c13c 0 2024-02-28T16:31:25.795 INFO:tasks.workunit.client.0.smithi001.stdout:8/920: dread d4/d7/f35 [0,4194304] 0 2024-02-28T16:31:25.796 INFO:tasks.workunit.client.0.smithi001.stdout:5/901: dwrite f7 [0,4194304] 0 2024-02-28T16:31:25.797 INFO:tasks.workunit.client.1.smithi066.stdout:3/676: rename db/dc/cc4 to db/d2a/d44/d48/d52/d77/cf4 0 2024-02-28T16:31:25.797 INFO:tasks.workunit.client.1.smithi066.stdout:8/557: write d0/d6/f1f [2378031,46802] 0 2024-02-28T16:31:25.799 INFO:tasks.workunit.client.1.smithi066.stdout:6/550: mkdir d2/da/d19/d75/dbd/dc6/dc7 0 2024-02-28T16:31:25.801 INFO:tasks.workunit.client.0.smithi001.stdout:2/863: rename d0/f83 to d0/d2/d3/de/d22/dbb/dc8/d117/d119/f120 0 2024-02-28T16:31:25.802 INFO:tasks.workunit.client.1.smithi066.stdout:6/551: stat d2/d26/d32/d46/f74 0 2024-02-28T16:31:25.803 INFO:tasks.workunit.client.1.smithi066.stdout:5/575: mknod d3/d4/dc/dd/cb8 0 2024-02-28T16:31:25.804 INFO:tasks.workunit.client.1.smithi066.stdout:2/604: unlink d0/d12/d1a/c24 0 2024-02-28T16:31:25.805 INFO:tasks.workunit.client.1.smithi066.stdout:4/704: mknod d1/d23/d25/d43/cea 0 2024-02-28T16:31:25.809 INFO:tasks.workunit.client.1.smithi066.stdout:1/561: creat d2/d27/d2b/d34/d9e/fae x:0 0 0 2024-02-28T16:31:25.809 INFO:tasks.workunit.client.0.smithi001.stdout:3/962: write d4/d1e/db7/fd0 [951086,3894] 0 2024-02-28T16:31:25.810 INFO:tasks.workunit.client.1.smithi066.stdout:2/605: truncate d0/d12/d1a/d23/d29/d21/d2f/d67/f97 72486 0 2024-02-28T16:31:25.813 INFO:tasks.workunit.client.1.smithi066.stdout:3/677: symlink db/dc/d35/d3d/d76/lf5 0 2024-02-28T16:31:25.813 INFO:tasks.workunit.client.0.smithi001.stdout:8/921: creat d4/d7/d22/d27/d4b/d80/f127 x:0 0 0 2024-02-28T16:31:25.814 INFO:tasks.workunit.client.1.smithi066.stdout:8/558: rename d0/db/f7d to d0/db/d12/d1d/d25/fb5 0 2024-02-28T16:31:25.817 INFO:tasks.workunit.client.0.smithi001.stdout:5/902: rename dc/d12/f14 to dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/f121 0 2024-02-28T16:31:25.819 INFO:tasks.workunit.client.1.smithi066.stdout:6/552: creat d2/da/d19/d75/dbd/dc6/fc8 x:0 0 0 2024-02-28T16:31:25.820 INFO:tasks.workunit.client.1.smithi066.stdout:0/708: dread d2/d27/d41/d76/d93/db2/db5/d52/f9a [0,4194304] 0 2024-02-28T16:31:25.822 INFO:tasks.workunit.client.1.smithi066.stdout:4/705: readlink d1/d18/d41/lc9 0 2024-02-28T16:31:25.822 INFO:tasks.workunit.client.0.smithi001.stdout:3/963: chown d4/d1e/d7c/da8/c119 0 1 2024-02-28T16:31:25.823 INFO:tasks.workunit.client.1.smithi066.stdout:2/606: unlink d0/d12/d2e/d3b/d66/c76 0 2024-02-28T16:31:25.827 INFO:tasks.workunit.client.1.smithi066.stdout:4/706: readlink d1/d18/d41/l6d 0 2024-02-28T16:31:25.829 INFO:tasks.workunit.client.1.smithi066.stdout:1/562: creat d2/d27/d2b/d43/faf x:0 0 0 2024-02-28T16:31:25.830 INFO:tasks.workunit.client.1.smithi066.stdout:3/678: symlink db/d13/d14/d2d/lf6 0 2024-02-28T16:31:25.831 INFO:tasks.workunit.client.1.smithi066.stdout:4/707: stat d1/d18/d41/fbf 0 2024-02-28T16:31:25.832 INFO:tasks.workunit.client.0.smithi001.stdout:5/903: mknod dc/d12/d41/c122 0 2024-02-28T16:31:25.833 INFO:tasks.workunit.client.0.smithi001.stdout:2/864: getdents d0/d2/d3/d73 0 2024-02-28T16:31:25.834 INFO:tasks.workunit.client.0.smithi001.stdout:5/904: chown dc/df/d79/d98 4 1 2024-02-28T16:31:25.834 INFO:tasks.workunit.client.1.smithi066.stdout:4/708: chown d1/f9b 30053 1 2024-02-28T16:31:25.835 INFO:tasks.workunit.client.0.smithi001.stdout:6/943: dwrite d9/db/d5f/dd4/dfb/d121/f33 [0,4194304] 0 2024-02-28T16:31:25.835 INFO:tasks.workunit.client.1.smithi066.stdout:8/559: chown d0/d6/dc/l94 5 1 2024-02-28T16:31:25.837 INFO:tasks.workunit.client.1.smithi066.stdout:0/709: truncate d2/d27/d41/d76/d93/db2/db5/d67/fe1 120267 0 2024-02-28T16:31:25.838 INFO:tasks.workunit.client.0.smithi001.stdout:6/944: chown d9/de/lfa 588925 1 2024-02-28T16:31:25.838 INFO:tasks.workunit.client.0.smithi001.stdout:2/865: mknod d0/d2/d3/de/d10f/c121 0 2024-02-28T16:31:25.839 INFO:tasks.workunit.client.1.smithi066.stdout:9/484: dwrite da/fb [0,4194304] 0 2024-02-28T16:31:25.844 INFO:tasks.workunit.client.1.smithi066.stdout:3/679: creat db/d2a/ff7 x:0 0 0 2024-02-28T16:31:25.849 INFO:tasks.workunit.client.1.smithi066.stdout:3/680: stat db/d13/d1a/d25/c79 0 2024-02-28T16:31:25.849 INFO:tasks.workunit.client.0.smithi001.stdout:5/905: rename dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/cef to dc/d12/d41/db7/c123 0 2024-02-28T16:31:25.849 INFO:tasks.workunit.client.0.smithi001.stdout:8/922: write d4/d47/fa0 [2355535,91779] 0 2024-02-28T16:31:25.852 INFO:tasks.workunit.client.1.smithi066.stdout:6/553: dwrite d2/d2c/d6e/f9d [0,4194304] 0 2024-02-28T16:31:25.853 INFO:tasks.workunit.client.1.smithi066.stdout:7/539: truncate dc/de/d17/d37/f72 4086772 0 2024-02-28T16:31:25.853 INFO:tasks.workunit.client.0.smithi001.stdout:5/906: chown dc/df/d79/f115 1258 1 2024-02-28T16:31:25.854 INFO:tasks.workunit.client.1.smithi066.stdout:2/607: rmdir d0/d12/d1a/d39/d7c/d87/daf 0 2024-02-28T16:31:25.855 INFO:tasks.workunit.client.1.smithi066.stdout:6/554: stat d2/da/f1a 0 2024-02-28T16:31:25.857 INFO:tasks.workunit.client.1.smithi066.stdout:9/485: creat da/d22/d46/d56/d8f/d28/d42/f9d x:0 0 0 2024-02-28T16:31:25.859 INFO:tasks.workunit.client.1.smithi066.stdout:3/681: chown db/d2a/d44/d48/d5d/fbe 2764 1 2024-02-28T16:31:25.859 INFO:tasks.workunit.client.0.smithi001.stdout:7/976: dwrite d3/d17/f10e [0,4194304] 0 2024-02-28T16:31:25.862 INFO:tasks.workunit.client.1.smithi066.stdout:7/540: read dc/de/d17/d1e/f98 [3813610,7682] 0 2024-02-28T16:31:25.864 INFO:tasks.workunit.client.0.smithi001.stdout:5/907: rmdir dc/df/d79 39 2024-02-28T16:31:25.865 INFO:tasks.workunit.client.0.smithi001.stdout:8/923: write d4/d32/dd6/fd2 [703539,13767] 0 2024-02-28T16:31:25.866 INFO:tasks.workunit.client.0.smithi001.stdout:7/977: mkdir d3/dc5/d13d 0 2024-02-28T16:31:25.867 INFO:tasks.workunit.client.1.smithi066.stdout:2/608: creat d0/d12/d1a/d39/d7c/d87/fc6 x:0 0 0 2024-02-28T16:31:25.868 INFO:tasks.workunit.client.1.smithi066.stdout:5/576: write d3/d4/f4e [1848545,39316] 0 2024-02-28T16:31:25.868 INFO:tasks.workunit.client.1.smithi066.stdout:3/682: readlink db/dc/d35/d3d/d76/d89/dcc/lc5 0 2024-02-28T16:31:25.870 INFO:tasks.workunit.client.1.smithi066.stdout:7/541: chown dc/de/d17/d37/c96 1747747069 1 2024-02-28T16:31:25.870 INFO:tasks.workunit.client.0.smithi001.stdout:3/964: dwrite d4/d6/d7/f63 [4194304,4194304] 0 2024-02-28T16:31:25.873 INFO:tasks.workunit.client.1.smithi066.stdout:1/563: write d2/f7 [3387160,29546] 0 2024-02-28T16:31:25.874 INFO:tasks.workunit.client.1.smithi066.stdout:4/709: write d1/d23/d25/d43/f65 [2714869,20759] 0 2024-02-28T16:31:25.875 INFO:tasks.workunit.client.1.smithi066.stdout:5/577: fsync d3/d4/dc/dd/d35/d92/fab 0 2024-02-28T16:31:25.877 INFO:tasks.workunit.client.0.smithi001.stdout:5/908: creat dc/d1a/d7d/d87/df3/d112/f124 x:0 0 0 2024-02-28T16:31:25.878 INFO:tasks.workunit.client.0.smithi001.stdout:7/978: unlink d3/d17/d3d/d5d/fc7 0 2024-02-28T16:31:25.878 INFO:tasks.workunit.client.1.smithi066.stdout:7/542: mknod dc/de/d17/d1e/d45/c9c 0 2024-02-28T16:31:25.880 INFO:tasks.workunit.client.1.smithi066.stdout:9/486: link da/d22/d46/d56/d8f/d28/d42/f9d da/d3a/f9e 0 2024-02-28T16:31:25.881 INFO:tasks.workunit.client.1.smithi066.stdout:3/683: read db/d13/f19 [1479511,91374] 0 2024-02-28T16:31:25.881 INFO:tasks.workunit.client.0.smithi001.stdout:5/909: truncate dc/d12/d41/f9c 6524777 0 2024-02-28T16:31:25.882 INFO:tasks.workunit.client.1.smithi066.stdout:4/710: chown d1/d18/d4e/l6c 117015567 1 2024-02-28T16:31:25.885 INFO:tasks.workunit.client.0.smithi001.stdout:2/866: write d0/d2/d3/f6c [2197560,120322] 0 2024-02-28T16:31:25.888 INFO:tasks.workunit.client.0.smithi001.stdout:3/965: truncate d4/d18/f11 2505115 0 2024-02-28T16:31:25.888 INFO:tasks.workunit.client.1.smithi066.stdout:5/578: creat d3/d1b/d7d/d9c/fb9 x:0 0 0 2024-02-28T16:31:25.888 INFO:tasks.workunit.client.0.smithi001.stdout:7/979: creat d3/d17/df0/f13e x:0 0 0 2024-02-28T16:31:25.889 INFO:tasks.workunit.client.1.smithi066.stdout:7/543: unlink dc/de/d17/d37/c96 0 2024-02-28T16:31:25.892 INFO:tasks.workunit.client.0.smithi001.stdout:5/910: symlink dc/d12/d56/d96/dd0/l125 0 2024-02-28T16:31:25.893 INFO:tasks.workunit.client.0.smithi001.stdout:2/867: creat d0/d2/d3/de/d22/dbb/f122 x:0 0 0 2024-02-28T16:31:25.893 INFO:tasks.workunit.client.1.smithi066.stdout:8/560: dwrite d0/db/d15/f44 [0,4194304] 0 2024-02-28T16:31:25.894 INFO:tasks.workunit.client.1.smithi066.stdout:3/684: write db/ff [678058,91598] 0 2024-02-28T16:31:25.896 INFO:tasks.workunit.client.0.smithi001.stdout:3/966: mknod d4/d6/d7/d23/c139 0 2024-02-28T16:31:25.900 INFO:tasks.workunit.client.0.smithi001.stdout:7/980: creat d3/d17/d3d/d59/d104/f13f x:0 0 0 2024-02-28T16:31:25.902 INFO:tasks.workunit.client.0.smithi001.stdout:6/945: dwrite d9/db/dd9/f11f [0,4194304] 0 2024-02-28T16:31:25.903 INFO:tasks.workunit.client.1.smithi066.stdout:4/711: creat d1/d18/d55/dc4/feb x:0 0 0 2024-02-28T16:31:25.904 INFO:tasks.workunit.client.1.smithi066.stdout:9/487: mknod da/d4b/d53/c9f 0 2024-02-28T16:31:25.904 INFO:tasks.workunit.client.0.smithi001.stdout:8/924: dwrite d4/d7/d22/d27/d4b/d80/f127 [0,4194304] 0 2024-02-28T16:31:25.909 INFO:tasks.workunit.client.0.smithi001.stdout:2/868: symlink d0/d2/d3/d73/l123 0 2024-02-28T16:31:25.910 INFO:tasks.workunit.client.0.smithi001.stdout:3/967: unlink d4/d6/d7/f88 0 2024-02-28T16:31:25.910 INFO:tasks.workunit.client.1.smithi066.stdout:7/544: mkdir dc/de/d17/d9d 0 2024-02-28T16:31:25.914 INFO:tasks.workunit.client.1.smithi066.stdout:3/685: rmdir db/d13 39 2024-02-28T16:31:25.914 INFO:tasks.workunit.client.0.smithi001.stdout:3/968: write d4/d1e/d7c/d7d/df8/f116 [107936,28790] 0 2024-02-28T16:31:25.915 INFO:tasks.workunit.client.1.smithi066.stdout:2/609: dwrite d0/d12/d1a/d23/d29/d21/d2f/f5f [0,4194304] 0 2024-02-28T16:31:25.918 INFO:tasks.workunit.client.0.smithi001.stdout:5/911: getdents dc/d12/d56 0 2024-02-28T16:31:25.922 INFO:tasks.workunit.client.1.smithi066.stdout:1/564: link d2/d27/d2b/l57 d2/d27/d2b/d37/d51/lb0 0 2024-02-28T16:31:25.923 INFO:tasks.workunit.client.0.smithi001.stdout:2/869: write d0/d2/d3/de/d61/dd5/f109 [561257,119986] 0 2024-02-28T16:31:25.923 INFO:tasks.workunit.client.1.smithi066.stdout:8/561: creat d0/d6/dc/d16/d4c/d42/da8/fb6 x:0 0 0 2024-02-28T16:31:25.924 INFO:tasks.workunit.client.0.smithi001.stdout:0/932: sync 2024-02-28T16:31:25.925 INFO:tasks.workunit.client.1.smithi066.stdout:3/686: rename db/d2a/c40 to db/dc/d11/d1f/d78/cf8 0 2024-02-28T16:31:25.926 INFO:tasks.workunit.client.1.smithi066.stdout:4/712: creat d1/d18/d41/fec x:0 0 0 2024-02-28T16:31:25.931 INFO:tasks.workunit.client.1.smithi066.stdout:1/565: mkdir d2/d18/d7a/db1 0 2024-02-28T16:31:25.932 INFO:tasks.workunit.client.0.smithi001.stdout:8/925: rename d4/d32/d93/ce1 to d4/d32/d69/d9a/dcd/c128 0 2024-02-28T16:31:25.933 INFO:tasks.workunit.client.1.smithi066.stdout:6/555: dwrite d2/da/f13 [0,4194304] 0 2024-02-28T16:31:25.933 INFO:tasks.workunit.client.0.smithi001.stdout:8/926: readlink d4/lb1 0 2024-02-28T16:31:25.936 INFO:tasks.workunit.client.0.smithi001.stdout:6/946: dwrite d9/db/d19/f1f [0,4194304] 0 2024-02-28T16:31:25.937 INFO:tasks.workunit.client.1.smithi066.stdout:5/579: dwrite d3/d4/dc/d2f/d55/f70 [4194304,4194304] 0 2024-02-28T16:31:25.942 INFO:tasks.workunit.client.0.smithi001.stdout:5/912: mknod dc/d1a/d7d/d87/df3/c126 0 2024-02-28T16:31:25.944 INFO:tasks.workunit.client.0.smithi001.stdout:5/913: fsync dc/d1a/d52/f89 0 2024-02-28T16:31:25.945 INFO:tasks.workunit.client.0.smithi001.stdout:7/981: mknod d3/d17/d52/d55/dca/d106/c140 0 2024-02-28T16:31:25.948 INFO:tasks.workunit.client.1.smithi066.stdout:7/545: dwrite dc/d29/f41 [0,4194304] 0 2024-02-28T16:31:25.951 INFO:tasks.workunit.client.1.smithi066.stdout:8/562: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/fb7 x:0 0 0 2024-02-28T16:31:25.953 INFO:tasks.workunit.client.0.smithi001.stdout:2/870: read d0/d2/d3/de/d61/d12/fe0 [2794781,126179] 0 2024-02-28T16:31:25.953 INFO:tasks.workunit.client.0.smithi001.stdout:3/969: rename d4/d1e/d7c/d7d/df8/f101 to d4/d1e/dcf/f13a 0 2024-02-28T16:31:25.956 INFO:tasks.workunit.client.0.smithi001.stdout:2/871: dread - d0/d2/da6/ff3 zero size 2024-02-28T16:31:25.960 INFO:tasks.workunit.client.1.smithi066.stdout:3/687: creat db/d2a/dd4/ff9 x:0 0 0 2024-02-28T16:31:25.960 INFO:tasks.workunit.client.1.smithi066.stdout:1/566: mkdir d2/d18/da5/db2 0 2024-02-28T16:31:25.961 INFO:tasks.workunit.client.1.smithi066.stdout:7/546: mknod dc/de/d17/d37/c9e 0 2024-02-28T16:31:25.962 INFO:tasks.workunit.client.0.smithi001.stdout:5/914: dread - dc/d12/d56/d96/db8/fbd zero size 2024-02-28T16:31:25.964 INFO:tasks.workunit.client.1.smithi066.stdout:4/713: getdents d1/d18/d4e/da0 0 2024-02-28T16:31:25.965 INFO:tasks.workunit.client.0.smithi001.stdout:8/927: rename d4/d7/d22/df6/f100 to d4/d9b/f129 0 2024-02-28T16:31:25.972 INFO:tasks.workunit.client.1.smithi066.stdout:3/688: chown db/dc/d11/daa/fd3 630326 1 2024-02-28T16:31:25.973 INFO:tasks.workunit.client.0.smithi001.stdout:2/872: creat d0/d2/d3/de/d22/dbb/f124 x:0 0 0 2024-02-28T16:31:25.976 INFO:tasks.workunit.client.0.smithi001.stdout:5/915: fdatasync dc/d1a/f8d 0 2024-02-28T16:31:25.976 INFO:tasks.workunit.client.0.smithi001.stdout:6/947: rename d9/d29/d6e/d53/l128 to d9/db/d5f/d62/d83/l143 0 2024-02-28T16:31:25.978 INFO:tasks.workunit.client.1.smithi066.stdout:7/547: mkdir dc/de/d17/d1e/d45/d9f 0 2024-02-28T16:31:25.979 INFO:tasks.workunit.client.1.smithi066.stdout:9/488: dwrite da/d22/d46/d56/d8f/d2d/f57 [0,4194304] 0 2024-02-28T16:31:25.983 INFO:tasks.workunit.client.1.smithi066.stdout:6/556: dwrite d2/da/d19/d75/f92 [8388608,4194304] 0 2024-02-28T16:31:25.983 INFO:tasks.workunit.client.1.smithi066.stdout:1/567: symlink d2/d27/d2b/lb3 0 2024-02-28T16:31:25.985 INFO:tasks.workunit.client.1.smithi066.stdout:3/689: stat db/d13/d14/dad/fc7 0 2024-02-28T16:31:25.987 INFO:tasks.workunit.client.0.smithi001.stdout:3/970: truncate d4/d6/f56 1203715 0 2024-02-28T16:31:25.990 INFO:tasks.workunit.client.1.smithi066.stdout:1/568: chown d2/d27/d2b/d43/d46/d80/l93 14 1 2024-02-28T16:31:25.990 INFO:tasks.workunit.client.1.smithi066.stdout:2/610: dwrite d0/d12/d1a/d23/d29/f6e [0,4194304] 0 2024-02-28T16:31:25.990 INFO:tasks.workunit.client.0.smithi001.stdout:8/928: mkdir d4/d7/d22/d27/d4b/d12a 0 2024-02-28T16:31:25.993 INFO:tasks.workunit.client.1.smithi066.stdout:4/714: truncate d1/d18/d41/f47 2999090 0 2024-02-28T16:31:25.995 INFO:tasks.workunit.client.0.smithi001.stdout:3/971: creat d4/d18/ddb/f13b x:0 0 0 2024-02-28T16:31:25.996 INFO:tasks.workunit.client.0.smithi001.stdout:5/916: fdatasync dc/d12/d56/d96/db9/dc2/dca/d3c/f46 0 2024-02-28T16:31:25.998 INFO:tasks.workunit.client.1.smithi066.stdout:7/548: symlink dc/de/d17/la0 0 2024-02-28T16:31:25.998 INFO:tasks.workunit.client.0.smithi001.stdout:5/917: readlink dc/d1a/d52/l114 0 2024-02-28T16:31:25.999 INFO:tasks.workunit.client.1.smithi066.stdout:3/690: creat db/dc/d35/d3d/d76/ffa x:0 0 0 2024-02-28T16:31:26.000 INFO:tasks.workunit.client.0.smithi001.stdout:8/929: creat d4/d32/dd6/f12b x:0 0 0 2024-02-28T16:31:26.001 INFO:tasks.workunit.client.0.smithi001.stdout:3/972: chown d4/d1e/def/f3e 925 1 2024-02-28T16:31:26.002 INFO:tasks.workunit.client.1.smithi066.stdout:8/563: link d0/d6/dc/d16/l30 d0/d6/dc/d16/d1a/d63/lb8 0 2024-02-28T16:31:26.008 INFO:tasks.workunit.client.0.smithi001.stdout:7/982: dread d3/ff [8388608,4194304] 0 2024-02-28T16:31:26.010 INFO:tasks.workunit.client.1.smithi066.stdout:1/569: mkdir d2/d27/d2b/d37/d51/db4 0 2024-02-28T16:31:26.010 INFO:tasks.workunit.client.1.smithi066.stdout:9/489: fsync da/d22/d46/d56/f9a 0 2024-02-28T16:31:26.014 INFO:tasks.workunit.client.0.smithi001.stdout:0/933: write d18/d22/d2b/d36/d46/df5/feb [584473,126046] 0 2024-02-28T16:31:26.014 INFO:tasks.workunit.client.1.smithi066.stdout:7/549: readlink dc/l64 0 2024-02-28T16:31:26.016 INFO:tasks.workunit.client.1.smithi066.stdout:9/490: fsync da/d22/d46/d56/d8f/f82 0 2024-02-28T16:31:26.019 INFO:tasks.workunit.client.0.smithi001.stdout:2/873: rename d0/d2/d3/de/d61/d12/d55/fcd to d0/d2/d3/de/f125 0 2024-02-28T16:31:26.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:25 smithi001 ceph-mon[28991]: pgmap v56: 65 pgs: 65 active+clean; 2.9 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 265 MiB/s wr, 518 op/s 2024-02-28T16:31:26.025 INFO:tasks.workunit.client.0.smithi001.stdout:5/918: fsync dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/f59 0 2024-02-28T16:31:26.026 INFO:tasks.workunit.client.1.smithi066.stdout:8/564: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/dad/db9 0 2024-02-28T16:31:26.027 INFO:tasks.workunit.client.1.smithi066.stdout:7/550: mkdir dc/de/d17/d1e/da1 0 2024-02-28T16:31:26.032 INFO:tasks.workunit.client.1.smithi066.stdout:8/565: creat d0/db/d15/fba x:0 0 0 2024-02-28T16:31:26.037 INFO:tasks.workunit.client.1.smithi066.stdout:2/611: dread d0/d12/d3e/f63 [0,4194304] 0 2024-02-28T16:31:26.039 INFO:tasks.workunit.client.0.smithi001.stdout:8/930: read d4/d7/d22/d27/d58/d94/dc9/fc1 [336459,37489] 0 2024-02-28T16:31:26.041 INFO:tasks.workunit.client.1.smithi066.stdout:8/566: chown d0/d6/dc/d16/d4c/d42/d73 21080988 1 2024-02-28T16:31:26.042 INFO:tasks.workunit.client.0.smithi001.stdout:0/934: mknod d18/d22/d105/c12c 0 2024-02-28T16:31:26.045 INFO:tasks.workunit.client.1.smithi066.stdout:3/691: truncate db/d13/d1a/d25/f5f 1205828 0 2024-02-28T16:31:26.047 INFO:tasks.workunit.client.0.smithi001.stdout:2/874: truncate d0/f91 793058 0 2024-02-28T16:31:26.047 INFO:tasks.workunit.client.0.smithi001.stdout:6/948: dwrite d9/d29/d6e/f31 [0,4194304] 0 2024-02-28T16:31:26.048 INFO:tasks.workunit.client.1.smithi066.stdout:1/570: dwrite d2/d27/d2b/d37/d54/f92 [0,4194304] 0 2024-02-28T16:31:26.051 INFO:tasks.workunit.client.1.smithi066.stdout:9/491: dwrite da/f20 [0,4194304] 0 2024-02-28T16:31:26.052 INFO:tasks.workunit.client.0.smithi001.stdout:2/875: readlink d0/d2/d3/de/d8a/l31 0 2024-02-28T16:31:26.056 INFO:tasks.workunit.client.1.smithi066.stdout:8/567: mkdir d0/db/d12/d1d/d25/d65/dbb 0 2024-02-28T16:31:26.056 INFO:tasks.workunit.client.1.smithi066.stdout:1/571: stat d2/da0/la7 0 2024-02-28T16:31:26.059 INFO:tasks.workunit.client.0.smithi001.stdout:5/919: mknod dc/d12/c127 0 2024-02-28T16:31:26.065 INFO:tasks.workunit.client.1.smithi066.stdout:6/557: dwrite d2/d2c/da1/d5d/d99/f49 [0,4194304] 0 2024-02-28T16:31:26.071 INFO:tasks.workunit.client.0.smithi001.stdout:7/983: rename d3/d17/d52/d7f/d99/dc1/d49/l58 to d3/l141 0 2024-02-28T16:31:26.073 INFO:tasks.workunit.client.0.smithi001.stdout:2/876: read - d0/d2/d3/de/d22/dbb/dc8/fef zero size 2024-02-28T16:31:26.075 INFO:tasks.workunit.client.1.smithi066.stdout:4/715: dwrite d1/d23/d25/d3e/f66 [0,4194304] 0 2024-02-28T16:31:26.077 INFO:tasks.workunit.client.1.smithi066.stdout:6/558: read d2/d26/d32/d46/f74 [1795871,36770] 0 2024-02-28T16:31:26.077 INFO:tasks.workunit.client.1.smithi066.stdout:9/492: creat da/d22/d46/d56/d8f/d94/fa0 x:0 0 0 2024-02-28T16:31:26.077 INFO:tasks.workunit.client.0.smithi001.stdout:5/920: creat dc/d1a/d21/de2/f128 x:0 0 0 2024-02-28T16:31:26.079 INFO:tasks.workunit.client.0.smithi001.stdout:8/931: symlink d4/d7/d22/d27/l12c 0 2024-02-28T16:31:26.080 INFO:tasks.workunit.client.1.smithi066.stdout:2/612: dwrite d0/d12/d1a/d23/d29/f6e [0,4194304] 0 2024-02-28T16:31:26.081 INFO:tasks.workunit.client.0.smithi001.stdout:3/973: dwrite d4/f76 [0,4194304] 0 2024-02-28T16:31:26.084 INFO:tasks.workunit.client.1.smithi066.stdout:6/559: chown d2/d2c/la7 22591572 1 2024-02-28T16:31:26.084 INFO:tasks.workunit.client.1.smithi066.stdout:9/493: fsync da/d22/d46/d56/d8f/d28/f59 0 2024-02-28T16:31:26.085 INFO:tasks.workunit.client.1.smithi066.stdout:3/692: link db/d13/d14/dad/fe1 db/dc/d11/d1f/d78/ffb 0 2024-02-28T16:31:26.088 INFO:tasks.workunit.client.1.smithi066.stdout:3/693: dread - db/d2a/ff7 zero size 2024-02-28T16:31:26.089 INFO:tasks.workunit.client.1.smithi066.stdout:1/572: link d2/d27/d2b/d43/d46/d80/f87 d2/d27/d2b/d43/d46/fb5 0 2024-02-28T16:31:26.089 INFO:tasks.workunit.client.1.smithi066.stdout:4/716: rmdir d1/d18/d4e/da0 39 2024-02-28T16:31:26.094 INFO:tasks.workunit.client.0.smithi001.stdout:8/932: mknod d4/d7/d22/d27/d58/d94/db4/c12d 0 2024-02-28T16:31:26.095 INFO:tasks.workunit.client.1.smithi066.stdout:7/551: dread dc/de/d17/d1e/f24 [0,4194304] 0 2024-02-28T16:31:26.098 INFO:tasks.workunit.client.1.smithi066.stdout:9/494: rmdir da/d3a/d15 39 2024-02-28T16:31:26.102 INFO:tasks.workunit.client.0.smithi001.stdout:7/984: dwrite d3/d1c/f51 [0,4194304] 0 2024-02-28T16:31:26.102 INFO:tasks.workunit.client.0.smithi001.stdout:8/933: mknod d4/d47/d8b/d108/c12e 0 2024-02-28T16:31:26.103 INFO:tasks.workunit.client.1.smithi066.stdout:2/613: mknod d0/d12/d1a/d39/d68/cc7 0 2024-02-28T16:31:26.103 INFO:tasks.workunit.client.0.smithi001.stdout:6/949: dread d9/db/fd8 [0,4194304] 0 2024-02-28T16:31:26.107 INFO:tasks.workunit.client.1.smithi066.stdout:4/717: stat d1/f12 0 2024-02-28T16:31:26.109 INFO:tasks.workunit.client.0.smithi001.stdout:6/950: chown d9/db/fd8 2333 1 2024-02-28T16:31:26.110 INFO:tasks.workunit.client.1.smithi066.stdout:4/718: readlink d1/l31 0 2024-02-28T16:31:26.110 INFO:tasks.workunit.client.1.smithi066.stdout:6/560: dwrite d2/da/d19/f41 [0,4194304] 0 2024-02-28T16:31:26.113 INFO:tasks.workunit.client.0.smithi001.stdout:7/985: symlink d3/dc5/l142 0 2024-02-28T16:31:26.119 INFO:tasks.workunit.client.0.smithi001.stdout:6/951: readlink d9/d29/d6e/d9a/l10e 0 2024-02-28T16:31:26.119 INFO:tasks.workunit.client.1.smithi066.stdout:7/552: creat dc/de/fa2 x:0 0 0 2024-02-28T16:31:26.120 INFO:tasks.workunit.client.1.smithi066.stdout:1/573: creat d2/d27/d2b/d43/d46/fb6 x:0 0 0 2024-02-28T16:31:26.121 INFO:tasks.workunit.client.1.smithi066.stdout:3/694: creat db/d13/d1a/d33/ffc x:0 0 0 2024-02-28T16:31:26.125 INFO:tasks.workunit.client.0.smithi001.stdout:5/921: dread dc/d12/d56/d96/db9/dc2/dca/f66 [0,4194304] 0 2024-02-28T16:31:26.125 INFO:tasks.workunit.client.0.smithi001.stdout:0/935: truncate d18/f59 3074690 0 2024-02-28T16:31:26.127 INFO:tasks.workunit.client.1.smithi066.stdout:9/495: rename da/d22/d46/d56/d8f/d28/d42/c4a to da/d3a/d58/d97/ca1 0 2024-02-28T16:31:26.127 INFO:tasks.workunit.client.0.smithi001.stdout:7/986: symlink d3/d17/d52/d7f/d99/dc1/db6/d134/l143 0 2024-02-28T16:31:26.128 INFO:tasks.workunit.client.1.smithi066.stdout:6/561: mknod d2/da/d19/cc9 0 2024-02-28T16:31:26.129 INFO:tasks.workunit.client.0.smithi001.stdout:2/877: write d0/d2/d3/de/d61/d12/fe0 [2723378,109499] 0 2024-02-28T16:31:26.130 INFO:tasks.workunit.client.0.smithi001.stdout:0/936: chown d18/d22/f3f 4 1 2024-02-28T16:31:26.130 INFO:tasks.workunit.client.1.smithi066.stdout:7/553: write dc/d29/f2e [2783357,120430] 0 2024-02-28T16:31:26.131 INFO:tasks.workunit.client.1.smithi066.stdout:1/574: symlink d2/d18/d7a/db1/lb7 0 2024-02-28T16:31:26.133 INFO:tasks.workunit.client.1.smithi066.stdout:6/562: creat d2/da/d19/d75/dbd/dc6/fca x:0 0 0 2024-02-28T16:31:26.134 INFO:tasks.workunit.client.0.smithi001.stdout:7/987: write d3/d17/d3d/d59/d96/dc9/f112 [85932,130746] 0 2024-02-28T16:31:26.134 INFO:tasks.workunit.client.0.smithi001.stdout:8/934: getdents d4/d7/da8 0 2024-02-28T16:31:26.135 INFO:tasks.workunit.client.1.smithi066.stdout:7/554: write dc/de/d17/d1e/f53 [4964250,76673] 0 2024-02-28T16:31:26.136 INFO:tasks.workunit.client.1.smithi066.stdout:2/614: dwrite d0/d12/d2e/d3b/d66/f7f [0,4194304] 0 2024-02-28T16:31:26.138 INFO:tasks.workunit.client.1.smithi066.stdout:6/563: fdatasync d2/d26/f2d 0 2024-02-28T16:31:26.143 INFO:tasks.workunit.client.1.smithi066.stdout:2/615: getdents d0/d12/d1a/d39/d7c 0 2024-02-28T16:31:26.144 INFO:tasks.workunit.client.0.smithi001.stdout:0/937: mknod d18/da9/dc9/c12d 0 2024-02-28T16:31:26.146 INFO:tasks.workunit.client.0.smithi001.stdout:3/974: dread d4/d97/fdc [0,4194304] 0 2024-02-28T16:31:26.149 INFO:tasks.workunit.client.0.smithi001.stdout:7/988: creat d3/d17/d52/d7f/d99/f144 x:0 0 0 2024-02-28T16:31:26.150 INFO:tasks.workunit.client.0.smithi001.stdout:8/935: truncate d4/d7/d22/d27/d58/d94/faa 30073 0 2024-02-28T16:31:26.150 INFO:tasks.workunit.client.0.smithi001.stdout:5/922: getdents dc/d12/d41/d54/d11a 0 2024-02-28T16:31:26.154 INFO:tasks.workunit.client.0.smithi001.stdout:7/989: write d3/d17/f122 [1011037,79556] 0 2024-02-28T16:31:26.157 INFO:tasks.workunit.client.0.smithi001.stdout:6/952: rename d9/db/d5f/d62/d7e/c12e to d9/db/d19/c144 0 2024-02-28T16:31:26.174 INFO:tasks.workunit.client.0.smithi001.stdout:7/990: mknod d3/d17/d3d/db3/c145 0 2024-02-28T16:31:26.178 INFO:tasks.workunit.client.1.smithi066.stdout:0/710: sync 2024-02-28T16:31:26.178 INFO:tasks.workunit.client.1.smithi066.stdout:7/555: rename dc/de/d17/d1e/f28 to dc/de/d17/d1e/fa3 0 2024-02-28T16:31:26.179 INFO:tasks.workunit.client.0.smithi001.stdout:2/878: rename d0/d2/d3/de/d61/d12/d6a to d0/d2/d126 0 2024-02-28T16:31:26.180 INFO:tasks.workunit.client.0.smithi001.stdout:2/879: readlink d0/d2/d3/d36/l108 0 2024-02-28T16:31:26.181 INFO:tasks.workunit.client.1.smithi066.stdout:4/719: write d1/f4d [17888,52599] 0 2024-02-28T16:31:26.181 INFO:tasks.workunit.client.0.smithi001.stdout:2/880: readlink d0/d2/d3/de/l65 0 2024-02-28T16:31:26.182 INFO:tasks.workunit.client.1.smithi066.stdout:2/616: creat d0/d12/d1a/d39/d7c/d87/dc4/fc8 x:0 0 0 2024-02-28T16:31:26.183 INFO:tasks.workunit.client.0.smithi001.stdout:2/881: chown d0/d2/d3/de 1715 1 2024-02-28T16:31:26.185 INFO:tasks.workunit.client.1.smithi066.stdout:7/556: write dc/de/d17/d1e/d45/f95 [237524,106281] 0 2024-02-28T16:31:26.187 INFO:tasks.workunit.client.0.smithi001.stdout:5/923: write dc/d1a/d7d/f88 [682168,56018] 0 2024-02-28T16:31:26.189 INFO:tasks.workunit.client.1.smithi066.stdout:4/720: truncate d1/d23/d25/d43/f8b 578648 0 2024-02-28T16:31:26.191 INFO:tasks.workunit.client.0.smithi001.stdout:2/882: mknod d0/d2/d3/d73/c127 0 2024-02-28T16:31:26.194 INFO:tasks.workunit.client.1.smithi066.stdout:7/557: mknod dc/d29/d92/ca4 0 2024-02-28T16:31:26.194 INFO:tasks.workunit.client.0.smithi001.stdout:6/953: write d9/d29/f87 [2237809,57261] 0 2024-02-28T16:31:26.196 INFO:tasks.workunit.client.1.smithi066.stdout:4/721: mkdir d1/d23/d25/d43/ded 0 2024-02-28T16:31:26.197 INFO:tasks.workunit.client.1.smithi066.stdout:2/617: link d0/d12/d2e/f84 d0/d12/d1a/d39/d68/fc9 0 2024-02-28T16:31:26.200 INFO:tasks.workunit.client.1.smithi066.stdout:7/558: write f1 [2841209,29106] 0 2024-02-28T16:31:26.201 INFO:tasks.workunit.client.0.smithi001.stdout:2/883: symlink d0/d2/d3/de/d61/l128 0 2024-02-28T16:31:26.201 INFO:tasks.workunit.client.0.smithi001.stdout:7/991: dread d3/d17/d3d/d59/fb1 [0,4194304] 0 2024-02-28T16:31:26.201 INFO:tasks.workunit.client.1.smithi066.stdout:4/722: chown d1/d18/d41/dba/fc6 108647501 1 2024-02-28T16:31:26.202 INFO:tasks.workunit.client.0.smithi001.stdout:5/924: dread dc/df/f13 [0,4194304] 0 2024-02-28T16:31:26.203 INFO:tasks.workunit.client.0.smithi001.stdout:8/936: dwrite d4/f13 [0,4194304] 0 2024-02-28T16:31:26.204 INFO:tasks.workunit.client.1.smithi066.stdout:6/564: dwrite d2/d26/d32/d46/f7c [0,4194304] 0 2024-02-28T16:31:26.204 INFO:tasks.workunit.client.0.smithi001.stdout:3/975: dwrite d4/d1e/d7c/d7d/df8/fff [0,4194304] 0 2024-02-28T16:31:26.206 INFO:tasks.workunit.client.1.smithi066.stdout:1/575: dread d2/f7 [0,4194304] 0 2024-02-28T16:31:26.207 INFO:tasks.workunit.client.1.smithi066.stdout:6/565: chown d2/d2c/d72/caf 465235736 1 2024-02-28T16:31:26.210 INFO:tasks.workunit.client.1.smithi066.stdout:2/618: mknod d0/d12/d1a/cca 0 2024-02-28T16:31:26.211 INFO:tasks.workunit.client.0.smithi001.stdout:6/954: fsync d9/db/d19/f90 0 2024-02-28T16:31:26.211 INFO:tasks.workunit.client.0.smithi001.stdout:5/925: creat dc/d1a/d7d/d87/df3/f129 x:0 0 0 2024-02-28T16:31:26.213 INFO:tasks.workunit.client.1.smithi066.stdout:7/559: symlink dc/de/d17/d37/la5 0 2024-02-28T16:31:26.218 INFO:tasks.workunit.client.1.smithi066.stdout:6/566: mkdir d2/da/dcb 0 2024-02-28T16:31:26.219 INFO:tasks.workunit.client.0.smithi001.stdout:7/992: creat d3/d17/d52/d7f/d99/dc1/db6/d134/f146 x:0 0 0 2024-02-28T16:31:26.220 INFO:tasks.workunit.client.0.smithi001.stdout:8/937: mknod d4/d7/c12f 0 2024-02-28T16:31:26.220 INFO:tasks.workunit.client.1.smithi066.stdout:6/567: fdatasync d2/da/d19/d9c/d3d/f89 0 2024-02-28T16:31:26.223 INFO:tasks.workunit.client.1.smithi066.stdout:7/560: mkdir dc/d29/d92/da6 0 2024-02-28T16:31:26.224 INFO:tasks.workunit.client.0.smithi001.stdout:2/884: truncate d0/d2/d3/de/d61/f23 425214 0 2024-02-28T16:31:26.224 INFO:tasks.workunit.client.1.smithi066.stdout:0/711: dwrite d2/d27/d41/d76/d93/db2/db5/d73/f90 [0,4194304] 0 2024-02-28T16:31:26.226 INFO:tasks.workunit.client.0.smithi001.stdout:0/938: dread d18/d22/d3d/f50 [0,4194304] 0 2024-02-28T16:31:26.227 INFO:tasks.workunit.client.1.smithi066.stdout:1/576: rename d2/d27/d2b/d43/d46/f88 to d2/d27/d2b/d43/d46/d80/fb8 0 2024-02-28T16:31:26.230 INFO:tasks.workunit.client.1.smithi066.stdout:2/619: readlink d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/l91 0 2024-02-28T16:31:26.230 INFO:tasks.workunit.client.1.smithi066.stdout:6/568: truncate d2/da/d19/d75/dbd/dc6/fca 728315 0 2024-02-28T16:31:26.231 INFO:tasks.workunit.client.0.smithi001.stdout:6/955: dread d9/db/d5f/d62/d83/fb6 [0,4194304] 0 2024-02-28T16:31:26.231 INFO:tasks.workunit.client.0.smithi001.stdout:5/926: getdents dc/d1a 0 2024-02-28T16:31:26.232 INFO:tasks.workunit.client.1.smithi066.stdout:0/712: chown d2/d27/d41/d76/d93/db2/db5/d51/f70 201 1 2024-02-28T16:31:26.233 INFO:tasks.workunit.client.0.smithi001.stdout:7/993: creat d3/d17/f147 x:0 0 0 2024-02-28T16:31:26.234 INFO:tasks.workunit.client.1.smithi066.stdout:1/577: getdents d2/d18/da5/db2 0 2024-02-28T16:31:26.234 INFO:tasks.workunit.client.1.smithi066.stdout:6/569: dread - d2/da/d19/d9c/f3f zero size 2024-02-28T16:31:26.237 INFO:tasks.workunit.client.0.smithi001.stdout:3/976: rename d4/d1e/d7c/fd3 to d4/d6/d7/d78/d81/da1/f13c 0 2024-02-28T16:31:26.237 INFO:tasks.workunit.client.0.smithi001.stdout:0/939: mknod d18/d22/d2b/d36/d48/c12e 0 2024-02-28T16:31:26.239 INFO:tasks.workunit.client.0.smithi001.stdout:6/956: creat d9/db/d5f/dd4/dfb/d121/f145 x:0 0 0 2024-02-28T16:31:26.240 INFO:tasks.workunit.client.1.smithi066.stdout:1/578: getdents d2/d27/d2b/d37/d51 0 2024-02-28T16:31:26.240 INFO:tasks.workunit.client.1.smithi066.stdout:2/620: creat d0/d12/d1a/d23/d29/d21/fcb x:0 0 0 2024-02-28T16:31:26.241 INFO:tasks.workunit.client.0.smithi001.stdout:8/938: link d4/d7/d22/d27/d4b/d80/f85 d4/d7/da8/f130 0 2024-02-28T16:31:26.242 INFO:tasks.workunit.client.1.smithi066.stdout:0/713: rename d2/d27/d41/d76/d93/db2/db5/d67/d94/de4/cd4 to d2/d27/d41/d76/d93/db2/db5/d73/ce9 0 2024-02-28T16:31:26.244 INFO:tasks.workunit.client.1.smithi066.stdout:1/579: mknod d2/d18/da5/cb9 0 2024-02-28T16:31:26.245 INFO:tasks.workunit.client.0.smithi001.stdout:7/994: symlink d3/d17/d52/d7f/l148 0 2024-02-28T16:31:26.246 INFO:tasks.workunit.client.1.smithi066.stdout:4/723: truncate d1/f4f 3785621 0 2024-02-28T16:31:26.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:25 smithi066 ceph-mon[36236]: pgmap v56: 65 pgs: 65 active+clean; 2.9 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 265 MiB/s wr, 518 op/s 2024-02-28T16:31:26.249 INFO:tasks.workunit.client.1.smithi066.stdout:7/561: truncate dc/de/d17/d1e/d45/f8f 1080133 0 2024-02-28T16:31:26.250 INFO:tasks.workunit.client.1.smithi066.stdout:1/580: mkdir d2/d18/d6b/dba 0 2024-02-28T16:31:26.250 INFO:tasks.workunit.client.1.smithi066.stdout:7/562: stat dc/de/f3f 0 2024-02-28T16:31:26.251 INFO:tasks.workunit.client.0.smithi001.stdout:2/885: rename d0/d2/d3/c3e to d0/d2/d3/de/d22/dbb/dc8/d117/c129 0 2024-02-28T16:31:26.253 INFO:tasks.workunit.client.0.smithi001.stdout:6/957: creat d9/de/f146 x:0 0 0 2024-02-28T16:31:26.255 INFO:tasks.workunit.client.0.smithi001.stdout:3/977: dwrite d4/d6/fb5 [0,4194304] 0 2024-02-28T16:31:26.255 INFO:tasks.workunit.client.0.smithi001.stdout:0/940: rmdir d18/d22/d2b/d36/d46/df5 39 2024-02-28T16:31:26.255 INFO:tasks.workunit.client.1.smithi066.stdout:4/724: read - d1/d18/d4e/da0/fc5 zero size 2024-02-28T16:31:26.256 INFO:tasks.workunit.client.1.smithi066.stdout:6/570: dwrite d2/d2c/f97 [0,4194304] 0 2024-02-28T16:31:26.262 INFO:tasks.workunit.client.0.smithi001.stdout:2/886: unlink d0/d2/d3/de/d22/fb9 0 2024-02-28T16:31:26.263 INFO:tasks.workunit.client.1.smithi066.stdout:2/621: link d0/d12/d1a/d23/d29/d21/d2f/cbf d0/d12/d1a/d39/d7c/ccc 0 2024-02-28T16:31:26.265 INFO:tasks.workunit.client.1.smithi066.stdout:2/622: chown d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/lb8 0 1 2024-02-28T16:31:26.266 INFO:tasks.workunit.client.0.smithi001.stdout:0/941: mkdir d18/da9/dc9/d12f 0 2024-02-28T16:31:26.271 INFO:tasks.workunit.client.1.smithi066.stdout:4/725: creat d1/d18/d7e/dbb/fee x:0 0 0 2024-02-28T16:31:26.272 INFO:tasks.workunit.client.0.smithi001.stdout:5/927: getdents dc/d12/daa 0 2024-02-28T16:31:26.275 INFO:tasks.workunit.client.1.smithi066.stdout:0/714: truncate d2/f5e 3553820 0 2024-02-28T16:31:26.276 INFO:tasks.workunit.client.0.smithi001.stdout:6/958: truncate d9/d29/d6e/d4e/fd0 3527288 0 2024-02-28T16:31:26.277 INFO:tasks.workunit.client.1.smithi066.stdout:2/623: creat d0/d12/d1a/d39/d7c/d87/dc4/fcd x:0 0 0 2024-02-28T16:31:26.279 INFO:tasks.workunit.client.0.smithi001.stdout:2/887: mknod d0/d2/d3/de/d22/dbb/dc8/c12a 0 2024-02-28T16:31:26.280 INFO:tasks.workunit.client.1.smithi066.stdout:4/726: symlink d1/d23/d25/d3e/d40/d9c/lef 0 2024-02-28T16:31:26.280 INFO:tasks.workunit.client.0.smithi001.stdout:0/942: chown d18/d22/d2f/c5b 5 1 2024-02-28T16:31:26.280 INFO:tasks.workunit.client.0.smithi001.stdout:5/928: rmdir dc/d12/d41 39 2024-02-28T16:31:26.282 INFO:tasks.workunit.client.0.smithi001.stdout:0/943: chown d18/da9/dc9/dd7 141374 1 2024-02-28T16:31:26.283 INFO:tasks.workunit.client.0.smithi001.stdout:7/995: creat d3/d17/d52/d7f/d99/dc1/f149 x:0 0 0 2024-02-28T16:31:26.283 INFO:tasks.workunit.client.0.smithi001.stdout:5/929: fsync dc/df/f3a 0 2024-02-28T16:31:26.284 INFO:tasks.workunit.client.0.smithi001.stdout:6/959: symlink d9/db/d10/l147 0 2024-02-28T16:31:26.285 INFO:tasks.workunit.client.0.smithi001.stdout:0/944: readlink d18/d22/d2b/d36/d4d/l91 0 2024-02-28T16:31:26.293 INFO:tasks.workunit.client.1.smithi066.stdout:0/715: rename d2/d27/d41/d76/d93/d10/d23/d6d/la7 to d2/d27/d41/d76/d93/db2/db5/d52/da2/lea 0 2024-02-28T16:31:26.298 INFO:tasks.workunit.client.0.smithi001.stdout:8/939: dread d4/d32/dd6/fe6 [0,4194304] 0 2024-02-28T16:31:26.301 INFO:tasks.workunit.client.1.smithi066.stdout:2/624: write d0/f1f [8218856,13231] 0 2024-02-28T16:31:26.301 INFO:tasks.workunit.client.1.smithi066.stdout:1/581: dwrite d2/d27/d2b/f3d [0,4194304] 0 2024-02-28T16:31:26.301 INFO:tasks.workunit.client.1.smithi066.stdout:8/568: sync 2024-02-28T16:31:26.301 INFO:tasks.workunit.client.1.smithi066.stdout:5/580: sync 2024-02-28T16:31:26.305 INFO:tasks.workunit.client.0.smithi001.stdout:0/945: creat d18/d22/d2b/d36/d46/df5/d4f/f130 x:0 0 0 2024-02-28T16:31:26.312 INFO:tasks.workunit.client.0.smithi001.stdout:7/996: dwrite d3/d17/f147 [0,4194304] 0 2024-02-28T16:31:26.312 INFO:tasks.workunit.client.0.smithi001.stdout:8/940: creat d4/d32/d69/d9a/f131 x:0 0 0 2024-02-28T16:31:26.315 INFO:tasks.workunit.client.0.smithi001.stdout:3/978: dwrite d4/d1e/d7c/d7d/df8/d82/fb0 [4194304,4194304] 0 2024-02-28T16:31:26.317 INFO:tasks.workunit.client.0.smithi001.stdout:2/888: getdents d0/d2/d126/d84 0 2024-02-28T16:31:26.318 INFO:tasks.workunit.client.0.smithi001.stdout:0/946: symlink d18/d22/d2f/d1c/d90/d9f/l131 0 2024-02-28T16:31:26.319 INFO:tasks.workunit.client.0.smithi001.stdout:3/979: chown d4/d18/d5f/f114 339013 1 2024-02-28T16:31:26.320 INFO:tasks.workunit.client.0.smithi001.stdout:5/930: link dc/d12/d41/f4c dc/d12/d41/d54/d11a/f12a 0 2024-02-28T16:31:26.321 INFO:tasks.workunit.client.0.smithi001.stdout:8/941: creat d4/d47/d8b/f132 x:0 0 0 2024-02-28T16:31:26.326 INFO:tasks.workunit.client.0.smithi001.stdout:0/947: truncate d18/d5a/f63 2245102 0 2024-02-28T16:31:26.334 INFO:tasks.workunit.client.0.smithi001.stdout:8/942: creat d4/d7/d22/d27/d58/d94/db4/f133 x:0 0 0 2024-02-28T16:31:26.336 INFO:tasks.workunit.client.0.smithi001.stdout:0/948: symlink d18/d22/d2b/d36/d46/df5/l132 0 2024-02-28T16:31:26.338 INFO:tasks.workunit.client.0.smithi001.stdout:3/980: dread d4/d1e/d7c/da8/fae [0,4194304] 0 2024-02-28T16:31:26.339 INFO:tasks.workunit.client.0.smithi001.stdout:6/960: dwrite d9/db/d5f/d8b/f110 [0,4194304] 0 2024-02-28T16:31:26.340 INFO:tasks.workunit.client.0.smithi001.stdout:8/943: creat d4/d47/df4/f134 x:0 0 0 2024-02-28T16:31:26.342 INFO:tasks.workunit.client.0.smithi001.stdout:8/944: fdatasync d4/d7/d22/d27/d4b/fa4 0 2024-02-28T16:31:26.347 INFO:tasks.workunit.client.1.smithi066.stdout:1/582: chown d2/f13 1 1 2024-02-28T16:31:26.349 INFO:tasks.workunit.client.0.smithi001.stdout:7/997: truncate d3/d17/f3b 722508 0 2024-02-28T16:31:26.352 INFO:tasks.workunit.client.1.smithi066.stdout:7/563: dwrite dc/de/d17/d1e/d45/f8f [0,4194304] 0 2024-02-28T16:31:26.356 INFO:tasks.workunit.client.0.smithi001.stdout:3/981: fdatasync d4/d1e/fc8 0 2024-02-28T16:31:26.356 INFO:tasks.workunit.client.1.smithi066.stdout:6/571: dwrite d2/d2c/da1/d5d/d99/f5e [0,4194304] 0 2024-02-28T16:31:26.356 INFO:tasks.workunit.client.1.smithi066.stdout:7/564: fsync dc/f4b 0 2024-02-28T16:31:26.361 INFO:tasks.workunit.client.1.smithi066.stdout:4/727: dwrite d1/d18/f87 [0,4194304] 0 2024-02-28T16:31:26.363 INFO:tasks.workunit.client.1.smithi066.stdout:5/581: read d3/d4/dc/dd/d2e/d18/f4a [829734,54121] 0 2024-02-28T16:31:26.368 INFO:tasks.workunit.client.1.smithi066.stdout:1/583: rename d2/d27/d2b/d34/l64 to d2/d18/d6b/dba/lbb 0 2024-02-28T16:31:26.369 INFO:tasks.workunit.client.1.smithi066.stdout:5/582: read d3/d4/dc/dd/d41/d5d/f7c [111277,61106] 0 2024-02-28T16:31:26.372 INFO:tasks.workunit.client.1.smithi066.stdout:1/584: fdatasync d2/d27/d2b/d34/d9e/fae 0 2024-02-28T16:31:26.373 INFO:tasks.workunit.client.1.smithi066.stdout:5/583: chown d3/d4/dc/dd/d41/d7a/dac/d57 97127 1 2024-02-28T16:31:26.374 INFO:tasks.workunit.client.1.smithi066.stdout:7/565: link dc/de/d17/f19 dc/d29/d92/da6/fa7 0 2024-02-28T16:31:26.377 INFO:tasks.workunit.client.0.smithi001.stdout:0/949: symlink d18/d22/d2b/d36/d48/d11a/l133 0 2024-02-28T16:31:26.377 INFO:tasks.workunit.client.1.smithi066.stdout:6/572: getdents d2/d2c/d6e 0 2024-02-28T16:31:26.377 INFO:tasks.workunit.client.1.smithi066.stdout:8/569: write d0/db/d12/d38/f54 [3976445,27068] 0 2024-02-28T16:31:26.378 INFO:tasks.workunit.client.1.smithi066.stdout:7/566: fdatasync dc/de/d87/f66 0 2024-02-28T16:31:26.379 INFO:tasks.workunit.client.0.smithi001.stdout:5/931: write dc/d1a/d7d/d87/df3/f10d [1585427,126147] 0 2024-02-28T16:31:26.381 INFO:tasks.workunit.client.1.smithi066.stdout:1/585: rmdir d2/d27/d2b 39 2024-02-28T16:31:26.383 INFO:tasks.workunit.client.1.smithi066.stdout:6/573: mkdir d2/da/dcc 0 2024-02-28T16:31:26.384 INFO:tasks.workunit.client.1.smithi066.stdout:5/584: rename d3/d4/dc/dd/d2e/l58 to d3/d4/dc/d2f/d55/d74/lba 0 2024-02-28T16:31:26.389 INFO:tasks.workunit.client.1.smithi066.stdout:1/586: fdatasync d2/d27/d2b/d43/f9d 0 2024-02-28T16:31:26.390 INFO:tasks.workunit.client.1.smithi066.stdout:7/567: symlink dc/de/d17/d1e/da1/la8 0 2024-02-28T16:31:26.391 INFO:tasks.workunit.client.0.smithi001.stdout:6/961: dread d9/d29/d6e/d4e/f131 [0,4194304] 0 2024-02-28T16:31:26.391 INFO:tasks.workunit.client.1.smithi066.stdout:8/570: mknod d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/dad/cbc 0 2024-02-28T16:31:26.393 INFO:tasks.workunit.client.1.smithi066.stdout:7/568: fsync dc/de/f3f 0 2024-02-28T16:31:26.393 INFO:tasks.workunit.client.1.smithi066.stdout:5/585: dread - d3/d4/dc/dd/d41/da2/f8d zero size 2024-02-28T16:31:26.393 INFO:tasks.workunit.client.1.smithi066.stdout:4/728: dwrite d1/d18/d41/f68 [0,4194304] 0 2024-02-28T16:31:26.397 INFO:tasks.workunit.client.1.smithi066.stdout:7/569: chown dc/de/d87/c76 294 1 2024-02-28T16:31:26.399 INFO:tasks.workunit.client.0.smithi001.stdout:2/889: dwrite d0/f91 [0,4194304] 0 2024-02-28T16:31:26.402 INFO:tasks.workunit.client.1.smithi066.stdout:4/729: creat d1/d18/d55/dc4/ff0 x:0 0 0 2024-02-28T16:31:26.402 INFO:tasks.workunit.client.1.smithi066.stdout:6/574: getdents d2/da/d19/d75/dbd 0 2024-02-28T16:31:26.404 INFO:tasks.workunit.client.1.smithi066.stdout:4/730: fdatasync d1/d23/d25/f6f 0 2024-02-28T16:31:26.404 INFO:tasks.workunit.client.1.smithi066.stdout:6/575: stat d2/da/d19/d9c/d3d 0 2024-02-28T16:31:26.406 INFO:tasks.workunit.client.1.smithi066.stdout:6/576: readlink d2/d26/d32/l87 0 2024-02-28T16:31:26.407 INFO:tasks.workunit.client.1.smithi066.stdout:1/587: mkdir d2/d18/d7a/dbc 0 2024-02-28T16:31:26.409 INFO:tasks.workunit.client.1.smithi066.stdout:4/731: write d1/d18/d7e/fab [3608371,3479] 0 2024-02-28T16:31:26.410 INFO:tasks.workunit.client.1.smithi066.stdout:5/586: creat d3/d4/dc/d2f/d55/d74/fbb x:0 0 0 2024-02-28T16:31:26.410 INFO:tasks.workunit.client.0.smithi001.stdout:0/950: dwrite d18/d22/d3d/fba [4194304,4194304] 0 2024-02-28T16:31:26.411 INFO:tasks.workunit.client.1.smithi066.stdout:7/570: creat dc/de/d87/fa9 x:0 0 0 2024-02-28T16:31:26.417 INFO:tasks.workunit.client.1.smithi066.stdout:5/587: chown d3/d4/dc/dd/d2e/d18/f4b 90 1 2024-02-28T16:31:26.418 INFO:tasks.workunit.client.1.smithi066.stdout:7/571: rmdir dc/de/d17/d37 39 2024-02-28T16:31:26.421 INFO:tasks.workunit.client.1.smithi066.stdout:7/572: fsync dc/f5d 0 2024-02-28T16:31:26.423 INFO:tasks.workunit.client.1.smithi066.stdout:3/695: sync 2024-02-28T16:31:26.423 INFO:tasks.workunit.client.1.smithi066.stdout:9/496: sync 2024-02-28T16:31:26.423 INFO:tasks.workunit.client.1.smithi066.stdout:2/625: dread d0/d12/d2e/d7a/f55 [0,4194304] 0 2024-02-28T16:31:26.425 INFO:tasks.workunit.client.0.smithi001.stdout:2/890: dwrite d0/f59 [0,4194304] 0 2024-02-28T16:31:26.425 INFO:tasks.workunit.client.1.smithi066.stdout:1/588: link d2/d27/d2b/lb3 d2/lbd 0 2024-02-28T16:31:26.426 INFO:tasks.workunit.client.1.smithi066.stdout:8/571: dwrite d0/d6/f9a [0,4194304] 0 2024-02-28T16:31:26.430 INFO:tasks.workunit.client.1.smithi066.stdout:3/696: chown db/d2a/d44/d48 1460718 1 2024-02-28T16:31:26.437 INFO:tasks.workunit.client.0.smithi001.stdout:7/998: truncate d3/d17/d3d/d42/ffb 442738 0 2024-02-28T16:31:26.438 INFO:tasks.workunit.client.1.smithi066.stdout:2/626: write d0/d12/d1a/d39/d7c/fbe [749508,49490] 0 2024-02-28T16:31:26.438 INFO:tasks.workunit.client.1.smithi066.stdout:1/589: readlink d2/lbd 0 2024-02-28T16:31:26.438 INFO:tasks.workunit.client.1.smithi066.stdout:7/573: dread dc/de/d17/d1e/d45/f95 [0,4194304] 0 2024-02-28T16:31:26.440 INFO:tasks.workunit.client.0.smithi001.stdout:3/982: rename d4/d18/d5f/fb8 to d4/d6/d7/d78/d81/dd6/f13d 0 2024-02-28T16:31:26.443 INFO:tasks.workunit.client.1.smithi066.stdout:8/572: dread d0/d6/dc/d16/d1a/d63/d58/d5a/f64 [0,4194304] 0 2024-02-28T16:31:26.445 INFO:tasks.workunit.client.1.smithi066.stdout:9/497: mknod da/d3a/d58/d97/ca2 0 2024-02-28T16:31:26.447 INFO:tasks.workunit.client.1.smithi066.stdout:6/577: write d2/d26/f47 [1243311,101673] 0 2024-02-28T16:31:26.449 INFO:tasks.workunit.client.1.smithi066.stdout:5/588: write d3/d4/dc/dd/d2e/f6f [1727251,6933] 0 2024-02-28T16:31:26.449 INFO:tasks.workunit.client.0.smithi001.stdout:5/932: mkdir dc/d1a/d21/d11b/d12b 0 2024-02-28T16:31:26.452 INFO:tasks.workunit.client.1.smithi066.stdout:2/627: mkdir d0/d12/d3e/d9e/dce 0 2024-02-28T16:31:26.453 INFO:tasks.workunit.client.1.smithi066.stdout:1/590: mkdir d2/d18/d19/dbe 0 2024-02-28T16:31:26.453 INFO:tasks.workunit.client.0.smithi001.stdout:7/999: rename d3/d17/d3d/d59/d6e/de2/f116 to d3/d17/df0/f14a 0 2024-02-28T16:31:26.454 INFO:tasks.workunit.client.1.smithi066.stdout:9/498: truncate da/d3a/f9b 966568 0 2024-02-28T16:31:26.455 INFO:tasks.workunit.client.1.smithi066.stdout:8/573: creat d0/d6/dc/d16/d4c/d42/fbd x:0 0 0 2024-02-28T16:31:26.456 INFO:tasks.workunit.client.0.smithi001.stdout:3/983: mknod d4/d1e/c13e 0 2024-02-28T16:31:26.457 INFO:tasks.workunit.client.1.smithi066.stdout:7/574: symlink dc/de/d17/laa 0 2024-02-28T16:31:26.457 INFO:tasks.workunit.client.1.smithi066.stdout:0/716: sync 2024-02-28T16:31:26.459 INFO:tasks.workunit.client.0.smithi001.stdout:5/933: unlink dc/d12/d41/d54/f118 0 2024-02-28T16:31:26.464 INFO:tasks.workunit.client.0.smithi001.stdout:6/962: link d9/db/d19/c144 d9/c148 0 2024-02-28T16:31:26.467 INFO:tasks.workunit.client.0.smithi001.stdout:0/951: dread d18/d22/d76/d93/f94 [0,4194304] 0 2024-02-28T16:31:26.467 INFO:tasks.workunit.client.1.smithi066.stdout:3/697: dwrite db/d2a/d44/d48/fd6 [0,4194304] 0 2024-02-28T16:31:26.468 INFO:tasks.workunit.client.0.smithi001.stdout:5/934: read dc/d12/d56/d96/db9/dc2/dca/d3c/f47 [2424179,126851] 0 2024-02-28T16:31:26.469 INFO:tasks.workunit.client.0.smithi001.stdout:6/963: read - d9/db/d5f/dd4/dfb/f102 zero size 2024-02-28T16:31:26.473 INFO:tasks.workunit.client.1.smithi066.stdout:1/591: mkdir d2/d27/d2b/d37/d51/db4/dbf 0 2024-02-28T16:31:26.474 INFO:tasks.workunit.client.1.smithi066.stdout:0/717: chown d2/d27/d41/d76/d93/db2/db5/d52/da2/lea 87 1 2024-02-28T16:31:26.474 INFO:tasks.workunit.client.1.smithi066.stdout:4/732: dread d1/d18/f2d [0,4194304] 0 2024-02-28T16:31:26.475 INFO:tasks.workunit.client.1.smithi066.stdout:2/628: write d0/d12/d1a/f3a [1449679,37426] 0 2024-02-28T16:31:26.480 INFO:tasks.workunit.client.0.smithi001.stdout:5/935: fsync dc/df/d79/f115 0 2024-02-28T16:31:26.480 INFO:tasks.workunit.client.1.smithi066.stdout:7/575: mknod dc/de/d17/d9d/cab 0 2024-02-28T16:31:26.481 INFO:tasks.workunit.client.1.smithi066.stdout:8/574: symlink d0/db/d15/d95/lbe 0 2024-02-28T16:31:26.481 INFO:tasks.workunit.client.0.smithi001.stdout:3/984: link d4/d1e/c6c d4/d6/d7/d78/d81/dd6/dd9/d11f/c13f 0 2024-02-28T16:31:26.483 INFO:tasks.workunit.client.0.smithi001.stdout:5/936: rmdir dc/d1a/d21 39 2024-02-28T16:31:26.484 INFO:tasks.workunit.client.1.smithi066.stdout:4/733: symlink d1/d23/d25/d43/lf1 0 2024-02-28T16:31:26.484 INFO:tasks.workunit.client.1.smithi066.stdout:5/589: getdents d3/d4/dc/dd/d41/d7a/dac 0 2024-02-28T16:31:26.484 INFO:tasks.workunit.client.1.smithi066.stdout:8/575: readlink d0/d6/dc/d16/d1a/d63/l2a 0 2024-02-28T16:31:26.484 INFO:tasks.workunit.client.0.smithi001.stdout:5/937: chown dc/d12/fa6 25408788 1 2024-02-28T16:31:26.486 INFO:tasks.workunit.client.0.smithi001.stdout:3/985: write d4/f14 [1575078,61745] 0 2024-02-28T16:31:26.488 INFO:tasks.workunit.client.1.smithi066.stdout:4/734: creat d1/d7a/ff2 x:0 0 0 2024-02-28T16:31:26.489 INFO:tasks.workunit.client.0.smithi001.stdout:2/891: write d0/f91 [4448425,12813] 0 2024-02-28T16:31:26.490 INFO:tasks.workunit.client.1.smithi066.stdout:6/578: dread d2/d2c/da1/d5d/d63/d6a/fb2 [0,4194304] 0 2024-02-28T16:31:26.490 INFO:tasks.workunit.client.0.smithi001.stdout:3/986: stat d4/d1e/d7c/d9a/cf3 0 2024-02-28T16:31:26.491 INFO:tasks.workunit.client.1.smithi066.stdout:5/590: dread - d3/d4/dc/d2f/d55/f5b zero size 2024-02-28T16:31:26.492 INFO:tasks.workunit.client.0.smithi001.stdout:2/892: stat d0/d2/d3/de/d61/d12/d55/db2/l110 0 2024-02-28T16:31:26.492 INFO:tasks.workunit.client.1.smithi066.stdout:8/576: dread - d0/d6/dc/d16/d1a/f8e zero size 2024-02-28T16:31:26.492 INFO:tasks.workunit.client.0.smithi001.stdout:5/938: unlink dc/d1a/d7d/c9e 0 2024-02-28T16:31:26.493 INFO:tasks.workunit.client.0.smithi001.stdout:2/893: chown d0/d2/da6 491 1 2024-02-28T16:31:26.495 INFO:tasks.workunit.client.0.smithi001.stdout:3/987: mknod d4/d6/d7/d78/d81/dd6/dd9/c140 0 2024-02-28T16:31:26.495 INFO:tasks.workunit.client.1.smithi066.stdout:0/718: link d2/d27/d41/d76/d93/db2/db5/d51/f70 d2/d85/d8d/feb 0 2024-02-28T16:31:26.497 INFO:tasks.workunit.client.1.smithi066.stdout:8/577: fsync d0/d6/dc/d16/d4c/f8f 0 2024-02-28T16:31:26.497 INFO:tasks.workunit.client.0.smithi001.stdout:5/939: symlink dc/d12/d41/dd6/l12c 0 2024-02-28T16:31:26.499 INFO:tasks.workunit.client.0.smithi001.stdout:5/940: write dc/d1a/d7d/f88 [1080042,96543] 0 2024-02-28T16:31:26.499 INFO:tasks.workunit.client.0.smithi001.stdout:2/894: mknod d0/d2/d3/de/d22/dbb/dc8/d117/c12b 0 2024-02-28T16:31:26.500 INFO:tasks.workunit.client.1.smithi066.stdout:0/719: creat d2/d27/d41/fec x:0 0 0 2024-02-28T16:31:26.501 INFO:tasks.workunit.client.0.smithi001.stdout:3/988: symlink d4/d1e/d7c/d7d/d80/l141 0 2024-02-28T16:31:26.502 INFO:tasks.workunit.client.0.smithi001.stdout:0/952: write d18/d22/d2f/d1c/f30 [5431269,63012] 0 2024-02-28T16:31:26.504 INFO:tasks.workunit.client.0.smithi001.stdout:6/964: write d9/db/d5f/d62/d83/f99 [811255,111157] 0 2024-02-28T16:31:26.507 INFO:tasks.workunit.client.1.smithi066.stdout:8/578: write d0/db/d15/f44 [4269964,95568] 0 2024-02-28T16:31:26.507 INFO:tasks.workunit.client.0.smithi001.stdout:3/989: symlink d4/d6/d7/d78/d81/l142 0 2024-02-28T16:31:26.508 INFO:tasks.workunit.client.0.smithi001.stdout:2/895: write d0/d2/d3/de/d61/d12/d55/db2/f106 [1017437,98651] 0 2024-02-28T16:31:26.508 INFO:tasks.workunit.client.1.smithi066.stdout:0/720: creat d2/d85/d8d/fed x:0 0 0 2024-02-28T16:31:26.511 INFO:tasks.workunit.client.0.smithi001.stdout:2/896: chown d0/d2/d3/de/d61/d12/d46/l8b 1843 1 2024-02-28T16:31:26.515 INFO:tasks.workunit.client.1.smithi066.stdout:2/629: truncate d0/f1e 1727171 0 2024-02-28T16:31:26.517 INFO:tasks.workunit.client.1.smithi066.stdout:0/721: dread - d2/d27/d41/d76/d93/db2/fe6 zero size 2024-02-28T16:31:26.519 INFO:tasks.workunit.client.0.smithi001.stdout:2/897: rename d0/d2/c47 to d0/d2/d3/de/d61/c12c 0 2024-02-28T16:31:26.521 INFO:tasks.workunit.client.0.smithi001.stdout:2/898: write d0/f91 [3683763,112946] 0 2024-02-28T16:31:26.522 INFO:tasks.workunit.client.1.smithi066.stdout:9/499: dread da/d22/d46/d56/d8f/f3b [0,4194304] 0 2024-02-28T16:31:26.522 INFO:tasks.workunit.client.0.smithi001.stdout:0/953: dwrite d18/d22/d3d/fd6 [0,4194304] 0 2024-02-28T16:31:26.523 INFO:tasks.workunit.client.1.smithi066.stdout:8/579: write d0/db/d12/d1d/d25/d65/f84 [614617,29837] 0 2024-02-28T16:31:26.524 INFO:tasks.workunit.client.0.smithi001.stdout:2/899: read d0/f17 [4041380,117738] 0 2024-02-28T16:31:26.527 INFO:tasks.workunit.client.1.smithi066.stdout:2/630: creat d0/fcf x:0 0 0 2024-02-28T16:31:26.528 INFO:tasks.workunit.client.0.smithi001.stdout:3/990: write d4/d1e/d7c/da8/fae [1155788,107530] 0 2024-02-28T16:31:26.529 INFO:tasks.workunit.client.1.smithi066.stdout:3/698: dread db/dc/d11/d1f/f2e [0,4194304] 0 2024-02-28T16:31:26.530 INFO:tasks.workunit.client.1.smithi066.stdout:7/576: dwrite f4 [0,4194304] 0 2024-02-28T16:31:26.532 INFO:tasks.workunit.client.1.smithi066.stdout:6/579: dwrite d2/d26/f47 [0,4194304] 0 2024-02-28T16:31:26.533 INFO:tasks.workunit.client.0.smithi001.stdout:5/941: dread dc/d1a/d7d/d87/f8f [0,4194304] 0 2024-02-28T16:31:26.534 INFO:tasks.workunit.client.1.smithi066.stdout:4/735: dwrite d1/d7a/f98 [0,4194304] 0 2024-02-28T16:31:26.534 INFO:tasks.workunit.client.1.smithi066.stdout:9/500: chown da/f20 9 1 2024-02-28T16:31:26.534 INFO:tasks.workunit.client.1.smithi066.stdout:2/631: chown d0/d12/d1a/d23/d5d/f4d 228681817 1 2024-02-28T16:31:26.537 INFO:tasks.workunit.client.0.smithi001.stdout:2/900: getdents d0 0 2024-02-28T16:31:26.539 INFO:tasks.workunit.client.1.smithi066.stdout:3/699: mknod db/d13/cfd 0 2024-02-28T16:31:26.542 INFO:tasks.workunit.client.1.smithi066.stdout:5/591: dread d3/d1b/d20/f63 [0,4194304] 0 2024-02-28T16:31:26.543 INFO:tasks.workunit.client.1.smithi066.stdout:4/736: creat d1/d23/d25/d77/ff3 x:0 0 0 2024-02-28T16:31:26.544 INFO:tasks.workunit.client.1.smithi066.stdout:8/580: read d0/db/d12/d1d/d25/fb5 [653862,90698] 0 2024-02-28T16:31:26.544 INFO:tasks.workunit.client.1.smithi066.stdout:2/632: write d0/d12/d3e/d9e/fa3 [2628968,38317] 0 2024-02-28T16:31:26.545 INFO:tasks.workunit.client.0.smithi001.stdout:2/901: mkdir d0/d12d 0 2024-02-28T16:31:26.546 INFO:tasks.workunit.client.1.smithi066.stdout:9/501: symlink da/d22/d46/d56/d8f/d94/la3 0 2024-02-28T16:31:26.547 INFO:tasks.workunit.client.0.smithi001.stdout:0/954: dwrite d18/d22/d2b/d5f/f85 [0,4194304] 0 2024-02-28T16:31:26.547 INFO:tasks.workunit.client.0.smithi001.stdout:5/942: dread dc/d12/d56/d96/db9/dc2/dca/f66 [0,4194304] 0 2024-02-28T16:31:26.548 INFO:tasks.workunit.client.1.smithi066.stdout:3/700: fdatasync db/dc/f55 0 2024-02-28T16:31:26.551 INFO:tasks.workunit.client.1.smithi066.stdout:5/592: chown d3/f98 88 1 2024-02-28T16:31:26.552 INFO:tasks.workunit.client.1.smithi066.stdout:2/633: read - d0/d12/d2e/d7a/fc5 zero size 2024-02-28T16:31:26.558 INFO:tasks.workunit.client.1.smithi066.stdout:0/722: rename d2/d27/d41/d76/d93/d33/d3f/d8e/c71 to d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/cee 0 2024-02-28T16:31:26.567 INFO:tasks.workunit.client.0.smithi001.stdout:0/955: dwrite f13 [4194304,4194304] 0 2024-02-28T16:31:26.568 INFO:tasks.workunit.client.1.smithi066.stdout:2/634: creat d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/fd0 x:0 0 0 2024-02-28T16:31:26.569 INFO:tasks.workunit.client.1.smithi066.stdout:6/580: getdents d2/da/d19/d75/dbd/dc6 0 2024-02-28T16:31:26.570 INFO:tasks.workunit.client.0.smithi001.stdout:0/956: dread - d18/d5a/fda zero size 2024-02-28T16:31:26.571 INFO:tasks.workunit.client.1.smithi066.stdout:0/723: unlink d2/d27/d41/d76/d93/db2/db5/f57 0 2024-02-28T16:31:26.572 INFO:tasks.workunit.client.1.smithi066.stdout:6/581: fsync d2/d2c/da1/fb7 0 2024-02-28T16:31:26.574 INFO:tasks.workunit.client.1.smithi066.stdout:5/593: unlink d3/d4/dc/dd/d2e/d8f/l95 0 2024-02-28T16:31:26.576 INFO:tasks.workunit.client.1.smithi066.stdout:4/737: dwrite d1/d18/d41/fbf [0,4194304] 0 2024-02-28T16:31:26.587 INFO:tasks.workunit.client.1.smithi066.stdout:2/635: write d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/fd0 [172074,27053] 0 2024-02-28T16:31:26.593 INFO:tasks.workunit.client.1.smithi066.stdout:5/594: creat d3/d4/dc/d56/fbc x:0 0 0 2024-02-28T16:31:26.597 INFO:tasks.workunit.client.1.smithi066.stdout:6/582: getdents d2/da/d19/d75/da6/daa/dbf 0 2024-02-28T16:31:26.598 INFO:tasks.workunit.client.1.smithi066.stdout:0/724: truncate d2/d27/d41/d76/d93/d10/d23/fc4 185701 0 2024-02-28T16:31:26.599 INFO:tasks.workunit.client.0.smithi001.stdout:5/943: dread dc/df/f81 [0,4194304] 0 2024-02-28T16:31:26.600 INFO:tasks.workunit.client.1.smithi066.stdout:2/636: mknod d0/d12/d1a/d39/d7c/cd1 0 2024-02-28T16:31:26.602 INFO:tasks.workunit.client.1.smithi066.stdout:8/581: dread d0/f7c [0,4194304] 0 2024-02-28T16:31:26.604 INFO:tasks.workunit.client.0.smithi001.stdout:5/944: readlink dc/d1a/d52/l114 0 2024-02-28T16:31:26.606 INFO:tasks.workunit.client.1.smithi066.stdout:9/502: write da/d4b/f54 [3889674,120785] 0 2024-02-28T16:31:26.606 INFO:tasks.workunit.client.1.smithi066.stdout:3/701: dwrite db/d2a/d44/d48/d52/fde [0,4194304] 0 2024-02-28T16:31:26.607 INFO:tasks.workunit.client.0.smithi001.stdout:3/991: dwrite d4/d6/d7/d78/d81/fc3 [0,4194304] 0 2024-02-28T16:31:26.610 INFO:tasks.workunit.client.0.smithi001.stdout:5/945: creat dc/d12/d56/f12d x:0 0 0 2024-02-28T16:31:26.611 INFO:tasks.workunit.client.1.smithi066.stdout:3/702: write db/dc/db3/fcd [667505,107503] 0 2024-02-28T16:31:26.618 INFO:tasks.workunit.client.0.smithi001.stdout:2/902: dwrite d0/d2/d3/de/d22/dbb/dc8/f116 [0,4194304] 0 2024-02-28T16:31:26.621 INFO:tasks.workunit.client.1.smithi066.stdout:1/592: sync 2024-02-28T16:31:26.623 INFO:tasks.workunit.client.0.smithi001.stdout:3/992: dread d4/d97/fed [0,4194304] 0 2024-02-28T16:31:26.623 INFO:tasks.workunit.client.0.smithi001.stdout:2/903: creat d0/d2/d3/de/d22/dbb/dc8/d117/d119/f12e x:0 0 0 2024-02-28T16:31:26.626 INFO:tasks.workunit.client.0.smithi001.stdout:2/904: symlink d0/d2/d3/de/d22/l12f 0 2024-02-28T16:31:26.626 INFO:tasks.workunit.client.1.smithi066.stdout:5/595: rename d3/d4/c99 to d3/d4/dc/d56/d93/cbd 0 2024-02-28T16:31:26.627 INFO:tasks.workunit.client.0.smithi001.stdout:3/993: truncate d4/d1e/f9e 495773 0 2024-02-28T16:31:26.627 INFO:tasks.workunit.client.1.smithi066.stdout:0/725: creat d2/d27/d41/d76/d93/db2/db5/d67/fef x:0 0 0 2024-02-28T16:31:26.628 INFO:tasks.workunit.client.1.smithi066.stdout:3/703: symlink db/dc/d11/d1f/d78/lfe 0 2024-02-28T16:31:26.629 INFO:tasks.workunit.client.0.smithi001.stdout:2/905: mknod d0/d2/d3/de/d61/d12/de8/c130 0 2024-02-28T16:31:26.630 INFO:tasks.workunit.client.0.smithi001.stdout:3/994: truncate d4/d18/d5f/f5b 1369167 0 2024-02-28T16:31:26.630 INFO:tasks.workunit.client.1.smithi066.stdout:7/577: dread dc/de/d17/d1e/f59 [0,4194304] 0 2024-02-28T16:31:26.635 INFO:tasks.workunit.client.0.smithi001.stdout:3/995: rename d4/d6/d7/f5d to d4/d1e/d7c/d7d/f143 0 2024-02-28T16:31:26.640 INFO:tasks.workunit.client.1.smithi066.stdout:6/583: dwrite d2/da/f1c [0,4194304] 0 2024-02-28T16:31:26.644 INFO:tasks.workunit.client.1.smithi066.stdout:8/582: mknod d0/d6/dc/d16/d4c/d6c/d6f/da7/cbf 0 2024-02-28T16:31:26.645 INFO:tasks.workunit.client.1.smithi066.stdout:9/503: creat da/d3a/d15/d48/fa4 x:0 0 0 2024-02-28T16:31:26.648 INFO:tasks.workunit.client.1.smithi066.stdout:8/583: write d0/d6/dc/d16/d4c/f8f [671769,33037] 0 2024-02-28T16:31:26.648 INFO:tasks.workunit.client.1.smithi066.stdout:5/596: mknod d3/d4/dc/dd/d2e/d8f/cbe 0 2024-02-28T16:31:26.650 INFO:tasks.workunit.client.1.smithi066.stdout:4/738: write d1/d18/d4e/f71 [263070,74936] 0 2024-02-28T16:31:26.651 INFO:tasks.workunit.client.1.smithi066.stdout:0/726: mkdir d2/d27/d41/d76/d93/df0 0 2024-02-28T16:31:26.655 INFO:tasks.workunit.client.0.smithi001.stdout:3/996: read d4/d18/f19 [8031362,29916] 0 2024-02-28T16:31:26.656 INFO:tasks.workunit.client.1.smithi066.stdout:5/597: dread d3/d1b/d39/fa6 [0,4194304] 0 2024-02-28T16:31:26.658 INFO:tasks.workunit.client.1.smithi066.stdout:3/704: mkdir db/dc/db3/dff 0 2024-02-28T16:31:26.664 INFO:tasks.workunit.client.1.smithi066.stdout:9/504: creat da/d22/d46/d56/d8f/d28/fa5 x:0 0 0 2024-02-28T16:31:26.667 INFO:tasks.workunit.client.0.smithi001.stdout:2/906: fsync d0/d2/d3/de/d22/dbb/dc8/f116 0 2024-02-28T16:31:26.668 INFO:tasks.workunit.client.0.smithi001.stdout:0/957: dread d18/d22/d2f/d1c/f21 [0,4194304] 0 2024-02-28T16:31:26.670 INFO:tasks.workunit.client.0.smithi001.stdout:0/958: rmdir d18/d22/d76/d7a 39 2024-02-28T16:31:26.671 INFO:tasks.workunit.client.1.smithi066.stdout:0/727: symlink d2/d27/d41/d76/d93/lf1 0 2024-02-28T16:31:26.672 INFO:tasks.workunit.client.1.smithi066.stdout:8/584: write d0/d6/dc/d16/d4c/d42/d73/f7f [1218038,103062] 0 2024-02-28T16:31:26.678 INFO:tasks.workunit.client.1.smithi066.stdout:2/637: dread d0/d12/d2e/d7a/f41 [0,4194304] 0 2024-02-28T16:31:26.680 INFO:tasks.workunit.client.1.smithi066.stdout:6/584: symlink d2/da/d19/d75/dbd/dc6/dc7/lcd 0 2024-02-28T16:31:26.684 INFO:tasks.workunit.client.1.smithi066.stdout:4/739: dwrite d1/d18/f2b [0,4194304] 0 2024-02-28T16:31:26.684 INFO:tasks.workunit.client.1.smithi066.stdout:2/638: chown d0/d12/f62 1636443676 1 2024-02-28T16:31:26.685 INFO:tasks.workunit.client.0.smithi001.stdout:2/907: dwrite d0/d2/d3/de/d8a/f51 [4194304,4194304] 0 2024-02-28T16:31:26.685 INFO:tasks.workunit.client.0.smithi001.stdout:0/959: dwrite d18/d5a/f109 [0,4194304] 0 2024-02-28T16:31:26.685 INFO:tasks.workunit.client.0.smithi001.stdout:2/908: symlink d0/d2/d3/de/d10f/l131 0 2024-02-28T16:31:26.685 INFO:tasks.workunit.client.0.smithi001.stdout:0/960: chown d18/ldd 9 1 2024-02-28T16:31:26.693 INFO:tasks.workunit.client.0.smithi001.stdout:2/909: creat d0/d2/d3/de/d22/dbb/dc8/d117/d119/f132 x:0 0 0 2024-02-28T16:31:26.695 INFO:tasks.workunit.client.0.smithi001.stdout:0/961: symlink d18/da9/dc9/d12f/l134 0 2024-02-28T16:31:26.696 INFO:tasks.workunit.client.1.smithi066.stdout:0/728: mknod d2/d27/d41/d76/d93/d33/d3f/d8e/cf2 0 2024-02-28T16:31:26.697 INFO:tasks.workunit.client.1.smithi066.stdout:4/740: write d1/d18/d41/fbf [5205481,10242] 0 2024-02-28T16:31:26.697 INFO:tasks.workunit.client.1.smithi066.stdout:5/598: link d3/l6 d3/d4/dc/dd/d41/d7f/d80/lbf 0 2024-02-28T16:31:26.698 INFO:tasks.workunit.client.0.smithi001.stdout:3/997: chown d4/d6/d7/d78/d81/dd6/dd9/f133 5304808 1 2024-02-28T16:31:26.700 INFO:tasks.workunit.client.0.smithi001.stdout:0/962: creat d18/f135 x:0 0 0 2024-02-28T16:31:26.703 INFO:tasks.workunit.client.0.smithi001.stdout:5/946: dwrite dc/d12/d56/f61 [0,4194304] 0 2024-02-28T16:31:26.703 INFO:tasks.workunit.client.1.smithi066.stdout:2/639: symlink d0/d12/d1a/d39/d7c/d87/ld2 0 2024-02-28T16:31:26.706 INFO:tasks.workunit.client.1.smithi066.stdout:3/705: dwrite db/dc/fca [0,4194304] 0 2024-02-28T16:31:26.706 INFO:tasks.workunit.client.1.smithi066.stdout:0/729: chown d2/d27/d41/d76/d93/l86 0 1 2024-02-28T16:31:26.708 INFO:tasks.workunit.client.0.smithi001.stdout:5/947: symlink dc/d12/d116/l12e 0 2024-02-28T16:31:26.711 INFO:tasks.workunit.client.1.smithi066.stdout:9/505: dwrite da/d22/d46/d56/d8f/d2d/f57 [0,4194304] 0 2024-02-28T16:31:26.712 INFO:tasks.workunit.client.0.smithi001.stdout:5/948: symlink dc/d12/d41/dd6/l12f 0 2024-02-28T16:31:26.714 INFO:tasks.workunit.client.0.smithi001.stdout:5/949: fdatasync dc/d12/fa6 0 2024-02-28T16:31:26.715 INFO:tasks.workunit.client.1.smithi066.stdout:5/599: rmdir d3/d4/dc/dd/d41/d7a/dac 39 2024-02-28T16:31:26.719 INFO:tasks.workunit.client.1.smithi066.stdout:2/640: stat d0/d12/d1a/d23/d5d/fbd 0 2024-02-28T16:31:26.720 INFO:tasks.workunit.client.0.smithi001.stdout:5/950: rename dc/d12/daa/lf0 to dc/d12/d41/dd6/l130 0 2024-02-28T16:31:26.735 INFO:tasks.workunit.client.0.smithi001.stdout:2/910: truncate d0/d2/d3/de/d8a/f6e 1488974 0 2024-02-28T16:31:26.735 INFO:tasks.workunit.client.1.smithi066.stdout:6/585: dwrite d2/f6f [0,4194304] 0 2024-02-28T16:31:26.735 INFO:tasks.workunit.client.1.smithi066.stdout:1/593: truncate d2/d18/d6b/f77 1818188 0 2024-02-28T16:31:26.739 INFO:tasks.workunit.client.0.smithi001.stdout:0/963: creat d18/d22/d76/d93/f136 x:0 0 0 2024-02-28T16:31:26.743 INFO:tasks.workunit.client.0.smithi001.stdout:0/964: creat d18/da9/dea/f137 x:0 0 0 2024-02-28T16:31:26.744 INFO:tasks.workunit.client.0.smithi001.stdout:3/998: write d4/d6/d7/d78/d81/dd6/f13d [262940,98896] 0 2024-02-28T16:31:26.744 INFO:tasks.workunit.client.1.smithi066.stdout:0/730: rename d2/d27/d41/dc5 to d2/d27/d41/d76/d93/df3 0 2024-02-28T16:31:26.745 INFO:tasks.workunit.client.0.smithi001.stdout:0/965: creat d18/da9/f138 x:0 0 0 2024-02-28T16:31:26.746 INFO:tasks.workunit.client.1.smithi066.stdout:8/585: truncate d0/db/d15/f37 2635938 0 2024-02-28T16:31:26.749 INFO:tasks.workunit.client.0.smithi001.stdout:0/966: creat d18/d22/d3d/f139 x:0 0 0 2024-02-28T16:31:26.750 INFO:tasks.workunit.client.0.smithi001.stdout:3/999: creat d4/d6/d7/d78/f144 x:0 0 0 2024-02-28T16:31:26.750 INFO:tasks.workunit.client.1.smithi066.stdout:6/586: creat d2/da/d19/d75/d88/fce x:0 0 0 2024-02-28T16:31:26.751 INFO:tasks.workunit.client.1.smithi066.stdout:9/506: rename da/d22/d46/d56/d8f/d28/l75 to da/d3a/d58/d97/la6 0 2024-02-28T16:31:26.755 INFO:tasks.workunit.client.1.smithi066.stdout:0/731: mkdir d2/d27/d41/df4 0 2024-02-28T16:31:26.757 INFO:tasks.workunit.client.1.smithi066.stdout:4/741: write d1/d7a/fd7 [1447819,83796] 0 2024-02-28T16:31:26.758 INFO:tasks.workunit.client.1.smithi066.stdout:1/594: rename d2/d27/d2b/d43/d46/d80/l93 to d2/d18/d7a/dbc/lc0 0 2024-02-28T16:31:26.758 INFO:tasks.workunit.client.1.smithi066.stdout:6/587: readlink d2/da/d19/lf 0 2024-02-28T16:31:26.758 INFO:tasks.workunit.client.1.smithi066.stdout:9/507: dread - da/df/d70/d90/f93 zero size 2024-02-28T16:31:26.759 INFO:tasks.workunit.client.1.smithi066.stdout:0/732: creat d2/d27/d41/ff5 x:0 0 0 2024-02-28T16:31:26.761 INFO:tasks.workunit.client.1.smithi066.stdout:4/742: readlink d1/d18/d41/ld8 0 2024-02-28T16:31:26.762 INFO:tasks.workunit.client.1.smithi066.stdout:8/586: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/fc0 x:0 0 0 2024-02-28T16:31:26.766 INFO:tasks.workunit.client.1.smithi066.stdout:2/641: dwrite d0/d12/d2e/d7a/f41 [4194304,4194304] 0 2024-02-28T16:31:26.772 INFO:tasks.workunit.client.0.smithi001.stdout:5/951: dwrite dc/df/f13 [0,4194304] 0 2024-02-28T16:31:26.773 INFO:tasks.workunit.client.1.smithi066.stdout:0/733: read d2/d27/d41/d76/d93/db2/db5/d73/fd5 [593006,121800] 0 2024-02-28T16:31:26.777 INFO:tasks.workunit.client.1.smithi066.stdout:8/587: stat d0/d6/dc/d16/d4c/d6c/d6f 0 2024-02-28T16:31:26.777 INFO:tasks.workunit.client.1.smithi066.stdout:9/508: creat da/d4b/d53/fa7 x:0 0 0 2024-02-28T16:31:26.781 INFO:tasks.workunit.client.1.smithi066.stdout:6/588: mknod d2/da/d19/d75/dbd/ccf 0 2024-02-28T16:31:26.782 INFO:tasks.workunit.client.0.smithi001.stdout:2/911: dwrite d0/d2/d3/de/d22/f6d [0,4194304] 0 2024-02-28T16:31:26.785 INFO:tasks.workunit.client.0.smithi001.stdout:0/967: write d18/d22/d3d/f67 [48414,112224] 0 2024-02-28T16:31:26.786 INFO:tasks.workunit.client.1.smithi066.stdout:7/578: dread dc/de/d17/d37/f72 [0,4194304] 0 2024-02-28T16:31:26.786 INFO:tasks.workunit.client.0.smithi001.stdout:2/912: stat d0/d2/d3/de/d22/dbb 0 2024-02-28T16:31:26.788 INFO:tasks.workunit.client.1.smithi066.stdout:0/734: readlink d2/d27/d41/d76/d93/d33/d3f/d8e/lbd 0 2024-02-28T16:31:26.801 INFO:tasks.workunit.client.0.smithi001.stdout:5/952: write dc/d12/d56/d96/db9/dc2/dca/d3c/f46 [2118412,36175] 0 2024-02-28T16:31:26.805 INFO:tasks.workunit.client.0.smithi001.stdout:2/913: dwrite d0/d2/d3/de/d22/f38 [0,4194304] 0 2024-02-28T16:31:26.807 INFO:tasks.workunit.client.1.smithi066.stdout:1/595: dwrite d2/f75 [0,4194304] 0 2024-02-28T16:31:26.810 INFO:tasks.workunit.client.1.smithi066.stdout:8/588: getdents d0/d6/dc/d16/d4c/d6c/daf 0 2024-02-28T16:31:26.813 INFO:tasks.workunit.client.1.smithi066.stdout:5/600: dwrite d3/d4/dc/dd/d2e/f79 [0,4194304] 0 2024-02-28T16:31:26.813 INFO:tasks.workunit.client.1.smithi066.stdout:3/706: dread db/d13/d14/d2d/d92/fac [0,4194304] 0 2024-02-28T16:31:26.814 INFO:tasks.workunit.client.0.smithi001.stdout:0/968: dwrite d18/d22/d2b/d5f/fde [0,4194304] 0 2024-02-28T16:31:26.820 INFO:tasks.workunit.client.1.smithi066.stdout:3/707: chown db/dc/d35/d3d/d76/l8d 159 1 2024-02-28T16:31:26.824 INFO:tasks.workunit.client.0.smithi001.stdout:0/969: rmdir d18/d22/d2b/d36/d48 39 2024-02-28T16:31:26.825 INFO:tasks.workunit.client.1.smithi066.stdout:2/642: getdents d0/d12/d1a/d39/d7c/d87 0 2024-02-28T16:31:26.825 INFO:tasks.workunit.client.1.smithi066.stdout:9/509: dwrite da/d3a/f5f [4194304,4194304] 0 2024-02-28T16:31:26.826 INFO:tasks.workunit.client.1.smithi066.stdout:3/708: readlink db/d2a/l53 0 2024-02-28T16:31:26.828 INFO:tasks.workunit.client.0.smithi001.stdout:2/914: dwrite d0/d2/d3/d36/d94/f10a [0,4194304] 0 2024-02-28T16:31:26.829 INFO:tasks.workunit.client.1.smithi066.stdout:6/589: dwrite d2/da/f1e [0,4194304] 0 2024-02-28T16:31:26.832 INFO:tasks.workunit.client.1.smithi066.stdout:6/590: chown d2/d2c/da1/fb7 466190109 1 2024-02-28T16:31:26.834 INFO:tasks.workunit.client.0.smithi001.stdout:5/953: write dc/d12/d41/d54/f9a [982602,103065] 0 2024-02-28T16:31:26.834 INFO:tasks.workunit.client.0.smithi001.stdout:2/915: fsync d0/d2/d3/de/f125 0 2024-02-28T16:31:26.837 INFO:tasks.workunit.client.1.smithi066.stdout:9/510: stat da/d4b/d53/fa7 0 2024-02-28T16:31:26.838 INFO:tasks.workunit.client.0.smithi001.stdout:0/970: fsync d18/d22/d2b/d36/f119 0 2024-02-28T16:31:26.839 INFO:tasks.workunit.client.1.smithi066.stdout:1/596: rename d2/lbd to d2/d18/d7a/db1/lc1 0 2024-02-28T16:31:26.841 INFO:tasks.workunit.client.0.smithi001.stdout:0/971: read - d18/d22/d2f/d1c/d90/f11e zero size 2024-02-28T16:31:26.841 INFO:tasks.workunit.client.0.smithi001.stdout:2/916: rename d0/f40 to d0/d2/d126/f133 0 2024-02-28T16:31:26.844 INFO:tasks.workunit.client.0.smithi001.stdout:5/954: rename dc/d12/daa to dc/d1a/d52/dfa/d131 0 2024-02-28T16:31:26.847 INFO:tasks.workunit.client.1.smithi066.stdout:3/709: unlink db/dc/d35/f58 0 2024-02-28T16:31:26.847 INFO:tasks.workunit.client.1.smithi066.stdout:6/591: write d2/da/d19/d75/dbd/dc6/fca [1429067,37193] 0 2024-02-28T16:31:26.847 INFO:tasks.workunit.client.0.smithi001.stdout:0/972: creat d18/da9/dea/d11d/f13a x:0 0 0 2024-02-28T16:31:26.847 INFO:tasks.workunit.client.0.smithi001.stdout:8/945: sync 2024-02-28T16:31:26.848 INFO:tasks.workunit.client.0.smithi001.stdout:6/965: sync 2024-02-28T16:31:26.848 INFO:tasks.workunit.client.1.smithi066.stdout:4/743: dread d1/d7a/fd7 [0,4194304] 0 2024-02-28T16:31:26.849 INFO:tasks.workunit.client.0.smithi001.stdout:6/966: chown d9/db/fd8 39430156 1 2024-02-28T16:31:26.850 INFO:tasks.workunit.client.1.smithi066.stdout:0/735: getdents d2/d27/d41/d76/d93/db2/db5/d52 0 2024-02-28T16:31:26.853 INFO:tasks.workunit.client.0.smithi001.stdout:8/946: mkdir d4/d32/d69/d8f/d135 0 2024-02-28T16:31:26.854 INFO:tasks.workunit.client.1.smithi066.stdout:1/597: rmdir d2/d27/d2b/d37 39 2024-02-28T16:31:26.855 INFO:tasks.workunit.client.1.smithi066.stdout:4/744: fdatasync d1/d18/d4e/fa5 0 2024-02-28T16:31:26.856 INFO:tasks.workunit.client.1.smithi066.stdout:7/579: dwrite dc/de/d17/d1e/f3c [0,4194304] 0 2024-02-28T16:31:26.862 INFO:tasks.workunit.client.0.smithi001.stdout:8/947: symlink d4/d32/d69/d8f/d135/l136 0 2024-02-28T16:31:26.863 INFO:tasks.workunit.client.1.smithi066.stdout:0/736: chown d2/d27/d41/d76/d93/d10/d23/d6d 1 1 2024-02-28T16:31:26.864 INFO:tasks.workunit.client.1.smithi066.stdout:2/643: rename d0/d12/d1a/d39/l96 to d0/d12/d1a/d23/d5d/ld3 0 2024-02-28T16:31:26.866 INFO:tasks.workunit.client.0.smithi001.stdout:8/948: read d4/d32/d69/d9a/f102 [425021,24656] 0 2024-02-28T16:31:26.866 INFO:tasks.workunit.client.1.smithi066.stdout:9/511: rename da/d22/d46/d56/d8f to da/d22/d46/d56/d8f/d94/da8 22 2024-02-28T16:31:26.869 INFO:tasks.workunit.client.0.smithi001.stdout:8/949: truncate d4/d7/d22/f48 717492 0 2024-02-28T16:31:26.874 INFO:tasks.workunit.client.1.smithi066.stdout:0/737: rmdir d2/d27/d41/d76 39 2024-02-28T16:31:26.874 INFO:tasks.workunit.client.1.smithi066.stdout:4/745: creat d1/d18/da8/ff4 x:0 0 0 2024-02-28T16:31:26.877 INFO:tasks.workunit.client.0.smithi001.stdout:8/950: getdents d4/d32 0 2024-02-28T16:31:26.879 INFO:tasks.workunit.client.1.smithi066.stdout:8/589: dwrite d0/db/d12/d1d/d25/fb5 [0,4194304] 0 2024-02-28T16:31:26.880 INFO:tasks.workunit.client.0.smithi001.stdout:2/917: creat d0/d2/d3/de/f134 x:0 0 0 2024-02-28T16:31:26.880 INFO:tasks.workunit.client.0.smithi001.stdout:8/951: creat d4/d47/df4/f137 x:0 0 0 2024-02-28T16:31:26.883 INFO:tasks.workunit.client.0.smithi001.stdout:5/955: dwrite dc/d12/f16 [0,4194304] 0 2024-02-28T16:31:26.886 INFO:tasks.workunit.client.0.smithi001.stdout:5/956: chown dc/d12/d56/d96/db9/dc2/dca/d91/l9f 5155 1 2024-02-28T16:31:26.887 INFO:tasks.workunit.client.1.smithi066.stdout:2/644: mknod d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/cd4 0 2024-02-28T16:31:26.887 INFO:tasks.workunit.client.1.smithi066.stdout:7/580: link dc/de/d17/c51 dc/cac 0 2024-02-28T16:31:26.887 INFO:tasks.workunit.client.1.smithi066.stdout:6/592: dwrite d2/d2c/f86 [0,4194304] 0 2024-02-28T16:31:26.888 INFO:tasks.workunit.client.0.smithi001.stdout:5/957: readlink dc/d1a/d52/l63 0 2024-02-28T16:31:26.888 INFO:tasks.workunit.client.1.smithi066.stdout:3/710: write db/d13/d14/dad/fe1 [323580,82231] 0 2024-02-28T16:31:26.889 INFO:tasks.workunit.client.1.smithi066.stdout:1/598: rename d2/d18/d7a/f73 to d2/d27/fc2 0 2024-02-28T16:31:26.890 INFO:tasks.workunit.client.0.smithi001.stdout:5/958: read - dc/d12/d41/dd6/f11c zero size 2024-02-28T16:31:26.894 INFO:tasks.workunit.client.1.smithi066.stdout:2/645: stat d0/d12/d2e/d7a/fba 0 2024-02-28T16:31:26.898 INFO:tasks.workunit.client.0.smithi001.stdout:2/918: dread d0/d2/d3/de/d61/d12/d55/db2/fe6 [0,4194304] 0 2024-02-28T16:31:26.899 INFO:tasks.workunit.client.1.smithi066.stdout:7/581: stat dc/de/l18 0 2024-02-28T16:31:26.903 INFO:tasks.workunit.client.1.smithi066.stdout:9/512: rename da/d22/d46/c73 to da/d22/d46/d56/d8f/d28/ca9 0 2024-02-28T16:31:26.905 INFO:tasks.workunit.client.1.smithi066.stdout:0/738: write d2/d27/d41/d76/d93/d10/d23/d6d/f79 [5739646,59138] 0 2024-02-28T16:31:26.905 INFO:tasks.workunit.client.1.smithi066.stdout:2/646: mkdir d0/d12/d3e/dd5 0 2024-02-28T16:31:26.907 INFO:tasks.workunit.client.1.smithi066.stdout:3/711: write db/dc/d35/d3d/f97 [4496089,30479] 0 2024-02-28T16:31:26.916 INFO:tasks.workunit.client.1.smithi066.stdout:1/599: truncate d2/d27/d2b/d37/f82 345587 0 2024-02-28T16:31:26.919 INFO:tasks.workunit.client.1.smithi066.stdout:8/590: rename d0/d6/dc/d16/d1a/d63/l2a to d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/lc1 0 2024-02-28T16:31:26.921 INFO:tasks.workunit.client.1.smithi066.stdout:1/600: write d2/d27/d2b/d43/f9d [108285,4750] 0 2024-02-28T16:31:26.922 INFO:tasks.workunit.client.1.smithi066.stdout:0/739: truncate d2/d27/d41/d76/d93/db2/db5/d67/fe1 148094 0 2024-02-28T16:31:26.926 INFO:tasks.workunit.client.0.smithi001.stdout:5/959: read f5 [2271546,121327] 0 2024-02-28T16:31:26.930 INFO:tasks.workunit.client.1.smithi066.stdout:2/647: fdatasync d0/d12/d2e/d7a/f55 0 2024-02-28T16:31:26.931 INFO:tasks.workunit.client.0.smithi001.stdout:5/960: mknod dc/c132 0 2024-02-28T16:31:26.932 INFO:tasks.workunit.client.0.smithi001.stdout:8/952: write d4/d7/d22/d27/d58/f87 [598638,42146] 0 2024-02-28T16:31:26.938 INFO:tasks.workunit.client.1.smithi066.stdout:6/593: dwrite d2/d26/fa3 [0,4194304] 0 2024-02-28T16:31:26.939 INFO:tasks.workunit.client.0.smithi001.stdout:5/961: write dc/d12/d56/d96/db9/dc2/dca/d3c/d4a/f82 [1168724,34065] 0 2024-02-28T16:31:26.941 INFO:tasks.workunit.client.1.smithi066.stdout:2/648: fdatasync d0/d12/d2e/d3b/f64 0 2024-02-28T16:31:26.941 INFO:tasks.workunit.client.1.smithi066.stdout:0/740: write d2/d27/d41/d76/d93/d10/d23/d6d/f79 [2720802,60552] 0 2024-02-28T16:31:26.941 INFO:tasks.workunit.client.0.smithi001.stdout:8/953: truncate d4/d7/da8/f130 4875638 0 2024-02-28T16:31:26.943 INFO:tasks.workunit.client.0.smithi001.stdout:2/919: dwrite d0/d2/da6/fdc [0,4194304] 0 2024-02-28T16:31:26.945 INFO:tasks.workunit.client.0.smithi001.stdout:5/962: mkdir dc/d12/d56/d96/db9/dc2/dca/d91/d133 0 2024-02-28T16:31:26.945 INFO:tasks.workunit.client.0.smithi001.stdout:8/954: chown d4/d7/d22/d27/f2f 3901 1 2024-02-28T16:31:26.948 INFO:tasks.workunit.client.1.smithi066.stdout:3/712: dwrite db/d2a/d44/d48/f85 [0,4194304] 0 2024-02-28T16:31:26.949 INFO:tasks.workunit.client.1.smithi066.stdout:4/746: write d1/d18/f38 [1445362,68286] 0 2024-02-28T16:31:26.950 INFO:tasks.workunit.client.1.smithi066.stdout:8/591: truncate d0/db/d12/d1d/d25/f75 209646 0 2024-02-28T16:31:26.951 INFO:tasks.workunit.client.0.smithi001.stdout:2/920: creat d0/d2/d126/f135 x:0 0 0 2024-02-28T16:31:26.951 INFO:tasks.workunit.client.0.smithi001.stdout:5/963: creat dc/d1a/d7d/d87/df3/d112/f134 x:0 0 0 2024-02-28T16:31:26.955 INFO:tasks.workunit.client.0.smithi001.stdout:8/955: mknod d4/d7/d22/d27/d58/d94/c138 0 2024-02-28T16:31:26.956 INFO:tasks.workunit.client.0.smithi001.stdout:2/921: truncate d0/d2/d3/d36/d94/f96 2588963 0 2024-02-28T16:31:26.958 INFO:tasks.workunit.client.1.smithi066.stdout:9/513: dread da/d4b/f54 [0,4194304] 0 2024-02-28T16:31:26.960 INFO:tasks.workunit.client.1.smithi066.stdout:7/582: dread dc/d29/f2e [0,4194304] 0 2024-02-28T16:31:26.961 INFO:tasks.workunit.client.0.smithi001.stdout:5/964: creat dc/d12/d56/d96/db9/dc2/dca/f135 x:0 0 0 2024-02-28T16:31:26.963 INFO:tasks.workunit.client.1.smithi066.stdout:6/594: rmdir d2/da/d19/d75/dbd/dc6/dc7 39 2024-02-28T16:31:26.963 INFO:tasks.workunit.client.0.smithi001.stdout:8/956: mkdir d4/d139 0 2024-02-28T16:31:26.965 INFO:tasks.workunit.client.1.smithi066.stdout:2/649: symlink d0/d12/d2e/d3b/ld6 0 2024-02-28T16:31:26.968 INFO:tasks.workunit.client.0.smithi001.stdout:5/965: creat dc/d1a/d7d/d87/f136 x:0 0 0 2024-02-28T16:31:26.972 INFO:tasks.workunit.client.1.smithi066.stdout:3/713: creat db/dc/d35/d3d/d76/d89/dcc/f100 x:0 0 0 2024-02-28T16:31:26.975 INFO:tasks.workunit.client.1.smithi066.stdout:1/601: truncate d2/d18/d19/f1c 600263 0 2024-02-28T16:31:26.976 INFO:tasks.workunit.client.1.smithi066.stdout:6/595: mkdir d2/d2c/d6e/dd0 0 2024-02-28T16:31:26.977 INFO:tasks.workunit.client.0.smithi001.stdout:5/966: unlink dc/d12/d56/d96/db9/dc2/dca/f77 0 2024-02-28T16:31:26.978 INFO:tasks.workunit.client.1.smithi066.stdout:6/596: read d2/f6f [2231802,77283] 0 2024-02-28T16:31:26.979 INFO:tasks.workunit.client.1.smithi066.stdout:4/747: mknod d1/d23/d25/dd9/cf5 0 2024-02-28T16:31:26.981 INFO:tasks.workunit.client.0.smithi001.stdout:5/967: symlink dc/d12/d116/l137 0 2024-02-28T16:31:26.983 INFO:tasks.workunit.client.1.smithi066.stdout:2/650: fsync d0/d12/d2e/d7a/f55 0 2024-02-28T16:31:26.986 INFO:tasks.workunit.client.1.smithi066.stdout:3/714: symlink db/d7a/l101 0 2024-02-28T16:31:26.987 INFO:tasks.workunit.client.0.smithi001.stdout:5/968: getdents dc/d12/d56/d96/db9/dc2/dca/d91/d133 0 2024-02-28T16:31:26.990 INFO:tasks.workunit.client.0.smithi001.stdout:5/969: dread - dc/d12/d56/d96/db9/dc2/dca/f135 zero size 2024-02-28T16:31:26.996 INFO:tasks.workunit.client.0.smithi001.stdout:5/970: read dc/d12/d56/d96/db9/dc2/dca/f9d [2479565,84720] 0 2024-02-28T16:31:26.997 INFO:tasks.workunit.client.1.smithi066.stdout:7/583: dread dc/de/d17/d91/f9b [0,4194304] 0 2024-02-28T16:31:26.999 INFO:tasks.workunit.client.0.smithi001.stdout:8/957: dread d4/d32/d11a/fb0 [0,4194304] 0 2024-02-28T16:31:27.000 INFO:tasks.workunit.client.1.smithi066.stdout:1/602: dwrite d2/d18/d19/f89 [0,4194304] 0 2024-02-28T16:31:27.001 INFO:tasks.workunit.client.1.smithi066.stdout:0/741: write d2/d27/d41/d76/d93/d10/d23/f7a [1143350,101093] 0 2024-02-28T16:31:27.001 INFO:tasks.workunit.client.0.smithi001.stdout:2/922: dread d0/d2/d3/de/d61/d12/d46/f66 [0,4194304] 0 2024-02-28T16:31:27.003 INFO:tasks.workunit.client.1.smithi066.stdout:1/603: chown d2/d27/d2b/l6a 449677 1 2024-02-28T16:31:27.004 INFO:tasks.workunit.client.0.smithi001.stdout:5/971: getdents dc/d12/d56/d96/db9/dc2/dca/d3c 0 2024-02-28T16:31:27.004 INFO:tasks.workunit.client.0.smithi001.stdout:8/958: truncate d4/d7/d22/d27/d4b/fa9 1099147 0 2024-02-28T16:31:27.006 INFO:tasks.workunit.client.0.smithi001.stdout:8/959: stat d4/d7/d22/fab 0 2024-02-28T16:31:27.006 INFO:tasks.workunit.client.0.smithi001.stdout:2/923: mknod d0/d2/d3/de/d61/d12/de8/c136 0 2024-02-28T16:31:27.007 INFO:tasks.workunit.client.0.smithi001.stdout:5/972: rmdir dc/d12/d41/db7 39 2024-02-28T16:31:27.009 INFO:tasks.workunit.client.0.smithi001.stdout:2/924: read - d0/d2/d126/f135 zero size 2024-02-28T16:31:27.010 INFO:tasks.workunit.client.0.smithi001.stdout:5/973: creat dc/d1a/d7d/d87/df3/d112/f138 x:0 0 0 2024-02-28T16:31:27.011 INFO:tasks.workunit.client.1.smithi066.stdout:8/592: getdents d0/d6/dc/d16/d4c/d42/da8 0 2024-02-28T16:31:27.012 INFO:tasks.workunit.client.1.smithi066.stdout:9/514: write da/d22/d46/d56/d8f/d2d/f62 [1278063,2795] 0 2024-02-28T16:31:27.013 INFO:tasks.workunit.client.0.smithi001.stdout:8/960: unlink d4/d32/d69/d9a/dcd/c128 0 2024-02-28T16:31:27.013 INFO:tasks.workunit.client.0.smithi001.stdout:2/925: symlink d0/d2/d3/de/d22/dbb/l137 0 2024-02-28T16:31:27.014 INFO:tasks.workunit.client.0.smithi001.stdout:5/974: write dc/d1a/d52/f89 [4723402,78929] 0 2024-02-28T16:31:27.017 INFO:tasks.workunit.client.1.smithi066.stdout:6/597: creat d2/da/d19/d75/dbd/dc6/fd1 x:0 0 0 2024-02-28T16:31:27.020 INFO:tasks.workunit.client.0.smithi001.stdout:8/961: mknod d4/d7/d22/d27/d4b/d80/c13a 0 2024-02-28T16:31:27.020 INFO:tasks.workunit.client.1.smithi066.stdout:4/748: dwrite d1/d18/d41/dba/f9d [0,4194304] 0 2024-02-28T16:31:27.021 INFO:tasks.workunit.client.1.smithi066.stdout:2/651: rename d0/d12/d3e/dd5 to d0/dd7 0 2024-02-28T16:31:27.024 INFO:tasks.workunit.client.0.smithi001.stdout:2/926: unlink d0/d2/d3/de/d61/c50 0 2024-02-28T16:31:27.025 INFO:tasks.workunit.client.0.smithi001.stdout:8/962: fsync d4/d7/d22/d27/d4b/f78 0 2024-02-28T16:31:27.026 INFO:tasks.workunit.client.1.smithi066.stdout:2/652: chown d0/d12/d2e/d7a/f55 1049170 1 2024-02-28T16:31:27.026 INFO:tasks.workunit.client.1.smithi066.stdout:5/601: sync 2024-02-28T16:31:27.026 INFO:tasks.workunit.client.1.smithi066.stdout:5/602: stat d3 0 2024-02-28T16:31:27.026 INFO:tasks.workunit.client.1.smithi066.stdout:3/715: truncate db/ff 71305 0 2024-02-28T16:31:27.029 INFO:tasks.workunit.client.1.smithi066.stdout:0/742: symlink d2/d27/d41/d76/d93/db2/db5/d51/lf6 0 2024-02-28T16:31:27.030 INFO:tasks.workunit.client.1.smithi066.stdout:2/653: stat d0/d12/d1a/d23/d29/l8e 0 2024-02-28T16:31:27.031 INFO:tasks.workunit.client.0.smithi001.stdout:2/927: write d0/d2/d3/de/d22/fd1 [1809135,117804] 0 2024-02-28T16:31:27.031 INFO:tasks.workunit.client.0.smithi001.stdout:8/963: read d4/d32/f7c [3165594,54212] 0 2024-02-28T16:31:27.032 INFO:tasks.workunit.client.0.smithi001.stdout:8/964: stat d4/d7/l54 0 2024-02-28T16:31:27.033 INFO:tasks.workunit.client.1.smithi066.stdout:7/584: dread dc/de/d17/d37/d47/f81 [0,4194304] 0 2024-02-28T16:31:27.035 INFO:tasks.workunit.client.0.smithi001.stdout:2/928: symlink d0/d2/d3/de/d61/d12/de8/l138 0 2024-02-28T16:31:27.038 INFO:tasks.workunit.client.1.smithi066.stdout:6/598: dread d2/d26/f47 [0,4194304] 0 2024-02-28T16:31:27.038 INFO:tasks.workunit.client.1.smithi066.stdout:7/585: write dc/de/f3f [5240172,130696] 0 2024-02-28T16:31:27.038 INFO:tasks.workunit.client.1.smithi066.stdout:9/515: mkdir da/df/d70/d90/d98/daa 0 2024-02-28T16:31:27.039 INFO:tasks.workunit.client.0.smithi001.stdout:8/965: rename d4/d7/d22/d27/d58/d94/dc9/fc1 to d4/d32/d69/d8f/f13b 0 2024-02-28T16:31:27.041 INFO:tasks.workunit.client.1.smithi066.stdout:6/599: stat d2/d2c/da1/d5d/d63/fc2 0 2024-02-28T16:31:27.042 INFO:tasks.workunit.client.0.smithi001.stdout:8/966: write d4/d7/d22/d27/d58/f87 [996125,1280] 0 2024-02-28T16:31:27.043 INFO:tasks.workunit.client.1.smithi066.stdout:3/716: mkdir db/d13/d14/dda/d102 0 2024-02-28T16:31:27.044 INFO:tasks.workunit.client.1.smithi066.stdout:8/593: creat d0/db/fc2 x:0 0 0 2024-02-28T16:31:27.044 INFO:tasks.workunit.client.1.smithi066.stdout:0/743: creat d2/d27/d41/d76/d93/db2/db5/d67/d94/ff7 x:0 0 0 2024-02-28T16:31:27.047 INFO:tasks.workunit.client.0.smithi001.stdout:2/929: link d0/d2/d126/d84/fd7 d0/d12d/f139 0 2024-02-28T16:31:27.051 INFO:tasks.workunit.client.1.smithi066.stdout:1/604: rename d2/fc to d2/d27/d2b/d37/fc3 0 2024-02-28T16:31:27.053 INFO:tasks.workunit.client.0.smithi001.stdout:5/975: dwrite dc/d12/f3f [0,4194304] 0 2024-02-28T16:31:27.053 INFO:tasks.workunit.client.0.smithi001.stdout:2/930: mknod d0/d2/d3/de/d8a/c13a 0 2024-02-28T16:31:27.054 INFO:tasks.workunit.client.1.smithi066.stdout:6/600: mknod d2/da/d19/d75/d88/cd2 0 2024-02-28T16:31:27.056 INFO:tasks.workunit.client.0.smithi001.stdout:5/976: stat dc/d12/d56/d96/db9/f105 0 2024-02-28T16:31:27.056 INFO:tasks.workunit.client.0.smithi001.stdout:2/931: stat d0/d2/d3/de/d8a/cb5 0 2024-02-28T16:31:27.058 INFO:tasks.workunit.client.1.smithi066.stdout:3/717: creat db/d13/d14/d84/f103 x:0 0 0 2024-02-28T16:31:27.059 INFO:tasks.workunit.client.1.smithi066.stdout:3/718: readlink db/d7a/l101 0 2024-02-28T16:31:27.061 INFO:tasks.workunit.client.1.smithi066.stdout:8/594: creat d0/d6/dc/d16/fc3 x:0 0 0 2024-02-28T16:31:27.062 INFO:tasks.workunit.client.1.smithi066.stdout:3/719: fsync db/d2a/ff7 0 2024-02-28T16:31:27.062 INFO:tasks.workunit.client.1.smithi066.stdout:7/586: creat dc/d56/fad x:0 0 0 2024-02-28T16:31:27.063 INFO:tasks.workunit.client.0.smithi001.stdout:2/932: symlink d0/d2/d3/d36/d94/l13b 0 2024-02-28T16:31:27.067 INFO:tasks.workunit.client.0.smithi001.stdout:2/933: read d0/d2/d3/de/d22/fd1 [969044,80261] 0 2024-02-28T16:31:27.067 INFO:tasks.workunit.client.1.smithi066.stdout:8/595: dread - d0/d6/dc/d16/d1a/d63/d58/d5a/f81 zero size 2024-02-28T16:31:27.068 INFO:tasks.workunit.client.1.smithi066.stdout:0/744: creat d2/d27/d41/d76/d93/db2/db5/d60/ff8 x:0 0 0 2024-02-28T16:31:27.073 INFO:tasks.workunit.client.1.smithi066.stdout:4/749: write d1/f10 [703562,87155] 0 2024-02-28T16:31:27.073 INFO:tasks.workunit.client.1.smithi066.stdout:6/601: mknod d2/d2c/da1/d5d/cd3 0 2024-02-28T16:31:27.074 INFO:tasks.workunit.client.0.smithi001.stdout:2/934: rename d0/d2/d3/de/f125 to d0/d2/d3/d73/f13c 0 2024-02-28T16:31:27.075 INFO:tasks.workunit.client.1.smithi066.stdout:3/720: mknod db/dc/db3/c104 0 2024-02-28T16:31:27.075 INFO:tasks.workunit.client.1.smithi066.stdout:8/596: chown d0/d6/dc/d16/d4c/c6a 218176 1 2024-02-28T16:31:27.077 INFO:tasks.workunit.client.0.smithi001.stdout:8/967: dwrite d4/d7/d22/d27/d4b/fd8 [0,4194304] 0 2024-02-28T16:31:27.077 INFO:tasks.workunit.client.1.smithi066.stdout:9/516: dread da/d3a/f47 [0,4194304] 0 2024-02-28T16:31:27.077 INFO:tasks.workunit.client.1.smithi066.stdout:5/603: getdents d3/d4/dc/dd 0 2024-02-28T16:31:27.078 INFO:tasks.workunit.client.1.smithi066.stdout:5/604: chown d3 212 1 2024-02-28T16:31:27.078 INFO:tasks.workunit.client.1.smithi066.stdout:6/602: chown d2/da/d19/d75 57481767 1 2024-02-28T16:31:27.082 INFO:tasks.workunit.client.1.smithi066.stdout:0/745: creat d2/d85/d8d/ff9 x:0 0 0 2024-02-28T16:31:27.084 INFO:tasks.workunit.client.1.smithi066.stdout:9/517: dread - da/d22/d46/d56/d8f/d28/d42/d68/f7a zero size 2024-02-28T16:31:27.085 INFO:tasks.workunit.client.0.smithi001.stdout:8/968: creat d4/d7/d22/d27/d4b/d84/f13c x:0 0 0 2024-02-28T16:31:27.085 INFO:tasks.workunit.client.1.smithi066.stdout:1/605: mkdir d2/d27/d2b/d37/d54/dc4 0 2024-02-28T16:31:27.088 INFO:tasks.workunit.client.1.smithi066.stdout:2/654: rename d0/d12/c18 to d0/d12/d3e/d9e/cd8 0 2024-02-28T16:31:27.090 INFO:tasks.workunit.client.1.smithi066.stdout:4/750: symlink d1/d18/d4e/da0/lf6 0 2024-02-28T16:31:27.091 INFO:tasks.workunit.client.0.smithi001.stdout:8/969: mkdir d4/d7/d22/d27/d4b/d13d 0 2024-02-28T16:31:27.091 INFO:tasks.workunit.client.1.smithi066.stdout:0/746: chown d2/d27/d41/d76/d93/d33/c91 45239 1 2024-02-28T16:31:27.093 INFO:tasks.workunit.client.0.smithi001.stdout:8/970: write d4/f13 [2326712,97471] 0 2024-02-28T16:31:27.096 INFO:tasks.workunit.client.0.smithi001.stdout:8/971: stat d4/d47/df4/f134 0 2024-02-28T16:31:27.097 INFO:tasks.workunit.client.1.smithi066.stdout:3/721: creat db/dc/d35/d3d/d76/d89/dcc/f105 x:0 0 0 2024-02-28T16:31:27.097 INFO:tasks.workunit.client.1.smithi066.stdout:9/518: chown da/d22/d46/d56/d8f/d28/d42/c79 0 1 2024-02-28T16:31:27.098 INFO:tasks.workunit.client.0.smithi001.stdout:2/935: dread d0/d2/d3/fca [0,4194304] 0 2024-02-28T16:31:27.099 INFO:tasks.workunit.client.1.smithi066.stdout:6/603: symlink d2/da/d19/d9c/ld4 0 2024-02-28T16:31:27.100 INFO:tasks.workunit.client.0.smithi001.stdout:5/977: dread dc/d1a/dba/fe9 [0,4194304] 0 2024-02-28T16:31:27.100 INFO:tasks.workunit.client.0.smithi001.stdout:8/972: mkdir d4/d7/d22/d27/d58/d94/dbc/d13e 0 2024-02-28T16:31:27.102 INFO:tasks.workunit.client.0.smithi001.stdout:2/936: symlink d0/d2/d3/de/d61/d12/de8/l13d 0 2024-02-28T16:31:27.104 INFO:tasks.workunit.client.0.smithi001.stdout:8/973: write d4/d7/d22/d27/d58/d94/db4/f133 [818600,26677] 0 2024-02-28T16:31:27.104 INFO:tasks.workunit.client.0.smithi001.stdout:5/978: creat dc/d1a/d7d/d87/df3/f139 x:0 0 0 2024-02-28T16:31:27.104 INFO:tasks.workunit.client.1.smithi066.stdout:4/751: fsync d1/d23/d25/d43/f8b 0 2024-02-28T16:31:27.106 INFO:tasks.workunit.client.0.smithi001.stdout:2/937: symlink d0/d2/d3/de/d22/l13e 0 2024-02-28T16:31:27.106 INFO:tasks.workunit.client.1.smithi066.stdout:3/722: chown db/d2a/d44/d48/d5d/fc1 1148605 1 2024-02-28T16:31:27.109 INFO:tasks.workunit.client.0.smithi001.stdout:5/979: creat dc/d12/d56/d96/db9/dc2/dca/d3c/da0/f13a x:0 0 0 2024-02-28T16:31:27.109 INFO:tasks.workunit.client.1.smithi066.stdout:1/606: link d2/d18/d19/f24 d2/d18/d7a/dbc/fc5 0 2024-02-28T16:31:27.110 INFO:tasks.workunit.client.0.smithi001.stdout:2/938: symlink d0/d2/d3/de/d61/d12/d46/l13f 0 2024-02-28T16:31:27.112 INFO:tasks.workunit.client.0.smithi001.stdout:2/939: read d0/d2/d3/f5 [7891695,115339] 0 2024-02-28T16:31:27.113 INFO:tasks.workunit.client.1.smithi066.stdout:9/519: mkdir da/d3a/d58/dab 0 2024-02-28T16:31:27.114 INFO:tasks.workunit.client.1.smithi066.stdout:0/747: getdents d2/d27/d41 0 2024-02-28T16:31:27.115 INFO:tasks.workunit.client.0.smithi001.stdout:2/940: truncate d0/d2/d3/de/f134 632222 0 2024-02-28T16:31:27.116 INFO:tasks.workunit.client.1.smithi066.stdout:4/752: mkdir d1/df7 0 2024-02-28T16:31:27.117 INFO:tasks.workunit.client.1.smithi066.stdout:6/604: creat d2/da/d19/d75/dbd/fd5 x:0 0 0 2024-02-28T16:31:27.118 INFO:tasks.workunit.client.0.smithi001.stdout:2/941: mknod d0/d2/d3/de/d61/dd5/c140 0 2024-02-28T16:31:27.119 INFO:tasks.workunit.client.1.smithi066.stdout:7/587: dwrite dc/de/d17/d91/f9b [4194304,4194304] 0 2024-02-28T16:31:27.119 INFO:tasks.workunit.client.0.smithi001.stdout:8/974: dwrite d4/d47/d8b/fe5 [4194304,4194304] 0 2024-02-28T16:31:27.123 INFO:tasks.workunit.client.1.smithi066.stdout:9/520: readlink da/d3a/d58/d97/la6 0 2024-02-28T16:31:27.123 INFO:tasks.workunit.client.0.smithi001.stdout:8/975: readlink d4/d32/d69/d9a/dcd/ldf 0 2024-02-28T16:31:27.125 INFO:tasks.workunit.client.0.smithi001.stdout:8/976: stat d4/d32/d69/d9a/cb6 0 2024-02-28T16:31:27.126 INFO:tasks.workunit.client.0.smithi001.stdout:5/980: dwrite dc/d1a/d52/f89 [0,4194304] 0 2024-02-28T16:31:27.128 INFO:tasks.workunit.client.1.smithi066.stdout:3/723: dread db/dc/d11/d1f/d78/fa4 [0,4194304] 0 2024-02-28T16:31:27.133 INFO:tasks.workunit.client.1.smithi066.stdout:8/597: dwrite d0/d6/dc/d16/d1a/d63/d3d/f62 [0,4194304] 0 2024-02-28T16:31:27.133 INFO:tasks.workunit.client.1.smithi066.stdout:3/724: readlink db/d2a/d44/d48/l8a 0 2024-02-28T16:31:27.134 INFO:tasks.workunit.client.1.smithi066.stdout:2/655: dread d0/d12/d3e/f4e [0,4194304] 0 2024-02-28T16:31:27.135 INFO:tasks.workunit.client.1.smithi066.stdout:6/605: mkdir d2/da/dcc/dd6 0 2024-02-28T16:31:27.135 INFO:tasks.workunit.client.1.smithi066.stdout:4/753: symlink d1/lf8 0 2024-02-28T16:31:27.135 INFO:tasks.workunit.client.0.smithi001.stdout:5/981: fsync dc/d12/d56/d96/db9/dc2/dca/d3c/f53 0 2024-02-28T16:31:27.139 INFO:tasks.workunit.client.1.smithi066.stdout:5/605: dwrite d3/d4/dc/dd/d2e/d18/f2c [0,4194304] 0 2024-02-28T16:31:27.146 INFO:tasks.workunit.client.0.smithi001.stdout:5/982: rename dc/d1a/d7d/d87/f8f to dc/d12/d56/d96/f13b 0 2024-02-28T16:31:27.148 INFO:tasks.workunit.client.0.smithi001.stdout:8/977: write d4/d47/d8b/fba [1569298,27980] 0 2024-02-28T16:31:27.148 INFO:tasks.workunit.client.0.smithi001.stdout:2/942: dwrite d0/d2/d3/de/d61/d12/d55/fb1 [0,4194304] 0 2024-02-28T16:31:27.150 INFO:tasks.workunit.client.1.smithi066.stdout:6/606: symlink d2/d2c/da1/d5d/ld7 0 2024-02-28T16:31:27.152 INFO:tasks.workunit.client.1.smithi066.stdout:1/607: dwrite d2/d18/d19/f1e [0,4194304] 0 2024-02-28T16:31:27.153 INFO:tasks.workunit.client.1.smithi066.stdout:3/725: read db/d13/d1a/fab [1302574,53497] 0 2024-02-28T16:31:27.157 INFO:tasks.workunit.client.1.smithi066.stdout:9/521: creat da/d3a/fac x:0 0 0 2024-02-28T16:31:27.158 INFO:tasks.workunit.client.0.smithi001.stdout:2/943: truncate d0/d2/d3/d36/d94/f11c 4456637 0 2024-02-28T16:31:27.159 INFO:tasks.workunit.client.1.smithi066.stdout:4/754: creat d1/d23/d25/d3e/d40/d9c/ff9 x:0 0 0 2024-02-28T16:31:27.160 INFO:tasks.workunit.client.0.smithi001.stdout:2/944: stat d0/d2/d3/de/d22/f52 0 2024-02-28T16:31:27.164 INFO:tasks.workunit.client.0.smithi001.stdout:5/983: dwrite dc/f108 [0,4194304] 0 2024-02-28T16:31:27.165 INFO:tasks.workunit.client.0.smithi001.stdout:2/945: creat d0/d2/d3/d36/f141 x:0 0 0 2024-02-28T16:31:27.168 INFO:tasks.workunit.client.1.smithi066.stdout:6/607: mknod d2/da/d19/cd8 0 2024-02-28T16:31:27.169 INFO:tasks.workunit.client.0.smithi001.stdout:5/984: rename dc/df/d79/d98 to dc/d12/d56/d96/db9/ddd/d13c 0 2024-02-28T16:31:27.170 INFO:tasks.workunit.client.1.smithi066.stdout:8/598: write d0/d6/f5d [634457,128340] 0 2024-02-28T16:31:27.172 INFO:tasks.workunit.client.0.smithi001.stdout:5/985: chown dc/d12/d56/d96/db9/dc2/dca/d3c/f47 8487 1 2024-02-28T16:31:27.173 INFO:tasks.workunit.client.1.smithi066.stdout:9/522: symlink da/d22/d46/d56/d8f/d28/d63/lad 0 2024-02-28T16:31:27.176 INFO:tasks.workunit.client.0.smithi001.stdout:5/986: mkdir dc/d1a/d7d/d13d 0 2024-02-28T16:31:27.177 INFO:tasks.workunit.client.0.smithi001.stdout:8/978: read d4/d47/d8b/fe5 [1661288,32484] 0 2024-02-28T16:31:27.180 INFO:tasks.workunit.client.1.smithi066.stdout:0/748: truncate d2/d85/f9f 495105 0 2024-02-28T16:31:27.181 INFO:tasks.workunit.client.1.smithi066.stdout:2/656: creat d0/d12/d1a/d23/d29/d21/fd9 x:0 0 0 2024-02-28T16:31:27.181 INFO:tasks.workunit.client.1.smithi066.stdout:7/588: link dc/de/d17/f19 dc/de/d17/d1e/d45/fae 0 2024-02-28T16:31:27.182 INFO:tasks.workunit.client.1.smithi066.stdout:3/726: dread db/d13/d14/d84/f9c [0,4194304] 0 2024-02-28T16:31:27.182 INFO:tasks.workunit.client.0.smithi001.stdout:5/987: mknod dc/d12/d56/d96/db9/dc2/dca/d3c/c13e 0 2024-02-28T16:31:27.184 INFO:tasks.workunit.client.1.smithi066.stdout:7/589: chown dc/d29/l39 63809 1 2024-02-28T16:31:27.185 INFO:tasks.workunit.client.1.smithi066.stdout:2/657: write d0/d12/d2e/f9d [2585779,113138] 0 2024-02-28T16:31:27.187 INFO:tasks.workunit.client.1.smithi066.stdout:7/590: chown dc/d56/l90 994 1 2024-02-28T16:31:27.189 INFO:tasks.workunit.client.1.smithi066.stdout:2/658: chown d0/d12/d1a/d23/d5d/fb5 48707563 1 2024-02-28T16:31:27.189 INFO:tasks.workunit.client.0.smithi001.stdout:6/967: sync 2024-02-28T16:31:27.189 INFO:tasks.workunit.client.0.smithi001.stdout:0/973: sync 2024-02-28T16:31:27.190 INFO:tasks.workunit.client.1.smithi066.stdout:9/523: fsync da/d22/d46/d56/d8f/d2d/f4c 0 2024-02-28T16:31:27.191 INFO:tasks.workunit.client.1.smithi066.stdout:2/659: readlink d0/d12/d1a/d39/lab 0 2024-02-28T16:31:27.193 INFO:tasks.workunit.client.1.smithi066.stdout:1/608: creat d2/d27/d2b/fc6 x:0 0 0 2024-02-28T16:31:27.195 INFO:tasks.workunit.client.0.smithi001.stdout:0/974: stat d18/d22/d76/d7a 0 2024-02-28T16:31:27.196 INFO:tasks.workunit.client.1.smithi066.stdout:3/727: creat db/dc/d35/d3d/d76/d89/dcc/f106 x:0 0 0 2024-02-28T16:31:27.196 INFO:tasks.workunit.client.0.smithi001.stdout:6/968: rename d9/db/d5f/dd4/dfb/f12b to d9/d29/d6a/f149 0 2024-02-28T16:31:27.197 INFO:tasks.workunit.client.1.smithi066.stdout:7/591: symlink dc/d29/d92/laf 0 2024-02-28T16:31:27.199 INFO:tasks.workunit.client.0.smithi001.stdout:6/969: chown d9/db/d5f/dd4/dfb/d121/fb9 0 1 2024-02-28T16:31:27.199 INFO:tasks.workunit.client.0.smithi001.stdout:0/975: chown d18/d22/d2f/fa4 29458 1 2024-02-28T16:31:27.200 INFO:tasks.workunit.client.1.smithi066.stdout:2/660: mknod d0/d12/d1a/d39/d7c/cda 0 2024-02-28T16:31:27.201 INFO:tasks.workunit.client.0.smithi001.stdout:6/970: dread - d9/db/dd9/f116 zero size 2024-02-28T16:31:27.203 INFO:tasks.workunit.client.1.smithi066.stdout:8/599: link d0/db/d12/d1d/d25/f74 d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/fc4 0 2024-02-28T16:31:27.204 INFO:tasks.workunit.client.0.smithi001.stdout:6/971: getdents d9/d39/dc4/df4 0 2024-02-28T16:31:27.204 INFO:tasks.workunit.client.0.smithi001.stdout:0/976: read fd [714499,73645] 0 2024-02-28T16:31:27.206 INFO:tasks.workunit.client.0.smithi001.stdout:6/972: chown d9/db/d10/l147 12619 1 2024-02-28T16:31:27.207 INFO:tasks.workunit.client.1.smithi066.stdout:2/661: write d0/d12/d1a/d23/d5d/fbb [949338,73157] 0 2024-02-28T16:31:27.207 INFO:tasks.workunit.client.0.smithi001.stdout:0/977: rmdir d18/d22/d76 39 2024-02-28T16:31:27.208 INFO:tasks.workunit.client.1.smithi066.stdout:7/592: symlink dc/d29/d92/da6/lb0 0 2024-02-28T16:31:27.210 INFO:tasks.workunit.client.1.smithi066.stdout:0/749: link d2/d27/l4d d2/d27/d41/d76/d93/d33/d3f/d8e/lfa 0 2024-02-28T16:31:27.212 INFO:tasks.workunit.client.1.smithi066.stdout:8/600: creat d0/d6/dc/d16/d4c/d6c/d6f/da7/fc5 x:0 0 0 2024-02-28T16:31:27.214 INFO:tasks.workunit.client.0.smithi001.stdout:0/978: mkdir d18/d22/d2b/d36/d46/d13b 0 2024-02-28T16:31:27.219 INFO:tasks.workunit.client.1.smithi066.stdout:7/593: creat dc/d29/d92/da6/fb1 x:0 0 0 2024-02-28T16:31:27.220 INFO:tasks.workunit.client.0.smithi001.stdout:5/988: chown dc/d12/d56/d96/db9/dc2/dca/d3c/f47 66405 1 2024-02-28T16:31:27.221 INFO:tasks.workunit.client.0.smithi001.stdout:2/946: dwrite d0/d2/d3/d73/f85 [0,4194304] 0 2024-02-28T16:31:27.221 INFO:tasks.workunit.client.0.smithi001.stdout:8/979: dwrite d4/d7/da8/d89/fce [0,4194304] 0 2024-02-28T16:31:27.223 INFO:tasks.workunit.client.0.smithi001.stdout:0/979: creat d18/d22/d2b/d36/d46/f13c x:0 0 0 2024-02-28T16:31:27.226 INFO:tasks.workunit.client.0.smithi001.stdout:8/980: dread - d4/d47/df4/df5/f118 zero size 2024-02-28T16:31:27.226 INFO:tasks.workunit.client.0.smithi001.stdout:2/947: stat d0/d2/d3/de/d61/d12/d55/f114 0 2024-02-28T16:31:27.230 INFO:tasks.workunit.client.1.smithi066.stdout:8/601: symlink d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/lc6 0 2024-02-28T16:31:27.231 INFO:tasks.workunit.client.0.smithi001.stdout:6/973: truncate d9/d29/d6a/ded/f10a 826499 0 2024-02-28T16:31:27.231 INFO:tasks.workunit.client.0.smithi001.stdout:5/989: dread - dc/d1a/d21/fcb zero size 2024-02-28T16:31:27.232 INFO:tasks.workunit.client.0.smithi001.stdout:0/980: mkdir d18/d22/d2f/d1c/d90/d9f/d13d 0 2024-02-28T16:31:27.233 INFO:tasks.workunit.client.0.smithi001.stdout:2/948: rmdir d0/d12d 39 2024-02-28T16:31:27.234 INFO:tasks.workunit.client.0.smithi001.stdout:8/981: truncate d4/d47/df4/f10e 1051104 0 2024-02-28T16:31:27.235 INFO:tasks.workunit.client.0.smithi001.stdout:6/974: symlink d9/d29/d6a/ded/d105/d106/l14a 0 2024-02-28T16:31:27.236 INFO:tasks.workunit.client.1.smithi066.stdout:7/594: creat dc/d29/d92/da6/fb2 x:0 0 0 2024-02-28T16:31:27.238 INFO:tasks.workunit.client.1.smithi066.stdout:2/662: creat d0/d12/fdb x:0 0 0 2024-02-28T16:31:27.241 INFO:tasks.workunit.client.0.smithi001.stdout:8/982: creat d4/d7/d22/d27/d8e/f13f x:0 0 0 2024-02-28T16:31:27.243 INFO:tasks.workunit.client.0.smithi001.stdout:5/990: truncate dc/df/f11 573195 0 2024-02-28T16:31:27.244 INFO:tasks.workunit.client.1.smithi066.stdout:9/524: dread da/d3a/f13 [4194304,4194304] 0 2024-02-28T16:31:27.245 INFO:tasks.workunit.client.1.smithi066.stdout:5/606: dwrite d3/d4/dc/dd/d41/d7a/dac/d60/fb2 [0,4194304] 0 2024-02-28T16:31:27.248 INFO:tasks.workunit.client.1.smithi066.stdout:8/602: dread - d0/d6/dc/d16/d1a/d63/d3d/fab zero size 2024-02-28T16:31:27.248 INFO:tasks.workunit.client.0.smithi001.stdout:8/983: dread - d4/d7/d22/d27/d4b/d77/f11f zero size 2024-02-28T16:31:27.249 INFO:tasks.workunit.client.0.smithi001.stdout:2/949: chown d0/d2/d3/d73/l123 124859217 1 2024-02-28T16:31:27.249 INFO:tasks.workunit.client.0.smithi001.stdout:8/984: truncate d4/d7/d22/d27/d58/f87 1246996 0 2024-02-28T16:31:27.249 INFO:tasks.workunit.client.1.smithi066.stdout:0/750: dwrite d2/d27/d41/d76/d93/db2/db5/d67/d94/de4/da1/fd2 [0,4194304] 0 2024-02-28T16:31:27.250 INFO:tasks.workunit.client.1.smithi066.stdout:2/663: chown d0/d12/d1a/f2b 87 1 2024-02-28T16:31:27.255 INFO:tasks.workunit.client.0.smithi001.stdout:8/985: fdatasync d4/f59 0 2024-02-28T16:31:27.257 INFO:tasks.workunit.client.1.smithi066.stdout:1/609: dread d2/d27/d2b/d37/d48/f81 [0,4194304] 0 2024-02-28T16:31:27.258 INFO:tasks.workunit.client.0.smithi001.stdout:2/950: creat d0/d2/d3/de/d61/d12/d46/f142 x:0 0 0 2024-02-28T16:31:27.260 INFO:tasks.workunit.client.1.smithi066.stdout:0/751: write d2/d27/d41/d76/d93/d33/f62 [5176118,83355] 0 2024-02-28T16:31:27.261 INFO:tasks.workunit.client.1.smithi066.stdout:2/664: write d0/d12/d1a/d39/d7c/d87/fa1 [239618,103964] 0 2024-02-28T16:31:27.266 INFO:tasks.workunit.client.1.smithi066.stdout:5/607: mkdir d3/d4/dc/d2f/dc0 0 2024-02-28T16:31:27.267 INFO:tasks.workunit.client.0.smithi001.stdout:8/986: write d4/d7/d22/d27/d4b/d77/f11f [364343,83646] 0 2024-02-28T16:31:27.272 INFO:tasks.workunit.client.1.smithi066.stdout:1/610: truncate d2/d27/d2b/f39 46656 0 2024-02-28T16:31:27.272 INFO:tasks.workunit.client.0.smithi001.stdout:5/991: truncate dc/d12/d56/d96/db9/dc2/dca/d3c/da0/f13a 973569 0 2024-02-28T16:31:27.273 INFO:tasks.workunit.client.1.smithi066.stdout:2/665: chown d0/d12/d1a/d23/d29/d21/d2f/d67/c81 91114 1 2024-02-28T16:31:27.274 INFO:tasks.workunit.client.0.smithi001.stdout:6/975: dwrite d9/db/d5f/d62/d127/f136 [0,4194304] 0 2024-02-28T16:31:27.275 INFO:tasks.workunit.client.1.smithi066.stdout:3/728: dread db/d2a/d44/d48/d5d/fa9 [0,4194304] 0 2024-02-28T16:31:27.275 INFO:tasks.workunit.client.0.smithi001.stdout:8/987: fdatasync d4/d7/f35 0 2024-02-28T16:31:27.280 INFO:tasks.workunit.client.1.smithi066.stdout:3/729: mkdir db/dc/d11/d107 0 2024-02-28T16:31:27.281 INFO:tasks.workunit.client.1.smithi066.stdout:2/666: write d0/d12/d1a/d23/d5d/fbd [894788,75381] 0 2024-02-28T16:31:27.281 INFO:tasks.workunit.client.0.smithi001.stdout:5/992: symlink dc/d1a/d7d/d13d/l13f 0 2024-02-28T16:31:27.282 INFO:tasks.workunit.client.1.smithi066.stdout:1/611: rename d2/d27/d2b/d37 to d2/d27/dc7 0 2024-02-28T16:31:27.285 INFO:tasks.workunit.client.0.smithi001.stdout:5/993: chown dc/d1a/d21/d11b/d12b 2392 1 2024-02-28T16:31:27.286 INFO:tasks.workunit.client.1.smithi066.stdout:3/730: mknod db/d2a/d44/d59/c108 0 2024-02-28T16:31:27.287 INFO:tasks.workunit.client.1.smithi066.stdout:1/612: fsync d2/d27/d2b/fc6 0 2024-02-28T16:31:27.289 INFO:tasks.workunit.client.0.smithi001.stdout:0/981: dwrite d18/d22/d2b/d36/d46/df5/f9e [0,4194304] 0 2024-02-28T16:31:27.290 INFO:tasks.workunit.client.0.smithi001.stdout:5/994: dread - dc/d1a/d21/fcb zero size 2024-02-28T16:31:27.291 INFO:tasks.workunit.client.1.smithi066.stdout:8/603: dwrite d0/d6/dc/d16/d1a/f88 [0,4194304] 0 2024-02-28T16:31:27.292 INFO:tasks.workunit.client.1.smithi066.stdout:7/595: truncate dc/de/d17/d1e/f20 1595598 0 2024-02-28T16:31:27.293 INFO:tasks.workunit.client.0.smithi001.stdout:2/951: write d0/d2/d3/d36/d94/ff8 [232463,124604] 0 2024-02-28T16:31:27.295 INFO:tasks.workunit.client.1.smithi066.stdout:9/525: dwrite da/d3a/fac [0,4194304] 0 2024-02-28T16:31:27.295 INFO:tasks.workunit.client.1.smithi066.stdout:3/731: mknod db/d2a/c109 0 2024-02-28T16:31:27.296 INFO:tasks.workunit.client.1.smithi066.stdout:2/667: mkdir d0/d12/d1a/d23/ddc 0 2024-02-28T16:31:27.303 INFO:tasks.workunit.client.1.smithi066.stdout:1/613: rmdir d2/d18/d7a 39 2024-02-28T16:31:27.304 INFO:tasks.workunit.client.0.smithi001.stdout:5/995: mkdir dc/d12/d56/d96/db9/d140 0 2024-02-28T16:31:27.305 INFO:tasks.workunit.client.1.smithi066.stdout:3/732: mknod db/dc/d35/c10a 0 2024-02-28T16:31:27.306 INFO:tasks.workunit.client.0.smithi001.stdout:2/952: write d0/d2/d3/de/d61/d12/d55/db2/fd3 [2668424,3986] 0 2024-02-28T16:31:27.306 INFO:tasks.workunit.client.0.smithi001.stdout:8/988: dwrite d4/d7/d22/d27/d4b/d84/f13c [0,4194304] 0 2024-02-28T16:31:27.310 INFO:tasks.workunit.client.0.smithi001.stdout:5/996: chown dc/d12/d56/d96/db9/dc2/dca/fb6 121275 1 2024-02-28T16:31:27.311 INFO:tasks.workunit.client.0.smithi001.stdout:8/989: chown d4/d32/d69/f6f 450588 1 2024-02-28T16:31:27.313 INFO:tasks.workunit.client.1.smithi066.stdout:8/604: unlink d0/d6/dc/d16/d1a/d63/d58/f48 0 2024-02-28T16:31:27.313 INFO:tasks.workunit.client.0.smithi001.stdout:5/997: fdatasync dc/df/f81 0 2024-02-28T16:31:27.315 INFO:tasks.workunit.client.0.smithi001.stdout:2/953: truncate d0/d2/d3/de/d8a/fbd 3751649 0 2024-02-28T16:31:27.317 INFO:tasks.workunit.client.1.smithi066.stdout:7/596: dread dc/d29/f2e [0,4194304] 0 2024-02-28T16:31:27.317 INFO:tasks.workunit.client.1.smithi066.stdout:2/668: rename d0/d12/d2e/d3b/d66/f92 to d0/d12/d1a/d39/d7c/fdd 0 2024-02-28T16:31:27.319 INFO:tasks.workunit.client.1.smithi066.stdout:8/605: fsync d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/fa4 0 2024-02-28T16:31:27.330 INFO:tasks.workunit.client.1.smithi066.stdout:2/669: write d0/d12/d1a/d23/d5d/fbb [1562992,91956] 0 2024-02-28T16:31:27.331 INFO:tasks.workunit.client.1.smithi066.stdout:8/606: creat d0/d8/fc7 x:0 0 0 2024-02-28T16:31:27.332 INFO:tasks.workunit.client.0.smithi001.stdout:8/990: dwrite d4/d7/d22/d27/d58/d94/db4/f133 [0,4194304] 0 2024-02-28T16:31:27.335 INFO:tasks.workunit.client.1.smithi066.stdout:7/597: mknod dc/d29/d92/cb3 0 2024-02-28T16:31:27.337 INFO:tasks.workunit.client.1.smithi066.stdout:2/670: fsync d0/d12/d1a/d23/fa6 0 2024-02-28T16:31:27.338 INFO:tasks.workunit.client.1.smithi066.stdout:7/598: write f5 [8898468,98085] 0 2024-02-28T16:31:27.341 INFO:tasks.workunit.client.0.smithi001.stdout:6/976: dread d9/db/d10/fe8 [0,4194304] 0 2024-02-28T16:31:27.342 INFO:tasks.workunit.client.1.smithi066.stdout:8/607: rmdir d0/db/d12/d1d/d25/d72 39 2024-02-28T16:31:27.344 INFO:tasks.workunit.client.0.smithi001.stdout:0/982: dread d18/d22/d2f/ff1 [0,4194304] 0 2024-02-28T16:31:27.346 INFO:tasks.workunit.client.0.smithi001.stdout:6/977: creat d9/d39/f14b x:0 0 0 2024-02-28T16:31:27.348 INFO:tasks.workunit.client.0.smithi001.stdout:0/983: readlink d18/d22/d76/d93/lfe 0 2024-02-28T16:31:27.349 INFO:tasks.workunit.client.1.smithi066.stdout:8/608: fsync d0/d6/ff 0 2024-02-28T16:31:27.357 INFO:tasks.workunit.client.1.smithi066.stdout:2/671: chown d0/d12/d1a/d23/d29/d21/f75 85525045 1 2024-02-28T16:31:27.357 INFO:tasks.workunit.client.1.smithi066.stdout:1/614: dwrite d2/d27/d2b/d43/f9b [0,4194304] 0 2024-02-28T16:31:27.358 INFO:tasks.workunit.client.1.smithi066.stdout:2/672: chown d0/lb9 991355647 1 2024-02-28T16:31:27.358 INFO:tasks.workunit.client.1.smithi066.stdout:9/526: dwrite da/d22/d46/d56/d8f/d2d/f62 [0,4194304] 0 2024-02-28T16:31:27.359 INFO:tasks.workunit.client.1.smithi066.stdout:7/599: rename c7 to dc/cb4 0 2024-02-28T16:31:27.361 INFO:tasks.workunit.client.1.smithi066.stdout:8/609: mkdir d0/dc8 0 2024-02-28T16:31:27.361 INFO:tasks.workunit.client.1.smithi066.stdout:7/600: dread - dc/d56/fad zero size 2024-02-28T16:31:27.363 INFO:tasks.workunit.client.1.smithi066.stdout:9/527: chown da/d22/d46/d56/d8f/d28/d42/c79 9586647 1 2024-02-28T16:31:27.367 INFO:tasks.workunit.client.1.smithi066.stdout:3/733: dwrite db/d13/d14/f15 [0,4194304] 0 2024-02-28T16:31:27.368 INFO:tasks.workunit.client.0.smithi001.stdout:0/984: symlink d18/d22/d2b/d36/d48/d11a/d7e/l13e 0 2024-02-28T16:31:27.370 INFO:tasks.workunit.client.1.smithi066.stdout:9/528: read da/d22/f51 [71443,120432] 0 2024-02-28T16:31:27.374 INFO:tasks.workunit.client.1.smithi066.stdout:7/601: creat dc/de/d17/d37/d47/d88/fb5 x:0 0 0 2024-02-28T16:31:27.379 INFO:tasks.workunit.client.1.smithi066.stdout:8/610: mkdir d0/d6/dc/d16/d4c/d6c/d6f/da7/dc9 0 2024-02-28T16:31:27.379 INFO:tasks.workunit.client.0.smithi001.stdout:2/954: write d0/d2/d3/de/d61/d12/d55/db2/f100 [135041,52735] 0 2024-02-28T16:31:27.381 INFO:tasks.workunit.client.0.smithi001.stdout:8/991: write d4/d7/d22/f3f [5817898,127535] 0 2024-02-28T16:31:27.384 INFO:tasks.workunit.client.1.smithi066.stdout:8/611: fsync d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/fa5 0 2024-02-28T16:31:27.385 INFO:tasks.workunit.client.1.smithi066.stdout:9/529: symlink da/d3a/d58/lae 0 2024-02-28T16:31:27.394 INFO:tasks.workunit.client.1.smithi066.stdout:3/734: read db/d2a/d44/d48/d5d/f91 [168782,102577] 0 2024-02-28T16:31:27.401 INFO:tasks.workunit.client.1.smithi066.stdout:3/735: readlink db/d2a/d44/d48/d52/d77/ldb 0 2024-02-28T16:31:27.402 INFO:tasks.workunit.client.0.smithi001.stdout:6/978: dwrite d9/db/d19/f37 [0,4194304] 0 2024-02-28T16:31:27.403 INFO:tasks.workunit.client.1.smithi066.stdout:7/602: rename dc/de/d17/d1e/c3d to dc/de/d17/d37/d47/cb6 0 2024-02-28T16:31:27.404 INFO:tasks.workunit.client.0.smithi001.stdout:5/998: dread dc/d1a/f8d [0,4194304] 0 2024-02-28T16:31:27.405 INFO:tasks.workunit.client.1.smithi066.stdout:1/615: dwrite d2/d18/d7a/d8d/f8f [4194304,4194304] 0 2024-02-28T16:31:27.413 INFO:tasks.workunit.client.1.smithi066.stdout:3/736: symlink db/d7a/l10b 0 2024-02-28T16:31:27.413 INFO:tasks.workunit.client.1.smithi066.stdout:7/603: mkdir dc/de/d17/d1e/da1/db7 0 2024-02-28T16:31:27.413 INFO:tasks.workunit.client.1.smithi066.stdout:8/612: creat d0/d6/dc/d16/d4c/d6c/d6f/da7/dc9/fca x:0 0 0 2024-02-28T16:31:27.415 INFO:tasks.workunit.client.1.smithi066.stdout:2/673: truncate d0/d12/d1a/d23/d5d/fbb 1288793 0 2024-02-28T16:31:27.416 INFO:tasks.workunit.client.1.smithi066.stdout:1/616: rename d2/d18/d19/dbe to d2/d27/d2b/d43/dc8 0 2024-02-28T16:31:27.417 INFO:tasks.workunit.client.1.smithi066.stdout:8/613: write d0/d6/f1f [3415857,120921] 0 2024-02-28T16:31:27.420 INFO:tasks.workunit.client.1.smithi066.stdout:2/674: write d0/d12/d2e/d7a/fba [701858,106932] 0 2024-02-28T16:31:27.427 INFO:tasks.workunit.client.1.smithi066.stdout:7/604: creat dc/de/fb8 x:0 0 0 2024-02-28T16:31:27.427 INFO:tasks.workunit.client.1.smithi066.stdout:1/617: symlink d2/d18/d6b/lc9 0 2024-02-28T16:31:27.428 INFO:tasks.workunit.client.1.smithi066.stdout:3/737: fdatasync db/ff 0 2024-02-28T16:31:27.428 INFO:tasks.workunit.client.0.smithi001.stdout:8/992: creat d4/d47/df4/f140 x:0 0 0 2024-02-28T16:31:27.428 INFO:tasks.workunit.client.0.smithi001.stdout:5/999: dread dc/f27 [0,4194304] 0 2024-02-28T16:31:27.429 INFO:tasks.workunit.client.1.smithi066.stdout:2/675: creat d0/dd7/fde x:0 0 0 2024-02-28T16:31:27.429 INFO:tasks.workunit.client.1.smithi066.stdout:8/614: truncate d0/db/d12/d1d/d25/d65/d93/f99 1013467 0 2024-02-28T16:31:27.430 INFO:tasks.workunit.client.0.smithi001.stdout:6/979: write d9/d29/d6a/f149 [280459,58437] 0 2024-02-28T16:31:27.431 INFO:tasks.workunit.client.1.smithi066.stdout:8/615: stat d0/d6/f9a 0 2024-02-28T16:31:27.434 INFO:tasks.workunit.client.0.smithi001.stdout:6/980: write d9/db/d5f/f65 [8318666,47798] 0 2024-02-28T16:31:27.434 INFO:tasks.workunit.client.1.smithi066.stdout:8/616: fsync d0/d6/f5d 0 2024-02-28T16:31:27.435 INFO:tasks.workunit.client.0.smithi001.stdout:8/993: unlink d4/d7/d22/d27/d58/f71 0 2024-02-28T16:31:27.443 INFO:tasks.workunit.client.1.smithi066.stdout:1/618: symlink d2/d27/lca 0 2024-02-28T16:31:27.445 INFO:tasks.workunit.client.1.smithi066.stdout:8/617: dread - d0/d6/dc/f6b zero size 2024-02-28T16:31:27.445 INFO:tasks.workunit.client.0.smithi001.stdout:6/981: rename d9/d29/l98 to d9/d29/d6e/d53/l14c 0 2024-02-28T16:31:27.448 INFO:tasks.workunit.client.0.smithi001.stdout:6/982: fdatasync d9/db/d5f/d89/ff8 0 2024-02-28T16:31:27.450 INFO:tasks.workunit.client.1.smithi066.stdout:3/738: creat db/dc/d11/d1f/f10c x:0 0 0 2024-02-28T16:31:27.452 INFO:tasks.workunit.client.0.smithi001.stdout:6/983: creat d9/db/d5f/d89/d8a/f14d x:0 0 0 2024-02-28T16:31:27.453 INFO:tasks.workunit.client.0.smithi001.stdout:6/984: readlink d9/d29/d6e/d4e/d96/lf1 0 2024-02-28T16:31:27.455 INFO:tasks.workunit.client.1.smithi066.stdout:7/605: dwrite dc/de/d17/d1e/d45/f8f [0,4194304] 0 2024-02-28T16:31:27.456 INFO:tasks.workunit.client.0.smithi001.stdout:6/985: creat d9/db/d5f/dd4/dfb/f14e x:0 0 0 2024-02-28T16:31:27.459 INFO:tasks.workunit.client.0.smithi001.stdout:6/986: mknod d9/db/d5f/d89/c14f 0 2024-02-28T16:31:27.461 INFO:tasks.workunit.client.1.smithi066.stdout:1/619: dread d2/d18/d19/f89 [0,4194304] 0 2024-02-28T16:31:27.463 INFO:tasks.workunit.client.1.smithi066.stdout:7/606: mkdir dc/de/d17/d91/db9 0 2024-02-28T16:31:27.464 INFO:tasks.workunit.client.1.smithi066.stdout:3/739: mknod db/d13/d14/dda/d102/c10d 0 2024-02-28T16:31:27.468 INFO:tasks.workunit.client.1.smithi066.stdout:7/607: stat dc/de/d17/d91 0 2024-02-28T16:31:27.470 INFO:tasks.workunit.client.1.smithi066.stdout:1/620: mknod d2/d18/d7a/d8d/ccb 0 2024-02-28T16:31:27.472 INFO:tasks.workunit.client.1.smithi066.stdout:8/618: dwrite d0/db/d12/d1d/d25/d65/f84 [0,4194304] 0 2024-02-28T16:31:27.472 INFO:tasks.workunit.client.1.smithi066.stdout:7/608: readlink dc/de/d17/d1e/d45/l5f 0 2024-02-28T16:31:27.474 INFO:tasks.workunit.client.1.smithi066.stdout:8/619: readlink d0/d6/dc/l36 0 2024-02-28T16:31:27.475 INFO:tasks.workunit.client.0.smithi001.stdout:0/985: dwrite d18/d22/d2b/d36/d46/df5/dbc/fd4 [0,4194304] 0 2024-02-28T16:31:27.476 INFO:tasks.workunit.client.1.smithi066.stdout:1/621: creat d2/d27/d2b/d34/fcc x:0 0 0 2024-02-28T16:31:27.477 INFO:tasks.workunit.client.1.smithi066.stdout:2/676: dwrite d0/d12/d2e/f59 [0,4194304] 0 2024-02-28T16:31:27.480 INFO:tasks.workunit.client.1.smithi066.stdout:7/609: unlink dc/de/d17/d37/d47/f81 0 2024-02-28T16:31:27.483 INFO:tasks.workunit.client.0.smithi001.stdout:2/955: dwrite d0/d2/d126/d84/ff9 [0,4194304] 0 2024-02-28T16:31:27.486 INFO:tasks.workunit.client.1.smithi066.stdout:1/622: creat d2/d27/d2b/d43/d46/fcd x:0 0 0 2024-02-28T16:31:27.488 INFO:tasks.workunit.client.0.smithi001.stdout:0/986: fdatasync d18/d22/d2b/d36/d48/f8a 0 2024-02-28T16:31:27.489 INFO:tasks.workunit.client.1.smithi066.stdout:3/740: dwrite db/d2a/d44/d48/f6c [0,4194304] 0 2024-02-28T16:31:27.489 INFO:tasks.workunit.client.0.smithi001.stdout:6/987: write d9/db/d5f/d89/fcb [906665,66203] 0 2024-02-28T16:31:27.492 INFO:tasks.workunit.client.1.smithi066.stdout:4/755: sync 2024-02-28T16:31:27.493 INFO:tasks.workunit.client.0.smithi001.stdout:2/956: chown d0/d2/d3/de/l7b 27896 1 2024-02-28T16:31:27.494 INFO:tasks.workunit.client.1.smithi066.stdout:7/610: mkdir dc/de/d17/d91/dba 0 2024-02-28T16:31:27.495 INFO:tasks.workunit.client.1.smithi066.stdout:4/756: chown d1/d23/d25/d43 0 1 2024-02-28T16:31:27.497 INFO:tasks.workunit.client.0.smithi001.stdout:6/988: fdatasync d9/de/f2c 0 2024-02-28T16:31:27.498 INFO:tasks.workunit.client.1.smithi066.stdout:7/611: stat dc/de/d17/d1e/l8c 0 2024-02-28T16:31:27.505 INFO:tasks.workunit.client.1.smithi066.stdout:1/623: unlink d2/d27/fc2 0 2024-02-28T16:31:27.505 INFO:tasks.workunit.client.0.smithi001.stdout:2/957: creat d0/f143 x:0 0 0 2024-02-28T16:31:27.505 INFO:tasks.workunit.client.0.smithi001.stdout:0/987: getdents d18/d22/d2b/d36 0 2024-02-28T16:31:27.507 INFO:tasks.workunit.client.0.smithi001.stdout:6/989: rename d9/db/d5f/d62/d83/f99 to d9/db/d5f/d62/d127/f150 0 2024-02-28T16:31:27.512 INFO:tasks.workunit.client.1.smithi066.stdout:3/741: dread db/d13/d14/d2d/f3e [0,4194304] 0 2024-02-28T16:31:27.512 INFO:tasks.workunit.client.0.smithi001.stdout:6/990: symlink d9/db/dd9/l151 0 2024-02-28T16:31:27.513 INFO:tasks.workunit.client.1.smithi066.stdout:1/624: mkdir d2/d27/dc7/d48/dce 0 2024-02-28T16:31:27.514 INFO:tasks.workunit.client.1.smithi066.stdout:4/757: creat d1/d23/ffa x:0 0 0 2024-02-28T16:31:27.517 INFO:tasks.workunit.client.0.smithi001.stdout:6/991: fsync d9/db/d5f/d62/d83/f109 0 2024-02-28T16:31:27.519 INFO:tasks.workunit.client.0.smithi001.stdout:2/958: link d0/d2/d3/de/d22/c9f d0/d2/d3/d36/d94/c144 0 2024-02-28T16:31:27.520 INFO:tasks.workunit.client.1.smithi066.stdout:3/742: write db/dc/d35/d3d/d76/f4a [2572729,104954] 0 2024-02-28T16:31:27.521 INFO:tasks.workunit.client.0.smithi001.stdout:6/992: symlink d9/d29/d6e/d139/l152 0 2024-02-28T16:31:27.521 INFO:tasks.workunit.client.1.smithi066.stdout:1/625: creat d2/d27/dc7/d54/dc4/fcf x:0 0 0 2024-02-28T16:31:27.523 INFO:tasks.workunit.client.1.smithi066.stdout:1/626: chown d2/d27/d2b/d43/d46/f6c 766359 1 2024-02-28T16:31:27.523 INFO:tasks.workunit.client.0.smithi001.stdout:2/959: mknod d0/d2/d126/c145 0 2024-02-28T16:31:27.524 INFO:tasks.workunit.client.0.smithi001.stdout:6/993: fdatasync d9/de/dbd/fdc 0 2024-02-28T16:31:27.524 INFO:tasks.workunit.client.1.smithi066.stdout:3/743: mknod db/d13/d14/dda/c10e 0 2024-02-28T16:31:27.525 INFO:tasks.workunit.client.0.smithi001.stdout:2/960: stat d0/d2/d3/de/d22/l12f 0 2024-02-28T16:31:27.527 INFO:tasks.workunit.client.0.smithi001.stdout:6/994: creat d9/db/d5f/d69/f153 x:0 0 0 2024-02-28T16:31:27.530 INFO:tasks.workunit.client.0.smithi001.stdout:6/995: mkdir d9/d29/d6a/d154 0 2024-02-28T16:31:27.530 INFO:tasks.workunit.client.1.smithi066.stdout:8/620: dread d0/db/d15/f37 [0,4194304] 0 2024-02-28T16:31:27.530 INFO:tasks.workunit.client.1.smithi066.stdout:3/744: write db/dc/d35/d3d/d76/d89/f9f [51164,3050] 0 2024-02-28T16:31:27.533 INFO:tasks.workunit.client.0.smithi001.stdout:6/996: creat d9/db/d5f/dd4/dfb/f155 x:0 0 0 2024-02-28T16:31:27.534 INFO:tasks.workunit.client.1.smithi066.stdout:1/627: truncate d2/d27/d2b/f39 803061 0 2024-02-28T16:31:27.535 INFO:tasks.workunit.client.0.smithi001.stdout:6/997: chown d9/db/d10/la1 5310 1 2024-02-28T16:31:27.537 INFO:tasks.workunit.client.1.smithi066.stdout:8/621: creat d0/db/d15/d95/fcb x:0 0 0 2024-02-28T16:31:27.538 INFO:tasks.workunit.client.0.smithi001.stdout:6/998: read d9/d39/f3a [523165,70775] 0 2024-02-28T16:31:27.542 INFO:tasks.workunit.client.1.smithi066.stdout:4/758: dwrite d1/d18/d4e/fe8 [0,4194304] 0 2024-02-28T16:31:27.543 INFO:tasks.workunit.client.0.smithi001.stdout:6/999: mknod d9/d29/d6e/c156 0 2024-02-28T16:31:27.543 INFO:tasks.workunit.client.1.smithi066.stdout:1/628: mkdir d2/dd0 0 2024-02-28T16:31:27.547 INFO:tasks.workunit.client.1.smithi066.stdout:1/629: getdents d2/dd0 0 2024-02-28T16:31:27.547 INFO:tasks.workunit.client.1.smithi066.stdout:4/759: readlink d1/d23/d25/d3e/laa 0 2024-02-28T16:31:27.548 INFO:tasks.workunit.client.0.smithi001.stdout:0/988: dwrite d18/f59 [0,4194304] 0 2024-02-28T16:31:27.548 INFO:tasks.workunit.client.1.smithi066.stdout:2/677: dread d0/d12/d1a/d23/f28 [0,4194304] 0 2024-02-28T16:31:27.551 INFO:tasks.workunit.client.1.smithi066.stdout:8/622: rename d0/d6/dc/f6b to d0/d6/dc/d16/d4c/d6c/d6f/da7/fcc 0 2024-02-28T16:31:27.553 INFO:tasks.workunit.client.1.smithi066.stdout:3/745: dwrite db/dc/db3/fcd [0,4194304] 0 2024-02-28T16:31:27.554 INFO:tasks.workunit.client.0.smithi001.stdout:2/961: dwrite d0/d2/dd9/f115 [0,4194304] 0 2024-02-28T16:31:27.557 INFO:tasks.workunit.client.0.smithi001.stdout:0/989: rmdir d18/da9/dc9/dd7/dfd 0 2024-02-28T16:31:27.561 INFO:tasks.workunit.client.1.smithi066.stdout:4/760: unlink d1/d23/c52 0 2024-02-28T16:31:27.563 INFO:tasks.workunit.client.0.smithi001.stdout:0/990: mknod d18/d22/d105/c13f 0 2024-02-28T16:31:27.564 INFO:tasks.workunit.client.1.smithi066.stdout:1/630: creat d2/fd1 x:0 0 0 2024-02-28T16:31:27.565 INFO:tasks.workunit.client.1.smithi066.stdout:3/746: read db/dc/d35/d3d/d76/d89/f9f [43351,45364] 0 2024-02-28T16:31:27.571 INFO:tasks.workunit.client.0.smithi001.stdout:2/962: mkdir d0/d2/d3/de/d61/d12/d55/d146 0 2024-02-28T16:31:27.574 INFO:tasks.workunit.client.1.smithi066.stdout:4/761: mkdir d1/dfb 0 2024-02-28T16:31:27.580 INFO:tasks.workunit.client.1.smithi066.stdout:4/762: creat d1/d23/d25/d3e/ffc x:0 0 0 2024-02-28T16:31:27.581 INFO:tasks.workunit.client.0.smithi001.stdout:2/963: rename d0/d2/d126/d84/ccc to d0/d2/c147 0 2024-02-28T16:31:27.581 INFO:tasks.workunit.client.1.smithi066.stdout:8/623: dwrite d0/d6/dc/d16/d4c/d6c/d6f/da7/fc5 [0,4194304] 0 2024-02-28T16:31:27.585 INFO:tasks.workunit.client.1.smithi066.stdout:8/624: write d0/d6/ff [2232111,31524] 0 2024-02-28T16:31:27.585 INFO:tasks.workunit.client.0.smithi001.stdout:2/964: mkdir d0/d2/d3/de/d22/dbb/dc8/d117/d148 0 2024-02-28T16:31:27.591 INFO:tasks.workunit.client.1.smithi066.stdout:8/625: dread - d0/d6/dc/d16/d1a/d63/d3d/f5b zero size 2024-02-28T16:31:27.592 INFO:tasks.workunit.client.0.smithi001.stdout:2/965: write d0/d2/d3/de/d61/d12/d46/f66 [1193723,62283] 0 2024-02-28T16:31:27.598 INFO:tasks.workunit.client.1.smithi066.stdout:3/747: dwrite db/d2a/d44/d48/d52/dcb/ff0 [0,4194304] 0 2024-02-28T16:31:27.599 INFO:tasks.workunit.client.1.smithi066.stdout:2/678: dread d0/d12/d1a/d39/f42 [0,4194304] 0 2024-02-28T16:31:27.599 INFO:tasks.workunit.client.0.smithi001.stdout:2/966: getdents d0/d2/d3/de/d10f 0 2024-02-28T16:31:27.606 INFO:tasks.workunit.client.1.smithi066.stdout:1/631: dread d2/d27/d2b/f2a [0,4194304] 0 2024-02-28T16:31:27.607 INFO:tasks.workunit.client.1.smithi066.stdout:4/763: dwrite d1/d23/d25/d43/f82 [0,4194304] 0 2024-02-28T16:31:27.609 INFO:tasks.workunit.client.1.smithi066.stdout:3/748: chown db/d13/d14/d2d/f9e 1008 1 2024-02-28T16:31:27.610 INFO:tasks.workunit.client.1.smithi066.stdout:5/608: sync 2024-02-28T16:31:27.610 INFO:tasks.workunit.client.1.smithi066.stdout:9/530: sync 2024-02-28T16:31:27.611 INFO:tasks.workunit.client.1.smithi066.stdout:0/752: sync 2024-02-28T16:31:27.611 INFO:tasks.workunit.client.1.smithi066.stdout:6/608: sync 2024-02-28T16:31:27.612 INFO:tasks.workunit.client.1.smithi066.stdout:4/764: fdatasync d1/d18/f86 0 2024-02-28T16:31:27.613 INFO:tasks.workunit.client.1.smithi066.stdout:3/749: read - db/d13/d1a/d33/ffc zero size 2024-02-28T16:31:27.620 INFO:tasks.workunit.client.1.smithi066.stdout:0/753: write d2/d27/d41/d76/d93/d33/d3f/d8e/fe3 [330366,48655] 0 2024-02-28T16:31:27.624 INFO:tasks.workunit.client.1.smithi066.stdout:9/531: symlink da/d4b/d53/laf 0 2024-02-28T16:31:27.625 INFO:tasks.workunit.client.1.smithi066.stdout:4/765: unlink d1/d7a/c8e 0 2024-02-28T16:31:27.630 INFO:tasks.workunit.client.1.smithi066.stdout:8/626: truncate d0/d6/dc/d16/d1a/f88 522810 0 2024-02-28T16:31:27.631 INFO:tasks.workunit.client.1.smithi066.stdout:0/754: fsync d2/d27/d41/d76/d93/db2/db5/d73/f90 0 2024-02-28T16:31:27.632 INFO:tasks.workunit.client.1.smithi066.stdout:1/632: truncate d2/d27/dc7/d51/f83 49270 0 2024-02-28T16:31:27.634 INFO:tasks.workunit.client.0.smithi001.stdout:2/967: dwrite d0/d2/d3/de/d61/fb3 [4194304,4194304] 0 2024-02-28T16:31:27.635 INFO:tasks.workunit.client.1.smithi066.stdout:6/609: dread d2/d26/f47 [0,4194304] 0 2024-02-28T16:31:27.635 INFO:tasks.workunit.client.1.smithi066.stdout:1/633: readlink d2/da0/la7 0 2024-02-28T16:31:27.647 INFO:tasks.workunit.client.0.smithi001.stdout:2/968: dread d0/d2/d3/de/d61/d12/d55/db2/f100 [0,4194304] 0 2024-02-28T16:31:27.651 INFO:tasks.workunit.client.1.smithi066.stdout:5/609: dread d3/d1b/d39/f44 [0,4194304] 0 2024-02-28T16:31:27.651 INFO:tasks.workunit.client.0.smithi001.stdout:2/969: write d0/d2/d3/fca [2633659,130815] 0 2024-02-28T16:31:27.656 INFO:tasks.workunit.client.0.smithi001.stdout:2/970: getdents d0/d2/d3/de/d22/dbb 0 2024-02-28T16:31:27.656 INFO:tasks.workunit.client.1.smithi066.stdout:2/679: dread d0/f71 [4194304,4194304] 0 2024-02-28T16:31:27.658 INFO:tasks.workunit.client.1.smithi066.stdout:3/750: dwrite db/d13/d14/d84/f9c [0,4194304] 0 2024-02-28T16:31:27.659 INFO:tasks.workunit.client.0.smithi001.stdout:2/971: mknod d0/d2/d3/d36/d94/c149 0 2024-02-28T16:31:27.661 INFO:tasks.workunit.client.1.smithi066.stdout:2/680: write d0/d12/d1a/d23/d29/d21/d2f/f5f [650276,105790] 0 2024-02-28T16:31:27.663 INFO:tasks.workunit.client.0.smithi001.stdout:2/972: mkdir d0/d2/d3/d73/d14a 0 2024-02-28T16:31:27.666 INFO:tasks.workunit.client.1.smithi066.stdout:2/681: truncate d0/d12/d1a/d39/d7c/d87/dc4/fc8 560837 0 2024-02-28T16:31:27.675 INFO:tasks.workunit.client.1.smithi066.stdout:7/612: sync 2024-02-28T16:31:27.696 INFO:tasks.workunit.client.0.smithi001.stdout:2/973: dread d0/d2/f14 [0,4194304] 0 2024-02-28T16:31:27.700 INFO:tasks.workunit.client.0.smithi001.stdout:2/974: symlink d0/d2/da6/l14b 0 2024-02-28T16:31:27.703 INFO:tasks.workunit.client.0.smithi001.stdout:2/975: write d0/f59 [1521829,106314] 0 2024-02-28T16:31:27.712 INFO:tasks.workunit.client.1.smithi066.stdout:7/613: dread dc/de/f13 [0,4194304] 0 2024-02-28T16:31:27.730 INFO:tasks.workunit.client.0.smithi001.stdout:2/976: dwrite d0/d2/d3/de/d61/f1f [0,4194304] 0 2024-02-28T16:31:27.735 INFO:tasks.workunit.client.1.smithi066.stdout:4/766: dread - d1/d23/d25/d77/fb6 zero size 2024-02-28T16:31:27.736 INFO:tasks.workunit.client.0.smithi001.stdout:2/977: creat d0/d2/d126/d84/f14c x:0 0 0 2024-02-28T16:31:27.736 INFO:tasks.workunit.client.1.smithi066.stdout:8/627: symlink d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/lcd 0 2024-02-28T16:31:27.739 INFO:tasks.workunit.client.1.smithi066.stdout:1/634: fdatasync d2/d27/d2b/f86 0 2024-02-28T16:31:27.741 INFO:tasks.workunit.client.1.smithi066.stdout:0/755: creat d2/d27/d41/d76/d93/d10/d23/d6d/ffb x:0 0 0 2024-02-28T16:31:27.751 INFO:tasks.workunit.client.1.smithi066.stdout:9/532: symlink da/df/lb0 0 2024-02-28T16:31:27.756 INFO:tasks.workunit.client.1.smithi066.stdout:4/767: mknod d1/d18/d4e/da0/cfd 0 2024-02-28T16:31:27.758 INFO:tasks.workunit.client.1.smithi066.stdout:6/610: symlink d2/d26/dbe/ld9 0 2024-02-28T16:31:27.758 INFO:tasks.workunit.client.0.smithi001.stdout:2/978: write d0/f7f [5041664,80361] 0 2024-02-28T16:31:27.761 INFO:tasks.workunit.client.1.smithi066.stdout:1/635: truncate d2/d18/d7a/f8b 95190 0 2024-02-28T16:31:27.763 INFO:tasks.workunit.client.1.smithi066.stdout:0/756: dwrite d2/d27/d41/d76/d93/db2/db5/d51/fd9 [0,4194304] 0 2024-02-28T16:31:27.764 INFO:tasks.workunit.client.1.smithi066.stdout:1/636: read - d2/d27/d2b/f86 zero size 2024-02-28T16:31:27.764 INFO:tasks.workunit.client.1.smithi066.stdout:2/682: creat d0/d12/d1a/d39/d7c/fdf x:0 0 0 2024-02-28T16:31:27.764 INFO:tasks.workunit.client.0.smithi001.stdout:2/979: creat d0/d2/d3/de/d61/f14d x:0 0 0 2024-02-28T16:31:27.765 INFO:tasks.workunit.client.1.smithi066.stdout:7/614: mkdir dc/de/d17/d91/dba/dbb 0 2024-02-28T16:31:27.767 INFO:tasks.workunit.client.1.smithi066.stdout:1/637: write d2/d27/dc7/d48/f66 [3180551,55910] 0 2024-02-28T16:31:27.781 INFO:tasks.workunit.client.0.smithi001.stdout:2/980: dread d0/d2/f14 [0,4194304] 0 2024-02-28T16:31:27.784 INFO:tasks.workunit.client.0.smithi001.stdout:2/981: write d0/d2/d3/de/d61/d12/d46/f142 [375018,56791] 0 2024-02-28T16:31:27.792 INFO:tasks.workunit.client.1.smithi066.stdout:0/757: write d2/d27/d41/d76/d93/d33/d3f/d8e/fe3 [239243,48928] 0 2024-02-28T16:31:27.793 INFO:tasks.workunit.client.1.smithi066.stdout:8/628: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db3/fce x:0 0 0 2024-02-28T16:31:27.794 INFO:tasks.workunit.client.1.smithi066.stdout:2/683: stat d0/d12/d2e/l7b 0 2024-02-28T16:31:27.795 INFO:tasks.workunit.client.1.smithi066.stdout:6/611: write d2/d26/f47 [3740915,101734] 0 2024-02-28T16:31:27.796 INFO:tasks.workunit.client.1.smithi066.stdout:8/629: dread - d0/db/fc2 zero size 2024-02-28T16:31:27.799 INFO:tasks.workunit.client.1.smithi066.stdout:0/758: readlink d2/d27/d41/d76/d93/db2/db5/d51/lf6 0 2024-02-28T16:31:27.801 INFO:tasks.workunit.client.1.smithi066.stdout:0/759: stat d2/d27/d41/ff5 0 2024-02-28T16:31:27.802 INFO:tasks.workunit.client.1.smithi066.stdout:2/684: write d0/d12/d1a/d39/d7c/d87/dc4/fcd [798349,103011] 0 2024-02-28T16:31:27.803 INFO:tasks.workunit.client.1.smithi066.stdout:7/615: dread dc/d29/f2e [0,4194304] 0 2024-02-28T16:31:27.803 INFO:tasks.workunit.client.1.smithi066.stdout:1/638: write d2/d27/d2b/d43/d46/f63 [116698,107021] 0 2024-02-28T16:31:27.804 INFO:tasks.workunit.client.1.smithi066.stdout:0/760: fsync d2/d27/d41/ff5 0 2024-02-28T16:31:27.807 INFO:tasks.workunit.client.1.smithi066.stdout:8/630: fdatasync d0/db/f5f 0 2024-02-28T16:31:27.811 INFO:tasks.workunit.client.0.smithi001.stdout:8/994: sync 2024-02-28T16:31:27.812 INFO:tasks.workunit.client.0.smithi001.stdout:2/982: dwrite d0/d2/d3/de/d22/dbb/dc8/d117/d119/f120 [0,4194304] 0 2024-02-28T16:31:27.817 INFO:tasks.workunit.client.1.smithi066.stdout:7/616: rename dc/de/d87/l8b to dc/de/d17/d91/dba/dbb/lbc 0 2024-02-28T16:31:27.817 INFO:tasks.workunit.client.0.smithi001.stdout:2/983: read d0/d2/d3/de/d8a/f51 [59699,19935] 0 2024-02-28T16:31:27.820 INFO:tasks.workunit.client.0.smithi001.stdout:8/995: creat d4/d7/d22/d27/d58/d94/db4/dd5/f141 x:0 0 0 2024-02-28T16:31:27.822 INFO:tasks.workunit.client.1.smithi066.stdout:2/685: dread d0/d12/d1a/d23/d5d/fbb [0,4194304] 0 2024-02-28T16:31:27.823 INFO:tasks.workunit.client.0.smithi001.stdout:8/996: readlink d4/d7/d22/d27/d58/d94/lc3 0 2024-02-28T16:31:27.826 INFO:tasks.workunit.client.0.smithi001.stdout:8/997: stat d4/d7/d22/d27/d58/f7b 0 2024-02-28T16:31:27.828 INFO:tasks.workunit.client.0.smithi001.stdout:2/984: rename d0/d2/d3/de/d61/d12/d55/d146 to d0/d2/d3/de/d22/dbb/dc8/d117/d119/d14e 0 2024-02-28T16:31:27.830 INFO:tasks.workunit.client.1.smithi066.stdout:2/686: write d0/d12/d1a/d39/d7c/d87/fa1 [2433279,52115] 0 2024-02-28T16:31:27.832 INFO:tasks.workunit.client.1.smithi066.stdout:7/617: mknod dc/d56/cbd 0 2024-02-28T16:31:27.832 INFO:tasks.workunit.client.0.smithi001.stdout:2/985: write d0/d2/d3/d36/d94/f10a [923303,75909] 0 2024-02-28T16:31:27.836 INFO:tasks.workunit.client.1.smithi066.stdout:0/761: unlink d2/d27/d41/d76/d93/db2/db5/d52/l80 0 2024-02-28T16:31:27.839 INFO:tasks.workunit.client.0.smithi001.stdout:2/986: creat d0/d2/d3/f14f x:0 0 0 2024-02-28T16:31:27.840 INFO:tasks.workunit.client.0.smithi001.stdout:8/998: dread d4/d7/d22/d27/d58/f87 [0,4194304] 0 2024-02-28T16:31:27.841 INFO:tasks.workunit.client.1.smithi066.stdout:2/687: fdatasync d0/d12/d1a/d23/d5d/f4d 0 2024-02-28T16:31:27.844 INFO:tasks.workunit.client.1.smithi066.stdout:2/688: readlink d0/d12/d2e/d7a/l6c 0 2024-02-28T16:31:27.844 INFO:tasks.workunit.client.1.smithi066.stdout:8/631: link d0/l49 d0/d6/dc/d16/d4c/d6c/lcf 0 2024-02-28T16:31:27.845 INFO:tasks.workunit.client.0.smithi001.stdout:8/999: mknod d4/d32/d93/c142 0 2024-02-28T16:31:27.848 INFO:tasks.workunit.client.1.smithi066.stdout:8/632: truncate d0/d8/fc7 721506 0 2024-02-28T16:31:27.849 INFO:tasks.workunit.client.1.smithi066.stdout:0/762: truncate d2/d85/f9f 8611 0 2024-02-28T16:31:27.849 INFO:tasks.workunit.client.1.smithi066.stdout:2/689: stat d0/d12/d1a/d39/d7c/fae 0 2024-02-28T16:31:27.853 INFO:tasks.workunit.client.1.smithi066.stdout:8/633: write d0/d6/dc/d16/fc3 [890679,96399] 0 2024-02-28T16:31:27.854 INFO:tasks.workunit.client.1.smithi066.stdout:2/690: write d0/f71 [2304770,6260] 0 2024-02-28T16:31:27.855 INFO:tasks.workunit.client.0.smithi001.stdout:2/987: dread d0/d2/d3/de/d61/f67 [0,4194304] 0 2024-02-28T16:31:27.857 INFO:tasks.workunit.client.0.smithi001.stdout:2/988: chown d0/d2/da6/fe9 0 1 2024-02-28T16:31:27.858 INFO:tasks.workunit.client.1.smithi066.stdout:3/751: creat db/d13/f10f x:0 0 0 2024-02-28T16:31:27.858 INFO:tasks.workunit.client.1.smithi066.stdout:8/634: dread - d0/db/d15/d95/fcb zero size 2024-02-28T16:31:27.858 INFO:tasks.workunit.client.1.smithi066.stdout:0/763: chown d2/lab 158086 1 2024-02-28T16:31:27.861 INFO:tasks.workunit.client.1.smithi066.stdout:3/752: dread db/d13/d14/d2d/f3e [0,4194304] 0 2024-02-28T16:31:27.866 INFO:tasks.workunit.client.1.smithi066.stdout:3/753: chown db/dc/d35/d3d/d76/d89/dcc/l28 17613 1 2024-02-28T16:31:27.868 INFO:tasks.workunit.client.1.smithi066.stdout:8/635: creat d0/db/d12/d1d/fd0 x:0 0 0 2024-02-28T16:31:27.870 INFO:tasks.workunit.client.1.smithi066.stdout:2/691: getdents d0/d12/d3e 0 2024-02-28T16:31:27.871 INFO:tasks.workunit.client.1.smithi066.stdout:7/618: dread dc/f5d [0,4194304] 0 2024-02-28T16:31:27.873 INFO:tasks.workunit.client.1.smithi066.stdout:8/636: dread - d0/db/d15/f7a zero size 2024-02-28T16:31:27.874 INFO:tasks.workunit.client.0.smithi001.stdout:2/989: dwrite d0/d2/d126/f7e [4194304,4194304] 0 2024-02-28T16:31:27.875 INFO:tasks.workunit.client.1.smithi066.stdout:3/754: chown db/d2a/d44/d48/d52/dcb/ff0 0 1 2024-02-28T16:31:27.877 INFO:tasks.workunit.client.1.smithi066.stdout:5/610: dwrite d3/d4/dc/d2f/d55/f5b [0,4194304] 0 2024-02-28T16:31:27.879 INFO:tasks.workunit.client.0.smithi001.stdout:2/990: mknod d0/d2/d3/d36/d94/c150 0 2024-02-28T16:31:27.883 INFO:tasks.workunit.client.1.smithi066.stdout:9/533: truncate da/d22/d46/d56/d8f/d2d/f62 3719674 0 2024-02-28T16:31:27.889 INFO:tasks.workunit.client.1.smithi066.stdout:3/755: creat db/d13/d1a/d25/f110 x:0 0 0 2024-02-28T16:31:27.897 INFO:tasks.workunit.client.1.smithi066.stdout:7/619: creat dc/de/d17/d1e/d45/d9f/fbe x:0 0 0 2024-02-28T16:31:27.899 INFO:tasks.workunit.client.1.smithi066.stdout:3/756: write db/d13/d1a/d25/fcf [4655990,90249] 0 2024-02-28T16:31:27.901 INFO:tasks.workunit.client.1.smithi066.stdout:9/534: creat da/d3a/d15/fb1 x:0 0 0 2024-02-28T16:31:27.901 INFO:tasks.workunit.client.0.smithi001.stdout:0/991: sync 2024-02-28T16:31:27.903 INFO:tasks.workunit.client.1.smithi066.stdout:8/637: dwrite d0/d6/dc/d16/d4c/d6c/d6f/da7/fc5 [4194304,4194304] 0 2024-02-28T16:31:27.904 INFO:tasks.workunit.client.1.smithi066.stdout:0/764: dread d2/d85/d8d/fd0 [0,4194304] 0 2024-02-28T16:31:27.905 INFO:tasks.workunit.client.1.smithi066.stdout:9/535: stat da/df/f3d 0 2024-02-28T16:31:27.908 INFO:tasks.workunit.client.0.smithi001.stdout:0/992: creat d18/d22/d2b/d5f/f140 x:0 0 0 2024-02-28T16:31:27.908 INFO:tasks.workunit.client.1.smithi066.stdout:1/639: truncate d2/d18/f6e 2036532 0 2024-02-28T16:31:27.910 INFO:tasks.workunit.client.1.smithi066.stdout:1/640: chown d2/d27/d2b/fc6 442092 1 2024-02-28T16:31:27.920 INFO:tasks.workunit.client.0.smithi001.stdout:2/991: dwrite d0/d2/d3/de/d61/d12/d55/db2/fc7 [4194304,4194304] 0 2024-02-28T16:31:27.926 INFO:tasks.workunit.client.0.smithi001.stdout:2/992: fdatasync d0/d2/d3/de/d22/dbb/dc8/fef 0 2024-02-28T16:31:27.927 INFO:tasks.workunit.client.1.smithi066.stdout:6/612: truncate d2/d2c/da1/d5d/d63/d6a/fb2 983948 0 2024-02-28T16:31:27.929 INFO:tasks.workunit.client.0.smithi001.stdout:0/993: dwrite d18/d22/d2b/d36/d4d/f65 [0,4194304] 0 2024-02-28T16:31:27.934 INFO:tasks.workunit.client.0.smithi001.stdout:0/994: write d18/da9/dea/d11d/f13a [717585,28924] 0 2024-02-28T16:31:27.940 INFO:tasks.workunit.client.1.smithi066.stdout:9/536: chown da/d3a/d15/f19 908 1 2024-02-28T16:31:27.942 INFO:tasks.workunit.client.1.smithi066.stdout:2/692: dwrite d0/d12/d1a/d39/d7c/fbe [0,4194304] 0 2024-02-28T16:31:27.944 INFO:tasks.workunit.client.0.smithi001.stdout:0/995: rename d18/d22/d2f/ff1 to d18/d22/d2b/d5f/f141 0 2024-02-28T16:31:27.947 INFO:tasks.workunit.client.0.smithi001.stdout:2/993: dread d0/d2/d3/de/d22/f6d [0,4194304] 0 2024-02-28T16:31:27.951 INFO:tasks.workunit.client.1.smithi066.stdout:8/638: getdents d0/db/d12/d1d/d25/d4f 0 2024-02-28T16:31:27.952 INFO:tasks.workunit.client.1.smithi066.stdout:9/537: stat da/d22/d46/d56/d8f/c7f 0 2024-02-28T16:31:27.956 INFO:tasks.workunit.client.1.smithi066.stdout:2/693: truncate d0/d12/d2e/fc1 143255 0 2024-02-28T16:31:27.956 INFO:tasks.workunit.client.1.smithi066.stdout:0/765: dwrite d2/d27/d41/d76/d93/d33/d3f/fbb [0,4194304] 0 2024-02-28T16:31:27.956 INFO:tasks.workunit.client.1.smithi066.stdout:6/613: link d2/d26/f53 d2/d26/d32/d46/fda 0 2024-02-28T16:31:27.961 INFO:tasks.workunit.client.1.smithi066.stdout:8/639: mknod d0/d6/cd1 0 2024-02-28T16:31:27.965 INFO:tasks.workunit.client.1.smithi066.stdout:3/757: truncate db/d2a/d44/d48/d5d/fbe 683549 0 2024-02-28T16:31:27.968 INFO:tasks.workunit.client.0.smithi001.stdout:0/996: write d18/f33 [587306,39852] 0 2024-02-28T16:31:27.971 INFO:tasks.workunit.client.1.smithi066.stdout:6/614: mknod d2/d2c/d82/cdb 0 2024-02-28T16:31:27.972 INFO:tasks.workunit.client.0.smithi001.stdout:2/994: write d0/d2/d3/d36/ffc [623500,45534] 0 2024-02-28T16:31:27.973 INFO:tasks.workunit.client.0.smithi001.stdout:0/997: chown d18/d22/d2f/d1c/c3b 12392080 1 2024-02-28T16:31:27.975 INFO:tasks.workunit.client.0.smithi001.stdout:2/995: fdatasync d0/d2/f8c 0 2024-02-28T16:31:27.978 INFO:tasks.workunit.client.1.smithi066.stdout:0/766: mkdir d2/d27/d41/d76/d93/db2/db5/d67/dfc 0 2024-02-28T16:31:27.979 INFO:tasks.workunit.client.1.smithi066.stdout:6/615: chown d2/d2c/d82/f8a 715 1 2024-02-28T16:31:27.981 INFO:tasks.workunit.client.1.smithi066.stdout:3/758: creat db/dc/d35/d3d/d76/d89/f111 x:0 0 0 2024-02-28T16:31:27.982 INFO:tasks.workunit.client.1.smithi066.stdout:8/640: symlink d0/d6/dc/d16/d4c/d6c/d6f/da7/dc9/ld2 0 2024-02-28T16:31:27.985 INFO:tasks.workunit.client.1.smithi066.stdout:8/641: fdatasync d0/db/d12/d8a/fa0 0 2024-02-28T16:31:27.985 INFO:tasks.workunit.client.1.smithi066.stdout:6/616: write d2/da/d19/f4f [6550855,70450] 0 2024-02-28T16:31:27.987 INFO:tasks.workunit.client.0.smithi001.stdout:0/998: dread d18/d22/d2f/fe9 [0,4194304] 0 2024-02-28T16:31:27.989 INFO:tasks.workunit.client.1.smithi066.stdout:3/759: readlink db/d13/d14/d2d/d92/le7 0 2024-02-28T16:31:27.990 INFO:tasks.workunit.client.0.smithi001.stdout:2/996: write d0/d2/d3/de/d22/dbb/dc8/fef [158327,36288] 0 2024-02-28T16:31:27.991 INFO:tasks.workunit.client.1.smithi066.stdout:0/767: mkdir d2/d27/d41/d76/d93/d10/d23/db6/de0/dfd 0 2024-02-28T16:31:27.991 INFO:tasks.workunit.client.0.smithi001.stdout:0/999: rmdir d18/d22/d2b/d36/d48/d11a/d7e 39 2024-02-28T16:31:27.992 INFO:tasks.workunit.client.0.smithi001.stdout:2/997: readlink d0/d2/d3/de/d22/l13e 0 2024-02-28T16:31:27.993 INFO:tasks.workunit.client.1.smithi066.stdout:7/620: dread dc/de/d17/d1e/d45/fae [0,4194304] 0 2024-02-28T16:31:27.994 INFO:tasks.workunit.client.1.smithi066.stdout:6/617: mknod d2/da/d19/d75/dbd/cdc 0 2024-02-28T16:31:27.996 INFO:tasks.workunit.client.0.smithi001.stdout:2/998: creat d0/d2/d3/de/d22/dbb/dc8/d117/f151 x:0 0 0 2024-02-28T16:31:27.999 INFO:tasks.workunit.client.1.smithi066.stdout:3/760: creat db/d13/d14/d84/f112 x:0 0 0 2024-02-28T16:31:28.001 INFO:tasks.workunit.client.1.smithi066.stdout:3/761: fsync db/d2a/dd4/ff9 0 2024-02-28T16:31:28.002 INFO:tasks.workunit.client.1.smithi066.stdout:0/768: readlink d2/d27/d41/d76/d93/df3/lcc 0 2024-02-28T16:31:28.003 INFO:tasks.workunit.client.0.smithi001.stdout:2/999: rename d0/d2/d3/de/d61/d12/fe3 to d0/d2/d3/de/d61/d12/f152 0 2024-02-28T16:31:28.003 INFO:tasks.workunit.client.1.smithi066.stdout:6/618: creat d2/da/d19/fdd x:0 0 0 2024-02-28T16:31:28.006 INFO:tasks.workunit.client.1.smithi066.stdout:7/621: mkdir dc/de/d17/d91/dbf 0 2024-02-28T16:31:28.010 INFO:tasks.workunit.client.1.smithi066.stdout:3/762: creat db/d13/d14/dda/f113 x:0 0 0 2024-02-28T16:31:28.011 INFO:tasks.workunit.client.0.smithi001.stderr:+ rm -rf -- ./tmp.r3r2qJkLtM 2024-02-28T16:31:28.014 INFO:tasks.workunit.client.1.smithi066.stdout:6/619: symlink d2/d2c/dba/lde 0 2024-02-28T16:31:28.016 INFO:tasks.workunit.client.1.smithi066.stdout:7/622: creat dc/d56/d97/fc0 x:0 0 0 2024-02-28T16:31:28.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:27 smithi001 ceph-mon[28991]: pgmap v57: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 88 MiB/s rd, 249 MiB/s wr, 511 op/s 2024-02-28T16:31:28.022 INFO:tasks.workunit.client.1.smithi066.stdout:3/763: rename db/dc/d11/daa/fd3 to db/d2a/d44/d48/d7d/db4/f114 0 2024-02-28T16:31:28.022 INFO:tasks.workunit.client.1.smithi066.stdout:2/694: dwrite d0/d12/d2e/d3b/fb1 [0,4194304] 0 2024-02-28T16:31:28.025 INFO:tasks.workunit.client.1.smithi066.stdout:3/764: fdatasync db/dc/d11/f22 0 2024-02-28T16:31:28.030 INFO:tasks.workunit.client.1.smithi066.stdout:6/620: symlink d2/ldf 0 2024-02-28T16:31:28.030 INFO:tasks.workunit.client.1.smithi066.stdout:7/623: creat dc/de/d17/d1e/da1/fc1 x:0 0 0 2024-02-28T16:31:28.030 INFO:tasks.workunit.client.1.smithi066.stdout:3/765: chown db/d2a/d44/d48/d52/lf2 31 1 2024-02-28T16:31:28.030 INFO:tasks.workunit.client.1.smithi066.stdout:4/768: sync 2024-02-28T16:31:28.030 INFO:tasks.workunit.client.1.smithi066.stdout:7/624: stat f1 0 2024-02-28T16:31:28.035 INFO:tasks.workunit.client.1.smithi066.stdout:7/625: chown dc/de/d17/c51 49 1 2024-02-28T16:31:28.036 INFO:tasks.workunit.client.1.smithi066.stdout:3/766: read - db/dc/d35/d3d/d76/d89/f111 zero size 2024-02-28T16:31:28.036 INFO:tasks.workunit.client.1.smithi066.stdout:8/642: write d0/d6/dc/d16/d4c/d6c/d6f/da7/fcc [770242,99830] 0 2024-02-28T16:31:28.037 INFO:tasks.workunit.client.1.smithi066.stdout:6/621: mkdir d2/d26/de0 0 2024-02-28T16:31:28.041 INFO:tasks.workunit.client.1.smithi066.stdout:4/769: link d1/d18/d4e/c8f d1/d18/d55/cfe 0 2024-02-28T16:31:28.046 INFO:tasks.workunit.client.1.smithi066.stdout:7/626: write dc/de/d17/d37/d47/d88/fb5 [548924,39329] 0 2024-02-28T16:31:28.046 INFO:tasks.workunit.client.1.smithi066.stdout:6/622: read d2/da/d19/d75/d88/f8d [114014,35988] 0 2024-02-28T16:31:28.046 INFO:tasks.workunit.client.1.smithi066.stdout:8/643: rmdir d0/db/d12/d1d/d25/d65/d93 39 2024-02-28T16:31:28.047 INFO:tasks.workunit.client.1.smithi066.stdout:0/769: dread d2/d27/d41/d76/d93/d10/f7f [0,4194304] 0 2024-02-28T16:31:28.052 INFO:tasks.workunit.client.1.smithi066.stdout:4/770: symlink d1/d18/d55/lff 0 2024-02-28T16:31:28.053 INFO:tasks.workunit.client.1.smithi066.stdout:2/695: dwrite d0/d12/d2e/d7a/f45 [0,4194304] 0 2024-02-28T16:31:28.055 INFO:tasks.workunit.client.1.smithi066.stdout:4/771: fdatasync d1/d7a/fd1 0 2024-02-28T16:31:28.059 INFO:tasks.workunit.client.1.smithi066.stdout:2/696: read d0/d12/d1a/d39/d7c/fbe [2972256,18169] 0 2024-02-28T16:31:28.061 INFO:tasks.workunit.client.1.smithi066.stdout:3/767: rename db/dc/d11/d107 to db/dc/d11/d1f/d115 0 2024-02-28T16:31:28.065 INFO:tasks.workunit.client.1.smithi066.stdout:4/772: mkdir d1/d18/d41/dba/d100 0 2024-02-28T16:31:28.069 INFO:tasks.workunit.client.1.smithi066.stdout:6/623: rename d2/da/f1a to d2/d2c/d72/fe1 0 2024-02-28T16:31:28.072 INFO:tasks.workunit.client.1.smithi066.stdout:3/768: rmdir db/dc/d11/daa/ded 39 2024-02-28T16:31:28.076 INFO:tasks.workunit.client.1.smithi066.stdout:7/627: rename dc/de/d17/d37/l6a to dc/de/d17/d1e/da1/db7/lc2 0 2024-02-28T16:31:28.077 INFO:tasks.workunit.client.1.smithi066.stdout:3/769: chown db/dc/d35/d3d/l7f 6 1 2024-02-28T16:31:28.077 INFO:tasks.workunit.client.1.smithi066.stdout:6/624: write d2/d2c/da1/d5d/d99/f85 [4227920,68926] 0 2024-02-28T16:31:28.079 INFO:tasks.workunit.client.1.smithi066.stdout:0/770: dwrite d2/d27/d41/d76/d93/d10/f2c [0,4194304] 0 2024-02-28T16:31:28.082 INFO:tasks.workunit.client.1.smithi066.stdout:2/697: dread d0/d12/d1a/d39/d7c/d87/dc4/fcd [0,4194304] 0 2024-02-28T16:31:28.088 INFO:tasks.workunit.client.1.smithi066.stdout:6/625: unlink d2/d2c/f97 0 2024-02-28T16:31:28.090 INFO:tasks.workunit.client.1.smithi066.stdout:3/770: mkdir db/dc/db3/dc9/de4/d116 0 2024-02-28T16:31:28.096 INFO:tasks.workunit.client.1.smithi066.stdout:6/626: dread - d2/f23 zero size 2024-02-28T16:31:28.096 INFO:tasks.workunit.client.1.smithi066.stdout:4/773: write d1/f12 [222104,53782] 0 2024-02-28T16:31:28.097 INFO:tasks.workunit.client.1.smithi066.stdout:8/644: dwrite d0/db/d12/d1d/d25/f75 [0,4194304] 0 2024-02-28T16:31:28.099 INFO:tasks.workunit.client.1.smithi066.stdout:7/628: write dc/de/d17/f19 [2153396,2704] 0 2024-02-28T16:31:28.101 INFO:tasks.workunit.client.1.smithi066.stdout:3/771: unlink db/dc/d11/l4d 0 2024-02-28T16:31:28.103 INFO:tasks.workunit.client.1.smithi066.stdout:2/698: chown d0/d12/d1a/d23/d29/d21/d2f/d67/f97 0 1 2024-02-28T16:31:28.104 INFO:tasks.workunit.client.1.smithi066.stdout:6/627: stat d2/d2c/f8c 0 2024-02-28T16:31:28.104 INFO:tasks.workunit.client.1.smithi066.stdout:7/629: mkdir dc/d56/dc3 0 2024-02-28T16:31:28.108 INFO:tasks.workunit.client.1.smithi066.stdout:3/772: dread - db/dc/d35/d3d/d76/ffa zero size 2024-02-28T16:31:28.109 INFO:tasks.workunit.client.1.smithi066.stdout:8/645: symlink d0/d6/dc/d16/d1a/d63/ld3 0 2024-02-28T16:31:28.116 INFO:tasks.workunit.client.1.smithi066.stdout:0/771: dwrite d2/d27/d41/d76/d93/db2/db5/d67/d94/de4/fdd [0,4194304] 0 2024-02-28T16:31:28.116 INFO:tasks.workunit.client.1.smithi066.stdout:7/630: mkdir dc/de/d17/d37/d47/d88/dc4 0 2024-02-28T16:31:28.117 INFO:tasks.workunit.client.1.smithi066.stdout:6/628: getdents d2/d2c/d6e/dd0 0 2024-02-28T16:31:28.119 INFO:tasks.workunit.client.1.smithi066.stdout:0/772: stat d2/d85/da9/cce 0 2024-02-28T16:31:28.120 INFO:tasks.workunit.client.1.smithi066.stdout:2/699: dread d0/d12/d3e/f63 [0,4194304] 0 2024-02-28T16:31:28.120 INFO:tasks.workunit.client.1.smithi066.stdout:6/629: dread - d2/d2c/da1/d5d/d63/fc2 zero size 2024-02-28T16:31:28.121 INFO:tasks.workunit.client.1.smithi066.stdout:8/646: rmdir d0/d6/dc/d16/d4c/d6c/d6f 39 2024-02-28T16:31:28.121 INFO:tasks.workunit.client.1.smithi066.stdout:3/773: mknod db/d2a/d44/d48/d52/dcb/c117 0 2024-02-28T16:31:28.124 INFO:tasks.workunit.client.1.smithi066.stdout:7/631: creat dc/de/fc5 x:0 0 0 2024-02-28T16:31:28.127 INFO:tasks.workunit.client.1.smithi066.stdout:0/773: symlink d2/d27/d41/d76/lfe 0 2024-02-28T16:31:28.128 INFO:tasks.workunit.client.1.smithi066.stdout:6/630: fdatasync d2/da/d19/d9c/d3d/f7f 0 2024-02-28T16:31:28.131 INFO:tasks.workunit.client.1.smithi066.stdout:8/647: mkdir d0/d6/dc/dd4 0 2024-02-28T16:31:28.138 INFO:tasks.workunit.client.1.smithi066.stdout:3/774: truncate db/d2a/d44/d48/d7d/db4/f114 142171 0 2024-02-28T16:31:28.139 INFO:tasks.workunit.client.1.smithi066.stdout:7/632: mkdir dc/de/d17/d1e/da1/db7/dc6 0 2024-02-28T16:31:28.139 INFO:tasks.workunit.client.1.smithi066.stdout:3/775: chown db/la8 34393 1 2024-02-28T16:31:28.139 INFO:tasks.workunit.client.1.smithi066.stdout:0/774: creat d2/d27/d41/d76/d93/d33/d3f/fff x:0 0 0 2024-02-28T16:31:28.142 INFO:tasks.workunit.client.1.smithi066.stdout:3/776: mknod db/dc/db3/c118 0 2024-02-28T16:31:28.146 INFO:tasks.workunit.client.1.smithi066.stdout:6/631: link d2/da/d19/d75/d88/fce d2/da/d19/d75/dbd/fe2 0 2024-02-28T16:31:28.146 INFO:tasks.workunit.client.1.smithi066.stdout:2/700: getdents d0/d12/d1a/d23/d29/d21/d2f/d67 0 2024-02-28T16:31:28.147 INFO:tasks.workunit.client.1.smithi066.stdout:0/775: creat d2/d27/d41/d76/d93/df3/f100 x:0 0 0 2024-02-28T16:31:28.149 INFO:tasks.workunit.client.1.smithi066.stdout:3/777: unlink db/d7a/l10b 0 2024-02-28T16:31:28.161 INFO:tasks.workunit.client.1.smithi066.stdout:4/774: dread d1/d23/f36 [0,4194304] 0 2024-02-28T16:31:28.161 INFO:tasks.workunit.client.1.smithi066.stdout:0/776: unlink d2/d27/d41/d76/d93/db2/db5/d52/f9a 0 2024-02-28T16:31:28.163 INFO:tasks.workunit.client.1.smithi066.stdout:2/701: mkdir d0/d12/d3e/d9e/dce/de0 0 2024-02-28T16:31:28.164 INFO:tasks.workunit.client.1.smithi066.stdout:8/648: write d0/db/f5f [698438,13367] 0 2024-02-28T16:31:28.167 INFO:tasks.workunit.client.1.smithi066.stdout:0/777: rmdir d2/d85 39 2024-02-28T16:31:28.167 INFO:tasks.workunit.client.1.smithi066.stdout:8/649: dread - d0/d6/dc/d16/d4c/d42/fbd zero size 2024-02-28T16:31:28.169 INFO:tasks.workunit.client.1.smithi066.stdout:4/775: creat d1/d18/d55/f101 x:0 0 0 2024-02-28T16:31:28.171 INFO:tasks.workunit.client.1.smithi066.stdout:2/702: truncate d0/d12/d1a/f3a 1513657 0 2024-02-28T16:31:28.172 INFO:tasks.workunit.client.1.smithi066.stdout:8/650: symlink d0/db/d12/d1d/d25/d4f/ld5 0 2024-02-28T16:31:28.173 INFO:tasks.workunit.client.1.smithi066.stdout:6/632: write d2/d2c/da1/d5d/d99/f90 [318261,128849] 0 2024-02-28T16:31:28.174 INFO:tasks.workunit.client.1.smithi066.stdout:0/778: mknod d2/d27/d41/d76/d93/db2/db5/d73/c101 0 2024-02-28T16:31:28.176 INFO:tasks.workunit.client.1.smithi066.stdout:6/633: stat d2/d2c/da1/fb7 0 2024-02-28T16:31:28.177 INFO:tasks.workunit.client.1.smithi066.stdout:8/651: write d0/db/d12/d1d/fd0 [487983,129779] 0 2024-02-28T16:31:28.180 INFO:tasks.workunit.client.1.smithi066.stdout:2/703: rmdir d0/d12/d3e/d9e/dce/de0 0 2024-02-28T16:31:28.187 INFO:tasks.workunit.client.1.smithi066.stdout:0/779: write d2/d27/d41/d76/d93/d10/f2c [2270020,50370] 0 2024-02-28T16:31:28.187 INFO:tasks.workunit.client.1.smithi066.stdout:8/652: fdatasync d0/d6/dc/d16/d4c/d42/fbd 0 2024-02-28T16:31:28.191 INFO:tasks.workunit.client.1.smithi066.stdout:3/778: dwrite db/f66 [0,4194304] 0 2024-02-28T16:31:28.194 INFO:tasks.workunit.client.1.smithi066.stdout:2/704: truncate d0/d12/d1a/d23/d5d/fbb 441632 0 2024-02-28T16:31:28.199 INFO:tasks.workunit.client.1.smithi066.stdout:3/779: chown db/d2a/d44/d48/fd6 3 1 2024-02-28T16:31:28.201 INFO:tasks.workunit.client.1.smithi066.stdout:6/634: link d2/d26/d32/d46/fda d2/da/d19/d75/da6/daa/dbf/fe3 0 2024-02-28T16:31:28.203 INFO:tasks.workunit.client.1.smithi066.stdout:3/780: symlink db/dc/d11/d1f/d115/l119 0 2024-02-28T16:31:28.204 INFO:tasks.workunit.client.1.smithi066.stdout:6/635: chown d2/da/f13 15 1 2024-02-28T16:31:28.205 INFO:tasks.workunit.client.1.smithi066.stdout:3/781: chown db/d13/d1a/d25/f110 0 1 2024-02-28T16:31:28.212 INFO:tasks.workunit.client.1.smithi066.stdout:6/636: rmdir d2/d2c/da1/d5d/d63 39 2024-02-28T16:31:28.213 INFO:tasks.workunit.client.1.smithi066.stdout:4/776: dread d1/d18/d4e/f53 [0,4194304] 0 2024-02-28T16:31:28.216 INFO:tasks.workunit.client.1.smithi066.stdout:4/777: chown d1/ce4 56254 1 2024-02-28T16:31:28.219 INFO:tasks.workunit.client.1.smithi066.stdout:6/637: mkdir d2/da/d19/d9c/de4 0 2024-02-28T16:31:28.222 INFO:tasks.workunit.client.1.smithi066.stdout:4/778: rename d1/d18 to d1/d18/d7e/d102 22 2024-02-28T16:31:28.229 INFO:tasks.workunit.client.1.smithi066.stdout:8/653: dwrite d0/db/d12/d1d/d25/d65/f84 [0,4194304] 0 2024-02-28T16:31:28.229 INFO:tasks.workunit.client.1.smithi066.stdout:6/638: read d2/da/d19/d9c/d3d/f48 [588963,65976] 0 2024-02-28T16:31:28.229 INFO:tasks.workunit.client.1.smithi066.stdout:4/779: creat d1/df7/f103 x:0 0 0 2024-02-28T16:31:28.235 INFO:tasks.workunit.client.1.smithi066.stdout:2/705: dwrite d0/d12/d2e/f59 [0,4194304] 0 2024-02-28T16:31:28.239 INFO:tasks.workunit.client.1.smithi066.stdout:8/654: creat d0/db/d15/fd6 x:0 0 0 2024-02-28T16:31:28.243 INFO:tasks.workunit.client.1.smithi066.stdout:0/780: dwrite d2/d27/d41/d76/d93/f4f [0,4194304] 0 2024-02-28T16:31:28.245 INFO:tasks.workunit.client.1.smithi066.stdout:2/706: creat d0/d12/d1a/d23/fe1 x:0 0 0 2024-02-28T16:31:28.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:27 smithi066 ceph-mon[36236]: pgmap v57: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 88 MiB/s rd, 249 MiB/s wr, 511 op/s 2024-02-28T16:31:28.248 INFO:tasks.workunit.client.1.smithi066.stdout:8/655: symlink d0/db/d12/d8a/ld7 0 2024-02-28T16:31:28.248 INFO:tasks.workunit.client.1.smithi066.stdout:3/782: dwrite db/ff [0,4194304] 0 2024-02-28T16:31:28.251 INFO:tasks.workunit.client.1.smithi066.stdout:3/783: fsync db/d13/d1a/d25/f96 0 2024-02-28T16:31:28.257 INFO:tasks.workunit.client.1.smithi066.stdout:3/784: mknod db/d13/d1a/c11a 0 2024-02-28T16:31:28.259 INFO:tasks.workunit.client.1.smithi066.stdout:3/785: write db/d2a/ff7 [238935,33403] 0 2024-02-28T16:31:28.261 INFO:tasks.workunit.client.1.smithi066.stdout:2/707: creat d0/fe2 x:0 0 0 2024-02-28T16:31:28.281 INFO:tasks.workunit.client.1.smithi066.stdout:8/656: link d0/d6/dc/d16/l5e d0/db/d12/d1d/d25/d72/ld8 0 2024-02-28T16:31:28.283 INFO:tasks.workunit.client.1.smithi066.stdout:3/786: link db/d13/d14/c4c db/dc/d11/daa/c11b 0 2024-02-28T16:31:28.286 INFO:tasks.workunit.client.1.smithi066.stdout:8/657: unlink d0/d6/f9a 0 2024-02-28T16:31:28.288 INFO:tasks.workunit.client.1.smithi066.stdout:4/780: write d1/d23/d25/f59 [575161,31768] 0 2024-02-28T16:31:28.290 INFO:tasks.workunit.client.1.smithi066.stdout:3/787: truncate db/d13/d1a/fab 5127545 0 2024-02-28T16:31:28.296 INFO:tasks.workunit.client.1.smithi066.stdout:4/781: mkdir d1/d7a/d104 0 2024-02-28T16:31:28.301 INFO:tasks.workunit.client.1.smithi066.stdout:5/611: sync 2024-02-28T16:31:28.302 INFO:tasks.workunit.client.1.smithi066.stdout:6/639: dwrite d2/f23 [0,4194304] 0 2024-02-28T16:31:28.305 INFO:tasks.workunit.client.1.smithi066.stdout:5/612: chown d3/c96 22849 1 2024-02-28T16:31:28.310 INFO:tasks.workunit.client.1.smithi066.stdout:4/782: write d1/d18/d41/da7/dae/fe3 [609118,65888] 0 2024-02-28T16:31:28.311 INFO:tasks.workunit.client.1.smithi066.stdout:6/640: dread - d2/da/d19/d75/dbd/fe2 zero size 2024-02-28T16:31:28.316 INFO:tasks.workunit.client.1.smithi066.stdout:5/613: fsync d3/d1b/f5f 0 2024-02-28T16:31:28.324 INFO:tasks.workunit.client.1.smithi066.stdout:4/783: symlink d1/d7a/l105 0 2024-02-28T16:31:28.327 INFO:tasks.workunit.client.1.smithi066.stdout:5/614: write d3/d4/dc/d2f/d55/d74/fbb [913486,113981] 0 2024-02-28T16:31:28.328 INFO:tasks.workunit.client.1.smithi066.stdout:0/781: dwrite d2/d27/d41/d76/d93/db2/db5/d60/f95 [4194304,4194304] 0 2024-02-28T16:31:28.334 INFO:tasks.workunit.client.1.smithi066.stdout:1/641: sync 2024-02-28T16:31:28.335 INFO:tasks.workunit.client.1.smithi066.stdout:4/784: write d1/d18/d7e/fab [3790874,48420] 0 2024-02-28T16:31:28.339 INFO:tasks.workunit.client.1.smithi066.stdout:2/708: dread d0/d12/d1a/d23/d29/d21/d2f/fa2 [0,4194304] 0 2024-02-28T16:31:28.344 INFO:tasks.workunit.client.1.smithi066.stdout:0/782: write d2/d27/d41/d76/d93/db2/db5/d51/f82 [1035374,12875] 0 2024-02-28T16:31:28.345 INFO:tasks.workunit.client.1.smithi066.stdout:3/788: dread db/dc/d35/d3d/f97 [0,4194304] 0 2024-02-28T16:31:28.348 INFO:tasks.workunit.client.1.smithi066.stdout:2/709: truncate d0/d12/d2e/d7a/fc0 453751 0 2024-02-28T16:31:28.348 INFO:tasks.workunit.client.1.smithi066.stdout:4/785: chown d1/d18/d4e/caf 87469 1 2024-02-28T16:31:28.349 INFO:tasks.workunit.client.1.smithi066.stdout:1/642: creat d2/d27/d2b/d34/d9e/fd2 x:0 0 0 2024-02-28T16:31:28.349 INFO:tasks.workunit.client.1.smithi066.stdout:5/615: truncate d3/d4/dc/d56/f75 3284108 0 2024-02-28T16:31:28.352 INFO:tasks.workunit.client.1.smithi066.stdout:4/786: stat d1/d18/d7e/fab 0 2024-02-28T16:31:28.367 INFO:tasks.workunit.client.1.smithi066.stdout:0/783: mkdir d2/d27/d41/d76/d93/d10/d23/db6/de0/dfd/d102 0 2024-02-28T16:31:28.368 INFO:tasks.workunit.client.1.smithi066.stdout:2/710: chown d0/d12/d1a/d23/d29/d21/d2f/c80 29 1 2024-02-28T16:31:28.368 INFO:tasks.workunit.client.1.smithi066.stdout:8/658: dread d0/db/d12/d1d/d25/fb5 [0,4194304] 0 2024-02-28T16:31:28.373 INFO:tasks.workunit.client.1.smithi066.stdout:0/784: chown d2/d27/d41/d76/d93/df3 11348980 1 2024-02-28T16:31:28.375 INFO:tasks.workunit.client.1.smithi066.stdout:6/641: dread d2/da/d19/d9c/d3d/f48 [0,4194304] 0 2024-02-28T16:31:28.376 INFO:tasks.workunit.client.1.smithi066.stdout:3/789: write db/dc/d35/d3d/d76/fb9 [3073126,91177] 0 2024-02-28T16:31:28.381 INFO:tasks.workunit.client.1.smithi066.stdout:3/790: fdatasync db/d2a/d44/d48/d52/dcb/fd7 0 2024-02-28T16:31:28.383 INFO:tasks.workunit.client.1.smithi066.stdout:5/616: dwrite d3/f98 [0,4194304] 0 2024-02-28T16:31:28.383 INFO:tasks.workunit.client.1.smithi066.stdout:1/643: write d2/d27/d2b/d43/f8e [259233,96410] 0 2024-02-28T16:31:28.387 INFO:tasks.workunit.client.1.smithi066.stdout:3/791: chown db/dc/d35/d3d/d76/d89/dcc 0 1 2024-02-28T16:31:28.391 INFO:tasks.workunit.client.1.smithi066.stdout:1/644: write d2/d27/d2b/d43/d46/d80/f9f [677850,36326] 0 2024-02-28T16:31:28.393 INFO:tasks.workunit.client.1.smithi066.stdout:3/792: mknod db/d2a/d44/d48/d52/d77/c11c 0 2024-02-28T16:31:28.395 INFO:tasks.workunit.client.1.smithi066.stdout:8/659: write d0/db/d15/f7a [512633,39369] 0 2024-02-28T16:31:28.397 INFO:tasks.workunit.client.1.smithi066.stdout:0/785: write d2/d27/d41/d76/d93/db2/db5/d52/f81 [2255606,120354] 0 2024-02-28T16:31:28.398 INFO:tasks.workunit.client.1.smithi066.stdout:1/645: chown d2/d27/d2b/f86 202221 1 2024-02-28T16:31:28.400 INFO:tasks.workunit.client.1.smithi066.stdout:1/646: stat d2 0 2024-02-28T16:31:28.405 INFO:tasks.workunit.client.1.smithi066.stdout:8/660: write d0/d6/dc/d16/d4c/d6c/f91 [1701560,29198] 0 2024-02-28T16:31:28.406 INFO:tasks.workunit.client.1.smithi066.stdout:0/786: readlink d2/d27/d41/d76/d93/d10/l17 0 2024-02-28T16:31:28.410 INFO:tasks.workunit.client.1.smithi066.stdout:4/787: dwrite d1/d18/f37 [4194304,4194304] 0 2024-02-28T16:31:28.414 INFO:tasks.workunit.client.1.smithi066.stdout:0/787: write d2/d85/d8d/fed [658854,88816] 0 2024-02-28T16:31:28.414 INFO:tasks.workunit.client.1.smithi066.stdout:1/647: link d2/d27/d2b/d34/d9e/fae d2/d27/d2b/d43/fd3 0 2024-02-28T16:31:28.415 INFO:tasks.workunit.client.1.smithi066.stdout:8/661: symlink d0/d6/dc/dd4/ld9 0 2024-02-28T16:31:28.416 INFO:tasks.workunit.client.1.smithi066.stdout:5/617: dwrite d3/d4/dc/dd/d2e/f38 [0,4194304] 0 2024-02-28T16:31:28.419 INFO:tasks.workunit.client.1.smithi066.stdout:4/788: truncate d1/d18/d4e/f53 4500774 0 2024-02-28T16:31:28.429 INFO:tasks.workunit.client.1.smithi066.stdout:4/789: creat d1/d18/d41/da7/f106 x:0 0 0 2024-02-28T16:31:28.430 INFO:tasks.workunit.client.1.smithi066.stdout:8/662: creat d0/db/d12/d1d/d25/d4f/fda x:0 0 0 2024-02-28T16:31:28.436 INFO:tasks.workunit.client.1.smithi066.stdout:0/788: getdents d2/d27/d41/d76/d93/db2/db5/d67 0 2024-02-28T16:31:28.438 INFO:tasks.workunit.client.1.smithi066.stdout:3/793: dwrite db/dc/d11/d1f/d78/f90 [0,4194304] 0 2024-02-28T16:31:28.438 INFO:tasks.workunit.client.1.smithi066.stdout:4/790: write d1/d23/d25/f59 [95868,12726] 0 2024-02-28T16:31:28.442 INFO:tasks.workunit.client.1.smithi066.stdout:8/663: unlink d0/d6/dc/l2e 0 2024-02-28T16:31:28.450 INFO:tasks.workunit.client.1.smithi066.stdout:4/791: unlink d1/d7a/fcd 0 2024-02-28T16:31:28.451 INFO:tasks.workunit.client.1.smithi066.stdout:1/648: dwrite d2/d18/d19/f89 [0,4194304] 0 2024-02-28T16:31:28.453 INFO:tasks.workunit.client.1.smithi066.stdout:0/789: write d2/d27/d41/d76/d93/db2/db5/d73/fc3 [9173689,11919] 0 2024-02-28T16:31:28.459 INFO:tasks.workunit.client.1.smithi066.stdout:8/664: rmdir d0/d6/dc/d16/d1a/d63/d3d 39 2024-02-28T16:31:28.459 INFO:tasks.workunit.client.1.smithi066.stdout:5/618: dwrite d3/d1b/d20/f63 [0,4194304] 0 2024-02-28T16:31:28.463 INFO:tasks.workunit.client.1.smithi066.stdout:8/665: write d0/d6/dc/d16/d4c/d42/fb2 [783144,84035] 0 2024-02-28T16:31:28.466 INFO:tasks.workunit.client.1.smithi066.stdout:8/666: fsync d0/db/d15/f7a 0 2024-02-28T16:31:28.466 INFO:tasks.workunit.client.1.smithi066.stdout:4/792: mknod d1/d7a/d104/c107 0 2024-02-28T16:31:28.469 INFO:tasks.workunit.client.1.smithi066.stdout:8/667: write d0/d6/f5d [783193,17423] 0 2024-02-28T16:31:28.471 INFO:tasks.workunit.client.1.smithi066.stdout:4/793: write d1/d23/d25/d43/f8b [669499,100187] 0 2024-02-28T16:31:28.473 INFO:tasks.workunit.client.1.smithi066.stdout:8/668: readlink d0/db/d12/d1d/d25/d65/l6d 0 2024-02-28T16:31:28.473 INFO:tasks.workunit.client.1.smithi066.stdout:1/649: mkdir d2/d27/dc7/da6/dd4 0 2024-02-28T16:31:28.476 INFO:tasks.workunit.client.1.smithi066.stdout:5/619: chown d3/d1b/c29 73 1 2024-02-28T16:31:28.477 INFO:tasks.workunit.client.1.smithi066.stdout:3/794: dread db/d2a/d44/d48/d52/dcb/ff0 [0,4194304] 0 2024-02-28T16:31:28.478 INFO:tasks.workunit.client.1.smithi066.stdout:8/669: write d0/db/d12/d1d/d25/d4f/f9f [305322,126586] 0 2024-02-28T16:31:28.480 INFO:tasks.workunit.client.1.smithi066.stdout:4/794: creat d1/d18/d41/dba/f108 x:0 0 0 2024-02-28T16:31:28.483 INFO:tasks.workunit.client.1.smithi066.stdout:8/670: chown d0/d6/dc/d16/d4c/c6a 155782 1 2024-02-28T16:31:28.485 INFO:tasks.workunit.client.1.smithi066.stdout:0/790: link d2/d27/d41/d76/d93/d33/d3f/faa d2/d27/d41/d76/d93/d10/d23/d6d/f103 0 2024-02-28T16:31:28.487 INFO:tasks.workunit.client.1.smithi066.stdout:1/650: unlink d2/d27/dc7/d48/la4 0 2024-02-28T16:31:28.491 INFO:tasks.workunit.client.1.smithi066.stdout:4/795: symlink d1/d18/d41/da7/dae/dc1/l109 0 2024-02-28T16:31:28.492 INFO:tasks.workunit.client.1.smithi066.stdout:8/671: creat d0/d6/dc/d16/d4c/fdb x:0 0 0 2024-02-28T16:31:28.492 INFO:tasks.workunit.client.1.smithi066.stdout:5/620: read d3/d1b/f31 [3134949,4049] 0 2024-02-28T16:31:28.494 INFO:tasks.workunit.client.1.smithi066.stdout:1/651: readlink d2/d18/d7a/db1/lc1 0 2024-02-28T16:31:28.495 INFO:tasks.workunit.client.1.smithi066.stdout:5/621: fdatasync d3/d4/dc/dd/d41/d5d/f69 0 2024-02-28T16:31:28.497 INFO:tasks.workunit.client.1.smithi066.stdout:3/795: rename db/d7a/f88 to db/dc/d11/d1f/f11d 0 2024-02-28T16:31:28.498 INFO:tasks.workunit.client.1.smithi066.stdout:5/622: write d3/d4/dc/dd/f9f [105078,123566] 0 2024-02-28T16:31:28.498 INFO:tasks.workunit.client.1.smithi066.stdout:4/796: mknod d1/d23/d25/d3e/d40/c10a 0 2024-02-28T16:31:28.501 INFO:tasks.workunit.client.1.smithi066.stdout:8/672: mknod d0/db/d12/cdc 0 2024-02-28T16:31:28.502 INFO:tasks.workunit.client.1.smithi066.stdout:3/796: read db/d2a/d44/d48/d5d/f91 [112799,118803] 0 2024-02-28T16:31:28.504 INFO:tasks.workunit.client.1.smithi066.stdout:5/623: mkdir d3/d1b/d20/dc1 0 2024-02-28T16:31:28.505 INFO:tasks.workunit.client.1.smithi066.stdout:1/652: creat d2/d27/fd5 x:0 0 0 2024-02-28T16:31:28.507 INFO:tasks.workunit.client.1.smithi066.stdout:5/624: fdatasync d3/d4/dc/d56/d93/fa7 0 2024-02-28T16:31:28.507 INFO:tasks.workunit.client.1.smithi066.stdout:3/797: mkdir db/dc/d35/d3d/d76/d89/dcc/d11e 0 2024-02-28T16:31:28.508 INFO:tasks.workunit.client.1.smithi066.stdout:8/673: write d0/d6/dc/d16/d4c/d6c/d6f/da7/fc5 [5450903,109685] 0 2024-02-28T16:31:28.511 INFO:tasks.workunit.client.1.smithi066.stdout:5/625: write d3/d4/dc/dd/d2e/f6f [2667182,56507] 0 2024-02-28T16:31:28.515 INFO:tasks.workunit.client.1.smithi066.stdout:1/653: truncate d2/f7 186347 0 2024-02-28T16:31:28.515 INFO:tasks.workunit.client.1.smithi066.stdout:3/798: read db/d13/d1a/d25/f5f [775304,120513] 0 2024-02-28T16:31:28.516 INFO:tasks.workunit.client.1.smithi066.stdout:8/674: fdatasync d0/d6/dc/d16/d4c/d6c/d6f/da7/dc9/fca 0 2024-02-28T16:31:28.523 INFO:tasks.workunit.client.1.smithi066.stdout:3/799: write db/d13/d1a/d33/f39 [6574973,38074] 0 2024-02-28T16:31:28.523 INFO:tasks.workunit.client.1.smithi066.stdout:1/654: chown d2/f7 18 1 2024-02-28T16:31:28.525 INFO:tasks.workunit.client.1.smithi066.stdout:8/675: creat d0/d6/dc/d16/d4c/d6c/d6f/da7/fdd x:0 0 0 2024-02-28T16:31:28.527 INFO:tasks.workunit.client.1.smithi066.stdout:1/655: mkdir d2/d27/dc7/d48/dd6 0 2024-02-28T16:31:28.529 INFO:tasks.workunit.client.1.smithi066.stdout:3/800: creat db/dc/db3/dc9/de4/f11f x:0 0 0 2024-02-28T16:31:28.532 INFO:tasks.workunit.client.1.smithi066.stdout:3/801: stat db/dc/d35/d3d/d76/d89/dcc/l28 0 2024-02-28T16:31:28.533 INFO:tasks.workunit.client.1.smithi066.stdout:7/633: sync 2024-02-28T16:31:28.533 INFO:tasks.workunit.client.1.smithi066.stdout:9/538: sync 2024-02-28T16:31:28.534 INFO:tasks.workunit.client.1.smithi066.stdout:1/656: truncate d2/d27/dc7/d51/f62 636237 0 2024-02-28T16:31:28.536 INFO:tasks.workunit.client.1.smithi066.stdout:7/634: chown dc/de/d17/d9d/cab 432318 1 2024-02-28T16:31:28.539 INFO:tasks.workunit.client.1.smithi066.stdout:3/802: stat db/d2a/d44/d48/d7d/db4/l9b 0 2024-02-28T16:31:28.544 INFO:tasks.workunit.client.1.smithi066.stdout:7/635: rmdir dc/de/d17/d1e 39 2024-02-28T16:31:28.549 INFO:tasks.workunit.client.1.smithi066.stdout:9/539: creat da/fb2 x:0 0 0 2024-02-28T16:31:28.549 INFO:tasks.workunit.client.1.smithi066.stdout:4/797: truncate d1/d23/d25/d77/fa3 4081116 0 2024-02-28T16:31:28.551 INFO:tasks.workunit.client.1.smithi066.stdout:7/636: creat dc/d29/fc7 x:0 0 0 2024-02-28T16:31:28.554 INFO:tasks.workunit.client.1.smithi066.stdout:1/657: dwrite d2/d18/d19/f1e [0,4194304] 0 2024-02-28T16:31:28.556 INFO:tasks.workunit.client.1.smithi066.stdout:9/540: write da/df/d70/d90/f93 [642230,91046] 0 2024-02-28T16:31:28.562 INFO:tasks.workunit.client.1.smithi066.stdout:8/676: dwrite d0/d6/dc/d16/d4c/d6c/f91 [8388608,4194304] 0 2024-02-28T16:31:28.564 INFO:tasks.workunit.client.1.smithi066.stdout:3/803: truncate db/dc/d35/d3d/d76/f4a 4038566 0 2024-02-28T16:31:28.568 INFO:tasks.workunit.client.1.smithi066.stdout:9/541: write da/df/d70/d90/f93 [1539334,111964] 0 2024-02-28T16:31:28.570 INFO:tasks.workunit.client.1.smithi066.stdout:7/637: mkdir dc/d6e/dc8 0 2024-02-28T16:31:28.573 INFO:tasks.workunit.client.1.smithi066.stdout:7/638: dread - dc/d56/f80 zero size 2024-02-28T16:31:28.573 INFO:tasks.workunit.client.1.smithi066.stdout:8/677: read d0/db/d12/d1d/d25/fb5 [1112588,130660] 0 2024-02-28T16:31:28.576 INFO:tasks.workunit.client.1.smithi066.stdout:7/639: stat dc/de/d17/c51 0 2024-02-28T16:31:28.579 INFO:tasks.workunit.client.1.smithi066.stdout:5/626: dwrite d3/d4/dc/dd/d41/d5d/f7c [0,4194304] 0 2024-02-28T16:31:28.583 INFO:tasks.workunit.client.1.smithi066.stdout:4/798: rename d1/d18/d4e/da0/fa2 to d1/d18/d41/dba/d100/f10b 0 2024-02-28T16:31:28.590 INFO:tasks.workunit.client.1.smithi066.stdout:1/658: mknod d2/d27/dc7/d54/d84/cd7 0 2024-02-28T16:31:28.591 INFO:tasks.workunit.client.1.smithi066.stdout:3/804: symlink db/d2a/d44/d48/d7d/l120 0 2024-02-28T16:31:28.594 INFO:tasks.workunit.client.1.smithi066.stdout:1/659: dread - d2/d27/d2b/d43/d46/d80/f98 zero size 2024-02-28T16:31:28.596 INFO:tasks.workunit.client.1.smithi066.stdout:1/660: stat d2/d27/d2b/d34/d9e/fae 0 2024-02-28T16:31:28.603 INFO:tasks.workunit.client.1.smithi066.stdout:5/627: symlink d3/d4/dc/d2f/d55/d74/lc2 0 2024-02-28T16:31:28.608 INFO:tasks.workunit.client.1.smithi066.stdout:7/640: dwrite dc/de/d17/d37/f54 [8388608,4194304] 0 2024-02-28T16:31:28.614 INFO:tasks.workunit.client.1.smithi066.stdout:9/542: symlink da/d3a/lb3 0 2024-02-28T16:31:28.614 INFO:tasks.workunit.client.1.smithi066.stdout:1/661: symlink d2/d27/dc7/d51/db4/ld8 0 2024-02-28T16:31:28.615 INFO:tasks.workunit.client.1.smithi066.stdout:4/799: rename d1/d23/c44 to d1/d23/d25/d3e/d40/d9c/c10c 0 2024-02-28T16:31:28.616 INFO:tasks.workunit.client.1.smithi066.stdout:1/662: fsync d2/d27/dc7/f49 0 2024-02-28T16:31:28.619 INFO:tasks.workunit.client.1.smithi066.stdout:9/543: readlink da/d22/d46/d56/d8f/d2d/d81/l85 0 2024-02-28T16:31:28.619 INFO:tasks.workunit.client.1.smithi066.stdout:1/663: read - d2/fd1 zero size 2024-02-28T16:31:28.620 INFO:tasks.workunit.client.1.smithi066.stdout:7/641: truncate dc/de/d17/d37/d47/d88/fb5 1551910 0 2024-02-28T16:31:28.625 INFO:tasks.workunit.client.1.smithi066.stdout:5/628: link d3/d4/dc/dd/d35/d92/fad d3/d4/fc3 0 2024-02-28T16:31:28.626 INFO:tasks.workunit.client.1.smithi066.stdout:3/805: rename db/d13/f18 to db/dc/d11/f121 0 2024-02-28T16:31:28.627 INFO:tasks.workunit.client.1.smithi066.stdout:9/544: creat da/d22/d46/d56/d8f/d28/d63/fb4 x:0 0 0 2024-02-28T16:31:28.629 INFO:tasks.workunit.client.1.smithi066.stdout:1/664: write d2/d27/d2b/d34/d9e/fd2 [323364,127559] 0 2024-02-28T16:31:28.639 INFO:tasks.workunit.client.1.smithi066.stdout:9/545: creat da/d22/d46/d56/fb5 x:0 0 0 2024-02-28T16:31:28.651 INFO:tasks.workunit.client.1.smithi066.stdout:5/629: dwrite d3/f98 [4194304,4194304] 0 2024-02-28T16:31:28.661 INFO:tasks.workunit.client.1.smithi066.stdout:4/800: dwrite d1/f9b [0,4194304] 0 2024-02-28T16:31:28.669 INFO:tasks.workunit.client.1.smithi066.stdout:3/806: dread db/dc/d11/d1f/f5e [0,4194304] 0 2024-02-28T16:31:28.671 INFO:tasks.workunit.client.1.smithi066.stdout:3/807: write db/dc/d11/d1f/fd0 [326365,23910] 0 2024-02-28T16:31:28.673 INFO:tasks.workunit.client.1.smithi066.stdout:9/546: symlink da/d22/d46/d56/lb6 0 2024-02-28T16:31:28.675 INFO:tasks.workunit.client.1.smithi066.stdout:5/630: mknod d3/d1b/d7d/cc4 0 2024-02-28T16:31:28.678 INFO:tasks.workunit.client.1.smithi066.stdout:4/801: read d1/d23/f27 [612243,17316] 0 2024-02-28T16:31:28.679 INFO:tasks.workunit.client.1.smithi066.stdout:1/665: getdents d2/d27/d2b/d34/d9e 0 2024-02-28T16:31:28.680 INFO:tasks.workunit.client.1.smithi066.stdout:5/631: mkdir d3/d4/dc/dd/d35/dc5 0 2024-02-28T16:31:28.683 INFO:tasks.workunit.client.1.smithi066.stdout:5/632: mknod d3/cc6 0 2024-02-28T16:31:28.684 INFO:tasks.workunit.client.1.smithi066.stdout:1/666: mknod d2/d27/d2b/d43/d46/cd9 0 2024-02-28T16:31:28.685 INFO:tasks.workunit.client.1.smithi066.stdout:5/633: write d3/d4/dc/d2f/d55/d74/fbb [1579714,4939] 0 2024-02-28T16:31:28.687 INFO:tasks.workunit.client.1.smithi066.stdout:4/802: creat d1/f10d x:0 0 0 2024-02-28T16:31:28.688 INFO:tasks.workunit.client.1.smithi066.stdout:1/667: mknod d2/d27/d2b/d43/cda 0 2024-02-28T16:31:28.689 INFO:tasks.workunit.client.1.smithi066.stdout:5/634: fdatasync d3/d4/dc/f21 0 2024-02-28T16:31:28.690 INFO:tasks.workunit.client.1.smithi066.stdout:2/711: sync 2024-02-28T16:31:28.692 INFO:tasks.workunit.client.1.smithi066.stdout:5/635: chown d3/d4/dc/dd/d2e/l6a 7246122 1 2024-02-28T16:31:28.696 INFO:tasks.workunit.client.1.smithi066.stdout:2/712: mkdir d0/d12/d1a/d23/d29/d21/d2f/de3 0 2024-02-28T16:31:28.700 INFO:tasks.workunit.client.1.smithi066.stdout:5/636: read d3/d4/dc/dd/d35/d86/f88 [4121258,59545] 0 2024-02-28T16:31:28.703 INFO:tasks.workunit.client.1.smithi066.stdout:4/803: dread d1/d18/d41/dba/d100/f10b [0,4194304] 0 2024-02-28T16:31:28.703 INFO:tasks.workunit.client.1.smithi066.stdout:2/713: creat d0/d12/d1a/fe4 x:0 0 0 2024-02-28T16:31:28.704 INFO:tasks.workunit.client.1.smithi066.stdout:4/804: chown d1/d23/d25/d3e/d40/fdd 1731822 1 2024-02-28T16:31:28.711 INFO:tasks.workunit.client.1.smithi066.stdout:2/714: creat d0/d12/d1a/d39/d7c/d87/fe5 x:0 0 0 2024-02-28T16:31:28.716 INFO:tasks.workunit.client.1.smithi066.stdout:4/805: symlink d1/d18/da8/dcb/l10e 0 2024-02-28T16:31:28.717 INFO:tasks.workunit.client.1.smithi066.stdout:2/715: unlink d0/d12/d1a/d23/d29/d21/l50 0 2024-02-28T16:31:28.723 INFO:tasks.workunit.client.1.smithi066.stdout:2/716: unlink d0/d12/d2e/f59 0 2024-02-28T16:31:28.726 INFO:tasks.workunit.client.1.smithi066.stdout:4/806: mknod d1/d18/c10f 0 2024-02-28T16:31:28.727 INFO:tasks.workunit.client.1.smithi066.stdout:2/717: symlink d0/d12/d1a/d23/le6 0 2024-02-28T16:31:28.729 INFO:tasks.workunit.client.1.smithi066.stdout:4/807: unlink d1/d23/d25/d3e/f3f 0 2024-02-28T16:31:28.730 INFO:tasks.workunit.client.1.smithi066.stdout:5/637: dread d3/d4/dc/dd/d2e/d18/f1a [4194304,4194304] 0 2024-02-28T16:31:28.731 INFO:tasks.workunit.client.1.smithi066.stdout:2/718: unlink d0/d12/d1a/d39/d7c/fae 0 2024-02-28T16:31:28.732 INFO:tasks.workunit.client.1.smithi066.stdout:4/808: symlink d1/d18/d7e/l110 0 2024-02-28T16:31:28.736 INFO:tasks.workunit.client.1.smithi066.stdout:5/638: creat d3/d4/dc/dd/d41/d7a/dac/d60/fc7 x:0 0 0 2024-02-28T16:31:28.738 INFO:tasks.workunit.client.1.smithi066.stdout:4/809: symlink d1/d18/d7e/l111 0 2024-02-28T16:31:28.739 INFO:tasks.workunit.client.1.smithi066.stdout:5/639: fdatasync d3/d4/dc/dd/f9f 0 2024-02-28T16:31:28.744 INFO:tasks.workunit.client.1.smithi066.stdout:3/808: write db/d13/d14/ff3 [690149,104574] 0 2024-02-28T16:31:28.764 INFO:tasks.workunit.client.1.smithi066.stdout:3/809: getdents db/d13/d14/dad 0 2024-02-28T16:31:28.765 INFO:tasks.workunit.client.1.smithi066.stdout:5/640: write d3/d4/dc/f1d [1875683,46984] 0 2024-02-28T16:31:28.767 INFO:tasks.workunit.client.1.smithi066.stdout:1/668: dwrite d2/d18/d7a/f8b [0,4194304] 0 2024-02-28T16:31:28.768 INFO:tasks.workunit.client.1.smithi066.stdout:2/719: dread d0/f1f [4194304,4194304] 0 2024-02-28T16:31:28.770 INFO:tasks.workunit.client.1.smithi066.stdout:5/641: stat d3/d4/dc/f1d 0 2024-02-28T16:31:28.772 INFO:tasks.workunit.client.1.smithi066.stdout:3/810: creat db/d13/d14/dda/f122 x:0 0 0 2024-02-28T16:31:28.773 INFO:tasks.workunit.client.1.smithi066.stdout:1/669: fsync d2/d18/d7a/d8d/f8f 0 2024-02-28T16:31:28.779 INFO:tasks.workunit.client.1.smithi066.stdout:5/642: symlink d3/d1b/d20/dc1/lc8 0 2024-02-28T16:31:28.780 INFO:tasks.workunit.client.1.smithi066.stdout:2/720: fdatasync d0/d12/d1a/d23/d5d/fbd 0 2024-02-28T16:31:28.783 INFO:tasks.workunit.client.1.smithi066.stdout:4/810: dwrite d1/d23/d25/d43/f5e [0,4194304] 0 2024-02-28T16:31:28.787 INFO:tasks.workunit.client.1.smithi066.stdout:4/811: truncate d1/d23/d25/f59 904701 0 2024-02-28T16:31:28.793 INFO:tasks.workunit.client.1.smithi066.stdout:5/643: creat d3/d4/dc/dd/d41/d7f/d80/fc9 x:0 0 0 2024-02-28T16:31:28.798 INFO:tasks.workunit.client.1.smithi066.stdout:1/670: getdents d2/d27/dc7/d54/dc4 0 2024-02-28T16:31:28.804 INFO:tasks.workunit.client.1.smithi066.stdout:4/812: unlink d1/d23/d25/c26 0 2024-02-28T16:31:28.807 INFO:tasks.workunit.client.1.smithi066.stdout:5/644: truncate d3/d1b/d39/f47 2224896 0 2024-02-28T16:31:28.807 INFO:tasks.workunit.client.1.smithi066.stdout:4/813: chown d1/d18/f4a 1231 1 2024-02-28T16:31:28.808 INFO:tasks.workunit.client.1.smithi066.stdout:1/671: read d2/d27/d2b/f39 [168249,61735] 0 2024-02-28T16:31:28.810 INFO:tasks.workunit.client.1.smithi066.stdout:5/645: chown d3/d4/dc/dd/l34 6635374 1 2024-02-28T16:31:28.810 INFO:tasks.workunit.client.1.smithi066.stdout:4/814: creat d1/d18/d7e/f112 x:0 0 0 2024-02-28T16:31:28.818 INFO:tasks.workunit.client.1.smithi066.stdout:1/672: creat d2/d27/d2b/d34/da9/fdb x:0 0 0 2024-02-28T16:31:28.819 INFO:tasks.workunit.client.1.smithi066.stdout:4/815: creat d1/d23/d25/d3e/d40/d9c/f113 x:0 0 0 2024-02-28T16:31:28.820 INFO:tasks.workunit.client.1.smithi066.stdout:3/811: truncate db/d2a/d44/d48/d5d/fa9 3598367 0 2024-02-28T16:31:28.828 INFO:tasks.workunit.client.1.smithi066.stdout:3/812: fsync db/d13/d14/d2d/f9e 0 2024-02-28T16:31:28.832 INFO:tasks.workunit.client.1.smithi066.stdout:4/816: creat d1/d23/d25/d43/ded/f114 x:0 0 0 2024-02-28T16:31:28.836 INFO:tasks.workunit.client.1.smithi066.stdout:5/646: dwrite d3/d4/dc/d2f/d55/f70 [0,4194304] 0 2024-02-28T16:31:28.843 INFO:tasks.workunit.client.1.smithi066.stdout:2/721: truncate d0/d12/d1a/d39/f42 2961475 0 2024-02-28T16:31:28.846 INFO:tasks.workunit.client.1.smithi066.stdout:0/791: sync 2024-02-28T16:31:28.846 INFO:tasks.workunit.client.1.smithi066.stdout:6/642: sync 2024-02-28T16:31:28.847 INFO:tasks.workunit.client.1.smithi066.stdout:5/647: symlink d3/d4/dc/dd/d41/d5d/lca 0 2024-02-28T16:31:28.856 INFO:tasks.workunit.client.1.smithi066.stdout:6/643: creat d2/d2c/d6e/fe5 x:0 0 0 2024-02-28T16:31:28.858 INFO:tasks.workunit.client.1.smithi066.stdout:5/648: fsync d3/d1b/d39/fa6 0 2024-02-28T16:31:28.858 INFO:tasks.workunit.client.1.smithi066.stdout:4/817: dread d1/f1e [0,4194304] 0 2024-02-28T16:31:28.864 INFO:tasks.workunit.client.1.smithi066.stdout:5/649: truncate d3/d4/dc/d2f/d55/d74/fbb 2504851 0 2024-02-28T16:31:28.864 INFO:tasks.workunit.client.1.smithi066.stdout:6/644: write d2/d26/d32/d46/f7c [1154491,93045] 0 2024-02-28T16:31:28.865 INFO:tasks.workunit.client.1.smithi066.stdout:0/792: symlink d2/d27/d41/d76/d93/d10/d23/l104 0 2024-02-28T16:31:28.869 INFO:tasks.workunit.client.1.smithi066.stdout:6/645: fsync d2/da/d19/d75/dbd/dc6/fd1 0 2024-02-28T16:31:28.870 INFO:tasks.workunit.client.1.smithi066.stdout:1/673: write d2/d27/dc7/fc3 [1813045,75016] 0 2024-02-28T16:31:28.881 INFO:tasks.workunit.client.1.smithi066.stdout:2/722: mknod d0/d12/d1a/d23/d29/d21/d2f/de3/ce7 0 2024-02-28T16:31:28.883 INFO:tasks.workunit.client.1.smithi066.stdout:1/674: mknod d2/d27/dc7/d51/cdc 0 2024-02-28T16:31:28.886 INFO:tasks.workunit.client.1.smithi066.stdout:4/818: dread d1/d18/d41/f49 [0,4194304] 0 2024-02-28T16:31:28.892 INFO:tasks.workunit.client.1.smithi066.stdout:6/646: mkdir d2/d2c/da1/d5d/d63/d6a/de6 0 2024-02-28T16:31:28.892 INFO:tasks.workunit.client.1.smithi066.stdout:5/650: link d3/d4/dc/dd/d2e/d18/f2c d3/d1b/fcb 0 2024-02-28T16:31:28.894 INFO:tasks.workunit.client.1.smithi066.stdout:6/647: fdatasync d2/da/f76 0 2024-02-28T16:31:28.895 INFO:tasks.workunit.client.1.smithi066.stdout:1/675: creat d2/d18/d6b/dba/fdd x:0 0 0 2024-02-28T16:31:28.896 INFO:tasks.workunit.client.1.smithi066.stdout:5/651: read d3/d4/f61 [326132,97759] 0 2024-02-28T16:31:28.902 INFO:tasks.workunit.client.1.smithi066.stdout:5/652: chown d3/d4/dc/dd/d41/d7a/dac/d60/fb2 51 1 2024-02-28T16:31:28.904 INFO:tasks.workunit.client.1.smithi066.stdout:1/676: unlink d2/d27/d2b/d34/f55 0 2024-02-28T16:31:28.906 INFO:tasks.workunit.client.1.smithi066.stdout:4/819: mkdir d1/d23/d25/d115 0 2024-02-28T16:31:28.908 INFO:tasks.workunit.client.1.smithi066.stdout:1/677: write d2/d27/d2b/d43/faf [84151,48098] 0 2024-02-28T16:31:28.908 INFO:tasks.workunit.client.1.smithi066.stdout:6/648: creat d2/da/dcb/fe7 x:0 0 0 2024-02-28T16:31:28.910 INFO:tasks.workunit.client.1.smithi066.stdout:0/793: link d2/d27/d41/d76/d93/db2/db5/d73/ce9 d2/d27/d41/d76/d93/db2/db5/d67/d94/de4/da1/c105 0 2024-02-28T16:31:28.912 INFO:tasks.workunit.client.1.smithi066.stdout:5/653: truncate d3/d4/fb 2763668 0 2024-02-28T16:31:28.914 INFO:tasks.workunit.client.1.smithi066.stdout:5/654: chown d3/d1b/d39/l3d 66732116 1 2024-02-28T16:31:28.918 INFO:tasks.workunit.client.1.smithi066.stdout:2/723: dread d0/d12/d1a/f2b [0,4194304] 0 2024-02-28T16:31:28.921 INFO:tasks.workunit.client.1.smithi066.stdout:6/649: unlink d2/da/f21 0 2024-02-28T16:31:28.927 INFO:tasks.workunit.client.1.smithi066.stdout:0/794: creat d2/d27/d41/d76/d93/f106 x:0 0 0 2024-02-28T16:31:28.931 INFO:tasks.workunit.client.1.smithi066.stdout:5/655: creat d3/d4/dc/dd/d35/fcc x:0 0 0 2024-02-28T16:31:28.935 INFO:tasks.workunit.client.1.smithi066.stdout:5/656: write d3/d4/dc/dd/d41/d5d/f7c [3953600,120216] 0 2024-02-28T16:31:28.942 INFO:tasks.workunit.client.1.smithi066.stdout:0/795: mkdir d2/d27/d41/d76/d93/d10/d107 0 2024-02-28T16:31:28.947 INFO:tasks.workunit.client.1.smithi066.stdout:4/820: dread d1/d18/d41/f47 [0,4194304] 0 2024-02-28T16:31:28.954 INFO:tasks.workunit.client.1.smithi066.stdout:2/724: rename d0/d12/d1a/d23/d29/f6e to d0/d12/d2e/fe8 0 2024-02-28T16:31:28.960 INFO:tasks.workunit.client.1.smithi066.stdout:8/678: sync 2024-02-28T16:31:28.961 INFO:tasks.workunit.client.1.smithi066.stdout:5/657: write d3/d4/dc/dd/d2e/d18/f4b [784518,129947] 0 2024-02-28T16:31:28.967 INFO:tasks.workunit.client.1.smithi066.stdout:6/650: dread d2/f6f [0,4194304] 0 2024-02-28T16:31:28.967 INFO:tasks.workunit.client.1.smithi066.stdout:4/821: mkdir d1/d18/d41/dba/d116 0 2024-02-28T16:31:28.967 INFO:tasks.workunit.client.1.smithi066.stdout:1/678: dwrite d2/d27/dc7/f82 [0,4194304] 0 2024-02-28T16:31:28.970 INFO:tasks.workunit.client.1.smithi066.stdout:6/651: chown d2/da/d19/d75/dbd/ccf 13173 1 2024-02-28T16:31:28.971 INFO:tasks.workunit.client.1.smithi066.stdout:4/822: chown d1/d18/d4e/l6c 61 1 2024-02-28T16:31:28.973 INFO:tasks.workunit.client.1.smithi066.stdout:2/725: creat d0/d12/d1a/d23/d29/d21/d2f/d67/fe9 x:0 0 0 2024-02-28T16:31:28.973 INFO:tasks.workunit.client.1.smithi066.stdout:1/679: chown d2/d27/dc7/d51/db4 14707 1 2024-02-28T16:31:28.986 INFO:tasks.workunit.client.1.smithi066.stdout:6/652: fsync d2/d2c/da1/d5d/d99/f52 0 2024-02-28T16:31:28.986 INFO:tasks.workunit.client.1.smithi066.stdout:4/823: fsync d1/d23/d25/f32 0 2024-02-28T16:31:28.991 INFO:tasks.workunit.client.1.smithi066.stdout:2/726: rename d0/d12/d2e/d3b/fb1 to d0/d12/d1a/d23/d5d/fea 0 2024-02-28T16:31:28.993 INFO:tasks.workunit.client.1.smithi066.stdout:0/796: truncate d2/d27/d41/d76/d93/db2/db5/d52/f81 2409911 0 2024-02-28T16:31:28.994 INFO:tasks.workunit.client.1.smithi066.stdout:7/642: sync 2024-02-28T16:31:28.995 INFO:tasks.workunit.client.1.smithi066.stdout:3/813: sync 2024-02-28T16:31:28.995 INFO:tasks.workunit.client.1.smithi066.stdout:8/679: write d0/d6/dc/d16/d1a/d63/d3d/f5b [345712,41682] 0 2024-02-28T16:31:28.996 INFO:tasks.workunit.client.1.smithi066.stdout:6/653: fdatasync d2/f6f 0 2024-02-28T16:31:28.996 INFO:tasks.workunit.client.1.smithi066.stdout:9/547: sync 2024-02-28T16:31:29.009 INFO:tasks.workunit.client.1.smithi066.stdout:4/824: getdents d1/d18/d55/dc4 0 2024-02-28T16:31:29.010 INFO:tasks.workunit.client.1.smithi066.stdout:9/548: read da/d3a/f5f [5898823,76247] 0 2024-02-28T16:31:29.010 INFO:tasks.workunit.client.1.smithi066.stdout:3/814: creat db/d2a/dd4/f123 x:0 0 0 2024-02-28T16:31:29.011 INFO:tasks.workunit.client.1.smithi066.stdout:8/680: read d0/db/f27 [650562,87487] 0 2024-02-28T16:31:29.014 INFO:tasks.workunit.client.1.smithi066.stdout:1/680: dwrite d2/d27/dc7/da6/faa [0,4194304] 0 2024-02-28T16:31:29.022 INFO:tasks.workunit.client.1.smithi066.stdout:7/643: write dc/de/d17/d1e/d45/f8f [4470400,71626] 0 2024-02-28T16:31:29.023 INFO:tasks.workunit.client.1.smithi066.stdout:5/658: dwrite d3/d1b/d39/f47 [0,4194304] 0 2024-02-28T16:31:29.033 INFO:tasks.workunit.client.1.smithi066.stdout:3/815: mkdir db/d13/d1a/d33/d124 0 2024-02-28T16:31:29.036 INFO:tasks.workunit.client.1.smithi066.stdout:3/816: fsync db/d13/d14/dad/fe1 0 2024-02-28T16:31:29.036 INFO:tasks.workunit.client.1.smithi066.stdout:2/727: creat d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/feb x:0 0 0 2024-02-28T16:31:29.036 INFO:tasks.workunit.client.1.smithi066.stdout:6/654: unlink d2/d26/c77 0 2024-02-28T16:31:29.041 INFO:tasks.workunit.client.1.smithi066.stdout:6/655: write d2/d2c/da1/d5d/d99/f5e [3848899,61647] 0 2024-02-28T16:31:29.042 INFO:tasks.workunit.client.1.smithi066.stdout:3/817: fdatasync db/d2a/ff7 0 2024-02-28T16:31:29.045 INFO:tasks.workunit.client.1.smithi066.stdout:5/659: creat d3/d4/dc/dd/d35/d86/fcd x:0 0 0 2024-02-28T16:31:29.046 INFO:tasks.workunit.client.1.smithi066.stdout:7/644: creat dc/de/d17/d1e/da1/fc9 x:0 0 0 2024-02-28T16:31:29.047 INFO:tasks.workunit.client.1.smithi066.stdout:2/728: unlink d0/d12/fdb 0 2024-02-28T16:31:29.056 INFO:tasks.workunit.client.1.smithi066.stdout:5/660: dread - d3/d4/dc/d56/fbc zero size 2024-02-28T16:31:29.056 INFO:tasks.workunit.client.1.smithi066.stdout:0/797: getdents d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/dd1 0 2024-02-28T16:31:29.062 INFO:tasks.workunit.client.1.smithi066.stdout:8/681: truncate d0/d6/dc/d16/d4c/d6c/d6f/f39 3932516 0 2024-02-28T16:31:29.062 INFO:tasks.workunit.client.1.smithi066.stdout:4/825: write d1/d23/d25/d3e/d40/d9c/fc7 [122348,1552] 0 2024-02-28T16:31:29.063 INFO:tasks.workunit.client.1.smithi066.stdout:2/729: stat d0/d12/d1a/d23/d29/fa8 0 2024-02-28T16:31:29.065 INFO:tasks.workunit.client.1.smithi066.stdout:6/656: truncate d2/d26/f31 1265811 0 2024-02-28T16:31:29.066 INFO:tasks.workunit.client.1.smithi066.stdout:5/661: read - d3/d4/dc/dd/d41/d7a/dac/d60/fc7 zero size 2024-02-28T16:31:29.070 INFO:tasks.workunit.client.1.smithi066.stdout:4/826: mknod d1/d18/d4e/c117 0 2024-02-28T16:31:29.072 INFO:tasks.workunit.client.1.smithi066.stdout:7/645: rmdir dc/de/d17/d1e/da1/db7/dc6 0 2024-02-28T16:31:29.072 INFO:tasks.workunit.client.1.smithi066.stdout:8/682: read d0/d6/dc/d16/d1a/d63/d58/f70 [2520719,107060] 0 2024-02-28T16:31:29.075 INFO:tasks.workunit.client.1.smithi066.stdout:9/549: dwrite da/d22/d46/d56/d8f/f65 [0,4194304] 0 2024-02-28T16:31:29.076 INFO:tasks.workunit.client.1.smithi066.stdout:4/827: dread - d1/d7a/fd1 zero size 2024-02-28T16:31:29.078 INFO:tasks.workunit.client.1.smithi066.stdout:3/818: dread db/ff [0,4194304] 0 2024-02-28T16:31:29.078 INFO:tasks.workunit.client.1.smithi066.stdout:2/730: mkdir d0/d12/d1a/d23/dec 0 2024-02-28T16:31:29.084 INFO:tasks.workunit.client.1.smithi066.stdout:5/662: creat d3/d4/dc/dd/d2e/fce x:0 0 0 2024-02-28T16:31:29.086 INFO:tasks.workunit.client.1.smithi066.stdout:6/657: symlink d2/d26/le8 0 2024-02-28T16:31:29.087 INFO:tasks.workunit.client.1.smithi066.stdout:3/819: write db/dc/d11/d1f/d78/fbf [4008512,2630] 0 2024-02-28T16:31:29.089 INFO:tasks.workunit.client.1.smithi066.stdout:7/646: creat dc/de/fca x:0 0 0 2024-02-28T16:31:29.090 INFO:tasks.workunit.client.1.smithi066.stdout:8/683: creat d0/db/d12/d1d/d25/d4f/fde x:0 0 0 2024-02-28T16:31:29.092 INFO:tasks.workunit.client.1.smithi066.stdout:7/647: readlink dc/de/l2f 0 2024-02-28T16:31:29.099 INFO:tasks.workunit.client.1.smithi066.stdout:1/681: dread d2/d27/d2b/d43/d46/fb5 [0,4194304] 0 2024-02-28T16:31:29.099 INFO:tasks.workunit.client.1.smithi066.stdout:5/663: creat d3/d1b/d20/dc1/fcf x:0 0 0 2024-02-28T16:31:29.101 INFO:tasks.workunit.client.1.smithi066.stdout:2/731: rmdir d0/d12/d1a/d23/dec 0 2024-02-28T16:31:29.102 INFO:tasks.workunit.client.1.smithi066.stdout:8/684: symlink d0/d6/dc/d16/d4c/d6c/d6f/da7/ldf 0 2024-02-28T16:31:29.104 INFO:tasks.workunit.client.1.smithi066.stdout:7/648: write dc/d29/f2e [3052856,59371] 0 2024-02-28T16:31:29.104 INFO:tasks.workunit.client.1.smithi066.stdout:4/828: link d1/d18/da8/fbe d1/d23/d25/d3e/f118 0 2024-02-28T16:31:29.108 INFO:tasks.workunit.client.1.smithi066.stdout:6/658: mknod d2/d26/d32/db9/ce9 0 2024-02-28T16:31:29.110 INFO:tasks.workunit.client.1.smithi066.stdout:1/682: read d2/d18/d19/f24 [2353676,97883] 0 2024-02-28T16:31:29.112 INFO:tasks.workunit.client.1.smithi066.stdout:4/829: chown d1/d18/c10f 6250397 1 2024-02-28T16:31:29.112 INFO:tasks.workunit.client.1.smithi066.stdout:3/820: creat db/d2a/d44/f125 x:0 0 0 2024-02-28T16:31:29.113 INFO:tasks.workunit.client.1.smithi066.stdout:2/732: creat d0/d12/d3e/d9e/fed x:0 0 0 2024-02-28T16:31:29.113 INFO:tasks.workunit.client.1.smithi066.stdout:5/664: symlink d3/d4/dc/ld0 0 2024-02-28T16:31:29.114 INFO:tasks.workunit.client.1.smithi066.stdout:1/683: creat d2/d18/d7a/dbc/fde x:0 0 0 2024-02-28T16:31:29.114 INFO:tasks.workunit.client.1.smithi066.stdout:7/649: write dc/d29/d92/da6/fa7 [2302453,46903] 0 2024-02-28T16:31:29.116 INFO:tasks.workunit.client.1.smithi066.stdout:3/821: dread - db/dc/d11/d1f/d78/fa3 zero size 2024-02-28T16:31:29.119 INFO:tasks.workunit.client.1.smithi066.stdout:5/665: dread - d3/d4/dc/dd/d35/d86/fcd zero size 2024-02-28T16:31:29.120 INFO:tasks.workunit.client.1.smithi066.stdout:6/659: symlink d2/da/d19/d9c/lea 0 2024-02-28T16:31:29.121 INFO:tasks.workunit.client.1.smithi066.stdout:2/733: write d0/d12/d1a/d39/d7c/d87/fe5 [955085,24858] 0 2024-02-28T16:31:29.128 INFO:tasks.workunit.client.1.smithi066.stdout:6/660: readlink d2/d2c/da1/d5d/l67 0 2024-02-28T16:31:29.136 INFO:tasks.workunit.client.1.smithi066.stdout:5/666: dread - d3/d4/dc/dd/d2e/d8f/f9a zero size 2024-02-28T16:31:29.136 INFO:tasks.workunit.client.1.smithi066.stdout:8/685: dwrite d0/d6/dc/d16/d1a/d63/d3d/f5b [0,4194304] 0 2024-02-28T16:31:29.136 INFO:tasks.workunit.client.1.smithi066.stdout:3/822: truncate db/d2a/d44/f4f 2617040 0 2024-02-28T16:31:29.137 INFO:tasks.workunit.client.1.smithi066.stdout:8/686: chown d0/d6/dc/d16/d4c/d6c/d6f/da7/dc9/fca 43906 1 2024-02-28T16:31:29.137 INFO:tasks.workunit.client.1.smithi066.stdout:2/734: symlink d0/d12/d3e/d9e/lee 0 2024-02-28T16:31:29.142 INFO:tasks.workunit.client.1.smithi066.stdout:8/687: chown d0/db/d12/c40 255323396 1 2024-02-28T16:31:29.147 INFO:tasks.workunit.client.1.smithi066.stdout:1/684: write d2/d27/d2b/f39 [20320,35366] 0 2024-02-28T16:31:29.150 INFO:tasks.workunit.client.1.smithi066.stdout:3/823: truncate db/dc/d11/d1f/f11d 617285 0 2024-02-28T16:31:29.163 INFO:tasks.workunit.client.1.smithi066.stdout:3/824: mknod db/dc/d35/c126 0 2024-02-28T16:31:29.163 INFO:tasks.workunit.client.1.smithi066.stdout:6/661: write d2/da/d19/d9c/d3d/f48 [2544663,20350] 0 2024-02-28T16:31:29.175 INFO:tasks.workunit.client.1.smithi066.stdout:7/650: dread f4 [0,4194304] 0 2024-02-28T16:31:29.179 INFO:tasks.workunit.client.1.smithi066.stdout:6/662: symlink d2/da/d19/d75/dbd/dc6/leb 0 2024-02-28T16:31:29.182 INFO:tasks.workunit.client.1.smithi066.stdout:2/735: dwrite d0/d12/d1a/d39/d7c/d87/fe5 [0,4194304] 0 2024-02-28T16:31:29.182 INFO:tasks.workunit.client.1.smithi066.stdout:8/688: dwrite d0/db/d12/d1d/d25/d72/f77 [0,4194304] 0 2024-02-28T16:31:29.183 INFO:tasks.workunit.client.1.smithi066.stdout:4/830: dread d1/f4f [0,4194304] 0 2024-02-28T16:31:29.188 INFO:tasks.workunit.client.1.smithi066.stdout:6/663: creat d2/d26/dbe/fec x:0 0 0 2024-02-28T16:31:29.188 INFO:tasks.workunit.client.1.smithi066.stdout:2/736: write d0/d12/d1a/f3a [1243873,5176] 0 2024-02-28T16:31:29.190 INFO:tasks.workunit.client.1.smithi066.stdout:4/831: chown d1/d7a/d104 196 1 2024-02-28T16:31:29.194 INFO:tasks.workunit.client.1.smithi066.stdout:6/664: fsync d2/da/d19/d75/f92 0 2024-02-28T16:31:29.196 INFO:tasks.workunit.client.1.smithi066.stdout:4/832: chown d1/d18/d4e/f71 1675181 1 2024-02-28T16:31:29.197 INFO:tasks.workunit.client.1.smithi066.stdout:1/685: dwrite d2/d27/f3c [0,4194304] 0 2024-02-28T16:31:29.197 INFO:tasks.workunit.client.1.smithi066.stdout:2/737: chown d0/d12/d1a/d23/d29/fa8 3283 1 2024-02-28T16:31:29.203 INFO:tasks.workunit.client.1.smithi066.stdout:6/665: readlink d2/da/l61 0 2024-02-28T16:31:29.207 INFO:tasks.workunit.client.1.smithi066.stdout:7/651: dwrite f1 [0,4194304] 0 2024-02-28T16:31:29.217 INFO:tasks.workunit.client.1.smithi066.stdout:1/686: getdents d2 0 2024-02-28T16:31:29.218 INFO:tasks.workunit.client.1.smithi066.stdout:2/738: dread d0/d12/d1a/d23/d29/d21/d2f/fa2 [0,4194304] 0 2024-02-28T16:31:29.223 INFO:tasks.workunit.client.1.smithi066.stdout:7/652: link dc/d29/d92/ca4 dc/de/d17/d1e/ccb 0 2024-02-28T16:31:29.224 INFO:tasks.workunit.client.1.smithi066.stdout:7/653: readlink dc/l42 0 2024-02-28T16:31:29.225 INFO:tasks.workunit.client.1.smithi066.stdout:1/687: mkdir d2/dd0/ddf 0 2024-02-28T16:31:29.227 INFO:tasks.workunit.client.1.smithi066.stdout:2/739: symlink d0/d12/d3e/d9e/dce/lef 0 2024-02-28T16:31:29.233 INFO:tasks.workunit.client.1.smithi066.stdout:7/654: fsync dc/de/d17/d1e/d45/f79 0 2024-02-28T16:31:29.234 INFO:tasks.workunit.client.1.smithi066.stdout:1/688: rename d2/d27/f5b to d2/d18/d7a/dbc/fe0 0 2024-02-28T16:31:29.237 INFO:tasks.workunit.client.1.smithi066.stdout:6/666: dwrite d2/da/d19/f93 [4194304,4194304] 0 2024-02-28T16:31:29.241 INFO:tasks.workunit.client.1.smithi066.stdout:7/655: creat dc/d6e/dc8/fcc x:0 0 0 2024-02-28T16:31:29.243 INFO:tasks.workunit.client.1.smithi066.stdout:8/689: dwrite d0/d6/f1f [0,4194304] 0 2024-02-28T16:31:29.244 INFO:tasks.workunit.client.1.smithi066.stdout:7/656: chown dc/de/d17/d1e/d45/l57 1793 1 2024-02-28T16:31:29.248 INFO:tasks.workunit.client.1.smithi066.stdout:2/740: write d0/d12/d2e/d7a/f99 [1840135,3444] 0 2024-02-28T16:31:29.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:28 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:29.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:28 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:29.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:28 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:29.254 INFO:tasks.workunit.client.1.smithi066.stdout:8/690: mkdir d0/db/d12/d1d/d25/d4f/de0 0 2024-02-28T16:31:29.261 INFO:tasks.workunit.client.1.smithi066.stdout:8/691: write d0/f43 [1645010,18582] 0 2024-02-28T16:31:29.268 INFO:tasks.workunit.client.1.smithi066.stdout:4/833: dread d1/d23/d25/d43/f5d [4194304,4194304] 0 2024-02-28T16:31:29.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:28 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:29.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:28 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:29.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:28 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:29.276 INFO:tasks.workunit.client.1.smithi066.stdout:8/692: rename d0/d6/dc/d16/d4c/d6c/d6f/da7/fdd to d0/d6/dc/d16/d1a/d63/fe1 0 2024-02-28T16:31:29.279 INFO:tasks.workunit.client.1.smithi066.stdout:2/741: dwrite d0/d12/d1a/d39/d7c/d87/dc4/fcd [0,4194304] 0 2024-02-28T16:31:29.291 INFO:tasks.workunit.client.1.smithi066.stdout:4/834: rename d1/d23/d25/c3b to d1/d18/d41/dba/c119 0 2024-02-28T16:31:29.295 INFO:tasks.workunit.client.1.smithi066.stdout:6/667: dread d2/da/f22 [4194304,4194304] 0 2024-02-28T16:31:29.296 INFO:tasks.workunit.client.1.smithi066.stdout:4/835: write d1/d7a/fd7 [1603772,2465] 0 2024-02-28T16:31:29.299 INFO:tasks.workunit.client.1.smithi066.stdout:7/657: dread fa [4194304,4194304] 0 2024-02-28T16:31:29.303 INFO:tasks.workunit.client.1.smithi066.stdout:2/742: creat d0/ff0 x:0 0 0 2024-02-28T16:31:29.303 INFO:tasks.workunit.client.1.smithi066.stdout:1/689: dread d2/d18/d19/f24 [4194304,4194304] 0 2024-02-28T16:31:29.305 INFO:tasks.workunit.client.1.smithi066.stdout:1/690: readlink d2/da0/la7 0 2024-02-28T16:31:29.309 INFO:tasks.workunit.client.1.smithi066.stdout:1/691: write d2/d18/f3a [4672970,36213] 0 2024-02-28T16:31:29.314 INFO:tasks.workunit.client.1.smithi066.stdout:1/692: write d2/d27/d2b/d34/f44 [312949,11359] 0 2024-02-28T16:31:29.317 INFO:tasks.workunit.client.1.smithi066.stdout:8/693: rename d0/d6/dc/d16/d4c/d6c/d6f/da7/dc9 to d0/db/d12/d1d/d25/de2 0 2024-02-28T16:31:29.317 INFO:tasks.workunit.client.1.smithi066.stdout:4/836: dread d1/d23/d25/d43/f5e [0,4194304] 0 2024-02-28T16:31:29.318 INFO:tasks.workunit.client.1.smithi066.stdout:6/668: fsync d2/d2c/f4a 0 2024-02-28T16:31:29.323 INFO:tasks.workunit.client.1.smithi066.stdout:2/743: creat d0/d12/d1a/d39/d7c/d87/dc4/ff1 x:0 0 0 2024-02-28T16:31:29.324 INFO:tasks.workunit.client.1.smithi066.stdout:1/693: mkdir d2/d18/d6b/dba/de1 0 2024-02-28T16:31:29.325 INFO:tasks.workunit.client.1.smithi066.stdout:8/694: stat d0/db/d12/d1d/d25/d65/d93 0 2024-02-28T16:31:29.327 INFO:tasks.workunit.client.1.smithi066.stdout:7/658: symlink dc/de/d17/d37/d47/d88/dc4/lcd 0 2024-02-28T16:31:29.332 INFO:tasks.workunit.client.1.smithi066.stdout:7/659: write dc/de/d17/d1e/f6c [487888,93085] 0 2024-02-28T16:31:29.332 INFO:tasks.workunit.client.1.smithi066.stdout:6/669: rename d2/d2c/dba/lde to d2/d26/de0/led 0 2024-02-28T16:31:29.334 INFO:tasks.workunit.client.1.smithi066.stdout:8/695: mknod d0/d6/dc/d16/d1a/d63/d58/d5a/ce3 0 2024-02-28T16:31:29.335 INFO:tasks.workunit.client.1.smithi066.stdout:1/694: mkdir d2/d27/dc7/de2 0 2024-02-28T16:31:29.340 INFO:tasks.workunit.client.1.smithi066.stdout:7/660: mkdir dc/d56/dce 0 2024-02-28T16:31:29.341 INFO:tasks.workunit.client.1.smithi066.stdout:6/670: chown d2/da/d19/d9c/f2f 335622 1 2024-02-28T16:31:29.341 INFO:tasks.workunit.client.1.smithi066.stdout:4/837: link d1/d18/d41/f95 d1/d23/d25/dd9/f11a 0 2024-02-28T16:31:29.341 INFO:tasks.workunit.client.1.smithi066.stdout:1/695: dread - d2/d27/d2b/f86 zero size 2024-02-28T16:31:29.341 INFO:tasks.workunit.client.1.smithi066.stdout:2/744: rename d0/d12/d1a/d23/d29/d21/d2f/c80 to d0/d12/d1a/d39/d68/cf2 0 2024-02-28T16:31:29.345 INFO:tasks.workunit.client.1.smithi066.stdout:4/838: truncate d1/d23/f36 1224709 0 2024-02-28T16:31:29.346 INFO:tasks.workunit.client.1.smithi066.stdout:2/745: chown d0/d12/d1a/d39/d7c/d87/dc4/ff1 34933 1 2024-02-28T16:31:29.351 INFO:tasks.workunit.client.1.smithi066.stdout:6/671: rename d2/d26/d6b/f70 to d2/da/d19/d9c/d3d/fee 0 2024-02-28T16:31:29.354 INFO:tasks.workunit.client.1.smithi066.stdout:4/839: creat d1/d23/d25/d3e/d40/f11b x:0 0 0 2024-02-28T16:31:29.355 INFO:tasks.workunit.client.1.smithi066.stdout:4/840: fsync d1/f4 0 2024-02-28T16:31:29.360 INFO:tasks.workunit.client.1.smithi066.stdout:2/746: symlink d0/d12/lf3 0 2024-02-28T16:31:29.360 INFO:tasks.workunit.client.1.smithi066.stdout:8/696: write d0/d6/dc/d16/f50 [1247130,9319] 0 2024-02-28T16:31:29.363 INFO:tasks.workunit.client.1.smithi066.stdout:0/798: sync 2024-02-28T16:31:29.365 INFO:tasks.workunit.client.1.smithi066.stdout:4/841: mkdir d1/d23/d25/d3e/d40/d9c/d11c 0 2024-02-28T16:31:29.366 INFO:tasks.workunit.client.1.smithi066.stdout:6/672: fsync d2/d2c/da1/d5d/d99/f85 0 2024-02-28T16:31:29.366 INFO:tasks.workunit.client.1.smithi066.stdout:1/696: write d2/d27/d2b/d43/d46/f61 [155760,104021] 0 2024-02-28T16:31:29.371 INFO:tasks.workunit.client.1.smithi066.stdout:2/747: rename d0/d12/d1a/d23/cb3 to d0/dd7/cf4 0 2024-02-28T16:31:29.372 INFO:tasks.workunit.client.1.smithi066.stdout:7/661: dread dc/de/d17/d1e/fa3 [0,4194304] 0 2024-02-28T16:31:29.372 INFO:tasks.workunit.client.1.smithi066.stdout:0/799: stat d2/d27/d41/d76/d93/d10/d23/d6d 0 2024-02-28T16:31:29.374 INFO:tasks.workunit.client.1.smithi066.stdout:4/842: dread - d1/d23/d25/d77/fe9 zero size 2024-02-28T16:31:29.374 INFO:tasks.workunit.client.1.smithi066.stdout:1/697: chown d2/d27/dc7/d54 677332 1 2024-02-28T16:31:29.382 INFO:tasks.workunit.client.1.smithi066.stdout:6/673: chown d2/d26/d32/d46/fda 3 1 2024-02-28T16:31:29.385 INFO:tasks.workunit.client.1.smithi066.stdout:1/698: mknod d2/d18/d7a/db1/ce3 0 2024-02-28T16:31:29.388 INFO:tasks.workunit.client.1.smithi066.stdout:8/697: rename d0/db/d15/f7a to d0/d6/dc/d16/d4c/d6c/fe4 0 2024-02-28T16:31:29.389 INFO:tasks.workunit.client.1.smithi066.stdout:7/662: symlink dc/de/d17/lcf 0 2024-02-28T16:31:29.395 INFO:tasks.workunit.client.1.smithi066.stdout:4/843: rename d1/d23/d25/d43 to d1/d23/d25/d3e/d40/d9c/d11c/d11d 0 2024-02-28T16:31:29.400 INFO:tasks.workunit.client.1.smithi066.stdout:1/699: mkdir d2/d27/dc7/de4 0 2024-02-28T16:31:29.403 INFO:tasks.workunit.client.1.smithi066.stdout:1/700: dread - d2/fd1 zero size 2024-02-28T16:31:29.404 INFO:tasks.workunit.client.1.smithi066.stdout:2/748: dwrite d0/d12/d1a/d23/f28 [4194304,4194304] 0 2024-02-28T16:31:29.405 INFO:tasks.workunit.client.1.smithi066.stdout:6/674: link d2/d26/f31 d2/da/d19/d75/d88/fef 0 2024-02-28T16:31:29.406 INFO:tasks.workunit.client.1.smithi066.stdout:8/698: creat d0/db/d12/d1d/fe5 x:0 0 0 2024-02-28T16:31:29.409 INFO:tasks.workunit.client.1.smithi066.stdout:7/663: dread dc/f7f [0,4194304] 0 2024-02-28T16:31:29.410 INFO:tasks.workunit.client.1.smithi066.stdout:8/699: chown d0/db/d15/fd6 2666571 1 2024-02-28T16:31:29.414 INFO:tasks.workunit.client.1.smithi066.stdout:1/701: rename l1 to d2/d18/d6b/dba/de1/le5 0 2024-02-28T16:31:29.415 INFO:tasks.workunit.client.1.smithi066.stdout:6/675: mknod d2/da/d19/cf0 0 2024-02-28T16:31:29.418 INFO:tasks.workunit.client.1.smithi066.stdout:1/702: chown d2/d18/d7a/db1/lc1 2660 1 2024-02-28T16:31:29.420 INFO:tasks.workunit.client.1.smithi066.stdout:2/749: mkdir d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/df5 0 2024-02-28T16:31:29.421 INFO:tasks.workunit.client.1.smithi066.stdout:7/664: rmdir dc/d29/d92/da6 39 2024-02-28T16:31:29.422 INFO:tasks.workunit.client.1.smithi066.stdout:1/703: fdatasync d2/d18/d6b/dba/fdd 0 2024-02-28T16:31:29.424 INFO:tasks.workunit.client.1.smithi066.stdout:2/750: chown d0/d12/d3e/l8f 433 1 2024-02-28T16:31:29.425 INFO:tasks.workunit.client.1.smithi066.stdout:4/844: dwrite d1/d18/d41/dba/f9d [0,4194304] 0 2024-02-28T16:31:29.428 INFO:tasks.workunit.client.1.smithi066.stdout:8/700: symlink d0/db/d12/d1d/d25/d65/dbb/le6 0 2024-02-28T16:31:29.428 INFO:tasks.workunit.client.1.smithi066.stdout:4/845: readlink d1/d23/d25/d3e/laa 0 2024-02-28T16:31:29.432 INFO:tasks.workunit.client.1.smithi066.stdout:7/665: rmdir dc/d6e/dc8 39 2024-02-28T16:31:29.439 INFO:tasks.workunit.client.1.smithi066.stdout:8/701: creat d0/db/d12/d38/fe7 x:0 0 0 2024-02-28T16:31:29.439 INFO:tasks.workunit.client.1.smithi066.stdout:4/846: unlink d1/f96 0 2024-02-28T16:31:29.440 INFO:tasks.workunit.client.1.smithi066.stdout:6/676: link d2/d26/dbe/ld9 d2/d26/d32/lf1 0 2024-02-28T16:31:29.441 INFO:tasks.workunit.client.1.smithi066.stdout:7/666: write dc/f4b [2307508,86668] 0 2024-02-28T16:31:29.443 INFO:tasks.workunit.client.1.smithi066.stdout:8/702: chown d0/db/d12/d1d/d25/d4f/fda 96 1 2024-02-28T16:31:29.447 INFO:tasks.workunit.client.1.smithi066.stdout:8/703: stat d0/d6/dc/d16/cb0 0 2024-02-28T16:31:29.447 INFO:tasks.workunit.client.1.smithi066.stdout:7/667: rmdir dc/de/d17/d1e/d45/d9f 39 2024-02-28T16:31:29.450 INFO:tasks.workunit.client.1.smithi066.stdout:8/704: fdatasync d0/db/d12/d1d/d25/d4f/f9f 0 2024-02-28T16:31:29.450 INFO:tasks.workunit.client.1.smithi066.stdout:6/677: fsync d2/da/d19/d9c/f3f 0 2024-02-28T16:31:29.452 INFO:tasks.workunit.client.1.smithi066.stdout:6/678: chown d2/d26/d32/d46 1740239 1 2024-02-28T16:31:29.456 INFO:tasks.workunit.client.1.smithi066.stdout:1/704: dread d2/d27/d2b/d43/d46/d80/fb8 [0,4194304] 0 2024-02-28T16:31:29.462 INFO:tasks.workunit.client.1.smithi066.stdout:8/705: rmdir d0/db/d12/d38 39 2024-02-28T16:31:29.463 INFO:tasks.workunit.client.1.smithi066.stdout:7/668: creat dc/de/d17/fd0 x:0 0 0 2024-02-28T16:31:29.464 INFO:tasks.workunit.client.1.smithi066.stdout:8/706: fdatasync d0/db/f80 0 2024-02-28T16:31:29.466 INFO:tasks.workunit.client.1.smithi066.stdout:6/679: mknod d2/d2c/da1/d5d/d63/d6a/de6/cf2 0 2024-02-28T16:31:29.471 INFO:tasks.workunit.client.1.smithi066.stdout:1/705: write d2/d27/d2b/d43/f4e [599838,90658] 0 2024-02-28T16:31:29.472 INFO:tasks.workunit.client.1.smithi066.stdout:2/751: dread d0/d12/d1a/d23/d29/d21/f75 [0,4194304] 0 2024-02-28T16:31:29.474 INFO:tasks.workunit.client.1.smithi066.stdout:4/847: dwrite d1/f10 [0,4194304] 0 2024-02-28T16:31:29.474 INFO:tasks.workunit.client.1.smithi066.stdout:1/706: fsync d2/d27/fd5 0 2024-02-28T16:31:29.476 INFO:tasks.workunit.client.1.smithi066.stdout:7/669: mkdir dc/d29/d92/dd1 0 2024-02-28T16:31:29.479 INFO:tasks.workunit.client.1.smithi066.stdout:4/848: dread - d1/d18/d55/f101 zero size 2024-02-28T16:31:29.482 INFO:tasks.workunit.client.1.smithi066.stdout:2/752: write d0/d12/d1a/d39/d7c/d87/fe5 [3841072,44777] 0 2024-02-28T16:31:29.482 INFO:tasks.workunit.client.1.smithi066.stdout:4/849: dread - d1/d7a/ff2 zero size 2024-02-28T16:31:29.483 INFO:tasks.workunit.client.1.smithi066.stdout:7/670: chown dc/de/d17/d1e/da1 4027328 1 2024-02-28T16:31:29.487 INFO:tasks.workunit.client.1.smithi066.stdout:2/753: truncate d0/d12/d2e/f84 4724005 0 2024-02-28T16:31:29.493 INFO:tasks.workunit.client.1.smithi066.stdout:4/850: rename d1/d18/f2b to d1/d23/d25/d3e/d40/d9c/d11c/f11e 0 2024-02-28T16:31:29.495 INFO:tasks.workunit.client.1.smithi066.stdout:7/671: dread - dc/de/d17/d37/d47/f86 zero size 2024-02-28T16:31:29.500 INFO:tasks.workunit.client.1.smithi066.stdout:6/680: dread d2/da/d19/d75/f92 [4194304,4194304] 0 2024-02-28T16:31:29.501 INFO:tasks.workunit.client.1.smithi066.stdout:4/851: chown d1/d18/d41/dba/c93 59219 1 2024-02-28T16:31:29.505 INFO:tasks.workunit.client.1.smithi066.stdout:7/672: mknod dc/de/d17/d37/d47/cd2 0 2024-02-28T16:31:29.506 INFO:tasks.workunit.client.1.smithi066.stdout:8/707: dwrite d0/db/f6e [0,4194304] 0 2024-02-28T16:31:29.507 INFO:tasks.workunit.client.1.smithi066.stdout:4/852: chown d1/d23/d25/lb7 47525098 1 2024-02-28T16:31:29.516 INFO:tasks.workunit.client.1.smithi066.stdout:4/853: symlink d1/d18/d55/l11f 0 2024-02-28T16:31:29.517 INFO:tasks.workunit.client.1.smithi066.stdout:7/673: truncate dc/d56/f77 1990320 0 2024-02-28T16:31:29.518 INFO:tasks.workunit.client.1.smithi066.stdout:1/707: dread d2/d27/dc7/d54/f92 [0,4194304] 0 2024-02-28T16:31:29.518 INFO:tasks.workunit.client.1.smithi066.stdout:2/754: dwrite d0/d12/d2e/fc1 [0,4194304] 0 2024-02-28T16:31:29.524 INFO:tasks.workunit.client.1.smithi066.stdout:1/708: write d2/d27/d2b/d43/d46/fb6 [486822,71989] 0 2024-02-28T16:31:29.524 INFO:tasks.workunit.client.1.smithi066.stdout:8/708: truncate d0/db/d15/f31 4006693 0 2024-02-28T16:31:29.531 INFO:tasks.workunit.client.1.smithi066.stdout:8/709: readlink d0/l49 0 2024-02-28T16:31:29.535 INFO:tasks.workunit.client.1.smithi066.stdout:4/854: getdents d1/d23/d25/d3e/d40/d9c 0 2024-02-28T16:31:29.537 INFO:tasks.workunit.client.1.smithi066.stdout:8/710: chown d0/db/d12/d1d/fe5 25671 1 2024-02-28T16:31:29.543 INFO:tasks.workunit.client.1.smithi066.stdout:1/709: link d2/c97 d2/d27/dc7/d54/d84/ce6 0 2024-02-28T16:31:29.543 INFO:tasks.workunit.client.1.smithi066.stdout:6/681: dwrite f1 [0,4194304] 0 2024-02-28T16:31:29.546 INFO:tasks.workunit.client.1.smithi066.stdout:7/674: dwrite dc/de/f3f [0,4194304] 0 2024-02-28T16:31:29.549 INFO:tasks.workunit.client.1.smithi066.stdout:8/711: mkdir d0/db/d12/d1d/d25/d4f/de0/de8 0 2024-02-28T16:31:29.554 INFO:tasks.workunit.client.1.smithi066.stdout:7/675: mknod dc/d56/cd3 0 2024-02-28T16:31:29.554 INFO:tasks.workunit.client.1.smithi066.stdout:8/712: truncate d0/db/d12/d1d/d25/d4f/fda 876918 0 2024-02-28T16:31:29.556 INFO:tasks.workunit.client.1.smithi066.stdout:1/710: mkdir d2/d27/dc7/d54/de7 0 2024-02-28T16:31:29.557 INFO:tasks.workunit.client.1.smithi066.stdout:7/676: read dc/d29/f41 [561605,15644] 0 2024-02-28T16:31:29.559 INFO:tasks.workunit.client.1.smithi066.stdout:1/711: chown d2/d27/dc7/d48/l70 24 1 2024-02-28T16:31:29.559 INFO:tasks.workunit.client.1.smithi066.stdout:8/713: rmdir d0/db/d12 39 2024-02-28T16:31:29.561 INFO:tasks.workunit.client.1.smithi066.stdout:6/682: write d2/d2c/f4a [3500380,63274] 0 2024-02-28T16:31:29.564 INFO:tasks.workunit.client.1.smithi066.stdout:7/677: creat dc/de/d17/d37/d47/d88/fd4 x:0 0 0 2024-02-28T16:31:29.564 INFO:tasks.workunit.client.1.smithi066.stdout:1/712: mknod d2/d18/d6b/ce8 0 2024-02-28T16:31:29.566 INFO:tasks.workunit.client.1.smithi066.stdout:4/855: truncate d1/d23/d25/d3e/d40/d9c/d11c/d11d/f65 2654101 0 2024-02-28T16:31:29.570 INFO:tasks.workunit.client.1.smithi066.stdout:4/856: fsync d1/d7a/f98 0 2024-02-28T16:31:29.570 INFO:tasks.workunit.client.1.smithi066.stdout:1/713: rmdir d2/d27/dc7/d51/db4 39 2024-02-28T16:31:29.577 INFO:tasks.workunit.client.1.smithi066.stdout:8/714: getdents d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/dad 0 2024-02-28T16:31:29.580 INFO:tasks.workunit.client.1.smithi066.stdout:1/714: rename d2/d27/d2b/d43/d46/d80 to d2/d27/d2b/d34/da9/de9 0 2024-02-28T16:31:29.582 INFO:tasks.workunit.client.1.smithi066.stdout:8/715: chown d0/d6/dc/d16/d4c/d42/d73/c85 3539 1 2024-02-28T16:31:29.604 INFO:tasks.workunit.client.1.smithi066.stdout:9/550: sync 2024-02-28T16:31:29.607 INFO:tasks.workunit.client.1.smithi066.stdout:6/683: write d2/da/f1c [4476170,109877] 0 2024-02-28T16:31:29.610 INFO:tasks.workunit.client.1.smithi066.stdout:8/716: creat d0/d6/dc/d16/d4c/d42/d73/fe9 x:0 0 0 2024-02-28T16:31:29.610 INFO:tasks.workunit.client.1.smithi066.stdout:1/715: rename d2/d18/d6b/f77 to d2/d18/d7a/db1/fea 0 2024-02-28T16:31:29.615 INFO:tasks.workunit.client.1.smithi066.stdout:9/551: rename da/d3a/lb3 to da/d22/d46/d56/d8f/d28/d42/lb7 0 2024-02-28T16:31:29.618 INFO:tasks.workunit.client.1.smithi066.stdout:6/684: truncate d2/da/f22 5719408 0 2024-02-28T16:31:29.619 INFO:tasks.workunit.client.1.smithi066.stdout:8/717: fdatasync d0/d6/dc/d16/d4c/d6c/fe4 0 2024-02-28T16:31:29.619 INFO:tasks.workunit.client.1.smithi066.stdout:1/716: symlink d2/d27/dc7/d48/dd6/leb 0 2024-02-28T16:31:29.623 INFO:tasks.workunit.client.1.smithi066.stdout:8/718: mkdir d0/d6/dea 0 2024-02-28T16:31:29.625 INFO:tasks.workunit.client.1.smithi066.stdout:4/857: dwrite d1/d18/d41/f95 [0,4194304] 0 2024-02-28T16:31:29.630 INFO:tasks.workunit.client.1.smithi066.stdout:4/858: creat d1/d18/da8/f120 x:0 0 0 2024-02-28T16:31:29.630 INFO:tasks.workunit.client.1.smithi066.stdout:8/719: truncate d0/d6/dc/d16/d1a/d63/d3d/fab 909135 0 2024-02-28T16:31:29.631 INFO:tasks.workunit.client.1.smithi066.stdout:6/685: getdents d2/da/d19/d75/dbd/dc6 0 2024-02-28T16:31:29.647 INFO:tasks.workunit.client.1.smithi066.stdout:8/720: symlink d0/d6/dc/d16/d1a/d63/d3d/leb 0 2024-02-28T16:31:29.649 INFO:tasks.workunit.client.1.smithi066.stdout:4/859: mknod d1/d23/d25/d3e/d40/c121 0 2024-02-28T16:31:29.657 INFO:tasks.workunit.client.1.smithi066.stdout:1/717: dwrite d2/d27/d2b/d34/da9/de9/fb8 [0,4194304] 0 2024-02-28T16:31:29.672 INFO:tasks.workunit.client.1.smithi066.stdout:6/686: getdents d2/da/d19/d9c/dc0 0 2024-02-28T16:31:29.672 INFO:tasks.workunit.client.1.smithi066.stdout:9/552: dread da/d22/d46/d56/d8f/d2d/f62 [0,4194304] 0 2024-02-28T16:31:29.673 INFO:tasks.workunit.client.1.smithi066.stdout:4/860: mknod d1/d18/d41/dba/d100/c122 0 2024-02-28T16:31:29.673 INFO:tasks.workunit.client.1.smithi066.stdout:1/718: dread d2/d27/d2b/d43/d46/f63 [0,4194304] 0 2024-02-28T16:31:29.674 INFO:tasks.workunit.client.1.smithi066.stdout:8/721: mkdir d0/d6/dea/dec 0 2024-02-28T16:31:29.679 INFO:tasks.workunit.client.1.smithi066.stdout:1/719: mknod d2/d18/d7a/cec 0 2024-02-28T16:31:29.680 INFO:tasks.workunit.client.1.smithi066.stdout:4/861: mkdir d1/d18/d41/dba/d116/d123 0 2024-02-28T16:31:29.682 INFO:tasks.workunit.client.1.smithi066.stdout:8/722: symlink d0/d6/dc/d16/d4c/led 0 2024-02-28T16:31:29.684 INFO:tasks.workunit.client.1.smithi066.stdout:9/553: rename da/d22/d46/d56/d8f/d28/d42/l6e to da/d22/d46/d56/d8f/d28/d63/lb8 0 2024-02-28T16:31:29.686 INFO:tasks.workunit.client.1.smithi066.stdout:4/862: truncate d1/d18/d41/f47 358656 0 2024-02-28T16:31:29.688 INFO:tasks.workunit.client.1.smithi066.stdout:1/720: symlink d2/d27/dc7/d51/led 0 2024-02-28T16:31:29.691 INFO:tasks.workunit.client.1.smithi066.stdout:8/723: truncate d0/db/d12/d1d/d25/d4f/fde 526393 0 2024-02-28T16:31:29.693 INFO:tasks.workunit.client.1.smithi066.stdout:4/863: mknod d1/d23/d25/d3e/d40/d9c/c124 0 2024-02-28T16:31:29.694 INFO:tasks.workunit.client.1.smithi066.stdout:1/721: creat d2/d18/d6b/dba/fee x:0 0 0 2024-02-28T16:31:29.695 INFO:tasks.workunit.client.1.smithi066.stdout:8/724: mknod d0/d6/dc/d16/d4c/d6c/d6f/cee 0 2024-02-28T16:31:29.700 INFO:tasks.workunit.client.1.smithi066.stdout:1/722: rename d2/l5e to d2/d27/d2b/d34/da9/de9/lef 0 2024-02-28T16:31:29.702 INFO:tasks.workunit.client.1.smithi066.stdout:8/725: truncate d0/db/d12/d1d/d25/f74 696989 0 2024-02-28T16:31:29.707 INFO:tasks.workunit.client.1.smithi066.stdout:8/726: rename d0/db/d15/c2d to d0/db/d12/d1d/d25/d4f/de0/cef 0 2024-02-28T16:31:29.713 INFO:tasks.workunit.client.1.smithi066.stdout:8/727: link d0/db/d12/c40 d0/d6/dc/d16/cf0 0 2024-02-28T16:31:29.715 INFO:tasks.workunit.client.1.smithi066.stdout:6/687: write d2/d26/f57 [916206,62536] 0 2024-02-28T16:31:29.717 INFO:tasks.workunit.client.1.smithi066.stdout:8/728: creat d0/db/ff1 x:0 0 0 2024-02-28T16:31:29.719 INFO:tasks.workunit.client.1.smithi066.stdout:9/554: rmdir da/d22/d46/d56 39 2024-02-28T16:31:29.723 INFO:tasks.workunit.client.1.smithi066.stdout:8/729: symlink d0/d6/dc/d16/d4c/d42/da8/lf2 0 2024-02-28T16:31:29.724 INFO:tasks.workunit.client.1.smithi066.stdout:6/688: getdents d2/d2c/da1/dad 0 2024-02-28T16:31:29.731 INFO:tasks.workunit.client.1.smithi066.stdout:9/555: creat da/d22/d46/d56/d8f/fb9 x:0 0 0 2024-02-28T16:31:29.734 INFO:tasks.workunit.client.1.smithi066.stdout:1/723: creat d2/d18/d7a/ff0 x:0 0 0 2024-02-28T16:31:29.735 INFO:tasks.workunit.client.1.smithi066.stdout:9/556: readlink da/d22/d46/d56/d8f/d2d/d81/l92 0 2024-02-28T16:31:29.735 INFO:tasks.workunit.client.1.smithi066.stdout:4/864: dwrite d1/d23/d25/d3e/d40/d9c/d11c/d11d/fa6 [0,4194304] 0 2024-02-28T16:31:29.739 INFO:tasks.workunit.client.1.smithi066.stdout:4/865: read d1/d18/f38 [39028,78904] 0 2024-02-28T16:31:29.740 INFO:tasks.workunit.client.1.smithi066.stdout:8/730: mkdir d0/d6/dc/d16/d4c/d6c/daf/df3 0 2024-02-28T16:31:29.747 INFO:tasks.workunit.client.1.smithi066.stdout:1/724: mknod d2/d27/dc7/d54/d84/cf1 0 2024-02-28T16:31:29.748 INFO:tasks.workunit.client.1.smithi066.stdout:6/689: write d2/d26/f31 [637267,120304] 0 2024-02-28T16:31:29.752 INFO:tasks.workunit.client.1.smithi066.stdout:4/866: creat d1/dfb/f125 x:0 0 0 2024-02-28T16:31:29.753 INFO:tasks.workunit.client.1.smithi066.stdout:9/557: write da/d22/d46/d56/d8f/d2d/f8d [262006,104959] 0 2024-02-28T16:31:29.760 INFO:tasks.workunit.client.1.smithi066.stdout:8/731: dread d0/db/d15/f37 [0,4194304] 0 2024-02-28T16:31:29.761 INFO:tasks.workunit.client.1.smithi066.stdout:4/867: mkdir d1/d23/d25/d3e/d40/d9c/d11c/d126 0 2024-02-28T16:31:29.761 INFO:tasks.workunit.client.1.smithi066.stdout:1/725: creat d2/d18/d19/ff2 x:0 0 0 2024-02-28T16:31:29.764 INFO:tasks.workunit.client.1.smithi066.stdout:6/690: write d2/da/f18 [6412295,96682] 0 2024-02-28T16:31:29.770 INFO:tasks.workunit.client.1.smithi066.stdout:6/691: mkdir d2/d2c/d6e/df3 0 2024-02-28T16:31:29.772 INFO:tasks.workunit.client.1.smithi066.stdout:5/667: sync 2024-02-28T16:31:29.772 INFO:tasks.workunit.client.1.smithi066.stdout:3/825: sync 2024-02-28T16:31:29.772 INFO:tasks.workunit.client.1.smithi066.stdout:0/800: sync 2024-02-28T16:31:29.774 INFO:tasks.workunit.client.1.smithi066.stdout:8/732: mknod d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/dad/cf4 0 2024-02-28T16:31:29.776 INFO:tasks.workunit.client.1.smithi066.stdout:4/868: write d1/d7a/fcf [338428,48056] 0 2024-02-28T16:31:29.776 INFO:tasks.workunit.client.1.smithi066.stdout:6/692: symlink d2/da/d19/d75/dbd/lf4 0 2024-02-28T16:31:29.777 INFO:tasks.workunit.client.1.smithi066.stdout:1/726: dread d2/d27/dc7/d54/d84/f8c [0,4194304] 0 2024-02-28T16:31:29.779 INFO:tasks.workunit.client.1.smithi066.stdout:0/801: fdatasync d2/f5e 0 2024-02-28T16:31:29.779 INFO:tasks.workunit.client.1.smithi066.stdout:6/693: readlink d2/da/d19/l4b 0 2024-02-28T16:31:29.781 INFO:tasks.workunit.client.1.smithi066.stdout:9/558: dwrite da/f2a [4194304,4194304] 0 2024-02-28T16:31:29.782 INFO:tasks.workunit.client.1.smithi066.stdout:4/869: chown d1/d7a/ce1 1935 1 2024-02-28T16:31:29.782 INFO:tasks.workunit.client.1.smithi066.stdout:3/826: creat db/d13/d14/dda/d102/f127 x:0 0 0 2024-02-28T16:31:29.783 INFO:tasks.workunit.client.1.smithi066.stdout:5/668: creat d3/d1b/d7d/fd1 x:0 0 0 2024-02-28T16:31:29.786 INFO:tasks.workunit.client.1.smithi066.stdout:5/669: chown d3/d4/dc/d2f/d55 17 1 2024-02-28T16:31:29.786 INFO:tasks.workunit.client.1.smithi066.stdout:3/827: write db/d2a/d44/dd8/feb [803032,40098] 0 2024-02-28T16:31:29.792 INFO:tasks.workunit.client.1.smithi066.stdout:1/727: chown d2/d18/d7a/db1/fea 27159369 1 2024-02-28T16:31:29.794 INFO:tasks.workunit.client.1.smithi066.stdout:1/728: write d2/d27/d2b/f3d [4595320,128327] 0 2024-02-28T16:31:29.794 INFO:tasks.workunit.client.1.smithi066.stdout:6/694: rename d2/d2c/f95 to d2/d2c/da1/dad/ff5 0 2024-02-28T16:31:29.796 INFO:tasks.workunit.client.1.smithi066.stdout:4/870: mknod d1/d18/d55/dc4/c127 0 2024-02-28T16:31:29.800 INFO:tasks.workunit.client.1.smithi066.stdout:5/670: symlink d3/d4/dc/dd/d2e/ld2 0 2024-02-28T16:31:29.800 INFO:tasks.workunit.client.1.smithi066.stdout:4/871: stat d1/d18/d4e/f71 0 2024-02-28T16:31:29.801 INFO:tasks.workunit.client.1.smithi066.stdout:5/671: readlink d3/d1b/d7d/l8a 0 2024-02-28T16:31:29.801 INFO:tasks.workunit.client.1.smithi066.stdout:3/828: unlink db/dc/d35/d3d/d76/d89/cb7 0 2024-02-28T16:31:29.804 INFO:tasks.workunit.client.1.smithi066.stdout:5/672: dread - d3/d4/dc/dd/d41/da2/f8d zero size 2024-02-28T16:31:29.807 INFO:tasks.workunit.client.1.smithi066.stdout:8/733: fdatasync d0/db/d15/f31 0 2024-02-28T16:31:29.812 INFO:tasks.workunit.client.1.smithi066.stdout:9/559: write da/d22/d46/d56/d8f/d28/d63/f60 [902364,26950] 0 2024-02-28T16:31:29.814 INFO:tasks.workunit.client.1.smithi066.stdout:3/829: rmdir db/d13/d14/dda/d102 39 2024-02-28T16:31:29.818 INFO:tasks.workunit.client.1.smithi066.stdout:1/729: mknod d2/d18/da5/db2/cf3 0 2024-02-28T16:31:29.823 INFO:tasks.workunit.client.1.smithi066.stdout:4/872: dwrite d1/d18/d41/f68 [0,4194304] 0 2024-02-28T16:31:29.823 INFO:tasks.workunit.client.1.smithi066.stdout:0/802: link d2/d27/d41/d76/d93/ca d2/d27/d41/d76/d93/d33/d3f/d8e/c108 0 2024-02-28T16:31:29.830 INFO:tasks.workunit.client.1.smithi066.stdout:8/734: mkdir d0/db/d12/d1d/d25/de2/df5 0 2024-02-28T16:31:29.830 INFO:tasks.workunit.client.1.smithi066.stdout:5/673: link d3/d4/dc/f1d d3/d4/dc/d56/fd3 0 2024-02-28T16:31:29.830 INFO:tasks.workunit.client.1.smithi066.stdout:9/560: rename da/d22/d46/d56/d8f/d28/d63 to da/d4b/d53/dba 0 2024-02-28T16:31:29.831 INFO:tasks.workunit.client.1.smithi066.stdout:1/730: symlink d2/d27/d2b/d34/da9/de9/lf4 0 2024-02-28T16:31:29.837 INFO:tasks.workunit.client.1.smithi066.stdout:6/695: truncate d2/d2c/da1/d5d/d99/f85 458505 0 2024-02-28T16:31:29.838 INFO:tasks.workunit.client.1.smithi066.stdout:9/561: getdents da/df/d70/d90/d98/daa 0 2024-02-28T16:31:29.848 INFO:tasks.workunit.client.1.smithi066.stdout:3/830: dwrite db/d13/d14/d84/f112 [0,4194304] 0 2024-02-28T16:31:29.848 INFO:tasks.workunit.client.1.smithi066.stdout:9/562: creat da/d4b/d53/dba/fbb x:0 0 0 2024-02-28T16:31:29.854 INFO:tasks.workunit.client.1.smithi066.stdout:6/696: symlink d2/d2c/da1/d5d/d63/d6a/lf6 0 2024-02-28T16:31:29.856 INFO:tasks.workunit.client.1.smithi066.stdout:2/755: sync 2024-02-28T16:31:29.856 INFO:tasks.workunit.client.1.smithi066.stdout:7/678: sync 2024-02-28T16:31:29.857 INFO:tasks.workunit.client.1.smithi066.stdout:9/563: mkdir da/d6b/dbc 0 2024-02-28T16:31:29.860 INFO:tasks.workunit.client.1.smithi066.stdout:1/731: link d2/d27/d2b/d34/c67 d2/d27/d2b/cf5 0 2024-02-28T16:31:29.867 INFO:tasks.workunit.client.1.smithi066.stdout:8/735: dwrite d0/d6/dc/d16/fc3 [0,4194304] 0 2024-02-28T16:31:29.872 INFO:tasks.workunit.client.1.smithi066.stdout:7/679: write dc/de/d17/fd0 [213762,92699] 0 2024-02-28T16:31:29.875 INFO:tasks.workunit.client.1.smithi066.stdout:3/831: rename db/d13/d1a/d25/f5f to db/dc/db3/dc9/de4/d116/f128 0 2024-02-28T16:31:29.877 INFO:tasks.workunit.client.1.smithi066.stdout:9/564: fdatasync da/d22/d46/d56/d8f/d28/d42/f80 0 2024-02-28T16:31:29.877 INFO:tasks.workunit.client.1.smithi066.stdout:4/873: dwrite d1/df7/f103 [0,4194304] 0 2024-02-28T16:31:29.882 INFO:tasks.workunit.client.1.smithi066.stdout:1/732: unlink d2/d18/d19/f1c 0 2024-02-28T16:31:29.884 INFO:tasks.workunit.client.1.smithi066.stdout:4/874: dread - d1/d18/d55/dc4/fd3 zero size 2024-02-28T16:31:29.888 INFO:tasks.workunit.client.1.smithi066.stdout:0/803: dread d2/d27/d41/d76/d93/d10/d23/d6d/faf [0,4194304] 0 2024-02-28T16:31:29.890 INFO:tasks.workunit.client.1.smithi066.stdout:4/875: write d1/d23/d25/d3e/d40/d9c/d11c/d11d/ded/f114 [945676,51467] 0 2024-02-28T16:31:29.892 INFO:tasks.workunit.client.1.smithi066.stdout:6/697: write d2/d2c/d82/f8a [946578,38997] 0 2024-02-28T16:31:29.893 INFO:tasks.workunit.client.1.smithi066.stdout:7/680: mkdir dc/d56/d97/dd5 0 2024-02-28T16:31:29.894 INFO:tasks.workunit.client.1.smithi066.stdout:6/698: fsync d2/fb5 0 2024-02-28T16:31:29.905 INFO:tasks.workunit.client.1.smithi066.stdout:1/733: symlink d2/d27/d2b/d34/d9e/lf6 0 2024-02-28T16:31:29.906 INFO:tasks.workunit.client.1.smithi066.stdout:8/736: dwrite d0/d6/dc/d16/d1a/f8e [0,4194304] 0 2024-02-28T16:31:29.909 INFO:tasks.workunit.client.1.smithi066.stdout:2/756: write d0/d12/d1a/d39/f42 [386511,7583] 0 2024-02-28T16:31:29.911 INFO:tasks.workunit.client.1.smithi066.stdout:0/804: mknod d2/d27/d41/d76/d93/db2/db5/d67/d94/de4/c109 0 2024-02-28T16:31:29.911 INFO:tasks.workunit.client.1.smithi066.stdout:7/681: symlink dc/d56/ld6 0 2024-02-28T16:31:29.914 INFO:tasks.workunit.client.1.smithi066.stdout:3/832: dread db/d2a/d44/f4f [0,4194304] 0 2024-02-28T16:31:29.916 INFO:tasks.workunit.client.1.smithi066.stdout:0/805: fdatasync d2/d27/d41/d76/d93/db2/db5/d51/f8a 0 2024-02-28T16:31:29.916 INFO:tasks.workunit.client.1.smithi066.stdout:6/699: truncate d2/da/d19/d75/d88/fce 600773 0 2024-02-28T16:31:29.922 INFO:tasks.workunit.client.1.smithi066.stdout:1/734: chown d2/f13 11 1 2024-02-28T16:31:29.923 INFO:tasks.workunit.client.1.smithi066.stdout:0/806: stat d2/d27/d41/d76/d93/db2/db5/d52/da2/lba 0 2024-02-28T16:31:29.926 INFO:tasks.workunit.client.1.smithi066.stdout:3/833: creat db/d2a/d44/d48/f129 x:0 0 0 2024-02-28T16:31:29.927 INFO:tasks.workunit.client.1.smithi066.stdout:1/735: chown d2/d27/dc7/d51/l90 890065 1 2024-02-28T16:31:29.933 INFO:tasks.workunit.client.1.smithi066.stdout:6/700: mkdir d2/d26/dbe/df7 0 2024-02-28T16:31:29.933 INFO:tasks.workunit.client.1.smithi066.stdout:2/757: fdatasync d0/f44 0 2024-02-28T16:31:29.938 INFO:tasks.workunit.client.1.smithi066.stdout:7/682: dwrite dc/d56/f8a [0,4194304] 0 2024-02-28T16:31:29.945 INFO:tasks.workunit.client.1.smithi066.stdout:0/807: mknod d2/c10a 0 2024-02-28T16:31:29.948 INFO:tasks.workunit.client.1.smithi066.stdout:6/701: creat d2/d2c/d82/ff8 x:0 0 0 2024-02-28T16:31:29.956 INFO:tasks.workunit.client.1.smithi066.stdout:0/808: rmdir d2/d27/d41/d76/d93/db2/db5/d52/da2 39 2024-02-28T16:31:29.956 INFO:tasks.workunit.client.1.smithi066.stdout:1/736: dread d2/d27/dc7/d51/f62 [0,4194304] 0 2024-02-28T16:31:29.956 INFO:tasks.workunit.client.1.smithi066.stdout:9/565: dread f3 [0,4194304] 0 2024-02-28T16:31:29.958 INFO:tasks.workunit.client.1.smithi066.stdout:8/737: dwrite d0/f13 [0,4194304] 0 2024-02-28T16:31:29.962 INFO:tasks.workunit.client.1.smithi066.stdout:2/758: creat d0/d12/d1a/d23/d29/ff6 x:0 0 0 2024-02-28T16:31:29.965 INFO:tasks.workunit.client.1.smithi066.stdout:6/702: creat d2/da/d19/d75/da6/ff9 x:0 0 0 2024-02-28T16:31:29.970 INFO:tasks.workunit.client.1.smithi066.stdout:9/566: symlink da/d22/d46/lbd 0 2024-02-28T16:31:29.973 INFO:tasks.workunit.client.1.smithi066.stdout:9/567: fsync da/f35 0 2024-02-28T16:31:29.973 INFO:tasks.workunit.client.1.smithi066.stdout:1/737: creat d2/d27/dc7/d54/de7/ff7 x:0 0 0 2024-02-28T16:31:29.973 INFO:tasks.workunit.client.1.smithi066.stdout:3/834: dwrite db/d13/f19 [0,4194304] 0 2024-02-28T16:31:29.977 INFO:tasks.workunit.client.1.smithi066.stdout:2/759: truncate d0/d12/d1a/d39/d7c/fdf 412468 0 2024-02-28T16:31:29.978 INFO:tasks.workunit.client.1.smithi066.stdout:8/738: creat d0/d6/dc/d16/d4c/d6c/d6f/ff6 x:0 0 0 2024-02-28T16:31:29.979 INFO:tasks.workunit.client.1.smithi066.stdout:0/809: symlink d2/d27/d41/d76/d93/d10/d107/l10b 0 2024-02-28T16:31:29.980 INFO:tasks.workunit.client.1.smithi066.stdout:3/835: write db/dc/d35/d3d/d76/d89/dcc/f106 [398571,83706] 0 2024-02-28T16:31:29.986 INFO:tasks.workunit.client.1.smithi066.stdout:6/703: rename d2/fb5 to d2/da/d19/d75/d88/dbc/ffa 0 2024-02-28T16:31:29.992 INFO:tasks.workunit.client.1.smithi066.stdout:7/683: link dc/c21 dc/d6e/dc8/cd7 0 2024-02-28T16:31:29.993 INFO:tasks.workunit.client.1.smithi066.stdout:8/739: creat d0/d6/dc/ff7 x:0 0 0 2024-02-28T16:31:29.997 INFO:tasks.workunit.client.1.smithi066.stdout:7/684: readlink dc/de/d17/d1e/d45/l57 0 2024-02-28T16:31:29.999 INFO:tasks.workunit.client.1.smithi066.stdout:1/738: truncate d2/d18/f6e 1081317 0 2024-02-28T16:31:30.001 INFO:tasks.workunit.client.1.smithi066.stdout:9/568: mkdir da/d22/d46/d56/d8f/dbe 0 2024-02-28T16:31:30.003 INFO:tasks.workunit.client.1.smithi066.stdout:2/760: creat d0/d12/d1a/ff7 x:0 0 0 2024-02-28T16:31:30.005 INFO:tasks.workunit.client.1.smithi066.stdout:3/836: dwrite db/dc/d11/d1f/fd0 [0,4194304] 0 2024-02-28T16:31:30.013 INFO:tasks.workunit.client.1.smithi066.stdout:6/704: creat d2/d2c/da1/d5d/ffb x:0 0 0 2024-02-28T16:31:30.019 INFO:tasks.workunit.client.1.smithi066.stdout:8/740: fdatasync d0/d6/dc/d16/d1a/f88 0 2024-02-28T16:31:30.024 INFO:tasks.workunit.client.1.smithi066.stdout:2/761: mkdir d0/d12/d1a/df8 0 2024-02-28T16:31:30.028 INFO:tasks.workunit.client.1.smithi066.stdout:0/810: dread d2/d27/d41/d76/d93/d33/d3f/faa [0,4194304] 0 2024-02-28T16:31:30.028 INFO:tasks.workunit.client.1.smithi066.stdout:6/705: creat d2/d26/de0/ffc x:0 0 0 2024-02-28T16:31:30.032 INFO:tasks.workunit.client.1.smithi066.stdout:8/741: write d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db3/fce [1023939,112695] 0 2024-02-28T16:31:30.034 INFO:tasks.workunit.client.1.smithi066.stdout:7/685: dwrite f5 [0,4194304] 0 2024-02-28T16:31:30.035 INFO:tasks.workunit.client.1.smithi066.stdout:1/739: write d2/d27/d2b/d43/d46/f61 [904835,11219] 0 2024-02-28T16:31:30.037 INFO:tasks.workunit.client.1.smithi066.stdout:9/569: write da/d22/d46/d56/d8f/d2d/f62 [118600,30840] 0 2024-02-28T16:31:30.039 INFO:tasks.workunit.client.1.smithi066.stdout:3/837: truncate db/dc/db3/fcd 4036586 0 2024-02-28T16:31:30.041 INFO:tasks.workunit.client.1.smithi066.stdout:1/740: readlink d2/d27/dc7/d54/l5c 0 2024-02-28T16:31:30.052 INFO:tasks.workunit.client.1.smithi066.stdout:9/570: symlink da/d4b/lbf 0 2024-02-28T16:31:30.053 INFO:tasks.workunit.client.1.smithi066.stdout:3/838: mknod db/dc/d11/c12a 0 2024-02-28T16:31:30.054 INFO:tasks.workunit.client.1.smithi066.stdout:2/762: dwrite d0/d12/d1a/d39/d7c/d87/dc4/fcd [4194304,4194304] 0 2024-02-28T16:31:30.055 INFO:tasks.workunit.client.1.smithi066.stdout:1/741: mknod d2/d27/dc7/d54/dc4/cf8 0 2024-02-28T16:31:30.060 INFO:tasks.workunit.client.1.smithi066.stdout:7/686: rename dc/de/d17 to dc/d29/d92/da6/dd8 0 2024-02-28T16:31:30.065 INFO:tasks.workunit.client.1.smithi066.stdout:8/742: creat d0/db/d12/d1d/d25/d65/ff8 x:0 0 0 2024-02-28T16:31:30.070 INFO:tasks.workunit.client.1.smithi066.stdout:0/811: getdents d2/d27/d41/d76/d93/db2/db5/d73 0 2024-02-28T16:31:30.074 INFO:tasks.workunit.client.1.smithi066.stdout:9/571: fsync da/d22/d46/d56/f72 0 2024-02-28T16:31:30.077 INFO:tasks.workunit.client.1.smithi066.stdout:9/572: chown da/d22/d46/d56 0 1 2024-02-28T16:31:30.082 INFO:tasks.workunit.client.1.smithi066.stdout:3/839: creat db/d13/f12b x:0 0 0 2024-02-28T16:31:30.082 INFO:tasks.workunit.client.1.smithi066.stdout:5/674: sync 2024-02-28T16:31:30.084 INFO:tasks.workunit.client.1.smithi066.stdout:8/743: mknod d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/dad/db9/cf9 0 2024-02-28T16:31:30.090 INFO:tasks.workunit.client.1.smithi066.stdout:6/706: dread d2/d26/f47 [0,4194304] 0 2024-02-28T16:31:30.090 INFO:tasks.workunit.client.1.smithi066.stdout:1/742: rename d2/d27/dc7/d54/dc4 to d2/d27/df9 0 2024-02-28T16:31:30.091 INFO:tasks.workunit.client.1.smithi066.stdout:8/744: chown d0/d6/dc/d16/d4c/d6c/l89 847 1 2024-02-28T16:31:30.092 INFO:tasks.workunit.client.1.smithi066.stdout:2/763: write d0/d12/d1a/d23/d5d/fbb [299438,108162] 0 2024-02-28T16:31:30.093 INFO:tasks.workunit.client.1.smithi066.stdout:0/812: creat d2/d27/d41/d76/dcf/f10c x:0 0 0 2024-02-28T16:31:30.093 INFO:tasks.workunit.client.1.smithi066.stdout:5/675: truncate d3/d4/f71 1409032 0 2024-02-28T16:31:30.097 INFO:tasks.workunit.client.1.smithi066.stdout:6/707: fdatasync d2/da/f1b 0 2024-02-28T16:31:30.101 INFO:tasks.workunit.client.1.smithi066.stdout:8/745: creat d0/d6/dc/d16/d1a/ffa x:0 0 0 2024-02-28T16:31:30.104 INFO:tasks.workunit.client.1.smithi066.stdout:7/687: dread dc/f4b [0,4194304] 0 2024-02-28T16:31:30.104 INFO:tasks.workunit.client.1.smithi066.stdout:2/764: read d0/d12/d1a/d39/d7c/d87/dc4/fc8 [230100,8846] 0 2024-02-28T16:31:30.106 INFO:tasks.workunit.client.1.smithi066.stdout:0/813: fsync d2/d27/d41/d76/d93/d10/d23/d6d/f103 0 2024-02-28T16:31:30.107 INFO:tasks.workunit.client.1.smithi066.stdout:8/746: fdatasync d0/db/d12/d1d/fd0 0 2024-02-28T16:31:30.108 INFO:tasks.workunit.client.1.smithi066.stdout:1/743: symlink d2/dd0/ddf/lfa 0 2024-02-28T16:31:30.110 INFO:tasks.workunit.client.1.smithi066.stdout:3/840: dwrite db/d2a/ff7 [0,4194304] 0 2024-02-28T16:31:30.111 INFO:tasks.workunit.client.1.smithi066.stdout:5/676: mknod d3/d4/dc/dd/d35/dc5/cd4 0 2024-02-28T16:31:30.113 INFO:tasks.workunit.client.1.smithi066.stdout:6/708: rename d2/d2c/d82/f81 to d2/d2c/da1/d5d/d63/d6a/de6/ffd 0 2024-02-28T16:31:30.114 INFO:tasks.workunit.client.1.smithi066.stdout:9/573: truncate da/d3a/f13 2382820 0 2024-02-28T16:31:30.116 INFO:tasks.workunit.client.1.smithi066.stdout:5/677: readlink d3/d4/dc/d2f/d55/d74/lc2 0 2024-02-28T16:31:30.117 INFO:tasks.workunit.client.1.smithi066.stdout:2/765: symlink d0/d12/d1a/d39/d73/lf9 0 2024-02-28T16:31:30.120 INFO:tasks.workunit.client.1.smithi066.stdout:0/814: read d2/d27/d41/d76/d93/db2/db5/d67/d94/de4/da1/fd2 [1782601,31847] 0 2024-02-28T16:31:30.123 INFO:tasks.workunit.client.1.smithi066.stdout:7/688: creat dc/d29/d92/da6/dd8/d91/dba/fd9 x:0 0 0 2024-02-28T16:31:30.128 INFO:tasks.workunit.client.1.smithi066.stdout:3/841: mkdir db/d2a/d44/d48/d52/d77/d12c 0 2024-02-28T16:31:30.129 INFO:tasks.workunit.client.1.smithi066.stdout:5/678: creat d3/d4/dc/dd/d2e/d18/fd5 x:0 0 0 2024-02-28T16:31:30.130 INFO:tasks.workunit.client.1.smithi066.stdout:3/842: dread - db/d2a/d44/f125 zero size 2024-02-28T16:31:30.131 INFO:tasks.workunit.client.1.smithi066.stdout:4/876: sync 2024-02-28T16:31:30.132 INFO:tasks.workunit.client.1.smithi066.stdout:2/766: rename d0/d12/d1a/d23/d5d/l52 to d0/d12/d1a/d23/d29/d21/d2f/lfa 0 2024-02-28T16:31:30.132 INFO:tasks.workunit.client.1.smithi066.stdout:1/744: getdents d2/d27/d2b/d43/dc8 0 2024-02-28T16:31:30.134 INFO:tasks.workunit.client.1.smithi066.stdout:7/689: symlink dc/d29/d92/da6/dd8/d37/lda 0 2024-02-28T16:31:30.138 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:29 smithi001 ceph-mon[28991]: pgmap v58: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 81 MiB/s rd, 217 MiB/s wr, 443 op/s 2024-02-28T16:31:30.138 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:29 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:30.138 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:29 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:30.138 INFO:tasks.workunit.client.1.smithi066.stdout:5/679: mkdir d3/d4/dc/d56/dd6 0 2024-02-28T16:31:30.138 INFO:tasks.workunit.client.1.smithi066.stdout:9/574: creat da/d4b/d5e/fc0 x:0 0 0 2024-02-28T16:31:30.139 INFO:tasks.workunit.client.1.smithi066.stdout:2/767: stat d0/d12/d2e/d7a/c4b 0 2024-02-28T16:31:30.143 INFO:tasks.workunit.client.1.smithi066.stdout:5/680: readlink d3/d4/dc/d2f/d55/d74/lc2 0 2024-02-28T16:31:30.144 INFO:tasks.workunit.client.1.smithi066.stdout:0/815: rename d2/d27/d41/d76/d93/db2/db5/d51/f8a to d2/d27/d41/d76/d93/df0/f10d 0 2024-02-28T16:31:30.148 INFO:tasks.workunit.client.1.smithi066.stdout:3/843: rename db to db/dc/db3/dc9/de4/d12d 22 2024-02-28T16:31:30.152 INFO:tasks.workunit.client.1.smithi066.stdout:8/747: dwrite d0/db/d12/d38/f54 [0,4194304] 0 2024-02-28T16:31:30.152 INFO:tasks.workunit.client.1.smithi066.stdout:6/709: write d2/f43 [1685434,103759] 0 2024-02-28T16:31:30.152 INFO:tasks.workunit.client.1.smithi066.stdout:7/690: mknod dc/d29/d92/cdb 0 2024-02-28T16:31:30.164 INFO:tasks.workunit.client.1.smithi066.stdout:2/768: creat d0/d12/d1a/ffb x:0 0 0 2024-02-28T16:31:30.166 INFO:tasks.workunit.client.1.smithi066.stdout:8/748: write d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/fa4 [2797955,90655] 0 2024-02-28T16:31:30.168 INFO:tasks.workunit.client.1.smithi066.stdout:6/710: truncate d2/d2c/da1/d5d/d99/f5e 4560721 0 2024-02-28T16:31:30.172 INFO:tasks.workunit.client.1.smithi066.stdout:9/575: getdents da/d22/d46/d56/d8f/d28 0 2024-02-28T16:31:30.173 INFO:tasks.workunit.client.1.smithi066.stdout:7/691: symlink dc/d56/dce/ldc 0 2024-02-28T16:31:30.176 INFO:tasks.workunit.client.1.smithi066.stdout:9/576: fdatasync da/d3a/d15/fb1 0 2024-02-28T16:31:30.178 INFO:tasks.workunit.client.1.smithi066.stdout:0/816: truncate d2/d27/d41/d76/d93/d10/f39 1638524 0 2024-02-28T16:31:30.178 INFO:tasks.workunit.client.1.smithi066.stdout:2/769: chown d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/feb 25141644 1 2024-02-28T16:31:30.179 INFO:tasks.workunit.client.1.smithi066.stdout:1/745: rename d2/d18/d19/f24 to d2/d27/dc7/d54/ffb 0 2024-02-28T16:31:30.183 INFO:tasks.workunit.client.1.smithi066.stdout:6/711: unlink d2/d26/d32/c65 0 2024-02-28T16:31:30.186 INFO:tasks.workunit.client.1.smithi066.stdout:8/749: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/ffb x:0 0 0 2024-02-28T16:31:30.188 INFO:tasks.workunit.client.1.smithi066.stdout:7/692: unlink dc/d29/d92/da6/dd8/d37/f54 0 2024-02-28T16:31:30.189 INFO:tasks.workunit.client.1.smithi066.stdout:2/770: write d0/d12/d3e/d9e/fed [213219,40002] 0 2024-02-28T16:31:30.196 INFO:tasks.workunit.client.1.smithi066.stdout:7/693: chown dc/c21 50831437 1 2024-02-28T16:31:30.197 INFO:tasks.workunit.client.1.smithi066.stdout:8/750: dread - d0/d6/dc/d16/d1a/d63/d58/dac/fb4 zero size 2024-02-28T16:31:30.200 INFO:tasks.workunit.client.1.smithi066.stdout:4/877: dread d1/d23/d25/f6f [0,4194304] 0 2024-02-28T16:31:30.200 INFO:tasks.workunit.client.1.smithi066.stdout:3/844: dwrite db/dc/fca [4194304,4194304] 0 2024-02-28T16:31:30.201 INFO:tasks.workunit.client.1.smithi066.stdout:5/681: dwrite d3/d4/dc/dd/f89 [0,4194304] 0 2024-02-28T16:31:30.204 INFO:tasks.workunit.client.1.smithi066.stdout:6/712: creat d2/d2c/da1/d5d/d63/d6a/ffe x:0 0 0 2024-02-28T16:31:30.207 INFO:tasks.workunit.client.1.smithi066.stdout:5/682: stat d3/d4/dc/dd/d35/fcc 0 2024-02-28T16:31:30.208 INFO:tasks.workunit.client.1.smithi066.stdout:2/771: mknod d0/d12/d1a/d23/d5d/cfc 0 2024-02-28T16:31:30.212 INFO:tasks.workunit.client.1.smithi066.stdout:9/577: getdents da/d22/d46 0 2024-02-28T16:31:30.214 INFO:tasks.workunit.client.1.smithi066.stdout:0/817: truncate d2/d27/d41/d76/d93/db2/db5/d51/f5a 1998709 0 2024-02-28T16:31:30.216 INFO:tasks.workunit.client.1.smithi066.stdout:7/694: rmdir dc/d29/d92/da6/dd8/d1e 39 2024-02-28T16:31:30.218 INFO:tasks.workunit.client.1.smithi066.stdout:8/751: mkdir d0/d6/dc/d16/dfc 0 2024-02-28T16:31:30.220 INFO:tasks.workunit.client.1.smithi066.stdout:4/878: creat d1/d7a/f128 x:0 0 0 2024-02-28T16:31:30.222 INFO:tasks.workunit.client.1.smithi066.stdout:2/772: rmdir d0/d12/d1a/d23/d29/d21/d2f/de3 39 2024-02-28T16:31:30.224 INFO:tasks.workunit.client.1.smithi066.stdout:1/746: rename d2/d27/dc7/d51/led to d2/d18/d19/lfc 0 2024-02-28T16:31:30.224 INFO:tasks.workunit.client.1.smithi066.stdout:8/752: read - d0/d6/dc/d16/d4c/d42/fbd zero size 2024-02-28T16:31:30.229 INFO:tasks.workunit.client.1.smithi066.stdout:9/578: stat da/d4b/d53/dba/lb8 0 2024-02-28T16:31:30.232 INFO:tasks.workunit.client.1.smithi066.stdout:5/683: fdatasync d3/d1b/f37 0 2024-02-28T16:31:30.237 INFO:tasks.workunit.client.1.smithi066.stdout:6/713: creat d2/d2c/da1/d5d/d99/fff x:0 0 0 2024-02-28T16:31:30.238 INFO:tasks.workunit.client.1.smithi066.stdout:2/773: truncate d0/d12/d1a/d23/d29/d21/fd9 763314 0 2024-02-28T16:31:30.244 INFO:tasks.workunit.client.1.smithi066.stdout:3/845: rename db/d2a/d44/d48/d7d/db4/ca5 to db/dc/db3/c12e 0 2024-02-28T16:31:30.244 INFO:tasks.workunit.client.1.smithi066.stdout:2/774: readlink d0/d12/d1a/d23/d29/d21/lc2 0 2024-02-28T16:31:30.246 INFO:tasks.workunit.client.1.smithi066.stdout:5/684: symlink d3/d4/dc/dd/d2e/ld7 0 2024-02-28T16:31:30.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:29 smithi066 ceph-mon[36236]: pgmap v58: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 81 MiB/s rd, 217 MiB/s wr, 443 op/s 2024-02-28T16:31:30.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:29 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:30.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:29 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:30.250 INFO:tasks.workunit.client.1.smithi066.stdout:9/579: getdents da/d6b/dbc 0 2024-02-28T16:31:30.251 INFO:tasks.workunit.client.1.smithi066.stdout:4/879: write d1/d23/d25/d77/fe9 [10712,121414] 0 2024-02-28T16:31:30.256 INFO:tasks.workunit.client.1.smithi066.stdout:6/714: mkdir d2/d2c/da1/d5d/d63/d6a/d100 0 2024-02-28T16:31:30.258 INFO:tasks.workunit.client.1.smithi066.stdout:2/775: symlink d0/d12/d1a/d39/d7c/d87/lfd 0 2024-02-28T16:31:30.259 INFO:tasks.workunit.client.1.smithi066.stdout:5/685: write d3/d1b/f37 [959257,36709] 0 2024-02-28T16:31:30.262 INFO:tasks.workunit.client.1.smithi066.stdout:8/753: dwrite d0/db/ff1 [0,4194304] 0 2024-02-28T16:31:30.270 INFO:tasks.workunit.client.1.smithi066.stdout:3/846: write db/d13/d14/d2d/d92/fac [2243910,77991] 0 2024-02-28T16:31:30.270 INFO:tasks.workunit.client.1.smithi066.stdout:4/880: mkdir d1/d23/d25/d3e/d40/d9c/d129 0 2024-02-28T16:31:30.274 INFO:tasks.workunit.client.1.smithi066.stdout:8/754: creat d0/db/d12/d38/ffd x:0 0 0 2024-02-28T16:31:30.276 INFO:tasks.workunit.client.1.smithi066.stdout:3/847: dread - db/dc/d35/d3d/d76/fdc zero size 2024-02-28T16:31:30.276 INFO:tasks.workunit.client.1.smithi066.stdout:7/695: rename dc/d29/d92/da6/dd8/d37/d47/d88/c94 to dc/cdd 0 2024-02-28T16:31:30.278 INFO:tasks.workunit.client.1.smithi066.stdout:9/580: write da/d3a/d58/f96 [2495681,8593] 0 2024-02-28T16:31:30.281 INFO:tasks.workunit.client.1.smithi066.stdout:0/818: dread d2/d27/d41/d76/d93/f1e [4194304,4194304] 0 2024-02-28T16:31:30.284 INFO:tasks.workunit.client.1.smithi066.stdout:2/776: rename d0/d12/d1a/d23/d29/d21/f75 to d0/d12/d1a/d23/d29/d21/ffe 0 2024-02-28T16:31:30.287 INFO:tasks.workunit.client.1.smithi066.stdout:7/696: write dc/de/fa2 [153945,43037] 0 2024-02-28T16:31:30.289 INFO:tasks.workunit.client.1.smithi066.stdout:1/747: dread d2/d27/d2b/d34/f44 [0,4194304] 0 2024-02-28T16:31:30.295 INFO:tasks.workunit.client.1.smithi066.stdout:8/755: symlink d0/d6/dc/d16/d4c/d6c/d6f/lfe 0 2024-02-28T16:31:30.302 INFO:tasks.workunit.client.1.smithi066.stdout:1/748: dread d2/d27/d2b/d43/faf [0,4194304] 0 2024-02-28T16:31:30.302 INFO:tasks.workunit.client.1.smithi066.stdout:2/777: write d0/d12/d1a/d39/d7c/fdd [4136430,128011] 0 2024-02-28T16:31:30.305 INFO:tasks.workunit.client.1.smithi066.stdout:7/697: mknod dc/d29/d92/da6/dd8/d91/cde 0 2024-02-28T16:31:30.311 INFO:tasks.workunit.client.1.smithi066.stdout:8/756: write d0/d6/dc/d16/d4c/d6c/d6f/da7/fcc [1437234,8854] 0 2024-02-28T16:31:30.319 INFO:tasks.workunit.client.1.smithi066.stdout:1/749: mkdir d2/d18/d7a/dfd 0 2024-02-28T16:31:30.320 INFO:tasks.workunit.client.1.smithi066.stdout:7/698: write f4 [110607,83736] 0 2024-02-28T16:31:30.321 INFO:tasks.workunit.client.1.smithi066.stdout:4/881: dwrite d1/d23/d25/d3e/d40/d9c/fb9 [0,4194304] 0 2024-02-28T16:31:30.324 INFO:tasks.workunit.client.1.smithi066.stdout:2/778: rmdir d0/d12/d1a/d23/d29/d21/d2f 39 2024-02-28T16:31:30.324 INFO:tasks.workunit.client.1.smithi066.stdout:9/581: dwrite da/d22/d46/d56/d8f/d2d/f57 [0,4194304] 0 2024-02-28T16:31:30.326 INFO:tasks.workunit.client.1.smithi066.stdout:6/715: dwrite d2/d2c/da1/d5d/d63/f84 [0,4194304] 0 2024-02-28T16:31:30.327 INFO:tasks.workunit.client.1.smithi066.stdout:4/882: dread - d1/d18/da8/f120 zero size 2024-02-28T16:31:30.329 INFO:tasks.workunit.client.1.smithi066.stdout:0/819: truncate d2/d27/d41/d76/d93/db2/db5/d67/d94/fb8 3731505 0 2024-02-28T16:31:30.330 INFO:tasks.workunit.client.1.smithi066.stdout:3/848: dwrite db/fbd [0,4194304] 0 2024-02-28T16:31:30.333 INFO:tasks.workunit.client.1.smithi066.stdout:4/883: stat d1/lf8 0 2024-02-28T16:31:30.340 INFO:tasks.workunit.client.1.smithi066.stdout:8/757: creat d0/db/d12/d1d/d25/d4f/fff x:0 0 0 2024-02-28T16:31:30.348 INFO:tasks.workunit.client.1.smithi066.stdout:9/582: chown da/d22/d46/d56/d8f/d28/d42/lb7 30 1 2024-02-28T16:31:30.353 INFO:tasks.workunit.client.1.smithi066.stdout:6/716: creat d2/da/dcb/f101 x:0 0 0 2024-02-28T16:31:30.363 INFO:tasks.workunit.client.1.smithi066.stdout:7/699: rmdir dc/d29/d92/dd1 0 2024-02-28T16:31:30.367 INFO:tasks.workunit.client.1.smithi066.stdout:3/849: creat db/f12f x:0 0 0 2024-02-28T16:31:30.375 INFO:tasks.workunit.client.1.smithi066.stdout:9/583: creat da/d3a/d58/dab/fc1 x:0 0 0 2024-02-28T16:31:30.376 INFO:tasks.workunit.client.1.smithi066.stdout:1/750: truncate d2/d27/dc7/da6/faa 2977951 0 2024-02-28T16:31:30.377 INFO:tasks.workunit.client.1.smithi066.stdout:6/717: dread d2/d26/f47 [0,4194304] 0 2024-02-28T16:31:30.378 INFO:tasks.workunit.client.1.smithi066.stdout:4/884: rmdir d1/d23/d25/d3e/d40/d9c/d129 0 2024-02-28T16:31:30.380 INFO:tasks.workunit.client.1.smithi066.stdout:9/584: chown da/f35 638436 1 2024-02-28T16:31:30.384 INFO:tasks.workunit.client.1.smithi066.stdout:7/700: mkdir dc/d56/ddf 0 2024-02-28T16:31:30.388 INFO:tasks.workunit.client.1.smithi066.stdout:3/850: creat db/d2a/d44/d48/d7d/f130 x:0 0 0 2024-02-28T16:31:30.389 INFO:tasks.workunit.client.1.smithi066.stdout:0/820: dwrite d2/d27/d41/d76/d93/d33/d3f/d8e/fb4 [0,4194304] 0 2024-02-28T16:31:30.392 INFO:tasks.workunit.client.1.smithi066.stdout:8/758: truncate d0/db/d12/d1d/d25/fb5 3953804 0 2024-02-28T16:31:30.402 INFO:tasks.workunit.client.1.smithi066.stdout:3/851: stat db/dc/d11/daa/ded 0 2024-02-28T16:31:30.402 INFO:tasks.workunit.client.1.smithi066.stdout:9/585: truncate da/d3a/d15/f4f 864504 0 2024-02-28T16:31:30.415 INFO:tasks.workunit.client.1.smithi066.stdout:1/751: getdents d2/d27/d2b/d43/d46 0 2024-02-28T16:31:30.418 INFO:tasks.workunit.client.1.smithi066.stdout:9/586: dread - da/d22/d46/d56/d8f/d28/d42/d68/f8c zero size 2024-02-28T16:31:30.420 INFO:tasks.workunit.client.1.smithi066.stdout:9/587: write da/d3a/fac [4733194,59029] 0 2024-02-28T16:31:30.421 INFO:tasks.workunit.client.1.smithi066.stdout:3/852: mkdir db/d2a/d44/d48/d7d/db4/d131 0 2024-02-28T16:31:30.426 INFO:tasks.workunit.client.1.smithi066.stdout:7/701: creat dc/d29/d92/da6/dd8/d1e/fe0 x:0 0 0 2024-02-28T16:31:30.426 INFO:tasks.workunit.client.1.smithi066.stdout:9/588: read - da/d3a/d15/d48/f66 zero size 2024-02-28T16:31:30.435 INFO:tasks.workunit.client.1.smithi066.stdout:0/821: link d2/d27/d41/d76/d93/db2/db5/d51/l8c d2/d85/da9/l10e 0 2024-02-28T16:31:30.438 INFO:tasks.workunit.client.1.smithi066.stdout:8/759: dread d0/f23 [0,4194304] 0 2024-02-28T16:31:30.445 INFO:tasks.workunit.client.1.smithi066.stdout:9/589: rename da/d22/d46/d56/d8f/d2d/d81/l92 to da/d3a/d15/lc2 0 2024-02-28T16:31:30.447 INFO:tasks.workunit.client.1.smithi066.stdout:1/752: dwrite d2/d27/df9/fcf [0,4194304] 0 2024-02-28T16:31:30.449 INFO:tasks.workunit.client.1.smithi066.stdout:0/822: fsync d2/d27/d41/d76/d93/d10/d23/f9d 0 2024-02-28T16:31:30.453 INFO:tasks.workunit.client.1.smithi066.stdout:0/823: chown d2/d27/d41/d76/d93/d10/l38 1266 1 2024-02-28T16:31:30.454 INFO:tasks.workunit.client.1.smithi066.stdout:8/760: unlink d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/la9 0 2024-02-28T16:31:30.457 INFO:tasks.workunit.client.1.smithi066.stdout:6/718: truncate d2/da/d19/d75/f92 2525283 0 2024-02-28T16:31:30.460 INFO:tasks.workunit.client.1.smithi066.stdout:4/885: truncate d1/d23/d25/d3e/d40/d9c/d11c/d11d/fa6 1740206 0 2024-02-28T16:31:30.463 INFO:tasks.workunit.client.1.smithi066.stdout:4/886: readlink d1/d23/d25/d3e/d40/ldb 0 2024-02-28T16:31:30.466 INFO:tasks.workunit.client.1.smithi066.stdout:4/887: stat d1/d23/d25/d3e/d40/lac 0 2024-02-28T16:31:30.468 INFO:tasks.workunit.client.1.smithi066.stdout:8/761: creat d0/d6/dc/dd4/f100 x:0 0 0 2024-02-28T16:31:30.470 INFO:tasks.workunit.client.1.smithi066.stdout:1/753: write d2/fd1 [781414,115683] 0 2024-02-28T16:31:30.474 INFO:tasks.workunit.client.1.smithi066.stdout:6/719: unlink d2/d2c/d72/c7e 0 2024-02-28T16:31:30.479 INFO:tasks.workunit.client.1.smithi066.stdout:0/824: rename d2/d27/d41/d76/d93/d33/d3f/d8e/lbd to d2/d27/d41/d76/d93/db2/db5/d67/dfc/l10f 0 2024-02-28T16:31:30.484 INFO:tasks.workunit.client.1.smithi066.stdout:8/762: write d0/d6/dc/d16/d4c/d6c/d6f/f39 [2199172,6352] 0 2024-02-28T16:31:30.486 INFO:tasks.workunit.client.1.smithi066.stdout:9/590: write da/d22/d46/d56/f72 [1954818,130146] 0 2024-02-28T16:31:30.487 INFO:tasks.workunit.client.1.smithi066.stdout:1/754: symlink d2/da0/lfe 0 2024-02-28T16:31:30.492 INFO:tasks.workunit.client.1.smithi066.stdout:4/888: rename d1/d23/d25/d3e/d40/c10a to d1/d23/d25/d3e/d40/d9c/d11c/d11d/c12a 0 2024-02-28T16:31:30.496 INFO:tasks.workunit.client.1.smithi066.stdout:7/702: dwrite dc/d29/d92/da6/dd8/d1e/d45/f62 [0,4194304] 0 2024-02-28T16:31:30.496 INFO:tasks.workunit.client.1.smithi066.stdout:4/889: stat d1/d7a/f128 0 2024-02-28T16:31:30.498 INFO:tasks.workunit.client.1.smithi066.stdout:9/591: symlink da/d3a/d58/lc3 0 2024-02-28T16:31:30.500 INFO:tasks.workunit.client.1.smithi066.stdout:7/703: write dc/d29/d92/da6/dd8/d37/f8d [786042,62613] 0 2024-02-28T16:31:30.501 INFO:tasks.workunit.client.1.smithi066.stdout:1/755: rmdir d2/d27/d2b/d34 39 2024-02-28T16:31:30.503 INFO:tasks.workunit.client.1.smithi066.stdout:9/592: read - da/d3a/d15/d48/fa4 zero size 2024-02-28T16:31:30.507 INFO:tasks.workunit.client.1.smithi066.stdout:0/825: dread d2/d27/d41/d76/d93/d33/d3f/d8e/fe3 [0,4194304] 0 2024-02-28T16:31:30.508 INFO:tasks.workunit.client.1.smithi066.stdout:6/720: getdents d2/d26/d32/d46 0 2024-02-28T16:31:30.509 INFO:tasks.workunit.client.1.smithi066.stdout:4/890: mknod d1/d18/d41/dba/d100/c12b 0 2024-02-28T16:31:30.512 INFO:tasks.workunit.client.1.smithi066.stdout:0/826: fsync d2/d27/d41/d76/d93/db2/db5/d73/fd5 0 2024-02-28T16:31:30.513 INFO:tasks.workunit.client.1.smithi066.stdout:4/891: fdatasync d1/d18/d41/da7/dae/fe3 0 2024-02-28T16:31:30.516 INFO:tasks.workunit.client.1.smithi066.stdout:4/892: dread - d1/d18/d41/dba/f108 zero size 2024-02-28T16:31:30.521 INFO:tasks.workunit.client.1.smithi066.stdout:4/893: read d1/d18/f2d [3813642,37036] 0 2024-02-28T16:31:30.527 INFO:tasks.workunit.client.1.smithi066.stdout:6/721: dwrite d2/d2c/da1/d5d/d99/f5e [0,4194304] 0 2024-02-28T16:31:30.528 INFO:tasks.workunit.client.1.smithi066.stdout:1/756: write d2/d18/d7a/db1/fea [2798916,46057] 0 2024-02-28T16:31:30.532 INFO:tasks.workunit.client.1.smithi066.stdout:8/763: dwrite d0/db/d15/f31 [0,4194304] 0 2024-02-28T16:31:30.536 INFO:tasks.workunit.client.1.smithi066.stdout:8/764: read - d0/d6/dc/d16/d1a/ffa zero size 2024-02-28T16:31:30.538 INFO:tasks.workunit.client.1.smithi066.stdout:7/704: creat dc/d29/d92/da6/dd8/fe1 x:0 0 0 2024-02-28T16:31:30.538 INFO:tasks.workunit.client.1.smithi066.stdout:9/593: write da/d3a/d15/f1d [1647475,25410] 0 2024-02-28T16:31:30.541 INFO:tasks.workunit.client.1.smithi066.stdout:6/722: readlink d2/d26/dbe/ld9 0 2024-02-28T16:31:30.544 INFO:tasks.workunit.client.1.smithi066.stdout:1/757: creat d2/d27/dc7/de4/fff x:0 0 0 2024-02-28T16:31:30.545 INFO:tasks.workunit.client.1.smithi066.stdout:7/705: creat dc/d29/d92/da6/dd8/d91/fe2 x:0 0 0 2024-02-28T16:31:30.548 INFO:tasks.workunit.client.1.smithi066.stdout:7/706: chown dc/d29/d92 496490 1 2024-02-28T16:31:30.553 INFO:tasks.workunit.client.1.smithi066.stdout:0/827: getdents d2/d27/d41/d76/d93/d33/d3f 0 2024-02-28T16:31:30.555 INFO:tasks.workunit.client.1.smithi066.stdout:4/894: getdents d1/d18/da8/dcb 0 2024-02-28T16:31:30.555 INFO:tasks.workunit.client.1.smithi066.stdout:7/707: chown dc/d29/d92/da6/dd8/d1e/d45/l57 20374 1 2024-02-28T16:31:30.557 INFO:tasks.workunit.client.1.smithi066.stdout:8/765: getdents d0/db/d12/d1d/d25/d72 0 2024-02-28T16:31:30.560 INFO:tasks.workunit.client.1.smithi066.stdout:1/758: mkdir d2/d27/dc7/de2/d100 0 2024-02-28T16:31:30.568 INFO:tasks.workunit.client.1.smithi066.stdout:1/759: write d2/d27/d2b/fc6 [740987,50510] 0 2024-02-28T16:31:30.568 INFO:tasks.workunit.client.1.smithi066.stdout:0/828: unlink d2/d27/d41/d76/cdc 0 2024-02-28T16:31:30.568 INFO:tasks.workunit.client.1.smithi066.stdout:7/708: chown dc/d29/d92/da6/dd8/d1e/f98 0 1 2024-02-28T16:31:30.568 INFO:tasks.workunit.client.1.smithi066.stdout:4/895: creat d1/d18/d41/f12c x:0 0 0 2024-02-28T16:31:30.568 INFO:tasks.workunit.client.1.smithi066.stdout:1/760: rmdir d2/d27/d2b 39 2024-02-28T16:31:30.570 INFO:tasks.workunit.client.1.smithi066.stdout:8/766: truncate d0/d6/dc/d16/d4c/d6c/f91 12914308 0 2024-02-28T16:31:30.575 INFO:tasks.workunit.client.1.smithi066.stdout:4/896: symlink d1/d23/d25/l12d 0 2024-02-28T16:31:30.578 INFO:tasks.workunit.client.1.smithi066.stdout:7/709: mkdir dc/d29/d92/da6/dd8/d91/dba/de3 0 2024-02-28T16:31:30.578 INFO:tasks.workunit.client.1.smithi066.stdout:0/829: write d2/d27/d41/d76/d93/d10/d23/d6d/f103 [2633315,96887] 0 2024-02-28T16:31:30.578 INFO:tasks.workunit.client.1.smithi066.stdout:6/723: dwrite d2/d2c/da1/d5d/f64 [0,4194304] 0 2024-02-28T16:31:30.583 INFO:tasks.workunit.client.1.smithi066.stdout:1/761: getdents d2/d27/d2b/d43/d46/d7c 0 2024-02-28T16:31:30.584 INFO:tasks.workunit.client.1.smithi066.stdout:9/594: dwrite da/d22/d46/d56/d8f/d28/d42/f80 [0,4194304] 0 2024-02-28T16:31:30.584 INFO:tasks.workunit.client.1.smithi066.stdout:8/767: fsync d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/ffb 0 2024-02-28T16:31:30.586 INFO:tasks.workunit.client.1.smithi066.stdout:0/830: stat d2/d27/d41/d76/d93/d10/ld3 0 2024-02-28T16:31:30.587 INFO:tasks.workunit.client.1.smithi066.stdout:6/724: mkdir d2/da/dcc/dd6/d102 0 2024-02-28T16:31:30.593 INFO:tasks.workunit.client.1.smithi066.stdout:6/725: rmdir d2/d2c/da1 39 2024-02-28T16:31:30.597 INFO:tasks.workunit.client.1.smithi066.stdout:0/831: mknod d2/c110 0 2024-02-28T16:31:30.600 INFO:tasks.workunit.client.1.smithi066.stdout:9/595: symlink da/df/d70/d90/lc4 0 2024-02-28T16:31:30.603 INFO:tasks.workunit.client.1.smithi066.stdout:1/762: getdents d2/d18/d7a 0 2024-02-28T16:31:30.605 INFO:tasks.workunit.client.1.smithi066.stdout:9/596: fsync da/d4b/d53/dba/fb4 0 2024-02-28T16:31:30.605 INFO:tasks.workunit.client.1.smithi066.stdout:7/710: getdents dc/d29/d92/da6/dd8/d1e/da1/db7 0 2024-02-28T16:31:30.609 INFO:tasks.workunit.client.1.smithi066.stdout:1/763: fdatasync d2/d27/fa3 0 2024-02-28T16:31:30.610 INFO:tasks.workunit.client.1.smithi066.stdout:4/897: dwrite d1/d18/da8/f120 [0,4194304] 0 2024-02-28T16:31:30.612 INFO:tasks.workunit.client.1.smithi066.stdout:6/726: link d2/da/fe d2/d26/d32/d46/f103 0 2024-02-28T16:31:30.616 INFO:tasks.workunit.client.1.smithi066.stdout:0/832: mknod d2/d27/d41/d76/d93/db2/db5/d51/c111 0 2024-02-28T16:31:30.616 INFO:tasks.workunit.client.1.smithi066.stdout:2/779: sync 2024-02-28T16:31:30.616 INFO:tasks.workunit.client.1.smithi066.stdout:5/686: sync 2024-02-28T16:31:30.618 INFO:tasks.workunit.client.1.smithi066.stdout:1/764: creat d2/d18/da5/f101 x:0 0 0 2024-02-28T16:31:30.621 INFO:tasks.workunit.client.1.smithi066.stdout:7/711: unlink dc/d29/d92/da6/dd8/d37/d47/f78 0 2024-02-28T16:31:30.623 INFO:tasks.workunit.client.1.smithi066.stdout:5/687: creat d3/fd8 x:0 0 0 2024-02-28T16:31:30.623 INFO:tasks.workunit.client.1.smithi066.stdout:1/765: rmdir d2 39 2024-02-28T16:31:30.631 INFO:tasks.workunit.client.1.smithi066.stdout:9/597: getdents da/df/d70 0 2024-02-28T16:31:30.631 INFO:tasks.workunit.client.1.smithi066.stdout:4/898: getdents d1/d18/d41/dba 0 2024-02-28T16:31:30.636 INFO:tasks.workunit.client.1.smithi066.stdout:8/768: dwrite d0/f24 [0,4194304] 0 2024-02-28T16:31:30.641 INFO:tasks.workunit.client.1.smithi066.stdout:1/766: chown d2/d27/d2b/d34/da9/fdb 9 1 2024-02-28T16:31:30.642 INFO:tasks.workunit.client.1.smithi066.stdout:4/899: creat d1/d23/d25/d3e/d40/d9c/d11c/d11d/f12e x:0 0 0 2024-02-28T16:31:30.643 INFO:tasks.workunit.client.1.smithi066.stdout:8/769: chown d0/db/d15/f44 18617593 1 2024-02-28T16:31:30.645 INFO:tasks.workunit.client.1.smithi066.stdout:9/598: truncate da/f35 2092827 0 2024-02-28T16:31:30.646 INFO:tasks.workunit.client.1.smithi066.stdout:4/900: fdatasync d1/d18/d55/dc4/fd3 0 2024-02-28T16:31:30.655 INFO:tasks.workunit.client.1.smithi066.stdout:3/853: sync 2024-02-28T16:31:30.661 INFO:tasks.workunit.client.1.smithi066.stdout:9/599: chown da/d3a/f9b 2511283 1 2024-02-28T16:31:30.662 INFO:tasks.workunit.client.1.smithi066.stdout:0/833: dwrite d2/d27/d41/d76/d93/db2/db5/d67/fe1 [0,4194304] 0 2024-02-28T16:31:30.670 INFO:tasks.workunit.client.1.smithi066.stdout:2/780: dread d0/d12/d1a/d39/d7c/d87/fe5 [0,4194304] 0 2024-02-28T16:31:30.674 INFO:tasks.workunit.client.1.smithi066.stdout:9/600: truncate da/d22/d46/d56/d8f/d2d/f4c 5693328 0 2024-02-28T16:31:30.675 INFO:tasks.workunit.client.1.smithi066.stdout:1/767: dwrite d2/d27/dc7/d54/f8a [0,4194304] 0 2024-02-28T16:31:30.677 INFO:tasks.workunit.client.1.smithi066.stdout:6/727: dwrite d2/d2c/da1/d5d/d99/f49 [0,4194304] 0 2024-02-28T16:31:30.680 INFO:tasks.workunit.client.1.smithi066.stdout:9/601: readlink da/d3a/d58/lae 0 2024-02-28T16:31:30.687 INFO:tasks.workunit.client.1.smithi066.stdout:1/768: dread d2/d27/d2b/f39 [0,4194304] 0 2024-02-28T16:31:30.688 INFO:tasks.workunit.client.1.smithi066.stdout:4/901: mknod d1/d18/d41/dba/c12f 0 2024-02-28T16:31:30.688 INFO:tasks.workunit.client.1.smithi066.stdout:3/854: symlink db/d2a/d44/d48/d52/l132 0 2024-02-28T16:31:30.688 INFO:tasks.workunit.client.1.smithi066.stdout:0/834: creat d2/d27/f112 x:0 0 0 2024-02-28T16:31:30.690 INFO:tasks.workunit.client.1.smithi066.stdout:6/728: mkdir d2/d26/dbe/d104 0 2024-02-28T16:31:30.690 INFO:tasks.workunit.client.1.smithi066.stdout:3/855: stat db/d7a 0 2024-02-28T16:31:30.690 INFO:tasks.workunit.client.1.smithi066.stdout:8/770: chown d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/lc1 174 1 2024-02-28T16:31:30.690 INFO:tasks.workunit.client.1.smithi066.stdout:9/602: symlink da/d22/d46/d56/d8f/d2d/lc5 0 2024-02-28T16:31:30.692 INFO:tasks.workunit.client.1.smithi066.stdout:3/856: readlink db/d2a/d44/d48/l8a 0 2024-02-28T16:31:30.693 INFO:tasks.workunit.client.1.smithi066.stdout:9/603: stat da/df/lb0 0 2024-02-28T16:31:30.744 INFO:tasks.workunit.client.1.smithi066.stdout:3/857: stat db/d2a/l53 0 2024-02-28T16:31:30.747 INFO:tasks.workunit.client.1.smithi066.stdout:8/771: creat d0/d6/dc/f101 x:0 0 0 2024-02-28T16:31:30.752 INFO:tasks.workunit.client.1.smithi066.stdout:0/835: rename d2/d27/d41/d76/d93/db2/db5/d67 to d2/d27/d41/d76/d93/d10/d23/d6d/d113 0 2024-02-28T16:31:30.755 INFO:tasks.workunit.client.1.smithi066.stdout:4/902: creat d1/d18/d4e/f130 x:0 0 0 2024-02-28T16:31:30.756 INFO:tasks.workunit.client.1.smithi066.stdout:1/769: link d2/d18/d6b/ce8 d2/d27/dc7/d54/d84/c102 0 2024-02-28T16:31:30.759 INFO:tasks.workunit.client.1.smithi066.stdout:6/729: getdents d2/d2c/dba 0 2024-02-28T16:31:30.759 INFO:tasks.workunit.client.1.smithi066.stdout:8/772: mknod d0/d6/dc/d16/c102 0 2024-02-28T16:31:30.760 INFO:tasks.workunit.client.1.smithi066.stdout:9/604: write da/d22/f51 [764177,108022] 0 2024-02-28T16:31:30.760 INFO:tasks.workunit.client.1.smithi066.stdout:3/858: symlink db/dc/d35/d3d/d76/d89/dcc/d11e/l133 0 2024-02-28T16:31:30.763 INFO:tasks.workunit.client.1.smithi066.stdout:6/730: chown d2/da/dcc/dd6/d102 33085 1 2024-02-28T16:31:30.764 INFO:tasks.workunit.client.1.smithi066.stdout:0/836: symlink d2/d85/da9/l114 0 2024-02-28T16:31:30.765 INFO:tasks.workunit.client.1.smithi066.stdout:4/903: symlink d1/d18/d41/da7/l131 0 2024-02-28T16:31:30.767 INFO:tasks.workunit.client.1.smithi066.stdout:6/731: readlink d2/d2c/d6e/l71 0 2024-02-28T16:31:30.768 INFO:tasks.workunit.client.1.smithi066.stdout:9/605: creat da/d22/d46/d56/d8f/d28/d42/fc6 x:0 0 0 2024-02-28T16:31:30.784 INFO:tasks.workunit.client.1.smithi066.stdout:3/859: dread db/dc/fca [4194304,4194304] 0 2024-02-28T16:31:30.784 INFO:tasks.workunit.client.1.smithi066.stdout:0/837: fdatasync d2/d27/d41/d76/d93/db2/db5/d52/f81 0 2024-02-28T16:31:30.786 INFO:tasks.workunit.client.1.smithi066.stdout:8/773: dwrite d0/db/d12/d1d/d25/d4f/f9f [0,4194304] 0 2024-02-28T16:31:30.786 INFO:tasks.workunit.client.1.smithi066.stdout:1/770: truncate d2/d27/d2b/d34/da9/de9/fb8 447946 0 2024-02-28T16:31:30.793 INFO:tasks.workunit.client.1.smithi066.stdout:0/838: unlink d2/f5e 0 2024-02-28T16:31:30.795 INFO:tasks.workunit.client.1.smithi066.stdout:3/860: write db/d2a/ff7 [4273732,22572] 0 2024-02-28T16:31:30.795 INFO:tasks.workunit.client.1.smithi066.stdout:1/771: chown d2/d27/dc7/d48/dd6 4 1 2024-02-28T16:31:30.801 INFO:tasks.workunit.client.1.smithi066.stdout:0/839: mkdir d2/d27/d41/d76/d93/d115 0 2024-02-28T16:31:30.802 INFO:tasks.workunit.client.1.smithi066.stdout:4/904: dwrite d1/d18/d41/fec [0,4194304] 0 2024-02-28T16:31:30.806 INFO:tasks.workunit.client.1.smithi066.stdout:1/772: fdatasync d2/d27/d2b/f86 0 2024-02-28T16:31:30.806 INFO:tasks.workunit.client.1.smithi066.stdout:3/861: dread - db/d13/d14/d84/f103 zero size 2024-02-28T16:31:30.810 INFO:tasks.workunit.client.1.smithi066.stdout:8/774: rename d0/d6/dc/l94 to d0/d6/dc/l103 0 2024-02-28T16:31:30.810 INFO:tasks.workunit.client.1.smithi066.stdout:6/732: dwrite d2/d26/f2d [0,4194304] 0 2024-02-28T16:31:30.817 INFO:tasks.workunit.client.1.smithi066.stdout:6/733: chown d2/d2c/da1/d5d/d63 7829322 1 2024-02-28T16:31:30.817 INFO:tasks.workunit.client.1.smithi066.stdout:3/862: fdatasync db/d13/d14/d2d/f9e 0 2024-02-28T16:31:30.825 INFO:tasks.workunit.client.1.smithi066.stdout:0/840: mknod d2/d27/d41/d76/d93/d10/d23/d6d/c116 0 2024-02-28T16:31:30.830 INFO:tasks.workunit.client.1.smithi066.stdout:6/734: dread d2/da/f79 [0,4194304] 0 2024-02-28T16:31:30.834 INFO:tasks.workunit.client.1.smithi066.stdout:0/841: read d2/d27/d41/d76/d93/d10/d23/f9d [372944,116655] 0 2024-02-28T16:31:30.835 INFO:tasks.workunit.client.1.smithi066.stdout:6/735: rename d2/f6f to d2/d26/de0/f105 0 2024-02-28T16:31:30.843 INFO:tasks.workunit.client.1.smithi066.stdout:4/905: dwrite d1/d23/d25/d77/fe9 [0,4194304] 0 2024-02-28T16:31:30.844 INFO:tasks.workunit.client.1.smithi066.stdout:3/863: write db/d2a/d44/d48/d5d/f81 [1006564,52795] 0 2024-02-28T16:31:30.848 INFO:tasks.workunit.client.1.smithi066.stdout:0/842: mknod d2/d27/d41/d76/d93/db2/db5/d51/c117 0 2024-02-28T16:31:30.853 INFO:tasks.workunit.client.1.smithi066.stdout:4/906: creat d1/d7a/f132 x:0 0 0 2024-02-28T16:31:30.857 INFO:tasks.workunit.client.1.smithi066.stdout:3/864: rename db/d13/d14/d2d/d92/l95 to db/dc/l134 0 2024-02-28T16:31:30.860 INFO:tasks.workunit.client.1.smithi066.stdout:4/907: mknod d1/d23/d25/dd9/c133 0 2024-02-28T16:31:30.863 INFO:tasks.workunit.client.1.smithi066.stdout:1/773: dread d2/d18/d7a/dbc/fc5 [0,4194304] 0 2024-02-28T16:31:30.864 INFO:tasks.workunit.client.1.smithi066.stdout:4/908: read - d1/d18/d55/dc4/ff0 zero size 2024-02-28T16:31:30.865 INFO:tasks.workunit.client.1.smithi066.stdout:0/843: link d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/de4/da1/fd2 d2/f118 0 2024-02-28T16:31:30.865 INFO:tasks.workunit.client.1.smithi066.stdout:3/865: creat db/d13/d1a/f135 x:0 0 0 2024-02-28T16:31:30.866 INFO:tasks.workunit.client.1.smithi066.stdout:8/775: dread d0/d6/dc/d16/d1a/d63/d58/f70 [0,4194304] 0 2024-02-28T16:31:30.866 INFO:tasks.workunit.client.1.smithi066.stdout:1/774: creat d2/d18/d7a/d8d/f103 x:0 0 0 2024-02-28T16:31:30.868 INFO:tasks.workunit.client.1.smithi066.stdout:3/866: creat db/dc/d11/f136 x:0 0 0 2024-02-28T16:31:30.870 INFO:tasks.workunit.client.1.smithi066.stdout:1/775: chown d2/d27/d2b/d43/cda 0 1 2024-02-28T16:31:30.876 INFO:tasks.workunit.client.1.smithi066.stdout:3/867: mknod db/d13/d14/d84/c137 0 2024-02-28T16:31:30.876 INFO:tasks.workunit.client.1.smithi066.stdout:1/776: mkdir d2/d18/d7a/d8d/d104 0 2024-02-28T16:31:30.877 INFO:tasks.workunit.client.1.smithi066.stdout:7/712: sync 2024-02-28T16:31:30.880 INFO:tasks.workunit.client.1.smithi066.stdout:0/844: mkdir d2/d27/d41/d76/d93/db2/db5/d52/da2/d119 0 2024-02-28T16:31:30.881 INFO:tasks.workunit.client.1.smithi066.stdout:8/776: rename d0/db/d12/d1d/d25/d65/f84 to d0/d6/dc/dd4/f104 0 2024-02-28T16:31:30.883 INFO:tasks.workunit.client.1.smithi066.stdout:3/868: chown db/dc/d35/d3d/d76/d89/dcc/lc5 1 1 2024-02-28T16:31:30.884 INFO:tasks.workunit.client.1.smithi066.stdout:6/736: dread d2/d2c/f86 [0,4194304] 0 2024-02-28T16:31:30.884 INFO:tasks.workunit.client.1.smithi066.stdout:0/845: chown d2/d27/d41/d76/d93/d33/c3d 983758780 1 2024-02-28T16:31:30.886 INFO:tasks.workunit.client.1.smithi066.stdout:7/713: chown dc/d29/d92/da6/dd8/d9d/cab 21849095 1 2024-02-28T16:31:30.886 INFO:tasks.workunit.client.1.smithi066.stdout:4/909: dread d1/f4f [0,4194304] 0 2024-02-28T16:31:30.893 INFO:tasks.workunit.client.1.smithi066.stdout:7/714: write f4 [854967,11869] 0 2024-02-28T16:31:30.893 INFO:tasks.workunit.client.1.smithi066.stdout:0/846: creat d2/d27/f11a x:0 0 0 2024-02-28T16:31:30.896 INFO:tasks.workunit.client.1.smithi066.stdout:3/869: rename db/dc/d35/d3d/d76/fdc to db/d13/d14/d2d/d92/f138 0 2024-02-28T16:31:30.898 INFO:tasks.workunit.client.1.smithi066.stdout:8/777: rmdir d0/d6/dc/d16/d4c/d6c/d6f/d92 39 2024-02-28T16:31:30.899 INFO:tasks.workunit.client.1.smithi066.stdout:6/737: creat d2/da/d19/d75/dbd/f106 x:0 0 0 2024-02-28T16:31:30.901 INFO:tasks.workunit.client.1.smithi066.stdout:6/738: chown d2/da/d19/d9c/ld4 3 1 2024-02-28T16:31:30.905 INFO:tasks.workunit.client.1.smithi066.stdout:6/739: dread d2/da/f79 [0,4194304] 0 2024-02-28T16:31:30.908 INFO:tasks.workunit.client.1.smithi066.stdout:7/715: mknod dc/d29/d92/da6/dd8/d1e/da1/db7/ce4 0 2024-02-28T16:31:30.914 INFO:tasks.workunit.client.1.smithi066.stdout:1/777: dread d2/d18/d7a/d8d/f8f [0,4194304] 0 2024-02-28T16:31:30.915 INFO:tasks.workunit.client.1.smithi066.stdout:6/740: creat d2/d26/de0/f107 x:0 0 0 2024-02-28T16:31:30.919 INFO:tasks.workunit.client.1.smithi066.stdout:8/778: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/f105 x:0 0 0 2024-02-28T16:31:30.921 INFO:tasks.workunit.client.1.smithi066.stdout:4/910: rename d1/d18/d4e/c8f to d1/d18/d4e/da0/c134 0 2024-02-28T16:31:30.924 INFO:tasks.workunit.client.1.smithi066.stdout:6/741: readlink d2/d2c/da1/d5d/d99/l4d 0 2024-02-28T16:31:30.928 INFO:tasks.workunit.client.1.smithi066.stdout:3/870: rename db/dc/d11/d1f/fdd to db/d2a/d44/d48/d7d/f139 0 2024-02-28T16:31:30.928 INFO:tasks.workunit.client.1.smithi066.stdout:6/742: stat d2/d2c/d6e/fe5 0 2024-02-28T16:31:30.935 INFO:tasks.workunit.client.1.smithi066.stdout:8/779: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/d106 0 2024-02-28T16:31:30.937 INFO:tasks.workunit.client.1.smithi066.stdout:7/716: truncate dc/d29/d92/da6/fa7 1316774 0 2024-02-28T16:31:30.937 INFO:tasks.workunit.client.1.smithi066.stdout:3/871: fdatasync db/d13/d14/dad/fc7 0 2024-02-28T16:31:30.943 INFO:tasks.workunit.client.1.smithi066.stdout:1/778: dread d2/d18/d7a/f8b [0,4194304] 0 2024-02-28T16:31:30.943 INFO:tasks.workunit.client.1.smithi066.stdout:3/872: write db/d13/d14/ff3 [962847,101808] 0 2024-02-28T16:31:30.943 INFO:tasks.workunit.client.1.smithi066.stdout:6/743: creat d2/da/d19/d75/da6/daa/dbf/f108 x:0 0 0 2024-02-28T16:31:30.945 INFO:tasks.workunit.client.1.smithi066.stdout:0/847: dwrite d2/d85/d8d/feb [0,4194304] 0 2024-02-28T16:31:30.955 INFO:tasks.workunit.client.1.smithi066.stdout:3/873: creat db/d13/d1a/f13a x:0 0 0 2024-02-28T16:31:30.959 INFO:tasks.workunit.client.1.smithi066.stdout:0/848: write d2/d27/d41/d76/d93/d10/d23/fc4 [1116044,119830] 0 2024-02-28T16:31:30.965 INFO:tasks.workunit.client.1.smithi066.stdout:3/874: fsync db/dc/d35/d3d/f97 0 2024-02-28T16:31:30.966 INFO:tasks.workunit.client.1.smithi066.stdout:7/717: link dc/l3b dc/d29/d92/da6/dd8/d91/db9/le5 0 2024-02-28T16:31:30.966 INFO:tasks.workunit.client.1.smithi066.stdout:9/606: sync 2024-02-28T16:31:30.966 INFO:tasks.workunit.client.1.smithi066.stdout:5/688: sync 2024-02-28T16:31:30.966 INFO:tasks.workunit.client.1.smithi066.stdout:2/781: sync 2024-02-28T16:31:30.968 INFO:tasks.workunit.client.1.smithi066.stdout:0/849: mkdir d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/de4/d11b 0 2024-02-28T16:31:30.971 INFO:tasks.workunit.client.1.smithi066.stdout:4/911: dread d1/d18/da8/fd6 [0,4194304] 0 2024-02-28T16:31:30.972 INFO:tasks.workunit.client.1.smithi066.stdout:3/875: mknod db/d2a/d44/d48/d52/c13b 0 2024-02-28T16:31:30.976 INFO:tasks.workunit.client.1.smithi066.stdout:7/718: write dc/d29/d92/da6/dd8/d91/f9b [1906863,3277] 0 2024-02-28T16:31:30.979 INFO:tasks.workunit.client.1.smithi066.stdout:8/780: dwrite d0/d6/dc/d16/d4c/d6c/d6f/d92/fc0 [0,4194304] 0 2024-02-28T16:31:30.979 INFO:tasks.workunit.client.1.smithi066.stdout:3/876: mkdir db/dc/d11/d1f/d115/d13c 0 2024-02-28T16:31:30.980 INFO:tasks.workunit.client.1.smithi066.stdout:1/779: dwrite d2/d27/d2b/d43/f9b [0,4194304] 0 2024-02-28T16:31:30.980 INFO:tasks.workunit.client.1.smithi066.stdout:9/607: creat da/d4b/d53/fc7 x:0 0 0 2024-02-28T16:31:30.985 INFO:tasks.workunit.client.1.smithi066.stdout:5/689: mkdir d3/d4/dc/dd9 0 2024-02-28T16:31:30.985 INFO:tasks.workunit.client.1.smithi066.stdout:7/719: dread - dc/de/fc5 zero size 2024-02-28T16:31:30.987 INFO:tasks.workunit.client.1.smithi066.stdout:0/850: mknod d2/d27/d41/d76/d93/d10/d23/db6/de0/dfd/c11c 0 2024-02-28T16:31:30.993 INFO:tasks.workunit.client.1.smithi066.stdout:2/782: creat d0/d12/d1a/d23/ddc/fff x:0 0 0 2024-02-28T16:31:30.993 INFO:tasks.workunit.client.1.smithi066.stdout:0/851: readlink d2/d27/d41/d76/d93/d10/d23/l8f 0 2024-02-28T16:31:30.993 INFO:tasks.workunit.client.1.smithi066.stdout:4/912: creat d1/d23/d25/d3e/d40/d9c/d11c/f135 x:0 0 0 2024-02-28T16:31:30.995 INFO:tasks.workunit.client.1.smithi066.stdout:3/877: write db/dc/d11/d1f/f2e [3438662,123753] 0 2024-02-28T16:31:30.999 INFO:tasks.workunit.client.1.smithi066.stdout:8/781: unlink d0/db/d12/d38/f54 0 2024-02-28T16:31:30.999 INFO:tasks.workunit.client.1.smithi066.stdout:9/608: stat da/d22/d46/d56/d8f/d28/d42/f80 0 2024-02-28T16:31:31.002 INFO:tasks.workunit.client.1.smithi066.stdout:0/852: dread - d2/d27/d41/d76/d93/db2/fe6 zero size 2024-02-28T16:31:31.002 INFO:tasks.workunit.client.1.smithi066.stdout:3/878: readlink db/d2a/l53 0 2024-02-28T16:31:31.003 INFO:tasks.workunit.client.1.smithi066.stdout:5/690: write d3/d1b/d39/f44 [387659,10023] 0 2024-02-28T16:31:31.005 INFO:tasks.workunit.client.1.smithi066.stdout:4/913: truncate d1/f10d 834682 0 2024-02-28T16:31:31.010 INFO:tasks.workunit.client.1.smithi066.stdout:9/609: chown da/d22/d46/d56/d8f/d2d/f57 21220493 1 2024-02-28T16:31:31.014 INFO:tasks.workunit.client.1.smithi066.stdout:4/914: write d1/d18/da8/ff4 [546764,59207] 0 2024-02-28T16:31:31.016 INFO:tasks.workunit.client.1.smithi066.stdout:5/691: truncate d3/d1b/d39/fa6 697897 0 2024-02-28T16:31:31.020 INFO:tasks.workunit.client.1.smithi066.stdout:1/780: truncate d2/d27/d2b/d34/f44 3793866 0 2024-02-28T16:31:31.020 INFO:tasks.workunit.client.1.smithi066.stdout:0/853: symlink d2/d27/d41/d76/d93/d10/d23/d6d/l11d 0 2024-02-28T16:31:31.022 INFO:tasks.workunit.client.1.smithi066.stdout:3/879: rename db/d13/d1a/d25/c79 to db/dc/d11/d1f/c13d 0 2024-02-28T16:31:31.024 INFO:tasks.workunit.client.1.smithi066.stdout:1/781: fdatasync d2/d27/d2b/d43/f96 0 2024-02-28T16:31:31.026 INFO:tasks.workunit.client.1.smithi066.stdout:4/915: creat d1/d23/d25/d3e/d40/d9c/d11c/d11d/f136 x:0 0 0 2024-02-28T16:31:31.027 INFO:tasks.workunit.client.1.smithi066.stdout:9/610: creat da/d22/d46/d56/d8f/fc8 x:0 0 0 2024-02-28T16:31:31.030 INFO:tasks.workunit.client.1.smithi066.stdout:0/854: unlink d2/d27/d41/d76/d93/d10/f98 0 2024-02-28T16:31:31.033 INFO:tasks.workunit.client.1.smithi066.stdout:3/880: chown db/d13/d14/d84/c137 115100047 1 2024-02-28T16:31:31.035 INFO:tasks.workunit.client.1.smithi066.stdout:2/783: dwrite d0/d12/d3e/f63 [0,4194304] 0 2024-02-28T16:31:31.036 INFO:tasks.workunit.client.1.smithi066.stdout:9/611: stat da/d3a/d58/d97 0 2024-02-28T16:31:31.038 INFO:tasks.workunit.client.1.smithi066.stdout:5/692: rename d3/d4/dc/dd/f89 to d3/d4/dc/fda 0 2024-02-28T16:31:31.038 INFO:tasks.workunit.client.1.smithi066.stdout:0/855: write d2/d85/d8d/fed [1699542,26202] 0 2024-02-28T16:31:31.040 INFO:tasks.workunit.client.1.smithi066.stdout:4/916: write d1/d18/d41/dba/f9d [1873253,31438] 0 2024-02-28T16:31:31.040 INFO:tasks.workunit.client.1.smithi066.stdout:1/782: stat d2/d18/d7a/dbc/lc0 0 2024-02-28T16:31:31.046 INFO:tasks.workunit.client.1.smithi066.stdout:8/782: dwrite d0/d6/dc/d16/d1a/ffa [0,4194304] 0 2024-02-28T16:31:31.049 INFO:tasks.workunit.client.1.smithi066.stdout:3/881: write db/dc/fca [7215098,51171] 0 2024-02-28T16:31:31.051 INFO:tasks.workunit.client.1.smithi066.stdout:9/612: unlink da/d3a/d58/dab/fc1 0 2024-02-28T16:31:31.053 INFO:tasks.workunit.client.1.smithi066.stdout:8/783: fdatasync d0/db/d12/d1d/d25/d72/f77 0 2024-02-28T16:31:31.053 INFO:tasks.workunit.client.1.smithi066.stdout:4/917: mkdir d1/d23/d25/dd9/d137 0 2024-02-28T16:31:31.061 INFO:tasks.workunit.client.1.smithi066.stdout:7/720: dread dc/d29/d92/da6/dd8/f5b [0,4194304] 0 2024-02-28T16:31:31.063 INFO:tasks.workunit.client.1.smithi066.stdout:9/613: mkdir da/df/d70/dc9 0 2024-02-28T16:31:31.065 INFO:tasks.workunit.client.1.smithi066.stdout:0/856: truncate d2/d27/d41/d76/d93/db2/db5/d73/fe8 511322 0 2024-02-28T16:31:31.065 INFO:tasks.workunit.client.1.smithi066.stdout:2/784: stat d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/df5 0 2024-02-28T16:31:31.068 INFO:tasks.workunit.client.1.smithi066.stdout:0/857: chown d2/f28 456 1 2024-02-28T16:31:31.068 INFO:tasks.workunit.client.1.smithi066.stdout:9/614: dread - da/d22/d46/d56/fb5 zero size 2024-02-28T16:31:31.071 INFO:tasks.workunit.client.1.smithi066.stdout:3/882: creat db/d13/d14/d2d/de3/f13e x:0 0 0 2024-02-28T16:31:31.076 INFO:tasks.workunit.client.1.smithi066.stdout:2/785: readlink d0/d12/d1a/d23/d29/l49 0 2024-02-28T16:31:31.077 INFO:tasks.workunit.client.1.smithi066.stdout:0/858: fdatasync d2/d27/d41/d76/d93/db2/db5/d52/f81 0 2024-02-28T16:31:31.078 INFO:tasks.workunit.client.1.smithi066.stdout:5/693: dread d3/d1b/d39/f44 [0,4194304] 0 2024-02-28T16:31:31.083 INFO:tasks.workunit.client.1.smithi066.stdout:1/783: truncate d2/f75 1313006 0 2024-02-28T16:31:31.086 INFO:tasks.workunit.client.1.smithi066.stdout:4/918: write d1/d23/d25/d3e/d40/d9c/d11c/d11d/f5d [8962836,127973] 0 2024-02-28T16:31:31.088 INFO:tasks.workunit.client.1.smithi066.stdout:5/694: fsync d3/d4/dc/dd/d41/f65 0 2024-02-28T16:31:31.088 INFO:tasks.workunit.client.1.smithi066.stdout:9/615: dread da/df/d70/d90/f93 [0,4194304] 0 2024-02-28T16:31:31.093 INFO:tasks.workunit.client.1.smithi066.stdout:1/784: mkdir d2/d18/d7a/d8d/d105 0 2024-02-28T16:31:31.094 INFO:tasks.workunit.client.1.smithi066.stdout:8/784: dwrite d0/d6/dc/d16/d4c/d6c/f91 [8388608,4194304] 0 2024-02-28T16:31:31.099 INFO:tasks.workunit.client.1.smithi066.stdout:9/616: creat da/d6b/dbc/fca x:0 0 0 2024-02-28T16:31:31.100 INFO:tasks.workunit.client.1.smithi066.stdout:6/744: sync 2024-02-28T16:31:31.101 INFO:tasks.workunit.client.1.smithi066.stdout:1/785: write d2/d27/d2b/d43/d46/f63 [4157317,37958] 0 2024-02-28T16:31:31.108 INFO:tasks.workunit.client.1.smithi066.stdout:1/786: chown d2/d27/dc7/d51/f99 7606969 1 2024-02-28T16:31:31.112 INFO:tasks.workunit.client.1.smithi066.stdout:5/695: link d3/d4/dc/d2f/ca1 d3/d4/dc/d56/d93/cdb 0 2024-02-28T16:31:31.113 INFO:tasks.workunit.client.1.smithi066.stdout:7/721: dread dc/d29/d92/da6/dd8/d1e/f59 [0,4194304] 0 2024-02-28T16:31:31.116 INFO:tasks.workunit.client.1.smithi066.stdout:9/617: mkdir da/d4b/d53/dcb 0 2024-02-28T16:31:31.117 INFO:tasks.workunit.client.1.smithi066.stdout:0/859: truncate d2/d85/d8d/fd0 4621205 0 2024-02-28T16:31:31.117 INFO:tasks.workunit.client.1.smithi066.stdout:2/786: dwrite d0/d12/d3e/d9e/fa3 [0,4194304] 0 2024-02-28T16:31:31.119 INFO:tasks.workunit.client.1.smithi066.stdout:8/785: mkdir d0/d6/dc/d16/d4c/d6c/d107 0 2024-02-28T16:31:31.121 INFO:tasks.workunit.client.1.smithi066.stdout:1/787: readlink d2/d18/d19/lfc 0 2024-02-28T16:31:31.129 INFO:tasks.workunit.client.1.smithi066.stdout:3/883: dread db/d13/d1a/d33/f39 [0,4194304] 0 2024-02-28T16:31:31.129 INFO:tasks.workunit.client.1.smithi066.stdout:4/919: dwrite d1/dfb/f125 [0,4194304] 0 2024-02-28T16:31:31.132 INFO:tasks.workunit.client.1.smithi066.stdout:0/860: fsync d2/d27/d41/d76/d93/d10/d23/d6d/f79 0 2024-02-28T16:31:31.132 INFO:tasks.workunit.client.1.smithi066.stdout:5/696: rename d3/d4/dc/d2f/f32 to d3/d4/dc/dd/d35/d86/fdc 0 2024-02-28T16:31:31.132 INFO:tasks.workunit.client.1.smithi066.stdout:8/786: dread d0/d8/fc7 [0,4194304] 0 2024-02-28T16:31:31.140 INFO:tasks.workunit.client.1.smithi066.stdout:1/788: read d2/d27/d2b/d43/d46/f85 [473924,104691] 0 2024-02-28T16:31:31.141 INFO:tasks.workunit.client.1.smithi066.stdout:5/697: truncate d3/d1b/d7d/d9c/fb9 327134 0 2024-02-28T16:31:31.145 INFO:tasks.workunit.client.1.smithi066.stdout:4/920: chown d1/d18/d4e/ldf 1038 1 2024-02-28T16:31:31.146 INFO:tasks.workunit.client.1.smithi066.stdout:5/698: read - d3/d4/dc/dd/d2e/d8f/f9a zero size 2024-02-28T16:31:31.146 INFO:tasks.workunit.client.1.smithi066.stdout:2/787: dread d0/d12/d3e/f63 [0,4194304] 0 2024-02-28T16:31:31.147 INFO:tasks.workunit.client.1.smithi066.stdout:8/787: mkdir d0/d6/dc/d16/d1a/d63/d58/d5a/d108 0 2024-02-28T16:31:31.148 INFO:tasks.workunit.client.1.smithi066.stdout:0/861: dread - d2/d27/d41/d76/d93/d33/f6b zero size 2024-02-28T16:31:31.149 INFO:tasks.workunit.client.1.smithi066.stdout:2/788: chown d0/d12/lf3 433533 1 2024-02-28T16:31:31.150 INFO:tasks.workunit.client.1.smithi066.stdout:3/884: creat db/d13/d1a/d33/d124/f13f x:0 0 0 2024-02-28T16:31:31.151 INFO:tasks.workunit.client.1.smithi066.stdout:8/788: chown d0/db/d15/f31 14119840 1 2024-02-28T16:31:31.152 INFO:tasks.workunit.client.1.smithi066.stdout:7/722: getdents dc/d29/d92/da6/dd8/d91 0 2024-02-28T16:31:31.153 INFO:tasks.workunit.client.1.smithi066.stdout:4/921: symlink d1/d23/d25/d77/l138 0 2024-02-28T16:31:31.157 INFO:tasks.workunit.client.1.smithi066.stdout:1/789: mknod d2/d27/dc7/c106 0 2024-02-28T16:31:31.159 INFO:tasks.workunit.client.1.smithi066.stdout:8/789: rmdir d0/d6/dc/d16/d4c/d6c/d6f/d92 39 2024-02-28T16:31:31.161 INFO:tasks.workunit.client.1.smithi066.stdout:7/723: symlink dc/d56/ddf/le6 0 2024-02-28T16:31:31.162 INFO:tasks.workunit.client.1.smithi066.stdout:3/885: mknod db/dc/db3/dff/c140 0 2024-02-28T16:31:31.166 INFO:tasks.workunit.client.1.smithi066.stdout:7/724: chown dc/de/d87/fa9 238189 1 2024-02-28T16:31:31.169 INFO:tasks.workunit.client.1.smithi066.stdout:8/790: readlink d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/lcd 0 2024-02-28T16:31:31.171 INFO:tasks.workunit.client.1.smithi066.stdout:3/886: dread - db/dc/d35/d3d/d76/d89/dcc/fb2 zero size 2024-02-28T16:31:31.172 INFO:tasks.workunit.client.1.smithi066.stdout:7/725: mknod dc/d56/d97/ce7 0 2024-02-28T16:31:31.173 INFO:tasks.workunit.client.1.smithi066.stdout:4/922: link d1/d23/d25/d3e/d40/c121 d1/c139 0 2024-02-28T16:31:31.175 INFO:tasks.workunit.client.1.smithi066.stdout:7/726: dread - dc/d29/d92/da6/fb1 zero size 2024-02-28T16:31:31.181 INFO:tasks.workunit.client.1.smithi066.stdout:5/699: dread d3/d1b/d20/f63 [0,4194304] 0 2024-02-28T16:31:31.181 INFO:tasks.workunit.client.1.smithi066.stdout:2/789: dwrite d0/f9 [4194304,4194304] 0 2024-02-28T16:31:31.182 INFO:tasks.workunit.client.1.smithi066.stdout:3/887: symlink db/d2a/d44/d48/d52/d77/l141 0 2024-02-28T16:31:31.182 INFO:tasks.workunit.client.1.smithi066.stdout:4/923: mkdir d1/d18/da8/dcb/d13a 0 2024-02-28T16:31:31.186 INFO:tasks.workunit.client.1.smithi066.stdout:7/727: creat dc/d56/d97/dd5/fe8 x:0 0 0 2024-02-28T16:31:31.189 INFO:tasks.workunit.client.1.smithi066.stdout:5/700: rename d3/d1b/d7d/d9c to d3/d1b/d20/ddd 0 2024-02-28T16:31:31.190 INFO:tasks.workunit.client.1.smithi066.stdout:0/862: dread d2/d27/d41/d76/d93/d10/f26 [0,4194304] 0 2024-02-28T16:31:31.194 INFO:tasks.workunit.client.1.smithi066.stdout:7/728: readlink dc/lf 0 2024-02-28T16:31:31.198 INFO:tasks.workunit.client.1.smithi066.stdout:3/888: getdents db/dc/d35/d3d 0 2024-02-28T16:31:31.199 INFO:tasks.workunit.client.1.smithi066.stdout:5/701: symlink d3/d4/dc/d2f/d55/lde 0 2024-02-28T16:31:31.201 INFO:tasks.workunit.client.1.smithi066.stdout:4/924: dread d1/f11 [0,4194304] 0 2024-02-28T16:31:31.202 INFO:tasks.workunit.client.1.smithi066.stdout:7/729: creat dc/d56/d97/dd5/fe9 x:0 0 0 2024-02-28T16:31:31.204 INFO:tasks.workunit.client.1.smithi066.stdout:8/791: dwrite d0/d8/fc7 [0,4194304] 0 2024-02-28T16:31:31.208 INFO:tasks.workunit.client.1.smithi066.stdout:5/702: read d3/d4/dc/dd/d2e/d18/f4b [657390,22113] 0 2024-02-28T16:31:31.208 INFO:tasks.workunit.client.1.smithi066.stdout:4/925: write d1/d18/d41/f95 [2618805,126938] 0 2024-02-28T16:31:31.209 INFO:tasks.workunit.client.1.smithi066.stdout:7/730: symlink dc/d29/d92/da6/lea 0 2024-02-28T16:31:31.214 INFO:tasks.workunit.client.1.smithi066.stdout:4/926: chown d1/d18/d41/f68 232567534 1 2024-02-28T16:31:31.222 INFO:tasks.workunit.client.1.smithi066.stdout:2/790: dwrite d0/d12/d3e/d9e/fb2 [0,4194304] 0 2024-02-28T16:31:31.222 INFO:tasks.workunit.client.1.smithi066.stdout:4/927: dread - d1/d18/d55/dc4/fd3 zero size 2024-02-28T16:31:31.223 INFO:tasks.workunit.client.1.smithi066.stdout:0/863: dwrite d2/f118 [0,4194304] 0 2024-02-28T16:31:31.232 INFO:tasks.workunit.client.1.smithi066.stdout:4/928: rename d1/d23/d25/d77/c8a to d1/c13b 0 2024-02-28T16:31:31.236 INFO:tasks.workunit.client.1.smithi066.stdout:3/889: dwrite db/dc/d35/d3d/d76/fb9 [4194304,4194304] 0 2024-02-28T16:31:31.240 INFO:tasks.workunit.client.1.smithi066.stdout:4/929: fsync d1/d18/d55/dc4/fd3 0 2024-02-28T16:31:31.249 INFO:tasks.workunit.client.1.smithi066.stdout:8/792: dwrite d0/d6/dc/d16/d4c/d6c/f78 [0,4194304] 0 2024-02-28T16:31:31.251 INFO:tasks.workunit.client.1.smithi066.stdout:4/930: chown d1/d18/c7d 1048511283 1 2024-02-28T16:31:31.257 INFO:tasks.workunit.client.1.smithi066.stdout:5/703: dwrite d3/d4/fc3 [0,4194304] 0 2024-02-28T16:31:31.257 INFO:tasks.workunit.client.1.smithi066.stdout:7/731: dwrite dc/d6e/f6f [0,4194304] 0 2024-02-28T16:31:31.261 INFO:tasks.workunit.client.1.smithi066.stdout:4/931: dread - d1/d18/da8/fbe zero size 2024-02-28T16:31:31.265 INFO:tasks.workunit.client.1.smithi066.stdout:8/793: chown d0/d6/dc/d16/d4c/d6c/d6f/d92/fc0 98 1 2024-02-28T16:31:31.265 INFO:tasks.workunit.client.1.smithi066.stdout:7/732: chown dc/de/c4f 0 1 2024-02-28T16:31:31.268 INFO:tasks.workunit.client.1.smithi066.stdout:5/704: symlink d3/d1b/d20/dc1/ldf 0 2024-02-28T16:31:31.268 INFO:tasks.workunit.client.1.smithi066.stdout:0/864: dwrite d2/d85/f9f [0,4194304] 0 2024-02-28T16:31:31.269 INFO:tasks.workunit.client.1.smithi066.stdout:2/791: dwrite d0/d12/d1a/d39/d73/fbc [4194304,4194304] 0 2024-02-28T16:31:31.279 INFO:tasks.workunit.client.1.smithi066.stdout:8/794: readlink d0/d6/dc/d16/d4c/led 0 2024-02-28T16:31:31.280 INFO:tasks.workunit.client.1.smithi066.stdout:3/890: truncate db/dc/d35/fd5 93725 0 2024-02-28T16:31:31.284 INFO:tasks.workunit.client.1.smithi066.stdout:5/705: symlink d3/d4/dc/dd/d2e/d8f/le0 0 2024-02-28T16:31:31.285 INFO:tasks.workunit.client.1.smithi066.stdout:0/865: write d2/d27/d41/d76/d93/d10/d23/f7a [271127,26128] 0 2024-02-28T16:31:31.285 INFO:tasks.workunit.client.1.smithi066.stdout:2/792: dread - d0/d12/d1a/d23/fe1 zero size 2024-02-28T16:31:31.293 INFO:tasks.workunit.client.1.smithi066.stdout:3/891: unlink db/fbd 0 2024-02-28T16:31:31.296 INFO:tasks.workunit.client.1.smithi066.stdout:7/733: dwrite dc/de/fa2 [0,4194304] 0 2024-02-28T16:31:31.298 INFO:tasks.workunit.client.1.smithi066.stdout:4/932: rename d1/d18/c34 to d1/d18/da8/dcb/d13a/c13c 0 2024-02-28T16:31:31.299 INFO:tasks.workunit.client.1.smithi066.stdout:7/734: chown dc/d29/d92/da6/dd8/d37/d47 1 1 2024-02-28T16:31:31.304 INFO:tasks.workunit.client.1.smithi066.stdout:8/795: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/f109 x:0 0 0 2024-02-28T16:31:31.304 INFO:tasks.workunit.client.1.smithi066.stdout:3/892: mkdir db/d2a/d142 0 2024-02-28T16:31:31.305 INFO:tasks.workunit.client.1.smithi066.stdout:2/793: write d0/d12/d2e/f84 [3885658,10865] 0 2024-02-28T16:31:31.309 INFO:tasks.workunit.client.1.smithi066.stdout:2/794: chown d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/l91 653058 1 2024-02-28T16:31:31.312 INFO:tasks.workunit.client.1.smithi066.stdout:2/795: write d0/ff0 [736759,23230] 0 2024-02-28T16:31:31.313 INFO:tasks.workunit.client.1.smithi066.stdout:8/796: mknod d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/dad/db9/c10a 0 2024-02-28T16:31:31.314 INFO:tasks.workunit.client.1.smithi066.stdout:0/866: link d2/d27/d41/d76/d93/db2/db5/d52/da2/cc8 d2/d85/d8d/c11e 0 2024-02-28T16:31:31.319 INFO:tasks.workunit.client.1.smithi066.stdout:2/796: rmdir d0/d12/d1a/d23/d29/d21/d2f 39 2024-02-28T16:31:31.321 INFO:tasks.workunit.client.1.smithi066.stdout:4/933: creat d1/d18/f13d x:0 0 0 2024-02-28T16:31:31.324 INFO:tasks.workunit.client.1.smithi066.stdout:4/934: rmdir d1/d23/d25/dd9 39 2024-02-28T16:31:31.325 INFO:tasks.workunit.client.1.smithi066.stdout:2/797: mknod d0/d12/d1a/d39/d7c/d87/dc4/c100 0 2024-02-28T16:31:31.327 INFO:tasks.workunit.client.1.smithi066.stdout:8/797: truncate d0/db/d12/d1d/d25/f74 300643 0 2024-02-28T16:31:31.327 INFO:tasks.workunit.client.1.smithi066.stdout:3/893: getdents db/d13/d14/d2d 0 2024-02-28T16:31:31.328 INFO:tasks.workunit.client.1.smithi066.stdout:2/798: write d0/f71 [5384653,106125] 0 2024-02-28T16:31:31.329 INFO:tasks.workunit.client.1.smithi066.stdout:4/935: creat d1/d23/d25/d3e/f13e x:0 0 0 2024-02-28T16:31:31.335 INFO:tasks.workunit.client.1.smithi066.stdout:8/798: mkdir d0/d6/dc/d16/d1a/d63/d58/d5a/d10b 0 2024-02-28T16:31:31.340 INFO:tasks.workunit.client.1.smithi066.stdout:2/799: symlink d0/d12/d1a/d23/d29/l101 0 2024-02-28T16:31:31.344 INFO:tasks.workunit.client.1.smithi066.stdout:5/706: write d3/d4/f71 [682608,130712] 0 2024-02-28T16:31:31.346 INFO:tasks.workunit.client.1.smithi066.stdout:7/735: dread dc/de/f3f [0,4194304] 0 2024-02-28T16:31:31.349 INFO:tasks.workunit.client.1.smithi066.stdout:4/936: rename d1/d18/d7e/dbb/ld2 to d1/d23/l13f 0 2024-02-28T16:31:31.354 INFO:tasks.workunit.client.1.smithi066.stdout:5/707: symlink d3/d4/dc/d2f/le1 0 2024-02-28T16:31:31.356 INFO:tasks.workunit.client.1.smithi066.stdout:2/800: rmdir d0/d12/d1a/d23/d29/d21/d2f/de3 39 2024-02-28T16:31:31.358 INFO:tasks.workunit.client.1.smithi066.stdout:3/894: dwrite db/dc/d35/d3d/d76/d89/dcc/f100 [0,4194304] 0 2024-02-28T16:31:31.361 INFO:tasks.workunit.client.1.smithi066.stdout:3/895: fdatasync db/dc/d11/d1f/f2e 0 2024-02-28T16:31:31.362 INFO:tasks.workunit.client.1.smithi066.stdout:4/937: rename d1/d7a/fd1 to d1/d18/da8/dcb/f140 0 2024-02-28T16:31:31.364 INFO:tasks.workunit.client.1.smithi066.stdout:2/801: symlink d0/d12/d1a/d39/d7c/l102 0 2024-02-28T16:31:31.369 INFO:tasks.workunit.client.1.smithi066.stdout:0/867: dwrite d2/d27/d41/d76/d93/d10/f47 [8388608,4194304] 0 2024-02-28T16:31:31.373 INFO:tasks.workunit.client.1.smithi066.stdout:7/736: dwrite dc/d29/d92/da6/dd8/d91/dba/fd9 [0,4194304] 0 2024-02-28T16:31:31.377 INFO:tasks.workunit.client.1.smithi066.stdout:3/896: symlink db/d13/d1a/l143 0 2024-02-28T16:31:31.377 INFO:tasks.workunit.client.1.smithi066.stdout:0/868: write d2/d27/d41/d76/d93/db2/db5/d60/f95 [7805144,5320] 0 2024-02-28T16:31:31.383 INFO:tasks.workunit.client.1.smithi066.stdout:2/802: fdatasync d0/d12/d1a/d23/d29/d21/d2f/d67/f97 0 2024-02-28T16:31:31.393 INFO:tasks.workunit.client.1.smithi066.stdout:0/869: mkdir d2/d27/d41/d76/dcf/d11f 0 2024-02-28T16:31:31.394 INFO:tasks.workunit.client.1.smithi066.stdout:8/799: dwrite d0/db/d12/d1d/d25/d4f/f82 [0,4194304] 0 2024-02-28T16:31:31.396 INFO:tasks.workunit.client.1.smithi066.stdout:0/870: fdatasync d2/d27/d41/d76/d93/db2/fe6 0 2024-02-28T16:31:31.400 INFO:tasks.workunit.client.1.smithi066.stdout:2/803: write d0/d12/d1a/d23/d29/d21/d2f/fa2 [3840373,72768] 0 2024-02-28T16:31:31.402 INFO:tasks.workunit.client.1.smithi066.stdout:4/938: link d1/d23/d25/d77/c92 d1/d18/d4e/da0/c141 0 2024-02-28T16:31:31.404 INFO:tasks.workunit.client.1.smithi066.stdout:3/897: rename db/dc/d35/d3d/d76/d89/f111 to db/d13/d1a/d25/dd1/f144 0 2024-02-28T16:31:31.406 INFO:tasks.workunit.client.1.smithi066.stdout:2/804: fsync d0/d12/d1a/d39/d7c/d87/dc4/ff1 0 2024-02-28T16:31:31.410 INFO:tasks.workunit.client.1.smithi066.stdout:8/800: creat d0/d6/dc/d16/d4c/d42/f10c x:0 0 0 2024-02-28T16:31:31.415 INFO:tasks.workunit.client.1.smithi066.stdout:5/708: read d3/d4/dc/dd/d2e/f79 [4308481,74417] 0 2024-02-28T16:31:31.417 INFO:tasks.workunit.client.1.smithi066.stdout:0/871: dread d2/d85/f9f [0,4194304] 0 2024-02-28T16:31:31.418 INFO:tasks.workunit.client.1.smithi066.stdout:2/805: mkdir d0/d12/d1a/d23/ddc/d103 0 2024-02-28T16:31:31.422 INFO:tasks.workunit.client.1.smithi066.stdout:2/806: dread - d0/d12/d1a/d23/fa6 zero size 2024-02-28T16:31:31.424 INFO:tasks.workunit.client.1.smithi066.stdout:3/898: fdatasync db/d2a/d44/d48/d7d/f139 0 2024-02-28T16:31:31.425 INFO:tasks.workunit.client.1.smithi066.stdout:8/801: mknod d0/d6/dc/d16/d1a/d63/d58/c10d 0 2024-02-28T16:31:31.432 INFO:tasks.workunit.client.1.smithi066.stdout:8/802: write d0/db/d15/fba [34721,124745] 0 2024-02-28T16:31:31.434 INFO:tasks.workunit.client.1.smithi066.stdout:6/745: sync 2024-02-28T16:31:31.434 INFO:tasks.workunit.client.1.smithi066.stdout:4/939: dwrite d1/d18/f13d [0,4194304] 0 2024-02-28T16:31:31.439 INFO:tasks.workunit.client.1.smithi066.stdout:8/803: read d0/db/f80 [430180,14067] 0 2024-02-28T16:31:31.440 INFO:tasks.workunit.client.1.smithi066.stdout:3/899: symlink db/dc/d11/l145 0 2024-02-28T16:31:31.443 INFO:tasks.workunit.client.1.smithi066.stdout:7/737: write dc/d29/d92/da6/dd8/f19 [105994,59571] 0 2024-02-28T16:31:31.446 INFO:tasks.workunit.client.1.smithi066.stdout:0/872: mkdir d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/de4/d120 0 2024-02-28T16:31:31.447 INFO:tasks.workunit.client.1.smithi066.stdout:0/873: write d2/d85/d8d/feb [4741016,22529] 0 2024-02-28T16:31:31.448 INFO:tasks.workunit.client.1.smithi066.stdout:6/746: fdatasync d2/da/d19/d9c/d3d/f7f 0 2024-02-28T16:31:31.452 INFO:tasks.workunit.client.1.smithi066.stdout:8/804: write d0/f7c [2457155,9659] 0 2024-02-28T16:31:31.456 INFO:tasks.workunit.client.1.smithi066.stdout:3/900: rename db/dc/d11/d1f/f10c to db/d2a/d44/d48/d52/d77/f146 0 2024-02-28T16:31:31.456 INFO:tasks.workunit.client.1.smithi066.stdout:8/805: dread - d0/d6/dc/f101 zero size 2024-02-28T16:31:31.457 INFO:tasks.workunit.client.1.smithi066.stdout:2/807: dread d0/d12/d1a/d23/d5d/fbd [0,4194304] 0 2024-02-28T16:31:31.458 INFO:tasks.workunit.client.1.smithi066.stdout:4/940: creat d1/d18/d41/dba/d116/d123/f142 x:0 0 0 2024-02-28T16:31:31.458 INFO:tasks.workunit.client.1.smithi066.stdout:6/747: read d2/d2c/da1/d5d/d99/f5e [863561,56905] 0 2024-02-28T16:31:31.460 INFO:tasks.workunit.client.1.smithi066.stdout:0/874: creat d2/d27/d41/d76/d93/df3/f121 x:0 0 0 2024-02-28T16:31:31.461 INFO:tasks.workunit.client.1.smithi066.stdout:4/941: chown d1/l17 1120 1 2024-02-28T16:31:31.462 INFO:tasks.workunit.client.1.smithi066.stdout:5/709: link d3/d4/dc/dd/d41/d7a/dac/f73 d3/d4/dc/dd/d2e/d8f/fe2 0 2024-02-28T16:31:31.462 INFO:tasks.workunit.client.1.smithi066.stdout:6/748: chown f1 1142 1 2024-02-28T16:31:31.467 INFO:tasks.workunit.client.1.smithi066.stdout:1/790: sync 2024-02-28T16:31:31.467 INFO:tasks.workunit.client.1.smithi066.stdout:9/618: sync 2024-02-28T16:31:31.467 INFO:tasks.workunit.client.1.smithi066.stdout:7/738: rename dc/d29/d92/da6/dd8/d9d to dc/d29/deb 0 2024-02-28T16:31:31.469 INFO:tasks.workunit.client.1.smithi066.stdout:7/739: fsync dc/de/fca 0 2024-02-28T16:31:31.469 INFO:tasks.workunit.client.1.smithi066.stdout:3/901: mkdir db/d13/d14/d2d/de3/d147 0 2024-02-28T16:31:31.477 INFO:tasks.workunit.client.1.smithi066.stdout:3/902: read db/d13/d14/d84/f112 [3699336,27783] 0 2024-02-28T16:31:31.481 INFO:tasks.workunit.client.1.smithi066.stdout:5/710: creat d3/d1b/d20/dc1/fe3 x:0 0 0 2024-02-28T16:31:31.491 INFO:tasks.workunit.client.1.smithi066.stdout:2/808: unlink d0/d12/f5c 0 2024-02-28T16:31:31.494 INFO:tasks.workunit.client.1.smithi066.stdout:1/791: rename d2/dd0/ddf to d2/d27/dc7/d54/de7/d107 0 2024-02-28T16:31:31.496 INFO:tasks.workunit.client.1.smithi066.stdout:7/740: mkdir dc/d29/d92/da6/dd8/d1e/da1/db7/dec 0 2024-02-28T16:31:31.497 INFO:tasks.workunit.client.1.smithi066.stdout:1/792: fdatasync d2/d27/dc7/f49 0 2024-02-28T16:31:31.499 INFO:tasks.workunit.client.1.smithi066.stdout:6/749: truncate d2/da/d19/d75/d88/fce 945468 0 2024-02-28T16:31:31.500 INFO:tasks.workunit.client.1.smithi066.stdout:9/619: unlink da/d3a/d58/d97/ca1 0 2024-02-28T16:31:31.501 INFO:tasks.workunit.client.1.smithi066.stdout:2/809: write d0/d12/d1a/d39/d7c/d87/dc4/ff1 [325603,1086] 0 2024-02-28T16:31:31.503 INFO:tasks.workunit.client.1.smithi066.stdout:4/942: dread d1/d18/d7e/fab [0,4194304] 0 2024-02-28T16:31:31.504 INFO:tasks.workunit.client.1.smithi066.stdout:7/741: dread - dc/d29/d92/da6/dd8/d1e/da1/fc1 zero size 2024-02-28T16:31:31.508 INFO:tasks.workunit.client.1.smithi066.stdout:5/711: symlink d3/d1b/le4 0 2024-02-28T16:31:31.510 INFO:tasks.workunit.client.1.smithi066.stdout:8/806: write d0/db/d12/d1d/d25/d65/f9b [235984,16715] 0 2024-02-28T16:31:31.511 INFO:tasks.workunit.client.1.smithi066.stdout:9/620: write da/d22/d46/d56/d8f/d2d/f57 [2980463,66437] 0 2024-02-28T16:31:31.518 INFO:tasks.workunit.client.1.smithi066.stdout:8/807: readlink d0/d6/dc/d16/d4c/d42/l57 0 2024-02-28T16:31:31.519 INFO:tasks.workunit.client.1.smithi066.stdout:1/793: creat d2/d18/d7a/dbc/f108 x:0 0 0 2024-02-28T16:31:31.520 INFO:tasks.workunit.client.1.smithi066.stdout:7/742: symlink dc/d29/led 0 2024-02-28T16:31:31.528 INFO:tasks.workunit.client.1.smithi066.stdout:4/943: mkdir d1/d23/d25/d143 0 2024-02-28T16:31:31.528 INFO:tasks.workunit.client.1.smithi066.stdout:9/621: read da/fc [3223802,91454] 0 2024-02-28T16:31:31.531 INFO:tasks.workunit.client.1.smithi066.stdout:5/712: rename d3/d4/dc/dd/d35/d92/fad to d3/d4/dc/dd/d35/d92/fe5 0 2024-02-28T16:31:31.535 INFO:tasks.workunit.client.1.smithi066.stdout:0/875: dwrite d2/d27/d41/d76/d93/f6f [0,4194304] 0 2024-02-28T16:31:31.536 INFO:tasks.workunit.client.1.smithi066.stdout:1/794: mknod d2/d18/da5/c109 0 2024-02-28T16:31:31.538 INFO:tasks.workunit.client.1.smithi066.stdout:6/750: dread d2/da/d19/d75/d88/fce [0,4194304] 0 2024-02-28T16:31:31.538 INFO:tasks.workunit.client.1.smithi066.stdout:4/944: chown d1/d18/d55/dc4/feb 0 1 2024-02-28T16:31:31.547 INFO:tasks.workunit.client.1.smithi066.stdout:7/743: mknod dc/d29/d92/cee 0 2024-02-28T16:31:31.549 INFO:tasks.workunit.client.1.smithi066.stdout:4/945: read - d1/d18/d41/dba/d116/d123/f142 zero size 2024-02-28T16:31:31.549 INFO:tasks.workunit.client.1.smithi066.stdout:5/713: rename d3/d4/dc/d56/dd6 to d3/d4/dc/dd/de6 0 2024-02-28T16:31:31.550 INFO:tasks.workunit.client.1.smithi066.stdout:9/622: truncate da/d3a/f41 2443636 0 2024-02-28T16:31:31.556 INFO:tasks.workunit.client.1.smithi066.stdout:6/751: chown d2/da/d19/cd8 3 1 2024-02-28T16:31:31.557 INFO:tasks.workunit.client.1.smithi066.stdout:9/623: dread - da/d22/d46/d56/fb5 zero size 2024-02-28T16:31:31.557 INFO:tasks.workunit.client.1.smithi066.stdout:3/903: dwrite db/dc/f2b [0,4194304] 0 2024-02-28T16:31:31.559 INFO:tasks.workunit.client.1.smithi066.stdout:0/876: write d2/d27/d41/d76/d93/df0/f10d [1617768,16726] 0 2024-02-28T16:31:31.562 INFO:tasks.workunit.client.1.smithi066.stdout:5/714: fdatasync d3/d4/dc/d56/f75 0 2024-02-28T16:31:31.567 INFO:tasks.workunit.client.1.smithi066.stdout:2/810: dwrite d0/d12/f62 [0,4194304] 0 2024-02-28T16:31:31.567 INFO:tasks.workunit.client.1.smithi066.stdout:3/904: fdatasync db/dc/fca 0 2024-02-28T16:31:31.572 INFO:tasks.workunit.client.1.smithi066.stdout:4/946: read d1/d23/d25/d3e/d40/d9c/fb9 [3744296,94042] 0 2024-02-28T16:31:31.576 INFO:tasks.workunit.client.1.smithi066.stdout:5/715: mknod d3/d4/dc/d56/d93/ce7 0 2024-02-28T16:31:31.586 INFO:tasks.workunit.client.1.smithi066.stdout:2/811: mkdir d0/d12/d1a/d39/d7c/d87/d104 0 2024-02-28T16:31:31.591 INFO:tasks.workunit.client.1.smithi066.stdout:9/624: creat da/d22/d46/d56/d8f/d28/d42/fcc x:0 0 0 2024-02-28T16:31:31.591 INFO:tasks.workunit.client.1.smithi066.stdout:8/808: dwrite d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db3/fce [0,4194304] 0 2024-02-28T16:31:31.594 INFO:tasks.workunit.client.1.smithi066.stdout:9/625: stat da/df/f29 0 2024-02-28T16:31:31.600 INFO:tasks.workunit.client.1.smithi066.stdout:3/905: dwrite db/d2a/d44/d48/d52/dcb/fd7 [0,4194304] 0 2024-02-28T16:31:31.607 INFO:tasks.workunit.client.1.smithi066.stdout:4/947: rename d1/d23/d25/dd9/d137 to d1/d18/da8/dcb/d13a/d144 0 2024-02-28T16:31:31.609 INFO:tasks.workunit.client.1.smithi066.stdout:6/752: dwrite d2/da/f1b [0,4194304] 0 2024-02-28T16:31:31.614 INFO:tasks.workunit.client.1.smithi066.stdout:6/753: write d2/d26/d32/d46/f7c [5216600,33906] 0 2024-02-28T16:31:31.618 INFO:tasks.workunit.client.1.smithi066.stdout:0/877: rmdir d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/de4/d120 0 2024-02-28T16:31:31.618 INFO:tasks.workunit.client.1.smithi066.stdout:3/906: rename db/d13/d14/ca7 to db/d13/d14/d84/c148 0 2024-02-28T16:31:31.625 INFO:tasks.workunit.client.1.smithi066.stdout:4/948: rmdir d1/d23/d25/d3e/d40 39 2024-02-28T16:31:31.629 INFO:tasks.workunit.client.1.smithi066.stdout:4/949: dread - d1/d18/d41/f12c zero size 2024-02-28T16:31:31.635 INFO:tasks.workunit.client.1.smithi066.stdout:6/754: creat d2/d2c/d72/f109 x:0 0 0 2024-02-28T16:31:31.637 INFO:tasks.workunit.client.1.smithi066.stdout:8/809: creat d0/db/d12/d1d/d25/f10e x:0 0 0 2024-02-28T16:31:31.640 INFO:tasks.workunit.client.1.smithi066.stdout:2/812: dwrite d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/feb [0,4194304] 0 2024-02-28T16:31:31.645 INFO:tasks.workunit.client.1.smithi066.stdout:0/878: rmdir d2/d27/d41/d76/d93/d10/d23/d6d/d113 39 2024-02-28T16:31:31.645 INFO:tasks.workunit.client.1.smithi066.stdout:2/813: write d0/d12/d2e/d7a/fc5 [178589,87416] 0 2024-02-28T16:31:31.646 INFO:tasks.workunit.client.1.smithi066.stdout:3/907: mknod db/d13/d1a/d25/dd1/c149 0 2024-02-28T16:31:31.658 INFO:tasks.workunit.client.1.smithi066.stdout:9/626: write da/d22/d46/d56/d8f/d28/f4d [1419386,116459] 0 2024-02-28T16:31:31.659 INFO:tasks.workunit.client.1.smithi066.stdout:5/716: dread d3/d4/dc/dd/d41/d7a/dac/fb4 [0,4194304] 0 2024-02-28T16:31:31.661 INFO:tasks.workunit.client.1.smithi066.stdout:6/755: creat d2/d2c/dba/f10a x:0 0 0 2024-02-28T16:31:31.662 INFO:tasks.workunit.client.1.smithi066.stdout:8/810: mknod d0/d6/dc/d16/d4c/d42/da8/c10f 0 2024-02-28T16:31:31.666 INFO:tasks.workunit.client.1.smithi066.stdout:0/879: read d2/d27/d41/d76/f9e [2596086,54271] 0 2024-02-28T16:31:31.668 INFO:tasks.workunit.client.1.smithi066.stdout:8/811: write d0/db/d12/d1d/d25/d4f/f82 [2342268,98032] 0 2024-02-28T16:31:31.679 INFO:tasks.workunit.client.1.smithi066.stdout:9/627: dwrite da/d22/d46/d56/d8f/d28/d42/d68/f7a [0,4194304] 0 2024-02-28T16:31:31.689 INFO:tasks.workunit.client.1.smithi066.stdout:2/814: dread d0/d12/d1a/d23/d29/d21/d2f/fa2 [0,4194304] 0 2024-02-28T16:31:31.701 INFO:tasks.workunit.client.1.smithi066.stdout:6/756: creat d2/da/d19/d75/dbd/dc6/dc7/f10b x:0 0 0 2024-02-28T16:31:31.703 INFO:tasks.workunit.client.1.smithi066.stdout:0/880: fsync d2/d27/d41/d76/f9e 0 2024-02-28T16:31:31.703 INFO:tasks.workunit.client.1.smithi066.stdout:8/812: mknod d0/db/d15/d95/c110 0 2024-02-28T16:31:31.710 INFO:tasks.workunit.client.1.smithi066.stdout:0/881: write d2/d27/d41/d76/d93/d10/d23/d6d/f79 [3029430,116228] 0 2024-02-28T16:31:31.710 INFO:tasks.workunit.client.1.smithi066.stdout:2/815: unlink d0/d12/d1a/f2b 0 2024-02-28T16:31:31.711 INFO:tasks.workunit.client.1.smithi066.stdout:8/813: readlink d0/d6/dc/d16/d4c/d6c/lcf 0 2024-02-28T16:31:31.714 INFO:tasks.workunit.client.1.smithi066.stdout:4/950: dread d1/d23/d25/d3e/d40/d9c/d11c/d11d/fa6 [0,4194304] 0 2024-02-28T16:31:31.720 INFO:tasks.workunit.client.1.smithi066.stdout:6/757: rename d2/d26/f47 to d2/d2c/d6e/f10c 0 2024-02-28T16:31:31.721 INFO:tasks.workunit.client.1.smithi066.stdout:3/908: link db/d13/d1a/d25/l73 db/d2a/d44/d48/d7d/l14a 0 2024-02-28T16:31:31.721 INFO:tasks.workunit.client.1.smithi066.stdout:5/717: creat d3/d1b/fe8 x:0 0 0 2024-02-28T16:31:31.725 INFO:tasks.workunit.client.1.smithi066.stdout:8/814: symlink d0/d6/dc/d16/d4c/d6c/daf/df3/l111 0 2024-02-28T16:31:31.725 INFO:tasks.workunit.client.1.smithi066.stdout:5/718: readlink d3/d4/dc/dd/d41/d7a/dac/d60/lb7 0 2024-02-28T16:31:31.728 INFO:tasks.workunit.client.1.smithi066.stdout:5/719: chown d3/c7b 699944 1 2024-02-28T16:31:31.729 INFO:tasks.workunit.client.1.smithi066.stdout:8/815: readlink d0/d6/dc/d16/d4c/d6c/lcf 0 2024-02-28T16:31:31.731 INFO:tasks.workunit.client.1.smithi066.stdout:6/758: unlink d2/da/d19/d75/dbd/dc6/dc7/lcd 0 2024-02-28T16:31:31.732 INFO:tasks.workunit.client.1.smithi066.stdout:8/816: dread - d0/db/d12/d1d/d25/de2/fca zero size 2024-02-28T16:31:31.733 INFO:tasks.workunit.client.1.smithi066.stdout:2/816: dwrite d0/d12/d1a/d39/d7c/fdd [0,4194304] 0 2024-02-28T16:31:31.734 INFO:tasks.workunit.client.1.smithi066.stdout:0/882: unlink d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/fb8 0 2024-02-28T16:31:31.736 INFO:tasks.workunit.client.1.smithi066.stdout:5/720: mkdir d3/d1b/d20/de9 0 2024-02-28T16:31:31.737 INFO:tasks.workunit.client.1.smithi066.stdout:6/759: stat d2/d26/f5f 0 2024-02-28T16:31:31.739 INFO:tasks.workunit.client.1.smithi066.stdout:8/817: symlink d0/d6/dc/d16/d1a/d63/d58/dac/l112 0 2024-02-28T16:31:31.741 INFO:tasks.workunit.client.1.smithi066.stdout:3/909: getdents db/dc/d35/d3d/d76 0 2024-02-28T16:31:31.749 INFO:tasks.workunit.client.1.smithi066.stdout:0/883: unlink d2/d27/d41/d76/d93/db2/db5/d73/fa8 0 2024-02-28T16:31:31.749 INFO:tasks.workunit.client.1.smithi066.stdout:2/817: read d0/d12/d2e/d3b/f61 [125291,128842] 0 2024-02-28T16:31:31.749 INFO:tasks.workunit.client.1.smithi066.stdout:5/721: mknod d3/d4/dc/dd/d2e/d8f/cea 0 2024-02-28T16:31:31.751 INFO:tasks.workunit.client.1.smithi066.stdout:6/760: creat d2/da/d19/d75/dbd/dc6/f10d x:0 0 0 2024-02-28T16:31:31.752 INFO:tasks.workunit.client.1.smithi066.stdout:0/884: readlink d2/d27/d41/d76/d93/df3/lcc 0 2024-02-28T16:31:31.759 INFO:tasks.workunit.client.1.smithi066.stdout:5/722: unlink d3/f98 0 2024-02-28T16:31:31.762 INFO:tasks.workunit.client.1.smithi066.stdout:3/910: dwrite db/d13/d14/ff3 [0,4194304] 0 2024-02-28T16:31:31.763 INFO:tasks.workunit.client.1.smithi066.stdout:2/818: rename d0/fcf to d0/d12/d3e/d9e/dce/f105 0 2024-02-28T16:31:31.764 INFO:tasks.workunit.client.1.smithi066.stdout:8/818: creat d0/d6/dc/d16/d4c/d6c/d107/f113 x:0 0 0 2024-02-28T16:31:31.768 INFO:tasks.workunit.client.1.smithi066.stdout:0/885: mkdir d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/d122 0 2024-02-28T16:31:31.769 INFO:tasks.workunit.client.1.smithi066.stdout:6/761: dread d2/f43 [0,4194304] 0 2024-02-28T16:31:31.772 INFO:tasks.workunit.client.1.smithi066.stdout:2/819: mkdir d0/d12/d2e/d7a/d106 0 2024-02-28T16:31:31.781 INFO:tasks.workunit.client.1.smithi066.stdout:0/886: mkdir d2/d27/d41/df4/d123 0 2024-02-28T16:31:31.785 INFO:tasks.workunit.client.1.smithi066.stdout:8/819: dread d0/f23 [0,4194304] 0 2024-02-28T16:31:31.786 INFO:tasks.workunit.client.1.smithi066.stdout:3/911: dread db/dc/db3/dc9/de4/d116/f128 [0,4194304] 0 2024-02-28T16:31:31.788 INFO:tasks.workunit.client.1.smithi066.stdout:9/628: dwrite da/d4b/d53/dba/f5c [0,4194304] 0 2024-02-28T16:31:31.790 INFO:tasks.workunit.client.1.smithi066.stdout:0/887: dread - d2/d27/d41/d76/d93/d10/d23/d6d/ffb zero size 2024-02-28T16:31:31.794 INFO:tasks.workunit.client.1.smithi066.stdout:3/912: creat db/dc/db3/f14b x:0 0 0 2024-02-28T16:31:31.797 INFO:tasks.workunit.client.1.smithi066.stdout:8/820: mknod d0/d6/dc/d16/c114 0 2024-02-28T16:31:31.802 INFO:tasks.workunit.client.1.smithi066.stdout:9/629: truncate da/d22/d46/d56/d8f/f82 760033 0 2024-02-28T16:31:31.810 INFO:tasks.workunit.client.1.smithi066.stdout:9/630: mknod da/d3a/d58/d97/ccd 0 2024-02-28T16:31:31.817 INFO:tasks.workunit.client.1.smithi066.stdout:3/913: link db/d2a/d44/d48/d5d/f91 db/d13/d1a/d25/dd1/f14c 0 2024-02-28T16:31:31.819 INFO:tasks.workunit.client.1.smithi066.stdout:6/762: fsync d2/da/d19/d75/dbd/dc6/dc7/f10b 0 2024-02-28T16:31:31.819 INFO:tasks.workunit.client.1.smithi066.stdout:9/631: mknod da/d3a/d58/cce 0 2024-02-28T16:31:31.821 INFO:tasks.workunit.client.1.smithi066.stdout:8/821: creat d0/d6/dc/d16/d1a/d63/f115 x:0 0 0 2024-02-28T16:31:31.822 INFO:tasks.workunit.client.1.smithi066.stdout:8/822: fdatasync d0/d6/f1f 0 2024-02-28T16:31:31.822 INFO:tasks.workunit.client.1.smithi066.stdout:3/914: fdatasync db/d13/d14/dad/fc7 0 2024-02-28T16:31:31.825 INFO:tasks.workunit.client.1.smithi066.stdout:6/763: chown d2/d26/d32/d46/f103 6 1 2024-02-28T16:31:31.826 INFO:tasks.workunit.client.1.smithi066.stdout:8/823: dread - d0/d6/dc/d16/d4c/d6c/d6f/ff6 zero size 2024-02-28T16:31:31.827 INFO:tasks.workunit.client.1.smithi066.stdout:6/764: read - d2/da/d19/d75/da6/ff9 zero size 2024-02-28T16:31:31.837 INFO:tasks.workunit.client.1.smithi066.stdout:8/824: truncate d0/db/d12/d1d/d25/d65/ff8 658426 0 2024-02-28T16:31:31.840 INFO:tasks.workunit.client.1.smithi066.stdout:0/888: dwrite d2/f96 [0,4194304] 0 2024-02-28T16:31:31.841 INFO:tasks.workunit.client.1.smithi066.stdout:6/765: write d2/d26/f4e [5584428,57706] 0 2024-02-28T16:31:31.844 INFO:tasks.workunit.client.1.smithi066.stdout:6/766: chown d2/d26/d32/db9 2837719 1 2024-02-28T16:31:31.845 INFO:tasks.workunit.client.1.smithi066.stdout:8/825: chown d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/dad/cf4 1606393 1 2024-02-28T16:31:31.846 INFO:tasks.workunit.client.1.smithi066.stdout:3/915: dwrite db/d2a/d44/d48/d7d/f130 [0,4194304] 0 2024-02-28T16:31:31.852 INFO:tasks.workunit.client.1.smithi066.stdout:8/826: chown d0/db/d12/d1d/d25/d65/ff8 38832577 1 2024-02-28T16:31:31.857 INFO:tasks.workunit.client.1.smithi066.stdout:0/889: rename d2/l31 to d2/d27/d41/d76/d93/d10/d23/db6/de0/dfd/l124 0 2024-02-28T16:31:31.858 INFO:tasks.workunit.client.1.smithi066.stdout:8/827: creat d0/db/d12/d8a/f116 x:0 0 0 2024-02-28T16:31:31.863 INFO:tasks.workunit.client.1.smithi066.stdout:6/767: rename d2/d2c/da1/d5d/d99/f59 to d2/da/d19/d75/dbd/dc6/dc7/f10e 0 2024-02-28T16:31:31.864 INFO:tasks.workunit.client.1.smithi066.stdout:8/828: chown d0/d6/dc/d16/d1a/d63/d58/c10d 37 1 2024-02-28T16:31:31.871 INFO:tasks.workunit.client.1.smithi066.stdout:8/829: mknod d0/d6/dc/d16/d4c/d42/da8/c117 0 2024-02-28T16:31:31.873 INFO:tasks.workunit.client.1.smithi066.stdout:0/890: symlink d2/d27/d41/d76/d93/d10/d23/d6d/d113/dfc/l125 0 2024-02-28T16:31:31.873 INFO:tasks.workunit.client.1.smithi066.stdout:6/768: dread d2/d26/f31 [0,4194304] 0 2024-02-28T16:31:31.876 INFO:tasks.workunit.client.1.smithi066.stdout:6/769: chown d2/da/d19/cf0 646716 1 2024-02-28T16:31:31.877 INFO:tasks.workunit.client.1.smithi066.stdout:3/916: dwrite db/dc/d11/d1f/d78/ffb [0,4194304] 0 2024-02-28T16:31:31.878 INFO:tasks.workunit.client.1.smithi066.stdout:6/770: readlink d2/da/d19/d9c/ld4 0 2024-02-28T16:31:31.886 INFO:tasks.workunit.client.1.smithi066.stdout:8/830: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2/d106/d118 0 2024-02-28T16:31:31.886 INFO:tasks.workunit.client.1.smithi066.stdout:6/771: creat d2/d2c/d6e/dd0/f10f x:0 0 0 2024-02-28T16:31:31.894 INFO:tasks.workunit.client.1.smithi066.stdout:8/831: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/f119 x:0 0 0 2024-02-28T16:31:31.897 INFO:tasks.workunit.client.1.smithi066.stdout:6/772: write d2/da/fe [1490496,21296] 0 2024-02-28T16:31:31.897 INFO:tasks.workunit.client.1.smithi066.stdout:0/891: dwrite d2/d27/d41/d76/d93/db2/db5/d51/f70 [0,4194304] 0 2024-02-28T16:31:31.897 INFO:tasks.workunit.client.1.smithi066.stdout:8/832: chown d0/db/d12/f4b 0 1 2024-02-28T16:31:31.909 INFO:tasks.workunit.client.1.smithi066.stdout:8/833: stat d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/c98 0 2024-02-28T16:31:31.910 INFO:tasks.workunit.client.1.smithi066.stdout:0/892: symlink d2/d27/d41/d76/d93/db2/l126 0 2024-02-28T16:31:31.914 INFO:tasks.workunit.client.1.smithi066.stdout:6/773: rename d2/da/d19/d9c/f3f to d2/d26/dbe/df7/f110 0 2024-02-28T16:31:31.915 INFO:tasks.workunit.client.1.smithi066.stdout:3/917: dwrite db/dc/d35/d3d/d76/d89/dcc/fb2 [0,4194304] 0 2024-02-28T16:31:31.922 INFO:tasks.workunit.client.1.smithi066.stdout:3/918: fsync db/d2a/dd4/ff9 0 2024-02-28T16:31:31.924 INFO:tasks.workunit.client.1.smithi066.stdout:3/919: chown db/d13/d14/d2d 1611 1 2024-02-28T16:31:31.927 INFO:tasks.workunit.client.1.smithi066.stdout:8/834: link d0/d6/dc/d16/d1a/ffa d0/d6/dc/d16/d4c/d6c/daf/f11a 0 2024-02-28T16:31:31.931 INFO:tasks.workunit.client.1.smithi066.stdout:3/920: mknod db/dc/d11/c14d 0 2024-02-28T16:31:31.932 INFO:tasks.workunit.client.1.smithi066.stdout:6/774: mknod d2/da/dcc/dd6/d102/c111 0 2024-02-28T16:31:31.937 INFO:tasks.workunit.client.1.smithi066.stdout:8/835: chown d0/db/d12/d38/c5c 15118 1 2024-02-28T16:31:31.946 INFO:tasks.workunit.client.1.smithi066.stdout:8/836: mkdir d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/dad/d11b 0 2024-02-28T16:31:31.949 INFO:tasks.workunit.client.1.smithi066.stdout:0/893: dwrite d2/d27/d41/d76/d93/df0/f10d [4194304,4194304] 0 2024-02-28T16:31:31.955 INFO:tasks.workunit.client.1.smithi066.stdout:6/775: write d2/da/d19/d75/d88/f8d [488724,4698] 0 2024-02-28T16:31:31.956 INFO:tasks.workunit.client.1.smithi066.stdout:8/837: fsync d0/d6/dc/d16/d4c/d6c/d6f/d92/fb7 0 2024-02-28T16:31:31.957 INFO:tasks.workunit.client.1.smithi066.stdout:0/894: creat d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f127 x:0 0 0 2024-02-28T16:31:31.960 INFO:tasks.workunit.client.1.smithi066.stdout:6/776: write d2/da/f76 [2131947,16999] 0 2024-02-28T16:31:31.963 INFO:tasks.workunit.client.1.smithi066.stdout:0/895: write d2/d27/d41/d76/d93/df0/f10d [1627699,103843] 0 2024-02-28T16:31:31.966 INFO:tasks.workunit.client.1.smithi066.stdout:8/838: chown d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/da2 245964 1 2024-02-28T16:31:31.967 INFO:tasks.workunit.client.1.smithi066.stdout:3/921: dwrite db/d13/d14/dda/fe6 [0,4194304] 0 2024-02-28T16:31:31.971 INFO:tasks.workunit.client.1.smithi066.stdout:0/896: rmdir d2/d85/d8d 39 2024-02-28T16:31:31.972 INFO:tasks.workunit.client.1.smithi066.stdout:6/777: rename d2/d2c/d6e/l71 to d2/da/dcb/l112 0 2024-02-28T16:31:31.975 INFO:tasks.workunit.client.1.smithi066.stdout:3/922: mkdir db/d2a/d44/d48/d52/d14e 0 2024-02-28T16:31:31.977 INFO:tasks.workunit.client.1.smithi066.stdout:0/897: write d2/d27/d41/d76/d93/db2/db5/d73/fe8 [1211106,83522] 0 2024-02-28T16:31:31.977 INFO:tasks.workunit.client.1.smithi066.stdout:8/839: creat d0/d6/dc/d16/d4c/d6c/d6f/d92/d96/d9d/db1/f11c x:0 0 0 2024-02-28T16:31:31.980 INFO:tasks.workunit.client.1.smithi066.stdout:3/923: rmdir db/d13/d14/dad 39 2024-02-28T16:31:31.984 INFO:tasks.workunit.client.1.smithi066.stdout:6/778: rename d2/d2c/da1/c7a to d2/d2c/da1/d5d/c113 0 2024-02-28T16:31:31.986 INFO:tasks.workunit.client.1.smithi066.stdout:0/898: mknod d2/d27/d41/d76/d93/d10/d23/db6/c128 0 2024-02-28T16:31:31.990 INFO:tasks.workunit.client.1.smithi066.stdout:3/924: symlink db/d13/d14/dda/d102/l14f 0 2024-02-28T16:31:31.997 INFO:tasks.workunit.client.1.smithi066.stdout:8/840: dwrite d0/db/ff1 [0,4194304] 0 2024-02-28T16:31:31.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:31 smithi066 ceph-mon[36236]: pgmap v59: 65 pgs: 65 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 121 MiB/s rd, 311 MiB/s wr, 656 op/s 2024-02-28T16:31:32.004 INFO:tasks.workunit.client.1.smithi066.stdout:0/899: truncate d2/d27/d41/d76/d93/d10/f2c 264112 0 2024-02-28T16:31:32.004 INFO:tasks.workunit.client.1.smithi066.stdout:8/841: creat d0/d6/dc/f11d x:0 0 0 2024-02-28T16:31:32.005 INFO:tasks.workunit.client.1.smithi066.stdout:6/779: dwrite d2/d26/de0/ffc [0,4194304] 0 2024-02-28T16:31:32.008 INFO:tasks.workunit.client.1.smithi066.stdout:0/900: mknod d2/d27/c129 0 2024-02-28T16:31:32.009 INFO:tasks.workunit.client.1.smithi066.stdout:6/780: chown d2/da/d19/d75/d88/fce 31504086 1 2024-02-28T16:31:32.010 INFO:tasks.workunit.client.1.smithi066.stdout:8/842: truncate d0/db/d12/d1d/d25/de2/fca 619110 0 2024-02-28T16:31:32.013 INFO:tasks.workunit.client.1.smithi066.stdout:6/781: mkdir d2/da/d19/d9c/d114 0 2024-02-28T16:31:32.016 INFO:tasks.workunit.client.1.smithi066.stdout:3/925: dread db/ff [0,4194304] 0 2024-02-28T16:31:32.016 INFO:tasks.workunit.client.1.smithi066.stdout:6/782: read d2/d26/de0/ffc [3723082,92319] 0 2024-02-28T16:31:32.020 INFO:tasks.workunit.client.1.smithi066.stdout:0/901: truncate d2/d27/d41/d76/d93/db2/db5/d51/f5a 2694527 0 2024-02-28T16:31:32.021 INFO:tasks.workunit.client.1.smithi066.stdout:3/926: creat db/dc/d35/d3d/d76/f150 x:0 0 0 2024-02-28T16:31:32.029 INFO:tasks.workunit.client.1.smithi066.stdout:3/927: unlink db/d13/d1a/d33/f39 0 2024-02-28T16:31:32.035 INFO:tasks.workunit.client.1.smithi066.stdout:8/843: dwrite d0/d6/dc/d16/d4c/d6c/d6f/da7/fcc [0,4194304] 0 2024-02-28T16:31:32.041 INFO:tasks.workunit.client.1.smithi066.stdout:8/844: truncate d0/db/d15/fd6 449395 0 2024-02-28T16:31:32.043 INFO:tasks.workunit.client.1.smithi066.stdout:0/902: dwrite d2/d27/d41/d76/d93/d10/f47 [4194304,4194304] 0 2024-02-28T16:31:32.046 INFO:tasks.workunit.client.1.smithi066.stdout:8/845: truncate d0/d6/dc/d16/d1a/ffa 4328929 0 2024-02-28T16:31:32.053 INFO:tasks.workunit.client.1.smithi066.stdout:8/846: readlink d0/db/d12/d1d/d25/d72/ld8 0 2024-02-28T16:31:32.057 INFO:tasks.workunit.client.1.smithi066.stdout:6/783: dread d2/d2c/d72/fe1 [0,4194304] 0 2024-02-28T16:31:32.058 INFO:tasks.workunit.client.1.smithi066.stdout:3/928: dwrite db/d13/d1a/d25/dd1/f14c [0,4194304] 0 2024-02-28T16:31:32.062 INFO:tasks.workunit.client.1.smithi066.stdout:8/847: fsync d0/d6/dc/d16/d4c/d6c/d6f/da7/fc5 0 2024-02-28T16:31:32.065 INFO:tasks.workunit.client.1.smithi066.stdout:0/903: write d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f72 [105731,4887] 0 2024-02-28T16:31:32.070 INFO:tasks.workunit.client.1.smithi066.stdout:3/929: rmdir db/dc/d11 39 2024-02-28T16:31:32.074 INFO:tasks.workunit.client.1.smithi066.stdout:0/904: symlink d2/d27/d41/d76/d93/df0/l12a 0 2024-02-28T16:31:32.076 INFO:tasks.workunit.client.1.smithi066.stdout:3/930: unlink l6 0 2024-02-28T16:31:32.083 INFO:tasks.workunit.client.1.smithi066.stdout:3/931: symlink db/dc/d11/daa/l151 0 2024-02-28T16:31:32.084 INFO:tasks.workunit.client.1.smithi066.stdout:6/784: truncate d2/d26/d32/d46/f7c 1001457 0 2024-02-28T16:31:32.087 INFO:tasks.workunit.client.1.smithi066.stdout:6/785: symlink d2/da/dcb/l115 0 2024-02-28T16:31:32.088 INFO:tasks.workunit.client.1.smithi066.stdout:3/932: creat db/d13/d1a/d25/f152 x:0 0 0 2024-02-28T16:31:32.089 INFO:tasks.workunit.client.1.smithi066.stdout:3/933: stat db/dc/d35/lb8 0 2024-02-28T16:31:32.091 INFO:tasks.workunit.client.1.smithi066.stdout:6/786: getdents d2/d26/d6b 0 2024-02-28T16:31:32.093 INFO:tasks.workunit.client.1.smithi066.stdout:0/905: dwrite d2/d27/d41/d76/d93/f1e [0,4194304] 0 2024-02-28T16:31:32.095 INFO:tasks.workunit.client.1.smithi066.stdout:8/848: dread d0/d6/dc/d16/d4c/d6c/d6f/d92/fc0 [0,4194304] 0 2024-02-28T16:31:32.097 INFO:tasks.workunit.client.1.smithi066.stdout:0/906: dread - d2/d27/d41/d76/d93/db2/db5/d60/ff8 zero size 2024-02-28T16:31:32.101 INFO:tasks.workunit.client.1.smithi066.stdout:0/907: dread - d2/d27/d41/d76/d93/df3/f100 zero size 2024-02-28T16:31:32.102 INFO:tasks.workunit.client.1.smithi066.stdout:6/787: creat d2/da/d19/d75/dbd/dc6/f116 x:0 0 0 2024-02-28T16:31:32.104 INFO:tasks.workunit.client.1.smithi066.stdout:0/908: chown d2/d27/d41/ff5 10633 1 2024-02-28T16:31:32.109 INFO:tasks.workunit.client.1.smithi066.stdout:8/849: rename d0/d6/dc/d16/d4c/d6c/d6f/d92/d96 to d0/db/d12/d1d/d25/d65/dbb/d11e 0 2024-02-28T16:31:32.111 INFO:tasks.workunit.client.1.smithi066.stdout:0/909: mkdir d2/d27/d41/d76/d93/df0/d12b 0 2024-02-28T16:31:32.115 INFO:tasks.workunit.client.1.smithi066.stdout:8/850: symlink d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db3/l11f 0 2024-02-28T16:31:32.116 INFO:tasks.workunit.client.1.smithi066.stdout:3/934: dread db/d13/d14/d2d/f9e [0,4194304] 0 2024-02-28T16:31:32.118 INFO:tasks.workunit.client.1.smithi066.stdout:8/851: read - d0/db/d12/d1d/fe5 zero size 2024-02-28T16:31:32.119 INFO:tasks.workunit.client.1.smithi066.stdout:3/935: dread - db/d2a/d44/d48/d7d/f139 zero size 2024-02-28T16:31:32.119 INFO:tasks.workunit.client.1.smithi066.stdout:0/910: mknod d2/d27/d41/d76/d93/db2/db5/d52/da2/c12c 0 2024-02-28T16:31:32.121 INFO:tasks.workunit.client.1.smithi066.stdout:6/788: write d2/d26/dbe/df7/f110 [194050,33827] 0 2024-02-28T16:31:32.123 INFO:tasks.workunit.client.1.smithi066.stdout:8/852: truncate d0/db/d15/fba 1087508 0 2024-02-28T16:31:32.124 INFO:tasks.workunit.client.1.smithi066.stdout:0/911: chown d2/d27/d41/d76/d93/d10/d23/d6d/d113/fc0 5058 1 2024-02-28T16:31:32.125 INFO:tasks.workunit.client.1.smithi066.stdout:6/789: symlink d2/da/dcc/l117 0 2024-02-28T16:31:32.140 INFO:tasks.workunit.client.1.smithi066.stdout:3/936: dwrite db/d2a/d44/d48/d52/d77/fb5 [0,4194304] 0 2024-02-28T16:31:32.151 INFO:tasks.workunit.client.1.smithi066.stdout:8/853: dwrite d0/d6/dc/d16/d1a/d63/d58/d5a/f64 [0,4194304] 0 2024-02-28T16:31:32.151 INFO:tasks.workunit.client.1.smithi066.stdout:3/937: truncate db/d2a/d44/d48/d52/d77/f74 305728 0 2024-02-28T16:31:32.152 INFO:tasks.workunit.client.1.smithi066.stdout:0/912: dwrite d2/d27/d41/d76/d93/d10/f39 [0,4194304] 0 2024-02-28T16:31:32.153 INFO:tasks.workunit.client.1.smithi066.stdout:8/854: stat d0/db/d15/c26 0 2024-02-28T16:31:32.156 INFO:tasks.workunit.client.1.smithi066.stdout:3/938: mkdir db/dc/d11/d1f/d115/d153 0 2024-02-28T16:31:32.160 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:31 smithi001 ceph-mon[28991]: pgmap v59: 65 pgs: 65 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 121 MiB/s rd, 311 MiB/s wr, 656 op/s 2024-02-28T16:31:32.162 INFO:tasks.workunit.client.1.smithi066.stdout:6/790: dread d2/da/d19/d9c/d3d/fee [0,4194304] 0 2024-02-28T16:31:32.165 INFO:tasks.workunit.client.1.smithi066.stdout:6/791: write d2/d2c/d6e/f10c [2601700,112786] 0 2024-02-28T16:31:32.166 INFO:tasks.workunit.client.1.smithi066.stdout:6/792: dread - d2/d2c/d6e/fe5 zero size 2024-02-28T16:31:32.173 INFO:tasks.workunit.client.1.smithi066.stdout:6/793: rmdir d2/da/dcc/dd6/d102 39 2024-02-28T16:31:32.173 INFO:tasks.workunit.client.1.smithi066.stdout:8/855: write d0/db/d12/d1d/d25/d4f/f86 [1634904,115871] 0 2024-02-28T16:31:32.173 INFO:tasks.workunit.client.1.smithi066.stdout:0/913: truncate d2/d85/f9f 3363230 0 2024-02-28T16:31:32.173 INFO:tasks.workunit.client.1.smithi066.stdout:6/794: rmdir d2/d26/d32 39 2024-02-28T16:31:32.177 INFO:tasks.workunit.client.1.smithi066.stdout:8/856: truncate d0/f23 575594 0 2024-02-28T16:31:32.179 INFO:tasks.workunit.client.1.smithi066.stdout:0/914: creat d2/d27/d41/d76/d93/d10/d23/f12d x:0 0 0 2024-02-28T16:31:32.179 INFO:tasks.workunit.client.1.smithi066.stdout:6/795: symlink d2/d2c/l118 0 2024-02-28T16:31:32.179 INFO:tasks.workunit.client.1.smithi066.stdout:1/795: sync 2024-02-28T16:31:32.182 INFO:tasks.workunit.client.1.smithi066.stdout:6/796: write d2/d2c/d82/f8a [1004647,74578] 0 2024-02-28T16:31:32.183 INFO:tasks.workunit.client.1.smithi066.stdout:8/857: symlink d0/db/d15/l120 0 2024-02-28T16:31:32.191 INFO:tasks.workunit.client.1.smithi066.stdout:8/858: fsync d0/d6/dc/d16/d1a/f88 0 2024-02-28T16:31:32.192 INFO:tasks.workunit.client.1.smithi066.stdout:1/796: getdents d2/d27/d2b 0 2024-02-28T16:31:32.194 INFO:tasks.workunit.client.1.smithi066.stdout:6/797: creat d2/d2c/da1/d5d/d63/d6a/d100/f119 x:0 0 0 2024-02-28T16:31:32.200 INFO:tasks.workunit.client.1.smithi066.stdout:6/798: mknod d2/da/d19/d9c/c11a 0 2024-02-28T16:31:32.208 INFO:tasks.workunit.client.1.smithi066.stdout:8/859: dread d0/db/f80 [0,4194304] 0 2024-02-28T16:31:32.211 INFO:tasks.workunit.client.1.smithi066.stdout:0/915: dwrite d2/d27/d41/d76/d93/d33/d3f/d8e/fe3 [0,4194304] 0 2024-02-28T16:31:32.212 INFO:tasks.workunit.client.1.smithi066.stdout:8/860: dread - d0/d6/dc/ff7 zero size 2024-02-28T16:31:32.213 INFO:tasks.workunit.client.1.smithi066.stdout:3/939: dread db/dc/d35/d3d/f97 [0,4194304] 0 2024-02-28T16:31:32.213 INFO:tasks.workunit.client.1.smithi066.stdout:0/916: dread - d2/d27/d41/d76/dcf/f10c zero size 2024-02-28T16:31:32.215 INFO:tasks.workunit.client.1.smithi066.stdout:8/861: fdatasync d0/d6/dc/d16/f50 0 2024-02-28T16:31:32.219 INFO:tasks.workunit.client.1.smithi066.stdout:8/862: chown d0/d6/dc/d16/d4c/d6c/d107 15508184 1 2024-02-28T16:31:32.220 INFO:tasks.workunit.client.1.smithi066.stdout:7/744: sync 2024-02-28T16:31:32.220 INFO:tasks.workunit.client.1.smithi066.stdout:9/632: sync 2024-02-28T16:31:32.220 INFO:tasks.workunit.client.1.smithi066.stdout:4/951: sync 2024-02-28T16:31:32.220 INFO:tasks.workunit.client.1.smithi066.stdout:2/820: sync 2024-02-28T16:31:32.221 INFO:tasks.workunit.client.1.smithi066.stdout:5/723: sync 2024-02-28T16:31:32.222 INFO:tasks.workunit.client.1.smithi066.stdout:0/917: creat d2/d27/d41/d76/d93/df0/d12b/f12e x:0 0 0 2024-02-28T16:31:32.223 INFO:tasks.workunit.client.1.smithi066.stdout:6/799: truncate d2/da/f1e 3034193 0 2024-02-28T16:31:32.228 INFO:tasks.workunit.client.1.smithi066.stdout:1/797: dread d2/d27/d2b/d34/f44 [0,4194304] 0 2024-02-28T16:31:32.229 INFO:tasks.workunit.client.1.smithi066.stdout:9/633: dread - da/d22/d46/d56/d8f/d28/d42/f9d zero size 2024-02-28T16:31:32.230 INFO:tasks.workunit.client.1.smithi066.stdout:5/724: symlink d3/d4/dc/dd/d35/dc5/leb 0 2024-02-28T16:31:32.233 INFO:tasks.workunit.client.1.smithi066.stdout:6/800: creat d2/d2c/da1/dad/f11b x:0 0 0 2024-02-28T16:31:32.236 INFO:tasks.workunit.client.1.smithi066.stdout:4/952: rename d1/d23/d25/d77/fe9 to d1/d18/d7e/dbb/f145 0 2024-02-28T16:31:32.239 INFO:tasks.workunit.client.1.smithi066.stdout:3/940: dread db/dc/d35/d3d/d76/f4a [0,4194304] 0 2024-02-28T16:31:32.240 INFO:tasks.workunit.client.1.smithi066.stdout:6/801: dread - d2/d26/fc3 zero size 2024-02-28T16:31:32.242 INFO:tasks.workunit.client.1.smithi066.stdout:5/725: fsync d3/d4/dc/dd/d41/f65 0 2024-02-28T16:31:32.247 INFO:tasks.workunit.client.1.smithi066.stdout:1/798: mkdir d2/d18/d7a/dfd/d10a 0 2024-02-28T16:31:32.247 INFO:tasks.workunit.client.1.smithi066.stdout:7/745: mkdir dc/d29/d92/da6/dd8/d37/d47/d88/def 0 2024-02-28T16:31:32.249 INFO:tasks.workunit.client.1.smithi066.stdout:1/799: fsync d2/d27/d2b/d43/d46/fcd 0 2024-02-28T16:31:32.252 INFO:tasks.workunit.client.1.smithi066.stdout:2/821: creat d0/d12/d1a/df8/f107 x:0 0 0 2024-02-28T16:31:32.255 INFO:tasks.workunit.client.1.smithi066.stdout:3/941: mknod db/d2a/d44/d48/d52/dcb/c154 0 2024-02-28T16:31:32.256 INFO:tasks.workunit.client.1.smithi066.stdout:0/918: rename d2/d27/d41/d76/d93/db2/db5/d52/da2/lea to d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/d122/l12f 0 2024-02-28T16:31:32.257 INFO:tasks.workunit.client.1.smithi066.stdout:4/953: write d1/d23/d25/d3e/d40/d9c/d11c/d11d/f5e [3337945,76608] 0 2024-02-28T16:31:32.258 INFO:tasks.workunit.client.1.smithi066.stdout:6/802: symlink d2/da/d19/d75/da6/daa/l11c 0 2024-02-28T16:31:32.261 INFO:tasks.workunit.client.1.smithi066.stdout:4/954: fsync d1/d18/d41/f68 0 2024-02-28T16:31:32.263 INFO:tasks.workunit.client.1.smithi066.stdout:7/746: mknod dc/d29/d92/da6/dd8/d37/d47/d88/dc4/cf0 0 2024-02-28T16:31:32.267 INFO:tasks.workunit.client.1.smithi066.stdout:8/863: dwrite d0/d6/dc/d16/d1a/d63/d58/dac/fb4 [0,4194304] 0 2024-02-28T16:31:32.268 INFO:tasks.workunit.client.1.smithi066.stdout:9/634: dwrite da/d22/d46/d56/d8f/d28/d42/d68/f8c [0,4194304] 0 2024-02-28T16:31:32.268 INFO:tasks.workunit.client.1.smithi066.stdout:5/726: mkdir d3/d1b/db3/dec 0 2024-02-28T16:31:32.271 INFO:tasks.workunit.client.1.smithi066.stdout:0/919: dread - d2/d27/d41/d76/d93/d10/d23/d6d/ffb zero size 2024-02-28T16:31:32.272 INFO:tasks.workunit.client.1.smithi066.stdout:7/747: chown dc/d29/d92/da6/dd8/d37/d47/d88 62800691 1 2024-02-28T16:31:32.276 INFO:tasks.workunit.client.1.smithi066.stdout:2/822: mkdir d0/d12/d2e/d7a/d106/d108 0 2024-02-28T16:31:32.277 INFO:tasks.workunit.client.1.smithi066.stdout:3/942: rename db/d2a/d44/d48/f6c to db/dc/d11/d1f/d115/d13c/f155 0 2024-02-28T16:31:32.278 INFO:tasks.workunit.client.1.smithi066.stdout:8/864: unlink d0/db/d12/f87 0 2024-02-28T16:31:32.283 INFO:tasks.workunit.client.1.smithi066.stdout:7/748: creat dc/d56/d97/ff1 x:0 0 0 2024-02-28T16:31:32.286 INFO:tasks.workunit.client.1.smithi066.stdout:9/635: rmdir da/d22/d46/d56/d8f/d28 39 2024-02-28T16:31:32.287 INFO:tasks.workunit.client.1.smithi066.stdout:2/823: readlink d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/lb8 0 2024-02-28T16:31:32.290 INFO:tasks.workunit.client.1.smithi066.stdout:1/800: getdents d2/d18/d6b 0 2024-02-28T16:31:32.294 INFO:tasks.workunit.client.1.smithi066.stdout:6/803: dwrite d2/d2c/da1/d5d/d63/fc2 [0,4194304] 0 2024-02-28T16:31:32.295 INFO:tasks.workunit.client.1.smithi066.stdout:2/824: readlink d0/d12/d1a/d23/d29/d21/l32 0 2024-02-28T16:31:32.298 INFO:tasks.workunit.client.1.smithi066.stdout:4/955: rename d1/d23/d25/l12d to d1/l146 0 2024-02-28T16:31:32.298 INFO:tasks.workunit.client.1.smithi066.stdout:1/801: write d2/d27/dc7/d48/f66 [2441138,2977] 0 2024-02-28T16:31:32.301 INFO:tasks.workunit.client.1.smithi066.stdout:6/804: write d2/d26/f2d [4460925,48406] 0 2024-02-28T16:31:32.304 INFO:tasks.workunit.client.1.smithi066.stdout:4/956: read d1/df7/f103 [719502,87044] 0 2024-02-28T16:31:32.306 INFO:tasks.workunit.client.1.smithi066.stdout:2/825: unlink d0/d12/d1a/ca9 0 2024-02-28T16:31:32.306 INFO:tasks.workunit.client.1.smithi066.stdout:9/636: truncate da/d22/d46/d56/d8f/d28/fa5 689194 0 2024-02-28T16:31:32.308 INFO:tasks.workunit.client.1.smithi066.stdout:1/802: rmdir d2/d27/dc7/da6 39 2024-02-28T16:31:32.308 INFO:tasks.workunit.client.1.smithi066.stdout:4/957: write d1/dfb/f125 [4487559,42014] 0 2024-02-28T16:31:32.312 INFO:tasks.workunit.client.1.smithi066.stdout:8/865: truncate d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/fc4 1205550 0 2024-02-28T16:31:32.314 INFO:tasks.workunit.client.1.smithi066.stdout:0/920: dread d2/d85/d8d/fed [0,4194304] 0 2024-02-28T16:31:32.319 INFO:tasks.workunit.client.1.smithi066.stdout:6/805: dread - d2/d2c/fbb zero size 2024-02-28T16:31:32.327 INFO:tasks.workunit.client.1.smithi066.stdout:1/803: truncate d2/d27/d2b/d34/fcc 1028572 0 2024-02-28T16:31:32.327 INFO:tasks.workunit.client.1.smithi066.stdout:9/637: dread da/d4b/d53/dba/f60 [0,4194304] 0 2024-02-28T16:31:32.328 INFO:tasks.workunit.client.1.smithi066.stdout:9/638: readlink da/d3a/d15/l23 0 2024-02-28T16:31:32.330 INFO:tasks.workunit.client.1.smithi066.stdout:6/806: unlink d2/d2c/d82/ff8 0 2024-02-28T16:31:32.331 INFO:tasks.workunit.client.1.smithi066.stdout:0/921: creat d2/d27/d41/d76/d93/db2/db5/d52/da2/f130 x:0 0 0 2024-02-28T16:31:32.333 INFO:tasks.workunit.client.1.smithi066.stdout:8/866: creat d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/d106/f121 x:0 0 0 2024-02-28T16:31:32.336 INFO:tasks.workunit.client.1.smithi066.stdout:9/639: mkdir da/d3a/d15/dcf 0 2024-02-28T16:31:32.338 INFO:tasks.workunit.client.1.smithi066.stdout:5/727: dread d3/d4/dc/dd/d2e/d18/f1a [4194304,4194304] 0 2024-02-28T16:31:32.341 INFO:tasks.workunit.client.1.smithi066.stdout:9/640: readlink da/d3a/d58/d97/la6 0 2024-02-28T16:31:32.342 INFO:tasks.workunit.client.1.smithi066.stdout:6/807: symlink d2/d2c/da1/d5d/d99/l11d 0 2024-02-28T16:31:32.344 INFO:tasks.workunit.client.1.smithi066.stdout:1/804: rename d2/d18/c41 to d2/c10b 0 2024-02-28T16:31:32.348 INFO:tasks.workunit.client.1.smithi066.stdout:0/922: link d2/d27/d41/d76/d93/df3/f100 d2/d27/d41/d76/d93/d10/d23/d6d/d113/f131 0 2024-02-28T16:31:32.349 INFO:tasks.workunit.client.1.smithi066.stdout:2/826: dread d0/d12/d1a/d23/d29/d21/ffe [4194304,4194304] 0 2024-02-28T16:31:32.353 INFO:tasks.workunit.client.1.smithi066.stdout:3/943: dread db/dc/d35/d3d/d76/fb9 [4194304,4194304] 0 2024-02-28T16:31:32.354 INFO:tasks.workunit.client.1.smithi066.stdout:9/641: creat da/d22/d46/d56/d8f/d94/fd0 x:0 0 0 2024-02-28T16:31:32.356 INFO:tasks.workunit.client.1.smithi066.stdout:5/728: mkdir d3/d4/dc/dd/d35/d86/ded 0 2024-02-28T16:31:32.363 INFO:tasks.workunit.client.1.smithi066.stdout:0/923: write d2/d27/d41/d76/d93/db2/db5/d51/f70 [252177,47342] 0 2024-02-28T16:31:32.363 INFO:tasks.workunit.client.1.smithi066.stdout:4/958: dwrite d1/d23/d25/d3e/d40/d9c/d11c/d11d/f82 [4194304,4194304] 0 2024-02-28T16:31:32.368 INFO:tasks.workunit.client.1.smithi066.stdout:8/867: dwrite d0/d6/dc/f11d [0,4194304] 0 2024-02-28T16:31:32.372 INFO:tasks.workunit.client.1.smithi066.stdout:1/805: write d2/d27/f3c [752158,58889] 0 2024-02-28T16:31:32.376 INFO:tasks.workunit.client.1.smithi066.stdout:4/959: creat d1/d18/d55/f147 x:0 0 0 2024-02-28T16:31:32.379 INFO:tasks.workunit.client.1.smithi066.stdout:1/806: chown d2/d27/d2b/l3f 48 1 2024-02-28T16:31:32.381 INFO:tasks.workunit.client.1.smithi066.stdout:6/808: truncate d2/da/d19/d75/da6/daa/dbf/fe3 1335578 0 2024-02-28T16:31:32.384 INFO:tasks.workunit.client.1.smithi066.stdout:0/924: mkdir d2/d27/d41/d76/d93/db2/d132 0 2024-02-28T16:31:32.385 INFO:tasks.workunit.client.1.smithi066.stdout:1/807: stat d2/d27/dc7/d54/d84/cf1 0 2024-02-28T16:31:32.393 INFO:tasks.workunit.client.1.smithi066.stdout:8/868: truncate d0/db/d12/d1d/d25/d4f/fff 155686 0 2024-02-28T16:31:32.394 INFO:tasks.workunit.client.1.smithi066.stdout:1/808: write d2/d18/f3a [5355144,95778] 0 2024-02-28T16:31:32.397 INFO:tasks.workunit.client.1.smithi066.stdout:3/944: dwrite db/d2a/d44/d48/d7d/f130 [0,4194304] 0 2024-02-28T16:31:32.399 INFO:tasks.workunit.client.1.smithi066.stdout:0/925: chown d2/d27/d41/d76/d93/db2/db5/cbe 0 1 2024-02-28T16:31:32.400 INFO:tasks.workunit.client.1.smithi066.stdout:2/827: dwrite d0/f44 [0,4194304] 0 2024-02-28T16:31:32.400 INFO:tasks.workunit.client.1.smithi066.stdout:4/960: getdents d1/d18/d41/da7/dae/dc1 0 2024-02-28T16:31:32.407 INFO:tasks.workunit.client.1.smithi066.stdout:3/945: chown db/d2a/c54 48330 1 2024-02-28T16:31:32.408 INFO:tasks.workunit.client.1.smithi066.stdout:6/809: symlink d2/da/d19/d9c/dc0/l11e 0 2024-02-28T16:31:32.415 INFO:tasks.workunit.client.1.smithi066.stdout:1/809: creat d2/da0/f10c x:0 0 0 2024-02-28T16:31:32.416 INFO:tasks.workunit.client.1.smithi066.stdout:6/810: write d2/da/dcb/fe7 [803138,19752] 0 2024-02-28T16:31:32.419 INFO:tasks.workunit.client.1.smithi066.stdout:1/810: write d2/d27/d2b/d43/d46/fb6 [821877,60402] 0 2024-02-28T16:31:32.419 INFO:tasks.workunit.client.1.smithi066.stdout:8/869: creat d0/db/d12/d1d/d25/d65/dbb/d11e/f122 x:0 0 0 2024-02-28T16:31:32.421 INFO:tasks.workunit.client.1.smithi066.stdout:0/926: rmdir d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/d122 39 2024-02-28T16:31:32.422 INFO:tasks.workunit.client.1.smithi066.stdout:4/961: mknod d1/d23/d25/d3e/d40/d9c/d11c/d126/c148 0 2024-02-28T16:31:32.426 INFO:tasks.workunit.client.1.smithi066.stdout:6/811: readlink d2/d26/d32/lb3 0 2024-02-28T16:31:32.427 INFO:tasks.workunit.client.1.smithi066.stdout:1/811: write d2/d27/dc7/fc3 [1508565,77481] 0 2024-02-28T16:31:32.431 INFO:tasks.workunit.client.1.smithi066.stdout:3/946: mknod db/dc/d11/d1f/c156 0 2024-02-28T16:31:32.438 INFO:tasks.workunit.client.1.smithi066.stdout:5/729: dread d3/d4/fb [0,4194304] 0 2024-02-28T16:31:32.442 INFO:tasks.workunit.client.1.smithi066.stdout:8/870: unlink d0/d6/dc/d16/d4c/d6c/fe4 0 2024-02-28T16:31:32.445 INFO:tasks.workunit.client.1.smithi066.stdout:6/812: rename d2/d2c/da1/d5d/d63/lb6 to d2/da/d19/d9c/d3d/l11f 0 2024-02-28T16:31:32.446 INFO:tasks.workunit.client.1.smithi066.stdout:4/962: fsync d1/d18/d7e/dbb/fee 0 2024-02-28T16:31:32.450 INFO:tasks.workunit.client.1.smithi066.stdout:2/828: dwrite d0/d12/d1a/d23/d29/fa8 [4194304,4194304] 0 2024-02-28T16:31:32.455 INFO:tasks.workunit.client.1.smithi066.stdout:6/813: write d2/da/dcb/f101 [422268,116154] 0 2024-02-28T16:31:32.458 INFO:tasks.workunit.client.1.smithi066.stdout:3/947: unlink db/dc/d11/d1f/lea 0 2024-02-28T16:31:32.464 INFO:tasks.workunit.client.1.smithi066.stdout:8/871: creat d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db1/f123 x:0 0 0 2024-02-28T16:31:32.475 INFO:tasks.workunit.client.1.smithi066.stdout:6/814: chown d2/d26/c9f 771 1 2024-02-28T16:31:32.478 INFO:tasks.workunit.client.1.smithi066.stdout:8/872: chown d0/db/d12/d1d/d25/d4f/f9f 4 1 2024-02-28T16:31:32.480 INFO:tasks.workunit.client.1.smithi066.stdout:4/963: dwrite d1/d23/ffa [0,4194304] 0 2024-02-28T16:31:32.489 INFO:tasks.workunit.client.1.smithi066.stdout:0/927: rename d2/d27/d41/d76/d93/d10/d23/d6d/c116 to d2/d27/d41/d76/d93/d10/d23/d6d/d113/dfc/c133 0 2024-02-28T16:31:32.495 INFO:tasks.workunit.client.1.smithi066.stdout:3/948: dread db/dc/f55 [0,4194304] 0 2024-02-28T16:31:32.496 INFO:tasks.workunit.client.1.smithi066.stdout:0/928: chown d2/d27/d41/d76/d93/d10/d23/d6d/d113 2372 1 2024-02-28T16:31:32.507 INFO:tasks.workunit.client.1.smithi066.stdout:5/730: write d3/d4/f4e [4976789,51985] 0 2024-02-28T16:31:32.509 INFO:tasks.workunit.client.1.smithi066.stdout:5/731: dread - d3/fd8 zero size 2024-02-28T16:31:32.511 INFO:tasks.workunit.client.1.smithi066.stdout:4/964: rename d1/d23/d25/d115 to d1/dfb/d149 0 2024-02-28T16:31:32.513 INFO:tasks.workunit.client.1.smithi066.stdout:5/732: fsync d3/d4/dc/dd/d35/f62 0 2024-02-28T16:31:32.527 INFO:tasks.workunit.client.1.smithi066.stdout:0/929: read d2/d27/d41/d76/d93/db2/db5/d51/f53 [1345827,28023] 0 2024-02-28T16:31:32.536 INFO:tasks.workunit.client.1.smithi066.stdout:0/930: write d2/d27/d41/d76/d93/db2/db5/d51/f70 [385293,11935] 0 2024-02-28T16:31:32.539 INFO:tasks.workunit.client.1.smithi066.stdout:2/829: getdents d0/d12/d3e/d9e/dce 0 2024-02-28T16:31:32.540 INFO:tasks.workunit.client.1.smithi066.stdout:1/812: getdents d2/d18/d7a 0 2024-02-28T16:31:32.544 INFO:tasks.workunit.client.1.smithi066.stdout:3/949: creat db/d2a/d44/d48/d52/d77/d12c/f157 x:0 0 0 2024-02-28T16:31:32.545 INFO:tasks.workunit.client.1.smithi066.stdout:6/815: dwrite d2/d2c/f30 [0,4194304] 0 2024-02-28T16:31:32.549 INFO:tasks.workunit.client.1.smithi066.stdout:4/965: dread d1/d18/da8/f120 [0,4194304] 0 2024-02-28T16:31:32.552 INFO:tasks.workunit.client.1.smithi066.stdout:3/950: chown db/dc/d11/d1f/d78/ffb 3 1 2024-02-28T16:31:32.552 INFO:tasks.workunit.client.1.smithi066.stdout:6/816: stat d2/da/d19/f93 0 2024-02-28T16:31:32.556 INFO:tasks.workunit.client.1.smithi066.stdout:0/931: read d2/d27/d41/d76/d93/df0/f10d [2443168,52294] 0 2024-02-28T16:31:32.558 INFO:tasks.workunit.client.1.smithi066.stdout:8/873: rename d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/fa4 to d0/db/d12/d1d/d25/d65/f124 0 2024-02-28T16:31:32.562 INFO:tasks.workunit.client.1.smithi066.stdout:1/813: unlink d2/l9 0 2024-02-28T16:31:32.562 INFO:tasks.workunit.client.1.smithi066.stdout:8/874: write d0/d6/dc/d16/d4c/d6c/f78 [2052535,50968] 0 2024-02-28T16:31:32.564 INFO:tasks.workunit.client.1.smithi066.stdout:4/966: read - d1/d18/d4e/da0/fb4 zero size 2024-02-28T16:31:32.571 INFO:tasks.workunit.client.1.smithi066.stdout:2/830: rename d0/d12/d1a/d39/lab to d0/dd7/l109 0 2024-02-28T16:31:32.613 INFO:tasks.workunit.client.1.smithi066.stdout:0/932: symlink d2/d85/l134 0 2024-02-28T16:31:32.614 INFO:tasks.workunit.client.1.smithi066.stdout:4/967: fsync d1/d23/d25/d3e/f66 0 2024-02-28T16:31:32.617 INFO:tasks.workunit.client.1.smithi066.stdout:8/875: mkdir d0/db/d12/d1d/d25/d4f/de0/de8/d125 0 2024-02-28T16:31:32.617 INFO:tasks.workunit.client.1.smithi066.stdout:1/814: creat d2/d27/df9/f10d x:0 0 0 2024-02-28T16:31:32.623 INFO:tasks.workunit.client.1.smithi066.stdout:1/815: write d2/d27/df9/fcf [1467728,50639] 0 2024-02-28T16:31:32.623 INFO:tasks.workunit.client.1.smithi066.stdout:4/968: creat d1/d23/d25/d3e/d40/f14a x:0 0 0 2024-02-28T16:31:32.626 INFO:tasks.workunit.client.1.smithi066.stdout:0/933: mkdir d2/d27/d41/d76/d93/d33/d3f/d135 0 2024-02-28T16:31:32.629 INFO:tasks.workunit.client.1.smithi066.stdout:1/816: creat d2/d18/d7a/f10e x:0 0 0 2024-02-28T16:31:32.630 INFO:tasks.workunit.client.1.smithi066.stdout:4/969: mkdir d1/dfb/d149/d14b 0 2024-02-28T16:31:32.633 INFO:tasks.workunit.client.1.smithi066.stdout:0/934: mknod d2/d27/d41/d76/dcf/d11f/c136 0 2024-02-28T16:31:32.639 INFO:tasks.workunit.client.1.smithi066.stdout:5/733: dread d3/d4/f9 [0,4194304] 0 2024-02-28T16:31:32.640 INFO:tasks.workunit.client.1.smithi066.stdout:6/817: dread d2/d2c/da1/d5d/d99/f5e [0,4194304] 0 2024-02-28T16:31:32.650 INFO:tasks.workunit.client.1.smithi066.stdout:2/831: dread d0/d12/d1a/d39/d7c/d87/fe5 [0,4194304] 0 2024-02-28T16:31:32.652 INFO:tasks.workunit.client.1.smithi066.stdout:2/832: write d0/ff0 [1138937,90212] 0 2024-02-28T16:31:32.666 INFO:tasks.workunit.client.1.smithi066.stdout:8/876: dread d0/d6/ff [4194304,4194304] 0 2024-02-28T16:31:32.685 INFO:tasks.workunit.client.1.smithi066.stdout:9/642: sync 2024-02-28T16:31:32.685 INFO:tasks.workunit.client.1.smithi066.stdout:7/749: sync 2024-02-28T16:31:32.689 INFO:tasks.workunit.client.1.smithi066.stdout:9/643: write da/d22/d46/d56/d8f/fb9 [884404,17945] 0 2024-02-28T16:31:32.691 INFO:tasks.workunit.client.1.smithi066.stdout:9/644: readlink da/df/l26 0 2024-02-28T16:31:32.696 INFO:tasks.workunit.client.1.smithi066.stdout:4/970: rename d1/d23/d25/d3e/d40/d9c/d11c/f11e to d1/d18/da8/dcb/d13a/f14c 0 2024-02-28T16:31:32.701 INFO:tasks.workunit.client.1.smithi066.stdout:0/935: mknod d2/d27/d41/d76/d93/d10/d23/db6/de0/c137 0 2024-02-28T16:31:32.704 INFO:tasks.workunit.client.1.smithi066.stdout:0/936: chown d2/d27/d41/d76/d93/db2/db5/d52/da2 126086 1 2024-02-28T16:31:32.707 INFO:tasks.workunit.client.1.smithi066.stdout:0/937: fdatasync d2/d27/d41/d76/d93/df0/d12b/f12e 0 2024-02-28T16:31:32.711 INFO:tasks.workunit.client.1.smithi066.stdout:0/938: chown d2/d27/d41/d76/d93/db2/db5/d73/fc3 840519506 1 2024-02-28T16:31:32.716 INFO:tasks.workunit.client.1.smithi066.stdout:4/971: dwrite d1/d23/d25/d3e/d40/d9c/d11c/d11d/ded/f114 [0,4194304] 0 2024-02-28T16:31:32.728 INFO:tasks.workunit.client.1.smithi066.stdout:3/951: sync 2024-02-28T16:31:32.730 INFO:tasks.workunit.client.1.smithi066.stdout:3/952: read db/ff [1935310,90927] 0 2024-02-28T16:31:32.732 INFO:tasks.workunit.client.1.smithi066.stdout:3/953: chown db/d13/d1a/l87 16201 1 2024-02-28T16:31:32.754 INFO:tasks.workunit.client.1.smithi066.stdout:7/750: rmdir dc/d29/d92 39 2024-02-28T16:31:32.757 INFO:tasks.workunit.client.1.smithi066.stdout:1/817: creat d2/d18/d7a/dfd/d10a/f10f x:0 0 0 2024-02-28T16:31:32.760 INFO:tasks.workunit.client.1.smithi066.stdout:9/645: rename da/df/d70 to da/d22/d46/d56/d8f/d2d/d81/dd1 0 2024-02-28T16:31:32.762 INFO:tasks.workunit.client.1.smithi066.stdout:9/646: chown da/d22/d46/d56/d8f/d94/la3 15269 1 2024-02-28T16:31:32.768 INFO:tasks.workunit.client.1.smithi066.stdout:8/877: dwrite d0/d6/dc/d16/d1a/d63/d58/f70 [0,4194304] 0 2024-02-28T16:31:32.773 INFO:tasks.workunit.client.1.smithi066.stdout:2/833: creat d0/d12/d1a/d23/ddc/d103/f10a x:0 0 0 2024-02-28T16:31:32.775 INFO:tasks.workunit.client.1.smithi066.stdout:1/818: dwrite d2/d18/d7a/d8d/f8f [0,4194304] 0 2024-02-28T16:31:32.777 INFO:tasks.workunit.client.1.smithi066.stdout:0/939: chown d2/d27/d41/d76/d93/d10/d23/d6d/f103 0 1 2024-02-28T16:31:32.778 INFO:tasks.workunit.client.1.smithi066.stdout:3/954: truncate db/d2a/d44/f4f 3285111 0 2024-02-28T16:31:32.779 INFO:tasks.workunit.client.1.smithi066.stdout:1/819: write d2/d27/d2b/fc6 [1187206,61578] 0 2024-02-28T16:31:32.783 INFO:tasks.workunit.client.1.smithi066.stdout:0/940: chown d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/de4/da1 11184958 1 2024-02-28T16:31:32.786 INFO:tasks.workunit.client.1.smithi066.stdout:8/878: mkdir d0/db/d12/d1d/d25/d4f/d126 0 2024-02-28T16:31:32.789 INFO:tasks.workunit.client.1.smithi066.stdout:4/972: symlink d1/d18/l14d 0 2024-02-28T16:31:32.791 INFO:tasks.workunit.client.1.smithi066.stdout:4/973: readlink d1/d18/d41/da7/dae/dc1/le2 0 2024-02-28T16:31:32.793 INFO:tasks.workunit.client.1.smithi066.stdout:3/955: mkdir db/d13/d14/d2d/de3/d158 0 2024-02-28T16:31:32.795 INFO:tasks.workunit.client.1.smithi066.stdout:4/974: write d1/d23/d25/d3e/d40/d9c/d11c/d11d/f5d [185813,32614] 0 2024-02-28T16:31:32.797 INFO:tasks.workunit.client.1.smithi066.stdout:7/751: chown dc/d29/d92/ca4 22879062 1 2024-02-28T16:31:32.797 INFO:tasks.workunit.client.1.smithi066.stdout:9/647: rename da/d3a/d15/f4f to da/d3a/d58/dab/fd2 0 2024-02-28T16:31:32.798 INFO:tasks.workunit.client.1.smithi066.stdout:9/648: fsync da/f2a 0 2024-02-28T16:31:32.801 INFO:tasks.workunit.client.1.smithi066.stdout:9/649: readlink da/d3a/d15/l23 0 2024-02-28T16:31:32.803 INFO:tasks.workunit.client.1.smithi066.stdout:4/975: write d1/d23/d25/d3e/d40/f11b [968961,94741] 0 2024-02-28T16:31:32.810 INFO:tasks.workunit.client.1.smithi066.stdout:8/879: dwrite d0/d6/dc/d16/fc3 [0,4194304] 0 2024-02-28T16:31:32.810 INFO:tasks.workunit.client.1.smithi066.stdout:9/650: symlink da/df/ld3 0 2024-02-28T16:31:32.812 INFO:tasks.workunit.client.1.smithi066.stdout:4/976: symlink d1/d18/d41/dba/d100/l14e 0 2024-02-28T16:31:32.813 INFO:tasks.workunit.client.1.smithi066.stdout:3/956: fdatasync db/d2a/d44/d48/d5d/fa9 0 2024-02-28T16:31:32.818 INFO:tasks.workunit.client.1.smithi066.stdout:0/941: creat d2/d27/d41/d76/d93/d10/d23/d6d/f138 x:0 0 0 2024-02-28T16:31:32.823 INFO:tasks.workunit.client.1.smithi066.stdout:4/977: stat d1/d7a/cb2 0 2024-02-28T16:31:32.823 INFO:tasks.workunit.client.1.smithi066.stdout:5/734: write d3/d4/f61 [3459806,48317] 0 2024-02-28T16:31:32.825 INFO:tasks.workunit.client.1.smithi066.stdout:8/880: dread d0/f43 [0,4194304] 0 2024-02-28T16:31:32.826 INFO:tasks.workunit.client.1.smithi066.stdout:4/978: chown d1/d23/d25/d3e/d40/d9c 21 1 2024-02-28T16:31:32.828 INFO:tasks.workunit.client.1.smithi066.stdout:8/881: readlink d0/d6/dc/d16/d4c/d6c/d6f/da7/ldf 0 2024-02-28T16:31:32.833 INFO:tasks.workunit.client.1.smithi066.stdout:8/882: write d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/f109 [334881,113072] 0 2024-02-28T16:31:32.834 INFO:tasks.workunit.client.1.smithi066.stdout:3/957: fsync db/d13/d14/dad/fef 0 2024-02-28T16:31:32.844 INFO:tasks.workunit.client.1.smithi066.stdout:4/979: creat d1/d18/d4e/f14f x:0 0 0 2024-02-28T16:31:32.847 INFO:tasks.workunit.client.1.smithi066.stdout:4/980: symlink d1/d18/d7e/dbb/l150 0 2024-02-28T16:31:32.848 INFO:tasks.workunit.client.1.smithi066.stdout:8/883: getdents d0/d6/dc/d16/d4c/d6c/daf 0 2024-02-28T16:31:32.853 INFO:tasks.workunit.client.1.smithi066.stdout:8/884: unlink d0/db/f6e 0 2024-02-28T16:31:32.854 INFO:tasks.workunit.client.1.smithi066.stdout:4/981: creat d1/d23/d25/f151 x:0 0 0 2024-02-28T16:31:32.858 INFO:tasks.workunit.client.1.smithi066.stdout:4/982: creat d1/d18/da8/f152 x:0 0 0 2024-02-28T16:31:32.858 INFO:tasks.workunit.client.1.smithi066.stdout:4/983: readlink d1/l21 0 2024-02-28T16:31:32.862 INFO:tasks.workunit.client.1.smithi066.stdout:4/984: symlink d1/d18/l153 0 2024-02-28T16:31:32.865 INFO:tasks.workunit.client.1.smithi066.stdout:7/752: mkdir dc/d29/d92/df2 0 2024-02-28T16:31:32.866 INFO:tasks.workunit.client.1.smithi066.stdout:4/985: symlink d1/dfb/d149/l154 0 2024-02-28T16:31:32.868 INFO:tasks.workunit.client.1.smithi066.stdout:7/753: readlink dc/d29/l85 0 2024-02-28T16:31:32.869 INFO:tasks.workunit.client.1.smithi066.stdout:4/986: fsync d1/d23/d25/d3e/d40/d9c/d11c/d11d/f8b 0 2024-02-28T16:31:32.873 INFO:tasks.workunit.client.1.smithi066.stdout:4/987: mknod d1/d18/c155 0 2024-02-28T16:31:32.875 INFO:tasks.workunit.client.1.smithi066.stdout:4/988: write d1/f4 [8301076,41505] 0 2024-02-28T16:31:32.881 INFO:tasks.workunit.client.1.smithi066.stdout:6/818: truncate d2/da/d19/f93 5770573 0 2024-02-28T16:31:32.896 INFO:tasks.workunit.client.1.smithi066.stdout:2/834: dwrite d0/d12/d2e/d3b/d66/f7f [0,4194304] 0 2024-02-28T16:31:32.896 INFO:tasks.workunit.client.1.smithi066.stdout:9/651: write da/fc [4616568,46321] 0 2024-02-28T16:31:32.897 INFO:tasks.workunit.client.1.smithi066.stdout:1/820: mkdir d2/d18/d7a/d110 0 2024-02-28T16:31:32.899 INFO:tasks.workunit.client.1.smithi066.stdout:2/835: readlink d0/lb9 0 2024-02-28T16:31:32.900 INFO:tasks.workunit.client.1.smithi066.stdout:3/958: write db/d13/d14/d84/f112 [3052021,86365] 0 2024-02-28T16:31:32.901 INFO:tasks.workunit.client.1.smithi066.stdout:7/754: dread dc/d29/d92/da6/dd8/f7c [0,4194304] 0 2024-02-28T16:31:32.912 INFO:tasks.workunit.client.1.smithi066.stdout:4/989: dread d1/d18/da8/fd6 [0,4194304] 0 2024-02-28T16:31:32.912 INFO:tasks.workunit.client.1.smithi066.stdout:8/885: truncate d0/d6/dc/f11d 969403 0 2024-02-28T16:31:32.913 INFO:tasks.workunit.client.1.smithi066.stdout:3/959: readlink db/d2a/d44/d48/d52/d77/ldb 0 2024-02-28T16:31:32.913 INFO:tasks.workunit.client.1.smithi066.stdout:2/836: rmdir d0/d12/d1a/d39/d68 39 2024-02-28T16:31:32.919 INFO:tasks.workunit.client.1.smithi066.stdout:4/990: chown d1/d18/da8/fbe 14846914 1 2024-02-28T16:31:32.921 INFO:tasks.workunit.client.1.smithi066.stdout:0/942: dwrite d2/d27/d41/d76/d93/db2/db5/d51/f5a [0,4194304] 0 2024-02-28T16:31:32.921 INFO:tasks.workunit.client.1.smithi066.stdout:3/960: stat db/dc/db3/dff/c140 0 2024-02-28T16:31:32.927 INFO:tasks.workunit.client.1.smithi066.stdout:5/735: dwrite d3/d4/f5c [4194304,4194304] 0 2024-02-28T16:31:32.929 INFO:tasks.workunit.client.1.smithi066.stdout:4/991: stat d1/d18/d41/dba/d100/f10b 0 2024-02-28T16:31:32.930 INFO:tasks.workunit.client.1.smithi066.stdout:6/819: write d2/d2c/d72/fe1 [1905893,38791] 0 2024-02-28T16:31:32.931 INFO:tasks.workunit.client.1.smithi066.stdout:8/886: rmdir d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/d106 39 2024-02-28T16:31:32.936 INFO:tasks.workunit.client.1.smithi066.stdout:4/992: creat d1/dfb/f156 x:0 0 0 2024-02-28T16:31:32.937 INFO:tasks.workunit.client.1.smithi066.stdout:3/961: mkdir db/d13/d14/d2d/d92/d159 0 2024-02-28T16:31:32.942 INFO:tasks.workunit.client.1.smithi066.stdout:1/821: dwrite d2/d27/f2f [0,4194304] 0 2024-02-28T16:31:32.944 INFO:tasks.workunit.client.1.smithi066.stdout:9/652: dread da/d22/d46/d56/d8f/d28/d42/d68/f8c [0,4194304] 0 2024-02-28T16:31:32.945 INFO:tasks.workunit.client.1.smithi066.stdout:0/943: unlink d2/d27/d41/d76/d93/d10/d23/l104 0 2024-02-28T16:31:32.948 INFO:tasks.workunit.client.1.smithi066.stdout:0/944: dread - d2/d27/d41/d76/dcf/f10c zero size 2024-02-28T16:31:32.948 INFO:tasks.workunit.client.1.smithi066.stdout:9/653: write da/d4b/d53/dba/fb4 [766646,112843] 0 2024-02-28T16:31:32.949 INFO:tasks.workunit.client.1.smithi066.stdout:6/820: mknod d2/d26/de0/c120 0 2024-02-28T16:31:32.950 INFO:tasks.workunit.client.1.smithi066.stdout:2/837: truncate d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/fd0 1212079 0 2024-02-28T16:31:32.952 INFO:tasks.workunit.client.1.smithi066.stdout:0/945: readlink d2/d27/d41/l48 0 2024-02-28T16:31:32.957 INFO:tasks.workunit.client.1.smithi066.stdout:1/822: mkdir d2/d18/d7a/dfd/d111 0 2024-02-28T16:31:32.962 INFO:tasks.workunit.client.1.smithi066.stdout:9/654: symlink da/d22/d46/d56/d8f/d94/ld4 0 2024-02-28T16:31:32.966 INFO:tasks.workunit.client.1.smithi066.stdout:7/755: dread dc/d29/d92/da6/dd8/d1e/d45/f8f [0,4194304] 0 2024-02-28T16:31:32.969 INFO:tasks.workunit.client.1.smithi066.stdout:4/993: dwrite d1/d7a/f132 [0,4194304] 0 2024-02-28T16:31:32.969 INFO:tasks.workunit.client.1.smithi066.stdout:8/887: write d0/d6/dc/d16/d4c/d42/da8/fb6 [523045,86366] 0 2024-02-28T16:31:32.971 INFO:tasks.workunit.client.1.smithi066.stdout:6/821: mknod d2/d2c/d6e/df3/c121 0 2024-02-28T16:31:32.974 INFO:tasks.workunit.client.1.smithi066.stdout:7/756: fsync dc/d6e/f6f 0 2024-02-28T16:31:32.975 INFO:tasks.workunit.client.1.smithi066.stdout:5/736: getdents d3/d4/dc/dd/d41/da2 0 2024-02-28T16:31:32.977 INFO:tasks.workunit.client.1.smithi066.stdout:1/823: mknod d2/d27/dc7/da6/dd4/c112 0 2024-02-28T16:31:32.979 INFO:tasks.workunit.client.1.smithi066.stdout:6/822: write d2/da/f76 [1238765,90020] 0 2024-02-28T16:31:32.986 INFO:tasks.workunit.client.1.smithi066.stdout:2/838: write d0/d12/d1a/d23/d29/d21/ffe [2359955,53027] 0 2024-02-28T16:31:32.988 INFO:tasks.workunit.client.1.smithi066.stdout:8/888: readlink d0/db/d12/d1d/l76 0 2024-02-28T16:31:32.988 INFO:tasks.workunit.client.1.smithi066.stdout:4/994: creat d1/d23/d25/dd9/f157 x:0 0 0 2024-02-28T16:31:32.990 INFO:tasks.workunit.client.1.smithi066.stdout:3/962: dwrite db/dc/d11/daa/fb6 [0,4194304] 0 2024-02-28T16:31:32.993 INFO:tasks.workunit.client.1.smithi066.stdout:6/823: fsync d2/d2c/da1/d5d/d63/d6a/d100/f119 0 2024-02-28T16:31:32.998 INFO:tasks.workunit.client.1.smithi066.stdout:7/757: creat dc/d56/ddf/ff3 x:0 0 0 2024-02-28T16:31:33.000 INFO:tasks.workunit.client.1.smithi066.stdout:6/824: dread - d2/d2c/da1/dad/f11b zero size 2024-02-28T16:31:33.002 INFO:tasks.workunit.client.1.smithi066.stdout:6/825: chown d2/da/d19 0 1 2024-02-28T16:31:33.007 INFO:tasks.workunit.client.1.smithi066.stdout:2/839: dread d0/d12/d1a/d39/d7c/d87/dc4/ff1 [0,4194304] 0 2024-02-28T16:31:33.007 INFO:tasks.workunit.client.1.smithi066.stdout:8/889: creat d0/d6/dc/d16/d1a/d63/d3d/f127 x:0 0 0 2024-02-28T16:31:33.009 INFO:tasks.workunit.client.1.smithi066.stdout:9/655: dread f3 [0,4194304] 0 2024-02-28T16:31:33.016 INFO:tasks.workunit.client.1.smithi066.stdout:2/840: read d0/d12/d1a/d39/d73/fbc [7823815,84261] 0 2024-02-28T16:31:33.016 INFO:tasks.workunit.client.1.smithi066.stdout:3/963: mkdir db/d2a/d44/d15a 0 2024-02-28T16:31:33.016 INFO:tasks.workunit.client.1.smithi066.stdout:6/826: symlink d2/d2c/d6e/l122 0 2024-02-28T16:31:33.021 INFO:tasks.workunit.client.1.smithi066.stdout:0/946: dread d2/d27/d41/d76/d93/db2/db5/d51/fd9 [0,4194304] 0 2024-02-28T16:31:33.023 INFO:tasks.workunit.client.1.smithi066.stdout:8/890: write d0/d6/dc/d16/d4c/d42/da8/fb6 [636475,34360] 0 2024-02-28T16:31:33.023 INFO:tasks.workunit.client.1.smithi066.stdout:6/827: stat d2/d2c/d6e/l9a 0 2024-02-28T16:31:33.027 INFO:tasks.workunit.client.1.smithi066.stdout:4/995: link d1/d7a/fd7 d1/d23/d25/d3e/d40/d9c/d11c/f158 0 2024-02-28T16:31:33.028 INFO:tasks.workunit.client.1.smithi066.stdout:7/758: mkdir dc/d56/dc3/df4 0 2024-02-28T16:31:33.030 INFO:tasks.workunit.client.1.smithi066.stdout:6/828: rmdir d2/d26/de0 39 2024-02-28T16:31:33.030 INFO:tasks.workunit.client.1.smithi066.stdout:4/996: chown d1/d23/d25/d3e 157 1 2024-02-28T16:31:33.033 INFO:tasks.workunit.client.1.smithi066.stdout:4/997: readlink d1/d18/l14d 0 2024-02-28T16:31:33.039 INFO:tasks.workunit.client.1.smithi066.stdout:5/737: dwrite d3/d4/dc/dd/d2e/d18/f4b [0,4194304] 0 2024-02-28T16:31:33.041 INFO:tasks.workunit.client.1.smithi066.stdout:1/824: dread d2/d18/f3a [0,4194304] 0 2024-02-28T16:31:33.042 INFO:tasks.workunit.client.1.smithi066.stdout:3/964: creat db/d13/d14/f15b x:0 0 0 2024-02-28T16:31:33.042 INFO:tasks.workunit.client.1.smithi066.stdout:0/947: rename d2/d27/d41/d76/d93/db2/db5/d52 to d2/d27/d41/d76/d93/d139 0 2024-02-28T16:31:33.045 INFO:tasks.workunit.client.1.smithi066.stdout:7/759: creat dc/d56/dc3/ff5 x:0 0 0 2024-02-28T16:31:33.049 INFO:tasks.workunit.client.1.smithi066.stdout:3/965: fsync db/dc/d11/d1f/d78/f90 0 2024-02-28T16:31:33.050 INFO:tasks.workunit.client.1.smithi066.stdout:7/760: fsync dc/d29/fc7 0 2024-02-28T16:31:33.052 INFO:tasks.workunit.client.1.smithi066.stdout:8/891: truncate d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db1/f11c 104837 0 2024-02-28T16:31:33.059 INFO:tasks.workunit.client.1.smithi066.stdout:2/841: dwrite d0/d12/d1a/d23/fe1 [0,4194304] 0 2024-02-28T16:31:33.059 INFO:tasks.workunit.client.1.smithi066.stdout:1/825: readlink d2/d18/d6b/dba/de1/le5 0 2024-02-28T16:31:33.059 INFO:tasks.workunit.client.1.smithi066.stdout:9/656: dread da/d22/d46/d56/d8f/d2d/f62 [0,4194304] 0 2024-02-28T16:31:33.066 INFO:tasks.workunit.client.1.smithi066.stdout:0/948: dread - d2/d85/d8d/ff9 zero size 2024-02-28T16:31:33.066 INFO:tasks.workunit.client.1.smithi066.stdout:5/738: mknod d3/d4/dc/dd/cee 0 2024-02-28T16:31:33.069 INFO:tasks.workunit.client.1.smithi066.stdout:6/829: rename d2/da/d19/cd8 to d2/da/dcc/dd6/d102/c123 0 2024-02-28T16:31:33.069 INFO:tasks.workunit.client.1.smithi066.stdout:2/842: mkdir d0/d12/d1a/d23/d10b 0 2024-02-28T16:31:33.072 INFO:tasks.workunit.client.1.smithi066.stdout:1/826: unlink d2/d27/dc7/c4b 0 2024-02-28T16:31:33.076 INFO:tasks.workunit.client.1.smithi066.stdout:8/892: creat d0/d6/dc/d16/d4c/d42/d73/f128 x:0 0 0 2024-02-28T16:31:33.078 INFO:tasks.workunit.client.1.smithi066.stdout:2/843: mkdir d0/dd7/d10c 0 2024-02-28T16:31:33.079 INFO:tasks.workunit.client.1.smithi066.stdout:4/998: write d1/d23/d25/d77/ff3 [780452,82662] 0 2024-02-28T16:31:33.081 INFO:tasks.workunit.client.1.smithi066.stdout:6/830: symlink d2/d26/dbe/df7/l124 0 2024-02-28T16:31:33.081 INFO:tasks.workunit.client.1.smithi066.stdout:8/893: readlink d0/db/d12/d1d/d25/d65/dbb/d11e/lc1 0 2024-02-28T16:31:33.088 INFO:tasks.workunit.client.1.smithi066.stdout:1/827: creat d2/d18/d6b/dba/de1/f113 x:0 0 0 2024-02-28T16:31:33.095 INFO:tasks.workunit.client.1.smithi066.stdout:6/831: dread - d2/d2c/da1/d5d/d63/d6a/d100/f119 zero size 2024-02-28T16:31:33.099 INFO:tasks.workunit.client.1.smithi066.stdout:1/828: creat d2/d18/d6b/dba/de1/f114 x:0 0 0 2024-02-28T16:31:33.100 INFO:tasks.workunit.client.1.smithi066.stdout:2/844: mknod d0/d12/d1a/d39/d68/c10d 0 2024-02-28T16:31:33.100 INFO:tasks.workunit.client.1.smithi066.stdout:8/894: write d0/d6/dc/d16/d4c/d6c/f91 [1439934,11219] 0 2024-02-28T16:31:33.101 INFO:tasks.workunit.client.1.smithi066.stdout:7/761: dread dc/d56/f77 [0,4194304] 0 2024-02-28T16:31:33.101 INFO:tasks.workunit.client.1.smithi066.stdout:9/657: dwrite da/d3a/d15/d48/fa4 [0,4194304] 0 2024-02-28T16:31:33.105 INFO:tasks.workunit.client.1.smithi066.stdout:5/739: write d3/d4/dc/dd/d41/d7a/dac/d60/fb2 [3770329,14374] 0 2024-02-28T16:31:33.109 INFO:tasks.workunit.client.1.smithi066.stdout:0/949: dread d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/de4/da1/fd2 [0,4194304] 0 2024-02-28T16:31:33.111 INFO:tasks.workunit.client.1.smithi066.stdout:2/845: mknod d0/d12/c10e 0 2024-02-28T16:31:33.113 INFO:tasks.workunit.client.1.smithi066.stdout:8/895: write d0/d6/dc/d16/d4c/d42/d73/f128 [760749,102779] 0 2024-02-28T16:31:33.115 INFO:tasks.workunit.client.1.smithi066.stdout:7/762: stat dc/d29/d92/da6/dd8/d91/fe2 0 2024-02-28T16:31:33.117 INFO:tasks.workunit.client.1.smithi066.stdout:2/846: fsync d0/d12/d1a/ff7 0 2024-02-28T16:31:33.118 INFO:tasks.workunit.client.1.smithi066.stdout:6/832: rename d2/da/d19/d75 to d2/da/d19/d75/dbd/dc6/dc7/d125 22 2024-02-28T16:31:33.119 INFO:tasks.workunit.client.1.smithi066.stdout:8/896: stat d0/db/d12/d8a 0 2024-02-28T16:31:33.120 INFO:tasks.workunit.client.1.smithi066.stdout:4/999: dread d1/d7a/f132 [0,4194304] 0 2024-02-28T16:31:33.120 INFO:tasks.workunit.client.1.smithi066.stdout:3/966: dread db/dc/d11/d1f/d78/f90 [0,4194304] 0 2024-02-28T16:31:33.121 INFO:tasks.workunit.client.1.smithi066.stdout:6/833: readlink d2/d26/d32/lf1 0 2024-02-28T16:31:33.129 INFO:tasks.workunit.client.1.smithi066.stdout:9/658: write da/df/f3d [3775833,71734] 0 2024-02-28T16:31:33.133 INFO:tasks.workunit.client.1.smithi066.stdout:3/967: mknod db/d13/d14/d2d/c15c 0 2024-02-28T16:31:33.135 INFO:tasks.workunit.client.1.smithi066.stdout:5/740: dwrite d3/d4/dc/dd/d2e/fce [0,4194304] 0 2024-02-28T16:31:33.137 INFO:tasks.workunit.client.1.smithi066.stdout:0/950: creat d2/d27/d41/d76/d93/d115/f13a x:0 0 0 2024-02-28T16:31:33.140 INFO:tasks.workunit.client.1.smithi066.stdout:3/968: readlink db/dc/d35/d3d/d76/d89/dcc/d11e/l133 0 2024-02-28T16:31:33.141 INFO:tasks.workunit.client.1.smithi066.stdout:9/659: mknod da/d6b/cd5 0 2024-02-28T16:31:33.141 INFO:tasks.workunit.client.1.smithi066.stdout:1/829: link d2/d27/dc7/d51/l90 d2/d27/d2b/d43/l115 0 2024-02-28T16:31:33.146 INFO:tasks.workunit.client.1.smithi066.stdout:5/741: symlink d3/d4/dc/dd/lef 0 2024-02-28T16:31:33.146 INFO:tasks.workunit.client.1.smithi066.stdout:9/660: chown da/d4b/d53/dba/lad 717844 1 2024-02-28T16:31:33.153 INFO:tasks.workunit.client.1.smithi066.stdout:7/763: getdents dc/de 0 2024-02-28T16:31:33.155 INFO:tasks.workunit.client.1.smithi066.stdout:0/951: truncate d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f127 1001281 0 2024-02-28T16:31:33.157 INFO:tasks.workunit.client.1.smithi066.stdout:6/834: truncate d2/d26/f2d 6016660 0 2024-02-28T16:31:33.158 INFO:tasks.workunit.client.1.smithi066.stdout:8/897: dwrite d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db1/f123 [0,4194304] 0 2024-02-28T16:31:33.162 INFO:tasks.workunit.client.1.smithi066.stdout:9/661: rename da/d4b/d53/dba/lad to da/d3a/d15/ld6 0 2024-02-28T16:31:33.162 INFO:tasks.workunit.client.1.smithi066.stdout:0/952: rmdir d2 39 2024-02-28T16:31:33.166 INFO:tasks.workunit.client.1.smithi066.stdout:3/969: creat db/d2a/d44/f15d x:0 0 0 2024-02-28T16:31:33.170 INFO:tasks.workunit.client.1.smithi066.stdout:0/953: fdatasync d2/d27/d41/d76/d93/d139/da2/f130 0 2024-02-28T16:31:33.171 INFO:tasks.workunit.client.1.smithi066.stdout:3/970: chown db/dc/d11/c12a 374 1 2024-02-28T16:31:33.173 INFO:tasks.workunit.client.1.smithi066.stdout:9/662: creat da/d22/d46/d56/d8f/d28/d42/d68/fd7 x:0 0 0 2024-02-28T16:31:33.173 INFO:tasks.workunit.client.1.smithi066.stdout:1/830: write d2/f5 [4943280,109810] 0 2024-02-28T16:31:33.176 INFO:tasks.workunit.client.1.smithi066.stdout:2/847: dread d0/d12/d1a/d39/d7c/d87/f94 [0,4194304] 0 2024-02-28T16:31:33.181 INFO:tasks.workunit.client.1.smithi066.stdout:7/764: rename dc/d29/d92/da6/dd8/c82 to dc/d29/d92/da6/dd8/d37/cf6 0 2024-02-28T16:31:33.183 INFO:tasks.workunit.client.1.smithi066.stdout:0/954: write d2/d27/d41/d76/d93/d10/d23/d6d/d113/fe1 [1704171,109651] 0 2024-02-28T16:31:33.185 INFO:tasks.workunit.client.1.smithi066.stdout:3/971: truncate db/d13/d14/d2d/de3/f13e 748145 0 2024-02-28T16:31:33.189 INFO:tasks.workunit.client.1.smithi066.stdout:8/898: dwrite d0/f13 [0,4194304] 0 2024-02-28T16:31:33.191 INFO:tasks.workunit.client.1.smithi066.stdout:0/955: read - d2/d27/d41/d76/d93/f106 zero size 2024-02-28T16:31:33.192 INFO:tasks.workunit.client.1.smithi066.stdout:1/831: unlink d2/d18/d7a/db1/fea 0 2024-02-28T16:31:33.196 INFO:tasks.workunit.client.1.smithi066.stdout:6/835: link d2/da/d19/d9c/d3d/f7f d2/d2c/da1/d5d/d63/d6a/d100/f126 0 2024-02-28T16:31:33.197 INFO:tasks.workunit.client.1.smithi066.stdout:0/956: write d2/d27/d41/d76/d93/d10/d23/f7a [557121,66781] 0 2024-02-28T16:31:33.209 INFO:tasks.workunit.client.1.smithi066.stdout:3/972: unlink db/dc/d11/d1f/d78/lfe 0 2024-02-28T16:31:33.213 INFO:tasks.workunit.client.1.smithi066.stdout:5/742: dwrite d3/d4/dc/dd/d41/d7a/dac/f73 [0,4194304] 0 2024-02-28T16:31:33.219 INFO:tasks.workunit.client.1.smithi066.stdout:6/836: mkdir d2/da/d19/d75/da6/d127 0 2024-02-28T16:31:33.220 INFO:tasks.workunit.client.1.smithi066.stdout:2/848: link d0/d12/d3e/d9e/fb2 d0/d12/d1a/df8/f10f 0 2024-02-28T16:31:33.226 INFO:tasks.workunit.client.1.smithi066.stdout:2/849: write d0/ff0 [1657380,97410] 0 2024-02-28T16:31:33.226 INFO:tasks.workunit.client.1.smithi066.stdout:9/663: dwrite da/fb [0,4194304] 0 2024-02-28T16:31:33.227 INFO:tasks.workunit.client.1.smithi066.stdout:3/973: creat db/dc/db3/f15e x:0 0 0 2024-02-28T16:31:33.231 INFO:tasks.workunit.client.1.smithi066.stdout:5/743: creat d3/d4/dc/dd/d41/d7a/dac/d60/ff0 x:0 0 0 2024-02-28T16:31:33.232 INFO:tasks.workunit.client.1.smithi066.stdout:1/832: creat d2/d27/d2b/d43/d46/d7c/f116 x:0 0 0 2024-02-28T16:31:33.232 INFO:tasks.workunit.client.1.smithi066.stdout:9/664: stat da/d22/d46/d56/d8f/d28/d42/c79 0 2024-02-28T16:31:33.234 INFO:tasks.workunit.client.1.smithi066.stdout:6/837: symlink d2/da/d19/d75/da6/daa/l128 0 2024-02-28T16:31:33.235 INFO:tasks.workunit.client.1.smithi066.stdout:8/899: link d0/db/d12/d38/ffd d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/d106/d118/f129 0 2024-02-28T16:31:33.240 INFO:tasks.workunit.client.1.smithi066.stdout:3/974: mkdir db/dc/db3/d15f 0 2024-02-28T16:31:33.242 INFO:tasks.workunit.client.1.smithi066.stdout:5/744: write d3/d4/dc/dd/d2e/d18/f1a [206156,65890] 0 2024-02-28T16:31:33.247 INFO:tasks.workunit.client.1.smithi066.stdout:1/833: creat d2/d27/d2b/d34/da9/f117 x:0 0 0 2024-02-28T16:31:33.247 INFO:tasks.workunit.client.1.smithi066.stdout:7/765: truncate f5 2813318 0 2024-02-28T16:31:33.247 INFO:tasks.workunit.client.1.smithi066.stdout:9/665: mkdir da/d6b/dd8 0 2024-02-28T16:31:33.247 INFO:tasks.workunit.client.1.smithi066.stdout:0/957: write d2/d27/d41/d76/d93/f75 [6177909,126211] 0 2024-02-28T16:31:33.249 INFO:tasks.workunit.client.1.smithi066.stdout:6/838: rename d2/d26/d32/d46/f60 to d2/da/d19/d75/da6/d127/f129 0 2024-02-28T16:31:33.254 INFO:tasks.workunit.client.1.smithi066.stdout:5/745: dread - d3/d4/dc/dd/d41/da2/f9b zero size 2024-02-28T16:31:33.258 INFO:tasks.workunit.client.1.smithi066.stdout:3/975: write db/d2a/d44/d48/d52/d77/f146 [197153,130439] 0 2024-02-28T16:31:33.261 INFO:tasks.workunit.client.1.smithi066.stdout:6/839: mknod d2/da/d19/d75/da6/c12a 0 2024-02-28T16:31:33.263 INFO:tasks.workunit.client.1.smithi066.stdout:2/850: write d0/d12/d1a/d39/d7c/fbe [5195627,63061] 0 2024-02-28T16:31:33.264 INFO:tasks.workunit.client.1.smithi066.stdout:6/840: chown d2/da/dcb 141 1 2024-02-28T16:31:33.271 INFO:tasks.workunit.client.1.smithi066.stdout:5/746: rmdir d3/d4/dc/dd/d41/d5d 39 2024-02-28T16:31:33.281 INFO:tasks.workunit.client.1.smithi066.stdout:6/841: rmdir d2/d26/dbe 39 2024-02-28T16:31:33.282 INFO:tasks.workunit.client.1.smithi066.stdout:1/834: dwrite d2/d27/d2b/d34/da9/de9/f9f [0,4194304] 0 2024-02-28T16:31:33.284 INFO:tasks.workunit.client.1.smithi066.stdout:5/747: mkdir d3/d1b/db3/df1 0 2024-02-28T16:31:33.287 INFO:tasks.workunit.client.1.smithi066.stdout:9/666: getdents da/d22/d46/d56/d8f/d2d/d81/dd1/d90/d98 0 2024-02-28T16:31:33.294 INFO:tasks.workunit.client.1.smithi066.stdout:8/900: dwrite d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/fa5 [0,4194304] 0 2024-02-28T16:31:33.294 INFO:tasks.workunit.client.1.smithi066.stdout:2/851: link d0/d12/d2e/d7a/l6c d0/d12/d2e/d7a/l110 0 2024-02-28T16:31:33.299 INFO:tasks.workunit.client.1.smithi066.stdout:1/835: write d2/d27/d2b/d34/da9/de9/f98 [886767,15259] 0 2024-02-28T16:31:33.300 INFO:tasks.workunit.client.1.smithi066.stdout:7/766: dwrite dc/d29/d92/da6/dd8/d1e/f59 [4194304,4194304] 0 2024-02-28T16:31:33.301 INFO:tasks.workunit.client.1.smithi066.stdout:0/958: link d2/d27/d41/d76/d93/db2/db5/d51/f70 d2/d27/d41/d76/d93/db2/db5/d51/f13b 0 2024-02-28T16:31:33.308 INFO:tasks.workunit.client.1.smithi066.stdout:5/748: fdatasync d3/d4/dc/f8b 0 2024-02-28T16:31:33.310 INFO:tasks.workunit.client.1.smithi066.stdout:9/667: unlink da/d22/d46/d56/d8f/d28/d42/fcc 0 2024-02-28T16:31:33.310 INFO:tasks.workunit.client.1.smithi066.stdout:3/976: dread db/d13/d14/ff3 [0,4194304] 0 2024-02-28T16:31:33.310 INFO:tasks.workunit.client.1.smithi066.stdout:7/767: chown dc/d29/d92/da6/dd8/d91/cde 232 1 2024-02-28T16:31:33.311 INFO:tasks.workunit.client.1.smithi066.stdout:1/836: truncate d2/d18/d7a/dbc/fde 567104 0 2024-02-28T16:31:33.315 INFO:tasks.workunit.client.1.smithi066.stdout:1/837: chown d2/d18/d7a/dfd/d10a 8785429 1 2024-02-28T16:31:33.316 INFO:tasks.workunit.client.1.smithi066.stdout:3/977: stat db/d13/d14/d2d/c15c 0 2024-02-28T16:31:33.323 INFO:tasks.workunit.client.1.smithi066.stdout:1/838: chown d2/d18/d7a/dfd/d10a/f10f 260513 1 2024-02-28T16:31:33.325 INFO:tasks.workunit.client.1.smithi066.stdout:6/842: getdents d2/da/dcc 0 2024-02-28T16:31:33.326 INFO:tasks.workunit.client.1.smithi066.stdout:7/768: dread - dc/de/d87/f66 zero size 2024-02-28T16:31:33.327 INFO:tasks.workunit.client.1.smithi066.stdout:2/852: write d0/d12/d2e/d3b/f64 [11289067,46384] 0 2024-02-28T16:31:33.331 INFO:tasks.workunit.client.1.smithi066.stdout:0/959: rename d2/d27/d41/d76/d93/db2/db5/d51/c111 to d2/d27/d41/d76/dcf/d11f/c13c 0 2024-02-28T16:31:33.331 INFO:tasks.workunit.client.1.smithi066.stdout:3/978: creat db/d2a/d44/dd8/f160 x:0 0 0 2024-02-28T16:31:33.335 INFO:tasks.workunit.client.1.smithi066.stdout:2/853: fdatasync d0/d12/d2e/d7a/fc0 0 2024-02-28T16:31:33.335 INFO:tasks.workunit.client.1.smithi066.stdout:1/839: unlink d2/d27/dc7/d51/f99 0 2024-02-28T16:31:33.335 INFO:tasks.workunit.client.1.smithi066.stdout:6/843: fdatasync d2/d2c/da1/d5d/d99/f90 0 2024-02-28T16:31:33.338 INFO:tasks.workunit.client.1.smithi066.stdout:8/901: dwrite d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/fa5 [0,4194304] 0 2024-02-28T16:31:33.338 INFO:tasks.workunit.client.1.smithi066.stdout:7/769: chown dc/d29/d92/da6/dd8/d1e/d45/f95 0 1 2024-02-28T16:31:33.340 INFO:tasks.workunit.client.1.smithi066.stdout:5/749: rename d3/d4/f61 to d3/d1b/db3/dec/ff2 0 2024-02-28T16:31:33.341 INFO:tasks.workunit.client.1.smithi066.stdout:7/770: stat dc/f5d 0 2024-02-28T16:31:33.344 INFO:tasks.workunit.client.1.smithi066.stdout:5/750: write d3/d4/f4e [3039439,60592] 0 2024-02-28T16:31:33.347 INFO:tasks.workunit.client.1.smithi066.stdout:0/960: mkdir d2/d27/d41/d76/d93/df3/d13d 0 2024-02-28T16:31:33.350 INFO:tasks.workunit.client.1.smithi066.stdout:8/902: unlink d0/d6/dc/d16/f50 0 2024-02-28T16:31:33.350 INFO:tasks.workunit.client.1.smithi066.stdout:6/844: mknod d2/d2c/da1/d5d/d63/c12b 0 2024-02-28T16:31:33.351 INFO:tasks.workunit.client.1.smithi066.stdout:7/771: chown dc/d29/d92/da6/dd8/d1e/f3c 129893192 1 2024-02-28T16:31:33.354 INFO:tasks.workunit.client.1.smithi066.stdout:2/854: truncate d0/d12/d3e/d9e/fed 1231447 0 2024-02-28T16:31:33.356 INFO:tasks.workunit.client.1.smithi066.stdout:7/772: rmdir dc/d29 39 2024-02-28T16:31:33.360 INFO:tasks.workunit.client.1.smithi066.stdout:3/979: dread db/dc/d11/d1f/d78/fa1 [0,4194304] 0 2024-02-28T16:31:33.365 INFO:tasks.workunit.client.1.smithi066.stdout:0/961: rename d2/d27/d41/d76/d93/c13 to d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/c13e 0 2024-02-28T16:31:33.370 INFO:tasks.workunit.client.1.smithi066.stdout:2/855: truncate d0/d12/f82 5012579 0 2024-02-28T16:31:33.378 INFO:tasks.workunit.client.1.smithi066.stdout:7/773: dread - dc/d29/d92/da6/dd8/d1e/da1/fc9 zero size 2024-02-28T16:31:33.378 INFO:tasks.workunit.client.1.smithi066.stdout:3/980: creat db/dc/d35/d3d/f161 x:0 0 0 2024-02-28T16:31:33.382 INFO:tasks.workunit.client.1.smithi066.stdout:9/668: dread da/d3a/f41 [0,4194304] 0 2024-02-28T16:31:33.389 INFO:tasks.workunit.client.1.smithi066.stdout:3/981: chown db/dc/d11/c14d 728929998 1 2024-02-28T16:31:33.390 INFO:tasks.workunit.client.1.smithi066.stdout:8/903: dwrite d0/db/d12/d1d/d25/d4f/f82 [0,4194304] 0 2024-02-28T16:31:33.397 INFO:tasks.workunit.client.1.smithi066.stdout:6/845: dread d2/d2c/da1/d5d/d63/fc2 [0,4194304] 0 2024-02-28T16:31:33.398 INFO:tasks.workunit.client.1.smithi066.stdout:5/751: dread d3/d4/dc/dd/d41/d7a/dac/f73 [0,4194304] 0 2024-02-28T16:31:33.402 INFO:tasks.workunit.client.1.smithi066.stdout:7/774: mkdir dc/d29/d92/da6/dd8/d37/df7 0 2024-02-28T16:31:33.408 INFO:tasks.workunit.client.1.smithi066.stdout:0/962: rename d2/d27/d41/d76/cdb to d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/d122/c13f 0 2024-02-28T16:31:33.411 INFO:tasks.workunit.client.1.smithi066.stdout:5/752: chown d3/d4/dc/dd/d41/d5d/f69 146 1 2024-02-28T16:31:33.416 INFO:tasks.workunit.client.1.smithi066.stdout:7/775: write dc/d29/d92/da6/dd8/d1e/d45/f79 [4029359,61654] 0 2024-02-28T16:31:33.416 INFO:tasks.workunit.client.1.smithi066.stdout:0/963: read d2/d27/d41/d76/d93/d139/f81 [1316953,103811] 0 2024-02-28T16:31:33.416 INFO:tasks.workunit.client.1.smithi066.stdout:8/904: rename d0/db/d12/d1d/l76 to d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db1/dad/l12a 0 2024-02-28T16:31:33.424 INFO:tasks.workunit.client.1.smithi066.stdout:5/753: rename d3/d1b/d39/f44 to d3/d1b/d20/de9/ff3 0 2024-02-28T16:31:33.428 INFO:tasks.workunit.client.1.smithi066.stdout:6/846: dwrite d2/d2c/dba/f10a [0,4194304] 0 2024-02-28T16:31:33.436 INFO:tasks.workunit.client.1.smithi066.stdout:7/776: write dc/f4b [935729,88356] 0 2024-02-28T16:31:33.436 INFO:tasks.workunit.client.1.smithi066.stdout:0/964: chown d2/d27/d41/d76/d93/d33/d3f/d8e/cf2 5 1 2024-02-28T16:31:33.445 INFO:tasks.workunit.client.1.smithi066.stdout:6/847: chown d2/da/d19/d9c/f2f 2667098 1 2024-02-28T16:31:33.449 INFO:tasks.workunit.client.1.smithi066.stdout:2/856: dwrite d0/d12/f82 [0,4194304] 0 2024-02-28T16:31:33.453 INFO:tasks.workunit.client.1.smithi066.stdout:5/754: dread - d3/d4/dc/dd/d41/d7f/d80/fc9 zero size 2024-02-28T16:31:33.454 INFO:tasks.workunit.client.1.smithi066.stdout:7/777: mkdir dc/d29/d92/da6/dd8/d91/dba/df8 0 2024-02-28T16:31:33.455 INFO:tasks.workunit.client.1.smithi066.stdout:1/840: dread d2/f5 [0,4194304] 0 2024-02-28T16:31:33.461 INFO:tasks.workunit.client.1.smithi066.stdout:9/669: dwrite da/d22/d46/d56/d8f/d2d/f62 [0,4194304] 0 2024-02-28T16:31:33.466 INFO:tasks.workunit.client.1.smithi066.stdout:0/965: rename d2/d27/d41/d76/d93/d10/d23/f12d to d2/d85/da9/f140 0 2024-02-28T16:31:33.468 INFO:tasks.workunit.client.1.smithi066.stdout:5/755: unlink d3/c7b 0 2024-02-28T16:31:33.471 INFO:tasks.workunit.client.1.smithi066.stdout:2/857: dread - d0/d12/d1a/d23/ddc/fff zero size 2024-02-28T16:31:33.472 INFO:tasks.workunit.client.1.smithi066.stdout:3/982: dwrite db/dc/f2b [0,4194304] 0 2024-02-28T16:31:33.482 INFO:tasks.workunit.client.1.smithi066.stdout:9/670: rmdir da/d4b/d53/dba 39 2024-02-28T16:31:33.482 INFO:tasks.workunit.client.1.smithi066.stdout:2/858: stat d0/d12/d1a/d23/ddc/fff 0 2024-02-28T16:31:33.486 INFO:tasks.workunit.client.1.smithi066.stdout:1/841: getdents d2/d27/d2b/d43/d46 0 2024-02-28T16:31:33.495 INFO:tasks.workunit.client.1.smithi066.stdout:5/756: mknod d3/d1b/db3/dec/cf4 0 2024-02-28T16:31:33.496 INFO:tasks.workunit.client.1.smithi066.stdout:8/905: dread d0/d6/dc/d16/d4c/d6c/d6f/da7/fcc [0,4194304] 0 2024-02-28T16:31:33.502 INFO:tasks.workunit.client.1.smithi066.stdout:2/859: rename d0/d12/d2e/d7a/d106/d108 to d0/d12/d3e/d9e/dce/d111 0 2024-02-28T16:31:33.509 INFO:tasks.workunit.client.1.smithi066.stdout:6/848: link d2/d2c/d72/caf d2/da/d19/d75/da6/d127/c12c 0 2024-02-28T16:31:33.509 INFO:tasks.workunit.client.1.smithi066.stdout:6/849: stat d2/da/d19/l4b 0 2024-02-28T16:31:33.516 INFO:tasks.workunit.client.1.smithi066.stdout:5/757: unlink d3/d1b/d39/fa6 0 2024-02-28T16:31:33.523 INFO:tasks.workunit.client.1.smithi066.stdout:3/983: dwrite db/d2a/d44/f15d [0,4194304] 0 2024-02-28T16:31:33.524 INFO:tasks.workunit.client.1.smithi066.stdout:8/906: mknod d0/db/d12/d1d/d25/d65/c12b 0 2024-02-28T16:31:33.532 INFO:tasks.workunit.client.1.smithi066.stdout:9/671: mkdir da/d4b/d53/dd9 0 2024-02-28T16:31:33.538 INFO:tasks.workunit.client.1.smithi066.stdout:9/672: write da/d22/d46/d56/d8f/d28/fa5 [74050,48848] 0 2024-02-28T16:31:33.542 INFO:tasks.workunit.client.1.smithi066.stdout:1/842: dwrite d2/d18/d19/ff2 [0,4194304] 0 2024-02-28T16:31:33.554 INFO:tasks.workunit.client.1.smithi066.stdout:6/850: dwrite d2/d2c/da1/d5d/d63/d6a/ffe [0,4194304] 0 2024-02-28T16:31:33.556 INFO:tasks.workunit.client.1.smithi066.stdout:2/860: mknod d0/d12/d1a/d23/d29/c112 0 2024-02-28T16:31:33.559 INFO:tasks.workunit.client.1.smithi066.stdout:5/758: creat d3/d4/dc/dd/d41/d7f/ff5 x:0 0 0 2024-02-28T16:31:33.567 INFO:tasks.workunit.client.1.smithi066.stdout:5/759: fsync d3/d4/dc/dd/d2e/f6f 0 2024-02-28T16:31:33.581 INFO:tasks.workunit.client.1.smithi066.stdout:7/778: dwrite dc/d29/d92/da6/dd8/d1e/f24 [0,4194304] 0 2024-02-28T16:31:33.585 INFO:tasks.workunit.client.1.smithi066.stdout:0/966: dwrite d2/d27/d41/d76/d93/d139/f81 [0,4194304] 0 2024-02-28T16:31:33.590 INFO:tasks.workunit.client.1.smithi066.stdout:8/907: dwrite d0/db/d12/d1d/fd0 [0,4194304] 0 2024-02-28T16:31:33.593 INFO:tasks.workunit.client.1.smithi066.stdout:7/779: rmdir dc/d56 39 2024-02-28T16:31:33.595 INFO:tasks.workunit.client.1.smithi066.stdout:3/984: getdents db/dc/d11/d1f/d115/d13c 0 2024-02-28T16:31:33.598 INFO:tasks.workunit.client.1.smithi066.stdout:2/861: write d0/d12/d2e/d7a/f55 [5196206,116747] 0 2024-02-28T16:31:33.603 INFO:tasks.workunit.client.1.smithi066.stdout:0/967: creat d2/d27/d41/df4/d123/f141 x:0 0 0 2024-02-28T16:31:33.604 INFO:tasks.workunit.client.1.smithi066.stdout:1/843: truncate d2/d27/d2b/d34/d9e/fd2 379746 0 2024-02-28T16:31:33.605 INFO:tasks.workunit.client.1.smithi066.stdout:8/908: mkdir d0/d6/d12c 0 2024-02-28T16:31:33.610 INFO:tasks.workunit.client.1.smithi066.stdout:3/985: unlink db/d13/d14/l8c 0 2024-02-28T16:31:33.612 INFO:tasks.workunit.client.1.smithi066.stdout:5/760: dwrite d3/d4/dc/dd/d41/d7f/ff5 [0,4194304] 0 2024-02-28T16:31:33.622 INFO:tasks.workunit.client.1.smithi066.stdout:8/909: write d0/d6/dc/d16/d1a/d63/d58/dac/fb4 [4001984,120668] 0 2024-02-28T16:31:33.623 INFO:tasks.workunit.client.1.smithi066.stdout:6/851: dwrite d2/da/d19/d9c/d3d/f89 [0,4194304] 0 2024-02-28T16:31:33.629 INFO:tasks.workunit.client.1.smithi066.stdout:7/780: rename dc/d56/cbd to dc/d29/d92/da6/dd8/d91/dba/df8/cf9 0 2024-02-28T16:31:33.631 INFO:tasks.workunit.client.1.smithi066.stdout:0/968: mkdir d2/d27/d41/d76/d93/d10/d23/db6/de0/d142 0 2024-02-28T16:31:33.632 INFO:tasks.workunit.client.1.smithi066.stdout:8/910: fdatasync d0/d6/dc/d16/d4c/d6c/d6f/da7/fc5 0 2024-02-28T16:31:33.635 INFO:tasks.workunit.client.1.smithi066.stdout:5/761: mknod d3/d4/dc/dd/d41/d5d/cf6 0 2024-02-28T16:31:33.638 INFO:tasks.workunit.client.1.smithi066.stdout:2/862: truncate d0/d12/d1a/d23/d29/d21/fb0 113952 0 2024-02-28T16:31:33.648 INFO:tasks.workunit.client.1.smithi066.stdout:6/852: read d2/da/f1e [1916918,113460] 0 2024-02-28T16:31:33.648 INFO:tasks.workunit.client.1.smithi066.stdout:1/844: rmdir d2/d18/d7a/d8d/d104 0 2024-02-28T16:31:33.651 INFO:tasks.workunit.client.1.smithi066.stdout:7/781: chown dc/d29/d92/da6/dd8/d1e/f53 1755735 1 2024-02-28T16:31:33.653 INFO:tasks.workunit.client.1.smithi066.stdout:7/782: dread - dc/d6e/dc8/fcc zero size 2024-02-28T16:31:33.656 INFO:tasks.workunit.client.1.smithi066.stdout:0/969: symlink d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/d122/l143 0 2024-02-28T16:31:33.657 INFO:tasks.workunit.client.1.smithi066.stdout:3/986: rename db/dc/d11/d1f/d78/cf8 to db/dc/d35/c162 0 2024-02-28T16:31:33.660 INFO:tasks.workunit.client.1.smithi066.stdout:6/853: dread - d2/d2c/da1/f91 zero size 2024-02-28T16:31:33.663 INFO:tasks.workunit.client.1.smithi066.stdout:7/783: truncate dc/d29/f44 239091 0 2024-02-28T16:31:33.669 INFO:tasks.workunit.client.1.smithi066.stdout:5/762: write d3/d4/f83 [157833,124945] 0 2024-02-28T16:31:33.669 INFO:tasks.workunit.client.1.smithi066.stdout:1/845: fsync d2/f1d 0 2024-02-28T16:31:33.672 INFO:tasks.workunit.client.1.smithi066.stdout:6/854: creat d2/d2c/da1/d5d/d63/f12d x:0 0 0 2024-02-28T16:31:33.683 INFO:tasks.workunit.client.1.smithi066.stdout:5/763: dread d3/d4/fb [0,4194304] 0 2024-02-28T16:31:33.689 INFO:tasks.workunit.client.1.smithi066.stdout:0/970: dwrite d2/d27/d41/d76/d93/d33/f62 [4194304,4194304] 0 2024-02-28T16:31:33.708 INFO:tasks.workunit.client.1.smithi066.stdout:8/911: mkdir d0/d6/dc/d16/d1a/d63/d58/d5a/d12d 0 2024-02-28T16:31:33.710 INFO:tasks.workunit.client.1.smithi066.stdout:2/863: rename d0/dd7/l109 to d0/d12/d1a/d23/ddc/l113 0 2024-02-28T16:31:33.711 INFO:tasks.workunit.client.1.smithi066.stdout:1/846: creat d2/d18/d6b/dba/de1/f118 x:0 0 0 2024-02-28T16:31:33.712 INFO:tasks.workunit.client.1.smithi066.stdout:7/784: truncate dc/d29/d92/da6/dd8/d1e/f3c 45224 0 2024-02-28T16:31:33.713 INFO:tasks.workunit.client.1.smithi066.stdout:1/847: read - d2/d18/da5/f101 zero size 2024-02-28T16:31:33.713 INFO:tasks.workunit.client.1.smithi066.stdout:5/764: creat d3/d4/dc/dd/de6/ff7 x:0 0 0 2024-02-28T16:31:33.716 INFO:tasks.workunit.client.1.smithi066.stdout:1/848: write d2/d27/f3c [208356,126484] 0 2024-02-28T16:31:33.719 INFO:tasks.workunit.client.1.smithi066.stdout:0/971: creat d2/d27/d41/d76/d93/f144 x:0 0 0 2024-02-28T16:31:33.719 INFO:tasks.workunit.client.1.smithi066.stdout:2/864: fdatasync d0/f1f 0 2024-02-28T16:31:33.720 INFO:tasks.workunit.client.1.smithi066.stdout:6/855: truncate d2/da/d19/d9c/d3d/f7f 3827100 0 2024-02-28T16:31:33.724 INFO:tasks.workunit.client.1.smithi066.stdout:7/785: creat dc/d29/d92/da6/dd8/d91/dba/dbb/ffa x:0 0 0 2024-02-28T16:31:33.731 INFO:tasks.workunit.client.1.smithi066.stdout:0/972: truncate d2/d27/d41/d76/d93/db2/db5/d73/fe8 2088470 0 2024-02-28T16:31:33.733 INFO:tasks.workunit.client.1.smithi066.stdout:7/786: symlink dc/d29/d92/df2/lfb 0 2024-02-28T16:31:33.735 INFO:tasks.workunit.client.1.smithi066.stdout:2/865: rename d0/d12/d1a/d39/d7c/d87/fe5 to d0/d12/d1a/d39/d7c/d87/d104/f114 0 2024-02-28T16:31:33.737 INFO:tasks.workunit.client.1.smithi066.stdout:0/973: fdatasync d2/d27/d41/d76/d93/d115/f13a 0 2024-02-28T16:31:33.738 INFO:tasks.workunit.client.1.smithi066.stdout:7/787: write dc/d56/d97/dd5/fe8 [312335,98461] 0 2024-02-28T16:31:33.744 INFO:tasks.workunit.client.1.smithi066.stdout:7/788: stat dc/d6e/c73 0 2024-02-28T16:31:33.744 INFO:tasks.workunit.client.1.smithi066.stdout:5/765: dwrite d3/d1b/d20/ddd/fb9 [0,4194304] 0 2024-02-28T16:31:33.746 INFO:tasks.workunit.client.1.smithi066.stdout:1/849: dread d2/d18/d19/f89 [0,4194304] 0 2024-02-28T16:31:33.746 INFO:tasks.workunit.client.1.smithi066.stdout:0/974: read d2/d27/d41/d76/d93/d10/d23/f7a [280654,11111] 0 2024-02-28T16:31:33.749 INFO:tasks.workunit.client.1.smithi066.stdout:5/766: readlink d3/d4/dc/dd/d2e/d8f/le0 0 2024-02-28T16:31:33.752 INFO:tasks.workunit.client.1.smithi066.stdout:1/850: chown d2/d27/d2b/d43/d46/fcd 14717156 1 2024-02-28T16:31:33.753 INFO:tasks.workunit.client.1.smithi066.stdout:7/789: mkdir dc/d29/d92/df2/dfc 0 2024-02-28T16:31:33.754 INFO:tasks.workunit.client.1.smithi066.stdout:5/767: write d3/d4/dc/dd/d2e/f38 [4482283,39801] 0 2024-02-28T16:31:33.770 INFO:tasks.workunit.client.1.smithi066.stdout:2/866: write d0/d12/d1a/d23/d29/d21/d2f/f9c [679374,11202] 0 2024-02-28T16:31:33.771 INFO:tasks.workunit.client.1.smithi066.stdout:2/867: readlink d0/d12/d2e/l9f 0 2024-02-28T16:31:33.771 INFO:tasks.workunit.client.1.smithi066.stdout:1/851: symlink d2/d27/d2b/d43/d46/l119 0 2024-02-28T16:31:33.771 INFO:tasks.workunit.client.1.smithi066.stdout:6/856: dread d2/d26/f57 [0,4194304] 0 2024-02-28T16:31:33.851 INFO:tasks.workunit.client.1.smithi066.stdout:5/768: getdents d3/d1b/d39 0 2024-02-28T16:31:33.851 INFO:tasks.workunit.client.1.smithi066.stdout:3/987: dwrite db/d2a/d44/d48/d52/d77/f74 [0,4194304] 0 2024-02-28T16:31:33.852 INFO:tasks.workunit.client.1.smithi066.stdout:2/868: mknod d0/dd7/c115 0 2024-02-28T16:31:33.860 INFO:tasks.workunit.client.1.smithi066.stdout:2/869: readlink d0/d12/l1b 0 2024-02-28T16:31:33.863 INFO:tasks.workunit.client.1.smithi066.stdout:1/852: dread d2/d27/dc7/d51/f83 [0,4194304] 0 2024-02-28T16:31:33.863 INFO:tasks.workunit.client.1.smithi066.stdout:3/988: fdatasync db/d13/d14/d84/f112 0 2024-02-28T16:31:33.872 INFO:tasks.workunit.client.1.smithi066.stdout:5/769: dread - d3/d4/dc/dd/d41/d7f/d80/fc9 zero size 2024-02-28T16:31:33.873 INFO:tasks.workunit.client.1.smithi066.stdout:1/853: read - d2/d27/d2b/d34/fac zero size 2024-02-28T16:31:33.873 INFO:tasks.workunit.client.1.smithi066.stdout:7/790: dwrite dc/d56/d97/ff1 [0,4194304] 0 2024-02-28T16:31:33.877 INFO:tasks.workunit.client.1.smithi066.stdout:5/770: write d3/d4/f71 [667800,105198] 0 2024-02-28T16:31:33.879 INFO:tasks.workunit.client.1.smithi066.stdout:0/975: dwrite d2/f28 [0,4194304] 0 2024-02-28T16:31:33.881 INFO:tasks.workunit.client.1.smithi066.stdout:1/854: write d2/d27/d2b/d43/faf [1179807,104150] 0 2024-02-28T16:31:33.883 INFO:tasks.workunit.client.1.smithi066.stdout:7/791: mknod dc/d29/d92/da6/dd8/d91/cfd 0 2024-02-28T16:31:33.890 INFO:tasks.workunit.client.1.smithi066.stdout:5/771: mknod d3/d4/cf8 0 2024-02-28T16:31:33.891 INFO:tasks.workunit.client.1.smithi066.stdout:7/792: chown dc/d29/d92/da6/lb0 0 1 2024-02-28T16:31:33.892 INFO:tasks.workunit.client.1.smithi066.stdout:3/989: link db/d13/d14/d2d/d92/laf db/d2a/d142/l163 0 2024-02-28T16:31:33.895 INFO:tasks.workunit.client.1.smithi066.stdout:7/793: chown dc/d29/d92/da6/dd8/d1e/l8c 13782292 1 2024-02-28T16:31:33.903 INFO:tasks.workunit.client.1.smithi066.stdout:0/976: creat d2/d27/d41/d76/d93/db2/f145 x:0 0 0 2024-02-28T16:31:33.904 INFO:tasks.workunit.client.1.smithi066.stdout:6/857: dread d2/da/d19/f20 [0,4194304] 0 2024-02-28T16:31:33.907 INFO:tasks.workunit.client.1.smithi066.stdout:7/794: truncate dc/d29/d92/da6/dd8/d91/dba/dbb/ffa 217647 0 2024-02-28T16:31:33.910 INFO:tasks.workunit.client.1.smithi066.stdout:6/858: dread - d2/da/d19/d75/da6/ff9 zero size 2024-02-28T16:31:33.911 INFO:tasks.workunit.client.1.smithi066.stdout:3/990: dread db/dc/d11/d1f/f11d [0,4194304] 0 2024-02-28T16:31:33.913 INFO:tasks.workunit.client.1.smithi066.stdout:1/855: dread d2/d27/d2b/d43/faf [0,4194304] 0 2024-02-28T16:31:33.917 INFO:tasks.workunit.client.1.smithi066.stdout:0/977: rename d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/f72 to d2/d27/d41/d76/dcf/d11f/f146 0 2024-02-28T16:31:33.920 INFO:tasks.workunit.client.1.smithi066.stdout:0/978: fsync d2/d27/d41/d76/d93/db2/fe6 0 2024-02-28T16:31:33.924 INFO:tasks.workunit.client.1.smithi066.stdout:6/859: dread d2/d26/f31 [0,4194304] 0 2024-02-28T16:31:33.924 INFO:tasks.workunit.client.1.smithi066.stdout:7/795: link dc/d29/d92/da6/dd8/f7c dc/d29/deb/ffe 0 2024-02-28T16:31:33.931 INFO:tasks.workunit.client.1.smithi066.stdout:6/860: symlink d2/da/dcc/l12e 0 2024-02-28T16:31:33.933 INFO:tasks.workunit.client.1.smithi066.stdout:0/979: getdents d2/d27/d41/d76/d93/df3 0 2024-02-28T16:31:33.934 INFO:tasks.workunit.client.1.smithi066.stdout:7/796: getdents dc/d56/dc3/df4 0 2024-02-28T16:31:33.937 INFO:tasks.workunit.client.1.smithi066.stdout:1/856: dwrite d2/d27/d2b/d34/da9/de9/f9f [4194304,4194304] 0 2024-02-28T16:31:33.946 INFO:tasks.workunit.client.1.smithi066.stdout:0/980: creat d2/d27/f147 x:0 0 0 2024-02-28T16:31:33.954 INFO:tasks.workunit.client.1.smithi066.stdout:1/857: rmdir d2/dd0 0 2024-02-28T16:31:33.957 INFO:tasks.workunit.client.1.smithi066.stdout:1/858: chown d2/d27/d2b/d43/d46/l119 36 1 2024-02-28T16:31:33.965 INFO:tasks.workunit.client.1.smithi066.stdout:1/859: mknod d2/d27/dc7/d54/de7/c11a 0 2024-02-28T16:31:33.967 INFO:tasks.workunit.client.1.smithi066.stdout:6/861: dwrite d2/d26/fa3 [0,4194304] 0 2024-02-28T16:31:33.968 INFO:tasks.workunit.client.1.smithi066.stdout:7/797: dread dc/d29/d92/da6/dd8/d1e/f59 [0,4194304] 0 2024-02-28T16:31:33.972 INFO:tasks.workunit.client.1.smithi066.stdout:1/860: write d2/d27/d2b/d34/da9/de9/f98 [444911,40526] 0 2024-02-28T16:31:33.976 INFO:tasks.workunit.client.1.smithi066.stdout:6/862: creat d2/da/d19/d75/da6/daa/f12f x:0 0 0 2024-02-28T16:31:33.978 INFO:tasks.workunit.client.1.smithi066.stdout:1/861: creat d2/d18/d7a/dbc/f11b x:0 0 0 2024-02-28T16:31:33.980 INFO:tasks.workunit.client.1.smithi066.stdout:7/798: rename dc/d56/d97/dd5 to dc/d56/dce/dff 0 2024-02-28T16:31:33.981 INFO:tasks.workunit.client.1.smithi066.stdout:6/863: write d2/da/d19/d75/da6/ff9 [271197,86000] 0 2024-02-28T16:31:33.984 INFO:tasks.workunit.client.1.smithi066.stdout:1/862: mkdir d2/d27/dc7/de4/d11c 0 2024-02-28T16:31:33.986 INFO:tasks.workunit.client.1.smithi066.stdout:0/981: dread d2/d27/d41/d76/d93/d33/d3f/fbb [0,4194304] 0 2024-02-28T16:31:33.991 INFO:tasks.workunit.client.1.smithi066.stdout:6/864: truncate d2/d26/dbe/fec 810579 0 2024-02-28T16:31:33.996 INFO:tasks.workunit.client.1.smithi066.stdout:6/865: mknod d2/da/dcc/dd6/c130 0 2024-02-28T16:31:34.005 INFO:tasks.workunit.client.1.smithi066.stdout:6/866: link d2/da/cae d2/da/dcc/dd6/c131 0 2024-02-28T16:31:34.008 INFO:tasks.workunit.client.1.smithi066.stdout:6/867: fdatasync d2/d26/d32/d46/f103 0 2024-02-28T16:31:34.008 INFO:tasks.workunit.client.1.smithi066.stdout:0/982: dwrite d2/d27/d41/d76/d93/f144 [0,4194304] 0 2024-02-28T16:31:34.010 INFO:tasks.workunit.client.1.smithi066.stdout:7/799: dwrite dc/d29/f41 [0,4194304] 0 2024-02-28T16:31:34.018 INFO:tasks.workunit.client.1.smithi066.stdout:6/868: mknod d2/da/d19/d75/dbd/dc6/c132 0 2024-02-28T16:31:34.019 INFO:tasks.workunit.client.1.smithi066.stdout:7/800: mkdir dc/d29/d92/da6/dd8/d37/df7/d100 0 2024-02-28T16:31:34.025 INFO:tasks.workunit.client.1.smithi066.stdout:6/869: rename d2/da/d19/f50 to d2/da/d19/d75/da6/f133 0 2024-02-28T16:31:34.026 INFO:tasks.workunit.client.1.smithi066.stdout:7/801: mknod dc/d6e/dc8/c101 0 2024-02-28T16:31:34.030 INFO:tasks.workunit.client.1.smithi066.stdout:6/870: creat d2/d26/de0/f134 x:0 0 0 2024-02-28T16:31:34.031 INFO:tasks.workunit.client.1.smithi066.stdout:7/802: dread - dc/d29/d92/da6/dd8/d37/d47/f86 zero size 2024-02-28T16:31:34.039 INFO:tasks.workunit.client.1.smithi066.stdout:7/803: rename dc/d29/d92/da6/dd8/d1e/da1 to dc/d56/dce/d102 0 2024-02-28T16:31:34.039 INFO:tasks.workunit.client.1.smithi066.stdout:0/983: dwrite d2/d27/d41/d76/d93/d10/f2c [0,4194304] 0 2024-02-28T16:31:34.045 INFO:tasks.workunit.client.1.smithi066.stdout:7/804: mkdir dc/d29/d92/da6/dd8/d37/df7/d100/d103 0 2024-02-28T16:31:34.047 INFO:tasks.workunit.client.1.smithi066.stdout:0/984: creat d2/d27/d41/d76/d93/df0/d12b/f148 x:0 0 0 2024-02-28T16:31:34.051 INFO:tasks.workunit.client.1.smithi066.stdout:7/805: write dc/d29/d92/da6/dd8/d91/fe2 [590740,65955] 0 2024-02-28T16:31:34.055 INFO:tasks.workunit.client.1.smithi066.stdout:6/871: dwrite d2/da/d19/f4f [4194304,4194304] 0 2024-02-28T16:31:34.056 INFO:tasks.workunit.client.1.smithi066.stdout:7/806: creat dc/d56/dce/d102/f104 x:0 0 0 2024-02-28T16:31:34.056 INFO:tasks.workunit.client.1.smithi066.stdout:0/985: mkdir d2/d27/d41/d76/d93/d10/d23/d6d/d113/d149 0 2024-02-28T16:31:34.060 INFO:tasks.workunit.client.1.smithi066.stdout:7/807: creat dc/d29/d92/da6/dd8/d37/df7/d100/f105 x:0 0 0 2024-02-28T16:31:34.061 INFO:tasks.workunit.client.1.smithi066.stdout:0/986: mkdir d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/dd1/d14a 0 2024-02-28T16:31:34.074 INFO:tasks.workunit.client.1.smithi066.stdout:0/987: link d2/d85/d8d/fd0 d2/d27/d41/d76/d93/d33/d3f/d8e/d6a/dd1/d14a/f14b 0 2024-02-28T16:31:34.078 INFO:tasks.workunit.client.1.smithi066.stdout:0/988: write d2/d27/d41/d76/d93/f6f [3840182,103018] 0 2024-02-28T16:31:34.081 INFO:tasks.workunit.client.1.smithi066.stdout:0/989: readlink d2/d27/d41/d76/d93/db2/l126 0 2024-02-28T16:31:34.084 INFO:tasks.workunit.client.1.smithi066.stdout:0/990: rmdir d2 39 2024-02-28T16:31:34.088 INFO:tasks.workunit.client.1.smithi066.stdout:7/808: dread dc/d29/d92/da6/dd8/d91/dba/fd9 [0,4194304] 0 2024-02-28T16:31:34.088 INFO:tasks.workunit.client.1.smithi066.stdout:6/872: dread d2/da/d19/d75/da6/f133 [0,4194304] 0 2024-02-28T16:31:34.089 INFO:tasks.workunit.client.1.smithi066.stdout:0/991: creat d2/d27/d41/d76/d93/d10/d23/d6d/d113/f14c x:0 0 0 2024-02-28T16:31:34.090 INFO:tasks.workunit.client.1.smithi066.stdout:6/873: mknod d2/d26/c135 0 2024-02-28T16:31:34.094 INFO:tasks.workunit.client.1.smithi066.stdout:0/992: creat d2/d27/d41/d76/d93/d10/d23/f14d x:0 0 0 2024-02-28T16:31:34.094 INFO:tasks.workunit.client.1.smithi066.stdout:7/809: getdents dc/d29/d92/da6/dd8/d91/db9 0 2024-02-28T16:31:34.096 INFO:tasks.workunit.client.1.smithi066.stdout:7/810: fsync dc/d56/dce/d102/fc1 0 2024-02-28T16:31:34.099 INFO:tasks.workunit.client.1.smithi066.stdout:7/811: write dc/de/fa2 [3716720,94623] 0 2024-02-28T16:31:34.101 INFO:tasks.workunit.client.1.smithi066.stdout:7/812: chown dc/d56/dce/d102/fc9 299 1 2024-02-28T16:31:34.104 INFO:tasks.workunit.client.1.smithi066.stdout:7/813: readlink dc/d29/d92/da6/dd8/d1e/d45/l5f 0 2024-02-28T16:31:34.105 INFO:tasks.workunit.client.1.smithi066.stdout:0/993: write d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/de4/fdd [1529681,106966] 0 2024-02-28T16:31:34.105 INFO:tasks.workunit.client.1.smithi066.stdout:7/814: readlink dc/d29/l85 0 2024-02-28T16:31:34.107 INFO:tasks.workunit.client.1.smithi066.stdout:7/815: write f4 [2612541,7160] 0 2024-02-28T16:31:34.107 INFO:tasks.workunit.client.1.smithi066.stdout:0/994: chown d2/d85/d8d/lcb 4891419 1 2024-02-28T16:31:34.108 INFO:tasks.workunit.client.1.smithi066.stdout:6/874: dwrite d2/d2c/f8e [0,4194304] 0 2024-02-28T16:31:34.112 INFO:tasks.workunit.client.1.smithi066.stdout:6/875: write d2/d26/dbe/fec [1545769,9745] 0 2024-02-28T16:31:34.122 INFO:tasks.workunit.client.1.smithi066.stdout:7/816: dread dc/d29/d92/da6/dd8/d91/dba/fd9 [0,4194304] 0 2024-02-28T16:31:34.124 INFO:tasks.workunit.client.1.smithi066.stdout:6/876: link d2/d2c/da1/d5d/d63/d6a/fb2 d2/d2c/da1/d5d/f136 0 2024-02-28T16:31:34.127 INFO:tasks.workunit.client.1.smithi066.stdout:6/877: fsync d2/da/d19/d75/d88/dbc/ffa 0 2024-02-28T16:31:34.128 INFO:tasks.workunit.client.1.smithi066.stdout:6/878: stat d2/da/dcc/dd6/d102/c111 0 2024-02-28T16:31:34.130 INFO:tasks.workunit.client.1.smithi066.stdout:6/879: readlink d2/da/d19/d75/da6/daa/l128 0 2024-02-28T16:31:34.132 INFO:tasks.workunit.client.1.smithi066.stdout:6/880: fdatasync d2/da/d19/d75/dbd/dc6/f116 0 2024-02-28T16:31:34.132 INFO:tasks.workunit.client.1.smithi066.stdout:6/881: readlink d2/da/l61 0 2024-02-28T16:31:34.139 INFO:tasks.workunit.client.1.smithi066.stdout:6/882: write d2/da/d19/fdd [121541,48990] 0 2024-02-28T16:31:34.142 INFO:tasks.workunit.client.1.smithi066.stdout:6/883: creat d2/d2c/da1/dad/f137 x:0 0 0 2024-02-28T16:31:34.143 INFO:tasks.workunit.client.1.smithi066.stdout:7/817: dread dc/de/fa2 [0,4194304] 0 2024-02-28T16:31:34.143 INFO:tasks.workunit.client.1.smithi066.stdout:6/884: readlink d2/d2c/d6e/l9a 0 2024-02-28T16:31:34.144 INFO:tasks.workunit.client.1.smithi066.stdout:7/818: chown dc/c7e 220860096 1 2024-02-28T16:31:34.147 INFO:tasks.workunit.client.1.smithi066.stdout:6/885: rename d2/d26/de0/c120 to d2/da/d19/d75/da6/c138 0 2024-02-28T16:31:34.148 INFO:tasks.workunit.client.1.smithi066.stdout:7/819: mkdir dc/d29/d92/da6/dd8/d91/dbf/d106 0 2024-02-28T16:31:34.154 INFO:tasks.workunit.client.1.smithi066.stdout:7/820: rename dc/d29/d92/da6/lea to dc/d29/d92/da6/dd8/d37/df7/d100/d103/l107 0 2024-02-28T16:31:34.155 INFO:tasks.workunit.client.1.smithi066.stdout:7/821: dread - dc/de/fb8 zero size 2024-02-28T16:31:34.160 INFO:tasks.workunit.client.1.smithi066.stdout:6/886: dwrite d2/d2c/f8e [0,4194304] 0 2024-02-28T16:31:34.162 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:33 smithi001 ceph-mon[28991]: pgmap v60: 65 pgs: 65 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 88 MiB/s rd, 200 MiB/s wr, 435 op/s 2024-02-28T16:31:34.163 INFO:tasks.workunit.client.1.smithi066.stdout:6/887: chown d2/da/d19/d75/dbd/dc6/f116 18426835 1 2024-02-28T16:31:34.166 INFO:tasks.workunit.client.1.smithi066.stdout:6/888: dread - d2/da/d19/d75/dbd/fd5 zero size 2024-02-28T16:31:34.170 INFO:tasks.workunit.client.1.smithi066.stdout:6/889: write d2/da/d19/d75/da6/daa/f12f [135077,73078] 0 2024-02-28T16:31:34.172 INFO:tasks.workunit.client.1.smithi066.stdout:6/890: chown d2/d26/dbe/df7/l124 3481419 1 2024-02-28T16:31:34.174 INFO:tasks.workunit.client.1.smithi066.stdout:7/822: dread fa [0,4194304] 0 2024-02-28T16:31:34.174 INFO:tasks.workunit.client.1.smithi066.stdout:6/891: chown d2/d2c/da1/d5d/d63/c98 5 1 2024-02-28T16:31:34.177 INFO:tasks.workunit.client.1.smithi066.stdout:6/892: creat d2/d2c/dba/f139 x:0 0 0 2024-02-28T16:31:34.178 INFO:tasks.workunit.client.1.smithi066.stdout:7/823: mkdir dc/d56/dc3/df4/d108 0 2024-02-28T16:31:34.180 INFO:tasks.workunit.client.1.smithi066.stdout:6/893: symlink d2/d26/de0/l13a 0 2024-02-28T16:31:34.185 INFO:tasks.workunit.client.1.smithi066.stdout:6/894: link d2/d2c/da1/d5d/ffb d2/d26/dbe/d104/f13b 0 2024-02-28T16:31:34.188 INFO:tasks.workunit.client.1.smithi066.stdout:6/895: getdents d2/da/d19/d9c/d114 0 2024-02-28T16:31:34.189 INFO:tasks.workunit.client.1.smithi066.stdout:6/896: read - d2/d2c/d72/f109 zero size 2024-02-28T16:31:34.194 INFO:tasks.workunit.client.1.smithi066.stdout:6/897: getdents d2/d2c 0 2024-02-28T16:31:34.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:33 smithi066 ceph-mon[36236]: pgmap v60: 65 pgs: 65 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 88 MiB/s rd, 200 MiB/s wr, 435 op/s 2024-02-28T16:31:34.591 INFO:tasks.workunit.client.1.smithi066.stdout:2/870: sync 2024-02-28T16:31:34.591 INFO:tasks.workunit.client.1.smithi066.stdout:3/991: sync 2024-02-28T16:31:34.591 INFO:tasks.workunit.client.1.smithi066.stdout:8/912: sync 2024-02-28T16:31:34.591 INFO:tasks.workunit.client.1.smithi066.stdout:9/673: sync 2024-02-28T16:31:34.591 INFO:tasks.workunit.client.1.smithi066.stdout:6/898: sync 2024-02-28T16:31:34.592 INFO:tasks.workunit.client.1.smithi066.stdout:5/772: sync 2024-02-28T16:31:34.592 INFO:tasks.workunit.client.1.smithi066.stdout:7/824: sync 2024-02-28T16:31:34.592 INFO:tasks.workunit.client.1.smithi066.stdout:0/995: sync 2024-02-28T16:31:34.592 INFO:tasks.workunit.client.1.smithi066.stdout:1/863: sync 2024-02-28T16:31:34.600 INFO:tasks.workunit.client.1.smithi066.stdout:2/871: write d0/d12/d1a/d23/fe1 [3707075,107102] 0 2024-02-28T16:31:34.600 INFO:tasks.workunit.client.1.smithi066.stdout:3/992: write db/d13/d1a/f13a [1033791,115059] 0 2024-02-28T16:31:34.602 INFO:tasks.workunit.client.1.smithi066.stdout:7/825: dread - dc/d29/d92/da6/dd8/d37/df7/d100/f105 zero size 2024-02-28T16:31:34.607 INFO:tasks.workunit.client.1.smithi066.stdout:6/899: creat d2/da/d19/d9c/d3d/f13c x:0 0 0 2024-02-28T16:31:34.607 INFO:tasks.workunit.client.1.smithi066.stdout:1/864: rmdir d2/d18/d7a 39 2024-02-28T16:31:34.609 INFO:tasks.workunit.client.1.smithi066.stdout:0/996: symlink d2/d27/d41/d76/d93/d10/d23/d6d/d113/d94/d122/l14e 0 2024-02-28T16:31:34.611 INFO:tasks.workunit.client.1.smithi066.stdout:2/872: mknod d0/d12/d1a/d23/ddc/c116 0 2024-02-28T16:31:34.613 INFO:tasks.workunit.client.1.smithi066.stdout:3/993: truncate db/dc/d35/d3d/d76/d89/dcc/fb2 4536476 0 2024-02-28T16:31:34.614 INFO:tasks.workunit.client.1.smithi066.stdout:6/900: chown d2/d2c/d82/c4c 74 1 2024-02-28T16:31:34.614 INFO:tasks.workunit.client.1.smithi066.stdout:0/997: fdatasync d2/d27/d41/d76/d93/f6f 0 2024-02-28T16:31:34.618 INFO:tasks.workunit.client.1.smithi066.stdout:1/865: fsync d2/d27/d2b/d34/da9/de9/fb8 0 2024-02-28T16:31:34.620 INFO:tasks.workunit.client.1.smithi066.stdout:5/773: dread d3/d4/dc/dd/d2e/fce [0,4194304] 0 2024-02-28T16:31:34.622 INFO:tasks.workunit.client.1.smithi066.stdout:9/674: write da/d3a/f41 [945831,45130] 0 2024-02-28T16:31:34.622 INFO:tasks.workunit.client.1.smithi066.stdout:3/994: read db/dc/d35/d3d/d76/d89/f9f [12162,17678] 0 2024-02-28T16:31:34.625 INFO:tasks.workunit.client.1.smithi066.stdout:9/675: truncate da/d3a/f41 2943160 0 2024-02-28T16:31:34.626 INFO:tasks.workunit.client.1.smithi066.stdout:7/826: getdents dc/d29/d92/da6/dd8/d91 0 2024-02-28T16:31:34.627 INFO:tasks.workunit.client.1.smithi066.stdout:3/995: chown db/dc/d11/d1f/d78/fbf 198224161 1 2024-02-28T16:31:34.628 INFO:tasks.workunit.client.1.smithi066.stdout:2/873: rename d0/d12/d3e/f63 to d0/d12/d1a/f117 0 2024-02-28T16:31:34.628 INFO:tasks.workunit.client.1.smithi066.stdout:7/827: read - dc/d56/dce/d102/f104 zero size 2024-02-28T16:31:34.630 INFO:tasks.workunit.client.1.smithi066.stdout:0/998: mkdir d2/d27/d41/d76/d93/d10/d23/d14f 0 2024-02-28T16:31:34.631 INFO:tasks.workunit.client.1.smithi066.stdout:7/828: chown dc/d56/dc3/ff5 239678350 1 2024-02-28T16:31:34.633 INFO:tasks.workunit.client.1.smithi066.stdout:8/913: truncate d0/db/d12/d1d/fd0 3466293 0 2024-02-28T16:31:34.637 INFO:tasks.workunit.client.1.smithi066.stdout:1/866: creat d2/d27/dc7/d48/dce/f11d x:0 0 0 2024-02-28T16:31:34.643 INFO:tasks.workunit.client.1.smithi066.stdout:6/901: rename d2/d26/dbe/df7 to d2/da/d19/d75/da6/d13d 0 2024-02-28T16:31:34.644 INFO:tasks.workunit.client.1.smithi066.stdout:2/874: dwrite d0/d12/d1a/d39/d7c/fdd [0,4194304] 0 2024-02-28T16:31:34.646 INFO:tasks.workunit.client.1.smithi066.stdout:5/774: write d3/d4/fb [2629474,73196] 0 2024-02-28T16:31:34.654 INFO:tasks.workunit.client.1.smithi066.stdout:2/875: truncate d0/d12/d1a/d23/d29/d21/fd9 955164 0 2024-02-28T16:31:34.655 INFO:tasks.workunit.client.1.smithi066.stdout:0/999: symlink d2/d27/d41/d76/d93/d139/da2/l150 0 2024-02-28T16:31:34.675 INFO:tasks.workunit.client.1.smithi066.stdout:9/676: dwrite da/df/f2f [4194304,4194304] 0 2024-02-28T16:31:34.675 INFO:tasks.workunit.client.1.smithi066.stdout:5/775: link d3/d4/dc/dd/d2e/d8f/fe2 d3/d4/dc/dd/d41/d7a/dac/d57/ff9 0 2024-02-28T16:31:34.675 INFO:tasks.workunit.client.1.smithi066.stdout:7/829: write dc/d29/d92/da6/dd8/d1e/d45/f84 [198477,42257] 0 2024-02-28T16:31:34.688 INFO:tasks.workunit.client.1.smithi066.stdout:2/876: dwrite d0/d12/d2e/fc1 [0,4194304] 0 2024-02-28T16:31:34.693 INFO:tasks.workunit.client.1.smithi066.stdout:1/867: dwrite d2/d27/d2b/d34/fac [0,4194304] 0 2024-02-28T16:31:34.697 INFO:tasks.workunit.client.1.smithi066.stdout:6/902: dwrite d2/d2c/da1/dad/ff5 [0,4194304] 0 2024-02-28T16:31:34.700 INFO:tasks.workunit.client.1.smithi066.stdout:1/868: dread - d2/d18/d6b/fa1 zero size 2024-02-28T16:31:34.705 INFO:tasks.workunit.client.1.smithi066.stdout:3/996: dwrite db/dc/d35/d3d/d76/d89/f9f [0,4194304] 0 2024-02-28T16:31:34.722 INFO:tasks.workunit.client.1.smithi066.stdout:9/677: rename da/d22/d46/d56/d8f/d2d/d81 to da/d22/d46/d56/d8f/d28/d42/d68/dda 0 2024-02-28T16:31:34.723 INFO:tasks.workunit.client.1.smithi066.stdout:8/914: getdents d0/db/d12/d1d/d25/d65/d93 0 2024-02-28T16:31:34.728 INFO:tasks.workunit.client.1.smithi066.stdout:5/776: getdents d3/d4/dc/dd/d35/d86/ded 0 2024-02-28T16:31:34.729 INFO:tasks.workunit.client.1.smithi066.stdout:3/997: mkdir db/dc/d11/d1f/d164 0 2024-02-28T16:31:34.737 INFO:tasks.workunit.client.1.smithi066.stdout:2/877: getdents d0/d12/d1a/d23/d29/d21/d2f/d67 0 2024-02-28T16:31:34.738 INFO:tasks.workunit.client.1.smithi066.stdout:5/777: symlink d3/lfa 0 2024-02-28T16:31:34.743 INFO:tasks.workunit.client.1.smithi066.stdout:9/678: creat da/fdb x:0 0 0 2024-02-28T16:31:34.748 INFO:tasks.workunit.client.1.smithi066.stdout:8/915: link d0/db/d12/d1d/d25/d65/dbb/d11e/f122 d0/d6/dc/d16/d4c/f12e 0 2024-02-28T16:31:34.748 INFO:tasks.workunit.client.1.smithi066.stdout:9/679: readlink da/df/ld3 0 2024-02-28T16:31:34.752 INFO:tasks.workunit.client.1.smithi066.stdout:5/778: dread - d3/d4/dc/d56/fbc zero size 2024-02-28T16:31:34.755 INFO:tasks.workunit.client.1.smithi066.stdout:5/779: fsync d3/d1b/fe8 0 2024-02-28T16:31:34.759 INFO:tasks.workunit.client.1.smithi066.stdout:5/780: dread - d3/d1b/fe8 zero size 2024-02-28T16:31:34.761 INFO:tasks.workunit.client.1.smithi066.stdout:9/680: symlink da/d4b/d5e/ldc 0 2024-02-28T16:31:34.764 INFO:tasks.workunit.client.1.smithi066.stdout:6/903: dread d2/d26/de0/f105 [0,4194304] 0 2024-02-28T16:31:34.768 INFO:tasks.workunit.client.1.smithi066.stdout:2/878: dwrite d0/d12/d1a/d23/d5d/fea [0,4194304] 0 2024-02-28T16:31:34.779 INFO:tasks.workunit.client.1.smithi066.stdout:6/904: dread d2/da/dcb/fe7 [0,4194304] 0 2024-02-28T16:31:34.783 INFO:tasks.workunit.client.1.smithi066.stdout:3/998: dread db/d13/d14/dad/fe1 [0,4194304] 0 2024-02-28T16:31:34.785 INFO:tasks.workunit.client.1.smithi066.stdout:2/879: mknod d0/d12/d3e/d9e/c118 0 2024-02-28T16:31:34.794 INFO:tasks.workunit.client.1.smithi066.stdout:1/869: dwrite d2/f13 [0,4194304] 0 2024-02-28T16:31:34.794 INFO:tasks.workunit.client.1.smithi066.stdout:7/830: dwrite dc/d29/d92/da6/dd8/d1e/d45/d9f/fbe [0,4194304] 0 2024-02-28T16:31:34.799 INFO:tasks.workunit.client.1.smithi066.stdout:9/681: link da/d22/d46/d56/d8f/d28/ca9 da/d22/d46/d56/cdd 0 2024-02-28T16:31:34.815 INFO:tasks.workunit.client.1.smithi066.stdout:5/781: dwrite d3/d1b/f5f [0,4194304] 0 2024-02-28T16:31:34.816 INFO:tasks.workunit.client.1.smithi066.stdout:6/905: rmdir d2/d26/d6b 0 2024-02-28T16:31:34.821 INFO:tasks.workunit.client.1.smithi066.stdout:5/782: chown d3/d4/dc/dd/d2e/lae 13 1 2024-02-28T16:31:34.822 INFO:tasks.workunit.client.1.smithi066.stdout:9/682: mkdir da/d22/d46/d56/d8f/d28/dde 0 2024-02-28T16:31:34.823 INFO:tasks.workunit.client.1.smithi066.stdout:7/831: dwrite f1 [8388608,4194304] 0 2024-02-28T16:31:34.826 INFO:tasks.workunit.client.1.smithi066.stdout:1/870: mkdir d2/d27/dc7/d54/d11e 0 2024-02-28T16:31:34.827 INFO:tasks.workunit.client.1.smithi066.stdout:3/999: truncate db/d13/d14/dda/fe6 962034 0 2024-02-28T16:31:34.832 INFO:tasks.workunit.client.1.smithi066.stdout:5/783: write d3/d4/dc/dd/d35/f62 [2777873,125624] 0 2024-02-28T16:31:34.833 INFO:tasks.workunit.client.1.smithi066.stdout:6/906: symlink d2/da/d19/d75/dbd/dc6/l13e 0 2024-02-28T16:31:34.836 INFO:tasks.workunit.client.1.smithi066.stdout:9/683: read da/d22/d46/d56/d8f/d28/d42/d68/f7a [1431649,118400] 0 2024-02-28T16:31:34.838 INFO:tasks.workunit.client.1.smithi066.stdout:7/832: mkdir dc/d29/d92/da6/d109 0 2024-02-28T16:31:34.842 INFO:tasks.workunit.client.1.smithi066.stdout:1/871: symlink d2/d18/d7a/dfd/l11f 0 2024-02-28T16:31:34.842 INFO:tasks.workunit.client.1.smithi066.stdout:6/907: mknod d2/d26/d32/c13f 0 2024-02-28T16:31:34.842 INFO:tasks.workunit.client.1.smithi066.stdout:5/784: chown d3/d4/dc/dd/d2e/d8f/cbe 128518306 1 2024-02-28T16:31:34.845 INFO:tasks.workunit.client.1.smithi066.stdout:7/833: mkdir dc/d56/dc3/df4/d10a 0 2024-02-28T16:31:34.847 INFO:tasks.workunit.client.1.smithi066.stdout:6/908: symlink d2/d26/d32/db9/l140 0 2024-02-28T16:31:34.848 INFO:tasks.workunit.client.1.smithi066.stdout:9/684: symlink da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/d90/ldf 0 2024-02-28T16:31:34.850 INFO:tasks.workunit.client.1.smithi066.stdout:1/872: rename d2/d27/dc7/d54/de7/ff7 to d2/d27/dc7/da6/dd4/f120 0 2024-02-28T16:31:34.851 INFO:tasks.workunit.client.1.smithi066.stdout:6/909: write d2/d26/d32/d46/f103 [2013859,87703] 0 2024-02-28T16:31:34.854 INFO:tasks.workunit.client.1.smithi066.stdout:7/834: read dc/d29/d92/da6/dd8/d1e/f3e [421043,68137] 0 2024-02-28T16:31:34.855 INFO:tasks.workunit.client.1.smithi066.stdout:5/785: rmdir d3/d4/dc/dd9 0 2024-02-28T16:31:34.868 INFO:tasks.workunit.client.1.smithi066.stdout:5/786: fdatasync d3/d4/dc/dd/d41/d7a/dac/d57/ff9 0 2024-02-28T16:31:34.871 INFO:tasks.workunit.client.1.smithi066.stdout:1/873: getdents d2/d27/dc7/da6 0 2024-02-28T16:31:34.872 INFO:tasks.workunit.client.1.smithi066.stdout:7/835: getdents dc/de/d87 0 2024-02-28T16:31:34.875 INFO:tasks.workunit.client.1.smithi066.stdout:5/787: fdatasync d3/d4/dc/dd/d35/fcc 0 2024-02-28T16:31:34.878 INFO:tasks.workunit.client.1.smithi066.stdout:1/874: mknod d2/d27/dc7/de4/d11c/c121 0 2024-02-28T16:31:34.882 INFO:tasks.workunit.client.1.smithi066.stdout:5/788: unlink d3/d4/dc/dd/d41/d7a/dac/fb4 0 2024-02-28T16:31:34.885 INFO:tasks.workunit.client.1.smithi066.stdout:1/875: unlink d2/d27/d2b/d34/da9/de9/fb8 0 2024-02-28T16:31:34.891 INFO:tasks.workunit.client.1.smithi066.stdout:9/685: dwrite da/d22/d46/d56/d8f/f6f [0,4194304] 0 2024-02-28T16:31:34.891 INFO:tasks.workunit.client.1.smithi066.stdout:6/910: dwrite d2/d26/f53 [0,4194304] 0 2024-02-28T16:31:34.902 INFO:tasks.workunit.client.1.smithi066.stdout:1/876: write d2/d18/d19/f3b [877541,34421] 0 2024-02-28T16:31:34.904 INFO:tasks.workunit.client.1.smithi066.stdout:6/911: mkdir d2/da/d19/d141 0 2024-02-28T16:31:34.906 INFO:tasks.workunit.client.1.smithi066.stdout:6/912: write d2/d26/d32/d46/f74 [4557640,49114] 0 2024-02-28T16:31:34.912 INFO:tasks.workunit.client.1.smithi066.stdout:1/877: creat d2/d27/d2b/d34/da9/f122 x:0 0 0 2024-02-28T16:31:34.914 INFO:tasks.workunit.client.1.smithi066.stdout:6/913: getdents d2/da/d19/d75/da6/d127 0 2024-02-28T16:31:34.916 INFO:tasks.workunit.client.1.smithi066.stdout:1/878: write d2/d18/d7a/dbc/f11b [749748,117744] 0 2024-02-28T16:31:34.921 INFO:tasks.workunit.client.1.smithi066.stdout:5/789: dread d3/d4/dc/dd/d2e/d18/f4b [0,4194304] 0 2024-02-28T16:31:34.921 INFO:tasks.workunit.client.1.smithi066.stdout:1/879: write d2/d27/d2b/d34/da9/f122 [501016,57510] 0 2024-02-28T16:31:34.928 INFO:tasks.workunit.client.1.smithi066.stdout:1/880: write d2/d27/d2b/d43/d46/f61 [567168,112834] 0 2024-02-28T16:31:34.928 INFO:tasks.workunit.client.1.smithi066.stdout:5/790: mknod d3/d4/dc/dd/d2e/d18/cfb 0 2024-02-28T16:31:34.932 INFO:tasks.workunit.client.1.smithi066.stdout:9/686: dread da/d22/d46/d56/d8f/d28/d42/d68/f8c [0,4194304] 0 2024-02-28T16:31:34.933 INFO:tasks.workunit.client.1.smithi066.stdout:1/881: dread - d2/d27/d2b/d34/da9/fdb zero size 2024-02-28T16:31:34.933 INFO:tasks.workunit.client.1.smithi066.stdout:5/791: symlink d3/d1b/d20/ddd/lfc 0 2024-02-28T16:31:34.934 INFO:tasks.workunit.client.1.smithi066.stdout:6/914: dread d2/da/d19/d75/da6/daa/dbf/fe3 [0,4194304] 0 2024-02-28T16:31:34.940 INFO:tasks.workunit.client.1.smithi066.stdout:5/792: mkdir d3/d4/dc/dd/d2e/d8f/dfd 0 2024-02-28T16:31:34.942 INFO:tasks.workunit.client.1.smithi066.stdout:5/793: readlink d3/d1b/d7d/l8a 0 2024-02-28T16:31:34.945 INFO:tasks.workunit.client.1.smithi066.stdout:5/794: read d3/d4/dc/dd/f9f [161398,27234] 0 2024-02-28T16:31:34.946 INFO:tasks.workunit.client.1.smithi066.stdout:9/687: link da/d22/d46/d56/d8f/d28/d42/c79 da/d22/d46/d56/d8f/d28/d42/d68/dda/ce0 0 2024-02-28T16:31:34.950 INFO:tasks.workunit.client.1.smithi066.stdout:5/795: mknod d3/d4/dc/dd/d35/d86/ded/cfe 0 2024-02-28T16:31:34.956 INFO:tasks.workunit.client.1.smithi066.stdout:1/882: dwrite d2/d27/d2b/d34/da9/de9/f87 [0,4194304] 0 2024-02-28T16:31:34.956 INFO:tasks.workunit.client.1.smithi066.stdout:6/915: dwrite d2/da/f13 [0,4194304] 0 2024-02-28T16:31:34.961 INFO:tasks.workunit.client.1.smithi066.stdout:5/796: link d3/d4/dc/dd/d2e/f79 d3/d4/dc/dd/d2e/d8f/dfd/fff 0 2024-02-28T16:31:34.976 INFO:tasks.workunit.client.1.smithi066.stdout:6/916: dread d2/da/d19/d9c/d3d/f89 [0,4194304] 0 2024-02-28T16:31:34.982 INFO:tasks.workunit.client.1.smithi066.stdout:1/883: write d2/d27/dc7/d54/ffb [2376728,54567] 0 2024-02-28T16:31:34.987 INFO:tasks.workunit.client.1.smithi066.stdout:1/884: symlink d2/d27/d2b/d43/dc8/l123 0 2024-02-28T16:31:34.990 INFO:tasks.workunit.client.1.smithi066.stdout:5/797: dwrite d3/d4/dc/dd/d35/d86/fdc [0,4194304] 0 2024-02-28T16:31:34.990 INFO:tasks.workunit.client.1.smithi066.stdout:1/885: mknod d2/d27/d2b/d34/c124 0 2024-02-28T16:31:34.992 INFO:tasks.workunit.client.1.smithi066.stdout:6/917: dwrite d2/d26/f5f [4194304,4194304] 0 2024-02-28T16:31:34.997 INFO:tasks.workunit.client.1.smithi066.stdout:6/918: symlink d2/da/d19/d75/da6/daa/l142 0 2024-02-28T16:31:34.998 INFO:tasks.workunit.client.1.smithi066.stdout:5/798: write d3/d4/f9 [4456789,44780] 0 2024-02-28T16:31:35.002 INFO:tasks.workunit.client.1.smithi066.stdout:6/919: read d2/da/d19/f41 [3393795,47344] 0 2024-02-28T16:31:35.003 INFO:tasks.workunit.client.1.smithi066.stdout:5/799: creat d3/d4/dc/dd/d41/d5d/f100 x:0 0 0 2024-02-28T16:31:35.008 INFO:tasks.workunit.client.1.smithi066.stdout:6/920: write d2/d26/f4e [3834196,107097] 0 2024-02-28T16:31:35.022 INFO:tasks.workunit.client.1.smithi066.stdout:5/800: truncate d3/d4/dc/dd/d2e/f90 1637462 0 2024-02-28T16:31:35.024 INFO:tasks.workunit.client.1.smithi066.stdout:5/801: chown d3/d4/dc/dd/d35/d86/fdc 3 1 2024-02-28T16:31:35.028 INFO:tasks.workunit.client.1.smithi066.stdout:5/802: rmdir d3/d1b/d20/dc1 39 2024-02-28T16:31:35.028 INFO:tasks.workunit.client.1.smithi066.stdout:6/921: dwrite d2/d2c/da1/d5d/d63/d6a/d100/f119 [0,4194304] 0 2024-02-28T16:31:35.034 INFO:tasks.workunit.client.1.smithi066.stdout:5/803: fdatasync d3/d4/dc/fda 0 2024-02-28T16:31:35.037 INFO:tasks.workunit.client.1.smithi066.stdout:6/922: creat d2/da/f143 x:0 0 0 2024-02-28T16:31:35.038 INFO:tasks.workunit.client.1.smithi066.stdout:5/804: dread - d3/d4/dc/dd/d2e/d8f/faa zero size 2024-02-28T16:31:35.041 INFO:tasks.workunit.client.1.smithi066.stdout:1/886: dread d2/d27/dc7/d48/f81 [0,4194304] 0 2024-02-28T16:31:35.043 INFO:tasks.workunit.client.1.smithi066.stdout:5/805: symlink d3/d4/dc/dd/d41/d5d/l101 0 2024-02-28T16:31:35.045 INFO:tasks.workunit.client.1.smithi066.stdout:1/887: mkdir d2/d27/d125 0 2024-02-28T16:31:35.046 INFO:tasks.workunit.client.1.smithi066.stdout:5/806: dread - d3/d4/dc/dd/d35/d86/fcd zero size 2024-02-28T16:31:35.054 INFO:tasks.workunit.client.1.smithi066.stdout:6/923: dwrite d2/da/d19/f41 [0,4194304] 0 2024-02-28T16:31:35.058 INFO:tasks.workunit.client.1.smithi066.stdout:6/924: symlink d2/d2c/l144 0 2024-02-28T16:31:35.062 INFO:tasks.workunit.client.1.smithi066.stdout:1/888: dwrite d2/d27/df9/f10d [0,4194304] 0 2024-02-28T16:31:35.065 INFO:tasks.workunit.client.1.smithi066.stdout:5/807: dwrite d3/d4/dc/d56/f75 [0,4194304] 0 2024-02-28T16:31:35.066 INFO:tasks.workunit.client.1.smithi066.stdout:1/889: write d2/d18/d7a/dbc/fc5 [6174695,112088] 0 2024-02-28T16:31:35.077 INFO:tasks.workunit.client.1.smithi066.stdout:1/890: symlink d2/l126 0 2024-02-28T16:31:35.077 INFO:tasks.workunit.client.1.smithi066.stdout:5/808: fsync d3/d4/dc/dd/d2e/d8f/dfd/fff 0 2024-02-28T16:31:35.082 INFO:tasks.workunit.client.1.smithi066.stdout:1/891: symlink d2/da0/l127 0 2024-02-28T16:31:35.085 INFO:tasks.workunit.client.1.smithi066.stdout:6/925: dwrite d2/da/d19/d75/da6/f133 [0,4194304] 0 2024-02-28T16:31:35.091 INFO:tasks.workunit.client.1.smithi066.stdout:1/892: rmdir d2/d27/dc7 39 2024-02-28T16:31:35.099 INFO:tasks.workunit.client.1.smithi066.stdout:1/893: mkdir d2/d27/dc7/d51/d128 0 2024-02-28T16:31:35.099 INFO:tasks.workunit.client.1.smithi066.stdout:5/809: dwrite d3/d4/dc/dd/d41/da2/f8d [0,4194304] 0 2024-02-28T16:31:35.112 INFO:tasks.workunit.client.1.smithi066.stdout:6/926: dwrite d2/d2c/da1/d5d/d63/f12d [0,4194304] 0 2024-02-28T16:31:35.115 INFO:tasks.workunit.client.1.smithi066.stdout:6/927: stat d2/da/f143 0 2024-02-28T16:31:35.119 INFO:tasks.workunit.client.1.smithi066.stdout:5/810: dwrite d3/d4/dc/fda [0,4194304] 0 2024-02-28T16:31:35.125 INFO:tasks.workunit.client.1.smithi066.stdout:5/811: symlink d3/d4/dc/dd/d35/d86/l102 0 2024-02-28T16:31:35.128 INFO:tasks.workunit.client.1.smithi066.stdout:5/812: dread - d3/d4/dc/dd/d41/d7a/dac/d60/fc7 zero size 2024-02-28T16:31:35.132 INFO:tasks.workunit.client.1.smithi066.stdout:6/928: dwrite d2/da/d19/f4f [0,4194304] 0 2024-02-28T16:31:35.132 INFO:tasks.workunit.client.1.smithi066.stdout:5/813: creat d3/d4/dc/dd/d35/d92/f103 x:0 0 0 2024-02-28T16:31:35.134 INFO:tasks.workunit.client.1.smithi066.stdout:5/814: truncate d3/d4/f83 843627 0 2024-02-28T16:31:35.136 INFO:tasks.workunit.client.1.smithi066.stdout:5/815: chown d3/d1b/d20 2324 1 2024-02-28T16:31:35.139 INFO:tasks.workunit.client.1.smithi066.stdout:6/929: symlink d2/d2c/da1/d5d/d63/d6a/d100/l145 0 2024-02-28T16:31:35.141 INFO:tasks.workunit.client.1.smithi066.stdout:5/816: unlink d3/d1b/f31 0 2024-02-28T16:31:35.145 INFO:tasks.workunit.client.1.smithi066.stdout:6/930: creat d2/d2c/d72/f146 x:0 0 0 2024-02-28T16:31:35.145 INFO:tasks.workunit.client.1.smithi066.stdout:1/894: dread d2/d27/d2b/fc6 [0,4194304] 0 2024-02-28T16:31:35.148 INFO:tasks.workunit.client.1.smithi066.stdout:1/895: truncate d2/d18/d19/ff2 4933597 0 2024-02-28T16:31:35.150 INFO:tasks.workunit.client.1.smithi066.stdout:6/931: mkdir d2/da/d19/d75/dbd/dc6/d147 0 2024-02-28T16:31:35.151 INFO:tasks.workunit.client.1.smithi066.stdout:1/896: mkdir d2/d27/dc7/da6/dd4/d129 0 2024-02-28T16:31:35.153 INFO:tasks.workunit.client.1.smithi066.stdout:6/932: dread - d2/da/d19/d75/dbd/dc6/dc7/f10b zero size 2024-02-28T16:31:35.153 INFO:tasks.workunit.client.1.smithi066.stdout:1/897: chown d2/d27/dc7/da6/f7e 281557 1 2024-02-28T16:31:35.154 INFO:tasks.workunit.client.1.smithi066.stdout:1/898: chown d2/f5 243332 1 2024-02-28T16:31:35.216 INFO:tasks.workunit.client.1.smithi066.stdout:1/899: dread d2/d27/d2b/d43/faf [0,4194304] 0 2024-02-28T16:31:35.222 INFO:tasks.workunit.client.1.smithi066.stdout:1/900: link d2/d27/d2b/d34/da9/de9/f87 d2/d18/da5/f12a 0 2024-02-28T16:31:35.229 INFO:tasks.workunit.client.1.smithi066.stdout:6/933: dwrite d2/da/d19/d9c/d3d/f48 [0,4194304] 0 2024-02-28T16:31:35.235 INFO:tasks.workunit.client.1.smithi066.stdout:6/934: mknod d2/da/d19/d75/c148 0 2024-02-28T16:31:35.246 INFO:tasks.workunit.client.1.smithi066.stdout:1/901: dread d2/d27/dc7/da6/faa [0,4194304] 0 2024-02-28T16:31:35.250 INFO:tasks.workunit.client.1.smithi066.stdout:1/902: truncate d2/d18/d7a/f10e 665206 0 2024-02-28T16:31:35.260 INFO:tasks.workunit.client.1.smithi066.stdout:6/935: dwrite d2/d2c/da1/d5d/d99/fff [0,4194304] 0 2024-02-28T16:31:35.267 INFO:tasks.workunit.client.1.smithi066.stdout:1/903: rename d2/d27/d2b/d34/c65 to d2/d18/d7a/d110/c12b 0 2024-02-28T16:31:35.273 INFO:tasks.workunit.client.1.smithi066.stdout:1/904: chown d2/d27/dc7/d48/dd6 1185579 1 2024-02-28T16:31:35.273 INFO:tasks.workunit.client.1.smithi066.stdout:1/905: fsync d2/d18/d19/f3b 0 2024-02-28T16:31:35.280 INFO:tasks.workunit.client.1.smithi066.stdout:6/936: truncate d2/da/dcb/f101 67632 0 2024-02-28T16:31:35.284 INFO:tasks.workunit.client.1.smithi066.stdout:1/906: getdents d2/d27/dc7/da6/dd4 0 2024-02-28T16:31:35.287 INFO:tasks.workunit.client.1.smithi066.stdout:6/937: dread d2/da/d19/d75/da6/daa/f12f [0,4194304] 0 2024-02-28T16:31:35.287 INFO:tasks.workunit.client.1.smithi066.stdout:1/907: write d2/d27/dc7/de4/fff [688025,106026] 0 2024-02-28T16:31:35.293 INFO:tasks.workunit.client.1.smithi066.stdout:1/908: creat d2/d18/d7a/db1/f12c x:0 0 0 2024-02-28T16:31:35.296 INFO:tasks.workunit.client.1.smithi066.stdout:6/938: creat d2/da/d19/d9c/f149 x:0 0 0 2024-02-28T16:31:35.298 INFO:tasks.workunit.client.1.smithi066.stdout:1/909: mkdir d2/d27/dc7/de2/d100/d12d 0 2024-02-28T16:31:35.299 INFO:tasks.workunit.client.1.smithi066.stdout:6/939: creat d2/d26/d32/d46/f14a x:0 0 0 2024-02-28T16:31:35.301 INFO:tasks.workunit.client.1.smithi066.stdout:1/910: dread - d2/d18/d6b/dba/de1/f114 zero size 2024-02-28T16:31:35.301 INFO:tasks.workunit.client.1.smithi066.stdout:6/940: rmdir d2/d2c/da1 39 2024-02-28T16:31:35.303 INFO:tasks.workunit.client.1.smithi066.stdout:1/911: mkdir d2/d27/dc7/da6/d12e 0 2024-02-28T16:31:35.305 INFO:tasks.workunit.client.1.smithi066.stdout:1/912: chown d2/f1d 31655 1 2024-02-28T16:31:35.306 INFO:tasks.workunit.client.1.smithi066.stdout:6/941: creat d2/d2c/f14b x:0 0 0 2024-02-28T16:31:35.323 INFO:tasks.workunit.client.1.smithi066.stdout:6/942: truncate d2/da/dcb/f101 737807 0 2024-02-28T16:31:35.323 INFO:tasks.workunit.client.1.smithi066.stdout:1/913: dwrite d2/d27/df9/f10d [0,4194304] 0 2024-02-28T16:31:35.331 INFO:tasks.workunit.client.1.smithi066.stdout:1/914: mknod d2/d27/d2b/d34/c12f 0 2024-02-28T16:31:35.336 INFO:tasks.workunit.client.1.smithi066.stdout:6/943: truncate d2/d2c/da1/d5d/d63/d6a/d100/f126 939170 0 2024-02-28T16:31:35.338 INFO:tasks.workunit.client.1.smithi066.stdout:1/915: rename d2/d18/d6b/dba/lbb to d2/d27/d2b/d34/da9/l130 0 2024-02-28T16:31:35.339 INFO:tasks.workunit.client.1.smithi066.stdout:6/944: rename d2/d2c/da1/d5d/d99 to d2/d2c/da1/d5d/d99/d14c 22 2024-02-28T16:31:35.341 INFO:tasks.workunit.client.1.smithi066.stdout:1/916: creat d2/d27/df9/f131 x:0 0 0 2024-02-28T16:31:35.342 INFO:tasks.workunit.client.1.smithi066.stdout:6/945: fsync d2/d2c/da1/d5d/d63/d6a/d100/f119 0 2024-02-28T16:31:35.344 INFO:tasks.workunit.client.1.smithi066.stdout:1/917: chown d2/d27/dc7/d48/f81 127648 1 2024-02-28T16:31:35.345 INFO:tasks.workunit.client.1.smithi066.stdout:6/946: symlink d2/l14d 0 2024-02-28T16:31:35.353 INFO:tasks.workunit.client.1.smithi066.stdout:1/918: rename d2/d27/dc7/d54/d84 to d2/d27/d132 0 2024-02-28T16:31:35.357 INFO:tasks.workunit.client.1.smithi066.stdout:6/947: dread d2/d26/d32/d46/f74 [0,4194304] 0 2024-02-28T16:31:35.362 INFO:tasks.workunit.client.1.smithi066.stdout:6/948: rename d2/d26/fa3 to d2/da/d19/d75/dbd/dc6/dc7/f14e 0 2024-02-28T16:31:35.375 INFO:tasks.workunit.client.1.smithi066.stdout:6/949: dwrite d2/da/f3b [0,4194304] 0 2024-02-28T16:31:35.385 INFO:tasks.workunit.client.1.smithi066.stdout:6/950: getdents d2/da/dcc/dd6/d102 0 2024-02-28T16:31:35.386 INFO:tasks.workunit.client.1.smithi066.stdout:6/951: chown d2/d26/f4e 3379874 1 2024-02-28T16:31:35.388 INFO:tasks.workunit.client.1.smithi066.stdout:6/952: readlink d2/da/d19/d75/da6/d13d/l124 0 2024-02-28T16:31:35.404 INFO:tasks.workunit.client.1.smithi066.stdout:6/953: dwrite d2/d26/d32/d46/fda [0,4194304] 0 2024-02-28T16:31:35.412 INFO:tasks.workunit.client.1.smithi066.stdout:6/954: unlink d2/d2c/f4a 0 2024-02-28T16:31:35.415 INFO:tasks.workunit.client.1.smithi066.stdout:6/955: symlink d2/da/d19/d9c/de4/l14f 0 2024-02-28T16:31:35.418 INFO:tasks.workunit.client.1.smithi066.stdout:6/956: write d2/da/d19/d75/da6/daa/dbf/fe3 [4485782,8676] 0 2024-02-28T16:31:35.422 INFO:tasks.workunit.client.1.smithi066.stdout:6/957: write d2/d2c/d82/f8a [1223901,64283] 0 2024-02-28T16:31:35.425 INFO:tasks.workunit.client.1.smithi066.stdout:6/958: getdents d2/da/d19/d75/dbd/dc6/d147 0 2024-02-28T16:31:35.435 INFO:tasks.workunit.client.1.smithi066.stdout:6/959: write d2/da/d19/d75/dbd/dc6/fd1 [704593,66707] 0 2024-02-28T16:31:35.441 INFO:tasks.workunit.client.1.smithi066.stdout:6/960: rename d2/da/cae to d2/da/d19/d75/dbd/dc6/d147/c150 0 2024-02-28T16:31:35.447 INFO:tasks.workunit.client.1.smithi066.stdout:6/961: link d2/d2c/f8e d2/d2c/f151 0 2024-02-28T16:31:35.450 INFO:tasks.workunit.client.1.smithi066.stdout:6/962: write d2/d2c/f30 [469122,83651] 0 2024-02-28T16:31:35.459 INFO:tasks.workunit.client.1.smithi066.stdout:6/963: dread d2/d2c/da1/d5d/f136 [0,4194304] 0 2024-02-28T16:31:35.461 INFO:tasks.workunit.client.1.smithi066.stdout:6/964: creat d2/da/d19/d75/da6/f152 x:0 0 0 2024-02-28T16:31:35.468 INFO:tasks.workunit.client.1.smithi066.stdout:6/965: write d2/da/d19/d9c/d3d/fee [5149279,10259] 0 2024-02-28T16:31:35.470 INFO:tasks.workunit.client.1.smithi066.stdout:6/966: chown d2/da/d19/d75/d88/dbc 790 1 2024-02-28T16:31:35.471 INFO:tasks.workunit.client.1.smithi066.stdout:2/880: sync 2024-02-28T16:31:35.471 INFO:tasks.workunit.client.1.smithi066.stdout:9/688: sync 2024-02-28T16:31:35.471 INFO:tasks.workunit.client.1.smithi066.stdout:7/836: sync 2024-02-28T16:31:35.471 INFO:tasks.workunit.client.1.smithi066.stdout:8/916: sync 2024-02-28T16:31:35.473 INFO:tasks.workunit.client.1.smithi066.stdout:7/837: rmdir dc 39 2024-02-28T16:31:35.474 INFO:tasks.workunit.client.1.smithi066.stdout:6/967: mkdir d2/da/d19/d9c/d3d/d153 0 2024-02-28T16:31:35.478 INFO:tasks.workunit.client.1.smithi066.stdout:9/689: symlink da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/d90/d98/le1 0 2024-02-28T16:31:35.481 INFO:tasks.workunit.client.1.smithi066.stdout:9/690: dread - da/d4b/d5e/fc0 zero size 2024-02-28T16:31:35.491 INFO:tasks.workunit.client.1.smithi066.stdout:9/691: rename da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/d90 to da/d22/d46/d56/d8f/d28/d42/d68/de2 0 2024-02-28T16:31:35.493 INFO:tasks.workunit.client.1.smithi066.stdout:2/881: dwrite d0/d12/d2e/d7a/f99 [0,4194304] 0 2024-02-28T16:31:35.500 INFO:tasks.workunit.client.1.smithi066.stdout:9/692: dread da/d3a/d15/f19 [0,4194304] 0 2024-02-28T16:31:35.501 INFO:tasks.workunit.client.1.smithi066.stdout:6/968: dwrite d2/da/d19/d75/dbd/dc6/fca [0,4194304] 0 2024-02-28T16:31:35.508 INFO:tasks.workunit.client.1.smithi066.stdout:7/838: dread dc/d29/d92/da6/dd8/d1e/f24 [0,4194304] 0 2024-02-28T16:31:35.508 INFO:tasks.workunit.client.1.smithi066.stdout:2/882: link d0/d12/d2e/d3b/d66/l65 d0/d12/d1a/d23/d10b/l119 0 2024-02-28T16:31:35.508 INFO:tasks.workunit.client.1.smithi066.stdout:6/969: write d2/d2c/da1/d5d/d63/f12d [1072179,880] 0 2024-02-28T16:31:35.510 INFO:tasks.workunit.client.1.smithi066.stdout:2/883: fdatasync d0/d12/d1a/d23/d29/d21/d2f/f9c 0 2024-02-28T16:31:35.517 INFO:tasks.workunit.client.1.smithi066.stdout:9/693: mkdir da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3 0 2024-02-28T16:31:35.519 INFO:tasks.workunit.client.1.smithi066.stdout:2/884: write d0/d12/d1a/f117 [6113601,68690] 0 2024-02-28T16:31:35.521 INFO:tasks.workunit.client.1.smithi066.stdout:7/839: rename dc/d29/d92/da6/dd8/d37/df7/d100/d103 to dc/d29/d92/da6/dd8/d91/dbf/d10b 0 2024-02-28T16:31:35.523 INFO:tasks.workunit.client.1.smithi066.stdout:7/840: stat dc/d56/dce/d102/db7 0 2024-02-28T16:31:35.528 INFO:tasks.workunit.client.1.smithi066.stdout:2/885: getdents d0/d12/d1a/df8 0 2024-02-28T16:31:35.530 INFO:tasks.workunit.client.1.smithi066.stdout:7/841: creat dc/d29/d92/da6/dd8/d1e/f10c x:0 0 0 2024-02-28T16:31:35.531 INFO:tasks.workunit.client.1.smithi066.stdout:2/886: fsync d0/d12/d2e/fc1 0 2024-02-28T16:31:35.531 INFO:tasks.workunit.client.1.smithi066.stdout:6/970: dwrite d2/da/d19/d75/da6/daa/dbf/f108 [0,4194304] 0 2024-02-28T16:31:35.539 INFO:tasks.workunit.client.1.smithi066.stdout:9/694: write da/d3a/d15/f9c [1019978,3473] 0 2024-02-28T16:31:35.541 INFO:tasks.workunit.client.1.smithi066.stdout:7/842: rename dc/d6e/c73 to dc/d29/d92/da6/dd8/d1e/d45/d9f/c10d 0 2024-02-28T16:31:35.544 INFO:tasks.workunit.client.1.smithi066.stdout:2/887: creat d0/d12/d1a/d23/d29/f11a x:0 0 0 2024-02-28T16:31:35.547 INFO:tasks.workunit.client.1.smithi066.stdout:7/843: write dc/d29/d92/da6/dd8/d1e/d45/f79 [3490897,63334] 0 2024-02-28T16:31:35.547 INFO:tasks.workunit.client.1.smithi066.stdout:2/888: write d0/d12/d1a/d39/d7c/fdd [3673411,70420] 0 2024-02-28T16:31:35.551 INFO:tasks.workunit.client.1.smithi066.stdout:9/695: write da/d4b/d53/fa7 [707745,106147] 0 2024-02-28T16:31:35.552 INFO:tasks.workunit.client.1.smithi066.stdout:7/844: read dc/d29/d92/da6/fa7 [1175097,85430] 0 2024-02-28T16:31:35.552 INFO:tasks.workunit.client.1.smithi066.stdout:2/889: chown d0/d12/d1a/d23/d29/f11a 3114 1 2024-02-28T16:31:35.557 INFO:tasks.workunit.client.1.smithi066.stdout:9/696: mknod da/d6b/dbc/ce4 0 2024-02-28T16:31:35.559 INFO:tasks.workunit.client.1.smithi066.stdout:9/697: readlink da/d3a/d58/d97/la6 0 2024-02-28T16:31:35.559 INFO:tasks.workunit.client.1.smithi066.stdout:7/845: symlink dc/d56/dce/d102/db7/dec/l10e 0 2024-02-28T16:31:35.560 INFO:tasks.workunit.client.1.smithi066.stdout:2/890: symlink d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/l11b 0 2024-02-28T16:31:35.561 INFO:tasks.workunit.client.1.smithi066.stdout:9/698: chown da/d22/d46/l4e 9182 1 2024-02-28T16:31:35.563 INFO:tasks.workunit.client.1.smithi066.stdout:7/846: rmdir dc/d56/dce/d102/db7/dec 39 2024-02-28T16:31:35.565 INFO:tasks.workunit.client.1.smithi066.stdout:2/891: unlink d0/d12/d1a/f117 0 2024-02-28T16:31:35.567 INFO:tasks.workunit.client.1.smithi066.stdout:9/699: creat da/d22/d46/d56/d8f/d28/d42/d68/de2/fe5 x:0 0 0 2024-02-28T16:31:35.567 INFO:tasks.workunit.client.1.smithi066.stdout:6/971: dread d2/d2c/f86 [0,4194304] 0 2024-02-28T16:31:35.568 INFO:tasks.workunit.client.1.smithi066.stdout:7/847: symlink dc/d29/d92/da6/dd8/d91/dbf/d10b/l10f 0 2024-02-28T16:31:35.569 INFO:tasks.workunit.client.1.smithi066.stdout:2/892: symlink d0/d12/d1a/d23/ddc/d103/l11c 0 2024-02-28T16:31:35.569 INFO:tasks.workunit.client.1.smithi066.stdout:7/848: write f4 [2035940,73306] 0 2024-02-28T16:31:35.572 INFO:tasks.workunit.client.1.smithi066.stdout:6/972: creat d2/da/dcc/dd6/d102/f154 x:0 0 0 2024-02-28T16:31:35.573 INFO:tasks.workunit.client.1.smithi066.stdout:9/700: truncate da/d4b/d53/dba/fbb 816740 0 2024-02-28T16:31:35.575 INFO:tasks.workunit.client.1.smithi066.stdout:2/893: dread d0/d12/d1a/d39/d7c/d87/dc4/fc8 [0,4194304] 0 2024-02-28T16:31:35.576 INFO:tasks.workunit.client.1.smithi066.stdout:7/849: creat dc/d56/dce/d102/db7/dec/f110 x:0 0 0 2024-02-28T16:31:35.579 INFO:tasks.workunit.client.1.smithi066.stdout:2/894: write d0/d12/d1a/df8/f107 [19046,100052] 0 2024-02-28T16:31:35.579 INFO:tasks.workunit.client.1.smithi066.stdout:6/973: fdatasync d2/d2c/da1/d5d/d63/d6a/d100/f126 0 2024-02-28T16:31:35.581 INFO:tasks.workunit.client.1.smithi066.stdout:7/850: rename dc/de/f3f to dc/de/d87/f111 0 2024-02-28T16:31:35.581 INFO:tasks.workunit.client.1.smithi066.stdout:2/895: symlink d0/d12/d1a/d39/d73/l11d 0 2024-02-28T16:31:35.582 INFO:tasks.workunit.client.1.smithi066.stdout:7/851: chown dc/d29/c40 1549286 1 2024-02-28T16:31:35.582 INFO:tasks.workunit.client.1.smithi066.stdout:6/974: mknod d2/da/d19/d75/c155 0 2024-02-28T16:31:35.592 INFO:tasks.workunit.client.1.smithi066.stdout:5/817: sync 2024-02-28T16:31:35.595 INFO:tasks.workunit.client.1.smithi066.stdout:5/818: chown d3/d4/dc/dd/d2e/lae 33317442 1 2024-02-28T16:31:35.598 INFO:tasks.workunit.client.1.smithi066.stdout:6/975: link d2/f23 d2/d26/dbe/d104/f156 0 2024-02-28T16:31:35.599 INFO:tasks.workunit.client.1.smithi066.stdout:7/852: getdents dc/d56/dce/d102/db7 0 2024-02-28T16:31:35.602 INFO:tasks.workunit.client.1.smithi066.stdout:5/819: truncate d3/d4/dc/dd/d2e/f79 5156661 0 2024-02-28T16:31:35.604 INFO:tasks.workunit.client.1.smithi066.stdout:6/976: rmdir d2/d2c/d6e/df3 39 2024-02-28T16:31:35.607 INFO:tasks.workunit.client.1.smithi066.stdout:5/820: fsync d3/d4/dc/dd/d41/d7a/dac/d57/ff9 0 2024-02-28T16:31:35.608 INFO:tasks.workunit.client.1.smithi066.stdout:7/853: creat dc/d29/f112 x:0 0 0 2024-02-28T16:31:35.609 INFO:tasks.workunit.client.1.smithi066.stdout:6/977: creat d2/da/d19/d75/da6/daa/f157 x:0 0 0 2024-02-28T16:31:35.614 INFO:tasks.workunit.client.1.smithi066.stdout:5/821: rename d3/d4/dc/dd/d2e/d8f to d3/d4/dc/d2f/d55/d74/d104 0 2024-02-28T16:31:35.616 INFO:tasks.workunit.client.1.smithi066.stdout:6/978: rmdir d2/da/d19/d75/d88/dbc 39 2024-02-28T16:31:35.618 INFO:tasks.workunit.client.1.smithi066.stdout:6/979: write d2/d2c/d6e/dd0/f10f [743831,31447] 0 2024-02-28T16:31:35.620 INFO:tasks.workunit.client.1.smithi066.stdout:5/822: creat d3/d4/dc/dd/d35/d86/ded/f105 x:0 0 0 2024-02-28T16:31:35.627 INFO:tasks.workunit.client.1.smithi066.stdout:5/823: symlink d3/d4/dc/dd/d35/l106 0 2024-02-28T16:31:35.631 INFO:tasks.workunit.client.1.smithi066.stdout:7/854: dwrite dc/d29/d92/da6/dd8/d1e/f59 [4194304,4194304] 0 2024-02-28T16:31:35.631 INFO:tasks.workunit.client.1.smithi066.stdout:2/896: dwrite d0/d12/d2e/f9d [0,4194304] 0 2024-02-28T16:31:35.643 INFO:tasks.workunit.client.1.smithi066.stdout:2/897: chown d0/d12/d2e/c88 23772 1 2024-02-28T16:31:35.645 INFO:tasks.workunit.client.1.smithi066.stdout:6/980: dwrite d2/d26/f9e [0,4194304] 0 2024-02-28T16:31:35.649 INFO:tasks.workunit.client.1.smithi066.stdout:2/898: chown d0/d12/d2e/f84 14794996 1 2024-02-28T16:31:35.651 INFO:tasks.workunit.client.1.smithi066.stdout:2/899: stat d0/d12/d1a/d23/d29/f11a 0 2024-02-28T16:31:35.658 INFO:tasks.workunit.client.1.smithi066.stdout:2/900: fsync d0/d12/d1a/d23/f7d 0 2024-02-28T16:31:35.658 INFO:tasks.workunit.client.1.smithi066.stdout:5/824: truncate d3/d4/dc/dd/d41/d7f/ff5 2475540 0 2024-02-28T16:31:35.660 INFO:tasks.workunit.client.1.smithi066.stdout:2/901: chown d0/d12/d3e/d9e/la0 25 1 2024-02-28T16:31:35.662 INFO:tasks.workunit.client.1.smithi066.stdout:7/855: write dc/f25 [1099655,16937] 0 2024-02-28T16:31:35.666 INFO:tasks.workunit.client.1.smithi066.stdout:2/902: mknod d0/d12/d3e/c11e 0 2024-02-28T16:31:35.666 INFO:tasks.workunit.client.1.smithi066.stdout:5/825: creat d3/d4/dc/d2f/d55/d74/f107 x:0 0 0 2024-02-28T16:31:35.666 INFO:tasks.workunit.client.1.smithi066.stdout:6/981: dread d2/da/f3b [0,4194304] 0 2024-02-28T16:31:35.669 INFO:tasks.workunit.client.1.smithi066.stdout:7/856: creat dc/d56/d97/f113 x:0 0 0 2024-02-28T16:31:35.671 INFO:tasks.workunit.client.1.smithi066.stdout:2/903: mknod d0/d12/d1a/d23/ddc/d103/c11f 0 2024-02-28T16:31:35.679 INFO:tasks.workunit.client.1.smithi066.stdout:5/826: mkdir d3/d4/dc/dd/de6/d108 0 2024-02-28T16:31:35.680 INFO:tasks.workunit.client.1.smithi066.stdout:7/857: unlink dc/de/d87/c76 0 2024-02-28T16:31:35.686 INFO:tasks.workunit.client.1.smithi066.stdout:6/982: dwrite d2/da/f3b [0,4194304] 0 2024-02-28T16:31:35.688 INFO:tasks.workunit.client.1.smithi066.stdout:6/983: chown d2/da/dcc/dd6/d102/c123 153 1 2024-02-28T16:31:35.690 INFO:tasks.workunit.client.1.smithi066.stdout:6/984: stat d2/da/d19/d75/da6/d127 0 2024-02-28T16:31:35.691 INFO:tasks.workunit.client.1.smithi066.stdout:6/985: write d2/d26/f9e [4069613,63806] 0 2024-02-28T16:31:35.710 INFO:tasks.workunit.client.1.smithi066.stdout:7/858: creat dc/d56/f114 x:0 0 0 2024-02-28T16:31:35.711 INFO:tasks.workunit.client.1.smithi066.stdout:2/904: mkdir d0/d12/d1a/df8/d120 0 2024-02-28T16:31:35.712 INFO:tasks.workunit.client.1.smithi066.stdout:5/827: rename d3/d4/dc/dd/d2e/fce to d3/d4/dc/dd/d41/d7f/d80/f109 0 2024-02-28T16:31:35.713 INFO:tasks.workunit.client.1.smithi066.stdout:7/859: dread - dc/d29/d92/da6/dd8/d1e/f10c zero size 2024-02-28T16:31:35.716 INFO:tasks.workunit.client.1.smithi066.stdout:6/986: mknod d2/d2c/d6e/c158 0 2024-02-28T16:31:35.717 INFO:tasks.workunit.client.1.smithi066.stdout:7/860: symlink dc/d56/dce/d102/db7/l115 0 2024-02-28T16:31:35.719 INFO:tasks.workunit.client.1.smithi066.stdout:7/861: stat dc/d29/l39 0 2024-02-28T16:31:35.723 INFO:tasks.workunit.client.1.smithi066.stdout:6/987: creat d2/da/dcc/f159 x:0 0 0 2024-02-28T16:31:35.727 INFO:tasks.workunit.client.1.smithi066.stdout:5/828: link d3/d4/dc/dd/d41/d7a/dac/d60/cb0 d3/d4/dc/dd/d41/d7a/dac/d57/c10a 0 2024-02-28T16:31:35.731 INFO:tasks.workunit.client.1.smithi066.stdout:1/919: sync 2024-02-28T16:31:35.731 INFO:tasks.workunit.client.1.smithi066.stdout:6/988: creat d2/da/d19/d9c/de4/f15a x:0 0 0 2024-02-28T16:31:35.731 INFO:tasks.workunit.client.1.smithi066.stdout:1/920: readlink d2/d18/l1b 0 2024-02-28T16:31:35.732 INFO:tasks.workunit.client.1.smithi066.stdout:5/829: mkdir d3/d4/dc/dd/d2e/d10b 0 2024-02-28T16:31:35.732 INFO:tasks.workunit.client.1.smithi066.stdout:6/989: write d2/d26/f2d [765314,89202] 0 2024-02-28T16:31:35.735 INFO:tasks.workunit.client.1.smithi066.stdout:1/921: write d2/d27/dc7/d48/f91 [4318557,35396] 0 2024-02-28T16:31:35.737 INFO:tasks.workunit.client.1.smithi066.stdout:5/830: readlink d3/d4/dc/dd/d2e/lae 0 2024-02-28T16:31:35.741 INFO:tasks.workunit.client.1.smithi066.stdout:6/990: mkdir d2/d26/d32/d15b 0 2024-02-28T16:31:35.742 INFO:tasks.workunit.client.1.smithi066.stdout:2/905: dread d0/f37 [0,4194304] 0 2024-02-28T16:31:35.747 INFO:tasks.workunit.client.1.smithi066.stdout:6/991: read d2/da/d19/d75/da6/daa/dbf/fe3 [2594015,61596] 0 2024-02-28T16:31:35.747 INFO:tasks.workunit.client.1.smithi066.stdout:1/922: creat d2/d18/d6b/dba/f133 x:0 0 0 2024-02-28T16:31:35.749 INFO:tasks.workunit.client.1.smithi066.stdout:2/906: creat d0/d12/d2e/d3b/d66/f121 x:0 0 0 2024-02-28T16:31:35.752 INFO:tasks.workunit.client.1.smithi066.stdout:7/862: dwrite dc/d29/d92/da6/dd8/f19 [0,4194304] 0 2024-02-28T16:31:35.757 INFO:tasks.workunit.client.1.smithi066.stdout:2/907: chown d0/d12/d1a/d23/d29/d21/l32 5846433 1 2024-02-28T16:31:35.758 INFO:tasks.workunit.client.1.smithi066.stdout:6/992: mknod d2/da/d19/c15c 0 2024-02-28T16:31:35.762 INFO:tasks.workunit.client.1.smithi066.stdout:1/923: mknod d2/d27/dc7/d48/c134 0 2024-02-28T16:31:35.766 INFO:tasks.workunit.client.1.smithi066.stdout:7/863: mknod dc/d29/d92/c116 0 2024-02-28T16:31:35.767 INFO:tasks.workunit.client.1.smithi066.stdout:6/993: truncate d2/da/d19/d75/d88/fce 1727612 0 2024-02-28T16:31:35.768 INFO:tasks.workunit.client.1.smithi066.stdout:9/701: sync 2024-02-28T16:31:35.768 INFO:tasks.workunit.client.1.smithi066.stdout:8/917: sync 2024-02-28T16:31:35.768 INFO:tasks.workunit.client.1.smithi066.stdout:1/924: creat d2/d18/da5/f135 x:0 0 0 2024-02-28T16:31:35.772 INFO:tasks.workunit.client.1.smithi066.stdout:8/918: write d0/d6/dc/ff7 [24185,54371] 0 2024-02-28T16:31:35.772 INFO:tasks.workunit.client.1.smithi066.stdout:7/864: mkdir dc/d56/d117 0 2024-02-28T16:31:35.773 INFO:tasks.workunit.client.1.smithi066.stdout:1/925: chown d2/f7 0 1 2024-02-28T16:31:35.777 INFO:tasks.workunit.client.1.smithi066.stdout:1/926: dread - d2/d27/d2b/d34/d9e/fae zero size 2024-02-28T16:31:35.777 INFO:tasks.workunit.client.1.smithi066.stdout:6/994: creat d2/d2c/da1/d5d/d63/d6a/d100/f15d x:0 0 0 2024-02-28T16:31:35.780 INFO:tasks.workunit.client.1.smithi066.stdout:6/995: fdatasync d2/d26/de0/f134 0 2024-02-28T16:31:35.781 INFO:tasks.workunit.client.1.smithi066.stdout:8/919: unlink d0/db/d12/d1d/d25/d4f/f86 0 2024-02-28T16:31:35.785 INFO:tasks.workunit.client.1.smithi066.stdout:1/927: creat d2/d18/d6b/f136 x:0 0 0 2024-02-28T16:31:35.787 INFO:tasks.workunit.client.1.smithi066.stdout:1/928: chown d2/d27/f2f 5 1 2024-02-28T16:31:35.794 INFO:tasks.workunit.client.1.smithi066.stdout:6/996: creat d2/da/f15e x:0 0 0 2024-02-28T16:31:35.798 INFO:tasks.workunit.client.1.smithi066.stdout:6/997: creat d2/da/d19/d9c/de4/f15f x:0 0 0 2024-02-28T16:31:35.810 INFO:tasks.workunit.client.1.smithi066.stdout:2/908: write d0/d12/d1a/f54 [880575,36845] 0 2024-02-28T16:31:35.812 INFO:tasks.workunit.client.1.smithi066.stdout:5/831: dwrite d3/d4/dc/dd/d2e/d18/fd5 [0,4194304] 0 2024-02-28T16:31:35.817 INFO:tasks.workunit.client.1.smithi066.stdout:7/865: truncate dc/d56/d97/ff1 2305133 0 2024-02-28T16:31:35.817 INFO:tasks.workunit.client.1.smithi066.stdout:1/929: dread d2/d27/f2f [0,4194304] 0 2024-02-28T16:31:35.819 INFO:tasks.workunit.client.1.smithi066.stdout:8/920: dread d0/db/d15/f31 [0,4194304] 0 2024-02-28T16:31:35.819 INFO:tasks.workunit.client.1.smithi066.stdout:5/832: readlink d3/d4/dc/dd/lef 0 2024-02-28T16:31:35.824 INFO:tasks.workunit.client.1.smithi066.stdout:9/702: dwrite da/d22/d46/f74 [4194304,4194304] 0 2024-02-28T16:31:35.824 INFO:tasks.workunit.client.1.smithi066.stdout:1/930: read d2/d27/d2b/f39 [390256,94100] 0 2024-02-28T16:31:35.824 INFO:tasks.workunit.client.1.smithi066.stdout:7/866: fsync dc/d56/fad 0 2024-02-28T16:31:35.826 INFO:tasks.workunit.client.1.smithi066.stdout:8/921: mkdir d0/d6/dea/d12f 0 2024-02-28T16:31:35.829 INFO:tasks.workunit.client.1.smithi066.stdout:2/909: rename d0/d12/d1a/d23/d29/d21/d2f/c69 to d0/d12/d1a/d23/ddc/d103/c122 0 2024-02-28T16:31:35.833 INFO:tasks.workunit.client.1.smithi066.stdout:1/931: mkdir d2/d27/d2b/d137 0 2024-02-28T16:31:35.833 INFO:tasks.workunit.client.1.smithi066.stdout:7/867: readlink dc/d29/d92/da6/dd8/lcf 0 2024-02-28T16:31:35.841 INFO:tasks.workunit.client.1.smithi066.stdout:6/998: dwrite d2/fc1 [0,4194304] 0 2024-02-28T16:31:35.841 INFO:tasks.workunit.client.1.smithi066.stdout:8/922: rmdir d0/d6/dc/d16/d4c/d6c 39 2024-02-28T16:31:35.847 INFO:tasks.workunit.client.1.smithi066.stdout:2/910: creat d0/d12/d1a/d39/d7c/d87/dc4/f123 x:0 0 0 2024-02-28T16:31:35.855 INFO:tasks.workunit.client.1.smithi066.stdout:2/911: chown d0/d12/d2e/d3b/d66/f7f 4017937 1 2024-02-28T16:31:35.856 INFO:tasks.workunit.client.1.smithi066.stdout:1/932: mkdir d2/d27/dc7/d51/db4/dbf/d138 0 2024-02-28T16:31:35.856 INFO:tasks.workunit.client.1.smithi066.stdout:6/999: symlink d2/da/d19/d9c/d3d/d153/l160 0 2024-02-28T16:31:35.861 INFO:tasks.workunit.client.1.smithi066.stdout:2/912: rmdir d0/d12/d1a/d39 39 2024-02-28T16:31:35.861 INFO:tasks.workunit.client.1.smithi066.stdout:5/833: dwrite d3/d4/dc/dd/d41/d7a/dac/d60/fb2 [0,4194304] 0 2024-02-28T16:31:35.864 INFO:tasks.workunit.client.1.smithi066.stdout:7/868: link dc/de/c68 dc/d29/d92/da6/dd8/d91/dba/de3/c118 0 2024-02-28T16:31:35.869 INFO:tasks.workunit.client.1.smithi066.stdout:9/703: dwrite da/d22/d46/d56/d8f/d94/fd0 [0,4194304] 0 2024-02-28T16:31:35.880 INFO:tasks.workunit.client.1.smithi066.stdout:8/923: dwrite d0/d6/dc/d16/d1a/ffa [4194304,4194304] 0 2024-02-28T16:31:35.883 INFO:tasks.workunit.client.1.smithi066.stdout:1/933: creat d2/d18/d7a/dfd/d111/f139 x:0 0 0 2024-02-28T16:31:35.893 INFO:tasks.workunit.client.1.smithi066.stdout:5/834: creat d3/d1b/d20/f10c x:0 0 0 2024-02-28T16:31:35.897 INFO:tasks.workunit.client.1.smithi066.stdout:1/934: read d2/d27/d132/f8c [259490,38363] 0 2024-02-28T16:31:35.899 INFO:tasks.workunit.client.1.smithi066.stdout:5/835: dread - d3/d4/dc/dd/d41/d7f/d80/fc9 zero size 2024-02-28T16:31:35.903 INFO:tasks.workunit.client.1.smithi066.stdout:2/913: dread d0/f44 [0,4194304] 0 2024-02-28T16:31:35.906 INFO:tasks.workunit.client.1.smithi066.stdout:1/935: rmdir d2/d27/dc7/d54/de7 39 2024-02-28T16:31:35.910 INFO:tasks.workunit.client.1.smithi066.stdout:7/869: dwrite dc/d29/d92/da6/dd8/d1e/f98 [4194304,4194304] 0 2024-02-28T16:31:35.917 INFO:tasks.workunit.client.1.smithi066.stdout:2/914: write d0/d12/d1a/d39/d7c/d87/d104/f114 [2127697,111900] 0 2024-02-28T16:31:35.921 INFO:tasks.workunit.client.1.smithi066.stdout:1/936: symlink d2/d18/d7a/d110/l13a 0 2024-02-28T16:31:35.925 INFO:tasks.workunit.client.1.smithi066.stdout:2/915: fdatasync d0/d12/d1a/d23/fa6 0 2024-02-28T16:31:35.925 INFO:tasks.workunit.client.1.smithi066.stdout:8/924: dwrite d0/d6/dc/d16/d1a/f88 [0,4194304] 0 2024-02-28T16:31:35.931 INFO:tasks.workunit.client.1.smithi066.stdout:9/704: dread da/f20 [0,4194304] 0 2024-02-28T16:31:35.931 INFO:tasks.workunit.client.1.smithi066.stdout:2/916: write d0/d12/f82 [1282994,29529] 0 2024-02-28T16:31:35.935 INFO:tasks.workunit.client.1.smithi066.stdout:5/836: dwrite d3/d1b/d7d/fd1 [0,4194304] 0 2024-02-28T16:31:35.943 INFO:tasks.workunit.client.1.smithi066.stdout:8/925: chown d0/db/d12/d38/l3e 716 1 2024-02-28T16:31:35.947 INFO:tasks.workunit.client.1.smithi066.stdout:1/937: truncate d2/d27/d2b/d43/d46/d7c/f116 842351 0 2024-02-28T16:31:35.949 INFO:tasks.workunit.client.1.smithi066.stdout:9/705: rename da/d3a/f47 to da/d6b/dbc/fe6 0 2024-02-28T16:31:35.952 INFO:tasks.workunit.client.1.smithi066.stdout:5/837: dread d3/d4/dc/f21 [0,4194304] 0 2024-02-28T16:31:35.955 INFO:tasks.workunit.client.1.smithi066.stdout:8/926: chown d0/d6/dc/d16/d1a/d63/l3b 15 1 2024-02-28T16:31:35.957 INFO:tasks.workunit.client.1.smithi066.stdout:5/838: write d3/d1b/d7d/fd1 [5206716,83416] 0 2024-02-28T16:31:35.958 INFO:tasks.workunit.client.1.smithi066.stdout:1/938: mkdir d2/d27/d2b/d43/d13b 0 2024-02-28T16:31:35.959 INFO:tasks.workunit.client.1.smithi066.stdout:7/870: dwrite dc/d29/d92/da6/dd8/d1e/d45/f8f [0,4194304] 0 2024-02-28T16:31:35.969 INFO:tasks.workunit.client.1.smithi066.stdout:7/871: dread - dc/d56/dce/d102/fc1 zero size 2024-02-28T16:31:35.969 INFO:tasks.workunit.client.1.smithi066.stdout:1/939: creat d2/d18/da5/db2/f13c x:0 0 0 2024-02-28T16:31:35.978 INFO:tasks.workunit.client.1.smithi066.stdout:2/917: dwrite d0/f1f [0,4194304] 0 2024-02-28T16:31:35.978 INFO:tasks.workunit.client.1.smithi066.stdout:7/872: chown dc/d29/d92/da6/dd8/d1e/l35 257950710 1 2024-02-28T16:31:35.987 INFO:tasks.workunit.client.1.smithi066.stdout:7/873: readlink dc/d29/d92/da6/dd8/l46 0 2024-02-28T16:31:35.988 INFO:tasks.workunit.client.1.smithi066.stdout:9/706: write da/d3a/d58/dab/fd2 [494035,100159] 0 2024-02-28T16:31:35.990 INFO:tasks.workunit.client.1.smithi066.stdout:2/918: creat d0/d12/d1a/d23/d29/d21/d2f/de3/f124 x:0 0 0 2024-02-28T16:31:35.991 INFO:tasks.workunit.client.1.smithi066.stdout:8/927: write d0/d6/dc/d16/d4c/d42/d73/fe9 [796765,19608] 0 2024-02-28T16:31:35.992 INFO:tasks.workunit.client.1.smithi066.stdout:7/874: readlink dc/d56/dce/ldc 0 2024-02-28T16:31:35.993 INFO:tasks.workunit.client.1.smithi066.stdout:9/707: symlink da/d3a/d58/le7 0 2024-02-28T16:31:36.001 INFO:tasks.workunit.client.1.smithi066.stdout:8/928: mknod d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db1/dad/db9/c130 0 2024-02-28T16:31:36.004 INFO:tasks.workunit.client.1.smithi066.stdout:2/919: write d0/d12/d2e/d7a/fba [1140103,33915] 0 2024-02-28T16:31:36.007 INFO:tasks.workunit.client.1.smithi066.stdout:1/940: dwrite d2/d27/dc7/d48/f81 [0,4194304] 0 2024-02-28T16:31:36.009 INFO:tasks.workunit.client.1.smithi066.stdout:8/929: creat d0/f131 x:0 0 0 2024-02-28T16:31:36.014 INFO:tasks.workunit.client.1.smithi066.stdout:9/708: dwrite da/d4b/d53/fa7 [0,4194304] 0 2024-02-28T16:31:36.015 INFO:tasks.workunit.client.1.smithi066.stdout:7/875: dread f5 [0,4194304] 0 2024-02-28T16:31:36.018 INFO:tasks.workunit.client.1.smithi066.stdout:8/930: dread d0/db/d12/d1d/d25/d65/ff8 [0,4194304] 0 2024-02-28T16:31:36.020 INFO:tasks.workunit.client.1.smithi066.stdout:9/709: chown da/d22/d46/d56/d8f/d28/dde 459911 1 2024-02-28T16:31:36.022 INFO:tasks.workunit.client.1.smithi066.stdout:1/941: mkdir d2/d27/dc7/d54/de7/d13d 0 2024-02-28T16:31:36.023 INFO:tasks.workunit.client.1.smithi066.stdout:7/876: creat dc/d56/f119 x:0 0 0 2024-02-28T16:31:36.025 INFO:tasks.workunit.client.1.smithi066.stdout:2/920: getdents d0/d12/d1a/d23/d29/d21/d2f/d67/d8d 0 2024-02-28T16:31:36.027 INFO:tasks.workunit.client.1.smithi066.stdout:1/942: symlink d2/d18/d7a/db1/l13e 0 2024-02-28T16:31:36.028 INFO:tasks.workunit.client.1.smithi066.stdout:2/921: stat d0/d12/d1a/d39/d7c/d87/dc4/ff1 0 2024-02-28T16:31:36.030 INFO:tasks.workunit.client.1.smithi066.stdout:7/877: rmdir dc/d29/d92/da6/dd8/d1e/d45/d9f 39 2024-02-28T16:31:36.032 INFO:tasks.workunit.client.1.smithi066.stdout:1/943: truncate d2/d27/fa3 4709401 0 2024-02-28T16:31:36.042 INFO:tasks.workunit.client.1.smithi066.stdout:1/944: symlink d2/d27/d2b/d34/l13f 0 2024-02-28T16:31:36.042 INFO:tasks.workunit.client.1.smithi066.stdout:9/710: dwrite da/fc [0,4194304] 0 2024-02-28T16:31:36.043 INFO:tasks.workunit.client.1.smithi066.stdout:8/931: truncate d0/d8/fc7 1359095 0 2024-02-28T16:31:36.047 INFO:tasks.workunit.client.1.smithi066.stdout:1/945: write d2/d18/d19/ff2 [3472281,121920] 0 2024-02-28T16:31:36.050 INFO:tasks.workunit.client.1.smithi066.stdout:8/932: write d0/d6/dc/d16/d4c/d42/f10c [562098,87101] 0 2024-02-28T16:31:36.051 INFO:tasks.workunit.client.1.smithi066.stdout:7/878: write dc/d29/d92/da6/dd8/d1e/f20 [312048,9734] 0 2024-02-28T16:31:36.052 INFO:tasks.workunit.client.1.smithi066.stdout:2/922: truncate d0/d12/d1a/d23/d29/fa8 4432309 0 2024-02-28T16:31:36.054 INFO:tasks.workunit.client.1.smithi066.stdout:9/711: read da/f64 [288904,101647] 0 2024-02-28T16:31:36.056 INFO:tasks.workunit.client.1.smithi066.stdout:8/933: stat d0/d6/dc/d16/d4c/d42/f10c 0 2024-02-28T16:31:36.057 INFO:tasks.workunit.client.1.smithi066.stdout:2/923: read - d0/d12/d2e/d3b/fb4 zero size 2024-02-28T16:31:36.058 INFO:tasks.workunit.client.1.smithi066.stdout:9/712: chown da/d3a/d58/d97/ca2 603184 1 2024-02-28T16:31:36.060 INFO:tasks.workunit.client.1.smithi066.stdout:1/946: dread d2/d18/d7a/dbc/f11b [0,4194304] 0 2024-02-28T16:31:36.061 INFO:tasks.workunit.client.1.smithi066.stdout:8/934: stat d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/d106/d118/f129 0 2024-02-28T16:31:36.062 INFO:tasks.workunit.client.1.smithi066.stdout:2/924: read d0/d12/d1a/d23/d29/d21/d2f/fa2 [2692272,10548] 0 2024-02-28T16:31:36.076 INFO:tasks.workunit.client.1.smithi066.stdout:9/713: rename da/d22/d46/d56/d8f/d28/d42/d68/c88 to da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3/ce8 0 2024-02-28T16:31:36.079 INFO:tasks.workunit.client.1.smithi066.stdout:2/925: write d0/d12/d1a/ffb [339889,9715] 0 2024-02-28T16:31:36.088 INFO:tasks.workunit.client.1.smithi066.stdout:9/714: mknod da/d3a/ce9 0 2024-02-28T16:31:36.089 INFO:tasks.workunit.client.1.smithi066.stdout:1/947: dwrite d2/d27/d2b/d43/d46/fb6 [0,4194304] 0 2024-02-28T16:31:36.091 INFO:tasks.workunit.client.1.smithi066.stdout:8/935: dwrite d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db1/f11c [0,4194304] 0 2024-02-28T16:31:36.097 INFO:tasks.workunit.client.1.smithi066.stdout:9/715: creat da/d22/d46/d56/d8f/d28/d42/d68/fea x:0 0 0 2024-02-28T16:31:36.100 INFO:tasks.workunit.client.1.smithi066.stdout:8/936: mknod d0/db/c132 0 2024-02-28T16:31:36.100 INFO:tasks.workunit.client.1.smithi066.stdout:7/879: dwrite dc/d56/f80 [0,4194304] 0 2024-02-28T16:31:36.101 INFO:tasks.workunit.client.1.smithi066.stdout:1/948: mknod d2/d27/dc7/da6/dd4/c140 0 2024-02-28T16:31:36.102 INFO:tasks.workunit.client.1.smithi066.stdout:1/949: write d2/d27/f3c [4922357,45716] 0 2024-02-28T16:31:36.104 INFO:tasks.workunit.client.1.smithi066.stdout:9/716: rmdir da/d6b/dbc 39 2024-02-28T16:31:36.113 INFO:tasks.workunit.client.1.smithi066.stdout:7/880: mkdir dc/de/d87/d11a 0 2024-02-28T16:31:36.116 INFO:tasks.workunit.client.1.smithi066.stdout:9/717: creat da/d22/d46/d56/d8f/d28/d42/feb x:0 0 0 2024-02-28T16:31:36.116 INFO:tasks.workunit.client.1.smithi066.stdout:2/926: dwrite d0/d12/d2e/d7a/fba [0,4194304] 0 2024-02-28T16:31:36.121 INFO:tasks.workunit.client.1.smithi066.stdout:7/881: dread - dc/d56/dce/dff/fe9 zero size 2024-02-28T16:31:36.125 INFO:tasks.workunit.client.1.smithi066.stdout:9/718: chown da/d3a/c17 3460 1 2024-02-28T16:31:36.132 INFO:tasks.workunit.client.1.smithi066.stdout:7/882: truncate dc/d29/d92/da6/dd8/fd0 113304 0 2024-02-28T16:31:36.133 INFO:tasks.workunit.client.1.smithi066.stdout:8/937: dwrite d0/f13 [0,4194304] 0 2024-02-28T16:31:36.134 INFO:tasks.workunit.client.1.smithi066.stdout:9/719: getdents da/d3a/d15/d48/d76 0 2024-02-28T16:31:36.143 INFO:tasks.workunit.client.1.smithi066.stdout:9/720: dread - da/fb2 zero size 2024-02-28T16:31:36.145 INFO:tasks.workunit.client.1.smithi066.stdout:7/883: readlink dc/d56/dce/d102/db7/lc2 0 2024-02-28T16:31:36.145 INFO:tasks.workunit.client.1.smithi066.stdout:1/950: dwrite d2/d27/d2b/d43/faf [0,4194304] 0 2024-02-28T16:31:36.148 INFO:tasks.workunit.client.1.smithi066.stdout:8/938: unlink d0/db/d12/d1d/d25/d4f/f82 0 2024-02-28T16:31:36.151 INFO:tasks.workunit.client.1.smithi066.stdout:9/721: creat da/d22/fec x:0 0 0 2024-02-28T16:31:36.155 INFO:tasks.workunit.client.1.smithi066.stdout:1/951: rename d2/d27/dc7/d48/dd6 to d2/d27/d125/d141 0 2024-02-28T16:31:36.156 INFO:tasks.workunit.client.1.smithi066.stdout:9/722: chown da/d22/d46/d56/d8f/d28/d42/f80 1 1 2024-02-28T16:31:36.159 INFO:tasks.workunit.client.1.smithi066.stdout:7/884: mknod dc/d29/c11b 0 2024-02-28T16:31:36.160 INFO:tasks.workunit.client.1.smithi066.stdout:8/939: creat d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/d106/f133 x:0 0 0 2024-02-28T16:31:36.161 INFO:tasks.workunit.client.1.smithi066.stdout:9/723: chown da/d22/d46/d56/d8f/d28/d42/fc6 6753281 1 2024-02-28T16:31:36.164 INFO:tasks.workunit.client.1.smithi066.stdout:1/952: creat d2/d27/dc7/da6/d12e/f142 x:0 0 0 2024-02-28T16:31:36.168 INFO:tasks.workunit.client.1.smithi066.stdout:8/940: stat d0/d6/dc/d16/d1a/d63/d58/d5a/d108 0 2024-02-28T16:31:36.168 INFO:tasks.workunit.client.1.smithi066.stdout:2/927: dread d0/d12/d1a/f54 [0,4194304] 0 2024-02-28T16:31:36.175 INFO:tasks.workunit.client.1.smithi066.stdout:8/941: write d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/ffb [523712,128366] 0 2024-02-28T16:31:36.176 INFO:tasks.workunit.client.1.smithi066.stdout:8/942: stat d0/db/d15/d95/lbe 0 2024-02-28T16:31:36.180 INFO:tasks.workunit.client.1.smithi066.stdout:8/943: chown d0/d6/dc/d16/d4c/d6c/d6f/da7 9908 1 2024-02-28T16:31:36.181 INFO:tasks.workunit.client.1.smithi066.stdout:8/944: read - d0/d6/dc/f101 zero size 2024-02-28T16:31:36.181 INFO:tasks.workunit.client.1.smithi066.stdout:7/885: dwrite dc/d29/d92/da6/dd8/d1e/f3e [0,4194304] 0 2024-02-28T16:31:36.186 INFO:tasks.workunit.client.1.smithi066.stdout:8/945: read - d0/d6/dc/d16/d1a/d63/d3d/f127 zero size 2024-02-28T16:31:36.232 INFO:tasks.workunit.client.1.smithi066.stdout:8/946: unlink d0/d6/f5d 0 2024-02-28T16:31:36.233 INFO:tasks.workunit.client.1.smithi066.stdout:8/947: chown d0/db/d15/f31 214 1 2024-02-28T16:31:36.239 INFO:tasks.workunit.client.1.smithi066.stdout:8/948: creat d0/d6/dc/d16/d1a/d63/d58/d5a/f134 x:0 0 0 2024-02-28T16:31:36.245 INFO:tasks.workunit.client.1.smithi066.stdout:7/886: dwrite dc/d29/d92/da6/dd8/d1e/f6c [0,4194304] 0 2024-02-28T16:31:36.246 INFO:tasks.workunit.client.1.smithi066.stdout:7/887: dread - dc/d56/dc3/ff5 zero size 2024-02-28T16:31:36.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:35 smithi066 ceph-mon[36236]: pgmap v61: 65 pgs: 65 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 101 MiB/s rd, 224 MiB/s wr, 490 op/s 2024-02-28T16:31:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:35 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:35 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:36.249 INFO:tasks.workunit.client.1.smithi066.stdout:7/888: write dc/d29/d92/da6/fb1 [689877,26576] 0 2024-02-28T16:31:36.249 INFO:tasks.workunit.client.1.smithi066.stdout:8/949: link d0/d6/dc/d16/d1a/d63/d58/d5a/c83 d0/db/d12/d1d/d25/de2/c135 0 2024-02-28T16:31:36.253 INFO:tasks.workunit.client.1.smithi066.stdout:8/950: fsync d0/db/d12/f4b 0 2024-02-28T16:31:36.259 INFO:tasks.workunit.client.1.smithi066.stdout:8/951: rmdir d0/db/d12/d1d/d25/d65/dbb 39 2024-02-28T16:31:36.261 INFO:tasks.workunit.client.1.smithi066.stdout:8/952: fdatasync d0/db/d12/d8a/f116 0 2024-02-28T16:31:36.264 INFO:tasks.workunit.client.1.smithi066.stdout:8/953: write d0/db/d12/d1d/d25/de2/fca [1355767,102793] 0 2024-02-28T16:31:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:35 smithi001 ceph-mon[28991]: pgmap v61: 65 pgs: 65 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 101 MiB/s rd, 224 MiB/s wr, 490 op/s 2024-02-28T16:31:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:35 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:35 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:36.270 INFO:tasks.workunit.client.1.smithi066.stdout:8/954: rename d0/d6/dc/d16/d1a/d63/d3d/fab to d0/db/d12/d1d/d25/de2/f136 0 2024-02-28T16:31:36.272 INFO:tasks.workunit.client.1.smithi066.stdout:7/889: dread dc/de/f13 [0,4194304] 0 2024-02-28T16:31:36.274 INFO:tasks.workunit.client.1.smithi066.stdout:8/955: unlink d0/d6/dc/d16/d4c/d42/d73/f128 0 2024-02-28T16:31:36.276 INFO:tasks.workunit.client.1.smithi066.stdout:7/890: creat dc/d56/dc3/f11c x:0 0 0 2024-02-28T16:31:36.279 INFO:tasks.workunit.client.1.smithi066.stdout:7/891: mknod dc/d29/d92/da6/dd8/d37/df7/d100/c11d 0 2024-02-28T16:31:36.282 INFO:tasks.workunit.client.1.smithi066.stdout:7/892: dread - dc/d56/dce/d102/db7/dec/f110 zero size 2024-02-28T16:31:36.283 INFO:tasks.workunit.client.1.smithi066.stdout:7/893: chown dc/d29/d92/da6/dd8/l1c 10 1 2024-02-28T16:31:36.288 INFO:tasks.workunit.client.1.smithi066.stdout:7/894: truncate dc/d29/d92/da6/fb2 586980 0 2024-02-28T16:31:36.289 INFO:tasks.workunit.client.1.smithi066.stdout:7/895: chown dc/d56/dc3/f11c 63 1 2024-02-28T16:31:36.291 INFO:tasks.workunit.client.1.smithi066.stdout:8/956: dwrite d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/ffb [0,4194304] 0 2024-02-28T16:31:36.295 INFO:tasks.workunit.client.1.smithi066.stdout:8/957: truncate d0/d6/dc/d16/d4c/d42/d73/fe9 1640915 0 2024-02-28T16:31:36.298 INFO:tasks.workunit.client.1.smithi066.stdout:7/896: dread dc/f7f [0,4194304] 0 2024-02-28T16:31:36.300 INFO:tasks.workunit.client.1.smithi066.stdout:8/958: mkdir d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db1/dad/d11b/d137 0 2024-02-28T16:31:36.301 INFO:tasks.workunit.client.1.smithi066.stdout:7/897: symlink dc/d56/l11e 0 2024-02-28T16:31:36.302 INFO:tasks.workunit.client.1.smithi066.stdout:8/959: write d0/f13 [3284153,65946] 0 2024-02-28T16:31:36.306 INFO:tasks.workunit.client.1.smithi066.stdout:7/898: rename dc/f4c to dc/d56/d97/f11f 0 2024-02-28T16:31:36.321 INFO:tasks.workunit.client.1.smithi066.stdout:7/899: dwrite dc/d6e/f6f [4194304,4194304] 0 2024-02-28T16:31:36.324 INFO:tasks.workunit.client.1.smithi066.stdout:7/900: chown dc/d56/f119 869 1 2024-02-28T16:31:36.329 INFO:tasks.workunit.client.1.smithi066.stdout:7/901: creat dc/d29/d92/da6/dd8/d91/dbf/d10b/f120 x:0 0 0 2024-02-28T16:31:36.332 INFO:tasks.workunit.client.1.smithi066.stdout:8/960: dwrite d0/db/d12/d1d/d25/f75 [0,4194304] 0 2024-02-28T16:31:36.336 INFO:tasks.workunit.client.1.smithi066.stdout:7/902: truncate dc/d29/d92/da6/dd8/f7c 4907319 0 2024-02-28T16:31:36.338 INFO:tasks.workunit.client.1.smithi066.stdout:8/961: read d0/db/ff1 [2678806,2591] 0 2024-02-28T16:31:36.340 INFO:tasks.workunit.client.1.smithi066.stdout:7/903: symlink dc/d56/dce/d102/l121 0 2024-02-28T16:31:36.341 INFO:tasks.workunit.client.1.smithi066.stdout:8/962: read - d0/d6/dc/d16/d1a/d63/d3d/f127 zero size 2024-02-28T16:31:36.348 INFO:tasks.workunit.client.1.smithi066.stdout:7/904: link dc/d29/d92/da6/dd8/d1e/f24 dc/d29/d92/da6/dd8/d91/dba/de3/f122 0 2024-02-28T16:31:36.353 INFO:tasks.workunit.client.1.smithi066.stdout:8/963: dread d0/db/d15/f31 [0,4194304] 0 2024-02-28T16:31:36.353 INFO:tasks.workunit.client.1.smithi066.stdout:7/905: getdents dc/d29/d92/da6/dd8/d37 0 2024-02-28T16:31:36.355 INFO:tasks.workunit.client.1.smithi066.stdout:8/964: chown d0/d6/cd1 450 1 2024-02-28T16:31:36.360 INFO:tasks.workunit.client.1.smithi066.stdout:8/965: rename d0/d8 to d0/db/d12/d138 0 2024-02-28T16:31:36.365 INFO:tasks.workunit.client.1.smithi066.stdout:7/906: dread dc/d29/d92/da6/dd8/d1e/f59 [4194304,4194304] 0 2024-02-28T16:31:36.368 INFO:tasks.workunit.client.1.smithi066.stdout:7/907: rmdir dc/d29/d92/da6/dd8/d91/dba/df8 39 2024-02-28T16:31:36.370 INFO:tasks.workunit.client.1.smithi066.stdout:8/966: truncate d0/db/d12/d1d/d25/f75 2448534 0 2024-02-28T16:31:36.373 INFO:tasks.workunit.client.1.smithi066.stdout:7/908: unlink dc/d29/d92/da6/dd8/d1e/f3c 0 2024-02-28T16:31:36.376 INFO:tasks.workunit.client.1.smithi066.stdout:7/909: write dc/d29/d92/da6/fb1 [1425135,1563] 0 2024-02-28T16:31:36.376 INFO:tasks.workunit.client.1.smithi066.stdout:8/967: creat d0/d6/dc/d16/d1a/d63/d58/d5a/d10b/f139 x:0 0 0 2024-02-28T16:31:36.379 INFO:tasks.workunit.client.1.smithi066.stdout:7/910: rename dc/d29/l39 to dc/d56/l123 0 2024-02-28T16:31:36.380 INFO:tasks.workunit.client.1.smithi066.stdout:8/968: read d0/d6/dc/d16/d4c/d42/fb2 [423882,26137] 0 2024-02-28T16:31:36.383 INFO:tasks.workunit.client.1.smithi066.stdout:7/911: symlink dc/d29/d92/da6/dd8/d1e/d45/d9f/l124 0 2024-02-28T16:31:36.384 INFO:tasks.workunit.client.1.smithi066.stdout:8/969: rmdir d0/d6/dc/d16/d4c/d6c/d6f/da7 39 2024-02-28T16:31:36.386 INFO:tasks.workunit.client.1.smithi066.stdout:7/912: creat dc/d56/dce/d102/f125 x:0 0 0 2024-02-28T16:31:36.387 INFO:tasks.workunit.client.1.smithi066.stdout:8/970: chown d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/db1/lc6 4267 1 2024-02-28T16:31:36.388 INFO:tasks.workunit.client.1.smithi066.stdout:8/971: write d0/d6/f1f [1391568,31964] 0 2024-02-28T16:31:36.392 INFO:tasks.workunit.client.1.smithi066.stdout:8/972: creat d0/d6/dea/d12f/f13a x:0 0 0 2024-02-28T16:31:36.404 INFO:tasks.workunit.client.1.smithi066.stdout:7/913: dread dc/d29/f2e [0,4194304] 0 2024-02-28T16:31:36.410 INFO:tasks.workunit.client.1.smithi066.stdout:7/914: rename dc/d29/d92/da6/dd8/d37/df7 to dc/d56/dce/d126 0 2024-02-28T16:31:36.411 INFO:tasks.workunit.client.1.smithi066.stdout:8/973: dwrite d0/db/d12/d1d/d25/d65/d93/f99 [0,4194304] 0 2024-02-28T16:31:36.416 INFO:tasks.workunit.client.1.smithi066.stdout:8/974: chown d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/fa5 5436 1 2024-02-28T16:31:36.420 INFO:tasks.workunit.client.1.smithi066.stdout:7/915: write dc/de/d87/fa9 [838277,15856] 0 2024-02-28T16:31:36.422 INFO:tasks.workunit.client.1.smithi066.stdout:7/916: chown dc/d56/dce/d126/d100/f105 1605 1 2024-02-28T16:31:36.423 INFO:tasks.workunit.client.1.smithi066.stdout:8/975: dread - d0/d6/dc/d16/d4c/fdb zero size 2024-02-28T16:31:36.425 INFO:tasks.workunit.client.1.smithi066.stdout:8/976: chown d0/d6/dc/d16/d1a/d63/d58/dac 235311106 1 2024-02-28T16:31:36.426 INFO:tasks.workunit.client.1.smithi066.stdout:7/917: write dc/d29/d92/da6/dd8/d91/dba/dbb/ffa [1211525,111078] 0 2024-02-28T16:31:36.430 INFO:tasks.workunit.client.1.smithi066.stdout:8/977: readlink d0/d6/dc/l103 0 2024-02-28T16:31:36.430 INFO:tasks.workunit.client.1.smithi066.stdout:7/918: mkdir dc/d29/d92/df2/dfc/d127 0 2024-02-28T16:31:36.431 INFO:tasks.workunit.client.1.smithi066.stdout:8/978: write d0/d6/dc/d16/d1a/d63/d3d/f127 [21253,82602] 0 2024-02-28T16:31:36.436 INFO:tasks.workunit.client.1.smithi066.stdout:8/979: mkdir d0/d6/dc/d16/d1a/d63/d3d/d13b 0 2024-02-28T16:31:36.445 INFO:tasks.workunit.client.1.smithi066.stdout:5/839: sync 2024-02-28T16:31:36.448 INFO:tasks.workunit.client.1.smithi066.stdout:7/919: dwrite dc/f5d [0,4194304] 0 2024-02-28T16:31:36.455 INFO:tasks.workunit.client.1.smithi066.stdout:8/980: dwrite d0/db/d12/d1d/d25/d4f/f9f [4194304,4194304] 0 2024-02-28T16:31:36.462 INFO:tasks.workunit.client.1.smithi066.stdout:5/840: rename d3/d4/dc/dd/d41/l78 to d3/d4/dc/d2f/d55/d74/d104/l10d 0 2024-02-28T16:31:36.462 INFO:tasks.workunit.client.1.smithi066.stdout:8/981: dread - d0/db/d15/d95/fcb zero size 2024-02-28T16:31:36.463 INFO:tasks.workunit.client.1.smithi066.stdout:8/982: chown d0/d6/dc/d16 550 1 2024-02-28T16:31:36.465 INFO:tasks.workunit.client.1.smithi066.stdout:5/841: dread - d3/d4/dc/dd/d41/d7f/d80/fc9 zero size 2024-02-28T16:31:36.478 INFO:tasks.workunit.client.1.smithi066.stdout:7/920: write dc/d29/d92/da6/dd8/d1e/fa3 [5546260,24154] 0 2024-02-28T16:31:36.484 INFO:tasks.workunit.client.1.smithi066.stdout:5/842: write d3/d1b/fcb [1079628,100534] 0 2024-02-28T16:31:36.488 INFO:tasks.workunit.client.1.smithi066.stdout:7/921: read - dc/d29/d92/da6/dd8/fe1 zero size 2024-02-28T16:31:36.489 INFO:tasks.workunit.client.1.smithi066.stdout:9/724: sync 2024-02-28T16:31:36.489 INFO:tasks.workunit.client.1.smithi066.stdout:1/953: sync 2024-02-28T16:31:36.490 INFO:tasks.workunit.client.1.smithi066.stdout:2/928: sync 2024-02-28T16:31:36.491 INFO:tasks.workunit.client.1.smithi066.stdout:5/843: creat d3/d4/dc/dd/d35/d86/f10e x:0 0 0 2024-02-28T16:31:36.497 INFO:tasks.workunit.client.1.smithi066.stdout:9/725: rmdir da/d22 39 2024-02-28T16:31:36.499 INFO:tasks.workunit.client.1.smithi066.stdout:7/922: dread - dc/d56/d97/fc0 zero size 2024-02-28T16:31:36.499 INFO:tasks.workunit.client.1.smithi066.stdout:8/983: dwrite d0/d6/dc/d16/d4c/f12e [0,4194304] 0 2024-02-28T16:31:36.502 INFO:tasks.workunit.client.1.smithi066.stdout:1/954: truncate d2/d27/d2b/d43/d46/fcd 397064 0 2024-02-28T16:31:36.505 INFO:tasks.workunit.client.1.smithi066.stdout:7/923: mknod dc/de/c128 0 2024-02-28T16:31:36.506 INFO:tasks.workunit.client.1.smithi066.stdout:9/726: stat da/d22/d46/d56/d8f/d28/d42/fc6 0 2024-02-28T16:31:36.509 INFO:tasks.workunit.client.1.smithi066.stdout:1/955: symlink d2/d27/df9/l143 0 2024-02-28T16:31:36.516 INFO:tasks.workunit.client.1.smithi066.stdout:1/956: chown d2/d27/d2b/l6a 12507798 1 2024-02-28T16:31:36.520 INFO:tasks.workunit.client.1.smithi066.stdout:5/844: rename d3/d4/dc/d2f/d55/d74/d104/dfd/fff to d3/d4/dc/d2f/dc0/f10f 0 2024-02-28T16:31:36.520 INFO:tasks.workunit.client.1.smithi066.stdout:2/929: truncate d0/d12/d2e/f9d 883554 0 2024-02-28T16:31:36.521 INFO:tasks.workunit.client.1.smithi066.stdout:1/957: read d2/d18/d7a/f8b [1251476,3344] 0 2024-02-28T16:31:36.526 INFO:tasks.workunit.client.1.smithi066.stdout:9/727: getdents da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/dc9 0 2024-02-28T16:31:36.526 INFO:tasks.workunit.client.1.smithi066.stdout:5/845: rmdir d3/d1b/d20/ddd 39 2024-02-28T16:31:36.529 INFO:tasks.workunit.client.1.smithi066.stdout:1/958: creat d2/d27/dc7/d48/f144 x:0 0 0 2024-02-28T16:31:36.530 INFO:tasks.workunit.client.1.smithi066.stdout:2/930: mknod d0/d12/d1a/d39/d7c/c125 0 2024-02-28T16:31:36.533 INFO:tasks.workunit.client.1.smithi066.stdout:7/924: dwrite dc/d29/d92/da6/dd8/d1e/d45/f8f [0,4194304] 0 2024-02-28T16:31:36.540 INFO:tasks.workunit.client.1.smithi066.stdout:9/728: truncate da/df/f29 551147 0 2024-02-28T16:31:36.543 INFO:tasks.workunit.client.1.smithi066.stdout:5/846: dread - d3/d4/dc/dd/d41/d5d/f100 zero size 2024-02-28T16:31:36.544 INFO:tasks.workunit.client.1.smithi066.stdout:8/984: dwrite d0/db/fc2 [0,4194304] 0 2024-02-28T16:31:36.545 INFO:tasks.workunit.client.1.smithi066.stdout:9/729: chown da/d3a/d15/f1d 31442786 1 2024-02-28T16:31:36.550 INFO:tasks.workunit.client.1.smithi066.stdout:7/925: truncate dc/d29/d92/da6/dd8/d1e/f20 1851921 0 2024-02-28T16:31:36.555 INFO:tasks.workunit.client.1.smithi066.stdout:7/926: mknod dc/d56/dce/d102/db7/dec/c129 0 2024-02-28T16:31:36.564 INFO:tasks.workunit.client.1.smithi066.stdout:1/959: dwrite d2/d18/d7a/dbc/f11b [0,4194304] 0 2024-02-28T16:31:36.570 INFO:tasks.workunit.client.1.smithi066.stdout:1/960: dread - d2/d27/dc7/da6/dd4/f120 zero size 2024-02-28T16:31:36.575 INFO:tasks.workunit.client.1.smithi066.stdout:1/961: fdatasync d2/d27/dc7/d51/f83 0 2024-02-28T16:31:36.575 INFO:tasks.workunit.client.1.smithi066.stdout:9/730: dwrite da/d3a/d58/dab/fd2 [0,4194304] 0 2024-02-28T16:31:36.577 INFO:tasks.workunit.client.1.smithi066.stdout:2/931: write d0/dd7/fde [341535,125052] 0 2024-02-28T16:31:36.579 INFO:tasks.workunit.client.1.smithi066.stdout:9/731: fsync da/d22/d46/d56/d8f/fc8 0 2024-02-28T16:31:36.581 INFO:tasks.workunit.client.1.smithi066.stdout:2/932: symlink d0/d12/d2e/d3b/l126 0 2024-02-28T16:31:36.582 INFO:tasks.workunit.client.1.smithi066.stdout:1/962: dread d2/d27/d2b/d34/fcc [0,4194304] 0 2024-02-28T16:31:36.584 INFO:tasks.workunit.client.1.smithi066.stdout:2/933: fdatasync d0/d12/d3e/d9e/fed 0 2024-02-28T16:31:36.587 INFO:tasks.workunit.client.1.smithi066.stdout:9/732: truncate da/d4b/d5e/fc0 60764 0 2024-02-28T16:31:36.589 INFO:tasks.workunit.client.1.smithi066.stdout:8/985: truncate d0/db/ff1 1333422 0 2024-02-28T16:31:36.591 INFO:tasks.workunit.client.1.smithi066.stdout:1/963: symlink d2/d27/d2b/d34/da9/de9/l145 0 2024-02-28T16:31:36.596 INFO:tasks.workunit.client.1.smithi066.stdout:9/733: creat da/df/fed x:0 0 0 2024-02-28T16:31:36.597 INFO:tasks.workunit.client.1.smithi066.stdout:8/986: fsync d0/db/d12/d1d/d25/d72/f77 0 2024-02-28T16:31:36.601 INFO:tasks.workunit.client.1.smithi066.stdout:1/964: mkdir d2/d27/dc7/d54/d11e/d146 0 2024-02-28T16:31:36.607 INFO:tasks.workunit.client.1.smithi066.stdout:1/965: chown d2/d27/d2b/d43/d46/f61 521 1 2024-02-28T16:31:36.609 INFO:tasks.workunit.client.1.smithi066.stdout:5/847: dread d3/d4/dc/dd/d41/d7a/dac/f73 [0,4194304] 0 2024-02-28T16:31:36.613 INFO:tasks.workunit.client.1.smithi066.stdout:8/987: dread d0/db/d12/d1d/d25/d65/dbb/d11e/d9d/da2/f109 [0,4194304] 0 2024-02-28T16:31:36.615 INFO:tasks.workunit.client.1.smithi066.stdout:9/734: write da/d22/d46/d56/d8f/d28/d42/d68/de2/f93 [1597864,46401] 0 2024-02-28T16:31:36.617 INFO:tasks.workunit.client.1.smithi066.stdout:1/966: mkdir d2/d18/d7a/db1/d147 0 2024-02-28T16:31:36.619 INFO:tasks.workunit.client.1.smithi066.stdout:2/934: write d0/d12/d1a/d39/d7c/d87/dc4/fc8 [214484,49085] 0 2024-02-28T16:31:36.622 INFO:tasks.workunit.client.1.smithi066.stdout:9/735: write da/d22/f51 [30236,120401] 0 2024-02-28T16:31:36.625 INFO:tasks.workunit.client.1.smithi066.stdout:8/988: readlink d0/d6/dc/d16/d4c/d6c/d6f/da7/ldf 0 2024-02-28T16:31:36.627 INFO:tasks.workunit.client.1.smithi066.stdout:5/848: creat d3/d1b/d20/f110 x:0 0 0 2024-02-28T16:31:36.631 INFO:tasks.workunit.client.1.smithi066.stdout:2/935: mkdir d0/d12/d2e/d3b/d127 0 2024-02-28T16:31:36.632 INFO:tasks.workunit.client.1.smithi066.stdout:7/927: dread f4 [0,4194304] 0 2024-02-28T16:31:36.633 INFO:tasks.workunit.client.1.smithi066.stdout:1/967: creat d2/d18/d7a/d8d/d105/f148 x:0 0 0 2024-02-28T16:31:36.636 INFO:tasks.workunit.client.1.smithi066.stdout:7/928: dread - dc/d29/d92/da6/dd8/d1e/f10c zero size 2024-02-28T16:31:36.638 INFO:tasks.workunit.client.1.smithi066.stdout:2/936: creat d0/d12/d3e/f128 x:0 0 0 2024-02-28T16:31:36.639 INFO:tasks.workunit.client.1.smithi066.stdout:7/929: readlink dc/d29/d92/da6/dd8/d37/la5 0 2024-02-28T16:31:36.641 INFO:tasks.workunit.client.1.smithi066.stdout:1/968: creat d2/d18/d7a/dfd/f149 x:0 0 0 2024-02-28T16:31:36.643 INFO:tasks.workunit.client.1.smithi066.stdout:9/736: dread da/d4b/d53/fa7 [0,4194304] 0 2024-02-28T16:31:36.643 INFO:tasks.workunit.client.1.smithi066.stdout:7/930: creat dc/d56/ddf/f12a x:0 0 0 2024-02-28T16:31:36.645 INFO:tasks.workunit.client.1.smithi066.stdout:7/931: dread - dc/d29/fc7 zero size 2024-02-28T16:31:36.645 INFO:tasks.workunit.client.1.smithi066.stdout:1/969: rmdir d2/d27/dc7/da6/d12e 39 2024-02-28T16:31:36.651 INFO:tasks.workunit.client.1.smithi066.stdout:8/989: getdents d0/db 0 2024-02-28T16:31:36.654 INFO:tasks.workunit.client.1.smithi066.stdout:9/737: unlink da/d3a/d58/cce 0 2024-02-28T16:31:36.656 INFO:tasks.workunit.client.1.smithi066.stdout:1/970: chown d2/d27/d2b/d43/l115 63 1 2024-02-28T16:31:36.660 INFO:tasks.workunit.client.1.smithi066.stdout:8/990: mknod d0/d6/dc/d16/d1a/d63/d58/c13c 0 2024-02-28T16:31:36.661 INFO:tasks.workunit.client.1.smithi066.stdout:9/738: mknod da/d22/d46/d56/d8f/d28/cee 0 2024-02-28T16:31:36.661 INFO:tasks.workunit.client.1.smithi066.stdout:7/932: dread dc/d29/d92/da6/dd8/d1e/f3e [0,4194304] 0 2024-02-28T16:31:36.662 INFO:tasks.workunit.client.1.smithi066.stdout:1/971: readlink d2/l36 0 2024-02-28T16:31:36.663 INFO:tasks.workunit.client.1.smithi066.stdout:8/991: write d0/db/d15/fd6 [746652,65982] 0 2024-02-28T16:31:36.667 INFO:tasks.workunit.client.1.smithi066.stdout:1/972: rmdir d2/d27/dc7/da6 39 2024-02-28T16:31:36.668 INFO:tasks.workunit.client.1.smithi066.stdout:5/849: dwrite d3/d1b/d20/dc1/fcf [0,4194304] 0 2024-02-28T16:31:36.669 INFO:tasks.workunit.client.1.smithi066.stdout:7/933: write dc/d29/d92/da6/dd8/d1e/d45/fae [3940125,100348] 0 2024-02-28T16:31:36.675 INFO:tasks.workunit.client.1.smithi066.stdout:8/992: rename d0/d6/f1f to d0/db/d12/d1d/d25/d4f/f13d 0 2024-02-28T16:31:36.693 INFO:tasks.workunit.client.1.smithi066.stdout:9/739: dread da/fb [0,4194304] 0 2024-02-28T16:31:36.698 INFO:tasks.workunit.client.1.smithi066.stdout:1/973: dwrite d2/d18/d6b/dba/fdd [0,4194304] 0 2024-02-28T16:31:36.712 INFO:tasks.workunit.client.1.smithi066.stdout:5/850: dread d3/d4/dc/dd/d41/d7a/dac/f49 [0,4194304] 0 2024-02-28T16:31:36.731 INFO:tasks.workunit.client.1.smithi066.stdout:7/934: mkdir dc/d29/d92/da6/dd8/d12b 0 2024-02-28T16:31:36.740 INFO:tasks.workunit.client.1.smithi066.stdout:9/740: write da/d4b/d53/dba/f60 [1229240,57798] 0 2024-02-28T16:31:36.740 INFO:tasks.workunit.client.1.smithi066.stdout:1/974: creat d2/d27/d2b/d34/da9/f14a x:0 0 0 2024-02-28T16:31:36.740 INFO:tasks.workunit.client.1.smithi066.stdout:7/935: creat dc/d29/d92/da6/dd8/d91/dba/dbb/f12c x:0 0 0 2024-02-28T16:31:36.741 INFO:tasks.workunit.client.1.smithi066.stdout:5/851: symlink d3/d4/l111 0 2024-02-28T16:31:36.748 INFO:tasks.workunit.client.1.smithi066.stdout:9/741: write da/d4b/d53/fc7 [48173,72777] 0 2024-02-28T16:31:36.751 INFO:tasks.workunit.client.1.smithi066.stdout:1/975: rename d2/d27/dc7/d51/d128 to d2/d27/d132/d14b 0 2024-02-28T16:31:36.752 INFO:tasks.workunit.client.1.smithi066.stdout:5/852: mkdir d3/d4/dc/dd/d35/d86/ded/d112 0 2024-02-28T16:31:36.757 INFO:tasks.workunit.client.1.smithi066.stdout:1/976: mknod d2/d27/d2b/d43/dc8/c14c 0 2024-02-28T16:31:36.763 INFO:tasks.workunit.client.1.smithi066.stdout:1/977: fsync d2/d27/d2b/f39 0 2024-02-28T16:31:36.768 INFO:tasks.workunit.client.1.smithi066.stdout:1/978: dread - d2/d18/d7a/d8d/d105/f148 zero size 2024-02-28T16:31:36.772 INFO:tasks.workunit.client.1.smithi066.stdout:1/979: symlink d2/d27/dc7/d51/l14d 0 2024-02-28T16:31:36.775 INFO:tasks.workunit.client.1.smithi066.stdout:9/742: dread da/f2a [4194304,4194304] 0 2024-02-28T16:31:36.778 INFO:tasks.workunit.client.1.smithi066.stdout:1/980: rename d2/d27/dc7/d51/cdc to d2/d27/d2b/d34/da9/de9/c14e 0 2024-02-28T16:31:36.782 INFO:tasks.workunit.client.1.smithi066.stdout:1/981: chown d2/d18/d7a/d8d/f103 93118 1 2024-02-28T16:31:36.783 INFO:tasks.workunit.client.1.smithi066.stdout:7/936: truncate dc/f5d 2469432 0 2024-02-28T16:31:36.794 INFO:tasks.workunit.client.1.smithi066.stdout:1/982: fsync d2/d27/dc7/d54/f8a 0 2024-02-28T16:31:36.794 INFO:tasks.workunit.client.1.smithi066.stdout:7/937: symlink dc/d29/d92/df2/dfc/l12d 0 2024-02-28T16:31:36.800 INFO:tasks.workunit.client.1.smithi066.stdout:7/938: chown dc/d29/d92/da6/dd8/d1e/d45/f8f 58686 1 2024-02-28T16:31:36.801 INFO:tasks.workunit.client.1.smithi066.stdout:5/853: dwrite d3/d4/dc/dd/d41/d5d/f69 [0,4194304] 0 2024-02-28T16:31:36.801 INFO:tasks.workunit.client.1.smithi066.stdout:8/993: dwrite d0/d6/dc/d16/d4c/d6c/d6f/d92/fb7 [0,4194304] 0 2024-02-28T16:31:36.802 INFO:tasks.workunit.client.1.smithi066.stdout:1/983: creat d2/d27/d132/d14b/f14f x:0 0 0 2024-02-28T16:31:36.803 INFO:tasks.workunit.client.1.smithi066.stdout:8/994: stat d0/db/f5f 0 2024-02-28T16:31:36.815 INFO:tasks.workunit.client.1.smithi066.stdout:5/854: fdatasync d3/d4/dc/d2f/d55/d74/d104/f9a 0 2024-02-28T16:31:36.823 INFO:tasks.workunit.client.1.smithi066.stdout:5/855: fsync d3/d4/dc/dd/d2e/d18/f2c 0 2024-02-28T16:31:36.825 INFO:tasks.workunit.client.1.smithi066.stdout:5/856: fdatasync d3/d4/dc/dd/d2e/d18/f4b 0 2024-02-28T16:31:36.825 INFO:tasks.workunit.client.1.smithi066.stdout:9/743: dread da/d3a/fac [0,4194304] 0 2024-02-28T16:31:36.828 INFO:tasks.workunit.client.1.smithi066.stdout:9/744: chown da/d3a/d15/dcf 3 1 2024-02-28T16:31:36.830 INFO:tasks.workunit.client.1.smithi066.stdout:5/857: unlink d3/d1b/d7d/cc4 0 2024-02-28T16:31:36.832 INFO:tasks.workunit.client.1.smithi066.stdout:9/745: write da/d22/d46/d56/d8f/fc8 [698489,130867] 0 2024-02-28T16:31:36.834 INFO:tasks.workunit.client.1.smithi066.stdout:7/939: truncate dc/d29/d92/da6/fa7 421695 0 2024-02-28T16:31:36.838 INFO:tasks.workunit.client.1.smithi066.stdout:9/746: rmdir da/d3a 39 2024-02-28T16:31:36.838 INFO:tasks.workunit.client.1.smithi066.stdout:7/940: read dc/d56/dce/dff/fe8 [29148,73479] 0 2024-02-28T16:31:36.844 INFO:tasks.workunit.client.1.smithi066.stdout:7/941: truncate dc/d56/dc3/ff5 419942 0 2024-02-28T16:31:36.846 INFO:tasks.workunit.client.1.smithi066.stdout:1/984: dwrite d2/d18/d7a/d8d/f103 [0,4194304] 0 2024-02-28T16:31:36.854 INFO:tasks.workunit.client.1.smithi066.stdout:7/942: creat dc/d29/d92/da6/dd8/d91/dba/dbb/f12e x:0 0 0 2024-02-28T16:31:36.857 INFO:tasks.workunit.client.1.smithi066.stdout:7/943: dread - dc/de/d87/f66 zero size 2024-02-28T16:31:36.861 INFO:tasks.workunit.client.1.smithi066.stdout:9/747: dwrite da/fc [4194304,4194304] 0 2024-02-28T16:31:36.868 INFO:tasks.workunit.client.1.smithi066.stdout:9/748: read da/d22/d46/d56/d8f/d28/d42/f49 [511467,119509] 0 2024-02-28T16:31:36.874 INFO:tasks.workunit.client.1.smithi066.stdout:7/944: dread dc/d29/d92/da6/fa7 [0,4194304] 0 2024-02-28T16:31:36.875 INFO:tasks.workunit.client.1.smithi066.stdout:9/749: unlink da/d22/d46/d56/d8f/d28/cee 0 2024-02-28T16:31:36.878 INFO:tasks.workunit.client.1.smithi066.stdout:1/985: dwrite d2/d27/d2b/d34/d9e/fa8 [0,4194304] 0 2024-02-28T16:31:36.878 INFO:tasks.workunit.client.1.smithi066.stdout:7/945: unlink dc/f7f 0 2024-02-28T16:31:36.890 INFO:tasks.workunit.client.1.smithi066.stdout:9/750: chown da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3/ce8 14027 1 2024-02-28T16:31:36.893 INFO:tasks.workunit.client.1.smithi066.stdout:9/751: chown da/d6b/c71 7151001 1 2024-02-28T16:31:36.897 INFO:tasks.workunit.client.1.smithi066.stdout:9/752: write da/d22/d46/d56/d8f/d28/d42/feb [368225,30862] 0 2024-02-28T16:31:36.899 INFO:tasks.workunit.client.1.smithi066.stdout:7/946: dwrite dc/d56/ddf/ff3 [0,4194304] 0 2024-02-28T16:31:36.907 INFO:tasks.workunit.client.1.smithi066.stdout:1/986: dwrite d2/d27/d2b/d34/da9/f14a [0,4194304] 0 2024-02-28T16:31:36.908 INFO:tasks.workunit.client.1.smithi066.stdout:9/753: symlink da/d4b/d53/dba/lef 0 2024-02-28T16:31:36.920 INFO:tasks.workunit.client.1.smithi066.stdout:9/754: creat da/d22/d46/d56/d8f/d28/d42/d68/ff0 x:0 0 0 2024-02-28T16:31:36.925 INFO:tasks.workunit.client.1.smithi066.stdout:9/755: creat da/d22/d46/d56/d8f/d2d/ff1 x:0 0 0 2024-02-28T16:31:36.928 INFO:tasks.workunit.client.1.smithi066.stdout:9/756: chown da/d22/l2e 0 1 2024-02-28T16:31:36.929 INFO:tasks.workunit.client.1.smithi066.stdout:1/987: dread d2/d27/dc7/d48/f81 [0,4194304] 0 2024-02-28T16:31:36.936 INFO:tasks.workunit.client.1.smithi066.stdout:9/757: creat da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/ff2 x:0 0 0 2024-02-28T16:31:36.937 INFO:tasks.workunit.client.1.smithi066.stdout:9/758: write da/fdb [469314,42689] 0 2024-02-28T16:31:36.940 INFO:tasks.workunit.client.1.smithi066.stdout:7/947: dread dc/d56/f8a [0,4194304] 0 2024-02-28T16:31:36.943 INFO:tasks.workunit.client.1.smithi066.stdout:7/948: dread - dc/d56/fad zero size 2024-02-28T16:31:36.944 INFO:tasks.workunit.client.1.smithi066.stdout:7/949: chown f1 8448 1 2024-02-28T16:31:36.945 INFO:tasks.workunit.client.1.smithi066.stdout:9/759: dread da/d22/d46/d56/d8f/d28/fa5 [0,4194304] 0 2024-02-28T16:31:36.945 INFO:tasks.workunit.client.1.smithi066.stdout:7/950: fsync dc/f4b 0 2024-02-28T16:31:36.948 INFO:tasks.workunit.client.1.smithi066.stdout:1/988: dwrite d2/d27/d2b/d34/da9/f117 [0,4194304] 0 2024-02-28T16:31:36.948 INFO:tasks.workunit.client.1.smithi066.stdout:7/951: unlink dc/c21 0 2024-02-28T16:31:36.954 INFO:tasks.workunit.client.1.smithi066.stdout:7/952: creat dc/d56/dce/d126/d100/f12f x:0 0 0 2024-02-28T16:31:36.958 INFO:tasks.workunit.client.1.smithi066.stdout:7/953: mkdir dc/d56/dc3/df4/d130 0 2024-02-28T16:31:36.960 INFO:tasks.workunit.client.1.smithi066.stdout:7/954: readlink dc/l4e 0 2024-02-28T16:31:36.964 INFO:tasks.workunit.client.1.smithi066.stdout:7/955: fdatasync dc/de/f13 0 2024-02-28T16:31:36.968 INFO:tasks.workunit.client.1.smithi066.stdout:7/956: chown dc/d29/d92/da6/dd8/d91 3861886 1 2024-02-28T16:31:36.968 INFO:tasks.workunit.client.1.smithi066.stdout:1/989: write d2/d27/d2b/d43/f9b [1831341,51598] 0 2024-02-28T16:31:36.970 INFO:tasks.workunit.client.1.smithi066.stdout:7/957: rename dc to dc/d56/dce/d102/d131 22 2024-02-28T16:31:36.974 INFO:tasks.workunit.client.1.smithi066.stdout:1/990: creat d2/d27/d2b/d137/f150 x:0 0 0 2024-02-28T16:31:36.976 INFO:tasks.workunit.client.1.smithi066.stdout:7/958: mknod dc/d56/d117/c132 0 2024-02-28T16:31:36.983 INFO:tasks.workunit.client.1.smithi066.stdout:7/959: rename dc/d29/d92/da6/dd8/d1e/f24 to dc/d29/d92/f133 0 2024-02-28T16:31:36.986 INFO:tasks.workunit.client.1.smithi066.stdout:7/960: creat dc/d29/d92/df2/f134 x:0 0 0 2024-02-28T16:31:36.988 INFO:tasks.workunit.client.1.smithi066.stdout:7/961: chown dc/de/c128 3 1 2024-02-28T16:31:36.994 INFO:tasks.workunit.client.1.smithi066.stdout:1/991: dwrite d2/d27/d2b/d43/f9d [0,4194304] 0 2024-02-28T16:31:36.998 INFO:tasks.workunit.client.1.smithi066.stdout:7/962: creat dc/d29/d92/da6/dd8/d37/d47/d88/def/f135 x:0 0 0 2024-02-28T16:31:36.999 INFO:tasks.workunit.client.1.smithi066.stdout:1/992: write d2/d27/d2b/d43/faf [1782739,37238] 0 2024-02-28T16:31:37.003 INFO:tasks.workunit.client.1.smithi066.stdout:7/963: mknod dc/d29/d92/df2/dfc/c136 0 2024-02-28T16:31:37.008 INFO:tasks.workunit.client.1.smithi066.stdout:1/993: truncate d2/d18/d7a/ff0 599905 0 2024-02-28T16:31:37.011 INFO:tasks.workunit.client.1.smithi066.stdout:1/994: creat d2/d18/d7a/dfd/f151 x:0 0 0 2024-02-28T16:31:37.013 INFO:tasks.workunit.client.1.smithi066.stdout:2/937: sync 2024-02-28T16:31:37.014 INFO:tasks.workunit.client.1.smithi066.stdout:1/995: mkdir d2/d27/dc7/d48/dce/d152 0 2024-02-28T16:31:37.018 INFO:tasks.workunit.client.1.smithi066.stdout:2/938: creat d0/d12/d1a/d39/d68/f129 x:0 0 0 2024-02-28T16:31:37.020 INFO:tasks.workunit.client.1.smithi066.stdout:1/996: rename d2/d27/dc7/de4 to d2/d27/d2b/d43/dc8/d153 0 2024-02-28T16:31:37.022 INFO:tasks.workunit.client.1.smithi066.stdout:7/964: dwrite dc/de/fc5 [0,4194304] 0 2024-02-28T16:31:37.111 INFO:tasks.workunit.client.1.smithi066.stdout:7/965: stat dc/d56/dce 0 2024-02-28T16:31:37.111 INFO:tasks.workunit.client.1.smithi066.stdout:2/939: write d0/fe2 [468311,85359] 0 2024-02-28T16:31:37.111 INFO:tasks.workunit.client.1.smithi066.stdout:7/966: creat dc/d29/f137 x:0 0 0 2024-02-28T16:31:37.111 INFO:tasks.workunit.client.1.smithi066.stdout:1/997: creat d2/d27/dc7/d54/f154 x:0 0 0 2024-02-28T16:31:37.111 INFO:tasks.workunit.client.1.smithi066.stdout:2/940: rmdir d0/d12/d1a/d23/ddc/d103 39 2024-02-28T16:31:37.111 INFO:tasks.workunit.client.1.smithi066.stdout:7/967: creat dc/d56/f138 x:0 0 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:1/998: truncate d2/d27/d2b/d137/f150 919713 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/941: creat d0/d12/d3e/f12a x:0 0 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:1/999: rename d2/d18/d6b/dba/fdd to d2/d18/d7a/dfd/d10a/f155 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/942: dwrite d0/d12/d2e/d7a/fba [0,4194304] 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/943: chown d0/d12/d1a/d23/l57 1029008 1 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/944: mkdir d0/d12/d1a/d39/d73/d12b 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/945: symlink d0/d12/d1a/l12c 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/946: write d0/d12/d1a/d39/d7c/fdf [1025761,35601] 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/947: symlink d0/d12/d1a/d23/d29/d21/l12d 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/948: dwrite d0/d12/d2e/d7a/f99 [0,4194304] 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/949: dwrite d0/d12/d2e/d7a/f55 [4194304,4194304] 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/950: stat d0/d12/d1a/d23/d29/d21 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/951: write d0/d12/d2e/d3b/f64 [239293,94936] 0 2024-02-28T16:31:37.112 INFO:tasks.workunit.client.1.smithi066.stdout:2/952: readlink d0/d12/d1a/d23/d29/d21/l12d 0 2024-02-28T16:31:37.117 INFO:tasks.workunit.client.1.smithi066.stdout:2/953: truncate d0/d12/d1a/d23/fe1 3453763 0 2024-02-28T16:31:37.195 INFO:tasks.workunit.client.1.smithi066.stdout:8/995: sync 2024-02-28T16:31:37.204 INFO:tasks.workunit.client.1.smithi066.stdout:8/996: truncate d0/db/d12/d38/fe7 963329 0 2024-02-28T16:31:37.211 INFO:tasks.workunit.client.1.smithi066.stdout:8/997: symlink d0/d6/dc/d16/d1a/d63/d58/d5a/l13e 0 2024-02-28T16:31:37.222 INFO:tasks.workunit.client.1.smithi066.stdout:8/998: rename d0/d6/dc/l36 to d0/d6/dc/d16/d1a/d63/d58/d5a/d12d/l13f 0 2024-02-28T16:31:37.228 INFO:tasks.workunit.client.1.smithi066.stdout:8/999: truncate d0/d6/dea/d12f/f13a 848044 0 2024-02-28T16:31:37.325 INFO:tasks.workunit.client.1.smithi066.stdout:9/760: sync 2024-02-28T16:31:37.325 INFO:tasks.workunit.client.1.smithi066.stdout:5/858: sync 2024-02-28T16:31:37.336 INFO:tasks.workunit.client.1.smithi066.stdout:5/859: unlink d3/d1b/db3/dec/ff2 0 2024-02-28T16:31:37.339 INFO:tasks.workunit.client.1.smithi066.stdout:5/860: readlink d3/d1b/d20/dc1/ldf 0 2024-02-28T16:31:37.341 INFO:tasks.workunit.client.1.smithi066.stdout:9/761: mknod da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3/cf3 0 2024-02-28T16:31:37.351 INFO:tasks.workunit.client.1.smithi066.stdout:5/861: write d3/d4/dc/dd/d41/d7f/d80/f109 [486727,116809] 0 2024-02-28T16:31:37.366 INFO:tasks.workunit.client.1.smithi066.stdout:7/968: sync 2024-02-28T16:31:37.366 INFO:tasks.workunit.client.1.smithi066.stdout:2/954: sync 2024-02-28T16:31:37.382 INFO:tasks.workunit.client.1.smithi066.stdout:7/969: mknod dc/d29/d92/c139 0 2024-02-28T16:31:37.390 INFO:tasks.workunit.client.1.smithi066.stdout:9/762: dwrite da/df/f3d [0,4194304] 0 2024-02-28T16:31:37.398 INFO:tasks.workunit.client.1.smithi066.stdout:5/862: dread d3/d4/dc/dd/d2e/d18/fd5 [0,4194304] 0 2024-02-28T16:31:37.403 INFO:tasks.workunit.client.1.smithi066.stdout:5/863: symlink d3/l113 0 2024-02-28T16:31:37.407 INFO:tasks.workunit.client.1.smithi066.stdout:7/970: rename dc/d29/d92/da6/dd8/d1e/fa3 to dc/d6e/f13a 0 2024-02-28T16:31:37.424 INFO:tasks.workunit.client.1.smithi066.stdout:5/864: dwrite d3/d1b/d7d/fd1 [0,4194304] 0 2024-02-28T16:31:37.426 INFO:tasks.workunit.client.1.smithi066.stdout:5/865: write d3/d1b/fcb [3206517,106674] 0 2024-02-28T16:31:37.434 INFO:tasks.workunit.client.1.smithi066.stdout:5/866: creat d3/d4/f114 x:0 0 0 2024-02-28T16:31:37.434 INFO:tasks.workunit.client.1.smithi066.stdout:7/971: dwrite dc/d29/d92/da6/dd8/d1e/d45/fae [0,4194304] 0 2024-02-28T16:31:37.437 INFO:tasks.workunit.client.1.smithi066.stdout:7/972: truncate dc/d29/fc7 212245 0 2024-02-28T16:31:37.443 INFO:tasks.workunit.client.1.smithi066.stdout:5/867: fsync d3/d4/dc/d2f/d55/d74/f107 0 2024-02-28T16:31:37.449 INFO:tasks.workunit.client.1.smithi066.stdout:5/868: mknod d3/d4/dc/dd/d41/d7a/dac/d60/c115 0 2024-02-28T16:31:37.455 INFO:tasks.workunit.client.1.smithi066.stdout:5/869: getdents d3/d4/dc/dd/de6/d108 0 2024-02-28T16:31:37.469 INFO:tasks.workunit.client.1.smithi066.stdout:7/973: dwrite dc/f5d [0,4194304] 0 2024-02-28T16:31:37.493 INFO:tasks.workunit.client.1.smithi066.stdout:7/974: dwrite dc/d56/dce/d102/fc1 [0,4194304] 0 2024-02-28T16:31:37.494 INFO:tasks.workunit.client.1.smithi066.stdout:5/870: dread d3/d4/dc/dd/d35/d86/f88 [0,4194304] 0 2024-02-28T16:31:37.498 INFO:tasks.workunit.client.1.smithi066.stdout:5/871: chown d3/d4/dc/dd/d35/d86/f10e 1504 1 2024-02-28T16:31:37.498 INFO:tasks.workunit.client.1.smithi066.stdout:7/975: mkdir dc/d29/d92/da6/dd8/d12b/d13b 0 2024-02-28T16:31:37.501 INFO:tasks.workunit.client.1.smithi066.stdout:5/872: write d3/d4/dc/dd/de6/ff7 [750230,78638] 0 2024-02-28T16:31:37.502 INFO:tasks.workunit.client.1.smithi066.stdout:7/976: getdents dc/d29/d92/da6/dd8/d12b/d13b 0 2024-02-28T16:31:37.507 INFO:tasks.workunit.client.1.smithi066.stdout:7/977: truncate dc/d56/f8a 5150350 0 2024-02-28T16:31:37.509 INFO:tasks.workunit.client.1.smithi066.stdout:5/873: chown d3/d4/dc/dd/d2e/f38 357858 1 2024-02-28T16:31:37.511 INFO:tasks.workunit.client.1.smithi066.stdout:7/978: read dc/d29/d92/da6/dd8/d1e/f3e [1593017,48198] 0 2024-02-28T16:31:37.513 INFO:tasks.workunit.client.1.smithi066.stdout:5/874: mkdir d3/d1b/d20/d116 0 2024-02-28T16:31:37.520 INFO:tasks.workunit.client.1.smithi066.stdout:5/875: symlink d3/d1b/d20/l117 0 2024-02-28T16:31:37.528 INFO:tasks.workunit.client.1.smithi066.stdout:5/876: unlink d3/d4/dc/dd/d35/d86/fdc 0 2024-02-28T16:31:37.539 INFO:tasks.workunit.client.1.smithi066.stdout:5/877: getdents d3/d4/dc/dd/d2e/d18 0 2024-02-28T16:31:37.544 INFO:tasks.workunit.client.1.smithi066.stdout:7/979: dwrite dc/d29/d92/da6/dd8/f7c [0,4194304] 0 2024-02-28T16:31:37.546 INFO:tasks.workunit.client.1.smithi066.stdout:5/878: truncate d3/d4/dc/d56/fd3 438934 0 2024-02-28T16:31:37.549 INFO:tasks.workunit.client.1.smithi066.stdout:2/955: sync 2024-02-28T16:31:37.549 INFO:tasks.workunit.client.1.smithi066.stdout:7/980: write dc/d29/d92/da6/dd8/d1e/f98 [1071191,113956] 0 2024-02-28T16:31:37.550 INFO:tasks.workunit.client.1.smithi066.stdout:5/879: read d3/d4/f83 [527243,15394] 0 2024-02-28T16:31:37.554 INFO:tasks.workunit.client.1.smithi066.stdout:2/956: mkdir d0/d12/d2e/d7a/d12e 0 2024-02-28T16:31:37.558 INFO:tasks.workunit.client.1.smithi066.stdout:5/880: creat d3/d4/dc/dd/f118 x:0 0 0 2024-02-28T16:31:37.566 INFO:tasks.workunit.client.1.smithi066.stdout:7/981: truncate dc/d29/d92/da6/dd8/d1e/f2d 986404 0 2024-02-28T16:31:37.567 INFO:tasks.workunit.client.1.smithi066.stdout:5/881: truncate d3/d4/dc/dd/d35/d86/f10e 307625 0 2024-02-28T16:31:37.568 INFO:tasks.workunit.client.1.smithi066.stdout:7/982: dread - dc/de/fb8 zero size 2024-02-28T16:31:37.570 INFO:tasks.workunit.client.1.smithi066.stdout:7/983: write dc/f5d [486388,61757] 0 2024-02-28T16:31:37.572 INFO:tasks.workunit.client.1.smithi066.stdout:5/882: getdents d3/d1b/d39 0 2024-02-28T16:31:37.575 INFO:tasks.workunit.client.1.smithi066.stdout:2/957: write d0/d12/d1a/d39/d7c/d87/fa1 [4883650,16431] 0 2024-02-28T16:31:37.578 INFO:tasks.workunit.client.1.smithi066.stdout:2/958: truncate d0/fe2 939218 0 2024-02-28T16:31:37.585 INFO:tasks.workunit.client.1.smithi066.stdout:5/883: link d3/d4/dc/f1d d3/d1b/d20/ddd/f119 0 2024-02-28T16:31:37.587 INFO:tasks.workunit.client.1.smithi066.stdout:2/959: truncate d0/d12/d2e/d7a/f41 5187254 0 2024-02-28T16:31:37.590 INFO:tasks.workunit.client.1.smithi066.stdout:5/884: write d3/d4/dc/dd/d35/d92/fe5 [4327006,44546] 0 2024-02-28T16:31:37.590 INFO:tasks.workunit.client.1.smithi066.stdout:2/960: read d0/d12/d2e/fe8 [1503282,11910] 0 2024-02-28T16:31:37.592 INFO:tasks.workunit.client.1.smithi066.stdout:7/984: dwrite dc/d29/fc7 [0,4194304] 0 2024-02-28T16:31:37.597 INFO:tasks.workunit.client.1.smithi066.stdout:5/885: mknod d3/d4/dc/dd/d2e/c11a 0 2024-02-28T16:31:37.598 INFO:tasks.workunit.client.1.smithi066.stdout:2/961: mknod d0/d12/d1a/d23/d29/d21/d2f/d67/c12f 0 2024-02-28T16:31:37.600 INFO:tasks.workunit.client.1.smithi066.stdout:7/985: creat dc/d56/dce/d102/f13c x:0 0 0 2024-02-28T16:31:37.604 INFO:tasks.workunit.client.1.smithi066.stdout:2/962: creat d0/d12/d3e/f130 x:0 0 0 2024-02-28T16:31:37.611 INFO:tasks.workunit.client.1.smithi066.stdout:9/763: sync 2024-02-28T16:31:37.613 INFO:tasks.workunit.client.1.smithi066.stdout:9/764: rmdir da/d22 39 2024-02-28T16:31:37.614 INFO:tasks.workunit.client.1.smithi066.stdout:5/886: dwrite d3/d4/dc/dd/d41/d7f/d80/f109 [0,4194304] 0 2024-02-28T16:31:37.617 INFO:tasks.workunit.client.1.smithi066.stdout:7/986: dread dc/d29/d92/da6/dd8/f7c [0,4194304] 0 2024-02-28T16:31:37.620 INFO:tasks.workunit.client.1.smithi066.stdout:9/765: rmdir da/d3a/d58 39 2024-02-28T16:31:37.628 INFO:tasks.workunit.client.1.smithi066.stdout:7/987: mknod dc/d6e/dc8/c13d 0 2024-02-28T16:31:37.630 INFO:tasks.workunit.client.1.smithi066.stdout:5/887: rename d3/d4/dc/d2f/d55/d74/d104/l10d to d3/d4/dc/dd/d41/d7a/dac/d60/l11b 0 2024-02-28T16:31:37.632 INFO:tasks.workunit.client.1.smithi066.stdout:5/888: dread - d3/d4/dc/dd/d41/d7f/fa9 zero size 2024-02-28T16:31:37.637 INFO:tasks.workunit.client.1.smithi066.stdout:5/889: write d3/d4/dc/dd/d35/f62 [4498637,108507] 0 2024-02-28T16:31:37.639 INFO:tasks.workunit.client.1.smithi066.stdout:5/890: fdatasync d3/d4/f4e 0 2024-02-28T16:31:37.642 INFO:tasks.workunit.client.1.smithi066.stdout:9/766: dread da/d3a/d15/f1d [0,4194304] 0 2024-02-28T16:31:37.731 INFO:tasks.workunit.client.1.smithi066.stdout:7/988: link dc/d56/ddf/le6 dc/de/d87/l13e 0 2024-02-28T16:31:37.734 INFO:tasks.workunit.client.1.smithi066.stdout:7/989: chown dc/d29/d92/da6/dd8/d37/f8d 275929 1 2024-02-28T16:31:37.736 INFO:tasks.workunit.client.1.smithi066.stdout:5/891: truncate d3/d4/dc/d2f/d55/d74/fbb 3505659 0 2024-02-28T16:31:37.754 INFO:tasks.workunit.client.1.smithi066.stdout:7/990: dwrite dc/d29/d92/da6/dd8/d91/fe2 [0,4194304] 0 2024-02-28T16:31:37.775 INFO:tasks.workunit.client.1.smithi066.stdout:7/991: dwrite dc/d29/d92/da6/dd8/d1e/d45/f8f [0,4194304] 0 2024-02-28T16:31:37.809 INFO:tasks.workunit.client.1.smithi066.stdout:2/963: sync 2024-02-28T16:31:37.815 INFO:tasks.workunit.client.1.smithi066.stdout:5/892: creat d3/d4/dc/dd/d41/d7a/dac/d60/f11c x:0 0 0 2024-02-28T16:31:37.815 INFO:tasks.workunit.client.1.smithi066.stdout:2/964: dread d0/d12/d1a/d39/d68/fc9 [4194304,4194304] 0 2024-02-28T16:31:37.822 INFO:tasks.workunit.client.1.smithi066.stdout:5/893: creat d3/d4/dc/d2f/d55/d74/d104/dfd/f11d x:0 0 0 2024-02-28T16:31:37.831 INFO:tasks.workunit.client.1.smithi066.stdout:5/894: rename d3/d4/dc/dd/d41/d7a/dac/d60/f11c to d3/d4/dc/d56/d93/f11e 0 2024-02-28T16:31:37.832 INFO:tasks.workunit.client.1.smithi066.stdout:5/895: chown d3/d4/dc/c87 1807 1 2024-02-28T16:31:37.836 INFO:tasks.workunit.client.1.smithi066.stdout:5/896: symlink d3/d1b/d20/ddd/l11f 0 2024-02-28T16:31:37.840 INFO:tasks.workunit.client.1.smithi066.stdout:5/897: creat d3/d4/dc/dd/d41/da2/f120 x:0 0 0 2024-02-28T16:31:37.898 INFO:tasks.workunit.client.1.smithi066.stdout:5/898: unlink d3/d4/dc/dd/d41/d7a/dac/cb5 0 2024-02-28T16:31:37.900 INFO:tasks.workunit.client.1.smithi066.stdout:9/767: rmdir da/d4b 39 2024-02-28T16:31:37.900 INFO:tasks.workunit.client.1.smithi066.stdout:5/899: stat d3/d4/dc/dd/d2e/d18/fd5 0 2024-02-28T16:31:37.900 INFO:tasks.workunit.client.1.smithi066.stdout:7/992: truncate dc/d29/d92/da6/dd8/d1e/f20 1394677 0 2024-02-28T16:31:37.903 INFO:tasks.workunit.client.1.smithi066.stdout:7/993: chown dc/l4e 15 1 2024-02-28T16:31:37.903 INFO:tasks.workunit.client.1.smithi066.stdout:9/768: read da/d6b/dbc/fe6 [961416,60187] 0 2024-02-28T16:31:37.906 INFO:tasks.workunit.client.1.smithi066.stdout:5/900: rmdir d3/d4/dc/dd/de6/d108 0 2024-02-28T16:31:37.908 INFO:tasks.workunit.client.1.smithi066.stdout:9/769: creat da/d22/d46/d56/d8f/d2d/ff4 x:0 0 0 2024-02-28T16:31:37.908 INFO:tasks.workunit.client.1.smithi066.stdout:7/994: truncate dc/d29/d92/da6/dd8/d1e/d45/f84 265664 0 2024-02-28T16:31:37.911 INFO:tasks.workunit.client.1.smithi066.stdout:7/995: truncate dc/d29/f137 555246 0 2024-02-28T16:31:37.912 INFO:tasks.workunit.client.1.smithi066.stdout:9/770: mkdir da/d3a/d15/df5 0 2024-02-28T16:31:37.917 INFO:tasks.workunit.client.1.smithi066.stdout:7/996: symlink dc/d29/d92/da6/dd8/d37/d47/l13f 0 2024-02-28T16:31:37.923 INFO:tasks.workunit.client.1.smithi066.stdout:9/771: link da/d3a/f13 da/d22/d46/d56/d8f/d28/d42/d68/ff6 0 2024-02-28T16:31:37.924 INFO:tasks.workunit.client.1.smithi066.stdout:9/772: readlink da/df/lb0 0 2024-02-28T16:31:37.924 INFO:tasks.workunit.client.1.smithi066.stdout:5/901: dwrite d3/d4/dc/d56/f75 [0,4194304] 0 2024-02-28T16:31:37.929 INFO:tasks.workunit.client.1.smithi066.stdout:9/773: chown da/d4b/d53/dba/f5c 24 1 2024-02-28T16:31:37.929 INFO:tasks.workunit.client.1.smithi066.stdout:7/997: dread dc/de/d87/f111 [4194304,4194304] 0 2024-02-28T16:31:37.933 INFO:tasks.workunit.client.1.smithi066.stdout:7/998: symlink dc/d29/d92/df2/dfc/l140 0 2024-02-28T16:31:37.936 INFO:tasks.workunit.client.1.smithi066.stdout:9/774: rename da/d3a/d15/l23 to da/d22/d46/d56/d8f/d28/d42/d68/dda/lf7 0 2024-02-28T16:31:37.939 INFO:tasks.workunit.client.1.smithi066.stdout:5/902: link d3/d4/dc/dd/d2e/c81 d3/d4/dc/dd/d35/d86/ded/d112/c121 0 2024-02-28T16:31:37.941 INFO:tasks.workunit.client.1.smithi066.stdout:9/775: unlink da/d22/d46/d56/d8f/d28/f4d 0 2024-02-28T16:31:37.944 INFO:tasks.workunit.client.1.smithi066.stdout:5/903: mkdir d3/d4/dc/dd/d2e/d10b/d122 0 2024-02-28T16:31:37.944 INFO:tasks.workunit.client.1.smithi066.stdout:9/776: write da/d22/d46/d56/d8f/d2d/ff4 [480535,103277] 0 2024-02-28T16:31:37.950 INFO:tasks.workunit.client.1.smithi066.stdout:5/904: getdents d3/d4/dc/dd/d35/d86/ded 0 2024-02-28T16:31:37.952 INFO:tasks.workunit.client.1.smithi066.stdout:9/777: creat da/d22/d46/d56/d8f/dbe/ff8 x:0 0 0 2024-02-28T16:31:37.956 INFO:tasks.workunit.client.1.smithi066.stdout:5/905: truncate d3/d1b/d7d/fd1 6120810 0 2024-02-28T16:31:37.957 INFO:tasks.workunit.client.1.smithi066.stdout:9/778: truncate da/d22/d46/d56/d8f/d2d/ff4 622353 0 2024-02-28T16:31:37.960 INFO:tasks.workunit.client.1.smithi066.stdout:7/999: dread dc/d29/d92/da6/dd8/d1e/d45/f79 [0,4194304] 0 2024-02-28T16:31:37.962 INFO:tasks.workunit.client.1.smithi066.stdout:9/779: mknod da/d22/d46/d56/d8f/cf9 0 2024-02-28T16:31:37.968 INFO:tasks.workunit.client.1.smithi066.stdout:5/906: truncate d3/d1b/fcb 2441074 0 2024-02-28T16:31:37.970 INFO:tasks.workunit.client.1.smithi066.stdout:5/907: creat d3/d4/dc/d56/d93/f123 x:0 0 0 2024-02-28T16:31:37.979 INFO:tasks.workunit.client.1.smithi066.stdout:5/908: write d3/d1b/d20/dc1/fe3 [116844,71545] 0 2024-02-28T16:31:37.986 INFO:tasks.workunit.client.1.smithi066.stdout:9/780: dread da/d22/d46/d56/d8f/d2d/f4c [0,4194304] 0 2024-02-28T16:31:37.986 INFO:tasks.workunit.client.1.smithi066.stdout:5/909: rename d3/d4/f9 to d3/d4/dc/f124 0 2024-02-28T16:31:37.989 INFO:tasks.workunit.client.1.smithi066.stdout:5/910: mknod d3/d4/dc/d2f/d55/d74/d104/c125 0 2024-02-28T16:31:37.990 INFO:tasks.workunit.client.1.smithi066.stdout:9/781: mkdir da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/dfa 0 2024-02-28T16:31:37.992 INFO:tasks.workunit.client.1.smithi066.stdout:5/911: write d3/d4/dc/dd/d2e/d18/fd5 [5019041,48135] 0 2024-02-28T16:31:37.993 INFO:tasks.workunit.client.1.smithi066.stdout:9/782: symlink da/d4b/d5e/lfb 0 2024-02-28T16:31:37.996 INFO:tasks.workunit.client.1.smithi066.stdout:9/783: read da/d3a/d15/f6a [2047705,31667] 0 2024-02-28T16:31:37.998 INFO:tasks.workunit.client.1.smithi066.stdout:9/784: creat da/d22/d46/d56/d8f/d94/ffc x:0 0 0 2024-02-28T16:31:38.002 INFO:tasks.workunit.client.1.smithi066.stdout:9/785: rename da/df to da/d3a/d58/dfd 0 2024-02-28T16:31:38.005 INFO:tasks.workunit.client.1.smithi066.stdout:9/786: mkdir da/d22/d46/d56/d8f/d28/d42/d68/dda/dfe 0 2024-02-28T16:31:38.008 INFO:tasks.workunit.client.1.smithi066.stdout:9/787: creat da/d4b/d53/fff x:0 0 0 2024-02-28T16:31:38.011 INFO:tasks.workunit.client.1.smithi066.stdout:9/788: rename da/d22/f51 to da/d6b/f100 0 2024-02-28T16:31:38.014 INFO:tasks.workunit.client.1.smithi066.stdout:9/789: write da/d22/d46/d56/d8f/fb9 [1787913,97594] 0 2024-02-28T16:31:38.029 INFO:tasks.workunit.client.1.smithi066.stdout:9/790: dwrite da/d3a/d58/dfd/fed [0,4194304] 0 2024-02-28T16:31:38.046 INFO:tasks.workunit.client.1.smithi066.stdout:9/791: dwrite da/d22/d46/d56/d8f/d28/d42/feb [0,4194304] 0 2024-02-28T16:31:38.065 INFO:tasks.workunit.client.1.smithi066.stdout:9/792: dwrite da/d22/d46/d56/d8f/d28/fa5 [0,4194304] 0 2024-02-28T16:31:38.069 INFO:tasks.workunit.client.1.smithi066.stdout:9/793: chown da/d22/d46/d56/d8f/d28/d42/feb 47401 1 2024-02-28T16:31:38.075 INFO:tasks.workunit.client.1.smithi066.stdout:9/794: link da/d4b/d53/dba/lef da/d22/d46/d56/d8f/d28/d42/l101 0 2024-02-28T16:31:38.079 INFO:tasks.workunit.client.1.smithi066.stdout:9/795: truncate da/d3a/d58/dab/fd2 4824707 0 2024-02-28T16:31:38.082 INFO:tasks.workunit.client.1.smithi066.stdout:9/796: mknod da/c102 0 2024-02-28T16:31:38.086 INFO:tasks.workunit.client.1.smithi066.stdout:9/797: symlink da/d3a/d15/dcf/l103 0 2024-02-28T16:31:38.090 INFO:tasks.workunit.client.1.smithi066.stdout:9/798: creat da/d22/d46/d56/d8f/d28/d42/d68/de2/f104 x:0 0 0 2024-02-28T16:31:38.092 INFO:tasks.workunit.client.1.smithi066.stdout:9/799: chown da/d3a/d58/dfd/f2f 57509 1 2024-02-28T16:31:38.099 INFO:tasks.workunit.client.1.smithi066.stdout:9/800: link da/d3a/f13 da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/dfa/f105 0 2024-02-28T16:31:38.102 INFO:tasks.workunit.client.1.smithi066.stdout:9/801: stat da/d22/d46/d56/d8f/f82 0 2024-02-28T16:31:38.113 INFO:tasks.workunit.client.1.smithi066.stdout:9/802: write da/d3a/d58/dfd/f29 [103740,28354] 0 2024-02-28T16:31:38.117 INFO:tasks.workunit.client.1.smithi066.stdout:9/803: creat da/d3a/d15/d48/d76/f106 x:0 0 0 2024-02-28T16:31:38.123 INFO:tasks.workunit.client.1.smithi066.stdout:9/804: truncate da/d22/d46/d56/d8f/d28/d42/fc6 279403 0 2024-02-28T16:31:38.142 INFO:tasks.workunit.client.1.smithi066.stdout:9/805: dwrite da/d3a/d58/dfd/f29 [0,4194304] 0 2024-02-28T16:31:38.155 INFO:tasks.workunit.client.1.smithi066.stdout:9/806: rename da/d22/d46/d56/d8f/f78 to da/d22/d46/d56/d8f/dbe/f107 0 2024-02-28T16:31:38.168 INFO:tasks.workunit.client.1.smithi066.stdout:9/807: write da/d22/d46/d56/d8f/d28/d42/d68/f7a [3363330,49068] 0 2024-02-28T16:31:38.170 INFO:tasks.workunit.client.1.smithi066.stdout:9/808: write da/d3a/d58/dfd/f29 [1945871,25127] 0 2024-02-28T16:31:38.180 INFO:tasks.workunit.client.1.smithi066.stdout:9/809: truncate da/d22/d46/d56/d8f/d2d/ff4 617564 0 2024-02-28T16:31:38.184 INFO:tasks.workunit.client.1.smithi066.stdout:9/810: write da/d3a/d58/dfd/f3d [1333093,4995] 0 2024-02-28T16:31:38.195 INFO:tasks.workunit.client.1.smithi066.stdout:9/811: write da/d3a/f13 [2742450,124531] 0 2024-02-28T16:31:38.196 INFO:tasks.workunit.client.1.smithi066.stdout:9/812: truncate da/d22/d46/d56/d8f/d2d/ff1 201871 0 2024-02-28T16:31:38.222 INFO:tasks.workunit.client.1.smithi066.stdout:9/813: dwrite da/d22/d46/d56/d8f/d2d/ff4 [0,4194304] 0 2024-02-28T16:31:38.223 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:37 smithi066 ceph-mon[36236]: pgmap v62: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 93 MiB/s rd, 212 MiB/s wr, 496 op/s 2024-02-28T16:31:38.228 INFO:tasks.workunit.client.1.smithi066.stdout:9/814: mkdir da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/d108 0 2024-02-28T16:31:38.233 INFO:tasks.workunit.client.1.smithi066.stdout:9/815: mknod da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/c109 0 2024-02-28T16:31:38.236 INFO:tasks.workunit.client.1.smithi066.stdout:9/816: mknod da/d22/d46/d56/d8f/d94/c10a 0 2024-02-28T16:31:38.237 INFO:tasks.workunit.client.1.smithi066.stdout:2/965: sync 2024-02-28T16:31:38.239 INFO:tasks.workunit.client.1.smithi066.stdout:2/966: creat d0/dd7/f131 x:0 0 0 2024-02-28T16:31:38.241 INFO:tasks.workunit.client.1.smithi066.stdout:9/817: getdents da/d4b/d53/dba 0 2024-02-28T16:31:38.241 INFO:tasks.workunit.client.1.smithi066.stdout:2/967: chown d0/d12/d1a/d39/d7c/d87/d104 68673001 1 2024-02-28T16:31:38.244 INFO:tasks.workunit.client.1.smithi066.stdout:9/818: mkdir da/d4b/d53/dcb/d10b 0 2024-02-28T16:31:38.245 INFO:tasks.workunit.client.1.smithi066.stdout:2/968: creat d0/d12/d1a/d39/d7c/d87/d104/f132 x:0 0 0 2024-02-28T16:31:38.247 INFO:tasks.workunit.client.1.smithi066.stdout:9/819: creat da/d3a/d58/f10c x:0 0 0 2024-02-28T16:31:38.251 INFO:tasks.workunit.client.1.smithi066.stdout:2/969: rename d0/d12/d1a/d23/d5d/fbb to d0/d12/d1a/d23/d5d/f133 0 2024-02-28T16:31:38.252 INFO:tasks.workunit.client.1.smithi066.stdout:2/970: fsync d0/d12/d2e/d3b/d66/f121 0 2024-02-28T16:31:38.254 INFO:tasks.workunit.client.1.smithi066.stdout:2/971: fsync d0/d12/d1a/df8/f107 0 2024-02-28T16:31:38.257 INFO:tasks.workunit.client.1.smithi066.stdout:2/972: mknod d0/d12/d2e/d3b/d127/c134 0 2024-02-28T16:31:38.259 INFO:tasks.workunit.client.1.smithi066.stdout:2/973: dread - d0/d12/d3e/f130 zero size 2024-02-28T16:31:38.263 INFO:tasks.workunit.client.1.smithi066.stdout:2/974: getdents d0/d12/d3e/d9e/dce/d111 0 2024-02-28T16:31:38.265 INFO:tasks.workunit.client.1.smithi066.stdout:9/820: dwrite da/d22/d46/d56/d8f/d28/d42/d68/f8c [4194304,4194304] 0 2024-02-28T16:31:38.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:37 smithi001 ceph-mon[28991]: pgmap v62: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 93 MiB/s rd, 212 MiB/s wr, 496 op/s 2024-02-28T16:31:38.271 INFO:tasks.workunit.client.1.smithi066.stdout:9/821: truncate da/d4b/d53/dba/fbb 1859533 0 2024-02-28T16:31:38.282 INFO:tasks.workunit.client.1.smithi066.stdout:2/975: dread d0/d12/d2e/d7a/fba [0,4194304] 0 2024-02-28T16:31:38.285 INFO:tasks.workunit.client.1.smithi066.stdout:2/976: rmdir d0/d12/d1a/d39/d73 39 2024-02-28T16:31:38.291 INFO:tasks.workunit.client.1.smithi066.stdout:2/977: truncate d0/d12/d1a/d23/f28 6212430 0 2024-02-28T16:31:38.293 INFO:tasks.workunit.client.1.smithi066.stdout:5/912: sync 2024-02-28T16:31:38.298 INFO:tasks.workunit.client.1.smithi066.stdout:2/978: truncate d0/d12/d1a/d39/d68/fc9 2837249 0 2024-02-28T16:31:38.299 INFO:tasks.workunit.client.1.smithi066.stdout:9/822: dwrite da/d22/d46/d56/d8f/d28/f59 [0,4194304] 0 2024-02-28T16:31:38.302 INFO:tasks.workunit.client.1.smithi066.stdout:5/913: getdents d3/d4/dc/dd/d41/d5d 0 2024-02-28T16:31:38.306 INFO:tasks.workunit.client.1.smithi066.stdout:9/823: mknod da/d22/d46/d56/d8f/d2d/c10d 0 2024-02-28T16:31:38.322 INFO:tasks.workunit.client.1.smithi066.stdout:5/914: link d3/d4/dc/ld0 d3/d4/dc/dd/d2e/d10b/d122/l126 0 2024-02-28T16:31:38.329 INFO:tasks.workunit.client.1.smithi066.stdout:9/824: dwrite da/fdb [0,4194304] 0 2024-02-28T16:31:38.332 INFO:tasks.workunit.client.1.smithi066.stdout:9/825: chown da/d3a/d15/d48/f66 14336753 1 2024-02-28T16:31:38.337 INFO:tasks.workunit.client.1.smithi066.stdout:9/826: creat da/d3a/d58/dfd/f10e x:0 0 0 2024-02-28T16:31:38.338 INFO:tasks.workunit.client.1.smithi066.stdout:5/915: dread d3/d4/f71 [0,4194304] 0 2024-02-28T16:31:38.342 INFO:tasks.workunit.client.1.smithi066.stdout:9/827: write da/d22/d46/d56/d8f/fc8 [23326,112617] 0 2024-02-28T16:31:38.344 INFO:tasks.workunit.client.1.smithi066.stdout:2/979: dwrite d0/d12/d1a/d23/f28 [4194304,4194304] 0 2024-02-28T16:31:38.347 INFO:tasks.workunit.client.1.smithi066.stdout:5/916: creat d3/d4/dc/dd/d41/da2/f127 x:0 0 0 2024-02-28T16:31:38.348 INFO:tasks.workunit.client.1.smithi066.stdout:5/917: chown d3/d4/dc/dd/d2e/d18/f4a 3624 1 2024-02-28T16:31:38.352 INFO:tasks.workunit.client.1.smithi066.stdout:2/980: creat d0/d12/d1a/d23/d29/d21/d2f/f135 x:0 0 0 2024-02-28T16:31:38.354 INFO:tasks.workunit.client.1.smithi066.stdout:5/918: mkdir d3/d1b/d20/de9/d128 0 2024-02-28T16:31:38.367 INFO:tasks.workunit.client.1.smithi066.stdout:2/981: chown d0/d12/d2e/f84 4942799 1 2024-02-28T16:31:38.367 INFO:tasks.workunit.client.1.smithi066.stdout:5/919: write d3/d4/dc/dd/d41/d7f/ff5 [1219963,20334] 0 2024-02-28T16:31:38.372 INFO:tasks.workunit.client.1.smithi066.stdout:2/982: write d0/d12/d1a/d39/d7c/d87/fa1 [1269762,72244] 0 2024-02-28T16:31:38.373 INFO:tasks.workunit.client.1.smithi066.stdout:5/920: fdatasync d3/d4/dc/dd/d2e/d18/f2b 0 2024-02-28T16:31:38.373 INFO:tasks.workunit.client.1.smithi066.stdout:9/828: dread da/d4b/f54 [0,4194304] 0 2024-02-28T16:31:38.377 INFO:tasks.workunit.client.1.smithi066.stdout:5/921: write d3/d4/dc/dd/d35/d86/f10e [436774,112826] 0 2024-02-28T16:31:38.379 INFO:tasks.workunit.client.1.smithi066.stdout:2/983: mknod d0/d12/d2e/d3b/d66/c136 0 2024-02-28T16:31:38.383 INFO:tasks.workunit.client.1.smithi066.stdout:2/984: creat d0/d12/d1a/d39/d7c/d87/d104/f137 x:0 0 0 2024-02-28T16:31:38.384 INFO:tasks.workunit.client.1.smithi066.stdout:5/922: getdents d3/d4/dc/d56/d93 0 2024-02-28T16:31:38.388 INFO:tasks.workunit.client.1.smithi066.stdout:5/923: mknod d3/d1b/db3/df1/c129 0 2024-02-28T16:31:38.494 INFO:tasks.workunit.client.1.smithi066.stdout:2/985: rename d0/d12/d1a/d23/ddc/d103/c11f to d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/c138 0 2024-02-28T16:31:38.494 INFO:tasks.workunit.client.1.smithi066.stdout:5/924: mkdir d3/d4/dc/d2f/d12a 0 2024-02-28T16:31:38.498 INFO:tasks.workunit.client.1.smithi066.stdout:2/986: mkdir d0/dd7/d139 0 2024-02-28T16:31:38.502 INFO:tasks.workunit.client.1.smithi066.stdout:5/925: write d3/d4/dc/d56/d93/f11e [5106,77339] 0 2024-02-28T16:31:38.504 INFO:tasks.workunit.client.1.smithi066.stdout:5/926: chown d3/d1b/f5f 1816 1 2024-02-28T16:31:38.508 INFO:tasks.workunit.client.1.smithi066.stdout:5/927: getdents d3/d4/dc/dd/d41 0 2024-02-28T16:31:38.509 INFO:tasks.workunit.client.1.smithi066.stdout:5/928: chown d3/d4/dc/d2f/d55/d74/d104 1 1 2024-02-28T16:31:38.511 INFO:tasks.workunit.client.1.smithi066.stdout:5/929: chown d3/d4/dc/dd/d35/dc5 26383 1 2024-02-28T16:31:38.515 INFO:tasks.workunit.client.1.smithi066.stdout:5/930: symlink d3/d4/dc/dd/d2e/d10b/l12b 0 2024-02-28T16:31:38.519 INFO:tasks.workunit.client.1.smithi066.stdout:5/931: creat d3/d4/dc/dd/d41/d7f/f12c x:0 0 0 2024-02-28T16:31:38.528 INFO:tasks.workunit.client.1.smithi066.stdout:2/987: dwrite d0/d12/d2e/d7a/f45 [0,4194304] 0 2024-02-28T16:31:38.531 INFO:tasks.workunit.client.1.smithi066.stdout:5/932: write d3/d4/dc/dd/d2e/d18/f4a [2857533,108829] 0 2024-02-28T16:31:38.538 INFO:tasks.workunit.client.1.smithi066.stdout:5/933: dread - d3/d4/dc/dd/d41/d7a/dac/d60/fc7 zero size 2024-02-28T16:31:38.538 INFO:tasks.workunit.client.1.smithi066.stdout:2/988: rename d0/d12/d1a/d39/d73/lf9 to d0/d12/d1a/df8/d120/l13a 0 2024-02-28T16:31:38.543 INFO:tasks.workunit.client.1.smithi066.stdout:5/934: truncate d3/d4/dc/d56/d93/f11e 243043 0 2024-02-28T16:31:38.544 INFO:tasks.workunit.client.1.smithi066.stdout:5/935: chown d3/d4/l1c 100 1 2024-02-28T16:31:38.546 INFO:tasks.workunit.client.1.smithi066.stdout:5/936: stat d3/d4/dc/dd/d41/d7a/dac/l4c 0 2024-02-28T16:31:38.551 INFO:tasks.workunit.client.1.smithi066.stdout:5/937: rmdir d3/d4/dc/d2f/d55/d74/d104/dfd 39 2024-02-28T16:31:38.554 INFO:tasks.workunit.client.1.smithi066.stdout:2/989: dread d0/d12/d1a/d39/d7c/d87/dc4/ff1 [0,4194304] 0 2024-02-28T16:31:38.555 INFO:tasks.workunit.client.1.smithi066.stdout:5/938: mkdir d3/d4/dc/dd/d2e/d18/d12d 0 2024-02-28T16:31:38.559 INFO:tasks.workunit.client.1.smithi066.stdout:2/990: chown d0/d12/d1a/d23/d29/d21/d2f/d67/c81 50 1 2024-02-28T16:31:38.563 INFO:tasks.workunit.client.1.smithi066.stdout:2/991: write d0/d12/d3e/f4e [442340,54220] 0 2024-02-28T16:31:38.565 INFO:tasks.workunit.client.1.smithi066.stdout:5/939: getdents d3/d4/dc/d2f/d55/d74/d104 0 2024-02-28T16:31:38.571 INFO:tasks.workunit.client.1.smithi066.stdout:5/940: unlink d3/l5a 0 2024-02-28T16:31:38.578 INFO:tasks.workunit.client.1.smithi066.stdout:5/941: rename d3/d1b/d20/de9 to d3/d4/dc/d2f/d55/d74/d104/dfd/d12e 0 2024-02-28T16:31:38.580 INFO:tasks.workunit.client.1.smithi066.stdout:2/992: dread d0/d12/d2e/d7a/fba [0,4194304] 0 2024-02-28T16:31:38.585 INFO:tasks.workunit.client.1.smithi066.stdout:2/993: unlink d0/d12/d2e/d3b/ld6 0 2024-02-28T16:31:38.587 INFO:tasks.workunit.client.1.smithi066.stdout:5/942: rename d3/d4/dc/d2f/l59 to d3/d4/dc/d2f/d55/d74/d104/dfd/d12e/l12f 0 2024-02-28T16:31:38.600 INFO:tasks.workunit.client.1.smithi066.stdout:2/994: dwrite d0/d12/d1a/f54 [0,4194304] 0 2024-02-28T16:31:38.600 INFO:tasks.workunit.client.1.smithi066.stdout:5/943: dwrite d3/d1b/f5f [0,4194304] 0 2024-02-28T16:31:38.608 INFO:tasks.workunit.client.1.smithi066.stdout:5/944: dread d3/d4/dc/f1d [0,4194304] 0 2024-02-28T16:31:38.613 INFO:tasks.workunit.client.1.smithi066.stdout:2/995: mknod d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/df5/c13b 0 2024-02-28T16:31:38.615 INFO:tasks.workunit.client.1.smithi066.stdout:9/829: sync 2024-02-28T16:31:38.616 INFO:tasks.workunit.client.1.smithi066.stdout:5/945: mkdir d3/d4/dc/d56/d93/d130 0 2024-02-28T16:31:38.620 INFO:tasks.workunit.client.1.smithi066.stdout:5/946: truncate d3/d1b/d20/f110 882056 0 2024-02-28T16:31:38.624 INFO:tasks.workunit.client.1.smithi066.stdout:9/830: getdents da/d6b/dbc 0 2024-02-28T16:31:38.626 INFO:tasks.workunit.client.1.smithi066.stdout:2/996: rename d0/d12/d1a/d23/d29/d21/d2f/d67/d8d/lad to d0/d12/d2e/d3b/l13c 0 2024-02-28T16:31:38.628 INFO:tasks.workunit.client.1.smithi066.stdout:5/947: truncate d3/d4/dc/d2f/d55/d74/d104/fe2 3699071 0 2024-02-28T16:31:38.629 INFO:tasks.workunit.client.1.smithi066.stdout:9/831: mknod da/d3a/d58/dfd/c10f 0 2024-02-28T16:31:38.630 INFO:tasks.workunit.client.1.smithi066.stdout:2/997: mknod d0/d12/d3e/d9e/dce/d111/c13d 0 2024-02-28T16:31:38.631 INFO:tasks.workunit.client.1.smithi066.stdout:9/832: read da/d4b/d53/fc7 [50569,29474] 0 2024-02-28T16:31:38.634 INFO:tasks.workunit.client.1.smithi066.stdout:2/998: write d0/d12/d1a/d39/d7c/d87/d104/f114 [3098086,92237] 0 2024-02-28T16:31:38.635 INFO:tasks.workunit.client.1.smithi066.stdout:5/948: creat d3/d4/dc/d2f/d55/f131 x:0 0 0 2024-02-28T16:31:38.637 INFO:tasks.workunit.client.1.smithi066.stdout:2/999: stat d0/d12/d2e/f9d 0 2024-02-28T16:31:38.637 INFO:tasks.workunit.client.1.smithi066.stdout:5/949: readlink d3/d4/dc/dd/l45 0 2024-02-28T16:31:38.640 INFO:tasks.workunit.client.1.smithi066.stdout:5/950: symlink d3/d1b/db3/df1/l132 0 2024-02-28T16:31:38.643 INFO:tasks.workunit.client.1.smithi066.stdout:5/951: rename d3/d4/dc/l4f to d3/d1b/d39/l133 0 2024-02-28T16:31:38.647 INFO:tasks.workunit.client.1.smithi066.stdout:5/952: dread d3/d4/f71 [0,4194304] 0 2024-02-28T16:31:38.650 INFO:tasks.workunit.client.1.smithi066.stdout:5/953: rename d3/d1b/d20/f10c to d3/d4/dc/dd/f134 0 2024-02-28T16:31:38.653 INFO:tasks.workunit.client.1.smithi066.stdout:5/954: creat d3/d4/dc/d56/f135 x:0 0 0 2024-02-28T16:31:38.654 INFO:tasks.workunit.client.1.smithi066.stdout:5/955: stat d3/d1b/d20/dc1/fe3 0 2024-02-28T16:31:38.657 INFO:tasks.workunit.client.1.smithi066.stdout:5/956: creat d3/d4/dc/dd/d35/dc5/f136 x:0 0 0 2024-02-28T16:31:38.659 INFO:tasks.workunit.client.1.smithi066.stdout:5/957: dread - d3/d4/dc/dd/d41/da2/f127 zero size 2024-02-28T16:31:38.904 INFO:tasks.workunit.client.1.smithi066.stdout:9/833: sync 2024-02-28T16:31:38.904 INFO:tasks.workunit.client.1.smithi066.stdout:5/958: sync 2024-02-28T16:31:38.910 INFO:tasks.workunit.client.1.smithi066.stdout:5/959: symlink d3/d4/dc/dd/de6/l137 0 2024-02-28T16:31:38.917 INFO:tasks.workunit.client.1.smithi066.stdout:9/834: link da/d3a/d58/d97/c95 da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3/c110 0 2024-02-28T16:31:38.921 INFO:tasks.workunit.client.1.smithi066.stdout:5/960: rename d3/d4/dc/dd/d2e/f6f to d3/d4/f138 0 2024-02-28T16:31:38.956 INFO:tasks.workunit.client.1.smithi066.stdout:9/835: truncate da/d22/d46/d56/d8f/d28/d42/d68/fd7 102198 0 2024-02-28T16:31:38.972 INFO:tasks.workunit.client.1.smithi066.stdout:5/961: rmdir d3/d4/dc/dd/de6 39 2024-02-28T16:31:38.990 INFO:tasks.workunit.client.1.smithi066.stdout:9/836: dwrite da/d3a/d58/f96 [0,4194304] 0 2024-02-28T16:31:38.997 INFO:tasks.workunit.client.1.smithi066.stdout:9/837: truncate da/d3a/f41 3457787 0 2024-02-28T16:31:38.999 INFO:tasks.workunit.client.1.smithi066.stdout:9/838: write da/d3a/f5f [2794868,84746] 0 2024-02-28T16:31:39.004 INFO:tasks.workunit.client.1.smithi066.stdout:5/962: dread d3/d4/dc/dd/d2e/f38 [0,4194304] 0 2024-02-28T16:31:39.005 INFO:tasks.workunit.client.1.smithi066.stdout:9/839: unlink da/d4b/d5e/fc0 0 2024-02-28T16:31:39.010 INFO:tasks.workunit.client.1.smithi066.stdout:5/963: mkdir d3/d1b/d20/d116/d139 0 2024-02-28T16:31:39.014 INFO:tasks.workunit.client.1.smithi066.stdout:5/964: read d3/d1b/d20/ddd/f119 [238536,49750] 0 2024-02-28T16:31:39.021 INFO:tasks.workunit.client.1.smithi066.stdout:5/965: symlink d3/d4/dc/dd/d35/d86/l13a 0 2024-02-28T16:31:39.026 INFO:tasks.workunit.client.1.smithi066.stdout:9/840: dwrite da/d3a/f5f [4194304,4194304] 0 2024-02-28T16:31:39.032 INFO:tasks.workunit.client.1.smithi066.stdout:9/841: truncate da/d22/d46/d56/f91 983337 0 2024-02-28T16:31:39.033 INFO:tasks.workunit.client.1.smithi066.stdout:5/966: dread d3/d4/dc/dd/de6/ff7 [0,4194304] 0 2024-02-28T16:31:39.036 INFO:tasks.workunit.client.1.smithi066.stdout:9/842: write da/d22/d46/d56/d8f/fc8 [585968,36358] 0 2024-02-28T16:31:39.038 INFO:tasks.workunit.client.1.smithi066.stdout:5/967: mknod d3/d1b/d39/c13b 0 2024-02-28T16:31:39.042 INFO:tasks.workunit.client.1.smithi066.stdout:5/968: readlink d3/d4/dc/d2f/d55/d74/lba 0 2024-02-28T16:31:39.046 INFO:tasks.workunit.client.1.smithi066.stdout:9/843: link da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3/cf3 da/d3a/d15/c111 0 2024-02-28T16:31:39.050 INFO:tasks.workunit.client.1.smithi066.stdout:9/844: dread da/d22/d46/d56/d8f/d28/d42/d68/fd7 [0,4194304] 0 2024-02-28T16:31:39.053 INFO:tasks.workunit.client.1.smithi066.stdout:9/845: chown da/d3a/fac 0 1 2024-02-28T16:31:39.055 INFO:tasks.workunit.client.1.smithi066.stdout:9/846: chown da/d4b/d53/c9f 29516554 1 2024-02-28T16:31:39.059 INFO:tasks.workunit.client.1.smithi066.stdout:5/969: dwrite d3/d4/dc/dd/d35/d92/fab [0,4194304] 0 2024-02-28T16:31:39.061 INFO:tasks.workunit.client.1.smithi066.stdout:9/847: creat da/d22/d46/d56/f112 x:0 0 0 2024-02-28T16:31:39.066 INFO:tasks.workunit.client.1.smithi066.stdout:9/848: creat da/d3a/d15/d48/d76/f113 x:0 0 0 2024-02-28T16:31:39.067 INFO:tasks.workunit.client.1.smithi066.stdout:5/970: fdatasync d3/d4/dc/dd/d2e/d18/f82 0 2024-02-28T16:31:39.074 INFO:tasks.workunit.client.1.smithi066.stdout:5/971: mkdir d3/d4/dc/dd/d2e/d18/d13c 0 2024-02-28T16:31:39.077 INFO:tasks.workunit.client.1.smithi066.stdout:5/972: mkdir d3/d13d 0 2024-02-28T16:31:39.081 INFO:tasks.workunit.client.1.smithi066.stdout:5/973: truncate d3/d4/f5c 1604459 0 2024-02-28T16:31:39.082 INFO:tasks.workunit.client.1.smithi066.stdout:9/849: write da/fb2 [825036,124105] 0 2024-02-28T16:31:39.083 INFO:tasks.workunit.client.1.smithi066.stdout:5/974: rmdir d3/d1b/d20 39 2024-02-28T16:31:39.085 INFO:tasks.workunit.client.1.smithi066.stdout:5/975: fsync d3/d4/dc/dd/d35/dc5/f136 0 2024-02-28T16:31:39.086 INFO:tasks.workunit.client.1.smithi066.stdout:9/850: read da/d4b/d53/fa7 [2168921,88061] 0 2024-02-28T16:31:39.092 INFO:tasks.workunit.client.1.smithi066.stdout:9/851: getdents da/d3a/d15 0 2024-02-28T16:31:39.094 INFO:tasks.workunit.client.1.smithi066.stdout:5/976: read d3/d1b/d20/ddd/fb9 [48548,130137] 0 2024-02-28T16:31:39.097 INFO:tasks.workunit.client.1.smithi066.stdout:9/852: creat da/d3a/d15/d48/d76/f114 x:0 0 0 2024-02-28T16:31:39.100 INFO:tasks.workunit.client.1.smithi066.stdout:5/977: mkdir d3/d4/dc/dd/d41/d7f/d13e 0 2024-02-28T16:31:39.106 INFO:tasks.workunit.client.1.smithi066.stdout:5/978: creat d3/d4/dc/dd/d41/d5d/f13f x:0 0 0 2024-02-28T16:31:39.124 INFO:tasks.workunit.client.1.smithi066.stdout:5/979: dread d3/d4/fc3 [0,4194304] 0 2024-02-28T16:31:39.128 INFO:tasks.workunit.client.1.smithi066.stdout:5/980: mkdir d3/d4/dc/dd/d41/d7a/dac/d60/db1/d140 0 2024-02-28T16:31:39.141 INFO:tasks.workunit.client.1.smithi066.stdout:5/981: dread d3/d1b/fcb [0,4194304] 0 2024-02-28T16:31:39.149 INFO:tasks.workunit.client.1.smithi066.stdout:5/982: link d3/d4/dc/dd/d2e/ld7 d3/d1b/db3/df1/l141 0 2024-02-28T16:31:39.153 INFO:tasks.workunit.client.1.smithi066.stdout:5/983: mkdir d3/d4/dc/dd/d35/d142 0 2024-02-28T16:31:39.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:39 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:39.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:39 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:39.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:39 smithi066 ceph-mon[36236]: pgmap v63: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 70 MiB/s rd, 151 MiB/s wr, 381 op/s 2024-02-28T16:31:39.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:39 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:31:39.366 INFO:tasks.workunit.client.1.smithi066.stdout:9/853: sync 2024-02-28T16:31:39.366 INFO:tasks.workunit.client.1.smithi066.stdout:5/984: sync 2024-02-28T16:31:39.378 INFO:tasks.workunit.client.1.smithi066.stdout:9/854: rename da/d3a/d58/dab/fd2 to da/d22/d46/d56/f115 0 2024-02-28T16:31:39.386 INFO:tasks.workunit.client.1.smithi066.stdout:5/985: link d3/d4/dc/dd/d41/d5d/lca d3/d4/dc/dd/d35/dc5/l143 0 2024-02-28T16:31:39.391 INFO:tasks.workunit.client.1.smithi066.stdout:5/986: chown d3/d4/fc3 12098400 1 2024-02-28T16:31:39.405 INFO:tasks.workunit.client.1.smithi066.stdout:9/855: getdents da/d3a/d15 0 2024-02-28T16:31:39.411 INFO:tasks.workunit.client.1.smithi066.stdout:5/987: creat d3/d4/dc/dd/d35/d86/ded/f144 x:0 0 0 2024-02-28T16:31:39.411 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:39 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:39.411 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:39 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:39.411 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:39 smithi001 ceph-mon[28991]: pgmap v63: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 70 MiB/s rd, 151 MiB/s wr, 381 op/s 2024-02-28T16:31:39.412 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:39 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:31:39.419 INFO:tasks.workunit.client.1.smithi066.stdout:9/856: write da/d4b/d53/fa7 [2462869,89312] 0 2024-02-28T16:31:39.423 INFO:tasks.workunit.client.1.smithi066.stdout:5/988: truncate d3/d1b/d20/f63 341087 0 2024-02-28T16:31:39.434 INFO:tasks.workunit.client.1.smithi066.stdout:5/989: fdatasync d3/d4/dc/d2f/d55/d74/d104/f9a 0 2024-02-28T16:31:39.437 INFO:tasks.workunit.client.1.smithi066.stdout:9/857: getdents da/d22/d46/d56/d8f/d28/d42 0 2024-02-28T16:31:39.441 INFO:tasks.workunit.client.1.smithi066.stdout:9/858: readlink da/d3a/d58/dfd/ld3 0 2024-02-28T16:31:39.461 INFO:tasks.workunit.client.1.smithi066.stdout:9/859: link da/d3a/d15/f19 da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/dc9/f116 0 2024-02-28T16:31:39.463 INFO:tasks.workunit.client.1.smithi066.stdout:9/860: chown da/d3a/d15/c8a 214251 1 2024-02-28T16:31:39.469 INFO:tasks.workunit.client.1.smithi066.stdout:5/990: dwrite d3/d4/dc/dd/d35/f64 [0,4194304] 0 2024-02-28T16:31:39.470 INFO:tasks.workunit.client.1.smithi066.stdout:9/861: fsync da/d4b/d53/dba/f5c 0 2024-02-28T16:31:39.478 INFO:tasks.workunit.client.1.smithi066.stdout:5/991: chown d3/d4/dc/dd/d35/d86/ded/cfe 13 1 2024-02-28T16:31:39.490 INFO:tasks.workunit.client.1.smithi066.stdout:5/992: unlink d3/d4/dc/dd/d41/d7f/f12c 0 2024-02-28T16:31:39.496 INFO:tasks.workunit.client.1.smithi066.stdout:5/993: unlink d3/d4/dc/f124 0 2024-02-28T16:31:39.501 INFO:tasks.workunit.client.1.smithi066.stdout:5/994: read d3/d4/dc/dd/d35/d92/fab [1417106,65236] 0 2024-02-28T16:31:39.506 INFO:tasks.workunit.client.1.smithi066.stdout:5/995: chown d3/d4/dc/dd/d41/da2/f127 4 1 2024-02-28T16:31:39.513 INFO:tasks.workunit.client.1.smithi066.stdout:5/996: fdatasync d3/d1b/d20/f63 0 2024-02-28T16:31:39.519 INFO:tasks.workunit.client.1.smithi066.stdout:5/997: write d3/d4/dc/fda [3389518,66167] 0 2024-02-28T16:31:39.525 INFO:tasks.workunit.client.1.smithi066.stdout:5/998: write d3/d4/dc/dd/d35/dc5/f136 [914251,118960] 0 2024-02-28T16:31:39.535 INFO:tasks.workunit.client.1.smithi066.stdout:9/862: dread f3 [0,4194304] 0 2024-02-28T16:31:39.545 INFO:tasks.workunit.client.1.smithi066.stdout:9/863: creat da/d22/d46/d56/d8f/d28/d42/f117 x:0 0 0 2024-02-28T16:31:39.549 INFO:tasks.workunit.client.1.smithi066.stdout:9/864: write da/d22/d46/d56/d8f/d2d/ff1 [1081521,78464] 0 2024-02-28T16:31:39.555 INFO:tasks.workunit.client.1.smithi066.stdout:9/865: chown da/d22/d46/d56/d8f/d28/d42/d68/dda/l85 4 1 2024-02-28T16:31:39.580 INFO:tasks.workunit.client.1.smithi066.stdout:9/866: dwrite da/d6b/dbc/fe6 [4194304,4194304] 0 2024-02-28T16:31:39.586 INFO:tasks.workunit.client.1.smithi066.stdout:9/867: write da/d3a/d15/d48/d76/f114 [840589,113735] 0 2024-02-28T16:31:39.588 INFO:tasks.workunit.client.1.smithi066.stdout:9/868: readlink da/d22/d46/d56/d8f/d2d/lc5 0 2024-02-28T16:31:39.615 INFO:tasks.workunit.client.1.smithi066.stdout:9/869: dwrite da/d22/d46/d56/d8f/d94/fa0 [0,4194304] 0 2024-02-28T16:31:39.622 INFO:tasks.workunit.client.1.smithi066.stdout:9/870: read da/d22/d46/d56/f72 [748027,18509] 0 2024-02-28T16:31:39.659 INFO:tasks.workunit.client.1.smithi066.stdout:9/871: dread da/f2a [0,4194304] 0 2024-02-28T16:31:39.666 INFO:tasks.workunit.client.1.smithi066.stdout:9/872: creat da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/f118 x:0 0 0 2024-02-28T16:31:39.672 INFO:tasks.workunit.client.1.smithi066.stdout:9/873: truncate da/d3a/d15/f6a 4937006 0 2024-02-28T16:31:39.675 INFO:tasks.workunit.client.1.smithi066.stdout:5/999: sync 2024-02-28T16:31:39.696 INFO:tasks.workunit.client.1.smithi066.stdout:9/874: dwrite da/d22/d46/d56/d8f/f82 [0,4194304] 0 2024-02-28T16:31:39.707 INFO:tasks.workunit.client.1.smithi066.stdout:9/875: write da/d22/d46/d56/d8f/fb9 [2140975,21103] 0 2024-02-28T16:31:39.772 INFO:tasks.workunit.client.1.smithi066.stdout:9/876: dread da/d3a/d58/dfd/f2f [4194304,4194304] 0 2024-02-28T16:31:39.778 INFO:tasks.workunit.client.1.smithi066.stdout:9/877: link da/d4b/d53/dba/f5c da/d6b/dd8/f119 0 2024-02-28T16:31:39.800 INFO:tasks.workunit.client.1.smithi066.stdout:9/878: truncate da/d22/d46/d56/d8f/d28/f2b 895472 0 2024-02-28T16:31:39.813 INFO:tasks.workunit.client.1.smithi066.stdout:9/879: dwrite da/d6b/f100 [0,4194304] 0 2024-02-28T16:31:39.821 INFO:tasks.workunit.client.1.smithi066.stdout:9/880: link da/d22/d46/d56/d8f/d28/d42/d68/de2/ldf da/d3a/d58/d97/l11a 0 2024-02-28T16:31:39.843 INFO:tasks.workunit.client.1.smithi066.stdout:9/881: dread da/d22/d46/f74 [4194304,4194304] 0 2024-02-28T16:31:39.862 INFO:tasks.workunit.client.1.smithi066.stdout:9/882: dwrite da/d22/d46/d56/f91 [0,4194304] 0 2024-02-28T16:31:39.883 INFO:tasks.workunit.client.1.smithi066.stdout:9/883: dwrite da/d22/d46/d56/d8f/d28/d42/d68/fd7 [0,4194304] 0 2024-02-28T16:31:39.889 INFO:tasks.workunit.client.1.smithi066.stdout:9/884: creat da/d22/d46/d56/d8f/d28/d42/d68/f11b x:0 0 0 2024-02-28T16:31:39.895 INFO:tasks.workunit.client.1.smithi066.stdout:9/885: mkdir da/d3a/d58/d11c 0 2024-02-28T16:31:39.903 INFO:tasks.workunit.client.1.smithi066.stdout:9/886: write da/d22/d46/d56/d8f/d28/d42/f49 [3163634,111341] 0 2024-02-28T16:31:39.907 INFO:tasks.workunit.client.1.smithi066.stdout:9/887: rmdir da/d22/d46/d56/d8f/d28/d42 39 2024-02-28T16:31:39.909 INFO:tasks.workunit.client.1.smithi066.stdout:9/888: symlink da/l11d 0 2024-02-28T16:31:39.911 INFO:tasks.workunit.client.1.smithi066.stdout:9/889: readlink da/d4b/d5e/ldc 0 2024-02-28T16:31:39.916 INFO:tasks.workunit.client.1.smithi066.stdout:9/890: getdents da/d22/d46/d56 0 2024-02-28T16:31:39.920 INFO:tasks.workunit.client.1.smithi066.stdout:9/891: creat da/f11e x:0 0 0 2024-02-28T16:31:39.924 INFO:tasks.workunit.client.1.smithi066.stdout:9/892: creat da/d4b/d53/dd9/f11f x:0 0 0 2024-02-28T16:31:40.229 INFO:tasks.workunit.client.1.smithi066.stdout:9/893: sync 2024-02-28T16:31:40.246 INFO:tasks.workunit.client.1.smithi066.stdout:9/894: truncate da/d6b/f100 1815224 0 2024-02-28T16:31:40.258 INFO:tasks.workunit.client.1.smithi066.stdout:9/895: symlink da/d22/d46/d56/d8f/d28/d42/d68/dda/dfe/l120 0 2024-02-28T16:31:40.268 INFO:tasks.workunit.client.1.smithi066.stdout:9/896: truncate da/d22/d46/d56/d8f/d28/d42/d68/f84 1008203 0 2024-02-28T16:31:40.271 INFO:tasks.workunit.client.1.smithi066.stdout:9/897: mknod da/d4b/d53/dd9/c121 0 2024-02-28T16:31:40.274 INFO:tasks.workunit.client.1.smithi066.stdout:9/898: fsync da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/ff2 0 2024-02-28T16:31:40.296 INFO:tasks.workunit.client.1.smithi066.stdout:9/899: dwrite da/d4b/d53/fc7 [0,4194304] 0 2024-02-28T16:31:40.358 INFO:tasks.workunit.client.1.smithi066.stdout:9/900: sync 2024-02-28T16:31:40.424 INFO:tasks.workunit.client.1.smithi066.stdout:9/901: sync 2024-02-28T16:31:40.444 INFO:tasks.workunit.client.1.smithi066.stdout:9/902: rename da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3/cf3 to da/d22/d46/d56/d8f/dbe/c122 0 2024-02-28T16:31:40.453 INFO:tasks.workunit.client.1.smithi066.stdout:9/903: link da/d22/d46/d56/d8f/dbe/f107 da/d4b/d53/dcb/d10b/f123 0 2024-02-28T16:31:40.457 INFO:tasks.workunit.client.1.smithi066.stdout:9/904: symlink da/d22/d46/d56/d8f/d28/d42/d68/dda/dfe/l124 0 2024-02-28T16:31:40.461 INFO:tasks.workunit.client.1.smithi066.stdout:9/905: creat da/d22/d46/d56/d8f/d28/d42/d68/dda/dd1/dc9/f125 x:0 0 0 2024-02-28T16:31:40.472 INFO:tasks.workunit.client.1.smithi066.stdout:9/906: link da/f64 da/d3a/d15/df5/f126 0 2024-02-28T16:31:40.475 INFO:tasks.workunit.client.1.smithi066.stdout:9/907: chown da/d22/d46/d56/d8f/d28/d42/d68/f11b 1 1 2024-02-28T16:31:40.477 INFO:tasks.workunit.client.1.smithi066.stdout:9/908: chown da/d4b/d53/c9f 7085 1 2024-02-28T16:31:40.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:40 smithi001 ceph-mon[28991]: Standby manager daemon smithi001.rmthtx restarted 2024-02-28T16:31:40.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:40 smithi001 ceph-mon[28991]: Standby manager daemon smithi001.rmthtx started 2024-02-28T16:31:40.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:40 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi001.rmthtx/crt"}]: dispatch 2024-02-28T16:31:40.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:40 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:31:40.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:40 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi001.rmthtx/key"}]: dispatch 2024-02-28T16:31:40.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:40 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:31:40.601 INFO:tasks.workunit.client.1.smithi066.stdout:9/909: sync 2024-02-28T16:31:40.604 INFO:tasks.workunit.client.1.smithi066.stdout:9/910: write da/d3a/d58/dfd/f10e [402275,54425] 0 2024-02-28T16:31:40.609 INFO:tasks.workunit.client.1.smithi066.stdout:9/911: unlink da/d4b/d53/fa7 0 2024-02-28T16:31:40.613 INFO:tasks.workunit.client.1.smithi066.stdout:9/912: readlink da/d22/d46/d56/d8f/d28/d42/d68/dda/lf7 0 2024-02-28T16:31:40.616 INFO:tasks.workunit.client.1.smithi066.stdout:9/913: mknod da/d4b/d53/dd9/c127 0 2024-02-28T16:31:40.619 INFO:tasks.workunit.client.1.smithi066.stdout:9/914: symlink da/d22/d46/d56/d8f/d28/d42/d68/dda/dfe/l128 0 2024-02-28T16:31:40.622 INFO:tasks.workunit.client.1.smithi066.stdout:9/915: mknod da/d6b/c129 0 2024-02-28T16:31:40.624 INFO:tasks.workunit.client.1.smithi066.stdout:9/916: fsync da/d22/d46/d56/d8f/d2d/f57 0 2024-02-28T16:31:40.638 INFO:tasks.workunit.client.1.smithi066.stdout:9/917: rmdir da/d22/d46/d56/d8f/d28 39 2024-02-28T16:31:40.641 INFO:tasks.workunit.client.1.smithi066.stdout:9/918: chown da/d22/d46/d56/d8f/d28/d42/d68/de2/d98 3664555 1 2024-02-28T16:31:40.721 INFO:tasks.workunit.client.1.smithi066.stdout:9/919: sync 2024-02-28T16:31:40.724 INFO:tasks.workunit.client.1.smithi066.stdout:9/920: readlink da/d22/d46/d56/d8f/d28/d42/d68/dda/lf7 0 2024-02-28T16:31:40.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:40 smithi066 ceph-mon[36236]: Standby manager daemon smithi001.rmthtx restarted 2024-02-28T16:31:40.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:40 smithi066 ceph-mon[36236]: Standby manager daemon smithi001.rmthtx started 2024-02-28T16:31:40.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:40 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi001.rmthtx/crt"}]: dispatch 2024-02-28T16:31:40.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:40 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:31:40.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:40 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi001.rmthtx/key"}]: dispatch 2024-02-28T16:31:40.745 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:40 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:31:40.765 INFO:tasks.workunit.client.1.smithi066.stdout:9/921: mknod da/d3a/d58/dab/c12a 0 2024-02-28T16:31:40.766 INFO:tasks.workunit.client.1.smithi066.stdout:9/922: chown da/d3a/d15/fb1 575251613 1 2024-02-28T16:31:40.868 INFO:tasks.workunit.client.1.smithi066.stdout:9/923: sync 2024-02-28T16:31:40.872 INFO:tasks.workunit.client.1.smithi066.stdout:9/924: chown da/d4b/d53/dba/f5c 144052657 1 2024-02-28T16:31:40.876 INFO:tasks.workunit.client.1.smithi066.stdout:9/925: mknod da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3/c12b 0 2024-02-28T16:31:40.891 INFO:tasks.workunit.client.1.smithi066.stdout:9/926: dwrite da/d4b/d53/dba/f60 [0,4194304] 0 2024-02-28T16:31:40.934 INFO:tasks.workunit.client.1.smithi066.stdout:9/927: truncate da/d22/d46/d56/d8f/d28/fa5 2484000 0 2024-02-28T16:31:40.936 INFO:tasks.workunit.client.1.smithi066.stdout:9/928: mknod da/d3a/d58/dfd/c12c 0 2024-02-28T16:31:40.953 INFO:tasks.workunit.client.1.smithi066.stdout:9/929: dwrite da/d6b/dd8/f119 [0,4194304] 0 2024-02-28T16:31:40.976 INFO:tasks.workunit.client.1.smithi066.stdout:9/930: dwrite da/d22/d46/d56/d8f/f82 [0,4194304] 0 2024-02-28T16:31:40.982 INFO:tasks.workunit.client.1.smithi066.stdout:9/931: truncate da/d3a/d15/f6a 2067395 0 2024-02-28T16:31:40.989 INFO:tasks.workunit.client.1.smithi066.stdout:9/932: dread da/d22/d46/d56/d8f/d28/f2b [0,4194304] 0 2024-02-28T16:31:40.998 INFO:tasks.workunit.client.1.smithi066.stdout:9/933: link da/d22/d46/d56/d8f/d28/c5b da/d4b/d53/dba/c12d 0 2024-02-28T16:31:41.020 INFO:tasks.workunit.client.1.smithi066.stdout:9/934: write da/d22/d46/d56/d8f/d2d/f62 [215806,13027] 0 2024-02-28T16:31:41.025 INFO:tasks.workunit.client.1.smithi066.stdout:9/935: creat da/d4b/d53/dd9/f12e x:0 0 0 2024-02-28T16:31:41.035 INFO:tasks.workunit.client.1.smithi066.stdout:9/936: write da/d22/d46/d56/d8f/dbe/f107 [4350680,23780] 0 2024-02-28T16:31:41.037 INFO:tasks.workunit.client.1.smithi066.stdout:9/937: chown da/d3a/d58/d97/ccd 873 1 2024-02-28T16:31:41.041 INFO:tasks.workunit.client.1.smithi066.stdout:9/938: truncate da/d22/d46/d56/d8f/d2d/f57 1935348 0 2024-02-28T16:31:41.052 INFO:tasks.workunit.client.1.smithi066.stdout:9/939: dread da/d22/d46/d56/d8f/f82 [0,4194304] 0 2024-02-28T16:31:41.054 INFO:tasks.workunit.client.1.smithi066.stdout:9/940: chown da/d3a/d58/dfd/c1f 66 1 2024-02-28T16:31:41.063 INFO:tasks.workunit.client.1.smithi066.stdout:9/941: dread da/d22/d46/d56/d8f/d2d/ff1 [0,4194304] 0 2024-02-28T16:31:41.064 INFO:tasks.workunit.client.1.smithi066.stdout:9/942: readlink da/d3a/d58/dfd/ld3 0 2024-02-28T16:31:41.079 INFO:tasks.workunit.client.1.smithi066.stdout:9/943: dread da/d3a/f13 [0,4194304] 0 2024-02-28T16:31:41.095 INFO:tasks.workunit.client.1.smithi066.stdout:9/944: dwrite da/d3a/d15/d48/d76/f106 [0,4194304] 0 2024-02-28T16:31:41.106 INFO:tasks.workunit.client.1.smithi066.stdout:9/945: mknod da/d22/d46/d56/d8f/d28/d42/d68/de2/d98/daa/de3/c12f 0 2024-02-28T16:31:41.122 INFO:tasks.workunit.client.1.smithi066.stdout:9/946: dwrite da/d4b/d53/dba/f60 [0,4194304] 0 2024-02-28T16:31:41.133 INFO:tasks.workunit.client.1.smithi066.stdout:9/947: link da/d6b/dbc/fca da/d22/d46/d56/d8f/d2d/f130 0 2024-02-28T16:31:41.138 INFO:tasks.workunit.client.1.smithi066.stdout:9/948: mkdir da/d4b/d53/dcb/d10b/d131 0 2024-02-28T16:31:41.145 INFO:tasks.workunit.client.1.smithi066.stdout:9/949: creat da/d22/d46/d56/d8f/d28/f132 x:0 0 0 2024-02-28T16:31:41.148 INFO:tasks.workunit.client.1.smithi066.stdout:9/950: symlink da/d3a/d15/l133 0 2024-02-28T16:31:41.153 INFO:tasks.workunit.client.1.smithi066.stdout:9/951: read da/d22/d46/d56/d8f/d28/d42/d68/de2/f93 [484390,30050] 0 2024-02-28T16:31:41.313 INFO:tasks.workunit.client.1.smithi066.stdout:9/952: sync 2024-02-28T16:31:41.324 INFO:tasks.workunit.client.1.smithi066.stdout:9/953: rmdir da/d22/d46/d56/d8f/d2d 39 2024-02-28T16:31:41.342 INFO:tasks.workunit.client.1.smithi066.stdout:9/954: dread da/f2a [0,4194304] 0 2024-02-28T16:31:41.346 INFO:tasks.workunit.client.1.smithi066.stdout:9/955: dread - da/d22/d46/d56/d8f/d28/f132 zero size 2024-02-28T16:31:41.356 INFO:tasks.workunit.client.1.smithi066.stdout:9/956: mkdir da/d4b/d53/dcb/d10b/d131/d134 0 2024-02-28T16:31:41.358 INFO:tasks.workunit.client.1.smithi066.stdout:9/957: chown da/d3a/d58/dab 1 1 2024-02-28T16:31:41.363 INFO:tasks.workunit.client.1.smithi066.stdout:9/958: write da/d6b/dd8/f119 [2974279,86901] 0 2024-02-28T16:31:41.402 INFO:tasks.workunit.client.1.smithi066.stdout:9/959: dread da/d22/d46/d56/d8f/d2d/f62 [0,4194304] 0 2024-02-28T16:31:41.417 INFO:tasks.workunit.client.1.smithi066.stdout:9/960: rename da/d22 to da/d3a/d58/dab/d135 0 2024-02-28T16:31:41.425 INFO:tasks.workunit.client.1.smithi066.stdout:9/961: link da/d3a/d15/d48/d76/f114 da/d3a/d58/dab/d135/f136 0 2024-02-28T16:31:41.429 INFO:tasks.workunit.client.1.smithi066.stdout:9/962: unlink da/d4b/d53/dba/f5c 0 2024-02-28T16:31:41.430 INFO:tasks.workunit.client.1.smithi066.stdout:9/963: stat da/d4b/d53/laf 0 2024-02-28T16:31:41.446 INFO:tasks.workunit.client.1.smithi066.stdout:9/964: dwrite da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/de2/f93 [0,4194304] 0 2024-02-28T16:31:41.451 INFO:tasks.workunit.client.1.smithi066.stdout:9/965: chown da/d3a/d58/lae 0 1 2024-02-28T16:31:41.561 INFO:tasks.workunit.client.1.smithi066.stdout:9/966: sync 2024-02-28T16:31:41.565 INFO:tasks.workunit.client.1.smithi066.stdout:9/967: mkdir da/d3a/d58/dfd/d137 0 2024-02-28T16:31:41.573 INFO:tasks.workunit.client.1.smithi066.stdout:9/968: symlink da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/dda/dd1/l138 0 2024-02-28T16:31:41.579 INFO:tasks.workunit.client.1.smithi066.stdout:9/969: write da/d3a/d58/dab/d135/d46/f74 [9050673,49674] 0 2024-02-28T16:31:41.584 INFO:tasks.workunit.client.1.smithi066.stdout:9/970: creat da/d6b/f139 x:0 0 0 2024-02-28T16:31:41.589 INFO:tasks.workunit.client.1.smithi066.stdout:9/971: symlink da/d3a/d58/dab/d135/d46/d56/d8f/l13a 0 2024-02-28T16:31:41.603 INFO:tasks.workunit.client.1.smithi066.stdout:9/972: dwrite da/d3a/fac [0,4194304] 0 2024-02-28T16:31:41.609 INFO:tasks.workunit.client.1.smithi066.stdout:9/973: getdents da/d4b 0 2024-02-28T16:31:41.614 INFO:tasks.workunit.client.1.smithi066.stdout:9/974: mknod da/d3a/d58/dab/d135/d46/d56/d8f/d28/c13b 0 2024-02-28T16:31:41.616 INFO:tasks.workunit.client.1.smithi066.stdout:9/975: mknod da/d4b/c13c 0 2024-02-28T16:31:41.622 INFO:tasks.workunit.client.1.smithi066.stdout:9/976: getdents da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/de2/d98/daa/d108 0 2024-02-28T16:31:41.630 INFO:tasks.workunit.client.1.smithi066.stdout:9/977: truncate da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/f9d 31385 0 2024-02-28T16:31:41.636 INFO:tasks.workunit.client.1.smithi066.stdout:9/978: write da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/fea [1016317,95992] 0 2024-02-28T16:31:41.637 INFO:tasks.workunit.client.1.smithi066.stdout:9/979: chown da/d3a/d58/dab/d135/d46/d56/f115 296 1 2024-02-28T16:31:41.638 INFO:tasks.workunit.client.1.smithi066.stdout:9/980: fsync da/fb2 0 2024-02-28T16:31:41.642 INFO:tasks.workunit.client.1.smithi066.stdout:9/981: rename da/d3a/d15/dcf to da/d3a/d58/dab/d13d 0 2024-02-28T16:31:41.648 INFO:tasks.workunit.client.1.smithi066.stdout:9/982: rename da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/f49 to da/d4b/d53/dcb/f13e 0 2024-02-28T16:31:41.654 INFO:tasks.workunit.client.1.smithi066.stdout:9/983: creat da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/dda/f13f x:0 0 0 2024-02-28T16:31:41.673 INFO:tasks.workunit.client.1.smithi066.stdout:9/984: truncate da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/feb 3699768 0 2024-02-28T16:31:41.677 INFO:tasks.workunit.client.1.smithi066.stdout:9/985: mknod da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/c140 0 2024-02-28T16:31:41.684 INFO:tasks.workunit.client.1.smithi066.stdout:9/986: getdents da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/dda/dd1/dc9 0 2024-02-28T16:31:41.687 INFO:tasks.workunit.client.1.smithi066.stdout:9/987: chown da/d3a/d58/dab/d135/d46/d56/cdd 1 1 2024-02-28T16:31:41.692 INFO:tasks.workunit.client.1.smithi066.stdout:9/988: creat da/d3a/d58/d11c/f141 x:0 0 0 2024-02-28T16:31:41.706 INFO:tasks.workunit.client.1.smithi066.stdout:9/989: truncate da/d3a/d58/dab/d135/d46/d56/d8f/dbe/f107 3819073 0 2024-02-28T16:31:41.733 INFO:tasks.workunit.client.1.smithi066.stdout:9/990: dwrite da/d3a/d15/fb1 [0,4194304] 0 2024-02-28T16:31:41.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:41 smithi066 ceph-mon[36236]: pgmap v64: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 82 MiB/s rd, 190 MiB/s wr, 505 op/s 2024-02-28T16:31:41.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:41 smithi066 ceph-mon[36236]: mgrmap e26: smithi066.ztraxe(active, since 2m), standbys: smithi001.rmthtx 2024-02-28T16:31:41.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:41 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:41.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:41 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:41.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:41 smithi001 ceph-mon[28991]: pgmap v64: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 82 MiB/s rd, 190 MiB/s wr, 505 op/s 2024-02-28T16:31:41.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:41 smithi001 ceph-mon[28991]: mgrmap e26: smithi066.ztraxe(active, since 2m), standbys: smithi001.rmthtx 2024-02-28T16:31:41.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:41 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:41.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:41 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:41.780 INFO:tasks.workunit.client.1.smithi066.stdout:9/991: creat da/d6b/dd8/f142 x:0 0 0 2024-02-28T16:31:41.784 INFO:tasks.workunit.client.1.smithi066.stdout:9/992: write da/d3a/d15/d48/d76/f113 [213322,87876] 0 2024-02-28T16:31:41.787 INFO:tasks.workunit.client.1.smithi066.stdout:9/993: mknod da/d4b/d53/dcb/d10b/d131/c143 0 2024-02-28T16:31:41.790 INFO:tasks.workunit.client.1.smithi066.stdout:9/994: fdatasync da/f2a 0 2024-02-28T16:31:41.794 INFO:tasks.workunit.client.1.smithi066.stdout:9/995: creat da/d4b/d5e/f144 x:0 0 0 2024-02-28T16:31:41.804 INFO:tasks.workunit.client.1.smithi066.stdout:9/996: write da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/de2/d98/dfa/f105 [1506945,118004] 0 2024-02-28T16:31:41.805 INFO:tasks.workunit.client.1.smithi066.stdout:9/997: chown da/d3a/d58/lae 588 1 2024-02-28T16:31:41.809 INFO:tasks.workunit.client.1.smithi066.stdout:9/998: chown da/d3a/d58/dab/d135/d46/d56/d8f/d28/d42/d68/f84 2 1 2024-02-28T16:31:41.818 INFO:tasks.workunit.client.1.smithi066.stdout:9/999: write da/d6b/dbc/fca [14989,15470] 0 2024-02-28T16:31:41.825 INFO:tasks.workunit.client.1.smithi066.stderr:+ rm -rf -- ./tmp.IEDwljntxx 2024-02-28T16:31:43.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:43 smithi066 ceph-mon[36236]: pgmap v65: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 42 MiB/s rd, 95 MiB/s wr, 293 op/s 2024-02-28T16:31:43.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:43 smithi001 ceph-mon[28991]: pgmap v65: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 42 MiB/s rd, 95 MiB/s wr, 293 op/s 2024-02-28T16:31:45.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:45.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr fail", "who": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr fail", "who": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: osdmap e47: 6 total, 6 up, 6 in 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd='[{"prefix": "mgr fail", "who": "smithi066.ztraxe"}]': finished 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: mgrmap e27: smithi001.rmthtx(active, starting, since 0.00894894s) 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:31:45.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: Manager daemon smithi001.rmthtx is now available 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:45.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:45.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:31:45.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:31:45.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/trash_purge_schedule"}]: dispatch 2024-02-28T16:31:45.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:45.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:45.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:31:45.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:31:45.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' 2024-02-28T16:31:45.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:45.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 172.21.15.66:0/1430919028' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr fail", "who": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:31:45.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "mgr fail", "who": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: osdmap e47: 6 total, 6 up, 6 in 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24347 ' entity='mgr.smithi066.ztraxe' cmd='[{"prefix": "mgr fail", "who": "smithi066.ztraxe"}]': finished 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: mgrmap e27: smithi001.rmthtx(active, starting, since 0.00894894s) 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:31:45.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: Manager daemon smithi001.rmthtx is now available 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:31:45.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/trash_purge_schedule"}]: dispatch 2024-02-28T16:31:46.606 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-28T16:31:46.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-28T16:31:46.707 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:46 smithi001 ceph-mon[28991]: [28/Feb/2024:16:31:45] ENGINE Bus STARTING 2024-02-28T16:31:46.707 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:46 smithi001 ceph-mon[28991]: mgrmap e28: smithi001.rmthtx(active, since 1.01273s) 2024-02-28T16:31:46.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:46 smithi066 ceph-mon[36236]: [28/Feb/2024:16:31:45] ENGINE Bus STARTING 2024-02-28T16:31:46.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:46 smithi066 ceph-mon[36236]: mgrmap e28: smithi001.rmthtx(active, since 1.01273s) 2024-02-28T16:31:47.352 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:47 smithi001 ceph-mon[28991]: [28/Feb/2024:16:31:45] ENGINE Serving on http://172.21.15.1:8765 2024-02-28T16:31:47.352 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:47 smithi001 ceph-mon[28991]: [28/Feb/2024:16:31:45] ENGINE Serving on https://172.21.15.1:7150 2024-02-28T16:31:47.352 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:47 smithi001 ceph-mon[28991]: [28/Feb/2024:16:31:45] ENGINE Bus STARTED 2024-02-28T16:31:47.353 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:47 smithi001 ceph-mon[28991]: pgmap v3: 65 pgs: 65 active+clean; 2.3 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail 2024-02-28T16:31:47.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:47 smithi066 ceph-mon[36236]: [28/Feb/2024:16:31:45] ENGINE Serving on http://172.21.15.1:8765 2024-02-28T16:31:47.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:47 smithi066 ceph-mon[36236]: [28/Feb/2024:16:31:45] ENGINE Serving on https://172.21.15.1:7150 2024-02-28T16:31:47.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:47 smithi066 ceph-mon[36236]: [28/Feb/2024:16:31:45] ENGINE Bus STARTED 2024-02-28T16:31:47.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:47 smithi066 ceph-mon[36236]: pgmap v3: 65 pgs: 65 active+clean; 2.3 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail 2024-02-28T16:31:48.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:48 smithi066 ceph-mon[36236]: pgmap v4: 65 pgs: 65 active+clean; 2.3 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail 2024-02-28T16:31:48.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:48 smithi066 ceph-mon[36236]: mgrmap e29: smithi001.rmthtx(active, since 2s) 2024-02-28T16:31:48.649 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:48 smithi001 ceph-mon[28991]: pgmap v4: 65 pgs: 65 active+clean; 2.3 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail 2024-02-28T16:31:48.650 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:48 smithi001 ceph-mon[28991]: mgrmap e29: smithi001.rmthtx(active, since 2s) 2024-02-28T16:31:50.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:50 smithi066 ceph-mon[36236]: pgmap v5: 65 pgs: 65 active+clean; 2.3 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail 2024-02-28T16:31:50.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:50 smithi066 ceph-mon[36236]: mgrmap e30: smithi001.rmthtx(active, since 4s) 2024-02-28T16:31:50.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:50 smithi001 ceph-mon[28991]: pgmap v5: 65 pgs: 65 active+clean; 2.3 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail 2024-02-28T16:31:50.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:50 smithi001 ceph-mon[28991]: mgrmap e30: smithi001.rmthtx(active, since 4s) 2024-02-28T16:31:51.459 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:51 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:51.459 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:51 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:51.459 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:51 smithi066 ceph-mon[36236]: pgmap v6: 65 pgs: 65 active+clean; 2.3 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail 2024-02-28T16:31:51.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:51 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:51.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:51 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:51.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:51 smithi001 ceph-mon[28991]: pgmap v6: 65 pgs: 65 active+clean; 2.3 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail 2024-02-28T16:31:53.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.359+0000 7f15ce200700 1 -- 172.21.15.1:0/3156188107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fcfb0 msgr2=0x7f15c80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:53.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.359+0000 7f15ce200700 1 --2- 172.21.15.1:0/3156188107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fcfb0 0x7f15c80fd420 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f15b4004940 tx=0x7f15b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:53.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.359+0000 7f15ce200700 1 -- 172.21.15.1:0/3156188107 shutdown_connections 2024-02-28T16:31:53.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.359+0000 7f15ce200700 1 --2- 172.21.15.1:0/3156188107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fcfb0 0x7f15c80fd420 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.359+0000 7f15ce200700 1 --2- 172.21.15.1:0/3156188107 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15c80fbe40 0x7f15c80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.359+0000 7f15ce200700 1 -- 172.21.15.1:0/3156188107 >> 172.21.15.1:0/3156188107 conn(0x7f15c80f74e0 msgr2=0x7f15c80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:53.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.359+0000 7f15ce200700 1 -- 172.21.15.1:0/3156188107 shutdown_connections 2024-02-28T16:31:53.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.359+0000 7f15ce200700 1 -- 172.21.15.1:0/3156188107 wait complete. 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.360+0000 7f15ce200700 1 Processor -- start 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.361+0000 7f15ce200700 1 -- start start 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.361+0000 7f15ce200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fbe40 0x7f15c810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.361+0000 7f15ce200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15c80fcfb0 0x7f15c81100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.361+0000 7f15ce200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c81106a0 con 0x7f15c80fbe40 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.361+0000 7f15ce200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c8110f40 con 0x7f15c80fcfb0 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.361+0000 7f15c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fbe40 0x7f15c810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.361+0000 7f15c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fbe40 0x7f15c810fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58136/0 (socket says 172.21.15.1:58136) 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.361+0000 7f15c7400700 1 -- 172.21.15.1:0/399879641 learned_addr learned my addr 172.21.15.1:0/399879641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.362+0000 7f15c6a00700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15c80fcfb0 0x7f15c81100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.362+0000 7f15c7400700 1 -- 172.21.15.1:0/399879641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15c80fcfb0 msgr2=0x7f15c81100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.362+0000 7f15c7400700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15c80fcfb0 0x7f15c81100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.362+0000 7f15c7400700 1 -- 172.21.15.1:0/399879641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4005040 con 0x7f15c80fbe40 2024-02-28T16:31:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.362+0000 7f15c6a00700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15c80fcfb0 0x7f15c81100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:31:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.362+0000 7f15c7400700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fbe40 0x7f15c810fb60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f15b8002a50 tx=0x7f15b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.363+0000 7f15bf400700 1 -- 172.21.15.1:0/399879641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15b8007cc0 con 0x7f15c80fbe40 2024-02-28T16:31:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.363+0000 7f15bf400700 1 -- 172.21.15.1:0/399879641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15b800f040 con 0x7f15c80fbe40 2024-02-28T16:31:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.363+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c8111220 con 0x7f15c80fbe40 2024-02-28T16:31:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.363+0000 7f15bf400700 1 -- 172.21.15.1:0/399879641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15b8009640 con 0x7f15c80fbe40 2024-02-28T16:31:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.363+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c8111760 con 0x7f15c80fbe40 2024-02-28T16:31:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.364+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c804bed0 con 0x7f15c80fbe40 2024-02-28T16:31:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.364+0000 7f15bf400700 1 -- 172.21.15.1:0/399879641 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 45926+0+0 (secure 0 0 0) 0x7f15b80097a0 con 0x7f15c80fbe40 2024-02-28T16:31:53.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.365+0000 7f15bf400700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f15ac038a70 0x7f15ac03af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:53.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.365+0000 7f15bf400700 1 -- 172.21.15.1:0/399879641 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f15b804c420 con 0x7f15c80fbe40 2024-02-28T16:31:53.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.365+0000 7f15c6a00700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f15ac038a70 0x7f15ac03af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:53.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.366+0000 7f15c6a00700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f15ac038a70 0x7f15ac03af20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f15b400b7a0 tx=0x7f15b4009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:53.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.372+0000 7f15bf400700 1 -- 172.21.15.1:0/399879641 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f15b8019630 con 0x7f15c80fbe40 2024-02-28T16:31:53.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.664+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f15c8065dd0 con 0x7f15ac038a70 2024-02-28T16:31:53.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.667+0000 7f15bf400700 1 -- 172.21.15.1:0/399879641 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f15c8065dd0 con 0x7f15ac038a70 2024-02-28T16:31:53.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f15ac038a70 msgr2=0x7f15ac03af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:53.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f15ac038a70 0x7f15ac03af20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f15b400b7a0 tx=0x7f15b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:53.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fbe40 msgr2=0x7f15c810fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:53.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fbe40 0x7f15c810fb60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f15b8002a50 tx=0x7f15b80046b0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 shutdown_connections 2024-02-28T16:31:53.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f15ac038a70 0x7f15ac03af20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15c80fbe40 0x7f15c810fb60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 --2- 172.21.15.1:0/399879641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15c80fcfb0 0x7f15c81100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 >> 172.21.15.1:0/399879641 conn(0x7f15c80f74e0 msgr2=0x7f15c80656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:53.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.671+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 shutdown_connections 2024-02-28T16:31:53.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.672+0000 7f15ce200700 1 -- 172.21.15.1:0/399879641 wait complete. 2024-02-28T16:31:53.686 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:31:53.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.885+0000 7fd359c00700 1 -- 172.21.15.1:0/3679091631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354072890 msgr2=0x7fd354072ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:53.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.885+0000 7fd359c00700 1 --2- 172.21.15.1:0/3679091631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354072890 0x7fd354072ca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd344004940 tx=0x7fd344009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:53.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.886+0000 7fd359c00700 1 -- 172.21.15.1:0/3679091631 shutdown_connections 2024-02-28T16:31:53.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.886+0000 7fd359c00700 1 --2- 172.21.15.1:0/3679091631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd354073af0 0x7fd354073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.886+0000 7fd359c00700 1 --2- 172.21.15.1:0/3679091631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354072890 0x7fd354072ca0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.886+0000 7fd359c00700 1 -- 172.21.15.1:0/3679091631 >> 172.21.15.1:0/3679091631 conn(0x7fd3540fa920 msgr2=0x7fd3540fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:53.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.886+0000 7fd359c00700 1 -- 172.21.15.1:0/3679091631 shutdown_connections 2024-02-28T16:31:53.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.886+0000 7fd359c00700 1 -- 172.21.15.1:0/3679091631 wait complete. 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.887+0000 7fd359c00700 1 Processor -- start 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.887+0000 7fd359c00700 1 -- start start 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.887+0000 7fd359c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd354072890 0x7fd354113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.887+0000 7fd359c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354073af0 0x7fd354114420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.887+0000 7fd359c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd354114a20 con 0x7fd354073af0 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.887+0000 7fd359c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3541152c0 con 0x7fd354072890 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.888+0000 7fd352000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354073af0 0x7fd354114420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.888+0000 7fd352000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354073af0 0x7fd354114420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58154/0 (socket says 172.21.15.1:58154) 2024-02-28T16:31:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.888+0000 7fd352000700 1 -- 172.21.15.1:0/1513398774 learned_addr learned my addr 172.21.15.1:0/1513398774 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.888+0000 7fd352a00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd354072890 0x7fd354113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.888+0000 7fd352000700 1 -- 172.21.15.1:0/1513398774 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd354072890 msgr2=0x7fd354113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.888+0000 7fd352000700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd354072890 0x7fd354113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.888+0000 7fd352000700 1 -- 172.21.15.1:0/1513398774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344005040 con 0x7fd354073af0 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.888+0000 7fd352000700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354073af0 0x7fd354114420 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd34000eb40 tx=0x7fd34000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.889+0000 7fd34aa00700 1 -- 172.21.15.1:0/1513398774 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd340007cc0 con 0x7fd354073af0 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.889+0000 7fd34aa00700 1 -- 172.21.15.1:0/1513398774 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd34000f040 con 0x7fd354073af0 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.889+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd354115540 con 0x7fd354073af0 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.889+0000 7fd34aa00700 1 -- 172.21.15.1:0/1513398774 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd34000ca10 con 0x7fd354073af0 2024-02-28T16:31:53.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.889+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd354115a80 con 0x7fd354073af0 2024-02-28T16:31:53.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.890+0000 7fd34aa00700 1 -- 172.21.15.1:0/1513398774 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 45926+0+0 (secure 0 0 0) 0x7fd34000cb70 con 0x7fd354073af0 2024-02-28T16:31:53.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.890+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35404bed0 con 0x7fd354073af0 2024-02-28T16:31:53.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.891+0000 7fd34aa00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd338038a70 0x7fd33803af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.891+0000 7fd34aa00700 1 -- 172.21.15.1:0/1513398774 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fd34004c290 con 0x7fd354073af0 2024-02-28T16:31:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.891+0000 7fd352a00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd338038a70 0x7fd33803af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:53.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.892+0000 7fd352a00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd338038a70 0x7fd33803af20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd3440136a0 tx=0x7fd344009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:53.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:53.896+0000 7fd34aa00700 1 -- 172.21.15.1:0/1513398774 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd340012d70 con 0x7fd354073af0 2024-02-28T16:31:54.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:54 smithi066 ceph-mon[36236]: pgmap v7: 65 pgs: 65 active+clean; 1.6 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 49 KiB/s rd, 791 KiB/s wr, 136 op/s 2024-02-28T16:31:54.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:54 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:54.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:54 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:54.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:54 smithi001 ceph-mon[28991]: pgmap v7: 65 pgs: 65 active+clean; 1.6 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 49 KiB/s rd, 791 KiB/s wr, 136 op/s 2024-02-28T16:31:54.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:54 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:54.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:54 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:54.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.319+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd35410ab80 con 0x7fd338038a70 2024-02-28T16:31:54.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.321+0000 7fd34aa00700 1 -- 172.21.15.1:0/1513398774 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fd35410ab80 con 0x7fd338038a70 2024-02-28T16:31:54.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.323+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd338038a70 msgr2=0x7fd33803af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:54.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.323+0000 7fd359c00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd338038a70 0x7fd33803af20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd3440136a0 tx=0x7fd344009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:54.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.323+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354073af0 msgr2=0x7fd354114420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:54.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.323+0000 7fd359c00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354073af0 0x7fd354114420 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd34000eb40 tx=0x7fd34000b040 comp rx=0 tx=0).stop 2024-02-28T16:31:54.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.324+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 shutdown_connections 2024-02-28T16:31:54.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.324+0000 7fd359c00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd354072890 0x7fd354113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.324+0000 7fd359c00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd338038a70 0x7fd33803af20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.324+0000 7fd359c00700 1 --2- 172.21.15.1:0/1513398774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd354073af0 0x7fd354114420 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.324+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 >> 172.21.15.1:0/1513398774 conn(0x7fd3540fa920 msgr2=0x7fd354109460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:54.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.324+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 shutdown_connections 2024-02-28T16:31:54.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.324+0000 7fd359c00700 1 -- 172.21.15.1:0/1513398774 wait complete. 2024-02-28T16:31:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.510+0000 7f2eaac00700 1 -- 172.21.15.1:0/2166555609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 msgr2=0x7f2ea4100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.510+0000 7f2eaac00700 1 --2- 172.21.15.1:0/2166555609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 0x7f2ea4100650 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2e90004940 tx=0x7f2e90009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.510+0000 7f2eaac00700 1 -- 172.21.15.1:0/2166555609 shutdown_connections 2024-02-28T16:31:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.510+0000 7f2eaac00700 1 --2- 172.21.15.1:0/2166555609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 0x7f2ea4100650 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.510+0000 7f2eaac00700 1 --2- 172.21.15.1:0/2166555609 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ea40fb870 0x7f2ea40fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.510+0000 7f2eaac00700 1 -- 172.21.15.1:0/2166555609 >> 172.21.15.1:0/2166555609 conn(0x7f2ea40f74e0 msgr2=0x7f2ea40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.510+0000 7f2eaac00700 1 -- 172.21.15.1:0/2166555609 shutdown_connections 2024-02-28T16:31:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.510+0000 7f2eaac00700 1 -- 172.21.15.1:0/2166555609 wait complete. 2024-02-28T16:31:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.511+0000 7f2eaac00700 1 Processor -- start 2024-02-28T16:31:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.511+0000 7f2eaac00700 1 -- start start 2024-02-28T16:31:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2eaac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ea40fb870 0x7f2ea4113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2eaac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 0x7f2ea41144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2eaac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea4114aa0 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2eaac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea4115340 con 0x7f2ea40fb870 2024-02-28T16:31:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2ea3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 0x7f2ea41144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2ea3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 0x7f2ea41144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58182/0 (socket says 172.21.15.1:58182) 2024-02-28T16:31:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2ea3400700 1 -- 172.21.15.1:0/935976902 learned_addr learned my addr 172.21.15.1:0/935976902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2ea3e00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ea40fb870 0x7f2ea4113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2ea3400700 1 -- 172.21.15.1:0/935976902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ea40fb870 msgr2=0x7f2ea4113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2ea3400700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ea40fb870 0x7f2ea4113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.512+0000 7f2ea3400700 1 -- 172.21.15.1:0/935976902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e90005040 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.513+0000 7f2ea3400700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 0x7f2ea41144a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009fd0 tx=0x7f2e90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:54.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.513+0000 7f2e9be00700 1 -- 172.21.15.1:0/935976902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e9000e040 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.513+0000 7f2e9be00700 1 -- 172.21.15.1:0/935976902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e9000bcc0 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.513+0000 7f2e9be00700 1 -- 172.21.15.1:0/935976902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e9001c630 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.513+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea41155c0 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.513+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea4115b00 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.515+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea410e050 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.515+0000 7f2e9be00700 1 -- 172.21.15.1:0/935976902 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 45926+0+0 (secure 0 0 0) 0x7f2e90013420 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.516+0000 7f2e9be00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2e88038a70 0x7f2e8803af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:54.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.516+0000 7f2e9be00700 1 -- 172.21.15.1:0/935976902 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f2e9004daf0 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.518+0000 7f2ea3e00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2e88038a70 0x7f2e8803af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:54.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.518+0000 7f2ea3e00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2e88038a70 0x7f2e8803af20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e94004f40 tx=0x7f2e9400b040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:54.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.522+0000 7f2e9be00700 1 -- 172.21.15.1:0/935976902 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f2e90025400 con 0x7f2ea40fe1d0 2024-02-28T16:31:54.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.802+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ea4115e50 con 0x7f2e88038a70 2024-02-28T16:31:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.823+0000 7f2e9be00700 1 -- 172.21.15.1:0/935976902 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f2ea4115e50 con 0x7f2e88038a70 2024-02-28T16:31:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:31:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (97s) 19s ago 12m 14.5M - 0.25.0 c8568f914cd2 04ac88629633 2024-02-28T16:31:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (12m) 19s ago 12m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:31:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (11m) 4s ago 11m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:31:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (73s) 19s ago 11m 73.8M - 9.4.7 954c08fa6188 1010f22bd792 2024-02-28T16:31:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (5m) 19s ago 5m 137M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:31:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (5m) 19s ago 5m 12.3M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:31:54.812 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (4m) 4s ago 4m 12.6M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (5m) 4s ago 5m 181M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (28s) 19s ago 14m 66.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283 running (2m) 4s ago 10m 70.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 30a2de341a4e 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (14m) 19s ago 14m 49.5M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (10m) 4s ago 10m 42.2M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (105s) 19s ago 12m 7172k - 1.5.0 0da6a335fe13 1a0f5b104896 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (52s) 4s ago 11m 7554k - 1.5.0 0da6a335fe13 5697e47a4eda 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (9m) 19s ago 9m 187M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (9m) 19s ago 9m 186M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (8m) 19s ago 8m 160M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (8m) 4s ago 8m 206M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (7m) 4s ago 7m 181M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (6m) 4s ago 6m 196M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:31:54.813 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (60s) 19s ago 11m 42.0M - 2.43.0 a07b618ecd1d 72e354bf7e21 2024-02-28T16:31:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2e88038a70 msgr2=0x7f2e8803af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2e88038a70 0x7f2e8803af20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e94004f40 tx=0x7f2e9400b040 comp rx=0 tx=0).stop 2024-02-28T16:31:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 msgr2=0x7f2ea41144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 0x7f2ea41144a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009fd0 tx=0x7f2e90009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 shutdown_connections 2024-02-28T16:31:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ea40fb870 0x7f2ea4113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2e88038a70 0x7f2e8803af20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 --2- 172.21.15.1:0/935976902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ea40fe1d0 0x7f2ea41144a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 >> 172.21.15.1:0/935976902 conn(0x7f2ea40f74e0 msgr2=0x7f2ea40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 shutdown_connections 2024-02-28T16:31:54.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.828+0000 7f2eaac00700 1 -- 172.21.15.1:0/935976902 wait complete. 2024-02-28T16:31:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.985+0000 7f261be00700 1 -- 172.21.15.1:0/2507390087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614066c50 msgr2=0x7f2614101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.985+0000 7f261be00700 1 --2- 172.21.15.1:0/2507390087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614066c50 0x7f2614101f10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2608004940 tx=0x7f2608009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.985+0000 7f261be00700 1 -- 172.21.15.1:0/2507390087 shutdown_connections 2024-02-28T16:31:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.985+0000 7f261be00700 1 --2- 172.21.15.1:0/2507390087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2614102450 0x7f2614104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.985+0000 7f261be00700 1 --2- 172.21.15.1:0/2507390087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614066c50 0x7f2614101f10 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.985+0000 7f261be00700 1 -- 172.21.15.1:0/2507390087 >> 172.21.15.1:0/2507390087 conn(0x7f26140f7500 msgr2=0x7f26140f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.986+0000 7f261be00700 1 -- 172.21.15.1:0/2507390087 shutdown_connections 2024-02-28T16:31:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.986+0000 7f261be00700 1 -- 172.21.15.1:0/2507390087 wait complete. 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f261be00700 1 Processor -- start 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f261be00700 1 -- start start 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f261be00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2614066c50 0x7f261410fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f261be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614102450 0x7f26141100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f261be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26141106d0 con 0x7f2614102450 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f261be00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2614110f70 con 0x7f2614066c50 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f2618a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614102450 0x7f26141100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f2618a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614102450 0x7f26141100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58202/0 (socket says 172.21.15.1:58202) 2024-02-28T16:31:54.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.987+0000 7f2618a00700 1 -- 172.21.15.1:0/2196092945 learned_addr learned my addr 172.21.15.1:0/2196092945 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.988+0000 7f2619400700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2614066c50 0x7f261410fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.988+0000 7f2618a00700 1 -- 172.21.15.1:0/2196092945 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2614066c50 msgr2=0x7f261410fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.988+0000 7f2618a00700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2614066c50 0x7f261410fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.988+0000 7f2618a00700 1 -- 172.21.15.1:0/2196092945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608005040 con 0x7f2614102450 2024-02-28T16:31:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.988+0000 7f2618a00700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614102450 0x7f26141100d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2600002a80 tx=0x7f2600004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.989+0000 7f2611600700 1 -- 172.21.15.1:0/2196092945 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2600007cc0 con 0x7f2614102450 2024-02-28T16:31:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.989+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2614111250 con 0x7f2614102450 2024-02-28T16:31:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.989+0000 7f2611600700 1 -- 172.21.15.1:0/2196092945 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f260000f040 con 0x7f2614102450 2024-02-28T16:31:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.989+0000 7f2611600700 1 -- 172.21.15.1:0/2196092945 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f260000daf0 con 0x7f2614102450 2024-02-28T16:31:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.989+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2614111790 con 0x7f2614102450 2024-02-28T16:31:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.991+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261404bed0 con 0x7f2614102450 2024-02-28T16:31:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.992+0000 7f2611600700 1 -- 172.21.15.1:0/2196092945 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 45926+0+0 (secure 0 0 0) 0x7f260000dc70 con 0x7f2614102450 2024-02-28T16:31:54.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.992+0000 7f2611600700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f25fc038a70 0x7f25fc03af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:54.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.992+0000 7f2611600700 1 -- 172.21.15.1:0/2196092945 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f260004ddd0 con 0x7f2614102450 2024-02-28T16:31:54.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.993+0000 7f2619400700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f25fc038a70 0x7f25fc03af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.994+0000 7f2619400700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f25fc038a70 0x7f25fc03af20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f260800b7a0 tx=0x7f2608009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:54.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:54.998+0000 7f2611600700 1 -- 172.21.15.1:0/2196092945 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f2600051060 con 0x7f2614102450 2024-02-28T16:31:55.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.365+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2614111dc0 con 0x7f2614102450 2024-02-28T16:31:55.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.366+0000 7f2611600700 1 -- 172.21.15.1:0/2196092945 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f2600051300 con 0x7f2614102450 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:31:55.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:31:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:31:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:31:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:31:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:31:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 12, 2024-02-28T16:31:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:31:55.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:31:55.356 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:31:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f25fc038a70 msgr2=0x7f25fc03af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f25fc038a70 0x7f25fc03af20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f260800b7a0 tx=0x7f2608009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614102450 msgr2=0x7f26141100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614102450 0x7f26141100d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2600002a80 tx=0x7f2600004630 comp rx=0 tx=0).stop 2024-02-28T16:31:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 shutdown_connections 2024-02-28T16:31:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2614066c50 0x7f261410fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f25fc038a70 0x7f25fc03af20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 --2- 172.21.15.1:0/2196092945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2614102450 0x7f26141100d0 secure :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2600002a80 tx=0x7f2600004630 comp rx=0 tx=0).stop 2024-02-28T16:31:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.369+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 >> 172.21.15.1:0/2196092945 conn(0x7f26140f7500 msgr2=0x7f26140f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.370+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 shutdown_connections 2024-02-28T16:31:55.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.370+0000 7f261be00700 1 -- 172.21.15.1:0/2196092945 wait complete. 2024-02-28T16:31:55.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.547+0000 7faac4e00700 1 -- 172.21.15.1:0/1055869131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac0071f10 msgr2=0x7faac0070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:55.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.547+0000 7faac4e00700 1 --2- 172.21.15.1:0/1055869131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac0071f10 0x7faac0070380 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faab0004940 tx=0x7faab0009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:55.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.547+0000 7faac4e00700 1 -- 172.21.15.1:0/1055869131 shutdown_connections 2024-02-28T16:31:55.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.547+0000 7faac4e00700 1 --2- 172.21.15.1:0/1055869131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faac00708c0 0x7faac0070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.547+0000 7faac4e00700 1 --2- 172.21.15.1:0/1055869131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac0071f10 0x7faac0070380 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.547+0000 7faac4e00700 1 -- 172.21.15.1:0/1055869131 >> 172.21.15.1:0/1055869131 conn(0x7faac00f8620 msgr2=0x7faac00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:55.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.547+0000 7faac4e00700 1 -- 172.21.15.1:0/1055869131 shutdown_connections 2024-02-28T16:31:55.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.547+0000 7faac4e00700 1 -- 172.21.15.1:0/1055869131 wait complete. 2024-02-28T16:31:55.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.548+0000 7faac4e00700 1 Processor -- start 2024-02-28T16:31:55.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.548+0000 7faac4e00700 1 -- start start 2024-02-28T16:31:55.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.548+0000 7faac4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac00708c0 0x7faac010fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:55.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.548+0000 7faac4e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faac0071f10 0x7faac01100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:55.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.548+0000 7faac4e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac01106e0 con 0x7faac00708c0 2024-02-28T16:31:55.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.548+0000 7faac4e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac0110f80 con 0x7faac0071f10 2024-02-28T16:31:55.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.549+0000 7faabde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac00708c0 0x7faac010fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:55.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.549+0000 7faabde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac00708c0 0x7faac010fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58216/0 (socket says 172.21.15.1:58216) 2024-02-28T16:31:55.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.549+0000 7faabde00700 1 -- 172.21.15.1:0/3573381515 learned_addr learned my addr 172.21.15.1:0/3573381515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:31:55.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.549+0000 7faabde00700 1 -- 172.21.15.1:0/3573381515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faac0071f10 msgr2=0x7faac01100e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:55.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.549+0000 7faabde00700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faac0071f10 0x7faac01100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.549+0000 7faabde00700 1 -- 172.21.15.1:0/3573381515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab0005040 con 0x7faac00708c0 2024-02-28T16:31:55.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.550+0000 7faabde00700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac00708c0 0x7faac010fba0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faab0000c00 tx=0x7faab0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:55.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.551+0000 7faab6000700 1 -- 172.21.15.1:0/3573381515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faab000e040 con 0x7faac00708c0 2024-02-28T16:31:55.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.551+0000 7faab6000700 1 -- 172.21.15.1:0/3573381515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faab0004290 con 0x7faac00708c0 2024-02-28T16:31:55.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.551+0000 7faac4e00700 1 -- 172.21.15.1:0/3573381515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac0111200 con 0x7faac00708c0 2024-02-28T16:31:55.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.551+0000 7faab6000700 1 -- 172.21.15.1:0/3573381515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faab001c630 con 0x7faac00708c0 2024-02-28T16:31:55.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.551+0000 7faac4e00700 1 -- 172.21.15.1:0/3573381515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac0111740 con 0x7faac00708c0 2024-02-28T16:31:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.553+0000 7faaab400700 1 -- 172.21.15.1:0/3573381515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa980052b0 con 0x7faac00708c0 2024-02-28T16:31:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.554+0000 7faab6000700 1 -- 172.21.15.1:0/3573381515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 45926+0+0 (secure 0 0 0) 0x7faab000bc60 con 0x7faac00708c0 2024-02-28T16:31:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.554+0000 7faab6000700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7faaa4038a60 0x7faaa403af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:31:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.554+0000 7faab6000700 1 -- 172.21.15.1:0/3573381515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7faab004da60 con 0x7faac00708c0 2024-02-28T16:31:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.555+0000 7faabd400700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7faaa4038a60 0x7faaa403af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:31:55.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.557+0000 7faabd400700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7faaa4038a60 0x7faaa403af10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faaac004d00 tx=0x7faaac009ce0 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:31:55.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.566+0000 7faab6000700 1 -- 172.21.15.1:0/3573381515 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7faab0011550 con 0x7faac00708c0 2024-02-28T16:31:55.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:55 smithi066 ceph-mon[36236]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:55.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:55 smithi066 ceph-mon[36236]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:55.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:55 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2196092945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:31:55.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:55 smithi001 ceph-mon[28991]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:55.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:55 smithi001 ceph-mon[28991]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:55.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:55 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2196092945' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:31:55.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.900+0000 7faaab400700 1 -- 172.21.15.1:0/3573381515 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faa98000bc0 con 0x7faaa4038a60 2024-02-28T16:31:55.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.902+0000 7faab6000700 1 -- 172.21.15.1:0/3573381515 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7faa98000bc0 con 0x7faaa4038a60 2024-02-28T16:31:55.890 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr" 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "2/2 daemons upgraded", 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "", 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:31:55.891 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:31:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 -- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7faaa4038a60 msgr2=0x7faaa403af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7faaa4038a60 0x7faaa403af10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faaac004d00 tx=0x7faaac009ce0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 -- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac00708c0 msgr2=0x7faac010fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:31:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac00708c0 0x7faac010fba0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faab0000c00 tx=0x7faab0009e30 comp rx=0 tx=0).stop 2024-02-28T16:31:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 -- 172.21.15.1:0/3573381515 shutdown_connections 2024-02-28T16:31:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7faaa4038a60 0x7faaa403af10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faac00708c0 0x7faac010fba0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 --2- 172.21.15.1:0/3573381515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faac0071f10 0x7faac01100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:31:55.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.905+0000 7faaab400700 1 -- 172.21.15.1:0/3573381515 >> 172.21.15.1:0/3573381515 conn(0x7faac00f8620 msgr2=0x7faac0102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:31:55.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.906+0000 7faaab400700 1 -- 172.21.15.1:0/3573381515 shutdown_connections 2024-02-28T16:31:55.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:31:55.906+0000 7faaab400700 1 -- 172.21.15.1:0/3573381515 wait complete. 2024-02-28T16:31:56.729 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:56 smithi066 ceph-mon[36236]: pgmap v8: 65 pgs: 65 active+clean; 1.6 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 38 KiB/s rd, 615 KiB/s wr, 106 op/s 2024-02-28T16:31:56.729 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:56 smithi066 ceph-mon[36236]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:56 smithi001 ceph-mon[28991]: pgmap v8: 65 pgs: 65 active+clean; 1.6 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 38 KiB/s rd, 615 KiB/s wr, 106 op/s 2024-02-28T16:31:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:56 smithi001 ceph-mon[28991]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:57 smithi066 ceph-mon[36236]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:57 smithi066 ceph-mon[36236]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:31:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:57 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/3307414075' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:31:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:57 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/3307414075' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:31:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:57 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/3307414075' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:31:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:57 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/3307414075' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:31:57.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:57 smithi001 ceph-mon[28991]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:31:57.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:57 smithi001 ceph-mon[28991]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:31:57.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:57 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/3307414075' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:31:57.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:57 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/3307414075' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:31:57.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:57 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/3307414075' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:31:57.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:57 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/3307414075' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:31:58.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:58 smithi066 ceph-mon[36236]: pgmap v9: 65 pgs: 65 active+clean; 1.2 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 40 KiB/s rd, 1.3 MiB/s wr, 127 op/s 2024-02-28T16:31:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:58 smithi066 ceph-mon[36236]: mgrmap e31: smithi001.rmthtx(active, since 12s), standbys: smithi066.ztraxe 2024-02-28T16:31:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:58 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi066.ztraxe", "id": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:31:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:58 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:31:58 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:58 smithi001 ceph-mon[28991]: pgmap v9: 65 pgs: 65 active+clean; 1.2 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 40 KiB/s rd, 1.3 MiB/s wr, 127 op/s 2024-02-28T16:31:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:58 smithi001 ceph-mon[28991]: mgrmap e31: smithi001.rmthtx(active, since 12s), standbys: smithi066.ztraxe 2024-02-28T16:31:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:58 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi066.ztraxe", "id": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:31:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:58 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:31:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:31:58 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:00 smithi066 ceph-mon[36236]: pgmap v10: 65 pgs: 65 active+clean; 1.2 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 36 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-02-28T16:32:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:00 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:00 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:00 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:32:00.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:00 smithi001 ceph-mon[28991]: pgmap v10: 65 pgs: 65 active+clean; 1.2 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 36 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-02-28T16:32:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:00 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:00 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:00 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:32:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:02 smithi066 ceph-mon[36236]: pgmap v11: 65 pgs: 65 active+clean; 1.2 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 36 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-02-28T16:32:02.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:02 smithi001 ceph-mon[28991]: pgmap v11: 65 pgs: 65 active+clean; 1.2 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 36 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-02-28T16:32:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:04 smithi066 ceph-mon[36236]: pgmap v12: 65 pgs: 65 active+clean; 846 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 48 KiB/s rd, 1.7 MiB/s wr, 154 op/s 2024-02-28T16:32:04.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:04 smithi001 ceph-mon[28991]: pgmap v12: 65 pgs: 65 active+clean; 846 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 48 KiB/s rd, 1.7 MiB/s wr, 154 op/s 2024-02-28T16:32:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:06 smithi066 ceph-mon[36236]: pgmap v13: 65 pgs: 65 active+clean; 846 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 19 KiB/s rd, 1.3 MiB/s wr, 74 op/s 2024-02-28T16:32:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:06 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:06 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:06 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:32:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:06 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:32:06.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:06 smithi001 ceph-mon[28991]: pgmap v13: 65 pgs: 65 active+clean; 846 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 19 KiB/s rd, 1.3 MiB/s wr, 74 op/s 2024-02-28T16:32:06.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:06 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:06.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:06 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:06.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:06 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:32:06.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:06 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:32:07.018 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-28T16:32:07.019 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-02-28T16:32:07.453 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:07 smithi066 ceph-mon[36236]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:32:07.453 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:07 smithi066 ceph-mon[36236]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:32:07.453 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:07 smithi066 ceph-mon[36236]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:32:07.453 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:07 smithi066 ceph-mon[36236]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:32:07.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:07 smithi001 ceph-mon[28991]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:32:07.796 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:07 smithi001 ceph-mon[28991]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:32:07.796 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:07 smithi001 ceph-mon[28991]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:32:07.796 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:07 smithi001 ceph-mon[28991]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:32:08.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:08 smithi066 ceph-mon[36236]: pgmap v14: 65 pgs: 65 active+clean; 570 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 30 KiB/s rd, 1.7 MiB/s wr, 110 op/s 2024-02-28T16:32:08.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:08 smithi066 ceph-mon[36236]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:32:08.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:08 smithi066 ceph-mon[36236]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:32:08.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:08 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:08.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:08 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:08.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:08 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:08.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:08 smithi001 ceph-mon[28991]: pgmap v14: 65 pgs: 65 active+clean; 570 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 30 KiB/s rd, 1.7 MiB/s wr, 110 op/s 2024-02-28T16:32:08.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:08 smithi001 ceph-mon[28991]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:32:08.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:08 smithi001 ceph-mon[28991]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:32:08.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:08 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:08.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:08 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:08.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:08 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:09.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:09 smithi066 ceph-mon[36236]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:32:09.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:09 smithi066 ceph-mon[36236]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:32:09.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:09 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:09.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:09 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:09.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:09 smithi066 ceph-mon[36236]: pgmap v15: 65 pgs: 65 active+clean; 570 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 22 KiB/s rd, 1005 KiB/s wr, 73 op/s 2024-02-28T16:32:09.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:09 smithi066 ceph-mon[36236]: Reconfiguring prometheus.smithi001 (dependencies changed)... 2024-02-28T16:32:09.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:09 smithi066 ceph-mon[36236]: Reconfiguring daemon prometheus.smithi001 on smithi001 2024-02-28T16:32:09.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:09 smithi001 ceph-mon[28991]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:32:09.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:09 smithi001 ceph-mon[28991]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:32:09.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:09 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:09.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:09 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:09.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:09 smithi001 ceph-mon[28991]: pgmap v15: 65 pgs: 65 active+clean; 570 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 22 KiB/s rd, 1005 KiB/s wr, 73 op/s 2024-02-28T16:32:09.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:09 smithi001 ceph-mon[28991]: Reconfiguring prometheus.smithi001 (dependencies changed)... 2024-02-28T16:32:09.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:09 smithi001 ceph-mon[28991]: Reconfiguring daemon prometheus.smithi001 on smithi001 2024-02-28T16:32:11.793 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:11 smithi001 ceph-mon[28991]: pgmap v16: 65 pgs: 65 active+clean; 570 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 22 KiB/s rd, 1005 KiB/s wr, 73 op/s 2024-02-28T16:32:12.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:11 smithi066 ceph-mon[36236]: pgmap v16: 65 pgs: 65 active+clean; 570 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 22 KiB/s rd, 1005 KiB/s wr, 73 op/s 2024-02-28T16:32:14.096 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:13 smithi001 ceph-mon[28991]: pgmap v17: 65 pgs: 65 active+clean; 308 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 1.6 MiB/s wr, 128 op/s 2024-02-28T16:32:14.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:13 smithi066 ceph-mon[36236]: pgmap v17: 65 pgs: 65 active+clean; 308 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 1.6 MiB/s wr, 128 op/s 2024-02-28T16:32:15.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:32:15.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:32:16.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:16 smithi001 ceph-mon[28991]: pgmap v18: 65 pgs: 65 active+clean; 308 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 1.2 MiB/s wr, 90 op/s 2024-02-28T16:32:16.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:16 smithi066 ceph-mon[36236]: pgmap v18: 65 pgs: 65 active+clean; 308 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 1.2 MiB/s wr, 90 op/s 2024-02-28T16:32:18.421 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:18 smithi066 ceph-mon[36236]: pgmap v19: 65 pgs: 65 active+clean; 314 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-02-28T16:32:18.421 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:18.421 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:18.421 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:32:18.421 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:18 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:32:18.421 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:32:18.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:18 smithi001 ceph-mon[28991]: pgmap v19: 65 pgs: 65 active+clean; 314 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 1.8 MiB/s wr, 129 op/s 2024-02-28T16:32:18.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:18.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:18.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:32:18.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:18 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:32:18.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:32:20.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:20 smithi066 ceph-mon[36236]: pgmap v20: 65 pgs: 65 active+clean; 314 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 25 KiB/s rd, 1.3 MiB/s wr, 94 op/s 2024-02-28T16:32:20.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:20 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:32:20.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:20 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:32:20.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:20 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:32:20.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:20 smithi001 ceph-mon[28991]: pgmap v20: 65 pgs: 65 active+clean; 314 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 25 KiB/s rd, 1.3 MiB/s wr, 94 op/s 2024-02-28T16:32:20.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:20 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:32:20.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:20 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:32:20.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:20 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:32:21.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:21 smithi066 ceph-mon[36236]: Upgrade: Updating mgr.smithi066.ztraxe 2024-02-28T16:32:21.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:21 smithi066 ceph-mon[36236]: Deploying daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:32:21.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:21 smithi001 ceph-mon[28991]: Upgrade: Updating mgr.smithi066.ztraxe 2024-02-28T16:32:21.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:21 smithi001 ceph-mon[28991]: Deploying daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:32:22.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:22 smithi066 ceph-mon[36236]: pgmap v21: 65 pgs: 65 active+clean; 314 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 25 KiB/s rd, 1.3 MiB/s wr, 94 op/s 2024-02-28T16:32:22.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:22 smithi001 ceph-mon[28991]: pgmap v21: 65 pgs: 65 active+clean; 314 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 25 KiB/s rd, 1.3 MiB/s wr, 94 op/s 2024-02-28T16:32:23.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:23 smithi001 ceph-mon[28991]: pgmap v22: 65 pgs: 65 active+clean; 312 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 2.0 MiB/s wr, 143 op/s 2024-02-28T16:32:23.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:23 smithi066 ceph-mon[36236]: pgmap v22: 65 pgs: 65 active+clean; 312 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 2.0 MiB/s wr, 143 op/s 2024-02-28T16:32:26.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.150+0000 7f7740c00700 1 -- 172.21.15.1:0/171622475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f773c06f430 msgr2=0x7f773c108fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:26.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.150+0000 7f7740c00700 1 --2- 172.21.15.1:0/171622475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f773c06f430 0x7f773c108fb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7728004a50 tx=0x7f772800af40 comp rx=0 tx=0).stop 2024-02-28T16:32:26.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.152+0000 7f7740c00700 1 -- 172.21.15.1:0/171622475 shutdown_connections 2024-02-28T16:32:26.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.152+0000 7f7740c00700 1 --2- 172.21.15.1:0/171622475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f773c06f430 0x7f773c108fb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.152+0000 7f7740c00700 1 --2- 172.21.15.1:0/171622475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773c06ea50 0x7f773c06ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.152+0000 7f7740c00700 1 -- 172.21.15.1:0/171622475 >> 172.21.15.1:0/171622475 conn(0x7f773c069e70 msgr2=0x7f773c06c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:26.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.153+0000 7f7740c00700 1 -- 172.21.15.1:0/171622475 shutdown_connections 2024-02-28T16:32:26.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.153+0000 7f7740c00700 1 -- 172.21.15.1:0/171622475 wait complete. 2024-02-28T16:32:26.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.154+0000 7f7740c00700 1 Processor -- start 2024-02-28T16:32:26.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.154+0000 7f7740c00700 1 -- start start 2024-02-28T16:32:26.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.155+0000 7f7740c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773c06ea50 0x7f773c1183a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:26.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.155+0000 7f7740c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f773c1188e0 0x7f773c119780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.155+0000 7f7740c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c118da0 con 0x7f773c06ea50 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.155+0000 7f7740c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c118ee0 con 0x7f773c1188e0 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.155+0000 7f7739e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773c06ea50 0x7f773c1183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.155+0000 7f7739e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773c06ea50 0x7f773c1183a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52792/0 (socket says 172.21.15.1:52792) 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.155+0000 7f7739e00700 1 -- 172.21.15.1:0/1637840075 learned_addr learned my addr 172.21.15.1:0/1637840075 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.155+0000 7f7739400700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f773c1188e0 0x7f773c119780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.156+0000 7f7739e00700 1 -- 172.21.15.1:0/1637840075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f773c1188e0 msgr2=0x7f773c119780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.156+0000 7f7739e00700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f773c1188e0 0x7f773c119780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.156+0000 7f7739e00700 1 -- 172.21.15.1:0/1637840075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7728005040 con 0x7f773c06ea50 2024-02-28T16:32:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.157+0000 7f7739e00700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773c06ea50 0x7f773c1183a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c830 tx=0x7f772c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.158+0000 7f7732000700 1 -- 172.21.15.1:0/1637840075 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f772c0106f0 con 0x7f773c06ea50 2024-02-28T16:32:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.158+0000 7f7732000700 1 -- 172.21.15.1:0/1637840075 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f772c010d20 con 0x7f773c06ea50 2024-02-28T16:32:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.158+0000 7f7732000700 1 -- 172.21.15.1:0/1637840075 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f772c0113a0 con 0x7f773c06ea50 2024-02-28T16:32:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.158+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c119de0 con 0x7f773c06ea50 2024-02-28T16:32:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.158+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c11a2f0 con 0x7f773c06ea50 2024-02-28T16:32:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.161+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c112490 con 0x7f773c06ea50 2024-02-28T16:32:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.163+0000 7f7732000700 1 -- 172.21.15.1:0/1637840075 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 31) v1 ==== 91299+0+0 (secure 0 0 0) 0x7f772c010850 con 0x7f773c06ea50 2024-02-28T16:32:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.166+0000 7f7732000700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f772006d310 0x7f772006f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.166+0000 7f7732000700 1 -- 172.21.15.1:0/1637840075 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f772c08c440 con 0x7f773c06ea50 2024-02-28T16:32:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.167+0000 7f7739400700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f772006d310 0x7f772006f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.169+0000 7f7739400700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f772006d310 0x7f772006f7c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f772800c3b0 tx=0x7f772800af40 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.178+0000 7f7732000700 1 -- 172.21.15.1:0/1637840075 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f772c05a080 con 0x7f773c06ea50 2024-02-28T16:32:26.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.557+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f773c06f4a0 con 0x7f772006d310 2024-02-28T16:32:26.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.559+0000 7f7732000700 1 -- 172.21.15.1:0/1637840075 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f773c06f4a0 con 0x7f772006d310 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.562+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f772006d310 msgr2=0x7f772006f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.562+0000 7f7740c00700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f772006d310 0x7f772006f7c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f772800c3b0 tx=0x7f772800af40 comp rx=0 tx=0).stop 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.562+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773c06ea50 msgr2=0x7f773c1183a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.562+0000 7f7740c00700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773c06ea50 0x7f773c1183a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c830 tx=0x7f772c004470 comp rx=0 tx=0).stop 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.563+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 shutdown_connections 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.563+0000 7f7740c00700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f772006d310 0x7f772006f7c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.563+0000 7f7740c00700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773c06ea50 0x7f773c1183a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.563+0000 7f7740c00700 1 --2- 172.21.15.1:0/1637840075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f773c1188e0 0x7f773c119780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.563+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 >> 172.21.15.1:0/1637840075 conn(0x7f773c069e70 msgr2=0x7f773c107830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.563+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 shutdown_connections 2024-02-28T16:32:26.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.563+0000 7f7740c00700 1 -- 172.21.15.1:0/1637840075 wait complete. 2024-02-28T16:32:26.568 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:32:26.699 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:26 smithi066 ceph-mon[36236]: pgmap v23: 65 pgs: 65 active+clean; 312 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 89 op/s 2024-02-28T16:32:26.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.722+0000 7fd9f0200700 1 -- 172.21.15.1:0/4197562653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 msgr2=0x7fd9e8070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:26.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.722+0000 7fd9f0200700 1 --2- 172.21.15.1:0/4197562653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 0x7fd9e8070380 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc004940 tx=0x7fd9dc009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:26.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:26 smithi001 ceph-mon[28991]: pgmap v23: 65 pgs: 65 active+clean; 312 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 89 op/s 2024-02-28T16:32:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.722+0000 7fd9f0200700 1 -- 172.21.15.1:0/4197562653 shutdown_connections 2024-02-28T16:32:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.722+0000 7fd9f0200700 1 --2- 172.21.15.1:0/4197562653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9e80708c0 0x7fd9e8070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.722+0000 7fd9f0200700 1 --2- 172.21.15.1:0/4197562653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 0x7fd9e8070380 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.722+0000 7fd9f0200700 1 -- 172.21.15.1:0/4197562653 >> 172.21.15.1:0/4197562653 conn(0x7fd9e80f8620 msgr2=0x7fd9e80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.722+0000 7fd9f0200700 1 -- 172.21.15.1:0/4197562653 shutdown_connections 2024-02-28T16:32:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.722+0000 7fd9f0200700 1 -- 172.21.15.1:0/4197562653 wait complete. 2024-02-28T16:32:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.723+0000 7fd9f0200700 1 Processor -- start 2024-02-28T16:32:26.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.723+0000 7fd9f0200700 1 -- start start 2024-02-28T16:32:26.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.724+0000 7fd9f0200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9e80708c0 0x7fd9e8113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:26.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.724+0000 7fd9f0200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 0x7fd9e8114400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:26.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.724+0000 7fd9f0200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e8114a00 con 0x7fd9e8071f10 2024-02-28T16:32:26.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.724+0000 7fd9f0200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e81152a0 con 0x7fd9e80708c0 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.724+0000 7fd9ece00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 0x7fd9e8114400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.724+0000 7fd9ece00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 0x7fd9e8114400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52806/0 (socket says 172.21.15.1:52806) 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.724+0000 7fd9ece00700 1 -- 172.21.15.1:0/2946856396 learned_addr learned my addr 172.21.15.1:0/2946856396 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9ed800700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9e80708c0 0x7fd9e8113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9ece00700 1 -- 172.21.15.1:0/2946856396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9e80708c0 msgr2=0x7fd9e8113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9ece00700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9e80708c0 0x7fd9e8113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9ece00700 1 -- 172.21.15.1:0/2946856396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc005040 con 0x7fd9e8071f10 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9ed800700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9e80708c0 0x7fd9e8113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:32:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9ece00700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 0x7fd9e8114400 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4002a80 tx=0x7fd9d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:26.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9e5600700 1 -- 172.21.15.1:0/2946856396 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9d4007cc0 con 0x7fd9e8071f10 2024-02-28T16:32:26.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9e5600700 1 -- 172.21.15.1:0/2946856396 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9d400f040 con 0x7fd9e8071f10 2024-02-28T16:32:26.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.725+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e8115580 con 0x7fd9e8071f10 2024-02-28T16:32:26.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.726+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e8115ac0 con 0x7fd9e8071f10 2024-02-28T16:32:26.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.726+0000 7fd9e5600700 1 -- 172.21.15.1:0/2946856396 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9d40096f0 con 0x7fd9e8071f10 2024-02-28T16:32:26.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.727+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c80052e0 con 0x7fd9e8071f10 2024-02-28T16:32:26.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.728+0000 7fd9e5600700 1 -- 172.21.15.1:0/2946856396 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 31) v1 ==== 91299+0+0 (secure 0 0 0) 0x7fd9d400d690 con 0x7fd9e8071f10 2024-02-28T16:32:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.729+0000 7fd9e5600700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd9d006d200 0x7fd9d006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.730+0000 7fd9e5600700 1 -- 172.21.15.1:0/2946856396 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fd9d408be40 con 0x7fd9e8071f10 2024-02-28T16:32:26.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.730+0000 7fd9ed800700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd9d006d200 0x7fd9d006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:26.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.731+0000 7fd9ed800700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd9d006d200 0x7fd9d006f6b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00b840 tx=0x7fd9dc009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:26.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:26.736+0000 7fd9e5600700 1 -- 172.21.15.1:0/2946856396 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd9d4059830 con 0x7fd9e8071f10 2024-02-28T16:32:27.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.123+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9c8000bf0 con 0x7fd9d006d200 2024-02-28T16:32:27.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.126+0000 7fd9e5600700 1 -- 172.21.15.1:0/2946856396 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fd9c8000bf0 con 0x7fd9d006d200 2024-02-28T16:32:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.131+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd9d006d200 msgr2=0x7fd9d006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.131+0000 7fd9f0200700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd9d006d200 0x7fd9d006f6b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00b840 tx=0x7fd9dc009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.131+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 msgr2=0x7fd9e8114400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.131+0000 7fd9f0200700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 0x7fd9e8114400 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4002a80 tx=0x7fd9d40046b0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.132+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 shutdown_connections 2024-02-28T16:32:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.132+0000 7fd9f0200700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9e80708c0 0x7fd9e8113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.132+0000 7fd9f0200700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd9d006d200 0x7fd9d006f6b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.132+0000 7fd9f0200700 1 --2- 172.21.15.1:0/2946856396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e8071f10 0x7fd9e8114400 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.132+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 >> 172.21.15.1:0/2946856396 conn(0x7fd9e80f8620 msgr2=0x7fd9e80fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.132+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 shutdown_connections 2024-02-28T16:32:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.132+0000 7fd9f0200700 1 -- 172.21.15.1:0/2946856396 wait complete. 2024-02-28T16:32:27.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.333+0000 7ff793c00700 1 -- 172.21.15.1:0/1457907577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c0fda80 msgr2=0x7ff78c0ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:27.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.333+0000 7ff793c00700 1 --2- 172.21.15.1:0/1457907577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c0fda80 0x7ff78c0ffe60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff780004940 tx=0x7ff780009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:27.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.333+0000 7ff793c00700 1 -- 172.21.15.1:0/1457907577 shutdown_connections 2024-02-28T16:32:27.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.333+0000 7ff793c00700 1 --2- 172.21.15.1:0/1457907577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff78c1003a0 0x7ff78c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.333+0000 7ff793c00700 1 --2- 172.21.15.1:0/1457907577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c0fda80 0x7ff78c0ffe60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.333+0000 7ff793c00700 1 -- 172.21.15.1:0/1457907577 >> 172.21.15.1:0/1457907577 conn(0x7ff78c0f74e0 msgr2=0x7ff78c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:27.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.334+0000 7ff793c00700 1 -- 172.21.15.1:0/1457907577 shutdown_connections 2024-02-28T16:32:27.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.334+0000 7ff793c00700 1 -- 172.21.15.1:0/1457907577 wait complete. 2024-02-28T16:32:27.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.335+0000 7ff793c00700 1 Processor -- start 2024-02-28T16:32:27.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.336+0000 7ff793c00700 1 -- start start 2024-02-28T16:32:27.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.336+0000 7ff793c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff78c0fda80 0x7ff78c10fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:27.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.336+0000 7ff793c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c1003a0 0x7ff78c110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:27.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.336+0000 7ff793c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c110670 con 0x7ff78c1003a0 2024-02-28T16:32:27.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.336+0000 7ff793c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c110f10 con 0x7ff78c0fda80 2024-02-28T16:32:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.337+0000 7ff791200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff78c0fda80 0x7ff78c10fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.337+0000 7ff78be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c1003a0 0x7ff78c110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.337+0000 7ff791200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff78c0fda80 0x7ff78c10fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:48582/0 (socket says 172.21.15.1:48582) 2024-02-28T16:32:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.337+0000 7ff791200700 1 -- 172.21.15.1:0/2596416862 learned_addr learned my addr 172.21.15.1:0/2596416862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:32:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.337+0000 7ff78be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c1003a0 0x7ff78c110070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52828/0 (socket says 172.21.15.1:52828) 2024-02-28T16:32:27.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.337+0000 7ff78be00700 1 -- 172.21.15.1:0/2596416862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff78c0fda80 msgr2=0x7ff78c10fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:27.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.337+0000 7ff78be00700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff78c0fda80 0x7ff78c10fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.337+0000 7ff78be00700 1 -- 172.21.15.1:0/2596416862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff780005040 con 0x7ff78c1003a0 2024-02-28T16:32:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.338+0000 7ff78be00700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c1003a0 0x7ff78c110070 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff778002820 tx=0x7ff778004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.338+0000 7ff788c00700 1 -- 172.21.15.1:0/2596416862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff778007cc0 con 0x7ff78c1003a0 2024-02-28T16:32:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.338+0000 7ff788c00700 1 -- 172.21.15.1:0/2596416862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff77800f040 con 0x7ff78c1003a0 2024-02-28T16:32:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.338+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c1111f0 con 0x7ff78c1003a0 2024-02-28T16:32:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.339+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c111730 con 0x7ff78c1003a0 2024-02-28T16:32:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.339+0000 7ff788c00700 1 -- 172.21.15.1:0/2596416862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff778013920 con 0x7ff78c1003a0 2024-02-28T16:32:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.340+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c0052e0 con 0x7ff78c1003a0 2024-02-28T16:32:27.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.342+0000 7ff788c00700 1 -- 172.21.15.1:0/2596416862 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 31) v1 ==== 91299+0+0 (secure 0 0 0) 0x7ff77801c400 con 0x7ff78c1003a0 2024-02-28T16:32:27.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.344+0000 7ff788c00700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff77406d2c0 0x7ff77406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:27.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.344+0000 7ff788c00700 1 -- 172.21.15.1:0/2596416862 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7ff77805e0b0 con 0x7ff78c1003a0 2024-02-28T16:32:27.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.345+0000 7ff791200700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff77406d2c0 0x7ff77406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:27.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.346+0000 7ff791200700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff77406d2c0 0x7ff77406f770 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff780009fd0 tx=0x7ff780009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:27.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.350+0000 7ff788c00700 1 -- 172.21.15.1:0/2596416862 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7ff778059de0 con 0x7ff78c1003a0 2024-02-28T16:32:27.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.615+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff76c000bf0 con 0x7ff77406d2c0 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.628+0000 7ff788c00700 1 -- 172.21.15.1:0/2596416862 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7ff76c000bf0 con 0x7ff77406d2c0 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (2m) 29s ago 12m 14.7M - 0.25.0 c8568f914cd2 04ac88629633 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (13m) 29s ago 13m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (11m) 37s ago 11m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (106s) 29s ago 12m 74.3M - 9.4.7 954c08fa6188 1010f22bd792 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (5m) 29s ago 5m 143M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (5m) 29s ago 5m 12.5M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:32:27.617 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (5m) 37s ago 5m 12.6M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (5m) 37s ago 5m 181M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (61s) 29s ago 14m 475M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 starting - - - - 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (14m) 29s ago 14m 50.6M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (11m) 37s ago 11m 42.2M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (2m) 29s ago 13m 7180k - 1.5.0 0da6a335fe13 1a0f5b104896 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (84s) 37s ago 11m 7554k - 1.5.0 0da6a335fe13 5697e47a4eda 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (10m) 29s ago 10m 179M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (9m) 29s ago 9m 182M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (9m) 29s ago 9m 141M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (8m) 37s ago 8m 206M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (8m) 37s ago 8m 181M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (7m) 37s ago 7m 196M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:32:27.618 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 starting - - - - 2024-02-28T16:32:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff77406d2c0 msgr2=0x7ff77406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff77406d2c0 0x7ff77406f770 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff780009fd0 tx=0x7ff780009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c1003a0 msgr2=0x7ff78c110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c1003a0 0x7ff78c110070 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff778002820 tx=0x7ff778004570 comp rx=0 tx=0).stop 2024-02-28T16:32:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 shutdown_connections 2024-02-28T16:32:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff78c0fda80 0x7ff78c10fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff77406d2c0 0x7ff77406f770 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 --2- 172.21.15.1:0/2596416862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff78c1003a0 0x7ff78c110070 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.631+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 >> 172.21.15.1:0/2596416862 conn(0x7ff78c0f74e0 msgr2=0x7ff78c0fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:27.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.632+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 shutdown_connections 2024-02-28T16:32:27.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.632+0000 7ff793c00700 1 -- 172.21.15.1:0/2596416862 wait complete. 2024-02-28T16:32:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.815+0000 7f7b76600700 1 -- 172.21.15.1:0/2270305532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b70102450 msgr2=0x7f7b70104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.815+0000 7f7b76600700 1 --2- 172.21.15.1:0/2270305532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b70102450 0x7f7b70104830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c004940 tx=0x7f7b5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.815+0000 7f7b76600700 1 -- 172.21.15.1:0/2270305532 shutdown_connections 2024-02-28T16:32:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.815+0000 7f7b76600700 1 --2- 172.21.15.1:0/2270305532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b70102450 0x7f7b70104830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.815+0000 7f7b76600700 1 --2- 172.21.15.1:0/2270305532 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b700663c0 0x7f7b70101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.815+0000 7f7b76600700 1 -- 172.21.15.1:0/2270305532 >> 172.21.15.1:0/2270305532 conn(0x7f7b700f74e0 msgr2=0x7f7b700f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.816+0000 7f7b76600700 1 -- 172.21.15.1:0/2270305532 shutdown_connections 2024-02-28T16:32:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.816+0000 7f7b76600700 1 -- 172.21.15.1:0/2270305532 wait complete. 2024-02-28T16:32:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.817+0000 7f7b76600700 1 Processor -- start 2024-02-28T16:32:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.817+0000 7f7b76600700 1 -- start start 2024-02-28T16:32:27.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.819+0000 7f7b76600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b700663c0 0x7f7b7010fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:27.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.819+0000 7f7b76600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b70102450 0x7f7b70110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:27.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.819+0000 7f7b76600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70110650 con 0x7f7b700663c0 2024-02-28T16:32:27.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.819+0000 7f7b76600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70110ef0 con 0x7f7b70102450 2024-02-28T16:32:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.820+0000 7f7b6ea00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b70102450 0x7f7b70110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.820+0000 7f7b6ea00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b70102450 0x7f7b70110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:48604/0 (socket says 172.21.15.1:48604) 2024-02-28T16:32:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.820+0000 7f7b6ea00700 1 -- 172.21.15.1:0/2509553100 learned_addr learned my addr 172.21.15.1:0/2509553100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:32:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.820+0000 7f7b6f400700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b700663c0 0x7f7b7010fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.820+0000 7f7b6ea00700 1 -- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b700663c0 msgr2=0x7f7b7010fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.821+0000 7f7b6ea00700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b700663c0 0x7f7b7010fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.821+0000 7f7b6ea00700 1 -- 172.21.15.1:0/2509553100 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c005040 con 0x7f7b70102450 2024-02-28T16:32:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.821+0000 7f7b6f400700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b700663c0 0x7f7b7010fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-28T16:32:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.821+0000 7f7b6ea00700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b70102450 0x7f7b70110050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009fd0 tx=0x7f7b5c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.822+0000 7f7b67400700 1 -- 172.21.15.1:0/2509553100 <== mon.1 v2:172.21.15.66:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5c00e070 con 0x7f7b70102450 2024-02-28T16:32:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.822+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b70111170 con 0x7f7b70102450 2024-02-28T16:32:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.822+0000 7f7b67400700 1 -- 172.21.15.1:0/2509553100 <== mon.1 v2:172.21.15.66:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b5c013de0 con 0x7f7b70102450 2024-02-28T16:32:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.822+0000 7f7b67400700 1 -- 172.21.15.1:0/2509553100 <== mon.1 v2:172.21.15.66:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5c00b410 con 0x7f7b70102450 2024-02-28T16:32:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.822+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b70111650 con 0x7f7b70102450 2024-02-28T16:32:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.823+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7004bed0 con 0x7f7b70102450 2024-02-28T16:32:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.824+0000 7f7b67400700 1 -- 172.21.15.1:0/2509553100 <== mon.1 v2:172.21.15.66:3300/0 4 ==== mgrmap(e 31) v1 ==== 91299+0+0 (secure 0 0 0) 0x7f7b5c01c400 con 0x7f7b70102450 2024-02-28T16:32:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.826+0000 7f7b67400700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f7b5406d200 0x7f7b5406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.826+0000 7f7b67400700 1 -- 172.21.15.1:0/2509553100 <== mon.1 v2:172.21.15.66:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f7b5c08de80 con 0x7f7b70102450 2024-02-28T16:32:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.826+0000 7f7b6f400700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f7b5406d200 0x7f7b5406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.827+0000 7f7b6f400700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f7b5406d200 0x7f7b5406f6b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009b80 tx=0x7f7b6000b040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:27.836+0000 7f7b67400700 1 -- 172.21.15.1:0/2509553100 <== mon.1 v2:172.21.15.66:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f7b5c05b8f0 con 0x7f7b70102450 2024-02-28T16:32:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:27 smithi066 ceph-mon[36236]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:32:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:27 smithi066 ceph-mon[36236]: pgmap v24: 65 pgs: 65 active+clean; 315 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-02-28T16:32:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:27 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:27 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:27 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:32:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:27 smithi066 ceph-mon[36236]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:32:28.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:27 smithi001 ceph-mon[28991]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:32:28.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:27 smithi001 ceph-mon[28991]: pgmap v24: 65 pgs: 65 active+clean; 315 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-02-28T16:32:28.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:27 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:28.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:27 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:28.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:27 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:32:28.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:27 smithi001 ceph-mon[28991]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:32:28.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.299+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b70107c20 con 0x7f7b70102450 2024-02-28T16:32:28.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.300+0000 7f7b67400700 1 -- 172.21.15.1:0/2509553100 <== mon.1 v2:172.21.15.66:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f7b5c05b480 con 0x7f7b70102450 2024-02-28T16:32:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:32:28.289 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:32:28.289 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:32:28.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 12, 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:32:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:32:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.303+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f7b5406d200 msgr2=0x7f7b5406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.303+0000 7f7b76600700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f7b5406d200 0x7f7b5406f6b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009b80 tx=0x7f7b6000b040 comp rx=0 tx=0).stop 2024-02-28T16:32:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.303+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b70102450 msgr2=0x7f7b70110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.303+0000 7f7b76600700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b70102450 0x7f7b70110050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009fd0 tx=0x7f7b5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.304+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 shutdown_connections 2024-02-28T16:32:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.304+0000 7f7b76600700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f7b5406d200 0x7f7b5406f6b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.304+0000 7f7b76600700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b700663c0 0x7f7b7010fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.304+0000 7f7b76600700 1 --2- 172.21.15.1:0/2509553100 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b70102450 0x7f7b70110050 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.304+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 >> 172.21.15.1:0/2509553100 conn(0x7f7b700f74e0 msgr2=0x7f7b700f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.304+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 shutdown_connections 2024-02-28T16:32:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.304+0000 7f7b76600700 1 -- 172.21.15.1:0/2509553100 wait complete. 2024-02-28T16:32:28.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.484+0000 7fa7e1600700 1 -- 172.21.15.1:0/393037347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc071f10 msgr2=0x7fa7dc070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:28.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.484+0000 7fa7e1600700 1 --2- 172.21.15.1:0/393037347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc071f10 0x7fa7dc070380 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc004940 tx=0x7fa7cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:28.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.484+0000 7fa7e1600700 1 -- 172.21.15.1:0/393037347 shutdown_connections 2024-02-28T16:32:28.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.484+0000 7fa7e1600700 1 --2- 172.21.15.1:0/393037347 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7dc0708c0 0x7fa7dc070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.484+0000 7fa7e1600700 1 --2- 172.21.15.1:0/393037347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc071f10 0x7fa7dc070380 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.484+0000 7fa7e1600700 1 -- 172.21.15.1:0/393037347 >> 172.21.15.1:0/393037347 conn(0x7fa7dc0f8620 msgr2=0x7fa7dc0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:28.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.485+0000 7fa7e1600700 1 -- 172.21.15.1:0/393037347 shutdown_connections 2024-02-28T16:32:28.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.485+0000 7fa7e1600700 1 -- 172.21.15.1:0/393037347 wait complete. 2024-02-28T16:32:28.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.485+0000 7fa7e1600700 1 Processor -- start 2024-02-28T16:32:28.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7e1600700 1 -- start start 2024-02-28T16:32:28.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7e1600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc0708c0 0x7fa7dc113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:28.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7e1600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7dc071f10 0x7fa7dc114470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:28.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7e1600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc114a70 con 0x7fa7dc0708c0 2024-02-28T16:32:28.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7e1600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc115310 con 0x7fa7dc071f10 2024-02-28T16:32:28.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7daa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc0708c0 0x7fa7dc113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:28.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7daa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc0708c0 0x7fa7dc113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52868/0 (socket says 172.21.15.1:52868) 2024-02-28T16:32:28.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7da000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7dc071f10 0x7fa7dc114470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:28.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.486+0000 7fa7daa00700 1 -- 172.21.15.1:0/1462402317 learned_addr learned my addr 172.21.15.1:0/1462402317 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:32:28.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.487+0000 7fa7daa00700 1 -- 172.21.15.1:0/1462402317 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7dc071f10 msgr2=0x7fa7dc114470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:28.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.487+0000 7fa7daa00700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7dc071f10 0x7fa7dc114470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.487+0000 7fa7daa00700 1 -- 172.21.15.1:0/1462402317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc005040 con 0x7fa7dc0708c0 2024-02-28T16:32:28.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.487+0000 7fa7daa00700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc0708c0 0x7fa7dc113f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00b3b0 tx=0x7fa7cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:28.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.488+0000 7fa7d2a00700 1 -- 172.21.15.1:0/1462402317 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7cc00e040 con 0x7fa7dc0708c0 2024-02-28T16:32:28.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.488+0000 7fa7d2a00700 1 -- 172.21.15.1:0/1462402317 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7cc00bd90 con 0x7fa7dc0708c0 2024-02-28T16:32:28.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.488+0000 7fa7d2a00700 1 -- 172.21.15.1:0/1462402317 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7cc01c630 con 0x7fa7dc0708c0 2024-02-28T16:32:28.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.488+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7dc115590 con 0x7fa7dc0708c0 2024-02-28T16:32:28.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.488+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7dc115ad0 con 0x7fa7dc0708c0 2024-02-28T16:32:28.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.489+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7dc04bed0 con 0x7fa7dc0708c0 2024-02-28T16:32:28.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.490+0000 7fa7d2a00700 1 -- 172.21.15.1:0/1462402317 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 31) v1 ==== 91299+0+0 (secure 0 0 0) 0x7fa7cc013980 con 0x7fa7dc0708c0 2024-02-28T16:32:28.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.492+0000 7fa7d2a00700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa7c006d200 0x7fa7c006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:28.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.492+0000 7fa7d2a00700 1 -- 172.21.15.1:0/1462402317 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fa7cc08dbc0 con 0x7fa7dc0708c0 2024-02-28T16:32:28.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.493+0000 7fa7da000700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa7c006d200 0x7fa7c006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:28.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.495+0000 7fa7da000700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa7c006d200 0x7fa7c006f6b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8004f60 tx=0x7fa7c800b040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.499+0000 7fa7d2a00700 1 -- 172.21.15.1:0/1462402317 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fa7cc05b630 con 0x7fa7dc0708c0 2024-02-28T16:32:28.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.857+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7dc1000a0 con 0x7fa7c006d200 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.859+0000 7fa7d2a00700 1 -- 172.21.15.1:0/1462402317 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fa7dc1000a0 con 0x7fa7c006d200 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [], 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "", 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mgr daemons", 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:32:28.848 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa7c006d200 msgr2=0x7fa7c006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa7c006d200 0x7fa7c006f6b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8004f60 tx=0x7fa7c800b040 comp rx=0 tx=0).stop 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc0708c0 msgr2=0x7fa7dc113f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc0708c0 0x7fa7dc113f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00b3b0 tx=0x7fa7cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 shutdown_connections 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa7c006d200 0x7fa7c006f6b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7dc0708c0 0x7fa7dc113f30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 --2- 172.21.15.1:0/1462402317 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7dc071f10 0x7fa7dc114470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.862+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 >> 172.21.15.1:0/1462402317 conn(0x7fa7dc0f8620 msgr2=0x7fa7dc0fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.863+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 shutdown_connections 2024-02-28T16:32:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:28.863+0000 7fa7e1600700 1 -- 172.21.15.1:0/1462402317 wait complete. 2024-02-28T16:32:29.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:28 smithi066 ceph-mon[36236]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:32:29.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:28 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2509553100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:29.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:28 smithi001 ceph-mon[28991]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:32:29.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:28 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2509553100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:30.707 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:30 smithi001 ceph-mon[28991]: pgmap v25: 65 pgs: 65 active+clean; 315 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 88 op/s 2024-02-28T16:32:30.708 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:30 smithi001 ceph-mon[28991]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:32:30.708 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:30 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:32:30.721 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:30 smithi066 ceph-mon[36236]: pgmap v25: 65 pgs: 65 active+clean; 315 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 88 op/s 2024-02-28T16:32:30.722 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:30 smithi066 ceph-mon[36236]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:32:30.722 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:30 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:32:32.741 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:32 smithi001 ceph-mon[28991]: pgmap v26: 65 pgs: 65 active+clean; 314 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-02-28T16:32:32.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:32 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:32.742 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:32 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:32.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:32 smithi066 ceph-mon[36236]: pgmap v26: 65 pgs: 65 active+clean; 314 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-02-28T16:32:32.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:32 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:32.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:32 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:34.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:34 smithi066 ceph-mon[36236]: pgmap v27: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 50 KiB/s rd, 2.3 MiB/s wr, 134 op/s 2024-02-28T16:32:34.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:34 smithi001 ceph-mon[28991]: pgmap v27: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 50 KiB/s rd, 2.3 MiB/s wr, 134 op/s 2024-02-28T16:32:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:35.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:35.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:36 smithi066 ceph-mon[36236]: pgmap v28: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-02-28T16:32:36.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:36 smithi001 ceph-mon[28991]: pgmap v28: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-02-28T16:32:37.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:37 smithi066 ceph-mon[36236]: pgmap v29: 65 pgs: 65 active+clean; 323 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 48 KiB/s rd, 2.2 MiB/s wr, 131 op/s 2024-02-28T16:32:37.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:37 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:37.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:37 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:38.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:37 smithi001 ceph-mon[28991]: pgmap v29: 65 pgs: 65 active+clean; 323 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 48 KiB/s rd, 2.2 MiB/s wr, 131 op/s 2024-02-28T16:32:38.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:37 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:38.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:37 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:39 smithi001 ceph-mon[28991]: pgmap v30: 65 pgs: 65 active+clean; 323 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 33 KiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-02-28T16:32:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:39 smithi001 ceph-mon[28991]: Standby manager daemon smithi066.ztraxe restarted 2024-02-28T16:32:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:39 smithi001 ceph-mon[28991]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:32:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:39 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/247787680' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:32:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:39 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/247787680' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:32:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:39 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/247787680' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:32:40.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:39 smithi001 ceph-mon[28991]: from='mgr.? 172.21.15.66:0/247787680' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:32:40.148 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:39 smithi066 ceph-mon[36236]: pgmap v30: 65 pgs: 65 active+clean; 323 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 33 KiB/s rd, 1.5 MiB/s wr, 92 op/s 2024-02-28T16:32:40.148 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:39 smithi066 ceph-mon[36236]: Standby manager daemon smithi066.ztraxe restarted 2024-02-28T16:32:40.148 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:39 smithi066 ceph-mon[36236]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:32:40.148 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:39 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/247787680' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:32:40.148 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:39 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/247787680' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:32:40.148 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:39 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/247787680' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:32:40.148 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:39 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/247787680' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: mgrmap e32: smithi001.rmthtx(active, since 55s), standbys: smithi066.ztraxe 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi001.rmthtx"}]: dispatch 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi001.rmthtx"}]': finished 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi066.ztraxe"}]: dispatch 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi066.ztraxe"}]': finished 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.250 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:40 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: mgrmap e32: smithi001.rmthtx(active, since 55s), standbys: smithi066.ztraxe 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi001.rmthtx"}]: dispatch 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi001.rmthtx"}]': finished 2024-02-28T16:32:41.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi066.ztraxe"}]: dispatch 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi066.ztraxe"}]': finished 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:41.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:32:41.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:40 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all mgr 2024-02-28T16:32:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all rgw 2024-02-28T16:32:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all rbd-mirror 2024-02-28T16:32:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-28T16:32:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all ceph-exporter 2024-02-28T16:32:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all iscsi 2024-02-28T16:32:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all nfs 2024-02-28T16:32:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all nvmeof 2024-02-28T16:32:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:41 smithi001 ceph-mon[28991]: pgmap v31: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-02-28T16:32:42.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all mgr 2024-02-28T16:32:42.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all rgw 2024-02-28T16:32:42.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all rbd-mirror 2024-02-28T16:32:42.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-28T16:32:42.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all ceph-exporter 2024-02-28T16:32:42.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all iscsi 2024-02-28T16:32:42.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all nfs 2024-02-28T16:32:42.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all nvmeof 2024-02-28T16:32:42.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:41 smithi066 ceph-mon[36236]: pgmap v31: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-02-28T16:32:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:43 smithi066 ceph-mon[36236]: pgmap v32: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 38 KiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-02-28T16:32:44.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:43 smithi001 ceph-mon[28991]: pgmap v32: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 38 KiB/s rd, 2.3 MiB/s wr, 143 op/s 2024-02-28T16:32:45.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:45 smithi066 ceph-mon[36236]: Upgrade: Updating node-exporter.smithi001 (1/2) 2024-02-28T16:32:45.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:45 smithi066 ceph-mon[36236]: Deploying daemon node-exporter.smithi001 on smithi001 2024-02-28T16:32:45.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:32:45.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:45 smithi001 ceph-mon[28991]: Upgrade: Updating node-exporter.smithi001 (1/2) 2024-02-28T16:32:45.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:45 smithi001 ceph-mon[28991]: Deploying daemon node-exporter.smithi001 on smithi001 2024-02-28T16:32:45.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:32:46.733 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:46 smithi001 ceph-mon[28991]: pgmap v33: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 28 KiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-02-28T16:32:46.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:46 smithi066 ceph-mon[36236]: pgmap v33: 65 pgs: 65 active+clean; 322 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 28 KiB/s rd, 1.3 MiB/s wr, 103 op/s 2024-02-28T16:32:47.745 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:47 smithi001 ceph-mon[28991]: pgmap v34: 65 pgs: 65 active+clean; 323 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 44 KiB/s rd, 2.0 MiB/s wr, 153 op/s 2024-02-28T16:32:47.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:47 smithi066 ceph-mon[36236]: pgmap v34: 65 pgs: 65 active+clean; 323 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 44 KiB/s rd, 2.0 MiB/s wr, 153 op/s 2024-02-28T16:32:49.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:49 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:49.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:49 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:49.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:49 smithi001 ceph-mon[28991]: pgmap v35: 65 pgs: 65 active+clean; 323 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 107 op/s 2024-02-28T16:32:49.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:49 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:49.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:49 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:49.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:49 smithi066 ceph-mon[36236]: pgmap v35: 65 pgs: 65 active+clean; 323 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 107 op/s 2024-02-28T16:32:51.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:51 smithi066 ceph-mon[36236]: pgmap v36: 65 pgs: 65 active+clean; 329 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 2.0 MiB/s wr, 146 op/s 2024-02-28T16:32:52.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:51 smithi001 ceph-mon[28991]: pgmap v36: 65 pgs: 65 active+clean; 329 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 2.0 MiB/s wr, 146 op/s 2024-02-28T16:32:53.138 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:52 smithi066 ceph-mon[36236]: Upgrade: Updating node-exporter.smithi066 (2/2) 2024-02-28T16:32:53.139 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:52 smithi066 ceph-mon[36236]: Deploying daemon node-exporter.smithi066 on smithi066 2024-02-28T16:32:53.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:52 smithi001 ceph-mon[28991]: Upgrade: Updating node-exporter.smithi066 (2/2) 2024-02-28T16:32:53.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:52 smithi001 ceph-mon[28991]: Deploying daemon node-exporter.smithi066 on smithi066 2024-02-28T16:32:53.984 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:53 smithi066 ceph-mon[36236]: pgmap v37: 65 pgs: 65 active+clean; 328 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 2.0 MiB/s wr, 143 op/s 2024-02-28T16:32:54.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:53 smithi001 ceph-mon[28991]: pgmap v37: 65 pgs: 65 active+clean; 328 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 2.0 MiB/s wr, 143 op/s 2024-02-28T16:32:56.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:56 smithi066 ceph-mon[36236]: pgmap v38: 65 pgs: 65 active+clean; 328 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 94 op/s 2024-02-28T16:32:56.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:56 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:56.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:56 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:56 smithi001 ceph-mon[28991]: pgmap v38: 65 pgs: 65 active+clean; 328 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 94 op/s 2024-02-28T16:32:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:56 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:56 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:32:57.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:57 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:32:57.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:57 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:32:58.651 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:32:58 smithi001 ceph-mon[28991]: pgmap v39: 65 pgs: 65 active+clean; 328 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.7 MiB/s wr, 145 op/s 2024-02-28T16:32:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:32:58 smithi066 ceph-mon[36236]: pgmap v39: 65 pgs: 65 active+clean; 328 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.7 MiB/s wr, 145 op/s 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.054+0000 7f1524400700 1 -- 172.21.15.1:0/1734674229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 msgr2=0x7f151c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.054+0000 7f1524400700 1 --2- 172.21.15.1:0/1734674229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 0x7f151c0fe550 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1508004940 tx=0x7f1508009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.054+0000 7f1524400700 1 -- 172.21.15.1:0/1734674229 shutdown_connections 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.054+0000 7f1524400700 1 --2- 172.21.15.1:0/1734674229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 0x7f151c0fe550 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.054+0000 7f1524400700 1 --2- 172.21.15.1:0/1734674229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f151c0fcef0 0x7f151c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.054+0000 7f1524400700 1 -- 172.21.15.1:0/1734674229 >> 172.21.15.1:0/1734674229 conn(0x7f151c0f8620 msgr2=0x7f151c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.054+0000 7f1524400700 1 -- 172.21.15.1:0/1734674229 shutdown_connections 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.054+0000 7f1524400700 1 -- 172.21.15.1:0/1734674229 wait complete. 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.055+0000 7f1524400700 1 Processor -- start 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.055+0000 7f1524400700 1 -- start start 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.055+0000 7f1524400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f151c0fcef0 0x7f151c10fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:59.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.055+0000 7f1524400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 0x7f151c1100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.055+0000 7f1524400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c1106a0 con 0x7f151c0fe0e0 2024-02-28T16:32:59.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.055+0000 7f1524400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c110f40 con 0x7f151c0fcef0 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.056+0000 7f1521000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 0x7f151c1100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.056+0000 7f1521000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 0x7f151c1100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55840/0 (socket says 172.21.15.1:55840) 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.056+0000 7f1521000700 1 -- 172.21.15.1:0/848915990 learned_addr learned my addr 172.21.15.1:0/848915990 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.056+0000 7f1521a00700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f151c0fcef0 0x7f151c10fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.056+0000 7f1521000700 1 -- 172.21.15.1:0/848915990 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f151c0fcef0 msgr2=0x7f151c10fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.056+0000 7f1521000700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f151c0fcef0 0x7f151c10fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.056+0000 7f1521000700 1 -- 172.21.15.1:0/848915990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1508005040 con 0x7f151c0fe0e0 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.057+0000 7f1521000700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 0x7f151c1100a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1508009fd0 tx=0x7f1508009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.057+0000 7f1519600700 1 -- 172.21.15.1:0/848915990 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f150800e040 con 0x7f151c0fe0e0 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.057+0000 7f1519600700 1 -- 172.21.15.1:0/848915990 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f150800bcc0 con 0x7f151c0fe0e0 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.057+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c1111c0 con 0x7f151c0fe0e0 2024-02-28T16:32:59.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.057+0000 7f1519600700 1 -- 172.21.15.1:0/848915990 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f150801c630 con 0x7f151c0fe0e0 2024-02-28T16:32:59.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.058+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c111700 con 0x7f151c0fe0e0 2024-02-28T16:32:59.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.059+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c102210 con 0x7f151c0fe0e0 2024-02-28T16:32:59.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.060+0000 7f1519600700 1 -- 172.21.15.1:0/848915990 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 32) v1 ==== 91299+0+0 (secure 0 0 0) 0x7f1508013980 con 0x7f151c0fe0e0 2024-02-28T16:32:59.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.061+0000 7f1519600700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f150406d200 0x7f150406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:59.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.061+0000 7f1519600700 1 -- 172.21.15.1:0/848915990 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f150808d930 con 0x7f151c0fe0e0 2024-02-28T16:32:59.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.062+0000 7f1521a00700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f150406d200 0x7f150406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:59.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.062+0000 7f1521a00700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f150406d200 0x7f150406f6b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1510009fd0 tx=0x7f1510009f00 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:59.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.068+0000 7f1519600700 1 -- 172.21.15.1:0/848915990 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f150805b460 con 0x7f151c0fe0e0 2024-02-28T16:32:59.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.346+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f151c02b3b0 con 0x7f150406d200 2024-02-28T16:32:59.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.350+0000 7f1519600700 1 -- 172.21.15.1:0/848915990 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f151c02b3b0 con 0x7f150406d200 2024-02-28T16:32:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.353+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f150406d200 msgr2=0x7f150406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.353+0000 7f1524400700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f150406d200 0x7f150406f6b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1510009fd0 tx=0x7f1510009f00 comp rx=0 tx=0).stop 2024-02-28T16:32:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.353+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 msgr2=0x7f151c1100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.353+0000 7f1524400700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 0x7f151c1100a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1508009fd0 tx=0x7f1508009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.354+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 shutdown_connections 2024-02-28T16:32:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.354+0000 7f1524400700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f151c0fcef0 0x7f151c10fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.354+0000 7f1524400700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f150406d200 0x7f150406f6b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.354+0000 7f1524400700 1 --2- 172.21.15.1:0/848915990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151c0fe0e0 0x7f151c1100a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.354+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 >> 172.21.15.1:0/848915990 conn(0x7f151c0f8620 msgr2=0x7f151c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.354+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 shutdown_connections 2024-02-28T16:32:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.354+0000 7f1524400700 1 -- 172.21.15.1:0/848915990 wait complete. 2024-02-28T16:32:59.360 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:32:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.535+0000 7fa77f800700 1 -- 172.21.15.1:0/1098282952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 msgr2=0x7fa778102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.535+0000 7fa77f800700 1 --2- 172.21.15.1:0/1098282952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 0x7fa778102780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa760004940 tx=0x7fa760009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.535+0000 7fa77f800700 1 -- 172.21.15.1:0/1098282952 shutdown_connections 2024-02-28T16:32:59.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.535+0000 7fa77f800700 1 --2- 172.21.15.1:0/1098282952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 0x7fa778102780 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.535+0000 7fa77f800700 1 --2- 172.21.15.1:0/1098282952 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7780fda80 0x7fa7780ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.535+0000 7fa77f800700 1 -- 172.21.15.1:0/1098282952 >> 172.21.15.1:0/1098282952 conn(0x7fa7780f74e0 msgr2=0x7fa7780f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:59.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.535+0000 7fa77f800700 1 -- 172.21.15.1:0/1098282952 shutdown_connections 2024-02-28T16:32:59.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.535+0000 7fa77f800700 1 -- 172.21.15.1:0/1098282952 wait complete. 2024-02-28T16:32:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.536+0000 7fa77f800700 1 Processor -- start 2024-02-28T16:32:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.536+0000 7fa77f800700 1 -- start start 2024-02-28T16:32:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.536+0000 7fa77f800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7780fda80 0x7fa77810fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa77f800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 0x7fa778110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa77f800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa778110690 con 0x7fa7781003a0 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa77f800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa778110f30 con 0x7fa7780fda80 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa777e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 0x7fa778110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa777e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 0x7fa778110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55860/0 (socket says 172.21.15.1:55860) 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa777e00700 1 -- 172.21.15.1:0/2638822981 learned_addr learned my addr 172.21.15.1:0/2638822981 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa77ce00700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7780fda80 0x7fa77810fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa777e00700 1 -- 172.21.15.1:0/2638822981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7780fda80 msgr2=0x7fa77810fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa777e00700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7780fda80 0x7fa77810fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa777e00700 1 -- 172.21.15.1:0/2638822981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa760005040 con 0x7fa7781003a0 2024-02-28T16:32:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.537+0000 7fa77ce00700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7780fda80 0x7fa77810fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:32:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.538+0000 7fa777e00700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 0x7fa778110090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa76000b3b0 tx=0x7fa760009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.538+0000 7fa774c00700 1 -- 172.21.15.1:0/2638822981 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa76000e040 con 0x7fa7781003a0 2024-02-28T16:32:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.538+0000 7fa774c00700 1 -- 172.21.15.1:0/2638822981 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa76000bd90 con 0x7fa7781003a0 2024-02-28T16:32:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.538+0000 7fa774c00700 1 -- 172.21.15.1:0/2638822981 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa76001c630 con 0x7fa7781003a0 2024-02-28T16:32:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.538+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7781111b0 con 0x7fa7781003a0 2024-02-28T16:32:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.538+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7781116f0 con 0x7fa7781003a0 2024-02-28T16:32:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.540+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77804bed0 con 0x7fa7781003a0 2024-02-28T16:32:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.541+0000 7fa774c00700 1 -- 172.21.15.1:0/2638822981 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 32) v1 ==== 91299+0+0 (secure 0 0 0) 0x7fa7600139f0 con 0x7fa7781003a0 2024-02-28T16:32:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.542+0000 7fa774c00700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa75c06d200 0x7fa75c06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:32:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.542+0000 7fa774c00700 1 -- 172.21.15.1:0/2638822981 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fa76008dbb0 con 0x7fa7781003a0 2024-02-28T16:32:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.543+0000 7fa77ce00700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa75c06d200 0x7fa75c06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:32:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.544+0000 7fa77ce00700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa75c06d200 0x7fa75c06f6b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa770007fd0 tx=0x7fa770009040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:32:59.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.552+0000 7fa774c00700 1 -- 172.21.15.1:0/2638822981 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fa760094080 con 0x7fa7781003a0 2024-02-28T16:32:59.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.938+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa778111b30 con 0x7fa75c06d200 2024-02-28T16:32:59.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.940+0000 7fa774c00700 1 -- 172.21.15.1:0/2638822981 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fa778111b30 con 0x7fa75c06d200 2024-02-28T16:32:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa75c06d200 msgr2=0x7fa75c06f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa75c06d200 0x7fa75c06f6b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa770007fd0 tx=0x7fa770009040 comp rx=0 tx=0).stop 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 msgr2=0x7fa778110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 0x7fa778110090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa76000b3b0 tx=0x7fa760009e30 comp rx=0 tx=0).stop 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 shutdown_connections 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7780fda80 0x7fa77810fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa75c06d200 0x7fa75c06f6b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 --2- 172.21.15.1:0/2638822981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7781003a0 0x7fa778110090 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.942+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 >> 172.21.15.1:0/2638822981 conn(0x7fa7780f74e0 msgr2=0x7fa7780fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.943+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 shutdown_connections 2024-02-28T16:32:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:32:59.943+0000 7fa77f800700 1 -- 172.21.15.1:0/2638822981 wait complete. 2024-02-28T16:33:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.100+0000 7f4dc8000700 1 -- 172.21.15.1:0/1319312407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 msgr2=0x7f4dc0070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.100+0000 7f4dc8000700 1 --2- 172.21.15.1:0/1319312407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 0x7f4dc0070380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4db4004940 tx=0x7f4db4009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.101+0000 7f4dc8000700 1 -- 172.21.15.1:0/1319312407 shutdown_connections 2024-02-28T16:33:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.101+0000 7f4dc8000700 1 --2- 172.21.15.1:0/1319312407 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc00708c0 0x7f4dc0070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.101+0000 7f4dc8000700 1 --2- 172.21.15.1:0/1319312407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 0x7f4dc0070380 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.101+0000 7f4dc8000700 1 -- 172.21.15.1:0/1319312407 >> 172.21.15.1:0/1319312407 conn(0x7f4dc00f8620 msgr2=0x7f4dc00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.101+0000 7f4dc8000700 1 -- 172.21.15.1:0/1319312407 shutdown_connections 2024-02-28T16:33:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.101+0000 7f4dc8000700 1 -- 172.21.15.1:0/1319312407 wait complete. 2024-02-28T16:33:00.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.102+0000 7f4dc8000700 1 Processor -- start 2024-02-28T16:33:00.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.102+0000 7f4dc8000700 1 -- start start 2024-02-28T16:33:00.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.103+0000 7f4dc8000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc00708c0 0x7f4dc010fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:00.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.103+0000 7f4dc8000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 0x7f4dc01100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:00.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.103+0000 7f4dc8000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc01106e0 con 0x7f4dc0071f10 2024-02-28T16:33:00.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.103+0000 7f4dc8000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc0110f80 con 0x7f4dc00708c0 2024-02-28T16:33:00.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.103+0000 7f4dc4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 0x7f4dc01100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:00.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.103+0000 7f4dc4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 0x7f4dc01100e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55874/0 (socket says 172.21.15.1:55874) 2024-02-28T16:33:00.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.103+0000 7f4dc4c00700 1 -- 172.21.15.1:0/1701109358 learned_addr learned my addr 172.21.15.1:0/1701109358 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:33:00.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.103+0000 7f4dc5600700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc00708c0 0x7f4dc010fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:00.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.104+0000 7f4dc4c00700 1 -- 172.21.15.1:0/1701109358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc00708c0 msgr2=0x7f4dc010fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:00.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.104+0000 7f4dc4c00700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc00708c0 0x7f4dc010fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.104+0000 7f4dc4c00700 1 -- 172.21.15.1:0/1701109358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4005040 con 0x7f4dc0071f10 2024-02-28T16:33:00.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.104+0000 7f4dc4c00700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 0x7f4dc01100e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c830 tx=0x7f4dac004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:00.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.105+0000 7f4dbd600700 1 -- 172.21.15.1:0/1701109358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dac00a760 con 0x7f4dc0071f10 2024-02-28T16:33:00.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.106+0000 7f4dbd600700 1 -- 172.21.15.1:0/1701109358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dac00ad90 con 0x7f4dc0071f10 2024-02-28T16:33:00.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.106+0000 7f4dbd600700 1 -- 172.21.15.1:0/1701109358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dac0113a0 con 0x7f4dc0071f10 2024-02-28T16:33:00.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.106+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc0111260 con 0x7f4dc0071f10 2024-02-28T16:33:00.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.106+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc01117a0 con 0x7f4dc0071f10 2024-02-28T16:33:00.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.107+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc004bed0 con 0x7f4dc0071f10 2024-02-28T16:33:00.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.108+0000 7f4dbd600700 1 -- 172.21.15.1:0/1701109358 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 32) v1 ==== 91299+0+0 (secure 0 0 0) 0x7f4dac012790 con 0x7f4dc0071f10 2024-02-28T16:33:00.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.111+0000 7f4dbd600700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4da806d200 0x7f4da806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:00.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.111+0000 7f4dbd600700 1 -- 172.21.15.1:0/1701109358 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f4dac08c680 con 0x7f4dc0071f10 2024-02-28T16:33:00.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.112+0000 7f4dc5600700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4da806d200 0x7f4da806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:00.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.114+0000 7f4dc5600700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4da806d200 0x7f4da806f6b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4db4004f90 tx=0x7f4db4009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.117+0000 7f4dbd600700 1 -- 172.21.15.1:0/1701109358 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f4dac05a200 con 0x7f4dc0071f10 2024-02-28T16:33:00.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.445+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dc0111ca0 con 0x7f4da806d200 2024-02-28T16:33:00.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.459+0000 7f4dbd600700 1 -- 172.21.15.1:0/1701109358 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4dc0111ca0 con 0x7f4da806d200 2024-02-28T16:33:00.447 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:33:00.447 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (2m) 23s ago 13m 15.3M - 0.25.0 c8568f914cd2 04ac88629633 2024-02-28T16:33:00.447 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (13m) 23s ago 13m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:33:00.447 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (12m) 28s ago 12m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:33:00.447 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (2m) 23s ago 13m 74.3M - 9.4.7 954c08fa6188 1010f22bd792 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (6m) 23s ago 6m 144M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (6m) 23s ago 6m 12.8M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (6m) 28s ago 6m 12.9M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (6m) 28s ago 6m 174M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (94s) 23s ago 15m 494M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (34s) 28s ago 12m 64.2M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (15m) 23s ago 15m 51.5M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (11m) 28s ago 11m 40.5M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 starting - - - - 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 starting - - - - 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (11m) 23s ago 11m 181M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (10m) 23s ago 10m 186M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:33:00.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (9m) 23s ago 9m 149M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:33:00.449 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (9m) 28s ago 9m 199M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:33:00.449 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (8m) 28s ago 8m 185M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:33:00.449 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (8m) 28s ago 8m 171M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:33:00.449 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (43s) 23s ago 12m 38.1M - 2.43.0 a07b618ecd1d b28dceb0f5b0 2024-02-28T16:33:00.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.462+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4da806d200 msgr2=0x7f4da806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:00.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.462+0000 7f4dc8000700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4da806d200 0x7f4da806f6b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4db4004f90 tx=0x7f4db4009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:00.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.462+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 msgr2=0x7f4dc01100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:00.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.462+0000 7f4dc8000700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 0x7f4dc01100e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c830 tx=0x7f4dac004470 comp rx=0 tx=0).stop 2024-02-28T16:33:00.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.463+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 shutdown_connections 2024-02-28T16:33:00.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.463+0000 7f4dc8000700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4dc00708c0 0x7f4dc010fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.463+0000 7f4dc8000700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4da806d200 0x7f4da806f6b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.463+0000 7f4dc8000700 1 --2- 172.21.15.1:0/1701109358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dc0071f10 0x7f4dc01100e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.463+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 >> 172.21.15.1:0/1701109358 conn(0x7f4dc00f8620 msgr2=0x7f4dc0102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:00.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.464+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 shutdown_connections 2024-02-28T16:33:00.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.464+0000 7f4dc8000700 1 -- 172.21.15.1:0/1701109358 wait complete. 2024-02-28T16:33:00.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.658+0000 7f8eb7c00700 1 -- 172.21.15.1:0/815283881 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 msgr2=0x7f8eb00fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:00.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.658+0000 7f8eb7c00700 1 --2- 172.21.15.1:0/815283881 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 0x7f8eb00fd420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c004880 tx=0x7f8e9c009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:00.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.658+0000 7f8eb7c00700 1 -- 172.21.15.1:0/815283881 shutdown_connections 2024-02-28T16:33:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.658+0000 7f8eb7c00700 1 --2- 172.21.15.1:0/815283881 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 0x7f8eb00fd420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.658+0000 7f8eb7c00700 1 --2- 172.21.15.1:0/815283881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb00fbe40 0x7f8eb00fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.658+0000 7f8eb7c00700 1 -- 172.21.15.1:0/815283881 >> 172.21.15.1:0/815283881 conn(0x7f8eb00f74e0 msgr2=0x7f8eb00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.659+0000 7f8eb7c00700 1 -- 172.21.15.1:0/815283881 shutdown_connections 2024-02-28T16:33:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.659+0000 7f8eb7c00700 1 -- 172.21.15.1:0/815283881 wait complete. 2024-02-28T16:33:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.660+0000 7f8eb7c00700 1 Processor -- start 2024-02-28T16:33:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.661+0000 7f8eb7c00700 1 -- start start 2024-02-28T16:33:00.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.662+0000 7f8eb7c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb00fbe40 0x7f8eb006e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:00.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.662+0000 7f8eb7c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 0x7f8eb006ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:00.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.662+0000 7f8eb7c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb0106760 con 0x7f8eb00fbe40 2024-02-28T16:33:00.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.662+0000 7f8eb7c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb006f360 con 0x7f8eb00fcfb0 2024-02-28T16:33:00.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.663+0000 7f8eafe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 0x7f8eb006ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:00.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.663+0000 7f8eb5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb00fbe40 0x7f8eb006e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:00.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.663+0000 7f8eafe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 0x7f8eb006ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:34356/0 (socket says 172.21.15.1:34356) 2024-02-28T16:33:00.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.663+0000 7f8eafe00700 1 -- 172.21.15.1:0/1681424362 learned_addr learned my addr 172.21.15.1:0/1681424362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:33:00.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.663+0000 7f8eb5200700 1 -- 172.21.15.1:0/1681424362 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 msgr2=0x7f8eb006ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:00.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.663+0000 7f8eb5200700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 0x7f8eb006ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:00.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.664+0000 7f8eb5200700 1 -- 172.21.15.1:0/1681424362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c005040 con 0x7f8eb00fbe40 2024-02-28T16:33:00.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.664+0000 7f8eb5200700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb00fbe40 0x7f8eb006e8e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009fd0 tx=0x7f8ea400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:00.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.664+0000 7f8eacc00700 1 -- 172.21.15.1:0/1681424362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea4007d90 con 0x7f8eb00fbe40 2024-02-28T16:33:00.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.665+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb011dd90 con 0x7f8eb00fbe40 2024-02-28T16:33:00.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.665+0000 7f8eacc00700 1 -- 172.21.15.1:0/1681424362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ea400f040 con 0x7f8eb00fbe40 2024-02-28T16:33:00.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.665+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb011e0e0 con 0x7f8eb00fbe40 2024-02-28T16:33:00.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.665+0000 7f8eacc00700 1 -- 172.21.15.1:0/1681424362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea400cad0 con 0x7f8eb00fbe40 2024-02-28T16:33:00.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.668+0000 7f8eacc00700 1 -- 172.21.15.1:0/1681424362 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 32) v1 ==== 91299+0+0 (secure 0 0 0) 0x7f8ea400cc30 con 0x7f8eb00fbe40 2024-02-28T16:33:00.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.669+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb004bed0 con 0x7f8eb00fbe40 2024-02-28T16:33:00.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.669+0000 7f8eacc00700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e9806d200 0x7f8e9806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:00.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.669+0000 7f8eacc00700 1 -- 172.21.15.1:0/1681424362 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f8ea408c300 con 0x7f8eb00fbe40 2024-02-28T16:33:00.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.672+0000 7f8eafe00700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e9806d200 0x7f8e9806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:00.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.675+0000 7f8eafe00700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e9806d200 0x7f8e9806f6b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009fd0 tx=0x7f8e9c009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:00.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:00.677+0000 7f8eacc00700 1 -- 172.21.15.1:0/1681424362 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f8ea4059910 con 0x7f8eb00fbe40 2024-02-28T16:33:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:00 smithi066 ceph-mon[36236]: pgmap v40: 65 pgs: 65 active+clean; 328 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 1.1 MiB/s wr, 95 op/s 2024-02-28T16:33:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:00 smithi066 ceph-mon[36236]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:00 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:33:00.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:00 smithi001 ceph-mon[28991]: pgmap v40: 65 pgs: 65 active+clean; 328 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 1.1 MiB/s wr, 95 op/s 2024-02-28T16:33:00.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:00 smithi001 ceph-mon[28991]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:00.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:00 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:33:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.126+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8eb011e550 con 0x7f8eb00fbe40 2024-02-28T16:33:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.127+0000 7f8eacc00700 1 -- 172.21.15.1:0/1681424362 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f8ea4059910 con 0x7f8eb00fbe40 2024-02-28T16:33:01.115 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:33:01.115 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:33:01.116 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:33:01.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 12, 2024-02-28T16:33:01.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:33:01.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:33:01.117 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:33:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e9806d200 msgr2=0x7f8e9806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e9806d200 0x7f8e9806f6b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009fd0 tx=0x7f8e9c009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb00fbe40 msgr2=0x7f8eb006e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb00fbe40 0x7f8eb006e8e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009fd0 tx=0x7f8ea400b040 comp rx=0 tx=0).stop 2024-02-28T16:33:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 shutdown_connections 2024-02-28T16:33:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e9806d200 0x7f8e9806f6b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb00fbe40 0x7f8eb006e8e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 --2- 172.21.15.1:0/1681424362 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb00fcfb0 0x7f8eb006ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 >> 172.21.15.1:0/1681424362 conn(0x7f8eb00f74e0 msgr2=0x7f8eb00f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.130+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 shutdown_connections 2024-02-28T16:33:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.131+0000 7f8eb7c00700 1 -- 172.21.15.1:0/1681424362 wait complete. 2024-02-28T16:33:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.314+0000 7f4769600700 1 -- 172.21.15.1:0/823717478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4764102430 msgr2=0x7f4764104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.314+0000 7f4769600700 1 --2- 172.21.15.1:0/823717478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4764102430 0x7f4764104810 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4750004940 tx=0x7f4750009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.315+0000 7f4769600700 1 -- 172.21.15.1:0/823717478 shutdown_connections 2024-02-28T16:33:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.315+0000 7f4769600700 1 --2- 172.21.15.1:0/823717478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4764102430 0x7f4764104810 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.315+0000 7f4769600700 1 --2- 172.21.15.1:0/823717478 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47640663e0 0x7f4764101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.315+0000 7f4769600700 1 -- 172.21.15.1:0/823717478 >> 172.21.15.1:0/823717478 conn(0x7f47640f74e0 msgr2=0x7f47640f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:01.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.315+0000 7f4769600700 1 -- 172.21.15.1:0/823717478 shutdown_connections 2024-02-28T16:33:01.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.315+0000 7f4769600700 1 -- 172.21.15.1:0/823717478 wait complete. 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.316+0000 7f4769600700 1 Processor -- start 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.316+0000 7f4769600700 1 -- start start 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.316+0000 7f4769600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47640663e0 0x7f4764113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.316+0000 7f4769600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4764102430 0x7f4764114450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.316+0000 7f4769600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4764114a50 con 0x7f47640663e0 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.316+0000 7f4769600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47641152f0 con 0x7f4764102430 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.317+0000 7f4762a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47640663e0 0x7f4764113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.317+0000 7f4762a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47640663e0 0x7f4764113f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55918/0 (socket says 172.21.15.1:55918) 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.317+0000 7f4762a00700 1 -- 172.21.15.1:0/59679352 learned_addr learned my addr 172.21.15.1:0/59679352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:33:01.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.317+0000 7f4762000700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4764102430 0x7f4764114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.317+0000 7f4762a00700 1 -- 172.21.15.1:0/59679352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4764102430 msgr2=0x7f4764114450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.317+0000 7f4762a00700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4764102430 0x7f4764114450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.317+0000 7f4762a00700 1 -- 172.21.15.1:0/59679352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4750005040 con 0x7f47640663e0 2024-02-28T16:33:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.318+0000 7f4762a00700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47640663e0 0x7f4764113f10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f475400eb70 tx=0x7f475400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.318+0000 7f475aa00700 1 -- 172.21.15.1:0/59679352 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4754007cc0 con 0x7f47640663e0 2024-02-28T16:33:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.318+0000 7f475aa00700 1 -- 172.21.15.1:0/59679352 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f475400f040 con 0x7f47640663e0 2024-02-28T16:33:01.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.318+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4764115570 con 0x7f47640663e0 2024-02-28T16:33:01.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.319+0000 7f475aa00700 1 -- 172.21.15.1:0/59679352 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f475400ca90 con 0x7f47640663e0 2024-02-28T16:33:01.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.319+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4764115ab0 con 0x7f47640663e0 2024-02-28T16:33:01.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.320+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4764100990 con 0x7f47640663e0 2024-02-28T16:33:01.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.321+0000 7f475aa00700 1 -- 172.21.15.1:0/59679352 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 32) v1 ==== 91299+0+0 (secure 0 0 0) 0x7f475400cbf0 con 0x7f47640663e0 2024-02-28T16:33:01.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.322+0000 7f475aa00700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f474806d200 0x7f474806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:01.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.322+0000 7f475aa00700 1 -- 172.21.15.1:0/59679352 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f475408c120 con 0x7f47640663e0 2024-02-28T16:33:01.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.323+0000 7f4762000700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f474806d200 0x7f474806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:01.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.324+0000 7f4762000700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f474806d200 0x7f474806f6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f475000b0d0 tx=0x7f4750009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:01.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.329+0000 7f475aa00700 1 -- 172.21.15.1:0/59679352 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f4754091060 con 0x7f47640663e0 2024-02-28T16:33:01.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.710+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f476402b3b0 con 0x7f474806d200 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr" 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "2/2 daemons upgraded", 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading node-exporter daemons", 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:33:01.700 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:33:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.711+0000 7f475aa00700 1 -- 172.21.15.1:0/59679352 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f476402b3b0 con 0x7f474806d200 2024-02-28T16:33:01.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f474806d200 msgr2=0x7f474806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:01.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f474806d200 0x7f474806f6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f475000b0d0 tx=0x7f4750009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:01.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47640663e0 msgr2=0x7f4764113f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:01.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47640663e0 0x7f4764113f10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f475400eb70 tx=0x7f475400b040 comp rx=0 tx=0).stop 2024-02-28T16:33:01.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 shutdown_connections 2024-02-28T16:33:01.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f474806d200 0x7f474806f6b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47640663e0 0x7f4764113f10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 --2- 172.21.15.1:0/59679352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4764102430 0x7f4764114450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:01.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 >> 172.21.15.1:0/59679352 conn(0x7f47640f74e0 msgr2=0x7f47640f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:01.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 shutdown_connections 2024-02-28T16:33:01.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:01.715+0000 7f4769600700 1 -- 172.21.15.1:0/59679352 wait complete. 2024-02-28T16:33:01.719 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:01 smithi066 ceph-mon[36236]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:01.719 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:01 smithi066 ceph-mon[36236]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:01.719 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:01 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1681424362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:01.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:01 smithi001 ceph-mon[28991]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:01.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:01 smithi001 ceph-mon[28991]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:01.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:01 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1681424362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:02.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:02 smithi066 ceph-mon[36236]: pgmap v41: 65 pgs: 65 active+clean; 330 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-02-28T16:33:02.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:02 smithi066 ceph-mon[36236]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:02.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:02 smithi001 ceph-mon[28991]: pgmap v41: 65 pgs: 65 active+clean; 330 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 35 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-02-28T16:33:02.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:02 smithi001 ceph-mon[28991]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:04.098 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:04 smithi001 ceph-mon[28991]: pgmap v42: 65 pgs: 65 active+clean; 330 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 22 KiB/s rd, 979 KiB/s wr, 122 op/s 2024-02-28T16:33:04.098 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:04 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:04.098 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:04 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:04.401 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:04 smithi066 ceph-mon[36236]: pgmap v42: 65 pgs: 65 active+clean; 330 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 22 KiB/s rd, 979 KiB/s wr, 122 op/s 2024-02-28T16:33:04.402 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:04 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:04.402 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:04 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:06.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:06 smithi066 ceph-mon[36236]: pgmap v43: 65 pgs: 65 active+clean; 330 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 22 KiB/s rd, 965 KiB/s wr, 116 op/s 2024-02-28T16:33:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:06 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:06 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:06 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:06 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:06.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:06 smithi001 ceph-mon[28991]: pgmap v43: 65 pgs: 65 active+clean; 330 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 22 KiB/s rd, 965 KiB/s wr, 116 op/s 2024-02-28T16:33:06.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:06 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:06.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:06 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:06.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:06 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:06.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:06 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:08.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:08 smithi066 ceph-mon[36236]: pgmap v44: 65 pgs: 65 active+clean; 332 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 36 KiB/s rd, 1.5 MiB/s wr, 209 op/s 2024-02-28T16:33:08.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:08 smithi001 ceph-mon[28991]: pgmap v44: 65 pgs: 65 active+clean; 332 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 36 KiB/s rd, 1.5 MiB/s wr, 209 op/s 2024-02-28T16:33:08.937 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-28T16:33:08.937 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-28T16:33:09.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:09 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:09.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:09 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:09.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:09 smithi001 ceph-mon[28991]: pgmap v45: 65 pgs: 65 active+clean; 332 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 25 KiB/s rd, 1.1 MiB/s wr, 158 op/s 2024-02-28T16:33:09.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:09 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:09.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:09 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:09.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:09 smithi066 ceph-mon[36236]: pgmap v45: 65 pgs: 65 active+clean; 332 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 25 KiB/s rd, 1.1 MiB/s wr, 158 op/s 2024-02-28T16:33:10.179 DEBUG:teuthology.parallel:result is None 2024-02-28T16:33:12.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:11 smithi066 ceph-mon[36236]: pgmap v46: 65 pgs: 65 active+clean; 331 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 39 KiB/s rd, 1.4 MiB/s wr, 205 op/s 2024-02-28T16:33:12.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:11 smithi001 ceph-mon[28991]: pgmap v46: 65 pgs: 65 active+clean; 331 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 39 KiB/s rd, 1.4 MiB/s wr, 205 op/s 2024-02-28T16:33:13.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:12 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:13.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:12 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:13.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:12 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:13.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:12 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:14.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:13 smithi001 ceph-mon[28991]: pgmap v47: 65 pgs: 65 active+clean; 330 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 28 KiB/s rd, 1.2 MiB/s wr, 170 op/s 2024-02-28T16:33:14.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:13 smithi066 ceph-mon[36236]: pgmap v47: 65 pgs: 65 active+clean; 330 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 28 KiB/s rd, 1.2 MiB/s wr, 170 op/s 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: pgmap v48: 65 pgs: 65 active+clean; 330 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 28 KiB/s rd, 1.1 MiB/s wr, 161 op/s 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:33:15.711 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.712 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:33:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:33:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:15.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: pgmap v48: 65 pgs: 65 active+clean; 330 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 28 KiB/s rd, 1.1 MiB/s wr, 161 op/s 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:16.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:17 smithi066 ceph-mon[36236]: mgrmap e33: smithi001.rmthtx(active, since 91s), standbys: smithi066.ztraxe 2024-02-28T16:33:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:17 smithi066 ceph-mon[36236]: pgmap v49: 65 pgs: 65 active+clean; 326 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 33 KiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-02-28T16:33:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:17 smithi066 ceph-mon[36236]: Upgrade: Updating prometheus.smithi001 2024-02-28T16:33:18.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:17 smithi001 ceph-mon[28991]: mgrmap e33: smithi001.rmthtx(active, since 91s), standbys: smithi066.ztraxe 2024-02-28T16:33:18.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:17 smithi001 ceph-mon[28991]: pgmap v49: 65 pgs: 65 active+clean; 326 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 33 KiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-02-28T16:33:18.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:17 smithi001 ceph-mon[28991]: Upgrade: Updating prometheus.smithi001 2024-02-28T16:33:18.970 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:18 smithi001 ceph-mon[28991]: Deploying daemon prometheus.smithi001 on smithi001 2024-02-28T16:33:18.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:18 smithi066 ceph-mon[36236]: Deploying daemon prometheus.smithi001 on smithi001 2024-02-28T16:33:20.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:20 smithi066 ceph-mon[36236]: pgmap v50: 65 pgs: 65 active+clean; 326 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 742 KiB/s wr, 113 op/s 2024-02-28T16:33:20.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:20 smithi001 ceph-mon[28991]: pgmap v50: 65 pgs: 65 active+clean; 326 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 742 KiB/s wr, 113 op/s 2024-02-28T16:33:21.604 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:21 smithi001 ceph-mon[28991]: pgmap v51: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 908 KiB/s wr, 143 op/s 2024-02-28T16:33:21.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:21 smithi066 ceph-mon[36236]: pgmap v51: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 908 KiB/s wr, 143 op/s 2024-02-28T16:33:24.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:23 smithi066 ceph-mon[36236]: pgmap v52: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.9 KiB/s rd, 544 KiB/s wr, 100 op/s 2024-02-28T16:33:24.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:23 smithi001 ceph-mon[28991]: pgmap v52: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.9 KiB/s rd, 544 KiB/s wr, 100 op/s 2024-02-28T16:33:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:26 smithi066 ceph-mon[36236]: pgmap v53: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.9 KiB/s rd, 381 KiB/s wr, 78 op/s 2024-02-28T16:33:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:26 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:26 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:26 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:33:26.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:26 smithi001 ceph-mon[28991]: pgmap v53: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.9 KiB/s rd, 381 KiB/s wr, 78 op/s 2024-02-28T16:33:26.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:26 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:26.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:26 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:26.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:26 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:33:28.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:28 smithi066 ceph-mon[36236]: pgmap v54: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 13 KiB/s rd, 548 KiB/s wr, 113 op/s 2024-02-28T16:33:28.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:28 smithi001 ceph-mon[28991]: pgmap v54: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 13 KiB/s rd, 548 KiB/s wr, 113 op/s 2024-02-28T16:33:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:30 smithi066 ceph-mon[36236]: pgmap v55: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.3 KiB/s rd, 333 KiB/s wr, 68 op/s 2024-02-28T16:33:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:30 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:33:30.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:30 smithi001 ceph-mon[28991]: pgmap v55: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.3 KiB/s rd, 333 KiB/s wr, 68 op/s 2024-02-28T16:33:30.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:30 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:33:31.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:31 smithi066 ceph-mon[36236]: pgmap v56: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 13 KiB/s rd, 522 KiB/s wr, 98 op/s 2024-02-28T16:33:31.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:31 smithi001 ceph-mon[28991]: pgmap v56: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 13 KiB/s rd, 522 KiB/s wr, 98 op/s 2024-02-28T16:33:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.891+0000 7f03a2800700 1 -- 172.21.15.1:0/3713642238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c1001d0 msgr2=0x7f039c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.891+0000 7f03a2800700 1 --2- 172.21.15.1:0/3713642238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c1001d0 0x7f039c1025b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0388004940 tx=0x7f0388009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.891+0000 7f03a2800700 1 -- 172.21.15.1:0/3713642238 shutdown_connections 2024-02-28T16:33:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.891+0000 7f03a2800700 1 --2- 172.21.15.1:0/3713642238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c1001d0 0x7f039c1025b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.891+0000 7f03a2800700 1 --2- 172.21.15.1:0/3713642238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f039c0fd8b0 0x7f039c0ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.891+0000 7f03a2800700 1 -- 172.21.15.1:0/3713642238 >> 172.21.15.1:0/3713642238 conn(0x7f039c0f7330 msgr2=0x7f039c0f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.891+0000 7f03a2800700 1 -- 172.21.15.1:0/3713642238 shutdown_connections 2024-02-28T16:33:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.891+0000 7f03a2800700 1 -- 172.21.15.1:0/3713642238 wait complete. 2024-02-28T16:33:31.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.892+0000 7f03a2800700 1 Processor -- start 2024-02-28T16:33:31.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.893+0000 7f03a2800700 1 -- start start 2024-02-28T16:33:31.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.893+0000 7f03a2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c0fd8b0 0x7f039c10d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:31.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.893+0000 7f03a2800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f039c1001d0 0x7f039c10dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:31.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.893+0000 7f03a2800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c10e2c0 con 0x7f039c0fd8b0 2024-02-28T16:33:31.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.893+0000 7f03a2800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c10eb00 con 0x7f039c1001d0 2024-02-28T16:33:31.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.893+0000 7f03a1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c0fd8b0 0x7f039c10d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:31.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.893+0000 7f03a1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c0fd8b0 0x7f039c10d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52606/0 (socket says 172.21.15.1:52606) 2024-02-28T16:33:31.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.893+0000 7f03a1400700 1 -- 172.21.15.1:0/916963564 learned_addr learned my addr 172.21.15.1:0/916963564 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:33:31.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.894+0000 7f03a0a00700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f039c1001d0 0x7f039c10dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:31.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.894+0000 7f03a1400700 1 -- 172.21.15.1:0/916963564 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f039c1001d0 msgr2=0x7f039c10dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.894+0000 7f03a1400700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f039c1001d0 0x7f039c10dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.894+0000 7f03a1400700 1 -- 172.21.15.1:0/916963564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0388005040 con 0x7f039c0fd8b0 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.894+0000 7f03a1400700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c0fd8b0 0x7f039c10d6f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f039000bbb0 tx=0x7f0390009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.895+0000 7f0399600700 1 -- 172.21.15.1:0/916963564 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03900092e0 con 0x7f039c0fd8b0 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.895+0000 7f0399600700 1 -- 172.21.15.1:0/916963564 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f039000cb20 con 0x7f039c0fd8b0 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.895+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c10ede0 con 0x7f039c0fd8b0 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.895+0000 7f0399600700 1 -- 172.21.15.1:0/916963564 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0390015d70 con 0x7f039c0fd8b0 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.895+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c10f3a0 con 0x7f039c0fd8b0 2024-02-28T16:33:31.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.896+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c107900 con 0x7f039c0fd8b0 2024-02-28T16:33:31.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.897+0000 7f0399600700 1 -- 172.21.15.1:0/916963564 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f0390012070 con 0x7f039c0fd8b0 2024-02-28T16:33:31.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.898+0000 7f0399600700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f038406d0f0 0x7f038406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:31.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.899+0000 7f0399600700 1 -- 172.21.15.1:0/916963564 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f039008c2c0 con 0x7f039c0fd8b0 2024-02-28T16:33:31.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.899+0000 7f03a0a00700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f038406d0f0 0x7f038406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:31.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.899+0000 7f03a0a00700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f038406d0f0 0x7f038406f5a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f03880043d0 tx=0x7f0388009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:31.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:31.903+0000 7f0399600700 1 -- 172.21.15.1:0/916963564 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f0390059e50 con 0x7f039c0fd8b0 2024-02-28T16:33:32.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.180+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f039c10f630 con 0x7f038406d0f0 2024-02-28T16:33:32.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.181+0000 7f0399600700 1 -- 172.21.15.1:0/916963564 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f039c10f630 con 0x7f038406d0f0 2024-02-28T16:33:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f038406d0f0 msgr2=0x7f038406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f038406d0f0 0x7f038406f5a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f03880043d0 tx=0x7f0388009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c0fd8b0 msgr2=0x7f039c10d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c0fd8b0 0x7f039c10d6f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f039000bbb0 tx=0x7f0390009840 comp rx=0 tx=0).stop 2024-02-28T16:33:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 shutdown_connections 2024-02-28T16:33:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f038406d0f0 0x7f038406f5a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f039c0fd8b0 0x7f039c10d6f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 --2- 172.21.15.1:0/916963564 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f039c1001d0 0x7f039c10dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 >> 172.21.15.1:0/916963564 conn(0x7f039c0f7330 msgr2=0x7f039c0f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:32.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 shutdown_connections 2024-02-28T16:33:32.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.184+0000 7f03a2800700 1 -- 172.21.15.1:0/916963564 wait complete. 2024-02-28T16:33:32.189 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:33:32.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.343+0000 7f8851000700 1 -- 172.21.15.1:0/2589459475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 msgr2=0x7f884c0fc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.343+0000 7f8851000700 1 --2- 172.21.15.1:0/2589459475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 0x7f884c0fc640 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8838004940 tx=0x7f8838009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.343+0000 7f8851000700 1 -- 172.21.15.1:0/2589459475 shutdown_connections 2024-02-28T16:33:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.343+0000 7f8851000700 1 --2- 172.21.15.1:0/2589459475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 0x7f884c0fc640 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.343+0000 7f8851000700 1 --2- 172.21.15.1:0/2589459475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f884c0fb880 0x7f884c0fbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.343+0000 7f8851000700 1 -- 172.21.15.1:0/2589459475 >> 172.21.15.1:0/2589459475 conn(0x7f884c0f7510 msgr2=0x7f884c0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.343+0000 7f8851000700 1 -- 172.21.15.1:0/2589459475 shutdown_connections 2024-02-28T16:33:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.343+0000 7f8851000700 1 -- 172.21.15.1:0/2589459475 wait complete. 2024-02-28T16:33:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.344+0000 7f8851000700 1 Processor -- start 2024-02-28T16:33:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.344+0000 7f8851000700 1 -- start start 2024-02-28T16:33:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8851000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f884c0fb880 0x7f884c10fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8851000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 0x7f884c1100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8851000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c1106b0 con 0x7f884c0fc1d0 2024-02-28T16:33:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8851000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c110f50 con 0x7f884c0fb880 2024-02-28T16:33:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8849400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 0x7f884c1100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8849400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 0x7f884c1100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52632/0 (socket says 172.21.15.1:52632) 2024-02-28T16:33:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8849400700 1 -- 172.21.15.1:0/1254724783 learned_addr learned my addr 172.21.15.1:0/1254724783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:33:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8849e00700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f884c0fb880 0x7f884c10fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.345+0000 7f8849400700 1 -- 172.21.15.1:0/1254724783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f884c0fb880 msgr2=0x7f884c10fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.346+0000 7f8849400700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f884c0fb880 0x7f884c10fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.346+0000 7f8849400700 1 -- 172.21.15.1:0/1254724783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838005040 con 0x7f884c0fc1d0 2024-02-28T16:33:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.346+0000 7f8849400700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 0x7f884c1100b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f883800b3b0 tx=0x7f8838009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.346+0000 7f8842000700 1 -- 172.21.15.1:0/1254724783 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f883800e040 con 0x7f884c0fc1d0 2024-02-28T16:33:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.346+0000 7f8842000700 1 -- 172.21.15.1:0/1254724783 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f883800bd90 con 0x7f884c0fc1d0 2024-02-28T16:33:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.346+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884c1111d0 con 0x7f884c0fc1d0 2024-02-28T16:33:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.346+0000 7f8842000700 1 -- 172.21.15.1:0/1254724783 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f883801c630 con 0x7f884c0fc1d0 2024-02-28T16:33:32.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.346+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f884c111710 con 0x7f884c0fc1d0 2024-02-28T16:33:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.348+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884c04bed0 con 0x7f884c0fc1d0 2024-02-28T16:33:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.349+0000 7f8842000700 1 -- 172.21.15.1:0/1254724783 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f88380139f0 con 0x7f884c0fc1d0 2024-02-28T16:33:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.350+0000 7f8842000700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f883006d0f0 0x7f883006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.350+0000 7f8842000700 1 -- 172.21.15.1:0/1254724783 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f883808d9e0 con 0x7f884c0fc1d0 2024-02-28T16:33:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.350+0000 7f8849e00700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f883006d0f0 0x7f883006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.351+0000 7f8849e00700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f883006d0f0 0x7f883006f5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f883c004990 tx=0x7f883c009ee0 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:32.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.355+0000 7f8842000700 1 -- 172.21.15.1:0/1254724783 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f883805b670 con 0x7f884c0fc1d0 2024-02-28T16:33:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.607+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f884c1055d0 con 0x7f883006d0f0 2024-02-28T16:33:32.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.609+0000 7f8842000700 1 -- 172.21.15.1:0/1254724783 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f884c1055d0 con 0x7f883006d0f0 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.611+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f883006d0f0 msgr2=0x7f883006f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.611+0000 7f8851000700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f883006d0f0 0x7f883006f5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f883c004990 tx=0x7f883c009ee0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.611+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 msgr2=0x7f884c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.611+0000 7f8851000700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 0x7f884c1100b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f883800b3b0 tx=0x7f8838009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.612+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 shutdown_connections 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.612+0000 7f8851000700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f884c0fb880 0x7f884c10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.612+0000 7f8851000700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f883006d0f0 0x7f883006f5a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.612+0000 7f8851000700 1 --2- 172.21.15.1:0/1254724783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f884c0fc1d0 0x7f884c1100b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.612+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 >> 172.21.15.1:0/1254724783 conn(0x7f884c0f7510 msgr2=0x7f884c103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.612+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 shutdown_connections 2024-02-28T16:33:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.612+0000 7f8851000700 1 -- 172.21.15.1:0/1254724783 wait complete. 2024-02-28T16:33:32.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:32 smithi066 ceph-mon[36236]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.770+0000 7fcbcc800700 1 -- 172.21.15.1:0/1503741020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 msgr2=0x7fcbc4103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.770+0000 7fcbcc800700 1 --2- 172.21.15.1:0/1503741020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 0x7fcbc4103950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8004940 tx=0x7fcbb8009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.770+0000 7fcbcc800700 1 -- 172.21.15.1:0/1503741020 shutdown_connections 2024-02-28T16:33:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.770+0000 7fcbcc800700 1 --2- 172.21.15.1:0/1503741020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc4104750 0x7fcbc4104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.770+0000 7fcbcc800700 1 --2- 172.21.15.1:0/1503741020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 0x7fcbc4103950 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.770+0000 7fcbcc800700 1 -- 172.21.15.1:0/1503741020 >> 172.21.15.1:0/1503741020 conn(0x7fcbc40727d0 msgr2=0x7fcbc4074c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:32.760 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:32 smithi001 ceph-mon[28991]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.770+0000 7fcbcc800700 1 -- 172.21.15.1:0/1503741020 shutdown_connections 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.770+0000 7fcbcc800700 1 -- 172.21.15.1:0/1503741020 wait complete. 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.771+0000 7fcbcc800700 1 Processor -- start 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.771+0000 7fcbcc800700 1 -- start start 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbcc800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 0x7fcbc4113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbcc800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc4104750 0x7fcbc41144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbcc800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4114aa0 con 0x7fcbc4103540 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbcc800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4115340 con 0x7fcbc4104750 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbc9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 0x7fcbc4113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbc9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 0x7fcbc4113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52640/0 (socket says 172.21.15.1:52640) 2024-02-28T16:33:32.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbc9e00700 1 -- 172.21.15.1:0/818955725 learned_addr learned my addr 172.21.15.1:0/818955725 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:33:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbc9e00700 1 -- 172.21.15.1:0/818955725 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc4104750 msgr2=0x7fcbc41144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbc9400700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc4104750 0x7fcbc41144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.772+0000 7fcbc9e00700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc4104750 0x7fcbc41144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.773+0000 7fcbc9e00700 1 -- 172.21.15.1:0/818955725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb8005040 con 0x7fcbc4103540 2024-02-28T16:33:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.773+0000 7fcbc9e00700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 0x7fcbc4113f60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8000c00 tx=0x7fcbb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.773+0000 7fcbbe000700 1 -- 172.21.15.1:0/818955725 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbb800e040 con 0x7fcbc4103540 2024-02-28T16:33:32.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.773+0000 7fcbbe000700 1 -- 172.21.15.1:0/818955725 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbb800bcc0 con 0x7fcbc4103540 2024-02-28T16:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.773+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc41155c0 con 0x7fcbc4103540 2024-02-28T16:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.773+0000 7fcbbe000700 1 -- 172.21.15.1:0/818955725 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbb8013ea0 con 0x7fcbc4103540 2024-02-28T16:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.773+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc4115b00 con 0x7fcbc4103540 2024-02-28T16:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.774+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc404bed0 con 0x7fcbc4103540 2024-02-28T16:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.776+0000 7fcbbe000700 1 -- 172.21.15.1:0/818955725 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7fcbb8013420 con 0x7fcbc4103540 2024-02-28T16:33:32.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.777+0000 7fcbbe000700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fcbac06d0f0 0x7fcbac06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:32.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.777+0000 7fcbbe000700 1 -- 172.21.15.1:0/818955725 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fcbb808d930 con 0x7fcbc4103540 2024-02-28T16:33:32.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.777+0000 7fcbc9400700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fcbac06d0f0 0x7fcbac06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:32.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.778+0000 7fcbc9400700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fcbac06d0f0 0x7fcbac06f5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40041d0 tx=0x7fcbb4009040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:32.782+0000 7fcbbe000700 1 -- 172.21.15.1:0/818955725 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fcbb805b4c0 con 0x7fcbc4103540 2024-02-28T16:33:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.045+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbc41090a0 con 0x7fcbac06d0f0 2024-02-28T16:33:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.067+0000 7fcbbe000700 1 -- 172.21.15.1:0/818955725 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fcbc41090a0 con 0x7fcbac06d0f0 2024-02-28T16:33:33.058 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:33:33.058 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (3m) 26s ago 13m 15.3M - 0.25.0 c8568f914cd2 04ac88629633 2024-02-28T16:33:33.058 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (14m) 26s ago 14m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:33:33.058 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (12m) 29s ago 12m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (2m) 26s ago 13m 73.7M - 9.4.7 954c08fa6188 1010f22bd792 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (6m) 26s ago 6m 131M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (6m) 26s ago 6m 13.0M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (6m) 29s ago 6m 13.0M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (6m) 29s ago 6m 152M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (2m) 26s ago 15m 496M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (66s) 29s ago 12m 446M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (15m) 26s ago 15m 51.8M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (12m) 29s ago 12m 42.5M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (45s) 26s ago 14m 6841k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:33:33.059 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (37s) 29s ago 12m 3053k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:33:33.060 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (11m) 26s ago 11m 191M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:33:33.060 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (11m) 26s ago 11m 194M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:33:33.060 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (10m) 26s ago 10m 160M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:33:33.060 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (9m) 29s ago 9m 215M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:33:33.060 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (9m) 29s ago 9m 197M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:33:33.060 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (8m) 29s ago 8m 181M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:33:33.061 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 starting - - - - 2024-02-28T16:33:33.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fcbac06d0f0 msgr2=0x7fcbac06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fcbac06d0f0 0x7fcbac06f5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40041d0 tx=0x7fcbb4009040 comp rx=0 tx=0).stop 2024-02-28T16:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 msgr2=0x7fcbc4113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 0x7fcbc4113f60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8000c00 tx=0x7fcbb8009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 shutdown_connections 2024-02-28T16:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fcbac06d0f0 0x7fcbac06f5a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbc4103540 0x7fcbc4113f60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 --2- 172.21.15.1:0/818955725 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbc4104750 0x7fcbc41144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 >> 172.21.15.1:0/818955725 conn(0x7fcbc40727d0 msgr2=0x7fcbc4107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 shutdown_connections 2024-02-28T16:33:33.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.071+0000 7fcbcc800700 1 -- 172.21.15.1:0/818955725 wait complete. 2024-02-28T16:33:33.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.265+0000 7f09a9e00700 1 -- 172.21.15.1:0/1392449200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 msgr2=0x7f09a40fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:33.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.265+0000 7f09a9e00700 1 --2- 172.21.15.1:0/1392449200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 0x7f09a40fd320 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0994004940 tx=0x7f0994009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:33.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.265+0000 7f09a9e00700 1 -- 172.21.15.1:0/1392449200 shutdown_connections 2024-02-28T16:33:33.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.265+0000 7f09a9e00700 1 --2- 172.21.15.1:0/1392449200 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09a40fe0e0 0x7f09a40fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.265+0000 7f09a9e00700 1 --2- 172.21.15.1:0/1392449200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 0x7f09a40fd320 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.265+0000 7f09a9e00700 1 -- 172.21.15.1:0/1392449200 >> 172.21.15.1:0/1392449200 conn(0x7f09a40f8620 msgr2=0x7f09a40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:33.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.265+0000 7f09a9e00700 1 -- 172.21.15.1:0/1392449200 shutdown_connections 2024-02-28T16:33:33.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.265+0000 7f09a9e00700 1 -- 172.21.15.1:0/1392449200 wait complete. 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.266+0000 7f09a9e00700 1 Processor -- start 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.266+0000 7f09a9e00700 1 -- start start 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 0x7f09a4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a9e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09a40fe0e0 0x7f09a40702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a9e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4070970 con 0x7f09a40fcef0 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a9e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4070ae0 con 0x7f09a40fe0e0 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 0x7f09a4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 0x7f09a4071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52664/0 (socket says 172.21.15.1:52664) 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a2a00700 1 -- 172.21.15.1:0/4128501651 learned_addr learned my addr 172.21.15.1:0/4128501651 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:33:33.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a2000700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09a40fe0e0 0x7f09a40702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:33.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a2a00700 1 -- 172.21.15.1:0/4128501651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09a40fe0e0 msgr2=0x7f09a40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:33.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a2a00700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09a40fe0e0 0x7f09a40702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.267+0000 7f09a2a00700 1 -- 172.21.15.1:0/4128501651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0994005040 con 0x7f09a40fcef0 2024-02-28T16:33:33.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.268+0000 7f09a2a00700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 0x7f09a4071e60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0994000c00 tx=0x7f0994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:33.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.268+0000 7f099aa00700 1 -- 172.21.15.1:0/4128501651 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f099400e030 con 0x7f09a40fcef0 2024-02-28T16:33:33.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.268+0000 7f099aa00700 1 -- 172.21.15.1:0/4128501651 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f099400bbf0 con 0x7f09a40fcef0 2024-02-28T16:33:33.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.268+0000 7f099aa00700 1 -- 172.21.15.1:0/4128501651 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f099401c400 con 0x7f09a40fcef0 2024-02-28T16:33:33.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.268+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a4070d60 con 0x7f09a40fcef0 2024-02-28T16:33:33.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.268+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a41159a0 con 0x7f09a40fcef0 2024-02-28T16:33:33.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.269+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a404bed0 con 0x7f09a40fcef0 2024-02-28T16:33:33.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.271+0000 7f099aa00700 1 -- 172.21.15.1:0/4128501651 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f0994013420 con 0x7f09a40fcef0 2024-02-28T16:33:33.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.272+0000 7f099aa00700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098806d140 0x7f098806f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:33.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.272+0000 7f099aa00700 1 -- 172.21.15.1:0/4128501651 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f099408d9f0 con 0x7f09a40fcef0 2024-02-28T16:33:33.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.272+0000 7f09a2000700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098806d140 0x7f098806f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:33.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.273+0000 7f09a2000700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098806d140 0x7f098806f5f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0990009bf0 tx=0x7f09900091a0 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:33.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.277+0000 7f099aa00700 1 -- 172.21.15.1:0/4128501651 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f099405b680 con 0x7f09a40fcef0 2024-02-28T16:33:33.438 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-28T16:33:33.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-02-28T16:33:33.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.591+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f09a4105ab0 con 0x7f09a40fcef0 2024-02-28T16:33:33.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.592+0000 7f099aa00700 1 -- 172.21.15.1:0/4128501651 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f099405b210 con 0x7f09a40fcef0 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:33:33.582 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:33:33.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:33:33.583 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:33:33.583 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 12, 2024-02-28T16:33:33.583 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:33:33.583 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:33:33.583 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:33:33.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098806d140 msgr2=0x7f098806f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:33.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098806d140 0x7f098806f5f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0990009bf0 tx=0x7f09900091a0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 msgr2=0x7f09a4071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:33.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 0x7f09a4071e60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0994000c00 tx=0x7f0994009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:33.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 shutdown_connections 2024-02-28T16:33:33.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098806d140 0x7f098806f5f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09a40fcef0 0x7f09a4071e60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 --2- 172.21.15.1:0/4128501651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f09a40fe0e0 0x7f09a40702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.595+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 >> 172.21.15.1:0/4128501651 conn(0x7f09a40f8620 msgr2=0x7f09a41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:33.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.596+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 shutdown_connections 2024-02-28T16:33:33.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.596+0000 7f09a9e00700 1 -- 172.21.15.1:0/4128501651 wait complete. 2024-02-28T16:33:33.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:33 smithi066 ceph-mon[36236]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:33.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:33 smithi066 ceph-mon[36236]: pgmap v57: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.7 KiB/s rd, 356 KiB/s wr, 73 op/s 2024-02-28T16:33:33.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:33 smithi066 ceph-mon[36236]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.766+0000 7f9d3c600700 1 -- 172.21.15.1:0/1396786648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34102430 msgr2=0x7f9d34104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.766+0000 7f9d3c600700 1 --2- 172.21.15.1:0/1396786648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34102430 0x7f9d34104810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9d20004940 tx=0x7f9d20009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.767+0000 7f9d3c600700 1 -- 172.21.15.1:0/1396786648 shutdown_connections 2024-02-28T16:33:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.767+0000 7f9d3c600700 1 --2- 172.21.15.1:0/1396786648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34102430 0x7f9d34104810 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.767+0000 7f9d3c600700 1 --2- 172.21.15.1:0/1396786648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d34066c70 0x7f9d34101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.767+0000 7f9d3c600700 1 -- 172.21.15.1:0/1396786648 >> 172.21.15.1:0/1396786648 conn(0x7f9d340f7500 msgr2=0x7f9d340f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.767+0000 7f9d3c600700 1 -- 172.21.15.1:0/1396786648 shutdown_connections 2024-02-28T16:33:33.756 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:33 smithi001 ceph-mon[28991]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:33.757 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:33 smithi001 ceph-mon[28991]: pgmap v57: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.7 KiB/s rd, 356 KiB/s wr, 73 op/s 2024-02-28T16:33:33.757 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:33 smithi001 ceph-mon[28991]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:33.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.767+0000 7f9d3c600700 1 -- 172.21.15.1:0/1396786648 wait complete. 2024-02-28T16:33:33.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.768+0000 7f9d3c600700 1 Processor -- start 2024-02-28T16:33:33.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.768+0000 7f9d3c600700 1 -- start start 2024-02-28T16:33:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.769+0000 7f9d3c600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34066c70 0x7f9d3410fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.769+0000 7f9d3c600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d34102430 0x7f9d341100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.769+0000 7f9d3c600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d341106b0 con 0x7f9d34066c70 2024-02-28T16:33:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.769+0000 7f9d3c600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d34110f50 con 0x7f9d34102430 2024-02-28T16:33:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.769+0000 7f9d39c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34066c70 0x7f9d3410fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.769+0000 7f9d39c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34066c70 0x7f9d3410fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52686/0 (socket says 172.21.15.1:52686) 2024-02-28T16:33:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.769+0000 7f9d39c00700 1 -- 172.21.15.1:0/3626133139 learned_addr learned my addr 172.21.15.1:0/3626133139 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:33:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.769+0000 7f9d39200700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d34102430 0x7f9d341100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.770+0000 7f9d39c00700 1 -- 172.21.15.1:0/3626133139 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d34102430 msgr2=0x7f9d341100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.770+0000 7f9d39c00700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d34102430 0x7f9d341100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.770+0000 7f9d39c00700 1 -- 172.21.15.1:0/3626133139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d20005040 con 0x7f9d34066c70 2024-02-28T16:33:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.770+0000 7f9d39c00700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34066c70 0x7f9d3410fb70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800c830 tx=0x7f9d28004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.771+0000 7f9d31600700 1 -- 172.21.15.1:0/3626133139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d280106f0 con 0x7f9d34066c70 2024-02-28T16:33:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.771+0000 7f9d31600700 1 -- 172.21.15.1:0/3626133139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d28010d20 con 0x7f9d34066c70 2024-02-28T16:33:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.771+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d34111230 con 0x7f9d34066c70 2024-02-28T16:33:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.771+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d34111770 con 0x7f9d34066c70 2024-02-28T16:33:33.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.772+0000 7f9d31600700 1 -- 172.21.15.1:0/3626133139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d280113a0 con 0x7f9d34066c70 2024-02-28T16:33:33.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.772+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3404bed0 con 0x7f9d34066c70 2024-02-28T16:33:33.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.774+0000 7f9d31600700 1 -- 172.21.15.1:0/3626133139 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f9d28012400 con 0x7f9d34066c70 2024-02-28T16:33:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.775+0000 7f9d31600700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f9d1c06d0f0 0x7f9d1c06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:33:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.775+0000 7f9d31600700 1 -- 172.21.15.1:0/3626133139 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f9d2808c540 con 0x7f9d34066c70 2024-02-28T16:33:33.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.775+0000 7f9d39200700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f9d1c06d0f0 0x7f9d1c06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:33:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.776+0000 7f9d39200700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f9d1c06d0f0 0x7f9d1c06f5a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000b840 tx=0x7f9d20009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:33:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:33.780+0000 7f9d31600700 1 -- 172.21.15.1:0/3626133139 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f9d2805a150 con 0x7f9d34066c70 2024-02-28T16:33:34.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.032+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d34111c60 con 0x7f9d1c06d0f0 2024-02-28T16:33:34.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.035+0000 7f9d31600700 1 -- 172.21.15.1:0/3626133139 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f9d34111c60 con 0x7f9d1c06d0f0 2024-02-28T16:33:34.024 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:33:34.024 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:33:34.024 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:33:34.024 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:33:34.025 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:33:34.025 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr" 2024-02-28T16:33:34.025 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:33:34.025 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "2/2 daemons upgraded", 2024-02-28T16:33:34.025 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading prometheus daemons", 2024-02-28T16:33:34.025 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:33:34.025 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:33:34.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f9d1c06d0f0 msgr2=0x7f9d1c06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:34.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f9d1c06d0f0 0x7f9d1c06f5a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000b840 tx=0x7f9d20009e30 comp rx=0 tx=0).stop 2024-02-28T16:33:34.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34066c70 msgr2=0x7f9d3410fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:33:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34066c70 0x7f9d3410fb70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800c830 tx=0x7f9d28004470 comp rx=0 tx=0).stop 2024-02-28T16:33:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 shutdown_connections 2024-02-28T16:33:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f9d1c06d0f0 0x7f9d1c06f5a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d34066c70 0x7f9d3410fb70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 --2- 172.21.15.1:0/3626133139 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d34102430 0x7f9d341100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:33:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 >> 172.21.15.1:0/3626133139 conn(0x7f9d340f7500 msgr2=0x7f9d340f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:33:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 shutdown_connections 2024-02-28T16:33:34.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:33:34.038+0000 7f9d3c600700 1 -- 172.21.15.1:0/3626133139 wait complete. 2024-02-28T16:33:34.548 DEBUG:teuthology.parallel:result is None 2024-02-28T16:33:34.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-02-28T16:33:34.644 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:34 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/4128501651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:34.644 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:34 smithi001 ceph-mon[28991]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:34.676 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-02-28T16:33:34.677 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-02-28T16:33:34.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:34 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/4128501651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:34 smithi066 ceph-mon[36236]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:33:34.847 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-02-28T16:33:34.848 DEBUG:teuthology.parallel:result is None 2024-02-28T16:33:35.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:35 smithi001 ceph-mon[28991]: pgmap v58: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.7 KiB/s rd, 356 KiB/s wr, 69 op/s 2024-02-28T16:33:35.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:35 smithi066 ceph-mon[36236]: pgmap v58: 65 pgs: 65 active+clean; 327 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 8.7 KiB/s rd, 356 KiB/s wr, 69 op/s 2024-02-28T16:33:36.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:36 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:36.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:36 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:36.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:36 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:36.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:36 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:37.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:37 smithi001 ceph-mon[28991]: pgmap v59: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 14 KiB/s rd, 502 KiB/s wr, 95 op/s 2024-02-28T16:33:37.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:37 smithi066 ceph-mon[36236]: pgmap v59: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 14 KiB/s rd, 502 KiB/s wr, 95 op/s 2024-02-28T16:33:38.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:38 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:38.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:38 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:38.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:38 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:38.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:38 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:39.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:39 smithi001 ceph-mon[28991]: pgmap v60: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 10 KiB/s rd, 335 KiB/s wr, 61 op/s 2024-02-28T16:33:39.921 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:39 smithi066 ceph-mon[36236]: pgmap v60: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 10 KiB/s rd, 335 KiB/s wr, 61 op/s 2024-02-28T16:33:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:41 smithi066 ceph-mon[36236]: pgmap v61: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 11 KiB/s rd, 337 KiB/s wr, 63 op/s 2024-02-28T16:33:42.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:41 smithi001 ceph-mon[28991]: pgmap v61: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 11 KiB/s rd, 337 KiB/s wr, 63 op/s 2024-02-28T16:33:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:43 smithi066 ceph-mon[36236]: pgmap v62: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 6.0 KiB/s rd, 147 KiB/s wr, 33 op/s 2024-02-28T16:33:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:43 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:43 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:43 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:33:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:43 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:33:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:43 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:44.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:43 smithi001 ceph-mon[28991]: pgmap v62: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 6.0 KiB/s rd, 147 KiB/s wr, 33 op/s 2024-02-28T16:33:44.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:43 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:44.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:43 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:44.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:43 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:33:44.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:43 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:33:44.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:43 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.132 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.133 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:44 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:33:45.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-28T16:33:45.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:33:45.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:44 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:33:45.991 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:45 smithi001 ceph-mon[28991]: pgmap v63: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 6.0 KiB/s rd, 147 KiB/s wr, 28 op/s 2024-02-28T16:33:45.991 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:33:46.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:45 smithi066 ceph-mon[36236]: pgmap v63: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 6.0 KiB/s rd, 147 KiB/s wr, 28 op/s 2024-02-28T16:33:46.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:33:47.117 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:46 smithi001 ceph-mon[28991]: Upgrade: Updating alertmanager.smithi001 2024-02-28T16:33:47.117 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:46 smithi001 ceph-mon[28991]: Deploying daemon alertmanager.smithi001 on smithi001 2024-02-28T16:33:47.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:46 smithi066 ceph-mon[36236]: Upgrade: Updating alertmanager.smithi001 2024-02-28T16:33:47.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:46 smithi066 ceph-mon[36236]: Deploying daemon alertmanager.smithi001 on smithi001 2024-02-28T16:33:48.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:47 smithi001 ceph-mon[28991]: pgmap v64: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 6.0 KiB/s rd, 148 KiB/s wr, 28 op/s 2024-02-28T16:33:48.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:47 smithi066 ceph-mon[36236]: pgmap v64: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 6.0 KiB/s rd, 148 KiB/s wr, 28 op/s 2024-02-28T16:33:50.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:49 smithi066 ceph-mon[36236]: pgmap v65: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 511 B/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-28T16:33:50.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:49 smithi001 ceph-mon[28991]: pgmap v65: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 511 B/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-28T16:33:52.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:51 smithi066 ceph-mon[36236]: pgmap v66: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 511 B/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-28T16:33:52.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:51 smithi001 ceph-mon[28991]: pgmap v66: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 511 B/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-28T16:33:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:53 smithi001 ceph-mon[28991]: pgmap v67: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:33:54.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:53 smithi066 ceph-mon[36236]: pgmap v67: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:33:55.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:54 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:55.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:54 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:55.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:54 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:33:55.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:54 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:55.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:54 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:33:55.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:54 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:33:56.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:55 smithi066 ceph-mon[36236]: pgmap v68: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:33:56.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:55 smithi001 ceph-mon[28991]: pgmap v68: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:33:58.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:33:58 smithi066 ceph-mon[36236]: pgmap v69: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:33:58.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:33:58 smithi001 ceph-mon[28991]: pgmap v69: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-28T16:34:00.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:00 smithi066 ceph-mon[36236]: pgmap v70: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:00.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:00 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:34:00.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:00 smithi001 ceph-mon[28991]: pgmap v70: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:00.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:00 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:34:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:02 smithi066 ceph-mon[36236]: pgmap v71: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:02 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:02 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:02.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:02 smithi001 ceph-mon[28991]: pgmap v71: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:02.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:02 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:02.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:02 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:04.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.208+0000 7fb934800700 1 -- 172.21.15.1:0/3206697088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 msgr2=0x7fb92c070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:04.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.209+0000 7fb934800700 1 --2- 172.21.15.1:0/3206697088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 0x7fb92c070cb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb91c004940 tx=0x7fb91c009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:04.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.209+0000 7fb934800700 1 -- 172.21.15.1:0/3206697088 shutdown_connections 2024-02-28T16:34:04.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.209+0000 7fb934800700 1 --2- 172.21.15.1:0/3206697088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 0x7fb92c070cb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.209+0000 7fb934800700 1 --2- 172.21.15.1:0/3206697088 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb92c071e90 0x7fb92c070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.209+0000 7fb934800700 1 -- 172.21.15.1:0/3206697088 >> 172.21.15.1:0/3206697088 conn(0x7fb92c0f8620 msgr2=0x7fb92c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.209+0000 7fb934800700 1 -- 172.21.15.1:0/3206697088 shutdown_connections 2024-02-28T16:34:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.209+0000 7fb934800700 1 -- 172.21.15.1:0/3206697088 wait complete. 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.210+0000 7fb934800700 1 Processor -- start 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.210+0000 7fb934800700 1 -- start start 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.210+0000 7fb934800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 0x7fb92c10fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.210+0000 7fb934800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb92c071e90 0x7fb92c1100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.210+0000 7fb934800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c1106a0 con 0x7fb92c070840 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.210+0000 7fb934800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c110f40 con 0x7fb92c071e90 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.210+0000 7fb931e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 0x7fb92c10fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.211+0000 7fb931e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 0x7fb92c10fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34972/0 (socket says 172.21.15.1:34972) 2024-02-28T16:34:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.211+0000 7fb931e00700 1 -- 172.21.15.1:0/1999238743 learned_addr learned my addr 172.21.15.1:0/1999238743 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.211+0000 7fb931400700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb92c071e90 0x7fb92c1100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.211+0000 7fb931e00700 1 -- 172.21.15.1:0/1999238743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb92c071e90 msgr2=0x7fb92c1100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.211+0000 7fb931e00700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb92c071e90 0x7fb92c1100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.211+0000 7fb931e00700 1 -- 172.21.15.1:0/1999238743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c005040 con 0x7fb92c070840 2024-02-28T16:34:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.211+0000 7fb931e00700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 0x7fb92c10fb60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb920002a50 tx=0x7fb9200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.212+0000 7fb926000700 1 -- 172.21.15.1:0/1999238743 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb920007cc0 con 0x7fb92c070840 2024-02-28T16:34:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.212+0000 7fb926000700 1 -- 172.21.15.1:0/1999238743 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb92000f040 con 0x7fb92c070840 2024-02-28T16:34:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.212+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92c111220 con 0x7fb92c070840 2024-02-28T16:34:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.212+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92c111760 con 0x7fb92c070840 2024-02-28T16:34:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.212+0000 7fb926000700 1 -- 172.21.15.1:0/1999238743 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb92000db70 con 0x7fb92c070840 2024-02-28T16:34:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.213+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9100052e0 con 0x7fb92c070840 2024-02-28T16:34:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.214+0000 7fb926000700 1 -- 172.21.15.1:0/1999238743 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7fb920009070 con 0x7fb92c070840 2024-02-28T16:34:04.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.215+0000 7fb926000700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb91406d0f0 0x7fb91406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:04.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.215+0000 7fb926000700 1 -- 172.21.15.1:0/1999238743 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fb92008bc40 con 0x7fb92c070840 2024-02-28T16:34:04.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.216+0000 7fb931400700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb91406d0f0 0x7fb91406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.216+0000 7fb931400700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb91406d0f0 0x7fb91406f5a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00b020 tx=0x7fb91c009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.221+0000 7fb926000700 1 -- 172.21.15.1:0/1999238743 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fb920059870 con 0x7fb92c070840 2024-02-28T16:34:04.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.480+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb910000bf0 con 0x7fb91406d0f0 2024-02-28T16:34:04.469 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:04 smithi001 ceph-mon[28991]: pgmap v72: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:04.469 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:04 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:04.469 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:04 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:04.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.481+0000 7fb926000700 1 -- 172.21.15.1:0/1999238743 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fb910000bf0 con 0x7fb91406d0f0 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.485+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb91406d0f0 msgr2=0x7fb91406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.485+0000 7fb934800700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb91406d0f0 0x7fb91406f5a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00b020 tx=0x7fb91c009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.485+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 msgr2=0x7fb92c10fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.485+0000 7fb934800700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 0x7fb92c10fb60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb920002a50 tx=0x7fb9200046b0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.486+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 shutdown_connections 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.486+0000 7fb934800700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb91406d0f0 0x7fb91406f5a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.486+0000 7fb934800700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c070840 0x7fb92c10fb60 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.486+0000 7fb934800700 1 --2- 172.21.15.1:0/1999238743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb92c071e90 0x7fb92c1100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.486+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 >> 172.21.15.1:0/1999238743 conn(0x7fb92c0f8620 msgr2=0x7fb92c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.486+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 shutdown_connections 2024-02-28T16:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.486+0000 7fb934800700 1 -- 172.21.15.1:0/1999238743 wait complete. 2024-02-28T16:34:04.492 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:34:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.665+0000 7f32af200700 1 -- 172.21.15.1:0/3927118158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fbe40 msgr2=0x7f32a80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.665+0000 7f32af200700 1 --2- 172.21.15.1:0/3927118158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fbe40 0x7f32a80fc250 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3298009ce0 tx=0x7f3298004d10 comp rx=0 tx=0).stop 2024-02-28T16:34:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.665+0000 7f32af200700 1 -- 172.21.15.1:0/3927118158 shutdown_connections 2024-02-28T16:34:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.665+0000 7f32af200700 1 --2- 172.21.15.1:0/3927118158 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a80fcfb0 0x7f32a80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.665+0000 7f32af200700 1 --2- 172.21.15.1:0/3927118158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fbe40 0x7f32a80fc250 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.665+0000 7f32af200700 1 -- 172.21.15.1:0/3927118158 >> 172.21.15.1:0/3927118158 conn(0x7f32a80f7520 msgr2=0x7f32a80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.665+0000 7f32af200700 1 -- 172.21.15.1:0/3927118158 shutdown_connections 2024-02-28T16:34:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.665+0000 7f32af200700 1 -- 172.21.15.1:0/3927118158 wait complete. 2024-02-28T16:34:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.666+0000 7f32af200700 1 Processor -- start 2024-02-28T16:34:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.666+0000 7f32af200700 1 -- start start 2024-02-28T16:34:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.666+0000 7f32af200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a80fbe40 0x7f32a810fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.666+0000 7f32af200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fcfb0 0x7f32a8110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.666+0000 7f32af200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8111050 con 0x7f32a80fcfb0 2024-02-28T16:34:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.666+0000 7f32af200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a81111c0 con 0x7f32a80fbe40 2024-02-28T16:34:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.667+0000 7f329f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fcfb0 0x7f32a8110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.667+0000 7f329f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fcfb0 0x7f32a8110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35000/0 (socket says 172.21.15.1:35000) 2024-02-28T16:34:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.667+0000 7f329f400700 1 -- 172.21.15.1:0/2112845285 learned_addr learned my addr 172.21.15.1:0/2112845285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.667+0000 7f32a7e00700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a80fbe40 0x7f32a810fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.667+0000 7f329f400700 1 -- 172.21.15.1:0/2112845285 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a80fbe40 msgr2=0x7f32a810fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.667+0000 7f329f400700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a80fbe40 0x7f32a810fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.667+0000 7f329f400700 1 -- 172.21.15.1:0/2112845285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3298009940 con 0x7f32a80fcfb0 2024-02-28T16:34:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.667+0000 7f329f400700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fcfb0 0x7f32a8110290 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f329000b790 tx=0x7f3290009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.668+0000 7f32a4c00700 1 -- 172.21.15.1:0/2112845285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329000f770 con 0x7f32a80fcfb0 2024-02-28T16:34:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.668+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a8111440 con 0x7f32a80fcfb0 2024-02-28T16:34:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.668+0000 7f32a4c00700 1 -- 172.21.15.1:0/2112845285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f329000fda0 con 0x7f32a80fcfb0 2024-02-28T16:34:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.668+0000 7f32a4c00700 1 -- 172.21.15.1:0/2112845285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329000d310 con 0x7f32a80fcfb0 2024-02-28T16:34:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.668+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a8111980 con 0x7f32a80fcfb0 2024-02-28T16:34:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.669+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a804bed0 con 0x7f32a80fcfb0 2024-02-28T16:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.670+0000 7f32a4c00700 1 -- 172.21.15.1:0/2112845285 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f3290010420 con 0x7f32a80fcfb0 2024-02-28T16:34:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.671+0000 7f32a4c00700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f328c06d0f0 0x7f328c06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.671+0000 7f32a4c00700 1 -- 172.21.15.1:0/2112845285 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f329008c5e0 con 0x7f32a80fcfb0 2024-02-28T16:34:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.671+0000 7f32a7e00700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f328c06d0f0 0x7f328c06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.672+0000 7f32a7e00700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f328c06d0f0 0x7f328c06f5a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f329800b740 tx=0x7f3298004d10 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:04.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.677+0000 7f32a4c00700 1 -- 172.21.15.1:0/2112845285 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f32900566c0 con 0x7f32a80fcfb0 2024-02-28T16:34:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:04 smithi066 ceph-mon[36236]: pgmap v72: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:04 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:04 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.945+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32a8111ee0 con 0x7f328c06d0f0 2024-02-28T16:34:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.946+0000 7f32a4c00700 1 -- 172.21.15.1:0/2112845285 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f32980097b0 con 0x7f328c06d0f0 2024-02-28T16:34:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f328c06d0f0 msgr2=0x7f328c06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f328c06d0f0 0x7f328c06f5a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f329800b740 tx=0x7f3298004d10 comp rx=0 tx=0).stop 2024-02-28T16:34:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fcfb0 msgr2=0x7f32a8110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fcfb0 0x7f32a8110290 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f329000b790 tx=0x7f3290009250 comp rx=0 tx=0).stop 2024-02-28T16:34:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 shutdown_connections 2024-02-28T16:34:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a80fbe40 0x7f32a810fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f328c06d0f0 0x7f328c06f5a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 --2- 172.21.15.1:0/2112845285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a80fcfb0 0x7f32a8110290 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 >> 172.21.15.1:0/2112845285 conn(0x7f32a80f7520 msgr2=0x7f32a8065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 shutdown_connections 2024-02-28T16:34:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:04.949+0000 7f32af200700 1 -- 172.21.15.1:0/2112845285 wait complete. 2024-02-28T16:34:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.104+0000 7fac99200700 1 -- 172.21.15.1:0/1754321860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94103540 msgr2=0x7fac94103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.104+0000 7fac99200700 1 --2- 172.21.15.1:0/1754321860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94103540 0x7fac94103950 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac84004940 tx=0x7fac84009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.104+0000 7fac99200700 1 -- 172.21.15.1:0/1754321860 shutdown_connections 2024-02-28T16:34:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.104+0000 7fac99200700 1 --2- 172.21.15.1:0/1754321860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac94104750 0x7fac94104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.104+0000 7fac99200700 1 --2- 172.21.15.1:0/1754321860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94103540 0x7fac94103950 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.104+0000 7fac99200700 1 -- 172.21.15.1:0/1754321860 >> 172.21.15.1:0/1754321860 conn(0x7fac940727d0 msgr2=0x7fac94074c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.104+0000 7fac99200700 1 -- 172.21.15.1:0/1754321860 shutdown_connections 2024-02-28T16:34:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.104+0000 7fac99200700 1 -- 172.21.15.1:0/1754321860 wait complete. 2024-02-28T16:34:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.105+0000 7fac99200700 1 Processor -- start 2024-02-28T16:34:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.105+0000 7fac99200700 1 -- start start 2024-02-28T16:34:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac99200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac94103540 0x7fac94113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac99200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94104750 0x7fac941144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac99200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94114aa0 con 0x7fac94104750 2024-02-28T16:34:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac99200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94115340 con 0x7fac94103540 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac91400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94104750 0x7fac941144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac91400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94104750 0x7fac941144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35018/0 (socket says 172.21.15.1:35018) 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac91400700 1 -- 172.21.15.1:0/1509759934 learned_addr learned my addr 172.21.15.1:0/1509759934 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac91e00700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac94103540 0x7fac94113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac91400700 1 -- 172.21.15.1:0/1509759934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac94103540 msgr2=0x7fac94113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac91400700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac94103540 0x7fac94113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.106+0000 7fac91400700 1 -- 172.21.15.1:0/1509759934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac84005040 con 0x7fac94104750 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.107+0000 7fac91400700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94104750 0x7fac941144a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00c830 tx=0x7fac7c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.107+0000 7fac8a000700 1 -- 172.21.15.1:0/1509759934 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac7c00a760 con 0x7fac94104750 2024-02-28T16:34:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.107+0000 7fac8a000700 1 -- 172.21.15.1:0/1509759934 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac7c00ad90 con 0x7fac94104750 2024-02-28T16:34:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.107+0000 7fac8a000700 1 -- 172.21.15.1:0/1509759934 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac7c0113a0 con 0x7fac94104750 2024-02-28T16:34:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.107+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac94115620 con 0x7fac94104750 2024-02-28T16:34:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.107+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac94115b60 con 0x7fac94104750 2024-02-28T16:34:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.108+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9404bed0 con 0x7fac94104750 2024-02-28T16:34:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.109+0000 7fac8a000700 1 -- 172.21.15.1:0/1509759934 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7fac7c012400 con 0x7fac94104750 2024-02-28T16:34:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.110+0000 7fac8a000700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fac7806d0f0 0x7fac7806f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.110+0000 7fac8a000700 1 -- 172.21.15.1:0/1509759934 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fac7c08c530 con 0x7fac94104750 2024-02-28T16:34:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.111+0000 7fac91e00700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fac7806d0f0 0x7fac7806f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:05.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.111+0000 7fac91e00700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fac7806d0f0 0x7fac7806f5a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac8400b840 tx=0x7fac84009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:05.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.115+0000 7fac8a000700 1 -- 172.21.15.1:0/1509759934 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fac7c05a1c0 con 0x7fac94104750 2024-02-28T16:34:05.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.352+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fac94116060 con 0x7fac7806d0f0 2024-02-28T16:34:05.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.366+0000 7fac8a000700 1 -- 172.21.15.1:0/1509759934 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fac94116060 con 0x7fac7806d0f0 2024-02-28T16:34:05.355 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:34:05.355 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (12s) 3s ago 14m 14.0M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:34:05.355 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (14m) 3s ago 14m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (13m) 62s ago 13m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (3m) 3s ago 14m 71.5M - 9.4.7 954c08fa6188 1010f22bd792 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (7m) 3s ago 7m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (7m) 3s ago 7m 13.2M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (7m) 62s ago 7m 13.0M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (7m) 62s ago 7m 152M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (2m) 3s ago 16m 500M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (99s) 62s ago 13m 446M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (16m) 3s ago 16m 54.0M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (13m) 62s ago 13m 42.5M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (77s) 3s ago 14m 7587k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (69s) 62s ago 13m 3053k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:34:05.356 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (12m) 3s ago 12m 206M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:34:05.357 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (11m) 3s ago 11m 207M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:34:05.357 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (10m) 3s ago 10m 167M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:34:05.357 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (10m) 62s ago 10m 215M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:34:05.357 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (9m) 62s ago 9m 197M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:34:05.357 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (9m) 62s ago 9m 181M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:34:05.357 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (40s) 3s ago 13m 44.9M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:34:05.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fac7806d0f0 msgr2=0x7fac7806f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fac7806d0f0 0x7fac7806f5a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac8400b840 tx=0x7fac84009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94104750 msgr2=0x7fac941144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94104750 0x7fac941144a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00c830 tx=0x7fac7c004470 comp rx=0 tx=0).stop 2024-02-28T16:34:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 shutdown_connections 2024-02-28T16:34:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac94103540 0x7fac94113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fac7806d0f0 0x7fac7806f5a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 --2- 172.21.15.1:0/1509759934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac94104750 0x7fac941144a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.369+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 >> 172.21.15.1:0/1509759934 conn(0x7fac940727d0 msgr2=0x7fac94107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.370+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 shutdown_connections 2024-02-28T16:34:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.370+0000 7fac99200700 1 -- 172.21.15.1:0/1509759934 wait complete. 2024-02-28T16:34:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.525+0000 7f0a26c00700 1 -- 172.21.15.1:0/4272043750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 msgr2=0x7f0a20070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.525+0000 7f0a26c00700 1 --2- 172.21.15.1:0/4272043750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 0x7f0a20070d10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0a08004940 tx=0x7f0a08009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:05.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.525+0000 7f0a26c00700 1 -- 172.21.15.1:0/4272043750 shutdown_connections 2024-02-28T16:34:05.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.525+0000 7f0a26c00700 1 --2- 172.21.15.1:0/4272043750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 0x7f0a20070d10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.525+0000 7f0a26c00700 1 --2- 172.21.15.1:0/4272043750 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a20071f10 0x7f0a20070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.525+0000 7f0a26c00700 1 -- 172.21.15.1:0/4272043750 >> 172.21.15.1:0/4272043750 conn(0x7f0a200f8620 msgr2=0x7f0a200faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:05.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.526+0000 7f0a26c00700 1 -- 172.21.15.1:0/4272043750 shutdown_connections 2024-02-28T16:34:05.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.526+0000 7f0a26c00700 1 -- 172.21.15.1:0/4272043750 wait complete. 2024-02-28T16:34:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.527+0000 7f0a26c00700 1 Processor -- start 2024-02-28T16:34:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.527+0000 7f0a26c00700 1 -- start start 2024-02-28T16:34:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.527+0000 7f0a26c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 0x7f0a2010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.527+0000 7f0a26c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a20071f10 0x7f0a201100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.527+0000 7f0a26c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a201106b0 con 0x7f0a200708c0 2024-02-28T16:34:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.527+0000 7f0a26c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20110f50 con 0x7f0a20071f10 2024-02-28T16:34:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.528+0000 7f0a1fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 0x7f0a2010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.528+0000 7f0a1fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 0x7f0a2010fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35042/0 (socket says 172.21.15.1:35042) 2024-02-28T16:34:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.528+0000 7f0a1fe00700 1 -- 172.21.15.1:0/4044167009 learned_addr learned my addr 172.21.15.1:0/4044167009 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.528+0000 7f0a17e00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a20071f10 0x7f0a201100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.528+0000 7f0a1fe00700 1 -- 172.21.15.1:0/4044167009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a20071f10 msgr2=0x7f0a201100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.528+0000 7f0a1fe00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a20071f10 0x7f0a201100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.528+0000 7f0a1fe00700 1 -- 172.21.15.1:0/4044167009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a08005040 con 0x7f0a200708c0 2024-02-28T16:34:05.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.529+0000 7f0a1fe00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 0x7f0a2010fb70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000c830 tx=0x7f0a10004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:05.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.529+0000 7f0a1cc00700 1 -- 172.21.15.1:0/4044167009 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a1000a760 con 0x7f0a200708c0 2024-02-28T16:34:05.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.529+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a20111230 con 0x7f0a200708c0 2024-02-28T16:34:05.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.529+0000 7f0a1cc00700 1 -- 172.21.15.1:0/4044167009 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a1000ad90 con 0x7f0a200708c0 2024-02-28T16:34:05.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.529+0000 7f0a1cc00700 1 -- 172.21.15.1:0/4044167009 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a100113a0 con 0x7f0a200708c0 2024-02-28T16:34:05.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.529+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a20111770 con 0x7f0a200708c0 2024-02-28T16:34:05.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.531+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2004bed0 con 0x7f0a200708c0 2024-02-28T16:34:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.532+0000 7f0a1cc00700 1 -- 172.21.15.1:0/4044167009 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f0a1000a8c0 con 0x7f0a200708c0 2024-02-28T16:34:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.533+0000 7f0a1cc00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f0a0406d0f0 0x7f0a0406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.533+0000 7f0a1cc00700 1 -- 172.21.15.1:0/4044167009 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f0a1008c410 con 0x7f0a200708c0 2024-02-28T16:34:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.533+0000 7f0a17e00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f0a0406d0f0 0x7f0a0406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.534+0000 7f0a17e00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f0a0406d0f0 0x7f0a0406f5a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800b3b0 tx=0x7f0a08009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.538+0000 7f0a1cc00700 1 -- 172.21.15.1:0/4044167009 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f0a1005a160 con 0x7f0a200708c0 2024-02-28T16:34:05.678 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:05 smithi001 ceph-mon[28991]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:05.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:05 smithi066 ceph-mon[36236]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.854+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0a201113c0 con 0x7f0a200708c0 2024-02-28T16:34:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.855+0000 7f0a1cc00700 1 -- 172.21.15.1:0/4044167009 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f0a201113c0 con 0x7f0a200708c0 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:34:05.844 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 12, 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:34:05.845 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:34:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f0a0406d0f0 msgr2=0x7f0a0406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f0a0406d0f0 0x7f0a0406f5a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800b3b0 tx=0x7f0a08009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 msgr2=0x7f0a2010fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 0x7f0a2010fb70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000c830 tx=0x7f0a10004470 comp rx=0 tx=0).stop 2024-02-28T16:34:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 shutdown_connections 2024-02-28T16:34:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f0a0406d0f0 0x7f0a0406f5a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a200708c0 0x7f0a2010fb70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 --2- 172.21.15.1:0/4044167009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a20071f10 0x7f0a201100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 >> 172.21.15.1:0/4044167009 conn(0x7f0a200f8620 msgr2=0x7f0a20102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.859+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 shutdown_connections 2024-02-28T16:34:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:05.860+0000 7f0a26c00700 1 -- 172.21.15.1:0/4044167009 wait complete. 2024-02-28T16:34:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.009+0000 7fa328800700 1 -- 172.21.15.1:0/2235640998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fcef0 msgr2=0x7fa3200fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.009+0000 7fa328800700 1 --2- 172.21.15.1:0/2235640998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fcef0 0x7fa3200fd320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa314004940 tx=0x7fa314009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.010+0000 7fa328800700 1 -- 172.21.15.1:0/2235640998 shutdown_connections 2024-02-28T16:34:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.010+0000 7fa328800700 1 --2- 172.21.15.1:0/2235640998 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3200fe0e0 0x7fa3200fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.010+0000 7fa328800700 1 --2- 172.21.15.1:0/2235640998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fcef0 0x7fa3200fd320 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.010+0000 7fa328800700 1 -- 172.21.15.1:0/2235640998 >> 172.21.15.1:0/2235640998 conn(0x7fa3200f8620 msgr2=0x7fa3200faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.010+0000 7fa328800700 1 -- 172.21.15.1:0/2235640998 shutdown_connections 2024-02-28T16:34:05.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.010+0000 7fa328800700 1 -- 172.21.15.1:0/2235640998 wait complete. 2024-02-28T16:34:05.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.011+0000 7fa328800700 1 Processor -- start 2024-02-28T16:34:05.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.011+0000 7fa328800700 1 -- start start 2024-02-28T16:34:05.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.011+0000 7fa328800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3200fcef0 0x7fa32010d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:05.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.011+0000 7fa328800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fe0e0 0x7fa32010ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:05.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.011+0000 7fa328800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32010e560 con 0x7fa3200fe0e0 2024-02-28T16:34:05.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.011+0000 7fa328800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32010eda0 con 0x7fa3200fcef0 2024-02-28T16:34:06.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.012+0000 7fa325400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fe0e0 0x7fa32010ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.012+0000 7fa325400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fe0e0 0x7fa32010ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35052/0 (socket says 172.21.15.1:35052) 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.012+0000 7fa325400700 1 -- 172.21.15.1:0/59573612 learned_addr learned my addr 172.21.15.1:0/59573612 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.012+0000 7fa325e00700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3200fcef0 0x7fa32010d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.012+0000 7fa325400700 1 -- 172.21.15.1:0/59573612 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3200fcef0 msgr2=0x7fa32010d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.012+0000 7fa325400700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3200fcef0 0x7fa32010d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.012+0000 7fa325400700 1 -- 172.21.15.1:0/59573612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa314005040 con 0x7fa3200fe0e0 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.012+0000 7fa325400700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fe0e0 0x7fa32010ded0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa31000b790 tx=0x7fa310009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.013+0000 7fa31a000700 1 -- 172.21.15.1:0/59573612 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa31000f770 con 0x7fa3200fe0e0 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.013+0000 7fa31a000700 1 -- 172.21.15.1:0/59573612 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa31000fda0 con 0x7fa3200fe0e0 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.013+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32010f080 con 0x7fa3200fe0e0 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.013+0000 7fa31a000700 1 -- 172.21.15.1:0/59573612 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa31000d310 con 0x7fa3200fe0e0 2024-02-28T16:34:06.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.013+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32010f5c0 con 0x7fa3200fe0e0 2024-02-28T16:34:06.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.014+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32004bed0 con 0x7fa3200fe0e0 2024-02-28T16:34:06.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.015+0000 7fa31a000700 1 -- 172.21.15.1:0/59573612 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7fa310010420 con 0x7fa3200fe0e0 2024-02-28T16:34:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.016+0000 7fa31a000700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa30806d0f0 0x7fa30806f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.016+0000 7fa31a000700 1 -- 172.21.15.1:0/59573612 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fa31008c520 con 0x7fa3200fe0e0 2024-02-28T16:34:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.016+0000 7fa325e00700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa30806d0f0 0x7fa30806f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.017+0000 7fa325e00700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa30806d0f0 0x7fa30806f5a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa31400b3b0 tx=0x7fa314009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.021+0000 7fa31a000700 1 -- 172.21.15.1:0/59573612 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fa31005a1b0 con 0x7fa3200fe0e0 2024-02-28T16:34:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.274+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa32010fb70 con 0x7fa30806d0f0 2024-02-28T16:34:06.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.276+0000 7fa31a000700 1 -- 172.21.15.1:0/59573612 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fa32010fb70 con 0x7fa30806d0f0 2024-02-28T16:34:06.264 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:34:06.264 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:34:06.264 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:34:06.264 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-28T16:34:06.264 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:34:06.264 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr" 2024-02-28T16:34:06.264 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:34:06.265 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "2/2 daemons upgraded", 2024-02-28T16:34:06.265 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading alertmanager daemons", 2024-02-28T16:34:06.265 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:34:06.265 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:34:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa30806d0f0 msgr2=0x7fa30806f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa30806d0f0 0x7fa30806f5a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa31400b3b0 tx=0x7fa314009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fe0e0 msgr2=0x7fa32010ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:06.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fe0e0 0x7fa32010ded0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa31000b790 tx=0x7fa310009250 comp rx=0 tx=0).stop 2024-02-28T16:34:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 shutdown_connections 2024-02-28T16:34:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3200fcef0 0x7fa32010d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa30806d0f0 0x7fa30806f5a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 --2- 172.21.15.1:0/59573612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3200fe0e0 0x7fa32010ded0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.279+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 >> 172.21.15.1:0/59573612 conn(0x7fa3200f8620 msgr2=0x7fa3201013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.280+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 shutdown_connections 2024-02-28T16:34:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:06.280+0000 7fa328800700 1 -- 172.21.15.1:0/59573612 wait complete. 2024-02-28T16:34:06.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:06 smithi001 ceph-mon[28991]: pgmap v73: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:06.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:06 smithi001 ceph-mon[28991]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:06.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:06 smithi001 ceph-mon[28991]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:06.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:06 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/4044167009' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:06 smithi066 ceph-mon[36236]: pgmap v73: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:06 smithi066 ceph-mon[36236]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:06 smithi066 ceph-mon[36236]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:06.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:06 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/4044167009' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:07.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:07 smithi066 ceph-mon[36236]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:07.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:07 smithi001 ceph-mon[28991]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:08.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:08 smithi066 ceph-mon[36236]: pgmap v74: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:08.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:08 smithi001 ceph-mon[28991]: pgmap v74: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:10.460 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: pgmap v75: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:10 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: pgmap v75: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:10.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:10.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:10 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:12.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:12 smithi066 ceph-mon[36236]: pgmap v76: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:12.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:12 smithi001 ceph-mon[28991]: pgmap v76: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:13.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:13 smithi001 ceph-mon[28991]: Upgrade: Updating grafana.smithi001 2024-02-28T16:34:13.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:13 smithi001 ceph-mon[28991]: Deploying daemon grafana.smithi001 on smithi001 2024-02-28T16:34:13.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:13 smithi066 ceph-mon[36236]: Upgrade: Updating grafana.smithi001 2024-02-28T16:34:13.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:13 smithi066 ceph-mon[36236]: Deploying daemon grafana.smithi001 on smithi001 2024-02-28T16:34:14.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:14 smithi001 ceph-mon[28991]: pgmap v77: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:14.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:14 smithi066 ceph-mon[36236]: pgmap v77: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:15.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:15 smithi066 ceph-mon[36236]: pgmap v78: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:15.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:34:15.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:15 smithi001 ceph-mon[28991]: pgmap v78: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:15.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:34:18.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:17 smithi001 ceph-mon[28991]: pgmap v79: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:18.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:17 smithi066 ceph-mon[36236]: pgmap v79: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:19.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:19 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:19.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:19 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:19.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:19 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:19.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:19 smithi001 ceph-mon[28991]: pgmap v80: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:19.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:19 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:19.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:19 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:19.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:19 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:19.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:19 smithi066 ceph-mon[36236]: pgmap v80: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:22.123 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:21 smithi001 ceph-mon[28991]: pgmap v81: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:22.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:21 smithi066 ceph-mon[36236]: pgmap v81: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:24.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:23 smithi001 ceph-mon[28991]: pgmap v82: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:24.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:23 smithi066 ceph-mon[36236]: pgmap v82: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:25.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:25 smithi001 ceph-mon[28991]: pgmap v83: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:25.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:25 smithi066 ceph-mon[36236]: pgmap v83: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:27.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:27 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:27.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:27 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:27.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:27 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:27.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:27 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:28.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:28 smithi066 ceph-mon[36236]: pgmap v84: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:28.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:28 smithi001 ceph-mon[28991]: pgmap v84: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:29.597 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:29 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:29.597 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:29 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:29.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:29 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:29.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:29 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:30.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:30 smithi001 ceph-mon[28991]: pgmap v85: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:30.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:30 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:34:30.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:30 smithi066 ceph-mon[36236]: pgmap v85: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:30 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:34:32.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:32 smithi066 ceph-mon[36236]: pgmap v86: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:32.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:32 smithi001 ceph-mon[28991]: pgmap v86: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: pgmap v87: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:34:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:34:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:34 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: pgmap v87: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:34:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:34:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:34.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:34 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:35.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: Upgrade: Finalizing container_image settings 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-28T16:34:35.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-28T16:34:35.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: Upgrade: Complete! 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:35.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:35.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:34:35.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:34:35.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:35.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:35 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:35.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-28T16:34:35.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: Upgrade: Finalizing container_image settings 2024-02-28T16:34:35.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: Upgrade: Complete! 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:34:35.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:35.775 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:35 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:34:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.451+0000 7f0012a00700 1 -- 172.21.15.1:0/1556291691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c0ff150 msgr2=0x7f000c0ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.451+0000 7f0012a00700 1 --2- 172.21.15.1:0/1556291691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c0ff150 0x7f000c0ff560 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efffc004940 tx=0x7efffc009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.451+0000 7f0012a00700 1 -- 172.21.15.1:0/1556291691 shutdown_connections 2024-02-28T16:34:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.451+0000 7f0012a00700 1 --2- 172.21.15.1:0/1556291691 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f000c100360 0x7f000c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.451+0000 7f0012a00700 1 --2- 172.21.15.1:0/1556291691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c0ff150 0x7f000c0ff560 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.451+0000 7f0012a00700 1 -- 172.21.15.1:0/1556291691 >> 172.21.15.1:0/1556291691 conn(0x7f000c0fa920 msgr2=0x7f000c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.452+0000 7f0012a00700 1 -- 172.21.15.1:0/1556291691 shutdown_connections 2024-02-28T16:34:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.452+0000 7f0012a00700 1 -- 172.21.15.1:0/1556291691 wait complete. 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.452+0000 7f0012a00700 1 Processor -- start 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.453+0000 7f0012a00700 1 -- start start 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.453+0000 7f0012a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f000c0ff150 0x7f000c10fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.453+0000 7f0012a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c100360 0x7f000c1100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.453+0000 7f0012a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c110780 con 0x7f000c100360 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.453+0000 7f0012a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c110fc0 con 0x7f000c0ff150 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.453+0000 7f000b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c100360 0x7f000c1100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.453+0000 7f000b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c100360 0x7f000c1100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51650/0 (socket says 172.21.15.1:51650) 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.453+0000 7f000b400700 1 -- 172.21.15.1:0/2734666368 learned_addr learned my addr 172.21.15.1:0/2734666368 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.454+0000 7f000be00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f000c0ff150 0x7f000c10fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.454+0000 7f000b400700 1 -- 172.21.15.1:0/2734666368 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f000c0ff150 msgr2=0x7f000c10fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.454+0000 7f000b400700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f000c0ff150 0x7f000c10fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.454+0000 7f000b400700 1 -- 172.21.15.1:0/2734666368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc005040 con 0x7f000c100360 2024-02-28T16:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.454+0000 7f000b400700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c100360 0x7f000c1100f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efff800b790 tx=0x7efff8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.454+0000 7f0003e00700 1 -- 172.21.15.1:0/2734666368 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff800f770 con 0x7f000c100360 2024-02-28T16:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.455+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000c1112a0 con 0x7f000c100360 2024-02-28T16:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.455+0000 7f0003e00700 1 -- 172.21.15.1:0/2734666368 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efff800fda0 con 0x7f000c100360 2024-02-28T16:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.455+0000 7f0003e00700 1 -- 172.21.15.1:0/2734666368 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff800d310 con 0x7f000c100360 2024-02-28T16:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.455+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c1117e0 con 0x7f000c100360 2024-02-28T16:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.456+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c04bed0 con 0x7f000c100360 2024-02-28T16:34:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.457+0000 7f0003e00700 1 -- 172.21.15.1:0/2734666368 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7efff8010420 con 0x7f000c100360 2024-02-28T16:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.458+0000 7f0003e00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7efff006d0e0 0x7efff006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.458+0000 7f0003e00700 1 -- 172.21.15.1:0/2734666368 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7efff808c5e0 con 0x7f000c100360 2024-02-28T16:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.459+0000 7f000be00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7efff006d0e0 0x7efff006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.459+0000 7f000be00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7efff006d0e0 0x7efff006f590 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efffc00b3b0 tx=0x7efffc009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.464+0000 7f0003e00700 1 -- 172.21.15.1:0/2734666368 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7efff805a1f0 con 0x7f000c100360 2024-02-28T16:34:36.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.708+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f000c111c90 con 0x7efff006d0e0 2024-02-28T16:34:36.698 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:36 smithi001 ceph-mon[28991]: pgmap v88: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:36.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.710+0000 7f0003e00700 1 -- 172.21.15.1:0/2734666368 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f000c111c90 con 0x7efff006d0e0 2024-02-28T16:34:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7efff006d0e0 msgr2=0x7efff006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7efff006d0e0 0x7efff006f590 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efffc00b3b0 tx=0x7efffc009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c100360 msgr2=0x7f000c1100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c100360 0x7f000c1100f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efff800b790 tx=0x7efff8009250 comp rx=0 tx=0).stop 2024-02-28T16:34:36.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 shutdown_connections 2024-02-28T16:34:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f000c0ff150 0x7f000c10fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7efff006d0e0 0x7efff006f590 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 --2- 172.21.15.1:0/2734666368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c100360 0x7f000c1100f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 >> 172.21.15.1:0/2734666368 conn(0x7f000c0fa920 msgr2=0x7f000c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 shutdown_connections 2024-02-28T16:34:36.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:36.713+0000 7f0012a00700 1 -- 172.21.15.1:0/2734666368 wait complete. 2024-02-28T16:34:36.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:36 smithi066 ceph-mon[36236]: pgmap v88: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:37.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-28T16:34:37.947 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:34:38.654 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:38 smithi001 ceph-mon[28991]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:38.654 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:38 smithi001 ceph-mon[28991]: pgmap v89: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:38.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:38 smithi066 ceph-mon[36236]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:38.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:38 smithi066 ceph-mon[36236]: pgmap v89: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.198+0000 7f3236c00700 1 -- 172.21.15.1:0/2844609592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105b10 msgr2=0x7f32300ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.198+0000 7f3236c00700 1 --2- 172.21.15.1:0/2844609592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105b10 0x7f32300ff180 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f321c004940 tx=0x7f321c009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.198+0000 7f3236c00700 1 -- 172.21.15.1:0/2844609592 shutdown_connections 2024-02-28T16:34:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.198+0000 7f3236c00700 1 --2- 172.21.15.1:0/2844609592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105b10 0x7f32300ff180 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.198+0000 7f3236c00700 1 --2- 172.21.15.1:0/2844609592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3230105170 0x7f3230105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.198+0000 7f3236c00700 1 -- 172.21.15.1:0/2844609592 >> 172.21.15.1:0/2844609592 conn(0x7f32300face0 msgr2=0x7f32300fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.199+0000 7f3236c00700 1 -- 172.21.15.1:0/2844609592 shutdown_connections 2024-02-28T16:34:40.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.199+0000 7f3236c00700 1 -- 172.21.15.1:0/2844609592 wait complete. 2024-02-28T16:34:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.199+0000 7f3236c00700 1 Processor -- start 2024-02-28T16:34:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.200+0000 7f3236c00700 1 -- start start 2024-02-28T16:34:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.200+0000 7f3236c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105170 0x7f3230115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.200+0000 7f3236c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3230105b10 0x7f3230116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:40.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.200+0000 7f3236c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3230116870 con 0x7f3230105170 2024-02-28T16:34:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.200+0000 7f3236c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323010fb80 con 0x7f3230105b10 2024-02-28T16:34:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.200+0000 7f322fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105170 0x7f3230115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.200+0000 7f322fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105170 0x7f3230115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51668/0 (socket says 172.21.15.1:51668) 2024-02-28T16:34:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.200+0000 7f322fe00700 1 -- 172.21.15.1:0/2084312130 learned_addr learned my addr 172.21.15.1:0/2084312130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:40.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.201+0000 7f322f400700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3230105b10 0x7f3230116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.201+0000 7f322fe00700 1 -- 172.21.15.1:0/2084312130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3230105b10 msgr2=0x7f3230116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.201+0000 7f322fe00700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3230105b10 0x7f3230116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.201+0000 7f322fe00700 1 -- 172.21.15.1:0/2084312130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c005040 con 0x7f3230105170 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.201+0000 7f322fe00700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105170 0x7f3230115ce0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3220002a50 tx=0x7f32200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.201+0000 7f3227e00700 1 -- 172.21.15.1:0/2084312130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3220007cc0 con 0x7f3230105170 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.202+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323010fe60 con 0x7f3230105170 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.202+0000 7f3227e00700 1 -- 172.21.15.1:0/2084312130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f322000f040 con 0x7f3230105170 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.202+0000 7f3227e00700 1 -- 172.21.15.1:0/2084312130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3220009640 con 0x7f3230105170 2024-02-28T16:34:40.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.202+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32301103a0 con 0x7f3230105170 2024-02-28T16:34:40.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.203+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3230107640 con 0x7f3230105170 2024-02-28T16:34:40.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.204+0000 7f3227e00700 1 -- 172.21.15.1:0/2084312130 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f32200097a0 con 0x7f3230105170 2024-02-28T16:34:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.205+0000 7f3227e00700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f321406d0e0 0x7f321406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.205+0000 7f3227e00700 1 -- 172.21.15.1:0/2084312130 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f322008c350 con 0x7f3230105170 2024-02-28T16:34:40.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.205+0000 7f322f400700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f321406d0e0 0x7f321406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:40.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.206+0000 7f322f400700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f321406d0e0 0x7f321406f590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b3b0 tx=0x7f321c009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:40.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.211+0000 7f3227e00700 1 -- 172.21.15.1:0/2084312130 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f3220059ee0 con 0x7f3230105170 2024-02-28T16:34:40.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.528+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f323004bed0 con 0x7f3230105170 2024-02-28T16:34:40.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:40 smithi001 ceph-mon[28991]: pgmap v90: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:40.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.529+0000 7f3227e00700 1 -- 172.21.15.1:0/2084312130 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f3220059a70 con 0x7f3230105170 2024-02-28T16:34:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f321406d0e0 msgr2=0x7f321406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f321406d0e0 0x7f321406f590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b3b0 tx=0x7f321c009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105170 msgr2=0x7f3230115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105170 0x7f3230115ce0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3220002a50 tx=0x7f32200046b0 comp rx=0 tx=0).stop 2024-02-28T16:34:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 shutdown_connections 2024-02-28T16:34:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f321406d0e0 0x7f321406f590 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3230105170 0x7f3230115ce0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 --2- 172.21.15.1:0/2084312130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3230105b10 0x7f3230116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 >> 172.21.15.1:0/2084312130 conn(0x7f32300face0 msgr2=0x7f32300fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.532+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 shutdown_connections 2024-02-28T16:34:40.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:40.533+0000 7f3236c00700 1 -- 172.21.15.1:0/2084312130 wait complete. 2024-02-28T16:34:40.537 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:34:40.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:40 smithi066 ceph-mon[36236]: pgmap v90: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:41.260 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-02-28T16:34:41.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:41 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/2084312130' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:41.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:41 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2084312130' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:41.775 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:34:42.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:42 smithi001 ceph-mon[28991]: pgmap v91: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:42.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:42 smithi066 ceph-mon[36236]: pgmap v91: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.020+0000 7f67dee00700 1 -- 172.21.15.1:0/3811491005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 msgr2=0x7f67d8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.020+0000 7f67dee00700 1 --2- 172.21.15.1:0/3811491005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 0x7f67d8104210 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f67c0004940 tx=0x7f67c0009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.020+0000 7f67dee00700 1 -- 172.21.15.1:0/3811491005 shutdown_connections 2024-02-28T16:34:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.020+0000 7f67dee00700 1 --2- 172.21.15.1:0/3811491005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 0x7f67d8104210 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.020+0000 7f67dee00700 1 --2- 172.21.15.1:0/3811491005 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f67d81095f0 0x7f67d81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.020+0000 7f67dee00700 1 -- 172.21.15.1:0/3811491005 >> 172.21.15.1:0/3811491005 conn(0x7f67d8072d50 msgr2=0x7f67d8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.020+0000 7f67dee00700 1 -- 172.21.15.1:0/3811491005 shutdown_connections 2024-02-28T16:34:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.020+0000 7f67dee00700 1 -- 172.21.15.1:0/3811491005 wait complete. 2024-02-28T16:34:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.021+0000 7f67dee00700 1 Processor -- start 2024-02-28T16:34:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.021+0000 7f67dee00700 1 -- start start 2024-02-28T16:34:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.022+0000 7f67dee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 0x7f67d811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.022+0000 7f67dee00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f67d81095f0 0x7f67d811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.022+0000 7f67dee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d811abd0 con 0x7f67d8103dc0 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.022+0000 7f67dee00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8113ee0 con 0x7f67d81095f0 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.022+0000 7f67d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 0x7f67d811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.022+0000 7f67d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 0x7f67d811a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52220/0 (socket says 172.21.15.1:52220) 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.022+0000 7f67d7e00700 1 -- 172.21.15.1:0/3360462856 learned_addr learned my addr 172.21.15.1:0/3360462856 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.022+0000 7f67cfe00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f67d81095f0 0x7f67d811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67d7e00700 1 -- 172.21.15.1:0/3360462856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f67d81095f0 msgr2=0x7f67d811a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67d7e00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f67d81095f0 0x7f67d811a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67d7e00700 1 -- 172.21.15.1:0/3360462856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c0005040 con 0x7f67d8103dc0 2024-02-28T16:34:44.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67d7e00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 0x7f67d811a040 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f67c800c8c0 tx=0x7f67c80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67d4c00700 1 -- 172.21.15.1:0/3360462856 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67c800a980 con 0x7f67d8103dc0 2024-02-28T16:34:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d81141c0 con 0x7f67d8103dc0 2024-02-28T16:34:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67d4c00700 1 -- 172.21.15.1:0/3360462856 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67c800ce60 con 0x7f67d8103dc0 2024-02-28T16:34:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67d4c00700 1 -- 172.21.15.1:0/3360462856 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67c80115e0 con 0x7f67d8103dc0 2024-02-28T16:34:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.023+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d8114700 con 0x7f67d8103dc0 2024-02-28T16:34:44.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.025+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d810b9c0 con 0x7f67d8103dc0 2024-02-28T16:34:44.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.026+0000 7f67d4c00700 1 -- 172.21.15.1:0/3360462856 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f67c8012400 con 0x7f67d8103dc0 2024-02-28T16:34:44.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.027+0000 7f67d4c00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f67bc06d0f0 0x7f67bc06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:44.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.027+0000 7f67d4c00700 1 -- 172.21.15.1:0/3360462856 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f67c808c670 con 0x7f67d8103dc0 2024-02-28T16:34:44.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.027+0000 7f67cfe00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f67bc06d0f0 0x7f67bc06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:44.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.028+0000 7f67cfe00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f67bc06d0f0 0x7f67bc06f5a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f67d81158e0 tx=0x7f67c0009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:44.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.032+0000 7f67d4c00700 1 -- 172.21.15.1:0/3360462856 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f67c805a280 con 0x7f67d8103dc0 2024-02-28T16:34:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.345+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f67d804bed0 con 0x7f67d8103dc0 2024-02-28T16:34:44.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.346+0000 7f67d4c00700 1 -- 172.21.15.1:0/3360462856 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f67c80126b0 con 0x7f67d8103dc0 2024-02-28T16:34:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f67bc06d0f0 msgr2=0x7f67bc06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f67bc06d0f0 0x7f67bc06f5a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f67d81158e0 tx=0x7f67c0009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 msgr2=0x7f67d811a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 0x7f67d811a040 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f67c800c8c0 tx=0x7f67c80044c0 comp rx=0 tx=0).stop 2024-02-28T16:34:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 shutdown_connections 2024-02-28T16:34:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f67bc06d0f0 0x7f67bc06f5a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67d8103dc0 0x7f67d811a040 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 --2- 172.21.15.1:0/3360462856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f67d81095f0 0x7f67d811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 >> 172.21.15.1:0/3360462856 conn(0x7f67d8072d50 msgr2=0x7f67d8074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 shutdown_connections 2024-02-28T16:34:44.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:44.349+0000 7f67dee00700 1 -- 172.21.15.1:0/3360462856 wait complete. 2024-02-28T16:34:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)" 2024-02-28T16:34:44.701 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:44 smithi001 ceph-mon[28991]: pgmap v92: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:44.702 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:44 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3360462856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:44.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:44 smithi066 ceph-mon[36236]: pgmap v92: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:44.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:44 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3360462856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:45.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-02-28T16:34:45.593 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:34:45.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:45 smithi001 ceph-mon[28991]: pgmap v93: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:45.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:45 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:34:45.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:45 smithi066 ceph-mon[36236]: pgmap v93: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:45.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:45 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:34:47.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.650+0000 7f5cd4400700 1 -- 172.21.15.1:0/2331850869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 msgr2=0x7f5ccc107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:47.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.650+0000 7f5cd4400700 1 --2- 172.21.15.1:0/2331850869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 0x7f5ccc107050 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8004970 tx=0x7f5cb8009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:47.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.651+0000 7f5cd4400700 1 -- 172.21.15.1:0/2331850869 shutdown_connections 2024-02-28T16:34:47.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.651+0000 7f5cd4400700 1 --2- 172.21.15.1:0/2331850869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 0x7f5ccc107050 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:47.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.651+0000 7f5cd4400700 1 --2- 172.21.15.1:0/2331850869 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ccc0fdfe0 0x7f5ccc0fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:47.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.651+0000 7f5cd4400700 1 -- 172.21.15.1:0/2331850869 >> 172.21.15.1:0/2331850869 conn(0x7f5ccc071d90 msgr2=0x7f5ccc072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:47.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.651+0000 7f5cd4400700 1 -- 172.21.15.1:0/2331850869 shutdown_connections 2024-02-28T16:34:47.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.651+0000 7f5cd4400700 1 -- 172.21.15.1:0/2331850869 wait complete. 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd4400700 1 Processor -- start 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd4400700 1 -- start start 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd4400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ccc0fdfe0 0x7f5ccc119ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd4400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 0x7f5ccc11a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd4400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc11ab80 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd4400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc113e90 con 0x7f5ccc0fdfe0 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 0x7f5ccc11a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 0x7f5ccc11a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52250/0 (socket says 172.21.15.1:52250) 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.652+0000 7f5cd1000700 1 -- 172.21.15.1:0/3442499414 learned_addr learned my addr 172.21.15.1:0/3442499414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.653+0000 7f5cd1a00700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ccc0fdfe0 0x7f5ccc119ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.653+0000 7f5cd1000700 1 -- 172.21.15.1:0/3442499414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ccc0fdfe0 msgr2=0x7f5ccc119ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:47.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.653+0000 7f5cd1000700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ccc0fdfe0 0x7f5ccc119ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.653+0000 7f5cd1000700 1 -- 172.21.15.1:0/3442499414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8005040 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.653+0000 7f5cd1a00700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ccc0fdfe0 0x7f5ccc119ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:34:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.653+0000 7f5cd1000700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 0x7f5ccc11a530 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8004f50 tx=0x7f5cb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.654+0000 7f5cc9600700 1 -- 172.21.15.1:0/3442499414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cb800e040 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.654+0000 7f5cc9600700 1 -- 172.21.15.1:0/3442499414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5cb8004290 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.654+0000 7f5cc9600700 1 -- 172.21.15.1:0/3442499414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cb801c630 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.654+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc114110 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.654+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc114650 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.655+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc102710 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.656+0000 7f5cc9600700 1 -- 172.21.15.1:0/3442499414 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f5cb800bc10 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.657+0000 7f5cc9600700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f5cb406d0f0 0x7f5cb406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.657+0000 7f5cc9600700 1 -- 172.21.15.1:0/3442499414 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f5cb808d820 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.658+0000 7f5cd1a00700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f5cb406d0f0 0x7f5cb406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:47.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.659+0000 7f5cd1a00700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f5cb406d0f0 0x7f5cb406f5a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0007fd0 tx=0x7f5cc0009040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:47.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.663+0000 7f5cc9600700 1 -- 172.21.15.1:0/3442499414 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f5cb805b570 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.983+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5ccc0738b0 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.985+0000 7f5cc9600700 1 -- 172.21.15.1:0/3442499414 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f5cb8013e60 con 0x7f5ccc0fe8f0 2024-02-28T16:34:47.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f5cb406d0f0 msgr2=0x7f5cb406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:47.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f5cb406d0f0 0x7f5cb406f5a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0007fd0 tx=0x7f5cc0009040 comp rx=0 tx=0).stop 2024-02-28T16:34:47.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 msgr2=0x7f5ccc11a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 0x7f5ccc11a530 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8004f50 tx=0x7f5cb8009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 shutdown_connections 2024-02-28T16:34:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ccc0fdfe0 0x7f5ccc119ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f5cb406d0f0 0x7f5cb406f5a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 --2- 172.21.15.1:0/3442499414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ccc0fe8f0 0x7f5ccc11a530 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 >> 172.21.15.1:0/3442499414 conn(0x7f5ccc071d90 msgr2=0x7f5ccc0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 shutdown_connections 2024-02-28T16:34:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:47.988+0000 7f5cd4400700 1 -- 172.21.15.1:0/3442499414 wait complete. 2024-02-28T16:34:47.992 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:34:48.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:47 smithi066 ceph-mon[36236]: pgmap v94: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:48.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:47 smithi001 ceph-mon[28991]: pgmap v94: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:48.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-02-28T16:34:49.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:48 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3442499414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:49.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:48 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3442499414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:34:49.268 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:34:50.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:49 smithi001 ceph-mon[28991]: pgmap v95: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:50.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:49 smithi066 ceph-mon[36236]: pgmap v95: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.414+0000 7f8e73c00700 1 -- 172.21.15.1:0/1885163872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 msgr2=0x7f8e6c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.414+0000 7f8e73c00700 1 --2- 172.21.15.1:0/1885163872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 0x7f8e6c103320 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009ce0 tx=0x7f8e5c004d10 comp rx=0 tx=0).stop 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.414+0000 7f8e73c00700 1 -- 172.21.15.1:0/1885163872 shutdown_connections 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.414+0000 7f8e73c00700 1 --2- 172.21.15.1:0/1885163872 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e6c1038f0 0x7f8e6c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.414+0000 7f8e73c00700 1 --2- 172.21.15.1:0/1885163872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 0x7f8e6c103320 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.414+0000 7f8e73c00700 1 -- 172.21.15.1:0/1885163872 >> 172.21.15.1:0/1885163872 conn(0x7f8e6c0f8ae0 msgr2=0x7f8e6c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.414+0000 7f8e73c00700 1 -- 172.21.15.1:0/1885163872 shutdown_connections 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.414+0000 7f8e73c00700 1 -- 172.21.15.1:0/1885163872 wait complete. 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.415+0000 7f8e73c00700 1 Processor -- start 2024-02-28T16:34:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.415+0000 7f8e73c00700 1 -- start start 2024-02-28T16:34:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.415+0000 7f8e73c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 0x7f8e6c115e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.415+0000 7f8e73c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e6c1038f0 0x7f8e6c116350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.415+0000 7f8e73c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c1169a0 con 0x7f8e6c102f50 2024-02-28T16:34:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.415+0000 7f8e73c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c10fcb0 con 0x7f8e6c1038f0 2024-02-28T16:34:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.416+0000 7f8e71200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 0x7f8e6c115e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.416+0000 7f8e71200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 0x7f8e6c115e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52268/0 (socket says 172.21.15.1:52268) 2024-02-28T16:34:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.416+0000 7f8e71200700 1 -- 172.21.15.1:0/3347747794 learned_addr learned my addr 172.21.15.1:0/3347747794 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:51.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.416+0000 7f8e71200700 1 -- 172.21.15.1:0/3347747794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e6c1038f0 msgr2=0x7f8e6c116350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:51.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.416+0000 7f8e71200700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e6c1038f0 0x7f8e6c116350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:51.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.416+0000 7f8e71200700 1 -- 172.21.15.1:0/3347747794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c009940 con 0x7f8e6c102f50 2024-02-28T16:34:51.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.416+0000 7f8e71200700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 0x7f8e6c115e10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c004b70 tx=0x7f8e5c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:51.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.417+0000 7f8e69600700 1 -- 172.21.15.1:0/3347747794 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e5c00e040 con 0x7f8e6c102f50 2024-02-28T16:34:51.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.417+0000 7f8e69600700 1 -- 172.21.15.1:0/3347747794 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e5c004610 con 0x7f8e6c102f50 2024-02-28T16:34:51.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.417+0000 7f8e69600700 1 -- 172.21.15.1:0/3347747794 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e5c01c810 con 0x7f8e6c102f50 2024-02-28T16:34:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.417+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c10ff30 con 0x7f8e6c102f50 2024-02-28T16:34:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.417+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c110470 con 0x7f8e6c102f50 2024-02-28T16:34:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.418+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c0fc6e0 con 0x7f8e6c102f50 2024-02-28T16:34:51.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.419+0000 7f8e69600700 1 -- 172.21.15.1:0/3347747794 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f8e5c012c00 con 0x7f8e6c102f50 2024-02-28T16:34:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.420+0000 7f8e69600700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e5006d0f0 0x7f8e5006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.420+0000 7f8e69600700 1 -- 172.21.15.1:0/3347747794 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f8e5c08e100 con 0x7f8e6c102f50 2024-02-28T16:34:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.421+0000 7f8e63e00700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e5006d0f0 0x7f8e5006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.421+0000 7f8e63e00700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e5006d0f0 0x7f8e5006f5a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c1116b0 tx=0x7f8e54007d40 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:51.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.425+0000 7f8e69600700 1 -- 172.21.15.1:0/3347747794 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f8e5c05bd10 con 0x7f8e6c102f50 2024-02-28T16:34:51.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:51.671+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e6c111490 con 0x7f8e5006d0f0 2024-02-28T16:34:52.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:51 smithi001 ceph-mon[28991]: pgmap v96: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:52.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:51 smithi066 ceph-mon[36236]: pgmap v96: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:52.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:52 smithi066 ceph-mon[36236]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:53.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:52 smithi001 ceph-mon[28991]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:53 smithi001 ceph-mon[28991]: pgmap v97: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:54.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:53 smithi066 ceph-mon[36236]: pgmap v97: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:54.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.750+0000 7f8e69600700 1 -- 172.21.15.1:0/3347747794 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5092 (secure 0 0 0) 0x7f8e6c111490 con 0x7f8e5006d0f0 2024-02-28T16:34:54.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e5006d0f0 msgr2=0x7f8e5006f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:54.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e5006d0f0 0x7f8e5006f5a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c1116b0 tx=0x7f8e54007d40 comp rx=0 tx=0).stop 2024-02-28T16:34:54.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 msgr2=0x7f8e6c115e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:54.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 0x7f8e6c115e10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c004b70 tx=0x7f8e5c004d10 comp rx=0 tx=0).stop 2024-02-28T16:34:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 shutdown_connections 2024-02-28T16:34:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f8e5006d0f0 0x7f8e5006f5a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c102f50 0x7f8e6c115e10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 --2- 172.21.15.1:0/3347747794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e6c1038f0 0x7f8e6c116350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.754+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 >> 172.21.15.1:0/3347747794 conn(0x7f8e6c0f8ae0 msgr2=0x7f8e6c0fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.755+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 shutdown_connections 2024-02-28T16:34:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:54.755+0000 7f8e73c00700 1 -- 172.21.15.1:0/3347747794 wait complete. 2024-02-28T16:34:54.761 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:34:55.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph orch ps' 2024-02-28T16:34:55.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:55 smithi001 ceph-mon[28991]: pgmap v98: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:55.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:55 smithi066 ceph-mon[36236]: pgmap v98: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:34:57.898 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:57 smithi001 ceph-mon[28991]: pgmap v99: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:58.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.199+0000 7fb523e00700 1 -- 172.21.15.1:0/335820510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c103dc0 msgr2=0x7fb51c104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:58.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.199+0000 7fb523e00700 1 --2- 172.21.15.1:0/335820510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c103dc0 0x7fb51c104210 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb508004940 tx=0x7fb508009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:58.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.199+0000 7fb523e00700 1 -- 172.21.15.1:0/335820510 shutdown_connections 2024-02-28T16:34:58.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.199+0000 7fb523e00700 1 --2- 172.21.15.1:0/335820510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c103dc0 0x7fb51c104210 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:58.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.199+0000 7fb523e00700 1 --2- 172.21.15.1:0/335820510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb51c1095f0 0x7fb51c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:58.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.199+0000 7fb523e00700 1 -- 172.21.15.1:0/335820510 >> 172.21.15.1:0/335820510 conn(0x7fb51c072d50 msgr2=0x7fb51c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:58.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.199+0000 7fb523e00700 1 -- 172.21.15.1:0/335820510 shutdown_connections 2024-02-28T16:34:58.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.199+0000 7fb523e00700 1 -- 172.21.15.1:0/335820510 wait complete. 2024-02-28T16:34:58.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.200+0000 7fb523e00700 1 Processor -- start 2024-02-28T16:34:58.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.200+0000 7fb523e00700 1 -- start start 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb523e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb51c103dc0 0x7fb51c11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb523e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c1095f0 0x7fb51c11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb523e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c11abd0 con 0x7fb51c1095f0 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb523e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c113ee0 con 0x7fb51c103dc0 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb520a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c1095f0 0x7fb51c11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb520a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c1095f0 0x7fb51c11a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59852/0 (socket says 172.21.15.1:59852) 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb520a00700 1 -- 172.21.15.1:0/4037749795 learned_addr learned my addr 172.21.15.1:0/4037749795 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb520a00700 1 -- 172.21.15.1:0/4037749795 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb51c103dc0 msgr2=0x7fb51c11a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb520a00700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb51c103dc0 0x7fb51c11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.201+0000 7fb520a00700 1 -- 172.21.15.1:0/4037749795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508005040 con 0x7fb51c1095f0 2024-02-28T16:34:58.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.202+0000 7fb520a00700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c1095f0 0x7fb51c11a580 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb508009150 tx=0x7fb508009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:58.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.202+0000 7fb519600700 1 -- 172.21.15.1:0/4037749795 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb50800e040 con 0x7fb51c1095f0 2024-02-28T16:34:58.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.202+0000 7fb519600700 1 -- 172.21.15.1:0/4037749795 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb50800bbf0 con 0x7fb51c1095f0 2024-02-28T16:34:58.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.202+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51c114160 con 0x7fb51c1095f0 2024-02-28T16:34:58.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.202+0000 7fb519600700 1 -- 172.21.15.1:0/4037749795 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb50801c630 con 0x7fb51c1095f0 2024-02-28T16:34:58.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.202+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51c1146a0 con 0x7fb51c1095f0 2024-02-28T16:34:58.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.203+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51c10b9c0 con 0x7fb51c1095f0 2024-02-28T16:34:58.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.204+0000 7fb519600700 1 -- 172.21.15.1:0/4037749795 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7fb50800bd60 con 0x7fb51c1095f0 2024-02-28T16:34:58.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.206+0000 7fb519600700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb50406d0f0 0x7fb50406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:34:58.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.206+0000 7fb519600700 1 -- 172.21.15.1:0/4037749795 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fb50808d8d0 con 0x7fb51c1095f0 2024-02-28T16:34:58.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.206+0000 7fb521400700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb50406d0f0 0x7fb50406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:34:58.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.207+0000 7fb521400700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb50406d0f0 0x7fb50406f5a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb510004d00 tx=0x7fb510009ce0 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:34:58.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.211+0000 7fb519600700 1 -- 172.21.15.1:0/4037749795 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fb50805b560 con 0x7fb51c1095f0 2024-02-28T16:34:58.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:57 smithi066 ceph-mon[36236]: pgmap v99: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:34:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.445+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb51c04bed0 con 0x7fb50406d0f0 2024-02-28T16:34:58.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.458+0000 7fb519600700 1 -- 172.21.15.1:0/4037749795 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb51c04bed0 con 0x7fb50406d0f0 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (65s) 32s ago 15m 14.0M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (15m) 32s ago 15m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (14m) 115s ago 14m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (40s) 32s ago 14m 51.7M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (8m) 32s ago 8m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (8m) 32s ago 8m 13.3M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (8m) 115s ago 8m 13.0M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (8m) 115s ago 8m 152M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (3m) 32s ago 17m 501M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:34:58.447 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (2m) 115s ago 14m 446M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (17m) 32s ago 17m 54.9M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (13m) 115s ago 13m 42.5M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (2m) 32s ago 15m 7268k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (2m) 115s ago 14m 3053k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (13m) 32s ago 13m 207M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (12m) 32s ago 12m 208M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (11m) 32s ago 11m 168M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (11m) 115s ago 11m 215M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (10m) 115s ago 10m 197M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (10m) 115s ago 10m 181M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:34:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (93s) 32s ago 14m 45.2M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:34:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb50406d0f0 msgr2=0x7fb50406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb50406d0f0 0x7fb50406f5a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb510004d00 tx=0x7fb510009ce0 comp rx=0 tx=0).stop 2024-02-28T16:34:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c1095f0 msgr2=0x7fb51c11a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:34:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c1095f0 0x7fb51c11a580 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb508009150 tx=0x7fb508009e30 comp rx=0 tx=0).stop 2024-02-28T16:34:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 shutdown_connections 2024-02-28T16:34:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb51c103dc0 0x7fb51c11a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fb50406d0f0 0x7fb50406f5a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 --2- 172.21.15.1:0/4037749795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb51c1095f0 0x7fb51c11a580 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:34:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.460+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 >> 172.21.15.1:0/4037749795 conn(0x7fb51c072d50 msgr2=0x7fb51c074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:34:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.461+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 shutdown_connections 2024-02-28T16:34:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:34:58.461+0000 7fb523e00700 1 -- 172.21.15.1:0/4037749795 wait complete. 2024-02-28T16:34:59.245 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:58 smithi001 ceph-mon[28991]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:59.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:58 smithi066 ceph-mon[36236]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:34:59.301 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-28T16:34:59.302 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:34:59.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-02-28T16:34:59.797 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:35:00.114 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:59 smithi001 ceph-mon[28991]: pgmap v100: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:00.114 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:34:59 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:00.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:59 smithi066 ceph-mon[36236]: pgmap v100: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:00.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:34:59 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:01.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.812+0000 7f2f32c00700 1 -- 172.21.15.1:0/2805536760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 msgr2=0x7f2f2c0fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:01.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.812+0000 7f2f32c00700 1 --2- 172.21.15.1:0/2805536760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 0x7f2f2c0fe160 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2f20004940 tx=0x7f2f20009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:01.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.812+0000 7f2f32c00700 1 -- 172.21.15.1:0/2805536760 shutdown_connections 2024-02-28T16:35:01.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.812+0000 7f2f32c00700 1 --2- 172.21.15.1:0/2805536760 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f2c0fe730 0x7f2f2c106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:01.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.812+0000 7f2f32c00700 1 --2- 172.21.15.1:0/2805536760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 0x7f2f2c0fe160 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:01.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.812+0000 7f2f32c00700 1 -- 172.21.15.1:0/2805536760 >> 172.21.15.1:0/2805536760 conn(0x7f2f2c0f7820 msgr2=0x7f2f2c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:01.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.812+0000 7f2f32c00700 1 -- 172.21.15.1:0/2805536760 shutdown_connections 2024-02-28T16:35:01.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.813+0000 7f2f32c00700 1 -- 172.21.15.1:0/2805536760 wait complete. 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.813+0000 7f2f32c00700 1 Processor -- start 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.814+0000 7f2f32c00700 1 -- start start 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.814+0000 7f2f32c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 0x7f2f2c113840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.814+0000 7f2f32c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f2c0fe730 0x7f2f2c113d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.814+0000 7f2f32c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c1143d0 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.814+0000 7f2f32c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c10d700 con 0x7f2f2c0fe730 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.814+0000 7f2f31800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 0x7f2f2c113840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.814+0000 7f2f31800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 0x7f2f2c113840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59856/0 (socket says 172.21.15.1:59856) 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.814+0000 7f2f31800700 1 -- 172.21.15.1:0/4157909898 learned_addr learned my addr 172.21.15.1:0/4157909898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f30e00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f2c0fe730 0x7f2f2c113d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f31800700 1 -- 172.21.15.1:0/4157909898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f2c0fe730 msgr2=0x7f2f2c113d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f31800700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f2c0fe730 0x7f2f2c113d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f31800700 1 -- 172.21.15.1:0/4157909898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20005040 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f30e00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f2c0fe730 0x7f2f2c113d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:35:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f31800700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 0x7f2f2c113840 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009180 tx=0x7f2f20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f29600700 1 -- 172.21.15.1:0/4157909898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2000e040 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f29600700 1 -- 172.21.15.1:0/4157909898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f2000bbf0 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2c10d980 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.815+0000 7f2f29600700 1 -- 172.21.15.1:0/4157909898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2001c630 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.816+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2c10dec0 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.817+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2c104380 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.818+0000 7f2f29600700 1 -- 172.21.15.1:0/4157909898 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f2f2000bd60 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.819+0000 7f2f29600700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2f1406d0f0 0x7f2f1406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.819+0000 7f2f29600700 1 -- 172.21.15.1:0/4157909898 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f2f2008d990 con 0x7f2f2c0fdd90 2024-02-28T16:35:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.820+0000 7f2f30e00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2f1406d0f0 0x7f2f1406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.820+0000 7f2f30e00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2f1406d0f0 0x7f2f1406f5a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f18007fd0 tx=0x7f2f18009040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:01.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:01.825+0000 7f2f29600700 1 -- 172.21.15.1:0/4157909898 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f2f2005b5a0 con 0x7f2f2c0fdd90 2024-02-28T16:35:02.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:01 smithi001 ceph-mon[28991]: pgmap v101: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:02.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.052+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f2f2c04bed0 con 0x7f2f2c0fdd90 2024-02-28T16:35:02.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.059+0000 7f2f29600700 1 -- 172.21.15.1:0/4157909898 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v47)=0 v47) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2f2005b130 con 0x7f2f2c0fdd90 2024-02-28T16:35:02.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.063+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2f1406d0f0 msgr2=0x7f2f1406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:02.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.063+0000 7f2f32c00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2f1406d0f0 0x7f2f1406f5a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f18007fd0 tx=0x7f2f18009040 comp rx=0 tx=0).stop 2024-02-28T16:35:02.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.063+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 msgr2=0x7f2f2c113840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:02.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.063+0000 7f2f32c00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 0x7f2f2c113840 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009180 tx=0x7f2f20009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:02.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.063+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 shutdown_connections 2024-02-28T16:35:02.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.063+0000 7f2f32c00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f2f1406d0f0 0x7f2f1406f5a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:02.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.063+0000 7f2f32c00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f2c0fdd90 0x7f2f2c113840 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:02.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.063+0000 7f2f32c00700 1 --2- 172.21.15.1:0/4157909898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2f2c0fe730 0x7f2f2c113d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:02.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.064+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 >> 172.21.15.1:0/4157909898 conn(0x7f2f2c0f7820 msgr2=0x7f2f2c101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:02.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.064+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 shutdown_connections 2024-02-28T16:35:02.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:02.064+0000 7f2f32c00700 1 -- 172.21.15.1:0/4157909898 wait complete. 2024-02-28T16:35:02.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:01 smithi066 ceph-mon[36236]: pgmap v101: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:02.857 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-28T16:35:02.858 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:35:02.858 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-28T16:35:03.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:03 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/4157909898' entity='client.admin' 2024-02-28T16:35:03.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:03 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:03.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:03 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:35:03.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:03 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:35:03.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:03 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:03.371 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:35:03.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:03 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/4157909898' entity='client.admin' 2024-02-28T16:35:03.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:03 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:03.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:03 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:35:03.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:03 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:35:03.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:03 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:04 smithi066 ceph-mon[36236]: pgmap v102: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:04.498 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:04 smithi001 ceph-mon[28991]: pgmap v102: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:05.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:05 smithi001 ceph-mon[28991]: pgmap v103: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:05.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.802+0000 7fa0f3600700 1 -- 172.21.15.1:0/1089343079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec1095f0 msgr2=0x7fa0ec1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:05.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.802+0000 7fa0f3600700 1 --2- 172.21.15.1:0/1089343079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec1095f0 0x7fa0ec1099c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0004940 tx=0x7fa0e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:05.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.802+0000 7fa0f3600700 1 -- 172.21.15.1:0/1089343079 shutdown_connections 2024-02-28T16:35:05.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.802+0000 7fa0f3600700 1 --2- 172.21.15.1:0/1089343079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0ec103dc0 0x7fa0ec104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:05.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.802+0000 7fa0f3600700 1 --2- 172.21.15.1:0/1089343079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec1095f0 0x7fa0ec1099c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:05.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.802+0000 7fa0f3600700 1 -- 172.21.15.1:0/1089343079 >> 172.21.15.1:0/1089343079 conn(0x7fa0ec072d50 msgr2=0x7fa0ec075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:05.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.802+0000 7fa0f3600700 1 -- 172.21.15.1:0/1089343079 shutdown_connections 2024-02-28T16:35:05.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.802+0000 7fa0f3600700 1 -- 172.21.15.1:0/1089343079 wait complete. 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.803+0000 7fa0f3600700 1 Processor -- start 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.803+0000 7fa0f3600700 1 -- start start 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.804+0000 7fa0f3600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec103dc0 0x7fa0ec11a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.804+0000 7fa0f3600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0ec1095f0 0x7fa0ec11a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.804+0000 7fa0f3600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec11ac70 con 0x7fa0ec103dc0 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.804+0000 7fa0f3600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec113ef0 con 0x7fa0ec1095f0 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.804+0000 7fa0f0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec103dc0 0x7fa0ec11a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.804+0000 7fa0f0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec103dc0 0x7fa0ec11a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36348/0 (socket says 172.21.15.1:36348) 2024-02-28T16:35:05.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.804+0000 7fa0f0c00700 1 -- 172.21.15.1:0/930116426 learned_addr learned my addr 172.21.15.1:0/930116426 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:05.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.804+0000 7fa0ebe00700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0ec1095f0 0x7fa0ec11a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:05.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0f0c00700 1 -- 172.21.15.1:0/930116426 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0ec1095f0 msgr2=0x7fa0ec11a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:05.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0f0c00700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0ec1095f0 0x7fa0ec11a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:05.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0f0c00700 1 -- 172.21.15.1:0/930116426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e0005040 con 0x7fa0ec103dc0 2024-02-28T16:35:05.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0ebe00700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0ec1095f0 0x7fa0ec11a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:35:05.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0f0c00700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec103dc0 0x7fa0ec11a050 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009100 tx=0x7fa0e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:05.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0e8c00700 1 -- 172.21.15.1:0/930116426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0e000e040 con 0x7fa0ec103dc0 2024-02-28T16:35:05.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0e8c00700 1 -- 172.21.15.1:0/930116426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0e0004290 con 0x7fa0ec103dc0 2024-02-28T16:35:05.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0e8c00700 1 -- 172.21.15.1:0/930116426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0e001c630 con 0x7fa0ec103dc0 2024-02-28T16:35:05.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.805+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ec114110 con 0x7fa0ec103dc0 2024-02-28T16:35:05.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.806+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ec114650 con 0x7fa0ec103dc0 2024-02-28T16:35:05.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.807+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ec075870 con 0x7fa0ec103dc0 2024-02-28T16:35:05.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.808+0000 7fa0e8c00700 1 -- 172.21.15.1:0/930116426 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7fa0e000bbf0 con 0x7fa0ec103dc0 2024-02-28T16:35:05.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.808+0000 7fa0e8c00700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa0d406d0e0 0x7fa0d406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:05.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.809+0000 7fa0e8c00700 1 -- 172.21.15.1:0/930116426 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fa0e008d810 con 0x7fa0ec103dc0 2024-02-28T16:35:05.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.809+0000 7fa0ebe00700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa0d406d0e0 0x7fa0d406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:05.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.810+0000 7fa0ebe00700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa0d406d0e0 0x7fa0d406f590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8006fd0 tx=0x7fa0d8008040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:05.814+0000 7fa0e8c00700 1 -- 172.21.15.1:0/930116426 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fa0e005b4a0 con 0x7fa0ec103dc0 2024-02-28T16:35:05.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:05 smithi066 ceph-mon[36236]: pgmap v103: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.035+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fa0ec1106a0 con 0x7fa0ec103dc0 2024-02-28T16:35:06.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.036+0000 7fa0e8c00700 1 -- 172.21.15.1:0/930116426 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v47)=0 v47) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa0e005b030 con 0x7fa0ec103dc0 2024-02-28T16:35:06.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa0d406d0e0 msgr2=0x7fa0d406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:06.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa0d406d0e0 0x7fa0d406f590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8006fd0 tx=0x7fa0d8008040 comp rx=0 tx=0).stop 2024-02-28T16:35:06.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec103dc0 msgr2=0x7fa0ec11a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:06.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec103dc0 0x7fa0ec11a050 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009100 tx=0x7fa0e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:06.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 shutdown_connections 2024-02-28T16:35:06.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fa0d406d0e0 0x7fa0d406f590 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:06.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec103dc0 0x7fa0ec11a050 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:06.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 --2- 172.21.15.1:0/930116426 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0ec1095f0 0x7fa0ec11a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:06.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 >> 172.21.15.1:0/930116426 conn(0x7fa0ec072d50 msgr2=0x7fa0ec0746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:06.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 shutdown_connections 2024-02-28T16:35:06.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:06.039+0000 7fa0f3600700 1 -- 172.21.15.1:0/930116426 wait complete. 2024-02-28T16:35:06.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-28T16:35:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:35:08.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:07 smithi066 ceph-mon[36236]: pgmap v104: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:08.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:07 smithi001 ceph-mon[28991]: pgmap v104: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:09.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.494+0000 7fd98ea00700 1 -- 172.21.15.1:0/2154274407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 msgr2=0x7fd988070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:09.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.494+0000 7fd98ea00700 1 --2- 172.21.15.1:0/2154274407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 0x7fd988070330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd978004970 tx=0x7fd978009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.494+0000 7fd98ea00700 1 -- 172.21.15.1:0/2154274407 shutdown_connections 2024-02-28T16:35:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.494+0000 7fd98ea00700 1 --2- 172.21.15.1:0/2154274407 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd988070870 0x7fd988070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.494+0000 7fd98ea00700 1 --2- 172.21.15.1:0/2154274407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 0x7fd988070330 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.494+0000 7fd98ea00700 1 -- 172.21.15.1:0/2154274407 >> 172.21.15.1:0/2154274407 conn(0x7fd9880f8ae0 msgr2=0x7fd9880faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.494+0000 7fd98ea00700 1 -- 172.21.15.1:0/2154274407 shutdown_connections 2024-02-28T16:35:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.494+0000 7fd98ea00700 1 -- 172.21.15.1:0/2154274407 wait complete. 2024-02-28T16:35:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.495+0000 7fd98ea00700 1 Processor -- start 2024-02-28T16:35:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.495+0000 7fd98ea00700 1 -- start start 2024-02-28T16:35:09.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.495+0000 7fd98ea00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd988070870 0x7fd98811a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:09.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.495+0000 7fd98ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 0x7fd98811a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:09.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.495+0000 7fd98ea00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98811acf0 con 0x7fd988071f60 2024-02-28T16:35:09.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.495+0000 7fd98ea00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd988113f70 con 0x7fd988070870 2024-02-28T16:35:09.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 0x7fd98811a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 0x7fd98811a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36366/0 (socket says 172.21.15.1:36366) 2024-02-28T16:35:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987400700 1 -- 172.21.15.1:0/184878156 learned_addr learned my addr 172.21.15.1:0/184878156 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987e00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd988070870 0x7fd98811a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987400700 1 -- 172.21.15.1:0/184878156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd988070870 msgr2=0x7fd98811a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987400700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd988070870 0x7fd98811a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987400700 1 -- 172.21.15.1:0/184878156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd978005040 con 0x7fd988071f60 2024-02-28T16:35:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987e00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd988070870 0x7fd98811a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:35:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.496+0000 7fd987400700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 0x7fd98811a610 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd97400b790 tx=0x7fd974009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:09.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.497+0000 7fd97fe00700 1 -- 172.21.15.1:0/184878156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd97400f770 con 0x7fd988071f60 2024-02-28T16:35:09.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.497+0000 7fd97fe00700 1 -- 172.21.15.1:0/184878156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd97400fda0 con 0x7fd988071f60 2024-02-28T16:35:09.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.497+0000 7fd97fe00700 1 -- 172.21.15.1:0/184878156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd97400d310 con 0x7fd988071f60 2024-02-28T16:35:09.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.497+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9881141f0 con 0x7fd988071f60 2024-02-28T16:35:09.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.497+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd988114730 con 0x7fd988071f60 2024-02-28T16:35:09.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.498+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9880fe400 con 0x7fd988071f60 2024-02-28T16:35:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.500+0000 7fd97fe00700 1 -- 172.21.15.1:0/184878156 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7fd974010420 con 0x7fd988071f60 2024-02-28T16:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.501+0000 7fd97fe00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd96c06d0e0 0x7fd96c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.501+0000 7fd97fe00700 1 -- 172.21.15.1:0/184878156 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7fd97408c520 con 0x7fd988071f60 2024-02-28T16:35:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.501+0000 7fd987e00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd96c06d0e0 0x7fd96c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.502+0000 7fd987e00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd96c06d0e0 0x7fd96c06f590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd978009fd0 tx=0x7fd978009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:09.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.506+0000 7fd97fe00700 1 -- 172.21.15.1:0/184878156 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd97405a1b0 con 0x7fd988071f60 2024-02-28T16:35:09.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.730+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fd9880fc6e0 con 0x7fd988071f60 2024-02-28T16:35:09.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.731+0000 7fd97fe00700 1 -- 172.21.15.1:0/184878156 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v47)=0 v47) v1 ==== 163+0+0 (secure 0 0 0) 0x7fd974059d40 con 0x7fd988071f60 2024-02-28T16:35:09.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd96c06d0e0 msgr2=0x7fd96c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:09.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd96c06d0e0 0x7fd96c06f590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd978009fd0 tx=0x7fd978009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:09.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 msgr2=0x7fd98811a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:09.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 0x7fd98811a610 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd97400b790 tx=0x7fd974009250 comp rx=0 tx=0).stop 2024-02-28T16:35:09.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 shutdown_connections 2024-02-28T16:35:09.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd988070870 0x7fd98811a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:09.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7fd96c06d0e0 0x7fd96c06f590 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:09.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 --2- 172.21.15.1:0/184878156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd988071f60 0x7fd98811a610 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:09.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.735+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 >> 172.21.15.1:0/184878156 conn(0x7fd9880f8ae0 msgr2=0x7fd9880fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:09.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.736+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 shutdown_connections 2024-02-28T16:35:09.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:09.736+0000 7fd98ea00700 1 -- 172.21.15.1:0/184878156 wait complete. 2024-02-28T16:35:10.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:09 smithi001 ceph-mon[28991]: pgmap v105: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:10.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:09 smithi066 ceph-mon[36236]: pgmap v105: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:10.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-28T16:35:11.223 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:35:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:11 smithi001 ceph-mon[28991]: pgmap v106: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:12.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:11 smithi066 ceph-mon[36236]: pgmap v106: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:13.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.616+0000 7f9890a00700 1 -- 172.21.15.1:0/2494550692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 msgr2=0x7f988c105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:13.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.616+0000 7f9890a00700 1 --2- 172.21.15.1:0/2494550692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 0x7f988c105520 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f987c004940 tx=0x7f987c009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:13.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.616+0000 7f9890a00700 1 -- 172.21.15.1:0/2494550692 shutdown_connections 2024-02-28T16:35:13.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.616+0000 7f9890a00700 1 --2- 172.21.15.1:0/2494550692 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f988c105af0 0x7f988c0ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:13.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.616+0000 7f9890a00700 1 --2- 172.21.15.1:0/2494550692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 0x7f988c105520 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:13.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.616+0000 7f9890a00700 1 -- 172.21.15.1:0/2494550692 >> 172.21.15.1:0/2494550692 conn(0x7f988c0face0 msgr2=0x7f988c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:13.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.616+0000 7f9890a00700 1 -- 172.21.15.1:0/2494550692 shutdown_connections 2024-02-28T16:35:13.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.616+0000 7f9890a00700 1 -- 172.21.15.1:0/2494550692 wait complete. 2024-02-28T16:35:13.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.617+0000 7f9890a00700 1 Processor -- start 2024-02-28T16:35:13.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.617+0000 7f9890a00700 1 -- start start 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9890a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 0x7f988c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9890a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f988c105af0 0x7f988c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9890a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c116800 con 0x7f988c105150 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9890a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c10fb30 con 0x7f988c105af0 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9889e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 0x7f988c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9889e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 0x7f988c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51688/0 (socket says 172.21.15.1:51688) 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9889e00700 1 -- 172.21.15.1:0/2825359454 learned_addr learned my addr 172.21.15.1:0/2825359454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9881400700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f988c105af0 0x7f988c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9889e00700 1 -- 172.21.15.1:0/2825359454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f988c105af0 msgr2=0x7f988c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9889e00700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f988c105af0 0x7f988c1161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.618+0000 7f9889e00700 1 -- 172.21.15.1:0/2825359454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c005040 con 0x7f988c105150 2024-02-28T16:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.619+0000 7f9889e00700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 0x7f988c115c70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f987c009150 tx=0x7f987c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:13.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.619+0000 7f9882a00700 1 -- 172.21.15.1:0/2825359454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f987c00e040 con 0x7f988c105150 2024-02-28T16:35:13.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.619+0000 7f9882a00700 1 -- 172.21.15.1:0/2825359454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f987c00bbf0 con 0x7f988c105150 2024-02-28T16:35:13.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.619+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c10fdb0 con 0x7f988c105150 2024-02-28T16:35:13.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.619+0000 7f9882a00700 1 -- 172.21.15.1:0/2825359454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f987c01c630 con 0x7f988c105150 2024-02-28T16:35:13.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.619+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c1102f0 con 0x7f988c105150 2024-02-28T16:35:13.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.620+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c0fe8e0 con 0x7f988c105150 2024-02-28T16:35:13.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.621+0000 7f9882a00700 1 -- 172.21.15.1:0/2825359454 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f987c00bd60 con 0x7f988c105150 2024-02-28T16:35:13.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.622+0000 7f9882a00700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f987006d0f0 0x7f987006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:13.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.622+0000 7f9882a00700 1 -- 172.21.15.1:0/2825359454 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f987c08d8d0 con 0x7f988c105150 2024-02-28T16:35:13.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.623+0000 7f9881400700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f987006d0f0 0x7f987006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:13.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.623+0000 7f9881400700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f987006d0f0 0x7f987006f5a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f988c1114a0 tx=0x7f987400b040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:13.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.627+0000 7f9882a00700 1 -- 172.21.15.1:0/2825359454 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f987c05b560 con 0x7f988c105150 2024-02-28T16:35:13.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.855+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f988c111250 con 0x7f988c105150 2024-02-28T16:35:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.856+0000 7f9882a00700 1 -- 172.21.15.1:0/2825359454 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v47)=0 v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7f987c05b0f0 con 0x7f988c105150 2024-02-28T16:35:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f987006d0f0 msgr2=0x7f987006f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f987006d0f0 0x7f987006f5a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f988c1114a0 tx=0x7f987400b040 comp rx=0 tx=0).stop 2024-02-28T16:35:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 msgr2=0x7f988c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 0x7f988c115c70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f987c009150 tx=0x7f987c009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 shutdown_connections 2024-02-28T16:35:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f987006d0f0 0x7f987006f5a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f988c105150 0x7f988c115c70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 --2- 172.21.15.1:0/2825359454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f988c105af0 0x7f988c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 >> 172.21.15.1:0/2825359454 conn(0x7f988c0face0 msgr2=0x7f988c0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 shutdown_connections 2024-02-28T16:35:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:13.859+0000 7f9890a00700 1 -- 172.21.15.1:0/2825359454 wait complete. 2024-02-28T16:35:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:13 smithi066 ceph-mon[36236]: pgmap v107: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:14.252 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:13 smithi001 ceph-mon[28991]: pgmap v107: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:14.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-28T16:35:15.230 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:35:15.685 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:15 smithi001 ceph-mon[28991]: pgmap v108: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:15.685 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:15 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:15.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:15 smithi066 ceph-mon[36236]: pgmap v108: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:15.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:15 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.688+0000 7f4b79800700 1 -- 172.21.15.1:0/1929947038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070c40 msgr2=0x7f4b74109050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.688+0000 7f4b79800700 1 --2- 172.21.15.1:0/1929947038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070c40 0x7f4b74109050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4b60004970 tx=0x7f4b60009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.688+0000 7f4b79800700 1 -- 172.21.15.1:0/1929947038 shutdown_connections 2024-02-28T16:35:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.688+0000 7f4b79800700 1 --2- 172.21.15.1:0/1929947038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070c40 0x7f4b74109050 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:17.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.688+0000 7f4b79800700 1 --2- 172.21.15.1:0/1929947038 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b74070330 0x7f4b74070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:17.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.688+0000 7f4b79800700 1 -- 172.21.15.1:0/1929947038 >> 172.21.15.1:0/1929947038 conn(0x7f4b740f88b0 msgr2=0x7f4b740facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:17.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.689+0000 7f4b79800700 1 -- 172.21.15.1:0/1929947038 shutdown_connections 2024-02-28T16:35:17.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.689+0000 7f4b79800700 1 -- 172.21.15.1:0/1929947038 wait complete. 2024-02-28T16:35:17.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.690+0000 7f4b79800700 1 Processor -- start 2024-02-28T16:35:17.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.690+0000 7f4b79800700 1 -- start start 2024-02-28T16:35:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.690+0000 7f4b79800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070330 0x7f4b74118d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.690+0000 7f4b79800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b74070c40 0x7f4b74119270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.690+0000 7f4b79800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b741198c0 con 0x7f4b74070330 2024-02-28T16:35:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.690+0000 7f4b79800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b74108200 con 0x7f4b74070c40 2024-02-28T16:35:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.691+0000 7f4b73e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070330 0x7f4b74118d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.691+0000 7f4b73e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070330 0x7f4b74118d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51698/0 (socket says 172.21.15.1:51698) 2024-02-28T16:35:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.691+0000 7f4b73e00700 1 -- 172.21.15.1:0/1908824189 learned_addr learned my addr 172.21.15.1:0/1908824189 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.691+0000 7f4b73400700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b74070c40 0x7f4b74119270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.691+0000 7f4b73e00700 1 -- 172.21.15.1:0/1908824189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b74070c40 msgr2=0x7f4b74119270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.691+0000 7f4b73e00700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b74070c40 0x7f4b74119270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.691+0000 7f4b73e00700 1 -- 172.21.15.1:0/1908824189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b60005040 con 0x7f4b74070330 2024-02-28T16:35:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.691+0000 7f4b73e00700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070330 0x7f4b74118d30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4b64002a50 tx=0x7f4b640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.692+0000 7f4b6be00700 1 -- 172.21.15.1:0/1908824189 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b64007cc0 con 0x7f4b74070330 2024-02-28T16:35:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.692+0000 7f4b6be00700 1 -- 172.21.15.1:0/1908824189 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b6400f040 con 0x7f4b74070330 2024-02-28T16:35:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.692+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b74108490 con 0x7f4b74070330 2024-02-28T16:35:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.692+0000 7f4b6be00700 1 -- 172.21.15.1:0/1908824189 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b64009640 con 0x7f4b74070330 2024-02-28T16:35:17.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.692+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b741089a0 con 0x7f4b74070330 2024-02-28T16:35:17.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.693+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b74102300 con 0x7f4b74070330 2024-02-28T16:35:17.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.694+0000 7f4b6be00700 1 -- 172.21.15.1:0/1908824189 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f4b640097a0 con 0x7f4b74070330 2024-02-28T16:35:17.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.695+0000 7f4b6be00700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4b5806d0f0 0x7f4b5806f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:17.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.695+0000 7f4b6be00700 1 -- 172.21.15.1:0/1908824189 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f4b6408c290 con 0x7f4b74070330 2024-02-28T16:35:17.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.695+0000 7f4b73400700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4b5806d0f0 0x7f4b5806f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:17.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.696+0000 7f4b73400700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4b5806d0f0 0x7f4b5806f5a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b74109340 tx=0x7f4b60009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:17.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.700+0000 7f4b6be00700 1 -- 172.21.15.1:0/1908824189 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f4b64059ea0 con 0x7f4b74070330 2024-02-28T16:35:17.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.948+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b741153d0 con 0x7f4b5806d0f0 2024-02-28T16:35:17.937 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:17 smithi001 ceph-mon[28991]: pgmap v109: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:17.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.957+0000 7f4b6be00700 1 -- 172.21.15.1:0/1908824189 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f4b741153d0 con 0x7f4b5806d0f0 2024-02-28T16:35:17.945 INFO:teuthology.orchestra.run.smithi001.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:35:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.959+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4b5806d0f0 msgr2=0x7f4b5806f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.959+0000 7f4b79800700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4b5806d0f0 0x7f4b5806f5a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b74109340 tx=0x7f4b60009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.959+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070330 msgr2=0x7f4b74118d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:17.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.959+0000 7f4b79800700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070330 0x7f4b74118d30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4b64002a50 tx=0x7f4b640046b0 comp rx=0 tx=0).stop 2024-02-28T16:35:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.960+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 shutdown_connections 2024-02-28T16:35:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.960+0000 7f4b79800700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f4b5806d0f0 0x7f4b5806f5a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.960+0000 7f4b79800700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b74070330 0x7f4b74118d30 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.960+0000 7f4b79800700 1 --2- 172.21.15.1:0/1908824189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4b74070c40 0x7f4b74119270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.960+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 >> 172.21.15.1:0/1908824189 conn(0x7f4b740f88b0 msgr2=0x7f4b740ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.960+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 shutdown_connections 2024-02-28T16:35:17.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:17.960+0000 7f4b79800700 1 -- 172.21.15.1:0/1908824189 wait complete. 2024-02-28T16:35:18.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:17 smithi066 ceph-mon[36236]: pgmap v109: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:18.795 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-28T16:35:18.796 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T16:35:18.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-28T16:35:19.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:18 smithi066 ceph-mon[36236]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:19.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:18 smithi066 ceph-mon[36236]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:35:19.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:19.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:19.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:35:19.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:35:19.251 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:18 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:19.252 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:18 smithi066 ceph-mon[36236]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:35:19.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:18 smithi001 ceph-mon[28991]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:19.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:18 smithi001 ceph-mon[28991]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:35:19.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:19.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:19.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:35:19.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:35:19.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:18 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:19.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:18 smithi001 ceph-mon[28991]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:1fb5d10608dccee73edd6ef8a2c7b804a097c324 2024-02-28T16:35:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:35:20.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:19 smithi066 ceph-mon[36236]: pgmap v110: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:20.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:19 smithi001 ceph-mon[28991]: pgmap v110: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:21.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.882+0000 7f16c7800700 1 -- 172.21.15.1:0/1960622899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 msgr2=0x7f16c01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:21.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.882+0000 7f16c7800700 1 --2- 172.21.15.1:0/1960622899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 0x7f16c01099c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004940 tx=0x7f16b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:21.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.882+0000 7f16c7800700 1 -- 172.21.15.1:0/1960622899 shutdown_connections 2024-02-28T16:35:21.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.882+0000 7f16c7800700 1 --2- 172.21.15.1:0/1960622899 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c0103dc0 0x7f16c0104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:21.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.882+0000 7f16c7800700 1 --2- 172.21.15.1:0/1960622899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 0x7f16c01099c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:21.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.882+0000 7f16c7800700 1 -- 172.21.15.1:0/1960622899 >> 172.21.15.1:0/1960622899 conn(0x7f16c0072d50 msgr2=0x7f16c0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:21.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.882+0000 7f16c7800700 1 -- 172.21.15.1:0/1960622899 shutdown_connections 2024-02-28T16:35:21.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.882+0000 7f16c7800700 1 -- 172.21.15.1:0/1960622899 wait complete. 2024-02-28T16:35:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.883+0000 7f16c7800700 1 Processor -- start 2024-02-28T16:35:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.883+0000 7f16c7800700 1 -- start start 2024-02-28T16:35:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16c7800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c0103dc0 0x7f16c011a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16c7800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 0x7f16c011a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16c7800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c011ac70 con 0x7f16c01095f0 2024-02-28T16:35:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16c7800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c0113ef0 con 0x7f16c0103dc0 2024-02-28T16:35:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16bfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 0x7f16c011a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16bfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 0x7f16c011a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44960/0 (socket says 172.21.15.1:44960) 2024-02-28T16:35:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16bfe00700 1 -- 172.21.15.1:0/1951991261 learned_addr learned my addr 172.21.15.1:0/1951991261 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16c4e00700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c0103dc0 0x7f16c011a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16bfe00700 1 -- 172.21.15.1:0/1951991261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c0103dc0 msgr2=0x7f16c011a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16bfe00700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c0103dc0 0x7f16c011a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.884+0000 7f16bfe00700 1 -- 172.21.15.1:0/1951991261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b4005040 con 0x7f16c01095f0 2024-02-28T16:35:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.885+0000 7f16bfe00700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 0x7f16c011a590 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b790 tx=0x7f16ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.885+0000 7f16bcc00700 1 -- 172.21.15.1:0/1951991261 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16ac00f770 con 0x7f16c01095f0 2024-02-28T16:35:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.885+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c0114170 con 0x7f16c01095f0 2024-02-28T16:35:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.885+0000 7f16bcc00700 1 -- 172.21.15.1:0/1951991261 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16ac00fda0 con 0x7f16c01095f0 2024-02-28T16:35:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.885+0000 7f16bcc00700 1 -- 172.21.15.1:0/1951991261 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16ac00d310 con 0x7f16c01095f0 2024-02-28T16:35:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.885+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c01146b0 con 0x7f16c01095f0 2024-02-28T16:35:21.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.886+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c0075870 con 0x7f16c01095f0 2024-02-28T16:35:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.887+0000 7f16bcc00700 1 -- 172.21.15.1:0/1951991261 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f16ac010420 con 0x7f16c01095f0 2024-02-28T16:35:21.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.889+0000 7f16bcc00700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f16a806d0f0 0x7f16a806f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:21.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.889+0000 7f16bcc00700 1 -- 172.21.15.1:0/1951991261 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f16ac08c590 con 0x7f16c01095f0 2024-02-28T16:35:21.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.889+0000 7f16c4e00700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f16a806d0f0 0x7f16a806f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:21.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.890+0000 7f16c4e00700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f16a806d0f0 0x7f16a806f5a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16b4000c00 tx=0x7f16b4009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:21.894+0000 7f16bcc00700 1 -- 172.21.15.1:0/1951991261 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f16ac05a1a0 con 0x7f16c01095f0 2024-02-28T16:35:22.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:21 smithi001 ceph-mon[28991]: pgmap v111: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.145+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f16c0115710 con 0x7f16a806d0f0 2024-02-28T16:35:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.146+0000 7f16bcc00700 1 -- 172.21.15.1:0/1951991261 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f16c0115710 con 0x7f16a806d0f0 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.149+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f16a806d0f0 msgr2=0x7f16a806f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.149+0000 7f16c7800700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f16a806d0f0 0x7f16a806f5a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16b4000c00 tx=0x7f16b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.149+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 msgr2=0x7f16c011a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.149+0000 7f16c7800700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 0x7f16c011a590 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b790 tx=0x7f16ac009250 comp rx=0 tx=0).stop 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.149+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 shutdown_connections 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.150+0000 7f16c7800700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16c0103dc0 0x7f16c011a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.150+0000 7f16c7800700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f16a806d0f0 0x7f16a806f5a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.150+0000 7f16c7800700 1 --2- 172.21.15.1:0/1951991261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16c01095f0 0x7f16c011a590 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.150+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 >> 172.21.15.1:0/1951991261 conn(0x7f16c0072d50 msgr2=0x7f16c00746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.150+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 shutdown_connections 2024-02-28T16:35:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.150+0000 7f16c7800700 1 -- 172.21.15.1:0/1951991261 wait complete. 2024-02-28T16:35:22.153 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:35:22.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:21 smithi066 ceph-mon[36236]: pgmap v111: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.312+0000 7ff8e2800700 1 -- 172.21.15.1:0/503951959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 msgr2=0x7ff8dc1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.312+0000 7ff8e2800700 1 --2- 172.21.15.1:0/503951959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 0x7ff8dc1099c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc004940 tx=0x7ff8cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.312+0000 7ff8e2800700 1 -- 172.21.15.1:0/503951959 shutdown_connections 2024-02-28T16:35:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.312+0000 7ff8e2800700 1 --2- 172.21.15.1:0/503951959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8dc103dc0 0x7ff8dc104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.312+0000 7ff8e2800700 1 --2- 172.21.15.1:0/503951959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 0x7ff8dc1099c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.312+0000 7ff8e2800700 1 -- 172.21.15.1:0/503951959 >> 172.21.15.1:0/503951959 conn(0x7ff8dc072d50 msgr2=0x7ff8dc075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.312+0000 7ff8e2800700 1 -- 172.21.15.1:0/503951959 shutdown_connections 2024-02-28T16:35:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.312+0000 7ff8e2800700 1 -- 172.21.15.1:0/503951959 wait complete. 2024-02-28T16:35:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.313+0000 7ff8e2800700 1 Processor -- start 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.313+0000 7ff8e2800700 1 -- start start 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.313+0000 7ff8e2800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8dc103dc0 0x7ff8dc11a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.313+0000 7ff8e2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 0x7ff8dc11a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.313+0000 7ff8e2800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc11ac70 con 0x7ff8dc1095f0 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.313+0000 7ff8e2800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc113ef0 con 0x7ff8dc103dc0 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.314+0000 7ff8daa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 0x7ff8dc11a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.314+0000 7ff8daa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 0x7ff8dc11a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44972/0 (socket says 172.21.15.1:44972) 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.314+0000 7ff8daa00700 1 -- 172.21.15.1:0/168130583 learned_addr learned my addr 172.21.15.1:0/168130583 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.314+0000 7ff8db400700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8dc103dc0 0x7ff8dc11a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.314+0000 7ff8daa00700 1 -- 172.21.15.1:0/168130583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8dc103dc0 msgr2=0x7ff8dc11a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.314+0000 7ff8daa00700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8dc103dc0 0x7ff8dc11a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.314+0000 7ff8daa00700 1 -- 172.21.15.1:0/168130583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc005040 con 0x7ff8dc1095f0 2024-02-28T16:35:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.315+0000 7ff8daa00700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 0x7ff8dc11a590 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800b790 tx=0x7ff8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.315+0000 7ff8d3400700 1 -- 172.21.15.1:0/168130583 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c800f770 con 0x7ff8dc1095f0 2024-02-28T16:35:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.315+0000 7ff8d3400700 1 -- 172.21.15.1:0/168130583 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8c800fda0 con 0x7ff8dc1095f0 2024-02-28T16:35:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.315+0000 7ff8d3400700 1 -- 172.21.15.1:0/168130583 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c800d310 con 0x7ff8dc1095f0 2024-02-28T16:35:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.315+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8dc114170 con 0x7ff8dc1095f0 2024-02-28T16:35:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.315+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc1146b0 con 0x7ff8dc1095f0 2024-02-28T16:35:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.316+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8dc075870 con 0x7ff8dc1095f0 2024-02-28T16:35:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.317+0000 7ff8d3400700 1 -- 172.21.15.1:0/168130583 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7ff8c8010420 con 0x7ff8dc1095f0 2024-02-28T16:35:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.318+0000 7ff8d3400700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff8c006d0f0 0x7ff8c006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.318+0000 7ff8d3400700 1 -- 172.21.15.1:0/168130583 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7ff8c808c520 con 0x7ff8dc1095f0 2024-02-28T16:35:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.319+0000 7ff8db400700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff8c006d0f0 0x7ff8c006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:22.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.319+0000 7ff8db400700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff8c006d0f0 0x7ff8c006f5a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00b020 tx=0x7ff8cc009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:22.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.323+0000 7ff8d3400700 1 -- 172.21.15.1:0/168130583 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7ff8c805a1b0 con 0x7ff8dc1095f0 2024-02-28T16:35:22.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.570+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8dc115710 con 0x7ff8c006d0f0 2024-02-28T16:35:22.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.572+0000 7ff8d3400700 1 -- 172.21.15.1:0/168130583 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7ff8dc115710 con 0x7ff8c006d0f0 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff8c006d0f0 msgr2=0x7ff8c006f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff8c006d0f0 0x7ff8c006f5a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00b020 tx=0x7ff8cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 msgr2=0x7ff8dc11a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 0x7ff8dc11a590 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800b790 tx=0x7ff8c8009250 comp rx=0 tx=0).stop 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 shutdown_connections 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff8dc103dc0 0x7ff8dc11a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff8c006d0f0 0x7ff8c006f5a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 --2- 172.21.15.1:0/168130583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8dc1095f0 0x7ff8dc11a590 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 >> 172.21.15.1:0/168130583 conn(0x7ff8dc072d50 msgr2=0x7ff8dc0746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:22.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 shutdown_connections 2024-02-28T16:35:22.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.575+0000 7ff8e2800700 1 -- 172.21.15.1:0/168130583 wait complete. 2024-02-28T16:35:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.732+0000 7f61e0c00700 1 -- 172.21.15.1:0/2414730947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 msgr2=0x7f61dc105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.732+0000 7f61e0c00700 1 --2- 172.21.15.1:0/2414730947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 0x7f61dc105540 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f61cc004940 tx=0x7f61cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.732+0000 7f61e0c00700 1 -- 172.21.15.1:0/2414730947 shutdown_connections 2024-02-28T16:35:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.732+0000 7f61e0c00700 1 --2- 172.21.15.1:0/2414730947 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61dc105b10 0x7f61dc0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.732+0000 7f61e0c00700 1 --2- 172.21.15.1:0/2414730947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 0x7f61dc105540 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.732+0000 7f61e0c00700 1 -- 172.21.15.1:0/2414730947 >> 172.21.15.1:0/2414730947 conn(0x7f61dc0face0 msgr2=0x7f61dc0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.732+0000 7f61e0c00700 1 -- 172.21.15.1:0/2414730947 shutdown_connections 2024-02-28T16:35:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.732+0000 7f61e0c00700 1 -- 172.21.15.1:0/2414730947 wait complete. 2024-02-28T16:35:22.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.733+0000 7f61e0c00700 1 Processor -- start 2024-02-28T16:35:22.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.733+0000 7f61e0c00700 1 -- start start 2024-02-28T16:35:22.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.734+0000 7f61e0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 0x7f61dc115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:22.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.734+0000 7f61e0c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61dc105b10 0x7f61dc1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:22.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.734+0000 7f61e0c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc1167f0 con 0x7f61dc105170 2024-02-28T16:35:22.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.734+0000 7f61e0c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc10fb90 con 0x7f61dc105b10 2024-02-28T16:35:22.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.734+0000 7f61d9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 0x7f61dc115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:22.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.734+0000 7f61d9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 0x7f61dc115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44988/0 (socket says 172.21.15.1:44988) 2024-02-28T16:35:22.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.734+0000 7f61d9e00700 1 -- 172.21.15.1:0/1382503155 learned_addr learned my addr 172.21.15.1:0/1382503155 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:22.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.735+0000 7f61d3e00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61dc105b10 0x7f61dc1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:22.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.735+0000 7f61d9e00700 1 -- 172.21.15.1:0/1382503155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61dc105b10 msgr2=0x7f61dc1161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.735+0000 7f61d9e00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61dc105b10 0x7f61dc1161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.735+0000 7f61d9e00700 1 -- 172.21.15.1:0/1382503155 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc005040 con 0x7f61dc105170 2024-02-28T16:35:22.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.735+0000 7f61d9e00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 0x7f61dc115c60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009fd0 tx=0x7f61cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.736+0000 7f61d2000700 1 -- 172.21.15.1:0/1382503155 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61cc00e040 con 0x7f61dc105170 2024-02-28T16:35:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.736+0000 7f61d2000700 1 -- 172.21.15.1:0/1382503155 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61cc00bbf0 con 0x7f61dc105170 2024-02-28T16:35:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.736+0000 7f61d2000700 1 -- 172.21.15.1:0/1382503155 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61cc01c3f0 con 0x7f61dc105170 2024-02-28T16:35:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.736+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc10fe10 con 0x7f61dc105170 2024-02-28T16:35:22.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.736+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61dc110350 con 0x7f61dc105170 2024-02-28T16:35:22.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.737+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61dc102480 con 0x7f61dc105170 2024-02-28T16:35:22.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.738+0000 7f61d2000700 1 -- 172.21.15.1:0/1382503155 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f61cc00bd60 con 0x7f61dc105170 2024-02-28T16:35:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.739+0000 7f61d2000700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f61c006d1a0 0x7f61c006f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.739+0000 7f61d2000700 1 -- 172.21.15.1:0/1382503155 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f61cc08d7b0 con 0x7f61dc105170 2024-02-28T16:35:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.740+0000 7f61d3e00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f61c006d1a0 0x7f61c006f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:22.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.740+0000 7f61d3e00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f61c006d1a0 0x7f61c006f650 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61c4007fd0 tx=0x7f61c4009040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:22.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.744+0000 7f61d2000700 1 -- 172.21.15.1:0/1382503155 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f61cc057910 con 0x7f61dc105170 2024-02-28T16:35:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:22.982+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f61dc04bf40 con 0x7f61c006d1a0 2024-02-28T16:35:22.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.001+0000 7f61d2000700 1 -- 172.21.15.1:0/1382503155 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f61dc04bf40 con 0x7f61c006d1a0 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (89s) 56s ago 15m 14.0M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (16m) 56s ago 16m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (14m) 2m ago 14m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (64s) 56s ago 15m 51.7M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (8m) 56s ago 8m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (8m) 56s ago 8m 13.3M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (8m) 2m ago 8m 13.0M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (8m) 2m ago 8m 152M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:35:22.990 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (3m) 56s ago 17m 501M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (2m) 2m ago 14m 446M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (17m) 56s ago 17m 54.9M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 4d5c15828f14 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (14m) 2m ago 14m 42.5M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (2m) 56s ago 15m 7268k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (2m) 2m ago 14m 3053k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (13m) 56s ago 13m 207M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (12m) 56s ago 12m 208M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (12m) 56s ago 12m 168M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (11m) 2m ago 11m 215M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (11m) 2m ago 11m 197M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (10m) 2m ago 10m 181M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:35:22.991 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (118s) 56s ago 15m 45.2M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:35:22.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f61c006d1a0 msgr2=0x7f61c006f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f61c006d1a0 0x7f61c006f650 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61c4007fd0 tx=0x7f61c4009040 comp rx=0 tx=0).stop 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 msgr2=0x7f61dc115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 0x7f61dc115c60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009fd0 tx=0x7f61cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 shutdown_connections 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f61c006d1a0 0x7f61c006f650 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc105170 0x7f61dc115c60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 --2- 172.21.15.1:0/1382503155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61dc105b10 0x7f61dc1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.005+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 >> 172.21.15.1:0/1382503155 conn(0x7f61dc0face0 msgr2=0x7f61dc0fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.006+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 shutdown_connections 2024-02-28T16:35:22.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.006+0000 7f61e0c00700 1 -- 172.21.15.1:0/1382503155 wait complete. 2024-02-28T16:35:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.162+0000 7f09a1200700 1 -- 172.21.15.1:0/684874934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fe8f0 msgr2=0x7f099c107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.162+0000 7f09a1200700 1 --2- 172.21.15.1:0/684874934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fe8f0 0x7f099c107050 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0988004970 tx=0x7f0988009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.162+0000 7f09a1200700 1 -- 172.21.15.1:0/684874934 shutdown_connections 2024-02-28T16:35:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.162+0000 7f09a1200700 1 --2- 172.21.15.1:0/684874934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fe8f0 0x7f099c107050 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.162+0000 7f09a1200700 1 --2- 172.21.15.1:0/684874934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099c0fdfe0 0x7f099c0fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.162+0000 7f09a1200700 1 -- 172.21.15.1:0/684874934 >> 172.21.15.1:0/684874934 conn(0x7f099c071d90 msgr2=0x7f099c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.162+0000 7f09a1200700 1 -- 172.21.15.1:0/684874934 shutdown_connections 2024-02-28T16:35:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.162+0000 7f09a1200700 1 -- 172.21.15.1:0/684874934 wait complete. 2024-02-28T16:35:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.163+0000 7f09a1200700 1 Processor -- start 2024-02-28T16:35:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.164+0000 7f09a1200700 1 -- start start 2024-02-28T16:35:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.164+0000 7f09a1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fdfe0 0x7f099c0fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.164+0000 7f09a1200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099c0fe8f0 0x7f099c0fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.164+0000 7f09a1200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099c0fc9e0 con 0x7f099c0fdfe0 2024-02-28T16:35:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.164+0000 7f09a1200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099c0fcb50 con 0x7f099c0fe8f0 2024-02-28T16:35:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.165+0000 7f0999e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fdfe0 0x7f099c0fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.165+0000 7f0999e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fdfe0 0x7f099c0fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45016/0 (socket says 172.21.15.1:45016) 2024-02-28T16:35:23.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.165+0000 7f0999e00700 1 -- 172.21.15.1:0/1936077104 learned_addr learned my addr 172.21.15.1:0/1936077104 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.165+0000 7f0999400700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099c0fe8f0 0x7f099c0fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.165+0000 7f0999e00700 1 -- 172.21.15.1:0/1936077104 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099c0fe8f0 msgr2=0x7f099c0fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.165+0000 7f0999e00700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099c0fe8f0 0x7f099c0fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.165+0000 7f0999e00700 1 -- 172.21.15.1:0/1936077104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0988005040 con 0x7f099c0fdfe0 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.166+0000 7f0999e00700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fdfe0 0x7f099c0fbea0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f098c002a50 tx=0x7f098c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.166+0000 7f0992000700 1 -- 172.21.15.1:0/1936077104 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f098c007cc0 con 0x7f099c0fdfe0 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.166+0000 7f0992000700 1 -- 172.21.15.1:0/1936077104 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f098c00f040 con 0x7f099c0fdfe0 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.166+0000 7f0992000700 1 -- 172.21.15.1:0/1936077104 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f098c009640 con 0x7f099c0fdfe0 2024-02-28T16:35:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.166+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099c119c30 con 0x7f099c0fdfe0 2024-02-28T16:35:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.166+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099c11a110 con 0x7f099c0fdfe0 2024-02-28T16:35:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.168+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099c1046c0 con 0x7f099c0fdfe0 2024-02-28T16:35:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.169+0000 7f0992000700 1 -- 172.21.15.1:0/1936077104 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f098c0097a0 con 0x7f099c0fdfe0 2024-02-28T16:35:23.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.170+0000 7f0992000700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098006d0f0 0x7f098006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:23.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.170+0000 7f0992000700 1 -- 172.21.15.1:0/1936077104 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f098c08c270 con 0x7f099c0fdfe0 2024-02-28T16:35:23.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.171+0000 7f0999400700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098006d0f0 0x7f098006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:23.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.172+0000 7f0999400700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098006d0f0 0x7f098006f5a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0988000c00 tx=0x7f0988009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:23.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.175+0000 7f0992000700 1 -- 172.21.15.1:0/1936077104 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f098c056350 con 0x7f099c0fdfe0 2024-02-28T16:35:23.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:22 smithi066 ceph-mon[36236]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:23.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:22 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:23.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:22 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:23.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:22 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:35:23.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:22 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:23.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:22 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-28T16:35:23.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:22 smithi001 ceph-mon[28991]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:23.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:22 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:23.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:22 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:23.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:22 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:35:23.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:22 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:23.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:22 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-28T16:35:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.522+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f099c0fd5e0 con 0x7f099c0fdfe0 2024-02-28T16:35:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.523+0000 7f0992000700 1 -- 172.21.15.1:0/1936077104 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f098c059a10 con 0x7f099c0fdfe0 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:35:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:35:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:35:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:35:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 12, 2024-02-28T16:35:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:35:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:35:23.512 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:35:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098006d0f0 msgr2=0x7f098006f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098006d0f0 0x7f098006f5a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0988000c00 tx=0x7f0988009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fdfe0 msgr2=0x7f099c0fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fdfe0 0x7f099c0fbea0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f098c002a50 tx=0x7f098c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 shutdown_connections 2024-02-28T16:35:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f098006d0f0 0x7f098006f5a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099c0fdfe0 0x7f099c0fbea0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 --2- 172.21.15.1:0/1936077104 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f099c0fe8f0 0x7f099c0fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.525+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 >> 172.21.15.1:0/1936077104 conn(0x7f099c071d90 msgr2=0x7f099c101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.526+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 shutdown_connections 2024-02-28T16:35:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.526+0000 7f09a1200700 1 -- 172.21.15.1:0/1936077104 wait complete. 2024-02-28T16:35:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.675+0000 7ff53e200700 1 -- 172.21.15.1:0/223895070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5381095f0 msgr2=0x7ff5381099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.675+0000 7ff53e200700 1 --2- 172.21.15.1:0/223895070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5381095f0 0x7ff5381099c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff528004940 tx=0x7ff528009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.675+0000 7ff53e200700 1 -- 172.21.15.1:0/223895070 shutdown_connections 2024-02-28T16:35:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.675+0000 7ff53e200700 1 --2- 172.21.15.1:0/223895070 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff538103dc0 0x7ff538104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.675+0000 7ff53e200700 1 --2- 172.21.15.1:0/223895070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5381095f0 0x7ff5381099c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.675+0000 7ff53e200700 1 -- 172.21.15.1:0/223895070 >> 172.21.15.1:0/223895070 conn(0x7ff538072d50 msgr2=0x7ff538075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:23.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.675+0000 7ff53e200700 1 -- 172.21.15.1:0/223895070 shutdown_connections 2024-02-28T16:35:23.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.675+0000 7ff53e200700 1 -- 172.21.15.1:0/223895070 wait complete. 2024-02-28T16:35:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.676+0000 7ff53e200700 1 Processor -- start 2024-02-28T16:35:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.676+0000 7ff53e200700 1 -- start start 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.676+0000 7ff53e200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff538103dc0 0x7ff53811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.676+0000 7ff53e200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5381095f0 0x7ff53811a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.676+0000 7ff53e200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53811ac70 con 0x7ff538103dc0 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.676+0000 7ff53e200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff538113ef0 con 0x7ff5381095f0 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff537400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff538103dc0 0x7ff53811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff537400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff538103dc0 0x7ff53811a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45042/0 (socket says 172.21.15.1:45042) 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff537400700 1 -- 172.21.15.1:0/1609805709 learned_addr learned my addr 172.21.15.1:0/1609805709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff536a00700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5381095f0 0x7ff53811a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff537400700 1 -- 172.21.15.1:0/1609805709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5381095f0 msgr2=0x7ff53811a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff537400700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5381095f0 0x7ff53811a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff537400700 1 -- 172.21.15.1:0/1609805709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff528005040 con 0x7ff538103dc0 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff536a00700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5381095f0 0x7ff53811a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:35:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.677+0000 7ff537400700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff538103dc0 0x7ff53811a050 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff528009150 tx=0x7ff528009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.678+0000 7ff52f400700 1 -- 172.21.15.1:0/1609805709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff52800e040 con 0x7ff538103dc0 2024-02-28T16:35:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.678+0000 7ff52f400700 1 -- 172.21.15.1:0/1609805709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff52800bbf0 con 0x7ff538103dc0 2024-02-28T16:35:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.678+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff538114110 con 0x7ff538103dc0 2024-02-28T16:35:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.678+0000 7ff52f400700 1 -- 172.21.15.1:0/1609805709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff52801c630 con 0x7ff538103dc0 2024-02-28T16:35:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.678+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff538114650 con 0x7ff538103dc0 2024-02-28T16:35:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.679+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff538075870 con 0x7ff538103dc0 2024-02-28T16:35:23.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.680+0000 7ff52f400700 1 -- 172.21.15.1:0/1609805709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7ff52800bd60 con 0x7ff538103dc0 2024-02-28T16:35:23.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.681+0000 7ff52f400700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff51c06d0f0 0x7ff51c06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:23.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.681+0000 7ff52f400700 1 -- 172.21.15.1:0/1609805709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7ff52808d9b0 con 0x7ff538103dc0 2024-02-28T16:35:23.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.681+0000 7ff536a00700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff51c06d0f0 0x7ff51c06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:23.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.682+0000 7ff536a00700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff51c06d0f0 0x7ff51c06f5a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff524006fd0 tx=0x7ff524008040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:23.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.686+0000 7ff52f400700 1 -- 172.21.15.1:0/1609805709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7ff52805b5c0 con 0x7ff538103dc0 2024-02-28T16:35:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.972+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff5381156d0 con 0x7ff538103dc0 2024-02-28T16:35:23.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.973+0000 7ff52f400700 1 -- 172.21.15.1:0/1609805709 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1844 (secure 0 0 0) 0x7ff52805b150 con 0x7ff538103dc0 2024-02-28T16:35:23.963 INFO:teuthology.orchestra.run.smithi001.stdout:e16 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 16 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:30:03.631200+0000 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:35:23.964 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:35:23.965 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 export targets 0 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:23.966 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:23.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.975+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff51c06d0f0 msgr2=0x7ff51c06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:23.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff51c06d0f0 0x7ff51c06f5a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff524006fd0 tx=0x7ff524008040 comp rx=0 tx=0).stop 2024-02-28T16:35:23.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff538103dc0 msgr2=0x7ff53811a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:23.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff538103dc0 0x7ff53811a050 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff528009150 tx=0x7ff528009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:23.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 shutdown_connections 2024-02-28T16:35:23.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff51c06d0f0 0x7ff51c06f5a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff538103dc0 0x7ff53811a050 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 --2- 172.21.15.1:0/1609805709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5381095f0 0x7ff53811a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:23.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 >> 172.21.15.1:0/1609805709 conn(0x7ff538072d50 msgr2=0x7ff5380746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:23.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 shutdown_connections 2024-02-28T16:35:23.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:23.976+0000 7ff53e200700 1 -- 172.21.15.1:0/1609805709 wait complete. 2024-02-28T16:35:23.968 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-02-28T16:35:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.133+0000 7ff33bc00700 1 -- 172.21.15.1:0/2651948024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 msgr2=0x7ff3340ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.133+0000 7ff33bc00700 1 --2- 172.21.15.1:0/2651948024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 0x7ff3340ff170 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff320004940 tx=0x7ff320009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.133+0000 7ff33bc00700 1 -- 172.21.15.1:0/2651948024 shutdown_connections 2024-02-28T16:35:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.133+0000 7ff33bc00700 1 --2- 172.21.15.1:0/2651948024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 0x7ff3340ff170 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.133+0000 7ff33bc00700 1 --2- 172.21.15.1:0/2651948024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff334105160 0x7ff334105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.133+0000 7ff33bc00700 1 -- 172.21.15.1:0/2651948024 >> 172.21.15.1:0/2651948024 conn(0x7ff3340fad30 msgr2=0x7ff3340fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.133+0000 7ff33bc00700 1 -- 172.21.15.1:0/2651948024 shutdown_connections 2024-02-28T16:35:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.133+0000 7ff33bc00700 1 -- 172.21.15.1:0/2651948024 wait complete. 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.134+0000 7ff33bc00700 1 Processor -- start 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.134+0000 7ff33bc00700 1 -- start start 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.134+0000 7ff33bc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff334105160 0x7ff334115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.134+0000 7ff33bc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 0x7ff3341161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.134+0000 7ff33bc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334116890 con 0x7ff334105b00 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.134+0000 7ff33bc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33410fb30 con 0x7ff334105160 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.135+0000 7ff333e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 0x7ff3341161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.135+0000 7ff333e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 0x7ff3341161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45052/0 (socket says 172.21.15.1:45052) 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.135+0000 7ff333e00700 1 -- 172.21.15.1:0/103937597 learned_addr learned my addr 172.21.15.1:0/103937597 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.135+0000 7ff339200700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff334105160 0x7ff334115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.135+0000 7ff333e00700 1 -- 172.21.15.1:0/103937597 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff334105160 msgr2=0x7ff334115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.135+0000 7ff333e00700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff334105160 0x7ff334115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.135+0000 7ff333e00700 1 -- 172.21.15.1:0/103937597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320005040 con 0x7ff334105b00 2024-02-28T16:35:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.136+0000 7ff333e00700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 0x7ff3341161b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff320009150 tx=0x7ff320009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.136+0000 7ff330c00700 1 -- 172.21.15.1:0/103937597 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32000e040 con 0x7ff334105b00 2024-02-28T16:35:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.136+0000 7ff330c00700 1 -- 172.21.15.1:0/103937597 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff32000bbf0 con 0x7ff334105b00 2024-02-28T16:35:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.136+0000 7ff330c00700 1 -- 172.21.15.1:0/103937597 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32001c630 con 0x7ff334105b00 2024-02-28T16:35:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.136+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33410fdb0 con 0x7ff334105b00 2024-02-28T16:35:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.136+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3341102f0 con 0x7ff334105b00 2024-02-28T16:35:24.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.137+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff334102470 con 0x7ff334105b00 2024-02-28T16:35:24.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.139+0000 7ff330c00700 1 -- 172.21.15.1:0/103937597 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7ff32000bd60 con 0x7ff334105b00 2024-02-28T16:35:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.140+0000 7ff330c00700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff31c06d0e0 0x7ff31c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.140+0000 7ff330c00700 1 -- 172.21.15.1:0/103937597 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7ff32008e400 con 0x7ff334105b00 2024-02-28T16:35:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.141+0000 7ff339200700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff31c06d0e0 0x7ff31c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.141+0000 7ff339200700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff31c06d0e0 0x7ff31c06f590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff328006fd0 tx=0x7ff328008040 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:24.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.144+0000 7ff330c00700 1 -- 172.21.15.1:0/103937597 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7ff32005c090 con 0x7ff334105b00 2024-02-28T16:35:24.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:23 smithi066 ceph-mon[36236]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:24.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:23 smithi066 ceph-mon[36236]: Upgrade: Target is version 19.0.0-1670-g1fb5d106 (squid) 2024-02-28T16:35:24.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:23 smithi066 ceph-mon[36236]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc'] 2024-02-28T16:35:24.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:23 smithi066 ceph-mon[36236]: Upgrade: Setting container_image for all mgr 2024-02-28T16:35:24.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:23 smithi066 ceph-mon[36236]: pgmap v112: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:24.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:23 smithi066 ceph-mon[36236]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:24.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:23 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1936077104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:35:24.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:23 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/1609805709' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:35:24.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:23 smithi001 ceph-mon[28991]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:24.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:23 smithi001 ceph-mon[28991]: Upgrade: Target is version 19.0.0-1670-g1fb5d106 (squid) 2024-02-28T16:35:24.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:23 smithi001 ceph-mon[28991]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc'] 2024-02-28T16:35:24.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:23 smithi001 ceph-mon[28991]: Upgrade: Setting container_image for all mgr 2024-02-28T16:35:24.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:23 smithi001 ceph-mon[28991]: pgmap v112: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:24.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:23 smithi001 ceph-mon[28991]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:24.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:23 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1936077104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:35:24.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:23 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/1609805709' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:35:24.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.395+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 --> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff334115940 con 0x7ff31c06d0e0 2024-02-28T16:35:24.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.396+0000 7ff330c00700 1 -- 172.21.15.1:0/103937597 <== mgr.24441 v2:172.21.15.1:6800/4056372531 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7ff334115940 con 0x7ff31c06d0e0 2024-02-28T16:35:24.384 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr" 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "2/21 daemons upgraded", 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mon daemons", 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:35:24.385 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:35:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.399+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff31c06d0e0 msgr2=0x7ff31c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.399+0000 7ff33bc00700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff31c06d0e0 0x7ff31c06f590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff328006fd0 tx=0x7ff328008040 comp rx=0 tx=0).stop 2024-02-28T16:35:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.399+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 msgr2=0x7ff3341161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:24.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.399+0000 7ff33bc00700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 0x7ff3341161b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff320009150 tx=0x7ff320009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.400+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 shutdown_connections 2024-02-28T16:35:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.400+0000 7ff33bc00700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff334105160 0x7ff334115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.400+0000 7ff33bc00700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7ff31c06d0e0 0x7ff31c06f590 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.400+0000 7ff33bc00700 1 --2- 172.21.15.1:0/103937597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff334105b00 0x7ff3341161b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.400+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 >> 172.21.15.1:0/103937597 conn(0x7ff3340fad30 msgr2=0x7ff3340fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:24.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.400+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 shutdown_connections 2024-02-28T16:35:24.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.401+0000 7ff33bc00700 1 -- 172.21.15.1:0/103937597 wait complete. 2024-02-28T16:35:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.548+0000 7f2134800700 1 -- 172.21.15.1:0/689613282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c1038f0 msgr2=0x7f212c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.548+0000 7f2134800700 1 --2- 172.21.15.1:0/689613282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c1038f0 0x7f212c0fcf60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f211c004940 tx=0x7f211c009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.548+0000 7f2134800700 1 -- 172.21.15.1:0/689613282 shutdown_connections 2024-02-28T16:35:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.548+0000 7f2134800700 1 --2- 172.21.15.1:0/689613282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c1038f0 0x7f212c0fcf60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.548+0000 7f2134800700 1 --2- 172.21.15.1:0/689613282 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f212c102f50 0x7f212c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.548+0000 7f2134800700 1 -- 172.21.15.1:0/689613282 >> 172.21.15.1:0/689613282 conn(0x7f212c0f8ae0 msgr2=0x7f212c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.548+0000 7f2134800700 1 -- 172.21.15.1:0/689613282 shutdown_connections 2024-02-28T16:35:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.548+0000 7f2134800700 1 -- 172.21.15.1:0/689613282 wait complete. 2024-02-28T16:35:24.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.549+0000 7f2134800700 1 Processor -- start 2024-02-28T16:35:24.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.549+0000 7f2134800700 1 -- start start 2024-02-28T16:35:24.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.550+0000 7f2134800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c102f50 0x7f212c113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:24.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.550+0000 7f2134800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f212c1038f0 0x7f212c113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:24.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.550+0000 7f2134800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c114590 con 0x7f212c102f50 2024-02-28T16:35:24.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.550+0000 7f2134800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c10d8c0 con 0x7f212c1038f0 2024-02-28T16:35:24.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.550+0000 7f2131e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c102f50 0x7f212c113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:24.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.550+0000 7f2131e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c102f50 0x7f212c113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45062/0 (socket says 172.21.15.1:45062) 2024-02-28T16:35:24.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.550+0000 7f2131e00700 1 -- 172.21.15.1:0/3385555614 learned_addr learned my addr 172.21.15.1:0/3385555614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:24.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2131e00700 1 -- 172.21.15.1:0/3385555614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f212c1038f0 msgr2=0x7f212c113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:24.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2131400700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f212c1038f0 0x7f212c113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:24.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2131e00700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f212c1038f0 0x7f212c113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2131e00700 1 -- 172.21.15.1:0/3385555614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c005040 con 0x7f212c102f50 2024-02-28T16:35:24.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2131e00700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c102f50 0x7f212c113a00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f212000eb40 tx=0x7f212000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:24.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2126000700 1 -- 172.21.15.1:0/3385555614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2120007cc0 con 0x7f212c102f50 2024-02-28T16:35:24.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2126000700 1 -- 172.21.15.1:0/3385555614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f212000f040 con 0x7f212c102f50 2024-02-28T16:35:24.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2126000700 1 -- 172.21.15.1:0/3385555614 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2120009620 con 0x7f212c102f50 2024-02-28T16:35:24.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212c10db40 con 0x7f212c102f50 2024-02-28T16:35:24.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.551+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212c10e080 con 0x7f212c102f50 2024-02-28T16:35:24.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.553+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212c105420 con 0x7f212c102f50 2024-02-28T16:35:24.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.554+0000 7f2126000700 1 -- 172.21.15.1:0/3385555614 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 91197+0+0 (secure 0 0 0) 0x7f2120009780 con 0x7f212c102f50 2024-02-28T16:35:24.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.555+0000 7f2126000700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f211406d0f0 0x7f211406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:24.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.555+0000 7f2126000700 1 -- 172.21.15.1:0/3385555614 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5980+0+0 (secure 0 0 0) 0x7f212008c170 con 0x7f212c102f50 2024-02-28T16:35:24.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.556+0000 7f2131400700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f211406d0f0 0x7f211406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:24.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.556+0000 7f2131400700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f211406d0f0 0x7f211406f5a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f211c009fd0 tx=0x7f211c009e30 comp rx=0 tx=0).ready entity=mgr.24441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:24.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.560+0000 7f2126000700 1 -- 172.21.15.1:0/3385555614 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f2120059e30 con 0x7f212c102f50 2024-02-28T16:35:24.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.875+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f212c04bf60 con 0x7f212c102f50 2024-02-28T16:35:24.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.876+0000 7f2126000700 1 -- 172.21.15.1:0/3385555614 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f21200599c0 con 0x7f212c102f50 2024-02-28T16:35:24.864 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:35:24.864 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:35:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:35:24.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f211406d0f0 msgr2=0x7f211406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:24.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f211406d0f0 0x7f211406f5a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f211c009fd0 tx=0x7f211c009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:24.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c102f50 msgr2=0x7f212c113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:24.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c102f50 0x7f212c113a00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f212000eb40 tx=0x7f212000b040 comp rx=0 tx=0).stop 2024-02-28T16:35:24.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 shutdown_connections 2024-02-28T16:35:24.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:6800/4056372531,v1:172.21.15.1:6801/4056372531] conn(0x7f211406d0f0 0x7f211406f5a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f212c102f50 0x7f212c113a00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 --2- 172.21.15.1:0/3385555614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f212c1038f0 0x7f212c113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:24.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 >> 172.21.15.1:0/3385555614 conn(0x7f212c0f8ae0 msgr2=0x7f212c0fa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:24.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.879+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 shutdown_connections 2024-02-28T16:35:24.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:24.880+0000 7f2134800700 1 -- 172.21.15.1:0/3385555614 wait complete. 2024-02-28T16:35:25.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:24 smithi066 ceph-mon[36236]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:25.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:24 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/3385555614' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:35:25.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:24 smithi001 ceph-mon[28991]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:25.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:24 smithi001 ceph-mon[28991]: from='client.? 172.21.15.1:0/3385555614' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:35:26.194 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:26 smithi001 ceph-mon[28991]: pgmap v113: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:26.194 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:26 smithi001 ceph-mon[28991]: Upgrade: Updating mon.smithi001 2024-02-28T16:35:26.194 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:26 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:26.194 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:26 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:35:26.194 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:26 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:35:26.194 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:26 smithi001 ceph-mon[28991]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:35:26.195 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:26 smithi001 ceph-mon[28991]: Deploying daemon mon.smithi001 on smithi001 2024-02-28T16:35:26.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:26 smithi066 ceph-mon[36236]: pgmap v113: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:26 smithi066 ceph-mon[36236]: Upgrade: Updating mon.smithi001 2024-02-28T16:35:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:26 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:26 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:35:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:26 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:35:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:26 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:35:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:26 smithi066 ceph-mon[36236]: Deploying daemon mon.smithi001 on smithi001 2024-02-28T16:35:28.017 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:27 smithi001 systemd[1]: Stopping Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:35:28.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:28 smithi001 ceph-mon[28991]: pgmap v114: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:28.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:28 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[28987]: 2024-02-28T16:35:28.159+0000 7f94a3400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi001 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:35:28.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:28 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[28987]: 2024-02-28T16:35:28.159+0000 7f94a3400700 -1 mon.smithi001@0(leader) e2 *** Got Signal Terminated *** 2024-02-28T16:35:28.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:28 smithi066 ceph-mon[36236]: pgmap v114: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:28.651 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:28 smithi001 podman[112772]: 2024-02-28 16:35:28.448058047 +0000 UTC m=+0.484328355 container died 4d5c15828f14e4d1fcfbdd566b05a1e164b76046386d9ea2029cede00c0fd2fc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, RELEASE=quincy-c0f02e6, org.label-schema.vendor=CentOS) 2024-02-28T16:35:29.206 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:28 smithi001 podman[112772]: 2024-02-28 16:35:28.93403631 +0000 UTC m=+0.970306611 container cleanup 4d5c15828f14e4d1fcfbdd566b05a1e164b76046386d9ea2029cede00c0fd2fc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=quincy-c0f02e6, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-02-28T16:35:29.207 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:28 smithi001 bash[112772]: ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001 2024-02-28T16:35:29.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:29 smithi001 podman[112798]: 2024-02-28 16:35:29.218411315 +0000 UTC m=+0.748449012 container remove 4d5c15828f14e4d1fcfbdd566b05a1e164b76046386d9ea2029cede00c0fd2fc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c0f02e6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-28T16:35:29.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:29 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service: Deactivated successfully. 2024-02-28T16:35:29.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:29 smithi001 systemd[1]: Stopped Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:35:29.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:29 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service: Consumed 31.548s CPU time. 2024-02-28T16:35:30.260 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:29 smithi001 systemd[1]: Starting Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:35:30.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 podman[112922]: 2024-02-28 16:35:30.177613636 +0000 UTC m=+0.033471004 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:35:30.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 podman[112922]: 2024-02-28 16:35:30.277871699 +0000 UTC m=+0.133729091 container create d04080cee273b377ffa020a7a6d8cb90785749c8b688b39b513c71cf1becd2a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-28T16:35:31.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 podman[112922]: 2024-02-28 16:35:30.582010795 +0000 UTC m=+0.437868230 container init d04080cee273b377ffa020a7a6d8cb90785749c8b688b39b513c71cf1becd2a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True) 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 podman[112922]: 2024-02-28 16:35:30.593103374 +0000 UTC m=+0.448960786 container start d04080cee273b377ffa020a7a6d8cb90785749c8b688b39b513c71cf1becd2a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: set uid:gid to 167:167 (ceph:ceph) 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev), process ceph-mon, pid 2 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: pidfile_write: ignore empty --pid-file 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: load: jerasure load: lrc 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: RocksDB version: 7.9.2 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Git sha 0 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Compile date 2024-02-23 16:20:17 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: DB SUMMARY 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: DB Session ID: 8WXAXPTP0PDV50HENM9Z 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: CURRENT file: CURRENT 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: IDENTITY file: IDENTITY 2024-02-28T16:35:31.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1354 Bytes 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi001/store.db dir, Total Num: 1, files: 000036.sst 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi001/store.db: 000034.log size: 931458 ; 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.error_if_exists: 0 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.create_if_missing: 0 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.paranoid_checks: 1 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.env: 0x563d8964db20 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.fs: PosixFileSystem 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.info_log: 0x563d8b00f7c0 2024-02-28T16:35:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_file_opening_threads: 16 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.statistics: (nil) 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.use_fsync: 0 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_log_file_size: 0 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.keep_log_file_num: 1000 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.recycle_log_file_num: 0 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.allow_fallocate: 1 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.allow_mmap_reads: 0 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.allow_mmap_writes: 0 2024-02-28T16:35:31.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.use_direct_reads: 0 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.create_missing_column_families: 0 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.db_log_dir: 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.wal_dir: 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-28T16:35:31.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.advise_random_on_open: 1 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.db_write_buffer_size: 0 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.write_buffer_manager: 0x563d8afe3400 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.rate_limiter: (nil) 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.wal_recovery_mode: 2 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.enable_thread_tracking: 0 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.enable_pipelined_write: 0 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.unordered_write: 0 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-28T16:35:31.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-28T16:35:31.026 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-28T16:35:31.026 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-28T16:35:31.026 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.row_cache: None 2024-02-28T16:35:31.026 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.wal_filter: None 2024-02-28T16:35:31.026 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.allow_ingest_behind: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.two_write_queues: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.manual_wal_flush: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.wal_compression: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.atomic_flush: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-28T16:35:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.log_readahead_size: 0 2024-02-28T16:35:31.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-28T16:35:31.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.best_efforts_recovery: 0 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.allow_data_in_errors: 0 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.db_host_id: __hostname__ 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_background_jobs: 2 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_background_compactions: -1 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_subcompactions: 1 2024-02-28T16:35:31.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_total_wal_size: 0 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_open_files: -1 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bytes_per_sync: 0 2024-02-28T16:35:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_readahead_size: 0 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_background_flushes: -1 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Compression algorithms supported: 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: kZSTD supported: 0 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: kXpressCompression supported: 0 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: kLZ4HCCompression supported: 1 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: kZlibCompression supported: 1 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: kSnappyCompression supported: 1 2024-02-28T16:35:31.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: kLZ4Compression supported: 1 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: kBZip2Compression supported: 0 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi001/store.db/MANIFEST-000015 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.merge_operator: 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_filter: None 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_filter_factory: None 2024-02-28T16:35:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.sst_partitioner_factory: None 2024-02-28T16:35:31.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-28T16:35:31.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-28T16:35:31.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563d8b00f8e0) 2024-02-28T16:35:31.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: cache_index_and_filter_blocks: 1 2024-02-28T16:35:31.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-28T16:35:31.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-28T16:35:31.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: pin_top_level_index_and_filter: 1 2024-02-28T16:35:31.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_type: 0 2024-02-28T16:35:31.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: data_block_index_type: 0 2024-02-28T16:35:31.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_shortening: 1 2024-02-28T16:35:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-28T16:35:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: checksum: 4 2024-02-28T16:35:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: no_block_cache: 0 2024-02-28T16:35:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache: 0x563d8b013090 2024-02-28T16:35:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_name: BinnedLRUCache 2024-02-28T16:35:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_options: 2024-02-28T16:35:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: capacity : 536870912 2024-02-28T16:35:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: num_shard_bits : 4 2024-02-28T16:35:31.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: strict_capacity_limit : 0 2024-02-28T16:35:31.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: high_pri_pool_ratio: 0.000 2024-02-28T16:35:31.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_compressed: (nil) 2024-02-28T16:35:31.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: persistent_cache: (nil) 2024-02-28T16:35:31.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_size: 4096 2024-02-28T16:35:31.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_size_deviation: 10 2024-02-28T16:35:31.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_restart_interval: 16 2024-02-28T16:35:31.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_block_restart_interval: 1 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: metadata_block_size: 4096 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: partition_filters: 0 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: use_delta_encoding: 1 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: filter_policy: bloomfilter 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: whole_key_filtering: 1 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: verify_compression: 0 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: read_amp_bytes_per_bit: 0 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: format_version: 5 2024-02-28T16:35:31.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: enable_index_compression: 1 2024-02-28T16:35:31.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_align: 0 2024-02-28T16:35:31.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: max_auto_readahead_size: 262144 2024-02-28T16:35:31.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: prepopulate_block_cache: 0 2024-02-28T16:35:31.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: initial_auto_readahead_size: 8192 2024-02-28T16:35:31.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: num_file_reads_for_auto_readahead: 2 2024-02-28T16:35:31.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.write_buffer_size: 33554432 2024-02-28T16:35:31.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_write_buffer_number: 2 2024-02-28T16:35:31.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression: NoCompression 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression: Disabled 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.prefix_extractor: nullptr 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.num_levels: 7 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-28T16:35:31.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.level: 32767 2024-02-28T16:35:31.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.strategy: 0 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.enabled: false 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.target_file_size_base: 67108864 2024-02-28T16:35:31.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-28T16:35:31.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-28T16:35:31.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-28T16:35:31.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-28T16:35:31.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-28T16:35:31.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-28T16:35:31.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-28T16:35:31.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-28T16:35:31.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-28T16:35:31.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-28T16:35:31.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-28T16:35:31.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.arena_block_size: 1048576 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.disable_auto_compactions: 0 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-28T16:35:31.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.table_properties_collectors: 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.inplace_update_support: 0 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-28T16:35:31.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.bloom_locality: 0 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.max_successive_merges: 0 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.paranoid_file_checks: 0 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.force_consistency_checks: 1 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.report_bg_io_stats: 0 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.ttl: 2592000 2024-02-28T16:35:31.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.enable_blob_files: false 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.min_blob_size: 0 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.blob_file_size: 268435456 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-28T16:35:31.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.blob_file_starting_level: 0 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 36.sst 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi001/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 38, last_sequence is 16460, log_number is 34,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 34 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dd7c6d90-7103-461d-9dee-f19804439cf4 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709138130650549, "job": 1, "event": "recovery_started", "wal_files": [34]} 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #34 mode 2 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709138130657776, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 39, "file_size": 852898, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 16461, "largest_seqno": 17674, "table_properties": {"data_size": 847757, "index_size": 2846, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1285, "raw_key_size": 12343, "raw_average_key_size": 24, "raw_value_size": 837124, "raw_average_value_size": 1641, "num_data_blocks": 129, "num_entries": 510, "num_filter_entries": 510, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709138130, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dd7c6d90-7103-461d-9dee-f19804439cf4", "db_session_id": "8WXAXPTP0PDV50HENM9Z", "orig_file_number": 39, "seqno_to_time_mapping": "N/A"}} 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709138130657939, "job": 1, "event": "recovery_finished"} 2024-02-28T16:35:31.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/version_set.cc:5047] Creating manifest 41 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi001/store.db/000034.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563d8b102000 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: DB pointer 0x563d8b0ee000 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** DB Stats ** 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-28T16:35:31.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: L0 1/0 832.91 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 196.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: L6 1/0 7.65 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Sum 2/0 8.46 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 196.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 196.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-28T16:35:31.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 196.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-28T16:35:31.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative compaction: 0.00 GB write, 67.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval compaction: 0.00 GB write, 67.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Block cache BinnedLRUCache@0x563d8b013090#2 capacity: 512.00 MB usage: 4.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: starting mon.smithi001 rank 0 at public addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] at bind addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi001 fsid c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: mon.smithi001@-1(???) e2 preinit fsid c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:35:31.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: mon.smithi001@-1(???).mds e16 new map 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: mon.smithi001@-1(???).mds e16 print_map 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: e16 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: legacy client fscid: 1 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Filesystem 'cephfs' (1) 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: fs_name cephfs 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: epoch 16 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:35:31.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: modified 2024-02-28T16:30:03.631200+0000 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: tableserver 0 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: root 0 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: session_timeout 60 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: session_autoclose 300 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: max_file_size 1099511627776 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: max_xattr_size 65536 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: required_client_features {} 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: last_failure 0 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: last_failure_osd_epoch 0 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: max_mds 2 2024-02-28T16:35:31.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: in 0,1 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: up {0=14464,1=24261} 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: failed 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: damaged 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: stopped 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: data_pools [3] 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: metadata_pool 2 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: inline_data enabled 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: balancer 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: bal_rank_mask -1 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: standby_count_wanted 1 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: [mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:31.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: [mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 export targets 0 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Standby daemons: 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: [mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: [mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: mon.smithi001@-1(???).osd e47 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: mon.smithi001@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: mon.smithi001@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: mon.smithi001@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-02-28T16:35:31.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 ceph-mon[112940]: mon.smithi001@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-02-28T16:35:31.057 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 bash[112922]: d04080cee273b377ffa020a7a6d8cb90785749c8b688b39b513c71cf1becd2a4 2024-02-28T16:35:31.057 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:30 smithi001 systemd[1]: Started Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:35:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:35:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:35:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: pgmap v115: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: mon.smithi001 calling monitor election 2024-02-28T16:35:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: mon.smithi001 is new leader, mons smithi001,smithi066 in quorum (ranks 0,1) 2024-02-28T16:35:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: monmap e2: 2 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],smithi066=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-28T16:35:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:35:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: osdmap e47: 6 total, 6 up, 6 in 2024-02-28T16:35:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: mgrmap e33: smithi001.rmthtx(active, since 3m), standbys: smithi066.ztraxe 2024-02-28T16:35:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:35:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:35:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:35:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: from='mgr.24441 ' entity='' 2024-02-28T16:35:31.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:31 smithi066 ceph-mon[36236]: mgrmap e34: smithi001.rmthtx(active, since 3m), standbys: smithi066.ztraxe 2024-02-28T16:35:31.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:35:31.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:35:31.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: pgmap v115: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: from='mgr.24441 172.21.15.1:0/4150415075' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: mon.smithi001 calling monitor election 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: mon.smithi001 is new leader, mons smithi001,smithi066 in quorum (ranks 0,1) 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: monmap e2: 2 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],smithi066=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: osdmap e47: 6 total, 6 up, 6 in 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: mgrmap e33: smithi001.rmthtx(active, since 3m), standbys: smithi066.ztraxe 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: from='mgr.24441 ' entity='' 2024-02-28T16:35:31.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:31 smithi001 ceph-mon[112940]: mgrmap e34: smithi001.rmthtx(active, since 3m), standbys: smithi066.ztraxe 2024-02-28T16:35:35.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:35 smithi001 ceph-mon[112940]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:35:35.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:35 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: Active manager daemon smithi001.rmthtx restarted 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: Activating manager daemon smithi001.rmthtx 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: osdmap e48: 6 total, 6 up, 6 in 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: mgrmap e35: smithi001.rmthtx(active, starting, since 0.0170035s), standbys: smithi066.ztraxe 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:35:40.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi066.ztraxe", "id": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: Standby manager daemon smithi066.ztraxe restarted 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:35:40.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:40 smithi066 ceph-mon[36236]: Manager daemon smithi001.rmthtx is now available 2024-02-28T16:35:40.857 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: Active manager daemon smithi001.rmthtx restarted 2024-02-28T16:35:40.857 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: Activating manager daemon smithi001.rmthtx 2024-02-28T16:35:40.857 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: osdmap e48: 6 total, 6 up, 6 in 2024-02-28T16:35:40.857 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: mgrmap e35: smithi001.rmthtx(active, starting, since 0.0170035s), standbys: smithi066.ztraxe 2024-02-28T16:35:40.857 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:35:40.857 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:35:40.857 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi001.rmthtx", "id": "smithi001.rmthtx"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr metadata", "who": "smithi066.ztraxe", "id": "smithi066.ztraxe"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-28T16:35:40.858 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: Standby manager daemon smithi066.ztraxe restarted 2024-02-28T16:35:40.859 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: Standby manager daemon smithi066.ztraxe started 2024-02-28T16:35:40.859 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:40 smithi001 ceph-mon[112940]: Manager daemon smithi001.rmthtx is now available 2024-02-28T16:35:41.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/1577085926' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/1577085926' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/1577085926' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.? 172.21.15.66:0/1577085926' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/trash_purge_schedule"}]: dispatch 2024-02-28T16:35:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:41 smithi066 ceph-mon[36236]: mgrmap e36: smithi001.rmthtx(active, since 1.0239s), standbys: smithi066.ztraxe 2024-02-28T16:35:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.? 172.21.15.66:0/1577085926' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/crt"}]: dispatch 2024-02-28T16:35:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.? 172.21.15.66:0/1577085926' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-28T16:35:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.? 172.21.15.66:0/1577085926' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi066.ztraxe/key"}]: dispatch 2024-02-28T16:35:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.? 172.21.15.66:0/1577085926' entity='mgr.smithi066.ztraxe' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-28T16:35:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:35:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/mirror_snapshot_schedule"}]: dispatch 2024-02-28T16:35:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.rmthtx/trash_purge_schedule"}]: dispatch 2024-02-28T16:35:42.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:41 smithi001 ceph-mon[112940]: mgrmap e36: smithi001.rmthtx(active, since 1.0239s), standbys: smithi066.ztraxe 2024-02-28T16:35:42.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:42 smithi066 ceph-mon[36236]: [28/Feb/2024:16:35:41] ENGINE Bus STARTING 2024-02-28T16:35:42.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:42 smithi066 ceph-mon[36236]: [28/Feb/2024:16:35:42] ENGINE Serving on https://172.21.15.1:7150 2024-02-28T16:35:42.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:42 smithi066 ceph-mon[36236]: [28/Feb/2024:16:35:42] ENGINE Serving on http://172.21.15.1:8765 2024-02-28T16:35:42.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:42 smithi066 ceph-mon[36236]: [28/Feb/2024:16:35:42] ENGINE Bus STARTED 2024-02-28T16:35:43.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:42 smithi001 ceph-mon[112940]: [28/Feb/2024:16:35:41] ENGINE Bus STARTING 2024-02-28T16:35:43.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:42 smithi001 ceph-mon[112940]: [28/Feb/2024:16:35:42] ENGINE Serving on https://172.21.15.1:7150 2024-02-28T16:35:43.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:42 smithi001 ceph-mon[112940]: [28/Feb/2024:16:35:42] ENGINE Serving on http://172.21.15.1:8765 2024-02-28T16:35:43.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:42 smithi001 ceph-mon[112940]: [28/Feb/2024:16:35:42] ENGINE Bus STARTED 2024-02-28T16:35:43.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:43 smithi066 ceph-mon[36236]: pgmap v4: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:43.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:43 smithi066 ceph-mon[36236]: mgrmap e37: smithi001.rmthtx(active, since 2s), standbys: smithi066.ztraxe 2024-02-28T16:35:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:43 smithi001 ceph-mon[112940]: pgmap v4: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:43 smithi001 ceph-mon[112940]: mgrmap e37: smithi001.rmthtx(active, since 2s), standbys: smithi066.ztraxe 2024-02-28T16:35:45.955 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:45 smithi001 ceph-mon[112940]: pgmap v5: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:45.956 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:45 smithi001 ceph-mon[112940]: mgrmap e38: smithi001.rmthtx(active, since 4s), standbys: smithi066.ztraxe 2024-02-28T16:35:45.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:45 smithi066 ceph-mon[36236]: pgmap v5: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:45.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:45 smithi066 ceph-mon[36236]: mgrmap e38: smithi001.rmthtx(active, since 4s), standbys: smithi066.ztraxe 2024-02-28T16:35:47.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:46 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:47.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:46 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:47.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:46 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:47.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:46 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:48.060 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:47 smithi001 ceph-mon[112940]: pgmap v6: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:48.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:47 smithi066 ceph-mon[36236]: pgmap v6: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:35:49.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:48 smithi001 ceph-mon[112940]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:35:49.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:49.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:49.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:49.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:49.109 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:48 smithi066 ceph-mon[36236]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:35:49.109 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:48 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:49.109 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:48 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:49.109 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:48 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:49.109 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:48 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:50.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:50 smithi066 ceph-mon[36236]: pgmap v7: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-28T16:35:50.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:50 smithi001 ceph-mon[112940]: pgmap v7: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-28T16:35:51.431 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:51 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:51.431 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:51 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:51.479 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:51 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:51.479 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:51 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:52.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:52 smithi066 ceph-mon[36236]: pgmap v8: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-28T16:35:52.511 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:52 smithi001 ceph-mon[112940]: pgmap v8: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-28T16:35:54.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:54 smithi066 ceph-mon[36236]: pgmap v9: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-28T16:35:54.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:54 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:54.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:54 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:54.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:54 smithi001 ceph-mon[112940]: pgmap v9: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-28T16:35:54.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:54 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:54.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:54 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.077+0000 7f75cf000700 1 -- 172.21.15.1:0/218149136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 msgr2=0x7f75c80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.077+0000 7f75cf000700 1 --2- 172.21.15.1:0/218149136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fe3b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f75b8004970 tx=0x7f75b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.078+0000 7f75cf000700 1 -- 172.21.15.1:0/218149136 shutdown_connections 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.078+0000 7f75cf000700 1 --2- 172.21.15.1:0/218149136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c8107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.078+0000 7f75cf000700 1 --2- 172.21.15.1:0/218149136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fe3b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.078+0000 7f75cf000700 1 -- 172.21.15.1:0/218149136 >> 172.21.15.1:0/218149136 conn(0x7f75c8071d90 msgr2=0x7f75c8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.078+0000 7f75cf000700 1 -- 172.21.15.1:0/218149136 shutdown_connections 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.078+0000 7f75cf000700 1 -- 172.21.15.1:0/218149136 wait complete. 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.079+0000 7f75cf000700 1 Processor -- start 2024-02-28T16:35:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.079+0000 7f75cf000700 1 -- start start 2024-02-28T16:35:55.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.079+0000 7f75cf000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75cf000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75cf000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80fc9e0 con 0x7f75c80fdfe0 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75cf000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80fcb50 con 0x7f75c80fe8f0 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75c7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75c7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33534/0 (socket says 172.21.15.1:33534) 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75c7e00700 1 -- 172.21.15.1:0/1990786404 learned_addr learned my addr 172.21.15.1:0/1990786404 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75c7400700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75c7e00700 1 -- 172.21.15.1:0/1990786404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 msgr2=0x7f75c80fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75c7e00700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.080+0000 7f75c7e00700 1 -- 172.21.15.1:0/1990786404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8005040 con 0x7f75c80fdfe0 2024-02-28T16:35:55.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.081+0000 7f75c7e00700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fbea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f75b8004f50 tx=0x7f75b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:55.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.081+0000 7f75bfe00700 1 -- 172.21.15.1:0/1990786404 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b800e040 con 0x7f75c80fdfe0 2024-02-28T16:35:55.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.081+0000 7f75bfe00700 1 -- 172.21.15.1:0/1990786404 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75b8004290 con 0x7f75c80fdfe0 2024-02-28T16:35:55.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.081+0000 7f75bfe00700 1 -- 172.21.15.1:0/1990786404 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b801c630 con 0x7f75c80fdfe0 2024-02-28T16:35:55.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.081+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c8119c90 con 0x7f75c80fdfe0 2024-02-28T16:35:55.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.081+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c811a0f0 con 0x7f75c80fdfe0 2024-02-28T16:35:55.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.083+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c81046c0 con 0x7f75c80fdfe0 2024-02-28T16:35:55.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.083+0000 7f75bfe00700 1 -- 172.21.15.1:0/1990786404 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f75b800bc10 con 0x7f75c80fdfe0 2024-02-28T16:35:55.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.084+0000 7f75bfe00700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d900 0x7f75ac06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.085+0000 7f75bfe00700 1 -- 172.21.15.1:0/1990786404 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f75b808e460 con 0x7f75c80fdfe0 2024-02-28T16:35:55.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.085+0000 7f75c7400700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d900 0x7f75ac06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.086+0000 7f75c7400700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d900 0x7f75ac06fdb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f75c80fd8a0 tx=0x7f75b400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:55.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.091+0000 7f75bfe00700 1 -- 172.21.15.1:0/1990786404 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f75b8057ba0 con 0x7f75c80fdfe0 2024-02-28T16:35:55.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.377+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75c80fd5e0 con 0x7f75ac06d900 2024-02-28T16:35:55.367 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:55 smithi001 ceph-mon[112940]: Detected new or changed devices on smithi066 2024-02-28T16:35:55.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.381+0000 7f75bfe00700 1 -- 172.21.15.1:0/1990786404 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f75c80fd5e0 con 0x7f75ac06d900 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.383+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d900 msgr2=0x7f75ac06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.383+0000 7f75cf000700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d900 0x7f75ac06fdb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f75c80fd8a0 tx=0x7f75b400b040 comp rx=0 tx=0).stop 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.383+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 msgr2=0x7f75c80fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.384+0000 7f75cf000700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fbea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f75b8004f50 tx=0x7f75b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.384+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 shutdown_connections 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.384+0000 7f75cf000700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d900 0x7f75ac06fdb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.384+0000 7f75cf000700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fbea0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.384+0000 7f75cf000700 1 --2- 172.21.15.1:0/1990786404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.384+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 >> 172.21.15.1:0/1990786404 conn(0x7f75c8071d90 msgr2=0x7f75c8101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:55.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.384+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 shutdown_connections 2024-02-28T16:35:55.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.384+0000 7f75cf000700 1 -- 172.21.15.1:0/1990786404 wait complete. 2024-02-28T16:35:55.390 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:35:55.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:55 smithi066 ceph-mon[36236]: Detected new or changed devices on smithi066 2024-02-28T16:35:55.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.538+0000 7f369d400700 1 -- 172.21.15.1:0/3718120161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 msgr2=0x7f36980ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.538+0000 7f369d400700 1 --2- 172.21.15.1:0/3718120161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 0x7f36980ff170 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3680004940 tx=0x7f3680009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:55.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.539+0000 7f369d400700 1 -- 172.21.15.1:0/3718120161 shutdown_connections 2024-02-28T16:35:55.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.539+0000 7f369d400700 1 --2- 172.21.15.1:0/3718120161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 0x7f36980ff170 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.539+0000 7f369d400700 1 --2- 172.21.15.1:0/3718120161 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3698105160 0x7f3698105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.539+0000 7f369d400700 1 -- 172.21.15.1:0/3718120161 >> 172.21.15.1:0/3718120161 conn(0x7f36980fad30 msgr2=0x7f36980fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:55.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.539+0000 7f369d400700 1 -- 172.21.15.1:0/3718120161 shutdown_connections 2024-02-28T16:35:55.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.539+0000 7f369d400700 1 -- 172.21.15.1:0/3718120161 wait complete. 2024-02-28T16:35:55.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.540+0000 7f369d400700 1 Processor -- start 2024-02-28T16:35:55.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.540+0000 7f369d400700 1 -- start start 2024-02-28T16:35:55.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.540+0000 7f369d400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3698105160 0x7f3698115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.540+0000 7f369d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 0x7f3698116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.540+0000 7f369d400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3698116860 con 0x7f3698105b00 2024-02-28T16:35:55.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.540+0000 7f369d400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369810fbb0 con 0x7f3698105160 2024-02-28T16:35:55.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.541+0000 7f368fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 0x7f3698116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.541+0000 7f368fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 0x7f3698116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33552/0 (socket says 172.21.15.1:33552) 2024-02-28T16:35:55.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.541+0000 7f368fe00700 1 -- 172.21.15.1:0/4081497277 learned_addr learned my addr 172.21.15.1:0/4081497277 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:55.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.541+0000 7f3696800700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3698105160 0x7f3698115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.541+0000 7f368fe00700 1 -- 172.21.15.1:0/4081497277 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3698105160 msgr2=0x7f3698115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.541+0000 7f368fe00700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3698105160 0x7f3698115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.541+0000 7f368fe00700 1 -- 172.21.15.1:0/4081497277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3680005040 con 0x7f3698105b00 2024-02-28T16:35:55.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.541+0000 7f368fe00700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 0x7f3698116210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3680009100 tx=0x7f3680009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:55.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.542+0000 7f368ea00700 1 -- 172.21.15.1:0/4081497277 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f368000e040 con 0x7f3698105b00 2024-02-28T16:35:55.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.542+0000 7f368ea00700 1 -- 172.21.15.1:0/4081497277 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3680004290 con 0x7f3698105b00 2024-02-28T16:35:55.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.542+0000 7f368ea00700 1 -- 172.21.15.1:0/4081497277 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f368001c3f0 con 0x7f3698105b00 2024-02-28T16:35:55.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.542+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369810fe30 con 0x7f3698105b00 2024-02-28T16:35:55.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.542+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3698110370 con 0x7f3698105b00 2024-02-28T16:35:55.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.543+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3698107630 con 0x7f3698105b00 2024-02-28T16:35:55.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.544+0000 7f368ea00700 1 -- 172.21.15.1:0/4081497277 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f368000bbf0 con 0x7f3698105b00 2024-02-28T16:35:55.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.545+0000 7f368ea00700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f367c06d900 0x7f367c06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.545+0000 7f368ea00700 1 -- 172.21.15.1:0/4081497277 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f368008e3b0 con 0x7f3698105b00 2024-02-28T16:35:55.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.545+0000 7f3696800700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f367c06d900 0x7f367c06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.546+0000 7f3696800700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f367c06d900 0x7f367c06fdb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3688004990 tx=0x7f3688009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:55.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.550+0000 7f368ea00700 1 -- 172.21.15.1:0/4081497277 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3680057af0 con 0x7f3698105b00 2024-02-28T16:35:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.805+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f369804bed0 con 0x7f367c06d900 2024-02-28T16:35:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.808+0000 7f368ea00700 1 -- 172.21.15.1:0/4081497277 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f369804bed0 con 0x7f367c06d900 2024-02-28T16:35:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.812+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f367c06d900 msgr2=0x7f367c06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.812+0000 7f369d400700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f367c06d900 0x7f367c06fdb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3688004990 tx=0x7f3688009ee0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.812+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 msgr2=0x7f3698116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.812+0000 7f369d400700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 0x7f3698116210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3680009100 tx=0x7f3680009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:55.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.813+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 shutdown_connections 2024-02-28T16:35:55.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.813+0000 7f369d400700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f367c06d900 0x7f367c06fdb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.813+0000 7f369d400700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3698105160 0x7f3698115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.813+0000 7f369d400700 1 --2- 172.21.15.1:0/4081497277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698105b00 0x7f3698116210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.813+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 >> 172.21.15.1:0/4081497277 conn(0x7f36980fad30 msgr2=0x7f36980fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:55.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.813+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 shutdown_connections 2024-02-28T16:35:55.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.813+0000 7f369d400700 1 -- 172.21.15.1:0/4081497277 wait complete. 2024-02-28T16:35:55.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.966+0000 7f7d3d600700 1 -- 172.21.15.1:0/491978703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 msgr2=0x7f7d380fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.966+0000 7f7d3d600700 1 --2- 172.21.15.1:0/491978703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 0x7f7d380fcd50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7d24004940 tx=0x7f7d24009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:55.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.966+0000 7f7d3d600700 1 -- 172.21.15.1:0/491978703 shutdown_connections 2024-02-28T16:35:55.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.966+0000 7f7d3d600700 1 --2- 172.21.15.1:0/491978703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 0x7f7d380fcd50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.966+0000 7f7d3d600700 1 --2- 172.21.15.1:0/491978703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d38102d40 0x7f7d38103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.966+0000 7f7d3d600700 1 -- 172.21.15.1:0/491978703 >> 172.21.15.1:0/491978703 conn(0x7f7d380f88b0 msgr2=0x7f7d380facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:55.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.966+0000 7f7d3d600700 1 -- 172.21.15.1:0/491978703 shutdown_connections 2024-02-28T16:35:55.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.966+0000 7f7d3d600700 1 -- 172.21.15.1:0/491978703 wait complete. 2024-02-28T16:35:55.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.967+0000 7f7d3d600700 1 Processor -- start 2024-02-28T16:35:55.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.967+0000 7f7d3d600700 1 -- start start 2024-02-28T16:35:55.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.968+0000 7f7d3d600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d38102d40 0x7f7d38115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.968+0000 7f7d3d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 0x7f7d38115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.968+0000 7f7d3d600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d381165e0 con 0x7f7d381036e0 2024-02-28T16:35:55.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.968+0000 7f7d3d600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3810f910 con 0x7f7d38102d40 2024-02-28T16:35:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.968+0000 7f7d37400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 0x7f7d38115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.968+0000 7f7d37400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 0x7f7d38115f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33576/0 (socket says 172.21.15.1:33576) 2024-02-28T16:35:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.968+0000 7f7d37400700 1 -- 172.21.15.1:0/1153108545 learned_addr learned my addr 172.21.15.1:0/1153108545 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d37e00700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d38102d40 0x7f7d38115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d37400700 1 -- 172.21.15.1:0/1153108545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d38102d40 msgr2=0x7f7d38115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d37400700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d38102d40 0x7f7d38115a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d37400700 1 -- 172.21.15.1:0/1153108545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d24005040 con 0x7f7d381036e0 2024-02-28T16:35:55.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d37400700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 0x7f7d38115f90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009150 tx=0x7f7d24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d2fe00700 1 -- 172.21.15.1:0/1153108545 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2400e040 con 0x7f7d381036e0 2024-02-28T16:35:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d2fe00700 1 -- 172.21.15.1:0/1153108545 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d2400bbf0 con 0x7f7d381036e0 2024-02-28T16:35:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d2fe00700 1 -- 172.21.15.1:0/1153108545 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2401c630 con 0x7f7d381036e0 2024-02-28T16:35:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.969+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3810fb90 con 0x7f7d381036e0 2024-02-28T16:35:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.970+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d38110150 con 0x7f7d381036e0 2024-02-28T16:35:55.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.971+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3805f5c0 con 0x7f7d381036e0 2024-02-28T16:35:55.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.972+0000 7f7d2fe00700 1 -- 172.21.15.1:0/1153108545 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f7d2400bd60 con 0x7f7d381036e0 2024-02-28T16:35:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.973+0000 7f7d2fe00700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7d1c06d900 0x7f7d1c06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:55.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.974+0000 7f7d2fe00700 1 -- 172.21.15.1:0/1153108545 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f7d2408e530 con 0x7f7d381036e0 2024-02-28T16:35:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.974+0000 7f7d37e00700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7d1c06d900 0x7f7d1c06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.974+0000 7f7d37e00700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7d1c06d900 0x7f7d1c06fdb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7d28004990 tx=0x7f7d28009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:55.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:55.980+0000 7f7d2fe00700 1 -- 172.21.15.1:0/1153108545 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d2405b720 con 0x7f7d381036e0 2024-02-28T16:35:56.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.218+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d3804bed0 con 0x7f7d1c06d900 2024-02-28T16:35:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.251+0000 7f7d2fe00700 1 -- 172.21.15.1:0/1153108545 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7d3804bed0 con 0x7f7d1c06d900 2024-02-28T16:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (2m) 8s ago 16m 14.6M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (16m) 8s ago 16m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (15m) 10s ago 15m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:35:56.240 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (98s) 8s ago 15m 74.7M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (9m) 8s ago 9m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (9m) 8s ago 9m 13.8M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (8m) 10s ago 8m 13.9M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (9m) 10s ago 9m 120M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (4m) 8s ago 18m 474M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (3m) 10s ago 15m 447M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (26s) 8s ago 18m 29.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (14m) 10s ago 14m 47.3M 2048M 17.2.7-535-gc0f02e60 1c37ed371348 7e952d59ccc8 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (3m) 8s ago 16m 7633k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (3m) 10s ago 15m 8232k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (13m) 8s ago 13m 207M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (13m) 8s ago 13m 209M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (12m) 8s ago 12m 169M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (12m) 10s ago 12m 222M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:35:56.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (11m) 10s ago 11m 213M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (11m) 10s ago 11m 188M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:35:56.242 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (2m) 8s ago 15m 48.7M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7d1c06d900 msgr2=0x7f7d1c06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7d1c06d900 0x7f7d1c06fdb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7d28004990 tx=0x7f7d28009ee0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 msgr2=0x7f7d38115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 0x7f7d38115f90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009150 tx=0x7f7d24009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 shutdown_connections 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7d1c06d900 0x7f7d1c06fdb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7d38102d40 0x7f7d38115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 --2- 172.21.15.1:0/1153108545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d381036e0 0x7f7d38115f90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 >> 172.21.15.1:0/1153108545 conn(0x7f7d380f88b0 msgr2=0x7f7d380fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 shutdown_connections 2024-02-28T16:35:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.254+0000 7f7d3d600700 1 -- 172.21.15.1:0/1153108545 wait complete. 2024-02-28T16:35:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.412+0000 7febe7a00700 1 -- 172.21.15.1:0/3642931443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fe8f0 msgr2=0x7febe0107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.412+0000 7febe7a00700 1 --2- 172.21.15.1:0/3642931443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fe8f0 0x7febe0107050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7febcc004970 tx=0x7febcc009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.413+0000 7febe7a00700 1 -- 172.21.15.1:0/3642931443 shutdown_connections 2024-02-28T16:35:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.413+0000 7febe7a00700 1 --2- 172.21.15.1:0/3642931443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fe8f0 0x7febe0107050 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.413+0000 7febe7a00700 1 --2- 172.21.15.1:0/3642931443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7febe00fdfe0 0x7febe00fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.413+0000 7febe7a00700 1 -- 172.21.15.1:0/3642931443 >> 172.21.15.1:0/3642931443 conn(0x7febe0071d90 msgr2=0x7febe0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.413+0000 7febe7a00700 1 -- 172.21.15.1:0/3642931443 shutdown_connections 2024-02-28T16:35:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.413+0000 7febe7a00700 1 -- 172.21.15.1:0/3642931443 wait complete. 2024-02-28T16:35:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.414+0000 7febe7a00700 1 Processor -- start 2024-02-28T16:35:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.414+0000 7febe7a00700 1 -- start start 2024-02-28T16:35:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.415+0000 7febe7a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fdfe0 0x7febe01139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.415+0000 7febe7a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7febe00fe8f0 0x7febe0113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.415+0000 7febe7a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe0114580 con 0x7febe00fdfe0 2024-02-28T16:35:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.415+0000 7febe7a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe010d8b0 con 0x7febe00fe8f0 2024-02-28T16:35:56.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.415+0000 7febe5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fdfe0 0x7febe01139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:56.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.415+0000 7febe5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fdfe0 0x7febe01139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33586/0 (socket says 172.21.15.1:33586) 2024-02-28T16:35:56.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.415+0000 7febe5000700 1 -- 172.21.15.1:0/2981464274 learned_addr learned my addr 172.21.15.1:0/2981464274 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:56.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.415+0000 7febdfe00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7febe00fe8f0 0x7febe0113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:56.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.416+0000 7febe5000700 1 -- 172.21.15.1:0/2981464274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7febe00fe8f0 msgr2=0x7febe0113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.416+0000 7febe5000700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7febe00fe8f0 0x7febe0113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.416+0000 7febe5000700 1 -- 172.21.15.1:0/2981464274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc005040 con 0x7febe00fdfe0 2024-02-28T16:35:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.416+0000 7febe5000700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fdfe0 0x7febe01139f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7febd400eb40 tx=0x7febd400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.416+0000 7febdcc00700 1 -- 172.21.15.1:0/2981464274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd4007cc0 con 0x7febe00fdfe0 2024-02-28T16:35:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.416+0000 7febdcc00700 1 -- 172.21.15.1:0/2981464274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febd400f040 con 0x7febe00fdfe0 2024-02-28T16:35:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.416+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe010db30 con 0x7febe00fdfe0 2024-02-28T16:35:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.416+0000 7febdcc00700 1 -- 172.21.15.1:0/2981464274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd400ca10 con 0x7febe00fdfe0 2024-02-28T16:35:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.417+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe010e040 con 0x7febe00fdfe0 2024-02-28T16:35:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.418+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe01046c0 con 0x7febe00fdfe0 2024-02-28T16:35:56.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.419+0000 7febdcc00700 1 -- 172.21.15.1:0/2981464274 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7febd400cb70 con 0x7febe00fdfe0 2024-02-28T16:35:56.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.420+0000 7febdcc00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febc806d900 0x7febc806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:56.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.420+0000 7febdcc00700 1 -- 172.21.15.1:0/2981464274 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7febd408cc30 con 0x7febe00fdfe0 2024-02-28T16:35:56.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.421+0000 7febdfe00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febc806d900 0x7febc806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.422+0000 7febdfe00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febc806d900 0x7febc806fdb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7febcc000c00 tx=0x7febcc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.425+0000 7febdcc00700 1 -- 172.21.15.1:0/2981464274 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7febd4059ed0 con 0x7febe00fdfe0 2024-02-28T16:35:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:56 smithi066 ceph-mon[36236]: pgmap v10: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-28T16:35:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:56 smithi066 ceph-mon[36236]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:56 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:56 smithi001 ceph-mon[112940]: pgmap v10: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-28T16:35:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:56 smithi001 ceph-mon[112940]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:35:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.760+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7febe00738b0 con 0x7febe00fdfe0 2024-02-28T16:35:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.760+0000 7febdcc00700 1 -- 172.21.15.1:0/2981464274 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7febd4059a60 con 0x7febe00fdfe0 2024-02-28T16:35:56.749 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:35:56.750 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 11, 2024-02-28T16:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T16:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:35:56.751 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:35:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febc806d900 msgr2=0x7febc806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febc806d900 0x7febc806fdb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7febcc000c00 tx=0x7febcc009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fdfe0 msgr2=0x7febe01139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fdfe0 0x7febe01139f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7febd400eb40 tx=0x7febd400b040 comp rx=0 tx=0).stop 2024-02-28T16:35:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 shutdown_connections 2024-02-28T16:35:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febc806d900 0x7febc806fdb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe00fdfe0 0x7febe01139f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 --2- 172.21.15.1:0/2981464274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7febe00fe8f0 0x7febe0113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.763+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 >> 172.21.15.1:0/2981464274 conn(0x7febe0071d90 msgr2=0x7febe0101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.764+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 shutdown_connections 2024-02-28T16:35:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.764+0000 7febe7a00700 1 -- 172.21.15.1:0/2981464274 wait complete. 2024-02-28T16:35:56.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.925+0000 7f21b5e00700 1 -- 172.21.15.1:0/1350147780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fbcc0 msgr2=0x7f21b00fc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:56.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.925+0000 7f21b5e00700 1 --2- 172.21.15.1:0/1350147780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fbcc0 0x7f21b00fc090 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f21a0004940 tx=0x7f21a0009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:56.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.925+0000 7f21b5e00700 1 -- 172.21.15.1:0/1350147780 shutdown_connections 2024-02-28T16:35:56.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.925+0000 7f21b5e00700 1 --2- 172.21.15.1:0/1350147780 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f21b00fc5d0 0x7f21b0109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.925+0000 7f21b5e00700 1 --2- 172.21.15.1:0/1350147780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fbcc0 0x7f21b00fc090 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.925+0000 7f21b5e00700 1 -- 172.21.15.1:0/1350147780 >> 172.21.15.1:0/1350147780 conn(0x7f21b00f7a10 msgr2=0x7f21b00f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:56.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.926+0000 7f21b5e00700 1 -- 172.21.15.1:0/1350147780 shutdown_connections 2024-02-28T16:35:56.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.926+0000 7f21b5e00700 1 -- 172.21.15.1:0/1350147780 wait complete. 2024-02-28T16:35:56.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.927+0000 7f21b5e00700 1 Processor -- start 2024-02-28T16:35:56.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.927+0000 7f21b5e00700 1 -- start start 2024-02-28T16:35:56.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.927+0000 7f21b5e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f21b00fbcc0 0x7f21b0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:56.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.927+0000 7f21b5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fc5d0 0x7f21b01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:56.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.927+0000 7f21b5e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0116810 con 0x7f21b00fc5d0 2024-02-28T16:35:56.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.927+0000 7f21b5e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b010fb40 con 0x7f21b00fbcc0 2024-02-28T16:35:56.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.928+0000 7f21ae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fc5d0 0x7f21b01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:56.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.928+0000 7f21ae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fc5d0 0x7f21b01161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33600/0 (socket says 172.21.15.1:33600) 2024-02-28T16:35:56.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.928+0000 7f21ae000700 1 -- 172.21.15.1:0/2685367112 learned_addr learned my addr 172.21.15.1:0/2685367112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:56.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.928+0000 7f21aea00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f21b00fbcc0 0x7f21b0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:56.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.928+0000 7f21ae000700 1 -- 172.21.15.1:0/2685367112 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f21b00fbcc0 msgr2=0x7f21b0115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:56.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.928+0000 7f21ae000700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f21b00fbcc0 0x7f21b0115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:56.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.928+0000 7f21ae000700 1 -- 172.21.15.1:0/2685367112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0005040 con 0x7f21b00fc5d0 2024-02-28T16:35:56.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.929+0000 7f21ae000700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fc5d0 0x7f21b01161c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f219c002a50 tx=0x7f219c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:56.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.929+0000 7f21a6a00700 1 -- 172.21.15.1:0/2685367112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f219c007cc0 con 0x7f21b00fc5d0 2024-02-28T16:35:56.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.929+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b010fe20 con 0x7f21b00fc5d0 2024-02-28T16:35:56.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.929+0000 7f21a6a00700 1 -- 172.21.15.1:0/2685367112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f219c00f040 con 0x7f21b00fc5d0 2024-02-28T16:35:56.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.929+0000 7f21a6a00700 1 -- 172.21.15.1:0/2685367112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f219c00da30 con 0x7f21b00fc5d0 2024-02-28T16:35:56.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.929+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b0110360 con 0x7f21b00fc5d0 2024-02-28T16:35:56.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.931+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b0106a60 con 0x7f21b00fc5d0 2024-02-28T16:35:56.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.932+0000 7f21a6a00700 1 -- 172.21.15.1:0/2685367112 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f219c00db90 con 0x7f21b00fc5d0 2024-02-28T16:35:56.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.933+0000 7f21a6a00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f219406d900 0x7f219406fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:56.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.933+0000 7f21a6a00700 1 -- 172.21.15.1:0/2685367112 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f219c08cd30 con 0x7f21b00fc5d0 2024-02-28T16:35:56.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.933+0000 7f21aea00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f219406d900 0x7f219406fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:56.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.934+0000 7f21aea00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f219406d900 0x7f219406fdb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f21a0009fd0 tx=0x7f21a0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:56.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:56.938+0000 7f21a6a00700 1 -- 172.21.15.1:0/2685367112 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f219c059fc0 con 0x7f21b00fc5d0 2024-02-28T16:35:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.255+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f21b00fb610 con 0x7f21b00fc5d0 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.256+0000 7f21a6a00700 1 -- 172.21.15.1:0/2685367112 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7f219c059b50 con 0x7f21b00fc5d0 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:35:57.245 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:35:57.246 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:35:57.247 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:57.248 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:35:57.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f219406d900 msgr2=0x7f219406fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:57.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f219406d900 0x7f219406fdb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f21a0009fd0 tx=0x7f21a0009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fc5d0 msgr2=0x7f21b01161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fc5d0 0x7f21b01161c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f219c002a50 tx=0x7f219c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 shutdown_connections 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f219406d900 0x7f219406fdb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f21b00fbcc0 0x7f21b0115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 --2- 172.21.15.1:0/2685367112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21b00fc5d0 0x7f21b01161c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 >> 172.21.15.1:0/2685367112 conn(0x7f21b00f7a10 msgr2=0x7f21b00f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 shutdown_connections 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.259+0000 7f21b5e00700 1 -- 172.21.15.1:0/2685367112 wait complete. 2024-02-28T16:35:57.249 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:35:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.427+0000 7f9aee400700 1 -- 172.21.15.1:0/1557847765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 msgr2=0x7f9ae8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.427+0000 7f9aee400700 1 --2- 172.21.15.1:0/1557847765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 0x7f9ae8103320 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8004940 tx=0x7f9ad8009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.427+0000 7f9aee400700 1 -- 172.21.15.1:0/1557847765 shutdown_connections 2024-02-28T16:35:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.427+0000 7f9aee400700 1 --2- 172.21.15.1:0/1557847765 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae81038f0 0x7f9ae80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.427+0000 7f9aee400700 1 --2- 172.21.15.1:0/1557847765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 0x7f9ae8103320 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.427+0000 7f9aee400700 1 -- 172.21.15.1:0/1557847765 >> 172.21.15.1:0/1557847765 conn(0x7f9ae80f8ae0 msgr2=0x7f9ae80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.427+0000 7f9aee400700 1 -- 172.21.15.1:0/1557847765 shutdown_connections 2024-02-28T16:35:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.427+0000 7f9aee400700 1 -- 172.21.15.1:0/1557847765 wait complete. 2024-02-28T16:35:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.428+0000 7f9aee400700 1 Processor -- start 2024-02-28T16:35:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.428+0000 7f9aee400700 1 -- start start 2024-02-28T16:35:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.429+0000 7f9aee400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 0x7f9ae8115be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.429+0000 7f9aee400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae81038f0 0x7f9ae8116120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.429+0000 7f9aee400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae8116770 con 0x7f9ae8102f50 2024-02-28T16:35:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.429+0000 7f9aee400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae810faa0 con 0x7f9ae81038f0 2024-02-28T16:35:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.429+0000 7f9ae7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 0x7f9ae8115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.429+0000 7f9ae7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 0x7f9ae8115be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33622/0 (socket says 172.21.15.1:33622) 2024-02-28T16:35:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.429+0000 7f9ae7400700 1 -- 172.21.15.1:0/1868322538 learned_addr learned my addr 172.21.15.1:0/1868322538 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.429+0000 7f9ae6a00700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae81038f0 0x7f9ae8116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.430+0000 7f9ae7400700 1 -- 172.21.15.1:0/1868322538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae81038f0 msgr2=0x7f9ae8116120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.430+0000 7f9ae7400700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae81038f0 0x7f9ae8116120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.430+0000 7f9ae7400700 1 -- 172.21.15.1:0/1868322538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8005040 con 0x7f9ae8102f50 2024-02-28T16:35:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.430+0000 7f9ae7400700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 0x7f9ae8115be0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009180 tx=0x7f9ad8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.430+0000 7f9adf400700 1 -- 172.21.15.1:0/1868322538 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad8021070 con 0x7f9ae8102f50 2024-02-28T16:35:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.430+0000 7f9adf400700 1 -- 172.21.15.1:0/1868322538 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ad8013d00 con 0x7f9ae8102f50 2024-02-28T16:35:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.430+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae810fd20 con 0x7f9ae8102f50 2024-02-28T16:35:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.430+0000 7f9adf400700 1 -- 172.21.15.1:0/1868322538 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad800e030 con 0x7f9ae8102f50 2024-02-28T16:35:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.431+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae8110260 con 0x7f9ae8102f50 2024-02-28T16:35:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.432+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae80fc6e0 con 0x7f9ae8102f50 2024-02-28T16:35:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.433+0000 7f9adf400700 1 -- 172.21.15.1:0/1868322538 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f9ad801ca40 con 0x7f9ae8102f50 2024-02-28T16:35:57.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.434+0000 7f9adf400700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9acc06d900 0x7f9acc06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:57.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.434+0000 7f9adf400700 1 -- 172.21.15.1:0/1868322538 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f9ad808e450 con 0x7f9ae8102f50 2024-02-28T16:35:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.434+0000 7f9ae6a00700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9acc06d900 0x7f9acc06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.435+0000 7f9ae6a00700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9acc06d900 0x7f9acc06fdb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8111410 tx=0x7f9ad400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:57.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.439+0000 7f9adf400700 1 -- 172.21.15.1:0/1868322538 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9ad805b780 con 0x7f9ae8102f50 2024-02-28T16:35:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: Detected new or changed devices on smithi001 2024-02-28T16:35:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:35:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:35:57.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:35:57.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:35:57.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2981464274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:35:57.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:57 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2685367112' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:35:57.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.687+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ae81111c0 con 0x7f9acc06d900 2024-02-28T16:35:57.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.689+0000 7f9adf400700 1 -- 172.21.15.1:0/1868322538 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f9ae81111c0 con 0x7f9acc06d900 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr" 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "3/21 daemons upgraded", 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "", 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:35:57.678 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:35:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.691+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9acc06d900 msgr2=0x7f9acc06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.691+0000 7f9aee400700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9acc06d900 0x7f9acc06fdb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8111410 tx=0x7f9ad400b040 comp rx=0 tx=0).stop 2024-02-28T16:35:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.691+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 msgr2=0x7f9ae8115be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:57.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.691+0000 7f9aee400700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 0x7f9ae8115be0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009180 tx=0x7f9ad8009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:57.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.692+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 shutdown_connections 2024-02-28T16:35:57.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.692+0000 7f9aee400700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9acc06d900 0x7f9acc06fdb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.692+0000 7f9aee400700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae8102f50 0x7f9ae8115be0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.692+0000 7f9aee400700 1 --2- 172.21.15.1:0/1868322538 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae81038f0 0x7f9ae8116120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.692+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 >> 172.21.15.1:0/1868322538 conn(0x7f9ae80f8ae0 msgr2=0x7f9ae80fa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:57.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.692+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 shutdown_connections 2024-02-28T16:35:57.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.692+0000 7f9aee400700 1 -- 172.21.15.1:0/1868322538 wait complete. 2024-02-28T16:35:57.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:57.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:57.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: Detected new or changed devices on smithi001 2024-02-28T16:35:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:35:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:35:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: Updating smithi001:/etc/ceph/ceph.conf 2024-02-28T16:35:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: Updating smithi066:/etc/ceph/ceph.conf 2024-02-28T16:35:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2981464274' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:35:57.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:57 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2685367112' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:35:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.849+0000 7fe819400700 1 -- 172.21.15.1:0/3065964442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8141038f0 msgr2=0x7fe8140fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.849+0000 7fe819400700 1 --2- 172.21.15.1:0/3065964442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8141038f0 0x7fe8140fcf60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe800004940 tx=0x7fe800009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.850+0000 7fe819400700 1 -- 172.21.15.1:0/3065964442 shutdown_connections 2024-02-28T16:35:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.850+0000 7fe819400700 1 --2- 172.21.15.1:0/3065964442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8141038f0 0x7fe8140fcf60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.850+0000 7fe819400700 1 --2- 172.21.15.1:0/3065964442 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe814102f50 0x7fe814103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.850+0000 7fe819400700 1 -- 172.21.15.1:0/3065964442 >> 172.21.15.1:0/3065964442 conn(0x7fe8140f8ae0 msgr2=0x7fe8140faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.850+0000 7fe819400700 1 -- 172.21.15.1:0/3065964442 shutdown_connections 2024-02-28T16:35:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.850+0000 7fe819400700 1 -- 172.21.15.1:0/3065964442 wait complete. 2024-02-28T16:35:57.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.851+0000 7fe819400700 1 Processor -- start 2024-02-28T16:35:57.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.851+0000 7fe819400700 1 -- start start 2024-02-28T16:35:57.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.851+0000 7fe819400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe814102f50 0x7fe814115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:57.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.851+0000 7fe819400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8141038f0 0x7fe8141161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:57.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.851+0000 7fe819400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe814116810 con 0x7fe814102f50 2024-02-28T16:35:57.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.851+0000 7fe819400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81410fb40 con 0x7fe8141038f0 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.852+0000 7fe812800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe814102f50 0x7fe814115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.852+0000 7fe812800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe814102f50 0x7fe814115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33648/0 (socket says 172.21.15.1:33648) 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.852+0000 7fe812800700 1 -- 172.21.15.1:0/2112394474 learned_addr learned my addr 172.21.15.1:0/2112394474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.852+0000 7fe811e00700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8141038f0 0x7fe8141161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.852+0000 7fe812800700 1 -- 172.21.15.1:0/2112394474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8141038f0 msgr2=0x7fe8141161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.852+0000 7fe812800700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8141038f0 0x7fe8141161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.852+0000 7fe812800700 1 -- 172.21.15.1:0/2112394474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe800005040 con 0x7fe814102f50 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.853+0000 7fe812800700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe814102f50 0x7fe814115c80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe804002a50 tx=0x7fe8040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.853+0000 7fe80aa00700 1 -- 172.21.15.1:0/2112394474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe804007cc0 con 0x7fe814102f50 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.853+0000 7fe80aa00700 1 -- 172.21.15.1:0/2112394474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe80400f040 con 0x7fe814102f50 2024-02-28T16:35:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.853+0000 7fe80aa00700 1 -- 172.21.15.1:0/2112394474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe80400da30 con 0x7fe814102f50 2024-02-28T16:35:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.853+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81410fe20 con 0x7fe814102f50 2024-02-28T16:35:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.853+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe814110330 con 0x7fe814102f50 2024-02-28T16:35:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.854+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8140fc6e0 con 0x7fe814102f50 2024-02-28T16:35:57.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.856+0000 7fe80aa00700 1 -- 172.21.15.1:0/2112394474 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7fe80400dbb0 con 0x7fe814102f50 2024-02-28T16:35:57.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.857+0000 7fe80aa00700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe7f806d900 0x7fe7f806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:35:57.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.857+0000 7fe80aa00700 1 -- 172.21.15.1:0/2112394474 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fe80408ce30 con 0x7fe814102f50 2024-02-28T16:35:57.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.858+0000 7fe811e00700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe7f806d900 0x7fe7f806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:35:57.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.858+0000 7fe811e00700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe7f806d900 0x7fe7f806fdb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe814110e70 tx=0x7fe800009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:35:57.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:57.862+0000 7fe80aa00700 1 -- 172.21.15.1:0/2112394474 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe80405a0d0 con 0x7fe814102f50 2024-02-28T16:35:58.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.184+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe81410a030 con 0x7fe814102f50 2024-02-28T16:35:58.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.185+0000 7fe80aa00700 1 -- 172.21.15.1:0/2112394474 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fe804059c60 con 0x7fe814102f50 2024-02-28T16:35:58.174 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:35:58.174 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:35:58.174 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:35:58.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe7f806d900 msgr2=0x7fe7f806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:58.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe7f806d900 0x7fe7f806fdb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe814110e70 tx=0x7fe800009e30 comp rx=0 tx=0).stop 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe814102f50 msgr2=0x7fe814115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe814102f50 0x7fe814115c80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe804002a50 tx=0x7fe8040046b0 comp rx=0 tx=0).stop 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 shutdown_connections 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe7f806d900 0x7fe7f806fdb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe814102f50 0x7fe814115c80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 --2- 172.21.15.1:0/2112394474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8141038f0 0x7fe8141161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.187+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 >> 172.21.15.1:0/2112394474 conn(0x7fe8140f8ae0 msgr2=0x7fe8140fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.188+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 shutdown_connections 2024-02-28T16:35:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:35:58.188+0000 7fe819400700 1 -- 172.21.15.1:0/2112394474 wait complete. 2024-02-28T16:35:58.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:58 smithi001 ceph-mon[112940]: pgmap v11: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-28T16:35:58.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:58 smithi001 ceph-mon[112940]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:35:58.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:58 smithi001 ceph-mon[112940]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:35:58.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:58 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2112394474' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:35:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:58 smithi066 ceph-mon[36236]: pgmap v11: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-28T16:35:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:58 smithi066 ceph-mon[36236]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:35:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:58 smithi066 ceph-mon[36236]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-28T16:35:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:58 smithi066 ceph-mon[36236]: from='client.? 172.21.15.1:0/2112394474' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:35:59.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:59 smithi066 ceph-mon[36236]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:35:59.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:59 smithi066 ceph-mon[36236]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:59.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:59 smithi066 ceph-mon[36236]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:35:59.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:59 smithi066 ceph-mon[36236]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:35:59.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:59 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:59.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:35:59 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:59 smithi001 ceph-mon[112940]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:35:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:59 smithi001 ceph-mon[112940]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:35:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:59 smithi001 ceph-mon[112940]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-02-28T16:35:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:59 smithi001 ceph-mon[112940]: Updating smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:35:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:59 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:35:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:35:59 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:00 smithi066 ceph-mon[36236]: pgmap v12: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-28T16:36:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:00 smithi066 ceph-mon[36236]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:36:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:00 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:00 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:00 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:00 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:36:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:00 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:36:00.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:00 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-28T16:36:00.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:00 smithi001 ceph-mon[112940]: pgmap v12: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-28T16:36:00.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:00 smithi001 ceph-mon[112940]: Updating smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-28T16:36:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:36:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:36:00.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-28T16:36:02.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:02 smithi066 ceph-mon[36236]: pgmap v13: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:02.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:02 smithi001 ceph-mon[112940]: pgmap v13: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:03.702 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:03 smithi066 ceph-mon[36236]: Upgrade: Updating mon.smithi066 2024-02-28T16:36:03.702 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:03 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:03.703 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:03 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:36:03.703 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:03 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:36:03.703 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:03 smithi066 ceph-mon[36236]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:03.703 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:03 smithi066 ceph-mon[36236]: Deploying daemon mon.smithi066 on smithi066 2024-02-28T16:36:03.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:03 smithi001 ceph-mon[112940]: Upgrade: Updating mon.smithi066 2024-02-28T16:36:03.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:03 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:03.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:03 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:36:03.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:03 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:36:03.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:03 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:03.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:03 smithi001 ceph-mon[112940]: Deploying daemon mon.smithi066 on smithi066 2024-02-28T16:36:04.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:04 smithi066 ceph-mon[36236]: pgmap v14: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:04.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:04 smithi001 ceph-mon[112940]: pgmap v14: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:05.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:05 smithi066 systemd[1]: Stopping Ceph mon.smithi066 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:36:05.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:05 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066[36231]: 2024-02-28T16:36:05.602+0000 7f53d7000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi066 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:36:05.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:05 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066[36231]: 2024-02-28T16:36:05.602+0000 7f53d7000700 -1 mon.smithi066@1(peon) e2 *** Got Signal Terminated *** 2024-02-28T16:36:05.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:05 smithi066 podman[85621]: 2024-02-28 16:36:05.850114276 +0000 UTC m=+0.374429557 container died 7e952d59ccc88fb0af61c30306e8c5ce5d440740e6626a2084e182e5fe934205 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, RELEASE=quincy-c0f02e6, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-02-28T16:36:06.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:06 smithi066 podman[85621]: 2024-02-28 16:36:06.294940078 +0000 UTC m=+0.819255356 container cleanup 7e952d59ccc88fb0af61c30306e8c5ce5d440740e6626a2084e182e5fe934205 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-c0f02e6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-28T16:36:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:06 smithi066 bash[85621]: ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066 2024-02-28T16:36:06.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:06 smithi066 podman[85635]: 2024-02-28 16:36:06.571215 +0000 UTC m=+0.705610981 container remove 7e952d59ccc88fb0af61c30306e8c5ce5d440740e6626a2084e182e5fe934205 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=quincy-c0f02e6, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-28T16:36:07.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:07 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi066.service: Deactivated successfully. 2024-02-28T16:36:07.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:07 smithi066 systemd[1]: Stopped Ceph mon.smithi066 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:36:07.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:07 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi066.service: Consumed 19.993s CPU time. 2024-02-28T16:36:07.891 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:07 smithi066 systemd[1]: Starting Ceph mon.smithi066 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:36:08.144 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:07 smithi066 podman[85762]: 2024-02-28 16:36:07.816346488 +0000 UTC m=+0.045730905 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:36:08.144 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:07 smithi066 podman[85762]: 2024-02-28 16:36:07.930676117 +0000 UTC m=+0.160060505 container create d17ce055575ed9becef55c650eb004794d1ea9bccaf9614b63cd819af5be0782 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-02-28T16:36:08.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 podman[85762]: 2024-02-28 16:36:08.242943055 +0000 UTC m=+0.472327487 container init d17ce055575ed9becef55c650eb004794d1ea9bccaf9614b63cd819af5be0782 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=main-1fb5d10) 2024-02-28T16:36:08.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 podman[85762]: 2024-02-28 16:36:08.254102598 +0000 UTC m=+0.483486958 container start d17ce055575ed9becef55c650eb004794d1ea9bccaf9614b63cd819af5be0782 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-28T16:36:08.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: set uid:gid to 167:167 (ceph:ceph) 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev), process ceph-mon, pid 2 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: pidfile_write: ignore empty --pid-file 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: load: jerasure load: lrc 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: RocksDB version: 7.9.2 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Git sha 0 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Compile date 2024-02-23 16:20:17 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: DB SUMMARY 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: DB Session ID: 1YJQMUBXM7MX4KC5CO32 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: CURRENT file: CURRENT 2024-02-28T16:36:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: IDENTITY file: IDENTITY 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1279 Bytes 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi066/store.db dir, Total Num: 1, files: 000030.sst 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi066/store.db: 000028.log size: 4142009 ; 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.error_if_exists: 0 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.create_if_missing: 0 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.paranoid_checks: 1 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.env: 0x55cb33cdfb20 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.fs: PosixFileSystem 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.info_log: 0x55cb34ff17c0 2024-02-28T16:36:08.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_file_opening_threads: 16 2024-02-28T16:36:08.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.statistics: (nil) 2024-02-28T16:36:08.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.use_fsync: 0 2024-02-28T16:36:08.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_log_file_size: 0 2024-02-28T16:36:08.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-28T16:36:08.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-28T16:36:08.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.keep_log_file_num: 1000 2024-02-28T16:36:08.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.recycle_log_file_num: 0 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.allow_fallocate: 1 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.allow_mmap_reads: 0 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.allow_mmap_writes: 0 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.use_direct_reads: 0 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.create_missing_column_families: 0 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.db_log_dir: 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.wal_dir: 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-28T16:36:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-28T16:36:08.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-28T16:36:08.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-28T16:36:08.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.advise_random_on_open: 1 2024-02-28T16:36:08.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.db_write_buffer_size: 0 2024-02-28T16:36:08.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.write_buffer_manager: 0x55cb34fc5400 2024-02-28T16:36:08.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-28T16:36:08.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-28T16:36:08.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.rate_limiter: (nil) 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.wal_recovery_mode: 2 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.enable_thread_tracking: 0 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.enable_pipelined_write: 0 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.unordered_write: 0 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.row_cache: None 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.wal_filter: None 2024-02-28T16:36:08.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.allow_ingest_behind: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.two_write_queues: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.manual_wal_flush: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.wal_compression: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.atomic_flush: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.log_readahead_size: 0 2024-02-28T16:36:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.best_efforts_recovery: 0 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.allow_data_in_errors: 0 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.db_host_id: __hostname__ 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_background_jobs: 2 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_background_compactions: -1 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_subcompactions: 1 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-28T16:36:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_total_wal_size: 0 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_open_files: -1 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bytes_per_sync: 0 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-28T16:36:08.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_readahead_size: 0 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_background_flushes: -1 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Compression algorithms supported: 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: kZSTD supported: 0 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: kXpressCompression supported: 0 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: kLZ4HCCompression supported: 1 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: kZlibCompression supported: 1 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: kSnappyCompression supported: 1 2024-02-28T16:36:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: kLZ4Compression supported: 1 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: kBZip2Compression supported: 0 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi066/store.db/MANIFEST-000009 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.merge_operator: 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_filter: None 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_filter_factory: None 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.sst_partitioner_factory: None 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-28T16:36:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cb34ff18e0) 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: cache_index_and_filter_blocks: 1 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: pin_top_level_index_and_filter: 1 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: index_type: 0 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: data_block_index_type: 0 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: index_shortening: 1 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: checksum: 4 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: no_block_cache: 0 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: block_cache: 0x55cb34ff5090 2024-02-28T16:36:08.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: block_cache_name: BinnedLRUCache 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: block_cache_options: 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: capacity : 536870912 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: num_shard_bits : 4 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: strict_capacity_limit : 0 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: high_pri_pool_ratio: 0.000 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: block_cache_compressed: (nil) 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: persistent_cache: (nil) 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: block_size: 4096 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: block_size_deviation: 10 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: block_restart_interval: 16 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: index_block_restart_interval: 1 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: metadata_block_size: 4096 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: partition_filters: 0 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: use_delta_encoding: 1 2024-02-28T16:36:08.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: filter_policy: bloomfilter 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: whole_key_filtering: 1 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: verify_compression: 0 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: read_amp_bytes_per_bit: 0 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: format_version: 5 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: enable_index_compression: 1 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: block_align: 0 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: max_auto_readahead_size: 262144 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: prepopulate_block_cache: 0 2024-02-28T16:36:08.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: initial_auto_readahead_size: 8192 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: num_file_reads_for_auto_readahead: 2 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.write_buffer_size: 33554432 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_write_buffer_number: 2 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression: NoCompression 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression: Disabled 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.prefix_extractor: nullptr 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.num_levels: 7 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-28T16:36:08.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.level: 32767 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.strategy: 0 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.enabled: false 2024-02-28T16:36:08.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.target_file_size_base: 67108864 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-28T16:36:08.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.arena_block_size: 1048576 2024-02-28T16:36:08.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.disable_auto_compactions: 0 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-28T16:36:08.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.table_properties_collectors: 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.inplace_update_support: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.bloom_locality: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.max_successive_merges: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.paranoid_file_checks: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.force_consistency_checks: 1 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.report_bg_io_stats: 0 2024-02-28T16:36:08.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.ttl: 2592000 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.enable_blob_files: false 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.min_blob_size: 0 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.blob_file_size: 268435456 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.blob_file_starting_level: 0 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-02-28T16:36:08.521 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi066/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 32, last_sequence is 16880, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e4bfb048-4a8d-4c6e-aa2a-47e3cd2095ed 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709138168314074, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709138168332409, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2581075, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 16885, "largest_seqno": 18873, "table_properties": {"data_size": 2573341, "index_size": 4730, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1989, "raw_key_size": 18605, "raw_average_key_size": 23, "raw_value_size": 2556860, "raw_average_value_size": 3273, "num_data_blocks": 215, "num_entries": 781, "num_filter_entries": 781, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709138168, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e4bfb048-4a8d-4c6e-aa2a-47e3cd2095ed", "db_session_id": "1YJQMUBXM7MX4KC5CO32", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709138168332611, "job": 1, "event": "recovery_finished"} 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi066/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cb350e4000 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: DB pointer 0x55cb350d0000 2024-02-28T16:36:08.522 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: ** DB Stats ** 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: ** Compaction Stats [default] ** 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-28T16:36:08.523 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: L0 1/0 2.46 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 404.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-28T16:36:08.524 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: L6 1/0 7.65 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-28T16:36:08.524 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Sum 2/0 10.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 404.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-28T16:36:08.524 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 404.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-28T16:36:08.524 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.524 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: ** Compaction Stats [default] ** 2024-02-28T16:36:08.524 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 404.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-28T16:36:08.525 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Cumulative compaction: 0.00 GB write, 97.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Interval compaction: 0.00 GB write, 97.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Block cache BinnedLRUCache@0x55cb34ff5090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.6e-05 secs_since: 0 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: starting mon.smithi066 rank 1 at public addrs [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] at bind addrs [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi066 fsid c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: mon.smithi066@-1(???) e2 preinit fsid c0118718-d654-11ee-95c3-87774f69a715 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: mon.smithi066@-1(???).mds e18 new map 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: mon.smithi066@-1(???).mds e18 print_map 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: e18 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:36:08.526 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: legacy client fscid: 1 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Filesystem 'cephfs' (1) 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: fs_name cephfs 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: epoch 18 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: tableserver 0 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: root 0 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: session_timeout 60 2024-02-28T16:36:08.527 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: session_autoclose 300 2024-02-28T16:36:08.528 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: max_file_size 1099511627776 2024-02-28T16:36:08.528 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: max_xattr_size 65536 2024-02-28T16:36:08.528 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: required_client_features {} 2024-02-28T16:36:08.528 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: last_failure 0 2024-02-28T16:36:08.528 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: last_failure_osd_epoch 0 2024-02-28T16:36:08.528 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:36:08.528 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: max_mds 2 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: in 0,1 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: up {0=14464,1=24261} 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: failed 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: damaged 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: stopped 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: data_pools [3] 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: metadata_pool 2 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: inline_data enabled 2024-02-28T16:36:08.529 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: balancer 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: bal_rank_mask -1 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: standby_count_wanted 1 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: [mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: [mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: Standby daemons: 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: [mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:36:08.530 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout: [mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:36:08.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: mon.smithi066@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-28T16:36:08.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: mon.smithi066@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-02-28T16:36:08.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: mon.smithi066@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-02-28T16:36:08.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: mon.smithi066@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-02-28T16:36:08.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 ceph-mon[85781]: mon.smithi066@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-02-28T16:36:08.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 bash[85762]: d17ce055575ed9becef55c650eb004794d1ea9bccaf9614b63cd819af5be0782 2024-02-28T16:36:08.531 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:08 smithi066 systemd[1]: Started Ceph mon.smithi066 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:36:09.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:09 smithi066 ceph-mon[85781]: pgmap v15: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:09.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:09 smithi001 ceph-mon[112940]: pgmap v15: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:10.650 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: pgmap v17: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:10.650 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: mon.smithi001 calling monitor election 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: mon.smithi066 calling monitor election 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: mon.smithi001 is new leader, mons smithi001,smithi066 in quorum (ranks 0,1) 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: monmap e3: 2 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],smithi066=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: osdmap e48: 6 total, 6 up, 6 in 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: mgrmap e38: smithi001.rmthtx(active, since 28s), standbys: smithi066.ztraxe 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:10.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: pgmap v17: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: mon.smithi001 calling monitor election 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: mon.smithi066 calling monitor election 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mon metadata", "id": "smithi066"}]: dispatch 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: mon.smithi001 is new leader, mons smithi001,smithi066 in quorum (ranks 0,1) 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: monmap e3: 2 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],smithi066=[v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi066.wfbdor=up:active} 2 up:standby 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: osdmap e48: 6 total, 6 up, 6 in 2024-02-28T16:36:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: mgrmap e38: smithi001.rmthtx(active, since 28s), standbys: smithi066.ztraxe 2024-02-28T16:36:10.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:36:10.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:36:10.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:36:10.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:10.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:10.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:36:11.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:36:11.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:36:12.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:12 smithi066 ceph-mon[85781]: pgmap v18: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:12.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:12 smithi001 ceph-mon[112940]: pgmap v18: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:14.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:14 smithi066 ceph-mon[85781]: pgmap v19: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:14.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:14 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:14.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:14 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:14.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:14 smithi001 ceph-mon[112940]: pgmap v19: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:14.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:14 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:14.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:14 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:16.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:16 smithi066 ceph-mon[85781]: pgmap v20: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:16.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:16 smithi001 ceph-mon[112940]: pgmap v20: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:17.623 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:17 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:17.623 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:17 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:17.671 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:17 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:17.671 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:17 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:18.696 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:18 smithi066 ceph-mon[85781]: pgmap v21: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:18.698 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:18 smithi001 ceph-mon[112940]: pgmap v21: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:20.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:20 smithi066 ceph-mon[85781]: pgmap v22: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:20.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:20 smithi001 ceph-mon[112940]: pgmap v22: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:22.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:22 smithi066 ceph-mon[85781]: pgmap v23: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:22.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:22 smithi001 ceph-mon[112940]: pgmap v23: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:23.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:23.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:23.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:23.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:36:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: pgmap v24: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: Reconfiguring mon.smithi001 (monmap changed)... 2024-02-28T16:36:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:36:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:36:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:23.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:23 smithi001 ceph-mon[112940]: Reconfiguring daemon mon.smithi001 on smithi001 2024-02-28T16:36:23.880 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: pgmap v24: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: Reconfiguring mon.smithi001 (monmap changed)... 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:23.881 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:23 smithi066 ceph-mon[85781]: Reconfiguring daemon mon.smithi001 on smithi001 2024-02-28T16:36:26.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: pgmap v25: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:26.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:26.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:26.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: Reconfiguring mgr.smithi001.rmthtx (monmap changed)... 2024-02-28T16:36:26.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.rmthtx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:36:26.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:36:26.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:26.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: Reconfiguring daemon mgr.smithi001.rmthtx on smithi001 2024-02-28T16:36:26.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:36:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: pgmap v25: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:26.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:26.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: Reconfiguring mgr.smithi001.rmthtx (monmap changed)... 2024-02-28T16:36:26.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.rmthtx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:36:26.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:36:26.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:26.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: Reconfiguring daemon mgr.smithi001.rmthtx on smithi001 2024-02-28T16:36:26.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:36:28.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.356+0000 7f2273c00700 1 -- 172.21.15.1:0/837257658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 msgr2=0x7f226c0fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:28.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.356+0000 7f2273c00700 1 --2- 172.21.15.1:0/837257658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 0x7f226c0fe320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2260009d10 tx=0x7f2260004d10 comp rx=0 tx=0).stop 2024-02-28T16:36:28.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.356+0000 7f2273c00700 1 -- 172.21.15.1:0/837257658 shutdown_connections 2024-02-28T16:36:28.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.356+0000 7f2273c00700 1 --2- 172.21.15.1:0/837257658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f226c0fe8f0 0x7f226c106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.356+0000 7f2273c00700 1 --2- 172.21.15.1:0/837257658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 0x7f226c0fe320 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.356+0000 7f2273c00700 1 -- 172.21.15.1:0/837257658 >> 172.21.15.1:0/837257658 conn(0x7f226c0f79d0 msgr2=0x7f226c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:28.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.356+0000 7f2273c00700 1 -- 172.21.15.1:0/837257658 shutdown_connections 2024-02-28T16:36:28.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.356+0000 7f2273c00700 1 -- 172.21.15.1:0/837257658 wait complete. 2024-02-28T16:36:28.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.357+0000 7f2273c00700 1 Processor -- start 2024-02-28T16:36:28.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.357+0000 7f2273c00700 1 -- start start 2024-02-28T16:36:28.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f2273c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 0x7f226c113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f2273c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f226c0fe8f0 0x7f226c114230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f2273c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c114910 con 0x7f226c0fdf50 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f2273c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c10db90 con 0x7f226c0fe8f0 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f2271200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 0x7f226c113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f2271200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 0x7f226c113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59988/0 (socket says 172.21.15.1:59988) 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f2271200700 1 -- 172.21.15.1:0/1407293997 learned_addr learned my addr 172.21.15.1:0/1407293997 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f226be00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f226c0fe8f0 0x7f226c114230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.358+0000 7f2271200700 1 -- 172.21.15.1:0/1407293997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f226c0fe8f0 msgr2=0x7f226c114230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.359+0000 7f2271200700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f226c0fe8f0 0x7f226c114230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.359+0000 7f2271200700 1 -- 172.21.15.1:0/1407293997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260009940 con 0x7f226c0fdf50 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.359+0000 7f226be00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f226c0fe8f0 0x7f226c114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:36:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.359+0000 7f2271200700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 0x7f226c113cf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2260008dd0 tx=0x7f2260004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.359+0000 7f2268c00700 1 -- 172.21.15.1:0/1407293997 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226000e040 con 0x7f226c0fdf50 2024-02-28T16:36:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.359+0000 7f2268c00700 1 -- 172.21.15.1:0/1407293997 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22600128e0 con 0x7f226c0fdf50 2024-02-28T16:36:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.359+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c10ddb0 con 0x7f226c0fdf50 2024-02-28T16:36:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.359+0000 7f2268c00700 1 -- 172.21.15.1:0/1407293997 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226001c8f0 con 0x7f226c0fdf50 2024-02-28T16:36:28.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.360+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c10e2f0 con 0x7f226c0fdf50 2024-02-28T16:36:28.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.361+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c104660 con 0x7f226c0fdf50 2024-02-28T16:36:28.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.362+0000 7f2268c00700 1 -- 172.21.15.1:0/1407293997 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f2260012a50 con 0x7f226c0fdf50 2024-02-28T16:36:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.363+0000 7f2268c00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f225406d900 0x7f225406fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.363+0000 7f2268c00700 1 -- 172.21.15.1:0/1407293997 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f226008edf0 con 0x7f226c0fdf50 2024-02-28T16:36:28.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.363+0000 7f226be00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f225406d900 0x7f225406fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:28.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.364+0000 7f226be00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f225406d900 0x7f225406fdb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f2258008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:28.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.369+0000 7f2268c00700 1 -- 172.21.15.1:0/1407293997 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f226005bf60 con 0x7f226c0fdf50 2024-02-28T16:36:28.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:28 smithi001 ceph-mon[112940]: pgmap v26: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:28.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:28 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:28.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:28 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:28.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:28 smithi001 ceph-mon[112940]: Reconfiguring crash.smithi001 (monmap changed)... 2024-02-28T16:36:28.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:28 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:36:28.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:28 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:28.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:28 smithi001 ceph-mon[112940]: Reconfiguring daemon crash.smithi001 on smithi001 2024-02-28T16:36:28.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.626+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f226c04bf40 con 0x7f225406d900 2024-02-28T16:36:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.628+0000 7f2268c00700 1 -- 172.21.15.1:0/1407293997 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f226c04bf40 con 0x7f225406d900 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f225406d900 msgr2=0x7f225406fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f225406d900 0x7f225406fdb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f2258008040 comp rx=0 tx=0).stop 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 msgr2=0x7f226c113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 0x7f226c113cf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2260008dd0 tx=0x7f2260004d10 comp rx=0 tx=0).stop 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 shutdown_connections 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f225406d900 0x7f225406fdb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f226c0fdf50 0x7f226c113cf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 --2- 172.21.15.1:0/1407293997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f226c0fe8f0 0x7f226c114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 >> 172.21.15.1:0/1407293997 conn(0x7f226c0f79d0 msgr2=0x7f226c1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 shutdown_connections 2024-02-28T16:36:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.632+0000 7f2273c00700 1 -- 172.21.15.1:0/1407293997 wait complete. 2024-02-28T16:36:28.638 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:36:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:28 smithi066 ceph-mon[85781]: pgmap v26: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:28 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:28 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:28 smithi066 ceph-mon[85781]: Reconfiguring crash.smithi001 (monmap changed)... 2024-02-28T16:36:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:28 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:36:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:28 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:28 smithi066 ceph-mon[85781]: Reconfiguring daemon crash.smithi001 on smithi001 2024-02-28T16:36:28.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.797+0000 7fcbb6c00700 1 -- 172.21.15.1:0/1856053821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 msgr2=0x7fcbb0070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:28.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.797+0000 7fcbb6c00700 1 --2- 172.21.15.1:0/1856053821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 0x7fcbb0070ce0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c004970 tx=0x7fcb9c009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:28.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.797+0000 7fcbb6c00700 1 -- 172.21.15.1:0/1856053821 shutdown_connections 2024-02-28T16:36:28.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.797+0000 7fcbb6c00700 1 --2- 172.21.15.1:0/1856053821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 0x7fcbb0070ce0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.797+0000 7fcbb6c00700 1 --2- 172.21.15.1:0/1856053821 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbb0071f60 0x7fcbb0070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.797+0000 7fcbb6c00700 1 -- 172.21.15.1:0/1856053821 >> 172.21.15.1:0/1856053821 conn(0x7fcbb00f88b0 msgr2=0x7fcbb00facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:28.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.797+0000 7fcbb6c00700 1 -- 172.21.15.1:0/1856053821 shutdown_connections 2024-02-28T16:36:28.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.798+0000 7fcbb6c00700 1 -- 172.21.15.1:0/1856053821 wait complete. 2024-02-28T16:36:28.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.798+0000 7fcbb6c00700 1 Processor -- start 2024-02-28T16:36:28.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.799+0000 7fcbb6c00700 1 -- start start 2024-02-28T16:36:28.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.799+0000 7fcbb6c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 0x7fcbb0119dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:28.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.799+0000 7fcbb6c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbb0071f60 0x7fcbb011a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:28.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.799+0000 7fcbb6c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb011a950 con 0x7fcbb0070870 2024-02-28T16:36:28.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.799+0000 7fcbb6c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb0113c60 con 0x7fcbb0071f60 2024-02-28T16:36:28.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.799+0000 7fcbb5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 0x7fcbb0119dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:28.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.799+0000 7fcbb5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 0x7fcbb0119dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60014/0 (socket says 172.21.15.1:60014) 2024-02-28T16:36:28.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.799+0000 7fcbb5800700 1 -- 172.21.15.1:0/2371185143 learned_addr learned my addr 172.21.15.1:0/2371185143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:36:28.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.800+0000 7fcbb4e00700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbb0071f60 0x7fcbb011a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:28.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.800+0000 7fcbb5800700 1 -- 172.21.15.1:0/2371185143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbb0071f60 msgr2=0x7fcbb011a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:28.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.800+0000 7fcbb5800700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbb0071f60 0x7fcbb011a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:28.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.800+0000 7fcbb5800700 1 -- 172.21.15.1:0/2371185143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c005040 con 0x7fcbb0070870 2024-02-28T16:36:28.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.800+0000 7fcbb5800700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 0x7fcbb0119dc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcba4002a50 tx=0x7fcba40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:28.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.800+0000 7fcbad600700 1 -- 172.21.15.1:0/2371185143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcba4007cc0 con 0x7fcbb0070870 2024-02-28T16:36:28.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.801+0000 7fcbad600700 1 -- 172.21.15.1:0/2371185143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcba400f040 con 0x7fcbb0070870 2024-02-28T16:36:28.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.801+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb0113f40 con 0x7fcbb0070870 2024-02-28T16:36:28.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.801+0000 7fcbad600700 1 -- 172.21.15.1:0/2371185143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcba4009640 con 0x7fcbb0070870 2024-02-28T16:36:28.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.801+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb0114480 con 0x7fcbb0070870 2024-02-28T16:36:28.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.802+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb00fe1f0 con 0x7fcbb0070870 2024-02-28T16:36:28.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.803+0000 7fcbad600700 1 -- 172.21.15.1:0/2371185143 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7fcba40097a0 con 0x7fcbb0070870 2024-02-28T16:36:28.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.804+0000 7fcbad600700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb9806d900 0x7fcb9806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:28.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.804+0000 7fcbad600700 1 -- 172.21.15.1:0/2371185143 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fcba408ce60 con 0x7fcbb0070870 2024-02-28T16:36:28.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.805+0000 7fcbb4e00700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb9806d900 0x7fcb9806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:28.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.805+0000 7fcbb4e00700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb9806d900 0x7fcb9806fdb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0115660 tx=0x7fcb9c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:28.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:28.810+0000 7fcbad600700 1 -- 172.21.15.1:0/2371185143 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcba405a070 con 0x7fcbb0070870 2024-02-28T16:36:29.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.057+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbb004bf40 con 0x7fcb9806d900 2024-02-28T16:36:29.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.059+0000 7fcbad600700 1 -- 172.21.15.1:0/2371185143 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fcbb004bf40 con 0x7fcb9806d900 2024-02-28T16:36:29.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb9806d900 msgr2=0x7fcb9806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb9806d900 0x7fcb9806fdb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0115660 tx=0x7fcb9c009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 msgr2=0x7fcbb0119dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 0x7fcbb0119dc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcba4002a50 tx=0x7fcba40046b0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 shutdown_connections 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb9806d900 0x7fcb9806fdb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbb0070870 0x7fcbb0119dc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 --2- 172.21.15.1:0/2371185143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbb0071f60 0x7fcbb011a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.061+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 >> 172.21.15.1:0/2371185143 conn(0x7fcbb00f88b0 msgr2=0x7fcbb00fa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.062+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 shutdown_connections 2024-02-28T16:36:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.062+0000 7fcbb6c00700 1 -- 172.21.15.1:0/2371185143 wait complete. 2024-02-28T16:36:29.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.206+0000 7f88bba00700 1 -- 172.21.15.1:0/2324855025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fb360 msgr2=0x7f88b40fb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.206+0000 7f88bba00700 1 --2- 172.21.15.1:0/2324855025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fb360 0x7f88b40fb730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f88a8004970 tx=0x7f88a8009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:29.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.206+0000 7f88bba00700 1 -- 172.21.15.1:0/2324855025 shutdown_connections 2024-02-28T16:36:29.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.206+0000 7f88bba00700 1 --2- 172.21.15.1:0/2324855025 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88b40fbd00 0x7f88b4106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.206+0000 7f88bba00700 1 --2- 172.21.15.1:0/2324855025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fb360 0x7f88b40fb730 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.206+0000 7f88bba00700 1 -- 172.21.15.1:0/2324855025 >> 172.21.15.1:0/2324855025 conn(0x7f88b4072040 msgr2=0x7f88b4072440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:29.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.206+0000 7f88bba00700 1 -- 172.21.15.1:0/2324855025 shutdown_connections 2024-02-28T16:36:29.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.206+0000 7f88bba00700 1 -- 172.21.15.1:0/2324855025 wait complete. 2024-02-28T16:36:29.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.207+0000 7f88bba00700 1 Processor -- start 2024-02-28T16:36:29.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.208+0000 7f88bba00700 1 -- start start 2024-02-28T16:36:29.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.208+0000 7f88bba00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88b40fb360 0x7f88b4115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:29.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.208+0000 7f88bba00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fbd00 0x7f88b4116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:29.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.208+0000 7f88bba00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4116780 con 0x7f88b40fbd00 2024-02-28T16:36:29.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.208+0000 7f88bba00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b410fab0 con 0x7f88b40fb360 2024-02-28T16:36:29.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.209+0000 7f88b3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fbd00 0x7f88b4116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:29.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.209+0000 7f88b3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fbd00 0x7f88b4116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60044/0 (socket says 172.21.15.1:60044) 2024-02-28T16:36:29.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.209+0000 7f88b3e00700 1 -- 172.21.15.1:0/3625457274 learned_addr learned my addr 172.21.15.1:0/3625457274 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:36:29.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.209+0000 7f88b9000700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88b40fb360 0x7f88b4115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:29.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.209+0000 7f88b3e00700 1 -- 172.21.15.1:0/3625457274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88b40fb360 msgr2=0x7f88b4115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.209+0000 7f88b3e00700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88b40fb360 0x7f88b4115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.209+0000 7f88b3e00700 1 -- 172.21.15.1:0/3625457274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a8005040 con 0x7f88b40fbd00 2024-02-28T16:36:29.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.209+0000 7f88b3e00700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fbd00 0x7f88b4116130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009b80 tx=0x7f88a0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:29.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.210+0000 7f88b0c00700 1 -- 172.21.15.1:0/3625457274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88a0007cc0 con 0x7f88b40fbd00 2024-02-28T16:36:29.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.210+0000 7f88b0c00700 1 -- 172.21.15.1:0/3625457274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88a000f040 con 0x7f88b40fbd00 2024-02-28T16:36:29.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.210+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b410fd90 con 0x7f88b40fbd00 2024-02-28T16:36:29.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.210+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b41102d0 con 0x7f88b40fbd00 2024-02-28T16:36:29.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.210+0000 7f88b0c00700 1 -- 172.21.15.1:0/3625457274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88a000dac0 con 0x7f88b40fbd00 2024-02-28T16:36:29.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.211+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b41021d0 con 0x7f88b40fbd00 2024-02-28T16:36:29.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.212+0000 7f88b0c00700 1 -- 172.21.15.1:0/3625457274 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f88a000dc20 con 0x7f88b40fbd00 2024-02-28T16:36:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.214+0000 7f88b0c00700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f889c06d900 0x7f889c06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.214+0000 7f88b0c00700 1 -- 172.21.15.1:0/3625457274 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f88a008cf80 con 0x7f88b40fbd00 2024-02-28T16:36:29.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.214+0000 7f88b9000700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f889c06d900 0x7f889c06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:29.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.215+0000 7f88b9000700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f889c06d900 0x7f889c06fdb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f88a8009fd0 tx=0x7f88a8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:29.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.218+0000 7f88b0c00700 1 -- 172.21.15.1:0/3625457274 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f88a005a0f0 con 0x7f88b40fbd00 2024-02-28T16:36:29.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.444+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f88b41158c0 con 0x7f889c06d900 2024-02-28T16:36:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.463+0000 7f88b0c00700 1 -- 172.21.15.1:0/3625457274 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f88b41158c0 con 0x7f889c06d900 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (2m) 41s ago 16m 14.6M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (17m) 41s ago 17m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (15m) 15s ago 15m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (2m) 41s ago 16m 74.7M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (9m) 41s ago 9m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (9m) 41s ago 9m 13.8M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (9m) 15s ago 9m 14.0M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:36:29.451 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (9m) 15s ago 9m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (5m) 41s ago 18m 474M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (4m) 15s ago 15m 447M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (59s) 41s ago 18m 29.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (21s) 15s ago 15m 19.7M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (3m) 41s ago 17m 7633k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (3m) 15s ago 15m 8132k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (14m) 41s ago 14m 207M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (13m) 41s ago 13m 209M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (13m) 41s ago 13m 169M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (12m) 15s ago 12m 222M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (12m) 15s ago 12m 213M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (11m) 15s ago 11m 188M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:36:29.452 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (3m) 41s ago 16m 48.7M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:36:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f889c06d900 msgr2=0x7f889c06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f889c06d900 0x7f889c06fdb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f88a8009fd0 tx=0x7f88a8009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fbd00 msgr2=0x7f88b4116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fbd00 0x7f88b4116130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009b80 tx=0x7f88a0004600 comp rx=0 tx=0).stop 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 shutdown_connections 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f889c06d900 0x7f889c06fdb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88b40fb360 0x7f88b4115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 --2- 172.21.15.1:0/3625457274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88b40fbd00 0x7f88b4116130 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.465+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 >> 172.21.15.1:0/3625457274 conn(0x7f88b4072040 msgr2=0x7f88b40ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.466+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 shutdown_connections 2024-02-28T16:36:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.466+0000 7f88bba00700 1 -- 172.21.15.1:0/3625457274 wait complete. 2024-02-28T16:36:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.616+0000 7f0177200700 1 -- 172.21.15.1:0/1673723375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105170 msgr2=0x7f0170105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.616+0000 7f0177200700 1 --2- 172.21.15.1:0/1673723375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105170 0x7f0170105540 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0160004940 tx=0x7f0160009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.616+0000 7f0177200700 1 -- 172.21.15.1:0/1673723375 shutdown_connections 2024-02-28T16:36:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.616+0000 7f0177200700 1 --2- 172.21.15.1:0/1673723375 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0170105b10 0x7f01700ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.616+0000 7f0177200700 1 --2- 172.21.15.1:0/1673723375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105170 0x7f0170105540 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.616+0000 7f0177200700 1 -- 172.21.15.1:0/1673723375 >> 172.21.15.1:0/1673723375 conn(0x7f01700face0 msgr2=0x7f01700fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.616+0000 7f0177200700 1 -- 172.21.15.1:0/1673723375 shutdown_connections 2024-02-28T16:36:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.617+0000 7f0177200700 1 -- 172.21.15.1:0/1673723375 wait complete. 2024-02-28T16:36:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.617+0000 7f0177200700 1 Processor -- start 2024-02-28T16:36:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.617+0000 7f0177200700 1 -- start start 2024-02-28T16:36:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.618+0000 7f0177200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0170105170 0x7f0170115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.618+0000 7f0177200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105b10 0x7f01701161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.618+0000 7f0177200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01701168d0 con 0x7f0170105b10 2024-02-28T16:36:29.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.618+0000 7f0177200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017010fb50 con 0x7f0170105170 2024-02-28T16:36:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.618+0000 7f016f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105b10 0x7f01701161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.618+0000 7f016f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105b10 0x7f01701161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60060/0 (socket says 172.21.15.1:60060) 2024-02-28T16:36:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.618+0000 7f016f400700 1 -- 172.21.15.1:0/1643721472 learned_addr learned my addr 172.21.15.1:0/1643721472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:36:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f016fe00700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0170105170 0x7f0170115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f016f400700 1 -- 172.21.15.1:0/1643721472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0170105170 msgr2=0x7f0170115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f016f400700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0170105170 0x7f0170115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f016f400700 1 -- 172.21.15.1:0/1643721472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0160005040 con 0x7f0170105b10 2024-02-28T16:36:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f016f400700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105b10 0x7f01701161f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f015c00b790 tx=0x7f015c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f0167e00700 1 -- 172.21.15.1:0/1643721472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f015c00f6f0 con 0x7f0170105b10 2024-02-28T16:36:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017010fdd0 con 0x7f0170105b10 2024-02-28T16:36:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f0167e00700 1 -- 172.21.15.1:0/1643721472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f015c00fd20 con 0x7f0170105b10 2024-02-28T16:36:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.619+0000 7f0167e00700 1 -- 172.21.15.1:0/1643721472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f015c0103a0 con 0x7f0170105b10 2024-02-28T16:36:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.620+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0170110310 con 0x7f0170105b10 2024-02-28T16:36:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.621+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01700fe900 con 0x7f0170105b10 2024-02-28T16:36:29.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.622+0000 7f0167e00700 1 -- 172.21.15.1:0/1643721472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f015c011400 con 0x7f0170105b10 2024-02-28T16:36:29.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.623+0000 7f0167e00700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f015406d900 0x7f015406fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:29.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.623+0000 7f0167e00700 1 -- 172.21.15.1:0/1643721472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f015c08ceb0 con 0x7f0170105b10 2024-02-28T16:36:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.623+0000 7f016fe00700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f015406d900 0x7f015406fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.624+0000 7f016fe00700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f015406d900 0x7f015406fdb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f016000b020 tx=0x7f0160009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:29.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.627+0000 7f0167e00700 1 -- 172.21.15.1:0/1643721472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f015c05a130 con 0x7f0170105b10 2024-02-28T16:36:29.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.937+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0170111290 con 0x7f0170105b10 2024-02-28T16:36:29.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.937+0000 7f0167e00700 1 -- 172.21.15.1:0/1643721472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f015c059cc0 con 0x7f0170105b10 2024-02-28T16:36:29.925 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:36:29.925 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:36:29.925 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:36:29.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:36:29.925 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:36:29.925 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:36:29.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 10, 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:36:29.926 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:36:29.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f015406d900 msgr2=0x7f015406fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f015406d900 0x7f015406fdb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f016000b020 tx=0x7f0160009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:29.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105b10 msgr2=0x7f01701161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:29.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105b10 0x7f01701161f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f015c00b790 tx=0x7f015c009220 comp rx=0 tx=0).stop 2024-02-28T16:36:29.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 shutdown_connections 2024-02-28T16:36:29.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f015406d900 0x7f015406fdb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0170105170 0x7f0170115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 --2- 172.21.15.1:0/1643721472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0170105b10 0x7f01701161f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 >> 172.21.15.1:0/1643721472 conn(0x7f01700face0 msgr2=0x7f01700fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.940+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 shutdown_connections 2024-02-28T16:36:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:29.941+0000 7f0177200700 1 -- 172.21.15.1:0/1643721472 wait complete. 2024-02-28T16:36:30.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.090+0000 7feeec000700 1 -- 172.21.15.1:0/38908705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 msgr2=0x7feee4107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.090+0000 7feeec000700 1 --2- 172.21.15.1:0/38908705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 0x7feee4107050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feed0004970 tx=0x7feed0009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:30.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.090+0000 7feeec000700 1 -- 172.21.15.1:0/38908705 shutdown_connections 2024-02-28T16:36:30.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.090+0000 7feeec000700 1 --2- 172.21.15.1:0/38908705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 0x7feee4107050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.090+0000 7feeec000700 1 --2- 172.21.15.1:0/38908705 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feee40fdfe0 0x7feee40fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.090+0000 7feeec000700 1 -- 172.21.15.1:0/38908705 >> 172.21.15.1:0/38908705 conn(0x7feee4071d90 msgr2=0x7feee4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:30.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.090+0000 7feeec000700 1 -- 172.21.15.1:0/38908705 shutdown_connections 2024-02-28T16:36:30.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.090+0000 7feeec000700 1 -- 172.21.15.1:0/38908705 wait complete. 2024-02-28T16:36:30.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.091+0000 7feeec000700 1 Processor -- start 2024-02-28T16:36:30.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.092+0000 7feeec000700 1 -- start start 2024-02-28T16:36:30.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.092+0000 7feeec000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feee40fdfe0 0x7feee4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.092+0000 7feeec000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 0x7feee4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.092+0000 7feeec000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee4116790 con 0x7feee40fe8f0 2024-02-28T16:36:30.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.092+0000 7feeec000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee410fac0 con 0x7feee40fdfe0 2024-02-28T16:36:30.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.092+0000 7feee8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 0x7feee4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:30.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.092+0000 7feee8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 0x7feee4116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60088/0 (socket says 172.21.15.1:60088) 2024-02-28T16:36:30.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.092+0000 7feee8c00700 1 -- 172.21.15.1:0/3768505911 learned_addr learned my addr 172.21.15.1:0/3768505911 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:36:30.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.093+0000 7feee9600700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feee40fdfe0 0x7feee4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.093+0000 7feee8c00700 1 -- 172.21.15.1:0/3768505911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feee40fdfe0 msgr2=0x7feee4115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.093+0000 7feee8c00700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feee40fdfe0 0x7feee4115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.093+0000 7feee8c00700 1 -- 172.21.15.1:0/3768505911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0005040 con 0x7feee40fe8f0 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.093+0000 7feee8c00700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 0x7feee4116140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feed0000c00 tx=0x7feed0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.094+0000 7feee1600700 1 -- 172.21.15.1:0/3768505911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed000e040 con 0x7feee40fe8f0 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.094+0000 7feee1600700 1 -- 172.21.15.1:0/3768505911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feed000bce0 con 0x7feee40fe8f0 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.094+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee410fd40 con 0x7feee40fe8f0 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.094+0000 7feee1600700 1 -- 172.21.15.1:0/3768505911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed001c630 con 0x7feee40fe8f0 2024-02-28T16:36:30.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.094+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee4110280 con 0x7feee40fe8f0 2024-02-28T16:36:30.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.095+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee4106b20 con 0x7feee40fe8f0 2024-02-28T16:36:30.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.096+0000 7feee1600700 1 -- 172.21.15.1:0/3768505911 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7feed00139f0 con 0x7feee40fe8f0 2024-02-28T16:36:30.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.098+0000 7feee1600700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feecc06d900 0x7feecc06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.098+0000 7feee1600700 1 -- 172.21.15.1:0/3768505911 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7feed008e330 con 0x7feee40fe8f0 2024-02-28T16:36:30.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.098+0000 7feee9600700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feecc06d900 0x7feecc06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:30.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.099+0000 7feee9600700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feecc06d900 0x7feecc06fdb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feed8004990 tx=0x7feed8009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:30.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.102+0000 7feee1600700 1 -- 172.21.15.1:0/3768505911 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feed005b660 con 0x7feee40fe8f0 2024-02-28T16:36:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.375+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feee410a030 con 0x7feee40fe8f0 2024-02-28T16:36:30.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.376+0000 7feee1600700 1 -- 172.21.15.1:0/3768505911 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7feed0004de0 con 0x7feee40fe8f0 2024-02-28T16:36:30.364 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:36:30.364 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:36:30.365 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:36:30.366 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:36:30.367 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:36:30.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feecc06d900 msgr2=0x7feecc06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feecc06d900 0x7feecc06fdb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7feed8004990 tx=0x7feed8009ee0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 msgr2=0x7feee4116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 0x7feee4116140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feed0000c00 tx=0x7feed0009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:30.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 shutdown_connections 2024-02-28T16:36:30.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feecc06d900 0x7feecc06fdb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feee40fdfe0 0x7feee4115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 --2- 172.21.15.1:0/3768505911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feee40fe8f0 0x7feee4116140 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 >> 172.21.15.1:0/3768505911 conn(0x7feee4071d90 msgr2=0x7feee40fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:30.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 shutdown_connections 2024-02-28T16:36:30.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.379+0000 7feeec000700 1 -- 172.21.15.1:0/3768505911 wait complete. 2024-02-28T16:36:30.369 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:36:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.537+0000 7fd691000700 1 -- 172.21.15.1:0/3584549180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fdfe0 msgr2=0x7fd68c0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.537+0000 7fd691000700 1 --2- 172.21.15.1:0/3584549180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fdfe0 0x7fd68c0fe3b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd67c004970 tx=0x7fd67c009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.537+0000 7fd691000700 1 -- 172.21.15.1:0/3584549180 shutdown_connections 2024-02-28T16:36:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.537+0000 7fd691000700 1 --2- 172.21.15.1:0/3584549180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd68c0fe8f0 0x7fd68c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.537+0000 7fd691000700 1 --2- 172.21.15.1:0/3584549180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fdfe0 0x7fd68c0fe3b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.537+0000 7fd691000700 1 -- 172.21.15.1:0/3584549180 >> 172.21.15.1:0/3584549180 conn(0x7fd68c071d90 msgr2=0x7fd68c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.537+0000 7fd691000700 1 -- 172.21.15.1:0/3584549180 shutdown_connections 2024-02-28T16:36:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.537+0000 7fd691000700 1 -- 172.21.15.1:0/3584549180 wait complete. 2024-02-28T16:36:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.538+0000 7fd691000700 1 Processor -- start 2024-02-28T16:36:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.538+0000 7fd691000700 1 -- start start 2024-02-28T16:36:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.538+0000 7fd691000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd68c0fdfe0 0x7fd68c0fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.538+0000 7fd691000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fe8f0 0x7fd68c0fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.538+0000 7fd691000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0fc9e0 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.538+0000 7fd691000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0fcb50 con 0x7fd68c0fdfe0 2024-02-28T16:36:30.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.539+0000 7fd689400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fe8f0 0x7fd68c0fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:30.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.539+0000 7fd689400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fe8f0 0x7fd68c0fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60098/0 (socket says 172.21.15.1:60098) 2024-02-28T16:36:30.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.539+0000 7fd689400700 1 -- 172.21.15.1:0/4269534652 learned_addr learned my addr 172.21.15.1:0/4269534652 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:36:30.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.539+0000 7fd689e00700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd68c0fdfe0 0x7fd68c0fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:30.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.539+0000 7fd689400700 1 -- 172.21.15.1:0/4269534652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd68c0fdfe0 msgr2=0x7fd68c0fbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.539+0000 7fd689400700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd68c0fdfe0 0x7fd68c0fbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.539+0000 7fd689400700 1 -- 172.21.15.1:0/4269534652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c005040 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.539+0000 7fd689e00700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd68c0fdfe0 0x7fd68c0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:36:30.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.540+0000 7fd689400700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fe8f0 0x7fd68c0fc3e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c830 tx=0x7fd678004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:30.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.540+0000 7fd682000700 1 -- 172.21.15.1:0/4269534652 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd67800a760 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.540+0000 7fd682000700 1 -- 172.21.15.1:0/4269534652 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd67800ad90 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.540+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c119c90 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.540+0000 7fd682000700 1 -- 172.21.15.1:0/4269534652 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6780113a0 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.540+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c11a170 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.542+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c1046c0 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.542+0000 7fd682000700 1 -- 172.21.15.1:0/4269534652 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7fd678012400 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.543+0000 7fd682000700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd67006d900 0x7fd67006fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.543+0000 7fd682000700 1 -- 172.21.15.1:0/4269534652 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fd67808cff0 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.543+0000 7fd689e00700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd67006d900 0x7fd67006fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:30.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.544+0000 7fd689e00700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd67006d900 0x7fd67006fdb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd67c000c00 tx=0x7fd67c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:30.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.548+0000 7fd682000700 1 -- 172.21.15.1:0/4269534652 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd6780c0d20 con 0x7fd68c0fe8f0 2024-02-28T16:36:30.672 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:36:30.672 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: pgmap v27: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:30.672 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:36:30.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:36:30.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:30.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:30.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-28T16:36:30.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:30.673 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1643721472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:36:30.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:36:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: pgmap v27: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:36:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:36:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-28T16:36:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:30.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1643721472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:36:30.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.794+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd68c0fd5e0 con 0x7fd67006d900 2024-02-28T16:36:30.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.797+0000 7fd682000700 1 -- 172.21.15.1:0/4269534652 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fd68c0fd5e0 con 0x7fd67006d900 2024-02-28T16:36:30.784 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:36:30.784 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "mon" 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "4/21 daemons upgraded", 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mon daemons", 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:36:30.785 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:36:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd67006d900 msgr2=0x7fd67006fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd67006d900 0x7fd67006fdb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd67c000c00 tx=0x7fd67c009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fe8f0 msgr2=0x7fd68c0fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fe8f0 0x7fd68c0fc3e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c830 tx=0x7fd678004470 comp rx=0 tx=0).stop 2024-02-28T16:36:30.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 shutdown_connections 2024-02-28T16:36:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd67006d900 0x7fd67006fdb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd68c0fdfe0 0x7fd68c0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 --2- 172.21.15.1:0/4269534652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd68c0fe8f0 0x7fd68c0fc3e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.799+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 >> 172.21.15.1:0/4269534652 conn(0x7fd68c071d90 msgr2=0x7fd68c101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.800+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 shutdown_connections 2024-02-28T16:36:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.800+0000 7fd691000700 1 -- 172.21.15.1:0/4269534652 wait complete. 2024-02-28T16:36:30.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.951+0000 7fb8aac00700 1 -- 172.21.15.1:0/3101663136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fc490 msgr2=0x7fb8a41091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.951+0000 7fb8aac00700 1 --2- 172.21.15.1:0/3101663136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fc490 0x7fb8a41091f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb890004940 tx=0x7fb890009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:30.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.951+0000 7fb8aac00700 1 -- 172.21.15.1:0/3101663136 shutdown_connections 2024-02-28T16:36:30.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.951+0000 7fb8aac00700 1 --2- 172.21.15.1:0/3101663136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fc490 0x7fb8a41091f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.951+0000 7fb8aac00700 1 --2- 172.21.15.1:0/3101663136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb8a40fbb80 0x7fb8a40fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.951+0000 7fb8aac00700 1 -- 172.21.15.1:0/3101663136 >> 172.21.15.1:0/3101663136 conn(0x7fb8a4073ab0 msgr2=0x7fb8a4073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.951+0000 7fb8aac00700 1 -- 172.21.15.1:0/3101663136 shutdown_connections 2024-02-28T16:36:30.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.951+0000 7fb8aac00700 1 -- 172.21.15.1:0/3101663136 wait complete. 2024-02-28T16:36:30.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.952+0000 7fb8aac00700 1 Processor -- start 2024-02-28T16:36:30.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.952+0000 7fb8aac00700 1 -- start start 2024-02-28T16:36:30.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8aac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fbb80 0x7fb8a4115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8aac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb8a40fc490 0x7fb8a4115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8aac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a41165d0 con 0x7fb8a40fbb80 2024-02-28T16:36:30.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8aac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a410f900 con 0x7fb8a40fc490 2024-02-28T16:36:30.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8a9800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fbb80 0x7fb8a4115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:30.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8a9800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fbb80 0x7fb8a4115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60110/0 (socket says 172.21.15.1:60110) 2024-02-28T16:36:30.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8a9800700 1 -- 172.21.15.1:0/769608216 learned_addr learned my addr 172.21.15.1:0/769608216 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:36:30.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8a9800700 1 -- 172.21.15.1:0/769608216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb8a40fc490 msgr2=0x7fb8a4115f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:30.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8a9800700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb8a40fc490 0x7fb8a4115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:30.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.953+0000 7fb8a9800700 1 -- 172.21.15.1:0/769608216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb890005040 con 0x7fb8a40fbb80 2024-02-28T16:36:30.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.954+0000 7fb8a9800700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fbb80 0x7fb8a4115a40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb89800c8c0 tx=0x7fb8980044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:30.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.954+0000 7fb8a1600700 1 -- 172.21.15.1:0/769608216 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb89800a980 con 0x7fb8a40fbb80 2024-02-28T16:36:30.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.954+0000 7fb8a1600700 1 -- 172.21.15.1:0/769608216 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb89800ce60 con 0x7fb8a40fbb80 2024-02-28T16:36:30.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.954+0000 7fb8a1600700 1 -- 172.21.15.1:0/769608216 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8980115e0 con 0x7fb8a40fbb80 2024-02-28T16:36:30.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.954+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a410fbe0 con 0x7fb8a40fbb80 2024-02-28T16:36:30.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.954+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a41101a0 con 0x7fb8a40fbb80 2024-02-28T16:36:30.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.955+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a4106920 con 0x7fb8a40fbb80 2024-02-28T16:36:30.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.957+0000 7fb8a1600700 1 -- 172.21.15.1:0/769608216 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7fb898012400 con 0x7fb8a40fbb80 2024-02-28T16:36:30.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.958+0000 7fb8a1600700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb88c06d900 0x7fb88c06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:36:30.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.958+0000 7fb8a1600700 1 -- 172.21.15.1:0/769608216 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fb89808cfe0 con 0x7fb8a40fbb80 2024-02-28T16:36:30.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.958+0000 7fb8a8e00700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb88c06d900 0x7fb88c06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:36:30.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.959+0000 7fb8a8e00700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb88c06d900 0x7fb88c06fdb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb89000b3b0 tx=0x7fb890009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:36:30.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:30.964+0000 7fb8a1600700 1 -- 172.21.15.1:0/769608216 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb89805a3c0 con 0x7fb8a40fbb80 2024-02-28T16:36:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.270+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb8a410c090 con 0x7fb8a40fbb80 2024-02-28T16:36:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.271+0000 7fb8a1600700 1 -- 172.21.15.1:0/769608216 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fb8980126b0 con 0x7fb8a40fbb80 2024-02-28T16:36:31.259 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:36:31.259 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:36:31.259 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:36:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.274+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb88c06d900 msgr2=0x7fb88c06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.274+0000 7fb8aac00700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb88c06d900 0x7fb88c06fdb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb89000b3b0 tx=0x7fb890009e30 comp rx=0 tx=0).stop 2024-02-28T16:36:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.274+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fbb80 msgr2=0x7fb8a4115a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:36:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.274+0000 7fb8aac00700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fbb80 0x7fb8a4115a40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb89800c8c0 tx=0x7fb8980044c0 comp rx=0 tx=0).stop 2024-02-28T16:36:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.274+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 shutdown_connections 2024-02-28T16:36:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.274+0000 7fb8aac00700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb88c06d900 0x7fb88c06fdb0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.275+0000 7fb8aac00700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a40fbb80 0x7fb8a4115a40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.275+0000 7fb8aac00700 1 --2- 172.21.15.1:0/769608216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb8a40fc490 0x7fb8a4115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:36:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.275+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 >> 172.21.15.1:0/769608216 conn(0x7fb8a4073ab0 msgr2=0x7fb8a40fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:36:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.275+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 shutdown_connections 2024-02-28T16:36:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:36:31.275+0000 7fb8aac00700 1 -- 172.21.15.1:0/769608216 wait complete. 2024-02-28T16:36:31.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:31 smithi001 ceph-mon[112940]: Reconfiguring osd.0 (monmap changed)... 2024-02-28T16:36:31.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:31 smithi001 ceph-mon[112940]: Reconfiguring daemon osd.0 on smithi001 2024-02-28T16:36:31.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:31 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3768505911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:36:31.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:31 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/769608216' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:36:31.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:31 smithi066 ceph-mon[85781]: Reconfiguring osd.0 (monmap changed)... 2024-02-28T16:36:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:31 smithi066 ceph-mon[85781]: Reconfiguring daemon osd.0 on smithi001 2024-02-28T16:36:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:31 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3768505911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:36:31.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:31 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/769608216' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:36:32.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:32 smithi066 ceph-mon[85781]: pgmap v28: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:32.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:32 smithi066 ceph-mon[85781]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:36:32.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:32.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:32.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-28T16:36:32.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:32.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:32 smithi001 ceph-mon[112940]: pgmap v28: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:32.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:32 smithi001 ceph-mon[112940]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:36:32.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:32.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:32.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-28T16:36:32.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:33.453 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:33 smithi001 ceph-mon[112940]: Reconfiguring osd.1 (monmap changed)... 2024-02-28T16:36:33.453 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:33 smithi001 ceph-mon[112940]: Reconfiguring daemon osd.1 on smithi001 2024-02-28T16:36:33.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:33 smithi066 ceph-mon[85781]: Reconfiguring osd.1 (monmap changed)... 2024-02-28T16:36:33.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:33 smithi066 ceph-mon[85781]: Reconfiguring daemon osd.1 on smithi001 2024-02-28T16:36:34.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:34 smithi001 ceph-mon[112940]: pgmap v29: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:34.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:34 smithi066 ceph-mon[85781]: pgmap v29: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:35.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:35 smithi001 ceph-mon[112940]: Reconfiguring osd.2 (monmap changed)... 2024-02-28T16:36:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-28T16:36:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:35 smithi001 ceph-mon[112940]: Reconfiguring daemon osd.2 on smithi001 2024-02-28T16:36:35.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:35 smithi001 ceph-mon[112940]: pgmap v30: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:35.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:35.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:35.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:35 smithi066 ceph-mon[85781]: Reconfiguring osd.2 (monmap changed)... 2024-02-28T16:36:35.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-28T16:36:35.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:35.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:35 smithi066 ceph-mon[85781]: Reconfiguring daemon osd.2 on smithi001 2024-02-28T16:36:35.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:35 smithi066 ceph-mon[85781]: pgmap v30: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:38.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:38 smithi001 ceph-mon[112940]: pgmap v31: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:38.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:38.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:38.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:38 smithi001 ceph-mon[112940]: Reconfiguring mds.cephfs.smithi001.awxxdo (monmap changed)... 2024-02-28T16:36:38.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.awxxdo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:36:38.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:38.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:38 smithi001 ceph-mon[112940]: Reconfiguring daemon mds.cephfs.smithi001.awxxdo on smithi001 2024-02-28T16:36:38.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:38 smithi066 ceph-mon[85781]: pgmap v31: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:38.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:38.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:38.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:38 smithi066 ceph-mon[85781]: Reconfiguring mds.cephfs.smithi001.awxxdo (monmap changed)... 2024-02-28T16:36:38.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.awxxdo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:36:38.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:38.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:38 smithi066 ceph-mon[85781]: Reconfiguring daemon mds.cephfs.smithi001.awxxdo on smithi001 2024-02-28T16:36:40.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:40 smithi001 ceph-mon[112940]: pgmap v32: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:40.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:40.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:40.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:40 smithi001 ceph-mon[112940]: Reconfiguring mds.cephfs.smithi001.rtdwhc (monmap changed)... 2024-02-28T16:36:40.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.rtdwhc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:36:40.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:40.466 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:40 smithi001 ceph-mon[112940]: Reconfiguring daemon mds.cephfs.smithi001.rtdwhc on smithi001 2024-02-28T16:36:40.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:40 smithi066 ceph-mon[85781]: pgmap v32: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:40 smithi066 ceph-mon[85781]: Reconfiguring mds.cephfs.smithi001.rtdwhc (monmap changed)... 2024-02-28T16:36:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.rtdwhc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:36:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:40.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:40 smithi066 ceph-mon[85781]: Reconfiguring daemon mds.cephfs.smithi001.rtdwhc on smithi001 2024-02-28T16:36:41.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:36:41.517 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:36:42.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:42 smithi066 ceph-mon[85781]: pgmap v33: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:42.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:42 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:42.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:42 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:42.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:42 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi066", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:36:42.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:42 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:42.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:42 smithi001 ceph-mon[112940]: pgmap v33: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:42 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:42 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:42 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi066", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:36:42.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:42 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:43.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:43 smithi066 ceph-mon[85781]: Reconfiguring crash.smithi066 (monmap changed)... 2024-02-28T16:36:43.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:43 smithi066 ceph-mon[85781]: Reconfiguring daemon crash.smithi066 on smithi066 2024-02-28T16:36:43.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:43 smithi001 ceph-mon[112940]: Reconfiguring crash.smithi066 (monmap changed)... 2024-02-28T16:36:43.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:43 smithi001 ceph-mon[112940]: Reconfiguring daemon crash.smithi066 on smithi066 2024-02-28T16:36:44.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:44 smithi066 ceph-mon[85781]: pgmap v34: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:44.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:44 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:44.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:44 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:44.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:44 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:36:44.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:44 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:36:44.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:44 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:44.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:44 smithi001 ceph-mon[112940]: pgmap v34: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:44.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:44 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:44.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:44 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:44.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:44 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi066.ztraxe", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-28T16:36:44.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:44 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-28T16:36:44.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:44 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:45.381 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:45 smithi066 ceph-mon[85781]: Reconfiguring mgr.smithi066.ztraxe (monmap changed)... 2024-02-28T16:36:45.382 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:45 smithi066 ceph-mon[85781]: Reconfiguring daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:36:45.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:45 smithi001 ceph-mon[112940]: Reconfiguring mgr.smithi066.ztraxe (monmap changed)... 2024-02-28T16:36:45.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:45 smithi001 ceph-mon[112940]: Reconfiguring daemon mgr.smithi066.ztraxe on smithi066 2024-02-28T16:36:46.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:46 smithi066 ceph-mon[85781]: pgmap v35: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:46.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:46 smithi001 ceph-mon[112940]: pgmap v35: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:47.680 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:47 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:47.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:47 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:47.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:47 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:36:47.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:47 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:36:47.681 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:47 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:47.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:47 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:47.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:47 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:47.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:47 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-28T16:36:47.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:47 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-28T16:36:47.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:47 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:48.705 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:48 smithi066 ceph-mon[85781]: Reconfiguring mon.smithi066 (monmap changed)... 2024-02-28T16:36:48.705 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:48 smithi066 ceph-mon[85781]: Reconfiguring daemon mon.smithi066 on smithi066 2024-02-28T16:36:48.706 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:48 smithi066 ceph-mon[85781]: pgmap v36: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:48.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:48 smithi001 ceph-mon[112940]: Reconfiguring mon.smithi066 (monmap changed)... 2024-02-28T16:36:48.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:48 smithi001 ceph-mon[112940]: Reconfiguring daemon mon.smithi066 on smithi066 2024-02-28T16:36:48.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:48 smithi001 ceph-mon[112940]: pgmap v36: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:50.132 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:49 smithi066 ceph-mon[85781]: pgmap v37: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:50.132 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:50.132 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:50.132 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:49 smithi066 ceph-mon[85781]: Reconfiguring osd.3 (monmap changed)... 2024-02-28T16:36:50.132 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-28T16:36:50.132 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:50.133 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:49 smithi066 ceph-mon[85781]: Reconfiguring daemon osd.3 on smithi066 2024-02-28T16:36:50.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:49 smithi001 ceph-mon[112940]: pgmap v37: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:50.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:50.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:50.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:49 smithi001 ceph-mon[112940]: Reconfiguring osd.3 (monmap changed)... 2024-02-28T16:36:50.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-28T16:36:50.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:50.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:49 smithi001 ceph-mon[112940]: Reconfiguring daemon osd.3 on smithi066 2024-02-28T16:36:52.676 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:52 smithi066 ceph-mon[85781]: pgmap v38: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:52.676 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:52.676 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:52.676 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-28T16:36:52.676 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:52.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:52 smithi001 ceph-mon[112940]: pgmap v38: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:52.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:52.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:52.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-28T16:36:52.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:53.739 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:53 smithi066 ceph-mon[85781]: Reconfiguring osd.4 (monmap changed)... 2024-02-28T16:36:53.739 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:53 smithi066 ceph-mon[85781]: Reconfiguring daemon osd.4 on smithi066 2024-02-28T16:36:53.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:53 smithi001 ceph-mon[112940]: Reconfiguring osd.4 (monmap changed)... 2024-02-28T16:36:53.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:53 smithi001 ceph-mon[112940]: Reconfiguring daemon osd.4 on smithi066 2024-02-28T16:36:54.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:54 smithi066 ceph-mon[85781]: pgmap v39: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:54.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:54 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:54.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:54 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:54.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:54 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-28T16:36:54.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:54 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:54.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:54 smithi001 ceph-mon[112940]: pgmap v39: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:54.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:54 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:54.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:54 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:54.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:54 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-28T16:36:54.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:54 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:55.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:55 smithi066 ceph-mon[85781]: Reconfiguring osd.5 (monmap changed)... 2024-02-28T16:36:55.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:55 smithi066 ceph-mon[85781]: Reconfiguring daemon osd.5 on smithi066 2024-02-28T16:36:55.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:55 smithi001 ceph-mon[112940]: Reconfiguring osd.5 (monmap changed)... 2024-02-28T16:36:55.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:55 smithi001 ceph-mon[112940]: Reconfiguring daemon osd.5 on smithi066 2024-02-28T16:36:56.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:56 smithi066 ceph-mon[85781]: pgmap v40: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:56.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:36:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:56 smithi001 ceph-mon[112940]: pgmap v40: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:36:57.846 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:57 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:57.847 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:57 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:57.847 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:57 smithi066 ceph-mon[85781]: Reconfiguring mds.cephfs.smithi066.wfbdor (monmap changed)... 2024-02-28T16:36:57.847 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:57 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.wfbdor", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:36:57.847 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:57 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:57.847 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:57 smithi066 ceph-mon[85781]: pgmap v41: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:57.847 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:36:57 smithi066 ceph-mon[85781]: Reconfiguring daemon mds.cephfs.smithi066.wfbdor on smithi066 2024-02-28T16:36:58.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:58.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:36:58.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:57 smithi001 ceph-mon[112940]: Reconfiguring mds.cephfs.smithi066.wfbdor (monmap changed)... 2024-02-28T16:36:58.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.wfbdor", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:36:58.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:36:58.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:57 smithi001 ceph-mon[112940]: pgmap v41: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:36:58.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:36:57 smithi001 ceph-mon[112940]: Reconfiguring daemon mds.cephfs.smithi066.wfbdor on smithi066 2024-02-28T16:37:00.428 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:00 smithi066 ceph-mon[85781]: pgmap v42: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:00.428 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:00 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:00.428 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:00 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:00.429 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:00 smithi066 ceph-mon[85781]: Reconfiguring mds.cephfs.smithi066.pivimo (monmap changed)... 2024-02-28T16:37:00.429 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:00 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.pivimo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:37:00.429 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:00 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:00.429 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:00 smithi066 ceph-mon[85781]: Reconfiguring daemon mds.cephfs.smithi066.pivimo on smithi066 2024-02-28T16:37:00.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:00 smithi001 ceph-mon[112940]: pgmap v42: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:00.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:00.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:00.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:00 smithi001 ceph-mon[112940]: Reconfiguring mds.cephfs.smithi066.pivimo (monmap changed)... 2024-02-28T16:37:00.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.pivimo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:37:00.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:00.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:00 smithi001 ceph-mon[112940]: Reconfiguring daemon mds.cephfs.smithi066.pivimo on smithi066 2024-02-28T16:37:01.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.444+0000 7f5d0f200700 1 -- 172.21.15.1:0/955162936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 msgr2=0x7f5d10109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:01.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.444+0000 7f5d0f200700 1 --2- 172.21.15.1:0/955162936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 0x7f5d10109750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d00004940 tx=0x7f5d00009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:01.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.444+0000 7f5d0f200700 1 -- 172.21.15.1:0/955162936 shutdown_connections 2024-02-28T16:37:01.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.444+0000 7f5d0f200700 1 --2- 172.21.15.1:0/955162936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d10109d20 0x7f5d101033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.444+0000 7f5d0f200700 1 --2- 172.21.15.1:0/955162936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 0x7f5d10109750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.444+0000 7f5d0f200700 1 -- 172.21.15.1:0/955162936 >> 172.21.15.1:0/955162936 conn(0x7f5d10072b50 msgr2=0x7f5d10074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:01.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.444+0000 7f5d0f200700 1 -- 172.21.15.1:0/955162936 shutdown_connections 2024-02-28T16:37:01.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.444+0000 7f5d0f200700 1 -- 172.21.15.1:0/955162936 wait complete. 2024-02-28T16:37:01.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.445+0000 7f5d0f200700 1 Processor -- start 2024-02-28T16:37:01.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.445+0000 7f5d0f200700 1 -- start start 2024-02-28T16:37:01.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.445+0000 7f5d0f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 0x7f5d10119ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:01.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.445+0000 7f5d0f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d10109d20 0x7f5d1011a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:01.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.445+0000 7f5d0f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1011aa30 con 0x7f5d10109380 2024-02-28T16:37:01.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.445+0000 7f5d0f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10113d40 con 0x7f5d10109d20 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.446+0000 7f5d0de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 0x7f5d10119ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.446+0000 7f5d0de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 0x7f5d10119ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50548/0 (socket says 172.21.15.1:50548) 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.446+0000 7f5d0de00700 1 -- 172.21.15.1:0/3636495892 learned_addr learned my addr 172.21.15.1:0/3636495892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.446+0000 7f5d0d400700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d10109d20 0x7f5d1011a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.446+0000 7f5d0de00700 1 -- 172.21.15.1:0/3636495892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d10109d20 msgr2=0x7f5d1011a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.446+0000 7f5d0de00700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d10109d20 0x7f5d1011a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.446+0000 7f5d0de00700 1 -- 172.21.15.1:0/3636495892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d00005040 con 0x7f5d10109380 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.446+0000 7f5d0d400700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d10109d20 0x7f5d1011a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.447+0000 7f5d0de00700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 0x7f5d10119ea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009150 tx=0x7f5d00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.447+0000 7f5d06000700 1 -- 172.21.15.1:0/3636495892 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d0000e040 con 0x7f5d10109380 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.447+0000 7f5d06000700 1 -- 172.21.15.1:0/3636495892 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d0000bbf0 con 0x7f5d10109380 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.447+0000 7f5d06000700 1 -- 172.21.15.1:0/3636495892 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d0001c630 con 0x7f5d10109380 2024-02-28T16:37:01.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.447+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d10113fc0 con 0x7f5d10109380 2024-02-28T16:37:01.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.447+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d10114500 con 0x7f5d10109380 2024-02-28T16:37:01.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.448+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d10106690 con 0x7f5d10109380 2024-02-28T16:37:01.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.449+0000 7f5d06000700 1 -- 172.21.15.1:0/3636495892 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f5d0000bd60 con 0x7f5d10109380 2024-02-28T16:37:01.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.450+0000 7f5d06000700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cf406d900 0x7f5cf406fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:01.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.450+0000 7f5d06000700 1 -- 172.21.15.1:0/3636495892 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f5d0008e3e0 con 0x7f5d10109380 2024-02-28T16:37:01.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.451+0000 7f5d0d400700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cf406d900 0x7f5cf406fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:01.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.451+0000 7f5d0d400700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cf406d900 0x7f5cf406fdb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007fd0 tx=0x7f5cfc009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:01.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.455+0000 7f5d06000700 1 -- 172.21.15.1:0/3636495892 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5d0005b710 con 0x7f5d10109380 2024-02-28T16:37:01.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.719+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d1004bf40 con 0x7f5cf406d900 2024-02-28T16:37:01.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.720+0000 7f5d06000700 1 -- 172.21.15.1:0/3636495892 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f5d1004bf40 con 0x7f5cf406d900 2024-02-28T16:37:01.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cf406d900 msgr2=0x7f5cf406fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:01.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cf406d900 0x7f5cf406fdb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007fd0 tx=0x7f5cfc009040 comp rx=0 tx=0).stop 2024-02-28T16:37:01.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 msgr2=0x7f5d10119ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:01.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 0x7f5d10119ea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009150 tx=0x7f5d00009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:01.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 shutdown_connections 2024-02-28T16:37:01.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cf406d900 0x7f5cf406fdb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d10109380 0x7f5d10119ea0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 --2- 172.21.15.1:0/3636495892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d10109d20 0x7f5d1011a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 >> 172.21.15.1:0/3636495892 conn(0x7f5d10072b50 msgr2=0x7f5d10075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:01.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 shutdown_connections 2024-02-28T16:37:01.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.723+0000 7f5d0f200700 1 -- 172.21.15.1:0/3636495892 wait complete. 2024-02-28T16:37:01.727 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:37:01.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.870+0000 7f890ee00700 1 -- 172.21.15.1:0/3456037667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 msgr2=0x7f8908107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:01.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.870+0000 7f890ee00700 1 --2- 172.21.15.1:0/3456037667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 0x7f8908107050 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f88f0004970 tx=0x7f88f0009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:01.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.870+0000 7f890ee00700 1 -- 172.21.15.1:0/3456037667 shutdown_connections 2024-02-28T16:37:01.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.870+0000 7f890ee00700 1 --2- 172.21.15.1:0/3456037667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 0x7f8908107050 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.870+0000 7f890ee00700 1 --2- 172.21.15.1:0/3456037667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89080fdfe0 0x7f89080fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.870+0000 7f890ee00700 1 -- 172.21.15.1:0/3456037667 >> 172.21.15.1:0/3456037667 conn(0x7f8908071d90 msgr2=0x7f8908072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:01.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.870+0000 7f890ee00700 1 -- 172.21.15.1:0/3456037667 shutdown_connections 2024-02-28T16:37:01.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.870+0000 7f890ee00700 1 -- 172.21.15.1:0/3456037667 wait complete. 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.871+0000 7f890ee00700 1 Processor -- start 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.871+0000 7f890ee00700 1 -- start start 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f890ee00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89080fdfe0 0x7f8908115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f890ee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 0x7f8908116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f890ee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908116790 con 0x7f89080fe8f0 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f890ee00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890810fac0 con 0x7f89080fdfe0 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f88ffe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 0x7f8908116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f88ffe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 0x7f8908116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34270/0 (socket says 172.21.15.1:34270) 2024-02-28T16:37:01.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f88ffe00700 1 -- 172.21.15.1:0/1959183815 learned_addr learned my addr 172.21.15.1:0/1959183815 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f8907e00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89080fdfe0 0x7f8908115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f88ffe00700 1 -- 172.21.15.1:0/1959183815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89080fdfe0 msgr2=0x7f8908115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.872+0000 7f88ffe00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89080fdfe0 0x7f8908115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.873+0000 7f88ffe00700 1 -- 172.21.15.1:0/1959183815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f0005040 con 0x7f89080fe8f0 2024-02-28T16:37:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.873+0000 7f88ffe00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 0x7f8908116140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f88f0000c00 tx=0x7f88f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.873+0000 7f8904c00700 1 -- 172.21.15.1:0/1959183815 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f000e040 con 0x7f89080fe8f0 2024-02-28T16:37:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.873+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890810fd40 con 0x7f89080fe8f0 2024-02-28T16:37:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.873+0000 7f8904c00700 1 -- 172.21.15.1:0/1959183815 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88f0004290 con 0x7f89080fe8f0 2024-02-28T16:37:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.873+0000 7f8904c00700 1 -- 172.21.15.1:0/1959183815 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f001c630 con 0x7f89080fe8f0 2024-02-28T16:37:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.873+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8908110280 con 0x7f89080fe8f0 2024-02-28T16:37:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.875+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8908106b20 con 0x7f89080fe8f0 2024-02-28T16:37:01.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.876+0000 7f8904c00700 1 -- 172.21.15.1:0/1959183815 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f88f0013980 con 0x7f89080fe8f0 2024-02-28T16:37:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.877+0000 7f8904c00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f88ec06d900 0x7f88ec06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.877+0000 7f8904c00700 1 -- 172.21.15.1:0/1959183815 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f88f008e480 con 0x7f89080fe8f0 2024-02-28T16:37:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.877+0000 7f8907e00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f88ec06d900 0x7f88ec06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:01.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.878+0000 7f8907e00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f88ec06d900 0x7f88ec06fdb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f88f8007fd0 tx=0x7f88f8009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:01.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:01.886+0000 7f8904c00700 1 -- 172.21.15.1:0/1959183815 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f88f005b670 con 0x7f89080fe8f0 2024-02-28T16:37:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.120+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f89081158d0 con 0x7f88ec06d900 2024-02-28T16:37:02.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.121+0000 7f8904c00700 1 -- 172.21.15.1:0/1959183815 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f89081158d0 con 0x7f88ec06d900 2024-02-28T16:37:02.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f88ec06d900 msgr2=0x7f88ec06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:02.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f88ec06d900 0x7f88ec06fdb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f88f8007fd0 tx=0x7f88f8009040 comp rx=0 tx=0).stop 2024-02-28T16:37:02.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 msgr2=0x7f8908116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:02.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 0x7f8908116140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f88f0000c00 tx=0x7f88f0009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:02.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 shutdown_connections 2024-02-28T16:37:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f88ec06d900 0x7f88ec06fdb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f89080fdfe0 0x7f8908115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 --2- 172.21.15.1:0/1959183815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89080fe8f0 0x7f8908116140 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 >> 172.21.15.1:0/1959183815 conn(0x7f8908071d90 msgr2=0x7f89080fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.124+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 shutdown_connections 2024-02-28T16:37:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.125+0000 7f890ee00700 1 -- 172.21.15.1:0/1959183815 wait complete. 2024-02-28T16:37:02.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.275+0000 7f0768600700 1 -- 172.21.15.1:0/2763062772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 msgr2=0x7f07600706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:02.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.275+0000 7f0768600700 1 --2- 172.21.15.1:0/2763062772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 0x7f07600706a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0754004970 tx=0x7f0754009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:02.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.275+0000 7f0768600700 1 -- 172.21.15.1:0/2763062772 shutdown_connections 2024-02-28T16:37:02.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.275+0000 7f0768600700 1 --2- 172.21.15.1:0/2763062772 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0760070be0 0x7f07601092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.275+0000 7f0768600700 1 --2- 172.21.15.1:0/2763062772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 0x7f07600706a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.275+0000 7f0768600700 1 -- 172.21.15.1:0/2763062772 >> 172.21.15.1:0/2763062772 conn(0x7f07600f8ae0 msgr2=0x7f07600faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.275+0000 7f0768600700 1 -- 172.21.15.1:0/2763062772 shutdown_connections 2024-02-28T16:37:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.275+0000 7f0768600700 1 -- 172.21.15.1:0/2763062772 wait complete. 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.276+0000 7f0768600700 1 Processor -- start 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.276+0000 7f0768600700 1 -- start start 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.277+0000 7f0768600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 0x7f0760115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.277+0000 7f0768600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0760070be0 0x7f0760116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.277+0000 7f0768600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07601167b0 con 0x7f07600702d0 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.277+0000 7f0768600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076010fae0 con 0x7f0760070be0 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.277+0000 7f0765c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 0x7f0760115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.277+0000 7f0765c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 0x7f0760115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34288/0 (socket says 172.21.15.1:34288) 2024-02-28T16:37:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.277+0000 7f0765c00700 1 -- 172.21.15.1:0/3325442212 learned_addr learned my addr 172.21.15.1:0/3325442212 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.277+0000 7f0765200700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0760070be0 0x7f0760116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.278+0000 7f0765c00700 1 -- 172.21.15.1:0/3325442212 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0760070be0 msgr2=0x7f0760116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.278+0000 7f0765c00700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0760070be0 0x7f0760116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.278+0000 7f0765c00700 1 -- 172.21.15.1:0/3325442212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754005040 con 0x7f07600702d0 2024-02-28T16:37:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.278+0000 7f0765c00700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 0x7f0760115c20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0754000c00 tx=0x7f0754009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.278+0000 7f075d600700 1 -- 172.21.15.1:0/3325442212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f075400e040 con 0x7f07600702d0 2024-02-28T16:37:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.278+0000 7f075d600700 1 -- 172.21.15.1:0/3325442212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0754004290 con 0x7f07600702d0 2024-02-28T16:37:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.278+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076010fd60 con 0x7f07600702d0 2024-02-28T16:37:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.278+0000 7f075d600700 1 -- 172.21.15.1:0/3325442212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f075401c630 con 0x7f07600702d0 2024-02-28T16:37:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.279+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07601102a0 con 0x7f07600702d0 2024-02-28T16:37:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.280+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0760106980 con 0x7f07600702d0 2024-02-28T16:37:02.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.281+0000 7f075d600700 1 -- 172.21.15.1:0/3325442212 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f0754013980 con 0x7f07600702d0 2024-02-28T16:37:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.282+0000 7f075d600700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f074806d900 0x7f074806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.282+0000 7f075d600700 1 -- 172.21.15.1:0/3325442212 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f075408e3e0 con 0x7f07600702d0 2024-02-28T16:37:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.282+0000 7f0765200700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f074806d900 0x7f074806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.283+0000 7f0765200700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f074806d900 0x7f074806fdb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f074c007fd0 tx=0x7f074c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:02.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.289+0000 7f075d600700 1 -- 172.21.15.1:0/3325442212 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f075405b670 con 0x7f07600702d0 2024-02-28T16:37:02.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.531+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f076004bed0 con 0x7f074806d900 2024-02-28T16:37:02.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.564+0000 7f075d600700 1 -- 172.21.15.1:0/3325442212 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f076004bed0 con 0x7f074806d900 2024-02-28T16:37:02.553 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:37:02.553 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (3m) 74s ago 17m 14.6M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (17m) 74s ago 17m 6660k - 17.2.7-535-gc0f02e60 1c37ed371348 83603dde54a9 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (16m) 48s ago 16m 6665k - 17.2.7-535-gc0f02e60 1c37ed371348 470a69ac72e6 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (2m) 74s ago 17m 74.7M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (10m) 74s ago 10m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (10m) 74s ago 10m 13.8M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (10m) 48s ago 10m 14.0M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (10m) 48s ago 10m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (5m) 74s ago 19m 474M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (4m) 48s ago 16m 447M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (92s) 74s ago 19m 29.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (54s) 48s ago 15m 19.7M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (4m) 74s ago 17m 7633k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (4m) 48s ago 16m 8132k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:37:02.554 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (15m) 74s ago 15m 207M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:37:02.555 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (14m) 74s ago 14m 209M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:37:02.555 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (13m) 74s ago 13m 169M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:37:02.555 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (13m) 48s ago 13m 222M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:37:02.555 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (12m) 48s ago 12m 213M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:37:02.555 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (12m) 48s ago 12m 188M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:37:02.555 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (3m) 74s ago 16m 48.7M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f074806d900 msgr2=0x7f074806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f074806d900 0x7f074806fdb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f074c007fd0 tx=0x7f074c009040 comp rx=0 tx=0).stop 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 msgr2=0x7f0760115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 0x7f0760115c20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0754000c00 tx=0x7f0754009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 shutdown_connections 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f074806d900 0x7f074806fdb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07600702d0 0x7f0760115c20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 --2- 172.21.15.1:0/3325442212 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0760070be0 0x7f0760116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 >> 172.21.15.1:0/3325442212 conn(0x7f07600f8ae0 msgr2=0x7f0760103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 shutdown_connections 2024-02-28T16:37:02.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.568+0000 7f0768600700 1 -- 172.21.15.1:0/3325442212 wait complete. 2024-02-28T16:37:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.715+0000 7f9dece00700 1 -- 172.21.15.1:0/3745234756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105af0 msgr2=0x7f9de80ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.715+0000 7f9dece00700 1 --2- 172.21.15.1:0/3745234756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105af0 0x7f9de80ff160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4004940 tx=0x7f9dd4009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.715+0000 7f9dece00700 1 -- 172.21.15.1:0/3745234756 shutdown_connections 2024-02-28T16:37:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.715+0000 7f9dece00700 1 --2- 172.21.15.1:0/3745234756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105af0 0x7f9de80ff160 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.715+0000 7f9dece00700 1 --2- 172.21.15.1:0/3745234756 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9de8105150 0x7f9de8105520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.715+0000 7f9dece00700 1 -- 172.21.15.1:0/3745234756 >> 172.21.15.1:0/3745234756 conn(0x7f9de80face0 msgr2=0x7f9de80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.715+0000 7f9dece00700 1 -- 172.21.15.1:0/3745234756 shutdown_connections 2024-02-28T16:37:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.715+0000 7f9dece00700 1 -- 172.21.15.1:0/3745234756 wait complete. 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: pgmap v43: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi001"}]: dispatch 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi001"}]': finished 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi066"}]: dispatch 2024-02-28T16:37:02.705 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:02 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi066"}]': finished 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.716+0000 7f9dece00700 1 Processor -- start 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.716+0000 7f9dece00700 1 -- start start 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.716+0000 7f9dece00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105150 0x7f9de8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.716+0000 7f9dece00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9de8105af0 0x7f9de8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.716+0000 7f9dece00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8116910 con 0x7f9de8105150 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.716+0000 7f9dece00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de810fb90 con 0x7f9de8105af0 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.716+0000 7f9de5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105150 0x7f9de8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9de5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105150 0x7f9de8115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34314/0 (socket says 172.21.15.1:34314) 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9de5e00700 1 -- 172.21.15.1:0/3309263980 learned_addr learned my addr 172.21.15.1:0/3309263980 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9de5e00700 1 -- 172.21.15.1:0/3309263980 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9de8105af0 msgr2=0x7f9de8116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9de5e00700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9de8105af0 0x7f9de8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9de5e00700 1 -- 172.21.15.1:0/3309263980 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd4005040 con 0x7f9de8105150 2024-02-28T16:37:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9de5e00700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105150 0x7f9de8115cf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800b790 tx=0x7f9dd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9dde000700 1 -- 172.21.15.1:0/3309263980 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd800f770 con 0x7f9de8105150 2024-02-28T16:37:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9dde000700 1 -- 172.21.15.1:0/3309263980 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dd800fda0 con 0x7f9de8105150 2024-02-28T16:37:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9dde000700 1 -- 172.21.15.1:0/3309263980 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd800d310 con 0x7f9de8105150 2024-02-28T16:37:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.717+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de810fe10 con 0x7f9de8105150 2024-02-28T16:37:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.718+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de8110350 con 0x7f9de8105150 2024-02-28T16:37:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.719+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de8102460 con 0x7f9de8105150 2024-02-28T16:37:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.719+0000 7f9dde000700 1 -- 172.21.15.1:0/3309263980 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f9dd8010420 con 0x7f9de8105150 2024-02-28T16:37:02.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.720+0000 7f9dde000700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9dcc06d900 0x7f9dcc06fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:02.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.720+0000 7f9dde000700 1 -- 172.21.15.1:0/3309263980 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f9dd808cf00 con 0x7f9de8105150 2024-02-28T16:37:02.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.721+0000 7f9de5400700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9dcc06d900 0x7f9dcc06fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.721+0000 7f9de5400700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9dcc06d900 0x7f9dcc06fdb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4004f90 tx=0x7f9dd4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:02.725+0000 7f9dde000700 1 -- 172.21.15.1:0/3309263980 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9dd805a340 con 0x7f9de8105150 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: pgmap v43: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi001"}]: dispatch 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi001"}]': finished 2024-02-28T16:37:02.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi066"}]: dispatch 2024-02-28T16:37:02.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:02 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi066"}]': finished 2024-02-28T16:37:03.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.049+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9de810c2a0 con 0x7f9de8105150 2024-02-28T16:37:03.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.050+0000 7f9dde000700 1 -- 172.21.15.1:0/3309263980 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f9dd8059ed0 con 0x7f9de8105150 2024-02-28T16:37:03.039 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:37:03.039 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:37:03.039 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:37:03.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:37:03.039 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:37:03.039 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 10, 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:37:03.040 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:37:03.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9dcc06d900 msgr2=0x7f9dcc06fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9dcc06d900 0x7f9dcc06fdb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4004f90 tx=0x7f9dd4009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:03.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105150 msgr2=0x7f9de8115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105150 0x7f9de8115cf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800b790 tx=0x7f9dd8009250 comp rx=0 tx=0).stop 2024-02-28T16:37:03.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 shutdown_connections 2024-02-28T16:37:03.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9dcc06d900 0x7f9dcc06fdb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de8105150 0x7f9de8115cf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 --2- 172.21.15.1:0/3309263980 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9de8105af0 0x7f9de8116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 >> 172.21.15.1:0/3309263980 conn(0x7f9de80face0 msgr2=0x7f9de80fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:03.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.053+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 shutdown_connections 2024-02-28T16:37:03.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.054+0000 7f9dece00700 1 -- 172.21.15.1:0/3309263980 wait complete. 2024-02-28T16:37:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.207+0000 7f87e2200700 1 -- 172.21.15.1:0/757646406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 msgr2=0x7f87dc104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.207+0000 7f87e2200700 1 --2- 172.21.15.1:0/757646406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 0x7f87dc104210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f87c4004940 tx=0x7f87c4009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.207+0000 7f87e2200700 1 -- 172.21.15.1:0/757646406 shutdown_connections 2024-02-28T16:37:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.207+0000 7f87e2200700 1 --2- 172.21.15.1:0/757646406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 0x7f87dc104210 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.207+0000 7f87e2200700 1 --2- 172.21.15.1:0/757646406 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87dc1095f0 0x7f87dc1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.207+0000 7f87e2200700 1 -- 172.21.15.1:0/757646406 >> 172.21.15.1:0/757646406 conn(0x7f87dc072d50 msgr2=0x7f87dc075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.207+0000 7f87e2200700 1 -- 172.21.15.1:0/757646406 shutdown_connections 2024-02-28T16:37:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.207+0000 7f87e2200700 1 -- 172.21.15.1:0/757646406 wait complete. 2024-02-28T16:37:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.208+0000 7f87e2200700 1 Processor -- start 2024-02-28T16:37:03.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.208+0000 7f87e2200700 1 -- start start 2024-02-28T16:37:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.208+0000 7f87e2200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 0x7f87dc11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.208+0000 7f87e2200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87dc1095f0 0x7f87dc11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87e2200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc11abd0 con 0x7f87dc103dc0 2024-02-28T16:37:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87e2200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc113ee0 con 0x7f87dc1095f0 2024-02-28T16:37:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87db400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 0x7f87dc11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87db400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 0x7f87dc11a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34340/0 (socket says 172.21.15.1:34340) 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87db400700 1 -- 172.21.15.1:0/2061441790 learned_addr learned my addr 172.21.15.1:0/2061441790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87db400700 1 -- 172.21.15.1:0/2061441790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87dc1095f0 msgr2=0x7f87dc11a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87db400700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87dc1095f0 0x7f87dc11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87db400700 1 -- 172.21.15.1:0/2061441790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c4005040 con 0x7f87dc103dc0 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.209+0000 7f87db400700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 0x7f87dc11a040 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f87cc002a50 tx=0x7f87cc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.210+0000 7f87d3400700 1 -- 172.21.15.1:0/2061441790 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87cc007cc0 con 0x7f87dc103dc0 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.210+0000 7f87d3400700 1 -- 172.21.15.1:0/2061441790 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87cc00f040 con 0x7f87dc103dc0 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.210+0000 7f87d3400700 1 -- 172.21.15.1:0/2061441790 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87cc009640 con 0x7f87dc103dc0 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.210+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc1141c0 con 0x7f87dc103dc0 2024-02-28T16:37:03.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.210+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc114700 con 0x7f87dc103dc0 2024-02-28T16:37:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.211+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87dc10b9c0 con 0x7f87dc103dc0 2024-02-28T16:37:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.212+0000 7f87d3400700 1 -- 172.21.15.1:0/2061441790 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f87cc0097a0 con 0x7f87dc103dc0 2024-02-28T16:37:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.213+0000 7f87d3400700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f87c006d900 0x7f87c006fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.213+0000 7f87d3400700 1 -- 172.21.15.1:0/2061441790 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f87cc08cda0 con 0x7f87dc103dc0 2024-02-28T16:37:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.213+0000 7f87d3e00700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f87c006d900 0x7f87c006fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.214+0000 7f87d3e00700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f87c006d900 0x7f87c006fdb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f87c4004f90 tx=0x7f87c4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:03.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.217+0000 7f87d3400700 1 -- 172.21.15.1:0/2061441790 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f87cc05a030 con 0x7f87dc103dc0 2024-02-28T16:37:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.479+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f87dc115620 con 0x7f87dc103dc0 2024-02-28T16:37:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.480+0000 7f87d3400700 1 -- 172.21.15.1:0/2061441790 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7f87cc059bc0 con 0x7f87dc103dc0 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:37:03.470 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:37:03.471 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:37:03.472 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:37:03.473 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:37:03.473 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:37:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f87c006d900 msgr2=0x7f87c006fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f87c006d900 0x7f87c006fdb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f87c4004f90 tx=0x7f87c4009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 msgr2=0x7f87dc11a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 0x7f87dc11a040 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f87cc002a50 tx=0x7f87cc0046b0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 shutdown_connections 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f87c006d900 0x7f87c006fdb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87dc103dc0 0x7f87dc11a040 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 --2- 172.21.15.1:0/2061441790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87dc1095f0 0x7f87dc11a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 >> 172.21.15.1:0/2061441790 conn(0x7f87dc072d50 msgr2=0x7f87dc074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 shutdown_connections 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.483+0000 7f87e2200700 1 -- 172.21.15.1:0/2061441790 wait complete. 2024-02-28T16:37:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:37:03.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.628+0000 7f57a8200700 1 -- 172.21.15.1:0/3408434626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 msgr2=0x7f57a00fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.628+0000 7f57a8200700 1 --2- 172.21.15.1:0/3408434626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 0x7f57a00fe3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5794004970 tx=0x7f5794009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:03.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.628+0000 7f57a8200700 1 -- 172.21.15.1:0/3408434626 shutdown_connections 2024-02-28T16:37:03.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.628+0000 7f57a8200700 1 --2- 172.21.15.1:0/3408434626 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57a00fe8f0 0x7f57a0107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.628+0000 7f57a8200700 1 --2- 172.21.15.1:0/3408434626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 0x7f57a00fe3b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.628+0000 7f57a8200700 1 -- 172.21.15.1:0/3408434626 >> 172.21.15.1:0/3408434626 conn(0x7f57a0071d90 msgr2=0x7f57a0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:03.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.628+0000 7f57a8200700 1 -- 172.21.15.1:0/3408434626 shutdown_connections 2024-02-28T16:37:03.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.628+0000 7f57a8200700 1 -- 172.21.15.1:0/3408434626 wait complete. 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.629+0000 7f57a8200700 1 Processor -- start 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.630+0000 7f57a8200700 1 -- start start 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.630+0000 7f57a8200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 0x7f57a0112940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.630+0000 7f57a8200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57a00fe8f0 0x7f57a010b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.630+0000 7f57a8200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a010bca0 con 0x7f57a00fdfe0 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.630+0000 7f57a8200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a010bde0 con 0x7f57a00fe8f0 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.630+0000 7f57a5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 0x7f57a0112940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.630+0000 7f57a5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 0x7f57a0112940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34368/0 (socket says 172.21.15.1:34368) 2024-02-28T16:37:03.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.630+0000 7f57a5800700 1 -- 172.21.15.1:0/2578684102 learned_addr learned my addr 172.21.15.1:0/2578684102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:03.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f57a4e00700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57a00fe8f0 0x7f57a010b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f57a5800700 1 -- 172.21.15.1:0/2578684102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57a00fe8f0 msgr2=0x7f57a010b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f57a5800700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57a00fe8f0 0x7f57a010b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f57a5800700 1 -- 172.21.15.1:0/2578684102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5794005040 con 0x7f57a00fdfe0 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f57a4e00700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57a00fe8f0 0x7f57a010b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f57a5800700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 0x7f57a0112940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5794004f50 tx=0x7f5794009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f579d600700 1 -- 172.21.15.1:0/2578684102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f579400e040 con 0x7f57a00fdfe0 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f579d600700 1 -- 172.21.15.1:0/2578684102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5794004290 con 0x7f57a00fdfe0 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a010c060 con 0x7f57a00fdfe0 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.631+0000 7f579d600700 1 -- 172.21.15.1:0/2578684102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f579401c630 con 0x7f57a00fdfe0 2024-02-28T16:37:03.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.632+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a010c5a0 con 0x7f57a00fdfe0 2024-02-28T16:37:03.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.633+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57800052e0 con 0x7f57a00fdfe0 2024-02-28T16:37:03.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.634+0000 7f579d600700 1 -- 172.21.15.1:0/2578684102 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7f579400bc10 con 0x7f57a00fdfe0 2024-02-28T16:37:03.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.635+0000 7f579d600700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f578806d900 0x7f578806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:03.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.635+0000 7f579d600700 1 -- 172.21.15.1:0/2578684102 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f579408e410 con 0x7f57a00fdfe0 2024-02-28T16:37:03.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.635+0000 7f57a4e00700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f578806d900 0x7f578806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:03.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.636+0000 7f57a4e00700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f578806d900 0x7f578806fdb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f578c006fd0 tx=0x7f578c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:03.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.639+0000 7f579d600700 1 -- 172.21.15.1:0/2578684102 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f579405b600 con 0x7f57a00fdfe0 2024-02-28T16:37:03.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:03 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all mon 2024-02-28T16:37:03.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:03 smithi001 ceph-mon[112940]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:03.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:03 smithi001 ceph-mon[112940]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:03.728 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:03 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3309263980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:03.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:03 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all mon 2024-02-28T16:37:03.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:03 smithi066 ceph-mon[85781]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:03.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:03 smithi066 ceph-mon[85781]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:03.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:03 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3309263980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.894+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5780000bf0 con 0x7f578806d900 2024-02-28T16:37:03.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.895+0000 7f579d600700 1 -- 172.21.15.1:0/2578684102 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f5780000bf0 con 0x7f578806d900 2024-02-28T16:37:03.884 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:37:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:37:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:37:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:37:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:37:03.884 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:37:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "mon" 2024-02-28T16:37:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:37:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "4/21 daemons upgraded", 2024-02-28T16:37:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading crash daemons", 2024-02-28T16:37:03.885 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:37:03.885 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f578806d900 msgr2=0x7f578806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f578806d900 0x7f578806fdb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f578c006fd0 tx=0x7f578c008040 comp rx=0 tx=0).stop 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 msgr2=0x7f57a0112940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 0x7f57a0112940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5794004f50 tx=0x7f5794009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 shutdown_connections 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f578806d900 0x7f578806fdb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57a00fdfe0 0x7f57a0112940 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 --2- 172.21.15.1:0/2578684102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57a00fe8f0 0x7f57a010b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 >> 172.21.15.1:0/2578684102 conn(0x7f57a0071d90 msgr2=0x7f57a00fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 shutdown_connections 2024-02-28T16:37:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:03.898+0000 7f57a8200700 1 -- 172.21.15.1:0/2578684102 wait complete. 2024-02-28T16:37:04.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.050+0000 7fd15b200700 1 -- 172.21.15.1:0/2860619631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1541038f0 msgr2=0x7fd1540fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:04.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.050+0000 7fd15b200700 1 --2- 172.21.15.1:0/2860619631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1541038f0 0x7fd1540fcf60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd140004940 tx=0x7fd140009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.050+0000 7fd15b200700 1 -- 172.21.15.1:0/2860619631 shutdown_connections 2024-02-28T16:37:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.050+0000 7fd15b200700 1 --2- 172.21.15.1:0/2860619631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1541038f0 0x7fd1540fcf60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.050+0000 7fd15b200700 1 --2- 172.21.15.1:0/2860619631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd154102f50 0x7fd154103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.050+0000 7fd15b200700 1 -- 172.21.15.1:0/2860619631 >> 172.21.15.1:0/2860619631 conn(0x7fd1540f8ae0 msgr2=0x7fd1540faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.051+0000 7fd15b200700 1 -- 172.21.15.1:0/2860619631 shutdown_connections 2024-02-28T16:37:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.051+0000 7fd15b200700 1 -- 172.21.15.1:0/2860619631 wait complete. 2024-02-28T16:37:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.052+0000 7fd15b200700 1 Processor -- start 2024-02-28T16:37:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.052+0000 7fd15b200700 1 -- start start 2024-02-28T16:37:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.052+0000 7fd15b200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd154102f50 0x7fd154113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.052+0000 7fd15b200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1541038f0 0x7fd154113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.052+0000 7fd15b200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd154114620 con 0x7fd154102f50 2024-02-28T16:37:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.052+0000 7fd15b200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15410d950 con 0x7fd1541038f0 2024-02-28T16:37:04.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.052+0000 7fd153e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd154102f50 0x7fd154113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:04.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.053+0000 7fd153e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd154102f50 0x7fd154113a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34388/0 (socket says 172.21.15.1:34388) 2024-02-28T16:37:04.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.053+0000 7fd153e00700 1 -- 172.21.15.1:0/3634349993 learned_addr learned my addr 172.21.15.1:0/3634349993 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:04.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.053+0000 7fd153400700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1541038f0 0x7fd154113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:04.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.053+0000 7fd153e00700 1 -- 172.21.15.1:0/3634349993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1541038f0 msgr2=0x7fd154113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:04.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.053+0000 7fd153e00700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1541038f0 0x7fd154113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:04.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.053+0000 7fd153e00700 1 -- 172.21.15.1:0/3634349993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd140005040 con 0x7fd154102f50 2024-02-28T16:37:04.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.053+0000 7fd153e00700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd154102f50 0x7fd154113a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd144002a50 tx=0x7fd1440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:04.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.054+0000 7fd14be00700 1 -- 172.21.15.1:0/3634349993 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd144007cc0 con 0x7fd154102f50 2024-02-28T16:37:04.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.054+0000 7fd14be00700 1 -- 172.21.15.1:0/3634349993 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd14400f040 con 0x7fd154102f50 2024-02-28T16:37:04.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.054+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15410dc30 con 0x7fd154102f50 2024-02-28T16:37:04.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.054+0000 7fd14be00700 1 -- 172.21.15.1:0/3634349993 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14400db70 con 0x7fd154102f50 2024-02-28T16:37:04.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.054+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15410e170 con 0x7fd154102f50 2024-02-28T16:37:04.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.055+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd154105420 con 0x7fd154102f50 2024-02-28T16:37:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.056+0000 7fd14be00700 1 -- 172.21.15.1:0/3634349993 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 91585+0+0 (secure 0 0 0) 0x7fd14400dd10 con 0x7fd154102f50 2024-02-28T16:37:04.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.058+0000 7fd14be00700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd13806d900 0x7fd13806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:04.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.058+0000 7fd14be00700 1 -- 172.21.15.1:0/3634349993 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fd14408c7b0 con 0x7fd154102f50 2024-02-28T16:37:04.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.058+0000 7fd153400700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd13806d900 0x7fd13806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:04.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.059+0000 7fd153400700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd13806d900 0x7fd13806fdb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd15410f2c0 tx=0x7fd140009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:04.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.066+0000 7fd14be00700 1 -- 172.21.15.1:0/3634349993 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd1440599c0 con 0x7fd154102f50 2024-02-28T16:37:04.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.378+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd15404bed0 con 0x7fd154102f50 2024-02-28T16:37:04.368 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:37:04.368 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:37:04.368 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:37:04.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.379+0000 7fd14be00700 1 -- 172.21.15.1:0/3634349993 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fd144059550 con 0x7fd154102f50 2024-02-28T16:37:04.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.381+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd13806d900 msgr2=0x7fd13806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:04.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.381+0000 7fd15b200700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd13806d900 0x7fd13806fdb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd15410f2c0 tx=0x7fd140009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:04.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.381+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd154102f50 msgr2=0x7fd154113a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:04.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.381+0000 7fd15b200700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd154102f50 0x7fd154113a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd144002a50 tx=0x7fd1440046b0 comp rx=0 tx=0).stop 2024-02-28T16:37:04.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.381+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 shutdown_connections 2024-02-28T16:37:04.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.381+0000 7fd15b200700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd13806d900 0x7fd13806fdb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:04.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.381+0000 7fd15b200700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd154102f50 0x7fd154113a90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:04.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.381+0000 7fd15b200700 1 --2- 172.21.15.1:0/3634349993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1541038f0 0x7fd154113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:04.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.382+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 >> 172.21.15.1:0/3634349993 conn(0x7fd1540f8ae0 msgr2=0x7fd1540fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:04.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.382+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 shutdown_connections 2024-02-28T16:37:04.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:04.382+0000 7fd15b200700 1 -- 172.21.15.1:0/3634349993 wait complete. 2024-02-28T16:37:04.661 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:04 smithi001 ceph-mon[112940]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:04.661 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:04 smithi001 ceph-mon[112940]: pgmap v44: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:04.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:04 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2061441790' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:37:04.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:04.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:37:04.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:04.662 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:04 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3634349993' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:37:04.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:04 smithi066 ceph-mon[85781]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:04 smithi066 ceph-mon[85781]: pgmap v44: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:04 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2061441790' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:37:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:37:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:04.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:04 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3634349993' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:37:05.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:05 smithi066 ceph-mon[85781]: Upgrade: Updating crash.smithi001 (1/2) 2024-02-28T16:37:05.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:05 smithi066 ceph-mon[85781]: Deploying daemon crash.smithi001 on smithi001 2024-02-28T16:37:05.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:05 smithi066 ceph-mon[85781]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:05.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:05 smithi001 ceph-mon[112940]: Upgrade: Updating crash.smithi001 (1/2) 2024-02-28T16:37:05.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:05 smithi001 ceph-mon[112940]: Deploying daemon crash.smithi001 on smithi001 2024-02-28T16:37:05.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:05 smithi001 ceph-mon[112940]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:06.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:06 smithi066 ceph-mon[85781]: pgmap v45: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:06.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:06 smithi001 ceph-mon[112940]: pgmap v45: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:07.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:07 smithi001 ceph-mon[112940]: pgmap v46: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:07.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:07 smithi066 ceph-mon[85781]: pgmap v46: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:10.683 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:10 smithi066 ceph-mon[85781]: pgmap v47: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:10.683 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:10.683 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:10 smithi001 ceph-mon[112940]: pgmap v47: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:10.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:11.638 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:37:11.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:37:12.715 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:12 smithi066 ceph-mon[85781]: pgmap v48: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:12.716 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:12 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:12.716 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:12 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi066", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:37:12.716 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:12 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:12.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:12 smithi001 ceph-mon[112940]: pgmap v48: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:12.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:12 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:12.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:12 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi066", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-28T16:37:12.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:12 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:13.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:13 smithi066 ceph-mon[85781]: Upgrade: Updating crash.smithi066 (2/2) 2024-02-28T16:37:13.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:13 smithi066 ceph-mon[85781]: Deploying daemon crash.smithi066 on smithi066 2024-02-28T16:37:13.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:13 smithi001 ceph-mon[112940]: Upgrade: Updating crash.smithi066 (2/2) 2024-02-28T16:37:13.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:13 smithi001 ceph-mon[112940]: Deploying daemon crash.smithi066 on smithi066 2024-02-28T16:37:14.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:14 smithi066 ceph-mon[85781]: pgmap v49: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:14.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:14 smithi066 ceph-mon[85781]: mgrmap e39: smithi001.rmthtx(active, since 92s), standbys: smithi066.ztraxe 2024-02-28T16:37:14.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:14 smithi001 ceph-mon[112940]: pgmap v49: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:14.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:14 smithi001 ceph-mon[112940]: mgrmap e39: smithi001.rmthtx(active, since 92s), standbys: smithi066.ztraxe 2024-02-28T16:37:16.660 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:16 smithi066 ceph-mon[85781]: pgmap v50: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:16.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:16 smithi001 ceph-mon[112940]: pgmap v50: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:18.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:18 smithi066 ceph-mon[85781]: pgmap v51: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:18.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:18 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:18.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:18 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:18.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:18 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:37:18.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:18 smithi001 ceph-mon[112940]: pgmap v51: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:18.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:18 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:18.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:18 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:18.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:18 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:37:20.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:20 smithi066 ceph-mon[85781]: pgmap v52: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:20.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:20 smithi001 ceph-mon[112940]: pgmap v52: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:22.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:22 smithi066 ceph-mon[85781]: pgmap v53: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:22.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:22 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:22.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:22 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:22.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:22 smithi001 ceph-mon[112940]: pgmap v53: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:22.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:22 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:22.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:22 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:24.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:24 smithi066 ceph-mon[85781]: pgmap v54: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:24.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:24 smithi001 ceph-mon[112940]: pgmap v54: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-28T16:37:25.742 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:25.742 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:25.742 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:25.742 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:25.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:25.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:25.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:25.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:26 smithi066 ceph-mon[85781]: pgmap v55: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:26.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:37:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:26 smithi001 ceph-mon[112940]: pgmap v55: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:37:28.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:28 smithi001 ceph-mon[112940]: pgmap v56: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:28.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:28 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:28.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:28 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:28.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:28 smithi066 ceph-mon[85781]: pgmap v56: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:28.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:28 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:28.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:28 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:30.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:30 smithi066 ceph-mon[85781]: pgmap v57: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:30.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:30 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:30.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:30 smithi001 ceph-mon[112940]: pgmap v57: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:30.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:30 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:31.324 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:31 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:31.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:31 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:32 smithi066 ceph-mon[85781]: pgmap v58: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:32.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:32 smithi001 ceph-mon[112940]: pgmap v58: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:33.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: pgmap v59: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:33.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:33.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all crash 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi001"}]: dispatch 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi001"}]': finished 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi066"}]: dispatch 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi066"}]': finished 2024-02-28T16:37:33.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-28T16:37:34.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-28T16:37:34.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:33 smithi066 ceph-mon[85781]: Upgrade: osd.0 is safe to restart 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: pgmap v59: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all crash 2024-02-28T16:37:34.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:34.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi001"}]: dispatch 2024-02-28T16:37:34.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi001"}]': finished 2024-02-28T16:37:34.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi066"}]: dispatch 2024-02-28T16:37:34.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi066"}]': finished 2024-02-28T16:37:34.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-28T16:37:34.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-28T16:37:34.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:33 smithi001 ceph-mon[112940]: Upgrade: osd.0 is safe to restart 2024-02-28T16:37:34.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.549+0000 7f692f000700 1 -- 172.21.15.1:0/2080993587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 msgr2=0x7f69280fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:34.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.549+0000 7f692f000700 1 --2- 172.21.15.1:0/2080993587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 0x7f69280fe3b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6918004970 tx=0x7f6918009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:34.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.549+0000 7f692f000700 1 -- 172.21.15.1:0/2080993587 shutdown_connections 2024-02-28T16:37:34.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.549+0000 7f692f000700 1 --2- 172.21.15.1:0/2080993587 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69280fe8f0 0x7f6928107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.549+0000 7f692f000700 1 --2- 172.21.15.1:0/2080993587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 0x7f69280fe3b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.549+0000 7f692f000700 1 -- 172.21.15.1:0/2080993587 >> 172.21.15.1:0/2080993587 conn(0x7f6928071d90 msgr2=0x7f6928072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:34.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.549+0000 7f692f000700 1 -- 172.21.15.1:0/2080993587 shutdown_connections 2024-02-28T16:37:34.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.549+0000 7f692f000700 1 -- 172.21.15.1:0/2080993587 wait complete. 2024-02-28T16:37:34.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.551+0000 7f692f000700 1 Processor -- start 2024-02-28T16:37:34.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.551+0000 7f692f000700 1 -- start start 2024-02-28T16:37:34.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.551+0000 7f692f000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 0x7f6928113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:34.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f692f000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69280fe8f0 0x7f6928113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:34.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f692f000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69281145e0 con 0x7f69280fdfe0 2024-02-28T16:37:34.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f692f000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692810d910 con 0x7f69280fe8f0 2024-02-28T16:37:34.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f6927e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 0x7f6928113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:34.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f6927e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 0x7f6928113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33626/0 (socket says 172.21.15.1:33626) 2024-02-28T16:37:34.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f6927e00700 1 -- 172.21.15.1:0/981084513 learned_addr learned my addr 172.21.15.1:0/981084513 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:34.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f6927400700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69280fe8f0 0x7f6928113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:34.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f6927e00700 1 -- 172.21.15.1:0/981084513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69280fe8f0 msgr2=0x7f6928113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:34.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f6927e00700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69280fe8f0 0x7f6928113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.552+0000 7f6927e00700 1 -- 172.21.15.1:0/981084513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6918005040 con 0x7f69280fdfe0 2024-02-28T16:37:34.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.553+0000 7f6927e00700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 0x7f6928113a50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6918009150 tx=0x7f6918009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:34.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.553+0000 7f691fe00700 1 -- 172.21.15.1:0/981084513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f691800e040 con 0x7f69280fdfe0 2024-02-28T16:37:34.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.553+0000 7f691fe00700 1 -- 172.21.15.1:0/981084513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6918004290 con 0x7f69280fdfe0 2024-02-28T16:37:34.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.553+0000 7f691fe00700 1 -- 172.21.15.1:0/981084513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f691801c630 con 0x7f69280fdfe0 2024-02-28T16:37:34.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.553+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692810db90 con 0x7f69280fdfe0 2024-02-28T16:37:34.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.553+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692810e0d0 con 0x7f69280fdfe0 2024-02-28T16:37:34.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.555+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69281046c0 con 0x7f69280fdfe0 2024-02-28T16:37:34.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.556+0000 7f691fe00700 1 -- 172.21.15.1:0/981084513 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6918013980 con 0x7f69280fdfe0 2024-02-28T16:37:34.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.557+0000 7f691fe00700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f690c06d7f0 0x7f690c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:34.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.557+0000 7f691fe00700 1 -- 172.21.15.1:0/981084513 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f691808e080 con 0x7f69280fdfe0 2024-02-28T16:37:34.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.558+0000 7f6927400700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f690c06d7f0 0x7f690c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:34.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.559+0000 7f6927400700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f690c06d7f0 0x7f690c06fca0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f692810f280 tx=0x7f691400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.563+0000 7f691fe00700 1 -- 172.21.15.1:0/981084513 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f691805b5d0 con 0x7f69280fdfe0 2024-02-28T16:37:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.822+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f692804bed0 con 0x7f690c06d7f0 2024-02-28T16:37:34.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.826+0000 7f691fe00700 1 -- 172.21.15.1:0/981084513 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f692804bed0 con 0x7f690c06d7f0 2024-02-28T16:37:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f690c06d7f0 msgr2=0x7f690c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f690c06d7f0 0x7f690c06fca0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f692810f280 tx=0x7f691400b040 comp rx=0 tx=0).stop 2024-02-28T16:37:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 msgr2=0x7f6928113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 0x7f6928113a50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6918009150 tx=0x7f6918009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 shutdown_connections 2024-02-28T16:37:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f690c06d7f0 0x7f690c06fca0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69280fdfe0 0x7f6928113a50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 --2- 172.21.15.1:0/981084513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f69280fe8f0 0x7f6928113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.828+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 >> 172.21.15.1:0/981084513 conn(0x7f6928071d90 msgr2=0x7f6928101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.829+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 shutdown_connections 2024-02-28T16:37:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.829+0000 7f692f000700 1 -- 172.21.15.1:0/981084513 wait complete. 2024-02-28T16:37:34.834 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:37:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.995+0000 7faa10a00700 1 -- 172.21.15.1:0/1494936523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fdf60 msgr2=0x7faa0c0fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.995+0000 7faa10a00700 1 --2- 172.21.15.1:0/1494936523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fdf60 0x7faa0c0fe330 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc004940 tx=0x7fa9fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.995+0000 7faa10a00700 1 -- 172.21.15.1:0/1494936523 shutdown_connections 2024-02-28T16:37:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.995+0000 7faa10a00700 1 --2- 172.21.15.1:0/1494936523 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa0c0fe900 0x7faa0c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.995+0000 7faa10a00700 1 --2- 172.21.15.1:0/1494936523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fdf60 0x7faa0c0fe330 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.995+0000 7faa10a00700 1 -- 172.21.15.1:0/1494936523 >> 172.21.15.1:0/1494936523 conn(0x7faa0c0f7a10 msgr2=0x7faa0c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:34.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.995+0000 7faa10a00700 1 -- 172.21.15.1:0/1494936523 shutdown_connections 2024-02-28T16:37:34.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.995+0000 7faa10a00700 1 -- 172.21.15.1:0/1494936523 wait complete. 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.996+0000 7faa10a00700 1 Processor -- start 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.997+0000 7faa10a00700 1 -- start start 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.997+0000 7faa10a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa0c0fdf60 0x7faa0c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.997+0000 7faa10a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fe900 0x7faa0c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.997+0000 7faa10a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c116810 con 0x7faa0c0fe900 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.997+0000 7faa10a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c10fb40 con 0x7faa0c0fdf60 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.997+0000 7faa01400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fe900 0x7faa0c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.997+0000 7faa01400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fe900 0x7faa0c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33644/0 (socket says 172.21.15.1:33644) 2024-02-28T16:37:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.997+0000 7faa01400700 1 -- 172.21.15.1:0/1216516164 learned_addr learned my addr 172.21.15.1:0/1216516164 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.998+0000 7faa09e00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa0c0fdf60 0x7faa0c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.998+0000 7faa01400700 1 -- 172.21.15.1:0/1216516164 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa0c0fdf60 msgr2=0x7faa0c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.998+0000 7faa01400700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa0c0fdf60 0x7faa0c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.998+0000 7faa01400700 1 -- 172.21.15.1:0/1216516164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc005040 con 0x7faa0c0fe900 2024-02-28T16:37:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.998+0000 7faa01400700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fe900 0x7faa0c1161c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4002a50 tx=0x7fa9f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.999+0000 7faa02a00700 1 -- 172.21.15.1:0/1216516164 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9f4007cc0 con 0x7faa0c0fe900 2024-02-28T16:37:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.999+0000 7faa02a00700 1 -- 172.21.15.1:0/1216516164 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9f400f040 con 0x7faa0c0fe900 2024-02-28T16:37:34.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.999+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c10fe20 con 0x7faa0c0fe900 2024-02-28T16:37:34.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.999+0000 7faa02a00700 1 -- 172.21.15.1:0/1216516164 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9f4009640 con 0x7faa0c0fe900 2024-02-28T16:37:34.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:34.999+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c110360 con 0x7faa0c0fe900 2024-02-28T16:37:34.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.000+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0c1069b0 con 0x7faa0c0fe900 2024-02-28T16:37:34.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.001+0000 7faa02a00700 1 -- 172.21.15.1:0/1216516164 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa9f40097a0 con 0x7faa0c0fe900 2024-02-28T16:37:34.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.002+0000 7faa02a00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9f006d7f0 0x7fa9f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:34.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.002+0000 7faa02a00700 1 -- 172.21.15.1:0/1216516164 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fa9f408cc70 con 0x7faa0c0fe900 2024-02-28T16:37:34.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.002+0000 7faa09e00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9f006d7f0 0x7fa9f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:34.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.003+0000 7faa09e00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9f006d7f0 0x7fa9f006fca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009fd0 tx=0x7fa9fc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:34.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.007+0000 7faa02a00700 1 -- 172.21.15.1:0/1216516164 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa9f4093080 con 0x7faa0c0fe900 2024-02-28T16:37:35.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.255+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faa0c04bed0 con 0x7fa9f006d7f0 2024-02-28T16:37:35.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.259+0000 7faa02a00700 1 -- 172.21.15.1:0/1216516164 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7faa0c04bed0 con 0x7fa9f006d7f0 2024-02-28T16:37:35.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.263+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9f006d7f0 msgr2=0x7fa9f006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:35.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.263+0000 7faa10a00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9f006d7f0 0x7fa9f006fca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009fd0 tx=0x7fa9fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:35.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.263+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fe900 msgr2=0x7faa0c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:35.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.263+0000 7faa10a00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fe900 0x7faa0c1161c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4002a50 tx=0x7fa9f40046b0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.264+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 shutdown_connections 2024-02-28T16:37:35.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.264+0000 7faa10a00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9f006d7f0 0x7fa9f006fca0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.264+0000 7faa10a00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa0c0fdf60 0x7faa0c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.264+0000 7faa10a00700 1 --2- 172.21.15.1:0/1216516164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0c0fe900 0x7faa0c1161c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.264+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 >> 172.21.15.1:0/1216516164 conn(0x7faa0c0f7a10 msgr2=0x7faa0c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:35.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.264+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 shutdown_connections 2024-02-28T16:37:35.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.264+0000 7faa10a00700 1 -- 172.21.15.1:0/1216516164 wait complete. 2024-02-28T16:37:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.421+0000 7fd46f600700 1 -- 172.21.15.1:0/3054340064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fdfe0 msgr2=0x7fd4680fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.421+0000 7fd46f600700 1 --2- 172.21.15.1:0/3054340064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fdfe0 0x7fd4680fe3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd460009d10 tx=0x7fd460004d10 comp rx=0 tx=0).stop 2024-02-28T16:37:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.421+0000 7fd46f600700 1 -- 172.21.15.1:0/3054340064 shutdown_connections 2024-02-28T16:37:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.421+0000 7fd46f600700 1 --2- 172.21.15.1:0/3054340064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4680fe8f0 0x7fd468107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.421+0000 7fd46f600700 1 --2- 172.21.15.1:0/3054340064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fdfe0 0x7fd4680fe3b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.421+0000 7fd46f600700 1 -- 172.21.15.1:0/3054340064 >> 172.21.15.1:0/3054340064 conn(0x7fd468071d90 msgr2=0x7fd468072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.421+0000 7fd46f600700 1 -- 172.21.15.1:0/3054340064 shutdown_connections 2024-02-28T16:37:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.422+0000 7fd46f600700 1 -- 172.21.15.1:0/3054340064 wait complete. 2024-02-28T16:37:35.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.423+0000 7fd46f600700 1 Processor -- start 2024-02-28T16:37:35.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.423+0000 7fd46f600700 1 -- start start 2024-02-28T16:37:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.424+0000 7fd46f600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4680fdfe0 0x7fd468115e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.424+0000 7fd46f600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fe8f0 0x7fd468116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.424+0000 7fd46f600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4681169b0 con 0x7fd4680fe8f0 2024-02-28T16:37:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.424+0000 7fd46f600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46810fcc0 con 0x7fd4680fdfe0 2024-02-28T16:37:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.424+0000 7fd467e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fe8f0 0x7fd468116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.424+0000 7fd467e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fe8f0 0x7fd468116360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33664/0 (socket says 172.21.15.1:33664) 2024-02-28T16:37:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.424+0000 7fd467e00700 1 -- 172.21.15.1:0/1193671931 learned_addr learned my addr 172.21.15.1:0/1193671931 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.424+0000 7fd46cc00700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4680fdfe0 0x7fd468115e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.425+0000 7fd467e00700 1 -- 172.21.15.1:0/1193671931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4680fdfe0 msgr2=0x7fd468115e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.425+0000 7fd467e00700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4680fdfe0 0x7fd468115e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.425+0000 7fd467e00700 1 -- 172.21.15.1:0/1193671931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd460009940 con 0x7fd4680fe8f0 2024-02-28T16:37:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.425+0000 7fd467e00700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fe8f0 0x7fd468116360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd458002a50 tx=0x7fd4580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.426+0000 7fd464c00700 1 -- 172.21.15.1:0/1193671931 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd458007cc0 con 0x7fd4680fe8f0 2024-02-28T16:37:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.426+0000 7fd464c00700 1 -- 172.21.15.1:0/1193671931 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd45800f040 con 0x7fd4680fe8f0 2024-02-28T16:37:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.426+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46810ffa0 con 0x7fd4680fe8f0 2024-02-28T16:37:35.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.426+0000 7fd464c00700 1 -- 172.21.15.1:0/1193671931 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd45800da30 con 0x7fd4680fe8f0 2024-02-28T16:37:35.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.426+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4681104e0 con 0x7fd4680fe8f0 2024-02-28T16:37:35.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.427+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd468106b20 con 0x7fd4680fe8f0 2024-02-28T16:37:35.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.428+0000 7fd464c00700 1 -- 172.21.15.1:0/1193671931 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd45800db90 con 0x7fd4680fe8f0 2024-02-28T16:37:35.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.429+0000 7fd464c00700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd44c06d7e0 0x7fd44c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:35.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.429+0000 7fd464c00700 1 -- 172.21.15.1:0/1193671931 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fd45808cc50 con 0x7fd4680fe8f0 2024-02-28T16:37:35.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.430+0000 7fd46cc00700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd44c06d7e0 0x7fd44c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:35.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.430+0000 7fd46cc00700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd44c06d7e0 0x7fd44c06fc90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd460003400 tx=0x7fd460004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:35.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.434+0000 7fd464c00700 1 -- 172.21.15.1:0/1193671931 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd458059fe0 con 0x7fd4680fe8f0 2024-02-28T16:37:35.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.671+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd46804bf40 con 0x7fd44c06d7e0 2024-02-28T16:37:35.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.684+0000 7fd464c00700 1 -- 172.21.15.1:0/1193671931 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd46804bf40 con 0x7fd44c06d7e0 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (3m) 10s ago 17m 14.8M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (26s) 10s ago 18m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (18s) 13s ago 16m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (3m) 10s ago 17m 75.7M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (10m) 10s ago 10m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (10m) 10s ago 10m 14.6M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (10m) 13s ago 10m 14.3M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:37:35.672 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (10m) 13s ago 10m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (6m) 10s ago 19m 496M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (5m) 13s ago 16m 449M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (2m) 10s ago 19m 43.3M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (87s) 13s ago 16m 35.0M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (4m) 10s ago 18m 7570k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (4m) 13s ago 16m 7612k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (15m) 10s ago 15m 208M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 77197415288f 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (15m) 10s ago 15m 208M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (14m) 10s ago 14m 169M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (13m) 13s ago 13m 222M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:37:35.673 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (13m) 13s ago 13m 214M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:37:35.674 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (12m) 13s ago 12m 189M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:37:35.674 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (4m) 10s ago 17m 51.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:37:35.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.687+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd44c06d7e0 msgr2=0x7fd44c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.687+0000 7fd46f600700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd44c06d7e0 0x7fd44c06fc90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd460003400 tx=0x7fd460004d10 comp rx=0 tx=0).stop 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.687+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fe8f0 msgr2=0x7fd468116360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.687+0000 7fd46f600700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fe8f0 0x7fd468116360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd458002a50 tx=0x7fd4580046b0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.688+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 shutdown_connections 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.688+0000 7fd46f600700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd44c06d7e0 0x7fd44c06fc90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.688+0000 7fd46f600700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4680fdfe0 0x7fd468115e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.688+0000 7fd46f600700 1 --2- 172.21.15.1:0/1193671931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4680fe8f0 0x7fd468116360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.688+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 >> 172.21.15.1:0/1193671931 conn(0x7fd468071d90 msgr2=0x7fd4680fba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.688+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 shutdown_connections 2024-02-28T16:37:35.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.688+0000 7fd46f600700 1 -- 172.21.15.1:0/1193671931 wait complete. 2024-02-28T16:37:35.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.854+0000 7fb100000700 1 -- 172.21.15.1:0/518911336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fe8f0 msgr2=0x7fb0f8107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:35.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.854+0000 7fb100000700 1 --2- 172.21.15.1:0/518911336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fe8f0 0x7fb0f8107050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4004970 tx=0x7fb0e4009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:35.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.854+0000 7fb100000700 1 -- 172.21.15.1:0/518911336 shutdown_connections 2024-02-28T16:37:35.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.854+0000 7fb100000700 1 --2- 172.21.15.1:0/518911336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fe8f0 0x7fb0f8107050 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.854+0000 7fb100000700 1 --2- 172.21.15.1:0/518911336 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0f80fdfe0 0x7fb0f80fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.854+0000 7fb100000700 1 -- 172.21.15.1:0/518911336 >> 172.21.15.1:0/518911336 conn(0x7fb0f8071d90 msgr2=0x7fb0f8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:35.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.854+0000 7fb100000700 1 -- 172.21.15.1:0/518911336 shutdown_connections 2024-02-28T16:37:35.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.854+0000 7fb100000700 1 -- 172.21.15.1:0/518911336 wait complete. 2024-02-28T16:37:35.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.855+0000 7fb100000700 1 Processor -- start 2024-02-28T16:37:35.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.855+0000 7fb100000700 1 -- start start 2024-02-28T16:37:35.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.856+0000 7fb100000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fdfe0 0x7fb0f8115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:35.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.856+0000 7fb100000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0f80fe8f0 0x7fb0f8116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:35.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.856+0000 7fb100000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f8116790 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.856+0000 7fb100000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f810fac0 con 0x7fb0f80fe8f0 2024-02-28T16:37:35.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.856+0000 7fb0fd600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fdfe0 0x7fb0f8115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:35.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.856+0000 7fb0fd600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fdfe0 0x7fb0f8115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33678/0 (socket says 172.21.15.1:33678) 2024-02-28T16:37:35.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.856+0000 7fb0fd600700 1 -- 172.21.15.1:0/920169365 learned_addr learned my addr 172.21.15.1:0/920169365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:35.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.857+0000 7fb0fcc00700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0f80fe8f0 0x7fb0f8116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:35.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.857+0000 7fb0fd600700 1 -- 172.21.15.1:0/920169365 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0f80fe8f0 msgr2=0x7fb0f8116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:35.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.857+0000 7fb0fd600700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0f80fe8f0 0x7fb0f8116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:35.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.857+0000 7fb0fd600700 1 -- 172.21.15.1:0/920169365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e4005040 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.857+0000 7fb0fd600700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fdfe0 0x7fb0f8115c00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec002a50 tx=0x7fb0ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:35.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.857+0000 7fb0f5600700 1 -- 172.21.15.1:0/920169365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0ec007cc0 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.857+0000 7fb0f5600700 1 -- 172.21.15.1:0/920169365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0ec00f040 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.857+0000 7fb0f5600700 1 -- 172.21.15.1:0/920169365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0ec009640 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.858+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f810fda0 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.858+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f81102b0 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.859+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f8106b20 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.861+0000 7fb0f5600700 1 -- 172.21.15.1:0/920169365 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb0ec0097c0 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.862+0000 7fb0f5600700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb0e006d7f0 0x7fb0e006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:35.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.862+0000 7fb0f5600700 1 -- 172.21.15.1:0/920169365 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fb0ec08cb70 con 0x7fb0f80fdfe0 2024-02-28T16:37:35.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.862+0000 7fb0fcc00700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb0e006d7f0 0x7fb0e006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.863+0000 7fb0fcc00700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb0e006d7f0 0x7fb0e006fca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8110df0 tx=0x7fb0e4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:35.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:35.872+0000 7fb0f5600700 1 -- 172.21.15.1:0/920169365 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb0ec059f80 con 0x7fb0f80fdfe0 2024-02-28T16:37:36.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.218+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb0f804bf20 con 0x7fb0f80fdfe0 2024-02-28T16:37:36.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.219+0000 7fb0f5600700 1 -- 172.21.15.1:0/920169365 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7fb0ec059b10 con 0x7fb0f80fdfe0 2024-02-28T16:37:36.207 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:37:36.207 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:37:36.208 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 10, 2024-02-28T16:37:36.209 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T16:37:36.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:37:36.209 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb0e006d7f0 msgr2=0x7fb0e006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb0e006d7f0 0x7fb0e006fca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8110df0 tx=0x7fb0e4009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fdfe0 msgr2=0x7fb0f8115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fdfe0 0x7fb0f8115c00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec002a50 tx=0x7fb0ec0046b0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 shutdown_connections 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb0e006d7f0 0x7fb0e006fca0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0f80fdfe0 0x7fb0f8115c00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 --2- 172.21.15.1:0/920169365 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0f80fe8f0 0x7fb0f8116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 >> 172.21.15.1:0/920169365 conn(0x7fb0f8071d90 msgr2=0x7fb0f80fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:36.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 shutdown_connections 2024-02-28T16:37:36.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.223+0000 7fb100000700 1 -- 172.21.15.1:0/920169365 wait complete. 2024-02-28T16:37:36.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.369+0000 7fbdee200700 1 -- 172.21.15.1:0/1975319500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8070870 msgr2=0x7fbde8070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:36.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.369+0000 7fbdee200700 1 --2- 172.21.15.1:0/1975319500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8070870 0x7fbde8070ce0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4004970 tx=0x7fbdd4009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:36.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.370+0000 7fbdee200700 1 -- 172.21.15.1:0/1975319500 shutdown_connections 2024-02-28T16:37:36.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.370+0000 7fbdee200700 1 --2- 172.21.15.1:0/1975319500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8070870 0x7fbde8070ce0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.370+0000 7fbdee200700 1 --2- 172.21.15.1:0/1975319500 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbde8071f60 0x7fbde8070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.370+0000 7fbdee200700 1 -- 172.21.15.1:0/1975319500 >> 172.21.15.1:0/1975319500 conn(0x7fbde80f8ae0 msgr2=0x7fbde80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:36.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.370+0000 7fbdee200700 1 -- 172.21.15.1:0/1975319500 shutdown_connections 2024-02-28T16:37:36.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.370+0000 7fbdee200700 1 -- 172.21.15.1:0/1975319500 wait complete. 2024-02-28T16:37:36.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.371+0000 7fbdee200700 1 Processor -- start 2024-02-28T16:37:36.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.371+0000 7fbdee200700 1 -- start start 2024-02-28T16:37:36.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.371+0000 7fbdee200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbde8070870 0x7fbde811a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:36.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.371+0000 7fbdee200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8071f60 0x7fbde811a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:36.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.371+0000 7fbdee200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde811aba0 con 0x7fbde8071f60 2024-02-28T16:37:36.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.371+0000 7fbdee200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8113eb0 con 0x7fbde8070870 2024-02-28T16:37:36.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.372+0000 7fbde6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8071f60 0x7fbde811a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:36.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.372+0000 7fbde6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8071f60 0x7fbde811a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33690/0 (socket says 172.21.15.1:33690) 2024-02-28T16:37:36.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.372+0000 7fbde6a00700 1 -- 172.21.15.1:0/4179850757 learned_addr learned my addr 172.21.15.1:0/4179850757 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:36.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.372+0000 7fbde7400700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbde8070870 0x7fbde811a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:36.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.372+0000 7fbde6a00700 1 -- 172.21.15.1:0/4179850757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbde8070870 msgr2=0x7fbde811a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:36.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.372+0000 7fbde6a00700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbde8070870 0x7fbde811a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.372+0000 7fbde6a00700 1 -- 172.21.15.1:0/4179850757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd4005040 con 0x7fbde8071f60 2024-02-28T16:37:36.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.372+0000 7fbde6a00700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8071f60 0x7fbde811a550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4004f50 tx=0x7fbdd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.373+0000 7fbddf400700 1 -- 172.21.15.1:0/4179850757 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdd400e040 con 0x7fbde8071f60 2024-02-28T16:37:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.373+0000 7fbddf400700 1 -- 172.21.15.1:0/4179850757 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdd400bc10 con 0x7fbde8071f60 2024-02-28T16:37:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.373+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde8114130 con 0x7fbde8071f60 2024-02-28T16:37:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.373+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde8114670 con 0x7fbde8071f60 2024-02-28T16:37:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.373+0000 7fbddf400700 1 -- 172.21.15.1:0/4179850757 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdd401c630 con 0x7fbde8071f60 2024-02-28T16:37:36.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.374+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde80fe400 con 0x7fbde8071f60 2024-02-28T16:37:36.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.375+0000 7fbddf400700 1 -- 172.21.15.1:0/4179850757 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbdd400bd80 con 0x7fbde8071f60 2024-02-28T16:37:36.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.376+0000 7fbddf400700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdcc06d7f0 0x7fbdcc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:36.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.376+0000 7fbddf400700 1 -- 172.21.15.1:0/4179850757 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fbdd408ed40 con 0x7fbde8071f60 2024-02-28T16:37:36.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.376+0000 7fbde7400700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdcc06d7f0 0x7fbdcc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:36.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.377+0000 7fbde7400700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdcc06d7f0 0x7fbdcc06fca0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8004990 tx=0x7fbdd8009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:36.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.380+0000 7fbddf400700 1 -- 172.21.15.1:0/4179850757 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbdd405c1d0 con 0x7fbde8071f60 2024-02-28T16:37:36.482 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: pgmap v60: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:36.482 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:36.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:36.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: Upgrade: Updating osd.0 2024-02-28T16:37:36.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:36.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-28T16:37:36.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:36.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: Deploying daemon osd.0 on smithi001 2024-02-28T16:37:36.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/920169365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:36.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.653+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbde81142c0 con 0x7fbde8071f60 2024-02-28T16:37:36.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.653+0000 7fbddf400700 1 -- 172.21.15.1:0/4179850757 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7fbde81142c0 con 0x7fbde8071f60 2024-02-28T16:37:36.641 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:37:36.641 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:37:36.641 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:37:36.641 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:37:36.642 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:37:36.643 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:37:36.644 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:37:36.644 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:37:36.644 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:37:36.644 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:37:36.644 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:37:36.644 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:37:36.644 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:37:36.644 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.656+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdcc06d7f0 msgr2=0x7fbdcc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.656+0000 7fbdee200700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdcc06d7f0 0x7fbdcc06fca0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8004990 tx=0x7fbdd8009ee0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.656+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8071f60 msgr2=0x7fbde811a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.656+0000 7fbdee200700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8071f60 0x7fbde811a550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4004f50 tx=0x7fbdd4009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.657+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 shutdown_connections 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.657+0000 7fbdee200700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdcc06d7f0 0x7fbdcc06fca0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.657+0000 7fbdee200700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbde8070870 0x7fbde811a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.657+0000 7fbdee200700 1 --2- 172.21.15.1:0/4179850757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbde8071f60 0x7fbde811a550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.657+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 >> 172.21.15.1:0/4179850757 conn(0x7fbde80f8ae0 msgr2=0x7fbde80fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:36.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.657+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 shutdown_connections 2024-02-28T16:37:36.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.657+0000 7fbdee200700 1 -- 172.21.15.1:0/4179850757 wait complete. 2024-02-28T16:37:36.646 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:37:36.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: pgmap v60: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:36.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:36.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: Upgrade: Updating osd.0 2024-02-28T16:37:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-28T16:37:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:37:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: Deploying daemon osd.0 on smithi001 2024-02-28T16:37:36.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/920169365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.817+0000 7f8223e00700 1 -- 172.21.15.1:0/4064761022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 msgr2=0x7f82240fef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.817+0000 7f8223e00700 1 --2- 172.21.15.1:0/4064761022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 0x7f82240fef90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8210004940 tx=0x7f8210009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.817+0000 7f8223e00700 1 -- 172.21.15.1:0/4064761022 shutdown_connections 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.817+0000 7f8223e00700 1 --2- 172.21.15.1:0/4064761022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 0x7f82240fef90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.817+0000 7f8223e00700 1 --2- 172.21.15.1:0/4064761022 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8224104f80 0x7f8224105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.817+0000 7f8223e00700 1 -- 172.21.15.1:0/4064761022 >> 172.21.15.1:0/4064761022 conn(0x7f82240fab10 msgr2=0x7f82240fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.817+0000 7f8223e00700 1 -- 172.21.15.1:0/4064761022 shutdown_connections 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.817+0000 7f8223e00700 1 -- 172.21.15.1:0/4064761022 wait complete. 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.818+0000 7f8223e00700 1 Processor -- start 2024-02-28T16:37:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.818+0000 7f8223e00700 1 -- start start 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8223e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8224104f80 0x7f8224115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8223e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 0x7f8224115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8223e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82241165b0 con 0x7f8224105920 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8223e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822410f8e0 con 0x7f8224104f80 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8222000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 0x7f8224115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8222000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 0x7f8224115f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33720/0 (socket says 172.21.15.1:33720) 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8222000700 1 -- 172.21.15.1:0/1508038040 learned_addr learned my addr 172.21.15.1:0/1508038040 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8222a00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8224104f80 0x7f8224115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8222000700 1 -- 172.21.15.1:0/1508038040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8224104f80 msgr2=0x7f8224115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8222000700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8224104f80 0x7f8224115a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8222000700 1 -- 172.21.15.1:0/1508038040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8210005040 con 0x7f8224105920 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.819+0000 7f8222a00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8224104f80 0x7f8224115a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.820+0000 7f8222000700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 0x7f8224115f60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8210009030 tx=0x7f8210009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.820+0000 7f821aa00700 1 -- 172.21.15.1:0/1508038040 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f821000e040 con 0x7f8224105920 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.820+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822410fb60 con 0x7f8224105920 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.820+0000 7f821aa00700 1 -- 172.21.15.1:0/1508038040 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f821000bbf0 con 0x7f8224105920 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.820+0000 7f821aa00700 1 -- 172.21.15.1:0/1508038040 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f821001c630 con 0x7f8224105920 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.820+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8224110120 con 0x7f8224105920 2024-02-28T16:37:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.822+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822405f5c0 con 0x7f8224105920 2024-02-28T16:37:36.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.824+0000 7f821aa00700 1 -- 172.21.15.1:0/1508038040 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f821000bd60 con 0x7f8224105920 2024-02-28T16:37:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.825+0000 7f821aa00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f820806d7f0 0x7f820806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.825+0000 7f821aa00700 1 -- 172.21.15.1:0/1508038040 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f821008e210 con 0x7f8224105920 2024-02-28T16:37:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.825+0000 7f8222a00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f820806d7f0 0x7f820806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.826+0000 7f8222a00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f820806d7f0 0x7f820806fca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8214007fd0 tx=0x7f8214009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:36.830+0000 7f821aa00700 1 -- 172.21.15.1:0/1508038040 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f821005b620 con 0x7f8224105920 2024-02-28T16:37:37.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.074+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82241156f0 con 0x7f820806d7f0 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.075+0000 7f821aa00700 1 -- 172.21.15.1:0/1508038040 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f82241156f0 con 0x7f820806d7f0 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: "crash" 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:37:37.063 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "6/21 daemons upgraded", 2024-02-28T16:37:37.064 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading osd daemons", 2024-02-28T16:37:37.064 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:37:37.064 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:37:37.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f820806d7f0 msgr2=0x7f820806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:37.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f820806d7f0 0x7f820806fca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8214007fd0 tx=0x7f8214009040 comp rx=0 tx=0).stop 2024-02-28T16:37:37.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 msgr2=0x7f8224115f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:37.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 0x7f8224115f60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8210009030 tx=0x7f8210009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 shutdown_connections 2024-02-28T16:37:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f820806d7f0 0x7f820806fca0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8224104f80 0x7f8224115a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 --2- 172.21.15.1:0/1508038040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8224105920 0x7f8224115f60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 >> 172.21.15.1:0/1508038040 conn(0x7f82240fab10 msgr2=0x7f82240fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 shutdown_connections 2024-02-28T16:37:37.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.078+0000 7f8223e00700 1 -- 172.21.15.1:0/1508038040 wait complete. 2024-02-28T16:37:37.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.225+0000 7f4d82a00700 1 -- 172.21.15.1:0/723834726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c101e40 msgr2=0x7f4d7c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:37.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.225+0000 7f4d82a00700 1 --2- 172.21.15.1:0/723834726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c101e40 0x7f4d7c102210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c004940 tx=0x7f4d6c009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:37.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.226+0000 7f4d82a00700 1 -- 172.21.15.1:0/723834726 shutdown_connections 2024-02-28T16:37:37.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.226+0000 7f4d82a00700 1 --2- 172.21.15.1:0/723834726 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d7c1027e0 0x7f4d7c0fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.226+0000 7f4d82a00700 1 --2- 172.21.15.1:0/723834726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c101e40 0x7f4d7c102210 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.226+0000 7f4d82a00700 1 -- 172.21.15.1:0/723834726 >> 172.21.15.1:0/723834726 conn(0x7f4d7c0f79d0 msgr2=0x7f4d7c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:37.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.226+0000 7f4d82a00700 1 -- 172.21.15.1:0/723834726 shutdown_connections 2024-02-28T16:37:37.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.226+0000 7f4d82a00700 1 -- 172.21.15.1:0/723834726 wait complete. 2024-02-28T16:37:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.227+0000 7f4d82a00700 1 Processor -- start 2024-02-28T16:37:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.227+0000 7f4d82a00700 1 -- start start 2024-02-28T16:37:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.227+0000 7f4d82a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d7c101e40 0x7f4d7c115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.227+0000 7f4d82a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c1027e0 0x7f4d7c1161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.227+0000 7f4d82a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c116820 con 0x7f4d7c1027e0 2024-02-28T16:37:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.227+0000 7f4d82a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c10fb50 con 0x7f4d7c101e40 2024-02-28T16:37:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.228+0000 7f4d7b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c1027e0 0x7f4d7c1161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.228+0000 7f4d7b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c1027e0 0x7f4d7c1161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33742/0 (socket says 172.21.15.1:33742) 2024-02-28T16:37:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.228+0000 7f4d7b400700 1 -- 172.21.15.1:0/2980794011 learned_addr learned my addr 172.21.15.1:0/2980794011 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:37:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.228+0000 7f4d7be00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d7c101e40 0x7f4d7c115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.228+0000 7f4d7b400700 1 -- 172.21.15.1:0/2980794011 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d7c101e40 msgr2=0x7f4d7c115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.228+0000 7f4d7b400700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d7c101e40 0x7f4d7c115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.228+0000 7f4d7b400700 1 -- 172.21.15.1:0/2980794011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c005040 con 0x7f4d7c1027e0 2024-02-28T16:37:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.229+0000 7f4d7b400700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c1027e0 0x7f4d7c1161d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d68002a50 tx=0x7f4d68004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:37.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.229+0000 7f4d73e00700 1 -- 172.21.15.1:0/2980794011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d68007cc0 con 0x7f4d7c1027e0 2024-02-28T16:37:37.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.229+0000 7f4d73e00700 1 -- 172.21.15.1:0/2980794011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d6800f040 con 0x7f4d7c1027e0 2024-02-28T16:37:37.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.229+0000 7f4d73e00700 1 -- 172.21.15.1:0/2980794011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d6800daf0 con 0x7f4d7c1027e0 2024-02-28T16:37:37.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.229+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7c10fe30 con 0x7f4d7c1027e0 2024-02-28T16:37:37.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.229+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7c110370 con 0x7f4d7c1027e0 2024-02-28T16:37:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.230+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d7c0fb5d0 con 0x7f4d7c1027e0 2024-02-28T16:37:37.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.233+0000 7f4d73e00700 1 -- 172.21.15.1:0/2980794011 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4d68009810 con 0x7f4d7c1027e0 2024-02-28T16:37:37.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.234+0000 7f4d73e00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d6006d7f0 0x7f4d6006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:37:37.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.234+0000 7f4d73e00700 1 -- 172.21.15.1:0/2980794011 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f4d6808cb80 con 0x7f4d7c1027e0 2024-02-28T16:37:37.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.235+0000 7f4d7be00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d6006d7f0 0x7f4d6006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:37:37.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.235+0000 7f4d7be00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d6006d7f0 0x7f4d6006fca0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00b020 tx=0x7f4d6c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:37:37.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.239+0000 7f4d73e00700 1 -- 172.21.15.1:0/2980794011 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4d68059f90 con 0x7f4d7c1027e0 2024-02-28T16:37:37.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.546+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4d7c108f40 con 0x7f4d7c1027e0 2024-02-28T16:37:37.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.546+0000 7f4d73e00700 1 -- 172.21.15.1:0/2980794011 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f4d68059b20 con 0x7f4d7c1027e0 2024-02-28T16:37:37.534 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:37:37.535 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:37:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d6006d7f0 msgr2=0x7f4d6006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d6006d7f0 0x7f4d6006fca0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00b020 tx=0x7f4d6c009e30 comp rx=0 tx=0).stop 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c1027e0 msgr2=0x7f4d7c1161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c1027e0 0x7f4d7c1161d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d68002a50 tx=0x7f4d68004600 comp rx=0 tx=0).stop 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 shutdown_connections 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d6006d7f0 0x7f4d6006fca0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d7c101e40 0x7f4d7c115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 --2- 172.21.15.1:0/2980794011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d7c1027e0 0x7f4d7c1161d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 >> 172.21.15.1:0/2980794011 conn(0x7f4d7c0f79d0 msgr2=0x7f4d7c066010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 shutdown_connections 2024-02-28T16:37:37.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:37:37.549+0000 7f4d82a00700 1 -- 172.21.15.1:0/2980794011 wait complete. 2024-02-28T16:37:37.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:37 smithi066 ceph-mon[85781]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:37.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:37 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4179850757' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:37:37.766 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:37 smithi001 ceph-mon[112940]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:37.767 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:37 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4179850757' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:37:38.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:38 smithi066 ceph-mon[85781]: pgmap v61: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:38.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:38 smithi066 ceph-mon[85781]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:38.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:38 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2980794011' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:37:38.759 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:38 smithi001 ceph-mon[112940]: pgmap v61: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:38.759 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:38 smithi001 ceph-mon[112940]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:37:38.759 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:38 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2980794011' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:37:40.269 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:39 smithi001 systemd[1]: Stopping Ceph osd.0 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:37:40.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:40 smithi066 ceph-mon[85781]: pgmap v62: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:40.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:40 smithi001 ceph-mon[112940]: pgmap v62: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:40.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:40 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[47317]: 2024-02-28T16:37:40.401+0000 7f8b09000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:37:40.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:40 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[47317]: 2024-02-28T16:37:40.401+0000 7f8b09000700 -1 osd.0 48 *** Got signal Terminated *** 2024-02-28T16:37:40.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:40 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[47317]: 2024-02-28T16:37:40.401+0000 7f8b09000700 -1 osd.0 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T16:37:41.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:41 smithi066 ceph-mon[85781]: osd.0 marked itself down and dead 2024-02-28T16:37:41.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:37:41.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:41 smithi001 ceph-mon[112940]: osd.0 marked itself down and dead 2024-02-28T16:37:41.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:37:42.199 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:41 smithi001 podman[124145]: 2024-02-28 16:37:41.941137654 +0000 UTC m=+1.673957106 container died 77197415288f5eca3e947af474bdfe4333d9fa7f1c1d3c89a792bf1be3f616fc (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c0f02e6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:37:42.465 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:42 smithi001 ceph-mon[112940]: pgmap v63: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:42.465 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:42 smithi001 ceph-mon[112940]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:37:42.465 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:42 smithi001 ceph-mon[112940]: osdmap e49: 6 total, 5 up, 6 in 2024-02-28T16:37:42.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:42 smithi066 ceph-mon[85781]: pgmap v63: 65 pgs: 65 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:42.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:42 smithi066 ceph-mon[85781]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:37:42.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:42 smithi066 ceph-mon[85781]: osdmap e49: 6 total, 5 up, 6 in 2024-02-28T16:37:42.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:42 smithi001 podman[124145]: 2024-02-28 16:37:42.477358806 +0000 UTC m=+2.210178295 container cleanup 77197415288f5eca3e947af474bdfe4333d9fa7f1c1d3c89a792bf1be3f616fc (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-c0f02e6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-02-28T16:37:42.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:42 smithi001 bash[124145]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0 2024-02-28T16:37:42.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:42 smithi001 podman[124165]: 2024-02-28 16:37:42.662133342 +0000 UTC m=+0.711371090 container remove 77197415288f5eca3e947af474bdfe4333d9fa7f1c1d3c89a792bf1be3f616fc (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-c0f02e6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-28T16:37:43.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:43 smithi066 ceph-mon[85781]: osdmap e50: 6 total, 5 up, 6 in 2024-02-28T16:37:43.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:43 smithi001 ceph-mon[112940]: osdmap e50: 6 total, 5 up, 6 in 2024-02-28T16:37:43.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:43 smithi001 podman[124254]: 2024-02-28 16:37:43.410469863 +0000 UTC m=+0.033472198 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:37:43.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:43 smithi001 podman[124254]: 2024-02-28 16:37:43.5048998 +0000 UTC m=+0.127902204 container create aeec44f6c63f535623a273d8e8630f0dfc8e3ded2bc5042793f7783082fb808c (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-28T16:37:44.269 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:44 smithi001 podman[124254]: 2024-02-28 16:37:44.00938541 +0000 UTC m=+0.632387787 container init aeec44f6c63f535623a273d8e8630f0dfc8e3ded2bc5042793f7783082fb808c (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:37:44.269 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:44 smithi001 podman[124254]: 2024-02-28 16:37:44.020982086 +0000 UTC m=+0.643984486 container start aeec44f6c63f535623a273d8e8630f0dfc8e3ded2bc5042793f7783082fb808c (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:37:44.269 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:44 smithi001 podman[124254]: 2024-02-28 16:37:44.116238488 +0000 UTC m=+0.739240885 container attach aeec44f6c63f535623a273d8e8630f0dfc8e3ded2bc5042793f7783082fb808c (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, RELEASE=main-1fb5d10, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-28T16:37:44.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:44 smithi066 ceph-mon[85781]: pgmap v66: 65 pgs: 34 peering, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:44.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:44 smithi066 ceph-mon[85781]: Health check failed: Reduced data availability: 32 pgs peering (PG_AVAILABILITY) 2024-02-28T16:37:44.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:44 smithi001 ceph-mon[112940]: pgmap v66: 65 pgs: 34 peering, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:44.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:44 smithi001 ceph-mon[112940]: Health check failed: Reduced data availability: 32 pgs peering (PG_AVAILABILITY) 2024-02-28T16:37:44.770 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:44 smithi001 podman[124254]: 2024-02-28 16:37:44.471695936 +0000 UTC m=+1.094698303 container died aeec44f6c63f535623a273d8e8630f0dfc8e3ded2bc5042793f7783082fb808c (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-28T16:37:45.524 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:45 smithi001 podman[124279]: 2024-02-28 16:37:45.169803071 +0000 UTC m=+0.676158222 container remove aeec44f6c63f535623a273d8e8630f0dfc8e3ded2bc5042793f7783082fb808c (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-02-28T16:37:45.525 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:45 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-28T16:37:45.525 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:45 smithi001 systemd[1]: Stopped Ceph osd.0 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:37:45.525 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:45 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.0.service: Consumed 1min 39.079s CPU time. 2024-02-28T16:37:45.994 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:45 smithi001 systemd[1]: Starting Ceph osd.0 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:37:46.269 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:46 smithi001 podman[124380]: 2024-02-28 16:37:45.913221842 +0000 UTC m=+0.038936084 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:37:46.269 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:46 smithi001 podman[124380]: 2024-02-28 16:37:46.045763032 +0000 UTC m=+0.171477310 container create 7143ab974345e99d1a3e10af006744ad21d66a9db1bd1d4626f162932c43ba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:37:46.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:46 smithi066 ceph-mon[85781]: pgmap v67: 65 pgs: 34 peering, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:46.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:46 smithi001 ceph-mon[112940]: pgmap v67: 65 pgs: 34 peering, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:37:46.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:46 smithi001 podman[124380]: 2024-02-28 16:37:46.324670455 +0000 UTC m=+0.450384777 container init 7143ab974345e99d1a3e10af006744ad21d66a9db1bd1d4626f162932c43ba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-28T16:37:46.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:46 smithi001 podman[124380]: 2024-02-28 16:37:46.337325655 +0000 UTC m=+0.463039904 container start 7143ab974345e99d1a3e10af006744ad21d66a9db1bd1d4626f162932c43ba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:37:46.769 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:46 smithi001 podman[124380]: 2024-02-28 16:37:46.415103931 +0000 UTC m=+0.540818216 container attach 7143ab974345e99d1a3e10af006744ad21d66a9db1bd1d4626f162932c43ba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=main-1fb5d10) 2024-02-28T16:37:47.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-mon[112940]: Health check failed: Degraded data redundancy: 74/372 objects degraded (19.892%), 19 pgs degraded (PG_DEGRADED) 2024-02-28T16:37:47.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-mon[112940]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 32 pgs peering) 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate[124395]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate[124395]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 bash[124380]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 bash[124380]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 bash[124380]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 bash[124380]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 bash[124380]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 bash[124380]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-28T16:37:47.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 bash[124380]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-28T16:37:47.521 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate[124395]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-28T16:37:47.521 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate[124395]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-28T16:37:47.521 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate[124395]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-28T16:37:47.521 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate[124395]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-28T16:37:47.521 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate[124395]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-28T16:37:47.521 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:47 smithi001 podman[124380]: 2024-02-28 16:37:47.303994136 +0000 UTC m=+1.429708421 container died 7143ab974345e99d1a3e10af006744ad21d66a9db1bd1d4626f162932c43ba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:37:47.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:47 smithi066 ceph-mon[85781]: Health check failed: Degraded data redundancy: 74/372 objects degraded (19.892%), 19 pgs degraded (PG_DEGRADED) 2024-02-28T16:37:47.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:47 smithi066 ceph-mon[85781]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 32 pgs peering) 2024-02-28T16:37:48.422 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:48 smithi001 podman[124529]: 2024-02-28 16:37:48.110949663 +0000 UTC m=+0.785261892 container remove 7143ab974345e99d1a3e10af006744ad21d66a9db1bd1d4626f162932c43ba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10) 2024-02-28T16:37:48.689 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:48 smithi001 ceph-mon[112940]: pgmap v68: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:48.690 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:48 smithi001 podman[124590]: 2024-02-28 16:37:48.370556818 +0000 UTC m=+0.037010486 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:37:48.690 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:48 smithi001 podman[124590]: 2024-02-28 16:37:48.545621035 +0000 UTC m=+0.212074732 container create 7cea5750b27a0a8d645276d26dbd51de3df9e4f3adce54760fa5d5553508eace (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-02-28T16:37:48.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:48 smithi066 ceph-mon[85781]: pgmap v68: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:49.046 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:48 smithi001 podman[124590]: 2024-02-28 16:37:48.799325996 +0000 UTC m=+0.465779729 container init 7cea5750b27a0a8d645276d26dbd51de3df9e4f3adce54760fa5d5553508eace (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10) 2024-02-28T16:37:49.046 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:48 smithi001 podman[124590]: 2024-02-28 16:37:48.811031059 +0000 UTC m=+0.477484722 container start 7cea5750b27a0a8d645276d26dbd51de3df9e4f3adce54760fa5d5553508eace (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-28T16:37:49.046 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:48 smithi001 ceph-osd[124609]: -- 172.21.15.1:0/3441925074 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5643199bd680 con 0x564319987000 2024-02-28T16:37:49.046 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:48 smithi001 bash[124590]: 7cea5750b27a0a8d645276d26dbd51de3df9e4f3adce54760fa5d5553508eace 2024-02-28T16:37:49.046 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:48 smithi001 systemd[1]: Started Ceph osd.0 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:37:49.721 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:49 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[124605]: 2024-02-28T16:37:49.380+0000 7fa4cae497c0 -1 Falling back to public interface 2024-02-28T16:37:50.346 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:50 smithi001 ceph-mon[112940]: pgmap v69: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:50.346 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:50 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:50.346 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:50 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:50.346 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:50 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:37:50.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:50 smithi066 ceph-mon[85781]: pgmap v69: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:50.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:50 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:50.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:50 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:50.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:50 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:37:52.353 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:52 smithi001 ceph-mon[112940]: pgmap v70: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:52.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:52 smithi066 ceph-mon[85781]: pgmap v70: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:54.269 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:53 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[124605]: 2024-02-28T16:37:53.985+0000 7fa4cae497c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-28T16:37:54.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:54 smithi001 ceph-mon[112940]: pgmap v71: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:54.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:54 smithi066 ceph-mon[85781]: pgmap v71: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:54.770 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:54 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[124605]: 2024-02-28T16:37:54.384+0000 7fa4cae497c0 -1 osd.0 48 log_to_monitors true 2024-02-28T16:37:55.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:55 smithi066 ceph-mon[85781]: from='osd.0 [v2:172.21.15.1:6802/524238602,v1:172.21.15.1:6803/524238602]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-28T16:37:55.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:55 smithi001 ceph-mon[112940]: from='osd.0 [v2:172.21.15.1:6802/524238602,v1:172.21.15.1:6803/524238602]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-28T16:37:55.520 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 16:37:55 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[124605]: 2024-02-28T16:37:55.098+0000 7fa4ba200700 -1 osd.0 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-28T16:37:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:56 smithi066 ceph-mon[85781]: pgmap v72: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:56 smithi066 ceph-mon[85781]: from='osd.0 [v2:172.21.15.1:6802/524238602,v1:172.21.15.1:6803/524238602]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-28T16:37:56.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:56 smithi066 ceph-mon[85781]: osdmap e51: 6 total, 5 up, 6 in 2024-02-28T16:37:56.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:56 smithi066 ceph-mon[85781]: from='osd.0 [v2:172.21.15.1:6802/524238602,v1:172.21.15.1:6803/524238602]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:37:56.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:56.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:37:56.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:56.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:56 smithi001 ceph-mon[112940]: pgmap v72: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 324 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:37:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:56 smithi001 ceph-mon[112940]: from='osd.0 [v2:172.21.15.1:6802/524238602,v1:172.21.15.1:6803/524238602]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-28T16:37:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:56 smithi001 ceph-mon[112940]: osdmap e51: 6 total, 5 up, 6 in 2024-02-28T16:37:56.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:56 smithi001 ceph-mon[112940]: from='osd.0 [v2:172.21.15.1:6802/524238602,v1:172.21.15.1:6803/524238602]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:37:56.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:56.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:37:56.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:56.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:57.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:57 smithi066 ceph-mon[85781]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:37:57.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:57 smithi066 ceph-mon[85781]: osd.0 [v2:172.21.15.1:6802/524238602,v1:172.21.15.1:6803/524238602] boot 2024-02-28T16:37:57.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:57 smithi066 ceph-mon[85781]: osdmap e52: 6 total, 6 up, 6 in 2024-02-28T16:37:57.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:57 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:37:57.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:57 smithi066 ceph-mon[85781]: osdmap e53: 6 total, 6 up, 6 in 2024-02-28T16:37:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:57 smithi001 ceph-mon[112940]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:37:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:57 smithi001 ceph-mon[112940]: osd.0 [v2:172.21.15.1:6802/524238602,v1:172.21.15.1:6803/524238602] boot 2024-02-28T16:37:57.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:57 smithi001 ceph-mon[112940]: osdmap e52: 6 total, 6 up, 6 in 2024-02-28T16:37:57.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-28T16:37:57.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:57 smithi001 ceph-mon[112940]: osdmap e53: 6 total, 6 up, 6 in 2024-02-28T16:37:58.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:58 smithi066 ceph-mon[85781]: pgmap v75: 65 pgs: 25 peering, 3 active+undersized+degraded, 6 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 14/372 objects degraded (3.763%) 2024-02-28T16:37:58.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:58 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 14/372 objects degraded (3.763%), 3 pgs degraded (PG_DEGRADED) 2024-02-28T16:37:58.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:58 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:58.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:37:58 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:58.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:58 smithi001 ceph-mon[112940]: pgmap v75: 65 pgs: 25 peering, 3 active+undersized+degraded, 6 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 14/372 objects degraded (3.763%) 2024-02-28T16:37:58.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:58 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 14/372 objects degraded (3.763%), 3 pgs degraded (PG_DEGRADED) 2024-02-28T16:37:58.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:58 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:37:58.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:37:58 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:00.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:00 smithi001 ceph-mon[112940]: pgmap v77: 65 pgs: 25 peering, 3 active+undersized+degraded, 6 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 14/372 objects degraded (3.763%) 2024-02-28T16:38:00.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:00 smithi066 ceph-mon[85781]: pgmap v77: 65 pgs: 25 peering, 3 active+undersized+degraded, 6 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 14/372 objects degraded (3.763%) 2024-02-28T16:38:01.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:01 smithi066 ceph-mon[85781]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/372 objects degraded (3.763%), 3 pgs degraded) 2024-02-28T16:38:01.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:01 smithi001 ceph-mon[112940]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/372 objects degraded (3.763%), 3 pgs degraded) 2024-02-28T16:38:02.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:02 smithi066 ceph-mon[85781]: pgmap v78: 65 pgs: 25 peering, 40 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:02.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:02 smithi001 ceph-mon[112940]: pgmap v78: 65 pgs: 25 peering, 40 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:04.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: pgmap v79: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:04 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: pgmap v79: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:04.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:04 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-28T16:38:05.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:05 smithi001 ceph-mon[112940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-28T16:38:05.264 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:05 smithi001 ceph-mon[112940]: Upgrade: osd.1 is safe to restart 2024-02-28T16:38:05.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:05 smithi066 ceph-mon[85781]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-28T16:38:05.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:05 smithi066 ceph-mon[85781]: Upgrade: osd.1 is safe to restart 2024-02-28T16:38:06.257 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:06 smithi001 ceph-mon[112940]: pgmap v80: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:06.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:06 smithi066 ceph-mon[85781]: pgmap v80: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:07.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:07 smithi001 ceph-mon[112940]: Upgrade: Updating osd.1 2024-02-28T16:38:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:07 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:07 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-28T16:38:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:07 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:38:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:07 smithi001 ceph-mon[112940]: Deploying daemon osd.1 on smithi001 2024-02-28T16:38:07.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.717+0000 7fe2afa00700 1 -- 172.21.15.1:0/923231693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fbcb0 msgr2=0x7fe2a80fc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.717+0000 7fe2afa00700 1 --2- 172.21.15.1:0/923231693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fbcb0 0x7fe2a80fc080 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe29c004940 tx=0x7fe29c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.717+0000 7fe2afa00700 1 -- 172.21.15.1:0/923231693 shutdown_connections 2024-02-28T16:38:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.717+0000 7fe2afa00700 1 --2- 172.21.15.1:0/923231693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2a80fc5c0 0x7fe2a8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.717+0000 7fe2afa00700 1 --2- 172.21.15.1:0/923231693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fbcb0 0x7fe2a80fc080 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.717+0000 7fe2afa00700 1 -- 172.21.15.1:0/923231693 >> 172.21.15.1:0/923231693 conn(0x7fe2a80f79e0 msgr2=0x7fe2a80f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.718+0000 7fe2afa00700 1 -- 172.21.15.1:0/923231693 shutdown_connections 2024-02-28T16:38:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.718+0000 7fe2afa00700 1 -- 172.21.15.1:0/923231693 wait complete. 2024-02-28T16:38:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.719+0000 7fe2afa00700 1 Processor -- start 2024-02-28T16:38:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.719+0000 7fe2afa00700 1 -- start start 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.719+0000 7fe2afa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2a80fbcb0 0x7fe2a8115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.719+0000 7fe2afa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fc5c0 0x7fe2a8116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.719+0000 7fe2afa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a8116790 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.719+0000 7fe2afa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a810fae0 con 0x7fe2a80fbcb0 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.720+0000 7fe2a7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fc5c0 0x7fe2a8116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.720+0000 7fe2a7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fc5c0 0x7fe2a8116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55710/0 (socket says 172.21.15.1:55710) 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.720+0000 7fe2a7e00700 1 -- 172.21.15.1:0/631097733 learned_addr learned my addr 172.21.15.1:0/631097733 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.720+0000 7fe2ad000700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2a80fbcb0 0x7fe2a8115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.720+0000 7fe2a7e00700 1 -- 172.21.15.1:0/631097733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2a80fbcb0 msgr2=0x7fe2a8115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.720+0000 7fe2a7e00700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2a80fbcb0 0x7fe2a8115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.720+0000 7fe2a7e00700 1 -- 172.21.15.1:0/631097733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c005040 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.720+0000 7fe2a7e00700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fc5c0 0x7fe2a8116140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe294002a50 tx=0x7fe2940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:07.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.721+0000 7fe2a4c00700 1 -- 172.21.15.1:0/631097733 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe294007cc0 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.721+0000 7fe2a4c00700 1 -- 172.21.15.1:0/631097733 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe29400f040 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.721+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a810fdc0 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.721+0000 7fe2a4c00700 1 -- 172.21.15.1:0/631097733 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe29400db70 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.721+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a8110300 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.722+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a8106a50 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.723+0000 7fe2a4c00700 1 -- 172.21.15.1:0/631097733 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe29400dcd0 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.724+0000 7fe2a4c00700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe29006d7f0 0x7fe29006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.725+0000 7fe2a4c00700 1 -- 172.21.15.1:0/631097733 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fe29408c520 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.725+0000 7fe2ad000700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe29006d7f0 0x7fe29006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:07.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.726+0000 7fe2ad000700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe29006d7f0 0x7fe29006fca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe29c00b3b0 tx=0x7fe29c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:07.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.730+0000 7fe2a4c00700 1 -- 172.21.15.1:0/631097733 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe294055e20 con 0x7fe2a80fc5c0 2024-02-28T16:38:07.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:07 smithi066 ceph-mon[85781]: Upgrade: Updating osd.1 2024-02-28T16:38:07.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:07 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:07.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:07 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-28T16:38:07.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:07 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:38:07.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:07 smithi066 ceph-mon[85781]: Deploying daemon osd.1 on smithi001 2024-02-28T16:38:07.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.990+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2a804bed0 con 0x7fe29006d7f0 2024-02-28T16:38:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.992+0000 7fe2a4c00700 1 -- 172.21.15.1:0/631097733 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe2a804bed0 con 0x7fe29006d7f0 2024-02-28T16:38:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe29006d7f0 msgr2=0x7fe29006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe29006d7f0 0x7fe29006fca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe29c00b3b0 tx=0x7fe29c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fc5c0 msgr2=0x7fe2a8116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fc5c0 0x7fe2a8116140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe294002a50 tx=0x7fe2940046b0 comp rx=0 tx=0).stop 2024-02-28T16:38:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 shutdown_connections 2024-02-28T16:38:07.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe29006d7f0 0x7fe29006fca0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2a80fbcb0 0x7fe2a8115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 --2- 172.21.15.1:0/631097733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2a80fc5c0 0x7fe2a8116140 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.995+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 >> 172.21.15.1:0/631097733 conn(0x7fe2a80f79e0 msgr2=0x7fe2a80f9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.996+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 shutdown_connections 2024-02-28T16:38:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:07.996+0000 7fe2afa00700 1 -- 172.21.15.1:0/631097733 wait complete. 2024-02-28T16:38:08.002 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:38:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.163+0000 7fcfe6a00700 1 -- 172.21.15.1:0/2098123506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe0102f50 msgr2=0x7fcfe0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.163+0000 7fcfe6a00700 1 --2- 172.21.15.1:0/2098123506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe0102f50 0x7fcfe0103320 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0004940 tx=0x7fcfd0009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.163+0000 7fcfe6a00700 1 -- 172.21.15.1:0/2098123506 shutdown_connections 2024-02-28T16:38:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.163+0000 7fcfe6a00700 1 --2- 172.21.15.1:0/2098123506 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfe01038f0 0x7fcfe00fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.163+0000 7fcfe6a00700 1 --2- 172.21.15.1:0/2098123506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe0102f50 0x7fcfe0103320 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.163+0000 7fcfe6a00700 1 -- 172.21.15.1:0/2098123506 >> 172.21.15.1:0/2098123506 conn(0x7fcfe00f8ae0 msgr2=0x7fcfe00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.163+0000 7fcfe6a00700 1 -- 172.21.15.1:0/2098123506 shutdown_connections 2024-02-28T16:38:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.163+0000 7fcfe6a00700 1 -- 172.21.15.1:0/2098123506 wait complete. 2024-02-28T16:38:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.164+0000 7fcfe6a00700 1 Processor -- start 2024-02-28T16:38:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.165+0000 7fcfe6a00700 1 -- start start 2024-02-28T16:38:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.165+0000 7fcfe6a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfe0102f50 0x7fcfe0113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:08.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.165+0000 7fcfe6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe01038f0 0x7fcfe0113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:08.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.165+0000 7fcfe6a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe01145e0 con 0x7fcfe01038f0 2024-02-28T16:38:08.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.165+0000 7fcfe6a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe010d910 con 0x7fcfe0102f50 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.166+0000 7fcfdf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe01038f0 0x7fcfe0113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.166+0000 7fcfdf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe01038f0 0x7fcfe0113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55724/0 (socket says 172.21.15.1:55724) 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.166+0000 7fcfdf400700 1 -- 172.21.15.1:0/3101920283 learned_addr learned my addr 172.21.15.1:0/3101920283 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.166+0000 7fcfdfe00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfe0102f50 0x7fcfe0113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.166+0000 7fcfdf400700 1 -- 172.21.15.1:0/3101920283 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfe0102f50 msgr2=0x7fcfe0113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.166+0000 7fcfdf400700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfe0102f50 0x7fcfe0113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.166+0000 7fcfdf400700 1 -- 172.21.15.1:0/3101920283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd0005040 con 0x7fcfe01038f0 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.166+0000 7fcfdfe00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfe0102f50 0x7fcfe0113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:38:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.167+0000 7fcfdf400700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe01038f0 0x7fcfe0113f90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc002a50 tx=0x7fcfcc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.167+0000 7fcfd7e00700 1 -- 172.21.15.1:0/3101920283 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfcc007cc0 con 0x7fcfe01038f0 2024-02-28T16:38:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.167+0000 7fcfd7e00700 1 -- 172.21.15.1:0/3101920283 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfcc00f040 con 0x7fcfe01038f0 2024-02-28T16:38:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.167+0000 7fcfd7e00700 1 -- 172.21.15.1:0/3101920283 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfcc00daf0 con 0x7fcfe01038f0 2024-02-28T16:38:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.167+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe010dbf0 con 0x7fcfe01038f0 2024-02-28T16:38:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.167+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe010e130 con 0x7fcfe01038f0 2024-02-28T16:38:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.168+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe00fc6e0 con 0x7fcfe01038f0 2024-02-28T16:38:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.170+0000 7fcfd7e00700 1 -- 172.21.15.1:0/3101920283 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcfcc009810 con 0x7fcfe01038f0 2024-02-28T16:38:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.171+0000 7fcfd7e00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcfc406d7e0 0x7fcfc406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.171+0000 7fcfd7e00700 1 -- 172.21.15.1:0/3101920283 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fcfcc08cc00 con 0x7fcfe01038f0 2024-02-28T16:38:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.172+0000 7fcfdfe00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcfc406d7e0 0x7fcfc406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:08.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.173+0000 7fcfdfe00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcfc406d7e0 0x7fcfc406fc90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0004f90 tx=0x7fcfd0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.176+0000 7fcfd7e00700 1 -- 172.21.15.1:0/3101920283 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcfcc059fd0 con 0x7fcfe01038f0 2024-02-28T16:38:08.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.436+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfe010e350 con 0x7fcfc406d7e0 2024-02-28T16:38:08.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.439+0000 7fcfd7e00700 1 -- 172.21.15.1:0/3101920283 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fcfe010e350 con 0x7fcfc406d7e0 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.441+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcfc406d7e0 msgr2=0x7fcfc406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.441+0000 7fcfe6a00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcfc406d7e0 0x7fcfc406fc90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0004f90 tx=0x7fcfd0009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.441+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe01038f0 msgr2=0x7fcfe0113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.441+0000 7fcfe6a00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe01038f0 0x7fcfe0113f90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc002a50 tx=0x7fcfcc004600 comp rx=0 tx=0).stop 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.442+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 shutdown_connections 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.442+0000 7fcfe6a00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcfc406d7e0 0x7fcfc406fc90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.442+0000 7fcfe6a00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfe0102f50 0x7fcfe0113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.442+0000 7fcfe6a00700 1 --2- 172.21.15.1:0/3101920283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfe01038f0 0x7fcfe0113f90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.442+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 >> 172.21.15.1:0/3101920283 conn(0x7fcfe00f8ae0 msgr2=0x7fcfe00fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.442+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 shutdown_connections 2024-02-28T16:38:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.442+0000 7fcfe6a00700 1 -- 172.21.15.1:0/3101920283 wait complete. 2024-02-28T16:38:08.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:08 smithi001 ceph-mon[112940]: pgmap v81: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.592+0000 7fea41000700 1 -- 172.21.15.1:0/1059026910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 msgr2=0x7fea3c107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.592+0000 7fea41000700 1 --2- 172.21.15.1:0/1059026910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 0x7fea3c107050 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fea28004970 tx=0x7fea28009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.592+0000 7fea41000700 1 -- 172.21.15.1:0/1059026910 shutdown_connections 2024-02-28T16:38:08.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.592+0000 7fea41000700 1 --2- 172.21.15.1:0/1059026910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 0x7fea3c107050 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.592+0000 7fea41000700 1 --2- 172.21.15.1:0/1059026910 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea3c0fdfe0 0x7fea3c0fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.592+0000 7fea41000700 1 -- 172.21.15.1:0/1059026910 >> 172.21.15.1:0/1059026910 conn(0x7fea3c071d90 msgr2=0x7fea3c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:08.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.592+0000 7fea41000700 1 -- 172.21.15.1:0/1059026910 shutdown_connections 2024-02-28T16:38:08.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.592+0000 7fea41000700 1 -- 172.21.15.1:0/1059026910 wait complete. 2024-02-28T16:38:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.593+0000 7fea41000700 1 Processor -- start 2024-02-28T16:38:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.593+0000 7fea41000700 1 -- start start 2024-02-28T16:38:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.594+0000 7fea41000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea3c0fdfe0 0x7fea3c115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.594+0000 7fea41000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 0x7fea3c116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.594+0000 7fea41000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c116780 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.594+0000 7fea41000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c10fab0 con 0x7fea3c0fdfe0 2024-02-28T16:38:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.594+0000 7fea39400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 0x7fea3c116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.594+0000 7fea39400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 0x7fea3c116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55738/0 (socket says 172.21.15.1:55738) 2024-02-28T16:38:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.594+0000 7fea39400700 1 -- 172.21.15.1:0/450285724 learned_addr learned my addr 172.21.15.1:0/450285724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.595+0000 7fea39e00700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea3c0fdfe0 0x7fea3c115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.595+0000 7fea39400700 1 -- 172.21.15.1:0/450285724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea3c0fdfe0 msgr2=0x7fea3c115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.595+0000 7fea39400700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea3c0fdfe0 0x7fea3c115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.595+0000 7fea39400700 1 -- 172.21.15.1:0/450285724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea28005040 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.595+0000 7fea39400700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 0x7fea3c116130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fea28000c00 tx=0x7fea28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.595+0000 7fea32000700 1 -- 172.21.15.1:0/450285724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea2800e040 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.595+0000 7fea32000700 1 -- 172.21.15.1:0/450285724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea28004290 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.595+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c10fd30 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.596+0000 7fea32000700 1 -- 172.21.15.1:0/450285724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea2801c630 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.596+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3c110270 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.598+0000 7fea32000700 1 -- 172.21.15.1:0/450285724 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fea28013980 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.598+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3c106ac0 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.599+0000 7fea32000700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2006d7e0 0x7fea2006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.599+0000 7fea32000700 1 -- 172.21.15.1:0/450285724 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fea2808e060 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.600+0000 7fea39e00700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2006d7e0 0x7fea2006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.600+0000 7fea39e00700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2006d7e0 0x7fea2006fc90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fea2c004990 tx=0x7fea2c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:08.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.608+0000 7fea32000700 1 -- 172.21.15.1:0/450285724 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fea2805b5b0 con 0x7fea3c0fe8f0 2024-02-28T16:38:08.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:08 smithi066 ceph-mon[85781]: pgmap v81: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:08.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.841+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fea3c10a030 con 0x7fea2006d7e0 2024-02-28T16:38:08.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.858+0000 7fea32000700 1 -- 172.21.15.1:0/450285724 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fea3c10a030 con 0x7fea2006d7e0 2024-02-28T16:38:08.847 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:38:08.847 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (4m) 12s ago 18m 14.9M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:38:08.847 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (60s) 12s ago 18m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:38:08.847 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (51s) 46s ago 17m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:38:08.847 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (3m) 12s ago 18m 74.8M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:38:08.847 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (11m) 12s ago 11m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (11m) 12s ago 11m 14.7M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (11m) 46s ago 11m 14.3M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (11m) 46s ago 11m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (6m) 12s ago 20m 498M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (5m) 46s ago 17m 449M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (2m) 12s ago 20m 44.6M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (2m) 46s ago 17m 35.0M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (5m) 12s ago 18m 7688k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (5m) 46s ago 17m 7612k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (20s) 12s ago 16m 11.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (15m) 12s ago 15m 208M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 457d6117edea 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (15m) 12s ago 15m 169M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (14m) 46s ago 14m 222M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:38:08.848 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (13m) 46s ago 13m 214M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:38:08.849 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (13m) 46s ago 13m 189M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:38:08.849 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (4m) 12s ago 17m 51.6M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:38:08.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.860+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2006d7e0 msgr2=0x7fea2006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.860+0000 7fea41000700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2006d7e0 0x7fea2006fc90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fea2c004990 tx=0x7fea2c009ee0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.860+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 msgr2=0x7fea3c116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.860+0000 7fea41000700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 0x7fea3c116130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fea28000c00 tx=0x7fea28009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.861+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 shutdown_connections 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.861+0000 7fea41000700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2006d7e0 0x7fea2006fc90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.861+0000 7fea41000700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea3c0fdfe0 0x7fea3c115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.861+0000 7fea41000700 1 --2- 172.21.15.1:0/450285724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3c0fe8f0 0x7fea3c116130 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.861+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 >> 172.21.15.1:0/450285724 conn(0x7fea3c071d90 msgr2=0x7fea3c103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.861+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 shutdown_connections 2024-02-28T16:38:08.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:08.861+0000 7fea41000700 1 -- 172.21.15.1:0/450285724 wait complete. 2024-02-28T16:38:08.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.005+0000 7fc766c00700 1 -- 172.21.15.1:0/3260518039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fe8f0 msgr2=0x7fc760107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.005+0000 7fc766c00700 1 --2- 172.21.15.1:0/3260518039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fe8f0 0x7fc760107050 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc74c004970 tx=0x7fc74c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:08.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.005+0000 7fc766c00700 1 -- 172.21.15.1:0/3260518039 shutdown_connections 2024-02-28T16:38:08.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.005+0000 7fc766c00700 1 --2- 172.21.15.1:0/3260518039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fe8f0 0x7fc760107050 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.005+0000 7fc766c00700 1 --2- 172.21.15.1:0/3260518039 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7600fdfe0 0x7fc7600fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.005+0000 7fc766c00700 1 -- 172.21.15.1:0/3260518039 >> 172.21.15.1:0/3260518039 conn(0x7fc760071d90 msgr2=0x7fc760072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.006+0000 7fc766c00700 1 -- 172.21.15.1:0/3260518039 shutdown_connections 2024-02-28T16:38:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.006+0000 7fc766c00700 1 -- 172.21.15.1:0/3260518039 wait complete. 2024-02-28T16:38:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.006+0000 7fc766c00700 1 Processor -- start 2024-02-28T16:38:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.007+0000 7fc766c00700 1 -- start start 2024-02-28T16:38:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.007+0000 7fc766c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fdfe0 0x7fc760115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.007+0000 7fc766c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7600fe8f0 0x7fc760116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.007+0000 7fc766c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760116790 con 0x7fc7600fdfe0 2024-02-28T16:38:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.007+0000 7fc766c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76010fac0 con 0x7fc7600fe8f0 2024-02-28T16:38:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.007+0000 7fc75fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fdfe0 0x7fc760115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.008+0000 7fc75fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fdfe0 0x7fc760115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55762/0 (socket says 172.21.15.1:55762) 2024-02-28T16:38:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.008+0000 7fc75fe00700 1 -- 172.21.15.1:0/2373053074 learned_addr learned my addr 172.21.15.1:0/2373053074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.008+0000 7fc75f400700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7600fe8f0 0x7fc760116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.008+0000 7fc75fe00700 1 -- 172.21.15.1:0/2373053074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7600fe8f0 msgr2=0x7fc760116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.008+0000 7fc75fe00700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7600fe8f0 0x7fc760116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.008+0000 7fc75fe00700 1 -- 172.21.15.1:0/2373053074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c005040 con 0x7fc7600fdfe0 2024-02-28T16:38:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.008+0000 7fc75fe00700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fdfe0 0x7fc760115c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc750002a50 tx=0x7fc7500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:08.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.009+0000 7fc757e00700 1 -- 172.21.15.1:0/2373053074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc750007cc0 con 0x7fc7600fdfe0 2024-02-28T16:38:08.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.009+0000 7fc757e00700 1 -- 172.21.15.1:0/2373053074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc75000f040 con 0x7fc7600fdfe0 2024-02-28T16:38:08.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.009+0000 7fc757e00700 1 -- 172.21.15.1:0/2373053074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc75000da30 con 0x7fc7600fdfe0 2024-02-28T16:38:08.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.009+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76010fda0 con 0x7fc7600fdfe0 2024-02-28T16:38:08.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.009+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7601102b0 con 0x7fc7600fdfe0 2024-02-28T16:38:08.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.010+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc760106b20 con 0x7fc7600fdfe0 2024-02-28T16:38:09.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.011+0000 7fc757e00700 1 -- 172.21.15.1:0/2373053074 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc75000dbb0 con 0x7fc7600fdfe0 2024-02-28T16:38:09.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.012+0000 7fc757e00700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc74406d7f0 0x7fc74406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:09.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.012+0000 7fc757e00700 1 -- 172.21.15.1:0/2373053074 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fc75008cb60 con 0x7fc7600fdfe0 2024-02-28T16:38:09.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.013+0000 7fc75f400700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc74406d7f0 0x7fc74406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:09.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.014+0000 7fc75f400700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc74406d7f0 0x7fc74406fca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc74c004940 tx=0x7fc74c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:09.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.017+0000 7fc757e00700 1 -- 172.21.15.1:0/2373053074 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc750059f70 con 0x7fc7600fdfe0 2024-02-28T16:38:09.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.351+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc76004bf20 con 0x7fc7600fdfe0 2024-02-28T16:38:09.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.352+0000 7fc757e00700 1 -- 172.21.15.1:0/2373053074 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7fc750059b00 con 0x7fc7600fdfe0 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 5, 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:38:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:38:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:38:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:38:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 9, 2024-02-28T16:38:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 5 2024-02-28T16:38:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:38:09.342 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:38:09.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc74406d7f0 msgr2=0x7fc74406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:09.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc74406d7f0 0x7fc74406fca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc74c004940 tx=0x7fc74c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:09.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fdfe0 msgr2=0x7fc760115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:09.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fdfe0 0x7fc760115c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc750002a50 tx=0x7fc7500046b0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 shutdown_connections 2024-02-28T16:38:09.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc74406d7f0 0x7fc74406fca0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7600fdfe0 0x7fc760115c00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 --2- 172.21.15.1:0/2373053074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7600fe8f0 0x7fc760116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 >> 172.21.15.1:0/2373053074 conn(0x7fc760071d90 msgr2=0x7fc7600fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 shutdown_connections 2024-02-28T16:38:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.355+0000 7fc766c00700 1 -- 172.21.15.1:0/2373053074 wait complete. 2024-02-28T16:38:09.486 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:09 smithi001 ceph-mon[112940]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:09.486 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:09 smithi001 ceph-mon[112940]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.507+0000 7fc4db600700 1 -- 172.21.15.1:0/296091065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 msgr2=0x7fc4d4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.507+0000 7fc4db600700 1 --2- 172.21.15.1:0/296091065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 0x7fc4d4103320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8004940 tx=0x7fc4c8009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.507+0000 7fc4db600700 1 -- 172.21.15.1:0/296091065 shutdown_connections 2024-02-28T16:38:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.507+0000 7fc4db600700 1 --2- 172.21.15.1:0/296091065 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4d41038f0 0x7fc4d40fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.507+0000 7fc4db600700 1 --2- 172.21.15.1:0/296091065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 0x7fc4d4103320 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.507+0000 7fc4db600700 1 -- 172.21.15.1:0/296091065 >> 172.21.15.1:0/296091065 conn(0x7fc4d40f8ae0 msgr2=0x7fc4d40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.507+0000 7fc4db600700 1 -- 172.21.15.1:0/296091065 shutdown_connections 2024-02-28T16:38:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.507+0000 7fc4db600700 1 -- 172.21.15.1:0/296091065 wait complete. 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.508+0000 7fc4db600700 1 Processor -- start 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.508+0000 7fc4db600700 1 -- start start 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.508+0000 7fc4db600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 0x7fc4d4070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.508+0000 7fc4db600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4d41038f0 0x7fc4d40707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.508+0000 7fc4db600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d4070dd0 con 0x7fc4d4102f50 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.508+0000 7fc4db600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d4070f40 con 0x7fc4d41038f0 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.509+0000 7fc4d8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 0x7fc4d4070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.509+0000 7fc4d8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 0x7fc4d4070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55776/0 (socket says 172.21.15.1:55776) 2024-02-28T16:38:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.509+0000 7fc4d8c00700 1 -- 172.21.15.1:0/74654257 learned_addr learned my addr 172.21.15.1:0/74654257 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.509+0000 7fc4d3e00700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4d41038f0 0x7fc4d40707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.509+0000 7fc4d8c00700 1 -- 172.21.15.1:0/74654257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4d41038f0 msgr2=0x7fc4d40707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.509+0000 7fc4d8c00700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4d41038f0 0x7fc4d40707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.509+0000 7fc4d8c00700 1 -- 172.21.15.1:0/74654257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c8005040 con 0x7fc4d4102f50 2024-02-28T16:38:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.509+0000 7fc4d3e00700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4d41038f0 0x7fc4d40707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:38:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.510+0000 7fc4d8c00700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 0x7fc4d4070290 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009100 tx=0x7fc4c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.510+0000 7fc4d0c00700 1 -- 172.21.15.1:0/74654257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c800e040 con 0x7fc4d4102f50 2024-02-28T16:38:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.510+0000 7fc4d0c00700 1 -- 172.21.15.1:0/74654257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4c8004290 con 0x7fc4d4102f50 2024-02-28T16:38:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.510+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d4119c90 con 0x7fc4d4102f50 2024-02-28T16:38:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.510+0000 7fc4d0c00700 1 -- 172.21.15.1:0/74654257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c801c630 con 0x7fc4d4102f50 2024-02-28T16:38:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.510+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d411a0f0 con 0x7fc4d4102f50 2024-02-28T16:38:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.511+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d40fc6e0 con 0x7fc4d4102f50 2024-02-28T16:38:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.513+0000 7fc4d0c00700 1 -- 172.21.15.1:0/74654257 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc4c800bbf0 con 0x7fc4d4102f50 2024-02-28T16:38:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.513+0000 7fc4d0c00700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc4bc06d7e0 0x7fc4bc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.514+0000 7fc4d0c00700 1 -- 172.21.15.1:0/74654257 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fc4c808e190 con 0x7fc4d4102f50 2024-02-28T16:38:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.514+0000 7fc4d3e00700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc4bc06d7e0 0x7fc4bc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:09.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.515+0000 7fc4d3e00700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc4bc06d7e0 0x7fc4bc06fc90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4071c90 tx=0x7fc4c0009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.518+0000 7fc4d0c00700 1 -- 172.21.15.1:0/74654257 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc4c8057b10 con 0x7fc4d4102f50 2024-02-28T16:38:09.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:09 smithi066 ceph-mon[85781]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:09.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:09 smithi066 ceph-mon[85781]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.788+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc4d411a330 con 0x7fc4d4102f50 2024-02-28T16:38:09.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.789+0000 7fc4d0c00700 1 -- 172.21.15.1:0/74654257 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7fc4c805b1d0 con 0x7fc4d4102f50 2024-02-28T16:38:09.778 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:38:09.778 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:38:09.778 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:38:09.778 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:38:09.778 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:38:09.778 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:38:09.778 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:38:09.778 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:38:09.779 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:38:09.780 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:38:09.781 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:38:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:38:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:38:09.781 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:38:09.781 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:38:09.781 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:38:09.781 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:38:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc4bc06d7e0 msgr2=0x7fc4bc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc4bc06d7e0 0x7fc4bc06fc90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4071c90 tx=0x7fc4c0009040 comp rx=0 tx=0).stop 2024-02-28T16:38:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 msgr2=0x7fc4d4070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 0x7fc4d4070290 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009100 tx=0x7fc4c8009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 shutdown_connections 2024-02-28T16:38:09.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc4bc06d7e0 0x7fc4bc06fc90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d4102f50 0x7fc4d4070290 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 --2- 172.21.15.1:0/74654257 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4d41038f0 0x7fc4d40707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.791+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 >> 172.21.15.1:0/74654257 conn(0x7fc4d40f8ae0 msgr2=0x7fc4d40fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:09.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.792+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 shutdown_connections 2024-02-28T16:38:09.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.792+0000 7fc4db600700 1 -- 172.21.15.1:0/74654257 wait complete. 2024-02-28T16:38:09.783 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.935+0000 7f1deea00700 1 -- 172.21.15.1:0/3694524832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 msgr2=0x7f1de80ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.935+0000 7f1deea00700 1 --2- 172.21.15.1:0/3694524832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 0x7f1de80ff180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0004940 tx=0x7f1dd0009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.936+0000 7f1deea00700 1 -- 172.21.15.1:0/3694524832 shutdown_connections 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.936+0000 7f1deea00700 1 --2- 172.21.15.1:0/3694524832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 0x7f1de80ff180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.936+0000 7f1deea00700 1 --2- 172.21.15.1:0/3694524832 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1de8105170 0x7f1de8105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.936+0000 7f1deea00700 1 -- 172.21.15.1:0/3694524832 >> 172.21.15.1:0/3694524832 conn(0x7f1de80face0 msgr2=0x7f1de80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.936+0000 7f1deea00700 1 -- 172.21.15.1:0/3694524832 shutdown_connections 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.936+0000 7f1deea00700 1 -- 172.21.15.1:0/3694524832 wait complete. 2024-02-28T16:38:09.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.936+0000 7f1deea00700 1 Processor -- start 2024-02-28T16:38:09.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.937+0000 7f1deea00700 1 -- start start 2024-02-28T16:38:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.937+0000 7f1deea00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1de8105170 0x7f1de8115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.937+0000 7f1deea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 0x7f1de8116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.937+0000 7f1deea00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8116870 con 0x7f1de8105b10 2024-02-28T16:38:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.937+0000 7f1deea00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de810fb80 con 0x7f1de8105170 2024-02-28T16:38:09.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.937+0000 7f1ddfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 0x7f1de8116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.937+0000 7f1ddfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 0x7f1de8116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55788/0 (socket says 172.21.15.1:55788) 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.937+0000 7f1ddfe00700 1 -- 172.21.15.1:0/3236857170 learned_addr learned my addr 172.21.15.1:0/3236857170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1de7e00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1de8105170 0x7f1de8115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1ddfe00700 1 -- 172.21.15.1:0/3236857170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1de8105170 msgr2=0x7f1de8115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1ddfe00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1de8105170 0x7f1de8115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1ddfe00700 1 -- 172.21.15.1:0/3236857170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd0005040 con 0x7f1de8105b10 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1de7e00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1de8105170 0x7f1de8115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1ddfe00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 0x7f1de8116220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009100 tx=0x7f1dd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1de4c00700 1 -- 172.21.15.1:0/3236857170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dd000e040 con 0x7f1de8105b10 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1de4c00700 1 -- 172.21.15.1:0/3236857170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1dd0004290 con 0x7f1de8105b10 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1de4c00700 1 -- 172.21.15.1:0/3236857170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dd001c3f0 con 0x7f1de8105b10 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de810fe00 con 0x7f1de8105b10 2024-02-28T16:38:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.938+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de8110340 con 0x7f1de8105b10 2024-02-28T16:38:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.939+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8107640 con 0x7f1de8105b10 2024-02-28T16:38:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.940+0000 7f1de4c00700 1 -- 172.21.15.1:0/3236857170 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1dd000bbf0 con 0x7f1de8105b10 2024-02-28T16:38:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.942+0000 7f1de4c00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1dcc06d7f0 0x7f1dcc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.942+0000 7f1de4c00700 1 -- 172.21.15.1:0/3236857170 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f1dd008e110 con 0x7f1de8105b10 2024-02-28T16:38:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.942+0000 7f1de7e00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1dcc06d7f0 0x7f1dcc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:09.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.943+0000 7f1de7e00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1dcc06d7f0 0x7f1dcc06fca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009fd0 tx=0x7f1dd800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:09.947+0000 7f1de4c00700 1 -- 172.21.15.1:0/3236857170 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1dd005b5a0 con 0x7f1de8105b10 2024-02-28T16:38:10.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.192+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1de81112c0 con 0x7f1dcc06d7f0 2024-02-28T16:38:10.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.195+0000 7f1de4c00700 1 -- 172.21.15.1:0/3236857170 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1de81112c0 con 0x7f1dcc06d7f0 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "crash" 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "7/21 daemons upgraded", 2024-02-28T16:38:10.184 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading osd daemons", 2024-02-28T16:38:10.185 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:38:10.185 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1dcc06d7f0 msgr2=0x7f1dcc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1dcc06d7f0 0x7f1dcc06fca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009fd0 tx=0x7f1dd800b040 comp rx=0 tx=0).stop 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 msgr2=0x7f1de8116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 0x7f1de8116220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009100 tx=0x7f1dd0009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 shutdown_connections 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1dcc06d7f0 0x7f1dcc06fca0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1de8105170 0x7f1de8115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 --2- 172.21.15.1:0/3236857170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1de8105b10 0x7f1de8116220 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 >> 172.21.15.1:0/3236857170 conn(0x7f1de80face0 msgr2=0x7f1de80fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 shutdown_connections 2024-02-28T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.198+0000 7f1deea00700 1 -- 172.21.15.1:0/3236857170 wait complete. 2024-02-28T16:38:10.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.358+0000 7f13b3600700 1 -- 172.21.15.1:0/2714119274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac1038f0 msgr2=0x7f13ac0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:10.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.358+0000 7f13b3600700 1 --2- 172.21.15.1:0/2714119274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac1038f0 0x7f13ac0fcf60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1398004940 tx=0x7f1398009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:10.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.358+0000 7f13b3600700 1 -- 172.21.15.1:0/2714119274 shutdown_connections 2024-02-28T16:38:10.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.358+0000 7f13b3600700 1 --2- 172.21.15.1:0/2714119274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac1038f0 0x7f13ac0fcf60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.358+0000 7f13b3600700 1 --2- 172.21.15.1:0/2714119274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13ac102f50 0x7f13ac103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.358+0000 7f13b3600700 1 -- 172.21.15.1:0/2714119274 >> 172.21.15.1:0/2714119274 conn(0x7f13ac0f8ae0 msgr2=0x7f13ac0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:10.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.358+0000 7f13b3600700 1 -- 172.21.15.1:0/2714119274 shutdown_connections 2024-02-28T16:38:10.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.358+0000 7f13b3600700 1 -- 172.21.15.1:0/2714119274 wait complete. 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.359+0000 7f13b3600700 1 Processor -- start 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.360+0000 7f13b3600700 1 -- start start 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.360+0000 7f13b3600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac102f50 0x7f13ac115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.360+0000 7f13b3600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13ac1038f0 0x7f13ac1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.360+0000 7f13b3600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac116810 con 0x7f13ac102f50 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.360+0000 7f13b3600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac10fb40 con 0x7f13ac1038f0 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.360+0000 7f13b0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac102f50 0x7f13ac115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.360+0000 7f13b0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac102f50 0x7f13ac115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55796/0 (socket says 172.21.15.1:55796) 2024-02-28T16:38:10.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.360+0000 7f13b0c00700 1 -- 172.21.15.1:0/4284893513 learned_addr learned my addr 172.21.15.1:0/4284893513 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:10.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13abe00700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13ac1038f0 0x7f13ac1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:10.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13b0c00700 1 -- 172.21.15.1:0/4284893513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13ac1038f0 msgr2=0x7f13ac1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:10.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13b0c00700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13ac1038f0 0x7f13ac1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13b0c00700 1 -- 172.21.15.1:0/4284893513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1398005040 con 0x7f13ac102f50 2024-02-28T16:38:10.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13b0c00700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac102f50 0x7f13ac115c80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f13a000c830 tx=0x7f13a0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:10.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13a8c00700 1 -- 172.21.15.1:0/4284893513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13a000a760 con 0x7f13ac102f50 2024-02-28T16:38:10.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13a8c00700 1 -- 172.21.15.1:0/4284893513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13a000ad90 con 0x7f13ac102f50 2024-02-28T16:38:10.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13a8c00700 1 -- 172.21.15.1:0/4284893513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13a00113a0 con 0x7f13ac102f50 2024-02-28T16:38:10.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac10fe20 con 0x7f13ac102f50 2024-02-28T16:38:10.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.361+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac110360 con 0x7f13ac102f50 2024-02-28T16:38:10.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.362+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac0fc6e0 con 0x7f13ac102f50 2024-02-28T16:38:10.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.364+0000 7f13a8c00700 1 -- 172.21.15.1:0/4284893513 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f13a0012400 con 0x7f13ac102f50 2024-02-28T16:38:10.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.365+0000 7f13a8c00700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f139406d7f0 0x7f139406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:10.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.365+0000 7f13a8c00700 1 -- 172.21.15.1:0/4284893513 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f13a008ce50 con 0x7f13ac102f50 2024-02-28T16:38:10.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.366+0000 7f13abe00700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f139406d7f0 0x7f139406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:10.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.366+0000 7f13abe00700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f139406d7f0 0x7f139406fca0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f13ac1114b0 tx=0x7f1398009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:10.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.372+0000 7f13a8c00700 1 -- 172.21.15.1:0/4284893513 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f13a005a2e0 con 0x7f13ac102f50 2024-02-28T16:38:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:10 smithi001 ceph-mon[112940]: pgmap v82: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:10 smithi001 ceph-mon[112940]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:10 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2373053074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:10 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/74654257' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:38:10.649 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:10 smithi066 ceph-mon[85781]: pgmap v82: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:10.649 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:10 smithi066 ceph-mon[85781]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:10.649 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:10 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2373053074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:10.649 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:10 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/74654257' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:38:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.674+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f13ac110630 con 0x7f13ac102f50 2024-02-28T16:38:10.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.674+0000 7f13a8c00700 1 -- 172.21.15.1:0/4284893513 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f13a0059e70 con 0x7f13ac102f50 2024-02-28T16:38:10.663 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:38:10.663 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:38:10.663 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:38:10.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f139406d7f0 msgr2=0x7f139406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:10.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f139406d7f0 0x7f139406fca0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f13ac1114b0 tx=0x7f1398009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:10.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac102f50 msgr2=0x7f13ac115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:10.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac102f50 0x7f13ac115c80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f13a000c830 tx=0x7f13a0004470 comp rx=0 tx=0).stop 2024-02-28T16:38:10.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 shutdown_connections 2024-02-28T16:38:10.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f139406d7f0 0x7f139406fca0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13ac102f50 0x7f13ac115c80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 --2- 172.21.15.1:0/4284893513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13ac1038f0 0x7f13ac1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:10.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 >> 172.21.15.1:0/4284893513 conn(0x7f13ac0f8ae0 msgr2=0x7f13ac0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:10.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 shutdown_connections 2024-02-28T16:38:10.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:10.677+0000 7f13b3600700 1 -- 172.21.15.1:0/4284893513 wait complete. 2024-02-28T16:38:11.519 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:11 smithi001 systemd[1]: Stopping Ceph osd.1 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:38:11.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:11 smithi001 ceph-mon[112940]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:11.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:11 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4284893513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:38:11.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:11.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:38:11.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:11.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:11 smithi066 ceph-mon[85781]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:11.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:11 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4284893513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:38:11.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:11.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:38:11.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:12.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:11 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[54704]: 2024-02-28T16:38:11.535+0000 7f0f50800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:38:12.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:11 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[54704]: 2024-02-28T16:38:11.535+0000 7f0f50800700 -1 osd.1 53 *** Got signal Terminated *** 2024-02-28T16:38:12.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:11 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[54704]: 2024-02-28T16:38:11.535+0000 7f0f50800700 -1 osd.1 53 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T16:38:12.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:12 smithi066 ceph-mon[85781]: pgmap v83: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:12.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:12 smithi066 ceph-mon[85781]: osd.1 marked itself down and dead 2024-02-28T16:38:12.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:12 smithi001 ceph-mon[112940]: pgmap v83: 65 pgs: 65 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:12.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:12 smithi001 ceph-mon[112940]: osd.1 marked itself down and dead 2024-02-28T16:38:13.248 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:12 smithi001 podman[129663]: 2024-02-28 16:38:12.957884606 +0000 UTC m=+1.558570267 container died 457d6117edea60084d632c4d8ac0b9bea3816dac6f60f6a230fd919dc98a1a7f (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c0f02e6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) 2024-02-28T16:38:13.519 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:13 smithi001 podman[129663]: 2024-02-28 16:38:13.463447152 +0000 UTC m=+2.064132819 container cleanup 457d6117edea60084d632c4d8ac0b9bea3816dac6f60f6a230fd919dc98a1a7f (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-c0f02e6, GIT_CLEAN=True) 2024-02-28T16:38:13.519 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:13 smithi001 bash[129663]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1 2024-02-28T16:38:13.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:13 smithi001 ceph-mon[112940]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:38:13.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:13 smithi001 ceph-mon[112940]: osdmap e54: 6 total, 5 up, 6 in 2024-02-28T16:38:13.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:13 smithi001 ceph-mon[112940]: osdmap e55: 6 total, 5 up, 6 in 2024-02-28T16:38:13.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:13 smithi066 ceph-mon[85781]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:38:13.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:13 smithi066 ceph-mon[85781]: osdmap e54: 6 total, 5 up, 6 in 2024-02-28T16:38:13.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:13 smithi066 ceph-mon[85781]: osdmap e55: 6 total, 5 up, 6 in 2024-02-28T16:38:14.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:13 smithi001 podman[129683]: 2024-02-28 16:38:13.740086209 +0000 UTC m=+0.774652166 container remove 457d6117edea60084d632c4d8ac0b9bea3816dac6f60f6a230fd919dc98a1a7f (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-c0f02e6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:38:14.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:14 smithi001 ceph-mon[112940]: pgmap v85: 65 pgs: 12 stale+active+clean, 53 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:14.688 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:14 smithi001 podman[129771]: 2024-02-28 16:38:14.547397034 +0000 UTC m=+0.039374075 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:38:14.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:14 smithi066 ceph-mon[85781]: pgmap v85: 65 pgs: 12 stale+active+clean, 53 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:15.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:14 smithi001 podman[129771]: 2024-02-28 16:38:14.699867462 +0000 UTC m=+0.191844561 container create a9c4a8eeaf11142ff2767ee52563af60ca81ef40cdc711258fc78d136b8d3df8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:38:15.519 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:15 smithi001 podman[129771]: 2024-02-28 16:38:15.136745967 +0000 UTC m=+0.628723083 container init a9c4a8eeaf11142ff2767ee52563af60ca81ef40cdc711258fc78d136b8d3df8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-02-28T16:38:15.519 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:15 smithi001 podman[129771]: 2024-02-28 16:38:15.14973911 +0000 UTC m=+0.641716213 container start a9c4a8eeaf11142ff2767ee52563af60ca81ef40cdc711258fc78d136b8d3df8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-28T16:38:15.519 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:15 smithi001 podman[129771]: 2024-02-28 16:38:15.260381286 +0000 UTC m=+0.752358366 container attach a9c4a8eeaf11142ff2767ee52563af60ca81ef40cdc711258fc78d136b8d3df8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=main-1fb5d10, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:38:15.900 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:15 smithi001 podman[129771]: 2024-02-28 16:38:15.606064597 +0000 UTC m=+1.098041710 container died a9c4a8eeaf11142ff2767ee52563af60ca81ef40cdc711258fc78d136b8d3df8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-1fb5d10) 2024-02-28T16:38:16.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:16 smithi066 ceph-mon[85781]: pgmap v87: 65 pgs: 12 stale+active+clean, 53 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:16.753 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:16 smithi001 podman[129796]: 2024-02-28 16:38:16.306035882 +0000 UTC m=+0.684820508 container remove a9c4a8eeaf11142ff2767ee52563af60ca81ef40cdc711258fc78d136b8d3df8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-28T16:38:16.753 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:16 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-28T16:38:16.753 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:16 smithi001 systemd[1]: Stopped Ceph osd.1 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:38:16.753 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:16 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.1.service: Consumed 1min 44.762s CPU time. 2024-02-28T16:38:16.754 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:16 smithi001 ceph-mon[112940]: pgmap v87: 65 pgs: 12 stale+active+clean, 53 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:17.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:16 smithi001 systemd[1]: Starting Ceph osd.1 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:38:17.424 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:17 smithi001 podman[129895]: 2024-02-28 16:38:17.022758433 +0000 UTC m=+0.035678504 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:38:17.424 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:17 smithi001 podman[129895]: 2024-02-28 16:38:17.140467348 +0000 UTC m=+0.153387455 container create aadc5d0bc73a0fd04c521be3389ef99e4716488f8eb33c6f2086e5af6e173e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-28T16:38:17.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:17 smithi066 ceph-mon[85781]: Health check failed: Degraded data redundancy: 58/372 objects degraded (15.591%), 16 pgs degraded (PG_DEGRADED) 2024-02-28T16:38:17.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:17 smithi001 ceph-mon[112940]: Health check failed: Degraded data redundancy: 58/372 objects degraded (15.591%), 16 pgs degraded (PG_DEGRADED) 2024-02-28T16:38:17.769 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:17 smithi001 podman[129895]: 2024-02-28 16:38:17.436169107 +0000 UTC m=+0.449089221 container init aadc5d0bc73a0fd04c521be3389ef99e4716488f8eb33c6f2086e5af6e173e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-02-28T16:38:17.769 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:17 smithi001 podman[129895]: 2024-02-28 16:38:17.447781735 +0000 UTC m=+0.460701869 container start aadc5d0bc73a0fd04c521be3389ef99e4716488f8eb33c6f2086e5af6e173e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-02-28T16:38:17.769 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:17 smithi001 podman[129895]: 2024-02-28 16:38:17.53472594 +0000 UTC m=+0.547646077 container attach aadc5d0bc73a0fd04c521be3389ef99e4716488f8eb33c6f2086e5af6e173e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=) 2024-02-28T16:38:18.672 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:18 smithi001 ceph-mon[112940]: pgmap v88: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:18.675 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate[129909]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-28T16:38:18.675 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate[129909]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-28T16:38:18.675 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 bash[129895]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-28T16:38:18.675 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 bash[129895]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-28T16:38:18.675 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 bash[129895]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-28T16:38:18.675 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 bash[129895]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 bash[129895]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 bash[129895]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 bash[129895]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate[129909]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate[129909]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate[129909]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate[129909]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate[129909]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-28T16:38:18.676 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:18 smithi001 podman[130038]: 2024-02-28 16:38:18.47185185 +0000 UTC m=+0.039926937 container died aadc5d0bc73a0fd04c521be3389ef99e4716488f8eb33c6f2086e5af6e173e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:38:18.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:18 smithi066 ceph-mon[85781]: pgmap v88: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:19.411 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:19 smithi001 podman[130038]: 2024-02-28 16:38:19.056947218 +0000 UTC m=+0.625022321 container remove aadc5d0bc73a0fd04c521be3389ef99e4716488f8eb33c6f2086e5af6e173e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=main-1fb5d10) 2024-02-28T16:38:19.709 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:19 smithi001 podman[130102]: 2024-02-28 16:38:19.332226753 +0000 UTC m=+0.042856315 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:38:19.709 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:19 smithi001 podman[130102]: 2024-02-28 16:38:19.482995699 +0000 UTC m=+0.193625304 container create 2ff850335691a3c4d4b4fe8af676c24de1d16801882275888a27ed8f5f7f5abc (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS) 2024-02-28T16:38:19.709 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:19 smithi001 podman[130102]: 2024-02-28 16:38:19.720445289 +0000 UTC m=+0.431074921 container init 2ff850335691a3c4d4b4fe8af676c24de1d16801882275888a27ed8f5f7f5abc (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS) 2024-02-28T16:38:20.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:19 smithi001 podman[130102]: 2024-02-28 16:38:19.732445354 +0000 UTC m=+0.443074969 container start 2ff850335691a3c4d4b4fe8af676c24de1d16801882275888a27ed8f5f7f5abc (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-1fb5d10, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-28T16:38:20.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:19 smithi001 ceph-osd[130121]: -- 172.21.15.1:0/2636550275 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de1747f680 con 0x55de17449000 2024-02-28T16:38:20.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:19 smithi001 bash[130102]: 2ff850335691a3c4d4b4fe8af676c24de1d16801882275888a27ed8f5f7f5abc 2024-02-28T16:38:20.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:19 smithi001 systemd[1]: Started Ceph osd.1 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:38:20.735 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:20 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[130116]: 2024-02-28T16:38:20.310+0000 7f34025f77c0 -1 Falling back to public interface 2024-02-28T16:38:20.735 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:20 smithi001 ceph-mon[112940]: pgmap v89: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:20.735 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:20.735 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:20.735 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:38:20.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:20 smithi066 ceph-mon[85781]: pgmap v89: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:20.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:20.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:20.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:38:22.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:22 smithi066 ceph-mon[85781]: pgmap v90: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:22.758 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:22 smithi001 ceph-mon[112940]: pgmap v90: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:24.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:24 smithi066 ceph-mon[85781]: pgmap v91: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:24.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:24 smithi001 ceph-mon[112940]: pgmap v91: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:25.519 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:25 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[130116]: 2024-02-28T16:38:25.199+0000 7f34025f77c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-28T16:38:26.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:25 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[130116]: 2024-02-28T16:38:25.732+0000 7f34025f77c0 -1 osd.1 53 log_to_monitors true 2024-02-28T16:38:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:26 smithi066 ceph-mon[85781]: pgmap v92: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:38:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:26 smithi066 ceph-mon[85781]: from='osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-28T16:38:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:26 smithi001 ceph-mon[112940]: pgmap v92: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:38:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:26 smithi001 ceph-mon[112940]: from='osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-28T16:38:27.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:27 smithi066 ceph-mon[85781]: pgmap v93: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:27.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:27 smithi066 ceph-mon[85781]: from='osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-28T16:38:27.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:27 smithi066 ceph-mon[85781]: osdmap e56: 6 total, 5 up, 6 in 2024-02-28T16:38:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:27 smithi066 ceph-mon[85781]: from='osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:38:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:27 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:27.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:27 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:28.019 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 16:38:27 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[130116]: 2024-02-28T16:38:27.785+0000 7f33eca00700 -1 osd.1 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-28T16:38:28.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:27 smithi001 ceph-mon[112940]: pgmap v93: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:28.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:27 smithi001 ceph-mon[112940]: from='osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-28T16:38:28.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:27 smithi001 ceph-mon[112940]: osdmap e56: 6 total, 5 up, 6 in 2024-02-28T16:38:28.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:27 smithi001 ceph-mon[112940]: from='osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:38:28.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:27 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:28.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:27 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:29.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:28 smithi066 ceph-mon[85781]: from='osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113]' entity='osd.1' 2024-02-28T16:38:29.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:28 smithi001 ceph-mon[112940]: from='osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113]' entity='osd.1' 2024-02-28T16:38:30.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:29 smithi066 ceph-mon[85781]: pgmap v95: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:30.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:29 smithi066 ceph-mon[85781]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:38:30.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:29 smithi066 ceph-mon[85781]: osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113] boot 2024-02-28T16:38:30.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:29 smithi066 ceph-mon[85781]: osdmap e57: 6 total, 6 up, 6 in 2024-02-28T16:38:30.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:29 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:38:30.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:29 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:30.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:29 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:30.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:29 smithi066 ceph-mon[85781]: osdmap e58: 6 total, 6 up, 6 in 2024-02-28T16:38:30.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:29 smithi001 ceph-mon[112940]: pgmap v95: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:30.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:29 smithi001 ceph-mon[112940]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:38:30.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:29 smithi001 ceph-mon[112940]: osd.1 [v2:172.21.15.1:6810/2672566113,v1:172.21.15.1:6811/2672566113] boot 2024-02-28T16:38:30.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:29 smithi001 ceph-mon[112940]: osdmap e57: 6 total, 6 up, 6 in 2024-02-28T16:38:30.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:29 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-28T16:38:30.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:29 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:30.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:29 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:30.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:29 smithi001 ceph-mon[112940]: osdmap e58: 6 total, 6 up, 6 in 2024-02-28T16:38:32.159 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:31 smithi001 ceph-mon[112940]: pgmap v98: 65 pgs: 16 active+undersized+degraded, 16 active+undersized, 33 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:32.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:31 smithi066 ceph-mon[85781]: pgmap v98: 65 pgs: 16 active+undersized+degraded, 16 active+undersized, 33 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 58/372 objects degraded (15.591%) 2024-02-28T16:38:33.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:32 smithi066 ceph-mon[85781]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/372 objects degraded (15.591%), 16 pgs degraded) 2024-02-28T16:38:33.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:32 smithi001 ceph-mon[112940]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/372 objects degraded (15.591%), 16 pgs degraded) 2024-02-28T16:38:34.162 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:33 smithi001 ceph-mon[112940]: pgmap v99: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:34.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:33 smithi066 ceph-mon[85781]: pgmap v99: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: pgmap v100: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-28T16:38:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-28T16:38:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:36 smithi001 ceph-mon[112940]: Upgrade: osd.2 is safe to restart 2024-02-28T16:38:36.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: pgmap v100: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:36.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-28T16:38:36.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:36 smithi066 ceph-mon[85781]: Upgrade: osd.2 is safe to restart 2024-02-28T16:38:38.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:38 smithi001 ceph-mon[112940]: pgmap v101: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:38.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:38.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-28T16:38:38.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:38:38.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:38 smithi066 ceph-mon[85781]: pgmap v101: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:38.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:38.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-28T16:38:38.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:38:39.118 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:39 smithi001 ceph-mon[112940]: Upgrade: Updating osd.2 2024-02-28T16:38:39.119 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:39 smithi001 ceph-mon[112940]: Deploying daemon osd.2 on smithi001 2024-02-28T16:38:39.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:39 smithi066 ceph-mon[85781]: Upgrade: Updating osd.2 2024-02-28T16:38:39.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:39 smithi066 ceph-mon[85781]: Deploying daemon osd.2 on smithi001 2024-02-28T16:38:40.301 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:40 smithi001 ceph-mon[112940]: pgmap v102: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:40.399 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:40 smithi066 ceph-mon[85781]: pgmap v102: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:40.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.878+0000 7fcf30c00700 1 -- 172.21.15.1:0/3046683229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 msgr2=0x7fcf2c102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:40.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.878+0000 7fcf30c00700 1 --2- 172.21.15.1:0/3046683229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 0x7fcf2c102230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c004940 tx=0x7fcf1c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:40.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.878+0000 7fcf30c00700 1 -- 172.21.15.1:0/3046683229 shutdown_connections 2024-02-28T16:38:40.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.878+0000 7fcf30c00700 1 --2- 172.21.15.1:0/3046683229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf2c102800 0x7fcf2c0fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:40.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.878+0000 7fcf30c00700 1 --2- 172.21.15.1:0/3046683229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 0x7fcf2c102230 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.878+0000 7fcf30c00700 1 -- 172.21.15.1:0/3046683229 >> 172.21.15.1:0/3046683229 conn(0x7fcf2c0f7a10 msgr2=0x7fcf2c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.878+0000 7fcf30c00700 1 -- 172.21.15.1:0/3046683229 shutdown_connections 2024-02-28T16:38:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.879+0000 7fcf30c00700 1 -- 172.21.15.1:0/3046683229 wait complete. 2024-02-28T16:38:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.879+0000 7fcf30c00700 1 Processor -- start 2024-02-28T16:38:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.879+0000 7fcf30c00700 1 -- start start 2024-02-28T16:38:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.880+0000 7fcf30c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 0x7fcf2c1030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.880+0000 7fcf30c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf2c102800 0x7fcf2c103630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.880+0000 7fcf30c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c103c80 con 0x7fcf2c101e60 2024-02-28T16:38:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.880+0000 7fcf30c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c103df0 con 0x7fcf2c102800 2024-02-28T16:38:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.880+0000 7fcf29e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 0x7fcf2c1030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.880+0000 7fcf29e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 0x7fcf2c1030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52932/0 (socket says 172.21.15.1:52932) 2024-02-28T16:38:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.880+0000 7fcf29e00700 1 -- 172.21.15.1:0/3404860985 learned_addr learned my addr 172.21.15.1:0/3404860985 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.880+0000 7fcf29400700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf2c102800 0x7fcf2c103630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf29e00700 1 -- 172.21.15.1:0/3404860985 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf2c102800 msgr2=0x7fcf2c103630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf29e00700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf2c102800 0x7fcf2c103630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf29e00700 1 -- 172.21.15.1:0/3404860985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c005040 con 0x7fcf2c101e60 2024-02-28T16:38:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf29400700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf2c102800 0x7fcf2c103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:38:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf29e00700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 0x7fcf2c1030d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009150 tx=0x7fcf1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf22000700 1 -- 172.21.15.1:0/3404860985 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf1c00e040 con 0x7fcf2c101e60 2024-02-28T16:38:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf22000700 1 -- 172.21.15.1:0/3404860985 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf1c00bbf0 con 0x7fcf2c101e60 2024-02-28T16:38:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c119d30 con 0x7fcf2c101e60 2024-02-28T16:38:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.881+0000 7fcf22000700 1 -- 172.21.15.1:0/3404860985 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf1c01c630 con 0x7fcf2c101e60 2024-02-28T16:38:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.882+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c11a210 con 0x7fcf2c101e60 2024-02-28T16:38:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.883+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c0fb610 con 0x7fcf2c101e60 2024-02-28T16:38:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.884+0000 7fcf22000700 1 -- 172.21.15.1:0/3404860985 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcf1c00bd60 con 0x7fcf2c101e60 2024-02-28T16:38:40.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.885+0000 7fcf22000700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1006d7f0 0x7fcf1006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:40.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.885+0000 7fcf22000700 1 -- 172.21.15.1:0/3404860985 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fcf1c08e150 con 0x7fcf2c101e60 2024-02-28T16:38:40.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.885+0000 7fcf29400700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1006d7f0 0x7fcf1006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:40.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.886+0000 7fcf29400700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1006d7f0 0x7fcf1006fca0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009fd0 tx=0x7fcf1800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:40.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:40.890+0000 7fcf22000700 1 -- 172.21.15.1:0/3404860985 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf1c057ab0 con 0x7fcf2c101e60 2024-02-28T16:38:41.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.140+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf2c104810 con 0x7fcf1006d7f0 2024-02-28T16:38:41.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.144+0000 7fcf22000700 1 -- 172.21.15.1:0/3404860985 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fcf2c104810 con 0x7fcf1006d7f0 2024-02-28T16:38:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1006d7f0 msgr2=0x7fcf1006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1006d7f0 0x7fcf1006fca0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009fd0 tx=0x7fcf1800b040 comp rx=0 tx=0).stop 2024-02-28T16:38:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 msgr2=0x7fcf2c1030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 0x7fcf2c1030d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009150 tx=0x7fcf1c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 shutdown_connections 2024-02-28T16:38:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1006d7f0 0x7fcf1006fca0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf2c101e60 0x7fcf2c1030d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 --2- 172.21.15.1:0/3404860985 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf2c102800 0x7fcf2c103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 >> 172.21.15.1:0/3404860985 conn(0x7fcf2c0f7a10 msgr2=0x7fcf2c105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.147+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 shutdown_connections 2024-02-28T16:38:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.148+0000 7fcf30c00700 1 -- 172.21.15.1:0/3404860985 wait complete. 2024-02-28T16:38:41.149 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:38:41.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.312+0000 7fdc6e800700 1 -- 172.21.15.1:0/1817266588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 msgr2=0x7fdc68105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.312+0000 7fdc6e800700 1 --2- 172.21.15.1:0/1817266588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 0x7fdc68105340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c004940 tx=0x7fdc5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:41.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.312+0000 7fdc6e800700 1 -- 172.21.15.1:0/1817266588 shutdown_connections 2024-02-28T16:38:41.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.312+0000 7fdc6e800700 1 --2- 172.21.15.1:0/1817266588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc68105910 0x7fdc680fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.312+0000 7fdc6e800700 1 --2- 172.21.15.1:0/1817266588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 0x7fdc68105340 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.312+0000 7fdc6e800700 1 -- 172.21.15.1:0/1817266588 >> 172.21.15.1:0/1817266588 conn(0x7fdc680faae0 msgr2=0x7fdc680fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:41.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.313+0000 7fdc6e800700 1 -- 172.21.15.1:0/1817266588 shutdown_connections 2024-02-28T16:38:41.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.313+0000 7fdc6e800700 1 -- 172.21.15.1:0/1817266588 wait complete. 2024-02-28T16:38:41.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6e800700 1 Processor -- start 2024-02-28T16:38:41.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6e800700 1 -- start start 2024-02-28T16:38:41.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 0x7fdc68115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:41.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6e800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc68105910 0x7fdc68115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:41.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6e800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc681165b0 con 0x7fdc68104f70 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6e800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6810f8e0 con 0x7fdc68105910 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 0x7fdc68115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 0x7fdc68115a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52938/0 (socket says 172.21.15.1:52938) 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.314+0000 7fdc6d400700 1 -- 172.21.15.1:0/395041394 learned_addr learned my addr 172.21.15.1:0/395041394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc6ca00700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc68105910 0x7fdc68115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc6d400700 1 -- 172.21.15.1:0/395041394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc68105910 msgr2=0x7fdc68115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc6d400700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc68105910 0x7fdc68115f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc6d400700 1 -- 172.21.15.1:0/395041394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c005040 con 0x7fdc68104f70 2024-02-28T16:38:41.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc6ca00700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc68105910 0x7fdc68115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:38:41.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc6d400700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 0x7fdc68115a20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009fd0 tx=0x7fdc5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:41.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc65600700 1 -- 172.21.15.1:0/395041394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc5c00e040 con 0x7fdc68104f70 2024-02-28T16:38:41.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc65600700 1 -- 172.21.15.1:0/395041394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc5c00bbf0 con 0x7fdc68104f70 2024-02-28T16:38:41.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6810fb60 con 0x7fdc68104f70 2024-02-28T16:38:41.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.315+0000 7fdc65600700 1 -- 172.21.15.1:0/395041394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc5c01c3f0 con 0x7fdc68104f70 2024-02-28T16:38:41.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.316+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc68110120 con 0x7fdc68104f70 2024-02-28T16:38:41.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.317+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6805f5c0 con 0x7fdc68104f70 2024-02-28T16:38:41.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.318+0000 7fdc65600700 1 -- 172.21.15.1:0/395041394 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdc5c00bd60 con 0x7fdc68104f70 2024-02-28T16:38:41.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.319+0000 7fdc65600700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc5006d7f0 0x7fdc5006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:41.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.319+0000 7fdc65600700 1 -- 172.21.15.1:0/395041394 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fdc5c08e160 con 0x7fdc68104f70 2024-02-28T16:38:41.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.319+0000 7fdc6ca00700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc5006d7f0 0x7fdc5006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:41.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.320+0000 7fdc6ca00700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc5006d7f0 0x7fdc5006fca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdc54009fd0 tx=0x7fdc5400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:41.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.323+0000 7fdc65600700 1 -- 172.21.15.1:0/395041394 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdc5c05b610 con 0x7fdc68104f70 2024-02-28T16:38:41.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.558+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc68111000 con 0x7fdc5006d7f0 2024-02-28T16:38:41.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.561+0000 7fdc65600700 1 -- 172.21.15.1:0/395041394 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fdc68111000 con 0x7fdc5006d7f0 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.563+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc5006d7f0 msgr2=0x7fdc5006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.563+0000 7fdc6e800700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc5006d7f0 0x7fdc5006fca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdc54009fd0 tx=0x7fdc5400b040 comp rx=0 tx=0).stop 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.563+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 msgr2=0x7fdc68115a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.563+0000 7fdc6e800700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 0x7fdc68115a20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009fd0 tx=0x7fdc5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.563+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 shutdown_connections 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.563+0000 7fdc6e800700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc5006d7f0 0x7fdc5006fca0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.563+0000 7fdc6e800700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc68104f70 0x7fdc68115a20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.563+0000 7fdc6e800700 1 --2- 172.21.15.1:0/395041394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc68105910 0x7fdc68115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.564+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 >> 172.21.15.1:0/395041394 conn(0x7fdc680faae0 msgr2=0x7fdc680fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.564+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 shutdown_connections 2024-02-28T16:38:41.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.564+0000 7fdc6e800700 1 -- 172.21.15.1:0/395041394 wait complete. 2024-02-28T16:38:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.705+0000 7f0a2ae00700 1 -- 172.21.15.1:0/4185438492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a241038f0 msgr2=0x7f0a240fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.705+0000 7f0a2ae00700 1 --2- 172.21.15.1:0/4185438492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a241038f0 0x7f0a240fcf60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0a10004940 tx=0x7f0a10009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.705+0000 7f0a2ae00700 1 -- 172.21.15.1:0/4185438492 shutdown_connections 2024-02-28T16:38:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.705+0000 7f0a2ae00700 1 --2- 172.21.15.1:0/4185438492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a241038f0 0x7f0a240fcf60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.705+0000 7f0a2ae00700 1 --2- 172.21.15.1:0/4185438492 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a24102f50 0x7f0a24103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.705+0000 7f0a2ae00700 1 -- 172.21.15.1:0/4185438492 >> 172.21.15.1:0/4185438492 conn(0x7f0a240f8ae0 msgr2=0x7f0a240faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.706+0000 7f0a2ae00700 1 -- 172.21.15.1:0/4185438492 shutdown_connections 2024-02-28T16:38:41.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.706+0000 7f0a2ae00700 1 -- 172.21.15.1:0/4185438492 wait complete. 2024-02-28T16:38:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.706+0000 7f0a2ae00700 1 Processor -- start 2024-02-28T16:38:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.707+0000 7f0a2ae00700 1 -- start start 2024-02-28T16:38:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.707+0000 7f0a2ae00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a24102f50 0x7f0a24115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.707+0000 7f0a2ae00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a241038f0 0x7f0a241161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.707+0000 7f0a2ae00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a24116810 con 0x7f0a24102f50 2024-02-28T16:38:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.707+0000 7f0a2ae00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2410fb40 con 0x7f0a241038f0 2024-02-28T16:38:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.707+0000 7f0a23e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a24102f50 0x7f0a24115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:41.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.707+0000 7f0a23e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a24102f50 0x7f0a24115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52942/0 (socket says 172.21.15.1:52942) 2024-02-28T16:38:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.707+0000 7f0a23e00700 1 -- 172.21.15.1:0/2642073675 learned_addr learned my addr 172.21.15.1:0/2642073675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.708+0000 7f0a23400700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a241038f0 0x7f0a241161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.708+0000 7f0a23e00700 1 -- 172.21.15.1:0/2642073675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a241038f0 msgr2=0x7f0a241161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.708+0000 7f0a23e00700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a241038f0 0x7f0a241161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.708+0000 7f0a23e00700 1 -- 172.21.15.1:0/2642073675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a10005040 con 0x7f0a24102f50 2024-02-28T16:38:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.708+0000 7f0a23e00700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a24102f50 0x7f0a24115c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a14002a50 tx=0x7f0a140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.708+0000 7f0a1be00700 1 -- 172.21.15.1:0/2642073675 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a14007cc0 con 0x7f0a24102f50 2024-02-28T16:38:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.708+0000 7f0a1be00700 1 -- 172.21.15.1:0/2642073675 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a1400f040 con 0x7f0a24102f50 2024-02-28T16:38:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.708+0000 7f0a1be00700 1 -- 172.21.15.1:0/2642073675 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a14009640 con 0x7f0a24102f50 2024-02-28T16:38:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.709+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2410fe20 con 0x7f0a24102f50 2024-02-28T16:38:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.709+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a24110360 con 0x7f0a24102f50 2024-02-28T16:38:41.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.710+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a240fc6e0 con 0x7f0a24102f50 2024-02-28T16:38:41.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.711+0000 7f0a1be00700 1 -- 172.21.15.1:0/2642073675 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0a140097c0 con 0x7f0a24102f50 2024-02-28T16:38:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.712+0000 7f0a1be00700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a0806d7f0 0x7f0a0806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.712+0000 7f0a1be00700 1 -- 172.21.15.1:0/2642073675 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f0a1408cb70 con 0x7f0a24102f50 2024-02-28T16:38:41.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.712+0000 7f0a23400700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a0806d7f0 0x7f0a0806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:41.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.713+0000 7f0a23400700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a0806d7f0 0x7f0a0806fca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0a241114b0 tx=0x7f0a10009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:41.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.716+0000 7f0a1be00700 1 -- 172.21.15.1:0/2642073675 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0a14059f80 con 0x7f0a24102f50 2024-02-28T16:38:41.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.960+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a241111f0 con 0x7f0a0806d7f0 2024-02-28T16:38:41.948 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:41 smithi001 ceph-mon[112940]: pgmap v103: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:41.948 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:41.948 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:38:41.948 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:41.948 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:41 smithi001 ceph-mon[112940]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.978+0000 7f0a1be00700 1 -- 172.21.15.1:0/2642073675 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0a241111f0 con 0x7f0a0806d7f0 2024-02-28T16:38:41.966 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:38:41.966 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (4m) 14s ago 19m 14.0M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (93s) 14s ago 19m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (85s) 79s ago 17m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (4m) 14s ago 18m 74.8M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (11m) 14s ago 11m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (11m) 14s ago 11m 14.8M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (11m) 79s ago 11m 14.3M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (11m) 79s ago 11m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (7m) 14s ago 20m 503M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (6m) 79s ago 17m 449M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (3m) 14s ago 20m 45.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (2m) 79s ago 17m 35.0M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (5m) 14s ago 19m 8212k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (5m) 79s ago 17m 7612k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:38:41.967 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (53s) 14s ago 16m 80.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:38:41.968 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (22s) 14s ago 16m 11.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:38:41.968 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (15m) 14s ago 15m 170M 1559M 17.2.7-535-gc0f02e60 1c37ed371348 f5b5e43f078b 2024-02-28T16:38:41.968 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (15m) 79s ago 14m 222M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:38:41.968 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (14m) 79s ago 14m 214M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:38:41.968 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (13m) 79s ago 13m 189M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:38:41.968 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (5m) 14s ago 18m 52.0M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:38:41.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a0806d7f0 msgr2=0x7f0a0806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a0806d7f0 0x7f0a0806fca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0a241114b0 tx=0x7f0a10009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a24102f50 msgr2=0x7f0a24115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a24102f50 0x7f0a24115c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a14002a50 tx=0x7f0a140046b0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 shutdown_connections 2024-02-28T16:38:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a0806d7f0 0x7f0a0806fca0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a24102f50 0x7f0a24115c80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 --2- 172.21.15.1:0/2642073675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a241038f0 0x7f0a241161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:41.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 >> 172.21.15.1:0/2642073675 conn(0x7f0a240f8ae0 msgr2=0x7f0a240fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:41.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 shutdown_connections 2024-02-28T16:38:41.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:41.982+0000 7f0a2ae00700 1 -- 172.21.15.1:0/2642073675 wait complete. 2024-02-28T16:38:41.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:41 smithi066 ceph-mon[85781]: pgmap v103: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:38:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:41.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:41 smithi066 ceph-mon[85781]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:42.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.134+0000 7f8c24800700 1 -- 172.21.15.1:0/1006692307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 msgr2=0x7f8c1c1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:42.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.134+0000 7f8c24800700 1 --2- 172.21.15.1:0/1006692307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 0x7f8c1c1022b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8c10004940 tx=0x7f8c10009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:42.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.134+0000 7f8c24800700 1 -- 172.21.15.1:0/1006692307 shutdown_connections 2024-02-28T16:38:42.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.134+0000 7f8c24800700 1 --2- 172.21.15.1:0/1006692307 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c1c102880 0x7f8c1c0fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.134+0000 7f8c24800700 1 --2- 172.21.15.1:0/1006692307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 0x7f8c1c1022b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.134+0000 7f8c24800700 1 -- 172.21.15.1:0/1006692307 >> 172.21.15.1:0/1006692307 conn(0x7f8c1c071560 msgr2=0x7f8c1c071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:42.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.134+0000 7f8c24800700 1 -- 172.21.15.1:0/1006692307 shutdown_connections 2024-02-28T16:38:42.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.134+0000 7f8c24800700 1 -- 172.21.15.1:0/1006692307 wait complete. 2024-02-28T16:38:42.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.135+0000 7f8c24800700 1 Processor -- start 2024-02-28T16:38:42.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.135+0000 7f8c24800700 1 -- start start 2024-02-28T16:38:42.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.135+0000 7f8c24800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 0x7f8c1c113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:42.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.135+0000 7f8c24800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c1c102880 0x7f8c1c113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.135+0000 7f8c24800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c1145e0 con 0x7f8c1c101ee0 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.135+0000 7f8c24800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c10d910 con 0x7f8c1c102880 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.136+0000 7f8c21e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 0x7f8c1c113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.136+0000 7f8c21e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 0x7f8c1c113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33276/0 (socket says 172.21.15.1:33276) 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.136+0000 7f8c21e00700 1 -- 172.21.15.1:0/914705816 learned_addr learned my addr 172.21.15.1:0/914705816 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.136+0000 7f8c21400700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c1c102880 0x7f8c1c113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.136+0000 7f8c21e00700 1 -- 172.21.15.1:0/914705816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c1c102880 msgr2=0x7f8c1c113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.136+0000 7f8c21e00700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c1c102880 0x7f8c1c113f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.136+0000 7f8c21e00700 1 -- 172.21.15.1:0/914705816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c10005040 con 0x7f8c1c101ee0 2024-02-28T16:38:42.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.136+0000 7f8c21400700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c1c102880 0x7f8c1c113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:38:42.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.137+0000 7f8c21e00700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 0x7f8c1c113a50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009150 tx=0x7f8c10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:42.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.137+0000 7f8c16000700 1 -- 172.21.15.1:0/914705816 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c1000e040 con 0x7f8c1c101ee0 2024-02-28T16:38:42.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.137+0000 7f8c16000700 1 -- 172.21.15.1:0/914705816 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c1000bbf0 con 0x7f8c1c101ee0 2024-02-28T16:38:42.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.137+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1c10db90 con 0x7f8c1c101ee0 2024-02-28T16:38:42.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.137+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1c10e0d0 con 0x7f8c1c101ee0 2024-02-28T16:38:42.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.137+0000 7f8c16000700 1 -- 172.21.15.1:0/914705816 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c1001c630 con 0x7f8c1c101ee0 2024-02-28T16:38:42.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.138+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1c100200 con 0x7f8c1c101ee0 2024-02-28T16:38:42.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.139+0000 7f8c16000700 1 -- 172.21.15.1:0/914705816 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8c1000bd60 con 0x7f8c1c101ee0 2024-02-28T16:38:42.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.140+0000 7f8c16000700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c0406d7f0 0x7f8c0406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:42.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.140+0000 7f8c16000700 1 -- 172.21.15.1:0/914705816 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f8c1008e230 con 0x7f8c1c101ee0 2024-02-28T16:38:42.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.141+0000 7f8c21400700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c0406d7f0 0x7f8c0406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:42.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.142+0000 7f8c21400700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c0406d7f0 0x7f8c0406fca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c007fd0 tx=0x7f8c0c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:42.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.145+0000 7f8c16000700 1 -- 172.21.15.1:0/914705816 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8c1005b6e0 con 0x7f8c1c101ee0 2024-02-28T16:38:42.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.461+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8c1c04bf20 con 0x7f8c1c101ee0 2024-02-28T16:38:42.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.462+0000 7f8c16000700 1 -- 172.21.15.1:0/914705816 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f8c1005b270 con 0x7f8c1c101ee0 2024-02-28T16:38:42.449 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4, 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:38:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:38:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 8, 2024-02-28T16:38:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:38:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:38:42.451 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c0406d7f0 msgr2=0x7f8c0406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c0406d7f0 0x7f8c0406fca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c007fd0 tx=0x7f8c0c009040 comp rx=0 tx=0).stop 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 msgr2=0x7f8c1c113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 0x7f8c1c113a50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009150 tx=0x7f8c10009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 shutdown_connections 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c0406d7f0 0x7f8c0406fca0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1c101ee0 0x7f8c1c113a50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 --2- 172.21.15.1:0/914705816 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8c1c102880 0x7f8c1c113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 >> 172.21.15.1:0/914705816 conn(0x7f8c1c071560 msgr2=0x7f8c1c106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 shutdown_connections 2024-02-28T16:38:42.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.464+0000 7f8c24800700 1 -- 172.21.15.1:0/914705816 wait complete. 2024-02-28T16:38:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.630+0000 7f672fc00700 1 -- 172.21.15.1:0/3431614074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 msgr2=0x7f6728070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.630+0000 7f672fc00700 1 --2- 172.21.15.1:0/3431614074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 0x7f6728070330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f671c004970 tx=0x7f671c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.630+0000 7f672fc00700 1 -- 172.21.15.1:0/3431614074 shutdown_connections 2024-02-28T16:38:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.630+0000 7f672fc00700 1 --2- 172.21.15.1:0/3431614074 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6728070870 0x7f6728070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.630+0000 7f672fc00700 1 --2- 172.21.15.1:0/3431614074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 0x7f6728070330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.630+0000 7f672fc00700 1 -- 172.21.15.1:0/3431614074 >> 172.21.15.1:0/3431614074 conn(0x7f67280f8ae0 msgr2=0x7f67280faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.630+0000 7f672fc00700 1 -- 172.21.15.1:0/3431614074 shutdown_connections 2024-02-28T16:38:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.630+0000 7f672fc00700 1 -- 172.21.15.1:0/3431614074 wait complete. 2024-02-28T16:38:42.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.631+0000 7f672fc00700 1 Processor -- start 2024-02-28T16:38:42.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.631+0000 7f672fc00700 1 -- start start 2024-02-28T16:38:42.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.632+0000 7f672fc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6728070870 0x7f672811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.632+0000 7f672fc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 0x7f672811a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.632+0000 7f672fc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672811abe0 con 0x7f6728071f60 2024-02-28T16:38:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.632+0000 7f672fc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6728113ef0 con 0x7f6728070870 2024-02-28T16:38:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.632+0000 7f6727e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 0x7f672811a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.632+0000 7f6727e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 0x7f672811a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33282/0 (socket says 172.21.15.1:33282) 2024-02-28T16:38:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.632+0000 7f6727e00700 1 -- 172.21.15.1:0/4271211883 learned_addr learned my addr 172.21.15.1:0/4271211883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.632+0000 7f672d200700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6728070870 0x7f672811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.633+0000 7f6727e00700 1 -- 172.21.15.1:0/4271211883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6728070870 msgr2=0x7f672811a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.633+0000 7f6727e00700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6728070870 0x7f672811a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.633+0000 7f6727e00700 1 -- 172.21.15.1:0/4271211883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c005040 con 0x7f6728071f60 2024-02-28T16:38:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.633+0000 7f6727e00700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 0x7f672811a590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f671400eb70 tx=0x7f671400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.633+0000 7f6724c00700 1 -- 172.21.15.1:0/4271211883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6714007cc0 con 0x7f6728071f60 2024-02-28T16:38:42.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.633+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6728114170 con 0x7f6728071f60 2024-02-28T16:38:42.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.634+0000 7f6724c00700 1 -- 172.21.15.1:0/4271211883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f671400f040 con 0x7f6728071f60 2024-02-28T16:38:42.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.634+0000 7f6724c00700 1 -- 172.21.15.1:0/4271211883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6714009640 con 0x7f6728071f60 2024-02-28T16:38:42.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.634+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67281146b0 con 0x7f6728071f60 2024-02-28T16:38:42.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.635+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67280fe400 con 0x7f6728071f60 2024-02-28T16:38:42.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.636+0000 7f6724c00700 1 -- 172.21.15.1:0/4271211883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f67140097a0 con 0x7f6728071f60 2024-02-28T16:38:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.637+0000 7f6724c00700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f671006d7f0 0x7f671006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.638+0000 7f6724c00700 1 -- 172.21.15.1:0/4271211883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f671408c9a0 con 0x7f6728071f60 2024-02-28T16:38:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.638+0000 7f672d200700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f671006d7f0 0x7f671006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:42.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.639+0000 7f672d200700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f671006d7f0 0x7f671006fca0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f671c009fd0 tx=0x7f671c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.642+0000 7f6724c00700 1 -- 172.21.15.1:0/4271211883 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6714059e50 con 0x7f6728071f60 2024-02-28T16:38:42.744 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:42 smithi001 systemd[1]: Stopping Ceph osd.2 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:38:42.745 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:42 smithi001 ceph-mon[112940]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:42.745 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:42 smithi001 ceph-mon[112940]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:42.745 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:42 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/914705816' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:42.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.940+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f67280fc6e0 con 0x7f6728071f60 2024-02-28T16:38:42.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.941+0000 7f6724c00700 1 -- 172.21.15.1:0/4271211883 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7f67140599e0 con 0x7f6728071f60 2024-02-28T16:38:42.929 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:38:42.930 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:38:42.931 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:38:42.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:38:42.933 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:38:42.933 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:38:42.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f671006d7f0 msgr2=0x7f671006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f671006d7f0 0x7f671006fca0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f671c009fd0 tx=0x7f671c009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 msgr2=0x7f672811a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 0x7f672811a590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f671400eb70 tx=0x7f671400b040 comp rx=0 tx=0).stop 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 shutdown_connections 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f671006d7f0 0x7f671006fca0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6728070870 0x7f672811a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 --2- 172.21.15.1:0/4271211883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6728071f60 0x7f672811a590 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 >> 172.21.15.1:0/4271211883 conn(0x7f67280f8ae0 msgr2=0x7f67280fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 shutdown_connections 2024-02-28T16:38:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:42.945+0000 7f672fc00700 1 -- 172.21.15.1:0/4271211883 wait complete. 2024-02-28T16:38:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:38:43.019 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:42 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[62451]: 2024-02-28T16:38:42.890+0000 7f588fc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:38:43.019 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:42 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[62451]: 2024-02-28T16:38:42.890+0000 7f588fc00700 -1 osd.2 58 *** Got signal Terminated *** 2024-02-28T16:38:43.019 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:42 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[62451]: 2024-02-28T16:38:42.890+0000 7f588fc00700 -1 osd.2 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T16:38:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.092+0000 7fcacc400700 1 -- 172.21.15.1:0/350696683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 msgr2=0x7fcac41099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.092+0000 7fcacc400700 1 --2- 172.21.15.1:0/350696683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 0x7fcac41099c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcab8004940 tx=0x7fcab8009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.092+0000 7fcacc400700 1 -- 172.21.15.1:0/350696683 shutdown_connections 2024-02-28T16:38:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.092+0000 7fcacc400700 1 --2- 172.21.15.1:0/350696683 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcac4103dc0 0x7fcac4104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.092+0000 7fcacc400700 1 --2- 172.21.15.1:0/350696683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 0x7fcac41099c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.092+0000 7fcacc400700 1 -- 172.21.15.1:0/350696683 >> 172.21.15.1:0/350696683 conn(0x7fcac4072d50 msgr2=0x7fcac4075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.092+0000 7fcacc400700 1 -- 172.21.15.1:0/350696683 shutdown_connections 2024-02-28T16:38:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.092+0000 7fcacc400700 1 -- 172.21.15.1:0/350696683 wait complete. 2024-02-28T16:38:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.093+0000 7fcacc400700 1 Processor -- start 2024-02-28T16:38:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.093+0000 7fcacc400700 1 -- start start 2024-02-28T16:38:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcacc400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcac4103dc0 0x7fcac411a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcacc400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 0x7fcac411a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcacc400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac411ab90 con 0x7fcac41095f0 2024-02-28T16:38:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcacc400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4113ee0 con 0x7fcac4103dc0 2024-02-28T16:38:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcac9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 0x7fcac411a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcac9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 0x7fcac411a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33300/0 (socket says 172.21.15.1:33300) 2024-02-28T16:38:43.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcac9000700 1 -- 172.21.15.1:0/3690009252 learned_addr learned my addr 172.21.15.1:0/3690009252 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcac9a00700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcac4103dc0 0x7fcac411a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcac9000700 1 -- 172.21.15.1:0/3690009252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcac4103dc0 msgr2=0x7fcac411a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcac9000700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcac4103dc0 0x7fcac411a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.094+0000 7fcac9000700 1 -- 172.21.15.1:0/3690009252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab8005040 con 0x7fcac41095f0 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.095+0000 7fcac9000700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 0x7fcac411a540 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcab000eb40 tx=0x7fcab000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.095+0000 7fcac1600700 1 -- 172.21.15.1:0/3690009252 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcab0007cc0 con 0x7fcac41095f0 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.095+0000 7fcac1600700 1 -- 172.21.15.1:0/3690009252 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcab000f040 con 0x7fcac41095f0 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.095+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac4114160 con 0x7fcac41095f0 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.095+0000 7fcac1600700 1 -- 172.21.15.1:0/3690009252 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcab0009620 con 0x7fcac41095f0 2024-02-28T16:38:43.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.095+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac41146a0 con 0x7fcac41095f0 2024-02-28T16:38:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.097+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac4075870 con 0x7fcac41095f0 2024-02-28T16:38:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.097+0000 7fcac1600700 1 -- 172.21.15.1:0/3690009252 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcab0009780 con 0x7fcac41095f0 2024-02-28T16:38:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.099+0000 7fcac1600700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcaac06d7f0 0x7fcaac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.099+0000 7fcac1600700 1 -- 172.21.15.1:0/3690009252 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fcab008cb50 con 0x7fcac41095f0 2024-02-28T16:38:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.099+0000 7fcac9a00700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcaac06d7f0 0x7fcaac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.099+0000 7fcac9a00700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcaac06d7f0 0x7fcaac06fca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcab8000c00 tx=0x7fcab8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.103+0000 7fcac1600700 1 -- 172.21.15.1:0/3690009252 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcab0056460 con 0x7fcac41095f0 2024-02-28T16:38:43.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:42 smithi066 ceph-mon[85781]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:43.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:42 smithi066 ceph-mon[85781]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:43.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:42 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/914705816' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:38:43.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.334+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcac4115620 con 0x7fcaac06d7f0 2024-02-28T16:38:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.337+0000 7fcac1600700 1 -- 172.21.15.1:0/3690009252 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fcac4115620 con 0x7fcaac06d7f0 2024-02-28T16:38:43.324 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "crash" 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "8/21 daemons upgraded", 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading osd daemons", 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:38:43.325 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:38:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcaac06d7f0 msgr2=0x7fcaac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcaac06d7f0 0x7fcaac06fca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcab8000c00 tx=0x7fcab8009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 msgr2=0x7fcac411a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 0x7fcac411a540 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcab000eb40 tx=0x7fcab000b040 comp rx=0 tx=0).stop 2024-02-28T16:38:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 shutdown_connections 2024-02-28T16:38:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcaac06d7f0 0x7fcaac06fca0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcac4103dc0 0x7fcac411a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 --2- 172.21.15.1:0/3690009252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcac41095f0 0x7fcac411a540 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.339+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 >> 172.21.15.1:0/3690009252 conn(0x7fcac4072d50 msgr2=0x7fcac40746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.340+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 shutdown_connections 2024-02-28T16:38:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.340+0000 7fcacc400700 1 -- 172.21.15.1:0/3690009252 wait complete. 2024-02-28T16:38:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.495+0000 7f46b2200700 1 -- 172.21.15.1:0/1282954121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 msgr2=0x7f46ac070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.495+0000 7f46b2200700 1 --2- 172.21.15.1:0/1282954121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 0x7f46ac070ce0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4698004970 tx=0x7f4698009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.495+0000 7f46b2200700 1 -- 172.21.15.1:0/1282954121 shutdown_connections 2024-02-28T16:38:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.495+0000 7f46b2200700 1 --2- 172.21.15.1:0/1282954121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 0x7f46ac070ce0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.495+0000 7f46b2200700 1 --2- 172.21.15.1:0/1282954121 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46ac071f60 0x7f46ac070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.495+0000 7f46b2200700 1 -- 172.21.15.1:0/1282954121 >> 172.21.15.1:0/1282954121 conn(0x7f46ac0f8af0 msgr2=0x7f46ac0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.495+0000 7f46b2200700 1 -- 172.21.15.1:0/1282954121 shutdown_connections 2024-02-28T16:38:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.496+0000 7f46b2200700 1 -- 172.21.15.1:0/1282954121 wait complete. 2024-02-28T16:38:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.497+0000 7f46b2200700 1 Processor -- start 2024-02-28T16:38:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.497+0000 7f46b2200700 1 -- start start 2024-02-28T16:38:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.498+0000 7f46b2200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 0x7f46ac11a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.498+0000 7f46b2200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46ac071f60 0x7f46ac11a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.498+0000 7f46b2200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac11ac20 con 0x7f46ac070870 2024-02-28T16:38:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.498+0000 7f46b2200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac113f30 con 0x7f46ac071f60 2024-02-28T16:38:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.498+0000 7f46ab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 0x7f46ac11a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.498+0000 7f46ab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 0x7f46ac11a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33306/0 (socket says 172.21.15.1:33306) 2024-02-28T16:38:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.498+0000 7f46ab400700 1 -- 172.21.15.1:0/3062351879 learned_addr learned my addr 172.21.15.1:0/3062351879 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:38:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.498+0000 7f46aaa00700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46ac071f60 0x7f46ac11a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46ab400700 1 -- 172.21.15.1:0/3062351879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46ac071f60 msgr2=0x7f46ac11a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46ab400700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46ac071f60 0x7f46ac11a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46ab400700 1 -- 172.21.15.1:0/3062351879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4698005040 con 0x7f46ac070870 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46aaa00700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46ac071f60 0x7f46ac11a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46ab400700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 0x7f46ac11a090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f469c002a50 tx=0x7f469c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46a3400700 1 -- 172.21.15.1:0/3062351879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f469c007cc0 con 0x7f46ac070870 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46a3400700 1 -- 172.21.15.1:0/3062351879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f469c00f040 con 0x7f46ac070870 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46a3400700 1 -- 172.21.15.1:0/3062351879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f469c009640 con 0x7f46ac070870 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.499+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ac114210 con 0x7f46ac070870 2024-02-28T16:38:43.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.500+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ac114750 con 0x7f46ac070870 2024-02-28T16:38:43.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.501+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ac0fe410 con 0x7f46ac070870 2024-02-28T16:38:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.502+0000 7f46a3400700 1 -- 172.21.15.1:0/3062351879 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f469c0097c0 con 0x7f46ac070870 2024-02-28T16:38:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.503+0000 7f46a3400700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f469006d7e0 0x7f469006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:38:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.504+0000 7f46a3400700 1 -- 172.21.15.1:0/3062351879 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f469c08cb70 con 0x7f46ac070870 2024-02-28T16:38:43.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.504+0000 7f46aaa00700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f469006d7e0 0x7f469006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:38:43.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.505+0000 7f46aaa00700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f469006d7e0 0x7f469006fc90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f46ac115930 tx=0x7f4698009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:38:43.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.509+0000 7f46a3400700 1 -- 172.21.15.1:0/3062351879 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f469c059f80 con 0x7f46ac070870 2024-02-28T16:38:43.743 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:43 smithi001 ceph-mon[112940]: pgmap v104: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:43.743 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:43 smithi001 ceph-mon[112940]: osd.2 marked itself down and dead 2024-02-28T16:38:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.844+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f46ac0fc6f0 con 0x7f46ac070870 2024-02-28T16:38:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.845+0000 7f46a3400700 1 -- 172.21.15.1:0/3062351879 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+290 (secure 0 0 0) 0x7f469c059b10 con 0x7f46ac070870 2024-02-28T16:38:43.833 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-02-28T16:38:43.833 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:38:43.833 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:38:43.833 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-28T16:38:43.833 INFO:teuthology.orchestra.run.smithi001.stdout: osd.2 (root=default,host=smithi001) is down 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f469006d7e0 msgr2=0x7f469006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f469006d7e0 0x7f469006fc90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f46ac115930 tx=0x7f4698009e30 comp rx=0 tx=0).stop 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 msgr2=0x7f46ac11a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 0x7f46ac11a090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f469c002a50 tx=0x7f469c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 shutdown_connections 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f469006d7e0 0x7f469006fc90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46ac070870 0x7f46ac11a090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 --2- 172.21.15.1:0/3062351879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f46ac071f60 0x7f46ac11a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:38:43.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.848+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 >> 172.21.15.1:0/3062351879 conn(0x7f46ac0f8af0 msgr2=0x7f46ac0fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:38:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.849+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 shutdown_connections 2024-02-28T16:38:43.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:38:43.849+0000 7f46b2200700 1 -- 172.21.15.1:0/3062351879 wait complete. 2024-02-28T16:38:43.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:43 smithi066 ceph-mon[85781]: pgmap v104: 65 pgs: 65 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:43.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:43 smithi066 ceph-mon[85781]: osd.2 marked itself down and dead 2024-02-28T16:38:43.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:43 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4271211883' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:38:43.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:43 smithi066 ceph-mon[85781]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:43 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4271211883' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:38:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:43 smithi001 ceph-mon[112940]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:38:44.519 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:44 smithi001 podman[135140]: 2024-02-28 16:38:44.261688697 +0000 UTC m=+1.536418034 container died f5b5e43f078b93e9839f7ea8eebc71c4ddf1edab1ab85cc59f455d3d677c90d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=quincy-c0f02e6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-28T16:38:44.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:44 smithi066 ceph-mon[85781]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:38:44.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:44 smithi066 ceph-mon[85781]: osdmap e59: 6 total, 5 up, 6 in 2024-02-28T16:38:44.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:44 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3062351879' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:38:45.006 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:44 smithi001 ceph-mon[112940]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:38:45.007 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:44 smithi001 ceph-mon[112940]: osdmap e59: 6 total, 5 up, 6 in 2024-02-28T16:38:45.007 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:44 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3062351879' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:38:45.007 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:44 smithi001 podman[135140]: 2024-02-28 16:38:44.816880109 +0000 UTC m=+2.091609451 container cleanup f5b5e43f078b93e9839f7ea8eebc71c4ddf1edab1ab85cc59f455d3d677c90d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c0f02e6, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-28T16:38:45.007 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:44 smithi001 bash[135140]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2 2024-02-28T16:38:45.269 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:45 smithi001 podman[135208]: 2024-02-28 16:38:45.018649921 +0000 UTC m=+0.741118617 container remove f5b5e43f078b93e9839f7ea8eebc71c4ddf1edab1ab85cc59f455d3d677c90d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-c0f02e6, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:38:46.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:45 smithi001 ceph-mon[112940]: pgmap v106: 65 pgs: 9 stale+active+clean, 56 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:46.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:45 smithi001 ceph-mon[112940]: osdmap e60: 6 total, 5 up, 6 in 2024-02-28T16:38:46.020 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:45 smithi001 podman[135299]: 2024-02-28 16:38:45.703097806 +0000 UTC m=+0.033876240 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:38:46.020 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:45 smithi001 podman[135299]: 2024-02-28 16:38:45.836340715 +0000 UTC m=+0.167119181 container create 503929a5e3fefbbdfd753c850ac606ed618d79919b2b42dc3baca329c6a05419 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-28T16:38:46.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:45 smithi066 ceph-mon[85781]: pgmap v106: 65 pgs: 9 stale+active+clean, 56 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:38:46.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:45 smithi066 ceph-mon[85781]: osdmap e60: 6 total, 5 up, 6 in 2024-02-28T16:38:46.519 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:46 smithi001 podman[135299]: 2024-02-28 16:38:46.13279551 +0000 UTC m=+0.463574017 container init 503929a5e3fefbbdfd753c850ac606ed618d79919b2b42dc3baca329c6a05419 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-28T16:38:46.519 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:46 smithi001 podman[135299]: 2024-02-28 16:38:46.145411621 +0000 UTC m=+0.476190108 container start 503929a5e3fefbbdfd753c850ac606ed618d79919b2b42dc3baca329c6a05419 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-28T16:38:46.519 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:46 smithi001 podman[135299]: 2024-02-28 16:38:46.271108988 +0000 UTC m=+0.601887458 container attach 503929a5e3fefbbdfd753c850ac606ed618d79919b2b42dc3baca329c6a05419 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-28T16:38:46.997 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:46 smithi001 podman[135299]: 2024-02-28 16:38:46.600787168 +0000 UTC m=+0.931565625 container died 503929a5e3fefbbdfd753c850ac606ed618d79919b2b42dc3baca329c6a05419 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-28T16:38:46.998 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:46 smithi001 ceph-mon[112940]: Health check failed: Degraded data redundancy: 32/372 objects degraded (8.602%), 9 pgs degraded (PG_DEGRADED) 2024-02-28T16:38:47.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:46 smithi066 ceph-mon[85781]: Health check failed: Degraded data redundancy: 32/372 objects degraded (8.602%), 9 pgs degraded (PG_DEGRADED) 2024-02-28T16:38:47.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:47 smithi001 podman[135325]: 2024-02-28 16:38:47.426569985 +0000 UTC m=+0.803451857 container remove 503929a5e3fefbbdfd753c850ac606ed618d79919b2b42dc3baca329c6a05419 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=main-1fb5d10, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-28T16:38:47.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:47 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-28T16:38:47.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:47 smithi001 systemd[1]: Stopped Ceph osd.2 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:38:47.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:47 smithi001 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.2.service: Consumed 1min 22.192s CPU time. 2024-02-28T16:38:48.192 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:47 smithi001 ceph-mon[112940]: pgmap v108: 65 pgs: 9 active+undersized+degraded, 4 stale+active+clean, 11 active+undersized, 41 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 32/372 objects degraded (8.602%) 2024-02-28T16:38:48.193 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:47 smithi001 systemd[1]: Starting Ceph osd.2 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:38:48.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:47 smithi066 ceph-mon[85781]: pgmap v108: 65 pgs: 9 active+undersized+degraded, 4 stale+active+clean, 11 active+undersized, 41 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 32/372 objects degraded (8.602%) 2024-02-28T16:38:48.494 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:48 smithi001 podman[135424]: 2024-02-28 16:38:48.108699347 +0000 UTC m=+0.040796216 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:38:48.495 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:48 smithi001 podman[135424]: 2024-02-28 16:38:48.219214142 +0000 UTC m=+0.151311057 container create 2afc448c5e674e926d02aa3a24b2cac8f82034b9df771400041fd6d477378772 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222) 2024-02-28T16:38:48.495 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:48 smithi001 podman[135424]: 2024-02-28 16:38:48.50670274 +0000 UTC m=+0.438799661 container init 2afc448c5e674e926d02aa3a24b2cac8f82034b9df771400041fd6d477378772 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-28T16:38:48.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:48 smithi001 podman[135424]: 2024-02-28 16:38:48.518716648 +0000 UTC m=+0.450813494 container start 2afc448c5e674e926d02aa3a24b2cac8f82034b9df771400041fd6d477378772 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-28T16:38:48.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:48 smithi001 podman[135424]: 2024-02-28 16:38:48.663910653 +0000 UTC m=+0.596007537 container attach 2afc448c5e674e926d02aa3a24b2cac8f82034b9df771400041fd6d477378772 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:38:49.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate[135439]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-28T16:38:49.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate[135439]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-28T16:38:49.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 bash[135424]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-28T16:38:49.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 bash[135424]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-28T16:38:49.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 bash[135424]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-28T16:38:49.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 bash[135424]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-28T16:38:49.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 bash[135424]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-28T16:38:49.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 bash[135424]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-28T16:38:49.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 bash[135424]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-28T16:38:49.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate[135439]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-28T16:38:49.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate[135439]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-28T16:38:49.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate[135439]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-28T16:38:49.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate[135439]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-28T16:38:49.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate[135439]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-28T16:38:49.770 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:49 smithi001 podman[135424]: 2024-02-28 16:38:49.463907716 +0000 UTC m=+1.396004604 container died 2afc448c5e674e926d02aa3a24b2cac8f82034b9df771400041fd6d477378772 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True, RELEASE=main-1fb5d10) 2024-02-28T16:38:50.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:49 smithi001 ceph-mon[112940]: pgmap v109: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:50.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:49 smithi066 ceph-mon[85781]: pgmap v109: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:50.519 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:50 smithi001 podman[135562]: 2024-02-28 16:38:50.200897282 +0000 UTC m=+0.715487305 container remove 2afc448c5e674e926d02aa3a24b2cac8f82034b9df771400041fd6d477378772 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-1fb5d10, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-02-28T16:38:50.843 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:50 smithi001 podman[135626]: 2024-02-28 16:38:50.4654664 +0000 UTC m=+0.036488770 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:38:50.843 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:50 smithi001 podman[135626]: 2024-02-28 16:38:50.601778718 +0000 UTC m=+0.172801145 container create 4cd5368c3ac635a883f4dbd0194e698e6513feb56723585c140a03aaa40e802b (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-28T16:38:50.843 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:50 smithi001 podman[135626]: 2024-02-28 16:38:50.855116067 +0000 UTC m=+0.426138475 container init 4cd5368c3ac635a883f4dbd0194e698e6513feb56723585c140a03aaa40e802b (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-02-28T16:38:51.269 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:50 smithi001 podman[135626]: 2024-02-28 16:38:50.868275002 +0000 UTC m=+0.439297410 container start 4cd5368c3ac635a883f4dbd0194e698e6513feb56723585c140a03aaa40e802b (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-1fb5d10, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:38:51.269 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:50 smithi001 ceph-osd[135645]: -- 172.21.15.1:0/2452068347 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5586f2e7f680 con 0x5586f2e49000 2024-02-28T16:38:51.269 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:50 smithi001 bash[135626]: 4cd5368c3ac635a883f4dbd0194e698e6513feb56723585c140a03aaa40e802b 2024-02-28T16:38:51.270 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:50 smithi001 systemd[1]: Started Ceph osd.2 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:38:51.769 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:51 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[135641]: 2024-02-28T16:38:51.434+0000 7fb3ff5417c0 -1 Falling back to public interface 2024-02-28T16:38:52.455 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:52 smithi001 ceph-mon[112940]: pgmap v110: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:52.455 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:52.455 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:52.455 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:38:52.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:52 smithi066 ceph-mon[85781]: pgmap v110: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:52.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:52.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:52.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:38:54.461 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:54 smithi001 ceph-mon[112940]: pgmap v111: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:54.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:54 smithi066 ceph-mon[85781]: pgmap v111: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:55.993 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:55 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[135641]: 2024-02-28T16:38:55.681+0000 7fb3ff5417c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-28T16:38:56.269 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:56 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[135641]: 2024-02-28T16:38:56.005+0000 7fb3ff5417c0 -1 osd.2 58 log_to_monitors true 2024-02-28T16:38:56.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:56 smithi001 ceph-mon[112940]: pgmap v112: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:56.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:56.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:38:56.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:56 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 45/372 objects degraded (12.097%), 12 pgs degraded (PG_DEGRADED) 2024-02-28T16:38:56.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:56 smithi001 ceph-mon[112940]: from='osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-28T16:38:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:56 smithi066 ceph-mon[85781]: pgmap v112: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:38:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:56 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 45/372 objects degraded (12.097%), 12 pgs degraded (PG_DEGRADED) 2024-02-28T16:38:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:56 smithi066 ceph-mon[85781]: from='osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-28T16:38:57.889 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:57 smithi001 ceph-mon[112940]: pgmap v113: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:57.889 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:57 smithi001 ceph-mon[112940]: from='osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-28T16:38:57.889 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:57 smithi001 ceph-mon[112940]: osdmap e61: 6 total, 5 up, 6 in 2024-02-28T16:38:57.889 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:57 smithi001 ceph-mon[112940]: from='osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:38:57.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:57 smithi066 ceph-mon[85781]: pgmap v113: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:38:57.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:57 smithi066 ceph-mon[85781]: from='osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-28T16:38:57.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:57 smithi066 ceph-mon[85781]: osdmap e61: 6 total, 5 up, 6 in 2024-02-28T16:38:57.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:57 smithi066 ceph-mon[85781]: from='osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-02-28T16:38:58.269 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 16:38:57 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[135641]: 2024-02-28T16:38:57.900+0000 7fb3e9800700 -1 osd.2 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-28T16:38:59.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:58 smithi066 ceph-mon[85781]: from='osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371]' entity='osd.2' 2024-02-28T16:38:59.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:58 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:59.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:58 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:59.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:58 smithi001 ceph-mon[112940]: from='osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371]' entity='osd.2' 2024-02-28T16:38:59.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:58 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:38:59.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:58 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:00.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:59 smithi066 ceph-mon[85781]: pgmap v115: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:39:00.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:59 smithi066 ceph-mon[85781]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:39:00.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:59 smithi066 ceph-mon[85781]: osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371] boot 2024-02-28T16:39:00.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:59 smithi066 ceph-mon[85781]: osdmap e62: 6 total, 6 up, 6 in 2024-02-28T16:39:00.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:59 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:39:00.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:38:59 smithi066 ceph-mon[85781]: osdmap e63: 6 total, 6 up, 6 in 2024-02-28T16:39:00.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:59 smithi001 ceph-mon[112940]: pgmap v115: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 45/372 objects degraded (12.097%) 2024-02-28T16:39:00.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:59 smithi001 ceph-mon[112940]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:39:00.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:59 smithi001 ceph-mon[112940]: osd.2 [v2:172.21.15.1:6818/4011428371,v1:172.21.15.1:6819/4011428371] boot 2024-02-28T16:39:00.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:59 smithi001 ceph-mon[112940]: osdmap e62: 6 total, 6 up, 6 in 2024-02-28T16:39:00.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:59 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-28T16:39:00.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:38:59 smithi001 ceph-mon[112940]: osdmap e63: 6 total, 6 up, 6 in 2024-02-28T16:39:01.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:01 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:01.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:01 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:01.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:01 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:01.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:01 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:02.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:02 smithi001 ceph-mon[112940]: pgmap v118: 65 pgs: 10 active+undersized+degraded, 12 active+undersized, 43 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 35/372 objects degraded (9.409%) 2024-02-28T16:39:02.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:02 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 35/372 objects degraded (9.409%), 10 pgs degraded (PG_DEGRADED) 2024-02-28T16:39:02.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:02 smithi066 ceph-mon[85781]: pgmap v118: 65 pgs: 10 active+undersized+degraded, 12 active+undersized, 43 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 35/372 objects degraded (9.409%) 2024-02-28T16:39:02.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:02 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 35/372 objects degraded (9.409%), 10 pgs degraded (PG_DEGRADED) 2024-02-28T16:39:03.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:03 smithi001 ceph-mon[112940]: pgmap v119: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:03.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:03 smithi001 ceph-mon[112940]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/372 objects degraded (9.409%), 10 pgs degraded) 2024-02-28T16:39:03.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:03 smithi066 ceph-mon[85781]: pgmap v119: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:03.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:03 smithi066 ceph-mon[85781]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/372 objects degraded (9.409%), 10 pgs degraded) 2024-02-28T16:39:06.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:05 smithi001 ceph-mon[112940]: pgmap v120: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:06.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:05 smithi066 ceph-mon[85781]: pgmap v120: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:07.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:07.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:07.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:39:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:39:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:39:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:07.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-28T16:39:07.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:07.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:07.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:07.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:39:07.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:39:07.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:07.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:39:07.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:07.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:07.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:07.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-28T16:39:08.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:08 smithi001 ceph-mon[112940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-28T16:39:08.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:08 smithi001 ceph-mon[112940]: Upgrade: osd.3 is safe to restart 2024-02-28T16:39:08.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:08 smithi001 ceph-mon[112940]: pgmap v121: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:08.532 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:08 smithi066 ceph-mon[85781]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-28T16:39:08.533 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:08 smithi066 ceph-mon[85781]: Upgrade: osd.3 is safe to restart 2024-02-28T16:39:08.533 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:08 smithi066 ceph-mon[85781]: pgmap v121: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:10.312 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:10 smithi066 ceph-mon[85781]: pgmap v122: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:10.312 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:10 smithi066 ceph-mon[85781]: Upgrade: Updating osd.3 2024-02-28T16:39:10.312 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:10.312 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-28T16:39:10.312 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:39:10.312 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:10 smithi066 ceph-mon[85781]: Deploying daemon osd.3 on smithi066 2024-02-28T16:39:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:10 smithi001 ceph-mon[112940]: pgmap v122: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:10 smithi001 ceph-mon[112940]: Upgrade: Updating osd.3 2024-02-28T16:39:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-28T16:39:10.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:39:10.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:10 smithi001 ceph-mon[112940]: Deploying daemon osd.3 on smithi066 2024-02-28T16:39:11.978 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:11 smithi066 ceph-mon[85781]: pgmap v123: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:11.978 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:11.978 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:39:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:11 smithi001 ceph-mon[112940]: pgmap v123: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:39:13.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:13 smithi066 ceph-mon[85781]: pgmap v124: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:14.008 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:13 smithi001 ceph-mon[112940]: pgmap v124: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:14.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.020+0000 7f786a200700 1 -- 172.21.15.1:0/2781269670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78641095f0 msgr2=0x7f78641099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.020+0000 7f786a200700 1 --2- 172.21.15.1:0/2781269670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78641095f0 0x7f78641099c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7854004940 tx=0x7f7854009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:14.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.020+0000 7f786a200700 1 -- 172.21.15.1:0/2781269670 shutdown_connections 2024-02-28T16:39:14.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.020+0000 7f786a200700 1 --2- 172.21.15.1:0/2781269670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7864103dc0 0x7f7864104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.020+0000 7f786a200700 1 --2- 172.21.15.1:0/2781269670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78641095f0 0x7f78641099c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.020+0000 7f786a200700 1 -- 172.21.15.1:0/2781269670 >> 172.21.15.1:0/2781269670 conn(0x7f7864072d50 msgr2=0x7f7864075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.020+0000 7f786a200700 1 -- 172.21.15.1:0/2781269670 shutdown_connections 2024-02-28T16:39:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.020+0000 7f786a200700 1 -- 172.21.15.1:0/2781269670 wait complete. 2024-02-28T16:39:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.022+0000 7f786a200700 1 Processor -- start 2024-02-28T16:39:14.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.022+0000 7f786a200700 1 -- start start 2024-02-28T16:39:14.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.022+0000 7f786a200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7864103dc0 0x7f786411a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.022+0000 7f786a200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78641095f0 0x7f786411a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.022+0000 7f786a200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786411ab90 con 0x7f7864103dc0 2024-02-28T16:39:14.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.022+0000 7f786a200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7864113ee0 con 0x7f78641095f0 2024-02-28T16:39:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.023+0000 7f7863400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7864103dc0 0x7f786411a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.023+0000 7f7863400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7864103dc0 0x7f786411a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38454/0 (socket says 172.21.15.1:38454) 2024-02-28T16:39:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.023+0000 7f7863400700 1 -- 172.21.15.1:0/550793414 learned_addr learned my addr 172.21.15.1:0/550793414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.023+0000 7f7862a00700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78641095f0 0x7f786411a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.023+0000 7f7863400700 1 -- 172.21.15.1:0/550793414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78641095f0 msgr2=0x7f786411a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.023+0000 7f7863400700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78641095f0 0x7f786411a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.023+0000 7f7863400700 1 -- 172.21.15.1:0/550793414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7854005040 con 0x7f7864103dc0 2024-02-28T16:39:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.023+0000 7f7863400700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7864103dc0 0x7f786411a000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7854009150 tx=0x7f7854009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:14.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.024+0000 7f785b400700 1 -- 172.21.15.1:0/550793414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f785400e040 con 0x7f7864103dc0 2024-02-28T16:39:14.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.024+0000 7f785b400700 1 -- 172.21.15.1:0/550793414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f785400bbf0 con 0x7f7864103dc0 2024-02-28T16:39:14.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.024+0000 7f785b400700 1 -- 172.21.15.1:0/550793414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f785401c630 con 0x7f7864103dc0 2024-02-28T16:39:14.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.024+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7864114160 con 0x7f7864103dc0 2024-02-28T16:39:14.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.024+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78641146a0 con 0x7f7864103dc0 2024-02-28T16:39:14.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.025+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7864075870 con 0x7f7864103dc0 2024-02-28T16:39:14.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.026+0000 7f785b400700 1 -- 172.21.15.1:0/550793414 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f785400bd60 con 0x7f7864103dc0 2024-02-28T16:39:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.027+0000 7f785b400700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f784806d7f0 0x7f784806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.027+0000 7f785b400700 1 -- 172.21.15.1:0/550793414 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f785408e210 con 0x7f7864103dc0 2024-02-28T16:39:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.028+0000 7f7862a00700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f784806d7f0 0x7f784806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:14.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.028+0000 7f7862a00700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f784806d7f0 0x7f784806fca0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7850009fd0 tx=0x7f7850009f60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:14.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.033+0000 7f785b400700 1 -- 172.21.15.1:0/550793414 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7854057b70 con 0x7f7864103dc0 2024-02-28T16:39:14.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.276+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7864115690 con 0x7f784806d7f0 2024-02-28T16:39:14.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.277+0000 7f785b400700 1 -- 172.21.15.1:0/550793414 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7864115690 con 0x7f784806d7f0 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f784806d7f0 msgr2=0x7f784806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f784806d7f0 0x7f784806fca0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7850009fd0 tx=0x7f7850009f60 comp rx=0 tx=0).stop 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7864103dc0 msgr2=0x7f786411a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7864103dc0 0x7f786411a000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7854009150 tx=0x7f7854009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 shutdown_connections 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f784806d7f0 0x7f784806fca0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7864103dc0 0x7f786411a000 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 --2- 172.21.15.1:0/550793414 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78641095f0 0x7f786411a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.280+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 >> 172.21.15.1:0/550793414 conn(0x7f7864072d50 msgr2=0x7f78640746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.281+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 shutdown_connections 2024-02-28T16:39:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.281+0000 7f786a200700 1 -- 172.21.15.1:0/550793414 wait complete. 2024-02-28T16:39:14.285 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:39:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.441+0000 7fd53f800700 1 -- 172.21.15.1:0/3198796179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 msgr2=0x7fd538109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.441+0000 7fd53f800700 1 --2- 172.21.15.1:0/3198796179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 0x7fd538109330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd524004940 tx=0x7fd524009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.441+0000 7fd53f800700 1 -- 172.21.15.1:0/3198796179 shutdown_connections 2024-02-28T16:39:14.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.441+0000 7fd53f800700 1 --2- 172.21.15.1:0/3198796179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 0x7fd538109330 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.441+0000 7fd53f800700 1 --2- 172.21.15.1:0/3198796179 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5380fbcc0 0x7fd5380fc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.441+0000 7fd53f800700 1 -- 172.21.15.1:0/3198796179 >> 172.21.15.1:0/3198796179 conn(0x7fd5380f7a10 msgr2=0x7fd5380f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:14.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.441+0000 7fd53f800700 1 -- 172.21.15.1:0/3198796179 shutdown_connections 2024-02-28T16:39:14.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.442+0000 7fd53f800700 1 -- 172.21.15.1:0/3198796179 wait complete. 2024-02-28T16:39:14.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.442+0000 7fd53f800700 1 Processor -- start 2024-02-28T16:39:14.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.443+0000 7fd53f800700 1 -- start start 2024-02-28T16:39:14.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.443+0000 7fd53f800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5380fbcc0 0x7fd538115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.443+0000 7fd53f800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 0x7fd5381161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.443+0000 7fd53f800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd538116820 con 0x7fd5380fc5d0 2024-02-28T16:39:14.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.443+0000 7fd53f800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53810fb50 con 0x7fd5380fbcc0 2024-02-28T16:39:14.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd537e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 0x7fd5381161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:14.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd537e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 0x7fd5381161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38468/0 (socket says 172.21.15.1:38468) 2024-02-28T16:39:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd537e00700 1 -- 172.21.15.1:0/1731208887 learned_addr learned my addr 172.21.15.1:0/1731208887 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd53ce00700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5380fbcc0 0x7fd538115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd537e00700 1 -- 172.21.15.1:0/1731208887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5380fbcc0 msgr2=0x7fd538115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd537e00700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5380fbcc0 0x7fd538115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd537e00700 1 -- 172.21.15.1:0/1731208887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd524005040 con 0x7fd5380fc5d0 2024-02-28T16:39:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd537e00700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 0x7fd5381161d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd524009180 tx=0x7fd524009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.444+0000 7fd534c00700 1 -- 172.21.15.1:0/1731208887 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd52400e040 con 0x7fd5380fc5d0 2024-02-28T16:39:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.445+0000 7fd534c00700 1 -- 172.21.15.1:0/1731208887 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd524004290 con 0x7fd5380fc5d0 2024-02-28T16:39:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.445+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53810fdd0 con 0x7fd5380fc5d0 2024-02-28T16:39:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.445+0000 7fd534c00700 1 -- 172.21.15.1:0/1731208887 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd52401c6c0 con 0x7fd5380fc5d0 2024-02-28T16:39:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.445+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd538110310 con 0x7fd5380fc5d0 2024-02-28T16:39:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.446+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd538106a60 con 0x7fd5380fc5d0 2024-02-28T16:39:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.447+0000 7fd534c00700 1 -- 172.21.15.1:0/1731208887 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd524013420 con 0x7fd5380fc5d0 2024-02-28T16:39:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.448+0000 7fd534c00700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd52006d7f0 0x7fd52006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.448+0000 7fd534c00700 1 -- 172.21.15.1:0/1731208887 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fd52408e1a0 con 0x7fd5380fc5d0 2024-02-28T16:39:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.448+0000 7fd53ce00700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd52006d7f0 0x7fd52006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:14.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.449+0000 7fd53ce00700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd52006d7f0 0x7fd52006fca0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd52c004990 tx=0x7fd52c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:14.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.453+0000 7fd534c00700 1 -- 172.21.15.1:0/1731208887 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd52405b6f0 con 0x7fd5380fc5d0 2024-02-28T16:39:14.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.695+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd53804bed0 con 0x7fd52006d7f0 2024-02-28T16:39:14.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.697+0000 7fd534c00700 1 -- 172.21.15.1:0/1731208887 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd53804bed0 con 0x7fd52006d7f0 2024-02-28T16:39:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.699+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd52006d7f0 msgr2=0x7fd52006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.699+0000 7fd53f800700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd52006d7f0 0x7fd52006fca0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd52c004990 tx=0x7fd52c009ee0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.699+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 msgr2=0x7fd5381161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.700+0000 7fd53f800700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 0x7fd5381161d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd524009180 tx=0x7fd524009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:14.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.700+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 shutdown_connections 2024-02-28T16:39:14.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.700+0000 7fd53f800700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd52006d7f0 0x7fd52006fca0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.700+0000 7fd53f800700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5380fbcc0 0x7fd538115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.700+0000 7fd53f800700 1 --2- 172.21.15.1:0/1731208887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5380fc5d0 0x7fd5381161d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.700+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 >> 172.21.15.1:0/1731208887 conn(0x7fd5380f7a10 msgr2=0x7fd5380f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:14.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.700+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 shutdown_connections 2024-02-28T16:39:14.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.700+0000 7fd53f800700 1 -- 172.21.15.1:0/1731208887 wait complete. 2024-02-28T16:39:14.733 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:14 smithi066 systemd[1]: Stopping Ceph osd.3 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:39:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.853+0000 7f9ae5a00700 1 -- 172.21.15.1:0/768833470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 msgr2=0x7f9ae0105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.853+0000 7f9ae5a00700 1 --2- 172.21.15.1:0/768833470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 0x7f9ae0105540 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0004940 tx=0x7f9ad0009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.853+0000 7f9ae5a00700 1 -- 172.21.15.1:0/768833470 shutdown_connections 2024-02-28T16:39:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.853+0000 7f9ae5a00700 1 --2- 172.21.15.1:0/768833470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae0105b10 0x7f9ae00ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.853+0000 7f9ae5a00700 1 --2- 172.21.15.1:0/768833470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 0x7f9ae0105540 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.853+0000 7f9ae5a00700 1 -- 172.21.15.1:0/768833470 >> 172.21.15.1:0/768833470 conn(0x7f9ae00face0 msgr2=0x7f9ae00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.853+0000 7f9ae5a00700 1 -- 172.21.15.1:0/768833470 shutdown_connections 2024-02-28T16:39:14.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.853+0000 7f9ae5a00700 1 -- 172.21.15.1:0/768833470 wait complete. 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.854+0000 7f9ae5a00700 1 Processor -- start 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.854+0000 7f9ae5a00700 1 -- start start 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9ae5a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 0x7f9ae0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9ae5a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae0105b10 0x7f9ae01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9ae5a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae01168a0 con 0x7f9ae0105170 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9ae5a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae010fb40 con 0x7f9ae0105b10 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9adea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 0x7f9ae0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9adea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 0x7f9ae0115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38480/0 (socket says 172.21.15.1:38480) 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9adea00700 1 -- 172.21.15.1:0/2272188231 learned_addr learned my addr 172.21.15.1:0/2272188231 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9adea00700 1 -- 172.21.15.1:0/2272188231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae0105b10 msgr2=0x7f9ae01161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:14.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9adea00700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae0105b10 0x7f9ae01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.855+0000 7f9adea00700 1 -- 172.21.15.1:0/2272188231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad0005040 con 0x7f9ae0105170 2024-02-28T16:39:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.856+0000 7f9adea00700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 0x7f9ae0115c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0009150 tx=0x7f9ad0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:14.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.856+0000 7f9ad6a00700 1 -- 172.21.15.1:0/2272188231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad000e040 con 0x7f9ae0105170 2024-02-28T16:39:14.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.856+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae010fdc0 con 0x7f9ae0105170 2024-02-28T16:39:14.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.856+0000 7f9ad6a00700 1 -- 172.21.15.1:0/2272188231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ad000bbf0 con 0x7f9ae0105170 2024-02-28T16:39:14.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.856+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae0110300 con 0x7f9ae0105170 2024-02-28T16:39:14.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.856+0000 7f9ad6a00700 1 -- 172.21.15.1:0/2272188231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad0013ea0 con 0x7f9ae0105170 2024-02-28T16:39:14.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.857+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae00fe900 con 0x7f9ae0105170 2024-02-28T16:39:14.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.858+0000 7f9ad6a00700 1 -- 172.21.15.1:0/2272188231 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9ad000bd60 con 0x7f9ae0105170 2024-02-28T16:39:14.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.859+0000 7f9ad6a00700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ac406d7f0 0x7f9ac406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:14.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.859+0000 7f9ad6a00700 1 -- 172.21.15.1:0/2272188231 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f9ad008e210 con 0x7f9ae0105170 2024-02-28T16:39:14.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.859+0000 7f9ade000700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ac406d7f0 0x7f9ac406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:14.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.860+0000 7f9ade000700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ac406d7f0 0x7f9ac406fca0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9acc006fd0 tx=0x7f9acc006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:14.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:14.864+0000 7f9ad6a00700 1 -- 172.21.15.1:0/2272188231 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9ad005b620 con 0x7f9ae0105170 2024-02-28T16:39:14.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:14 smithi066 ceph-mon[85781]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:14.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:14 smithi066 ceph-mon[85781]: osd.3 marked itself down and dead 2024-02-28T16:39:14.998 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:14 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[41749]: 2024-02-28T16:39:14.743+0000 7f542f800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:39:14.998 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:14 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[41749]: 2024-02-28T16:39:14.743+0000 7f542f800700 -1 osd.3 63 *** Got signal Terminated *** 2024-02-28T16:39:14.998 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:14 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[41749]: 2024-02-28T16:39:14.743+0000 7f542f800700 -1 osd.3 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T16:39:15.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:14 smithi001 ceph-mon[112940]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:15.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:14 smithi001 ceph-mon[112940]: osd.3 marked itself down and dead 2024-02-28T16:39:15.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.098+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ae0111290 con 0x7f9ac406d7f0 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.129+0000 7f9ad6a00700 1 -- 172.21.15.1:0/2272188231 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f9ae0111290 con 0x7f9ac406d7f0 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (5m) 16s ago 19m 14.1M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (2m) 16s ago 19m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (118s) 112s ago 18m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (4m) 16s ago 19m 74.9M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (12m) 16s ago 12m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (12m) 16s ago 12m 15.0M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:39:15.117 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (12m) 112s ago 12m 14.3M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (12m) 112s ago 12m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (7m) 16s ago 21m 503M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (6m) 112s ago 18m 449M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (3m) 16s ago 21m 46.7M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (3m) 112s ago 18m 35.0M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (6m) 16s ago 19m 8108k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (6m) 112s ago 18m 7612k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (86s) 16s ago 17m 80.8M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (55s) 16s ago 16m 88.0M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (24s) 16s ago 16m 11.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (15m) 112s ago 15m 222M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 38fe956daf9a 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (14m) 112s ago 14m 214M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:39:15.118 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (14m) 112s ago 14m 189M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:39:15.119 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (5m) 16s ago 18m 52.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ac406d7f0 msgr2=0x7f9ac406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ac406d7f0 0x7f9ac406fca0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9acc006fd0 tx=0x7f9acc006db0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 msgr2=0x7f9ae0115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 0x7f9ae0115c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0009150 tx=0x7f9ad0009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 shutdown_connections 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ac406d7f0 0x7f9ac406fca0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ae0105170 0x7f9ae0115c80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 --2- 172.21.15.1:0/2272188231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9ae0105b10 0x7f9ae01161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 >> 172.21.15.1:0/2272188231 conn(0x7f9ae00face0 msgr2=0x7f9ae00fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 shutdown_connections 2024-02-28T16:39:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.132+0000 7f9ae5a00700 1 -- 172.21.15.1:0/2272188231 wait complete. 2024-02-28T16:39:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.273+0000 7f8194200700 1 -- 172.21.15.1:0/457327160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 msgr2=0x7f818c109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.273+0000 7f8194200700 1 --2- 172.21.15.1:0/457327160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 0x7f818c109260 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8178004970 tx=0x7f8178009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.273+0000 7f8194200700 1 -- 172.21.15.1:0/457327160 shutdown_connections 2024-02-28T16:39:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.273+0000 7f8194200700 1 --2- 172.21.15.1:0/457327160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 0x7f818c109260 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.273+0000 7f8194200700 1 --2- 172.21.15.1:0/457327160 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f818c070330 0x7f818c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.273+0000 7f8194200700 1 -- 172.21.15.1:0/457327160 >> 172.21.15.1:0/457327160 conn(0x7f818c0f8ae0 msgr2=0x7f818c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.273+0000 7f8194200700 1 -- 172.21.15.1:0/457327160 shutdown_connections 2024-02-28T16:39:15.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.273+0000 7f8194200700 1 -- 172.21.15.1:0/457327160 wait complete. 2024-02-28T16:39:15.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.274+0000 7f8194200700 1 Processor -- start 2024-02-28T16:39:15.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.274+0000 7f8194200700 1 -- start start 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.274+0000 7f8194200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f818c070330 0x7f818c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.274+0000 7f8194200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 0x7f818c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.274+0000 7f8194200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c116790 con 0x7f818c070c40 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.274+0000 7f8194200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c10fae0 con 0x7f818c070330 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.274+0000 7f8190e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 0x7f818c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.275+0000 7f8191800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f818c070330 0x7f818c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.275+0000 7f8190e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 0x7f818c116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38508/0 (socket says 172.21.15.1:38508) 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.275+0000 7f8190e00700 1 -- 172.21.15.1:0/601062661 learned_addr learned my addr 172.21.15.1:0/601062661 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.275+0000 7f8190e00700 1 -- 172.21.15.1:0/601062661 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f818c070330 msgr2=0x7f818c115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.275+0000 7f8190e00700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f818c070330 0x7f818c115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.275+0000 7f8190e00700 1 -- 172.21.15.1:0/601062661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8178005040 con 0x7f818c070c40 2024-02-28T16:39:15.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.275+0000 7f8190e00700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 0x7f818c116140 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8178005010 tx=0x7f8178009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.276+0000 7f8189600700 1 -- 172.21.15.1:0/601062661 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817800e040 con 0x7f818c070c40 2024-02-28T16:39:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.276+0000 7f8189600700 1 -- 172.21.15.1:0/601062661 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8178004290 con 0x7f818c070c40 2024-02-28T16:39:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.276+0000 7f8189600700 1 -- 172.21.15.1:0/601062661 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817801c3f0 con 0x7f818c070c40 2024-02-28T16:39:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.276+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818c10fd60 con 0x7f818c070c40 2024-02-28T16:39:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.276+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818c110250 con 0x7f818c070c40 2024-02-28T16:39:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.277+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f818c106920 con 0x7f818c070c40 2024-02-28T16:39:15.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.278+0000 7f8189600700 1 -- 172.21.15.1:0/601062661 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8178013420 con 0x7f818c070c40 2024-02-28T16:39:15.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.279+0000 7f8189600700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f817406d7f0 0x7f817406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:15.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.279+0000 7f8189600700 1 -- 172.21.15.1:0/601062661 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f817808e0f0 con 0x7f818c070c40 2024-02-28T16:39:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.279+0000 7f8191800700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f817406d7f0 0x7f817406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.280+0000 7f8191800700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f817406d7f0 0x7f817406fca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8180009b80 tx=0x7f818000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:15.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.283+0000 7f8189600700 1 -- 172.21.15.1:0/601062661 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f81780579d0 con 0x7f818c070c40 2024-02-28T16:39:15.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.590+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f818c04bed0 con 0x7f818c070c40 2024-02-28T16:39:15.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.592+0000 7f8189600700 1 -- 172.21.15.1:0/601062661 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f817805b090 con 0x7f818c070c40 2024-02-28T16:39:15.579 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:39:15.579 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:39:15.580 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:39:15.581 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6, 2024-02-28T16:39:15.581 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 7 2024-02-28T16:39:15.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:39:15.581 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:39:15.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.594+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f817406d7f0 msgr2=0x7f817406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:15.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.594+0000 7f8194200700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f817406d7f0 0x7f817406fca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8180009b80 tx=0x7f818000b040 comp rx=0 tx=0).stop 2024-02-28T16:39:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.594+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 msgr2=0x7f818c116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.594+0000 7f8194200700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 0x7f818c116140 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8178005010 tx=0x7f8178009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.595+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 shutdown_connections 2024-02-28T16:39:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.595+0000 7f8194200700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f817406d7f0 0x7f817406fca0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.595+0000 7f8194200700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f818c070330 0x7f818c115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.595+0000 7f8194200700 1 --2- 172.21.15.1:0/601062661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f818c070c40 0x7f818c116140 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.595+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 >> 172.21.15.1:0/601062661 conn(0x7f818c0f8ae0 msgr2=0x7f818c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.595+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 shutdown_connections 2024-02-28T16:39:15.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.595+0000 7f8194200700 1 -- 172.21.15.1:0/601062661 wait complete. 2024-02-28T16:39:15.654 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:15 smithi066 podman[95146]: 2024-02-28 16:39:15.340402109 +0000 UTC m=+0.715500357 container died 38fe956daf9a70df1cd0e2687b6fbf70fbaeb7fc33686aa1bf6c5db2daa94b18 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c0f02e6, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-28T16:39:15.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.751+0000 7f8898e00700 1 -- 172.21.15.1:0/155948073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 msgr2=0x7f8894109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:15.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.751+0000 7f8898e00700 1 --2- 172.21.15.1:0/155948073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 0x7f8894109750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8884004940 tx=0x7f8884009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:15.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.751+0000 7f8898e00700 1 -- 172.21.15.1:0/155948073 shutdown_connections 2024-02-28T16:39:15.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.751+0000 7f8898e00700 1 --2- 172.21.15.1:0/155948073 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8894109d20 0x7f88941033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.751+0000 7f8898e00700 1 --2- 172.21.15.1:0/155948073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 0x7f8894109750 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.751+0000 7f8898e00700 1 -- 172.21.15.1:0/155948073 >> 172.21.15.1:0/155948073 conn(0x7f8894072b50 msgr2=0x7f8894074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:15.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.752+0000 7f8898e00700 1 -- 172.21.15.1:0/155948073 shutdown_connections 2024-02-28T16:39:15.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.752+0000 7f8898e00700 1 -- 172.21.15.1:0/155948073 wait complete. 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8898e00700 1 Processor -- start 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8898e00700 1 -- start start 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8898e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 0x7f8894119e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8898e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8894109d20 0x7f889411a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8898e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889411a9c0 con 0x7f8894109380 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8898e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8894113d10 con 0x7f8894109d20 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8893400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 0x7f8894119e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8893400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 0x7f8894119e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38532/0 (socket says 172.21.15.1:38532) 2024-02-28T16:39:15.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.753+0000 7f8893400700 1 -- 172.21.15.1:0/2934289568 learned_addr learned my addr 172.21.15.1:0/2934289568 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:15.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.754+0000 7f8892a00700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8894109d20 0x7f889411a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:15.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.754+0000 7f8893400700 1 -- 172.21.15.1:0/2934289568 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8894109d20 msgr2=0x7f889411a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:15.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.754+0000 7f8893400700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8894109d20 0x7f889411a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:15.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.754+0000 7f8893400700 1 -- 172.21.15.1:0/2934289568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8884005040 con 0x7f8894109380 2024-02-28T16:39:15.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.754+0000 7f8893400700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 0x7f8894119e30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8884009150 tx=0x7f8884009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:15.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.754+0000 7f888b400700 1 -- 172.21.15.1:0/2934289568 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f888400e040 con 0x7f8894109380 2024-02-28T16:39:15.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.755+0000 7f888b400700 1 -- 172.21.15.1:0/2934289568 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f888400bbf0 con 0x7f8894109380 2024-02-28T16:39:15.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.755+0000 7f888b400700 1 -- 172.21.15.1:0/2934289568 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f888401c630 con 0x7f8894109380 2024-02-28T16:39:15.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.755+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8894113f90 con 0x7f8894109380 2024-02-28T16:39:15.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.755+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8894114550 con 0x7f8894109380 2024-02-28T16:39:15.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.756+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889405f5c0 con 0x7f8894109380 2024-02-28T16:39:15.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.757+0000 7f888b400700 1 -- 172.21.15.1:0/2934289568 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f888400bd60 con 0x7f8894109380 2024-02-28T16:39:15.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.758+0000 7f888b400700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887806d7f0 0x7f887806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:15.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.758+0000 7f888b400700 1 -- 172.21.15.1:0/2934289568 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f888408e1f0 con 0x7f8894109380 2024-02-28T16:39:15.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.758+0000 7f8892a00700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887806d7f0 0x7f887806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:15.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.759+0000 7f8892a00700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887806d7f0 0x7f887806fca0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8894115710 tx=0x7f888000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:15.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:15.763+0000 7f888b400700 1 -- 172.21.15.1:0/2934289568 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8884057b50 con 0x7f8894109380 2024-02-28T16:39:15.950 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:15 smithi066 podman[95146]: 2024-02-28 16:39:15.786818181 +0000 UTC m=+1.161916453 container cleanup 38fe956daf9a70df1cd0e2687b6fbf70fbaeb7fc33686aa1bf6c5db2daa94b18 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c0f02e6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-28T16:39:15.950 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:15 smithi066 bash[95146]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3 2024-02-28T16:39:15.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:15 smithi066 ceph-mon[85781]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:15.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:15 smithi066 ceph-mon[85781]: pgmap v125: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:15.950 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:15 smithi066 ceph-mon[85781]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:39:15.951 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:15 smithi066 ceph-mon[85781]: osdmap e64: 6 total, 5 up, 6 in 2024-02-28T16:39:15.951 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:15 smithi066 ceph-mon[85781]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:15.951 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:15 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/601062661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:16.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:15 smithi001 ceph-mon[112940]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:16.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:15 smithi001 ceph-mon[112940]: pgmap v125: 65 pgs: 65 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-02-28T16:39:16.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:15 smithi001 ceph-mon[112940]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:39:16.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:15 smithi001 ceph-mon[112940]: osdmap e64: 6 total, 5 up, 6 in 2024-02-28T16:39:16.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:15 smithi001 ceph-mon[112940]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:16.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:15 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/601062661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:16.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.049+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8894115450 con 0x7f8894109380 2024-02-28T16:39:16.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.050+0000 7f888b400700 1 -- 172.21.15.1:0/2934289568 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7f888405b210 con 0x7f8894109380 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:39:16.041 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:39:16.042 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:39:16.043 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:39:16.044 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:39:16.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:39:16.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:39:16.044 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:39:16.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:39:16.044 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:39:16.044 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.052+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887806d7f0 msgr2=0x7f887806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.052+0000 7f8898e00700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887806d7f0 0x7f887806fca0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8894115710 tx=0x7f888000b040 comp rx=0 tx=0).stop 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 msgr2=0x7f8894119e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 0x7f8894119e30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8884009150 tx=0x7f8884009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 shutdown_connections 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887806d7f0 0x7f887806fca0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8894109380 0x7f8894119e30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 --2- 172.21.15.1:0/2934289568 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8894109d20 0x7f889411a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 >> 172.21.15.1:0/2934289568 conn(0x7f8894072b50 msgr2=0x7f88940756b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:16.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 shutdown_connections 2024-02-28T16:39:16.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.053+0000 7f8898e00700 1 -- 172.21.15.1:0/2934289568 wait complete. 2024-02-28T16:39:16.046 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:39:16.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.197+0000 7f2e0cc00700 1 -- 172.21.15.1:0/62885511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 msgr2=0x7f2e080fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.197+0000 7f2e0cc00700 1 --2- 172.21.15.1:0/62885511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 0x7f2e080fc1a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009ce0 tx=0x7f2df8004d10 comp rx=0 tx=0).stop 2024-02-28T16:39:16.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.197+0000 7f2e0cc00700 1 -- 172.21.15.1:0/62885511 shutdown_connections 2024-02-28T16:39:16.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.197+0000 7f2e0cc00700 1 --2- 172.21.15.1:0/62885511 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e080fc6e0 0x7f2e08109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.197+0000 7f2e0cc00700 1 --2- 172.21.15.1:0/62885511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 0x7f2e080fc1a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.197+0000 7f2e0cc00700 1 -- 172.21.15.1:0/62885511 >> 172.21.15.1:0/62885511 conn(0x7f2e08071d90 msgr2=0x7f2e08072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:16.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.197+0000 7f2e0cc00700 1 -- 172.21.15.1:0/62885511 shutdown_connections 2024-02-28T16:39:16.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.197+0000 7f2e0cc00700 1 -- 172.21.15.1:0/62885511 wait complete. 2024-02-28T16:39:16.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.198+0000 7f2e0cc00700 1 Processor -- start 2024-02-28T16:39:16.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.199+0000 7f2e0cc00700 1 -- start start 2024-02-28T16:39:16.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.199+0000 7f2e0cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 0x7f2e08115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:16.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.199+0000 7f2e0cc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e080fc6e0 0x7f2e08116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:16.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.199+0000 7f2e0cc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e08116860 con 0x7f2e080fbdd0 2024-02-28T16:39:16.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.199+0000 7f2e0cc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0810fb70 con 0x7f2e080fc6e0 2024-02-28T16:39:16.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.199+0000 7f2e05e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 0x7f2e08115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:16.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.199+0000 7f2e05e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 0x7f2e08115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38548/0 (socket says 172.21.15.1:38548) 2024-02-28T16:39:16.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.199+0000 7f2e05e00700 1 -- 172.21.15.1:0/936206974 learned_addr learned my addr 172.21.15.1:0/936206974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:16.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.200+0000 7f2e05e00700 1 -- 172.21.15.1:0/936206974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e080fc6e0 msgr2=0x7f2e08116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.200+0000 7f2e05e00700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e080fc6e0 0x7f2e08116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.200+0000 7f2e05e00700 1 -- 172.21.15.1:0/936206974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df8009940 con 0x7f2e080fbdd0 2024-02-28T16:39:16.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.200+0000 7f2e05e00700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 0x7f2e08115cd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2df8004b70 tx=0x7f2df8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.200+0000 7f2dfea00700 1 -- 172.21.15.1:0/936206974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2df800e040 con 0x7f2e080fbdd0 2024-02-28T16:39:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.200+0000 7f2dfea00700 1 -- 172.21.15.1:0/936206974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2df80128e0 con 0x7f2e080fbdd0 2024-02-28T16:39:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.200+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0810fdf0 con 0x7f2e080fbdd0 2024-02-28T16:39:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.200+0000 7f2dfea00700 1 -- 172.21.15.1:0/936206974 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2df801c830 con 0x7f2e080fbdd0 2024-02-28T16:39:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.201+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e08110330 con 0x7f2e080fbdd0 2024-02-28T16:39:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.202+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e08106b70 con 0x7f2e080fbdd0 2024-02-28T16:39:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.203+0000 7f2dfea00700 1 -- 172.21.15.1:0/936206974 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2df8012a50 con 0x7f2e080fbdd0 2024-02-28T16:39:16.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.204+0000 7f2dfea00700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2dec06d7f0 0x7f2dec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:16.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.204+0000 7f2dfea00700 1 -- 172.21.15.1:0/936206974 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f2df808eac0 con 0x7f2e080fbdd0 2024-02-28T16:39:16.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.204+0000 7f2dfd400700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2dec06d7f0 0x7f2dec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:16.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.205+0000 7f2dfd400700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2dec06d7f0 0x7f2dec06fca0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e08111570 tx=0x7f2df0009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:16.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.209+0000 7f2dfea00700 1 -- 172.21.15.1:0/936206974 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2df805bed0 con 0x7f2e080fbdd0 2024-02-28T16:39:16.248 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:15 smithi066 podman[95164]: 2024-02-28 16:39:15.962158719 +0000 UTC m=+0.604858373 container remove 38fe956daf9a70df1cd0e2687b6fbf70fbaeb7fc33686aa1bf6c5db2daa94b18 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-c0f02e6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-28T16:39:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.443+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e0804bf70 con 0x7f2dec06d7f0 2024-02-28T16:39:16.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.447+0000 7f2dfea00700 1 -- 172.21.15.1:0/936206974 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2df000f040 con 0x7f2dec06d7f0 2024-02-28T16:39:16.434 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "crash" 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "9/21 daemons upgraded", 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading osd daemons", 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:39:16.435 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:39:16.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.450+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2dec06d7f0 msgr2=0x7f2dec06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.450+0000 7f2e0cc00700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2dec06d7f0 0x7f2dec06fca0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e08111570 tx=0x7f2df0009ce0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.450+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 msgr2=0x7f2e08115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.450+0000 7f2e0cc00700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 0x7f2e08115cd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2df8004b70 tx=0x7f2df8004d10 comp rx=0 tx=0).stop 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.451+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 shutdown_connections 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.451+0000 7f2e0cc00700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2dec06d7f0 0x7f2dec06fca0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.451+0000 7f2e0cc00700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e080fbdd0 0x7f2e08115cd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.451+0000 7f2e0cc00700 1 --2- 172.21.15.1:0/936206974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e080fc6e0 0x7f2e08116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.451+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 >> 172.21.15.1:0/936206974 conn(0x7f2e08071d90 msgr2=0x7f2e080faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.451+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 shutdown_connections 2024-02-28T16:39:16.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.451+0000 7f2e0cc00700 1 -- 172.21.15.1:0/936206974 wait complete. 2024-02-28T16:39:16.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.603+0000 7f923d400700 1 -- 172.21.15.1:0/1027648279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 msgr2=0x7f9238070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.603+0000 7f923d400700 1 --2- 172.21.15.1:0/1027648279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 0x7f9238070700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9228004970 tx=0x7f9228009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.603+0000 7f923d400700 1 -- 172.21.15.1:0/1027648279 shutdown_connections 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.603+0000 7f923d400700 1 --2- 172.21.15.1:0/1027648279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9238070c40 0x7f9238109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.603+0000 7f923d400700 1 --2- 172.21.15.1:0/1027648279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 0x7f9238070700 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.603+0000 7f923d400700 1 -- 172.21.15.1:0/1027648279 >> 172.21.15.1:0/1027648279 conn(0x7f92380f8ae0 msgr2=0x7f92380faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.603+0000 7f923d400700 1 -- 172.21.15.1:0/1027648279 shutdown_connections 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.603+0000 7f923d400700 1 -- 172.21.15.1:0/1027648279 wait complete. 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.604+0000 7f923d400700 1 Processor -- start 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.604+0000 7f923d400700 1 -- start start 2024-02-28T16:39:16.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f923d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 0x7f9238115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f923d400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9238070c40 0x7f9238116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f923d400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92381167b0 con 0x7f9238070330 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f923d400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923810fae0 con 0x7f9238070c40 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f9236800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 0x7f9238115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f9236800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 0x7f9238115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38566/0 (socket says 172.21.15.1:38566) 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f9236800700 1 -- 172.21.15.1:0/3008438140 learned_addr learned my addr 172.21.15.1:0/3008438140 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f922de00700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9238070c40 0x7f9238116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f9236800700 1 -- 172.21.15.1:0/3008438140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9238070c40 msgr2=0x7f9238116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f9236800700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9238070c40 0x7f9238116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.605+0000 7f9236800700 1 -- 172.21.15.1:0/3008438140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9228005040 con 0x7f9238070330 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.606+0000 7f9236800700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 0x7f9238115c20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9228009150 tx=0x7f9228009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.606+0000 7f922f400700 1 -- 172.21.15.1:0/3008438140 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922800e040 con 0x7f9238070330 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.606+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923810fd60 con 0x7f9238070330 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.606+0000 7f922f400700 1 -- 172.21.15.1:0/3008438140 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9228004290 con 0x7f9238070330 2024-02-28T16:39:16.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.606+0000 7f922f400700 1 -- 172.21.15.1:0/3008438140 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922801c630 con 0x7f9238070330 2024-02-28T16:39:16.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.606+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92381102a0 con 0x7f9238070330 2024-02-28T16:39:16.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.607+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9238106920 con 0x7f9238070330 2024-02-28T16:39:16.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.608+0000 7f922f400700 1 -- 172.21.15.1:0/3008438140 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9228013980 con 0x7f9238070330 2024-02-28T16:39:16.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.609+0000 7f922f400700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f921c06d7f0 0x7f921c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:16.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.609+0000 7f922f400700 1 -- 172.21.15.1:0/3008438140 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f922808e160 con 0x7f9238070330 2024-02-28T16:39:16.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.609+0000 7f922de00700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f921c06d7f0 0x7f921c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:16.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.610+0000 7f922de00700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f921c06d7f0 0x7f921c06fca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9220007fd0 tx=0x7f9220009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:16.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.614+0000 7f922f400700 1 -- 172.21.15.1:0/3008438140 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f922805b5f0 con 0x7f9238070330 2024-02-28T16:39:16.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.919+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f923804bed0 con 0x7f9238070330 2024-02-28T16:39:16.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.919+0000 7f922f400700 1 -- 172.21.15.1:0/3008438140 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1301 (secure 0 0 0) 0x7f922805b180 con 0x7f9238070330 2024-02-28T16:39:16.907 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; Reduced data availability: 3 pgs peering; Degraded data redundancy: 42/372 objects degraded (11.290%), 9 pgs degraded 2024-02-28T16:39:16.907 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:39:16.907 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:39:16.907 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-28T16:39:16.907 INFO:teuthology.orchestra.run.smithi001.stdout: osd.3 (root=default,host=smithi066) is down 2024-02-28T16:39:16.907 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs peering 2024-02-28T16:39:16.907 INFO:teuthology.orchestra.run.smithi001.stdout: pg 1.0 is stuck peering for 3m, current state peering, last acting [0,1] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.5 is stuck peering for 3m, current state peering, last acting [0,4] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 3.15 is stuck peering for 5m, current state peering, last acting [0,4] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 42/372 objects degraded (11.290%), 9 pgs degraded 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.7 is active+undersized+degraded, acting [4,2] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.8 is active+undersized+degraded, acting [5,0] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.14 is active+undersized+degraded, acting [4,5] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.1a is active+undersized+degraded, acting [4,5] 2024-02-28T16:39:16.908 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.1d is active+undersized+degraded, acting [5,0] 2024-02-28T16:39:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.922+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f921c06d7f0 msgr2=0x7f921c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.922+0000 7f923d400700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f921c06d7f0 0x7f921c06fca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9220007fd0 tx=0x7f9220009040 comp rx=0 tx=0).stop 2024-02-28T16:39:16.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.922+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 msgr2=0x7f9238115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:16.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.922+0000 7f923d400700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 0x7f9238115c20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9228009150 tx=0x7f9228009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:16.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.923+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 shutdown_connections 2024-02-28T16:39:16.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.923+0000 7f923d400700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f921c06d7f0 0x7f921c06fca0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.923+0000 7f923d400700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9238070330 0x7f9238115c20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.923+0000 7f923d400700 1 --2- 172.21.15.1:0/3008438140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9238070c40 0x7f9238116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:16.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.923+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 >> 172.21.15.1:0/3008438140 conn(0x7f92380f8ae0 msgr2=0x7f9238103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:16.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.923+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 shutdown_connections 2024-02-28T16:39:16.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:16.923+0000 7f923d400700 1 -- 172.21.15.1:0/3008438140 wait complete. 2024-02-28T16:39:17.071 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:16 smithi066 ceph-mon[85781]: osdmap e65: 6 total, 5 up, 6 in 2024-02-28T16:39:17.071 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:16 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2934289568' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:39:17.071 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:16 smithi066 ceph-mon[85781]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:17.071 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:16 smithi066 podman[95256]: 2024-02-28 16:39:16.747127862 +0000 UTC m=+0.034169276 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:39:17.071 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:16 smithi066 podman[95256]: 2024-02-28 16:39:16.921529181 +0000 UTC m=+0.208570632 container create d5fcb98bf4e07d45c821ae982f6da4190695174b797ac5b2a0eeeec167919811 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, RELEASE=main-1fb5d10, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-28T16:39:17.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:16 smithi001 ceph-mon[112940]: osdmap e65: 6 total, 5 up, 6 in 2024-02-28T16:39:17.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:16 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2934289568' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:39:17.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:16 smithi001 ceph-mon[112940]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:17.497 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:17 smithi066 podman[95256]: 2024-02-28 16:39:17.243911296 +0000 UTC m=+0.530952804 container init d5fcb98bf4e07d45c821ae982f6da4190695174b797ac5b2a0eeeec167919811 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-28T16:39:17.497 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:17 smithi066 podman[95256]: 2024-02-28 16:39:17.256613953 +0000 UTC m=+0.543655378 container start d5fcb98bf4e07d45c821ae982f6da4190695174b797ac5b2a0eeeec167919811 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, GIT_CLEAN=True) 2024-02-28T16:39:17.497 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:17 smithi066 podman[95256]: 2024-02-28 16:39:17.332505066 +0000 UTC m=+0.619546544 container attach d5fcb98bf4e07d45c821ae982f6da4190695174b797ac5b2a0eeeec167919811 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10) 2024-02-28T16:39:17.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:17 smithi066 ceph-mon[85781]: pgmap v128: 65 pgs: 9 active+undersized+degraded, 7 peering, 3 stale+active+clean, 9 active+undersized, 37 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 42/372 objects degraded (11.290%) 2024-02-28T16:39:17.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:17 smithi066 ceph-mon[85781]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-02-28T16:39:17.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:17 smithi066 ceph-mon[85781]: Health check failed: Degraded data redundancy: 42/372 objects degraded (11.290%), 9 pgs degraded (PG_DEGRADED) 2024-02-28T16:39:17.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:17 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3008438140' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:39:17.997 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:17 smithi066 podman[95256]: 2024-02-28 16:39:17.710644483 +0000 UTC m=+0.997685954 container died d5fcb98bf4e07d45c821ae982f6da4190695174b797ac5b2a0eeeec167919811 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:39:18.177 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:17 smithi001 ceph-mon[112940]: pgmap v128: 65 pgs: 9 active+undersized+degraded, 7 peering, 3 stale+active+clean, 9 active+undersized, 37 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 42/372 objects degraded (11.290%) 2024-02-28T16:39:18.178 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:17 smithi001 ceph-mon[112940]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-02-28T16:39:18.178 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:17 smithi001 ceph-mon[112940]: Health check failed: Degraded data redundancy: 42/372 objects degraded (11.290%), 9 pgs degraded (PG_DEGRADED) 2024-02-28T16:39:18.178 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:17 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3008438140' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:39:18.747 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:18 smithi066 podman[95282]: 2024-02-28 16:39:18.473139972 +0000 UTC m=+0.743927672 container remove d5fcb98bf4e07d45c821ae982f6da4190695174b797ac5b2a0eeeec167919811 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-28T16:39:18.747 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:18 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-28T16:39:18.748 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:18 smithi066 systemd[1]: Stopped Ceph osd.3 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:39:18.748 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:18 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.3.service: Consumed 1min 57.950s CPU time. 2024-02-28T16:39:19.247 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:18 smithi066 systemd[1]: Starting Ceph osd.3 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:39:19.572 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:19 smithi066 podman[95384]: 2024-02-28 16:39:19.233710149 +0000 UTC m=+0.035911284 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:39:19.572 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:19 smithi066 podman[95384]: 2024-02-28 16:39:19.353363979 +0000 UTC m=+0.155565151 container create 1c7281a323fb116cfdeb561bafbf726d9056b6549636d74ae0fac9aa7ee1814a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-1fb5d10) 2024-02-28T16:39:19.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:19 smithi066 ceph-mon[85781]: pgmap v129: 65 pgs: 16 active+undersized+degraded, 7 peering, 15 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:39:19.997 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:19 smithi066 podman[95384]: 2024-02-28 16:39:19.665161051 +0000 UTC m=+0.467362266 container init 1c7281a323fb116cfdeb561bafbf726d9056b6549636d74ae0fac9aa7ee1814a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-28T16:39:19.997 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:19 smithi066 podman[95384]: 2024-02-28 16:39:19.678286974 +0000 UTC m=+0.480488118 container start 1c7281a323fb116cfdeb561bafbf726d9056b6549636d74ae0fac9aa7ee1814a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-02-28T16:39:19.997 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:19 smithi066 podman[95384]: 2024-02-28 16:39:19.756037506 +0000 UTC m=+0.558238728 container attach 1c7281a323fb116cfdeb561bafbf726d9056b6549636d74ae0fac9aa7ee1814a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-02-28T16:39:20.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:19 smithi001 ceph-mon[112940]: pgmap v129: 65 pgs: 16 active+undersized+degraded, 7 peering, 15 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:39:20.839 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate[95399]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-28T16:39:20.839 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate[95399]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 bash[95384]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 bash[95384]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 bash[95384]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 bash[95384]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 bash[95384]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 bash[95384]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 bash[95384]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate[95399]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate[95399]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate[95399]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate[95399]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-28T16:39:20.840 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate[95399]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-28T16:39:20.841 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:20 smithi066 podman[95534]: 2024-02-28 16:39:20.683836418 +0000 UTC m=+0.038295553 container died 1c7281a323fb116cfdeb561bafbf726d9056b6549636d74ae0fac9aa7ee1814a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=main-1fb5d10) 2024-02-28T16:39:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:20 smithi066 ceph-mon[85781]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-02-28T16:39:21.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:20 smithi001 ceph-mon[112940]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-02-28T16:39:21.747 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:21 smithi066 podman[95534]: 2024-02-28 16:39:21.426148638 +0000 UTC m=+0.780607813 container remove 1c7281a323fb116cfdeb561bafbf726d9056b6549636d74ae0fac9aa7ee1814a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10) 2024-02-28T16:39:22.049 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:21 smithi066 ceph-mon[85781]: pgmap v130: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:22.050 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:21 smithi066 podman[95598]: 2024-02-28 16:39:21.720679262 +0000 UTC m=+0.041054382 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:39:22.050 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:21 smithi066 podman[95598]: 2024-02-28 16:39:21.893596059 +0000 UTC m=+0.213971185 container create 4953ba0214066081e61c1b229adf3c4ce493878ad1d546f502fe9c8fbad7a3b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-28T16:39:22.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:21 smithi001 ceph-mon[112940]: pgmap v130: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:22.498 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:22 smithi066 podman[95598]: 2024-02-28 16:39:22.205550665 +0000 UTC m=+0.525925808 container init 4953ba0214066081e61c1b229adf3c4ce493878ad1d546f502fe9c8fbad7a3b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-28T16:39:22.498 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:22 smithi066 podman[95598]: 2024-02-28 16:39:22.217265349 +0000 UTC m=+0.537640476 container start 4953ba0214066081e61c1b229adf3c4ce493878ad1d546f502fe9c8fbad7a3b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-1fb5d10) 2024-02-28T16:39:22.498 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:22 smithi066 ceph-osd[95617]: -- 172.21.15.66:0/2932193921 <== mon.1 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e0796d680 con 0x559e07936c00 2024-02-28T16:39:22.498 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:22 smithi066 bash[95598]: 4953ba0214066081e61c1b229adf3c4ce493878ad1d546f502fe9c8fbad7a3b6 2024-02-28T16:39:22.498 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:22 smithi066 systemd[1]: Started Ceph osd.3 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:39:23.246 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:22 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T16:39:22.788+0000 7f126c2c27c0 -1 Falling back to public interface 2024-02-28T16:39:23.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:23.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:23.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:39:23.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:23.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:23.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:39:24.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:24 smithi066 ceph-mon[85781]: pgmap v131: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:24.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:24 smithi001 ceph-mon[112940]: pgmap v131: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:26 smithi066 ceph-mon[85781]: pgmap v132: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:39:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:26 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 74/372 objects degraded (19.892%), 19 pgs degraded (PG_DEGRADED) 2024-02-28T16:39:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:26 smithi001 ceph-mon[112940]: pgmap v132: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:39:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:26 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 74/372 objects degraded (19.892%), 19 pgs degraded (PG_DEGRADED) 2024-02-28T16:39:28.247 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:27 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T16:39:27.907+0000 7f126c2c27c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-28T16:39:28.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:27 smithi066 ceph-mon[85781]: pgmap v133: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:28.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:27 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:28.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:27 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:28.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:27 smithi001 ceph-mon[112940]: pgmap v133: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:28.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:27 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:28.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:27 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:28.747 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:28 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T16:39:28.340+0000 7f126c2c27c0 -1 osd.3 63 log_to_monitors true 2024-02-28T16:39:29.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:28 smithi066 ceph-mon[85781]: from='osd.3 [v2:172.21.15.66:6800/2991662728,v1:172.21.15.66:6801/2991662728]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-28T16:39:29.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:28 smithi066 ceph-mon[85781]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-28T16:39:29.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:28 smithi001 ceph-mon[112940]: from='osd.3 [v2:172.21.15.66:6800/2991662728,v1:172.21.15.66:6801/2991662728]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-28T16:39:29.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:28 smithi001 ceph-mon[112940]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-28T16:39:30.196 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 16:39:29 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T16:39:29.984+0000 7f1256600700 -1 osd.3 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-28T16:39:30.197 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:29 smithi066 ceph-mon[85781]: pgmap v134: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:30.197 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:29 smithi066 ceph-mon[85781]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-28T16:39:30.197 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:29 smithi066 ceph-mon[85781]: from='osd.3 [v2:172.21.15.66:6800/2991662728,v1:172.21.15.66:6801/2991662728]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:39:30.198 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:29 smithi066 ceph-mon[85781]: osdmap e66: 6 total, 5 up, 6 in 2024-02-28T16:39:30.198 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:29 smithi066 ceph-mon[85781]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:39:30.198 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:29 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:30.198 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:29 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:30.253 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:29 smithi001 ceph-mon[112940]: pgmap v134: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:30.253 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:29 smithi001 ceph-mon[112940]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-28T16:39:30.253 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:29 smithi001 ceph-mon[112940]: from='osd.3 [v2:172.21.15.66:6800/2991662728,v1:172.21.15.66:6801/2991662728]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:39:30.253 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:29 smithi001 ceph-mon[112940]: osdmap e66: 6 total, 5 up, 6 in 2024-02-28T16:39:30.253 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:29 smithi001 ceph-mon[112940]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:39:30.253 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:29 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:30.254 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:29 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:31.245 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:30 smithi066 ceph-mon[85781]: from='osd.3 ' entity='osd.3' 2024-02-28T16:39:31.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:30 smithi001 ceph-mon[112940]: from='osd.3 ' entity='osd.3' 2024-02-28T16:39:32.204 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:32 smithi066 ceph-mon[85781]: pgmap v136: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:32.204 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:32 smithi066 ceph-mon[85781]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:39:32.204 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:32 smithi066 ceph-mon[85781]: osd.3 [v2:172.21.15.66:6800/2991662728,v1:172.21.15.66:6801/2991662728] boot 2024-02-28T16:39:32.205 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:32 smithi066 ceph-mon[85781]: osdmap e67: 6 total, 6 up, 6 in 2024-02-28T16:39:32.205 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:39:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:32 smithi001 ceph-mon[112940]: pgmap v136: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 74/372 objects degraded (19.892%) 2024-02-28T16:39:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:32 smithi001 ceph-mon[112940]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:39:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:32 smithi001 ceph-mon[112940]: osd.3 [v2:172.21.15.66:6800/2991662728,v1:172.21.15.66:6801/2991662728] boot 2024-02-28T16:39:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:32 smithi001 ceph-mon[112940]: osdmap e67: 6 total, 6 up, 6 in 2024-02-28T16:39:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-28T16:39:33.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:33 smithi001 ceph-mon[112940]: osdmap e68: 6 total, 6 up, 6 in 2024-02-28T16:39:33.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:33 smithi066 ceph-mon[85781]: osdmap e68: 6 total, 6 up, 6 in 2024-02-28T16:39:34.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:34 smithi001 ceph-mon[112940]: pgmap v139: 65 pgs: 18 peering, 11 active+undersized+degraded, 9 active+undersized, 27 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 49/372 objects degraded (13.172%) 2024-02-28T16:39:34.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:34 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 49/372 objects degraded (13.172%), 11 pgs degraded (PG_DEGRADED) 2024-02-28T16:39:34.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:34 smithi066 ceph-mon[85781]: pgmap v139: 65 pgs: 18 peering, 11 active+undersized+degraded, 9 active+undersized, 27 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 49/372 objects degraded (13.172%) 2024-02-28T16:39:34.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:34 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 49/372 objects degraded (13.172%), 11 pgs degraded (PG_DEGRADED) 2024-02-28T16:39:36.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: pgmap v140: 65 pgs: 18 peering, 1 active+undersized+degraded, 3 active+undersized, 43 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4/372 objects degraded (1.075%) 2024-02-28T16:39:36.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:36.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-28T16:39:36.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:35 smithi066 ceph-mon[85781]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: pgmap v140: 65 pgs: 18 peering, 1 active+undersized+degraded, 3 active+undersized, 43 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4/372 objects degraded (1.075%) 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:36.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:36.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-28T16:39:36.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-28T16:39:36.271 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:35 smithi001 ceph-mon[112940]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-02-28T16:39:37.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:37 smithi001 ceph-mon[112940]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/372 objects degraded (1.075%), 1 pg degraded) 2024-02-28T16:39:37.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:37 smithi066 ceph-mon[85781]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/372 objects degraded (1.075%), 1 pg degraded) 2024-02-28T16:39:38.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:38 smithi001 ceph-mon[112940]: pgmap v141: 65 pgs: 10 peering, 55 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:38.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:38 smithi066 ceph-mon[85781]: pgmap v141: 65 pgs: 10 peering, 55 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:40.300 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:40 smithi066 ceph-mon[85781]: pgmap v142: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:40.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:40 smithi001 ceph-mon[112940]: pgmap v142: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:41.984 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:41 smithi066 ceph-mon[85781]: pgmap v143: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:41.984 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:41.984 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:39:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:41 smithi001 ceph-mon[112940]: pgmap v143: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:42.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:39:43.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:43 smithi066 ceph-mon[85781]: pgmap v144: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:43 smithi001 ceph-mon[112940]: pgmap v144: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:45.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:45 smithi066 ceph-mon[85781]: pgmap v145: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:46.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:45 smithi001 ceph-mon[112940]: pgmap v145: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.092+0000 7fb303000700 1 -- 172.21.15.1:0/2641884925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc070be0 msgr2=0x7fb2fc1090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.092+0000 7fb303000700 1 --2- 172.21.15.1:0/2641884925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc070be0 0x7fb2fc1090e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8004970 tx=0x7fb2e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.092+0000 7fb303000700 1 -- 172.21.15.1:0/2641884925 shutdown_connections 2024-02-28T16:39:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.092+0000 7fb303000700 1 --2- 172.21.15.1:0/2641884925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc070be0 0x7fb2fc1090e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.092+0000 7fb303000700 1 --2- 172.21.15.1:0/2641884925 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2fc0702d0 0x7fb2fc0706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.092+0000 7fb303000700 1 -- 172.21.15.1:0/2641884925 >> 172.21.15.1:0/2641884925 conn(0x7fb2fc0f88e0 msgr2=0x7fb2fc0facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.092+0000 7fb303000700 1 -- 172.21.15.1:0/2641884925 shutdown_connections 2024-02-28T16:39:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.092+0000 7fb303000700 1 -- 172.21.15.1:0/2641884925 wait complete. 2024-02-28T16:39:47.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.093+0000 7fb303000700 1 Processor -- start 2024-02-28T16:39:47.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.093+0000 7fb303000700 1 -- start start 2024-02-28T16:39:47.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb303000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc0702d0 0x7fb2fc1159d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb303000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2fc070be0 0x7fb2fc115f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb303000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc116560 con 0x7fb2fc0702d0 2024-02-28T16:39:47.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb303000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc10f890 con 0x7fb2fc070be0 2024-02-28T16:39:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb301c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc0702d0 0x7fb2fc1159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb301c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc0702d0 0x7fb2fc1159d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55874/0 (socket says 172.21.15.1:55874) 2024-02-28T16:39:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb301c00700 1 -- 172.21.15.1:0/2959987701 learned_addr learned my addr 172.21.15.1:0/2959987701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb301200700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2fc070be0 0x7fb2fc115f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb301c00700 1 -- 172.21.15.1:0/2959987701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2fc070be0 msgr2=0x7fb2fc115f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb301c00700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2fc070be0 0x7fb2fc115f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb301c00700 1 -- 172.21.15.1:0/2959987701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8005040 con 0x7fb2fc0702d0 2024-02-28T16:39:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.094+0000 7fb301200700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2fc070be0 0x7fb2fc115f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:39:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.095+0000 7fb301c00700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc0702d0 0x7fb2fc1159d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000cac0 tx=0x7fb2f0007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.095+0000 7fb2f9600700 1 -- 172.21.15.1:0/2959987701 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2f000aa50 con 0x7fb2fc0702d0 2024-02-28T16:39:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.095+0000 7fb2f9600700 1 -- 172.21.15.1:0/2959987701 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2f000abb0 con 0x7fb2fc0702d0 2024-02-28T16:39:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.095+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc10fb70 con 0x7fb2fc0702d0 2024-02-28T16:39:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.095+0000 7fb2f9600700 1 -- 172.21.15.1:0/2959987701 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2f00115e0 con 0x7fb2fc0702d0 2024-02-28T16:39:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.095+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc1100b0 con 0x7fb2fc0702d0 2024-02-28T16:39:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.097+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc1067a0 con 0x7fb2fc0702d0 2024-02-28T16:39:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.097+0000 7fb2f9600700 1 -- 172.21.15.1:0/2959987701 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb2f000ad20 con 0x7fb2fc0702d0 2024-02-28T16:39:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.099+0000 7fb2f9600700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2e406d7f0 0x7fb2e406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.099+0000 7fb2f9600700 1 -- 172.21.15.1:0/2959987701 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fb2f008d530 con 0x7fb2fc0702d0 2024-02-28T16:39:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.099+0000 7fb301200700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2e406d7f0 0x7fb2e406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.100+0000 7fb301200700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2e406d7f0 0x7fb2e406fca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8000c00 tx=0x7fb2e8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:47.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.104+0000 7fb2f9600700 1 -- 172.21.15.1:0/2959987701 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb2f005a960 con 0x7fb2fc0702d0 2024-02-28T16:39:47.093 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:46 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:47.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:46 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.345+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2fc04bf40 con 0x7fb2e406d7f0 2024-02-28T16:39:47.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.347+0000 7fb2f9600700 1 -- 172.21.15.1:0/2959987701 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fb2fc04bf40 con 0x7fb2e406d7f0 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2e406d7f0 msgr2=0x7fb2e406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2e406d7f0 0x7fb2e406fca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8000c00 tx=0x7fb2e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc0702d0 msgr2=0x7fb2fc1159d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc0702d0 0x7fb2fc1159d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000cac0 tx=0x7fb2f0007ee0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 shutdown_connections 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2e406d7f0 0x7fb2e406fca0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc0702d0 0x7fb2fc1159d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 --2- 172.21.15.1:0/2959987701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2fc070be0 0x7fb2fc115f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 >> 172.21.15.1:0/2959987701 conn(0x7fb2fc0f88e0 msgr2=0x7fb2fc103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 shutdown_connections 2024-02-28T16:39:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.350+0000 7fb303000700 1 -- 172.21.15.1:0/2959987701 wait complete. 2024-02-28T16:39:47.354 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:39:47.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.509+0000 7f157c400700 1 -- 172.21.15.1:0/4112249214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105be0 msgr2=0x7f15740ff250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.509+0000 7f157c400700 1 --2- 172.21.15.1:0/4112249214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105be0 0x7f15740ff250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1560004940 tx=0x7f1560009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.509+0000 7f157c400700 1 -- 172.21.15.1:0/4112249214 shutdown_connections 2024-02-28T16:39:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.509+0000 7f157c400700 1 --2- 172.21.15.1:0/4112249214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105be0 0x7f15740ff250 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.509+0000 7f157c400700 1 --2- 172.21.15.1:0/4112249214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574105240 0x7f1574105610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.509+0000 7f157c400700 1 -- 172.21.15.1:0/4112249214 >> 172.21.15.1:0/4112249214 conn(0x7f15740fadf0 msgr2=0x7f15740fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.509+0000 7f157c400700 1 -- 172.21.15.1:0/4112249214 shutdown_connections 2024-02-28T16:39:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.509+0000 7f157c400700 1 -- 172.21.15.1:0/4112249214 wait complete. 2024-02-28T16:39:47.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.510+0000 7f157c400700 1 Processor -- start 2024-02-28T16:39:47.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.510+0000 7f157c400700 1 -- start start 2024-02-28T16:39:47.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.510+0000 7f157c400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105240 0x7f1574115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.510+0000 7f157c400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574105be0 0x7f1574116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.510+0000 7f157c400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1574116880 con 0x7f1574105240 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.510+0000 7f157c400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157410fb90 con 0x7f1574105be0 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.511+0000 7f1579a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105240 0x7f1574115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.511+0000 7f1579a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105240 0x7f1574115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55896/0 (socket says 172.21.15.1:55896) 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.511+0000 7f1579a00700 1 -- 172.21.15.1:0/2675548957 learned_addr learned my addr 172.21.15.1:0/2675548957 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.511+0000 7f1579000700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574105be0 0x7f1574116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.511+0000 7f1579a00700 1 -- 172.21.15.1:0/2675548957 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574105be0 msgr2=0x7f1574116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.511+0000 7f1579a00700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574105be0 0x7f1574116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.511+0000 7f1579a00700 1 -- 172.21.15.1:0/2675548957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560005040 con 0x7f1574105240 2024-02-28T16:39:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.512+0000 7f1579a00700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105240 0x7f1574115cf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f156800c830 tx=0x7f1568004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.512+0000 7f1571600700 1 -- 172.21.15.1:0/2675548957 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f156800a760 con 0x7f1574105240 2024-02-28T16:39:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.512+0000 7f1571600700 1 -- 172.21.15.1:0/2675548957 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f156800ad90 con 0x7f1574105240 2024-02-28T16:39:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.512+0000 7f1571600700 1 -- 172.21.15.1:0/2675548957 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15680113a0 con 0x7f1574105240 2024-02-28T16:39:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.512+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157410fe70 con 0x7f1574105240 2024-02-28T16:39:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.512+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15741103b0 con 0x7f1574105240 2024-02-28T16:39:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.513+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1574107710 con 0x7f1574105240 2024-02-28T16:39:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.514+0000 7f1571600700 1 -- 172.21.15.1:0/2675548957 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1568012400 con 0x7f1574105240 2024-02-28T16:39:47.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.515+0000 7f1571600700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155c06d7e0 0x7f155c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.515+0000 7f1571600700 1 -- 172.21.15.1:0/2675548957 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f156808ce50 con 0x7f1574105240 2024-02-28T16:39:47.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.515+0000 7f1579000700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155c06d7e0 0x7f155c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.516+0000 7f1579000700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155c06d7e0 0x7f155c06fc90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f156000b3b0 tx=0x7f1560009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:47.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.520+0000 7f1571600700 1 -- 172.21.15.1:0/2675548957 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f156805a300 con 0x7f1574105240 2024-02-28T16:39:47.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.764+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f15740fe9c0 con 0x7f155c06d7e0 2024-02-28T16:39:47.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.769+0000 7f1571600700 1 -- 172.21.15.1:0/2675548957 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f15740fe9c0 con 0x7f155c06d7e0 2024-02-28T16:39:47.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155c06d7e0 msgr2=0x7f155c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155c06d7e0 0x7f155c06fc90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f156000b3b0 tx=0x7f1560009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:47.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105240 msgr2=0x7f1574115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105240 0x7f1574115cf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f156800c830 tx=0x7f1568004470 comp rx=0 tx=0).stop 2024-02-28T16:39:47.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 shutdown_connections 2024-02-28T16:39:47.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155c06d7e0 0x7f155c06fc90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574105240 0x7f1574115cf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 --2- 172.21.15.1:0/2675548957 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574105be0 0x7f1574116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.771+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 >> 172.21.15.1:0/2675548957 conn(0x7f15740fadf0 msgr2=0x7f15740fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:47.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.772+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 shutdown_connections 2024-02-28T16:39:47.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.772+0000 7f157c400700 1 -- 172.21.15.1:0/2675548957 wait complete. 2024-02-28T16:39:47.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.922+0000 7fe953e00700 1 -- 172.21.15.1:0/3572338617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070c40 msgr2=0x7fe94c109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.922+0000 7fe953e00700 1 --2- 172.21.15.1:0/3572338617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070c40 0x7fe94c109260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe938004970 tx=0x7fe938009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:47.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.922+0000 7fe953e00700 1 -- 172.21.15.1:0/3572338617 shutdown_connections 2024-02-28T16:39:47.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.922+0000 7fe953e00700 1 --2- 172.21.15.1:0/3572338617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070c40 0x7fe94c109260 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.922+0000 7fe953e00700 1 --2- 172.21.15.1:0/3572338617 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe94c070330 0x7fe94c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.922+0000 7fe953e00700 1 -- 172.21.15.1:0/3572338617 >> 172.21.15.1:0/3572338617 conn(0x7fe94c0f8ae0 msgr2=0x7fe94c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:47.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.922+0000 7fe953e00700 1 -- 172.21.15.1:0/3572338617 shutdown_connections 2024-02-28T16:39:47.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.922+0000 7fe953e00700 1 -- 172.21.15.1:0/3572338617 wait complete. 2024-02-28T16:39:47.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.923+0000 7fe953e00700 1 Processor -- start 2024-02-28T16:39:47.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.923+0000 7fe953e00700 1 -- start start 2024-02-28T16:39:47.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.923+0000 7fe953e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070330 0x7fe94c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.923+0000 7fe953e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe94c070c40 0x7fe94c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.923+0000 7fe953e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c116790 con 0x7fe94c070330 2024-02-28T16:39:47.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.924+0000 7fe953e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c10fae0 con 0x7fe94c070c40 2024-02-28T16:39:47.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.924+0000 7fe951400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070330 0x7fe94c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.924+0000 7fe951400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070330 0x7fe94c115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55914/0 (socket says 172.21.15.1:55914) 2024-02-28T16:39:47.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.924+0000 7fe951400700 1 -- 172.21.15.1:0/2618769644 learned_addr learned my addr 172.21.15.1:0/2618769644 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:47.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.924+0000 7fe950a00700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe94c070c40 0x7fe94c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.924+0000 7fe951400700 1 -- 172.21.15.1:0/2618769644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe94c070c40 msgr2=0x7fe94c116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:47.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.924+0000 7fe951400700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe94c070c40 0x7fe94c116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:47.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.924+0000 7fe951400700 1 -- 172.21.15.1:0/2618769644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe938005040 con 0x7fe94c070330 2024-02-28T16:39:47.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.925+0000 7fe951400700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070330 0x7fe94c115c00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe93c002a50 tx=0x7fe93c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:47.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.925+0000 7fe949600700 1 -- 172.21.15.1:0/2618769644 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe93c007cc0 con 0x7fe94c070330 2024-02-28T16:39:47.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.925+0000 7fe949600700 1 -- 172.21.15.1:0/2618769644 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe93c00f040 con 0x7fe94c070330 2024-02-28T16:39:47.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.925+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c10fdc0 con 0x7fe94c070330 2024-02-28T16:39:47.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.925+0000 7fe949600700 1 -- 172.21.15.1:0/2618769644 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe93c009640 con 0x7fe94c070330 2024-02-28T16:39:47.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.925+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c110300 con 0x7fe94c070330 2024-02-28T16:39:47.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.926+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c106920 con 0x7fe94c070330 2024-02-28T16:39:47.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.928+0000 7fe949600700 1 -- 172.21.15.1:0/2618769644 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe93c0097c0 con 0x7fe94c070330 2024-02-28T16:39:47.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.929+0000 7fe949600700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe93006d7f0 0x7fe93006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:47.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.929+0000 7fe949600700 1 -- 172.21.15.1:0/2618769644 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fe93c08cb70 con 0x7fe94c070330 2024-02-28T16:39:47.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.929+0000 7fe950a00700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe93006d7f0 0x7fe93006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:47.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.930+0000 7fe950a00700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe93006d7f0 0x7fe93006fca0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe938000c00 tx=0x7fe938009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:47.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:47.934+0000 7fe949600700 1 -- 172.21.15.1:0/2618769644 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe93c056450 con 0x7fe94c070330 2024-02-28T16:39:48.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:47 smithi001 ceph-mon[112940]: pgmap v146: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:48.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:47 smithi001 ceph-mon[112940]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.166+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe94c04bf40 con 0x7fe93006d7f0 2024-02-28T16:39:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.180+0000 7fe949600700 1 -- 172.21.15.1:0/2618769644 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fe94c04bf40 con 0x7fe93006d7f0 2024-02-28T16:39:48.170 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:39:48.170 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (5m) 49s ago 20m 14.1M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:39:48.170 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (2m) 49s ago 20m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:39:48.170 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (2m) 21s ago 19m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:39:48.170 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (5m) 49s ago 19m 74.9M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:39:48.170 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (13m) 49s ago 13m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:39:48.171 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (12m) 49s ago 12m 15.0M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:39:48.171 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (12m) 21s ago 12m 14.9M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:39:48.171 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (13m) 21s ago 13m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:39:48.171 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (8m) 49s ago 21m 503M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:39:48.171 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (7m) 21s ago 18m 450M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:39:48.171 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (4m) 49s ago 22m 46.7M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:39:48.172 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (3m) 21s ago 18m 37.0M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:39:48.172 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (7m) 49s ago 20m 8108k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:39:48.172 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (6m) 21s ago 18m 8010k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:39:48.172 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (119s) 49s ago 17m 80.8M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:39:48.172 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (88s) 49s ago 17m 88.0M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:39:48.173 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (57s) 49s ago 16m 11.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:39:48.173 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (26s) 21s ago 16m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:39:48.173 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (15m) 21s ago 15m 215M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 5c3ff2016db5 2024-02-28T16:39:48.173 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (14m) 21s ago 14m 190M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:39:48.173 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (6m) 49s ago 19m 52.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:39:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe93006d7f0 msgr2=0x7fe93006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe93006d7f0 0x7fe93006fca0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe938000c00 tx=0x7fe938009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070330 msgr2=0x7fe94c115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070330 0x7fe94c115c00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe93c002a50 tx=0x7fe93c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 shutdown_connections 2024-02-28T16:39:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe93006d7f0 0x7fe93006fca0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe94c070330 0x7fe94c115c00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 --2- 172.21.15.1:0/2618769644 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe94c070c40 0x7fe94c116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 >> 172.21.15.1:0/2618769644 conn(0x7fe94c0f8ae0 msgr2=0x7fe94c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.183+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 shutdown_connections 2024-02-28T16:39:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.184+0000 7fe953e00700 1 -- 172.21.15.1:0/2618769644 wait complete. 2024-02-28T16:39:48.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:47 smithi066 ceph-mon[85781]: pgmap v146: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:48.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:47 smithi066 ceph-mon[85781]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.337+0000 7f6d28200700 1 -- 172.21.15.1:0/2361364046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d20103dc0 msgr2=0x7f6d20104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.337+0000 7f6d28200700 1 --2- 172.21.15.1:0/2361364046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d20103dc0 0x7f6d20104210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c004940 tx=0x7f6d0c009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.338+0000 7f6d28200700 1 -- 172.21.15.1:0/2361364046 shutdown_connections 2024-02-28T16:39:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.338+0000 7f6d28200700 1 --2- 172.21.15.1:0/2361364046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d20103dc0 0x7f6d20104210 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.338+0000 7f6d28200700 1 --2- 172.21.15.1:0/2361364046 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d201095f0 0x7f6d201099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.338+0000 7f6d28200700 1 -- 172.21.15.1:0/2361364046 >> 172.21.15.1:0/2361364046 conn(0x7f6d20072d50 msgr2=0x7f6d20075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.338+0000 7f6d28200700 1 -- 172.21.15.1:0/2361364046 shutdown_connections 2024-02-28T16:39:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.338+0000 7f6d28200700 1 -- 172.21.15.1:0/2361364046 wait complete. 2024-02-28T16:39:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.339+0000 7f6d28200700 1 Processor -- start 2024-02-28T16:39:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.340+0000 7f6d28200700 1 -- start start 2024-02-28T16:39:48.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.340+0000 7f6d28200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d20103dc0 0x7f6d2011a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.340+0000 7f6d28200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d201095f0 0x7f6d2011a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.340+0000 7f6d28200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2011abd0 con 0x7f6d201095f0 2024-02-28T16:39:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.340+0000 7f6d28200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20113ee0 con 0x7f6d20103dc0 2024-02-28T16:39:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.341+0000 7f6d24e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d201095f0 0x7f6d2011a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.341+0000 7f6d24e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d201095f0 0x7f6d2011a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55924/0 (socket says 172.21.15.1:55924) 2024-02-28T16:39:48.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.341+0000 7f6d24e00700 1 -- 172.21.15.1:0/995396102 learned_addr learned my addr 172.21.15.1:0/995396102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.341+0000 7f6d25800700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d20103dc0 0x7f6d2011a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.341+0000 7f6d24e00700 1 -- 172.21.15.1:0/995396102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d20103dc0 msgr2=0x7f6d2011a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.341+0000 7f6d24e00700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d20103dc0 0x7f6d2011a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.341+0000 7f6d24e00700 1 -- 172.21.15.1:0/995396102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0c005040 con 0x7f6d201095f0 2024-02-28T16:39:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.341+0000 7f6d24e00700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d201095f0 0x7f6d2011a580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009fd0 tx=0x7f6d0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:48.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.342+0000 7f6d1d600700 1 -- 172.21.15.1:0/995396102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d0c00e040 con 0x7f6d201095f0 2024-02-28T16:39:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.342+0000 7f6d1d600700 1 -- 172.21.15.1:0/995396102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d0c00bbf0 con 0x7f6d201095f0 2024-02-28T16:39:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.342+0000 7f6d1d600700 1 -- 172.21.15.1:0/995396102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d0c01c3f0 con 0x7f6d201095f0 2024-02-28T16:39:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.342+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d20114160 con 0x7f6d201095f0 2024-02-28T16:39:48.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.342+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d201146a0 con 0x7f6d201095f0 2024-02-28T16:39:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.343+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2010b9c0 con 0x7f6d201095f0 2024-02-28T16:39:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.345+0000 7f6d1d600700 1 -- 172.21.15.1:0/995396102 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6d0c00bd60 con 0x7f6d201095f0 2024-02-28T16:39:48.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.346+0000 7f6d1d600700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d0806d7f0 0x7f6d0806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.346+0000 7f6d1d600700 1 -- 172.21.15.1:0/995396102 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f6d0c08e110 con 0x7f6d201095f0 2024-02-28T16:39:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.346+0000 7f6d25800700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d0806d7f0 0x7f6d0806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:48.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.347+0000 7f6d25800700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d0806d7f0 0x7f6d0806fca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d14004550 tx=0x7f6d14009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:48.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.351+0000 7f6d1d600700 1 -- 172.21.15.1:0/995396102 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d0c05b5a0 con 0x7f6d201095f0 2024-02-28T16:39:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.652+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6d20075870 con 0x7f6d201095f0 2024-02-28T16:39:48.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.654+0000 7f6d1d600700 1 -- 172.21.15.1:0/995396102 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f6d0c05b130 con 0x7f6d201095f0 2024-02-28T16:39:48.642 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:39:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:39:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 6, 2024-02-28T16:39:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 8 2024-02-28T16:39:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:39:48.644 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:39:48.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d0806d7f0 msgr2=0x7f6d0806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d0806d7f0 0x7f6d0806fca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d14004550 tx=0x7f6d14009040 comp rx=0 tx=0).stop 2024-02-28T16:39:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d201095f0 msgr2=0x7f6d2011a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d201095f0 0x7f6d2011a580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009fd0 tx=0x7f6d0c009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 shutdown_connections 2024-02-28T16:39:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d0806d7f0 0x7f6d0806fca0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d20103dc0 0x7f6d2011a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 --2- 172.21.15.1:0/995396102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d201095f0 0x7f6d2011a580 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.656+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 >> 172.21.15.1:0/995396102 conn(0x7f6d20072d50 msgr2=0x7f6d20074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.657+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 shutdown_connections 2024-02-28T16:39:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.657+0000 7f6d28200700 1 -- 172.21.15.1:0/995396102 wait complete. 2024-02-28T16:39:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.809+0000 7fc40bc00700 1 -- 172.21.15.1:0/2512131150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 msgr2=0x7fc4041099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.809+0000 7fc40bc00700 1 --2- 172.21.15.1:0/2512131150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 0x7fc4041099c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8004940 tx=0x7fc3f8009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.809+0000 7fc40bc00700 1 -- 172.21.15.1:0/2512131150 shutdown_connections 2024-02-28T16:39:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.809+0000 7fc40bc00700 1 --2- 172.21.15.1:0/2512131150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc404103dc0 0x7fc404104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.809+0000 7fc40bc00700 1 --2- 172.21.15.1:0/2512131150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 0x7fc4041099c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.809+0000 7fc40bc00700 1 -- 172.21.15.1:0/2512131150 >> 172.21.15.1:0/2512131150 conn(0x7fc404072d50 msgr2=0x7fc404075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.809+0000 7fc40bc00700 1 -- 172.21.15.1:0/2512131150 shutdown_connections 2024-02-28T16:39:48.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.809+0000 7fc40bc00700 1 -- 172.21.15.1:0/2512131150 wait complete. 2024-02-28T16:39:48.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.810+0000 7fc40bc00700 1 Processor -- start 2024-02-28T16:39:48.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.810+0000 7fc40bc00700 1 -- start start 2024-02-28T16:39:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc40bc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc404103dc0 0x7fc40411a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc40bc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 0x7fc40411a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc40bc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40411ac70 con 0x7fc4041095f0 2024-02-28T16:39:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc40bc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc404113f80 con 0x7fc404103dc0 2024-02-28T16:39:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc403e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 0x7fc40411a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc403e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 0x7fc40411a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55950/0 (socket says 172.21.15.1:55950) 2024-02-28T16:39:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc403e00700 1 -- 172.21.15.1:0/2033853349 learned_addr learned my addr 172.21.15.1:0/2033853349 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc409200700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc404103dc0 0x7fc40411a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:48.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc403e00700 1 -- 172.21.15.1:0/2033853349 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc404103dc0 msgr2=0x7fc40411a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:48.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc403e00700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc404103dc0 0x7fc40411a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:48.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc403e00700 1 -- 172.21.15.1:0/2033853349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f8005040 con 0x7fc4041095f0 2024-02-28T16:39:48.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.811+0000 7fc409200700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc404103dc0 0x7fc40411a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:39:48.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.812+0000 7fc403e00700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 0x7fc40411a620 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000c830 tx=0x7fc3f0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:48.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.812+0000 7fc400c00700 1 -- 172.21.15.1:0/2033853349 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3f000a760 con 0x7fc4041095f0 2024-02-28T16:39:48.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.812+0000 7fc400c00700 1 -- 172.21.15.1:0/2033853349 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3f000ad90 con 0x7fc4041095f0 2024-02-28T16:39:48.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.812+0000 7fc400c00700 1 -- 172.21.15.1:0/2033853349 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3f00113a0 con 0x7fc4041095f0 2024-02-28T16:39:48.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.812+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc404114260 con 0x7fc4041095f0 2024-02-28T16:39:48.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.812+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4041147a0 con 0x7fc4041095f0 2024-02-28T16:39:48.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.813+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc404106850 con 0x7fc4041095f0 2024-02-28T16:39:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.814+0000 7fc400c00700 1 -- 172.21.15.1:0/2033853349 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc3f0012400 con 0x7fc4041095f0 2024-02-28T16:39:48.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.816+0000 7fc400c00700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc3ec06d7f0 0x7fc3ec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:48.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.816+0000 7fc400c00700 1 -- 172.21.15.1:0/2033853349 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fc3f008ce50 con 0x7fc4041095f0 2024-02-28T16:39:48.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.816+0000 7fc409200700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc3ec06d7f0 0x7fc3ec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:48.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.816+0000 7fc409200700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc3ec06d7f0 0x7fc3ec06fca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009fd0 tx=0x7fc3f8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:48.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:48.820+0000 7fc400c00700 1 -- 172.21.15.1:0/2033853349 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc3f005a2e0 con 0x7fc4041095f0 2024-02-28T16:39:49.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:48 smithi001 ceph-mon[112940]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:49.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:48 smithi001 ceph-mon[112940]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:49.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:48 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/995396102' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:49.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.089+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc4041156c0 con 0x7fc4041095f0 2024-02-28T16:39:49.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.090+0000 7fc400c00700 1 -- 172.21.15.1:0/2033853349 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7fc3f0059e70 con 0x7fc4041095f0 2024-02-28T16:39:49.079 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:39:49.079 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:39:49.080 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:39:49.081 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:39:49.082 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:39:49.083 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:39:49.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc3ec06d7f0 msgr2=0x7fc3ec06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc3ec06d7f0 0x7fc3ec06fca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009fd0 tx=0x7fc3f8009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 msgr2=0x7fc40411a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 0x7fc40411a620 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000c830 tx=0x7fc3f0004470 comp rx=0 tx=0).stop 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 shutdown_connections 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc3ec06d7f0 0x7fc3ec06fca0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc404103dc0 0x7fc40411a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 --2- 172.21.15.1:0/2033853349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4041095f0 0x7fc40411a620 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 >> 172.21.15.1:0/2033853349 conn(0x7fc404072d50 msgr2=0x7fc404074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 shutdown_connections 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.093+0000 7fc40bc00700 1 -- 172.21.15.1:0/2033853349 wait complete. 2024-02-28T16:39:49.084 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:39:49.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.242+0000 7fea4ce00700 1 -- 172.21.15.1:0/4072989065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070330 msgr2=0x7fea48070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.242+0000 7fea4ce00700 1 --2- 172.21.15.1:0/4072989065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070330 0x7fea48070700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fea38004970 tx=0x7fea38009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:49.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.242+0000 7fea4ce00700 1 -- 172.21.15.1:0/4072989065 shutdown_connections 2024-02-28T16:39:49.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.242+0000 7fea4ce00700 1 --2- 172.21.15.1:0/4072989065 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea48070c40 0x7fea48109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.242+0000 7fea4ce00700 1 --2- 172.21.15.1:0/4072989065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070330 0x7fea48070700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.242+0000 7fea4ce00700 1 -- 172.21.15.1:0/4072989065 >> 172.21.15.1:0/4072989065 conn(0x7fea480f8ae0 msgr2=0x7fea480faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:49.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.242+0000 7fea4ce00700 1 -- 172.21.15.1:0/4072989065 shutdown_connections 2024-02-28T16:39:49.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.243+0000 7fea4ce00700 1 -- 172.21.15.1:0/4072989065 wait complete. 2024-02-28T16:39:49.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.243+0000 7fea4ce00700 1 Processor -- start 2024-02-28T16:39:49.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.243+0000 7fea4ce00700 1 -- start start 2024-02-28T16:39:49.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.244+0000 7fea4ce00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea48070330 0x7fea48115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:49.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.244+0000 7fea4ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070c40 0x7fea48116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:49.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.244+0000 7fea4ce00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea481167b0 con 0x7fea48070c40 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.244+0000 7fea4ce00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4810fae0 con 0x7fea48070330 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.244+0000 7fea3fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070c40 0x7fea48116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.244+0000 7fea3fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070c40 0x7fea48116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55972/0 (socket says 172.21.15.1:55972) 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.244+0000 7fea3fe00700 1 -- 172.21.15.1:0/3375248208 learned_addr learned my addr 172.21.15.1:0/3375248208 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.244+0000 7fea45e00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea48070330 0x7fea48115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.245+0000 7fea3fe00700 1 -- 172.21.15.1:0/3375248208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea48070330 msgr2=0x7fea48115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.245+0000 7fea3fe00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea48070330 0x7fea48115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.245+0000 7fea3fe00700 1 -- 172.21.15.1:0/3375248208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38005040 con 0x7fea48070c40 2024-02-28T16:39:49.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.245+0000 7fea3fe00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070c40 0x7fea48116160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fea3000c830 tx=0x7fea30004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.245+0000 7fea3e000700 1 -- 172.21.15.1:0/3375248208 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea3000a760 con 0x7fea48070c40 2024-02-28T16:39:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.245+0000 7fea3e000700 1 -- 172.21.15.1:0/3375248208 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea3000ad90 con 0x7fea48070c40 2024-02-28T16:39:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.245+0000 7fea3e000700 1 -- 172.21.15.1:0/3375248208 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea300113a0 con 0x7fea48070c40 2024-02-28T16:39:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.245+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4810fdc0 con 0x7fea48070c40 2024-02-28T16:39:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.246+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea48110300 con 0x7fea48070c40 2024-02-28T16:39:49.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.247+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea48106920 con 0x7fea48070c40 2024-02-28T16:39:49.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.247+0000 7fea3e000700 1 -- 172.21.15.1:0/3375248208 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fea30012400 con 0x7fea48070c40 2024-02-28T16:39:49.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.248+0000 7fea3e000700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2c06d7f0 0x7fea2c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:49.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.248+0000 7fea3e000700 1 -- 172.21.15.1:0/3375248208 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fea3008cd90 con 0x7fea48070c40 2024-02-28T16:39:49.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.249+0000 7fea45e00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2c06d7f0 0x7fea2c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:49.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.249+0000 7fea45e00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2c06d7f0 0x7fea2c06fca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fea38004ee0 tx=0x7fea38009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:49.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.253+0000 7fea3e000700 1 -- 172.21.15.1:0/3375248208 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fea3005a2e0 con 0x7fea48070c40 2024-02-28T16:39:49.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:48 smithi066 ceph-mon[85781]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:49.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:48 smithi066 ceph-mon[85781]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:49.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:48 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/995396102' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:39:49.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.488+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea4810c240 con 0x7fea2c06d7f0 2024-02-28T16:39:49.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.491+0000 7fea3e000700 1 -- 172.21.15.1:0/3375248208 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fea4810c240 con 0x7fea2c06d7f0 2024-02-28T16:39:49.479 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:39:49.479 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:39:49.479 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: "crash" 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "10/21 daemons upgraded", 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading osd daemons", 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:39:49.480 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:39:49.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2c06d7f0 msgr2=0x7fea2c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2c06d7f0 0x7fea2c06fca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fea38004ee0 tx=0x7fea38009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:49.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070c40 msgr2=0x7fea48116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070c40 0x7fea48116160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fea3000c830 tx=0x7fea30004470 comp rx=0 tx=0).stop 2024-02-28T16:39:49.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 shutdown_connections 2024-02-28T16:39:49.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea2c06d7f0 0x7fea2c06fca0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea48070330 0x7fea48115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 --2- 172.21.15.1:0/3375248208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea48070c40 0x7fea48116160 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 >> 172.21.15.1:0/3375248208 conn(0x7fea480f8ae0 msgr2=0x7fea48103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:49.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 shutdown_connections 2024-02-28T16:39:49.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.494+0000 7fea4ce00700 1 -- 172.21.15.1:0/3375248208 wait complete. 2024-02-28T16:39:49.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.636+0000 7f5c59c00700 1 -- 172.21.15.1:0/2215503296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 msgr2=0x7f5c540fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.636+0000 7f5c59c00700 1 --2- 172.21.15.1:0/2215503296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 0x7f5c540fcd80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c40004940 tx=0x7f5c40009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:49.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.636+0000 7f5c59c00700 1 -- 172.21.15.1:0/2215503296 shutdown_connections 2024-02-28T16:39:49.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.636+0000 7f5c59c00700 1 --2- 172.21.15.1:0/2215503296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 0x7f5c540fcd80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.636+0000 7f5c59c00700 1 --2- 172.21.15.1:0/2215503296 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c54102d70 0x7f5c54103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.636+0000 7f5c59c00700 1 -- 172.21.15.1:0/2215503296 >> 172.21.15.1:0/2215503296 conn(0x7f5c540f88e0 msgr2=0x7f5c540facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.637+0000 7f5c59c00700 1 -- 172.21.15.1:0/2215503296 shutdown_connections 2024-02-28T16:39:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.637+0000 7f5c59c00700 1 -- 172.21.15.1:0/2215503296 wait complete. 2024-02-28T16:39:49.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.637+0000 7f5c59c00700 1 Processor -- start 2024-02-28T16:39:49.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c59c00700 1 -- start start 2024-02-28T16:39:49.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c59c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c54102d70 0x7f5c541138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:49.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c59c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 0x7f5c54113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:49.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c59c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54114460 con 0x7f5c54103710 2024-02-28T16:39:49.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c59c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5410d790 con 0x7f5c54102d70 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c53400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 0x7f5c54113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c53400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 0x7f5c54113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55984/0 (socket says 172.21.15.1:55984) 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c53400700 1 -- 172.21.15.1:0/3212564378 learned_addr learned my addr 172.21.15.1:0/3212564378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.638+0000 7f5c53e00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c54102d70 0x7f5c541138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c53400700 1 -- 172.21.15.1:0/3212564378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c54102d70 msgr2=0x7f5c541138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c53400700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c54102d70 0x7f5c541138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c53400700 1 -- 172.21.15.1:0/3212564378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c40005040 con 0x7f5c54103710 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c53e00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c54102d70 0x7f5c541138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:39:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c53400700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 0x7f5c54113e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009030 tx=0x7f5c40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c4be00700 1 -- 172.21.15.1:0/3212564378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c4000e040 con 0x7f5c54103710 2024-02-28T16:39:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5410da10 con 0x7f5c54103710 2024-02-28T16:39:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c4be00700 1 -- 172.21.15.1:0/3212564378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c4000bbf0 con 0x7f5c54103710 2024-02-28T16:39:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.639+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5410dfd0 con 0x7f5c54103710 2024-02-28T16:39:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.640+0000 7f5c4be00700 1 -- 172.21.15.1:0/3212564378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c400255e0 con 0x7f5c54103710 2024-02-28T16:39:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.641+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54105240 con 0x7f5c54103710 2024-02-28T16:39:49.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.642+0000 7f5c4be00700 1 -- 172.21.15.1:0/3212564378 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5c4000bd60 con 0x7f5c54103710 2024-02-28T16:39:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.642+0000 7f5c4be00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c3806d7f0 0x7f5c3806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:39:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.643+0000 7f5c4be00700 1 -- 172.21.15.1:0/3212564378 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f5c4008e1e0 con 0x7f5c54103710 2024-02-28T16:39:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.643+0000 7f5c53e00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c3806d7f0 0x7f5c3806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:39:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.644+0000 7f5c53e00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c3806d7f0 0x7f5c3806fca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5c44007fd0 tx=0x7f5c44009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:39:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.647+0000 7f5c4be00700 1 -- 172.21.15.1:0/3212564378 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5c4005b5f0 con 0x7f5c54103710 2024-02-28T16:39:49.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.970+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5c54109e50 con 0x7f5c54103710 2024-02-28T16:39:49.959 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:49 smithi001 ceph-mon[112940]: pgmap v147: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:49.959 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:49 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2033853349' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:39:49.959 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:49 smithi001 ceph-mon[112940]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:49.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.971+0000 7f5c4be00700 1 -- 172.21.15.1:0/3212564378 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f5c4005b180 con 0x7f5c54103710 2024-02-28T16:39:49.960 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:39:49.960 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:39:49.960 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:39:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c3806d7f0 msgr2=0x7f5c3806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c3806d7f0 0x7f5c3806fca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5c44007fd0 tx=0x7f5c44009040 comp rx=0 tx=0).stop 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 msgr2=0x7f5c54113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 0x7f5c54113e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009030 tx=0x7f5c40009e30 comp rx=0 tx=0).stop 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 shutdown_connections 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c3806d7f0 0x7f5c3806fca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c54102d70 0x7f5c541138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 --2- 172.21.15.1:0/3212564378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c54103710 0x7f5c54113e10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 >> 172.21.15.1:0/3212564378 conn(0x7f5c540f88e0 msgr2=0x7f5c540fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 shutdown_connections 2024-02-28T16:39:49.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:39:49.974+0000 7f5c59c00700 1 -- 172.21.15.1:0/3212564378 wait complete. 2024-02-28T16:39:50.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:49 smithi066 ceph-mon[85781]: pgmap v147: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:50.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:49 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2033853349' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:39:50.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:49 smithi066 ceph-mon[85781]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:39:51.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:50 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3212564378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:39:51.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:50 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-28T16:39:51.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:50 smithi001 ceph-mon[112940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-28T16:39:51.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:50 smithi001 ceph-mon[112940]: Upgrade: osd.4 is safe to restart 2024-02-28T16:39:51.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:50 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3212564378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:39:51.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:50 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-28T16:39:51.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:50 smithi066 ceph-mon[85781]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-28T16:39:51.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:50 smithi066 ceph-mon[85781]: Upgrade: osd.4 is safe to restart 2024-02-28T16:39:51.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:51 smithi066 ceph-mon[85781]: pgmap v148: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:52.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:51 smithi001 ceph-mon[112940]: pgmap v148: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:53.910 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:53 smithi066 ceph-mon[85781]: Upgrade: Updating osd.4 2024-02-28T16:39:53.910 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:53 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:53.911 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:53 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-28T16:39:53.911 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:53 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:39:53.911 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:53 smithi066 ceph-mon[85781]: Deploying daemon osd.4 on smithi066 2024-02-28T16:39:53.911 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:53 smithi066 ceph-mon[85781]: pgmap v149: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:53 smithi001 ceph-mon[112940]: Upgrade: Updating osd.4 2024-02-28T16:39:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:53 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:39:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:53 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-28T16:39:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:53 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:39:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:53 smithi001 ceph-mon[112940]: Deploying daemon osd.4 on smithi066 2024-02-28T16:39:54.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:53 smithi001 ceph-mon[112940]: pgmap v149: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:55.882 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:55 smithi066 ceph-mon[85781]: pgmap v150: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:55.882 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:39:56.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:55 smithi001 ceph-mon[112940]: pgmap v150: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:56.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:39:57.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:57 smithi066 ceph-mon[85781]: pgmap v151: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:57.997 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:39:57 smithi066 systemd[1]: Stopping Ceph osd.4 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:39:58.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:57 smithi001 ceph-mon[112940]: pgmap v151: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:39:58.749 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:39:58 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[48175]: 2024-02-28T16:39:58.276+0000 7faab7400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:39:58.749 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:39:58 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[48175]: 2024-02-28T16:39:58.276+0000 7faab7400700 -1 osd.4 68 *** Got signal Terminated *** 2024-02-28T16:39:58.749 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:39:58 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[48175]: 2024-02-28T16:39:58.276+0000 7faab7400700 -1 osd.4 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T16:39:59.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:58 smithi066 ceph-mon[85781]: osd.4 marked itself down and dead 2024-02-28T16:39:59.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:58 smithi001 ceph-mon[112940]: osd.4 marked itself down and dead 2024-02-28T16:39:59.747 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:39:59 smithi066 podman[100043]: 2024-02-28 16:39:59.501412704 +0000 UTC m=+1.423044921 container died 5c3ff2016db5647ce4c5ea02d4cd077216dcb4892875514614584e2977037d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, RELEASE=quincy-c0f02e6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:40:00.072 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:59 smithi001 ceph-mon[112940]: pgmap v152: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:00.072 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:59 smithi001 ceph-mon[112940]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:40:00.072 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:39:59 smithi001 ceph-mon[112940]: osdmap e69: 6 total, 5 up, 6 in 2024-02-28T16:40:00.199 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:39:59 smithi066 podman[100043]: 2024-02-28 16:39:59.993837591 +0000 UTC m=+1.915469804 container cleanup 5c3ff2016db5647ce4c5ea02d4cd077216dcb4892875514614584e2977037d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c0f02e6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-28T16:40:00.199 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:39:59 smithi066 bash[100043]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4 2024-02-28T16:40:00.200 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:59 smithi066 ceph-mon[85781]: pgmap v152: 65 pgs: 65 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:00.200 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:59 smithi066 ceph-mon[85781]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:40:00.200 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:39:59 smithi066 ceph-mon[85781]: osdmap e69: 6 total, 5 up, 6 in 2024-02-28T16:40:00.497 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:00 smithi066 podman[100064]: 2024-02-28 16:40:00.211417204 +0000 UTC m=+0.692726246 container remove 5c3ff2016db5647ce4c5ea02d4cd077216dcb4892875514614584e2977037d0a (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c0f02e6, ceph=True) 2024-02-28T16:40:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:00 smithi001 ceph-mon[112940]: osdmap e70: 6 total, 5 up, 6 in 2024-02-28T16:40:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:00 smithi001 ceph-mon[112940]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-02-28T16:40:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:00 smithi001 ceph-mon[112940]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:40:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:00 smithi001 ceph-mon[112940]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:40:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:00 smithi001 ceph-mon[112940]: [WRN] OSD_DOWN: 1 osds down 2024-02-28T16:40:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:00 smithi001 ceph-mon[112940]: osd.4 (root=default,host=smithi066) is down 2024-02-28T16:40:01.077 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:00 smithi066 ceph-mon[85781]: osdmap e70: 6 total, 5 up, 6 in 2024-02-28T16:40:01.077 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:00 smithi066 ceph-mon[85781]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-02-28T16:40:01.077 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:00 smithi066 ceph-mon[85781]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:40:01.077 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:00 smithi066 ceph-mon[85781]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:40:01.077 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:00 smithi066 ceph-mon[85781]: [WRN] OSD_DOWN: 1 osds down 2024-02-28T16:40:01.077 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:00 smithi066 ceph-mon[85781]: osd.4 (root=default,host=smithi066) is down 2024-02-28T16:40:01.331 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:01 smithi066 podman[100154]: 2024-02-28 16:40:00.994412263 +0000 UTC m=+0.038160466 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:40:01.331 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:01 smithi066 podman[100154]: 2024-02-28 16:40:01.120812204 +0000 UTC m=+0.164560447 container create 32bbb37db3148b72683591fc4497e74ba792e80596006f895f85f4ae6782058a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-28T16:40:01.747 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:01 smithi066 podman[100154]: 2024-02-28 16:40:01.451291721 +0000 UTC m=+0.495040014 container init 32bbb37db3148b72683591fc4497e74ba792e80596006f895f85f4ae6782058a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , RELEASE=main-1fb5d10, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-28T16:40:01.747 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:01 smithi066 podman[100154]: 2024-02-28 16:40:01.464721712 +0000 UTC m=+0.508469922 container start 32bbb37db3148b72683591fc4497e74ba792e80596006f895f85f4ae6782058a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-02-28T16:40:01.748 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:01 smithi066 podman[100154]: 2024-02-28 16:40:01.581552728 +0000 UTC m=+0.625301018 container attach 32bbb37db3148b72683591fc4497e74ba792e80596006f895f85f4ae6782058a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-02-28T16:40:02.218 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:01 smithi066 podman[100154]: 2024-02-28 16:40:01.912488422 +0000 UTC m=+0.956236669 container died 32bbb37db3148b72683591fc4497e74ba792e80596006f895f85f4ae6782058a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-02-28T16:40:02.219 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:01 smithi066 ceph-mon[85781]: pgmap v155: 65 pgs: 4 active+undersized+degraded, 11 peering, 3 stale+active+clean, 4 active+undersized, 43 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17/372 objects degraded (4.570%) 2024-02-28T16:40:02.219 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:01 smithi066 ceph-mon[85781]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-28T16:40:02.219 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:01 smithi066 ceph-mon[85781]: Health check failed: Degraded data redundancy: 17/372 objects degraded (4.570%), 4 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:02.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:01 smithi001 ceph-mon[112940]: pgmap v155: 65 pgs: 4 active+undersized+degraded, 11 peering, 3 stale+active+clean, 4 active+undersized, 43 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17/372 objects degraded (4.570%) 2024-02-28T16:40:02.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:01 smithi001 ceph-mon[112940]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-28T16:40:02.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:01 smithi001 ceph-mon[112940]: Health check failed: Degraded data redundancy: 17/372 objects degraded (4.570%), 4 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:02.997 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:02 smithi066 podman[100180]: 2024-02-28 16:40:02.62709946 +0000 UTC m=+0.695318657 container remove 32bbb37db3148b72683591fc4497e74ba792e80596006f895f85f4ae6782058a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-28T16:40:02.998 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:02 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-28T16:40:02.998 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:02 smithi066 systemd[1]: Stopped Ceph osd.4 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:40:02.998 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:02 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.4.service: Consumed 1min 45.673s CPU time. 2024-02-28T16:40:03.440 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:03 smithi066 systemd[1]: Starting Ceph osd.4 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:40:03.747 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:03 smithi066 podman[100279]: 2024-02-28 16:40:03.357845459 +0000 UTC m=+0.037295599 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:40:03.747 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:03 smithi066 podman[100279]: 2024-02-28 16:40:03.511319516 +0000 UTC m=+0.190769688 container create 487e0a7c419d553e63c8ed088525e8970aef886d5e2d495053fafb979df03379 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-1fb5d10, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-28T16:40:04.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:03 smithi066 ceph-mon[85781]: pgmap v156: 65 pgs: 10 active+undersized+degraded, 11 peering, 13 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 36/372 objects degraded (9.677%) 2024-02-28T16:40:04.247 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:03 smithi066 podman[100279]: 2024-02-28 16:40:03.773759881 +0000 UTC m=+0.453210076 container init 487e0a7c419d553e63c8ed088525e8970aef886d5e2d495053fafb979df03379 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-02-28T16:40:04.247 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:03 smithi066 podman[100279]: 2024-02-28 16:40:03.786260265 +0000 UTC m=+0.465710415 container start 487e0a7c419d553e63c8ed088525e8970aef886d5e2d495053fafb979df03379 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-28T16:40:04.247 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:03 smithi066 podman[100279]: 2024-02-28 16:40:03.922107294 +0000 UTC m=+0.601557510 container attach 487e0a7c419d553e63c8ed088525e8970aef886d5e2d495053fafb979df03379 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-28T16:40:04.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:03 smithi001 ceph-mon[112940]: pgmap v156: 65 pgs: 10 active+undersized+degraded, 11 peering, 13 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 36/372 objects degraded (9.677%) 2024-02-28T16:40:04.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:04 smithi066 ceph-mon[85781]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-28T16:40:04.998 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate[100293]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-28T16:40:04.998 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate[100293]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-28T16:40:04.998 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 bash[100279]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-28T16:40:04.998 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 bash[100279]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-28T16:40:04.998 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 bash[100279]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 bash[100279]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 bash[100279]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 bash[100279]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 bash[100279]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate[100293]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate[100293]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate[100293]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate[100293]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate[100293]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-28T16:40:04.999 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:04 smithi066 podman[100413]: 2024-02-28 16:40:04.845215081 +0000 UTC m=+0.036580885 container died 487e0a7c419d553e63c8ed088525e8970aef886d5e2d495053fafb979df03379 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-28T16:40:05.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:04 smithi001 ceph-mon[112940]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-28T16:40:05.889 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:05 smithi066 podman[100413]: 2024-02-28 16:40:05.56738137 +0000 UTC m=+0.758747215 container remove 487e0a7c419d553e63c8ed088525e8970aef886d5e2d495053fafb979df03379 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:40:06.234 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:05 smithi066 ceph-mon[85781]: pgmap v157: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:06.235 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:05 smithi066 podman[100473]: 2024-02-28 16:40:05.858603594 +0000 UTC m=+0.040381028 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:40:06.235 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:05 smithi066 podman[100473]: 2024-02-28 16:40:05.985372847 +0000 UTC m=+0.167150340 container create 1e0b6d867560cd5b9e87bf5d0a2c4c343c1b336d999d6618856d8feaf485c270 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-28T16:40:06.235 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:06 smithi066 podman[100473]: 2024-02-28 16:40:06.247212364 +0000 UTC m=+0.428989878 container init 1e0b6d867560cd5b9e87bf5d0a2c4c343c1b336d999d6618856d8feaf485c270 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-28T16:40:06.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:05 smithi001 ceph-mon[112940]: pgmap v157: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:06.497 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:06 smithi066 podman[100473]: 2024-02-28 16:40:06.258621709 +0000 UTC m=+0.440399149 container start 1e0b6d867560cd5b9e87bf5d0a2c4c343c1b336d999d6618856d8feaf485c270 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-1fb5d10) 2024-02-28T16:40:06.497 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:06 smithi066 ceph-osd[100492]: -- 172.21.15.66:0/2848845257 <== mon.1 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1b591b680 con 0x55f1b58e5000 2024-02-28T16:40:06.497 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:06 smithi066 bash[100473]: 1e0b6d867560cd5b9e87bf5d0a2c4c343c1b336d999d6618856d8feaf485c270 2024-02-28T16:40:06.497 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:06 smithi066 systemd[1]: Started Ceph osd.4 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:40:07.246 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:06 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T16:40:06.828+0000 7f6f3af3c7c0 -1 Falling back to public interface 2024-02-28T16:40:07.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:07 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:07.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:07 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:07.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:07 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:40:07.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:07 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:07.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:07 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:07.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:07 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:40:08.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:08 smithi001 ceph-mon[112940]: pgmap v158: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:08.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:08.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:08.954 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:08 smithi066 ceph-mon[85781]: pgmap v158: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:08.954 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:08.954 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:10.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:10 smithi066 ceph-mon[85781]: pgmap v159: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:10.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:10 smithi001 ceph-mon[112940]: pgmap v159: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:11.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:11 smithi066 ceph-mon[85781]: pgmap v160: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:12.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:12.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:40:12.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:11 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 65/372 objects degraded (17.473%), 16 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:12.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:12.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:11 smithi001 ceph-mon[112940]: pgmap v160: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:40:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:11 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 65/372 objects degraded (17.473%), 16 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:12.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:12.247 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:12 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T16:40:12.006+0000 7f6f3af3c7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-28T16:40:12.747 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:12 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T16:40:12.430+0000 7f6f3af3c7c0 -1 osd.4 68 log_to_monitors true 2024-02-28T16:40:13.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:12 smithi001 ceph-mon[112940]: from='osd.4 [v2:172.21.15.66:6808/2169044357,v1:172.21.15.66:6809/2169044357]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-28T16:40:13.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:12 smithi001 ceph-mon[112940]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-28T16:40:13.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:12 smithi066 ceph-mon[85781]: from='osd.4 [v2:172.21.15.66:6808/2169044357,v1:172.21.15.66:6809/2169044357]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-28T16:40:13.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:12 smithi066 ceph-mon[85781]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-28T16:40:13.247 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 16:40:12 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T16:40:12.794+0000 7f6f2a200700 -1 osd.4 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-28T16:40:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:13 smithi066 ceph-mon[85781]: pgmap v161: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:13 smithi066 ceph-mon[85781]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-28T16:40:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:13 smithi066 ceph-mon[85781]: from='osd.4 [v2:172.21.15.66:6808/2169044357,v1:172.21.15.66:6809/2169044357]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:40:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:13 smithi066 ceph-mon[85781]: osdmap e71: 6 total, 5 up, 6 in 2024-02-28T16:40:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:13 smithi066 ceph-mon[85781]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:40:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:13 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:13 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:14.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:13 smithi001 ceph-mon[112940]: pgmap v161: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 324 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/372 objects degraded (17.473%) 2024-02-28T16:40:14.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:13 smithi001 ceph-mon[112940]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-28T16:40:14.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:13 smithi001 ceph-mon[112940]: from='osd.4 [v2:172.21.15.66:6808/2169044357,v1:172.21.15.66:6809/2169044357]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:40:14.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:13 smithi001 ceph-mon[112940]: osdmap e71: 6 total, 5 up, 6 in 2024-02-28T16:40:14.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:13 smithi001 ceph-mon[112940]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:40:14.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:13 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:14.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:13 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:15.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:14 smithi066 ceph-mon[85781]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:40:15.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:14 smithi066 ceph-mon[85781]: osd.4 [v2:172.21.15.66:6808/2169044357,v1:172.21.15.66:6809/2169044357] boot 2024-02-28T16:40:15.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:14 smithi066 ceph-mon[85781]: osdmap e72: 6 total, 6 up, 6 in 2024-02-28T16:40:15.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:14 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:40:15.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:14 smithi066 ceph-mon[85781]: osdmap e73: 6 total, 6 up, 6 in 2024-02-28T16:40:15.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:14 smithi001 ceph-mon[112940]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:40:15.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:14 smithi001 ceph-mon[112940]: osd.4 [v2:172.21.15.66:6808/2169044357,v1:172.21.15.66:6809/2169044357] boot 2024-02-28T16:40:15.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:14 smithi001 ceph-mon[112940]: osdmap e72: 6 total, 6 up, 6 in 2024-02-28T16:40:15.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:14 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-28T16:40:15.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:14 smithi001 ceph-mon[112940]: osdmap e73: 6 total, 6 up, 6 in 2024-02-28T16:40:15.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:15 smithi066 ceph-mon[85781]: pgmap v164: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 39/372 objects degraded (10.484%) 2024-02-28T16:40:15.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:15 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 39/372 objects degraded (10.484%), 11 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:16.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:15 smithi001 ceph-mon[112940]: pgmap v164: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 39/372 objects degraded (10.484%) 2024-02-28T16:40:16.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:15 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 39/372 objects degraded (10.484%), 11 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:18.071 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:17 smithi001 ceph-mon[112940]: pgmap v166: 65 pgs: 10 peering, 7 active+undersized+degraded, 7 active+undersized, 41 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 22/372 objects degraded (5.914%) 2024-02-28T16:40:18.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:17 smithi066 ceph-mon[85781]: pgmap v166: 65 pgs: 10 peering, 7 active+undersized+degraded, 7 active+undersized, 41 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 22/372 objects degraded (5.914%) 2024-02-28T16:40:19.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:18 smithi066 ceph-mon[85781]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/372 objects degraded (5.914%), 7 pgs degraded) 2024-02-28T16:40:19.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:18 smithi001 ceph-mon[112940]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/372 objects degraded (5.914%), 7 pgs degraded) 2024-02-28T16:40:20.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.153+0000 7f0f06600700 1 -- 172.21.15.1:0/2139461998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00105900 msgr2=0x7f0f000fef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.153+0000 7f0f06600700 1 --2- 172.21.15.1:0/2139461998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00105900 0x7f0f000fef70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0eec004940 tx=0x7f0eec009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:20.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.153+0000 7f0f06600700 1 -- 172.21.15.1:0/2139461998 shutdown_connections 2024-02-28T16:40:20.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.153+0000 7f0f06600700 1 --2- 172.21.15.1:0/2139461998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00105900 0x7f0f000fef70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.153+0000 7f0f06600700 1 --2- 172.21.15.1:0/2139461998 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00104f60 0x7f0f00105330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.153+0000 7f0f06600700 1 -- 172.21.15.1:0/2139461998 >> 172.21.15.1:0/2139461998 conn(0x7f0f000fab10 msgr2=0x7f0f000fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:20.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.153+0000 7f0f06600700 1 -- 172.21.15.1:0/2139461998 shutdown_connections 2024-02-28T16:40:20.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.153+0000 7f0f06600700 1 -- 172.21.15.1:0/2139461998 wait complete. 2024-02-28T16:40:20.143 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:19 smithi001 ceph-mon[112940]: pgmap v167: 65 pgs: 10 peering, 55 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:20.143 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:19 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:20.143 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:19 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:20.143 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:19 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:40:20.143 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:19 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:40:20.143 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:19 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.154+0000 7f0f06600700 1 Processor -- start 2024-02-28T16:40:20.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.155+0000 7f0f06600700 1 -- start start 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.155+0000 7f0f06600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00104f60 0x7f0f00115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.155+0000 7f0f06600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00105900 0x7f0f00115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.155+0000 7f0f06600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00116670 con 0x7f0f00104f60 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.155+0000 7f0f06600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0010f910 con 0x7f0f00105900 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.155+0000 7f0f05200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00104f60 0x7f0f00115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.156+0000 7f0f05200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00104f60 0x7f0f00115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60390/0 (socket says 172.21.15.1:60390) 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.156+0000 7f0f05200700 1 -- 172.21.15.1:0/2219601048 learned_addr learned my addr 172.21.15.1:0/2219601048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.156+0000 7f0effe00700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00105900 0x7f0f00115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.156+0000 7f0f05200700 1 -- 172.21.15.1:0/2219601048 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00105900 msgr2=0x7f0f00115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.156+0000 7f0f05200700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00105900 0x7f0f00115f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.156+0000 7f0f05200700 1 -- 172.21.15.1:0/2219601048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec005040 con 0x7f0f00104f60 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.156+0000 7f0f05200700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00104f60 0x7f0f00115a50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400b790 tx=0x7f0ef4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:20.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.157+0000 7f0efcc00700 1 -- 172.21.15.1:0/2219601048 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ef400f770 con 0x7f0f00104f60 2024-02-28T16:40:20.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.157+0000 7f0efcc00700 1 -- 172.21.15.1:0/2219601048 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ef400fda0 con 0x7f0f00104f60 2024-02-28T16:40:20.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.157+0000 7f0efcc00700 1 -- 172.21.15.1:0/2219601048 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ef400d310 con 0x7f0f00104f60 2024-02-28T16:40:20.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.157+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0010fbf0 con 0x7f0f00104f60 2024-02-28T16:40:20.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.157+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f001101b0 con 0x7f0f00104f60 2024-02-28T16:40:20.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.158+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0005f5c0 con 0x7f0f00104f60 2024-02-28T16:40:20.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.159+0000 7f0efcc00700 1 -- 172.21.15.1:0/2219601048 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0ef4010420 con 0x7f0f00104f60 2024-02-28T16:40:20.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.160+0000 7f0efcc00700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0ee806d7f0 0x7f0ee806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.160+0000 7f0efcc00700 1 -- 172.21.15.1:0/2219601048 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f0ef408cd80 con 0x7f0f00104f60 2024-02-28T16:40:20.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.160+0000 7f0effe00700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0ee806d7f0 0x7f0ee806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.161+0000 7f0effe00700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0ee806d7f0 0x7f0ee806fca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0f00111280 tx=0x7f0eec009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:20.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.165+0000 7f0efcc00700 1 -- 172.21.15.1:0/2219601048 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0ef405a2d0 con 0x7f0f00104f60 2024-02-28T16:40:20.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:19 smithi066 ceph-mon[85781]: pgmap v167: 65 pgs: 10 peering, 55 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:20.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:19 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:20.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:19 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:20.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:19 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:40:20.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:19 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:40:20.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:19 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:20.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.412+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f001110a0 con 0x7f0ee806d7f0 2024-02-28T16:40:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.415+0000 7f0efcc00700 1 -- 172.21.15.1:0/2219601048 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f0f001110a0 con 0x7f0ee806d7f0 2024-02-28T16:40:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0ee806d7f0 msgr2=0x7f0ee806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0ee806d7f0 0x7f0ee806fca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0f00111280 tx=0x7f0eec009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00104f60 msgr2=0x7f0f00115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00104f60 0x7f0f00115a50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400b790 tx=0x7f0ef4009250 comp rx=0 tx=0).stop 2024-02-28T16:40:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 shutdown_connections 2024-02-28T16:40:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0ee806d7f0 0x7f0ee806fca0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f00104f60 0x7f0f00115a50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 --2- 172.21.15.1:0/2219601048 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f00105900 0x7f0f00115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 >> 172.21.15.1:0/2219601048 conn(0x7f0f000fab10 msgr2=0x7f0f000fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 shutdown_connections 2024-02-28T16:40:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.418+0000 7f0f06600700 1 -- 172.21.15.1:0/2219601048 wait complete. 2024-02-28T16:40:20.423 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:40:20.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.581+0000 7f6ca5000700 1 -- 172.21.15.1:0/3753669176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01038f0 msgr2=0x7f6ca00fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.581+0000 7f6ca5000700 1 --2- 172.21.15.1:0/3753669176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01038f0 0x7f6ca00fcf60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c88004940 tx=0x7f6c88009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:20.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.581+0000 7f6ca5000700 1 -- 172.21.15.1:0/3753669176 shutdown_connections 2024-02-28T16:40:20.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.581+0000 7f6ca5000700 1 --2- 172.21.15.1:0/3753669176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01038f0 0x7f6ca00fcf60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.581+0000 7f6ca5000700 1 --2- 172.21.15.1:0/3753669176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ca0102f50 0x7f6ca0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.581+0000 7f6ca5000700 1 -- 172.21.15.1:0/3753669176 >> 172.21.15.1:0/3753669176 conn(0x7f6ca00f8ae0 msgr2=0x7f6ca00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:20.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.582+0000 7f6ca5000700 1 -- 172.21.15.1:0/3753669176 shutdown_connections 2024-02-28T16:40:20.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.582+0000 7f6ca5000700 1 -- 172.21.15.1:0/3753669176 wait complete. 2024-02-28T16:40:20.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.583+0000 7f6ca5000700 1 Processor -- start 2024-02-28T16:40:20.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.583+0000 7f6ca5000700 1 -- start start 2024-02-28T16:40:20.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.583+0000 7f6ca5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca0102f50 0x7f6ca0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.583+0000 7f6ca5000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ca01038f0 0x7f6ca01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.584+0000 7f6ca5000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca0116810 con 0x7f6ca0102f50 2024-02-28T16:40:20.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.584+0000 7f6ca5000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca010fb40 con 0x7f6ca01038f0 2024-02-28T16:40:20.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.584+0000 7f6c9de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca0102f50 0x7f6ca0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:20.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.584+0000 7f6c9de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca0102f50 0x7f6ca0115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60402/0 (socket says 172.21.15.1:60402) 2024-02-28T16:40:20.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.584+0000 7f6c9de00700 1 -- 172.21.15.1:0/1885982244 learned_addr learned my addr 172.21.15.1:0/1885982244 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:20.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.584+0000 7f6c97e00700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ca01038f0 0x7f6ca01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.585+0000 7f6c9de00700 1 -- 172.21.15.1:0/1885982244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ca01038f0 msgr2=0x7f6ca01161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.585+0000 7f6c9de00700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ca01038f0 0x7f6ca01161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.585+0000 7f6c9de00700 1 -- 172.21.15.1:0/1885982244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88005040 con 0x7f6ca0102f50 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.585+0000 7f6c9de00700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca0102f50 0x7f6ca0115c80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6c90002a50 tx=0x7f6c900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.585+0000 7f6c96000700 1 -- 172.21.15.1:0/1885982244 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c90007cc0 con 0x7f6ca0102f50 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.585+0000 7f6c96000700 1 -- 172.21.15.1:0/1885982244 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c9000f040 con 0x7f6ca0102f50 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.585+0000 7f6c96000700 1 -- 172.21.15.1:0/1885982244 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c9000da30 con 0x7f6ca0102f50 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.585+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca010fe20 con 0x7f6ca0102f50 2024-02-28T16:40:20.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.586+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca0110330 con 0x7f6ca0102f50 2024-02-28T16:40:20.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.587+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca00fc6e0 con 0x7f6ca0102f50 2024-02-28T16:40:20.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.588+0000 7f6c96000700 1 -- 172.21.15.1:0/1885982244 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6c9000db90 con 0x7f6ca0102f50 2024-02-28T16:40:20.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.589+0000 7f6c96000700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c8406d7f0 0x7f6c8406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.589+0000 7f6c96000700 1 -- 172.21.15.1:0/1885982244 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f6c9008caa0 con 0x7f6ca0102f50 2024-02-28T16:40:20.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.589+0000 7f6c97e00700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c8406d7f0 0x7f6c8406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:20.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.590+0000 7f6c97e00700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c8406d7f0 0x7f6c8406fca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0110e70 tx=0x7f6c88009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:20.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.594+0000 7f6c96000700 1 -- 172.21.15.1:0/1885982244 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c90059f50 con 0x7f6ca0102f50 2024-02-28T16:40:20.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.837+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ca0111290 con 0x7f6c8406d7f0 2024-02-28T16:40:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.839+0000 7f6c96000700 1 -- 172.21.15.1:0/1885982244 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f6ca0111290 con 0x7f6c8406d7f0 2024-02-28T16:40:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c8406d7f0 msgr2=0x7f6c8406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c8406d7f0 0x7f6c8406fca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0110e70 tx=0x7f6c88009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca0102f50 msgr2=0x7f6ca0115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca0102f50 0x7f6ca0115c80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6c90002a50 tx=0x7f6c900046b0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 shutdown_connections 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c8406d7f0 0x7f6c8406fca0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca0102f50 0x7f6ca0115c80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 --2- 172.21.15.1:0/1885982244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ca01038f0 0x7f6ca01161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 >> 172.21.15.1:0/1885982244 conn(0x7f6ca00f8ae0 msgr2=0x7f6ca00fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 shutdown_connections 2024-02-28T16:40:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:20.842+0000 7f6ca5000700 1 -- 172.21.15.1:0/1885982244 wait complete. 2024-02-28T16:40:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.000+0000 7fdb51200700 1 -- 172.21.15.1:0/42628172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 msgr2=0x7fdb4c070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.000+0000 7fdb51200700 1 --2- 172.21.15.1:0/42628172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 0x7fdb4c070ce0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb38004970 tx=0x7fdb38009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.000+0000 7fdb51200700 1 -- 172.21.15.1:0/42628172 shutdown_connections 2024-02-28T16:40:20.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.000+0000 7fdb51200700 1 --2- 172.21.15.1:0/42628172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 0x7fdb4c070ce0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.000+0000 7fdb51200700 1 --2- 172.21.15.1:0/42628172 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb4c071f60 0x7fdb4c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.000+0000 7fdb51200700 1 -- 172.21.15.1:0/42628172 >> 172.21.15.1:0/42628172 conn(0x7fdb4c0f8ae0 msgr2=0x7fdb4c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.000+0000 7fdb51200700 1 -- 172.21.15.1:0/42628172 shutdown_connections 2024-02-28T16:40:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.000+0000 7fdb51200700 1 -- 172.21.15.1:0/42628172 wait complete. 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.001+0000 7fdb51200700 1 Processor -- start 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.001+0000 7fdb51200700 1 -- start start 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.001+0000 7fdb51200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 0x7fdb4c11a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.001+0000 7fdb51200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb4c071f60 0x7fdb4c11a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.001+0000 7fdb51200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c11aba0 con 0x7fdb4c070870 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.001+0000 7fdb51200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c113eb0 con 0x7fdb4c071f60 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.002+0000 7fdb49e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 0x7fdb4c11a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.002+0000 7fdb49e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 0x7fdb4c11a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60430/0 (socket says 172.21.15.1:60430) 2024-02-28T16:40:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.002+0000 7fdb49e00700 1 -- 172.21.15.1:0/2421147483 learned_addr learned my addr 172.21.15.1:0/2421147483 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:20.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.002+0000 7fdb49e00700 1 -- 172.21.15.1:0/2421147483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb4c071f60 msgr2=0x7fdb4c11a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:20.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.002+0000 7fdb49e00700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb4c071f60 0x7fdb4c11a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:20.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.002+0000 7fdb49e00700 1 -- 172.21.15.1:0/2421147483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb38005040 con 0x7fdb4c070870 2024-02-28T16:40:20.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.002+0000 7fdb49e00700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 0x7fdb4c11a010 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00c830 tx=0x7fdb3c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:20.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.003+0000 7fdb42000700 1 -- 172.21.15.1:0/2421147483 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3c00a760 con 0x7fdb4c070870 2024-02-28T16:40:20.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.003+0000 7fdb42000700 1 -- 172.21.15.1:0/2421147483 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb3c00ad90 con 0x7fdb4c070870 2024-02-28T16:40:20.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.003+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4c114190 con 0x7fdb4c070870 2024-02-28T16:40:20.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.003+0000 7fdb42000700 1 -- 172.21.15.1:0/2421147483 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3c0113a0 con 0x7fdb4c070870 2024-02-28T16:40:20.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.003+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c1146d0 con 0x7fdb4c070870 2024-02-28T16:40:20.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.004+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4c0fe400 con 0x7fdb4c070870 2024-02-28T16:40:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.005+0000 7fdb42000700 1 -- 172.21.15.1:0/2421147483 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdb3c012400 con 0x7fdb4c070870 2024-02-28T16:40:20.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.006+0000 7fdb42000700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdb3006d7e0 0x7fdb3006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:20.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.006+0000 7fdb42000700 1 -- 172.21.15.1:0/2421147483 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fdb3c08ce50 con 0x7fdb4c070870 2024-02-28T16:40:20.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.006+0000 7fdb49400700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdb3006d7e0 0x7fdb3006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:20.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.007+0000 7fdb49400700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdb3006d7e0 0x7fdb3006fc90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb38004890 tx=0x7fdb38009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:20.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.011+0000 7fdb42000700 1 -- 172.21.15.1:0/2421147483 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdb3c05a2e0 con 0x7fdb4c070870 2024-02-28T16:40:21.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.247+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb4c1155f0 con 0x7fdb3006d7e0 2024-02-28T16:40:21.235 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:40:21.235 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:21.235 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:21.235 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:21.235 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-28T16:40:21.236 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:20 smithi001 ceph-mon[112940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-28T16:40:21.236 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:20 smithi001 ceph-mon[112940]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-28T16:40:21.236 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:20 smithi001 ceph-mon[112940]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:40:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-28T16:40:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:20 smithi066 ceph-mon[85781]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-28T16:40:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:20 smithi066 ceph-mon[85781]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-28T16:40:21.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:20 smithi066 ceph-mon[85781]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.259+0000 7fdb42000700 1 -- 172.21.15.1:0/2421147483 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fdb4c1155f0 con 0x7fdb3006d7e0 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (6m) 82s ago 20m 14.1M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (3m) 82s ago 21m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (3m) 10s ago 19m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (6m) 82s ago 20m 74.9M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (13m) 82s ago 13m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (13m) 82s ago 13m 15.0M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:40:21.248 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (13m) 10s ago 13m 15.0M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (13m) 10s ago 13m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (8m) 82s ago 22m 503M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (7m) 10s ago 19m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (4m) 82s ago 22m 46.7M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (4m) 10s ago 19m 37.3M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (7m) 82s ago 20m 8108k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (7m) 10s ago 19m 8112k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (2m) 82s ago 18m 80.8M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (2m) 82s ago 17m 88.0M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (90s) 82s ago 17m 11.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (59s) 10s ago 16m 100M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (15s) 10s ago 16m 11.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (15m) 10s ago 15m 190M 2582M 17.2.7-535-gc0f02e60 1c37ed371348 4199ecd808fd 2024-02-28T16:40:21.249 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (6m) 82s ago 20m 52.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:40:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.262+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdb3006d7e0 msgr2=0x7fdb3006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.262+0000 7fdb51200700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdb3006d7e0 0x7fdb3006fc90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb38004890 tx=0x7fdb38009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.262+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 msgr2=0x7fdb4c11a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.262+0000 7fdb51200700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 0x7fdb4c11a010 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00c830 tx=0x7fdb3c004470 comp rx=0 tx=0).stop 2024-02-28T16:40:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.263+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 shutdown_connections 2024-02-28T16:40:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.263+0000 7fdb51200700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdb3006d7e0 0x7fdb3006fc90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.263+0000 7fdb51200700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb4c070870 0x7fdb4c11a010 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.263+0000 7fdb51200700 1 --2- 172.21.15.1:0/2421147483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdb4c071f60 0x7fdb4c11a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.263+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 >> 172.21.15.1:0/2421147483 conn(0x7fdb4c0f8ae0 msgr2=0x7fdb4c0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.263+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 shutdown_connections 2024-02-28T16:40:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.263+0000 7fdb51200700 1 -- 172.21.15.1:0/2421147483 wait complete. 2024-02-28T16:40:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.418+0000 7fa54fa00700 1 -- 172.21.15.1:0/252615120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105b10 msgr2=0x7fa5480ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.418+0000 7fa54fa00700 1 --2- 172.21.15.1:0/252615120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105b10 0x7fa5480ff180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa534004940 tx=0x7fa534009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.419+0000 7fa54fa00700 1 -- 172.21.15.1:0/252615120 shutdown_connections 2024-02-28T16:40:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.419+0000 7fa54fa00700 1 --2- 172.21.15.1:0/252615120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105b10 0x7fa5480ff180 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.419+0000 7fa54fa00700 1 --2- 172.21.15.1:0/252615120 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa548105170 0x7fa548105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.419+0000 7fa54fa00700 1 -- 172.21.15.1:0/252615120 >> 172.21.15.1:0/252615120 conn(0x7fa5480face0 msgr2=0x7fa5480fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.419+0000 7fa54fa00700 1 -- 172.21.15.1:0/252615120 shutdown_connections 2024-02-28T16:40:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.419+0000 7fa54fa00700 1 -- 172.21.15.1:0/252615120 wait complete. 2024-02-28T16:40:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.419+0000 7fa54fa00700 1 Processor -- start 2024-02-28T16:40:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.420+0000 7fa54fa00700 1 -- start start 2024-02-28T16:40:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.420+0000 7fa54fa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105170 0x7fa548115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.420+0000 7fa54fa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa548105b10 0x7fa548116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.420+0000 7fa54fa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa548116870 con 0x7fa548105170 2024-02-28T16:40:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.420+0000 7fa54fa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54810fb80 con 0x7fa548105b10 2024-02-28T16:40:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.420+0000 7fa54d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105170 0x7fa548115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.420+0000 7fa54d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105170 0x7fa548115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60440/0 (socket says 172.21.15.1:60440) 2024-02-28T16:40:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.420+0000 7fa54d000700 1 -- 172.21.15.1:0/1824716270 learned_addr learned my addr 172.21.15.1:0/1824716270 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa547e00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa548105b10 0x7fa548116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa54d000700 1 -- 172.21.15.1:0/1824716270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa548105b10 msgr2=0x7fa548116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa54d000700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa548105b10 0x7fa548116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa54d000700 1 -- 172.21.15.1:0/1824716270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa534005040 con 0x7fa548105170 2024-02-28T16:40:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa54d000700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105170 0x7fa548115ce0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00c830 tx=0x7fa53c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa544c00700 1 -- 172.21.15.1:0/1824716270 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa53c00a760 con 0x7fa548105170 2024-02-28T16:40:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa544c00700 1 -- 172.21.15.1:0/1824716270 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa53c00ad90 con 0x7fa548105170 2024-02-28T16:40:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa544c00700 1 -- 172.21.15.1:0/1824716270 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa53c0113a0 con 0x7fa548105170 2024-02-28T16:40:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.421+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54810fe60 con 0x7fa548105170 2024-02-28T16:40:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.422+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5481103a0 con 0x7fa548105170 2024-02-28T16:40:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.423+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa548107640 con 0x7fa548105170 2024-02-28T16:40:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.424+0000 7fa544c00700 1 -- 172.21.15.1:0/1824716270 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa53c012400 con 0x7fa548105170 2024-02-28T16:40:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.425+0000 7fa544c00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa53006d7f0 0x7fa53006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.425+0000 7fa544c00700 1 -- 172.21.15.1:0/1824716270 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fa53c08ce50 con 0x7fa548105170 2024-02-28T16:40:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.425+0000 7fa547e00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa53006d7f0 0x7fa53006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.426+0000 7fa547e00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa53006d7f0 0x7fa53006fca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa53400b020 tx=0x7fa534009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.430+0000 7fa544c00700 1 -- 172.21.15.1:0/1824716270 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa53c05a2e0 con 0x7fa548105170 2024-02-28T16:40:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.729+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa5481112c0 con 0x7fa548105170 2024-02-28T16:40:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.730+0000 7fa544c00700 1 -- 172.21.15.1:0/1824716270 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7fa53c059e70 con 0x7fa548105170 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:40:21.718 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 5 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 5, 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 9 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:40:21.719 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:40:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa53006d7f0 msgr2=0x7fa53006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa53006d7f0 0x7fa53006fca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa53400b020 tx=0x7fa534009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105170 msgr2=0x7fa548115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105170 0x7fa548115ce0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00c830 tx=0x7fa53c004470 comp rx=0 tx=0).stop 2024-02-28T16:40:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 shutdown_connections 2024-02-28T16:40:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa53006d7f0 0x7fa53006fca0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa548105170 0x7fa548115ce0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 --2- 172.21.15.1:0/1824716270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa548105b10 0x7fa548116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 >> 172.21.15.1:0/1824716270 conn(0x7fa5480face0 msgr2=0x7fa5480fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.733+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 shutdown_connections 2024-02-28T16:40:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.734+0000 7fa54fa00700 1 -- 172.21.15.1:0/1824716270 wait complete. 2024-02-28T16:40:21.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.890+0000 7fa536200700 1 -- 172.21.15.1:0/3006784769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530070870 msgr2=0x7fa530070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:21.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.890+0000 7fa536200700 1 --2- 172.21.15.1:0/3006784769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530070870 0x7fa530070ce0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa51c004970 tx=0x7fa51c009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:21.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.891+0000 7fa536200700 1 -- 172.21.15.1:0/3006784769 shutdown_connections 2024-02-28T16:40:21.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.891+0000 7fa536200700 1 --2- 172.21.15.1:0/3006784769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530070870 0x7fa530070ce0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.891+0000 7fa536200700 1 --2- 172.21.15.1:0/3006784769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa530071f60 0x7fa530070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.891+0000 7fa536200700 1 -- 172.21.15.1:0/3006784769 >> 172.21.15.1:0/3006784769 conn(0x7fa5300f8ae0 msgr2=0x7fa5300faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:21.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.891+0000 7fa536200700 1 -- 172.21.15.1:0/3006784769 shutdown_connections 2024-02-28T16:40:21.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.891+0000 7fa536200700 1 -- 172.21.15.1:0/3006784769 wait complete. 2024-02-28T16:40:21.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.892+0000 7fa536200700 1 Processor -- start 2024-02-28T16:40:21.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.892+0000 7fa536200700 1 -- start start 2024-02-28T16:40:21.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.892+0000 7fa536200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa530070870 0x7fa530119fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:21.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.892+0000 7fa536200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530071f60 0x7fa53011a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:21.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.892+0000 7fa536200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53011ab30 con 0x7fa530071f60 2024-02-28T16:40:21.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.892+0000 7fa536200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa530113eb0 con 0x7fa530070870 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.892+0000 7fa52ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530071f60 0x7fa53011a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.893+0000 7fa52ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530071f60 0x7fa53011a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60156/0 (socket says 172.21.15.1:60156) 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.893+0000 7fa52ea00700 1 -- 172.21.15.1:0/3600286328 learned_addr learned my addr 172.21.15.1:0/3600286328 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.893+0000 7fa52f400700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa530070870 0x7fa530119fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.893+0000 7fa52ea00700 1 -- 172.21.15.1:0/3600286328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa530070870 msgr2=0x7fa530119fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.893+0000 7fa52ea00700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa530070870 0x7fa530119fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.893+0000 7fa52ea00700 1 -- 172.21.15.1:0/3600286328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c005040 con 0x7fa530071f60 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.893+0000 7fa52f400700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa530070870 0x7fa530119fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:40:21.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.893+0000 7fa52ea00700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530071f60 0x7fa53011a4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa51c000c00 tx=0x7fa51c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:21.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.895+0000 7fa527400700 1 -- 172.21.15.1:0/3600286328 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51c00e040 con 0x7fa530071f60 2024-02-28T16:40:21.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.895+0000 7fa527400700 1 -- 172.21.15.1:0/3600286328 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa51c013420 con 0x7fa530071f60 2024-02-28T16:40:21.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.895+0000 7fa527400700 1 -- 172.21.15.1:0/3600286328 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51c01c5f0 con 0x7fa530071f60 2024-02-28T16:40:21.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.895+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa530114130 con 0x7fa530071f60 2024-02-28T16:40:21.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.895+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa530114670 con 0x7fa530071f60 2024-02-28T16:40:21.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.896+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5300fe400 con 0x7fa530071f60 2024-02-28T16:40:21.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.897+0000 7fa527400700 1 -- 172.21.15.1:0/3600286328 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa51c013590 con 0x7fa530071f60 2024-02-28T16:40:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.898+0000 7fa527400700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa51406d7f0 0x7fa51406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.898+0000 7fa527400700 1 -- 172.21.15.1:0/3600286328 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fa51c08e190 con 0x7fa530071f60 2024-02-28T16:40:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.899+0000 7fa52f400700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa51406d7f0 0x7fa51406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.899+0000 7fa52f400700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa51406d7f0 0x7fa51406fca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa520009fd0 tx=0x7fa52000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:21.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:21.903+0000 7fa527400700 1 -- 172.21.15.1:0/3600286328 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa51c05b620 con 0x7fa530071f60 2024-02-28T16:40:22.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:21 smithi001 ceph-mon[112940]: pgmap v168: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:22.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:21 smithi001 ceph-mon[112940]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:22.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:21 smithi001 ceph-mon[112940]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:22.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:21 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1824716270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.171+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa53004bed0 con 0x7fa530071f60 2024-02-28T16:40:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.172+0000 7fa527400700 1 -- 172.21.15.1:0/3600286328 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7fa51c05b1b0 con 0x7fa530071f60 2024-02-28T16:40:22.161 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:40:22.161 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:40:22.161 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:40:22.161 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:40:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:40:22.162 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:40:22.163 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:40:22.164 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:40:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:40:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:40:22.164 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:40:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:40:22.164 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:40:22.164 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa51406d7f0 msgr2=0x7fa51406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa51406d7f0 0x7fa51406fca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa520009fd0 tx=0x7fa52000b040 comp rx=0 tx=0).stop 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530071f60 msgr2=0x7fa53011a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530071f60 0x7fa53011a4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa51c000c00 tx=0x7fa51c009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 shutdown_connections 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa51406d7f0 0x7fa51406fca0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa530070870 0x7fa530119fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 --2- 172.21.15.1:0/3600286328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa530071f60 0x7fa53011a4e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 >> 172.21.15.1:0/3600286328 conn(0x7fa5300f8ae0 msgr2=0x7fa5300fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 shutdown_connections 2024-02-28T16:40:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.175+0000 7fa536200700 1 -- 172.21.15.1:0/3600286328 wait complete. 2024-02-28T16:40:22.166 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:40:22.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:21 smithi066 ceph-mon[85781]: pgmap v168: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:22.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:21 smithi066 ceph-mon[85781]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:22.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:21 smithi066 ceph-mon[85781]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:22.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:21 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1824716270' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.318+0000 7f2b73000700 1 -- 172.21.15.1:0/2299172697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 msgr2=0x7f2b6c070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.318+0000 7f2b73000700 1 --2- 172.21.15.1:0/2299172697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 0x7f2b6c070330 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c004970 tx=0x7f2b5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.318+0000 7f2b73000700 1 -- 172.21.15.1:0/2299172697 shutdown_connections 2024-02-28T16:40:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.318+0000 7f2b73000700 1 --2- 172.21.15.1:0/2299172697 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b6c070870 0x7f2b6c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.318+0000 7f2b73000700 1 --2- 172.21.15.1:0/2299172697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 0x7f2b6c070330 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.318+0000 7f2b73000700 1 -- 172.21.15.1:0/2299172697 >> 172.21.15.1:0/2299172697 conn(0x7f2b6c0f8ae0 msgr2=0x7f2b6c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.319+0000 7f2b73000700 1 -- 172.21.15.1:0/2299172697 shutdown_connections 2024-02-28T16:40:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.319+0000 7f2b73000700 1 -- 172.21.15.1:0/2299172697 wait complete. 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.319+0000 7f2b73000700 1 Processor -- start 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.319+0000 7f2b73000700 1 -- start start 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b73000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b6c070870 0x7f2b6c11a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b73000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 0x7f2b6c11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b73000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c11ac00 con 0x7f2b6c071f60 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b73000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c113f10 con 0x7f2b6c070870 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b6b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 0x7f2b6c11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b6b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 0x7f2b6c11a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60174/0 (socket says 172.21.15.1:60174) 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b6b400700 1 -- 172.21.15.1:0/2410910843 learned_addr learned my addr 172.21.15.1:0/2410910843 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b6be00700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b6c070870 0x7f2b6c11a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b6b400700 1 -- 172.21.15.1:0/2410910843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b6c070870 msgr2=0x7f2b6c11a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b6b400700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b6c070870 0x7f2b6c11a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.320+0000 7f2b6b400700 1 -- 172.21.15.1:0/2410910843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c005040 con 0x7f2b6c071f60 2024-02-28T16:40:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.321+0000 7f2b6b400700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 0x7f2b6c11a5b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2b58002a50 tx=0x7f2b580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.321+0000 7f2b63e00700 1 -- 172.21.15.1:0/2410910843 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b58007cc0 con 0x7f2b6c071f60 2024-02-28T16:40:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.321+0000 7f2b63e00700 1 -- 172.21.15.1:0/2410910843 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b5800f040 con 0x7f2b6c071f60 2024-02-28T16:40:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.321+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6c1141f0 con 0x7f2b6c071f60 2024-02-28T16:40:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.321+0000 7f2b63e00700 1 -- 172.21.15.1:0/2410910843 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b58009640 con 0x7f2b6c071f60 2024-02-28T16:40:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.321+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6c114730 con 0x7f2b6c071f60 2024-02-28T16:40:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.322+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6c0fe400 con 0x7f2b6c071f60 2024-02-28T16:40:22.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.324+0000 7f2b63e00700 1 -- 172.21.15.1:0/2410910843 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2b580097a0 con 0x7f2b6c071f60 2024-02-28T16:40:22.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.325+0000 7f2b63e00700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b5006d7f0 0x7f2b5006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:22.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.325+0000 7f2b63e00700 1 -- 172.21.15.1:0/2410910843 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f2b5808cc70 con 0x7f2b6c071f60 2024-02-28T16:40:22.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.325+0000 7f2b6be00700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b5006d7f0 0x7f2b5006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:22.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.325+0000 7f2b6be00700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b5006d7f0 0x7f2b5006fca0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c000c00 tx=0x7f2b5c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:22.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.329+0000 7f2b63e00700 1 -- 172.21.15.1:0/2410910843 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b58093080 con 0x7f2b6c071f60 2024-02-28T16:40:22.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.562+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b6c04bf40 con 0x7f2b5006d7f0 2024-02-28T16:40:22.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.565+0000 7f2b63e00700 1 -- 172.21.15.1:0/2410910843 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f2b6c04bf40 con 0x7f2b5006d7f0 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "crash" 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "11/21 daemons upgraded", 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading osd daemons", 2024-02-28T16:40:22.553 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:40:22.554 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:40:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.567+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b5006d7f0 msgr2=0x7f2b5006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.567+0000 7f2b73000700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b5006d7f0 0x7f2b5006fca0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c000c00 tx=0x7f2b5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.567+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 msgr2=0x7f2b6c11a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.567+0000 7f2b73000700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 0x7f2b6c11a5b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2b58002a50 tx=0x7f2b580046b0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.567+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 shutdown_connections 2024-02-28T16:40:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.568+0000 7f2b73000700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b5006d7f0 0x7f2b5006fca0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.568+0000 7f2b73000700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b6c070870 0x7f2b6c11a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.568+0000 7f2b73000700 1 --2- 172.21.15.1:0/2410910843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b6c071f60 0x7f2b6c11a5b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.568+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 >> 172.21.15.1:0/2410910843 conn(0x7f2b6c0f8ae0 msgr2=0x7f2b6c0fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:22.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.568+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 shutdown_connections 2024-02-28T16:40:22.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.568+0000 7f2b73000700 1 -- 172.21.15.1:0/2410910843 wait complete. 2024-02-28T16:40:22.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.710+0000 7fc69bc00700 1 -- 172.21.15.1:0/954555253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 msgr2=0x7fc6940fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:22.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.710+0000 7fc69bc00700 1 --2- 172.21.15.1:0/954555253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 0x7fc6940fcf60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc680004940 tx=0x7fc680009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:22.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.710+0000 7fc69bc00700 1 -- 172.21.15.1:0/954555253 shutdown_connections 2024-02-28T16:40:22.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.710+0000 7fc69bc00700 1 --2- 172.21.15.1:0/954555253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 0x7fc6940fcf60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.710+0000 7fc69bc00700 1 --2- 172.21.15.1:0/954555253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc694102f50 0x7fc694103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.710+0000 7fc69bc00700 1 -- 172.21.15.1:0/954555253 >> 172.21.15.1:0/954555253 conn(0x7fc6940f8ae0 msgr2=0x7fc6940faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:22.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.710+0000 7fc69bc00700 1 -- 172.21.15.1:0/954555253 shutdown_connections 2024-02-28T16:40:22.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.710+0000 7fc69bc00700 1 -- 172.21.15.1:0/954555253 wait complete. 2024-02-28T16:40:22.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.711+0000 7fc69bc00700 1 Processor -- start 2024-02-28T16:40:22.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.711+0000 7fc69bc00700 1 -- start start 2024-02-28T16:40:22.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.711+0000 7fc69bc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc694102f50 0x7fc694115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:22.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.711+0000 7fc69bc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 0x7fc694116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.711+0000 7fc69bc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc694116870 con 0x7fc6941038f0 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.711+0000 7fc69bc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69410fb80 con 0x7fc694102f50 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.712+0000 7fc693e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 0x7fc694116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.712+0000 7fc693e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 0x7fc694116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60180/0 (socket says 172.21.15.1:60180) 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.712+0000 7fc693e00700 1 -- 172.21.15.1:0/3689376591 learned_addr learned my addr 172.21.15.1:0/3689376591 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.712+0000 7fc699200700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc694102f50 0x7fc694115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.712+0000 7fc693e00700 1 -- 172.21.15.1:0/3689376591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc694102f50 msgr2=0x7fc694115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.712+0000 7fc693e00700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc694102f50 0x7fc694115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.712+0000 7fc693e00700 1 -- 172.21.15.1:0/3689376591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680005040 con 0x7fc6941038f0 2024-02-28T16:40:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.712+0000 7fc693e00700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 0x7fc694116220 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc680009100 tx=0x7fc680009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.713+0000 7fc690c00700 1 -- 172.21.15.1:0/3689376591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc68000e040 con 0x7fc6941038f0 2024-02-28T16:40:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.713+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69410fe00 con 0x7fc6941038f0 2024-02-28T16:40:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.713+0000 7fc690c00700 1 -- 172.21.15.1:0/3689376591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc680004290 con 0x7fc6941038f0 2024-02-28T16:40:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.713+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc694110340 con 0x7fc6941038f0 2024-02-28T16:40:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.713+0000 7fc690c00700 1 -- 172.21.15.1:0/3689376591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6800255e0 con 0x7fc6941038f0 2024-02-28T16:40:22.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.714+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6940fc6e0 con 0x7fc6941038f0 2024-02-28T16:40:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.715+0000 7fc690c00700 1 -- 172.21.15.1:0/3689376591 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc68000bbf0 con 0x7fc6941038f0 2024-02-28T16:40:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.716+0000 7fc690c00700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67c06d7f0 0x7fc67c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.716+0000 7fc690c00700 1 -- 172.21.15.1:0/3689376591 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fc68008e180 con 0x7fc6941038f0 2024-02-28T16:40:22.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.716+0000 7fc699200700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67c06d7f0 0x7fc67c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:22.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.717+0000 7fc699200700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67c06d7f0 0x7fc67c06fca0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc688004d60 tx=0x7fc68800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:22.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:22.721+0000 7fc690c00700 1 -- 172.21.15.1:0/3689376591 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc68005b590 con 0x7fc6941038f0 2024-02-28T16:40:22.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:22 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3600286328' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:40:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.027+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc6941112c0 con 0x7fc6941038f0 2024-02-28T16:40:23.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.028+0000 7fc690c00700 1 -- 172.21.15.1:0/3689376591 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fc680013e60 con 0x7fc6941038f0 2024-02-28T16:40:23.015 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T16:40:23.015 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:40:23.016 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:40:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67c06d7f0 msgr2=0x7fc67c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:23.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67c06d7f0 0x7fc67c06fca0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc688004d60 tx=0x7fc68800b040 comp rx=0 tx=0).stop 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 msgr2=0x7fc694116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 0x7fc694116220 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc680009100 tx=0x7fc680009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 shutdown_connections 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67c06d7f0 0x7fc67c06fca0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc694102f50 0x7fc694115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 --2- 172.21.15.1:0/3689376591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6941038f0 0x7fc694116220 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.031+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 >> 172.21.15.1:0/3689376591 conn(0x7fc6940f8ae0 msgr2=0x7fc6940fa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.032+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 shutdown_connections 2024-02-28T16:40:23.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:23.032+0000 7fc69bc00700 1 -- 172.21.15.1:0/3689376591 wait complete. 2024-02-28T16:40:23.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:22 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3600286328' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:40:24.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:23 smithi066 ceph-mon[85781]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:24.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:23 smithi066 ceph-mon[85781]: pgmap v169: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:24.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:23 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3689376591' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:40:24.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:23 smithi001 ceph-mon[112940]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:24.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:23 smithi001 ceph-mon[112940]: pgmap v169: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:24.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:23 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3689376591' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:40:26.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:25 smithi066 ceph-mon[85781]: pgmap v170: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:26.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:26.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:40:26.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:26.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:25 smithi001 ceph-mon[112940]: pgmap v170: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:26.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:26.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:40:26.270 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:28.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:27 smithi066 ceph-mon[85781]: pgmap v171: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:28.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:27 smithi001 ceph-mon[112940]: pgmap v171: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:30.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:29 smithi066 ceph-mon[85781]: pgmap v172: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:30.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:29 smithi001 ceph-mon[112940]: pgmap v172: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:32.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:31 smithi066 ceph-mon[85781]: pgmap v173: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:32.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:31 smithi001 ceph-mon[112940]: pgmap v173: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:34.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:33 smithi066 ceph-mon[85781]: pgmap v174: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:34.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:33 smithi001 ceph-mon[112940]: pgmap v174: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:36.183 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:35 smithi066 ceph-mon[85781]: pgmap v175: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:36.183 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-28T16:40:36.183 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:35 smithi066 ceph-mon[85781]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-28T16:40:36.183 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:35 smithi066 ceph-mon[85781]: Upgrade: osd.5 is safe to restart 2024-02-28T16:40:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:35 smithi001 ceph-mon[112940]: pgmap v175: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-28T16:40:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:35 smithi001 ceph-mon[112940]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-28T16:40:36.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:35 smithi001 ceph-mon[112940]: Upgrade: osd.5 is safe to restart 2024-02-28T16:40:38.672 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:38 smithi066 ceph-mon[85781]: pgmap v176: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:38.672 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:38 smithi066 ceph-mon[85781]: Upgrade: Updating osd.5 2024-02-28T16:40:38.672 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:38.672 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-28T16:40:38.672 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:38 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:40:38.672 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:38 smithi066 ceph-mon[85781]: Deploying daemon osd.5 on smithi066 2024-02-28T16:40:38.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:38 smithi001 ceph-mon[112940]: pgmap v176: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:38.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:38 smithi001 ceph-mon[112940]: Upgrade: Updating osd.5 2024-02-28T16:40:38.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:38.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-28T16:40:38.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:38 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:40:38.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:38 smithi001 ceph-mon[112940]: Deploying daemon osd.5 on smithi066 2024-02-28T16:40:40.520 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:40 smithi066 ceph-mon[85781]: pgmap v177: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:40.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:40 smithi001 ceph-mon[112940]: pgmap v177: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:41.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:40:41.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:40:42.655 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:42 smithi066 ceph-mon[85781]: pgmap v178: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:42.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:42 smithi001 ceph-mon[112940]: pgmap v178: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:42.997 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:42 smithi066 systemd[1]: Stopping Ceph osd.5 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:40:43.497 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:43 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[54754]: 2024-02-28T16:40:43.120+0000 7f19bec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T16:40:43.497 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:43 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[54754]: 2024-02-28T16:40:43.120+0000 7f19bec00700 -1 osd.5 73 *** Got signal Terminated *** 2024-02-28T16:40:43.498 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:43 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[54754]: 2024-02-28T16:40:43.120+0000 7f19bec00700 -1 osd.5 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T16:40:43.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:43 smithi001 ceph-mon[112940]: osd.5 marked itself down and dead 2024-02-28T16:40:43.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:43 smithi066 ceph-mon[85781]: osd.5 marked itself down and dead 2024-02-28T16:40:44.420 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:44 smithi066 podman[104903]: 2024-02-28 16:40:44.091334538 +0000 UTC m=+1.096506882 container died 4199ecd808fdc5b0979c457af9c7736db476b7e2ccdd2936922b203262cbfe68 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , RELEASE=quincy-c0f02e6, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-28T16:40:44.742 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:44 smithi066 ceph-mon[85781]: pgmap v179: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:44.742 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:44 smithi066 ceph-mon[85781]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:40:44.742 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:44 smithi066 ceph-mon[85781]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-28T16:40:44.742 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:44 smithi066 ceph-mon[85781]: osdmap e74: 6 total, 5 up, 6 in 2024-02-28T16:40:44.743 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:44 smithi066 podman[104903]: 2024-02-28 16:40:44.602773916 +0000 UTC m=+1.607946260 container cleanup 4199ecd808fdc5b0979c457af9c7736db476b7e2ccdd2936922b203262cbfe68 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=quincy-c0f02e6, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS) 2024-02-28T16:40:44.743 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:44 smithi066 bash[104903]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5 2024-02-28T16:40:44.997 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:44 smithi066 podman[104923]: 2024-02-28 16:40:44.753650428 +0000 UTC m=+0.641977724 container remove 4199ecd808fdc5b0979c457af9c7736db476b7e2ccdd2936922b203262cbfe68 (image=quay.ceph.io/ceph-ci/ceph@sha256:021981da540536081f6780ee5f46f271a73eedf115b483788e5c5e3dab60ff13, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-c0f02e6, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:40:45.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:44 smithi001 ceph-mon[112940]: pgmap v179: 65 pgs: 65 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:45.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:44 smithi001 ceph-mon[112940]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-28T16:40:45.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:44 smithi001 ceph-mon[112940]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-28T16:40:45.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:44 smithi001 ceph-mon[112940]: osdmap e74: 6 total, 5 up, 6 in 2024-02-28T16:40:45.831 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:45 smithi066 ceph-mon[85781]: osdmap e75: 6 total, 5 up, 6 in 2024-02-28T16:40:45.832 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:45 smithi066 podman[105009]: 2024-02-28 16:40:45.528016871 +0000 UTC m=+0.036406334 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:40:45.832 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:45 smithi066 podman[105009]: 2024-02-28 16:40:45.671429435 +0000 UTC m=+0.179818923 container create eed633675f952247491eb0d700a174b05409cb0adf103f449c052d27368be6ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-28T16:40:46.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:45 smithi001 ceph-mon[112940]: osdmap e75: 6 total, 5 up, 6 in 2024-02-28T16:40:46.247 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:45 smithi066 podman[105009]: 2024-02-28 16:40:45.959051463 +0000 UTC m=+0.467440998 container init eed633675f952247491eb0d700a174b05409cb0adf103f449c052d27368be6ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T16:40:46.247 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:45 smithi066 podman[105009]: 2024-02-28 16:40:45.971369 +0000 UTC m=+0.479758458 container start eed633675f952247491eb0d700a174b05409cb0adf103f449c052d27368be6ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-28T16:40:46.247 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:46 smithi066 podman[105009]: 2024-02-28 16:40:46.057390318 +0000 UTC m=+0.565779869 container attach eed633675f952247491eb0d700a174b05409cb0adf103f449c052d27368be6ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-28T16:40:46.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:46 smithi066 ceph-mon[85781]: pgmap v182: 65 pgs: 10 peering, 10 stale+active+clean, 45 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:46.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:46 smithi066 ceph-mon[85781]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-02-28T16:40:46.747 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:46 smithi066 podman[105009]: 2024-02-28 16:40:46.425728307 +0000 UTC m=+0.934117845 container died eed633675f952247491eb0d700a174b05409cb0adf103f449c052d27368be6ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-28T16:40:47.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:46 smithi001 ceph-mon[112940]: pgmap v182: 65 pgs: 10 peering, 10 stale+active+clean, 45 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:47.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:46 smithi001 ceph-mon[112940]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-02-28T16:40:47.678 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:47 smithi066 podman[105035]: 2024-02-28 16:40:47.27734464 +0000 UTC m=+0.835612895 container remove eed633675f952247491eb0d700a174b05409cb0adf103f449c052d27368be6ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:40:47.679 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:47 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-28T16:40:47.679 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:47 smithi066 systemd[1]: Stopped Ceph osd.5 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:40:47.679 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:47 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.5.service: Consumed 1min 42.010s CPU time. 2024-02-28T16:40:47.679 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:47 smithi066 systemd[1]: Starting Ceph osd.5 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T16:40:48.370 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:48 smithi066 podman[105135]: 2024-02-28 16:40:47.979272021 +0000 UTC m=+0.034109242 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:40:48.370 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:48 smithi066 podman[105135]: 2024-02-28 16:40:48.08653723 +0000 UTC m=+0.141374492 container create a009f8437bed48e5f4ae0e2b720ce15bf7a0b367e729e11f9179b03292f2ec95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-02-28T16:40:48.370 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:48 smithi066 podman[105135]: 2024-02-28 16:40:48.381705154 +0000 UTC m=+0.436542420 container init a009f8437bed48e5f4ae0e2b720ce15bf7a0b367e729e11f9179b03292f2ec95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, GIT_BRANCH=HEAD) 2024-02-28T16:40:48.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:48 smithi066 ceph-mon[85781]: pgmap v183: 65 pgs: 10 peering, 8 stale+active+clean, 3 active+undersized, 44 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:48.747 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:48 smithi066 podman[105135]: 2024-02-28 16:40:48.393415577 +0000 UTC m=+0.448252815 container start a009f8437bed48e5f4ae0e2b720ce15bf7a0b367e729e11f9179b03292f2ec95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-28T16:40:48.747 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:48 smithi066 podman[105135]: 2024-02-28 16:40:48.547260799 +0000 UTC m=+0.602098049 container attach a009f8437bed48e5f4ae0e2b720ce15bf7a0b367e729e11f9179b03292f2ec95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T16:40:48.821 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:48 smithi001 ceph-mon[112940]: pgmap v183: 65 pgs: 10 peering, 8 stale+active+clean, 3 active+undersized, 44 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:40:49.594 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate[105149]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-28T16:40:49.594 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate[105149]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-28T16:40:49.594 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 bash[105135]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 bash[105135]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 bash[105135]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 bash[105135]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 bash[105135]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 bash[105135]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 bash[105135]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate[105149]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate[105149]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate[105149]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate[105149]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate[105149]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-28T16:40:49.595 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:49 smithi066 podman[105268]: 2024-02-28 16:40:49.458795987 +0000 UTC m=+0.041082346 container died a009f8437bed48e5f4ae0e2b720ce15bf7a0b367e729e11f9179b03292f2ec95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-1fb5d10, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-28T16:40:49.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:49 smithi001 ceph-mon[112940]: pgmap v184: 65 pgs: 8 active+undersized+degraded, 10 peering, 10 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 24/372 objects degraded (6.452%) 2024-02-28T16:40:49.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:49 smithi001 ceph-mon[112940]: Health check failed: Degraded data redundancy: 24/372 objects degraded (6.452%), 8 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:49.854 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-mon[85781]: pgmap v184: 65 pgs: 8 active+undersized+degraded, 10 peering, 10 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 24/372 objects degraded (6.452%) 2024-02-28T16:40:49.854 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:49 smithi066 ceph-mon[85781]: Health check failed: Degraded data redundancy: 24/372 objects degraded (6.452%), 8 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:50.653 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:50 smithi066 podman[105268]: 2024-02-28 16:40:50.293050829 +0000 UTC m=+0.875337210 container remove a009f8437bed48e5f4ae0e2b720ce15bf7a0b367e729e11f9179b03292f2ec95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-02-28T16:40:50.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:50 smithi066 ceph-mon[85781]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-02-28T16:40:50.997 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:50 smithi066 podman[105328]: 2024-02-28 16:40:50.572108233 +0000 UTC m=+0.035967728 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T16:40:50.997 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:50 smithi066 podman[105328]: 2024-02-28 16:40:50.710612185 +0000 UTC m=+0.174471711 container create 7cf5aef03cba4182eb8263f5a2f5a77c251a9d8d1b51af801137d6c16d56236a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-28T16:40:51.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:50 smithi001 ceph-mon[112940]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-02-28T16:40:51.497 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:51 smithi066 podman[105328]: 2024-02-28 16:40:51.114301898 +0000 UTC m=+0.578161411 container init 7cf5aef03cba4182eb8263f5a2f5a77c251a9d8d1b51af801137d6c16d56236a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-02-28T16:40:51.497 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:51 smithi066 podman[105328]: 2024-02-28 16:40:51.126192307 +0000 UTC m=+0.590051820 container start 7cf5aef03cba4182eb8263f5a2f5a77c251a9d8d1b51af801137d6c16d56236a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-28T16:40:51.497 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:51 smithi066 ceph-osd[105347]: -- 172.21.15.66:0/3003477080 <== mon.1 v2:172.21.15.66:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a60839680 con 0x556a60803000 2024-02-28T16:40:51.497 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:51 smithi066 bash[105328]: 7cf5aef03cba4182eb8263f5a2f5a77c251a9d8d1b51af801137d6c16d56236a 2024-02-28T16:40:51.497 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:51 smithi066 systemd[1]: Started Ceph osd.5 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T16:40:51.996 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:51 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T16:40:51.703+0000 7f82230f57c0 -1 Falling back to public interface 2024-02-28T16:40:52.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:52 smithi066 ceph-mon[85781]: pgmap v185: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:52.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:52.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:52.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:40:52.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:52 smithi001 ceph-mon[112940]: pgmap v185: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:52.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:52.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:52.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:40:53.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.201+0000 7fb01f200700 1 -- 172.21.15.1:0/3645298123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 msgr2=0x7fb018105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.201+0000 7fb01f200700 1 --2- 172.21.15.1:0/3645298123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 0x7fb018105540 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb008004940 tx=0x7fb008009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.201+0000 7fb01f200700 1 -- 172.21.15.1:0/3645298123 shutdown_connections 2024-02-28T16:40:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.201+0000 7fb01f200700 1 --2- 172.21.15.1:0/3645298123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb018105b10 0x7fb0180ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.201+0000 7fb01f200700 1 --2- 172.21.15.1:0/3645298123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 0x7fb018105540 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.201+0000 7fb01f200700 1 -- 172.21.15.1:0/3645298123 >> 172.21.15.1:0/3645298123 conn(0x7fb0180face0 msgr2=0x7fb0180fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.201+0000 7fb01f200700 1 -- 172.21.15.1:0/3645298123 shutdown_connections 2024-02-28T16:40:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.201+0000 7fb01f200700 1 -- 172.21.15.1:0/3645298123 wait complete. 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.202+0000 7fb01f200700 1 Processor -- start 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.202+0000 7fb01f200700 1 -- start start 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.203+0000 7fb01f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 0x7fb018115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.203+0000 7fb01f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb018105b10 0x7fb0181161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.203+0000 7fb01f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0181167f0 con 0x7fb018105170 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.203+0000 7fb01f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01810fb40 con 0x7fb018105b10 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.203+0000 7fb017e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 0x7fb018115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.203+0000 7fb017e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 0x7fb018115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47780/0 (socket says 172.21.15.1:47780) 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.203+0000 7fb017e00700 1 -- 172.21.15.1:0/1367259275 learned_addr learned my addr 172.21.15.1:0/1367259275 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb017400700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb018105b10 0x7fb0181161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb017e00700 1 -- 172.21.15.1:0/1367259275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb018105b10 msgr2=0x7fb0181161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb017e00700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb018105b10 0x7fb0181161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb017e00700 1 -- 172.21.15.1:0/1367259275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008005040 con 0x7fb018105170 2024-02-28T16:40:53.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb017400700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb018105b10 0x7fb0181161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:40:53.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb017e00700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 0x7fb018115c60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb00800b3b0 tx=0x7fb008009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:53.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb00fe00700 1 -- 172.21.15.1:0/1367259275 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb00800e040 con 0x7fb018105170 2024-02-28T16:40:53.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb00fe00700 1 -- 172.21.15.1:0/1367259275 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb00800bbf0 con 0x7fb018105170 2024-02-28T16:40:53.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01810fdc0 con 0x7fb018105170 2024-02-28T16:40:53.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.204+0000 7fb00fe00700 1 -- 172.21.15.1:0/1367259275 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb00801c630 con 0x7fb018105170 2024-02-28T16:40:53.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.205+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb018110300 con 0x7fb018105170 2024-02-28T16:40:53.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.206+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb018102480 con 0x7fb018105170 2024-02-28T16:40:53.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.207+0000 7fb00fe00700 1 -- 172.21.15.1:0/1367259275 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb00800bd60 con 0x7fb018105170 2024-02-28T16:40:53.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.208+0000 7fb00fe00700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faffc06d8b0 0x7faffc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:53.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.208+0000 7fb00fe00700 1 -- 172.21.15.1:0/1367259275 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fb00808e180 con 0x7fb018105170 2024-02-28T16:40:53.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.208+0000 7fb017400700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faffc06d8b0 0x7faffc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:53.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.209+0000 7fb017400700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faffc06d8b0 0x7faffc06fd60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb004007fd0 tx=0x7fb004009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:53.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.213+0000 7fb00fe00700 1 -- 172.21.15.1:0/1367259275 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb008057b90 con 0x7fb018105170 2024-02-28T16:40:53.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.454+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb01804bf70 con 0x7faffc06d8b0 2024-02-28T16:40:53.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.456+0000 7fb00fe00700 1 -- 172.21.15.1:0/1367259275 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fb004004920 con 0x7faffc06d8b0 2024-02-28T16:40:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.459+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faffc06d8b0 msgr2=0x7faffc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.459+0000 7fb01f200700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faffc06d8b0 0x7faffc06fd60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb004007fd0 tx=0x7fb004009040 comp rx=0 tx=0).stop 2024-02-28T16:40:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.459+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 msgr2=0x7fb018115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.459+0000 7fb01f200700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 0x7fb018115c60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb00800b3b0 tx=0x7fb008009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:53.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.459+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 shutdown_connections 2024-02-28T16:40:53.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.459+0000 7fb01f200700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faffc06d8b0 0x7faffc06fd60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.460+0000 7fb01f200700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb018105170 0x7fb018115c60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.460+0000 7fb01f200700 1 --2- 172.21.15.1:0/1367259275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb018105b10 0x7fb0181161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.460+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 >> 172.21.15.1:0/1367259275 conn(0x7fb0180face0 msgr2=0x7fb0180fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:53.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.460+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 shutdown_connections 2024-02-28T16:40:53.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.460+0000 7fb01f200700 1 -- 172.21.15.1:0/1367259275 wait complete. 2024-02-28T16:40:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:40:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.605+0000 7f9c33400700 1 -- 172.21.15.1:0/3670259671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c102f50 msgr2=0x7f9c2c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.605+0000 7f9c33400700 1 --2- 172.21.15.1:0/3670259671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c102f50 0x7f9c2c103320 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009ce0 tx=0x7f9c20004d10 comp rx=0 tx=0).stop 2024-02-28T16:40:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.605+0000 7f9c33400700 1 -- 172.21.15.1:0/3670259671 shutdown_connections 2024-02-28T16:40:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.605+0000 7f9c33400700 1 --2- 172.21.15.1:0/3670259671 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c2c1038f0 0x7f9c2c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.605+0000 7f9c33400700 1 --2- 172.21.15.1:0/3670259671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c102f50 0x7f9c2c103320 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.605+0000 7f9c33400700 1 -- 172.21.15.1:0/3670259671 >> 172.21.15.1:0/3670259671 conn(0x7f9c2c0f8ae0 msgr2=0x7f9c2c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.605+0000 7f9c33400700 1 -- 172.21.15.1:0/3670259671 shutdown_connections 2024-02-28T16:40:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.605+0000 7f9c33400700 1 -- 172.21.15.1:0/3670259671 wait complete. 2024-02-28T16:40:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.606+0000 7f9c33400700 1 Processor -- start 2024-02-28T16:40:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.606+0000 7f9c33400700 1 -- start start 2024-02-28T16:40:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.607+0000 7f9c33400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c2c102f50 0x7f9c2c070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.607+0000 7f9c33400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c1038f0 0x7f9c2c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.607+0000 7f9c33400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c070dd0 con 0x7f9c2c1038f0 2024-02-28T16:40:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.607+0000 7f9c33400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c070f40 con 0x7f9c2c102f50 2024-02-28T16:40:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.607+0000 7f9c2be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c1038f0 0x7f9c2c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.607+0000 7f9c2be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c1038f0 0x7f9c2c0707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47790/0 (socket says 172.21.15.1:47790) 2024-02-28T16:40:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.607+0000 7f9c2be00700 1 -- 172.21.15.1:0/3627826663 learned_addr learned my addr 172.21.15.1:0/3627826663 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.607+0000 7f9c30a00700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c2c102f50 0x7f9c2c070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c2be00700 1 -- 172.21.15.1:0/3627826663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c2c102f50 msgr2=0x7f9c2c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c2be00700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c2c102f50 0x7f9c2c070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c2be00700 1 -- 172.21.15.1:0/3627826663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c20009940 con 0x7f9c2c1038f0 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c2be00700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c1038f0 0x7f9c2c0707d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c18002a80 tx=0x7f9c18004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c28c00700 1 -- 172.21.15.1:0/3627826663 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c18007cc0 con 0x7f9c2c1038f0 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c28c00700 1 -- 172.21.15.1:0/3627826663 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c1800f040 con 0x7f9c2c1038f0 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c119e70 con 0x7f9c2c1038f0 2024-02-28T16:40:53.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c28c00700 1 -- 172.21.15.1:0/3627826663 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c1800daf0 con 0x7f9c2c1038f0 2024-02-28T16:40:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.608+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c11a320 con 0x7f9c2c1038f0 2024-02-28T16:40:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.609+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c100260 con 0x7f9c2c1038f0 2024-02-28T16:40:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.610+0000 7f9c28c00700 1 -- 172.21.15.1:0/3627826663 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9c1800dc90 con 0x7f9c2c1038f0 2024-02-28T16:40:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.611+0000 7f9c28c00700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c1406d7f0 0x7f9c1406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.611+0000 7f9c28c00700 1 -- 172.21.15.1:0/3627826663 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f9c1808c560 con 0x7f9c2c1038f0 2024-02-28T16:40:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.612+0000 7f9c30a00700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c1406d7f0 0x7f9c1406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.612+0000 7f9c30a00700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c1406d7f0 0x7f9c1406fca0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000b740 tx=0x7f9c20004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:53.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.616+0000 7f9c28c00700 1 -- 172.21.15.1:0/3627826663 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9c18055dc0 con 0x7f9c2c1038f0 2024-02-28T16:40:53.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.860+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c2c0719d0 con 0x7f9c1406d7f0 2024-02-28T16:40:53.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.862+0000 7f9c28c00700 1 -- 172.21.15.1:0/3627826663 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f9c2c0719d0 con 0x7f9c1406d7f0 2024-02-28T16:40:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.864+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c1406d7f0 msgr2=0x7f9c1406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.864+0000 7f9c33400700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c1406d7f0 0x7f9c1406fca0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000b740 tx=0x7f9c20004d10 comp rx=0 tx=0).stop 2024-02-28T16:40:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.864+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c1038f0 msgr2=0x7f9c2c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.864+0000 7f9c33400700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c1038f0 0x7f9c2c0707d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c18002a80 tx=0x7f9c18004630 comp rx=0 tx=0).stop 2024-02-28T16:40:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.865+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 shutdown_connections 2024-02-28T16:40:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.865+0000 7f9c33400700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c1406d7f0 0x7f9c1406fca0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.865+0000 7f9c33400700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c2c102f50 0x7f9c2c070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.865+0000 7f9c33400700 1 --2- 172.21.15.1:0/3627826663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c2c1038f0 0x7f9c2c0707d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.865+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 >> 172.21.15.1:0/3627826663 conn(0x7f9c2c0f8ae0 msgr2=0x7f9c2c0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.865+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 shutdown_connections 2024-02-28T16:40:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:53.865+0000 7f9c33400700 1 -- 172.21.15.1:0/3627826663 wait complete. 2024-02-28T16:40:53.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.008+0000 7f9243a00700 1 -- 172.21.15.1:0/3790328745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 msgr2=0x7f923c070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.008+0000 7f9243a00700 1 --2- 172.21.15.1:0/3790328745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 0x7f923c070700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9230004970 tx=0x7f9230009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.008+0000 7f9243a00700 1 -- 172.21.15.1:0/3790328745 shutdown_connections 2024-02-28T16:40:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.008+0000 7f9243a00700 1 --2- 172.21.15.1:0/3790328745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f923c070c40 0x7f923c109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.008+0000 7f9243a00700 1 --2- 172.21.15.1:0/3790328745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 0x7f923c070700 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.008+0000 7f9243a00700 1 -- 172.21.15.1:0/3790328745 >> 172.21.15.1:0/3790328745 conn(0x7f923c0f8ae0 msgr2=0x7f923c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.008+0000 7f9243a00700 1 -- 172.21.15.1:0/3790328745 shutdown_connections 2024-02-28T16:40:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.008+0000 7f9243a00700 1 -- 172.21.15.1:0/3790328745 wait complete. 2024-02-28T16:40:54.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.009+0000 7f9243a00700 1 Processor -- start 2024-02-28T16:40:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.009+0000 7f9243a00700 1 -- start start 2024-02-28T16:40:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9243a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 0x7f923c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9243a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f923c070c40 0x7f923c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9243a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c116790 con 0x7f923c070330 2024-02-28T16:40:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9243a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c10fac0 con 0x7f923c070c40 2024-02-28T16:40:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9241000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 0x7f923c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9241000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 0x7f923c115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47812/0 (socket says 172.21.15.1:47812) 2024-02-28T16:40:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9241000700 1 -- 172.21.15.1:0/1124330259 learned_addr learned my addr 172.21.15.1:0/1124330259 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f923be00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f923c070c40 0x7f923c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9241000700 1 -- 172.21.15.1:0/1124330259 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f923c070c40 msgr2=0x7f923c116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9241000700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f923c070c40 0x7f923c116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.010+0000 7f9241000700 1 -- 172.21.15.1:0/1124330259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9230005040 con 0x7f923c070330 2024-02-28T16:40:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.011+0000 7f9241000700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 0x7f923c115c00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9230000c00 tx=0x7f9230009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:54.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.011+0000 7f9238c00700 1 -- 172.21.15.1:0/1124330259 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f923000e040 con 0x7f923c070330 2024-02-28T16:40:54.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.011+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923c10fd40 con 0x7f923c070330 2024-02-28T16:40:54.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.011+0000 7f9238c00700 1 -- 172.21.15.1:0/1124330259 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9230004290 con 0x7f923c070330 2024-02-28T16:40:54.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.011+0000 7f9238c00700 1 -- 172.21.15.1:0/1124330259 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f923001c630 con 0x7f923c070330 2024-02-28T16:40:54.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.011+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923c110280 con 0x7f923c070330 2024-02-28T16:40:54.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.012+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923c106920 con 0x7f923c070330 2024-02-28T16:40:54.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.013+0000 7f9238c00700 1 -- 172.21.15.1:0/1124330259 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9230013980 con 0x7f923c070330 2024-02-28T16:40:54.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.014+0000 7f9238c00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922406d7e0 0x7f922406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.014+0000 7f9238c00700 1 -- 172.21.15.1:0/1124330259 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f923008e120 con 0x7f923c070330 2024-02-28T16:40:54.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.014+0000 7f923be00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922406d7e0 0x7f922406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.015+0000 7f923be00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922406d7e0 0x7f922406fc90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9228007fd0 tx=0x7f9228009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:54.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.019+0000 7f9238c00700 1 -- 172.21.15.1:0/1124330259 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f923005b5b0 con 0x7f923c070330 2024-02-28T16:40:54.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.251+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f923c04bed0 con 0x7f922406d7e0 2024-02-28T16:40:54.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.264+0000 7f9238c00700 1 -- 172.21.15.1:0/1124330259 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f923c04bed0 con 0x7f922406d7e0 2024-02-28T16:40:54.253 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:40:54.253 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (7m) 115s ago 21m 14.1M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:40:54.253 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (3m) 115s ago 21m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:40:54.253 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (3m) 43s ago 20m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (6m) 115s ago 20m 74.9M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (14m) 115s ago 14m 117M - 17.2.7-535-gc0f02e60 1c37ed371348 741e686e6eb4 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (14m) 115s ago 14m 15.0M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (13m) 43s ago 13m 15.0M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (14m) 43s ago 14m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (9m) 115s ago 23m 503M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (8m) 43s ago 19m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (5m) 115s ago 23m 46.7M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (4m) 43s ago 19m 37.3M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (8m) 115s ago 21m 8108k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (7m) 43s ago 20m 8112k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (3m) 115s ago 18m 80.8M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (2m) 115s ago 18m 88.0M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (2m) 115s ago 17m 11.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:40:54.254 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (92s) 43s ago 17m 100M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:40:54.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (48s) 43s ago 16m 11.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:40:54.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 starting - - - 2582M 2024-02-28T16:40:54.255 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (7m) 115s ago 20m 52.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:40:54.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.267+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922406d7e0 msgr2=0x7f922406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.267+0000 7f9243a00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922406d7e0 0x7f922406fc90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9228007fd0 tx=0x7f9228009040 comp rx=0 tx=0).stop 2024-02-28T16:40:54.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.267+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 msgr2=0x7f923c115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.267+0000 7f9243a00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 0x7f923c115c00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9230000c00 tx=0x7f9230009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:54.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.268+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 shutdown_connections 2024-02-28T16:40:54.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.268+0000 7f9243a00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922406d7e0 0x7f922406fc90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.268+0000 7f9243a00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f923c070330 0x7f923c115c00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.268+0000 7f9243a00700 1 --2- 172.21.15.1:0/1124330259 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f923c070c40 0x7f923c116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.268+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 >> 172.21.15.1:0/1124330259 conn(0x7f923c0f8ae0 msgr2=0x7f923c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:54.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.268+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 shutdown_connections 2024-02-28T16:40:54.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.268+0000 7f9243a00700 1 -- 172.21.15.1:0/1124330259 wait complete. 2024-02-28T16:40:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.421+0000 7f3571800700 1 -- 172.21.15.1:0/2493557320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fdd90 msgr2=0x7f356c0fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.421+0000 7f3571800700 1 --2- 172.21.15.1:0/2493557320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fdd90 0x7f356c0fe160 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f355c004940 tx=0x7f355c009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.421+0000 7f3571800700 1 -- 172.21.15.1:0/2493557320 shutdown_connections 2024-02-28T16:40:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.421+0000 7f3571800700 1 --2- 172.21.15.1:0/2493557320 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f356c0fe730 0x7f356c106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.421+0000 7f3571800700 1 --2- 172.21.15.1:0/2493557320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fdd90 0x7f356c0fe160 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.421+0000 7f3571800700 1 -- 172.21.15.1:0/2493557320 >> 172.21.15.1:0/2493557320 conn(0x7f356c073ab0 msgr2=0x7f356c073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.421+0000 7f3571800700 1 -- 172.21.15.1:0/2493557320 shutdown_connections 2024-02-28T16:40:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.421+0000 7f3571800700 1 -- 172.21.15.1:0/2493557320 wait complete. 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.422+0000 7f3571800700 1 Processor -- start 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.422+0000 7f3571800700 1 -- start start 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.422+0000 7f3571800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f356c0fdd90 0x7f356c115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.422+0000 7f3571800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fe730 0x7f356c115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.422+0000 7f3571800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c116600 con 0x7f356c0fe730 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.422+0000 7f3571800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c10f930 con 0x7f356c0fdd90 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fe730 0x7f356c115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fe730 0x7f356c115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47828/0 (socket says 172.21.15.1:47828) 2024-02-28T16:40:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356b400700 1 -- 172.21.15.1:0/4082680628 learned_addr learned my addr 172.21.15.1:0/4082680628 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356be00700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f356c0fdd90 0x7f356c115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356b400700 1 -- 172.21.15.1:0/4082680628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f356c0fdd90 msgr2=0x7f356c115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356b400700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f356c0fdd90 0x7f356c115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356b400700 1 -- 172.21.15.1:0/4082680628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c005040 con 0x7f356c0fe730 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356be00700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f356c0fdd90 0x7f356c115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.423+0000 7f356b400700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fe730 0x7f356c115fb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3558002a80 tx=0x7f35580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.424+0000 7f3563e00700 1 -- 172.21.15.1:0/4082680628 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3558007cc0 con 0x7f356c0fe730 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.424+0000 7f3563e00700 1 -- 172.21.15.1:0/4082680628 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f355800f040 con 0x7f356c0fe730 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.424+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c10fc10 con 0x7f356c0fe730 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.424+0000 7f3563e00700 1 -- 172.21.15.1:0/4082680628 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35580096f0 con 0x7f356c0fe730 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.424+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c110150 con 0x7f356c0fe730 2024-02-28T16:40:54.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.425+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c1068c0 con 0x7f356c0fe730 2024-02-28T16:40:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.427+0000 7f3563e00700 1 -- 172.21.15.1:0/4082680628 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f355800d690 con 0x7f356c0fe730 2024-02-28T16:40:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.427+0000 7f3563e00700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355006d7e0 0x7f355006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.428+0000 7f3563e00700 1 -- 172.21.15.1:0/4082680628 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f355808c560 con 0x7f356c0fe730 2024-02-28T16:40:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.428+0000 7f356be00700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355006d7e0 0x7f355006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.428+0000 7f356be00700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355006d7e0 0x7f355006fc90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f355c00b3b0 tx=0x7f355c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:54.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.432+0000 7f3563e00700 1 -- 172.21.15.1:0/4082680628 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3558059970 con 0x7f356c0fe730 2024-02-28T16:40:54.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:54 smithi001 ceph-mon[112940]: pgmap v186: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:54.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:54 smithi001 ceph-mon[112940]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:54.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.743+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f356c109e00 con 0x7f356c0fe730 2024-02-28T16:40:54.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.744+0000 7f3563e00700 1 -- 172.21.15.1:0/4082680628 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+732 (secure 0 0 0) 0x7f3558059500 con 0x7f356c0fe730 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 5 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:40:54.733 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 4, 2024-02-28T16:40:54.734 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 9 2024-02-28T16:40:54.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:40:54.734 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:40:54.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355006d7e0 msgr2=0x7f355006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355006d7e0 0x7f355006fc90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f355c00b3b0 tx=0x7f355c009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:54.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fe730 msgr2=0x7f356c115fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fe730 0x7f356c115fb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3558002a80 tx=0x7f35580046b0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 shutdown_connections 2024-02-28T16:40:54.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355006d7e0 0x7f355006fc90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f356c0fdd90 0x7f356c115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 --2- 172.21.15.1:0/4082680628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f356c0fe730 0x7f356c115fb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 >> 172.21.15.1:0/4082680628 conn(0x7f356c073ab0 msgr2=0x7f356c103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:54.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 shutdown_connections 2024-02-28T16:40:54.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.747+0000 7f3571800700 1 -- 172.21.15.1:0/4082680628 wait complete. 2024-02-28T16:40:54.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:54 smithi066 ceph-mon[85781]: pgmap v186: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:54.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:54 smithi066 ceph-mon[85781]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.909+0000 7fbd16e00700 1 -- 172.21.15.1:0/2258958508 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbd10070330 msgr2=0x7fbd10070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.909+0000 7fbd16e00700 1 --2- 172.21.15.1:0/2258958508 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbd10070330 0x7fbd10070700 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbd00004940 tx=0x7fbd00009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.909+0000 7fbd16e00700 1 -- 172.21.15.1:0/2258958508 shutdown_connections 2024-02-28T16:40:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.909+0000 7fbd16e00700 1 --2- 172.21.15.1:0/2258958508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd10070c40 0x7fbd10109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.909+0000 7fbd16e00700 1 --2- 172.21.15.1:0/2258958508 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbd10070330 0x7fbd10070700 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.909+0000 7fbd16e00700 1 -- 172.21.15.1:0/2258958508 >> 172.21.15.1:0/2258958508 conn(0x7fbd100f8ae0 msgr2=0x7fbd100faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.910+0000 7fbd16e00700 1 -- 172.21.15.1:0/2258958508 shutdown_connections 2024-02-28T16:40:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.910+0000 7fbd16e00700 1 -- 172.21.15.1:0/2258958508 wait complete. 2024-02-28T16:40:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.910+0000 7fbd16e00700 1 Processor -- start 2024-02-28T16:40:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd16e00700 1 -- start start 2024-02-28T16:40:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd16e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbd10070330 0x7fbd10115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd16e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd10070c40 0x7fbd10116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd16e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd101167b0 con 0x7fbd10070c40 2024-02-28T16:40:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd16e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1010fae0 con 0x7fbd10070330 2024-02-28T16:40:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd07400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd10070c40 0x7fbd10116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd0fe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbd10070330 0x7fbd10115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd07400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd10070c40 0x7fbd10116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47844/0 (socket says 172.21.15.1:47844) 2024-02-28T16:40:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.911+0000 7fbd07400700 1 -- 172.21.15.1:0/4189510817 learned_addr learned my addr 172.21.15.1:0/4189510817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.912+0000 7fbd07400700 1 -- 172.21.15.1:0/4189510817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbd10070330 msgr2=0x7fbd10115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.912+0000 7fbd07400700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbd10070330 0x7fbd10115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.912+0000 7fbd07400700 1 -- 172.21.15.1:0/4189510817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd00005040 con 0x7fbd10070c40 2024-02-28T16:40:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.912+0000 7fbd07400700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd10070c40 0x7fbd10116160 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800eb40 tx=0x7fbcf800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.912+0000 7fbd0cc00700 1 -- 172.21.15.1:0/4189510817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcf8007cc0 con 0x7fbd10070c40 2024-02-28T16:40:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.912+0000 7fbd0cc00700 1 -- 172.21.15.1:0/4189510817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcf800f040 con 0x7fbd10070c40 2024-02-28T16:40:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.912+0000 7fbd0cc00700 1 -- 172.21.15.1:0/4189510817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcf800ca10 con 0x7fbd10070c40 2024-02-28T16:40:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.912+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1010fd60 con 0x7fbd10070c40 2024-02-28T16:40:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.913+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd101102a0 con 0x7fbd10070c40 2024-02-28T16:40:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.914+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd10106920 con 0x7fbd10070c40 2024-02-28T16:40:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.915+0000 7fbd0cc00700 1 -- 172.21.15.1:0/4189510817 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbcf800cb70 con 0x7fbd10070c40 2024-02-28T16:40:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.916+0000 7fbd0cc00700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbcf406d7e0 0x7fbcf406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.916+0000 7fbd0cc00700 1 -- 172.21.15.1:0/4189510817 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6238+0+0 (secure 0 0 0) 0x7fbcf808c9a0 con 0x7fbd10070c40 2024-02-28T16:40:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.916+0000 7fbd0fe00700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbcf406d7e0 0x7fbcf406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.917+0000 7fbd0fe00700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbcf406d7e0 0x7fbcf406fc90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009fd0 tx=0x7fbd00009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:54.921+0000 7fbd0cc00700 1 -- 172.21.15.1:0/4189510817 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbcf8059e60 con 0x7fbd10070c40 2024-02-28T16:40:55.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.191+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbd1010c240 con 0x7fbd10070c40 2024-02-28T16:40:55.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.192+0000 7fbd0cc00700 1 -- 172.21.15.1:0/4189510817 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1848 (secure 0 0 0) 0x7fbcf80599f0 con 0x7fbd10070c40 2024-02-28T16:40:55.181 INFO:teuthology.orchestra.run.smithi001.stdout:e18 2024-02-28T16:40:55.181 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:40:55.181 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 18 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:35:47.667892+0000 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:40:55.182 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=14464,1=24261} 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:40:55.183 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:40:55.184 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:14464} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.1:6826/1940162712,v1:172.21.15.1:6827/1940162712] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:40:55.184 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{1:24261} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.66:6824/1481832428,v1:172.21.15.66:6825/1481832428] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:40:55.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:40:55.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:40:55.184 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:40:55.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:40:55.184 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:40:55.184 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:40:55.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.195+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbcf406d7e0 msgr2=0x7fbcf406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:55.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.195+0000 7fbd16e00700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbcf406d7e0 0x7fbcf406fc90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009fd0 tx=0x7fbd00009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:55.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.195+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd10070c40 msgr2=0x7fbd10116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:55.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.195+0000 7fbd16e00700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd10070c40 0x7fbd10116160 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800eb40 tx=0x7fbcf800b040 comp rx=0 tx=0).stop 2024-02-28T16:40:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.196+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 shutdown_connections 2024-02-28T16:40:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.196+0000 7fbd16e00700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbcf406d7e0 0x7fbcf406fc90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.196+0000 7fbd16e00700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbd10070330 0x7fbd10115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.196+0000 7fbd16e00700 1 --2- 172.21.15.1:0/4189510817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd10070c40 0x7fbd10116160 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.196+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 >> 172.21.15.1:0/4189510817 conn(0x7fbd100f8ae0 msgr2=0x7fbd10103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.196+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 shutdown_connections 2024-02-28T16:40:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.196+0000 7fbd16e00700 1 -- 172.21.15.1:0/4189510817 wait complete. 2024-02-28T16:40:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T16:40:55.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.341+0000 7f1899e00700 1 -- 172.21.15.1:0/2455746940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 msgr2=0x7f18940ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:55.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.341+0000 7f1899e00700 1 --2- 172.21.15.1:0/2455746940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 0x7f18940ff180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f187c004940 tx=0x7f187c009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:55.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.341+0000 7f1899e00700 1 -- 172.21.15.1:0/2455746940 shutdown_connections 2024-02-28T16:40:55.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.341+0000 7f1899e00700 1 --2- 172.21.15.1:0/2455746940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 0x7f18940ff180 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.341+0000 7f1899e00700 1 --2- 172.21.15.1:0/2455746940 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1894105170 0x7f1894105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.341+0000 7f1899e00700 1 -- 172.21.15.1:0/2455746940 >> 172.21.15.1:0/2455746940 conn(0x7f18940face0 msgr2=0x7f18940fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:55.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.341+0000 7f1899e00700 1 -- 172.21.15.1:0/2455746940 shutdown_connections 2024-02-28T16:40:55.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.342+0000 7f1899e00700 1 -- 172.21.15.1:0/2455746940 wait complete. 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.342+0000 7f1899e00700 1 Processor -- start 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.342+0000 7f1899e00700 1 -- start start 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.343+0000 7f1899e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1894105170 0x7f1894115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.343+0000 7f1899e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 0x7f1894116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.343+0000 7f1899e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18941168e0 con 0x7f1894105b10 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.343+0000 7f1899e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189410fbf0 con 0x7f1894105170 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.343+0000 7f188be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 0x7f1894116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.343+0000 7f188be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 0x7f1894116290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47862/0 (socket says 172.21.15.1:47862) 2024-02-28T16:40:55.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.343+0000 7f188be00700 1 -- 172.21.15.1:0/1205492023 learned_addr learned my addr 172.21.15.1:0/1205492023 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f1892a00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1894105170 0x7f1894115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f188be00700 1 -- 172.21.15.1:0/1205492023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1894105170 msgr2=0x7f1894115d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f188be00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1894105170 0x7f1894115d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f188be00700 1 -- 172.21.15.1:0/1205492023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c005040 con 0x7f1894105b10 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f1892a00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1894105170 0x7f1894115d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f188be00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 0x7f1894116290 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f187c000c00 tx=0x7f187c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f188aa00700 1 -- 172.21.15.1:0/1205492023 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187c021070 con 0x7f1894105b10 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f188aa00700 1 -- 172.21.15.1:0/1205492023 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f187c013d00 con 0x7f1894105b10 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189410fe70 con 0x7f1894105b10 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.344+0000 7f188aa00700 1 -- 172.21.15.1:0/1205492023 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187c00e070 con 0x7f1894105b10 2024-02-28T16:40:55.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.345+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1894110350 con 0x7f1894105b10 2024-02-28T16:40:55.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.346+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18940fe900 con 0x7f1894105b10 2024-02-28T16:40:55.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.347+0000 7f188aa00700 1 -- 172.21.15.1:0/1205492023 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f187c01cb60 con 0x7f1894105b10 2024-02-28T16:40:55.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.348+0000 7f188aa00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f187806d7f0 0x7f187806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:55.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.348+0000 7f188aa00700 1 -- 172.21.15.1:0/1205492023 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f187c08e3a0 con 0x7f1894105b10 2024-02-28T16:40:55.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.348+0000 7f1892a00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f187806d7f0 0x7f187806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:55.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.349+0000 7f1892a00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f187806d7f0 0x7f187806fca0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1884004600 tx=0x7f188400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.352+0000 7f188aa00700 1 -- 172.21.15.1:0/1205492023 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f187c05b7b0 con 0x7f1894105b10 2024-02-28T16:40:55.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:55 smithi001 ceph-mon[112940]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:55.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:55 smithi001 ceph-mon[112940]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:55.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:55 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4082680628' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:55.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:55 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4189510817' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:40:55.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.590+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1894111330 con 0x7f187806d7f0 2024-02-28T16:40:55.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.591+0000 7f188aa00700 1 -- 172.21.15.1:0/1205492023 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f1894111330 con 0x7f187806d7f0 2024-02-28T16:40:55.580 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:40:55.580 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:40:55.580 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:40:55.580 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:40:55.580 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:40:55.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:40:55.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:40:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: "crash" 2024-02-28T16:40:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:40:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "11/21 daemons upgraded", 2024-02-28T16:40:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading osd daemons", 2024-02-28T16:40:55.581 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:40:55.581 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f187806d7f0 msgr2=0x7f187806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f187806d7f0 0x7f187806fca0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1884004600 tx=0x7f188400b040 comp rx=0 tx=0).stop 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 msgr2=0x7f1894116290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 0x7f1894116290 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f187c000c00 tx=0x7f187c009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 shutdown_connections 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f187806d7f0 0x7f187806fca0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1894105170 0x7f1894115d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 --2- 172.21.15.1:0/1205492023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1894105b10 0x7f1894116290 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.594+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 >> 172.21.15.1:0/1205492023 conn(0x7f18940face0 msgr2=0x7f18940fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:55.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.595+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 shutdown_connections 2024-02-28T16:40:55.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.595+0000 7f1899e00700 1 -- 172.21.15.1:0/1205492023 wait complete. 2024-02-28T16:40:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.751+0000 7f6c73200700 1 -- 172.21.15.1:0/1058204098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fe8f0 msgr2=0x7f6c6c107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.751+0000 7f6c73200700 1 --2- 172.21.15.1:0/1058204098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fe8f0 0x7f6c6c107050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c58004970 tx=0x7f6c58009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.751+0000 7f6c73200700 1 -- 172.21.15.1:0/1058204098 shutdown_connections 2024-02-28T16:40:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.751+0000 7f6c73200700 1 --2- 172.21.15.1:0/1058204098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fe8f0 0x7f6c6c107050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.751+0000 7f6c73200700 1 --2- 172.21.15.1:0/1058204098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c6c0fdfe0 0x7f6c6c0fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.751+0000 7f6c73200700 1 -- 172.21.15.1:0/1058204098 >> 172.21.15.1:0/1058204098 conn(0x7f6c6c071d90 msgr2=0x7f6c6c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.751+0000 7f6c73200700 1 -- 172.21.15.1:0/1058204098 shutdown_connections 2024-02-28T16:40:55.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.751+0000 7f6c73200700 1 -- 172.21.15.1:0/1058204098 wait complete. 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.752+0000 7f6c73200700 1 Processor -- start 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.752+0000 7f6c73200700 1 -- start start 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.753+0000 7f6c73200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fdfe0 0x7f6c6c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.753+0000 7f6c73200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c6c0fe8f0 0x7f6c6c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.753+0000 7f6c73200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c116790 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.753+0000 7f6c73200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c10fac0 con 0x7f6c6c0fe8f0 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.753+0000 7f6c6be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fdfe0 0x7f6c6c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.753+0000 7f6c6be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fdfe0 0x7f6c6c115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47872/0 (socket says 172.21.15.1:47872) 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.753+0000 7f6c6be00700 1 -- 172.21.15.1:0/178916934 learned_addr learned my addr 172.21.15.1:0/178916934 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.753+0000 7f6c6b400700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c6c0fe8f0 0x7f6c6c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:55.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.754+0000 7f6c6be00700 1 -- 172.21.15.1:0/178916934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c6c0fe8f0 msgr2=0x7f6c6c116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.754+0000 7f6c6be00700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c6c0fe8f0 0x7f6c6c116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.754+0000 7f6c6be00700 1 -- 172.21.15.1:0/178916934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c58005040 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.754+0000 7f6c6be00700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fdfe0 0x7f6c6c115c00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00eb40 tx=0x7f6c5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:55.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.754+0000 7f6c63e00700 1 -- 172.21.15.1:0/178916934 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c5c007cc0 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.754+0000 7f6c63e00700 1 -- 172.21.15.1:0/178916934 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c5c00f040 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.754+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c10fd40 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.755+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c110250 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.755+0000 7f6c63e00700 1 -- 172.21.15.1:0/178916934 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c5c00ca10 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.756+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c106b20 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.757+0000 7f6c63e00700 1 -- 172.21.15.1:0/178916934 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6c5c00cc80 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.758+0000 7f6c63e00700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c5006d7f0 0x7f6c5006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:40:55.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.758+0000 7f6c63e00700 1 -- 172.21.15.1:0/178916934 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6238+0+0 (secure 0 0 0) 0x7f6c5c08ca50 con 0x7f6c6c0fdfe0 2024-02-28T16:40:55.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:55 smithi066 ceph-mon[85781]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:55.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:55 smithi066 ceph-mon[85781]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:55.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:55 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4082680628' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:40:55.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:55 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4189510817' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:40:55.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.758+0000 7f6c6b400700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c5006d7f0 0x7f6c5006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:40:55.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.759+0000 7f6c6b400700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c5006d7f0 0x7f6c5006fca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c110df0 tx=0x7f6c58009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:40:55.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:55.763+0000 7f6c63e00700 1 -- 172.21.15.1:0/178916934 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c5c059e90 con 0x7f6c6c0fdfe0 2024-02-28T16:40:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.071+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6c6c04bed0 con 0x7f6c6c0fdfe0 2024-02-28T16:40:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.071+0000 7f6c63e00700 1 -- 172.21.15.1:0/178916934 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1480 (secure 0 0 0) 0x7f6c5c059a20 con 0x7f6c6c0fdfe0 2024-02-28T16:40:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 56/372 objects degraded (15.054%), 14 pgs degraded 2024-02-28T16:40:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:40:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:40:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-28T16:40:56.060 INFO:teuthology.orchestra.run.smithi001.stdout: osd.5 (root=default,host=smithi066) is down 2024-02-28T16:40:56.060 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 56/372 objects degraded (15.054%), 14 pgs degraded 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.2 is active+undersized+degraded, acting [1,0] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.3 is active+undersized+degraded, acting [2,1] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.8 is active+undersized+degraded, acting [3,0] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.b is active+undersized+degraded, acting [3,4] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.16 is active+undersized+degraded, acting [3,2] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.17 is active+undersized+degraded, acting [2,1] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.18 is active+undersized+degraded, acting [4,3] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.1a is active+undersized+degraded, acting [3,4] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.1b is active+undersized+degraded, acting [1,0] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.1c is active+undersized+degraded, acting [4,2] 2024-02-28T16:40:56.061 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-02-28T16:40:56.062 INFO:teuthology.orchestra.run.smithi001.stdout: pg 2.1e is active+undersized+degraded, acting [2,0] 2024-02-28T16:40:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.075+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c5006d7f0 msgr2=0x7f6c5006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.075+0000 7f6c73200700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c5006d7f0 0x7f6c5006fca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c110df0 tx=0x7f6c58009e30 comp rx=0 tx=0).stop 2024-02-28T16:40:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.075+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fdfe0 msgr2=0x7f6c6c115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:40:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.075+0000 7f6c73200700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fdfe0 0x7f6c6c115c00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00eb40 tx=0x7f6c5c00b040 comp rx=0 tx=0).stop 2024-02-28T16:40:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.076+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 shutdown_connections 2024-02-28T16:40:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.076+0000 7f6c73200700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6c5006d7f0 0x7f6c5006fca0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.076+0000 7f6c73200700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6c0fdfe0 0x7f6c6c115c00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.076+0000 7f6c73200700 1 --2- 172.21.15.1:0/178916934 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6c6c0fe8f0 0x7f6c6c116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:40:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.076+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 >> 172.21.15.1:0/178916934 conn(0x7f6c6c071d90 msgr2=0x7f6c6c0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:40:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.076+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 shutdown_connections 2024-02-28T16:40:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:40:56.076+0000 7f6c73200700 1 -- 172.21.15.1:0/178916934 wait complete. 2024-02-28T16:40:56.561 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T16:40:56.260+0000 7f82230f57c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-28T16:40:56.561 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T16:40:56.573+0000 7f82230f57c0 -1 osd.5 73 log_to_monitors true 2024-02-28T16:40:56.561 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-mon[85781]: pgmap v187: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:56.561 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:56.562 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:40:56.562 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 56/372 objects degraded (15.054%), 14 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:56.562 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:56.562 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:56.562 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:56 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/178916934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:40:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:56 smithi001 ceph-mon[112940]: pgmap v187: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:40:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:56 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 56/372 objects degraded (15.054%), 14 pgs degraded (PG_DEGRADED) 2024-02-28T16:40:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:56.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:56 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/178916934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:40:57.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:57 smithi066 ceph-mon[85781]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:57.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:57 smithi066 ceph-mon[85781]: from='osd.5 [v2:172.21.15.66:6816/2636222889,v1:172.21.15.66:6817/2636222889]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-28T16:40:57.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:57 smithi066 ceph-mon[85781]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-28T16:40:57.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:57 smithi001 ceph-mon[112940]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:40:57.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:57 smithi001 ceph-mon[112940]: from='osd.5 [v2:172.21.15.66:6816/2636222889,v1:172.21.15.66:6817/2636222889]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-28T16:40:57.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:57 smithi001 ceph-mon[112940]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-28T16:40:58.747 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 16:40:58 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T16:40:58.597+0000 7f820d400700 -1 osd.5 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-28T16:40:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:58 smithi066 ceph-mon[85781]: pgmap v188: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:58 smithi066 ceph-mon[85781]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-28T16:40:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:58 smithi066 ceph-mon[85781]: from='osd.5 [v2:172.21.15.66:6816/2636222889,v1:172.21.15.66:6817/2636222889]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:40:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:58 smithi066 ceph-mon[85781]: osdmap e76: 6 total, 5 up, 6 in 2024-02-28T16:40:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:58 smithi066 ceph-mon[85781]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:40:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:58 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:58 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:58 smithi001 ceph-mon[112940]: pgmap v188: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:58 smithi001 ceph-mon[112940]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-28T16:40:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:58 smithi001 ceph-mon[112940]: from='osd.5 [v2:172.21.15.66:6816/2636222889,v1:172.21.15.66:6817/2636222889]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:40:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:58 smithi001 ceph-mon[112940]: osdmap e76: 6 total, 5 up, 6 in 2024-02-28T16:40:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:58 smithi001 ceph-mon[112940]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-02-28T16:40:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:58 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:58.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:58 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:40:59.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:59 smithi066 ceph-mon[85781]: from='osd.5 ' entity='osd.5' 2024-02-28T16:40:59.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:40:59 smithi066 ceph-mon[85781]: pgmap v190: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:40:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:59 smithi001 ceph-mon[112940]: from='osd.5 ' entity='osd.5' 2024-02-28T16:40:59.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:40:59 smithi001 ceph-mon[112940]: pgmap v190: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:41:00.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:00 smithi066 ceph-mon[85781]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:41:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:00 smithi066 ceph-mon[85781]: osd.5 [v2:172.21.15.66:6816/2636222889,v1:172.21.15.66:6817/2636222889] boot 2024-02-28T16:41:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:00 smithi066 ceph-mon[85781]: osdmap e77: 6 total, 6 up, 6 in 2024-02-28T16:41:00.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:00 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:41:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:00 smithi001 ceph-mon[112940]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-28T16:41:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:00 smithi001 ceph-mon[112940]: osd.5 [v2:172.21.15.66:6816/2636222889,v1:172.21.15.66:6817/2636222889] boot 2024-02-28T16:41:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:00 smithi001 ceph-mon[112940]: osdmap e77: 6 total, 6 up, 6 in 2024-02-28T16:41:01.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-28T16:41:01.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:01 smithi066 ceph-mon[85781]: osdmap e78: 6 total, 6 up, 6 in 2024-02-28T16:41:01.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:01 smithi066 ceph-mon[85781]: pgmap v193: 65 pgs: 1 peering, 14 active+undersized+degraded, 13 active+undersized, 37 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:41:02.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:01 smithi001 ceph-mon[112940]: osdmap e78: 6 total, 6 up, 6 in 2024-02-28T16:41:02.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:01 smithi001 ceph-mon[112940]: pgmap v193: 65 pgs: 1 peering, 14 active+undersized+degraded, 13 active+undersized, 37 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/372 objects degraded (15.054%) 2024-02-28T16:41:03.134 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:02 smithi066 ceph-mon[85781]: Health check update: Degraded data redundancy: 33/372 objects degraded (8.871%), 7 pgs degraded (PG_DEGRADED) 2024-02-28T16:41:03.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:02 smithi001 ceph-mon[112940]: Health check update: Degraded data redundancy: 33/372 objects degraded (8.871%), 7 pgs degraded (PG_DEGRADED) 2024-02-28T16:41:03.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:03 smithi066 ceph-mon[85781]: pgmap v194: 65 pgs: 1 peering, 7 active+undersized+degraded, 3 active+undersized, 54 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 33/372 objects degraded (8.871%) 2024-02-28T16:41:04.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:03 smithi001 ceph-mon[112940]: pgmap v194: 65 pgs: 1 peering, 7 active+undersized+degraded, 3 active+undersized, 54 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 33/372 objects degraded (8.871%) 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-28T16:41:05.499 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-28T16:41:05.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:05.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-28T16:41:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-28T16:41:05.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-28T16:41:06.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:05 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:41:05.748+0000 7f7e3d800700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-28T16:41:06.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all osd 2024-02-28T16:41:06.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: Upgrade: Setting require_osd_release to 19 squid 2024-02-28T16:41:06.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: pgmap v195: 65 pgs: 1 peering, 64 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:06.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/372 objects degraded (8.871%), 7 pgs degraded) 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: osdmap e79: 6 total, 6 up, 6 in 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: osdmap e80: 6 total, 6 up, 6 in 2024-02-28T16:41:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:06 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-02-28T16:41:06.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all osd 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: Upgrade: Setting require_osd_release to 19 squid 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: pgmap v195: 65 pgs: 1 peering, 64 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/372 objects degraded (8.871%), 7 pgs degraded) 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: osdmap e79: 6 total, 6 up, 6 in 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: osdmap e80: 6 total, 6 up, 6 in 2024-02-28T16:41:06.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:06 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-02-28T16:41:07.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:07 smithi066 ceph-mon[85781]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-02-28T16:41:07.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:07 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] up:boot 2024-02-28T16:41:07.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:07 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6826/996956384,v1:172.21.15.1:6827/996956384] up:boot 2024-02-28T16:41:07.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:07 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T16:41:07.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:07 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:41:07.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:07 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:41:07.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:07 smithi001 ceph-mon[112940]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-02-28T16:41:07.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:07 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] up:boot 2024-02-28T16:41:07.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:07 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6826/996956384,v1:172.21.15.1:6827/996956384] up:boot 2024-02-28T16:41:07.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:07 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T16:41:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:07 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:41:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:07 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:41:08.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:08 smithi001 ceph-mon[112940]: pgmap v198: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:08.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:08.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.awxxdo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:41:08.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:41:08.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:08 smithi066 ceph-mon[85781]: pgmap v198: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:08.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:08.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.awxxdo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:41:08.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:41:09.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:09 smithi001 ceph-mon[112940]: Upgrade: Updating mds.cephfs.smithi001.awxxdo 2024-02-28T16:41:09.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:09 smithi001 ceph-mon[112940]: Deploying daemon mds.cephfs.smithi001.awxxdo on smithi001 2024-02-28T16:41:09.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:09 smithi066 ceph-mon[85781]: Upgrade: Updating mds.cephfs.smithi001.awxxdo 2024-02-28T16:41:09.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:09 smithi066 ceph-mon[85781]: Deploying daemon mds.cephfs.smithi001.awxxdo on smithi001 2024-02-28T16:41:10.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:10 smithi001 ceph-mon[112940]: pgmap v199: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:10.649 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:10 smithi066 ceph-mon[85781]: pgmap v199: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:41:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:11 smithi001 ceph-mon[112940]: pgmap v200: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:12.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:12.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:12.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:41:12.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:11 smithi066 ceph-mon[85781]: pgmap v200: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:12.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:13.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:12 smithi001 ceph-mon[112940]: osdmap e81: 6 total, 6 up, 6 in 2024-02-28T16:41:13.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:12 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-02-28T16:41:13.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:12 smithi066 ceph-mon[85781]: osdmap e81: 6 total, 6 up, 6 in 2024-02-28T16:41:13.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:12 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-02-28T16:41:14.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:13 smithi001 ceph-mon[112940]: pgmap v202: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:14.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:13 smithi066 ceph-mon[85781]: pgmap v202: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:16.014 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:15 smithi001 ceph-mon[112940]: pgmap v203: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:16.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:15 smithi066 ceph-mon[85781]: pgmap v203: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:17.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:17 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:17.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:17 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:17.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:17 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:41:17.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:17 smithi001 ceph-mon[112940]: pgmap v204: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:17.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:17 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:boot 2024-02-28T16:41:17.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:17 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T16:41:17.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:17 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:41:17.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:17 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:17.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:17 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:17.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:17 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:41:17.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:17 smithi066 ceph-mon[85781]: pgmap v204: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:17 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:boot 2024-02-28T16:41:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:17 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T16:41:17.998 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:17 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T16:41:20.105 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:19 smithi001 ceph-mon[112940]: pgmap v205: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:20.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:19 smithi066 ceph-mon[85781]: pgmap v205: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:21.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:21 smithi066 ceph-mon[85781]: pgmap v206: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:22.080 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:21 smithi001 ceph-mon[112940]: pgmap v206: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:24.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:23 smithi066 ceph-mon[85781]: pgmap v207: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:24.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:23 smithi001 ceph-mon[112940]: pgmap v207: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:25.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:24 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:25.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:24 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:25.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:24 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:25.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:24 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.241+0000 7f87b2400700 1 -- 172.21.15.1:0/1201964777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 msgr2=0x7f87ac103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.241+0000 7f87b2400700 1 --2- 172.21.15.1:0/1201964777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 0x7f87ac103320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f879c004940 tx=0x7f879c009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.241+0000 7f87b2400700 1 -- 172.21.15.1:0/1201964777 shutdown_connections 2024-02-28T16:41:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.241+0000 7f87b2400700 1 --2- 172.21.15.1:0/1201964777 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87ac1038f0 0x7f87ac0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.241+0000 7f87b2400700 1 --2- 172.21.15.1:0/1201964777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 0x7f87ac103320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.241+0000 7f87b2400700 1 -- 172.21.15.1:0/1201964777 >> 172.21.15.1:0/1201964777 conn(0x7f87ac0f8ae0 msgr2=0x7f87ac0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.242+0000 7f87b2400700 1 -- 172.21.15.1:0/1201964777 shutdown_connections 2024-02-28T16:41:26.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.242+0000 7f87b2400700 1 -- 172.21.15.1:0/1201964777 wait complete. 2024-02-28T16:41:26.230 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:25 smithi001 ceph-mon[112940]: pgmap v208: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:26.230 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:41:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.243+0000 7f87b2400700 1 Processor -- start 2024-02-28T16:41:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.243+0000 7f87b2400700 1 -- start start 2024-02-28T16:41:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.243+0000 7f87b2400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 0x7f87ac115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.243+0000 7f87b2400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87ac1038f0 0x7f87ac1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.243+0000 7f87b2400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac116890 con 0x7f87ac102f50 2024-02-28T16:41:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.243+0000 7f87b2400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac10fb30 con 0x7f87ac1038f0 2024-02-28T16:41:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.243+0000 7f87ab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 0x7f87ac115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.244+0000 7f87ab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 0x7f87ac115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39640/0 (socket says 172.21.15.1:39640) 2024-02-28T16:41:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.244+0000 7f87ab400700 1 -- 172.21.15.1:0/2798018744 learned_addr learned my addr 172.21.15.1:0/2798018744 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.244+0000 7f87aaa00700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87ac1038f0 0x7f87ac1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.244+0000 7f87ab400700 1 -- 172.21.15.1:0/2798018744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87ac1038f0 msgr2=0x7f87ac1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.244+0000 7f87ab400700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87ac1038f0 0x7f87ac1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.244+0000 7f87ab400700 1 -- 172.21.15.1:0/2798018744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c005040 con 0x7f87ac102f50 2024-02-28T16:41:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.244+0000 7f87ab400700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 0x7f87ac115c70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f879c009100 tx=0x7f879c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.245+0000 7f87a3400700 1 -- 172.21.15.1:0/2798018744 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c00e040 con 0x7f87ac102f50 2024-02-28T16:41:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.245+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ac10fdb0 con 0x7f87ac102f50 2024-02-28T16:41:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.245+0000 7f87a3400700 1 -- 172.21.15.1:0/2798018744 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f879c004290 con 0x7f87ac102f50 2024-02-28T16:41:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.245+0000 7f87a3400700 1 -- 172.21.15.1:0/2798018744 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c01c630 con 0x7f87ac102f50 2024-02-28T16:41:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.245+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ac1102f0 con 0x7f87ac102f50 2024-02-28T16:41:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.246+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ac0fc6e0 con 0x7f87ac102f50 2024-02-28T16:41:26.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.247+0000 7f87a3400700 1 -- 172.21.15.1:0/2798018744 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f879c00bbf0 con 0x7f87ac102f50 2024-02-28T16:41:26.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.248+0000 7f87a3400700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f879006d7e0 0x7f879006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.248+0000 7f87a3400700 1 -- 172.21.15.1:0/2798018744 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f879c08e260 con 0x7f87ac102f50 2024-02-28T16:41:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.249+0000 7f87aaa00700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f879006d7e0 0x7f879006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.250+0000 7f87aaa00700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f879006d7e0 0x7f879006fc90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8798006fd0 tx=0x7f8798008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.254+0000 7f87a3400700 1 -- 172.21.15.1:0/2798018744 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f879c05b660 con 0x7f87ac102f50 2024-02-28T16:41:26.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:25 smithi066 ceph-mon[85781]: pgmap v208: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:26.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:41:26.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.500+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87ac111280 con 0x7f879006d7e0 2024-02-28T16:41:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.502+0000 7f87a3400700 1 -- 172.21.15.1:0/2798018744 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f87ac111280 con 0x7f879006d7e0 2024-02-28T16:41:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f879006d7e0 msgr2=0x7f879006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f879006d7e0 0x7f879006fc90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8798006fd0 tx=0x7f8798008040 comp rx=0 tx=0).stop 2024-02-28T16:41:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 msgr2=0x7f87ac115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 0x7f87ac115c70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f879c009100 tx=0x7f879c009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 shutdown_connections 2024-02-28T16:41:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f879006d7e0 0x7f879006fc90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ac102f50 0x7f87ac115c70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 --2- 172.21.15.1:0/2798018744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f87ac1038f0 0x7f87ac1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.506+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 >> 172.21.15.1:0/2798018744 conn(0x7f87ac0f8ae0 msgr2=0x7f87ac0fa320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.507+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 shutdown_connections 2024-02-28T16:41:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.507+0000 7f87b2400700 1 -- 172.21.15.1:0/2798018744 wait complete. 2024-02-28T16:41:26.512 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:41:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.668+0000 7f5331800700 1 -- 172.21.15.1:0/4270170785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 msgr2=0x7f532c109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.668+0000 7f5331800700 1 --2- 172.21.15.1:0/4270170785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 0x7f532c109370 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5318004940 tx=0x7f5318009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.668+0000 7f5331800700 1 -- 172.21.15.1:0/4270170785 shutdown_connections 2024-02-28T16:41:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.668+0000 7f5331800700 1 --2- 172.21.15.1:0/4270170785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 0x7f532c109370 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.668+0000 7f5331800700 1 --2- 172.21.15.1:0/4270170785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f532c0fbd00 0x7f532c0fc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.668+0000 7f5331800700 1 -- 172.21.15.1:0/4270170785 >> 172.21.15.1:0/4270170785 conn(0x7f532c0f79d0 msgr2=0x7f532c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.668+0000 7f5331800700 1 -- 172.21.15.1:0/4270170785 shutdown_connections 2024-02-28T16:41:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.668+0000 7f5331800700 1 -- 172.21.15.1:0/4270170785 wait complete. 2024-02-28T16:41:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.669+0000 7f5331800700 1 Processor -- start 2024-02-28T16:41:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.669+0000 7f5331800700 1 -- start start 2024-02-28T16:41:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.670+0000 7f5331800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f532c0fbd00 0x7f532c115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.670+0000 7f5331800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 0x7f532c116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.670+0000 7f5331800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c116780 con 0x7f532c0fc610 2024-02-28T16:41:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.670+0000 7f5331800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c10fab0 con 0x7f532c0fbd00 2024-02-28T16:41:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.670+0000 7f532a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 0x7f532c116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.670+0000 7f532a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 0x7f532c116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39662/0 (socket says 172.21.15.1:39662) 2024-02-28T16:41:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.670+0000 7f532a000700 1 -- 172.21.15.1:0/3828749424 learned_addr learned my addr 172.21.15.1:0/3828749424 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.670+0000 7f532aa00700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f532c0fbd00 0x7f532c115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.671+0000 7f532a000700 1 -- 172.21.15.1:0/3828749424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f532c0fbd00 msgr2=0x7f532c115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.671+0000 7f532a000700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f532c0fbd00 0x7f532c115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.671+0000 7f532a000700 1 -- 172.21.15.1:0/3828749424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5318005040 con 0x7f532c0fc610 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.671+0000 7f532a000700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 0x7f532c116130 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5318009180 tx=0x7f5318009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.671+0000 7f5322a00700 1 -- 172.21.15.1:0/3828749424 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f531800e040 con 0x7f532c0fc610 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.671+0000 7f5322a00700 1 -- 172.21.15.1:0/3828749424 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f531800bbf0 con 0x7f532c0fc610 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.671+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c10fd90 con 0x7f532c0fc610 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.671+0000 7f5322a00700 1 -- 172.21.15.1:0/3828749424 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f531801c630 con 0x7f532c0fc610 2024-02-28T16:41:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.672+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c1102d0 con 0x7f532c0fc610 2024-02-28T16:41:26.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.673+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532c106aa0 con 0x7f532c0fc610 2024-02-28T16:41:26.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.674+0000 7f5322a00700 1 -- 172.21.15.1:0/3828749424 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f531800bd60 con 0x7f532c0fc610 2024-02-28T16:41:26.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.675+0000 7f5322a00700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f531006d7f0 0x7f531006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:26.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.675+0000 7f5322a00700 1 -- 172.21.15.1:0/3828749424 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f531808e370 con 0x7f532c0fc610 2024-02-28T16:41:26.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.675+0000 7f532aa00700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f531006d7f0 0x7f531006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:26.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.676+0000 7f532aa00700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f531006d7f0 0x7f531006fca0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f531c004990 tx=0x7f531c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.681+0000 7f5322a00700 1 -- 172.21.15.1:0/3828749424 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f531805b6d0 con 0x7f532c0fc610 2024-02-28T16:41:26.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.949+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f532c10c150 con 0x7f531006d7f0 2024-02-28T16:41:26.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.951+0000 7f5322a00700 1 -- 172.21.15.1:0/3828749424 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f532c10c150 con 0x7f531006d7f0 2024-02-28T16:41:26.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f531006d7f0 msgr2=0x7f531006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:26.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f531006d7f0 0x7f531006fca0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f531c004990 tx=0x7f531c009ee0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 msgr2=0x7f532c116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:26.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 0x7f532c116130 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5318009180 tx=0x7f5318009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 shutdown_connections 2024-02-28T16:41:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f531006d7f0 0x7f531006fca0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f532c0fbd00 0x7f532c115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 --2- 172.21.15.1:0/3828749424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f532c0fc610 0x7f532c116130 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 >> 172.21.15.1:0/3828749424 conn(0x7f532c0f79d0 msgr2=0x7f532c0f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 shutdown_connections 2024-02-28T16:41:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:26.954+0000 7f5331800700 1 -- 172.21.15.1:0/3828749424 wait complete. 2024-02-28T16:41:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.104+0000 7effdf200700 1 -- 172.21.15.1:0/407101156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 msgr2=0x7effd80ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.104+0000 7effdf200700 1 --2- 172.21.15.1:0/407101156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 0x7effd80ff180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7effc4004940 tx=0x7effc4009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.104+0000 7effdf200700 1 -- 172.21.15.1:0/407101156 shutdown_connections 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.104+0000 7effdf200700 1 --2- 172.21.15.1:0/407101156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 0x7effd80ff180 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.104+0000 7effdf200700 1 --2- 172.21.15.1:0/407101156 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effd8105170 0x7effd8105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.104+0000 7effdf200700 1 -- 172.21.15.1:0/407101156 >> 172.21.15.1:0/407101156 conn(0x7effd80face0 msgr2=0x7effd80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.104+0000 7effdf200700 1 -- 172.21.15.1:0/407101156 shutdown_connections 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.104+0000 7effdf200700 1 -- 172.21.15.1:0/407101156 wait complete. 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.105+0000 7effdf200700 1 Processor -- start 2024-02-28T16:41:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.105+0000 7effdf200700 1 -- start start 2024-02-28T16:41:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.106+0000 7effdf200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effd8105170 0x7effd8114970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.106+0000 7effdf200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 0x7effd8075510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.106+0000 7effdf200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd8115000 con 0x7effd8105b10 2024-02-28T16:41:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.106+0000 7effdf200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd8075a50 con 0x7effd8105170 2024-02-28T16:41:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.106+0000 7effd7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 0x7effd8075510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.106+0000 7effd7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 0x7effd8075510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39678/0 (socket says 172.21.15.1:39678) 2024-02-28T16:41:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.106+0000 7effd7400700 1 -- 172.21.15.1:0/3080598380 learned_addr learned my addr 172.21.15.1:0/3080598380 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effd7e00700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effd8105170 0x7effd8114970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effd7400700 1 -- 172.21.15.1:0/3080598380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effd8105170 msgr2=0x7effd8114970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effd7400700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effd8105170 0x7effd8114970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effd7400700 1 -- 172.21.15.1:0/3080598380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc4005040 con 0x7effd8105b10 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effd7e00700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effd8105170 0x7effd8114970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effd7400700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 0x7effd8075510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7effc4009030 tx=0x7effc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effcfe00700 1 -- 172.21.15.1:0/3080598380 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effc400e040 con 0x7effd8105b10 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effcfe00700 1 -- 172.21.15.1:0/3080598380 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc400bbf0 con 0x7effd8105b10 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd8075cd0 con 0x7effd8105b10 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.107+0000 7effcfe00700 1 -- 172.21.15.1:0/3080598380 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effc401c630 con 0x7effd8105b10 2024-02-28T16:41:27.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.108+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd8076210 con 0x7effd8105b10 2024-02-28T16:41:27.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.109+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd80fe900 con 0x7effd8105b10 2024-02-28T16:41:27.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.110+0000 7effcfe00700 1 -- 172.21.15.1:0/3080598380 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7effc400bd60 con 0x7effd8105b10 2024-02-28T16:41:27.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.111+0000 7effcfe00700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effbc06d7f0 0x7effbc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:27.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.111+0000 7effcfe00700 1 -- 172.21.15.1:0/3080598380 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6410+0+0 (secure 0 0 0) 0x7effc408e1f0 con 0x7effd8105b10 2024-02-28T16:41:27.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.111+0000 7effd7e00700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effbc06d7f0 0x7effbc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:27.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.112+0000 7effd7e00700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effbc06d7f0 0x7effbc06fca0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7effc8007fd0 tx=0x7effc8009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.116+0000 7effcfe00700 1 -- 172.21.15.1:0/3080598380 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7effc405b5f0 con 0x7effd8105b10 2024-02-28T16:41:27.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:27 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:27.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:27 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:27.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:27 smithi001 ceph-mon[112940]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:41:27.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.357+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7effd810ce90 con 0x7effbc06d7f0 2024-02-28T16:41:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.380+0000 7effcfe00700 1 -- 172.21.15.1:0/3080598380 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7effd810ce90 con 0x7effbc06d7f0 2024-02-28T16:41:27.368 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:41:27.368 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (7m) 3s ago 21m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (4m) 3s ago 22m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (4m) 31s ago 20m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (7m) 3s ago 21m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (11s) 3s ago 14m 10.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (14m) 3s ago 14m 15.6M - 17.2.7-535-gc0f02e60 1c37ed371348 c85f1b8bf4e3 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (14m) 31s ago 14m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (14m) 31s ago 14m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (10m) 3s ago 23m 507M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (9m) 31s ago 20m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (5m) 3s ago 23m 52.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (5m) 31s ago 20m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:41:27.369 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (8m) 3s ago 22m 8040k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:41:27.370 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (8m) 31s ago 20m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:41:27.370 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (3m) 3s ago 19m 86.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:41:27.370 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (3m) 3s ago 18m 92.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:41:27.370 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (2m) 3s ago 18m 82.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:41:27.370 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (2m) 31s ago 17m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:41:27.370 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (81s) 31s ago 17m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:41:27.370 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (36s) 31s ago 16m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:41:27.370 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (8m) 3s ago 21m 50.2M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:41:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effbc06d7f0 msgr2=0x7effbc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effbc06d7f0 0x7effbc06fca0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7effc8007fd0 tx=0x7effc8009040 comp rx=0 tx=0).stop 2024-02-28T16:41:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 msgr2=0x7effd8075510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 0x7effd8075510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7effc4009030 tx=0x7effc4009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 shutdown_connections 2024-02-28T16:41:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effbc06d7f0 0x7effbc06fca0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effd8105170 0x7effd8114970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 --2- 172.21.15.1:0/3080598380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd8105b10 0x7effd8075510 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.383+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 >> 172.21.15.1:0/3080598380 conn(0x7effd80face0 msgr2=0x7effd80fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.384+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 shutdown_connections 2024-02-28T16:41:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.384+0000 7effdf200700 1 -- 172.21.15.1:0/3080598380 wait complete. 2024-02-28T16:41:27.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:27 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:27.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:27 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:27.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:27 smithi066 ceph-mon[85781]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:41:27.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.524+0000 7fed0f800700 1 -- 172.21.15.1:0/2402163278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 msgr2=0x7fed08104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:27.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.524+0000 7fed0f800700 1 --2- 172.21.15.1:0/2402163278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 0x7fed08104210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fecf4004940 tx=0x7fecf4009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:27.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.524+0000 7fed0f800700 1 -- 172.21.15.1:0/2402163278 shutdown_connections 2024-02-28T16:41:27.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.524+0000 7fed0f800700 1 --2- 172.21.15.1:0/2402163278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 0x7fed08104210 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.524+0000 7fed0f800700 1 --2- 172.21.15.1:0/2402163278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed081095f0 0x7fed081099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.524+0000 7fed0f800700 1 -- 172.21.15.1:0/2402163278 >> 172.21.15.1:0/2402163278 conn(0x7fed08072d50 msgr2=0x7fed08075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:27.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.525+0000 7fed0f800700 1 -- 172.21.15.1:0/2402163278 shutdown_connections 2024-02-28T16:41:27.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.525+0000 7fed0f800700 1 -- 172.21.15.1:0/2402163278 wait complete. 2024-02-28T16:41:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.526+0000 7fed0f800700 1 Processor -- start 2024-02-28T16:41:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.526+0000 7fed0f800700 1 -- start start 2024-02-28T16:41:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.526+0000 7fed0f800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 0x7fed0811a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.526+0000 7fed0f800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed081095f0 0x7fed0811a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.526+0000 7fed0f800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0811ac50 con 0x7fed08103dc0 2024-02-28T16:41:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.526+0000 7fed0f800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08113ed0 con 0x7fed081095f0 2024-02-28T16:41:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.526+0000 7fed0ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 0x7fed0811a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.527+0000 7fed0ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 0x7fed0811a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39690/0 (socket says 172.21.15.1:39690) 2024-02-28T16:41:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.527+0000 7fed0ce00700 1 -- 172.21.15.1:0/653237351 learned_addr learned my addr 172.21.15.1:0/653237351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.527+0000 7fed0ce00700 1 -- 172.21.15.1:0/653237351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed081095f0 msgr2=0x7fed0811a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.527+0000 7fed0ce00700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed081095f0 0x7fed0811a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.527+0000 7fed0ce00700 1 -- 172.21.15.1:0/653237351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf4005040 con 0x7fed08103dc0 2024-02-28T16:41:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.527+0000 7fed0ce00700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 0x7fed0811a030 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00b790 tx=0x7fecfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.528+0000 7fed04c00700 1 -- 172.21.15.1:0/653237351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecfc00f770 con 0x7fed08103dc0 2024-02-28T16:41:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.528+0000 7fed04c00700 1 -- 172.21.15.1:0/653237351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecfc00fda0 con 0x7fed08103dc0 2024-02-28T16:41:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.528+0000 7fed04c00700 1 -- 172.21.15.1:0/653237351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecfc00d310 con 0x7fed08103dc0 2024-02-28T16:41:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.528+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed08114150 con 0x7fed08103dc0 2024-02-28T16:41:27.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.528+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed08114690 con 0x7fed08103dc0 2024-02-28T16:41:27.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.529+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed08075870 con 0x7fed08103dc0 2024-02-28T16:41:27.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.530+0000 7fed04c00700 1 -- 172.21.15.1:0/653237351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fecfc010420 con 0x7fed08103dc0 2024-02-28T16:41:27.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.531+0000 7fed04c00700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fecf006d7f0 0x7fecf006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.531+0000 7fed04c00700 1 -- 172.21.15.1:0/653237351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fecfc08cef0 con 0x7fed08103dc0 2024-02-28T16:41:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.531+0000 7fed07e00700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fecf006d7f0 0x7fecf006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.532+0000 7fed07e00700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fecf006d7f0 0x7fecf006fca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fecf4004f90 tx=0x7fecf4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.536+0000 7fed04c00700 1 -- 172.21.15.1:0/653237351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fecfc05a2d0 con 0x7fed08103dc0 2024-02-28T16:41:27.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.854+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fed081156b0 con 0x7fed08103dc0 2024-02-28T16:41:27.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.855+0000 7fed04c00700 1 -- 172.21.15.1:0/653237351 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fecfc059e60 con 0x7fed08103dc0 2024-02-28T16:41:27.842 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 3, 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 1 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:41:27.843 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:41:27.844 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 3, 2024-02-28T16:41:27.844 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 11 2024-02-28T16:41:27.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:41:27.844 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fecf006d7f0 msgr2=0x7fecf006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fecf006d7f0 0x7fecf006fca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fecf4004f90 tx=0x7fecf4009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 msgr2=0x7fed0811a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 0x7fed0811a030 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00b790 tx=0x7fecfc009250 comp rx=0 tx=0).stop 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 shutdown_connections 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fecf006d7f0 0x7fecf006fca0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed08103dc0 0x7fed0811a030 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 --2- 172.21.15.1:0/653237351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fed081095f0 0x7fed0811a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.858+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 >> 172.21.15.1:0/653237351 conn(0x7fed08072d50 msgr2=0x7fed080746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.859+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 shutdown_connections 2024-02-28T16:41:27.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:27.859+0000 7fed0f800700 1 -- 172.21.15.1:0/653237351 wait complete. 2024-02-28T16:41:28.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.021+0000 7efcef200700 1 -- 172.21.15.1:0/3605275547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf0070be0 msgr2=0x7efcf01090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.021+0000 7efcef200700 1 --2- 172.21.15.1:0/3605275547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf0070be0 0x7efcf01090e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efcdc004970 tx=0x7efcdc009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:28.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.021+0000 7efcef200700 1 -- 172.21.15.1:0/3605275547 shutdown_connections 2024-02-28T16:41:28.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.021+0000 7efcef200700 1 --2- 172.21.15.1:0/3605275547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf0070be0 0x7efcf01090e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.021+0000 7efcef200700 1 --2- 172.21.15.1:0/3605275547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efcf00702d0 0x7efcf00706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.021+0000 7efcef200700 1 -- 172.21.15.1:0/3605275547 >> 172.21.15.1:0/3605275547 conn(0x7efcf00f8920 msgr2=0x7efcf00fad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:28.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.021+0000 7efcef200700 1 -- 172.21.15.1:0/3605275547 shutdown_connections 2024-02-28T16:41:28.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.021+0000 7efcef200700 1 -- 172.21.15.1:0/3605275547 wait complete. 2024-02-28T16:41:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.022+0000 7efcef200700 1 Processor -- start 2024-02-28T16:41:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.022+0000 7efcef200700 1 -- start start 2024-02-28T16:41:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.022+0000 7efcef200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf00702d0 0x7efcf01159f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.022+0000 7efcef200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efcf0070be0 0x7efcf0115f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.022+0000 7efcef200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf0116580 con 0x7efcf00702d0 2024-02-28T16:41:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.022+0000 7efcef200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf010f8b0 con 0x7efcf0070be0 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.023+0000 7efcede00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf00702d0 0x7efcf01159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.023+0000 7efcede00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf00702d0 0x7efcf01159f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39716/0 (socket says 172.21.15.1:39716) 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.023+0000 7efcede00700 1 -- 172.21.15.1:0/436206556 learned_addr learned my addr 172.21.15.1:0/436206556 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.023+0000 7efced400700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efcf0070be0 0x7efcf0115f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.023+0000 7efcede00700 1 -- 172.21.15.1:0/436206556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efcf0070be0 msgr2=0x7efcf0115f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.023+0000 7efcede00700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efcf0070be0 0x7efcf0115f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.023+0000 7efcede00700 1 -- 172.21.15.1:0/436206556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc005040 con 0x7efcf00702d0 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.023+0000 7efced400700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efcf0070be0 0x7efcf0115f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.024+0000 7efcede00700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf00702d0 0x7efcf01159f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efce000c830 tx=0x7efce0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.024+0000 7efce6000700 1 -- 172.21.15.1:0/436206556 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce000a760 con 0x7efcf00702d0 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.024+0000 7efce6000700 1 -- 172.21.15.1:0/436206556 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efce000ad90 con 0x7efcf00702d0 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.024+0000 7efce6000700 1 -- 172.21.15.1:0/436206556 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce00113a0 con 0x7efcf00702d0 2024-02-28T16:41:28.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.024+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf010fb90 con 0x7efcf00702d0 2024-02-28T16:41:28.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.024+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf01100d0 con 0x7efcf00702d0 2024-02-28T16:41:28.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.025+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf01067a0 con 0x7efcf00702d0 2024-02-28T16:41:28.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.026+0000 7efce6000700 1 -- 172.21.15.1:0/436206556 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7efce0012400 con 0x7efcf00702d0 2024-02-28T16:41:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.027+0000 7efce6000700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efcd406d7f0 0x7efcd406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.027+0000 7efce6000700 1 -- 172.21.15.1:0/436206556 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efce008cfa0 con 0x7efcf00702d0 2024-02-28T16:41:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.028+0000 7efced400700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efcd406d7f0 0x7efcd406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.028+0000 7efced400700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efcd406d7f0 0x7efcd406fca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efcdc000c00 tx=0x7efcdc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:28.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.032+0000 7efce6000700 1 -- 172.21.15.1:0/436206556 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efce005a300 con 0x7efcf00702d0 2024-02-28T16:41:28.171 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:28 smithi001 ceph-mon[112940]: pgmap v209: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:28.171 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:28 smithi001 ceph-mon[112940]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:41:28.171 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:28 smithi001 ceph-mon[112940]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:41:28.171 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:28 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/653237351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:28.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.300+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efcf004bed0 con 0x7efcf00702d0 2024-02-28T16:41:28.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.301+0000 7efce6000700 1 -- 172.21.15.1:0/436206556 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1832 (secure 0 0 0) 0x7efce0059e90 con 0x7efcf00702d0 2024-02-28T16:41:28.288 INFO:teuthology.orchestra.run.smithi001.stdout:e22 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:41:28.289 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:41:28.290 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:14474} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.1:6828/1371327686,v1:172.21.15.1:6829/1371327686] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:41:28.291 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:41:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efcd406d7f0 msgr2=0x7efcd406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efcd406d7f0 0x7efcd406fca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efcdc000c00 tx=0x7efcdc009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf00702d0 msgr2=0x7efcf01159f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf00702d0 0x7efcf01159f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efce000c830 tx=0x7efce0004470 comp rx=0 tx=0).stop 2024-02-28T16:41:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 shutdown_connections 2024-02-28T16:41:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efcd406d7f0 0x7efcd406fca0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf00702d0 0x7efcf01159f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 --2- 172.21.15.1:0/436206556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efcf0070be0 0x7efcf0115f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.303+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 >> 172.21.15.1:0/436206556 conn(0x7efcf00f8920 msgr2=0x7efcf0103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.304+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 shutdown_connections 2024-02-28T16:41:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.304+0000 7efcef200700 1 -- 172.21.15.1:0/436206556 wait complete. 2024-02-28T16:41:28.293 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 22 2024-02-28T16:41:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.469+0000 7f8382200700 1 -- 172.21.15.1:0/1171817834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c1038f0 msgr2=0x7f837c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.469+0000 7f8382200700 1 --2- 172.21.15.1:0/1171817834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c1038f0 0x7f837c0fcf60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8368004940 tx=0x7f8368009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.469+0000 7f8382200700 1 -- 172.21.15.1:0/1171817834 shutdown_connections 2024-02-28T16:41:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.469+0000 7f8382200700 1 --2- 172.21.15.1:0/1171817834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c1038f0 0x7f837c0fcf60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.469+0000 7f8382200700 1 --2- 172.21.15.1:0/1171817834 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f837c102f50 0x7f837c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.469+0000 7f8382200700 1 -- 172.21.15.1:0/1171817834 >> 172.21.15.1:0/1171817834 conn(0x7f837c0f8ae0 msgr2=0x7f837c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.469+0000 7f8382200700 1 -- 172.21.15.1:0/1171817834 shutdown_connections 2024-02-28T16:41:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.469+0000 7f8382200700 1 -- 172.21.15.1:0/1171817834 wait complete. 2024-02-28T16:41:28.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.470+0000 7f8382200700 1 Processor -- start 2024-02-28T16:41:28.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.470+0000 7f8382200700 1 -- start start 2024-02-28T16:41:28.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.471+0000 7f8382200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c102f50 0x7f837c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.471+0000 7f8382200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f837c1038f0 0x7f837c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.471+0000 7f8382200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c1168f0 con 0x7f837c102f50 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.471+0000 7f8382200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c10fb70 con 0x7f837c1038f0 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.471+0000 7f837b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c102f50 0x7f837c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.471+0000 7f837b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c102f50 0x7f837c115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39722/0 (socket says 172.21.15.1:39722) 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.471+0000 7f837b400700 1 -- 172.21.15.1:0/3654582141 learned_addr learned my addr 172.21.15.1:0/3654582141 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f837aa00700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f837c1038f0 0x7f837c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f837b400700 1 -- 172.21.15.1:0/3654582141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f837c1038f0 msgr2=0x7f837c116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f837b400700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f837c1038f0 0x7f837c116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f837b400700 1 -- 172.21.15.1:0/3654582141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8368005040 con 0x7f837c102f50 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f837b400700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c102f50 0x7f837c115cd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f836c00b790 tx=0x7f836c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f8373400700 1 -- 172.21.15.1:0/3654582141 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f836c00f770 con 0x7f837c102f50 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f8373400700 1 -- 172.21.15.1:0/3654582141 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f836c00fda0 con 0x7f837c102f50 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f8373400700 1 -- 172.21.15.1:0/3654582141 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f836c00d310 con 0x7f837c102f50 2024-02-28T16:41:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c10fdf0 con 0x7f837c102f50 2024-02-28T16:41:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.472+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837c110330 con 0x7f837c102f50 2024-02-28T16:41:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.473+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837c0fc6e0 con 0x7f837c102f50 2024-02-28T16:41:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.474+0000 7f8373400700 1 -- 172.21.15.1:0/3654582141 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f836c010420 con 0x7f837c102f50 2024-02-28T16:41:28.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.475+0000 7f8373400700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f836006d7f0 0x7f836006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.475+0000 7f8373400700 1 -- 172.21.15.1:0/3654582141 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f836c08cef0 con 0x7f837c102f50 2024-02-28T16:41:28.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.476+0000 7f837aa00700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f836006d7f0 0x7f836006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.476+0000 7f837aa00700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f836006d7f0 0x7f836006fca0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f836800b3b0 tx=0x7f8368009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:28.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.480+0000 7f8373400700 1 -- 172.21.15.1:0/3654582141 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f836c05a2d0 con 0x7f837c102f50 2024-02-28T16:41:28.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:28 smithi066 ceph-mon[85781]: pgmap v209: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:28.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:28 smithi066 ceph-mon[85781]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:41:28.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:28 smithi066 ceph-mon[85781]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:41:28.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:28 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/653237351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.729+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f837c110600 con 0x7f836006d7f0 2024-02-28T16:41:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.731+0000 7f8373400700 1 -- 172.21.15.1:0/3654582141 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f837c110600 con 0x7f836006d7f0 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:41:28.719 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:41:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "13/21 daemons upgraded", 2024-02-28T16:41:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:41:28.720 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:41:28.720 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.734+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f836006d7f0 msgr2=0x7f836006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.734+0000 7f8382200700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f836006d7f0 0x7f836006fca0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f836800b3b0 tx=0x7f8368009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.734+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c102f50 msgr2=0x7f837c115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.734+0000 7f8382200700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c102f50 0x7f837c115cd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f836c00b790 tx=0x7f836c009250 comp rx=0 tx=0).stop 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.735+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 shutdown_connections 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.735+0000 7f8382200700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f836006d7f0 0x7f836006fca0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.735+0000 7f8382200700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f837c102f50 0x7f837c115cd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.735+0000 7f8382200700 1 --2- 172.21.15.1:0/3654582141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f837c1038f0 0x7f837c116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.735+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 >> 172.21.15.1:0/3654582141 conn(0x7f837c0f8ae0 msgr2=0x7f837c0fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.735+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 shutdown_connections 2024-02-28T16:41:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.735+0000 7f8382200700 1 -- 172.21.15.1:0/3654582141 wait complete. 2024-02-28T16:41:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.892+0000 7fbb43600700 1 -- 172.21.15.1:0/2659458736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c102f50 msgr2=0x7fbb3c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.892+0000 7fbb43600700 1 --2- 172.21.15.1:0/2659458736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c102f50 0x7fbb3c103320 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb30004940 tx=0x7fbb30009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.893+0000 7fbb43600700 1 -- 172.21.15.1:0/2659458736 shutdown_connections 2024-02-28T16:41:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.893+0000 7fbb43600700 1 --2- 172.21.15.1:0/2659458736 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb3c1038f0 0x7fbb3c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.893+0000 7fbb43600700 1 --2- 172.21.15.1:0/2659458736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c102f50 0x7fbb3c103320 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.893+0000 7fbb43600700 1 -- 172.21.15.1:0/2659458736 >> 172.21.15.1:0/2659458736 conn(0x7fbb3c0f8ae0 msgr2=0x7fbb3c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:28.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.893+0000 7fbb43600700 1 -- 172.21.15.1:0/2659458736 shutdown_connections 2024-02-28T16:41:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.893+0000 7fbb43600700 1 -- 172.21.15.1:0/2659458736 wait complete. 2024-02-28T16:41:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.894+0000 7fbb43600700 1 Processor -- start 2024-02-28T16:41:28.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.894+0000 7fbb43600700 1 -- start start 2024-02-28T16:41:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.895+0000 7fbb43600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb3c102f50 0x7fbb3c113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.895+0000 7fbb43600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c1038f0 0x7fbb3c113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.895+0000 7fbb43600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1145e0 con 0x7fbb3c1038f0 2024-02-28T16:41:28.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.895+0000 7fbb43600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c10d910 con 0x7fbb3c102f50 2024-02-28T16:41:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.895+0000 7fbb3be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c1038f0 0x7fbb3c113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.895+0000 7fbb3be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c1038f0 0x7fbb3c113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39730/0 (socket says 172.21.15.1:39730) 2024-02-28T16:41:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.895+0000 7fbb3be00700 1 -- 172.21.15.1:0/1220769664 learned_addr learned my addr 172.21.15.1:0/1220769664 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.896+0000 7fbb40c00700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb3c102f50 0x7fbb3c113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.896+0000 7fbb3be00700 1 -- 172.21.15.1:0/1220769664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb3c102f50 msgr2=0x7fbb3c113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.896+0000 7fbb3be00700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb3c102f50 0x7fbb3c113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:28.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.896+0000 7fbb3be00700 1 -- 172.21.15.1:0/1220769664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30005040 con 0x7fbb3c1038f0 2024-02-28T16:41:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.896+0000 7fbb3be00700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c1038f0 0x7fbb3c113f90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb28002a50 tx=0x7fbb28004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:28.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.896+0000 7fbb38c00700 1 -- 172.21.15.1:0/1220769664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb28007cc0 con 0x7fbb3c1038f0 2024-02-28T16:41:28.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.897+0000 7fbb38c00700 1 -- 172.21.15.1:0/1220769664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb2800f040 con 0x7fbb3c1038f0 2024-02-28T16:41:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.897+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c10dbf0 con 0x7fbb3c1038f0 2024-02-28T16:41:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.897+0000 7fbb38c00700 1 -- 172.21.15.1:0/1220769664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb2800daf0 con 0x7fbb3c1038f0 2024-02-28T16:41:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.897+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c10e130 con 0x7fbb3c1038f0 2024-02-28T16:41:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.898+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3c0fc6e0 con 0x7fbb3c1038f0 2024-02-28T16:41:28.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.899+0000 7fbb38c00700 1 -- 172.21.15.1:0/1220769664 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbb28009810 con 0x7fbb3c1038f0 2024-02-28T16:41:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.900+0000 7fbb38c00700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb2406d7f0 0x7fbb2406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.901+0000 7fbb38c00700 1 -- 172.21.15.1:0/1220769664 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbb2808ccf0 con 0x7fbb3c1038f0 2024-02-28T16:41:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.901+0000 7fbb40c00700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb2406d7f0 0x7fbb2406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:28.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.901+0000 7fbb40c00700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb2406d7f0 0x7fbb2406fca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000b020 tx=0x7fbb30009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:28.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:28.905+0000 7fbb38c00700 1 -- 172.21.15.1:0/1220769664 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbb28059fd0 con 0x7fbb3c1038f0 2024-02-28T16:41:29.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.225+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbb3c10e350 con 0x7fbb3c1038f0 2024-02-28T16:41:29.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.225+0000 7fbb38c00700 1 -- 172.21.15.1:0/1220769664 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fbb28059b60 con 0x7fbb3c1038f0 2024-02-28T16:41:29.213 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:41:29.213 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:41:29.213 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:41:29.213 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:41:29.213 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:41:29.213 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:41:29.213 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:41:29.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.228+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb2406d7f0 msgr2=0x7fbb2406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:29.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.228+0000 7fbb43600700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb2406d7f0 0x7fbb2406fca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000b020 tx=0x7fbb30009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:29.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.228+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c1038f0 msgr2=0x7fbb3c113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:29.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.228+0000 7fbb43600700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c1038f0 0x7fbb3c113f90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb28002a50 tx=0x7fbb28004600 comp rx=0 tx=0).stop 2024-02-28T16:41:29.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.229+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 shutdown_connections 2024-02-28T16:41:29.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.229+0000 7fbb43600700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb2406d7f0 0x7fbb2406fca0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:29.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.229+0000 7fbb43600700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb3c102f50 0x7fbb3c113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:29.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.229+0000 7fbb43600700 1 --2- 172.21.15.1:0/1220769664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c1038f0 0x7fbb3c113f90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:29.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.229+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 >> 172.21.15.1:0/1220769664 conn(0x7fbb3c0f8ae0 msgr2=0x7fbb3c0fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:29.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.229+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 shutdown_connections 2024-02-28T16:41:29.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:29.229+0000 7fbb43600700 1 -- 172.21.15.1:0/1220769664 wait complete. 2024-02-28T16:41:29.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:29 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/436206556' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:41:29.517 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:29 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/436206556' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:41:30.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:30 smithi066 ceph-mon[85781]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:41:30.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:30 smithi066 ceph-mon[85781]: pgmap v210: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:30.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1220769664' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:41:30.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:30 smithi001 ceph-mon[112940]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:41:30.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:30 smithi001 ceph-mon[112940]: pgmap v210: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:30.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1220769664' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:41:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:32 smithi066 ceph-mon[85781]: pgmap v211: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:41:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:41:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:32.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:32 smithi001 ceph-mon[112940]: pgmap v211: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:32.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:32.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:32.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:41:32.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:41:32.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:33.344 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:41:33.344 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:33.345 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:33.345 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:33.345 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:33 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:33.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:41:33.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:33.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:33.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:33.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:33 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:41:34.411 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:34 smithi001 ceph-mon[112940]: pgmap v212: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:34.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:34 smithi066 ceph-mon[85781]: pgmap v212: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:35.746 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:35 smithi001 ceph-mon[112940]: Upgrade: Updating mds.cephfs.smithi001.rtdwhc 2024-02-28T16:41:35.746 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:35.746 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.rtdwhc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:41:35.746 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:35 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:41:35.746 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:35 smithi001 ceph-mon[112940]: Deploying daemon mds.cephfs.smithi001.rtdwhc on smithi001 2024-02-28T16:41:35.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:35 smithi066 ceph-mon[85781]: Upgrade: Updating mds.cephfs.smithi001.rtdwhc 2024-02-28T16:41:35.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:35.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi001.rtdwhc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:41:35.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:35 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:41:35.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:35 smithi066 ceph-mon[85781]: Deploying daemon mds.cephfs.smithi001.rtdwhc on smithi001 2024-02-28T16:41:36.766 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:36 smithi001 ceph-mon[112940]: pgmap v213: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:36.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:36 smithi066 ceph-mon[85781]: pgmap v213: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:38.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:38 smithi066 ceph-mon[85781]: pgmap v214: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:39.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:38 smithi001 ceph-mon[112940]: pgmap v214: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:39.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:39 smithi001 ceph-mon[112940]: osdmap e82: 6 total, 6 up, 6 in 2024-02-28T16:41:39.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:39 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-02-28T16:41:39.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:39 smithi066 ceph-mon[85781]: osdmap e82: 6 total, 6 up, 6 in 2024-02-28T16:41:39.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:39 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-02-28T16:41:40.865 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:40 smithi001 ceph-mon[112940]: pgmap v216: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:40.898 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:40 smithi066 ceph-mon[85781]: pgmap v216: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:41.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:41:42.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:41:42.688 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:42 smithi001 ceph-mon[112940]: pgmap v217: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:42.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:42 smithi066 ceph-mon[85781]: pgmap v217: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: pgmap v218: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:boot 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:44.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:43 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:44.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: pgmap v218: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:44.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:44.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:44.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:41:44.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:boot 2024-02-28T16:41:44.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T16:41:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T16:41:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:44.248 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:43 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:46.232 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:45 smithi001 ceph-mon[112940]: pgmap v219: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:46.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:45 smithi066 ceph-mon[85781]: pgmap v219: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:48.133 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:47 smithi001 ceph-mon[112940]: pgmap v220: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:48.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:47 smithi066 ceph-mon[85781]: pgmap v220: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:50.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:50 smithi066 ceph-mon[85781]: pgmap v221: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:50.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:50 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:50.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:50 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:50.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:50 smithi001 ceph-mon[112940]: pgmap v221: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:50.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:50 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:50.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:50 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:52.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:52 smithi066 ceph-mon[85781]: pgmap v222: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:52.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:52.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:52.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:52 smithi001 ceph-mon[112940]: pgmap v222: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:52.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:52.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:41:54.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:54 smithi001 ceph-mon[112940]: pgmap v223: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:54.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:54 smithi066 ceph-mon[85781]: pgmap v223: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:56.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:56 smithi001 ceph-mon[112940]: pgmap v224: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:41:56.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:56 smithi066 ceph-mon[85781]: pgmap v224: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:56.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:41:58.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:41:58 smithi001 ceph-mon[112940]: pgmap v225: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:58.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:41:58 smithi066 ceph-mon[85781]: pgmap v225: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:41:59.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.402+0000 7fd308200700 1 -- 172.21.15.1:0/610339866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 msgr2=0x7fd300103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:59.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.402+0000 7fd308200700 1 --2- 172.21.15.1:0/610339866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 0x7fd300103320 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4004940 tx=0x7fd2f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.402+0000 7fd308200700 1 -- 172.21.15.1:0/610339866 shutdown_connections 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.402+0000 7fd308200700 1 --2- 172.21.15.1:0/610339866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3001038f0 0x7fd3000fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.402+0000 7fd308200700 1 --2- 172.21.15.1:0/610339866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 0x7fd300103320 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.402+0000 7fd308200700 1 -- 172.21.15.1:0/610339866 >> 172.21.15.1:0/610339866 conn(0x7fd3000f8ae0 msgr2=0x7fd3000faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.403+0000 7fd308200700 1 -- 172.21.15.1:0/610339866 shutdown_connections 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.403+0000 7fd308200700 1 -- 172.21.15.1:0/610339866 wait complete. 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.403+0000 7fd308200700 1 Processor -- start 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.404+0000 7fd308200700 1 -- start start 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.404+0000 7fd308200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 0x7fd300115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.404+0000 7fd308200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3001038f0 0x7fd3001161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.404+0000 7fd308200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd300116800 con 0x7fd300102f50 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.404+0000 7fd308200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30010fb30 con 0x7fd3001038f0 2024-02-28T16:41:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.404+0000 7fd305800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 0x7fd300115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.404+0000 7fd305800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 0x7fd300115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48554/0 (socket says 172.21.15.1:48554) 2024-02-28T16:41:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.404+0000 7fd305800700 1 -- 172.21.15.1:0/1210690025 learned_addr learned my addr 172.21.15.1:0/1210690025 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.405+0000 7fd305800700 1 -- 172.21.15.1:0/1210690025 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3001038f0 msgr2=0x7fd3001161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.405+0000 7fd305800700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3001038f0 0x7fd3001161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.405+0000 7fd305800700 1 -- 172.21.15.1:0/1210690025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4005040 con 0x7fd300102f50 2024-02-28T16:41:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.405+0000 7fd305800700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 0x7fd300115c70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009fd0 tx=0x7fd2f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.405+0000 7fd2fd600700 1 -- 172.21.15.1:0/1210690025 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2f400e040 con 0x7fd300102f50 2024-02-28T16:41:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.405+0000 7fd2fd600700 1 -- 172.21.15.1:0/1210690025 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2f400bbf0 con 0x7fd300102f50 2024-02-28T16:41:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.405+0000 7fd2fd600700 1 -- 172.21.15.1:0/1210690025 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2f401c3f0 con 0x7fd300102f50 2024-02-28T16:41:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.405+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30010fdb0 con 0x7fd300102f50 2024-02-28T16:41:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.406+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3001102f0 con 0x7fd300102f50 2024-02-28T16:41:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.407+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3000fc6e0 con 0x7fd300102f50 2024-02-28T16:41:59.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.408+0000 7fd2fd600700 1 -- 172.21.15.1:0/1210690025 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd2f400bd60 con 0x7fd300102f50 2024-02-28T16:41:59.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.409+0000 7fd2fd600700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd2e806d7f0 0x7fd2e806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:59.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.409+0000 7fd2fd600700 1 -- 172.21.15.1:0/1210690025 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd2f408e1c0 con 0x7fd300102f50 2024-02-28T16:41:59.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.409+0000 7fd304e00700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd2e806d7f0 0x7fd2e806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.410+0000 7fd304e00700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd2e806d7f0 0x7fd2e806fca0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd3001114a0 tx=0x7fd2ec009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:59.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.414+0000 7fd2fd600700 1 -- 172.21.15.1:0/1210690025 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd2f405b5a0 con 0x7fd300102f50 2024-02-28T16:41:59.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.662+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3001112f0 con 0x7fd2e806d7f0 2024-02-28T16:41:59.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.664+0000 7fd2fd600700 1 -- 172.21.15.1:0/1210690025 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd3001112f0 con 0x7fd2e806d7f0 2024-02-28T16:41:59.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd2e806d7f0 msgr2=0x7fd2e806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:59.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd2e806d7f0 0x7fd2e806fca0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd3001114a0 tx=0x7fd2ec009ce0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 msgr2=0x7fd300115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:59.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 0x7fd300115c70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009fd0 tx=0x7fd2f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:59.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 shutdown_connections 2024-02-28T16:41:59.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd2e806d7f0 0x7fd2e806fca0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300102f50 0x7fd300115c70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 --2- 172.21.15.1:0/1210690025 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3001038f0 0x7fd3001161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 >> 172.21.15.1:0/1210690025 conn(0x7fd3000f8ae0 msgr2=0x7fd300070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:59.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 shutdown_connections 2024-02-28T16:41:59.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.667+0000 7fd308200700 1 -- 172.21.15.1:0/1210690025 wait complete. 2024-02-28T16:41:59.670 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:41:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.826+0000 7f364f400700 1 -- 172.21.15.1:0/1827181726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070c40 msgr2=0x7f3648109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.826+0000 7f364f400700 1 --2- 172.21.15.1:0/1827181726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070c40 0x7f3648109260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3634004970 tx=0x7f3634009e30 comp rx=0 tx=0).stop 2024-02-28T16:41:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.826+0000 7f364f400700 1 -- 172.21.15.1:0/1827181726 shutdown_connections 2024-02-28T16:41:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.826+0000 7f364f400700 1 --2- 172.21.15.1:0/1827181726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070c40 0x7f3648109260 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.826+0000 7f364f400700 1 --2- 172.21.15.1:0/1827181726 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3648070330 0x7f3648070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.826+0000 7f364f400700 1 -- 172.21.15.1:0/1827181726 >> 172.21.15.1:0/1827181726 conn(0x7f36480f8ae0 msgr2=0x7f36480faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:41:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.826+0000 7f364f400700 1 -- 172.21.15.1:0/1827181726 shutdown_connections 2024-02-28T16:41:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.826+0000 7f364f400700 1 -- 172.21.15.1:0/1827181726 wait complete. 2024-02-28T16:41:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.827+0000 7f364f400700 1 Processor -- start 2024-02-28T16:41:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.827+0000 7f364f400700 1 -- start start 2024-02-28T16:41:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.827+0000 7f364f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070330 0x7f3648115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.827+0000 7f364f400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3648070c40 0x7f3648116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.827+0000 7f364f400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36481167b0 con 0x7f3648070330 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.827+0000 7f364f400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364810fae0 con 0x7f3648070c40 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.828+0000 7f364ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070330 0x7f3648115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.828+0000 7f364ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070330 0x7f3648115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48578/0 (socket says 172.21.15.1:48578) 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.828+0000 7f364ca00700 1 -- 172.21.15.1:0/2683227118 learned_addr learned my addr 172.21.15.1:0/2683227118 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.828+0000 7f3647e00700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3648070c40 0x7f3648116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.828+0000 7f364ca00700 1 -- 172.21.15.1:0/2683227118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3648070c40 msgr2=0x7f3648116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.828+0000 7f364ca00700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3648070c40 0x7f3648116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:41:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.828+0000 7f364ca00700 1 -- 172.21.15.1:0/2683227118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3634005040 con 0x7f3648070330 2024-02-28T16:41:59.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.828+0000 7f364ca00700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070330 0x7f3648115c20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f363c00eb40 tx=0x7f363c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:59.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.829+0000 7f3644c00700 1 -- 172.21.15.1:0/2683227118 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363c007cc0 con 0x7f3648070330 2024-02-28T16:41:59.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.829+0000 7f3644c00700 1 -- 172.21.15.1:0/2683227118 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f363c00f040 con 0x7f3648070330 2024-02-28T16:41:59.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.829+0000 7f3644c00700 1 -- 172.21.15.1:0/2683227118 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363c009620 con 0x7f3648070330 2024-02-28T16:41:59.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.829+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364810fd60 con 0x7f3648070330 2024-02-28T16:41:59.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.829+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36481102a0 con 0x7f3648070330 2024-02-28T16:41:59.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.830+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3648106920 con 0x7f3648070330 2024-02-28T16:41:59.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.831+0000 7f3644c00700 1 -- 172.21.15.1:0/2683227118 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f363c009780 con 0x7f3648070330 2024-02-28T16:41:59.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.832+0000 7f3644c00700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f363006d7f0 0x7f363006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:41:59.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.832+0000 7f3644c00700 1 -- 172.21.15.1:0/2683227118 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f363c08caa0 con 0x7f3648070330 2024-02-28T16:41:59.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.832+0000 7f3647e00700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f363006d7f0 0x7f363006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:41:59.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.833+0000 7f3647e00700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f363006d7f0 0x7f363006fca0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f363400b370 tx=0x7f3634009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:41:59.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:41:59.837+0000 7f3644c00700 1 -- 172.21.15.1:0/2683227118 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f363c059f50 con 0x7f3648070330 2024-02-28T16:42:00.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.084+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f364804bed0 con 0x7f363006d7f0 2024-02-28T16:42:00.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.088+0000 7f3644c00700 1 -- 172.21.15.1:0/2683227118 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f364804bed0 con 0x7f363006d7f0 2024-02-28T16:42:00.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f363006d7f0 msgr2=0x7f363006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:00.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f363006d7f0 0x7f363006fca0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f363400b370 tx=0x7f3634009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070330 msgr2=0x7f3648115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070330 0x7f3648115c20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f363c00eb40 tx=0x7f363c00b040 comp rx=0 tx=0).stop 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 shutdown_connections 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f363006d7f0 0x7f363006fca0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3648070330 0x7f3648115c20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 --2- 172.21.15.1:0/2683227118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3648070c40 0x7f3648116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 >> 172.21.15.1:0/2683227118 conn(0x7f36480f8ae0 msgr2=0x7f3648103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.091+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 shutdown_connections 2024-02-28T16:42:00.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.092+0000 7f364f400700 1 -- 172.21.15.1:0/2683227118 wait complete. 2024-02-28T16:42:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.250+0000 7fb599600700 1 -- 172.21.15.1:0/2058226105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 msgr2=0x7fb594105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.250+0000 7fb599600700 1 --2- 172.21.15.1:0/2058226105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 0x7fb594105530 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb584004940 tx=0x7fb584009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.250+0000 7fb599600700 1 -- 172.21.15.1:0/2058226105 shutdown_connections 2024-02-28T16:42:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.251+0000 7fb599600700 1 --2- 172.21.15.1:0/2058226105 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb594105b00 0x7fb5940ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.251+0000 7fb599600700 1 --2- 172.21.15.1:0/2058226105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 0x7fb594105530 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.251+0000 7fb599600700 1 -- 172.21.15.1:0/2058226105 >> 172.21.15.1:0/2058226105 conn(0x7fb5940fad30 msgr2=0x7fb5940fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:00.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.251+0000 7fb599600700 1 -- 172.21.15.1:0/2058226105 shutdown_connections 2024-02-28T16:42:00.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.251+0000 7fb599600700 1 -- 172.21.15.1:0/2058226105 wait complete. 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.251+0000 7fb599600700 1 Processor -- start 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.252+0000 7fb599600700 1 -- start start 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.252+0000 7fb599600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 0x7fb594115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.252+0000 7fb599600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb594105b00 0x7fb594116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.252+0000 7fb599600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594116830 con 0x7fb594105160 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.252+0000 7fb599600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59410faf0 con 0x7fb594105b00 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.252+0000 7fb592a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 0x7fb594115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.252+0000 7fb592a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 0x7fb594115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48592/0 (socket says 172.21.15.1:48592) 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.252+0000 7fb592a00700 1 -- 172.21.15.1:0/3522471862 learned_addr learned my addr 172.21.15.1:0/3522471862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.253+0000 7fb592a00700 1 -- 172.21.15.1:0/3522471862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb594105b00 msgr2=0x7fb594116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.253+0000 7fb592a00700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb594105b00 0x7fb594116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.253+0000 7fb592a00700 1 -- 172.21.15.1:0/3522471862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb584005040 con 0x7fb594105160 2024-02-28T16:42:00.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.253+0000 7fb592a00700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 0x7fb594115c10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb584009150 tx=0x7fb584009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.253+0000 7fb58aa00700 1 -- 172.21.15.1:0/3522471862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb58400e040 con 0x7fb594105160 2024-02-28T16:42:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.253+0000 7fb58aa00700 1 -- 172.21.15.1:0/3522471862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb58400bbf0 con 0x7fb594105160 2024-02-28T16:42:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.253+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59410fd70 con 0x7fb594105160 2024-02-28T16:42:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.253+0000 7fb58aa00700 1 -- 172.21.15.1:0/3522471862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb58401c630 con 0x7fb594105160 2024-02-28T16:42:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.254+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5941102b0 con 0x7fb594105160 2024-02-28T16:42:00.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.255+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5940fe930 con 0x7fb594105160 2024-02-28T16:42:00.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.256+0000 7fb58aa00700 1 -- 172.21.15.1:0/3522471862 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb58400bd60 con 0x7fb594105160 2024-02-28T16:42:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.256+0000 7fb58aa00700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb57806d7f0 0x7fb57806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.257+0000 7fb58aa00700 1 -- 172.21.15.1:0/3522471862 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb58408e320 con 0x7fb594105160 2024-02-28T16:42:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.257+0000 7fb58be00700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb57806d7f0 0x7fb57806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:00.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.258+0000 7fb58be00700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb57806d7f0 0x7fb57806fca0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb57c006fd0 tx=0x7fb57c006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:00.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.261+0000 7fb58aa00700 1 -- 172.21.15.1:0/3522471862 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb584057bf0 con 0x7fb594105160 2024-02-28T16:42:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.494+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5941104d0 con 0x7fb57806d7f0 2024-02-28T16:42:00.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:00 smithi001 ceph-mon[112940]: pgmap v226: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:00.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.525+0000 7fb58aa00700 1 -- 172.21.15.1:0/3522471862 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fb5941104d0 con 0x7fb57806d7f0 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (8m) 11s ago 22m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (4m) 11s ago 22m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (4m) 64s ago 21m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (7m) 11s ago 22m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (44s) 11s ago 15m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (18s) 11s ago 15m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (15m) 64s ago 15m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (15m) 64s ago 15m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (10m) 11s ago 24m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:42:00.514 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (9m) 64s ago 21m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (6m) 11s ago 24m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (5m) 64s ago 20m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (9m) 11s ago 22m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (9m) 64s ago 21m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (4m) 11s ago 20m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (3m) 11s ago 19m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (3m) 11s ago 18m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (2m) 64s ago 18m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (114s) 64s ago 17m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (69s) 64s ago 17m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:42:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (8m) 11s ago 21m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:42:00.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb57806d7f0 msgr2=0x7fb57806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:00.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb57806d7f0 0x7fb57806fca0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb57c006fd0 tx=0x7fb57c006db0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 msgr2=0x7fb594115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:00.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 0x7fb594115c10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb584009150 tx=0x7fb584009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:00.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 shutdown_connections 2024-02-28T16:42:00.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb57806d7f0 0x7fb57806fca0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb594105160 0x7fb594115c10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 --2- 172.21.15.1:0/3522471862 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb594105b00 0x7fb594116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.528+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 >> 172.21.15.1:0/3522471862 conn(0x7fb5940fad30 msgr2=0x7fb5940fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:00.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.529+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 shutdown_connections 2024-02-28T16:42:00.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.529+0000 7fb599600700 1 -- 172.21.15.1:0/3522471862 wait complete. 2024-02-28T16:42:00.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.684+0000 7ff12c400700 1 -- 172.21.15.1:0/4233327266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 msgr2=0x7ff1240ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.684+0000 7ff12c400700 1 --2- 172.21.15.1:0/4233327266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 0x7ff1240ff180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff110004940 tx=0x7ff110009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.684+0000 7ff12c400700 1 -- 172.21.15.1:0/4233327266 shutdown_connections 2024-02-28T16:42:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.684+0000 7ff12c400700 1 --2- 172.21.15.1:0/4233327266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 0x7ff1240ff180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.684+0000 7ff12c400700 1 --2- 172.21.15.1:0/4233327266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff124105170 0x7ff124105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.684+0000 7ff12c400700 1 -- 172.21.15.1:0/4233327266 >> 172.21.15.1:0/4233327266 conn(0x7ff1240face0 msgr2=0x7ff1240fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.684+0000 7ff12c400700 1 -- 172.21.15.1:0/4233327266 shutdown_connections 2024-02-28T16:42:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.684+0000 7ff12c400700 1 -- 172.21.15.1:0/4233327266 wait complete. 2024-02-28T16:42:00.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.685+0000 7ff12c400700 1 Processor -- start 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.685+0000 7ff12c400700 1 -- start start 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff12c400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff124105170 0x7ff124115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff12c400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 0x7ff124116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff12c400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff124116870 con 0x7ff124105b10 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff12c400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12410fb80 con 0x7ff124105170 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff129000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 0x7ff124116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff129000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 0x7ff124116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48618/0 (socket says 172.21.15.1:48618) 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff129000700 1 -- 172.21.15.1:0/3207721835 learned_addr learned my addr 172.21.15.1:0/3207721835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff129a00700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff124105170 0x7ff124115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff129000700 1 -- 172.21.15.1:0/3207721835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff124105170 msgr2=0x7ff124115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff129000700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff124105170 0x7ff124115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.686+0000 7ff129000700 1 -- 172.21.15.1:0/3207721835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110005040 con 0x7ff124105b10 2024-02-28T16:42:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.687+0000 7ff129000700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 0x7ff124116220 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff110009030 tx=0x7ff110009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.687+0000 7ff121600700 1 -- 172.21.15.1:0/3207721835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff11000e040 con 0x7ff124105b10 2024-02-28T16:42:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.687+0000 7ff121600700 1 -- 172.21.15.1:0/3207721835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff11000bbf0 con 0x7ff124105b10 2024-02-28T16:42:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.687+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12410fe00 con 0x7ff124105b10 2024-02-28T16:42:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.687+0000 7ff121600700 1 -- 172.21.15.1:0/3207721835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff11001c630 con 0x7ff124105b10 2024-02-28T16:42:00.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.687+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff124110340 con 0x7ff124105b10 2024-02-28T16:42:00.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.689+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff124107640 con 0x7ff124105b10 2024-02-28T16:42:00.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.690+0000 7ff121600700 1 -- 172.21.15.1:0/3207721835 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff11000bd60 con 0x7ff124105b10 2024-02-28T16:42:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.690+0000 7ff121600700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff10c06d7f0 0x7ff10c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.691+0000 7ff121600700 1 -- 172.21.15.1:0/3207721835 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff11008e2c0 con 0x7ff124105b10 2024-02-28T16:42:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.691+0000 7ff129a00700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff10c06d7f0 0x7ff10c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.692+0000 7ff129a00700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff10c06d7f0 0x7ff10c06fca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff118004d60 tx=0x7ff11800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:00.696+0000 7ff121600700 1 -- 172.21.15.1:0/3207721835 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff11005b620 con 0x7ff124105b10 2024-02-28T16:42:00.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:00 smithi066 ceph-mon[85781]: pgmap v226: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:00.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.009+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff12404bed0 con 0x7ff124105b10 2024-02-28T16:42:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.010+0000 7ff121600700 1 -- 172.21.15.1:0/3207721835 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7ff11005b1b0 con 0x7ff124105b10 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:42:00.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:42:01.000 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.013+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff10c06d7f0 msgr2=0x7ff10c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.013+0000 7ff12c400700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff10c06d7f0 0x7ff10c06fca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff118004d60 tx=0x7ff11800b040 comp rx=0 tx=0).stop 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.013+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 msgr2=0x7ff124116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.013+0000 7ff12c400700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 0x7ff124116220 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff110009030 tx=0x7ff110009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.013+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 shutdown_connections 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.014+0000 7ff12c400700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff10c06d7f0 0x7ff10c06fca0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.014+0000 7ff12c400700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff124105170 0x7ff124115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.014+0000 7ff12c400700 1 --2- 172.21.15.1:0/3207721835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff124105b10 0x7ff124116220 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.014+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 >> 172.21.15.1:0/3207721835 conn(0x7ff1240face0 msgr2=0x7ff1240fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.014+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 shutdown_connections 2024-02-28T16:42:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.014+0000 7ff12c400700 1 -- 172.21.15.1:0/3207721835 wait complete. 2024-02-28T16:42:01.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.169+0000 7fd3a9000700 1 -- 172.21.15.1:0/1926815301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a4101e40 msgr2=0x7fd3a4102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.169+0000 7fd3a9000700 1 --2- 172.21.15.1:0/1926815301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a4101e40 0x7fd3a4102210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd394004940 tx=0x7fd394009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:01.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.169+0000 7fd3a9000700 1 -- 172.21.15.1:0/1926815301 shutdown_connections 2024-02-28T16:42:01.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.169+0000 7fd3a9000700 1 --2- 172.21.15.1:0/1926815301 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3a41027e0 0x7fd3a40fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.169+0000 7fd3a9000700 1 --2- 172.21.15.1:0/1926815301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a4101e40 0x7fd3a4102210 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.169+0000 7fd3a9000700 1 -- 172.21.15.1:0/1926815301 >> 172.21.15.1:0/1926815301 conn(0x7fd3a4071d90 msgr2=0x7fd3a4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:01.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.169+0000 7fd3a9000700 1 -- 172.21.15.1:0/1926815301 shutdown_connections 2024-02-28T16:42:01.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.169+0000 7fd3a9000700 1 -- 172.21.15.1:0/1926815301 wait complete. 2024-02-28T16:42:01.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.170+0000 7fd3a9000700 1 Processor -- start 2024-02-28T16:42:01.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.171+0000 7fd3a9000700 1 -- start start 2024-02-28T16:42:01.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.171+0000 7fd3a9000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3a4101e40 0x7fd3a4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.171+0000 7fd3a9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a41027e0 0x7fd3a4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.171+0000 7fd3a9000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a4114600 con 0x7fd3a41027e0 2024-02-28T16:42:01.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.171+0000 7fd3a9000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a410d930 con 0x7fd3a4101e40 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.171+0000 7fd3a1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a41027e0 0x7fd3a4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.171+0000 7fd3a1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a41027e0 0x7fd3a4113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48636/0 (socket says 172.21.15.1:48636) 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.171+0000 7fd3a1400700 1 -- 172.21.15.1:0/1988051762 learned_addr learned my addr 172.21.15.1:0/1988051762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.172+0000 7fd3a1e00700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3a4101e40 0x7fd3a4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.172+0000 7fd3a1400700 1 -- 172.21.15.1:0/1988051762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3a4101e40 msgr2=0x7fd3a4113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.172+0000 7fd3a1400700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3a4101e40 0x7fd3a4113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.172+0000 7fd3a1400700 1 -- 172.21.15.1:0/1988051762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd394005040 con 0x7fd3a41027e0 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.172+0000 7fd3a1e00700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3a4101e40 0x7fd3a4113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.172+0000 7fd3a1400700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a41027e0 0x7fd3a4113fb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd390002a50 tx=0x7fd390004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.173+0000 7fd39a000700 1 -- 172.21.15.1:0/1988051762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd390007cc0 con 0x7fd3a41027e0 2024-02-28T16:42:01.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.173+0000 7fd39a000700 1 -- 172.21.15.1:0/1988051762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd39000f040 con 0x7fd3a41027e0 2024-02-28T16:42:01.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.173+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a410dc10 con 0x7fd3a41027e0 2024-02-28T16:42:01.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.173+0000 7fd39a000700 1 -- 172.21.15.1:0/1988051762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd39000daf0 con 0x7fd3a41027e0 2024-02-28T16:42:01.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.173+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a410e150 con 0x7fd3a41027e0 2024-02-28T16:42:01.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.174+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a40738b0 con 0x7fd3a41027e0 2024-02-28T16:42:01.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.175+0000 7fd39a000700 1 -- 172.21.15.1:0/1988051762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd390009810 con 0x7fd3a41027e0 2024-02-28T16:42:01.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.176+0000 7fd39a000700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd38806d7f0 0x7fd38806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.176+0000 7fd39a000700 1 -- 172.21.15.1:0/1988051762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd39008cc20 con 0x7fd3a41027e0 2024-02-28T16:42:01.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.176+0000 7fd3a1e00700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd38806d7f0 0x7fd38806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.177+0000 7fd3a1e00700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd38806d7f0 0x7fd38806fca0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd394009fd0 tx=0x7fd394009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:01.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.181+0000 7fd39a000700 1 -- 172.21.15.1:0/1988051762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd390059f80 con 0x7fd3a41027e0 2024-02-28T16:42:01.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.454+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd3a404bed0 con 0x7fd3a41027e0 2024-02-28T16:42:01.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.455+0000 7fd39a000700 1 -- 172.21.15.1:0/1988051762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fd390059b10 con 0x7fd3a41027e0 2024-02-28T16:42:01.444 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:42:01.444 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:42:01.444 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:42:01.444 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:42:01.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:42:01.444 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:42:01.445 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:42:01.446 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:42:01.447 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:42:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.457+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd38806d7f0 msgr2=0x7fd38806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.457+0000 7fd3a9000700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd38806d7f0 0x7fd38806fca0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd394009fd0 tx=0x7fd394009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.457+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a41027e0 msgr2=0x7fd3a4113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.457+0000 7fd3a9000700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a41027e0 0x7fd3a4113fb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd390002a50 tx=0x7fd390004600 comp rx=0 tx=0).stop 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.458+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 shutdown_connections 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.458+0000 7fd3a9000700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd38806d7f0 0x7fd38806fca0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.458+0000 7fd3a9000700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3a4101e40 0x7fd3a4113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.458+0000 7fd3a9000700 1 --2- 172.21.15.1:0/1988051762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3a41027e0 0x7fd3a4113fb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.458+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 >> 172.21.15.1:0/1988051762 conn(0x7fd3a4071d90 msgr2=0x7fd3a40fae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.458+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 shutdown_connections 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.458+0000 7fd3a9000700 1 -- 172.21.15.1:0/1988051762 wait complete. 2024-02-28T16:42:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:42:01.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.606+0000 7f0611600700 1 -- 172.21.15.1:0/2558242729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 msgr2=0x7f060c105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.606+0000 7f0611600700 1 --2- 172.21.15.1:0/2558242729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 0x7f060c105530 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f05fc004940 tx=0x7f05fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:01.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.607+0000 7f0611600700 1 -- 172.21.15.1:0/2558242729 shutdown_connections 2024-02-28T16:42:01.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.607+0000 7f0611600700 1 --2- 172.21.15.1:0/2558242729 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f060c105b00 0x7f060c0ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.607+0000 7f0611600700 1 --2- 172.21.15.1:0/2558242729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 0x7f060c105530 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.607+0000 7f0611600700 1 -- 172.21.15.1:0/2558242729 >> 172.21.15.1:0/2558242729 conn(0x7f060c0facb0 msgr2=0x7f060c0fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:01.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.607+0000 7f0611600700 1 -- 172.21.15.1:0/2558242729 shutdown_connections 2024-02-28T16:42:01.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.607+0000 7f0611600700 1 -- 172.21.15.1:0/2558242729 wait complete. 2024-02-28T16:42:01.596 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:01 smithi001 ceph-mon[112940]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:01.596 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:01 smithi001 ceph-mon[112940]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:01.596 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:01 smithi001 ceph-mon[112940]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:01.596 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:01 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3207721835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:42:01.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f0611600700 1 Processor -- start 2024-02-28T16:42:01.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f0611600700 1 -- start start 2024-02-28T16:42:01.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f0611600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 0x7f060c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f0611600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f060c105b00 0x7f060c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f0611600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c116800 con 0x7f060c105160 2024-02-28T16:42:01.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f0611600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c10fb30 con 0x7f060c105b00 2024-02-28T16:42:01.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f060aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 0x7f060c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f060aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 0x7f060c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48650/0 (socket says 172.21.15.1:48650) 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.608+0000 7f060aa00700 1 -- 172.21.15.1:0/2646408754 learned_addr learned my addr 172.21.15.1:0/2646408754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.609+0000 7f060a000700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f060c105b00 0x7f060c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.609+0000 7f060aa00700 1 -- 172.21.15.1:0/2646408754 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f060c105b00 msgr2=0x7f060c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.609+0000 7f060aa00700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f060c105b00 0x7f060c1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.609+0000 7f060aa00700 1 -- 172.21.15.1:0/2646408754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc005040 con 0x7f060c105160 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.609+0000 7f060a000700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f060c105b00 0x7f060c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.609+0000 7f060aa00700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 0x7f060c115c70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00b3b0 tx=0x7f05fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.610+0000 7f0602a00700 1 -- 172.21.15.1:0/2646408754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05fc00e040 con 0x7f060c105160 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.610+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c10fdb0 con 0x7f060c105160 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.610+0000 7f0602a00700 1 -- 172.21.15.1:0/2646408754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05fc00bbf0 con 0x7f060c105160 2024-02-28T16:42:01.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.610+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c1102f0 con 0x7f060c105160 2024-02-28T16:42:01.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.610+0000 7f0602a00700 1 -- 172.21.15.1:0/2646408754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05fc0255e0 con 0x7f060c105160 2024-02-28T16:42:01.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.611+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c0fe8f0 con 0x7f060c105160 2024-02-28T16:42:01.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.613+0000 7f0602a00700 1 -- 172.21.15.1:0/2646408754 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f05fc00bd60 con 0x7f060c105160 2024-02-28T16:42:01.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.613+0000 7f0602a00700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f05f006d7e0 0x7f05f006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.614+0000 7f0602a00700 1 -- 172.21.15.1:0/2646408754 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f05fc08ecd0 con 0x7f060c105160 2024-02-28T16:42:01.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.614+0000 7f060a000700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f05f006d7e0 0x7f05f006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.615+0000 7f060a000700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f05f006d7e0 0x7f05f006fc90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009fd0 tx=0x7f05f800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:01.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.617+0000 7f0602a00700 1 -- 172.21.15.1:0/2646408754 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f05fc05c170 con 0x7f060c105160 2024-02-28T16:42:01.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:01 smithi066 ceph-mon[85781]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:01.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:01 smithi066 ceph-mon[85781]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:01.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:01 smithi066 ceph-mon[85781]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:01.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:01 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3207721835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:42:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.853+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f060c111280 con 0x7f05f006d7e0 2024-02-28T16:42:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.856+0000 7f0602a00700 1 -- 172.21.15.1:0/2646408754 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f060c111280 con 0x7f05f006d7e0 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:42:01.845 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:42:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f05f006d7e0 msgr2=0x7f05f006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f05f006d7e0 0x7f05f006fc90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009fd0 tx=0x7f05f800b040 comp rx=0 tx=0).stop 2024-02-28T16:42:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 msgr2=0x7f060c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 0x7f060c115c70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00b3b0 tx=0x7f05fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 shutdown_connections 2024-02-28T16:42:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f05f006d7e0 0x7f05f006fc90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f060c105160 0x7f060c115c70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 --2- 172.21.15.1:0/2646408754 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f060c105b00 0x7f060c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 >> 172.21.15.1:0/2646408754 conn(0x7f060c0facb0 msgr2=0x7f060c0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 shutdown_connections 2024-02-28T16:42:01.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:01.859+0000 7f0611600700 1 -- 172.21.15.1:0/2646408754 wait complete. 2024-02-28T16:42:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.003+0000 7fd9fd600700 1 -- 172.21.15.1:0/1996606111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105160 msgr2=0x7fd9f8105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.003+0000 7fd9fd600700 1 --2- 172.21.15.1:0/1996606111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105160 0x7fd9f8105530 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8004940 tx=0x7fd9e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.003+0000 7fd9fd600700 1 -- 172.21.15.1:0/1996606111 shutdown_connections 2024-02-28T16:42:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.003+0000 7fd9fd600700 1 --2- 172.21.15.1:0/1996606111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9f8105b00 0x7fd9f80ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.003+0000 7fd9fd600700 1 --2- 172.21.15.1:0/1996606111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105160 0x7fd9f8105530 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.003+0000 7fd9fd600700 1 -- 172.21.15.1:0/1996606111 >> 172.21.15.1:0/1996606111 conn(0x7fd9f80fad30 msgr2=0x7fd9f80fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.003+0000 7fd9fd600700 1 -- 172.21.15.1:0/1996606111 shutdown_connections 2024-02-28T16:42:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.003+0000 7fd9fd600700 1 -- 172.21.15.1:0/1996606111 wait complete. 2024-02-28T16:42:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.005+0000 7fd9fd600700 1 Processor -- start 2024-02-28T16:42:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.005+0000 7fd9fd600700 1 -- start start 2024-02-28T16:42:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.005+0000 7fd9fd600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9f8105160 0x7fd9f8115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.005+0000 7fd9fd600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105b00 0x7fd9f81161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.005+0000 7fd9fd600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f81167f0 con 0x7fd9f8105b00 2024-02-28T16:42:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.005+0000 7fd9fd600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f810fb40 con 0x7fd9f8105160 2024-02-28T16:42:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.006+0000 7fd9f6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105b00 0x7fd9f81161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.006+0000 7fd9f6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105b00 0x7fd9f81161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54402/0 (socket says 172.21.15.1:54402) 2024-02-28T16:42:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.006+0000 7fd9f6000700 1 -- 172.21.15.1:0/997822797 learned_addr learned my addr 172.21.15.1:0/997822797 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.006+0000 7fd9f6a00700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9f8105160 0x7fd9f8115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.006+0000 7fd9f6000700 1 -- 172.21.15.1:0/997822797 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9f8105160 msgr2=0x7fd9f8115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.006+0000 7fd9f6000700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9f8105160 0x7fd9f8115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.006+0000 7fd9f6000700 1 -- 172.21.15.1:0/997822797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e8005040 con 0x7fd9f8105b00 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.007+0000 7fd9f6000700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105b00 0x7fd9f81161a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400c800 tx=0x7fd9e4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.007+0000 7fd9eea00700 1 -- 172.21.15.1:0/997822797 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9e400a760 con 0x7fd9f8105b00 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.007+0000 7fd9eea00700 1 -- 172.21.15.1:0/997822797 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9e400ad90 con 0x7fd9f8105b00 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.007+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f810fe20 con 0x7fd9f8105b00 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.007+0000 7fd9eea00700 1 -- 172.21.15.1:0/997822797 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9e40113a0 con 0x7fd9f8105b00 2024-02-28T16:42:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.007+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f8110360 con 0x7fd9f8105b00 2024-02-28T16:42:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.008+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f8102470 con 0x7fd9f8105b00 2024-02-28T16:42:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.009+0000 7fd9eea00700 1 -- 172.21.15.1:0/997822797 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd9e4012400 con 0x7fd9f8105b00 2024-02-28T16:42:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.010+0000 7fd9eea00700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd9dc06d7f0 0x7fd9dc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.010+0000 7fd9eea00700 1 -- 172.21.15.1:0/997822797 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd9e408cdc0 con 0x7fd9f8105b00 2024-02-28T16:42:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.010+0000 7fd9f6a00700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd9dc06d7f0 0x7fd9dc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:01.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.011+0000 7fd9f6a00700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd9dc06d7f0 0x7fd9dc06fca0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8000c00 tx=0x7fd9e8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:02.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.015+0000 7fd9eea00700 1 -- 172.21.15.1:0/997822797 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd9e4056730 con 0x7fd9f8105b00 2024-02-28T16:42:02.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.320+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd9f804bed0 con 0x7fd9f8105b00 2024-02-28T16:42:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.321+0000 7fd9eea00700 1 -- 172.21.15.1:0/997822797 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fd9e4059df0 con 0x7fd9f8105b00 2024-02-28T16:42:02.309 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:42:02.309 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:42:02.309 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:42:02.309 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:42:02.309 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:42:02.309 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:42:02.309 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.323+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd9dc06d7f0 msgr2=0x7fd9dc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.323+0000 7fd9fd600700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd9dc06d7f0 0x7fd9dc06fca0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8000c00 tx=0x7fd9e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.323+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105b00 msgr2=0x7fd9f81161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.323+0000 7fd9fd600700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105b00 0x7fd9f81161a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400c800 tx=0x7fd9e4004470 comp rx=0 tx=0).stop 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.323+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 shutdown_connections 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.323+0000 7fd9fd600700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd9dc06d7f0 0x7fd9dc06fca0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.323+0000 7fd9fd600700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd9f8105160 0x7fd9f8115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.323+0000 7fd9fd600700 1 --2- 172.21.15.1:0/997822797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f8105b00 0x7fd9f81161a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.324+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 >> 172.21.15.1:0/997822797 conn(0x7fd9f80fad30 msgr2=0x7fd9f80fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.324+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 shutdown_connections 2024-02-28T16:42:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:02.324+0000 7fd9fd600700 1 -- 172.21.15.1:0/997822797 wait complete. 2024-02-28T16:42:02.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:02 smithi066 ceph-mon[85781]: pgmap v227: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:02.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:02 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1988051762' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:42:02.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:02 smithi001 ceph-mon[112940]: pgmap v227: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:02.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:02 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1988051762' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:42:03.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:03 smithi066 ceph-mon[85781]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:03.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:03 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/997822797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:42:03.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:03 smithi001 ceph-mon[112940]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:03.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:03 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/997822797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:42:04.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:04 smithi066 ceph-mon[85781]: pgmap v228: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:04.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:04 smithi001 ceph-mon[112940]: pgmap v228: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:06.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:06 smithi066 ceph-mon[85781]: pgmap v229: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:06.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:06 smithi001 ceph-mon[112940]: pgmap v229: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:08.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:08 smithi066 ceph-mon[85781]: pgmap v230: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:08.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:08 smithi001 ceph-mon[112940]: pgmap v230: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:10.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:10 smithi066 ceph-mon[85781]: pgmap v231: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:10.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:10 smithi001 ceph-mon[112940]: pgmap v231: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:11.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:42:11.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:42:12.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:12 smithi066 ceph-mon[85781]: pgmap v232: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:12.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:12 smithi001 ceph-mon[112940]: pgmap v232: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:14.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:14 smithi066 ceph-mon[85781]: pgmap v233: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:14.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:14 smithi001 ceph-mon[112940]: pgmap v233: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:16.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:16 smithi066 ceph-mon[85781]: pgmap v234: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:16.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:16 smithi001 ceph-mon[112940]: pgmap v234: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:18.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:18 smithi066 ceph-mon[85781]: pgmap v235: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:18.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:18 smithi001 ceph-mon[112940]: pgmap v235: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:20.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:20 smithi001 ceph-mon[112940]: pgmap v236: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:20.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:20 smithi066 ceph-mon[85781]: pgmap v236: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:22.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:22 smithi066 ceph-mon[85781]: pgmap v237: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:22.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:22 smithi001 ceph-mon[112940]: pgmap v237: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:24.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:24 smithi066 ceph-mon[85781]: pgmap v238: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:24.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:24 smithi001 ceph-mon[112940]: pgmap v238: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:26.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:26 smithi066 ceph-mon[85781]: pgmap v239: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:26.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:42:26.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:26 smithi001 ceph-mon[112940]: pgmap v239: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:26.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:42:28.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:28 smithi066 ceph-mon[85781]: pgmap v240: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:28.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:28 smithi001 ceph-mon[112940]: pgmap v240: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:30.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:30 smithi066 ceph-mon[85781]: pgmap v241: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:30.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:30 smithi001 ceph-mon[112940]: pgmap v241: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.498+0000 7f58c1600700 1 -- 172.21.15.1:0/1099651670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 msgr2=0x7f58bc1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.498+0000 7f58c1600700 1 --2- 172.21.15.1:0/1099651670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 0x7f58bc1091f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f58a8004940 tx=0x7f58a8009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.498+0000 7f58c1600700 1 -- 172.21.15.1:0/1099651670 shutdown_connections 2024-02-28T16:42:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.498+0000 7f58c1600700 1 --2- 172.21.15.1:0/1099651670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 0x7f58bc1091f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.498+0000 7f58c1600700 1 --2- 172.21.15.1:0/1099651670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f58bc0fbb80 0x7f58bc0fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.498+0000 7f58c1600700 1 -- 172.21.15.1:0/1099651670 >> 172.21.15.1:0/1099651670 conn(0x7f58bc073ab0 msgr2=0x7f58bc073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.498+0000 7f58c1600700 1 -- 172.21.15.1:0/1099651670 shutdown_connections 2024-02-28T16:42:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.498+0000 7f58c1600700 1 -- 172.21.15.1:0/1099651670 wait complete. 2024-02-28T16:42:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.499+0000 7f58c1600700 1 Processor -- start 2024-02-28T16:42:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.499+0000 7f58c1600700 1 -- start start 2024-02-28T16:42:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.499+0000 7f58c1600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f58bc0fbb80 0x7f58bc115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.499+0000 7f58c1600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 0x7f58bc115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.499+0000 7f58c1600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1165d0 con 0x7f58bc0fc490 2024-02-28T16:42:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.499+0000 7f58c1600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc10f900 con 0x7f58bc0fbb80 2024-02-28T16:42:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.500+0000 7f58bb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 0x7f58bc115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.500+0000 7f58bb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 0x7f58bc115f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37088/0 (socket says 172.21.15.1:37088) 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.500+0000 7f58bb400700 1 -- 172.21.15.1:0/1887690575 learned_addr learned my addr 172.21.15.1:0/1887690575 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.500+0000 7f58bbe00700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f58bc0fbb80 0x7f58bc115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.500+0000 7f58bb400700 1 -- 172.21.15.1:0/1887690575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f58bc0fbb80 msgr2=0x7f58bc115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.500+0000 7f58bb400700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f58bc0fbb80 0x7f58bc115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.500+0000 7f58bb400700 1 -- 172.21.15.1:0/1887690575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a8005040 con 0x7f58bc0fc490 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.501+0000 7f58bb400700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 0x7f58bc115f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009100 tx=0x7f58a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.501+0000 7f58b3e00700 1 -- 172.21.15.1:0/1887690575 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58a800e040 con 0x7f58bc0fc490 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.501+0000 7f58b3e00700 1 -- 172.21.15.1:0/1887690575 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58a8004290 con 0x7f58bc0fc490 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.501+0000 7f58b3e00700 1 -- 172.21.15.1:0/1887690575 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58a801c3f0 con 0x7f58bc0fc490 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.501+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc10fb80 con 0x7f58bc0fc490 2024-02-28T16:42:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.501+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc110140 con 0x7f58bc0fc490 2024-02-28T16:42:32.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.502+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc106920 con 0x7f58bc0fc490 2024-02-28T16:42:32.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.503+0000 7f58b3e00700 1 -- 172.21.15.1:0/1887690575 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f58a800bbf0 con 0x7f58bc0fc490 2024-02-28T16:42:32.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.504+0000 7f58b3e00700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f58a006d7f0 0x7f58a006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:32.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.505+0000 7f58b3e00700 1 -- 172.21.15.1:0/1887690575 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f58a808e1c0 con 0x7f58bc0fc490 2024-02-28T16:42:32.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.505+0000 7f58bbe00700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f58a006d7f0 0x7f58a006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:32.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.506+0000 7f58bbe00700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f58a006d7f0 0x7f58a006fca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f58ac004d60 tx=0x7f58ac00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:32.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.510+0000 7f58b3e00700 1 -- 172.21.15.1:0/1887690575 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f58a805b5a0 con 0x7f58bc0fc490 2024-02-28T16:42:32.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.758+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f58bc10c060 con 0x7f58a006d7f0 2024-02-28T16:42:32.746 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:32 smithi001 ceph-mon[112940]: pgmap v242: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:32.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:32 smithi066 ceph-mon[85781]: pgmap v242: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.761+0000 7f58b3e00700 1 -- 172.21.15.1:0/1887690575 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f58bc10c060 con 0x7f58a006d7f0 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.763+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f58a006d7f0 msgr2=0x7f58a006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.763+0000 7f58c1600700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f58a006d7f0 0x7f58a006fca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f58ac004d60 tx=0x7f58ac00b040 comp rx=0 tx=0).stop 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.763+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 msgr2=0x7f58bc115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.763+0000 7f58c1600700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 0x7f58bc115f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009100 tx=0x7f58a8009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.764+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 shutdown_connections 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.764+0000 7f58c1600700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f58a006d7f0 0x7f58a006fca0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.764+0000 7f58c1600700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f58bc0fbb80 0x7f58bc115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.764+0000 7f58c1600700 1 --2- 172.21.15.1:0/1887690575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58bc0fc490 0x7f58bc115f80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.764+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 >> 172.21.15.1:0/1887690575 conn(0x7f58bc073ab0 msgr2=0x7f58bc0fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.764+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 shutdown_connections 2024-02-28T16:42:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.764+0000 7f58c1600700 1 -- 172.21.15.1:0/1887690575 wait complete. 2024-02-28T16:42:32.767 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:42:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.923+0000 7f32a8e00700 1 -- 172.21.15.1:0/122804010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a4102f50 msgr2=0x7f32a4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.923+0000 7f32a8e00700 1 --2- 172.21.15.1:0/122804010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a4102f50 0x7f32a4103320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3294004940 tx=0x7f3294009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.924+0000 7f32a8e00700 1 -- 172.21.15.1:0/122804010 shutdown_connections 2024-02-28T16:42:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.924+0000 7f32a8e00700 1 --2- 172.21.15.1:0/122804010 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a41038f0 0x7f32a40fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.924+0000 7f32a8e00700 1 --2- 172.21.15.1:0/122804010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a4102f50 0x7f32a4103320 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.924+0000 7f32a8e00700 1 -- 172.21.15.1:0/122804010 >> 172.21.15.1:0/122804010 conn(0x7f32a40f8ae0 msgr2=0x7f32a40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.924+0000 7f32a8e00700 1 -- 172.21.15.1:0/122804010 shutdown_connections 2024-02-28T16:42:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.924+0000 7f32a8e00700 1 -- 172.21.15.1:0/122804010 wait complete. 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.925+0000 7f32a8e00700 1 Processor -- start 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.925+0000 7f32a8e00700 1 -- start start 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.925+0000 7f32a8e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a4102f50 0x7f32a4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.925+0000 7f32a8e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a41038f0 0x7f32a4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.925+0000 7f32a8e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a4116790 con 0x7f32a41038f0 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.925+0000 7f32a8e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a410fae0 con 0x7f32a4102f50 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.926+0000 7f32a1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a41038f0 0x7f32a4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.926+0000 7f32a1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a41038f0 0x7f32a4116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37112/0 (socket says 172.21.15.1:37112) 2024-02-28T16:42:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.926+0000 7f32a1400700 1 -- 172.21.15.1:0/1767731302 learned_addr learned my addr 172.21.15.1:0/1767731302 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.926+0000 7f32a1e00700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a4102f50 0x7f32a4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.926+0000 7f32a1400700 1 -- 172.21.15.1:0/1767731302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a4102f50 msgr2=0x7f32a4115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.926+0000 7f32a1400700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a4102f50 0x7f32a4115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.926+0000 7f32a1400700 1 -- 172.21.15.1:0/1767731302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3294005040 con 0x7f32a41038f0 2024-02-28T16:42:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.927+0000 7f32a1400700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a41038f0 0x7f32a4116140 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3290002a50 tx=0x7f3290004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.927+0000 7f329a000700 1 -- 172.21.15.1:0/1767731302 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3290007cc0 con 0x7f32a41038f0 2024-02-28T16:42:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.927+0000 7f329a000700 1 -- 172.21.15.1:0/1767731302 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f329000f040 con 0x7f32a41038f0 2024-02-28T16:42:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.927+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a410fdc0 con 0x7f32a41038f0 2024-02-28T16:42:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.927+0000 7f329a000700 1 -- 172.21.15.1:0/1767731302 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329000daf0 con 0x7f32a41038f0 2024-02-28T16:42:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.927+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a4110300 con 0x7f32a41038f0 2024-02-28T16:42:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.928+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a40fc6e0 con 0x7f32a41038f0 2024-02-28T16:42:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.929+0000 7f329a000700 1 -- 172.21.15.1:0/1767731302 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3290009810 con 0x7f32a41038f0 2024-02-28T16:42:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.930+0000 7f329a000700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f328806d7f0 0x7f328806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.930+0000 7f329a000700 1 -- 172.21.15.1:0/1767731302 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f329008ccf0 con 0x7f32a41038f0 2024-02-28T16:42:32.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.931+0000 7f32a1e00700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f328806d7f0 0x7f328806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.931+0000 7f32a1e00700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f328806d7f0 0x7f328806fca0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f329400b3b0 tx=0x7f3294009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:32.935+0000 7f329a000700 1 -- 172.21.15.1:0/1767731302 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f32900564a0 con 0x7f32a41038f0 2024-02-28T16:42:33.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.183+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32a4111250 con 0x7f328806d7f0 2024-02-28T16:42:33.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.186+0000 7f329a000700 1 -- 172.21.15.1:0/1767731302 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f32a4111250 con 0x7f328806d7f0 2024-02-28T16:42:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.188+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f328806d7f0 msgr2=0x7f328806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.188+0000 7f32a8e00700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f328806d7f0 0x7f328806fca0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f329400b3b0 tx=0x7f3294009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.188+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a41038f0 msgr2=0x7f32a4116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.188+0000 7f32a8e00700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a41038f0 0x7f32a4116140 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3290002a50 tx=0x7f3290004600 comp rx=0 tx=0).stop 2024-02-28T16:42:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.189+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 shutdown_connections 2024-02-28T16:42:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.189+0000 7f32a8e00700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f328806d7f0 0x7f328806fca0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.189+0000 7f32a8e00700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32a4102f50 0x7f32a4115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.189+0000 7f32a8e00700 1 --2- 172.21.15.1:0/1767731302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a41038f0 0x7f32a4116140 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.189+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 >> 172.21.15.1:0/1767731302 conn(0x7f32a40f8ae0 msgr2=0x7f32a40fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:33.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.189+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 shutdown_connections 2024-02-28T16:42:33.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.189+0000 7f32a8e00700 1 -- 172.21.15.1:0/1767731302 wait complete. 2024-02-28T16:42:33.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.337+0000 7f82c3600700 1 -- 172.21.15.1:0/3193070614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc072dd0 msgr2=0x7f82bc10d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:33.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.337+0000 7f82c3600700 1 --2- 172.21.15.1:0/3193070614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc072dd0 0x7f82bc10d8e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f82a8004940 tx=0x7f82a8009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:33.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.337+0000 7f82c3600700 1 -- 172.21.15.1:0/3193070614 shutdown_connections 2024-02-28T16:42:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.337+0000 7f82c3600700 1 --2- 172.21.15.1:0/3193070614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc072dd0 0x7f82bc10d8e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.337+0000 7f82c3600700 1 --2- 172.21.15.1:0/3193070614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82bc0724c0 0x7f82bc072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.337+0000 7f82c3600700 1 -- 172.21.15.1:0/3193070614 >> 172.21.15.1:0/3193070614 conn(0x7f82bc0face0 msgr2=0x7f82bc0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.337+0000 7f82c3600700 1 -- 172.21.15.1:0/3193070614 shutdown_connections 2024-02-28T16:42:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.337+0000 7f82c3600700 1 -- 172.21.15.1:0/3193070614 wait complete. 2024-02-28T16:42:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.338+0000 7f82c3600700 1 Processor -- start 2024-02-28T16:42:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.338+0000 7f82c3600700 1 -- start start 2024-02-28T16:42:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.339+0000 7f82c3600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc0724c0 0x7f82bc11a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.339+0000 7f82c3600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82bc072dd0 0x7f82bc11a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.339+0000 7f82c3600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc11abc0 con 0x7f82bc0724c0 2024-02-28T16:42:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.339+0000 7f82c3600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc113ed0 con 0x7f82bc072dd0 2024-02-28T16:42:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.339+0000 7f82c0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc0724c0 0x7f82bc11a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.339+0000 7f82c0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc0724c0 0x7f82bc11a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37130/0 (socket says 172.21.15.1:37130) 2024-02-28T16:42:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.339+0000 7f82c0c00700 1 -- 172.21.15.1:0/1913945629 learned_addr learned my addr 172.21.15.1:0/1913945629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82bbe00700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82bc072dd0 0x7f82bc11a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82c0c00700 1 -- 172.21.15.1:0/1913945629 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82bc072dd0 msgr2=0x7f82bc11a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82c0c00700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82bc072dd0 0x7f82bc11a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82c0c00700 1 -- 172.21.15.1:0/1913945629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a8005040 con 0x7f82bc0724c0 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82c0c00700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc0724c0 0x7f82bc11a030 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f82b0002a50 tx=0x7f82b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82b8c00700 1 -- 172.21.15.1:0/1913945629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b0007cc0 con 0x7f82bc0724c0 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82b8c00700 1 -- 172.21.15.1:0/1913945629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82b000f040 con 0x7f82bc0724c0 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc1141b0 con 0x7f82bc0724c0 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82b8c00700 1 -- 172.21.15.1:0/1913945629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b000db70 con 0x7f82bc0724c0 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.340+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc1146f0 con 0x7f82bc0724c0 2024-02-28T16:42:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.342+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc10b010 con 0x7f82bc0724c0 2024-02-28T16:42:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.343+0000 7f82b8c00700 1 -- 172.21.15.1:0/1913945629 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f82b000dd10 con 0x7f82bc0724c0 2024-02-28T16:42:33.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.344+0000 7f82b8c00700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82a406d7e0 0x7f82a406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:33.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.344+0000 7f82b8c00700 1 -- 172.21.15.1:0/1913945629 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f82b008c590 con 0x7f82bc0724c0 2024-02-28T16:42:33.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.344+0000 7f82bbe00700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82a406d7e0 0x7f82a406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:33.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.344+0000 7f82bbe00700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82a406d7e0 0x7f82a406fc90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f82a8004f90 tx=0x7f82a8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:33.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.348+0000 7f82b8c00700 1 -- 172.21.15.1:0/1913945629 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82b00598f0 con 0x7f82bc0724c0 2024-02-28T16:42:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.575+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f82bc115610 con 0x7f82a406d7e0 2024-02-28T16:42:33.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.585+0000 7f82b8c00700 1 -- 172.21.15.1:0/1913945629 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f82bc115610 con 0x7f82a406d7e0 2024-02-28T16:42:33.573 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:42:33.573 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (8m) 44s ago 22m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:42:33.573 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (5m) 44s ago 23m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:42:33.573 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (5m) 97s ago 21m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:42:33.573 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (8m) 44s ago 22m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:42:33.573 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (77s) 44s ago 15m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (51s) 44s ago 15m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (15m) 97s ago 15m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (15m) 97s ago 15m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (11m) 44s ago 24m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (10m) 97s ago 21m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (7m) 44s ago 24m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (6m) 97s ago 21m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (9m) 44s ago 23m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (9m) 97s ago 21m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (4m) 44s ago 20m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (4m) 44s ago 20m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (3m) 44s ago 19m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:42:33.574 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (3m) 97s ago 18m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:42:33.575 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (2m) 97s ago 18m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:42:33.575 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (103s) 97s ago 17m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:42:33.575 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (9m) 44s ago 22m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:42:33.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82a406d7e0 msgr2=0x7f82a406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82a406d7e0 0x7f82a406fc90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f82a8004f90 tx=0x7f82a8009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc0724c0 msgr2=0x7f82bc11a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc0724c0 0x7f82bc11a030 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f82b0002a50 tx=0x7f82b00046b0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 shutdown_connections 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82a406d7e0 0x7f82a406fc90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc0724c0 0x7f82bc11a030 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 --2- 172.21.15.1:0/1913945629 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82bc072dd0 0x7f82bc11a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 >> 172.21.15.1:0/1913945629 conn(0x7f82bc0face0 msgr2=0x7f82bc0fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 shutdown_connections 2024-02-28T16:42:33.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.588+0000 7f82c3600700 1 -- 172.21.15.1:0/1913945629 wait complete. 2024-02-28T16:42:33.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.748+0000 7f3574c00700 1 -- 172.21.15.1:0/3811992313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 msgr2=0x7f35700ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:33.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.748+0000 7f3574c00700 1 --2- 172.21.15.1:0/3811992313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 0x7f35700ff180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f355c004940 tx=0x7f355c009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:33.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.748+0000 7f3574c00700 1 -- 172.21.15.1:0/3811992313 shutdown_connections 2024-02-28T16:42:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.748+0000 7f3574c00700 1 --2- 172.21.15.1:0/3811992313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 0x7f35700ff180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.748+0000 7f3574c00700 1 --2- 172.21.15.1:0/3811992313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3570105170 0x7f3570105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.748+0000 7f3574c00700 1 -- 172.21.15.1:0/3811992313 >> 172.21.15.1:0/3811992313 conn(0x7f35700face0 msgr2=0x7f35700fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.748+0000 7f3574c00700 1 -- 172.21.15.1:0/3811992313 shutdown_connections 2024-02-28T16:42:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.748+0000 7f3574c00700 1 -- 172.21.15.1:0/3811992313 wait complete. 2024-02-28T16:42:33.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.749+0000 7f3574c00700 1 Processor -- start 2024-02-28T16:42:33.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.749+0000 7f3574c00700 1 -- start start 2024-02-28T16:42:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.749+0000 7f3574c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3570105170 0x7f3570115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f3574c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 0x7f3570116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f3574c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570116870 con 0x7f3570105b10 2024-02-28T16:42:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f3574c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357010fb80 con 0x7f3570105170 2024-02-28T16:42:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f356d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 0x7f3570116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f356d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 0x7f3570116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37148/0 (socket says 172.21.15.1:37148) 2024-02-28T16:42:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f356d400700 1 -- 172.21.15.1:0/2944838280 learned_addr learned my addr 172.21.15.1:0/2944838280 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f356de00700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3570105170 0x7f3570115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f356d400700 1 -- 172.21.15.1:0/2944838280 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3570105170 msgr2=0x7f3570115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f356d400700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3570105170 0x7f3570115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.750+0000 7f356d400700 1 -- 172.21.15.1:0/2944838280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c005040 con 0x7f3570105b10 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.751+0000 7f356d400700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 0x7f3570116220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f355c009030 tx=0x7f355c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.751+0000 7f3566000700 1 -- 172.21.15.1:0/2944838280 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f355c00e040 con 0x7f3570105b10 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.751+0000 7f3566000700 1 -- 172.21.15.1:0/2944838280 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f355c00bbf0 con 0x7f3570105b10 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.751+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357010fe00 con 0x7f3570105b10 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.751+0000 7f3566000700 1 -- 172.21.15.1:0/2944838280 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f355c01c630 con 0x7f3570105b10 2024-02-28T16:42:33.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.751+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3570110340 con 0x7f3570105b10 2024-02-28T16:42:33.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.752+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570107640 con 0x7f3570105b10 2024-02-28T16:42:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.753+0000 7f3566000700 1 -- 172.21.15.1:0/2944838280 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f355c00bd60 con 0x7f3570105b10 2024-02-28T16:42:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.754+0000 7f3566000700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355406d7f0 0x7f355406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.754+0000 7f3566000700 1 -- 172.21.15.1:0/2944838280 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f355c08e1f0 con 0x7f3570105b10 2024-02-28T16:42:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.754+0000 7f356de00700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355406d7f0 0x7f355406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.755+0000 7f356de00700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355406d7f0 0x7f355406fca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3560004d60 tx=0x7f356000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:33.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:33.759+0000 7f3566000700 1 -- 172.21.15.1:0/2944838280 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f355c05b5d0 con 0x7f3570105b10 2024-02-28T16:42:34.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.079+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f357004bed0 con 0x7f3570105b10 2024-02-28T16:42:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.080+0000 7f3566000700 1 -- 172.21.15.1:0/2944838280 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f355c05b160 con 0x7f3570105b10 2024-02-28T16:42:34.067 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:42:34.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:42:34.069 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:42:34.069 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:42:34.069 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:42:34.069 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:42:34.069 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355406d7f0 msgr2=0x7f355406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355406d7f0 0x7f355406fca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3560004d60 tx=0x7f356000b040 comp rx=0 tx=0).stop 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 msgr2=0x7f3570116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 0x7f3570116220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f355c009030 tx=0x7f355c009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 shutdown_connections 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f355406d7f0 0x7f355406fca0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3570105170 0x7f3570115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 --2- 172.21.15.1:0/2944838280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3570105b10 0x7f3570116220 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.084+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 >> 172.21.15.1:0/2944838280 conn(0x7f35700face0 msgr2=0x7f35700fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.085+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 shutdown_connections 2024-02-28T16:42:34.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.085+0000 7f3574c00700 1 -- 172.21.15.1:0/2944838280 wait complete. 2024-02-28T16:42:34.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.248+0000 7f9562000700 1 -- 172.21.15.1:0/1445048273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c070be0 msgr2=0x7f955c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.248+0000 7f9562000700 1 --2- 172.21.15.1:0/1445048273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c070be0 0x7f955c1092c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9548004970 tx=0x7f9548009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:34.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.248+0000 7f9562000700 1 -- 172.21.15.1:0/1445048273 shutdown_connections 2024-02-28T16:42:34.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.248+0000 7f9562000700 1 --2- 172.21.15.1:0/1445048273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c070be0 0x7f955c1092c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.248+0000 7f9562000700 1 --2- 172.21.15.1:0/1445048273 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f955c0702d0 0x7f955c0706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.248+0000 7f9562000700 1 -- 172.21.15.1:0/1445048273 >> 172.21.15.1:0/1445048273 conn(0x7f955c0f8ae0 msgr2=0x7f955c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:34.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.249+0000 7f9562000700 1 -- 172.21.15.1:0/1445048273 shutdown_connections 2024-02-28T16:42:34.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.249+0000 7f9562000700 1 -- 172.21.15.1:0/1445048273 wait complete. 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.249+0000 7f9562000700 1 Processor -- start 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.250+0000 7f9562000700 1 -- start start 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.250+0000 7f9562000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c0702d0 0x7f955c115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.250+0000 7f9562000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f955c070be0 0x7f955c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.250+0000 7f9562000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c1167a0 con 0x7f955c0702d0 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.250+0000 7f9562000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c10faf0 con 0x7f955c070be0 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.250+0000 7f955b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c0702d0 0x7f955c115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.251+0000 7f955b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c0702d0 0x7f955c115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37176/0 (socket says 172.21.15.1:37176) 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.251+0000 7f955b400700 1 -- 172.21.15.1:0/3444704355 learned_addr learned my addr 172.21.15.1:0/3444704355 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:34.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.251+0000 7f955aa00700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f955c070be0 0x7f955c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:34.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.251+0000 7f955b400700 1 -- 172.21.15.1:0/3444704355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f955c070be0 msgr2=0x7f955c116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.251+0000 7f955b400700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f955c070be0 0x7f955c116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.251+0000 7f955b400700 1 -- 172.21.15.1:0/3444704355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9548005040 con 0x7f955c0702d0 2024-02-28T16:42:34.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.251+0000 7f955aa00700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f955c070be0 0x7f955c116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:42:34.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.251+0000 7f955b400700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c0702d0 0x7f955c115c10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f954c00eb40 tx=0x7f954c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:34.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.252+0000 7f9553400700 1 -- 172.21.15.1:0/3444704355 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f954c007cc0 con 0x7f955c0702d0 2024-02-28T16:42:34.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.252+0000 7f9553400700 1 -- 172.21.15.1:0/3444704355 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f954c00f040 con 0x7f955c0702d0 2024-02-28T16:42:34.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.252+0000 7f9553400700 1 -- 172.21.15.1:0/3444704355 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f954c009620 con 0x7f955c0702d0 2024-02-28T16:42:34.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.252+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c10fd70 con 0x7f955c0702d0 2024-02-28T16:42:34.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.252+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c1102b0 con 0x7f955c0702d0 2024-02-28T16:42:34.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.253+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c106980 con 0x7f955c0702d0 2024-02-28T16:42:34.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.254+0000 7f9553400700 1 -- 172.21.15.1:0/3444704355 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f954c009780 con 0x7f955c0702d0 2024-02-28T16:42:34.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.255+0000 7f9553400700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f954006d7f0 0x7f954006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:34.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.255+0000 7f9553400700 1 -- 172.21.15.1:0/3444704355 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f954c08cb40 con 0x7f955c0702d0 2024-02-28T16:42:34.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.255+0000 7f955aa00700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f954006d7f0 0x7f954006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:34.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.256+0000 7f955aa00700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f954006d7f0 0x7f954006fca0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f955c1114b0 tx=0x7f9548009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:34.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.260+0000 7f9553400700 1 -- 172.21.15.1:0/3444704355 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f954c056420 con 0x7f955c0702d0 2024-02-28T16:42:34.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.532+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f955c1111f0 con 0x7f955c0702d0 2024-02-28T16:42:34.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.533+0000 7f9553400700 1 -- 172.21.15.1:0/3444704355 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f954c019740 con 0x7f955c0702d0 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:42:34.521 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:42:34.522 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:42:34.523 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:42:34.524 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:42:34.524 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:42:34.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f954006d7f0 msgr2=0x7f954006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f954006d7f0 0x7f954006fca0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f955c1114b0 tx=0x7f9548009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c0702d0 msgr2=0x7f955c115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c0702d0 0x7f955c115c10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f954c00eb40 tx=0x7f954c00b040 comp rx=0 tx=0).stop 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 shutdown_connections 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f954006d7f0 0x7f954006fca0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f955c0702d0 0x7f955c115c10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 --2- 172.21.15.1:0/3444704355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f955c070be0 0x7f955c116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 >> 172.21.15.1:0/3444704355 conn(0x7f955c0f8ae0 msgr2=0x7f955c103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 shutdown_connections 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.536+0000 7f9562000700 1 -- 172.21.15.1:0/3444704355 wait complete. 2024-02-28T16:42:34.525 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:42:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.699+0000 7fa6ac400700 1 -- 172.21.15.1:0/2039426612 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a40fe8f0 msgr2=0x7fa6a4107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.699+0000 7fa6ac400700 1 --2- 172.21.15.1:0/2039426612 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a40fe8f0 0x7fa6a4107050 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa690004880 tx=0x7fa690009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:34.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:34 smithi001 ceph-mon[112940]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:34.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:34 smithi001 ceph-mon[112940]: pgmap v243: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:34.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:34 smithi001 ceph-mon[112940]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:34.687 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:34 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2944838280' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:42:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.699+0000 7fa6ac400700 1 -- 172.21.15.1:0/2039426612 shutdown_connections 2024-02-28T16:42:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.699+0000 7fa6ac400700 1 --2- 172.21.15.1:0/2039426612 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a40fe8f0 0x7fa6a4107050 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.699+0000 7fa6ac400700 1 --2- 172.21.15.1:0/2039426612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a40fdfe0 0x7fa6a40fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.699+0000 7fa6ac400700 1 -- 172.21.15.1:0/2039426612 >> 172.21.15.1:0/2039426612 conn(0x7fa6a4071d90 msgr2=0x7fa6a4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.700+0000 7fa6ac400700 1 -- 172.21.15.1:0/2039426612 shutdown_connections 2024-02-28T16:42:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.700+0000 7fa6ac400700 1 -- 172.21.15.1:0/2039426612 wait complete. 2024-02-28T16:42:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.701+0000 7fa6ac400700 1 Processor -- start 2024-02-28T16:42:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.701+0000 7fa6ac400700 1 -- start start 2024-02-28T16:42:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.702+0000 7fa6ac400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a40fdfe0 0x7fa6a4115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.702+0000 7fa6ac400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a40fe8f0 0x7fa6a41161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.702+0000 7fa6ac400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a4116840 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.702+0000 7fa6ac400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a410fb50 con 0x7fa6a40fdfe0 2024-02-28T16:42:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.702+0000 7fa6a9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a40fe8f0 0x7fa6a41161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.702+0000 7fa6a9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a40fe8f0 0x7fa6a41161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37196/0 (socket says 172.21.15.1:37196) 2024-02-28T16:42:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.702+0000 7fa6a9000700 1 -- 172.21.15.1:0/1964014343 learned_addr learned my addr 172.21.15.1:0/1964014343 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.703+0000 7fa6a9a00700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a40fdfe0 0x7fa6a4115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.703+0000 7fa6a9000700 1 -- 172.21.15.1:0/1964014343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a40fdfe0 msgr2=0x7fa6a4115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.703+0000 7fa6a9000700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a40fdfe0 0x7fa6a4115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.703+0000 7fa6a9000700 1 -- 172.21.15.1:0/1964014343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa690005040 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.704+0000 7fa6a9000700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a40fe8f0 0x7fa6a41161f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa6900134a0 tx=0x7fa690009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.704+0000 7fa6a1600700 1 -- 172.21.15.1:0/1964014343 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa69000e030 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.704+0000 7fa6a1600700 1 -- 172.21.15.1:0/1964014343 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa690013e90 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.704+0000 7fa6a1600700 1 -- 172.21.15.1:0/1964014343 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6900123c0 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.704+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a410fdd0 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.705+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a4110310 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.706+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a4106ac0 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.708+0000 7fa6a1600700 1 -- 172.21.15.1:0/1964014343 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa69001c400 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.710+0000 7fa6a1600700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68c06d7e0 0x7fa68c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:34.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.710+0000 7fa6a1600700 1 -- 172.21.15.1:0/1964014343 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa69008e6a0 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.711+0000 7fa6a9a00700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68c06d7e0 0x7fa68c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:34.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.711+0000 7fa6a9a00700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68c06d7e0 0x7fa68c06fc90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa698009b80 tx=0x7fa69800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:34.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.715+0000 7fa6a1600700 1 -- 172.21.15.1:0/1964014343 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa69005bb40 con 0x7fa6a40fe8f0 2024-02-28T16:42:34.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:34 smithi066 ceph-mon[85781]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:34.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:34 smithi066 ceph-mon[85781]: pgmap v243: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:34.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:34 smithi066 ceph-mon[85781]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:34.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:34 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2944838280' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:42:34.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.956+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6a4111290 con 0x7fa68c06d7e0 2024-02-28T16:42:34.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.960+0000 7fa6a1600700 1 -- 172.21.15.1:0/1964014343 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa6a4111290 con 0x7fa68c06d7e0 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:42:34.948 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68c06d7e0 msgr2=0x7fa68c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68c06d7e0 0x7fa68c06fc90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa698009b80 tx=0x7fa69800b040 comp rx=0 tx=0).stop 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a40fe8f0 msgr2=0x7fa6a41161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a40fe8f0 0x7fa6a41161f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa6900134a0 tx=0x7fa690009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 shutdown_connections 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68c06d7e0 0x7fa68c06fc90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a40fdfe0 0x7fa6a4115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 --2- 172.21.15.1:0/1964014343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a40fe8f0 0x7fa6a41161f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:34.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 >> 172.21.15.1:0/1964014343 conn(0x7fa6a4071d90 msgr2=0x7fa6a4103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:34.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 shutdown_connections 2024-02-28T16:42:34.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:34.963+0000 7fa6ac400700 1 -- 172.21.15.1:0/1964014343 wait complete. 2024-02-28T16:42:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.105+0000 7fc5aec00700 1 -- 172.21.15.1:0/2036221294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105170 msgr2=0x7fc5a8105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.105+0000 7fc5aec00700 1 --2- 172.21.15.1:0/2036221294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105170 0x7fc5a8105540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc598004940 tx=0x7fc598009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.105+0000 7fc5aec00700 1 -- 172.21.15.1:0/2036221294 shutdown_connections 2024-02-28T16:42:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.105+0000 7fc5aec00700 1 --2- 172.21.15.1:0/2036221294 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a8105b10 0x7fc5a80ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.105+0000 7fc5aec00700 1 --2- 172.21.15.1:0/2036221294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105170 0x7fc5a8105540 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.105+0000 7fc5aec00700 1 -- 172.21.15.1:0/2036221294 >> 172.21.15.1:0/2036221294 conn(0x7fc5a80face0 msgr2=0x7fc5a80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.105+0000 7fc5aec00700 1 -- 172.21.15.1:0/2036221294 shutdown_connections 2024-02-28T16:42:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.105+0000 7fc5aec00700 1 -- 172.21.15.1:0/2036221294 wait complete. 2024-02-28T16:42:35.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.106+0000 7fc5aec00700 1 Processor -- start 2024-02-28T16:42:35.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.106+0000 7fc5aec00700 1 -- start start 2024-02-28T16:42:35.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5aec00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a8105170 0x7fc5a8115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:35.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5aec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105b10 0x7fc5a8116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:35.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5aec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a81168e0 con 0x7fc5a8105b10 2024-02-28T16:42:35.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5aec00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a810fbf0 con 0x7fc5a8105170 2024-02-28T16:42:35.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5a7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105b10 0x7fc5a8116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5a7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105b10 0x7fc5a8116290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37208/0 (socket says 172.21.15.1:37208) 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5a7400700 1 -- 172.21.15.1:0/2936917968 learned_addr learned my addr 172.21.15.1:0/2936917968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5a7400700 1 -- 172.21.15.1:0/2936917968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a8105170 msgr2=0x7fc5a8115d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5a7e00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a8105170 0x7fc5a8115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5a7400700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a8105170 0x7fc5a8115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.107+0000 7fc5a7400700 1 -- 172.21.15.1:0/2936917968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598005040 con 0x7fc5a8105b10 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.108+0000 7fc5a7400700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105b10 0x7fc5a8116290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc594002a80 tx=0x7fc594004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.108+0000 7fc59fe00700 1 -- 172.21.15.1:0/2936917968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc594007cc0 con 0x7fc5a8105b10 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.108+0000 7fc59fe00700 1 -- 172.21.15.1:0/2936917968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc59400f040 con 0x7fc5a8105b10 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.108+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a810fed0 con 0x7fc5a8105b10 2024-02-28T16:42:35.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.108+0000 7fc59fe00700 1 -- 172.21.15.1:0/2936917968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc59400daf0 con 0x7fc5a8105b10 2024-02-28T16:42:35.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.108+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a8110410 con 0x7fc5a8105b10 2024-02-28T16:42:35.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.109+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a8102480 con 0x7fc5a8105b10 2024-02-28T16:42:35.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.111+0000 7fc59fe00700 1 -- 172.21.15.1:0/2936917968 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc59400dc90 con 0x7fc5a8105b10 2024-02-28T16:42:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.112+0000 7fc59fe00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58c06d7f0 0x7fc58c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:42:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.112+0000 7fc59fe00700 1 -- 172.21.15.1:0/2936917968 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc59408c660 con 0x7fc5a8105b10 2024-02-28T16:42:35.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.112+0000 7fc5a7e00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58c06d7f0 0x7fc58c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:42:35.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.113+0000 7fc5a7e00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58c06d7f0 0x7fc58c06fca0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc59800b3b0 tx=0x7fc598009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:42:35.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.117+0000 7fc59fe00700 1 -- 172.21.15.1:0/2936917968 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc594059940 con 0x7fc5a8105b10 2024-02-28T16:42:35.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.412+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc5a8111330 con 0x7fc5a8105b10 2024-02-28T16:42:35.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.418+0000 7fc59fe00700 1 -- 172.21.15.1:0/2936917968 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fc5940594d0 con 0x7fc5a8105b10 2024-02-28T16:42:35.406 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:42:35.406 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:42:35.406 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:42:35.406 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:42:35.406 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:42:35.406 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:42:35.406 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:42:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.421+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58c06d7f0 msgr2=0x7fc58c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.421+0000 7fc5aec00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58c06d7f0 0x7fc58c06fca0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc59800b3b0 tx=0x7fc598009e30 comp rx=0 tx=0).stop 2024-02-28T16:42:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.421+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105b10 msgr2=0x7fc5a8116290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:42:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.421+0000 7fc5aec00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105b10 0x7fc5a8116290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc594002a80 tx=0x7fc594004630 comp rx=0 tx=0).stop 2024-02-28T16:42:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.422+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 shutdown_connections 2024-02-28T16:42:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.422+0000 7fc5aec00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58c06d7f0 0x7fc58c06fca0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.422+0000 7fc5aec00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a8105170 0x7fc5a8115d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.422+0000 7fc5aec00700 1 --2- 172.21.15.1:0/2936917968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a8105b10 0x7fc5a8116290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:42:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.422+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 >> 172.21.15.1:0/2936917968 conn(0x7fc5a80face0 msgr2=0x7fc5a80fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:42:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.422+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 shutdown_connections 2024-02-28T16:42:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:42:35.422+0000 7fc5aec00700 1 -- 172.21.15.1:0/2936917968 wait complete. 2024-02-28T16:42:35.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:35 smithi066 ceph-mon[85781]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:35.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:35 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3444704355' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:42:35.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:35 smithi001 ceph-mon[112940]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:35.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:35 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3444704355' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:42:36.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:36 smithi066 ceph-mon[85781]: pgmap v244: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:36.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:36 smithi066 ceph-mon[85781]: from='client.34522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:36.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2936917968' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:42:36.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:36 smithi001 ceph-mon[112940]: pgmap v244: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:36.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:36 smithi001 ceph-mon[112940]: from='client.34522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:42:36.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2936917968' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:42:38.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:38 smithi066 ceph-mon[85781]: pgmap v245: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:38.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:38 smithi001 ceph-mon[112940]: pgmap v245: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:40.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:40 smithi066 ceph-mon[85781]: pgmap v246: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:40.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:40 smithi001 ceph-mon[112940]: pgmap v246: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:41.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:42:41.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:42:42.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:42 smithi066 ceph-mon[85781]: pgmap v247: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:42.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:42 smithi001 ceph-mon[112940]: pgmap v247: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:44.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:44 smithi066 ceph-mon[85781]: pgmap v248: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:44.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:44 smithi001 ceph-mon[112940]: pgmap v248: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:46.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:46 smithi066 ceph-mon[85781]: pgmap v249: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:46.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:46 smithi001 ceph-mon[112940]: pgmap v249: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:48.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:48 smithi066 ceph-mon[85781]: pgmap v250: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:48.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:48 smithi001 ceph-mon[112940]: pgmap v250: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:50.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:50 smithi066 ceph-mon[85781]: pgmap v251: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:50.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:50 smithi001 ceph-mon[112940]: pgmap v251: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:52.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:52 smithi066 ceph-mon[85781]: pgmap v252: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:52.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:52 smithi001 ceph-mon[112940]: pgmap v252: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:54.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:54 smithi066 ceph-mon[85781]: pgmap v253: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:54.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:54 smithi001 ceph-mon[112940]: pgmap v253: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:56.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:56 smithi066 ceph-mon[85781]: pgmap v254: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:56.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:42:56.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:56 smithi001 ceph-mon[112940]: pgmap v254: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:56.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:42:58.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:42:58 smithi066 ceph-mon[85781]: pgmap v255: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:42:58.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:42:58 smithi001 ceph-mon[112940]: pgmap v255: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:00.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:00 smithi066 ceph-mon[85781]: pgmap v256: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:00.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:00 smithi001 ceph-mon[112940]: pgmap v256: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:02.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:02 smithi001 ceph-mon[112940]: pgmap v257: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:02.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:02 smithi066 ceph-mon[85781]: pgmap v257: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:04.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:04 smithi001 ceph-mon[112940]: pgmap v258: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:04.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:04 smithi066 ceph-mon[85781]: pgmap v258: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:05.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.593+0000 7fa4ba800700 1 -- 172.21.15.1:0/4261290040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 msgr2=0x7fa4b40fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:05.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.593+0000 7fa4ba800700 1 --2- 172.21.15.1:0/4261290040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 0x7fa4b40fcf60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0004940 tx=0x7fa4a0009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:05.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.593+0000 7fa4ba800700 1 -- 172.21.15.1:0/4261290040 shutdown_connections 2024-02-28T16:43:05.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.593+0000 7fa4ba800700 1 --2- 172.21.15.1:0/4261290040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 0x7fa4b40fcf60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:05.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.593+0000 7fa4ba800700 1 --2- 172.21.15.1:0/4261290040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4b4102f50 0x7fa4b4103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:05.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.593+0000 7fa4ba800700 1 -- 172.21.15.1:0/4261290040 >> 172.21.15.1:0/4261290040 conn(0x7fa4b40f8ae0 msgr2=0x7fa4b40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:05.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.594+0000 7fa4ba800700 1 -- 172.21.15.1:0/4261290040 shutdown_connections 2024-02-28T16:43:05.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.594+0000 7fa4ba800700 1 -- 172.21.15.1:0/4261290040 wait complete. 2024-02-28T16:43:05.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.594+0000 7fa4ba800700 1 Processor -- start 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.594+0000 7fa4ba800700 1 -- start start 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.595+0000 7fa4ba800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4b4102f50 0x7fa4b4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.595+0000 7fa4ba800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 0x7fa4b4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.595+0000 7fa4ba800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b4114600 con 0x7fa4b41038f0 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.595+0000 7fa4ba800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b410d930 con 0x7fa4b4102f50 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.595+0000 7fa4b2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 0x7fa4b4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.595+0000 7fa4b2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 0x7fa4b4113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60370/0 (socket says 172.21.15.1:60370) 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.595+0000 7fa4b2a00700 1 -- 172.21.15.1:0/802879851 learned_addr learned my addr 172.21.15.1:0/802879851 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.595+0000 7fa4b3400700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4b4102f50 0x7fa4b4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.596+0000 7fa4b2a00700 1 -- 172.21.15.1:0/802879851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4b4102f50 msgr2=0x7fa4b4113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.596+0000 7fa4b2a00700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4b4102f50 0x7fa4b4113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.596+0000 7fa4b2a00700 1 -- 172.21.15.1:0/802879851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a0005040 con 0x7fa4b41038f0 2024-02-28T16:43:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.596+0000 7fa4b2a00700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 0x7fa4b4113fb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009100 tx=0x7fa4a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.596+0000 7fa4ab400700 1 -- 172.21.15.1:0/802879851 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4a000e040 con 0x7fa4b41038f0 2024-02-28T16:43:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.596+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b410dbb0 con 0x7fa4b41038f0 2024-02-28T16:43:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.596+0000 7fa4ab400700 1 -- 172.21.15.1:0/802879851 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4a0004290 con 0x7fa4b41038f0 2024-02-28T16:43:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.596+0000 7fa4ab400700 1 -- 172.21.15.1:0/802879851 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4a001c3f0 con 0x7fa4b41038f0 2024-02-28T16:43:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.597+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b410e0f0 con 0x7fa4b41038f0 2024-02-28T16:43:05.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.598+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b4105420 con 0x7fa4b41038f0 2024-02-28T16:43:05.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.599+0000 7fa4ab400700 1 -- 172.21.15.1:0/802879851 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa4a000bbf0 con 0x7fa4b41038f0 2024-02-28T16:43:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.600+0000 7fa4ab400700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa49806d7f0 0x7fa49806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.600+0000 7fa4ab400700 1 -- 172.21.15.1:0/802879851 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa4a008e1c0 con 0x7fa4b41038f0 2024-02-28T16:43:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.600+0000 7fa4b3400700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa49806d7f0 0x7fa49806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.601+0000 7fa4b3400700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa49806d7f0 0x7fa49806fca0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4007fd0 tx=0x7fa4a4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:05.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.605+0000 7fa4ab400700 1 -- 172.21.15.1:0/802879851 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa4a005b5a0 con 0x7fa4b41038f0 2024-02-28T16:43:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.855+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4b410a030 con 0x7fa49806d7f0 2024-02-28T16:43:05.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.857+0000 7fa4ab400700 1 -- 172.21.15.1:0/802879851 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa4b410a030 con 0x7fa49806d7f0 2024-02-28T16:43:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa49806d7f0 msgr2=0x7fa49806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa49806d7f0 0x7fa49806fca0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4007fd0 tx=0x7fa4a4009040 comp rx=0 tx=0).stop 2024-02-28T16:43:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 msgr2=0x7fa4b4113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 0x7fa4b4113fb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009100 tx=0x7fa4a0009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 shutdown_connections 2024-02-28T16:43:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa49806d7f0 0x7fa49806fca0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa4b4102f50 0x7fa4b4113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 --2- 172.21.15.1:0/802879851 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b41038f0 0x7fa4b4113fb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.859+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 >> 172.21.15.1:0/802879851 conn(0x7fa4b40f8ae0 msgr2=0x7fa4b40fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.860+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 shutdown_connections 2024-02-28T16:43:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:05.860+0000 7fa4ba800700 1 -- 172.21.15.1:0/802879851 wait complete. 2024-02-28T16:43:05.865 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:43:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.015+0000 7fd6c8e00700 1 -- 172.21.15.1:0/116404547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070c40 msgr2=0x7fd6c4109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.015+0000 7fd6c8e00700 1 --2- 172.21.15.1:0/116404547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070c40 0x7fd6c4109260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0004970 tx=0x7fd6b0009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.016+0000 7fd6c8e00700 1 -- 172.21.15.1:0/116404547 shutdown_connections 2024-02-28T16:43:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.016+0000 7fd6c8e00700 1 --2- 172.21.15.1:0/116404547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070c40 0x7fd6c4109260 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.016+0000 7fd6c8e00700 1 --2- 172.21.15.1:0/116404547 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6c4070330 0x7fd6c4070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.016+0000 7fd6c8e00700 1 -- 172.21.15.1:0/116404547 >> 172.21.15.1:0/116404547 conn(0x7fd6c40f8ae0 msgr2=0x7fd6c40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:06.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.016+0000 7fd6c8e00700 1 -- 172.21.15.1:0/116404547 shutdown_connections 2024-02-28T16:43:06.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.016+0000 7fd6c8e00700 1 -- 172.21.15.1:0/116404547 wait complete. 2024-02-28T16:43:06.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.017+0000 7fd6c8e00700 1 Processor -- start 2024-02-28T16:43:06.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.017+0000 7fd6c8e00700 1 -- start start 2024-02-28T16:43:06.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.017+0000 7fd6c8e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070330 0x7fd6c4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.017+0000 7fd6c8e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6c4070c40 0x7fd6c4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.017+0000 7fd6c8e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c4116790 con 0x7fd6c4070330 2024-02-28T16:43:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.017+0000 7fd6c8e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c410fae0 con 0x7fd6c4070c40 2024-02-28T16:43:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.018+0000 7fd6c1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070330 0x7fd6c4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.018+0000 7fd6c1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070330 0x7fd6c4115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60386/0 (socket says 172.21.15.1:60386) 2024-02-28T16:43:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.018+0000 7fd6c1e00700 1 -- 172.21.15.1:0/3666334317 learned_addr learned my addr 172.21.15.1:0/3666334317 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.018+0000 7fd6c1400700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6c4070c40 0x7fd6c4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.018+0000 7fd6c1e00700 1 -- 172.21.15.1:0/3666334317 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6c4070c40 msgr2=0x7fd6c4116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.018+0000 7fd6c1e00700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6c4070c40 0x7fd6c4116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.018+0000 7fd6c1e00700 1 -- 172.21.15.1:0/3666334317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b0005040 con 0x7fd6c4070330 2024-02-28T16:43:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.019+0000 7fd6c1e00700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070330 0x7fd6c4115c00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400c830 tx=0x7fd6b4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.019+0000 7fd6ba000700 1 -- 172.21.15.1:0/3666334317 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6b40106f0 con 0x7fd6c4070330 2024-02-28T16:43:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.019+0000 7fd6ba000700 1 -- 172.21.15.1:0/3666334317 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6b4010d20 con 0x7fd6c4070330 2024-02-28T16:43:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.019+0000 7fd6ba000700 1 -- 172.21.15.1:0/3666334317 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6b40113a0 con 0x7fd6c4070330 2024-02-28T16:43:06.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.019+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c410fdc0 con 0x7fd6c4070330 2024-02-28T16:43:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.019+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c41102d0 con 0x7fd6c4070330 2024-02-28T16:43:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.020+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c4106920 con 0x7fd6c4070330 2024-02-28T16:43:06.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.022+0000 7fd6ba000700 1 -- 172.21.15.1:0/3666334317 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd6b4012400 con 0x7fd6c4070330 2024-02-28T16:43:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.023+0000 7fd6ba000700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6a806d7e0 0x7fd6a806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.023+0000 7fd6ba000700 1 -- 172.21.15.1:0/3666334317 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd6b408cd90 con 0x7fd6c4070330 2024-02-28T16:43:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.023+0000 7fd6c1400700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6a806d7e0 0x7fd6a806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.024+0000 7fd6c1400700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6a806d7e0 0x7fd6a806fc90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0000c00 tx=0x7fd6b0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.028+0000 7fd6ba000700 1 -- 172.21.15.1:0/3666334317 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd6b4056700 con 0x7fd6c4070330 2024-02-28T16:43:06.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.268+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6c404bf70 con 0x7fd6a806d7e0 2024-02-28T16:43:06.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.271+0000 7fd6ba000700 1 -- 172.21.15.1:0/3666334317 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd6b000b700 con 0x7fd6a806d7e0 2024-02-28T16:43:06.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.273+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6a806d7e0 msgr2=0x7fd6a806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.273+0000 7fd6c8e00700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6a806d7e0 0x7fd6a806fc90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0000c00 tx=0x7fd6b0009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.273+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070330 msgr2=0x7fd6c4115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.273+0000 7fd6c8e00700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070330 0x7fd6c4115c00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400c830 tx=0x7fd6b4004470 comp rx=0 tx=0).stop 2024-02-28T16:43:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.274+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 shutdown_connections 2024-02-28T16:43:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.274+0000 7fd6c8e00700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6a806d7e0 0x7fd6a806fc90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.274+0000 7fd6c8e00700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6c4070330 0x7fd6c4115c00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.274+0000 7fd6c8e00700 1 --2- 172.21.15.1:0/3666334317 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6c4070c40 0x7fd6c4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.274+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 >> 172.21.15.1:0/3666334317 conn(0x7fd6c40f8ae0 msgr2=0x7fd6c4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.274+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 shutdown_connections 2024-02-28T16:43:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.274+0000 7fd6c8e00700 1 -- 172.21.15.1:0/3666334317 wait complete. 2024-02-28T16:43:06.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.430+0000 7f9d9fe00700 1 -- 172.21.15.1:0/2684773417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 msgr2=0x7f9da00fef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.430+0000 7f9d9fe00700 1 --2- 172.21.15.1:0/2684773417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 0x7f9da00fef90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c004940 tx=0x7f9d8c009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.430+0000 7f9d9fe00700 1 -- 172.21.15.1:0/2684773417 shutdown_connections 2024-02-28T16:43:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.430+0000 7f9d9fe00700 1 --2- 172.21.15.1:0/2684773417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 0x7f9da00fef90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.430+0000 7f9d9fe00700 1 --2- 172.21.15.1:0/2684773417 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da0104f80 0x7f9da0105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.430+0000 7f9d9fe00700 1 -- 172.21.15.1:0/2684773417 >> 172.21.15.1:0/2684773417 conn(0x7f9da00fab10 msgr2=0x7f9da00fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.430+0000 7f9d9fe00700 1 -- 172.21.15.1:0/2684773417 shutdown_connections 2024-02-28T16:43:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.430+0000 7f9d9fe00700 1 -- 172.21.15.1:0/2684773417 wait complete. 2024-02-28T16:43:06.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.431+0000 7f9d9fe00700 1 Processor -- start 2024-02-28T16:43:06.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.431+0000 7f9d9fe00700 1 -- start start 2024-02-28T16:43:06.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.432+0000 7f9d9fe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da0104f80 0x7f9da00724a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.432+0000 7f9d9fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 0x7f9da00729e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.432+0000 7f9d9fe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da0072f20 con 0x7f9da0105920 2024-02-28T16:43:06.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.432+0000 7f9d9fe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da0073060 con 0x7f9da0104f80 2024-02-28T16:43:06.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.432+0000 7f9d9e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 0x7f9da00729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.432+0000 7f9d9e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 0x7f9da00729e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60394/0 (socket says 172.21.15.1:60394) 2024-02-28T16:43:06.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.432+0000 7f9d9e000700 1 -- 172.21.15.1:0/1467289989 learned_addr learned my addr 172.21.15.1:0/1467289989 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d9ea00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da0104f80 0x7f9da00724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d9e000700 1 -- 172.21.15.1:0/1467289989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da0104f80 msgr2=0x7f9da00724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d9e000700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da0104f80 0x7f9da00724a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d9e000700 1 -- 172.21.15.1:0/1467289989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c005040 con 0x7f9da0105920 2024-02-28T16:43:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d9ea00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da0104f80 0x7f9da00724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:43:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d9e000700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 0x7f9da00729e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009150 tx=0x7f9d8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d96a00700 1 -- 172.21.15.1:0/1467289989 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d8c00e040 con 0x7f9da0105920 2024-02-28T16:43:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d96a00700 1 -- 172.21.15.1:0/1467289989 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d8c00bbf0 con 0x7f9da0105920 2024-02-28T16:43:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da0073280 con 0x7f9da0105920 2024-02-28T16:43:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.433+0000 7f9d96a00700 1 -- 172.21.15.1:0/1467289989 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d8c01c630 con 0x7f9da0105920 2024-02-28T16:43:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.434+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da011e440 con 0x7f9da0105920 2024-02-28T16:43:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.435+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da005f5c0 con 0x7f9da0105920 2024-02-28T16:43:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.436+0000 7f9d96a00700 1 -- 172.21.15.1:0/1467289989 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9d8c00bd60 con 0x7f9da0105920 2024-02-28T16:43:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.437+0000 7f9d96a00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d8406d7e0 0x7f9d8406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.437+0000 7f9d96a00700 1 -- 172.21.15.1:0/1467289989 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9d8c08e330 con 0x7f9da0105920 2024-02-28T16:43:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.437+0000 7f9d9ea00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d8406d7e0 0x7f9d8406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.438+0000 7f9d9ea00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d8406d7e0 0x7f9d8406fc90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9d90006fd0 tx=0x7f9d90008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.442+0000 7f9d96a00700 1 -- 172.21.15.1:0/1467289989 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d8c057b60 con 0x7f9da0105920 2024-02-28T16:43:06.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.677+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9da0073cf0 con 0x7f9d8406d7e0 2024-02-28T16:43:06.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.708+0000 7f9d96a00700 1 -- 172.21.15.1:0/1467289989 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f9da0073cf0 con 0x7f9d8406d7e0 2024-02-28T16:43:06.697 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:43:06.697 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (9m) 77s ago 23m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:43:06.697 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (5m) 77s ago 23m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:43:06.697 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (5m) 2m ago 22m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:43:06.697 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (8m) 77s ago 23m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:43:06.697 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (110s) 77s ago 16m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (84s) 77s ago 16m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (16m) 2m ago 16m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (16m) 2m ago 16m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (11m) 77s ago 25m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (10m) 2m ago 22m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (7m) 77s ago 25m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (6m) 2m ago 22m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (10m) 77s ago 23m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (10m) 2m ago 22m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (5m) 77s ago 21m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (4m) 77s ago 20m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (4m) 77s ago 19m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (3m) 2m ago 19m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (3m) 2m ago 18m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:43:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (2m) 2m ago 18m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:43:06.699 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (9m) 77s ago 22m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:43:06.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.711+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d8406d7e0 msgr2=0x7f9d8406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.711+0000 7f9d9fe00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d8406d7e0 0x7f9d8406fc90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9d90006fd0 tx=0x7f9d90008040 comp rx=0 tx=0).stop 2024-02-28T16:43:06.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 msgr2=0x7f9da00729e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 0x7f9da00729e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009150 tx=0x7f9d8c009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:06.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 shutdown_connections 2024-02-28T16:43:06.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d8406d7e0 0x7f9d8406fc90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9da0104f80 0x7f9da00724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 --2- 172.21.15.1:0/1467289989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9da0105920 0x7f9da00729e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 >> 172.21.15.1:0/1467289989 conn(0x7f9da00fab10 msgr2=0x7f9da00fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:06.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 shutdown_connections 2024-02-28T16:43:06.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.712+0000 7f9d9fe00700 1 -- 172.21.15.1:0/1467289989 wait complete. 2024-02-28T16:43:06.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:06 smithi001 ceph-mon[112940]: pgmap v259: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:06.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.872+0000 7fba28200700 1 -- 172.21.15.1:0/2360711198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 msgr2=0x7fba20105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.872+0000 7fba28200700 1 --2- 172.21.15.1:0/2360711198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 0x7fba20105540 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fba14009ce0 tx=0x7fba14004d10 comp rx=0 tx=0).stop 2024-02-28T16:43:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.872+0000 7fba28200700 1 -- 172.21.15.1:0/2360711198 shutdown_connections 2024-02-28T16:43:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.872+0000 7fba28200700 1 --2- 172.21.15.1:0/2360711198 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba20105b10 0x7fba200ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.872+0000 7fba28200700 1 --2- 172.21.15.1:0/2360711198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 0x7fba20105540 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.872+0000 7fba28200700 1 -- 172.21.15.1:0/2360711198 >> 172.21.15.1:0/2360711198 conn(0x7fba200face0 msgr2=0x7fba200fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.872+0000 7fba28200700 1 -- 172.21.15.1:0/2360711198 shutdown_connections 2024-02-28T16:43:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.872+0000 7fba28200700 1 -- 172.21.15.1:0/2360711198 wait complete. 2024-02-28T16:43:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.873+0000 7fba28200700 1 Processor -- start 2024-02-28T16:43:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.874+0000 7fba28200700 1 -- start start 2024-02-28T16:43:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.874+0000 7fba28200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 0x7fba20115e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.874+0000 7fba28200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba20105b10 0x7fba201163b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.874+0000 7fba28200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba20116a00 con 0x7fba20105170 2024-02-28T16:43:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.874+0000 7fba28200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2010fd50 con 0x7fba20105b10 2024-02-28T16:43:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.874+0000 7fba25800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 0x7fba20115e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.874+0000 7fba25800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 0x7fba20115e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60414/0 (socket says 172.21.15.1:60414) 2024-02-28T16:43:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.874+0000 7fba25800700 1 -- 172.21.15.1:0/1805133837 learned_addr learned my addr 172.21.15.1:0/1805133837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.875+0000 7fba24e00700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba20105b10 0x7fba201163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.875+0000 7fba25800700 1 -- 172.21.15.1:0/1805133837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba20105b10 msgr2=0x7fba201163b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.875+0000 7fba25800700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba20105b10 0x7fba201163b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.875+0000 7fba25800700 1 -- 172.21.15.1:0/1805133837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba14009940 con 0x7fba20105170 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.875+0000 7fba25800700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 0x7fba20115e70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fba14004b70 tx=0x7fba14004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.875+0000 7fba1d600700 1 -- 172.21.15.1:0/1805133837 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba1400e040 con 0x7fba20105170 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.876+0000 7fba1d600700 1 -- 172.21.15.1:0/1805133837 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba14004610 con 0x7fba20105170 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.876+0000 7fba1d600700 1 -- 172.21.15.1:0/1805133837 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba1401c750 con 0x7fba20105170 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.876+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2010ffd0 con 0x7fba20105170 2024-02-28T16:43:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.876+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba20110510 con 0x7fba20105170 2024-02-28T16:43:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.877+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba20102480 con 0x7fba20105170 2024-02-28T16:43:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.878+0000 7fba1d600700 1 -- 172.21.15.1:0/1805133837 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fba14012410 con 0x7fba20105170 2024-02-28T16:43:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.879+0000 7fba1d600700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba0806d7f0 0x7fba0806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.880+0000 7fba1d600700 1 -- 172.21.15.1:0/1805133837 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fba1408ea40 con 0x7fba20105170 2024-02-28T16:43:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.880+0000 7fba24e00700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba0806d7f0 0x7fba0806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.880+0000 7fba24e00700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba0806d7f0 0x7fba0806fca0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba20111750 tx=0x7fba0c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:06.884+0000 7fba1d600700 1 -- 172.21.15.1:0/1805133837 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba14058270 con 0x7fba20105170 2024-02-28T16:43:06.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:06 smithi066 ceph-mon[85781]: pgmap v259: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:07.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.200+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fba2004bf00 con 0x7fba20105170 2024-02-28T16:43:07.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.201+0000 7fba1d600700 1 -- 172.21.15.1:0/1805133837 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fba1405b930 con 0x7fba20105170 2024-02-28T16:43:07.190 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:43:07.190 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:43:07.190 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:43:07.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:43:07.190 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:43:07.190 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:43:07.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:43:07.191 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:43:07.191 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:43:07.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:43:07.191 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:43:07.191 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:43:07.192 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:43:07.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:43:07.192 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:43:07.192 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:43:07.192 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:43:07.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:43:07.193 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:43:07.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba0806d7f0 msgr2=0x7fba0806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:07.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba0806d7f0 0x7fba0806fca0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba20111750 tx=0x7fba0c00b040 comp rx=0 tx=0).stop 2024-02-28T16:43:07.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 msgr2=0x7fba20115e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:07.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 0x7fba20115e70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fba14004b70 tx=0x7fba14004d10 comp rx=0 tx=0).stop 2024-02-28T16:43:07.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 shutdown_connections 2024-02-28T16:43:07.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba0806d7f0 0x7fba0806fca0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba20105170 0x7fba20115e70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 --2- 172.21.15.1:0/1805133837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba20105b10 0x7fba201163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 >> 172.21.15.1:0/1805133837 conn(0x7fba200face0 msgr2=0x7fba200fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:07.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 shutdown_connections 2024-02-28T16:43:07.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.204+0000 7fba28200700 1 -- 172.21.15.1:0/1805133837 wait complete. 2024-02-28T16:43:07.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.366+0000 7fa775200700 1 -- 172.21.15.1:0/495468313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 msgr2=0x7fa7700ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.366+0000 7fa775200700 1 --2- 172.21.15.1:0/495468313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 0x7fa7700ff150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa760004940 tx=0x7fa760009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.366+0000 7fa775200700 1 -- 172.21.15.1:0/495468313 shutdown_connections 2024-02-28T16:43:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.366+0000 7fa775200700 1 --2- 172.21.15.1:0/495468313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7700fb350 0x7fa7700fb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.366+0000 7fa775200700 1 --2- 172.21.15.1:0/495468313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 0x7fa7700ff150 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.366+0000 7fa775200700 1 -- 172.21.15.1:0/495468313 >> 172.21.15.1:0/495468313 conn(0x7fa770071340 msgr2=0x7fa770071740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.366+0000 7fa775200700 1 -- 172.21.15.1:0/495468313 shutdown_connections 2024-02-28T16:43:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.366+0000 7fa775200700 1 -- 172.21.15.1:0/495468313 wait complete. 2024-02-28T16:43:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.367+0000 7fa775200700 1 Processor -- start 2024-02-28T16:43:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.368+0000 7fa775200700 1 -- start start 2024-02-28T16:43:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.368+0000 7fa775200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7700fb350 0x7fa770115a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.368+0000 7fa775200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 0x7fa770115f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.368+0000 7fa775200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa770116590 con 0x7fa7700fed80 2024-02-28T16:43:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.368+0000 7fa775200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77010f8c0 con 0x7fa7700fb350 2024-02-28T16:43:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.368+0000 7fa76ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 0x7fa770115f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.369+0000 7fa76ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 0x7fa770115f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60436/0 (socket says 172.21.15.1:60436) 2024-02-28T16:43:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.369+0000 7fa76ea00700 1 -- 172.21.15.1:0/1081833464 learned_addr learned my addr 172.21.15.1:0/1081833464 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.369+0000 7fa76f400700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7700fb350 0x7fa770115a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.369+0000 7fa76ea00700 1 -- 172.21.15.1:0/1081833464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7700fb350 msgr2=0x7fa770115a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.369+0000 7fa76ea00700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7700fb350 0x7fa770115a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.369+0000 7fa76ea00700 1 -- 172.21.15.1:0/1081833464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa760005040 con 0x7fa7700fed80 2024-02-28T16:43:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.369+0000 7fa76ea00700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 0x7fa770115f40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa75c002a50 tx=0x7fa75c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.370+0000 7fa767400700 1 -- 172.21.15.1:0/1081833464 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75c007cc0 con 0x7fa7700fed80 2024-02-28T16:43:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.370+0000 7fa767400700 1 -- 172.21.15.1:0/1081833464 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa75c00f040 con 0x7fa7700fed80 2024-02-28T16:43:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.370+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77010fba0 con 0x7fa7700fed80 2024-02-28T16:43:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.370+0000 7fa767400700 1 -- 172.21.15.1:0/1081833464 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75c009640 con 0x7fa7700fed80 2024-02-28T16:43:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.370+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7701100e0 con 0x7fa7700fed80 2024-02-28T16:43:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.371+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77005f5c0 con 0x7fa7700fed80 2024-02-28T16:43:07.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.372+0000 7fa767400700 1 -- 172.21.15.1:0/1081833464 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa75c0097a0 con 0x7fa7700fed80 2024-02-28T16:43:07.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.373+0000 7fa767400700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa75406d7e0 0x7fa75406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:07.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.373+0000 7fa767400700 1 -- 172.21.15.1:0/1081833464 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa75c08cbc0 con 0x7fa7700fed80 2024-02-28T16:43:07.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.374+0000 7fa76f400700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa75406d7e0 0x7fa75406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:07.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.374+0000 7fa76f400700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa75406d7e0 0x7fa75406fc90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa760004f90 tx=0x7fa760009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:07.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.378+0000 7fa767400700 1 -- 172.21.15.1:0/1081833464 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa75c059fc0 con 0x7fa7700fed80 2024-02-28T16:43:07.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.646+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa77004bed0 con 0x7fa7700fed80 2024-02-28T16:43:07.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.647+0000 7fa767400700 1 -- 172.21.15.1:0/1081833464 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fa75c059b50 con 0x7fa7700fed80 2024-02-28T16:43:07.636 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:43:07.636 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:43:07.636 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:43:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:43:07.638 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:43:07.639 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:43:07.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa75406d7e0 msgr2=0x7fa75406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:07.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa75406d7e0 0x7fa75406fc90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa760004f90 tx=0x7fa760009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:07.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 msgr2=0x7fa770115f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:07.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 0x7fa770115f40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa75c002a50 tx=0x7fa75c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 shutdown_connections 2024-02-28T16:43:07.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa75406d7e0 0x7fa75406fc90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7700fb350 0x7fa770115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 --2- 172.21.15.1:0/1081833464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7700fed80 0x7fa770115f40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 >> 172.21.15.1:0/1081833464 conn(0x7fa770071340 msgr2=0x7fa770071b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:07.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.650+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 shutdown_connections 2024-02-28T16:43:07.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.651+0000 7fa775200700 1 -- 172.21.15.1:0/1081833464 wait complete. 2024-02-28T16:43:07.641 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:43:07.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.806+0000 7f1688a00700 1 -- 172.21.15.1:0/3980966016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684105890 msgr2=0x7f16840fef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:07.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.806+0000 7f1688a00700 1 --2- 172.21.15.1:0/3980966016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684105890 0x7f16840fef90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1670004940 tx=0x7f1670009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:07.795 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:07 smithi001 ceph-mon[112940]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:07.795 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:07 smithi001 ceph-mon[112940]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:07.795 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:07 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1805133837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:43:07.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.806+0000 7f1688a00700 1 -- 172.21.15.1:0/3980966016 shutdown_connections 2024-02-28T16:43:07.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.806+0000 7f1688a00700 1 --2- 172.21.15.1:0/3980966016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684105890 0x7f16840fef90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.806+0000 7f1688a00700 1 --2- 172.21.15.1:0/3980966016 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684104ef0 0x7f16841052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.806+0000 7f1688a00700 1 -- 172.21.15.1:0/3980966016 >> 172.21.15.1:0/3980966016 conn(0x7f16840fab10 msgr2=0x7f16840fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:07.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.806+0000 7f1688a00700 1 -- 172.21.15.1:0/3980966016 shutdown_connections 2024-02-28T16:43:07.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.806+0000 7f1688a00700 1 -- 172.21.15.1:0/3980966016 wait complete. 2024-02-28T16:43:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.807+0000 7f1688a00700 1 Processor -- start 2024-02-28T16:43:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.807+0000 7f1688a00700 1 -- start start 2024-02-28T16:43:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.808+0000 7f1688a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684104ef0 0x7f1684115a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.808+0000 7f1688a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684105890 0x7f1684115f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.808+0000 7f1688a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16841165c0 con 0x7f1684104ef0 2024-02-28T16:43:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.808+0000 7f1688a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168410f8f0 con 0x7f1684105890 2024-02-28T16:43:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.808+0000 7f1683400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684104ef0 0x7f1684115a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:07.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.808+0000 7f1683400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684104ef0 0x7f1684115a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60446/0 (socket says 172.21.15.1:60446) 2024-02-28T16:43:07.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.808+0000 7f1683400700 1 -- 172.21.15.1:0/2410153593 learned_addr learned my addr 172.21.15.1:0/2410153593 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:07.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.808+0000 7f1682a00700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684105890 0x7f1684115f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:07.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.809+0000 7f1683400700 1 -- 172.21.15.1:0/2410153593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684105890 msgr2=0x7f1684115f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:07.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.809+0000 7f1683400700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684105890 0x7f1684115f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:07.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.809+0000 7f1683400700 1 -- 172.21.15.1:0/2410153593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1670005040 con 0x7f1684104ef0 2024-02-28T16:43:07.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.809+0000 7f1683400700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684104ef0 0x7f1684115a30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f167400c8c0 tx=0x7f16740044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:07.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.809+0000 7f167b400700 1 -- 172.21.15.1:0/2410153593 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f167400a980 con 0x7f1684104ef0 2024-02-28T16:43:07.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.809+0000 7f167b400700 1 -- 172.21.15.1:0/2410153593 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f167400ce60 con 0x7f1684104ef0 2024-02-28T16:43:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.809+0000 7f167b400700 1 -- 172.21.15.1:0/2410153593 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16740115e0 con 0x7f1684104ef0 2024-02-28T16:43:07.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.809+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168410fbd0 con 0x7f1684104ef0 2024-02-28T16:43:07.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.810+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1684110190 con 0x7f1684104ef0 2024-02-28T16:43:07.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.811+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168405f5c0 con 0x7f1684104ef0 2024-02-28T16:43:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.812+0000 7f167b400700 1 -- 172.21.15.1:0/2410153593 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1674012400 con 0x7f1684104ef0 2024-02-28T16:43:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.813+0000 7f167b400700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.813+0000 7f167b400700 1 -- 172.21.15.1:0/2410153593 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f167408cf70 con 0x7f1684104ef0 2024-02-28T16:43:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.814+0000 7f1682a00700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.814+0000 7f1682a00700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1684111260 tx=0x7f1670009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:07.818+0000 7f167b400700 1 -- 172.21.15.1:0/2410153593 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f167405a350 con 0x7f1684104ef0 2024-02-28T16:43:07.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:07 smithi066 ceph-mon[85781]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:07.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:07 smithi066 ceph-mon[85781]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:07.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:07 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1805133837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:43:08.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.066+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1684111010 con 0x7f166806d7f0 2024-02-28T16:43:08.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.068+0000 7f167b400700 1 -- 172.21.15.1:0/2410153593 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1684111010 con 0x7f166806d7f0 2024-02-28T16:43:08.056 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:43:08.056 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:43:08.057 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:43:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 msgr2=0x7f166806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1684111260 tx=0x7f1670009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684104ef0 msgr2=0x7f1684115a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684104ef0 0x7f1684115a30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f167400c8c0 tx=0x7f16740044c0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 shutdown_connections 2024-02-28T16:43:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684104ef0 0x7f1684115a30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 --2- 172.21.15.1:0/2410153593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684105890 0x7f1684115f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 >> 172.21.15.1:0/2410153593 conn(0x7f16840fab10 msgr2=0x7f16840fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 shutdown_connections 2024-02-28T16:43:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.071+0000 7f1688a00700 1 -- 172.21.15.1:0/2410153593 wait complete. 2024-02-28T16:43:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.220+0000 7faa54e00700 1 -- 172.21.15.1:0/1907171583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 msgr2=0x7faa50105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.220+0000 7faa54e00700 1 --2- 172.21.15.1:0/1907171583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 0x7faa50105340 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa40004940 tx=0x7faa40009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.220+0000 7faa54e00700 1 -- 172.21.15.1:0/1907171583 shutdown_connections 2024-02-28T16:43:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.220+0000 7faa54e00700 1 --2- 172.21.15.1:0/1907171583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa50105910 0x7faa500fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.220+0000 7faa54e00700 1 --2- 172.21.15.1:0/1907171583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 0x7faa50105340 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.220+0000 7faa54e00700 1 -- 172.21.15.1:0/1907171583 >> 172.21.15.1:0/1907171583 conn(0x7faa500faae0 msgr2=0x7faa500fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.221+0000 7faa54e00700 1 -- 172.21.15.1:0/1907171583 shutdown_connections 2024-02-28T16:43:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.221+0000 7faa54e00700 1 -- 172.21.15.1:0/1907171583 wait complete. 2024-02-28T16:43:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.221+0000 7faa54e00700 1 Processor -- start 2024-02-28T16:43:08.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.221+0000 7faa54e00700 1 -- start start 2024-02-28T16:43:08.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.222+0000 7faa54e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 0x7faa50115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:08.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.222+0000 7faa54e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa50105910 0x7faa50115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:08.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.222+0000 7faa54e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa501165b0 con 0x7faa50104f70 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.222+0000 7faa54e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5010f8e0 con 0x7faa50105910 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.222+0000 7faa4f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 0x7faa50115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.222+0000 7faa4f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 0x7faa50115a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60466/0 (socket says 172.21.15.1:60466) 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.222+0000 7faa4f400700 1 -- 172.21.15.1:0/401816942 learned_addr learned my addr 172.21.15.1:0/401816942 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa4ea00700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa50105910 0x7faa50115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa4f400700 1 -- 172.21.15.1:0/401816942 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa50105910 msgr2=0x7faa50115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa4f400700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa50105910 0x7faa50115f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa4f400700 1 -- 172.21.15.1:0/401816942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40005040 con 0x7faa50104f70 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa4f400700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 0x7faa50115a20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa40009fd0 tx=0x7faa40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa47400700 1 -- 172.21.15.1:0/401816942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa4000e040 con 0x7faa50104f70 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa47400700 1 -- 172.21.15.1:0/401816942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa4000bbf0 con 0x7faa50104f70 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa47400700 1 -- 172.21.15.1:0/401816942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa4001c3f0 con 0x7faa50104f70 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.223+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5010fb60 con 0x7faa50104f70 2024-02-28T16:43:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.224+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa50110120 con 0x7faa50104f70 2024-02-28T16:43:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.225+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5005f5c0 con 0x7faa50104f70 2024-02-28T16:43:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.226+0000 7faa47400700 1 -- 172.21.15.1:0/401816942 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7faa4000bd60 con 0x7faa50104f70 2024-02-28T16:43:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.227+0000 7faa47400700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa3406d7f0 0x7faa3406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.227+0000 7faa47400700 1 -- 172.21.15.1:0/401816942 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7faa4008e1c0 con 0x7faa50104f70 2024-02-28T16:43:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.227+0000 7faa4ea00700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa3406d7f0 0x7faa3406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.227+0000 7faa4ea00700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa3406d7f0 0x7faa3406fca0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faa50111250 tx=0x7faa3c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:08.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.231+0000 7faa47400700 1 -- 172.21.15.1:0/401816942 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faa4005b5a0 con 0x7faa50104f70 2024-02-28T16:43:08.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.545+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faa50111000 con 0x7faa50104f70 2024-02-28T16:43:08.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.546+0000 7faa47400700 1 -- 172.21.15.1:0/401816942 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7faa4005b130 con 0x7faa50104f70 2024-02-28T16:43:08.534 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:43:08.534 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:43:08.534 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:43:08.534 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:43:08.534 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:43:08.534 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:43:08.534 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.548+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa3406d7f0 msgr2=0x7faa3406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.548+0000 7faa54e00700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa3406d7f0 0x7faa3406fca0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faa50111250 tx=0x7faa3c009ee0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 msgr2=0x7faa50115a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 0x7faa50115a20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa40009fd0 tx=0x7faa40009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 shutdown_connections 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa3406d7f0 0x7faa3406fca0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa50104f70 0x7faa50115a20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 --2- 172.21.15.1:0/401816942 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faa50105910 0x7faa50115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 >> 172.21.15.1:0/401816942 conn(0x7faa500faae0 msgr2=0x7faa500fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 shutdown_connections 2024-02-28T16:43:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:08.549+0000 7faa54e00700 1 -- 172.21.15.1:0/401816942 wait complete. 2024-02-28T16:43:08.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:08 smithi066 ceph-mon[85781]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:08.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:08 smithi066 ceph-mon[85781]: pgmap v260: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:08.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:08 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1081833464' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:43:09.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:08 smithi001 ceph-mon[112940]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:09.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:08 smithi001 ceph-mon[112940]: pgmap v260: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:09.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:08 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1081833464' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:43:09.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:09 smithi001 ceph-mon[112940]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:09.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:09 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/401816942' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:43:09.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:09 smithi066 ceph-mon[85781]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:09.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:09 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/401816942' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:43:10.898 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:10 smithi066 ceph-mon[85781]: pgmap v261: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:11.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:10 smithi001 ceph-mon[112940]: pgmap v261: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:11.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:43:12.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:43:12.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:12 smithi066 ceph-mon[85781]: pgmap v262: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:13.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:12 smithi001 ceph-mon[112940]: pgmap v262: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:14.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:14 smithi066 ceph-mon[85781]: pgmap v263: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:15.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:14 smithi001 ceph-mon[112940]: pgmap v263: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:16.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:16 smithi066 ceph-mon[85781]: pgmap v264: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:17.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:16 smithi001 ceph-mon[112940]: pgmap v264: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:17.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:17 smithi066 ceph-mon[85781]: pgmap v265: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:18.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:17 smithi001 ceph-mon[112940]: pgmap v265: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:20.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:19 smithi066 ceph-mon[85781]: pgmap v266: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:20.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:19 smithi001 ceph-mon[112940]: pgmap v266: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:22.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:21 smithi066 ceph-mon[85781]: pgmap v267: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:22.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:21 smithi001 ceph-mon[112940]: pgmap v267: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:24.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:23 smithi066 ceph-mon[85781]: pgmap v268: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:24.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:23 smithi001 ceph-mon[112940]: pgmap v268: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:26.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:25 smithi066 ceph-mon[85781]: pgmap v269: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:26.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:43:26.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:25 smithi001 ceph-mon[112940]: pgmap v269: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:26.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:43:28.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:27 smithi066 ceph-mon[85781]: pgmap v270: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:28.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:27 smithi001 ceph-mon[112940]: pgmap v270: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:30.249 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:29 smithi066 ceph-mon[85781]: pgmap v271: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:30.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:29 smithi001 ceph-mon[112940]: pgmap v271: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:32.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:31 smithi066 ceph-mon[85781]: pgmap v272: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:32.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:31 smithi001 ceph-mon[112940]: pgmap v272: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:34.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:33 smithi066 ceph-mon[85781]: pgmap v273: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:34.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:33 smithi001 ceph-mon[112940]: pgmap v273: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:36.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:35 smithi066 ceph-mon[85781]: pgmap v274: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:36.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:35 smithi001 ceph-mon[112940]: pgmap v274: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:38.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:37 smithi066 ceph-mon[85781]: pgmap v275: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:38.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:37 smithi001 ceph-mon[112940]: pgmap v275: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:38.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.722+0000 7fac20600700 1 -- 172.21.15.1:0/3923724110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 msgr2=0x7fac181099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:38.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.722+0000 7fac20600700 1 --2- 172.21.15.1:0/3923724110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 0x7fac181099c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fac0c004940 tx=0x7fac0c009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:38.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.722+0000 7fac20600700 1 -- 172.21.15.1:0/3923724110 shutdown_connections 2024-02-28T16:43:38.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.722+0000 7fac20600700 1 --2- 172.21.15.1:0/3923724110 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac18103dc0 0x7fac18104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:38.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.722+0000 7fac20600700 1 --2- 172.21.15.1:0/3923724110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 0x7fac181099c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:38.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.722+0000 7fac20600700 1 -- 172.21.15.1:0/3923724110 >> 172.21.15.1:0/3923724110 conn(0x7fac18072d50 msgr2=0x7fac18075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:38.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.723+0000 7fac20600700 1 -- 172.21.15.1:0/3923724110 shutdown_connections 2024-02-28T16:43:38.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.723+0000 7fac20600700 1 -- 172.21.15.1:0/3923724110 wait complete. 2024-02-28T16:43:38.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.724+0000 7fac20600700 1 Processor -- start 2024-02-28T16:43:38.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.724+0000 7fac20600700 1 -- start start 2024-02-28T16:43:38.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.724+0000 7fac20600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac18103dc0 0x7fac1811a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:38.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.724+0000 7fac20600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 0x7fac1811a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.724+0000 7fac20600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1811ab90 con 0x7fac181095f0 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.724+0000 7fac20600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac18113ea0 con 0x7fac18103dc0 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.725+0000 7fac1d200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 0x7fac1811a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.725+0000 7fac1d200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 0x7fac1811a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55742/0 (socket says 172.21.15.1:55742) 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.725+0000 7fac1d200700 1 -- 172.21.15.1:0/159244572 learned_addr learned my addr 172.21.15.1:0/159244572 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.725+0000 7fac1dc00700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac18103dc0 0x7fac1811a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.725+0000 7fac1d200700 1 -- 172.21.15.1:0/159244572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac18103dc0 msgr2=0x7fac1811a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.725+0000 7fac1d200700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac18103dc0 0x7fac1811a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.725+0000 7fac1d200700 1 -- 172.21.15.1:0/159244572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c005040 con 0x7fac181095f0 2024-02-28T16:43:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.726+0000 7fac1d200700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 0x7fac1811a540 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fac0400eb70 tx=0x7fac0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.726+0000 7fac15600700 1 -- 172.21.15.1:0/159244572 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac04007cc0 con 0x7fac181095f0 2024-02-28T16:43:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.726+0000 7fac15600700 1 -- 172.21.15.1:0/159244572 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac0400f040 con 0x7fac181095f0 2024-02-28T16:43:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.726+0000 7fac15600700 1 -- 172.21.15.1:0/159244572 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac04009640 con 0x7fac181095f0 2024-02-28T16:43:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.726+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac18114120 con 0x7fac181095f0 2024-02-28T16:43:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.726+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac18114660 con 0x7fac181095f0 2024-02-28T16:43:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.727+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac18075870 con 0x7fac181095f0 2024-02-28T16:43:38.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.728+0000 7fac15600700 1 -- 172.21.15.1:0/159244572 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fac040097a0 con 0x7fac181095f0 2024-02-28T16:43:38.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.730+0000 7fac15600700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fac0006d7f0 0x7fac0006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:38.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.731+0000 7fac15600700 1 -- 172.21.15.1:0/159244572 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fac0408ca30 con 0x7fac181095f0 2024-02-28T16:43:38.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.731+0000 7fac1dc00700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fac0006d7f0 0x7fac0006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:38.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.732+0000 7fac1dc00700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fac0006d7f0 0x7fac0006fca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00b0d0 tx=0x7fac0c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:38.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.735+0000 7fac15600700 1 -- 172.21.15.1:0/159244572 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac04059e30 con 0x7fac181095f0 2024-02-28T16:43:38.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.985+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac181156f0 con 0x7fac0006d7f0 2024-02-28T16:43:38.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.987+0000 7fac15600700 1 -- 172.21.15.1:0/159244572 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fac181156f0 con 0x7fac0006d7f0 2024-02-28T16:43:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.989+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fac0006d7f0 msgr2=0x7fac0006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.989+0000 7fac20600700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fac0006d7f0 0x7fac0006fca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00b0d0 tx=0x7fac0c009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.989+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 msgr2=0x7fac1811a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.989+0000 7fac20600700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 0x7fac1811a540 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fac0400eb70 tx=0x7fac0400b040 comp rx=0 tx=0).stop 2024-02-28T16:43:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.990+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 shutdown_connections 2024-02-28T16:43:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.990+0000 7fac20600700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fac0006d7f0 0x7fac0006fca0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.990+0000 7fac20600700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac18103dc0 0x7fac1811a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.990+0000 7fac20600700 1 --2- 172.21.15.1:0/159244572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac181095f0 0x7fac1811a540 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.990+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 >> 172.21.15.1:0/159244572 conn(0x7fac18072d50 msgr2=0x7fac180746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.990+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 shutdown_connections 2024-02-28T16:43:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:38.990+0000 7fac20600700 1 -- 172.21.15.1:0/159244572 wait complete. 2024-02-28T16:43:38.994 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:43:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.152+0000 7fc17ac00700 1 -- 172.21.15.1:0/1750250911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 msgr2=0x7fc174105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.152+0000 7fc17ac00700 1 --2- 172.21.15.1:0/1750250911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 0x7fc174105540 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc164004940 tx=0x7fc164009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.153+0000 7fc17ac00700 1 -- 172.21.15.1:0/1750250911 shutdown_connections 2024-02-28T16:43:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.153+0000 7fc17ac00700 1 --2- 172.21.15.1:0/1750250911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc174105b10 0x7fc1740ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.153+0000 7fc17ac00700 1 --2- 172.21.15.1:0/1750250911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 0x7fc174105540 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.153+0000 7fc17ac00700 1 -- 172.21.15.1:0/1750250911 >> 172.21.15.1:0/1750250911 conn(0x7fc1740face0 msgr2=0x7fc1740fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.153+0000 7fc17ac00700 1 -- 172.21.15.1:0/1750250911 shutdown_connections 2024-02-28T16:43:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.153+0000 7fc17ac00700 1 -- 172.21.15.1:0/1750250911 wait complete. 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.154+0000 7fc17ac00700 1 Processor -- start 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.154+0000 7fc17ac00700 1 -- start start 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.154+0000 7fc17ac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 0x7fc174115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.154+0000 7fc17ac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc174105b10 0x7fc1741161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.154+0000 7fc17ac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1741168a0 con 0x7fc174105170 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.154+0000 7fc17ac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17410fb40 con 0x7fc174105b10 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.154+0000 7fc173e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 0x7fc174115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.155+0000 7fc173e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 0x7fc174115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55758/0 (socket says 172.21.15.1:55758) 2024-02-28T16:43:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.155+0000 7fc173e00700 1 -- 172.21.15.1:0/4235379358 learned_addr learned my addr 172.21.15.1:0/4235379358 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.155+0000 7fc173400700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc174105b10 0x7fc1741161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.155+0000 7fc173e00700 1 -- 172.21.15.1:0/4235379358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc174105b10 msgr2=0x7fc1741161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.155+0000 7fc173e00700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc174105b10 0x7fc1741161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.155+0000 7fc173e00700 1 -- 172.21.15.1:0/4235379358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164005040 con 0x7fc174105170 2024-02-28T16:43:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.155+0000 7fc173e00700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 0x7fc174115c80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc164009fd0 tx=0x7fc164009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.156+0000 7fc16be00700 1 -- 172.21.15.1:0/4235379358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16400e040 con 0x7fc174105170 2024-02-28T16:43:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.156+0000 7fc16be00700 1 -- 172.21.15.1:0/4235379358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc16400bbf0 con 0x7fc174105170 2024-02-28T16:43:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.156+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17410fdc0 con 0x7fc174105170 2024-02-28T16:43:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.156+0000 7fc16be00700 1 -- 172.21.15.1:0/4235379358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16401c3f0 con 0x7fc174105170 2024-02-28T16:43:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.156+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc174110300 con 0x7fc174105170 2024-02-28T16:43:39.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.157+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1740fe900 con 0x7fc174105170 2024-02-28T16:43:39.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.158+0000 7fc16be00700 1 -- 172.21.15.1:0/4235379358 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc16400bd60 con 0x7fc174105170 2024-02-28T16:43:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.159+0000 7fc16be00700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15806d7f0 0x7fc15806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.159+0000 7fc16be00700 1 -- 172.21.15.1:0/4235379358 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc16408e210 con 0x7fc174105170 2024-02-28T16:43:39.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.160+0000 7fc173400700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15806d7f0 0x7fc15806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:39.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.160+0000 7fc173400700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15806d7f0 0x7fc15806fca0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc160006fd0 tx=0x7fc160008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:39.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.164+0000 7fc16be00700 1 -- 172.21.15.1:0/4235379358 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc16405b610 con 0x7fc174105170 2024-02-28T16:43:39.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.410+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc174111300 con 0x7fc15806d7f0 2024-02-28T16:43:39.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.413+0000 7fc16be00700 1 -- 172.21.15.1:0/4235379358 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc174111300 con 0x7fc15806d7f0 2024-02-28T16:43:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15806d7f0 msgr2=0x7fc15806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15806d7f0 0x7fc15806fca0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc160006fd0 tx=0x7fc160008040 comp rx=0 tx=0).stop 2024-02-28T16:43:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 msgr2=0x7fc174115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 0x7fc174115c80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc164009fd0 tx=0x7fc164009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:39.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 shutdown_connections 2024-02-28T16:43:39.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15806d7f0 0x7fc15806fca0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc174105170 0x7fc174115c80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 --2- 172.21.15.1:0/4235379358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc174105b10 0x7fc1741161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 >> 172.21.15.1:0/4235379358 conn(0x7fc1740face0 msgr2=0x7fc1740fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:39.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 shutdown_connections 2024-02-28T16:43:39.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.416+0000 7fc17ac00700 1 -- 172.21.15.1:0/4235379358 wait complete. 2024-02-28T16:43:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.577+0000 7f6159000700 1 -- 172.21.15.1:0/2164774079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fdd80 msgr2=0x7f61540fe150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.577+0000 7f6159000700 1 --2- 172.21.15.1:0/2164774079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fdd80 0x7f61540fe150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6144004940 tx=0x7f6144009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.577+0000 7f6159000700 1 -- 172.21.15.1:0/2164774079 shutdown_connections 2024-02-28T16:43:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.577+0000 7f6159000700 1 --2- 172.21.15.1:0/2164774079 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61540fe720 0x7f6154106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.577+0000 7f6159000700 1 --2- 172.21.15.1:0/2164774079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fdd80 0x7f61540fe150 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.577+0000 7f6159000700 1 -- 172.21.15.1:0/2164774079 >> 172.21.15.1:0/2164774079 conn(0x7f61540f7800 msgr2=0x7f61540f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.577+0000 7f6159000700 1 -- 172.21.15.1:0/2164774079 shutdown_connections 2024-02-28T16:43:39.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.577+0000 7f6159000700 1 -- 172.21.15.1:0/2164774079 wait complete. 2024-02-28T16:43:39.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.578+0000 7f6159000700 1 Processor -- start 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.579+0000 7f6159000700 1 -- start start 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.579+0000 7f6159000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61540fdd80 0x7f61540fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.579+0000 7f6159000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fe720 0x7f61540fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.579+0000 7f6159000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540fc780 con 0x7f61540fe720 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.579+0000 7f6159000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540fc8f0 con 0x7f61540fdd80 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.579+0000 7f614aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fe720 0x7f61540fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.580+0000 7f614aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fe720 0x7f61540fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55764/0 (socket says 172.21.15.1:55764) 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.580+0000 7f614aa00700 1 -- 172.21.15.1:0/2476078836 learned_addr learned my addr 172.21.15.1:0/2476078836 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.580+0000 7f6153400700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61540fdd80 0x7f61540fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.580+0000 7f614aa00700 1 -- 172.21.15.1:0/2476078836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61540fdd80 msgr2=0x7f61540fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.580+0000 7f614aa00700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61540fdd80 0x7f61540fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.580+0000 7f614aa00700 1 -- 172.21.15.1:0/2476078836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144005040 con 0x7f61540fe720 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.580+0000 7f6153400700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61540fdd80 0x7f61540fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.580+0000 7f614aa00700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fe720 0x7f61540fc180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f613c001d80 tx=0x7f613c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.581+0000 7f614be00700 1 -- 172.21.15.1:0/2476078836 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613c007d10 con 0x7f61540fe720 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.581+0000 7f614be00700 1 -- 172.21.15.1:0/2476078836 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f613c007e70 con 0x7f61540fe720 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.581+0000 7f614be00700 1 -- 172.21.15.1:0/2476078836 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613c014640 con 0x7f61540fe720 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.581+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6154119ac0 con 0x7f61540fe720 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.581+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6154119fa0 con 0x7f61540fe720 2024-02-28T16:43:39.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.582+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61541044b0 con 0x7f61540fe720 2024-02-28T16:43:39.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.583+0000 7f614be00700 1 -- 172.21.15.1:0/2476078836 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f613c00e960 con 0x7f61540fe720 2024-02-28T16:43:39.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.584+0000 7f614be00700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f613806d7f0 0x7f613806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.584+0000 7f614be00700 1 -- 172.21.15.1:0/2476078836 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f613c08cc20 con 0x7f61540fe720 2024-02-28T16:43:39.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.585+0000 7f6153400700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f613806d7f0 0x7f613806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:39.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.585+0000 7f6153400700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f613806d7f0 0x7f613806fca0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f614400b3b0 tx=0x7f6144009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:39.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.589+0000 7f614be00700 1 -- 172.21.15.1:0/2476078836 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f613c0565a0 con 0x7f61540fe720 2024-02-28T16:43:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.814+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f61540fd380 con 0x7f613806d7f0 2024-02-28T16:43:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.833+0000 7f614be00700 1 -- 172.21.15.1:0/2476078836 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f61540fd380 con 0x7f613806d7f0 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (9m) 110s ago 24m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (6m) 110s ago 24m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (6m) 2m ago 22m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (9m) 110s ago 23m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (2m) 110s ago 16m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (117s) 110s ago 16m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (16m) 2m ago 16m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (16m) 2m ago 16m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (12m) 110s ago 25m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (11m) 2m ago 22m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (8m) 110s ago 25m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (7m) 2m ago 22m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (10m) 110s ago 24m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:43:39.821 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (10m) 2m ago 22m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:43:39.822 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (5m) 110s ago 21m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:43:39.822 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (5m) 110s ago 21m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:43:39.822 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (4m) 110s ago 20m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:43:39.822 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (4m) 2m ago 19m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:43:39.822 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (3m) 2m ago 19m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:43:39.822 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (2m) 2m ago 18m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:43:39.822 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (10m) 110s ago 23m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:43:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f613806d7f0 msgr2=0x7f613806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f613806d7f0 0x7f613806fca0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f614400b3b0 tx=0x7f6144009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fe720 msgr2=0x7f61540fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fe720 0x7f61540fc180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f613c001d80 tx=0x7f613c009ec0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 shutdown_connections 2024-02-28T16:43:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f613806d7f0 0x7f613806fca0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61540fdd80 0x7f61540fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 --2- 172.21.15.1:0/2476078836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61540fe720 0x7f61540fc180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 >> 172.21.15.1:0/2476078836 conn(0x7f61540f7800 msgr2=0x7f6154101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 shutdown_connections 2024-02-28T16:43:39.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.836+0000 7f6159000700 1 -- 172.21.15.1:0/2476078836 wait complete. 2024-02-28T16:43:39.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.980+0000 7fb873e00700 1 -- 172.21.15.1:0/1954732782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 msgr2=0x7fb874072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.980+0000 7fb873e00700 1 --2- 172.21.15.1:0/1954732782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 0x7fb874072890 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb864004940 tx=0x7fb864009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:39.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.980+0000 7fb873e00700 1 -- 172.21.15.1:0/1954732782 shutdown_connections 2024-02-28T16:43:39.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.980+0000 7fb873e00700 1 --2- 172.21.15.1:0/1954732782 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb874072dd0 0x7fb87410d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.980+0000 7fb873e00700 1 --2- 172.21.15.1:0/1954732782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 0x7fb874072890 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.980+0000 7fb873e00700 1 -- 172.21.15.1:0/1954732782 >> 172.21.15.1:0/1954732782 conn(0x7fb8740faae0 msgr2=0x7fb8740fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:39.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.980+0000 7fb873e00700 1 -- 172.21.15.1:0/1954732782 shutdown_connections 2024-02-28T16:43:39.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.980+0000 7fb873e00700 1 -- 172.21.15.1:0/1954732782 wait complete. 2024-02-28T16:43:39.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.981+0000 7fb873e00700 1 Processor -- start 2024-02-28T16:43:39.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.981+0000 7fb873e00700 1 -- start start 2024-02-28T16:43:39.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.981+0000 7fb873e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 0x7fb874119e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.981+0000 7fb873e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb874072dd0 0x7fb87411a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.981+0000 7fb873e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87411a9d0 con 0x7fb8740724c0 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.981+0000 7fb873e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb874113ce0 con 0x7fb874072dd0 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.982+0000 7fb872a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 0x7fb874119e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.982+0000 7fb872a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 0x7fb874119e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55778/0 (socket says 172.21.15.1:55778) 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.982+0000 7fb872a00700 1 -- 172.21.15.1:0/3621522230 learned_addr learned my addr 172.21.15.1:0/3621522230 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.982+0000 7fb872a00700 1 -- 172.21.15.1:0/3621522230 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb874072dd0 msgr2=0x7fb87411a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.982+0000 7fb872a00700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb874072dd0 0x7fb87411a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.982+0000 7fb872a00700 1 -- 172.21.15.1:0/3621522230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb864005040 con 0x7fb8740724c0 2024-02-28T16:43:39.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.982+0000 7fb872a00700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 0x7fb874119e40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb864009100 tx=0x7fb864009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.983+0000 7fb86aa00700 1 -- 172.21.15.1:0/3621522230 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86400e040 con 0x7fb8740724c0 2024-02-28T16:43:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.983+0000 7fb86aa00700 1 -- 172.21.15.1:0/3621522230 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb864004290 con 0x7fb8740724c0 2024-02-28T16:43:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.983+0000 7fb86aa00700 1 -- 172.21.15.1:0/3621522230 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86401c630 con 0x7fb8740724c0 2024-02-28T16:43:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.983+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb874113f60 con 0x7fb8740724c0 2024-02-28T16:43:39.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.983+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb874114520 con 0x7fb8740724c0 2024-02-28T16:43:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.984+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87410ae70 con 0x7fb8740724c0 2024-02-28T16:43:39.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.985+0000 7fb86aa00700 1 -- 172.21.15.1:0/3621522230 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb86400bbf0 con 0x7fb8740724c0 2024-02-28T16:43:39.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.986+0000 7fb86aa00700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb85806d7e0 0x7fb85806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:39.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.986+0000 7fb86aa00700 1 -- 172.21.15.1:0/3621522230 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb86408e1e0 con 0x7fb8740724c0 2024-02-28T16:43:39.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.986+0000 7fb872000700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb85806d7e0 0x7fb85806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:39.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.987+0000 7fb872000700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb85806d7e0 0x7fb85806fc90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb8741156e0 tx=0x7fb860007c30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:39.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:39.990+0000 7fb86aa00700 1 -- 172.21.15.1:0/3621522230 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb86405b5c0 con 0x7fb8740724c0 2024-02-28T16:43:40.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:39 smithi066 ceph-mon[85781]: pgmap v276: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:40.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:39 smithi066 ceph-mon[85781]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:40.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:39 smithi066 ceph-mon[85781]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:40.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:39 smithi001 ceph-mon[112940]: pgmap v276: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:40.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:39 smithi001 ceph-mon[112940]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:40.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:39 smithi001 ceph-mon[112940]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.290+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb87404bed0 con 0x7fb8740724c0 2024-02-28T16:43:40.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.291+0000 7fb86aa00700 1 -- 172.21.15.1:0/3621522230 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fb86405b150 con 0x7fb8740724c0 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:43:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:43:40.280 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:43:40.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb85806d7e0 msgr2=0x7fb85806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb85806d7e0 0x7fb85806fc90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb8741156e0 tx=0x7fb860007c30 comp rx=0 tx=0).stop 2024-02-28T16:43:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 msgr2=0x7fb874119e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 0x7fb874119e40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb864009100 tx=0x7fb864009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 shutdown_connections 2024-02-28T16:43:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb85806d7e0 0x7fb85806fc90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8740724c0 0x7fb874119e40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 --2- 172.21.15.1:0/3621522230 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb874072dd0 0x7fb87411a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 >> 172.21.15.1:0/3621522230 conn(0x7fb8740faae0 msgr2=0x7fb8740fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 shutdown_connections 2024-02-28T16:43:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.296+0000 7fb873e00700 1 -- 172.21.15.1:0/3621522230 wait complete. 2024-02-28T16:43:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.444+0000 7fa7f0600700 1 -- 172.21.15.1:0/3631816989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 msgr2=0x7fa7e8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.444+0000 7fa7f0600700 1 --2- 172.21.15.1:0/3631816989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 0x7fa7e8104210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4004940 tx=0x7fa7d4009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:40.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.444+0000 7fa7f0600700 1 -- 172.21.15.1:0/3631816989 shutdown_connections 2024-02-28T16:43:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.444+0000 7fa7f0600700 1 --2- 172.21.15.1:0/3631816989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 0x7fa7e8104210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.444+0000 7fa7f0600700 1 --2- 172.21.15.1:0/3631816989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7e81095f0 0x7fa7e81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.444+0000 7fa7f0600700 1 -- 172.21.15.1:0/3631816989 >> 172.21.15.1:0/3631816989 conn(0x7fa7e8072d50 msgr2=0x7fa7e8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.444+0000 7fa7f0600700 1 -- 172.21.15.1:0/3631816989 shutdown_connections 2024-02-28T16:43:40.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.444+0000 7fa7f0600700 1 -- 172.21.15.1:0/3631816989 wait complete. 2024-02-28T16:43:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.445+0000 7fa7f0600700 1 Processor -- start 2024-02-28T16:43:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.445+0000 7fa7f0600700 1 -- start start 2024-02-28T16:43:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.446+0000 7fa7f0600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 0x7fa7e811a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.446+0000 7fa7f0600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7e81095f0 0x7fa7e811a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.446+0000 7fa7f0600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e811ac50 con 0x7fa7e8103dc0 2024-02-28T16:43:40.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.446+0000 7fa7f0600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e8113ed0 con 0x7fa7e81095f0 2024-02-28T16:43:40.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.446+0000 7fa7edc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 0x7fa7e811a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:40.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.446+0000 7fa7edc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 0x7fa7e811a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55790/0 (socket says 172.21.15.1:55790) 2024-02-28T16:43:40.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.446+0000 7fa7edc00700 1 -- 172.21.15.1:0/2371507346 learned_addr learned my addr 172.21.15.1:0/2371507346 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:40.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.446+0000 7fa7edc00700 1 -- 172.21.15.1:0/2371507346 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7e81095f0 msgr2=0x7fa7e811a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:40.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.447+0000 7fa7edc00700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7e81095f0 0x7fa7e811a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.447+0000 7fa7edc00700 1 -- 172.21.15.1:0/2371507346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d4005040 con 0x7fa7e8103dc0 2024-02-28T16:43:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.447+0000 7fa7edc00700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 0x7fa7e811a030 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00b790 tx=0x7fa7dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.447+0000 7fa7e5600700 1 -- 172.21.15.1:0/2371507346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7dc00f770 con 0x7fa7e8103dc0 2024-02-28T16:43:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.447+0000 7fa7e5600700 1 -- 172.21.15.1:0/2371507346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7dc00fda0 con 0x7fa7e8103dc0 2024-02-28T16:43:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.447+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e8114150 con 0x7fa7e8103dc0 2024-02-28T16:43:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.447+0000 7fa7e5600700 1 -- 172.21.15.1:0/2371507346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7dc00d310 con 0x7fa7e8103dc0 2024-02-28T16:43:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.447+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e8114690 con 0x7fa7e8103dc0 2024-02-28T16:43:40.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.448+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e8075870 con 0x7fa7e8103dc0 2024-02-28T16:43:40.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.449+0000 7fa7e5600700 1 -- 172.21.15.1:0/2371507346 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa7dc010420 con 0x7fa7e8103dc0 2024-02-28T16:43:40.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.450+0000 7fa7e5600700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7d006d7f0 0x7fa7d006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:40.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.450+0000 7fa7e5600700 1 -- 172.21.15.1:0/2371507346 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa7dc08cef0 con 0x7fa7e8103dc0 2024-02-28T16:43:40.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.450+0000 7fa7ed200700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7d006d7f0 0x7fa7d006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:40.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.451+0000 7fa7ed200700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7d006d7f0 0x7fa7d006fca0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4004f90 tx=0x7fa7d4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.455+0000 7fa7e5600700 1 -- 172.21.15.1:0/2371507346 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa7dc05a2d0 con 0x7fa7e8103dc0 2024-02-28T16:43:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.717+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa7e81156b0 con 0x7fa7e8103dc0 2024-02-28T16:43:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.717+0000 7fa7e5600700 1 -- 172.21.15.1:0/2371507346 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fa7dc059e60 con 0x7fa7e8103dc0 2024-02-28T16:43:40.705 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:43:40.705 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:43:40.705 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:43:40.705 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:43:40.705 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:43:40.705 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:43:40.706 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:43:40.706 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:43:40.706 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:43:40.706 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:43:40.706 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:43:40.706 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:43:40.706 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:43:40.707 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:43:40.708 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:43:40.709 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:43:40.709 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:43:40.709 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7d006d7f0 msgr2=0x7fa7d006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7d006d7f0 0x7fa7d006fca0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4004f90 tx=0x7fa7d4009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 msgr2=0x7fa7e811a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 0x7fa7e811a030 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00b790 tx=0x7fa7dc009250 comp rx=0 tx=0).stop 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 shutdown_connections 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7d006d7f0 0x7fa7d006fca0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7e8103dc0 0x7fa7e811a030 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 --2- 172.21.15.1:0/2371507346 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa7e81095f0 0x7fa7e811a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.720+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 >> 172.21.15.1:0/2371507346 conn(0x7fa7e8072d50 msgr2=0x7fa7e80746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.721+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 shutdown_connections 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.721+0000 7fa7f0600700 1 -- 172.21.15.1:0/2371507346 wait complete. 2024-02-28T16:43:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:43:40.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.870+0000 7fbb7f800700 1 -- 172.21.15.1:0/3635305898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 msgr2=0x7fbb78103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:40.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.870+0000 7fbb7f800700 1 --2- 172.21.15.1:0/3635305898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 0x7fbb78103320 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c004940 tx=0x7fbb6c009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:40.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.870+0000 7fbb7f800700 1 -- 172.21.15.1:0/3635305898 shutdown_connections 2024-02-28T16:43:40.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.870+0000 7fbb7f800700 1 --2- 172.21.15.1:0/3635305898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb781038f0 0x7fbb780fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.870+0000 7fbb7f800700 1 --2- 172.21.15.1:0/3635305898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 0x7fbb78103320 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.870+0000 7fbb7f800700 1 -- 172.21.15.1:0/3635305898 >> 172.21.15.1:0/3635305898 conn(0x7fbb780f8ae0 msgr2=0x7fbb780faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:40.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.870+0000 7fbb7f800700 1 -- 172.21.15.1:0/3635305898 shutdown_connections 2024-02-28T16:43:40.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.870+0000 7fbb7f800700 1 -- 172.21.15.1:0/3635305898 wait complete. 2024-02-28T16:43:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7f800700 1 Processor -- start 2024-02-28T16:43:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7f800700 1 -- start start 2024-02-28T16:43:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7f800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 0x7fbb78113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7f800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb781038f0 0x7fbb78114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7f800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78114700 con 0x7fbb78102f50 2024-02-28T16:43:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7f800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7810d9a0 con 0x7fbb781038f0 2024-02-28T16:43:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 0x7fbb78113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 0x7fbb78113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55812/0 (socket says 172.21.15.1:55812) 2024-02-28T16:43:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.871+0000 7fbb7ce00700 1 -- 172.21.15.1:0/576665363 learned_addr learned my addr 172.21.15.1:0/576665363 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.872+0000 7fbb7ce00700 1 -- 172.21.15.1:0/576665363 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb781038f0 msgr2=0x7fbb78114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.872+0000 7fbb7ce00700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb781038f0 0x7fbb78114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.872+0000 7fbb7ce00700 1 -- 172.21.15.1:0/576665363 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c005040 con 0x7fbb78102f50 2024-02-28T16:43:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.872+0000 7fbb7ce00700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 0x7fbb78113ae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009fd0 tx=0x7fbb6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:40.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.872+0000 7fbb74c00700 1 -- 172.21.15.1:0/576665363 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb6c00e040 con 0x7fbb78102f50 2024-02-28T16:43:40.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.872+0000 7fbb74c00700 1 -- 172.21.15.1:0/576665363 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb6c00bbf0 con 0x7fbb78102f50 2024-02-28T16:43:40.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.872+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7810dc20 con 0x7fbb78102f50 2024-02-28T16:43:40.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.872+0000 7fbb74c00700 1 -- 172.21.15.1:0/576665363 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb6c01c3f0 con 0x7fbb78102f50 2024-02-28T16:43:40.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.873+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7810e160 con 0x7fbb78102f50 2024-02-28T16:43:40.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.874+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb780fc6e0 con 0x7fbb78102f50 2024-02-28T16:43:40.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.875+0000 7fbb74c00700 1 -- 172.21.15.1:0/576665363 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbb6c00bd60 con 0x7fbb78102f50 2024-02-28T16:43:40.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.875+0000 7fbb74c00700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb6006d7f0 0x7fbb6006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.875+0000 7fbb74c00700 1 -- 172.21.15.1:0/576665363 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbb6c08e2b0 con 0x7fbb78102f50 2024-02-28T16:43:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.876+0000 7fbb77e00700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb6006d7f0 0x7fbb6006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.876+0000 7fbb77e00700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb6006d7f0 0x7fbb6006fca0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbb64006fd0 tx=0x7fbb64006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:40.881+0000 7fbb74c00700 1 -- 172.21.15.1:0/576665363 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbb6c05b610 con 0x7fbb78102f50 2024-02-28T16:43:41.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:40 smithi001 ceph-mon[112940]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:41.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3621522230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:43:41.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2371507346' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:43:41.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:43:41.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.119+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb7804bf40 con 0x7fbb6006d7f0 2024-02-28T16:43:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.121+0000 7fbb74c00700 1 -- 172.21.15.1:0/576665363 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbb7804bf40 con 0x7fbb6006d7f0 2024-02-28T16:43:41.109 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:43:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:43:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:43:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:43:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:43:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:43:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:43:41.109 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:43:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:43:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:43:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:43:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:43:41.110 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:43:41.110 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:43:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb6006d7f0 msgr2=0x7fbb6006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb6006d7f0 0x7fbb6006fca0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbb64006fd0 tx=0x7fbb64006db0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 msgr2=0x7fbb78113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 0x7fbb78113ae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009fd0 tx=0x7fbb6c009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 shutdown_connections 2024-02-28T16:43:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb6006d7f0 0x7fbb6006fca0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb78102f50 0x7fbb78113ae0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 --2- 172.21.15.1:0/576665363 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb781038f0 0x7fbb78114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 >> 172.21.15.1:0/576665363 conn(0x7fbb780f8ae0 msgr2=0x7fbb780fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 shutdown_connections 2024-02-28T16:43:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.125+0000 7fbb7f800700 1 -- 172.21.15.1:0/576665363 wait complete. 2024-02-28T16:43:41.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:40 smithi066 ceph-mon[85781]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:41.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3621522230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:43:41.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2371507346' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:43:41.247 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:43:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.284+0000 7fb852a00700 1 -- 172.21.15.1:0/2606297596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105b10 msgr2=0x7fb84c0ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.284+0000 7fb852a00700 1 --2- 172.21.15.1:0/2606297596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105b10 0x7fb84c0ff180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb838004940 tx=0x7fb838009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.285+0000 7fb852a00700 1 -- 172.21.15.1:0/2606297596 shutdown_connections 2024-02-28T16:43:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.285+0000 7fb852a00700 1 --2- 172.21.15.1:0/2606297596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105b10 0x7fb84c0ff180 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.285+0000 7fb852a00700 1 --2- 172.21.15.1:0/2606297596 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb84c105170 0x7fb84c105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.285+0000 7fb852a00700 1 -- 172.21.15.1:0/2606297596 >> 172.21.15.1:0/2606297596 conn(0x7fb84c0face0 msgr2=0x7fb84c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.285+0000 7fb852a00700 1 -- 172.21.15.1:0/2606297596 shutdown_connections 2024-02-28T16:43:41.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.285+0000 7fb852a00700 1 -- 172.21.15.1:0/2606297596 wait complete. 2024-02-28T16:43:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.286+0000 7fb852a00700 1 Processor -- start 2024-02-28T16:43:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.286+0000 7fb852a00700 1 -- start start 2024-02-28T16:43:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.287+0000 7fb852a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105170 0x7fb84c115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.287+0000 7fb852a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb84c105b10 0x7fb84c116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.287+0000 7fb852a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c116870 con 0x7fb84c105170 2024-02-28T16:43:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.287+0000 7fb852a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c10fb80 con 0x7fb84c105b10 2024-02-28T16:43:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.287+0000 7fb84be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105170 0x7fb84c115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.287+0000 7fb84be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105170 0x7fb84c115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55836/0 (socket says 172.21.15.1:55836) 2024-02-28T16:43:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.287+0000 7fb84be00700 1 -- 172.21.15.1:0/1014650148 learned_addr learned my addr 172.21.15.1:0/1014650148 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:43:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb84b400700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb84c105b10 0x7fb84c116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb84be00700 1 -- 172.21.15.1:0/1014650148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb84c105b10 msgr2=0x7fb84c116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb84be00700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb84c105b10 0x7fb84c116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb84be00700 1 -- 172.21.15.1:0/1014650148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838005040 con 0x7fb84c105170 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb84b400700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb84c105b10 0x7fb84c116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb84be00700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105170 0x7fb84c115ce0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb83c00c8c0 tx=0x7fb83c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb843e00700 1 -- 172.21.15.1:0/1014650148 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb83c00a980 con 0x7fb84c105170 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb843e00700 1 -- 172.21.15.1:0/1014650148 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb83c00ce60 con 0x7fb84c105170 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c10fe60 con 0x7fb84c105170 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.288+0000 7fb843e00700 1 -- 172.21.15.1:0/1014650148 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb83c0115e0 con 0x7fb84c105170 2024-02-28T16:43:41.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.289+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c1103a0 con 0x7fb84c105170 2024-02-28T16:43:41.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.290+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84c107640 con 0x7fb84c105170 2024-02-28T16:43:41.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.291+0000 7fb843e00700 1 -- 172.21.15.1:0/1014650148 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb83c012400 con 0x7fb84c105170 2024-02-28T16:43:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.292+0000 7fb843e00700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb83006d7f0 0x7fb83006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:43:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.292+0000 7fb843e00700 1 -- 172.21.15.1:0/1014650148 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb83c08cdb0 con 0x7fb84c105170 2024-02-28T16:43:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.292+0000 7fb84b400700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb83006d7f0 0x7fb83006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:43:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.293+0000 7fb84b400700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb83006d7f0 0x7fb83006fca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb83800b3b0 tx=0x7fb838009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:43:41.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.297+0000 7fb843e00700 1 -- 172.21.15.1:0/1014650148 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb83c05a250 con 0x7fb84c105170 2024-02-28T16:43:41.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.604+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb84c0fe900 con 0x7fb84c105170 2024-02-28T16:43:41.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.604+0000 7fb843e00700 1 -- 172.21.15.1:0/1014650148 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fb83c0126b0 con 0x7fb84c105170 2024-02-28T16:43:41.591 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:43:41.591 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:43:41.591 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:43:41.592 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:43:41.592 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:43:41.592 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:43:41.592 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:43:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb83006d7f0 msgr2=0x7fb83006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb83006d7f0 0x7fb83006fca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb83800b3b0 tx=0x7fb838009e30 comp rx=0 tx=0).stop 2024-02-28T16:43:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105170 msgr2=0x7fb84c115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:43:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105170 0x7fb84c115ce0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb83c00c8c0 tx=0x7fb83c0044c0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 shutdown_connections 2024-02-28T16:43:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb83006d7f0 0x7fb83006fca0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84c105170 0x7fb84c115ce0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 --2- 172.21.15.1:0/1014650148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb84c105b10 0x7fb84c116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:43:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 >> 172.21.15.1:0/1014650148 conn(0x7fb84c0face0 msgr2=0x7fb84c0fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:43:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 shutdown_connections 2024-02-28T16:43:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:43:41.607+0000 7fb852a00700 1 -- 172.21.15.1:0/1014650148 wait complete. 2024-02-28T16:43:42.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:41 smithi066 ceph-mon[85781]: pgmap v277: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:42.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:41 smithi066 ceph-mon[85781]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:42.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:41 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1014650148' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:43:42.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:41 smithi001 ceph-mon[112940]: pgmap v277: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:42.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:41 smithi001 ceph-mon[112940]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:43:42.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:41 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1014650148' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:43:44.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:43 smithi066 ceph-mon[85781]: pgmap v278: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:44.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:43 smithi001 ceph-mon[112940]: pgmap v278: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:46.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:45 smithi066 ceph-mon[85781]: pgmap v279: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:46.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:45 smithi001 ceph-mon[112940]: pgmap v279: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:48.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:47 smithi066 ceph-mon[85781]: pgmap v280: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:48.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:47 smithi001 ceph-mon[112940]: pgmap v280: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:50.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:49 smithi066 ceph-mon[85781]: pgmap v281: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:50.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:49 smithi001 ceph-mon[112940]: pgmap v281: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:52.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:51 smithi066 ceph-mon[85781]: pgmap v282: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:52.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:51 smithi001 ceph-mon[112940]: pgmap v282: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:54.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:53 smithi066 ceph-mon[85781]: pgmap v283: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:54.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:53 smithi001 ceph-mon[112940]: pgmap v283: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:56.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:56 smithi066 ceph-mon[85781]: pgmap v284: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:56.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:43:56.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:55 smithi001 ceph-mon[112940]: pgmap v284: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:56.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:43:58.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:43:58 smithi066 ceph-mon[85781]: pgmap v285: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:43:58.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:43:58 smithi001 ceph-mon[112940]: pgmap v285: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:00.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:00 smithi001 ceph-mon[112940]: pgmap v286: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:00.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:00 smithi066 ceph-mon[85781]: pgmap v286: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:02.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:02 smithi001 ceph-mon[112940]: pgmap v287: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:02.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:02 smithi066 ceph-mon[85781]: pgmap v287: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:04.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:04 smithi001 ceph-mon[112940]: pgmap v288: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:04.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:04 smithi066 ceph-mon[85781]: pgmap v288: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:06.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:06 smithi001 ceph-mon[112940]: pgmap v289: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:06.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:06 smithi066 ceph-mon[85781]: pgmap v289: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:08.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:08 smithi066 ceph-mon[85781]: pgmap v290: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:08.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:08 smithi001 ceph-mon[112940]: pgmap v290: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:10.299 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:10 smithi066 ceph-mon[85781]: pgmap v291: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:10.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:10 smithi001 ceph-mon[112940]: pgmap v291: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:11.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:44:11.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:44:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.777+0000 7fe192c00700 1 -- 172.21.15.1:0/2070332136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 msgr2=0x7fe18c105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.777+0000 7fe192c00700 1 --2- 172.21.15.1:0/2070332136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 0x7fe18c105540 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe17c004940 tx=0x7fe17c009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.777+0000 7fe192c00700 1 -- 172.21.15.1:0/2070332136 shutdown_connections 2024-02-28T16:44:11.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.777+0000 7fe192c00700 1 --2- 172.21.15.1:0/2070332136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe18c105b10 0x7fe18c0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:11.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.777+0000 7fe192c00700 1 --2- 172.21.15.1:0/2070332136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 0x7fe18c105540 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:11.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.777+0000 7fe192c00700 1 -- 172.21.15.1:0/2070332136 >> 172.21.15.1:0/2070332136 conn(0x7fe18c0face0 msgr2=0x7fe18c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:11.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.777+0000 7fe192c00700 1 -- 172.21.15.1:0/2070332136 shutdown_connections 2024-02-28T16:44:11.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.777+0000 7fe192c00700 1 -- 172.21.15.1:0/2070332136 wait complete. 2024-02-28T16:44:11.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.778+0000 7fe192c00700 1 Processor -- start 2024-02-28T16:44:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.778+0000 7fe192c00700 1 -- start start 2024-02-28T16:44:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.778+0000 7fe192c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 0x7fe18c115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.778+0000 7fe192c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe18c105b10 0x7fe18c1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.778+0000 7fe192c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c1167f0 con 0x7fe18c105170 2024-02-28T16:44:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.778+0000 7fe192c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c10fb40 con 0x7fe18c105b10 2024-02-28T16:44:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.779+0000 7fe18be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 0x7fe18c115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.779+0000 7fe18be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 0x7fe18c115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50690/0 (socket says 172.21.15.1:50690) 2024-02-28T16:44:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.779+0000 7fe18be00700 1 -- 172.21.15.1:0/252890504 learned_addr learned my addr 172.21.15.1:0/252890504 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.779+0000 7fe18be00700 1 -- 172.21.15.1:0/252890504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe18c105b10 msgr2=0x7fe18c1161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.779+0000 7fe18be00700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe18c105b10 0x7fe18c1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.779+0000 7fe18be00700 1 -- 172.21.15.1:0/252890504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c005040 con 0x7fe18c105170 2024-02-28T16:44:11.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.779+0000 7fe18be00700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 0x7fe18c115c60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009150 tx=0x7fe17c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.780+0000 7fe183e00700 1 -- 172.21.15.1:0/252890504 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe17c00e040 con 0x7fe18c105170 2024-02-28T16:44:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.780+0000 7fe183e00700 1 -- 172.21.15.1:0/252890504 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe17c00bbf0 con 0x7fe18c105170 2024-02-28T16:44:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.780+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c10fdc0 con 0x7fe18c105170 2024-02-28T16:44:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.780+0000 7fe183e00700 1 -- 172.21.15.1:0/252890504 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe17c01c630 con 0x7fe18c105170 2024-02-28T16:44:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.780+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c110300 con 0x7fe18c105170 2024-02-28T16:44:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.781+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c102480 con 0x7fe18c105170 2024-02-28T16:44:11.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.782+0000 7fe183e00700 1 -- 172.21.15.1:0/252890504 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe17c00bd60 con 0x7fe18c105170 2024-02-28T16:44:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.783+0000 7fe183e00700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe17006d7f0 0x7fe17006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:11.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.783+0000 7fe183e00700 1 -- 172.21.15.1:0/252890504 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe17c08e390 con 0x7fe18c105170 2024-02-28T16:44:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.783+0000 7fe18b400700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe17006d7f0 0x7fe17006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.784+0000 7fe18b400700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe17006d7f0 0x7fe17006fca0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe18c111540 tx=0x7fe178007d40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:11.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:11.788+0000 7fe183e00700 1 -- 172.21.15.1:0/252890504 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe17c057bc0 con 0x7fe18c105170 2024-02-28T16:44:12.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.033+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe18c04bf70 con 0x7fe17006d7f0 2024-02-28T16:44:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.036+0000 7fe183e00700 1 -- 172.21.15.1:0/252890504 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe17800c5f0 con 0x7fe17006d7f0 2024-02-28T16:44:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe17006d7f0 msgr2=0x7fe17006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe17006d7f0 0x7fe17006fca0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe18c111540 tx=0x7fe178007d40 comp rx=0 tx=0).stop 2024-02-28T16:44:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 msgr2=0x7fe18c115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 0x7fe18c115c60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009150 tx=0x7fe17c009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 shutdown_connections 2024-02-28T16:44:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe17006d7f0 0x7fe17006fca0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18c105170 0x7fe18c115c60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 --2- 172.21.15.1:0/252890504 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe18c105b10 0x7fe18c1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 >> 172.21.15.1:0/252890504 conn(0x7fe18c0face0 msgr2=0x7fe18c0fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 shutdown_connections 2024-02-28T16:44:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.039+0000 7fe192c00700 1 -- 172.21.15.1:0/252890504 wait complete. 2024-02-28T16:44:12.043 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:44:12.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.200+0000 7f5a9dc00700 1 -- 172.21.15.1:0/4035977760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 msgr2=0x7f5a98070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.200+0000 7f5a9dc00700 1 --2- 172.21.15.1:0/4035977760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 0x7f5a98070ce0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5a84004970 tx=0x7f5a84009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:12.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.200+0000 7f5a9dc00700 1 -- 172.21.15.1:0/4035977760 shutdown_connections 2024-02-28T16:44:12.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.200+0000 7f5a9dc00700 1 --2- 172.21.15.1:0/4035977760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 0x7f5a98070ce0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.200+0000 7f5a9dc00700 1 --2- 172.21.15.1:0/4035977760 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a98071f60 0x7f5a98070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.200+0000 7f5a9dc00700 1 -- 172.21.15.1:0/4035977760 >> 172.21.15.1:0/4035977760 conn(0x7f5a980f8af0 msgr2=0x7f5a980faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:12.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.200+0000 7f5a9dc00700 1 -- 172.21.15.1:0/4035977760 shutdown_connections 2024-02-28T16:44:12.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.200+0000 7f5a9dc00700 1 -- 172.21.15.1:0/4035977760 wait complete. 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.201+0000 7f5a9dc00700 1 Processor -- start 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.201+0000 7f5a9dc00700 1 -- start start 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.201+0000 7f5a9dc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 0x7f5a9811a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.201+0000 7f5a9dc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a98071f60 0x7f5a9811a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.201+0000 7f5a9dc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9811ac20 con 0x7f5a98070870 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.201+0000 7f5a9dc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98113f30 con 0x7f5a98071f60 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.202+0000 7f5a96a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 0x7f5a9811a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.202+0000 7f5a96a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 0x7f5a9811a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46638/0 (socket says 172.21.15.1:46638) 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.202+0000 7f5a96a00700 1 -- 172.21.15.1:0/918159289 learned_addr learned my addr 172.21.15.1:0/918159289 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:12.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.202+0000 7f5a96000700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a98071f60 0x7f5a9811a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:12.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.202+0000 7f5a96a00700 1 -- 172.21.15.1:0/918159289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a98071f60 msgr2=0x7f5a9811a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.202+0000 7f5a96a00700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a98071f60 0x7f5a9811a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.202+0000 7f5a96a00700 1 -- 172.21.15.1:0/918159289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a84005040 con 0x7f5a98070870 2024-02-28T16:44:12.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.203+0000 7f5a96a00700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 0x7f5a9811a090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a8800cac0 tx=0x7f5a88007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:12.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.203+0000 7f5a8ea00700 1 -- 172.21.15.1:0/918159289 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a8800aa50 con 0x7f5a98070870 2024-02-28T16:44:12.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.203+0000 7f5a8ea00700 1 -- 172.21.15.1:0/918159289 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a8800abb0 con 0x7f5a98070870 2024-02-28T16:44:12.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.203+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a98114210 con 0x7f5a98070870 2024-02-28T16:44:12.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.203+0000 7f5a8ea00700 1 -- 172.21.15.1:0/918159289 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a880115e0 con 0x7f5a98070870 2024-02-28T16:44:12.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.203+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a98114750 con 0x7f5a98070870 2024-02-28T16:44:12.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.204+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a980fe410 con 0x7f5a98070870 2024-02-28T16:44:12.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.205+0000 7f5a8ea00700 1 -- 172.21.15.1:0/918159289 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5a8800ad20 con 0x7f5a98070870 2024-02-28T16:44:12.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.206+0000 7f5a8ea00700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a7c06d7e0 0x7f5a7c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:12.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.206+0000 7f5a8ea00700 1 -- 172.21.15.1:0/918159289 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5a8808d5e0 con 0x7f5a98070870 2024-02-28T16:44:12.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.207+0000 7f5a96000700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a7c06d7e0 0x7f5a7c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:12.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.207+0000 7f5a96000700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a7c06d7e0 0x7f5a7c06fc90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a98115930 tx=0x7f5a84009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:12.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.211+0000 7f5a8ea00700 1 -- 172.21.15.1:0/918159289 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a8805a960 con 0x7f5a98070870 2024-02-28T16:44:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.454+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a9804bed0 con 0x7f5a7c06d7e0 2024-02-28T16:44:12.443 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:12 smithi001 ceph-mon[112940]: pgmap v292: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:12.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.457+0000 7f5a8ea00700 1 -- 172.21.15.1:0/918159289 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5a9804bed0 con 0x7f5a7c06d7e0 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.461+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a7c06d7e0 msgr2=0x7f5a7c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.461+0000 7f5a9dc00700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a7c06d7e0 0x7f5a7c06fc90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a98115930 tx=0x7f5a84009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.461+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 msgr2=0x7f5a9811a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.461+0000 7f5a9dc00700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 0x7f5a9811a090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a8800cac0 tx=0x7f5a88007ee0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.462+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 shutdown_connections 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.462+0000 7f5a9dc00700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a7c06d7e0 0x7f5a7c06fc90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.462+0000 7f5a9dc00700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a98070870 0x7f5a9811a090 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.462+0000 7f5a9dc00700 1 --2- 172.21.15.1:0/918159289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a98071f60 0x7f5a9811a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.462+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 >> 172.21.15.1:0/918159289 conn(0x7f5a980f8af0 msgr2=0x7f5a980fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.462+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 shutdown_connections 2024-02-28T16:44:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.462+0000 7f5a9dc00700 1 -- 172.21.15.1:0/918159289 wait complete. 2024-02-28T16:44:12.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:12 smithi066 ceph-mon[85781]: pgmap v292: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:12.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.616+0000 7fcf36400700 1 -- 172.21.15.1:0/1910191407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 msgr2=0x7fcf30104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.616+0000 7fcf36400700 1 --2- 172.21.15.1:0/1910191407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 0x7fcf30104210 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c004940 tx=0x7fcf1c009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.616+0000 7fcf36400700 1 -- 172.21.15.1:0/1910191407 shutdown_connections 2024-02-28T16:44:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.616+0000 7fcf36400700 1 --2- 172.21.15.1:0/1910191407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 0x7fcf30104210 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.616+0000 7fcf36400700 1 --2- 172.21.15.1:0/1910191407 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf301095f0 0x7fcf301099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.616+0000 7fcf36400700 1 -- 172.21.15.1:0/1910191407 >> 172.21.15.1:0/1910191407 conn(0x7fcf30072d50 msgr2=0x7fcf30075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.617+0000 7fcf36400700 1 -- 172.21.15.1:0/1910191407 shutdown_connections 2024-02-28T16:44:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.617+0000 7fcf36400700 1 -- 172.21.15.1:0/1910191407 wait complete. 2024-02-28T16:44:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.618+0000 7fcf36400700 1 Processor -- start 2024-02-28T16:44:12.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.618+0000 7fcf36400700 1 -- start start 2024-02-28T16:44:12.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.618+0000 7fcf36400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 0x7fcf3011a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:12.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.618+0000 7fcf36400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf301095f0 0x7fcf3011a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:12.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.618+0000 7fcf36400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3011abd0 con 0x7fcf30103dc0 2024-02-28T16:44:12.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.618+0000 7fcf36400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf30113ee0 con 0x7fcf301095f0 2024-02-28T16:44:12.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.619+0000 7fcf2f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 0x7fcf3011a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:12.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.619+0000 7fcf2f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 0x7fcf3011a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46660/0 (socket says 172.21.15.1:46660) 2024-02-28T16:44:12.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.619+0000 7fcf2f400700 1 -- 172.21.15.1:0/443109677 learned_addr learned my addr 172.21.15.1:0/443109677 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.619+0000 7fcf2ea00700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf301095f0 0x7fcf3011a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.619+0000 7fcf2f400700 1 -- 172.21.15.1:0/443109677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf301095f0 msgr2=0x7fcf3011a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.619+0000 7fcf2f400700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf301095f0 0x7fcf3011a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.619+0000 7fcf2f400700 1 -- 172.21.15.1:0/443109677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c005040 con 0x7fcf30103dc0 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.619+0000 7fcf2ea00700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf301095f0 0x7fcf3011a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.620+0000 7fcf2f400700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 0x7fcf3011a040 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcf20002a50 tx=0x7fcf200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.620+0000 7fcf27400700 1 -- 172.21.15.1:0/443109677 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf20007cc0 con 0x7fcf30103dc0 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.620+0000 7fcf27400700 1 -- 172.21.15.1:0/443109677 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf2000f040 con 0x7fcf30103dc0 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.620+0000 7fcf27400700 1 -- 172.21.15.1:0/443109677 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf20009640 con 0x7fcf30103dc0 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.620+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf301141c0 con 0x7fcf30103dc0 2024-02-28T16:44:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.620+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf30114700 con 0x7fcf30103dc0 2024-02-28T16:44:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.621+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3010b9c0 con 0x7fcf30103dc0 2024-02-28T16:44:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.622+0000 7fcf27400700 1 -- 172.21.15.1:0/443109677 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcf200097a0 con 0x7fcf30103dc0 2024-02-28T16:44:12.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.623+0000 7fcf27400700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1406d7f0 0x7fcf1406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:12.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.623+0000 7fcf27400700 1 -- 172.21.15.1:0/443109677 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcf2008cc60 con 0x7fcf30103dc0 2024-02-28T16:44:12.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.623+0000 7fcf2ea00700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1406d7f0 0x7fcf1406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:12.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.624+0000 7fcf2ea00700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1406d7f0 0x7fcf1406fca0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009fd0 tx=0x7fcf1c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:12.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.628+0000 7fcf27400700 1 -- 172.21.15.1:0/443109677 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf20059fc0 con 0x7fcf30103dc0 2024-02-28T16:44:12.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.853+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf30115620 con 0x7fcf1406d7f0 2024-02-28T16:44:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.878+0000 7fcf27400700 1 -- 172.21.15.1:0/443109677 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fcf30115620 con 0x7fcf1406d7f0 2024-02-28T16:44:12.866 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (10m) 2m ago 24m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (7m) 2m ago 24m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (6m) 3m ago 23m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (9m) 2m ago 24m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (2m) 2m ago 17m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (2m) 2m ago 17m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (17m) 3m ago 17m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (17m) 3m ago 17m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (12m) 2m ago 26m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (11m) 3m ago 23m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (8m) 2m ago 26m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (8m) 3m ago 23m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (11m) 2m ago 24m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:44:12.867 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (11m) 3m ago 23m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:44:12.868 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (6m) 2m ago 22m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:44:12.868 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (5m) 2m ago 21m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:44:12.868 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (5m) 2m ago 21m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:44:12.868 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (4m) 3m ago 20m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:44:12.868 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (4m) 3m ago 19m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:44:12.868 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (3m) 3m ago 19m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:44:12.868 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (10m) 2m ago 23m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:44:12.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1406d7f0 msgr2=0x7fcf1406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1406d7f0 0x7fcf1406fca0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009fd0 tx=0x7fcf1c009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 msgr2=0x7fcf3011a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 0x7fcf3011a040 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcf20002a50 tx=0x7fcf200046b0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 shutdown_connections 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf1406d7f0 0x7fcf1406fca0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf30103dc0 0x7fcf3011a040 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 --2- 172.21.15.1:0/443109677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf301095f0 0x7fcf3011a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 >> 172.21.15.1:0/443109677 conn(0x7fcf30072d50 msgr2=0x7fcf30074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 shutdown_connections 2024-02-28T16:44:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:12.881+0000 7fcf36400700 1 -- 172.21.15.1:0/443109677 wait complete. 2024-02-28T16:44:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.036+0000 7f63d5000700 1 -- 172.21.15.1:0/1746926469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d0072dd0 msgr2=0x7f63d010d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.036+0000 7f63d5000700 1 --2- 172.21.15.1:0/1746926469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d0072dd0 0x7f63d010d700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f63bc004940 tx=0x7f63bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.036+0000 7f63d5000700 1 -- 172.21.15.1:0/1746926469 shutdown_connections 2024-02-28T16:44:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.036+0000 7f63d5000700 1 --2- 172.21.15.1:0/1746926469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d0072dd0 0x7f63d010d700 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.036+0000 7f63d5000700 1 --2- 172.21.15.1:0/1746926469 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f63d00724c0 0x7f63d0072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.036+0000 7f63d5000700 1 -- 172.21.15.1:0/1746926469 >> 172.21.15.1:0/1746926469 conn(0x7f63d00fab40 msgr2=0x7f63d00fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.036+0000 7f63d5000700 1 -- 172.21.15.1:0/1746926469 shutdown_connections 2024-02-28T16:44:13.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.036+0000 7f63d5000700 1 -- 172.21.15.1:0/1746926469 wait complete. 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.037+0000 7f63d5000700 1 Processor -- start 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.037+0000 7f63d5000700 1 -- start start 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63d5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d00724c0 0x7f63d0119d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63d5000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f63d0072dd0 0x7f63d011a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63d5000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d011a8e0 con 0x7f63d00724c0 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63d5000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0113bf0 con 0x7f63d0072dd0 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63cf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d00724c0 0x7f63d0119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63cf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d00724c0 0x7f63d0119d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46684/0 (socket says 172.21.15.1:46684) 2024-02-28T16:44:13.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63cf400700 1 -- 172.21.15.1:0/2513714383 learned_addr learned my addr 172.21.15.1:0/2513714383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63cf400700 1 -- 172.21.15.1:0/2513714383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f63d0072dd0 msgr2=0x7f63d011a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63cf400700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f63d0072dd0 0x7f63d011a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.038+0000 7f63cf400700 1 -- 172.21.15.1:0/2513714383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc005040 con 0x7f63d00724c0 2024-02-28T16:44:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.039+0000 7f63cf400700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d00724c0 0x7f63d0119d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f63c0002a50 tx=0x7f63c00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.039+0000 7f63c7400700 1 -- 172.21.15.1:0/2513714383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63c0007cc0 con 0x7f63d00724c0 2024-02-28T16:44:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.039+0000 7f63c7400700 1 -- 172.21.15.1:0/2513714383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63c000f040 con 0x7f63d00724c0 2024-02-28T16:44:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.039+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d0113ed0 con 0x7f63d00724c0 2024-02-28T16:44:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.039+0000 7f63c7400700 1 -- 172.21.15.1:0/2513714383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63c0009640 con 0x7f63d00724c0 2024-02-28T16:44:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.039+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d0114490 con 0x7f63d00724c0 2024-02-28T16:44:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.040+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d010ae30 con 0x7f63d00724c0 2024-02-28T16:44:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.041+0000 7f63c7400700 1 -- 172.21.15.1:0/2513714383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f63c00097a0 con 0x7f63d00724c0 2024-02-28T16:44:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.043+0000 7f63c7400700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f63b406d7e0 0x7f63b406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.043+0000 7f63c7400700 1 -- 172.21.15.1:0/2513714383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f63c008cbc0 con 0x7f63d00724c0 2024-02-28T16:44:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.043+0000 7f63cea00700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f63b406d7e0 0x7f63b406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.044+0000 7f63cea00700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f63b406d7e0 0x7f63b406fc90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f63bc004f90 tx=0x7f63bc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:13.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.048+0000 7f63c7400700 1 -- 172.21.15.1:0/2513714383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f63c0059fc0 con 0x7f63d00724c0 2024-02-28T16:44:13.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:13 smithi001 ceph-mon[112940]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:13.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:13 smithi001 ceph-mon[112940]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.361+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f63d0115330 con 0x7f63d00724c0 2024-02-28T16:44:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.362+0000 7f63c7400700 1 -- 172.21.15.1:0/2513714383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f63c0059b50 con 0x7f63d00724c0 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:44:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:44:13.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:44:13.351 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:44:13.351 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:44:13.351 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:44:13.351 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:44:13.351 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:44:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.364+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f63b406d7e0 msgr2=0x7f63b406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.364+0000 7f63d5000700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f63b406d7e0 0x7f63b406fc90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f63bc004f90 tx=0x7f63bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d00724c0 msgr2=0x7f63d0119d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d00724c0 0x7f63d0119d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f63c0002a50 tx=0x7f63c00046b0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 shutdown_connections 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f63b406d7e0 0x7f63b406fc90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63d00724c0 0x7f63d0119d50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 --2- 172.21.15.1:0/2513714383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f63d0072dd0 0x7f63d011a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 >> 172.21.15.1:0/2513714383 conn(0x7f63d00fab40 msgr2=0x7f63d00fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 shutdown_connections 2024-02-28T16:44:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.365+0000 7f63d5000700 1 -- 172.21.15.1:0/2513714383 wait complete. 2024-02-28T16:44:13.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:13 smithi066 ceph-mon[85781]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:13.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:13 smithi066 ceph-mon[85781]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:13.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.524+0000 7f8d08600700 1 -- 172.21.15.1:0/146775551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 msgr2=0x7f8d001092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.524+0000 7f8d08600700 1 --2- 172.21.15.1:0/146775551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 0x7f8d001092c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8cec004970 tx=0x7f8cec009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:13.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.525+0000 7f8d08600700 1 -- 172.21.15.1:0/146775551 shutdown_connections 2024-02-28T16:44:13.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.525+0000 7f8d08600700 1 --2- 172.21.15.1:0/146775551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 0x7f8d001092c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.525+0000 7f8d08600700 1 --2- 172.21.15.1:0/146775551 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d000702d0 0x7f8d000706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.525+0000 7f8d08600700 1 -- 172.21.15.1:0/146775551 >> 172.21.15.1:0/146775551 conn(0x7f8d000f8ae0 msgr2=0x7f8d000faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:13.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.525+0000 7f8d08600700 1 -- 172.21.15.1:0/146775551 shutdown_connections 2024-02-28T16:44:13.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.525+0000 7f8d08600700 1 -- 172.21.15.1:0/146775551 wait complete. 2024-02-28T16:44:13.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.526+0000 7f8d08600700 1 Processor -- start 2024-02-28T16:44:13.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.526+0000 7f8d08600700 1 -- start start 2024-02-28T16:44:13.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.526+0000 7f8d08600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d000702d0 0x7f8d00115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.526+0000 7f8d08600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 0x7f8d00116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.526+0000 7f8d08600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d001167a0 con 0x7f8d00070be0 2024-02-28T16:44:13.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.526+0000 7f8d08600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0010faf0 con 0x7f8d000702d0 2024-02-28T16:44:13.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.527+0000 7f8d05200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 0x7f8d00116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:13.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.527+0000 7f8d05200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 0x7f8d00116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46704/0 (socket says 172.21.15.1:46704) 2024-02-28T16:44:13.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.527+0000 7f8d05200700 1 -- 172.21.15.1:0/4278856641 learned_addr learned my addr 172.21.15.1:0/4278856641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.527+0000 7f8d05200700 1 -- 172.21.15.1:0/4278856641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d000702d0 msgr2=0x7f8d00115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.527+0000 7f8d05200700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d000702d0 0x7f8d00115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.527+0000 7f8d05200700 1 -- 172.21.15.1:0/4278856641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec005040 con 0x7f8d00070be0 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.527+0000 7f8d05200700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 0x7f8d00116150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009150 tx=0x7f8cec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.528+0000 7f8cfd600700 1 -- 172.21.15.1:0/4278856641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cec00e040 con 0x7f8d00070be0 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.528+0000 7f8cfd600700 1 -- 172.21.15.1:0/4278856641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8cec004290 con 0x7f8d00070be0 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.528+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0010fd70 con 0x7f8d00070be0 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.528+0000 7f8cfd600700 1 -- 172.21.15.1:0/4278856641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cec01c630 con 0x7f8d00070be0 2024-02-28T16:44:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.528+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d001102b0 con 0x7f8d00070be0 2024-02-28T16:44:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.529+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d00106980 con 0x7f8d00070be0 2024-02-28T16:44:13.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.530+0000 7f8cfd600700 1 -- 172.21.15.1:0/4278856641 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8cec013980 con 0x7f8d00070be0 2024-02-28T16:44:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.531+0000 7f8cfd600700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ce806d7f0 0x7f8ce806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.531+0000 7f8cfd600700 1 -- 172.21.15.1:0/4278856641 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8cec08e210 con 0x7f8d00070be0 2024-02-28T16:44:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.531+0000 7f8d05c00700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ce806d7f0 0x7f8ce806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.532+0000 7f8d05c00700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ce806d7f0 0x7f8ce806fca0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4004d00 tx=0x7f8cf4009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:13.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.535+0000 7f8cfd600700 1 -- 172.21.15.1:0/4278856641 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8cec057ac0 con 0x7f8d00070be0 2024-02-28T16:44:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.813+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8d0004bed0 con 0x7f8d00070be0 2024-02-28T16:44:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.814+0000 7f8cfd600700 1 -- 172.21.15.1:0/4278856641 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f8cec05b180 con 0x7f8d00070be0 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:44:13.803 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:44:13.804 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:44:13.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:44:13.806 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:44:13.806 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:44:13.806 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:44:13.806 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:44:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.818+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ce806d7f0 msgr2=0x7f8ce806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.818+0000 7f8d08600700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ce806d7f0 0x7f8ce806fca0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4004d00 tx=0x7f8cf4009ce0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.818+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 msgr2=0x7f8d00116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.818+0000 7f8d08600700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 0x7f8d00116150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009150 tx=0x7f8cec009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.819+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 shutdown_connections 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.819+0000 7f8d08600700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ce806d7f0 0x7f8ce806fca0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.819+0000 7f8d08600700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d000702d0 0x7f8d00115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.819+0000 7f8d08600700 1 --2- 172.21.15.1:0/4278856641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d00070be0 0x7f8d00116150 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.819+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 >> 172.21.15.1:0/4278856641 conn(0x7f8d000f8ae0 msgr2=0x7f8d00103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.819+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 shutdown_connections 2024-02-28T16:44:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.819+0000 7f8d08600700 1 -- 172.21.15.1:0/4278856641 wait complete. 2024-02-28T16:44:13.808 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:44:13.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.969+0000 7f98cf200700 1 -- 172.21.15.1:0/2810477655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 msgr2=0x7f98c81099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.969+0000 7f98cf200700 1 --2- 172.21.15.1:0/2810477655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 0x7f98c81099c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f98b8004940 tx=0x7f98b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:13.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.969+0000 7f98cf200700 1 -- 172.21.15.1:0/2810477655 shutdown_connections 2024-02-28T16:44:13.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.969+0000 7f98cf200700 1 --2- 172.21.15.1:0/2810477655 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98c8103dc0 0x7f98c8104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.969+0000 7f98cf200700 1 --2- 172.21.15.1:0/2810477655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 0x7f98c81099c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.969+0000 7f98cf200700 1 -- 172.21.15.1:0/2810477655 >> 172.21.15.1:0/2810477655 conn(0x7f98c8072d50 msgr2=0x7f98c8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:13.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.970+0000 7f98cf200700 1 -- 172.21.15.1:0/2810477655 shutdown_connections 2024-02-28T16:44:13.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.970+0000 7f98cf200700 1 -- 172.21.15.1:0/2810477655 wait complete. 2024-02-28T16:44:13.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.971+0000 7f98cf200700 1 Processor -- start 2024-02-28T16:44:13.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.971+0000 7f98cf200700 1 -- start start 2024-02-28T16:44:13.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.971+0000 7f98cf200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98c8103dc0 0x7f98c811a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.971+0000 7f98cf200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 0x7f98c811a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.971+0000 7f98cf200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c811abf0 con 0x7f98c81095f0 2024-02-28T16:44:13.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.971+0000 7f98cf200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8113f00 con 0x7f98c8103dc0 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.972+0000 7f98c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 0x7f98c811a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.972+0000 7f98c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 0x7f98c811a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46734/0 (socket says 172.21.15.1:46734) 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.972+0000 7f98c7400700 1 -- 172.21.15.1:0/3638130664 learned_addr learned my addr 172.21.15.1:0/3638130664 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.972+0000 7f98c7e00700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98c8103dc0 0x7f98c811a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.972+0000 7f98c7400700 1 -- 172.21.15.1:0/3638130664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98c8103dc0 msgr2=0x7f98c811a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.972+0000 7f98c7400700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98c8103dc0 0x7f98c811a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.972+0000 7f98c7400700 1 -- 172.21.15.1:0/3638130664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8005040 con 0x7f98c81095f0 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.972+0000 7f98c7e00700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98c8103dc0 0x7f98c811a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.973+0000 7f98c7400700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 0x7f98c811a5a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f98b4002a50 tx=0x7f98b40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.973+0000 7f98bfe00700 1 -- 172.21.15.1:0/3638130664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98b4007cc0 con 0x7f98c81095f0 2024-02-28T16:44:13.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.973+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c81141e0 con 0x7f98c81095f0 2024-02-28T16:44:13.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.973+0000 7f98bfe00700 1 -- 172.21.15.1:0/3638130664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98b400f040 con 0x7f98c81095f0 2024-02-28T16:44:13.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.973+0000 7f98bfe00700 1 -- 172.21.15.1:0/3638130664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98b400db70 con 0x7f98c81095f0 2024-02-28T16:44:13.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.973+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c8114720 con 0x7f98c81095f0 2024-02-28T16:44:13.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.974+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c8075870 con 0x7f98c81095f0 2024-02-28T16:44:13.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.976+0000 7f98bfe00700 1 -- 172.21.15.1:0/3638130664 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f98b400dd10 con 0x7f98c81095f0 2024-02-28T16:44:13.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.977+0000 7f98bfe00700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98ac06d7f0 0x7f98ac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:13.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.977+0000 7f98bfe00700 1 -- 172.21.15.1:0/3638130664 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f98b408c4f0 con 0x7f98c81095f0 2024-02-28T16:44:13.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.977+0000 7f98c7e00700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98ac06d7f0 0x7f98ac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:13.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.978+0000 7f98c7e00700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98ac06d7f0 0x7f98ac06fca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009fd0 tx=0x7f98b8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:13.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:13.982+0000 7f98bfe00700 1 -- 172.21.15.1:0/3638130664 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98b40598f0 con 0x7f98c81095f0 2024-02-28T16:44:14.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.224+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98c8115720 con 0x7f98ac06d7f0 2024-02-28T16:44:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.226+0000 7f98bfe00700 1 -- 172.21.15.1:0/3638130664 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f98c8115720 con 0x7f98ac06d7f0 2024-02-28T16:44:14.214 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:44:14.214 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:44:14.215 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:44:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.229+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98ac06d7f0 msgr2=0x7f98ac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.229+0000 7f98cf200700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98ac06d7f0 0x7f98ac06fca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009fd0 tx=0x7f98b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:14.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 msgr2=0x7f98c811a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:14.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 0x7f98c811a5a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f98b4002a50 tx=0x7f98b40046b0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 shutdown_connections 2024-02-28T16:44:14.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98ac06d7f0 0x7f98ac06fca0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f98c8103dc0 0x7f98c811a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 --2- 172.21.15.1:0/3638130664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c81095f0 0x7f98c811a5a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 >> 172.21.15.1:0/3638130664 conn(0x7f98c8072d50 msgr2=0x7f98c8074760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:14.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 shutdown_connections 2024-02-28T16:44:14.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.230+0000 7f98cf200700 1 -- 172.21.15.1:0/3638130664 wait complete. 2024-02-28T16:44:14.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.388+0000 7f79fee00700 1 -- 172.21.15.1:0/3101904987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 msgr2=0x7f79f8070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:14.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.388+0000 7f79fee00700 1 --2- 172.21.15.1:0/3101904987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 0x7f79f8070330 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f79e8004970 tx=0x7f79e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:14.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.388+0000 7f79fee00700 1 -- 172.21.15.1:0/3101904987 shutdown_connections 2024-02-28T16:44:14.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.388+0000 7f79fee00700 1 --2- 172.21.15.1:0/3101904987 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79f8070870 0x7f79f8070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.388+0000 7f79fee00700 1 --2- 172.21.15.1:0/3101904987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 0x7f79f8070330 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.388+0000 7f79fee00700 1 -- 172.21.15.1:0/3101904987 >> 172.21.15.1:0/3101904987 conn(0x7f79f80f8ae0 msgr2=0x7f79f80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:14.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.388+0000 7f79fee00700 1 -- 172.21.15.1:0/3101904987 shutdown_connections 2024-02-28T16:44:14.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.388+0000 7f79fee00700 1 -- 172.21.15.1:0/3101904987 wait complete. 2024-02-28T16:44:14.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:14 smithi001 ceph-mon[112940]: pgmap v293: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:14.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:14 smithi001 ceph-mon[112940]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:14.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:14 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2513714383' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:44:14.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:14 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4278856641' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.389+0000 7f79fee00700 1 Processor -- start 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.389+0000 7f79fee00700 1 -- start start 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.389+0000 7f79fee00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79f8070870 0x7f79f811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.389+0000 7f79fee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 0x7f79f811a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.389+0000 7f79fee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f811abe0 con 0x7f79f8071f60 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.389+0000 7f79fee00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f8113ef0 con 0x7f79f8070870 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.390+0000 7f79f7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 0x7f79f811a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.390+0000 7f79f7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 0x7f79f811a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46750/0 (socket says 172.21.15.1:46750) 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.390+0000 7f79f7400700 1 -- 172.21.15.1:0/3932639781 learned_addr learned my addr 172.21.15.1:0/3932639781 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.390+0000 7f79f7e00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79f8070870 0x7f79f811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.390+0000 7f79f7400700 1 -- 172.21.15.1:0/3932639781 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79f8070870 msgr2=0x7f79f811a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.390+0000 7f79f7400700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79f8070870 0x7f79f811a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.390+0000 7f79f7400700 1 -- 172.21.15.1:0/3932639781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8005040 con 0x7f79f8071f60 2024-02-28T16:44:14.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.391+0000 7f79f7400700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 0x7f79f811a590 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f79e4002a50 tx=0x7f79e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:14.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.391+0000 7f79efe00700 1 -- 172.21.15.1:0/3932639781 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e4007cc0 con 0x7f79f8071f60 2024-02-28T16:44:14.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.391+0000 7f79efe00700 1 -- 172.21.15.1:0/3932639781 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79e400f040 con 0x7f79f8071f60 2024-02-28T16:44:14.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.391+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f81141d0 con 0x7f79f8071f60 2024-02-28T16:44:14.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.391+0000 7f79efe00700 1 -- 172.21.15.1:0/3932639781 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e400db70 con 0x7f79f8071f60 2024-02-28T16:44:14.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.391+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f8114710 con 0x7f79f8071f60 2024-02-28T16:44:14.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.392+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f80fe400 con 0x7f79f8071f60 2024-02-28T16:44:14.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.395+0000 7f79efe00700 1 -- 172.21.15.1:0/3932639781 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f79e400dcd0 con 0x7f79f8071f60 2024-02-28T16:44:14.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.396+0000 7f79efe00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f79dc06d7f0 0x7f79dc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:14.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.396+0000 7f79efe00700 1 -- 172.21.15.1:0/3932639781 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f79e408c840 con 0x7f79f8071f60 2024-02-28T16:44:14.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.397+0000 7f79f7e00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f79dc06d7f0 0x7f79dc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:14.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.398+0000 7f79f7e00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f79dc06d7f0 0x7f79dc06fca0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f79e8004f50 tx=0x7f79e8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.400+0000 7f79efe00700 1 -- 172.21.15.1:0/3932639781 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f79e4059ba0 con 0x7f79f8071f60 2024-02-28T16:44:14.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:14 smithi066 ceph-mon[85781]: pgmap v293: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:14.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:14 smithi066 ceph-mon[85781]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:14.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:14 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2513714383' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:44:14.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:14 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4278856641' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:44:14.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.715+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f79f80fc6e0 con 0x7f79f8071f60 2024-02-28T16:44:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.716+0000 7f79efe00700 1 -- 172.21.15.1:0/3932639781 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f79e4059730 con 0x7f79f8071f60 2024-02-28T16:44:14.704 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:44:14.705 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:44:14.705 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:44:14.705 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:44:14.705 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:44:14.705 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:44:14.705 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.719+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f79dc06d7f0 msgr2=0x7f79dc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.719+0000 7f79fee00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f79dc06d7f0 0x7f79dc06fca0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f79e8004f50 tx=0x7f79e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.719+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 msgr2=0x7f79f811a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.719+0000 7f79fee00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 0x7f79f811a590 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f79e4002a50 tx=0x7f79e40046b0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.720+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 shutdown_connections 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.720+0000 7f79fee00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f79dc06d7f0 0x7f79dc06fca0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.720+0000 7f79fee00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f79f8070870 0x7f79f811a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.720+0000 7f79fee00700 1 --2- 172.21.15.1:0/3932639781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79f8071f60 0x7f79f811a590 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.720+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 >> 172.21.15.1:0/3932639781 conn(0x7f79f80f8ae0 msgr2=0x7f79f80fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.720+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 shutdown_connections 2024-02-28T16:44:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:14.720+0000 7f79fee00700 1 -- 172.21.15.1:0/3932639781 wait complete. 2024-02-28T16:44:15.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:15 smithi066 ceph-mon[85781]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:15.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:15 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3932639781' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:44:15.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:15 smithi001 ceph-mon[112940]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:15.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:15 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3932639781' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:44:16.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:16 smithi066 ceph-mon[85781]: pgmap v294: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:16.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:16 smithi001 ceph-mon[112940]: pgmap v294: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:18.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:18 smithi066 ceph-mon[85781]: pgmap v295: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:18.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:18 smithi001 ceph-mon[112940]: pgmap v295: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:20.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:20 smithi066 ceph-mon[85781]: pgmap v296: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:20.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:20 smithi001 ceph-mon[112940]: pgmap v296: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:22.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:22 smithi066 ceph-mon[85781]: pgmap v297: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:22.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:22 smithi001 ceph-mon[112940]: pgmap v297: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:24.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:24 smithi066 ceph-mon[85781]: pgmap v298: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:24.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:24 smithi001 ceph-mon[112940]: pgmap v298: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:26.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:26 smithi066 ceph-mon[85781]: pgmap v299: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:26.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:44:26.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:26 smithi001 ceph-mon[112940]: pgmap v299: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:26.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:44:28.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:28 smithi066 ceph-mon[85781]: pgmap v300: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:28.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:28 smithi001 ceph-mon[112940]: pgmap v300: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:30.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:30 smithi066 ceph-mon[85781]: pgmap v301: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:30.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:30 smithi001 ceph-mon[112940]: pgmap v301: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:32.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:32 smithi066 ceph-mon[85781]: pgmap v302: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:32.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:32 smithi001 ceph-mon[112940]: pgmap v302: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:34.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:34 smithi066 ceph-mon[85781]: pgmap v303: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:34.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:34 smithi001 ceph-mon[112940]: pgmap v303: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:36.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:36 smithi066 ceph-mon[85781]: pgmap v304: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:36.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:36 smithi001 ceph-mon[112940]: pgmap v304: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:38.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:38 smithi066 ceph-mon[85781]: pgmap v305: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:38.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:38 smithi001 ceph-mon[112940]: pgmap v305: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:40.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:40 smithi066 ceph-mon[85781]: pgmap v306: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:40.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:40 smithi001 ceph-mon[112940]: pgmap v306: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:41.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:44:41.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:44:42.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:42 smithi066 ceph-mon[85781]: pgmap v307: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:42.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:42 smithi001 ceph-mon[112940]: pgmap v307: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:44.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:44 smithi066 ceph-mon[85781]: pgmap v308: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:44.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:44 smithi001 ceph-mon[112940]: pgmap v308: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.893+0000 7f6e6b000700 1 -- 172.21.15.1:0/2317871682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 msgr2=0x7f6e64103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.893+0000 7f6e6b000700 1 --2- 172.21.15.1:0/2317871682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 0x7f6e64103320 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6e54004940 tx=0x7f6e54009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.894+0000 7f6e6b000700 1 -- 172.21.15.1:0/2317871682 shutdown_connections 2024-02-28T16:44:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.894+0000 7f6e6b000700 1 --2- 172.21.15.1:0/2317871682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e641038f0 0x7f6e640fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.894+0000 7f6e6b000700 1 --2- 172.21.15.1:0/2317871682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 0x7f6e64103320 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.894+0000 7f6e6b000700 1 -- 172.21.15.1:0/2317871682 >> 172.21.15.1:0/2317871682 conn(0x7f6e640f8ae0 msgr2=0x7f6e640faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.894+0000 7f6e6b000700 1 -- 172.21.15.1:0/2317871682 shutdown_connections 2024-02-28T16:44:44.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.894+0000 7f6e6b000700 1 -- 172.21.15.1:0/2317871682 wait complete. 2024-02-28T16:44:44.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.895+0000 7f6e6b000700 1 Processor -- start 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.895+0000 7f6e6b000700 1 -- start start 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.895+0000 7f6e6b000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 0x7f6e64115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.895+0000 7f6e6b000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e641038f0 0x7f6e64116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.895+0000 7f6e6b000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e641167a0 con 0x7f6e64102f50 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.895+0000 7f6e6b000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6410faf0 con 0x7f6e641038f0 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.896+0000 7f6e63e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 0x7f6e64115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.896+0000 7f6e63e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 0x7f6e64115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38770/0 (socket says 172.21.15.1:38770) 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.896+0000 7f6e63e00700 1 -- 172.21.15.1:0/1755648709 learned_addr learned my addr 172.21.15.1:0/1755648709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.896+0000 7f6e63400700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e641038f0 0x7f6e64116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.896+0000 7f6e63e00700 1 -- 172.21.15.1:0/1755648709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e641038f0 msgr2=0x7f6e64116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.896+0000 7f6e63e00700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e641038f0 0x7f6e64116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:44.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.896+0000 7f6e63e00700 1 -- 172.21.15.1:0/1755648709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54005040 con 0x7f6e64102f50 2024-02-28T16:44:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.896+0000 7f6e63e00700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 0x7f6e64115c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009fd0 tx=0x7f6e54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.897+0000 7f6e5be00700 1 -- 172.21.15.1:0/1755648709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e5400e040 con 0x7f6e64102f50 2024-02-28T16:44:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.897+0000 7f6e5be00700 1 -- 172.21.15.1:0/1755648709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e5400bbf0 con 0x7f6e64102f50 2024-02-28T16:44:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.897+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6410fd70 con 0x7f6e64102f50 2024-02-28T16:44:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.897+0000 7f6e5be00700 1 -- 172.21.15.1:0/1755648709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e5401c3f0 con 0x7f6e64102f50 2024-02-28T16:44:44.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.897+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e641102b0 con 0x7f6e64102f50 2024-02-28T16:44:44.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.898+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e640fc6e0 con 0x7f6e64102f50 2024-02-28T16:44:44.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.899+0000 7f6e5be00700 1 -- 172.21.15.1:0/1755648709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6e5400bd60 con 0x7f6e64102f50 2024-02-28T16:44:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.900+0000 7f6e5be00700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e4806d7f0 0x7f6e4806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:44.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.900+0000 7f6e5be00700 1 -- 172.21.15.1:0/1755648709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6e5408e2b0 con 0x7f6e64102f50 2024-02-28T16:44:44.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.901+0000 7f6e63400700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e4806d7f0 0x7f6e4806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:44.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.901+0000 7f6e63400700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e4806d7f0 0x7f6e4806fca0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6e641114b0 tx=0x7f6e50009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:44.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:44.906+0000 7f6e5be00700 1 -- 172.21.15.1:0/1755648709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e54057ae0 con 0x7f6e64102f50 2024-02-28T16:44:45.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.158+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e64111290 con 0x7f6e4806d7f0 2024-02-28T16:44:45.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.162+0000 7f6e5be00700 1 -- 172.21.15.1:0/1755648709 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6e64111290 con 0x7f6e4806d7f0 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e4806d7f0 msgr2=0x7f6e4806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e4806d7f0 0x7f6e4806fca0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6e641114b0 tx=0x7f6e50009040 comp rx=0 tx=0).stop 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 msgr2=0x7f6e64115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 0x7f6e64115c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009fd0 tx=0x7f6e54009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 shutdown_connections 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e4806d7f0 0x7f6e4806fca0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e64102f50 0x7f6e64115c10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 --2- 172.21.15.1:0/1755648709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e641038f0 0x7f6e64116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 >> 172.21.15.1:0/1755648709 conn(0x7f6e640f8ae0 msgr2=0x7f6e640fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:45.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 shutdown_connections 2024-02-28T16:44:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.165+0000 7f6e6b000700 1 -- 172.21.15.1:0/1755648709 wait complete. 2024-02-28T16:44:45.169 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:44:45.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.323+0000 7fc9f9e00700 1 -- 172.21.15.1:0/228536127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105b10 msgr2=0x7fc9f40ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.323+0000 7fc9f9e00700 1 --2- 172.21.15.1:0/228536127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105b10 0x7fc9f40ff180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0004940 tx=0x7fc9e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:45.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.323+0000 7fc9f9e00700 1 -- 172.21.15.1:0/228536127 shutdown_connections 2024-02-28T16:44:45.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.323+0000 7fc9f9e00700 1 --2- 172.21.15.1:0/228536127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105b10 0x7fc9f40ff180 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.323+0000 7fc9f9e00700 1 --2- 172.21.15.1:0/228536127 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9f4105170 0x7fc9f4105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.323+0000 7fc9f9e00700 1 -- 172.21.15.1:0/228536127 >> 172.21.15.1:0/228536127 conn(0x7fc9f40face0 msgr2=0x7fc9f40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:45.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.323+0000 7fc9f9e00700 1 -- 172.21.15.1:0/228536127 shutdown_connections 2024-02-28T16:44:45.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.323+0000 7fc9f9e00700 1 -- 172.21.15.1:0/228536127 wait complete. 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f9e00700 1 Processor -- start 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f9e00700 1 -- start start 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105170 0x7fc9f4115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f9e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9f4105b10 0x7fc9f4116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f9e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f4116870 con 0x7fc9f4105170 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f9e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f410fb80 con 0x7fc9f4105b10 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105170 0x7fc9f4115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105170 0x7fc9f4115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38784/0 (socket says 172.21.15.1:38784) 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.325+0000 7fc9f2a00700 1 -- 172.21.15.1:0/1306151250 learned_addr learned my addr 172.21.15.1:0/1306151250 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:45.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.326+0000 7fc9f2000700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9f4105b10 0x7fc9f4116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:45.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.326+0000 7fc9f2a00700 1 -- 172.21.15.1:0/1306151250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9f4105b10 msgr2=0x7fc9f4116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.326+0000 7fc9f2a00700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9f4105b10 0x7fc9f4116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.326+0000 7fc9f2a00700 1 -- 172.21.15.1:0/1306151250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e0005040 con 0x7fc9f4105170 2024-02-28T16:44:45.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.326+0000 7fc9f2a00700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105170 0x7fc9f4115ce0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4002a50 tx=0x7fc9e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:45.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.326+0000 7fc9eaa00700 1 -- 172.21.15.1:0/1306151250 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9e4007cc0 con 0x7fc9f4105170 2024-02-28T16:44:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.327+0000 7fc9eaa00700 1 -- 172.21.15.1:0/1306151250 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9e400f040 con 0x7fc9f4105170 2024-02-28T16:44:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.327+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f410fe60 con 0x7fc9f4105170 2024-02-28T16:44:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.327+0000 7fc9eaa00700 1 -- 172.21.15.1:0/1306151250 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9e4009640 con 0x7fc9f4105170 2024-02-28T16:44:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.327+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f41103a0 con 0x7fc9f4105170 2024-02-28T16:44:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.329+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f4107640 con 0x7fc9f4105170 2024-02-28T16:44:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.330+0000 7fc9eaa00700 1 -- 172.21.15.1:0/1306151250 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc9e40097a0 con 0x7fc9f4105170 2024-02-28T16:44:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.330+0000 7fc9eaa00700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc9d806d7f0 0x7fc9d806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.330+0000 7fc9eaa00700 1 -- 172.21.15.1:0/1306151250 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc9e408cc60 con 0x7fc9f4105170 2024-02-28T16:44:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.330+0000 7fc9f2000700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc9d806d7f0 0x7fc9d806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:45.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.332+0000 7fc9f2000700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc9d806d7f0 0x7fc9d806fca0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4111580 tx=0x7fc9e0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:45.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.335+0000 7fc9eaa00700 1 -- 172.21.15.1:0/1306151250 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc9e4059fc0 con 0x7fc9f4105170 2024-02-28T16:44:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.584+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9f41112c0 con 0x7fc9d806d7f0 2024-02-28T16:44:45.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.586+0000 7fc9eaa00700 1 -- 172.21.15.1:0/1306151250 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc9f41112c0 con 0x7fc9d806d7f0 2024-02-28T16:44:45.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc9d806d7f0 msgr2=0x7fc9d806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc9d806d7f0 0x7fc9d806fca0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4111580 tx=0x7fc9e0009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105170 msgr2=0x7fc9f4115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105170 0x7fc9f4115ce0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4002a50 tx=0x7fc9e40046b0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 shutdown_connections 2024-02-28T16:44:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc9d806d7f0 0x7fc9d806fca0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f4105170 0x7fc9f4115ce0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 --2- 172.21.15.1:0/1306151250 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9f4105b10 0x7fc9f4116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 >> 172.21.15.1:0/1306151250 conn(0x7fc9f40face0 msgr2=0x7fc9f40fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 shutdown_connections 2024-02-28T16:44:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.589+0000 7fc9f9e00700 1 -- 172.21.15.1:0/1306151250 wait complete. 2024-02-28T16:44:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.743+0000 7f4e85800700 1 -- 172.21.15.1:0/2402124796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e80103dc0 msgr2=0x7f4e80104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.743+0000 7f4e85800700 1 --2- 172.21.15.1:0/2402124796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e80103dc0 0x7f4e80104210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4e68004940 tx=0x7f4e68009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.743+0000 7f4e85800700 1 -- 172.21.15.1:0/2402124796 shutdown_connections 2024-02-28T16:44:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.743+0000 7f4e85800700 1 --2- 172.21.15.1:0/2402124796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e80103dc0 0x7f4e80104210 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.743+0000 7f4e85800700 1 --2- 172.21.15.1:0/2402124796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e801095f0 0x7f4e801099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.743+0000 7f4e85800700 1 -- 172.21.15.1:0/2402124796 >> 172.21.15.1:0/2402124796 conn(0x7f4e80072d50 msgr2=0x7f4e80075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.743+0000 7f4e85800700 1 -- 172.21.15.1:0/2402124796 shutdown_connections 2024-02-28T16:44:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.743+0000 7f4e85800700 1 -- 172.21.15.1:0/2402124796 wait complete. 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.744+0000 7f4e85800700 1 Processor -- start 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.744+0000 7f4e85800700 1 -- start start 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.744+0000 7f4e85800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e80103dc0 0x7f4e8011a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.744+0000 7f4e85800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e801095f0 0x7f4e8011a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.744+0000 7f4e85800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8011ac50 con 0x7f4e801095f0 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.744+0000 7f4e85800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e80113ed0 con 0x7f4e80103dc0 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.745+0000 7f4e77e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e801095f0 0x7f4e8011a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.745+0000 7f4e77e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e801095f0 0x7f4e8011a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38800/0 (socket says 172.21.15.1:38800) 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.745+0000 7f4e77e00700 1 -- 172.21.15.1:0/2905806531 learned_addr learned my addr 172.21.15.1:0/2905806531 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.745+0000 7f4e7ea00700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e80103dc0 0x7f4e8011a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.745+0000 7f4e77e00700 1 -- 172.21.15.1:0/2905806531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e80103dc0 msgr2=0x7f4e8011a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.745+0000 7f4e77e00700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e80103dc0 0x7f4e8011a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.745+0000 7f4e77e00700 1 -- 172.21.15.1:0/2905806531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68005040 con 0x7f4e801095f0 2024-02-28T16:44:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.745+0000 7f4e77e00700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e801095f0 0x7f4e8011a570 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009150 tx=0x7f4e68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.746+0000 7f4e76a00700 1 -- 172.21.15.1:0/2905806531 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e6800e040 con 0x7f4e801095f0 2024-02-28T16:44:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.746+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e801140f0 con 0x7f4e801095f0 2024-02-28T16:44:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.746+0000 7f4e76a00700 1 -- 172.21.15.1:0/2905806531 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e6800bbf0 con 0x7f4e801095f0 2024-02-28T16:44:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.746+0000 7f4e76a00700 1 -- 172.21.15.1:0/2905806531 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e6801c630 con 0x7f4e801095f0 2024-02-28T16:44:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.746+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e80114630 con 0x7f4e801095f0 2024-02-28T16:44:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.747+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e80075870 con 0x7f4e801095f0 2024-02-28T16:44:45.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.748+0000 7f4e76a00700 1 -- 172.21.15.1:0/2905806531 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4e6800bd60 con 0x7f4e801095f0 2024-02-28T16:44:45.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.749+0000 7f4e76a00700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e6406d7f0 0x7f4e6406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:45.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.749+0000 7f4e76a00700 1 -- 172.21.15.1:0/2905806531 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4e6808e1e0 con 0x7f4e801095f0 2024-02-28T16:44:45.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.749+0000 7f4e7ea00700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e6406d7f0 0x7f4e6406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:45.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.750+0000 7f4e7ea00700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e6406d7f0 0x7f4e6406fca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006fd0 tx=0x7f4e70008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:45.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.754+0000 7f4e76a00700 1 -- 172.21.15.1:0/2905806531 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4e6805b680 con 0x7f4e801095f0 2024-02-28T16:44:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:45.992+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e80115680 con 0x7f4e6406d7f0 2024-02-28T16:44:45.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.005+0000 7f4e76a00700 1 -- 172.21.15.1:0/2905806531 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4e80115680 con 0x7f4e6406d7f0 2024-02-28T16:44:45.992 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:44:45.992 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (10m) 2m ago 25m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (7m) 2m ago 25m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (7m) 3m ago 24m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (10m) 2m ago 24m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (3m) 2m ago 18m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (3m) 2m ago 17m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (17m) 3m ago 17m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (17m) 3m ago 17m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (13m) 2m ago 26m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (12m) 3m ago 23m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (9m) 2m ago 27m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:44:45.993 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (8m) 3m ago 23m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (11m) 2m ago 25m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (11m) 3m ago 23m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (6m) 2m ago 22m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (6m) 2m ago 22m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (5m) 2m ago 21m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (5m) 3m ago 21m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (4m) 3m ago 20m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (3m) 3m ago 19m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:44:45.994 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (11m) 2m ago 24m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:44:45.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e6406d7f0 msgr2=0x7f4e6406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e6406d7f0 0x7f4e6406fca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006fd0 tx=0x7f4e70008040 comp rx=0 tx=0).stop 2024-02-28T16:44:45.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e801095f0 msgr2=0x7f4e8011a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:45.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e801095f0 0x7f4e8011a570 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009150 tx=0x7f4e68009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:45.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 shutdown_connections 2024-02-28T16:44:45.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e6406d7f0 0x7f4e6406fca0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e80103dc0 0x7f4e8011a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 --2- 172.21.15.1:0/2905806531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e801095f0 0x7f4e8011a570 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:45.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.008+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 >> 172.21.15.1:0/2905806531 conn(0x7f4e80072d50 msgr2=0x7f4e800746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:45.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.009+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 shutdown_connections 2024-02-28T16:44:45.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.009+0000 7f4e85800700 1 -- 172.21.15.1:0/2905806531 wait complete. 2024-02-28T16:44:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.162+0000 7fb36e400700 1 -- 172.21.15.1:0/2623671510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 msgr2=0x7fb3680fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.162+0000 7fb36e400700 1 --2- 172.21.15.1:0/2623671510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 0x7fb3680fbe70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb354004940 tx=0x7fb354009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.162+0000 7fb36e400700 1 -- 172.21.15.1:0/2623671510 shutdown_connections 2024-02-28T16:44:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.162+0000 7fb36e400700 1 --2- 172.21.15.1:0/2623671510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 0x7fb3680fbe70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.162+0000 7fb36e400700 1 --2- 172.21.15.1:0/2623671510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb368101e60 0x7fb368102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.162+0000 7fb36e400700 1 -- 172.21.15.1:0/2623671510 >> 172.21.15.1:0/2623671510 conn(0x7fb3680f7a10 msgr2=0x7fb3680f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.162+0000 7fb36e400700 1 -- 172.21.15.1:0/2623671510 shutdown_connections 2024-02-28T16:44:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.163+0000 7fb36e400700 1 -- 172.21.15.1:0/2623671510 wait complete. 2024-02-28T16:44:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.164+0000 7fb36e400700 1 Processor -- start 2024-02-28T16:44:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.164+0000 7fb36e400700 1 -- start start 2024-02-28T16:44:46.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.164+0000 7fb36e400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb368101e60 0x7fb368115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:46.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.164+0000 7fb36e400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 0x7fb368116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:46.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.164+0000 7fb36e400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368116870 con 0x7fb368102800 2024-02-28T16:44:46.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.164+0000 7fb36e400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36810fb80 con 0x7fb368101e60 2024-02-28T16:44:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.165+0000 7fb366a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 0x7fb368116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.165+0000 7fb366a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 0x7fb368116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38808/0 (socket says 172.21.15.1:38808) 2024-02-28T16:44:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.165+0000 7fb366a00700 1 -- 172.21.15.1:0/3004919972 learned_addr learned my addr 172.21.15.1:0/3004919972 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.165+0000 7fb367400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb368101e60 0x7fb368115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.165+0000 7fb366a00700 1 -- 172.21.15.1:0/3004919972 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb368101e60 msgr2=0x7fb368115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.165+0000 7fb366a00700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb368101e60 0x7fb368115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.165+0000 7fb366a00700 1 -- 172.21.15.1:0/3004919972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb354005040 con 0x7fb368102800 2024-02-28T16:44:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.166+0000 7fb366a00700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 0x7fb368116220 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb354009100 tx=0x7fb354009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.166+0000 7fb35f400700 1 -- 172.21.15.1:0/3004919972 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb35400e040 con 0x7fb368102800 2024-02-28T16:44:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.166+0000 7fb35f400700 1 -- 172.21.15.1:0/3004919972 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb354004290 con 0x7fb368102800 2024-02-28T16:44:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.166+0000 7fb35f400700 1 -- 172.21.15.1:0/3004919972 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb35401c3f0 con 0x7fb368102800 2024-02-28T16:44:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.166+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36810fe00 con 0x7fb368102800 2024-02-28T16:44:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.166+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb368110340 con 0x7fb368102800 2024-02-28T16:44:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.168+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3680fb610 con 0x7fb368102800 2024-02-28T16:44:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.168+0000 7fb35f400700 1 -- 172.21.15.1:0/3004919972 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb35400bbf0 con 0x7fb368102800 2024-02-28T16:44:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.169+0000 7fb35f400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb34c06d7f0 0x7fb34c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.170+0000 7fb35f400700 1 -- 172.21.15.1:0/3004919972 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb35408e1d0 con 0x7fb368102800 2024-02-28T16:44:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.170+0000 7fb367400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb34c06d7f0 0x7fb34c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:46.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.170+0000 7fb367400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb34c06d7f0 0x7fb34c06fca0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb358004990 tx=0x7fb358009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:46.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.175+0000 7fb35f400700 1 -- 172.21.15.1:0/3004919972 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb35405b5b0 con 0x7fb368102800 2024-02-28T16:44:46.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:46 smithi001 ceph-mon[112940]: pgmap v309: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:46.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:46 smithi001 ceph-mon[112940]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:46.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:46 smithi001 ceph-mon[112940]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:46.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.492+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb3681112c0 con 0x7fb368102800 2024-02-28T16:44:46.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.493+0000 7fb35f400700 1 -- 172.21.15.1:0/3004919972 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fb35405b140 con 0x7fb368102800 2024-02-28T16:44:46.481 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:44:46.481 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:44:46.481 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:44:46.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:44:46.481 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:44:46.481 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:44:46.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:44:46.482 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:44:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb34c06d7f0 msgr2=0x7fb34c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb34c06d7f0 0x7fb34c06fca0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb358004990 tx=0x7fb358009ee0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 msgr2=0x7fb368116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 0x7fb368116220 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb354009100 tx=0x7fb354009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 shutdown_connections 2024-02-28T16:44:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb34c06d7f0 0x7fb34c06fca0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb368101e60 0x7fb368115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 --2- 172.21.15.1:0/3004919972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb368102800 0x7fb368116220 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 >> 172.21.15.1:0/3004919972 conn(0x7fb3680f7a10 msgr2=0x7fb368066070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 shutdown_connections 2024-02-28T16:44:46.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.496+0000 7fb36e400700 1 -- 172.21.15.1:0/3004919972 wait complete. 2024-02-28T16:44:46.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:46 smithi066 ceph-mon[85781]: pgmap v309: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:46.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:46 smithi066 ceph-mon[85781]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:46.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:46 smithi066 ceph-mon[85781]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:46.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.654+0000 7f0858e00700 1 -- 172.21.15.1:0/168835830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105b00 msgr2=0x7f08540ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:46.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.654+0000 7f0858e00700 1 --2- 172.21.15.1:0/168835830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105b00 0x7f08540ff170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0840004940 tx=0x7f0840009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:46.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.654+0000 7f0858e00700 1 -- 172.21.15.1:0/168835830 shutdown_connections 2024-02-28T16:44:46.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.654+0000 7f0858e00700 1 --2- 172.21.15.1:0/168835830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105b00 0x7f08540ff170 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.654+0000 7f0858e00700 1 --2- 172.21.15.1:0/168835830 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0854105160 0x7f0854105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.654+0000 7f0858e00700 1 -- 172.21.15.1:0/168835830 >> 172.21.15.1:0/168835830 conn(0x7f08540facb0 msgr2=0x7f08540fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:46.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.654+0000 7f0858e00700 1 -- 172.21.15.1:0/168835830 shutdown_connections 2024-02-28T16:44:46.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.654+0000 7f0858e00700 1 -- 172.21.15.1:0/168835830 wait complete. 2024-02-28T16:44:46.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.655+0000 7f0858e00700 1 Processor -- start 2024-02-28T16:44:46.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.655+0000 7f0858e00700 1 -- start start 2024-02-28T16:44:46.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.656+0000 7f0858e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105160 0x7f0854115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:46.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.656+0000 7f0858e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0854105b00 0x7f08541161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:46.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.656+0000 7f0858e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0854116820 con 0x7f0854105160 2024-02-28T16:44:46.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.656+0000 7f0858e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085410fb50 con 0x7f0854105b00 2024-02-28T16:44:46.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.656+0000 7f0851e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105160 0x7f0854115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.656+0000 7f0851e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105160 0x7f0854115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38816/0 (socket says 172.21.15.1:38816) 2024-02-28T16:44:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.656+0000 7f0851e00700 1 -- 172.21.15.1:0/363274001 learned_addr learned my addr 172.21.15.1:0/363274001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f0851400700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0854105b00 0x7f08541161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f0851e00700 1 -- 172.21.15.1:0/363274001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0854105b00 msgr2=0x7f08541161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f0851e00700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0854105b00 0x7f08541161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f0851e00700 1 -- 172.21.15.1:0/363274001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0840005040 con 0x7f0854105160 2024-02-28T16:44:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f0851400700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0854105b00 0x7f08541161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:44:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f0851e00700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105160 0x7f0854115c90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f084400c8c0 tx=0x7f08440044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f084a000700 1 -- 172.21.15.1:0/363274001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f084400a980 con 0x7f0854105160 2024-02-28T16:44:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f084a000700 1 -- 172.21.15.1:0/363274001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f084400ce60 con 0x7f0854105160 2024-02-28T16:44:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085410fe30 con 0x7f0854105160 2024-02-28T16:44:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.657+0000 7f084a000700 1 -- 172.21.15.1:0/363274001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08440115e0 con 0x7f0854105160 2024-02-28T16:44:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.658+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0854110370 con 0x7f0854105160 2024-02-28T16:44:46.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.659+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0854107630 con 0x7f0854105160 2024-02-28T16:44:46.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.660+0000 7f084a000700 1 -- 172.21.15.1:0/363274001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0844012400 con 0x7f0854105160 2024-02-28T16:44:46.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.661+0000 7f084a000700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f083806d7f0 0x7f083806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:46.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.661+0000 7f084a000700 1 -- 172.21.15.1:0/363274001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f084408cfe0 con 0x7f0854105160 2024-02-28T16:44:46.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.661+0000 7f0851400700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f083806d7f0 0x7f083806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.662+0000 7f0851400700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f083806d7f0 0x7f083806fca0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f084000b3b0 tx=0x7f0840009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:46.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.665+0000 7f084a000700 1 -- 172.21.15.1:0/363274001 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f084405a3a0 con 0x7f0854105160 2024-02-28T16:44:46.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.952+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f085404bed0 con 0x7f0854105160 2024-02-28T16:44:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.953+0000 7f084a000700 1 -- 172.21.15.1:0/363274001 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f08440126b0 con 0x7f0854105160 2024-02-28T16:44:46.941 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:44:46.941 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:44:46.941 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:44:46.941 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:44:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:44:46.942 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:44:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:44:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.956+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f083806d7f0 msgr2=0x7f083806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.956+0000 7f0858e00700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f083806d7f0 0x7f083806fca0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f084000b3b0 tx=0x7f0840009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.956+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105160 msgr2=0x7f0854115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.956+0000 7f0858e00700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105160 0x7f0854115c90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f084400c8c0 tx=0x7f08440044c0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.957+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 shutdown_connections 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.957+0000 7f0858e00700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f083806d7f0 0x7f083806fca0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.957+0000 7f0858e00700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0854105160 0x7f0854115c90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.957+0000 7f0858e00700 1 --2- 172.21.15.1:0/363274001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0854105b00 0x7f08541161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.957+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 >> 172.21.15.1:0/363274001 conn(0x7f08540facb0 msgr2=0x7f08540fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.957+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 shutdown_connections 2024-02-28T16:44:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:46.957+0000 7f0858e00700 1 -- 172.21.15.1:0/363274001 wait complete. 2024-02-28T16:44:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:44:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.104+0000 7f5f47c00700 1 -- 172.21.15.1:0/1897553854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f401095f0 msgr2=0x7f5f401099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.104+0000 7f5f47c00700 1 --2- 172.21.15.1:0/1897553854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f401095f0 0x7f5f401099c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5f34004940 tx=0x7f5f34009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.104+0000 7f5f47c00700 1 -- 172.21.15.1:0/1897553854 shutdown_connections 2024-02-28T16:44:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.104+0000 7f5f47c00700 1 --2- 172.21.15.1:0/1897553854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f40103dc0 0x7f5f40104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.104+0000 7f5f47c00700 1 --2- 172.21.15.1:0/1897553854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f401095f0 0x7f5f401099c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.104+0000 7f5f47c00700 1 -- 172.21.15.1:0/1897553854 >> 172.21.15.1:0/1897553854 conn(0x7f5f40072d50 msgr2=0x7f5f40075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.105+0000 7f5f47c00700 1 -- 172.21.15.1:0/1897553854 shutdown_connections 2024-02-28T16:44:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.105+0000 7f5f47c00700 1 -- 172.21.15.1:0/1897553854 wait complete. 2024-02-28T16:44:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.105+0000 7f5f47c00700 1 Processor -- start 2024-02-28T16:44:47.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.106+0000 7f5f47c00700 1 -- start start 2024-02-28T16:44:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.106+0000 7f5f47c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f40103dc0 0x7f5f4011a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.106+0000 7f5f47c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f4011a6b0 0x7f5f40114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.106+0000 7f5f47c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4011ac30 con 0x7f5f4011a6b0 2024-02-28T16:44:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.106+0000 7f5f47c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4011ada0 con 0x7f5f40103dc0 2024-02-28T16:44:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.106+0000 7f5f3fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f4011a6b0 0x7f5f40114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:47.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.106+0000 7f5f3fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f4011a6b0 0x7f5f40114080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38830/0 (socket says 172.21.15.1:38830) 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.106+0000 7f5f3fe00700 1 -- 172.21.15.1:0/662976132 learned_addr learned my addr 172.21.15.1:0/662976132 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f45200700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f40103dc0 0x7f5f4011a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f3fe00700 1 -- 172.21.15.1:0/662976132 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f40103dc0 msgr2=0x7f5f4011a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f3fe00700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f40103dc0 0x7f5f4011a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f3fe00700 1 -- 172.21.15.1:0/662976132 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34005040 con 0x7f5f4011a6b0 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f3fe00700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f4011a6b0 0x7f5f40114080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00c8c0 tx=0x7f5f2c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f3cc00700 1 -- 172.21.15.1:0/662976132 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f2c00a980 con 0x7f5f4011a6b0 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f3cc00700 1 -- 172.21.15.1:0/662976132 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f2c00ce60 con 0x7f5f4011a6b0 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40114680 con 0x7f5f4011a6b0 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f3cc00700 1 -- 172.21.15.1:0/662976132 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f2c0115e0 con 0x7f5f4011a6b0 2024-02-28T16:44:47.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.107+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f40114bc0 con 0x7f5f4011a6b0 2024-02-28T16:44:47.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.108+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f40114ea0 con 0x7f5f4011a6b0 2024-02-28T16:44:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.110+0000 7f5f3cc00700 1 -- 172.21.15.1:0/662976132 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5f2c012400 con 0x7f5f4011a6b0 2024-02-28T16:44:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.111+0000 7f5f3cc00700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5f2806d7f0 0x7f5f2806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.111+0000 7f5f3cc00700 1 -- 172.21.15.1:0/662976132 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5f2c08d0b0 con 0x7f5f4011a6b0 2024-02-28T16:44:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.111+0000 7f5f45200700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5f2806d7f0 0x7f5f2806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:47.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.112+0000 7f5f45200700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5f2806d7f0 0x7f5f2806fca0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5f34000c00 tx=0x7f5f34009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:47.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.115+0000 7f5f3cc00700 1 -- 172.21.15.1:0/662976132 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5f2c05a410 con 0x7f5f4011a6b0 2024-02-28T16:44:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.350+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f40115850 con 0x7f5f2806d7f0 2024-02-28T16:44:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.352+0000 7f5f3cc00700 1 -- 172.21.15.1:0/662976132 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5f34007d50 con 0x7f5f2806d7f0 2024-02-28T16:44:47.339 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:44:47.340 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:44:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.355+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5f2806d7f0 msgr2=0x7f5f2806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.355+0000 7f5f47c00700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5f2806d7f0 0x7f5f2806fca0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5f34000c00 tx=0x7f5f34009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.355+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f4011a6b0 msgr2=0x7f5f40114080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.355+0000 7f5f47c00700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f4011a6b0 0x7f5f40114080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00c8c0 tx=0x7f5f2c0044c0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.356+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 shutdown_connections 2024-02-28T16:44:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.356+0000 7f5f47c00700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5f2806d7f0 0x7f5f2806fca0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.356+0000 7f5f47c00700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5f40103dc0 0x7f5f4011a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.356+0000 7f5f47c00700 1 --2- 172.21.15.1:0/662976132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f4011a6b0 0x7f5f40114080 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.356+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 >> 172.21.15.1:0/662976132 conn(0x7f5f40072d50 msgr2=0x7f5f40074af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.356+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 shutdown_connections 2024-02-28T16:44:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.356+0000 7f5f47c00700 1 -- 172.21.15.1:0/662976132 wait complete. 2024-02-28T16:44:47.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:47 smithi066 ceph-mon[85781]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:47.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:47 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3004919972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:44:47.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:47 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/363274001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:44:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.511+0000 7fe171c00700 1 -- 172.21.15.1:0/979347642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c070870 msgr2=0x7fe16c070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.511+0000 7fe171c00700 1 --2- 172.21.15.1:0/979347642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c070870 0x7fe16c070ce0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe158004970 tx=0x7fe158009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.511+0000 7fe171c00700 1 -- 172.21.15.1:0/979347642 shutdown_connections 2024-02-28T16:44:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.511+0000 7fe171c00700 1 --2- 172.21.15.1:0/979347642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c070870 0x7fe16c070ce0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.511+0000 7fe171c00700 1 --2- 172.21.15.1:0/979347642 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe16c071f60 0x7fe16c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.511+0000 7fe171c00700 1 -- 172.21.15.1:0/979347642 >> 172.21.15.1:0/979347642 conn(0x7fe16c0f8ae0 msgr2=0x7fe16c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:47.499 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:47 smithi001 ceph-mon[112940]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:47.499 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:47 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3004919972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:44:47.499 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:47 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/363274001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:44:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.512+0000 7fe171c00700 1 -- 172.21.15.1:0/979347642 shutdown_connections 2024-02-28T16:44:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.512+0000 7fe171c00700 1 -- 172.21.15.1:0/979347642 wait complete. 2024-02-28T16:44:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.512+0000 7fe171c00700 1 Processor -- start 2024-02-28T16:44:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.513+0000 7fe171c00700 1 -- start start 2024-02-28T16:44:47.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.513+0000 7fe171c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe16c070870 0x7fe16c11a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.513+0000 7fe171c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c071f60 0x7fe16c11a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.513+0000 7fe171c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c11aba0 con 0x7fe16c071f60 2024-02-28T16:44:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.513+0000 7fe171c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c113eb0 con 0x7fe16c070870 2024-02-28T16:44:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.513+0000 7fe16a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c071f60 0x7fe16c11a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.513+0000 7fe16a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c071f60 0x7fe16c11a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38846/0 (socket says 172.21.15.1:38846) 2024-02-28T16:44:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.514+0000 7fe16a000700 1 -- 172.21.15.1:0/2353986527 learned_addr learned my addr 172.21.15.1:0/2353986527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:44:47.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.514+0000 7fe16aa00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe16c070870 0x7fe16c11a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.514+0000 7fe16a000700 1 -- 172.21.15.1:0/2353986527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe16c070870 msgr2=0x7fe16c11a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.514+0000 7fe16a000700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe16c070870 0x7fe16c11a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.514+0000 7fe16a000700 1 -- 172.21.15.1:0/2353986527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe158005040 con 0x7fe16c071f60 2024-02-28T16:44:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.514+0000 7fe16a000700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c071f60 0x7fe16c11a550 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe158004f50 tx=0x7fe158009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.515+0000 7fe162a00700 1 -- 172.21.15.1:0/2353986527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe15800e040 con 0x7fe16c071f60 2024-02-28T16:44:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.515+0000 7fe162a00700 1 -- 172.21.15.1:0/2353986527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe15800bc10 con 0x7fe16c071f60 2024-02-28T16:44:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.515+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c114130 con 0x7fe16c071f60 2024-02-28T16:44:47.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.515+0000 7fe162a00700 1 -- 172.21.15.1:0/2353986527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe15801c630 con 0x7fe16c071f60 2024-02-28T16:44:47.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.515+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c114670 con 0x7fe16c071f60 2024-02-28T16:44:47.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.516+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c0fe400 con 0x7fe16c071f60 2024-02-28T16:44:47.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.517+0000 7fe162a00700 1 -- 172.21.15.1:0/2353986527 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe15800bd80 con 0x7fe16c071f60 2024-02-28T16:44:47.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.518+0000 7fe162a00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe15006d7f0 0x7fe15006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:44:47.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.518+0000 7fe162a00700 1 -- 172.21.15.1:0/2353986527 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe15808e1e0 con 0x7fe16c071f60 2024-02-28T16:44:47.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.519+0000 7fe16aa00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe15006d7f0 0x7fe15006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:44:47.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.519+0000 7fe16aa00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe15006d7f0 0x7fe15006fca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe15c007fd0 tx=0x7fe15c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:44:47.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.523+0000 7fe162a00700 1 -- 172.21.15.1:0/2353986527 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe15805b680 con 0x7fe16c071f60 2024-02-28T16:44:47.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.837+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe16c0fc6e0 con 0x7fe16c071f60 2024-02-28T16:44:47.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.838+0000 7fe162a00700 1 -- 172.21.15.1:0/2353986527 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fe15805b210 con 0x7fe16c071f60 2024-02-28T16:44:47.825 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:44:47.825 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:44:47.825 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:44:47.825 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:44:47.825 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:44:47.825 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:44:47.825 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:44:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe15006d7f0 msgr2=0x7fe15006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe15006d7f0 0x7fe15006fca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe15c007fd0 tx=0x7fe15c009040 comp rx=0 tx=0).stop 2024-02-28T16:44:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c071f60 msgr2=0x7fe16c11a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:44:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c071f60 0x7fe16c11a550 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe158004f50 tx=0x7fe158009e30 comp rx=0 tx=0).stop 2024-02-28T16:44:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 shutdown_connections 2024-02-28T16:44:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe15006d7f0 0x7fe15006fca0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe16c070870 0x7fe16c11a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 --2- 172.21.15.1:0/2353986527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe16c071f60 0x7fe16c11a550 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:44:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 >> 172.21.15.1:0/2353986527 conn(0x7fe16c0f8ae0 msgr2=0x7fe16c0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:44:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 shutdown_connections 2024-02-28T16:44:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:44:47.841+0000 7fe171c00700 1 -- 172.21.15.1:0/2353986527 wait complete. 2024-02-28T16:44:48.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:48 smithi066 ceph-mon[85781]: pgmap v310: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:48.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:48 smithi066 ceph-mon[85781]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:48.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:48 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2353986527' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:44:48.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:48 smithi001 ceph-mon[112940]: pgmap v310: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:48.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:48 smithi001 ceph-mon[112940]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:44:48.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:48 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2353986527' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:44:50.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:50 smithi066 ceph-mon[85781]: pgmap v311: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:50.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:50 smithi001 ceph-mon[112940]: pgmap v311: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:52.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:52 smithi066 ceph-mon[85781]: pgmap v312: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:52.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:52 smithi001 ceph-mon[112940]: pgmap v312: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:54.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:54 smithi066 ceph-mon[85781]: pgmap v313: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:54.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:54 smithi001 ceph-mon[112940]: pgmap v313: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:56.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:56 smithi066 ceph-mon[85781]: pgmap v314: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:56.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:44:56.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:56 smithi001 ceph-mon[112940]: pgmap v314: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:56.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:44:58.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:44:58 smithi066 ceph-mon[85781]: pgmap v315: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:44:58.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:44:58 smithi001 ceph-mon[112940]: pgmap v315: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:00.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:00 smithi066 ceph-mon[85781]: pgmap v316: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:00.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:00 smithi001 ceph-mon[112940]: pgmap v316: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:02.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:02 smithi066 ceph-mon[85781]: pgmap v317: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:02.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:02 smithi001 ceph-mon[112940]: pgmap v317: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:04.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:04 smithi066 ceph-mon[85781]: pgmap v318: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:04.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:04 smithi001 ceph-mon[112940]: pgmap v318: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:06.504 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:06 smithi001 ceph-mon[112940]: pgmap v319: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:06.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:06 smithi066 ceph-mon[85781]: pgmap v319: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:08.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:08 smithi001 ceph-mon[112940]: pgmap v320: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:08.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:08 smithi066 ceph-mon[85781]: pgmap v320: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:10.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:10 smithi001 ceph-mon[112940]: pgmap v321: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:10.549 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:10 smithi066 ceph-mon[85781]: pgmap v321: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:11.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:45:11.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:45:12.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:12 smithi066 ceph-mon[85781]: pgmap v322: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:12.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:12 smithi001 ceph-mon[112940]: pgmap v322: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:14.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:14 smithi066 ceph-mon[85781]: pgmap v323: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:14.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:14 smithi001 ceph-mon[112940]: pgmap v323: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:16.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:16 smithi066 ceph-mon[85781]: pgmap v324: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:16.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:16 smithi001 ceph-mon[112940]: pgmap v324: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:18.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.017+0000 7f9d50400700 1 -- 172.21.15.1:0/491339289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48070870 msgr2=0x7f9d48070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.017+0000 7f9d50400700 1 --2- 172.21.15.1:0/491339289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48070870 0x7f9d48070ce0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d34004970 tx=0x7f9d34009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:18.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.017+0000 7f9d50400700 1 -- 172.21.15.1:0/491339289 shutdown_connections 2024-02-28T16:45:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.017+0000 7f9d50400700 1 --2- 172.21.15.1:0/491339289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48070870 0x7f9d48070ce0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.017+0000 7f9d50400700 1 --2- 172.21.15.1:0/491339289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d48071f60 0x7f9d48070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.017+0000 7f9d50400700 1 -- 172.21.15.1:0/491339289 >> 172.21.15.1:0/491339289 conn(0x7f9d480f8ae0 msgr2=0x7f9d480faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.018+0000 7f9d50400700 1 -- 172.21.15.1:0/491339289 shutdown_connections 2024-02-28T16:45:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.018+0000 7f9d50400700 1 -- 172.21.15.1:0/491339289 wait complete. 2024-02-28T16:45:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.019+0000 7f9d50400700 1 Processor -- start 2024-02-28T16:45:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.019+0000 7f9d50400700 1 -- start start 2024-02-28T16:45:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.019+0000 7f9d50400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d48070870 0x7f9d4811a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.019+0000 7f9d50400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48071f60 0x7f9d4811a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.020+0000 7f9d50400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4811aba0 con 0x7f9d48071f60 2024-02-28T16:45:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.020+0000 7f9d50400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d48113eb0 con 0x7f9d48070870 2024-02-28T16:45:18.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.020+0000 7f9d4d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48071f60 0x7f9d4811a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.020+0000 7f9d4d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48071f60 0x7f9d4811a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53428/0 (socket says 172.21.15.1:53428) 2024-02-28T16:45:18.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.020+0000 7f9d4d000700 1 -- 172.21.15.1:0/1798913313 learned_addr learned my addr 172.21.15.1:0/1798913313 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:18.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.020+0000 7f9d4da00700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d48070870 0x7f9d4811a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.020+0000 7f9d4d000700 1 -- 172.21.15.1:0/1798913313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d48070870 msgr2=0x7f9d4811a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.020+0000 7f9d4d000700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d48070870 0x7f9d4811a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.021+0000 7f9d4d000700 1 -- 172.21.15.1:0/1798913313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34005040 con 0x7f9d48071f60 2024-02-28T16:45:18.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.021+0000 7f9d4da00700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d48070870 0x7f9d4811a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:45:18.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.021+0000 7f9d4d000700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48071f60 0x7f9d4811a550 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9d34004f50 tx=0x7f9d34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:18.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.021+0000 7f9d45600700 1 -- 172.21.15.1:0/1798913313 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d3400e040 con 0x7f9d48071f60 2024-02-28T16:45:18.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.021+0000 7f9d45600700 1 -- 172.21.15.1:0/1798913313 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d3400bc10 con 0x7f9d48071f60 2024-02-28T16:45:18.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.021+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d48114130 con 0x7f9d48071f60 2024-02-28T16:45:18.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.021+0000 7f9d45600700 1 -- 172.21.15.1:0/1798913313 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d3401c630 con 0x7f9d48071f60 2024-02-28T16:45:18.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.022+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d48114670 con 0x7f9d48071f60 2024-02-28T16:45:18.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.023+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d480fe400 con 0x7f9d48071f60 2024-02-28T16:45:18.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.024+0000 7f9d45600700 1 -- 172.21.15.1:0/1798913313 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9d3400bd80 con 0x7f9d48071f60 2024-02-28T16:45:18.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.025+0000 7f9d45600700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d3006d7f0 0x7f9d3006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.025+0000 7f9d45600700 1 -- 172.21.15.1:0/1798913313 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9d3408e2a0 con 0x7f9d48071f60 2024-02-28T16:45:18.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.026+0000 7f9d4da00700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d3006d7f0 0x7f9d3006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.026+0000 7f9d4da00700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d3006d7f0 0x7f9d3006fca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c007fd0 tx=0x7f9d3c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:18.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.031+0000 7f9d45600700 1 -- 172.21.15.1:0/1798913313 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d3405b680 con 0x7f9d48071f60 2024-02-28T16:45:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.278+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d4804bf40 con 0x7f9d3006d7f0 2024-02-28T16:45:18.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.282+0000 7f9d45600700 1 -- 172.21.15.1:0/1798913313 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9d4804bf40 con 0x7f9d3006d7f0 2024-02-28T16:45:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d3006d7f0 msgr2=0x7f9d3006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d3006d7f0 0x7f9d3006fca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c007fd0 tx=0x7f9d3c009040 comp rx=0 tx=0).stop 2024-02-28T16:45:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48071f60 msgr2=0x7f9d4811a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48071f60 0x7f9d4811a550 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9d34004f50 tx=0x7f9d34009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 shutdown_connections 2024-02-28T16:45:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9d3006d7f0 0x7f9d3006fca0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9d48070870 0x7f9d4811a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 --2- 172.21.15.1:0/1798913313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d48071f60 0x7f9d4811a550 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 >> 172.21.15.1:0/1798913313 conn(0x7f9d480f8ae0 msgr2=0x7f9d480fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.286+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 shutdown_connections 2024-02-28T16:45:18.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.287+0000 7f9d50400700 1 -- 172.21.15.1:0/1798913313 wait complete. 2024-02-28T16:45:18.289 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:45:18.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.447+0000 7fd3dfa00700 1 -- 172.21.15.1:0/508058154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8103dc0 msgr2=0x7fd3d8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.447+0000 7fd3dfa00700 1 --2- 172.21.15.1:0/508058154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8103dc0 0x7fd3d8104210 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4004940 tx=0x7fd3c4009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:18.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.448+0000 7fd3dfa00700 1 -- 172.21.15.1:0/508058154 shutdown_connections 2024-02-28T16:45:18.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.448+0000 7fd3dfa00700 1 --2- 172.21.15.1:0/508058154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d8103dc0 0x7fd3d8104210 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.448+0000 7fd3dfa00700 1 --2- 172.21.15.1:0/508058154 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d81095f0 0x7fd3d81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.448+0000 7fd3dfa00700 1 -- 172.21.15.1:0/508058154 >> 172.21.15.1:0/508058154 conn(0x7fd3d8072d50 msgr2=0x7fd3d8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:18.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.448+0000 7fd3dfa00700 1 -- 172.21.15.1:0/508058154 shutdown_connections 2024-02-28T16:45:18.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.448+0000 7fd3dfa00700 1 -- 172.21.15.1:0/508058154 wait complete. 2024-02-28T16:45:18.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.449+0000 7fd3dfa00700 1 Processor -- start 2024-02-28T16:45:18.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.449+0000 7fd3dfa00700 1 -- start start 2024-02-28T16:45:18.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.450+0000 7fd3dfa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d8103dc0 0x7fd3d811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.450+0000 7fd3dfa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d81095f0 0x7fd3d811a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.450+0000 7fd3dfa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d811abe0 con 0x7fd3d81095f0 2024-02-28T16:45:18.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.450+0000 7fd3dfa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d8113ef0 con 0x7fd3d8103dc0 2024-02-28T16:45:18.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.450+0000 7fd3d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d81095f0 0x7fd3d811a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.450+0000 7fd3d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d81095f0 0x7fd3d811a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53436/0 (socket says 172.21.15.1:53436) 2024-02-28T16:45:18.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.450+0000 7fd3d7e00700 1 -- 172.21.15.1:0/3512796125 learned_addr learned my addr 172.21.15.1:0/3512796125 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:18.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.450+0000 7fd3dd000700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d8103dc0 0x7fd3d811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.451+0000 7fd3d7e00700 1 -- 172.21.15.1:0/3512796125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d8103dc0 msgr2=0x7fd3d811a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.451+0000 7fd3d7e00700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d8103dc0 0x7fd3d811a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.451+0000 7fd3d7e00700 1 -- 172.21.15.1:0/3512796125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c4005040 con 0x7fd3d81095f0 2024-02-28T16:45:18.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.451+0000 7fd3d7e00700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d81095f0 0x7fd3d811a590 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009150 tx=0x7fd3c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:18.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.451+0000 7fd3d4c00700 1 -- 172.21.15.1:0/3512796125 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3c400e040 con 0x7fd3d81095f0 2024-02-28T16:45:18.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.451+0000 7fd3d4c00700 1 -- 172.21.15.1:0/3512796125 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3c400bbf0 con 0x7fd3d81095f0 2024-02-28T16:45:18.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.451+0000 7fd3d4c00700 1 -- 172.21.15.1:0/3512796125 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3c401c3f0 con 0x7fd3d81095f0 2024-02-28T16:45:18.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.451+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d8114170 con 0x7fd3d81095f0 2024-02-28T16:45:18.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.452+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d81146b0 con 0x7fd3d81095f0 2024-02-28T16:45:18.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.453+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d810b9c0 con 0x7fd3d81095f0 2024-02-28T16:45:18.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.454+0000 7fd3d4c00700 1 -- 172.21.15.1:0/3512796125 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd3c400bd60 con 0x7fd3d81095f0 2024-02-28T16:45:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.455+0000 7fd3d4c00700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3c006d7f0 0x7fd3c006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.455+0000 7fd3d4c00700 1 -- 172.21.15.1:0/3512796125 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd3c408e130 con 0x7fd3d81095f0 2024-02-28T16:45:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.455+0000 7fd3dd000700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3c006d7f0 0x7fd3c006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.456+0000 7fd3dd000700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3c006d7f0 0x7fd3c006fca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc004f60 tx=0x7fd3cc00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:18.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.460+0000 7fd3d4c00700 1 -- 172.21.15.1:0/3512796125 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd3c405b510 con 0x7fd3d81095f0 2024-02-28T16:45:18.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:18 smithi001 ceph-mon[112940]: pgmap v325: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.704+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3d804bed0 con 0x7fd3c006d7f0 2024-02-28T16:45:18.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.706+0000 7fd3d4c00700 1 -- 172.21.15.1:0/3512796125 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd3d804bed0 con 0x7fd3c006d7f0 2024-02-28T16:45:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3c006d7f0 msgr2=0x7fd3c006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3c006d7f0 0x7fd3c006fca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc004f60 tx=0x7fd3cc00b040 comp rx=0 tx=0).stop 2024-02-28T16:45:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d81095f0 msgr2=0x7fd3d811a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d81095f0 0x7fd3d811a590 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4009150 tx=0x7fd3c4009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 shutdown_connections 2024-02-28T16:45:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3c006d7f0 0x7fd3c006fca0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3d8103dc0 0x7fd3d811a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 --2- 172.21.15.1:0/3512796125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3d81095f0 0x7fd3d811a590 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 >> 172.21.15.1:0/3512796125 conn(0x7fd3d8072d50 msgr2=0x7fd3d8074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 shutdown_connections 2024-02-28T16:45:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.709+0000 7fd3dfa00700 1 -- 172.21.15.1:0/3512796125 wait complete. 2024-02-28T16:45:18.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:18 smithi066 ceph-mon[85781]: pgmap v325: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:18.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.872+0000 7f4aafc00700 1 -- 172.21.15.1:0/1432705571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 msgr2=0x7f4aa80fc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.872+0000 7f4aafc00700 1 --2- 172.21.15.1:0/1432705571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 0x7f4aa80fc100 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c004940 tx=0x7f4a9c009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:18.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.872+0000 7f4aafc00700 1 -- 172.21.15.1:0/1432705571 shutdown_connections 2024-02-28T16:45:18.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.872+0000 7f4aafc00700 1 --2- 172.21.15.1:0/1432705571 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4aa80fc6d0 0x7f4aa8100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.872+0000 7f4aafc00700 1 --2- 172.21.15.1:0/1432705571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 0x7f4aa80fc100 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.872+0000 7f4aafc00700 1 -- 172.21.15.1:0/1432705571 >> 172.21.15.1:0/1432705571 conn(0x7f4aa80f79d0 msgr2=0x7f4aa80f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:18.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.872+0000 7f4aafc00700 1 -- 172.21.15.1:0/1432705571 shutdown_connections 2024-02-28T16:45:18.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.872+0000 7f4aafc00700 1 -- 172.21.15.1:0/1432705571 wait complete. 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.873+0000 7f4aafc00700 1 Processor -- start 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.873+0000 7f4aafc00700 1 -- start start 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aafc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 0x7f4aa811a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aafc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4aa80fc6d0 0x7f4aa811a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aafc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa811ac20 con 0x7f4aa80fbd30 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aafc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8113f30 con 0x7f4aa80fc6d0 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aad200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 0x7f4aa811a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aad200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 0x7f4aa811a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53464/0 (socket says 172.21.15.1:53464) 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aad200700 1 -- 172.21.15.1:0/541153477 learned_addr learned my addr 172.21.15.1:0/541153477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aa7e00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4aa80fc6d0 0x7f4aa811a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aad200700 1 -- 172.21.15.1:0/541153477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4aa80fc6d0 msgr2=0x7f4aa811a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:18.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aad200700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4aa80fc6d0 0x7f4aa811a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.874+0000 7f4aad200700 1 -- 172.21.15.1:0/541153477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c005040 con 0x7f4aa80fbd30 2024-02-28T16:45:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.875+0000 7f4aad200700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 0x7f4aa811a090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009150 tx=0x7f4a9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.875+0000 7f4aa4c00700 1 -- 172.21.15.1:0/541153477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a9c00e040 con 0x7f4aa80fbd30 2024-02-28T16:45:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.875+0000 7f4aa4c00700 1 -- 172.21.15.1:0/541153477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a9c00bbf0 con 0x7f4aa80fbd30 2024-02-28T16:45:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.875+0000 7f4aa4c00700 1 -- 172.21.15.1:0/541153477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a9c01c630 con 0x7f4aa80fbd30 2024-02-28T16:45:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.875+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa81141b0 con 0x7f4aa80fbd30 2024-02-28T16:45:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.875+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa81146f0 con 0x7f4aa80fbd30 2024-02-28T16:45:18.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.876+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa80fe410 con 0x7f4aa80fbd30 2024-02-28T16:45:18.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.877+0000 7f4aa4c00700 1 -- 172.21.15.1:0/541153477 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4a9c00bd60 con 0x7f4aa80fbd30 2024-02-28T16:45:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.878+0000 7f4aa4c00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a9006d7f0 0x7f4a9006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.878+0000 7f4aa4c00700 1 -- 172.21.15.1:0/541153477 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4a9c08e380 con 0x7f4aa80fbd30 2024-02-28T16:45:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.878+0000 7f4aa7e00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a9006d7f0 0x7f4a9006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:18.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.879+0000 7f4aa7e00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a9006d7f0 0x7f4a9006fca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8115930 tx=0x7f4a94009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:18.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:18.883+0000 7f4aa4c00700 1 -- 172.21.15.1:0/541153477 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a9c05b6e0 con 0x7f4aa80fbd30 2024-02-28T16:45:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.122+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4aa804bf40 con 0x7f4a9006d7f0 2024-02-28T16:45:19.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.152+0000 7f4aa4c00700 1 -- 172.21.15.1:0/541153477 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4aa804bf40 con 0x7f4a9006d7f0 2024-02-28T16:45:19.139 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:45:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (11m) 3m ago 25m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:45:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (8m) 3m ago 26m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:45:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (8m) 4m ago 24m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:45:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (11m) 3m ago 25m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:45:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (4m) 3m ago 18m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:45:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (3m) 3m ago 18m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:45:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (18m) 4m ago 18m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:45:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (18m) 4m ago 18m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (13m) 3m ago 27m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (12m) 4m ago 24m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (9m) 3m ago 27m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (9m) 4m ago 24m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (12m) 3m ago 25m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (12m) 4m ago 24m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (7m) 3m ago 23m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (6m) 3m ago 22m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (6m) 3m ago 22m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (5m) 4m ago 21m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (5m) 4m ago 21m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (4m) 4m ago 20m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (11m) 3m ago 25m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:45:19.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a9006d7f0 msgr2=0x7f4a9006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:19.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a9006d7f0 0x7f4a9006fca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8115930 tx=0x7f4a94009040 comp rx=0 tx=0).stop 2024-02-28T16:45:19.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 msgr2=0x7f4aa811a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:19.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 0x7f4aa811a090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009150 tx=0x7f4a9c009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 shutdown_connections 2024-02-28T16:45:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a9006d7f0 0x7f4a9006fca0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4aa80fbd30 0x7f4aa811a090 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 --2- 172.21.15.1:0/541153477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4aa80fc6d0 0x7f4aa811a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.155+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 >> 172.21.15.1:0/541153477 conn(0x7f4aa80f79d0 msgr2=0x7f4aa810d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.156+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 shutdown_connections 2024-02-28T16:45:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.156+0000 7f4aafc00700 1 -- 172.21.15.1:0/541153477 wait complete. 2024-02-28T16:45:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.319+0000 7f15e5800700 1 -- 172.21.15.1:0/1053549356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0070870 msgr2=0x7f15e0070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.319+0000 7f15e5800700 1 --2- 172.21.15.1:0/1053549356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0070870 0x7f15e0070ce0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f15cc004970 tx=0x7f15cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.319+0000 7f15e5800700 1 -- 172.21.15.1:0/1053549356 shutdown_connections 2024-02-28T16:45:19.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.319+0000 7f15e5800700 1 --2- 172.21.15.1:0/1053549356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0070870 0x7f15e0070ce0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.319+0000 7f15e5800700 1 --2- 172.21.15.1:0/1053549356 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15e0071f60 0x7f15e0070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.319+0000 7f15e5800700 1 -- 172.21.15.1:0/1053549356 >> 172.21.15.1:0/1053549356 conn(0x7f15e00f88b0 msgr2=0x7f15e00facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.319+0000 7f15e5800700 1 -- 172.21.15.1:0/1053549356 shutdown_connections 2024-02-28T16:45:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.319+0000 7f15e5800700 1 -- 172.21.15.1:0/1053549356 wait complete. 2024-02-28T16:45:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.320+0000 7f15e5800700 1 Processor -- start 2024-02-28T16:45:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.321+0000 7f15e5800700 1 -- start start 2024-02-28T16:45:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.321+0000 7f15e5800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15e0070870 0x7f15e0119dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.321+0000 7f15e5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0071f60 0x7f15e011a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.321+0000 7f15e5800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e011a950 con 0x7f15e0071f60 2024-02-28T16:45:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.321+0000 7f15e5800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e0113c60 con 0x7f15e0070870 2024-02-28T16:45:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.321+0000 7f15df400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0071f60 0x7f15e011a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.322+0000 7f15df400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0071f60 0x7f15e011a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53488/0 (socket says 172.21.15.1:53488) 2024-02-28T16:45:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.322+0000 7f15df400700 1 -- 172.21.15.1:0/731696542 learned_addr learned my addr 172.21.15.1:0/731696542 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.322+0000 7f15dfe00700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15e0070870 0x7f15e0119dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.322+0000 7f15df400700 1 -- 172.21.15.1:0/731696542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15e0070870 msgr2=0x7f15e0119dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.322+0000 7f15df400700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15e0070870 0x7f15e0119dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.322+0000 7f15df400700 1 -- 172.21.15.1:0/731696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15cc005040 con 0x7f15e0071f60 2024-02-28T16:45:19.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.322+0000 7f15df400700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0071f60 0x7f15e011a300 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f15cc004f50 tx=0x7f15cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.323+0000 7f15d7e00700 1 -- 172.21.15.1:0/731696542 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15cc00e040 con 0x7f15e0071f60 2024-02-28T16:45:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.323+0000 7f15d7e00700 1 -- 172.21.15.1:0/731696542 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15cc00bc10 con 0x7f15e0071f60 2024-02-28T16:45:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.323+0000 7f15d7e00700 1 -- 172.21.15.1:0/731696542 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15cc01c630 con 0x7f15e0071f60 2024-02-28T16:45:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.323+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e0113ee0 con 0x7f15e0071f60 2024-02-28T16:45:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.323+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e0114420 con 0x7f15e0071f60 2024-02-28T16:45:19.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.324+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e00fe1f0 con 0x7f15e0071f60 2024-02-28T16:45:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.325+0000 7f15d7e00700 1 -- 172.21.15.1:0/731696542 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f15cc00bd80 con 0x7f15e0071f60 2024-02-28T16:45:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.326+0000 7f15d7e00700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15c406d7f0 0x7f15c406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.326+0000 7f15d7e00700 1 -- 172.21.15.1:0/731696542 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f15cc08e280 con 0x7f15e0071f60 2024-02-28T16:45:19.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.327+0000 7f15dfe00700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15c406d7f0 0x7f15c406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:19.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.327+0000 7f15dfe00700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15c406d7f0 0x7f15c406fca0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15d0004990 tx=0x7f15d0009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.331+0000 7f15d7e00700 1 -- 172.21.15.1:0/731696542 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f15cc05b660 con 0x7f15e0071f60 2024-02-28T16:45:19.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.645+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f15e01153a0 con 0x7f15e0071f60 2024-02-28T16:45:19.632 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:19 smithi001 ceph-mon[112940]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:19.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.646+0000 7f15d7e00700 1 -- 172.21.15.1:0/731696542 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f15cc05b1f0 con 0x7f15e0071f60 2024-02-28T16:45:19.633 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:45:19.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:45:19.633 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:45:19.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:45:19.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:45:19.633 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:45:19.634 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15c406d7f0 msgr2=0x7f15c406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15c406d7f0 0x7f15c406fca0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15d0004990 tx=0x7f15d0009ee0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0071f60 msgr2=0x7f15e011a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0071f60 0x7f15e011a300 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f15cc004f50 tx=0x7f15cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 shutdown_connections 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15c406d7f0 0x7f15c406fca0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15e0070870 0x7f15e0119dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 --2- 172.21.15.1:0/731696542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e0071f60 0x7f15e011a300 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 >> 172.21.15.1:0/731696542 conn(0x7f15e00f88b0 msgr2=0x7f15e00fa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:19.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 shutdown_connections 2024-02-28T16:45:19.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.649+0000 7f15e5800700 1 -- 172.21.15.1:0/731696542 wait complete. 2024-02-28T16:45:19.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:19 smithi066 ceph-mon[85781]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.806+0000 7fe284c00700 1 -- 172.21.15.1:0/1356238960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2801095f0 msgr2=0x7fe2801099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.806+0000 7fe284c00700 1 --2- 172.21.15.1:0/1356238960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2801095f0 0x7fe2801099c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe270004940 tx=0x7fe270009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.807+0000 7fe284c00700 1 -- 172.21.15.1:0/1356238960 shutdown_connections 2024-02-28T16:45:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.807+0000 7fe284c00700 1 --2- 172.21.15.1:0/1356238960 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe280103dc0 0x7fe280104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.807+0000 7fe284c00700 1 --2- 172.21.15.1:0/1356238960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2801095f0 0x7fe2801099c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.807+0000 7fe284c00700 1 -- 172.21.15.1:0/1356238960 >> 172.21.15.1:0/1356238960 conn(0x7fe280072d50 msgr2=0x7fe280075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.807+0000 7fe284c00700 1 -- 172.21.15.1:0/1356238960 shutdown_connections 2024-02-28T16:45:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.807+0000 7fe284c00700 1 -- 172.21.15.1:0/1356238960 wait complete. 2024-02-28T16:45:19.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.808+0000 7fe284c00700 1 Processor -- start 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.808+0000 7fe284c00700 1 -- start start 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.808+0000 7fe284c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe280103dc0 0x7fe28011a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.808+0000 7fe284c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2801095f0 0x7fe28011a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.808+0000 7fe284c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28011aba0 con 0x7fe280103dc0 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.808+0000 7fe284c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280113eb0 con 0x7fe2801095f0 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.809+0000 7fe27de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe280103dc0 0x7fe28011a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.809+0000 7fe27de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe280103dc0 0x7fe28011a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53516/0 (socket says 172.21.15.1:53516) 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.809+0000 7fe27de00700 1 -- 172.21.15.1:0/137394704 learned_addr learned my addr 172.21.15.1:0/137394704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.809+0000 7fe27d400700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2801095f0 0x7fe28011a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.809+0000 7fe27de00700 1 -- 172.21.15.1:0/137394704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2801095f0 msgr2=0x7fe28011a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.809+0000 7fe27de00700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2801095f0 0x7fe28011a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:19.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.809+0000 7fe27de00700 1 -- 172.21.15.1:0/137394704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270005040 con 0x7fe280103dc0 2024-02-28T16:45:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.809+0000 7fe27de00700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe280103dc0 0x7fe28011a010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe270000c00 tx=0x7fe270009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.810+0000 7fe276000700 1 -- 172.21.15.1:0/137394704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe270021070 con 0x7fe280103dc0 2024-02-28T16:45:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.810+0000 7fe276000700 1 -- 172.21.15.1:0/137394704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe270013dd0 con 0x7fe280103dc0 2024-02-28T16:45:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.810+0000 7fe276000700 1 -- 172.21.15.1:0/137394704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe27000e070 con 0x7fe280103dc0 2024-02-28T16:45:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.810+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe280114130 con 0x7fe280103dc0 2024-02-28T16:45:19.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.810+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe280114610 con 0x7fe280103dc0 2024-02-28T16:45:19.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.811+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe280075870 con 0x7fe280103dc0 2024-02-28T16:45:19.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.812+0000 7fe276000700 1 -- 172.21.15.1:0/137394704 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe27001c400 con 0x7fe280103dc0 2024-02-28T16:45:19.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.813+0000 7fe276000700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe26406d7f0 0x7fe26406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.813+0000 7fe276000700 1 -- 172.21.15.1:0/137394704 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe27008e380 con 0x7fe280103dc0 2024-02-28T16:45:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.813+0000 7fe27d400700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe26406d7f0 0x7fe26406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:19.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.814+0000 7fe27d400700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe26406d7f0 0x7fe26406fca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe26c004600 tx=0x7fe26c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:19.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:19.818+0000 7fe276000700 1 -- 172.21.15.1:0/137394704 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe27005b760 con 0x7fe280103dc0 2024-02-28T16:45:20.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.081+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe280115690 con 0x7fe280103dc0 2024-02-28T16:45:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.082+0000 7fe276000700 1 -- 172.21.15.1:0/137394704 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fe27005b2f0 con 0x7fe280103dc0 2024-02-28T16:45:20.069 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:45:20.070 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:45:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:45:20.072 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:45:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe26406d7f0 msgr2=0x7fe26406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe26406d7f0 0x7fe26406fca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe26c004600 tx=0x7fe26c00b040 comp rx=0 tx=0).stop 2024-02-28T16:45:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe280103dc0 msgr2=0x7fe28011a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe280103dc0 0x7fe28011a010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe270000c00 tx=0x7fe270009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 shutdown_connections 2024-02-28T16:45:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe26406d7f0 0x7fe26406fca0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe280103dc0 0x7fe28011a010 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 --2- 172.21.15.1:0/137394704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe2801095f0 0x7fe28011a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.085+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 >> 172.21.15.1:0/137394704 conn(0x7fe280072d50 msgr2=0x7fe2800746d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:20.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.086+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 shutdown_connections 2024-02-28T16:45:20.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.086+0000 7fe284c00700 1 -- 172.21.15.1:0/137394704 wait complete. 2024-02-28T16:45:20.074 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:45:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.237+0000 7f9723600700 1 -- 172.21.15.1:0/137298362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 msgr2=0x7f971c104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.237+0000 7f9723600700 1 --2- 172.21.15.1:0/137298362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 0x7f971c104230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9708004940 tx=0x7f9708009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.237+0000 7f9723600700 1 -- 172.21.15.1:0/137298362 shutdown_connections 2024-02-28T16:45:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.237+0000 7f9723600700 1 --2- 172.21.15.1:0/137298362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 0x7f971c104230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.237+0000 7f9723600700 1 --2- 172.21.15.1:0/137298362 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f971c1095b0 0x7f971c109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.237+0000 7f9723600700 1 -- 172.21.15.1:0/137298362 >> 172.21.15.1:0/137298362 conn(0x7f971c072d70 msgr2=0x7f971c075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.237+0000 7f9723600700 1 -- 172.21.15.1:0/137298362 shutdown_connections 2024-02-28T16:45:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.237+0000 7f9723600700 1 -- 172.21.15.1:0/137298362 wait complete. 2024-02-28T16:45:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.238+0000 7f9723600700 1 Processor -- start 2024-02-28T16:45:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.238+0000 7f9723600700 1 -- start start 2024-02-28T16:45:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.239+0000 7f9723600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 0x7f971c11a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.239+0000 7f9723600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f971c1095b0 0x7f971c11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.239+0000 7f9723600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c11ac00 con 0x7f971c103dc0 2024-02-28T16:45:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.239+0000 7f9723600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c113f10 con 0x7f971c1095b0 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.239+0000 7f9720c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 0x7f971c11a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.239+0000 7f9720c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 0x7f971c11a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53530/0 (socket says 172.21.15.1:53530) 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.239+0000 7f9720c00700 1 -- 172.21.15.1:0/2744126968 learned_addr learned my addr 172.21.15.1:0/2744126968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.239+0000 7f971be00700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f971c1095b0 0x7f971c11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9720c00700 1 -- 172.21.15.1:0/2744126968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f971c1095b0 msgr2=0x7f971c11a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9720c00700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f971c1095b0 0x7f971c11a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9720c00700 1 -- 172.21.15.1:0/2744126968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9708005040 con 0x7f971c103dc0 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9720c00700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 0x7f971c11a070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f971000c8b0 tx=0x7f9710004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9718c00700 1 -- 172.21.15.1:0/2744126968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f971000a760 con 0x7f971c103dc0 2024-02-28T16:45:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9718c00700 1 -- 172.21.15.1:0/2744126968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f971000ad90 con 0x7f971c103dc0 2024-02-28T16:45:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c1141f0 con 0x7f971c103dc0 2024-02-28T16:45:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9718c00700 1 -- 172.21.15.1:0/2744126968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97100113a0 con 0x7f971c103dc0 2024-02-28T16:45:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.240+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c114730 con 0x7f971c103dc0 2024-02-28T16:45:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.242+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971c10b990 con 0x7f971c103dc0 2024-02-28T16:45:20.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.243+0000 7f9718c00700 1 -- 172.21.15.1:0/2744126968 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9710012400 con 0x7f971c103dc0 2024-02-28T16:45:20.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.244+0000 7f9718c00700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f970406d7f0 0x7f970406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:20.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.244+0000 7f9718c00700 1 -- 172.21.15.1:0/2744126968 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f971008ce60 con 0x7f971c103dc0 2024-02-28T16:45:20.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.245+0000 7f971be00700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f970406d7f0 0x7f970406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:20.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.245+0000 7f971be00700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f970406d7f0 0x7f970406fca0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f970800b020 tx=0x7f9708009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:20.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.249+0000 7f9718c00700 1 -- 172.21.15.1:0/2744126968 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f971005a240 con 0x7f971c103dc0 2024-02-28T16:45:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.495+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f971c115650 con 0x7f970406d7f0 2024-02-28T16:45:20.482 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:20 smithi001 ceph-mon[112940]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:20.482 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:20 smithi001 ceph-mon[112940]: pgmap v326: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:20.482 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:20 smithi001 ceph-mon[112940]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:20.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:20 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/731696542' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:45:20.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:20 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/137394704' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:45:20.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.497+0000 7f9718c00700 1 -- 172.21.15.1:0/2744126968 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f971c115650 con 0x7f970406d7f0 2024-02-28T16:45:20.484 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:45:20.484 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:45:20.484 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:45:20.484 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:45:20.484 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:45:20.484 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:45:20.484 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:45:20.485 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:45:20.485 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:45:20.485 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:45:20.485 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:45:20.485 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:45:20.485 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:45:20.485 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:45:20.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f970406d7f0 msgr2=0x7f970406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f970406d7f0 0x7f970406fca0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f970800b020 tx=0x7f9708009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:20.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 msgr2=0x7f971c11a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 0x7f971c11a070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f971000c8b0 tx=0x7f9710004470 comp rx=0 tx=0).stop 2024-02-28T16:45:20.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 shutdown_connections 2024-02-28T16:45:20.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f970406d7f0 0x7f970406fca0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971c103dc0 0x7f971c11a070 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 --2- 172.21.15.1:0/2744126968 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f971c1095b0 0x7f971c11a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.500+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 >> 172.21.15.1:0/2744126968 conn(0x7f971c072d70 msgr2=0x7f971c074960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:20.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.501+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 shutdown_connections 2024-02-28T16:45:20.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.501+0000 7f9723600700 1 -- 172.21.15.1:0/2744126968 wait complete. 2024-02-28T16:45:20.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.661+0000 7f0f53000700 1 -- 172.21.15.1:0/3853259235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 msgr2=0x7f0f4c0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.661+0000 7f0f53000700 1 --2- 172.21.15.1:0/3853259235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 0x7f0f4c0fe3b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c004970 tx=0x7f0f3c009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.661+0000 7f0f53000700 1 -- 172.21.15.1:0/3853259235 shutdown_connections 2024-02-28T16:45:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.661+0000 7f0f53000700 1 --2- 172.21.15.1:0/3853259235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f4c0fe8f0 0x7f0f4c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.661+0000 7f0f53000700 1 --2- 172.21.15.1:0/3853259235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 0x7f0f4c0fe3b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.661+0000 7f0f53000700 1 -- 172.21.15.1:0/3853259235 >> 172.21.15.1:0/3853259235 conn(0x7f0f4c071d90 msgr2=0x7f0f4c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.661+0000 7f0f53000700 1 -- 172.21.15.1:0/3853259235 shutdown_connections 2024-02-28T16:45:20.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.662+0000 7f0f53000700 1 -- 172.21.15.1:0/3853259235 wait complete. 2024-02-28T16:45:20.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.662+0000 7f0f53000700 1 Processor -- start 2024-02-28T16:45:20.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.663+0000 7f0f53000700 1 -- start start 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.663+0000 7f0f53000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 0x7f0f4c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.663+0000 7f0f53000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f4c0fe8f0 0x7f0f4c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.663+0000 7f0f53000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c116800 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.663+0000 7f0f53000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c10fb30 con 0x7f0f4c0fe8f0 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.663+0000 7f0f4be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 0x7f0f4c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.663+0000 7f0f4be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 0x7f0f4c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53544/0 (socket says 172.21.15.1:53544) 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.663+0000 7f0f4be00700 1 -- 172.21.15.1:0/3213379193 learned_addr learned my addr 172.21.15.1:0/3213379193 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f4b400700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f4c0fe8f0 0x7f0f4c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f4be00700 1 -- 172.21.15.1:0/3213379193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f4c0fe8f0 msgr2=0x7f0f4c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f4be00700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f4c0fe8f0 0x7f0f4c1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f4be00700 1 -- 172.21.15.1:0/3213379193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c005040 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f4b400700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f4c0fe8f0 0x7f0f4c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:45:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f4be00700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 0x7f0f4c115c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009150 tx=0x7f0f3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f43e00700 1 -- 172.21.15.1:0/3213379193 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3c00e040 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f43e00700 1 -- 172.21.15.1:0/3213379193 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f3c004290 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4c10fdb0 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.664+0000 7f0f43e00700 1 -- 172.21.15.1:0/3213379193 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3c01c630 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.665+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4c1102f0 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.666+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4c106ac0 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.667+0000 7f0f43e00700 1 -- 172.21.15.1:0/3213379193 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0f3c013980 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.668+0000 7f0f43e00700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0f3006d7f0 0x7f0f3006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.668+0000 7f0f43e00700 1 -- 172.21.15.1:0/3213379193 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0f3c08e150 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.668+0000 7f0f4b400700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0f3006d7f0 0x7f0f3006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.669+0000 7f0f4b400700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0f3006d7f0 0x7f0f3006fca0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009fd0 tx=0x7f0f3800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.672+0000 7f0f43e00700 1 -- 172.21.15.1:0/3213379193 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0f3c05b5f0 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:20 smithi066 ceph-mon[85781]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:20.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:20 smithi066 ceph-mon[85781]: pgmap v326: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:20.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:20 smithi066 ceph-mon[85781]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:20.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:20 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/731696542' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:45:20.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:20 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/137394704' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:45:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.993+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0f4c04bed0 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.996+0000 7f0f43e00700 1 -- 172.21.15.1:0/3213379193 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f0f3c05b180 con 0x7f0f4c0fdfe0 2024-02-28T16:45:20.983 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:45:20.983 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:45:20.983 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:45:20.983 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:45:20.983 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:45:20.983 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:45:20.983 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.998+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0f3006d7f0 msgr2=0x7f0f3006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.998+0000 7f0f53000700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0f3006d7f0 0x7f0f3006fca0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009fd0 tx=0x7f0f3800b040 comp rx=0 tx=0).stop 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.998+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 msgr2=0x7f0f4c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.998+0000 7f0f53000700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 0x7f0f4c115c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009150 tx=0x7f0f3c009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.999+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 shutdown_connections 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.999+0000 7f0f53000700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0f3006d7f0 0x7f0f3006fca0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.999+0000 7f0f53000700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f4c0fdfe0 0x7f0f4c115c70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.999+0000 7f0f53000700 1 --2- 172.21.15.1:0/3213379193 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0f4c0fe8f0 0x7f0f4c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.999+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 >> 172.21.15.1:0/3213379193 conn(0x7f0f4c071d90 msgr2=0x7f0f4c103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.999+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 shutdown_connections 2024-02-28T16:45:20.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:20.999+0000 7f0f53000700 1 -- 172.21.15.1:0/3213379193 wait complete. 2024-02-28T16:45:21.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:21 smithi066 ceph-mon[85781]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:21.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:21 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3213379193' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:45:21.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:21 smithi001 ceph-mon[112940]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:21.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:21 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3213379193' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:45:22.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:22 smithi066 ceph-mon[85781]: pgmap v327: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:22.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:22 smithi001 ceph-mon[112940]: pgmap v327: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:24.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:24 smithi066 ceph-mon[85781]: pgmap v328: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:24.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:24 smithi001 ceph-mon[112940]: pgmap v328: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:26.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:26 smithi066 ceph-mon[85781]: pgmap v329: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:26.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:45:26.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:26 smithi001 ceph-mon[112940]: pgmap v329: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:26.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:45:28.747 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:28 smithi066 ceph-mon[85781]: pgmap v330: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:28.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:28 smithi001 ceph-mon[112940]: pgmap v330: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:30.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:30 smithi066 ceph-mon[85781]: pgmap v331: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:30.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:30 smithi001 ceph-mon[112940]: pgmap v331: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:32.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:32 smithi066 ceph-mon[85781]: pgmap v332: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:32.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:32 smithi001 ceph-mon[112940]: pgmap v332: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:34.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:34 smithi066 ceph-mon[85781]: pgmap v333: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:34.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:34 smithi001 ceph-mon[112940]: pgmap v333: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:36.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:36 smithi066 ceph-mon[85781]: pgmap v334: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:36.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:36 smithi001 ceph-mon[112940]: pgmap v334: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:38.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:38 smithi066 ceph-mon[85781]: pgmap v335: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:38.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:38 smithi001 ceph-mon[112940]: pgmap v335: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:40.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:40 smithi066 ceph-mon[85781]: pgmap v336: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:40.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:40 smithi001 ceph-mon[112940]: pgmap v336: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:41.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:45:41.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:45:42.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:42 smithi066 ceph-mon[85781]: pgmap v337: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:42.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:42 smithi001 ceph-mon[112940]: pgmap v337: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:44.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:44 smithi066 ceph-mon[85781]: pgmap v338: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:44.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:44 smithi001 ceph-mon[112940]: pgmap v338: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:46.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:46 smithi066 ceph-mon[85781]: pgmap v339: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:46.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:46 smithi001 ceph-mon[112940]: pgmap v339: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:48.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:48 smithi066 ceph-mon[85781]: pgmap v340: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:48.780 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:48 smithi001 ceph-mon[112940]: pgmap v340: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:50.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:50 smithi066 ceph-mon[85781]: pgmap v341: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:50.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:50 smithi001 ceph-mon[112940]: pgmap v341: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:51.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.169+0000 7f0aa4e00700 1 -- 172.21.15.1:0/3329221269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 msgr2=0x7f0aa00fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.169+0000 7f0aa4e00700 1 --2- 172.21.15.1:0/3329221269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 0x7f0aa00fcf60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c004940 tx=0x7f0a8c009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:51.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.169+0000 7f0aa4e00700 1 -- 172.21.15.1:0/3329221269 shutdown_connections 2024-02-28T16:45:51.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.169+0000 7f0aa4e00700 1 --2- 172.21.15.1:0/3329221269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 0x7f0aa00fcf60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.169+0000 7f0aa4e00700 1 --2- 172.21.15.1:0/3329221269 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0aa0102f50 0x7f0aa0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.169+0000 7f0aa4e00700 1 -- 172.21.15.1:0/3329221269 >> 172.21.15.1:0/3329221269 conn(0x7f0aa00f8ae0 msgr2=0x7f0aa00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:51.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.169+0000 7f0aa4e00700 1 -- 172.21.15.1:0/3329221269 shutdown_connections 2024-02-28T16:45:51.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.169+0000 7f0aa4e00700 1 -- 172.21.15.1:0/3329221269 wait complete. 2024-02-28T16:45:51.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.170+0000 7f0aa4e00700 1 Processor -- start 2024-02-28T16:45:51.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.170+0000 7f0aa4e00700 1 -- start start 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0aa4e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0aa0102f50 0x7f0aa0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0aa4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 0x7f0aa01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0aa4e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0116810 con 0x7f0aa01038f0 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0aa4e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa010fb40 con 0x7f0aa0102f50 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0a9d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 0x7f0aa01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0a9d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 0x7f0aa01161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43080/0 (socket says 172.21.15.1:43080) 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0a9d400700 1 -- 172.21.15.1:0/2764289385 learned_addr learned my addr 172.21.15.1:0/2764289385 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0a9d400700 1 -- 172.21.15.1:0/2764289385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0aa0102f50 msgr2=0x7f0aa0115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0a9d400700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0aa0102f50 0x7f0aa0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.171+0000 7f0a9d400700 1 -- 172.21.15.1:0/2764289385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c005040 con 0x7f0aa01038f0 2024-02-28T16:45:51.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.172+0000 7f0a9d400700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 0x7f0aa01161c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009030 tx=0x7f0a8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.172+0000 7f0a96000700 1 -- 172.21.15.1:0/2764289385 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a8c00e040 con 0x7f0aa01038f0 2024-02-28T16:45:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.172+0000 7f0a96000700 1 -- 172.21.15.1:0/2764289385 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a8c00bbf0 con 0x7f0aa01038f0 2024-02-28T16:45:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.172+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa010fdc0 con 0x7f0aa01038f0 2024-02-28T16:45:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.172+0000 7f0a96000700 1 -- 172.21.15.1:0/2764289385 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a8c01c630 con 0x7f0aa01038f0 2024-02-28T16:45:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.172+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa0110300 con 0x7f0aa01038f0 2024-02-28T16:45:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.173+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa00fc6e0 con 0x7f0aa01038f0 2024-02-28T16:45:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.175+0000 7f0a96000700 1 -- 172.21.15.1:0/2764289385 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0a8c00bd60 con 0x7f0aa01038f0 2024-02-28T16:45:51.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.175+0000 7f0a96000700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a8406d7f0 0x7f0a8406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.176+0000 7f0a96000700 1 -- 172.21.15.1:0/2764289385 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0a8c08e2c0 con 0x7f0aa01038f0 2024-02-28T16:45:51.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.176+0000 7f0a9de00700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a8406d7f0 0x7f0a8406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:51.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.177+0000 7f0a9de00700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a8406d7f0 0x7f0a8406fca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0a90004d00 tx=0x7f0a90009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.181+0000 7f0a96000700 1 -- 172.21.15.1:0/2764289385 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0a8c05b620 con 0x7f0aa01038f0 2024-02-28T16:45:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.425+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0aa0111260 con 0x7f0a8406d7f0 2024-02-28T16:45:51.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.428+0000 7f0a96000700 1 -- 172.21.15.1:0/2764289385 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0aa0111260 con 0x7f0a8406d7f0 2024-02-28T16:45:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a8406d7f0 msgr2=0x7f0a8406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a8406d7f0 0x7f0a8406fca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0a90004d00 tx=0x7f0a90009ce0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 msgr2=0x7f0aa01161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 0x7f0aa01161c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009030 tx=0x7f0a8c009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 shutdown_connections 2024-02-28T16:45:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a8406d7f0 0x7f0a8406fca0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0aa0102f50 0x7f0aa0115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 --2- 172.21.15.1:0/2764289385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0aa01038f0 0x7f0aa01161c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 >> 172.21.15.1:0/2764289385 conn(0x7f0aa00f8ae0 msgr2=0x7f0aa00fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.431+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 shutdown_connections 2024-02-28T16:45:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.432+0000 7f0aa4e00700 1 -- 172.21.15.1:0/2764289385 wait complete. 2024-02-28T16:45:51.435 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:45:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.586+0000 7f94a5200700 1 -- 172.21.15.1:0/2149781780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 msgr2=0x7f94a0104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.586+0000 7f94a5200700 1 --2- 172.21.15.1:0/2149781780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 0x7f94a0104210 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9488004940 tx=0x7f9488009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.586+0000 7f94a5200700 1 -- 172.21.15.1:0/2149781780 shutdown_connections 2024-02-28T16:45:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.586+0000 7f94a5200700 1 --2- 172.21.15.1:0/2149781780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 0x7f94a0104210 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.586+0000 7f94a5200700 1 --2- 172.21.15.1:0/2149781780 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94a01095f0 0x7f94a01099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.586+0000 7f94a5200700 1 -- 172.21.15.1:0/2149781780 >> 172.21.15.1:0/2149781780 conn(0x7f94a0072d50 msgr2=0x7f94a0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.586+0000 7f94a5200700 1 -- 172.21.15.1:0/2149781780 shutdown_connections 2024-02-28T16:45:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.586+0000 7f94a5200700 1 -- 172.21.15.1:0/2149781780 wait complete. 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.587+0000 7f94a5200700 1 Processor -- start 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.587+0000 7f94a5200700 1 -- start start 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.587+0000 7f94a5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 0x7f94a011a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.587+0000 7f94a5200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94a01095f0 0x7f94a011a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.587+0000 7f94a5200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a011abd0 con 0x7f94a0103dc0 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.587+0000 7f94a5200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0113ee0 con 0x7f94a01095f0 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.588+0000 7f949de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 0x7f94a011a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.588+0000 7f949de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 0x7f94a011a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43098/0 (socket says 172.21.15.1:43098) 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.588+0000 7f949de00700 1 -- 172.21.15.1:0/508255241 learned_addr learned my addr 172.21.15.1:0/508255241 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:51.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.588+0000 7f949de00700 1 -- 172.21.15.1:0/508255241 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94a01095f0 msgr2=0x7f94a011a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.588+0000 7f949de00700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94a01095f0 0x7f94a011a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.588+0000 7f949de00700 1 -- 172.21.15.1:0/508255241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488005040 con 0x7f94a0103dc0 2024-02-28T16:45:51.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.588+0000 7f949de00700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 0x7f94a011a040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f949000c830 tx=0x7f9490004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.589+0000 7f9496000700 1 -- 172.21.15.1:0/508255241 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f949000a760 con 0x7f94a0103dc0 2024-02-28T16:45:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.589+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a01141c0 con 0x7f94a0103dc0 2024-02-28T16:45:51.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.589+0000 7f9496000700 1 -- 172.21.15.1:0/508255241 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f949000ad90 con 0x7f94a0103dc0 2024-02-28T16:45:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.589+0000 7f9496000700 1 -- 172.21.15.1:0/508255241 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94900113a0 con 0x7f94a0103dc0 2024-02-28T16:45:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.589+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a0114700 con 0x7f94a0103dc0 2024-02-28T16:45:51.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.590+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a010b9c0 con 0x7f94a0103dc0 2024-02-28T16:45:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.591+0000 7f9496000700 1 -- 172.21.15.1:0/508255241 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9490012400 con 0x7f94a0103dc0 2024-02-28T16:45:51.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.592+0000 7f9496000700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f948406d7e0 0x7f948406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.592+0000 7f9496000700 1 -- 172.21.15.1:0/508255241 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f949008cfa0 con 0x7f94a0103dc0 2024-02-28T16:45:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.592+0000 7f9497e00700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f948406d7e0 0x7f948406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:51.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.593+0000 7f9497e00700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f948406d7e0 0x7f948406fc90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9488004f90 tx=0x7f9488009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:51.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.597+0000 7f9496000700 1 -- 172.21.15.1:0/508255241 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f949005a300 con 0x7f94a0103dc0 2024-02-28T16:45:51.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.844+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94a0115620 con 0x7f948406d7e0 2024-02-28T16:45:51.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.847+0000 7f9496000700 1 -- 172.21.15.1:0/508255241 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f94a0115620 con 0x7f948406d7e0 2024-02-28T16:45:51.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f948406d7e0 msgr2=0x7f948406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f948406d7e0 0x7f948406fc90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9488004f90 tx=0x7f9488009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 msgr2=0x7f94a011a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 0x7f94a011a040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f949000c830 tx=0x7f9490004470 comp rx=0 tx=0).stop 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 shutdown_connections 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f948406d7e0 0x7f948406fc90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94a0103dc0 0x7f94a011a040 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 --2- 172.21.15.1:0/508255241 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94a01095f0 0x7f94a011a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.850+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 >> 172.21.15.1:0/508255241 conn(0x7f94a0072d50 msgr2=0x7f94a0074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.851+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 shutdown_connections 2024-02-28T16:45:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.851+0000 7f94a5200700 1 -- 172.21.15.1:0/508255241 wait complete. 2024-02-28T16:45:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.997+0000 7f7774000700 1 -- 172.21.15.1:0/1804381244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fdfe0 msgr2=0x7f776c0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.997+0000 7f7774000700 1 --2- 172.21.15.1:0/1804381244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fdfe0 0x7f776c0fe3b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7760004970 tx=0x7f7760009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.997+0000 7f7774000700 1 -- 172.21.15.1:0/1804381244 shutdown_connections 2024-02-28T16:45:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.997+0000 7f7774000700 1 --2- 172.21.15.1:0/1804381244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0fe8f0 0x7f776c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.997+0000 7f7774000700 1 --2- 172.21.15.1:0/1804381244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fdfe0 0x7f776c0fe3b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.997+0000 7f7774000700 1 -- 172.21.15.1:0/1804381244 >> 172.21.15.1:0/1804381244 conn(0x7f776c071d90 msgr2=0x7f776c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:51.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.997+0000 7f7774000700 1 -- 172.21.15.1:0/1804381244 shutdown_connections 2024-02-28T16:45:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.997+0000 7f7774000700 1 -- 172.21.15.1:0/1804381244 wait complete. 2024-02-28T16:45:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.998+0000 7f7774000700 1 Processor -- start 2024-02-28T16:45:51.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.998+0000 7f7774000700 1 -- start start 2024-02-28T16:45:51.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7774000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0fdfe0 0x7f776c113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7774000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fe8f0 0x7f776c113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7774000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c1145e0 con 0x7f776c0fe8f0 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7774000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c10d910 con 0x7f776c0fdfe0 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7770c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fe8f0 0x7f776c113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7770c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fe8f0 0x7f776c113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48376/0 (socket says 172.21.15.1:48376) 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7770c00700 1 -- 172.21.15.1:0/1458660323 learned_addr learned my addr 172.21.15.1:0/1458660323 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7771600700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0fdfe0 0x7f776c113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7770c00700 1 -- 172.21.15.1:0/1458660323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0fdfe0 msgr2=0x7f776c113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:51.999+0000 7f7770c00700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0fdfe0 0x7f776c113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.000+0000 7f7770c00700 1 -- 172.21.15.1:0/1458660323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7760005040 con 0x7f776c0fe8f0 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.000+0000 7f7770c00700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fe8f0 0x7f776c113f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f775800c8c0 tx=0x7f77580044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.000+0000 7f7769600700 1 -- 172.21.15.1:0/1458660323 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f775800a980 con 0x7f776c0fe8f0 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.000+0000 7f7769600700 1 -- 172.21.15.1:0/1458660323 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f775800ce60 con 0x7f776c0fe8f0 2024-02-28T16:45:51.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.000+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776c10dbf0 con 0x7f776c0fe8f0 2024-02-28T16:45:51.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.000+0000 7f7769600700 1 -- 172.21.15.1:0/1458660323 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77580115e0 con 0x7f776c0fe8f0 2024-02-28T16:45:51.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.000+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c10e130 con 0x7f776c0fe8f0 2024-02-28T16:45:51.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.001+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776c1046c0 con 0x7f776c0fe8f0 2024-02-28T16:45:51.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.002+0000 7f7769600700 1 -- 172.21.15.1:0/1458660323 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7758012400 con 0x7f776c0fe8f0 2024-02-28T16:45:51.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.004+0000 7f7769600700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775406d7f0 0x7f775406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:51.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.004+0000 7f7769600700 1 -- 172.21.15.1:0/1458660323 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f775808ceb0 con 0x7f776c0fe8f0 2024-02-28T16:45:51.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.004+0000 7f7771600700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775406d7f0 0x7f775406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:51.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.005+0000 7f7771600700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775406d7f0 0x7f775406fca0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7760000c00 tx=0x7f7760009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:51.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.008+0000 7f7769600700 1 -- 172.21.15.1:0/1458660323 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f775805a350 con 0x7f776c0fe8f0 2024-02-28T16:45:52.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.239+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f776c113720 con 0x7f775406d7f0 2024-02-28T16:45:52.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.251+0000 7f7769600700 1 -- 172.21.15.1:0/1458660323 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f776c113720 con 0x7f775406d7f0 2024-02-28T16:45:52.239 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (11m) 4m ago 26m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (8m) 4m ago 26m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (8m) 4m ago 25m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (11m) 4m ago 25m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (4m) 4m ago 19m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (4m) 4m ago 18m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (18m) 4m ago 18m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (19m) 4m ago 19m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (14m) 4m ago 28m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (13m) 4m ago 24m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (10m) 4m ago 28m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:45:52.240 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (9m) 4m ago 24m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (13m) 4m ago 26m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (12m) 4m ago 25m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (8m) 4m ago 23m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (7m) 4m ago 23m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (7m) 4m ago 22m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (6m) 4m ago 22m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (5m) 4m ago 21m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (5m) 4m ago 20m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:45:52.241 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (12m) 4m ago 25m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:45:52.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775406d7f0 msgr2=0x7f775406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:52.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775406d7f0 0x7f775406fca0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7760000c00 tx=0x7f7760009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:52.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fe8f0 msgr2=0x7f776c113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:52.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fe8f0 0x7f776c113f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f775800c8c0 tx=0x7f77580044c0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 shutdown_connections 2024-02-28T16:45:52.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775406d7f0 0x7f775406fca0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0fdfe0 0x7f776c113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 --2- 172.21.15.1:0/1458660323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0fe8f0 0x7f776c113f90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 >> 172.21.15.1:0/1458660323 conn(0x7f776c071d90 msgr2=0x7f776c101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:52.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.254+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 shutdown_connections 2024-02-28T16:45:52.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.255+0000 7f7774000700 1 -- 172.21.15.1:0/1458660323 wait complete. 2024-02-28T16:45:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.412+0000 7f0640e00700 1 -- 172.21.15.1:0/2060535856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c103dc0 msgr2=0x7f063c104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.412+0000 7f0640e00700 1 --2- 172.21.15.1:0/2060535856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c103dc0 0x7f063c104210 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0628004940 tx=0x7f0628009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.413+0000 7f0640e00700 1 -- 172.21.15.1:0/2060535856 shutdown_connections 2024-02-28T16:45:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.413+0000 7f0640e00700 1 --2- 172.21.15.1:0/2060535856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c103dc0 0x7f063c104210 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.413+0000 7f0640e00700 1 --2- 172.21.15.1:0/2060535856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f063c1095f0 0x7f063c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.413+0000 7f0640e00700 1 -- 172.21.15.1:0/2060535856 >> 172.21.15.1:0/2060535856 conn(0x7f063c072d50 msgr2=0x7f063c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.413+0000 7f0640e00700 1 -- 172.21.15.1:0/2060535856 shutdown_connections 2024-02-28T16:45:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.413+0000 7f0640e00700 1 -- 172.21.15.1:0/2060535856 wait complete. 2024-02-28T16:45:52.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.414+0000 7f0640e00700 1 Processor -- start 2024-02-28T16:45:52.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.414+0000 7f0640e00700 1 -- start start 2024-02-28T16:45:52.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.415+0000 7f0640e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f063c103dc0 0x7f063c11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:52.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.415+0000 7f0640e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c1095f0 0x7f063c11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:52.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.415+0000 7f0640e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063c11abd0 con 0x7f063c1095f0 2024-02-28T16:45:52.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.415+0000 7f0640e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063c113ee0 con 0x7f063c103dc0 2024-02-28T16:45:52.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.415+0000 7f0639400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c1095f0 0x7f063c11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:52.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.415+0000 7f0639400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c1095f0 0x7f063c11a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48398/0 (socket says 172.21.15.1:48398) 2024-02-28T16:45:52.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.415+0000 7f0639400700 1 -- 172.21.15.1:0/844467686 learned_addr learned my addr 172.21.15.1:0/844467686 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:52.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.416+0000 7f0639e00700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f063c103dc0 0x7f063c11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:52.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.416+0000 7f0639400700 1 -- 172.21.15.1:0/844467686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f063c103dc0 msgr2=0x7f063c11a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:52.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.416+0000 7f0639400700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f063c103dc0 0x7f063c11a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.416+0000 7f0639400700 1 -- 172.21.15.1:0/844467686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0628005040 con 0x7f063c1095f0 2024-02-28T16:45:52.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.416+0000 7f0639400700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c1095f0 0x7f063c11a580 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0628009150 tx=0x7f0628009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:52.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.416+0000 7f0632000700 1 -- 172.21.15.1:0/844467686 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f062800e040 con 0x7f063c1095f0 2024-02-28T16:45:52.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.417+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063c114160 con 0x7f063c1095f0 2024-02-28T16:45:52.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.417+0000 7f0632000700 1 -- 172.21.15.1:0/844467686 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f062800bbf0 con 0x7f063c1095f0 2024-02-28T16:45:52.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.417+0000 7f0632000700 1 -- 172.21.15.1:0/844467686 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f062801c630 con 0x7f063c1095f0 2024-02-28T16:45:52.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.417+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063c1146a0 con 0x7f063c1095f0 2024-02-28T16:45:52.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.418+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063c10b9c0 con 0x7f063c1095f0 2024-02-28T16:45:52.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.419+0000 7f0632000700 1 -- 172.21.15.1:0/844467686 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f062800bd60 con 0x7f063c1095f0 2024-02-28T16:45:52.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.420+0000 7f0632000700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f062006d7f0 0x7f062006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:52.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.420+0000 7f0632000700 1 -- 172.21.15.1:0/844467686 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f062808e1d0 con 0x7f063c1095f0 2024-02-28T16:45:52.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.420+0000 7f0639e00700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f062006d7f0 0x7f062006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:52.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.421+0000 7f0639e00700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f062006d7f0 0x7f062006fca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f062c004990 tx=0x7f062c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:52.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.425+0000 7f0632000700 1 -- 172.21.15.1:0/844467686 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f062805b670 con 0x7f063c1095f0 2024-02-28T16:45:52.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:52 smithi001 ceph-mon[112940]: pgmap v342: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:52.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:52 smithi001 ceph-mon[112940]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:52.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.724+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f063c1142f0 con 0x7f063c1095f0 2024-02-28T16:45:52.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.725+0000 7f0632000700 1 -- 172.21.15.1:0/844467686 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f063c1142f0 con 0x7f063c1095f0 2024-02-28T16:45:52.713 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:45:52.713 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:45:52.713 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:45:52.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:45:52.714 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:45:52.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f062006d7f0 msgr2=0x7f062006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:52.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f062006d7f0 0x7f062006fca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f062c004990 tx=0x7f062c009ee0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c1095f0 msgr2=0x7f063c11a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:52.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c1095f0 0x7f063c11a580 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0628009150 tx=0x7f0628009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:52.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 shutdown_connections 2024-02-28T16:45:52.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f062006d7f0 0x7f062006fca0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f063c103dc0 0x7f063c11a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 --2- 172.21.15.1:0/844467686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063c1095f0 0x7f063c11a580 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 >> 172.21.15.1:0/844467686 conn(0x7f063c072d50 msgr2=0x7f063c074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:52.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.728+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 shutdown_connections 2024-02-28T16:45:52.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.729+0000 7f0640e00700 1 -- 172.21.15.1:0/844467686 wait complete. 2024-02-28T16:45:52.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:52 smithi066 ceph-mon[85781]: pgmap v342: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:52.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:52 smithi066 ceph-mon[85781]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:52.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.876+0000 7f538de00700 1 -- 172.21.15.1:0/236266906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fdfe0 msgr2=0x7f53880fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.876+0000 7f538de00700 1 --2- 172.21.15.1:0/236266906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fdfe0 0x7f53880fe3b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5378004970 tx=0x7f5378009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.876+0000 7f538de00700 1 -- 172.21.15.1:0/236266906 shutdown_connections 2024-02-28T16:45:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.876+0000 7f538de00700 1 --2- 172.21.15.1:0/236266906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53880fe8f0 0x7f5388107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.876+0000 7f538de00700 1 --2- 172.21.15.1:0/236266906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fdfe0 0x7f53880fe3b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.876+0000 7f538de00700 1 -- 172.21.15.1:0/236266906 >> 172.21.15.1:0/236266906 conn(0x7f5388071d90 msgr2=0x7f5388072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.876+0000 7f538de00700 1 -- 172.21.15.1:0/236266906 shutdown_connections 2024-02-28T16:45:52.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.876+0000 7f538de00700 1 -- 172.21.15.1:0/236266906 wait complete. 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.877+0000 7f538de00700 1 Processor -- start 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.877+0000 7f538de00700 1 -- start start 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.877+0000 7f538de00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53880fdfe0 0x7f5388115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.877+0000 7f538de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fe8f0 0x7f5388116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.877+0000 7f538de00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5388116900 con 0x7f53880fe8f0 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.877+0000 7f538de00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538810fb80 con 0x7f53880fdfe0 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.877+0000 7f5386000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fe8f0 0x7f5388116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.878+0000 7f5386000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fe8f0 0x7f5388116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48416/0 (socket says 172.21.15.1:48416) 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.878+0000 7f5386000700 1 -- 172.21.15.1:0/2787592253 learned_addr learned my addr 172.21.15.1:0/2787592253 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.878+0000 7f5386a00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53880fdfe0 0x7f5388115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.878+0000 7f5386000700 1 -- 172.21.15.1:0/2787592253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53880fdfe0 msgr2=0x7f5388115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.878+0000 7f5386000700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53880fdfe0 0x7f5388115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:52.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.878+0000 7f5386000700 1 -- 172.21.15.1:0/2787592253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5378005040 con 0x7f53880fe8f0 2024-02-28T16:45:52.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.878+0000 7f5386000700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fe8f0 0x7f5388116220 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f537400b790 tx=0x7f5374009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:52.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.879+0000 7f537ea00700 1 -- 172.21.15.1:0/2787592253 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f537400f770 con 0x7f53880fe8f0 2024-02-28T16:45:52.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.879+0000 7f537ea00700 1 -- 172.21.15.1:0/2787592253 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f537400fda0 con 0x7f53880fe8f0 2024-02-28T16:45:52.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.879+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538810fe00 con 0x7f53880fe8f0 2024-02-28T16:45:52.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.879+0000 7f537ea00700 1 -- 172.21.15.1:0/2787592253 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f537400d310 con 0x7f53880fe8f0 2024-02-28T16:45:52.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.879+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5388110340 con 0x7f53880fe8f0 2024-02-28T16:45:52.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.880+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5388106b20 con 0x7f53880fe8f0 2024-02-28T16:45:52.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.881+0000 7f537ea00700 1 -- 172.21.15.1:0/2787592253 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5374010420 con 0x7f53880fe8f0 2024-02-28T16:45:52.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.882+0000 7f537ea00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f536c06d7f0 0x7f536c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:52.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.882+0000 7f537ea00700 1 -- 172.21.15.1:0/2787592253 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f537408ce30 con 0x7f53880fe8f0 2024-02-28T16:45:52.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.882+0000 7f5386a00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f536c06d7f0 0x7f536c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:52.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.883+0000 7f5386a00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f536c06d7f0 0x7f536c06fca0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5378004ee0 tx=0x7f5378009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:52.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:52.887+0000 7f537ea00700 1 -- 172.21.15.1:0/2787592253 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f537405a2d0 con 0x7f53880fe8f0 2024-02-28T16:45:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.152+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f53880738b0 con 0x7f53880fe8f0 2024-02-28T16:45:53.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.153+0000 7f537ea00700 1 -- 172.21.15.1:0/2787592253 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f5374059e60 con 0x7f53880fe8f0 2024-02-28T16:45:53.141 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:45:53.141 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:45:53.141 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:45:53.141 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:45:53.141 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:45:53.141 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:45:53.141 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:45:53.142 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:45:53.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:45:53.144 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:45:53.144 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:45:53.144 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:45:53.144 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f536c06d7f0 msgr2=0x7f536c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f536c06d7f0 0x7f536c06fca0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5378004ee0 tx=0x7f5378009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fe8f0 msgr2=0x7f5388116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fe8f0 0x7f5388116220 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f537400b790 tx=0x7f5374009250 comp rx=0 tx=0).stop 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 shutdown_connections 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f536c06d7f0 0x7f536c06fca0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53880fdfe0 0x7f5388115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 --2- 172.21.15.1:0/2787592253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53880fe8f0 0x7f5388116220 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.156+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 >> 172.21.15.1:0/2787592253 conn(0x7f5388071d90 msgr2=0x7f53880fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.157+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 shutdown_connections 2024-02-28T16:45:53.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.157+0000 7f538de00700 1 -- 172.21.15.1:0/2787592253 wait complete. 2024-02-28T16:45:53.146 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:45:53.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.301+0000 7fc1a9000700 1 -- 172.21.15.1:0/1576817140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 msgr2=0x7fc1a4105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.301+0000 7fc1a9000700 1 --2- 172.21.15.1:0/1576817140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 0x7fc1a4105500 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc194004940 tx=0x7fc194009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.301+0000 7fc1a9000700 1 -- 172.21.15.1:0/1576817140 shutdown_connections 2024-02-28T16:45:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.301+0000 7fc1a9000700 1 --2- 172.21.15.1:0/1576817140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1a4105ad0 0x7fc1a40ff140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.301+0000 7fc1a9000700 1 --2- 172.21.15.1:0/1576817140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 0x7fc1a4105500 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.301+0000 7fc1a9000700 1 -- 172.21.15.1:0/1576817140 >> 172.21.15.1:0/1576817140 conn(0x7fc1a40fad00 msgr2=0x7fc1a40fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.301+0000 7fc1a9000700 1 -- 172.21.15.1:0/1576817140 shutdown_connections 2024-02-28T16:45:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.301+0000 7fc1a9000700 1 -- 172.21.15.1:0/1576817140 wait complete. 2024-02-28T16:45:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.302+0000 7fc1a9000700 1 Processor -- start 2024-02-28T16:45:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.302+0000 7fc1a9000700 1 -- start start 2024-02-28T16:45:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.303+0000 7fc1a9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 0x7fc1a4115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.303+0000 7fc1a9000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1a4105ad0 0x7fc1a4116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.303+0000 7fc1a9000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a4116890 con 0x7fc1a4105130 2024-02-28T16:45:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.303+0000 7fc1a9000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a410fba0 con 0x7fc1a4105ad0 2024-02-28T16:45:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.303+0000 7fc1a1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 0x7fc1a4115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.303+0000 7fc1a1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 0x7fc1a4115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48434/0 (socket says 172.21.15.1:48434) 2024-02-28T16:45:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.303+0000 7fc1a1e00700 1 -- 172.21.15.1:0/1146490648 learned_addr learned my addr 172.21.15.1:0/1146490648 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.303+0000 7fc1a1e00700 1 -- 172.21.15.1:0/1146490648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1a4105ad0 msgr2=0x7fc1a4116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc1a1400700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1a4105ad0 0x7fc1a4116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc1a1e00700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1a4105ad0 0x7fc1a4116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc1a1e00700 1 -- 172.21.15.1:0/1146490648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194005040 con 0x7fc1a4105130 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc1a1e00700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 0x7fc1a4115d00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc194009150 tx=0x7fc194009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc19a000700 1 -- 172.21.15.1:0/1146490648 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19400e040 con 0x7fc1a4105130 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc19a000700 1 -- 172.21.15.1:0/1146490648 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc19400bbf0 con 0x7fc1a4105130 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a410fe20 con 0x7fc1a4105130 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc19a000700 1 -- 172.21.15.1:0/1146490648 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19401c630 con 0x7fc1a4105130 2024-02-28T16:45:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.304+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a4110360 con 0x7fc1a4105130 2024-02-28T16:45:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.305+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a4102440 con 0x7fc1a4105130 2024-02-28T16:45:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.306+0000 7fc19a000700 1 -- 172.21.15.1:0/1146490648 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc19400bd60 con 0x7fc1a4105130 2024-02-28T16:45:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.307+0000 7fc19a000700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc18806d7f0 0x7fc18806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.307+0000 7fc19a000700 1 -- 172.21.15.1:0/1146490648 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc19408e380 con 0x7fc1a4105130 2024-02-28T16:45:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.308+0000 7fc1a1400700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc18806d7f0 0x7fc18806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.308+0000 7fc1a1400700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc18806d7f0 0x7fc18806fca0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1a41115a0 tx=0x7fc190009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:53.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.312+0000 7fc19a000700 1 -- 172.21.15.1:0/1146490648 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc19405b6e0 con 0x7fc1a4105130 2024-02-28T16:45:53.429 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:53 smithi001 ceph-mon[112940]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:53.429 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:53 smithi001 ceph-mon[112940]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:53.429 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:53 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/844467686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:45:53.429 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:53 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2787592253' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:45:53.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.546+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1a404bf40 con 0x7fc18806d7f0 2024-02-28T16:45:53.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.549+0000 7fc19a000700 1 -- 172.21.15.1:0/1146490648 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc1a404bf40 con 0x7fc18806d7f0 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:45:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:45:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:45:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:45:53.538 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:45:53.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.551+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc18806d7f0 msgr2=0x7fc18806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:53.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.551+0000 7fc1a9000700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc18806d7f0 0x7fc18806fca0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1a41115a0 tx=0x7fc190009040 comp rx=0 tx=0).stop 2024-02-28T16:45:53.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.551+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 msgr2=0x7fc1a4115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:53.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.551+0000 7fc1a9000700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 0x7fc1a4115d00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc194009150 tx=0x7fc194009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:53.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.552+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 shutdown_connections 2024-02-28T16:45:53.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.552+0000 7fc1a9000700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc18806d7f0 0x7fc18806fca0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.552+0000 7fc1a9000700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1a4105130 0x7fc1a4115d00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.552+0000 7fc1a9000700 1 --2- 172.21.15.1:0/1146490648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1a4105ad0 0x7fc1a4116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.552+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 >> 172.21.15.1:0/1146490648 conn(0x7fc1a40fad00 msgr2=0x7fc1a40fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:53.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.552+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 shutdown_connections 2024-02-28T16:45:53.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.552+0000 7fc1a9000700 1 -- 172.21.15.1:0/1146490648 wait complete. 2024-02-28T16:45:53.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.704+0000 7f6933a00700 1 -- 172.21.15.1:0/3526410857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c1095f0 msgr2=0x7f692c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:53.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.704+0000 7f6933a00700 1 --2- 172.21.15.1:0/3526410857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c1095f0 0x7f692c1099c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6920004940 tx=0x7f6920009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:53.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.704+0000 7f6933a00700 1 -- 172.21.15.1:0/3526410857 shutdown_connections 2024-02-28T16:45:53.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.704+0000 7f6933a00700 1 --2- 172.21.15.1:0/3526410857 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f692c103dc0 0x7f692c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.704+0000 7f6933a00700 1 --2- 172.21.15.1:0/3526410857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c1095f0 0x7f692c1099c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.704+0000 7f6933a00700 1 -- 172.21.15.1:0/3526410857 >> 172.21.15.1:0/3526410857 conn(0x7f692c072d50 msgr2=0x7f692c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:53.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.705+0000 7f6933a00700 1 -- 172.21.15.1:0/3526410857 shutdown_connections 2024-02-28T16:45:53.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.705+0000 7f6933a00700 1 -- 172.21.15.1:0/3526410857 wait complete. 2024-02-28T16:45:53.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.706+0000 7f6933a00700 1 Processor -- start 2024-02-28T16:45:53.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.706+0000 7f6933a00700 1 -- start start 2024-02-28T16:45:53.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6933a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c103dc0 0x7f692c11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:53.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6933a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f692c1095f0 0x7f692c11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:53.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6933a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c11ac70 con 0x7f692c103dc0 2024-02-28T16:45:53.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6933a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c113f80 con 0x7f692c1095f0 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6931000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c103dc0 0x7f692c11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6931000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c103dc0 0x7f692c11a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48454/0 (socket says 172.21.15.1:48454) 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6931000700 1 -- 172.21.15.1:0/1039145231 learned_addr learned my addr 172.21.15.1:0/1039145231 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f692be00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f692c1095f0 0x7f692c11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6931000700 1 -- 172.21.15.1:0/1039145231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f692c1095f0 msgr2=0x7f692c11a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6931000700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f692c1095f0 0x7f692c11a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f6931000700 1 -- 172.21.15.1:0/1039145231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6920005040 con 0x7f692c103dc0 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.707+0000 7f692be00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f692c1095f0 0x7f692c11a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.708+0000 7f6931000700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c103dc0 0x7f692c11a0e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6920009fd0 tx=0x7f6920009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.708+0000 7f6928c00700 1 -- 172.21.15.1:0/1039145231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f692000e040 con 0x7f692c103dc0 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.708+0000 7f6928c00700 1 -- 172.21.15.1:0/1039145231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f692000bbf0 con 0x7f692c103dc0 2024-02-28T16:45:53.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.708+0000 7f6928c00700 1 -- 172.21.15.1:0/1039145231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f692001c3f0 con 0x7f692c103dc0 2024-02-28T16:45:53.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.708+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c114200 con 0x7f692c103dc0 2024-02-28T16:45:53.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.708+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c1146f0 con 0x7f692c103dc0 2024-02-28T16:45:53.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.710+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692c106850 con 0x7f692c103dc0 2024-02-28T16:45:53.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.711+0000 7f6928c00700 1 -- 172.21.15.1:0/1039145231 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f692000bd60 con 0x7f692c103dc0 2024-02-28T16:45:53.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.712+0000 7f6928c00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f691406d7f0 0x7f691406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:45:53.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.712+0000 7f6928c00700 1 -- 172.21.15.1:0/1039145231 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f692008e210 con 0x7f692c103dc0 2024-02-28T16:45:53.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.712+0000 7f692be00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f691406d7f0 0x7f691406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:45:53.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.713+0000 7f692be00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f691406d7f0 0x7f691406fca0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6918009fd0 tx=0x7f691800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:45:53.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:53.717+0000 7f6928c00700 1 -- 172.21.15.1:0/1039145231 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f692005b610 con 0x7f692c103dc0 2024-02-28T16:45:53.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:53 smithi066 ceph-mon[85781]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:53.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:53 smithi066 ceph-mon[85781]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:53.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:53 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/844467686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:45:53.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:53 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2787592253' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:45:54.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.033+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f692c04bed0 con 0x7f692c103dc0 2024-02-28T16:45:54.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.034+0000 7f6928c00700 1 -- 172.21.15.1:0/1039145231 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f692005b1a0 con 0x7f692c103dc0 2024-02-28T16:45:54.021 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:45:54.022 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:45:54.022 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:45:54.022 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:45:54.022 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:45:54.022 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:45:54.022 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:45:54.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f691406d7f0 msgr2=0x7f691406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:54.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f691406d7f0 0x7f691406fca0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6918009fd0 tx=0x7f691800b040 comp rx=0 tx=0).stop 2024-02-28T16:45:54.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c103dc0 msgr2=0x7f692c11a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:45:54.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c103dc0 0x7f692c11a0e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6920009fd0 tx=0x7f6920009e30 comp rx=0 tx=0).stop 2024-02-28T16:45:54.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 shutdown_connections 2024-02-28T16:45:54.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f691406d7f0 0x7f691406fca0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:54.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f692c103dc0 0x7f692c11a0e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:54.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 --2- 172.21.15.1:0/1039145231 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f692c1095f0 0x7f692c11a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:45:54.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.037+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 >> 172.21.15.1:0/1039145231 conn(0x7f692c072d50 msgr2=0x7f692c074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:45:54.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.038+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 shutdown_connections 2024-02-28T16:45:54.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:45:54.038+0000 7f6933a00700 1 -- 172.21.15.1:0/1039145231 wait complete. 2024-02-28T16:45:54.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:54 smithi066 ceph-mon[85781]: pgmap v343: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:54.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:54 smithi066 ceph-mon[85781]: from='client.34690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:54.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:54 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1039145231' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:45:54.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:54 smithi001 ceph-mon[112940]: pgmap v343: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:54.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:54 smithi001 ceph-mon[112940]: from='client.34690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:45:54.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:54 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1039145231' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:45:56.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:56 smithi066 ceph-mon[85781]: pgmap v344: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:56.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:45:56.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:56 smithi001 ceph-mon[112940]: pgmap v344: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:56.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:45:58.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:45:58 smithi066 ceph-mon[85781]: pgmap v345: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:45:58.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:45:58 smithi001 ceph-mon[112940]: pgmap v345: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:00.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:00 smithi066 ceph-mon[85781]: pgmap v346: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:00.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:00 smithi001 ceph-mon[112940]: pgmap v346: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:02.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:02 smithi066 ceph-mon[85781]: pgmap v347: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:02.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:02 smithi001 ceph-mon[112940]: pgmap v347: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:04.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:04 smithi066 ceph-mon[85781]: pgmap v348: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:04.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:04 smithi001 ceph-mon[112940]: pgmap v348: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:06.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:06 smithi066 ceph-mon[85781]: pgmap v349: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:06.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:06 smithi001 ceph-mon[112940]: pgmap v349: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:08.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:08 smithi066 ceph-mon[85781]: pgmap v350: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:08.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:08 smithi001 ceph-mon[112940]: pgmap v350: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:10.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:10 smithi066 ceph-mon[85781]: pgmap v351: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:10.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:10 smithi001 ceph-mon[112940]: pgmap v351: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:11.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:46:11.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:46:12.746 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:12 smithi066 ceph-mon[85781]: pgmap v352: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:12.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:12 smithi001 ceph-mon[112940]: pgmap v352: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:14.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:14 smithi001 ceph-mon[112940]: pgmap v353: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:14.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:14 smithi066 ceph-mon[85781]: pgmap v353: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:16.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:16 smithi001 ceph-mon[112940]: pgmap v354: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:16.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:16 smithi066 ceph-mon[85781]: pgmap v354: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:18.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:18 smithi001 ceph-mon[112940]: pgmap v355: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:18.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:18 smithi066 ceph-mon[85781]: pgmap v355: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:20.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:20 smithi001 ceph-mon[112940]: pgmap v356: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:20.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:20 smithi066 ceph-mon[85781]: pgmap v356: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:22.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:22 smithi066 ceph-mon[85781]: pgmap v357: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:23.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:22 smithi001 ceph-mon[112940]: pgmap v357: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:24.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.209+0000 7f2071800700 1 -- 172.21.15.1:0/807759227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 msgr2=0x7f206c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:24.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.209+0000 7f2071800700 1 --2- 172.21.15.1:0/807759227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 0x7f206c103320 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f205c004940 tx=0x7f205c009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:24.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.209+0000 7f2071800700 1 -- 172.21.15.1:0/807759227 shutdown_connections 2024-02-28T16:46:24.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.209+0000 7f2071800700 1 --2- 172.21.15.1:0/807759227 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f206c1038f0 0x7f206c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.209+0000 7f2071800700 1 --2- 172.21.15.1:0/807759227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 0x7f206c103320 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.209+0000 7f2071800700 1 -- 172.21.15.1:0/807759227 >> 172.21.15.1:0/807759227 conn(0x7f206c0f8ae0 msgr2=0x7f206c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.209+0000 7f2071800700 1 -- 172.21.15.1:0/807759227 shutdown_connections 2024-02-28T16:46:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.209+0000 7f2071800700 1 -- 172.21.15.1:0/807759227 wait complete. 2024-02-28T16:46:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.210+0000 7f2071800700 1 Processor -- start 2024-02-28T16:46:24.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.211+0000 7f2071800700 1 -- start start 2024-02-28T16:46:24.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.211+0000 7f2071800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 0x7f206c070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:24.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.211+0000 7f2071800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f206c1038f0 0x7f206c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:24.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.211+0000 7f2071800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c070dd0 con 0x7f206c102f50 2024-02-28T16:46:24.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.211+0000 7f2071800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c070f40 con 0x7f206c1038f0 2024-02-28T16:46:24.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.211+0000 7f206aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 0x7f206c070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:24.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.211+0000 7f206aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 0x7f206c070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35568/0 (socket says 172.21.15.1:35568) 2024-02-28T16:46:24.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.211+0000 7f206aa00700 1 -- 172.21.15.1:0/2518779463 learned_addr learned my addr 172.21.15.1:0/2518779463 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f206a000700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f206c1038f0 0x7f206c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f206aa00700 1 -- 172.21.15.1:0/2518779463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f206c1038f0 msgr2=0x7f206c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f206aa00700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f206c1038f0 0x7f206c0707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f206aa00700 1 -- 172.21.15.1:0/2518779463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c005040 con 0x7f206c102f50 2024-02-28T16:46:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f206a000700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f206c1038f0 0x7f206c0707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:46:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f206aa00700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 0x7f206c070290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f205c009150 tx=0x7f205c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f2062a00700 1 -- 172.21.15.1:0/2518779463 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f205c00e040 con 0x7f206c102f50 2024-02-28T16:46:24.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f2062a00700 1 -- 172.21.15.1:0/2518779463 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f205c00bbf0 con 0x7f206c102f50 2024-02-28T16:46:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.212+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c119c90 con 0x7f206c102f50 2024-02-28T16:46:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.213+0000 7f2062a00700 1 -- 172.21.15.1:0/2518779463 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f205c01c630 con 0x7f206c102f50 2024-02-28T16:46:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.213+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c11a0f0 con 0x7f206c102f50 2024-02-28T16:46:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.214+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206c0fc6e0 con 0x7f206c102f50 2024-02-28T16:46:24.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.215+0000 7f2062a00700 1 -- 172.21.15.1:0/2518779463 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f205c00bd60 con 0x7f206c102f50 2024-02-28T16:46:24.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.216+0000 7f2062a00700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f205006d7f0 0x7f205006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:24.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.216+0000 7f2062a00700 1 -- 172.21.15.1:0/2518779463 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f205c08e280 con 0x7f206c102f50 2024-02-28T16:46:24.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.216+0000 7f206a000700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f205006d7f0 0x7f205006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:24.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.217+0000 7f206a000700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f205006d7f0 0x7f205006fca0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2058007fd0 tx=0x7f2058009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:24.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.221+0000 7f2062a00700 1 -- 172.21.15.1:0/2518779463 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f205c057b30 con 0x7f206c102f50 2024-02-28T16:46:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.463+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f206c0fac10 con 0x7f205006d7f0 2024-02-28T16:46:24.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.465+0000 7f2062a00700 1 -- 172.21.15.1:0/2518779463 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f206c0fac10 con 0x7f205006d7f0 2024-02-28T16:46:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f205006d7f0 msgr2=0x7f205006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f205006d7f0 0x7f205006fca0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2058007fd0 tx=0x7f2058009040 comp rx=0 tx=0).stop 2024-02-28T16:46:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 msgr2=0x7f206c070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 0x7f206c070290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f205c009150 tx=0x7f205c009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 shutdown_connections 2024-02-28T16:46:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f205006d7f0 0x7f205006fca0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f206c102f50 0x7f206c070290 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 --2- 172.21.15.1:0/2518779463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f206c1038f0 0x7f206c0707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.468+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 >> 172.21.15.1:0/2518779463 conn(0x7f206c0f8ae0 msgr2=0x7f206c0fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.469+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 shutdown_connections 2024-02-28T16:46:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.469+0000 7f2071800700 1 -- 172.21.15.1:0/2518779463 wait complete. 2024-02-28T16:46:24.471 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:46:24.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.628+0000 7f2d48600700 1 -- 172.21.15.1:0/630791949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 msgr2=0x7f2d40070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.628+0000 7f2d48600700 1 --2- 172.21.15.1:0/630791949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 0x7f2d40070680 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d34004940 tx=0x7f2d34009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.628+0000 7f2d48600700 1 -- 172.21.15.1:0/630791949 shutdown_connections 2024-02-28T16:46:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.628+0000 7f2d48600700 1 --2- 172.21.15.1:0/630791949 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d40070bc0 0x7f2d401092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.628+0000 7f2d48600700 1 --2- 172.21.15.1:0/630791949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 0x7f2d40070680 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.628+0000 7f2d48600700 1 -- 172.21.15.1:0/630791949 >> 172.21.15.1:0/630791949 conn(0x7f2d400f8af0 msgr2=0x7f2d400faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:24.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.628+0000 7f2d48600700 1 -- 172.21.15.1:0/630791949 shutdown_connections 2024-02-28T16:46:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.628+0000 7f2d48600700 1 -- 172.21.15.1:0/630791949 wait complete. 2024-02-28T16:46:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.629+0000 7f2d48600700 1 Processor -- start 2024-02-28T16:46:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.629+0000 7f2d48600700 1 -- start start 2024-02-28T16:46:24.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.630+0000 7f2d48600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 0x7f2d400fcf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:24.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.630+0000 7f2d48600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d40070bc0 0x7f2d400fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.630+0000 7f2d48600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d400fd9b0 con 0x7f2d400702b0 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.630+0000 7f2d48600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d400fdaf0 con 0x7f2d40070bc0 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.630+0000 7f2d45c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 0x7f2d400fcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.630+0000 7f2d45c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 0x7f2d400fcf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35598/0 (socket says 172.21.15.1:35598) 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.630+0000 7f2d45c00700 1 -- 172.21.15.1:0/180713676 learned_addr learned my addr 172.21.15.1:0/180713676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.630+0000 7f2d45200700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d40070bc0 0x7f2d400fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d45c00700 1 -- 172.21.15.1:0/180713676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d40070bc0 msgr2=0x7f2d400fd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d45c00700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d40070bc0 0x7f2d400fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d45c00700 1 -- 172.21.15.1:0/180713676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d34005040 con 0x7f2d400702b0 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d45c00700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 0x7f2d400fcf30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009fd0 tx=0x7f2d34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d3d600700 1 -- 172.21.15.1:0/180713676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d3400e040 con 0x7f2d400702b0 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d3d600700 1 -- 172.21.15.1:0/180713676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d3400bbf0 con 0x7f2d400702b0 2024-02-28T16:46:24.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d400fdd70 con 0x7f2d400702b0 2024-02-28T16:46:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d3d600700 1 -- 172.21.15.1:0/180713676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d3401c3f0 con 0x7f2d400702b0 2024-02-28T16:46:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.631+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4011e580 con 0x7f2d400702b0 2024-02-28T16:46:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.633+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d40106990 con 0x7f2d400702b0 2024-02-28T16:46:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.634+0000 7f2d3d600700 1 -- 172.21.15.1:0/180713676 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2d3400bd60 con 0x7f2d400702b0 2024-02-28T16:46:24.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.635+0000 7f2d3d600700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d2806d7f0 0x7f2d2806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.635+0000 7f2d3d600700 1 -- 172.21.15.1:0/180713676 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2d3408e270 con 0x7f2d400702b0 2024-02-28T16:46:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.635+0000 7f2d45200700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d2806d7f0 0x7f2d2806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:24.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.636+0000 7f2d45200700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d2806d7f0 0x7f2d2806fca0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2d400fe930 tx=0x7f2d2c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.639+0000 7f2d3d600700 1 -- 172.21.15.1:0/180713676 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2d34057aa0 con 0x7f2d400702b0 2024-02-28T16:46:24.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:24 smithi001 ceph-mon[112940]: pgmap v358: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:24.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.888+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d400fe670 con 0x7f2d2806d7f0 2024-02-28T16:46:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.891+0000 7f2d3d600700 1 -- 172.21.15.1:0/180713676 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2d400fe670 con 0x7f2d2806d7f0 2024-02-28T16:46:24.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.893+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d2806d7f0 msgr2=0x7f2d2806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:24.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.893+0000 7f2d48600700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d2806d7f0 0x7f2d2806fca0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2d400fe930 tx=0x7f2d2c00b040 comp rx=0 tx=0).stop 2024-02-28T16:46:24.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.893+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 msgr2=0x7f2d400fcf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:24.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.893+0000 7f2d48600700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 0x7f2d400fcf30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009fd0 tx=0x7f2d34009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:24.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.893+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 shutdown_connections 2024-02-28T16:46:24.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.894+0000 7f2d48600700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d2806d7f0 0x7f2d2806fca0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.894+0000 7f2d48600700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d400702b0 0x7f2d400fcf30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.894+0000 7f2d48600700 1 --2- 172.21.15.1:0/180713676 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d40070bc0 0x7f2d400fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:24.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.894+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 >> 172.21.15.1:0/180713676 conn(0x7f2d400f8af0 msgr2=0x7f2d40103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:24.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.894+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 shutdown_connections 2024-02-28T16:46:24.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:24.894+0000 7f2d48600700 1 -- 172.21.15.1:0/180713676 wait complete. 2024-02-28T16:46:24.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:24 smithi066 ceph-mon[85781]: pgmap v358: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.047+0000 7f1fc3e00700 1 -- 172.21.15.1:0/50042669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 msgr2=0x7f1fc40fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.047+0000 7f1fc3e00700 1 --2- 172.21.15.1:0/50042669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 0x7f1fc40fcda0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1fac004940 tx=0x7f1fac009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.047+0000 7f1fc3e00700 1 -- 172.21.15.1:0/50042669 shutdown_connections 2024-02-28T16:46:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.047+0000 7f1fc3e00700 1 --2- 172.21.15.1:0/50042669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 0x7f1fc40fcda0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.047+0000 7f1fc3e00700 1 --2- 172.21.15.1:0/50042669 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fc4102d90 0x7f1fc4103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.047+0000 7f1fc3e00700 1 -- 172.21.15.1:0/50042669 >> 172.21.15.1:0/50042669 conn(0x7f1fc40f8940 msgr2=0x7f1fc40fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:25.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.047+0000 7f1fc3e00700 1 -- 172.21.15.1:0/50042669 shutdown_connections 2024-02-28T16:46:25.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.047+0000 7f1fc3e00700 1 -- 172.21.15.1:0/50042669 wait complete. 2024-02-28T16:46:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.048+0000 7f1fc3e00700 1 Processor -- start 2024-02-28T16:46:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.048+0000 7f1fc3e00700 1 -- start start 2024-02-28T16:46:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fc3e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fc4102d90 0x7f1fc4115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fc3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 0x7f1fc4115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fc3e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc41166d0 con 0x7f1fc4103730 2024-02-28T16:46:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fc3e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc410f970 con 0x7f1fc4102d90 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fbbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 0x7f1fc4115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fbbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 0x7f1fc4115ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35618/0 (socket says 172.21.15.1:35618) 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fbbe00700 1 -- 172.21.15.1:0/3268542607 learned_addr learned my addr 172.21.15.1:0/3268542607 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fbbe00700 1 -- 172.21.15.1:0/3268542607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fc4102d90 msgr2=0x7f1fc4115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.049+0000 7f1fc2a00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fc4102d90 0x7f1fc4115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fbbe00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fc4102d90 0x7f1fc4115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fbbe00700 1 -- 172.21.15.1:0/3268542607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac005040 con 0x7f1fc4103730 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fc2a00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fc4102d90 0x7f1fc4115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:46:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fbbe00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 0x7f1fc4115ff0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009030 tx=0x7f1fac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:25.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fbaa00700 1 -- 172.21.15.1:0/3268542607 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fac00e040 con 0x7f1fc4103730 2024-02-28T16:46:25.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fbaa00700 1 -- 172.21.15.1:0/3268542607 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fac00bbf0 con 0x7f1fc4103730 2024-02-28T16:46:25.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc410fbf0 con 0x7f1fc4103730 2024-02-28T16:46:25.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fbaa00700 1 -- 172.21.15.1:0/3268542607 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fac01c630 con 0x7f1fc4103730 2024-02-28T16:46:25.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.050+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc41101b0 con 0x7f1fc4103730 2024-02-28T16:46:25.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.051+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc405f5c0 con 0x7f1fc4103730 2024-02-28T16:46:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.052+0000 7f1fbaa00700 1 -- 172.21.15.1:0/3268542607 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1fac00bd60 con 0x7f1fc4103730 2024-02-28T16:46:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.054+0000 7f1fbaa00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fa806d7f0 0x7f1fa806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.054+0000 7f1fbaa00700 1 -- 172.21.15.1:0/3268542607 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1fac08e1f0 con 0x7f1fc4103730 2024-02-28T16:46:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.054+0000 7f1fc2a00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fa806d7f0 0x7f1fa806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.055+0000 7f1fc2a00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fa806d7f0 0x7f1fa806fca0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4006fd0 tx=0x7f1fb4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:25.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.058+0000 7f1fbaa00700 1 -- 172.21.15.1:0/3268542607 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1fac05b5d0 con 0x7f1fc4103730 2024-02-28T16:46:25.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.293+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fc404bed0 con 0x7f1fa806d7f0 2024-02-28T16:46:25.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.310+0000 7f1fbaa00700 1 -- 172.21.15.1:0/3268542607 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1fc404bed0 con 0x7f1fa806d7f0 2024-02-28T16:46:25.297 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:46:25.297 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (12m) 4m ago 26m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:46:25.297 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (9m) 4m ago 27m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:46:25.297 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (9m) 5m ago 25m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:46:25.297 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (12m) 4m ago 26m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:46:25.297 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (5m) 4m ago 19m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (4m) 4m ago 19m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (19m) 5m ago 19m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (19m) 5m ago 19m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (14m) 4m ago 28m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (13m) 5m ago 25m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (10m) 4m ago 28m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (10m) 5m ago 25m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (13m) 4m ago 26m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (13m) 5m ago 25m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (8m) 4m ago 24m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (8m) 4m ago 23m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (7m) 4m ago 23m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (7m) 5m ago 22m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:46:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (6m) 5m ago 22m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:46:25.299 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (5m) 5m ago 21m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:46:25.299 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (13m) 4m ago 26m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:46:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fa806d7f0 msgr2=0x7f1fa806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fa806d7f0 0x7f1fa806fca0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4006fd0 tx=0x7f1fb4008040 comp rx=0 tx=0).stop 2024-02-28T16:46:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 msgr2=0x7f1fc4115ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 0x7f1fc4115ff0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009030 tx=0x7f1fac009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 shutdown_connections 2024-02-28T16:46:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fa806d7f0 0x7f1fa806fca0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fc4102d90 0x7f1fc4115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 --2- 172.21.15.1:0/3268542607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fc4103730 0x7f1fc4115ff0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 >> 172.21.15.1:0/3268542607 conn(0x7f1fc40f8940 msgr2=0x7f1fc40fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 shutdown_connections 2024-02-28T16:46:25.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.313+0000 7f1fc3e00700 1 -- 172.21.15.1:0/3268542607 wait complete. 2024-02-28T16:46:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.460+0000 7fd1f5800700 1 -- 172.21.15.1:0/2608752114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 msgr2=0x7fd1f0100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.460+0000 7fd1f5800700 1 --2- 172.21.15.1:0/2608752114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 0x7fd1f0100d10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc004940 tx=0x7fd1dc009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.460+0000 7fd1f5800700 1 -- 172.21.15.1:0/2608752114 shutdown_connections 2024-02-28T16:46:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.460+0000 7fd1f5800700 1 --2- 172.21.15.1:0/2608752114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 0x7fd1f0100d10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.460+0000 7fd1f5800700 1 --2- 172.21.15.1:0/2608752114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1f00fbd50 0x7fd1f00fc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.460+0000 7fd1f5800700 1 -- 172.21.15.1:0/2608752114 >> 172.21.15.1:0/2608752114 conn(0x7fd1f00f7a10 msgr2=0x7fd1f00f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.460+0000 7fd1f5800700 1 -- 172.21.15.1:0/2608752114 shutdown_connections 2024-02-28T16:46:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.460+0000 7fd1f5800700 1 -- 172.21.15.1:0/2608752114 wait complete. 2024-02-28T16:46:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.461+0000 7fd1f5800700 1 Processor -- start 2024-02-28T16:46:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.461+0000 7fd1f5800700 1 -- start start 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.461+0000 7fd1f5800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1f00fbd50 0x7fd1f011a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.461+0000 7fd1f5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 0x7fd1f011a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.461+0000 7fd1f5800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f011ad50 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.461+0000 7fd1f5800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f0114060 con 0x7fd1f00fbd50 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.462+0000 7fd1ee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 0x7fd1f011a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.462+0000 7fd1ee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 0x7fd1f011a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35626/0 (socket says 172.21.15.1:35626) 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.462+0000 7fd1ee000700 1 -- 172.21.15.1:0/2297573064 learned_addr learned my addr 172.21.15.1:0/2297573064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.462+0000 7fd1eea00700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1f00fbd50 0x7fd1f011a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.462+0000 7fd1ee000700 1 -- 172.21.15.1:0/2297573064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1f00fbd50 msgr2=0x7fd1f011a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.462+0000 7fd1ee000700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1f00fbd50 0x7fd1f011a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.462+0000 7fd1ee000700 1 -- 172.21.15.1:0/2297573064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc005040 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.462+0000 7fd1ee000700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 0x7fd1f011a700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009150 tx=0x7fd1dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:25.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.463+0000 7fd1e6a00700 1 -- 172.21.15.1:0/2297573064 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1dc00e040 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.463+0000 7fd1e6a00700 1 -- 172.21.15.1:0/2297573064 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1dc00bbf0 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.463+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f01142e0 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.463+0000 7fd1e6a00700 1 -- 172.21.15.1:0/2297573064 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1dc01c630 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.463+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f0114820 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.464+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f00fe430 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.465+0000 7fd1e6a00700 1 -- 172.21.15.1:0/2297573064 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd1dc00bd60 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.466+0000 7fd1e6a00700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd1d406d7f0 0x7fd1d406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.466+0000 7fd1e6a00700 1 -- 172.21.15.1:0/2297573064 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd1dc08e360 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.466+0000 7fd1eea00700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd1d406d7f0 0x7fd1d406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.467+0000 7fd1eea00700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd1d406d7f0 0x7fd1d406fca0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00041d0 tx=0x7fd1e0009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:25.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.470+0000 7fd1e6a00700 1 -- 172.21.15.1:0/2297573064 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd1dc05b6c0 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.782+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd1f0114470 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.784+0000 7fd1e6a00700 1 -- 172.21.15.1:0/2297573064 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fd1f0114470 con 0x7fd1f00fc6f0 2024-02-28T16:46:25.770 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:46:25.771 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:46:25.772 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:46:25.772 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:46:25.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:46:25.772 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:46:25.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd1d406d7f0 msgr2=0x7fd1d406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd1d406d7f0 0x7fd1d406fca0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00041d0 tx=0x7fd1e0009040 comp rx=0 tx=0).stop 2024-02-28T16:46:25.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 msgr2=0x7fd1f011a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 0x7fd1f011a700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009150 tx=0x7fd1dc009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:25.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 shutdown_connections 2024-02-28T16:46:25.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd1d406d7f0 0x7fd1d406fca0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd1f00fbd50 0x7fd1f011a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 --2- 172.21.15.1:0/2297573064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f00fc6f0 0x7fd1f011a700 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.787+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 >> 172.21.15.1:0/2297573064 conn(0x7fd1f00f7a10 msgr2=0x7fd1f010d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.788+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 shutdown_connections 2024-02-28T16:46:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.788+0000 7fd1f5800700 1 -- 172.21.15.1:0/2297573064 wait complete. 2024-02-28T16:46:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.943+0000 7f30ca800700 1 -- 172.21.15.1:0/1401783694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 msgr2=0x7f30c41099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.943+0000 7f30ca800700 1 --2- 172.21.15.1:0/1401783694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 0x7f30c41099c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f30b4004940 tx=0x7f30b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.943+0000 7f30ca800700 1 -- 172.21.15.1:0/1401783694 shutdown_connections 2024-02-28T16:46:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.943+0000 7f30ca800700 1 --2- 172.21.15.1:0/1401783694 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30c4103dc0 0x7f30c4104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.943+0000 7f30ca800700 1 --2- 172.21.15.1:0/1401783694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 0x7f30c41099c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.943+0000 7f30ca800700 1 -- 172.21.15.1:0/1401783694 >> 172.21.15.1:0/1401783694 conn(0x7f30c4072d50 msgr2=0x7f30c4075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:25.930 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:25 smithi001 ceph-mon[112940]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.943+0000 7f30ca800700 1 -- 172.21.15.1:0/1401783694 shutdown_connections 2024-02-28T16:46:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.943+0000 7f30ca800700 1 -- 172.21.15.1:0/1401783694 wait complete. 2024-02-28T16:46:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.944+0000 7f30ca800700 1 Processor -- start 2024-02-28T16:46:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.944+0000 7f30ca800700 1 -- start start 2024-02-28T16:46:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30ca800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30c4103dc0 0x7f30c411a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30ca800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 0x7f30c411a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30ca800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c411ac70 con 0x7f30c41095f0 2024-02-28T16:46:25.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30ca800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4113ef0 con 0x7f30c4103dc0 2024-02-28T16:46:25.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30c2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 0x7f30c411a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30c2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 0x7f30c411a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35644/0 (socket says 172.21.15.1:35644) 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30c2a00700 1 -- 172.21.15.1:0/2521445075 learned_addr learned my addr 172.21.15.1:0/2521445075 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30c3400700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30c4103dc0 0x7f30c411a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.945+0000 7f30c2a00700 1 -- 172.21.15.1:0/2521445075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30c4103dc0 msgr2=0x7f30c411a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.946+0000 7f30c2a00700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30c4103dc0 0x7f30c411a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.946+0000 7f30c2a00700 1 -- 172.21.15.1:0/2521445075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4005040 con 0x7f30c41095f0 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.946+0000 7f30c2a00700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 0x7f30c411a590 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f30b000b790 tx=0x7f30b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.946+0000 7f30bb400700 1 -- 172.21.15.1:0/2521445075 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b000f770 con 0x7f30c41095f0 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.946+0000 7f30bb400700 1 -- 172.21.15.1:0/2521445075 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30b000fda0 con 0x7f30c41095f0 2024-02-28T16:46:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.946+0000 7f30bb400700 1 -- 172.21.15.1:0/2521445075 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b000d310 con 0x7f30c41095f0 2024-02-28T16:46:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.946+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c4114170 con 0x7f30c41095f0 2024-02-28T16:46:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.946+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c41146b0 con 0x7f30c41095f0 2024-02-28T16:46:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.948+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c4075870 con 0x7f30c41095f0 2024-02-28T16:46:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.949+0000 7f30bb400700 1 -- 172.21.15.1:0/2521445075 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f30b0010420 con 0x7f30c41095f0 2024-02-28T16:46:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.950+0000 7f30bb400700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f30a806d7f0 0x7f30a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.950+0000 7f30bb400700 1 -- 172.21.15.1:0/2521445075 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f30b008cec0 con 0x7f30c41095f0 2024-02-28T16:46:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.950+0000 7f30c3400700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f30a806d7f0 0x7f30a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:25.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.951+0000 7f30c3400700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f30a806d7f0 0x7f30a806fca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f30b400b020 tx=0x7f30b4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:25.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:25.955+0000 7f30bb400700 1 -- 172.21.15.1:0/2521445075 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f30b005a2a0 con 0x7f30c41095f0 2024-02-28T16:46:25.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:25 smithi066 ceph-mon[85781]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.214+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f30c41106a0 con 0x7f30c41095f0 2024-02-28T16:46:26.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.215+0000 7f30bb400700 1 -- 172.21.15.1:0/2521445075 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f30b0059e30 con 0x7f30c41095f0 2024-02-28T16:46:26.202 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:46:26.203 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:46:26.204 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:46:26.204 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:46:26.204 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:46:26.204 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:46:26.204 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:46:26.204 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:46:26.205 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:46:26.205 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:46:26.205 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:46:26.205 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:46:26.205 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:46:26.205 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:46:26.205 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:46:26.205 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:46:26.206 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:46:26.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f30a806d7f0 msgr2=0x7f30a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:26.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f30a806d7f0 0x7f30a806fca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f30b400b020 tx=0x7f30b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:26.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 msgr2=0x7f30c411a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:26.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 0x7f30c411a590 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f30b000b790 tx=0x7f30b0009250 comp rx=0 tx=0).stop 2024-02-28T16:46:26.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 shutdown_connections 2024-02-28T16:46:26.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f30a806d7f0 0x7f30a806fca0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30c4103dc0 0x7f30c411a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 --2- 172.21.15.1:0/2521445075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41095f0 0x7f30c411a590 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 >> 172.21.15.1:0/2521445075 conn(0x7f30c4072d50 msgr2=0x7f30c40746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 shutdown_connections 2024-02-28T16:46:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.218+0000 7f30ca800700 1 -- 172.21.15.1:0/2521445075 wait complete. 2024-02-28T16:46:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:46:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.374+0000 7f8554600700 1 -- 172.21.15.1:0/3397944562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 msgr2=0x7f854c070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.374+0000 7f8554600700 1 --2- 172.21.15.1:0/3397944562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 0x7f854c070330 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8540004970 tx=0x7f8540009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.374+0000 7f8554600700 1 -- 172.21.15.1:0/3397944562 shutdown_connections 2024-02-28T16:46:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.374+0000 7f8554600700 1 --2- 172.21.15.1:0/3397944562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f854c070870 0x7f854c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.374+0000 7f8554600700 1 --2- 172.21.15.1:0/3397944562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 0x7f854c070330 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.374+0000 7f8554600700 1 -- 172.21.15.1:0/3397944562 >> 172.21.15.1:0/3397944562 conn(0x7f854c0f8ae0 msgr2=0x7f854c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.374+0000 7f8554600700 1 -- 172.21.15.1:0/3397944562 shutdown_connections 2024-02-28T16:46:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.374+0000 7f8554600700 1 -- 172.21.15.1:0/3397944562 wait complete. 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.375+0000 7f8554600700 1 Processor -- start 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.375+0000 7f8554600700 1 -- start start 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.375+0000 7f8554600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f854c070870 0x7f854c11a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.375+0000 7f8554600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 0x7f854c11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.375+0000 7f8554600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c11ac00 con 0x7f854c071f60 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.375+0000 7f8554600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c113f10 con 0x7f854c070870 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.376+0000 7f8551200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 0x7f854c11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.376+0000 7f8551200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 0x7f854c11a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35668/0 (socket says 172.21.15.1:35668) 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.376+0000 7f8551200700 1 -- 172.21.15.1:0/3325270573 learned_addr learned my addr 172.21.15.1:0/3325270573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.376+0000 7f8551c00700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f854c070870 0x7f854c11a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.376+0000 7f8551200700 1 -- 172.21.15.1:0/3325270573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f854c070870 msgr2=0x7f854c11a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.376+0000 7f8551200700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f854c070870 0x7f854c11a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.376+0000 7f8551200700 1 -- 172.21.15.1:0/3325270573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8540005040 con 0x7f854c071f60 2024-02-28T16:46:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.376+0000 7f8551c00700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f854c070870 0x7f854c11a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:46:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.377+0000 7f8551200700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 0x7f854c11a5b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8538002a50 tx=0x7f85380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.377+0000 7f8549600700 1 -- 172.21.15.1:0/3325270573 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8538007cc0 con 0x7f854c071f60 2024-02-28T16:46:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.377+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c1141f0 con 0x7f854c071f60 2024-02-28T16:46:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.377+0000 7f8549600700 1 -- 172.21.15.1:0/3325270573 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f853800f040 con 0x7f854c071f60 2024-02-28T16:46:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.377+0000 7f8549600700 1 -- 172.21.15.1:0/3325270573 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8538009640 con 0x7f854c071f60 2024-02-28T16:46:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.377+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c114730 con 0x7f854c071f60 2024-02-28T16:46:26.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.378+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c0fe400 con 0x7f854c071f60 2024-02-28T16:46:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.379+0000 7f8549600700 1 -- 172.21.15.1:0/3325270573 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f85380097a0 con 0x7f854c071f60 2024-02-28T16:46:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.381+0000 7f8549600700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853406d7f0 0x7f853406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.381+0000 7f8549600700 1 -- 172.21.15.1:0/3325270573 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f853808cc80 con 0x7f854c071f60 2024-02-28T16:46:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.381+0000 7f8551c00700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853406d7f0 0x7f853406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.382+0000 7f8551c00700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853406d7f0 0x7f853406fca0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8540000c00 tx=0x7f8540009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:26.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.385+0000 7f8549600700 1 -- 172.21.15.1:0/3325270573 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f853805a000 con 0x7f854c071f60 2024-02-28T16:46:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.625+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f854c04bf40 con 0x7f853406d7f0 2024-02-28T16:46:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.627+0000 7f8549600700 1 -- 172.21.15.1:0/3325270573 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f854c04bf40 con 0x7f853406d7f0 2024-02-28T16:46:26.614 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:46:26.615 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:46:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853406d7f0 msgr2=0x7f853406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853406d7f0 0x7f853406fca0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8540000c00 tx=0x7f8540009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 msgr2=0x7f854c11a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 0x7f854c11a5b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8538002a50 tx=0x7f85380046b0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 shutdown_connections 2024-02-28T16:46:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853406d7f0 0x7f853406fca0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f854c070870 0x7f854c11a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 --2- 172.21.15.1:0/3325270573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c071f60 0x7f854c11a5b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 >> 172.21.15.1:0/3325270573 conn(0x7f854c0f8ae0 msgr2=0x7f854c0fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:26.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 shutdown_connections 2024-02-28T16:46:26.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.630+0000 7f8554600700 1 -- 172.21.15.1:0/3325270573 wait complete. 2024-02-28T16:46:26.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.785+0000 7f90ae800700 1 -- 172.21.15.1:0/891607627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8070870 msgr2=0x7f90a8070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:26.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.785+0000 7f90ae800700 1 --2- 172.21.15.1:0/891607627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8070870 0x7f90a8070ce0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9094004970 tx=0x7f9094009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:26.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.786+0000 7f90ae800700 1 -- 172.21.15.1:0/891607627 shutdown_connections 2024-02-28T16:46:26.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.786+0000 7f90ae800700 1 --2- 172.21.15.1:0/891607627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8070870 0x7f90a8070ce0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.786+0000 7f90ae800700 1 --2- 172.21.15.1:0/891607627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90a8071f60 0x7f90a8070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.786+0000 7f90ae800700 1 -- 172.21.15.1:0/891607627 >> 172.21.15.1:0/891607627 conn(0x7f90a80f8940 msgr2=0x7f90a80fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:26.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.786+0000 7f90ae800700 1 -- 172.21.15.1:0/891607627 shutdown_connections 2024-02-28T16:46:26.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.786+0000 7f90ae800700 1 -- 172.21.15.1:0/891607627 wait complete. 2024-02-28T16:46:26.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.787+0000 7f90ae800700 1 Processor -- start 2024-02-28T16:46:26.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.787+0000 7f90ae800700 1 -- start start 2024-02-28T16:46:26.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.787+0000 7f90ae800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90a8070870 0x7f90a8119ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:26.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90ae800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8071f60 0x7f90a811a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:26.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90ae800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a811aae0 con 0x7f90a8071f60 2024-02-28T16:46:26.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90ae800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8113d60 con 0x7f90a8070870 2024-02-28T16:46:26.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90aca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8071f60 0x7f90a811a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:26.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90aca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8071f60 0x7f90a811a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35682/0 (socket says 172.21.15.1:35682) 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90aca00700 1 -- 172.21.15.1:0/3778803673 learned_addr learned my addr 172.21.15.1:0/3778803673 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90ad400700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90a8070870 0x7f90a8119ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90aca00700 1 -- 172.21.15.1:0/3778803673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90a8070870 msgr2=0x7f90a8119ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90aca00700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90a8070870 0x7f90a8119ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.788+0000 7f90aca00700 1 -- 172.21.15.1:0/3778803673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9094005040 con 0x7f90a8071f60 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.789+0000 7f90aca00700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8071f60 0x7f90a811a400 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9094004f50 tx=0x7f9094009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.789+0000 7f90a5600700 1 -- 172.21.15.1:0/3778803673 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f909400e040 con 0x7f90a8071f60 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.789+0000 7f90a5600700 1 -- 172.21.15.1:0/3778803673 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f909400bc10 con 0x7f90a8071f60 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.789+0000 7f90a5600700 1 -- 172.21.15.1:0/3778803673 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f909401c630 con 0x7f90a8071f60 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.789+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a8113f80 con 0x7f90a8071f60 2024-02-28T16:46:26.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.789+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a81144c0 con 0x7f90a8071f60 2024-02-28T16:46:26.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.790+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a80fe260 con 0x7f90a8071f60 2024-02-28T16:46:26.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.791+0000 7f90a5600700 1 -- 172.21.15.1:0/3778803673 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f909400bd80 con 0x7f90a8071f60 2024-02-28T16:46:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.792+0000 7f90a5600700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f909006d7f0 0x7f909006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.792+0000 7f90a5600700 1 -- 172.21.15.1:0/3778803673 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f909408dfd0 con 0x7f90a8071f60 2024-02-28T16:46:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.792+0000 7f90ad400700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f909006d7f0 0x7f909006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.793+0000 7f90ad400700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f909006d7f0 0x7f909006fca0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:26.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:26.797+0000 7f90a5600700 1 -- 172.21.15.1:0/3778803673 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f909405b3d0 con 0x7f90a8071f60 2024-02-28T16:46:26.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:26 smithi066 ceph-mon[85781]: from='client.34702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:26.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:26 smithi066 ceph-mon[85781]: pgmap v359: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:26.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:26 smithi066 ceph-mon[85781]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:26.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:46:26.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:26 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2297573064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:46:26.997 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:26 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2521445075' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:46:27.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:26 smithi001 ceph-mon[112940]: from='client.34702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:27.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:26 smithi001 ceph-mon[112940]: pgmap v359: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:27.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:26 smithi001 ceph-mon[112940]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:27.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:46:27.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:26 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2297573064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:46:27.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:26 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2521445075' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:46:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.116+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f90a81154a0 con 0x7f90a8071f60 2024-02-28T16:46:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.117+0000 7f90a5600700 1 -- 172.21.15.1:0/3778803673 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f9094057020 con 0x7f90a8071f60 2024-02-28T16:46:27.103 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:46:27.103 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:46:27.103 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:46:27.104 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:46:27.104 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:46:27.104 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:46:27.104 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:46:27.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f909006d7f0 msgr2=0x7f909006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f909006d7f0 0x7f909006fca0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c008040 comp rx=0 tx=0).stop 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8071f60 msgr2=0x7f90a811a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8071f60 0x7f90a811a400 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9094004f50 tx=0x7f9094009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 shutdown_connections 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f909006d7f0 0x7f909006fca0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90a8070870 0x7f90a8119ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 --2- 172.21.15.1:0/3778803673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a8071f60 0x7f90a811a400 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 >> 172.21.15.1:0/3778803673 conn(0x7f90a80f8940 msgr2=0x7f90a80fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 shutdown_connections 2024-02-28T16:46:27.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:27.120+0000 7f90ae800700 1 -- 172.21.15.1:0/3778803673 wait complete. 2024-02-28T16:46:27.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:27 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3778803673' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:46:28.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:27 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3778803673' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:46:28.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:28 smithi066 ceph-mon[85781]: from='client.34718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:28.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:28 smithi066 ceph-mon[85781]: pgmap v360: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:29.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:28 smithi001 ceph-mon[112940]: from='client.34718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:29.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:28 smithi001 ceph-mon[112940]: pgmap v360: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:30.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:30 smithi066 ceph-mon[85781]: pgmap v361: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:31.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:30 smithi001 ceph-mon[112940]: pgmap v361: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:32.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:32 smithi066 ceph-mon[85781]: pgmap v362: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:33.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:32 smithi001 ceph-mon[112940]: pgmap v362: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:34.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:34 smithi066 ceph-mon[85781]: pgmap v363: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:35.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:34 smithi001 ceph-mon[112940]: pgmap v363: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:36.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:36 smithi066 ceph-mon[85781]: pgmap v364: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:37.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:36 smithi001 ceph-mon[112940]: pgmap v364: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:38.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:38 smithi066 ceph-mon[85781]: pgmap v365: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:39.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:38 smithi001 ceph-mon[112940]: pgmap v365: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:40.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:40 smithi066 ceph-mon[85781]: pgmap v366: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:41.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:40 smithi001 ceph-mon[112940]: pgmap v366: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:41.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:46:42.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:46:42.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:42 smithi066 ceph-mon[85781]: pgmap v367: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:43.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:42 smithi001 ceph-mon[112940]: pgmap v367: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:44.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:44 smithi066 ceph-mon[85781]: pgmap v368: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:45.018 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:44 smithi001 ceph-mon[112940]: pgmap v368: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:45.996 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:45 smithi066 ceph-mon[85781]: pgmap v369: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:46.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:45 smithi001 ceph-mon[112940]: pgmap v369: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:48.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:47 smithi066 ceph-mon[85781]: pgmap v370: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:48.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:47 smithi001 ceph-mon[112940]: pgmap v370: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:50.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:49 smithi066 ceph-mon[85781]: pgmap v371: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:50.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:49 smithi001 ceph-mon[112940]: pgmap v371: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:52.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:51 smithi066 ceph-mon[85781]: pgmap v372: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:52.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:51 smithi001 ceph-mon[112940]: pgmap v372: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:54.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:53 smithi066 ceph-mon[85781]: pgmap v373: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:54.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:53 smithi001 ceph-mon[112940]: pgmap v373: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:56.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:55 smithi066 ceph-mon[85781]: pgmap v374: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:56.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:46:56.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:55 smithi001 ceph-mon[112940]: pgmap v374: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:56.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:46:57.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.292+0000 7f6a4b200700 1 -- 172.21.15.1:0/425367187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a44102f50 msgr2=0x7f6a44103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:57.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.292+0000 7f6a4b200700 1 --2- 172.21.15.1:0/425367187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a44102f50 0x7f6a44103320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a34004940 tx=0x7f6a34009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:57.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.292+0000 7f6a4b200700 1 -- 172.21.15.1:0/425367187 shutdown_connections 2024-02-28T16:46:57.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.292+0000 7f6a4b200700 1 --2- 172.21.15.1:0/425367187 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a441038f0 0x7f6a440fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.292+0000 7f6a4b200700 1 --2- 172.21.15.1:0/425367187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a44102f50 0x7f6a44103320 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.292+0000 7f6a4b200700 1 -- 172.21.15.1:0/425367187 >> 172.21.15.1:0/425367187 conn(0x7f6a440f8ae0 msgr2=0x7f6a440faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.292+0000 7f6a4b200700 1 -- 172.21.15.1:0/425367187 shutdown_connections 2024-02-28T16:46:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.292+0000 7f6a4b200700 1 -- 172.21.15.1:0/425367187 wait complete. 2024-02-28T16:46:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.293+0000 7f6a4b200700 1 Processor -- start 2024-02-28T16:46:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.293+0000 7f6a4b200700 1 -- start start 2024-02-28T16:46:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.294+0000 7f6a4b200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a44102f50 0x7f6a44115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.294+0000 7f6a4b200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a441038f0 0x7f6a44116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.294+0000 7f6a4b200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a44116780 con 0x7f6a441038f0 2024-02-28T16:46:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.294+0000 7f6a4b200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4410fad0 con 0x7f6a44102f50 2024-02-28T16:46:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.294+0000 7f6a43400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a441038f0 0x7f6a44116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.294+0000 7f6a43400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a441038f0 0x7f6a44116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58928/0 (socket says 172.21.15.1:58928) 2024-02-28T16:46:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.294+0000 7f6a43400700 1 -- 172.21.15.1:0/3974343741 learned_addr learned my addr 172.21.15.1:0/3974343741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.295+0000 7f6a43e00700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a44102f50 0x7f6a44115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.295+0000 7f6a43400700 1 -- 172.21.15.1:0/3974343741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a44102f50 msgr2=0x7f6a44115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.295+0000 7f6a43400700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a44102f50 0x7f6a44115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.295+0000 7f6a43400700 1 -- 172.21.15.1:0/3974343741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a34005040 con 0x7f6a441038f0 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.295+0000 7f6a43400700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a441038f0 0x7f6a44116130 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000c800 tx=0x7f6a30004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.296+0000 7f6a3be00700 1 -- 172.21.15.1:0/3974343741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a300106f0 con 0x7f6a441038f0 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.296+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4410fdb0 con 0x7f6a441038f0 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.296+0000 7f6a3be00700 1 -- 172.21.15.1:0/3974343741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a30010d20 con 0x7f6a441038f0 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.296+0000 7f6a3be00700 1 -- 172.21.15.1:0/3974343741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a300113a0 con 0x7f6a441038f0 2024-02-28T16:46:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.296+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a441102f0 con 0x7f6a441038f0 2024-02-28T16:46:57.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.297+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a440fc6e0 con 0x7f6a441038f0 2024-02-28T16:46:57.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.299+0000 7f6a3be00700 1 -- 172.21.15.1:0/3974343741 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6a30012400 con 0x7f6a441038f0 2024-02-28T16:46:57.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.300+0000 7f6a3be00700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a2806d7e0 0x7f6a2806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.300+0000 7f6a3be00700 1 -- 172.21.15.1:0/3974343741 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6a3008ce80 con 0x7f6a441038f0 2024-02-28T16:46:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.300+0000 7f6a43e00700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a2806d7e0 0x7f6a2806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.301+0000 7f6a43e00700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a2806d7e0 0x7f6a2806fc90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6a34004f90 tx=0x7f6a34009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:57.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.305+0000 7f6a3be00700 1 -- 172.21.15.1:0/3974343741 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a300566b0 con 0x7f6a441038f0 2024-02-28T16:46:57.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.552+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a441112e0 con 0x7f6a2806d7e0 2024-02-28T16:46:57.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.554+0000 7f6a3be00700 1 -- 172.21.15.1:0/3974343741 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6a441112e0 con 0x7f6a2806d7e0 2024-02-28T16:46:57.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a2806d7e0 msgr2=0x7f6a2806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:57.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a2806d7e0 0x7f6a2806fc90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6a34004f90 tx=0x7f6a34009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:57.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a441038f0 msgr2=0x7f6a44116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:57.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a441038f0 0x7f6a44116130 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000c800 tx=0x7f6a30004470 comp rx=0 tx=0).stop 2024-02-28T16:46:57.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 shutdown_connections 2024-02-28T16:46:57.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a2806d7e0 0x7f6a2806fc90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a44102f50 0x7f6a44115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 --2- 172.21.15.1:0/3974343741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a441038f0 0x7f6a44116130 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 >> 172.21.15.1:0/3974343741 conn(0x7f6a440f8ae0 msgr2=0x7f6a440fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:57.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 shutdown_connections 2024-02-28T16:46:57.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.557+0000 7f6a4b200700 1 -- 172.21.15.1:0/3974343741 wait complete. 2024-02-28T16:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:46:57.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.718+0000 7f99ccc00700 1 -- 172.21.15.1:0/4021535813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 msgr2=0x7f99c8105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:57.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.718+0000 7f99ccc00700 1 --2- 172.21.15.1:0/4021535813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 0x7f99c8105540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f99b0004940 tx=0x7f99b0009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:57.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.718+0000 7f99ccc00700 1 -- 172.21.15.1:0/4021535813 shutdown_connections 2024-02-28T16:46:57.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.718+0000 7f99ccc00700 1 --2- 172.21.15.1:0/4021535813 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99c8105b10 0x7f99c80ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.718+0000 7f99ccc00700 1 --2- 172.21.15.1:0/4021535813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 0x7f99c8105540 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.718+0000 7f99ccc00700 1 -- 172.21.15.1:0/4021535813 >> 172.21.15.1:0/4021535813 conn(0x7f99c80face0 msgr2=0x7f99c80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:57.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.719+0000 7f99ccc00700 1 -- 172.21.15.1:0/4021535813 shutdown_connections 2024-02-28T16:46:57.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.719+0000 7f99ccc00700 1 -- 172.21.15.1:0/4021535813 wait complete. 2024-02-28T16:46:57.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.719+0000 7f99ccc00700 1 Processor -- start 2024-02-28T16:46:57.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.720+0000 7f99ccc00700 1 -- start start 2024-02-28T16:46:57.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.720+0000 7f99ccc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 0x7f99c8115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:57.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.720+0000 7f99ccc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99c8105b10 0x7f99c81161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:57.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.720+0000 7f99ccc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c81167f0 con 0x7f99c8105170 2024-02-28T16:46:57.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.720+0000 7f99ccc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c810fb40 con 0x7f99c8105b10 2024-02-28T16:46:57.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.720+0000 7f99c5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 0x7f99c8115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:57.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.720+0000 7f99c5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 0x7f99c8115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58942/0 (socket says 172.21.15.1:58942) 2024-02-28T16:46:57.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.720+0000 7f99c5e00700 1 -- 172.21.15.1:0/247053003 learned_addr learned my addr 172.21.15.1:0/247053003 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:57.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.721+0000 7f99bd400700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99c8105b10 0x7f99c81161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:57.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.721+0000 7f99c5e00700 1 -- 172.21.15.1:0/247053003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99c8105b10 msgr2=0x7f99c81161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:57.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.721+0000 7f99c5e00700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99c8105b10 0x7f99c81161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.721+0000 7f99c5e00700 1 -- 172.21.15.1:0/247053003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b0005040 con 0x7f99c8105170 2024-02-28T16:46:57.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.721+0000 7f99c5e00700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 0x7f99c8115c60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009fd0 tx=0x7f99b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:57.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.721+0000 7f99bea00700 1 -- 172.21.15.1:0/247053003 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b000e040 con 0x7f99c8105170 2024-02-28T16:46:57.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.722+0000 7f99bea00700 1 -- 172.21.15.1:0/247053003 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99b000bbf0 con 0x7f99c8105170 2024-02-28T16:46:57.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.722+0000 7f99bea00700 1 -- 172.21.15.1:0/247053003 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b001c3f0 con 0x7f99c8105170 2024-02-28T16:46:57.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.722+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c810fdc0 con 0x7f99c8105170 2024-02-28T16:46:57.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.722+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c8110300 con 0x7f99c8105170 2024-02-28T16:46:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.723+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c8102480 con 0x7f99c8105170 2024-02-28T16:46:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.724+0000 7f99bea00700 1 -- 172.21.15.1:0/247053003 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f99b000bd60 con 0x7f99c8105170 2024-02-28T16:46:57.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.725+0000 7f99bea00700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99a806d7f0 0x7f99a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.725+0000 7f99bea00700 1 -- 172.21.15.1:0/247053003 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f99b008e1c0 con 0x7f99c8105170 2024-02-28T16:46:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.726+0000 7f99bd400700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99a806d7f0 0x7f99a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.726+0000 7f99bd400700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99a806d7f0 0x7f99a806fca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f99c8111540 tx=0x7f99b8009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:57.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.730+0000 7f99bea00700 1 -- 172.21.15.1:0/247053003 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99b0057a70 con 0x7f99c8105170 2024-02-28T16:46:57.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.972+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99c804bf70 con 0x7f99a806d7f0 2024-02-28T16:46:57.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.974+0000 7f99bea00700 1 -- 172.21.15.1:0/247053003 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f99b8004e30 con 0x7f99a806d7f0 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99a806d7f0 msgr2=0x7f99a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99a806d7f0 0x7f99a806fca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f99c8111540 tx=0x7f99b8009ee0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 msgr2=0x7f99c8115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 0x7f99c8115c60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009fd0 tx=0x7f99b0009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 shutdown_connections 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99a806d7f0 0x7f99a806fca0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99c8105170 0x7f99c8115c60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 --2- 172.21.15.1:0/247053003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99c8105b10 0x7f99c81161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 >> 172.21.15.1:0/247053003 conn(0x7f99c80face0 msgr2=0x7f99c80fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:57.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 shutdown_connections 2024-02-28T16:46:57.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:57.977+0000 7f99ccc00700 1 -- 172.21.15.1:0/247053003 wait complete. 2024-02-28T16:46:58.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.131+0000 7ff822c00700 1 -- 172.21.15.1:0/897088272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105170 msgr2=0x7ff81c105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:58.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.131+0000 7ff822c00700 1 --2- 172.21.15.1:0/897088272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105170 0x7ff81c105540 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff80c004940 tx=0x7ff80c009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:58.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.131+0000 7ff822c00700 1 -- 172.21.15.1:0/897088272 shutdown_connections 2024-02-28T16:46:58.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.131+0000 7ff822c00700 1 --2- 172.21.15.1:0/897088272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c105b10 0x7ff81c0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.131+0000 7ff822c00700 1 --2- 172.21.15.1:0/897088272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105170 0x7ff81c105540 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.131+0000 7ff822c00700 1 -- 172.21.15.1:0/897088272 >> 172.21.15.1:0/897088272 conn(0x7ff81c0face0 msgr2=0x7ff81c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:58.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.131+0000 7ff822c00700 1 -- 172.21.15.1:0/897088272 shutdown_connections 2024-02-28T16:46:58.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.131+0000 7ff822c00700 1 -- 172.21.15.1:0/897088272 wait complete. 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.132+0000 7ff822c00700 1 Processor -- start 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.132+0000 7ff822c00700 1 -- start start 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.133+0000 7ff822c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c105170 0x7ff81c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.133+0000 7ff822c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105b10 0x7ff81c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.133+0000 7ff822c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c1168a0 con 0x7ff81c105b10 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.133+0000 7ff822c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c10fb40 con 0x7ff81c105170 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.133+0000 7ff81b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105b10 0x7ff81c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.133+0000 7ff81b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105b10 0x7ff81c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58964/0 (socket says 172.21.15.1:58964) 2024-02-28T16:46:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.133+0000 7ff81b400700 1 -- 172.21.15.1:0/3140313781 learned_addr learned my addr 172.21.15.1:0/3140313781 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:58.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.133+0000 7ff81be00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c105170 0x7ff81c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:58.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.134+0000 7ff81b400700 1 -- 172.21.15.1:0/3140313781 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c105170 msgr2=0x7ff81c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:58.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.134+0000 7ff81b400700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c105170 0x7ff81c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.134+0000 7ff81b400700 1 -- 172.21.15.1:0/3140313781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c005040 con 0x7ff81c105b10 2024-02-28T16:46:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.134+0000 7ff81b400700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105b10 0x7ff81c1161c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff80800b790 tx=0x7ff808009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.134+0000 7ff813e00700 1 -- 172.21.15.1:0/3140313781 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff80800f6f0 con 0x7ff81c105b10 2024-02-28T16:46:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.134+0000 7ff813e00700 1 -- 172.21.15.1:0/3140313781 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff80800fd20 con 0x7ff81c105b10 2024-02-28T16:46:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.134+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81c10fe20 con 0x7ff81c105b10 2024-02-28T16:46:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.134+0000 7ff813e00700 1 -- 172.21.15.1:0/3140313781 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8080103a0 con 0x7ff81c105b10 2024-02-28T16:46:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.135+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c110360 con 0x7ff81c105b10 2024-02-28T16:46:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.136+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81c0fe900 con 0x7ff81c105b10 2024-02-28T16:46:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.137+0000 7ff813e00700 1 -- 172.21.15.1:0/3140313781 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff808011400 con 0x7ff81c105b10 2024-02-28T16:46:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.138+0000 7ff813e00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff80006d7f0 0x7ff80006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.138+0000 7ff813e00700 1 -- 172.21.15.1:0/3140313781 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff80808cc90 con 0x7ff81c105b10 2024-02-28T16:46:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.138+0000 7ff81be00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff80006d7f0 0x7ff80006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.139+0000 7ff81be00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff80006d7f0 0x7ff80006fca0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00b020 tx=0x7ff80c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:58.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.142+0000 7ff813e00700 1 -- 172.21.15.1:0/3140313781 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff80805a090 con 0x7ff81c105b10 2024-02-28T16:46:58.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:57 smithi066 ceph-mon[85781]: pgmap v375: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:58.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:57 smithi066 ceph-mon[85781]: from='client.34726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:58.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:57 smithi001 ceph-mon[112940]: pgmap v375: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:46:58.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:57 smithi001 ceph-mon[112940]: from='client.34726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:58.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.376+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff81c111260 con 0x7ff80006d7f0 2024-02-28T16:46:58.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.403+0000 7ff813e00700 1 -- 172.21.15.1:0/3140313781 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff81c111260 con 0x7ff80006d7f0 2024-02-28T16:46:58.391 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:46:58.391 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (13m) 5m ago 27m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:46:58.391 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (9m) 5m ago 27m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (9m) 6m ago 26m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (12m) 5m ago 26m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (5m) 5m ago 20m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (5m) 5m ago 20m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (20m) 6m ago 20m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (20m) 6m ago 20m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (15m) 5m ago 29m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (14m) 6m ago 26m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (11m) 5m ago 29m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (10m) 6m ago 25m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (14m) 5m ago 27m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (14m) 6m ago 26m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (9m) 5m ago 25m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:46:58.392 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (8m) 5m ago 24m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:46:58.393 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (8m) 5m ago 23m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:46:58.393 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (7m) 6m ago 23m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:46:58.393 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (6m) 6m ago 22m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:46:58.393 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (6m) 6m ago 22m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:46:58.393 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (13m) 5m ago 26m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:46:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff80006d7f0 msgr2=0x7ff80006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff80006d7f0 0x7ff80006fca0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00b020 tx=0x7ff80c009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105b10 msgr2=0x7ff81c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105b10 0x7ff81c1161c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff80800b790 tx=0x7ff808009220 comp rx=0 tx=0).stop 2024-02-28T16:46:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 shutdown_connections 2024-02-28T16:46:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff80006d7f0 0x7ff80006fca0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff81c105170 0x7ff81c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 --2- 172.21.15.1:0/3140313781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff81c105b10 0x7ff81c1161c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.407+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 >> 172.21.15.1:0/3140313781 conn(0x7ff81c0face0 msgr2=0x7ff81c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.408+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 shutdown_connections 2024-02-28T16:46:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.408+0000 7ff822c00700 1 -- 172.21.15.1:0/3140313781 wait complete. 2024-02-28T16:46:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.566+0000 7f3871c00700 1 -- 172.21.15.1:0/3883499769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fe910 msgr2=0x7f386c106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.566+0000 7f3871c00700 1 --2- 172.21.15.1:0/3883499769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fe910 0x7f386c106fe0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3858004970 tx=0x7f3858009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:58.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.566+0000 7f3871c00700 1 -- 172.21.15.1:0/3883499769 shutdown_connections 2024-02-28T16:46:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.566+0000 7f3871c00700 1 --2- 172.21.15.1:0/3883499769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fe910 0x7f386c106fe0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.566+0000 7f3871c00700 1 --2- 172.21.15.1:0/3883499769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f386c0fdf70 0x7f386c0fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.566+0000 7f3871c00700 1 -- 172.21.15.1:0/3883499769 >> 172.21.15.1:0/3883499769 conn(0x7f386c0f7a10 msgr2=0x7f386c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.566+0000 7f3871c00700 1 -- 172.21.15.1:0/3883499769 shutdown_connections 2024-02-28T16:46:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.567+0000 7f3871c00700 1 -- 172.21.15.1:0/3883499769 wait complete. 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.567+0000 7f3871c00700 1 Processor -- start 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.568+0000 7f3871c00700 1 -- start start 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.568+0000 7f3871c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fdf70 0x7f386c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.568+0000 7f3871c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f386c0fe910 0x7f386c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.568+0000 7f3871c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c1168c0 con 0x7f386c0fdf70 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.568+0000 7f3871c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c10fb40 con 0x7f386c0fe910 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.568+0000 7f386aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fdf70 0x7f386c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.568+0000 7f386aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fdf70 0x7f386c115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58974/0 (socket says 172.21.15.1:58974) 2024-02-28T16:46:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.568+0000 7f386aa00700 1 -- 172.21.15.1:0/236001930 learned_addr learned my addr 172.21.15.1:0/236001930 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f386a000700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f386c0fe910 0x7f386c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f386aa00700 1 -- 172.21.15.1:0/236001930 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f386c0fe910 msgr2=0x7f386c1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f386aa00700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f386c0fe910 0x7f386c1161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f386aa00700 1 -- 172.21.15.1:0/236001930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3858005040 con 0x7f386c0fdf70 2024-02-28T16:46:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f386aa00700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fdf70 0x7f386c115ca0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f385c00b790 tx=0x7f385c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f3862a00700 1 -- 172.21.15.1:0/236001930 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385c00f770 con 0x7f386c0fdf70 2024-02-28T16:46:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f3862a00700 1 -- 172.21.15.1:0/236001930 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f385c00fda0 con 0x7f386c0fdf70 2024-02-28T16:46:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386c10fdc0 con 0x7f386c0fdf70 2024-02-28T16:46:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.569+0000 7f3862a00700 1 -- 172.21.15.1:0/236001930 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385c00d310 con 0x7f386c0fdf70 2024-02-28T16:46:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.570+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386c110300 con 0x7f386c0fdf70 2024-02-28T16:46:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.571+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386c106a80 con 0x7f386c0fdf70 2024-02-28T16:46:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.572+0000 7f3862a00700 1 -- 172.21.15.1:0/236001930 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f385c010420 con 0x7f386c0fdf70 2024-02-28T16:46:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.573+0000 7f3862a00700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f385006d7f0 0x7f385006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.573+0000 7f3862a00700 1 -- 172.21.15.1:0/236001930 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f385c08cfb0 con 0x7f386c0fdf70 2024-02-28T16:46:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.573+0000 7f386a000700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f385006d7f0 0x7f385006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.574+0000 7f386a000700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f385006d7f0 0x7f385006fca0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3858000c00 tx=0x7f3858009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.577+0000 7f3862a00700 1 -- 172.21.15.1:0/236001930 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f385c0c0d20 con 0x7f386c0fdf70 2024-02-28T16:46:58.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.887+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f386c111280 con 0x7f386c0fdf70 2024-02-28T16:46:58.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.888+0000 7f3862a00700 1 -- 172.21.15.1:0/236001930 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f385c05a310 con 0x7f386c0fdf70 2024-02-28T16:46:58.875 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:46:58.875 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:46:58.876 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:46:58.877 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:46:58.877 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:46:58.877 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:46:58.877 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:46:58.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f385006d7f0 msgr2=0x7f385006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:58.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f385006d7f0 0x7f385006fca0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3858000c00 tx=0x7f3858009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:58.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fdf70 msgr2=0x7f386c115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:58.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fdf70 0x7f386c115ca0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f385c00b790 tx=0x7f385c009250 comp rx=0 tx=0).stop 2024-02-28T16:46:58.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 shutdown_connections 2024-02-28T16:46:58.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f385006d7f0 0x7f385006fca0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f386c0fdf70 0x7f386c115ca0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 --2- 172.21.15.1:0/236001930 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f386c0fe910 0x7f386c1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:58.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.890+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 >> 172.21.15.1:0/236001930 conn(0x7f386c0f7a10 msgr2=0x7f386c103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:58.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.891+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 shutdown_connections 2024-02-28T16:46:58.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:58.891+0000 7f3871c00700 1 -- 172.21.15.1:0/236001930 wait complete. 2024-02-28T16:46:59.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.048+0000 7efff5800700 1 -- 172.21.15.1:0/345661790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff0071f60 msgr2=0x7efff0070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.048+0000 7efff5800700 1 --2- 172.21.15.1:0/345661790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff0071f60 0x7efff0070330 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7effe0004970 tx=0x7effe0009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.048+0000 7efff5800700 1 -- 172.21.15.1:0/345661790 shutdown_connections 2024-02-28T16:46:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.048+0000 7efff5800700 1 --2- 172.21.15.1:0/345661790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efff0070870 0x7efff0070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.048+0000 7efff5800700 1 --2- 172.21.15.1:0/345661790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff0071f60 0x7efff0070330 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.048+0000 7efff5800700 1 -- 172.21.15.1:0/345661790 >> 172.21.15.1:0/345661790 conn(0x7efff00f8ae0 msgr2=0x7efff00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.049+0000 7efff5800700 1 -- 172.21.15.1:0/345661790 shutdown_connections 2024-02-28T16:46:59.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.049+0000 7efff5800700 1 -- 172.21.15.1:0/345661790 wait complete. 2024-02-28T16:46:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.050+0000 7efff5800700 1 Processor -- start 2024-02-28T16:46:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.050+0000 7efff5800700 1 -- start start 2024-02-28T16:46:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.050+0000 7efff5800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efff0070870 0x7efff011a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.050+0000 7efff5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff011a730 0x7efff0114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.050+0000 7efff5800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff011acb0 con 0x7efff011a730 2024-02-28T16:46:59.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.050+0000 7efff5800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff011ae20 con 0x7efff0070870 2024-02-28T16:46:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.051+0000 7effee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff011a730 0x7efff0114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.051+0000 7effee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff011a730 0x7efff0114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59000/0 (socket says 172.21.15.1:59000) 2024-02-28T16:46:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.051+0000 7effee000700 1 -- 172.21.15.1:0/961764710 learned_addr learned my addr 172.21.15.1:0/961764710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.051+0000 7effee000700 1 -- 172.21.15.1:0/961764710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efff0070870 msgr2=0x7efff011a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.051+0000 7effee000700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efff0070870 0x7efff011a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.051+0000 7effee000700 1 -- 172.21.15.1:0/961764710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe0005040 con 0x7efff011a730 2024-02-28T16:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.051+0000 7effee000700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff011a730 0x7efff0114090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7effdc00c830 tx=0x7effdc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.052+0000 7effe6a00700 1 -- 172.21.15.1:0/961764710 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effdc00a760 con 0x7efff011a730 2024-02-28T16:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.052+0000 7effe6a00700 1 -- 172.21.15.1:0/961764710 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effdc00ad90 con 0x7efff011a730 2024-02-28T16:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.052+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff0114690 con 0x7efff011a730 2024-02-28T16:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.052+0000 7effe6a00700 1 -- 172.21.15.1:0/961764710 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effdc0113a0 con 0x7efff011a730 2024-02-28T16:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.052+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff0114bd0 con 0x7efff011a730 2024-02-28T16:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.053+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff0114eb0 con 0x7efff011a730 2024-02-28T16:46:59.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.054+0000 7effe6a00700 1 -- 172.21.15.1:0/961764710 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7effdc012400 con 0x7efff011a730 2024-02-28T16:46:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.055+0000 7effe6a00700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effd406d7f0 0x7effd406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.055+0000 7effe6a00700 1 -- 172.21.15.1:0/961764710 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7effdc08cfd0 con 0x7efff011a730 2024-02-28T16:46:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.055+0000 7effeea00700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effd406d7f0 0x7effd406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:59.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.056+0000 7effeea00700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effd406d7f0 0x7effd406fca0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7effe0000c00 tx=0x7effe0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:59.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.060+0000 7effe6a00700 1 -- 172.21.15.1:0/961764710 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7effdc0c0d20 con 0x7efff011a730 2024-02-28T16:46:59.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:59 smithi066 ceph-mon[85781]: from='client.34730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:59.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:59 smithi066 ceph-mon[85781]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:59.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:46:59 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/236001930' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:46:59.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:59 smithi001 ceph-mon[112940]: from='client.34730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:59.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:59 smithi001 ceph-mon[112940]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:46:59.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:46:59 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/236001930' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:46:59.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.327+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efff004bed0 con 0x7efff011a730 2024-02-28T16:46:59.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.328+0000 7effe6a00700 1 -- 172.21.15.1:0/961764710 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7effdc05a330 con 0x7efff011a730 2024-02-28T16:46:59.316 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:46:59.316 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:46:59.316 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:46:59.316 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:46:59.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:46:59.316 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:46:59.316 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:46:59.317 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:46:59.318 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:46:59.319 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:46:59.319 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:46:59.319 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:46:59.319 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:46:59.319 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:46:59.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effd406d7f0 msgr2=0x7effd406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effd406d7f0 0x7effd406fca0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7effe0000c00 tx=0x7effe0009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:59.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff011a730 msgr2=0x7efff0114090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff011a730 0x7efff0114090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7effdc00c830 tx=0x7effdc004470 comp rx=0 tx=0).stop 2024-02-28T16:46:59.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 shutdown_connections 2024-02-28T16:46:59.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effd406d7f0 0x7effd406fca0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efff0070870 0x7efff011a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 --2- 172.21.15.1:0/961764710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff011a730 0x7efff0114090 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 >> 172.21.15.1:0/961764710 conn(0x7efff00f8ae0 msgr2=0x7efff010c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:59.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 shutdown_connections 2024-02-28T16:46:59.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.332+0000 7efff5800700 1 -- 172.21.15.1:0/961764710 wait complete. 2024-02-28T16:46:59.321 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:46:59.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.495+0000 7f74c7400700 1 -- 172.21.15.1:0/1887184907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 msgr2=0x7f74c0070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.495+0000 7f74c7400700 1 --2- 172.21.15.1:0/1887184907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 0x7f74c0070ce0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f74ac004970 tx=0x7f74ac009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:59.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.495+0000 7f74c7400700 1 -- 172.21.15.1:0/1887184907 shutdown_connections 2024-02-28T16:46:59.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.496+0000 7f74c7400700 1 --2- 172.21.15.1:0/1887184907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 0x7f74c0070ce0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.496+0000 7f74c7400700 1 --2- 172.21.15.1:0/1887184907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74c0071f60 0x7f74c0070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.496+0000 7f74c7400700 1 -- 172.21.15.1:0/1887184907 >> 172.21.15.1:0/1887184907 conn(0x7f74c00f8af0 msgr2=0x7f74c00faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:59.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.496+0000 7f74c7400700 1 -- 172.21.15.1:0/1887184907 shutdown_connections 2024-02-28T16:46:59.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.496+0000 7f74c7400700 1 -- 172.21.15.1:0/1887184907 wait complete. 2024-02-28T16:46:59.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.496+0000 7f74c7400700 1 Processor -- start 2024-02-28T16:46:59.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.497+0000 7f74c7400700 1 -- start start 2024-02-28T16:46:59.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.497+0000 7f74c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 0x7f74c011a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.497+0000 7f74c7400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74c0071f60 0x7f74c011a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.497+0000 7f74c7400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c011ac20 con 0x7f74c0070870 2024-02-28T16:46:59.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.497+0000 7f74c7400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0113f30 con 0x7f74c0071f60 2024-02-28T16:46:59.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.497+0000 7f74c4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 0x7f74c011a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:59.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.497+0000 7f74c4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 0x7f74c011a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59024/0 (socket says 172.21.15.1:59024) 2024-02-28T16:46:59.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.497+0000 7f74c4a00700 1 -- 172.21.15.1:0/872096472 learned_addr learned my addr 172.21.15.1:0/872096472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:59.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74bfe00700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74c0071f60 0x7f74c011a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:59.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74c4a00700 1 -- 172.21.15.1:0/872096472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74c0071f60 msgr2=0x7f74c011a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74c4a00700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74c0071f60 0x7f74c011a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74c4a00700 1 -- 172.21.15.1:0/872096472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac005040 con 0x7f74c0070870 2024-02-28T16:46:59.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74c4a00700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 0x7f74c011a090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f74b4002a50 tx=0x7f74b40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74bcc00700 1 -- 172.21.15.1:0/872096472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74b4007cc0 con 0x7f74c0070870 2024-02-28T16:46:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74bcc00700 1 -- 172.21.15.1:0/872096472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74b400f040 con 0x7f74c0070870 2024-02-28T16:46:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74bcc00700 1 -- 172.21.15.1:0/872096472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74b4009640 con 0x7f74c0070870 2024-02-28T16:46:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.498+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c0114210 con 0x7f74c0070870 2024-02-28T16:46:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.499+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c0114750 con 0x7f74c0070870 2024-02-28T16:46:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.500+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c00fe410 con 0x7f74c0070870 2024-02-28T16:46:59.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.501+0000 7f74bcc00700 1 -- 172.21.15.1:0/872096472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f74b40097a0 con 0x7f74c0070870 2024-02-28T16:46:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.502+0000 7f74bcc00700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f74a806d7f0 0x7f74a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.502+0000 7f74bcc00700 1 -- 172.21.15.1:0/872096472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f74b408cc60 con 0x7f74c0070870 2024-02-28T16:46:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.502+0000 7f74bfe00700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f74a806d7f0 0x7f74a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:59.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.503+0000 7f74bfe00700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f74a806d7f0 0x7f74a806fca0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f74ac000c00 tx=0x7f74ac009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.506+0000 7f74bcc00700 1 -- 172.21.15.1:0/872096472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f74b4059fc0 con 0x7f74c0070870 2024-02-28T16:46:59.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.753+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f74c0115670 con 0x7f74a806d7f0 2024-02-28T16:46:59.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.756+0000 7f74bcc00700 1 -- 172.21.15.1:0/872096472 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f74c0115670 con 0x7f74a806d7f0 2024-02-28T16:46:59.744 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:46:59.744 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:46:59.744 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:46:59.744 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:46:59.744 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:46:59.744 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:46:59.745 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:46:59.745 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:46:59.745 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:46:59.745 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:46:59.745 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:46:59.745 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:46:59.745 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:46:59.745 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:46:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.759+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f74a806d7f0 msgr2=0x7f74a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.759+0000 7f74c7400700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f74a806d7f0 0x7f74a806fca0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f74ac000c00 tx=0x7f74ac009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.759+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 msgr2=0x7f74c011a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.759+0000 7f74c7400700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 0x7f74c011a090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f74b4002a50 tx=0x7f74b40046b0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.760+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 shutdown_connections 2024-02-28T16:46:59.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.760+0000 7f74c7400700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f74a806d7f0 0x7f74a806fca0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.760+0000 7f74c7400700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74c0070870 0x7f74c011a090 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.760+0000 7f74c7400700 1 --2- 172.21.15.1:0/872096472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f74c0071f60 0x7f74c011a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.760+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 >> 172.21.15.1:0/872096472 conn(0x7f74c00f8af0 msgr2=0x7f74c00fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:59.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.760+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 shutdown_connections 2024-02-28T16:46:59.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.760+0000 7f74c7400700 1 -- 172.21.15.1:0/872096472 wait complete. 2024-02-28T16:46:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.920+0000 7f8e6ea00700 1 -- 172.21.15.1:0/1849782560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e681038e0 msgr2=0x7f8e680fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.920+0000 7f8e6ea00700 1 --2- 172.21.15.1:0/1849782560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e681038e0 0x7f8e680fcf50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8e50004940 tx=0x7f8e50009e30 comp rx=0 tx=0).stop 2024-02-28T16:46:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.920+0000 7f8e6ea00700 1 -- 172.21.15.1:0/1849782560 shutdown_connections 2024-02-28T16:46:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.920+0000 7f8e6ea00700 1 --2- 172.21.15.1:0/1849782560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e681038e0 0x7f8e680fcf50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.920+0000 7f8e6ea00700 1 --2- 172.21.15.1:0/1849782560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e68102f40 0x7f8e68103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.920+0000 7f8e6ea00700 1 -- 172.21.15.1:0/1849782560 >> 172.21.15.1:0/1849782560 conn(0x7f8e680f8af0 msgr2=0x7f8e680faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:46:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.920+0000 7f8e6ea00700 1 -- 172.21.15.1:0/1849782560 shutdown_connections 2024-02-28T16:46:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.920+0000 7f8e6ea00700 1 -- 172.21.15.1:0/1849782560 wait complete. 2024-02-28T16:46:59.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.921+0000 7f8e6ea00700 1 Processor -- start 2024-02-28T16:46:59.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.921+0000 7f8e6ea00700 1 -- start start 2024-02-28T16:46:59.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.921+0000 7f8e6ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e68102f40 0x7f8e68113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e6ea00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e681038e0 0x7f8e68113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e6ea00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e681145f0 con 0x7f8e68102f40 2024-02-28T16:46:59.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e6ea00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6810d920 con 0x7f8e681038e0 2024-02-28T16:46:59.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e67e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e68102f40 0x7f8e68113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:59.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e67e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e68102f40 0x7f8e68113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59046/0 (socket says 172.21.15.1:59046) 2024-02-28T16:46:59.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e67e00700 1 -- 172.21.15.1:0/3663438427 learned_addr learned my addr 172.21.15.1:0/3663438427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:46:59.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e5fe00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e681038e0 0x7f8e68113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:59.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e67e00700 1 -- 172.21.15.1:0/3663438427 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e681038e0 msgr2=0x7f8e68113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:46:59.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e67e00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e681038e0 0x7f8e68113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:46:59.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.922+0000 7f8e67e00700 1 -- 172.21.15.1:0/3663438427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e50005040 con 0x7f8e68102f40 2024-02-28T16:46:59.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.923+0000 7f8e67e00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e68102f40 0x7f8e68113a60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800c8c0 tx=0x7f8e580044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:59.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.923+0000 7f8e64c00700 1 -- 172.21.15.1:0/3663438427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e5800a980 con 0x7f8e68102f40 2024-02-28T16:46:59.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.923+0000 7f8e64c00700 1 -- 172.21.15.1:0/3663438427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e5800ce60 con 0x7f8e68102f40 2024-02-28T16:46:59.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.923+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6810dc00 con 0x7f8e68102f40 2024-02-28T16:46:59.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.923+0000 7f8e64c00700 1 -- 172.21.15.1:0/3663438427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e580115e0 con 0x7f8e68102f40 2024-02-28T16:46:59.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.923+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6810e140 con 0x7f8e68102f40 2024-02-28T16:46:59.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.924+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e68105410 con 0x7f8e68102f40 2024-02-28T16:46:59.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.925+0000 7f8e64c00700 1 -- 172.21.15.1:0/3663438427 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8e58012400 con 0x7f8e68102f40 2024-02-28T16:46:59.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.926+0000 7f8e64c00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4c06d7f0 0x7f8e4c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:46:59.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.927+0000 7f8e64c00700 1 -- 172.21.15.1:0/3663438427 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8e5808cfa0 con 0x7f8e68102f40 2024-02-28T16:46:59.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.927+0000 7f8e5fe00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4c06d7f0 0x7f8e4c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:46:59.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.928+0000 7f8e5fe00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4c06d7f0 0x7f8e4c06fca0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8e6810f290 tx=0x7f8e50009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:46:59.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:46:59.931+0000 7f8e64c00700 1 -- 172.21.15.1:0/3663438427 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8e5805a3a0 con 0x7f8e68102f40 2024-02-28T16:47:00.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.245+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8e6804bed0 con 0x7f8e68102f40 2024-02-28T16:47:00.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.245+0000 7f8e64c00700 1 -- 172.21.15.1:0/3663438427 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f8e580126b0 con 0x7f8e68102f40 2024-02-28T16:47:00.233 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:47:00.233 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:47:00.233 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:47:00.233 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:47:00.233 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:47:00.234 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:47:00.234 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:47:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4c06d7f0 msgr2=0x7f8e4c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4c06d7f0 0x7f8e4c06fca0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8e6810f290 tx=0x7f8e50009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e68102f40 msgr2=0x7f8e68113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e68102f40 0x7f8e68113a60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800c8c0 tx=0x7f8e580044c0 comp rx=0 tx=0).stop 2024-02-28T16:47:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 shutdown_connections 2024-02-28T16:47:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4c06d7f0 0x7f8e4c06fca0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e68102f40 0x7f8e68113a60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:00.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 --2- 172.21.15.1:0/3663438427 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e681038e0 0x7f8e68113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 >> 172.21.15.1:0/3663438427 conn(0x7f8e680f8af0 msgr2=0x7f8e680fa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 shutdown_connections 2024-02-28T16:47:00.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:00.248+0000 7f8e6ea00700 1 -- 172.21.15.1:0/3663438427 wait complete. 2024-02-28T16:47:00.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:00 smithi066 ceph-mon[85781]: pgmap v376: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:00.246 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:00 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/961764710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:47:00.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:00 smithi001 ceph-mon[112940]: pgmap v376: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:00.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:00 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/961764710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:47:01.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:01 smithi001 ceph-mon[112940]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:01.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:01 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3663438427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:47:01.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:01 smithi066 ceph-mon[85781]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:01.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:01 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3663438427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:47:02.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:02 smithi001 ceph-mon[112940]: pgmap v377: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:02.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:02 smithi066 ceph-mon[85781]: pgmap v377: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:04.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:04 smithi001 ceph-mon[112940]: pgmap v378: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:04.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:04 smithi066 ceph-mon[85781]: pgmap v378: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:06.268 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:06 smithi001 ceph-mon[112940]: pgmap v379: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:06.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:06 smithi066 ceph-mon[85781]: pgmap v379: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:08.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:08 smithi066 ceph-mon[85781]: pgmap v380: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:08.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:08 smithi001 ceph-mon[112940]: pgmap v380: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:10.299 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:10 smithi066 ceph-mon[85781]: pgmap v381: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:10.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:10 smithi001 ceph-mon[112940]: pgmap v381: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:11.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:47:11.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:47:12.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:12 smithi066 ceph-mon[85781]: pgmap v382: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:12.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:12 smithi001 ceph-mon[112940]: pgmap v382: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:14.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:14 smithi066 ceph-mon[85781]: pgmap v383: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:14.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:14 smithi001 ceph-mon[112940]: pgmap v383: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:16.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:16 smithi066 ceph-mon[85781]: pgmap v384: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:16.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:16 smithi001 ceph-mon[112940]: pgmap v384: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:18.496 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:18 smithi066 ceph-mon[85781]: pgmap v385: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:18.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:18 smithi001 ceph-mon[112940]: pgmap v385: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:20.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:20 smithi066 ceph-mon[85781]: pgmap v386: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:20.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:20 smithi001 ceph-mon[112940]: pgmap v386: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:22.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:22 smithi066 ceph-mon[85781]: pgmap v387: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:22.518 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:22 smithi001 ceph-mon[112940]: pgmap v387: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:24.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:24 smithi066 ceph-mon[85781]: pgmap v388: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:24.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:24 smithi001 ceph-mon[112940]: pgmap v388: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:26.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:26 smithi066 ceph-mon[85781]: pgmap v389: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:26.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:47:26.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:26 smithi001 ceph-mon[112940]: pgmap v389: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:26.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:47:28.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:28 smithi066 ceph-mon[85781]: pgmap v390: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:28.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:28 smithi001 ceph-mon[112940]: pgmap v390: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:30.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.419+0000 7fc9af400700 1 -- 172.21.15.1:0/509140063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 msgr2=0x7fc9a80ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:30.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.419+0000 7fc9af400700 1 --2- 172.21.15.1:0/509140063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 0x7fc9a80ff180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc994004940 tx=0x7fc994009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:30.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.419+0000 7fc9af400700 1 -- 172.21.15.1:0/509140063 shutdown_connections 2024-02-28T16:47:30.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.419+0000 7fc9af400700 1 --2- 172.21.15.1:0/509140063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 0x7fc9a80ff180 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.419+0000 7fc9af400700 1 --2- 172.21.15.1:0/509140063 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9a8105170 0x7fc9a8105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.419+0000 7fc9af400700 1 -- 172.21.15.1:0/509140063 >> 172.21.15.1:0/509140063 conn(0x7fc9a80face0 msgr2=0x7fc9a80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:30.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.419+0000 7fc9af400700 1 -- 172.21.15.1:0/509140063 shutdown_connections 2024-02-28T16:47:30.409 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:30 smithi001 ceph-mon[112940]: pgmap v391: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:30.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.419+0000 7fc9af400700 1 -- 172.21.15.1:0/509140063 wait complete. 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.420+0000 7fc9af400700 1 Processor -- start 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.420+0000 7fc9af400700 1 -- start start 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.421+0000 7fc9af400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9a8105170 0x7fc9a8115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.421+0000 7fc9af400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 0x7fc9a8116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.421+0000 7fc9af400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8116890 con 0x7fc9a8105b10 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.421+0000 7fc9af400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a810fba0 con 0x7fc9a8105170 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.421+0000 7fc9a7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 0x7fc9a8116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.421+0000 7fc9a7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 0x7fc9a8116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58318/0 (socket says 172.21.15.1:58318) 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.421+0000 7fc9a7e00700 1 -- 172.21.15.1:0/3939137578 learned_addr learned my addr 172.21.15.1:0/3939137578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.421+0000 7fc9aca00700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9a8105170 0x7fc9a8115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:30.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.422+0000 7fc9a7e00700 1 -- 172.21.15.1:0/3939137578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9a8105170 msgr2=0x7fc9a8115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.422+0000 7fc9a7e00700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9a8105170 0x7fc9a8115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.422+0000 7fc9a7e00700 1 -- 172.21.15.1:0/3939137578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc994005040 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.422+0000 7fc9a7e00700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 0x7fc9a8116240 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc994009030 tx=0x7fc994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.422+0000 7fc9a4c00700 1 -- 172.21.15.1:0/3939137578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99400e040 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.422+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a810fe20 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.422+0000 7fc9a4c00700 1 -- 172.21.15.1:0/3939137578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc99400bbf0 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.422+0000 7fc9a4c00700 1 -- 172.21.15.1:0/3939137578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99401c630 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.423+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a8110360 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.424+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a8107640 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.425+0000 7fc9a4c00700 1 -- 172.21.15.1:0/3939137578 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc99400bd60 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.425+0000 7fc9a4c00700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc99006d7f0 0x7fc99006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.426+0000 7fc9a4c00700 1 -- 172.21.15.1:0/3939137578 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc99408e2c0 con 0x7fc9a8105b10 2024-02-28T16:47:30.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.426+0000 7fc9aca00700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc99006d7f0 0x7fc99006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:30.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.427+0000 7fc9aca00700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc99006d7f0 0x7fc99006fca0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009fd0 tx=0x7fc99c009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:30.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.431+0000 7fc9a4c00700 1 -- 172.21.15.1:0/3939137578 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc99405b620 con 0x7fc9a8105b10 2024-02-28T16:47:30.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:30 smithi066 ceph-mon[85781]: pgmap v391: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.676+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9a81112e0 con 0x7fc99006d7f0 2024-02-28T16:47:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.678+0000 7fc9a4c00700 1 -- 172.21.15.1:0/3939137578 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc9a81112e0 con 0x7fc99006d7f0 2024-02-28T16:47:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.680+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc99006d7f0 msgr2=0x7fc99006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.680+0000 7fc9af400700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc99006d7f0 0x7fc99006fca0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009fd0 tx=0x7fc99c009f20 comp rx=0 tx=0).stop 2024-02-28T16:47:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.680+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 msgr2=0x7fc9a8116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.680+0000 7fc9af400700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 0x7fc9a8116240 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc994009030 tx=0x7fc994009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.681+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 shutdown_connections 2024-02-28T16:47:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.681+0000 7fc9af400700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc99006d7f0 0x7fc99006fca0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.681+0000 7fc9af400700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc9a8105170 0x7fc9a8115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.681+0000 7fc9af400700 1 --2- 172.21.15.1:0/3939137578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9a8105b10 0x7fc9a8116240 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.681+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 >> 172.21.15.1:0/3939137578 conn(0x7fc9a80face0 msgr2=0x7fc9a80fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.681+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 shutdown_connections 2024-02-28T16:47:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.681+0000 7fc9af400700 1 -- 172.21.15.1:0/3939137578 wait complete. 2024-02-28T16:47:30.684 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:47:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.844+0000 7f2400e00700 1 -- 172.21.15.1:0/2741990911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fdf70 msgr2=0x7f23fc0fe340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.844+0000 7f2400e00700 1 --2- 172.21.15.1:0/2741990911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fdf70 0x7f23fc0fe340 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f23ec004940 tx=0x7f23ec009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.845+0000 7f2400e00700 1 -- 172.21.15.1:0/2741990911 shutdown_connections 2024-02-28T16:47:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.845+0000 7f2400e00700 1 --2- 172.21.15.1:0/2741990911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23fc0fe910 0x7f23fc106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.845+0000 7f2400e00700 1 --2- 172.21.15.1:0/2741990911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fdf70 0x7f23fc0fe340 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.845+0000 7f2400e00700 1 -- 172.21.15.1:0/2741990911 >> 172.21.15.1:0/2741990911 conn(0x7f23fc0f79d0 msgr2=0x7f23fc0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.845+0000 7f2400e00700 1 -- 172.21.15.1:0/2741990911 shutdown_connections 2024-02-28T16:47:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.845+0000 7f2400e00700 1 -- 172.21.15.1:0/2741990911 wait complete. 2024-02-28T16:47:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.846+0000 7f2400e00700 1 Processor -- start 2024-02-28T16:47:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.846+0000 7f2400e00700 1 -- start start 2024-02-28T16:47:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.846+0000 7f2400e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23fc0fdf70 0x7f23fc115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.846+0000 7f2400e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fe910 0x7f23fc116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.846+0000 7f2400e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc116850 con 0x7f23fc0fe910 2024-02-28T16:47:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.846+0000 7f2400e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc10fb60 con 0x7f23fc0fdf70 2024-02-28T16:47:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.847+0000 7f23f9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fe910 0x7f23fc116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.847+0000 7f23f9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fe910 0x7f23fc116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58332/0 (socket says 172.21.15.1:58332) 2024-02-28T16:47:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.847+0000 7f23f9400700 1 -- 172.21.15.1:0/3066451703 learned_addr learned my addr 172.21.15.1:0/3066451703 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:47:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.847+0000 7f23f9e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23fc0fdf70 0x7f23fc115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.847+0000 7f23f9400700 1 -- 172.21.15.1:0/3066451703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23fc0fdf70 msgr2=0x7f23fc115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.847+0000 7f23f9400700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23fc0fdf70 0x7f23fc115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.847+0000 7f23f9400700 1 -- 172.21.15.1:0/3066451703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ec005040 con 0x7f23fc0fe910 2024-02-28T16:47:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.847+0000 7f23f9e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23fc0fdf70 0x7f23fc115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:47:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.848+0000 7f23f9400700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fe910 0x7f23fc116200 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f23e8002a50 tx=0x7f23e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.848+0000 7f23f2000700 1 -- 172.21.15.1:0/3066451703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e8007cc0 con 0x7f23fc0fe910 2024-02-28T16:47:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.848+0000 7f23f2000700 1 -- 172.21.15.1:0/3066451703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23e800f040 con 0x7f23fc0fe910 2024-02-28T16:47:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.848+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc10fe40 con 0x7f23fc0fe910 2024-02-28T16:47:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.848+0000 7f23f2000700 1 -- 172.21.15.1:0/3066451703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e800db70 con 0x7f23fc0fe910 2024-02-28T16:47:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.848+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc110380 con 0x7f23fc0fe910 2024-02-28T16:47:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.849+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23fc106b00 con 0x7f23fc0fe910 2024-02-28T16:47:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.850+0000 7f23f2000700 1 -- 172.21.15.1:0/3066451703 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f23e800dd10 con 0x7f23fc0fe910 2024-02-28T16:47:30.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.851+0000 7f23f2000700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23e006d7f0 0x7f23e006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:30.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.851+0000 7f23f2000700 1 -- 172.21.15.1:0/3066451703 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f23e808c590 con 0x7f23fc0fe910 2024-02-28T16:47:30.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.852+0000 7f23f9e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23e006d7f0 0x7f23e006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:30.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.852+0000 7f23f9e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23e006d7f0 0x7f23e006fca0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00b3b0 tx=0x7f23ec009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:30.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:30.856+0000 7f23f2000700 1 -- 172.21.15.1:0/3066451703 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f23e80598f0 con 0x7f23fc0fe910 2024-02-28T16:47:31.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.098+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23fc1112a0 con 0x7f23e006d7f0 2024-02-28T16:47:31.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.101+0000 7f23f2000700 1 -- 172.21.15.1:0/3066451703 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f23fc1112a0 con 0x7f23e006d7f0 2024-02-28T16:47:31.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23e006d7f0 msgr2=0x7f23e006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23e006d7f0 0x7f23e006fca0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00b3b0 tx=0x7f23ec009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:31.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fe910 msgr2=0x7f23fc116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fe910 0x7f23fc116200 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f23e8002a50 tx=0x7f23e80046b0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 shutdown_connections 2024-02-28T16:47:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23e006d7f0 0x7f23e006fca0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23fc0fdf70 0x7f23fc115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 --2- 172.21.15.1:0/3066451703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23fc0fe910 0x7f23fc116200 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 >> 172.21.15.1:0/3066451703 conn(0x7f23fc0f79d0 msgr2=0x7f23fc0fc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 shutdown_connections 2024-02-28T16:47:31.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.104+0000 7f2400e00700 1 -- 172.21.15.1:0/3066451703 wait complete. 2024-02-28T16:47:31.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.261+0000 7f6850000700 1 -- 172.21.15.1:0/220625103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fe8f0 msgr2=0x7f6848107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.261+0000 7f6850000700 1 --2- 172.21.15.1:0/220625103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fe8f0 0x7f6848107050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6834004970 tx=0x7f6834009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:31.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.261+0000 7f6850000700 1 -- 172.21.15.1:0/220625103 shutdown_connections 2024-02-28T16:47:31.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.261+0000 7f6850000700 1 --2- 172.21.15.1:0/220625103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fe8f0 0x7f6848107050 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.261+0000 7f6850000700 1 --2- 172.21.15.1:0/220625103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68480fdfe0 0x7f68480fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.261+0000 7f6850000700 1 -- 172.21.15.1:0/220625103 >> 172.21.15.1:0/220625103 conn(0x7f6848071d90 msgr2=0x7f6848072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:31.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.261+0000 7f6850000700 1 -- 172.21.15.1:0/220625103 shutdown_connections 2024-02-28T16:47:31.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.261+0000 7f6850000700 1 -- 172.21.15.1:0/220625103 wait complete. 2024-02-28T16:47:31.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.262+0000 7f6850000700 1 Processor -- start 2024-02-28T16:47:31.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.263+0000 7f6850000700 1 -- start start 2024-02-28T16:47:31.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.263+0000 7f6850000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fdfe0 0x7f6848115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.263+0000 7f6850000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68480fe8f0 0x7f6848116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.263+0000 7f6850000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6848116790 con 0x7f68480fdfe0 2024-02-28T16:47:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.263+0000 7f6850000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684810fac0 con 0x7f68480fe8f0 2024-02-28T16:47:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.263+0000 7f684d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fdfe0 0x7f6848115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.263+0000 7f684d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fdfe0 0x7f6848115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58358/0 (socket says 172.21.15.1:58358) 2024-02-28T16:47:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.263+0000 7f684d600700 1 -- 172.21.15.1:0/2702824611 learned_addr learned my addr 172.21.15.1:0/2702824611 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:47:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.264+0000 7f684cc00700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68480fe8f0 0x7f6848116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.264+0000 7f684d600700 1 -- 172.21.15.1:0/2702824611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68480fe8f0 msgr2=0x7f6848116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.264+0000 7f684d600700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68480fe8f0 0x7f6848116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.264+0000 7f684d600700 1 -- 172.21.15.1:0/2702824611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6834005040 con 0x7f68480fdfe0 2024-02-28T16:47:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.264+0000 7f684d600700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fdfe0 0x7f6848115c00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f683c001d80 tx=0x7f683c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.264+0000 7f6845600700 1 -- 172.21.15.1:0/2702824611 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f683c007d10 con 0x7f68480fdfe0 2024-02-28T16:47:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.264+0000 7f6845600700 1 -- 172.21.15.1:0/2702824611 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f683c007e70 con 0x7f68480fdfe0 2024-02-28T16:47:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.265+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684810fda0 con 0x7f68480fdfe0 2024-02-28T16:47:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.265+0000 7f6845600700 1 -- 172.21.15.1:0/2702824611 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f683c014640 con 0x7f68480fdfe0 2024-02-28T16:47:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.265+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68481102b0 con 0x7f68480fdfe0 2024-02-28T16:47:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.266+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6848106b20 con 0x7f68480fdfe0 2024-02-28T16:47:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.267+0000 7f6845600700 1 -- 172.21.15.1:0/2702824611 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f683c00e960 con 0x7f68480fdfe0 2024-02-28T16:47:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.268+0000 7f6845600700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f683006d7e0 0x7f683006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.268+0000 7f6845600700 1 -- 172.21.15.1:0/2702824611 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f683c08ccd0 con 0x7f68480fdfe0 2024-02-28T16:47:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.268+0000 7f684cc00700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f683006d7e0 0x7f683006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.269+0000 7f684cc00700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f683006d7e0 0x7f683006fc90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6848110df0 tx=0x7f6834009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.273+0000 7f6845600700 1 -- 172.21.15.1:0/2702824611 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f683c05a100 con 0x7f68480fdfe0 2024-02-28T16:47:31.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.505+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f684804bed0 con 0x7f683006d7e0 2024-02-28T16:47:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.524+0000 7f6845600700 1 -- 172.21.15.1:0/2702824611 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f684804bed0 con 0x7f683006d7e0 2024-02-28T16:47:31.512 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:47:31.512 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (13m) 5m ago 27m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:47:31.512 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (10m) 5m ago 28m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:47:31.512 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (10m) 6m ago 26m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:47:31.512 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (13m) 5m ago 27m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:47:31.512 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (6m) 5m ago 20m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (5m) 5m ago 20m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (20m) 6m ago 20m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (20m) 6m ago 20m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (16m) 5m ago 29m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (15m) 6m ago 26m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (12m) 5m ago 29m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (11m) 6m ago 26m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (14m) 5m ago 28m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (14m) 6m ago 26m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (9m) 5m ago 25m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (9m) 5m ago 25m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (8m) 5m ago 24m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (8m) 6m ago 23m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:47:31.513 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (7m) 6m ago 23m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:47:31.514 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (6m) 6m ago 22m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:47:31.514 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (14m) 5m ago 27m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:47:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.526+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f683006d7e0 msgr2=0x7f683006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.526+0000 7f6850000700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f683006d7e0 0x7f683006fc90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6848110df0 tx=0x7f6834009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.526+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fdfe0 msgr2=0x7f6848115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.527+0000 7f6850000700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fdfe0 0x7f6848115c00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f683c001d80 tx=0x7f683c009ec0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.527+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 shutdown_connections 2024-02-28T16:47:31.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.527+0000 7f6850000700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f683006d7e0 0x7f683006fc90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.527+0000 7f6850000700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68480fdfe0 0x7f6848115c00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.527+0000 7f6850000700 1 --2- 172.21.15.1:0/2702824611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68480fe8f0 0x7f6848116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.527+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 >> 172.21.15.1:0/2702824611 conn(0x7f6848071d90 msgr2=0x7f68480fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:31.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.527+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 shutdown_connections 2024-02-28T16:47:31.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.527+0000 7f6850000700 1 -- 172.21.15.1:0/2702824611 wait complete. 2024-02-28T16:47:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.678+0000 7fa3c8600700 1 -- 172.21.15.1:0/544472207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 msgr2=0x7fa3c0107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.678+0000 7fa3c8600700 1 --2- 172.21.15.1:0/544472207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 0x7fa3c0107050 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac004970 tx=0x7fa3ac009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.679+0000 7fa3c8600700 1 -- 172.21.15.1:0/544472207 shutdown_connections 2024-02-28T16:47:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.679+0000 7fa3c8600700 1 --2- 172.21.15.1:0/544472207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 0x7fa3c0107050 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.679+0000 7fa3c8600700 1 --2- 172.21.15.1:0/544472207 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3c00fdfe0 0x7fa3c00fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.679+0000 7fa3c8600700 1 -- 172.21.15.1:0/544472207 >> 172.21.15.1:0/544472207 conn(0x7fa3c0071d90 msgr2=0x7fa3c0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.679+0000 7fa3c8600700 1 -- 172.21.15.1:0/544472207 shutdown_connections 2024-02-28T16:47:31.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.679+0000 7fa3c8600700 1 -- 172.21.15.1:0/544472207 wait complete. 2024-02-28T16:47:31.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.680+0000 7fa3c8600700 1 Processor -- start 2024-02-28T16:47:31.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.680+0000 7fa3c8600700 1 -- start start 2024-02-28T16:47:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.680+0000 7fa3c8600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3c00fdfe0 0x7fa3c0115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.680+0000 7fa3c8600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 0x7fa3c01161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.680+0000 7fa3c8600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c0116890 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.680+0000 7fa3c8600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c010fb30 con 0x7fa3c00fdfe0 2024-02-28T16:47:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.680+0000 7fa3c5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 0x7fa3c01161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.681+0000 7fa3c5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 0x7fa3c01161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58376/0 (socket says 172.21.15.1:58376) 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.681+0000 7fa3c5200700 1 -- 172.21.15.1:0/1231810882 learned_addr learned my addr 172.21.15.1:0/1231810882 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.681+0000 7fa3c5c00700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3c00fdfe0 0x7fa3c0115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.681+0000 7fa3c5200700 1 -- 172.21.15.1:0/1231810882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3c00fdfe0 msgr2=0x7fa3c0115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.681+0000 7fa3c5200700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3c00fdfe0 0x7fa3c0115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.681+0000 7fa3c5200700 1 -- 172.21.15.1:0/1231810882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac005040 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.681+0000 7fa3c5c00700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3c00fdfe0 0x7fa3c0115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.681+0000 7fa3c5200700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 0x7fa3c01161b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac004f50 tx=0x7fa3ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.682+0000 7fa3bd600700 1 -- 172.21.15.1:0/1231810882 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3ac00e040 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.682+0000 7fa3bd600700 1 -- 172.21.15.1:0/1231810882 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3ac004290 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.682+0000 7fa3bd600700 1 -- 172.21.15.1:0/1231810882 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3ac01c630 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.682+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c010fdb0 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.682+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c01102f0 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.683+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c0106ac0 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.684+0000 7fa3bd600700 1 -- 172.21.15.1:0/1231810882 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa3ac00bc10 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.685+0000 7fa3bd600700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa3a806d7f0 0x7fa3a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.685+0000 7fa3bd600700 1 -- 172.21.15.1:0/1231810882 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa3ac08e220 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.685+0000 7fa3c5c00700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa3a806d7f0 0x7fa3a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.686+0000 7fa3c5c00700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa3a806d7f0 0x7fa3a806fca0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009b10 tx=0x7fa3b4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.689+0000 7fa3bd600700 1 -- 172.21.15.1:0/1231810882 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa3ac05b600 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.994+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa3c00738b0 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.995+0000 7fa3bd600700 1 -- 172.21.15.1:0/1231810882 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fa3ac013e60 con 0x7fa3c00fe8f0 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:47:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:47:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:47:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:47:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:47:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:47:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:47:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:47:31.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa3a806d7f0 msgr2=0x7fa3a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa3a806d7f0 0x7fa3a806fca0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009b10 tx=0x7fa3b4008040 comp rx=0 tx=0).stop 2024-02-28T16:47:31.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 msgr2=0x7fa3c01161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:31.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 0x7fa3c01161b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac004f50 tx=0x7fa3ac009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:31.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 shutdown_connections 2024-02-28T16:47:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa3a806d7f0 0x7fa3a806fca0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa3c00fdfe0 0x7fa3c0115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 --2- 172.21.15.1:0/1231810882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c00fe8f0 0x7fa3c01161b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.997+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 >> 172.21.15.1:0/1231810882 conn(0x7fa3c0071d90 msgr2=0x7fa3c0103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.998+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 shutdown_connections 2024-02-28T16:47:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:31.998+0000 7fa3c8600700 1 -- 172.21.15.1:0/1231810882 wait complete. 2024-02-28T16:47:32.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.154+0000 7fead2c00700 1 -- 172.21.15.1:0/3929876199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 msgr2=0x7feacc109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.154+0000 7fead2c00700 1 --2- 172.21.15.1:0/3929876199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 0x7feacc109250 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7feab8004970 tx=0x7feab8009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.154+0000 7fead2c00700 1 -- 172.21.15.1:0/3929876199 shutdown_connections 2024-02-28T16:47:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.154+0000 7fead2c00700 1 --2- 172.21.15.1:0/3929876199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 0x7feacc109250 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.154+0000 7fead2c00700 1 --2- 172.21.15.1:0/3929876199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feacc070330 0x7feacc070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.154+0000 7fead2c00700 1 -- 172.21.15.1:0/3929876199 >> 172.21.15.1:0/3929876199 conn(0x7feacc0f8af0 msgr2=0x7feacc0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.154+0000 7fead2c00700 1 -- 172.21.15.1:0/3929876199 shutdown_connections 2024-02-28T16:47:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.154+0000 7fead2c00700 1 -- 172.21.15.1:0/3929876199 wait complete. 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.155+0000 7fead2c00700 1 Processor -- start 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.155+0000 7fead2c00700 1 -- start start 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.155+0000 7fead2c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feacc070330 0x7feacc115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.155+0000 7fead2c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 0x7feacc116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.155+0000 7fead2c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc116780 con 0x7feacc070c40 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.155+0000 7fead2c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc10fad0 con 0x7feacc070330 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.156+0000 7feacb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 0x7feacc116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.156+0000 7feacb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 0x7feacc116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45056/0 (socket says 172.21.15.1:45056) 2024-02-28T16:47:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.156+0000 7feacb400700 1 -- 172.21.15.1:0/3193619439 learned_addr learned my addr 172.21.15.1:0/3193619439 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:47:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.156+0000 7feacbe00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feacc070330 0x7feacc115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.156+0000 7feacb400700 1 -- 172.21.15.1:0/3193619439 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feacc070330 msgr2=0x7feacc115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.156+0000 7feacb400700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feacc070330 0x7feacc115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.156+0000 7feacb400700 1 -- 172.21.15.1:0/3193619439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab8005040 con 0x7feacc070c40 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.156+0000 7feacbe00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feacc070330 0x7feacc115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.157+0000 7feacb400700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 0x7feacc116130 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7feab8000c00 tx=0x7feab8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.157+0000 7feac3e00700 1 -- 172.21.15.1:0/3193619439 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feab800e040 con 0x7feacc070c40 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.157+0000 7feac3e00700 1 -- 172.21.15.1:0/3193619439 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feab8004290 con 0x7feacc070c40 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.157+0000 7feac3e00700 1 -- 172.21.15.1:0/3193619439 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feab801c3f0 con 0x7feacc070c40 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.157+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc10fd50 con 0x7feacc070c40 2024-02-28T16:47:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.157+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc110290 con 0x7feacc070c40 2024-02-28T16:47:32.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.158+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc106910 con 0x7feacc070c40 2024-02-28T16:47:32.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.159+0000 7feac3e00700 1 -- 172.21.15.1:0/3193619439 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7feab8013420 con 0x7feacc070c40 2024-02-28T16:47:32.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.160+0000 7feac3e00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feab006d7f0 0x7feab006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.160+0000 7feac3e00700 1 -- 172.21.15.1:0/3193619439 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7feab808e1e0 con 0x7feacc070c40 2024-02-28T16:47:32.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.160+0000 7feacbe00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feab006d7f0 0x7feab006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:32.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.161+0000 7feacbe00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feab006d7f0 0x7feab006fca0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feabc009fd0 tx=0x7feabc00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:32.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.165+0000 7feac3e00700 1 -- 172.21.15.1:0/3193619439 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feab8057a90 con 0x7feacc070c40 2024-02-28T16:47:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:32 smithi001 ceph-mon[112940]: from='client.34754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:32 smithi001 ceph-mon[112940]: pgmap v392: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:32 smithi001 ceph-mon[112940]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:32 smithi001 ceph-mon[112940]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:32.269 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:32 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1231810882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:47:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.432+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feacc10fee0 con 0x7feacc070c40 2024-02-28T16:47:32.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.433+0000 7feac3e00700 1 -- 172.21.15.1:0/3193619439 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7feacc10fee0 con 0x7feacc070c40 2024-02-28T16:47:32.421 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:47:32.421 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:47:32.422 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:47:32.422 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:47:32.422 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:47:32.422 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:47:32.422 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:47:32.422 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:47:32.423 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:47:32.423 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:47:32.423 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:47:32.423 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:47:32.423 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:47:32.423 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:47:32.424 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:47:32.424 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:47:32.424 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:47:32.424 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:47:32.424 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:47:32.424 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:47:32.425 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:47:32.425 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:47:32.425 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:47:32.425 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:47:32.425 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:47:32.425 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:47:32.426 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:47:32.426 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:47:32.426 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:47:32.426 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:47:32.426 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:47:32.426 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:47:32.427 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:47:32.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:47:32.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:47:32.427 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:47:32.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:47:32.427 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:47:32.428 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:47:32.428 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:47:32.428 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:47:32.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feab006d7f0 msgr2=0x7feab006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feab006d7f0 0x7feab006fca0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feabc009fd0 tx=0x7feabc00b040 comp rx=0 tx=0).stop 2024-02-28T16:47:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 msgr2=0x7feacc116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 0x7feacc116130 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7feab8000c00 tx=0x7feab8009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 shutdown_connections 2024-02-28T16:47:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feab006d7f0 0x7feab006fca0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feacc070330 0x7feacc115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 --2- 172.21.15.1:0/3193619439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feacc070c40 0x7feacc116130 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.435+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 >> 172.21.15.1:0/3193619439 conn(0x7feacc0f8af0 msgr2=0x7feacc103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.436+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 shutdown_connections 2024-02-28T16:47:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.436+0000 7fead2c00700 1 -- 172.21.15.1:0/3193619439 wait complete. 2024-02-28T16:47:32.431 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:47:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:32 smithi066 ceph-mon[85781]: from='client.34754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:32 smithi066 ceph-mon[85781]: pgmap v392: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:32 smithi066 ceph-mon[85781]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:32 smithi066 ceph-mon[85781]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:32.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:32 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1231810882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:47:32.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.584+0000 7f4c73200700 1 -- 172.21.15.1:0/1261342699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 msgr2=0x7f4c74103380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.584+0000 7f4c73200700 1 --2- 172.21.15.1:0/1261342699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 0x7f4c74103380 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4c60004940 tx=0x7f4c60009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:32.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.584+0000 7f4c73200700 1 -- 172.21.15.1:0/1261342699 shutdown_connections 2024-02-28T16:47:32.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.584+0000 7f4c73200700 1 --2- 172.21.15.1:0/1261342699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 0x7f4c74103380 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.584+0000 7f4c73200700 1 --2- 172.21.15.1:0/1261342699 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c74109370 0x7f4c74109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.584+0000 7f4c73200700 1 -- 172.21.15.1:0/1261342699 >> 172.21.15.1:0/1261342699 conn(0x7f4c74072b80 msgr2=0x7f4c74074f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:32.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.584+0000 7f4c73200700 1 -- 172.21.15.1:0/1261342699 shutdown_connections 2024-02-28T16:47:32.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.585+0000 7f4c73200700 1 -- 172.21.15.1:0/1261342699 wait complete. 2024-02-28T16:47:32.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.585+0000 7f4c73200700 1 Processor -- start 2024-02-28T16:47:32.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.585+0000 7f4c73200700 1 -- start start 2024-02-28T16:47:32.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c73200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c74109370 0x7f4c74119e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c73200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 0x7f4c7411a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c73200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7411a990 con 0x7f4c74109d10 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c73200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74113ce0 con 0x7f4c74109370 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c71400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 0x7f4c7411a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c71400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 0x7f4c7411a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45070/0 (socket says 172.21.15.1:45070) 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c71400700 1 -- 172.21.15.1:0/1976102856 learned_addr learned my addr 172.21.15.1:0/1976102856 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c71400700 1 -- 172.21.15.1:0/1976102856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c74109370 msgr2=0x7f4c74119e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.586+0000 7f4c71e00700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c74109370 0x7f4c74119e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c71400700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c74109370 0x7f4c74119e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c71400700 1 -- 172.21.15.1:0/1976102856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60005040 con 0x7f4c74109d10 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c71e00700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c74109370 0x7f4c74119e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:47:32.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c71400700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 0x7f4c7411a340 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009150 tx=0x7f4c60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:32.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c6a000700 1 -- 172.21.15.1:0/1976102856 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c6000e040 con 0x7f4c74109d10 2024-02-28T16:47:32.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c6a000700 1 -- 172.21.15.1:0/1976102856 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c6000bbf0 con 0x7f4c74109d10 2024-02-28T16:47:32.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c6a000700 1 -- 172.21.15.1:0/1976102856 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c6001c630 con 0x7f4c74109d10 2024-02-28T16:47:32.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c74113f60 con 0x7f4c74109d10 2024-02-28T16:47:32.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.587+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c74114520 con 0x7f4c74109d10 2024-02-28T16:47:32.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.588+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7405f5c0 con 0x7f4c74109d10 2024-02-28T16:47:32.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.589+0000 7f4c6a000700 1 -- 172.21.15.1:0/1976102856 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4c6000bd60 con 0x7f4c74109d10 2024-02-28T16:47:32.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.590+0000 7f4c6a000700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5806d7e0 0x7f4c5806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.590+0000 7f4c6a000700 1 -- 172.21.15.1:0/1976102856 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4c6008e280 con 0x7f4c74109d10 2024-02-28T16:47:32.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.591+0000 7f4c71e00700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5806d7e0 0x7f4c5806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:32.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.591+0000 7f4c71e00700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5806d7e0 0x7f4c5806fc90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4c64007fd0 tx=0x7f4c64009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:32.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.595+0000 7f4c6a000700 1 -- 172.21.15.1:0/1976102856 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c60057b30 con 0x7f4c74109d10 2024-02-28T16:47:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.841+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c74115420 con 0x7f4c5806d7e0 2024-02-28T16:47:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.843+0000 7f4c6a000700 1 -- 172.21.15.1:0/1976102856 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4c74115420 con 0x7f4c5806d7e0 2024-02-28T16:47:32.831 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:47:32.831 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:47:32.831 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:47:32.831 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:47:32.831 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:47:32.831 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:47:32.831 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:47:32.832 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:47:32.832 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:47:32.832 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:47:32.832 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:47:32.832 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:47:32.832 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:47:32.832 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:47:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5806d7e0 msgr2=0x7f4c5806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5806d7e0 0x7f4c5806fc90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4c64007fd0 tx=0x7f4c64009040 comp rx=0 tx=0).stop 2024-02-28T16:47:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 msgr2=0x7f4c7411a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 0x7f4c7411a340 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009150 tx=0x7f4c60009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 shutdown_connections 2024-02-28T16:47:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5806d7e0 0x7f4c5806fc90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c74109370 0x7f4c74119e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 --2- 172.21.15.1:0/1976102856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c74109d10 0x7f4c7411a340 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 >> 172.21.15.1:0/1976102856 conn(0x7f4c74072b80 msgr2=0x7f4c74075670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 shutdown_connections 2024-02-28T16:47:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:32.846+0000 7f4c73200700 1 -- 172.21.15.1:0/1976102856 wait complete. 2024-02-28T16:47:32.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.004+0000 7f61ee000700 1 -- 172.21.15.1:0/176518669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e8102f50 msgr2=0x7f61e8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.004+0000 7f61ee000700 1 --2- 172.21.15.1:0/176518669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e8102f50 0x7f61e8103320 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f61d8004940 tx=0x7f61d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:32.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.004+0000 7f61ee000700 1 -- 172.21.15.1:0/176518669 shutdown_connections 2024-02-28T16:47:32.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.004+0000 7f61ee000700 1 --2- 172.21.15.1:0/176518669 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61e81038f0 0x7f61e80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.004+0000 7f61ee000700 1 --2- 172.21.15.1:0/176518669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e8102f50 0x7f61e8103320 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.004+0000 7f61ee000700 1 -- 172.21.15.1:0/176518669 >> 172.21.15.1:0/176518669 conn(0x7f61e80f8ae0 msgr2=0x7f61e80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:32.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.004+0000 7f61ee000700 1 -- 172.21.15.1:0/176518669 shutdown_connections 2024-02-28T16:47:32.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.004+0000 7f61ee000700 1 -- 172.21.15.1:0/176518669 wait complete. 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.005+0000 7f61ee000700 1 Processor -- start 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.005+0000 7f61ee000700 1 -- start start 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.006+0000 7f61ee000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61e8102f50 0x7f61e8113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.006+0000 7f61ee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e81038f0 0x7f61e8113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.006+0000 7f61ee000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e81145e0 con 0x7f61e81038f0 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.006+0000 7f61ee000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e810d910 con 0x7f61e8102f50 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.006+0000 7f61dfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e81038f0 0x7f61e8113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.006+0000 7f61dfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e81038f0 0x7f61e8113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45086/0 (socket says 172.21.15.1:45086) 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.006+0000 7f61dfe00700 1 -- 172.21.15.1:0/1218376458 learned_addr learned my addr 172.21.15.1:0/1218376458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:47:32.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.006+0000 7f61e7400700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61e8102f50 0x7f61e8113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:32.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61dfe00700 1 -- 172.21.15.1:0/1218376458 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61e8102f50 msgr2=0x7f61e8113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:32.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61dfe00700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61e8102f50 0x7f61e8113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61dfe00700 1 -- 172.21.15.1:0/1218376458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d8005040 con 0x7f61e81038f0 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61e7400700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61e8102f50 0x7f61e8113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61dfe00700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e81038f0 0x7f61e8113f90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c890 tx=0x7f61d00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61df400700 1 -- 172.21.15.1:0/1218376458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d000a7d0 con 0x7f61e81038f0 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61df400700 1 -- 172.21.15.1:0/1218376458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61d000ae00 con 0x7f61e81038f0 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61df400700 1 -- 172.21.15.1:0/1218376458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d00113a0 con 0x7f61e81038f0 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.007+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e810dbf0 con 0x7f61e81038f0 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.008+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e810e0e0 con 0x7f61e81038f0 2024-02-28T16:47:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.009+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e80fc6e0 con 0x7f61e81038f0 2024-02-28T16:47:32.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.010+0000 7f61df400700 1 -- 172.21.15.1:0/1218376458 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f61d0012400 con 0x7f61e81038f0 2024-02-28T16:47:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.011+0000 7f61df400700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f61cc06d7e0 0x7f61cc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:47:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.011+0000 7f61df400700 1 -- 172.21.15.1:0/1218376458 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f61d008d030 con 0x7f61e81038f0 2024-02-28T16:47:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.012+0000 7f61e7400700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f61cc06d7e0 0x7f61cc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:47:33.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.012+0000 7f61e7400700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f61cc06d7e0 0x7f61cc06fc90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f61d8004f90 tx=0x7f61d8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:47:33.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.016+0000 7f61df400700 1 -- 172.21.15.1:0/1218376458 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f61d00c0d20 con 0x7f61e81038f0 2024-02-28T16:47:33.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.333+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f61e810f030 con 0x7f61e81038f0 2024-02-28T16:47:33.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.334+0000 7f61df400700 1 -- 172.21.15.1:0/1218376458 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f61d005a390 con 0x7f61e81038f0 2024-02-28T16:47:33.321 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:47:33.322 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:47:33.322 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:47:33.322 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:47:33.322 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:47:33.322 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:47:33.322 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:47:33.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f61cc06d7e0 msgr2=0x7f61cc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:33.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f61cc06d7e0 0x7f61cc06fc90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f61d8004f90 tx=0x7f61d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:47:33.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e81038f0 msgr2=0x7f61e8113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:47:33.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e81038f0 0x7f61e8113f90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c890 tx=0x7f61d00044c0 comp rx=0 tx=0).stop 2024-02-28T16:47:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 shutdown_connections 2024-02-28T16:47:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f61cc06d7e0 0x7f61cc06fc90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61e8102f50 0x7f61e8113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 --2- 172.21.15.1:0/1218376458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61e81038f0 0x7f61e8113f90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:47:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 >> 172.21.15.1:0/1218376458 conn(0x7f61e80f8ae0 msgr2=0x7f61e80fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:47:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 shutdown_connections 2024-02-28T16:47:33.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:47:33.337+0000 7f61ee000700 1 -- 172.21.15.1:0/1218376458 wait complete. 2024-02-28T16:47:33.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:33 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3193619439' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:47:33.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:33 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3193619439' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:47:34.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:34 smithi066 ceph-mon[85781]: from='client.34774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:34.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:34 smithi066 ceph-mon[85781]: pgmap v393: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:34.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:34 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1218376458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:47:34.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:34 smithi001 ceph-mon[112940]: from='client.34774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:47:34.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:34 smithi001 ceph-mon[112940]: pgmap v393: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:34.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:34 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1218376458' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:47:36.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:36 smithi066 ceph-mon[85781]: pgmap v394: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:36.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:36 smithi001 ceph-mon[112940]: pgmap v394: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:38.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:38 smithi066 ceph-mon[85781]: pgmap v395: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:38.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:38 smithi001 ceph-mon[112940]: pgmap v395: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:40.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:40 smithi066 ceph-mon[85781]: pgmap v396: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:40.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:40 smithi001 ceph-mon[112940]: pgmap v396: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:41.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:47:41.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:47:42.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:42 smithi066 ceph-mon[85781]: pgmap v397: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:42.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:42 smithi001 ceph-mon[112940]: pgmap v397: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:44.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:44 smithi066 ceph-mon[85781]: pgmap v398: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:44.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:44 smithi001 ceph-mon[112940]: pgmap v398: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:46.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:46 smithi066 ceph-mon[85781]: pgmap v399: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:46.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:46 smithi001 ceph-mon[112940]: pgmap v399: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:48.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:48 smithi066 ceph-mon[85781]: pgmap v400: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:48.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:48 smithi001 ceph-mon[112940]: pgmap v400: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:50.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:50 smithi066 ceph-mon[85781]: pgmap v401: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:50.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:50 smithi001 ceph-mon[112940]: pgmap v401: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:52.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:52 smithi066 ceph-mon[85781]: pgmap v402: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:52.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:52 smithi001 ceph-mon[112940]: pgmap v402: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:54.497 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:54 smithi066 ceph-mon[85781]: pgmap v403: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:54.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:54 smithi001 ceph-mon[112940]: pgmap v403: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:56.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:56 smithi066 ceph-mon[85781]: pgmap v404: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:56.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:47:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:56 smithi001 ceph-mon[112940]: pgmap v404: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:56.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:47:58.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:47:58 smithi066 ceph-mon[85781]: pgmap v405: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:47:58.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:47:58 smithi001 ceph-mon[112940]: pgmap v405: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:00.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:00 smithi066 ceph-mon[85781]: pgmap v406: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:00.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:00 smithi001 ceph-mon[112940]: pgmap v406: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:02.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:02 smithi066 ceph-mon[85781]: pgmap v407: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:02.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:02 smithi001 ceph-mon[112940]: pgmap v407: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:03.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.510+0000 7fa87b600700 1 -- 172.21.15.1:0/3555580932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 msgr2=0x7fa874106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.510+0000 7fa87b600700 1 --2- 172.21.15.1:0/3555580932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 0x7fa874106fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa860004940 tx=0x7fa860009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.510+0000 7fa87b600700 1 -- 172.21.15.1:0/3555580932 shutdown_connections 2024-02-28T16:48:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.510+0000 7fa87b600700 1 --2- 172.21.15.1:0/3555580932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 0x7fa874106fa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.510+0000 7fa87b600700 1 --2- 172.21.15.1:0/3555580932 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8740fdfc0 0x7fa8740fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.510+0000 7fa87b600700 1 -- 172.21.15.1:0/3555580932 >> 172.21.15.1:0/3555580932 conn(0x7fa8740f7a10 msgr2=0x7fa8740f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.510+0000 7fa87b600700 1 -- 172.21.15.1:0/3555580932 shutdown_connections 2024-02-28T16:48:03.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.510+0000 7fa87b600700 1 -- 172.21.15.1:0/3555580932 wait complete. 2024-02-28T16:48:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.511+0000 7fa87b600700 1 Processor -- start 2024-02-28T16:48:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.511+0000 7fa87b600700 1 -- start start 2024-02-28T16:48:03.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.512+0000 7fa87b600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8740fdfc0 0x7fa874115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.512+0000 7fa87b600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 0x7fa874116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.512+0000 7fa87b600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8741167b0 con 0x7fa8740fe960 2024-02-28T16:48:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.512+0000 7fa87b600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87410fb00 con 0x7fa8740fdfc0 2024-02-28T16:48:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.512+0000 7fa873e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 0x7fa874116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.512+0000 7fa873e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 0x7fa874116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56028/0 (socket says 172.21.15.1:56028) 2024-02-28T16:48:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.512+0000 7fa873e00700 1 -- 172.21.15.1:0/1341514517 learned_addr learned my addr 172.21.15.1:0/1341514517 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.512+0000 7fa878c00700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8740fdfc0 0x7fa874115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:03.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.513+0000 7fa873e00700 1 -- 172.21.15.1:0/1341514517 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8740fdfc0 msgr2=0x7fa874115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:03.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.513+0000 7fa873e00700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8740fdfc0 0x7fa874115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.513+0000 7fa873e00700 1 -- 172.21.15.1:0/1341514517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860005040 con 0x7fa8740fe960 2024-02-28T16:48:03.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.513+0000 7fa873e00700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 0x7fa874116160 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa860009180 tx=0x7fa860009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.513+0000 7fa870c00700 1 -- 172.21.15.1:0/1341514517 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa86000e040 con 0x7fa8740fe960 2024-02-28T16:48:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.513+0000 7fa870c00700 1 -- 172.21.15.1:0/1341514517 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa860004290 con 0x7fa8740fe960 2024-02-28T16:48:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.513+0000 7fa870c00700 1 -- 172.21.15.1:0/1341514517 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa86001c630 con 0x7fa8740fe960 2024-02-28T16:48:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.513+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87410fde0 con 0x7fa8740fe960 2024-02-28T16:48:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.514+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8741102d0 con 0x7fa8740fe960 2024-02-28T16:48:03.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.515+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa874106a10 con 0x7fa8740fe960 2024-02-28T16:48:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.516+0000 7fa870c00700 1 -- 172.21.15.1:0/1341514517 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa8600139f0 con 0x7fa8740fe960 2024-02-28T16:48:03.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.517+0000 7fa870c00700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa85c06d7f0 0x7fa85c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:03.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.517+0000 7fa870c00700 1 -- 172.21.15.1:0/1341514517 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa86008e400 con 0x7fa8740fe960 2024-02-28T16:48:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.518+0000 7fa878c00700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa85c06d7f0 0x7fa85c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.518+0000 7fa878c00700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa85c06d7f0 0x7fa85c06fca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa868007fd0 tx=0x7fa868009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.522+0000 7fa870c00700 1 -- 172.21.15.1:0/1341514517 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa860057c30 con 0x7fa8740fe960 2024-02-28T16:48:03.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.762+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa87404bed0 con 0x7fa85c06d7f0 2024-02-28T16:48:03.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.765+0000 7fa870c00700 1 -- 172.21.15.1:0/1341514517 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa87404bed0 con 0x7fa85c06d7f0 2024-02-28T16:48:03.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa85c06d7f0 msgr2=0x7fa85c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa85c06d7f0 0x7fa85c06fca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa868007fd0 tx=0x7fa868009040 comp rx=0 tx=0).stop 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 msgr2=0x7fa874116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 0x7fa874116160 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa860009180 tx=0x7fa860009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 shutdown_connections 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa85c06d7f0 0x7fa85c06fca0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8740fdfc0 0x7fa874115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 --2- 172.21.15.1:0/1341514517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8740fe960 0x7fa874116160 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 >> 172.21.15.1:0/1341514517 conn(0x7fa8740f7a10 msgr2=0x7fa8740fc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 shutdown_connections 2024-02-28T16:48:03.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.768+0000 7fa87b600700 1 -- 172.21.15.1:0/1341514517 wait complete. 2024-02-28T16:48:03.773 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:48:03.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.925+0000 7f0d52000700 1 -- 172.21.15.1:0/2067903260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 msgr2=0x7f0d4c10d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:03.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.925+0000 7f0d52000700 1 --2- 172.21.15.1:0/2067903260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 0x7f0d4c10d700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0d38004940 tx=0x7f0d38009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:03.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.926+0000 7f0d52000700 1 -- 172.21.15.1:0/2067903260 shutdown_connections 2024-02-28T16:48:03.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.926+0000 7f0d52000700 1 --2- 172.21.15.1:0/2067903260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 0x7f0d4c10d700 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.926+0000 7f0d52000700 1 --2- 172.21.15.1:0/2067903260 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d4c0724c0 0x7f0d4c072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.926+0000 7f0d52000700 1 -- 172.21.15.1:0/2067903260 >> 172.21.15.1:0/2067903260 conn(0x7f0d4c0fab40 msgr2=0x7f0d4c0fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:03.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.926+0000 7f0d52000700 1 -- 172.21.15.1:0/2067903260 shutdown_connections 2024-02-28T16:48:03.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.926+0000 7f0d52000700 1 -- 172.21.15.1:0/2067903260 wait complete. 2024-02-28T16:48:03.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.927+0000 7f0d52000700 1 Processor -- start 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.927+0000 7f0d52000700 1 -- start start 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.927+0000 7f0d52000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d4c0724c0 0x7f0d4c119d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.927+0000 7f0d52000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 0x7f0d4c11a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.927+0000 7f0d52000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c11a8e0 con 0x7f0d4c072dd0 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.927+0000 7f0d52000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c113bf0 con 0x7f0d4c0724c0 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.928+0000 7f0d4be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 0x7f0d4c11a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.928+0000 7f0d4be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 0x7f0d4c11a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56048/0 (socket says 172.21.15.1:56048) 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.928+0000 7f0d4be00700 1 -- 172.21.15.1:0/441048233 learned_addr learned my addr 172.21.15.1:0/441048233 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.928+0000 7f0d50c00700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d4c0724c0 0x7f0d4c119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.928+0000 7f0d4be00700 1 -- 172.21.15.1:0/441048233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d4c0724c0 msgr2=0x7f0d4c119d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.928+0000 7f0d4be00700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d4c0724c0 0x7f0d4c119d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:03.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.928+0000 7f0d4be00700 1 -- 172.21.15.1:0/441048233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d38005040 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.928+0000 7f0d4be00700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 0x7f0d4c11a290 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009180 tx=0x7f0d38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.929+0000 7f0d48c00700 1 -- 172.21.15.1:0/441048233 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3800e040 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.929+0000 7f0d48c00700 1 -- 172.21.15.1:0/441048233 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d38004290 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.929+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c113e70 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.929+0000 7f0d48c00700 1 -- 172.21.15.1:0/441048233 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3801c630 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.929+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c114430 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.930+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c10ae30 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.931+0000 7f0d48c00700 1 -- 172.21.15.1:0/441048233 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0d380139f0 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.932+0000 7f0d48c00700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d3406d7f0 0x7f0d3406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.932+0000 7f0d48c00700 1 -- 172.21.15.1:0/441048233 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0d3808e2f0 con 0x7f0d4c072dd0 2024-02-28T16:48:03.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.932+0000 7f0d50c00700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d3406d7f0 0x7f0d3406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:03.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.933+0000 7f0d50c00700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d3406d7f0 0x7f0d3406fca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0d40004990 tx=0x7f0d40009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:03.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:03.937+0000 7f0d48c00700 1 -- 172.21.15.1:0/441048233 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0d3805b6d0 con 0x7f0d4c072dd0 2024-02-28T16:48:04.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.186+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d4c115330 con 0x7f0d3406d7f0 2024-02-28T16:48:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.188+0000 7f0d48c00700 1 -- 172.21.15.1:0/441048233 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0d4c115330 con 0x7f0d3406d7f0 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d3406d7f0 msgr2=0x7f0d3406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d3406d7f0 0x7f0d3406fca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0d40004990 tx=0x7f0d40009ee0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 msgr2=0x7f0d4c11a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 0x7f0d4c11a290 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009180 tx=0x7f0d38009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 shutdown_connections 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d3406d7f0 0x7f0d3406fca0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0d4c0724c0 0x7f0d4c119d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 --2- 172.21.15.1:0/441048233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d4c072dd0 0x7f0d4c11a290 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 >> 172.21.15.1:0/441048233 conn(0x7f0d4c0fab40 msgr2=0x7f0d4c0fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 shutdown_connections 2024-02-28T16:48:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.191+0000 7f0d52000700 1 -- 172.21.15.1:0/441048233 wait complete. 2024-02-28T16:48:04.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.354+0000 7fba78000700 1 -- 172.21.15.1:0/329225863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba701038f0 msgr2=0x7fba700fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:04.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.354+0000 7fba78000700 1 --2- 172.21.15.1:0/329225863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba701038f0 0x7fba700fcf60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fba5c004940 tx=0x7fba5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.354+0000 7fba78000700 1 -- 172.21.15.1:0/329225863 shutdown_connections 2024-02-28T16:48:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.354+0000 7fba78000700 1 --2- 172.21.15.1:0/329225863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba701038f0 0x7fba700fcf60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.354+0000 7fba78000700 1 --2- 172.21.15.1:0/329225863 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba70102f50 0x7fba70103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.354+0000 7fba78000700 1 -- 172.21.15.1:0/329225863 >> 172.21.15.1:0/329225863 conn(0x7fba700f8ae0 msgr2=0x7fba700faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.354+0000 7fba78000700 1 -- 172.21.15.1:0/329225863 shutdown_connections 2024-02-28T16:48:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.354+0000 7fba78000700 1 -- 172.21.15.1:0/329225863 wait complete. 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.355+0000 7fba78000700 1 Processor -- start 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.355+0000 7fba78000700 1 -- start start 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.355+0000 7fba78000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70102f50 0x7fba70115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.355+0000 7fba78000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba701038f0 0x7fba701161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.355+0000 7fba78000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba70116810 con 0x7fba70102f50 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.355+0000 7fba78000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7010fb40 con 0x7fba701038f0 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.356+0000 7fba75600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70102f50 0x7fba70115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.356+0000 7fba75600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70102f50 0x7fba70115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56062/0 (socket says 172.21.15.1:56062) 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.356+0000 7fba75600700 1 -- 172.21.15.1:0/1521660437 learned_addr learned my addr 172.21.15.1:0/1521660437 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.356+0000 7fba74c00700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba701038f0 0x7fba701161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.356+0000 7fba75600700 1 -- 172.21.15.1:0/1521660437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba701038f0 msgr2=0x7fba701161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.356+0000 7fba75600700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba701038f0 0x7fba701161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.356+0000 7fba75600700 1 -- 172.21.15.1:0/1521660437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c005040 con 0x7fba70102f50 2024-02-28T16:48:04.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.357+0000 7fba75600700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70102f50 0x7fba70115c80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fba64002a50 tx=0x7fba640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:04.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.357+0000 7fba6d600700 1 -- 172.21.15.1:0/1521660437 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba64007cc0 con 0x7fba70102f50 2024-02-28T16:48:04.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.357+0000 7fba6d600700 1 -- 172.21.15.1:0/1521660437 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba6400f040 con 0x7fba70102f50 2024-02-28T16:48:04.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.357+0000 7fba6d600700 1 -- 172.21.15.1:0/1521660437 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba64009640 con 0x7fba70102f50 2024-02-28T16:48:04.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.357+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7010fe20 con 0x7fba70102f50 2024-02-28T16:48:04.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.357+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba70110360 con 0x7fba70102f50 2024-02-28T16:48:04.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.358+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba700fc6e0 con 0x7fba70102f50 2024-02-28T16:48:04.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.359+0000 7fba6d600700 1 -- 172.21.15.1:0/1521660437 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fba640097a0 con 0x7fba70102f50 2024-02-28T16:48:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.360+0000 7fba6d600700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba5806d7f0 0x7fba5806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.360+0000 7fba6d600700 1 -- 172.21.15.1:0/1521660437 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fba6408cbc0 con 0x7fba70102f50 2024-02-28T16:48:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.361+0000 7fba74c00700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba5806d7f0 0x7fba5806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.361+0000 7fba74c00700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba5806d7f0 0x7fba5806fca0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fba701114b0 tx=0x7fba5c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.365+0000 7fba6d600700 1 -- 172.21.15.1:0/1521660437 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba64059fc0 con 0x7fba70102f50 2024-02-28T16:48:04.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:04 smithi066 ceph-mon[85781]: pgmap v408: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:04.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:04 smithi001 ceph-mon[112940]: pgmap v408: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:04.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.599+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fba70111290 con 0x7fba5806d7f0 2024-02-28T16:48:04.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.613+0000 7fba6d600700 1 -- 172.21.15.1:0/1521660437 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fba70111290 con 0x7fba5806d7f0 2024-02-28T16:48:04.603 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:48:04.603 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (14m) 6m ago 28m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:48:04.603 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (10m) 6m ago 28m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (10m) 7m ago 27m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (13m) 6m ago 28m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (6m) 6m ago 21m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (6m) 6m ago 21m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (21m) 7m ago 21m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (21m) 7m ago 21m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (16m) 6m ago 30m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (15m) 7m ago 27m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (12m) 6m ago 30m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (11m) 7m ago 27m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (15m) 6m ago 28m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (15m) 7m ago 27m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (10m) 6m ago 26m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:48:04.604 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (9m) 6m ago 25m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:48:04.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (9m) 6m ago 24m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:48:04.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (8m) 7m ago 24m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:48:04.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (7m) 7m ago 23m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:48:04.605 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (7m) 7m ago 23m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:48:04.605 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (14m) 6m ago 27m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba5806d7f0 msgr2=0x7fba5806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba5806d7f0 0x7fba5806fca0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fba701114b0 tx=0x7fba5c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70102f50 msgr2=0x7fba70115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70102f50 0x7fba70115c80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fba64002a50 tx=0x7fba640046b0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 shutdown_connections 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba5806d7f0 0x7fba5806fca0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70102f50 0x7fba70115c80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 --2- 172.21.15.1:0/1521660437 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba701038f0 0x7fba701161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 >> 172.21.15.1:0/1521660437 conn(0x7fba700f8ae0 msgr2=0x7fba700fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.616+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 shutdown_connections 2024-02-28T16:48:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.617+0000 7fba78000700 1 -- 172.21.15.1:0/1521660437 wait complete. 2024-02-28T16:48:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.772+0000 7f12abe00700 1 -- 172.21.15.1:0/2491234562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fc520 msgr2=0x7f12ac100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.772+0000 7f12abe00700 1 --2- 172.21.15.1:0/2491234562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fc520 0x7f12ac100bd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1298004940 tx=0x7f1298009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.773+0000 7f12abe00700 1 -- 172.21.15.1:0/2491234562 shutdown_connections 2024-02-28T16:48:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.773+0000 7f12abe00700 1 --2- 172.21.15.1:0/2491234562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fc520 0x7f12ac100bd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.773+0000 7f12abe00700 1 --2- 172.21.15.1:0/2491234562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12ac0fbb80 0x7f12ac0fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.773+0000 7f12abe00700 1 -- 172.21.15.1:0/2491234562 >> 172.21.15.1:0/2491234562 conn(0x7f12ac073ab0 msgr2=0x7f12ac073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.773+0000 7f12abe00700 1 -- 172.21.15.1:0/2491234562 shutdown_connections 2024-02-28T16:48:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.773+0000 7f12abe00700 1 -- 172.21.15.1:0/2491234562 wait complete. 2024-02-28T16:48:04.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.774+0000 7f12abe00700 1 Processor -- start 2024-02-28T16:48:04.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.774+0000 7f12abe00700 1 -- start start 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.774+0000 7f12abe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fbb80 0x7f12ac119f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12abe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12ac0fc520 0x7f12ac11a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12abe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac11aab0 con 0x7f12ac0fbb80 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12abe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac113dc0 con 0x7f12ac0fc520 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12aaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fbb80 0x7f12ac119f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12aaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fbb80 0x7f12ac119f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56086/0 (socket says 172.21.15.1:56086) 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12aaa00700 1 -- 172.21.15.1:0/1227832238 learned_addr learned my addr 172.21.15.1:0/1227832238 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12aa000700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12ac0fc520 0x7f12ac11a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12aaa00700 1 -- 172.21.15.1:0/1227832238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12ac0fc520 msgr2=0x7f12ac11a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12aaa00700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12ac0fc520 0x7f12ac11a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12aaa00700 1 -- 172.21.15.1:0/1227832238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298005040 con 0x7f12ac0fbb80 2024-02-28T16:48:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.775+0000 7f12aa000700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12ac0fc520 0x7f12ac11a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:48:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.776+0000 7f12aaa00700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fbb80 0x7f12ac119f20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f129c00cb10 tx=0x7f129c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.776+0000 7f12a2a00700 1 -- 172.21.15.1:0/1227832238 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f129c004200 con 0x7f12ac0fbb80 2024-02-28T16:48:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.776+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac1140a0 con 0x7f12ac0fbb80 2024-02-28T16:48:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.776+0000 7f12a2a00700 1 -- 172.21.15.1:0/1227832238 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f129c00acc0 con 0x7f12ac0fbb80 2024-02-28T16:48:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.776+0000 7f12a2a00700 1 -- 172.21.15.1:0/1227832238 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f129c003d90 con 0x7f12ac0fbb80 2024-02-28T16:48:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.776+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac1145e0 con 0x7f12ac0fbb80 2024-02-28T16:48:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.777+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac0fe2f0 con 0x7f12ac0fbb80 2024-02-28T16:48:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.778+0000 7f12a2a00700 1 -- 172.21.15.1:0/1227832238 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f129c003430 con 0x7f12ac0fbb80 2024-02-28T16:48:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.779+0000 7f12a2a00700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f129006d7f0 0x7f129006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.780+0000 7f12a2a00700 1 -- 172.21.15.1:0/1227832238 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f129c08d430 con 0x7f12ac0fbb80 2024-02-28T16:48:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.780+0000 7f12aa000700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f129006d7f0 0x7f129006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.780+0000 7f12aa000700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f129006d7f0 0x7f129006fca0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1298009fd0 tx=0x7f1298009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:04.784+0000 7f12a2a00700 1 -- 172.21.15.1:0/1227832238 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f129c05a790 con 0x7f12ac0fbb80 2024-02-28T16:48:05.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.099+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f12ac04bed0 con 0x7f12ac0fbb80 2024-02-28T16:48:05.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.099+0000 7f12a2a00700 1 -- 172.21.15.1:0/1227832238 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f129c05a320 con 0x7f12ac0fbb80 2024-02-28T16:48:05.089 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:48:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:48:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:48:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:48:05.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:48:05.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:48:05.091 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:48:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f129006d7f0 msgr2=0x7f129006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f129006d7f0 0x7f129006fca0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1298009fd0 tx=0x7f1298009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fbb80 msgr2=0x7f12ac119f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fbb80 0x7f12ac119f20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f129c00cb10 tx=0x7f129c007f90 comp rx=0 tx=0).stop 2024-02-28T16:48:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 shutdown_connections 2024-02-28T16:48:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f129006d7f0 0x7f129006fca0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12ac0fbb80 0x7f12ac119f20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 --2- 172.21.15.1:0/1227832238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12ac0fc520 0x7f12ac11a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.102+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 >> 172.21.15.1:0/1227832238 conn(0x7f12ac073ab0 msgr2=0x7f12ac10d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.103+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 shutdown_connections 2024-02-28T16:48:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.103+0000 7f12abe00700 1 -- 172.21.15.1:0/1227832238 wait complete. 2024-02-28T16:48:05.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.254+0000 7f2ca8000700 1 -- 172.21.15.1:0/306371754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 msgr2=0x7f2ca00fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.254+0000 7f2ca8000700 1 --2- 172.21.15.1:0/306371754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 0x7f2ca00fcf60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c004940 tx=0x7f2c8c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:05.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.255+0000 7f2ca8000700 1 -- 172.21.15.1:0/306371754 shutdown_connections 2024-02-28T16:48:05.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.255+0000 7f2ca8000700 1 --2- 172.21.15.1:0/306371754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 0x7f2ca00fcf60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.255+0000 7f2ca8000700 1 --2- 172.21.15.1:0/306371754 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ca0102f50 0x7f2ca0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.255+0000 7f2ca8000700 1 -- 172.21.15.1:0/306371754 >> 172.21.15.1:0/306371754 conn(0x7f2ca00f8ae0 msgr2=0x7f2ca00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:05.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.255+0000 7f2ca8000700 1 -- 172.21.15.1:0/306371754 shutdown_connections 2024-02-28T16:48:05.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.255+0000 7f2ca8000700 1 -- 172.21.15.1:0/306371754 wait complete. 2024-02-28T16:48:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca8000700 1 Processor -- start 2024-02-28T16:48:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca8000700 1 -- start start 2024-02-28T16:48:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca8000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ca0102f50 0x7f2ca0115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca8000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 0x7f2ca01161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca8000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca01168d0 con 0x7f2ca01038f0 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca8000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca010fb50 con 0x7f2ca0102f50 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 0x7f2ca01161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 0x7f2ca01161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56100/0 (socket says 172.21.15.1:56100) 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.256+0000 7f2ca4c00700 1 -- 172.21.15.1:0/770937032 learned_addr learned my addr 172.21.15.1:0/770937032 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.257+0000 7f2ca4c00700 1 -- 172.21.15.1:0/770937032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ca0102f50 msgr2=0x7f2ca0115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.257+0000 7f2ca5600700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ca0102f50 0x7f2ca0115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.257+0000 7f2ca4c00700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ca0102f50 0x7f2ca0115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.257+0000 7f2ca4c00700 1 -- 172.21.15.1:0/770937032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c005040 con 0x7f2ca01038f0 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.257+0000 7f2ca5600700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ca0102f50 0x7f2ca0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.257+0000 7f2ca4c00700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 0x7f2ca01161f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009030 tx=0x7f2c8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.257+0000 7f2c9d600700 1 -- 172.21.15.1:0/770937032 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c8c00e040 con 0x7f2ca01038f0 2024-02-28T16:48:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.258+0000 7f2c9d600700 1 -- 172.21.15.1:0/770937032 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c8c00bbf0 con 0x7f2ca01038f0 2024-02-28T16:48:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.258+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca010fd70 con 0x7f2ca01038f0 2024-02-28T16:48:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.258+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca01102b0 con 0x7f2ca01038f0 2024-02-28T16:48:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.258+0000 7f2c9d600700 1 -- 172.21.15.1:0/770937032 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c8c0255e0 con 0x7f2ca01038f0 2024-02-28T16:48:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.259+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca00fc6e0 con 0x7f2ca01038f0 2024-02-28T16:48:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.260+0000 7f2c9d600700 1 -- 172.21.15.1:0/770937032 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2c8c00bd60 con 0x7f2ca01038f0 2024-02-28T16:48:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.261+0000 7f2c9d600700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8806d7f0 0x7f2c8806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.261+0000 7f2c9d600700 1 -- 172.21.15.1:0/770937032 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2c8c08e260 con 0x7f2ca01038f0 2024-02-28T16:48:05.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.261+0000 7f2ca5600700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8806d7f0 0x7f2c8806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:05.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.261+0000 7f2ca5600700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8806d7f0 0x7f2c8806fca0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c94006fd0 tx=0x7f2c94008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.265+0000 7f2c9d600700 1 -- 172.21.15.1:0/770937032 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c8c05b5c0 con 0x7f2ca01038f0 2024-02-28T16:48:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:05 smithi066 ceph-mon[85781]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:05 smithi066 ceph-mon[85781]: from='client.34786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:05 smithi066 ceph-mon[85781]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:05.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:05 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1227832238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:48:05.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:05 smithi001 ceph-mon[112940]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:05 smithi001 ceph-mon[112940]: from='client.34786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:05 smithi001 ceph-mon[112940]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:05.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:05 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1227832238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:48:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.535+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2ca01112c0 con 0x7f2ca01038f0 2024-02-28T16:48:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.536+0000 7f2c9d600700 1 -- 172.21.15.1:0/770937032 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f2c8c05b150 con 0x7f2ca01038f0 2024-02-28T16:48:05.526 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:48:05.526 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:48:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:48:05.528 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:48:05.529 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:48:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.539+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8806d7f0 msgr2=0x7f2c8806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.539+0000 7f2ca8000700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8806d7f0 0x7f2c8806fca0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c94006fd0 tx=0x7f2c94008040 comp rx=0 tx=0).stop 2024-02-28T16:48:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.539+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 msgr2=0x7f2ca01161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.539+0000 7f2ca8000700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 0x7f2ca01161f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009030 tx=0x7f2c8c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.539+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 shutdown_connections 2024-02-28T16:48:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.539+0000 7f2ca8000700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8806d7f0 0x7f2c8806fca0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.539+0000 7f2ca8000700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ca0102f50 0x7f2ca0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.539+0000 7f2ca8000700 1 --2- 172.21.15.1:0/770937032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca01038f0 0x7f2ca01161f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.540+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 >> 172.21.15.1:0/770937032 conn(0x7f2ca00f8ae0 msgr2=0x7f2ca00fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.540+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 shutdown_connections 2024-02-28T16:48:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.540+0000 7f2ca8000700 1 -- 172.21.15.1:0/770937032 wait complete. 2024-02-28T16:48:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:48:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.686+0000 7fade6e00700 1 -- 172.21.15.1:0/1859658345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 msgr2=0x7fade01092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.686+0000 7fade6e00700 1 --2- 172.21.15.1:0/1859658345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 0x7fade01092e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fadcc004940 tx=0x7fadcc009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.687+0000 7fade6e00700 1 -- 172.21.15.1:0/1859658345 shutdown_connections 2024-02-28T16:48:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.687+0000 7fade6e00700 1 --2- 172.21.15.1:0/1859658345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 0x7fade01092e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.687+0000 7fade6e00700 1 --2- 172.21.15.1:0/1859658345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fade00702b0 0x7fade0070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.687+0000 7fade6e00700 1 -- 172.21.15.1:0/1859658345 >> 172.21.15.1:0/1859658345 conn(0x7fade00f8ae0 msgr2=0x7fade00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.687+0000 7fade6e00700 1 -- 172.21.15.1:0/1859658345 shutdown_connections 2024-02-28T16:48:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.687+0000 7fade6e00700 1 -- 172.21.15.1:0/1859658345 wait complete. 2024-02-28T16:48:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.688+0000 7fade6e00700 1 Processor -- start 2024-02-28T16:48:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.688+0000 7fade6e00700 1 -- start start 2024-02-28T16:48:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.688+0000 7fade6e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fade00702b0 0x7fade0115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.688+0000 7fade6e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 0x7fade01161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.688+0000 7fade6e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade01168c0 con 0x7fade0070bc0 2024-02-28T16:48:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.688+0000 7fade6e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade010fb40 con 0x7fade00702b0 2024-02-28T16:48:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.689+0000 7faddf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 0x7fade01161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.689+0000 7faddf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 0x7fade01161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56118/0 (socket says 172.21.15.1:56118) 2024-02-28T16:48:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.689+0000 7faddf400700 1 -- 172.21.15.1:0/572875701 learned_addr learned my addr 172.21.15.1:0/572875701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:05.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.689+0000 7faddf400700 1 -- 172.21.15.1:0/572875701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fade00702b0 msgr2=0x7fade0115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.689+0000 7faddfe00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fade00702b0 0x7fade0115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.689+0000 7faddf400700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fade00702b0 0x7fade0115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.689+0000 7faddf400700 1 -- 172.21.15.1:0/572875701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc005040 con 0x7fade0070bc0 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.689+0000 7faddfe00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fade00702b0 0x7fade0115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.690+0000 7faddf400700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 0x7fade01161e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009100 tx=0x7fadcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.690+0000 7fadd7e00700 1 -- 172.21.15.1:0/572875701 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadcc00e040 con 0x7fade0070bc0 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.690+0000 7fadd7e00700 1 -- 172.21.15.1:0/572875701 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadcc004290 con 0x7fade0070bc0 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.690+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade010fd60 con 0x7fade0070bc0 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.690+0000 7fadd7e00700 1 -- 172.21.15.1:0/572875701 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadcc01c630 con 0x7fade0070bc0 2024-02-28T16:48:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.690+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade01102a0 con 0x7fade0070bc0 2024-02-28T16:48:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.691+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade01069a0 con 0x7fade0070bc0 2024-02-28T16:48:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.692+0000 7fadd7e00700 1 -- 172.21.15.1:0/572875701 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fadcc00bbf0 con 0x7fade0070bc0 2024-02-28T16:48:05.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.693+0000 7fadd7e00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadc406d7e0 0x7fadc406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:05.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.693+0000 7fadd7e00700 1 -- 172.21.15.1:0/572875701 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fadcc08e1d0 con 0x7fade0070bc0 2024-02-28T16:48:05.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.693+0000 7faddfe00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadc406d7e0 0x7fadc406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:05.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.694+0000 7faddfe00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadc406d7e0 0x7fadc406fc90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fadd0006fd0 tx=0x7fadd0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:05.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.698+0000 7fadd7e00700 1 -- 172.21.15.1:0/572875701 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fadcc05b5b0 con 0x7fade0070bc0 2024-02-28T16:48:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.935+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fade0111280 con 0x7fadc406d7e0 2024-02-28T16:48:05.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.938+0000 7fadd7e00700 1 -- 172.21.15.1:0/572875701 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fade0111280 con 0x7fadc406d7e0 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:48:05.928 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:48:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:48:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:48:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:48:05.929 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:48:05.929 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:48:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadc406d7e0 msgr2=0x7fadc406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadc406d7e0 0x7fadc406fc90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fadd0006fd0 tx=0x7fadd0008040 comp rx=0 tx=0).stop 2024-02-28T16:48:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 msgr2=0x7fade01161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:05.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 0x7fade01161e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009100 tx=0x7fadcc009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 shutdown_connections 2024-02-28T16:48:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadc406d7e0 0x7fadc406fc90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fade00702b0 0x7fade0115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 --2- 172.21.15.1:0/572875701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade0070bc0 0x7fade01161e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 >> 172.21.15.1:0/572875701 conn(0x7fade00f8ae0 msgr2=0x7fade0103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 shutdown_connections 2024-02-28T16:48:05.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:05.941+0000 7fade6e00700 1 -- 172.21.15.1:0/572875701 wait complete. 2024-02-28T16:48:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.096+0000 7ffa4e400700 1 -- 172.21.15.1:0/3245941053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105b10 msgr2=0x7ffa480ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.096+0000 7ffa4e400700 1 --2- 172.21.15.1:0/3245941053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105b10 0x7ffa480ff180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffa34004940 tx=0x7ffa34009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.096+0000 7ffa4e400700 1 -- 172.21.15.1:0/3245941053 shutdown_connections 2024-02-28T16:48:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.096+0000 7ffa4e400700 1 --2- 172.21.15.1:0/3245941053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105b10 0x7ffa480ff180 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.096+0000 7ffa4e400700 1 --2- 172.21.15.1:0/3245941053 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa48105170 0x7ffa48105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.096+0000 7ffa4e400700 1 -- 172.21.15.1:0/3245941053 >> 172.21.15.1:0/3245941053 conn(0x7ffa480face0 msgr2=0x7ffa480fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.097+0000 7ffa4e400700 1 -- 172.21.15.1:0/3245941053 shutdown_connections 2024-02-28T16:48:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.097+0000 7ffa4e400700 1 -- 172.21.15.1:0/3245941053 wait complete. 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.097+0000 7ffa4e400700 1 Processor -- start 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.098+0000 7ffa4e400700 1 -- start start 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.098+0000 7ffa4e400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105170 0x7ffa48115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.098+0000 7ffa4e400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa48105b10 0x7ffa48116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.098+0000 7ffa4e400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa48116870 con 0x7ffa48105170 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.098+0000 7ffa4e400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4810fb80 con 0x7ffa48105b10 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.098+0000 7ffa47400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105170 0x7ffa48115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.098+0000 7ffa47400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105170 0x7ffa48115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56132/0 (socket says 172.21.15.1:56132) 2024-02-28T16:48:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.098+0000 7ffa47400700 1 -- 172.21.15.1:0/1027546457 learned_addr learned my addr 172.21.15.1:0/1027546457 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.099+0000 7ffa46a00700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa48105b10 0x7ffa48116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.099+0000 7ffa47400700 1 -- 172.21.15.1:0/1027546457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa48105b10 msgr2=0x7ffa48116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.099+0000 7ffa47400700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa48105b10 0x7ffa48116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.099+0000 7ffa47400700 1 -- 172.21.15.1:0/1027546457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa34005040 con 0x7ffa48105170 2024-02-28T16:48:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.099+0000 7ffa47400700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105170 0x7ffa48115ce0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800c830 tx=0x7ffa38004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.099+0000 7ffa3f400700 1 -- 172.21.15.1:0/1027546457 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa3800a760 con 0x7ffa48105170 2024-02-28T16:48:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.100+0000 7ffa3f400700 1 -- 172.21.15.1:0/1027546457 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa3800ad90 con 0x7ffa48105170 2024-02-28T16:48:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.100+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4810fe60 con 0x7ffa48105170 2024-02-28T16:48:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.100+0000 7ffa3f400700 1 -- 172.21.15.1:0/1027546457 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa380113a0 con 0x7ffa48105170 2024-02-28T16:48:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.100+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa481103a0 con 0x7ffa48105170 2024-02-28T16:48:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.101+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa48107640 con 0x7ffa48105170 2024-02-28T16:48:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.102+0000 7ffa3f400700 1 -- 172.21.15.1:0/1027546457 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffa38012400 con 0x7ffa48105170 2024-02-28T16:48:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.103+0000 7ffa3f400700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa2c06d7f0 0x7ffa2c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.103+0000 7ffa3f400700 1 -- 172.21.15.1:0/1027546457 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffa3808cf00 con 0x7ffa48105170 2024-02-28T16:48:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.103+0000 7ffa46a00700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa2c06d7f0 0x7ffa2c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:06.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.104+0000 7ffa46a00700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa2c06d7f0 0x7ffa2c06fca0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa48111580 tx=0x7ffa34009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.108+0000 7ffa3f400700 1 -- 172.21.15.1:0/1027546457 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffa3805a300 con 0x7ffa48105170 2024-02-28T16:48:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.429+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffa480fe900 con 0x7ffa48105170 2024-02-28T16:48:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.430+0000 7ffa3f400700 1 -- 172.21.15.1:0/1027546457 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ffa480fe900 con 0x7ffa48105170 2024-02-28T16:48:06.420 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:48:06.420 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:48:06.420 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:48:06.420 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:48:06.420 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:48:06.420 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:48:06.420 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:48:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa2c06d7f0 msgr2=0x7ffa2c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:06.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa2c06d7f0 0x7ffa2c06fca0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa48111580 tx=0x7ffa34009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105170 msgr2=0x7ffa48115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105170 0x7ffa48115ce0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800c830 tx=0x7ffa38004470 comp rx=0 tx=0).stop 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 shutdown_connections 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa2c06d7f0 0x7ffa2c06fca0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa48105170 0x7ffa48115ce0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 --2- 172.21.15.1:0/1027546457 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa48105b10 0x7ffa48116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 >> 172.21.15.1:0/1027546457 conn(0x7ffa480face0 msgr2=0x7ffa480fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 shutdown_connections 2024-02-28T16:48:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:06.433+0000 7ffa4e400700 1 -- 172.21.15.1:0/1027546457 wait complete. 2024-02-28T16:48:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:06 smithi066 ceph-mon[85781]: pgmap v409: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:06.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:06 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/770937032' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:48:06.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:06 smithi001 ceph-mon[112940]: pgmap v409: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:06.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:06 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/770937032' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:48:07.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:07 smithi066 ceph-mon[85781]: from='client.34802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:07.498 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:07 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1027546457' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:48:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:07 smithi001 ceph-mon[112940]: from='client.34802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:07.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:07 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1027546457' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:48:08.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:08 smithi001 ceph-mon[112940]: pgmap v410: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:08.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:08 smithi066 ceph-mon[85781]: pgmap v410: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:10.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:10 smithi001 ceph-mon[112940]: pgmap v411: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:10.550 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:10 smithi066 ceph-mon[85781]: pgmap v411: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:11.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:48:11.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:48:12.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:12 smithi066 ceph-mon[85781]: pgmap v412: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:12.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:12 smithi001 ceph-mon[112940]: pgmap v412: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:14.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:14 smithi066 ceph-mon[85781]: pgmap v413: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:14.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:14 smithi001 ceph-mon[112940]: pgmap v413: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:16.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:16 smithi066 ceph-mon[85781]: pgmap v414: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:16.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:16 smithi001 ceph-mon[112940]: pgmap v414: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:18.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:18 smithi066 ceph-mon[85781]: pgmap v415: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:18.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:18 smithi001 ceph-mon[112940]: pgmap v415: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:20.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:20 smithi066 ceph-mon[85781]: pgmap v416: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:20.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:20 smithi001 ceph-mon[112940]: pgmap v416: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:22.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:22 smithi066 ceph-mon[85781]: pgmap v417: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:22.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:22 smithi001 ceph-mon[112940]: pgmap v417: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:24.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:24 smithi066 ceph-mon[85781]: pgmap v418: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:24.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:24 smithi001 ceph-mon[112940]: pgmap v418: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:26.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:26 smithi066 ceph-mon[85781]: pgmap v419: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:26.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:48:26.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:26 smithi001 ceph-mon[112940]: pgmap v419: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:26.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:48:28.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:28 smithi066 ceph-mon[85781]: pgmap v420: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:28.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:28 smithi001 ceph-mon[112940]: pgmap v420: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:30.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:30 smithi066 ceph-mon[85781]: pgmap v421: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:30.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:30 smithi001 ceph-mon[112940]: pgmap v421: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:32.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:32 smithi066 ceph-mon[85781]: pgmap v422: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:32.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:32 smithi001 ceph-mon[112940]: pgmap v422: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:34.748 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:34 smithi066 ceph-mon[85781]: pgmap v423: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:34.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:34 smithi001 ceph-mon[112940]: pgmap v423: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:36.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.607+0000 7fc691400700 1 -- 172.21.15.1:0/1210632100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 msgr2=0x7fc68c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:36.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.607+0000 7fc691400700 1 --2- 172.21.15.1:0/1210632100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 0x7fc68c102210 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc67c004940 tx=0x7fc67c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:36.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.608+0000 7fc691400700 1 -- 172.21.15.1:0/1210632100 shutdown_connections 2024-02-28T16:48:36.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.608+0000 7fc691400700 1 --2- 172.21.15.1:0/1210632100 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc68c1027e0 0x7fc68c0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:36.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.608+0000 7fc691400700 1 --2- 172.21.15.1:0/1210632100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 0x7fc68c102210 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:36.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.608+0000 7fc691400700 1 -- 172.21.15.1:0/1210632100 >> 172.21.15.1:0/1210632100 conn(0x7fc68c071d90 msgr2=0x7fc68c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:36.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.608+0000 7fc691400700 1 -- 172.21.15.1:0/1210632100 shutdown_connections 2024-02-28T16:48:36.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.608+0000 7fc691400700 1 -- 172.21.15.1:0/1210632100 wait complete. 2024-02-28T16:48:36.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.609+0000 7fc691400700 1 Processor -- start 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.609+0000 7fc691400700 1 -- start start 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.609+0000 7fc691400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 0x7fc68c113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.609+0000 7fc691400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc68c1027e0 0x7fc68c113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.609+0000 7fc691400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c114590 con 0x7fc68c101e40 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.609+0000 7fc691400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c10d8c0 con 0x7fc68c1027e0 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.610+0000 7fc68a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 0x7fc68c113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.610+0000 7fc68a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 0x7fc68c113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47230/0 (socket says 172.21.15.1:47230) 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.610+0000 7fc68a800700 1 -- 172.21.15.1:0/2155958939 learned_addr learned my addr 172.21.15.1:0/2155958939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.610+0000 7fc689e00700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc68c1027e0 0x7fc68c113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.610+0000 7fc68a800700 1 -- 172.21.15.1:0/2155958939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc68c1027e0 msgr2=0x7fc68c113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:36.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.610+0000 7fc68a800700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc68c1027e0 0x7fc68c113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:36.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.610+0000 7fc68a800700 1 -- 172.21.15.1:0/2155958939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67c005040 con 0x7fc68c101e40 2024-02-28T16:48:36.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.610+0000 7fc689e00700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc68c1027e0 0x7fc68c113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:48:36.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.611+0000 7fc68a800700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 0x7fc68c113a00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc67c000c00 tx=0x7fc67c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:36.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.611+0000 7fc682a00700 1 -- 172.21.15.1:0/2155958939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc67c021070 con 0x7fc68c101e40 2024-02-28T16:48:36.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.611+0000 7fc682a00700 1 -- 172.21.15.1:0/2155958939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc67c013dd0 con 0x7fc68c101e40 2024-02-28T16:48:36.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.611+0000 7fc682a00700 1 -- 172.21.15.1:0/2155958939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc67c00e070 con 0x7fc68c101e40 2024-02-28T16:48:36.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.611+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68c10db40 con 0x7fc68c101e40 2024-02-28T16:48:36.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.611+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68c10e020 con 0x7fc68c101e40 2024-02-28T16:48:36.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.612+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c0738b0 con 0x7fc68c101e40 2024-02-28T16:48:36.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.613+0000 7fc682a00700 1 -- 172.21.15.1:0/2155958939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc67c01c400 con 0x7fc68c101e40 2024-02-28T16:48:36.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.614+0000 7fc682a00700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67006d7f0 0x7fc67006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:36.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.614+0000 7fc682a00700 1 -- 172.21.15.1:0/2155958939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc67c08e3d0 con 0x7fc68c101e40 2024-02-28T16:48:36.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.615+0000 7fc689e00700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67006d7f0 0x7fc67006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:36.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.615+0000 7fc689e00700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67006d7f0 0x7fc67006fca0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc678004600 tx=0x7fc67800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:36.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.620+0000 7fc682a00700 1 -- 172.21.15.1:0/2155958939 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc67c05b7b0 con 0x7fc68c101e40 2024-02-28T16:48:36.609 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:36 smithi001 ceph-mon[112940]: pgmap v424: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:36.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:36 smithi066 ceph-mon[85781]: pgmap v424: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:36.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.869+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc68c10efe0 con 0x7fc67006d7f0 2024-02-28T16:48:36.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.871+0000 7fc682a00700 1 -- 172.21.15.1:0/2155958939 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc68c10efe0 con 0x7fc67006d7f0 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.874+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67006d7f0 msgr2=0x7fc67006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.874+0000 7fc691400700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67006d7f0 0x7fc67006fca0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc678004600 tx=0x7fc67800b040 comp rx=0 tx=0).stop 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.874+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 msgr2=0x7fc68c113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.874+0000 7fc691400700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 0x7fc68c113a00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc67c000c00 tx=0x7fc67c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.875+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 shutdown_connections 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.875+0000 7fc691400700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc67006d7f0 0x7fc67006fca0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.875+0000 7fc691400700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc68c101e40 0x7fc68c113a00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.875+0000 7fc691400700 1 --2- 172.21.15.1:0/2155958939 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc68c1027e0 0x7fc68c113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:36.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.875+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 >> 172.21.15.1:0/2155958939 conn(0x7fc68c071d90 msgr2=0x7fc68c0656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:36.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.875+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 shutdown_connections 2024-02-28T16:48:36.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:36.875+0000 7fc691400700 1 -- 172.21.15.1:0/2155958939 wait complete. 2024-02-28T16:48:36.881 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:48:37.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.039+0000 7fbf7de00700 1 -- 172.21.15.1:0/2902814443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 msgr2=0x7fbf78104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.039+0000 7fbf7de00700 1 --2- 172.21.15.1:0/2902814443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 0x7fbf78104210 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbf60004940 tx=0x7fbf60009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.039+0000 7fbf7de00700 1 -- 172.21.15.1:0/2902814443 shutdown_connections 2024-02-28T16:48:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.039+0000 7fbf7de00700 1 --2- 172.21.15.1:0/2902814443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 0x7fbf78104210 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.039+0000 7fbf7de00700 1 --2- 172.21.15.1:0/2902814443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf781095f0 0x7fbf781099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.039+0000 7fbf7de00700 1 -- 172.21.15.1:0/2902814443 >> 172.21.15.1:0/2902814443 conn(0x7fbf78072d50 msgr2=0x7fbf78075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.039+0000 7fbf7de00700 1 -- 172.21.15.1:0/2902814443 shutdown_connections 2024-02-28T16:48:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.039+0000 7fbf7de00700 1 -- 172.21.15.1:0/2902814443 wait complete. 2024-02-28T16:48:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.040+0000 7fbf7de00700 1 Processor -- start 2024-02-28T16:48:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.040+0000 7fbf7de00700 1 -- start start 2024-02-28T16:48:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf7de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 0x7fbf7811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf7de00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf781095f0 0x7fbf7811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf7de00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7811abd0 con 0x7fbf78103dc0 2024-02-28T16:48:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf7de00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78113ee0 con 0x7fbf781095f0 2024-02-28T16:48:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf76a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 0x7fbf7811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf76a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 0x7fbf7811a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47238/0 (socket says 172.21.15.1:47238) 2024-02-28T16:48:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf76a00700 1 -- 172.21.15.1:0/351655548 learned_addr learned my addr 172.21.15.1:0/351655548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf6e000700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf781095f0 0x7fbf7811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf76a00700 1 -- 172.21.15.1:0/351655548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf781095f0 msgr2=0x7fbf7811a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf76a00700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf781095f0 0x7fbf7811a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf76a00700 1 -- 172.21.15.1:0/351655548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60005040 con 0x7fbf78103dc0 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.041+0000 7fbf6e000700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf781095f0 0x7fbf7811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.042+0000 7fbf76a00700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 0x7fbf7811a040 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800cb10 tx=0x7fbf68007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.042+0000 7fbf6f400700 1 -- 172.21.15.1:0/351655548 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf68004200 con 0x7fbf78103dc0 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.042+0000 7fbf6f400700 1 -- 172.21.15.1:0/351655548 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf6800acc0 con 0x7fbf78103dc0 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.042+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf781141c0 con 0x7fbf78103dc0 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.042+0000 7fbf6f400700 1 -- 172.21.15.1:0/351655548 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf68003d90 con 0x7fbf78103dc0 2024-02-28T16:48:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.042+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf78114700 con 0x7fbf78103dc0 2024-02-28T16:48:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.043+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7810b9c0 con 0x7fbf78103dc0 2024-02-28T16:48:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.045+0000 7fbf6f400700 1 -- 172.21.15.1:0/351655548 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbf68003430 con 0x7fbf78103dc0 2024-02-28T16:48:37.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.046+0000 7fbf6f400700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbf5c06d7f0 0x7fbf5c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.046+0000 7fbf6f400700 1 -- 172.21.15.1:0/351655548 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbf6808d4f0 con 0x7fbf78103dc0 2024-02-28T16:48:37.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.046+0000 7fbf6e000700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbf5c06d7f0 0x7fbf5c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:37.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.047+0000 7fbf6e000700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbf5c06d7f0 0x7fbf5c06fca0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009fd0 tx=0x7fbf60009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:37.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.051+0000 7fbf6f400700 1 -- 172.21.15.1:0/351655548 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbf6805a7d0 con 0x7fbf78103dc0 2024-02-28T16:48:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.299+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf78115620 con 0x7fbf5c06d7f0 2024-02-28T16:48:37.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.301+0000 7fbf6f400700 1 -- 172.21.15.1:0/351655548 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbf78115620 con 0x7fbf5c06d7f0 2024-02-28T16:48:37.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.304+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbf5c06d7f0 msgr2=0x7fbf5c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.304+0000 7fbf7de00700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbf5c06d7f0 0x7fbf5c06fca0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009fd0 tx=0x7fbf60009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:37.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.304+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 msgr2=0x7fbf7811a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.304+0000 7fbf7de00700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 0x7fbf7811a040 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800cb10 tx=0x7fbf68007f90 comp rx=0 tx=0).stop 2024-02-28T16:48:37.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.305+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 shutdown_connections 2024-02-28T16:48:37.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.305+0000 7fbf7de00700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbf5c06d7f0 0x7fbf5c06fca0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.305+0000 7fbf7de00700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf78103dc0 0x7fbf7811a040 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.305+0000 7fbf7de00700 1 --2- 172.21.15.1:0/351655548 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf781095f0 0x7fbf7811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.305+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 >> 172.21.15.1:0/351655548 conn(0x7fbf78072d50 msgr2=0x7fbf78074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:37.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.305+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 shutdown_connections 2024-02-28T16:48:37.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.305+0000 7fbf7de00700 1 -- 172.21.15.1:0/351655548 wait complete. 2024-02-28T16:48:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.459+0000 7f666f200700 1 -- 172.21.15.1:0/378026898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 msgr2=0x7f667010d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.459+0000 7f666f200700 1 --2- 172.21.15.1:0/378026898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 0x7f667010d700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f665c004940 tx=0x7f665c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.459+0000 7f666f200700 1 -- 172.21.15.1:0/378026898 shutdown_connections 2024-02-28T16:48:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.459+0000 7f666f200700 1 --2- 172.21.15.1:0/378026898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 0x7f667010d700 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.459+0000 7f666f200700 1 --2- 172.21.15.1:0/378026898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66700724c0 0x7f6670072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.459+0000 7f666f200700 1 -- 172.21.15.1:0/378026898 >> 172.21.15.1:0/378026898 conn(0x7f66700fab40 msgr2=0x7f66700fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.459+0000 7f666f200700 1 -- 172.21.15.1:0/378026898 shutdown_connections 2024-02-28T16:48:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.459+0000 7f666f200700 1 -- 172.21.15.1:0/378026898 wait complete. 2024-02-28T16:48:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.460+0000 7f666f200700 1 Processor -- start 2024-02-28T16:48:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.461+0000 7f666f200700 1 -- start start 2024-02-28T16:48:37.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.461+0000 7f666f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66700724c0 0x7f6670119d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.461+0000 7f666f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 0x7f667011a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.461+0000 7f666f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667011a8e0 con 0x7f6670072dd0 2024-02-28T16:48:37.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.461+0000 7f666f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6670113bf0 con 0x7f66700724c0 2024-02-28T16:48:37.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.462+0000 7f666d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 0x7f667011a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:37.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.462+0000 7f666d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 0x7f667011a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47256/0 (socket says 172.21.15.1:47256) 2024-02-28T16:48:37.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.462+0000 7f666d400700 1 -- 172.21.15.1:0/2660323733 learned_addr learned my addr 172.21.15.1:0/2660323733 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.462+0000 7f666d400700 1 -- 172.21.15.1:0/2660323733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66700724c0 msgr2=0x7f6670119d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.462+0000 7f666d400700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66700724c0 0x7f6670119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.462+0000 7f666d400700 1 -- 172.21.15.1:0/2660323733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c005040 con 0x7f6670072dd0 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.462+0000 7f666d400700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 0x7f667011a290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f665c004890 tx=0x7f665c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.463+0000 7f6666000700 1 -- 172.21.15.1:0/2660323733 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f665c00e040 con 0x7f6670072dd0 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.463+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6670113e70 con 0x7f6670072dd0 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.463+0000 7f6666000700 1 -- 172.21.15.1:0/2660323733 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f665c004290 con 0x7f6670072dd0 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.463+0000 7f6666000700 1 -- 172.21.15.1:0/2660323733 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f665c01c630 con 0x7f6670072dd0 2024-02-28T16:48:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.463+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6670114430 con 0x7f6670072dd0 2024-02-28T16:48:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.464+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667010ae30 con 0x7f6670072dd0 2024-02-28T16:48:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.465+0000 7f6666000700 1 -- 172.21.15.1:0/2660323733 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f665c00bbf0 con 0x7f6670072dd0 2024-02-28T16:48:37.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.466+0000 7f6666000700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f665406d7e0 0x7f665406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.466+0000 7f6666000700 1 -- 172.21.15.1:0/2660323733 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f665c08e310 con 0x7f6670072dd0 2024-02-28T16:48:37.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.466+0000 7f666de00700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f665406d7e0 0x7f665406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.467+0000 7f666de00700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f665406d7e0 0x7f665406fc90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6660004d00 tx=0x7f6660009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:37.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.470+0000 7f6666000700 1 -- 172.21.15.1:0/2660323733 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f665c05b6f0 con 0x7f6670072dd0 2024-02-28T16:48:37.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.699+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6670115330 con 0x7f665406d7e0 2024-02-28T16:48:37.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.710+0000 7f6666000700 1 -- 172.21.15.1:0/2660323733 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6670115330 con 0x7f665406d7e0 2024-02-28T16:48:37.699 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (14m) 6m ago 28m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (11m) 6m ago 29m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (11m) 7m ago 27m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (14m) 6m ago 28m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (7m) 6m ago 21m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (6m) 6m ago 21m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (21m) 7m ago 21m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (21m) 7m ago 21m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (17m) 6m ago 30m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (16m) 7m ago 27m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (13m) 6m ago 30m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (12m) 7m ago 27m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:48:37.700 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (15m) 6m ago 29m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:48:37.701 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (15m) 7m ago 27m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:48:37.701 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (10m) 6m ago 26m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:48:37.701 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (10m) 6m ago 26m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:48:37.701 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (9m) 6m ago 25m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:48:37.701 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (9m) 7m ago 24m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:48:37.701 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (8m) 7m ago 24m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:48:37.701 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (7m) 7m ago 23m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:48:37.701 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (15m) 6m ago 28m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:48:37.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f665406d7e0 msgr2=0x7f665406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f665406d7e0 0x7f665406fc90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6660004d00 tx=0x7f6660009ce0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 msgr2=0x7f667011a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 0x7f667011a290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f665c004890 tx=0x7f665c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:37.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 shutdown_connections 2024-02-28T16:48:37.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f665406d7e0 0x7f665406fc90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66700724c0 0x7f6670119d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 --2- 172.21.15.1:0/2660323733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6670072dd0 0x7f667011a290 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.713+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 >> 172.21.15.1:0/2660323733 conn(0x7f66700fab40 msgr2=0x7f66700fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:37.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.714+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 shutdown_connections 2024-02-28T16:48:37.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.714+0000 7f666f200700 1 -- 172.21.15.1:0/2660323733 wait complete. 2024-02-28T16:48:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.874+0000 7f3689c00700 1 -- 172.21.15.1:0/1651404207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 msgr2=0x7f36840fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.874+0000 7f3689c00700 1 --2- 172.21.15.1:0/1651404207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 0x7f36840fcd50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f366c004940 tx=0x7f366c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.875+0000 7f3689c00700 1 -- 172.21.15.1:0/1651404207 shutdown_connections 2024-02-28T16:48:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.875+0000 7f3689c00700 1 --2- 172.21.15.1:0/1651404207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 0x7f36840fcd50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.875+0000 7f3689c00700 1 --2- 172.21.15.1:0/1651404207 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3684102d40 0x7f3684103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.875+0000 7f3689c00700 1 -- 172.21.15.1:0/1651404207 >> 172.21.15.1:0/1651404207 conn(0x7f36840f88b0 msgr2=0x7f36840facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.875+0000 7f3689c00700 1 -- 172.21.15.1:0/1651404207 shutdown_connections 2024-02-28T16:48:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.875+0000 7f3689c00700 1 -- 172.21.15.1:0/1651404207 wait complete. 2024-02-28T16:48:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.876+0000 7f3689c00700 1 Processor -- start 2024-02-28T16:48:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.876+0000 7f3689c00700 1 -- start start 2024-02-28T16:48:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f3689c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3684102d40 0x7f3684115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f3689c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 0x7f3684115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f3689c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3684116600 con 0x7f36841036e0 2024-02-28T16:48:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f3689c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368410f930 con 0x7f3684102d40 2024-02-28T16:48:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f367be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 0x7f3684115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f367be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 0x7f3684115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47278/0 (socket says 172.21.15.1:47278) 2024-02-28T16:48:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f367be00700 1 -- 172.21.15.1:0/319308423 learned_addr learned my addr 172.21.15.1:0/319308423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f3683e00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3684102d40 0x7f3684115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.877+0000 7f367be00700 1 -- 172.21.15.1:0/319308423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3684102d40 msgr2=0x7f3684115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.878+0000 7f367be00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3684102d40 0x7f3684115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.878+0000 7f367be00700 1 -- 172.21.15.1:0/319308423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c005040 con 0x7f36841036e0 2024-02-28T16:48:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.878+0000 7f367be00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 0x7f3684115fb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f366c009030 tx=0x7f366c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:37.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.878+0000 7f3680c00700 1 -- 172.21.15.1:0/319308423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366c00e040 con 0x7f36841036e0 2024-02-28T16:48:37.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.878+0000 7f3680c00700 1 -- 172.21.15.1:0/319308423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f366c00bbf0 con 0x7f36841036e0 2024-02-28T16:48:37.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.878+0000 7f3680c00700 1 -- 172.21.15.1:0/319308423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366c01c630 con 0x7f36841036e0 2024-02-28T16:48:37.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.878+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368410fbb0 con 0x7f36841036e0 2024-02-28T16:48:37.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.878+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3684110170 con 0x7f36841036e0 2024-02-28T16:48:37.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.880+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368405f5c0 con 0x7f36841036e0 2024-02-28T16:48:37.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.881+0000 7f3680c00700 1 -- 172.21.15.1:0/319308423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f366c00bd60 con 0x7f36841036e0 2024-02-28T16:48:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.881+0000 7f3680c00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f366806d7f0 0x7f366806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.882+0000 7f3680c00700 1 -- 172.21.15.1:0/319308423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f366c08e1f0 con 0x7f36841036e0 2024-02-28T16:48:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.882+0000 7f3683e00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f366806d7f0 0x7f366806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.883+0000 7f3683e00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f366806d7f0 0x7f366806fca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3674007fd0 tx=0x7f3674009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:37.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:37.886+0000 7f3680c00700 1 -- 172.21.15.1:0/319308423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f366c05b5d0 con 0x7f36841036e0 2024-02-28T16:48:38.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.200+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f368404bed0 con 0x7f36841036e0 2024-02-28T16:48:38.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.202+0000 7f3680c00700 1 -- 172.21.15.1:0/319308423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f366c05b160 con 0x7f36841036e0 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:48:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:48:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:48:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:48:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:48:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:48:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:48:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:48:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:48:38.193 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:48:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.205+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f366806d7f0 msgr2=0x7f366806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f366806d7f0 0x7f366806fca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3674007fd0 tx=0x7f3674009040 comp rx=0 tx=0).stop 2024-02-28T16:48:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 msgr2=0x7f3684115fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 0x7f3684115fb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f366c009030 tx=0x7f366c009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:38.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 shutdown_connections 2024-02-28T16:48:38.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f366806d7f0 0x7f366806fca0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3684102d40 0x7f3684115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 --2- 172.21.15.1:0/319308423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36841036e0 0x7f3684115fb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 >> 172.21.15.1:0/319308423 conn(0x7f36840f88b0 msgr2=0x7f36840fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:38.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 shutdown_connections 2024-02-28T16:48:38.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.206+0000 7f3689c00700 1 -- 172.21.15.1:0/319308423 wait complete. 2024-02-28T16:48:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.364+0000 7f4c76e00700 1 -- 172.21.15.1:0/2584869829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fdfb0 msgr2=0x7f4c700fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.364+0000 7f4c76e00700 1 --2- 172.21.15.1:0/2584869829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fdfb0 0x7f4c700fe380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4c60004940 tx=0x7f4c60009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.364+0000 7f4c76e00700 1 -- 172.21.15.1:0/2584869829 shutdown_connections 2024-02-28T16:48:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.364+0000 7f4c76e00700 1 --2- 172.21.15.1:0/2584869829 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c700fe950 0x7f4c70106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.364+0000 7f4c76e00700 1 --2- 172.21.15.1:0/2584869829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fdfb0 0x7f4c700fe380 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.364+0000 7f4c76e00700 1 -- 172.21.15.1:0/2584869829 >> 172.21.15.1:0/2584869829 conn(0x7f4c700f79e0 msgr2=0x7f4c700f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.364+0000 7f4c76e00700 1 -- 172.21.15.1:0/2584869829 shutdown_connections 2024-02-28T16:48:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.365+0000 7f4c76e00700 1 -- 172.21.15.1:0/2584869829 wait complete. 2024-02-28T16:48:38.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.365+0000 7f4c76e00700 1 Processor -- start 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.366+0000 7f4c76e00700 1 -- start start 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.366+0000 7f4c76e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c700fdfb0 0x7f4c700fbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.366+0000 7f4c76e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fe950 0x7f4c700fc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.366+0000 7f4c76e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700fc9b0 con 0x7f4c700fe950 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.366+0000 7f4c76e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700fcb20 con 0x7f4c700fdfb0 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.366+0000 7f4c6f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fe950 0x7f4c700fc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.366+0000 7f4c6f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fe950 0x7f4c700fc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47288/0 (socket says 172.21.15.1:47288) 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.366+0000 7f4c6f400700 1 -- 172.21.15.1:0/3434391618 learned_addr learned my addr 172.21.15.1:0/3434391618 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c6fe00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c700fdfb0 0x7f4c700fbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c6f400700 1 -- 172.21.15.1:0/3434391618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c700fdfb0 msgr2=0x7f4c700fbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c6f400700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c700fdfb0 0x7f4c700fbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c6f400700 1 -- 172.21.15.1:0/3434391618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60005040 con 0x7f4c700fe950 2024-02-28T16:48:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c6f400700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fe950 0x7f4c700fc320 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b790 tx=0x7f4c5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c67e00700 1 -- 172.21.15.1:0/3434391618 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5c00f770 con 0x7f4c700fe950 2024-02-28T16:48:38.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c67e00700 1 -- 172.21.15.1:0/3434391618 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c5c00fda0 con 0x7f4c700fe950 2024-02-28T16:48:38.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c70119d70 con 0x7f4c700fe950 2024-02-28T16:48:38.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.367+0000 7f4c67e00700 1 -- 172.21.15.1:0/3434391618 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5c00d310 con 0x7f4c700fe950 2024-02-28T16:48:38.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.368+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7011a260 con 0x7f4c700fe950 2024-02-28T16:48:38.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.369+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c701045a0 con 0x7f4c700fe950 2024-02-28T16:48:38.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.370+0000 7f4c67e00700 1 -- 172.21.15.1:0/3434391618 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4c5c010420 con 0x7f4c700fe950 2024-02-28T16:48:38.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.371+0000 7f4c67e00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5406d7f0 0x7f4c5406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:38.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.371+0000 7f4c67e00700 1 -- 172.21.15.1:0/3434391618 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4c5c08ce80 con 0x7f4c700fe950 2024-02-28T16:48:38.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.371+0000 7f4c6fe00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5406d7f0 0x7f4c5406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:38.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.372+0000 7f4c6fe00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5406d7f0 0x7f4c5406fca0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009fd0 tx=0x7f4c60009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:38.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.376+0000 7f4c67e00700 1 -- 172.21.15.1:0/3434391618 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c5c056730 con 0x7f4c700fe950 2024-02-28T16:48:38.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:38 smithi001 ceph-mon[112940]: from='client.34810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:38.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:38 smithi001 ceph-mon[112940]: pgmap v425: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:38.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:38 smithi001 ceph-mon[112940]: from='client.34814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:38.521 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:38 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/319308423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:48:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.640+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4c7004bed0 con 0x7f4c700fe950 2024-02-28T16:48:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.640+0000 7f4c67e00700 1 -- 172.21.15.1:0/3434391618 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f4c5c059df0 con 0x7f4c700fe950 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:48:38.630 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:48:38.631 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:48:38.632 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:48:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:48:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.643+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5406d7f0 msgr2=0x7f4c5406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.643+0000 7f4c76e00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5406d7f0 0x7f4c5406fca0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009fd0 tx=0x7f4c60009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.643+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fe950 msgr2=0x7f4c700fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.643+0000 7f4c76e00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fe950 0x7f4c700fc320 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b790 tx=0x7f4c5c009250 comp rx=0 tx=0).stop 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.644+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 shutdown_connections 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.644+0000 7f4c76e00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c5406d7f0 0x7f4c5406fca0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.644+0000 7f4c76e00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c700fdfb0 0x7f4c700fbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.644+0000 7f4c76e00700 1 --2- 172.21.15.1:0/3434391618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700fe950 0x7f4c700fc320 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.644+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 >> 172.21.15.1:0/3434391618 conn(0x7f4c700f79e0 msgr2=0x7f4c70101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.644+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 shutdown_connections 2024-02-28T16:48:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.644+0000 7f4c76e00700 1 -- 172.21.15.1:0/3434391618 wait complete. 2024-02-28T16:48:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:48:38.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:38 smithi066 ceph-mon[85781]: from='client.34810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:38.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:38 smithi066 ceph-mon[85781]: pgmap v425: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:38.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:38 smithi066 ceph-mon[85781]: from='client.34814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:38.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:38 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/319308423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:48:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.794+0000 7f4a2ce00700 1 -- 172.21.15.1:0/2614749553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 msgr2=0x7f4a28072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.794+0000 7f4a2ce00700 1 --2- 172.21.15.1:0/2614749553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 0x7f4a28072890 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4a18004940 tx=0x7f4a18009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.794+0000 7f4a2ce00700 1 -- 172.21.15.1:0/2614749553 shutdown_connections 2024-02-28T16:48:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.794+0000 7f4a2ce00700 1 --2- 172.21.15.1:0/2614749553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a28072dd0 0x7f4a2810d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.794+0000 7f4a2ce00700 1 --2- 172.21.15.1:0/2614749553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 0x7f4a28072890 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.794+0000 7f4a2ce00700 1 -- 172.21.15.1:0/2614749553 >> 172.21.15.1:0/2614749553 conn(0x7f4a280fab10 msgr2=0x7f4a280fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.795+0000 7f4a2ce00700 1 -- 172.21.15.1:0/2614749553 shutdown_connections 2024-02-28T16:48:38.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.795+0000 7f4a2ce00700 1 -- 172.21.15.1:0/2614749553 wait complete. 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.795+0000 7f4a2ce00700 1 Processor -- start 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.796+0000 7f4a2ce00700 1 -- start start 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.796+0000 7f4a2ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 0x7f4a28119d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.796+0000 7f4a2ce00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a28072dd0 0x7f4a2811a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.796+0000 7f4a2ce00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2811a920 con 0x7f4a280724c0 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.796+0000 7f4a2ce00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a28113c30 con 0x7f4a28072dd0 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.796+0000 7f4a27400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 0x7f4a28119d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.796+0000 7f4a27400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 0x7f4a28119d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47308/0 (socket says 172.21.15.1:47308) 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.796+0000 7f4a27400700 1 -- 172.21.15.1:0/401725784 learned_addr learned my addr 172.21.15.1:0/401725784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.797+0000 7f4a26a00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a28072dd0 0x7f4a2811a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.797+0000 7f4a27400700 1 -- 172.21.15.1:0/401725784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a28072dd0 msgr2=0x7f4a2811a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.797+0000 7f4a27400700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a28072dd0 0x7f4a2811a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.797+0000 7f4a27400700 1 -- 172.21.15.1:0/401725784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a18005040 con 0x7f4a280724c0 2024-02-28T16:48:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.797+0000 7f4a26a00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a28072dd0 0x7f4a2811a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:48:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.797+0000 7f4a27400700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 0x7f4a28119d90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009180 tx=0x7f4a18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.797+0000 7f4a1f400700 1 -- 172.21.15.1:0/401725784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a1800e040 con 0x7f4a280724c0 2024-02-28T16:48:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.797+0000 7f4a1f400700 1 -- 172.21.15.1:0/401725784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a18004290 con 0x7f4a280724c0 2024-02-28T16:48:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.798+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a28113eb0 con 0x7f4a280724c0 2024-02-28T16:48:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.798+0000 7f4a1f400700 1 -- 172.21.15.1:0/401725784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a1801c630 con 0x7f4a280724c0 2024-02-28T16:48:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.798+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a28114470 con 0x7f4a280724c0 2024-02-28T16:48:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.799+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2810ae20 con 0x7f4a280724c0 2024-02-28T16:48:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.800+0000 7f4a1f400700 1 -- 172.21.15.1:0/401725784 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4a180139f0 con 0x7f4a280724c0 2024-02-28T16:48:38.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.801+0000 7f4a1f400700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:38.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.801+0000 7f4a1f400700 1 -- 172.21.15.1:0/401725784 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4a1808e2f0 con 0x7f4a280724c0 2024-02-28T16:48:38.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.801+0000 7f4a26a00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:38.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.802+0000 7f4a26a00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009fd0 tx=0x7f4a1400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:38.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:38.805+0000 7f4a1f400700 1 -- 172.21.15.1:0/401725784 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a1805b6d0 con 0x7f4a280724c0 2024-02-28T16:48:39.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.050+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a2804bed0 con 0x7f4a0c06d7f0 2024-02-28T16:48:39.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.053+0000 7f4a1f400700 1 -- 172.21.15.1:0/401725784 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4a2804bed0 con 0x7f4a0c06d7f0 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:48:39.043 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 msgr2=0x7f4a0c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009fd0 tx=0x7f4a1400b040 comp rx=0 tx=0).stop 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 msgr2=0x7f4a28119d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 0x7f4a28119d90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009180 tx=0x7f4a18009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 shutdown_connections 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a280724c0 0x7f4a28119d90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 --2- 172.21.15.1:0/401725784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a28072dd0 0x7f4a2811a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.056+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 >> 172.21.15.1:0/401725784 conn(0x7f4a280fab10 msgr2=0x7f4a280fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:39.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.057+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 shutdown_connections 2024-02-28T16:48:39.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.057+0000 7f4a2ce00700 1 -- 172.21.15.1:0/401725784 wait complete. 2024-02-28T16:48:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.200+0000 7f872f000700 1 -- 172.21.15.1:0/2958691526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728071f60 msgr2=0x7f8728070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.200+0000 7f872f000700 1 --2- 172.21.15.1:0/2958691526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728071f60 0x7f8728070330 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8718004970 tx=0x7f8718009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.200+0000 7f872f000700 1 -- 172.21.15.1:0/2958691526 shutdown_connections 2024-02-28T16:48:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.200+0000 7f872f000700 1 --2- 172.21.15.1:0/2958691526 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8728070870 0x7f8728070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.200+0000 7f872f000700 1 --2- 172.21.15.1:0/2958691526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728071f60 0x7f8728070330 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.200+0000 7f872f000700 1 -- 172.21.15.1:0/2958691526 >> 172.21.15.1:0/2958691526 conn(0x7f87280f8a70 msgr2=0x7f87280fae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.200+0000 7f872f000700 1 -- 172.21.15.1:0/2958691526 shutdown_connections 2024-02-28T16:48:39.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.200+0000 7f872f000700 1 -- 172.21.15.1:0/2958691526 wait complete. 2024-02-28T16:48:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.201+0000 7f872f000700 1 Processor -- start 2024-02-28T16:48:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.201+0000 7f872f000700 1 -- start start 2024-02-28T16:48:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.202+0000 7f872f000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728070870 0x7f8728119f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.202+0000 7f872f000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8728071f60 0x7f872811a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.202+0000 7f872f000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872811aaa0 con 0x7f8728070870 2024-02-28T16:48:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.202+0000 7f872f000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8728113df0 con 0x7f8728071f60 2024-02-28T16:48:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.202+0000 7f8727e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728070870 0x7f8728119f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.202+0000 7f8727e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728070870 0x7f8728119f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47326/0 (socket says 172.21.15.1:47326) 2024-02-28T16:48:39.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.202+0000 7f8727e00700 1 -- 172.21.15.1:0/1528322590 learned_addr learned my addr 172.21.15.1:0/1528322590 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f8727e00700 1 -- 172.21.15.1:0/1528322590 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8728071f60 msgr2=0x7f872811a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f8727e00700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8728071f60 0x7f872811a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f8727e00700 1 -- 172.21.15.1:0/1528322590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8718005040 con 0x7f8728070870 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f8727e00700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728070870 0x7f8728119f10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8718004f50 tx=0x7f8718009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f871fe00700 1 -- 172.21.15.1:0/1528322590 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f871800e040 con 0x7f8728070870 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f871fe00700 1 -- 172.21.15.1:0/1528322590 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f871800bc10 con 0x7f8728070870 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8728114070 con 0x7f8728070870 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f871fe00700 1 -- 172.21.15.1:0/1528322590 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f871801c630 con 0x7f8728070870 2024-02-28T16:48:39.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.203+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87281145b0 con 0x7f8728070870 2024-02-28T16:48:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.204+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87280fe390 con 0x7f8728070870 2024-02-28T16:48:39.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.206+0000 7f871fe00700 1 -- 172.21.15.1:0/1528322590 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f871800bd80 con 0x7f8728070870 2024-02-28T16:48:39.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.206+0000 7f871fe00700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f870c06d7f0 0x7f870c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:48:39.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.207+0000 7f871fe00700 1 -- 172.21.15.1:0/1528322590 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f871808e370 con 0x7f8728070870 2024-02-28T16:48:39.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.207+0000 7f8727400700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f870c06d7f0 0x7f870c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:48:39.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.207+0000 7f8727400700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f870c06d7f0 0x7f870c06fca0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f87281157f0 tx=0x7f8714007cc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:48:39.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.211+0000 7f871fe00700 1 -- 172.21.15.1:0/1528322590 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8718057ba0 con 0x7f8728070870 2024-02-28T16:48:39.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.519+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f872804bed0 con 0x7f8728070870 2024-02-28T16:48:39.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.520+0000 7f871fe00700 1 -- 172.21.15.1:0/1528322590 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f871805b260 con 0x7f8728070870 2024-02-28T16:48:39.509 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:48:39.510 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:48:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:48:39.510 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:48:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:48:39.510 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:48:39.510 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:48:39.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f870c06d7f0 msgr2=0x7f870c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:39.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f870c06d7f0 0x7f870c06fca0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f87281157f0 tx=0x7f8714007cc0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728070870 msgr2=0x7f8728119f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:48:39.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728070870 0x7f8728119f10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8718004f50 tx=0x7f8718009e30 comp rx=0 tx=0).stop 2024-02-28T16:48:39.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 shutdown_connections 2024-02-28T16:48:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f870c06d7f0 0x7f870c06fca0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8728070870 0x7f8728119f10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 --2- 172.21.15.1:0/1528322590 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8728071f60 0x7f872811a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:48:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 >> 172.21.15.1:0/1528322590 conn(0x7f87280f8a70 msgr2=0x7f87280fa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:48:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 shutdown_connections 2024-02-28T16:48:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:48:39.523+0000 7f872f000700 1 -- 172.21.15.1:0/1528322590 wait complete. 2024-02-28T16:48:39.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:39 smithi066 ceph-mon[85781]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:39.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:39 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3434391618' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:48:39.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:39 smithi001 ceph-mon[112940]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:39.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:39 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3434391618' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:48:40.650 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:40 smithi066 ceph-mon[85781]: pgmap v426: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:40.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:40 smithi066 ceph-mon[85781]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:40.651 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1528322590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:48:40.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:40 smithi001 ceph-mon[112940]: pgmap v426: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:40.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:40 smithi001 ceph-mon[112940]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:48:40.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1528322590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:48:41.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:48:41.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:48:42.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:42 smithi066 ceph-mon[85781]: pgmap v427: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:42.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:42 smithi001 ceph-mon[112940]: pgmap v427: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:44.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:44 smithi066 ceph-mon[85781]: pgmap v428: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:44.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:44 smithi001 ceph-mon[112940]: pgmap v428: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:46.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:46 smithi066 ceph-mon[85781]: pgmap v429: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:46.770 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:46 smithi001 ceph-mon[112940]: pgmap v429: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:48.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:48 smithi066 ceph-mon[85781]: pgmap v430: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:48.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:48 smithi001 ceph-mon[112940]: pgmap v430: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:50.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:50 smithi066 ceph-mon[85781]: pgmap v431: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:50.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:50 smithi001 ceph-mon[112940]: pgmap v431: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:52.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:52 smithi066 ceph-mon[85781]: pgmap v432: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:52.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:52 smithi001 ceph-mon[112940]: pgmap v432: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:54.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:54 smithi066 ceph-mon[85781]: pgmap v433: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:54.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:54 smithi001 ceph-mon[112940]: pgmap v433: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:56.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:56 smithi066 ceph-mon[85781]: pgmap v434: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:56.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:48:56.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:56 smithi001 ceph-mon[112940]: pgmap v434: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:56.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:48:58.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:48:58 smithi066 ceph-mon[85781]: pgmap v435: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:48:58.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:48:58 smithi001 ceph-mon[112940]: pgmap v435: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:00.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:00 smithi066 ceph-mon[85781]: pgmap v436: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:00.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:00 smithi001 ceph-mon[112940]: pgmap v436: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:02.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:02 smithi066 ceph-mon[85781]: pgmap v437: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:02.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:02 smithi001 ceph-mon[112940]: pgmap v437: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:04.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:04 smithi066 ceph-mon[85781]: pgmap v438: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:04.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:04 smithi001 ceph-mon[112940]: pgmap v438: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:06.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:06 smithi066 ceph-mon[85781]: pgmap v439: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:06.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:06 smithi001 ceph-mon[112940]: pgmap v439: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:08.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:08 smithi066 ceph-mon[85781]: pgmap v440: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:08.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:08 smithi001 ceph-mon[112940]: pgmap v440: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:09.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.694+0000 7fe9d3400700 1 -- 172.21.15.1:0/2024333465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fc6d0 msgr2=0x7fe9cc100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:09.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.694+0000 7fe9d3400700 1 --2- 172.21.15.1:0/2024333465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fc6d0 0x7fe9cc100cf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8004940 tx=0x7fe9b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:09.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.695+0000 7fe9d3400700 1 -- 172.21.15.1:0/2024333465 shutdown_connections 2024-02-28T16:49:09.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.695+0000 7fe9d3400700 1 --2- 172.21.15.1:0/2024333465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fc6d0 0x7fe9cc100cf0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:09.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.695+0000 7fe9d3400700 1 --2- 172.21.15.1:0/2024333465 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe9cc0fbd30 0x7fe9cc0fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:09.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.695+0000 7fe9d3400700 1 -- 172.21.15.1:0/2024333465 >> 172.21.15.1:0/2024333465 conn(0x7fe9cc0f79d0 msgr2=0x7fe9cc0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:09.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.695+0000 7fe9d3400700 1 -- 172.21.15.1:0/2024333465 shutdown_connections 2024-02-28T16:49:09.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.695+0000 7fe9d3400700 1 -- 172.21.15.1:0/2024333465 wait complete. 2024-02-28T16:49:09.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.697+0000 7fe9d3400700 1 Processor -- start 2024-02-28T16:49:09.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.697+0000 7fe9d3400700 1 -- start start 2024-02-28T16:49:09.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.697+0000 7fe9d3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fbd30 0x7fe9cc11a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:09.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.697+0000 7fe9d3400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe9cc0fc6d0 0x7fe9cc11a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:09.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.697+0000 7fe9d3400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc11ac80 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.697+0000 7fe9d3400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc113f90 con 0x7fe9cc0fc6d0 2024-02-28T16:49:09.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.698+0000 7fe9d0a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fbd30 0x7fe9cc11a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:09.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.698+0000 7fe9d0a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fbd30 0x7fe9cc11a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43266/0 (socket says 172.21.15.1:43266) 2024-02-28T16:49:09.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.698+0000 7fe9d0a00700 1 -- 172.21.15.1:0/1685124089 learned_addr learned my addr 172.21.15.1:0/1685124089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:09.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.698+0000 7fe9d0a00700 1 -- 172.21.15.1:0/1685124089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe9cc0fc6d0 msgr2=0x7fe9cc11a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:09.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.698+0000 7fe9cbe00700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe9cc0fc6d0 0x7fe9cc11a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:09.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.698+0000 7fe9d0a00700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe9cc0fc6d0 0x7fe9cc11a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:09.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.698+0000 7fe9d0a00700 1 -- 172.21.15.1:0/1685124089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b8005040 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.698+0000 7fe9cbe00700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe9cc0fc6d0 0x7fe9cc11a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:49:09.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.699+0000 7fe9d0a00700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fbd30 0x7fe9cc11a0f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0002a50 tx=0x7fe9c00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:09.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.699+0000 7fe9c8c00700 1 -- 172.21.15.1:0/1685124089 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c0007cc0 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.699+0000 7fe9c8c00700 1 -- 172.21.15.1:0/1685124089 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9c000f040 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.699+0000 7fe9c8c00700 1 -- 172.21.15.1:0/1685124089 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c0009640 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.699+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc114270 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.699+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc1147b0 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.701+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc0fe410 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.702+0000 7fe9c8c00700 1 -- 172.21.15.1:0/1685124089 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe9c00097a0 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.702+0000 7fe9c8c00700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe9b406d7f0 0x7fe9b406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:09.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.703+0000 7fe9c8c00700 1 -- 172.21.15.1:0/1685124089 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe9c008cbc0 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.703+0000 7fe9cbe00700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe9b406d7f0 0x7fe9b406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:09.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.704+0000 7fe9cbe00700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe9b406d7f0 0x7fe9b406fca0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800b3b0 tx=0x7fe9b8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:09.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.708+0000 7fe9c8c00700 1 -- 172.21.15.1:0/1685124089 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe9c0059fc0 con 0x7fe9cc0fbd30 2024-02-28T16:49:09.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.950+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9cc04bf40 con 0x7fe9b406d7f0 2024-02-28T16:49:09.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.953+0000 7fe9c8c00700 1 -- 172.21.15.1:0/1685124089 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe9cc04bf40 con 0x7fe9b406d7f0 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe9b406d7f0 msgr2=0x7fe9b406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe9b406d7f0 0x7fe9b406fca0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800b3b0 tx=0x7fe9b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fbd30 msgr2=0x7fe9cc11a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fbd30 0x7fe9cc11a0f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0002a50 tx=0x7fe9c00046b0 comp rx=0 tx=0).stop 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 shutdown_connections 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe9b406d7f0 0x7fe9b406fca0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9cc0fbd30 0x7fe9cc11a0f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 --2- 172.21.15.1:0/1685124089 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe9cc0fc6d0 0x7fe9cc11a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.955+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 >> 172.21.15.1:0/1685124089 conn(0x7fe9cc0f79d0 msgr2=0x7fe9cc10d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.956+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 shutdown_connections 2024-02-28T16:49:09.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:09.956+0000 7fe9d3400700 1 -- 172.21.15.1:0/1685124089 wait complete. 2024-02-28T16:49:09.962 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:49:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.117+0000 7f2c9be00700 1 -- 172.21.15.1:0/2355498471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c105930 msgr2=0x7f2c9c0fefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.117+0000 7f2c9be00700 1 --2- 172.21.15.1:0/2355498471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c105930 0x7f2c9c0fefa0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2c84004940 tx=0x7f2c84009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.117+0000 7f2c9be00700 1 -- 172.21.15.1:0/2355498471 shutdown_connections 2024-02-28T16:49:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.117+0000 7f2c9be00700 1 --2- 172.21.15.1:0/2355498471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c105930 0x7f2c9c0fefa0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.117+0000 7f2c9be00700 1 --2- 172.21.15.1:0/2355498471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c9c104f90 0x7f2c9c105360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.117+0000 7f2c9be00700 1 -- 172.21.15.1:0/2355498471 >> 172.21.15.1:0/2355498471 conn(0x7f2c9c0fab40 msgr2=0x7f2c9c0fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.117+0000 7f2c9be00700 1 -- 172.21.15.1:0/2355498471 shutdown_connections 2024-02-28T16:49:10.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.117+0000 7f2c9be00700 1 -- 172.21.15.1:0/2355498471 wait complete. 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.118+0000 7f2c9be00700 1 Processor -- start 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.118+0000 7f2c9be00700 1 -- start start 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.118+0000 7f2c9be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c104f90 0x7f2c9c115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.118+0000 7f2c9be00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c9c105930 0x7f2c9c115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.118+0000 7f2c9be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c1165e0 con 0x7f2c9c104f90 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.118+0000 7f2c9be00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c10f910 con 0x7f2c9c105930 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.119+0000 7f2c9aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c104f90 0x7f2c9c115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.119+0000 7f2c9aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c104f90 0x7f2c9c115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43280/0 (socket says 172.21.15.1:43280) 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.119+0000 7f2c9aa00700 1 -- 172.21.15.1:0/139630575 learned_addr learned my addr 172.21.15.1:0/139630575 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:10.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.119+0000 7f2c9aa00700 1 -- 172.21.15.1:0/139630575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c9c105930 msgr2=0x7f2c9c115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.119+0000 7f2c9aa00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c9c105930 0x7f2c9c115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.119+0000 7f2c9aa00700 1 -- 172.21.15.1:0/139630575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c84005040 con 0x7f2c9c104f90 2024-02-28T16:49:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.119+0000 7f2c9aa00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c104f90 0x7f2c9c115a50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c002a50 tx=0x7f2c8c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.120+0000 7f2c92a00700 1 -- 172.21.15.1:0/139630575 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c8c007cc0 con 0x7f2c9c104f90 2024-02-28T16:49:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.120+0000 7f2c92a00700 1 -- 172.21.15.1:0/139630575 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c8c00f040 con 0x7f2c9c104f90 2024-02-28T16:49:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.120+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c10fbf0 con 0x7f2c9c104f90 2024-02-28T16:49:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.120+0000 7f2c92a00700 1 -- 172.21.15.1:0/139630575 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c8c009640 con 0x7f2c9c104f90 2024-02-28T16:49:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.120+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c1101b0 con 0x7f2c9c104f90 2024-02-28T16:49:10.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.121+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c107460 con 0x7f2c9c104f90 2024-02-28T16:49:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.122+0000 7f2c92a00700 1 -- 172.21.15.1:0/139630575 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2c8c0097a0 con 0x7f2c9c104f90 2024-02-28T16:49:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.123+0000 7f2c92a00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8006d7f0 0x7f2c8006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.123+0000 7f2c92a00700 1 -- 172.21.15.1:0/139630575 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2c8c08cd20 con 0x7f2c9c104f90 2024-02-28T16:49:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.123+0000 7f2c93e00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8006d7f0 0x7f2c8006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.124+0000 7f2c93e00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8006d7f0 0x7f2c8006fca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2c84004f90 tx=0x7f2c84009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.128+0000 7f2c92a00700 1 -- 172.21.15.1:0/139630575 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c8c05a000 con 0x7f2c9c104f90 2024-02-28T16:49:10.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.375+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c9c04bed0 con 0x7f2c8006d7f0 2024-02-28T16:49:10.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.377+0000 7f2c92a00700 1 -- 172.21.15.1:0/139630575 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2c9c04bed0 con 0x7f2c8006d7f0 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8006d7f0 msgr2=0x7f2c8006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8006d7f0 0x7f2c8006fca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2c84004f90 tx=0x7f2c84009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c104f90 msgr2=0x7f2c9c115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c104f90 0x7f2c9c115a50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c002a50 tx=0x7f2c8c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 shutdown_connections 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c8006d7f0 0x7f2c8006fca0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c9c104f90 0x7f2c9c115a50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 --2- 172.21.15.1:0/139630575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c9c105930 0x7f2c9c115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 >> 172.21.15.1:0/139630575 conn(0x7f2c9c0fab40 msgr2=0x7f2c9c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:10.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 shutdown_connections 2024-02-28T16:49:10.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.380+0000 7f2c9be00700 1 -- 172.21.15.1:0/139630575 wait complete. 2024-02-28T16:49:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.536+0000 7f9490a00700 1 -- 172.21.15.1:0/4176325394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 msgr2=0x7f948c100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.536+0000 7f9490a00700 1 --2- 172.21.15.1:0/4176325394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 0x7f948c100d10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9474004940 tx=0x7f9474009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.536+0000 7f9490a00700 1 -- 172.21.15.1:0/4176325394 shutdown_connections 2024-02-28T16:49:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.536+0000 7f9490a00700 1 --2- 172.21.15.1:0/4176325394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 0x7f948c100d10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.536+0000 7f9490a00700 1 --2- 172.21.15.1:0/4176325394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f948c0fbd50 0x7f948c0fc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.536+0000 7f9490a00700 1 -- 172.21.15.1:0/4176325394 >> 172.21.15.1:0/4176325394 conn(0x7f948c0f7a10 msgr2=0x7f948c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.536+0000 7f9490a00700 1 -- 172.21.15.1:0/4176325394 shutdown_connections 2024-02-28T16:49:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.536+0000 7f9490a00700 1 -- 172.21.15.1:0/4176325394 wait complete. 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.537+0000 7f9490a00700 1 Processor -- start 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.537+0000 7f9490a00700 1 -- start start 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9490a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f948c0fbd50 0x7f948c11a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9490a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 0x7f948c11a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9490a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c11ad70 con 0x7f948c0fc6f0 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9490a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c114080 con 0x7f948c0fbd50 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9481400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 0x7f948c11a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9481400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 0x7f948c11a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43300/0 (socket says 172.21.15.1:43300) 2024-02-28T16:49:10.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9481400700 1 -- 172.21.15.1:0/931001308 learned_addr learned my addr 172.21.15.1:0/931001308 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:10.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9481400700 1 -- 172.21.15.1:0/931001308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f948c0fbd50 msgr2=0x7f948c11a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9489e00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f948c0fbd50 0x7f948c11a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:10.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9481400700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f948c0fbd50 0x7f948c11a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9481400700 1 -- 172.21.15.1:0/931001308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9474005040 con 0x7f948c0fc6f0 2024-02-28T16:49:10.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.538+0000 7f9489e00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f948c0fbd50 0x7f948c11a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:49:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.539+0000 7f9481400700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 0x7f948c11a720 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9474009150 tx=0x7f9474009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.539+0000 7f9482a00700 1 -- 172.21.15.1:0/931001308 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f947400e040 con 0x7f948c0fc6f0 2024-02-28T16:49:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.539+0000 7f9482a00700 1 -- 172.21.15.1:0/931001308 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f947400bbf0 con 0x7f948c0fc6f0 2024-02-28T16:49:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.539+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948c114300 con 0x7f948c0fc6f0 2024-02-28T16:49:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.539+0000 7f9482a00700 1 -- 172.21.15.1:0/931001308 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f947401c630 con 0x7f948c0fc6f0 2024-02-28T16:49:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.539+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948c114840 con 0x7f948c0fc6f0 2024-02-28T16:49:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.540+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f948c0fe430 con 0x7f948c0fc6f0 2024-02-28T16:49:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.542+0000 7f9482a00700 1 -- 172.21.15.1:0/931001308 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f947400bd60 con 0x7f948c0fc6f0 2024-02-28T16:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.543+0000 7f9482a00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f947006d7f0 0x7f947006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.543+0000 7f9482a00700 1 -- 172.21.15.1:0/931001308 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f947408e390 con 0x7f948c0fc6f0 2024-02-28T16:49:10.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.543+0000 7f9489e00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f947006d7f0 0x7f947006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:10.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.544+0000 7f9489e00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f947006d7f0 0x7f947006fca0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f947c007fd0 tx=0x7f947c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:10.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.548+0000 7f9482a00700 1 -- 172.21.15.1:0/931001308 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f947405b6f0 con 0x7f948c0fc6f0 2024-02-28T16:49:10.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:10 smithi066 ceph-mon[85781]: pgmap v441: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.777+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f948c04bed0 con 0x7f947006d7f0 2024-02-28T16:49:10.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:10 smithi001 ceph-mon[112940]: pgmap v441: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:10.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.816+0000 7f9482a00700 1 -- 172.21.15.1:0/931001308 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f948c04bed0 con 0x7f947006d7f0 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (15m) 7m ago 29m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (12m) 7m ago 29m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (11m) 8m ago 28m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (14m) 7m ago 29m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (7m) 7m ago 22m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (7m) 7m ago 22m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (22m) 8m ago 22m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:49:10.808 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (22m) 8m ago 22m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (17m) 7m ago 31m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (16m) 8m ago 28m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (13m) 7m ago 31m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (13m) 8m ago 28m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (16m) 7m ago 29m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (16m) 8m ago 28m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (11m) 7m ago 27m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (10m) 7m ago 26m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (10m) 7m ago 26m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (9m) 8m ago 25m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (9m) 8m ago 24m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (8m) 8m ago 24m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:49:10.809 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (15m) 7m ago 28m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:49:10.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f947006d7f0 msgr2=0x7f947006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f947006d7f0 0x7f947006fca0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f947c007fd0 tx=0x7f947c009040 comp rx=0 tx=0).stop 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 msgr2=0x7f948c11a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 0x7f948c11a720 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9474009150 tx=0x7f9474009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 shutdown_connections 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f947006d7f0 0x7f947006fca0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f948c0fbd50 0x7f948c11a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 --2- 172.21.15.1:0/931001308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948c0fc6f0 0x7f948c11a720 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 >> 172.21.15.1:0/931001308 conn(0x7f948c0f7a10 msgr2=0x7f948c10d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 shutdown_connections 2024-02-28T16:49:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.819+0000 7f9490a00700 1 -- 172.21.15.1:0/931001308 wait complete. 2024-02-28T16:49:10.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.971+0000 7f854d400700 1 -- 172.21.15.1:0/3182775585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105b00 msgr2=0x7f85480ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.971+0000 7f854d400700 1 --2- 172.21.15.1:0/3182775585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105b00 0x7f85480ff170 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8534004940 tx=0x7f8534009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:10.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.971+0000 7f854d400700 1 -- 172.21.15.1:0/3182775585 shutdown_connections 2024-02-28T16:49:10.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.971+0000 7f854d400700 1 --2- 172.21.15.1:0/3182775585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105b00 0x7f85480ff170 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.971+0000 7f854d400700 1 --2- 172.21.15.1:0/3182775585 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8548105160 0x7f8548105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.971+0000 7f854d400700 1 -- 172.21.15.1:0/3182775585 >> 172.21.15.1:0/3182775585 conn(0x7f85480facb0 msgr2=0x7f85480fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:10.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.971+0000 7f854d400700 1 -- 172.21.15.1:0/3182775585 shutdown_connections 2024-02-28T16:49:10.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.971+0000 7f854d400700 1 -- 172.21.15.1:0/3182775585 wait complete. 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.972+0000 7f854d400700 1 Processor -- start 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.972+0000 7f854d400700 1 -- start start 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.972+0000 7f854d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105160 0x7f8548115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.972+0000 7f854d400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8548105b00 0x7f85481161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f854d400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8548116840 con 0x7f8548105160 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f854d400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854810fb50 con 0x7f8548105b00 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f8546800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105160 0x7f8548115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f8546800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105160 0x7f8548115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43312/0 (socket says 172.21.15.1:43312) 2024-02-28T16:49:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f8546800700 1 -- 172.21.15.1:0/2863815024 learned_addr learned my addr 172.21.15.1:0/2863815024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f8545e00700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8548105b00 0x7f85481161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f8546800700 1 -- 172.21.15.1:0/2863815024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8548105b00 msgr2=0x7f85481161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f8546800700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8548105b00 0x7f85481161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.973+0000 7f8546800700 1 -- 172.21.15.1:0/2863815024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8534005040 con 0x7f8548105160 2024-02-28T16:49:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.974+0000 7f8546800700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105160 0x7f8548115cb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8538002a50 tx=0x7f85380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.974+0000 7f853ea00700 1 -- 172.21.15.1:0/2863815024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8538007cc0 con 0x7f8548105160 2024-02-28T16:49:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.974+0000 7f853ea00700 1 -- 172.21.15.1:0/2863815024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f853800f040 con 0x7f8548105160 2024-02-28T16:49:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.974+0000 7f853ea00700 1 -- 172.21.15.1:0/2863815024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f853800db70 con 0x7f8548105160 2024-02-28T16:49:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.974+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854810fe30 con 0x7f8548105160 2024-02-28T16:49:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.974+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8548110370 con 0x7f8548105160 2024-02-28T16:49:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.975+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8548107630 con 0x7f8548105160 2024-02-28T16:49:10.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.976+0000 7f853ea00700 1 -- 172.21.15.1:0/2863815024 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f853800dd10 con 0x7f8548105160 2024-02-28T16:49:10.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.977+0000 7f853ea00700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f852c06d7f0 0x7f852c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:10.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.977+0000 7f853ea00700 1 -- 172.21.15.1:0/2863815024 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f853808c590 con 0x7f8548105160 2024-02-28T16:49:10.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.977+0000 7f8545e00700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f852c06d7f0 0x7f852c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.978+0000 7f8545e00700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f852c06d7f0 0x7f852c06fca0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8548111550 tx=0x7f8534009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:10.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:10.982+0000 7f853ea00700 1 -- 172.21.15.1:0/2863815024 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f85380598f0 con 0x7f8548105160 2024-02-28T16:49:11.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.291+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8548111290 con 0x7f8548105160 2024-02-28T16:49:11.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.292+0000 7f853ea00700 1 -- 172.21.15.1:0/2863815024 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f8538059480 con 0x7f8548105160 2024-02-28T16:49:11.283 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:49:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:49:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:49:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:49:11.283 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:49:11.284 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:49:11.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:49:11.285 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:49:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f852c06d7f0 msgr2=0x7f852c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f852c06d7f0 0x7f852c06fca0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8548111550 tx=0x7f8534009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105160 msgr2=0x7f8548115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105160 0x7f8548115cb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8538002a50 tx=0x7f85380046b0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 shutdown_connections 2024-02-28T16:49:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f852c06d7f0 0x7f852c06fca0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8548105160 0x7f8548115cb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 --2- 172.21.15.1:0/2863815024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8548105b00 0x7f85481161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 >> 172.21.15.1:0/2863815024 conn(0x7f85480facb0 msgr2=0x7f85480fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:11.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 shutdown_connections 2024-02-28T16:49:11.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.295+0000 7f854d400700 1 -- 172.21.15.1:0/2863815024 wait complete. 2024-02-28T16:49:11.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.448+0000 7f2e89000700 1 -- 172.21.15.1:0/200756224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 msgr2=0x7f2e840fef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:11.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.448+0000 7f2e89000700 1 --2- 172.21.15.1:0/200756224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 0x7f2e840fef80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2e70004940 tx=0x7f2e70009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:11.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.448+0000 7f2e89000700 1 -- 172.21.15.1:0/200756224 shutdown_connections 2024-02-28T16:49:11.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.448+0000 7f2e89000700 1 --2- 172.21.15.1:0/200756224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 0x7f2e840fef80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.448+0000 7f2e89000700 1 --2- 172.21.15.1:0/200756224 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e84104f70 0x7f2e84105340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.448+0000 7f2e89000700 1 -- 172.21.15.1:0/200756224 >> 172.21.15.1:0/200756224 conn(0x7f2e840faae0 msgr2=0x7f2e840fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.448+0000 7f2e89000700 1 -- 172.21.15.1:0/200756224 shutdown_connections 2024-02-28T16:49:11.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.448+0000 7f2e89000700 1 -- 172.21.15.1:0/200756224 wait complete. 2024-02-28T16:49:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.449+0000 7f2e89000700 1 Processor -- start 2024-02-28T16:49:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.449+0000 7f2e89000700 1 -- start start 2024-02-28T16:49:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.449+0000 7f2e89000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e84104f70 0x7f2e84115a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.449+0000 7f2e89000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 0x7f2e84115f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.449+0000 7f2e89000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e841165c0 con 0x7f2e84105910 2024-02-28T16:49:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.449+0000 7f2e89000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8410f8f0 con 0x7f2e84104f70 2024-02-28T16:49:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.449+0000 7f2e82a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 0x7f2e84115f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:11.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e82a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 0x7f2e84115f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43322/0 (socket says 172.21.15.1:43322) 2024-02-28T16:49:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e82a00700 1 -- 172.21.15.1:0/1636651076 learned_addr learned my addr 172.21.15.1:0/1636651076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e82a00700 1 -- 172.21.15.1:0/1636651076 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e84104f70 msgr2=0x7f2e84115a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e83400700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e84104f70 0x7f2e84115a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e82a00700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e84104f70 0x7f2e84115a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e82a00700 1 -- 172.21.15.1:0/1636651076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e70005040 con 0x7f2e84105910 2024-02-28T16:49:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e83400700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e84104f70 0x7f2e84115a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:49:11.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e82a00700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 0x7f2e84115f70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009030 tx=0x7f2e70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:11.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.450+0000 7f2e7b400700 1 -- 172.21.15.1:0/1636651076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e7000e040 con 0x7f2e84105910 2024-02-28T16:49:11.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.451+0000 7f2e7b400700 1 -- 172.21.15.1:0/1636651076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e7000bbf0 con 0x7f2e84105910 2024-02-28T16:49:11.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.451+0000 7f2e7b400700 1 -- 172.21.15.1:0/1636651076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e7001c630 con 0x7f2e84105910 2024-02-28T16:49:11.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.451+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8410fb70 con 0x7f2e84105910 2024-02-28T16:49:11.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.451+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e84110130 con 0x7f2e84105910 2024-02-28T16:49:11.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.452+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8405f5c0 con 0x7f2e84105910 2024-02-28T16:49:11.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.453+0000 7f2e7b400700 1 -- 172.21.15.1:0/1636651076 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2e7000bd60 con 0x7f2e84105910 2024-02-28T16:49:11.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.454+0000 7f2e7b400700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e6806d7f0 0x7f2e6806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.454+0000 7f2e7b400700 1 -- 172.21.15.1:0/1636651076 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2e7008e1f0 con 0x7f2e84105910 2024-02-28T16:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.454+0000 7f2e83400700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e6806d7f0 0x7f2e6806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:11.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.455+0000 7f2e83400700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e6806d7f0 0x7f2e6806fca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2e74007fd0 tx=0x7f2e74009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.458+0000 7f2e7b400700 1 -- 172.21.15.1:0/1636651076 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2e7005b5d0 con 0x7f2e84105910 2024-02-28T16:49:11.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.727+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2e8404bed0 con 0x7f2e84105910 2024-02-28T16:49:11.718 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:11 smithi001 ceph-mon[112940]: from='client.34838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:11.719 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:11 smithi001 ceph-mon[112940]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:11.719 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:49:11.719 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:11 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2863815024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:49:11.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.728+0000 7f2e7b400700 1 -- 172.21.15.1:0/1636651076 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f2e7005b160 con 0x7f2e84105910 2024-02-28T16:49:11.719 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:49:11.719 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:49:11.720 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:49:11.721 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:49:11.722 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:49:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.731+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e6806d7f0 msgr2=0x7f2e6806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.731+0000 7f2e89000700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e6806d7f0 0x7f2e6806fca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2e74007fd0 tx=0x7f2e74009040 comp rx=0 tx=0).stop 2024-02-28T16:49:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.731+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 msgr2=0x7f2e84115f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.731+0000 7f2e89000700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 0x7f2e84115f70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009030 tx=0x7f2e70009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.731+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 shutdown_connections 2024-02-28T16:49:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.731+0000 7f2e89000700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e6806d7f0 0x7f2e6806fca0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.731+0000 7f2e89000700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e84104f70 0x7f2e84115a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.731+0000 7f2e89000700 1 --2- 172.21.15.1:0/1636651076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e84105910 0x7f2e84115f70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.732+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 >> 172.21.15.1:0/1636651076 conn(0x7f2e840faae0 msgr2=0x7f2e840fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:11.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.732+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 shutdown_connections 2024-02-28T16:49:11.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.732+0000 7f2e89000700 1 -- 172.21.15.1:0/1636651076 wait complete. 2024-02-28T16:49:11.724 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:49:11.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:11 smithi066 ceph-mon[85781]: from='client.34838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:11.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:11 smithi066 ceph-mon[85781]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:11.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:49:11.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:11 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2863815024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:49:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.887+0000 7f95d8200700 1 -- 172.21.15.1:0/691351422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d0103dc0 msgr2=0x7f95d0104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:11.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.887+0000 7f95d8200700 1 --2- 172.21.15.1:0/691351422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d0103dc0 0x7f95d0104210 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f95bc004940 tx=0x7f95bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:11.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.887+0000 7f95d8200700 1 -- 172.21.15.1:0/691351422 shutdown_connections 2024-02-28T16:49:11.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.887+0000 7f95d8200700 1 --2- 172.21.15.1:0/691351422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d0103dc0 0x7f95d0104210 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.887+0000 7f95d8200700 1 --2- 172.21.15.1:0/691351422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95d01095f0 0x7f95d01099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.887+0000 7f95d8200700 1 -- 172.21.15.1:0/691351422 >> 172.21.15.1:0/691351422 conn(0x7f95d0072d50 msgr2=0x7f95d0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:11.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.887+0000 7f95d8200700 1 -- 172.21.15.1:0/691351422 shutdown_connections 2024-02-28T16:49:11.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.887+0000 7f95d8200700 1 -- 172.21.15.1:0/691351422 wait complete. 2024-02-28T16:49:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.888+0000 7f95d8200700 1 Processor -- start 2024-02-28T16:49:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.888+0000 7f95d8200700 1 -- start start 2024-02-28T16:49:11.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d8200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95d0103dc0 0x7f95d011a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d8200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d01095f0 0x7f95d011a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d8200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d011abd0 con 0x7f95d01095f0 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d8200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d0113ee0 con 0x7f95d0103dc0 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d01095f0 0x7f95d011a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d01095f0 0x7f95d011a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59466/0 (socket says 172.21.15.1:59466) 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d4e00700 1 -- 172.21.15.1:0/841448244 learned_addr learned my addr 172.21.15.1:0/841448244 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d5800700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95d0103dc0 0x7f95d011a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d4e00700 1 -- 172.21.15.1:0/841448244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95d0103dc0 msgr2=0x7f95d011a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d4e00700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95d0103dc0 0x7f95d011a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d4e00700 1 -- 172.21.15.1:0/841448244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc005040 con 0x7f95d01095f0 2024-02-28T16:49:11.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.889+0000 7f95d5800700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95d0103dc0 0x7f95d011a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:49:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.890+0000 7f95d4e00700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d01095f0 0x7f95d011a580 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009150 tx=0x7f95bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.890+0000 7f95cd600700 1 -- 172.21.15.1:0/841448244 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95bc00e040 con 0x7f95d01095f0 2024-02-28T16:49:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.890+0000 7f95cd600700 1 -- 172.21.15.1:0/841448244 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95bc00bbf0 con 0x7f95d01095f0 2024-02-28T16:49:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.890+0000 7f95cd600700 1 -- 172.21.15.1:0/841448244 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95bc01c630 con 0x7f95d01095f0 2024-02-28T16:49:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.890+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d0114160 con 0x7f95d01095f0 2024-02-28T16:49:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.890+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d01146a0 con 0x7f95d01095f0 2024-02-28T16:49:11.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.891+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d010b9c0 con 0x7f95d01095f0 2024-02-28T16:49:11.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.892+0000 7f95cd600700 1 -- 172.21.15.1:0/841448244 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f95bc00bd60 con 0x7f95d01095f0 2024-02-28T16:49:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.894+0000 7f95cd600700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f95b806d7f0 0x7f95b806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.894+0000 7f95cd600700 1 -- 172.21.15.1:0/841448244 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f95bc08e290 con 0x7f95d01095f0 2024-02-28T16:49:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.894+0000 7f95d5800700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f95b806d7f0 0x7f95b806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:11.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.895+0000 7f95d5800700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f95b806d7f0 0x7f95b806fca0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f95c4007fd0 tx=0x7f95c4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:11.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:11.898+0000 7f95cd600700 1 -- 172.21.15.1:0/841448244 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f95bc05b670 con 0x7f95d01095f0 2024-02-28T16:49:12.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.138+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f95d004bed0 con 0x7f95b806d7f0 2024-02-28T16:49:12.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.141+0000 7f95cd600700 1 -- 172.21.15.1:0/841448244 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f95d004bed0 con 0x7f95b806d7f0 2024-02-28T16:49:12.132 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:49:12.132 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:49:12.132 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:49:12.132 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:49:12.132 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:49:12.133 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:49:12.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f95b806d7f0 msgr2=0x7f95b806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:12.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f95b806d7f0 0x7f95b806fca0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f95c4007fd0 tx=0x7f95c4009040 comp rx=0 tx=0).stop 2024-02-28T16:49:12.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d01095f0 msgr2=0x7f95d011a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:12.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d01095f0 0x7f95d011a580 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009150 tx=0x7f95bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:12.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 shutdown_connections 2024-02-28T16:49:12.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f95b806d7f0 0x7f95b806fca0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f95d0103dc0 0x7f95d011a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 --2- 172.21.15.1:0/841448244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95d01095f0 0x7f95d011a580 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 >> 172.21.15.1:0/841448244 conn(0x7f95d0072d50 msgr2=0x7f95d0074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:12.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.144+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 shutdown_connections 2024-02-28T16:49:12.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.145+0000 7f95d8200700 1 -- 172.21.15.1:0/841448244 wait complete. 2024-02-28T16:49:12.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.289+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3571035716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d040702d0 msgr2=0x7f8d040706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:12.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.289+0000 7f8d0ba00700 1 --2- 172.21.15.1:0/3571035716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d040702d0 0x7f8d040706a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8004970 tx=0x7f8cf8009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:12.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.290+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3571035716 shutdown_connections 2024-02-28T16:49:12.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.290+0000 7f8d0ba00700 1 --2- 172.21.15.1:0/3571035716 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d04070be0 0x7f8d041092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.290+0000 7f8d0ba00700 1 --2- 172.21.15.1:0/3571035716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d040702d0 0x7f8d040706a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.290+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3571035716 >> 172.21.15.1:0/3571035716 conn(0x7f8d040f8af0 msgr2=0x7f8d040faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:12.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.290+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3571035716 shutdown_connections 2024-02-28T16:49:12.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.290+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3571035716 wait complete. 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.290+0000 7f8d0ba00700 1 Processor -- start 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.291+0000 7f8d0ba00700 1 -- start start 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.291+0000 7f8d0ba00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d040702d0 0x7f8d04115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.291+0000 7f8d0ba00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d04070be0 0x7f8d04116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.291+0000 7f8d0ba00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d04116780 con 0x7f8d04070be0 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.291+0000 7f8d0ba00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0410fad0 con 0x7f8d040702d0 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.291+0000 7f8d03e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d04070be0 0x7f8d04116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.291+0000 7f8d03e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d04070be0 0x7f8d04116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59482/0 (socket says 172.21.15.1:59482) 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.291+0000 7f8d03e00700 1 -- 172.21.15.1:0/3727451543 learned_addr learned my addr 172.21.15.1:0/3727451543 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d09000700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d040702d0 0x7f8d04115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d03e00700 1 -- 172.21.15.1:0/3727451543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d040702d0 msgr2=0x7f8d04115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d03e00700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d040702d0 0x7f8d04115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d03e00700 1 -- 172.21.15.1:0/3727451543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf8005040 con 0x7f8d04070be0 2024-02-28T16:49:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d09000700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d040702d0 0x7f8d04115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:49:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d03e00700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d04070be0 0x7f8d04116130 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000c830 tx=0x7f8cf0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d00c00700 1 -- 172.21.15.1:0/3727451543 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cf000a760 con 0x7f8d04070be0 2024-02-28T16:49:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d00c00700 1 -- 172.21.15.1:0/3727451543 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8cf000ad90 con 0x7f8d04070be0 2024-02-28T16:49:12.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0410fdb0 con 0x7f8d04070be0 2024-02-28T16:49:12.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.292+0000 7f8d00c00700 1 -- 172.21.15.1:0/3727451543 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cf00113a0 con 0x7f8d04070be0 2024-02-28T16:49:12.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.293+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d041102f0 con 0x7f8d04070be0 2024-02-28T16:49:12.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.294+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d04106970 con 0x7f8d04070be0 2024-02-28T16:49:12.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.295+0000 7f8d00c00700 1 -- 172.21.15.1:0/3727451543 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8cf0012400 con 0x7f8d04070be0 2024-02-28T16:49:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.296+0000 7f8d00c00700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8cec06d7f0 0x7f8cec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.296+0000 7f8d00c00700 1 -- 172.21.15.1:0/3727451543 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8cf008ce40 con 0x7f8d04070be0 2024-02-28T16:49:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.296+0000 7f8d09000700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8cec06d7f0 0x7f8cec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.297+0000 7f8d09000700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8cec06d7f0 0x7f8cec06fca0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8000c00 tx=0x7f8cf8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:12.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.302+0000 7f8d00c00700 1 -- 172.21.15.1:0/3727451543 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8cf00567b0 con 0x7f8d04070be0 2024-02-28T16:49:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.618+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8d0404bed0 con 0x7f8d04070be0 2024-02-28T16:49:12.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.619+0000 7f8d00c00700 1 -- 172.21.15.1:0/3727451543 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f8cf0059e70 con 0x7f8d04070be0 2024-02-28T16:49:12.610 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:49:12.610 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:49:12.610 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:49:12.611 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:49:12.611 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:49:12.611 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:49:12.611 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:49:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8cec06d7f0 msgr2=0x7f8cec06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8cec06d7f0 0x7f8cec06fca0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8000c00 tx=0x7f8cf8009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d04070be0 msgr2=0x7f8d04116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d04070be0 0x7f8d04116130 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000c830 tx=0x7f8cf0004470 comp rx=0 tx=0).stop 2024-02-28T16:49:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 shutdown_connections 2024-02-28T16:49:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8cec06d7f0 0x7f8cec06fca0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d040702d0 0x7f8d04115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 --2- 172.21.15.1:0/3727451543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d04070be0 0x7f8d04116130 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 >> 172.21.15.1:0/3727451543 conn(0x7f8d040f8af0 msgr2=0x7f8d04103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 shutdown_connections 2024-02-28T16:49:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:12.622+0000 7f8d0ba00700 1 -- 172.21.15.1:0/3727451543 wait complete. 2024-02-28T16:49:12.749 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:12 smithi066 ceph-mon[85781]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:12.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:12 smithi066 ceph-mon[85781]: pgmap v442: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:12.750 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:12 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1636651076' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:49:12.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:12 smithi001 ceph-mon[112940]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:12.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:12 smithi001 ceph-mon[112940]: pgmap v442: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:12.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:12 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1636651076' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:49:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:13 smithi001 ceph-mon[112940]: from='client.34858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:13.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:13 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3727451543' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:49:14.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:13 smithi066 ceph-mon[85781]: from='client.34858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:14.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:13 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3727451543' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:49:14.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:14 smithi001 ceph-mon[112940]: pgmap v443: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:14.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:14 smithi066 ceph-mon[85781]: pgmap v443: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:16.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:16 smithi001 ceph-mon[112940]: pgmap v444: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:16.999 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:16 smithi066 ceph-mon[85781]: pgmap v444: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:18.771 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:18 smithi001 ceph-mon[112940]: pgmap v445: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:19.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:18 smithi066 ceph-mon[85781]: pgmap v445: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:21.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:20 smithi066 ceph-mon[85781]: pgmap v446: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:21.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:20 smithi001 ceph-mon[112940]: pgmap v446: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:23.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:22 smithi066 ceph-mon[85781]: pgmap v447: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:23.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:22 smithi001 ceph-mon[112940]: pgmap v447: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:25.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:24 smithi066 ceph-mon[85781]: pgmap v448: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:25.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:24 smithi001 ceph-mon[112940]: pgmap v448: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:27.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:26 smithi066 ceph-mon[85781]: pgmap v449: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:27.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:49:27.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:26 smithi001 ceph-mon[112940]: pgmap v449: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:27.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:49:29.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:28 smithi066 ceph-mon[85781]: pgmap v450: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:29.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:28 smithi001 ceph-mon[112940]: pgmap v450: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:31.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:30 smithi066 ceph-mon[85781]: pgmap v451: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:31.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:30 smithi001 ceph-mon[112940]: pgmap v451: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:33.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:32 smithi066 ceph-mon[85781]: pgmap v452: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:33.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:32 smithi001 ceph-mon[112940]: pgmap v452: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:35.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:34 smithi066 ceph-mon[85781]: pgmap v453: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:35.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:34 smithi001 ceph-mon[112940]: pgmap v453: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:37.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:36 smithi066 ceph-mon[85781]: pgmap v454: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:37.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:36 smithi001 ceph-mon[112940]: pgmap v454: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:39.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:38 smithi066 ceph-mon[85781]: pgmap v455: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:39.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:38 smithi001 ceph-mon[112940]: pgmap v455: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:41.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:40 smithi066 ceph-mon[85781]: pgmap v456: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:41.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:40 smithi001 ceph-mon[112940]: pgmap v456: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:42.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:49:42.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:49:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.793+0000 7f1711200700 1 -- 172.21.15.1:0/3131185386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 msgr2=0x7f170c105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.793+0000 7f1711200700 1 --2- 172.21.15.1:0/3131185386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 0x7f170c105360 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f16fc004940 tx=0x7f16fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.793+0000 7f1711200700 1 -- 172.21.15.1:0/3131185386 shutdown_connections 2024-02-28T16:49:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.793+0000 7f1711200700 1 --2- 172.21.15.1:0/3131185386 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f170c105930 0x7f170c0fefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.793+0000 7f1711200700 1 --2- 172.21.15.1:0/3131185386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 0x7f170c105360 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.793+0000 7f1711200700 1 -- 172.21.15.1:0/3131185386 >> 172.21.15.1:0/3131185386 conn(0x7f170c0fab40 msgr2=0x7f170c0fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.793+0000 7f1711200700 1 -- 172.21.15.1:0/3131185386 shutdown_connections 2024-02-28T16:49:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.794+0000 7f1711200700 1 -- 172.21.15.1:0/3131185386 wait complete. 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.794+0000 7f1711200700 1 Processor -- start 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.795+0000 7f1711200700 1 -- start start 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.795+0000 7f1711200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 0x7f170c115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.795+0000 7f1711200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f170c105930 0x7f170c115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.795+0000 7f1711200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c1165a0 con 0x7f170c104f90 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.795+0000 7f1711200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c10f8d0 con 0x7f170c105930 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.795+0000 7f170b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 0x7f170c115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.796+0000 7f170b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 0x7f170c115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36448/0 (socket says 172.21.15.1:36448) 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.796+0000 7f170b400700 1 -- 172.21.15.1:0/3992262837 learned_addr learned my addr 172.21.15.1:0/3992262837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.796+0000 7f170aa00700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f170c105930 0x7f170c115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.796+0000 7f170b400700 1 -- 172.21.15.1:0/3992262837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f170c105930 msgr2=0x7f170c115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.796+0000 7f170b400700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f170c105930 0x7f170c115f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.796+0000 7f170b400700 1 -- 172.21.15.1:0/3992262837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc005040 con 0x7f170c104f90 2024-02-28T16:49:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.796+0000 7f170b400700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 0x7f170c115a10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009fd0 tx=0x7f16fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.797+0000 7f1703400700 1 -- 172.21.15.1:0/3992262837 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16fc00e040 con 0x7f170c104f90 2024-02-28T16:49:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.797+0000 7f1703400700 1 -- 172.21.15.1:0/3992262837 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16fc00bbf0 con 0x7f170c104f90 2024-02-28T16:49:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.797+0000 7f1703400700 1 -- 172.21.15.1:0/3992262837 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16fc01c3f0 con 0x7f170c104f90 2024-02-28T16:49:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.797+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c10fb50 con 0x7f170c104f90 2024-02-28T16:49:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.797+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c110110 con 0x7f170c104f90 2024-02-28T16:49:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.798+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c05f5c0 con 0x7f170c104f90 2024-02-28T16:49:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.799+0000 7f1703400700 1 -- 172.21.15.1:0/3992262837 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f16fc00bd60 con 0x7f170c104f90 2024-02-28T16:49:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.800+0000 7f1703400700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16f006d7f0 0x7f16f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.800+0000 7f1703400700 1 -- 172.21.15.1:0/3992262837 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f16fc08e1c0 con 0x7f170c104f90 2024-02-28T16:49:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.801+0000 7f170aa00700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16f006d7f0 0x7f16f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.801+0000 7f170aa00700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16f006d7f0 0x7f16f006fca0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f170c111240 tx=0x7f16f8009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:42.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:42.805+0000 7f1703400700 1 -- 172.21.15.1:0/3992262837 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f16fc05b5a0 con 0x7f170c104f90 2024-02-28T16:49:43.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:42 smithi066 ceph-mon[85781]: pgmap v457: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:43.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:42 smithi001 ceph-mon[112940]: pgmap v457: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:43.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.054+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f170c111060 con 0x7f16f006d7f0 2024-02-28T16:49:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.057+0000 7f1703400700 1 -- 172.21.15.1:0/3992262837 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f170c111060 con 0x7f16f006d7f0 2024-02-28T16:49:43.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.059+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16f006d7f0 msgr2=0x7f16f006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.059+0000 7f1711200700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16f006d7f0 0x7f16f006fca0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f170c111240 tx=0x7f16f8009ee0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 msgr2=0x7f170c115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 0x7f170c115a10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009fd0 tx=0x7f16fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 shutdown_connections 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16f006d7f0 0x7f16f006fca0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f170c104f90 0x7f170c115a10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 --2- 172.21.15.1:0/3992262837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f170c105930 0x7f170c115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 >> 172.21.15.1:0/3992262837 conn(0x7f170c0fab40 msgr2=0x7f170c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 shutdown_connections 2024-02-28T16:49:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.060+0000 7f1711200700 1 -- 172.21.15.1:0/3992262837 wait complete. 2024-02-28T16:49:43.068 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:49:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.226+0000 7f5d62a00700 1 -- 172.21.15.1:0/661845734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 msgr2=0x7f5d5c105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.226+0000 7f5d62a00700 1 --2- 172.21.15.1:0/661845734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 0x7f5d5c105540 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004940 tx=0x7f5d4c009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.227+0000 7f5d62a00700 1 -- 172.21.15.1:0/661845734 shutdown_connections 2024-02-28T16:49:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.227+0000 7f5d62a00700 1 --2- 172.21.15.1:0/661845734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d5c105b10 0x7f5d5c0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.227+0000 7f5d62a00700 1 --2- 172.21.15.1:0/661845734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 0x7f5d5c105540 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.227+0000 7f5d62a00700 1 -- 172.21.15.1:0/661845734 >> 172.21.15.1:0/661845734 conn(0x7f5d5c0face0 msgr2=0x7f5d5c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.227+0000 7f5d62a00700 1 -- 172.21.15.1:0/661845734 shutdown_connections 2024-02-28T16:49:43.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.227+0000 7f5d62a00700 1 -- 172.21.15.1:0/661845734 wait complete. 2024-02-28T16:49:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.228+0000 7f5d62a00700 1 Processor -- start 2024-02-28T16:49:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.228+0000 7f5d62a00700 1 -- start start 2024-02-28T16:49:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.228+0000 7f5d62a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 0x7f5d5c115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.228+0000 7f5d62a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d5c105b10 0x7f5d5c1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.228+0000 7f5d62a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1167f0 con 0x7f5d5c105170 2024-02-28T16:49:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.228+0000 7f5d62a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c10fb40 con 0x7f5d5c105b10 2024-02-28T16:49:43.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.228+0000 7f5d5be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 0x7f5d5c115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.229+0000 7f5d5be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 0x7f5d5c115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36460/0 (socket says 172.21.15.1:36460) 2024-02-28T16:49:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.229+0000 7f5d5be00700 1 -- 172.21.15.1:0/3646229002 learned_addr learned my addr 172.21.15.1:0/3646229002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:43.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.229+0000 7f5d5b400700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d5c105b10 0x7f5d5c1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.229+0000 7f5d5be00700 1 -- 172.21.15.1:0/3646229002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d5c105b10 msgr2=0x7f5d5c1161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.229+0000 7f5d5be00700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d5c105b10 0x7f5d5c1161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.229+0000 7f5d5be00700 1 -- 172.21.15.1:0/3646229002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c005040 con 0x7f5d5c105170 2024-02-28T16:49:43.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.229+0000 7f5d5be00700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 0x7f5d5c115c60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009150 tx=0x7f5d4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.230+0000 7f5d53e00700 1 -- 172.21.15.1:0/3646229002 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d4c00e040 con 0x7f5d5c105170 2024-02-28T16:49:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.230+0000 7f5d53e00700 1 -- 172.21.15.1:0/3646229002 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d4c00bbf0 con 0x7f5d5c105170 2024-02-28T16:49:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.230+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c10fdc0 con 0x7f5d5c105170 2024-02-28T16:49:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.230+0000 7f5d53e00700 1 -- 172.21.15.1:0/3646229002 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d4c01c630 con 0x7f5d5c105170 2024-02-28T16:49:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.230+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c110300 con 0x7f5d5c105170 2024-02-28T16:49:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.231+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c102480 con 0x7f5d5c105170 2024-02-28T16:49:43.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.232+0000 7f5d53e00700 1 -- 172.21.15.1:0/3646229002 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5d4c00bd60 con 0x7f5d5c105170 2024-02-28T16:49:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.233+0000 7f5d53e00700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d4006d7f0 0x7f5d4006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.233+0000 7f5d53e00700 1 -- 172.21.15.1:0/3646229002 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5d4c08e360 con 0x7f5d5c105170 2024-02-28T16:49:43.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.233+0000 7f5d5b400700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d4006d7f0 0x7f5d4006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.234+0000 7f5d5b400700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d4006d7f0 0x7f5d4006fca0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5d48007fd0 tx=0x7f5d48009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:43.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.238+0000 7f5d53e00700 1 -- 172.21.15.1:0/3646229002 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5d4c057b90 con 0x7f5d5c105170 2024-02-28T16:49:43.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.484+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d5c111280 con 0x7f5d4006d7f0 2024-02-28T16:49:43.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.487+0000 7f5d53e00700 1 -- 172.21.15.1:0/3646229002 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5d5c111280 con 0x7f5d4006d7f0 2024-02-28T16:49:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.490+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d4006d7f0 msgr2=0x7f5d4006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.490+0000 7f5d62a00700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d4006d7f0 0x7f5d4006fca0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5d48007fd0 tx=0x7f5d48009040 comp rx=0 tx=0).stop 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.490+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 msgr2=0x7f5d5c115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.490+0000 7f5d62a00700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 0x7f5d5c115c60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009150 tx=0x7f5d4c009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.491+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 shutdown_connections 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.491+0000 7f5d62a00700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d4006d7f0 0x7f5d4006fca0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.491+0000 7f5d62a00700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d5c105170 0x7f5d5c115c60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.491+0000 7f5d62a00700 1 --2- 172.21.15.1:0/3646229002 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d5c105b10 0x7f5d5c1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.491+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 >> 172.21.15.1:0/3646229002 conn(0x7f5d5c0face0 msgr2=0x7f5d5c0fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.491+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 shutdown_connections 2024-02-28T16:49:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.491+0000 7f5d62a00700 1 -- 172.21.15.1:0/3646229002 wait complete. 2024-02-28T16:49:43.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.650+0000 7fb46f400700 1 -- 172.21.15.1:0/3157540362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb468101e50 msgr2=0x7fb468102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.650+0000 7fb46f400700 1 --2- 172.21.15.1:0/3157540362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb468101e50 0x7fb468102220 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb45c004940 tx=0x7fb45c009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:43.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.650+0000 7fb46f400700 1 -- 172.21.15.1:0/3157540362 shutdown_connections 2024-02-28T16:49:43.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.650+0000 7fb46f400700 1 --2- 172.21.15.1:0/3157540362 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4681027f0 0x7fb4680fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.650+0000 7fb46f400700 1 --2- 172.21.15.1:0/3157540362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb468101e50 0x7fb468102220 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.650+0000 7fb46f400700 1 -- 172.21.15.1:0/3157540362 >> 172.21.15.1:0/3157540362 conn(0x7fb4680f79e0 msgr2=0x7fb4680f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.650+0000 7fb46f400700 1 -- 172.21.15.1:0/3157540362 shutdown_connections 2024-02-28T16:49:43.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.650+0000 7fb46f400700 1 -- 172.21.15.1:0/3157540362 wait complete. 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.651+0000 7fb46f400700 1 Processor -- start 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.651+0000 7fb46f400700 1 -- start start 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.652+0000 7fb46f400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb468101e50 0x7fb468115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.652+0000 7fb46f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4681027f0 0x7fb4681161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.652+0000 7fb46f400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb468116820 con 0x7fb4681027f0 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.652+0000 7fb46f400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46810fb50 con 0x7fb468101e50 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.652+0000 7fb467e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4681027f0 0x7fb4681161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.652+0000 7fb467e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4681027f0 0x7fb4681161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36490/0 (socket says 172.21.15.1:36490) 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.652+0000 7fb467e00700 1 -- 172.21.15.1:0/3907821378 learned_addr learned my addr 172.21.15.1:0/3907821378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:43.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.652+0000 7fb46ca00700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb468101e50 0x7fb468115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.653+0000 7fb467e00700 1 -- 172.21.15.1:0/3907821378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb468101e50 msgr2=0x7fb468115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.653+0000 7fb467e00700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb468101e50 0x7fb468115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.653+0000 7fb467e00700 1 -- 172.21.15.1:0/3907821378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c005040 con 0x7fb4681027f0 2024-02-28T16:49:43.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.653+0000 7fb467e00700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4681027f0 0x7fb4681161d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb45400c800 tx=0x7fb454004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.653+0000 7fb464c00700 1 -- 172.21.15.1:0/3907821378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb45400a760 con 0x7fb4681027f0 2024-02-28T16:49:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.653+0000 7fb464c00700 1 -- 172.21.15.1:0/3907821378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb45400ad90 con 0x7fb4681027f0 2024-02-28T16:49:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.653+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46810fe30 con 0x7fb4681027f0 2024-02-28T16:49:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.653+0000 7fb464c00700 1 -- 172.21.15.1:0/3907821378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4540113a0 con 0x7fb4681027f0 2024-02-28T16:49:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.654+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb468110370 con 0x7fb4681027f0 2024-02-28T16:49:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.655+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb468064b00 con 0x7fb4681027f0 2024-02-28T16:49:43.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.656+0000 7fb464c00700 1 -- 172.21.15.1:0/3907821378 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb454012400 con 0x7fb4681027f0 2024-02-28T16:49:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.657+0000 7fb464c00700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb45006d7e0 0x7fb45006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.657+0000 7fb464c00700 1 -- 172.21.15.1:0/3907821378 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb45408ce80 con 0x7fb4681027f0 2024-02-28T16:49:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.657+0000 7fb46ca00700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb45006d7e0 0x7fb45006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.658+0000 7fb46ca00700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb45006d7e0 0x7fb45006fc90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb45c004f90 tx=0x7fb45c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:43.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.662+0000 7fb464c00700 1 -- 172.21.15.1:0/3907821378 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb45405a260 con 0x7fb4681027f0 2024-02-28T16:49:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.889+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4681112c0 con 0x7fb45006d7e0 2024-02-28T16:49:43.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.909+0000 7fb464c00700 1 -- 172.21.15.1:0/3907821378 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb4681112c0 con 0x7fb45006d7e0 2024-02-28T16:49:43.900 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:49:43.900 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (15m) 7m ago 30m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:49:43.900 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (12m) 7m ago 30m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:49:43.900 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (12m) 8m ago 28m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:49:43.900 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (15m) 7m ago 29m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:49:43.900 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (8m) 7m ago 23m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:49:43.900 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (8m) 7m ago 22m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:49:43.900 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (22m) 8m ago 22m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (22m) 8m ago 22m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (18m) 7m ago 31m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (17m) 8m ago 28m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (14m) 7m ago 32m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (13m) 8m ago 28m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (16m) 7m ago 30m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (16m) 8m ago 28m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (11m) 7m ago 27m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (11m) 7m ago 27m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (10m) 7m ago 26m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (10m) 8m ago 26m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (9m) 8m ago 25m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (8m) 8m ago 24m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:49:43.901 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (16m) 7m ago 29m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:49:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.911+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb45006d7e0 msgr2=0x7fb45006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.911+0000 7fb46f400700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb45006d7e0 0x7fb45006fc90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb45c004f90 tx=0x7fb45c009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.911+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4681027f0 msgr2=0x7fb4681161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.911+0000 7fb46f400700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4681027f0 0x7fb4681161d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb45400c800 tx=0x7fb454004470 comp rx=0 tx=0).stop 2024-02-28T16:49:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.912+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 shutdown_connections 2024-02-28T16:49:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.912+0000 7fb46f400700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb45006d7e0 0x7fb45006fc90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.912+0000 7fb46f400700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb468101e50 0x7fb468115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.912+0000 7fb46f400700 1 --2- 172.21.15.1:0/3907821378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4681027f0 0x7fb4681161d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.912+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 >> 172.21.15.1:0/3907821378 conn(0x7fb4680f79e0 msgr2=0x7fb468065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.912+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 shutdown_connections 2024-02-28T16:49:43.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:43.912+0000 7fb46f400700 1 -- 172.21.15.1:0/3907821378 wait complete. 2024-02-28T16:49:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.063+0000 7fd278e00700 1 -- 172.21.15.1:0/956095399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 msgr2=0x7fd2740ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.063+0000 7fd278e00700 1 --2- 172.21.15.1:0/956095399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 0x7fd2740ff170 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd260004940 tx=0x7fd260009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.064+0000 7fd278e00700 1 -- 172.21.15.1:0/956095399 shutdown_connections 2024-02-28T16:49:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.064+0000 7fd278e00700 1 --2- 172.21.15.1:0/956095399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 0x7fd2740ff170 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.064+0000 7fd278e00700 1 --2- 172.21.15.1:0/956095399 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd274105160 0x7fd274105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.064+0000 7fd278e00700 1 -- 172.21.15.1:0/956095399 >> 172.21.15.1:0/956095399 conn(0x7fd2740facb0 msgr2=0x7fd2740fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.064+0000 7fd278e00700 1 -- 172.21.15.1:0/956095399 shutdown_connections 2024-02-28T16:49:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.064+0000 7fd278e00700 1 -- 172.21.15.1:0/956095399 wait complete. 2024-02-28T16:49:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.064+0000 7fd278e00700 1 Processor -- start 2024-02-28T16:49:44.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.065+0000 7fd278e00700 1 -- start start 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.065+0000 7fd278e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd274105160 0x7fd274115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.065+0000 7fd278e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 0x7fd2741161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.065+0000 7fd278e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd274116820 con 0x7fd274105b00 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.065+0000 7fd278e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27410fb50 con 0x7fd274105160 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.065+0000 7fd271400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 0x7fd2741161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.065+0000 7fd271400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 0x7fd2741161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36498/0 (socket says 172.21.15.1:36498) 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.065+0000 7fd271400700 1 -- 172.21.15.1:0/3123580742 learned_addr learned my addr 172.21.15.1:0/3123580742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd271e00700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd274105160 0x7fd274115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd271400700 1 -- 172.21.15.1:0/3123580742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd274105160 msgr2=0x7fd274115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd271400700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd274105160 0x7fd274115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd271400700 1 -- 172.21.15.1:0/3123580742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd260005040 con 0x7fd274105b00 2024-02-28T16:49:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd271400700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 0x7fd2741161d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd260009030 tx=0x7fd260009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd26a000700 1 -- 172.21.15.1:0/3123580742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd26000e040 con 0x7fd274105b00 2024-02-28T16:49:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd26a000700 1 -- 172.21.15.1:0/3123580742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd26000bbf0 con 0x7fd274105b00 2024-02-28T16:49:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27410fdd0 con 0x7fd274105b00 2024-02-28T16:49:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.066+0000 7fd26a000700 1 -- 172.21.15.1:0/3123580742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd26001c630 con 0x7fd274105b00 2024-02-28T16:49:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.067+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd274110310 con 0x7fd274105b00 2024-02-28T16:49:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.068+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd274107630 con 0x7fd274105b00 2024-02-28T16:49:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.069+0000 7fd26a000700 1 -- 172.21.15.1:0/3123580742 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd26000bd60 con 0x7fd274105b00 2024-02-28T16:49:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.069+0000 7fd26a000700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25806d7f0 0x7fd25806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.070+0000 7fd26a000700 1 -- 172.21.15.1:0/3123580742 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd26008e2c0 con 0x7fd274105b00 2024-02-28T16:49:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.070+0000 7fd271e00700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25806d7f0 0x7fd25806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.071+0000 7fd271e00700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25806d7f0 0x7fd25806fca0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd264004c40 tx=0x7fd264009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.075+0000 7fd26a000700 1 -- 172.21.15.1:0/3123580742 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd26005b620 con 0x7fd274105b00 2024-02-28T16:49:44.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.393+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd27404bed0 con 0x7fd274105b00 2024-02-28T16:49:44.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.394+0000 7fd26a000700 1 -- 172.21.15.1:0/3123580742 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fd26005b1b0 con 0x7fd274105b00 2024-02-28T16:49:44.385 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:49:44.385 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:49:44.386 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:49:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:49:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:49:44.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:49:44.387 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:49:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25806d7f0 msgr2=0x7fd25806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25806d7f0 0x7fd25806fca0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd264004c40 tx=0x7fd264009040 comp rx=0 tx=0).stop 2024-02-28T16:49:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 msgr2=0x7fd2741161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 0x7fd2741161d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd260009030 tx=0x7fd260009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 shutdown_connections 2024-02-28T16:49:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25806d7f0 0x7fd25806fca0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd274105160 0x7fd274115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 --2- 172.21.15.1:0/3123580742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd274105b00 0x7fd2741161d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.397+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 >> 172.21.15.1:0/3123580742 conn(0x7fd2740facb0 msgr2=0x7fd2740fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.398+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 shutdown_connections 2024-02-28T16:49:44.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.398+0000 7fd278e00700 1 -- 172.21.15.1:0/3123580742 wait complete. 2024-02-28T16:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.553+0000 7f6eade00700 1 -- 172.21.15.1:0/2943523362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105170 msgr2=0x7f6ea8105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.553+0000 7f6eade00700 1 --2- 172.21.15.1:0/2943523362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105170 0x7f6ea8105540 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6e98004940 tx=0x7f6e98009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.553+0000 7f6eade00700 1 -- 172.21.15.1:0/2943523362 shutdown_connections 2024-02-28T16:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.554+0000 7f6eade00700 1 --2- 172.21.15.1:0/2943523362 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ea8105b10 0x7f6ea80ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.554+0000 7f6eade00700 1 --2- 172.21.15.1:0/2943523362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105170 0x7f6ea8105540 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.554+0000 7f6eade00700 1 -- 172.21.15.1:0/2943523362 >> 172.21.15.1:0/2943523362 conn(0x7f6ea80face0 msgr2=0x7f6ea80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.554+0000 7f6eade00700 1 -- 172.21.15.1:0/2943523362 shutdown_connections 2024-02-28T16:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.554+0000 7f6eade00700 1 -- 172.21.15.1:0/2943523362 wait complete. 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.554+0000 7f6eade00700 1 Processor -- start 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.555+0000 7f6eade00700 1 -- start start 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.555+0000 7f6eade00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ea8105170 0x7f6ea8115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.555+0000 7f6eade00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105b10 0x7f6ea81161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.555+0000 7f6eade00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea81167f0 con 0x7f6ea8105b10 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.555+0000 7f6eade00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea810fb40 con 0x7f6ea8105170 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.555+0000 7f6ea6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105b10 0x7f6ea81161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.556+0000 7f6ea6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105b10 0x7f6ea81161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36518/0 (socket says 172.21.15.1:36518) 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.556+0000 7f6ea6000700 1 -- 172.21.15.1:0/1667658933 learned_addr learned my addr 172.21.15.1:0/1667658933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.556+0000 7f6ea6a00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ea8105170 0x7f6ea8115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.556+0000 7f6ea6000700 1 -- 172.21.15.1:0/1667658933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ea8105170 msgr2=0x7f6ea8115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.556+0000 7f6ea6000700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ea8105170 0x7f6ea8115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.556+0000 7f6ea6000700 1 -- 172.21.15.1:0/1667658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98005040 con 0x7f6ea8105b10 2024-02-28T16:49:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.556+0000 7f6ea6000700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105b10 0x7f6ea81161a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6e94002a50 tx=0x7f6e94004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.557+0000 7f6e9ea00700 1 -- 172.21.15.1:0/1667658933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e94007cc0 con 0x7f6ea8105b10 2024-02-28T16:49:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.557+0000 7f6e9ea00700 1 -- 172.21.15.1:0/1667658933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e9400f040 con 0x7f6ea8105b10 2024-02-28T16:49:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.557+0000 7f6e9ea00700 1 -- 172.21.15.1:0/1667658933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9400daf0 con 0x7f6ea8105b10 2024-02-28T16:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.557+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea810fe20 con 0x7f6ea8105b10 2024-02-28T16:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.557+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea8110360 con 0x7f6ea8105b10 2024-02-28T16:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.558+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea8102480 con 0x7f6ea8105b10 2024-02-28T16:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.559+0000 7f6e9ea00700 1 -- 172.21.15.1:0/1667658933 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6e94009810 con 0x7f6ea8105b10 2024-02-28T16:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.560+0000 7f6e9ea00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e8c06d7f0 0x7f6e8c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.560+0000 7f6e9ea00700 1 -- 172.21.15.1:0/1667658933 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6e9408cc30 con 0x7f6ea8105b10 2024-02-28T16:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.560+0000 7f6ea6a00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e8c06d7f0 0x7f6e8c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.561+0000 7f6ea6a00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e8c06d7f0 0x7f6e8c06fca0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b3b0 tx=0x7f6e98009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.564+0000 7f6e9ea00700 1 -- 172.21.15.1:0/1667658933 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e94056460 con 0x7f6ea8105b10 2024-02-28T16:49:44.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:44 smithi001 ceph-mon[112940]: pgmap v458: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:44.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:44 smithi001 ceph-mon[112940]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:44.772 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:44 smithi001 ceph-mon[112940]: from='client.34870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:44.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:44 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3123580742' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:49:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.834+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6ea804bed0 con 0x7f6ea8105b10 2024-02-28T16:49:44.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.835+0000 7f6e9ea00700 1 -- 172.21.15.1:0/1667658933 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f6e94059b20 con 0x7f6ea8105b10 2024-02-28T16:49:44.827 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:49:44.827 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:49:44.827 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:49:44.827 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:49:44.827 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:49:44.827 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:49:44.827 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:49:44.828 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:49:44.829 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:49:44.830 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:49:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.839+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e8c06d7f0 msgr2=0x7f6e8c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.839+0000 7f6eade00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e8c06d7f0 0x7f6e8c06fca0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800b3b0 tx=0x7f6e98009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.839+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105b10 msgr2=0x7f6ea81161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.839+0000 7f6eade00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105b10 0x7f6ea81161a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6e94002a50 tx=0x7f6e94004600 comp rx=0 tx=0).stop 2024-02-28T16:49:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.839+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 shutdown_connections 2024-02-28T16:49:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.840+0000 7f6eade00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e8c06d7f0 0x7f6e8c06fca0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.840+0000 7f6eade00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ea8105170 0x7f6ea8115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.840+0000 7f6eade00700 1 --2- 172.21.15.1:0/1667658933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ea8105b10 0x7f6ea81161a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.840+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 >> 172.21.15.1:0/1667658933 conn(0x7f6ea80face0 msgr2=0x7f6ea80fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.840+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 shutdown_connections 2024-02-28T16:49:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.840+0000 7f6eade00700 1 -- 172.21.15.1:0/1667658933 wait complete. 2024-02-28T16:49:44.833 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:49:44.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.983+0000 7f188a000700 1 -- 172.21.15.1:0/2318883308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 msgr2=0x7f1884070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.983+0000 7f188a000700 1 --2- 172.21.15.1:0/2318883308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 0x7f1884070ce0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1870004970 tx=0x7f1870009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:44.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.983+0000 7f188a000700 1 -- 172.21.15.1:0/2318883308 shutdown_connections 2024-02-28T16:49:44.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.983+0000 7f188a000700 1 --2- 172.21.15.1:0/2318883308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 0x7f1884070ce0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.983+0000 7f188a000700 1 --2- 172.21.15.1:0/2318883308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1884071f60 0x7f1884070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.983+0000 7f188a000700 1 -- 172.21.15.1:0/2318883308 >> 172.21.15.1:0/2318883308 conn(0x7f18840f8ae0 msgr2=0x7f18840faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:44.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.983+0000 7f188a000700 1 -- 172.21.15.1:0/2318883308 shutdown_connections 2024-02-28T16:49:44.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.984+0000 7f188a000700 1 -- 172.21.15.1:0/2318883308 wait complete. 2024-02-28T16:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.984+0000 7f188a000700 1 Processor -- start 2024-02-28T16:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.985+0000 7f188a000700 1 -- start start 2024-02-28T16:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.985+0000 7f188a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 0x7f188411a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.985+0000 7f188a000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1884071f60 0x7f188411a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.985+0000 7f188a000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188411aba0 con 0x7f1884070870 2024-02-28T16:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.985+0000 7f188a000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1884113eb0 con 0x7f1884071f60 2024-02-28T16:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.986+0000 7f1883400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 0x7f188411a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.986+0000 7f1883400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 0x7f188411a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36534/0 (socket says 172.21.15.1:36534) 2024-02-28T16:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.986+0000 7f1883400700 1 -- 172.21.15.1:0/1208633490 learned_addr learned my addr 172.21.15.1:0/1208633490 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.986+0000 7f1882a00700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1884071f60 0x7f188411a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.986+0000 7f1883400700 1 -- 172.21.15.1:0/1208633490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1884071f60 msgr2=0x7f188411a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.986+0000 7f1883400700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1884071f60 0x7f188411a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.986+0000 7f1883400700 1 -- 172.21.15.1:0/1208633490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870005040 con 0x7f1884070870 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.986+0000 7f1883400700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 0x7f188411a010 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f187400cb10 tx=0x7f1874007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.987+0000 7f187b400700 1 -- 172.21.15.1:0/1208633490 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1874004200 con 0x7f1884070870 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.987+0000 7f187b400700 1 -- 172.21.15.1:0/1208633490 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f187400acc0 con 0x7f1884070870 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.987+0000 7f187b400700 1 -- 172.21.15.1:0/1208633490 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1874003d90 con 0x7f1884070870 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.987+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1884114190 con 0x7f1884070870 2024-02-28T16:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.987+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18841146d0 con 0x7f1884070870 2024-02-28T16:49:44.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.988+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18840fe400 con 0x7f1884070870 2024-02-28T16:49:44.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.989+0000 7f187b400700 1 -- 172.21.15.1:0/1208633490 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1874003430 con 0x7f1884070870 2024-02-28T16:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.990+0000 7f187b400700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.990+0000 7f187b400700 1 -- 172.21.15.1:0/1208633490 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f187408d490 con 0x7f1884070870 2024-02-28T16:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.991+0000 7f1882a00700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.991+0000 7f1882a00700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1870000c00 tx=0x7f1870009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:44.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:44.995+0000 7f187b400700 1 -- 172.21.15.1:0/1208633490 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f187405a7f0 con 0x7f1884070870 2024-02-28T16:49:45.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:44 smithi066 ceph-mon[85781]: pgmap v458: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:45.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:44 smithi066 ceph-mon[85781]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:45.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:44 smithi066 ceph-mon[85781]: from='client.34870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:45.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:44 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3123580742' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:49:45.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.228+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f188404bf40 con 0x7f186806d7f0 2024-02-28T16:49:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.231+0000 7f187b400700 1 -- 172.21.15.1:0/1208633490 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f188404bf40 con 0x7f186806d7f0 2024-02-28T16:49:45.222 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:49:45.222 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:49:45.222 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:49:45.222 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:49:45.222 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:49:45.222 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:49:45.223 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:49:45.223 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:49:45.223 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:49:45.223 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:49:45.223 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:49:45.223 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:49:45.223 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:49:45.223 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:49:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 msgr2=0x7f186806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1870000c00 tx=0x7f1870009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 msgr2=0x7f188411a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 0x7f188411a010 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f187400cb10 tx=0x7f1874007f90 comp rx=0 tx=0).stop 2024-02-28T16:49:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 shutdown_connections 2024-02-28T16:49:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884070870 0x7f188411a010 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 --2- 172.21.15.1:0/1208633490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1884071f60 0x7f188411a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 >> 172.21.15.1:0/1208633490 conn(0x7f18840f8ae0 msgr2=0x7f18840fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.234+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 shutdown_connections 2024-02-28T16:49:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.235+0000 7f188a000700 1 -- 172.21.15.1:0/1208633490 wait complete. 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.391+0000 7f38ac400700 1 -- 172.21.15.1:0/3613566018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a41027e0 msgr2=0x7f38a40fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.391+0000 7f38ac400700 1 --2- 172.21.15.1:0/3613566018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a41027e0 0x7f38a40fbea0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3890004940 tx=0x7f3890009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.391+0000 7f38ac400700 1 -- 172.21.15.1:0/3613566018 shutdown_connections 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.391+0000 7f38ac400700 1 --2- 172.21.15.1:0/3613566018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a41027e0 0x7f38a40fbea0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.391+0000 7f38ac400700 1 --2- 172.21.15.1:0/3613566018 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38a4101e40 0x7f38a4102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.391+0000 7f38ac400700 1 -- 172.21.15.1:0/3613566018 >> 172.21.15.1:0/3613566018 conn(0x7f38a4071d90 msgr2=0x7f38a4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.392+0000 7f38ac400700 1 -- 172.21.15.1:0/3613566018 shutdown_connections 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.392+0000 7f38ac400700 1 -- 172.21.15.1:0/3613566018 wait complete. 2024-02-28T16:49:45.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.392+0000 7f38ac400700 1 Processor -- start 2024-02-28T16:49:45.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.392+0000 7f38ac400700 1 -- start start 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.393+0000 7f38ac400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a4101e40 0x7f38a4115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.393+0000 7f38ac400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38a41027e0 0x7f38a41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.393+0000 7f38ac400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a4116810 con 0x7f38a4101e40 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.393+0000 7f38ac400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a410fb40 con 0x7f38a41027e0 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.393+0000 7f38a9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a4101e40 0x7f38a4115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.393+0000 7f38a9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a4101e40 0x7f38a4115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36556/0 (socket says 172.21.15.1:36556) 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.393+0000 7f38a9a00700 1 -- 172.21.15.1:0/2130850740 learned_addr learned my addr 172.21.15.1:0/2130850740 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38a9000700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38a41027e0 0x7f38a41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:45.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38a9a00700 1 -- 172.21.15.1:0/2130850740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38a41027e0 msgr2=0x7f38a41161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38a9a00700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38a41027e0 0x7f38a41161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38a9a00700 1 -- 172.21.15.1:0/2130850740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3890005040 con 0x7f38a4101e40 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38a9a00700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a4101e40 0x7f38a4115c80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f389800c8c0 tx=0x7f38980044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38a1600700 1 -- 172.21.15.1:0/2130850740 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f389800a980 con 0x7f38a4101e40 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38a1600700 1 -- 172.21.15.1:0/2130850740 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f389800ce60 con 0x7f38a4101e40 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a410fe20 con 0x7f38a4101e40 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38a1600700 1 -- 172.21.15.1:0/2130850740 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38980115e0 con 0x7f38a4101e40 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.394+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a4110360 con 0x7f38a4101e40 2024-02-28T16:49:45.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.395+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a40738b0 con 0x7f38a4101e40 2024-02-28T16:49:45.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.396+0000 7f38a1600700 1 -- 172.21.15.1:0/2130850740 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3898012400 con 0x7f38a4101e40 2024-02-28T16:49:45.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.397+0000 7f38a1600700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f388c06d7f0 0x7f388c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:49:45.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.398+0000 7f38a1600700 1 -- 172.21.15.1:0/2130850740 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f389808cf70 con 0x7f38a4101e40 2024-02-28T16:49:45.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.398+0000 7f38a9000700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f388c06d7f0 0x7f388c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:49:45.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.398+0000 7f38a9000700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f388c06d7f0 0x7f388c06fca0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38a41114b0 tx=0x7f3890009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:49:45.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.402+0000 7f38a1600700 1 -- 172.21.15.1:0/2130850740 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f389805a350 con 0x7f38a4101e40 2024-02-28T16:49:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.706+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f38a41111f0 con 0x7f38a4101e40 2024-02-28T16:49:45.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.706+0000 7f38a1600700 1 -- 172.21.15.1:0/2130850740 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f38980126b0 con 0x7f38a4101e40 2024-02-28T16:49:45.697 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:49:45.697 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:49:45.697 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:49:45.698 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:49:45.698 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:49:45.698 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:49:45.698 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:49:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f388c06d7f0 msgr2=0x7f388c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f388c06d7f0 0x7f388c06fca0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38a41114b0 tx=0x7f3890009e30 comp rx=0 tx=0).stop 2024-02-28T16:49:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a4101e40 msgr2=0x7f38a4115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:49:45.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a4101e40 0x7f38a4115c80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f389800c8c0 tx=0x7f38980044c0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 shutdown_connections 2024-02-28T16:49:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f388c06d7f0 0x7f388c06fca0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38a4101e40 0x7f38a4115c80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 --2- 172.21.15.1:0/2130850740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38a41027e0 0x7f38a41161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:49:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 >> 172.21.15.1:0/2130850740 conn(0x7f38a4071d90 msgr2=0x7f38a4065740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:49:45.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 shutdown_connections 2024-02-28T16:49:45.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:49:45.709+0000 7f38ac400700 1 -- 172.21.15.1:0/2130850740 wait complete. 2024-02-28T16:49:46.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:45 smithi066 ceph-mon[85781]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:46.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:45 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1667658933' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:49:46.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:45 smithi001 ceph-mon[112940]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:46.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:45 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1667658933' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:49:47.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:46 smithi066 ceph-mon[85781]: pgmap v459: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:47.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:46 smithi066 ceph-mon[85781]: from='client.34886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:47.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:46 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2130850740' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:49:47.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:46 smithi001 ceph-mon[112940]: pgmap v459: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:47.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:46 smithi001 ceph-mon[112940]: from='client.34886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:49:47.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:46 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2130850740' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:49:49.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:48 smithi066 ceph-mon[85781]: pgmap v460: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:49.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:48 smithi001 ceph-mon[112940]: pgmap v460: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:51.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:50 smithi066 ceph-mon[85781]: pgmap v461: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:51.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:50 smithi001 ceph-mon[112940]: pgmap v461: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:53.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:52 smithi066 ceph-mon[85781]: pgmap v462: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:53.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:52 smithi001 ceph-mon[112940]: pgmap v462: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:55.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:54 smithi066 ceph-mon[85781]: pgmap v463: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:55.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:54 smithi001 ceph-mon[112940]: pgmap v463: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:57.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:56 smithi066 ceph-mon[85781]: pgmap v464: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:57.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:49:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:56 smithi001 ceph-mon[112940]: pgmap v464: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:57.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:49:59.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:49:58 smithi066 ceph-mon[85781]: pgmap v465: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:49:59.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:49:58 smithi001 ceph-mon[112940]: pgmap v465: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:00.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:49:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:50:00.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:49:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:50:00.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:49:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-02-28T16:50:00.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:49:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:50:00.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:49:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:50:00.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:49:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:50:00.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:49:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-02-28T16:50:01.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:00 smithi066 ceph-mon[85781]: pgmap v466: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:01.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:00 smithi066 ceph-mon[85781]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:50:01.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:00 smithi066 ceph-mon[85781]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:50:01.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:00 smithi066 ceph-mon[85781]: fs cephfs is degraded 2024-02-28T16:50:01.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:00 smithi066 ceph-mon[85781]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:50:01.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:00 smithi066 ceph-mon[85781]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:50:01.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:00 smithi066 ceph-mon[85781]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:50:01.001 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:00 smithi066 ceph-mon[85781]: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:50:01.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-mon[112940]: pgmap v466: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:01.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-mon[112940]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:50:01.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-mon[112940]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:50:01.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-mon[112940]: fs cephfs is degraded 2024-02-28T16:50:01.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-mon[112940]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:50:01.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-mon[112940]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:50:01.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-mon[112940]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:50:01.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:00 smithi001 ceph-mon[112940]: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:50:02.000 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:01 smithi066 ceph-mon[85781]: pgmap v467: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:02.022 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:01 smithi001 ceph-mon[112940]: pgmap v467: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:04.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:04 smithi066 ceph-mon[85781]: pgmap v468: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:04.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:04 smithi001 ceph-mon[112940]: pgmap v468: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:06.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:06 smithi066 ceph-mon[85781]: pgmap v469: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:06.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:06 smithi001 ceph-mon[112940]: pgmap v469: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:08.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:08 smithi066 ceph-mon[85781]: pgmap v470: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:08.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:08 smithi001 ceph-mon[112940]: pgmap v470: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:10.303 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:10 smithi066 ceph-mon[85781]: pgmap v471: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:10.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:10 smithi001 ceph-mon[112940]: pgmap v471: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:11.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:50:11.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:50:12.500 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:12 smithi066 ceph-mon[85781]: pgmap v472: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:12.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:12 smithi001 ceph-mon[112940]: pgmap v472: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:14.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:14 smithi066 ceph-mon[85781]: pgmap v473: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:14.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:14 smithi001 ceph-mon[112940]: pgmap v473: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.881+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2501080239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 msgr2=0x7f4c9c0faf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.881+0000 7f4ca0c00700 1 --2- 172.21.15.1:0/2501080239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 0x7f4c9c0faf80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4c88004940 tx=0x7f4c88009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.881+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2501080239 shutdown_connections 2024-02-28T16:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.881+0000 7f4ca0c00700 1 --2- 172.21.15.1:0/2501080239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 0x7f4c9c0faf80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.881+0000 7f4ca0c00700 1 --2- 172.21.15.1:0/2501080239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c9c100f70 0x7f4c9c101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.881+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2501080239 >> 172.21.15.1:0/2501080239 conn(0x7f4c9c0f6aa0 msgr2=0x7f4c9c0f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.882+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2501080239 shutdown_connections 2024-02-28T16:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.882+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2501080239 wait complete. 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.882+0000 7f4ca0c00700 1 Processor -- start 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.883+0000 7f4ca0c00700 1 -- start start 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.883+0000 7f4ca0c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c9c100f70 0x7f4c9c115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.883+0000 7f4ca0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 0x7f4c9c116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.883+0000 7f4ca0c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c116650 con 0x7f4c9c101910 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.883+0000 7f4ca0c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c10f980 con 0x7f4c9c100f70 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.884+0000 7f4c9aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 0x7f4c9c116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.884+0000 7f4c9aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 0x7f4c9c116000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35012/0 (socket says 172.21.15.1:35012) 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.884+0000 7f4c9aa00700 1 -- 172.21.15.1:0/2677377811 learned_addr learned my addr 172.21.15.1:0/2677377811 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.884+0000 7f4c9b400700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c9c100f70 0x7f4c9c115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.884+0000 7f4c9aa00700 1 -- 172.21.15.1:0/2677377811 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c9c100f70 msgr2=0x7f4c9c115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.884+0000 7f4c9aa00700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c9c100f70 0x7f4c9c115ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.884+0000 7f4c9aa00700 1 -- 172.21.15.1:0/2677377811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c88005040 con 0x7f4c9c101910 2024-02-28T16:50:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.885+0000 7f4c9aa00700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 0x7f4c9c116000 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009030 tx=0x7f4c88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.885+0000 7f4c93400700 1 -- 172.21.15.1:0/2677377811 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c8800e040 con 0x7f4c9c101910 2024-02-28T16:50:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.885+0000 7f4c93400700 1 -- 172.21.15.1:0/2677377811 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c88009400 con 0x7f4c9c101910 2024-02-28T16:50:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.885+0000 7f4c93400700 1 -- 172.21.15.1:0/2677377811 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c8801c3f0 con 0x7f4c9c101910 2024-02-28T16:50:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.885+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9c10fc00 con 0x7f4c9c101910 2024-02-28T16:50:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.885+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9c110140 con 0x7f4c9c101910 2024-02-28T16:50:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.886+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9c0ff9a0 con 0x7f4c9c101910 2024-02-28T16:50:15.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.888+0000 7f4c93400700 1 -- 172.21.15.1:0/2677377811 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4c8800bbf0 con 0x7f4c9c101910 2024-02-28T16:50:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.889+0000 7f4c93400700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c8006d7f0 0x7f4c8006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.889+0000 7f4c93400700 1 -- 172.21.15.1:0/2677377811 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4c8808e130 con 0x7f4c9c101910 2024-02-28T16:50:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.889+0000 7f4c9b400700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c8006d7f0 0x7f4c8006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.890+0000 7f4c9b400700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c8006d7f0 0x7f4c8006fca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c0041d0 tx=0x7f4c8c0043c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:15.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:15.894+0000 7f4c93400700 1 -- 172.21.15.1:0/2677377811 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c8805b5d0 con 0x7f4c9c101910 2024-02-28T16:50:16.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.139+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c9c1060d0 con 0x7f4c8006d7f0 2024-02-28T16:50:16.131 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:16 smithi001 ceph-mon[112940]: pgmap v474: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.141+0000 7f4c93400700 1 -- 172.21.15.1:0/2677377811 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4c9c1060d0 con 0x7f4c8006d7f0 2024-02-28T16:50:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c8006d7f0 msgr2=0x7f4c8006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c8006d7f0 0x7f4c8006fca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c0041d0 tx=0x7f4c8c0043c0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 msgr2=0x7f4c9c116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 0x7f4c9c116000 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009030 tx=0x7f4c88009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:16.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 shutdown_connections 2024-02-28T16:50:16.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c8006d7f0 0x7f4c8006fca0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c9c100f70 0x7f4c9c115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 --2- 172.21.15.1:0/2677377811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c9c101910 0x7f4c9c116000 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 >> 172.21.15.1:0/2677377811 conn(0x7f4c9c0f6aa0 msgr2=0x7f4c9c0fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:16.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 shutdown_connections 2024-02-28T16:50:16.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.145+0000 7f4ca0c00700 1 -- 172.21.15.1:0/2677377811 wait complete. 2024-02-28T16:50:16.154 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:50:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.308+0000 7ff9e5600700 1 -- 172.21.15.1:0/1069536672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 msgr2=0x7ff9e0070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.308+0000 7ff9e5600700 1 --2- 172.21.15.1:0/1069536672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 0x7ff9e0070ce0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8004970 tx=0x7ff9c8009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.308+0000 7ff9e5600700 1 -- 172.21.15.1:0/1069536672 shutdown_connections 2024-02-28T16:50:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.308+0000 7ff9e5600700 1 --2- 172.21.15.1:0/1069536672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 0x7ff9e0070ce0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.308+0000 7ff9e5600700 1 --2- 172.21.15.1:0/1069536672 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff9e0071f60 0x7ff9e0070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.308+0000 7ff9e5600700 1 -- 172.21.15.1:0/1069536672 >> 172.21.15.1:0/1069536672 conn(0x7ff9e00f8af0 msgr2=0x7ff9e00faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.308+0000 7ff9e5600700 1 -- 172.21.15.1:0/1069536672 shutdown_connections 2024-02-28T16:50:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.308+0000 7ff9e5600700 1 -- 172.21.15.1:0/1069536672 wait complete. 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.309+0000 7ff9e5600700 1 Processor -- start 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.309+0000 7ff9e5600700 1 -- start start 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.310+0000 7ff9e5600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 0x7ff9e011a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.310+0000 7ff9e5600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff9e0071f60 0x7ff9e011a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.310+0000 7ff9e5600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e011ac20 con 0x7ff9e0070870 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.310+0000 7ff9e5600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0113f30 con 0x7ff9e0071f60 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.310+0000 7ff9dea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 0x7ff9e011a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.310+0000 7ff9dea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 0x7ff9e011a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35032/0 (socket says 172.21.15.1:35032) 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.310+0000 7ff9dea00700 1 -- 172.21.15.1:0/2292415319 learned_addr learned my addr 172.21.15.1:0/2292415319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.310+0000 7ff9d7e00700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff9e0071f60 0x7ff9e011a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:16.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9dea00700 1 -- 172.21.15.1:0/2292415319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff9e0071f60 msgr2=0x7ff9e011a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:16.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9dea00700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff9e0071f60 0x7ff9e011a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9dea00700 1 -- 172.21.15.1:0/2292415319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8005040 con 0x7ff9e0070870 2024-02-28T16:50:16.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9dea00700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 0x7ff9e011a090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0002a50 tx=0x7ff9d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:16.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9d6a00700 1 -- 172.21.15.1:0/2292415319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d0007cc0 con 0x7ff9e0070870 2024-02-28T16:50:16.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9d6a00700 1 -- 172.21.15.1:0/2292415319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9d000f040 con 0x7ff9e0070870 2024-02-28T16:50:16.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e0114210 con 0x7ff9e0070870 2024-02-28T16:50:16.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9d6a00700 1 -- 172.21.15.1:0/2292415319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d0009640 con 0x7ff9e0070870 2024-02-28T16:50:16.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.311+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e0114750 con 0x7ff9e0070870 2024-02-28T16:50:16.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.313+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e00fe410 con 0x7ff9e0070870 2024-02-28T16:50:16.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.314+0000 7ff9d6a00700 1 -- 172.21.15.1:0/2292415319 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff9d00097a0 con 0x7ff9e0070870 2024-02-28T16:50:16.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.315+0000 7ff9d6a00700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff9c406d7f0 0x7ff9c406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:16.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.315+0000 7ff9d6a00700 1 -- 172.21.15.1:0/2292415319 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff9d008cc60 con 0x7ff9e0070870 2024-02-28T16:50:16.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.315+0000 7ff9d7e00700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff9c406d7f0 0x7ff9c406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:16.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.316+0000 7ff9d7e00700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff9c406d7f0 0x7ff9c406fca0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0115930 tx=0x7ff9c8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:16.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.321+0000 7ff9d6a00700 1 -- 172.21.15.1:0/2292415319 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff9d0059fc0 con 0x7ff9e0070870 2024-02-28T16:50:16.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:16 smithi066 ceph-mon[85781]: pgmap v474: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.564+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9e004bed0 con 0x7ff9c406d7f0 2024-02-28T16:50:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.566+0000 7ff9d6a00700 1 -- 172.21.15.1:0/2292415319 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff9e004bed0 con 0x7ff9c406d7f0 2024-02-28T16:50:16.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.569+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff9c406d7f0 msgr2=0x7ff9c406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.569+0000 7ff9e5600700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff9c406d7f0 0x7ff9c406fca0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0115930 tx=0x7ff9c8009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.569+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 msgr2=0x7ff9e011a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.569+0000 7ff9e5600700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 0x7ff9e011a090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0002a50 tx=0x7ff9d00046b0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.570+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 shutdown_connections 2024-02-28T16:50:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.570+0000 7ff9e5600700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff9c406d7f0 0x7ff9c406fca0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.570+0000 7ff9e5600700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9e0070870 0x7ff9e011a090 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.570+0000 7ff9e5600700 1 --2- 172.21.15.1:0/2292415319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff9e0071f60 0x7ff9e011a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.570+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 >> 172.21.15.1:0/2292415319 conn(0x7ff9e00f8af0 msgr2=0x7ff9e00fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.570+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 shutdown_connections 2024-02-28T16:50:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.570+0000 7ff9e5600700 1 -- 172.21.15.1:0/2292415319 wait complete. 2024-02-28T16:50:16.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.725+0000 7fa0e9a00700 1 -- 172.21.15.1:0/1735393718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4071f60 msgr2=0x7fa0e4070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:16.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.725+0000 7fa0e9a00700 1 --2- 172.21.15.1:0/1735393718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4071f60 0x7fa0e4070330 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4004970 tx=0x7fa0d4009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:16.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.725+0000 7fa0e9a00700 1 -- 172.21.15.1:0/1735393718 shutdown_connections 2024-02-28T16:50:16.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.725+0000 7fa0e9a00700 1 --2- 172.21.15.1:0/1735393718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0e4070870 0x7fa0e4070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.725+0000 7fa0e9a00700 1 --2- 172.21.15.1:0/1735393718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4071f60 0x7fa0e4070330 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.725+0000 7fa0e9a00700 1 -- 172.21.15.1:0/1735393718 >> 172.21.15.1:0/1735393718 conn(0x7fa0e40f8940 msgr2=0x7fa0e40fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:16.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.725+0000 7fa0e9a00700 1 -- 172.21.15.1:0/1735393718 shutdown_connections 2024-02-28T16:50:16.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.725+0000 7fa0e9a00700 1 -- 172.21.15.1:0/1735393718 wait complete. 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.726+0000 7fa0e9a00700 1 Processor -- start 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.726+0000 7fa0e9a00700 1 -- start start 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.726+0000 7fa0e9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4070870 0x7fa0e4119e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.726+0000 7fa0e9a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0e4071f60 0x7fa0e411a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.726+0000 7fa0e9a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e411a990 con 0x7fa0e4070870 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.726+0000 7fa0e9a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4113ce0 con 0x7fa0e4071f60 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.727+0000 7fa0e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4070870 0x7fa0e4119e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.727+0000 7fa0e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4070870 0x7fa0e4119e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35042/0 (socket says 172.21.15.1:35042) 2024-02-28T16:50:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.727+0000 7fa0e3e00700 1 -- 172.21.15.1:0/4109029549 learned_addr learned my addr 172.21.15.1:0/4109029549 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.727+0000 7fa0e3e00700 1 -- 172.21.15.1:0/4109029549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0e4071f60 msgr2=0x7fa0e411a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.727+0000 7fa0e3e00700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0e4071f60 0x7fa0e411a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.727+0000 7fa0e3e00700 1 -- 172.21.15.1:0/4109029549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d4005040 con 0x7fa0e4070870 2024-02-28T16:50:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.727+0000 7fa0e3e00700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4070870 0x7fa0e4119e00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009150 tx=0x7fa0d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.728+0000 7fa0dbe00700 1 -- 172.21.15.1:0/4109029549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d400e040 con 0x7fa0e4070870 2024-02-28T16:50:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.728+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e4113f60 con 0x7fa0e4070870 2024-02-28T16:50:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.728+0000 7fa0dbe00700 1 -- 172.21.15.1:0/4109029549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0d400bc10 con 0x7fa0e4070870 2024-02-28T16:50:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.728+0000 7fa0dbe00700 1 -- 172.21.15.1:0/4109029549 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d401c630 con 0x7fa0e4070870 2024-02-28T16:50:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.728+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e41144a0 con 0x7fa0e4070870 2024-02-28T16:50:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.729+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e40fe260 con 0x7fa0e4070870 2024-02-28T16:50:16.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.730+0000 7fa0dbe00700 1 -- 172.21.15.1:0/4109029549 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa0d400bd80 con 0x7fa0e4070870 2024-02-28T16:50:16.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.731+0000 7fa0dbe00700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa0c806d7f0 0x7fa0c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:16.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.732+0000 7fa0dbe00700 1 -- 172.21.15.1:0/4109029549 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa0d408e310 con 0x7fa0e4070870 2024-02-28T16:50:16.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.732+0000 7fa0e3400700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa0c806d7f0 0x7fa0c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:16.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.732+0000 7fa0e3400700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa0c806d7f0 0x7fa0c806fca0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa0e41156e0 tx=0x7fa0d0009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:16.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.736+0000 7fa0dbe00700 1 -- 172.21.15.1:0/4109029549 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa0d4057be0 con 0x7fa0e4070870 2024-02-28T16:50:16.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:16.974+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0e4115420 con 0x7fa0c806d7f0 2024-02-28T16:50:16.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.006+0000 7fa0dbe00700 1 -- 172.21.15.1:0/4109029549 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa0e4115420 con 0x7fa0c806d7f0 2024-02-28T16:50:16.998 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:50:16.998 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (16m) 8m ago 30m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:50:16.998 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (13m) 8m ago 30m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:50:16.998 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (13m) 9m ago 29m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:50:16.998 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (15m) 8m ago 30m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:50:16.998 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (9m) 8m ago 23m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:50:16.998 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (8m) 8m ago 23m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:50:16.998 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (23m) 9m ago 23m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (23m) 9m ago 23m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (18m) 8m ago 32m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (17m) 9m ago 29m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (14m) 8m ago 32m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (14m) 9m ago 29m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (17m) 8m ago 30m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (17m) 9m ago 29m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (12m) 8m ago 28m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (11m) 8m ago 27m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (11m) 8m ago 27m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (10m) 9m ago 26m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (10m) 9m ago 25m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (9m) 9m ago 25m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:50:16.999 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (16m) 8m ago 30m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:50:17.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa0c806d7f0 msgr2=0x7fa0c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa0c806d7f0 0x7fa0c806fca0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa0e41156e0 tx=0x7fa0d0009ce0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4070870 msgr2=0x7fa0e4119e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4070870 0x7fa0e4119e00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009150 tx=0x7fa0d4009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:17.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 shutdown_connections 2024-02-28T16:50:17.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa0c806d7f0 0x7fa0c806fca0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0e4070870 0x7fa0e4119e00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 --2- 172.21.15.1:0/4109029549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0e4071f60 0x7fa0e411a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.009+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 >> 172.21.15.1:0/4109029549 conn(0x7fa0e40f8940 msgr2=0x7fa0e40fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:17.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.010+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 shutdown_connections 2024-02-28T16:50:17.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.010+0000 7fa0e9a00700 1 -- 172.21.15.1:0/4109029549 wait complete. 2024-02-28T16:50:17.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.168+0000 7f813e000700 1 -- 172.21.15.1:0/2583370835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 msgr2=0x7f8138105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.168+0000 7f813e000700 1 --2- 172.21.15.1:0/2583370835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 0x7f8138105540 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8128004940 tx=0x7f8128009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:17.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.168+0000 7f813e000700 1 -- 172.21.15.1:0/2583370835 shutdown_connections 2024-02-28T16:50:17.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.168+0000 7f813e000700 1 --2- 172.21.15.1:0/2583370835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8138105b10 0x7f81380ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.168+0000 7f813e000700 1 --2- 172.21.15.1:0/2583370835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 0x7f8138105540 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.168+0000 7f813e000700 1 -- 172.21.15.1:0/2583370835 >> 172.21.15.1:0/2583370835 conn(0x7f81380face0 msgr2=0x7f81380fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:17.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.168+0000 7f813e000700 1 -- 172.21.15.1:0/2583370835 shutdown_connections 2024-02-28T16:50:17.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.168+0000 7f813e000700 1 -- 172.21.15.1:0/2583370835 wait complete. 2024-02-28T16:50:17.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.169+0000 7f813e000700 1 Processor -- start 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.169+0000 7f813e000700 1 -- start start 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.169+0000 7f813e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 0x7f8138115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.169+0000 7f813e000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8138105b10 0x7f81381161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.169+0000 7f813e000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381167f0 con 0x7f8138105170 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.169+0000 7f813e000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813810fb40 con 0x7f8138105b10 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.170+0000 7f8137400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 0x7f8138115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.170+0000 7f8137400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 0x7f8138115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35060/0 (socket says 172.21.15.1:35060) 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.170+0000 7f8137400700 1 -- 172.21.15.1:0/1756144433 learned_addr learned my addr 172.21.15.1:0/1756144433 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.170+0000 7f8136a00700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8138105b10 0x7f81381161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.170+0000 7f8137400700 1 -- 172.21.15.1:0/1756144433 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8138105b10 msgr2=0x7f81381161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.170+0000 7f8137400700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8138105b10 0x7f81381161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.170+0000 7f8137400700 1 -- 172.21.15.1:0/1756144433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8128005040 con 0x7f8138105170 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.170+0000 7f8136a00700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8138105b10 0x7f81381161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:50:17.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.171+0000 7f8137400700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 0x7f8138115c60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8128009fd0 tx=0x7f8128009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:17.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.171+0000 7f812f400700 1 -- 172.21.15.1:0/1756144433 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f812800e040 con 0x7f8138105170 2024-02-28T16:50:17.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.171+0000 7f812f400700 1 -- 172.21.15.1:0/1756144433 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f812800bbf0 con 0x7f8138105170 2024-02-28T16:50:17.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.171+0000 7f812f400700 1 -- 172.21.15.1:0/1756144433 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f812801c3f0 con 0x7f8138105170 2024-02-28T16:50:17.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.171+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813810fdc0 con 0x7f8138105170 2024-02-28T16:50:17.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.171+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8138110300 con 0x7f8138105170 2024-02-28T16:50:17.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.172+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8138102480 con 0x7f8138105170 2024-02-28T16:50:17.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.173+0000 7f812f400700 1 -- 172.21.15.1:0/1756144433 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f812800bd60 con 0x7f8138105170 2024-02-28T16:50:17.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.174+0000 7f812f400700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f811c06d7f0 0x7f811c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:17.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.174+0000 7f812f400700 1 -- 172.21.15.1:0/1756144433 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f812808e2b0 con 0x7f8138105170 2024-02-28T16:50:17.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.174+0000 7f8136a00700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f811c06d7f0 0x7f811c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:17.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.175+0000 7f8136a00700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f811c06d7f0 0x7f811c06fca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8124009fd0 tx=0x7f812400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:17.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.179+0000 7f812f400700 1 -- 172.21.15.1:0/1756144433 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8128057ae0 con 0x7f8138105170 2024-02-28T16:50:17.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:17 smithi001 ceph-mon[112940]: from='client.34894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:17.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:17 smithi001 ceph-mon[112940]: from='client.34898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:17.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.489+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f813804bf00 con 0x7f8138105170 2024-02-28T16:50:17.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.490+0000 7f812f400700 1 -- 172.21.15.1:0/1756144433 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f812805b1a0 con 0x7f8138105170 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:50:17.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:50:17.483 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:50:17.483 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:50:17.483 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:50:17.483 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:50:17.483 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:50:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f811c06d7f0 msgr2=0x7f811c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f811c06d7f0 0x7f811c06fca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8124009fd0 tx=0x7f812400b040 comp rx=0 tx=0).stop 2024-02-28T16:50:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 msgr2=0x7f8138115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 0x7f8138115c60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8128009fd0 tx=0x7f8128009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 shutdown_connections 2024-02-28T16:50:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f811c06d7f0 0x7f811c06fca0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8138105170 0x7f8138115c60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 --2- 172.21.15.1:0/1756144433 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8138105b10 0x7f81381161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 >> 172.21.15.1:0/1756144433 conn(0x7f81380face0 msgr2=0x7f81380fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:17.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 shutdown_connections 2024-02-28T16:50:17.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.493+0000 7f813e000700 1 -- 172.21.15.1:0/1756144433 wait complete. 2024-02-28T16:50:17.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:17 smithi066 ceph-mon[85781]: from='client.34894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:17.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:17 smithi066 ceph-mon[85781]: from='client.34898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.645+0000 7f9c5ea00700 1 -- 172.21.15.1:0/2898061472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 msgr2=0x7f9c580fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.645+0000 7f9c5ea00700 1 --2- 172.21.15.1:0/2898061472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 0x7f9c580fcda0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9c44004940 tx=0x7f9c44009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.646+0000 7f9c5ea00700 1 -- 172.21.15.1:0/2898061472 shutdown_connections 2024-02-28T16:50:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.646+0000 7f9c5ea00700 1 --2- 172.21.15.1:0/2898061472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 0x7f9c580fcda0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.646+0000 7f9c5ea00700 1 --2- 172.21.15.1:0/2898061472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c58102d90 0x7f9c58103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.646+0000 7f9c5ea00700 1 -- 172.21.15.1:0/2898061472 >> 172.21.15.1:0/2898061472 conn(0x7f9c580f8940 msgr2=0x7f9c580fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.646+0000 7f9c5ea00700 1 -- 172.21.15.1:0/2898061472 shutdown_connections 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.646+0000 7f9c5ea00700 1 -- 172.21.15.1:0/2898061472 wait complete. 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5ea00700 1 Processor -- start 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5ea00700 1 -- start start 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5ea00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c58102d90 0x7f9c58115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 0x7f9c58116030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5ea00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c58116710 con 0x7f9c58103730 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5ea00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5810f9b0 con 0x7f9c58102d90 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 0x7f9c58116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 0x7f9c58116030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35072/0 (socket says 172.21.15.1:35072) 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.647+0000 7f9c5cc00700 1 -- 172.21.15.1:0/1531365769 learned_addr learned my addr 172.21.15.1:0/1531365769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c5d600700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c58102d90 0x7f9c58115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:17.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c5cc00700 1 -- 172.21.15.1:0/1531365769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c58102d90 msgr2=0x7f9c58115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c5cc00700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c58102d90 0x7f9c58115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c5cc00700 1 -- 172.21.15.1:0/1531365769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c44005040 con 0x7f9c58103730 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c5d600700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c58102d90 0x7f9c58115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c5cc00700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 0x7f9c58116030 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009150 tx=0x7f9c44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c55600700 1 -- 172.21.15.1:0/1531365769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c4400e040 con 0x7f9c58103730 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5810fc30 con 0x7f9c58103730 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c55600700 1 -- 172.21.15.1:0/1531365769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c4400bbf0 con 0x7f9c58103730 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.648+0000 7f9c55600700 1 -- 172.21.15.1:0/1531365769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c4401c630 con 0x7f9c58103730 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.649+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c58110170 con 0x7f9c58103730 2024-02-28T16:50:17.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.650+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5805f5c0 con 0x7f9c58103730 2024-02-28T16:50:17.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.651+0000 7f9c55600700 1 -- 172.21.15.1:0/1531365769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9c4400bd60 con 0x7f9c58103730 2024-02-28T16:50:17.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.652+0000 7f9c55600700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c4006d7e0 0x7f9c4006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:17.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.652+0000 7f9c55600700 1 -- 172.21.15.1:0/1531365769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9c4408e2b0 con 0x7f9c58103730 2024-02-28T16:50:17.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.652+0000 7f9c5d600700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c4006d7e0 0x7f9c4006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:17.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.653+0000 7f9c5d600700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c4006d7e0 0x7f9c4006fc90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c006fd0 tx=0x7f9c4c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:17.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.656+0000 7f9c55600700 1 -- 172.21.15.1:0/1531365769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9c4405b6b0 con 0x7f9c58103730 2024-02-28T16:50:17.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.926+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9c5804bed0 con 0x7f9c58103730 2024-02-28T16:50:17.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.926+0000 7f9c55600700 1 -- 172.21.15.1:0/1531365769 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f9c4405b240 con 0x7f9c58103730 2024-02-28T16:50:17.918 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:50:17.918 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:50:17.918 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:50:17.918 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:50:17.918 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:50:17.919 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:50:17.920 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:50:17.921 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:50:17.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c4006d7e0 msgr2=0x7f9c4006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c4006d7e0 0x7f9c4006fc90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c006fd0 tx=0x7f9c4c008040 comp rx=0 tx=0).stop 2024-02-28T16:50:17.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 msgr2=0x7f9c58116030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:17.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 0x7f9c58116030 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009150 tx=0x7f9c44009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:17.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 shutdown_connections 2024-02-28T16:50:17.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9c4006d7e0 0x7f9c4006fc90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9c58102d90 0x7f9c58115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 --2- 172.21.15.1:0/1531365769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58103730 0x7f9c58116030 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:17.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 >> 172.21.15.1:0/1531365769 conn(0x7f9c580f8940 msgr2=0x7f9c580fa160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:17.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.929+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 shutdown_connections 2024-02-28T16:50:17.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:17.930+0000 7f9c5ea00700 1 -- 172.21.15.1:0/1531365769 wait complete. 2024-02-28T16:50:17.923 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:50:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.088+0000 7f9bd2800700 1 -- 172.21.15.1:0/301061284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 msgr2=0x7f9bcc1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.088+0000 7f9bd2800700 1 --2- 172.21.15.1:0/301061284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 0x7f9bcc1099c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc004940 tx=0x7f9bbc009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:18.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.088+0000 7f9bd2800700 1 -- 172.21.15.1:0/301061284 shutdown_connections 2024-02-28T16:50:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.088+0000 7f9bd2800700 1 --2- 172.21.15.1:0/301061284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9bcc103dc0 0x7f9bcc104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.088+0000 7f9bd2800700 1 --2- 172.21.15.1:0/301061284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 0x7f9bcc1099c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.088+0000 7f9bd2800700 1 -- 172.21.15.1:0/301061284 >> 172.21.15.1:0/301061284 conn(0x7f9bcc072d50 msgr2=0x7f9bcc075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.089+0000 7f9bd2800700 1 -- 172.21.15.1:0/301061284 shutdown_connections 2024-02-28T16:50:18.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.089+0000 7f9bd2800700 1 -- 172.21.15.1:0/301061284 wait complete. 2024-02-28T16:50:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.089+0000 7f9bd2800700 1 Processor -- start 2024-02-28T16:50:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.090+0000 7f9bd2800700 1 -- start start 2024-02-28T16:50:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.090+0000 7f9bd2800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9bcc103dc0 0x7f9bcc11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.090+0000 7f9bd2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 0x7f9bcc11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.090+0000 7f9bd2800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc11ac70 con 0x7f9bcc1095f0 2024-02-28T16:50:18.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.090+0000 7f9bd2800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc113f80 con 0x7f9bcc103dc0 2024-02-28T16:50:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.090+0000 7f9bc3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 0x7f9bcc11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.090+0000 7f9bc3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 0x7f9bcc11a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35090/0 (socket says 172.21.15.1:35090) 2024-02-28T16:50:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.090+0000 7f9bc3e00700 1 -- 172.21.15.1:0/4265030854 learned_addr learned my addr 172.21.15.1:0/4265030854 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bcb400700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9bcc103dc0 0x7f9bcc11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bc3e00700 1 -- 172.21.15.1:0/4265030854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9bcc103dc0 msgr2=0x7f9bcc11a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bc3e00700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9bcc103dc0 0x7f9bcc11a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bc3e00700 1 -- 172.21.15.1:0/4265030854 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc005040 con 0x7f9bcc1095f0 2024-02-28T16:50:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bc3e00700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 0x7f9bcc11a620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400c830 tx=0x7f9bb4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:18.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bc3400700 1 -- 172.21.15.1:0/4265030854 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb400a760 con 0x7f9bcc1095f0 2024-02-28T16:50:18.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bc3400700 1 -- 172.21.15.1:0/4265030854 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bb400ad90 con 0x7f9bcc1095f0 2024-02-28T16:50:18.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc114260 con 0x7f9bcc1095f0 2024-02-28T16:50:18.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.091+0000 7f9bc3400700 1 -- 172.21.15.1:0/4265030854 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb40113a0 con 0x7f9bcc1095f0 2024-02-28T16:50:18.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.092+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc1147a0 con 0x7f9bcc1095f0 2024-02-28T16:50:18.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.093+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc106850 con 0x7f9bcc1095f0 2024-02-28T16:50:18.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.094+0000 7f9bc3400700 1 -- 172.21.15.1:0/4265030854 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9bb4012400 con 0x7f9bcc1095f0 2024-02-28T16:50:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.095+0000 7f9bc3400700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9bb006d7f0 0x7f9bb006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.095+0000 7f9bc3400700 1 -- 172.21.15.1:0/4265030854 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9bb408cf00 con 0x7f9bcc1095f0 2024-02-28T16:50:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.095+0000 7f9bcb400700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9bb006d7f0 0x7f9bb006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.096+0000 7f9bcb400700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9bb006d7f0 0x7f9bb006fca0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00b3b0 tx=0x7f9bbc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.099+0000 7f9bc3400700 1 -- 172.21.15.1:0/4265030854 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9bb405a2e0 con 0x7f9bcc1095f0 2024-02-28T16:50:18.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:18 smithi001 ceph-mon[112940]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:18.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:18 smithi001 ceph-mon[112940]: pgmap v475: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:18.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:18 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1756144433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:50:18.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:18 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1531365769' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:50:18.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.350+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bcc04bf40 con 0x7f9bb006d7f0 2024-02-28T16:50:18.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.352+0000 7f9bc3400700 1 -- 172.21.15.1:0/4265030854 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9bcc04bf40 con 0x7f9bb006d7f0 2024-02-28T16:50:18.344 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:50:18.344 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:50:18.344 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:50:18.344 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:50:18.344 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:50:18.344 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:50:18.345 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:50:18.345 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:50:18.345 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:50:18.345 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:50:18.345 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:50:18.345 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:50:18.345 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:50:18.345 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:50:18.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.354+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9bb006d7f0 msgr2=0x7f9bb006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:18.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.354+0000 7f9bd2800700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9bb006d7f0 0x7f9bb006fca0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00b3b0 tx=0x7f9bbc009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:18.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.354+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 msgr2=0x7f9bcc11a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:18.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.354+0000 7f9bd2800700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 0x7f9bcc11a620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400c830 tx=0x7f9bb4004470 comp rx=0 tx=0).stop 2024-02-28T16:50:18.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.355+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 shutdown_connections 2024-02-28T16:50:18.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.355+0000 7f9bd2800700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9bb006d7f0 0x7f9bb006fca0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.355+0000 7f9bd2800700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9bcc103dc0 0x7f9bcc11a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.355+0000 7f9bd2800700 1 --2- 172.21.15.1:0/4265030854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bcc1095f0 0x7f9bcc11a620 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.355+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 >> 172.21.15.1:0/4265030854 conn(0x7f9bcc072d50 msgr2=0x7f9bcc074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:18.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.355+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 shutdown_connections 2024-02-28T16:50:18.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.355+0000 7f9bd2800700 1 -- 172.21.15.1:0/4265030854 wait complete. 2024-02-28T16:50:18.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:18 smithi066 ceph-mon[85781]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:18.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:18 smithi066 ceph-mon[85781]: pgmap v475: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:18.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:18 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1756144433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:50:18.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:18 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1531365769' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:50:18.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.510+0000 7ff766a00700 1 -- 172.21.15.1:0/2605554985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105b10 msgr2=0x7ff7600ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:18.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.510+0000 7ff766a00700 1 --2- 172.21.15.1:0/2605554985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105b10 0x7ff7600ff180 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff748004940 tx=0x7ff748009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:18.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.511+0000 7ff766a00700 1 -- 172.21.15.1:0/2605554985 shutdown_connections 2024-02-28T16:50:18.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.511+0000 7ff766a00700 1 --2- 172.21.15.1:0/2605554985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105b10 0x7ff7600ff180 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.511+0000 7ff766a00700 1 --2- 172.21.15.1:0/2605554985 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760105170 0x7ff760105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.511+0000 7ff766a00700 1 -- 172.21.15.1:0/2605554985 >> 172.21.15.1:0/2605554985 conn(0x7ff7600face0 msgr2=0x7ff7600fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:18.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.511+0000 7ff766a00700 1 -- 172.21.15.1:0/2605554985 shutdown_connections 2024-02-28T16:50:18.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.511+0000 7ff766a00700 1 -- 172.21.15.1:0/2605554985 wait complete. 2024-02-28T16:50:18.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff766a00700 1 Processor -- start 2024-02-28T16:50:18.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff766a00700 1 -- start start 2024-02-28T16:50:18.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff766a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105170 0x7ff760115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff766a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760105b10 0x7ff760116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff766a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff760116870 con 0x7ff760105170 2024-02-28T16:50:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff766a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76010fb80 con 0x7ff760105b10 2024-02-28T16:50:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff75fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105170 0x7ff760115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff75fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105170 0x7ff760115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35110/0 (socket says 172.21.15.1:35110) 2024-02-28T16:50:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.512+0000 7ff75fe00700 1 -- 172.21.15.1:0/867944247 learned_addr learned my addr 172.21.15.1:0/867944247 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.513+0000 7ff757400700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760105b10 0x7ff760116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.513+0000 7ff75fe00700 1 -- 172.21.15.1:0/867944247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760105b10 msgr2=0x7ff760116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.513+0000 7ff75fe00700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760105b10 0x7ff760116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.513+0000 7ff75fe00700 1 -- 172.21.15.1:0/867944247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748005040 con 0x7ff760105170 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.513+0000 7ff75fe00700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105170 0x7ff760115ce0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff750001d80 tx=0x7ff750009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.514+0000 7ff75cc00700 1 -- 172.21.15.1:0/867944247 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff750007d10 con 0x7ff760105170 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.514+0000 7ff75cc00700 1 -- 172.21.15.1:0/867944247 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff750007e70 con 0x7ff760105170 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.514+0000 7ff75cc00700 1 -- 172.21.15.1:0/867944247 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff750014640 con 0x7ff760105170 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.514+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76010fe60 con 0x7ff760105170 2024-02-28T16:50:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.514+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7601103a0 con 0x7ff760105170 2024-02-28T16:50:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.515+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff760107640 con 0x7ff760105170 2024-02-28T16:50:18.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.516+0000 7ff75cc00700 1 -- 172.21.15.1:0/867944247 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff75000e960 con 0x7ff760105170 2024-02-28T16:50:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.517+0000 7ff75cc00700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff74406d8b0 0x7ff74406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.517+0000 7ff75cc00700 1 -- 172.21.15.1:0/867944247 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff75008ccc0 con 0x7ff760105170 2024-02-28T16:50:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.517+0000 7ff757400700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff74406d8b0 0x7ff74406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.518+0000 7ff757400700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff74406d8b0 0x7ff74406fd60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff74800b3b0 tx=0x7ff748009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.522+0000 7ff75cc00700 1 -- 172.21.15.1:0/867944247 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff75005a0d0 con 0x7ff760105170 2024-02-28T16:50:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.830+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff7600fe900 con 0x7ff760105170 2024-02-28T16:50:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.831+0000 7ff75cc00700 1 -- 172.21.15.1:0/867944247 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ff7500191c0 con 0x7ff760105170 2024-02-28T16:50:18.823 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:50:18.823 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:50:18.824 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:50:18.824 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:50:18.824 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:50:18.824 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:50:18.824 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:50:18.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff74406d8b0 msgr2=0x7ff74406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff74406d8b0 0x7ff74406fd60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff74800b3b0 tx=0x7ff748009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105170 msgr2=0x7ff760115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105170 0x7ff760115ce0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff750001d80 tx=0x7ff750009ec0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 shutdown_connections 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff74406d8b0 0x7ff74406fd60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff760105170 0x7ff760115ce0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 --2- 172.21.15.1:0/867944247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff760105b10 0x7ff760116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 >> 172.21.15.1:0/867944247 conn(0x7ff7600face0 msgr2=0x7ff7600fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 shutdown_connections 2024-02-28T16:50:18.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:18.834+0000 7ff766a00700 1 -- 172.21.15.1:0/867944247 wait complete. 2024-02-28T16:50:19.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:19 smithi066 ceph-mon[85781]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:19.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:19 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/867944247' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:50:19.519 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:19 smithi001 ceph-mon[112940]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:19.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:19 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/867944247' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:50:20.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:20 smithi066 ceph-mon[85781]: pgmap v476: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:20.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:20 smithi001 ceph-mon[112940]: pgmap v476: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:22.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:22 smithi066 ceph-mon[85781]: pgmap v477: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:22.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:22 smithi001 ceph-mon[112940]: pgmap v477: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:24.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:24 smithi066 ceph-mon[85781]: pgmap v478: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:24.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:24 smithi001 ceph-mon[112940]: pgmap v478: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:26.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:26 smithi066 ceph-mon[85781]: pgmap v479: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:26.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:50:26.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:26 smithi001 ceph-mon[112940]: pgmap v479: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:26.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:50:28.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:28 smithi066 ceph-mon[85781]: pgmap v480: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:28.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:28 smithi001 ceph-mon[112940]: pgmap v480: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:30.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:30 smithi066 ceph-mon[85781]: pgmap v481: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:30.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:30 smithi001 ceph-mon[112940]: pgmap v481: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:32.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:32 smithi066 ceph-mon[85781]: pgmap v482: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:32.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:32 smithi001 ceph-mon[112940]: pgmap v482: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:34.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:34 smithi066 ceph-mon[85781]: pgmap v483: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:34.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:34 smithi001 ceph-mon[112940]: pgmap v483: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:36.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:36 smithi066 ceph-mon[85781]: pgmap v484: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:36.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:36 smithi001 ceph-mon[112940]: pgmap v484: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:38.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:38 smithi066 ceph-mon[85781]: pgmap v485: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:38.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:38 smithi001 ceph-mon[112940]: pgmap v485: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:40.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:40 smithi066 ceph-mon[85781]: pgmap v486: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:40.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:40 smithi001 ceph-mon[112940]: pgmap v486: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:41.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:50:41.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:50:42.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:42 smithi066 ceph-mon[85781]: pgmap v487: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:42.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:42 smithi001 ceph-mon[112940]: pgmap v487: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:44.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:44 smithi066 ceph-mon[85781]: pgmap v488: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:44.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:44 smithi001 ceph-mon[112940]: pgmap v488: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:46.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:46 smithi066 ceph-mon[85781]: pgmap v489: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:46.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:46 smithi001 ceph-mon[112940]: pgmap v489: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:48.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:48 smithi066 ceph-mon[85781]: pgmap v490: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:48.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:48 smithi001 ceph-mon[112940]: pgmap v490: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:49.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.007+0000 7f3c5d000700 1 -- 172.21.15.1:0/3528905341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 msgr2=0x7f3c58105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.007+0000 7f3c5d000700 1 --2- 172.21.15.1:0/3528905341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 0x7f3c58105540 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3c48004940 tx=0x7f3c48009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:49.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.007+0000 7f3c5d000700 1 -- 172.21.15.1:0/3528905341 shutdown_connections 2024-02-28T16:50:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.007+0000 7f3c5d000700 1 --2- 172.21.15.1:0/3528905341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c58105b10 0x7f3c580ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.007+0000 7f3c5d000700 1 --2- 172.21.15.1:0/3528905341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 0x7f3c58105540 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.007+0000 7f3c5d000700 1 -- 172.21.15.1:0/3528905341 >> 172.21.15.1:0/3528905341 conn(0x7f3c580face0 msgr2=0x7f3c580fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.007+0000 7f3c5d000700 1 -- 172.21.15.1:0/3528905341 shutdown_connections 2024-02-28T16:50:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.007+0000 7f3c5d000700 1 -- 172.21.15.1:0/3528905341 wait complete. 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.008+0000 7f3c5d000700 1 Processor -- start 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.008+0000 7f3c5d000700 1 -- start start 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.009+0000 7f3c5d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 0x7f3c58115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.009+0000 7f3c5d000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c58105b10 0x7f3c581161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.009+0000 7f3c5d000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581168a0 con 0x7f3c58105170 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.009+0000 7f3c5d000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5810fb40 con 0x7f3c58105b10 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.009+0000 7f3c55e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 0x7f3c58115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.009+0000 7f3c55e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 0x7f3c58115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40752/0 (socket says 172.21.15.1:40752) 2024-02-28T16:50:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.009+0000 7f3c55e00700 1 -- 172.21.15.1:0/3994448214 learned_addr learned my addr 172.21.15.1:0/3994448214 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:49.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c55e00700 1 -- 172.21.15.1:0/3994448214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c58105b10 msgr2=0x7f3c581161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c55400700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c58105b10 0x7f3c581161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c55e00700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c58105b10 0x7f3c581161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c55e00700 1 -- 172.21.15.1:0/3994448214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c48005040 con 0x7f3c58105170 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c55400700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c58105b10 0x7f3c581161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c55e00700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 0x7f3c58115c80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009100 tx=0x7f3c48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c4e000700 1 -- 172.21.15.1:0/3994448214 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4800e040 con 0x7f3c58105170 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c4e000700 1 -- 172.21.15.1:0/3994448214 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c48004290 con 0x7f3c58105170 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c4e000700 1 -- 172.21.15.1:0/3994448214 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4801c630 con 0x7f3c58105170 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.010+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5810fdc0 con 0x7f3c58105170 2024-02-28T16:50:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.011+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c58110300 con 0x7f3c58105170 2024-02-28T16:50:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.012+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c580fe900 con 0x7f3c58105170 2024-02-28T16:50:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.013+0000 7f3c4e000700 1 -- 172.21.15.1:0/3994448214 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3c4800bbf0 con 0x7f3c58105170 2024-02-28T16:50:49.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.014+0000 7f3c4e000700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c3c06d7e0 0x7f3c3c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.014+0000 7f3c4e000700 1 -- 172.21.15.1:0/3994448214 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3c4808e120 con 0x7f3c58105170 2024-02-28T16:50:49.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.015+0000 7f3c55400700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c3c06d7e0 0x7f3c3c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.015+0000 7f3c55400700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c3c06d7e0 0x7f3c3c06fc90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3c44006fd0 tx=0x7f3c44008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:49.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.020+0000 7f3c4e000700 1 -- 172.21.15.1:0/3994448214 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c4805b5c0 con 0x7f3c58105170 2024-02-28T16:50:49.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.260+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c58111300 con 0x7f3c3c06d7e0 2024-02-28T16:50:49.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.262+0000 7f3c4e000700 1 -- 172.21.15.1:0/3994448214 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3c58111300 con 0x7f3c3c06d7e0 2024-02-28T16:50:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.264+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c3c06d7e0 msgr2=0x7f3c3c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.264+0000 7f3c5d000700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c3c06d7e0 0x7f3c3c06fc90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3c44006fd0 tx=0x7f3c44008040 comp rx=0 tx=0).stop 2024-02-28T16:50:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.264+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 msgr2=0x7f3c58115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.265+0000 7f3c5d000700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 0x7f3c58115c80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009100 tx=0x7f3c48009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.265+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 shutdown_connections 2024-02-28T16:50:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.265+0000 7f3c5d000700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c3c06d7e0 0x7f3c3c06fc90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.265+0000 7f3c5d000700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c58105170 0x7f3c58115c80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.265+0000 7f3c5d000700 1 --2- 172.21.15.1:0/3994448214 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c58105b10 0x7f3c581161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.265+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 >> 172.21.15.1:0/3994448214 conn(0x7f3c580face0 msgr2=0x7f3c580fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.265+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 shutdown_connections 2024-02-28T16:50:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.265+0000 7f3c5d000700 1 -- 172.21.15.1:0/3994448214 wait complete. 2024-02-28T16:50:49.274 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:50:49.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.425+0000 7f8a25600700 1 -- 172.21.15.1:0/556836572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 msgr2=0x7f8a20104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.425+0000 7f8a25600700 1 --2- 172.21.15.1:0/556836572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 0x7f8a20104210 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8a08004940 tx=0x7f8a08009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.425+0000 7f8a25600700 1 -- 172.21.15.1:0/556836572 shutdown_connections 2024-02-28T16:50:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.425+0000 7f8a25600700 1 --2- 172.21.15.1:0/556836572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 0x7f8a20104210 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.425+0000 7f8a25600700 1 --2- 172.21.15.1:0/556836572 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a201095f0 0x7f8a201099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.425+0000 7f8a25600700 1 -- 172.21.15.1:0/556836572 >> 172.21.15.1:0/556836572 conn(0x7f8a20072d50 msgr2=0x7f8a20075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.426+0000 7f8a25600700 1 -- 172.21.15.1:0/556836572 shutdown_connections 2024-02-28T16:50:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.426+0000 7f8a25600700 1 -- 172.21.15.1:0/556836572 wait complete. 2024-02-28T16:50:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.427+0000 7f8a25600700 1 Processor -- start 2024-02-28T16:50:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.427+0000 7f8a25600700 1 -- start start 2024-02-28T16:50:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.427+0000 7f8a25600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 0x7f8a2011a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.427+0000 7f8a25600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a201095f0 0x7f8a2011a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.427+0000 7f8a25600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2011abd0 con 0x7f8a20103dc0 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.427+0000 7f8a25600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a20113ee0 con 0x7f8a201095f0 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.427+0000 7f8a1ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 0x7f8a2011a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.428+0000 7f8a1ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 0x7f8a2011a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40768/0 (socket says 172.21.15.1:40768) 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.428+0000 7f8a1ea00700 1 -- 172.21.15.1:0/3951336842 learned_addr learned my addr 172.21.15.1:0/3951336842 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.428+0000 7f8a17e00700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a201095f0 0x7f8a2011a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.428+0000 7f8a1ea00700 1 -- 172.21.15.1:0/3951336842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a201095f0 msgr2=0x7f8a2011a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.428+0000 7f8a1ea00700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a201095f0 0x7f8a2011a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.428+0000 7f8a1ea00700 1 -- 172.21.15.1:0/3951336842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a08005040 con 0x7f8a20103dc0 2024-02-28T16:50:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.428+0000 7f8a1ea00700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 0x7f8a2011a040 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8a10002a50 tx=0x7f8a100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.429+0000 7f8a16a00700 1 -- 172.21.15.1:0/3951336842 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a10007cc0 con 0x7f8a20103dc0 2024-02-28T16:50:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.429+0000 7f8a16a00700 1 -- 172.21.15.1:0/3951336842 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a1000f040 con 0x7f8a20103dc0 2024-02-28T16:50:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.429+0000 7f8a16a00700 1 -- 172.21.15.1:0/3951336842 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a10009640 con 0x7f8a20103dc0 2024-02-28T16:50:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.429+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a201141c0 con 0x7f8a20103dc0 2024-02-28T16:50:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.429+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a20114700 con 0x7f8a20103dc0 2024-02-28T16:50:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.430+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2010b9c0 con 0x7f8a20103dc0 2024-02-28T16:50:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.431+0000 7f8a16a00700 1 -- 172.21.15.1:0/3951336842 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8a100097a0 con 0x7f8a20103dc0 2024-02-28T16:50:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.432+0000 7f8a16a00700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a0406d7f0 0x7f8a0406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.432+0000 7f8a16a00700 1 -- 172.21.15.1:0/3951336842 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8a1008cbc0 con 0x7f8a20103dc0 2024-02-28T16:50:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.432+0000 7f8a17e00700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a0406d7f0 0x7f8a0406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.433+0000 7f8a17e00700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a0406d7f0 0x7f8a0406fca0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800b3b0 tx=0x7f8a08009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.437+0000 7f8a16a00700 1 -- 172.21.15.1:0/3951336842 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8a10059fc0 con 0x7f8a20103dc0 2024-02-28T16:50:49.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.687+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a20115620 con 0x7f8a0406d7f0 2024-02-28T16:50:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.689+0000 7f8a16a00700 1 -- 172.21.15.1:0/3951336842 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8a20115620 con 0x7f8a0406d7f0 2024-02-28T16:50:49.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a0406d7f0 msgr2=0x7f8a0406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a0406d7f0 0x7f8a0406fca0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800b3b0 tx=0x7f8a08009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:49.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 msgr2=0x7f8a2011a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 0x7f8a2011a040 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8a10002a50 tx=0x7f8a100046b0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 shutdown_connections 2024-02-28T16:50:49.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a0406d7f0 0x7f8a0406fca0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a20103dc0 0x7f8a2011a040 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 --2- 172.21.15.1:0/3951336842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8a201095f0 0x7f8a2011a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 >> 172.21.15.1:0/3951336842 conn(0x7f8a20072d50 msgr2=0x7f8a20074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:49.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 shutdown_connections 2024-02-28T16:50:49.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.692+0000 7f8a25600700 1 -- 172.21.15.1:0/3951336842 wait complete. 2024-02-28T16:50:49.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.852+0000 7f6186000700 1 -- 172.21.15.1:0/3060737982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6180103050 msgr2=0x7f6180103420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.852+0000 7f6186000700 1 --2- 172.21.15.1:0/3060737982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6180103050 0x7f6180103420 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6170004940 tx=0x7f6170009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.852+0000 7f6186000700 1 -- 172.21.15.1:0/3060737982 shutdown_connections 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.852+0000 7f6186000700 1 --2- 172.21.15.1:0/3060737982 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61800fb5b0 0x7f61800fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.852+0000 7f6186000700 1 --2- 172.21.15.1:0/3060737982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6180103050 0x7f6180103420 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.852+0000 7f6186000700 1 -- 172.21.15.1:0/3060737982 >> 172.21.15.1:0/3060737982 conn(0x7f6180071560 msgr2=0x7f6180071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.853+0000 7f6186000700 1 -- 172.21.15.1:0/3060737982 shutdown_connections 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.853+0000 7f6186000700 1 -- 172.21.15.1:0/3060737982 wait complete. 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.853+0000 7f6186000700 1 Processor -- start 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.854+0000 7f6186000700 1 -- start start 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.854+0000 7f6186000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800fb5b0 0x7f6180115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.854+0000 7f6186000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6180103050 0x7f6180116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.854+0000 7f6186000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6180116970 con 0x7f61800fb5b0 2024-02-28T16:50:49.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.854+0000 7f6186000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618010fbf0 con 0x7f6180103050 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.854+0000 7f617f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800fb5b0 0x7f6180115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.854+0000 7f617f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800fb5b0 0x7f6180115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40788/0 (socket says 172.21.15.1:40788) 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.854+0000 7f617f400700 1 -- 172.21.15.1:0/3194666670 learned_addr learned my addr 172.21.15.1:0/3194666670 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f617f400700 1 -- 172.21.15.1:0/3194666670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6180103050 msgr2=0x7f6180116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f6177e00700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6180103050 0x7f6180116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f617f400700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6180103050 0x7f6180116290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f617f400700 1 -- 172.21.15.1:0/3194666670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6170005040 con 0x7f61800fb5b0 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f617f400700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800fb5b0 0x7f6180115d50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6170009150 tx=0x7f6170009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:49.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f6177400700 1 -- 172.21.15.1:0/3194666670 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617000e040 con 0x7f61800fb5b0 2024-02-28T16:50:49.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f6177400700 1 -- 172.21.15.1:0/3194666670 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f617000bbf0 con 0x7f61800fb5b0 2024-02-28T16:50:49.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618010fe10 con 0x7f61800fb5b0 2024-02-28T16:50:49.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f6177400700 1 -- 172.21.15.1:0/3194666670 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617001c630 con 0x7f61800fb5b0 2024-02-28T16:50:49.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.855+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6180110350 con 0x7f61800fb5b0 2024-02-28T16:50:49.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.856+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6180073080 con 0x7f61800fb5b0 2024-02-28T16:50:49.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.857+0000 7f6177400700 1 -- 172.21.15.1:0/3194666670 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f617000bd60 con 0x7f61800fb5b0 2024-02-28T16:50:49.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.858+0000 7f6177400700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f616006d7f0 0x7f616006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:49.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.858+0000 7f6177400700 1 -- 172.21.15.1:0/3194666670 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f617008e2f0 con 0x7f61800fb5b0 2024-02-28T16:50:49.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.859+0000 7f6177e00700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f616006d7f0 0x7f616006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:49.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.859+0000 7f6177e00700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f616006d7f0 0x7f616006fca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6168006fd0 tx=0x7f6168008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:49.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:49.863+0000 7f6177400700 1 -- 172.21.15.1:0/3194666670 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f617005b6f0 con 0x7f61800fb5b0 2024-02-28T16:50:50.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.099+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6180111330 con 0x7f616006d7f0 2024-02-28T16:50:50.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.129+0000 7f6177400700 1 -- 172.21.15.1:0/3194666670 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6180111330 con 0x7f616006d7f0 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (16m) 9m ago 31m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (13m) 9m ago 31m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (13m) 9m ago 30m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (16m) 9m ago 30m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (9m) 9m ago 24m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (9m) 9m ago 23m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (23m) 9m ago 23m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:50:50.123 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (24m) 9m ago 24m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (19m) 9m ago 32m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (18m) 9m ago 29m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (15m) 9m ago 33m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (14m) 9m ago 29m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (18m) 9m ago 31m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (17m) 9m ago 29m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (13m) 9m ago 28m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (12m) 9m ago 28m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (11m) 9m ago 27m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (11m) 9m ago 27m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (10m) 9m ago 26m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (9m) 9m ago 25m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:50:50.124 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (17m) 9m ago 30m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:50:50.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.132+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f616006d7f0 msgr2=0x7f616006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:50.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.132+0000 7f6186000700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f616006d7f0 0x7f616006fca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6168006fd0 tx=0x7f6168008040 comp rx=0 tx=0).stop 2024-02-28T16:50:50.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.132+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800fb5b0 msgr2=0x7f6180115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:50.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.132+0000 7f6186000700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800fb5b0 0x7f6180115d50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6170009150 tx=0x7f6170009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:50.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.133+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 shutdown_connections 2024-02-28T16:50:50.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.133+0000 7f6186000700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f616006d7f0 0x7f616006fca0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.133+0000 7f6186000700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800fb5b0 0x7f6180115d50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.133+0000 7f6186000700 1 --2- 172.21.15.1:0/3194666670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6180103050 0x7f6180116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.133+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 >> 172.21.15.1:0/3194666670 conn(0x7f6180071560 msgr2=0x7f61800fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:50.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.133+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 shutdown_connections 2024-02-28T16:50:50.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.133+0000 7f6186000700 1 -- 172.21.15.1:0/3194666670 wait complete. 2024-02-28T16:50:50.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.291+0000 7f7df9800700 1 -- 172.21.15.1:0/2570037345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fb5e0 msgr2=0x7f7df40fb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:50.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.291+0000 7f7df9800700 1 --2- 172.21.15.1:0/2570037345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fb5e0 0x7f7df40fb9b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7de4004970 tx=0x7f7de4009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:50.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.291+0000 7f7df9800700 1 -- 172.21.15.1:0/2570037345 shutdown_connections 2024-02-28T16:50:50.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.291+0000 7f7df9800700 1 --2- 172.21.15.1:0/2570037345 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7df40fbef0 0x7f7df410d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.291+0000 7f7df9800700 1 --2- 172.21.15.1:0/2570037345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fb5e0 0x7f7df40fb9b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.291+0000 7f7df9800700 1 -- 172.21.15.1:0/2570037345 >> 172.21.15.1:0/2570037345 conn(0x7f7df4072270 msgr2=0x7f7df4072670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:50.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.292+0000 7f7df9800700 1 -- 172.21.15.1:0/2570037345 shutdown_connections 2024-02-28T16:50:50.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.292+0000 7f7df9800700 1 -- 172.21.15.1:0/2570037345 wait complete. 2024-02-28T16:50:50.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.292+0000 7f7df9800700 1 Processor -- start 2024-02-28T16:50:50.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.292+0000 7f7df9800700 1 -- start start 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.293+0000 7f7df9800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7df40fb5e0 0x7f7df4119dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.293+0000 7f7df9800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fbef0 0x7f7df411a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.293+0000 7f7df9800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df411a950 con 0x7f7df40fbef0 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.293+0000 7f7df9800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df4113c60 con 0x7f7df40fb5e0 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.293+0000 7f7df3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fbef0 0x7f7df411a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.293+0000 7f7df3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fbef0 0x7f7df411a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40814/0 (socket says 172.21.15.1:40814) 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.293+0000 7f7df3400700 1 -- 172.21.15.1:0/3314680229 learned_addr learned my addr 172.21.15.1:0/3314680229 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7df3e00700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7df40fb5e0 0x7f7df4119dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7df3400700 1 -- 172.21.15.1:0/3314680229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7df40fb5e0 msgr2=0x7f7df4119dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7df3400700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7df40fb5e0 0x7f7df4119dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7df3400700 1 -- 172.21.15.1:0/3314680229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de4005040 con 0x7f7df40fbef0 2024-02-28T16:50:50.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7df3400700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fbef0 0x7f7df411a300 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7de0002a50 tx=0x7f7de00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:50.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7debe00700 1 -- 172.21.15.1:0/3314680229 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7de0007cc0 con 0x7f7df40fbef0 2024-02-28T16:50:50.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7debe00700 1 -- 172.21.15.1:0/3314680229 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7de000f040 con 0x7f7df40fbef0 2024-02-28T16:50:50.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df4113f40 con 0x7f7df40fbef0 2024-02-28T16:50:50.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7debe00700 1 -- 172.21.15.1:0/3314680229 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7de0009640 con 0x7f7df40fbef0 2024-02-28T16:50:50.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.294+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df4114480 con 0x7f7df40fbef0 2024-02-28T16:50:50.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.295+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df405f5c0 con 0x7f7df40fbef0 2024-02-28T16:50:50.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.297+0000 7f7debe00700 1 -- 172.21.15.1:0/3314680229 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7de00097a0 con 0x7f7df40fbef0 2024-02-28T16:50:50.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.298+0000 7f7debe00700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7dd806d7f0 0x7f7dd806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:50.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.298+0000 7f7debe00700 1 -- 172.21.15.1:0/3314680229 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7de008cc60 con 0x7f7df40fbef0 2024-02-28T16:50:50.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.298+0000 7f7df3e00700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7dd806d7f0 0x7f7dd806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:50.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.299+0000 7f7df3e00700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7dd806d7f0 0x7f7dd806fca0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7de4000c00 tx=0x7f7de4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:50.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.303+0000 7f7debe00700 1 -- 172.21.15.1:0/3314680229 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7de0059fc0 con 0x7f7df40fbef0 2024-02-28T16:50:50.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:50 smithi066 ceph-mon[85781]: pgmap v491: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:50.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:50 smithi066 ceph-mon[85781]: from='client.34922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:50.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:50 smithi001 ceph-mon[112940]: pgmap v491: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:50.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:50 smithi001 ceph-mon[112940]: from='client.34922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:50.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.619+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7df4115440 con 0x7f7df40fbef0 2024-02-28T16:50:50.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.620+0000 7f7debe00700 1 -- 172.21.15.1:0/3314680229 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f7de0059b50 con 0x7f7df40fbef0 2024-02-28T16:50:50.613 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:50:50.613 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:50:50.613 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:50:50.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:50:50.615 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:50:50.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7dd806d7f0 msgr2=0x7f7dd806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:50.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7dd806d7f0 0x7f7dd806fca0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7de4000c00 tx=0x7f7de4009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:50.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fbef0 msgr2=0x7f7df411a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:50.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fbef0 0x7f7df411a300 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7de0002a50 tx=0x7f7de00046b0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 shutdown_connections 2024-02-28T16:50:50.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7dd806d7f0 0x7f7dd806fca0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7df40fb5e0 0x7f7df4119dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 --2- 172.21.15.1:0/3314680229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7df40fbef0 0x7f7df411a300 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 >> 172.21.15.1:0/3314680229 conn(0x7f7df4072270 msgr2=0x7f7df40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:50.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 shutdown_connections 2024-02-28T16:50:50.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.623+0000 7f7df9800700 1 -- 172.21.15.1:0/3314680229 wait complete. 2024-02-28T16:50:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.767+0000 7fbadb000700 1 -- 172.21.15.1:0/1929862205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 msgr2=0x7fbad40fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.767+0000 7fbadb000700 1 --2- 172.21.15.1:0/1929862205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 0x7fbad40fbea0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbac0004940 tx=0x7fbac0009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.767+0000 7fbadb000700 1 -- 172.21.15.1:0/1929862205 shutdown_connections 2024-02-28T16:50:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.767+0000 7fbadb000700 1 --2- 172.21.15.1:0/1929862205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 0x7fbad40fbea0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.767+0000 7fbadb000700 1 --2- 172.21.15.1:0/1929862205 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbad4101e40 0x7fbad4102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.767+0000 7fbadb000700 1 -- 172.21.15.1:0/1929862205 >> 172.21.15.1:0/1929862205 conn(0x7fbad4071d90 msgr2=0x7fbad4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.767+0000 7fbadb000700 1 -- 172.21.15.1:0/1929862205 shutdown_connections 2024-02-28T16:50:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.767+0000 7fbadb000700 1 -- 172.21.15.1:0/1929862205 wait complete. 2024-02-28T16:50:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.768+0000 7fbadb000700 1 Processor -- start 2024-02-28T16:50:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.768+0000 7fbadb000700 1 -- start start 2024-02-28T16:50:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.769+0000 7fbadb000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbad4101e40 0x7fbad4115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.769+0000 7fbadb000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 0x7fbad41161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.769+0000 7fbadb000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad4116830 con 0x7fbad41027e0 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.769+0000 7fbadb000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad410fb40 con 0x7fbad4101e40 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.769+0000 7fbad3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 0x7fbad41161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.769+0000 7fbad3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 0x7fbad41161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40838/0 (socket says 172.21.15.1:40838) 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.769+0000 7fbad3400700 1 -- 172.21.15.1:0/1958086355 learned_addr learned my addr 172.21.15.1:0/1958086355 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbad3e00700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbad4101e40 0x7fbad4115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbad3400700 1 -- 172.21.15.1:0/1958086355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbad4101e40 msgr2=0x7fbad4115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbad3400700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbad4101e40 0x7fbad4115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbad3400700 1 -- 172.21.15.1:0/1958086355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0005040 con 0x7fbad41027e0 2024-02-28T16:50:50.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbad3400700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 0x7fbad41161e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009100 tx=0x7fbac0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:50.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbacbe00700 1 -- 172.21.15.1:0/1958086355 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbac000e040 con 0x7fbad41027e0 2024-02-28T16:50:50.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbacbe00700 1 -- 172.21.15.1:0/1958086355 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbac0004290 con 0x7fbad41027e0 2024-02-28T16:50:50.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad410fdc0 con 0x7fbad41027e0 2024-02-28T16:50:50.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.770+0000 7fbacbe00700 1 -- 172.21.15.1:0/1958086355 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbac001c3f0 con 0x7fbad41027e0 2024-02-28T16:50:50.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.771+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad4110300 con 0x7fbad41027e0 2024-02-28T16:50:50.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.771+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad40738b0 con 0x7fbad41027e0 2024-02-28T16:50:50.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.773+0000 7fbacbe00700 1 -- 172.21.15.1:0/1958086355 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbac000bbf0 con 0x7fbad41027e0 2024-02-28T16:50:50.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.774+0000 7fbacbe00700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbab806d7f0 0x7fbab806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:50.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.774+0000 7fbacbe00700 1 -- 172.21.15.1:0/1958086355 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbac008e1d0 con 0x7fbad41027e0 2024-02-28T16:50:50.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.774+0000 7fbad3e00700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbab806d7f0 0x7fbab806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:50.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.775+0000 7fbad3e00700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbab806d7f0 0x7fbab806fca0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbac40041d0 tx=0x7fbac4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:50.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:50.778+0000 7fbacbe00700 1 -- 172.21.15.1:0/1958086355 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbac005b5b0 con 0x7fbad41027e0 2024-02-28T16:50:51.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.040+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbad4111280 con 0x7fbad41027e0 2024-02-28T16:50:51.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.041+0000 7fbacbe00700 1 -- 172.21.15.1:0/1958086355 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fbac005b140 con 0x7fbad41027e0 2024-02-28T16:50:51.035 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:50:51.035 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:50:51.036 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:50:51.036 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:50:51.036 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:50:51.036 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:50:51.036 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:50:51.037 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:50:51.037 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:50:51.037 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:50:51.037 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:50:51.037 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:50:51.037 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:50:51.038 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:50:51.038 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:50:51.038 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:50:51.038 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:50:51.038 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:50:51.038 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:50:51.039 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:50:51.039 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:50:51.039 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:50:51.039 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:50:51.039 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:50:51.039 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:50:51.040 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:50:51.040 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:50:51.040 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:50:51.040 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:50:51.040 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:50:51.041 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:50:51.041 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:50:51.041 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:50:51.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:50:51.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:50:51.041 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:50:51.042 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:50:51.042 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:50:51.042 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:50:51.042 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:50:51.042 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:50:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbab806d7f0 msgr2=0x7fbab806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbab806d7f0 0x7fbab806fca0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbac40041d0 tx=0x7fbac4009040 comp rx=0 tx=0).stop 2024-02-28T16:50:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 msgr2=0x7fbad41161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 0x7fbad41161e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009100 tx=0x7fbac0009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 shutdown_connections 2024-02-28T16:50:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbab806d7f0 0x7fbab806fca0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbad4101e40 0x7fbad4115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 --2- 172.21.15.1:0/1958086355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbad41027e0 0x7fbad41161e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 >> 172.21.15.1:0/1958086355 conn(0x7fbad4071d90 msgr2=0x7fbad41089b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.044+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 shutdown_connections 2024-02-28T16:50:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.045+0000 7fbadb000700 1 -- 172.21.15.1:0/1958086355 wait complete. 2024-02-28T16:50:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:50:51.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.189+0000 7f59cbe00700 1 -- 172.21.15.1:0/875073836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 msgr2=0x7f59cc105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.189+0000 7f59cbe00700 1 --2- 172.21.15.1:0/875073836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 0x7f59cc105350 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f59bc004940 tx=0x7f59bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:51.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.189+0000 7f59cbe00700 1 -- 172.21.15.1:0/875073836 shutdown_connections 2024-02-28T16:50:51.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.189+0000 7f59cbe00700 1 --2- 172.21.15.1:0/875073836 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc105920 0x7f59cc0fef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.190+0000 7f59cbe00700 1 --2- 172.21.15.1:0/875073836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 0x7f59cc105350 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.190+0000 7f59cbe00700 1 -- 172.21.15.1:0/875073836 >> 172.21.15.1:0/875073836 conn(0x7f59cc0fab10 msgr2=0x7f59cc0fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:51.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.190+0000 7f59cbe00700 1 -- 172.21.15.1:0/875073836 shutdown_connections 2024-02-28T16:50:51.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.190+0000 7f59cbe00700 1 -- 172.21.15.1:0/875073836 wait complete. 2024-02-28T16:50:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.191+0000 7f59cbe00700 1 Processor -- start 2024-02-28T16:50:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.191+0000 7f59cbe00700 1 -- start start 2024-02-28T16:50:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.191+0000 7f59cbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 0x7f59cc115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.191+0000 7f59cbe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc105920 0x7f59cc115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.191+0000 7f59cbe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc1165b0 con 0x7f59cc104f80 2024-02-28T16:50:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.191+0000 7f59cbe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc10f8e0 con 0x7f59cc105920 2024-02-28T16:50:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.192+0000 7f59caa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 0x7f59cc115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.192+0000 7f59caa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 0x7f59cc115a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40844/0 (socket says 172.21.15.1:40844) 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.192+0000 7f59caa00700 1 -- 172.21.15.1:0/2612948921 learned_addr learned my addr 172.21.15.1:0/2612948921 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.192+0000 7f59caa00700 1 -- 172.21.15.1:0/2612948921 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc105920 msgr2=0x7f59cc115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.192+0000 7f59ca000700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc105920 0x7f59cc115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.192+0000 7f59caa00700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc105920 0x7f59cc115f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.192+0000 7f59caa00700 1 -- 172.21.15.1:0/2612948921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59bc005040 con 0x7f59cc104f80 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.192+0000 7f59caa00700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 0x7f59cc115a20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00b3b0 tx=0x7f59bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.193+0000 7f59c2a00700 1 -- 172.21.15.1:0/2612948921 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59bc00e040 con 0x7f59cc104f80 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.193+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc10fb60 con 0x7f59cc104f80 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.193+0000 7f59c2a00700 1 -- 172.21.15.1:0/2612948921 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59bc00bbf0 con 0x7f59cc104f80 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.193+0000 7f59c2a00700 1 -- 172.21.15.1:0/2612948921 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59bc01c630 con 0x7f59cc104f80 2024-02-28T16:50:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.193+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc110120 con 0x7f59cc104f80 2024-02-28T16:50:51.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.194+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc05f5c0 con 0x7f59cc104f80 2024-02-28T16:50:51.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.195+0000 7f59c2a00700 1 -- 172.21.15.1:0/2612948921 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f59bc00bd60 con 0x7f59cc104f80 2024-02-28T16:50:51.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.196+0000 7f59c2a00700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f59b006d7e0 0x7f59b006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:51.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.196+0000 7f59c2a00700 1 -- 172.21.15.1:0/2612948921 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f59bc08e2a0 con 0x7f59cc104f80 2024-02-28T16:50:51.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.196+0000 7f59ca000700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f59b006d7e0 0x7f59b006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:51.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.197+0000 7f59ca000700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f59b006d7e0 0x7f59b006fc90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f59cc111250 tx=0x7f59b800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:51.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.200+0000 7f59c2a00700 1 -- 172.21.15.1:0/2612948921 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f59bc05b680 con 0x7f59cc104f80 2024-02-28T16:50:51.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.441+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f59cc111000 con 0x7f59b006d7e0 2024-02-28T16:50:51.435 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:51 smithi001 ceph-mon[112940]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:51.435 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:51 smithi001 ceph-mon[112940]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:51.435 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:51 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3314680229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:50:51.435 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:51 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1958086355' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.444+0000 7f59c2a00700 1 -- 172.21.15.1:0/2612948921 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f59cc111000 con 0x7f59b006d7e0 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:50:51.437 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:50:51.438 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:50:51.438 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:50:51.438 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:50:51.438 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:50:51.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f59b006d7e0 msgr2=0x7f59b006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f59b006d7e0 0x7f59b006fc90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f59cc111250 tx=0x7f59b800b040 comp rx=0 tx=0).stop 2024-02-28T16:50:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 msgr2=0x7f59cc115a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 0x7f59cc115a20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00b3b0 tx=0x7f59bc009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 shutdown_connections 2024-02-28T16:50:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f59b006d7e0 0x7f59b006fc90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59cc104f80 0x7f59cc115a20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 --2- 172.21.15.1:0/2612948921 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f59cc105920 0x7f59cc115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 >> 172.21.15.1:0/2612948921 conn(0x7f59cc0fab10 msgr2=0x7f59cc0fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:51.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 shutdown_connections 2024-02-28T16:50:51.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.447+0000 7f59cbe00700 1 -- 172.21.15.1:0/2612948921 wait complete. 2024-02-28T16:50:51.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:51 smithi066 ceph-mon[85781]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:51.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:51 smithi066 ceph-mon[85781]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:51.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:51 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3314680229' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:50:51.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:51 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1958086355' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:50:51.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.606+0000 7f0db6e00700 1 -- 172.21.15.1:0/4197327740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fdd90 msgr2=0x7f0db00fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.606+0000 7f0db6e00700 1 --2- 172.21.15.1:0/4197327740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fdd90 0x7f0db00fe160 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0da4004940 tx=0x7f0da4009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:51.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.606+0000 7f0db6e00700 1 -- 172.21.15.1:0/4197327740 shutdown_connections 2024-02-28T16:50:51.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.606+0000 7f0db6e00700 1 --2- 172.21.15.1:0/4197327740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db00fe730 0x7f0db0106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.606+0000 7f0db6e00700 1 --2- 172.21.15.1:0/4197327740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fdd90 0x7f0db00fe160 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.606+0000 7f0db6e00700 1 -- 172.21.15.1:0/4197327740 >> 172.21.15.1:0/4197327740 conn(0x7f0db0073ab0 msgr2=0x7f0db0073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:51.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.606+0000 7f0db6e00700 1 -- 172.21.15.1:0/4197327740 shutdown_connections 2024-02-28T16:50:51.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.606+0000 7f0db6e00700 1 -- 172.21.15.1:0/4197327740 wait complete. 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.607+0000 7f0db6e00700 1 Processor -- start 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.607+0000 7f0db6e00700 1 -- start start 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db6e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db00fdd90 0x7f0db00fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db6e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fe730 0x7f0db00fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db6e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00fc790 con 0x7f0db00fe730 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db6e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00fc900 con 0x7f0db00fdd90 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fe730 0x7f0db00fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fe730 0x7f0db00fc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40870/0 (socket says 172.21.15.1:40870) 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db5000700 1 -- 172.21.15.1:0/62689589 learned_addr learned my addr 172.21.15.1:0/62689589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db5000700 1 -- 172.21.15.1:0/62689589 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db00fdd90 msgr2=0x7f0db00fbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db5000700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db00fdd90 0x7f0db00fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.608+0000 7f0db5000700 1 -- 172.21.15.1:0/62689589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da4005040 con 0x7f0db00fe730 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.609+0000 7f0db5000700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fe730 0x7f0db00fc190 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00c8c0 tx=0x7f0d9c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:51.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.609+0000 7f0dad600700 1 -- 172.21.15.1:0/62689589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d9c00a980 con 0x7f0db00fe730 2024-02-28T16:50:51.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.609+0000 7f0dad600700 1 -- 172.21.15.1:0/62689589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d9c00ce60 con 0x7f0db00fe730 2024-02-28T16:50:51.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.609+0000 7f0dad600700 1 -- 172.21.15.1:0/62689589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d9c0115e0 con 0x7f0db00fe730 2024-02-28T16:50:51.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.609+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db0119ac0 con 0x7f0db00fe730 2024-02-28T16:50:51.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.609+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db0119fa0 con 0x7f0db00fe730 2024-02-28T16:50:51.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.610+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db01044c0 con 0x7f0db00fe730 2024-02-28T16:50:51.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.611+0000 7f0dad600700 1 -- 172.21.15.1:0/62689589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0d9c012400 con 0x7f0db00fe730 2024-02-28T16:50:51.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.613+0000 7f0dad600700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:50:51.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.613+0000 7f0dad600700 1 -- 172.21.15.1:0/62689589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0d9c08ceb0 con 0x7f0db00fe730 2024-02-28T16:50:51.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.613+0000 7f0db5a00700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:50:51.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.614+0000 7f0db5a00700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0da4004f90 tx=0x7f0da4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:50:51.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.617+0000 7f0dad600700 1 -- 172.21.15.1:0/62689589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0d9c056820 con 0x7f0db00fe730 2024-02-28T16:50:51.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.920+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0db005f5c0 con 0x7f0db00fe730 2024-02-28T16:50:51.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.921+0000 7f0dad600700 1 -- 172.21.15.1:0/62689589 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f0d9c0126b0 con 0x7f0db00fe730 2024-02-28T16:50:51.914 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:50:51.914 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:50:51.914 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:50:51.914 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:50:51.914 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:50:51.914 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:50:51.914 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 msgr2=0x7f0d9806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0da4004f90 tx=0x7f0da4009e30 comp rx=0 tx=0).stop 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fe730 msgr2=0x7f0db00fc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fe730 0x7f0db00fc190 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00c8c0 tx=0x7f0d9c0044c0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 shutdown_connections 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db00fdd90 0x7f0db00fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 --2- 172.21.15.1:0/62689589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db00fe730 0x7f0db00fc190 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:50:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 >> 172.21.15.1:0/62689589 conn(0x7f0db0073ab0 msgr2=0x7f0db0101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:50:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 shutdown_connections 2024-02-28T16:50:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:50:51.924+0000 7f0db6e00700 1 -- 172.21.15.1:0/62689589 wait complete. 2024-02-28T16:50:52.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:52 smithi066 ceph-mon[85781]: pgmap v492: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:52.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:52 smithi066 ceph-mon[85781]: from='client.34942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:52.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:52 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/62689589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:50:52.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:52 smithi001 ceph-mon[112940]: pgmap v492: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:52.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:52 smithi001 ceph-mon[112940]: from='client.34942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:50:52.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:52 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/62689589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:50:54.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:54 smithi066 ceph-mon[85781]: pgmap v493: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:54.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:54 smithi001 ceph-mon[112940]: pgmap v493: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:56.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:56 smithi066 ceph-mon[85781]: pgmap v494: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:56.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:50:56.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:56 smithi001 ceph-mon[112940]: pgmap v494: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:56.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:50:58.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:50:58 smithi066 ceph-mon[85781]: pgmap v495: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:50:58.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:50:58 smithi001 ceph-mon[112940]: pgmap v495: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:00.502 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:00 smithi066 ceph-mon[85781]: pgmap v496: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:00.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:00 smithi001 ceph-mon[112940]: pgmap v496: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:02.501 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:02 smithi066 ceph-mon[85781]: pgmap v497: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:02.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:02 smithi001 ceph-mon[112940]: pgmap v497: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:04.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:04 smithi001 ceph-mon[112940]: pgmap v498: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:04.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:04 smithi066 ceph-mon[85781]: pgmap v498: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:06.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:06 smithi001 ceph-mon[112940]: pgmap v499: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:06.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:06 smithi066 ceph-mon[85781]: pgmap v499: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:08.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:08 smithi001 ceph-mon[112940]: pgmap v500: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:08.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:08 smithi066 ceph-mon[85781]: pgmap v500: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:10.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:10 smithi001 ceph-mon[112940]: pgmap v501: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:10.554 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:10 smithi066 ceph-mon[85781]: pgmap v501: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:11.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:51:11.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:51:12.751 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:12 smithi066 ceph-mon[85781]: pgmap v502: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:12.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:12 smithi001 ceph-mon[112940]: pgmap v502: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:14.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:14 smithi066 ceph-mon[85781]: pgmap v503: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:14.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:14 smithi001 ceph-mon[112940]: pgmap v503: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:16.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:16 smithi066 ceph-mon[85781]: pgmap v504: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:16.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:16 smithi001 ceph-mon[112940]: pgmap v504: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:18.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:18 smithi066 ceph-mon[85781]: pgmap v505: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:18.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:18 smithi001 ceph-mon[112940]: pgmap v505: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:20.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:20 smithi066 ceph-mon[85781]: pgmap v506: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:20.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:20 smithi001 ceph-mon[112940]: pgmap v506: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:22.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.098+0000 7f84f2c00700 1 -- 172.21.15.1:0/1129965616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec103dc0 msgr2=0x7f84ec104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.098+0000 7f84f2c00700 1 --2- 172.21.15.1:0/1129965616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec103dc0 0x7f84ec104210 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f84d8004940 tx=0x7f84d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:22.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.098+0000 7f84f2c00700 1 -- 172.21.15.1:0/1129965616 shutdown_connections 2024-02-28T16:51:22.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.098+0000 7f84f2c00700 1 --2- 172.21.15.1:0/1129965616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec103dc0 0x7f84ec104210 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.098+0000 7f84f2c00700 1 --2- 172.21.15.1:0/1129965616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84ec1095f0 0x7f84ec1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.098+0000 7f84f2c00700 1 -- 172.21.15.1:0/1129965616 >> 172.21.15.1:0/1129965616 conn(0x7f84ec072d50 msgr2=0x7f84ec075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:22.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.098+0000 7f84f2c00700 1 -- 172.21.15.1:0/1129965616 shutdown_connections 2024-02-28T16:51:22.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.098+0000 7f84f2c00700 1 -- 172.21.15.1:0/1129965616 wait complete. 2024-02-28T16:51:22.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.099+0000 7f84f2c00700 1 Processor -- start 2024-02-28T16:51:22.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.099+0000 7f84f2c00700 1 -- start start 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.099+0000 7f84f2c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84ec103dc0 0x7f84ec11a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.099+0000 7f84f2c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec1095f0 0x7f84ec11a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.099+0000 7f84f2c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec11abe0 con 0x7f84ec1095f0 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.099+0000 7f84f2c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec113ef0 con 0x7f84ec103dc0 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.100+0000 7f84eb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec1095f0 0x7f84ec11a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.100+0000 7f84eb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec1095f0 0x7f84ec11a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42720/0 (socket says 172.21.15.1:42720) 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.100+0000 7f84eb400700 1 -- 172.21.15.1:0/3797856361 learned_addr learned my addr 172.21.15.1:0/3797856361 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.100+0000 7f84ebe00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84ec103dc0 0x7f84ec11a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.100+0000 7f84eb400700 1 -- 172.21.15.1:0/3797856361 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84ec103dc0 msgr2=0x7f84ec11a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.100+0000 7f84eb400700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84ec103dc0 0x7f84ec11a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.100+0000 7f84eb400700 1 -- 172.21.15.1:0/3797856361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d8005040 con 0x7f84ec1095f0 2024-02-28T16:51:22.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.100+0000 7f84eb400700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec1095f0 0x7f84ec11a590 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009150 tx=0x7f84d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:22.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.101+0000 7f84e3e00700 1 -- 172.21.15.1:0/3797856361 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d800e040 con 0x7f84ec1095f0 2024-02-28T16:51:22.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.101+0000 7f84e3e00700 1 -- 172.21.15.1:0/3797856361 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84d800bbf0 con 0x7f84ec1095f0 2024-02-28T16:51:22.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.101+0000 7f84e3e00700 1 -- 172.21.15.1:0/3797856361 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d801c630 con 0x7f84ec1095f0 2024-02-28T16:51:22.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.101+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec114170 con 0x7f84ec1095f0 2024-02-28T16:51:22.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.101+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec1146b0 con 0x7f84ec1095f0 2024-02-28T16:51:22.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.102+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ec10b9c0 con 0x7f84ec1095f0 2024-02-28T16:51:22.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.103+0000 7f84e3e00700 1 -- 172.21.15.1:0/3797856361 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f84d800bd60 con 0x7f84ec1095f0 2024-02-28T16:51:22.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.104+0000 7f84e3e00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f84d006d7f0 0x7f84d006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.104+0000 7f84e3e00700 1 -- 172.21.15.1:0/3797856361 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f84d808e2a0 con 0x7f84ec1095f0 2024-02-28T16:51:22.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.104+0000 7f84ebe00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f84d006d7f0 0x7f84d006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.105+0000 7f84ebe00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f84d006d7f0 0x7f84d006fca0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f84dc004d60 tx=0x7f84dc00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:22.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.109+0000 7f84e3e00700 1 -- 172.21.15.1:0/3797856361 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f84d805b680 con 0x7f84ec1095f0 2024-02-28T16:51:22.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.354+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f84ec04bed0 con 0x7f84d006d7f0 2024-02-28T16:51:22.347 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:22 smithi001 ceph-mon[112940]: pgmap v507: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:22.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.356+0000 7f84e3e00700 1 -- 172.21.15.1:0/3797856361 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f84ec04bed0 con 0x7f84d006d7f0 2024-02-28T16:51:22.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.358+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f84d006d7f0 msgr2=0x7f84d006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.358+0000 7f84f2c00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f84d006d7f0 0x7f84d006fca0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f84dc004d60 tx=0x7f84dc00b040 comp rx=0 tx=0).stop 2024-02-28T16:51:22.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.358+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec1095f0 msgr2=0x7f84ec11a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.358+0000 7f84f2c00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec1095f0 0x7f84ec11a590 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009150 tx=0x7f84d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:22.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.359+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 shutdown_connections 2024-02-28T16:51:22.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.359+0000 7f84f2c00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f84d006d7f0 0x7f84d006fca0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.359+0000 7f84f2c00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f84ec103dc0 0x7f84ec11a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.359+0000 7f84f2c00700 1 --2- 172.21.15.1:0/3797856361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84ec1095f0 0x7f84ec11a590 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.359+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 >> 172.21.15.1:0/3797856361 conn(0x7f84ec072d50 msgr2=0x7f84ec074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:22.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.359+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 shutdown_connections 2024-02-28T16:51:22.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.359+0000 7f84f2c00700 1 -- 172.21.15.1:0/3797856361 wait complete. 2024-02-28T16:51:22.368 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:51:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.514+0000 7f0bea800700 1 -- 172.21.15.1:0/704255597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 msgr2=0x7f0be4105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.514+0000 7f0bea800700 1 --2- 172.21.15.1:0/704255597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 0x7f0be4105540 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4004940 tx=0x7f0bd4009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.514+0000 7f0bea800700 1 -- 172.21.15.1:0/704255597 shutdown_connections 2024-02-28T16:51:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.514+0000 7f0bea800700 1 --2- 172.21.15.1:0/704255597 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0be4105b10 0x7f0be40ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.514+0000 7f0bea800700 1 --2- 172.21.15.1:0/704255597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 0x7f0be4105540 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.514+0000 7f0bea800700 1 -- 172.21.15.1:0/704255597 >> 172.21.15.1:0/704255597 conn(0x7f0be40face0 msgr2=0x7f0be40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.515+0000 7f0bea800700 1 -- 172.21.15.1:0/704255597 shutdown_connections 2024-02-28T16:51:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.515+0000 7f0bea800700 1 -- 172.21.15.1:0/704255597 wait complete. 2024-02-28T16:51:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.515+0000 7f0bea800700 1 Processor -- start 2024-02-28T16:51:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.516+0000 7f0bea800700 1 -- start start 2024-02-28T16:51:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.516+0000 7f0bea800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 0x7f0be4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.516+0000 7f0bea800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0be4105b10 0x7f0be41161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.516+0000 7f0bea800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be41167f0 con 0x7f0be4105170 2024-02-28T16:51:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.516+0000 7f0bea800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be410fb40 con 0x7f0be4105b10 2024-02-28T16:51:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.516+0000 7f0be3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 0x7f0be4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.516+0000 7f0be3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 0x7f0be4115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42732/0 (socket says 172.21.15.1:42732) 2024-02-28T16:51:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.516+0000 7f0be3400700 1 -- 172.21.15.1:0/3747806818 learned_addr learned my addr 172.21.15.1:0/3747806818 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.517+0000 7f0be2a00700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0be4105b10 0x7f0be41161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.517+0000 7f0be3400700 1 -- 172.21.15.1:0/3747806818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0be4105b10 msgr2=0x7f0be41161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.517+0000 7f0be3400700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0be4105b10 0x7f0be41161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.517+0000 7f0be3400700 1 -- 172.21.15.1:0/3747806818 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd4005040 con 0x7f0be4105170 2024-02-28T16:51:22.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.517+0000 7f0be3400700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 0x7f0be4115c60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009fd0 tx=0x7f0bd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.517+0000 7f0bdb400700 1 -- 172.21.15.1:0/3747806818 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd400e040 con 0x7f0be4105170 2024-02-28T16:51:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.517+0000 7f0bdb400700 1 -- 172.21.15.1:0/3747806818 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bd400bbf0 con 0x7f0be4105170 2024-02-28T16:51:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.518+0000 7f0bdb400700 1 -- 172.21.15.1:0/3747806818 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd401c3f0 con 0x7f0be4105170 2024-02-28T16:51:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.518+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be410fdc0 con 0x7f0be4105170 2024-02-28T16:51:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.518+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be4110300 con 0x7f0be4105170 2024-02-28T16:51:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.519+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be4102480 con 0x7f0be4105170 2024-02-28T16:51:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.520+0000 7f0bdb400700 1 -- 172.21.15.1:0/3747806818 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0bd400bd60 con 0x7f0be4105170 2024-02-28T16:51:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.521+0000 7f0bdb400700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc806d7f0 0x7f0bc806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.521+0000 7f0bdb400700 1 -- 172.21.15.1:0/3747806818 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0bd408e2b0 con 0x7f0be4105170 2024-02-28T16:51:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.521+0000 7f0be2a00700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc806d7f0 0x7f0bc806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.522+0000 7f0be2a00700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc806d7f0 0x7f0bc806fca0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0be4111540 tx=0x7f0bd0009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:22.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.526+0000 7f0bdb400700 1 -- 172.21.15.1:0/3747806818 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0bd4057ae0 con 0x7f0be4105170 2024-02-28T16:51:22.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:22 smithi066 ceph-mon[85781]: pgmap v507: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.772+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0be404bf70 con 0x7f0bc806d7f0 2024-02-28T16:51:22.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.774+0000 7f0bdb400700 1 -- 172.21.15.1:0/3747806818 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0bd0004e30 con 0x7f0bc806d7f0 2024-02-28T16:51:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.777+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc806d7f0 msgr2=0x7f0bc806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.777+0000 7f0bea800700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc806d7f0 0x7f0bc806fca0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0be4111540 tx=0x7f0bd0009ee0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.777+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 msgr2=0x7f0be4115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.777+0000 7f0bea800700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 0x7f0be4115c60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009fd0 tx=0x7f0bd4009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:22.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.777+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 shutdown_connections 2024-02-28T16:51:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.778+0000 7f0bea800700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc806d7f0 0x7f0bc806fca0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.778+0000 7f0bea800700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0be4105170 0x7f0be4115c60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.778+0000 7f0bea800700 1 --2- 172.21.15.1:0/3747806818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0be4105b10 0x7f0be41161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.778+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 >> 172.21.15.1:0/3747806818 conn(0x7f0be40face0 msgr2=0x7f0be40fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.778+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 shutdown_connections 2024-02-28T16:51:22.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.778+0000 7f0bea800700 1 -- 172.21.15.1:0/3747806818 wait complete. 2024-02-28T16:51:22.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.946+0000 7f93a3e00700 1 -- 172.21.15.1:0/1142915783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 msgr2=0x7f939c104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.946+0000 7f93a3e00700 1 --2- 172.21.15.1:0/1142915783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 0x7f939c104210 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f938c004940 tx=0x7f938c009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.946+0000 7f93a3e00700 1 -- 172.21.15.1:0/1142915783 shutdown_connections 2024-02-28T16:51:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.946+0000 7f93a3e00700 1 --2- 172.21.15.1:0/1142915783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 0x7f939c104210 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.946+0000 7f93a3e00700 1 --2- 172.21.15.1:0/1142915783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f939c1095f0 0x7f939c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.946+0000 7f93a3e00700 1 -- 172.21.15.1:0/1142915783 >> 172.21.15.1:0/1142915783 conn(0x7f939c072d50 msgr2=0x7f939c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.946+0000 7f93a3e00700 1 -- 172.21.15.1:0/1142915783 shutdown_connections 2024-02-28T16:51:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.946+0000 7f93a3e00700 1 -- 172.21.15.1:0/1142915783 wait complete. 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.947+0000 7f93a3e00700 1 Processor -- start 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.947+0000 7f93a3e00700 1 -- start start 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 0x7f939c11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a3e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f939c1095f0 0x7f939c11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a3e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c11abd0 con 0x7f939c103dc0 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a3e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c113ee0 con 0x7f939c1095f0 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 0x7f939c11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 0x7f939c11a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42762/0 (socket says 172.21.15.1:42762) 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a1400700 1 -- 172.21.15.1:0/308905078 learned_addr learned my addr 172.21.15.1:0/308905078 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a0a00700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f939c1095f0 0x7f939c11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a1400700 1 -- 172.21.15.1:0/308905078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f939c1095f0 msgr2=0x7f939c11a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a1400700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f939c1095f0 0x7f939c11a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.948+0000 7f93a1400700 1 -- 172.21.15.1:0/308905078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c005040 con 0x7f939c103dc0 2024-02-28T16:51:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.949+0000 7f93a1400700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 0x7f939c11a040 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9390002a50 tx=0x7f93900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.949+0000 7f9399600700 1 -- 172.21.15.1:0/308905078 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9390007cc0 con 0x7f939c103dc0 2024-02-28T16:51:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.949+0000 7f9399600700 1 -- 172.21.15.1:0/308905078 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f939000f040 con 0x7f939c103dc0 2024-02-28T16:51:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.949+0000 7f9399600700 1 -- 172.21.15.1:0/308905078 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9390009640 con 0x7f939c103dc0 2024-02-28T16:51:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.949+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c1141c0 con 0x7f939c103dc0 2024-02-28T16:51:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.949+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c114700 con 0x7f939c103dc0 2024-02-28T16:51:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.950+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939c10b9c0 con 0x7f939c103dc0 2024-02-28T16:51:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.951+0000 7f9399600700 1 -- 172.21.15.1:0/308905078 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f93900097a0 con 0x7f939c103dc0 2024-02-28T16:51:22.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.952+0000 7f9399600700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f938406d7f0 0x7f938406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:22.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.952+0000 7f9399600700 1 -- 172.21.15.1:0/308905078 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f939008cbc0 con 0x7f939c103dc0 2024-02-28T16:51:22.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.953+0000 7f93a0a00700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f938406d7f0 0x7f938406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:22.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.953+0000 7f93a0a00700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f938406d7f0 0x7f938406fca0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f938c00b3b0 tx=0x7f938c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:22.958+0000 7f9399600700 1 -- 172.21.15.1:0/308905078 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9390059fc0 con 0x7f939c103dc0 2024-02-28T16:51:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.192+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f939c115620 con 0x7f938406d7f0 2024-02-28T16:51:23.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.209+0000 7f9399600700 1 -- 172.21.15.1:0/308905078 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f939c115620 con 0x7f938406d7f0 2024-02-28T16:51:23.202 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:51:23.202 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (17m) 9m ago 31m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:51:23.202 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (14m) 9m ago 32m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:51:23.202 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (14m) 10m ago 30m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:51:23.202 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (17m) 9m ago 31m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:51:23.202 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (10m) 9m ago 24m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:51:23.202 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (9m) 9m ago 24m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (24m) 10m ago 24m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (24m) 10m ago 24m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (19m) 9m ago 33m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (18m) 10m ago 30m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (15m) 9m ago 33m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (15m) 10m ago 30m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (18m) 9m ago 31m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (18m) 10m ago 30m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (13m) 9m ago 29m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (13m) 9m ago 28m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (12m) 9m ago 28m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (12m) 10m ago 27m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (11m) 10m ago 27m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (10m) 10m ago 26m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:51:23.203 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (17m) 9m ago 31m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:51:23.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.212+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f938406d7f0 msgr2=0x7f938406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:23.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.212+0000 7f93a3e00700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f938406d7f0 0x7f938406fca0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f938c00b3b0 tx=0x7f938c009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:23.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.212+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 msgr2=0x7f939c11a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:23.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.212+0000 7f93a3e00700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 0x7f939c11a040 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9390002a50 tx=0x7f93900046b0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.212+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 shutdown_connections 2024-02-28T16:51:23.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.212+0000 7f93a3e00700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f938406d7f0 0x7f938406fca0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.213+0000 7f93a3e00700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f939c103dc0 0x7f939c11a040 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.213+0000 7f93a3e00700 1 --2- 172.21.15.1:0/308905078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f939c1095f0 0x7f939c11a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.213+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 >> 172.21.15.1:0/308905078 conn(0x7f939c072d50 msgr2=0x7f939c074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:23.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.213+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 shutdown_connections 2024-02-28T16:51:23.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.213+0000 7f93a3e00700 1 -- 172.21.15.1:0/308905078 wait complete. 2024-02-28T16:51:23.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.376+0000 7f99f3000700 1 -- 172.21.15.1:0/1611124586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 msgr2=0x7f99ec0fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:23.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.376+0000 7f99f3000700 1 --2- 172.21.15.1:0/1611124586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 0x7f99ec0fcd50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f99d8004940 tx=0x7f99d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:23.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.376+0000 7f99f3000700 1 -- 172.21.15.1:0/1611124586 shutdown_connections 2024-02-28T16:51:23.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.376+0000 7f99f3000700 1 --2- 172.21.15.1:0/1611124586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 0x7f99ec0fcd50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.376+0000 7f99f3000700 1 --2- 172.21.15.1:0/1611124586 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ec102d40 0x7f99ec103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.376+0000 7f99f3000700 1 -- 172.21.15.1:0/1611124586 >> 172.21.15.1:0/1611124586 conn(0x7f99ec0f88b0 msgr2=0x7f99ec0facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:23.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.376+0000 7f99f3000700 1 -- 172.21.15.1:0/1611124586 shutdown_connections 2024-02-28T16:51:23.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.376+0000 7f99f3000700 1 -- 172.21.15.1:0/1611124586 wait complete. 2024-02-28T16:51:23.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.377+0000 7f99f3000700 1 Processor -- start 2024-02-28T16:51:23.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.377+0000 7f99f3000700 1 -- start start 2024-02-28T16:51:23.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f3000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ec102d40 0x7f99ec113840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:23.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f3000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 0x7f99ec113d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:23.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f3000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec114460 con 0x7f99ec1036e0 2024-02-28T16:51:23.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f3000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec10d700 con 0x7f99ec102d40 2024-02-28T16:51:23.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 0x7f99ec113d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:23.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 0x7f99ec113d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42776/0 (socket says 172.21.15.1:42776) 2024-02-28T16:51:23.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f1200700 1 -- 172.21.15.1:0/1217171593 learned_addr learned my addr 172.21.15.1:0/1217171593 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:23.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f1c00700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ec102d40 0x7f99ec113840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f1200700 1 -- 172.21.15.1:0/1217171593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ec102d40 msgr2=0x7f99ec113840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.378+0000 7f99f1200700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ec102d40 0x7f99ec113840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.379+0000 7f99f1200700 1 -- 172.21.15.1:0/1217171593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d8005040 con 0x7f99ec1036e0 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.379+0000 7f99f1200700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 0x7f99ec113d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009030 tx=0x7f99d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.379+0000 7f99e9600700 1 -- 172.21.15.1:0/1217171593 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d800e040 con 0x7f99ec1036e0 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.379+0000 7f99e9600700 1 -- 172.21.15.1:0/1217171593 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99d800bbf0 con 0x7f99ec1036e0 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.379+0000 7f99e9600700 1 -- 172.21.15.1:0/1217171593 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d801c630 con 0x7f99ec1036e0 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.379+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ec10d980 con 0x7f99ec1036e0 2024-02-28T16:51:23.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.379+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec10df40 con 0x7f99ec1036e0 2024-02-28T16:51:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.380+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ec05f5c0 con 0x7f99ec1036e0 2024-02-28T16:51:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.382+0000 7f99e9600700 1 -- 172.21.15.1:0/1217171593 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f99d800bd60 con 0x7f99ec1036e0 2024-02-28T16:51:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.383+0000 7f99e9600700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99d406d7f0 0x7f99d406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.383+0000 7f99e9600700 1 -- 172.21.15.1:0/1217171593 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f99d808e1f0 con 0x7f99ec1036e0 2024-02-28T16:51:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.383+0000 7f99f1c00700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99d406d7f0 0x7f99d406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:23.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.384+0000 7f99f1c00700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99d406d7f0 0x7f99d406fca0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f99e0006fd0 tx=0x7f99e0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.387+0000 7f99e9600700 1 -- 172.21.15.1:0/1217171593 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99d805b5d0 con 0x7f99ec1036e0 2024-02-28T16:51:23.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:23 smithi001 ceph-mon[112940]: from='client.34950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:23.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.703+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f99ec10ee20 con 0x7f99ec1036e0 2024-02-28T16:51:23.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.704+0000 7f99e9600700 1 -- 172.21.15.1:0/1217171593 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f99d805b160 con 0x7f99ec1036e0 2024-02-28T16:51:23.696 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:51:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:51:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:51:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:51:23.697 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:51:23.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.706+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99d406d7f0 msgr2=0x7f99d406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:23.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.706+0000 7f99f3000700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99d406d7f0 0x7f99d406fca0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f99e0006fd0 tx=0x7f99e0008040 comp rx=0 tx=0).stop 2024-02-28T16:51:23.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.706+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 msgr2=0x7f99ec113d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:23.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.706+0000 7f99f3000700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 0x7f99ec113d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009030 tx=0x7f99d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:23.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.706+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 shutdown_connections 2024-02-28T16:51:23.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.706+0000 7f99f3000700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99d406d7f0 0x7f99d406fca0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.707+0000 7f99f3000700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ec102d40 0x7f99ec113840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.707+0000 7f99f3000700 1 --2- 172.21.15.1:0/1217171593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ec1036e0 0x7f99ec113d80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.707+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 >> 172.21.15.1:0/1217171593 conn(0x7f99ec0f88b0 msgr2=0x7f99ec0fa250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:23.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.707+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 shutdown_connections 2024-02-28T16:51:23.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.707+0000 7f99f3000700 1 -- 172.21.15.1:0/1217171593 wait complete. 2024-02-28T16:51:23.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:23 smithi066 ceph-mon[85781]: from='client.34950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:23.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.864+0000 7f6db6600700 1 -- 172.21.15.1:0/2257381555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105160 msgr2=0x7f6db0105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.864+0000 7f6db6600700 1 --2- 172.21.15.1:0/2257381555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105160 0x7f6db0105530 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6da0004940 tx=0x7f6da0009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.864+0000 7f6db6600700 1 -- 172.21.15.1:0/2257381555 shutdown_connections 2024-02-28T16:51:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.864+0000 7f6db6600700 1 --2- 172.21.15.1:0/2257381555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6db0105b00 0x7f6db00ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.864+0000 7f6db6600700 1 --2- 172.21.15.1:0/2257381555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105160 0x7f6db0105530 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.864+0000 7f6db6600700 1 -- 172.21.15.1:0/2257381555 >> 172.21.15.1:0/2257381555 conn(0x7f6db00facb0 msgr2=0x7f6db00fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.865+0000 7f6db6600700 1 -- 172.21.15.1:0/2257381555 shutdown_connections 2024-02-28T16:51:23.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.865+0000 7f6db6600700 1 -- 172.21.15.1:0/2257381555 wait complete. 2024-02-28T16:51:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.866+0000 7f6db6600700 1 Processor -- start 2024-02-28T16:51:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.866+0000 7f6db6600700 1 -- start start 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.867+0000 7f6db6600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6db0105160 0x7f6db0115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.867+0000 7f6db6600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105b00 0x7f6db01161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.867+0000 7f6db6600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db0116800 con 0x7f6db0105b00 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.867+0000 7f6db6600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db010fb30 con 0x7f6db0105160 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.867+0000 7f6daea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105b00 0x7f6db01161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.867+0000 7f6daea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105b00 0x7f6db01161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42786/0 (socket says 172.21.15.1:42786) 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.867+0000 7f6daea00700 1 -- 172.21.15.1:0/3970311339 learned_addr learned my addr 172.21.15.1:0/3970311339 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6daf400700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6db0105160 0x7f6db0115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:23.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6daea00700 1 -- 172.21.15.1:0/3970311339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6db0105160 msgr2=0x7f6db0115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6daea00700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6db0105160 0x7f6db0115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6daea00700 1 -- 172.21.15.1:0/3970311339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da0005040 con 0x7f6db0105b00 2024-02-28T16:51:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6daea00700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105b00 0x7f6db01161b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c002a50 tx=0x7f6d9c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6da7400700 1 -- 172.21.15.1:0/3970311339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d9c007cc0 con 0x7f6db0105b00 2024-02-28T16:51:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6da7400700 1 -- 172.21.15.1:0/3970311339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d9c00f040 con 0x7f6db0105b00 2024-02-28T16:51:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db010fe10 con 0x7f6db0105b00 2024-02-28T16:51:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6da7400700 1 -- 172.21.15.1:0/3970311339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d9c00daf0 con 0x7f6db0105b00 2024-02-28T16:51:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.868+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db0110350 con 0x7f6db0105b00 2024-02-28T16:51:23.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.870+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db00fe8f0 con 0x7f6db0105b00 2024-02-28T16:51:23.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.871+0000 7f6da7400700 1 -- 172.21.15.1:0/3970311339 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6d9c009810 con 0x7f6db0105b00 2024-02-28T16:51:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.871+0000 7f6da7400700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d9406d7f0 0x7f6d9406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.872+0000 7f6da7400700 1 -- 172.21.15.1:0/3970311339 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6d9c08cc30 con 0x7f6db0105b00 2024-02-28T16:51:23.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.872+0000 7f6daf400700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d9406d7f0 0x7f6d9406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:23.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.872+0000 7f6daf400700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d9406d7f0 0x7f6d9406fca0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6da0000c00 tx=0x7f6da0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:23.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:23.877+0000 7f6da7400700 1 -- 172.21.15.1:0/3970311339 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d9c059f90 con 0x7f6db0105b00 2024-02-28T16:51:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.154+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6db010ffa0 con 0x7f6db0105b00 2024-02-28T16:51:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.155+0000 7f6da7400700 1 -- 172.21.15.1:0/3970311339 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f6db010ffa0 con 0x7f6db0105b00 2024-02-28T16:51:24.148 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:51:24.148 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:51:24.148 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:51:24.149 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:51:24.150 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:51:24.151 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:51:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d9406d7f0 msgr2=0x7f6d9406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d9406d7f0 0x7f6d9406fca0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6da0000c00 tx=0x7f6da0009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105b00 msgr2=0x7f6db01161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105b00 0x7f6db01161b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c002a50 tx=0x7f6d9c004600 comp rx=0 tx=0).stop 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 shutdown_connections 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d9406d7f0 0x7f6d9406fca0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6db0105160 0x7f6db0115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 --2- 172.21.15.1:0/3970311339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6db0105b00 0x7f6db01161b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 >> 172.21.15.1:0/3970311339 conn(0x7f6db00facb0 msgr2=0x7f6db00fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 shutdown_connections 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.158+0000 7f6db6600700 1 -- 172.21.15.1:0/3970311339 wait complete. 2024-02-28T16:51:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:51:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.320+0000 7f25e1200700 1 -- 172.21.15.1:0/1556724667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fe900 msgr2=0x7f25dc106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.320+0000 7f25e1200700 1 --2- 172.21.15.1:0/1556724667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fe900 0x7f25dc106f40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f25c8004940 tx=0x7f25c8009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.320+0000 7f25e1200700 1 -- 172.21.15.1:0/1556724667 shutdown_connections 2024-02-28T16:51:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.320+0000 7f25e1200700 1 --2- 172.21.15.1:0/1556724667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fe900 0x7f25dc106f40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.320+0000 7f25e1200700 1 --2- 172.21.15.1:0/1556724667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25dc0fdf60 0x7f25dc0fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.320+0000 7f25e1200700 1 -- 172.21.15.1:0/1556724667 >> 172.21.15.1:0/1556724667 conn(0x7f25dc0f7a10 msgr2=0x7f25dc0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.320+0000 7f25e1200700 1 -- 172.21.15.1:0/1556724667 shutdown_connections 2024-02-28T16:51:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.320+0000 7f25e1200700 1 -- 172.21.15.1:0/1556724667 wait complete. 2024-02-28T16:51:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.321+0000 7f25e1200700 1 Processor -- start 2024-02-28T16:51:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.322+0000 7f25e1200700 1 -- start start 2024-02-28T16:51:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.322+0000 7f25e1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fdf60 0x7f25dc0fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.322+0000 7f25e1200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25dc0fe900 0x7f25dc0fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.322+0000 7f25e1200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0fc8d0 con 0x7f25dc0fdf60 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.322+0000 7f25e1200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0fca40 con 0x7f25dc0fe900 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.322+0000 7f25d9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fdf60 0x7f25dc0fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.322+0000 7f25d9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fdf60 0x7f25dc0fbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42812/0 (socket says 172.21.15.1:42812) 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.322+0000 7f25d9e00700 1 -- 172.21.15.1:0/2193520526 learned_addr learned my addr 172.21.15.1:0/2193520526 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25d9400700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25dc0fe900 0x7f25dc0fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25d9e00700 1 -- 172.21.15.1:0/2193520526 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25dc0fe900 msgr2=0x7f25dc0fc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25d9e00700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25dc0fe900 0x7f25dc0fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25d9e00700 1 -- 172.21.15.1:0/2193520526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c8005040 con 0x7f25dc0fdf60 2024-02-28T16:51:24.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25d9e00700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fdf60 0x7f25dc0fbd90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00c8c0 tx=0x7f25cc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:24.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25d2000700 1 -- 172.21.15.1:0/2193520526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25cc00a980 con 0x7f25dc0fdf60 2024-02-28T16:51:24.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25d2000700 1 -- 172.21.15.1:0/2193520526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25cc00ce60 con 0x7f25dc0fdf60 2024-02-28T16:51:24.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25d2000700 1 -- 172.21.15.1:0/2193520526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25cc0115e0 con 0x7f25dc0fdf60 2024-02-28T16:51:24.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.323+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc119d70 con 0x7f25dc0fdf60 2024-02-28T16:51:24.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.324+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc11a250 con 0x7f25dc0fdf60 2024-02-28T16:51:24.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.325+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc104550 con 0x7f25dc0fdf60 2024-02-28T16:51:24.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.326+0000 7f25d2000700 1 -- 172.21.15.1:0/2193520526 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f25cc012400 con 0x7f25dc0fdf60 2024-02-28T16:51:24.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.327+0000 7f25d2000700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25c006d7e0 0x7f25c006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:24.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.327+0000 7f25d2000700 1 -- 172.21.15.1:0/2193520526 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f25cc08ce70 con 0x7f25dc0fdf60 2024-02-28T16:51:24.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.327+0000 7f25d9400700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25c006d7e0 0x7f25c006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:24.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.328+0000 7f25d9400700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25c006d7e0 0x7f25c006fc90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b3b0 tx=0x7f25c8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:24.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.333+0000 7f25d2000700 1 -- 172.21.15.1:0/2193520526 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f25cc0567e0 con 0x7f25dc0fdf60 2024-02-28T16:51:24.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:24 smithi001 ceph-mon[112940]: from='client.34954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:24.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:24 smithi001 ceph-mon[112940]: pgmap v508: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:24.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:24 smithi001 ceph-mon[112940]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:24.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:24 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1217171593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:51:24.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:24 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3970311339' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:51:24.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.579+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25dc04bed0 con 0x7f25c006d7e0 2024-02-28T16:51:24.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.582+0000 7f25d2000700 1 -- 172.21.15.1:0/2193520526 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f25dc04bed0 con 0x7f25c006d7e0 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:51:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:51:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:51:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:51:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:51:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:51:24.576 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:51:24.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25c006d7e0 msgr2=0x7f25c006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:24.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25c006d7e0 0x7f25c006fc90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b3b0 tx=0x7f25c8009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:24.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fdf60 msgr2=0x7f25dc0fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:24.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fdf60 0x7f25dc0fbd90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00c8c0 tx=0x7f25cc0044c0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 shutdown_connections 2024-02-28T16:51:24.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25c006d7e0 0x7f25c006fc90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25dc0fdf60 0x7f25dc0fbd90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 --2- 172.21.15.1:0/2193520526 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25dc0fe900 0x7f25dc0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 >> 172.21.15.1:0/2193520526 conn(0x7f25dc0f7a10 msgr2=0x7f25dc1016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:24.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.585+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 shutdown_connections 2024-02-28T16:51:24.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.586+0000 7f25e1200700 1 -- 172.21.15.1:0/2193520526 wait complete. 2024-02-28T16:51:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.748+0000 7f5dad600700 1 -- 172.21.15.1:0/3479044719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da8102f50 msgr2=0x7f5da8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.748+0000 7f5dad600700 1 --2- 172.21.15.1:0/3479044719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da8102f50 0x7f5da8103320 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5d98004940 tx=0x7f5d98009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.748+0000 7f5dad600700 1 -- 172.21.15.1:0/3479044719 shutdown_connections 2024-02-28T16:51:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.748+0000 7f5dad600700 1 --2- 172.21.15.1:0/3479044719 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da81038f0 0x7f5da80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.748+0000 7f5dad600700 1 --2- 172.21.15.1:0/3479044719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da8102f50 0x7f5da8103320 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.748+0000 7f5dad600700 1 -- 172.21.15.1:0/3479044719 >> 172.21.15.1:0/3479044719 conn(0x7f5da80f8ae0 msgr2=0x7f5da80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.749+0000 7f5dad600700 1 -- 172.21.15.1:0/3479044719 shutdown_connections 2024-02-28T16:51:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.749+0000 7f5dad600700 1 -- 172.21.15.1:0/3479044719 wait complete. 2024-02-28T16:51:24.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.749+0000 7f5dad600700 1 Processor -- start 2024-02-28T16:51:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.750+0000 7f5dad600700 1 -- start start 2024-02-28T16:51:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.750+0000 7f5dad600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da8102f50 0x7f5da8070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.750+0000 7f5dad600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da81038f0 0x7f5da80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.750+0000 7f5dad600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8070dd0 con 0x7f5da81038f0 2024-02-28T16:51:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.750+0000 7f5dad600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8070f40 con 0x7f5da8102f50 2024-02-28T16:51:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.750+0000 7f5da6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da81038f0 0x7f5da80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.750+0000 7f5da6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da81038f0 0x7f5da80707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42828/0 (socket says 172.21.15.1:42828) 2024-02-28T16:51:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.750+0000 7f5da6000700 1 -- 172.21.15.1:0/3310746731 learned_addr learned my addr 172.21.15.1:0/3310746731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.751+0000 7f5da6a00700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da8102f50 0x7f5da8070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.751+0000 7f5da6000700 1 -- 172.21.15.1:0/3310746731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da8102f50 msgr2=0x7f5da8070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.751+0000 7f5da6000700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da8102f50 0x7f5da8070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.751+0000 7f5da6000700 1 -- 172.21.15.1:0/3310746731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d98005040 con 0x7f5da81038f0 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.751+0000 7f5da6000700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da81038f0 0x7f5da80707d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5d94002a50 tx=0x7f5d94004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.752+0000 7f5d9ea00700 1 -- 172.21.15.1:0/3310746731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d94007cc0 con 0x7f5da81038f0 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.752+0000 7f5d9ea00700 1 -- 172.21.15.1:0/3310746731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d9400f040 con 0x7f5da81038f0 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.752+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da8119c90 con 0x7f5da81038f0 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.752+0000 7f5d9ea00700 1 -- 172.21.15.1:0/3310746731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d9400daf0 con 0x7f5da81038f0 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.752+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da811a170 con 0x7f5da81038f0 2024-02-28T16:51:24.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.753+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da80fc6e0 con 0x7f5da81038f0 2024-02-28T16:51:24.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.755+0000 7f5d9ea00700 1 -- 172.21.15.1:0/3310746731 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5d94009810 con 0x7f5da81038f0 2024-02-28T16:51:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.756+0000 7f5d9ea00700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7e0 0x7f5d8c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.756+0000 7f5d9ea00700 1 -- 172.21.15.1:0/3310746731 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5d9408ccd0 con 0x7f5da81038f0 2024-02-28T16:51:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.756+0000 7f5da6a00700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7e0 0x7f5d8c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.757+0000 7f5da6a00700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7e0 0x7f5d8c06fc90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5d98004f90 tx=0x7f5d98009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:24.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:24 smithi066 ceph-mon[85781]: from='client.34954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:24.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:24 smithi066 ceph-mon[85781]: pgmap v508: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:24.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:24 smithi066 ceph-mon[85781]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:24.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:24 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1217171593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:51:24.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:24 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3970311339' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:51:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:24.762+0000 7f5d9ea00700 1 -- 172.21.15.1:0/3310746731 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5d94056480 con 0x7f5da81038f0 2024-02-28T16:51:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.074+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5da811a360 con 0x7f5da81038f0 2024-02-28T16:51:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.075+0000 7f5d9ea00700 1 -- 172.21.15.1:0/3310746731 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f5d94059b40 con 0x7f5da81038f0 2024-02-28T16:51:25.068 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:51:25.068 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:51:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:51:25.068 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:51:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:51:25.068 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:51:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:51:25.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.077+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7e0 msgr2=0x7f5d8c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.077+0000 7f5dad600700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7e0 0x7f5d8c06fc90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5d98004f90 tx=0x7f5d98009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da81038f0 msgr2=0x7f5da80707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da81038f0 0x7f5da80707d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5d94002a50 tx=0x7f5d94004600 comp rx=0 tx=0).stop 2024-02-28T16:51:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 shutdown_connections 2024-02-28T16:51:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7e0 0x7f5d8c06fc90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da8102f50 0x7f5da8070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 --2- 172.21.15.1:0/3310746731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da81038f0 0x7f5da80707d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:25.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 >> 172.21.15.1:0/3310746731 conn(0x7f5da80f8ae0 msgr2=0x7f5da80fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 shutdown_connections 2024-02-28T16:51:25.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:25.078+0000 7f5dad600700 1 -- 172.21.15.1:0/3310746731 wait complete. 2024-02-28T16:51:25.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:25 smithi066 ceph-mon[85781]: from='client.34970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:25.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:25 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3310746731' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:51:25.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:25 smithi001 ceph-mon[112940]: from='client.34970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:25.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:25 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3310746731' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:51:26.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:26 smithi066 ceph-mon[85781]: pgmap v509: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:26.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:51:26.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:26 smithi001 ceph-mon[112940]: pgmap v509: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:26.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:51:28.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:28 smithi066 ceph-mon[85781]: pgmap v510: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:28.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:28 smithi001 ceph-mon[112940]: pgmap v510: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:30.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:30 smithi066 ceph-mon[85781]: pgmap v511: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:30.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:30 smithi001 ceph-mon[112940]: pgmap v511: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:32.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:32 smithi066 ceph-mon[85781]: pgmap v512: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:32.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:32 smithi001 ceph-mon[112940]: pgmap v512: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:34.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:34 smithi066 ceph-mon[85781]: pgmap v513: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:34.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:34 smithi001 ceph-mon[112940]: pgmap v513: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:36.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:36 smithi066 ceph-mon[85781]: pgmap v514: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:36.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:36 smithi001 ceph-mon[112940]: pgmap v514: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:38.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:38 smithi066 ceph-mon[85781]: pgmap v515: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:38.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:38 smithi001 ceph-mon[112940]: pgmap v515: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:40.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:40 smithi066 ceph-mon[85781]: pgmap v516: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:40.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:40 smithi001 ceph-mon[112940]: pgmap v516: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:41.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:51:41.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:51:42.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:42 smithi066 ceph-mon[85781]: pgmap v517: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:42.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:42 smithi001 ceph-mon[112940]: pgmap v517: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:44.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:44 smithi066 ceph-mon[85781]: pgmap v518: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:44.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:44 smithi001 ceph-mon[112940]: pgmap v518: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:46.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:46 smithi066 ceph-mon[85781]: pgmap v519: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:46.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:46 smithi001 ceph-mon[112940]: pgmap v519: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:48.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:48 smithi066 ceph-mon[85781]: pgmap v520: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:48.775 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:48 smithi001 ceph-mon[112940]: pgmap v520: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:50.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:50 smithi066 ceph-mon[85781]: pgmap v521: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:50.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:50 smithi001 ceph-mon[112940]: pgmap v521: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:52.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:52 smithi066 ceph-mon[85781]: pgmap v522: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:52.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:52 smithi001 ceph-mon[112940]: pgmap v522: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:54.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:54 smithi066 ceph-mon[85781]: pgmap v523: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:54.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:54 smithi001 ceph-mon[112940]: pgmap v523: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:55.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.249+0000 7fdf5be00700 1 -- 172.21.15.1:0/1087623176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c072dd0 msgr2=0x7fdf5c10d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:55.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.249+0000 7fdf5be00700 1 --2- 172.21.15.1:0/1087623176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c072dd0 0x7fdf5c10d700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdf44004940 tx=0x7fdf44009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:55.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.249+0000 7fdf5be00700 1 -- 172.21.15.1:0/1087623176 shutdown_connections 2024-02-28T16:51:55.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.249+0000 7fdf5be00700 1 --2- 172.21.15.1:0/1087623176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c072dd0 0x7fdf5c10d700 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.249+0000 7fdf5be00700 1 --2- 172.21.15.1:0/1087623176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf5c0724c0 0x7fdf5c072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.249+0000 7fdf5be00700 1 -- 172.21.15.1:0/1087623176 >> 172.21.15.1:0/1087623176 conn(0x7fdf5c0fab40 msgr2=0x7fdf5c0fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:55.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.249+0000 7fdf5be00700 1 -- 172.21.15.1:0/1087623176 shutdown_connections 2024-02-28T16:51:55.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.249+0000 7fdf5be00700 1 -- 172.21.15.1:0/1087623176 wait complete. 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.250+0000 7fdf5be00700 1 Processor -- start 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.250+0000 7fdf5be00700 1 -- start start 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c0724c0 0x7fdf5c119d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5be00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf5c072dd0 0x7fdf5c11a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c11a8e0 con 0x7fdf5c0724c0 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5be00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c113bf0 con 0x7fdf5c072dd0 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c0724c0 0x7fdf5c119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c0724c0 0x7fdf5c119d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59808/0 (socket says 172.21.15.1:59808) 2024-02-28T16:51:55.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5aa00700 1 -- 172.21.15.1:0/35662810 learned_addr learned my addr 172.21.15.1:0/35662810 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:55.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5aa00700 1 -- 172.21.15.1:0/35662810 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf5c072dd0 msgr2=0x7fdf5c11a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:55.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf52000700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf5c072dd0 0x7fdf5c11a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5aa00700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf5c072dd0 0x7fdf5c11a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.251+0000 7fdf5aa00700 1 -- 172.21.15.1:0/35662810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44005040 con 0x7fdf5c0724c0 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.252+0000 7fdf5aa00700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c0724c0 0x7fdf5c119d50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c002a50 tx=0x7fdf4c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.252+0000 7fdf53400700 1 -- 172.21.15.1:0/35662810 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf4c007cc0 con 0x7fdf5c0724c0 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.252+0000 7fdf53400700 1 -- 172.21.15.1:0/35662810 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf4c00f040 con 0x7fdf5c0724c0 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.252+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c113ed0 con 0x7fdf5c0724c0 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.252+0000 7fdf53400700 1 -- 172.21.15.1:0/35662810 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf4c009640 con 0x7fdf5c0724c0 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.252+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c114490 con 0x7fdf5c0724c0 2024-02-28T16:51:55.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.253+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c10ae30 con 0x7fdf5c0724c0 2024-02-28T16:51:55.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.254+0000 7fdf53400700 1 -- 172.21.15.1:0/35662810 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdf4c0097a0 con 0x7fdf5c0724c0 2024-02-28T16:51:55.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.255+0000 7fdf53400700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf4006d7f0 0x7fdf4006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:55.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.256+0000 7fdf53400700 1 -- 172.21.15.1:0/35662810 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdf4c08cd20 con 0x7fdf5c0724c0 2024-02-28T16:51:55.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.256+0000 7fdf52000700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf4006d7f0 0x7fdf4006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:55.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.257+0000 7fdf52000700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf4006d7f0 0x7fdf4006fca0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdf44000c00 tx=0x7fdf44009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:55.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.261+0000 7fdf53400700 1 -- 172.21.15.1:0/35662810 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdf4c05a000 con 0x7fdf5c0724c0 2024-02-28T16:51:55.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.507+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf5c115330 con 0x7fdf4006d7f0 2024-02-28T16:51:55.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.509+0000 7fdf53400700 1 -- 172.21.15.1:0/35662810 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdf5c115330 con 0x7fdf4006d7f0 2024-02-28T16:51:55.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf4006d7f0 msgr2=0x7fdf4006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:55.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf4006d7f0 0x7fdf4006fca0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdf44000c00 tx=0x7fdf44009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:55.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c0724c0 msgr2=0x7fdf5c119d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:55.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c0724c0 0x7fdf5c119d50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c002a50 tx=0x7fdf4c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 shutdown_connections 2024-02-28T16:51:55.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf4006d7f0 0x7fdf4006fca0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5c0724c0 0x7fdf5c119d50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 --2- 172.21.15.1:0/35662810 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf5c072dd0 0x7fdf5c11a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 >> 172.21.15.1:0/35662810 conn(0x7fdf5c0fab40 msgr2=0x7fdf5c0fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 shutdown_connections 2024-02-28T16:51:55.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.512+0000 7fdf5be00700 1 -- 172.21.15.1:0/35662810 wait complete. 2024-02-28T16:51:55.522 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:51:55.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.664+0000 7fc42ec00700 1 -- 172.21.15.1:0/1841432067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fbdd0 msgr2=0x7fc4280fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.664+0000 7fc42ec00700 1 --2- 172.21.15.1:0/1841432067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fbdd0 0x7fc4280fc1a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc418004940 tx=0x7fc418009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.664+0000 7fc42ec00700 1 -- 172.21.15.1:0/1841432067 shutdown_connections 2024-02-28T16:51:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.664+0000 7fc42ec00700 1 --2- 172.21.15.1:0/1841432067 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4280fc6e0 0x7fc428109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.664+0000 7fc42ec00700 1 --2- 172.21.15.1:0/1841432067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fbdd0 0x7fc4280fc1a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.664+0000 7fc42ec00700 1 -- 172.21.15.1:0/1841432067 >> 172.21.15.1:0/1841432067 conn(0x7fc428071d90 msgr2=0x7fc428072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.664+0000 7fc42ec00700 1 -- 172.21.15.1:0/1841432067 shutdown_connections 2024-02-28T16:51:55.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.664+0000 7fc42ec00700 1 -- 172.21.15.1:0/1841432067 wait complete. 2024-02-28T16:51:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.665+0000 7fc42ec00700 1 Processor -- start 2024-02-28T16:51:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.665+0000 7fc42ec00700 1 -- start start 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.666+0000 7fc42ec00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4280fbdd0 0x7fc428115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.666+0000 7fc42ec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fc6e0 0x7fc428116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.666+0000 7fc42ec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc428116860 con 0x7fc4280fc6e0 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.666+0000 7fc42ec00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42810fb70 con 0x7fc4280fbdd0 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.666+0000 7fc427400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fc6e0 0x7fc428116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.666+0000 7fc427400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fc6e0 0x7fc428116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59822/0 (socket says 172.21.15.1:59822) 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.666+0000 7fc427400700 1 -- 172.21.15.1:0/3237920882 learned_addr learned my addr 172.21.15.1:0/3237920882 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.666+0000 7fc427e00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4280fbdd0 0x7fc428115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.667+0000 7fc427400700 1 -- 172.21.15.1:0/3237920882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4280fbdd0 msgr2=0x7fc428115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.667+0000 7fc427400700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4280fbdd0 0x7fc428115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.667+0000 7fc427400700 1 -- 172.21.15.1:0/3237920882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc418005040 con 0x7fc4280fc6e0 2024-02-28T16:51:55.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.667+0000 7fc427400700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fc6e0 0x7fc428116210 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc41400c8c0 tx=0x7fc4140044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:55.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.667+0000 7fc41fe00700 1 -- 172.21.15.1:0/3237920882 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc41400a980 con 0x7fc4280fc6e0 2024-02-28T16:51:55.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.667+0000 7fc41fe00700 1 -- 172.21.15.1:0/3237920882 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc41400ce60 con 0x7fc4280fc6e0 2024-02-28T16:51:55.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.667+0000 7fc41fe00700 1 -- 172.21.15.1:0/3237920882 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4140115e0 con 0x7fc4280fc6e0 2024-02-28T16:51:55.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.667+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42810fe50 con 0x7fc4280fc6e0 2024-02-28T16:51:55.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.668+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc428110390 con 0x7fc4280fc6e0 2024-02-28T16:51:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.669+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc428106b70 con 0x7fc4280fc6e0 2024-02-28T16:51:55.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.670+0000 7fc41fe00700 1 -- 172.21.15.1:0/3237920882 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc414012400 con 0x7fc4280fc6e0 2024-02-28T16:51:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.672+0000 7fc41fe00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40c06d7f0 0x7fc40c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.672+0000 7fc41fe00700 1 -- 172.21.15.1:0/3237920882 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc41408cfa0 con 0x7fc4280fc6e0 2024-02-28T16:51:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.672+0000 7fc427e00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40c06d7f0 0x7fc40c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:55.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.673+0000 7fc427e00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40c06d7f0 0x7fc40c06fca0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc41800b3b0 tx=0x7fc418009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:55.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.677+0000 7fc41fe00700 1 -- 172.21.15.1:0/3237920882 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc41405a3a0 con 0x7fc4280fc6e0 2024-02-28T16:51:55.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.917+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc42804bed0 con 0x7fc40c06d7f0 2024-02-28T16:51:55.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.920+0000 7fc41fe00700 1 -- 172.21.15.1:0/3237920882 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc42804bed0 con 0x7fc40c06d7f0 2024-02-28T16:51:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40c06d7f0 msgr2=0x7fc40c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40c06d7f0 0x7fc40c06fca0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc41800b3b0 tx=0x7fc418009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fc6e0 msgr2=0x7fc428116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fc6e0 0x7fc428116210 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc41400c8c0 tx=0x7fc4140044c0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 shutdown_connections 2024-02-28T16:51:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40c06d7f0 0x7fc40c06fca0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc4280fbdd0 0x7fc428115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 --2- 172.21.15.1:0/3237920882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4280fc6e0 0x7fc428116210 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 >> 172.21.15.1:0/3237920882 conn(0x7fc428071d90 msgr2=0x7fc4280faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.922+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 shutdown_connections 2024-02-28T16:51:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:55.923+0000 7fc42ec00700 1 -- 172.21.15.1:0/3237920882 wait complete. 2024-02-28T16:51:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.088+0000 7fbe93000700 1 -- 172.21.15.1:0/1676375949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 msgr2=0x7fbe8c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.088+0000 7fbe93000700 1 --2- 172.21.15.1:0/1676375949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 0x7fbe8c0fcf60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbe78004940 tx=0x7fbe78009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:56.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.088+0000 7fbe93000700 1 -- 172.21.15.1:0/1676375949 shutdown_connections 2024-02-28T16:51:56.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.088+0000 7fbe93000700 1 --2- 172.21.15.1:0/1676375949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 0x7fbe8c0fcf60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.088+0000 7fbe93000700 1 --2- 172.21.15.1:0/1676375949 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe8c102f50 0x7fbe8c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.088+0000 7fbe93000700 1 -- 172.21.15.1:0/1676375949 >> 172.21.15.1:0/1676375949 conn(0x7fbe8c0f8ae0 msgr2=0x7fbe8c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:56.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.088+0000 7fbe93000700 1 -- 172.21.15.1:0/1676375949 shutdown_connections 2024-02-28T16:51:56.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.089+0000 7fbe93000700 1 -- 172.21.15.1:0/1676375949 wait complete. 2024-02-28T16:51:56.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.089+0000 7fbe93000700 1 Processor -- start 2024-02-28T16:51:56.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.090+0000 7fbe93000700 1 -- start start 2024-02-28T16:51:56.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.090+0000 7fbe93000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe8c102f50 0x7fbe8c115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.090+0000 7fbe93000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 0x7fbe8c1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.090+0000 7fbe93000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c1168d0 con 0x7fbe8c1038f0 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.090+0000 7fbe93000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c10fb50 con 0x7fbe8c102f50 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.090+0000 7fbe8b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 0x7fbe8c1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.090+0000 7fbe8b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 0x7fbe8c1161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59840/0 (socket says 172.21.15.1:59840) 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.090+0000 7fbe8b400700 1 -- 172.21.15.1:0/614444430 learned_addr learned my addr 172.21.15.1:0/614444430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe8be00700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe8c102f50 0x7fbe8c115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe8b400700 1 -- 172.21.15.1:0/614444430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe8c102f50 msgr2=0x7fbe8c115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe8b400700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe8c102f50 0x7fbe8c115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe8b400700 1 -- 172.21.15.1:0/614444430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe78005040 con 0x7fbe8c1038f0 2024-02-28T16:51:56.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe8b400700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 0x7fbe8c1161f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009100 tx=0x7fbe78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:56.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe83e00700 1 -- 172.21.15.1:0/614444430 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe7800e040 con 0x7fbe8c1038f0 2024-02-28T16:51:56.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe83e00700 1 -- 172.21.15.1:0/614444430 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe78004290 con 0x7fbe8c1038f0 2024-02-28T16:51:56.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe83e00700 1 -- 172.21.15.1:0/614444430 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe7801c3f0 con 0x7fbe8c1038f0 2024-02-28T16:51:56.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.091+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8c10fd70 con 0x7fbe8c1038f0 2024-02-28T16:51:56.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.092+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8c1102b0 con 0x7fbe8c1038f0 2024-02-28T16:51:56.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.092+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8c0fc6e0 con 0x7fbe8c1038f0 2024-02-28T16:51:56.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.094+0000 7fbe83e00700 1 -- 172.21.15.1:0/614444430 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbe7800bbf0 con 0x7fbe8c1038f0 2024-02-28T16:51:56.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.094+0000 7fbe83e00700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe7006d7f0 0x7fbe7006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.095+0000 7fbe83e00700 1 -- 172.21.15.1:0/614444430 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbe7808e1c0 con 0x7fbe8c1038f0 2024-02-28T16:51:56.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.095+0000 7fbe8be00700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe7006d7f0 0x7fbe7006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:56.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.095+0000 7fbe8be00700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe7006d7f0 0x7fbe7006fca0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:56.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.099+0000 7fbe83e00700 1 -- 172.21.15.1:0/614444430 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe7805b5a0 con 0x7fbe8c1038f0 2024-02-28T16:51:56.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.332+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe8c1113a0 con 0x7fbe7006d7f0 2024-02-28T16:51:56.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.353+0000 7fbe83e00700 1 -- 172.21.15.1:0/614444430 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbe8c1113a0 con 0x7fbe7006d7f0 2024-02-28T16:51:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:51:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (18m) 10m ago 32m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:51:56.347 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (14m) 10m ago 32m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (14m) 11m ago 31m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (17m) 10m ago 31m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (10m) 10m ago 25m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (10m) 10m ago 25m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (25m) 11m ago 24m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (25m) 11m ago 25m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (20m) 10m ago 34m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (19m) 11m ago 31m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (16m) 10m ago 34m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (15m) 11m ago 30m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (19m) 10m ago 32m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (19m) 11m ago 31m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (14m) 10m ago 29m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:51:56.348 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (13m) 10m ago 29m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:51:56.349 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (13m) 10m ago 28m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:51:56.349 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (12m) 11m ago 28m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:51:56.349 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (11m) 11m ago 27m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:51:56.349 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (11m) 11m ago 27m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:51:56.349 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (18m) 10m ago 31m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:51:56.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe7006d7f0 msgr2=0x7fbe7006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe7006d7f0 0x7fbe7006fca0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c008040 comp rx=0 tx=0).stop 2024-02-28T16:51:56.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 msgr2=0x7fbe8c1161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 0x7fbe8c1161f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009100 tx=0x7fbe78009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:56.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 shutdown_connections 2024-02-28T16:51:56.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe7006d7f0 0x7fbe7006fca0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe8c102f50 0x7fbe8c115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 --2- 172.21.15.1:0/614444430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe8c1038f0 0x7fbe8c1161f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 >> 172.21.15.1:0/614444430 conn(0x7fbe8c0f8ae0 msgr2=0x7fbe8c0fa380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:56.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 shutdown_connections 2024-02-28T16:51:56.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.356+0000 7fbe93000700 1 -- 172.21.15.1:0/614444430 wait complete. 2024-02-28T16:51:56.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.499+0000 7fb412c00700 1 -- 172.21.15.1:0/1199918818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c1095f0 msgr2=0x7fb40c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.499+0000 7fb412c00700 1 --2- 172.21.15.1:0/1199918818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c1095f0 0x7fb40c1099c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc004940 tx=0x7fb3fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:56.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.499+0000 7fb412c00700 1 -- 172.21.15.1:0/1199918818 shutdown_connections 2024-02-28T16:51:56.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.499+0000 7fb412c00700 1 --2- 172.21.15.1:0/1199918818 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb40c103dc0 0x7fb40c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.499+0000 7fb412c00700 1 --2- 172.21.15.1:0/1199918818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c1095f0 0x7fb40c1099c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.499+0000 7fb412c00700 1 -- 172.21.15.1:0/1199918818 >> 172.21.15.1:0/1199918818 conn(0x7fb40c072d50 msgr2=0x7fb40c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.499+0000 7fb412c00700 1 -- 172.21.15.1:0/1199918818 shutdown_connections 2024-02-28T16:51:56.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.499+0000 7fb412c00700 1 -- 172.21.15.1:0/1199918818 wait complete. 2024-02-28T16:51:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.500+0000 7fb412c00700 1 Processor -- start 2024-02-28T16:51:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.500+0000 7fb412c00700 1 -- start start 2024-02-28T16:51:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.500+0000 7fb412c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c103dc0 0x7fb40c11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.500+0000 7fb412c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb40c1095f0 0x7fb40c11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.501+0000 7fb412c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c11ac70 con 0x7fb40c103dc0 2024-02-28T16:51:56.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.501+0000 7fb412c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c113f80 con 0x7fb40c1095f0 2024-02-28T16:51:56.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.501+0000 7fb40be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c103dc0 0x7fb40c11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:56.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.501+0000 7fb40be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c103dc0 0x7fb40c11a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59862/0 (socket says 172.21.15.1:59862) 2024-02-28T16:51:56.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.501+0000 7fb40be00700 1 -- 172.21.15.1:0/2705623778 learned_addr learned my addr 172.21.15.1:0/2705623778 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:56.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.501+0000 7fb40be00700 1 -- 172.21.15.1:0/2705623778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb40c1095f0 msgr2=0x7fb40c11a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.501+0000 7fb40be00700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb40c1095f0 0x7fb40c11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.501+0000 7fb40be00700 1 -- 172.21.15.1:0/2705623778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc005040 con 0x7fb40c103dc0 2024-02-28T16:51:56.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.502+0000 7fb40be00700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c103dc0 0x7fb40c11a0e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009fd0 tx=0x7fb3fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:56.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.502+0000 7fb403e00700 1 -- 172.21.15.1:0/2705623778 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3fc00e040 con 0x7fb40c103dc0 2024-02-28T16:51:56.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.502+0000 7fb403e00700 1 -- 172.21.15.1:0/2705623778 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3fc00bbf0 con 0x7fb40c103dc0 2024-02-28T16:51:56.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.502+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c114200 con 0x7fb40c103dc0 2024-02-28T16:51:56.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.502+0000 7fb403e00700 1 -- 172.21.15.1:0/2705623778 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3fc01c3f0 con 0x7fb40c103dc0 2024-02-28T16:51:56.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.502+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c114740 con 0x7fb40c103dc0 2024-02-28T16:51:56.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.503+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c106850 con 0x7fb40c103dc0 2024-02-28T16:51:56.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.504+0000 7fb403e00700 1 -- 172.21.15.1:0/2705623778 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb3fc00bd60 con 0x7fb40c103dc0 2024-02-28T16:51:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.505+0000 7fb403e00700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3f006d7f0 0x7fb3f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.505+0000 7fb403e00700 1 -- 172.21.15.1:0/2705623778 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb3fc08e2b0 con 0x7fb40c103dc0 2024-02-28T16:51:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.505+0000 7fb40b400700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3f006d7f0 0x7fb3f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:56.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.506+0000 7fb40b400700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3f006d7f0 0x7fb3f006fca0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb40c115980 tx=0x7fb3f8009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:56.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.509+0000 7fb403e00700 1 -- 172.21.15.1:0/2705623778 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb3fc05b610 con 0x7fb40c103dc0 2024-02-28T16:51:56.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:56 smithi066 ceph-mon[85781]: pgmap v524: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:56.753 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:56 smithi066 ceph-mon[85781]: from='client.34978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:56.753 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:51:56.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:56 smithi001 ceph-mon[112940]: pgmap v524: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:56.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:56 smithi001 ceph-mon[112940]: from='client.34978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:56.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:51:56.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.831+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb40c1156c0 con 0x7fb40c103dc0 2024-02-28T16:51:56.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.832+0000 7fb403e00700 1 -- 172.21.15.1:0/2705623778 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fb3fc05b1a0 con 0x7fb40c103dc0 2024-02-28T16:51:56.826 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:51:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:51:56.826 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:51:56.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:51:56.828 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.834+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3f006d7f0 msgr2=0x7fb3f006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3f006d7f0 0x7fb3f006fca0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb40c115980 tx=0x7fb3f8009ce0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c103dc0 msgr2=0x7fb40c11a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c103dc0 0x7fb40c11a0e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009fd0 tx=0x7fb3fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 shutdown_connections 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3f006d7f0 0x7fb3f006fca0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40c103dc0 0x7fb40c11a0e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 --2- 172.21.15.1:0/2705623778 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb40c1095f0 0x7fb40c11a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 >> 172.21.15.1:0/2705623778 conn(0x7fb40c072d50 msgr2=0x7fb40c074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 shutdown_connections 2024-02-28T16:51:56.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.835+0000 7fb412c00700 1 -- 172.21.15.1:0/2705623778 wait complete. 2024-02-28T16:51:56.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.987+0000 7f75cec00700 1 -- 172.21.15.1:0/87867097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 msgr2=0x7f75c80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.987+0000 7f75cec00700 1 --2- 172.21.15.1:0/87867097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fe3b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f75b8004970 tx=0x7f75b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:56.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.987+0000 7f75cec00700 1 -- 172.21.15.1:0/87867097 shutdown_connections 2024-02-28T16:51:56.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.987+0000 7f75cec00700 1 --2- 172.21.15.1:0/87867097 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c8107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.987+0000 7f75cec00700 1 --2- 172.21.15.1:0/87867097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c80fe3b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.987+0000 7f75cec00700 1 -- 172.21.15.1:0/87867097 >> 172.21.15.1:0/87867097 conn(0x7f75c8071d90 msgr2=0x7f75c8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:56.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.987+0000 7f75cec00700 1 -- 172.21.15.1:0/87867097 shutdown_connections 2024-02-28T16:51:56.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.987+0000 7f75cec00700 1 -- 172.21.15.1:0/87867097 wait complete. 2024-02-28T16:51:56.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.988+0000 7f75cec00700 1 Processor -- start 2024-02-28T16:51:56.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.988+0000 7f75cec00700 1 -- start start 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75cec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c8115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75cec00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c8116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75cec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c8116900 con 0x7f75c80fdfe0 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75cec00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c810fb80 con 0x7f75c80fe8f0 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75c7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c8115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75c7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c8115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59884/0 (socket says 172.21.15.1:59884) 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75c7e00700 1 -- 172.21.15.1:0/3495709529 learned_addr learned my addr 172.21.15.1:0/3495709529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75c7400700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c8116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75c7e00700 1 -- 172.21.15.1:0/3495709529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 msgr2=0x7f75c8116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75c7e00700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c8116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75c7e00700 1 -- 172.21.15.1:0/3495709529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8005040 con 0x7f75c80fdfe0 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.989+0000 7f75c7400700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c8116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:51:56.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.990+0000 7f75c7e00700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c8115ce0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009150 tx=0x7f75b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:56.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.990+0000 7f75bfe00700 1 -- 172.21.15.1:0/3495709529 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b800e040 con 0x7f75c80fdfe0 2024-02-28T16:51:56.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.990+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c810fda0 con 0x7f75c80fdfe0 2024-02-28T16:51:56.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.990+0000 7f75bfe00700 1 -- 172.21.15.1:0/3495709529 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75b8004290 con 0x7f75c80fdfe0 2024-02-28T16:51:56.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.990+0000 7f75bfe00700 1 -- 172.21.15.1:0/3495709529 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b801c630 con 0x7f75c80fdfe0 2024-02-28T16:51:56.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.990+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c81102e0 con 0x7f75c80fdfe0 2024-02-28T16:51:56.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.991+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c8106b20 con 0x7f75c80fdfe0 2024-02-28T16:51:56.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.993+0000 7f75bfe00700 1 -- 172.21.15.1:0/3495709529 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f75b8013980 con 0x7f75c80fdfe0 2024-02-28T16:51:56.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.993+0000 7f75bfe00700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d7f0 0x7f75ac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:56.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.994+0000 7f75bfe00700 1 -- 172.21.15.1:0/3495709529 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f75b808e230 con 0x7f75c80fdfe0 2024-02-28T16:51:56.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.994+0000 7f75c7400700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d7f0 0x7f75ac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:56.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.994+0000 7f75c7400700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d7f0 0x7f75ac06fca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f75b4006fd0 tx=0x7f75b4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:56.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:56.998+0000 7f75bfe00700 1 -- 172.21.15.1:0/3495709529 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f75b805b590 con 0x7f75c80fdfe0 2024-02-28T16:51:57.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.267+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f75c81112c0 con 0x7f75c80fdfe0 2024-02-28T16:51:57.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.267+0000 7f75bfe00700 1 -- 172.21.15.1:0/3495709529 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f75b805b120 con 0x7f75c80fdfe0 2024-02-28T16:51:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:51:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:51:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:51:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:51:57.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:51:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:51:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:51:57.262 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:51:57.263 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:51:57.264 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:51:57.265 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:51:57.265 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:51:57.265 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:51:57.265 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:51:57.265 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:51:57.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.270+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d7f0 msgr2=0x7f75ac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.270+0000 7f75cec00700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d7f0 0x7f75ac06fca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f75b4006fd0 tx=0x7f75b4008040 comp rx=0 tx=0).stop 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.270+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 msgr2=0x7f75c8115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.270+0000 7f75cec00700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c8115ce0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009150 tx=0x7f75b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.271+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 shutdown_connections 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.271+0000 7f75cec00700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75ac06d7f0 0x7f75ac06fca0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.271+0000 7f75cec00700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75c80fdfe0 0x7f75c8115ce0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.271+0000 7f75cec00700 1 --2- 172.21.15.1:0/3495709529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75c80fe8f0 0x7f75c8116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.271+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 >> 172.21.15.1:0/3495709529 conn(0x7f75c8071d90 msgr2=0x7f75c80fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.271+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 shutdown_connections 2024-02-28T16:51:57.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.271+0000 7f75cec00700 1 -- 172.21.15.1:0/3495709529 wait complete. 2024-02-28T16:51:57.267 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:51:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.419+0000 7f1b8f000700 1 -- 172.21.15.1:0/700880022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 msgr2=0x7f1b88106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.419+0000 7f1b8f000700 1 --2- 172.21.15.1:0/700880022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 0x7f1b88106f30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1b74004940 tx=0x7f1b74009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.419+0000 7f1b8f000700 1 -- 172.21.15.1:0/700880022 shutdown_connections 2024-02-28T16:51:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.419+0000 7f1b8f000700 1 --2- 172.21.15.1:0/700880022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 0x7f1b88106f30 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.419+0000 7f1b8f000700 1 --2- 172.21.15.1:0/700880022 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b880fdf50 0x7f1b880fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.419+0000 7f1b8f000700 1 -- 172.21.15.1:0/700880022 >> 172.21.15.1:0/700880022 conn(0x7f1b880f79e0 msgr2=0x7f1b880f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.419+0000 7f1b8f000700 1 -- 172.21.15.1:0/700880022 shutdown_connections 2024-02-28T16:51:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.419+0000 7f1b8f000700 1 -- 172.21.15.1:0/700880022 wait complete. 2024-02-28T16:51:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.420+0000 7f1b8f000700 1 Processor -- start 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.420+0000 7f1b8f000700 1 -- start start 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.420+0000 7f1b8f000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b880fdf50 0x7f1b88113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.420+0000 7f1b8f000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 0x7f1b88114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.421+0000 7f1b8f000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88114670 con 0x7f1b880fe8f0 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.421+0000 7f1b8f000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8810d9a0 con 0x7f1b880fdf50 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.421+0000 7f1b87400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 0x7f1b88114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.421+0000 7f1b87400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 0x7f1b88114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59906/0 (socket says 172.21.15.1:59906) 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.421+0000 7f1b87400700 1 -- 172.21.15.1:0/2931579109 learned_addr learned my addr 172.21.15.1:0/2931579109 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.421+0000 7f1b87400700 1 -- 172.21.15.1:0/2931579109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b880fdf50 msgr2=0x7f1b88113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.421+0000 7f1b87400700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b880fdf50 0x7f1b88113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.421+0000 7f1b87400700 1 -- 172.21.15.1:0/2931579109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b74005040 con 0x7f1b880fe8f0 2024-02-28T16:51:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.422+0000 7f1b87400700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 0x7f1b88114020 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009180 tx=0x7f1b74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.422+0000 7f1b7fe00700 1 -- 172.21.15.1:0/2931579109 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b7400e040 con 0x7f1b880fe8f0 2024-02-28T16:51:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.422+0000 7f1b7fe00700 1 -- 172.21.15.1:0/2931579109 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b74004290 con 0x7f1b880fe8f0 2024-02-28T16:51:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.422+0000 7f1b7fe00700 1 -- 172.21.15.1:0/2931579109 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b7401c630 con 0x7f1b880fe8f0 2024-02-28T16:51:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.422+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8810dc20 con 0x7f1b880fe8f0 2024-02-28T16:51:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.422+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8810e160 con 0x7f1b880fe8f0 2024-02-28T16:51:57.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.423+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b88104540 con 0x7f1b880fe8f0 2024-02-28T16:51:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.424+0000 7f1b7fe00700 1 -- 172.21.15.1:0/2931579109 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1b740139f0 con 0x7f1b880fe8f0 2024-02-28T16:51:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.425+0000 7f1b7fe00700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b6c06d7f0 0x7f1b6c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.425+0000 7f1b7fe00700 1 -- 172.21.15.1:0/2931579109 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1b7408e2f0 con 0x7f1b880fe8f0 2024-02-28T16:51:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.425+0000 7f1b87e00700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b6c06d7f0 0x7f1b6c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.426+0000 7f1b87e00700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b6c06d7f0 0x7f1b6c06fca0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1b78004ad0 tx=0x7f1b78007d40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.429+0000 7f1b7fe00700 1 -- 172.21.15.1:0/2931579109 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b7405b6d0 con 0x7f1b880fe8f0 2024-02-28T16:51:57.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:57 smithi001 ceph-mon[112940]: from='client.34982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:57.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:57 smithi001 ceph-mon[112940]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:57.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:57 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2705623778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:51:57.524 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:57 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3495709529' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:51:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.667+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b88109fe0 con 0x7f1b6c06d7f0 2024-02-28T16:51:57.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.670+0000 7f1b7fe00700 1 -- 172.21.15.1:0/2931579109 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1b88109fe0 con 0x7f1b6c06d7f0 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:51:57.664 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:51:57.665 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:51:57.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b6c06d7f0 msgr2=0x7f1b6c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b6c06d7f0 0x7f1b6c06fca0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1b78004ad0 tx=0x7f1b78007d40 comp rx=0 tx=0).stop 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 msgr2=0x7f1b88114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 0x7f1b88114020 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009180 tx=0x7f1b74009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 shutdown_connections 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b6c06d7f0 0x7f1b6c06fca0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b880fdf50 0x7f1b88113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 --2- 172.21.15.1:0/2931579109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b880fe8f0 0x7f1b88114020 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.672+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 >> 172.21.15.1:0/2931579109 conn(0x7f1b880f79e0 msgr2=0x7f1b881016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.673+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 shutdown_connections 2024-02-28T16:51:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.673+0000 7f1b8f000700 1 -- 172.21.15.1:0/2931579109 wait complete. 2024-02-28T16:51:57.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:57 smithi066 ceph-mon[85781]: from='client.34982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:57.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:57 smithi066 ceph-mon[85781]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:57.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:57 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2705623778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:51:57.753 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:57 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3495709529' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:51:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.829+0000 7f9a89200700 1 -- 172.21.15.1:0/3309008060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a84070be0 msgr2=0x7f9a841092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.829+0000 7f9a89200700 1 --2- 172.21.15.1:0/3309008060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a84070be0 0x7f9a841092b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9a70004970 tx=0x7f9a70009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.829+0000 7f9a89200700 1 -- 172.21.15.1:0/3309008060 shutdown_connections 2024-02-28T16:51:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.829+0000 7f9a89200700 1 --2- 172.21.15.1:0/3309008060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a84070be0 0x7f9a841092b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.829+0000 7f9a89200700 1 --2- 172.21.15.1:0/3309008060 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a840702d0 0x7f9a840706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.829+0000 7f9a89200700 1 -- 172.21.15.1:0/3309008060 >> 172.21.15.1:0/3309008060 conn(0x7f9a840f8af0 msgr2=0x7f9a840faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.830+0000 7f9a89200700 1 -- 172.21.15.1:0/3309008060 shutdown_connections 2024-02-28T16:51:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.830+0000 7f9a89200700 1 -- 172.21.15.1:0/3309008060 wait complete. 2024-02-28T16:51:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.830+0000 7f9a89200700 1 Processor -- start 2024-02-28T16:51:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.831+0000 7f9a89200700 1 -- start start 2024-02-28T16:51:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.831+0000 7f9a89200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a840702d0 0x7f9a84115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.831+0000 7f9a89200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a84070be0 0x7f9a84116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.831+0000 7f9a89200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a84116780 con 0x7f9a840702d0 2024-02-28T16:51:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.831+0000 7f9a89200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8410fad0 con 0x7f9a84070be0 2024-02-28T16:51:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.831+0000 7f9a81e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a840702d0 0x7f9a84115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:57.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.831+0000 7f9a81e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a840702d0 0x7f9a84115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59916/0 (socket says 172.21.15.1:59916) 2024-02-28T16:51:57.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.832+0000 7f9a81e00700 1 -- 172.21.15.1:0/2042414976 learned_addr learned my addr 172.21.15.1:0/2042414976 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.832+0000 7f9a81400700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a84070be0 0x7f9a84116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.832+0000 7f9a81e00700 1 -- 172.21.15.1:0/2042414976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a84070be0 msgr2=0x7f9a84116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.832+0000 7f9a81e00700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a84070be0 0x7f9a84116130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.832+0000 7f9a81e00700 1 -- 172.21.15.1:0/2042414976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70005040 con 0x7f9a840702d0 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.832+0000 7f9a81400700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a84070be0 0x7f9a84116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.832+0000 7f9a81e00700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a840702d0 0x7f9a84115bf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9a74002a50 tx=0x7f9a740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.833+0000 7f9a7a000700 1 -- 172.21.15.1:0/2042414976 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a74007cc0 con 0x7f9a840702d0 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.833+0000 7f9a7a000700 1 -- 172.21.15.1:0/2042414976 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a7400f040 con 0x7f9a840702d0 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.833+0000 7f9a7a000700 1 -- 172.21.15.1:0/2042414976 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a74009640 con 0x7f9a840702d0 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.833+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8410fdb0 con 0x7f9a840702d0 2024-02-28T16:51:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.833+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a841102f0 con 0x7f9a840702d0 2024-02-28T16:51:57.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.834+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a84106970 con 0x7f9a840702d0 2024-02-28T16:51:57.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.835+0000 7f9a7a000700 1 -- 172.21.15.1:0/2042414976 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9a740097a0 con 0x7f9a840702d0 2024-02-28T16:51:57.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.836+0000 7f9a7a000700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9a6806d7f0 0x7f9a6806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:51:57.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.836+0000 7f9a7a000700 1 -- 172.21.15.1:0/2042414976 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9a7408cbc0 con 0x7f9a840702d0 2024-02-28T16:51:57.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.836+0000 7f9a81400700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9a6806d7f0 0x7f9a6806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:51:57.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.837+0000 7f9a81400700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9a6806d7f0 0x7f9a6806fca0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a70000c00 tx=0x7f9a70009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:51:57.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:57.841+0000 7f9a7a000700 1 -- 172.21.15.1:0/2042414976 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9a74056490 con 0x7f9a840702d0 2024-02-28T16:51:58.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.137+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9a8404bed0 con 0x7f9a840702d0 2024-02-28T16:51:58.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.138+0000 7f9a7a000700 1 -- 172.21.15.1:0/2042414976 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f9a74059b50 con 0x7f9a840702d0 2024-02-28T16:51:58.132 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:51:58.132 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:51:58.132 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:51:58.133 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:51:58.133 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:51:58.133 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:51:58.133 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:51:58.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9a6806d7f0 msgr2=0x7f9a6806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:58.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9a6806d7f0 0x7f9a6806fca0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a70000c00 tx=0x7f9a70009e30 comp rx=0 tx=0).stop 2024-02-28T16:51:58.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a840702d0 msgr2=0x7f9a84115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:51:58.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a840702d0 0x7f9a84115bf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9a74002a50 tx=0x7f9a740046b0 comp rx=0 tx=0).stop 2024-02-28T16:51:58.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 shutdown_connections 2024-02-28T16:51:58.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9a6806d7f0 0x7f9a6806fca0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:58.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a840702d0 0x7f9a84115bf0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:58.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 --2- 172.21.15.1:0/2042414976 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9a84070be0 0x7f9a84116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:51:58.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 >> 172.21.15.1:0/2042414976 conn(0x7f9a840f8af0 msgr2=0x7f9a84103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:51:58.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 shutdown_connections 2024-02-28T16:51:58.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:51:58.141+0000 7f9a89200700 1 -- 172.21.15.1:0/2042414976 wait complete. 2024-02-28T16:51:58.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:58 smithi066 ceph-mon[85781]: pgmap v525: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:58.753 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:58 smithi066 ceph-mon[85781]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:58.753 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:51:58 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2042414976' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:51:58.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:58 smithi001 ceph-mon[112940]: pgmap v525: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:51:58.775 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:58 smithi001 ceph-mon[112940]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:51:58.775 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:51:58 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2042414976' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:52:00.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:00 smithi066 ceph-mon[85781]: pgmap v526: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:00.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:00 smithi001 ceph-mon[112940]: pgmap v526: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:02.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:02 smithi066 ceph-mon[85781]: pgmap v527: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:02.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:02 smithi001 ceph-mon[112940]: pgmap v527: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:04.753 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:04 smithi066 ceph-mon[85781]: pgmap v528: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:04.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:04 smithi001 ceph-mon[112940]: pgmap v528: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:06.753 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:06 smithi066 ceph-mon[85781]: pgmap v529: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:06.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:06 smithi001 ceph-mon[112940]: pgmap v529: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:08.752 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:08 smithi066 ceph-mon[85781]: pgmap v530: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:08.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:08 smithi001 ceph-mon[112940]: pgmap v530: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:10.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:10 smithi001 ceph-mon[112940]: pgmap v531: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:11.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:10 smithi066 ceph-mon[85781]: pgmap v531: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:11.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:52:12.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:52:12.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:12 smithi001 ceph-mon[112940]: pgmap v532: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:13.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:12 smithi066 ceph-mon[85781]: pgmap v532: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:14.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:14 smithi001 ceph-mon[112940]: pgmap v533: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:15.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:14 smithi066 ceph-mon[85781]: pgmap v533: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:17.002 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:16 smithi066 ceph-mon[85781]: pgmap v534: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:17.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:16 smithi001 ceph-mon[112940]: pgmap v534: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:19.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:18 smithi066 ceph-mon[85781]: pgmap v535: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:19.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:18 smithi001 ceph-mon[112940]: pgmap v535: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:21.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:20 smithi066 ceph-mon[85781]: pgmap v536: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:21.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:20 smithi001 ceph-mon[112940]: pgmap v536: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:23.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:22 smithi066 ceph-mon[85781]: pgmap v537: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:23.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:22 smithi001 ceph-mon[112940]: pgmap v537: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:25.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:24 smithi066 ceph-mon[85781]: pgmap v538: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:25.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:24 smithi001 ceph-mon[112940]: pgmap v538: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:27.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:26 smithi066 ceph-mon[85781]: pgmap v539: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:27.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:52:27.024 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:26 smithi001 ceph-mon[112940]: pgmap v539: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:27.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:52:28.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.313+0000 7f5d0ce00700 1 -- 172.21.15.1:0/2504224480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d081095f0 msgr2=0x7f5d081099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:28.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.313+0000 7f5d0ce00700 1 --2- 172.21.15.1:0/2504224480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d081095f0 0x7f5d081099c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8004940 tx=0x7f5cf8009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.313+0000 7f5d0ce00700 1 -- 172.21.15.1:0/2504224480 shutdown_connections 2024-02-28T16:52:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.313+0000 7f5d0ce00700 1 --2- 172.21.15.1:0/2504224480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d08103dc0 0x7f5d08104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.313+0000 7f5d0ce00700 1 --2- 172.21.15.1:0/2504224480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d081095f0 0x7f5d081099c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.313+0000 7f5d0ce00700 1 -- 172.21.15.1:0/2504224480 >> 172.21.15.1:0/2504224480 conn(0x7f5d08072d50 msgr2=0x7f5d08075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.314+0000 7f5d0ce00700 1 -- 172.21.15.1:0/2504224480 shutdown_connections 2024-02-28T16:52:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.314+0000 7f5d0ce00700 1 -- 172.21.15.1:0/2504224480 wait complete. 2024-02-28T16:52:28.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.314+0000 7f5d0ce00700 1 Processor -- start 2024-02-28T16:52:28.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.315+0000 7f5d0ce00700 1 -- start start 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.315+0000 7f5d0ce00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d08103dc0 0x7f5d0811a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.315+0000 7f5d0ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d0811a6b0 0x7f5d08114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.315+0000 7f5d0ce00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0811ac30 con 0x7f5d0811a6b0 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.315+0000 7f5d0ce00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0811ada0 con 0x7f5d08103dc0 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.315+0000 7f5d05400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d0811a6b0 0x7f5d08114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.316+0000 7f5d05400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d0811a6b0 0x7f5d08114080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34214/0 (socket says 172.21.15.1:34214) 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.316+0000 7f5d05400700 1 -- 172.21.15.1:0/3179596583 learned_addr learned my addr 172.21.15.1:0/3179596583 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.316+0000 7f5d05e00700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d08103dc0 0x7f5d0811a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.316+0000 7f5d05400700 1 -- 172.21.15.1:0/3179596583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d08103dc0 msgr2=0x7f5d0811a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.316+0000 7f5d05400700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d08103dc0 0x7f5d0811a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.316+0000 7f5d05400700 1 -- 172.21.15.1:0/3179596583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8005040 con 0x7f5d0811a6b0 2024-02-28T16:52:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.316+0000 7f5d05400700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d0811a6b0 0x7f5d08114080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4002a50 tx=0x7f5cf40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.317+0000 7f5cfe000700 1 -- 172.21.15.1:0/3179596583 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cf4007cc0 con 0x7f5d0811a6b0 2024-02-28T16:52:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.317+0000 7f5cfe000700 1 -- 172.21.15.1:0/3179596583 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5cf400f040 con 0x7f5d0811a6b0 2024-02-28T16:52:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.317+0000 7f5cfe000700 1 -- 172.21.15.1:0/3179596583 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cf4009640 con 0x7f5d0811a6b0 2024-02-28T16:52:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.317+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d08114680 con 0x7f5d0811a6b0 2024-02-28T16:52:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.317+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d08114bc0 con 0x7f5d0811a6b0 2024-02-28T16:52:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.318+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d08106850 con 0x7f5d0811a6b0 2024-02-28T16:52:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.319+0000 7f5cfe000700 1 -- 172.21.15.1:0/3179596583 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5cf4009800 con 0x7f5d0811a6b0 2024-02-28T16:52:28.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.320+0000 7f5cfe000700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cec06d7f0 0x7f5cec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:28.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.320+0000 7f5cfe000700 1 -- 172.21.15.1:0/3179596583 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5cf408cc40 con 0x7f5d0811a6b0 2024-02-28T16:52:28.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.320+0000 7f5d05e00700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cec06d7f0 0x7f5cec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:28.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.321+0000 7f5d05e00700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cec06d7f0 0x7f5cec06fca0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8000c00 tx=0x7f5cf8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.325+0000 7f5cfe000700 1 -- 172.21.15.1:0/3179596583 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5cf4059fa0 con 0x7f5d0811a6b0 2024-02-28T16:52:28.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.563+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d08110620 con 0x7f5cec06d7f0 2024-02-28T16:52:28.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.566+0000 7f5cfe000700 1 -- 172.21.15.1:0/3179596583 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5d08110620 con 0x7f5cec06d7f0 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cec06d7f0 msgr2=0x7f5cec06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cec06d7f0 0x7f5cec06fca0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8000c00 tx=0x7f5cf8009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d0811a6b0 msgr2=0x7f5d08114080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d0811a6b0 0x7f5d08114080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4002a50 tx=0x7f5cf40046b0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 shutdown_connections 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cec06d7f0 0x7f5cec06fca0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d08103dc0 0x7f5d0811a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 --2- 172.21.15.1:0/3179596583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d0811a6b0 0x7f5d08114080 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 >> 172.21.15.1:0/3179596583 conn(0x7f5d08072d50 msgr2=0x7f5d08074af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 shutdown_connections 2024-02-28T16:52:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.569+0000 7f5d0ce00700 1 -- 172.21.15.1:0/3179596583 wait complete. 2024-02-28T16:52:28.579 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:52:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.726+0000 7fa256000700 1 -- 172.21.15.1:0/857440936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fe8f0 msgr2=0x7fa250107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.726+0000 7fa256000700 1 --2- 172.21.15.1:0/857440936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fe8f0 0x7fa250107050 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa238009d10 tx=0x7fa238004d10 comp rx=0 tx=0).stop 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.726+0000 7fa256000700 1 -- 172.21.15.1:0/857440936 shutdown_connections 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.726+0000 7fa256000700 1 --2- 172.21.15.1:0/857440936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fe8f0 0x7fa250107050 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.727+0000 7fa256000700 1 --2- 172.21.15.1:0/857440936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2500fdfe0 0x7fa2500fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.727+0000 7fa256000700 1 -- 172.21.15.1:0/857440936 >> 172.21.15.1:0/857440936 conn(0x7fa250071d90 msgr2=0x7fa250072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.727+0000 7fa256000700 1 -- 172.21.15.1:0/857440936 shutdown_connections 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.727+0000 7fa256000700 1 -- 172.21.15.1:0/857440936 wait complete. 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.727+0000 7fa256000700 1 Processor -- start 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.728+0000 7fa256000700 1 -- start start 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.728+0000 7fa256000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fdfe0 0x7fa250115e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.728+0000 7fa256000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2500fe8f0 0x7fa2501163d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.728+0000 7fa256000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa250116a20 con 0x7fa2500fdfe0 2024-02-28T16:52:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.728+0000 7fa256000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25010fd30 con 0x7fa2500fe8f0 2024-02-28T16:52:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.728+0000 7fa24f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fdfe0 0x7fa250115e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.728+0000 7fa24f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fdfe0 0x7fa250115e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34218/0 (socket says 172.21.15.1:34218) 2024-02-28T16:52:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.728+0000 7fa24f400700 1 -- 172.21.15.1:0/2983181678 learned_addr learned my addr 172.21.15.1:0/2983181678 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:52:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa246a00700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2500fe8f0 0x7fa2501163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa24f400700 1 -- 172.21.15.1:0/2983181678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2500fe8f0 msgr2=0x7fa2501163d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa24f400700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2500fe8f0 0x7fa2501163d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa24f400700 1 -- 172.21.15.1:0/2983181678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa238009940 con 0x7fa2500fdfe0 2024-02-28T16:52:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa246a00700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2500fe8f0 0x7fa2501163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:52:28.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa24f400700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fdfe0 0x7fa250115e90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa240002a50 tx=0x7fa2400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa247e00700 1 -- 172.21.15.1:0/2983181678 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa240007cc0 con 0x7fa2500fdfe0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa247e00700 1 -- 172.21.15.1:0/2983181678 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa24000f040 con 0x7fa2500fdfe0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa250110010 con 0x7fa2500fdfe0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.729+0000 7fa247e00700 1 -- 172.21.15.1:0/2983181678 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa240009640 con 0x7fa2500fdfe0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.730+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa250110550 con 0x7fa2500fdfe0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.731+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa250106b20 con 0x7fa2500fdfe0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.732+0000 7fa247e00700 1 -- 172.21.15.1:0/2983181678 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa2400097a0 con 0x7fa2500fdfe0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.732+0000 7fa247e00700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa23406d7f0 0x7fa23406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.733+0000 7fa247e00700 1 -- 172.21.15.1:0/2983181678 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa24008c960 con 0x7fa2500fdfe0 2024-02-28T16:52:28.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.733+0000 7fa246a00700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa23406d7f0 0x7fa23406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.733+0000 7fa246a00700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa23406d7f0 0x7fa23406fca0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa238005010 tx=0x7fa238004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.737+0000 7fa247e00700 1 -- 172.21.15.1:0/2983181678 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa240059d60 con 0x7fa2500fdfe0 2024-02-28T16:52:28.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.978+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa250111470 con 0x7fa23406d7f0 2024-02-28T16:52:28.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:28 smithi001 ceph-mon[112940]: pgmap v540: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.980+0000 7fa247e00700 1 -- 172.21.15.1:0/2983181678 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa250111470 con 0x7fa23406d7f0 2024-02-28T16:52:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa23406d7f0 msgr2=0x7fa23406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa23406d7f0 0x7fa23406fca0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa238005010 tx=0x7fa238004d10 comp rx=0 tx=0).stop 2024-02-28T16:52:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fdfe0 msgr2=0x7fa250115e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fdfe0 0x7fa250115e90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa240002a50 tx=0x7fa2400046b0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 shutdown_connections 2024-02-28T16:52:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa23406d7f0 0x7fa23406fca0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2500fdfe0 0x7fa250115e90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 --2- 172.21.15.1:0/2983181678 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2500fe8f0 0x7fa2501163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 >> 172.21.15.1:0/2983181678 conn(0x7fa250071d90 msgr2=0x7fa2500fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 shutdown_connections 2024-02-28T16:52:28.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:28.983+0000 7fa256000700 1 -- 172.21.15.1:0/2983181678 wait complete. 2024-02-28T16:52:29.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:28 smithi066 ceph-mon[85781]: pgmap v540: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.140+0000 7f6d86400700 1 -- 172.21.15.1:0/3429681026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fdf60 msgr2=0x7f6d800fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.140+0000 7f6d86400700 1 --2- 172.21.15.1:0/3429681026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fdf60 0x7f6d800fe330 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6d70004940 tx=0x7f6d70009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.140+0000 7f6d86400700 1 -- 172.21.15.1:0/3429681026 shutdown_connections 2024-02-28T16:52:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.140+0000 7f6d86400700 1 --2- 172.21.15.1:0/3429681026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d800fe900 0x7f6d80106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.140+0000 7f6d86400700 1 --2- 172.21.15.1:0/3429681026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fdf60 0x7f6d800fe330 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.140+0000 7f6d86400700 1 -- 172.21.15.1:0/3429681026 >> 172.21.15.1:0/3429681026 conn(0x7f6d800f7a10 msgr2=0x7f6d800f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.140+0000 7f6d86400700 1 -- 172.21.15.1:0/3429681026 shutdown_connections 2024-02-28T16:52:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.140+0000 7f6d86400700 1 -- 172.21.15.1:0/3429681026 wait complete. 2024-02-28T16:52:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.141+0000 7f6d86400700 1 Processor -- start 2024-02-28T16:52:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.141+0000 7f6d86400700 1 -- start start 2024-02-28T16:52:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.142+0000 7f6d86400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d800fdf60 0x7f6d80115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.142+0000 7f6d86400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fe900 0x7f6d801161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.142+0000 7f6d86400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d80116890 con 0x7f6d800fe900 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.142+0000 7f6d86400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8010fb30 con 0x7f6d800fdf60 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.142+0000 7f6d7ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fe900 0x7f6d801161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.142+0000 7f6d7ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fe900 0x7f6d801161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34240/0 (socket says 172.21.15.1:34240) 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.142+0000 7f6d7ea00700 1 -- 172.21.15.1:0/3028179935 learned_addr learned my addr 172.21.15.1:0/3028179935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.142+0000 7f6d7f400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d800fdf60 0x7f6d80115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d7ea00700 1 -- 172.21.15.1:0/3028179935 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d800fdf60 msgr2=0x7f6d80115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d7ea00700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d800fdf60 0x7f6d80115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d7ea00700 1 -- 172.21.15.1:0/3028179935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d70005040 con 0x7f6d800fe900 2024-02-28T16:52:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d7ea00700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fe900 0x7f6d801161b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00b790 tx=0x7f6d6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d77400700 1 -- 172.21.15.1:0/3028179935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d6c00f770 con 0x7f6d800fe900 2024-02-28T16:52:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d77400700 1 -- 172.21.15.1:0/3028179935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d6c00fda0 con 0x7f6d800fe900 2024-02-28T16:52:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8010fe10 con 0x7f6d800fe900 2024-02-28T16:52:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d77400700 1 -- 172.21.15.1:0/3028179935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d6c00d310 con 0x7f6d800fe900 2024-02-28T16:52:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.143+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d80110350 con 0x7f6d800fe900 2024-02-28T16:52:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.145+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d80106950 con 0x7f6d800fe900 2024-02-28T16:52:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.146+0000 7f6d77400700 1 -- 172.21.15.1:0/3028179935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6d6c010420 con 0x7f6d800fe900 2024-02-28T16:52:29.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.146+0000 7f6d77400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.147+0000 7f6d77400700 1 -- 172.21.15.1:0/3028179935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6d6c08cfb0 con 0x7f6d800fe900 2024-02-28T16:52:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.147+0000 7f6d7f400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.147+0000 7f6d7f400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000b3b0 tx=0x7f6d70009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:29.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.151+0000 7f6d77400700 1 -- 172.21.15.1:0/3028179935 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d6c0c0d20 con 0x7f6d800fe900 2024-02-28T16:52:29.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.383+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d8004bf40 con 0x7f6d6406d7f0 2024-02-28T16:52:29.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.396+0000 7f6d77400700 1 -- 172.21.15.1:0/3028179935 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6d8004bf40 con 0x7f6d6406d7f0 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (18m) 10m ago 32m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (15m) 10m ago 33m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (15m) 11m ago 31m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (18m) 10m ago 32m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (11m) 10m ago 25m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (10m) 10m ago 25m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (25m) 11m ago 25m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (25m) 11m ago 25m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (21m) 10m ago 34m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:52:29.390 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (20m) 11m ago 31m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (16m) 10m ago 34m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (16m) 11m ago 31m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (19m) 10m ago 33m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (19m) 11m ago 31m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (14m) 10m ago 30m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (14m) 10m ago 29m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (13m) 10m ago 29m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (13m) 11m ago 28m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (12m) 11m ago 28m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (11m) 11m ago 27m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:52:29.391 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (19m) 10m ago 32m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:52:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 msgr2=0x7f6d6406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000b3b0 tx=0x7f6d70009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fe900 msgr2=0x7f6d801161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fe900 0x7f6d801161b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00b790 tx=0x7f6d6c009250 comp rx=0 tx=0).stop 2024-02-28T16:52:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 shutdown_connections 2024-02-28T16:52:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d800fdf60 0x7f6d80115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 --2- 172.21.15.1:0/3028179935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d800fe900 0x7f6d801161b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.399+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 >> 172.21.15.1:0/3028179935 conn(0x7f6d800f7a10 msgr2=0x7f6d80103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.400+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 shutdown_connections 2024-02-28T16:52:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.400+0000 7f6d86400700 1 -- 172.21.15.1:0/3028179935 wait complete. 2024-02-28T16:52:29.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.551+0000 7f5be9a00700 1 -- 172.21.15.1:0/2654506797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41038f0 msgr2=0x7f5be40fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:29.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.551+0000 7f5be9a00700 1 --2- 172.21.15.1:0/2654506797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41038f0 0x7f5be40fcf60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0004940 tx=0x7f5bd0009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:29.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.551+0000 7f5be9a00700 1 -- 172.21.15.1:0/2654506797 shutdown_connections 2024-02-28T16:52:29.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.551+0000 7f5be9a00700 1 --2- 172.21.15.1:0/2654506797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be41038f0 0x7f5be40fcf60 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.551+0000 7f5be9a00700 1 --2- 172.21.15.1:0/2654506797 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5be4102f50 0x7f5be4103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.551+0000 7f5be9a00700 1 -- 172.21.15.1:0/2654506797 >> 172.21.15.1:0/2654506797 conn(0x7f5be40f8ae0 msgr2=0x7f5be40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.552+0000 7f5be9a00700 1 -- 172.21.15.1:0/2654506797 shutdown_connections 2024-02-28T16:52:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.552+0000 7f5be9a00700 1 -- 172.21.15.1:0/2654506797 wait complete. 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.552+0000 7f5be9a00700 1 Processor -- start 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.553+0000 7f5be9a00700 1 -- start start 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.553+0000 7f5be9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4102f50 0x7f5be4115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.553+0000 7f5be9a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5be41038f0 0x7f5be41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.553+0000 7f5be9a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4116810 con 0x7f5be4102f50 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.553+0000 7f5be9a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be410fb40 con 0x7f5be41038f0 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.553+0000 7f5be2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4102f50 0x7f5be4115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.553+0000 7f5be2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4102f50 0x7f5be4115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34268/0 (socket says 172.21.15.1:34268) 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.553+0000 7f5be2a00700 1 -- 172.21.15.1:0/2276413068 learned_addr learned my addr 172.21.15.1:0/2276413068 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:52:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.554+0000 7f5be2000700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5be41038f0 0x7f5be41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:29.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.554+0000 7f5be2a00700 1 -- 172.21.15.1:0/2276413068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5be41038f0 msgr2=0x7f5be41161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:29.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.554+0000 7f5be2a00700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5be41038f0 0x7f5be41161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.554+0000 7f5be2a00700 1 -- 172.21.15.1:0/2276413068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0005040 con 0x7f5be4102f50 2024-02-28T16:52:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.554+0000 7f5be2a00700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4102f50 0x7f5be4115c80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4002a50 tx=0x7f5bd40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.554+0000 7f5bdaa00700 1 -- 172.21.15.1:0/2276413068 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bd4007cc0 con 0x7f5be4102f50 2024-02-28T16:52:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.555+0000 7f5bdaa00700 1 -- 172.21.15.1:0/2276413068 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bd400f040 con 0x7f5be4102f50 2024-02-28T16:52:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.555+0000 7f5bdaa00700 1 -- 172.21.15.1:0/2276413068 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bd400da30 con 0x7f5be4102f50 2024-02-28T16:52:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.555+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be410fe20 con 0x7f5be4102f50 2024-02-28T16:52:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.555+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be4110330 con 0x7f5be4102f50 2024-02-28T16:52:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.556+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be40fc6e0 con 0x7f5be4102f50 2024-02-28T16:52:29.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.557+0000 7f5bdaa00700 1 -- 172.21.15.1:0/2276413068 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5bd400db90 con 0x7f5be4102f50 2024-02-28T16:52:29.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.558+0000 7f5bdaa00700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5bc806d7e0 0x7f5bc806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:29.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.558+0000 7f5bdaa00700 1 -- 172.21.15.1:0/2276413068 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5bd408cbf0 con 0x7f5be4102f50 2024-02-28T16:52:29.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.558+0000 7f5be2000700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5bc806d7e0 0x7f5bc806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:29.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.559+0000 7f5be2000700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5bc806d7e0 0x7f5bc806fc90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5be4110e70 tx=0x7f5bd0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:29.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.563+0000 7f5bdaa00700 1 -- 172.21.15.1:0/2276413068 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5bd4059f50 con 0x7f5be4102f50 2024-02-28T16:52:29.774 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:29 smithi001 ceph-mon[112940]: from='client.35006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:52:29.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.879+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5be4111220 con 0x7f5be4102f50 2024-02-28T16:52:29.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.880+0000 7f5bdaa00700 1 -- 172.21.15.1:0/2276413068 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f5bd4059ae0 con 0x7f5be4102f50 2024-02-28T16:52:29.873 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:52:29.873 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:52:29.874 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:52:29.875 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:52:29.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:52:29.875 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:52:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5bc806d7e0 msgr2=0x7f5bc806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5bc806d7e0 0x7f5bc806fc90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5be4110e70 tx=0x7f5bd0009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4102f50 msgr2=0x7f5be4115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4102f50 0x7f5be4115c80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4002a50 tx=0x7f5bd40046b0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 shutdown_connections 2024-02-28T16:52:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5bc806d7e0 0x7f5bc806fc90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be4102f50 0x7f5be4115c80 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 --2- 172.21.15.1:0/2276413068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5be41038f0 0x7f5be41161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:29.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.882+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 >> 172.21.15.1:0/2276413068 conn(0x7f5be40f8ae0 msgr2=0x7f5be40fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:29.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.883+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 shutdown_connections 2024-02-28T16:52:29.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:29.883+0000 7f5be9a00700 1 -- 172.21.15.1:0/2276413068 wait complete. 2024-02-28T16:52:30.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:29 smithi066 ceph-mon[85781]: from='client.35006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:52:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.038+0000 7f34fd400700 1 -- 172.21.15.1:0/651036556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f81095f0 msgr2=0x7f34f81099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.038+0000 7f34fd400700 1 --2- 172.21.15.1:0/651036556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f81095f0 0x7f34f81099c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f34e8004940 tx=0x7f34e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.038+0000 7f34fd400700 1 -- 172.21.15.1:0/651036556 shutdown_connections 2024-02-28T16:52:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.038+0000 7f34fd400700 1 --2- 172.21.15.1:0/651036556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34f8103dc0 0x7f34f8104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.038+0000 7f34fd400700 1 --2- 172.21.15.1:0/651036556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f81095f0 0x7f34f81099c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.038+0000 7f34fd400700 1 -- 172.21.15.1:0/651036556 >> 172.21.15.1:0/651036556 conn(0x7f34f8072d50 msgr2=0x7f34f8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.038+0000 7f34fd400700 1 -- 172.21.15.1:0/651036556 shutdown_connections 2024-02-28T16:52:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.038+0000 7f34fd400700 1 -- 172.21.15.1:0/651036556 wait complete. 2024-02-28T16:52:30.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.039+0000 7f34fd400700 1 Processor -- start 2024-02-28T16:52:30.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.039+0000 7f34fd400700 1 -- start start 2024-02-28T16:52:30.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.040+0000 7f34fd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f8103dc0 0x7f34f811a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.040+0000 7f34fd400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34f811a740 0x7f34f81140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.040+0000 7f34fd400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f811acc0 con 0x7f34f8103dc0 2024-02-28T16:52:30.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.040+0000 7f34fd400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f811ae30 con 0x7f34f811a740 2024-02-28T16:52:30.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.040+0000 7f34f6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f8103dc0 0x7f34f811a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:30.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.040+0000 7f34f6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f8103dc0 0x7f34f811a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34290/0 (socket says 172.21.15.1:34290) 2024-02-28T16:52:30.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.040+0000 7f34f6800700 1 -- 172.21.15.1:0/265198909 learned_addr learned my addr 172.21.15.1:0/265198909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34f6800700 1 -- 172.21.15.1:0/265198909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34f811a740 msgr2=0x7f34f81140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34f6800700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34f811a740 0x7f34f81140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34f6800700 1 -- 172.21.15.1:0/265198909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e8005040 con 0x7f34f8103dc0 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34f6800700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f8103dc0 0x7f34f811a200 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f34e8000c00 tx=0x7f34e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34eea00700 1 -- 172.21.15.1:0/265198909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e800e040 con 0x7f34f8103dc0 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34eea00700 1 -- 172.21.15.1:0/265198909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34e8009430 con 0x7f34f8103dc0 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34eea00700 1 -- 172.21.15.1:0/265198909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e801c630 con 0x7f34f8103dc0 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f81146a0 con 0x7f34f8103dc0 2024-02-28T16:52:30.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.041+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f8114be0 con 0x7f34f8103dc0 2024-02-28T16:52:30.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.042+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8114ec0 con 0x7f34f8103dc0 2024-02-28T16:52:30.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.044+0000 7f34eea00700 1 -- 172.21.15.1:0/265198909 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f34e800bbf0 con 0x7f34f8103dc0 2024-02-28T16:52:30.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.044+0000 7f34eea00700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34dc06d7f0 0x7f34dc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.045+0000 7f34eea00700 1 -- 172.21.15.1:0/265198909 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f34e808e330 con 0x7f34f8103dc0 2024-02-28T16:52:30.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.045+0000 7f34f5e00700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34dc06d7f0 0x7f34dc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:30.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.045+0000 7f34f5e00700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34dc06d7f0 0x7f34dc06fca0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f34e4004d00 tx=0x7f34e4009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:30.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.049+0000 7f34eea00700 1 -- 172.21.15.1:0/265198909 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f34e805b690 con 0x7f34f8103dc0 2024-02-28T16:52:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.316+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f34f804bed0 con 0x7f34f8103dc0 2024-02-28T16:52:30.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.316+0000 7f34eea00700 1 -- 172.21.15.1:0/265198909 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f34e805b220 con 0x7f34f8103dc0 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:52:30.311 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:52:30.312 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:52:30.313 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:52:30.314 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:52:30.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34dc06d7f0 msgr2=0x7f34dc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34dc06d7f0 0x7f34dc06fca0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f34e4004d00 tx=0x7f34e4009ce0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f8103dc0 msgr2=0x7f34f811a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f8103dc0 0x7f34f811a200 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f34e8000c00 tx=0x7f34e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:30.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 shutdown_connections 2024-02-28T16:52:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34dc06d7f0 0x7f34dc06fca0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34f8103dc0 0x7f34f811a200 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 --2- 172.21.15.1:0/265198909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34f811a740 0x7f34f81140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 >> 172.21.15.1:0/265198909 conn(0x7f34f8072d50 msgr2=0x7f34f810cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 shutdown_connections 2024-02-28T16:52:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.319+0000 7f34fd400700 1 -- 172.21.15.1:0/265198909 wait complete. 2024-02-28T16:52:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:52:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.470+0000 7f1c59a00700 1 -- 172.21.15.1:0/4186137606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105170 msgr2=0x7f1c54105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.470+0000 7f1c59a00700 1 --2- 172.21.15.1:0/4186137606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105170 0x7f1c54105540 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1c44004940 tx=0x7f1c44009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.470+0000 7f1c59a00700 1 -- 172.21.15.1:0/4186137606 shutdown_connections 2024-02-28T16:52:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.470+0000 7f1c59a00700 1 --2- 172.21.15.1:0/4186137606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c54105b10 0x7f1c540ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.470+0000 7f1c59a00700 1 --2- 172.21.15.1:0/4186137606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105170 0x7f1c54105540 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.470+0000 7f1c59a00700 1 -- 172.21.15.1:0/4186137606 >> 172.21.15.1:0/4186137606 conn(0x7f1c540face0 msgr2=0x7f1c540fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.470+0000 7f1c59a00700 1 -- 172.21.15.1:0/4186137606 shutdown_connections 2024-02-28T16:52:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.471+0000 7f1c59a00700 1 -- 172.21.15.1:0/4186137606 wait complete. 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.471+0000 7f1c59a00700 1 Processor -- start 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.471+0000 7f1c59a00700 1 -- start start 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.472+0000 7f1c59a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c54105170 0x7f1c54115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.472+0000 7f1c59a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105b10 0x7f1c541161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.472+0000 7f1c59a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c541167f0 con 0x7f1c54105b10 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.472+0000 7f1c59a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5410fb40 con 0x7f1c54105170 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.472+0000 7f1c52000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105b10 0x7f1c541161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.472+0000 7f1c52000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105b10 0x7f1c541161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34312/0 (socket says 172.21.15.1:34312) 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.472+0000 7f1c52000700 1 -- 172.21.15.1:0/526429434 learned_addr learned my addr 172.21.15.1:0/526429434 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:52:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c52000700 1 -- 172.21.15.1:0/526429434 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c54105170 msgr2=0x7f1c54115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c52000700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c54105170 0x7f1c54115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c52000700 1 -- 172.21.15.1:0/526429434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c44005040 con 0x7f1c54105b10 2024-02-28T16:52:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c52000700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105b10 0x7f1c541161a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1c40002a50 tx=0x7f1c40004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c4aa00700 1 -- 172.21.15.1:0/526429434 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c40007cc0 con 0x7f1c54105b10 2024-02-28T16:52:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c4aa00700 1 -- 172.21.15.1:0/526429434 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c4000f040 con 0x7f1c54105b10 2024-02-28T16:52:30.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5410fe20 con 0x7f1c54105b10 2024-02-28T16:52:30.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c4aa00700 1 -- 172.21.15.1:0/526429434 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c4000daf0 con 0x7f1c54105b10 2024-02-28T16:52:30.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.473+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c54110360 con 0x7f1c54105b10 2024-02-28T16:52:30.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.474+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c54102480 con 0x7f1c54105b10 2024-02-28T16:52:30.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.476+0000 7f1c4aa00700 1 -- 172.21.15.1:0/526429434 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1c40009810 con 0x7f1c54105b10 2024-02-28T16:52:30.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.477+0000 7f1c4aa00700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1c3806d7f0 0x7f1c3806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.477+0000 7f1c4aa00700 1 -- 172.21.15.1:0/526429434 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1c4008cc20 con 0x7f1c54105b10 2024-02-28T16:52:30.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.477+0000 7f1c52a00700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1c3806d7f0 0x7f1c3806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:30.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.477+0000 7f1c52a00700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1c3806d7f0 0x7f1c3806fca0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1c44004f90 tx=0x7f1c44009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:30.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.481+0000 7f1c4aa00700 1 -- 172.21.15.1:0/526429434 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1c40056450 con 0x7f1c54105b10 2024-02-28T16:52:30.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.714+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c54111280 con 0x7f1c3806d7f0 2024-02-28T16:52:30.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.716+0000 7f1c4aa00700 1 -- 172.21.15.1:0/526429434 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1c54111280 con 0x7f1c3806d7f0 2024-02-28T16:52:30.710 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:52:30.710 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:52:30.710 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:52:30.710 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:52:30.710 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:52:30.710 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:52:30.711 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:52:30.711 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:52:30.711 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:52:30.711 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:52:30.711 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:52:30.711 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:52:30.711 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:52:30.711 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1c3806d7f0 msgr2=0x7f1c3806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1c3806d7f0 0x7f1c3806fca0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1c44004f90 tx=0x7f1c44009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105b10 msgr2=0x7f1c541161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105b10 0x7f1c541161a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1c40002a50 tx=0x7f1c40004600 comp rx=0 tx=0).stop 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 shutdown_connections 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1c3806d7f0 0x7f1c3806fca0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1c54105170 0x7f1c54115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 --2- 172.21.15.1:0/526429434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c54105b10 0x7f1c541161a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 >> 172.21.15.1:0/526429434 conn(0x7f1c540face0 msgr2=0x7f1c540fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.719+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 shutdown_connections 2024-02-28T16:52:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.720+0000 7f1c59a00700 1 -- 172.21.15.1:0/526429434 wait complete. 2024-02-28T16:52:30.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.879+0000 7fae93200700 1 -- 172.21.15.1:0/1957146616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fe720 msgr2=0x7fae94106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.879+0000 7fae93200700 1 --2- 172.21.15.1:0/1957146616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fe720 0x7fae94106df0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fae80004970 tx=0x7fae80009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:30.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.879+0000 7fae93200700 1 -- 172.21.15.1:0/1957146616 shutdown_connections 2024-02-28T16:52:30.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.879+0000 7fae93200700 1 --2- 172.21.15.1:0/1957146616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fe720 0x7fae94106df0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.879+0000 7fae93200700 1 --2- 172.21.15.1:0/1957146616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae940fdd80 0x7fae940fe150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.874 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:30 smithi001 ceph-mon[112940]: from='client.35010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:52:30.874 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:30 smithi001 ceph-mon[112940]: pgmap v541: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:30.874 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:30 smithi001 ceph-mon[112940]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:52:30.874 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2276413068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:52:30.874 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/265198909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:52:30.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.879+0000 7fae93200700 1 -- 172.21.15.1:0/1957146616 >> 172.21.15.1:0/1957146616 conn(0x7fae940f77c0 msgr2=0x7fae940f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.880+0000 7fae93200700 1 -- 172.21.15.1:0/1957146616 shutdown_connections 2024-02-28T16:52:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.880+0000 7fae93200700 1 -- 172.21.15.1:0/1957146616 wait complete. 2024-02-28T16:52:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.881+0000 7fae93200700 1 Processor -- start 2024-02-28T16:52:30.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.881+0000 7fae93200700 1 -- start start 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.881+0000 7fae93200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fdd80 0x7fae941159f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.881+0000 7fae93200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae940fe720 0x7fae94115f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.881+0000 7fae93200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae94116580 con 0x7fae940fdd80 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.881+0000 7fae93200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9410f8b0 con 0x7fae940fe720 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.882+0000 7fae91e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fdd80 0x7fae941159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.882+0000 7fae91e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fdd80 0x7fae941159f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34326/0 (socket says 172.21.15.1:34326) 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.882+0000 7fae91e00700 1 -- 172.21.15.1:0/3885969924 learned_addr learned my addr 172.21.15.1:0/3885969924 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.882+0000 7fae91400700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae940fe720 0x7fae94115f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.882+0000 7fae91e00700 1 -- 172.21.15.1:0/3885969924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae940fe720 msgr2=0x7fae94115f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.882+0000 7fae91e00700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae940fe720 0x7fae94115f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.882+0000 7fae91e00700 1 -- 172.21.15.1:0/3885969924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80005040 con 0x7fae940fdd80 2024-02-28T16:52:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.882+0000 7fae91400700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae940fe720 0x7fae94115f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:52:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.883+0000 7fae91e00700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fdd80 0x7fae941159f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fae8400c810 tx=0x7fae84004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.883+0000 7fae8a000700 1 -- 172.21.15.1:0/3885969924 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae84012810 con 0x7fae940fdd80 2024-02-28T16:52:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.883+0000 7fae8a000700 1 -- 172.21.15.1:0/3885969924 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae84012e40 con 0x7fae940fdd80 2024-02-28T16:52:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.883+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9410fb90 con 0x7fae940fdd80 2024-02-28T16:52:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.883+0000 7fae8a000700 1 -- 172.21.15.1:0/3885969924 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae8400a370 con 0x7fae940fdd80 2024-02-28T16:52:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.883+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae941100d0 con 0x7fae940fdd80 2024-02-28T16:52:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.885+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae941068f0 con 0x7fae940fdd80 2024-02-28T16:52:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.886+0000 7fae8a000700 1 -- 172.21.15.1:0/3885969924 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fae84011810 con 0x7fae940fdd80 2024-02-28T16:52:30.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.887+0000 7fae8a000700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae7806d7f0 0x7fae7806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:52:30.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.887+0000 7fae8a000700 1 -- 172.21.15.1:0/3885969924 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fae8408d350 con 0x7fae940fdd80 2024-02-28T16:52:30.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.887+0000 7fae91400700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae7806d7f0 0x7fae7806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:52:30.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.888+0000 7fae91400700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae7806d7f0 0x7fae7806fca0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fae80000c00 tx=0x7fae80009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:52:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:30.892+0000 7fae8a000700 1 -- 172.21.15.1:0/3885969924 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae8405a6d0 con 0x7fae940fdd80 2024-02-28T16:52:31.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:30 smithi066 ceph-mon[85781]: from='client.35010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:52:31.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:30 smithi066 ceph-mon[85781]: pgmap v541: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:31.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:30 smithi066 ceph-mon[85781]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:52:31.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2276413068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:52:31.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/265198909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:52:31.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.206+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fae9404bf60 con 0x7fae940fdd80 2024-02-28T16:52:31.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.208+0000 7fae8a000700 1 -- 172.21.15.1:0/3885969924 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fae8405a260 con 0x7fae940fdd80 2024-02-28T16:52:31.201 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:52:31.201 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:52:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:52:31.201 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:52:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:52:31.201 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:52:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:52:31.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.210+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae7806d7f0 msgr2=0x7fae7806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:31.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.210+0000 7fae93200700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae7806d7f0 0x7fae7806fca0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fae80000c00 tx=0x7fae80009e30 comp rx=0 tx=0).stop 2024-02-28T16:52:31.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.210+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fdd80 msgr2=0x7fae941159f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:52:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.210+0000 7fae93200700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fdd80 0x7fae941159f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fae8400c810 tx=0x7fae84004470 comp rx=0 tx=0).stop 2024-02-28T16:52:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.211+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 shutdown_connections 2024-02-28T16:52:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.211+0000 7fae93200700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae7806d7f0 0x7fae7806fca0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.211+0000 7fae93200700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae940fdd80 0x7fae941159f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.211+0000 7fae93200700 1 --2- 172.21.15.1:0/3885969924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae940fe720 0x7fae94115f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:52:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.211+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 >> 172.21.15.1:0/3885969924 conn(0x7fae940f77c0 msgr2=0x7fae940fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:52:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.211+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 shutdown_connections 2024-02-28T16:52:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:52:31.211+0000 7fae93200700 1 -- 172.21.15.1:0/3885969924 wait complete. 2024-02-28T16:52:32.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:31 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3885969924' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:52:32.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:31 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3885969924' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:52:33.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:32 smithi066 ceph-mon[85781]: from='client.35026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:52:33.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:32 smithi066 ceph-mon[85781]: pgmap v542: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:33.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:32 smithi001 ceph-mon[112940]: from='client.35026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:52:33.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:32 smithi001 ceph-mon[112940]: pgmap v542: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:35.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:34 smithi066 ceph-mon[85781]: pgmap v543: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:35.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:34 smithi001 ceph-mon[112940]: pgmap v543: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:37.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:36 smithi066 ceph-mon[85781]: pgmap v544: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:37.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:36 smithi001 ceph-mon[112940]: pgmap v544: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:39.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:38 smithi066 ceph-mon[85781]: pgmap v545: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:39.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:38 smithi001 ceph-mon[112940]: pgmap v545: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:41.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:40 smithi066 ceph-mon[85781]: pgmap v546: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:41.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:40 smithi001 ceph-mon[112940]: pgmap v546: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:42.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:52:42.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:52:43.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:42 smithi066 ceph-mon[85781]: pgmap v547: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:43.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:42 smithi001 ceph-mon[112940]: pgmap v547: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:45.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:44 smithi066 ceph-mon[85781]: pgmap v548: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:45.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:44 smithi001 ceph-mon[112940]: pgmap v548: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:47.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:46 smithi066 ceph-mon[85781]: pgmap v549: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:47.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:46 smithi001 ceph-mon[112940]: pgmap v549: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:49.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:48 smithi066 ceph-mon[85781]: pgmap v550: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:49.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:48 smithi001 ceph-mon[112940]: pgmap v550: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:51.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:50 smithi066 ceph-mon[85781]: pgmap v551: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:51.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:50 smithi001 ceph-mon[112940]: pgmap v551: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:53.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:52 smithi066 ceph-mon[85781]: pgmap v552: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:53.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:52 smithi001 ceph-mon[112940]: pgmap v552: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:55.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:54 smithi066 ceph-mon[85781]: pgmap v553: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:55.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:54 smithi001 ceph-mon[112940]: pgmap v553: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:57.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:56 smithi066 ceph-mon[85781]: pgmap v554: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:57.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:52:57.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:56 smithi001 ceph-mon[112940]: pgmap v554: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:57.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:52:59.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:52:58 smithi066 ceph-mon[85781]: pgmap v555: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:52:59.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:52:58 smithi001 ceph-mon[112940]: pgmap v555: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:01.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:00 smithi066 ceph-mon[85781]: pgmap v556: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:01.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:00 smithi001 ceph-mon[112940]: pgmap v556: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:01.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.382+0000 7fef7e400700 1 -- 172.21.15.1:0/3606135826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef781095f0 msgr2=0x7fef781099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:01.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.382+0000 7fef7e400700 1 --2- 172.21.15.1:0/3606135826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef781095f0 0x7fef781099c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fef68004940 tx=0x7fef68009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:01.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.382+0000 7fef7e400700 1 -- 172.21.15.1:0/3606135826 shutdown_connections 2024-02-28T16:53:01.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.382+0000 7fef7e400700 1 --2- 172.21.15.1:0/3606135826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef78103dc0 0x7fef78104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.382+0000 7fef7e400700 1 --2- 172.21.15.1:0/3606135826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef781095f0 0x7fef781099c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.382+0000 7fef7e400700 1 -- 172.21.15.1:0/3606135826 >> 172.21.15.1:0/3606135826 conn(0x7fef78072d50 msgr2=0x7fef78075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:01.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.382+0000 7fef7e400700 1 -- 172.21.15.1:0/3606135826 shutdown_connections 2024-02-28T16:53:01.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.382+0000 7fef7e400700 1 -- 172.21.15.1:0/3606135826 wait complete. 2024-02-28T16:53:01.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.383+0000 7fef7e400700 1 Processor -- start 2024-02-28T16:53:01.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.384+0000 7fef7e400700 1 -- start start 2024-02-28T16:53:01.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.384+0000 7fef7e400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef78103dc0 0x7fef7811a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:01.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.384+0000 7fef7e400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef781095f0 0x7fef7811a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:01.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.384+0000 7fef7e400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7811ac70 con 0x7fef78103dc0 2024-02-28T16:53:01.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.384+0000 7fef7e400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef78113f80 con 0x7fef781095f0 2024-02-28T16:53:01.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.384+0000 7fef77400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef78103dc0 0x7fef7811a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:01.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.384+0000 7fef77400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef78103dc0 0x7fef7811a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37800/0 (socket says 172.21.15.1:37800) 2024-02-28T16:53:01.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.384+0000 7fef77400700 1 -- 172.21.15.1:0/847278997 learned_addr learned my addr 172.21.15.1:0/847278997 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:01.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.385+0000 7fef6ea00700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef781095f0 0x7fef7811a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:01.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.385+0000 7fef77400700 1 -- 172.21.15.1:0/847278997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef781095f0 msgr2=0x7fef7811a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:01.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.385+0000 7fef77400700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef781095f0 0x7fef7811a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.385+0000 7fef77400700 1 -- 172.21.15.1:0/847278997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68005040 con 0x7fef78103dc0 2024-02-28T16:53:01.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.385+0000 7fef77400700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef78103dc0 0x7fef7811a0e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fef68009150 tx=0x7fef68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:01.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.385+0000 7fef6fe00700 1 -- 172.21.15.1:0/847278997 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef6800e040 con 0x7fef78103dc0 2024-02-28T16:53:01.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.386+0000 7fef6fe00700 1 -- 172.21.15.1:0/847278997 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef6800bbf0 con 0x7fef78103dc0 2024-02-28T16:53:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.386+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef78114200 con 0x7fef78103dc0 2024-02-28T16:53:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.386+0000 7fef6fe00700 1 -- 172.21.15.1:0/847278997 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef6801c630 con 0x7fef78103dc0 2024-02-28T16:53:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.386+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef78114740 con 0x7fef78103dc0 2024-02-28T16:53:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.387+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef78106850 con 0x7fef78103dc0 2024-02-28T16:53:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.388+0000 7fef6fe00700 1 -- 172.21.15.1:0/847278997 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fef6800bd60 con 0x7fef78103dc0 2024-02-28T16:53:01.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.389+0000 7fef6fe00700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fef5806d7f0 0x7fef5806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:01.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.389+0000 7fef6fe00700 1 -- 172.21.15.1:0/847278997 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fef6808e2c0 con 0x7fef78103dc0 2024-02-28T16:53:01.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.389+0000 7fef6ea00700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fef5806d7f0 0x7fef5806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:01.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.390+0000 7fef6ea00700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fef5806d7f0 0x7fef5806fca0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fef78115980 tx=0x7fef60009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:01.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.394+0000 7fef6fe00700 1 -- 172.21.15.1:0/847278997 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef6805b6a0 con 0x7fef78103dc0 2024-02-28T16:53:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.636+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef7804bed0 con 0x7fef5806d7f0 2024-02-28T16:53:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.638+0000 7fef6fe00700 1 -- 172.21.15.1:0/847278997 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fef7804bed0 con 0x7fef5806d7f0 2024-02-28T16:53:01.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fef5806d7f0 msgr2=0x7fef5806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:01.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fef5806d7f0 0x7fef5806fca0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fef78115980 tx=0x7fef60009040 comp rx=0 tx=0).stop 2024-02-28T16:53:01.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef78103dc0 msgr2=0x7fef7811a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:01.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef78103dc0 0x7fef7811a0e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fef68009150 tx=0x7fef68009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:01.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 shutdown_connections 2024-02-28T16:53:01.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fef5806d7f0 0x7fef5806fca0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef78103dc0 0x7fef7811a0e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 --2- 172.21.15.1:0/847278997 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fef781095f0 0x7fef7811a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.641+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 >> 172.21.15.1:0/847278997 conn(0x7fef78072d50 msgr2=0x7fef78074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:01.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.642+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 shutdown_connections 2024-02-28T16:53:01.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.642+0000 7fef7e400700 1 -- 172.21.15.1:0/847278997 wait complete. 2024-02-28T16:53:01.651 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:53:01.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.793+0000 7f2e4ca00700 1 -- 172.21.15.1:0/3882389532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 msgr2=0x7f2e48103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:01.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.793+0000 7f2e4ca00700 1 --2- 172.21.15.1:0/3882389532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 0x7f2e48103320 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2e38004940 tx=0x7f2e38009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:01.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.793+0000 7f2e4ca00700 1 -- 172.21.15.1:0/3882389532 shutdown_connections 2024-02-28T16:53:01.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.793+0000 7f2e4ca00700 1 --2- 172.21.15.1:0/3882389532 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e481038f0 0x7f2e480fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.793+0000 7f2e4ca00700 1 --2- 172.21.15.1:0/3882389532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 0x7f2e48103320 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.793+0000 7f2e4ca00700 1 -- 172.21.15.1:0/3882389532 >> 172.21.15.1:0/3882389532 conn(0x7f2e480f8ae0 msgr2=0x7f2e480faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.793+0000 7f2e4ca00700 1 -- 172.21.15.1:0/3882389532 shutdown_connections 2024-02-28T16:53:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.793+0000 7f2e4ca00700 1 -- 172.21.15.1:0/3882389532 wait complete. 2024-02-28T16:53:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.794+0000 7f2e4ca00700 1 Processor -- start 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.794+0000 7f2e4ca00700 1 -- start start 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.794+0000 7f2e4ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 0x7f2e48115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.794+0000 7f2e4ca00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e481038f0 0x7f2e48116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.794+0000 7f2e4ca00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e48116790 con 0x7f2e48102f50 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.794+0000 7f2e4ca00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4810fae0 con 0x7f2e481038f0 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.795+0000 7f2e45e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 0x7f2e48115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.795+0000 7f2e45e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 0x7f2e48115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37812/0 (socket says 172.21.15.1:37812) 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.795+0000 7f2e45e00700 1 -- 172.21.15.1:0/4180438338 learned_addr learned my addr 172.21.15.1:0/4180438338 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.795+0000 7f2e45e00700 1 -- 172.21.15.1:0/4180438338 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e481038f0 msgr2=0x7f2e48116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.795+0000 7f2e45e00700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e481038f0 0x7f2e48116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.795+0000 7f2e45e00700 1 -- 172.21.15.1:0/4180438338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e38005040 con 0x7f2e48102f50 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.795+0000 7f2e45e00700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 0x7f2e48115c00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009fd0 tx=0x7f2e38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.796+0000 7f2e3e000700 1 -- 172.21.15.1:0/4180438338 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e3800e040 con 0x7f2e48102f50 2024-02-28T16:53:01.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.796+0000 7f2e3e000700 1 -- 172.21.15.1:0/4180438338 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e3800bbf0 con 0x7f2e48102f50 2024-02-28T16:53:01.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.796+0000 7f2e3e000700 1 -- 172.21.15.1:0/4180438338 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e3801c3f0 con 0x7f2e48102f50 2024-02-28T16:53:01.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.796+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4810fd60 con 0x7f2e48102f50 2024-02-28T16:53:01.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.796+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e481102a0 con 0x7f2e48102f50 2024-02-28T16:53:01.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.797+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e480fc6e0 con 0x7f2e48102f50 2024-02-28T16:53:01.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.798+0000 7f2e3e000700 1 -- 172.21.15.1:0/4180438338 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2e3800bd60 con 0x7f2e48102f50 2024-02-28T16:53:01.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.799+0000 7f2e3e000700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2c06d7f0 0x7f2e2c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:01.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.799+0000 7f2e3e000700 1 -- 172.21.15.1:0/4180438338 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2e3808e1c0 con 0x7f2e48102f50 2024-02-28T16:53:01.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.799+0000 7f2e45400700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2c06d7f0 0x7f2e2c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:01.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.800+0000 7f2e45400700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2c06d7f0 0x7f2e2c06fca0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2e481114a0 tx=0x7f2e34007d40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:01.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:01.803+0000 7f2e3e000700 1 -- 172.21.15.1:0/4180438338 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2e38057a70 con 0x7f2e48102f50 2024-02-28T16:53:02.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.034+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e48111250 con 0x7f2e2c06d7f0 2024-02-28T16:53:02.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.036+0000 7f2e3e000700 1 -- 172.21.15.1:0/4180438338 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2e48111250 con 0x7f2e2c06d7f0 2024-02-28T16:53:02.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2c06d7f0 msgr2=0x7f2e2c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2c06d7f0 0x7f2e2c06fca0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2e481114a0 tx=0x7f2e34007d40 comp rx=0 tx=0).stop 2024-02-28T16:53:02.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 msgr2=0x7f2e48115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 0x7f2e48115c00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009fd0 tx=0x7f2e38009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:02.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 shutdown_connections 2024-02-28T16:53:02.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2c06d7f0 0x7f2e2c06fca0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e48102f50 0x7f2e48115c00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 --2- 172.21.15.1:0/4180438338 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e481038f0 0x7f2e48116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 >> 172.21.15.1:0/4180438338 conn(0x7f2e480f8ae0 msgr2=0x7f2e480fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:02.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 shutdown_connections 2024-02-28T16:53:02.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.039+0000 7f2e4ca00700 1 -- 172.21.15.1:0/4180438338 wait complete. 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.184+0000 7f7bb5600700 1 -- 172.21.15.1:0/233531269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 msgr2=0x7f7bb00ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.184+0000 7f7bb5600700 1 --2- 172.21.15.1:0/233531269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 0x7f7bb00ff170 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c004940 tx=0x7f7b9c009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.184+0000 7f7bb5600700 1 -- 172.21.15.1:0/233531269 shutdown_connections 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.184+0000 7f7bb5600700 1 --2- 172.21.15.1:0/233531269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 0x7f7bb00ff170 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.184+0000 7f7bb5600700 1 --2- 172.21.15.1:0/233531269 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb0105160 0x7f7bb0105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.184+0000 7f7bb5600700 1 -- 172.21.15.1:0/233531269 >> 172.21.15.1:0/233531269 conn(0x7f7bb00fad30 msgr2=0x7f7bb00fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.184+0000 7f7bb5600700 1 -- 172.21.15.1:0/233531269 shutdown_connections 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.184+0000 7f7bb5600700 1 -- 172.21.15.1:0/233531269 wait complete. 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.185+0000 7f7bb5600700 1 Processor -- start 2024-02-28T16:53:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.185+0000 7f7bb5600700 1 -- start start 2024-02-28T16:53:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.185+0000 7f7bb5600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb0105160 0x7f7bb0115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.185+0000 7f7bb5600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 0x7f7bb0116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.185+0000 7f7bb5600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0116860 con 0x7f7bb0105b00 2024-02-28T16:53:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.185+0000 7f7bb5600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb010fbb0 con 0x7f7bb0105160 2024-02-28T16:53:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.186+0000 7f7bae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 0x7f7bb0116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.186+0000 7f7bae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 0x7f7bb0116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40030/0 (socket says 172.21.15.1:40030) 2024-02-28T16:53:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.186+0000 7f7bae000700 1 -- 172.21.15.1:0/379858652 learned_addr learned my addr 172.21.15.1:0/379858652 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.186+0000 7f7baea00700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb0105160 0x7f7bb0115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.186+0000 7f7bae000700 1 -- 172.21.15.1:0/379858652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb0105160 msgr2=0x7f7bb0115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.186+0000 7f7bae000700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb0105160 0x7f7bb0115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.186+0000 7f7bae000700 1 -- 172.21.15.1:0/379858652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c005040 con 0x7f7bb0105b00 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.186+0000 7f7bae000700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 0x7f7bb0116210 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009100 tx=0x7f7b9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.187+0000 7f7ba6a00700 1 -- 172.21.15.1:0/379858652 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9c00e040 con 0x7f7bb0105b00 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.187+0000 7f7ba6a00700 1 -- 172.21.15.1:0/379858652 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b9c004290 con 0x7f7bb0105b00 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.187+0000 7f7ba6a00700 1 -- 172.21.15.1:0/379858652 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9c01c3f0 con 0x7f7bb0105b00 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.187+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb010fe30 con 0x7f7bb0105b00 2024-02-28T16:53:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.187+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb0110370 con 0x7f7bb0105b00 2024-02-28T16:53:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.188+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb0107630 con 0x7f7bb0105b00 2024-02-28T16:53:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.189+0000 7f7ba6a00700 1 -- 172.21.15.1:0/379858652 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7b9c00bbf0 con 0x7f7bb0105b00 2024-02-28T16:53:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.190+0000 7f7ba6a00700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9406d7f0 0x7f7b9406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.190+0000 7f7ba6a00700 1 -- 172.21.15.1:0/379858652 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7b9c08e1c0 con 0x7f7bb0105b00 2024-02-28T16:53:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.190+0000 7f7baea00700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9406d7f0 0x7f7b9406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.191+0000 7f7baea00700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9406d7f0 0x7f7b9406fca0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0004d60 tx=0x7f7ba000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.195+0000 7f7ba6a00700 1 -- 172.21.15.1:0/379858652 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7b9c057a70 con 0x7f7bb0105b00 2024-02-28T16:53:02.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.420+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bb004bed0 con 0x7f7b9406d7f0 2024-02-28T16:53:02.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.455+0000 7f7ba6a00700 1 -- 172.21.15.1:0/379858652 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7bb004bed0 con 0x7f7b9406d7f0 2024-02-28T16:53:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:53:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (19m) 11m ago 33m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:53:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (15m) 11m ago 33m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:53:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (15m) 12m ago 32m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:53:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (18m) 11m ago 33m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:53:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (11m) 11m ago 26m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:53:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (11m) 11m ago 26m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:53:02.451 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (26m) 12m ago 26m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (26m) 12m ago 26m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (21m) 11m ago 35m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (20m) 12m ago 32m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (17m) 11m ago 35m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (16m) 12m ago 31m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (20m) 11m ago 33m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (20m) 12m ago 32m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (15m) 11m ago 31m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (14m) 11m ago 30m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (14m) 11m ago 29m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (13m) 12m ago 29m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:53:02.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (12m) 12m ago 28m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:53:02.453 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (12m) 12m ago 28m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:53:02.453 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (19m) 11m ago 32m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9406d7f0 msgr2=0x7f7b9406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9406d7f0 0x7f7b9406fca0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0004d60 tx=0x7f7ba000b040 comp rx=0 tx=0).stop 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 msgr2=0x7f7bb0116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 0x7f7bb0116210 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009100 tx=0x7f7b9c009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 shutdown_connections 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9406d7f0 0x7f7b9406fca0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb0105160 0x7f7bb0115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 --2- 172.21.15.1:0/379858652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb0105b00 0x7f7bb0116210 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 >> 172.21.15.1:0/379858652 conn(0x7f7bb00fad30 msgr2=0x7f7bb00fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 shutdown_connections 2024-02-28T16:53:02.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.459+0000 7f7bb5600700 1 -- 172.21.15.1:0/379858652 wait complete. 2024-02-28T16:53:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.614+0000 7f2e3f200700 1 -- 172.21.15.1:0/2041445912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 msgr2=0x7f2e381033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.614+0000 7f2e3f200700 1 --2- 172.21.15.1:0/2041445912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 0x7f2e381033a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2e28004940 tx=0x7f2e28009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.615+0000 7f2e3f200700 1 -- 172.21.15.1:0/2041445912 shutdown_connections 2024-02-28T16:53:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.615+0000 7f2e3f200700 1 --2- 172.21.15.1:0/2041445912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 0x7f2e381033a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.615+0000 7f2e3f200700 1 --2- 172.21.15.1:0/2041445912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e38109380 0x7f2e38109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.615+0000 7f2e3f200700 1 -- 172.21.15.1:0/2041445912 >> 172.21.15.1:0/2041445912 conn(0x7f2e38072b50 msgr2=0x7f2e38074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.615+0000 7f2e3f200700 1 -- 172.21.15.1:0/2041445912 shutdown_connections 2024-02-28T16:53:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.615+0000 7f2e3f200700 1 -- 172.21.15.1:0/2041445912 wait complete. 2024-02-28T16:53:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.615+0000 7f2e3f200700 1 Processor -- start 2024-02-28T16:53:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.616+0000 7f2e3f200700 1 -- start start 2024-02-28T16:53:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.616+0000 7f2e3f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e38109380 0x7f2e38119db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.616+0000 7f2e3f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 0x7f2e3811a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.616+0000 7f2e3f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3811a940 con 0x7f2e38109d20 2024-02-28T16:53:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.616+0000 7f2e3f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e38113c90 con 0x7f2e38109380 2024-02-28T16:53:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.616+0000 7f2e3d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 0x7f2e3811a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.616+0000 7f2e3d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 0x7f2e3811a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40042/0 (socket says 172.21.15.1:40042) 2024-02-28T16:53:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.616+0000 7f2e3d400700 1 -- 172.21.15.1:0/4018151835 learned_addr learned my addr 172.21.15.1:0/4018151835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.617+0000 7f2e3de00700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e38109380 0x7f2e38119db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.617+0000 7f2e3d400700 1 -- 172.21.15.1:0/4018151835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e38109380 msgr2=0x7f2e38119db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.617+0000 7f2e3d400700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e38109380 0x7f2e38119db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.617+0000 7f2e3d400700 1 -- 172.21.15.1:0/4018151835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28005040 con 0x7f2e38109d20 2024-02-28T16:53:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.617+0000 7f2e3d400700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 0x7f2e3811a2f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2e280090d0 tx=0x7f2e28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.617+0000 7f2e32000700 1 -- 172.21.15.1:0/4018151835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e2800e040 con 0x7f2e38109d20 2024-02-28T16:53:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.617+0000 7f2e32000700 1 -- 172.21.15.1:0/4018151835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e2800bbf0 con 0x7f2e38109d20 2024-02-28T16:53:02.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.617+0000 7f2e32000700 1 -- 172.21.15.1:0/4018151835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e2801c630 con 0x7f2e38109d20 2024-02-28T16:53:02.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.618+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e38113f10 con 0x7f2e38109d20 2024-02-28T16:53:02.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.618+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e381144d0 con 0x7f2e38109d20 2024-02-28T16:53:02.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.619+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3805f5c0 con 0x7f2e38109d20 2024-02-28T16:53:02.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.620+0000 7f2e32000700 1 -- 172.21.15.1:0/4018151835 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2e2800bd60 con 0x7f2e38109d20 2024-02-28T16:53:02.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.621+0000 7f2e32000700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2006d7e0 0x7f2e2006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:02.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.621+0000 7f2e32000700 1 -- 172.21.15.1:0/4018151835 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2e2808e1e0 con 0x7f2e38109d20 2024-02-28T16:53:02.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.621+0000 7f2e3de00700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2006d7e0 0x7f2e2006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:02.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.622+0000 7f2e3de00700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2006d7e0 0x7f2e2006fc90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c004bc0 tx=0x7f2e2c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:02.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.625+0000 7f2e32000700 1 -- 172.21.15.1:0/4018151835 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2e28057a90 con 0x7f2e38109d20 2024-02-28T16:53:02.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.949+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2e381153d0 con 0x7f2e38109d20 2024-02-28T16:53:02.945 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:02 smithi001 ceph-mon[112940]: pgmap v557: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:02.945 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:02 smithi001 ceph-mon[112940]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.950+0000 7f2e32000700 1 -- 172.21.15.1:0/4018151835 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f2e28013e60 con 0x7f2e38109d20 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:53:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:53:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:53:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:53:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:53:02.947 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:53:02.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2006d7e0 msgr2=0x7f2e2006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2006d7e0 0x7f2e2006fc90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c004bc0 tx=0x7f2e2c009040 comp rx=0 tx=0).stop 2024-02-28T16:53:02.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 msgr2=0x7f2e3811a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:02.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 0x7f2e3811a2f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2e280090d0 tx=0x7f2e28009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 shutdown_connections 2024-02-28T16:53:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2e2006d7e0 0x7f2e2006fc90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2e38109380 0x7f2e38119db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 --2- 172.21.15.1:0/4018151835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38109d20 0x7f2e3811a2f0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.953+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 >> 172.21.15.1:0/4018151835 conn(0x7f2e38072b50 msgr2=0x7f2e38075600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.954+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 shutdown_connections 2024-02-28T16:53:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:02.954+0000 7f2e3f200700 1 -- 172.21.15.1:0/4018151835 wait complete. 2024-02-28T16:53:03.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:02 smithi066 ceph-mon[85781]: pgmap v557: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:03.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:02 smithi066 ceph-mon[85781]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.112+0000 7f659b200700 1 -- 172.21.15.1:0/3768685694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 msgr2=0x7f6594103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.112+0000 7f659b200700 1 --2- 172.21.15.1:0/3768685694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 0x7f6594103310 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6584004940 tx=0x7f6584009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.112+0000 7f659b200700 1 -- 172.21.15.1:0/3768685694 shutdown_connections 2024-02-28T16:53:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.112+0000 7f659b200700 1 --2- 172.21.15.1:0/3768685694 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65941038e0 0x7f65940fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.112+0000 7f659b200700 1 --2- 172.21.15.1:0/3768685694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 0x7f6594103310 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.112+0000 7f659b200700 1 -- 172.21.15.1:0/3768685694 >> 172.21.15.1:0/3768685694 conn(0x7f65940f8af0 msgr2=0x7f65940faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.112+0000 7f659b200700 1 -- 172.21.15.1:0/3768685694 shutdown_connections 2024-02-28T16:53:03.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.112+0000 7f659b200700 1 -- 172.21.15.1:0/3768685694 wait complete. 2024-02-28T16:53:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.113+0000 7f659b200700 1 Processor -- start 2024-02-28T16:53:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.113+0000 7f659b200700 1 -- start start 2024-02-28T16:53:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.113+0000 7f659b200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 0x7f6594115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.113+0000 7f659b200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65941038e0 0x7f65941161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.113+0000 7f659b200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6594116800 con 0x7f6594102f40 2024-02-28T16:53:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.113+0000 7f659b200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659410fb30 con 0x7f65941038e0 2024-02-28T16:53:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.114+0000 7f6593e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 0x7f6594115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.114+0000 7f6593e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 0x7f6594115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40058/0 (socket says 172.21.15.1:40058) 2024-02-28T16:53:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.114+0000 7f6593e00700 1 -- 172.21.15.1:0/1717700098 learned_addr learned my addr 172.21.15.1:0/1717700098 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.114+0000 7f658be00700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65941038e0 0x7f65941161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.114+0000 7f6593e00700 1 -- 172.21.15.1:0/1717700098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65941038e0 msgr2=0x7f65941161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.114+0000 7f6593e00700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65941038e0 0x7f65941161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.114+0000 7f6593e00700 1 -- 172.21.15.1:0/1717700098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584005040 con 0x7f6594102f40 2024-02-28T16:53:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.114+0000 7f6593e00700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 0x7f6594115c70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6584009fd0 tx=0x7f6584009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.115+0000 7f6590c00700 1 -- 172.21.15.1:0/1717700098 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f658400e040 con 0x7f6594102f40 2024-02-28T16:53:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.115+0000 7f6590c00700 1 -- 172.21.15.1:0/1717700098 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f658400bbf0 con 0x7f6594102f40 2024-02-28T16:53:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.115+0000 7f6590c00700 1 -- 172.21.15.1:0/1717700098 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f658401c3f0 con 0x7f6594102f40 2024-02-28T16:53:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.115+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659410fdb0 con 0x7f6594102f40 2024-02-28T16:53:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.115+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65941102f0 con 0x7f6594102f40 2024-02-28T16:53:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.116+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65940fc6f0 con 0x7f6594102f40 2024-02-28T16:53:03.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.117+0000 7f6590c00700 1 -- 172.21.15.1:0/1717700098 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f658400bd60 con 0x7f6594102f40 2024-02-28T16:53:03.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.118+0000 7f6590c00700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f657806d7f0 0x7f657806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.118+0000 7f6590c00700 1 -- 172.21.15.1:0/1717700098 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f658408e1c0 con 0x7f6594102f40 2024-02-28T16:53:03.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.118+0000 7f658be00700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f657806d7f0 0x7f657806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.119+0000 7f658be00700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f657806d7f0 0x7f657806fca0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f65941114a0 tx=0x7f657c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.123+0000 7f6590c00700 1 -- 172.21.15.1:0/1717700098 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f658405b5a0 con 0x7f6594102f40 2024-02-28T16:53:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.397+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f659410a0e0 con 0x7f6594102f40 2024-02-28T16:53:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.398+0000 7f6590c00700 1 -- 172.21.15.1:0/1717700098 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f659410a0e0 con 0x7f6594102f40 2024-02-28T16:53:03.393 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:53:03.393 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:53:03.393 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:53:03.393 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:53:03.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:53:03.393 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:53:03.393 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:53:03.394 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:53:03.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:53:03.396 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:53:03.396 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:53:03.396 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:53:03.396 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:53:03.396 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:53:03.396 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.400+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f657806d7f0 msgr2=0x7f657806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.400+0000 7f659b200700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f657806d7f0 0x7f657806fca0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f65941114a0 tx=0x7f657c00b040 comp rx=0 tx=0).stop 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.400+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 msgr2=0x7f6594115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.400+0000 7f659b200700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 0x7f6594115c70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6584009fd0 tx=0x7f6584009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.401+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 shutdown_connections 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.401+0000 7f659b200700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f657806d7f0 0x7f657806fca0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.401+0000 7f659b200700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6594102f40 0x7f6594115c70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.401+0000 7f659b200700 1 --2- 172.21.15.1:0/1717700098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f65941038e0 0x7f65941161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.401+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 >> 172.21.15.1:0/1717700098 conn(0x7f65940f8af0 msgr2=0x7f65940fa380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.401+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 shutdown_connections 2024-02-28T16:53:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.401+0000 7f659b200700 1 -- 172.21.15.1:0/1717700098 wait complete. 2024-02-28T16:53:03.398 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:53:03.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.558+0000 7fb265000700 1 -- 172.21.15.1:0/4193363671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070c40 msgr2=0x7fb260109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.558+0000 7fb265000700 1 --2- 172.21.15.1:0/4193363671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070c40 0x7fb260109260 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb24c004970 tx=0x7fb24c009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:03.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.558+0000 7fb265000700 1 -- 172.21.15.1:0/4193363671 shutdown_connections 2024-02-28T16:53:03.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.558+0000 7fb265000700 1 --2- 172.21.15.1:0/4193363671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070c40 0x7fb260109260 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.558+0000 7fb265000700 1 --2- 172.21.15.1:0/4193363671 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070330 0x7fb260070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.558+0000 7fb265000700 1 -- 172.21.15.1:0/4193363671 >> 172.21.15.1:0/4193363671 conn(0x7fb2600f8ae0 msgr2=0x7fb2600faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:03.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.559+0000 7fb265000700 1 -- 172.21.15.1:0/4193363671 shutdown_connections 2024-02-28T16:53:03.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.559+0000 7fb265000700 1 -- 172.21.15.1:0/4193363671 wait complete. 2024-02-28T16:53:03.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.560+0000 7fb265000700 1 Processor -- start 2024-02-28T16:53:03.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.560+0000 7fb265000700 1 -- start start 2024-02-28T16:53:03.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.560+0000 7fb265000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070330 0x7fb260115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.560+0000 7fb265000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070c40 0x7fb260116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.560+0000 7fb265000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb260116790 con 0x7fb260070330 2024-02-28T16:53:03.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.560+0000 7fb265000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26010fae0 con 0x7fb260070c40 2024-02-28T16:53:03.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.561+0000 7fb25de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070330 0x7fb260115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.561+0000 7fb25de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070330 0x7fb260115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40074/0 (socket says 172.21.15.1:40074) 2024-02-28T16:53:03.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.561+0000 7fb25de00700 1 -- 172.21.15.1:0/3044412535 learned_addr learned my addr 172.21.15.1:0/3044412535 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:03.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.561+0000 7fb25d400700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070c40 0x7fb260116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.561+0000 7fb25de00700 1 -- 172.21.15.1:0/3044412535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070c40 msgr2=0x7fb260116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.561+0000 7fb25de00700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070c40 0x7fb260116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.561+0000 7fb25de00700 1 -- 172.21.15.1:0/3044412535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c005040 con 0x7fb260070330 2024-02-28T16:53:03.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.561+0000 7fb25de00700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070330 0x7fb260115c00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb250002a50 tx=0x7fb2500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:03.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.562+0000 7fb256000700 1 -- 172.21.15.1:0/3044412535 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb250007cc0 con 0x7fb260070330 2024-02-28T16:53:03.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.562+0000 7fb256000700 1 -- 172.21.15.1:0/3044412535 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb25000f040 con 0x7fb260070330 2024-02-28T16:53:03.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.562+0000 7fb256000700 1 -- 172.21.15.1:0/3044412535 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb250009640 con 0x7fb260070330 2024-02-28T16:53:03.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.562+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26010fdc0 con 0x7fb260070330 2024-02-28T16:53:03.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.562+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb260110300 con 0x7fb260070330 2024-02-28T16:53:03.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.563+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb260106920 con 0x7fb260070330 2024-02-28T16:53:03.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.564+0000 7fb256000700 1 -- 172.21.15.1:0/3044412535 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb2500097a0 con 0x7fb260070330 2024-02-28T16:53:03.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.565+0000 7fb256000700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.565+0000 7fb256000700 1 -- 172.21.15.1:0/3044412535 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb25008cc60 con 0x7fb260070330 2024-02-28T16:53:03.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.565+0000 7fb25d400700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.566+0000 7fb25d400700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb24c000c00 tx=0x7fb24c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:03.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.570+0000 7fb256000700 1 -- 172.21.15.1:0/3044412535 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb250056490 con 0x7fb260070330 2024-02-28T16:53:03.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.814+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2601111e0 con 0x7fb24406d7f0 2024-02-28T16:53:03.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.817+0000 7fb256000700 1 -- 172.21.15.1:0/3044412535 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb2601111e0 con 0x7fb24406d7f0 2024-02-28T16:53:03.812 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:53:03.813 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 msgr2=0x7fb24406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb24c000c00 tx=0x7fb24c009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070330 msgr2=0x7fb260115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070330 0x7fb260115c00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb250002a50 tx=0x7fb2500046b0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 shutdown_connections 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb260070330 0x7fb260115c00 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 --2- 172.21.15.1:0/3044412535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070c40 0x7fb260116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.820+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 >> 172.21.15.1:0/3044412535 conn(0x7fb2600f8ae0 msgr2=0x7fb260103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.821+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 shutdown_connections 2024-02-28T16:53:03.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.821+0000 7fb265000700 1 -- 172.21.15.1:0/3044412535 wait complete. 2024-02-28T16:53:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.981+0000 7fbddfa00700 1 -- 172.21.15.1:0/440443556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fe8e0 msgr2=0x7fbdd8106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.981+0000 7fbddfa00700 1 --2- 172.21.15.1:0/440443556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fe8e0 0x7fbdd8106f20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4004940 tx=0x7fbdc4009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.981+0000 7fbddfa00700 1 -- 172.21.15.1:0/440443556 shutdown_connections 2024-02-28T16:53:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.981+0000 7fbddfa00700 1 --2- 172.21.15.1:0/440443556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fe8e0 0x7fbdd8106f20 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.981+0000 7fbddfa00700 1 --2- 172.21.15.1:0/440443556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbdd80fdf40 0x7fbdd80fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.981+0000 7fbddfa00700 1 -- 172.21.15.1:0/440443556 >> 172.21.15.1:0/440443556 conn(0x7fbdd80f79d0 msgr2=0x7fbdd80f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:03.978 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:03 smithi001 ceph-mon[112940]: from='client.35038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:03.978 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:03 smithi001 ceph-mon[112940]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:03.978 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:03 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4018151835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:53:03.978 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:03 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1717700098' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:53:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.982+0000 7fbddfa00700 1 -- 172.21.15.1:0/440443556 shutdown_connections 2024-02-28T16:53:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.982+0000 7fbddfa00700 1 -- 172.21.15.1:0/440443556 wait complete. 2024-02-28T16:53:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.983+0000 7fbddfa00700 1 Processor -- start 2024-02-28T16:53:03.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.983+0000 7fbddfa00700 1 -- start start 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.983+0000 7fbddfa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fdf40 0x7fbdd8115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.983+0000 7fbddfa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbdd80fe8e0 0x7fbdd8116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.983+0000 7fbddfa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd8116790 con 0x7fbdd80fdf40 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.983+0000 7fbddfa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd810fac0 con 0x7fbdd80fe8e0 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.984+0000 7fbddd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fdf40 0x7fbdd8115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.984+0000 7fbddd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fdf40 0x7fbdd8115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40086/0 (socket says 172.21.15.1:40086) 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.984+0000 7fbddd000700 1 -- 172.21.15.1:0/466512846 learned_addr learned my addr 172.21.15.1:0/466512846 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.984+0000 7fbdd7e00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbdd80fe8e0 0x7fbdd8116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.984+0000 7fbddd000700 1 -- 172.21.15.1:0/466512846 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbdd80fe8e0 msgr2=0x7fbdd8116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.984+0000 7fbddd000700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbdd80fe8e0 0x7fbdd8116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:03.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.984+0000 7fbddd000700 1 -- 172.21.15.1:0/466512846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc4005040 con 0x7fbdd80fdf40 2024-02-28T16:53:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.985+0000 7fbddd000700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fdf40 0x7fbdd8115c00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00c8c0 tx=0x7fbdcc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.985+0000 7fbdd4c00700 1 -- 172.21.15.1:0/466512846 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdcc00a980 con 0x7fbdd80fdf40 2024-02-28T16:53:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.985+0000 7fbdd4c00700 1 -- 172.21.15.1:0/466512846 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdcc00ce60 con 0x7fbdd80fdf40 2024-02-28T16:53:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.985+0000 7fbdd4c00700 1 -- 172.21.15.1:0/466512846 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdcc0115e0 con 0x7fbdd80fdf40 2024-02-28T16:53:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.985+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd810fda0 con 0x7fbdd80fdf40 2024-02-28T16:53:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.985+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd81102e0 con 0x7fbdd80fdf40 2024-02-28T16:53:03.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.987+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd8106930 con 0x7fbdd80fdf40 2024-02-28T16:53:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.987+0000 7fbdd4c00700 1 -- 172.21.15.1:0/466512846 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbdcc012400 con 0x7fbdd80fdf40 2024-02-28T16:53:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.988+0000 7fbdd4c00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdc006d7f0 0x7fbdc006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:03.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.989+0000 7fbdd4c00700 1 -- 172.21.15.1:0/466512846 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbdcc08ceb0 con 0x7fbdd80fdf40 2024-02-28T16:53:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.989+0000 7fbdd7e00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdc006d7f0 0x7fbdc006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:03.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.989+0000 7fbdd7e00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdc006d7f0 0x7fbdc006fca0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8111430 tx=0x7fbdc4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:03.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:03.994+0000 7fbdd4c00700 1 -- 172.21.15.1:0/466512846 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbdcc05a350 con 0x7fbdd80fdf40 2024-02-28T16:53:04.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:03 smithi066 ceph-mon[85781]: from='client.35038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:04.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:03 smithi066 ceph-mon[85781]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:04.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:03 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4018151835' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:53:04.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:03 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1717700098' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:53:04.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.310+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbdd804bed0 con 0x7fbdd80fdf40 2024-02-28T16:53:04.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.311+0000 7fbdd4c00700 1 -- 172.21.15.1:0/466512846 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fbdcc0126b0 con 0x7fbdd80fdf40 2024-02-28T16:53:04.306 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:53:04.306 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:53:04.306 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:53:04.306 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:53:04.306 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:53:04.306 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:53:04.306 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:53:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdc006d7f0 msgr2=0x7fbdc006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdc006d7f0 0x7fbdc006fca0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8111430 tx=0x7fbdc4009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fdf40 msgr2=0x7fbdd8115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fdf40 0x7fbdd8115c00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00c8c0 tx=0x7fbdcc0044c0 comp rx=0 tx=0).stop 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 shutdown_connections 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbdc006d7f0 0x7fbdc006fca0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdd80fdf40 0x7fbdd8115c00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 --2- 172.21.15.1:0/466512846 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbdd80fe8e0 0x7fbdd8116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.313+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 >> 172.21.15.1:0/466512846 conn(0x7fbdd80f79d0 msgr2=0x7fbdd8103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.314+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 shutdown_connections 2024-02-28T16:53:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:04.314+0000 7fbddfa00700 1 -- 172.21.15.1:0/466512846 wait complete. 2024-02-28T16:53:05.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:04 smithi066 ceph-mon[85781]: pgmap v558: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:05.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:04 smithi066 ceph-mon[85781]: from='client.35054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:05.004 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:04 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/466512846' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:53:05.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:04 smithi001 ceph-mon[112940]: pgmap v558: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:05.026 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:04 smithi001 ceph-mon[112940]: from='client.35054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:05.026 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:04 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/466512846' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:53:06.003 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:05 smithi066 ceph-mon[85781]: pgmap v559: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:06.025 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:05 smithi001 ceph-mon[112940]: pgmap v559: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:08.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:08 smithi066 ceph-mon[85781]: pgmap v560: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:08.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:08 smithi001 ceph-mon[112940]: pgmap v560: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:10.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:10 smithi066 ceph-mon[85781]: pgmap v561: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:10.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:10 smithi001 ceph-mon[112940]: pgmap v561: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:11.503 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:53:11.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:53:12.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:12 smithi066 ceph-mon[85781]: pgmap v562: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:12.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:12 smithi001 ceph-mon[112940]: pgmap v562: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:14.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:14 smithi066 ceph-mon[85781]: pgmap v563: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:14.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:14 smithi001 ceph-mon[112940]: pgmap v563: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:16.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:16 smithi066 ceph-mon[85781]: pgmap v564: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:16.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:16 smithi001 ceph-mon[112940]: pgmap v564: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:18.169 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:18 smithi001 ceph-mon[112940]: pgmap v565: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:18.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:18 smithi066 ceph-mon[85781]: pgmap v565: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:20.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:20 smithi066 ceph-mon[85781]: pgmap v566: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:20.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:20 smithi001 ceph-mon[112940]: pgmap v566: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:22.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:22 smithi066 ceph-mon[85781]: pgmap v567: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:22.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:22 smithi001 ceph-mon[112940]: pgmap v567: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:24.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:24 smithi066 ceph-mon[85781]: pgmap v568: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:24.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:24 smithi001 ceph-mon[112940]: pgmap v568: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:26.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:26 smithi066 ceph-mon[85781]: pgmap v569: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:26.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:53:26.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:26 smithi001 ceph-mon[112940]: pgmap v569: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:26.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:53:28.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:28 smithi066 ceph-mon[85781]: pgmap v570: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:28.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:28 smithi001 ceph-mon[112940]: pgmap v570: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:30.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:30 smithi066 ceph-mon[85781]: pgmap v571: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:30.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:30 smithi001 ceph-mon[112940]: pgmap v571: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:32.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:32 smithi066 ceph-mon[85781]: pgmap v572: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:32.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:32 smithi001 ceph-mon[112940]: pgmap v572: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:34.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.488+0000 7f6bc7800700 1 -- 172.21.15.1:0/2045107643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070330 msgr2=0x7f6bc0070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.488+0000 7f6bc7800700 1 --2- 172.21.15.1:0/2045107643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070330 0x7f6bc0070700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4004970 tx=0x7f6bb4009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.488+0000 7f6bc7800700 1 -- 172.21.15.1:0/2045107643 shutdown_connections 2024-02-28T16:53:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.488+0000 7f6bc7800700 1 --2- 172.21.15.1:0/2045107643 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bc0070c40 0x7f6bc0109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.488+0000 7f6bc7800700 1 --2- 172.21.15.1:0/2045107643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070330 0x7f6bc0070700 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.488+0000 7f6bc7800700 1 -- 172.21.15.1:0/2045107643 >> 172.21.15.1:0/2045107643 conn(0x7f6bc00f8ae0 msgr2=0x7f6bc00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:34.483 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:34 smithi001 ceph-mon[112940]: pgmap v573: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:34.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.488+0000 7f6bc7800700 1 -- 172.21.15.1:0/2045107643 shutdown_connections 2024-02-28T16:53:34.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.488+0000 7f6bc7800700 1 -- 172.21.15.1:0/2045107643 wait complete. 2024-02-28T16:53:34.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.490+0000 7f6bc7800700 1 Processor -- start 2024-02-28T16:53:34.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.490+0000 7f6bc7800700 1 -- start start 2024-02-28T16:53:34.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.490+0000 7f6bc7800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bc0070330 0x7f6bc0115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:34.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.490+0000 7f6bc7800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070c40 0x7f6bc0116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:34.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bc7800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc01167b0 con 0x7f6bc0070c40 2024-02-28T16:53:34.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bc7800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc010fae0 con 0x7f6bc0070330 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bbfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070c40 0x7f6bc0116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bbfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070c40 0x7f6bc0116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33970/0 (socket says 172.21.15.1:33970) 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bbfe00700 1 -- 172.21.15.1:0/527688331 learned_addr learned my addr 172.21.15.1:0/527688331 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bc4e00700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bc0070330 0x7f6bc0115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bbfe00700 1 -- 172.21.15.1:0/527688331 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bc0070330 msgr2=0x7f6bc0115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bbfe00700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bc0070330 0x7f6bc0115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bbfe00700 1 -- 172.21.15.1:0/527688331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb4005040 con 0x7f6bc0070c40 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.491+0000 7f6bc4e00700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bc0070330 0x7f6bc0115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.492+0000 7f6bbfe00700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070c40 0x7f6bc0116160 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6bac002a50 tx=0x7f6bac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.492+0000 7f6bbcc00700 1 -- 172.21.15.1:0/527688331 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bac007cc0 con 0x7f6bc0070c40 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.492+0000 7f6bbcc00700 1 -- 172.21.15.1:0/527688331 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bac00f040 con 0x7f6bc0070c40 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.492+0000 7f6bbcc00700 1 -- 172.21.15.1:0/527688331 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bac009640 con 0x7f6bc0070c40 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.492+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc010fdc0 con 0x7f6bc0070c40 2024-02-28T16:53:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.492+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc0110300 con 0x7f6bc0070c40 2024-02-28T16:53:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.493+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc0106920 con 0x7f6bc0070c40 2024-02-28T16:53:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.495+0000 7f6bbcc00700 1 -- 172.21.15.1:0/527688331 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6bac0097a0 con 0x7f6bc0070c40 2024-02-28T16:53:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.496+0000 7f6bbcc00700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ba806d7f0 0x7f6ba806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.496+0000 7f6bbcc00700 1 -- 172.21.15.1:0/527688331 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6bac08cc60 con 0x7f6bc0070c40 2024-02-28T16:53:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.496+0000 7f6bc4e00700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ba806d7f0 0x7f6ba806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.497+0000 7f6bc4e00700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ba806d7f0 0x7f6ba806fca0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4000c00 tx=0x7f6bb4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.501+0000 7f6bbcc00700 1 -- 172.21.15.1:0/527688331 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6bac059fc0 con 0x7f6bc0070c40 2024-02-28T16:53:34.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:34 smithi066 ceph-mon[85781]: pgmap v573: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:34.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.742+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bc01158f0 con 0x7f6ba806d7f0 2024-02-28T16:53:34.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.745+0000 7f6bbcc00700 1 -- 172.21.15.1:0/527688331 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6bc01158f0 con 0x7f6ba806d7f0 2024-02-28T16:53:34.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ba806d7f0 msgr2=0x7f6ba806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:34.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ba806d7f0 0x7f6ba806fca0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4000c00 tx=0x7f6bb4009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:34.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070c40 msgr2=0x7f6bc0116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:34.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070c40 0x7f6bc0116160 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6bac002a50 tx=0x7f6bac0046b0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 shutdown_connections 2024-02-28T16:53:34.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ba806d7f0 0x7f6ba806fca0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6bc0070330 0x7f6bc0115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 --2- 172.21.15.1:0/527688331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc0070c40 0x7f6bc0116160 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.749+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 >> 172.21.15.1:0/527688331 conn(0x7f6bc00f8ae0 msgr2=0x7f6bc0103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:34.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.750+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 shutdown_connections 2024-02-28T16:53:34.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.750+0000 7f6bc7800700 1 -- 172.21.15.1:0/527688331 wait complete. 2024-02-28T16:53:34.759 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:53:34.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.913+0000 7f13cb200700 1 -- 172.21.15.1:0/496672101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 msgr2=0x7f13c4105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:34.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.913+0000 7f13cb200700 1 --2- 172.21.15.1:0/496672101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 0x7f13c4105540 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f13b4004940 tx=0x7f13b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:34.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.913+0000 7f13cb200700 1 -- 172.21.15.1:0/496672101 shutdown_connections 2024-02-28T16:53:34.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.913+0000 7f13cb200700 1 --2- 172.21.15.1:0/496672101 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13c4105b10 0x7f13c40ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.913+0000 7f13cb200700 1 --2- 172.21.15.1:0/496672101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 0x7f13c4105540 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.913+0000 7f13cb200700 1 -- 172.21.15.1:0/496672101 >> 172.21.15.1:0/496672101 conn(0x7f13c40face0 msgr2=0x7f13c40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.913+0000 7f13cb200700 1 -- 172.21.15.1:0/496672101 shutdown_connections 2024-02-28T16:53:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.913+0000 7f13cb200700 1 -- 172.21.15.1:0/496672101 wait complete. 2024-02-28T16:53:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.914+0000 7f13cb200700 1 Processor -- start 2024-02-28T16:53:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.915+0000 7f13cb200700 1 -- start start 2024-02-28T16:53:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.915+0000 7f13cb200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 0x7f13c4115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.915+0000 7f13cb200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13c4105b10 0x7f13c41161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.915+0000 7f13cb200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c41168d0 con 0x7f13c4105170 2024-02-28T16:53:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.915+0000 7f13cb200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c410fb50 con 0x7f13c4105b10 2024-02-28T16:53:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.915+0000 7f13c3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 0x7f13c4115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.915+0000 7f13c3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 0x7f13c4115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33996/0 (socket says 172.21.15.1:33996) 2024-02-28T16:53:34.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.915+0000 7f13c3e00700 1 -- 172.21.15.1:0/2985385482 learned_addr learned my addr 172.21.15.1:0/2985385482 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:34.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.916+0000 7f13c3400700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13c4105b10 0x7f13c41161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:34.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.916+0000 7f13c3e00700 1 -- 172.21.15.1:0/2985385482 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13c4105b10 msgr2=0x7f13c41161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:34.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.916+0000 7f13c3e00700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13c4105b10 0x7f13c41161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.916+0000 7f13c3e00700 1 -- 172.21.15.1:0/2985385482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4005040 con 0x7f13c4105170 2024-02-28T16:53:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.916+0000 7f13c3e00700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 0x7f13c4115cb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009150 tx=0x7f13b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.916+0000 7f13bbe00700 1 -- 172.21.15.1:0/2985385482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13b400e040 con 0x7f13c4105170 2024-02-28T16:53:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.916+0000 7f13bbe00700 1 -- 172.21.15.1:0/2985385482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13b400bbf0 con 0x7f13c4105170 2024-02-28T16:53:34.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.917+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c410fd70 con 0x7f13c4105170 2024-02-28T16:53:34.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.917+0000 7f13bbe00700 1 -- 172.21.15.1:0/2985385482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13b401c630 con 0x7f13c4105170 2024-02-28T16:53:34.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.917+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c41102b0 con 0x7f13c4105170 2024-02-28T16:53:34.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.918+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c40fe900 con 0x7f13c4105170 2024-02-28T16:53:34.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.919+0000 7f13bbe00700 1 -- 172.21.15.1:0/2985385482 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f13b400bd60 con 0x7f13c4105170 2024-02-28T16:53:34.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.920+0000 7f13bbe00700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f13a806d7f0 0x7f13a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:34.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.920+0000 7f13bbe00700 1 -- 172.21.15.1:0/2985385482 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f13b408e360 con 0x7f13c4105170 2024-02-28T16:53:34.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.920+0000 7f13c3400700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f13a806d7f0 0x7f13a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:34.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.921+0000 7f13c3400700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f13a806d7f0 0x7f13a806fca0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f13b0006fd0 tx=0x7f13b0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:34.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:34.924+0000 7f13bbe00700 1 -- 172.21.15.1:0/2985385482 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f13b405b6c0 con 0x7f13c4105170 2024-02-28T16:53:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.169+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13c41113a0 con 0x7f13a806d7f0 2024-02-28T16:53:35.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.171+0000 7f13bbe00700 1 -- 172.21.15.1:0/2985385482 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f13c41113a0 con 0x7f13a806d7f0 2024-02-28T16:53:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f13a806d7f0 msgr2=0x7f13a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f13a806d7f0 0x7f13a806fca0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f13b0006fd0 tx=0x7f13b0008040 comp rx=0 tx=0).stop 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 msgr2=0x7f13c4115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 0x7f13c4115cb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009150 tx=0x7f13b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 shutdown_connections 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f13a806d7f0 0x7f13a806fca0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13c4105170 0x7f13c4115cb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 --2- 172.21.15.1:0/2985385482 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13c4105b10 0x7f13c41161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 >> 172.21.15.1:0/2985385482 conn(0x7f13c40face0 msgr2=0x7f13c40fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 shutdown_connections 2024-02-28T16:53:35.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.174+0000 7f13cb200700 1 -- 172.21.15.1:0/2985385482 wait complete. 2024-02-28T16:53:35.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.337+0000 7fb4c7800700 1 -- 172.21.15.1:0/2023292665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 msgr2=0x7fb4c00fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:35.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.337+0000 7fb4c7800700 1 --2- 172.21.15.1:0/2023292665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 0x7fb4c00fe380 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4004940 tx=0x7fb4b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:35.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.337+0000 7fb4c7800700 1 -- 172.21.15.1:0/2023292665 shutdown_connections 2024-02-28T16:53:35.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.337+0000 7fb4c7800700 1 --2- 172.21.15.1:0/2023292665 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4c00fe950 0x7fb4c0106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.337+0000 7fb4c7800700 1 --2- 172.21.15.1:0/2023292665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 0x7fb4c00fe380 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.337+0000 7fb4c7800700 1 -- 172.21.15.1:0/2023292665 >> 172.21.15.1:0/2023292665 conn(0x7fb4c00f79e0 msgr2=0x7fb4c00f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:35.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.338+0000 7fb4c7800700 1 -- 172.21.15.1:0/2023292665 shutdown_connections 2024-02-28T16:53:35.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.338+0000 7fb4c7800700 1 -- 172.21.15.1:0/2023292665 wait complete. 2024-02-28T16:53:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.339+0000 7fb4c7800700 1 Processor -- start 2024-02-28T16:53:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.339+0000 7fb4c7800700 1 -- start start 2024-02-28T16:53:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.339+0000 7fb4c7800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 0x7fb4c0115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.339+0000 7fb4c7800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4c00fe950 0x7fb4c0116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.339+0000 7fb4c7800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c0116790 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.339+0000 7fb4c7800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c010fac0 con 0x7fb4c00fe950 2024-02-28T16:53:35.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.339+0000 7fb4c4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 0x7fb4c0115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:35.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.340+0000 7fb4c4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 0x7fb4c0115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34016/0 (socket says 172.21.15.1:34016) 2024-02-28T16:53:35.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.340+0000 7fb4c4e00700 1 -- 172.21.15.1:0/1547291798 learned_addr learned my addr 172.21.15.1:0/1547291798 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:35.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.340+0000 7fb4bfe00700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4c00fe950 0x7fb4c0116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:35.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.340+0000 7fb4c4e00700 1 -- 172.21.15.1:0/1547291798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4c00fe950 msgr2=0x7fb4c0116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:35.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.340+0000 7fb4c4e00700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4c00fe950 0x7fb4c0116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.340+0000 7fb4c4e00700 1 -- 172.21.15.1:0/1547291798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4005040 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.340+0000 7fb4c4e00700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 0x7fb4c0115c00 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4000c00 tx=0x7fb4b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.341+0000 7fb4bcc00700 1 -- 172.21.15.1:0/1547291798 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b400e040 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.341+0000 7fb4bcc00700 1 -- 172.21.15.1:0/1547291798 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4b4009af0 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.341+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c010fda0 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.341+0000 7fb4bcc00700 1 -- 172.21.15.1:0/1547291798 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b401c630 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.341+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c01102e0 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.342+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c01069a0 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.343+0000 7fb4bcc00700 1 -- 172.21.15.1:0/1547291798 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb4b400bbf0 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.344+0000 7fb4bcc00700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb4a806d7e0 0x7fb4a806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:35.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.344+0000 7fb4bcc00700 1 -- 172.21.15.1:0/1547291798 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb4b408e1a0 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.344+0000 7fb4bfe00700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb4a806d7e0 0x7fb4a806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:35.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.345+0000 7fb4bfe00700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb4a806d7e0 0x7fb4a806fc90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0111430 tx=0x7fb4ac00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:35.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.348+0000 7fb4bcc00700 1 -- 172.21.15.1:0/1547291798 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb4b405b640 con 0x7fb4c00fdfb0 2024-02-28T16:53:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.584+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4c004bf40 con 0x7fb4a806d7e0 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.600+0000 7fb4bcc00700 1 -- 172.21.15.1:0/1547291798 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb4c004bf40 con 0x7fb4a806d7e0 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (19m) 11m ago 33m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (16m) 11m ago 34m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (16m) 12m ago 32m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (19m) 11m ago 33m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (12m) 11m ago 26m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (11m) 11m ago 26m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (26m) 12m ago 26m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:53:35.595 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (26m) 12m ago 26m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (22m) 11m ago 35m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (21m) 12m ago 32m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (18m) 11m ago 35m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (17m) 12m ago 32m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (20m) 11m ago 34m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (20m) 12m ago 32m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (15m) 11m ago 31m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (15m) 11m ago 31m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (14m) 11m ago 30m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (14m) 12m ago 29m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (13m) 12m ago 29m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:53:35.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (12m) 12m ago 28m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:53:35.597 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (20m) 11m ago 33m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:53:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb4a806d7e0 msgr2=0x7fb4a806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb4a806d7e0 0x7fb4a806fc90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0111430 tx=0x7fb4ac00b040 comp rx=0 tx=0).stop 2024-02-28T16:53:35.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 msgr2=0x7fb4c0115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 0x7fb4c0115c00 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4000c00 tx=0x7fb4b4009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 shutdown_connections 2024-02-28T16:53:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb4a806d7e0 0x7fb4a806fc90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c00fdfb0 0x7fb4c0115c00 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 --2- 172.21.15.1:0/1547291798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4c00fe950 0x7fb4c0116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 >> 172.21.15.1:0/1547291798 conn(0x7fb4c00f79e0 msgr2=0x7fb4c0103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 shutdown_connections 2024-02-28T16:53:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.603+0000 7fb4c7800700 1 -- 172.21.15.1:0/1547291798 wait complete. 2024-02-28T16:53:35.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.753+0000 7f876f200700 1 -- 172.21.15.1:0/2236355242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 msgr2=0x7f8770105370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:35.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.753+0000 7f876f200700 1 --2- 172.21.15.1:0/2236355242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 0x7f8770105370 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8760004940 tx=0x7f8760009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:35.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.754+0000 7f876f200700 1 -- 172.21.15.1:0/2236355242 shutdown_connections 2024-02-28T16:53:35.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.754+0000 7f876f200700 1 --2- 172.21.15.1:0/2236355242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8770105940 0x7f87700fefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.754+0000 7f876f200700 1 --2- 172.21.15.1:0/2236355242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 0x7f8770105370 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.754+0000 7f876f200700 1 -- 172.21.15.1:0/2236355242 >> 172.21.15.1:0/2236355242 conn(0x7f87700fab10 msgr2=0x7f87700fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:35.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.754+0000 7f876f200700 1 -- 172.21.15.1:0/2236355242 shutdown_connections 2024-02-28T16:53:35.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.754+0000 7f876f200700 1 -- 172.21.15.1:0/2236355242 wait complete. 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.755+0000 7f876f200700 1 Processor -- start 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.755+0000 7f876f200700 1 -- start start 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.755+0000 7f876f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 0x7f8770115a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.755+0000 7f876f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8770105940 0x7f8770115fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.755+0000 7f876f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87701165f0 con 0x7f8770104fa0 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.755+0000 7f876f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877010f920 con 0x7f8770105940 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.755+0000 7f876de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 0x7f8770115a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.756+0000 7f876de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 0x7f8770115a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34034/0 (socket says 172.21.15.1:34034) 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.756+0000 7f876de00700 1 -- 172.21.15.1:0/1423246639 learned_addr learned my addr 172.21.15.1:0/1423246639 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.756+0000 7f876d400700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8770105940 0x7f8770115fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.756+0000 7f876de00700 1 -- 172.21.15.1:0/1423246639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8770105940 msgr2=0x7f8770115fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:35.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.756+0000 7f876de00700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8770105940 0x7f8770115fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.756+0000 7f876de00700 1 -- 172.21.15.1:0/1423246639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8760005040 con 0x7f8770104fa0 2024-02-28T16:53:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.756+0000 7f876de00700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 0x7f8770115a60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8760009150 tx=0x7f8760009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.757+0000 7f8766000700 1 -- 172.21.15.1:0/1423246639 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f876000e040 con 0x7f8770104fa0 2024-02-28T16:53:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.757+0000 7f8766000700 1 -- 172.21.15.1:0/1423246639 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f876000bbf0 con 0x7f8770104fa0 2024-02-28T16:53:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.757+0000 7f8766000700 1 -- 172.21.15.1:0/1423246639 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f876001c630 con 0x7f8770104fa0 2024-02-28T16:53:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.757+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877010fba0 con 0x7f8770104fa0 2024-02-28T16:53:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.757+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8770110160 con 0x7f8770104fa0 2024-02-28T16:53:35.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.758+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877005f5c0 con 0x7f8770104fa0 2024-02-28T16:53:35.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.759+0000 7f8766000700 1 -- 172.21.15.1:0/1423246639 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f876000bd60 con 0x7f8770104fa0 2024-02-28T16:53:35.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.760+0000 7f8766000700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875406d7f0 0x7f875406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:35.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.760+0000 7f8766000700 1 -- 172.21.15.1:0/1423246639 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f876008e290 con 0x7f8770104fa0 2024-02-28T16:53:35.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.760+0000 7f876d400700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875406d7f0 0x7f875406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:35.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.761+0000 7f876d400700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875406d7f0 0x7f875406fca0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f875c007fd0 tx=0x7f875c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:35.765+0000 7f8766000700 1 -- 172.21.15.1:0/1423246639 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f876005b670 con 0x7f8770104fa0 2024-02-28T16:53:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.069+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8770111040 con 0x7f8770104fa0 2024-02-28T16:53:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.070+0000 7f8766000700 1 -- 172.21.15.1:0/1423246639 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f876005b200 con 0x7f8770104fa0 2024-02-28T16:53:36.065 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:53:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:53:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:53:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:53:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:53:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:53:36.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:53:36.066 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875406d7f0 msgr2=0x7f875406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875406d7f0 0x7f875406fca0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f875c007fd0 tx=0x7f875c009040 comp rx=0 tx=0).stop 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 msgr2=0x7f8770115a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 0x7f8770115a60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8760009150 tx=0x7f8760009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 shutdown_connections 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875406d7f0 0x7f875406fca0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8770104fa0 0x7f8770115a60 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 --2- 172.21.15.1:0/1423246639 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8770105940 0x7f8770115fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 >> 172.21.15.1:0/1423246639 conn(0x7f87700fab10 msgr2=0x7f87700fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.073+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 shutdown_connections 2024-02-28T16:53:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.074+0000 7f876f200700 1 -- 172.21.15.1:0/1423246639 wait complete. 2024-02-28T16:53:36.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.226+0000 7f2872e00700 1 -- 172.21.15.1:0/1498995668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fdd90 msgr2=0x7f286c0fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.226+0000 7f2872e00700 1 --2- 172.21.15.1:0/1498995668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fdd90 0x7f286c0fe160 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2860004940 tx=0x7f2860009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:36.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.226+0000 7f2872e00700 1 -- 172.21.15.1:0/1498995668 shutdown_connections 2024-02-28T16:53:36.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.226+0000 7f2872e00700 1 --2- 172.21.15.1:0/1498995668 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f286c0fe730 0x7f286c106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.226+0000 7f2872e00700 1 --2- 172.21.15.1:0/1498995668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fdd90 0x7f286c0fe160 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.226+0000 7f2872e00700 1 -- 172.21.15.1:0/1498995668 >> 172.21.15.1:0/1498995668 conn(0x7f286c073ab0 msgr2=0x7f286c073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:36.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.227+0000 7f2872e00700 1 -- 172.21.15.1:0/1498995668 shutdown_connections 2024-02-28T16:53:36.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.227+0000 7f2872e00700 1 -- 172.21.15.1:0/1498995668 wait complete. 2024-02-28T16:53:36.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.227+0000 7f2872e00700 1 Processor -- start 2024-02-28T16:53:36.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.227+0000 7f2872e00700 1 -- start start 2024-02-28T16:53:36.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.228+0000 7f2872e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f286c0fdd90 0x7f286c115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:36.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.228+0000 7f2872e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fe730 0x7f286c116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:36.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.228+0000 7f2872e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c116700 con 0x7f286c0fe730 2024-02-28T16:53:36.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.228+0000 7f2872e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c10f9a0 con 0x7f286c0fdd90 2024-02-28T16:53:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.228+0000 7f2871000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fe730 0x7f286c116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.228+0000 7f2871000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fe730 0x7f286c116020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34052/0 (socket says 172.21.15.1:34052) 2024-02-28T16:53:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.228+0000 7f2871000700 1 -- 172.21.15.1:0/2303669238 learned_addr learned my addr 172.21.15.1:0/2303669238 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.228+0000 7f2871a00700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f286c0fdd90 0x7f286c115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2871000700 1 -- 172.21.15.1:0/2303669238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f286c0fdd90 msgr2=0x7f286c115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2871000700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f286c0fdd90 0x7f286c115ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2871000700 1 -- 172.21.15.1:0/2303669238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2860005040 con 0x7f286c0fe730 2024-02-28T16:53:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2871000700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fe730 0x7f286c116020 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f285800b790 tx=0x7f2858009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2869600700 1 -- 172.21.15.1:0/2303669238 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f285800f770 con 0x7f286c0fe730 2024-02-28T16:53:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c10fc80 con 0x7f286c0fe730 2024-02-28T16:53:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2869600700 1 -- 172.21.15.1:0/2303669238 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f285800fda0 con 0x7f286c0fe730 2024-02-28T16:53:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2869600700 1 -- 172.21.15.1:0/2303669238 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f285801a640 con 0x7f286c0fe730 2024-02-28T16:53:36.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.229+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c1101c0 con 0x7f286c0fe730 2024-02-28T16:53:36.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.231+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c106920 con 0x7f286c0fe730 2024-02-28T16:53:36.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.231+0000 7f2869600700 1 -- 172.21.15.1:0/2303669238 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2858010420 con 0x7f286c0fe730 2024-02-28T16:53:36.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.233+0000 7f2869600700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f285406d7f0 0x7f285406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:36.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.233+0000 7f2869600700 1 -- 172.21.15.1:0/2303669238 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f285808ce80 con 0x7f286c0fe730 2024-02-28T16:53:36.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.233+0000 7f2871a00700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f285406d7f0 0x7f285406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:36.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.234+0000 7f2871a00700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f285406d7f0 0x7f285406fca0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2860004890 tx=0x7f2860009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:36.231 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:36 smithi001 ceph-mon[112940]: from='client.35062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:36.232 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:36 smithi001 ceph-mon[112940]: pgmap v574: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:36.232 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:36 smithi001 ceph-mon[112940]: from='client.35066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:36.232 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:36 smithi001 ceph-mon[112940]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:36.232 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1423246639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:53:36.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.242+0000 7f2869600700 1 -- 172.21.15.1:0/2303669238 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f285805a280 con 0x7f286c0fe730 2024-02-28T16:53:36.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:36 smithi066 ceph-mon[85781]: from='client.35062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:36.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:36 smithi066 ceph-mon[85781]: pgmap v574: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:36.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:36 smithi066 ceph-mon[85781]: from='client.35066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:36.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:36 smithi066 ceph-mon[85781]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:36.505 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1423246639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:53:36.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.515+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f286c05f5c0 con 0x7f286c0fe730 2024-02-28T16:53:36.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.516+0000 7f2869600700 1 -- 172.21.15.1:0/2303669238 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f2858059e10 con 0x7f286c0fe730 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:53:36.511 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:53:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:53:36.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:53:36.514 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:53:36.514 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:53:36.514 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:53:36.514 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:53:36.514 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:53:36.514 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:53:36.514 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f285406d7f0 msgr2=0x7f285406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f285406d7f0 0x7f285406fca0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2860004890 tx=0x7f2860009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fe730 msgr2=0x7f286c116020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fe730 0x7f286c116020 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f285800b790 tx=0x7f2858009250 comp rx=0 tx=0).stop 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 shutdown_connections 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f285406d7f0 0x7f285406fca0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f286c0fdd90 0x7f286c115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 --2- 172.21.15.1:0/2303669238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c0fe730 0x7f286c116020 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.518+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 >> 172.21.15.1:0/2303669238 conn(0x7f286c073ab0 msgr2=0x7f286c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.519+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 shutdown_connections 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.519+0000 7f2872e00700 1 -- 172.21.15.1:0/2303669238 wait complete. 2024-02-28T16:53:36.515 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:53:36.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.681+0000 7f6e3a800700 1 -- 172.21.15.1:0/1125378825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e34103dc0 msgr2=0x7f6e34104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.681+0000 7f6e3a800700 1 --2- 172.21.15.1:0/1125378825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e34103dc0 0x7f6e34104210 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009ce0 tx=0x7f6e1c004d10 comp rx=0 tx=0).stop 2024-02-28T16:53:36.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.681+0000 7f6e3a800700 1 -- 172.21.15.1:0/1125378825 shutdown_connections 2024-02-28T16:53:36.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.681+0000 7f6e3a800700 1 --2- 172.21.15.1:0/1125378825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e34103dc0 0x7f6e34104210 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.681+0000 7f6e3a800700 1 --2- 172.21.15.1:0/1125378825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e341095f0 0x7f6e341099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.681+0000 7f6e3a800700 1 -- 172.21.15.1:0/1125378825 >> 172.21.15.1:0/1125378825 conn(0x7f6e34072d50 msgr2=0x7f6e34075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:36.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.682+0000 7f6e3a800700 1 -- 172.21.15.1:0/1125378825 shutdown_connections 2024-02-28T16:53:36.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.682+0000 7f6e3a800700 1 -- 172.21.15.1:0/1125378825 wait complete. 2024-02-28T16:53:36.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.682+0000 7f6e3a800700 1 Processor -- start 2024-02-28T16:53:36.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.683+0000 7f6e3a800700 1 -- start start 2024-02-28T16:53:36.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.683+0000 7f6e3a800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e34103dc0 0x7f6e3411a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:36.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.683+0000 7f6e3a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e341095f0 0x7f6e3411a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:36.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.683+0000 7f6e3a800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3411ad30 con 0x7f6e341095f0 2024-02-28T16:53:36.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.683+0000 7f6e3a800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e34114040 con 0x7f6e34103dc0 2024-02-28T16:53:36.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.683+0000 7f6e2aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e341095f0 0x7f6e3411a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:36.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.683+0000 7f6e2aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e341095f0 0x7f6e3411a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34080/0 (socket says 172.21.15.1:34080) 2024-02-28T16:53:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.683+0000 7f6e2aa00700 1 -- 172.21.15.1:0/2361243522 learned_addr learned my addr 172.21.15.1:0/2361243522 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.684+0000 7f6e33400700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e34103dc0 0x7f6e3411a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.684+0000 7f6e2aa00700 1 -- 172.21.15.1:0/2361243522 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e34103dc0 msgr2=0x7f6e3411a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.684+0000 7f6e2aa00700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e34103dc0 0x7f6e3411a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.684+0000 7f6e2aa00700 1 -- 172.21.15.1:0/2361243522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c009940 con 0x7f6e341095f0 2024-02-28T16:53:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.684+0000 7f6e2aa00700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e341095f0 0x7f6e3411a6e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00be90 tx=0x7f6e1c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.684+0000 7f6e2be00700 1 -- 172.21.15.1:0/2361243522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e1c00e040 con 0x7f6e341095f0 2024-02-28T16:53:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.685+0000 7f6e2be00700 1 -- 172.21.15.1:0/2361243522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e1c008ea0 con 0x7f6e341095f0 2024-02-28T16:53:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.685+0000 7f6e2be00700 1 -- 172.21.15.1:0/2361243522 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e1c01c7a0 con 0x7f6e341095f0 2024-02-28T16:53:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.685+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e341142c0 con 0x7f6e341095f0 2024-02-28T16:53:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.685+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e34114800 con 0x7f6e341095f0 2024-02-28T16:53:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.686+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3410b9c0 con 0x7f6e341095f0 2024-02-28T16:53:36.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.687+0000 7f6e2be00700 1 -- 172.21.15.1:0/2361243522 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6e1c012410 con 0x7f6e341095f0 2024-02-28T16:53:36.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.688+0000 7f6e2be00700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e1806d7e0 0x7f6e1806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:36.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.688+0000 7f6e2be00700 1 -- 172.21.15.1:0/2361243522 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6e1c08eae0 con 0x7f6e341095f0 2024-02-28T16:53:36.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.688+0000 7f6e33400700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e1806d7e0 0x7f6e1806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:36.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.689+0000 7f6e33400700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e1806d7e0 0x7f6e1806fc90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6e24004990 tx=0x7f6e24009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:36.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.693+0000 7f6e2be00700 1 -- 172.21.15.1:0/2361243522 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e1c05be40 con 0x7f6e341095f0 2024-02-28T16:53:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.939+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e3404bf40 con 0x7f6e1806d7e0 2024-02-28T16:53:36.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.942+0000 7f6e2be00700 1 -- 172.21.15.1:0/2361243522 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6e3404bf40 con 0x7f6e1806d7e0 2024-02-28T16:53:36.937 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:53:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:53:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:53:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:53:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:53:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:53:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:53:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:53:36.938 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:53:36.938 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:53:36.938 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:53:36.938 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:53:36.938 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:53:36.938 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:53:36.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.944+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e1806d7e0 msgr2=0x7f6e1806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.944+0000 7f6e3a800700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e1806d7e0 0x7f6e1806fc90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6e24004990 tx=0x7f6e24009ee0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.944+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e341095f0 msgr2=0x7f6e3411a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:36.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.944+0000 7f6e3a800700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e341095f0 0x7f6e3411a6e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00be90 tx=0x7f6e1c004d10 comp rx=0 tx=0).stop 2024-02-28T16:53:36.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.945+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 shutdown_connections 2024-02-28T16:53:36.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.945+0000 7f6e3a800700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6e1806d7e0 0x7f6e1806fc90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.945+0000 7f6e3a800700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6e34103dc0 0x7f6e3411a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.945+0000 7f6e3a800700 1 --2- 172.21.15.1:0/2361243522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e341095f0 0x7f6e3411a6e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:36.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.945+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 >> 172.21.15.1:0/2361243522 conn(0x7f6e34072d50 msgr2=0x7f6e34074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:36.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.945+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 shutdown_connections 2024-02-28T16:53:36.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:36.945+0000 7f6e3a800700 1 -- 172.21.15.1:0/2361243522 wait complete. 2024-02-28T16:53:37.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.106+0000 7f68cae00700 1 -- 172.21.15.1:0/3066136631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 msgr2=0x7f68c40fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.106+0000 7f68cae00700 1 --2- 172.21.15.1:0/3066136631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 0x7f68c40fcf50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f68b0004940 tx=0x7f68b0009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.106+0000 7f68cae00700 1 -- 172.21.15.1:0/3066136631 shutdown_connections 2024-02-28T16:53:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.106+0000 7f68cae00700 1 --2- 172.21.15.1:0/3066136631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 0x7f68c40fcf50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.106+0000 7f68cae00700 1 --2- 172.21.15.1:0/3066136631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c4102f40 0x7f68c4103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.106+0000 7f68cae00700 1 -- 172.21.15.1:0/3066136631 >> 172.21.15.1:0/3066136631 conn(0x7f68c40f8af0 msgr2=0x7f68c40faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.106+0000 7f68cae00700 1 -- 172.21.15.1:0/3066136631 shutdown_connections 2024-02-28T16:53:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.106+0000 7f68cae00700 1 -- 172.21.15.1:0/3066136631 wait complete. 2024-02-28T16:53:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.107+0000 7f68cae00700 1 Processor -- start 2024-02-28T16:53:37.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.108+0000 7f68cae00700 1 -- start start 2024-02-28T16:53:37.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.108+0000 7f68cae00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c4102f40 0x7f68c4115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:37.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.108+0000 7f68cae00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 0x7f68c41161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:37.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.108+0000 7f68cae00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c4116830 con 0x7f68c41038e0 2024-02-28T16:53:37.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.108+0000 7f68cae00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c410fb40 con 0x7f68c4102f40 2024-02-28T16:53:37.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.109+0000 7f68c3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 0x7f68c41161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:37.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.109+0000 7f68c3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 0x7f68c41161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34102/0 (socket says 172.21.15.1:34102) 2024-02-28T16:53:37.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.109+0000 7f68c3400700 1 -- 172.21.15.1:0/2640378970 learned_addr learned my addr 172.21.15.1:0/2640378970 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:53:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.109+0000 7f68c3e00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c4102f40 0x7f68c4115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.109+0000 7f68c3400700 1 -- 172.21.15.1:0/2640378970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c4102f40 msgr2=0x7f68c4115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.109+0000 7f68c3400700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c4102f40 0x7f68c4115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.109+0000 7f68c3400700 1 -- 172.21.15.1:0/2640378970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b0005040 con 0x7f68c41038e0 2024-02-28T16:53:37.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.109+0000 7f68c3400700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 0x7f68c41161e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009030 tx=0x7f68b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:37.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.110+0000 7f68bbe00700 1 -- 172.21.15.1:0/2640378970 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b000e040 con 0x7f68c41038e0 2024-02-28T16:53:37.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.110+0000 7f68bbe00700 1 -- 172.21.15.1:0/2640378970 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68b000bbf0 con 0x7f68c41038e0 2024-02-28T16:53:37.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.110+0000 7f68bbe00700 1 -- 172.21.15.1:0/2640378970 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b001c630 con 0x7f68c41038e0 2024-02-28T16:53:37.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.110+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c410fdc0 con 0x7f68c41038e0 2024-02-28T16:53:37.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.110+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c41102b0 con 0x7f68c41038e0 2024-02-28T16:53:37.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.111+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c40fc6f0 con 0x7f68c41038e0 2024-02-28T16:53:37.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.113+0000 7f68bbe00700 1 -- 172.21.15.1:0/2640378970 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f68b000bd60 con 0x7f68c41038e0 2024-02-28T16:53:37.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.113+0000 7f68bbe00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f68a806d7f0 0x7f68a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:53:37.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.114+0000 7f68bbe00700 1 -- 172.21.15.1:0/2640378970 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f68b008e220 con 0x7f68c41038e0 2024-02-28T16:53:37.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.114+0000 7f68c3e00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f68a806d7f0 0x7f68a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:53:37.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.115+0000 7f68c3e00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f68a806d7f0 0x7f68a806fca0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f68b4004990 tx=0x7f68b4009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:53:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.119+0000 7f68bbe00700 1 -- 172.21.15.1:0/2640378970 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f68b005b620 con 0x7f68c41038e0 2024-02-28T16:53:37.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.431+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f68c4111280 con 0x7f68c41038e0 2024-02-28T16:53:37.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.432+0000 7f68bbe00700 1 -- 172.21.15.1:0/2640378970 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f68b005b1b0 con 0x7f68c41038e0 2024-02-28T16:53:37.427 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:53:37.427 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:53:37.427 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:53:37.427 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:53:37.427 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:53:37.427 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:53:37.427 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f68a806d7f0 msgr2=0x7f68a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f68a806d7f0 0x7f68a806fca0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f68b4004990 tx=0x7f68b4009ee0 comp rx=0 tx=0).stop 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 msgr2=0x7f68c41161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 0x7f68c41161e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009030 tx=0x7f68b0009e30 comp rx=0 tx=0).stop 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 shutdown_connections 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f68a806d7f0 0x7f68a806fca0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68c4102f40 0x7f68c4115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 --2- 172.21.15.1:0/2640378970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68c41038e0 0x7f68c41161e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:53:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.435+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 >> 172.21.15.1:0/2640378970 conn(0x7f68c40f8af0 msgr2=0x7f68c40fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:53:37.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.436+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 shutdown_connections 2024-02-28T16:53:37.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:53:37.436+0000 7f68cae00700 1 -- 172.21.15.1:0/2640378970 wait complete. 2024-02-28T16:53:37.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:37 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2303669238' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:53:37.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:37 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2303669238' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:53:38.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:38 smithi066 ceph-mon[85781]: from='client.35082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:38.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:38 smithi066 ceph-mon[85781]: pgmap v575: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:38.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:38 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2640378970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:53:38.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:38 smithi001 ceph-mon[112940]: from='client.35082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:53:38.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:38 smithi001 ceph-mon[112940]: pgmap v575: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:38.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:38 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2640378970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:53:40.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:40 smithi066 ceph-mon[85781]: pgmap v576: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:40.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:40 smithi001 ceph-mon[112940]: pgmap v576: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:41.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:53:41.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:53:42.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:42 smithi066 ceph-mon[85781]: pgmap v577: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:42.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:42 smithi001 ceph-mon[112940]: pgmap v577: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:44.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:44 smithi066 ceph-mon[85781]: pgmap v578: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:44.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:44 smithi001 ceph-mon[112940]: pgmap v578: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:46.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:46 smithi066 ceph-mon[85781]: pgmap v579: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:46.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:46 smithi001 ceph-mon[112940]: pgmap v579: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:48.504 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:48 smithi066 ceph-mon[85781]: pgmap v580: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:48.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:48 smithi001 ceph-mon[112940]: pgmap v580: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:50.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:50 smithi001 ceph-mon[112940]: pgmap v581: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:50.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:50 smithi066 ceph-mon[85781]: pgmap v581: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:52.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:52 smithi001 ceph-mon[112940]: pgmap v582: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:52.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:52 smithi066 ceph-mon[85781]: pgmap v582: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:54.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:54 smithi001 ceph-mon[112940]: pgmap v583: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:54.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:54 smithi066 ceph-mon[85781]: pgmap v583: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:56.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:56 smithi066 ceph-mon[85781]: pgmap v584: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:56.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:53:56.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:56 smithi001 ceph-mon[112940]: pgmap v584: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:56.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:53:58.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:53:58 smithi066 ceph-mon[85781]: pgmap v585: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:53:58.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:53:58 smithi001 ceph-mon[112940]: pgmap v585: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:00.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:00 smithi066 ceph-mon[85781]: pgmap v586: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:00.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:00 smithi001 ceph-mon[112940]: pgmap v586: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:02.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:02 smithi066 ceph-mon[85781]: pgmap v587: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:02.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:02 smithi001 ceph-mon[112940]: pgmap v587: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:04.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:04 smithi066 ceph-mon[85781]: pgmap v588: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:04.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:04 smithi001 ceph-mon[112940]: pgmap v588: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:06.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:06 smithi066 ceph-mon[85781]: pgmap v589: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:06.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:06 smithi001 ceph-mon[112940]: pgmap v589: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.607+0000 7f75e8600700 1 -- 172.21.15.1:0/1953604795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 msgr2=0x7f75e00ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.607+0000 7f75e8600700 1 --2- 172.21.15.1:0/1953604795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 0x7f75e00ff180 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f75cc004940 tx=0x7f75cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.607+0000 7f75e8600700 1 -- 172.21.15.1:0/1953604795 shutdown_connections 2024-02-28T16:54:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.607+0000 7f75e8600700 1 --2- 172.21.15.1:0/1953604795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 0x7f75e00ff180 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.607+0000 7f75e8600700 1 --2- 172.21.15.1:0/1953604795 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75e0105170 0x7f75e0105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.607+0000 7f75e8600700 1 -- 172.21.15.1:0/1953604795 >> 172.21.15.1:0/1953604795 conn(0x7f75e00face0 msgr2=0x7f75e00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.607+0000 7f75e8600700 1 -- 172.21.15.1:0/1953604795 shutdown_connections 2024-02-28T16:54:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.607+0000 7f75e8600700 1 -- 172.21.15.1:0/1953604795 wait complete. 2024-02-28T16:54:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.608+0000 7f75e8600700 1 Processor -- start 2024-02-28T16:54:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.608+0000 7f75e8600700 1 -- start start 2024-02-28T16:54:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e8600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75e0105170 0x7f75e0115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e8600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 0x7f75e0116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e8600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0116870 con 0x7f75e0105b10 2024-02-28T16:54:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e8600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e010fb80 con 0x7f75e0105170 2024-02-28T16:54:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 0x7f75e0116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 0x7f75e0116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37150/0 (socket says 172.21.15.1:37150) 2024-02-28T16:54:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e5200700 1 -- 172.21.15.1:0/73239734 learned_addr learned my addr 172.21.15.1:0/73239734 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e5c00700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75e0105170 0x7f75e0115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e5200700 1 -- 172.21.15.1:0/73239734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75e0105170 msgr2=0x7f75e0115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e5200700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75e0105170 0x7f75e0115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.609+0000 7f75e5200700 1 -- 172.21.15.1:0/73239734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc005040 con 0x7f75e0105b10 2024-02-28T16:54:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.610+0000 7f75e5200700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 0x7f75e0116220 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009030 tx=0x7f75cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:07.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.610+0000 7f75dd600700 1 -- 172.21.15.1:0/73239734 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75cc00e040 con 0x7f75e0105b10 2024-02-28T16:54:07.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.610+0000 7f75dd600700 1 -- 172.21.15.1:0/73239734 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75cc00bbf0 con 0x7f75e0105b10 2024-02-28T16:54:07.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.610+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e010fe00 con 0x7f75e0105b10 2024-02-28T16:54:07.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.610+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e0110340 con 0x7f75e0105b10 2024-02-28T16:54:07.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.610+0000 7f75dd600700 1 -- 172.21.15.1:0/73239734 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75cc01c630 con 0x7f75e0105b10 2024-02-28T16:54:07.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.611+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c00052b0 con 0x7f75e0105b10 2024-02-28T16:54:07.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.612+0000 7f75dd600700 1 -- 172.21.15.1:0/73239734 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f75cc00bd60 con 0x7f75e0105b10 2024-02-28T16:54:07.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.613+0000 7f75dd600700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75c806d7f0 0x7f75c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:07.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.613+0000 7f75dd600700 1 -- 172.21.15.1:0/73239734 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f75cc08e0c0 con 0x7f75e0105b10 2024-02-28T16:54:07.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.614+0000 7f75e5c00700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75c806d7f0 0x7f75c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:07.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.615+0000 7f75e5c00700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75c806d7f0 0x7f75c806fca0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f75d4004d60 tx=0x7f75d400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.619+0000 7f75dd600700 1 -- 172.21.15.1:0/73239734 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f75cc05b560 con 0x7f75e0105b10 2024-02-28T16:54:07.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.869+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75c0000bc0 con 0x7f75c806d7f0 2024-02-28T16:54:07.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.871+0000 7f75dd600700 1 -- 172.21.15.1:0/73239734 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f75c0000bc0 con 0x7f75c806d7f0 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75c806d7f0 msgr2=0x7f75c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75c806d7f0 0x7f75c806fca0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f75d4004d60 tx=0x7f75d400b040 comp rx=0 tx=0).stop 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 msgr2=0x7f75e0116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 0x7f75e0116220 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009030 tx=0x7f75cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 shutdown_connections 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f75c806d7f0 0x7f75c806fca0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f75e0105170 0x7f75e0115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 --2- 172.21.15.1:0/73239734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75e0105b10 0x7f75e0116220 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 >> 172.21.15.1:0/73239734 conn(0x7f75e00face0 msgr2=0x7f75e00fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 shutdown_connections 2024-02-28T16:54:07.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:07.874+0000 7f75e8600700 1 -- 172.21.15.1:0/73239734 wait complete. 2024-02-28T16:54:07.887 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.031+0000 7f3804400700 1 -- 172.21.15.1:0/2679016434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 msgr2=0x7f37fc109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.031+0000 7f3804400700 1 --2- 172.21.15.1:0/2679016434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 0x7f37fc109260 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f37e8004970 tx=0x7f37e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.031+0000 7f3804400700 1 -- 172.21.15.1:0/2679016434 shutdown_connections 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.031+0000 7f3804400700 1 --2- 172.21.15.1:0/2679016434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 0x7f37fc109260 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.031+0000 7f3804400700 1 --2- 172.21.15.1:0/2679016434 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37fc070330 0x7f37fc070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.031+0000 7f3804400700 1 -- 172.21.15.1:0/2679016434 >> 172.21.15.1:0/2679016434 conn(0x7f37fc0f8ae0 msgr2=0x7f37fc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.031+0000 7f3804400700 1 -- 172.21.15.1:0/2679016434 shutdown_connections 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.031+0000 7f3804400700 1 -- 172.21.15.1:0/2679016434 wait complete. 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.032+0000 7f3804400700 1 Processor -- start 2024-02-28T16:54:08.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.032+0000 7f3804400700 1 -- start start 2024-02-28T16:54:08.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.032+0000 7f3804400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37fc070330 0x7f37fc115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.032+0000 7f3804400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 0x7f37fc116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.032+0000 7f3804400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc116790 con 0x7f37fc070c40 2024-02-28T16:54:08.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.032+0000 7f3804400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc10fae0 con 0x7f37fc070330 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.033+0000 7f3801000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 0x7f37fc116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.033+0000 7f3801000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 0x7f37fc116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37166/0 (socket says 172.21.15.1:37166) 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.033+0000 7f3801000700 1 -- 172.21.15.1:0/421569509 learned_addr learned my addr 172.21.15.1:0/421569509 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.033+0000 7f3801a00700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37fc070330 0x7f37fc115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.033+0000 7f3801000700 1 -- 172.21.15.1:0/421569509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37fc070330 msgr2=0x7f37fc115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.033+0000 7f3801000700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37fc070330 0x7f37fc115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.033+0000 7f3801000700 1 -- 172.21.15.1:0/421569509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e8005040 con 0x7f37fc070c40 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.034+0000 7f3801000700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 0x7f37fc116140 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f37e800be70 tx=0x7f37e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.034+0000 7f37f9600700 1 -- 172.21.15.1:0/421569509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37e800e040 con 0x7f37fc070c40 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.034+0000 7f37f9600700 1 -- 172.21.15.1:0/421569509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37e8004290 con 0x7f37fc070c40 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.034+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc10fd60 con 0x7f37fc070c40 2024-02-28T16:54:08.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.034+0000 7f37f9600700 1 -- 172.21.15.1:0/421569509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37e801c3f0 con 0x7f37fc070c40 2024-02-28T16:54:08.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.034+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc1102a0 con 0x7f37fc070c40 2024-02-28T16:54:08.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.035+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc106920 con 0x7f37fc070c40 2024-02-28T16:54:08.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.036+0000 7f37f9600700 1 -- 172.21.15.1:0/421569509 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f37e8013820 con 0x7f37fc070c40 2024-02-28T16:54:08.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.038+0000 7f37f9600700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37e406d7e0 0x7f37e406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.038+0000 7f37f9600700 1 -- 172.21.15.1:0/421569509 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f37e808e260 con 0x7f37fc070c40 2024-02-28T16:54:08.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.038+0000 7f3801a00700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37e406d7e0 0x7f37e406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.039+0000 7f3801a00700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37e406d7e0 0x7f37e406fc90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009fd0 tx=0x7f37f000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:08.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.042+0000 7f37f9600700 1 -- 172.21.15.1:0/421569509 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f37e8057b30 con 0x7f37fc070c40 2024-02-28T16:54:08.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.288+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37fc04bed0 con 0x7f37e406d7e0 2024-02-28T16:54:08.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.290+0000 7f37f9600700 1 -- 172.21.15.1:0/421569509 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f37fc04bed0 con 0x7f37e406d7e0 2024-02-28T16:54:08.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.292+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37e406d7e0 msgr2=0x7f37e406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.292+0000 7f3804400700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37e406d7e0 0x7f37e406fc90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009fd0 tx=0x7f37f000b040 comp rx=0 tx=0).stop 2024-02-28T16:54:08.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.292+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 msgr2=0x7f37fc116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.292+0000 7f3804400700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 0x7f37fc116140 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f37e800be70 tx=0x7f37e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:08.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.293+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 shutdown_connections 2024-02-28T16:54:08.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.293+0000 7f3804400700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37e406d7e0 0x7f37e406fc90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.293+0000 7f3804400700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37fc070330 0x7f37fc115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.293+0000 7f3804400700 1 --2- 172.21.15.1:0/421569509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37fc070c40 0x7f37fc116140 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.293+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 >> 172.21.15.1:0/421569509 conn(0x7f37fc0f8ae0 msgr2=0x7f37fc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:08.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.293+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 shutdown_connections 2024-02-28T16:54:08.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.293+0000 7f3804400700 1 -- 172.21.15.1:0/421569509 wait complete. 2024-02-28T16:54:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.453+0000 7f2ecc600700 1 -- 172.21.15.1:0/2036891045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070c40 msgr2=0x7f2ec4109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.453+0000 7f2ecc600700 1 --2- 172.21.15.1:0/2036891045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070c40 0x7f2ec4109260 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0004970 tx=0x7f2eb0009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.453+0000 7f2ecc600700 1 -- 172.21.15.1:0/2036891045 shutdown_connections 2024-02-28T16:54:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.453+0000 7f2ecc600700 1 --2- 172.21.15.1:0/2036891045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070c40 0x7f2ec4109260 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.453+0000 7f2ecc600700 1 --2- 172.21.15.1:0/2036891045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ec4070330 0x7f2ec4070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.453+0000 7f2ecc600700 1 -- 172.21.15.1:0/2036891045 >> 172.21.15.1:0/2036891045 conn(0x7f2ec40f8ae0 msgr2=0x7f2ec40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.453+0000 7f2ecc600700 1 -- 172.21.15.1:0/2036891045 shutdown_connections 2024-02-28T16:54:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.453+0000 7f2ecc600700 1 -- 172.21.15.1:0/2036891045 wait complete. 2024-02-28T16:54:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.454+0000 7f2ecc600700 1 Processor -- start 2024-02-28T16:54:08.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.455+0000 7f2ecc600700 1 -- start start 2024-02-28T16:54:08.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.455+0000 7f2ecc600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070330 0x7f2ec4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.455+0000 7f2ecc600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ec4070c40 0x7f2ec4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.455+0000 7f2ecc600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec4116790 con 0x7f2ec4070330 2024-02-28T16:54:08.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.455+0000 7f2ecc600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec410fae0 con 0x7f2ec4070c40 2024-02-28T16:54:08.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.456+0000 7f2ec9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070330 0x7f2ec4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.456+0000 7f2ec9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070330 0x7f2ec4115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37190/0 (socket says 172.21.15.1:37190) 2024-02-28T16:54:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.456+0000 7f2ec9c00700 1 -- 172.21.15.1:0/1627004555 learned_addr learned my addr 172.21.15.1:0/1627004555 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.456+0000 7f2ec9200700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ec4070c40 0x7f2ec4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.456+0000 7f2ec9c00700 1 -- 172.21.15.1:0/1627004555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ec4070c40 msgr2=0x7f2ec4116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.456+0000 7f2ec9c00700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ec4070c40 0x7f2ec4116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.456+0000 7f2ec9c00700 1 -- 172.21.15.1:0/1627004555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0005040 con 0x7f2ec4070330 2024-02-28T16:54:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.456+0000 7f2ec9c00700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070330 0x7f2ec4115c00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c830 tx=0x7f2eb8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.457+0000 7f2ec1600700 1 -- 172.21.15.1:0/1627004555 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eb800a760 con 0x7f2ec4070330 2024-02-28T16:54:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.457+0000 7f2ec1600700 1 -- 172.21.15.1:0/1627004555 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2eb800ad90 con 0x7f2ec4070330 2024-02-28T16:54:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.457+0000 7f2ec1600700 1 -- 172.21.15.1:0/1627004555 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eb80113a0 con 0x7f2ec4070330 2024-02-28T16:54:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.457+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec410fdc0 con 0x7f2ec4070330 2024-02-28T16:54:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.457+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec4110300 con 0x7f2ec4070330 2024-02-28T16:54:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.458+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec4106920 con 0x7f2ec4070330 2024-02-28T16:54:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.459+0000 7f2ec1600700 1 -- 172.21.15.1:0/1627004555 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2eb8012400 con 0x7f2ec4070330 2024-02-28T16:54:08.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.460+0000 7f2ec1600700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2eac06d7f0 0x7f2eac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.460+0000 7f2ec1600700 1 -- 172.21.15.1:0/1627004555 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2eb808cfa0 con 0x7f2ec4070330 2024-02-28T16:54:08.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.461+0000 7f2ec9200700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2eac06d7f0 0x7f2eac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.461+0000 7f2ec9200700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2eac06d7f0 0x7f2eac06fca0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000b370 tx=0x7f2eb0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:08.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.465+0000 7f2ec1600700 1 -- 172.21.15.1:0/1627004555 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2eb80567d0 con 0x7f2ec4070330 2024-02-28T16:54:08.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.696+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ec41111e0 con 0x7f2eac06d7f0 2024-02-28T16:54:08.692 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:08 smithi001 ceph-mon[112940]: pgmap v590: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:08.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.710+0000 7f2ec1600700 1 -- 172.21.15.1:0/1627004555 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2ec41111e0 con 0x7f2eac06d7f0 2024-02-28T16:54:08.706 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:54:08.706 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (20m) 12m ago 34m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (16m) 12m ago 34m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (16m) 13m ago 33m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (19m) 12m ago 34m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (12m) 12m ago 27m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (12m) 12m ago 27m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (27m) 13m ago 27m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (27m) 13m ago 27m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (22m) 12m ago 36m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (21m) 13m ago 33m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (18m) 12m ago 36m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (18m) 13m ago 33m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (21m) 12m ago 34m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (21m) 13m ago 33m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:54:08.707 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (16m) 12m ago 32m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:54:08.708 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (15m) 12m ago 31m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:54:08.708 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (15m) 12m ago 31m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:54:08.708 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (14m) 13m ago 30m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:54:08.708 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (14m) 13m ago 29m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:54:08.708 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (13m) 13m ago 29m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:54:08.708 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (20m) 12m ago 33m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:54:08.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.712+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2eac06d7f0 msgr2=0x7f2eac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.712+0000 7f2ecc600700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2eac06d7f0 0x7f2eac06fca0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000b370 tx=0x7f2eb0009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.712+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070330 msgr2=0x7f2ec4115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.712+0000 7f2ecc600700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070330 0x7f2ec4115c00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c830 tx=0x7f2eb8004470 comp rx=0 tx=0).stop 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.712+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 shutdown_connections 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.712+0000 7f2ecc600700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2eac06d7f0 0x7f2eac06fca0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.712+0000 7f2ecc600700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ec4070330 0x7f2ec4115c00 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.712+0000 7f2ecc600700 1 --2- 172.21.15.1:0/1627004555 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2ec4070c40 0x7f2ec4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.713+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 >> 172.21.15.1:0/1627004555 conn(0x7f2ec40f8ae0 msgr2=0x7f2ec4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.713+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 shutdown_connections 2024-02-28T16:54:08.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.713+0000 7f2ecc600700 1 -- 172.21.15.1:0/1627004555 wait complete. 2024-02-28T16:54:08.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:08 smithi066 ceph-mon[85781]: pgmap v590: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:08.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.862+0000 7fedd3a00700 1 -- 172.21.15.1:0/922709396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc103dc0 msgr2=0x7fedcc104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.862+0000 7fedd3a00700 1 --2- 172.21.15.1:0/922709396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc103dc0 0x7fedcc104210 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fedb8004940 tx=0x7fedb8009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:08.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.863+0000 7fedd3a00700 1 -- 172.21.15.1:0/922709396 shutdown_connections 2024-02-28T16:54:08.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.863+0000 7fedd3a00700 1 --2- 172.21.15.1:0/922709396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc103dc0 0x7fedcc104210 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.863+0000 7fedd3a00700 1 --2- 172.21.15.1:0/922709396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fedcc1095f0 0x7fedcc1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.863+0000 7fedd3a00700 1 -- 172.21.15.1:0/922709396 >> 172.21.15.1:0/922709396 conn(0x7fedcc072d50 msgr2=0x7fedcc075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:08.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.863+0000 7fedd3a00700 1 -- 172.21.15.1:0/922709396 shutdown_connections 2024-02-28T16:54:08.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.863+0000 7fedd3a00700 1 -- 172.21.15.1:0/922709396 wait complete. 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.864+0000 7fedd3a00700 1 Processor -- start 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.864+0000 7fedd3a00700 1 -- start start 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.864+0000 7fedd3a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fedcc103dc0 0x7fedcc11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.864+0000 7fedd3a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc1095f0 0x7fedcc11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.864+0000 7fedd3a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc11abd0 con 0x7fedcc1095f0 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.864+0000 7fedd3a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc113ee0 con 0x7fedcc103dc0 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.865+0000 7fedcbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc1095f0 0x7fedcc11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.865+0000 7fedcbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc1095f0 0x7fedcc11a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37208/0 (socket says 172.21.15.1:37208) 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.865+0000 7fedcbe00700 1 -- 172.21.15.1:0/3509192974 learned_addr learned my addr 172.21.15.1:0/3509192974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.865+0000 7fedd1000700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fedcc103dc0 0x7fedcc11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.865+0000 7fedcbe00700 1 -- 172.21.15.1:0/3509192974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fedcc103dc0 msgr2=0x7fedcc11a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.865+0000 7fedcbe00700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fedcc103dc0 0x7fedcc11a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.865+0000 7fedcbe00700 1 -- 172.21.15.1:0/3509192974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8005040 con 0x7fedcc1095f0 2024-02-28T16:54:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.866+0000 7fedcbe00700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc1095f0 0x7fedcc11a580 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009fd0 tx=0x7fedb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.866+0000 7fedc8c00700 1 -- 172.21.15.1:0/3509192974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedb800e040 con 0x7fedcc1095f0 2024-02-28T16:54:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.866+0000 7fedc8c00700 1 -- 172.21.15.1:0/3509192974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedb800bbf0 con 0x7fedcc1095f0 2024-02-28T16:54:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.866+0000 7fedc8c00700 1 -- 172.21.15.1:0/3509192974 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedb801c3f0 con 0x7fedcc1095f0 2024-02-28T16:54:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.866+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedcc114160 con 0x7fedcc1095f0 2024-02-28T16:54:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.866+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedcc1146a0 con 0x7fedcc1095f0 2024-02-28T16:54:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.867+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedcc10b9c0 con 0x7fedcc1095f0 2024-02-28T16:54:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.868+0000 7fedc8c00700 1 -- 172.21.15.1:0/3509192974 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fedb800bd60 con 0x7fedcc1095f0 2024-02-28T16:54:08.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.870+0000 7fedc8c00700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fedb406d7f0 0x7fedb406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:08.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.870+0000 7fedc8c00700 1 -- 172.21.15.1:0/3509192974 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fedb808e100 con 0x7fedcc1095f0 2024-02-28T16:54:08.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.870+0000 7fedd1000700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fedb406d7f0 0x7fedb406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:08.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.871+0000 7fedd1000700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fedb406d7f0 0x7fedb406fca0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fedc0007fd0 tx=0x7fedc0009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:08.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:08.874+0000 7fedc8c00700 1 -- 172.21.15.1:0/3509192974 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fedb805b5a0 con 0x7fedcc1095f0 2024-02-28T16:54:09.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.184+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fedcc1142f0 con 0x7fedcc1095f0 2024-02-28T16:54:09.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.186+0000 7fedc8c00700 1 -- 172.21.15.1:0/3509192974 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fedcc1142f0 con 0x7fedcc1095f0 2024-02-28T16:54:09.182 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:54:09.182 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:54:09.183 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:54:09.184 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:54:09.184 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.188+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fedb406d7f0 msgr2=0x7fedb406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.188+0000 7fedd3a00700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fedb406d7f0 0x7fedb406fca0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fedc0007fd0 tx=0x7fedc0009040 comp rx=0 tx=0).stop 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.188+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc1095f0 msgr2=0x7fedcc11a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.188+0000 7fedd3a00700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc1095f0 0x7fedcc11a580 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009fd0 tx=0x7fedb8009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.189+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 shutdown_connections 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.189+0000 7fedd3a00700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fedb406d7f0 0x7fedb406fca0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.189+0000 7fedd3a00700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fedcc103dc0 0x7fedcc11a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.189+0000 7fedd3a00700 1 --2- 172.21.15.1:0/3509192974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedcc1095f0 0x7fedcc11a580 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.189+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 >> 172.21.15.1:0/3509192974 conn(0x7fedcc072d50 msgr2=0x7fedcc074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.189+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 shutdown_connections 2024-02-28T16:54:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.189+0000 7fedd3a00700 1 -- 172.21.15.1:0/3509192974 wait complete. 2024-02-28T16:54:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.349+0000 7fd341200700 1 -- 172.21.15.1:0/4159865329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070330 msgr2=0x7fd33c070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.349+0000 7fd341200700 1 --2- 172.21.15.1:0/4159865329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070330 0x7fd33c070700 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd32c004970 tx=0x7fd32c009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.349+0000 7fd341200700 1 -- 172.21.15.1:0/4159865329 shutdown_connections 2024-02-28T16:54:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.349+0000 7fd341200700 1 --2- 172.21.15.1:0/4159865329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd33c070c40 0x7fd33c109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.349+0000 7fd341200700 1 --2- 172.21.15.1:0/4159865329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070330 0x7fd33c070700 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.349+0000 7fd341200700 1 -- 172.21.15.1:0/4159865329 >> 172.21.15.1:0/4159865329 conn(0x7fd33c0f8ae0 msgr2=0x7fd33c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:09.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.349+0000 7fd341200700 1 -- 172.21.15.1:0/4159865329 shutdown_connections 2024-02-28T16:54:09.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.349+0000 7fd341200700 1 -- 172.21.15.1:0/4159865329 wait complete. 2024-02-28T16:54:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.350+0000 7fd341200700 1 Processor -- start 2024-02-28T16:54:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.350+0000 7fd341200700 1 -- start start 2024-02-28T16:54:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.350+0000 7fd341200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd33c070330 0x7fd33c115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.350+0000 7fd341200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070c40 0x7fd33c116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.350+0000 7fd341200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c1168e0 con 0x7fd33c070c40 2024-02-28T16:54:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.350+0000 7fd341200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c10fb60 con 0x7fd33c070330 2024-02-28T16:54:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070c40 0x7fd33c116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070c40 0x7fd33c116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37236/0 (socket says 172.21.15.1:37236) 2024-02-28T16:54:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339400700 1 -- 172.21.15.1:0/348215170 learned_addr learned my addr 172.21.15.1:0/348215170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339e00700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd33c070330 0x7fd33c115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339400700 1 -- 172.21.15.1:0/348215170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd33c070330 msgr2=0x7fd33c115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339400700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd33c070330 0x7fd33c115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339400700 1 -- 172.21.15.1:0/348215170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c005040 con 0x7fd33c070c40 2024-02-28T16:54:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339e00700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd33c070330 0x7fd33c115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:54:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.351+0000 7fd339400700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070c40 0x7fd33c116200 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd32800b790 tx=0x7fd328009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.352+0000 7fd332000700 1 -- 172.21.15.1:0/348215170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32800f770 con 0x7fd33c070c40 2024-02-28T16:54:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.352+0000 7fd332000700 1 -- 172.21.15.1:0/348215170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd32800fda0 con 0x7fd33c070c40 2024-02-28T16:54:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.352+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c10fde0 con 0x7fd33c070c40 2024-02-28T16:54:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.352+0000 7fd332000700 1 -- 172.21.15.1:0/348215170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32800d310 con 0x7fd33c070c40 2024-02-28T16:54:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.352+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c110320 con 0x7fd33c070c40 2024-02-28T16:54:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.353+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c106920 con 0x7fd33c070c40 2024-02-28T16:54:09.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.354+0000 7fd332000700 1 -- 172.21.15.1:0/348215170 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd328010420 con 0x7fd33c070c40 2024-02-28T16:54:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.355+0000 7fd332000700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd32006d7f0 0x7fd32006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.356+0000 7fd332000700 1 -- 172.21.15.1:0/348215170 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd32808ceb0 con 0x7fd33c070c40 2024-02-28T16:54:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.356+0000 7fd339e00700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd32006d7f0 0x7fd32006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.357+0000 7fd339e00700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd32006d7f0 0x7fd32006fca0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd32c000c00 tx=0x7fd32c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.360+0000 7fd332000700 1 -- 172.21.15.1:0/348215170 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd32805a290 con 0x7fd33c070c40 2024-02-28T16:54:09.522 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:09 smithi001 ceph-mon[112940]: from='client.35090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:09.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:09 smithi001 ceph-mon[112940]: from='client.35094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:09.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:09 smithi001 ceph-mon[112940]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:09.523 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:09 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3509192974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:54:09.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.636+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd33c0fc6e0 con 0x7fd33c070c40 2024-02-28T16:54:09.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.637+0000 7fd332000700 1 -- 172.21.15.1:0/348215170 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fd328059e20 con 0x7fd33c070c40 2024-02-28T16:54:09.633 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:54:09.633 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:54:09.633 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:54:09.633 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:54:09.634 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:54:09.635 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:54:09.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:54:09.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:54:09.636 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:54:09.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:54:09.636 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:54:09.636 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:54:09.636 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:54:09.636 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.639+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd32006d7f0 msgr2=0x7fd32006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.639+0000 7fd341200700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd32006d7f0 0x7fd32006fca0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd32c000c00 tx=0x7fd32c009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.639+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070c40 msgr2=0x7fd33c116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.639+0000 7fd341200700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070c40 0x7fd33c116200 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd32800b790 tx=0x7fd328009250 comp rx=0 tx=0).stop 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.640+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 shutdown_connections 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.640+0000 7fd341200700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd32006d7f0 0x7fd32006fca0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.640+0000 7fd341200700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd33c070330 0x7fd33c115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.640+0000 7fd341200700 1 --2- 172.21.15.1:0/348215170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd33c070c40 0x7fd33c116200 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.640+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 >> 172.21.15.1:0/348215170 conn(0x7fd33c0f8ae0 msgr2=0x7fd33c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.640+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 shutdown_connections 2024-02-28T16:54:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.640+0000 7fd341200700 1 -- 172.21.15.1:0/348215170 wait complete. 2024-02-28T16:54:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:54:09.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:09 smithi066 ceph-mon[85781]: from='client.35090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:09.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:09 smithi066 ceph-mon[85781]: from='client.35094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:09.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:09 smithi066 ceph-mon[85781]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:09.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:09 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3509192974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:54:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.797+0000 7fd633200700 1 -- 172.21.15.1:0/2252113526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 msgr2=0x7fd6340fe140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.797+0000 7fd633200700 1 --2- 172.21.15.1:0/2252113526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 0x7fd6340fe140 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd624004940 tx=0x7fd624009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.797+0000 7fd633200700 1 -- 172.21.15.1:0/2252113526 shutdown_connections 2024-02-28T16:54:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.797+0000 7fd633200700 1 --2- 172.21.15.1:0/2252113526 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6340fe710 0x7fd634106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.797+0000 7fd633200700 1 --2- 172.21.15.1:0/2252113526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 0x7fd6340fe140 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.797+0000 7fd633200700 1 -- 172.21.15.1:0/2252113526 >> 172.21.15.1:0/2252113526 conn(0x7fd6340f77c0 msgr2=0x7fd6340f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:09.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.797+0000 7fd633200700 1 -- 172.21.15.1:0/2252113526 shutdown_connections 2024-02-28T16:54:09.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.797+0000 7fd633200700 1 -- 172.21.15.1:0/2252113526 wait complete. 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.798+0000 7fd633200700 1 Processor -- start 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.798+0000 7fd633200700 1 -- start start 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.798+0000 7fd633200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 0x7fd634115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd633200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6340fe710 0x7fd634115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd633200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6341165a0 con 0x7fd6340fdd70 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd633200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63410f8d0 con 0x7fd6340fe710 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd631e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 0x7fd634115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd631e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 0x7fd634115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37246/0 (socket says 172.21.15.1:37246) 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd631e00700 1 -- 172.21.15.1:0/2504409129 learned_addr learned my addr 172.21.15.1:0/2504409129 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd631e00700 1 -- 172.21.15.1:0/2504409129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6340fe710 msgr2=0x7fd634115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:09.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd631e00700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6340fe710 0x7fd634115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:09.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.799+0000 7fd631e00700 1 -- 172.21.15.1:0/2504409129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624005040 con 0x7fd6340fdd70 2024-02-28T16:54:09.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.800+0000 7fd631e00700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 0x7fd634115a10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd624009180 tx=0x7fd624009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:09.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.800+0000 7fd62a000700 1 -- 172.21.15.1:0/2504409129 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd62400e040 con 0x7fd6340fdd70 2024-02-28T16:54:09.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.800+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63410fb50 con 0x7fd6340fdd70 2024-02-28T16:54:09.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.800+0000 7fd62a000700 1 -- 172.21.15.1:0/2504409129 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd62400bbf0 con 0x7fd6340fdd70 2024-02-28T16:54:09.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.800+0000 7fd62a000700 1 -- 172.21.15.1:0/2504409129 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd62401c630 con 0x7fd6340fdd70 2024-02-28T16:54:09.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.800+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd634110090 con 0x7fd6340fdd70 2024-02-28T16:54:09.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.801+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6341067c0 con 0x7fd6340fdd70 2024-02-28T16:54:09.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.802+0000 7fd62a000700 1 -- 172.21.15.1:0/2504409129 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd62400bd60 con 0x7fd6340fdd70 2024-02-28T16:54:09.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.803+0000 7fd62a000700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd61806d7f0 0x7fd61806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:09.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.803+0000 7fd62a000700 1 -- 172.21.15.1:0/2504409129 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd62408e360 con 0x7fd6340fdd70 2024-02-28T16:54:09.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.804+0000 7fd631400700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd61806d7f0 0x7fd61806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.804+0000 7fd631400700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd61806d7f0 0x7fd61806fca0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd634111240 tx=0x7fd61c009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:09.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:09.808+0000 7fd62a000700 1 -- 172.21.15.1:0/2504409129 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd62405b6c0 con 0x7fd6340fdd70 2024-02-28T16:54:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.042+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd634109e70 con 0x7fd61806d7f0 2024-02-28T16:54:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.045+0000 7fd62a000700 1 -- 172.21.15.1:0/2504409129 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd634109e70 con 0x7fd61806d7f0 2024-02-28T16:54:10.041 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:54:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:54:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:54:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:54:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:54:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:54:10.041 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:54:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:54:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:54:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:54:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:54:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:54:10.042 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:54:10.042 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:54:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.047+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd61806d7f0 msgr2=0x7fd61806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.047+0000 7fd633200700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd61806d7f0 0x7fd61806fca0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd634111240 tx=0x7fd61c009ce0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.047+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 msgr2=0x7fd634115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.047+0000 7fd633200700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 0x7fd634115a10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd624009180 tx=0x7fd624009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.047+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 shutdown_connections 2024-02-28T16:54:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.048+0000 7fd633200700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd61806d7f0 0x7fd61806fca0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.048+0000 7fd633200700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6340fdd70 0x7fd634115a10 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.048+0000 7fd633200700 1 --2- 172.21.15.1:0/2504409129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6340fe710 0x7fd634115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.048+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 >> 172.21.15.1:0/2504409129 conn(0x7fd6340f77c0 msgr2=0x7fd6340fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.048+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 shutdown_connections 2024-02-28T16:54:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.048+0000 7fd633200700 1 -- 172.21.15.1:0/2504409129 wait complete. 2024-02-28T16:54:10.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.203+0000 7f26f4200700 1 -- 172.21.15.1:0/2478310837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec1038f0 msgr2=0x7f26ec0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:10.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.203+0000 7f26f4200700 1 --2- 172.21.15.1:0/2478310837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec1038f0 0x7f26ec0fcf60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f26d8004940 tx=0x7f26d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:10.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.203+0000 7f26f4200700 1 -- 172.21.15.1:0/2478310837 shutdown_connections 2024-02-28T16:54:10.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.203+0000 7f26f4200700 1 --2- 172.21.15.1:0/2478310837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec1038f0 0x7f26ec0fcf60 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.203+0000 7f26f4200700 1 --2- 172.21.15.1:0/2478310837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26ec102f50 0x7f26ec103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.203+0000 7f26f4200700 1 -- 172.21.15.1:0/2478310837 >> 172.21.15.1:0/2478310837 conn(0x7f26ec0f8ae0 msgr2=0x7f26ec0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:10.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.203+0000 7f26f4200700 1 -- 172.21.15.1:0/2478310837 shutdown_connections 2024-02-28T16:54:10.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.203+0000 7f26f4200700 1 -- 172.21.15.1:0/2478310837 wait complete. 2024-02-28T16:54:10.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.204+0000 7f26f4200700 1 Processor -- start 2024-02-28T16:54:10.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.204+0000 7f26f4200700 1 -- start start 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.205+0000 7f26f4200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec102f50 0x7f26ec113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.205+0000 7f26f4200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26ec1038f0 0x7f26ec113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.205+0000 7f26f4200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec114600 con 0x7f26ec102f50 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.205+0000 7f26f4200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec10d930 con 0x7f26ec1038f0 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.205+0000 7f26f1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec102f50 0x7f26ec113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.205+0000 7f26f1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec102f50 0x7f26ec113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37258/0 (socket says 172.21.15.1:37258) 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.205+0000 7f26f1800700 1 -- 172.21.15.1:0/2580395357 learned_addr learned my addr 172.21.15.1:0/2580395357 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26f0e00700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26ec1038f0 0x7f26ec113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26f1800700 1 -- 172.21.15.1:0/2580395357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26ec1038f0 msgr2=0x7f26ec113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26f1800700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26ec1038f0 0x7f26ec113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26f1800700 1 -- 172.21.15.1:0/2580395357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d8005040 con 0x7f26ec102f50 2024-02-28T16:54:10.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26f0e00700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26ec1038f0 0x7f26ec113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T16:54:10.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26f1800700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec102f50 0x7f26ec113a70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f26e0001d80 tx=0x7f26e0009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:10.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26e9600700 1 -- 172.21.15.1:0/2580395357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26e0007d10 con 0x7f26ec102f50 2024-02-28T16:54:10.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26e9600700 1 -- 172.21.15.1:0/2580395357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26e0007e70 con 0x7f26ec102f50 2024-02-28T16:54:10.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26e9600700 1 -- 172.21.15.1:0/2580395357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26e0014640 con 0x7f26ec102f50 2024-02-28T16:54:10.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.206+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ec10dc10 con 0x7f26ec102f50 2024-02-28T16:54:10.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.207+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ec10e150 con 0x7f26ec102f50 2024-02-28T16:54:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.208+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ec105420 con 0x7f26ec102f50 2024-02-28T16:54:10.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.209+0000 7f26e9600700 1 -- 172.21.15.1:0/2580395357 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f26e000e960 con 0x7f26ec102f50 2024-02-28T16:54:10.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.210+0000 7f26e9600700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f26d406d7f0 0x7f26d406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:10.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.210+0000 7f26e9600700 1 -- 172.21.15.1:0/2580395357 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f26e008ccc0 con 0x7f26ec102f50 2024-02-28T16:54:10.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.210+0000 7f26f0e00700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f26d406d7f0 0x7f26d406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:10.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.211+0000 7f26f0e00700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f26d406d7f0 0x7f26d406fca0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f26ec10f2a0 tx=0x7f26d8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:10.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.215+0000 7f26e9600700 1 -- 172.21.15.1:0/2580395357 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f26e0092020 con 0x7f26ec102f50 2024-02-28T16:54:10.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.525+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f26ec04bf60 con 0x7f26ec102f50 2024-02-28T16:54:10.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.526+0000 7f26e9600700 1 -- 172.21.15.1:0/2580395357 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f26e00191c0 con 0x7f26ec102f50 2024-02-28T16:54:10.522 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:54:10.522 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:54:10.522 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:54:10.522 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:54:10.523 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:54:10.523 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:54:10.523 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:54:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f26d406d7f0 msgr2=0x7f26d406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f26d406d7f0 0x7f26d406fca0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f26ec10f2a0 tx=0x7f26d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec102f50 msgr2=0x7f26ec113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec102f50 0x7f26ec113a70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f26e0001d80 tx=0x7f26e0009ec0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 shutdown_connections 2024-02-28T16:54:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f26d406d7f0 0x7f26d406fca0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec102f50 0x7f26ec113a70 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 --2- 172.21.15.1:0/2580395357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26ec1038f0 0x7f26ec113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.529+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 >> 172.21.15.1:0/2580395357 conn(0x7f26ec0f8ae0 msgr2=0x7f26ec0fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.530+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 shutdown_connections 2024-02-28T16:54:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:10.530+0000 7f26f4200700 1 -- 172.21.15.1:0/2580395357 wait complete. 2024-02-28T16:54:10.656 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:10 smithi066 ceph-mon[85781]: pgmap v591: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:10.656 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:10 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/348215170' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:54:10.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:10 smithi001 ceph-mon[112940]: pgmap v591: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:10.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:10 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/348215170' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:54:11.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:11 smithi066 ceph-mon[85781]: from='client.35110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:11.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:11 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2580395357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:54:11.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:54:11.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:11 smithi001 ceph-mon[112940]: from='client.35110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:11.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:11 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2580395357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:54:11.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:54:12.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:12 smithi066 ceph-mon[85781]: pgmap v592: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:12.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:12 smithi001 ceph-mon[112940]: pgmap v592: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:14.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:14 smithi066 ceph-mon[85781]: pgmap v593: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:14.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:14 smithi001 ceph-mon[112940]: pgmap v593: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:16.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:16 smithi066 ceph-mon[85781]: pgmap v594: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:16.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:16 smithi001 ceph-mon[112940]: pgmap v594: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:18.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:18 smithi066 ceph-mon[85781]: pgmap v595: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:18.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:18 smithi001 ceph-mon[112940]: pgmap v595: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:20.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:20 smithi066 ceph-mon[85781]: pgmap v596: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:20.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:20 smithi001 ceph-mon[112940]: pgmap v596: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:22.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:22 smithi066 ceph-mon[85781]: pgmap v597: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:22.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:22 smithi001 ceph-mon[112940]: pgmap v597: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:24.754 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:24 smithi066 ceph-mon[85781]: pgmap v598: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:24.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:24 smithi001 ceph-mon[112940]: pgmap v598: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:26.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:26 smithi066 ceph-mon[85781]: pgmap v599: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:26.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:54:26.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:26 smithi001 ceph-mon[112940]: pgmap v599: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:26.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:54:28.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:28 smithi066 ceph-mon[85781]: pgmap v600: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:28.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:28 smithi001 ceph-mon[112940]: pgmap v600: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:30.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:30 smithi066 ceph-mon[85781]: pgmap v601: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:30.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:30 smithi001 ceph-mon[112940]: pgmap v601: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:32.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:32 smithi066 ceph-mon[85781]: pgmap v602: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:32.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:32 smithi001 ceph-mon[112940]: pgmap v602: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:34.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:34 smithi066 ceph-mon[85781]: pgmap v603: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:34.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:34 smithi001 ceph-mon[112940]: pgmap v603: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:36.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:36 smithi066 ceph-mon[85781]: pgmap v604: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:36.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:36 smithi001 ceph-mon[112940]: pgmap v604: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:38.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:38 smithi066 ceph-mon[85781]: pgmap v605: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:38.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:38 smithi001 ceph-mon[112940]: pgmap v605: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.700+0000 7f4a6c800700 1 -- 172.21.15.1:0/4198547034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 msgr2=0x7f4a64104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.700+0000 7f4a6c800700 1 --2- 172.21.15.1:0/4198547034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 0x7f4a64104210 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4a54004940 tx=0x7f4a54009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:40.696 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:40 smithi001 ceph-mon[112940]: pgmap v606: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:40.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.700+0000 7f4a6c800700 1 -- 172.21.15.1:0/4198547034 shutdown_connections 2024-02-28T16:54:40.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.700+0000 7f4a6c800700 1 --2- 172.21.15.1:0/4198547034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 0x7f4a64104210 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:40.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.700+0000 7f4a6c800700 1 --2- 172.21.15.1:0/4198547034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a641095f0 0x7f4a641099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.701+0000 7f4a6c800700 1 -- 172.21.15.1:0/4198547034 >> 172.21.15.1:0/4198547034 conn(0x7f4a64072d50 msgr2=0x7f4a64075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.701+0000 7f4a6c800700 1 -- 172.21.15.1:0/4198547034 shutdown_connections 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.701+0000 7f4a6c800700 1 -- 172.21.15.1:0/4198547034 wait complete. 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.701+0000 7f4a6c800700 1 Processor -- start 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.702+0000 7f4a6c800700 1 -- start start 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.702+0000 7f4a6c800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 0x7f4a6411a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.702+0000 7f4a6c800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a641095f0 0x7f4a6411a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.702+0000 7f4a6c800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6411ac50 con 0x7f4a64103dc0 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.702+0000 7f4a6c800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64113ed0 con 0x7f4a641095f0 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.702+0000 7f4a69e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 0x7f4a6411a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.702+0000 7f4a69e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 0x7f4a6411a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53472/0 (socket says 172.21.15.1:53472) 2024-02-28T16:54:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.702+0000 7f4a69e00700 1 -- 172.21.15.1:0/2549275734 learned_addr learned my addr 172.21.15.1:0/2549275734 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:40.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.703+0000 7f4a69400700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a641095f0 0x7f4a6411a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:40.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.703+0000 7f4a69e00700 1 -- 172.21.15.1:0/2549275734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a641095f0 msgr2=0x7f4a6411a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:40.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.703+0000 7f4a69e00700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a641095f0 0x7f4a6411a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.703+0000 7f4a69e00700 1 -- 172.21.15.1:0/2549275734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a54005040 con 0x7f4a64103dc0 2024-02-28T16:54:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.703+0000 7f4a69e00700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 0x7f4a6411a030 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4a5800b790 tx=0x7f4a58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.703+0000 7f4a5e000700 1 -- 172.21.15.1:0/2549275734 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5800f770 con 0x7f4a64103dc0 2024-02-28T16:54:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.703+0000 7f4a5e000700 1 -- 172.21.15.1:0/2549275734 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a5800fda0 con 0x7f4a64103dc0 2024-02-28T16:54:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.703+0000 7f4a5e000700 1 -- 172.21.15.1:0/2549275734 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5800d310 con 0x7f4a64103dc0 2024-02-28T16:54:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.704+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a64114150 con 0x7f4a64103dc0 2024-02-28T16:54:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.704+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a64114690 con 0x7f4a64103dc0 2024-02-28T16:54:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.705+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a64075870 con 0x7f4a64103dc0 2024-02-28T16:54:40.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.706+0000 7f4a5e000700 1 -- 172.21.15.1:0/2549275734 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4a58010420 con 0x7f4a64103dc0 2024-02-28T16:54:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.707+0000 7f4a5e000700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a4c06d7f0 0x7f4a4c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.707+0000 7f4a5e000700 1 -- 172.21.15.1:0/2549275734 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4a5808ce30 con 0x7f4a64103dc0 2024-02-28T16:54:40.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.707+0000 7f4a69400700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a4c06d7f0 0x7f4a4c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:40.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.708+0000 7f4a69400700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a4c06d7f0 0x7f4a4c06fca0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4a641158d0 tx=0x7f4a54009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.712+0000 7f4a5e000700 1 -- 172.21.15.1:0/2549275734 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a5805a2d0 con 0x7f4a64103dc0 2024-02-28T16:54:40.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:40 smithi066 ceph-mon[85781]: pgmap v606: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.958+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a64114960 con 0x7f4a4c06d7f0 2024-02-28T16:54:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.962+0000 7f4a5e000700 1 -- 172.21.15.1:0/2549275734 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4a64114960 con 0x7f4a4c06d7f0 2024-02-28T16:54:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a4c06d7f0 msgr2=0x7f4a4c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a4c06d7f0 0x7f4a4c06fca0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4a641158d0 tx=0x7f4a54009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 msgr2=0x7f4a6411a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 0x7f4a6411a030 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4a5800b790 tx=0x7f4a58009250 comp rx=0 tx=0).stop 2024-02-28T16:54:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 shutdown_connections 2024-02-28T16:54:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a4c06d7f0 0x7f4a4c06fca0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a64103dc0 0x7f4a6411a030 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 --2- 172.21.15.1:0/2549275734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a641095f0 0x7f4a6411a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 >> 172.21.15.1:0/2549275734 conn(0x7f4a64072d50 msgr2=0x7f4a640746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 shutdown_connections 2024-02-28T16:54:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:40.965+0000 7f4a6c800700 1 -- 172.21.15.1:0/2549275734 wait complete. 2024-02-28T16:54:40.977 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:54:41.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.127+0000 7f0435800700 1 -- 172.21.15.1:0/2872985684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 msgr2=0x7f0430109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.127+0000 7f0435800700 1 --2- 172.21.15.1:0/2872985684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 0x7f0430109260 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f041c004970 tx=0x7f041c009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:41.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.127+0000 7f0435800700 1 -- 172.21.15.1:0/2872985684 shutdown_connections 2024-02-28T16:54:41.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.127+0000 7f0435800700 1 --2- 172.21.15.1:0/2872985684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 0x7f0430109260 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.127+0000 7f0435800700 1 --2- 172.21.15.1:0/2872985684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0430070330 0x7f0430070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.127+0000 7f0435800700 1 -- 172.21.15.1:0/2872985684 >> 172.21.15.1:0/2872985684 conn(0x7f04300f8ae0 msgr2=0x7f04300faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:41.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.128+0000 7f0435800700 1 -- 172.21.15.1:0/2872985684 shutdown_connections 2024-02-28T16:54:41.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.128+0000 7f0435800700 1 -- 172.21.15.1:0/2872985684 wait complete. 2024-02-28T16:54:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.128+0000 7f0435800700 1 Processor -- start 2024-02-28T16:54:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.129+0000 7f0435800700 1 -- start start 2024-02-28T16:54:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.129+0000 7f0435800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0430070330 0x7f0430115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.129+0000 7f0435800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 0x7f0430116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.129+0000 7f0435800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0430116790 con 0x7f0430070c40 2024-02-28T16:54:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.129+0000 7f0435800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043010fae0 con 0x7f0430070330 2024-02-28T16:54:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.129+0000 7f042e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 0x7f0430116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.129+0000 7f042e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 0x7f0430116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53490/0 (socket says 172.21.15.1:53490) 2024-02-28T16:54:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.129+0000 7f042e000700 1 -- 172.21.15.1:0/2381300327 learned_addr learned my addr 172.21.15.1:0/2381300327 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f042ea00700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0430070330 0x7f0430115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f042e000700 1 -- 172.21.15.1:0/2381300327 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0430070330 msgr2=0x7f0430115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f042e000700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0430070330 0x7f0430115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f042e000700 1 -- 172.21.15.1:0/2381300327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c005040 con 0x7f0430070c40 2024-02-28T16:54:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f042ea00700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0430070330 0x7f0430115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:54:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f042e000700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 0x7f0430116140 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f041c004f50 tx=0x7f041c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:41.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f0426a00700 1 -- 172.21.15.1:0/2381300327 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f041c00e040 con 0x7f0430070c40 2024-02-28T16:54:41.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f0426a00700 1 -- 172.21.15.1:0/2381300327 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f041c004290 con 0x7f0430070c40 2024-02-28T16:54:41.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043010fd60 con 0x7f0430070c40 2024-02-28T16:54:41.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.130+0000 7f0426a00700 1 -- 172.21.15.1:0/2381300327 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f041c01c630 con 0x7f0430070c40 2024-02-28T16:54:41.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.131+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04301102a0 con 0x7f0430070c40 2024-02-28T16:54:41.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.132+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0430106920 con 0x7f0430070c40 2024-02-28T16:54:41.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.133+0000 7f0426a00700 1 -- 172.21.15.1:0/2381300327 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f041c00bc10 con 0x7f0430070c40 2024-02-28T16:54:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.133+0000 7f0426a00700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f041406d7f0 0x7f041406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.134+0000 7f0426a00700 1 -- 172.21.15.1:0/2381300327 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f041c08e320 con 0x7f0430070c40 2024-02-28T16:54:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.134+0000 7f042ea00700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f041406d7f0 0x7f041406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.135+0000 7f042ea00700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f041406d7f0 0x7f041406fca0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0420007fd0 tx=0x7f0420009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.139+0000 7f0426a00700 1 -- 172.21.15.1:0/2381300327 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f041c057b50 con 0x7f0430070c40 2024-02-28T16:54:41.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.378+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f043004bed0 con 0x7f041406d7f0 2024-02-28T16:54:41.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.381+0000 7f0426a00700 1 -- 172.21.15.1:0/2381300327 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f043004bed0 con 0x7f041406d7f0 2024-02-28T16:54:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.383+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f041406d7f0 msgr2=0x7f041406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f041406d7f0 0x7f041406fca0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0420007fd0 tx=0x7f0420009040 comp rx=0 tx=0).stop 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 msgr2=0x7f0430116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 0x7f0430116140 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f041c004f50 tx=0x7f041c009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 shutdown_connections 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f041406d7f0 0x7f041406fca0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0430070330 0x7f0430115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 --2- 172.21.15.1:0/2381300327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430070c40 0x7f0430116140 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 >> 172.21.15.1:0/2381300327 conn(0x7f04300f8ae0 msgr2=0x7f0430103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 shutdown_connections 2024-02-28T16:54:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.384+0000 7f0435800700 1 -- 172.21.15.1:0/2381300327 wait complete. 2024-02-28T16:54:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.543+0000 7f9513200700 1 -- 172.21.15.1:0/2881305398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c0702d0 msgr2=0x7f950c0706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.543+0000 7f9513200700 1 --2- 172.21.15.1:0/2881305398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c0702d0 0x7f950c0706a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f94fc004970 tx=0x7f94fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:41.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.543+0000 7f9513200700 1 -- 172.21.15.1:0/2881305398 shutdown_connections 2024-02-28T16:54:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.543+0000 7f9513200700 1 --2- 172.21.15.1:0/2881305398 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f950c070be0 0x7f950c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.543+0000 7f9513200700 1 --2- 172.21.15.1:0/2881305398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c0702d0 0x7f950c0706a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.543+0000 7f9513200700 1 -- 172.21.15.1:0/2881305398 >> 172.21.15.1:0/2881305398 conn(0x7f950c0f8ae0 msgr2=0x7f950c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.543+0000 7f9513200700 1 -- 172.21.15.1:0/2881305398 shutdown_connections 2024-02-28T16:54:41.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.543+0000 7f9513200700 1 -- 172.21.15.1:0/2881305398 wait complete. 2024-02-28T16:54:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.544+0000 7f9513200700 1 Processor -- start 2024-02-28T16:54:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.544+0000 7f9513200700 1 -- start start 2024-02-28T16:54:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.544+0000 7f9513200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f950c0702d0 0x7f950c115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.544+0000 7f9513200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c070be0 0x7f950c116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.544+0000 7f9513200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c1167b0 con 0x7f950c070be0 2024-02-28T16:54:41.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.544+0000 7f9513200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c10fae0 con 0x7f950c0702d0 2024-02-28T16:54:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.545+0000 7f9503400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c070be0 0x7f950c116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.545+0000 7f9503400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c070be0 0x7f950c116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53522/0 (socket says 172.21.15.1:53522) 2024-02-28T16:54:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.545+0000 7f9503400700 1 -- 172.21.15.1:0/566932718 learned_addr learned my addr 172.21.15.1:0/566932718 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.545+0000 7f950be00700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f950c0702d0 0x7f950c115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.545+0000 7f9503400700 1 -- 172.21.15.1:0/566932718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f950c0702d0 msgr2=0x7f950c115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.545+0000 7f9503400700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f950c0702d0 0x7f950c115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.545+0000 7f9503400700 1 -- 172.21.15.1:0/566932718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc005040 con 0x7f950c070be0 2024-02-28T16:54:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.545+0000 7f9503400700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c070be0 0x7f950c116160 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f94f400cb10 tx=0x7f94f4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.546+0000 7f9508c00700 1 -- 172.21.15.1:0/566932718 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94f4004200 con 0x7f950c070be0 2024-02-28T16:54:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.546+0000 7f9508c00700 1 -- 172.21.15.1:0/566932718 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94f400acc0 con 0x7f950c070be0 2024-02-28T16:54:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.546+0000 7f9508c00700 1 -- 172.21.15.1:0/566932718 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94f4003d90 con 0x7f950c070be0 2024-02-28T16:54:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.546+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c10fdc0 con 0x7f950c070be0 2024-02-28T16:54:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.546+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c110300 con 0x7f950c070be0 2024-02-28T16:54:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.547+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c106980 con 0x7f950c070be0 2024-02-28T16:54:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.548+0000 7f9508c00700 1 -- 172.21.15.1:0/566932718 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f94f4003430 con 0x7f950c070be0 2024-02-28T16:54:41.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.549+0000 7f9508c00700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f94f006d7f0 0x7f94f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.549+0000 7f9508c00700 1 -- 172.21.15.1:0/566932718 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f94f408d490 con 0x7f950c070be0 2024-02-28T16:54:41.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.549+0000 7f950be00700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f94f006d7f0 0x7f94f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.550+0000 7f950be00700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f94f006d7f0 0x7f94f006fca0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f94fc000c00 tx=0x7f94fc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:41.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.554+0000 7f9508c00700 1 -- 172.21.15.1:0/566932718 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f94f405a7f0 con 0x7f950c070be0 2024-02-28T16:54:41.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:54:41.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:54:41.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.787+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f950c10c240 con 0x7f94f006d7f0 2024-02-28T16:54:41.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.800+0000 7f9508c00700 1 -- 172.21.15.1:0/566932718 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f950c10c240 con 0x7f94f006d7f0 2024-02-28T16:54:41.796 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:54:41.796 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (20m) 12m ago 35m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:54:41.796 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (17m) 12m ago 35m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:54:41.796 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (17m) 13m ago 33m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:54:41.796 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (20m) 12m ago 34m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:54:41.796 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (13m) 12m ago 27m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:54:41.796 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (12m) 12m ago 27m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (27m) 13m ago 27m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (27m) 13m ago 27m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (23m) 12m ago 36m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (22m) 13m ago 33m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (19m) 12m ago 36m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (18m) 13m ago 33m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (21m) 12m ago 35m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (21m) 13m ago 33m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (16m) 12m ago 32m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (16m) 12m ago 32m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (15m) 12m ago 31m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (15m) 13m ago 30m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (14m) 13m ago 30m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:54:41.797 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (13m) 13m ago 29m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:54:41.798 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (21m) 12m ago 34m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:54:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.803+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f94f006d7f0 msgr2=0x7f94f006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.803+0000 7f9513200700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f94f006d7f0 0x7f94f006fca0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f94fc000c00 tx=0x7f94fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.803+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c070be0 msgr2=0x7f950c116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.803+0000 7f9513200700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c070be0 0x7f950c116160 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f94f400cb10 tx=0x7f94f4007f90 comp rx=0 tx=0).stop 2024-02-28T16:54:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.804+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 shutdown_connections 2024-02-28T16:54:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.804+0000 7f9513200700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f94f006d7f0 0x7f94f006fca0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.804+0000 7f9513200700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f950c0702d0 0x7f950c115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.804+0000 7f9513200700 1 --2- 172.21.15.1:0/566932718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f950c070be0 0x7f950c116160 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.804+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 >> 172.21.15.1:0/566932718 conn(0x7f950c0f8ae0 msgr2=0x7f950c103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.804+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 shutdown_connections 2024-02-28T16:54:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.804+0000 7f9513200700 1 -- 172.21.15.1:0/566932718 wait complete. 2024-02-28T16:54:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.958+0000 7f9594600700 1 -- 172.21.15.1:0/438247857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 msgr2=0x7f958c105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.958+0000 7f9594600700 1 --2- 172.21.15.1:0/438247857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 0x7f958c105540 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9580004940 tx=0x7f9580009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.958+0000 7f9594600700 1 -- 172.21.15.1:0/438247857 shutdown_connections 2024-02-28T16:54:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.958+0000 7f9594600700 1 --2- 172.21.15.1:0/438247857 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f958c105b10 0x7f958c0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.958+0000 7f9594600700 1 --2- 172.21.15.1:0/438247857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 0x7f958c105540 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.958+0000 7f9594600700 1 -- 172.21.15.1:0/438247857 >> 172.21.15.1:0/438247857 conn(0x7f958c0face0 msgr2=0x7f958c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.958+0000 7f9594600700 1 -- 172.21.15.1:0/438247857 shutdown_connections 2024-02-28T16:54:41.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.958+0000 7f9594600700 1 -- 172.21.15.1:0/438247857 wait complete. 2024-02-28T16:54:41.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.959+0000 7f9594600700 1 Processor -- start 2024-02-28T16:54:41.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.959+0000 7f9594600700 1 -- start start 2024-02-28T16:54:41.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9594600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 0x7f958c115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9594600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f958c105b10 0x7f958c1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9594600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c1167f0 con 0x7f958c105170 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9594600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c10fb40 con 0x7f958c105b10 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9591c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 0x7f958c115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9591c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 0x7f958c115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44254/0 (socket says 172.21.15.1:44254) 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9591c00700 1 -- 172.21.15.1:0/1360891799 learned_addr learned my addr 172.21.15.1:0/1360891799 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9591200700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f958c105b10 0x7f958c1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9591c00700 1 -- 172.21.15.1:0/1360891799 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f958c105b10 msgr2=0x7f958c1161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9591c00700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f958c105b10 0x7f958c1161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9591c00700 1 -- 172.21.15.1:0/1360891799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580005040 con 0x7f958c105170 2024-02-28T16:54:41.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.960+0000 7f9591200700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f958c105b10 0x7f958c1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:54:41.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.961+0000 7f9591c00700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 0x7f958c115c60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f958000b3b0 tx=0x7f9580009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:41.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.961+0000 7f9589600700 1 -- 172.21.15.1:0/1360891799 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f958000e040 con 0x7f958c105170 2024-02-28T16:54:41.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.961+0000 7f9589600700 1 -- 172.21.15.1:0/1360891799 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f958000bbf0 con 0x7f958c105170 2024-02-28T16:54:41.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.961+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c10fdc0 con 0x7f958c105170 2024-02-28T16:54:41.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.961+0000 7f9589600700 1 -- 172.21.15.1:0/1360891799 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f958001c630 con 0x7f958c105170 2024-02-28T16:54:41.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.961+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c110300 con 0x7f958c105170 2024-02-28T16:54:41.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.962+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958c102480 con 0x7f958c105170 2024-02-28T16:54:41.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.963+0000 7f9589600700 1 -- 172.21.15.1:0/1360891799 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f958000bd60 con 0x7f958c105170 2024-02-28T16:54:41.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.964+0000 7f9589600700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f957406d7e0 0x7f957406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:41.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.964+0000 7f9589600700 1 -- 172.21.15.1:0/1360891799 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f958008e380 con 0x7f958c105170 2024-02-28T16:54:41.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.964+0000 7f9591200700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f957406d7e0 0x7f957406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:41.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.965+0000 7f9591200700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f957406d7e0 0x7f957406fc90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9578009fd0 tx=0x7f957800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:41.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:41.969+0000 7f9589600700 1 -- 172.21.15.1:0/1360891799 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9580057bb0 con 0x7f958c105170 2024-02-28T16:54:42.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.273+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f958c04bf00 con 0x7f958c105170 2024-02-28T16:54:42.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.274+0000 7f9589600700 1 -- 172.21.15.1:0/1360891799 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f958005b270 con 0x7f958c105170 2024-02-28T16:54:42.270 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:54:42.270 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:54:42.270 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:54:42.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:54:42.270 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:54:42.270 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:54:42.271 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:54:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f957406d7e0 msgr2=0x7f957406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f957406d7e0 0x7f957406fc90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9578009fd0 tx=0x7f957800b040 comp rx=0 tx=0).stop 2024-02-28T16:54:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 msgr2=0x7f958c115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 0x7f958c115c60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f958000b3b0 tx=0x7f9580009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 shutdown_connections 2024-02-28T16:54:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f957406d7e0 0x7f957406fc90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f958c105170 0x7f958c115c60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 --2- 172.21.15.1:0/1360891799 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f958c105b10 0x7f958c1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 >> 172.21.15.1:0/1360891799 conn(0x7f958c0face0 msgr2=0x7f958c0fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 shutdown_connections 2024-02-28T16:54:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.277+0000 7f9594600700 1 -- 172.21.15.1:0/1360891799 wait complete. 2024-02-28T16:54:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.424+0000 7f2443a00700 1 -- 172.21.15.1:0/1057380714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c1027e0 msgr2=0x7f243c0fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.424+0000 7f2443a00700 1 --2- 172.21.15.1:0/1057380714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c1027e0 0x7f243c0fbea0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2428004940 tx=0x7f2428009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.424+0000 7f2443a00700 1 -- 172.21.15.1:0/1057380714 shutdown_connections 2024-02-28T16:54:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.424+0000 7f2443a00700 1 --2- 172.21.15.1:0/1057380714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c1027e0 0x7f243c0fbea0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.424+0000 7f2443a00700 1 --2- 172.21.15.1:0/1057380714 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f243c101e40 0x7f243c102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.424+0000 7f2443a00700 1 -- 172.21.15.1:0/1057380714 >> 172.21.15.1:0/1057380714 conn(0x7f243c071d90 msgr2=0x7f243c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.424+0000 7f2443a00700 1 -- 172.21.15.1:0/1057380714 shutdown_connections 2024-02-28T16:54:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.424+0000 7f2443a00700 1 -- 172.21.15.1:0/1057380714 wait complete. 2024-02-28T16:54:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.425+0000 7f2443a00700 1 Processor -- start 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.425+0000 7f2443a00700 1 -- start start 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2443a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c101e40 0x7f243c0fdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2443a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f243c1027e0 0x7f243c0fe000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2443a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c0fe600 con 0x7f243c101e40 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2443a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c0fe770 con 0x7f243c1027e0 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2441000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c101e40 0x7f243c0fdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2441000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c101e40 0x7f243c0fdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44270/0 (socket says 172.21.15.1:44270) 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2441000700 1 -- 172.21.15.1:0/1756573580 learned_addr learned my addr 172.21.15.1:0/1756573580 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f243be00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f243c1027e0 0x7f243c0fe000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2441000700 1 -- 172.21.15.1:0/1756573580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f243c1027e0 msgr2=0x7f243c0fe000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2441000700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f243c1027e0 0x7f243c0fe000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f2441000700 1 -- 172.21.15.1:0/1756573580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428005040 con 0x7f243c101e40 2024-02-28T16:54:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.426+0000 7f243be00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f243c1027e0 0x7f243c0fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:54:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.427+0000 7f2441000700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c101e40 0x7f243c0fdac0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2430002a50 tx=0x7f24300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.427+0000 7f2438c00700 1 -- 172.21.15.1:0/1756573580 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2430007cc0 con 0x7f243c101e40 2024-02-28T16:54:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.427+0000 7f2438c00700 1 -- 172.21.15.1:0/1756573580 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f243000f040 con 0x7f243c101e40 2024-02-28T16:54:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.427+0000 7f2438c00700 1 -- 172.21.15.1:0/1756573580 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2430009640 con 0x7f243c101e40 2024-02-28T16:54:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.427+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243c119cb0 con 0x7f243c101e40 2024-02-28T16:54:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.427+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243c11a190 con 0x7f243c101e40 2024-02-28T16:54:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.428+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243c0738b0 con 0x7f243c101e40 2024-02-28T16:54:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.429+0000 7f2438c00700 1 -- 172.21.15.1:0/1756573580 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f24300097a0 con 0x7f243c101e40 2024-02-28T16:54:42.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.430+0000 7f2438c00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f242406d7f0 0x7f242406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:42.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.430+0000 7f2438c00700 1 -- 172.21.15.1:0/1756573580 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f243008cc40 con 0x7f243c101e40 2024-02-28T16:54:42.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.431+0000 7f243be00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f242406d7f0 0x7f242406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:42.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.431+0000 7f243be00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f242406d7f0 0x7f242406fca0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f242800b3b0 tx=0x7f2428009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.435+0000 7f2438c00700 1 -- 172.21.15.1:0/1756573580 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2430056470 con 0x7f243c101e40 2024-02-28T16:54:42.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:42 smithi001 ceph-mon[112940]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:42.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:42 smithi001 ceph-mon[112940]: pgmap v607: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:42.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:42 smithi001 ceph-mon[112940]: from='client.35122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:42.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:42 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1360891799' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:54:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.696+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f243c0ff300 con 0x7f243c101e40 2024-02-28T16:54:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.697+0000 7f2438c00700 1 -- 172.21.15.1:0/1756573580 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f2430059b30 con 0x7f243c101e40 2024-02-28T16:54:42.693 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:54:42.693 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:54:42.693 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:54:42.693 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:54:42.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:54:42.693 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:54:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:54:42.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:54:42.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:54:42.696 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:54:42.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:54:42.696 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:54:42.696 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:54:42.696 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:54:42.696 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.699+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f242406d7f0 msgr2=0x7f242406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.699+0000 7f2443a00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f242406d7f0 0x7f242406fca0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f242800b3b0 tx=0x7f2428009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c101e40 msgr2=0x7f243c0fdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c101e40 0x7f243c0fdac0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2430002a50 tx=0x7f24300046b0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 shutdown_connections 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f242406d7f0 0x7f242406fca0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f243c101e40 0x7f243c0fdac0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 --2- 172.21.15.1:0/1756573580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f243c1027e0 0x7f243c0fe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 >> 172.21.15.1:0/1756573580 conn(0x7f243c071d90 msgr2=0x7f243c065760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 shutdown_connections 2024-02-28T16:54:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.700+0000 7f2443a00700 1 -- 172.21.15.1:0/1756573580 wait complete. 2024-02-28T16:54:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:54:42.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:42 smithi066 ceph-mon[85781]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:42.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:42 smithi066 ceph-mon[85781]: pgmap v607: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:42.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:42 smithi066 ceph-mon[85781]: from='client.35122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:42.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:42 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1360891799' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:54:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.855+0000 7f920a800700 1 -- 172.21.15.1:0/980908959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 msgr2=0x7f9204104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.855+0000 7f920a800700 1 --2- 172.21.15.1:0/980908959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 0x7f9204104230 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f91f0004940 tx=0x7f91f0009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.855+0000 7f920a800700 1 -- 172.21.15.1:0/980908959 shutdown_connections 2024-02-28T16:54:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.855+0000 7f920a800700 1 --2- 172.21.15.1:0/980908959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 0x7f9204104230 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.855+0000 7f920a800700 1 --2- 172.21.15.1:0/980908959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92041095b0 0x7f9204109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.855+0000 7f920a800700 1 -- 172.21.15.1:0/980908959 >> 172.21.15.1:0/980908959 conn(0x7f9204072d70 msgr2=0x7f9204075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:42.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.855+0000 7f920a800700 1 -- 172.21.15.1:0/980908959 shutdown_connections 2024-02-28T16:54:42.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.855+0000 7f920a800700 1 -- 172.21.15.1:0/980908959 wait complete. 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.856+0000 7f920a800700 1 Processor -- start 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.856+0000 7f920a800700 1 -- start start 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f920a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 0x7f920411a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f920a800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92041095b0 0x7f920411a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f920a800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920411ac00 con 0x7f9204103dc0 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f920a800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9204113f10 con 0x7f92041095b0 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f9203400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 0x7f920411a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f9203400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 0x7f920411a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44296/0 (socket says 172.21.15.1:44296) 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f9203400700 1 -- 172.21.15.1:0/2595753119 learned_addr learned my addr 172.21.15.1:0/2595753119 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f9202a00700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92041095b0 0x7f920411a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f9203400700 1 -- 172.21.15.1:0/2595753119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92041095b0 msgr2=0x7f920411a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.857+0000 7f9203400700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92041095b0 0x7f920411a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.858+0000 7f9203400700 1 -- 172.21.15.1:0/2595753119 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0005040 con 0x7f9204103dc0 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.858+0000 7f9202a00700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92041095b0 0x7f920411a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.858+0000 7f9203400700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 0x7f920411a070 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f91f4002a80 tx=0x7f91f4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.858+0000 7f91fb400700 1 -- 172.21.15.1:0/2595753119 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91f4007cc0 con 0x7f9204103dc0 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.858+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92041141f0 con 0x7f9204103dc0 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.858+0000 7f91fb400700 1 -- 172.21.15.1:0/2595753119 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91f400f040 con 0x7f9204103dc0 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.858+0000 7f91fb400700 1 -- 172.21.15.1:0/2595753119 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91f400dad0 con 0x7f9204103dc0 2024-02-28T16:54:42.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.858+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9204114730 con 0x7f9204103dc0 2024-02-28T16:54:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.859+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920410b990 con 0x7f9204103dc0 2024-02-28T16:54:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.860+0000 7f91fb400700 1 -- 172.21.15.1:0/2595753119 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f91f400dc30 con 0x7f9204103dc0 2024-02-28T16:54:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.861+0000 7f91fb400700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f91e806d7f0 0x7f91e806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.861+0000 7f91fb400700 1 -- 172.21.15.1:0/2595753119 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f91f408cd20 con 0x7f9204103dc0 2024-02-28T16:54:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.862+0000 7f9202a00700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f91e806d7f0 0x7f91e806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.862+0000 7f9202a00700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f91e806d7f0 0x7f91e806fca0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009fd0 tx=0x7f91f0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:42.866+0000 7f91fb400700 1 -- 172.21.15.1:0/2595753119 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f91f405a000 con 0x7f9204103dc0 2024-02-28T16:54:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.098+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9204115650 con 0x7f91e806d7f0 2024-02-28T16:54:43.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.100+0000 7f91fb400700 1 -- 172.21.15.1:0/2595753119 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9204115650 con 0x7f91e806d7f0 2024-02-28T16:54:43.096 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:54:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:54:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:54:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:54:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:54:43.096 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:54:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:54:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:54:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:54:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:54:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:54:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:54:43.097 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:54:43.097 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:54:43.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f91e806d7f0 msgr2=0x7f91e806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:43.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f91e806d7f0 0x7f91e806fca0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009fd0 tx=0x7f91f0009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:43.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 msgr2=0x7f920411a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:43.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 0x7f920411a070 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f91f4002a80 tx=0x7f91f4004600 comp rx=0 tx=0).stop 2024-02-28T16:54:43.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 shutdown_connections 2024-02-28T16:54:43.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f91e806d7f0 0x7f91e806fca0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9204103dc0 0x7f920411a070 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 --2- 172.21.15.1:0/2595753119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92041095b0 0x7f920411a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.103+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 >> 172.21.15.1:0/2595753119 conn(0x7f9204072d70 msgr2=0x7f9204074960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:43.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.104+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 shutdown_connections 2024-02-28T16:54:43.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.104+0000 7f920a800700 1 -- 172.21.15.1:0/2595753119 wait complete. 2024-02-28T16:54:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.246+0000 7f8863600700 1 -- 172.21.15.1:0/585083216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 msgr2=0x7f885c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.246+0000 7f8863600700 1 --2- 172.21.15.1:0/585083216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 0x7f885c103320 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8850004940 tx=0x7f8850009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.247+0000 7f8863600700 1 -- 172.21.15.1:0/585083216 shutdown_connections 2024-02-28T16:54:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.247+0000 7f8863600700 1 --2- 172.21.15.1:0/585083216 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f885c1038f0 0x7f885c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.247+0000 7f8863600700 1 --2- 172.21.15.1:0/585083216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 0x7f885c103320 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.247+0000 7f8863600700 1 -- 172.21.15.1:0/585083216 >> 172.21.15.1:0/585083216 conn(0x7f885c0f8ae0 msgr2=0x7f885c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.247+0000 7f8863600700 1 -- 172.21.15.1:0/585083216 shutdown_connections 2024-02-28T16:54:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.247+0000 7f8863600700 1 -- 172.21.15.1:0/585083216 wait complete. 2024-02-28T16:54:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.248+0000 7f8863600700 1 Processor -- start 2024-02-28T16:54:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.248+0000 7f8863600700 1 -- start start 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.248+0000 7f8863600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 0x7f885c113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8863600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f885c1038f0 0x7f885c113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8863600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1145e0 con 0x7f885c102f50 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8863600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c10d910 con 0x7f885c1038f0 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8860c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 0x7f885c113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8860c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 0x7f885c113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44320/0 (socket says 172.21.15.1:44320) 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8860c00700 1 -- 172.21.15.1:0/2391689118 learned_addr learned my addr 172.21.15.1:0/2391689118 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8860c00700 1 -- 172.21.15.1:0/2391689118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f885c1038f0 msgr2=0x7f885c113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f885be00700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f885c1038f0 0x7f885c113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:43.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8860c00700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f885c1038f0 0x7f885c113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.249+0000 7f8860c00700 1 -- 172.21.15.1:0/2391689118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8850005040 con 0x7f885c102f50 2024-02-28T16:54:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.250+0000 7f8860c00700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 0x7f885c113a50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8850009150 tx=0x7f8850009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.250+0000 7f8858c00700 1 -- 172.21.15.1:0/2391689118 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f885000e040 con 0x7f885c102f50 2024-02-28T16:54:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.250+0000 7f8858c00700 1 -- 172.21.15.1:0/2391689118 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f885000bbf0 con 0x7f885c102f50 2024-02-28T16:54:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.250+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c10db90 con 0x7f885c102f50 2024-02-28T16:54:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.250+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885c10e0d0 con 0x7f885c102f50 2024-02-28T16:54:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.250+0000 7f8858c00700 1 -- 172.21.15.1:0/2391689118 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8850013ea0 con 0x7f885c102f50 2024-02-28T16:54:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.251+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885c0fc6e0 con 0x7f885c102f50 2024-02-28T16:54:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.252+0000 7f8858c00700 1 -- 172.21.15.1:0/2391689118 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f885000bd60 con 0x7f885c102f50 2024-02-28T16:54:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.254+0000 7f8858c00700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f884406d7f0 0x7f884406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:54:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.254+0000 7f8858c00700 1 -- 172.21.15.1:0/2391689118 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f885008e250 con 0x7f885c102f50 2024-02-28T16:54:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.254+0000 7f885be00700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f884406d7f0 0x7f884406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:54:43.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.255+0000 7f885be00700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f884406d7f0 0x7f884406fca0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f885c10f280 tx=0x7f8848009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:54:43.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.258+0000 7f8858c00700 1 -- 172.21.15.1:0/2391689118 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f885005b650 con 0x7f885c102f50 2024-02-28T16:54:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.567+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f885c10e340 con 0x7f885c102f50 2024-02-28T16:54:43.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.568+0000 7f8858c00700 1 -- 172.21.15.1:0/2391689118 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f885005b1e0 con 0x7f885c102f50 2024-02-28T16:54:43.564 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:54:43.564 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:54:43.564 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:54:43.564 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:54:43.564 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:54:43.564 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:54:43.564 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:54:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f884406d7f0 msgr2=0x7f884406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f884406d7f0 0x7f884406fca0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f885c10f280 tx=0x7f8848009040 comp rx=0 tx=0).stop 2024-02-28T16:54:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 msgr2=0x7f885c113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:54:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 0x7f885c113a50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8850009150 tx=0x7f8850009e30 comp rx=0 tx=0).stop 2024-02-28T16:54:43.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 shutdown_connections 2024-02-28T16:54:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f884406d7f0 0x7f884406fca0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f885c102f50 0x7f885c113a50 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 --2- 172.21.15.1:0/2391689118 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f885c1038f0 0x7f885c113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:54:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 >> 172.21.15.1:0/2391689118 conn(0x7f885c0f8ae0 msgr2=0x7f885c0fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:54:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 shutdown_connections 2024-02-28T16:54:43.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:54:43.571+0000 7f8863600700 1 -- 172.21.15.1:0/2391689118 wait complete. 2024-02-28T16:54:43.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:43 smithi066 ceph-mon[85781]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:43.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:43 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1756573580' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:54:43.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:43 smithi001 ceph-mon[112940]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:43.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:43 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1756573580' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:54:44.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:44 smithi066 ceph-mon[85781]: pgmap v608: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:44.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:44 smithi066 ceph-mon[85781]: from='client.35138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:44.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:44 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2391689118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:54:44.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:44 smithi001 ceph-mon[112940]: pgmap v608: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:44.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:44 smithi001 ceph-mon[112940]: from='client.35138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:54:44.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:44 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2391689118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:54:46.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:46 smithi066 ceph-mon[85781]: pgmap v609: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:46.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:46 smithi001 ceph-mon[112940]: pgmap v609: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:48.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:48 smithi066 ceph-mon[85781]: pgmap v610: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:48.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:48 smithi001 ceph-mon[112940]: pgmap v610: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:50.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:50 smithi066 ceph-mon[85781]: pgmap v611: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:50.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:50 smithi001 ceph-mon[112940]: pgmap v611: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:52.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:52 smithi066 ceph-mon[85781]: pgmap v612: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:52.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:52 smithi001 ceph-mon[112940]: pgmap v612: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:54.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:54 smithi066 ceph-mon[85781]: pgmap v613: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:54.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:54 smithi001 ceph-mon[112940]: pgmap v613: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:56.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:56 smithi066 ceph-mon[85781]: pgmap v614: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:56.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:54:56.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:56 smithi001 ceph-mon[112940]: pgmap v614: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:56.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:54:58.755 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:54:58 smithi066 ceph-mon[85781]: pgmap v615: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:54:58.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:54:58 smithi001 ceph-mon[112940]: pgmap v615: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:00.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:00 smithi001 ceph-mon[112940]: pgmap v616: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:01.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:00 smithi066 ceph-mon[85781]: pgmap v616: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:02.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:02 smithi001 ceph-mon[112940]: pgmap v617: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:03.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:02 smithi066 ceph-mon[85781]: pgmap v617: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:04.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:04 smithi001 ceph-mon[112940]: pgmap v618: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:05.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:04 smithi066 ceph-mon[85781]: pgmap v618: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:07.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:06 smithi066 ceph-mon[85781]: pgmap v619: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:07.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:06 smithi001 ceph-mon[112940]: pgmap v619: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:09.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:08 smithi066 ceph-mon[85781]: pgmap v620: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:09.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:08 smithi001 ceph-mon[112940]: pgmap v620: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:10.808 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:10 smithi066 ceph-mon[85781]: pgmap v621: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:11.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:10 smithi001 ceph-mon[112940]: pgmap v621: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:12.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:55:12.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:55:13.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:12 smithi066 ceph-mon[85781]: pgmap v622: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:13.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:12 smithi001 ceph-mon[112940]: pgmap v622: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:13.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.747+0000 7fc100c00700 1 -- 172.21.15.1:0/3765565766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 msgr2=0x7fc0fc100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:13.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.747+0000 7fc100c00700 1 --2- 172.21.15.1:0/3765565766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 0x7fc0fc100bd0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8004940 tx=0x7fc0e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:13.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.747+0000 7fc100c00700 1 -- 172.21.15.1:0/3765565766 shutdown_connections 2024-02-28T16:55:13.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.747+0000 7fc100c00700 1 --2- 172.21.15.1:0/3765565766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 0x7fc0fc100bd0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:13.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.747+0000 7fc100c00700 1 --2- 172.21.15.1:0/3765565766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0fc0fbb80 0x7fc0fc0fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:13.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.747+0000 7fc100c00700 1 -- 172.21.15.1:0/3765565766 >> 172.21.15.1:0/3765565766 conn(0x7fc0fc073ab0 msgr2=0x7fc0fc073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:13.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.747+0000 7fc100c00700 1 -- 172.21.15.1:0/3765565766 shutdown_connections 2024-02-28T16:55:13.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.747+0000 7fc100c00700 1 -- 172.21.15.1:0/3765565766 wait complete. 2024-02-28T16:55:13.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.748+0000 7fc100c00700 1 Processor -- start 2024-02-28T16:55:13.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.748+0000 7fc100c00700 1 -- start start 2024-02-28T16:55:13.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.749+0000 7fc100c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0fc0fbb80 0x7fc0fc119f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:13.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.749+0000 7fc100c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 0x7fc0fc11a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:13.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.749+0000 7fc100c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc11aab0 con 0x7fc0fc0fc520 2024-02-28T16:55:13.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.749+0000 7fc100c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc113dc0 con 0x7fc0fc0fbb80 2024-02-28T16:55:13.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.749+0000 7fc0faa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 0x7fc0fc11a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:13.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.749+0000 7fc0faa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 0x7fc0fc11a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55402/0 (socket says 172.21.15.1:55402) 2024-02-28T16:55:13.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.749+0000 7fc0faa00700 1 -- 172.21.15.1:0/1978655048 learned_addr learned my addr 172.21.15.1:0/1978655048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:13.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc0faa00700 1 -- 172.21.15.1:0/1978655048 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0fc0fbb80 msgr2=0x7fc0fc119f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:13.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc0faa00700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0fc0fbb80 0x7fc0fc119f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:13.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc0faa00700 1 -- 172.21.15.1:0/1978655048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e8005040 con 0x7fc0fc0fc520 2024-02-28T16:55:13.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc0faa00700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 0x7fc0fc11a460 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009100 tx=0x7fc0e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:13.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc0f3400700 1 -- 172.21.15.1:0/1978655048 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e800e040 con 0x7fc0fc0fc520 2024-02-28T16:55:13.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc0f3400700 1 -- 172.21.15.1:0/1978655048 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0e8004290 con 0x7fc0fc0fc520 2024-02-28T16:55:13.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0fc114040 con 0x7fc0fc0fc520 2024-02-28T16:55:13.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc0f3400700 1 -- 172.21.15.1:0/1978655048 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e801c630 con 0x7fc0fc0fc520 2024-02-28T16:55:13.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.750+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0fc114580 con 0x7fc0fc0fc520 2024-02-28T16:55:13.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.752+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0fc0fe2f0 con 0x7fc0fc0fc520 2024-02-28T16:55:13.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.753+0000 7fc0f3400700 1 -- 172.21.15.1:0/1978655048 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc0e800bbf0 con 0x7fc0fc0fc520 2024-02-28T16:55:13.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.754+0000 7fc0f3400700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0e006d7e0 0x7fc0e006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:13.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.754+0000 7fc0f3400700 1 -- 172.21.15.1:0/1978655048 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc0e808e300 con 0x7fc0fc0fc520 2024-02-28T16:55:13.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.754+0000 7fc0fb400700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0e006d7e0 0x7fc0e006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:13.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.755+0000 7fc0fb400700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0e006d7e0 0x7fc0e006fc90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0041d0 tx=0x7fc0ec004160 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:13.759+0000 7fc0f3400700 1 -- 172.21.15.1:0/1978655048 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc0e805b660 con 0x7fc0fc0fc520 2024-02-28T16:55:14.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.015+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0fc04bed0 con 0x7fc0e006d7e0 2024-02-28T16:55:14.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.017+0000 7fc0f3400700 1 -- 172.21.15.1:0/1978655048 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc0fc04bed0 con 0x7fc0e006d7e0 2024-02-28T16:55:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0e006d7e0 msgr2=0x7fc0e006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0e006d7e0 0x7fc0e006fc90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0041d0 tx=0x7fc0ec004160 comp rx=0 tx=0).stop 2024-02-28T16:55:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 msgr2=0x7fc0fc11a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 0x7fc0fc11a460 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009100 tx=0x7fc0e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:14.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 shutdown_connections 2024-02-28T16:55:14.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0e006d7e0 0x7fc0e006fc90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0fc0fbb80 0x7fc0fc119f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 --2- 172.21.15.1:0/1978655048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0fc0fc520 0x7fc0fc11a460 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.020+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 >> 172.21.15.1:0/1978655048 conn(0x7fc0fc073ab0 msgr2=0x7fc0fc10d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:14.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.021+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 shutdown_connections 2024-02-28T16:55:14.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.021+0000 7fc100c00700 1 -- 172.21.15.1:0/1978655048 wait complete. 2024-02-28T16:55:14.033 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:55:14.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.182+0000 7efd0b000700 1 -- 172.21.15.1:0/2557146603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 msgr2=0x7efd04109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.182+0000 7efd0b000700 1 --2- 172.21.15.1:0/2557146603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 0x7efd04109260 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efcec004970 tx=0x7efcec009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:14.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.183+0000 7efd0b000700 1 -- 172.21.15.1:0/2557146603 shutdown_connections 2024-02-28T16:55:14.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.183+0000 7efd0b000700 1 --2- 172.21.15.1:0/2557146603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 0x7efd04109260 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.183+0000 7efd0b000700 1 --2- 172.21.15.1:0/2557146603 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd04070330 0x7efd04070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.183+0000 7efd0b000700 1 -- 172.21.15.1:0/2557146603 >> 172.21.15.1:0/2557146603 conn(0x7efd040f8ae0 msgr2=0x7efd040faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:14.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.183+0000 7efd0b000700 1 -- 172.21.15.1:0/2557146603 shutdown_connections 2024-02-28T16:55:14.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.183+0000 7efd0b000700 1 -- 172.21.15.1:0/2557146603 wait complete. 2024-02-28T16:55:14.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.184+0000 7efd0b000700 1 Processor -- start 2024-02-28T16:55:14.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.184+0000 7efd0b000700 1 -- start start 2024-02-28T16:55:14.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.184+0000 7efd0b000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd04070330 0x7efd04115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:14.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.184+0000 7efd0b000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 0x7efd04116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:14.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.184+0000 7efd0b000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd04116790 con 0x7efd04070c40 2024-02-28T16:55:14.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.184+0000 7efd0b000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0410fae0 con 0x7efd04070330 2024-02-28T16:55:14.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.185+0000 7efcfb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 0x7efd04116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.185+0000 7efcfb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 0x7efd04116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55412/0 (socket says 172.21.15.1:55412) 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.185+0000 7efcfb400700 1 -- 172.21.15.1:0/2170141030 learned_addr learned my addr 172.21.15.1:0/2170141030 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.185+0000 7efd03e00700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd04070330 0x7efd04115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.185+0000 7efcfb400700 1 -- 172.21.15.1:0/2170141030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd04070330 msgr2=0x7efd04115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.185+0000 7efcfb400700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd04070330 0x7efd04115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.185+0000 7efcfb400700 1 -- 172.21.15.1:0/2170141030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec005040 con 0x7efd04070c40 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.186+0000 7efcfb400700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 0x7efd04116140 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efcec009150 tx=0x7efcec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.186+0000 7efd00c00700 1 -- 172.21.15.1:0/2170141030 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcec00e040 con 0x7efd04070c40 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.186+0000 7efd00c00700 1 -- 172.21.15.1:0/2170141030 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efcec004290 con 0x7efd04070c40 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.186+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0410fd60 con 0x7efd04070c40 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.186+0000 7efd00c00700 1 -- 172.21.15.1:0/2170141030 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcec01c630 con 0x7efd04070c40 2024-02-28T16:55:14.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.186+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd041102a0 con 0x7efd04070c40 2024-02-28T16:55:14.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.187+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd04106920 con 0x7efd04070c40 2024-02-28T16:55:14.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.188+0000 7efd00c00700 1 -- 172.21.15.1:0/2170141030 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7efcec013980 con 0x7efd04070c40 2024-02-28T16:55:14.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.189+0000 7efd00c00700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efce806d7f0 0x7efce806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:14.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.189+0000 7efd00c00700 1 -- 172.21.15.1:0/2170141030 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efcec08e210 con 0x7efd04070c40 2024-02-28T16:55:14.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.190+0000 7efd03e00700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efce806d7f0 0x7efce806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:14.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.191+0000 7efd03e00700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efce806d7f0 0x7efce806fca0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efcf4004f30 tx=0x7efcf400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:14.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.194+0000 7efd00c00700 1 -- 172.21.15.1:0/2170141030 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efcec057ac0 con 0x7efd04070c40 2024-02-28T16:55:14.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.440+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd0404bed0 con 0x7efce806d7f0 2024-02-28T16:55:14.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.442+0000 7efd00c00700 1 -- 172.21.15.1:0/2170141030 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efd0404bed0 con 0x7efce806d7f0 2024-02-28T16:55:14.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efce806d7f0 msgr2=0x7efce806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efce806d7f0 0x7efce806fca0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efcf4004f30 tx=0x7efcf400b040 comp rx=0 tx=0).stop 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 msgr2=0x7efd04116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 0x7efd04116140 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efcec009150 tx=0x7efcec009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 shutdown_connections 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efce806d7f0 0x7efce806fca0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd04070330 0x7efd04115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 --2- 172.21.15.1:0/2170141030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd04070c40 0x7efd04116140 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.445+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 >> 172.21.15.1:0/2170141030 conn(0x7efd040f8ae0 msgr2=0x7efd04103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.446+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 shutdown_connections 2024-02-28T16:55:14.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.446+0000 7efd0b000700 1 -- 172.21.15.1:0/2170141030 wait complete. 2024-02-28T16:55:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.609+0000 7f37a5200700 1 -- 172.21.15.1:0/884902783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a0102f50 msgr2=0x7f37a0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.609+0000 7f37a5200700 1 --2- 172.21.15.1:0/884902783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a0102f50 0x7f37a0103320 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3790004940 tx=0x7f3790009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.609+0000 7f37a5200700 1 -- 172.21.15.1:0/884902783 shutdown_connections 2024-02-28T16:55:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.609+0000 7f37a5200700 1 --2- 172.21.15.1:0/884902783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37a01038f0 0x7f37a00fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.609+0000 7f37a5200700 1 --2- 172.21.15.1:0/884902783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a0102f50 0x7f37a0103320 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.609+0000 7f37a5200700 1 -- 172.21.15.1:0/884902783 >> 172.21.15.1:0/884902783 conn(0x7f37a00f8ae0 msgr2=0x7f37a00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.609+0000 7f37a5200700 1 -- 172.21.15.1:0/884902783 shutdown_connections 2024-02-28T16:55:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.609+0000 7f37a5200700 1 -- 172.21.15.1:0/884902783 wait complete. 2024-02-28T16:55:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.610+0000 7f37a5200700 1 Processor -- start 2024-02-28T16:55:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.610+0000 7f37a5200700 1 -- start start 2024-02-28T16:55:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.611+0000 7f37a5200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37a0102f50 0x7f37a0113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.611+0000 7f37a5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a01038f0 0x7f37a0113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.611+0000 7f37a5200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a01145e0 con 0x7f37a01038f0 2024-02-28T16:55:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.611+0000 7f37a5200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a010d910 con 0x7f37a0102f50 2024-02-28T16:55:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.611+0000 7f379d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a01038f0 0x7f37a0113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.611+0000 7f379d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a01038f0 0x7f37a0113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55430/0 (socket says 172.21.15.1:55430) 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.611+0000 7f379d400700 1 -- 172.21.15.1:0/4136836575 learned_addr learned my addr 172.21.15.1:0/4136836575 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.611+0000 7f379de00700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37a0102f50 0x7f37a0113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.612+0000 7f379d400700 1 -- 172.21.15.1:0/4136836575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37a0102f50 msgr2=0x7f37a0113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.612+0000 7f379d400700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37a0102f50 0x7f37a0113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.612+0000 7f379d400700 1 -- 172.21.15.1:0/4136836575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3790005040 con 0x7f37a01038f0 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.612+0000 7f379d400700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a01038f0 0x7f37a0113f90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c800 tx=0x7f378c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.612+0000 7f3796000700 1 -- 172.21.15.1:0/4136836575 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f378c00a760 con 0x7f37a01038f0 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.612+0000 7f3796000700 1 -- 172.21.15.1:0/4136836575 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f378c00ad90 con 0x7f37a01038f0 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.612+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a010dbf0 con 0x7f37a01038f0 2024-02-28T16:55:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.612+0000 7f3796000700 1 -- 172.21.15.1:0/4136836575 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f378c0113a0 con 0x7f37a01038f0 2024-02-28T16:55:14.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.613+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a010e130 con 0x7f37a01038f0 2024-02-28T16:55:14.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.614+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a00fc6e0 con 0x7f37a01038f0 2024-02-28T16:55:14.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.615+0000 7f3796000700 1 -- 172.21.15.1:0/4136836575 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f378c012400 con 0x7f37a01038f0 2024-02-28T16:55:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.616+0000 7f3796000700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f378406d7f0 0x7f378406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.616+0000 7f3796000700 1 -- 172.21.15.1:0/4136836575 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f378c08cfa0 con 0x7f37a01038f0 2024-02-28T16:55:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.616+0000 7f379de00700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f378406d7f0 0x7f378406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.617+0000 7f379de00700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f378406d7f0 0x7f378406fca0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f379000b020 tx=0x7f3790009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:14.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.621+0000 7f3796000700 1 -- 172.21.15.1:0/4136836575 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f378c05a300 con 0x7f37a01038f0 2024-02-28T16:55:14.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:14 smithi001 ceph-mon[112940]: pgmap v623: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:14.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.858+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f37a004bf70 con 0x7f378406d7f0 2024-02-28T16:55:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.887+0000 7f3796000700 1 -- 172.21.15.1:0/4136836575 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f379000b5c0 con 0x7f378406d7f0 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (21m) 13m ago 35m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (18m) 13m ago 35m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (17m) 14m ago 34m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (20m) 13m ago 35m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (13m) 13m ago 28m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (13m) 13m ago 28m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (28m) 14m ago 28m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (28m) 14m ago 28m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:55:14.884 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (23m) 13m ago 37m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (22m) 14m ago 34m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (19m) 13m ago 37m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (19m) 14m ago 34m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (22m) 13m ago 35m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (22m) 14m ago 34m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (17m) 13m ago 33m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (16m) 13m ago 32m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (16m) 13m ago 32m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (15m) 14m ago 31m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (15m) 14m ago 30m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (14m) 14m ago 30m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:55:14.885 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (21m) 13m ago 34m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:55:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.890+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f378406d7f0 msgr2=0x7f378406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.890+0000 7f37a5200700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f378406d7f0 0x7f378406fca0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f379000b020 tx=0x7f3790009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.890+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a01038f0 msgr2=0x7f37a0113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.890+0000 7f37a5200700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a01038f0 0x7f37a0113f90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c800 tx=0x7f378c004470 comp rx=0 tx=0).stop 2024-02-28T16:55:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.891+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 shutdown_connections 2024-02-28T16:55:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.891+0000 7f37a5200700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f378406d7f0 0x7f378406fca0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.891+0000 7f37a5200700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37a0102f50 0x7f37a0113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.891+0000 7f37a5200700 1 --2- 172.21.15.1:0/4136836575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37a01038f0 0x7f37a0113f90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.891+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 >> 172.21.15.1:0/4136836575 conn(0x7f37a00f8ae0 msgr2=0x7f37a00fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.891+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 shutdown_connections 2024-02-28T16:55:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:14.891+0000 7f37a5200700 1 -- 172.21.15.1:0/4136836575 wait complete. 2024-02-28T16:55:15.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:14 smithi066 ceph-mon[85781]: pgmap v623: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.045+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2308566362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 msgr2=0x7ffb180fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.045+0000 7ffb1ee00700 1 --2- 172.21.15.1:0/2308566362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 0x7ffb180fcf60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffb04004940 tx=0x7ffb04009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.045+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2308566362 shutdown_connections 2024-02-28T16:55:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.045+0000 7ffb1ee00700 1 --2- 172.21.15.1:0/2308566362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 0x7ffb180fcf60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.045+0000 7ffb1ee00700 1 --2- 172.21.15.1:0/2308566362 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb18102f50 0x7ffb18103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.045+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2308566362 >> 172.21.15.1:0/2308566362 conn(0x7ffb180f8ae0 msgr2=0x7ffb180faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.046+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2308566362 shutdown_connections 2024-02-28T16:55:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.046+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2308566362 wait complete. 2024-02-28T16:55:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.046+0000 7ffb1ee00700 1 Processor -- start 2024-02-28T16:55:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.047+0000 7ffb1ee00700 1 -- start start 2024-02-28T16:55:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.047+0000 7ffb1ee00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb18102f50 0x7ffb18115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.047+0000 7ffb1ee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 0x7ffb181161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.047+0000 7ffb1ee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18116810 con 0x7ffb181038f0 2024-02-28T16:55:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.047+0000 7ffb1ee00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1810fb40 con 0x7ffb18102f50 2024-02-28T16:55:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.047+0000 7ffb17400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 0x7ffb181161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.047+0000 7ffb17400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 0x7ffb181161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55450/0 (socket says 172.21.15.1:55450) 2024-02-28T16:55:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.047+0000 7ffb17400700 1 -- 172.21.15.1:0/2553682765 learned_addr learned my addr 172.21.15.1:0/2553682765 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb17e00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb18102f50 0x7ffb18115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb17400700 1 -- 172.21.15.1:0/2553682765 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb18102f50 msgr2=0x7ffb18115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb17400700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb18102f50 0x7ffb18115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb17400700 1 -- 172.21.15.1:0/2553682765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04005040 con 0x7ffb181038f0 2024-02-28T16:55:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb17e00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb18102f50 0x7ffb18115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:55:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb17400700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 0x7ffb181161c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009030 tx=0x7ffb04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb0fe00700 1 -- 172.21.15.1:0/2553682765 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0400e040 con 0x7ffb181038f0 2024-02-28T16:55:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb0fe00700 1 -- 172.21.15.1:0/2553682765 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb0400bbf0 con 0x7ffb181038f0 2024-02-28T16:55:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb0fe00700 1 -- 172.21.15.1:0/2553682765 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0401c630 con 0x7ffb181038f0 2024-02-28T16:55:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.048+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1810fdc0 con 0x7ffb181038f0 2024-02-28T16:55:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.049+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb18110300 con 0x7ffb181038f0 2024-02-28T16:55:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.050+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb180fc6e0 con 0x7ffb181038f0 2024-02-28T16:55:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.051+0000 7ffb0fe00700 1 -- 172.21.15.1:0/2553682765 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffb0400bd60 con 0x7ffb181038f0 2024-02-28T16:55:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.051+0000 7ffb0fe00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffafc06d7f0 0x7ffafc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.052+0000 7ffb0fe00700 1 -- 172.21.15.1:0/2553682765 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffb0408e220 con 0x7ffb181038f0 2024-02-28T16:55:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.052+0000 7ffb17e00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffafc06d7f0 0x7ffafc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.052+0000 7ffb17e00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffafc06d7f0 0x7ffafc06fca0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffb08007fd0 tx=0x7ffb08009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:15.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.056+0000 7ffb0fe00700 1 -- 172.21.15.1:0/2553682765 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffb0405b620 con 0x7ffb181038f0 2024-02-28T16:55:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.373+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffb181111f0 con 0x7ffb181038f0 2024-02-28T16:55:15.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.374+0000 7ffb0fe00700 1 -- 172.21.15.1:0/2553682765 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7ffb0405b1b0 con 0x7ffb181038f0 2024-02-28T16:55:15.370 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:55:15.371 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:55:15.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:55:15.372 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:55:15.372 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:55:15.372 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:55:15.372 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:55:15.372 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffafc06d7f0 msgr2=0x7ffafc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffafc06d7f0 0x7ffafc06fca0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffb08007fd0 tx=0x7ffb08009040 comp rx=0 tx=0).stop 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 msgr2=0x7ffb181161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 0x7ffb181161c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009030 tx=0x7ffb04009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 shutdown_connections 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffafc06d7f0 0x7ffafc06fca0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb18102f50 0x7ffb18115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 --2- 172.21.15.1:0/2553682765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb181038f0 0x7ffb181161c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 >> 172.21.15.1:0/2553682765 conn(0x7ffb180f8ae0 msgr2=0x7ffb180fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 shutdown_connections 2024-02-28T16:55:15.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.378+0000 7ffb1ee00700 1 -- 172.21.15.1:0/2553682765 wait complete. 2024-02-28T16:55:15.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.538+0000 7fec0b400700 1 -- 172.21.15.1:0/128674301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fe8f0 msgr2=0x7fec04107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.538+0000 7fec0b400700 1 --2- 172.21.15.1:0/128674301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fe8f0 0x7fec04107050 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7febf0004970 tx=0x7febf0009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.538+0000 7fec0b400700 1 -- 172.21.15.1:0/128674301 shutdown_connections 2024-02-28T16:55:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.538+0000 7fec0b400700 1 --2- 172.21.15.1:0/128674301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fe8f0 0x7fec04107050 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.538+0000 7fec0b400700 1 --2- 172.21.15.1:0/128674301 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec040fdfe0 0x7fec040fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.538+0000 7fec0b400700 1 -- 172.21.15.1:0/128674301 >> 172.21.15.1:0/128674301 conn(0x7fec04071d90 msgr2=0x7fec04072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.539+0000 7fec0b400700 1 -- 172.21.15.1:0/128674301 shutdown_connections 2024-02-28T16:55:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.539+0000 7fec0b400700 1 -- 172.21.15.1:0/128674301 wait complete. 2024-02-28T16:55:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.540+0000 7fec0b400700 1 Processor -- start 2024-02-28T16:55:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.540+0000 7fec0b400700 1 -- start start 2024-02-28T16:55:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.540+0000 7fec0b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fdfe0 0x7fec04115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.540+0000 7fec0b400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec040fe8f0 0x7fec04116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.540+0000 7fec0b400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec04116780 con 0x7fec040fdfe0 2024-02-28T16:55:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.540+0000 7fec0b400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0410fab0 con 0x7fec040fe8f0 2024-02-28T16:55:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec08a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fdfe0 0x7fec04115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec08a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fdfe0 0x7fec04115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55470/0 (socket says 172.21.15.1:55470) 2024-02-28T16:55:15.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec08a00700 1 -- 172.21.15.1:0/3956645215 learned_addr learned my addr 172.21.15.1:0/3956645215 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec03e00700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec040fe8f0 0x7fec04116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec08a00700 1 -- 172.21.15.1:0/3956645215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec040fe8f0 msgr2=0x7fec04116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec08a00700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec040fe8f0 0x7fec04116130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec08a00700 1 -- 172.21.15.1:0/3956645215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf0005040 con 0x7fec040fdfe0 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec03e00700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec040fe8f0 0x7fec04116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.541+0000 7fec08a00700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fdfe0 0x7fec04115bf0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7febf800c830 tx=0x7febf8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.542+0000 7fec00c00700 1 -- 172.21.15.1:0/3956645215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febf800a760 con 0x7fec040fdfe0 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.542+0000 7fec00c00700 1 -- 172.21.15.1:0/3956645215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febf800ad90 con 0x7fec040fdfe0 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.542+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0410fd90 con 0x7fec040fdfe0 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.542+0000 7fec00c00700 1 -- 172.21.15.1:0/3956645215 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febf80113a0 con 0x7fec040fdfe0 2024-02-28T16:55:15.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.542+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec041102d0 con 0x7fec040fdfe0 2024-02-28T16:55:15.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.543+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec04106ac0 con 0x7fec040fdfe0 2024-02-28T16:55:15.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.544+0000 7fec00c00700 1 -- 172.21.15.1:0/3956645215 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7febf8012400 con 0x7fec040fdfe0 2024-02-28T16:55:15.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.545+0000 7fec00c00700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febec06d7e0 0x7febec06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.545+0000 7fec00c00700 1 -- 172.21.15.1:0/3956645215 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7febf808cfa0 con 0x7fec040fdfe0 2024-02-28T16:55:15.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.545+0000 7fec03e00700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febec06d7e0 0x7febec06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.546+0000 7fec03e00700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febec06d7e0 0x7febec06fc90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7febf0009fd0 tx=0x7febf0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:15.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.550+0000 7fec00c00700 1 -- 172.21.15.1:0/3956645215 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7febf805a300 con 0x7fec040fdfe0 2024-02-28T16:55:15.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:15 smithi001 ceph-mon[112940]: from='client.35146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:15.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:15 smithi001 ceph-mon[112940]: from='client.35150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:15.777 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:15 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2553682765' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:55:15.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.824+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fec0404bf00 con 0x7fec040fdfe0 2024-02-28T16:55:15.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.825+0000 7fec00c00700 1 -- 172.21.15.1:0/3956645215 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7febf8059e90 con 0x7fec040fdfe0 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:55:15.821 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:55:15.822 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:55:15.823 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:55:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:55:15.824 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:55:15.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:55:15.824 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:55:15.824 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:55:15.824 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:55:15.824 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febec06d7e0 msgr2=0x7febec06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febec06d7e0 0x7febec06fc90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7febf0009fd0 tx=0x7febf0009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fdfe0 msgr2=0x7fec04115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fdfe0 0x7fec04115bf0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7febf800c830 tx=0x7febf8004470 comp rx=0 tx=0).stop 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 shutdown_connections 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7febec06d7e0 0x7febec06fc90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec040fdfe0 0x7fec04115bf0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 --2- 172.21.15.1:0/3956645215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec040fe8f0 0x7fec04116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 >> 172.21.15.1:0/3956645215 conn(0x7fec04071d90 msgr2=0x7fec04103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:15.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.828+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 shutdown_connections 2024-02-28T16:55:15.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.829+0000 7fec0b400700 1 -- 172.21.15.1:0/3956645215 wait complete. 2024-02-28T16:55:15.826 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:55:15.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.988+0000 7f34d2200700 1 -- 172.21.15.1:0/950306663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 msgr2=0x7f34cc105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.988+0000 7f34d2200700 1 --2- 172.21.15.1:0/950306663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 0x7f34cc105530 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f34b8004940 tx=0x7f34b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:15.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.988+0000 7f34d2200700 1 -- 172.21.15.1:0/950306663 shutdown_connections 2024-02-28T16:55:15.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.988+0000 7f34d2200700 1 --2- 172.21.15.1:0/950306663 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34cc105b00 0x7f34cc0ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.988+0000 7f34d2200700 1 --2- 172.21.15.1:0/950306663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 0x7f34cc105530 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.988+0000 7f34d2200700 1 -- 172.21.15.1:0/950306663 >> 172.21.15.1:0/950306663 conn(0x7f34cc0fad30 msgr2=0x7f34cc0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:15.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.988+0000 7f34d2200700 1 -- 172.21.15.1:0/950306663 shutdown_connections 2024-02-28T16:55:15.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.988+0000 7f34d2200700 1 -- 172.21.15.1:0/950306663 wait complete. 2024-02-28T16:55:15.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.989+0000 7f34d2200700 1 Processor -- start 2024-02-28T16:55:15.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.989+0000 7f34d2200700 1 -- start start 2024-02-28T16:55:15.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.990+0000 7f34d2200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 0x7f34cc115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.990+0000 7f34d2200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34cc105b00 0x7f34cc1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.990+0000 7f34d2200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc1167f0 con 0x7f34cc105160 2024-02-28T16:55:15.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.990+0000 7f34d2200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc10fb40 con 0x7f34cc105b00 2024-02-28T16:55:15.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.990+0000 7f34cb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 0x7f34cc115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.990+0000 7f34cb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 0x7f34cc115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55498/0 (socket says 172.21.15.1:55498) 2024-02-28T16:55:15.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.990+0000 7f34cb400700 1 -- 172.21.15.1:0/3274046140 learned_addr learned my addr 172.21.15.1:0/3274046140 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:15.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34cb400700 1 -- 172.21.15.1:0/3274046140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34cc105b00 msgr2=0x7f34cc1161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:15.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34caa00700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34cc105b00 0x7f34cc1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34cb400700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34cc105b00 0x7f34cc1161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:15.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34cb400700 1 -- 172.21.15.1:0/3274046140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b8005040 con 0x7f34cc105160 2024-02-28T16:55:15.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34cb400700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 0x7f34cc115c60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009fd0 tx=0x7f34b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:15.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34c3400700 1 -- 172.21.15.1:0/3274046140 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34b800e040 con 0x7f34cc105160 2024-02-28T16:55:15.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34c3400700 1 -- 172.21.15.1:0/3274046140 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34b800bbf0 con 0x7f34cc105160 2024-02-28T16:55:15.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc10fdc0 con 0x7f34cc105160 2024-02-28T16:55:15.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.991+0000 7f34c3400700 1 -- 172.21.15.1:0/3274046140 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34b801c3f0 con 0x7f34cc105160 2024-02-28T16:55:15.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.992+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc110300 con 0x7f34cc105160 2024-02-28T16:55:15.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.993+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc102470 con 0x7f34cc105160 2024-02-28T16:55:15.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.994+0000 7f34c3400700 1 -- 172.21.15.1:0/3274046140 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f34b800bd60 con 0x7f34cc105160 2024-02-28T16:55:15.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.995+0000 7f34c3400700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34b006d7f0 0x7f34b006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:15.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.995+0000 7f34c3400700 1 -- 172.21.15.1:0/3274046140 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f34b808e1c0 con 0x7f34cc105160 2024-02-28T16:55:15.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.995+0000 7f34caa00700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34b006d7f0 0x7f34b006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:15.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.996+0000 7f34caa00700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34b006d7f0 0x7f34b006fca0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f34cc111540 tx=0x7f34bc00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:15.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:15.999+0000 7f34c3400700 1 -- 172.21.15.1:0/3274046140 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f34b8057a70 con 0x7f34cc105160 2024-02-28T16:55:16.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:15 smithi066 ceph-mon[85781]: from='client.35146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:16.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:15 smithi066 ceph-mon[85781]: from='client.35150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:16.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:15 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2553682765' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:55:16.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.249+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f34cc04bf40 con 0x7f34b006d7f0 2024-02-28T16:55:16.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.252+0000 7f34c3400700 1 -- 172.21.15.1:0/3274046140 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f34cc04bf40 con 0x7f34b006d7f0 2024-02-28T16:55:16.248 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:55:16.248 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:55:16.249 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:55:16.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34b006d7f0 msgr2=0x7f34b006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:16.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34b006d7f0 0x7f34b006fca0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f34cc111540 tx=0x7f34bc00b040 comp rx=0 tx=0).stop 2024-02-28T16:55:16.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 msgr2=0x7f34cc115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:16.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 0x7f34cc115c60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009fd0 tx=0x7f34b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:16.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 shutdown_connections 2024-02-28T16:55:16.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f34b006d7f0 0x7f34b006fca0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34cc105160 0x7f34cc115c60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 --2- 172.21.15.1:0/3274046140 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f34cc105b00 0x7f34cc1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 >> 172.21.15.1:0/3274046140 conn(0x7f34cc0fad30 msgr2=0x7f34cc0fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:16.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 shutdown_connections 2024-02-28T16:55:16.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.256+0000 7f34d2200700 1 -- 172.21.15.1:0/3274046140 wait complete. 2024-02-28T16:55:16.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.416+0000 7fd011200700 1 -- 172.21.15.1:0/1748786806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105b10 msgr2=0x7fd00c0ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:16.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.416+0000 7fd011200700 1 --2- 172.21.15.1:0/1748786806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105b10 0x7fd00c0ff180 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fcff4004940 tx=0x7fcff4009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:16.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.416+0000 7fd011200700 1 -- 172.21.15.1:0/1748786806 shutdown_connections 2024-02-28T16:55:16.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.416+0000 7fd011200700 1 --2- 172.21.15.1:0/1748786806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105b10 0x7fd00c0ff180 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.416+0000 7fd011200700 1 --2- 172.21.15.1:0/1748786806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd00c105170 0x7fd00c105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.416+0000 7fd011200700 1 -- 172.21.15.1:0/1748786806 >> 172.21.15.1:0/1748786806 conn(0x7fd00c0face0 msgr2=0x7fd00c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:16.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.416+0000 7fd011200700 1 -- 172.21.15.1:0/1748786806 shutdown_connections 2024-02-28T16:55:16.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.416+0000 7fd011200700 1 -- 172.21.15.1:0/1748786806 wait complete. 2024-02-28T16:55:16.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.417+0000 7fd011200700 1 Processor -- start 2024-02-28T16:55:16.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.418+0000 7fd011200700 1 -- start start 2024-02-28T16:55:16.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.418+0000 7fd011200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105170 0x7fd00c115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:16.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.418+0000 7fd011200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd00c105b10 0x7fd00c116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.418+0000 7fd011200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c116870 con 0x7fd00c105170 2024-02-28T16:55:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.418+0000 7fd011200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c10fb80 con 0x7fd00c105b10 2024-02-28T16:55:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd009e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105170 0x7fd00c115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd009e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105170 0x7fd00c115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55510/0 (socket says 172.21.15.1:55510) 2024-02-28T16:55:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd009e00700 1 -- 172.21.15.1:0/1522170378 learned_addr learned my addr 172.21.15.1:0/1522170378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd003e00700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd00c105b10 0x7fd00c116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd009e00700 1 -- 172.21.15.1:0/1522170378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd00c105b10 msgr2=0x7fd00c116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd009e00700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd00c105b10 0x7fd00c116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd009e00700 1 -- 172.21.15.1:0/1522170378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff4005040 con 0x7fd00c105170 2024-02-28T16:55:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd003e00700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd00c105b10 0x7fd00c116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:55:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.419+0000 7fd009e00700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105170 0x7fd00c115ce0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00c8c0 tx=0x7fcffc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.420+0000 7fd002000700 1 -- 172.21.15.1:0/1522170378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcffc00a980 con 0x7fd00c105170 2024-02-28T16:55:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.420+0000 7fd002000700 1 -- 172.21.15.1:0/1522170378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcffc00ce60 con 0x7fd00c105170 2024-02-28T16:55:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.420+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00c10fe60 con 0x7fd00c105170 2024-02-28T16:55:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.420+0000 7fd002000700 1 -- 172.21.15.1:0/1522170378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcffc0115e0 con 0x7fd00c105170 2024-02-28T16:55:16.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.420+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00c1103a0 con 0x7fd00c105170 2024-02-28T16:55:16.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.421+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00c107640 con 0x7fd00c105170 2024-02-28T16:55:16.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.422+0000 7fd002000700 1 -- 172.21.15.1:0/1522170378 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcffc012400 con 0x7fd00c105170 2024-02-28T16:55:16.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.423+0000 7fd002000700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcff006d7f0 0x7fcff006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:16.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.424+0000 7fd002000700 1 -- 172.21.15.1:0/1522170378 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcffc08d040 con 0x7fd00c105170 2024-02-28T16:55:16.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.424+0000 7fd003e00700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcff006d7f0 0x7fcff006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:16.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.424+0000 7fd003e00700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcff006d7f0 0x7fcff006fca0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b3b0 tx=0x7fcff4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.429+0000 7fd002000700 1 -- 172.21.15.1:0/1522170378 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcffc05a3a0 con 0x7fd00c105170 2024-02-28T16:55:16.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.749+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd00c0fe900 con 0x7fd00c105170 2024-02-28T16:55:16.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.750+0000 7fd002000700 1 -- 172.21.15.1:0/1522170378 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fd00c0fe900 con 0x7fd00c105170 2024-02-28T16:55:16.746 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:55:16.746 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:55:16.746 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:55:16.746 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:55:16.746 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:55:16.746 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:55:16.746 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcff006d7f0 msgr2=0x7fcff006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcff006d7f0 0x7fcff006fca0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b3b0 tx=0x7fcff4009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105170 msgr2=0x7fd00c115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105170 0x7fd00c115ce0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00c8c0 tx=0x7fcffc0044c0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 shutdown_connections 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcff006d7f0 0x7fcff006fca0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd00c105170 0x7fd00c115ce0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 --2- 172.21.15.1:0/1522170378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd00c105b10 0x7fd00c116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.753+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 >> 172.21.15.1:0/1522170378 conn(0x7fd00c0face0 msgr2=0x7fd00c0fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:16.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.754+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 shutdown_connections 2024-02-28T16:55:16.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:16.754+0000 7fd011200700 1 -- 172.21.15.1:0/1522170378 wait complete. 2024-02-28T16:55:17.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:16 smithi066 ceph-mon[85781]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:17.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:16 smithi066 ceph-mon[85781]: pgmap v624: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:17.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:16 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3956645215' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:55:17.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:16 smithi001 ceph-mon[112940]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:17.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:16 smithi001 ceph-mon[112940]: pgmap v624: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:17.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:16 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3956645215' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:55:18.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:17 smithi066 ceph-mon[85781]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:18.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:17 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1522170378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:55:18.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:17 smithi001 ceph-mon[112940]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:18.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:17 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1522170378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:55:19.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:18 smithi066 ceph-mon[85781]: pgmap v625: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:19.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:18 smithi001 ceph-mon[112940]: pgmap v625: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:20.849 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:20 smithi001 ceph-mon[112940]: pgmap v626: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:21.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:20 smithi066 ceph-mon[85781]: pgmap v626: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:23.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:22 smithi066 ceph-mon[85781]: pgmap v627: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:23.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:22 smithi001 ceph-mon[112940]: pgmap v627: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:25.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:24 smithi066 ceph-mon[85781]: pgmap v628: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:25.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:24 smithi001 ceph-mon[112940]: pgmap v628: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:27.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:26 smithi066 ceph-mon[85781]: pgmap v629: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:27.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:55:27.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:26 smithi001 ceph-mon[112940]: pgmap v629: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:27.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:55:29.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:28 smithi066 ceph-mon[85781]: pgmap v630: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:29.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:28 smithi001 ceph-mon[112940]: pgmap v630: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:31.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:30 smithi066 ceph-mon[85781]: pgmap v631: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:31.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:30 smithi001 ceph-mon[112940]: pgmap v631: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:33.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:32 smithi066 ceph-mon[85781]: pgmap v632: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:33.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:32 smithi001 ceph-mon[112940]: pgmap v632: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:35.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:34 smithi066 ceph-mon[85781]: pgmap v633: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:35.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:34 smithi001 ceph-mon[112940]: pgmap v633: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:37.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:36 smithi066 ceph-mon[85781]: pgmap v634: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:37.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:36 smithi001 ceph-mon[112940]: pgmap v634: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:39.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:38 smithi066 ceph-mon[85781]: pgmap v635: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:39.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:38 smithi001 ceph-mon[112940]: pgmap v635: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:41.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:40 smithi066 ceph-mon[85781]: pgmap v636: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:41.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:40 smithi001 ceph-mon[112940]: pgmap v636: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:42.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:55:42.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:55:43.005 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:42 smithi066 ceph-mon[85781]: pgmap v637: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:43.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:42 smithi001 ceph-mon[112940]: pgmap v637: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:45.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:44 smithi066 ceph-mon[85781]: pgmap v638: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:45.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:44 smithi001 ceph-mon[112940]: pgmap v638: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.926+0000 7f9ccac00700 1 -- 172.21.15.1:0/250847848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070330 msgr2=0x7f9cc4070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.926+0000 7f9ccac00700 1 --2- 172.21.15.1:0/250847848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070330 0x7f9cc4070700 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4004970 tx=0x7f9cb4009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.926+0000 7f9ccac00700 1 -- 172.21.15.1:0/250847848 shutdown_connections 2024-02-28T16:55:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.926+0000 7f9ccac00700 1 --2- 172.21.15.1:0/250847848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9cc4070c40 0x7f9cc4109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.926+0000 7f9ccac00700 1 --2- 172.21.15.1:0/250847848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070330 0x7f9cc4070700 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.926+0000 7f9ccac00700 1 -- 172.21.15.1:0/250847848 >> 172.21.15.1:0/250847848 conn(0x7f9cc40f8af0 msgr2=0x7f9cc40faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.927+0000 7f9ccac00700 1 -- 172.21.15.1:0/250847848 shutdown_connections 2024-02-28T16:55:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.927+0000 7f9ccac00700 1 -- 172.21.15.1:0/250847848 wait complete. 2024-02-28T16:55:46.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.927+0000 7f9ccac00700 1 Processor -- start 2024-02-28T16:55:46.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.928+0000 7f9ccac00700 1 -- start start 2024-02-28T16:55:46.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.928+0000 7f9ccac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9cc4070330 0x7f9cc4115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:46.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.928+0000 7f9ccac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070c40 0x7f9cc4116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.928+0000 7f9ccac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc4116780 con 0x7f9cc4070c40 2024-02-28T16:55:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.928+0000 7f9ccac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc410fad0 con 0x7f9cc4070330 2024-02-28T16:55:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.928+0000 7f9cc3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070c40 0x7f9cc4116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.928+0000 7f9cc3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070c40 0x7f9cc4116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44158/0 (socket says 172.21.15.1:44158) 2024-02-28T16:55:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.928+0000 7f9cc3400700 1 -- 172.21.15.1:0/3218562770 learned_addr learned my addr 172.21.15.1:0/3218562770 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.929+0000 7f9cc3e00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9cc4070330 0x7f9cc4115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.929+0000 7f9cc3400700 1 -- 172.21.15.1:0/3218562770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9cc4070330 msgr2=0x7f9cc4115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.929+0000 7f9cc3400700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9cc4070330 0x7f9cc4115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.929+0000 7f9cc3400700 1 -- 172.21.15.1:0/3218562770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb4005040 con 0x7f9cc4070c40 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.929+0000 7f9cc3400700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070c40 0x7f9cc4116130 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0002a50 tx=0x7f9cb00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.929+0000 7f9cbbe00700 1 -- 172.21.15.1:0/3218562770 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cb0007cc0 con 0x7f9cc4070c40 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.930+0000 7f9cbbe00700 1 -- 172.21.15.1:0/3218562770 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cb000f040 con 0x7f9cc4070c40 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.930+0000 7f9cbbe00700 1 -- 172.21.15.1:0/3218562770 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cb0009640 con 0x7f9cc4070c40 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.930+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc410fdb0 con 0x7f9cc4070c40 2024-02-28T16:55:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.930+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc41102f0 con 0x7f9cc4070c40 2024-02-28T16:55:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.931+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc4106910 con 0x7f9cc4070c40 2024-02-28T16:55:46.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.932+0000 7f9cbbe00700 1 -- 172.21.15.1:0/3218562770 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9cb00097a0 con 0x7f9cc4070c40 2024-02-28T16:55:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.933+0000 7f9cbbe00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ca806d7e0 0x7f9ca806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.933+0000 7f9cbbe00700 1 -- 172.21.15.1:0/3218562770 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9cb008cc60 con 0x7f9cc4070c40 2024-02-28T16:55:46.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.934+0000 7f9cc3e00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ca806d7e0 0x7f9ca806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:46.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.934+0000 7f9cc3e00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ca806d7e0 0x7f9ca806fc90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4000c00 tx=0x7f9cb4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:46.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:46.938+0000 7f9cbbe00700 1 -- 172.21.15.1:0/3218562770 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9cb0056490 con 0x7f9cc4070c40 2024-02-28T16:55:46.936 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:46 smithi001 ceph-mon[112940]: pgmap v639: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:47.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:46 smithi066 ceph-mon[85781]: pgmap v639: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:47.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.182+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cc404bed0 con 0x7f9ca806d7e0 2024-02-28T16:55:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.185+0000 7f9cbbe00700 1 -- 172.21.15.1:0/3218562770 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9cc404bed0 con 0x7f9ca806d7e0 2024-02-28T16:55:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.187+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ca806d7e0 msgr2=0x7f9ca806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.187+0000 7f9ccac00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ca806d7e0 0x7f9ca806fc90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4000c00 tx=0x7f9cb4009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.187+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070c40 msgr2=0x7f9cc4116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.187+0000 7f9ccac00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070c40 0x7f9cc4116130 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0002a50 tx=0x7f9cb00046b0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.188+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 shutdown_connections 2024-02-28T16:55:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.188+0000 7f9ccac00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9ca806d7e0 0x7f9ca806fc90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.188+0000 7f9ccac00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9cc4070330 0x7f9cc4115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.188+0000 7f9ccac00700 1 --2- 172.21.15.1:0/3218562770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc4070c40 0x7f9cc4116130 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.188+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 >> 172.21.15.1:0/3218562770 conn(0x7f9cc40f8af0 msgr2=0x7f9cc4103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.188+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 shutdown_connections 2024-02-28T16:55:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.188+0000 7f9ccac00700 1 -- 172.21.15.1:0/3218562770 wait complete. 2024-02-28T16:55:47.202 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:55:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.344+0000 7f8e77200700 1 -- 172.21.15.1:0/57309707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e701038f0 msgr2=0x7f8e700fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.344+0000 7f8e77200700 1 --2- 172.21.15.1:0/57309707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e701038f0 0x7f8e700fcf60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8e58004940 tx=0x7f8e58009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.344+0000 7f8e77200700 1 -- 172.21.15.1:0/57309707 shutdown_connections 2024-02-28T16:55:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.344+0000 7f8e77200700 1 --2- 172.21.15.1:0/57309707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e701038f0 0x7f8e700fcf60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.344+0000 7f8e77200700 1 --2- 172.21.15.1:0/57309707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e70102f50 0x7f8e70103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.344+0000 7f8e77200700 1 -- 172.21.15.1:0/57309707 >> 172.21.15.1:0/57309707 conn(0x7f8e700f8ae0 msgr2=0x7f8e700faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.345+0000 7f8e77200700 1 -- 172.21.15.1:0/57309707 shutdown_connections 2024-02-28T16:55:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.345+0000 7f8e77200700 1 -- 172.21.15.1:0/57309707 wait complete. 2024-02-28T16:55:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.345+0000 7f8e77200700 1 Processor -- start 2024-02-28T16:55:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.346+0000 7f8e77200700 1 -- start start 2024-02-28T16:55:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.346+0000 7f8e77200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e70102f50 0x7f8e70113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.346+0000 7f8e77200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e701038f0 0x7f8e70113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.346+0000 7f8e77200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70114600 con 0x7f8e70102f50 2024-02-28T16:55:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.346+0000 7f8e77200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7010d930 con 0x7f8e701038f0 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.346+0000 7f8e6fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e70102f50 0x7f8e70113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.346+0000 7f8e6fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e70102f50 0x7f8e70113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44186/0 (socket says 172.21.15.1:44186) 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.346+0000 7f8e6fe00700 1 -- 172.21.15.1:0/3913123341 learned_addr learned my addr 172.21.15.1:0/3913123341 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e67e00700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e701038f0 0x7f8e70113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e6fe00700 1 -- 172.21.15.1:0/3913123341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e701038f0 msgr2=0x7f8e70113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e6fe00700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e701038f0 0x7f8e70113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e6fe00700 1 -- 172.21.15.1:0/3913123341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58005040 con 0x7f8e70102f50 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e6fe00700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e70102f50 0x7f8e70113a70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8e60002a50 tx=0x7f8e600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e6cc00700 1 -- 172.21.15.1:0/3913123341 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e60007cc0 con 0x7f8e70102f50 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e6cc00700 1 -- 172.21.15.1:0/3913123341 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e6000f040 con 0x7f8e70102f50 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e6cc00700 1 -- 172.21.15.1:0/3913123341 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e60009640 con 0x7f8e70102f50 2024-02-28T16:55:47.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.347+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7010dc10 con 0x7f8e70102f50 2024-02-28T16:55:47.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.348+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7010e150 con 0x7f8e70102f50 2024-02-28T16:55:47.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.349+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e70105420 con 0x7f8e70102f50 2024-02-28T16:55:47.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.350+0000 7f8e6cc00700 1 -- 172.21.15.1:0/3913123341 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8e600097a0 con 0x7f8e70102f50 2024-02-28T16:55:47.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.351+0000 7f8e6cc00700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e5406d7f0 0x7f8e5406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:47.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.351+0000 7f8e6cc00700 1 -- 172.21.15.1:0/3913123341 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8e6008cc60 con 0x7f8e70102f50 2024-02-28T16:55:47.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.351+0000 7f8e67e00700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e5406d7f0 0x7f8e5406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:47.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.352+0000 7f8e67e00700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e5406d7f0 0x7f8e5406fca0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8e7010f2a0 tx=0x7f8e58009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.356+0000 7f8e6cc00700 1 -- 172.21.15.1:0/3913123341 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8e60059fc0 con 0x7f8e70102f50 2024-02-28T16:55:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.600+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e7010f050 con 0x7f8e5406d7f0 2024-02-28T16:55:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.602+0000 7f8e6cc00700 1 -- 172.21.15.1:0/3913123341 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8e7010f050 con 0x7f8e5406d7f0 2024-02-28T16:55:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e5406d7f0 msgr2=0x7f8e5406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e5406d7f0 0x7f8e5406fca0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8e7010f2a0 tx=0x7f8e58009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e70102f50 msgr2=0x7f8e70113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e70102f50 0x7f8e70113a70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8e60002a50 tx=0x7f8e600046b0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 shutdown_connections 2024-02-28T16:55:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e5406d7f0 0x7f8e5406fca0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e70102f50 0x7f8e70113a70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 --2- 172.21.15.1:0/3913123341 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e701038f0 0x7f8e70113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 >> 172.21.15.1:0/3913123341 conn(0x7f8e700f8ae0 msgr2=0x7f8e700fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 shutdown_connections 2024-02-28T16:55:47.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.605+0000 7f8e77200700 1 -- 172.21.15.1:0/3913123341 wait complete. 2024-02-28T16:55:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.750+0000 7f28cc600700 1 -- 172.21.15.1:0/2319187129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 msgr2=0x7f28c4070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.750+0000 7f28cc600700 1 --2- 172.21.15.1:0/2319187129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 0x7f28c4070700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f28b8004970 tx=0x7f28b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.750+0000 7f28cc600700 1 -- 172.21.15.1:0/2319187129 shutdown_connections 2024-02-28T16:55:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.750+0000 7f28cc600700 1 --2- 172.21.15.1:0/2319187129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28c4070c40 0x7f28c4109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.751+0000 7f28cc600700 1 --2- 172.21.15.1:0/2319187129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 0x7f28c4070700 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.751+0000 7f28cc600700 1 -- 172.21.15.1:0/2319187129 >> 172.21.15.1:0/2319187129 conn(0x7f28c40f8ae0 msgr2=0x7f28c40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.751+0000 7f28cc600700 1 -- 172.21.15.1:0/2319187129 shutdown_connections 2024-02-28T16:55:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.751+0000 7f28cc600700 1 -- 172.21.15.1:0/2319187129 wait complete. 2024-02-28T16:55:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.751+0000 7f28cc600700 1 Processor -- start 2024-02-28T16:55:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.752+0000 7f28cc600700 1 -- start start 2024-02-28T16:55:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.752+0000 7f28cc600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 0x7f28c4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.752+0000 7f28cc600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28c4070c40 0x7f28c4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.752+0000 7f28cc600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c4116790 con 0x7f28c4070330 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.752+0000 7f28cc600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c410fae0 con 0x7f28c4070c40 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.752+0000 7f28c9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 0x7f28c4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.752+0000 7f28c9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 0x7f28c4115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44212/0 (socket says 172.21.15.1:44212) 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.752+0000 7f28c9c00700 1 -- 172.21.15.1:0/558656228 learned_addr learned my addr 172.21.15.1:0/558656228 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28c9c00700 1 -- 172.21.15.1:0/558656228 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28c4070c40 msgr2=0x7f28c4116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28c9200700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28c4070c40 0x7f28c4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28c9c00700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28c4070c40 0x7f28c4116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28c9c00700 1 -- 172.21.15.1:0/558656228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8005040 con 0x7f28c4070330 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28c9c00700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 0x7f28c4115c00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f28b8004f50 tx=0x7f28b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28c1600700 1 -- 172.21.15.1:0/558656228 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28b800e040 con 0x7f28c4070330 2024-02-28T16:55:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28c1600700 1 -- 172.21.15.1:0/558656228 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28b8004290 con 0x7f28c4070330 2024-02-28T16:55:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28c1600700 1 -- 172.21.15.1:0/558656228 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28b801c630 con 0x7f28c4070330 2024-02-28T16:55:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c410fd60 con 0x7f28c4070330 2024-02-28T16:55:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.753+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c41102a0 con 0x7f28c4070330 2024-02-28T16:55:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.754+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c4106920 con 0x7f28c4070330 2024-02-28T16:55:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.755+0000 7f28c1600700 1 -- 172.21.15.1:0/558656228 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f28b800bc10 con 0x7f28c4070330 2024-02-28T16:55:47.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.756+0000 7f28c1600700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28ac06d7f0 0x7f28ac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:47.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.756+0000 7f28c1600700 1 -- 172.21.15.1:0/558656228 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f28b808e1c0 con 0x7f28c4070330 2024-02-28T16:55:47.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.757+0000 7f28c9200700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28ac06d7f0 0x7f28ac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:47.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.757+0000 7f28c9200700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28ac06d7f0 0x7f28ac06fca0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f28c41114a0 tx=0x7f28b000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:47.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.761+0000 7f28c1600700 1 -- 172.21.15.1:0/558656228 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f28b8057b30 con 0x7f28c4070330 2024-02-28T16:55:47.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:47.985+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f28c404bed0 con 0x7f28ac06d7f0 2024-02-28T16:55:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.023+0000 7f28c1600700 1 -- 172.21.15.1:0/558656228 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f28c404bed0 con 0x7f28ac06d7f0 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (21m) 13m ago 36m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (18m) 13m ago 36m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (18m) 14m ago 35m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (21m) 13m ago 35m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (14m) 13m ago 29m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (14m) 13m ago 28m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (28m) 14m ago 28m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (29m) 14m ago 29m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (24m) 13m ago 37m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:55:48.021 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (23m) 14m ago 34m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (20m) 13m ago 38m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (19m) 14m ago 34m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (22m) 13m ago 36m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (22m) 14m ago 34m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (17m) 13m ago 33m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (17m) 13m ago 33m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (16m) 13m ago 32m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (16m) 14m ago 32m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (15m) 14m ago 31m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (14m) 14m ago 30m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:55:48.022 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (22m) 13m ago 35m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:55:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.025+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28ac06d7f0 msgr2=0x7f28ac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28ac06d7f0 0x7f28ac06fca0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f28c41114a0 tx=0x7f28b000b040 comp rx=0 tx=0).stop 2024-02-28T16:55:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 msgr2=0x7f28c4115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 0x7f28c4115c00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f28b8004f50 tx=0x7f28b8009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 shutdown_connections 2024-02-28T16:55:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28ac06d7f0 0x7f28ac06fca0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c4070330 0x7f28c4115c00 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 --2- 172.21.15.1:0/558656228 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28c4070c40 0x7f28c4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 >> 172.21.15.1:0/558656228 conn(0x7f28c40f8ae0 msgr2=0x7f28c4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 shutdown_connections 2024-02-28T16:55:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.026+0000 7f28cc600700 1 -- 172.21.15.1:0/558656228 wait complete. 2024-02-28T16:55:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.179+0000 7f611c200700 1 -- 172.21.15.1:0/438423369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61141038e0 msgr2=0x7f61140fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.179+0000 7f611c200700 1 --2- 172.21.15.1:0/438423369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61141038e0 0x7f61140fcf50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6100004940 tx=0x7f6100009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.179+0000 7f611c200700 1 -- 172.21.15.1:0/438423369 shutdown_connections 2024-02-28T16:55:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.179+0000 7f611c200700 1 --2- 172.21.15.1:0/438423369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61141038e0 0x7f61140fcf50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.179+0000 7f611c200700 1 --2- 172.21.15.1:0/438423369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6114102f40 0x7f6114103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.179+0000 7f611c200700 1 -- 172.21.15.1:0/438423369 >> 172.21.15.1:0/438423369 conn(0x7f61140f8af0 msgr2=0x7f61140faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.179+0000 7f611c200700 1 -- 172.21.15.1:0/438423369 shutdown_connections 2024-02-28T16:55:48.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.179+0000 7f611c200700 1 -- 172.21.15.1:0/438423369 wait complete. 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.180+0000 7f611c200700 1 Processor -- start 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.180+0000 7f611c200700 1 -- start start 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.180+0000 7f611c200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114102f40 0x7f6114113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.180+0000 7f611c200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61141038e0 0x7f6114113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f611c200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6114114680 con 0x7f6114102f40 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f611c200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611410d920 con 0x7f61141038e0 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f6119800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114102f40 0x7f6114113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f6119800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114102f40 0x7f6114113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44236/0 (socket says 172.21.15.1:44236) 2024-02-28T16:55:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f6119800700 1 -- 172.21.15.1:0/3129839328 learned_addr learned my addr 172.21.15.1:0/3129839328 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f6118e00700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61141038e0 0x7f6114113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f6119800700 1 -- 172.21.15.1:0/3129839328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61141038e0 msgr2=0x7f6114113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f6119800700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61141038e0 0x7f6114113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.181+0000 7f6119800700 1 -- 172.21.15.1:0/3129839328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6100005040 con 0x7f6114102f40 2024-02-28T16:55:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.182+0000 7f6119800700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114102f40 0x7f6114113a60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f610800b790 tx=0x7f6108009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.182+0000 7f6111600700 1 -- 172.21.15.1:0/3129839328 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610800f770 con 0x7f6114102f40 2024-02-28T16:55:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.182+0000 7f6111600700 1 -- 172.21.15.1:0/3129839328 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f610800fda0 con 0x7f6114102f40 2024-02-28T16:55:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.182+0000 7f6111600700 1 -- 172.21.15.1:0/3129839328 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610800d310 con 0x7f6114102f40 2024-02-28T16:55:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.182+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611410dc00 con 0x7f6114102f40 2024-02-28T16:55:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.182+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611410e140 con 0x7f6114102f40 2024-02-28T16:55:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.183+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61140fc6f0 con 0x7f6114102f40 2024-02-28T16:55:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.184+0000 7f6111600700 1 -- 172.21.15.1:0/3129839328 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6108010420 con 0x7f6114102f40 2024-02-28T16:55:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.185+0000 7f6111600700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60fc06d8a0 0x7f60fc06fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.185+0000 7f6111600700 1 -- 172.21.15.1:0/3129839328 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f610808ce70 con 0x7f6114102f40 2024-02-28T16:55:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.186+0000 7f6118e00700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60fc06d8a0 0x7f60fc06fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.186+0000 7f6118e00700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60fc06d8a0 0x7f60fc06fd50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f611410f290 tx=0x7f6100009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:48.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.190+0000 7f6111600700 1 -- 172.21.15.1:0/3129839328 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f610805a250 con 0x7f6114102f40 2024-02-28T16:55:48.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.493+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f611410a110 con 0x7f6114102f40 2024-02-28T16:55:48.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.494+0000 7f6111600700 1 -- 172.21.15.1:0/3129839328 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f611410a110 con 0x7f6114102f40 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:55:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:55:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:55:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:55:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:55:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:55:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:55:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:55:48.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:55:48.493 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:55:48.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60fc06d8a0 msgr2=0x7f60fc06fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60fc06d8a0 0x7f60fc06fd50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f611410f290 tx=0x7f6100009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114102f40 msgr2=0x7f6114113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114102f40 0x7f6114113a60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f610800b790 tx=0x7f6108009250 comp rx=0 tx=0).stop 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 shutdown_connections 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60fc06d8a0 0x7f60fc06fd50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114102f40 0x7f6114113a60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 --2- 172.21.15.1:0/3129839328 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f61141038e0 0x7f6114113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.497+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 >> 172.21.15.1:0/3129839328 conn(0x7f61140f8af0 msgr2=0x7f61140fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.498+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 shutdown_connections 2024-02-28T16:55:48.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.498+0000 7f611c200700 1 -- 172.21.15.1:0/3129839328 wait complete. 2024-02-28T16:55:48.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.648+0000 7fcf14000700 1 -- 172.21.15.1:0/1007544337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c1095f0 msgr2=0x7fcf0c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.648+0000 7fcf14000700 1 --2- 172.21.15.1:0/1007544337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c1095f0 0x7fcf0c1099c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fcf00004940 tx=0x7fcf00009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.648+0000 7fcf14000700 1 -- 172.21.15.1:0/1007544337 shutdown_connections 2024-02-28T16:55:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.648+0000 7fcf14000700 1 --2- 172.21.15.1:0/1007544337 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf0c103dc0 0x7fcf0c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.648+0000 7fcf14000700 1 --2- 172.21.15.1:0/1007544337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c1095f0 0x7fcf0c1099c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.648+0000 7fcf14000700 1 -- 172.21.15.1:0/1007544337 >> 172.21.15.1:0/1007544337 conn(0x7fcf0c072d50 msgr2=0x7fcf0c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.648+0000 7fcf14000700 1 -- 172.21.15.1:0/1007544337 shutdown_connections 2024-02-28T16:55:48.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.649+0000 7fcf14000700 1 -- 172.21.15.1:0/1007544337 wait complete. 2024-02-28T16:55:48.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.649+0000 7fcf14000700 1 Processor -- start 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.649+0000 7fcf14000700 1 -- start start 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.650+0000 7fcf14000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c103dc0 0x7fcf0c11a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.650+0000 7fcf14000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf0c1095f0 0x7fcf0c11a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.650+0000 7fcf14000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c11ab90 con 0x7fcf0c103dc0 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.650+0000 7fcf14000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c113ee0 con 0x7fcf0c1095f0 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.650+0000 7fcf11600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c103dc0 0x7fcf0c11a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.650+0000 7fcf11600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c103dc0 0x7fcf0c11a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44242/0 (socket says 172.21.15.1:44242) 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.650+0000 7fcf11600700 1 -- 172.21.15.1:0/3577908032 learned_addr learned my addr 172.21.15.1:0/3577908032 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.650+0000 7fcf10c00700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf0c1095f0 0x7fcf0c11a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf11600700 1 -- 172.21.15.1:0/3577908032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf0c1095f0 msgr2=0x7fcf0c11a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf11600700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf0c1095f0 0x7fcf0c11a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf11600700 1 -- 172.21.15.1:0/3577908032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00005040 con 0x7fcf0c103dc0 2024-02-28T16:55:48.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf11600700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c103dc0 0x7fcf0c11a000 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fcf00000c00 tx=0x7fcf00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:48.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf09600700 1 -- 172.21.15.1:0/3577908032 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf00021070 con 0x7fcf0c103dc0 2024-02-28T16:55:48.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf09600700 1 -- 172.21.15.1:0/3577908032 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf00013dd0 con 0x7fcf0c103dc0 2024-02-28T16:55:48.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf09600700 1 -- 172.21.15.1:0/3577908032 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf0000e070 con 0x7fcf0c103dc0 2024-02-28T16:55:48.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c114160 con 0x7fcf0c103dc0 2024-02-28T16:55:48.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.651+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c114640 con 0x7fcf0c103dc0 2024-02-28T16:55:48.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.652+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0c075870 con 0x7fcf0c103dc0 2024-02-28T16:55:48.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.653+0000 7fcf09600700 1 -- 172.21.15.1:0/3577908032 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcf0001c400 con 0x7fcf0c103dc0 2024-02-28T16:55:48.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.654+0000 7fcf09600700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcef406d7f0 0x7fcef406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:48.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.654+0000 7fcf09600700 1 -- 172.21.15.1:0/3577908032 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcf0008e380 con 0x7fcf0c103dc0 2024-02-28T16:55:48.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.654+0000 7fcf10c00700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcef406d7f0 0x7fcef406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:48.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.655+0000 7fcf10c00700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcef406d7f0 0x7fcef406fca0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcef8004600 tx=0x7fcef800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.659+0000 7fcf09600700 1 -- 172.21.15.1:0/3577908032 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf00057c30 con 0x7fcf0c103dc0 2024-02-28T16:55:48.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:48 smithi001 ceph-mon[112940]: pgmap v640: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:48.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:48 smithi001 ceph-mon[112940]: from='client.35174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:48.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:48 smithi001 ceph-mon[112940]: from='client.35178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:48.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:48 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3129839328' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:55:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.934+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcf0c1156c0 con 0x7fcf0c103dc0 2024-02-28T16:55:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.935+0000 7fcf09600700 1 -- 172.21.15.1:0/3577908032 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fcf0005b2f0 con 0x7fcf0c103dc0 2024-02-28T16:55:48.933 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:55:48.933 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:55:48.933 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:55:48.933 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:55:48.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:55:48.933 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:55:48.933 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:55:48.934 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:55:48.935 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:55:48.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:55:48.936 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:55:48.936 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:55:48.936 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:55:48.936 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.938+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcef406d7f0 msgr2=0x7fcef406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.938+0000 7fcf14000700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcef406d7f0 0x7fcef406fca0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcef8004600 tx=0x7fcef800b040 comp rx=0 tx=0).stop 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.938+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c103dc0 msgr2=0x7fcf0c11a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.938+0000 7fcf14000700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c103dc0 0x7fcf0c11a000 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fcf00000c00 tx=0x7fcf00009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.939+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 shutdown_connections 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.939+0000 7fcf14000700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcef406d7f0 0x7fcef406fca0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.939+0000 7fcf14000700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf0c103dc0 0x7fcf0c11a000 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.939+0000 7fcf14000700 1 --2- 172.21.15.1:0/3577908032 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf0c1095f0 0x7fcf0c11a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.939+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 >> 172.21.15.1:0/3577908032 conn(0x7fcf0c072d50 msgr2=0x7fcf0c0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.939+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 shutdown_connections 2024-02-28T16:55:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:48.939+0000 7fcf14000700 1 -- 172.21.15.1:0/3577908032 wait complete. 2024-02-28T16:55:48.938 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:55:49.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:48 smithi066 ceph-mon[85781]: pgmap v640: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:49.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:48 smithi066 ceph-mon[85781]: from='client.35174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:49.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:48 smithi066 ceph-mon[85781]: from='client.35178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:49.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:48 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3129839328' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.091+0000 7f36b0800700 1 -- 172.21.15.1:0/2583019097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 msgr2=0x7f36a81092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.091+0000 7f36b0800700 1 --2- 172.21.15.1:0/2583019097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 0x7f36a81092c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3698004970 tx=0x7f3698009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.091+0000 7f36b0800700 1 -- 172.21.15.1:0/2583019097 shutdown_connections 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.091+0000 7f36b0800700 1 --2- 172.21.15.1:0/2583019097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 0x7f36a81092c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.091+0000 7f36b0800700 1 --2- 172.21.15.1:0/2583019097 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36a80702d0 0x7f36a80706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.091+0000 7f36b0800700 1 -- 172.21.15.1:0/2583019097 >> 172.21.15.1:0/2583019097 conn(0x7f36a80f8ae0 msgr2=0x7f36a80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.091+0000 7f36b0800700 1 -- 172.21.15.1:0/2583019097 shutdown_connections 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.091+0000 7f36b0800700 1 -- 172.21.15.1:0/2583019097 wait complete. 2024-02-28T16:55:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.092+0000 7f36b0800700 1 Processor -- start 2024-02-28T16:55:49.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.092+0000 7f36b0800700 1 -- start start 2024-02-28T16:55:49.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.092+0000 7f36b0800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36a80702d0 0x7f36a8115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:49.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.092+0000 7f36b0800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 0x7f36a8116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:49.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.092+0000 7f36b0800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a81167a0 con 0x7f36a8070be0 2024-02-28T16:55:49.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.092+0000 7f36b0800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a810faf0 con 0x7f36a80702d0 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.093+0000 7f36ad400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 0x7f36a8116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.093+0000 7f36ad400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 0x7f36a8116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44248/0 (socket says 172.21.15.1:44248) 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.093+0000 7f36ad400700 1 -- 172.21.15.1:0/3201848413 learned_addr learned my addr 172.21.15.1:0/3201848413 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.093+0000 7f36ade00700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36a80702d0 0x7f36a8115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.093+0000 7f36ad400700 1 -- 172.21.15.1:0/3201848413 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36a80702d0 msgr2=0x7f36a8115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.093+0000 7f36ad400700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36a80702d0 0x7f36a8115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.093+0000 7f36ad400700 1 -- 172.21.15.1:0/3201848413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3698005040 con 0x7f36a8070be0 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.094+0000 7f36ad400700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 0x7f36a8116150 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3698004f50 tx=0x7f3698009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.094+0000 7f36a2000700 1 -- 172.21.15.1:0/3201848413 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f369800e040 con 0x7f36a8070be0 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.094+0000 7f36a2000700 1 -- 172.21.15.1:0/3201848413 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3698004290 con 0x7f36a8070be0 2024-02-28T16:55:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.094+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a810fd70 con 0x7f36a8070be0 2024-02-28T16:55:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.094+0000 7f36a2000700 1 -- 172.21.15.1:0/3201848413 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f369801c630 con 0x7f36a8070be0 2024-02-28T16:55:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.094+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a81102b0 con 0x7f36a8070be0 2024-02-28T16:55:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.095+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a8106980 con 0x7f36a8070be0 2024-02-28T16:55:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.096+0000 7f36a2000700 1 -- 172.21.15.1:0/3201848413 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f369800bc10 con 0x7f36a8070be0 2024-02-28T16:55:49.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.097+0000 7f36a2000700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f369006d7f0 0x7f369006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:49.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.097+0000 7f36a2000700 1 -- 172.21.15.1:0/3201848413 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f369808e280 con 0x7f36a8070be0 2024-02-28T16:55:49.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.097+0000 7f36ade00700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f369006d7f0 0x7f369006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:49.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.098+0000 7f36ade00700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f369006d7f0 0x7f369006fca0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f369c0041d0 tx=0x7f369c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:49.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.102+0000 7f36a2000700 1 -- 172.21.15.1:0/3201848413 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3698057b30 con 0x7f36a8070be0 2024-02-28T16:55:49.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.337+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36a804bed0 con 0x7f369006d7f0 2024-02-28T16:55:49.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.339+0000 7f36a2000700 1 -- 172.21.15.1:0/3201848413 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f36a804bed0 con 0x7f369006d7f0 2024-02-28T16:55:49.339 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:55:49.340 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:55:49.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f369006d7f0 msgr2=0x7f369006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:49.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f369006d7f0 0x7f369006fca0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f369c0041d0 tx=0x7f369c009040 comp rx=0 tx=0).stop 2024-02-28T16:55:49.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 msgr2=0x7f36a8116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:49.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 0x7f36a8116150 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3698004f50 tx=0x7f3698009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 shutdown_connections 2024-02-28T16:55:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f369006d7f0 0x7f369006fca0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f36a80702d0 0x7f36a8115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 --2- 172.21.15.1:0/3201848413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36a8070be0 0x7f36a8116150 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 >> 172.21.15.1:0/3201848413 conn(0x7f36a80f8ae0 msgr2=0x7f36a8103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 shutdown_connections 2024-02-28T16:55:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.342+0000 7f36b0800700 1 -- 172.21.15.1:0/3201848413 wait complete. 2024-02-28T16:55:49.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.487+0000 7f15b1c00700 1 -- 172.21.15.1:0/2614074260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109d20 msgr2=0x7f15ac1033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:49.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.487+0000 7f15b1c00700 1 --2- 172.21.15.1:0/2614074260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109d20 0x7f15ac1033a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1594004940 tx=0x7f1594009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:49.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.487+0000 7f15b1c00700 1 -- 172.21.15.1:0/2614074260 shutdown_connections 2024-02-28T16:55:49.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.487+0000 7f15b1c00700 1 --2- 172.21.15.1:0/2614074260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109d20 0x7f15ac1033a0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.487+0000 7f15b1c00700 1 --2- 172.21.15.1:0/2614074260 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15ac109380 0x7f15ac109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.487+0000 7f15b1c00700 1 -- 172.21.15.1:0/2614074260 >> 172.21.15.1:0/2614074260 conn(0x7f15ac072b50 msgr2=0x7f15ac074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:49.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.487+0000 7f15b1c00700 1 -- 172.21.15.1:0/2614074260 shutdown_connections 2024-02-28T16:55:49.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.487+0000 7f15b1c00700 1 -- 172.21.15.1:0/2614074260 wait complete. 2024-02-28T16:55:49.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.488+0000 7f15b1c00700 1 Processor -- start 2024-02-28T16:55:49.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.488+0000 7f15b1c00700 1 -- start start 2024-02-28T16:55:49.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.488+0000 7f15b1c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109380 0x7f15ac119db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.488+0000 7f15b1c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15ac109d20 0x7f15ac11a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.488+0000 7f15b1c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac11a940 con 0x7f15ac109380 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.488+0000 7f15b1c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac113c90 con 0x7f15ac109d20 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15abe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109380 0x7f15ac119db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15abe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109380 0x7f15ac119db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44264/0 (socket says 172.21.15.1:44264) 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15abe00700 1 -- 172.21.15.1:0/289831335 learned_addr learned my addr 172.21.15.1:0/289831335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15a3e00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15ac109d20 0x7f15ac11a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15abe00700 1 -- 172.21.15.1:0/289831335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15ac109d20 msgr2=0x7f15ac11a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15abe00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15ac109d20 0x7f15ac11a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15abe00700 1 -- 172.21.15.1:0/289831335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1594005040 con 0x7f15ac109380 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15a3e00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15ac109d20 0x7f15ac11a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:55:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.489+0000 7f15abe00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109380 0x7f15ac119db0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f159c00c830 tx=0x7f159c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:49.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.490+0000 7f15a8c00700 1 -- 172.21.15.1:0/289831335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f159c00a760 con 0x7f15ac109380 2024-02-28T16:55:49.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.490+0000 7f15a8c00700 1 -- 172.21.15.1:0/289831335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f159c00ad90 con 0x7f15ac109380 2024-02-28T16:55:49.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.490+0000 7f15a8c00700 1 -- 172.21.15.1:0/289831335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f159c0113a0 con 0x7f15ac109380 2024-02-28T16:55:49.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.490+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac113f70 con 0x7f15ac109380 2024-02-28T16:55:49.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.490+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac1144e0 con 0x7f15ac109380 2024-02-28T16:55:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.491+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac05f5c0 con 0x7f15ac109380 2024-02-28T16:55:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.492+0000 7f15a8c00700 1 -- 172.21.15.1:0/289831335 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f159c012400 con 0x7f15ac109380 2024-02-28T16:55:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.493+0000 7f15a8c00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f159006d7f0 0x7f159006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:55:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.493+0000 7f15a8c00700 1 -- 172.21.15.1:0/289831335 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f159c08cfa0 con 0x7f15ac109380 2024-02-28T16:55:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.493+0000 7f15a3e00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f159006d7f0 0x7f159006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:55:49.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.494+0000 7f15a3e00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f159006d7f0 0x7f159006fca0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f159400b3b0 tx=0x7f1594009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:55:49.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.498+0000 7f15a8c00700 1 -- 172.21.15.1:0/289831335 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f159c0567d0 con 0x7f15ac109380 2024-02-28T16:55:49.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.810+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f15ac115470 con 0x7f15ac109380 2024-02-28T16:55:49.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.810+0000 7f15a8c00700 1 -- 172.21.15.1:0/289831335 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f159c059e90 con 0x7f15ac109380 2024-02-28T16:55:49.808 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:55:49.808 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:55:49.808 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:55:49.808 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:55:49.808 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:55:49.808 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:55:49.808 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:55:49.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f159006d7f0 msgr2=0x7f159006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:49.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f159006d7f0 0x7f159006fca0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f159400b3b0 tx=0x7f1594009e30 comp rx=0 tx=0).stop 2024-02-28T16:55:49.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109380 msgr2=0x7f15ac119db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:55:49.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109380 0x7f15ac119db0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f159c00c830 tx=0x7f159c004470 comp rx=0 tx=0).stop 2024-02-28T16:55:49.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 shutdown_connections 2024-02-28T16:55:49.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f159006d7f0 0x7f159006fca0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac109380 0x7f15ac119db0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 --2- 172.21.15.1:0/289831335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f15ac109d20 0x7f15ac11a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:55:49.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 >> 172.21.15.1:0/289831335 conn(0x7f15ac072b50 msgr2=0x7f15ac075600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:55:49.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.813+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 shutdown_connections 2024-02-28T16:55:49.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:55:49.814+0000 7f15b1c00700 1 -- 172.21.15.1:0/289831335 wait complete. 2024-02-28T16:55:50.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:49 smithi066 ceph-mon[85781]: from='client.35182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:50.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:49 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3577908032' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:55:50.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:49 smithi001 ceph-mon[112940]: from='client.35182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:50.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:49 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3577908032' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:55:51.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:50 smithi066 ceph-mon[85781]: pgmap v641: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:51.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:50 smithi066 ceph-mon[85781]: from='client.35194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:51.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:50 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/289831335' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:55:51.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:50 smithi001 ceph-mon[112940]: pgmap v641: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:51.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:50 smithi001 ceph-mon[112940]: from='client.35194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:55:51.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:50 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/289831335' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:55:53.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:52 smithi066 ceph-mon[85781]: pgmap v642: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:53.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:52 smithi001 ceph-mon[112940]: pgmap v642: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:55.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:54 smithi066 ceph-mon[85781]: pgmap v643: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:55.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:54 smithi001 ceph-mon[112940]: pgmap v643: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:57.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:56 smithi066 ceph-mon[85781]: pgmap v644: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:57.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:55:57.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:56 smithi001 ceph-mon[112940]: pgmap v644: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:57.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:55:59.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:58 smithi066 ceph-mon[85781]: pgmap v645: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:55:59.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:58 smithi001 ceph-mon[112940]: pgmap v645: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:00.256 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:55:59 smithi066 ceph-mon[85781]: pgmap v646: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:00.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:55:59 smithi001 ceph-mon[112940]: pgmap v646: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:02.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:02 smithi066 ceph-mon[85781]: pgmap v647: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:02.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:02 smithi001 ceph-mon[112940]: pgmap v647: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:04.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:04 smithi066 ceph-mon[85781]: pgmap v648: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:04.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:04 smithi001 ceph-mon[112940]: pgmap v648: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:06.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:06 smithi066 ceph-mon[85781]: pgmap v649: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:06.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:06 smithi001 ceph-mon[112940]: pgmap v649: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:08.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:08 smithi066 ceph-mon[85781]: pgmap v650: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:08.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:08 smithi001 ceph-mon[112940]: pgmap v650: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:10.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:10 smithi066 ceph-mon[85781]: pgmap v651: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:10.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:10 smithi001 ceph-mon[112940]: pgmap v651: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:11.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:56:11.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:56:12.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:12 smithi066 ceph-mon[85781]: pgmap v652: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:12.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:12 smithi001 ceph-mon[112940]: pgmap v652: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:14.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:14 smithi066 ceph-mon[85781]: pgmap v653: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:14.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:14 smithi001 ceph-mon[112940]: pgmap v653: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:16.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:16 smithi066 ceph-mon[85781]: pgmap v654: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:16.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:16 smithi001 ceph-mon[112940]: pgmap v654: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:18.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:18 smithi066 ceph-mon[85781]: pgmap v655: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:18.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:18 smithi001 ceph-mon[112940]: pgmap v655: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.986+0000 7ff604c00700 1 -- 172.21.15.1:0/2475714700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 msgr2=0x7ff600103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.986+0000 7ff604c00700 1 --2- 172.21.15.1:0/2475714700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 0x7ff600103320 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0004940 tx=0x7ff5f0009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.986+0000 7ff604c00700 1 -- 172.21.15.1:0/2475714700 shutdown_connections 2024-02-28T16:56:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.986+0000 7ff604c00700 1 --2- 172.21.15.1:0/2475714700 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6001038f0 0x7ff6000fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.986+0000 7ff604c00700 1 --2- 172.21.15.1:0/2475714700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 0x7ff600103320 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.986+0000 7ff604c00700 1 -- 172.21.15.1:0/2475714700 >> 172.21.15.1:0/2475714700 conn(0x7ff6000f8ae0 msgr2=0x7ff6000faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.987+0000 7ff604c00700 1 -- 172.21.15.1:0/2475714700 shutdown_connections 2024-02-28T16:56:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.987+0000 7ff604c00700 1 -- 172.21.15.1:0/2475714700 wait complete. 2024-02-28T16:56:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.988+0000 7ff604c00700 1 Processor -- start 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.988+0000 7ff604c00700 1 -- start start 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.988+0000 7ff604c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 0x7ff60010a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.988+0000 7ff604c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6001038f0 0x7ff60010ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.988+0000 7ff604c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60010b390 con 0x7ff600102f50 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.988+0000 7ff604c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60010b4d0 con 0x7ff6001038f0 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.989+0000 7ff5fde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 0x7ff60010a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.989+0000 7ff5fde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 0x7ff60010a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55300/0 (socket says 172.21.15.1:55300) 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.989+0000 7ff5fde00700 1 -- 172.21.15.1:0/4039322199 learned_addr learned my addr 172.21.15.1:0/4039322199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.989+0000 7ff5fde00700 1 -- 172.21.15.1:0/4039322199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6001038f0 msgr2=0x7ff60010ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.989+0000 7ff5fde00700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6001038f0 0x7ff60010ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.989+0000 7ff5fde00700 1 -- 172.21.15.1:0/4039322199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f0005040 con 0x7ff600102f50 2024-02-28T16:56:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.989+0000 7ff5fde00700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 0x7ff60010a910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009fd0 tx=0x7ff5f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.990+0000 7ff5f6000700 1 -- 172.21.15.1:0/4039322199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5f000e040 con 0x7ff600102f50 2024-02-28T16:56:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.990+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60010b750 con 0x7ff600102f50 2024-02-28T16:56:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.990+0000 7ff5f6000700 1 -- 172.21.15.1:0/4039322199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5f000bbf0 con 0x7ff600102f50 2024-02-28T16:56:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.990+0000 7ff5f6000700 1 -- 172.21.15.1:0/4039322199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5f001c3f0 con 0x7ff600102f50 2024-02-28T16:56:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.990+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60011e580 con 0x7ff600102f50 2024-02-28T16:56:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.991+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6000fc6e0 con 0x7ff600102f50 2024-02-28T16:56:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.993+0000 7ff5f6000700 1 -- 172.21.15.1:0/4039322199 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff5f000bd60 con 0x7ff600102f50 2024-02-28T16:56:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.993+0000 7ff5f6000700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5e406d7f0 0x7ff5e406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.994+0000 7ff5f6000700 1 -- 172.21.15.1:0/4039322199 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff5f008e0d0 con 0x7ff600102f50 2024-02-28T16:56:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.994+0000 7ff5fd400700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5e406d7f0 0x7ff5e406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.995+0000 7ff5fd400700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5e406d7f0 0x7ff5e406fca0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff60010c310 tx=0x7ff5ec009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:19.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:19.999+0000 7ff5f6000700 1 -- 172.21.15.1:0/4039322199 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff5f0057a40 con 0x7ff600102f50 2024-02-28T16:56:20.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.247+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff600070c20 con 0x7ff5e406d7f0 2024-02-28T16:56:20.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.252+0000 7ff5f6000700 1 -- 172.21.15.1:0/4039322199 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff600070c20 con 0x7ff5e406d7f0 2024-02-28T16:56:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5e406d7f0 msgr2=0x7ff5e406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5e406d7f0 0x7ff5e406fca0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff60010c310 tx=0x7ff5ec009ce0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 msgr2=0x7ff60010a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 0x7ff60010a910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009fd0 tx=0x7ff5f0009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:20.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 shutdown_connections 2024-02-28T16:56:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5e406d7f0 0x7ff5e406fca0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff600102f50 0x7ff60010a910 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 --2- 172.21.15.1:0/4039322199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6001038f0 0x7ff60010ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 >> 172.21.15.1:0/4039322199 conn(0x7ff6000f8ae0 msgr2=0x7ff600070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 shutdown_connections 2024-02-28T16:56:20.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.255+0000 7ff604c00700 1 -- 172.21.15.1:0/4039322199 wait complete. 2024-02-28T16:56:20.269 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:56:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.411+0000 7fe3e1400700 1 -- 172.21.15.1:0/2391248360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105b10 msgr2=0x7fe3dc0ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.411+0000 7fe3e1400700 1 --2- 172.21.15.1:0/2391248360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105b10 0x7fe3dc0ff180 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4004940 tx=0x7fe3c4009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.411+0000 7fe3e1400700 1 -- 172.21.15.1:0/2391248360 shutdown_connections 2024-02-28T16:56:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.411+0000 7fe3e1400700 1 --2- 172.21.15.1:0/2391248360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105b10 0x7fe3dc0ff180 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.411+0000 7fe3e1400700 1 --2- 172.21.15.1:0/2391248360 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3dc105170 0x7fe3dc105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.411+0000 7fe3e1400700 1 -- 172.21.15.1:0/2391248360 >> 172.21.15.1:0/2391248360 conn(0x7fe3dc0face0 msgr2=0x7fe3dc0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.412+0000 7fe3e1400700 1 -- 172.21.15.1:0/2391248360 shutdown_connections 2024-02-28T16:56:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.412+0000 7fe3e1400700 1 -- 172.21.15.1:0/2391248360 wait complete. 2024-02-28T16:56:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.413+0000 7fe3e1400700 1 Processor -- start 2024-02-28T16:56:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.413+0000 7fe3e1400700 1 -- start start 2024-02-28T16:56:20.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.413+0000 7fe3e1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105170 0x7fe3dc115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:20.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.413+0000 7fe3e1400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3dc105b10 0x7fe3dc116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:20.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.413+0000 7fe3e1400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc116870 con 0x7fe3dc105170 2024-02-28T16:56:20.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.413+0000 7fe3e1400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc10fb80 con 0x7fe3dc105b10 2024-02-28T16:56:20.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.413+0000 7fe3da800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105170 0x7fe3dc115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:20.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.413+0000 7fe3da800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105170 0x7fe3dc115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55316/0 (socket says 172.21.15.1:55316) 2024-02-28T16:56:20.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.414+0000 7fe3da800700 1 -- 172.21.15.1:0/4139640441 learned_addr learned my addr 172.21.15.1:0/4139640441 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.414+0000 7fe3da800700 1 -- 172.21.15.1:0/4139640441 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3dc105b10 msgr2=0x7fe3dc116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.414+0000 7fe3da800700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3dc105b10 0x7fe3dc116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.414+0000 7fe3da800700 1 -- 172.21.15.1:0/4139640441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4005040 con 0x7fe3dc105170 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.414+0000 7fe3da800700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105170 0x7fe3dc115ce0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc002a50 tx=0x7fe3cc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.415+0000 7fe3d2a00700 1 -- 172.21.15.1:0/4139640441 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3cc007cc0 con 0x7fe3dc105170 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.415+0000 7fe3d2a00700 1 -- 172.21.15.1:0/4139640441 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3cc00f040 con 0x7fe3dc105170 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.415+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc10fe60 con 0x7fe3dc105170 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.415+0000 7fe3d2a00700 1 -- 172.21.15.1:0/4139640441 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3cc009640 con 0x7fe3dc105170 2024-02-28T16:56:20.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.415+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc1103a0 con 0x7fe3dc105170 2024-02-28T16:56:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.416+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc107640 con 0x7fe3dc105170 2024-02-28T16:56:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.417+0000 7fe3d2a00700 1 -- 172.21.15.1:0/4139640441 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe3cc0097a0 con 0x7fe3dc105170 2024-02-28T16:56:20.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.418+0000 7fe3d2a00700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe3c006d7f0 0x7fe3c006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:20.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.419+0000 7fe3d2a00700 1 -- 172.21.15.1:0/4139640441 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe3cc08cbc0 con 0x7fe3dc105170 2024-02-28T16:56:20.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.419+0000 7fe3d3e00700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe3c006d7f0 0x7fe3c006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:20.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.420+0000 7fe3d3e00700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe3c006d7f0 0x7fe3c006fca0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4004f90 tx=0x7fe3c4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.423+0000 7fe3d2a00700 1 -- 172.21.15.1:0/4139640441 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe3cc059fc0 con 0x7fe3dc105170 2024-02-28T16:56:20.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:20 smithi066 ceph-mon[85781]: pgmap v656: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:20.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:20 smithi001 ceph-mon[112940]: pgmap v656: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.662+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3dc1112c0 con 0x7fe3c006d7f0 2024-02-28T16:56:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.665+0000 7fe3d2a00700 1 -- 172.21.15.1:0/4139640441 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe3dc1112c0 con 0x7fe3c006d7f0 2024-02-28T16:56:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.667+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe3c006d7f0 msgr2=0x7fe3c006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.667+0000 7fe3e1400700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe3c006d7f0 0x7fe3c006fca0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4004f90 tx=0x7fe3c4009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.667+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105170 msgr2=0x7fe3dc115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.667+0000 7fe3e1400700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105170 0x7fe3dc115ce0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc002a50 tx=0x7fe3cc0046b0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.668+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 shutdown_connections 2024-02-28T16:56:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.668+0000 7fe3e1400700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe3c006d7f0 0x7fe3c006fca0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.668+0000 7fe3e1400700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3dc105170 0x7fe3dc115ce0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.668+0000 7fe3e1400700 1 --2- 172.21.15.1:0/4139640441 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe3dc105b10 0x7fe3dc116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.668+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 >> 172.21.15.1:0/4139640441 conn(0x7fe3dc0face0 msgr2=0x7fe3dc0fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.668+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 shutdown_connections 2024-02-28T16:56:20.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.668+0000 7fe3e1400700 1 -- 172.21.15.1:0/4139640441 wait complete. 2024-02-28T16:56:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.829+0000 7f2210000700 1 -- 172.21.15.1:0/1581221358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 msgr2=0x7f2208104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.829+0000 7f2210000700 1 --2- 172.21.15.1:0/1581221358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 0x7f2208104210 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f21f4004940 tx=0x7f21f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.829+0000 7f2210000700 1 -- 172.21.15.1:0/1581221358 shutdown_connections 2024-02-28T16:56:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.829+0000 7f2210000700 1 --2- 172.21.15.1:0/1581221358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 0x7f2208104210 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.829+0000 7f2210000700 1 --2- 172.21.15.1:0/1581221358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22081095f0 0x7f22081099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.829+0000 7f2210000700 1 -- 172.21.15.1:0/1581221358 >> 172.21.15.1:0/1581221358 conn(0x7f2208072d50 msgr2=0x7f2208075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.829+0000 7f2210000700 1 -- 172.21.15.1:0/1581221358 shutdown_connections 2024-02-28T16:56:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.829+0000 7f2210000700 1 -- 172.21.15.1:0/1581221358 wait complete. 2024-02-28T16:56:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.830+0000 7f2210000700 1 Processor -- start 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.830+0000 7f2210000700 1 -- start start 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f2210000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 0x7f220811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f2210000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22081095f0 0x7f220811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f2210000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220811abd0 con 0x7f2208103dc0 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f2210000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2208113ee0 con 0x7f22081095f0 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f220d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 0x7f220811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f220d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 0x7f220811a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55326/0 (socket says 172.21.15.1:55326) 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f220d600700 1 -- 172.21.15.1:0/2447328303 learned_addr learned my addr 172.21.15.1:0/2447328303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f220cc00700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22081095f0 0x7f220811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f220d600700 1 -- 172.21.15.1:0/2447328303 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22081095f0 msgr2=0x7f220811a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f220d600700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22081095f0 0x7f220811a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.831+0000 7f220d600700 1 -- 172.21.15.1:0/2447328303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f4005040 con 0x7f2208103dc0 2024-02-28T16:56:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.832+0000 7f220d600700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 0x7f220811a040 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c8c0 tx=0x7f21fc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.832+0000 7f2205600700 1 -- 172.21.15.1:0/2447328303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21fc00a980 con 0x7f2208103dc0 2024-02-28T16:56:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.832+0000 7f2205600700 1 -- 172.21.15.1:0/2447328303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21fc00ce60 con 0x7f2208103dc0 2024-02-28T16:56:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.832+0000 7f2205600700 1 -- 172.21.15.1:0/2447328303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21fc0115e0 con 0x7f2208103dc0 2024-02-28T16:56:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.832+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22081141c0 con 0x7f2208103dc0 2024-02-28T16:56:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.832+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2208114700 con 0x7f2208103dc0 2024-02-28T16:56:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.833+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220810b9c0 con 0x7f2208103dc0 2024-02-28T16:56:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.834+0000 7f2205600700 1 -- 172.21.15.1:0/2447328303 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f21fc012400 con 0x7f2208103dc0 2024-02-28T16:56:20.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.835+0000 7f2205600700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f21f006d7f0 0x7f21f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:20.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.835+0000 7f2205600700 1 -- 172.21.15.1:0/2447328303 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f21fc08cfa0 con 0x7f2208103dc0 2024-02-28T16:56:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.836+0000 7f220cc00700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f21f006d7f0 0x7f21f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:20.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.836+0000 7f220cc00700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f21f006d7f0 0x7f21f006fca0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f21f400b3b0 tx=0x7f21f4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:20.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:20.840+0000 7f2205600700 1 -- 172.21.15.1:0/2447328303 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f21fc05a3a0 con 0x7f2208103dc0 2024-02-28T16:56:21.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.074+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f220804bf40 con 0x7f21f006d7f0 2024-02-28T16:56:21.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.104+0000 7f2205600700 1 -- 172.21.15.1:0/2447328303 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f220804bf40 con 0x7f21f006d7f0 2024-02-28T16:56:21.101 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:56:21.101 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (22m) 14m ago 36m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:56:21.101 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (19m) 14m ago 37m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:56:21.101 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (19m) 15m ago 35m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:56:21.101 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (22m) 14m ago 36m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:56:21.101 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (15m) 14m ago 29m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (14m) 14m ago 29m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (29m) 15m ago 29m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (29m) 15m ago 29m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (24m) 14m ago 38m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (23m) 15m ago 35m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (20m) 14m ago 38m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (20m) 15m ago 35m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (23m) 14m ago 36m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (23m) 15m ago 35m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (18m) 14m ago 34m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (18m) 14m ago 33m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (17m) 14m ago 33m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (16m) 15m ago 32m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:56:21.102 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (16m) 15m ago 32m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:56:21.103 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (15m) 15m ago 31m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:56:21.103 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (22m) 14m ago 36m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:56:21.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f21f006d7f0 msgr2=0x7f21f006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:21.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f21f006d7f0 0x7f21f006fca0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f21f400b3b0 tx=0x7f21f4009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:21.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 msgr2=0x7f220811a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:21.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 0x7f220811a040 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c8c0 tx=0x7f21fc0044c0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 shutdown_connections 2024-02-28T16:56:21.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f21f006d7f0 0x7f21f006fca0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2208103dc0 0x7f220811a040 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 --2- 172.21.15.1:0/2447328303 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22081095f0 0x7f220811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 >> 172.21.15.1:0/2447328303 conn(0x7f2208072d50 msgr2=0x7f2208074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:21.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 shutdown_connections 2024-02-28T16:56:21.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.107+0000 7f2210000700 1 -- 172.21.15.1:0/2447328303 wait complete. 2024-02-28T16:56:21.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.252+0000 7f4c8ee00700 1 -- 172.21.15.1:0/3043688352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c881038e0 msgr2=0x7f4c880fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:21.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.252+0000 7f4c8ee00700 1 --2- 172.21.15.1:0/3043688352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c881038e0 0x7f4c880fcf50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4c74004940 tx=0x7f4c74009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:21.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.252+0000 7f4c8ee00700 1 -- 172.21.15.1:0/3043688352 shutdown_connections 2024-02-28T16:56:21.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.252+0000 7f4c8ee00700 1 --2- 172.21.15.1:0/3043688352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c881038e0 0x7f4c880fcf50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.252+0000 7f4c8ee00700 1 --2- 172.21.15.1:0/3043688352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c88102f40 0x7f4c88103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.252+0000 7f4c8ee00700 1 -- 172.21.15.1:0/3043688352 >> 172.21.15.1:0/3043688352 conn(0x7f4c880f8af0 msgr2=0x7f4c880faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:21.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.252+0000 7f4c8ee00700 1 -- 172.21.15.1:0/3043688352 shutdown_connections 2024-02-28T16:56:21.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.252+0000 7f4c8ee00700 1 -- 172.21.15.1:0/3043688352 wait complete. 2024-02-28T16:56:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.253+0000 7f4c8ee00700 1 Processor -- start 2024-02-28T16:56:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.253+0000 7f4c8ee00700 1 -- start start 2024-02-28T16:56:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.253+0000 7f4c8ee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c88102f40 0x7f4c88070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.253+0000 7f4c8ee00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c881038e0 0x7f4c880707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.253+0000 7f4c8ee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88070dd0 con 0x7f4c88102f40 2024-02-28T16:56:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.253+0000 7f4c8ee00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88070f40 con 0x7f4c881038e0 2024-02-28T16:56:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.254+0000 7f4c87e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c88102f40 0x7f4c88070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:21.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.254+0000 7f4c87e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c88102f40 0x7f4c88070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55342/0 (socket says 172.21.15.1:55342) 2024-02-28T16:56:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.254+0000 7f4c87e00700 1 -- 172.21.15.1:0/1304411081 learned_addr learned my addr 172.21.15.1:0/1304411081 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.254+0000 7f4c87400700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c881038e0 0x7f4c880707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.254+0000 7f4c87e00700 1 -- 172.21.15.1:0/1304411081 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c881038e0 msgr2=0x7f4c880707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.254+0000 7f4c87e00700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c881038e0 0x7f4c880707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.254+0000 7f4c87e00700 1 -- 172.21.15.1:0/1304411081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c74005040 con 0x7f4c88102f40 2024-02-28T16:56:21.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.255+0000 7f4c87e00700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c88102f40 0x7f4c88070290 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800ec90 tx=0x7f4c7800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:21.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.257+0000 7f4c7fe00700 1 -- 172.21.15.1:0/1304411081 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c7800eea0 con 0x7f4c88102f40 2024-02-28T16:56:21.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.257+0000 7f4c7fe00700 1 -- 172.21.15.1:0/1304411081 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c7800f040 con 0x7f4c88102f40 2024-02-28T16:56:21.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.257+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c88119c30 con 0x7f4c88102f40 2024-02-28T16:56:21.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.257+0000 7f4c7fe00700 1 -- 172.21.15.1:0/1304411081 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c78009640 con 0x7f4c88102f40 2024-02-28T16:56:21.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.257+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8811a0e0 con 0x7f4c88102f40 2024-02-28T16:56:21.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.258+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c88105410 con 0x7f4c88102f40 2024-02-28T16:56:21.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.259+0000 7f4c7fe00700 1 -- 172.21.15.1:0/1304411081 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4c78007b30 con 0x7f4c88102f40 2024-02-28T16:56:21.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.260+0000 7f4c7fe00700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c6c06d7e0 0x7f4c6c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:21.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.260+0000 7f4c7fe00700 1 -- 172.21.15.1:0/1304411081 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4c7808d450 con 0x7f4c88102f40 2024-02-28T16:56:21.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.260+0000 7f4c87400700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c6c06d7e0 0x7f4c6c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:21.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.261+0000 7f4c87400700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c6c06d7e0 0x7f4c6c06fc90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009b30 tx=0x7f4c74009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:21.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.265+0000 7f4c7fe00700 1 -- 172.21.15.1:0/1304411081 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c78056c80 con 0x7f4c88102f40 2024-02-28T16:56:21.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:21 smithi066 ceph-mon[85781]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:21.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:21 smithi066 ceph-mon[85781]: from='client.35206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:21.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:21 smithi001 ceph-mon[112940]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:21.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:21 smithi001 ceph-mon[112940]: from='client.35206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:21.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.564+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4c880719d0 con 0x7f4c88102f40 2024-02-28T16:56:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.566+0000 7f4c7fe00700 1 -- 172.21.15.1:0/1304411081 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f4c7805a340 con 0x7f4c88102f40 2024-02-28T16:56:21.562 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:56:21.562 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:56:21.562 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:56:21.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 2, 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:56:21.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:56:21.564 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:56:21.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.568+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c6c06d7e0 msgr2=0x7f4c6c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:21.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.568+0000 7f4c8ee00700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c6c06d7e0 0x7f4c6c06fc90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009b30 tx=0x7f4c74009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:21.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.568+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c88102f40 msgr2=0x7f4c88070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:21.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.568+0000 7f4c8ee00700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c88102f40 0x7f4c88070290 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800ec90 tx=0x7f4c7800b040 comp rx=0 tx=0).stop 2024-02-28T16:56:21.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.568+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 shutdown_connections 2024-02-28T16:56:21.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.568+0000 7f4c8ee00700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c6c06d7e0 0x7f4c6c06fc90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.569+0000 7f4c8ee00700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c88102f40 0x7f4c88070290 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.569+0000 7f4c8ee00700 1 --2- 172.21.15.1:0/1304411081 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c881038e0 0x7f4c880707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.569+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 >> 172.21.15.1:0/1304411081 conn(0x7f4c880f8af0 msgr2=0x7f4c880fa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:21.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.569+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 shutdown_connections 2024-02-28T16:56:21.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.569+0000 7f4c8ee00700 1 -- 172.21.15.1:0/1304411081 wait complete. 2024-02-28T16:56:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.713+0000 7f4892400700 1 -- 172.21.15.1:0/563304431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fe8e0 msgr2=0x7f488c106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.713+0000 7f4892400700 1 --2- 172.21.15.1:0/563304431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fe8e0 0x7f488c106f20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4878004940 tx=0x7f4878009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.713+0000 7f4892400700 1 -- 172.21.15.1:0/563304431 shutdown_connections 2024-02-28T16:56:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.713+0000 7f4892400700 1 --2- 172.21.15.1:0/563304431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fe8e0 0x7f488c106f20 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.713+0000 7f4892400700 1 --2- 172.21.15.1:0/563304431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f488c0fdf40 0x7f488c0fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.713+0000 7f4892400700 1 -- 172.21.15.1:0/563304431 >> 172.21.15.1:0/563304431 conn(0x7f488c0f79d0 msgr2=0x7f488c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.713+0000 7f4892400700 1 -- 172.21.15.1:0/563304431 shutdown_connections 2024-02-28T16:56:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.713+0000 7f4892400700 1 -- 172.21.15.1:0/563304431 wait complete. 2024-02-28T16:56:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.714+0000 7f4892400700 1 Processor -- start 2024-02-28T16:56:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.715+0000 7f4892400700 1 -- start start 2024-02-28T16:56:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.715+0000 7f4892400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fdf40 0x7f488c115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.715+0000 7f4892400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f488c0fe8e0 0x7f488c116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.715+0000 7f4892400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c116780 con 0x7f488c0fdf40 2024-02-28T16:56:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.715+0000 7f4892400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c10fab0 con 0x7f488c0fe8e0 2024-02-28T16:56:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.715+0000 7f488b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fdf40 0x7f488c115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.716+0000 7f488b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fdf40 0x7f488c115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55358/0 (socket says 172.21.15.1:55358) 2024-02-28T16:56:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.716+0000 7f488b400700 1 -- 172.21.15.1:0/3271888141 learned_addr learned my addr 172.21.15.1:0/3271888141 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.716+0000 7f488aa00700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f488c0fe8e0 0x7f488c116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.716+0000 7f488b400700 1 -- 172.21.15.1:0/3271888141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f488c0fe8e0 msgr2=0x7f488c116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.716+0000 7f488b400700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f488c0fe8e0 0x7f488c116130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.716+0000 7f488b400700 1 -- 172.21.15.1:0/3271888141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878005040 con 0x7f488c0fdf40 2024-02-28T16:56:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.716+0000 7f488aa00700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f488c0fe8e0 0x7f488c116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:56:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.716+0000 7f488b400700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fdf40 0x7f488c115bf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f487c00c8c0 tx=0x7f487c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.717+0000 7f4883400700 1 -- 172.21.15.1:0/3271888141 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487c00a980 con 0x7f488c0fdf40 2024-02-28T16:56:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.717+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c10fd90 con 0x7f488c0fdf40 2024-02-28T16:56:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.717+0000 7f4883400700 1 -- 172.21.15.1:0/3271888141 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f487c00ce60 con 0x7f488c0fdf40 2024-02-28T16:56:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.717+0000 7f4883400700 1 -- 172.21.15.1:0/3271888141 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487c0115e0 con 0x7f488c0fdf40 2024-02-28T16:56:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.717+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c1102d0 con 0x7f488c0fdf40 2024-02-28T16:56:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.718+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c106990 con 0x7f488c0fdf40 2024-02-28T16:56:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.719+0000 7f4883400700 1 -- 172.21.15.1:0/3271888141 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f487c012400 con 0x7f488c0fdf40 2024-02-28T16:56:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.720+0000 7f4883400700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f487006d7e0 0x7f487006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.720+0000 7f4883400700 1 -- 172.21.15.1:0/3271888141 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f487c08cfe0 con 0x7f488c0fdf40 2024-02-28T16:56:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.720+0000 7f488aa00700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f487006d7e0 0x7f487006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.721+0000 7f488aa00700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f487006d7e0 0x7f487006fc90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4878004560 tx=0x7f4878009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.725+0000 7f4883400700 1 -- 172.21.15.1:0/3271888141 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f487c05a3a0 con 0x7f488c0fdf40 2024-02-28T16:56:21.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:21.999+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f488c04bed0 con 0x7f488c0fdf40 2024-02-28T16:56:21.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.001+0000 7f4883400700 1 -- 172.21.15.1:0/3271888141 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f487c0126b0 con 0x7f488c0fdf40 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout:e24 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:56:21.999 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:56:22.000 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:56:22.001 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:56:22.002 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/1243103917,v1:172.21.15.66:6825/1243103917] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:56:22.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f487006d7e0 msgr2=0x7f487006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f487006d7e0 0x7f487006fc90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4878004560 tx=0x7f4878009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:22.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fdf40 msgr2=0x7f488c115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fdf40 0x7f488c115bf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f487c00c8c0 tx=0x7f487c0044c0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 shutdown_connections 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f487006d7e0 0x7f487006fc90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f488c0fdf40 0x7f488c115bf0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 --2- 172.21.15.1:0/3271888141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f488c0fe8e0 0x7f488c116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.004+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 >> 172.21.15.1:0/3271888141 conn(0x7f488c0f79d0 msgr2=0x7f488c0fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.005+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 shutdown_connections 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.005+0000 7f4892400700 1 -- 172.21.15.1:0/3271888141 wait complete. 2024-02-28T16:56:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T16:56:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.162+0000 7f746d800700 1 -- 172.21.15.1:0/3680179672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 msgr2=0x7f74680fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.162+0000 7f746d800700 1 --2- 172.21.15.1:0/3680179672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 0x7f74680fcf60 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7450004940 tx=0x7f7450009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.163+0000 7f746d800700 1 -- 172.21.15.1:0/3680179672 shutdown_connections 2024-02-28T16:56:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.163+0000 7f746d800700 1 --2- 172.21.15.1:0/3680179672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 0x7f74680fcf60 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.163+0000 7f746d800700 1 --2- 172.21.15.1:0/3680179672 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7468102f50 0x7f7468103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.163+0000 7f746d800700 1 -- 172.21.15.1:0/3680179672 >> 172.21.15.1:0/3680179672 conn(0x7f74680f8ae0 msgr2=0x7f74680faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.163+0000 7f746d800700 1 -- 172.21.15.1:0/3680179672 shutdown_connections 2024-02-28T16:56:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.163+0000 7f746d800700 1 -- 172.21.15.1:0/3680179672 wait complete. 2024-02-28T16:56:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.164+0000 7f746d800700 1 Processor -- start 2024-02-28T16:56:22.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.164+0000 7f746d800700 1 -- start start 2024-02-28T16:56:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.165+0000 7f746d800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7468102f50 0x7f7468113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.165+0000 7f746d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 0x7f7468113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.165+0000 7f746d800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7468114600 con 0x7f74681038f0 2024-02-28T16:56:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.165+0000 7f746d800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746810d930 con 0x7f7468102f50 2024-02-28T16:56:22.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.165+0000 7f745e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 0x7f7468113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:22.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.165+0000 7f745e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 0x7f7468113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42586/0 (socket says 172.21.15.1:42586) 2024-02-28T16:56:22.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.165+0000 7f745e000700 1 -- 172.21.15.1:0/3238264580 learned_addr learned my addr 172.21.15.1:0/3238264580 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:22.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f7466a00700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7468102f50 0x7f7468113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:22.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f745e000700 1 -- 172.21.15.1:0/3238264580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7468102f50 msgr2=0x7f7468113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f745e000700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7468102f50 0x7f7468113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f745e000700 1 -- 172.21.15.1:0/3238264580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7450005040 con 0x7f74681038f0 2024-02-28T16:56:22.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f745e000700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 0x7f7468113fb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7450009030 tx=0x7f7450009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:22.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f745f400700 1 -- 172.21.15.1:0/3238264580 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f745000e040 con 0x7f74681038f0 2024-02-28T16:56:22.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f745f400700 1 -- 172.21.15.1:0/3238264580 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f745000bbf0 con 0x7f74681038f0 2024-02-28T16:56:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f745f400700 1 -- 172.21.15.1:0/3238264580 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f745001c630 con 0x7f74681038f0 2024-02-28T16:56:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.166+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746810dbb0 con 0x7f74681038f0 2024-02-28T16:56:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.167+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746810e0f0 con 0x7f74681038f0 2024-02-28T16:56:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.168+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7468105420 con 0x7f74681038f0 2024-02-28T16:56:22.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.169+0000 7f745f400700 1 -- 172.21.15.1:0/3238264580 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f745000bd60 con 0x7f74681038f0 2024-02-28T16:56:22.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.170+0000 7f745f400700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f744c06d7f0 0x7f744c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:22.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.170+0000 7f745f400700 1 -- 172.21.15.1:0/3238264580 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f745008e130 con 0x7f74681038f0 2024-02-28T16:56:22.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.170+0000 7f7466a00700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f744c06d7f0 0x7f744c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:22.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.171+0000 7f7466a00700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f744c06d7f0 0x7f744c06fca0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7458004fa0 tx=0x7f745800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:22.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.175+0000 7f745f400700 1 -- 172.21.15.1:0/3238264580 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f745005b5d0 con 0x7f74681038f0 2024-02-28T16:56:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.417+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7468113740 con 0x7f744c06d7f0 2024-02-28T16:56:22.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.420+0000 7f745f400700 1 -- 172.21.15.1:0/3238264580 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7468113740 con 0x7f744c06d7f0 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:56:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:56:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:56:22.418 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f744c06d7f0 msgr2=0x7f744c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f744c06d7f0 0x7f744c06fca0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7458004fa0 tx=0x7f745800b040 comp rx=0 tx=0).stop 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 msgr2=0x7f7468113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 0x7f7468113fb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7450009030 tx=0x7f7450009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 shutdown_connections 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f744c06d7f0 0x7f744c06fca0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7468102f50 0x7f7468113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 --2- 172.21.15.1:0/3238264580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74681038f0 0x7f7468113fb0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.423+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 >> 172.21.15.1:0/3238264580 conn(0x7f74680f8ae0 msgr2=0x7f74680fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.424+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 shutdown_connections 2024-02-28T16:56:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.424+0000 7f746d800700 1 -- 172.21.15.1:0/3238264580 wait complete. 2024-02-28T16:56:22.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:22 smithi066 ceph-mon[85781]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:22.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:22 smithi066 ceph-mon[85781]: pgmap v657: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:22.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:22 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1304411081' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:22.506 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:22 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3271888141' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:56:22.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:22 smithi001 ceph-mon[112940]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:22.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:22 smithi001 ceph-mon[112940]: pgmap v657: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:22.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:22 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1304411081' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:22.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:22 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3271888141' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:56:22.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.581+0000 7f51aae00700 1 -- 172.21.15.1:0/3916972234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 msgr2=0x7f51a4105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.581+0000 7f51aae00700 1 --2- 172.21.15.1:0/3916972234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 0x7f51a4105540 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5194004940 tx=0x7f5194009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:22.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.581+0000 7f51aae00700 1 -- 172.21.15.1:0/3916972234 shutdown_connections 2024-02-28T16:56:22.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.581+0000 7f51aae00700 1 --2- 172.21.15.1:0/3916972234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51a4105b10 0x7f51a40ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.581+0000 7f51aae00700 1 --2- 172.21.15.1:0/3916972234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 0x7f51a4105540 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.581+0000 7f51aae00700 1 -- 172.21.15.1:0/3916972234 >> 172.21.15.1:0/3916972234 conn(0x7f51a40face0 msgr2=0x7f51a40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:22.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.581+0000 7f51aae00700 1 -- 172.21.15.1:0/3916972234 shutdown_connections 2024-02-28T16:56:22.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.581+0000 7f51aae00700 1 -- 172.21.15.1:0/3916972234 wait complete. 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.582+0000 7f51aae00700 1 Processor -- start 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.582+0000 7f51aae00700 1 -- start start 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.582+0000 7f51aae00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 0x7f51a4115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.582+0000 7f51aae00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51a4105b10 0x7f51a4116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.582+0000 7f51aae00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a41168e0 con 0x7f51a4105170 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.582+0000 7f51aae00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a410fbf0 con 0x7f51a4105b10 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.583+0000 7f51a3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 0x7f51a4115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.583+0000 7f51a3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 0x7f51a4115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42612/0 (socket says 172.21.15.1:42612) 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.583+0000 7f51a3e00700 1 -- 172.21.15.1:0/1216526403 learned_addr learned my addr 172.21.15.1:0/1216526403 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.583+0000 7f51a3e00700 1 -- 172.21.15.1:0/1216526403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51a4105b10 msgr2=0x7f51a4116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.583+0000 7f51a3e00700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51a4105b10 0x7f51a4116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.583+0000 7f51a3e00700 1 -- 172.21.15.1:0/1216526403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5194005040 con 0x7f51a4105170 2024-02-28T16:56:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.583+0000 7f51a3e00700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 0x7f51a4115d50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f519400b3b0 tx=0x7f5194009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.584+0000 7f519be00700 1 -- 172.21.15.1:0/1216526403 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519400e040 con 0x7f51a4105170 2024-02-28T16:56:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.584+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a410fe70 con 0x7f51a4105170 2024-02-28T16:56:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.584+0000 7f519be00700 1 -- 172.21.15.1:0/1216526403 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f519400bbf0 con 0x7f51a4105170 2024-02-28T16:56:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.584+0000 7f519be00700 1 -- 172.21.15.1:0/1216526403 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519401c630 con 0x7f51a4105170 2024-02-28T16:56:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.584+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a41103b0 con 0x7f51a4105170 2024-02-28T16:56:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.585+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a4102480 con 0x7f51a4105170 2024-02-28T16:56:22.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.586+0000 7f519be00700 1 -- 172.21.15.1:0/1216526403 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f519400bd60 con 0x7f51a4105170 2024-02-28T16:56:22.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.587+0000 7f519be00700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f518806d7e0 0x7f518806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:22.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.587+0000 7f519be00700 1 -- 172.21.15.1:0/1216526403 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f519408e2e0 con 0x7f51a4105170 2024-02-28T16:56:22.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.587+0000 7f51a3400700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f518806d7e0 0x7f518806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:22.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.588+0000 7f51a3400700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f518806d7e0 0x7f518806fc90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f51a41115f0 tx=0x7f5190009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.591+0000 7f519be00700 1 -- 172.21.15.1:0/1216526403 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f519405b6e0 con 0x7f51a4105170 2024-02-28T16:56:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.891+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f51a404bed0 con 0x7f51a4105170 2024-02-28T16:56:22.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.891+0000 7f519be00700 1 -- 172.21.15.1:0/1216526403 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f519405b270 con 0x7f51a4105170 2024-02-28T16:56:22.888 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:56:22.888 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:56:22.888 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:56:22.888 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:56:22.888 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:56:22.889 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:56:22.889 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:56:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f518806d7e0 msgr2=0x7f518806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f518806d7e0 0x7f518806fc90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f51a41115f0 tx=0x7f5190009ce0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 msgr2=0x7f51a4115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 0x7f51a4115d50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f519400b3b0 tx=0x7f5194009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 shutdown_connections 2024-02-28T16:56:22.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f518806d7e0 0x7f518806fc90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51a4105170 0x7f51a4115d50 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 --2- 172.21.15.1:0/1216526403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51a4105b10 0x7f51a4116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.894+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 >> 172.21.15.1:0/1216526403 conn(0x7f51a40face0 msgr2=0x7f51a40fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.895+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 shutdown_connections 2024-02-28T16:56:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:22.895+0000 7f51aae00700 1 -- 172.21.15.1:0/1216526403 wait complete. 2024-02-28T16:56:23.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:23 smithi001 ceph-mon[112940]: from='client.35222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:23.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:23 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1216526403' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:56:23.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:23 smithi066 ceph-mon[85781]: from='client.35222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:23.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:23 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1216526403' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:56:24.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:24 smithi001 ceph-mon[112940]: pgmap v658: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:24.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:24 smithi066 ceph-mon[85781]: pgmap v658: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:26.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:26 smithi001 ceph-mon[112940]: pgmap v659: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:26.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:56:26.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:26 smithi066 ceph-mon[85781]: pgmap v659: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:26.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:56:28.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:28 smithi066 ceph-mon[85781]: pgmap v660: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:28.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:28 smithi001 ceph-mon[112940]: pgmap v660: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:30.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:30 smithi066 ceph-mon[85781]: pgmap v661: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:30.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:30 smithi001 ceph-mon[112940]: pgmap v661: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:32.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:32 smithi066 ceph-mon[85781]: pgmap v662: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:32.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:32 smithi001 ceph-mon[112940]: pgmap v662: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:34.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:34 smithi066 ceph-mon[85781]: pgmap v663: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:34.932 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:34 smithi001 ceph-mon[112940]: pgmap v663: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:36.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:36 smithi066 ceph-mon[85781]: pgmap v664: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:36.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:36 smithi001 ceph-mon[112940]: pgmap v664: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:38.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:38 smithi066 ceph-mon[85781]: pgmap v665: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:38.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:38 smithi001 ceph-mon[112940]: pgmap v665: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:40.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:40 smithi066 ceph-mon[85781]: pgmap v666: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:40.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:40 smithi001 ceph-mon[112940]: pgmap v666: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:41.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:56:41.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:56:42.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:42 smithi066 ceph-mon[85781]: pgmap v667: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:42.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:42 smithi001 ceph-mon[112940]: pgmap v667: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:44.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:44 smithi066 ceph-mon[85781]: pgmap v668: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:44.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:44 smithi001 ceph-mon[112940]: pgmap v668: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:46.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:46 smithi066 ceph-mon[85781]: pgmap v669: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:46.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:46 smithi001 ceph-mon[112940]: pgmap v669: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:48.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:48 smithi066 ceph-mon[85781]: pgmap v670: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:48.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:48 smithi001 ceph-mon[112940]: pgmap v670: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:49.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:56:49.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:56:49.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:56:49.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:56:49.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:49.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:49.681 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:49.682 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:49 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:49.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:56:49.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T16:56:49.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:56:49.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:56:49.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:49.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:49.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:49.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:49 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:50.451 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:50 smithi066 ceph-mon[85781]: pgmap v671: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:50.451 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:50 smithi066 ceph-mon[85781]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-02-28T16:56:50.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:50 smithi001 ceph-mon[112940]: pgmap v671: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:50.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:50 smithi001 ceph-mon[112940]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-02-28T16:56:52.725 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:52 smithi066 ceph-mon[85781]: pgmap v672: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:52.725 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:52 smithi066 ceph-mon[85781]: Upgrade: Updating mds.cephfs.smithi066.wfbdor 2024-02-28T16:56:52.725 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:56:52.725 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.wfbdor", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:56:52.725 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:52 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:56:52.725 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:52 smithi066 ceph-mon[85781]: Deploying daemon mds.cephfs.smithi066.wfbdor on smithi066 2024-02-28T16:56:52.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:52 smithi001 ceph-mon[112940]: pgmap v672: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:52.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:52 smithi001 ceph-mon[112940]: Upgrade: Updating mds.cephfs.smithi066.wfbdor 2024-02-28T16:56:52.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:56:52.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.wfbdor", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T16:56:52.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:52 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T16:56:52.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:52 smithi001 ceph-mon[112940]: Deploying daemon mds.cephfs.smithi066.wfbdor on smithi066 2024-02-28T16:56:53.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.066+0000 7fce46000700 1 -- 172.21.15.1:0/1449042509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 msgr2=0x7fce401099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.066+0000 7fce46000700 1 --2- 172.21.15.1:0/1449042509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 0x7fce401099c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fce30004940 tx=0x7fce30009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.067+0000 7fce46000700 1 -- 172.21.15.1:0/1449042509 shutdown_connections 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.067+0000 7fce46000700 1 --2- 172.21.15.1:0/1449042509 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce40103dc0 0x7fce40104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.067+0000 7fce46000700 1 --2- 172.21.15.1:0/1449042509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 0x7fce401099c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.067+0000 7fce46000700 1 -- 172.21.15.1:0/1449042509 >> 172.21.15.1:0/1449042509 conn(0x7fce40072d50 msgr2=0x7fce40075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.067+0000 7fce46000700 1 -- 172.21.15.1:0/1449042509 shutdown_connections 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.067+0000 7fce46000700 1 -- 172.21.15.1:0/1449042509 wait complete. 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.068+0000 7fce46000700 1 Processor -- start 2024-02-28T16:56:53.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.068+0000 7fce46000700 1 -- start start 2024-02-28T16:56:53.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.068+0000 7fce46000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce40103dc0 0x7fce4011a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.068+0000 7fce46000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 0x7fce4011a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.068+0000 7fce46000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4011ac70 con 0x7fce401095f0 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.068+0000 7fce46000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce40113f80 con 0x7fce40103dc0 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.069+0000 7fce3ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 0x7fce4011a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.069+0000 7fce3f400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce40103dc0 0x7fce4011a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.069+0000 7fce3ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 0x7fce4011a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59210/0 (socket says 172.21.15.1:59210) 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.069+0000 7fce3ea00700 1 -- 172.21.15.1:0/1187911737 learned_addr learned my addr 172.21.15.1:0/1187911737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.069+0000 7fce3ea00700 1 -- 172.21.15.1:0/1187911737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce40103dc0 msgr2=0x7fce4011a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.069+0000 7fce3ea00700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce40103dc0 0x7fce4011a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.069+0000 7fce3ea00700 1 -- 172.21.15.1:0/1187911737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce30005040 con 0x7fce401095f0 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.069+0000 7fce3ea00700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 0x7fce4011a620 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fce2c002a50 tx=0x7fce2c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:53.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.070+0000 7fce37400700 1 -- 172.21.15.1:0/1187911737 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce2c007cc0 con 0x7fce401095f0 2024-02-28T16:56:53.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.070+0000 7fce37400700 1 -- 172.21.15.1:0/1187911737 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce2c00f040 con 0x7fce401095f0 2024-02-28T16:56:53.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.070+0000 7fce37400700 1 -- 172.21.15.1:0/1187911737 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce2c00da30 con 0x7fce401095f0 2024-02-28T16:56:53.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.070+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce40114260 con 0x7fce401095f0 2024-02-28T16:56:53.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.070+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce401147a0 con 0x7fce401095f0 2024-02-28T16:56:53.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.071+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce40106850 con 0x7fce401095f0 2024-02-28T16:56:53.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.072+0000 7fce37400700 1 -- 172.21.15.1:0/1187911737 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fce2c00db90 con 0x7fce401095f0 2024-02-28T16:56:53.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.073+0000 7fce37400700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce2406d7f0 0x7fce2406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.073+0000 7fce37400700 1 -- 172.21.15.1:0/1187911737 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fce2c08cbf0 con 0x7fce401095f0 2024-02-28T16:56:53.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.074+0000 7fce3f400700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce2406d7f0 0x7fce2406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.074+0000 7fce3f400700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce2406d7f0 0x7fce2406fca0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fce3000b3b0 tx=0x7fce30009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:53.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.079+0000 7fce37400700 1 -- 172.21.15.1:0/1187911737 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fce2c059f50 con 0x7fce401095f0 2024-02-28T16:56:53.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.323+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce401156c0 con 0x7fce2406d7f0 2024-02-28T16:56:53.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.326+0000 7fce37400700 1 -- 172.21.15.1:0/1187911737 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fce401156c0 con 0x7fce2406d7f0 2024-02-28T16:56:53.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.329+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce2406d7f0 msgr2=0x7fce2406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.329+0000 7fce46000700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce2406d7f0 0x7fce2406fca0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fce3000b3b0 tx=0x7fce30009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:53.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.329+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 msgr2=0x7fce4011a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.329+0000 7fce46000700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 0x7fce4011a620 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fce2c002a50 tx=0x7fce2c0046b0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.330+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 shutdown_connections 2024-02-28T16:56:53.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.330+0000 7fce46000700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce2406d7f0 0x7fce2406fca0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.330+0000 7fce46000700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce40103dc0 0x7fce4011a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.330+0000 7fce46000700 1 --2- 172.21.15.1:0/1187911737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce401095f0 0x7fce4011a620 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.330+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 >> 172.21.15.1:0/1187911737 conn(0x7fce40072d50 msgr2=0x7fce40074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:53.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.330+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 shutdown_connections 2024-02-28T16:56:53.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.330+0000 7fce46000700 1 -- 172.21.15.1:0/1187911737 wait complete. 2024-02-28T16:56:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:56:53.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.491+0000 7fa183a00700 1 -- 172.21.15.1:0/2586356363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fe900 msgr2=0x7fa17c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.491+0000 7fa183a00700 1 --2- 172.21.15.1:0/2586356363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fe900 0x7fa17c106f40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa168004940 tx=0x7fa168009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.491+0000 7fa183a00700 1 -- 172.21.15.1:0/2586356363 shutdown_connections 2024-02-28T16:56:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.491+0000 7fa183a00700 1 --2- 172.21.15.1:0/2586356363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fe900 0x7fa17c106f40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.491+0000 7fa183a00700 1 --2- 172.21.15.1:0/2586356363 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa17c0fdf60 0x7fa17c0fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.491+0000 7fa183a00700 1 -- 172.21.15.1:0/2586356363 >> 172.21.15.1:0/2586356363 conn(0x7fa17c0f7a10 msgr2=0x7fa17c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.491+0000 7fa183a00700 1 -- 172.21.15.1:0/2586356363 shutdown_connections 2024-02-28T16:56:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.491+0000 7fa183a00700 1 -- 172.21.15.1:0/2586356363 wait complete. 2024-02-28T16:56:53.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.492+0000 7fa183a00700 1 Processor -- start 2024-02-28T16:56:53.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.492+0000 7fa183a00700 1 -- start start 2024-02-28T16:56:53.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.493+0000 7fa183a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fdf60 0x7fa17c0fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.493+0000 7fa183a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa17c0fe900 0x7fa17c0fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.493+0000 7fa183a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0fc8d0 con 0x7fa17c0fdf60 2024-02-28T16:56:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.493+0000 7fa183a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0fca40 con 0x7fa17c0fe900 2024-02-28T16:56:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.493+0000 7fa181000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fdf60 0x7fa17c0fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.493+0000 7fa181000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fdf60 0x7fa17c0fbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59216/0 (socket says 172.21.15.1:59216) 2024-02-28T16:56:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.493+0000 7fa181000700 1 -- 172.21.15.1:0/3621842977 learned_addr learned my addr 172.21.15.1:0/3621842977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.493+0000 7fa17be00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa17c0fe900 0x7fa17c0fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.494+0000 7fa181000700 1 -- 172.21.15.1:0/3621842977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa17c0fe900 msgr2=0x7fa17c0fc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.494+0000 7fa181000700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa17c0fe900 0x7fa17c0fc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.494+0000 7fa181000700 1 -- 172.21.15.1:0/3621842977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa168005040 con 0x7fa17c0fdf60 2024-02-28T16:56:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.494+0000 7fa181000700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fdf60 0x7fa17c0fbd90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa170002a50 tx=0x7fa1700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.494+0000 7fa178c00700 1 -- 172.21.15.1:0/3621842977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa170007cc0 con 0x7fa17c0fdf60 2024-02-28T16:56:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.494+0000 7fa178c00700 1 -- 172.21.15.1:0/3621842977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa17000f040 con 0x7fa17c0fdf60 2024-02-28T16:56:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.494+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c119d90 con 0x7fa17c0fdf60 2024-02-28T16:56:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.494+0000 7fa178c00700 1 -- 172.21.15.1:0/3621842977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17000db70 con 0x7fa17c0fdf60 2024-02-28T16:56:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.495+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c11a270 con 0x7fa17c0fdf60 2024-02-28T16:56:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.496+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c104550 con 0x7fa17c0fdf60 2024-02-28T16:56:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.497+0000 7fa178c00700 1 -- 172.21.15.1:0/3621842977 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa17000dd10 con 0x7fa17c0fdf60 2024-02-28T16:56:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.498+0000 7fa178c00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa16406d7f0 0x7fa16406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.498+0000 7fa178c00700 1 -- 172.21.15.1:0/3621842977 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa17008c540 con 0x7fa17c0fdf60 2024-02-28T16:56:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.498+0000 7fa17be00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa16406d7f0 0x7fa16406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.499+0000 7fa17be00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa16406d7f0 0x7fa16406fca0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0fd790 tx=0x7fa168009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:53.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.503+0000 7fa178c00700 1 -- 172.21.15.1:0/3621842977 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa170055d70 con 0x7fa17c0fdf60 2024-02-28T16:56:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.750+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa17c0fd4d0 con 0x7fa16406d7f0 2024-02-28T16:56:53.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.752+0000 7fa178c00700 1 -- 172.21.15.1:0/3621842977 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa17c0fd4d0 con 0x7fa16406d7f0 2024-02-28T16:56:53.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.755+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa16406d7f0 msgr2=0x7fa16406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.755+0000 7fa183a00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa16406d7f0 0x7fa16406fca0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0fd790 tx=0x7fa168009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:53.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.755+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fdf60 msgr2=0x7fa17c0fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.755+0000 7fa183a00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fdf60 0x7fa17c0fbd90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa170002a50 tx=0x7fa1700046b0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.756+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 shutdown_connections 2024-02-28T16:56:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.756+0000 7fa183a00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa16406d7f0 0x7fa16406fca0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.756+0000 7fa183a00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17c0fdf60 0x7fa17c0fbd90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.756+0000 7fa183a00700 1 --2- 172.21.15.1:0/3621842977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa17c0fe900 0x7fa17c0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.756+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 >> 172.21.15.1:0/3621842977 conn(0x7fa17c0f7a10 msgr2=0x7fa17c1016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.756+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 shutdown_connections 2024-02-28T16:56:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.756+0000 7fa183a00700 1 -- 172.21.15.1:0/3621842977 wait complete. 2024-02-28T16:56:53.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.911+0000 7f1b2c800700 1 -- 172.21.15.1:0/3083692225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 msgr2=0x7f1b24107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.911+0000 7f1b2c800700 1 --2- 172.21.15.1:0/3083692225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 0x7f1b24107050 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1b14004970 tx=0x7f1b14009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:53.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.912+0000 7f1b2c800700 1 -- 172.21.15.1:0/3083692225 shutdown_connections 2024-02-28T16:56:53.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.912+0000 7f1b2c800700 1 --2- 172.21.15.1:0/3083692225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 0x7f1b24107050 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.912+0000 7f1b2c800700 1 --2- 172.21.15.1:0/3083692225 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b240fdfe0 0x7f1b240fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.912+0000 7f1b2c800700 1 -- 172.21.15.1:0/3083692225 >> 172.21.15.1:0/3083692225 conn(0x7f1b24071d90 msgr2=0x7f1b24072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:53.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.912+0000 7f1b2c800700 1 -- 172.21.15.1:0/3083692225 shutdown_connections 2024-02-28T16:56:53.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.912+0000 7f1b2c800700 1 -- 172.21.15.1:0/3083692225 wait complete. 2024-02-28T16:56:53.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.913+0000 7f1b2c800700 1 Processor -- start 2024-02-28T16:56:53.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.913+0000 7f1b2c800700 1 -- start start 2024-02-28T16:56:53.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b2c800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b240fdfe0 0x7f1b24115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b2c800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 0x7f1b24116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b2c800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b24116790 con 0x7f1b240fe8f0 2024-02-28T16:56:53.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b2c800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2410fac0 con 0x7f1b240fdfe0 2024-02-28T16:56:53.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b29400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 0x7f1b24116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b29400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 0x7f1b24116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59234/0 (socket says 172.21.15.1:59234) 2024-02-28T16:56:53.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b29400700 1 -- 172.21.15.1:0/3624548594 learned_addr learned my addr 172.21.15.1:0/3624548594 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:53.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b29e00700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b240fdfe0 0x7f1b24115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.914+0000 7f1b29400700 1 -- 172.21.15.1:0/3624548594 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b240fdfe0 msgr2=0x7f1b24115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:53.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.915+0000 7f1b29400700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b240fdfe0 0x7f1b24115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:53.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.915+0000 7f1b29400700 1 -- 172.21.15.1:0/3624548594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b14005040 con 0x7f1b240fe8f0 2024-02-28T16:56:53.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.915+0000 7f1b29400700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 0x7f1b24116140 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009150 tx=0x7f1b14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:53.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.915+0000 7f1b1e000700 1 -- 172.21.15.1:0/3624548594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b1400e040 con 0x7f1b240fe8f0 2024-02-28T16:56:53.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.915+0000 7f1b1e000700 1 -- 172.21.15.1:0/3624548594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b14004290 con 0x7f1b240fe8f0 2024-02-28T16:56:53.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.915+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2410fd40 con 0x7f1b240fe8f0 2024-02-28T16:56:53.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.915+0000 7f1b1e000700 1 -- 172.21.15.1:0/3624548594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b1401c3f0 con 0x7f1b240fe8f0 2024-02-28T16:56:53.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.915+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b24110280 con 0x7f1b240fe8f0 2024-02-28T16:56:53.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.917+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b24106b20 con 0x7f1b240fe8f0 2024-02-28T16:56:53.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.918+0000 7f1b1e000700 1 -- 172.21.15.1:0/3624548594 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1b1400bc10 con 0x7f1b240fe8f0 2024-02-28T16:56:53.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.919+0000 7f1b1e000700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b0c06d7f0 0x7f1b0c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:53.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.919+0000 7f1b1e000700 1 -- 172.21.15.1:0/3624548594 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1b1408e220 con 0x7f1b240fe8f0 2024-02-28T16:56:53.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.919+0000 7f1b29e00700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b0c06d7f0 0x7f1b0c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.920+0000 7f1b29e00700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b0c06d7f0 0x7f1b0c06fca0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1b18007fd0 tx=0x7f1b18009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:53.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:53.924+0000 7f1b1e000700 1 -- 172.21.15.1:0/3624548594 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b1405b580 con 0x7f1b240fe8f0 2024-02-28T16:56:54.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.156+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b24111280 con 0x7f1b0c06d7f0 2024-02-28T16:56:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.174+0000 7f1b1e000700 1 -- 172.21.15.1:0/3624548594 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1b24111280 con 0x7f1b0c06d7f0 2024-02-28T16:56:54.172 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:56:54.172 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (23m) 15m ago 37m 14.7M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (19m) 15m ago 37m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (19m) 15m ago 36m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (22m) 15m ago 36m 76.0M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (15m) 15m ago 30m 11.0M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (15m) 15m ago 30m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (29m) 15m ago 29m 15.2M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (30m) 15m ago 30m 119M - 17.2.7-535-gc0f02e60 1c37ed371348 133bf7406108 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (25m) 15m ago 39m 508M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (24m) 15m ago 35m 451M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (21m) 15m ago 39m 52.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (20m) 15m ago 35m 38.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (24m) 15m ago 37m 8074k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:56:54.173 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (23m) 15m ago 36m 8326k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:56:54.174 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (19m) 15m ago 34m 87.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:56:54.174 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (18m) 15m ago 34m 92.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:56:54.174 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (18m) 15m ago 33m 82.2M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:56:54.174 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (17m) 15m ago 33m 101M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:56:54.174 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (16m) 15m ago 32m 83.4M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:56:54.174 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (16m) 15m ago 31m 11.1M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:56:54.174 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (23m) 15m ago 36m 47.1M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:56:54.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b0c06d7f0 msgr2=0x7f1b0c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:54.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b0c06d7f0 0x7f1b0c06fca0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1b18007fd0 tx=0x7f1b18009040 comp rx=0 tx=0).stop 2024-02-28T16:56:54.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 msgr2=0x7f1b24116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 0x7f1b24116140 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009150 tx=0x7f1b14009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 shutdown_connections 2024-02-28T16:56:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b0c06d7f0 0x7f1b0c06fca0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b240fdfe0 0x7f1b24115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 --2- 172.21.15.1:0/3624548594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b240fe8f0 0x7f1b24116140 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.177+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 >> 172.21.15.1:0/3624548594 conn(0x7f1b24071d90 msgr2=0x7f1b240fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.178+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 shutdown_connections 2024-02-28T16:56:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.178+0000 7f1b2c800700 1 -- 172.21.15.1:0/3624548594 wait complete. 2024-02-28T16:56:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.330+0000 7f02efc00700 1 -- 172.21.15.1:0/424215021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e8103dc0 msgr2=0x7f02e8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.330+0000 7f02efc00700 1 --2- 172.21.15.1:0/424215021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e8103dc0 0x7f02e8104210 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f02d4004940 tx=0x7f02d4009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.330+0000 7f02efc00700 1 -- 172.21.15.1:0/424215021 shutdown_connections 2024-02-28T16:56:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.330+0000 7f02efc00700 1 --2- 172.21.15.1:0/424215021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e8103dc0 0x7f02e8104210 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.330+0000 7f02efc00700 1 --2- 172.21.15.1:0/424215021 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e81095f0 0x7f02e81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.330+0000 7f02efc00700 1 -- 172.21.15.1:0/424215021 >> 172.21.15.1:0/424215021 conn(0x7f02e8072d50 msgr2=0x7f02e8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.330+0000 7f02efc00700 1 -- 172.21.15.1:0/424215021 shutdown_connections 2024-02-28T16:56:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.330+0000 7f02efc00700 1 -- 172.21.15.1:0/424215021 wait complete. 2024-02-28T16:56:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.331+0000 7f02efc00700 1 Processor -- start 2024-02-28T16:56:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.332+0000 7f02efc00700 1 -- start start 2024-02-28T16:56:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.332+0000 7f02efc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e8103dc0 0x7f02e811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.332+0000 7f02efc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e81095f0 0x7f02e811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.332+0000 7f02efc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e811abd0 con 0x7f02e81095f0 2024-02-28T16:56:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.332+0000 7f02efc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8113ee0 con 0x7f02e8103dc0 2024-02-28T16:56:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.332+0000 7f02e7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e81095f0 0x7f02e811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.332+0000 7f02e7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e81095f0 0x7f02e811a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59256/0 (socket says 172.21.15.1:59256) 2024-02-28T16:56:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.332+0000 7f02e7e00700 1 -- 172.21.15.1:0/3865059544 learned_addr learned my addr 172.21.15.1:0/3865059544 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02ed200700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e8103dc0 0x7f02e811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02e7e00700 1 -- 172.21.15.1:0/3865059544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e8103dc0 msgr2=0x7f02e811a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02e7e00700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e8103dc0 0x7f02e811a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02e7e00700 1 -- 172.21.15.1:0/3865059544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d4005040 con 0x7f02e81095f0 2024-02-28T16:56:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02e7e00700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e81095f0 0x7f02e811a580 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009fd0 tx=0x7f02d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02e4c00700 1 -- 172.21.15.1:0/3865059544 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d400e040 con 0x7f02e81095f0 2024-02-28T16:56:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02e4c00700 1 -- 172.21.15.1:0/3865059544 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02d400bbf0 con 0x7f02e81095f0 2024-02-28T16:56:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02e4c00700 1 -- 172.21.15.1:0/3865059544 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d401c3f0 con 0x7f02e81095f0 2024-02-28T16:56:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.333+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e8114160 con 0x7f02e81095f0 2024-02-28T16:56:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.334+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e81146a0 con 0x7f02e81095f0 2024-02-28T16:56:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.335+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e810b9c0 con 0x7f02e81095f0 2024-02-28T16:56:54.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.336+0000 7f02e4c00700 1 -- 172.21.15.1:0/3865059544 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f02d400bd60 con 0x7f02e81095f0 2024-02-28T16:56:54.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.336+0000 7f02e4c00700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02d006d7f0 0x7f02d006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:54.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.337+0000 7f02e4c00700 1 -- 172.21.15.1:0/3865059544 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f02d408e100 con 0x7f02e81095f0 2024-02-28T16:56:54.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.337+0000 7f02ed200700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02d006d7f0 0x7f02d006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:54.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.338+0000 7f02ed200700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02d006d7f0 0x7f02d006fca0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f02dc004990 tx=0x7f02dc009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:54.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.341+0000 7f02e4c00700 1 -- 172.21.15.1:0/3865059544 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f02d405b5a0 con 0x7f02e81095f0 2024-02-28T16:56:54.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.652+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f02e804bed0 con 0x7f02e81095f0 2024-02-28T16:56:54.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.653+0000 7f02e4c00700 1 -- 172.21.15.1:0/3865059544 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f02d405b130 con 0x7f02e81095f0 2024-02-28T16:56:54.651 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:56:54.652 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:56:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:56:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 12 2024-02-28T16:56:54.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:56:54.653 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:56:54.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.656+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02d006d7f0 msgr2=0x7f02d006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:54.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.656+0000 7f02efc00700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02d006d7f0 0x7f02d006fca0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f02dc004990 tx=0x7f02dc009ee0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.656+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e81095f0 msgr2=0x7f02e811a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:54.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.656+0000 7f02efc00700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e81095f0 0x7f02e811a580 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009fd0 tx=0x7f02d4009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:54.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.657+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 shutdown_connections 2024-02-28T16:56:54.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.657+0000 7f02efc00700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02d006d7f0 0x7f02d006fca0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.657+0000 7f02efc00700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e8103dc0 0x7f02e811a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.657+0000 7f02efc00700 1 --2- 172.21.15.1:0/3865059544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e81095f0 0x7f02e811a580 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.657+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 >> 172.21.15.1:0/3865059544 conn(0x7f02e8072d50 msgr2=0x7f02e8074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:54.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.657+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 shutdown_connections 2024-02-28T16:56:54.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.657+0000 7f02efc00700 1 -- 172.21.15.1:0/3865059544 wait complete. 2024-02-28T16:56:54.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:54 smithi066 ceph-mon[85781]: pgmap v673: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:54.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:54 smithi066 ceph-mon[85781]: from='client.35230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:54.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:54 smithi066 ceph-mon[85781]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:54.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:54 smithi001 ceph-mon[112940]: pgmap v673: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:54.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:54 smithi001 ceph-mon[112940]: from='client.35230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:54.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:54 smithi001 ceph-mon[112940]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.810+0000 7f83fb400700 1 -- 172.21.15.1:0/2107424973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fbdd0 msgr2=0x7f83f40fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.810+0000 7f83fb400700 1 --2- 172.21.15.1:0/2107424973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fbdd0 0x7f83f40fc1a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f83e8004940 tx=0x7f83e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.810+0000 7f83fb400700 1 -- 172.21.15.1:0/2107424973 shutdown_connections 2024-02-28T16:56:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.810+0000 7f83fb400700 1 --2- 172.21.15.1:0/2107424973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83f40fc6e0 0x7f83f4100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.810+0000 7f83fb400700 1 --2- 172.21.15.1:0/2107424973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fbdd0 0x7f83f40fc1a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.810+0000 7f83fb400700 1 -- 172.21.15.1:0/2107424973 >> 172.21.15.1:0/2107424973 conn(0x7f83f4071d90 msgr2=0x7f83f4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.810+0000 7f83fb400700 1 -- 172.21.15.1:0/2107424973 shutdown_connections 2024-02-28T16:56:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.811+0000 7f83fb400700 1 -- 172.21.15.1:0/2107424973 wait complete. 2024-02-28T16:56:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.811+0000 7f83fb400700 1 Processor -- start 2024-02-28T16:56:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.811+0000 7f83fb400700 1 -- start start 2024-02-28T16:56:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.812+0000 7f83fb400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83f40fbdd0 0x7f83f411a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.812+0000 7f83fb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fc6e0 0x7f83f411a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.812+0000 7f83fb400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f411ac30 con 0x7f83f40fc6e0 2024-02-28T16:56:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.812+0000 7f83fb400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f4113f40 con 0x7f83f40fbdd0 2024-02-28T16:56:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.812+0000 7f83f3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fc6e0 0x7f83f411a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.812+0000 7f83f3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fc6e0 0x7f83f411a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59272/0 (socket says 172.21.15.1:59272) 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.812+0000 7f83f3e00700 1 -- 172.21.15.1:0/315786807 learned_addr learned my addr 172.21.15.1:0/315786807 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f8a00700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83f40fbdd0 0x7f83f411a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f3e00700 1 -- 172.21.15.1:0/315786807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83f40fbdd0 msgr2=0x7f83f411a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f3e00700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83f40fbdd0 0x7f83f411a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f3e00700 1 -- 172.21.15.1:0/315786807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8005040 con 0x7f83f40fc6e0 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f8a00700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83f40fbdd0 0x7f83f411a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f3e00700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fc6e0 0x7f83f411a5e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f83e000c830 tx=0x7f83e0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f0c00700 1 -- 172.21.15.1:0/315786807 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83e000a760 con 0x7f83f40fc6e0 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f0c00700 1 -- 172.21.15.1:0/315786807 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83e000ad90 con 0x7f83f40fc6e0 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83f0c00700 1 -- 172.21.15.1:0/315786807 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83e00113a0 con 0x7f83f40fc6e0 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.813+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f4114220 con 0x7f83f40fc6e0 2024-02-28T16:56:54.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.814+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f4114760 con 0x7f83f40fc6e0 2024-02-28T16:56:54.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.815+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f40fe540 con 0x7f83f40fc6e0 2024-02-28T16:56:54.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.816+0000 7f83f0c00700 1 -- 172.21.15.1:0/315786807 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f83e0012400 con 0x7f83f40fc6e0 2024-02-28T16:56:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.818+0000 7f83f0c00700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83dc06d7f0 0x7f83dc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.818+0000 7f83f0c00700 1 -- 172.21.15.1:0/315786807 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f83e008cf00 con 0x7f83f40fc6e0 2024-02-28T16:56:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.818+0000 7f83f8a00700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83dc06d7f0 0x7f83dc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:54.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.819+0000 7f83f8a00700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83dc06d7f0 0x7f83dc06fca0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f83e800b3b0 tx=0x7f83e8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:54.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:54.822+0000 7f83f0c00700 1 -- 172.21.15.1:0/315786807 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f83e005a2e0 con 0x7f83f40fc6e0 2024-02-28T16:56:55.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.101+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f83f40738b0 con 0x7f83f40fc6e0 2024-02-28T16:56:55.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.103+0000 7f83f0c00700 1 -- 172.21.15.1:0/315786807 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1656 (secure 0 0 0) 0x7f83e0059e70 con 0x7f83f40fc6e0 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:e25 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:56:55.102 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:56:55.103 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:56:55.104 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:56:55.105 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:56:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83dc06d7f0 msgr2=0x7f83dc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83dc06d7f0 0x7f83dc06fca0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f83e800b3b0 tx=0x7f83e8009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fc6e0 msgr2=0x7f83f411a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fc6e0 0x7f83f411a5e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f83e000c830 tx=0x7f83e0004470 comp rx=0 tx=0).stop 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 shutdown_connections 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83dc06d7f0 0x7f83dc06fca0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83f40fbdd0 0x7f83f411a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 --2- 172.21.15.1:0/315786807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83f40fc6e0 0x7f83f411a5e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.106+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 >> 172.21.15.1:0/315786807 conn(0x7f83f4071d90 msgr2=0x7f83f410d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.107+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 shutdown_connections 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.107+0000 7f83fb400700 1 -- 172.21.15.1:0/315786807 wait complete. 2024-02-28T16:56:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-02-28T16:56:55.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.269+0000 7f1efa000700 1 -- 172.21.15.1:0/1949783252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070c40 msgr2=0x7f1ef4109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:55.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.269+0000 7f1efa000700 1 --2- 172.21.15.1:0/1949783252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070c40 0x7f1ef4109260 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1edc004970 tx=0x7f1edc009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:55.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.269+0000 7f1efa000700 1 -- 172.21.15.1:0/1949783252 shutdown_connections 2024-02-28T16:56:55.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.269+0000 7f1efa000700 1 --2- 172.21.15.1:0/1949783252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070c40 0x7f1ef4109260 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.269+0000 7f1efa000700 1 --2- 172.21.15.1:0/1949783252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1ef4070330 0x7f1ef4070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.269+0000 7f1efa000700 1 -- 172.21.15.1:0/1949783252 >> 172.21.15.1:0/1949783252 conn(0x7f1ef40f8ae0 msgr2=0x7f1ef40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:55.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.269+0000 7f1efa000700 1 -- 172.21.15.1:0/1949783252 shutdown_connections 2024-02-28T16:56:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.269+0000 7f1efa000700 1 -- 172.21.15.1:0/1949783252 wait complete. 2024-02-28T16:56:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.270+0000 7f1efa000700 1 Processor -- start 2024-02-28T16:56:55.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.270+0000 7f1efa000700 1 -- start start 2024-02-28T16:56:55.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1efa000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070330 0x7f1ef4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:55.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1efa000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1ef4070c40 0x7f1ef4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:55.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1efa000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef4116790 con 0x7f1ef4070330 2024-02-28T16:56:55.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1efa000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef410fae0 con 0x7f1ef4070c40 2024-02-28T16:56:55.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1ef3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070330 0x7f1ef4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:55.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1ef3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070330 0x7f1ef4115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59290/0 (socket says 172.21.15.1:59290) 2024-02-28T16:56:55.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1ef3400700 1 -- 172.21.15.1:0/2203921928 learned_addr learned my addr 172.21.15.1:0/2203921928 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:55.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1eebe00700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1ef4070c40 0x7f1ef4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:55.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1ef3400700 1 -- 172.21.15.1:0/2203921928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1ef4070c40 msgr2=0x7f1ef4116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.271+0000 7f1ef3400700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1ef4070c40 0x7f1ef4116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.272+0000 7f1ef3400700 1 -- 172.21.15.1:0/2203921928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc005040 con 0x7f1ef4070330 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.272+0000 7f1eebe00700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1ef4070c40 0x7f1ef4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.272+0000 7f1ef3400700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070330 0x7f1ef4115c00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4002a50 tx=0x7f1ee40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.272+0000 7f1eeb400700 1 -- 172.21.15.1:0/2203921928 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ee4007cc0 con 0x7f1ef4070330 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.272+0000 7f1eeb400700 1 -- 172.21.15.1:0/2203921928 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ee400f040 con 0x7f1ef4070330 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.272+0000 7f1eeb400700 1 -- 172.21.15.1:0/2203921928 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ee400db70 con 0x7f1ef4070330 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.272+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef410fdc0 con 0x7f1ef4070330 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.272+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef4110300 con 0x7f1ef4070330 2024-02-28T16:56:55.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.274+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef4106920 con 0x7f1ef4070330 2024-02-28T16:56:55.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.275+0000 7f1eeb400700 1 -- 172.21.15.1:0/2203921928 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1ee400dd10 con 0x7f1ef4070330 2024-02-28T16:56:55.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.276+0000 7f1eeb400700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1ed806d7f0 0x7f1ed806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:55.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.276+0000 7f1eeb400700 1 -- 172.21.15.1:0/2203921928 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1ee408c5d0 con 0x7f1ef4070330 2024-02-28T16:56:55.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.276+0000 7f1eebe00700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1ed806d7f0 0x7f1ed806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:55.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.277+0000 7f1eebe00700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1ed806d7f0 0x7f1ed806fca0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1edc000c00 tx=0x7f1edc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:55.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.281+0000 7f1eeb400700 1 -- 172.21.15.1:0/2203921928 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1ee4055e00 con 0x7f1ef4070330 2024-02-28T16:56:55.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.523+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ef404bf40 con 0x7f1ed806d7f0 2024-02-28T16:56:55.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.525+0000 7f1eeb400700 1 -- 172.21.15.1:0/2203921928 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1ef404bf40 con 0x7f1ed806d7f0 2024-02-28T16:56:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:56:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:56:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:56:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:56:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "14/21 daemons upgraded", 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:56:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:56:55.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1ed806d7f0 msgr2=0x7f1ed806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1ed806d7f0 0x7f1ed806fca0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1edc000c00 tx=0x7f1edc009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070330 msgr2=0x7f1ef4115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070330 0x7f1ef4115c00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4002a50 tx=0x7f1ee40046b0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 shutdown_connections 2024-02-28T16:56:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1ed806d7f0 0x7f1ed806fca0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ef4070330 0x7f1ef4115c00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 --2- 172.21.15.1:0/2203921928 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1ef4070c40 0x7f1ef4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 >> 172.21.15.1:0/2203921928 conn(0x7f1ef40f8ae0 msgr2=0x7f1ef4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:55.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.528+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 shutdown_connections 2024-02-28T16:56:55.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.529+0000 7f1efa000700 1 -- 172.21.15.1:0/2203921928 wait complete. 2024-02-28T16:56:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.686+0000 7f353c000700 1 -- 172.21.15.1:0/2766702607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 msgr2=0x7f3534103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.686+0000 7f353c000700 1 --2- 172.21.15.1:0/2766702607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 0x7f3534103320 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3528004940 tx=0x7f3528009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.686+0000 7f353c000700 1 -- 172.21.15.1:0/2766702607 shutdown_connections 2024-02-28T16:56:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.686+0000 7f353c000700 1 --2- 172.21.15.1:0/2766702607 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35341038f0 0x7f35340fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.686+0000 7f353c000700 1 --2- 172.21.15.1:0/2766702607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 0x7f3534103320 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.686+0000 7f353c000700 1 -- 172.21.15.1:0/2766702607 >> 172.21.15.1:0/2766702607 conn(0x7f35340f8ae0 msgr2=0x7f35340faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:55.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.686+0000 7f353c000700 1 -- 172.21.15.1:0/2766702607 shutdown_connections 2024-02-28T16:56:55.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.686+0000 7f353c000700 1 -- 172.21.15.1:0/2766702607 wait complete. 2024-02-28T16:56:55.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.687+0000 7f353c000700 1 Processor -- start 2024-02-28T16:56:55.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.687+0000 7f353c000700 1 -- start start 2024-02-28T16:56:55.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.687+0000 7f353c000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 0x7f3534070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:55.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.687+0000 7f353c000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35341038f0 0x7f35340707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:55.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.687+0000 7f353c000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3534070e60 con 0x7f3534102f50 2024-02-28T16:56:55.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.687+0000 7f353c000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3534070fd0 con 0x7f35341038f0 2024-02-28T16:56:55.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.688+0000 7f3539600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 0x7f3534070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:55.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.688+0000 7f3539600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 0x7f3534070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59312/0 (socket says 172.21.15.1:59312) 2024-02-28T16:56:55.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.688+0000 7f3539600700 1 -- 172.21.15.1:0/481079653 learned_addr learned my addr 172.21.15.1:0/481079653 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:56:55.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.688+0000 7f3538c00700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35341038f0 0x7f35340707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:55.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.688+0000 7f3539600700 1 -- 172.21.15.1:0/481079653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35341038f0 msgr2=0x7f35340707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:55.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.688+0000 7f3539600700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35341038f0 0x7f35340707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:55.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.688+0000 7f3539600700 1 -- 172.21.15.1:0/481079653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3528005040 con 0x7f3534102f50 2024-02-28T16:56:55.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.689+0000 7f3539600700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 0x7f3534070290 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3528009fd0 tx=0x7f3528009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.689+0000 7f3531600700 1 -- 172.21.15.1:0/481079653 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f352800e040 con 0x7f3534102f50 2024-02-28T16:56:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.689+0000 7f3531600700 1 -- 172.21.15.1:0/481079653 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f352800bbf0 con 0x7f3534102f50 2024-02-28T16:56:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.689+0000 7f3531600700 1 -- 172.21.15.1:0/481079653 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f352801c3f0 con 0x7f3534102f50 2024-02-28T16:56:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.689+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3534119ce0 con 0x7f3534102f50 2024-02-28T16:56:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.689+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353411a1d0 con 0x7f3534102f50 2024-02-28T16:56:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.691+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35340fc6e0 con 0x7f3534102f50 2024-02-28T16:56:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.692+0000 7f3531600700 1 -- 172.21.15.1:0/481079653 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f352800bd60 con 0x7f3534102f50 2024-02-28T16:56:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.692+0000 7f3531600700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f351c06d7f0 0x7f351c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:56:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.693+0000 7f3531600700 1 -- 172.21.15.1:0/481079653 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f352808e0d0 con 0x7f3534102f50 2024-02-28T16:56:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.693+0000 7f3538c00700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f351c06d7f0 0x7f351c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:56:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.694+0000 7f3538c00700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f351c06d7f0 0x7f351c06fca0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3520006fd0 tx=0x7f3520008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:56:55.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:55.697+0000 7f3531600700 1 -- 172.21.15.1:0/481079653 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3528057a40 con 0x7f3534102f50 2024-02-28T16:56:55.778 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:55 smithi001 ceph-mon[112940]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:55.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:55 smithi001 ceph-mon[112940]: osdmap e83: 6 total, 6 up, 6 in 2024-02-28T16:56:55.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:55 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-02-28T16:56:55.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:55 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3865059544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:55.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:55 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/315786807' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:56:56.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:55 smithi066 ceph-mon[85781]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:56.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:55 smithi066 ceph-mon[85781]: osdmap e83: 6 total, 6 up, 6 in 2024-02-28T16:56:56.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:55 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-02-28T16:56:56.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:55 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3865059544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:56:56.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:55 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/315786807' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:56:56.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.011+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f353411a410 con 0x7f3534102f50 2024-02-28T16:56:56.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.012+0000 7f3531600700 1 -- 172.21.15.1:0/481079653 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f352805b100 con 0x7f3534102f50 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T16:56:56.011 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:56:56.012 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:56:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f351c06d7f0 msgr2=0x7f351c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f351c06d7f0 0x7f351c06fca0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3520006fd0 tx=0x7f3520008040 comp rx=0 tx=0).stop 2024-02-28T16:56:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 msgr2=0x7f3534070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:56:56.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 0x7f3534070290 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3528009fd0 tx=0x7f3528009e30 comp rx=0 tx=0).stop 2024-02-28T16:56:56.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 shutdown_connections 2024-02-28T16:56:56.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f351c06d7f0 0x7f351c06fca0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:56.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3534102f50 0x7f3534070290 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:56.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 --2- 172.21.15.1:0/481079653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35341038f0 0x7f35340707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:56:56.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.015+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 >> 172.21.15.1:0/481079653 conn(0x7f35340f8ae0 msgr2=0x7f35340fa460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:56:56.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.016+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 shutdown_connections 2024-02-28T16:56:56.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:56:56.016+0000 7f353c000700 1 -- 172.21.15.1:0/481079653 wait complete. 2024-02-28T16:56:56.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:56 smithi001 ceph-mon[112940]: pgmap v675: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:56.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:56 smithi001 ceph-mon[112940]: from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:56.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:56:56.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:56:56.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:56 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/481079653' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:56:57.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:56 smithi066 ceph-mon[85781]: pgmap v675: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:57.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:56 smithi066 ceph-mon[85781]: from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:56:57.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:56:57.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:56:57.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:56 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/481079653' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:56:57.986 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:57 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:56:58.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:57 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:56:58.648 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:56:58 smithi066 ceph-mon[85781]: pgmap v676: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:56:59.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:56:58 smithi001 ceph-mon[112940]: pgmap v676: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:01.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:00 smithi066 ceph-mon[85781]: pgmap v677: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:01.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:00 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:01.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:00 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:01.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:00 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:57:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:00 smithi001 ceph-mon[112940]: pgmap v677: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:00 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T16:57:02.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:01 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] up:boot 2024-02-28T16:57:02.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:01 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T16:57:02.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:01 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:57:02.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:01 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:02.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:01 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:02.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:01 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:02.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:01 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:02.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:01 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] up:boot 2024-02-28T16:57:02.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:01 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T16:57:02.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:01 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T16:57:02.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:01 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:02.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:01 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:02.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:01 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:02.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:01 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:02.832 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:02 smithi001 ceph-mon[112940]: pgmap v678: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:03.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:02 smithi066 ceph-mon[85781]: pgmap v678: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:04.756 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:04 smithi066 ceph-mon[85781]: pgmap v679: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:04.942 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:04 smithi001 ceph-mon[112940]: pgmap v679: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:06.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:06 smithi001 ceph-mon[112940]: pgmap v680: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:06.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:06.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:07.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:06 smithi066 ceph-mon[85781]: pgmap v680: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:07.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:07.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:09.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:08 smithi066 ceph-mon[85781]: pgmap v681: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:09.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:09.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:09.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:09.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:09.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:08 smithi001 ceph-mon[112940]: pgmap v681: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:09.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:09.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:09.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:09.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T16:57:11.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:10 smithi066 ceph-mon[85781]: pgmap v682: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:11.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:10 smithi001 ceph-mon[112940]: pgmap v682: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:11.884 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:57:12.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:57:13.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:12 smithi066 ceph-mon[85781]: pgmap v683: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:13.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:12 smithi001 ceph-mon[112940]: pgmap v683: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:15.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:14 smithi066 ceph-mon[85781]: pgmap v684: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:15.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:14 smithi001 ceph-mon[112940]: pgmap v684: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:17.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:16 smithi066 ceph-mon[85781]: pgmap v685: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:17.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:16 smithi001 ceph-mon[112940]: pgmap v685: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:19.006 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:18 smithi066 ceph-mon[85781]: pgmap v686: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:19.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:18 smithi001 ceph-mon[112940]: pgmap v686: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:21.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:20 smithi001 ceph-mon[112940]: pgmap v687: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:21.256 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:20 smithi066 ceph-mon[85781]: pgmap v687: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:22.028 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:21 smithi001 ceph-mon[112940]: pgmap v688: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:22.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:21 smithi066 ceph-mon[85781]: pgmap v688: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:23.934 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:23 smithi001 ceph-mon[112940]: pgmap v689: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:24.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:23 smithi066 ceph-mon[85781]: pgmap v689: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:26.188 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:25 smithi001 ceph-mon[112940]: pgmap v690: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:26.189 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:57:26.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.191+0000 7f1971c00700 1 -- 172.21.15.1:0/783148019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c1095f0 msgr2=0x7f196c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:26.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.191+0000 7f1971c00700 1 --2- 172.21.15.1:0/783148019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c1095f0 0x7f196c1099c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f195c004940 tx=0x7f195c009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.191+0000 7f1971c00700 1 -- 172.21.15.1:0/783148019 shutdown_connections 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.191+0000 7f1971c00700 1 --2- 172.21.15.1:0/783148019 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f196c103dc0 0x7f196c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.191+0000 7f1971c00700 1 --2- 172.21.15.1:0/783148019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c1095f0 0x7f196c1099c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.191+0000 7f1971c00700 1 -- 172.21.15.1:0/783148019 >> 172.21.15.1:0/783148019 conn(0x7f196c072d50 msgr2=0x7f196c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.191+0000 7f1971c00700 1 -- 172.21.15.1:0/783148019 shutdown_connections 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.191+0000 7f1971c00700 1 -- 172.21.15.1:0/783148019 wait complete. 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.192+0000 7f1971c00700 1 Processor -- start 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.192+0000 7f1971c00700 1 -- start start 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f1971c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c103dc0 0x7f196c11a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f1971c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f196c1095f0 0x7f196c11a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f1971c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c11ac70 con 0x7f196c103dc0 2024-02-28T16:57:26.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f1971c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c113ef0 con 0x7f196c1095f0 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f196aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c103dc0 0x7f196c11a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f196aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c103dc0 0x7f196c11a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44770/0 (socket says 172.21.15.1:44770) 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f196aa00700 1 -- 172.21.15.1:0/2764500480 learned_addr learned my addr 172.21.15.1:0/2764500480 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f196aa00700 1 -- 172.21.15.1:0/2764500480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f196c1095f0 msgr2=0x7f196c11a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f196a000700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f196c1095f0 0x7f196c11a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f196aa00700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f196c1095f0 0x7f196c11a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.193+0000 7f196aa00700 1 -- 172.21.15.1:0/2764500480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c005040 con 0x7f196c103dc0 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.194+0000 7f196aa00700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c103dc0 0x7f196c11a050 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f195c009150 tx=0x7f195c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.194+0000 7f1962a00700 1 -- 172.21.15.1:0/2764500480 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f195c00e040 con 0x7f196c103dc0 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.194+0000 7f1962a00700 1 -- 172.21.15.1:0/2764500480 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f195c00bbf0 con 0x7f196c103dc0 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.194+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196c114110 con 0x7f196c103dc0 2024-02-28T16:57:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.194+0000 7f1962a00700 1 -- 172.21.15.1:0/2764500480 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f195c01c630 con 0x7f196c103dc0 2024-02-28T16:57:26.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.194+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196c114650 con 0x7f196c103dc0 2024-02-28T16:57:26.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.195+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196c075870 con 0x7f196c103dc0 2024-02-28T16:57:26.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.196+0000 7f1962a00700 1 -- 172.21.15.1:0/2764500480 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f195c00bd60 con 0x7f196c103dc0 2024-02-28T16:57:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.197+0000 7f1962a00700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f195006d7f0 0x7f195006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.198+0000 7f1962a00700 1 -- 172.21.15.1:0/2764500480 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f195c08e2a0 con 0x7f196c103dc0 2024-02-28T16:57:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.198+0000 7f196a000700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f195006d7f0 0x7f195006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.199+0000 7f196a000700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f195006d7f0 0x7f195006fca0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1958006fd0 tx=0x7f1958008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.203+0000 7f1962a00700 1 -- 172.21.15.1:0/2764500480 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f195c05b680 con 0x7f196c103dc0 2024-02-28T16:57:26.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:25 smithi066 ceph-mon[85781]: pgmap v690: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:26.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:57:26.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.450+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f196c114980 con 0x7f195006d7f0 2024-02-28T16:57:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.452+0000 7f1962a00700 1 -- 172.21.15.1:0/2764500480 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f196c114980 con 0x7f195006d7f0 2024-02-28T16:57:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.455+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f195006d7f0 msgr2=0x7f195006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.455+0000 7f1971c00700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f195006d7f0 0x7f195006fca0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1958006fd0 tx=0x7f1958008040 comp rx=0 tx=0).stop 2024-02-28T16:57:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.455+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c103dc0 msgr2=0x7f196c11a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.455+0000 7f1971c00700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c103dc0 0x7f196c11a050 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f195c009150 tx=0x7f195c009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.456+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 shutdown_connections 2024-02-28T16:57:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.456+0000 7f1971c00700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f195006d7f0 0x7f195006fca0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.456+0000 7f1971c00700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f196c103dc0 0x7f196c11a050 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.456+0000 7f1971c00700 1 --2- 172.21.15.1:0/2764500480 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f196c1095f0 0x7f196c11a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.456+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 >> 172.21.15.1:0/2764500480 conn(0x7f196c072d50 msgr2=0x7f196c0746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.456+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 shutdown_connections 2024-02-28T16:57:26.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.456+0000 7f1971c00700 1 -- 172.21.15.1:0/2764500480 wait complete. 2024-02-28T16:57:26.467 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:57:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.614+0000 7fe312200700 1 -- 172.21.15.1:0/2509578128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fbb60 msgr2=0x7fe30c0fbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.614+0000 7fe312200700 1 --2- 172.21.15.1:0/2509578128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fbb60 0x7fe30c0fbf30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe300004940 tx=0x7fe300009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.614+0000 7fe312200700 1 -- 172.21.15.1:0/2509578128 shutdown_connections 2024-02-28T16:57:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.614+0000 7fe312200700 1 --2- 172.21.15.1:0/2509578128 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe30c0fc500 0x7fe30c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.614+0000 7fe312200700 1 --2- 172.21.15.1:0/2509578128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fbb60 0x7fe30c0fbf30 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.614+0000 7fe312200700 1 -- 172.21.15.1:0/2509578128 >> 172.21.15.1:0/2509578128 conn(0x7fe30c0f77c0 msgr2=0x7fe30c0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.615+0000 7fe312200700 1 -- 172.21.15.1:0/2509578128 shutdown_connections 2024-02-28T16:57:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.615+0000 7fe312200700 1 -- 172.21.15.1:0/2509578128 wait complete. 2024-02-28T16:57:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.616+0000 7fe312200700 1 Processor -- start 2024-02-28T16:57:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.616+0000 7fe312200700 1 -- start start 2024-02-28T16:57:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.616+0000 7fe312200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe30c0fbb60 0x7fe30c119eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.616+0000 7fe312200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fc500 0x7fe30c11a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.616+0000 7fe312200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c11aa40 con 0x7fe30c0fc500 2024-02-28T16:57:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.616+0000 7fe312200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c113d50 con 0x7fe30c0fbb60 2024-02-28T16:57:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.617+0000 7fe30be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fc500 0x7fe30c11a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.617+0000 7fe30be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fc500 0x7fe30c11a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44790/0 (socket says 172.21.15.1:44790) 2024-02-28T16:57:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.617+0000 7fe30be00700 1 -- 172.21.15.1:0/1859338321 learned_addr learned my addr 172.21.15.1:0/1859338321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:26.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.617+0000 7fe310e00700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe30c0fbb60 0x7fe30c119eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.617+0000 7fe30be00700 1 -- 172.21.15.1:0/1859338321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe30c0fbb60 msgr2=0x7fe30c119eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.617+0000 7fe30be00700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe30c0fbb60 0x7fe30c119eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.617+0000 7fe30be00700 1 -- 172.21.15.1:0/1859338321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe300005040 con 0x7fe30c0fc500 2024-02-28T16:57:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.618+0000 7fe30be00700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fc500 0x7fe30c11a3f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8002a50 tx=0x7fe2f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.618+0000 7fe308c00700 1 -- 172.21.15.1:0/1859338321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2f8007cc0 con 0x7fe30c0fc500 2024-02-28T16:57:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.618+0000 7fe308c00700 1 -- 172.21.15.1:0/1859338321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2f800f040 con 0x7fe30c0fc500 2024-02-28T16:57:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.618+0000 7fe308c00700 1 -- 172.21.15.1:0/1859338321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2f8009640 con 0x7fe30c0fc500 2024-02-28T16:57:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.618+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c114030 con 0x7fe30c0fc500 2024-02-28T16:57:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.618+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30c114570 con 0x7fe30c0fc500 2024-02-28T16:57:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.619+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30c0fe240 con 0x7fe30c0fc500 2024-02-28T16:57:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.620+0000 7fe308c00700 1 -- 172.21.15.1:0/1859338321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe2f80097a0 con 0x7fe30c0fc500 2024-02-28T16:57:26.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.622+0000 7fe308c00700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe2f406d7f0 0x7fe2f406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:26.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.622+0000 7fe308c00700 1 -- 172.21.15.1:0/1859338321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe2f808cc60 con 0x7fe30c0fc500 2024-02-28T16:57:26.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.622+0000 7fe310e00700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe2f406d7f0 0x7fe2f406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:26.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.623+0000 7fe310e00700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe2f406d7f0 0x7fe2f406fca0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe30000b3b0 tx=0x7fe300009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:26.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.626+0000 7fe308c00700 1 -- 172.21.15.1:0/1859338321 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe2f8059fc0 con 0x7fe30c0fc500 2024-02-28T16:57:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.876+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe30c115490 con 0x7fe2f406d7f0 2024-02-28T16:57:26.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.878+0000 7fe308c00700 1 -- 172.21.15.1:0/1859338321 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe30c115490 con 0x7fe2f406d7f0 2024-02-28T16:57:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.881+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe2f406d7f0 msgr2=0x7fe2f406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:26.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.881+0000 7fe312200700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe2f406d7f0 0x7fe2f406fca0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe30000b3b0 tx=0x7fe300009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.881+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fc500 msgr2=0x7fe30c11a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.881+0000 7fe312200700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fc500 0x7fe30c11a3f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8002a50 tx=0x7fe2f80046b0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.882+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 shutdown_connections 2024-02-28T16:57:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.882+0000 7fe312200700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe2f406d7f0 0x7fe2f406fca0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.882+0000 7fe312200700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe30c0fbb60 0x7fe30c119eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.882+0000 7fe312200700 1 --2- 172.21.15.1:0/1859338321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe30c0fc500 0x7fe30c11a3f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.882+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 >> 172.21.15.1:0/1859338321 conn(0x7fe30c0f77c0 msgr2=0x7fe30c10d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.882+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 shutdown_connections 2024-02-28T16:57:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:26.882+0000 7fe312200700 1 -- 172.21.15.1:0/1859338321 wait complete. 2024-02-28T16:57:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.036+0000 7fc129800700 1 -- 172.21.15.1:0/4046359318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 msgr2=0x7fc12410d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.036+0000 7fc129800700 1 --2- 172.21.15.1:0/4046359318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 0x7fc12410d740 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc10c004940 tx=0x7fc10c009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.036+0000 7fc129800700 1 -- 172.21.15.1:0/4046359318 shutdown_connections 2024-02-28T16:57:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.036+0000 7fc129800700 1 --2- 172.21.15.1:0/4046359318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 0x7fc12410d740 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.036+0000 7fc129800700 1 --2- 172.21.15.1:0/4046359318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1240724c0 0x7fc124072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.036+0000 7fc129800700 1 -- 172.21.15.1:0/4046359318 >> 172.21.15.1:0/4046359318 conn(0x7fc1240faae0 msgr2=0x7fc1240fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:27.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.036+0000 7fc129800700 1 -- 172.21.15.1:0/4046359318 shutdown_connections 2024-02-28T16:57:27.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.036+0000 7fc129800700 1 -- 172.21.15.1:0/4046359318 wait complete. 2024-02-28T16:57:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.037+0000 7fc129800700 1 Processor -- start 2024-02-28T16:57:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.037+0000 7fc129800700 1 -- start start 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.037+0000 7fc129800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1240724c0 0x7fc124119e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.037+0000 7fc129800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 0x7fc12411a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.037+0000 7fc129800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12411aa40 con 0x7fc124072dd0 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.037+0000 7fc129800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124113cc0 con 0x7fc1240724c0 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.038+0000 7fc11be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 0x7fc12411a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.038+0000 7fc11be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 0x7fc12411a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44802/0 (socket says 172.21.15.1:44802) 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.038+0000 7fc11be00700 1 -- 172.21.15.1:0/1847233394 learned_addr learned my addr 172.21.15.1:0/1847233394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.038+0000 7fc123e00700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1240724c0 0x7fc124119e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.038+0000 7fc11be00700 1 -- 172.21.15.1:0/1847233394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1240724c0 msgr2=0x7fc124119e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.038+0000 7fc11be00700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1240724c0 0x7fc124119e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.038+0000 7fc11be00700 1 -- 172.21.15.1:0/1847233394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c005040 con 0x7fc124072dd0 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.038+0000 7fc11be00700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 0x7fc12411a360 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009150 tx=0x7fc10c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.039+0000 7fc120c00700 1 -- 172.21.15.1:0/1847233394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc10c00e040 con 0x7fc124072dd0 2024-02-28T16:57:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.039+0000 7fc120c00700 1 -- 172.21.15.1:0/1847233394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc10c00bbf0 con 0x7fc124072dd0 2024-02-28T16:57:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.039+0000 7fc120c00700 1 -- 172.21.15.1:0/1847233394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc10c01c630 con 0x7fc124072dd0 2024-02-28T16:57:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.039+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc124113ee0 con 0x7fc124072dd0 2024-02-28T16:57:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.039+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1241144a0 con 0x7fc124072dd0 2024-02-28T16:57:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.040+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12410ae70 con 0x7fc124072dd0 2024-02-28T16:57:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.041+0000 7fc120c00700 1 -- 172.21.15.1:0/1847233394 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc10c00bd60 con 0x7fc124072dd0 2024-02-28T16:57:27.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.042+0000 7fc120c00700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc10806d7f0 0x7fc10806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.042+0000 7fc120c00700 1 -- 172.21.15.1:0/1847233394 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc10c08e2c0 con 0x7fc124072dd0 2024-02-28T16:57:27.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.042+0000 7fc123e00700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc10806d7f0 0x7fc10806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.043+0000 7fc123e00700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc10806d7f0 0x7fc10806fca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc114006fd0 tx=0x7fc114008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:27.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.047+0000 7fc120c00700 1 -- 172.21.15.1:0/1847233394 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc10c05b6a0 con 0x7fc124072dd0 2024-02-28T16:57:27.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:26 smithi066 ceph-mon[85781]: from='client.35262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:57:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.276+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc124115400 con 0x7fc10806d7f0 2024-02-28T16:57:27.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:26 smithi001 ceph-mon[112940]: from='client.35262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:57:27.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.309+0000 7fc120c00700 1 -- 172.21.15.1:0/1847233394 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc124115400 con 0x7fc10806d7f0 2024-02-28T16:57:27.306 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:57:27.306 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (23m) 19s ago 37m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:57:27.306 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (20m) 19s ago 38m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (20m) 21s ago 36m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (23m) 19s ago 37m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (16m) 19s ago 30m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (15m) 19s ago 30m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (30m) 21s ago 30m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (27s) 21s ago 30m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (26m) 19s ago 39m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (25m) 21s ago 36m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (21m) 19s ago 39m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (21m) 21s ago 36m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (24m) 19s ago 38m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (24m) 21s ago 36m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (19m) 19s ago 35m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:57:27.307 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (19m) 19s ago 34m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:57:27.308 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (18m) 19s ago 34m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:57:27.308 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (18m) 21s ago 33m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:57:27.308 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (17m) 21s ago 33m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:57:27.308 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (16m) 21s ago 32m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:57:27.308 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (24m) 19s ago 37m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:57:27.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.311+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc10806d7f0 msgr2=0x7fc10806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.311+0000 7fc129800700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc10806d7f0 0x7fc10806fca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc114006fd0 tx=0x7fc114008040 comp rx=0 tx=0).stop 2024-02-28T16:57:27.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 msgr2=0x7fc12411a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 0x7fc12411a360 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009150 tx=0x7fc10c009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:27.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 shutdown_connections 2024-02-28T16:57:27.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc10806d7f0 0x7fc10806fca0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc1240724c0 0x7fc124119e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 --2- 172.21.15.1:0/1847233394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc124072dd0 0x7fc12411a360 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 >> 172.21.15.1:0/1847233394 conn(0x7fc1240faae0 msgr2=0x7fc1240fb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:27.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 shutdown_connections 2024-02-28T16:57:27.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.312+0000 7fc129800700 1 -- 172.21.15.1:0/1847233394 wait complete. 2024-02-28T16:57:27.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.467+0000 7f5444400700 1 -- 172.21.15.1:0/3508140621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 msgr2=0x7f543c070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.467+0000 7f5444400700 1 --2- 172.21.15.1:0/3508140621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 0x7f543c070330 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5430004970 tx=0x7f5430009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:27.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.468+0000 7f5444400700 1 -- 172.21.15.1:0/3508140621 shutdown_connections 2024-02-28T16:57:27.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.468+0000 7f5444400700 1 --2- 172.21.15.1:0/3508140621 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f543c070870 0x7f543c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.468+0000 7f5444400700 1 --2- 172.21.15.1:0/3508140621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 0x7f543c070330 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.468+0000 7f5444400700 1 -- 172.21.15.1:0/3508140621 >> 172.21.15.1:0/3508140621 conn(0x7f543c0f8ac0 msgr2=0x7f543c0faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:27.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.468+0000 7f5444400700 1 -- 172.21.15.1:0/3508140621 shutdown_connections 2024-02-28T16:57:27.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.468+0000 7f5444400700 1 -- 172.21.15.1:0/3508140621 wait complete. 2024-02-28T16:57:27.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.469+0000 7f5444400700 1 Processor -- start 2024-02-28T16:57:27.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.469+0000 7f5444400700 1 -- start start 2024-02-28T16:57:27.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.469+0000 7f5444400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f543c070870 0x7f543c11a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.469+0000 7f5444400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 0x7f543c11a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.469+0000 7f5444400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c11ac30 con 0x7f543c071f60 2024-02-28T16:57:27.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.469+0000 7f5444400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c113f80 con 0x7f543c070870 2024-02-28T16:57:27.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.469+0000 7f5441000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 0x7f543c11a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5441000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 0x7f543c11a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44824/0 (socket says 172.21.15.1:44824) 2024-02-28T16:57:27.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5441000700 1 -- 172.21.15.1:0/4235939016 learned_addr learned my addr 172.21.15.1:0/4235939016 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:27.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5441a00700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f543c070870 0x7f543c11a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5441000700 1 -- 172.21.15.1:0/4235939016 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f543c070870 msgr2=0x7f543c11a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5441000700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f543c070870 0x7f543c11a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5441000700 1 -- 172.21.15.1:0/4235939016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5430005040 con 0x7f543c071f60 2024-02-28T16:57:27.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5441a00700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f543c070870 0x7f543c11a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:57:27.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5441000700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 0x7f543c11a5e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f542800c830 tx=0x7f5428004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:27.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.470+0000 7f5439600700 1 -- 172.21.15.1:0/4235939016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f542800a760 con 0x7f543c071f60 2024-02-28T16:57:27.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.471+0000 7f5439600700 1 -- 172.21.15.1:0/4235939016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f542800ad90 con 0x7f543c071f60 2024-02-28T16:57:27.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.471+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c114260 con 0x7f543c071f60 2024-02-28T16:57:27.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.471+0000 7f5439600700 1 -- 172.21.15.1:0/4235939016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54280113a0 con 0x7f543c071f60 2024-02-28T16:57:27.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.471+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c1147a0 con 0x7f543c071f60 2024-02-28T16:57:27.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.472+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c0fe400 con 0x7f543c071f60 2024-02-28T16:57:27.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.473+0000 7f5439600700 1 -- 172.21.15.1:0/4235939016 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5428012400 con 0x7f543c071f60 2024-02-28T16:57:27.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.475+0000 7f5439600700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f542406d7f0 0x7f542406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.475+0000 7f5439600700 1 -- 172.21.15.1:0/4235939016 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f542808ce40 con 0x7f543c071f60 2024-02-28T16:57:27.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.476+0000 7f5441a00700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f542406d7f0 0x7f542406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.476+0000 7f5441a00700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f542406d7f0 0x7f542406fca0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5430000c00 tx=0x7f5430009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:27.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.480+0000 7f5439600700 1 -- 172.21.15.1:0/4235939016 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f54280567b0 con 0x7f543c071f60 2024-02-28T16:57:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.794+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f543c04bed0 con 0x7f543c071f60 2024-02-28T16:57:27.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.795+0000 7f5439600700 1 -- 172.21.15.1:0/4235939016 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f5428059e70 con 0x7f543c071f60 2024-02-28T16:57:27.793 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:57:27.793 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:57:27.793 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:57:27.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:57:27.793 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:57:27.793 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:57:27.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:57:27.794 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T16:57:27.795 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:57:27.795 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:57:27.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.797+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f542406d7f0 msgr2=0x7f542406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.797+0000 7f5444400700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f542406d7f0 0x7f542406fca0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5430000c00 tx=0x7f5430009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:27.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.797+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 msgr2=0x7f543c11a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.797+0000 7f5444400700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 0x7f543c11a5e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f542800c830 tx=0x7f5428004470 comp rx=0 tx=0).stop 2024-02-28T16:57:27.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.798+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 shutdown_connections 2024-02-28T16:57:27.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.798+0000 7f5444400700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f542406d7f0 0x7f542406fca0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.798+0000 7f5444400700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f543c070870 0x7f543c11a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.798+0000 7f5444400700 1 --2- 172.21.15.1:0/4235939016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f543c071f60 0x7f543c11a5e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.798+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 >> 172.21.15.1:0/4235939016 conn(0x7f543c0f8ac0 msgr2=0x7f543c0fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:27.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.798+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 shutdown_connections 2024-02-28T16:57:27.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.798+0000 7f5444400700 1 -- 172.21.15.1:0/4235939016 wait complete. 2024-02-28T16:57:27.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.950+0000 7f855da00700 1 -- 172.21.15.1:0/1723186223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 msgr2=0x7f85580fef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.950+0000 7f855da00700 1 --2- 172.21.15.1:0/1723186223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 0x7f85580fef80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8544004940 tx=0x7f8544009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:27.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.950+0000 7f855da00700 1 -- 172.21.15.1:0/1723186223 shutdown_connections 2024-02-28T16:57:27.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.950+0000 7f855da00700 1 --2- 172.21.15.1:0/1723186223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 0x7f85580fef80 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.950+0000 7f855da00700 1 --2- 172.21.15.1:0/1723186223 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8558104f70 0x7f8558105340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.950+0000 7f855da00700 1 -- 172.21.15.1:0/1723186223 >> 172.21.15.1:0/1723186223 conn(0x7f85580faae0 msgr2=0x7f85580fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:27.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.950+0000 7f855da00700 1 -- 172.21.15.1:0/1723186223 shutdown_connections 2024-02-28T16:57:27.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.950+0000 7f855da00700 1 -- 172.21.15.1:0/1723186223 wait complete. 2024-02-28T16:57:27.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.951+0000 7f855da00700 1 Processor -- start 2024-02-28T16:57:27.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.951+0000 7f855da00700 1 -- start start 2024-02-28T16:57:27.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.952+0000 7f855da00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8558104f70 0x7f8558115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.952+0000 7f855da00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 0x7f8558115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.952+0000 7f855da00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85581165a0 con 0x7f8558105910 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.952+0000 7f855da00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855810f8d0 con 0x7f8558104f70 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.952+0000 7f8557400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 0x7f8558115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.952+0000 7f8557400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 0x7f8558115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44846/0 (socket says 172.21.15.1:44846) 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.952+0000 7f8557400700 1 -- 172.21.15.1:0/2395902087 learned_addr learned my addr 172.21.15.1:0/2395902087 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.952+0000 7f8557e00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8558104f70 0x7f8558115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.953+0000 7f8557400700 1 -- 172.21.15.1:0/2395902087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8558104f70 msgr2=0x7f8558115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.953+0000 7f8557400700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8558104f70 0x7f8558115a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:27.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.953+0000 7f8557400700 1 -- 172.21.15.1:0/2395902087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8544005040 con 0x7f8558105910 2024-02-28T16:57:27.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.953+0000 7f8557400700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 0x7f8558115f50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8544009030 tx=0x7f8544009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:27.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.953+0000 7f854fe00700 1 -- 172.21.15.1:0/2395902087 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f854400e040 con 0x7f8558105910 2024-02-28T16:57:27.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.953+0000 7f854fe00700 1 -- 172.21.15.1:0/2395902087 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f854400bbf0 con 0x7f8558105910 2024-02-28T16:57:27.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.953+0000 7f854fe00700 1 -- 172.21.15.1:0/2395902087 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f854401c630 con 0x7f8558105910 2024-02-28T16:57:27.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.953+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855810fb50 con 0x7f8558105910 2024-02-28T16:57:27.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.954+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8558110110 con 0x7f8558105910 2024-02-28T16:57:27.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.955+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855805f5c0 con 0x7f8558105910 2024-02-28T16:57:27.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.956+0000 7f854fe00700 1 -- 172.21.15.1:0/2395902087 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f854400bd60 con 0x7f8558105910 2024-02-28T16:57:27.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.957+0000 7f854fe00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853c06d7f0 0x7f853c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:27.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.957+0000 7f854fe00700 1 -- 172.21.15.1:0/2395902087 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f854408e1f0 con 0x7f8558105910 2024-02-28T16:57:27.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.957+0000 7f8557e00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853c06d7f0 0x7f853c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:27.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.958+0000 7f8557e00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853c06d7f0 0x7f853c06fca0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8548007fd0 tx=0x7f8548009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:27.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:27.962+0000 7f854fe00700 1 -- 172.21.15.1:0/2395902087 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f854405b5d0 con 0x7f8558105910 2024-02-28T16:57:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.218+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f855804bed0 con 0x7f8558105910 2024-02-28T16:57:28.216 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:27 smithi001 ceph-mon[112940]: pgmap v691: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:28.216 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:27 smithi001 ceph-mon[112940]: from='client.35266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:57:28.216 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:27 smithi001 ceph-mon[112940]: from='client.35270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:57:28.217 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:27 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4235939016' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:57:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.219+0000 7f854fe00700 1 -- 172.21.15.1:0/2395902087 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f854405b160 con 0x7f8558105910 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:57:28.218 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:57:28.219 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:57:28.220 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:57:28.221 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:57:28.221 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:57:28.221 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:57:28.221 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:57:28.221 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:57:28.221 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:57:28.221 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:57:28.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853c06d7f0 msgr2=0x7f853c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:28.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853c06d7f0 0x7f853c06fca0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8548007fd0 tx=0x7f8548009040 comp rx=0 tx=0).stop 2024-02-28T16:57:28.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 msgr2=0x7f8558115f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:28.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 0x7f8558115f50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8544009030 tx=0x7f8544009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:28.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 shutdown_connections 2024-02-28T16:57:28.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f853c06d7f0 0x7f853c06fca0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8558104f70 0x7f8558115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 --2- 172.21.15.1:0/2395902087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8558105910 0x7f8558115f50 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 >> 172.21.15.1:0/2395902087 conn(0x7f85580faae0 msgr2=0x7f85580fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:28.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 shutdown_connections 2024-02-28T16:57:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.222+0000 7f855da00700 1 -- 172.21.15.1:0/2395902087 wait complete. 2024-02-28T16:57:28.224 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T16:57:28.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:27 smithi066 ceph-mon[85781]: pgmap v691: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:28.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:27 smithi066 ceph-mon[85781]: from='client.35266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:57:28.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:27 smithi066 ceph-mon[85781]: from='client.35270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:57:28.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:27 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4235939016' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:57:28.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.367+0000 7fe435a00700 1 -- 172.21.15.1:0/3370255056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105b10 msgr2=0x7fe4300ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:28.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.367+0000 7fe435a00700 1 --2- 172.21.15.1:0/3370255056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105b10 0x7fe4300ff180 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe41c004940 tx=0x7fe41c009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:28.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.367+0000 7fe435a00700 1 -- 172.21.15.1:0/3370255056 shutdown_connections 2024-02-28T16:57:28.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.367+0000 7fe435a00700 1 --2- 172.21.15.1:0/3370255056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105b10 0x7fe4300ff180 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.367+0000 7fe435a00700 1 --2- 172.21.15.1:0/3370255056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe430105170 0x7fe430105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.367+0000 7fe435a00700 1 -- 172.21.15.1:0/3370255056 >> 172.21.15.1:0/3370255056 conn(0x7fe4300face0 msgr2=0x7fe4300fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:28.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.367+0000 7fe435a00700 1 -- 172.21.15.1:0/3370255056 shutdown_connections 2024-02-28T16:57:28.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.367+0000 7fe435a00700 1 -- 172.21.15.1:0/3370255056 wait complete. 2024-02-28T16:57:28.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.368+0000 7fe435a00700 1 Processor -- start 2024-02-28T16:57:28.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.368+0000 7fe435a00700 1 -- start start 2024-02-28T16:57:28.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.368+0000 7fe435a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105170 0x7fe430115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:28.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.368+0000 7fe435a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe430105b10 0x7fe430116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:28.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.368+0000 7fe435a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe430116870 con 0x7fe430105170 2024-02-28T16:57:28.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.368+0000 7fe435a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43010fb80 con 0x7fe430105b10 2024-02-28T16:57:28.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.369+0000 7fe42ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105170 0x7fe430115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:28.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.369+0000 7fe42ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105170 0x7fe430115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44866/0 (socket says 172.21.15.1:44866) 2024-02-28T16:57:28.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.369+0000 7fe42ea00700 1 -- 172.21.15.1:0/2055376167 learned_addr learned my addr 172.21.15.1:0/2055376167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:28.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.369+0000 7fe42e000700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe430105b10 0x7fe430116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:28.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.369+0000 7fe42ea00700 1 -- 172.21.15.1:0/2055376167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe430105b10 msgr2=0x7fe430116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:28.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.369+0000 7fe42ea00700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe430105b10 0x7fe430116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.369+0000 7fe42ea00700 1 -- 172.21.15.1:0/2055376167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c005040 con 0x7fe430105170 2024-02-28T16:57:28.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.370+0000 7fe42ea00700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105170 0x7fe430115ce0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe42000c830 tx=0x7fe420004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:28.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.370+0000 7fe426a00700 1 -- 172.21.15.1:0/2055376167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe42000a760 con 0x7fe430105170 2024-02-28T16:57:28.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.370+0000 7fe426a00700 1 -- 172.21.15.1:0/2055376167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe42000ad90 con 0x7fe430105170 2024-02-28T16:57:28.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.370+0000 7fe426a00700 1 -- 172.21.15.1:0/2055376167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4200113a0 con 0x7fe430105170 2024-02-28T16:57:28.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.370+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43010fe60 con 0x7fe430105170 2024-02-28T16:57:28.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.370+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4301103a0 con 0x7fe430105170 2024-02-28T16:57:28.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.371+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe430107640 con 0x7fe430105170 2024-02-28T16:57:28.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.372+0000 7fe426a00700 1 -- 172.21.15.1:0/2055376167 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe420012400 con 0x7fe430105170 2024-02-28T16:57:28.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.373+0000 7fe426a00700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe41406d7f0 0x7fe41406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:28.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.373+0000 7fe426a00700 1 -- 172.21.15.1:0/2055376167 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe42008cf00 con 0x7fe430105170 2024-02-28T16:57:28.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.373+0000 7fe42e000700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe41406d7f0 0x7fe41406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:28.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.374+0000 7fe42e000700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe41406d7f0 0x7fe41406fca0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe430111580 tx=0x7fe41c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:28.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.378+0000 7fe426a00700 1 -- 172.21.15.1:0/2055376167 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe42005a2e0 con 0x7fe430105170 2024-02-28T16:57:28.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.613+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe43004bf40 con 0x7fe41406d7f0 2024-02-28T16:57:28.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.615+0000 7fe426a00700 1 -- 172.21.15.1:0/2055376167 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe43004bf40 con 0x7fe41406d7f0 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:57:28.612 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T16:57:28.613 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:57:28.613 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:57:28.613 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:57:28.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe41406d7f0 msgr2=0x7fe41406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:28.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe41406d7f0 0x7fe41406fca0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe430111580 tx=0x7fe41c009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:28.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105170 msgr2=0x7fe430115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:28.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105170 0x7fe430115ce0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe42000c830 tx=0x7fe420004470 comp rx=0 tx=0).stop 2024-02-28T16:57:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 shutdown_connections 2024-02-28T16:57:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe41406d7f0 0x7fe41406fca0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe430105170 0x7fe430115ce0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 --2- 172.21.15.1:0/2055376167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe430105b10 0x7fe430116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 >> 172.21.15.1:0/2055376167 conn(0x7fe4300face0 msgr2=0x7fe4300fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.618+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 shutdown_connections 2024-02-28T16:57:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.619+0000 7fe435a00700 1 -- 172.21.15.1:0/2055376167 wait complete. 2024-02-28T16:57:28.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.773+0000 7f8b6e600700 1 -- 172.21.15.1:0/2558393313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 msgr2=0x7f8b68072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.773+0000 7f8b6e600700 1 --2- 172.21.15.1:0/2558393313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 0x7f8b68072890 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8b58004940 tx=0x7f8b58009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.773+0000 7f8b6e600700 1 -- 172.21.15.1:0/2558393313 shutdown_connections 2024-02-28T16:57:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.773+0000 7f8b6e600700 1 --2- 172.21.15.1:0/2558393313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b68072dd0 0x7f8b6810d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.773+0000 7f8b6e600700 1 --2- 172.21.15.1:0/2558393313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 0x7f8b68072890 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.773+0000 7f8b6e600700 1 -- 172.21.15.1:0/2558393313 >> 172.21.15.1:0/2558393313 conn(0x7f8b680face0 msgr2=0x7f8b680fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.773+0000 7f8b6e600700 1 -- 172.21.15.1:0/2558393313 shutdown_connections 2024-02-28T16:57:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.773+0000 7f8b6e600700 1 -- 172.21.15.1:0/2558393313 wait complete. 2024-02-28T16:57:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.774+0000 7f8b6e600700 1 Processor -- start 2024-02-28T16:57:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.775+0000 7f8b6e600700 1 -- start start 2024-02-28T16:57:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.775+0000 7f8b6e600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 0x7f8b6811a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.775+0000 7f8b6e600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b68072dd0 0x7f8b6811a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.775+0000 7f8b6e600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6811ac50 con 0x7f8b680724c0 2024-02-28T16:57:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.775+0000 7f8b6e600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b68113f60 con 0x7f8b68072dd0 2024-02-28T16:57:28.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.775+0000 7f8b67400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 0x7f8b6811a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:28.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.775+0000 7f8b67400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 0x7f8b6811a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44872/0 (socket says 172.21.15.1:44872) 2024-02-28T16:57:28.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.775+0000 7f8b67400700 1 -- 172.21.15.1:0/80211868 learned_addr learned my addr 172.21.15.1:0/80211868 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:28.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.776+0000 7f8b66a00700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b68072dd0 0x7f8b6811a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:28.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.776+0000 7f8b67400700 1 -- 172.21.15.1:0/80211868 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b68072dd0 msgr2=0x7f8b6811a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:28.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.776+0000 7f8b67400700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b68072dd0 0x7f8b6811a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:28.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.776+0000 7f8b67400700 1 -- 172.21.15.1:0/80211868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b58005040 con 0x7f8b680724c0 2024-02-28T16:57:28.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.776+0000 7f8b67400700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 0x7f8b6811a0c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009180 tx=0x7f8b58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:28.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.776+0000 7f8b5f400700 1 -- 172.21.15.1:0/80211868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5800e040 con 0x7f8b680724c0 2024-02-28T16:57:28.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.776+0000 7f8b5f400700 1 -- 172.21.15.1:0/80211868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b58004290 con 0x7f8b680724c0 2024-02-28T16:57:28.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.776+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b681141e0 con 0x7f8b680724c0 2024-02-28T16:57:28.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.777+0000 7f8b5f400700 1 -- 172.21.15.1:0/80211868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5801c630 con 0x7f8b680724c0 2024-02-28T16:57:28.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.777+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b68114720 con 0x7f8b680724c0 2024-02-28T16:57:28.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.778+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6810b010 con 0x7f8b680724c0 2024-02-28T16:57:28.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.779+0000 7f8b5f400700 1 -- 172.21.15.1:0/80211868 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8b580139f0 con 0x7f8b680724c0 2024-02-28T16:57:28.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.780+0000 7f8b5f400700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b4c06d7f0 0x7f8b4c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:28.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.780+0000 7f8b5f400700 1 -- 172.21.15.1:0/80211868 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8b5808e3e0 con 0x7f8b680724c0 2024-02-28T16:57:28.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.780+0000 7f8b66a00700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b4c06d7f0 0x7f8b4c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:28.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.781+0000 7f8b66a00700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b4c06d7f0 0x7f8b4c06fca0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8b68115960 tx=0x7f8b5400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:28.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:28.785+0000 7f8b5f400700 1 -- 172.21.15.1:0/80211868 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8b5805b7e0 con 0x7f8b680724c0 2024-02-28T16:57:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.089+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8b6804bed0 con 0x7f8b680724c0 2024-02-28T16:57:29.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.089+0000 7f8b5f400700 1 -- 172.21.15.1:0/80211868 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f8b5805b370 con 0x7f8b680724c0 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T16:57:29.087 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:57:29.088 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:57:29.089 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:57:29.089 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:57:29.089 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b4c06d7f0 msgr2=0x7f8b4c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b4c06d7f0 0x7f8b4c06fca0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8b68115960 tx=0x7f8b5400b040 comp rx=0 tx=0).stop 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 msgr2=0x7f8b6811a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 0x7f8b6811a0c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009180 tx=0x7f8b58009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 shutdown_connections 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b4c06d7f0 0x7f8b4c06fca0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b680724c0 0x7f8b6811a0c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 --2- 172.21.15.1:0/80211868 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b68072dd0 0x7f8b6811a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 >> 172.21.15.1:0/80211868 conn(0x7f8b680face0 msgr2=0x7f8b680fb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:29.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 shutdown_connections 2024-02-28T16:57:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:29.093+0000 7f8b6e600700 1 -- 172.21.15.1:0/80211868 wait complete. 2024-02-28T16:57:29.256 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:28 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2395902087' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:57:29.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:28 smithi066 ceph-mon[85781]: from='client.35282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:57:29.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:28 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2395902087' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:57:29.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:28 smithi001 ceph-mon[112940]: from='client.35282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:57:30.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:29 smithi066 ceph-mon[85781]: pgmap v692: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:30.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:29 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/80211868' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:57:30.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:29 smithi001 ceph-mon[112940]: pgmap v692: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:30.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:29 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/80211868' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:57:32.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:31 smithi066 ceph-mon[85781]: pgmap v693: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:32.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:31 smithi001 ceph-mon[112940]: pgmap v693: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:34.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:33 smithi066 ceph-mon[85781]: pgmap v694: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:34.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:33 smithi001 ceph-mon[112940]: pgmap v694: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:36.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:35 smithi066 ceph-mon[85781]: pgmap v695: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:36.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:35 smithi001 ceph-mon[112940]: pgmap v695: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:38.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:37 smithi066 ceph-mon[85781]: pgmap v696: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:38.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:37 smithi001 ceph-mon[112940]: pgmap v696: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:40.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:39 smithi066 ceph-mon[85781]: pgmap v697: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:40.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:39 smithi001 ceph-mon[112940]: pgmap v697: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:41.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:57:41.591 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:57:42.007 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:41 smithi066 ceph-mon[85781]: pgmap v698: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:42.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:41 smithi001 ceph-mon[112940]: pgmap v698: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:44.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:43 smithi066 ceph-mon[85781]: pgmap v699: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:44.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:43 smithi001 ceph-mon[112940]: pgmap v699: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:46.257 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:45 smithi066 ceph-mon[85781]: pgmap v700: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:46.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:45 smithi001 ceph-mon[112940]: pgmap v700: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:48.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:48 smithi066 ceph-mon[85781]: pgmap v701: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:48.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:48 smithi001 ceph-mon[112940]: pgmap v701: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:50.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:50 smithi066 ceph-mon[85781]: pgmap v702: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:50.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:50 smithi001 ceph-mon[112940]: pgmap v702: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:52.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:52 smithi066 ceph-mon[85781]: pgmap v703: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:52.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:52 smithi001 ceph-mon[112940]: pgmap v703: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:54.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:54 smithi066 ceph-mon[85781]: pgmap v704: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:54.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:54 smithi001 ceph-mon[112940]: pgmap v704: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:56.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:56 smithi066 ceph-mon[85781]: pgmap v705: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:56.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:57:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:56 smithi001 ceph-mon[112940]: pgmap v705: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:56.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:57:58.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:57:58 smithi066 ceph-mon[85781]: pgmap v706: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:58.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:57:58 smithi001 ceph-mon[112940]: pgmap v706: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:57:59.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.263+0000 7f60cbc00700 1 -- 172.21.15.1:0/4040707842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c4102f50 msgr2=0x7f60c4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:59.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.263+0000 7f60cbc00700 1 --2- 172.21.15.1:0/4040707842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c4102f50 0x7f60c4103320 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009ce0 tx=0x7f60b8004d10 comp rx=0 tx=0).stop 2024-02-28T16:57:59.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.264+0000 7f60cbc00700 1 -- 172.21.15.1:0/4040707842 shutdown_connections 2024-02-28T16:57:59.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.264+0000 7f60cbc00700 1 --2- 172.21.15.1:0/4040707842 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60c41038f0 0x7f60c40fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.264+0000 7f60cbc00700 1 --2- 172.21.15.1:0/4040707842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c4102f50 0x7f60c4103320 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.264+0000 7f60cbc00700 1 -- 172.21.15.1:0/4040707842 >> 172.21.15.1:0/4040707842 conn(0x7f60c40f8ae0 msgr2=0x7f60c40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:59.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.264+0000 7f60cbc00700 1 -- 172.21.15.1:0/4040707842 shutdown_connections 2024-02-28T16:57:59.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.264+0000 7f60cbc00700 1 -- 172.21.15.1:0/4040707842 wait complete. 2024-02-28T16:57:59.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.265+0000 7f60cbc00700 1 Processor -- start 2024-02-28T16:57:59.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.265+0000 7f60cbc00700 1 -- start start 2024-02-28T16:57:59.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.265+0000 7f60cbc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c41038f0 0x7f60c4115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:59.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.265+0000 7f60cbc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60c4116530 0x7f60c410fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.265+0000 7f60cbc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c4116ab0 con 0x7f60c41038f0 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.265+0000 7f60cbc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c4116c20 con 0x7f60c4116530 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c9200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c41038f0 0x7f60c4115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c9200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c41038f0 0x7f60c4115ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46202/0 (socket says 172.21.15.1:46202) 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c9200700 1 -- 172.21.15.1:0/3877420964 learned_addr learned my addr 172.21.15.1:0/3877420964 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c3e00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60c4116530 0x7f60c410fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c9200700 1 -- 172.21.15.1:0/3877420964 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60c4116530 msgr2=0x7f60c410fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c9200700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60c4116530 0x7f60c410fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c9200700 1 -- 172.21.15.1:0/3877420964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b8009940 con 0x7f60c41038f0 2024-02-28T16:57:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c3e00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60c4116530 0x7f60c410fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:57:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.266+0000 7f60c9200700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c41038f0 0x7f60c4115ff0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f60b8015040 tx=0x7f60b8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.267+0000 7f60c0c00700 1 -- 172.21.15.1:0/3877420964 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60b800e040 con 0x7f60c41038f0 2024-02-28T16:57:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.267+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c4110490 con 0x7f60c41038f0 2024-02-28T16:57:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.267+0000 7f60c0c00700 1 -- 172.21.15.1:0/3877420964 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60b80049b0 con 0x7f60c41038f0 2024-02-28T16:57:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.267+0000 7f60c0c00700 1 -- 172.21.15.1:0/3877420964 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60b801c620 con 0x7f60c41038f0 2024-02-28T16:57:59.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.267+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c41109d0 con 0x7f60c41038f0 2024-02-28T16:57:59.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.268+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c4110cb0 con 0x7f60c41038f0 2024-02-28T16:57:59.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.269+0000 7f60c0c00700 1 -- 172.21.15.1:0/3877420964 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f60b8012a50 con 0x7f60c41038f0 2024-02-28T16:57:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.270+0000 7f60c0c00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60ac06d7f0 0x7f60ac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.270+0000 7f60c0c00700 1 -- 172.21.15.1:0/3877420964 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f60b808e990 con 0x7f60c41038f0 2024-02-28T16:57:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.270+0000 7f60c3e00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60ac06d7f0 0x7f60ac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.271+0000 7f60c3e00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60ac06d7f0 0x7f60ac06fca0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009fd0 tx=0x7f60b000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.275+0000 7f60c0c00700 1 -- 172.21.15.1:0/3877420964 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f60b805bc70 con 0x7f60c41038f0 2024-02-28T16:57:59.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.519+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60c410a0f0 con 0x7f60ac06d7f0 2024-02-28T16:57:59.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.521+0000 7f60c0c00700 1 -- 172.21.15.1:0/3877420964 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f60b000f040 con 0x7f60ac06d7f0 2024-02-28T16:57:59.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60ac06d7f0 msgr2=0x7f60ac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60ac06d7f0 0x7f60ac06fca0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009fd0 tx=0x7f60b000b040 comp rx=0 tx=0).stop 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c41038f0 msgr2=0x7f60c4115ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c41038f0 0x7f60c4115ff0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f60b8015040 tx=0x7f60b8004d10 comp rx=0 tx=0).stop 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 shutdown_connections 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60ac06d7f0 0x7f60ac06fca0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60c41038f0 0x7f60c4115ff0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 --2- 172.21.15.1:0/3877420964 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f60c4116530 0x7f60c410fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 >> 172.21.15.1:0/3877420964 conn(0x7f60c40f8ae0 msgr2=0x7f60c40fa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 shutdown_connections 2024-02-28T16:57:59.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.524+0000 7f60cbc00700 1 -- 172.21.15.1:0/3877420964 wait complete. 2024-02-28T16:57:59.540 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:57:59.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.689+0000 7f44eb600700 1 -- 172.21.15.1:0/1023761152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 msgr2=0x7f44e40ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:59.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.689+0000 7f44eb600700 1 --2- 172.21.15.1:0/1023761152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 0x7f44e40ff170 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f44d0004940 tx=0x7f44d0009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:59.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.689+0000 7f44eb600700 1 -- 172.21.15.1:0/1023761152 shutdown_connections 2024-02-28T16:57:59.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.689+0000 7f44eb600700 1 --2- 172.21.15.1:0/1023761152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 0x7f44e40ff170 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.689+0000 7f44eb600700 1 --2- 172.21.15.1:0/1023761152 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44e4105160 0x7f44e4105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.689+0000 7f44eb600700 1 -- 172.21.15.1:0/1023761152 >> 172.21.15.1:0/1023761152 conn(0x7f44e40fad30 msgr2=0x7f44e40fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:59.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.689+0000 7f44eb600700 1 -- 172.21.15.1:0/1023761152 shutdown_connections 2024-02-28T16:57:59.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.689+0000 7f44eb600700 1 -- 172.21.15.1:0/1023761152 wait complete. 2024-02-28T16:57:59.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.690+0000 7f44eb600700 1 Processor -- start 2024-02-28T16:57:59.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.690+0000 7f44eb600700 1 -- start start 2024-02-28T16:57:59.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.691+0000 7f44eb600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44e4105160 0x7f44e4115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:59.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.691+0000 7f44eb600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 0x7f44e4116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:59.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.691+0000 7f44eb600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e4116860 con 0x7f44e4105b00 2024-02-28T16:57:59.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.691+0000 7f44eb600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e410fbb0 con 0x7f44e4105160 2024-02-28T16:57:59.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.691+0000 7f44e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 0x7f44e4116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:59.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.691+0000 7f44e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 0x7f44e4116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46222/0 (socket says 172.21.15.1:46222) 2024-02-28T16:57:59.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.691+0000 7f44e3e00700 1 -- 172.21.15.1:0/1805824195 learned_addr learned my addr 172.21.15.1:0/1805824195 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:57:59.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.691+0000 7f44e8c00700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44e4105160 0x7f44e4115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:59.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.692+0000 7f44e3e00700 1 -- 172.21.15.1:0/1805824195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44e4105160 msgr2=0x7f44e4115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:59.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.692+0000 7f44e3e00700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44e4105160 0x7f44e4115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.692+0000 7f44e3e00700 1 -- 172.21.15.1:0/1805824195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0005040 con 0x7f44e4105b00 2024-02-28T16:57:59.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.692+0000 7f44e3e00700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 0x7f44e4116210 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009150 tx=0x7f44d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:59.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.692+0000 7f44e0c00700 1 -- 172.21.15.1:0/1805824195 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44d000e040 con 0x7f44e4105b00 2024-02-28T16:57:59.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.692+0000 7f44e0c00700 1 -- 172.21.15.1:0/1805824195 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44d000bbf0 con 0x7f44e4105b00 2024-02-28T16:57:59.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.692+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e410fe30 con 0x7f44e4105b00 2024-02-28T16:57:59.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.692+0000 7f44e0c00700 1 -- 172.21.15.1:0/1805824195 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44d001c630 con 0x7f44e4105b00 2024-02-28T16:57:59.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.693+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e4110370 con 0x7f44e4105b00 2024-02-28T16:57:59.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.694+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e4107630 con 0x7f44e4105b00 2024-02-28T16:57:59.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.695+0000 7f44e0c00700 1 -- 172.21.15.1:0/1805824195 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f44d000bd60 con 0x7f44e4105b00 2024-02-28T16:57:59.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.696+0000 7f44e0c00700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44cc06d7e0 0x7f44cc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:57:59.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.696+0000 7f44e0c00700 1 -- 172.21.15.1:0/1805824195 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f44d008e350 con 0x7f44e4105b00 2024-02-28T16:57:59.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.696+0000 7f44e8c00700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44cc06d7e0 0x7f44cc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:57:59.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.697+0000 7f44e8c00700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44cc06d7e0 0x7f44cc06fc90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f44d8004990 tx=0x7f44d8009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:57:59.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.700+0000 7f44e0c00700 1 -- 172.21.15.1:0/1805824195 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44d0057b80 con 0x7f44e4105b00 2024-02-28T16:57:59.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.943+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44e404bed0 con 0x7f44cc06d7e0 2024-02-28T16:57:59.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.946+0000 7f44e0c00700 1 -- 172.21.15.1:0/1805824195 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f44e404bed0 con 0x7f44cc06d7e0 2024-02-28T16:57:59.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.948+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44cc06d7e0 msgr2=0x7f44cc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:59.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.948+0000 7f44eb600700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44cc06d7e0 0x7f44cc06fc90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f44d8004990 tx=0x7f44d8009ee0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.948+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 msgr2=0x7f44e4116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.948+0000 7f44eb600700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 0x7f44e4116210 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009150 tx=0x7f44d0009e30 comp rx=0 tx=0).stop 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.949+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 shutdown_connections 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.949+0000 7f44eb600700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44cc06d7e0 0x7f44cc06fc90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.949+0000 7f44eb600700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44e4105160 0x7f44e4115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.949+0000 7f44eb600700 1 --2- 172.21.15.1:0/1805824195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44e4105b00 0x7f44e4116210 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.949+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 >> 172.21.15.1:0/1805824195 conn(0x7f44e40fad30 msgr2=0x7f44e40fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.949+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 shutdown_connections 2024-02-28T16:57:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:57:59.949+0000 7f44eb600700 1 -- 172.21.15.1:0/1805824195 wait complete. 2024-02-28T16:58:00.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.104+0000 7f16f1400700 1 -- 172.21.15.1:0/3008057971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 msgr2=0x7f16ec070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.104+0000 7f16f1400700 1 --2- 172.21.15.1:0/3008057971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 0x7f16ec070ce0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f16d8004970 tx=0x7f16d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.104+0000 7f16f1400700 1 -- 172.21.15.1:0/3008057971 shutdown_connections 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.104+0000 7f16f1400700 1 --2- 172.21.15.1:0/3008057971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 0x7f16ec070ce0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.104+0000 7f16f1400700 1 --2- 172.21.15.1:0/3008057971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16ec071f60 0x7f16ec070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.104+0000 7f16f1400700 1 -- 172.21.15.1:0/3008057971 >> 172.21.15.1:0/3008057971 conn(0x7f16ec0f8ae0 msgr2=0x7f16ec0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.104+0000 7f16f1400700 1 -- 172.21.15.1:0/3008057971 shutdown_connections 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.105+0000 7f16f1400700 1 -- 172.21.15.1:0/3008057971 wait complete. 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.105+0000 7f16f1400700 1 Processor -- start 2024-02-28T16:58:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.106+0000 7f16f1400700 1 -- start start 2024-02-28T16:58:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.106+0000 7f16f1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 0x7f16ec11a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.106+0000 7f16f1400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16ec071f60 0x7f16ec11a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.106+0000 7f16f1400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec11abc0 con 0x7f16ec070870 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.106+0000 7f16f1400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec113ed0 con 0x7f16ec071f60 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.106+0000 7f16ea800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 0x7f16ec11a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.106+0000 7f16ea800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 0x7f16ec11a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46240/0 (socket says 172.21.15.1:46240) 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.106+0000 7f16ea800700 1 -- 172.21.15.1:0/53070950 learned_addr learned my addr 172.21.15.1:0/53070950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16ea800700 1 -- 172.21.15.1:0/53070950 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16ec071f60 msgr2=0x7f16ec11a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16ea800700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16ec071f60 0x7f16ec11a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16ea800700 1 -- 172.21.15.1:0/53070950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8005040 con 0x7f16ec070870 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16ea800700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 0x7f16ec11a030 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00c8c0 tx=0x7f16dc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16e2a00700 1 -- 172.21.15.1:0/53070950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16dc00a980 con 0x7f16ec070870 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16e2a00700 1 -- 172.21.15.1:0/53070950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16dc00ce60 con 0x7f16ec070870 2024-02-28T16:58:00.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16e2a00700 1 -- 172.21.15.1:0/53070950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16dc0115e0 con 0x7f16ec070870 2024-02-28T16:58:00.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec1141b0 con 0x7f16ec070870 2024-02-28T16:58:00.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.107+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec1146f0 con 0x7f16ec070870 2024-02-28T16:58:00.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.109+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec0fe400 con 0x7f16ec070870 2024-02-28T16:58:00.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.110+0000 7f16e2a00700 1 -- 172.21.15.1:0/53070950 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f16dc012400 con 0x7f16ec070870 2024-02-28T16:58:00.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.111+0000 7f16e2a00700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16d006d7f0 0x7f16d006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:00.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.111+0000 7f16e2a00700 1 -- 172.21.15.1:0/53070950 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f16dc08cf70 con 0x7f16ec070870 2024-02-28T16:58:00.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.111+0000 7f16e9e00700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16d006d7f0 0x7f16d006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:00.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.112+0000 7f16e9e00700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16d006d7f0 0x7f16d006fca0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f16d800b370 tx=0x7f16d8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:00.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.116+0000 7f16e2a00700 1 -- 172.21.15.1:0/53070950 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f16dc05a350 con 0x7f16ec070870 2024-02-28T16:58:00.278 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:00 smithi001 ceph-mon[112940]: pgmap v707: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:00.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:00 smithi001 ceph-mon[112940]: from='client.35290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:00.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.350+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f16ec04bf40 con 0x7f16d006d7f0 2024-02-28T16:58:00.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.367+0000 7f16e2a00700 1 -- 172.21.15.1:0/53070950 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f16ec04bf40 con 0x7f16d006d7f0 2024-02-28T16:58:00.366 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:58:00.366 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (24m) 52s ago 38m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:58:00.366 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (20m) 52s ago 38m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:58:00.366 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (20m) 54s ago 37m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:58:00.366 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (23m) 52s ago 38m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:58:00.366 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (16m) 52s ago 31m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (16m) 52s ago 31m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (31m) 54s ago 31m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (60s) 54s ago 31m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (26m) 52s ago 40m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (25m) 54s ago 37m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (22m) 52s ago 40m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (21m) 54s ago 36m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (25m) 52s ago 38m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:58:00.367 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (25m) 54s ago 37m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:58:00.368 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (20m) 52s ago 36m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:58:00.368 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (19m) 52s ago 35m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:58:00.368 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (19m) 52s ago 34m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:58:00.368 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (18m) 54s ago 34m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:58:00.368 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (17m) 54s ago 33m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:58:00.368 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (17m) 54s ago 33m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:58:00.368 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (24m) 52s ago 37m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16d006d7f0 msgr2=0x7f16d006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16d006d7f0 0x7f16d006fca0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f16d800b370 tx=0x7f16d8009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 msgr2=0x7f16ec11a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 0x7f16ec11a030 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00c8c0 tx=0x7f16dc0044c0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 shutdown_connections 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f16d006d7f0 0x7f16d006fca0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec070870 0x7f16ec11a030 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 --2- 172.21.15.1:0/53070950 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f16ec071f60 0x7f16ec11a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.370+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 >> 172.21.15.1:0/53070950 conn(0x7f16ec0f8ae0 msgr2=0x7f16ec0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.371+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 shutdown_connections 2024-02-28T16:58:00.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.371+0000 7f16f1400700 1 -- 172.21.15.1:0/53070950 wait complete. 2024-02-28T16:58:00.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:00 smithi066 ceph-mon[85781]: pgmap v707: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:00.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:00 smithi066 ceph-mon[85781]: from='client.35290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.527+0000 7f924d600700 1 -- 172.21.15.1:0/5610920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 msgr2=0x7f9248109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.527+0000 7f924d600700 1 --2- 172.21.15.1:0/5610920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 0x7f9248109100 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9230004970 tx=0x7f9230009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.527+0000 7f924d600700 1 -- 172.21.15.1:0/5610920 shutdown_connections 2024-02-28T16:58:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.527+0000 7f924d600700 1 --2- 172.21.15.1:0/5610920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 0x7f9248109100 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.527+0000 7f924d600700 1 --2- 172.21.15.1:0/5610920 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92480702d0 0x7f92480706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.527+0000 7f924d600700 1 -- 172.21.15.1:0/5610920 >> 172.21.15.1:0/5610920 conn(0x7f92480f8940 msgr2=0x7f92480fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.528+0000 7f924d600700 1 -- 172.21.15.1:0/5610920 shutdown_connections 2024-02-28T16:58:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.528+0000 7f924d600700 1 -- 172.21.15.1:0/5610920 wait complete. 2024-02-28T16:58:00.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.528+0000 7f924d600700 1 Processor -- start 2024-02-28T16:58:00.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.529+0000 7f924d600700 1 -- start start 2024-02-28T16:58:00.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.529+0000 7f924d600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92480702d0 0x7f9248115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:00.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.529+0000 7f924d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 0x7f9248115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:00.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.529+0000 7f924d600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92481166b0 con 0x7f9248070be0 2024-02-28T16:58:00.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.529+0000 7f924d600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924810f950 con 0x7f92480702d0 2024-02-28T16:58:00.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.529+0000 7f923fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 0x7f9248115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:00.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.529+0000 7f923fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 0x7f9248115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46254/0 (socket says 172.21.15.1:46254) 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.529+0000 7f923fe00700 1 -- 172.21.15.1:0/507630658 learned_addr learned my addr 172.21.15.1:0/507630658 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.530+0000 7f9247e00700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92480702d0 0x7f9248115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.530+0000 7f923fe00700 1 -- 172.21.15.1:0/507630658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92480702d0 msgr2=0x7f9248115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.530+0000 7f923fe00700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92480702d0 0x7f9248115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.530+0000 7f923fe00700 1 -- 172.21.15.1:0/507630658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9230005040 con 0x7f9248070be0 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.530+0000 7f923fe00700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 0x7f9248115fd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9230009150 tx=0x7f9230009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.530+0000 7f9244c00700 1 -- 172.21.15.1:0/507630658 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f923000e040 con 0x7f9248070be0 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.531+0000 7f9244c00700 1 -- 172.21.15.1:0/507630658 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9230004290 con 0x7f9248070be0 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.531+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924810fbd0 con 0x7f9248070be0 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.531+0000 7f9244c00700 1 -- 172.21.15.1:0/507630658 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f923001c630 con 0x7f9248070be0 2024-02-28T16:58:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.531+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9248110110 con 0x7f9248070be0 2024-02-28T16:58:00.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.532+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92481067c0 con 0x7f9248070be0 2024-02-28T16:58:00.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.533+0000 7f9244c00700 1 -- 172.21.15.1:0/507630658 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9230013980 con 0x7f9248070be0 2024-02-28T16:58:00.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.534+0000 7f9244c00700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922c06d7f0 0x7f922c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:00.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.534+0000 7f9244c00700 1 -- 172.21.15.1:0/507630658 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f923008e0c0 con 0x7f9248070be0 2024-02-28T16:58:00.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.534+0000 7f9247e00700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922c06d7f0 0x7f922c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:00.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.535+0000 7f9247e00700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922c06d7f0 0x7f922c06fca0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9238006fd0 tx=0x7f9238008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:00.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.539+0000 7f9244c00700 1 -- 172.21.15.1:0/507630658 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f923005b560 con 0x7f9248070be0 2024-02-28T16:58:00.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.854+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f924805f5c0 con 0x7f9248070be0 2024-02-28T16:58:00.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.855+0000 7f9244c00700 1 -- 172.21.15.1:0/507630658 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f923005b0f0 con 0x7f9248070be0 2024-02-28T16:58:00.854 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:58:00.854 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:58:00.854 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:58:00.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:58:00.854 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:58:00.854 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:58:00.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:58:00.855 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:58:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922c06d7f0 msgr2=0x7f922c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922c06d7f0 0x7f922c06fca0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9238006fd0 tx=0x7f9238008040 comp rx=0 tx=0).stop 2024-02-28T16:58:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 msgr2=0x7f9248115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:00.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 0x7f9248115fd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9230009150 tx=0x7f9230009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 shutdown_connections 2024-02-28T16:58:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f922c06d7f0 0x7f922c06fca0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f92480702d0 0x7f9248115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 --2- 172.21.15.1:0/507630658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9248070be0 0x7f9248115fd0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 >> 172.21.15.1:0/507630658 conn(0x7f92480f8940 msgr2=0x7f9248103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 shutdown_connections 2024-02-28T16:58:00.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:00.859+0000 7f924d600700 1 -- 172.21.15.1:0/507630658 wait complete. 2024-02-28T16:58:01.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.015+0000 7fc92a200700 1 -- 172.21.15.1:0/2565503466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924104f90 msgr2=0x7fc924105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.015+0000 7fc92a200700 1 --2- 172.21.15.1:0/2565503466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924104f90 0x7fc924105360 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc918004940 tx=0x7fc918009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:01.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.015+0000 7fc92a200700 1 -- 172.21.15.1:0/2565503466 shutdown_connections 2024-02-28T16:58:01.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.015+0000 7fc92a200700 1 --2- 172.21.15.1:0/2565503466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc924105930 0x7fc9240fefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.015+0000 7fc92a200700 1 --2- 172.21.15.1:0/2565503466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924104f90 0x7fc924105360 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.015+0000 7fc92a200700 1 -- 172.21.15.1:0/2565503466 >> 172.21.15.1:0/2565503466 conn(0x7fc9240fab40 msgr2=0x7fc9240fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:01.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.015+0000 7fc92a200700 1 -- 172.21.15.1:0/2565503466 shutdown_connections 2024-02-28T16:58:01.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.015+0000 7fc92a200700 1 -- 172.21.15.1:0/2565503466 wait complete. 2024-02-28T16:58:01.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.016+0000 7fc92a200700 1 Processor -- start 2024-02-28T16:58:01.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.016+0000 7fc92a200700 1 -- start start 2024-02-28T16:58:01.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.016+0000 7fc92a200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc924104f90 0x7fc924115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.016+0000 7fc92a200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924105930 0x7fc924115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.016+0000 7fc92a200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241165a0 con 0x7fc924105930 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.016+0000 7fc92a200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92410f8d0 con 0x7fc924104f90 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.017+0000 7fc923e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924105930 0x7fc924115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.017+0000 7fc923e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924105930 0x7fc924115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46258/0 (socket says 172.21.15.1:46258) 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.017+0000 7fc923e00700 1 -- 172.21.15.1:0/1616171515 learned_addr learned my addr 172.21.15.1:0/1616171515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.017+0000 7fc928e00700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc924104f90 0x7fc924115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.017+0000 7fc923e00700 1 -- 172.21.15.1:0/1616171515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc924104f90 msgr2=0x7fc924115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.017+0000 7fc923e00700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc924104f90 0x7fc924115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.017+0000 7fc923e00700 1 -- 172.21.15.1:0/1616171515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc918005040 con 0x7fc924105930 2024-02-28T16:58:01.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.017+0000 7fc923e00700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924105930 0x7fc924115f50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc910002a50 tx=0x7fc910004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:01.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.018+0000 7fc920c00700 1 -- 172.21.15.1:0/1616171515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc910007cc0 con 0x7fc924105930 2024-02-28T16:58:01.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.018+0000 7fc920c00700 1 -- 172.21.15.1:0/1616171515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc91000f040 con 0x7fc924105930 2024-02-28T16:58:01.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.018+0000 7fc920c00700 1 -- 172.21.15.1:0/1616171515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc91000daf0 con 0x7fc924105930 2024-02-28T16:58:01.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.018+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92410fbb0 con 0x7fc924105930 2024-02-28T16:58:01.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.018+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc924110170 con 0x7fc924105930 2024-02-28T16:58:01.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.019+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92405f5c0 con 0x7fc924105930 2024-02-28T16:58:01.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.020+0000 7fc920c00700 1 -- 172.21.15.1:0/1616171515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc910009810 con 0x7fc924105930 2024-02-28T16:58:01.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.021+0000 7fc920c00700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc90c06d7f0 0x7fc90c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.021+0000 7fc920c00700 1 -- 172.21.15.1:0/1616171515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc91008cb80 con 0x7fc924105930 2024-02-28T16:58:01.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.021+0000 7fc928e00700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc90c06d7f0 0x7fc90c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.022+0000 7fc928e00700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc90c06d7f0 0x7fc90c06fca0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc91800b3b0 tx=0x7fc918009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:01.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.026+0000 7fc920c00700 1 -- 172.21.15.1:0/1616171515 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc910059f80 con 0x7fc924105930 2024-02-28T16:58:01.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.285+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc92404bed0 con 0x7fc924105930 2024-02-28T16:58:01.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.286+0000 7fc920c00700 1 -- 172.21.15.1:0/1616171515 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fc910059b10 con 0x7fc924105930 2024-02-28T16:58:01.284 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:58:01.285 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:58:01.286 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:58:01.287 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:58:01.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.288+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc90c06d7f0 msgr2=0x7fc90c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.288+0000 7fc92a200700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc90c06d7f0 0x7fc90c06fca0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc91800b3b0 tx=0x7fc918009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:01.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924105930 msgr2=0x7fc924115f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924105930 0x7fc924115f50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc910002a50 tx=0x7fc910004600 comp rx=0 tx=0).stop 2024-02-28T16:58:01.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 shutdown_connections 2024-02-28T16:58:01.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc90c06d7f0 0x7fc90c06fca0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc924104f90 0x7fc924115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 --2- 172.21.15.1:0/1616171515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc924105930 0x7fc924115f50 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 >> 172.21.15.1:0/1616171515 conn(0x7fc9240fab40 msgr2=0x7fc9240fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:01.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 shutdown_connections 2024-02-28T16:58:01.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.289+0000 7fc92a200700 1 -- 172.21.15.1:0/1616171515 wait complete. 2024-02-28T16:58:01.289 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T16:58:01.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.445+0000 7f6635a00700 1 -- 172.21.15.1:0/3747945854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fbdd0 msgr2=0x7f66300fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.445+0000 7f6635a00700 1 --2- 172.21.15.1:0/3747945854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fbdd0 0x7f66300fc1a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6620004940 tx=0x7f6620009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:01.444 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:01 smithi001 ceph-mon[112940]: from='client.35294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:01.444 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:01 smithi001 ceph-mon[112940]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:01.444 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:01 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/507630658' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:58:01.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.446+0000 7f6635a00700 1 -- 172.21.15.1:0/3747945854 shutdown_connections 2024-02-28T16:58:01.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.446+0000 7f6635a00700 1 --2- 172.21.15.1:0/3747945854 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66300fc6e0 0x7f6630100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.446+0000 7f6635a00700 1 --2- 172.21.15.1:0/3747945854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fbdd0 0x7f66300fc1a0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.446+0000 7f6635a00700 1 -- 172.21.15.1:0/3747945854 >> 172.21.15.1:0/3747945854 conn(0x7f6630071d90 msgr2=0x7f6630072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:01.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.446+0000 7f6635a00700 1 -- 172.21.15.1:0/3747945854 shutdown_connections 2024-02-28T16:58:01.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.446+0000 7f6635a00700 1 -- 172.21.15.1:0/3747945854 wait complete. 2024-02-28T16:58:01.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.447+0000 7f6635a00700 1 Processor -- start 2024-02-28T16:58:01.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.447+0000 7f6635a00700 1 -- start start 2024-02-28T16:58:01.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.448+0000 7f6635a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66300fbdd0 0x7f663011a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.448+0000 7f6635a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fc6e0 0x7f663011a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.448+0000 7f6635a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663011ac30 con 0x7f66300fc6e0 2024-02-28T16:58:01.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.448+0000 7f6635a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6630113f40 con 0x7f66300fbdd0 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.448+0000 7f6626000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fc6e0 0x7f663011a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.448+0000 7f6626000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fc6e0 0x7f663011a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46290/0 (socket says 172.21.15.1:46290) 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.448+0000 7f6626000700 1 -- 172.21.15.1:0/3251172353 learned_addr learned my addr 172.21.15.1:0/3251172353 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.448+0000 7f662ea00700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66300fbdd0 0x7f663011a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6626000700 1 -- 172.21.15.1:0/3251172353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66300fbdd0 msgr2=0x7f663011a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6626000700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66300fbdd0 0x7f663011a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6626000700 1 -- 172.21.15.1:0/3251172353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6620005040 con 0x7f66300fc6e0 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6626000700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fc6e0 0x7f663011a5e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f661800c8c0 tx=0x7f66180044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6627400700 1 -- 172.21.15.1:0/3251172353 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661800a980 con 0x7f66300fc6e0 2024-02-28T16:58:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6627400700 1 -- 172.21.15.1:0/3251172353 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f661800ce60 con 0x7f66300fc6e0 2024-02-28T16:58:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6627400700 1 -- 172.21.15.1:0/3251172353 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66180115e0 con 0x7f66300fc6e0 2024-02-28T16:58:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6630114220 con 0x7f66300fc6e0 2024-02-28T16:58:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.449+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6630114760 con 0x7f66300fc6e0 2024-02-28T16:58:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.451+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66300fe540 con 0x7f66300fc6e0 2024-02-28T16:58:01.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.453+0000 7f6627400700 1 -- 172.21.15.1:0/3251172353 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6618012400 con 0x7f66300fc6e0 2024-02-28T16:58:01.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.453+0000 7f6627400700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f661406d7f0 0x7f661406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.454+0000 7f6627400700 1 -- 172.21.15.1:0/3251172353 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f661808cf70 con 0x7f66300fc6e0 2024-02-28T16:58:01.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.454+0000 7f662ea00700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f661406d7f0 0x7f661406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.454+0000 7f662ea00700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f661406d7f0 0x7f661406fca0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f662000b020 tx=0x7f6620009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:01.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.459+0000 7f6627400700 1 -- 172.21.15.1:0/3251172353 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f661805a350 con 0x7f66300fc6e0 2024-02-28T16:58:01.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:01 smithi066 ceph-mon[85781]: from='client.35294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:01.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:01 smithi066 ceph-mon[85781]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:01.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:01 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/507630658' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:58:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.703+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f663004bed0 con 0x7f661406d7f0 2024-02-28T16:58:01.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.705+0000 7f6627400700 1 -- 172.21.15.1:0/3251172353 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f663004bed0 con 0x7f661406d7f0 2024-02-28T16:58:01.703 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:58:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:58:01.703 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:58:01.704 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.707+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f661406d7f0 msgr2=0x7f661406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.707+0000 7f6635a00700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f661406d7f0 0x7f661406fca0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f662000b020 tx=0x7f6620009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.707+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fc6e0 msgr2=0x7f663011a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.707+0000 7f6635a00700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fc6e0 0x7f663011a5e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f661800c8c0 tx=0x7f66180044c0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.708+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 shutdown_connections 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.708+0000 7f6635a00700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f661406d7f0 0x7f661406fca0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.708+0000 7f6635a00700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66300fbdd0 0x7f663011a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.708+0000 7f6635a00700 1 --2- 172.21.15.1:0/3251172353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66300fc6e0 0x7f663011a5e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.708+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 >> 172.21.15.1:0/3251172353 conn(0x7f6630071d90 msgr2=0x7f663010d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:01.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.708+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 shutdown_connections 2024-02-28T16:58:01.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.708+0000 7f6635a00700 1 -- 172.21.15.1:0/3251172353 wait complete. 2024-02-28T16:58:01.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.866+0000 7fc7a9200700 1 -- 172.21.15.1:0/3837628877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105b10 msgr2=0x7fc7a40ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.866+0000 7fc7a9200700 1 --2- 172.21.15.1:0/3837628877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105b10 0x7fc7a40ff180 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc78c004940 tx=0x7fc78c009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.866+0000 7fc7a9200700 1 -- 172.21.15.1:0/3837628877 shutdown_connections 2024-02-28T16:58:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.866+0000 7fc7a9200700 1 --2- 172.21.15.1:0/3837628877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105b10 0x7fc7a40ff180 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.866+0000 7fc7a9200700 1 --2- 172.21.15.1:0/3837628877 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a4105170 0x7fc7a4105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.866+0000 7fc7a9200700 1 -- 172.21.15.1:0/3837628877 >> 172.21.15.1:0/3837628877 conn(0x7fc7a40face0 msgr2=0x7fc7a40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.866+0000 7fc7a9200700 1 -- 172.21.15.1:0/3837628877 shutdown_connections 2024-02-28T16:58:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.866+0000 7fc7a9200700 1 -- 172.21.15.1:0/3837628877 wait complete. 2024-02-28T16:58:01.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.867+0000 7fc7a9200700 1 Processor -- start 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.867+0000 7fc7a9200700 1 -- start start 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.867+0000 7fc7a9200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105170 0x7fc7a4115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.867+0000 7fc7a9200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a4105b10 0x7fc7a4116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.867+0000 7fc7a9200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a4116870 con 0x7fc7a4105170 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.867+0000 7fc7a9200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a410fb80 con 0x7fc7a4105b10 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc7a1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105170 0x7fc7a4115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc7a1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105170 0x7fc7a4115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48414/0 (socket says 172.21.15.1:48414) 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc7a1e00700 1 -- 172.21.15.1:0/3873486217 learned_addr learned my addr 172.21.15.1:0/3873486217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc799400700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a4105b10 0x7fc7a4116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc7a1e00700 1 -- 172.21.15.1:0/3873486217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a4105b10 msgr2=0x7fc7a4116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc7a1e00700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a4105b10 0x7fc7a4116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc7a1e00700 1 -- 172.21.15.1:0/3873486217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c005040 con 0x7fc7a4105170 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc799400700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a4105b10 0x7fc7a4116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:58:01.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.868+0000 7fc7a1e00700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105170 0x7fc7a4115ce0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc79400cac0 tx=0x7fc794007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:01.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.869+0000 7fc79aa00700 1 -- 172.21.15.1:0/3873486217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc79400aa50 con 0x7fc7a4105170 2024-02-28T16:58:01.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.869+0000 7fc79aa00700 1 -- 172.21.15.1:0/3873486217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc79400abb0 con 0x7fc7a4105170 2024-02-28T16:58:01.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.869+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a410fe60 con 0x7fc7a4105170 2024-02-28T16:58:01.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.869+0000 7fc79aa00700 1 -- 172.21.15.1:0/3873486217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7940115e0 con 0x7fc7a4105170 2024-02-28T16:58:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.869+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a41103a0 con 0x7fc7a4105170 2024-02-28T16:58:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.870+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a4107640 con 0x7fc7a4105170 2024-02-28T16:58:01.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.871+0000 7fc79aa00700 1 -- 172.21.15.1:0/3873486217 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc79400ad20 con 0x7fc7a4105170 2024-02-28T16:58:01.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.873+0000 7fc79aa00700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78806d7f0 0x7fc78806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:01.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.873+0000 7fc79aa00700 1 -- 172.21.15.1:0/3873486217 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc79408d6b0 con 0x7fc7a4105170 2024-02-28T16:58:01.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.873+0000 7fc799400700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78806d7f0 0x7fc78806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:01.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.874+0000 7fc799400700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78806d7f0 0x7fc78806fca0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009fd0 tx=0x7fc78c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:01.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:01.877+0000 7fc79aa00700 1 -- 172.21.15.1:0/3873486217 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc79405a990 con 0x7fc7a4105170 2024-02-28T16:58:02.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.196+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc7a40fe900 con 0x7fc7a4105170 2024-02-28T16:58:02.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.197+0000 7fc79aa00700 1 -- 172.21.15.1:0/3873486217 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fc79405a520 con 0x7fc7a4105170 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T16:58:02.196 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:58:02.197 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:58:02.198 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:58:02.198 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:58:02.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78806d7f0 msgr2=0x7fc78806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:02.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78806d7f0 0x7fc78806fca0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009fd0 tx=0x7fc78c009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:02.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105170 msgr2=0x7fc7a4115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105170 0x7fc7a4115ce0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc79400cac0 tx=0x7fc794007ee0 comp rx=0 tx=0).stop 2024-02-28T16:58:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 shutdown_connections 2024-02-28T16:58:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78806d7f0 0x7fc78806fca0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a4105170 0x7fc7a4115ce0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 --2- 172.21.15.1:0/3873486217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a4105b10 0x7fc7a4116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 >> 172.21.15.1:0/3873486217 conn(0x7fc7a40face0 msgr2=0x7fc7a40fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 shutdown_connections 2024-02-28T16:58:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:02.200+0000 7fc7a9200700 1 -- 172.21.15.1:0/3873486217 wait complete. 2024-02-28T16:58:02.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:02 smithi066 ceph-mon[85781]: pgmap v708: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:02.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:02 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1616171515' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:58:02.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:02 smithi066 ceph-mon[85781]: from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:02.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:02 smithi001 ceph-mon[112940]: pgmap v708: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:02.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:02 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1616171515' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:58:02.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:02 smithi001 ceph-mon[112940]: from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:03.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:03 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3873486217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:58:03.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:03 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3873486217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:58:04.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:04 smithi066 ceph-mon[85781]: pgmap v709: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:04.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:04 smithi001 ceph-mon[112940]: pgmap v709: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:06.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:06 smithi066 ceph-mon[85781]: pgmap v710: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:06.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:06 smithi001 ceph-mon[112940]: pgmap v710: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:08.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:08 smithi066 ceph-mon[85781]: pgmap v711: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:08.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:08 smithi001 ceph-mon[112940]: pgmap v711: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:10.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:10 smithi066 ceph-mon[85781]: pgmap v712: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:10.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:10 smithi001 ceph-mon[112940]: pgmap v712: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:11.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:58:11.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:58:12.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:12 smithi066 ceph-mon[85781]: pgmap v713: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:12.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:12 smithi001 ceph-mon[112940]: pgmap v713: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:14.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:14 smithi066 ceph-mon[85781]: pgmap v714: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:14.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:14 smithi001 ceph-mon[112940]: pgmap v714: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:16.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:16 smithi066 ceph-mon[85781]: pgmap v715: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:16.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:16 smithi001 ceph-mon[112940]: pgmap v715: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:18.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:18 smithi066 ceph-mon[85781]: pgmap v716: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:18.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:18 smithi001 ceph-mon[112940]: pgmap v716: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:20.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:20 smithi066 ceph-mon[85781]: pgmap v717: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:20.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:20 smithi001 ceph-mon[112940]: pgmap v717: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:22.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:22 smithi066 ceph-mon[85781]: pgmap v718: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:22.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:22 smithi001 ceph-mon[112940]: pgmap v718: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:24.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:24 smithi066 ceph-mon[85781]: pgmap v719: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:24.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:24 smithi001 ceph-mon[112940]: pgmap v719: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:26.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:26 smithi066 ceph-mon[85781]: pgmap v720: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:26.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:58:26.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:26 smithi001 ceph-mon[112940]: pgmap v720: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:26.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:58:28.507 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:28 smithi066 ceph-mon[85781]: pgmap v721: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:28.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:28 smithi001 ceph-mon[112940]: pgmap v721: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:30.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:30 smithi001 ceph-mon[112940]: pgmap v722: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:30.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:30 smithi066 ceph-mon[85781]: pgmap v722: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.376+0000 7fb89c400700 1 -- 172.21.15.1:0/771348577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105130 msgr2=0x7fb894105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:32.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.376+0000 7fb89c400700 1 --2- 172.21.15.1:0/771348577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105130 0x7fb894105500 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb888004940 tx=0x7fb888009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:32.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.376+0000 7fb89c400700 1 -- 172.21.15.1:0/771348577 shutdown_connections 2024-02-28T16:58:32.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.376+0000 7fb89c400700 1 --2- 172.21.15.1:0/771348577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb894105ad0 0x7fb8940ff140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.376+0000 7fb89c400700 1 --2- 172.21.15.1:0/771348577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105130 0x7fb894105500 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.376+0000 7fb89c400700 1 -- 172.21.15.1:0/771348577 >> 172.21.15.1:0/771348577 conn(0x7fb8940fad00 msgr2=0x7fb8940fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:32.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.376+0000 7fb89c400700 1 -- 172.21.15.1:0/771348577 shutdown_connections 2024-02-28T16:58:32.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.376+0000 7fb89c400700 1 -- 172.21.15.1:0/771348577 wait complete. 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.377+0000 7fb89c400700 1 Processor -- start 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.377+0000 7fb89c400700 1 -- start start 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.378+0000 7fb89c400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb894105130 0x7fb894115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.378+0000 7fb89c400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105ad0 0x7fb894116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.378+0000 7fb89c400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb894116850 con 0x7fb894105ad0 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.378+0000 7fb89c400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89410fb60 con 0x7fb894105130 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.378+0000 7fb899000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105ad0 0x7fb894116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.378+0000 7fb899000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105ad0 0x7fb894116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34278/0 (socket says 172.21.15.1:34278) 2024-02-28T16:58:32.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.378+0000 7fb899000700 1 -- 172.21.15.1:0/3205574987 learned_addr learned my addr 172.21.15.1:0/3205574987 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.379+0000 7fb899a00700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb894105130 0x7fb894115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.379+0000 7fb899000700 1 -- 172.21.15.1:0/3205574987 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb894105130 msgr2=0x7fb894115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.379+0000 7fb899000700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb894105130 0x7fb894115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.379+0000 7fb899000700 1 -- 172.21.15.1:0/3205574987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb888005040 con 0x7fb894105ad0 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.379+0000 7fb899000700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105ad0 0x7fb894116200 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb88000c890 tx=0x7fb8800044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.380+0000 7fb891600700 1 -- 172.21.15.1:0/3205574987 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb88000a7d0 con 0x7fb894105ad0 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.380+0000 7fb891600700 1 -- 172.21.15.1:0/3205574987 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb88000ae00 con 0x7fb894105ad0 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.380+0000 7fb891600700 1 -- 172.21.15.1:0/3205574987 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8800113a0 con 0x7fb894105ad0 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.380+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89410fe40 con 0x7fb894105ad0 2024-02-28T16:58:32.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.380+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb894110380 con 0x7fb894105ad0 2024-02-28T16:58:32.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.381+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb894102440 con 0x7fb894105ad0 2024-02-28T16:58:32.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.382+0000 7fb891600700 1 -- 172.21.15.1:0/3205574987 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb880012400 con 0x7fb894105ad0 2024-02-28T16:58:32.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.384+0000 7fb891600700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb87c06d7f0 0x7fb87c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:32.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.384+0000 7fb891600700 1 -- 172.21.15.1:0/3205574987 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb88008ceb0 con 0x7fb894105ad0 2024-02-28T16:58:32.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.384+0000 7fb899a00700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb87c06d7f0 0x7fb87c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:32.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.385+0000 7fb899a00700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb87c06d7f0 0x7fb87c06fca0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb88800b020 tx=0x7fb888009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:32.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.388+0000 7fb891600700 1 -- 172.21.15.1:0/3205574987 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb88005a350 con 0x7fb894105ad0 2024-02-28T16:58:32.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:32 smithi001 ceph-mon[112940]: pgmap v723: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:32.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.634+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb89404bf40 con 0x7fb87c06d7f0 2024-02-28T16:58:32.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.637+0000 7fb891600700 1 -- 172.21.15.1:0/3205574987 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb89404bf40 con 0x7fb87c06d7f0 2024-02-28T16:58:32.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb87c06d7f0 msgr2=0x7fb87c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:32.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb87c06d7f0 0x7fb87c06fca0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb88800b020 tx=0x7fb888009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:32.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105ad0 msgr2=0x7fb894116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:32.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105ad0 0x7fb894116200 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb88000c890 tx=0x7fb8800044c0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 shutdown_connections 2024-02-28T16:58:32.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb87c06d7f0 0x7fb87c06fca0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb894105130 0x7fb894115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 --2- 172.21.15.1:0/3205574987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb894105ad0 0x7fb894116200 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 >> 172.21.15.1:0/3205574987 conn(0x7fb8940fad00 msgr2=0x7fb8940fc840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:32.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.640+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 shutdown_connections 2024-02-28T16:58:32.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.641+0000 7fb89c400700 1 -- 172.21.15.1:0/3205574987 wait complete. 2024-02-28T16:58:32.653 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:58:32.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:32 smithi066 ceph-mon[85781]: pgmap v723: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.796+0000 7fbff6600700 1 -- 172.21.15.1:0/70550916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff01036e0 msgr2=0x7fbff00fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.796+0000 7fbff6600700 1 --2- 172.21.15.1:0/70550916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff01036e0 0x7fbff00fcd50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0004940 tx=0x7fbfe0009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.796+0000 7fbff6600700 1 -- 172.21.15.1:0/70550916 shutdown_connections 2024-02-28T16:58:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.796+0000 7fbff6600700 1 --2- 172.21.15.1:0/70550916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff01036e0 0x7fbff00fcd50 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.796+0000 7fbff6600700 1 --2- 172.21.15.1:0/70550916 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbff0102d40 0x7fbff0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.796+0000 7fbff6600700 1 -- 172.21.15.1:0/70550916 >> 172.21.15.1:0/70550916 conn(0x7fbff00f88b0 msgr2=0x7fbff00facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:32.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.797+0000 7fbff6600700 1 -- 172.21.15.1:0/70550916 shutdown_connections 2024-02-28T16:58:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.797+0000 7fbff6600700 1 -- 172.21.15.1:0/70550916 wait complete. 2024-02-28T16:58:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.798+0000 7fbff6600700 1 Processor -- start 2024-02-28T16:58:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.798+0000 7fbff6600700 1 -- start start 2024-02-28T16:58:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.799+0000 7fbff6600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff0102d40 0x7fbff0070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.799+0000 7fbff6600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbff01036e0 0x7fbff00707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.799+0000 7fbff6600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0070eb0 con 0x7fbff0102d40 2024-02-28T16:58:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.799+0000 7fbff6600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0070ff0 con 0x7fbff01036e0 2024-02-28T16:58:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.799+0000 7fbff5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff0102d40 0x7fbff0070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.799+0000 7fbff5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff0102d40 0x7fbff0070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34290/0 (socket says 172.21.15.1:34290) 2024-02-28T16:58:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.799+0000 7fbff5200700 1 -- 172.21.15.1:0/2322397860 learned_addr learned my addr 172.21.15.1:0/2322397860 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.799+0000 7fbfe7e00700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbff01036e0 0x7fbff00707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.800+0000 7fbff5200700 1 -- 172.21.15.1:0/2322397860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbff01036e0 msgr2=0x7fbff00707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.800+0000 7fbff5200700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbff01036e0 0x7fbff00707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:32.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.800+0000 7fbff5200700 1 -- 172.21.15.1:0/2322397860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0005040 con 0x7fbff0102d40 2024-02-28T16:58:32.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.800+0000 7fbff5200700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff0102d40 0x7fbff0070290 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800b790 tx=0x7fbfd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:32.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.800+0000 7fbfed600700 1 -- 172.21.15.1:0/2322397860 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd800f770 con 0x7fbff0102d40 2024-02-28T16:58:32.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.800+0000 7fbfed600700 1 -- 172.21.15.1:0/2322397860 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfd800fda0 con 0x7fbff0102d40 2024-02-28T16:58:32.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.800+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff0119ad0 con 0x7fbff0102d40 2024-02-28T16:58:32.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.800+0000 7fbfed600700 1 -- 172.21.15.1:0/2322397860 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd800d310 con 0x7fbff0102d40 2024-02-28T16:58:32.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.801+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff011a060 con 0x7fbff0102d40 2024-02-28T16:58:32.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.802+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff0100050 con 0x7fbff0102d40 2024-02-28T16:58:32.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.803+0000 7fbfed600700 1 -- 172.21.15.1:0/2322397860 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbfd8010420 con 0x7fbff0102d40 2024-02-28T16:58:32.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.804+0000 7fbfed600700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbfd006d7f0 0x7fbfd006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:32.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.804+0000 7fbfed600700 1 -- 172.21.15.1:0/2322397860 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbfd808cde0 con 0x7fbff0102d40 2024-02-28T16:58:32.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.804+0000 7fbfe7e00700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbfd006d7f0 0x7fbfd006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:32.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.805+0000 7fbfe7e00700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbfd006d7f0 0x7fbfd006fca0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbff0071c40 tx=0x7fbfe0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:32.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:32.809+0000 7fbfed600700 1 -- 172.21.15.1:0/2322397860 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbfd8056750 con 0x7fbff0102d40 2024-02-28T16:58:33.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.051+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbff0071980 con 0x7fbfd006d7f0 2024-02-28T16:58:33.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.054+0000 7fbfed600700 1 -- 172.21.15.1:0/2322397860 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbff0071980 con 0x7fbfd006d7f0 2024-02-28T16:58:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.056+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbfd006d7f0 msgr2=0x7fbfd006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.056+0000 7fbff6600700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbfd006d7f0 0x7fbfd006fca0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbff0071c40 tx=0x7fbfe0009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.056+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff0102d40 msgr2=0x7fbff0070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.056+0000 7fbff6600700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff0102d40 0x7fbff0070290 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800b790 tx=0x7fbfd8009250 comp rx=0 tx=0).stop 2024-02-28T16:58:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.057+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 shutdown_connections 2024-02-28T16:58:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.057+0000 7fbff6600700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbfd006d7f0 0x7fbfd006fca0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.057+0000 7fbff6600700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff0102d40 0x7fbff0070290 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.057+0000 7fbff6600700 1 --2- 172.21.15.1:0/2322397860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbff01036e0 0x7fbff00707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.057+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 >> 172.21.15.1:0/2322397860 conn(0x7fbff00f88b0 msgr2=0x7fbff00fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.057+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 shutdown_connections 2024-02-28T16:58:33.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.057+0000 7fbff6600700 1 -- 172.21.15.1:0/2322397860 wait complete. 2024-02-28T16:58:33.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.212+0000 7f2d63000700 1 -- 172.21.15.1:0/2378999484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 msgr2=0x7f2d5c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.212+0000 7f2d63000700 1 --2- 172.21.15.1:0/2378999484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 0x7f2d5c1099c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c004940 tx=0x7f2d4c009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.212+0000 7f2d63000700 1 -- 172.21.15.1:0/2378999484 shutdown_connections 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.212+0000 7f2d63000700 1 --2- 172.21.15.1:0/2378999484 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d5c103dc0 0x7f2d5c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.212+0000 7f2d63000700 1 --2- 172.21.15.1:0/2378999484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 0x7f2d5c1099c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.212+0000 7f2d63000700 1 -- 172.21.15.1:0/2378999484 >> 172.21.15.1:0/2378999484 conn(0x7f2d5c072d50 msgr2=0x7f2d5c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.212+0000 7f2d63000700 1 -- 172.21.15.1:0/2378999484 shutdown_connections 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.212+0000 7f2d63000700 1 -- 172.21.15.1:0/2378999484 wait complete. 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.213+0000 7f2d63000700 1 Processor -- start 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.213+0000 7f2d63000700 1 -- start start 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d63000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d5c103dc0 0x7f2d5c11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d63000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 0x7f2d5c11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d63000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c11ac70 con 0x7f2d5c1095f0 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d63000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c113f80 con 0x7f2d5c103dc0 2024-02-28T16:58:33.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 0x7f2d5c11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 0x7f2d5c11a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34316/0 (socket says 172.21.15.1:34316) 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d5b400700 1 -- 172.21.15.1:0/2787389221 learned_addr learned my addr 172.21.15.1:0/2787389221 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d5be00700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d5c103dc0 0x7f2d5c11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d5b400700 1 -- 172.21.15.1:0/2787389221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d5c103dc0 msgr2=0x7f2d5c11a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d5b400700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d5c103dc0 0x7f2d5c11a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.214+0000 7f2d5b400700 1 -- 172.21.15.1:0/2787389221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c005040 con 0x7f2d5c1095f0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.215+0000 7f2d5be00700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d5c103dc0 0x7f2d5c11a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.215+0000 7f2d5b400700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 0x7f2d5c11a620 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800c830 tx=0x7f2d48004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.215+0000 7f2d53e00700 1 -- 172.21.15.1:0/2787389221 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4800a760 con 0x7f2d5c1095f0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.215+0000 7f2d53e00700 1 -- 172.21.15.1:0/2787389221 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d4800ad90 con 0x7f2d5c1095f0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.215+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c114260 con 0x7f2d5c1095f0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.215+0000 7f2d53e00700 1 -- 172.21.15.1:0/2787389221 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d480113a0 con 0x7f2d5c1095f0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.215+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c1147a0 con 0x7f2d5c1095f0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.216+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c106850 con 0x7f2d5c1095f0 2024-02-28T16:58:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.218+0000 7f2d53e00700 1 -- 172.21.15.1:0/2787389221 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2d48012400 con 0x7f2d5c1095f0 2024-02-28T16:58:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.219+0000 7f2d53e00700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d4006d7f0 0x7f2d4006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.219+0000 7f2d53e00700 1 -- 172.21.15.1:0/2787389221 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2d4808cfa0 con 0x7f2d5c1095f0 2024-02-28T16:58:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.219+0000 7f2d5be00700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d4006d7f0 0x7f2d4006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.220+0000 7f2d5be00700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d4006d7f0 0x7f2d4006fca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00b3b0 tx=0x7f2d4c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.224+0000 7f2d53e00700 1 -- 172.21.15.1:0/2787389221 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2d4805a300 con 0x7f2d5c1095f0 2024-02-28T16:58:33.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.457+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d5c04bf70 con 0x7f2d4006d7f0 2024-02-28T16:58:33.456 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:33 smithi001 ceph-mon[112940]: from='client.35318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:33.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.470+0000 7f2d53e00700 1 -- 172.21.15.1:0/2787389221 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2d4c00b6e0 con 0x7f2d4006d7f0 2024-02-28T16:58:33.467 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (24m) 85s ago 38m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (21m) 85s ago 39m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (21m) 87s ago 37m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (24m) 85s ago 38m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (17m) 85s ago 31m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (16m) 85s ago 31m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (31m) 87s ago 31m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (93s) 87s ago 31m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (27m) 85s ago 40m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (26m) 87s ago 37m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (23m) 85s ago 40m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:58:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (22m) 87s ago 37m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (25m) 85s ago 39m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (25m) 87s ago 37m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (20m) 85s ago 36m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (20m) 85s ago 36m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (19m) 85s ago 35m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (19m) 87s ago 34m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (18m) 87s ago 34m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (17m) 87s ago 33m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (25m) 85s ago 38m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:58:33.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d4006d7f0 msgr2=0x7f2d4006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d4006d7f0 0x7f2d4006fca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00b3b0 tx=0x7f2d4c009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 msgr2=0x7f2d5c11a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 0x7f2d5c11a620 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800c830 tx=0x7f2d48004470 comp rx=0 tx=0).stop 2024-02-28T16:58:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 shutdown_connections 2024-02-28T16:58:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2d4006d7f0 0x7f2d4006fca0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2d5c103dc0 0x7f2d5c11a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 --2- 172.21.15.1:0/2787389221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c1095f0 0x7f2d5c11a620 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.472+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 >> 172.21.15.1:0/2787389221 conn(0x7f2d5c072d50 msgr2=0x7f2d5c074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:33.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.473+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 shutdown_connections 2024-02-28T16:58:33.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.473+0000 7f2d63000700 1 -- 172.21.15.1:0/2787389221 wait complete. 2024-02-28T16:58:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.621+0000 7f040fe00700 1 -- 172.21.15.1:0/3162366553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 msgr2=0x7f0408105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.621+0000 7f040fe00700 1 --2- 172.21.15.1:0/3162366553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 0x7f0408105540 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f03fc004940 tx=0x7f03fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.621+0000 7f040fe00700 1 -- 172.21.15.1:0/3162366553 shutdown_connections 2024-02-28T16:58:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.621+0000 7f040fe00700 1 --2- 172.21.15.1:0/3162366553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105b10 0x7f04080ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.621+0000 7f040fe00700 1 --2- 172.21.15.1:0/3162366553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 0x7f0408105540 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.621+0000 7f040fe00700 1 -- 172.21.15.1:0/3162366553 >> 172.21.15.1:0/3162366553 conn(0x7f04080face0 msgr2=0x7f04080fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.621+0000 7f040fe00700 1 -- 172.21.15.1:0/3162366553 shutdown_connections 2024-02-28T16:58:33.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.621+0000 7f040fe00700 1 -- 172.21.15.1:0/3162366553 wait complete. 2024-02-28T16:58:33.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.622+0000 7f040fe00700 1 Processor -- start 2024-02-28T16:58:33.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.622+0000 7f040fe00700 1 -- start start 2024-02-28T16:58:33.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.622+0000 7f040fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 0x7f0408115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:33.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.622+0000 7f040fe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105b10 0x7f04081161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:33.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.622+0000 7f040fe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081167f0 con 0x7f0408105170 2024-02-28T16:58:33.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.622+0000 7f040fe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040810fb40 con 0x7f0408105b10 2024-02-28T16:58:33.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.622+0000 7f040d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 0x7f0408115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:33.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.623+0000 7f040d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 0x7f0408115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34346/0 (socket says 172.21.15.1:34346) 2024-02-28T16:58:33.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.623+0000 7f040d400700 1 -- 172.21.15.1:0/1957529388 learned_addr learned my addr 172.21.15.1:0/1957529388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:33.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.623+0000 7f040ca00700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105b10 0x7f04081161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.623+0000 7f040d400700 1 -- 172.21.15.1:0/1957529388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105b10 msgr2=0x7f04081161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.623+0000 7f040d400700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105b10 0x7f04081161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.623+0000 7f040d400700 1 -- 172.21.15.1:0/1957529388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc005040 con 0x7f0408105170 2024-02-28T16:58:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.623+0000 7f040d400700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 0x7f0408115c60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f03fc009fd0 tx=0x7f03fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.623+0000 7f0405600700 1 -- 172.21.15.1:0/1957529388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03fc00e040 con 0x7f0408105170 2024-02-28T16:58:33.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.624+0000 7f0405600700 1 -- 172.21.15.1:0/1957529388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03fc00bbf0 con 0x7f0408105170 2024-02-28T16:58:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.624+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040810fdc0 con 0x7f0408105170 2024-02-28T16:58:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.624+0000 7f0405600700 1 -- 172.21.15.1:0/1957529388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03fc01c3f0 con 0x7f0408105170 2024-02-28T16:58:33.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.624+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0408110300 con 0x7f0408105170 2024-02-28T16:58:33.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.625+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0408102480 con 0x7f0408105170 2024-02-28T16:58:33.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.626+0000 7f0405600700 1 -- 172.21.15.1:0/1957529388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f03fc00bd60 con 0x7f0408105170 2024-02-28T16:58:33.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.627+0000 7f0405600700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03f006d7f0 0x7f03f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:33.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.627+0000 7f0405600700 1 -- 172.21.15.1:0/1957529388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f03fc08e1e0 con 0x7f0408105170 2024-02-28T16:58:33.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.627+0000 7f040ca00700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03f006d7f0 0x7f03f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:33.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.628+0000 7f040ca00700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03f006d7f0 0x7f03f006fca0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0408111540 tx=0x7f03f400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.632+0000 7f0405600700 1 -- 172.21.15.1:0/1957529388 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f03fc057a90 con 0x7f0408105170 2024-02-28T16:58:33.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:33 smithi066 ceph-mon[85781]: from='client.35318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:33.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.942+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0408111280 con 0x7f0408105170 2024-02-28T16:58:33.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.943+0000 7f0405600700 1 -- 172.21.15.1:0/1957529388 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f03fc05b150 con 0x7f0408105170 2024-02-28T16:58:33.941 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:58:33.941 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:58:33.941 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:58:33.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:58:33.941 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:58:33.941 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:58:33.942 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:58:33.943 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T16:58:33.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:58:33.943 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:58:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03f006d7f0 msgr2=0x7f03f006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03f006d7f0 0x7f03f006fca0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0408111540 tx=0x7f03f400b040 comp rx=0 tx=0).stop 2024-02-28T16:58:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 msgr2=0x7f0408115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 0x7f0408115c60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f03fc009fd0 tx=0x7f03fc009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 shutdown_connections 2024-02-28T16:58:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03f006d7f0 0x7f03f006fca0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105170 0x7f0408115c60 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 --2- 172.21.15.1:0/1957529388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105b10 0x7f04081161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.945+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 >> 172.21.15.1:0/1957529388 conn(0x7f04080face0 msgr2=0x7f04080fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.946+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 shutdown_connections 2024-02-28T16:58:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:33.946+0000 7f040fe00700 1 -- 172.21.15.1:0/1957529388 wait complete. 2024-02-28T16:58:34.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.101+0000 7f45dfe00700 1 -- 172.21.15.1:0/2299288059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 msgr2=0x7f45d80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.101+0000 7f45dfe00700 1 --2- 172.21.15.1:0/2299288059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 0x7f45d80fe3b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f45cc004970 tx=0x7f45cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.101+0000 7f45dfe00700 1 -- 172.21.15.1:0/2299288059 shutdown_connections 2024-02-28T16:58:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.102+0000 7f45dfe00700 1 --2- 172.21.15.1:0/2299288059 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45d80fe8f0 0x7f45d8107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.102+0000 7f45dfe00700 1 --2- 172.21.15.1:0/2299288059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 0x7f45d80fe3b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.102+0000 7f45dfe00700 1 -- 172.21.15.1:0/2299288059 >> 172.21.15.1:0/2299288059 conn(0x7f45d8071d90 msgr2=0x7f45d8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.102+0000 7f45dfe00700 1 -- 172.21.15.1:0/2299288059 shutdown_connections 2024-02-28T16:58:34.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.102+0000 7f45dfe00700 1 -- 172.21.15.1:0/2299288059 wait complete. 2024-02-28T16:58:34.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.102+0000 7f45dfe00700 1 Processor -- start 2024-02-28T16:58:34.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.103+0000 7f45dfe00700 1 -- start start 2024-02-28T16:58:34.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.103+0000 7f45dfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 0x7f45d80fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.103+0000 7f45dfe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45d80fe8f0 0x7f45d80fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.103+0000 7f45dfe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d80fc9e0 con 0x7f45d80fdfe0 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.103+0000 7f45dfe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d80fcb50 con 0x7f45d80fe8f0 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.103+0000 7f45dd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 0x7f45d80fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.103+0000 7f45dd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 0x7f45d80fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34358/0 (socket says 172.21.15.1:34358) 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.103+0000 7f45dd400700 1 -- 172.21.15.1:0/704866471 learned_addr learned my addr 172.21.15.1:0/704866471 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.104+0000 7f45dd400700 1 -- 172.21.15.1:0/704866471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45d80fe8f0 msgr2=0x7f45d80fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.104+0000 7f45dca00700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45d80fe8f0 0x7f45d80fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.104+0000 7f45dd400700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45d80fe8f0 0x7f45d80fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.104+0000 7f45dd400700 1 -- 172.21.15.1:0/704866471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc005040 con 0x7f45d80fdfe0 2024-02-28T16:58:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.104+0000 7f45dd400700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 0x7f45d80fbea0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f45cc000c00 tx=0x7f45cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:34.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.104+0000 7f45d5600700 1 -- 172.21.15.1:0/704866471 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45cc00e040 con 0x7f45d80fdfe0 2024-02-28T16:58:34.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.104+0000 7f45d5600700 1 -- 172.21.15.1:0/704866471 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45cc004290 con 0x7f45d80fdfe0 2024-02-28T16:58:34.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.104+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d8119c90 con 0x7f45d80fdfe0 2024-02-28T16:58:34.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.105+0000 7f45d5600700 1 -- 172.21.15.1:0/704866471 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45cc01c630 con 0x7f45d80fdfe0 2024-02-28T16:58:34.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.105+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d811a0f0 con 0x7f45d80fdfe0 2024-02-28T16:58:34.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.106+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d81046c0 con 0x7f45d80fdfe0 2024-02-28T16:58:34.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.107+0000 7f45d5600700 1 -- 172.21.15.1:0/704866471 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f45cc013980 con 0x7f45d80fdfe0 2024-02-28T16:58:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.108+0000 7f45d5600700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45c006d7e0 0x7f45c006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.108+0000 7f45d5600700 1 -- 172.21.15.1:0/704866471 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f45cc08e1d0 con 0x7f45d80fdfe0 2024-02-28T16:58:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.108+0000 7f45dca00700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45c006d7e0 0x7f45c006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:34.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.109+0000 7f45dca00700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45c006d7e0 0x7f45c006fc90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f45d80fd8a0 tx=0x7f45c400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.112+0000 7f45d5600700 1 -- 172.21.15.1:0/704866471 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f45cc057a80 con 0x7f45d80fdfe0 2024-02-28T16:58:34.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.391+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f45d80fd5e0 con 0x7f45d80fdfe0 2024-02-28T16:58:34.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.392+0000 7f45d5600700 1 -- 172.21.15.1:0/704866471 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f45cc05b140 con 0x7f45d80fdfe0 2024-02-28T16:58:34.390 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:58:34.391 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:58:34.392 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:58:34.393 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:58:34.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45c006d7e0 msgr2=0x7f45c006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45c006d7e0 0x7f45c006fc90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f45d80fd8a0 tx=0x7f45c400b040 comp rx=0 tx=0).stop 2024-02-28T16:58:34.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 msgr2=0x7f45d80fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 0x7f45d80fbea0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f45cc000c00 tx=0x7f45cc009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:34.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 shutdown_connections 2024-02-28T16:58:34.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45c006d7e0 0x7f45c006fc90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45d80fdfe0 0x7f45d80fbea0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 --2- 172.21.15.1:0/704866471 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45d80fe8f0 0x7f45d80fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 >> 172.21.15.1:0/704866471 conn(0x7f45d8071d90 msgr2=0x7f45d8101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:34.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 shutdown_connections 2024-02-28T16:58:34.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.395+0000 7f45dfe00700 1 -- 172.21.15.1:0/704866471 wait complete. 2024-02-28T16:58:34.395 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T16:58:34.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:34 smithi001 ceph-mon[112940]: pgmap v724: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:34.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:34 smithi001 ceph-mon[112940]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:34.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:34 smithi001 ceph-mon[112940]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:34.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:34 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1957529388' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:58:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.551+0000 7fcbac600700 1 -- 172.21.15.1:0/1871441708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070c40 msgr2=0x7fcba4109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.551+0000 7fcbac600700 1 --2- 172.21.15.1:0/1871441708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070c40 0x7fcba4109260 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcb90004970 tx=0x7fcb90009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.551+0000 7fcbac600700 1 -- 172.21.15.1:0/1871441708 shutdown_connections 2024-02-28T16:58:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.551+0000 7fcbac600700 1 --2- 172.21.15.1:0/1871441708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070c40 0x7fcba4109260 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.551+0000 7fcbac600700 1 --2- 172.21.15.1:0/1871441708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcba4070330 0x7fcba4070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.551+0000 7fcbac600700 1 -- 172.21.15.1:0/1871441708 >> 172.21.15.1:0/1871441708 conn(0x7fcba40f8ae0 msgr2=0x7fcba40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.551+0000 7fcbac600700 1 -- 172.21.15.1:0/1871441708 shutdown_connections 2024-02-28T16:58:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.551+0000 7fcbac600700 1 -- 172.21.15.1:0/1871441708 wait complete. 2024-02-28T16:58:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.552+0000 7fcbac600700 1 Processor -- start 2024-02-28T16:58:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.552+0000 7fcbac600700 1 -- start start 2024-02-28T16:58:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcbac600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070330 0x7fcba4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcbac600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcba4070c40 0x7fcba4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcbac600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba4116790 con 0x7fcba4070330 2024-02-28T16:58:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcbac600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba410fae0 con 0x7fcba4070c40 2024-02-28T16:58:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcba9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070330 0x7fcba4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcba9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070330 0x7fcba4115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34376/0 (socket says 172.21.15.1:34376) 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcba9c00700 1 -- 172.21.15.1:0/2609135015 learned_addr learned my addr 172.21.15.1:0/2609135015 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcba9c00700 1 -- 172.21.15.1:0/2609135015 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcba4070c40 msgr2=0x7fcba4116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcba9c00700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcba4070c40 0x7fcba4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.553+0000 7fcba9c00700 1 -- 172.21.15.1:0/2609135015 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb90005040 con 0x7fcba4070330 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.554+0000 7fcba9c00700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070330 0x7fcba4115c00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcb98002a50 tx=0x7fcb980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.554+0000 7fcba1600700 1 -- 172.21.15.1:0/2609135015 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb98007cc0 con 0x7fcba4070330 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.554+0000 7fcba1600700 1 -- 172.21.15.1:0/2609135015 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb9800f040 con 0x7fcba4070330 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.554+0000 7fcba1600700 1 -- 172.21.15.1:0/2609135015 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb9800db70 con 0x7fcba4070330 2024-02-28T16:58:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.554+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba410fdc0 con 0x7fcba4070330 2024-02-28T16:58:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.554+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba4110300 con 0x7fcba4070330 2024-02-28T16:58:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.555+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba4106920 con 0x7fcba4070330 2024-02-28T16:58:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.557+0000 7fcba1600700 1 -- 172.21.15.1:0/2609135015 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcb9800dd10 con 0x7fcba4070330 2024-02-28T16:58:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.557+0000 7fcba1600700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb8c06d7f0 0x7fcb8c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.558+0000 7fcba1600700 1 -- 172.21.15.1:0/2609135015 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcb9808c670 con 0x7fcba4070330 2024-02-28T16:58:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.558+0000 7fcba9200700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb8c06d7f0 0x7fcb8c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.559+0000 7fcba9200700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb8c06d7f0 0x7fcb8c06fca0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000b370 tx=0x7fcb90009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:34.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.562+0000 7fcba1600700 1 -- 172.21.15.1:0/2609135015 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcb98055e20 con 0x7fcba4070330 2024-02-28T16:58:34.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:34 smithi066 ceph-mon[85781]: pgmap v724: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:34.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:34 smithi066 ceph-mon[85781]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:34.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:34 smithi066 ceph-mon[85781]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:34.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:34 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1957529388' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:58:34.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.798+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcba404bf70 con 0x7fcb8c06d7f0 2024-02-28T16:58:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.800+0000 7fcba1600700 1 -- 172.21.15.1:0/2609135015 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcb90009710 con 0x7fcb8c06d7f0 2024-02-28T16:58:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:58:34.798 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:58:34.798 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:58:34.798 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:58:34.798 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:58:34.799 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:58:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb8c06d7f0 msgr2=0x7fcb8c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb8c06d7f0 0x7fcb8c06fca0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000b370 tx=0x7fcb90009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070330 msgr2=0x7fcba4115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070330 0x7fcba4115c00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcb98002a50 tx=0x7fcb980046b0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 shutdown_connections 2024-02-28T16:58:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb8c06d7f0 0x7fcb8c06fca0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4070330 0x7fcba4115c00 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 --2- 172.21.15.1:0/2609135015 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcba4070c40 0x7fcba4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 >> 172.21.15.1:0/2609135015 conn(0x7fcba40f8ae0 msgr2=0x7fcba4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 shutdown_connections 2024-02-28T16:58:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.803+0000 7fcbac600700 1 -- 172.21.15.1:0/2609135015 wait complete. 2024-02-28T16:58:34.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.960+0000 7f3c95400700 1 -- 172.21.15.1:0/2822217100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 msgr2=0x7f3c90105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.960+0000 7f3c95400700 1 --2- 172.21.15.1:0/2822217100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 0x7f3c90105530 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3c80004940 tx=0x7f3c80009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:34.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.960+0000 7f3c95400700 1 -- 172.21.15.1:0/2822217100 shutdown_connections 2024-02-28T16:58:34.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.960+0000 7f3c95400700 1 --2- 172.21.15.1:0/2822217100 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c90105b00 0x7f3c900ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.960+0000 7f3c95400700 1 --2- 172.21.15.1:0/2822217100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 0x7f3c90105530 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.960+0000 7f3c95400700 1 -- 172.21.15.1:0/2822217100 >> 172.21.15.1:0/2822217100 conn(0x7f3c900fad30 msgr2=0x7f3c900fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:34.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.960+0000 7f3c95400700 1 -- 172.21.15.1:0/2822217100 shutdown_connections 2024-02-28T16:58:34.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.960+0000 7f3c95400700 1 -- 172.21.15.1:0/2822217100 wait complete. 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.961+0000 7f3c95400700 1 Processor -- start 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.961+0000 7f3c95400700 1 -- start start 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.962+0000 7f3c95400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 0x7f3c90115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.962+0000 7f3c95400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c90105b00 0x7f3c901161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.962+0000 7f3c95400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c901167f0 con 0x7f3c90105160 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.962+0000 7f3c95400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9010fb40 con 0x7f3c90105b00 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.962+0000 7f3c8e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 0x7f3c90115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.962+0000 7f3c8e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 0x7f3c90115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34400/0 (socket says 172.21.15.1:34400) 2024-02-28T16:58:34.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.962+0000 7f3c8e800700 1 -- 172.21.15.1:0/3899298757 learned_addr learned my addr 172.21.15.1:0/3899298757 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:58:34.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.963+0000 7f3c8e800700 1 -- 172.21.15.1:0/3899298757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c90105b00 msgr2=0x7f3c901161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:34.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.963+0000 7f3c8e800700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c90105b00 0x7f3c901161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:34.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.963+0000 7f3c8e800700 1 -- 172.21.15.1:0/3899298757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80005040 con 0x7f3c90105160 2024-02-28T16:58:34.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.963+0000 7f3c8e800700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 0x7f3c90115c60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009150 tx=0x7f3c80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:34.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.963+0000 7f3c86a00700 1 -- 172.21.15.1:0/3899298757 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c8000e040 con 0x7f3c90105160 2024-02-28T16:58:34.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.963+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9010fdc0 con 0x7f3c90105160 2024-02-28T16:58:34.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.963+0000 7f3c86a00700 1 -- 172.21.15.1:0/3899298757 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c8000bbf0 con 0x7f3c90105160 2024-02-28T16:58:34.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.963+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c90110300 con 0x7f3c90105160 2024-02-28T16:58:34.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.964+0000 7f3c86a00700 1 -- 172.21.15.1:0/3899298757 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c80013ea0 con 0x7f3c90105160 2024-02-28T16:58:34.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.965+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c90102470 con 0x7f3c90105160 2024-02-28T16:58:34.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.966+0000 7f3c86a00700 1 -- 172.21.15.1:0/3899298757 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3c8000bd60 con 0x7f3c90105160 2024-02-28T16:58:34.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.966+0000 7f3c86a00700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c7406d7f0 0x7f3c7406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:58:34.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.967+0000 7f3c86a00700 1 -- 172.21.15.1:0/3899298757 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3c8008e320 con 0x7f3c90105160 2024-02-28T16:58:34.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.967+0000 7f3c8de00700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c7406d7f0 0x7f3c7406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:58:34.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.967+0000 7f3c8de00700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c7406d7f0 0x7f3c7406fca0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3c90111540 tx=0x7f3c7c007d40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:58:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:34.973+0000 7f3c86a00700 1 -- 172.21.15.1:0/3899298757 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c80057b50 con 0x7f3c90105160 2024-02-28T16:58:35.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.290+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3c900fe930 con 0x7f3c90105160 2024-02-28T16:58:35.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.294+0000 7f3c86a00700 1 -- 172.21.15.1:0/3899298757 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f3c8005b210 con 0x7f3c90105160 2024-02-28T16:58:35.291 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T16:58:35.292 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:58:35.293 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:58:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.296+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c7406d7f0 msgr2=0x7f3c7406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.296+0000 7f3c95400700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c7406d7f0 0x7f3c7406fca0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3c90111540 tx=0x7f3c7c007d40 comp rx=0 tx=0).stop 2024-02-28T16:58:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.296+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 msgr2=0x7f3c90115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:58:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.296+0000 7f3c95400700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 0x7f3c90115c60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009150 tx=0x7f3c80009e30 comp rx=0 tx=0).stop 2024-02-28T16:58:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.297+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 shutdown_connections 2024-02-28T16:58:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.297+0000 7f3c95400700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3c7406d7f0 0x7f3c7406fca0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.297+0000 7f3c95400700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c90105160 0x7f3c90115c60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.297+0000 7f3c95400700 1 --2- 172.21.15.1:0/3899298757 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c90105b00 0x7f3c901161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:58:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.297+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 >> 172.21.15.1:0/3899298757 conn(0x7f3c900fad30 msgr2=0x7f3c900fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:58:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.297+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 shutdown_connections 2024-02-28T16:58:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:58:35.297+0000 7f3c95400700 1 -- 172.21.15.1:0/3899298757 wait complete. 2024-02-28T16:58:35.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:35 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/704866471' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:58:35.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:35 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/704866471' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:58:36.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:36 smithi066 ceph-mon[85781]: from='client.35338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:36.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:36 smithi066 ceph-mon[85781]: pgmap v725: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:36.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3899298757' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:58:36.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:36 smithi001 ceph-mon[112940]: from='client.35338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:58:36.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:36 smithi001 ceph-mon[112940]: pgmap v725: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:36.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3899298757' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:58:38.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:38 smithi066 ceph-mon[85781]: pgmap v726: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:38.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:38 smithi001 ceph-mon[112940]: pgmap v726: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:40.560 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:40 smithi066 ceph-mon[85781]: pgmap v727: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:40.780 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:40 smithi001 ceph-mon[112940]: pgmap v727: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:41.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:58:41.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:58:42.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:42 smithi066 ceph-mon[85781]: pgmap v728: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:42.852 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:42 smithi001 ceph-mon[112940]: pgmap v728: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:44.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:44 smithi066 ceph-mon[85781]: pgmap v729: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:44.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:44 smithi001 ceph-mon[112940]: pgmap v729: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:46.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:46 smithi066 ceph-mon[85781]: pgmap v730: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:46.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:46 smithi001 ceph-mon[112940]: pgmap v730: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:48.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:48 smithi066 ceph-mon[85781]: pgmap v731: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:48.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:48 smithi001 ceph-mon[112940]: pgmap v731: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:50.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:50 smithi066 ceph-mon[85781]: pgmap v732: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:50.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:50 smithi001 ceph-mon[112940]: pgmap v732: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:52.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:52 smithi066 ceph-mon[85781]: pgmap v733: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:52.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:52 smithi001 ceph-mon[112940]: pgmap v733: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:54.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:54 smithi066 ceph-mon[85781]: pgmap v734: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:54.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:54 smithi001 ceph-mon[112940]: pgmap v734: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:56.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:56 smithi066 ceph-mon[85781]: pgmap v735: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:56.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:58:56.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:56 smithi001 ceph-mon[112940]: pgmap v735: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:56.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:58:58.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:58:58 smithi066 ceph-mon[85781]: pgmap v736: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:58:58.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:58:58 smithi001 ceph-mon[112940]: pgmap v736: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:00.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:00 smithi066 ceph-mon[85781]: pgmap v737: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:00.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:00 smithi001 ceph-mon[112940]: pgmap v737: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:02.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:02 smithi066 ceph-mon[85781]: pgmap v738: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:02.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:02 smithi001 ceph-mon[112940]: pgmap v738: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:04.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:04 smithi066 ceph-mon[85781]: pgmap v739: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:04.780 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:04 smithi001 ceph-mon[112940]: pgmap v739: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:05.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.467+0000 7fc42a200700 1 -- 172.21.15.1:0/827544054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424071f60 msgr2=0x7fc424070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:05.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.467+0000 7fc42a200700 1 --2- 172.21.15.1:0/827544054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424071f60 0x7fc424070330 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc414004970 tx=0x7fc414009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:05.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.467+0000 7fc42a200700 1 -- 172.21.15.1:0/827544054 shutdown_connections 2024-02-28T16:59:05.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.467+0000 7fc42a200700 1 --2- 172.21.15.1:0/827544054 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc424070870 0x7fc424070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.467+0000 7fc42a200700 1 --2- 172.21.15.1:0/827544054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424071f60 0x7fc424070330 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.467+0000 7fc42a200700 1 -- 172.21.15.1:0/827544054 >> 172.21.15.1:0/827544054 conn(0x7fc4240f8ae0 msgr2=0x7fc4240faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:05.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.467+0000 7fc42a200700 1 -- 172.21.15.1:0/827544054 shutdown_connections 2024-02-28T16:59:05.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.467+0000 7fc42a200700 1 -- 172.21.15.1:0/827544054 wait complete. 2024-02-28T16:59:05.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.468+0000 7fc42a200700 1 Processor -- start 2024-02-28T16:59:05.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.468+0000 7fc42a200700 1 -- start start 2024-02-28T16:59:05.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc42a200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424070870 0x7fc42411a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:05.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc42a200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc424071f60 0x7fc42411a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:05.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc42a200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42411ad10 con 0x7fc424070870 2024-02-28T16:59:05.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc42a200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc424113f90 con 0x7fc424071f60 2024-02-28T16:59:05.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc423400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424070870 0x7fc42411a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:05.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc423400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424070870 0x7fc42411a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50324/0 (socket says 172.21.15.1:50324) 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc423400700 1 -- 172.21.15.1:0/2445430438 learned_addr learned my addr 172.21.15.1:0/2445430438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc422a00700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc424071f60 0x7fc42411a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc423400700 1 -- 172.21.15.1:0/2445430438 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc424071f60 msgr2=0x7fc42411a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc423400700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc424071f60 0x7fc42411a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc423400700 1 -- 172.21.15.1:0/2445430438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414005040 con 0x7fc424070870 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.469+0000 7fc422a00700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc424071f60 0x7fc42411a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.470+0000 7fc423400700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424070870 0x7fc42411a0f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc414004f50 tx=0x7fc414009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.470+0000 7fc41b400700 1 -- 172.21.15.1:0/2445430438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc41400e040 con 0x7fc424070870 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.470+0000 7fc41b400700 1 -- 172.21.15.1:0/2445430438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc41400bc10 con 0x7fc424070870 2024-02-28T16:59:05.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.470+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4241141b0 con 0x7fc424070870 2024-02-28T16:59:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.470+0000 7fc41b400700 1 -- 172.21.15.1:0/2445430438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc41401c630 con 0x7fc424070870 2024-02-28T16:59:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.470+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4241146f0 con 0x7fc424070870 2024-02-28T16:59:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.472+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4240fe400 con 0x7fc424070870 2024-02-28T16:59:05.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.473+0000 7fc41b400700 1 -- 172.21.15.1:0/2445430438 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc41400bd80 con 0x7fc424070870 2024-02-28T16:59:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.474+0000 7fc41b400700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40806d7f0 0x7fc40806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.474+0000 7fc41b400700 1 -- 172.21.15.1:0/2445430438 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc41408e2c0 con 0x7fc424070870 2024-02-28T16:59:05.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.474+0000 7fc422a00700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40806d7f0 0x7fc40806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:05.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.475+0000 7fc422a00700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40806d7f0 0x7fc40806fca0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc410006fd0 tx=0x7fc410008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.479+0000 7fc41b400700 1 -- 172.21.15.1:0/2445430438 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc41405b620 con 0x7fc424070870 2024-02-28T16:59:05.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.721+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4241156d0 con 0x7fc40806d7f0 2024-02-28T16:59:05.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.724+0000 7fc41b400700 1 -- 172.21.15.1:0/2445430438 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc4241156d0 con 0x7fc40806d7f0 2024-02-28T16:59:05.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40806d7f0 msgr2=0x7fc40806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40806d7f0 0x7fc40806fca0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc410006fd0 tx=0x7fc410008040 comp rx=0 tx=0).stop 2024-02-28T16:59:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424070870 msgr2=0x7fc42411a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424070870 0x7fc42411a0f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc414004f50 tx=0x7fc414009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 shutdown_connections 2024-02-28T16:59:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc40806d7f0 0x7fc40806fca0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc424070870 0x7fc42411a0f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 --2- 172.21.15.1:0/2445430438 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc424071f60 0x7fc42411a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 >> 172.21.15.1:0/2445430438 conn(0x7fc4240f8ae0 msgr2=0x7fc4240fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 shutdown_connections 2024-02-28T16:59:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.727+0000 7fc42a200700 1 -- 172.21.15.1:0/2445430438 wait complete. 2024-02-28T16:59:05.741 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:59:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.885+0000 7f4c5f200700 1 -- 172.21.15.1:0/516175809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 msgr2=0x7f4c581092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.885+0000 7f4c5f200700 1 --2- 172.21.15.1:0/516175809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 0x7f4c581092c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4c40004970 tx=0x7f4c40009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.885+0000 7f4c5f200700 1 -- 172.21.15.1:0/516175809 shutdown_connections 2024-02-28T16:59:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.885+0000 7f4c5f200700 1 --2- 172.21.15.1:0/516175809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 0x7f4c581092c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.885+0000 7f4c5f200700 1 --2- 172.21.15.1:0/516175809 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c580702d0 0x7f4c580706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.885+0000 7f4c5f200700 1 -- 172.21.15.1:0/516175809 >> 172.21.15.1:0/516175809 conn(0x7f4c580f8ae0 msgr2=0x7f4c580faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.885+0000 7f4c5f200700 1 -- 172.21.15.1:0/516175809 shutdown_connections 2024-02-28T16:59:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.885+0000 7f4c5f200700 1 -- 172.21.15.1:0/516175809 wait complete. 2024-02-28T16:59:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.886+0000 7f4c5f200700 1 Processor -- start 2024-02-28T16:59:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.886+0000 7f4c5f200700 1 -- start start 2024-02-28T16:59:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.886+0000 7f4c5f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c580702d0 0x7f4c58115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.886+0000 7f4c5f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 0x7f4c58116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.886+0000 7f4c5f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581167a0 con 0x7f4c58070be0 2024-02-28T16:59:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.886+0000 7f4c5f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5810faf0 con 0x7f4c580702d0 2024-02-28T16:59:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.887+0000 7f4c4fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 0x7f4c58116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.887+0000 7f4c4fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 0x7f4c58116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50336/0 (socket says 172.21.15.1:50336) 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.887+0000 7f4c4fe00700 1 -- 172.21.15.1:0/2532934795 learned_addr learned my addr 172.21.15.1:0/2532934795 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.887+0000 7f4c57e00700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c580702d0 0x7f4c58115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.887+0000 7f4c4fe00700 1 -- 172.21.15.1:0/2532934795 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c580702d0 msgr2=0x7f4c58115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.887+0000 7f4c4fe00700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c580702d0 0x7f4c58115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.887+0000 7f4c4fe00700 1 -- 172.21.15.1:0/2532934795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40005040 con 0x7f4c58070be0 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.887+0000 7f4c4fe00700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 0x7f4c58116150 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4c40004f50 tx=0x7f4c40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.888+0000 7f4c54c00700 1 -- 172.21.15.1:0/2532934795 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4000e040 con 0x7f4c58070be0 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.888+0000 7f4c54c00700 1 -- 172.21.15.1:0/2532934795 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c40004290 con 0x7f4c58070be0 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.888+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5810fd70 con 0x7f4c58070be0 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.888+0000 7f4c54c00700 1 -- 172.21.15.1:0/2532934795 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4001c630 con 0x7f4c58070be0 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.888+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581102b0 con 0x7f4c58070be0 2024-02-28T16:59:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.889+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c58106980 con 0x7f4c58070be0 2024-02-28T16:59:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.890+0000 7f4c54c00700 1 -- 172.21.15.1:0/2532934795 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4c4000bc10 con 0x7f4c58070be0 2024-02-28T16:59:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.891+0000 7f4c54c00700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c3c06d7f0 0x7f4c3c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.891+0000 7f4c54c00700 1 -- 172.21.15.1:0/2532934795 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4c4008e2b0 con 0x7f4c58070be0 2024-02-28T16:59:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.891+0000 7f4c57e00700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c3c06d7f0 0x7f4c3c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.892+0000 7f4c57e00700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c3c06d7f0 0x7f4c3c06fca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4c48004990 tx=0x7f4c48009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:05.896+0000 7f4c54c00700 1 -- 172.21.15.1:0/2532934795 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c40057b60 con 0x7f4c58070be0 2024-02-28T16:59:06.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.133+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c5804bf40 con 0x7f4c3c06d7f0 2024-02-28T16:59:06.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.135+0000 7f4c54c00700 1 -- 172.21.15.1:0/2532934795 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4c5804bf40 con 0x7f4c3c06d7f0 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c3c06d7f0 msgr2=0x7f4c3c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c3c06d7f0 0x7f4c3c06fca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4c48004990 tx=0x7f4c48009ee0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 msgr2=0x7f4c58116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 0x7f4c58116150 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4c40004f50 tx=0x7f4c40009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 shutdown_connections 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c3c06d7f0 0x7f4c3c06fca0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c580702d0 0x7f4c58115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 --2- 172.21.15.1:0/2532934795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c58070be0 0x7f4c58116150 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.137+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 >> 172.21.15.1:0/2532934795 conn(0x7f4c580f8ae0 msgr2=0x7f4c58103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:06.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.138+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 shutdown_connections 2024-02-28T16:59:06.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.138+0000 7f4c5f200700 1 -- 172.21.15.1:0/2532934795 wait complete. 2024-02-28T16:59:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.292+0000 7f147ba00700 1 -- 172.21.15.1:0/2182206232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 msgr2=0x7f1474105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:06.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.292+0000 7f147ba00700 1 --2- 172.21.15.1:0/2182206232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 0x7f1474105540 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1468004940 tx=0x7f1468009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:06.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.293+0000 7f147ba00700 1 -- 172.21.15.1:0/2182206232 shutdown_connections 2024-02-28T16:59:06.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.293+0000 7f147ba00700 1 --2- 172.21.15.1:0/2182206232 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1474105b10 0x7f14740ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.293+0000 7f147ba00700 1 --2- 172.21.15.1:0/2182206232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 0x7f1474105540 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.293+0000 7f147ba00700 1 -- 172.21.15.1:0/2182206232 >> 172.21.15.1:0/2182206232 conn(0x7f14740face0 msgr2=0x7f14740fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:06.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.293+0000 7f147ba00700 1 -- 172.21.15.1:0/2182206232 shutdown_connections 2024-02-28T16:59:06.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.293+0000 7f147ba00700 1 -- 172.21.15.1:0/2182206232 wait complete. 2024-02-28T16:59:06.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.294+0000 7f147ba00700 1 Processor -- start 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.294+0000 7f147ba00700 1 -- start start 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.294+0000 7f147ba00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 0x7f1474115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.294+0000 7f147ba00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1474105b10 0x7f14741161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.294+0000 7f147ba00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14741167f0 con 0x7f1474105170 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.294+0000 7f147ba00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147410fb40 con 0x7f1474105b10 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.295+0000 7f1479000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 0x7f1474115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.295+0000 7f1479000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 0x7f1474115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50356/0 (socket says 172.21.15.1:50356) 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.295+0000 7f1479000700 1 -- 172.21.15.1:0/3634164246 learned_addr learned my addr 172.21.15.1:0/3634164246 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.295+0000 7f1479000700 1 -- 172.21.15.1:0/3634164246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1474105b10 msgr2=0x7f14741161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.295+0000 7f1479000700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1474105b10 0x7f14741161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.295+0000 7f1479000700 1 -- 172.21.15.1:0/3634164246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1468005040 con 0x7f1474105170 2024-02-28T16:59:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.296+0000 7f1479000700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 0x7f1474115c60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1468009150 tx=0x7f1468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.296+0000 7f1470c00700 1 -- 172.21.15.1:0/3634164246 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146800e040 con 0x7f1474105170 2024-02-28T16:59:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.296+0000 7f1470c00700 1 -- 172.21.15.1:0/3634164246 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f146800bbf0 con 0x7f1474105170 2024-02-28T16:59:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.296+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147410fdc0 con 0x7f1474105170 2024-02-28T16:59:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.296+0000 7f1470c00700 1 -- 172.21.15.1:0/3634164246 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146801c630 con 0x7f1474105170 2024-02-28T16:59:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.296+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1474110300 con 0x7f1474105170 2024-02-28T16:59:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.297+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1474102480 con 0x7f1474105170 2024-02-28T16:59:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.298+0000 7f1470c00700 1 -- 172.21.15.1:0/3634164246 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f146800bd60 con 0x7f1474105170 2024-02-28T16:59:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.299+0000 7f1470c00700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f145c06d7f0 0x7f145c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.299+0000 7f1470c00700 1 -- 172.21.15.1:0/3634164246 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f146808e200 con 0x7f1474105170 2024-02-28T16:59:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.299+0000 7f1473e00700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f145c06d7f0 0x7f145c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.300+0000 7f1473e00700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f145c06d7f0 0x7f145c06fca0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1474111540 tx=0x7f1460009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.304+0000 7f1470c00700 1 -- 172.21.15.1:0/3634164246 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1468057b70 con 0x7f1474105170 2024-02-28T16:59:06.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.540+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f147404bf70 con 0x7f145c06d7f0 2024-02-28T16:59:06.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.568+0000 7f1470c00700 1 -- 172.21.15.1:0/3634164246 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f146000f040 con 0x7f145c06d7f0 2024-02-28T16:59:06.567 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:59:06.567 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (25m) 118s ago 39m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (21m) 118s ago 39m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (21m) 2m ago 38m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (24m) 118s ago 39m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (17m) 118s ago 32m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (17m) 118s ago 32m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (32m) 2m ago 32m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (2m) 2m ago 32m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (27m) 118s ago 41m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (26m) 2m ago 38m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (23m) 118s ago 41m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (22m) 2m ago 38m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:59:06.568 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (26m) 118s ago 39m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:59:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (26m) 2m ago 38m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:59:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (21m) 118s ago 37m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:59:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (20m) 118s ago 36m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:59:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (20m) 118s ago 35m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:59:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (19m) 2m ago 35m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:59:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (19m) 2m ago 34m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:59:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (18m) 2m ago 34m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:59:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (25m) 118s ago 38m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:59:06.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f145c06d7f0 msgr2=0x7f145c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f145c06d7f0 0x7f145c06fca0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1474111540 tx=0x7f1460009ce0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 msgr2=0x7f1474115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 0x7f1474115c60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1468009150 tx=0x7f1468009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 shutdown_connections 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f145c06d7f0 0x7f145c06fca0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1474105170 0x7f1474115c60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 --2- 172.21.15.1:0/3634164246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1474105b10 0x7f14741161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.571+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 >> 172.21.15.1:0/3634164246 conn(0x7f14740face0 msgr2=0x7f14740fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.572+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 shutdown_connections 2024-02-28T16:59:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.572+0000 7f147ba00700 1 -- 172.21.15.1:0/3634164246 wait complete. 2024-02-28T16:59:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.729+0000 7fcb39c00700 1 -- 172.21.15.1:0/65279440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fc6e0 msgr2=0x7fcb34100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.729+0000 7fcb39c00700 1 --2- 172.21.15.1:0/65279440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fc6e0 0x7fcb34100e20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcb20004940 tx=0x7fcb20009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.729+0000 7fcb39c00700 1 -- 172.21.15.1:0/65279440 shutdown_connections 2024-02-28T16:59:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.729+0000 7fcb39c00700 1 --2- 172.21.15.1:0/65279440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fc6e0 0x7fcb34100e20 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.729+0000 7fcb39c00700 1 --2- 172.21.15.1:0/65279440 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb340fbdd0 0x7fcb340fc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.729+0000 7fcb39c00700 1 -- 172.21.15.1:0/65279440 >> 172.21.15.1:0/65279440 conn(0x7fcb34071d90 msgr2=0x7fcb34072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:06.729 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:06 smithi001 ceph-mon[112940]: pgmap v740: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:06.729 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:06 smithi001 ceph-mon[112940]: from='client.35346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:06.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.729+0000 7fcb39c00700 1 -- 172.21.15.1:0/65279440 shutdown_connections 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.729+0000 7fcb39c00700 1 -- 172.21.15.1:0/65279440 wait complete. 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.730+0000 7fcb39c00700 1 Processor -- start 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.730+0000 7fcb39c00700 1 -- start start 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb39c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fbdd0 0x7fcb3411a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb39c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb340fc6e0 0x7fcb3411a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb39c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3411ac80 con 0x7fcb340fbdd0 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb39c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34113f90 con 0x7fcb340fc6e0 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb32a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fbdd0 0x7fcb3411a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb32a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fbdd0 0x7fcb3411a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50368/0 (socket says 172.21.15.1:50368) 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb32a00700 1 -- 172.21.15.1:0/1156003379 learned_addr learned my addr 172.21.15.1:0/1156003379 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb32a00700 1 -- 172.21.15.1:0/1156003379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb340fc6e0 msgr2=0x7fcb3411a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb32000700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb340fc6e0 0x7fcb3411a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.731+0000 7fcb32a00700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb340fc6e0 0x7fcb3411a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.732+0000 7fcb32a00700 1 -- 172.21.15.1:0/1156003379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb20005040 con 0x7fcb340fbdd0 2024-02-28T16:59:06.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.732+0000 7fcb32a00700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fbdd0 0x7fcb3411a0f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400c8c0 tx=0x7fcb240044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:06.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.732+0000 7fcb2aa00700 1 -- 172.21.15.1:0/1156003379 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb2400a980 con 0x7fcb340fbdd0 2024-02-28T16:59:06.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.732+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb34114270 con 0x7fcb340fbdd0 2024-02-28T16:59:06.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.732+0000 7fcb2aa00700 1 -- 172.21.15.1:0/1156003379 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb2400ce60 con 0x7fcb340fbdd0 2024-02-28T16:59:06.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.732+0000 7fcb2aa00700 1 -- 172.21.15.1:0/1156003379 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb240115e0 con 0x7fcb340fbdd0 2024-02-28T16:59:06.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.732+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb341147b0 con 0x7fcb340fbdd0 2024-02-28T16:59:06.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.734+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb340fe540 con 0x7fcb340fbdd0 2024-02-28T16:59:06.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.735+0000 7fcb2aa00700 1 -- 172.21.15.1:0/1156003379 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcb24012400 con 0x7fcb340fbdd0 2024-02-28T16:59:06.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.736+0000 7fcb2aa00700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb1806d7f0 0x7fcb1806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:06.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.736+0000 7fcb2aa00700 1 -- 172.21.15.1:0/1156003379 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcb2408cf70 con 0x7fcb340fbdd0 2024-02-28T16:59:06.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.736+0000 7fcb32000700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb1806d7f0 0x7fcb1806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:06.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.737+0000 7fcb32000700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb1806d7f0 0x7fcb1806fca0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fcb34115990 tx=0x7fcb20009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:06.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:06.741+0000 7fcb2aa00700 1 -- 172.21.15.1:0/1156003379 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcb2405a350 con 0x7fcb340fbdd0 2024-02-28T16:59:06.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:06 smithi066 ceph-mon[85781]: pgmap v740: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:06.757 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:06 smithi066 ceph-mon[85781]: from='client.35346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:07.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.053+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcb3404bed0 con 0x7fcb340fbdd0 2024-02-28T16:59:07.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.054+0000 7fcb2aa00700 1 -- 172.21.15.1:0/1156003379 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fcb240126b0 con 0x7fcb340fbdd0 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:59:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T16:59:07.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:59:07.055 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:59:07.055 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:59:07.055 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T16:59:07.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:59:07.055 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:59:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.057+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb1806d7f0 msgr2=0x7fcb1806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.057+0000 7fcb39c00700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb1806d7f0 0x7fcb1806fca0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fcb34115990 tx=0x7fcb20009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.057+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fbdd0 msgr2=0x7fcb3411a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.057+0000 7fcb39c00700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fbdd0 0x7fcb3411a0f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400c8c0 tx=0x7fcb240044c0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.057+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 shutdown_connections 2024-02-28T16:59:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.057+0000 7fcb39c00700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcb1806d7f0 0x7fcb1806fca0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.057+0000 7fcb39c00700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb340fbdd0 0x7fcb3411a0f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.057+0000 7fcb39c00700 1 --2- 172.21.15.1:0/1156003379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcb340fc6e0 0x7fcb3411a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.058+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 >> 172.21.15.1:0/1156003379 conn(0x7fcb34071d90 msgr2=0x7fcb3410d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:07.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.058+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 shutdown_connections 2024-02-28T16:59:07.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.058+0000 7fcb39c00700 1 -- 172.21.15.1:0/1156003379 wait complete. 2024-02-28T16:59:07.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.208+0000 7f502d800700 1 -- 172.21.15.1:0/927083062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 msgr2=0x7f5028070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.208+0000 7f502d800700 1 --2- 172.21.15.1:0/927083062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 0x7f5028070700 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5018004970 tx=0x7f5018009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.208+0000 7f502d800700 1 -- 172.21.15.1:0/927083062 shutdown_connections 2024-02-28T16:59:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.208+0000 7f502d800700 1 --2- 172.21.15.1:0/927083062 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5028070c40 0x7f5028109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.208+0000 7f502d800700 1 --2- 172.21.15.1:0/927083062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 0x7f5028070700 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.208+0000 7f502d800700 1 -- 172.21.15.1:0/927083062 >> 172.21.15.1:0/927083062 conn(0x7f50280f8ae0 msgr2=0x7f50280faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.208+0000 7f502d800700 1 -- 172.21.15.1:0/927083062 shutdown_connections 2024-02-28T16:59:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.209+0000 7f502d800700 1 -- 172.21.15.1:0/927083062 wait complete. 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.209+0000 7f502d800700 1 Processor -- start 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.210+0000 7f502d800700 1 -- start start 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.210+0000 7f502d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 0x7f5028115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.210+0000 7f502d800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5028070c40 0x7f5028116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.210+0000 7f502d800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50281168e0 con 0x7f5028070330 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.210+0000 7f502d800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502810fb60 con 0x7f5028070c40 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.210+0000 7f5026a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 0x7f5028115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.210+0000 7f5026a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 0x7f5028115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50374/0 (socket says 172.21.15.1:50374) 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.210+0000 7f5026a00700 1 -- 172.21.15.1:0/2165657037 learned_addr learned my addr 172.21.15.1:0/2165657037 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.211+0000 7f5026a00700 1 -- 172.21.15.1:0/2165657037 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5028070c40 msgr2=0x7f5028116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.211+0000 7f5026a00700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5028070c40 0x7f5028116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.211+0000 7f5026a00700 1 -- 172.21.15.1:0/2165657037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5018005040 con 0x7f5028070330 2024-02-28T16:59:07.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.211+0000 7f5026a00700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 0x7f5028115cc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5018004f50 tx=0x7f5018009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:07.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.211+0000 7f501ea00700 1 -- 172.21.15.1:0/2165657037 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f501800e040 con 0x7f5028070330 2024-02-28T16:59:07.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.211+0000 7f501ea00700 1 -- 172.21.15.1:0/2165657037 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5018004290 con 0x7f5028070330 2024-02-28T16:59:07.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.211+0000 7f501ea00700 1 -- 172.21.15.1:0/2165657037 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f501801c630 con 0x7f5028070330 2024-02-28T16:59:07.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.211+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502810fd80 con 0x7f5028070330 2024-02-28T16:59:07.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.212+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50281102c0 con 0x7f5028070330 2024-02-28T16:59:07.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.213+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5028106920 con 0x7f5028070330 2024-02-28T16:59:07.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.214+0000 7f501ea00700 1 -- 172.21.15.1:0/2165657037 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f501800bc10 con 0x7f5028070330 2024-02-28T16:59:07.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.215+0000 7f501ea00700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f500c06d7f0 0x7f500c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:07.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.215+0000 7f501ea00700 1 -- 172.21.15.1:0/2165657037 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f501808e220 con 0x7f5028070330 2024-02-28T16:59:07.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.215+0000 7f5026000700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f500c06d7f0 0x7f500c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:07.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.216+0000 7f5026000700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f500c06d7f0 0x7f500c06fca0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5014006fd0 tx=0x7f5014006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:07.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.219+0000 7f501ea00700 1 -- 172.21.15.1:0/2165657037 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f501805b600 con 0x7f5028070330 2024-02-28T16:59:07.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.491+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f50281112a0 con 0x7f5028070330 2024-02-28T16:59:07.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.492+0000 7f501ea00700 1 -- 172.21.15.1:0/2165657037 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f5018013e60 con 0x7f5028070330 2024-02-28T16:59:07.491 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T16:59:07.491 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:59:07.491 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:59:07.491 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:59:07.491 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:59:07.491 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:59:07.492 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:59:07.493 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:59:07.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:59:07.494 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:59:07.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:59:07.494 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:59:07.494 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:59:07.494 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:59:07.494 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f500c06d7f0 msgr2=0x7f500c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f500c06d7f0 0x7f500c06fca0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5014006fd0 tx=0x7f5014006db0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 msgr2=0x7f5028115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 0x7f5028115cc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5018004f50 tx=0x7f5018009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 shutdown_connections 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f500c06d7f0 0x7f500c06fca0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5028070330 0x7f5028115cc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 --2- 172.21.15.1:0/2165657037 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5028070c40 0x7f5028116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 >> 172.21.15.1:0/2165657037 conn(0x7f50280f8ae0 msgr2=0x7f5028103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 shutdown_connections 2024-02-28T16:59:07.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.495+0000 7f502d800700 1 -- 172.21.15.1:0/2165657037 wait complete. 2024-02-28T16:59:07.496 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T16:59:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.652+0000 7f3e7e800700 1 -- 172.21.15.1:0/1362214565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 msgr2=0x7f3e780ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.652+0000 7f3e7e800700 1 --2- 172.21.15.1:0/1362214565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 0x7f3e780ff180 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3e64004940 tx=0x7f3e64009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.652+0000 7f3e7e800700 1 -- 172.21.15.1:0/1362214565 shutdown_connections 2024-02-28T16:59:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.652+0000 7f3e7e800700 1 --2- 172.21.15.1:0/1362214565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 0x7f3e780ff180 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.652+0000 7f3e7e800700 1 --2- 172.21.15.1:0/1362214565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e78105170 0x7f3e78105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.652+0000 7f3e7e800700 1 -- 172.21.15.1:0/1362214565 >> 172.21.15.1:0/1362214565 conn(0x7f3e780face0 msgr2=0x7f3e780fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.652+0000 7f3e7e800700 1 -- 172.21.15.1:0/1362214565 shutdown_connections 2024-02-28T16:59:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.652+0000 7f3e7e800700 1 -- 172.21.15.1:0/1362214565 wait complete. 2024-02-28T16:59:07.652 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:07 smithi001 ceph-mon[112940]: from='client.35350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:07.652 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:07 smithi001 ceph-mon[112940]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:07.652 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:07 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1156003379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:59:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.653+0000 7f3e7e800700 1 Processor -- start 2024-02-28T16:59:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.653+0000 7f3e7e800700 1 -- start start 2024-02-28T16:59:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.653+0000 7f3e7e800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e78105170 0x7f3e78115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.653+0000 7f3e7e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 0x7f3e78116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.653+0000 7f3e7e800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e78116a00 con 0x7f3e78105b10 2024-02-28T16:59:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.653+0000 7f3e7e800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7810fc80 con 0x7f3e78105170 2024-02-28T16:59:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.654+0000 7f3e76a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 0x7f3e78116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.654+0000 7f3e76a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 0x7f3e78116320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50394/0 (socket says 172.21.15.1:50394) 2024-02-28T16:59:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.654+0000 7f3e76a00700 1 -- 172.21.15.1:0/3285621455 learned_addr learned my addr 172.21.15.1:0/3285621455 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.654+0000 7f3e77400700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e78105170 0x7f3e78115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.654+0000 7f3e76a00700 1 -- 172.21.15.1:0/3285621455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e78105170 msgr2=0x7f3e78115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.654+0000 7f3e76a00700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e78105170 0x7f3e78115de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.654+0000 7f3e76a00700 1 -- 172.21.15.1:0/3285621455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e64005040 con 0x7f3e78105b10 2024-02-28T16:59:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.654+0000 7f3e76a00700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 0x7f3e78116320 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009100 tx=0x7f3e64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.655+0000 7f3e6f400700 1 -- 172.21.15.1:0/3285621455 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6400e040 con 0x7f3e78105b10 2024-02-28T16:59:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.655+0000 7f3e6f400700 1 -- 172.21.15.1:0/3285621455 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e64004290 con 0x7f3e78105b10 2024-02-28T16:59:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.655+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7810fea0 con 0x7f3e78105b10 2024-02-28T16:59:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.655+0000 7f3e6f400700 1 -- 172.21.15.1:0/3285621455 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6401c3f0 con 0x7f3e78105b10 2024-02-28T16:59:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.655+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e781103e0 con 0x7f3e78105b10 2024-02-28T16:59:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.656+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e780fe900 con 0x7f3e78105b10 2024-02-28T16:59:07.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.658+0000 7f3e6f400700 1 -- 172.21.15.1:0/3285621455 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3e6400bbf0 con 0x7f3e78105b10 2024-02-28T16:59:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.659+0000 7f3e6f400700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3e5c06d7f0 0x7f3e5c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.659+0000 7f3e6f400700 1 -- 172.21.15.1:0/3285621455 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3e6408e1c0 con 0x7f3e78105b10 2024-02-28T16:59:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.659+0000 7f3e77400700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3e5c06d7f0 0x7f3e5c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.660+0000 7f3e77400700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3e5c06d7f0 0x7f3e5c06fca0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3e68006fd0 tx=0x7f3e68008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:07.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.663+0000 7f3e6f400700 1 -- 172.21.15.1:0/3285621455 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e6405b5a0 con 0x7f3e78105b10 2024-02-28T16:59:07.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:07 smithi066 ceph-mon[85781]: from='client.35350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:07.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:07 smithi066 ceph-mon[85781]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:07.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:07 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1156003379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:59:07.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.897+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e78111430 con 0x7f3e5c06d7f0 2024-02-28T16:59:07.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.899+0000 7f3e6f400700 1 -- 172.21.15.1:0/3285621455 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3e78111430 con 0x7f3e5c06d7f0 2024-02-28T16:59:07.899 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:59:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:59:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:59:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:59:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:59:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:59:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:59:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:59:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:59:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:59:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T16:59:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:59:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:59:07.900 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3e5c06d7f0 msgr2=0x7f3e5c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3e5c06d7f0 0x7f3e5c06fca0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3e68006fd0 tx=0x7f3e68008040 comp rx=0 tx=0).stop 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 msgr2=0x7f3e78116320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 0x7f3e78116320 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009100 tx=0x7f3e64009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 shutdown_connections 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3e5c06d7f0 0x7f3e5c06fca0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e78105170 0x7f3e78115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 --2- 172.21.15.1:0/3285621455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e78105b10 0x7f3e78116320 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.902+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 >> 172.21.15.1:0/3285621455 conn(0x7f3e780face0 msgr2=0x7f3e780fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:07.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.903+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 shutdown_connections 2024-02-28T16:59:07.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:07.903+0000 7f3e7e800700 1 -- 172.21.15.1:0/3285621455 wait complete. 2024-02-28T16:59:08.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.056+0000 7f18b8000700 1 -- 172.21.15.1:0/1590545016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fbdd0 msgr2=0x7f18b00fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:08.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.056+0000 7f18b8000700 1 --2- 172.21.15.1:0/1590545016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fbdd0 0x7f18b00fc1a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f18a4004940 tx=0x7f18a4009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:08.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.057+0000 7f18b8000700 1 -- 172.21.15.1:0/1590545016 shutdown_connections 2024-02-28T16:59:08.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.057+0000 7f18b8000700 1 --2- 172.21.15.1:0/1590545016 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18b00fc6e0 0x7f18b0109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:08.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.057+0000 7f18b8000700 1 --2- 172.21.15.1:0/1590545016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fbdd0 0x7f18b00fc1a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:08.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.057+0000 7f18b8000700 1 -- 172.21.15.1:0/1590545016 >> 172.21.15.1:0/1590545016 conn(0x7f18b0071d90 msgr2=0x7f18b0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:08.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.057+0000 7f18b8000700 1 -- 172.21.15.1:0/1590545016 shutdown_connections 2024-02-28T16:59:08.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.057+0000 7f18b8000700 1 -- 172.21.15.1:0/1590545016 wait complete. 2024-02-28T16:59:08.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.058+0000 7f18b8000700 1 Processor -- start 2024-02-28T16:59:08.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.058+0000 7f18b8000700 1 -- start start 2024-02-28T16:59:08.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.058+0000 7f18b8000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18b00fbdd0 0x7f18b0115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b8000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fc6e0 0x7f18b01160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b8000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b0116740 con 0x7f18b00fc6e0 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b8000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b010fa70 con 0x7f18b00fbdd0 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fc6e0 0x7f18b01160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fc6e0 0x7f18b01160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50412/0 (socket says 172.21.15.1:50412) 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b4c00700 1 -- 172.21.15.1:0/1507339753 learned_addr learned my addr 172.21.15.1:0/1507339753 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b5600700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18b00fbdd0 0x7f18b0115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b4c00700 1 -- 172.21.15.1:0/1507339753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18b00fbdd0 msgr2=0x7f18b0115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b4c00700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18b00fbdd0 0x7f18b0115bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.059+0000 7f18b4c00700 1 -- 172.21.15.1:0/1507339753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a4005040 con 0x7f18b00fc6e0 2024-02-28T16:59:08.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.060+0000 7f18b4c00700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fc6e0 0x7f18b01160f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f189c00eb40 tx=0x7f189c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.060+0000 7f18ad600700 1 -- 172.21.15.1:0/1507339753 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f189c007cc0 con 0x7f18b00fc6e0 2024-02-28T16:59:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.060+0000 7f18ad600700 1 -- 172.21.15.1:0/1507339753 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f189c00f040 con 0x7f18b00fc6e0 2024-02-28T16:59:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.060+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b010fcf0 con 0x7f18b00fc6e0 2024-02-28T16:59:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.060+0000 7f18ad600700 1 -- 172.21.15.1:0/1507339753 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f189c009620 con 0x7f18b00fc6e0 2024-02-28T16:59:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.060+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b0110230 con 0x7f18b00fc6e0 2024-02-28T16:59:08.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.061+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b0106b70 con 0x7f18b00fc6e0 2024-02-28T16:59:08.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.062+0000 7f18ad600700 1 -- 172.21.15.1:0/1507339753 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f189c009780 con 0x7f18b00fc6e0 2024-02-28T16:59:08.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.064+0000 7f18ad600700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f189806d7f0 0x7f189806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:08.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.064+0000 7f18ad600700 1 -- 172.21.15.1:0/1507339753 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f189c08cc00 con 0x7f18b00fc6e0 2024-02-28T16:59:08.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.064+0000 7f18b5600700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f189806d7f0 0x7f189806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:08.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.065+0000 7f18b5600700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f189806d7f0 0x7f189806fca0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b0d0 tx=0x7f18a4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.068+0000 7f18ad600700 1 -- 172.21.15.1:0/1507339753 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f189c059f90 con 0x7f18b00fc6e0 2024-02-28T16:59:08.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.375+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f18b004bed0 con 0x7f18b00fc6e0 2024-02-28T16:59:08.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.376+0000 7f18ad600700 1 -- 172.21.15.1:0/1507339753 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f189c059b20 con 0x7f18b00fc6e0 2024-02-28T16:59:08.376 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:59:08.376 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T16:59:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:59:08.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T16:59:08.376 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:59:08.377 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:59:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:59:08.378 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:59:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:59:08.378 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:59:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f189806d7f0 msgr2=0x7f189806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f189806d7f0 0x7f189806fca0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b0d0 tx=0x7f18a4009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fc6e0 msgr2=0x7f18b01160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fc6e0 0x7f18b01160f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f189c00eb40 tx=0x7f189c00b040 comp rx=0 tx=0).stop 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 shutdown_connections 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f189806d7f0 0x7f189806fca0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18b00fbdd0 0x7f18b0115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 --2- 172.21.15.1:0/1507339753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b00fc6e0 0x7f18b01160f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 >> 172.21.15.1:0/1507339753 conn(0x7f18b0071d90 msgr2=0x7f18b00fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.379+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 shutdown_connections 2024-02-28T16:59:08.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:08.380+0000 7f18b8000700 1 -- 172.21.15.1:0/1507339753 wait complete. 2024-02-28T16:59:08.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:08 smithi066 ceph-mon[85781]: pgmap v741: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:08.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:08 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2165657037' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:59:08.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:08 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1507339753' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:59:08.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:08 smithi001 ceph-mon[112940]: pgmap v741: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:08.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:08 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2165657037' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:59:08.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:08 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1507339753' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:59:09.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:09 smithi066 ceph-mon[85781]: from='client.35366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:09.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:09 smithi001 ceph-mon[112940]: from='client.35366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:10.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:10 smithi066 ceph-mon[85781]: pgmap v742: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:10.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:10 smithi001 ceph-mon[112940]: pgmap v742: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:11.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:59:11.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:59:12.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:12 smithi066 ceph-mon[85781]: pgmap v743: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:12.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:12 smithi001 ceph-mon[112940]: pgmap v743: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:14.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:14 smithi066 ceph-mon[85781]: pgmap v744: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:14.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:14 smithi001 ceph-mon[112940]: pgmap v744: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:16.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:16 smithi066 ceph-mon[85781]: pgmap v745: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:16.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:16 smithi001 ceph-mon[112940]: pgmap v745: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:18.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:18 smithi066 ceph-mon[85781]: pgmap v746: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:18.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:18 smithi001 ceph-mon[112940]: pgmap v746: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:20.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:20 smithi066 ceph-mon[85781]: pgmap v747: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:20.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:20 smithi001 ceph-mon[112940]: pgmap v747: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:22.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:22 smithi066 ceph-mon[85781]: pgmap v748: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:22.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:22 smithi001 ceph-mon[112940]: pgmap v748: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:24.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:24 smithi066 ceph-mon[85781]: pgmap v749: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:24.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:24 smithi001 ceph-mon[112940]: pgmap v749: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:26.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:26 smithi001 ceph-mon[112940]: pgmap v750: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:26.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:59:26.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:26 smithi066 ceph-mon[85781]: pgmap v750: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:26.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:59:28.758 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:28 smithi066 ceph-mon[85781]: pgmap v751: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:28.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:28 smithi001 ceph-mon[112940]: pgmap v751: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:30.780 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:30 smithi001 ceph-mon[112940]: pgmap v752: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:31.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:30 smithi066 ceph-mon[85781]: pgmap v752: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:32.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:32 smithi001 ceph-mon[112940]: pgmap v753: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:33.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:32 smithi066 ceph-mon[85781]: pgmap v753: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:34.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:34 smithi001 ceph-mon[112940]: pgmap v754: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:35.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:34 smithi066 ceph-mon[85781]: pgmap v754: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:36.780 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:36 smithi001 ceph-mon[112940]: pgmap v755: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:37.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:36 smithi066 ceph-mon[85781]: pgmap v755: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:38.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.555+0000 7f5c9fa00700 1 -- 172.21.15.1:0/2476017208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c981038f0 msgr2=0x7f5c980fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:38.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.555+0000 7f5c9fa00700 1 --2- 172.21.15.1:0/2476017208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c981038f0 0x7f5c980fcf60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5c84004940 tx=0x7f5c84009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:38.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.555+0000 7f5c9fa00700 1 -- 172.21.15.1:0/2476017208 shutdown_connections 2024-02-28T16:59:38.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.555+0000 7f5c9fa00700 1 --2- 172.21.15.1:0/2476017208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c981038f0 0x7f5c980fcf60 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:38.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.555+0000 7f5c9fa00700 1 --2- 172.21.15.1:0/2476017208 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c98102f50 0x7f5c98103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:38.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.555+0000 7f5c9fa00700 1 -- 172.21.15.1:0/2476017208 >> 172.21.15.1:0/2476017208 conn(0x7f5c980f8ae0 msgr2=0x7f5c980faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:38.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.555+0000 7f5c9fa00700 1 -- 172.21.15.1:0/2476017208 shutdown_connections 2024-02-28T16:59:38.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.555+0000 7f5c9fa00700 1 -- 172.21.15.1:0/2476017208 wait complete. 2024-02-28T16:59:38.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.556+0000 7f5c9fa00700 1 Processor -- start 2024-02-28T16:59:38.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.556+0000 7f5c9fa00700 1 -- start start 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.556+0000 7f5c9fa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98102f50 0x7f5c98115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.556+0000 7f5c9fa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c981038f0 0x7f5c98116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c9fa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c981168f0 con 0x7f5c98102f50 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c9fa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9810fb70 con 0x7f5c981038f0 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c9d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98102f50 0x7f5c98115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c9d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98102f50 0x7f5c98115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41634/0 (socket says 172.21.15.1:41634) 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c9d000700 1 -- 172.21.15.1:0/4031270848 learned_addr learned my addr 172.21.15.1:0/4031270848 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c97e00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c981038f0 0x7f5c98116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c9d000700 1 -- 172.21.15.1:0/4031270848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c981038f0 msgr2=0x7f5c98116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c9d000700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c981038f0 0x7f5c98116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.557+0000 7f5c9d000700 1 -- 172.21.15.1:0/4031270848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c84005040 con 0x7f5c98102f50 2024-02-28T16:59:38.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.558+0000 7f5c9d000700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98102f50 0x7f5c98115cd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00bbb0 tx=0x7f5c8c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:38.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.558+0000 7f5c94c00700 1 -- 172.21.15.1:0/4031270848 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8c0092e0 con 0x7f5c98102f50 2024-02-28T16:59:38.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.558+0000 7f5c94c00700 1 -- 172.21.15.1:0/4031270848 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c8c00cbd0 con 0x7f5c98102f50 2024-02-28T16:59:38.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.558+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9810fdf0 con 0x7f5c98102f50 2024-02-28T16:59:38.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.558+0000 7f5c94c00700 1 -- 172.21.15.1:0/4031270848 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8c007400 con 0x7f5c98102f50 2024-02-28T16:59:38.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.558+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c98110330 con 0x7f5c98102f50 2024-02-28T16:59:38.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.559+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c980fc6e0 con 0x7f5c98102f50 2024-02-28T16:59:38.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.561+0000 7f5c94c00700 1 -- 172.21.15.1:0/4031270848 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5c8c012070 con 0x7f5c98102f50 2024-02-28T16:59:38.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.562+0000 7f5c94c00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c8006d7f0 0x7f5c8006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:38.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.562+0000 7f5c94c00700 1 -- 172.21.15.1:0/4031270848 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5c8c08cc00 con 0x7f5c98102f50 2024-02-28T16:59:38.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.562+0000 7f5c97e00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c8006d7f0 0x7f5c8006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:38.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.563+0000 7f5c97e00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c8006d7f0 0x7f5c8006fca0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400b3b0 tx=0x7f5c84009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:38.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.567+0000 7f5c94c00700 1 -- 172.21.15.1:0/4031270848 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5c8c059ee0 con 0x7f5c98102f50 2024-02-28T16:59:38.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.810+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c98110550 con 0x7f5c8006d7f0 2024-02-28T16:59:38.809 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:38 smithi001 ceph-mon[112940]: pgmap v756: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:38.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.812+0000 7f5c94c00700 1 -- 172.21.15.1:0/4031270848 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5c98110550 con 0x7f5c8006d7f0 2024-02-28T16:59:38.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.814+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c8006d7f0 msgr2=0x7f5c8006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:38.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.814+0000 7f5c9fa00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c8006d7f0 0x7f5c8006fca0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400b3b0 tx=0x7f5c84009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:38.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.814+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98102f50 msgr2=0x7f5c98115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:38.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.814+0000 7f5c9fa00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98102f50 0x7f5c98115cd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00bbb0 tx=0x7f5c8c009840 comp rx=0 tx=0).stop 2024-02-28T16:59:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.815+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 shutdown_connections 2024-02-28T16:59:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.815+0000 7f5c9fa00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c8006d7f0 0x7f5c8006fca0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.815+0000 7f5c9fa00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c98102f50 0x7f5c98115cd0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.815+0000 7f5c9fa00700 1 --2- 172.21.15.1:0/4031270848 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c981038f0 0x7f5c98116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.815+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 >> 172.21.15.1:0/4031270848 conn(0x7f5c980f8ae0 msgr2=0x7f5c980fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.815+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 shutdown_connections 2024-02-28T16:59:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.815+0000 7f5c9fa00700 1 -- 172.21.15.1:0/4031270848 wait complete. 2024-02-28T16:59:38.828 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T16:59:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.961+0000 7fc2a0800700 1 -- 172.21.15.1:0/3101326864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fe8f0 msgr2=0x7fc298107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.961+0000 7fc2a0800700 1 --2- 172.21.15.1:0/3101326864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fe8f0 0x7fc298107050 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc288004970 tx=0x7fc288009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.961+0000 7fc2a0800700 1 -- 172.21.15.1:0/3101326864 shutdown_connections 2024-02-28T16:59:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.961+0000 7fc2a0800700 1 --2- 172.21.15.1:0/3101326864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fe8f0 0x7fc298107050 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.961+0000 7fc2a0800700 1 --2- 172.21.15.1:0/3101326864 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2980fdfe0 0x7fc2980fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.961+0000 7fc2a0800700 1 -- 172.21.15.1:0/3101326864 >> 172.21.15.1:0/3101326864 conn(0x7fc298071d90 msgr2=0x7fc298072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.961+0000 7fc2a0800700 1 -- 172.21.15.1:0/3101326864 shutdown_connections 2024-02-28T16:59:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.961+0000 7fc2a0800700 1 -- 172.21.15.1:0/3101326864 wait complete. 2024-02-28T16:59:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.962+0000 7fc2a0800700 1 Processor -- start 2024-02-28T16:59:39.113 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:38 smithi066 ceph-mon[85781]: pgmap v756: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.962+0000 7fc2a0800700 1 -- start start 2024-02-28T16:59:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc2a0800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fdfe0 0x7fc298115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc2a0800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2980fe8f0 0x7fc298116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc2a0800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc298116790 con 0x7fc2980fdfe0 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc2a0800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29810fac0 con 0x7fc2980fe8f0 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc29de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fdfe0 0x7fc298115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc29de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fdfe0 0x7fc298115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41650/0 (socket says 172.21.15.1:41650) 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc29de00700 1 -- 172.21.15.1:0/33489454 learned_addr learned my addr 172.21.15.1:0/33489454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc29de00700 1 -- 172.21.15.1:0/33489454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2980fe8f0 msgr2=0x7fc298116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc29de00700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2980fe8f0 0x7fc298116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.963+0000 7fc29de00700 1 -- 172.21.15.1:0/33489454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc288005040 con 0x7fc2980fdfe0 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.964+0000 7fc29de00700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fdfe0 0x7fc298115c00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00c830 tx=0x7fc28c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:39.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.964+0000 7fc292000700 1 -- 172.21.15.1:0/33489454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc28c00a760 con 0x7fc2980fdfe0 2024-02-28T16:59:39.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.964+0000 7fc292000700 1 -- 172.21.15.1:0/33489454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc28c00ad90 con 0x7fc2980fdfe0 2024-02-28T16:59:39.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.964+0000 7fc292000700 1 -- 172.21.15.1:0/33489454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc28c0113a0 con 0x7fc2980fdfe0 2024-02-28T16:59:39.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.964+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29810fda0 con 0x7fc2980fdfe0 2024-02-28T16:59:39.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.964+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2981102e0 con 0x7fc2980fdfe0 2024-02-28T16:59:39.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.965+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc298106b20 con 0x7fc2980fdfe0 2024-02-28T16:59:39.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.966+0000 7fc292000700 1 -- 172.21.15.1:0/33489454 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc28c012400 con 0x7fc2980fdfe0 2024-02-28T16:59:39.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.967+0000 7fc292000700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc28006d7f0 0x7fc28006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.967+0000 7fc292000700 1 -- 172.21.15.1:0/33489454 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc28c08cfa0 con 0x7fc2980fdfe0 2024-02-28T16:59:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.967+0000 7fc29d400700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc28006d7f0 0x7fc28006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.968+0000 7fc29d400700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc28006d7f0 0x7fc28006fca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc28800b370 tx=0x7fc288009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:39.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:38.972+0000 7fc292000700 1 -- 172.21.15.1:0/33489454 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc28c05a300 con 0x7fc2980fdfe0 2024-02-28T16:59:39.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.204+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc29804bed0 con 0x7fc28006d7f0 2024-02-28T16:59:39.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.206+0000 7fc292000700 1 -- 172.21.15.1:0/33489454 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc29804bed0 con 0x7fc28006d7f0 2024-02-28T16:59:39.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc28006d7f0 msgr2=0x7fc28006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc28006d7f0 0x7fc28006fca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc28800b370 tx=0x7fc288009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:39.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fdfe0 msgr2=0x7fc298115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fdfe0 0x7fc298115c00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00c830 tx=0x7fc28c004470 comp rx=0 tx=0).stop 2024-02-28T16:59:39.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 shutdown_connections 2024-02-28T16:59:39.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc28006d7f0 0x7fc28006fca0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2980fdfe0 0x7fc298115c00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 --2- 172.21.15.1:0/33489454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2980fe8f0 0x7fc298116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 >> 172.21.15.1:0/33489454 conn(0x7fc298071d90 msgr2=0x7fc2980fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:39.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 shutdown_connections 2024-02-28T16:59:39.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.209+0000 7fc2a0800700 1 -- 172.21.15.1:0/33489454 wait complete. 2024-02-28T16:59:39.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.364+0000 7f3e08200700 1 -- 172.21.15.1:0/1971172835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 msgr2=0x7f3e00105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.364+0000 7f3e08200700 1 --2- 172.21.15.1:0/1971172835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 0x7f3e00105540 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3df4004940 tx=0x7f3df4009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:39.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.364+0000 7f3e08200700 1 -- 172.21.15.1:0/1971172835 shutdown_connections 2024-02-28T16:59:39.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.364+0000 7f3e08200700 1 --2- 172.21.15.1:0/1971172835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e00105b10 0x7f3e000ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.364+0000 7f3e08200700 1 --2- 172.21.15.1:0/1971172835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 0x7f3e00105540 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.364+0000 7f3e08200700 1 -- 172.21.15.1:0/1971172835 >> 172.21.15.1:0/1971172835 conn(0x7f3e000face0 msgr2=0x7f3e000fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:39.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.365+0000 7f3e08200700 1 -- 172.21.15.1:0/1971172835 shutdown_connections 2024-02-28T16:59:39.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.365+0000 7f3e08200700 1 -- 172.21.15.1:0/1971172835 wait complete. 2024-02-28T16:59:39.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.365+0000 7f3e08200700 1 Processor -- start 2024-02-28T16:59:39.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.366+0000 7f3e08200700 1 -- start start 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.366+0000 7f3e08200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 0x7f3e00115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.366+0000 7f3e08200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e00105b10 0x7f3e001161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.366+0000 7f3e08200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e001167f0 con 0x7f3e00105170 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.366+0000 7f3e08200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0010fb40 con 0x7f3e00105b10 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.366+0000 7f3e05800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 0x7f3e00115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.366+0000 7f3e05800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 0x7f3e00115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41662/0 (socket says 172.21.15.1:41662) 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.366+0000 7f3e05800700 1 -- 172.21.15.1:0/3871987793 learned_addr learned my addr 172.21.15.1:0/3871987793 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.367+0000 7f3e05800700 1 -- 172.21.15.1:0/3871987793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e00105b10 msgr2=0x7f3e001161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.367+0000 7f3e05800700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e00105b10 0x7f3e001161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.367+0000 7f3e05800700 1 -- 172.21.15.1:0/3871987793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df4005040 con 0x7f3e00105170 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.367+0000 7f3e05800700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 0x7f3e00115c60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009fd0 tx=0x7f3df4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.367+0000 7f3dfd600700 1 -- 172.21.15.1:0/3871987793 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df400e040 con 0x7f3e00105170 2024-02-28T16:59:39.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.367+0000 7f3dfd600700 1 -- 172.21.15.1:0/3871987793 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3df400bbf0 con 0x7f3e00105170 2024-02-28T16:59:39.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.367+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0010fdc0 con 0x7f3e00105170 2024-02-28T16:59:39.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.367+0000 7f3dfd600700 1 -- 172.21.15.1:0/3871987793 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df401c3f0 con 0x7f3e00105170 2024-02-28T16:59:39.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.368+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e00110300 con 0x7f3e00105170 2024-02-28T16:59:39.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.369+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e00102480 con 0x7f3e00105170 2024-02-28T16:59:39.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.369+0000 7f3dfd600700 1 -- 172.21.15.1:0/3871987793 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3df400bd60 con 0x7f3e00105170 2024-02-28T16:59:39.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.370+0000 7f3dfd600700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de806d7f0 0x7f3de806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.371+0000 7f3dfd600700 1 -- 172.21.15.1:0/3871987793 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3df408e2b0 con 0x7f3e00105170 2024-02-28T16:59:39.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.371+0000 7f3e04e00700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de806d7f0 0x7f3de806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:39.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.371+0000 7f3e04e00700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de806d7f0 0x7f3de806fca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3e00111540 tx=0x7f3dec009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:39.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.375+0000 7f3dfd600700 1 -- 172.21.15.1:0/3871987793 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3df4057ae0 con 0x7f3e00105170 2024-02-28T16:59:39.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.611+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e00111280 con 0x7f3de806d7f0 2024-02-28T16:59:39.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.628+0000 7f3dfd600700 1 -- 172.21.15.1:0/3871987793 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3e00111280 con 0x7f3de806d7f0 2024-02-28T16:59:39.627 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T16:59:39.627 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (25m) 2m ago 40m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T16:59:39.628 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (22m) 2m ago 40m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T16:59:39.628 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (22m) 2m ago 38m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T16:59:39.628 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (25m) 2m ago 39m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T16:59:39.628 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (18m) 2m ago 32m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T16:59:39.628 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (17m) 2m ago 32m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T16:59:39.629 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (32m) 2m ago 32m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T16:59:39.629 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (2m) 2m ago 32m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T16:59:39.629 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (28m) 2m ago 41m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T16:59:39.629 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (27m) 2m ago 38m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T16:59:39.629 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (24m) 2m ago 41m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T16:59:39.630 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (23m) 2m ago 38m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T16:59:39.630 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (26m) 2m ago 40m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T16:59:39.630 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (26m) 2m ago 38m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T16:59:39.630 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (21m) 2m ago 37m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T16:59:39.630 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (21m) 2m ago 37m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T16:59:39.630 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (20m) 2m ago 36m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T16:59:39.630 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (20m) 2m ago 35m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T16:59:39.630 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (19m) 2m ago 35m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T16:59:39.631 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (18m) 2m ago 34m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T16:59:39.631 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (26m) 2m ago 39m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T16:59:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.630+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de806d7f0 msgr2=0x7f3de806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.630+0000 7f3e08200700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de806d7f0 0x7f3de806fca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3e00111540 tx=0x7f3dec009ce0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 msgr2=0x7f3e00115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 0x7f3e00115c60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009fd0 tx=0x7f3df4009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 shutdown_connections 2024-02-28T16:59:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de806d7f0 0x7f3de806fca0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00105170 0x7f3e00115c60 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 --2- 172.21.15.1:0/3871987793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e00105b10 0x7f3e001161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 >> 172.21.15.1:0/3871987793 conn(0x7f3e000face0 msgr2=0x7f3e000fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 shutdown_connections 2024-02-28T16:59:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.631+0000 7f3e08200700 1 -- 172.21.15.1:0/3871987793 wait complete. 2024-02-28T16:59:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.788+0000 7f1b73c00700 1 -- 172.21.15.1:0/248394005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 msgr2=0x7f1b6c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.788+0000 7f1b73c00700 1 --2- 172.21.15.1:0/248394005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 0x7f1b6c1092c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1b58004970 tx=0x7f1b58009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.788+0000 7f1b73c00700 1 -- 172.21.15.1:0/248394005 shutdown_connections 2024-02-28T16:59:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.788+0000 7f1b73c00700 1 --2- 172.21.15.1:0/248394005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 0x7f1b6c1092c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.788+0000 7f1b73c00700 1 --2- 172.21.15.1:0/248394005 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b6c0702d0 0x7f1b6c0706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.788+0000 7f1b73c00700 1 -- 172.21.15.1:0/248394005 >> 172.21.15.1:0/248394005 conn(0x7f1b6c0f8ae0 msgr2=0x7f1b6c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.788+0000 7f1b73c00700 1 -- 172.21.15.1:0/248394005 shutdown_connections 2024-02-28T16:59:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.788+0000 7f1b73c00700 1 -- 172.21.15.1:0/248394005 wait complete. 2024-02-28T16:59:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.789+0000 7f1b73c00700 1 Processor -- start 2024-02-28T16:59:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.789+0000 7f1b73c00700 1 -- start start 2024-02-28T16:59:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.790+0000 7f1b73c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b6c0702d0 0x7f1b6c115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.790+0000 7f1b73c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 0x7f1b6c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.790+0000 7f1b73c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c1167a0 con 0x7f1b6c070be0 2024-02-28T16:59:39.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.790+0000 7f1b73c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c10faf0 con 0x7f1b6c0702d0 2024-02-28T16:59:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.790+0000 7f1b6be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 0x7f1b6c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.790+0000 7f1b6be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 0x7f1b6c116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41686/0 (socket says 172.21.15.1:41686) 2024-02-28T16:59:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.790+0000 7f1b6be00700 1 -- 172.21.15.1:0/2961147587 learned_addr learned my addr 172.21.15.1:0/2961147587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.790+0000 7f1b71200700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b6c0702d0 0x7f1b6c115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b6be00700 1 -- 172.21.15.1:0/2961147587 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b6c0702d0 msgr2=0x7f1b6c115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b6be00700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b6c0702d0 0x7f1b6c115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b6be00700 1 -- 172.21.15.1:0/2961147587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b58005040 con 0x7f1b6c070be0 2024-02-28T16:59:39.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b6be00700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 0x7f1b6c116150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1b58000c00 tx=0x7f1b58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b68c00700 1 -- 172.21.15.1:0/2961147587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b5800e040 con 0x7f1b6c070be0 2024-02-28T16:59:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b68c00700 1 -- 172.21.15.1:0/2961147587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b58004290 con 0x7f1b6c070be0 2024-02-28T16:59:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c10fd70 con 0x7f1b6c070be0 2024-02-28T16:59:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b68c00700 1 -- 172.21.15.1:0/2961147587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b5801c630 con 0x7f1b6c070be0 2024-02-28T16:59:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.791+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c1102b0 con 0x7f1b6c070be0 2024-02-28T16:59:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.793+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c106980 con 0x7f1b6c070be0 2024-02-28T16:59:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.794+0000 7f1b68c00700 1 -- 172.21.15.1:0/2961147587 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1b58013980 con 0x7f1b6c070be0 2024-02-28T16:59:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.795+0000 7f1b68c00700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b5406d7e0 0x7f1b5406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.795+0000 7f1b68c00700 1 -- 172.21.15.1:0/2961147587 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1b5808e2c0 con 0x7f1b6c070be0 2024-02-28T16:59:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.795+0000 7f1b71200700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b5406d7e0 0x7f1b5406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.796+0000 7f1b71200700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b5406d7e0 0x7f1b5406fc90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1b60007fd0 tx=0x7f1b60009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:39.799+0000 7f1b68c00700 1 -- 172.21.15.1:0/2961147587 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b58057af0 con 0x7f1b6c070be0 2024-02-28T16:59:40.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.113+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1b6c1111f0 con 0x7f1b6c070be0 2024-02-28T16:59:40.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.114+0000 7f1b68c00700 1 -- 172.21.15.1:0/2961147587 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f1b5805b1b0 con 0x7f1b6c070be0 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:59:40.113 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T16:59:40.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T16:59:40.114 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T16:59:40.114 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T16:59:40.114 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T16:59:40.114 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T16:59:40.114 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.117+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b5406d7e0 msgr2=0x7f1b5406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.117+0000 7f1b73c00700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b5406d7e0 0x7f1b5406fc90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1b60007fd0 tx=0x7f1b60009040 comp rx=0 tx=0).stop 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.117+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 msgr2=0x7f1b6c116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.117+0000 7f1b73c00700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 0x7f1b6c116150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1b58000c00 tx=0x7f1b58009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.118+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 shutdown_connections 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.118+0000 7f1b73c00700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b5406d7e0 0x7f1b5406fc90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.118+0000 7f1b73c00700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b6c0702d0 0x7f1b6c115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.118+0000 7f1b73c00700 1 --2- 172.21.15.1:0/2961147587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b6c070be0 0x7f1b6c116150 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.118+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 >> 172.21.15.1:0/2961147587 conn(0x7f1b6c0f8ae0 msgr2=0x7f1b6c103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:40.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.118+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 shutdown_connections 2024-02-28T16:59:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.118+0000 7f1b73c00700 1 -- 172.21.15.1:0/2961147587 wait complete. 2024-02-28T16:59:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.276+0000 7fc79f200700 1 -- 172.21.15.1:0/1577980183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a01036e0 msgr2=0x7fc7a00fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.276+0000 7fc79f200700 1 --2- 172.21.15.1:0/1577980183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a01036e0 0x7fc7a00fcd50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc78c004940 tx=0x7fc78c009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.276+0000 7fc79f200700 1 -- 172.21.15.1:0/1577980183 shutdown_connections 2024-02-28T16:59:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.276+0000 7fc79f200700 1 --2- 172.21.15.1:0/1577980183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a01036e0 0x7fc7a00fcd50 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.276+0000 7fc79f200700 1 --2- 172.21.15.1:0/1577980183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a0102d40 0x7fc7a0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.276+0000 7fc79f200700 1 -- 172.21.15.1:0/1577980183 >> 172.21.15.1:0/1577980183 conn(0x7fc7a00f88b0 msgr2=0x7fc7a00facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.276+0000 7fc79f200700 1 -- 172.21.15.1:0/1577980183 shutdown_connections 2024-02-28T16:59:40.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.276+0000 7fc79f200700 1 -- 172.21.15.1:0/1577980183 wait complete. 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.277+0000 7fc79f200700 1 Processor -- start 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.277+0000 7fc79f200700 1 -- start start 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a0102d40 0x7fc7a0115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a01036e0 0x7fc7a0115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a01165e0 con 0x7fc7a0102d40 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a010f910 con 0x7fc7a01036e0 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a0102d40 0x7fc7a0115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a0102d40 0x7fc7a0115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41704/0 (socket says 172.21.15.1:41704) 2024-02-28T16:59:40.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79de00700 1 -- 172.21.15.1:0/2848157164 learned_addr learned my addr 172.21.15.1:0/2848157164 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:40.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79d400700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a01036e0 0x7fc7a0115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:40.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79de00700 1 -- 172.21.15.1:0/2848157164 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a01036e0 msgr2=0x7fc7a0115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79de00700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a01036e0 0x7fc7a0115f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.278+0000 7fc79de00700 1 -- 172.21.15.1:0/2848157164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c005040 con 0x7fc7a0102d40 2024-02-28T16:59:40.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.279+0000 7fc79de00700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a0102d40 0x7fc7a0115a50 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc79000c830 tx=0x7fc790004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.279+0000 7fc796000700 1 -- 172.21.15.1:0/2848157164 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc79000a760 con 0x7fc7a0102d40 2024-02-28T16:59:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.279+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a010fbf0 con 0x7fc7a0102d40 2024-02-28T16:59:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.279+0000 7fc796000700 1 -- 172.21.15.1:0/2848157164 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc79000ad90 con 0x7fc7a0102d40 2024-02-28T16:59:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.279+0000 7fc796000700 1 -- 172.21.15.1:0/2848157164 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7900113a0 con 0x7fc7a0102d40 2024-02-28T16:59:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.279+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a01101b0 con 0x7fc7a0102d40 2024-02-28T16:59:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.280+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a005f5c0 con 0x7fc7a0102d40 2024-02-28T16:59:40.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.282+0000 7fc796000700 1 -- 172.21.15.1:0/2848157164 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc790012400 con 0x7fc7a0102d40 2024-02-28T16:59:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.282+0000 7fc796000700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78406d7f0 0x7fc78406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.283+0000 7fc796000700 1 -- 172.21.15.1:0/2848157164 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc79008da30 con 0x7fc7a0102d40 2024-02-28T16:59:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.283+0000 7fc79d400700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78406d7f0 0x7fc78406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:40.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.284+0000 7fc79d400700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78406d7f0 0x7fc78406fca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b3b0 tx=0x7fc78c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.286+0000 7fc796000700 1 -- 172.21.15.1:0/2848157164 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc79005ae10 con 0x7fc7a0102d40 2024-02-28T16:59:40.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.560+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc7a01110c0 con 0x7fc7a0102d40 2024-02-28T16:59:40.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.561+0000 7fc796000700 1 -- 172.21.15.1:0/2848157164 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fc79005a9a0 con 0x7fc7a0102d40 2024-02-28T16:59:40.559 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T16:59:40.560 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T16:59:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:59:40.562 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T16:59:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.563+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78406d7f0 msgr2=0x7fc78406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.563+0000 7fc79f200700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78406d7f0 0x7fc78406fca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b3b0 tx=0x7fc78c009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.563+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a0102d40 msgr2=0x7fc7a0115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.563+0000 7fc79f200700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a0102d40 0x7fc7a0115a50 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc79000c830 tx=0x7fc790004470 comp rx=0 tx=0).stop 2024-02-28T16:59:40.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.564+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 shutdown_connections 2024-02-28T16:59:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.564+0000 7fc79f200700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78406d7f0 0x7fc78406fca0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.564+0000 7fc79f200700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a0102d40 0x7fc7a0115a50 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.564+0000 7fc79f200700 1 --2- 172.21.15.1:0/2848157164 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a01036e0 0x7fc7a0115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.564+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 >> 172.21.15.1:0/2848157164 conn(0x7fc7a00f88b0 msgr2=0x7fc7a00fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.564+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 shutdown_connections 2024-02-28T16:59:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.564+0000 7fc79f200700 1 -- 172.21.15.1:0/2848157164 wait complete. 2024-02-28T16:59:40.564 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T16:59:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.718+0000 7fac07e00700 1 -- 172.21.15.1:0/1760456368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105b10 msgr2=0x7fac000ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.718+0000 7fac07e00700 1 --2- 172.21.15.1:0/1760456368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105b10 0x7fac000ff180 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fabec004940 tx=0x7fabec009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.719+0000 7fac07e00700 1 -- 172.21.15.1:0/1760456368 shutdown_connections 2024-02-28T16:59:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.719+0000 7fac07e00700 1 --2- 172.21.15.1:0/1760456368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105b10 0x7fac000ff180 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.719+0000 7fac07e00700 1 --2- 172.21.15.1:0/1760456368 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac00105170 0x7fac00105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.719+0000 7fac07e00700 1 -- 172.21.15.1:0/1760456368 >> 172.21.15.1:0/1760456368 conn(0x7fac000face0 msgr2=0x7fac000fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.719+0000 7fac07e00700 1 -- 172.21.15.1:0/1760456368 shutdown_connections 2024-02-28T16:59:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.719+0000 7fac07e00700 1 -- 172.21.15.1:0/1760456368 wait complete. 2024-02-28T16:59:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.720+0000 7fac07e00700 1 Processor -- start 2024-02-28T16:59:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.720+0000 7fac07e00700 1 -- start start 2024-02-28T16:59:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac07e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105170 0x7fac00115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac07e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac00105b10 0x7fac00116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac07e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac00116870 con 0x7fac00105170 2024-02-28T16:59:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac07e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0010fb80 con 0x7fac00105b10 2024-02-28T16:59:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac05400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105170 0x7fac00115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac05400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105170 0x7fac00115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41722/0 (socket says 172.21.15.1:41722) 2024-02-28T16:59:40.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac05400700 1 -- 172.21.15.1:0/3084535501 learned_addr learned my addr 172.21.15.1:0/3084535501 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac04a00700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac00105b10 0x7fac00116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:40.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac05400700 1 -- 172.21.15.1:0/3084535501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac00105b10 msgr2=0x7fac00116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.721+0000 7fac05400700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac00105b10 0x7fac00116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.722+0000 7fac05400700 1 -- 172.21.15.1:0/3084535501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec005040 con 0x7fac00105170 2024-02-28T16:59:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.722+0000 7fac05400700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105170 0x7fac00115ce0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fabf4002a50 tx=0x7fabf40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.722+0000 7fabfd600700 1 -- 172.21.15.1:0/3084535501 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabf4007cc0 con 0x7fac00105170 2024-02-28T16:59:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.722+0000 7fabfd600700 1 -- 172.21.15.1:0/3084535501 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabf400f040 con 0x7fac00105170 2024-02-28T16:59:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.722+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0010fe60 con 0x7fac00105170 2024-02-28T16:59:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.722+0000 7fabfd600700 1 -- 172.21.15.1:0/3084535501 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabf4009640 con 0x7fac00105170 2024-02-28T16:59:40.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.722+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac001103a0 con 0x7fac00105170 2024-02-28T16:59:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.724+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac00107640 con 0x7fac00105170 2024-02-28T16:59:40.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.725+0000 7fabfd600700 1 -- 172.21.15.1:0/3084535501 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fabf40097a0 con 0x7fac00105170 2024-02-28T16:59:40.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.726+0000 7fabfd600700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabe806d7f0 0x7fabe806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.726+0000 7fabfd600700 1 -- 172.21.15.1:0/3084535501 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fabf408cc60 con 0x7fac00105170 2024-02-28T16:59:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.726+0000 7fac04a00700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabe806d7f0 0x7fabe806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.727+0000 7fac04a00700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabe806d7f0 0x7fabe806fca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fac00111580 tx=0x7fabec009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:40.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.731+0000 7fabfd600700 1 -- 172.21.15.1:0/3084535501 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fabf4059fc0 con 0x7fac00105170 2024-02-28T16:59:40.811 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:40 smithi066 ceph-mon[85781]: from='client.35374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:40.811 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:40 smithi066 ceph-mon[85781]: pgmap v757: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:40.812 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:40 smithi066 ceph-mon[85781]: from='client.35378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:40.812 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:40 smithi066 ceph-mon[85781]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:40.812 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2961147587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:59:40.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.968+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac0004bf40 con 0x7fabe806d7f0 2024-02-28T16:59:40.967 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:40 smithi001 ceph-mon[112940]: from='client.35374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:40.967 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:40 smithi001 ceph-mon[112940]: pgmap v757: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:40.967 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:40 smithi001 ceph-mon[112940]: from='client.35378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:40.968 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:40 smithi001 ceph-mon[112940]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:40.968 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2961147587' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T16:59:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.970+0000 7fabfd600700 1 -- 172.21.15.1:0/3084535501 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fac0004bf40 con 0x7fabe806d7f0 2024-02-28T16:59:40.968 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T16:59:40.968 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T16:59:40.968 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T16:59:40.969 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T16:59:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabe806d7f0 msgr2=0x7fabe806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabe806d7f0 0x7fabe806fca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fac00111580 tx=0x7fabec009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105170 msgr2=0x7fac00115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105170 0x7fac00115ce0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fabf4002a50 tx=0x7fabf40046b0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 shutdown_connections 2024-02-28T16:59:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabe806d7f0 0x7fabe806fca0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac00105170 0x7fac00115ce0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 --2- 172.21.15.1:0/3084535501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fac00105b10 0x7fac00116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 >> 172.21.15.1:0/3084535501 conn(0x7fac000face0 msgr2=0x7fac000fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.973+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 shutdown_connections 2024-02-28T16:59:40.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:40.974+0000 7fac07e00700 1 -- 172.21.15.1:0/3084535501 wait complete. 2024-02-28T16:59:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.137+0000 7f5079600700 1 -- 172.21.15.1:0/2476458559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 msgr2=0x7f5074105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.137+0000 7f5079600700 1 --2- 172.21.15.1:0/2476458559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 0x7f5074105540 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5064004940 tx=0x7f5064009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.137+0000 7f5079600700 1 -- 172.21.15.1:0/2476458559 shutdown_connections 2024-02-28T16:59:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.137+0000 7f5079600700 1 --2- 172.21.15.1:0/2476458559 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5074105b10 0x7f50740ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.137+0000 7f5079600700 1 --2- 172.21.15.1:0/2476458559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 0x7f5074105540 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.137+0000 7f5079600700 1 -- 172.21.15.1:0/2476458559 >> 172.21.15.1:0/2476458559 conn(0x7f50740face0 msgr2=0x7f50740fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.137+0000 7f5079600700 1 -- 172.21.15.1:0/2476458559 shutdown_connections 2024-02-28T16:59:41.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.137+0000 7f5079600700 1 -- 172.21.15.1:0/2476458559 wait complete. 2024-02-28T16:59:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.138+0000 7f5079600700 1 Processor -- start 2024-02-28T16:59:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.138+0000 7f5079600700 1 -- start start 2024-02-28T16:59:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.139+0000 7f5079600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 0x7f5074115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.139+0000 7f5079600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5074105b10 0x7f5074116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.139+0000 7f5079600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50741168e0 con 0x7f5074105170 2024-02-28T16:59:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.139+0000 7f5079600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507410fbf0 con 0x7f5074105b10 2024-02-28T16:59:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.139+0000 7f5072a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 0x7f5074115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.139+0000 7f5072a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 0x7f5074115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41738/0 (socket says 172.21.15.1:41738) 2024-02-28T16:59:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.139+0000 7f5072a00700 1 -- 172.21.15.1:0/3859818804 learned_addr learned my addr 172.21.15.1:0/3859818804 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T16:59:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f5072a00700 1 -- 172.21.15.1:0/3859818804 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5074105b10 msgr2=0x7f5074116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f5072a00700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5074105b10 0x7f5074116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f5072a00700 1 -- 172.21.15.1:0/3859818804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5064005040 con 0x7f5074105170 2024-02-28T16:59:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f5072a00700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 0x7f5074115d50 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5064009fd0 tx=0x7f5064009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f506aa00700 1 -- 172.21.15.1:0/3859818804 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f506400e040 con 0x7f5074105170 2024-02-28T16:59:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507410fe70 con 0x7f5074105170 2024-02-28T16:59:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f506aa00700 1 -- 172.21.15.1:0/3859818804 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f506400bbf0 con 0x7f5074105170 2024-02-28T16:59:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f506aa00700 1 -- 172.21.15.1:0/3859818804 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f506401c3f0 con 0x7f5074105170 2024-02-28T16:59:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.140+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50741103b0 con 0x7f5074105170 2024-02-28T16:59:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.142+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5074102480 con 0x7f5074105170 2024-02-28T16:59:41.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.143+0000 7f506aa00700 1 -- 172.21.15.1:0/3859818804 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f506400bd60 con 0x7f5074105170 2024-02-28T16:59:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.144+0000 7f506aa00700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505806d7f0 0x7f505806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T16:59:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.144+0000 7f506aa00700 1 -- 172.21.15.1:0/3859818804 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f506408e2b0 con 0x7f5074105170 2024-02-28T16:59:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.144+0000 7f5072000700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505806d7f0 0x7f505806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T16:59:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.145+0000 7f5072000700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505806d7f0 0x7f505806fca0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f50741115f0 tx=0x7f5060009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T16:59:41.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.149+0000 7f506aa00700 1 -- 172.21.15.1:0/3859818804 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f506405b610 con 0x7f5074105170 2024-02-28T16:59:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.466+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f50740fe900 con 0x7f5074105170 2024-02-28T16:59:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.466+0000 7f506aa00700 1 -- 172.21.15.1:0/3859818804 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f506405b1a0 con 0x7f5074105170 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T16:59:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T16:59:41.466 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T16:59:41.467 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T16:59:41.467 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T16:59:41.467 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T16:59:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505806d7f0 msgr2=0x7f505806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505806d7f0 0x7f505806fca0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f50741115f0 tx=0x7f5060009ce0 comp rx=0 tx=0).stop 2024-02-28T16:59:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 msgr2=0x7f5074115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T16:59:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 0x7f5074115d50 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5064009fd0 tx=0x7f5064009e30 comp rx=0 tx=0).stop 2024-02-28T16:59:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 shutdown_connections 2024-02-28T16:59:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505806d7f0 0x7f505806fca0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5074105170 0x7f5074115d50 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 --2- 172.21.15.1:0/3859818804 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5074105b10 0x7f5074116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T16:59:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 >> 172.21.15.1:0/3859818804 conn(0x7f50740face0 msgr2=0x7f50740fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T16:59:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 shutdown_connections 2024-02-28T16:59:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T16:59:41.469+0000 7f5079600700 1 -- 172.21.15.1:0/3859818804 wait complete. 2024-02-28T16:59:41.780 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:41 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2848157164' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:59:41.780 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:59:41.780 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:41 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3859818804' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:59:42.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:41 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2848157164' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T16:59:42.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:59:42.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:41 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3859818804' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T16:59:43.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:42 smithi066 ceph-mon[85781]: pgmap v758: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:43.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:42 smithi066 ceph-mon[85781]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:43.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:42 smithi001 ceph-mon[112940]: pgmap v758: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:43.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:42 smithi001 ceph-mon[112940]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T16:59:45.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:44 smithi066 ceph-mon[85781]: pgmap v759: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:45.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:44 smithi001 ceph-mon[112940]: pgmap v759: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:47.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:46 smithi066 ceph-mon[85781]: pgmap v760: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:47.029 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:46 smithi001 ceph-mon[112940]: pgmap v760: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:49.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:48 smithi066 ceph-mon[85781]: pgmap v761: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:49.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:48 smithi001 ceph-mon[112940]: pgmap v761: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:51.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:50 smithi066 ceph-mon[85781]: pgmap v762: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:51.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:50 smithi001 ceph-mon[112940]: pgmap v762: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:53.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:52 smithi066 ceph-mon[85781]: pgmap v763: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:53.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:52 smithi001 ceph-mon[112940]: pgmap v763: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:55.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:54 smithi066 ceph-mon[85781]: pgmap v764: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:55.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:54 smithi001 ceph-mon[112940]: pgmap v764: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:57.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:56 smithi066 ceph-mon[85781]: pgmap v765: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:57.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:59:57.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:56 smithi001 ceph-mon[112940]: pgmap v765: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:57.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T16:59:59.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 16:59:58 smithi066 ceph-mon[85781]: pgmap v766: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T16:59:59.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 16:59:58 smithi001 ceph-mon[112940]: pgmap v766: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:00.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:00:00.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:00:00.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:00:00.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-02-28T17:00:00.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:00.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-02-28T17:00:00.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:00:00.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : return _run_code(code, main_globals, None, 2024-02-28T17:00:00.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:00:00.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-02-28T17:00:00.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:00:00.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:00:00.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T16:59:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-02-28T17:00:01.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: pgmap v767: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:01.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:00:01.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:00:01.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: /usr/bin/podman: timeout after 895 seconds 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: Traceback (most recent call last): 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: return _run_code(code, main_globals, None, 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: exec(code, run_globals) 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:00:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: fs cephfs is degraded 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:00:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:00 smithi066 ceph-mon[85781]: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: pgmap v767: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: /usr/bin/podman: timeout after 895 seconds 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: Traceback (most recent call last): 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: return _run_code(code, main_globals, None, 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: exec(code, run_globals) 2024-02-28T17:00:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:00:01.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: fs cephfs is degraded 2024-02-28T17:00:01.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:00:01.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:00:01.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:00:01.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:00 smithi001 ceph-mon[112940]: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:00:03.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:02 smithi066 ceph-mon[85781]: pgmap v768: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:03.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:02 smithi001 ceph-mon[112940]: pgmap v768: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:05.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:04 smithi066 ceph-mon[85781]: pgmap v769: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:05.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:04 smithi001 ceph-mon[112940]: pgmap v769: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:07.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:06 smithi066 ceph-mon[85781]: pgmap v770: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:07.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:06 smithi001 ceph-mon[112940]: pgmap v770: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:09.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:08 smithi066 ceph-mon[85781]: pgmap v771: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:09.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:08 smithi001 ceph-mon[112940]: pgmap v771: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:11.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:10 smithi066 ceph-mon[85781]: pgmap v772: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:11.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:10 smithi001 ceph-mon[112940]: pgmap v772: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.641+0000 7f7668600700 1 -- 172.21.15.1:0/750555464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 msgr2=0x7f76601099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.641+0000 7f7668600700 1 --2- 172.21.15.1:0/750555464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 0x7f76601099c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7654004940 tx=0x7f7654009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.641+0000 7f7668600700 1 -- 172.21.15.1:0/750555464 shutdown_connections 2024-02-28T17:00:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.641+0000 7f7668600700 1 --2- 172.21.15.1:0/750555464 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7660103dc0 0x7f7660104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.641+0000 7f7668600700 1 --2- 172.21.15.1:0/750555464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 0x7f76601099c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.641+0000 7f7668600700 1 -- 172.21.15.1:0/750555464 >> 172.21.15.1:0/750555464 conn(0x7f7660072d50 msgr2=0x7f7660075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.641+0000 7f7668600700 1 -- 172.21.15.1:0/750555464 shutdown_connections 2024-02-28T17:00:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.641+0000 7f7668600700 1 -- 172.21.15.1:0/750555464 wait complete. 2024-02-28T17:00:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.642+0000 7f7668600700 1 Processor -- start 2024-02-28T17:00:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.642+0000 7f7668600700 1 -- start start 2024-02-28T17:00:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.643+0000 7f7668600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7660103dc0 0x7f766011a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.643+0000 7f7668600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 0x7f766011a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.643+0000 7f7668600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766011ac70 con 0x7f76601095f0 2024-02-28T17:00:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.643+0000 7f7668600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7660113ef0 con 0x7f7660103dc0 2024-02-28T17:00:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.643+0000 7f7665200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 0x7f766011a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.643+0000 7f7665200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 0x7f766011a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46192/0 (socket says 172.21.15.1:46192) 2024-02-28T17:00:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.643+0000 7f7665200700 1 -- 172.21.15.1:0/1384932479 learned_addr learned my addr 172.21.15.1:0/1384932479 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.644+0000 7f7665c00700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7660103dc0 0x7f766011a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.644+0000 7f7665200700 1 -- 172.21.15.1:0/1384932479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7660103dc0 msgr2=0x7f766011a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.644+0000 7f7665200700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7660103dc0 0x7f766011a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.644+0000 7f7665200700 1 -- 172.21.15.1:0/1384932479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654005040 con 0x7f76601095f0 2024-02-28T17:00:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.644+0000 7f7665200700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 0x7f766011a590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f764c00b790 tx=0x7f764c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.646+0000 7f765d600700 1 -- 172.21.15.1:0/1384932479 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f764c00f770 con 0x7f76601095f0 2024-02-28T17:00:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.646+0000 7f765d600700 1 -- 172.21.15.1:0/1384932479 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f764c00fda0 con 0x7f76601095f0 2024-02-28T17:00:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.646+0000 7f765d600700 1 -- 172.21.15.1:0/1384932479 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f764c00d310 con 0x7f76601095f0 2024-02-28T17:00:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.646+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7660114170 con 0x7f76601095f0 2024-02-28T17:00:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.646+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76601146b0 con 0x7f76601095f0 2024-02-28T17:00:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.647+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7660075870 con 0x7f76601095f0 2024-02-28T17:00:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.648+0000 7f765d600700 1 -- 172.21.15.1:0/1384932479 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f764c010420 con 0x7f76601095f0 2024-02-28T17:00:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.650+0000 7f765d600700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764806d7f0 0x7f764806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.650+0000 7f765d600700 1 -- 172.21.15.1:0/1384932479 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f764c08cf10 con 0x7f76601095f0 2024-02-28T17:00:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.650+0000 7f7665c00700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764806d7f0 0x7f764806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.651+0000 7f7665c00700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764806d7f0 0x7f764806fca0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f765400b020 tx=0x7f7654009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:11.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.655+0000 7f765d600700 1 -- 172.21.15.1:0/1384932479 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f764c05a310 con 0x7f76601095f0 2024-02-28T17:00:11.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.905+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7660115710 con 0x7f764806d7f0 2024-02-28T17:00:11.905 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:00:11.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.907+0000 7f765d600700 1 -- 172.21.15.1:0/1384932479 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7660115710 con 0x7f764806d7f0 2024-02-28T17:00:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.910+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764806d7f0 msgr2=0x7f764806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.910+0000 7f7668600700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764806d7f0 0x7f764806fca0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f765400b020 tx=0x7f7654009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.910+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 msgr2=0x7f766011a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.910+0000 7f7668600700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 0x7f766011a590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f764c00b790 tx=0x7f764c009250 comp rx=0 tx=0).stop 2024-02-28T17:00:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.911+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 shutdown_connections 2024-02-28T17:00:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.911+0000 7f7668600700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764806d7f0 0x7f764806fca0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:11.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.911+0000 7f7668600700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7660103dc0 0x7f766011a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:11.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.911+0000 7f7668600700 1 --2- 172.21.15.1:0/1384932479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76601095f0 0x7f766011a590 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:11.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.911+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 >> 172.21.15.1:0/1384932479 conn(0x7f7660072d50 msgr2=0x7f76600746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:11.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.911+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 shutdown_connections 2024-02-28T17:00:11.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:11.911+0000 7f7668600700 1 -- 172.21.15.1:0/1384932479 wait complete. 2024-02-28T17:00:11.928 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:00:12.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:00:12.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.071+0000 7f8893200700 1 -- 172.21.15.1:0/3419910215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 msgr2=0x7f888c0ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.071+0000 7f8893200700 1 --2- 172.21.15.1:0/3419910215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 0x7f888c0ff180 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f887c004940 tx=0x7f887c009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:12.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.071+0000 7f8893200700 1 -- 172.21.15.1:0/3419910215 shutdown_connections 2024-02-28T17:00:12.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.071+0000 7f8893200700 1 --2- 172.21.15.1:0/3419910215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 0x7f888c0ff180 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.071+0000 7f8893200700 1 --2- 172.21.15.1:0/3419910215 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f888c105170 0x7f888c105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.071+0000 7f8893200700 1 -- 172.21.15.1:0/3419910215 >> 172.21.15.1:0/3419910215 conn(0x7f888c0face0 msgr2=0x7f888c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:12.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.071+0000 7f8893200700 1 -- 172.21.15.1:0/3419910215 shutdown_connections 2024-02-28T17:00:12.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.071+0000 7f8893200700 1 -- 172.21.15.1:0/3419910215 wait complete. 2024-02-28T17:00:12.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.072+0000 7f8893200700 1 Processor -- start 2024-02-28T17:00:12.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.072+0000 7f8893200700 1 -- start start 2024-02-28T17:00:12.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8893200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f888c105170 0x7f888c115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8893200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 0x7f888c116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8893200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c116870 con 0x7f888c105b10 2024-02-28T17:00:12.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8893200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c10fb80 con 0x7f888c105170 2024-02-28T17:00:12.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8883400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 0x7f888c116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:12.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8883400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 0x7f888c116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44178/0 (socket says 172.21.15.1:44178) 2024-02-28T17:00:12.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8883400700 1 -- 172.21.15.1:0/4120874856 learned_addr learned my addr 172.21.15.1:0/4120874856 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:12.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f888be00700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f888c105170 0x7f888c115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:12.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8883400700 1 -- 172.21.15.1:0/4120874856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f888c105170 msgr2=0x7f888c115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8883400700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f888c105170 0x7f888c115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.073+0000 7f8883400700 1 -- 172.21.15.1:0/4120874856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c005040 con 0x7f888c105b10 2024-02-28T17:00:12.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.074+0000 7f888be00700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f888c105170 0x7f888c115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:00:12.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.074+0000 7f8883400700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 0x7f888c116220 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f887c009030 tx=0x7f887c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:12.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.074+0000 7f8888c00700 1 -- 172.21.15.1:0/4120874856 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f887c00e040 con 0x7f888c105b10 2024-02-28T17:00:12.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.074+0000 7f8888c00700 1 -- 172.21.15.1:0/4120874856 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f887c00bbf0 con 0x7f888c105b10 2024-02-28T17:00:12.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.074+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c10fe00 con 0x7f888c105b10 2024-02-28T17:00:12.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.074+0000 7f8888c00700 1 -- 172.21.15.1:0/4120874856 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f887c01c630 con 0x7f888c105b10 2024-02-28T17:00:12.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.074+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c110340 con 0x7f888c105b10 2024-02-28T17:00:12.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.076+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c107640 con 0x7f888c105b10 2024-02-28T17:00:12.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.076+0000 7f8888c00700 1 -- 172.21.15.1:0/4120874856 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f887c00bd60 con 0x7f888c105b10 2024-02-28T17:00:12.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.077+0000 7f8888c00700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887006d7f0 0x7f887006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.077+0000 7f8888c00700 1 -- 172.21.15.1:0/4120874856 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f887c08e1f0 con 0x7f888c105b10 2024-02-28T17:00:12.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.078+0000 7f888be00700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887006d7f0 0x7f887006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:12.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.078+0000 7f888be00700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887006d7f0 0x7f887006fca0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8874007fd0 tx=0x7f8874009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.082+0000 7f8888c00700 1 -- 172.21.15.1:0/4120874856 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f887c05b5d0 con 0x7f888c105b10 2024-02-28T17:00:12.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.328+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f888c04bf40 con 0x7f887006d7f0 2024-02-28T17:00:12.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.330+0000 7f8888c00700 1 -- 172.21.15.1:0/4120874856 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f888c04bf40 con 0x7f887006d7f0 2024-02-28T17:00:12.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887006d7f0 msgr2=0x7f887006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887006d7f0 0x7f887006fca0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8874007fd0 tx=0x7f8874009040 comp rx=0 tx=0).stop 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 msgr2=0x7f888c116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 0x7f888c116220 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f887c009030 tx=0x7f887c009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 shutdown_connections 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f887006d7f0 0x7f887006fca0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f888c105170 0x7f888c115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 --2- 172.21.15.1:0/4120874856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f888c105b10 0x7f888c116220 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 >> 172.21.15.1:0/4120874856 conn(0x7f888c0face0 msgr2=0x7f888c0fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 shutdown_connections 2024-02-28T17:00:12.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.333+0000 7f8893200700 1 -- 172.21.15.1:0/4120874856 wait complete. 2024-02-28T17:00:12.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.476+0000 7fe124800700 1 -- 172.21.15.1:0/2128863511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070c40 msgr2=0x7fe11c109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.476+0000 7fe124800700 1 --2- 172.21.15.1:0/2128863511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070c40 0x7fe11c109260 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe10c004970 tx=0x7fe10c009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.476+0000 7fe124800700 1 -- 172.21.15.1:0/2128863511 shutdown_connections 2024-02-28T17:00:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.476+0000 7fe124800700 1 --2- 172.21.15.1:0/2128863511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070c40 0x7fe11c109260 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.476+0000 7fe124800700 1 --2- 172.21.15.1:0/2128863511 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c070330 0x7fe11c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.476+0000 7fe124800700 1 -- 172.21.15.1:0/2128863511 >> 172.21.15.1:0/2128863511 conn(0x7fe11c0f8ae0 msgr2=0x7fe11c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.477+0000 7fe124800700 1 -- 172.21.15.1:0/2128863511 shutdown_connections 2024-02-28T17:00:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.477+0000 7fe124800700 1 -- 172.21.15.1:0/2128863511 wait complete. 2024-02-28T17:00:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.477+0000 7fe124800700 1 Processor -- start 2024-02-28T17:00:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.477+0000 7fe124800700 1 -- start start 2024-02-28T17:00:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe124800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070330 0x7fe11c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe124800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c070c40 0x7fe11c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe124800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c116790 con 0x7fe11c070330 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe124800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c10fae0 con 0x7fe11c070c40 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe121e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070330 0x7fe11c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe121e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070330 0x7fe11c115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44204/0 (socket says 172.21.15.1:44204) 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe121e00700 1 -- 172.21.15.1:0/3417790149 learned_addr learned my addr 172.21.15.1:0/3417790149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe121e00700 1 -- 172.21.15.1:0/3417790149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c070c40 msgr2=0x7fe11c116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe121e00700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c070c40 0x7fe11c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.478+0000 7fe121e00700 1 -- 172.21.15.1:0/3417790149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c005040 con 0x7fe11c070330 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.479+0000 7fe121e00700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070330 0x7fe11c115c00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c8c0 tx=0x7fe1100044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.479+0000 7fe116000700 1 -- 172.21.15.1:0/3417790149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe11000a980 con 0x7fe11c070330 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.479+0000 7fe116000700 1 -- 172.21.15.1:0/3417790149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe11000ce60 con 0x7fe11c070330 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.479+0000 7fe116000700 1 -- 172.21.15.1:0/3417790149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1100115e0 con 0x7fe11c070330 2024-02-28T17:00:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.479+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c10fdc0 con 0x7fe11c070330 2024-02-28T17:00:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.479+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c110300 con 0x7fe11c070330 2024-02-28T17:00:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.480+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c106920 con 0x7fe11c070330 2024-02-28T17:00:12.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.481+0000 7fe116000700 1 -- 172.21.15.1:0/3417790149 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe110012400 con 0x7fe11c070330 2024-02-28T17:00:12.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.482+0000 7fe116000700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe10406d7f0 0x7fe10406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.482+0000 7fe116000700 1 -- 172.21.15.1:0/3417790149 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe11008d040 con 0x7fe11c070330 2024-02-28T17:00:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.482+0000 7fe121400700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe10406d7f0 0x7fe10406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:12.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.483+0000 7fe121400700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe10406d7f0 0x7fe10406fca0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00b370 tx=0x7fe10c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:12.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.487+0000 7fe116000700 1 -- 172.21.15.1:0/3417790149 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe110056870 con 0x7fe11c070330 2024-02-28T17:00:12.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.715+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe11c04bed0 con 0x7fe10406d7f0 2024-02-28T17:00:12.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.749+0000 7fe116000700 1 -- 172.21.15.1:0/3417790149 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe11c04bed0 con 0x7fe10406d7f0 2024-02-28T17:00:12.749 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:00:12.749 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (26m) 3m ago 40m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:00:12.749 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (23m) 3m ago 40m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:00:12.749 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (22m) 3m ago 39m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (25m) 3m ago 40m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (18m) 3m ago 33m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (18m) 3m ago 33m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (33m) 3m ago 33m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (3m) 3m ago 33m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (28m) 3m ago 42m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (27m) 3m ago 39m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (24m) 3m ago 42m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (24m) 3m ago 39m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (27m) 3m ago 40m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (27m) 3m ago 39m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (22m) 3m ago 38m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (21m) 3m ago 37m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (21m) 3m ago 37m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:00:12.750 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (20m) 3m ago 36m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:00:12.751 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (20m) 3m ago 35m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:00:12.751 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (19m) 3m ago 35m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:00:12.751 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (26m) 3m ago 39m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:00:12.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.752+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe10406d7f0 msgr2=0x7fe10406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.752+0000 7fe124800700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe10406d7f0 0x7fe10406fca0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00b370 tx=0x7fe10c009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:12.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.752+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070330 msgr2=0x7fe11c115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.752+0000 7fe124800700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070330 0x7fe11c115c00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c8c0 tx=0x7fe1100044c0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.753+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 shutdown_connections 2024-02-28T17:00:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.753+0000 7fe124800700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe10406d7f0 0x7fe10406fca0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.753+0000 7fe124800700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe11c070330 0x7fe11c115c00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.753+0000 7fe124800700 1 --2- 172.21.15.1:0/3417790149 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe11c070c40 0x7fe11c116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.753+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 >> 172.21.15.1:0/3417790149 conn(0x7fe11c0f8ae0 msgr2=0x7fe11c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.753+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 shutdown_connections 2024-02-28T17:00:12.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.753+0000 7fe124800700 1 -- 172.21.15.1:0/3417790149 wait complete. 2024-02-28T17:00:12.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.907+0000 7fc490e00700 1 -- 172.21.15.1:0/2595630648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c103dc0 msgr2=0x7fc48c104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.907+0000 7fc490e00700 1 --2- 172.21.15.1:0/2595630648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c103dc0 0x7fc48c104210 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc478004940 tx=0x7fc478009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.907+0000 7fc490e00700 1 -- 172.21.15.1:0/2595630648 shutdown_connections 2024-02-28T17:00:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.907+0000 7fc490e00700 1 --2- 172.21.15.1:0/2595630648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c103dc0 0x7fc48c104210 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.907+0000 7fc490e00700 1 --2- 172.21.15.1:0/2595630648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc48c1095f0 0x7fc48c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.907+0000 7fc490e00700 1 -- 172.21.15.1:0/2595630648 >> 172.21.15.1:0/2595630648 conn(0x7fc48c072d50 msgr2=0x7fc48c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.907+0000 7fc490e00700 1 -- 172.21.15.1:0/2595630648 shutdown_connections 2024-02-28T17:00:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.907+0000 7fc490e00700 1 -- 172.21.15.1:0/2595630648 wait complete. 2024-02-28T17:00:12.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.908+0000 7fc490e00700 1 Processor -- start 2024-02-28T17:00:12.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.908+0000 7fc490e00700 1 -- start start 2024-02-28T17:00:12.908 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:12 smithi001 ceph-mon[112940]: pgmap v773: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.908+0000 7fc490e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc48c103dc0 0x7fc48c11a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.908+0000 7fc490e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c1095f0 0x7fc48c11a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.908+0000 7fc490e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c11ac50 con 0x7fc48c1095f0 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.908+0000 7fc490e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c113ed0 con 0x7fc48c103dc0 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.909+0000 7fc489400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c1095f0 0x7fc48c11a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.909+0000 7fc489400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c1095f0 0x7fc48c11a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44222/0 (socket says 172.21.15.1:44222) 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.909+0000 7fc489400700 1 -- 172.21.15.1:0/1701033873 learned_addr learned my addr 172.21.15.1:0/1701033873 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.909+0000 7fc489e00700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc48c103dc0 0x7fc48c11a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.909+0000 7fc489400700 1 -- 172.21.15.1:0/1701033873 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc48c103dc0 msgr2=0x7fc48c11a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.909+0000 7fc489400700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc48c103dc0 0x7fc48c11a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.909+0000 7fc489400700 1 -- 172.21.15.1:0/1701033873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc478005040 con 0x7fc48c1095f0 2024-02-28T17:00:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.910+0000 7fc489400700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c1095f0 0x7fc48c11a570 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc478009150 tx=0x7fc478009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.910+0000 7fc482000700 1 -- 172.21.15.1:0/1701033873 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc47800e040 con 0x7fc48c1095f0 2024-02-28T17:00:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.910+0000 7fc482000700 1 -- 172.21.15.1:0/1701033873 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc47800bbf0 con 0x7fc48c1095f0 2024-02-28T17:00:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.910+0000 7fc482000700 1 -- 172.21.15.1:0/1701033873 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc47801c630 con 0x7fc48c1095f0 2024-02-28T17:00:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.910+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48c1140f0 con 0x7fc48c1095f0 2024-02-28T17:00:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.910+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48c114630 con 0x7fc48c1095f0 2024-02-28T17:00:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.912+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48c075870 con 0x7fc48c1095f0 2024-02-28T17:00:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.913+0000 7fc482000700 1 -- 172.21.15.1:0/1701033873 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc47800bd60 con 0x7fc48c1095f0 2024-02-28T17:00:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.914+0000 7fc482000700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc47006d7f0 0x7fc47006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.914+0000 7fc482000700 1 -- 172.21.15.1:0/1701033873 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc47808e1d0 con 0x7fc48c1095f0 2024-02-28T17:00:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.914+0000 7fc489e00700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc47006d7f0 0x7fc47006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:12.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.915+0000 7fc489e00700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc47006d7f0 0x7fc47006fca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc47c006fd0 tx=0x7fc47c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:12.918+0000 7fc482000700 1 -- 172.21.15.1:0/1701033873 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc47805b670 con 0x7fc48c1095f0 2024-02-28T17:00:13.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:12 smithi066 ceph-mon[85781]: pgmap v773: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:13.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.232+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc48c1106a0 con 0x7fc48c1095f0 2024-02-28T17:00:13.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.234+0000 7fc482000700 1 -- 172.21.15.1:0/1701033873 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fc47805b200 con 0x7fc48c1095f0 2024-02-28T17:00:13.233 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:00:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:00:13.233 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:00:13.234 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:00:13.235 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:00:13.235 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:00:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.236+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc47006d7f0 msgr2=0x7fc47006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.236+0000 7fc490e00700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc47006d7f0 0x7fc47006fca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc47c006fd0 tx=0x7fc47c008040 comp rx=0 tx=0).stop 2024-02-28T17:00:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.236+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c1095f0 msgr2=0x7fc48c11a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.236+0000 7fc490e00700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c1095f0 0x7fc48c11a570 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc478009150 tx=0x7fc478009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.237+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 shutdown_connections 2024-02-28T17:00:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.237+0000 7fc490e00700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc47006d7f0 0x7fc47006fca0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.237+0000 7fc490e00700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc48c103dc0 0x7fc48c11a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.237+0000 7fc490e00700 1 --2- 172.21.15.1:0/1701033873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc48c1095f0 0x7fc48c11a570 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.237+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 >> 172.21.15.1:0/1701033873 conn(0x7fc48c072d50 msgr2=0x7fc48c0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.237+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 shutdown_connections 2024-02-28T17:00:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.237+0000 7fc490e00700 1 -- 172.21.15.1:0/1701033873 wait complete. 2024-02-28T17:00:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.391+0000 7f26ad400700 1 -- 172.21.15.1:0/582095359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 msgr2=0x7f26a8070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.391+0000 7f26ad400700 1 --2- 172.21.15.1:0/582095359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 0x7f26a8070700 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2698004970 tx=0x7f2698009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.391+0000 7f26ad400700 1 -- 172.21.15.1:0/582095359 shutdown_connections 2024-02-28T17:00:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.391+0000 7f26ad400700 1 --2- 172.21.15.1:0/582095359 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26a8070c40 0x7f26a8109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.391+0000 7f26ad400700 1 --2- 172.21.15.1:0/582095359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 0x7f26a8070700 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.391+0000 7f26ad400700 1 -- 172.21.15.1:0/582095359 >> 172.21.15.1:0/582095359 conn(0x7f26a80f8ae0 msgr2=0x7f26a80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.391+0000 7f26ad400700 1 -- 172.21.15.1:0/582095359 shutdown_connections 2024-02-28T17:00:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.391+0000 7f26ad400700 1 -- 172.21.15.1:0/582095359 wait complete. 2024-02-28T17:00:13.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.392+0000 7f26ad400700 1 Processor -- start 2024-02-28T17:00:13.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.392+0000 7f26ad400700 1 -- start start 2024-02-28T17:00:13.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.392+0000 7f26ad400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 0x7f26a8115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:13.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.392+0000 7f26ad400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26a8070c40 0x7f26a8116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:13.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.392+0000 7f26ad400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a81167b0 con 0x7f26a8070330 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.392+0000 7f26ad400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a810fae0 con 0x7f26a8070c40 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.393+0000 7f26a6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 0x7f26a8115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.393+0000 7f26a6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 0x7f26a8115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44244/0 (socket says 172.21.15.1:44244) 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.393+0000 7f26a6800700 1 -- 172.21.15.1:0/1860007648 learned_addr learned my addr 172.21.15.1:0/1860007648 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.393+0000 7f26a6800700 1 -- 172.21.15.1:0/1860007648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26a8070c40 msgr2=0x7f26a8116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.393+0000 7f26a6800700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26a8070c40 0x7f26a8116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.393+0000 7f26a6800700 1 -- 172.21.15.1:0/1860007648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2698005040 con 0x7f26a8070330 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.393+0000 7f26a6800700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 0x7f26a8115c20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2698004f50 tx=0x7f2698009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.394+0000 7f269ea00700 1 -- 172.21.15.1:0/1860007648 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f269800e040 con 0x7f26a8070330 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.394+0000 7f269ea00700 1 -- 172.21.15.1:0/1860007648 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2698004290 con 0x7f26a8070330 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.394+0000 7f269ea00700 1 -- 172.21.15.1:0/1860007648 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f269801c630 con 0x7f26a8070330 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.394+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a810fd60 con 0x7f26a8070330 2024-02-28T17:00:13.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.394+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a81102a0 con 0x7f26a8070330 2024-02-28T17:00:13.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.395+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a8106920 con 0x7f26a8070330 2024-02-28T17:00:13.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.396+0000 7f269ea00700 1 -- 172.21.15.1:0/1860007648 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f269800bc10 con 0x7f26a8070330 2024-02-28T17:00:13.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.397+0000 7f269ea00700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f268c06d7f0 0x7f268c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:13.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.397+0000 7f269ea00700 1 -- 172.21.15.1:0/1860007648 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f269808e1f0 con 0x7f26a8070330 2024-02-28T17:00:13.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.397+0000 7f269fe00700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f268c06d7f0 0x7f268c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:13.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.398+0000 7f269fe00700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f268c06d7f0 0x7f268c06fca0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f26a8111450 tx=0x7f2690007cc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:13.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.402+0000 7f269ea00700 1 -- 172.21.15.1:0/1860007648 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f269805b690 con 0x7f26a8070330 2024-02-28T17:00:13.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.673+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f26a810c240 con 0x7f26a8070330 2024-02-28T17:00:13.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.674+0000 7f269ea00700 1 -- 172.21.15.1:0/1860007648 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f2698013e60 con 0x7f26a8070330 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:00:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:00:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:00:13.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:00:13.678 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:00:13.678 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:00:13.678 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:00:13.678 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:00:13.678 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:00:13.678 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:00:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f268c06d7f0 msgr2=0x7f268c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f268c06d7f0 0x7f268c06fca0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f26a8111450 tx=0x7f2690007cc0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 msgr2=0x7f26a8115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 0x7f26a8115c20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2698004f50 tx=0x7f2698009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 shutdown_connections 2024-02-28T17:00:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f268c06d7f0 0x7f268c06fca0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a8070330 0x7f26a8115c20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 --2- 172.21.15.1:0/1860007648 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f26a8070c40 0x7f26a8116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.677+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 >> 172.21.15.1:0/1860007648 conn(0x7f26a80f8ae0 msgr2=0x7f26a8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.678+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 shutdown_connections 2024-02-28T17:00:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.678+0000 7f26ad400700 1 -- 172.21.15.1:0/1860007648 wait complete. 2024-02-28T17:00:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:00:13.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.835+0000 7f07df200700 1 -- 172.21.15.1:0/2967145641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fe730 msgr2=0x7f07e0106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.835+0000 7f07df200700 1 --2- 172.21.15.1:0/2967145641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fe730 0x7f07e0106e00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f07cc004940 tx=0x7f07cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.836+0000 7f07df200700 1 -- 172.21.15.1:0/2967145641 shutdown_connections 2024-02-28T17:00:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.836+0000 7f07df200700 1 --2- 172.21.15.1:0/2967145641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fe730 0x7f07e0106e00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.836+0000 7f07df200700 1 --2- 172.21.15.1:0/2967145641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07e00fdd90 0x7f07e00fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.836+0000 7f07df200700 1 -- 172.21.15.1:0/2967145641 >> 172.21.15.1:0/2967145641 conn(0x7f07e0073ab0 msgr2=0x7f07e0073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.836+0000 7f07df200700 1 -- 172.21.15.1:0/2967145641 shutdown_connections 2024-02-28T17:00:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.836+0000 7f07df200700 1 -- 172.21.15.1:0/2967145641 wait complete. 2024-02-28T17:00:13.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.837+0000 7f07df200700 1 Processor -- start 2024-02-28T17:00:13.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.837+0000 7f07df200700 1 -- start start 2024-02-28T17:00:13.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.837+0000 7f07df200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fdd90 0x7f07e00fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.837+0000 7f07df200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07e00fe730 0x7f07e00fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.837+0000 7f07df200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e00fc790 con 0x7f07e00fdd90 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.837+0000 7f07df200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e00fc900 con 0x7f07e00fe730 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fdd90 0x7f07e00fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fdd90 0x7f07e00fbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44266/0 (socket says 172.21.15.1:44266) 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dde00700 1 -- 172.21.15.1:0/3245578315 learned_addr learned my addr 172.21.15.1:0/3245578315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dd400700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07e00fe730 0x7f07e00fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dde00700 1 -- 172.21.15.1:0/3245578315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07e00fe730 msgr2=0x7f07e00fc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dde00700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07e00fe730 0x7f07e00fc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:13.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dde00700 1 -- 172.21.15.1:0/3245578315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc005040 con 0x7f07e00fdd90 2024-02-28T17:00:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dd400700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07e00fe730 0x7f07e00fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:00:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.838+0000 7f07dde00700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fdd90 0x7f07e00fbc50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f07d0002a50 tx=0x7f07d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.839+0000 7f07d6000700 1 -- 172.21.15.1:0/3245578315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07d0007cc0 con 0x7f07e00fdd90 2024-02-28T17:00:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.839+0000 7f07d6000700 1 -- 172.21.15.1:0/3245578315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07d000f040 con 0x7f07e00fdd90 2024-02-28T17:00:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.839+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e0119b30 con 0x7f07e00fdd90 2024-02-28T17:00:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.839+0000 7f07d6000700 1 -- 172.21.15.1:0/3245578315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07d0009640 con 0x7f07e00fdd90 2024-02-28T17:00:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.839+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e011a010 con 0x7f07e00fdd90 2024-02-28T17:00:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.840+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e01044c0 con 0x7f07e00fdd90 2024-02-28T17:00:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.841+0000 7f07d6000700 1 -- 172.21.15.1:0/3245578315 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f07d00097a0 con 0x7f07e00fdd90 2024-02-28T17:00:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.842+0000 7f07d6000700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07c406d7e0 0x7f07c406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.842+0000 7f07d6000700 1 -- 172.21.15.1:0/3245578315 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f07d008cc40 con 0x7f07e00fdd90 2024-02-28T17:00:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.843+0000 7f07dd400700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07c406d7e0 0x7f07c406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:13.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.843+0000 7f07dd400700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07c406d7e0 0x7f07c406fc90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f07cc00b3b0 tx=0x7f07cc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:13.847+0000 7f07d6000700 1 -- 172.21.15.1:0/3245578315 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f07d0056470 con 0x7f07e00fdd90 2024-02-28T17:00:14.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:13 smithi066 ceph-mon[85781]: from='client.35402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:14.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:13 smithi066 ceph-mon[85781]: from='client.35406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:14.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:13 smithi066 ceph-mon[85781]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:14.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:13 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1701033873' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:00:14.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:13 smithi001 ceph-mon[112940]: from='client.35402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:14.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:13 smithi001 ceph-mon[112940]: from='client.35406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:14.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:13 smithi001 ceph-mon[112940]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:14.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:13 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1701033873' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:00:14.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.083+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07e00fd390 con 0x7f07c406d7e0 2024-02-28T17:00:14.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.085+0000 7f07d6000700 1 -- 172.21.15.1:0/3245578315 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f07e00fd390 con 0x7f07c406d7e0 2024-02-28T17:00:14.085 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:00:14.085 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:00:14.085 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:00:14.086 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:00:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07c406d7e0 msgr2=0x7f07c406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07c406d7e0 0x7f07c406fc90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f07cc00b3b0 tx=0x7f07cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fdd90 msgr2=0x7f07e00fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fdd90 0x7f07e00fbc50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f07d0002a50 tx=0x7f07d00046b0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 shutdown_connections 2024-02-28T17:00:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07c406d7e0 0x7f07c406fc90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e00fdd90 0x7f07e00fbc50 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 --2- 172.21.15.1:0/3245578315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07e00fe730 0x7f07e00fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 >> 172.21.15.1:0/3245578315 conn(0x7f07e0073ab0 msgr2=0x7f07e0101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 shutdown_connections 2024-02-28T17:00:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.088+0000 7f07df200700 1 -- 172.21.15.1:0/3245578315 wait complete. 2024-02-28T17:00:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.242+0000 7fe71cc00700 1 -- 172.21.15.1:0/18707721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7180724c0 msgr2=0x7fe718072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.242+0000 7fe71cc00700 1 --2- 172.21.15.1:0/18707721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7180724c0 0x7fe718072890 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe708004940 tx=0x7fe708009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.242+0000 7fe71cc00700 1 -- 172.21.15.1:0/18707721 shutdown_connections 2024-02-28T17:00:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.242+0000 7fe71cc00700 1 --2- 172.21.15.1:0/18707721 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe718072dd0 0x7fe71810d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.242+0000 7fe71cc00700 1 --2- 172.21.15.1:0/18707721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7180724c0 0x7fe718072890 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.242+0000 7fe71cc00700 1 -- 172.21.15.1:0/18707721 >> 172.21.15.1:0/18707721 conn(0x7fe7180fab40 msgr2=0x7fe7180fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:14.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.242+0000 7fe71cc00700 1 -- 172.21.15.1:0/18707721 shutdown_connections 2024-02-28T17:00:14.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.242+0000 7fe71cc00700 1 -- 172.21.15.1:0/18707721 wait complete. 2024-02-28T17:00:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.243+0000 7fe71cc00700 1 Processor -- start 2024-02-28T17:00:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.243+0000 7fe71cc00700 1 -- start start 2024-02-28T17:00:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.244+0000 7fe71cc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7180724c0 0x7fe718119dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.244+0000 7fe71cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe718072dd0 0x7fe71811a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.244+0000 7fe71cc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71811a950 con 0x7fe718072dd0 2024-02-28T17:00:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.244+0000 7fe71cc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe718113c60 con 0x7fe7180724c0 2024-02-28T17:00:14.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.244+0000 7fe716a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe718072dd0 0x7fe71811a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:14.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.244+0000 7fe716a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe718072dd0 0x7fe71811a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44294/0 (socket says 172.21.15.1:44294) 2024-02-28T17:00:14.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.244+0000 7fe716a00700 1 -- 172.21.15.1:0/3800859556 learned_addr learned my addr 172.21.15.1:0/3800859556 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:14.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.244+0000 7fe717400700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7180724c0 0x7fe718119dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:14.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe716a00700 1 -- 172.21.15.1:0/3800859556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7180724c0 msgr2=0x7fe718119dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:14.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe716a00700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7180724c0 0x7fe718119dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe716a00700 1 -- 172.21.15.1:0/3800859556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe708005040 con 0x7fe718072dd0 2024-02-28T17:00:14.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe716a00700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe718072dd0 0x7fe71811a300 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe704002a50 tx=0x7fe7040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:14.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe70f400700 1 -- 172.21.15.1:0/3800859556 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe704007cc0 con 0x7fe718072dd0 2024-02-28T17:00:14.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe718113f40 con 0x7fe718072dd0 2024-02-28T17:00:14.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe70f400700 1 -- 172.21.15.1:0/3800859556 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe70400f040 con 0x7fe718072dd0 2024-02-28T17:00:14.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe70f400700 1 -- 172.21.15.1:0/3800859556 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe70400db70 con 0x7fe718072dd0 2024-02-28T17:00:14.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.245+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe718114500 con 0x7fe718072dd0 2024-02-28T17:00:14.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.246+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71810ae30 con 0x7fe718072dd0 2024-02-28T17:00:14.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.248+0000 7fe70f400700 1 -- 172.21.15.1:0/3800859556 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe70400dd10 con 0x7fe718072dd0 2024-02-28T17:00:14.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.248+0000 7fe70f400700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe6fc06d7f0 0x7fe6fc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:14.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.249+0000 7fe70f400700 1 -- 172.21.15.1:0/3800859556 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe70408c670 con 0x7fe718072dd0 2024-02-28T17:00:14.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.249+0000 7fe717400700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe6fc06d7f0 0x7fe6fc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:14.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.249+0000 7fe717400700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe6fc06d7f0 0x7fe6fc06fca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe70800b3b0 tx=0x7fe708009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.253+0000 7fe70f400700 1 -- 172.21.15.1:0/3800859556 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe704059950 con 0x7fe718072dd0 2024-02-28T17:00:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.560+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe71804bed0 con 0x7fe718072dd0 2024-02-28T17:00:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.561+0000 7fe70f400700 1 -- 172.21.15.1:0/3800859556 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fe7040594e0 con 0x7fe718072dd0 2024-02-28T17:00:14.560 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:00:14.561 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:00:14.562 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:00:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.563+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe6fc06d7f0 msgr2=0x7fe6fc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:14.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.563+0000 7fe71cc00700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe6fc06d7f0 0x7fe6fc06fca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe70800b3b0 tx=0x7fe708009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.563+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe718072dd0 msgr2=0x7fe71811a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.563+0000 7fe71cc00700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe718072dd0 0x7fe71811a300 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe704002a50 tx=0x7fe7040046b0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.564+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 shutdown_connections 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.564+0000 7fe71cc00700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe6fc06d7f0 0x7fe6fc06fca0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.564+0000 7fe71cc00700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7180724c0 0x7fe718119dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.564+0000 7fe71cc00700 1 --2- 172.21.15.1:0/3800859556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe718072dd0 0x7fe71811a300 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.564+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 >> 172.21.15.1:0/3800859556 conn(0x7fe7180fab40 msgr2=0x7fe7180fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.564+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 shutdown_connections 2024-02-28T17:00:14.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:14.564+0000 7fe71cc00700 1 -- 172.21.15.1:0/3800859556 wait complete. 2024-02-28T17:00:15.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:14 smithi066 ceph-mon[85781]: pgmap v774: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:15.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:14 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1860007648' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:00:15.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:14 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3800859556' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:00:15.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:14 smithi001 ceph-mon[112940]: pgmap v774: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:15.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:14 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1860007648' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:00:15.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:14 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3800859556' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:00:16.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:15 smithi066 ceph-mon[85781]: from='client.35422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:16.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:15 smithi001 ceph-mon[112940]: from='client.35422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:17.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:16 smithi066 ceph-mon[85781]: pgmap v775: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:17.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:16 smithi001 ceph-mon[112940]: pgmap v775: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:19.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:18 smithi066 ceph-mon[85781]: pgmap v776: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:19.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:18 smithi001 ceph-mon[112940]: pgmap v776: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:21.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:20 smithi066 ceph-mon[85781]: pgmap v777: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:21.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:20 smithi001 ceph-mon[112940]: pgmap v777: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:23.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:22 smithi066 ceph-mon[85781]: pgmap v778: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:23.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:22 smithi001 ceph-mon[112940]: pgmap v778: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:25.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:24 smithi066 ceph-mon[85781]: pgmap v779: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:25.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:24 smithi001 ceph-mon[112940]: pgmap v779: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:27.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:26 smithi066 ceph-mon[85781]: pgmap v780: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:27.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:00:27.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:26 smithi001 ceph-mon[112940]: pgmap v780: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:27.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:00:29.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:28 smithi066 ceph-mon[85781]: pgmap v781: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:29.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:28 smithi001 ceph-mon[112940]: pgmap v781: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:31.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:30 smithi066 ceph-mon[85781]: pgmap v782: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:31.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:30 smithi001 ceph-mon[112940]: pgmap v782: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:33.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:32 smithi066 ceph-mon[85781]: pgmap v783: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:33.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:32 smithi001 ceph-mon[112940]: pgmap v783: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:35.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:34 smithi066 ceph-mon[85781]: pgmap v784: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:35.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:34 smithi001 ceph-mon[112940]: pgmap v784: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:37.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:36 smithi066 ceph-mon[85781]: pgmap v785: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:37.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:36 smithi001 ceph-mon[112940]: pgmap v785: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:39.008 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:38 smithi066 ceph-mon[85781]: pgmap v786: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:39.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:38 smithi001 ceph-mon[112940]: pgmap v786: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:41.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:40 smithi001 ceph-mon[112940]: pgmap v787: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:41.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:40 smithi066 ceph-mon[85781]: pgmap v787: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:42.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:00:42.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:00:43.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:42 smithi001 ceph-mon[112940]: pgmap v788: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:43.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:42 smithi066 ceph-mon[85781]: pgmap v788: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.740+0000 7f4105e00700 1 -- 172.21.15.1:0/2747086263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100101e40 msgr2=0x7f4100102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.740+0000 7f4105e00700 1 --2- 172.21.15.1:0/2747086263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100101e40 0x7f4100102210 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f40f0004940 tx=0x7f40f0009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.741+0000 7f4105e00700 1 -- 172.21.15.1:0/2747086263 shutdown_connections 2024-02-28T17:00:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.741+0000 7f4105e00700 1 --2- 172.21.15.1:0/2747086263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41001027e0 0x7f41000fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.741+0000 7f4105e00700 1 --2- 172.21.15.1:0/2747086263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100101e40 0x7f4100102210 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.741+0000 7f4105e00700 1 -- 172.21.15.1:0/2747086263 >> 172.21.15.1:0/2747086263 conn(0x7f4100071d90 msgr2=0x7f4100072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:44.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.741+0000 7f4105e00700 1 -- 172.21.15.1:0/2747086263 shutdown_connections 2024-02-28T17:00:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.741+0000 7f4105e00700 1 -- 172.21.15.1:0/2747086263 wait complete. 2024-02-28T17:00:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.742+0000 7f4105e00700 1 Processor -- start 2024-02-28T17:00:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.742+0000 7f4105e00700 1 -- start start 2024-02-28T17:00:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.743+0000 7f4105e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41001027e0 0x7f4100115e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.743+0000 7f4105e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100116370 0x7f410010fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.743+0000 7f4105e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41001168f0 con 0x7f4100116370 2024-02-28T17:00:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.743+0000 7f4105e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4100116a60 con 0x7f41001027e0 2024-02-28T17:00:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.743+0000 7f40f6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100116370 0x7f410010fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.743+0000 7f40f6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100116370 0x7f410010fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42780/0 (socket says 172.21.15.1:42780) 2024-02-28T17:00:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.743+0000 7f40f6000700 1 -- 172.21.15.1:0/3403436929 learned_addr learned my addr 172.21.15.1:0/3403436929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.744+0000 7f40fea00700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41001027e0 0x7f4100115e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.744+0000 7f40f6000700 1 -- 172.21.15.1:0/3403436929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41001027e0 msgr2=0x7f4100115e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.744+0000 7f40f6000700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41001027e0 0x7f4100115e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.744+0000 7f40f6000700 1 -- 172.21.15.1:0/3403436929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f0005040 con 0x7f4100116370 2024-02-28T17:00:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.744+0000 7f40f6000700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100116370 0x7f410010fcd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f40e8002a80 tx=0x7f40e8004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.744+0000 7f40f7400700 1 -- 172.21.15.1:0/3403436929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40e8007cc0 con 0x7f4100116370 2024-02-28T17:00:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.745+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41001102d0 con 0x7f4100116370 2024-02-28T17:00:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.745+0000 7f40f7400700 1 -- 172.21.15.1:0/3403436929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40e800f040 con 0x7f4100116370 2024-02-28T17:00:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.745+0000 7f40f7400700 1 -- 172.21.15.1:0/3403436929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40e800daf0 con 0x7f4100116370 2024-02-28T17:00:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.745+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4100110810 con 0x7f4100116370 2024-02-28T17:00:44.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.746+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4100110ae0 con 0x7f4100116370 2024-02-28T17:00:44.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.748+0000 7f40f7400700 1 -- 172.21.15.1:0/3403436929 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f40e800dcd0 con 0x7f4100116370 2024-02-28T17:00:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.749+0000 7f40f7400700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f40e406d7e0 0x7f40e406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.749+0000 7f40f7400700 1 -- 172.21.15.1:0/3403436929 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f40e808c590 con 0x7f4100116370 2024-02-28T17:00:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.749+0000 7f40fea00700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f40e406d7e0 0x7f40e406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.750+0000 7f40fea00700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f40e406d7e0 0x7f40e406fc90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f40f0000c00 tx=0x7f40f0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:44.755+0000 7f40f7400700 1 -- 172.21.15.1:0/3403436929 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f40e80598f0 con 0x7f4100116370 2024-02-28T17:00:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.001+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4100111410 con 0x7f40e406d7e0 2024-02-28T17:00:45.000 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:44 smithi001 ceph-mon[112940]: pgmap v789: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.003+0000 7f40f7400700 1 -- 172.21.15.1:0/3403436929 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4100111410 con 0x7f40e406d7e0 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f40e406d7e0 msgr2=0x7f40e406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f40e406d7e0 0x7f40e406fc90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f40f0000c00 tx=0x7f40f0009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100116370 msgr2=0x7f410010fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100116370 0x7f410010fcd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f40e8002a80 tx=0x7f40e8004630 comp rx=0 tx=0).stop 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 shutdown_connections 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f40e406d7e0 0x7f40e406fc90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f41001027e0 0x7f4100115e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 --2- 172.21.15.1:0/3403436929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4100116370 0x7f410010fcd0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 >> 172.21.15.1:0/3403436929 conn(0x7f4100071d90 msgr2=0x7f4100065860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 shutdown_connections 2024-02-28T17:00:45.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.006+0000 7f4105e00700 1 -- 172.21.15.1:0/3403436929 wait complete. 2024-02-28T17:00:45.021 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:00:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.166+0000 7f235a000700 1 -- 172.21.15.1:0/3246023919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 msgr2=0x7f23540fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.166+0000 7f235a000700 1 --2- 172.21.15.1:0/3246023919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 0x7f23540fcf60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2340004940 tx=0x7f2340009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.166+0000 7f235a000700 1 -- 172.21.15.1:0/3246023919 shutdown_connections 2024-02-28T17:00:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.166+0000 7f235a000700 1 --2- 172.21.15.1:0/3246023919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 0x7f23540fcf60 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.166+0000 7f235a000700 1 --2- 172.21.15.1:0/3246023919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2354102f50 0x7f2354103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.166+0000 7f235a000700 1 -- 172.21.15.1:0/3246023919 >> 172.21.15.1:0/3246023919 conn(0x7f23540f8ae0 msgr2=0x7f23540faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.166+0000 7f235a000700 1 -- 172.21.15.1:0/3246023919 shutdown_connections 2024-02-28T17:00:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.166+0000 7f235a000700 1 -- 172.21.15.1:0/3246023919 wait complete. 2024-02-28T17:00:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.167+0000 7f235a000700 1 Processor -- start 2024-02-28T17:00:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.167+0000 7f235a000700 1 -- start start 2024-02-28T17:00:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f235a000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2354102f50 0x7f2354070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f235a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 0x7f23540707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f235a000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2354070e60 con 0x7f23541038f0 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f235a000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2354070fd0 con 0x7f2354102f50 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f2352a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 0x7f23540707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f2352a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 0x7f23540707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42802/0 (socket says 172.21.15.1:42802) 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f2352a00700 1 -- 172.21.15.1:0/3939987407 learned_addr learned my addr 172.21.15.1:0/3939987407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f2353400700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2354102f50 0x7f2354070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f2352a00700 1 -- 172.21.15.1:0/3939987407 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2354102f50 msgr2=0x7f2354070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f2352a00700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2354102f50 0x7f2354070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.168+0000 7f2352a00700 1 -- 172.21.15.1:0/3939987407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2340005040 con 0x7f23541038f0 2024-02-28T17:00:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.169+0000 7f2352a00700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 0x7f23540707d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2340009030 tx=0x7f2340009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.169+0000 7f234b400700 1 -- 172.21.15.1:0/3939987407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f234000e040 con 0x7f23541038f0 2024-02-28T17:00:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.169+0000 7f234b400700 1 -- 172.21.15.1:0/3939987407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f234000bbf0 con 0x7f23541038f0 2024-02-28T17:00:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.169+0000 7f234b400700 1 -- 172.21.15.1:0/3939987407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f234001c630 con 0x7f23541038f0 2024-02-28T17:00:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.169+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2354119d00 con 0x7f23541038f0 2024-02-28T17:00:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.169+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235411a1f0 con 0x7f23541038f0 2024-02-28T17:00:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.170+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23540fc6e0 con 0x7f23541038f0 2024-02-28T17:00:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.172+0000 7f234b400700 1 -- 172.21.15.1:0/3939987407 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f234000bd60 con 0x7f23541038f0 2024-02-28T17:00:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.172+0000 7f234b400700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f233806d7f0 0x7f233806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.173+0000 7f234b400700 1 -- 172.21.15.1:0/3939987407 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f234008e190 con 0x7f23541038f0 2024-02-28T17:00:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.173+0000 7f2353400700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f233806d7f0 0x7f233806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.173+0000 7f2353400700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f233806d7f0 0x7f233806fca0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2344006fd0 tx=0x7f2344008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.177+0000 7f234b400700 1 -- 172.21.15.1:0/3939987407 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2340057a40 con 0x7f23541038f0 2024-02-28T17:00:45.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:44 smithi066 ceph-mon[85781]: pgmap v789: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:45.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.413+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23540fab90 con 0x7f233806d7f0 2024-02-28T17:00:45.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.415+0000 7f234b400700 1 -- 172.21.15.1:0/3939987407 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f23540fab90 con 0x7f233806d7f0 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f233806d7f0 msgr2=0x7f233806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f233806d7f0 0x7f233806fca0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2344006fd0 tx=0x7f2344008040 comp rx=0 tx=0).stop 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 msgr2=0x7f23540707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 0x7f23540707d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2340009030 tx=0x7f2340009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 shutdown_connections 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f233806d7f0 0x7f233806fca0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2354102f50 0x7f2354070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 --2- 172.21.15.1:0/3939987407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23541038f0 0x7f23540707d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 >> 172.21.15.1:0/3939987407 conn(0x7f23540f8ae0 msgr2=0x7f23540fa480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 shutdown_connections 2024-02-28T17:00:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.418+0000 7f235a000700 1 -- 172.21.15.1:0/3939987407 wait complete. 2024-02-28T17:00:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.577+0000 7f52ff600700 1 -- 172.21.15.1:0/2505797036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 msgr2=0x7f52f81099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.577+0000 7f52ff600700 1 --2- 172.21.15.1:0/2505797036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 0x7f52f81099c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f52ec004940 tx=0x7f52ec009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.577+0000 7f52ff600700 1 -- 172.21.15.1:0/2505797036 shutdown_connections 2024-02-28T17:00:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.577+0000 7f52ff600700 1 --2- 172.21.15.1:0/2505797036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52f8103dc0 0x7f52f8104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.577+0000 7f52ff600700 1 --2- 172.21.15.1:0/2505797036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 0x7f52f81099c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.577+0000 7f52ff600700 1 -- 172.21.15.1:0/2505797036 >> 172.21.15.1:0/2505797036 conn(0x7f52f8072d50 msgr2=0x7f52f8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.577+0000 7f52ff600700 1 -- 172.21.15.1:0/2505797036 shutdown_connections 2024-02-28T17:00:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.577+0000 7f52ff600700 1 -- 172.21.15.1:0/2505797036 wait complete. 2024-02-28T17:00:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.578+0000 7f52ff600700 1 Processor -- start 2024-02-28T17:00:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.578+0000 7f52ff600700 1 -- start start 2024-02-28T17:00:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.579+0000 7f52ff600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52f8103dc0 0x7f52f811a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:45.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.579+0000 7f52ff600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 0x7f52f811a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:45.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.579+0000 7f52ff600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f811ac70 con 0x7f52f81095f0 2024-02-28T17:00:45.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.579+0000 7f52ff600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f8113f80 con 0x7f52f8103dc0 2024-02-28T17:00:45.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.579+0000 7f52f7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 0x7f52f811a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:45.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.579+0000 7f52f7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 0x7f52f811a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42822/0 (socket says 172.21.15.1:42822) 2024-02-28T17:00:45.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.579+0000 7f52f7e00700 1 -- 172.21.15.1:0/2733632167 learned_addr learned my addr 172.21.15.1:0/2733632167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:45.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.579+0000 7f52fcc00700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52f8103dc0 0x7f52f811a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:45.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52f7e00700 1 -- 172.21.15.1:0/2733632167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52f8103dc0 msgr2=0x7f52f811a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52f7e00700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52f8103dc0 0x7f52f811a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52f7e00700 1 -- 172.21.15.1:0/2733632167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec005040 con 0x7f52f81095f0 2024-02-28T17:00:45.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52fcc00700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52f8103dc0 0x7f52f811a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:00:45.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52f7e00700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 0x7f52f811a620 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f52e4002a50 tx=0x7f52e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52f4c00700 1 -- 172.21.15.1:0/2733632167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e4007cc0 con 0x7f52f81095f0 2024-02-28T17:00:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52f4c00700 1 -- 172.21.15.1:0/2733632167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52e400f040 con 0x7f52f81095f0 2024-02-28T17:00:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52f4c00700 1 -- 172.21.15.1:0/2733632167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e4009640 con 0x7f52f81095f0 2024-02-28T17:00:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f8114260 con 0x7f52f81095f0 2024-02-28T17:00:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.580+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f81147a0 con 0x7f52f81095f0 2024-02-28T17:00:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.582+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f8106850 con 0x7f52f81095f0 2024-02-28T17:00:45.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.583+0000 7f52f4c00700 1 -- 172.21.15.1:0/2733632167 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f52e40097c0 con 0x7f52f81095f0 2024-02-28T17:00:45.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.585+0000 7f52f4c00700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f52e006d7f0 0x7f52e006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:45.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.585+0000 7f52f4c00700 1 -- 172.21.15.1:0/2733632167 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f52e408cb80 con 0x7f52f81095f0 2024-02-28T17:00:45.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.585+0000 7f52fcc00700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f52e006d7f0 0x7f52e006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:45.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.586+0000 7f52fcc00700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f52e006d7f0 0x7f52e006fca0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009fd0 tx=0x7f52ec009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:45.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.590+0000 7f52f4c00700 1 -- 172.21.15.1:0/2733632167 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f52e4059f80 con 0x7f52f81095f0 2024-02-28T17:00:45.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.831+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f52f804bed0 con 0x7f52e006d7f0 2024-02-28T17:00:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.860+0000 7f52f4c00700 1 -- 172.21.15.1:0/2733632167 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f52f804bed0 con 0x7f52e006d7f0 2024-02-28T17:00:45.859 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:00:45.859 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (26m) 3m ago 41m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:00:45.859 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (23m) 3m ago 41m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:00:45.859 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (23m) 3m ago 40m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:00:45.859 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (26m) 3m ago 40m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (19m) 3m ago 34m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (19m) 3m ago 33m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (33m) 3m ago 33m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (3m) 3m ago 33m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (29m) 3m ago 42m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (28m) 3m ago 39m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (25m) 3m ago 43m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (24m) 3m ago 39m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (27m) 3m ago 41m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (27m) 3m ago 39m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (22m) 3m ago 38m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (22m) 3m ago 38m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (21m) 3m ago 37m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:00:45.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (21m) 3m ago 37m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:00:45.861 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (20m) 3m ago 36m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:00:45.861 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (19m) 3m ago 35m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:00:45.861 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (27m) 3m ago 40m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.862+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f52e006d7f0 msgr2=0x7f52e006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f52e006d7f0 0x7f52e006fca0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009fd0 tx=0x7f52ec009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 msgr2=0x7f52f811a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 0x7f52f811a620 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f52e4002a50 tx=0x7f52e40046b0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 shutdown_connections 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f52e006d7f0 0x7f52e006fca0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52f8103dc0 0x7f52f811a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 --2- 172.21.15.1:0/2733632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52f81095f0 0x7f52f811a620 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 >> 172.21.15.1:0/2733632167 conn(0x7f52f8072d50 msgr2=0x7f52f8074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:45.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 shutdown_connections 2024-02-28T17:00:45.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:45.863+0000 7f52ff600700 1 -- 172.21.15.1:0/2733632167 wait complete. 2024-02-28T17:00:46.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.019+0000 7fca5e200700 1 -- 172.21.15.1:0/796991808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 msgr2=0x7fca58103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.019+0000 7fca5e200700 1 --2- 172.21.15.1:0/796991808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 0x7fca58103320 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca48004940 tx=0x7fca48009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:46.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.019+0000 7fca5e200700 1 -- 172.21.15.1:0/796991808 shutdown_connections 2024-02-28T17:00:46.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.019+0000 7fca5e200700 1 --2- 172.21.15.1:0/796991808 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca581038f0 0x7fca580fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.019+0000 7fca5e200700 1 --2- 172.21.15.1:0/796991808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 0x7fca58103320 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.019+0000 7fca5e200700 1 -- 172.21.15.1:0/796991808 >> 172.21.15.1:0/796991808 conn(0x7fca580f8ae0 msgr2=0x7fca580faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:46.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.020+0000 7fca5e200700 1 -- 172.21.15.1:0/796991808 shutdown_connections 2024-02-28T17:00:46.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.020+0000 7fca5e200700 1 -- 172.21.15.1:0/796991808 wait complete. 2024-02-28T17:00:46.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.020+0000 7fca5e200700 1 Processor -- start 2024-02-28T17:00:46.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.021+0000 7fca5e200700 1 -- start start 2024-02-28T17:00:46.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.021+0000 7fca5e200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 0x7fca58115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.021+0000 7fca5e200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca581038f0 0x7fca58116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.021+0000 7fca5e200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca581167a0 con 0x7fca58102f50 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.021+0000 7fca5e200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5810faf0 con 0x7fca581038f0 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.021+0000 7fca57400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 0x7fca58115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.021+0000 7fca57400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 0x7fca58115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42846/0 (socket says 172.21.15.1:42846) 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.021+0000 7fca57400700 1 -- 172.21.15.1:0/2919356741 learned_addr learned my addr 172.21.15.1:0/2919356741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.022+0000 7fca57400700 1 -- 172.21.15.1:0/2919356741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca581038f0 msgr2=0x7fca58116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.022+0000 7fca4fe00700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca581038f0 0x7fca58116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.022+0000 7fca57400700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca581038f0 0x7fca58116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.022+0000 7fca57400700 1 -- 172.21.15.1:0/2919356741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca48005040 con 0x7fca58102f50 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.022+0000 7fca4fe00700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca581038f0 0x7fca58116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.022+0000 7fca57400700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 0x7fca58115c10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fca4800b3b0 tx=0x7fca48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.022+0000 7fca4f400700 1 -- 172.21.15.1:0/2919356741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca4800e040 con 0x7fca58102f50 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.023+0000 7fca4f400700 1 -- 172.21.15.1:0/2919356741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca4800bbf0 con 0x7fca58102f50 2024-02-28T17:00:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.023+0000 7fca4f400700 1 -- 172.21.15.1:0/2919356741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca4801c630 con 0x7fca58102f50 2024-02-28T17:00:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.023+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5810fd70 con 0x7fca58102f50 2024-02-28T17:00:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.023+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca581102b0 con 0x7fca58102f50 2024-02-28T17:00:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.024+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca580fc6e0 con 0x7fca58102f50 2024-02-28T17:00:46.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.025+0000 7fca4f400700 1 -- 172.21.15.1:0/2919356741 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fca4800bd60 con 0x7fca58102f50 2024-02-28T17:00:46.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.026+0000 7fca4f400700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca3c06d7e0 0x7fca3c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.026+0000 7fca4f400700 1 -- 172.21.15.1:0/2919356741 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fca4808e2a0 con 0x7fca58102f50 2024-02-28T17:00:46.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.027+0000 7fca4fe00700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca3c06d7e0 0x7fca3c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.027+0000 7fca4fe00700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca3c06d7e0 0x7fca3c06fc90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fca40007fd0 tx=0x7fca40009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:46.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.031+0000 7fca4f400700 1 -- 172.21.15.1:0/2919356741 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fca48057b50 con 0x7fca58102f50 2024-02-28T17:00:46.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.343+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fca5810a050 con 0x7fca58102f50 2024-02-28T17:00:46.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.345+0000 7fca4f400700 1 -- 172.21.15.1:0/2919356741 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fca4805b210 con 0x7fca58102f50 2024-02-28T17:00:46.344 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:00:46.344 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:00:46.344 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:00:46.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:00:46.345 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:00:46.346 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.347+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca3c06d7e0 msgr2=0x7fca3c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.347+0000 7fca5e200700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca3c06d7e0 0x7fca3c06fc90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fca40007fd0 tx=0x7fca40009040 comp rx=0 tx=0).stop 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.347+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 msgr2=0x7fca58115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.347+0000 7fca5e200700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 0x7fca58115c10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fca4800b3b0 tx=0x7fca48009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.348+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 shutdown_connections 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.348+0000 7fca5e200700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca3c06d7e0 0x7fca3c06fc90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.348+0000 7fca5e200700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca58102f50 0x7fca58115c10 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.348+0000 7fca5e200700 1 --2- 172.21.15.1:0/2919356741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca581038f0 0x7fca58116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.348+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 >> 172.21.15.1:0/2919356741 conn(0x7fca580f8ae0 msgr2=0x7fca580fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.348+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 shutdown_connections 2024-02-28T17:00:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.348+0000 7fca5e200700 1 -- 172.21.15.1:0/2919356741 wait complete. 2024-02-28T17:00:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.492+0000 7fa790c00700 1 -- 172.21.15.1:0/1075212879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c102f50 msgr2=0x7fa78c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.492+0000 7fa790c00700 1 --2- 172.21.15.1:0/1075212879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c102f50 0x7fa78c103320 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa77c004940 tx=0x7fa77c009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:46.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.492+0000 7fa790c00700 1 -- 172.21.15.1:0/1075212879 shutdown_connections 2024-02-28T17:00:46.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.492+0000 7fa790c00700 1 --2- 172.21.15.1:0/1075212879 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa78c1038f0 0x7fa78c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.492+0000 7fa790c00700 1 --2- 172.21.15.1:0/1075212879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c102f50 0x7fa78c103320 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.492+0000 7fa790c00700 1 -- 172.21.15.1:0/1075212879 >> 172.21.15.1:0/1075212879 conn(0x7fa78c0f8ae0 msgr2=0x7fa78c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:46.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.492+0000 7fa790c00700 1 -- 172.21.15.1:0/1075212879 shutdown_connections 2024-02-28T17:00:46.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.492+0000 7fa790c00700 1 -- 172.21.15.1:0/1075212879 wait complete. 2024-02-28T17:00:46.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.493+0000 7fa790c00700 1 Processor -- start 2024-02-28T17:00:46.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.493+0000 7fa790c00700 1 -- start start 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.494+0000 7fa790c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa78c102f50 0x7fa78c115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.494+0000 7fa790c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c1038f0 0x7fa78c1160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.494+0000 7fa790c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c116710 con 0x7fa78c1038f0 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.494+0000 7fa790c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c10fad0 con 0x7fa78c102f50 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.494+0000 7fa789400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c1038f0 0x7fa78c1160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.494+0000 7fa789400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c1038f0 0x7fa78c1160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42866/0 (socket says 172.21.15.1:42866) 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.494+0000 7fa789400700 1 -- 172.21.15.1:0/2286203319 learned_addr learned my addr 172.21.15.1:0/2286203319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa789e00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa78c102f50 0x7fa78c115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa789400700 1 -- 172.21.15.1:0/2286203319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa78c102f50 msgr2=0x7fa78c115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa789400700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa78c102f50 0x7fa78c115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa789400700 1 -- 172.21.15.1:0/2286203319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c005040 con 0x7fa78c1038f0 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa789e00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa78c102f50 0x7fa78c115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:00:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa789400700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c1038f0 0x7fa78c1160c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa778002a50 tx=0x7fa778004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa782000700 1 -- 172.21.15.1:0/2286203319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa778007cc0 con 0x7fa78c1038f0 2024-02-28T17:00:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa782000700 1 -- 172.21.15.1:0/2286203319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa77800f040 con 0x7fa78c1038f0 2024-02-28T17:00:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa782000700 1 -- 172.21.15.1:0/2286203319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa77800da40 con 0x7fa78c1038f0 2024-02-28T17:00:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c10fdb0 con 0x7fa78c1038f0 2024-02-28T17:00:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.495+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c1102f0 con 0x7fa78c1038f0 2024-02-28T17:00:46.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.496+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78c0fc6e0 con 0x7fa78c1038f0 2024-02-28T17:00:46.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.498+0000 7fa782000700 1 -- 172.21.15.1:0/2286203319 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa7780097c0 con 0x7fa78c1038f0 2024-02-28T17:00:46.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.498+0000 7fa782000700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa77006d7f0 0x7fa77006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.499+0000 7fa782000700 1 -- 172.21.15.1:0/2286203319 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa77808cd50 con 0x7fa78c1038f0 2024-02-28T17:00:46.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.499+0000 7fa789e00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa77006d7f0 0x7fa77006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.499+0000 7fa789e00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa77006d7f0 0x7fa77006fca0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009fd0 tx=0x7fa77c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:46.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.504+0000 7fa782000700 1 -- 172.21.15.1:0/2286203319 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa77805a030 con 0x7fa78c1038f0 2024-02-28T17:00:46.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.763+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa78c04bed0 con 0x7fa78c1038f0 2024-02-28T17:00:46.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.764+0000 7fa782000700 1 -- 172.21.15.1:0/2286203319 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fa778059bc0 con 0x7fa78c1038f0 2024-02-28T17:00:46.764 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:00:46.764 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:00:46.764 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:00:46.764 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:00:46.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:00:46.765 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:00:46.766 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:00:46.767 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:00:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.767+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa77006d7f0 msgr2=0x7fa77006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.767+0000 7fa790c00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa77006d7f0 0x7fa77006fca0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009fd0 tx=0x7fa77c009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.767+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c1038f0 msgr2=0x7fa78c1160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.767+0000 7fa790c00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c1038f0 0x7fa78c1160c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa778002a50 tx=0x7fa778004600 comp rx=0 tx=0).stop 2024-02-28T17:00:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.768+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 shutdown_connections 2024-02-28T17:00:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.768+0000 7fa790c00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa77006d7f0 0x7fa77006fca0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.768+0000 7fa790c00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa78c102f50 0x7fa78c115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.768+0000 7fa790c00700 1 --2- 172.21.15.1:0/2286203319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c1038f0 0x7fa78c1160c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.768+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 >> 172.21.15.1:0/2286203319 conn(0x7fa78c0f8ae0 msgr2=0x7fa78c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.768+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 shutdown_connections 2024-02-28T17:00:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.768+0000 7fa790c00700 1 -- 172.21.15.1:0/2286203319 wait complete. 2024-02-28T17:00:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:00:46.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.921+0000 7f28e4800700 1 -- 172.21.15.1:0/605279417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fdf40 msgr2=0x7f28dc0fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.921+0000 7f28e4800700 1 --2- 172.21.15.1:0/605279417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fdf40 0x7f28dc0fe310 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f28d0004940 tx=0x7f28d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:46.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.921+0000 7f28e4800700 1 -- 172.21.15.1:0/605279417 shutdown_connections 2024-02-28T17:00:46.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.921+0000 7f28e4800700 1 --2- 172.21.15.1:0/605279417 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28dc0fe8e0 0x7f28dc106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.921+0000 7f28e4800700 1 --2- 172.21.15.1:0/605279417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fdf40 0x7f28dc0fe310 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.921+0000 7f28e4800700 1 -- 172.21.15.1:0/605279417 >> 172.21.15.1:0/605279417 conn(0x7f28dc0f79d0 msgr2=0x7f28dc0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:46.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.921+0000 7f28e4800700 1 -- 172.21.15.1:0/605279417 shutdown_connections 2024-02-28T17:00:46.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.921+0000 7f28e4800700 1 -- 172.21.15.1:0/605279417 wait complete. 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.922+0000 7f28e4800700 1 Processor -- start 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.922+0000 7f28e4800700 1 -- start start 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.922+0000 7f28e4800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28dc0fdf40 0x7f28dc115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.922+0000 7f28e4800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fe8e0 0x7f28dc116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e4800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc116820 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e4800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc10fae0 con 0x7f28dc0fdf40 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fe8e0 0x7f28dc116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fe8e0 0x7f28dc116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42876/0 (socket says 172.21.15.1:42876) 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e1400700 1 -- 172.21.15.1:0/3285362233 learned_addr learned my addr 172.21.15.1:0/3285362233 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e1e00700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28dc0fdf40 0x7f28dc115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e1400700 1 -- 172.21.15.1:0/3285362233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28dc0fdf40 msgr2=0x7f28dc115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e1400700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28dc0fdf40 0x7f28dc115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e1400700 1 -- 172.21.15.1:0/3285362233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d0005040 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.923+0000 7f28e1e00700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28dc0fdf40 0x7f28dc115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:00:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.924+0000 7f28e1400700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fe8e0 0x7f28dc116140 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f28cc00b790 tx=0x7f28cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.924+0000 7f28d6000700 1 -- 172.21.15.1:0/3285362233 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28cc00f770 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.924+0000 7f28d6000700 1 -- 172.21.15.1:0/3285362233 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28cc00fda0 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.924+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc10fdc0 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.924+0000 7f28d6000700 1 -- 172.21.15.1:0/3285362233 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28cc00d310 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.924+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc110300 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.925+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc106990 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.926+0000 7f28d6000700 1 -- 172.21.15.1:0/3285362233 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f28cc010420 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.927+0000 7f28d6000700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28c406d7f0 0x7f28c406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.927+0000 7f28d6000700 1 -- 172.21.15.1:0/3285362233 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f28cc08cf70 con 0x7f28dc0fe8e0 2024-02-28T17:00:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.927+0000 7f28e1e00700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28c406d7f0 0x7f28c406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.928+0000 7f28e1e00700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28c406d7f0 0x7f28c406fca0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f28d0004f90 tx=0x7f28d0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:46.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:46.932+0000 7f28d6000700 1 -- 172.21.15.1:0/3285362233 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f28cc0567a0 con 0x7f28dc0fe8e0 2024-02-28T17:00:47.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:46 smithi001 ceph-mon[112940]: pgmap v790: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:47.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:46 smithi001 ceph-mon[112940]: from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:47.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:46 smithi001 ceph-mon[112940]: from='client.35434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:47.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:46 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2919356741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:00:47.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:46 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2286203319' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:00:47.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.176+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28dc04bf40 con 0x7f28c406d7f0 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.178+0000 7f28d6000700 1 -- 172.21.15.1:0/3285362233 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f28dc04bf40 con 0x7f28c406d7f0 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:00:47.178 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:00:47.179 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:00:47.179 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:00:47.179 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:00:47.179 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:00:47.179 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:00:47.179 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:00:47.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28c406d7f0 msgr2=0x7f28c406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28c406d7f0 0x7f28c406fca0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f28d0004f90 tx=0x7f28d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fe8e0 msgr2=0x7f28dc116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fe8e0 0x7f28dc116140 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f28cc00b790 tx=0x7f28cc009250 comp rx=0 tx=0).stop 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 shutdown_connections 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28c406d7f0 0x7f28c406fca0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28dc0fdf40 0x7f28dc115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 --2- 172.21.15.1:0/3285362233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28dc0fe8e0 0x7f28dc116140 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.181+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 >> 172.21.15.1:0/3285362233 conn(0x7f28dc0f79d0 msgr2=0x7f28dc0fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.182+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 shutdown_connections 2024-02-28T17:00:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.182+0000 7f28e4800700 1 -- 172.21.15.1:0/3285362233 wait complete. 2024-02-28T17:00:47.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:46 smithi066 ceph-mon[85781]: pgmap v790: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:47.259 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:46 smithi066 ceph-mon[85781]: from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:47.259 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:46 smithi066 ceph-mon[85781]: from='client.35434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:47.259 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:46 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2919356741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:00:47.259 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:46 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2286203319' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:00:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.336+0000 7f6a3c400700 1 -- 172.21.15.1:0/4018231908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 msgr2=0x7f6a340fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.336+0000 7f6a3c400700 1 --2- 172.21.15.1:0/4018231908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 0x7f6a340fe3b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6a28004970 tx=0x7f6a28009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.337+0000 7f6a3c400700 1 -- 172.21.15.1:0/4018231908 shutdown_connections 2024-02-28T17:00:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.337+0000 7f6a3c400700 1 --2- 172.21.15.1:0/4018231908 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a340fe8f0 0x7f6a34107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.337+0000 7f6a3c400700 1 --2- 172.21.15.1:0/4018231908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 0x7f6a340fe3b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.337+0000 7f6a3c400700 1 -- 172.21.15.1:0/4018231908 >> 172.21.15.1:0/4018231908 conn(0x7f6a34071d90 msgr2=0x7f6a34072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.337+0000 7f6a3c400700 1 -- 172.21.15.1:0/4018231908 shutdown_connections 2024-02-28T17:00:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.337+0000 7f6a3c400700 1 -- 172.21.15.1:0/4018231908 wait complete. 2024-02-28T17:00:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.338+0000 7f6a3c400700 1 Processor -- start 2024-02-28T17:00:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.338+0000 7f6a3c400700 1 -- start start 2024-02-28T17:00:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.339+0000 7f6a3c400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 0x7f6a34115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.339+0000 7f6a3c400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a340fe8f0 0x7f6a34116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.339+0000 7f6a3c400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a34116790 con 0x7f6a340fdfe0 2024-02-28T17:00:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.339+0000 7f6a3c400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3410fae0 con 0x7f6a340fe8f0 2024-02-28T17:00:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.339+0000 7f6a39a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 0x7f6a34115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.339+0000 7f6a39a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 0x7f6a34115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42906/0 (socket says 172.21.15.1:42906) 2024-02-28T17:00:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.339+0000 7f6a39a00700 1 -- 172.21.15.1:0/2549158806 learned_addr learned my addr 172.21.15.1:0/2549158806 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:00:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a39000700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a340fe8f0 0x7f6a34116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a39a00700 1 -- 172.21.15.1:0/2549158806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a340fe8f0 msgr2=0x7f6a34116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a39a00700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a340fe8f0 0x7f6a34116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a39a00700 1 -- 172.21.15.1:0/2549158806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a28005040 con 0x7f6a340fdfe0 2024-02-28T17:00:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a39000700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a340fe8f0 0x7f6a34116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:00:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a39a00700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 0x7f6a34115c00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009150 tx=0x7f6a28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:47.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a31600700 1 -- 172.21.15.1:0/2549158806 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a2800e040 con 0x7f6a340fdfe0 2024-02-28T17:00:47.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a31600700 1 -- 172.21.15.1:0/2549158806 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a28004290 con 0x7f6a340fdfe0 2024-02-28T17:00:47.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a31600700 1 -- 172.21.15.1:0/2549158806 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a2801c630 con 0x7f6a340fdfe0 2024-02-28T17:00:47.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.340+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3410fd60 con 0x7f6a340fdfe0 2024-02-28T17:00:47.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.341+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a341102a0 con 0x7f6a340fdfe0 2024-02-28T17:00:47.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.342+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a34106b20 con 0x7f6a340fdfe0 2024-02-28T17:00:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.343+0000 7f6a31600700 1 -- 172.21.15.1:0/2549158806 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6a28013980 con 0x7f6a340fdfe0 2024-02-28T17:00:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.344+0000 7f6a31600700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a1c06d7f0 0x7f6a1c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:00:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.344+0000 7f6a31600700 1 -- 172.21.15.1:0/2549158806 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6a2808e210 con 0x7f6a340fdfe0 2024-02-28T17:00:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.344+0000 7f6a39000700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a1c06d7f0 0x7f6a1c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:00:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.345+0000 7f6a39000700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a1c06d7f0 0x7f6a1c06fca0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6a20007fd0 tx=0x7f6a20009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:00:47.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.348+0000 7f6a31600700 1 -- 172.21.15.1:0/2549158806 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a28057ac0 con 0x7f6a340fdfe0 2024-02-28T17:00:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.660+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6a3404bed0 con 0x7f6a340fdfe0 2024-02-28T17:00:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.661+0000 7f6a31600700 1 -- 172.21.15.1:0/2549158806 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f6a2805b180 con 0x7f6a340fdfe0 2024-02-28T17:00:47.660 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:00:47.660 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:00:47.660 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:00:47.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:00:47.661 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:00:47.662 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:00:47.662 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:00:47.662 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:00:47.662 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:00:47.662 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:00:47.662 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:00:47.662 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:00:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a1c06d7f0 msgr2=0x7f6a1c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a1c06d7f0 0x7f6a1c06fca0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6a20007fd0 tx=0x7f6a20009040 comp rx=0 tx=0).stop 2024-02-28T17:00:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 msgr2=0x7f6a34115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:00:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 0x7f6a34115c00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009150 tx=0x7f6a28009e30 comp rx=0 tx=0).stop 2024-02-28T17:00:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 shutdown_connections 2024-02-28T17:00:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a1c06d7f0 0x7f6a1c06fca0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a340fdfe0 0x7f6a34115c00 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 --2- 172.21.15.1:0/2549158806 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a340fe8f0 0x7f6a34116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:00:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 >> 172.21.15.1:0/2549158806 conn(0x7f6a34071d90 msgr2=0x7f6a340fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:00:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 shutdown_connections 2024-02-28T17:00:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:00:47.664+0000 7f6a3c400700 1 -- 172.21.15.1:0/2549158806 wait complete. 2024-02-28T17:00:48.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:47 smithi001 ceph-mon[112940]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:48.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:47 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2549158806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:00:48.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:47 smithi066 ceph-mon[85781]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:48.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:47 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2549158806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:00:49.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:48 smithi066 ceph-mon[85781]: pgmap v791: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:49.259 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:48 smithi066 ceph-mon[85781]: from='client.35450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:48 smithi001 ceph-mon[112940]: pgmap v791: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:49.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:48 smithi001 ceph-mon[112940]: from='client.35450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:00:50.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:49 smithi066 ceph-mon[85781]: pgmap v792: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:50.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:49 smithi001 ceph-mon[112940]: pgmap v792: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:52.258 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:52 smithi066 ceph-mon[85781]: pgmap v793: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:52.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:52 smithi001 ceph-mon[112940]: pgmap v793: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:54.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:54 smithi001 ceph-mon[112940]: pgmap v794: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:54.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:54 smithi066 ceph-mon[85781]: pgmap v794: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:56.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:56 smithi001 ceph-mon[112940]: pgmap v795: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:56.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:00:56.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:56 smithi066 ceph-mon[85781]: pgmap v795: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:56.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:00:58.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:00:58 smithi001 ceph-mon[112940]: pgmap v796: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:00:58.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:00:58 smithi066 ceph-mon[85781]: pgmap v796: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:00.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:00 smithi066 ceph-mon[85781]: pgmap v797: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:00.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:00 smithi001 ceph-mon[112940]: pgmap v797: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:02.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:02 smithi001 ceph-mon[112940]: pgmap v798: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:02.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:02 smithi066 ceph-mon[85781]: pgmap v798: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:04.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:04 smithi066 ceph-mon[85781]: pgmap v799: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:04.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:04 smithi001 ceph-mon[112940]: pgmap v799: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:06.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:06 smithi066 ceph-mon[85781]: pgmap v800: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:06.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:06 smithi001 ceph-mon[112940]: pgmap v800: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:08.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:08 smithi066 ceph-mon[85781]: pgmap v801: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:08.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:08 smithi001 ceph-mon[112940]: pgmap v801: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:10.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:10 smithi066 ceph-mon[85781]: pgmap v802: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:10.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:10 smithi001 ceph-mon[112940]: pgmap v802: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:11.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:01:11.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:01:12.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:12 smithi066 ceph-mon[85781]: pgmap v803: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:12.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:12 smithi001 ceph-mon[112940]: pgmap v803: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:14.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:14 smithi066 ceph-mon[85781]: pgmap v804: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:14.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:14 smithi001 ceph-mon[112940]: pgmap v804: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:16.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:16 smithi066 ceph-mon[85781]: pgmap v805: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:16.543 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:16 smithi001 ceph-mon[112940]: pgmap v805: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:17.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.835+0000 7f6d82e00700 1 -- 172.21.15.1:0/1053196475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070c40 msgr2=0x7f6d7c109050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:17.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.835+0000 7f6d82e00700 1 --2- 172.21.15.1:0/1053196475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070c40 0x7f6d7c109050 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6d68004970 tx=0x7f6d68009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:17.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.836+0000 7f6d82e00700 1 -- 172.21.15.1:0/1053196475 shutdown_connections 2024-02-28T17:01:17.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.836+0000 7f6d82e00700 1 --2- 172.21.15.1:0/1053196475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070c40 0x7f6d7c109050 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:17.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.836+0000 7f6d82e00700 1 --2- 172.21.15.1:0/1053196475 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d7c070330 0x7f6d7c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:17.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.836+0000 7f6d82e00700 1 -- 172.21.15.1:0/1053196475 >> 172.21.15.1:0/1053196475 conn(0x7f6d7c0f88b0 msgr2=0x7f6d7c0facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:17.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.836+0000 7f6d82e00700 1 -- 172.21.15.1:0/1053196475 shutdown_connections 2024-02-28T17:01:17.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.836+0000 7f6d82e00700 1 -- 172.21.15.1:0/1053196475 wait complete. 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.837+0000 7f6d82e00700 1 Processor -- start 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.837+0000 7f6d82e00700 1 -- start start 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d82e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070330 0x7f6d7c115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d82e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d7c070c40 0x7f6d7c115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d82e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c116690 con 0x7f6d7c070330 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d82e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c10f930 con 0x7f6d7c070c40 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d81a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070330 0x7f6d7c115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d81a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070330 0x7f6d7c115a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45908/0 (socket says 172.21.15.1:45908) 2024-02-28T17:01:17.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d81a00700 1 -- 172.21.15.1:0/1106790991 learned_addr learned my addr 172.21.15.1:0/1106790991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:17.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d81000700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d7c070c40 0x7f6d7c115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:17.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d81a00700 1 -- 172.21.15.1:0/1106790991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d7c070c40 msgr2=0x7f6d7c115fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d81a00700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d7c070c40 0x7f6d7c115fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.838+0000 7f6d81a00700 1 -- 172.21.15.1:0/1106790991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d68005040 con 0x7f6d7c070330 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.839+0000 7f6d81a00700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070330 0x7f6d7c115a70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000b790 tx=0x7f6d70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.839+0000 7f6d79600700 1 -- 172.21.15.1:0/1106790991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d7000f770 con 0x7f6d7c070330 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.839+0000 7f6d79600700 1 -- 172.21.15.1:0/1106790991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d7000fda0 con 0x7f6d7c070330 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.839+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7c10fc10 con 0x7f6d7c070330 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.839+0000 7f6d79600700 1 -- 172.21.15.1:0/1106790991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d7000d310 con 0x7f6d7c070330 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.839+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7c110120 con 0x7f6d7c070330 2024-02-28T17:01:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.840+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7c106710 con 0x7f6d7c070330 2024-02-28T17:01:17.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.841+0000 7f6d79600700 1 -- 172.21.15.1:0/1106790991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6d70010420 con 0x7f6d7c070330 2024-02-28T17:01:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.842+0000 7f6d79600700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.843+0000 7f6d79600700 1 -- 172.21.15.1:0/1106790991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6d7008cf10 con 0x7f6d7c070330 2024-02-28T17:01:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.843+0000 7f6d81000700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.844+0000 7f6d81000700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c110c60 tx=0x7f6d68009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:17.847+0000 7f6d79600700 1 -- 172.21.15.1:0/1106790991 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d7005a310 con 0x7f6d7c070330 2024-02-28T17:01:18.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.092+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d7c04bf70 con 0x7f6d6406d7f0 2024-02-28T17:01:18.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.094+0000 7f6d79600700 1 -- 172.21.15.1:0/1106790991 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6d68008e80 con 0x7f6d6406d7f0 2024-02-28T17:01:18.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 msgr2=0x7f6d6406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c110c60 tx=0x7f6d68009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:18.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070330 msgr2=0x7f6d7c115a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070330 0x7f6d7c115a70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000b790 tx=0x7f6d70009250 comp rx=0 tx=0).stop 2024-02-28T17:01:18.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 shutdown_connections 2024-02-28T17:01:18.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6d6406d7f0 0x7f6d6406fca0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d7c070330 0x7f6d7c115a70 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 --2- 172.21.15.1:0/1106790991 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6d7c070c40 0x7f6d7c115fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.098+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 >> 172.21.15.1:0/1106790991 conn(0x7f6d7c0f88b0 msgr2=0x7f6d7c103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:18.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.099+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 shutdown_connections 2024-02-28T17:01:18.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.099+0000 7f6d82e00700 1 -- 172.21.15.1:0/1106790991 wait complete. 2024-02-28T17:01:18.115 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:01:18.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.257+0000 7feca8e00700 1 -- 172.21.15.1:0/1084991392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fe8e0 msgr2=0x7feca4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.258+0000 7feca8e00700 1 --2- 172.21.15.1:0/1084991392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fe8e0 0x7feca4106f20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fec90004940 tx=0x7fec90009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.258+0000 7feca8e00700 1 -- 172.21.15.1:0/1084991392 shutdown_connections 2024-02-28T17:01:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.258+0000 7feca8e00700 1 --2- 172.21.15.1:0/1084991392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fe8e0 0x7feca4106f20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.258+0000 7feca8e00700 1 --2- 172.21.15.1:0/1084991392 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feca40fdf40 0x7feca40fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.258+0000 7feca8e00700 1 -- 172.21.15.1:0/1084991392 >> 172.21.15.1:0/1084991392 conn(0x7feca40f79d0 msgr2=0x7feca40f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.258+0000 7feca8e00700 1 -- 172.21.15.1:0/1084991392 shutdown_connections 2024-02-28T17:01:18.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.258+0000 7feca8e00700 1 -- 172.21.15.1:0/1084991392 wait complete. 2024-02-28T17:01:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.259+0000 7feca8e00700 1 Processor -- start 2024-02-28T17:01:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.259+0000 7feca8e00700 1 -- start start 2024-02-28T17:01:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.259+0000 7feca8e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fdf40 0x7feca4115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.259+0000 7feca8e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feca40fe8e0 0x7feca4116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.259+0000 7feca8e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca41167b0 con 0x7feca40fdf40 2024-02-28T17:01:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.259+0000 7feca8e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca410fb00 con 0x7feca40fe8e0 2024-02-28T17:01:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.260+0000 7feca1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fdf40 0x7feca4115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:18.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.260+0000 7feca1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fdf40 0x7feca4115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45930/0 (socket says 172.21.15.1:45930) 2024-02-28T17:01:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.260+0000 7feca1e00700 1 -- 172.21.15.1:0/1187496151 learned_addr learned my addr 172.21.15.1:0/1187496151 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.260+0000 7feca1400700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feca40fe8e0 0x7feca4116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.260+0000 7feca1e00700 1 -- 172.21.15.1:0/1187496151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feca40fe8e0 msgr2=0x7feca4116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.260+0000 7feca1e00700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feca40fe8e0 0x7feca4116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.260+0000 7feca1e00700 1 -- 172.21.15.1:0/1187496151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec90005040 con 0x7feca40fdf40 2024-02-28T17:01:18.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.261+0000 7feca1e00700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fdf40 0x7feca4115c20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fec9400c830 tx=0x7fec94004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:18.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.261+0000 7fec9a000700 1 -- 172.21.15.1:0/1187496151 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec9400a760 con 0x7feca40fdf40 2024-02-28T17:01:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.261+0000 7fec9a000700 1 -- 172.21.15.1:0/1187496151 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec9400ad90 con 0x7feca40fdf40 2024-02-28T17:01:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.261+0000 7fec9a000700 1 -- 172.21.15.1:0/1187496151 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec940113a0 con 0x7feca40fdf40 2024-02-28T17:01:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.261+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca410fde0 con 0x7feca40fdf40 2024-02-28T17:01:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.261+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca4110320 con 0x7feca40fdf40 2024-02-28T17:01:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.262+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca4106930 con 0x7feca40fdf40 2024-02-28T17:01:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.263+0000 7fec9a000700 1 -- 172.21.15.1:0/1187496151 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fec94012400 con 0x7feca40fdf40 2024-02-28T17:01:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.264+0000 7fec9a000700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec8806d7f0 0x7fec8806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.264+0000 7fec9a000700 1 -- 172.21.15.1:0/1187496151 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fec9408ce40 con 0x7feca40fdf40 2024-02-28T17:01:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.265+0000 7feca1400700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec8806d7f0 0x7fec8806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:18.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.265+0000 7feca1400700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec8806d7f0 0x7fec8806fca0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fec90000c00 tx=0x7fec90009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:18.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.270+0000 7fec9a000700 1 -- 172.21.15.1:0/1187496151 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec940567b0 con 0x7feca40fdf40 2024-02-28T17:01:18.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:18 smithi066 ceph-mon[85781]: pgmap v806: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.510+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feca40fb5a0 con 0x7fec8806d7f0 2024-02-28T17:01:18.510 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:18 smithi001 ceph-mon[112940]: pgmap v806: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:18.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.513+0000 7fec9a000700 1 -- 172.21.15.1:0/1187496151 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7feca40fb5a0 con 0x7fec8806d7f0 2024-02-28T17:01:18.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec8806d7f0 msgr2=0x7fec8806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec8806d7f0 0x7fec8806fca0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fec90000c00 tx=0x7fec90009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:18.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fdf40 msgr2=0x7feca4115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fdf40 0x7feca4115c20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fec9400c830 tx=0x7fec94004470 comp rx=0 tx=0).stop 2024-02-28T17:01:18.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 shutdown_connections 2024-02-28T17:01:18.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec8806d7f0 0x7fec8806fca0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feca40fdf40 0x7feca4115c20 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 --2- 172.21.15.1:0/1187496151 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feca40fe8e0 0x7feca4116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 >> 172.21.15.1:0/1187496151 conn(0x7feca40f79d0 msgr2=0x7feca4103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:18.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 shutdown_connections 2024-02-28T17:01:18.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.516+0000 7feca8e00700 1 -- 172.21.15.1:0/1187496151 wait complete. 2024-02-28T17:01:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.661+0000 7f1b0ee00700 1 -- 172.21.15.1:0/471369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105170 msgr2=0x7f1b08105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.661+0000 7f1b0ee00700 1 --2- 172.21.15.1:0/471369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105170 0x7f1b08105540 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1af8004940 tx=0x7f1af8009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.661+0000 7f1b0ee00700 1 -- 172.21.15.1:0/471369052 shutdown_connections 2024-02-28T17:01:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.661+0000 7f1b0ee00700 1 --2- 172.21.15.1:0/471369052 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b08105b10 0x7f1b080ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.661+0000 7f1b0ee00700 1 --2- 172.21.15.1:0/471369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105170 0x7f1b08105540 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.661+0000 7f1b0ee00700 1 -- 172.21.15.1:0/471369052 >> 172.21.15.1:0/471369052 conn(0x7f1b080face0 msgr2=0x7f1b080fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.661+0000 7f1b0ee00700 1 -- 172.21.15.1:0/471369052 shutdown_connections 2024-02-28T17:01:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.661+0000 7f1b0ee00700 1 -- 172.21.15.1:0/471369052 wait complete. 2024-02-28T17:01:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b0ee00700 1 Processor -- start 2024-02-28T17:01:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b0ee00700 1 -- start start 2024-02-28T17:01:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b0ee00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b08105170 0x7f1b08115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b0ee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105b10 0x7f1b081161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b0ee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b081167f0 con 0x7f1b08105b10 2024-02-28T17:01:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b0ee00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0810fb40 con 0x7f1b08105170 2024-02-28T17:01:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b07400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105b10 0x7f1b081161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b07400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105b10 0x7f1b081161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45946/0 (socket says 172.21.15.1:45946) 2024-02-28T17:01:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.662+0000 7f1b07400700 1 -- 172.21.15.1:0/3931511841 learned_addr learned my addr 172.21.15.1:0/3931511841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1b07e00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b08105170 0x7f1b08115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1b07400700 1 -- 172.21.15.1:0/3931511841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b08105170 msgr2=0x7f1b08115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1b07400700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b08105170 0x7f1b08115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1b07400700 1 -- 172.21.15.1:0/3931511841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af8005040 con 0x7f1b08105b10 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1b07400700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105b10 0x7f1b081161a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1af400c800 tx=0x7f1af4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1affe00700 1 -- 172.21.15.1:0/3931511841 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1af400a760 con 0x7f1b08105b10 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1affe00700 1 -- 172.21.15.1:0/3931511841 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1af400ad90 con 0x7f1b08105b10 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0810fe20 con 0x7f1b08105b10 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.663+0000 7f1affe00700 1 -- 172.21.15.1:0/3931511841 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1af40113a0 con 0x7f1b08105b10 2024-02-28T17:01:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.664+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b08110360 con 0x7f1b08105b10 2024-02-28T17:01:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.665+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b08102480 con 0x7f1b08105b10 2024-02-28T17:01:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.666+0000 7f1affe00700 1 -- 172.21.15.1:0/3931511841 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1af4012400 con 0x7f1b08105b10 2024-02-28T17:01:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.667+0000 7f1affe00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1aec06d7f0 0x7f1aec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.667+0000 7f1affe00700 1 -- 172.21.15.1:0/3931511841 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1af408cdc0 con 0x7f1b08105b10 2024-02-28T17:01:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.667+0000 7f1b07e00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1aec06d7f0 0x7f1aec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.668+0000 7f1b07e00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1aec06d7f0 0x7f1aec06fca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1af800b020 tx=0x7f1af8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:18.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.672+0000 7f1affe00700 1 -- 172.21.15.1:0/3931511841 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1af4056730 con 0x7f1b08105b10 2024-02-28T17:01:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.906+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b08111280 con 0x7f1aec06d7f0 2024-02-28T17:01:18.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.917+0000 7f1affe00700 1 -- 172.21.15.1:0/3931511841 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1b08111280 con 0x7f1aec06d7f0 2024-02-28T17:01:18.917 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:01:18.917 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (27m) 4m ago 41m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:01:18.917 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (24m) 4m ago 42m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (24m) 4m ago 40m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (27m) 4m ago 41m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (20m) 4m ago 34m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (19m) 4m ago 34m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (34m) 4m ago 34m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (4m) 4m ago 34m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (29m) 4m ago 43m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (28m) 4m ago 40m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (25m) 4m ago 43m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (25m) 4m ago 40m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (28m) 4m ago 41m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (28m) 4m ago 40m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (23m) 4m ago 39m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (22m) 4m ago 38m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:01:18.918 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (22m) 4m ago 38m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:01:18.919 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (21m) 4m ago 37m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:01:18.919 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (21m) 4m ago 37m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:01:18.919 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (20m) 4m ago 36m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:01:18.919 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (27m) 4m ago 41m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:01:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.919+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1aec06d7f0 msgr2=0x7f1aec06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.919+0000 7f1b0ee00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1aec06d7f0 0x7f1aec06fca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1af800b020 tx=0x7f1af8009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.919+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105b10 msgr2=0x7f1b081161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.919+0000 7f1b0ee00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105b10 0x7f1b081161a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1af400c800 tx=0x7f1af4004470 comp rx=0 tx=0).stop 2024-02-28T17:01:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.920+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 shutdown_connections 2024-02-28T17:01:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.920+0000 7f1b0ee00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1aec06d7f0 0x7f1aec06fca0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.920+0000 7f1b0ee00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1b08105170 0x7f1b08115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.920+0000 7f1b0ee00700 1 --2- 172.21.15.1:0/3931511841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b08105b10 0x7f1b081161a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:18.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.920+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 >> 172.21.15.1:0/3931511841 conn(0x7f1b080face0 msgr2=0x7f1b080fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:18.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.920+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 shutdown_connections 2024-02-28T17:01:18.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:18.920+0000 7f1b0ee00700 1 -- 172.21.15.1:0/3931511841 wait complete. 2024-02-28T17:01:19.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.076+0000 7fdd99e00700 1 -- 172.21.15.1:0/3794651510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 msgr2=0x7fdd94105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.076+0000 7fdd99e00700 1 --2- 172.21.15.1:0/3794651510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 0x7fdd94105530 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fdd84004940 tx=0x7fdd84009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:19.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.076+0000 7fdd99e00700 1 -- 172.21.15.1:0/3794651510 shutdown_connections 2024-02-28T17:01:19.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.076+0000 7fdd99e00700 1 --2- 172.21.15.1:0/3794651510 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd94105b00 0x7fdd940ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.076+0000 7fdd99e00700 1 --2- 172.21.15.1:0/3794651510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 0x7fdd94105530 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.076+0000 7fdd99e00700 1 -- 172.21.15.1:0/3794651510 >> 172.21.15.1:0/3794651510 conn(0x7fdd940fad30 msgr2=0x7fdd940fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:19.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.076+0000 7fdd99e00700 1 -- 172.21.15.1:0/3794651510 shutdown_connections 2024-02-28T17:01:19.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.076+0000 7fdd99e00700 1 -- 172.21.15.1:0/3794651510 wait complete. 2024-02-28T17:01:19.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.077+0000 7fdd99e00700 1 Processor -- start 2024-02-28T17:01:19.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.077+0000 7fdd99e00700 1 -- start start 2024-02-28T17:01:19.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd99e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 0x7fdd94115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd99e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd94105b00 0x7fdd941161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd99e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd941167f0 con 0x7fdd94105160 2024-02-28T17:01:19.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd99e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9410fb40 con 0x7fdd94105b00 2024-02-28T17:01:19.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd92a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 0x7fdd94115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd92a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 0x7fdd94115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45956/0 (socket says 172.21.15.1:45956) 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd92a00700 1 -- 172.21.15.1:0/1164327197 learned_addr learned my addr 172.21.15.1:0/1164327197 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd92a00700 1 -- 172.21.15.1:0/1164327197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd94105b00 msgr2=0x7fdd941161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.078+0000 7fdd8be00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd94105b00 0x7fdd941161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd92a00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd94105b00 0x7fdd941161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd92a00700 1 -- 172.21.15.1:0/1164327197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84005040 con 0x7fdd94105160 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd8be00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd94105b00 0x7fdd941161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd92a00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 0x7fdd94115c60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009150 tx=0x7fdd84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd8aa00700 1 -- 172.21.15.1:0/1164327197 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd8400e040 con 0x7fdd94105160 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd8aa00700 1 -- 172.21.15.1:0/1164327197 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd8400bbf0 con 0x7fdd94105160 2024-02-28T17:01:19.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9410fdc0 con 0x7fdd94105160 2024-02-28T17:01:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd8aa00700 1 -- 172.21.15.1:0/1164327197 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd8401c630 con 0x7fdd94105160 2024-02-28T17:01:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.079+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd94110300 con 0x7fdd94105160 2024-02-28T17:01:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.081+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd94102470 con 0x7fdd94105160 2024-02-28T17:01:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.082+0000 7fdd8aa00700 1 -- 172.21.15.1:0/1164327197 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdd8400bd60 con 0x7fdd94105160 2024-02-28T17:01:19.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.083+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd7806d7f0 0x7fdd7806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.083+0000 7fdd8aa00700 1 -- 172.21.15.1:0/1164327197 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdd8408e360 con 0x7fdd94105160 2024-02-28T17:01:19.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.083+0000 7fdd8be00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd7806d7f0 0x7fdd7806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.084+0000 7fdd8be00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd7806d7f0 0x7fdd7806fca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c007fd0 tx=0x7fdd7c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:19.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.088+0000 7fdd8aa00700 1 -- 172.21.15.1:0/1164327197 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdd84057b90 con 0x7fdd94105160 2024-02-28T17:01:19.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:19 smithi001 ceph-mon[112940]: from='client.35458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:19.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:19 smithi001 ceph-mon[112940]: from='client.35462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.402+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd9404bf00 con 0x7fdd94105160 2024-02-28T17:01:19.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.403+0000 7fdd8aa00700 1 -- 172.21.15.1:0/1164327197 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fdd8405b250 con 0x7fdd94105160 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:01:19.404 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:01:19.405 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:01:19.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:01:19.405 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:01:19.405 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:01:19.405 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:01:19.405 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:01:19.405 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:01:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd7806d7f0 msgr2=0x7fdd7806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd7806d7f0 0x7fdd7806fca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c007fd0 tx=0x7fdd7c009040 comp rx=0 tx=0).stop 2024-02-28T17:01:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 msgr2=0x7fdd94115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 0x7fdd94115c60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009150 tx=0x7fdd84009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 shutdown_connections 2024-02-28T17:01:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd7806d7f0 0x7fdd7806fca0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd94105160 0x7fdd94115c60 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 --2- 172.21.15.1:0/1164327197 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd94105b00 0x7fdd941161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 >> 172.21.15.1:0/1164327197 conn(0x7fdd940fad30 msgr2=0x7fdd940fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 shutdown_connections 2024-02-28T17:01:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.406+0000 7fdd99e00700 1 -- 172.21.15.1:0/1164327197 wait complete. 2024-02-28T17:01:19.508 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:19 smithi066 ceph-mon[85781]: from='client.35458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:19.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:19 smithi066 ceph-mon[85781]: from='client.35462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.557+0000 7f8bca000700 1 -- 172.21.15.1:0/1616338393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 msgr2=0x7f8bc4105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.557+0000 7f8bca000700 1 --2- 172.21.15.1:0/1616338393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 0x7f8bc4105540 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4004940 tx=0x7f8bb4009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.558+0000 7f8bca000700 1 -- 172.21.15.1:0/1616338393 shutdown_connections 2024-02-28T17:01:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.558+0000 7f8bca000700 1 --2- 172.21.15.1:0/1616338393 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc4105b10 0x7f8bc40ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.558+0000 7f8bca000700 1 --2- 172.21.15.1:0/1616338393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 0x7f8bc4105540 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.558+0000 7f8bca000700 1 -- 172.21.15.1:0/1616338393 >> 172.21.15.1:0/1616338393 conn(0x7f8bc40face0 msgr2=0x7f8bc40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.558+0000 7f8bca000700 1 -- 172.21.15.1:0/1616338393 shutdown_connections 2024-02-28T17:01:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.558+0000 7f8bca000700 1 -- 172.21.15.1:0/1616338393 wait complete. 2024-02-28T17:01:19.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.559+0000 7f8bca000700 1 Processor -- start 2024-02-28T17:01:19.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.559+0000 7f8bca000700 1 -- start start 2024-02-28T17:01:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.560+0000 7f8bca000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 0x7f8bc4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.560+0000 7f8bca000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc4105b10 0x7f8bc41161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.560+0000 7f8bca000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc41167f0 con 0x7f8bc4105170 2024-02-28T17:01:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.560+0000 7f8bca000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc410fb40 con 0x7f8bc4105b10 2024-02-28T17:01:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.560+0000 7f8bc3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 0x7f8bc4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.560+0000 7f8bc3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 0x7f8bc4115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45968/0 (socket says 172.21.15.1:45968) 2024-02-28T17:01:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.560+0000 7f8bc3400700 1 -- 172.21.15.1:0/1900281241 learned_addr learned my addr 172.21.15.1:0/1900281241 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:19.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.560+0000 7f8bc3400700 1 -- 172.21.15.1:0/1900281241 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc4105b10 msgr2=0x7f8bc41161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bc2a00700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc4105b10 0x7f8bc41161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bc3400700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc4105b10 0x7f8bc41161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bc3400700 1 -- 172.21.15.1:0/1900281241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb4005040 con 0x7f8bc4105170 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bc3400700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 0x7f8bc4115c60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009fd0 tx=0x7f8bb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bbb400700 1 -- 172.21.15.1:0/1900281241 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb400e040 con 0x7f8bc4105170 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bbb400700 1 -- 172.21.15.1:0/1900281241 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bb400bbf0 con 0x7f8bc4105170 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bbb400700 1 -- 172.21.15.1:0/1900281241 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb401c3f0 con 0x7f8bc4105170 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc410fdc0 con 0x7f8bc4105170 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.561+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc4110300 con 0x7f8bc4105170 2024-02-28T17:01:19.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.563+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc4102480 con 0x7f8bc4105170 2024-02-28T17:01:19.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.564+0000 7f8bbb400700 1 -- 172.21.15.1:0/1900281241 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8bb400bd60 con 0x7f8bc4105170 2024-02-28T17:01:19.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.565+0000 7f8bbb400700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ba806d7f0 0x7f8ba806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.565+0000 7f8bbb400700 1 -- 172.21.15.1:0/1900281241 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8bb408e1c0 con 0x7f8bc4105170 2024-02-28T17:01:19.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.565+0000 7f8bc2a00700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ba806d7f0 0x7f8ba806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.566+0000 7f8bc2a00700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ba806d7f0 0x7f8ba806fca0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4111540 tx=0x7f8bac00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:19.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.569+0000 7f8bbb400700 1 -- 172.21.15.1:0/1900281241 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8bb4057a70 con 0x7f8bc4105170 2024-02-28T17:01:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.828+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8bc404bf00 con 0x7f8bc4105170 2024-02-28T17:01:19.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.829+0000 7f8bbb400700 1 -- 172.21.15.1:0/1900281241 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f8bb405b130 con 0x7f8bc4105170 2024-02-28T17:01:19.829 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:01:19.830 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:01:19.831 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:01:19.832 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:01:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.831+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ba806d7f0 msgr2=0x7f8ba806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.831+0000 7f8bca000700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ba806d7f0 0x7f8ba806fca0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4111540 tx=0x7f8bac00b040 comp rx=0 tx=0).stop 2024-02-28T17:01:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 msgr2=0x7f8bc4115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 0x7f8bc4115c60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009fd0 tx=0x7f8bb4009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 shutdown_connections 2024-02-28T17:01:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8ba806d7f0 0x7f8ba806fca0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc4105170 0x7f8bc4115c60 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 --2- 172.21.15.1:0/1900281241 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc4105b10 0x7f8bc41161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 >> 172.21.15.1:0/1900281241 conn(0x7f8bc40face0 msgr2=0x7f8bc40fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 shutdown_connections 2024-02-28T17:01:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.832+0000 7f8bca000700 1 -- 172.21.15.1:0/1900281241 wait complete. 2024-02-28T17:01:19.834 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:01:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.984+0000 7f68a9800700 1 -- 172.21.15.1:0/1980871122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 msgr2=0x7f68a4106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.984+0000 7f68a9800700 1 --2- 172.21.15.1:0/1980871122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 0x7f68a4106f40 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6890004940 tx=0x7f6890009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.984+0000 7f68a9800700 1 -- 172.21.15.1:0/1980871122 shutdown_connections 2024-02-28T17:01:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.984+0000 7f68a9800700 1 --2- 172.21.15.1:0/1980871122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 0x7f68a4106f40 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.984+0000 7f68a9800700 1 --2- 172.21.15.1:0/1980871122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68a40fdf60 0x7f68a40fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.984+0000 7f68a9800700 1 -- 172.21.15.1:0/1980871122 >> 172.21.15.1:0/1980871122 conn(0x7f68a40f7a10 msgr2=0x7f68a40f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.984+0000 7f68a9800700 1 -- 172.21.15.1:0/1980871122 shutdown_connections 2024-02-28T17:01:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.984+0000 7f68a9800700 1 -- 172.21.15.1:0/1980871122 wait complete. 2024-02-28T17:01:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.985+0000 7f68a9800700 1 Processor -- start 2024-02-28T17:01:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.985+0000 7f68a9800700 1 -- start start 2024-02-28T17:01:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.985+0000 7f68a9800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68a40fdf60 0x7f68a4115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.985+0000 7f68a9800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 0x7f68a4116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.985+0000 7f68a9800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a41168c0 con 0x7f68a40fe900 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.985+0000 7f68a9800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a410fbd0 con 0x7f68a40fdf60 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.986+0000 7f68a2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 0x7f68a4116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.986+0000 7f68a2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 0x7f68a4116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45992/0 (socket says 172.21.15.1:45992) 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.986+0000 7f68a2000700 1 -- 172.21.15.1:0/2263422915 learned_addr learned my addr 172.21.15.1:0/2263422915 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.986+0000 7f68a2a00700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68a40fdf60 0x7f68a4115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.986+0000 7f68a2000700 1 -- 172.21.15.1:0/2263422915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68a40fdf60 msgr2=0x7f68a4115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.986+0000 7f68a2000700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68a40fdf60 0x7f68a4115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.986+0000 7f68a2000700 1 -- 172.21.15.1:0/2263422915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6890005040 con 0x7f68a40fe900 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.986+0000 7f68a2a00700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68a40fdf60 0x7f68a4115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:01:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.987+0000 7f68a2000700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 0x7f68a4116270 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6890000c00 tx=0x7f6890009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.987+0000 7f689aa00700 1 -- 172.21.15.1:0/2263422915 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689000e040 con 0x7f68a40fe900 2024-02-28T17:01:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.987+0000 7f689aa00700 1 -- 172.21.15.1:0/2263422915 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6890009af0 con 0x7f68a40fe900 2024-02-28T17:01:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.987+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a410fe50 con 0x7f68a40fe900 2024-02-28T17:01:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.987+0000 7f689aa00700 1 -- 172.21.15.1:0/2263422915 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689001c630 con 0x7f68a40fe900 2024-02-28T17:01:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.987+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a4110390 con 0x7f68a40fe900 2024-02-28T17:01:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.988+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a4106950 con 0x7f68a40fe900 2024-02-28T17:01:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.989+0000 7f689aa00700 1 -- 172.21.15.1:0/2263422915 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f689000bbf0 con 0x7f68a40fe900 2024-02-28T17:01:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.990+0000 7f689aa00700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f688806d7e0 0x7f688806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.990+0000 7f689aa00700 1 -- 172.21.15.1:0/2263422915 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f689008e270 con 0x7f68a40fe900 2024-02-28T17:01:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.990+0000 7f68a2a00700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f688806d7e0 0x7f688806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.991+0000 7f68a2a00700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f688806d7e0 0x7f688806fc90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6894009fd0 tx=0x7f689400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:19.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:19.995+0000 7f689aa00700 1 -- 172.21.15.1:0/2263422915 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f689005b650 con 0x7f68a40fe900 2024-02-28T17:01:20.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.239+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f68a404bed0 con 0x7f688806d7e0 2024-02-28T17:01:20.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.241+0000 7f689aa00700 1 -- 172.21.15.1:0/2263422915 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f68a404bed0 con 0x7f688806d7e0 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:01:20.242 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:01:20.243 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:01:20.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.244+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f688806d7e0 msgr2=0x7f688806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:20.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.244+0000 7f68a9800700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f688806d7e0 0x7f688806fc90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6894009fd0 tx=0x7f689400b040 comp rx=0 tx=0).stop 2024-02-28T17:01:20.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.244+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 msgr2=0x7f68a4116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:20.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.244+0000 7f68a9800700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 0x7f68a4116270 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6890000c00 tx=0x7f6890009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:20.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.245+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 shutdown_connections 2024-02-28T17:01:20.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.245+0000 7f68a9800700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f688806d7e0 0x7f688806fc90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.245+0000 7f68a9800700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f68a40fdf60 0x7f68a4115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.245+0000 7f68a9800700 1 --2- 172.21.15.1:0/2263422915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a40fe900 0x7f68a4116270 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.245+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 >> 172.21.15.1:0/2263422915 conn(0x7f68a40f7a10 msgr2=0x7f68a4103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:20.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.245+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 shutdown_connections 2024-02-28T17:01:20.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.245+0000 7f68a9800700 1 -- 172.21.15.1:0/2263422915 wait complete. 2024-02-28T17:01:20.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.400+0000 7fae65c00700 1 -- 172.21.15.1:0/2640086763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 msgr2=0x7fae60103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:20.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.400+0000 7fae65c00700 1 --2- 172.21.15.1:0/2640086763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 0x7fae60103320 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fae50004940 tx=0x7fae50009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:20.401 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:20 smithi001 ceph-mon[112940]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:20.402 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:20 smithi001 ceph-mon[112940]: pgmap v807: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:20.402 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:20 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1164327197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:01:20.402 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:20 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1900281241' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:01:20.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.401+0000 7fae65c00700 1 -- 172.21.15.1:0/2640086763 shutdown_connections 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.401+0000 7fae65c00700 1 --2- 172.21.15.1:0/2640086763 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae601038f0 0x7fae600fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.401+0000 7fae65c00700 1 --2- 172.21.15.1:0/2640086763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 0x7fae60103320 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.401+0000 7fae65c00700 1 -- 172.21.15.1:0/2640086763 >> 172.21.15.1:0/2640086763 conn(0x7fae600f8ae0 msgr2=0x7fae600faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.401+0000 7fae65c00700 1 -- 172.21.15.1:0/2640086763 shutdown_connections 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.401+0000 7fae65c00700 1 -- 172.21.15.1:0/2640086763 wait complete. 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.402+0000 7fae65c00700 1 Processor -- start 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.402+0000 7fae65c00700 1 -- start start 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.402+0000 7fae65c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 0x7fae60115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.402+0000 7fae65c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae601038f0 0x7fae601161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.402+0000 7fae65c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae60116800 con 0x7fae60102f50 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.402+0000 7fae65c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6010fb30 con 0x7fae601038f0 2024-02-28T17:01:20.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.403+0000 7fae5ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 0x7fae60115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.403+0000 7fae5ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 0x7fae60115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46014/0 (socket says 172.21.15.1:46014) 2024-02-28T17:01:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.403+0000 7fae5ea00700 1 -- 172.21.15.1:0/1338750133 learned_addr learned my addr 172.21.15.1:0/1338750133 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.403+0000 7fae57e00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae601038f0 0x7fae601161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.403+0000 7fae5ea00700 1 -- 172.21.15.1:0/1338750133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae601038f0 msgr2=0x7fae601161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.403+0000 7fae5ea00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae601038f0 0x7fae601161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.403+0000 7fae5ea00700 1 -- 172.21.15.1:0/1338750133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae50005040 con 0x7fae60102f50 2024-02-28T17:01:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.404+0000 7fae5ea00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 0x7fae60115c70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fae50009fd0 tx=0x7fae50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.404+0000 7fae56a00700 1 -- 172.21.15.1:0/1338750133 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae5000e040 con 0x7fae60102f50 2024-02-28T17:01:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.404+0000 7fae56a00700 1 -- 172.21.15.1:0/1338750133 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae5000bbf0 con 0x7fae60102f50 2024-02-28T17:01:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.404+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6010fdb0 con 0x7fae60102f50 2024-02-28T17:01:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.404+0000 7fae56a00700 1 -- 172.21.15.1:0/1338750133 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae5001c3f0 con 0x7fae60102f50 2024-02-28T17:01:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.404+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae601102f0 con 0x7fae60102f50 2024-02-28T17:01:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.405+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae600fc6e0 con 0x7fae60102f50 2024-02-28T17:01:20.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.407+0000 7fae56a00700 1 -- 172.21.15.1:0/1338750133 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fae5000bd60 con 0x7fae60102f50 2024-02-28T17:01:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.408+0000 7fae56a00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae4406d7f0 0x7fae4406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.408+0000 7fae56a00700 1 -- 172.21.15.1:0/1338750133 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fae5008e2b0 con 0x7fae60102f50 2024-02-28T17:01:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.408+0000 7fae57e00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae4406d7f0 0x7fae4406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.409+0000 7fae57e00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae4406d7f0 0x7fae4406fca0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fae601114a0 tx=0x7fae48009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.412+0000 7fae56a00700 1 -- 172.21.15.1:0/1338750133 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae5005b610 con 0x7fae60102f50 2024-02-28T17:01:20.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:20 smithi066 ceph-mon[85781]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:20.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:20 smithi066 ceph-mon[85781]: pgmap v807: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:20.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:20 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1164327197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:01:20.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:20 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1900281241' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:01:20.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.709+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fae60111280 con 0x7fae60102f50 2024-02-28T17:01:20.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.710+0000 7fae56a00700 1 -- 172.21.15.1:0/1338750133 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fae5005b1a0 con 0x7fae60102f50 2024-02-28T17:01:20.711 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:01:20.711 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:01:20.711 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:01:20.711 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:01:20.711 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:01:20.711 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:01:20.711 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:01:20.712 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:01:20.713 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:01:20.713 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:01:20.713 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:01:20.713 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:01:20.713 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:01:20.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae4406d7f0 msgr2=0x7fae4406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae4406d7f0 0x7fae4406fca0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fae601114a0 tx=0x7fae48009ee0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 msgr2=0x7fae60115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 0x7fae60115c70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fae50009fd0 tx=0x7fae50009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 shutdown_connections 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae4406d7f0 0x7fae4406fca0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae60102f50 0x7fae60115c70 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 --2- 172.21.15.1:0/1338750133 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae601038f0 0x7fae601161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 >> 172.21.15.1:0/1338750133 conn(0x7fae600f8ae0 msgr2=0x7fae60070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 shutdown_connections 2024-02-28T17:01:20.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:20.713+0000 7fae65c00700 1 -- 172.21.15.1:0/1338750133 wait complete. 2024-02-28T17:01:21.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:21 smithi066 ceph-mon[85781]: from='client.35478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:21.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:21 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1338750133' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:01:21.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:21 smithi001 ceph-mon[112940]: from='client.35478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:21.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:21 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1338750133' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:01:22.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:22 smithi066 ceph-mon[85781]: pgmap v808: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:22.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:22 smithi001 ceph-mon[112940]: pgmap v808: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:24.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:24 smithi066 ceph-mon[85781]: pgmap v809: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:24.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:24 smithi001 ceph-mon[112940]: pgmap v809: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:26.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:26 smithi066 ceph-mon[85781]: pgmap v810: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:26.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:01:26.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:26 smithi001 ceph-mon[112940]: pgmap v810: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:26.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:01:28.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:28 smithi066 ceph-mon[85781]: pgmap v811: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:28.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:28 smithi001 ceph-mon[112940]: pgmap v811: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:30.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:30 smithi066 ceph-mon[85781]: pgmap v812: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:30.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:30 smithi001 ceph-mon[112940]: pgmap v812: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:32.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:32 smithi066 ceph-mon[85781]: pgmap v813: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:32.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:32 smithi001 ceph-mon[112940]: pgmap v813: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:34.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:34 smithi066 ceph-mon[85781]: pgmap v814: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:34.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:34 smithi001 ceph-mon[112940]: pgmap v814: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:36.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:36 smithi066 ceph-mon[85781]: pgmap v815: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:36.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:36 smithi001 ceph-mon[112940]: pgmap v815: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:38.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:38 smithi066 ceph-mon[85781]: pgmap v816: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:38.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:38 smithi001 ceph-mon[112940]: pgmap v816: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:40.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:40 smithi066 ceph-mon[85781]: pgmap v817: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:40.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:40 smithi001 ceph-mon[112940]: pgmap v817: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:41.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:01:41.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:01:42.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:42 smithi066 ceph-mon[85781]: pgmap v818: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:42.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:42 smithi001 ceph-mon[112940]: pgmap v818: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:44.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:44 smithi066 ceph-mon[85781]: pgmap v819: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:44.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:44 smithi001 ceph-mon[112940]: pgmap v819: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:46.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:46 smithi066 ceph-mon[85781]: pgmap v820: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:46.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:46 smithi001 ceph-mon[112940]: pgmap v820: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:48.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:48 smithi066 ceph-mon[85781]: pgmap v821: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:48.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:48 smithi001 ceph-mon[112940]: pgmap v821: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:50.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:50 smithi066 ceph-mon[85781]: pgmap v822: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:50.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:50 smithi001 ceph-mon[112940]: pgmap v822: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.882+0000 7ff850e00700 1 -- 172.21.15.1:0/3072544218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 msgr2=0x7ff84c104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.882+0000 7ff850e00700 1 --2- 172.21.15.1:0/3072544218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 0x7ff84c104230 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff838004940 tx=0x7ff838009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.882+0000 7ff850e00700 1 -- 172.21.15.1:0/3072544218 shutdown_connections 2024-02-28T17:01:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.882+0000 7ff850e00700 1 --2- 172.21.15.1:0/3072544218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 0x7ff84c104230 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.882+0000 7ff850e00700 1 --2- 172.21.15.1:0/3072544218 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff84c1095b0 0x7ff84c109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.882+0000 7ff850e00700 1 -- 172.21.15.1:0/3072544218 >> 172.21.15.1:0/3072544218 conn(0x7ff84c072d70 msgr2=0x7ff84c075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.882+0000 7ff850e00700 1 -- 172.21.15.1:0/3072544218 shutdown_connections 2024-02-28T17:01:50.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.882+0000 7ff850e00700 1 -- 172.21.15.1:0/3072544218 wait complete. 2024-02-28T17:01:50.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.883+0000 7ff850e00700 1 Processor -- start 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.883+0000 7ff850e00700 1 -- start start 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.884+0000 7ff850e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 0x7ff84c11a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.884+0000 7ff850e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff84c1095b0 0x7ff84c11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.884+0000 7ff850e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c11ac00 con 0x7ff84c103dc0 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.884+0000 7ff850e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c113f10 con 0x7ff84c1095b0 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.884+0000 7ff849e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 0x7ff84c11a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.884+0000 7ff849e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 0x7ff84c11a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42742/0 (socket says 172.21.15.1:42742) 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.884+0000 7ff849e00700 1 -- 172.21.15.1:0/1545167903 learned_addr learned my addr 172.21.15.1:0/1545167903 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:50.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.885+0000 7ff849400700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff84c1095b0 0x7ff84c11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.885+0000 7ff849e00700 1 -- 172.21.15.1:0/1545167903 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff84c1095b0 msgr2=0x7ff84c11a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.885+0000 7ff849e00700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff84c1095b0 0x7ff84c11a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.885+0000 7ff849e00700 1 -- 172.21.15.1:0/1545167903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff838005040 con 0x7ff84c103dc0 2024-02-28T17:01:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.885+0000 7ff849e00700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 0x7ff84c11a070 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00c8b0 tx=0x7ff83c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.886+0000 7ff842000700 1 -- 172.21.15.1:0/1545167903 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff83c00a760 con 0x7ff84c103dc0 2024-02-28T17:01:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.886+0000 7ff842000700 1 -- 172.21.15.1:0/1545167903 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff83c00ad90 con 0x7ff84c103dc0 2024-02-28T17:01:50.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.886+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c1141f0 con 0x7ff84c103dc0 2024-02-28T17:01:50.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.886+0000 7ff842000700 1 -- 172.21.15.1:0/1545167903 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff83c0113a0 con 0x7ff84c103dc0 2024-02-28T17:01:50.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.886+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c114730 con 0x7ff84c103dc0 2024-02-28T17:01:50.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.887+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c10b990 con 0x7ff84c103dc0 2024-02-28T17:01:50.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.888+0000 7ff842000700 1 -- 172.21.15.1:0/1545167903 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff83c012400 con 0x7ff84c103dc0 2024-02-28T17:01:50.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.890+0000 7ff842000700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff83006d7f0 0x7ff83006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:50.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.890+0000 7ff842000700 1 -- 172.21.15.1:0/1545167903 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff83c08ce70 con 0x7ff84c103dc0 2024-02-28T17:01:50.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.890+0000 7ff849400700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff83006d7f0 0x7ff83006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:50.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.891+0000 7ff849400700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff83006d7f0 0x7ff83006fca0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff84c115910 tx=0x7ff838009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:50.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:50.895+0000 7ff842000700 1 -- 172.21.15.1:0/1545167903 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff83c05a250 con 0x7ff84c103dc0 2024-02-28T17:01:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.136+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff84c04bf70 con 0x7ff83006d7f0 2024-02-28T17:01:51.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.138+0000 7ff842000700 1 -- 172.21.15.1:0/1545167903 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff838008e30 con 0x7ff83006d7f0 2024-02-28T17:01:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.140+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff83006d7f0 msgr2=0x7ff83006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.140+0000 7ff850e00700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff83006d7f0 0x7ff83006fca0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff84c115910 tx=0x7ff838009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.140+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 msgr2=0x7ff84c11a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.140+0000 7ff850e00700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 0x7ff84c11a070 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00c8b0 tx=0x7ff83c004470 comp rx=0 tx=0).stop 2024-02-28T17:01:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.141+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 shutdown_connections 2024-02-28T17:01:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.141+0000 7ff850e00700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff83006d7f0 0x7ff83006fca0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.141+0000 7ff850e00700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff84c103dc0 0x7ff84c11a070 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.141+0000 7ff850e00700 1 --2- 172.21.15.1:0/1545167903 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff84c1095b0 0x7ff84c11a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.141+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 >> 172.21.15.1:0/1545167903 conn(0x7ff84c072d70 msgr2=0x7ff84c074960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:51.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.141+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 shutdown_connections 2024-02-28T17:01:51.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.141+0000 7ff850e00700 1 -- 172.21.15.1:0/1545167903 wait complete. 2024-02-28T17:01:51.155 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:01:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.300+0000 7fcea7400700 1 -- 172.21.15.1:0/2688731246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 msgr2=0x7fcea01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.300+0000 7fcea7400700 1 --2- 172.21.15.1:0/2688731246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 0x7fcea01099c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fce94004940 tx=0x7fce94009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.300+0000 7fcea7400700 1 -- 172.21.15.1:0/2688731246 shutdown_connections 2024-02-28T17:01:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.300+0000 7fcea7400700 1 --2- 172.21.15.1:0/2688731246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea0103dc0 0x7fcea0104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.300+0000 7fcea7400700 1 --2- 172.21.15.1:0/2688731246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 0x7fcea01099c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.300+0000 7fcea7400700 1 -- 172.21.15.1:0/2688731246 >> 172.21.15.1:0/2688731246 conn(0x7fcea0072d50 msgr2=0x7fcea0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.300+0000 7fcea7400700 1 -- 172.21.15.1:0/2688731246 shutdown_connections 2024-02-28T17:01:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.301+0000 7fcea7400700 1 -- 172.21.15.1:0/2688731246 wait complete. 2024-02-28T17:01:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.301+0000 7fcea7400700 1 Processor -- start 2024-02-28T17:01:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.301+0000 7fcea7400700 1 -- start start 2024-02-28T17:01:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.302+0000 7fcea7400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea0103dc0 0x7fcea011a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.302+0000 7fcea7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 0x7fcea011a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.302+0000 7fcea7400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea011ac70 con 0x7fcea01095f0 2024-02-28T17:01:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.302+0000 7fcea7400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0113f80 con 0x7fcea0103dc0 2024-02-28T17:01:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.302+0000 7fce9fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 0x7fcea011a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:51.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.302+0000 7fce9fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 0x7fcea011a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42756/0 (socket says 172.21.15.1:42756) 2024-02-28T17:01:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.302+0000 7fce9fe00700 1 -- 172.21.15.1:0/2168239270 learned_addr learned my addr 172.21.15.1:0/2168239270 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fcea4a00700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea0103dc0 0x7fcea011a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fce9fe00700 1 -- 172.21.15.1:0/2168239270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea0103dc0 msgr2=0x7fcea011a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fce9fe00700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea0103dc0 0x7fcea011a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fce9fe00700 1 -- 172.21.15.1:0/2168239270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce94005040 con 0x7fcea01095f0 2024-02-28T17:01:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fcea4a00700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea0103dc0 0x7fcea011a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:01:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fce9fe00700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 0x7fcea011a620 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00c8c0 tx=0x7fce8c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fce9cc00700 1 -- 172.21.15.1:0/2168239270 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8c00a980 con 0x7fcea01095f0 2024-02-28T17:01:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fce9cc00700 1 -- 172.21.15.1:0/2168239270 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce8c00ce60 con 0x7fcea01095f0 2024-02-28T17:01:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea0114260 con 0x7fcea01095f0 2024-02-28T17:01:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fce9cc00700 1 -- 172.21.15.1:0/2168239270 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8c0115e0 con 0x7fcea01095f0 2024-02-28T17:01:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.303+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01147a0 con 0x7fcea01095f0 2024-02-28T17:01:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.305+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea0106850 con 0x7fcea01095f0 2024-02-28T17:01:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.306+0000 7fce9cc00700 1 -- 172.21.15.1:0/2168239270 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fce8c012400 con 0x7fcea01095f0 2024-02-28T17:01:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.307+0000 7fce9cc00700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce8806d7f0 0x7fce8806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.307+0000 7fce9cc00700 1 -- 172.21.15.1:0/2168239270 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fce8c08cf70 con 0x7fcea01095f0 2024-02-28T17:01:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.307+0000 7fcea4a00700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce8806d7f0 0x7fce8806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.308+0000 7fcea4a00700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce8806d7f0 0x7fce8806fca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fce9400b3b0 tx=0x7fce94009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.312+0000 7fce9cc00700 1 -- 172.21.15.1:0/2168239270 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fce8c05a350 con 0x7fcea01095f0 2024-02-28T17:01:51.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.557+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcea004bf40 con 0x7fce8806d7f0 2024-02-28T17:01:51.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.560+0000 7fce9cc00700 1 -- 172.21.15.1:0/2168239270 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcea004bf40 con 0x7fce8806d7f0 2024-02-28T17:01:51.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce8806d7f0 msgr2=0x7fce8806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce8806d7f0 0x7fce8806fca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fce9400b3b0 tx=0x7fce94009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:51.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 msgr2=0x7fcea011a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 0x7fcea011a620 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00c8c0 tx=0x7fce8c0044c0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 shutdown_connections 2024-02-28T17:01:51.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fce8806d7f0 0x7fce8806fca0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcea0103dc0 0x7fcea011a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 --2- 172.21.15.1:0/2168239270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea01095f0 0x7fcea011a620 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.563+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 >> 172.21.15.1:0/2168239270 conn(0x7fcea0072d50 msgr2=0x7fcea0074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:51.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.564+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 shutdown_connections 2024-02-28T17:01:51.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.564+0000 7fcea7400700 1 -- 172.21.15.1:0/2168239270 wait complete. 2024-02-28T17:01:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.722+0000 7f4d2a000700 1 -- 172.21.15.1:0/2418324684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070330 msgr2=0x7f4d24070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.722+0000 7f4d2a000700 1 --2- 172.21.15.1:0/2418324684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070330 0x7f4d24070700 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4d18004970 tx=0x7f4d18009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.722+0000 7f4d2a000700 1 -- 172.21.15.1:0/2418324684 shutdown_connections 2024-02-28T17:01:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.722+0000 7f4d2a000700 1 --2- 172.21.15.1:0/2418324684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d24070c40 0x7f4d24109050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.722+0000 7f4d2a000700 1 --2- 172.21.15.1:0/2418324684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070330 0x7f4d24070700 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.722+0000 7f4d2a000700 1 -- 172.21.15.1:0/2418324684 >> 172.21.15.1:0/2418324684 conn(0x7f4d240f88b0 msgr2=0x7f4d240facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.722+0000 7f4d2a000700 1 -- 172.21.15.1:0/2418324684 shutdown_connections 2024-02-28T17:01:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.722+0000 7f4d2a000700 1 -- 172.21.15.1:0/2418324684 wait complete. 2024-02-28T17:01:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.723+0000 7f4d2a000700 1 Processor -- start 2024-02-28T17:01:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.723+0000 7f4d2a000700 1 -- start start 2024-02-28T17:01:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.724+0000 7f4d2a000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d24070330 0x7f4d241159b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.724+0000 7f4d2a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070c40 0x7f4d24115ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:51.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.724+0000 7f4d2a000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d24116540 con 0x7f4d24070c40 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.724+0000 7f4d2a000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2410f870 con 0x7f4d24070330 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.724+0000 7f4d23e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070c40 0x7f4d24115ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.724+0000 7f4d23e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070c40 0x7f4d24115ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42762/0 (socket says 172.21.15.1:42762) 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.724+0000 7f4d23e00700 1 -- 172.21.15.1:0/1105548789 learned_addr learned my addr 172.21.15.1:0/1105548789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d28c00700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d24070330 0x7f4d241159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d23e00700 1 -- 172.21.15.1:0/1105548789 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d24070330 msgr2=0x7f4d241159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d23e00700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d24070330 0x7f4d241159b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d23e00700 1 -- 172.21.15.1:0/1105548789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d18005040 con 0x7f4d24070c40 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d28c00700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d24070330 0x7f4d241159b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d23e00700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070c40 0x7f4d24115ef0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000c8c0 tx=0x7f4d100044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:51.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d20c00700 1 -- 172.21.15.1:0/1105548789 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1000a980 con 0x7f4d24070c40 2024-02-28T17:01:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2410fb50 con 0x7f4d24070c40 2024-02-28T17:01:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d20c00700 1 -- 172.21.15.1:0/1105548789 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d1000ce60 con 0x7f4d24070c40 2024-02-28T17:01:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d20c00700 1 -- 172.21.15.1:0/1105548789 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d100115e0 con 0x7f4d24070c40 2024-02-28T17:01:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.725+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d24110090 con 0x7f4d24070c40 2024-02-28T17:01:51.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.727+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d24106710 con 0x7f4d24070c40 2024-02-28T17:01:51.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.728+0000 7f4d20c00700 1 -- 172.21.15.1:0/1105548789 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4d10012400 con 0x7f4d24070c40 2024-02-28T17:01:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.730+0000 7f4d20c00700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0c06d7f0 0x7f4d0c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.730+0000 7f4d20c00700 1 -- 172.21.15.1:0/1105548789 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4d1008cfe0 con 0x7f4d24070c40 2024-02-28T17:01:51.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.730+0000 7f4d28c00700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0c06d7f0 0x7f4d0c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:51.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.731+0000 7f4d28c00700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0c06d7f0 0x7f4d0c06fca0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4d18000c00 tx=0x7f4d18009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:51.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.734+0000 7f4d20c00700 1 -- 172.21.15.1:0/1105548789 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4d1005a3a0 con 0x7f4d24070c40 2024-02-28T17:01:51.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.957+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d2410c030 con 0x7f4d0c06d7f0 2024-02-28T17:01:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.969+0000 7f4d20c00700 1 -- 172.21.15.1:0/1105548789 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4d2410c030 con 0x7f4d0c06d7f0 2024-02-28T17:01:51.968 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (27m) 4m ago 42m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (24m) 4m ago 42m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (24m) 4m ago 41m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (27m) 4m ago 41m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (20m) 4m ago 35m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (20m) 4m ago 34m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (34m) 4m ago 34m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (4m) 4m ago 35m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (30m) 4m ago 43m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (29m) 4m ago 40m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (26m) 4m ago 44m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (25m) 4m ago 40m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (29m) 4m ago 42m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:01:51.969 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (28m) 4m ago 41m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:01:51.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (24m) 4m ago 39m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:01:51.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (23m) 4m ago 39m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:01:51.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (23m) 4m ago 38m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:01:51.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (22m) 4m ago 38m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:01:51.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (21m) 4m ago 37m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:01:51.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (21m) 4m ago 36m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:01:51.970 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (28m) 4m ago 41m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:01:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.971+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0c06d7f0 msgr2=0x7f4d0c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.971+0000 7f4d2a000700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0c06d7f0 0x7f4d0c06fca0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4d18000c00 tx=0x7f4d18009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:51.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.971+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070c40 msgr2=0x7f4d24115ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.971+0000 7f4d2a000700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070c40 0x7f4d24115ef0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000c8c0 tx=0x7f4d100044c0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.972+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 shutdown_connections 2024-02-28T17:01:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.972+0000 7f4d2a000700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0c06d7f0 0x7f4d0c06fca0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.972+0000 7f4d2a000700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d24070330 0x7f4d241159b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.972+0000 7f4d2a000700 1 --2- 172.21.15.1:0/1105548789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d24070c40 0x7f4d24115ef0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.972+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 >> 172.21.15.1:0/1105548789 conn(0x7f4d240f88b0 msgr2=0x7f4d24103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.972+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 shutdown_connections 2024-02-28T17:01:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:51.972+0000 7f4d2a000700 1 -- 172.21.15.1:0/1105548789 wait complete. 2024-02-28T17:01:52.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.123+0000 7fd0ea800700 1 -- 172.21.15.1:0/3905471898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105b10 msgr2=0x7fd0e40ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:52.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.123+0000 7fd0ea800700 1 --2- 172.21.15.1:0/3905471898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105b10 0x7fd0e40ff180 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0004940 tx=0x7fd0d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:52.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.123+0000 7fd0ea800700 1 -- 172.21.15.1:0/3905471898 shutdown_connections 2024-02-28T17:01:52.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.123+0000 7fd0ea800700 1 --2- 172.21.15.1:0/3905471898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105b10 0x7fd0e40ff180 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.123+0000 7fd0ea800700 1 --2- 172.21.15.1:0/3905471898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0e4105170 0x7fd0e4105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.123+0000 7fd0ea800700 1 -- 172.21.15.1:0/3905471898 >> 172.21.15.1:0/3905471898 conn(0x7fd0e40face0 msgr2=0x7fd0e40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:52.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.124+0000 7fd0ea800700 1 -- 172.21.15.1:0/3905471898 shutdown_connections 2024-02-28T17:01:52.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.124+0000 7fd0ea800700 1 -- 172.21.15.1:0/3905471898 wait complete. 2024-02-28T17:01:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0ea800700 1 Processor -- start 2024-02-28T17:01:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0ea800700 1 -- start start 2024-02-28T17:01:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0ea800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105170 0x7fd0e4115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0ea800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0e4105b10 0x7fd0e4116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0ea800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4116870 con 0x7fd0e4105170 2024-02-28T17:01:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0ea800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e410fb80 con 0x7fd0e4105b10 2024-02-28T17:01:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105170 0x7fd0e4115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105170 0x7fd0e4115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33998/0 (socket says 172.21.15.1:33998) 2024-02-28T17:01:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.125+0000 7fd0e3400700 1 -- 172.21.15.1:0/1626186693 learned_addr learned my addr 172.21.15.1:0/1626186693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:52.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.126+0000 7fd0e3400700 1 -- 172.21.15.1:0/1626186693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0e4105b10 msgr2=0x7fd0e4116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.126+0000 7fd0e3400700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0e4105b10 0x7fd0e4116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.126+0000 7fd0e3400700 1 -- 172.21.15.1:0/1626186693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d0005040 con 0x7fd0e4105170 2024-02-28T17:01:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.126+0000 7fd0e3400700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105170 0x7fd0e4115ce0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4002a50 tx=0x7fd0d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:52.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.126+0000 7fd0db400700 1 -- 172.21.15.1:0/1626186693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0d4007cc0 con 0x7fd0e4105170 2024-02-28T17:01:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.126+0000 7fd0db400700 1 -- 172.21.15.1:0/1626186693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0d400f040 con 0x7fd0e4105170 2024-02-28T17:01:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.126+0000 7fd0db400700 1 -- 172.21.15.1:0/1626186693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0d4009640 con 0x7fd0e4105170 2024-02-28T17:01:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.126+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e410fe60 con 0x7fd0e4105170 2024-02-28T17:01:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.127+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e41103a0 con 0x7fd0e4105170 2024-02-28T17:01:52.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.128+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e4107640 con 0x7fd0e4105170 2024-02-28T17:01:52.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.129+0000 7fd0db400700 1 -- 172.21.15.1:0/1626186693 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd0d40097a0 con 0x7fd0e4105170 2024-02-28T17:01:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.129+0000 7fd0db400700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd0c806d7f0 0x7fd0c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.130+0000 7fd0db400700 1 -- 172.21.15.1:0/1626186693 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd0d408cc60 con 0x7fd0e4105170 2024-02-28T17:01:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.130+0000 7fd0e2a00700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd0c806d7f0 0x7fd0c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.131+0000 7fd0e2a00700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd0c806d7f0 0x7fd0c806fca0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0004f90 tx=0x7fd0d0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:52.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.134+0000 7fd0db400700 1 -- 172.21.15.1:0/1626186693 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd0d4059fc0 con 0x7fd0e4105170 2024-02-28T17:01:52.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.439+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd0e404bed0 con 0x7fd0e4105170 2024-02-28T17:01:52.552 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:52 smithi066 ceph-mon[85781]: pgmap v823: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:52.552 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:52 smithi066 ceph-mon[85781]: from='client.35486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:52.552 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:52 smithi066 ceph-mon[85781]: from='client.35490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:52.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.440+0000 7fd0db400700 1 -- 172.21.15.1:0/1626186693 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fd0d4059b50 con 0x7fd0e4105170 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:01:52.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:01:52.554 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:01:52.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd0c806d7f0 msgr2=0x7fd0c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:52.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd0c806d7f0 0x7fd0c806fca0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0004f90 tx=0x7fd0d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:52.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105170 msgr2=0x7fd0e4115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:52.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105170 0x7fd0e4115ce0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4002a50 tx=0x7fd0d40046b0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 shutdown_connections 2024-02-28T17:01:52.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd0c806d7f0 0x7fd0c806fca0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e4105170 0x7fd0e4115ce0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 --2- 172.21.15.1:0/1626186693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0e4105b10 0x7fd0e4116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.442+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 >> 172.21.15.1:0/1626186693 conn(0x7fd0e40face0 msgr2=0x7fd0e40fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:52.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.443+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 shutdown_connections 2024-02-28T17:01:52.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.443+0000 7fd0ea800700 1 -- 172.21.15.1:0/1626186693 wait complete. 2024-02-28T17:01:52.556 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:52 smithi001 ceph-mon[112940]: pgmap v823: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:52.557 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:52 smithi001 ceph-mon[112940]: from='client.35486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:52.557 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:52 smithi001 ceph-mon[112940]: from='client.35490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:52.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.586+0000 7fbb78200700 1 -- 172.21.15.1:0/2066921821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fc5b0 msgr2=0x7fbb70109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.586+0000 7fbb78200700 1 --2- 172.21.15.1:0/2066921821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fc5b0 0x7fbb70109310 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c004940 tx=0x7fbb5c009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.586+0000 7fbb78200700 1 -- 172.21.15.1:0/2066921821 shutdown_connections 2024-02-28T17:01:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.586+0000 7fbb78200700 1 --2- 172.21.15.1:0/2066921821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fc5b0 0x7fbb70109310 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.586+0000 7fbb78200700 1 --2- 172.21.15.1:0/2066921821 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb700fbca0 0x7fbb700fc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.586+0000 7fbb78200700 1 -- 172.21.15.1:0/2066921821 >> 172.21.15.1:0/2066921821 conn(0x7fbb700f79d0 msgr2=0x7fbb700f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.586+0000 7fbb78200700 1 -- 172.21.15.1:0/2066921821 shutdown_connections 2024-02-28T17:01:52.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.586+0000 7fbb78200700 1 -- 172.21.15.1:0/2066921821 wait complete. 2024-02-28T17:01:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.587+0000 7fbb78200700 1 Processor -- start 2024-02-28T17:01:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.587+0000 7fbb78200700 1 -- start start 2024-02-28T17:01:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.587+0000 7fbb78200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fbca0 0x7fbb70115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.587+0000 7fbb78200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb700fc5b0 0x7fbb70116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.587+0000 7fbb78200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb70116840 con 0x7fbb700fbca0 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.587+0000 7fbb78200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7010fae0 con 0x7fbb700fc5b0 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.588+0000 7fbb75800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fbca0 0x7fbb70115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.588+0000 7fbb75800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fbca0 0x7fbb70115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34018/0 (socket says 172.21.15.1:34018) 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.588+0000 7fbb75800700 1 -- 172.21.15.1:0/2045685578 learned_addr learned my addr 172.21.15.1:0/2045685578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.588+0000 7fbb75800700 1 -- 172.21.15.1:0/2045685578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb700fc5b0 msgr2=0x7fbb70116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.588+0000 7fbb74e00700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb700fc5b0 0x7fbb70116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.588+0000 7fbb75800700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb700fc5b0 0x7fbb70116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.588+0000 7fbb75800700 1 -- 172.21.15.1:0/2045685578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c005040 con 0x7fbb700fbca0 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.589+0000 7fbb75800700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fbca0 0x7fbb70115c20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400b790 tx=0x7fbb64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:52.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.589+0000 7fbb6d600700 1 -- 172.21.15.1:0/2045685578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb6400f770 con 0x7fbb700fbca0 2024-02-28T17:01:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.589+0000 7fbb6d600700 1 -- 172.21.15.1:0/2045685578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb6400fda0 con 0x7fbb700fbca0 2024-02-28T17:01:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.589+0000 7fbb6d600700 1 -- 172.21.15.1:0/2045685578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb6400d310 con 0x7fbb700fbca0 2024-02-28T17:01:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.589+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7010fdc0 con 0x7fbb700fbca0 2024-02-28T17:01:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.589+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb70110300 con 0x7fbb700fbca0 2024-02-28T17:01:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.590+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb70106a40 con 0x7fbb700fbca0 2024-02-28T17:01:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.591+0000 7fbb6d600700 1 -- 172.21.15.1:0/2045685578 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbb64010420 con 0x7fbb700fbca0 2024-02-28T17:01:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.592+0000 7fbb6d600700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb5806d7f0 0x7fbb5806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.593+0000 7fbb6d600700 1 -- 172.21.15.1:0/2045685578 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbb6408cef0 con 0x7fbb700fbca0 2024-02-28T17:01:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.593+0000 7fbb74e00700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb5806d7f0 0x7fbb5806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.593+0000 7fbb74e00700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb5806d7f0 0x7fbb5806fca0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c000c00 tx=0x7fbb5c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:52.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.597+0000 7fbb6d600700 1 -- 172.21.15.1:0/2045685578 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbb6405a2d0 con 0x7fbb700fbca0 2024-02-28T17:01:52.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.871+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbb70111200 con 0x7fbb700fbca0 2024-02-28T17:01:52.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.872+0000 7fbb6d600700 1 -- 172.21.15.1:0/2045685578 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fbb64059e60 con 0x7fbb700fbca0 2024-02-28T17:01:52.872 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:01:52.872 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:01:52.872 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:01:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:01:52.874 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:01:52.875 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:01:52.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.874+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb5806d7f0 msgr2=0x7fbb5806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:52.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.874+0000 7fbb78200700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb5806d7f0 0x7fbb5806fca0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c000c00 tx=0x7fbb5c009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:52.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fbca0 msgr2=0x7fbb70115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:52.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fbca0 0x7fbb70115c20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400b790 tx=0x7fbb64009250 comp rx=0 tx=0).stop 2024-02-28T17:01:52.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 shutdown_connections 2024-02-28T17:01:52.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbb5806d7f0 0x7fbb5806fca0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb700fbca0 0x7fbb70115c20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 --2- 172.21.15.1:0/2045685578 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb700fc5b0 0x7fbb70116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:52.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 >> 172.21.15.1:0/2045685578 conn(0x7fbb700f79d0 msgr2=0x7fbb700f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:52.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 shutdown_connections 2024-02-28T17:01:52.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:52.875+0000 7fbb78200700 1 -- 172.21.15.1:0/2045685578 wait complete. 2024-02-28T17:01:52.877 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:01:53.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.030+0000 7fa00da00700 1 -- 172.21.15.1:0/4169421344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109d20 msgr2=0x7fa0081033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:53.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.030+0000 7fa00da00700 1 --2- 172.21.15.1:0/4169421344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109d20 0x7fa0081033a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4004940 tx=0x7f9ff4009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:53.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.030+0000 7fa00da00700 1 -- 172.21.15.1:0/4169421344 shutdown_connections 2024-02-28T17:01:53.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.030+0000 7fa00da00700 1 --2- 172.21.15.1:0/4169421344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109d20 0x7fa0081033a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.030+0000 7fa00da00700 1 --2- 172.21.15.1:0/4169421344 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa008109380 0x7fa008109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.030+0000 7fa00da00700 1 -- 172.21.15.1:0/4169421344 >> 172.21.15.1:0/4169421344 conn(0x7fa008072b50 msgr2=0x7fa008074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:53.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.031+0000 7fa00da00700 1 -- 172.21.15.1:0/4169421344 shutdown_connections 2024-02-28T17:01:53.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.031+0000 7fa00da00700 1 -- 172.21.15.1:0/4169421344 wait complete. 2024-02-28T17:01:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.031+0000 7fa00da00700 1 Processor -- start 2024-02-28T17:01:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.032+0000 7fa00da00700 1 -- start start 2024-02-28T17:01:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.032+0000 7fa00da00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109380 0x7fa008119db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.032+0000 7fa00da00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa008109d20 0x7fa00811a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.032+0000 7fa00da00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00811a940 con 0x7fa008109380 2024-02-28T17:01:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.032+0000 7fa00da00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa008113c90 con 0x7fa008109d20 2024-02-28T17:01:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.032+0000 7fa007e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109380 0x7fa008119db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.032+0000 7fa007e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109380 0x7fa008119db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34028/0 (socket says 172.21.15.1:34028) 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.032+0000 7fa007e00700 1 -- 172.21.15.1:0/3052635034 learned_addr learned my addr 172.21.15.1:0/3052635034 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7fa007400700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa008109d20 0x7fa00811a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7fa007e00700 1 -- 172.21.15.1:0/3052635034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa008109d20 msgr2=0x7fa00811a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7fa007e00700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa008109d20 0x7fa00811a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7fa007e00700 1 -- 172.21.15.1:0/3052635034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4005040 con 0x7fa008109380 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7fa007e00700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109380 0x7fa008119db0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800c8c0 tx=0x7f9ff80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7f9fffe00700 1 -- 172.21.15.1:0/3052635034 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ff800a980 con 0x7fa008109380 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7f9fffe00700 1 -- 172.21.15.1:0/3052635034 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ff800ce60 con 0x7fa008109380 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7f9fffe00700 1 -- 172.21.15.1:0/3052635034 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ff80115e0 con 0x7fa008109380 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.033+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa008113f70 con 0x7fa008109380 2024-02-28T17:01:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.034+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa008114530 con 0x7fa008109380 2024-02-28T17:01:53.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.035+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00805f5c0 con 0x7fa008109380 2024-02-28T17:01:53.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.036+0000 7f9fffe00700 1 -- 172.21.15.1:0/3052635034 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9ff8012400 con 0x7fa008109380 2024-02-28T17:01:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.037+0000 7f9fffe00700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9fec06d7e0 0x7f9fec06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.037+0000 7f9fffe00700 1 -- 172.21.15.1:0/3052635034 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9ff808cf70 con 0x7fa008109380 2024-02-28T17:01:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.037+0000 7fa007400700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9fec06d7e0 0x7f9fec06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.038+0000 7fa007400700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9fec06d7e0 0x7f9fec06fc90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b3b0 tx=0x7f9ff4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.041+0000 7f9fffe00700 1 -- 172.21.15.1:0/3052635034 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9ff8056820 con 0x7fa008109380 2024-02-28T17:01:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.285+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa008115440 con 0x7f9fec06d7e0 2024-02-28T17:01:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.288+0000 7f9fffe00700 1 -- 172.21.15.1:0/3052635034 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa008115440 con 0x7f9fec06d7e0 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:01:53.288 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:01:53.289 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:01:53.289 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:01:53.289 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:01:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.290+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9fec06d7e0 msgr2=0x7f9fec06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.290+0000 7fa00da00700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9fec06d7e0 0x7f9fec06fc90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b3b0 tx=0x7f9ff4009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.290+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109380 msgr2=0x7fa008119db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:53.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.291+0000 7fa00da00700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109380 0x7fa008119db0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800c8c0 tx=0x7f9ff80044c0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.291+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 shutdown_connections 2024-02-28T17:01:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.291+0000 7fa00da00700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9fec06d7e0 0x7f9fec06fc90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.291+0000 7fa00da00700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008109380 0x7fa008119db0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.291+0000 7fa00da00700 1 --2- 172.21.15.1:0/3052635034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa008109d20 0x7fa00811a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.291+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 >> 172.21.15.1:0/3052635034 conn(0x7fa008072b50 msgr2=0x7fa008075600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.291+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 shutdown_connections 2024-02-28T17:01:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.291+0000 7fa00da00700 1 -- 172.21.15.1:0/3052635034 wait complete. 2024-02-28T17:01:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.447+0000 7f66c1000700 1 -- 172.21.15.1:0/3076031159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 msgr2=0x7f66bc070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.447+0000 7f66c1000700 1 --2- 172.21.15.1:0/3076031159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 0x7f66bc070330 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f66ac004970 tx=0x7f66ac009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.447+0000 7f66c1000700 1 -- 172.21.15.1:0/3076031159 shutdown_connections 2024-02-28T17:01:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.447+0000 7f66c1000700 1 --2- 172.21.15.1:0/3076031159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66bc070870 0x7f66bc070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.447+0000 7f66c1000700 1 --2- 172.21.15.1:0/3076031159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 0x7f66bc070330 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.447+0000 7f66c1000700 1 -- 172.21.15.1:0/3076031159 >> 172.21.15.1:0/3076031159 conn(0x7f66bc0f8ae0 msgr2=0x7f66bc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.447+0000 7f66c1000700 1 -- 172.21.15.1:0/3076031159 shutdown_connections 2024-02-28T17:01:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.447+0000 7f66c1000700 1 -- 172.21.15.1:0/3076031159 wait complete. 2024-02-28T17:01:53.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.448+0000 7f66c1000700 1 Processor -- start 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.448+0000 7f66c1000700 1 -- start start 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66c1000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66bc070870 0x7f66bc11a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66c1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 0x7f66bc11a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66c1000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc11abe0 con 0x7f66bc071f60 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66c1000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc113ef0 con 0x7f66bc070870 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66b9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 0x7f66bc11a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66b9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 0x7f66bc11a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34036/0 (socket says 172.21.15.1:34036) 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66b9400700 1 -- 172.21.15.1:0/3102092853 learned_addr learned my addr 172.21.15.1:0/3102092853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66b9400700 1 -- 172.21.15.1:0/3102092853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66bc070870 msgr2=0x7f66bc11a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66b9400700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66bc070870 0x7f66bc11a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66b9400700 1 -- 172.21.15.1:0/3102092853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac005040 con 0x7f66bc071f60 2024-02-28T17:01:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.449+0000 7f66b9400700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 0x7f66bc11a590 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f66a8002a50 tx=0x7f66a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:53.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.450+0000 7f66b2000700 1 -- 172.21.15.1:0/3102092853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a8007cc0 con 0x7f66bc071f60 2024-02-28T17:01:53.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.450+0000 7f66b2000700 1 -- 172.21.15.1:0/3102092853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66a800f040 con 0x7f66bc071f60 2024-02-28T17:01:53.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.450+0000 7f66b2000700 1 -- 172.21.15.1:0/3102092853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a8009640 con 0x7f66bc071f60 2024-02-28T17:01:53.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.450+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66bc1141d0 con 0x7f66bc071f60 2024-02-28T17:01:53.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.450+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66bc114710 con 0x7f66bc071f60 2024-02-28T17:01:53.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.451+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66bc0fe400 con 0x7f66bc071f60 2024-02-28T17:01:53.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.452+0000 7f66b2000700 1 -- 172.21.15.1:0/3102092853 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f66a80097a0 con 0x7f66bc071f60 2024-02-28T17:01:53.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.453+0000 7f66b2000700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f66a006d7e0 0x7f66a006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:01:53.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.453+0000 7f66b2000700 1 -- 172.21.15.1:0/3102092853 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f66a808cbc0 con 0x7f66bc071f60 2024-02-28T17:01:53.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.453+0000 7f66b9e00700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f66a006d7e0 0x7f66a006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:01:53.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.454+0000 7f66b9e00700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f66a006d7e0 0x7f66a006fc90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66ac004540 tx=0x7f66ac009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:01:53.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.458+0000 7f66b2000700 1 -- 172.21.15.1:0/3102092853 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f66a8059fc0 con 0x7f66bc071f60 2024-02-28T17:01:53.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:53 smithi066 ceph-mon[85781]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:53.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:53 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1626186693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:01:53.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:53 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2045685578' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:01:53.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:53 smithi001 ceph-mon[112940]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:53.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:53 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1626186693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:01:53.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:53 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2045685578' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:01:53.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.773+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f66bc0fc6e0 con 0x7f66bc071f60 2024-02-28T17:01:53.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.774+0000 7f66b2000700 1 -- 172.21.15.1:0/3102092853 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f66a8059b50 con 0x7f66bc071f60 2024-02-28T17:01:53.774 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:01:53.774 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:01:53.774 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:01:53.774 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:01:53.774 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:01:53.774 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:01:53.774 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:01:53.774 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:01:53.775 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:01:53.776 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:01:53.776 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:01:53.776 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:01:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f66a006d7e0 msgr2=0x7f66a006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f66a006d7e0 0x7f66a006fc90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66ac004540 tx=0x7f66ac009e30 comp rx=0 tx=0).stop 2024-02-28T17:01:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 msgr2=0x7f66bc11a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:01:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 0x7f66bc11a590 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f66a8002a50 tx=0x7f66a80046b0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 shutdown_connections 2024-02-28T17:01:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f66a006d7e0 0x7f66a006fc90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66bc070870 0x7f66bc11a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 --2- 172.21.15.1:0/3102092853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66bc071f60 0x7f66bc11a590 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:01:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 >> 172.21.15.1:0/3102092853 conn(0x7f66bc0f8ae0 msgr2=0x7f66bc0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:01:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.777+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 shutdown_connections 2024-02-28T17:01:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:01:53.778+0000 7f66c1000700 1 -- 172.21.15.1:0/3102092853 wait complete. 2024-02-28T17:01:54.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:54 smithi066 ceph-mon[85781]: pgmap v824: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:54.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:54 smithi066 ceph-mon[85781]: from='client.35506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:54.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:54 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3102092853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:01:54.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:54 smithi001 ceph-mon[112940]: pgmap v824: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:54.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:54 smithi001 ceph-mon[112940]: from='client.35506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:01:54.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:54 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3102092853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:01:56.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:56 smithi066 ceph-mon[85781]: pgmap v825: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:56.509 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:01:56.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:56 smithi001 ceph-mon[112940]: pgmap v825: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:56.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:01:58.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:01:58 smithi001 ceph-mon[112940]: pgmap v826: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:01:58.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:01:58 smithi066 ceph-mon[85781]: pgmap v826: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:00.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:00 smithi001 ceph-mon[112940]: pgmap v827: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:00.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:00 smithi066 ceph-mon[85781]: pgmap v827: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:02.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:02 smithi001 ceph-mon[112940]: pgmap v828: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:02.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:02 smithi066 ceph-mon[85781]: pgmap v828: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:04.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:04 smithi001 ceph-mon[112940]: pgmap v829: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:04.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:04 smithi066 ceph-mon[85781]: pgmap v829: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:06.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:06 smithi066 ceph-mon[85781]: pgmap v830: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:06.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:06 smithi001 ceph-mon[112940]: pgmap v830: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:08.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:08 smithi066 ceph-mon[85781]: pgmap v831: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:08.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:08 smithi001 ceph-mon[112940]: pgmap v831: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:10.562 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:10 smithi066 ceph-mon[85781]: pgmap v832: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:10.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:10 smithi001 ceph-mon[112940]: pgmap v832: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:11.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:02:11.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:02:12.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:12 smithi066 ceph-mon[85781]: pgmap v833: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:12.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:12 smithi001 ceph-mon[112940]: pgmap v833: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:14.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:14 smithi066 ceph-mon[85781]: pgmap v834: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:14.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:14 smithi001 ceph-mon[112940]: pgmap v834: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:16.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:16 smithi066 ceph-mon[85781]: pgmap v835: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:16.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:16 smithi001 ceph-mon[112940]: pgmap v835: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:18.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:18 smithi066 ceph-mon[85781]: pgmap v836: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:18.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:18 smithi001 ceph-mon[112940]: pgmap v836: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:20.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:20 smithi066 ceph-mon[85781]: pgmap v837: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:20.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:20 smithi001 ceph-mon[112940]: pgmap v837: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:22.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:22 smithi066 ceph-mon[85781]: pgmap v838: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:22.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:22 smithi001 ceph-mon[112940]: pgmap v838: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:23.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.947+0000 7fbab4400700 1 -- 172.21.15.1:0/2104547173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 msgr2=0x7fbaac0706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:23.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.947+0000 7fbab4400700 1 --2- 172.21.15.1:0/2104547173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 0x7fbaac0706a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0004970 tx=0x7fbaa0009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:23.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.947+0000 7fbab4400700 1 -- 172.21.15.1:0/2104547173 shutdown_connections 2024-02-28T17:02:23.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.947+0000 7fbab4400700 1 --2- 172.21.15.1:0/2104547173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaac070be0 0x7fbaac1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:23.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.947+0000 7fbab4400700 1 --2- 172.21.15.1:0/2104547173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 0x7fbaac0706a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:23.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.947+0000 7fbab4400700 1 -- 172.21.15.1:0/2104547173 >> 172.21.15.1:0/2104547173 conn(0x7fbaac0f8ae0 msgr2=0x7fbaac0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:23.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.947+0000 7fbab4400700 1 -- 172.21.15.1:0/2104547173 shutdown_connections 2024-02-28T17:02:23.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.947+0000 7fbab4400700 1 -- 172.21.15.1:0/2104547173 wait complete. 2024-02-28T17:02:23.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.948+0000 7fbab4400700 1 Processor -- start 2024-02-28T17:02:23.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.948+0000 7fbab4400700 1 -- start start 2024-02-28T17:02:23.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.949+0000 7fbab4400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 0x7fbaac115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:23.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.949+0000 7fbab4400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaac070be0 0x7fbaac116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:23.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.949+0000 7fbab4400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac1167b0 con 0x7fbaac0702d0 2024-02-28T17:02:23.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.949+0000 7fbab4400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac10fb00 con 0x7fbaac070be0 2024-02-28T17:02:23.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.949+0000 7fbab1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 0x7fbaac115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:23.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.949+0000 7fbab1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 0x7fbaac115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51628/0 (socket says 172.21.15.1:51628) 2024-02-28T17:02:23.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.949+0000 7fbab1a00700 1 -- 172.21.15.1:0/75437234 learned_addr learned my addr 172.21.15.1:0/75437234 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:23.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.949+0000 7fbab1000700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaac070be0 0x7fbaac116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:23.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbab1a00700 1 -- 172.21.15.1:0/75437234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaac070be0 msgr2=0x7fbaac116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:23.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbab1a00700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaac070be0 0x7fbaac116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:23.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbab1a00700 1 -- 172.21.15.1:0/75437234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa0005040 con 0x7fbaac0702d0 2024-02-28T17:02:23.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbab1a00700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 0x7fbaac115c20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0005010 tx=0x7fbaa0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:23.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbaa9600700 1 -- 172.21.15.1:0/75437234 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbaa000e040 con 0x7fbaac0702d0 2024-02-28T17:02:23.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbaa9600700 1 -- 172.21.15.1:0/75437234 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbaa0004290 con 0x7fbaac0702d0 2024-02-28T17:02:23.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaac10fd80 con 0x7fbaac0702d0 2024-02-28T17:02:23.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbaa9600700 1 -- 172.21.15.1:0/75437234 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbaa001c3f0 con 0x7fbaac0702d0 2024-02-28T17:02:23.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.950+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaac1102c0 con 0x7fbaac0702d0 2024-02-28T17:02:23.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.952+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaac106980 con 0x7fbaac0702d0 2024-02-28T17:02:23.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.953+0000 7fbaa9600700 1 -- 172.21.15.1:0/75437234 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbaa0013820 con 0x7fbaac0702d0 2024-02-28T17:02:23.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.954+0000 7fbaa9600700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba9406d7f0 0x7fba9406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:23.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.954+0000 7fbaa9600700 1 -- 172.21.15.1:0/75437234 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbaa008e220 con 0x7fbaac0702d0 2024-02-28T17:02:23.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.954+0000 7fbab1000700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba9406d7f0 0x7fba9406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:23.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.955+0000 7fbab1000700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba9406d7f0 0x7fba9406fca0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fbaac1114c0 tx=0x7fba9800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:23.959+0000 7fbaa9600700 1 -- 172.21.15.1:0/75437234 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbaa0057a50 con 0x7fbaac0702d0 2024-02-28T17:02:24.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.204+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaac04bf40 con 0x7fba9406d7f0 2024-02-28T17:02:24.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.208+0000 7fbaa9600700 1 -- 172.21.15.1:0/75437234 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbaac04bf40 con 0x7fba9406d7f0 2024-02-28T17:02:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba9406d7f0 msgr2=0x7fba9406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba9406d7f0 0x7fba9406fca0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fbaac1114c0 tx=0x7fba9800b040 comp rx=0 tx=0).stop 2024-02-28T17:02:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 msgr2=0x7fbaac115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 0x7fbaac115c20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0005010 tx=0x7fbaa0009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 shutdown_connections 2024-02-28T17:02:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba9406d7f0 0x7fba9406fca0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaac0702d0 0x7fbaac115c20 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 --2- 172.21.15.1:0/75437234 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaac070be0 0x7fbaac116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.210+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 >> 172.21.15.1:0/75437234 conn(0x7fbaac0f8ae0 msgr2=0x7fbaac103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.211+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 shutdown_connections 2024-02-28T17:02:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.211+0000 7fbab4400700 1 -- 172.21.15.1:0/75437234 wait complete. 2024-02-28T17:02:24.226 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:02:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.367+0000 7f8b84200700 1 -- 172.21.15.1:0/2761706622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 msgr2=0x7f8b7c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.367+0000 7f8b84200700 1 --2- 172.21.15.1:0/2761706622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 0x7f8b7c103320 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8b70004940 tx=0x7f8b70009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.367+0000 7f8b84200700 1 -- 172.21.15.1:0/2761706622 shutdown_connections 2024-02-28T17:02:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.367+0000 7f8b84200700 1 --2- 172.21.15.1:0/2761706622 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b7c1038f0 0x7f8b7c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.367+0000 7f8b84200700 1 --2- 172.21.15.1:0/2761706622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 0x7f8b7c103320 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.367+0000 7f8b84200700 1 -- 172.21.15.1:0/2761706622 >> 172.21.15.1:0/2761706622 conn(0x7f8b7c0f8ae0 msgr2=0x7f8b7c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.367+0000 7f8b84200700 1 -- 172.21.15.1:0/2761706622 shutdown_connections 2024-02-28T17:02:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.367+0000 7f8b84200700 1 -- 172.21.15.1:0/2761706622 wait complete. 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.368+0000 7f8b84200700 1 Processor -- start 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.368+0000 7f8b84200700 1 -- start start 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.368+0000 7f8b84200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 0x7f8b7c115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.368+0000 7f8b84200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b7c1038f0 0x7f8b7c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.368+0000 7f8b84200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c1167a0 con 0x7f8b7c102f50 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.368+0000 7f8b84200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c10faf0 con 0x7f8b7c1038f0 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.369+0000 7f8b81800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 0x7f8b7c115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.369+0000 7f8b81800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 0x7f8b7c115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51632/0 (socket says 172.21.15.1:51632) 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.369+0000 7f8b81800700 1 -- 172.21.15.1:0/1190404567 learned_addr learned my addr 172.21.15.1:0/1190404567 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.369+0000 7f8b81800700 1 -- 172.21.15.1:0/1190404567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b7c1038f0 msgr2=0x7f8b7c116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.369+0000 7f8b81800700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b7c1038f0 0x7f8b7c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.369+0000 7f8b81800700 1 -- 172.21.15.1:0/1190404567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b70005040 con 0x7f8b7c102f50 2024-02-28T17:02:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.369+0000 7f8b81800700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 0x7f8b7c115c10 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009150 tx=0x7f8b70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:24.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.370+0000 7f8b79600700 1 -- 172.21.15.1:0/1190404567 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b7000e040 con 0x7f8b7c102f50 2024-02-28T17:02:24.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.370+0000 7f8b79600700 1 -- 172.21.15.1:0/1190404567 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b7000bbf0 con 0x7f8b7c102f50 2024-02-28T17:02:24.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.370+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7c10fd70 con 0x7f8b7c102f50 2024-02-28T17:02:24.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.370+0000 7f8b79600700 1 -- 172.21.15.1:0/1190404567 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b7001c630 con 0x7f8b7c102f50 2024-02-28T17:02:24.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.370+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7c1102b0 con 0x7f8b7c102f50 2024-02-28T17:02:24.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.371+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7c0fc6e0 con 0x7f8b7c102f50 2024-02-28T17:02:24.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.372+0000 7f8b79600700 1 -- 172.21.15.1:0/1190404567 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8b7000bd60 con 0x7f8b7c102f50 2024-02-28T17:02:24.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.373+0000 7f8b79600700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b6406d7f0 0x7f8b6406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:24.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.373+0000 7f8b79600700 1 -- 172.21.15.1:0/1190404567 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8b7008e2c0 con 0x7f8b7c102f50 2024-02-28T17:02:24.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.373+0000 7f8b80e00700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b6406d7f0 0x7f8b6406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:24.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.374+0000 7f8b80e00700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b6406d7f0 0x7f8b6406fca0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c1114b0 tx=0x7f8b68009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:24.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.378+0000 7f8b79600700 1 -- 172.21.15.1:0/1190404567 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8b70057b70 con 0x7f8b7c102f50 2024-02-28T17:02:24.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:24 smithi001 ceph-mon[112940]: pgmap v839: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:24.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.612+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b7c111290 con 0x7f8b6406d7f0 2024-02-28T17:02:24.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.614+0000 7f8b79600700 1 -- 172.21.15.1:0/1190404567 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8b7c111290 con 0x7f8b6406d7f0 2024-02-28T17:02:24.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.616+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b6406d7f0 msgr2=0x7f8b6406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.616+0000 7f8b84200700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b6406d7f0 0x7f8b6406fca0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c1114b0 tx=0x7f8b68009ce0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 msgr2=0x7f8b7c115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 0x7f8b7c115c10 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009150 tx=0x7f8b70009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 shutdown_connections 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8b6406d7f0 0x7f8b6406fca0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b7c102f50 0x7f8b7c115c10 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 --2- 172.21.15.1:0/1190404567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8b7c1038f0 0x7f8b7c116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 >> 172.21.15.1:0/1190404567 conn(0x7f8b7c0f8ae0 msgr2=0x7f8b7c0fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 shutdown_connections 2024-02-28T17:02:24.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.617+0000 7f8b84200700 1 -- 172.21.15.1:0/1190404567 wait complete. 2024-02-28T17:02:24.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:24 smithi066 ceph-mon[85781]: pgmap v839: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.771+0000 7f4564400700 1 -- 172.21.15.1:0/2103995448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 msgr2=0x7f455c107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.771+0000 7f4564400700 1 --2- 172.21.15.1:0/2103995448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 0x7f455c107050 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4548004970 tx=0x7f4548009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.771+0000 7f4564400700 1 -- 172.21.15.1:0/2103995448 shutdown_connections 2024-02-28T17:02:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.771+0000 7f4564400700 1 --2- 172.21.15.1:0/2103995448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 0x7f455c107050 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.771+0000 7f4564400700 1 --2- 172.21.15.1:0/2103995448 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f455c0fdfe0 0x7f455c0fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.771+0000 7f4564400700 1 -- 172.21.15.1:0/2103995448 >> 172.21.15.1:0/2103995448 conn(0x7f455c071d90 msgr2=0x7f455c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.771+0000 7f4564400700 1 -- 172.21.15.1:0/2103995448 shutdown_connections 2024-02-28T17:02:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.771+0000 7f4564400700 1 -- 172.21.15.1:0/2103995448 wait complete. 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.772+0000 7f4564400700 1 Processor -- start 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.772+0000 7f4564400700 1 -- start start 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.773+0000 7f4564400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f455c0fdfe0 0x7f455c1139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.773+0000 7f4564400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 0x7f455c113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.773+0000 7f4564400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c114580 con 0x7f455c0fe8f0 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.773+0000 7f4564400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c10d8b0 con 0x7f455c0fdfe0 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.773+0000 7f4561000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 0x7f455c113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.773+0000 7f4561000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 0x7f455c113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51648/0 (socket says 172.21.15.1:51648) 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.773+0000 7f4561000700 1 -- 172.21.15.1:0/1727540576 learned_addr learned my addr 172.21.15.1:0/1727540576 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4561000700 1 -- 172.21.15.1:0/1727540576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f455c0fdfe0 msgr2=0x7f455c1139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:24.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4561a00700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f455c0fdfe0 0x7f455c1139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:24.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4561000700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f455c0fdfe0 0x7f455c1139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:24.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4561000700 1 -- 172.21.15.1:0/1727540576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4548005040 con 0x7f455c0fe8f0 2024-02-28T17:02:24.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4561000700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 0x7f455c113f30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4548009150 tx=0x7f4548009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:24.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4559600700 1 -- 172.21.15.1:0/1727540576 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f454800e040 con 0x7f455c0fe8f0 2024-02-28T17:02:24.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4559600700 1 -- 172.21.15.1:0/1727540576 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4548004290 con 0x7f455c0fe8f0 2024-02-28T17:02:24.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455c10db30 con 0x7f455c0fe8f0 2024-02-28T17:02:24.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4559600700 1 -- 172.21.15.1:0/1727540576 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f454801c630 con 0x7f455c0fe8f0 2024-02-28T17:02:24.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.774+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455c10e070 con 0x7f455c0fe8f0 2024-02-28T17:02:24.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.775+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455c1046c0 con 0x7f455c0fe8f0 2024-02-28T17:02:24.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.777+0000 7f4559600700 1 -- 172.21.15.1:0/1727540576 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4548013980 con 0x7f455c0fe8f0 2024-02-28T17:02:24.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.777+0000 7f4559600700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f454406d7f0 0x7f454406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:24.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.778+0000 7f4559600700 1 -- 172.21.15.1:0/1727540576 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f454808e2c0 con 0x7f455c0fe8f0 2024-02-28T17:02:24.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.778+0000 7f4561a00700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f454406d7f0 0x7f454406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:24.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.778+0000 7f4561a00700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f454406d7f0 0x7f454406fca0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4550004c40 tx=0x7f4550009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:24.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:24.782+0000 7f4559600700 1 -- 172.21.15.1:0/1727540576 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f454805b620 con 0x7f455c0fe8f0 2024-02-28T17:02:25.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.013+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f455c1136c0 con 0x7f454406d7f0 2024-02-28T17:02:25.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.036+0000 7f4559600700 1 -- 172.21.15.1:0/1727540576 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f455c1136c0 con 0x7f454406d7f0 2024-02-28T17:02:25.037 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:02:25.037 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (28m) 5m ago 42m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:02:25.037 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (25m) 5m ago 43m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:02:25.037 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (25m) 5m ago 41m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:02:25.037 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (28m) 5m ago 42m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:02:25.037 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (21m) 5m ago 35m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (20m) 5m ago 35m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (35m) 5m ago 35m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (5m) 5m ago 35m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (30m) 5m ago 44m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (29m) 5m ago 41m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (26m) 5m ago 44m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (26m) 5m ago 41m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (29m) 5m ago 42m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (29m) 5m ago 41m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (24m) 5m ago 40m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (24m) 5m ago 39m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (23m) 5m ago 39m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (23m) 5m ago 38m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:02:25.038 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (22m) 5m ago 38m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:02:25.039 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (21m) 5m ago 37m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:02:25.039 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (29m) 5m ago 42m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:02:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f454406d7f0 msgr2=0x7f454406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f454406d7f0 0x7f454406fca0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4550004c40 tx=0x7f4550009040 comp rx=0 tx=0).stop 2024-02-28T17:02:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 msgr2=0x7f455c113f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 0x7f455c113f30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4548009150 tx=0x7f4548009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 shutdown_connections 2024-02-28T17:02:25.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f454406d7f0 0x7f454406fca0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f455c0fdfe0 0x7f455c1139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 --2- 172.21.15.1:0/1727540576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c0fe8f0 0x7f455c113f30 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 >> 172.21.15.1:0/1727540576 conn(0x7f455c071d90 msgr2=0x7f455c101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 shutdown_connections 2024-02-28T17:02:25.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.039+0000 7f4564400700 1 -- 172.21.15.1:0/1727540576 wait complete. 2024-02-28T17:02:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.191+0000 7ff57c800700 1 -- 172.21.15.1:0/2867384223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574103dc0 msgr2=0x7ff574104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.191+0000 7ff57c800700 1 --2- 172.21.15.1:0/2867384223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574103dc0 0x7ff574104210 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff564004940 tx=0x7ff564009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.191+0000 7ff57c800700 1 -- 172.21.15.1:0/2867384223 shutdown_connections 2024-02-28T17:02:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.191+0000 7ff57c800700 1 --2- 172.21.15.1:0/2867384223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574103dc0 0x7ff574104210 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.191+0000 7ff57c800700 1 --2- 172.21.15.1:0/2867384223 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff5741095f0 0x7ff5741099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.191+0000 7ff57c800700 1 -- 172.21.15.1:0/2867384223 >> 172.21.15.1:0/2867384223 conn(0x7ff574072d50 msgr2=0x7ff574075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.191+0000 7ff57c800700 1 -- 172.21.15.1:0/2867384223 shutdown_connections 2024-02-28T17:02:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.191+0000 7ff57c800700 1 -- 172.21.15.1:0/2867384223 wait complete. 2024-02-28T17:02:25.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.192+0000 7ff57c800700 1 Processor -- start 2024-02-28T17:02:25.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.192+0000 7ff57c800700 1 -- start start 2024-02-28T17:02:25.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff57c800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff574103dc0 0x7ff57411a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:25.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff57c800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5741095f0 0x7ff57411a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff57c800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57411abd0 con 0x7ff5741095f0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff57c800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff574113ee0 con 0x7ff574103dc0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff579400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5741095f0 0x7ff57411a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff579400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5741095f0 0x7ff57411a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51662/0 (socket says 172.21.15.1:51662) 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff579400700 1 -- 172.21.15.1:0/3408453519 learned_addr learned my addr 172.21.15.1:0/3408453519 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff579400700 1 -- 172.21.15.1:0/3408453519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff574103dc0 msgr2=0x7ff57411a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff579400700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff574103dc0 0x7ff57411a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff579400700 1 -- 172.21.15.1:0/3408453519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff564005040 con 0x7ff5741095f0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.193+0000 7ff579400700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5741095f0 0x7ff57411a580 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff564009fd0 tx=0x7ff564009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.194+0000 7ff56e000700 1 -- 172.21.15.1:0/3408453519 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff56400e040 con 0x7ff5741095f0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.194+0000 7ff56e000700 1 -- 172.21.15.1:0/3408453519 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff56400bbf0 con 0x7ff5741095f0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.194+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff574114160 con 0x7ff5741095f0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.194+0000 7ff56e000700 1 -- 172.21.15.1:0/3408453519 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff56401c3f0 con 0x7ff5741095f0 2024-02-28T17:02:25.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.194+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5741146a0 con 0x7ff5741095f0 2024-02-28T17:02:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.195+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57410b9c0 con 0x7ff5741095f0 2024-02-28T17:02:25.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.196+0000 7ff56e000700 1 -- 172.21.15.1:0/3408453519 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff56400bd60 con 0x7ff5741095f0 2024-02-28T17:02:25.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.197+0000 7ff56e000700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff55c06d7f0 0x7ff55c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:25.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.197+0000 7ff56e000700 1 -- 172.21.15.1:0/3408453519 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff56408e2b0 con 0x7ff5741095f0 2024-02-28T17:02:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.197+0000 7ff579e00700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff55c06d7f0 0x7ff55c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.198+0000 7ff579e00700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff55c06d7f0 0x7ff55c06fca0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff568004b70 tx=0x7ff568007cc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:25.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.202+0000 7ff56e000700 1 -- 172.21.15.1:0/3408453519 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff56405b610 con 0x7ff5741095f0 2024-02-28T17:02:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.509+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff57404bed0 con 0x7ff5741095f0 2024-02-28T17:02:25.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.511+0000 7ff56e000700 1 -- 172.21.15.1:0/3408453519 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7ff56405b1a0 con 0x7ff5741095f0 2024-02-28T17:02:25.511 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:02:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:02:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:02:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:02:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:02:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:02:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:02:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:02:25.513 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:02:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff55c06d7f0 msgr2=0x7ff55c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff55c06d7f0 0x7ff55c06fca0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff568004b70 tx=0x7ff568007cc0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5741095f0 msgr2=0x7ff57411a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5741095f0 0x7ff57411a580 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff564009fd0 tx=0x7ff564009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 shutdown_connections 2024-02-28T17:02:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff55c06d7f0 0x7ff55c06fca0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff574103dc0 0x7ff57411a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 --2- 172.21.15.1:0/3408453519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5741095f0 0x7ff57411a580 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.513+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 >> 172.21.15.1:0/3408453519 conn(0x7ff574072d50 msgr2=0x7ff574074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.514+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 shutdown_connections 2024-02-28T17:02:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.514+0000 7ff57c800700 1 -- 172.21.15.1:0/3408453519 wait complete. 2024-02-28T17:02:25.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.657+0000 7f664f600700 1 -- 172.21.15.1:0/1008716516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6648102f50 msgr2=0x7f6648103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.657+0000 7f664f600700 1 --2- 172.21.15.1:0/1008716516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6648102f50 0x7f6648103320 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f663c004940 tx=0x7f663c009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:25.658 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:25 smithi001 ceph-mon[112940]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:25.658 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:25 smithi001 ceph-mon[112940]: from='client.35518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.657+0000 7f664f600700 1 -- 172.21.15.1:0/1008716516 shutdown_connections 2024-02-28T17:02:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.657+0000 7f664f600700 1 --2- 172.21.15.1:0/1008716516 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66481038f0 0x7f66480fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.657+0000 7f664f600700 1 --2- 172.21.15.1:0/1008716516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6648102f50 0x7f6648103320 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.657+0000 7f664f600700 1 -- 172.21.15.1:0/1008716516 >> 172.21.15.1:0/1008716516 conn(0x7f66480f8ae0 msgr2=0x7f66480faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.658+0000 7f664f600700 1 -- 172.21.15.1:0/1008716516 shutdown_connections 2024-02-28T17:02:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.658+0000 7f664f600700 1 -- 172.21.15.1:0/1008716516 wait complete. 2024-02-28T17:02:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.658+0000 7f664f600700 1 Processor -- start 2024-02-28T17:02:25.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.659+0000 7f664f600700 1 -- start start 2024-02-28T17:02:25.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.659+0000 7f664f600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6648102f50 0x7f6648070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:25.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.659+0000 7f664f600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66481038f0 0x7f66480707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.659+0000 7f664f600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648070dd0 con 0x7f66481038f0 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.659+0000 7f664f600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648070f40 con 0x7f6648102f50 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.659+0000 7f6647e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66481038f0 0x7f66480707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.659+0000 7f6647e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66481038f0 0x7f66480707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51688/0 (socket says 172.21.15.1:51688) 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.659+0000 7f6647e00700 1 -- 172.21.15.1:0/542802698 learned_addr learned my addr 172.21.15.1:0/542802698 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f664cc00700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6648102f50 0x7f6648070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f6647e00700 1 -- 172.21.15.1:0/542802698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6648102f50 msgr2=0x7f6648070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f6647e00700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6648102f50 0x7f6648070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f6647e00700 1 -- 172.21.15.1:0/542802698 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c005040 con 0x7f66481038f0 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f664cc00700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6648102f50 0x7f6648070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:02:25.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f6647e00700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66481038f0 0x7f66480707d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6634002a50 tx=0x7f6634004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f6644c00700 1 -- 172.21.15.1:0/542802698 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6634007cc0 con 0x7f66481038f0 2024-02-28T17:02:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f6644c00700 1 -- 172.21.15.1:0/542802698 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f663400f040 con 0x7f66481038f0 2024-02-28T17:02:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6648119ce0 con 0x7f66481038f0 2024-02-28T17:02:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.660+0000 7f6644c00700 1 -- 172.21.15.1:0/542802698 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663400da40 con 0x7f66481038f0 2024-02-28T17:02:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.661+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664811a1c0 con 0x7f66481038f0 2024-02-28T17:02:25.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.662+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6648100260 con 0x7f66481038f0 2024-02-28T17:02:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.663+0000 7f6644c00700 1 -- 172.21.15.1:0/542802698 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f66340097c0 con 0x7f66481038f0 2024-02-28T17:02:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.664+0000 7f6644c00700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663006d7f0 0x7f663006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.664+0000 7f6644c00700 1 -- 172.21.15.1:0/542802698 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f663408cc00 con 0x7f66481038f0 2024-02-28T17:02:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.665+0000 7f664cc00700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663006d7f0 0x7f663006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:25.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.665+0000 7f664cc00700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663006d7f0 0x7f663006fca0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f663c00b3b0 tx=0x7f663c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.669+0000 7f6644c00700 1 -- 172.21.15.1:0/542802698 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6634092020 con 0x7f66481038f0 2024-02-28T17:02:25.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:25 smithi066 ceph-mon[85781]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:25.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:25 smithi066 ceph-mon[85781]: from='client.35518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.929+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f66480719d0 con 0x7f66481038f0 2024-02-28T17:02:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.930+0000 7f6644c00700 1 -- 172.21.15.1:0/542802698 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f663405a000 con 0x7f66481038f0 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:02:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:02:25.933 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:02:25.934 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:02:25.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:02:25.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663006d7f0 msgr2=0x7f663006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663006d7f0 0x7f663006fca0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f663c00b3b0 tx=0x7f663c009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66481038f0 msgr2=0x7f66480707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66481038f0 0x7f66480707d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6634002a50 tx=0x7f6634004600 comp rx=0 tx=0).stop 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 shutdown_connections 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663006d7f0 0x7f663006fca0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6648102f50 0x7f6648070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 --2- 172.21.15.1:0/542802698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66481038f0 0x7f66480707d0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 >> 172.21.15.1:0/542802698 conn(0x7f66480f8ae0 msgr2=0x7f66480fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 shutdown_connections 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:25.933+0000 7f664f600700 1 -- 172.21.15.1:0/542802698 wait complete. 2024-02-28T17:02:25.936 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:02:26.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.086+0000 7f2972e00700 1 -- 172.21.15.1:0/1214939210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 msgr2=0x7f296c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:26.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.086+0000 7f2972e00700 1 --2- 172.21.15.1:0/1214939210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 0x7f296c103320 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f295c004940 tx=0x7f295c009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:26.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.087+0000 7f2972e00700 1 -- 172.21.15.1:0/1214939210 shutdown_connections 2024-02-28T17:02:26.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.087+0000 7f2972e00700 1 --2- 172.21.15.1:0/1214939210 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f296c1038f0 0x7f296c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.087+0000 7f2972e00700 1 --2- 172.21.15.1:0/1214939210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 0x7f296c103320 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.087+0000 7f2972e00700 1 -- 172.21.15.1:0/1214939210 >> 172.21.15.1:0/1214939210 conn(0x7f296c0f8ae0 msgr2=0x7f296c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:26.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.087+0000 7f2972e00700 1 -- 172.21.15.1:0/1214939210 shutdown_connections 2024-02-28T17:02:26.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.087+0000 7f2972e00700 1 -- 172.21.15.1:0/1214939210 wait complete. 2024-02-28T17:02:26.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.088+0000 7f2972e00700 1 Processor -- start 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.088+0000 7f2972e00700 1 -- start start 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.088+0000 7f2972e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 0x7f296c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.088+0000 7f2972e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f296c1038f0 0x7f296c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.088+0000 7f2972e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c116800 con 0x7f296c102f50 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.088+0000 7f2972e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c10fb30 con 0x7f296c1038f0 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.088+0000 7f296be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 0x7f296c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f296be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 0x7f296c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51698/0 (socket says 172.21.15.1:51698) 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f296be00700 1 -- 172.21.15.1:0/3668371286 learned_addr learned my addr 172.21.15.1:0/3668371286 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f296be00700 1 -- 172.21.15.1:0/3668371286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f296c1038f0 msgr2=0x7f296c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f296b400700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f296c1038f0 0x7f296c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f296be00700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f296c1038f0 0x7f296c1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f296be00700 1 -- 172.21.15.1:0/3668371286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c005040 con 0x7f296c102f50 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f296b400700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f296c1038f0 0x7f296c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:02:26.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f296be00700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 0x7f296c115c70 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f295c009fd0 tx=0x7f295c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.089+0000 7f2963e00700 1 -- 172.21.15.1:0/3668371286 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f295c00e040 con 0x7f296c102f50 2024-02-28T17:02:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.090+0000 7f2963e00700 1 -- 172.21.15.1:0/3668371286 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f295c00bbf0 con 0x7f296c102f50 2024-02-28T17:02:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.090+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c10fdb0 con 0x7f296c102f50 2024-02-28T17:02:26.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.090+0000 7f2963e00700 1 -- 172.21.15.1:0/3668371286 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f295c01c3f0 con 0x7f296c102f50 2024-02-28T17:02:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.090+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c1102f0 con 0x7f296c102f50 2024-02-28T17:02:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.091+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c0fc6e0 con 0x7f296c102f50 2024-02-28T17:02:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.092+0000 7f2963e00700 1 -- 172.21.15.1:0/3668371286 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f295c00bd60 con 0x7f296c102f50 2024-02-28T17:02:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.093+0000 7f2963e00700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f295006d7f0 0x7f295006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.093+0000 7f2963e00700 1 -- 172.21.15.1:0/3668371286 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f295c08e100 con 0x7f296c102f50 2024-02-28T17:02:26.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.093+0000 7f296b400700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f295006d7f0 0x7f295006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.094+0000 7f296b400700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f295006d7f0 0x7f295006fca0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2958007fd0 tx=0x7f2958009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:26.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.098+0000 7f2963e00700 1 -- 172.21.15.1:0/3668371286 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f295c05b5a0 con 0x7f296c102f50 2024-02-28T17:02:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.335+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f296c1112c0 con 0x7f295006d7f0 2024-02-28T17:02:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.337+0000 7f2963e00700 1 -- 172.21.15.1:0/3668371286 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f296c1112c0 con 0x7f295006d7f0 2024-02-28T17:02:26.338 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:02:26.338 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:02:26.338 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:02:26.338 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:02:26.338 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:02:26.338 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:02:26.338 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:02:26.339 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:02:26.339 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:02:26.339 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:02:26.339 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:02:26.339 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:02:26.339 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:02:26.339 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:02:26.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f295006d7f0 msgr2=0x7f295006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f295006d7f0 0x7f295006fca0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2958007fd0 tx=0x7f2958009040 comp rx=0 tx=0).stop 2024-02-28T17:02:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 msgr2=0x7f296c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 0x7f296c115c70 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f295c009fd0 tx=0x7f295c009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 shutdown_connections 2024-02-28T17:02:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f295006d7f0 0x7f295006fca0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f296c102f50 0x7f296c115c70 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 --2- 172.21.15.1:0/3668371286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f296c1038f0 0x7f296c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.340+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 >> 172.21.15.1:0/3668371286 conn(0x7f296c0f8ae0 msgr2=0x7f296c070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.341+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 shutdown_connections 2024-02-28T17:02:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.341+0000 7f2972e00700 1 -- 172.21.15.1:0/3668371286 wait complete. 2024-02-28T17:02:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.495+0000 7f3c0d000700 1 -- 172.21.15.1:0/3814674107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 msgr2=0x7f3c081099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.495+0000 7f3c0d000700 1 --2- 172.21.15.1:0/3814674107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 0x7f3c081099c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8004940 tx=0x7f3bf8009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.495+0000 7f3c0d000700 1 -- 172.21.15.1:0/3814674107 shutdown_connections 2024-02-28T17:02:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.495+0000 7f3c0d000700 1 --2- 172.21.15.1:0/3814674107 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c08103dc0 0x7f3c08104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.495+0000 7f3c0d000700 1 --2- 172.21.15.1:0/3814674107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 0x7f3c081099c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.495+0000 7f3c0d000700 1 -- 172.21.15.1:0/3814674107 >> 172.21.15.1:0/3814674107 conn(0x7f3c08072d50 msgr2=0x7f3c08075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.495+0000 7f3c0d000700 1 -- 172.21.15.1:0/3814674107 shutdown_connections 2024-02-28T17:02:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.495+0000 7f3c0d000700 1 -- 172.21.15.1:0/3814674107 wait complete. 2024-02-28T17:02:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.496+0000 7f3c0d000700 1 Processor -- start 2024-02-28T17:02:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.496+0000 7f3c0d000700 1 -- start start 2024-02-28T17:02:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c0d000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c08103dc0 0x7f3c0811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c0d000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 0x7f3c0811a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c0d000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0811ac70 con 0x7f3c081095f0 2024-02-28T17:02:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c0d000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08113ef0 con 0x7f3c08103dc0 2024-02-28T17:02:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c05400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 0x7f3c0811a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c05400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 0x7f3c0811a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51724/0 (socket says 172.21.15.1:51724) 2024-02-28T17:02:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c05400700 1 -- 172.21.15.1:0/3325569865 learned_addr learned my addr 172.21.15.1:0/3325569865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c05e00700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c08103dc0 0x7f3c0811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c05400700 1 -- 172.21.15.1:0/3325569865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c08103dc0 msgr2=0x7f3c0811a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c05400700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c08103dc0 0x7f3c0811a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.497+0000 7f3c05400700 1 -- 172.21.15.1:0/3325569865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8005040 con 0x7f3c081095f0 2024-02-28T17:02:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.498+0000 7f3c05e00700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c08103dc0 0x7f3c0811a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:02:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.498+0000 7f3c05400700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 0x7f3c0811a590 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400b790 tx=0x7f3bf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.498+0000 7f3bfe000700 1 -- 172.21.15.1:0/3325569865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bf400f770 con 0x7f3c081095f0 2024-02-28T17:02:26.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.498+0000 7f3bfe000700 1 -- 172.21.15.1:0/3325569865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3bf400fda0 con 0x7f3c081095f0 2024-02-28T17:02:26.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.498+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c08114170 con 0x7f3c081095f0 2024-02-28T17:02:26.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.498+0000 7f3bfe000700 1 -- 172.21.15.1:0/3325569865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bf400d310 con 0x7f3c081095f0 2024-02-28T17:02:26.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.498+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c081146b0 con 0x7f3c081095f0 2024-02-28T17:02:26.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.499+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c08075870 con 0x7f3c081095f0 2024-02-28T17:02:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.501+0000 7f3bfe000700 1 -- 172.21.15.1:0/3325569865 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3bf4010420 con 0x7f3c081095f0 2024-02-28T17:02:26.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.501+0000 7f3bfe000700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3bec06d7f0 0x7f3bec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:26.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.501+0000 7f3bfe000700 1 -- 172.21.15.1:0/3325569865 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3bf408ce30 con 0x7f3c081095f0 2024-02-28T17:02:26.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.502+0000 7f3c05e00700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3bec06d7f0 0x7f3bec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:26.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.502+0000 7f3c05e00700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3bec06d7f0 0x7f3bec06fca0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800b3b0 tx=0x7f3bf8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.506+0000 7f3bfe000700 1 -- 172.21.15.1:0/3325569865 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3bf405a2d0 con 0x7f3c081095f0 2024-02-28T17:02:26.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:26 smithi066 ceph-mon[85781]: pgmap v840: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:26.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:26 smithi066 ceph-mon[85781]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:26.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:26 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3408453519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:02:26.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:02:26.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:26 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/542802698' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:02:26.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:26 smithi001 ceph-mon[112940]: pgmap v840: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:26.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:26 smithi001 ceph-mon[112940]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:26.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:26 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3408453519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:02:26.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:02:26.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:26 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/542802698' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:02:26.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.818+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3c08115630 con 0x7f3c081095f0 2024-02-28T17:02:26.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.819+0000 7f3bfe000700 1 -- 172.21.15.1:0/3325569865 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f3bf4059e60 con 0x7f3c081095f0 2024-02-28T17:02:26.820 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:02:26.821 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:02:26.822 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:02:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3bec06d7f0 msgr2=0x7f3bec06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3bec06d7f0 0x7f3bec06fca0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800b3b0 tx=0x7f3bf8009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 msgr2=0x7f3c0811a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 0x7f3c0811a590 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400b790 tx=0x7f3bf4009250 comp rx=0 tx=0).stop 2024-02-28T17:02:26.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 shutdown_connections 2024-02-28T17:02:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3bec06d7f0 0x7f3bec06fca0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3c08103dc0 0x7f3c0811a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 --2- 172.21.15.1:0/3325569865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c081095f0 0x7f3c0811a590 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.822+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 >> 172.21.15.1:0/3325569865 conn(0x7f3c08072d50 msgr2=0x7f3c080746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.823+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 shutdown_connections 2024-02-28T17:02:26.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:26.823+0000 7f3c0d000700 1 -- 172.21.15.1:0/3325569865 wait complete. 2024-02-28T17:02:27.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:27 smithi066 ceph-mon[85781]: from='client.35534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:27.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:27 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3325569865' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:02:27.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:27 smithi001 ceph-mon[112940]: from='client.35534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:27.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:27 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3325569865' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:02:28.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:28 smithi066 ceph-mon[85781]: pgmap v841: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:28.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:28 smithi001 ceph-mon[112940]: pgmap v841: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:30.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:30 smithi066 ceph-mon[85781]: pgmap v842: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:30.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:30 smithi001 ceph-mon[112940]: pgmap v842: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:32.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:32 smithi066 ceph-mon[85781]: pgmap v843: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:32.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:32 smithi001 ceph-mon[112940]: pgmap v843: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:34.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:34 smithi066 ceph-mon[85781]: pgmap v844: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:34.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:34 smithi001 ceph-mon[112940]: pgmap v844: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:36.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:36 smithi066 ceph-mon[85781]: pgmap v845: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:36.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:36 smithi001 ceph-mon[112940]: pgmap v845: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:38.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:38 smithi066 ceph-mon[85781]: pgmap v846: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:38.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:38 smithi001 ceph-mon[112940]: pgmap v846: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:40.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:40 smithi066 ceph-mon[85781]: pgmap v847: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:40.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:40 smithi001 ceph-mon[112940]: pgmap v847: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:41.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:02:41.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:02:42.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:42 smithi066 ceph-mon[85781]: pgmap v848: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:42.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:42 smithi001 ceph-mon[112940]: pgmap v848: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:44.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:44 smithi066 ceph-mon[85781]: pgmap v849: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:44.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:44 smithi001 ceph-mon[112940]: pgmap v849: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:46.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:46 smithi066 ceph-mon[85781]: pgmap v850: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:46.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:46 smithi001 ceph-mon[112940]: pgmap v850: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:48.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:48 smithi066 ceph-mon[85781]: pgmap v851: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:48.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:48 smithi001 ceph-mon[112940]: pgmap v851: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:50.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:50 smithi066 ceph-mon[85781]: pgmap v852: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:50.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:50 smithi001 ceph-mon[112940]: pgmap v852: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:52.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:52 smithi066 ceph-mon[85781]: pgmap v853: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:52.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:52 smithi001 ceph-mon[112940]: pgmap v853: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:54.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:54 smithi066 ceph-mon[85781]: pgmap v854: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:54.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:54 smithi001 ceph-mon[112940]: pgmap v854: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:56.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:56 smithi066 ceph-mon[85781]: pgmap v855: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:56.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:02:56.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:56 smithi001 ceph-mon[112940]: pgmap v855: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:56.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:02:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:56.999+0000 7f317aa00700 1 -- 172.21.15.1:0/1355169936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 msgr2=0x7f31741099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:56.999+0000 7f317aa00700 1 --2- 172.21.15.1:0/1355169936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 0x7f31741099c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3164004940 tx=0x7f3164009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:56.999+0000 7f317aa00700 1 -- 172.21.15.1:0/1355169936 shutdown_connections 2024-02-28T17:02:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:56.999+0000 7f317aa00700 1 --2- 172.21.15.1:0/1355169936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3174103dc0 0x7f3174104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:56.999+0000 7f317aa00700 1 --2- 172.21.15.1:0/1355169936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 0x7f31741099c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:56.999+0000 7f317aa00700 1 -- 172.21.15.1:0/1355169936 >> 172.21.15.1:0/1355169936 conn(0x7f3174072d50 msgr2=0x7f3174075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.000+0000 7f317aa00700 1 -- 172.21.15.1:0/1355169936 shutdown_connections 2024-02-28T17:02:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.000+0000 7f317aa00700 1 -- 172.21.15.1:0/1355169936 wait complete. 2024-02-28T17:02:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.000+0000 7f317aa00700 1 Processor -- start 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.001+0000 7f317aa00700 1 -- start start 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.001+0000 7f317aa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3174103dc0 0x7f317411a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.001+0000 7f317aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 0x7f317411a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.001+0000 7f317aa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317411ac70 con 0x7f31741095f0 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.001+0000 7f317aa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3174113f80 con 0x7f3174103dc0 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.001+0000 7f3173400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 0x7f317411a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.001+0000 7f3173400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 0x7f317411a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47010/0 (socket says 172.21.15.1:47010) 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.002+0000 7f3173400700 1 -- 172.21.15.1:0/638733311 learned_addr learned my addr 172.21.15.1:0/638733311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.002+0000 7f3173e00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3174103dc0 0x7f317411a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.002+0000 7f3173400700 1 -- 172.21.15.1:0/638733311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3174103dc0 msgr2=0x7f317411a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.002+0000 7f3173400700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3174103dc0 0x7f317411a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.002+0000 7f3173400700 1 -- 172.21.15.1:0/638733311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3164005040 con 0x7f31741095f0 2024-02-28T17:02:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.002+0000 7f3173e00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3174103dc0 0x7f317411a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:02:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.002+0000 7f3173400700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 0x7f317411a620 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f316000c8c0 tx=0x7f31600044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.003+0000 7f316be00700 1 -- 172.21.15.1:0/638733311 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f316000a980 con 0x7f31741095f0 2024-02-28T17:02:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.003+0000 7f316be00700 1 -- 172.21.15.1:0/638733311 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f316000ce60 con 0x7f31741095f0 2024-02-28T17:02:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.003+0000 7f316be00700 1 -- 172.21.15.1:0/638733311 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31600115e0 con 0x7f31741095f0 2024-02-28T17:02:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.003+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3174114260 con 0x7f31741095f0 2024-02-28T17:02:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.003+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31741147a0 con 0x7f31741095f0 2024-02-28T17:02:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.004+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3174106850 con 0x7f31741095f0 2024-02-28T17:02:57.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.006+0000 7f316be00700 1 -- 172.21.15.1:0/638733311 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3160012400 con 0x7f31741095f0 2024-02-28T17:02:57.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.007+0000 7f316be00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f315806d7f0 0x7f315806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.007+0000 7f316be00700 1 -- 172.21.15.1:0/638733311 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f316008cf60 con 0x7f31741095f0 2024-02-28T17:02:57.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.007+0000 7f3173e00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f315806d7f0 0x7f315806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.008+0000 7f3173e00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f315806d7f0 0x7f315806fca0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3164009fd0 tx=0x7f3164009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:57.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.012+0000 7f316be00700 1 -- 172.21.15.1:0/638733311 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f316005a340 con 0x7f31741095f0 2024-02-28T17:02:57.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.257+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f317404bf40 con 0x7f315806d7f0 2024-02-28T17:02:57.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.259+0000 7f316be00700 1 -- 172.21.15.1:0/638733311 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f317404bf40 con 0x7f315806d7f0 2024-02-28T17:02:57.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f315806d7f0 msgr2=0x7f315806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f315806d7f0 0x7f315806fca0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3164009fd0 tx=0x7f3164009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:57.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 msgr2=0x7f317411a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 0x7f317411a620 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f316000c8c0 tx=0x7f31600044c0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 shutdown_connections 2024-02-28T17:02:57.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f315806d7f0 0x7f315806fca0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3174103dc0 0x7f317411a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 --2- 172.21.15.1:0/638733311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31741095f0 0x7f317411a620 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 >> 172.21.15.1:0/638733311 conn(0x7f3174072d50 msgr2=0x7f3174074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 shutdown_connections 2024-02-28T17:02:57.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.262+0000 7f317aa00700 1 -- 172.21.15.1:0/638733311 wait complete. 2024-02-28T17:02:57.278 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:02:57.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.409+0000 7fd922c00700 1 -- 172.21.15.1:0/870516516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 msgr2=0x7fd91c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.409+0000 7fd922c00700 1 --2- 172.21.15.1:0/870516516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 0x7fd91c0fcf60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd908004940 tx=0x7fd908009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:57.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.409+0000 7fd922c00700 1 -- 172.21.15.1:0/870516516 shutdown_connections 2024-02-28T17:02:57.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.409+0000 7fd922c00700 1 --2- 172.21.15.1:0/870516516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 0x7fd91c0fcf60 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.409+0000 7fd922c00700 1 --2- 172.21.15.1:0/870516516 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd91c102f50 0x7fd91c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.409+0000 7fd922c00700 1 -- 172.21.15.1:0/870516516 >> 172.21.15.1:0/870516516 conn(0x7fd91c0f8ae0 msgr2=0x7fd91c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:57.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.409+0000 7fd922c00700 1 -- 172.21.15.1:0/870516516 shutdown_connections 2024-02-28T17:02:57.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.409+0000 7fd922c00700 1 -- 172.21.15.1:0/870516516 wait complete. 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.410+0000 7fd922c00700 1 Processor -- start 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.410+0000 7fd922c00700 1 -- start start 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.411+0000 7fd922c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd91c102f50 0x7fd91c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.411+0000 7fd922c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 0x7fd91c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.411+0000 7fd922c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c116810 con 0x7fd91c1038f0 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.411+0000 7fd922c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c10fb40 con 0x7fd91c102f50 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.411+0000 7fd91b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 0x7fd91c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.411+0000 7fd91b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 0x7fd91c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47030/0 (socket says 172.21.15.1:47030) 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.411+0000 7fd91b400700 1 -- 172.21.15.1:0/3904595938 learned_addr learned my addr 172.21.15.1:0/3904595938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:57.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.411+0000 7fd91be00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd91c102f50 0x7fd91c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd91b400700 1 -- 172.21.15.1:0/3904595938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd91c102f50 msgr2=0x7fd91c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd91b400700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd91c102f50 0x7fd91c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd91b400700 1 -- 172.21.15.1:0/3904595938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd908005040 con 0x7fd91c1038f0 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd91be00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd91c102f50 0x7fd91c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd91b400700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 0x7fd91c1161c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd908009030 tx=0x7fd908009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd913e00700 1 -- 172.21.15.1:0/3904595938 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd90800e040 con 0x7fd91c1038f0 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd913e00700 1 -- 172.21.15.1:0/3904595938 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd90800bbf0 con 0x7fd91c1038f0 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd913e00700 1 -- 172.21.15.1:0/3904595938 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd90801c630 con 0x7fd91c1038f0 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c10fdc0 con 0x7fd91c1038f0 2024-02-28T17:02:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.412+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c110300 con 0x7fd91c1038f0 2024-02-28T17:02:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.414+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c0fc6e0 con 0x7fd91c1038f0 2024-02-28T17:02:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.414+0000 7fd913e00700 1 -- 172.21.15.1:0/3904595938 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd90800bd60 con 0x7fd91c1038f0 2024-02-28T17:02:57.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.415+0000 7fd913e00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd90006d7f0 0x7fd90006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.415+0000 7fd913e00700 1 -- 172.21.15.1:0/3904595938 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd90808e130 con 0x7fd91c1038f0 2024-02-28T17:02:57.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.416+0000 7fd91be00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd90006d7f0 0x7fd90006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.416+0000 7fd91be00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd90006d7f0 0x7fd90006fca0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009fd0 tx=0x7fd90c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.420+0000 7fd913e00700 1 -- 172.21.15.1:0/3904595938 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd90805b5d0 con 0x7fd91c1038f0 2024-02-28T17:02:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.661+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd91c1111f0 con 0x7fd90006d7f0 2024-02-28T17:02:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.664+0000 7fd913e00700 1 -- 172.21.15.1:0/3904595938 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd91c1111f0 con 0x7fd90006d7f0 2024-02-28T17:02:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd90006d7f0 msgr2=0x7fd90006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd90006d7f0 0x7fd90006fca0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009fd0 tx=0x7fd90c00b040 comp rx=0 tx=0).stop 2024-02-28T17:02:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 msgr2=0x7fd91c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 0x7fd91c1161c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd908009030 tx=0x7fd908009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:57.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 shutdown_connections 2024-02-28T17:02:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd90006d7f0 0x7fd90006fca0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd91c102f50 0x7fd91c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 --2- 172.21.15.1:0/3904595938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd91c1038f0 0x7fd91c1161c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 >> 172.21.15.1:0/3904595938 conn(0x7fd91c0f8ae0 msgr2=0x7fd91c0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 shutdown_connections 2024-02-28T17:02:57.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.667+0000 7fd922c00700 1 -- 172.21.15.1:0/3904595938 wait complete. 2024-02-28T17:02:57.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.820+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2038937362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 msgr2=0x7fe8ec065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.820+0000 7fe8f0a00700 1 --2- 172.21.15.1:0/2038937362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 0x7fe8ec065bd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8004940 tx=0x7fe8d8009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:57.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.820+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2038937362 shutdown_connections 2024-02-28T17:02:57.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.820+0000 7fe8f0a00700 1 --2- 172.21.15.1:0/2038937362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 0x7fe8ec065bd0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.820+0000 7fe8f0a00700 1 --2- 172.21.15.1:0/2038937362 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8ec102f30 0x7fe8ec103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.820+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2038937362 >> 172.21.15.1:0/2038937362 conn(0x7fe8ec071d50 msgr2=0x7fe8ec072150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:57.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.820+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2038937362 shutdown_connections 2024-02-28T17:02:57.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.820+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2038937362 wait complete. 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.821+0000 7fe8f0a00700 1 Processor -- start 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.821+0000 7fe8f0a00700 1 -- start start 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.821+0000 7fe8f0a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 0x7fe8ec115b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.821+0000 7fe8f0a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8ec102f30 0x7fe8ec1160b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.821+0000 7fe8f0a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec116700 con 0x7fe8ec065760 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.821+0000 7fe8f0a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec10fa30 con 0x7fe8ec102f30 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.821+0000 7fe8e9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 0x7fe8ec115b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.822+0000 7fe8e9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 0x7fe8ec115b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47054/0 (socket says 172.21.15.1:47054) 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.822+0000 7fe8e9e00700 1 -- 172.21.15.1:0/2228749920 learned_addr learned my addr 172.21.15.1:0/2228749920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.822+0000 7fe8e9e00700 1 -- 172.21.15.1:0/2228749920 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8ec102f30 msgr2=0x7fe8ec1160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.822+0000 7fe8e9400700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8ec102f30 0x7fe8ec1160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.822+0000 7fe8e9e00700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8ec102f30 0x7fe8ec1160b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.822+0000 7fe8e9e00700 1 -- 172.21.15.1:0/2228749920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d8005040 con 0x7fe8ec065760 2024-02-28T17:02:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.822+0000 7fe8e9e00700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 0x7fe8ec115b70 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc002a50 tx=0x7fe8dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.822+0000 7fe8e2000700 1 -- 172.21.15.1:0/2228749920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8dc007cc0 con 0x7fe8ec065760 2024-02-28T17:02:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.823+0000 7fe8e2000700 1 -- 172.21.15.1:0/2228749920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8dc00f040 con 0x7fe8ec065760 2024-02-28T17:02:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.823+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ec10fd10 con 0x7fe8ec065760 2024-02-28T17:02:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.823+0000 7fe8e2000700 1 -- 172.21.15.1:0/2228749920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8dc009640 con 0x7fe8ec065760 2024-02-28T17:02:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.823+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ec1102d0 con 0x7fe8ec065760 2024-02-28T17:02:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.824+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ec075ca0 con 0x7fe8ec065760 2024-02-28T17:02:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.825+0000 7fe8e2000700 1 -- 172.21.15.1:0/2228749920 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe8dc0097a0 con 0x7fe8ec065760 2024-02-28T17:02:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.826+0000 7fe8e2000700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8d006d7f0 0x7fe8d006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:57.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.826+0000 7fe8e2000700 1 -- 172.21.15.1:0/2228749920 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe8dc08cc80 con 0x7fe8ec065760 2024-02-28T17:02:57.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.826+0000 7fe8e9400700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8d006d7f0 0x7fe8d006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:57.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.827+0000 7fe8e9400700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8d006d7f0 0x7fe8d006fca0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800b3b0 tx=0x7fe8d8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:57.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:57.830+0000 7fe8e2000700 1 -- 172.21.15.1:0/2228749920 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe8dc05a000 con 0x7fe8ec065760 2024-02-28T17:02:58.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.056+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8ec1104f0 con 0x7fe8d006d7f0 2024-02-28T17:02:58.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.085+0000 7fe8e2000700 1 -- 172.21.15.1:0/2228749920 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe8ec1104f0 con 0x7fe8d006d7f0 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (29m) 5m ago 43m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (25m) 5m ago 43m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (25m) 5m ago 42m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (28m) 5m ago 42m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (21m) 5m ago 36m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (21m) 5m ago 36m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (36m) 5m ago 36m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (5m) 5m ago 36m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (31m) 5m ago 45m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (30m) 5m ago 42m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:02:58.086 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (27m) 5m ago 45m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (26m) 5m ago 41m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (30m) 5m ago 43m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (30m) 5m ago 42m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (25m) 5m ago 41m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (24m) 5m ago 40m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (24m) 5m ago 39m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (23m) 5m ago 39m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (22m) 5m ago 38m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (22m) 5m ago 38m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:02:58.087 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (29m) 5m ago 42m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:02:58.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8d006d7f0 msgr2=0x7fe8d006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8d006d7f0 0x7fe8d006fca0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800b3b0 tx=0x7fe8d8009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:58.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 msgr2=0x7fe8ec115b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 0x7fe8ec115b70 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc002a50 tx=0x7fe8dc0046b0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 shutdown_connections 2024-02-28T17:02:58.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8d006d7f0 0x7fe8d006fca0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8ec065760 0x7fe8ec115b70 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 --2- 172.21.15.1:0/2228749920 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe8ec102f30 0x7fe8ec1160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 >> 172.21.15.1:0/2228749920 conn(0x7fe8ec071d50 msgr2=0x7fe8ec0faac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:58.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.088+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 shutdown_connections 2024-02-28T17:02:58.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.089+0000 7fe8f0a00700 1 -- 172.21.15.1:0/2228749920 wait complete. 2024-02-28T17:02:58.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.229+0000 7ffa1fe00700 1 -- 172.21.15.1:0/1437545322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa20102d40 msgr2=0x7ffa20103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.229+0000 7ffa1fe00700 1 --2- 172.21.15.1:0/1437545322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa20102d40 0x7ffa20103110 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ffa10004940 tx=0x7ffa10009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:58.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.229+0000 7ffa1fe00700 1 -- 172.21.15.1:0/1437545322 shutdown_connections 2024-02-28T17:02:58.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.229+0000 7ffa1fe00700 1 --2- 172.21.15.1:0/1437545322 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa201036e0 0x7ffa200fcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.229+0000 7ffa1fe00700 1 --2- 172.21.15.1:0/1437545322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa20102d40 0x7ffa20103110 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.229+0000 7ffa1fe00700 1 -- 172.21.15.1:0/1437545322 >> 172.21.15.1:0/1437545322 conn(0x7ffa200f88b0 msgr2=0x7ffa200facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:58.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.229+0000 7ffa1fe00700 1 -- 172.21.15.1:0/1437545322 shutdown_connections 2024-02-28T17:02:58.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.229+0000 7ffa1fe00700 1 -- 172.21.15.1:0/1437545322 wait complete. 2024-02-28T17:02:58.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.230+0000 7ffa1fe00700 1 Processor -- start 2024-02-28T17:02:58.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.230+0000 7ffa1fe00700 1 -- start start 2024-02-28T17:02:58.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.230+0000 7ffa1fe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20102d40 0x7ffa20115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:58.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.230+0000 7ffa1fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201036e0 0x7ffa20115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:58.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.230+0000 7ffa1fe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20116620 con 0x7ffa201036e0 2024-02-28T17:02:58.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.230+0000 7ffa1fe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2010f950 con 0x7ffa20102d40 2024-02-28T17:02:58.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.231+0000 7ffa17e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201036e0 0x7ffa20115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:58.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.231+0000 7ffa17e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201036e0 0x7ffa20115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47086/0 (socket says 172.21.15.1:47086) 2024-02-28T17:02:58.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.231+0000 7ffa17e00700 1 -- 172.21.15.1:0/3002151923 learned_addr learned my addr 172.21.15.1:0/3002151923 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.231+0000 7ffa17e00700 1 -- 172.21.15.1:0/3002151923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20102d40 msgr2=0x7ffa20115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.231+0000 7ffa17e00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20102d40 0x7ffa20115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.231+0000 7ffa17e00700 1 -- 172.21.15.1:0/3002151923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10005040 con 0x7ffa201036e0 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.231+0000 7ffa17e00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201036e0 0x7ffa20115fd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ffa08002a50 tx=0x7ffa08004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.232+0000 7ffa16a00700 1 -- 172.21.15.1:0/3002151923 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa08007cc0 con 0x7ffa201036e0 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.232+0000 7ffa16a00700 1 -- 172.21.15.1:0/3002151923 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa0800f040 con 0x7ffa201036e0 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.232+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2010fc30 con 0x7ffa201036e0 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.232+0000 7ffa16a00700 1 -- 172.21.15.1:0/3002151923 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa0800da40 con 0x7ffa201036e0 2024-02-28T17:02:58.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.232+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa20110170 con 0x7ffa201036e0 2024-02-28T17:02:58.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.233+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2005f5c0 con 0x7ffa201036e0 2024-02-28T17:02:58.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.234+0000 7ffa16a00700 1 -- 172.21.15.1:0/3002151923 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffa080097c0 con 0x7ffa201036e0 2024-02-28T17:02:58.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.235+0000 7ffa16a00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa0406d7f0 0x7ffa0406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:58.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.235+0000 7ffa16a00700 1 -- 172.21.15.1:0/3002151923 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffa0808cca0 con 0x7ffa201036e0 2024-02-28T17:02:58.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.236+0000 7ffa1ea00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa0406d7f0 0x7ffa0406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:58.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.236+0000 7ffa1ea00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa0406d7f0 0x7ffa0406fca0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ffa10004f90 tx=0x7ffa10009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:58.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.240+0000 7ffa16a00700 1 -- 172.21.15.1:0/3002151923 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffa08059d70 con 0x7ffa201036e0 2024-02-28T17:02:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.553+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffa2004bed0 con 0x7ffa201036e0 2024-02-28T17:02:58.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.554+0000 7ffa16a00700 1 -- 172.21.15.1:0/3002151923 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7ffa08059b90 con 0x7ffa201036e0 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:02:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:02:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:02:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:02:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:02:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:02:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:02:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:02:58.555 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:02:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa0406d7f0 msgr2=0x7ffa0406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa0406d7f0 0x7ffa0406fca0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ffa10004f90 tx=0x7ffa10009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:58.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201036e0 msgr2=0x7ffa20115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201036e0 0x7ffa20115fd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ffa08002a50 tx=0x7ffa08004600 comp rx=0 tx=0).stop 2024-02-28T17:02:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 shutdown_connections 2024-02-28T17:02:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa0406d7f0 0x7ffa0406fca0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa20102d40 0x7ffa20115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 --2- 172.21.15.1:0/3002151923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201036e0 0x7ffa20115fd0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.556+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 >> 172.21.15.1:0/3002151923 conn(0x7ffa200f88b0 msgr2=0x7ffa200705d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.557+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 shutdown_connections 2024-02-28T17:02:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.557+0000 7ffa1fe00700 1 -- 172.21.15.1:0/3002151923 wait complete. 2024-02-28T17:02:58.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.709+0000 7fe03b200700 1 -- 172.21.15.1:0/629741497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 msgr2=0x7fe034105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.709+0000 7fe03b200700 1 --2- 172.21.15.1:0/629741497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 0x7fe034105540 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe024004940 tx=0x7fe024009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:58.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.709+0000 7fe03b200700 1 -- 172.21.15.1:0/629741497 shutdown_connections 2024-02-28T17:02:58.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.709+0000 7fe03b200700 1 --2- 172.21.15.1:0/629741497 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe034105b10 0x7fe0340ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.709+0000 7fe03b200700 1 --2- 172.21.15.1:0/629741497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 0x7fe034105540 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.709+0000 7fe03b200700 1 -- 172.21.15.1:0/629741497 >> 172.21.15.1:0/629741497 conn(0x7fe0340face0 msgr2=0x7fe0340fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:58.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.709+0000 7fe03b200700 1 -- 172.21.15.1:0/629741497 shutdown_connections 2024-02-28T17:02:58.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.709+0000 7fe03b200700 1 -- 172.21.15.1:0/629741497 wait complete. 2024-02-28T17:02:58.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe03b200700 1 Processor -- start 2024-02-28T17:02:58.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe03b200700 1 -- start start 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe03b200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 0x7fe034115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe03b200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe034105b10 0x7fe034116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe03b200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0341168e0 con 0x7fe034105170 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe03b200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03410fbf0 con 0x7fe034105b10 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe033e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 0x7fe034115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe033e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 0x7fe034115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47110/0 (socket says 172.21.15.1:47110) 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.710+0000 7fe033e00700 1 -- 172.21.15.1:0/347447912 learned_addr learned my addr 172.21.15.1:0/347447912 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe02be00700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe034105b10 0x7fe034116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe033e00700 1 -- 172.21.15.1:0/347447912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe034105b10 msgr2=0x7fe034116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe033e00700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe034105b10 0x7fe034116290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe033e00700 1 -- 172.21.15.1:0/347447912 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe024005040 con 0x7fe034105170 2024-02-28T17:02:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe033e00700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 0x7fe034115d50 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe024009150 tx=0x7fe024009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:58.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe030c00700 1 -- 172.21.15.1:0/347447912 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe02400e040 con 0x7fe034105170 2024-02-28T17:02:58.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe030c00700 1 -- 172.21.15.1:0/347447912 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe02400bbf0 con 0x7fe034105170 2024-02-28T17:02:58.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe030c00700 1 -- 172.21.15.1:0/347447912 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe02401c630 con 0x7fe034105170 2024-02-28T17:02:58.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03410fe70 con 0x7fe034105170 2024-02-28T17:02:58.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.711+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0341103b0 con 0x7fe034105170 2024-02-28T17:02:58.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.712+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe034102480 con 0x7fe034105170 2024-02-28T17:02:58.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.714+0000 7fe030c00700 1 -- 172.21.15.1:0/347447912 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe02400bd60 con 0x7fe034105170 2024-02-28T17:02:58.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.714+0000 7fe030c00700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe01806d7f0 0x7fe01806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:58.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.714+0000 7fe030c00700 1 -- 172.21.15.1:0/347447912 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe02408e290 con 0x7fe034105170 2024-02-28T17:02:58.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.715+0000 7fe02be00700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe01806d7f0 0x7fe01806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.715+0000 7fe02be00700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe01806d7f0 0x7fe01806fca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe0341115f0 tx=0x7fe01c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:58.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.719+0000 7fe030c00700 1 -- 172.21.15.1:0/347447912 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe02405b670 con 0x7fe034105170 2024-02-28T17:02:58.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:58 smithi066 ceph-mon[85781]: pgmap v856: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:58.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:58 smithi066 ceph-mon[85781]: from='client.35542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:58.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:58 smithi066 ceph-mon[85781]: from='client.35546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:58.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:58 smithi001 ceph-mon[112940]: pgmap v856: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:02:58.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:58 smithi001 ceph-mon[112940]: from='client.35542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:58.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:58 smithi001 ceph-mon[112940]: from='client.35546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:58.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.979+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe03404bed0 con 0x7fe034105170 2024-02-28T17:02:58.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.980+0000 7fe030c00700 1 -- 172.21.15.1:0/347447912 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fe02405b200 con 0x7fe034105170 2024-02-28T17:02:58.981 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:02:58.981 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:02:58.981 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:02:58.981 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:02:58.981 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:02:58.982 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:02:58.982 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:02:58.982 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:02:58.982 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:02:58.982 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:02:58.982 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:02:58.983 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:02:58.983 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:02:58.983 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:02:58.983 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:02:58.983 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:02:58.983 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:02:58.984 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:02:58.984 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:02:58.984 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:02:58.984 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:02:58.984 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:02:58.984 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:02:58.985 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:02:58.985 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:02:58.985 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:02:58.985 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:02:58.985 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:02:58.985 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:02:58.985 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:02:58.986 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:02:58.986 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:02:58.986 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:02:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:02:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:02:58.986 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:02:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:02:58.987 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:02:58.987 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:02:58.987 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:02:58.987 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:02:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe01806d7f0 msgr2=0x7fe01806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe01806d7f0 0x7fe01806fca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe0341115f0 tx=0x7fe01c009040 comp rx=0 tx=0).stop 2024-02-28T17:02:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 msgr2=0x7fe034115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 0x7fe034115d50 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe024009150 tx=0x7fe024009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 shutdown_connections 2024-02-28T17:02:58.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe01806d7f0 0x7fe01806fca0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe034105170 0x7fe034115d50 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 --2- 172.21.15.1:0/347447912 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe034105b10 0x7fe034116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:58.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.984+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 >> 172.21.15.1:0/347447912 conn(0x7fe0340face0 msgr2=0x7fe0340fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:58.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.985+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 shutdown_connections 2024-02-28T17:02:58.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:58.985+0000 7fe03b200700 1 -- 172.21.15.1:0/347447912 wait complete. 2024-02-28T17:02:58.990 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:02:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.145+0000 7f8eae800700 1 -- 172.21.15.1:0/2314126911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea81095f0 msgr2=0x7f8ea81099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:59.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.145+0000 7f8eae800700 1 --2- 172.21.15.1:0/2314126911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea81095f0 0x7f8ea81099c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f8e98004940 tx=0x7f8e98009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:59.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.145+0000 7f8eae800700 1 -- 172.21.15.1:0/2314126911 shutdown_connections 2024-02-28T17:02:59.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.145+0000 7f8eae800700 1 --2- 172.21.15.1:0/2314126911 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea8103dc0 0x7f8ea8104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.145+0000 7f8eae800700 1 --2- 172.21.15.1:0/2314126911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea81095f0 0x7f8ea81099c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.145+0000 7f8eae800700 1 -- 172.21.15.1:0/2314126911 >> 172.21.15.1:0/2314126911 conn(0x7f8ea8072d50 msgr2=0x7f8ea8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:59.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.145+0000 7f8eae800700 1 -- 172.21.15.1:0/2314126911 shutdown_connections 2024-02-28T17:02:59.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.145+0000 7f8eae800700 1 -- 172.21.15.1:0/2314126911 wait complete. 2024-02-28T17:02:59.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.146+0000 7f8eae800700 1 Processor -- start 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.146+0000 7f8eae800700 1 -- start start 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.147+0000 7f8eae800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea8103dc0 0x7f8ea811a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.147+0000 7f8eae800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea81095f0 0x7f8ea811a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.147+0000 7f8eae800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea811ac70 con 0x7f8ea8103dc0 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.147+0000 7f8eae800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8113f80 con 0x7f8ea81095f0 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.147+0000 7f8ea7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea8103dc0 0x7f8ea811a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.147+0000 7f8ea7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea8103dc0 0x7f8ea811a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47134/0 (socket says 172.21.15.1:47134) 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.147+0000 7f8ea7400700 1 -- 172.21.15.1:0/425251949 learned_addr learned my addr 172.21.15.1:0/425251949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.147+0000 7f8ea6a00700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea81095f0 0x7f8ea811a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:59.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8ea7400700 1 -- 172.21.15.1:0/425251949 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea81095f0 msgr2=0x7f8ea811a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8ea7400700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea81095f0 0x7f8ea811a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8ea7400700 1 -- 172.21.15.1:0/425251949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e98005040 con 0x7f8ea8103dc0 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8ea6a00700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea81095f0 0x7f8ea811a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8ea7400700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea8103dc0 0x7f8ea811a0e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009150 tx=0x7f8e98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8e9f400700 1 -- 172.21.15.1:0/425251949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9800e040 con 0x7f8ea8103dc0 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8e9f400700 1 -- 172.21.15.1:0/425251949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e9800bbf0 con 0x7f8ea8103dc0 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea8114200 con 0x7f8ea8103dc0 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8e9f400700 1 -- 172.21.15.1:0/425251949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9801c630 con 0x7f8ea8103dc0 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.148+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea8114740 con 0x7f8ea8103dc0 2024-02-28T17:02:59.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.149+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea8106850 con 0x7f8ea8103dc0 2024-02-28T17:02:59.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.151+0000 7f8e9f400700 1 -- 172.21.15.1:0/425251949 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8e9800bd60 con 0x7f8ea8103dc0 2024-02-28T17:02:59.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.152+0000 7f8e9f400700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e8c06d7f0 0x7f8e8c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:59.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.152+0000 7f8e9f400700 1 -- 172.21.15.1:0/425251949 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8e9808e290 con 0x7f8ea8103dc0 2024-02-28T17:02:59.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.152+0000 7f8ea6a00700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e8c06d7f0 0x7f8e8c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:59.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.153+0000 7f8ea6a00700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e8c06d7f0 0x7f8e8c06fca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8e94007fd0 tx=0x7f8e94009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:59.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.156+0000 7f8e9f400700 1 -- 172.21.15.1:0/425251949 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8e9805b670 con 0x7f8ea8103dc0 2024-02-28T17:02:59.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.410+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ea804bf40 con 0x7f8e8c06d7f0 2024-02-28T17:02:59.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.412+0000 7f8e9f400700 1 -- 172.21.15.1:0/425251949 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8ea804bf40 con 0x7f8e8c06d7f0 2024-02-28T17:02:59.412 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:02:59.413 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.415+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e8c06d7f0 msgr2=0x7f8e8c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.415+0000 7f8eae800700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e8c06d7f0 0x7f8e8c06fca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8e94007fd0 tx=0x7f8e94009040 comp rx=0 tx=0).stop 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.415+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea8103dc0 msgr2=0x7f8ea811a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.415+0000 7f8eae800700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea8103dc0 0x7f8ea811a0e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009150 tx=0x7f8e98009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.415+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 shutdown_connections 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.415+0000 7f8eae800700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e8c06d7f0 0x7f8e8c06fca0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.415+0000 7f8eae800700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ea8103dc0 0x7f8ea811a0e0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.415+0000 7f8eae800700 1 --2- 172.21.15.1:0/425251949 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ea81095f0 0x7f8ea811a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.416+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 >> 172.21.15.1:0/425251949 conn(0x7f8ea8072d50 msgr2=0x7f8ea8074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.416+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 shutdown_connections 2024-02-28T17:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.416+0000 7f8eae800700 1 -- 172.21.15.1:0/425251949 wait complete. 2024-02-28T17:02:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.573+0000 7f57eac00700 1 -- 172.21.15.1:0/692776793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4071f60 msgr2=0x7f57e4070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.573+0000 7f57eac00700 1 --2- 172.21.15.1:0/692776793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4071f60 0x7f57e4070330 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f57d4004970 tx=0x7f57d4009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.573+0000 7f57eac00700 1 -- 172.21.15.1:0/692776793 shutdown_connections 2024-02-28T17:02:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.573+0000 7f57eac00700 1 --2- 172.21.15.1:0/692776793 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57e4070870 0x7f57e4070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.573+0000 7f57eac00700 1 --2- 172.21.15.1:0/692776793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4071f60 0x7f57e4070330 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.573+0000 7f57eac00700 1 -- 172.21.15.1:0/692776793 >> 172.21.15.1:0/692776793 conn(0x7f57e40f8ae0 msgr2=0x7f57e40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.573+0000 7f57eac00700 1 -- 172.21.15.1:0/692776793 shutdown_connections 2024-02-28T17:02:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.573+0000 7f57eac00700 1 -- 172.21.15.1:0/692776793 wait complete. 2024-02-28T17:02:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.574+0000 7f57eac00700 1 Processor -- start 2024-02-28T17:02:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.574+0000 7f57eac00700 1 -- start start 2024-02-28T17:02:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57eac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4070870 0x7f57e411a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57eac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57e4071f60 0x7f57e411a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57eac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e411ac00 con 0x7f57e4070870 2024-02-28T17:02:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57eac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e4113f10 con 0x7f57e4071f60 2024-02-28T17:02:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4070870 0x7f57e411a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4070870 0x7f57e411a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47152/0 (socket says 172.21.15.1:47152) 2024-02-28T17:02:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57e3e00700 1 -- 172.21.15.1:0/2375074470 learned_addr learned my addr 172.21.15.1:0/2375074470 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:02:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57e3e00700 1 -- 172.21.15.1:0/2375074470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57e4071f60 msgr2=0x7f57e411a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57e3400700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57e4071f60 0x7f57e411a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57e3e00700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57e4071f60 0x7f57e411a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.575+0000 7f57e3e00700 1 -- 172.21.15.1:0/2375074470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d4005040 con 0x7f57e4070870 2024-02-28T17:02:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.576+0000 7f57e3e00700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4070870 0x7f57e411a070 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f57d4004f50 tx=0x7f57d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:59.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.576+0000 7f57dbe00700 1 -- 172.21.15.1:0/2375074470 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57d400e040 con 0x7f57e4070870 2024-02-28T17:02:59.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.576+0000 7f57dbe00700 1 -- 172.21.15.1:0/2375074470 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57d400bc10 con 0x7f57e4070870 2024-02-28T17:02:59.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.576+0000 7f57dbe00700 1 -- 172.21.15.1:0/2375074470 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57d401c630 con 0x7f57e4070870 2024-02-28T17:02:59.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.576+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e4114190 con 0x7f57e4070870 2024-02-28T17:02:59.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.576+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e41146d0 con 0x7f57e4070870 2024-02-28T17:02:59.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.577+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e40fe400 con 0x7f57e4070870 2024-02-28T17:02:59.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.578+0000 7f57dbe00700 1 -- 172.21.15.1:0/2375074470 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f57d400bd80 con 0x7f57e4070870 2024-02-28T17:02:59.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.579+0000 7f57dbe00700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57c806d7f0 0x7f57c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:02:59.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.579+0000 7f57dbe00700 1 -- 172.21.15.1:0/2375074470 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f57d408e3b0 con 0x7f57e4070870 2024-02-28T17:02:59.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.579+0000 7f57e3400700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57c806d7f0 0x7f57c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:02:59.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.580+0000 7f57e3400700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57c806d7f0 0x7f57c806fca0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f57e4115910 tx=0x7f57d0009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:02:59.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.583+0000 7f57dbe00700 1 -- 172.21.15.1:0/2375074470 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f57d405b710 con 0x7f57e4070870 2024-02-28T17:02:59.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:59 smithi066 ceph-mon[85781]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:59.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:59 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3002151923' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:02:59.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:02:59 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/347447912' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:02:59.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:59 smithi001 ceph-mon[112940]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:02:59.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:59 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3002151923' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:02:59.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:02:59 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/347447912' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:02:59.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.887+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f57e404bed0 con 0x7f57e4070870 2024-02-28T17:02:59.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.888+0000 7f57dbe00700 1 -- 172.21.15.1:0/2375074470 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f57d405b2a0 con 0x7f57e4070870 2024-02-28T17:02:59.888 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:02:59.888 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:02:59.888 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:02:59.889 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:02:59.890 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:02:59.890 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:02:59.890 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:02:59.890 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:02:59.890 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:02:59.890 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:02:59.890 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:02:59.890 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.890+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57c806d7f0 msgr2=0x7f57c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.890+0000 7f57eac00700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57c806d7f0 0x7f57c806fca0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f57e4115910 tx=0x7f57d0009040 comp rx=0 tx=0).stop 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.890+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4070870 msgr2=0x7f57e411a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.890+0000 7f57eac00700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4070870 0x7f57e411a070 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f57d4004f50 tx=0x7f57d4009e30 comp rx=0 tx=0).stop 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.890+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 shutdown_connections 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.891+0000 7f57eac00700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57c806d7f0 0x7f57c806fca0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.891+0000 7f57eac00700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e4070870 0x7f57e411a070 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.891+0000 7f57eac00700 1 --2- 172.21.15.1:0/2375074470 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f57e4071f60 0x7f57e411a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.891+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 >> 172.21.15.1:0/2375074470 conn(0x7f57e40f8ae0 msgr2=0x7f57e40fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.891+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 shutdown_connections 2024-02-28T17:02:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:02:59.891+0000 7f57eac00700 1 -- 172.21.15.1:0/2375074470 wait complete. 2024-02-28T17:03:00.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:00 smithi066 ceph-mon[85781]: pgmap v857: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:00.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:00 smithi066 ceph-mon[85781]: from='client.35562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:00.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:00 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2375074470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:03:00.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:00 smithi001 ceph-mon[112940]: pgmap v857: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:00.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:00 smithi001 ceph-mon[112940]: from='client.35562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:00.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:00 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2375074470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:03:02.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:02 smithi066 ceph-mon[85781]: pgmap v858: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:02.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:02 smithi001 ceph-mon[112940]: pgmap v858: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:04.759 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:04 smithi066 ceph-mon[85781]: pgmap v859: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:04.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:04 smithi001 ceph-mon[112940]: pgmap v859: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:06.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:06 smithi001 ceph-mon[112940]: pgmap v860: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:07.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:06 smithi066 ceph-mon[85781]: pgmap v860: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:08.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:08 smithi001 ceph-mon[112940]: pgmap v861: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:09.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:08 smithi066 ceph-mon[85781]: pgmap v861: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:10.781 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:10 smithi001 ceph-mon[112940]: pgmap v862: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:10.812 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:10 smithi066 ceph-mon[85781]: pgmap v862: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:12.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:03:12.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:03:13.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:12 smithi066 ceph-mon[85781]: pgmap v863: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:13.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:12 smithi001 ceph-mon[112940]: pgmap v863: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:15.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:14 smithi066 ceph-mon[85781]: pgmap v864: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:15.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:14 smithi001 ceph-mon[112940]: pgmap v864: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:17.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:16 smithi066 ceph-mon[85781]: pgmap v865: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:17.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:16 smithi001 ceph-mon[112940]: pgmap v865: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:19.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:18 smithi066 ceph-mon[85781]: pgmap v866: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:19.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:18 smithi001 ceph-mon[112940]: pgmap v866: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:21.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:20 smithi066 ceph-mon[85781]: pgmap v867: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:21.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:20 smithi001 ceph-mon[112940]: pgmap v867: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:23.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:22 smithi066 ceph-mon[85781]: pgmap v868: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:23.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:22 smithi001 ceph-mon[112940]: pgmap v868: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:25.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:24 smithi066 ceph-mon[85781]: pgmap v869: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:25.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:24 smithi001 ceph-mon[112940]: pgmap v869: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:27.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:26 smithi066 ceph-mon[85781]: pgmap v870: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:27.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:03:27.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:26 smithi001 ceph-mon[112940]: pgmap v870: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:27.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:03:29.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:28 smithi066 ceph-mon[85781]: pgmap v871: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:29.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:28 smithi001 ceph-mon[112940]: pgmap v871: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:30.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.060+0000 7f4625c00700 1 -- 172.21.15.1:0/3484380933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 msgr2=0x7f4620109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.060+0000 7f4625c00700 1 --2- 172.21.15.1:0/3484380933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 0x7f4620109910 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4610004940 tx=0x7f4610009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:30.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.061+0000 7f4625c00700 1 -- 172.21.15.1:0/3484380933 shutdown_connections 2024-02-28T17:03:30.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.061+0000 7f4625c00700 1 --2- 172.21.15.1:0/3484380933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4620103dc0 0x7f4620104230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.061+0000 7f4625c00700 1 --2- 172.21.15.1:0/3484380933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 0x7f4620109910 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.061+0000 7f4625c00700 1 -- 172.21.15.1:0/3484380933 >> 172.21.15.1:0/3484380933 conn(0x7f4620072cf0 msgr2=0x7f4620075100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:30.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.061+0000 7f4625c00700 1 -- 172.21.15.1:0/3484380933 shutdown_connections 2024-02-28T17:03:30.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.061+0000 7f4625c00700 1 -- 172.21.15.1:0/3484380933 wait complete. 2024-02-28T17:03:30.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.062+0000 7f4625c00700 1 Processor -- start 2024-02-28T17:03:30.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.062+0000 7f4625c00700 1 -- start start 2024-02-28T17:03:30.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.062+0000 7f4625c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4620103dc0 0x7f462011a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.062+0000 7f4625c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 0x7f462011a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.062+0000 7f4625c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462011ab90 con 0x7f4620109540 2024-02-28T17:03:30.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.062+0000 7f4625c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4620113ea0 con 0x7f4620103dc0 2024-02-28T17:03:30.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.063+0000 7f4617e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 0x7f462011a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.063+0000 7f4617e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 0x7f462011a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49816/0 (socket says 172.21.15.1:49816) 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.063+0000 7f4617e00700 1 -- 172.21.15.1:0/3140547995 learned_addr learned my addr 172.21.15.1:0/3140547995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.063+0000 7f461ea00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4620103dc0 0x7f462011a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.063+0000 7f4617e00700 1 -- 172.21.15.1:0/3140547995 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4620103dc0 msgr2=0x7f462011a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.063+0000 7f4617e00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4620103dc0 0x7f462011a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.063+0000 7f4617e00700 1 -- 172.21.15.1:0/3140547995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610005040 con 0x7f4620109540 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.063+0000 7f4617e00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 0x7f462011a540 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4608002a50 tx=0x7f46080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.064+0000 7f4616a00700 1 -- 172.21.15.1:0/3140547995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4608007cc0 con 0x7f4620109540 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.064+0000 7f4616a00700 1 -- 172.21.15.1:0/3140547995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f460800f040 con 0x7f4620109540 2024-02-28T17:03:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.064+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4620114180 con 0x7f4620109540 2024-02-28T17:03:30.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.064+0000 7f4616a00700 1 -- 172.21.15.1:0/3140547995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4608009640 con 0x7f4620109540 2024-02-28T17:03:30.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.064+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46201146c0 con 0x7f4620109540 2024-02-28T17:03:30.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.065+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4620075810 con 0x7f4620109540 2024-02-28T17:03:30.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.066+0000 7f4616a00700 1 -- 172.21.15.1:0/3140547995 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f46080097a0 con 0x7f4620109540 2024-02-28T17:03:30.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.067+0000 7f4616a00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f460406d7f0 0x7f460406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.067+0000 7f4616a00700 1 -- 172.21.15.1:0/3140547995 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f460808cc60 con 0x7f4620109540 2024-02-28T17:03:30.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.067+0000 7f461ea00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f460406d7f0 0x7f460406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.068+0000 7f461ea00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f460406d7f0 0x7f460406fca0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f461000b020 tx=0x7f4610009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:30.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.072+0000 7f4616a00700 1 -- 172.21.15.1:0/3140547995 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4608059fc0 con 0x7f4620109540 2024-02-28T17:03:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.314+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f46201156f0 con 0x7f460406d7f0 2024-02-28T17:03:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.317+0000 7f4616a00700 1 -- 172.21.15.1:0/3140547995 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f46201156f0 con 0x7f460406d7f0 2024-02-28T17:03:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.319+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f460406d7f0 msgr2=0x7f460406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.319+0000 7f4625c00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f460406d7f0 0x7f460406fca0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f461000b020 tx=0x7f4610009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.319+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 msgr2=0x7f462011a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.319+0000 7f4625c00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 0x7f462011a540 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4608002a50 tx=0x7f46080046b0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.320+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 shutdown_connections 2024-02-28T17:03:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.320+0000 7f4625c00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f460406d7f0 0x7f460406fca0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.320+0000 7f4625c00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4620103dc0 0x7f462011a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.320+0000 7f4625c00700 1 --2- 172.21.15.1:0/3140547995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4620109540 0x7f462011a540 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.320+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 >> 172.21.15.1:0/3140547995 conn(0x7f4620072cf0 msgr2=0x7f4620074600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.320+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 shutdown_connections 2024-02-28T17:03:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.320+0000 7f4625c00700 1 -- 172.21.15.1:0/3140547995 wait complete. 2024-02-28T17:03:30.337 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:03:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.479+0000 7f8dd0400700 1 -- 172.21.15.1:0/199203893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 msgr2=0x7f8dc8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.479+0000 7f8dd0400700 1 --2- 172.21.15.1:0/199203893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 0x7f8dc8104210 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8db4004940 tx=0x7f8db4009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.480+0000 7f8dd0400700 1 -- 172.21.15.1:0/199203893 shutdown_connections 2024-02-28T17:03:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.480+0000 7f8dd0400700 1 --2- 172.21.15.1:0/199203893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 0x7f8dc8104210 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.480+0000 7f8dd0400700 1 --2- 172.21.15.1:0/199203893 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8dc81095f0 0x7f8dc81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.480+0000 7f8dd0400700 1 -- 172.21.15.1:0/199203893 >> 172.21.15.1:0/199203893 conn(0x7f8dc8072d50 msgr2=0x7f8dc8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.480+0000 7f8dd0400700 1 -- 172.21.15.1:0/199203893 shutdown_connections 2024-02-28T17:03:30.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.480+0000 7f8dd0400700 1 -- 172.21.15.1:0/199203893 wait complete. 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.481+0000 7f8dd0400700 1 Processor -- start 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.481+0000 7f8dd0400700 1 -- start start 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dd0400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 0x7f8dc811a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dd0400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8dc81095f0 0x7f8dc811a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dd0400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc811ac50 con 0x7f8dc8103dc0 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dd0400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8113ed0 con 0x7f8dc81095f0 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dcda00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 0x7f8dc811a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dcda00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 0x7f8dc811a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49834/0 (socket says 172.21.15.1:49834) 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dcda00700 1 -- 172.21.15.1:0/2045095989 learned_addr learned my addr 172.21.15.1:0/2045095989 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:03:30.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dcd000700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8dc81095f0 0x7f8dc811a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dcda00700 1 -- 172.21.15.1:0/2045095989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8dc81095f0 msgr2=0x7f8dc811a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dcda00700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8dc81095f0 0x7f8dc811a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.482+0000 7f8dcda00700 1 -- 172.21.15.1:0/2045095989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4005040 con 0x7f8dc8103dc0 2024-02-28T17:03:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.483+0000 7f8dcda00700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 0x7f8dc811a030 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00b790 tx=0x7f8dbc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.483+0000 7f8dc5600700 1 -- 172.21.15.1:0/2045095989 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dbc00f770 con 0x7f8dc8103dc0 2024-02-28T17:03:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.483+0000 7f8dc5600700 1 -- 172.21.15.1:0/2045095989 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8dbc00fda0 con 0x7f8dc8103dc0 2024-02-28T17:03:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.483+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc8114150 con 0x7f8dc8103dc0 2024-02-28T17:03:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.483+0000 7f8dc5600700 1 -- 172.21.15.1:0/2045095989 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dbc00d310 con 0x7f8dc8103dc0 2024-02-28T17:03:30.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.483+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc8114690 con 0x7f8dc8103dc0 2024-02-28T17:03:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.484+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc8075870 con 0x7f8dc8103dc0 2024-02-28T17:03:30.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.485+0000 7f8dc5600700 1 -- 172.21.15.1:0/2045095989 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8dbc010420 con 0x7f8dc8103dc0 2024-02-28T17:03:30.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.486+0000 7f8dc5600700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8db006d7f0 0x7f8db006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.486+0000 7f8dc5600700 1 -- 172.21.15.1:0/2045095989 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8dbc08ced0 con 0x7f8dc8103dc0 2024-02-28T17:03:30.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.487+0000 7f8dcd000700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8db006d7f0 0x7f8db006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.487+0000 7f8dcd000700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8db006d7f0 0x7f8db006fca0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8dc81158d0 tx=0x7f8db4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:30.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.491+0000 7f8dc5600700 1 -- 172.21.15.1:0/2045095989 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8dbc05a2d0 con 0x7f8dc8103dc0 2024-02-28T17:03:30.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.738+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dc8115720 con 0x7f8db006d7f0 2024-02-28T17:03:30.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.741+0000 7f8dc5600700 1 -- 172.21.15.1:0/2045095989 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8dc8115720 con 0x7f8db006d7f0 2024-02-28T17:03:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8db006d7f0 msgr2=0x7f8db006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8db006d7f0 0x7f8db006fca0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8dc81158d0 tx=0x7f8db4009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 msgr2=0x7f8dc811a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 0x7f8dc811a030 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00b790 tx=0x7f8dbc009250 comp rx=0 tx=0).stop 2024-02-28T17:03:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 shutdown_connections 2024-02-28T17:03:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8db006d7f0 0x7f8db006fca0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc8103dc0 0x7f8dc811a030 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 --2- 172.21.15.1:0/2045095989 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8dc81095f0 0x7f8dc811a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 >> 172.21.15.1:0/2045095989 conn(0x7f8dc8072d50 msgr2=0x7f8dc80746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:30.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.744+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 shutdown_connections 2024-02-28T17:03:30.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.745+0000 7f8dd0400700 1 -- 172.21.15.1:0/2045095989 wait complete. 2024-02-28T17:03:30.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.908+0000 7fa80be00700 1 -- 172.21.15.1:0/1369043303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 msgr2=0x7fa80c109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.908+0000 7fa80be00700 1 --2- 172.21.15.1:0/1369043303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 0x7fa80c109100 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8004970 tx=0x7fa7f8009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:30.909 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:30 smithi001 ceph-mon[112940]: pgmap v872: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:30.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.908+0000 7fa80be00700 1 -- 172.21.15.1:0/1369043303 shutdown_connections 2024-02-28T17:03:30.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.908+0000 7fa80be00700 1 --2- 172.21.15.1:0/1369043303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 0x7fa80c109100 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.908+0000 7fa80be00700 1 --2- 172.21.15.1:0/1369043303 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa80c0702d0 0x7fa80c0706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.908+0000 7fa80be00700 1 -- 172.21.15.1:0/1369043303 >> 172.21.15.1:0/1369043303 conn(0x7fa80c0f8940 msgr2=0x7fa80c0fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.908+0000 7fa80be00700 1 -- 172.21.15.1:0/1369043303 shutdown_connections 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.908+0000 7fa80be00700 1 -- 172.21.15.1:0/1369043303 wait complete. 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.909+0000 7fa80be00700 1 Processor -- start 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.909+0000 7fa80be00700 1 -- start start 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80be00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa80c0702d0 0x7fa80c115a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 0x7fa80c115f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c1165c0 con 0x7fa80c070be0 2024-02-28T17:03:30.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80be00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c10f8f0 con 0x7fa80c0702d0 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 0x7fa80c115f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 0x7fa80c115f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49842/0 (socket says 172.21.15.1:49842) 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80a000700 1 -- 172.21.15.1:0/3234570546 learned_addr learned my addr 172.21.15.1:0/3234570546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80aa00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa80c0702d0 0x7fa80c115a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80a000700 1 -- 172.21.15.1:0/3234570546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa80c0702d0 msgr2=0x7fa80c115a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80a000700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa80c0702d0 0x7fa80c115a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.910+0000 7fa80a000700 1 -- 172.21.15.1:0/3234570546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8005040 con 0x7fa80c070be0 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.911+0000 7fa80a000700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 0x7fa80c115f70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8004f50 tx=0x7fa7f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.911+0000 7fa802a00700 1 -- 172.21.15.1:0/3234570546 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7f800e040 con 0x7fa80c070be0 2024-02-28T17:03:30.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.911+0000 7fa802a00700 1 -- 172.21.15.1:0/3234570546 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7f8004290 con 0x7fa80c070be0 2024-02-28T17:03:30.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.911+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80c10fb70 con 0x7fa80c070be0 2024-02-28T17:03:30.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.911+0000 7fa802a00700 1 -- 172.21.15.1:0/3234570546 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7f801c630 con 0x7fa80c070be0 2024-02-28T17:03:30.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.911+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80c1100b0 con 0x7fa80c070be0 2024-02-28T17:03:30.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.913+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c1067c0 con 0x7fa80c070be0 2024-02-28T17:03:30.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.914+0000 7fa802a00700 1 -- 172.21.15.1:0/3234570546 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa7f800bc10 con 0x7fa80c070be0 2024-02-28T17:03:30.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.915+0000 7fa802a00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7f006d7f0 0x7fa7f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:30.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.915+0000 7fa802a00700 1 -- 172.21.15.1:0/3234570546 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa7f808e1c0 con 0x7fa80c070be0 2024-02-28T17:03:30.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.915+0000 7fa80aa00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7f006d7f0 0x7fa7f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:30.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.916+0000 7fa80aa00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7f006d7f0 0x7fa7f006fca0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc004990 tx=0x7fa7fc009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:30.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:30.919+0000 7fa802a00700 1 -- 172.21.15.1:0/3234570546 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa7f805b660 con 0x7fa80c070be0 2024-02-28T17:03:31.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:30 smithi066 ceph-mon[85781]: pgmap v872: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:31.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.159+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa80c10c090 con 0x7fa7f006d7f0 2024-02-28T17:03:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.174+0000 7fa802a00700 1 -- 172.21.15.1:0/3234570546 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa80c10c090 con 0x7fa7f006d7f0 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (29m) 6m ago 43m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (26m) 6m ago 44m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (26m) 6m ago 42m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (29m) 6m ago 43m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (22m) 6m ago 36m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (21m) 6m ago 36m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (36m) 6m ago 36m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (6m) 6m ago 36m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (32m) 6m ago 45m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (31m) 6m ago 42m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:03:31.176 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (28m) 6m ago 45m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (27m) 6m ago 42m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (30m) 6m ago 44m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (30m) 6m ago 42m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (25m) 6m ago 41m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (25m) 6m ago 40m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (24m) 6m ago 40m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (24m) 6m ago 39m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (23m) 6m ago 39m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (22m) 6m ago 38m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:03:31.177 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (30m) 6m ago 43m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:03:31.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7f006d7f0 msgr2=0x7fa7f006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:31.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7f006d7f0 0x7fa7f006fca0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc004990 tx=0x7fa7fc009ee0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 msgr2=0x7fa80c115f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:31.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 0x7fa80c115f70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8004f50 tx=0x7fa7f8009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:31.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 shutdown_connections 2024-02-28T17:03:31.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa7f006d7f0 0x7fa7f006fca0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa80c0702d0 0x7fa80c115a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 --2- 172.21.15.1:0/3234570546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa80c070be0 0x7fa80c115f70 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 >> 172.21.15.1:0/3234570546 conn(0x7fa80c0f8940 msgr2=0x7fa80c103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:31.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 shutdown_connections 2024-02-28T17:03:31.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.178+0000 7fa80be00700 1 -- 172.21.15.1:0/3234570546 wait complete. 2024-02-28T17:03:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.337+0000 7fbe25800700 1 -- 172.21.15.1:0/3257599256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe20103dc0 msgr2=0x7fbe20104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.337+0000 7fbe25800700 1 --2- 172.21.15.1:0/3257599256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe20103dc0 0x7fbe20104210 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c004940 tx=0x7fbe0c009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.338+0000 7fbe25800700 1 -- 172.21.15.1:0/3257599256 shutdown_connections 2024-02-28T17:03:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.338+0000 7fbe25800700 1 --2- 172.21.15.1:0/3257599256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe20103dc0 0x7fbe20104210 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.338+0000 7fbe25800700 1 --2- 172.21.15.1:0/3257599256 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe201095f0 0x7fbe201099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.338+0000 7fbe25800700 1 -- 172.21.15.1:0/3257599256 >> 172.21.15.1:0/3257599256 conn(0x7fbe20072d50 msgr2=0x7fbe20075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.338+0000 7fbe25800700 1 -- 172.21.15.1:0/3257599256 shutdown_connections 2024-02-28T17:03:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.338+0000 7fbe25800700 1 -- 172.21.15.1:0/3257599256 wait complete. 2024-02-28T17:03:31.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.339+0000 7fbe25800700 1 Processor -- start 2024-02-28T17:03:31.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.339+0000 7fbe25800700 1 -- start start 2024-02-28T17:03:31.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.339+0000 7fbe25800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe20103dc0 0x7fbe2011a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:31.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe25800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe201095f0 0x7fbe2011a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:31.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe25800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2011abe0 con 0x7fbe201095f0 2024-02-28T17:03:31.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe25800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20113ef0 con 0x7fbe20103dc0 2024-02-28T17:03:31.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe1e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe201095f0 0x7fbe2011a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:31.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe1e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe201095f0 0x7fbe2011a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49862/0 (socket says 172.21.15.1:49862) 2024-02-28T17:03:31.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe1e000700 1 -- 172.21.15.1:0/4082301745 learned_addr learned my addr 172.21.15.1:0/4082301745 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:03:31.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe1ea00700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe20103dc0 0x7fbe2011a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:31.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe1e000700 1 -- 172.21.15.1:0/4082301745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe20103dc0 msgr2=0x7fbe2011a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:31.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe1e000700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe20103dc0 0x7fbe2011a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.340+0000 7fbe1e000700 1 -- 172.21.15.1:0/4082301745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c005040 con 0x7fbe201095f0 2024-02-28T17:03:31.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.341+0000 7fbe1e000700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe201095f0 0x7fbe2011a590 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009150 tx=0x7fbe0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.341+0000 7fbe16a00700 1 -- 172.21.15.1:0/4082301745 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe0c00e040 con 0x7fbe201095f0 2024-02-28T17:03:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.341+0000 7fbe16a00700 1 -- 172.21.15.1:0/4082301745 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe0c00bbf0 con 0x7fbe201095f0 2024-02-28T17:03:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.341+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe20114170 con 0x7fbe201095f0 2024-02-28T17:03:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.341+0000 7fbe16a00700 1 -- 172.21.15.1:0/4082301745 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe0c01c630 con 0x7fbe201095f0 2024-02-28T17:03:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.341+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe201146b0 con 0x7fbe201095f0 2024-02-28T17:03:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.343+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2010b9c0 con 0x7fbe201095f0 2024-02-28T17:03:31.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.344+0000 7fbe16a00700 1 -- 172.21.15.1:0/4082301745 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbe0c00bd60 con 0x7fbe201095f0 2024-02-28T17:03:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.345+0000 7fbe16a00700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe0406d7f0 0x7fbe0406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.345+0000 7fbe16a00700 1 -- 172.21.15.1:0/4082301745 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbe0c08e290 con 0x7fbe201095f0 2024-02-28T17:03:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.345+0000 7fbe1ea00700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe0406d7f0 0x7fbe0406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.346+0000 7fbe1ea00700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe0406d7f0 0x7fbe0406fca0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbe10009fd0 tx=0x7fbe10009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:31.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.350+0000 7fbe16a00700 1 -- 172.21.15.1:0/4082301745 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe0c05b670 con 0x7fbe201095f0 2024-02-28T17:03:31.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.658+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbe20114300 con 0x7fbe201095f0 2024-02-28T17:03:31.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.660+0000 7fbe16a00700 1 -- 172.21.15.1:0/4082301745 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fbe20114300 con 0x7fbe201095f0 2024-02-28T17:03:31.661 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:03:31.661 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:03:31.661 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:03:31.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:03:31.661 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:03:31.661 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:03:31.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:03:31.662 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:03:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe0406d7f0 msgr2=0x7fbe0406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe0406d7f0 0x7fbe0406fca0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbe10009fd0 tx=0x7fbe10009f20 comp rx=0 tx=0).stop 2024-02-28T17:03:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe201095f0 msgr2=0x7fbe2011a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe201095f0 0x7fbe2011a590 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009150 tx=0x7fbe0c009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 shutdown_connections 2024-02-28T17:03:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe0406d7f0 0x7fbe0406fca0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe20103dc0 0x7fbe2011a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 --2- 172.21.15.1:0/4082301745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe201095f0 0x7fbe2011a590 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 >> 172.21.15.1:0/4082301745 conn(0x7fbe20072d50 msgr2=0x7fbe20074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 shutdown_connections 2024-02-28T17:03:31.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.663+0000 7fbe25800700 1 -- 172.21.15.1:0/4082301745 wait complete. 2024-02-28T17:03:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.807+0000 7f8cb2800700 1 -- 172.21.15.1:0/3335491154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 msgr2=0x7f8cac1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.807+0000 7f8cb2800700 1 --2- 172.21.15.1:0/3335491154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 0x7f8cac1099c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c004940 tx=0x7f8c9c009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.807+0000 7f8cb2800700 1 -- 172.21.15.1:0/3335491154 shutdown_connections 2024-02-28T17:03:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.807+0000 7f8cb2800700 1 --2- 172.21.15.1:0/3335491154 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8cac103dc0 0x7f8cac104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.807+0000 7f8cb2800700 1 --2- 172.21.15.1:0/3335491154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 0x7f8cac1099c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.807+0000 7f8cb2800700 1 -- 172.21.15.1:0/3335491154 >> 172.21.15.1:0/3335491154 conn(0x7f8cac072d50 msgr2=0x7f8cac075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.807+0000 7f8cb2800700 1 -- 172.21.15.1:0/3335491154 shutdown_connections 2024-02-28T17:03:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.807+0000 7f8cb2800700 1 -- 172.21.15.1:0/3335491154 wait complete. 2024-02-28T17:03:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.808+0000 7f8cb2800700 1 Processor -- start 2024-02-28T17:03:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.809+0000 7f8cb2800700 1 -- start start 2024-02-28T17:03:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.809+0000 7f8cb2800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8cac103dc0 0x7f8cac11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.809+0000 7f8cb2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 0x7f8cac11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.809+0000 7f8cb2800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac11ac70 con 0x7f8cac1095f0 2024-02-28T17:03:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.809+0000 7f8cb2800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac113f80 con 0x7f8cac103dc0 2024-02-28T17:03:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.810+0000 7f8ca3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 0x7f8cac11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:31.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.810+0000 7f8ca3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 0x7f8cac11a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49882/0 (socket says 172.21.15.1:49882) 2024-02-28T17:03:31.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.810+0000 7f8ca3e00700 1 -- 172.21.15.1:0/10807189 learned_addr learned my addr 172.21.15.1:0/10807189 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:03:31.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.810+0000 7f8cab400700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8cac103dc0 0x7f8cac11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:31.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.810+0000 7f8ca3e00700 1 -- 172.21.15.1:0/10807189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8cac103dc0 msgr2=0x7f8cac11a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:31.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.810+0000 7f8ca3e00700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8cac103dc0 0x7f8cac11a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:31.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.810+0000 7f8ca3e00700 1 -- 172.21.15.1:0/10807189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c005040 con 0x7f8cac1095f0 2024-02-28T17:03:31.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.810+0000 7f8ca3e00700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 0x7f8cac11a620 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8c94002a50 tx=0x7f8c940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.811+0000 7f8ca3400700 1 -- 172.21.15.1:0/10807189 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c94007cc0 con 0x7f8cac1095f0 2024-02-28T17:03:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.811+0000 7f8ca3400700 1 -- 172.21.15.1:0/10807189 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c9400f040 con 0x7f8cac1095f0 2024-02-28T17:03:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.811+0000 7f8ca3400700 1 -- 172.21.15.1:0/10807189 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c94009640 con 0x7f8cac1095f0 2024-02-28T17:03:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.811+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac114260 con 0x7f8cac1095f0 2024-02-28T17:03:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.811+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac1147a0 con 0x7f8cac1095f0 2024-02-28T17:03:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.812+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac106850 con 0x7f8cac1095f0 2024-02-28T17:03:31.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.813+0000 7f8ca3400700 1 -- 172.21.15.1:0/10807189 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8c940097a0 con 0x7f8cac1095f0 2024-02-28T17:03:31.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.814+0000 7f8ca3400700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c9006d7f0 0x7f8c9006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:31.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.814+0000 7f8ca3400700 1 -- 172.21.15.1:0/10807189 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8c9408cc60 con 0x7f8cac1095f0 2024-02-28T17:03:31.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.814+0000 7f8cab400700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c9006d7f0 0x7f8c9006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.815+0000 7f8cab400700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c9006d7f0 0x7f8c9006fca0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00b3b0 tx=0x7f8c9c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:31.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:31.819+0000 7f8ca3400700 1 -- 172.21.15.1:0/10807189 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8c94059fc0 con 0x7f8cac1095f0 2024-02-28T17:03:32.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:31 smithi066 ceph-mon[85781]: from='client.35570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:32.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:31 smithi066 ceph-mon[85781]: from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:32.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:31 smithi001 ceph-mon[112940]: from='client.35570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:32.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:31 smithi001 ceph-mon[112940]: from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:32.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.085+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8cac1156c0 con 0x7f8cac1095f0 2024-02-28T17:03:32.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.086+0000 7f8ca3400700 1 -- 172.21.15.1:0/10807189 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f8c94059b50 con 0x7f8cac1095f0 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:03:32.088 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:03:32.089 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:03:32.090 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:03:32.091 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:03:32.091 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:03:32.091 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:03:32.091 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c9006d7f0 msgr2=0x7f8c9006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c9006d7f0 0x7f8c9006fca0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00b3b0 tx=0x7f8c9c009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 msgr2=0x7f8cac11a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 0x7f8cac11a620 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8c94002a50 tx=0x7f8c940046b0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 shutdown_connections 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8c9006d7f0 0x7f8c9006fca0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8cac103dc0 0x7f8cac11a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 --2- 172.21.15.1:0/10807189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cac1095f0 0x7f8cac11a620 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 >> 172.21.15.1:0/10807189 conn(0x7f8cac072d50 msgr2=0x7f8cac074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 shutdown_connections 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.089+0000 7f8cb2800700 1 -- 172.21.15.1:0/10807189 wait complete. 2024-02-28T17:03:32.092 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:03:32.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.246+0000 7f71ed400700 1 -- 172.21.15.1:0/3830115284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fbd20 msgr2=0x7f71e80fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.246+0000 7f71ed400700 1 --2- 172.21.15.1:0/3830115284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fbd20 0x7f71e80fc0f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f71d8004940 tx=0x7f71d8009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.246+0000 7f71ed400700 1 -- 172.21.15.1:0/3830115284 shutdown_connections 2024-02-28T17:03:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.246+0000 7f71ed400700 1 --2- 172.21.15.1:0/3830115284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71e80fc630 0x7f71e8109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.246+0000 7f71ed400700 1 --2- 172.21.15.1:0/3830115284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fbd20 0x7f71e80fc0f0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.246+0000 7f71ed400700 1 -- 172.21.15.1:0/3830115284 >> 172.21.15.1:0/3830115284 conn(0x7f71e80f7a10 msgr2=0x7f71e80f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.246+0000 7f71ed400700 1 -- 172.21.15.1:0/3830115284 shutdown_connections 2024-02-28T17:03:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.246+0000 7f71ed400700 1 -- 172.21.15.1:0/3830115284 wait complete. 2024-02-28T17:03:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.247+0000 7f71ed400700 1 Processor -- start 2024-02-28T17:03:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.248+0000 7f71ed400700 1 -- start start 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.248+0000 7f71ed400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71e80fbd20 0x7f71e8115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.248+0000 7f71ed400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fc630 0x7f71e81160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.248+0000 7f71ed400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e81167d0 con 0x7f71e80fc630 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.248+0000 7f71ed400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e810fa70 con 0x7f71e80fbd20 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.248+0000 7f71e5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fc630 0x7f71e81160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.249+0000 7f71e5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fc630 0x7f71e81160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52110/0 (socket says 172.21.15.1:52110) 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.249+0000 7f71e5e00700 1 -- 172.21.15.1:0/1406361217 learned_addr learned my addr 172.21.15.1:0/1406361217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.249+0000 7f71e6800700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71e80fbd20 0x7f71e8115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.249+0000 7f71e5e00700 1 -- 172.21.15.1:0/1406361217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71e80fbd20 msgr2=0x7f71e8115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.249+0000 7f71e5e00700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71e80fbd20 0x7f71e8115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.249+0000 7f71e5e00700 1 -- 172.21.15.1:0/1406361217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d8005040 con 0x7f71e80fc630 2024-02-28T17:03:32.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.249+0000 7f71e5e00700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fc630 0x7f71e81160f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f71d400b790 tx=0x7f71d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:32.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.250+0000 7f71dea00700 1 -- 172.21.15.1:0/1406361217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71d400f770 con 0x7f71e80fc630 2024-02-28T17:03:32.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.250+0000 7f71dea00700 1 -- 172.21.15.1:0/1406361217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71d400fda0 con 0x7f71e80fc630 2024-02-28T17:03:32.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.250+0000 7f71dea00700 1 -- 172.21.15.1:0/1406361217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71d400d310 con 0x7f71e80fc630 2024-02-28T17:03:32.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.250+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e810fd50 con 0x7f71e80fc630 2024-02-28T17:03:32.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.250+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e8110290 con 0x7f71e80fc630 2024-02-28T17:03:32.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.251+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e8106ac0 con 0x7f71e80fc630 2024-02-28T17:03:32.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.252+0000 7f71dea00700 1 -- 172.21.15.1:0/1406361217 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f71d4010420 con 0x7f71e80fc630 2024-02-28T17:03:32.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.253+0000 7f71dea00700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f71cc06d7e0 0x7f71cc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:32.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.253+0000 7f71dea00700 1 -- 172.21.15.1:0/1406361217 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f71d408cef0 con 0x7f71e80fc630 2024-02-28T17:03:32.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.254+0000 7f71e6800700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f71cc06d7e0 0x7f71cc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:32.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.254+0000 7f71e6800700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f71cc06d7e0 0x7f71cc06fc90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f71d8004f90 tx=0x7f71d8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:32.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.258+0000 7f71dea00700 1 -- 172.21.15.1:0/1406361217 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f71d405a2d0 con 0x7f71e80fc630 2024-02-28T17:03:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.499+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71e810c260 con 0x7f71cc06d7e0 2024-02-28T17:03:32.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.501+0000 7f71dea00700 1 -- 172.21.15.1:0/1406361217 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f71e810c260 con 0x7f71cc06d7e0 2024-02-28T17:03:32.502 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:03:32.503 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:03:32.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f71cc06d7e0 msgr2=0x7f71cc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f71cc06d7e0 0x7f71cc06fc90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f71d8004f90 tx=0x7f71d8009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fc630 msgr2=0x7f71e81160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fc630 0x7f71e81160f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f71d400b790 tx=0x7f71d4009250 comp rx=0 tx=0).stop 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 shutdown_connections 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f71cc06d7e0 0x7f71cc06fc90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71e80fbd20 0x7f71e8115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 --2- 172.21.15.1:0/1406361217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80fc630 0x7f71e81160f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 >> 172.21.15.1:0/1406361217 conn(0x7f71e80f7a10 msgr2=0x7f71e80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 shutdown_connections 2024-02-28T17:03:32.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.504+0000 7f71ed400700 1 -- 172.21.15.1:0/1406361217 wait complete. 2024-02-28T17:03:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.651+0000 7fb45ba00700 1 -- 172.21.15.1:0/2859295615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fdfe0 msgr2=0x7fb4540fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.651+0000 7fb45ba00700 1 --2- 172.21.15.1:0/2859295615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fdfe0 0x7fb4540fe3b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb448004970 tx=0x7fb448009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.651+0000 7fb45ba00700 1 -- 172.21.15.1:0/2859295615 shutdown_connections 2024-02-28T17:03:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.651+0000 7fb45ba00700 1 --2- 172.21.15.1:0/2859295615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4540fe8f0 0x7fb454107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.651+0000 7fb45ba00700 1 --2- 172.21.15.1:0/2859295615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fdfe0 0x7fb4540fe3b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.651+0000 7fb45ba00700 1 -- 172.21.15.1:0/2859295615 >> 172.21.15.1:0/2859295615 conn(0x7fb454071d90 msgr2=0x7fb454072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.651+0000 7fb45ba00700 1 -- 172.21.15.1:0/2859295615 shutdown_connections 2024-02-28T17:03:32.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.651+0000 7fb45ba00700 1 -- 172.21.15.1:0/2859295615 wait complete. 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.652+0000 7fb45ba00700 1 Processor -- start 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.652+0000 7fb45ba00700 1 -- start start 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.652+0000 7fb45ba00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4540fdfe0 0x7fb4540fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.652+0000 7fb45ba00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fe8f0 0x7fb4540fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.652+0000 7fb45ba00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4540fca70 con 0x7fb4540fe8f0 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.652+0000 7fb45ba00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4540fcbe0 con 0x7fb4540fdfe0 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.653+0000 7fb453e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fe8f0 0x7fb4540fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.653+0000 7fb453e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fe8f0 0x7fb4540fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52138/0 (socket says 172.21.15.1:52138) 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.653+0000 7fb453e00700 1 -- 172.21.15.1:0/715617853 learned_addr learned my addr 172.21.15.1:0/715617853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.653+0000 7fb459000700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4540fdfe0 0x7fb4540fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.653+0000 7fb453e00700 1 -- 172.21.15.1:0/715617853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4540fdfe0 msgr2=0x7fb4540fbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.653+0000 7fb453e00700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4540fdfe0 0x7fb4540fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.653+0000 7fb453e00700 1 -- 172.21.15.1:0/715617853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb448005040 con 0x7fb4540fe8f0 2024-02-28T17:03:32.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.653+0000 7fb453e00700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fe8f0 0x7fb4540fc3e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb44000b790 tx=0x7fb440009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:32.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.654+0000 7fb450c00700 1 -- 172.21.15.1:0/715617853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb44000f770 con 0x7fb4540fe8f0 2024-02-28T17:03:32.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.654+0000 7fb450c00700 1 -- 172.21.15.1:0/715617853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb44000fda0 con 0x7fb4540fe8f0 2024-02-28T17:03:32.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.654+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb454119d70 con 0x7fb4540fe8f0 2024-02-28T17:03:32.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.654+0000 7fb450c00700 1 -- 172.21.15.1:0/715617853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb44000d310 con 0x7fb4540fe8f0 2024-02-28T17:03:32.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.654+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45411a260 con 0x7fb4540fe8f0 2024-02-28T17:03:32.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.655+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4541046c0 con 0x7fb4540fe8f0 2024-02-28T17:03:32.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.656+0000 7fb450c00700 1 -- 172.21.15.1:0/715617853 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb440010420 con 0x7fb4540fe8f0 2024-02-28T17:03:32.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.657+0000 7fb450c00700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb43c06d7f0 0x7fb43c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:03:32.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.657+0000 7fb450c00700 1 -- 172.21.15.1:0/715617853 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb44008cfb0 con 0x7fb4540fe8f0 2024-02-28T17:03:32.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.658+0000 7fb459000700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb43c06d7f0 0x7fb43c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:03:32.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.658+0000 7fb459000700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb43c06d7f0 0x7fb43c06fca0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb448000c00 tx=0x7fb448009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:03:32.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.662+0000 7fb450c00700 1 -- 172.21.15.1:0/715617853 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb4400c0d20 con 0x7fb4540fe8f0 2024-02-28T17:03:32.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.970+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb4540fd5e0 con 0x7fb4540fe8f0 2024-02-28T17:03:32.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:32 smithi001 ceph-mon[112940]: pgmap v873: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:32.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:32 smithi001 ceph-mon[112940]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:32.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:32 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4082301745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:03:32.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:32 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/10807189' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.971+0000 7fb450c00700 1 -- 172.21.15.1:0/715617853 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fb4400567e0 con 0x7fb4540fe8f0 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:03:32.973 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:03:32.974 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:03:32.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.975+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb43c06d7f0 msgr2=0x7fb43c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.975+0000 7fb45ba00700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb43c06d7f0 0x7fb43c06fca0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb448000c00 tx=0x7fb448009e30 comp rx=0 tx=0).stop 2024-02-28T17:03:32.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.975+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fe8f0 msgr2=0x7fb4540fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:03:32.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.975+0000 7fb45ba00700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fe8f0 0x7fb4540fc3e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb44000b790 tx=0x7fb440009250 comp rx=0 tx=0).stop 2024-02-28T17:03:32.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.975+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 shutdown_connections 2024-02-28T17:03:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.976+0000 7fb45ba00700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb43c06d7f0 0x7fb43c06fca0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.976+0000 7fb45ba00700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb4540fdfe0 0x7fb4540fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.976+0000 7fb45ba00700 1 --2- 172.21.15.1:0/715617853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540fe8f0 0x7fb4540fc3e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:03:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.976+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 >> 172.21.15.1:0/715617853 conn(0x7fb454071d90 msgr2=0x7fb454101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:03:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.976+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 shutdown_connections 2024-02-28T17:03:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:03:32.976+0000 7fb45ba00700 1 -- 172.21.15.1:0/715617853 wait complete. 2024-02-28T17:03:33.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:32 smithi066 ceph-mon[85781]: pgmap v873: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:33.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:32 smithi066 ceph-mon[85781]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:33.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:32 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4082301745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:03:33.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:32 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/10807189' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:03:34.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:33 smithi066 ceph-mon[85781]: from='client.35590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:34.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:33 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/715617853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:03:34.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:33 smithi001 ceph-mon[112940]: from='client.35590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:03:34.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:33 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/715617853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:03:35.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:34 smithi066 ceph-mon[85781]: pgmap v874: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:35.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:34 smithi001 ceph-mon[112940]: pgmap v874: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:37.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:36 smithi066 ceph-mon[85781]: pgmap v875: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:37.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:36 smithi001 ceph-mon[112940]: pgmap v875: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:39.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:38 smithi066 ceph-mon[85781]: pgmap v876: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:39.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:38 smithi001 ceph-mon[112940]: pgmap v876: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:41.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:40 smithi066 ceph-mon[85781]: pgmap v877: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:41.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:40 smithi001 ceph-mon[112940]: pgmap v877: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:42.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:03:42.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:03:43.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:42 smithi066 ceph-mon[85781]: pgmap v878: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:43.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:42 smithi001 ceph-mon[112940]: pgmap v878: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:45.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:44 smithi066 ceph-mon[85781]: pgmap v879: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:45.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:44 smithi001 ceph-mon[112940]: pgmap v879: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:47.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:46 smithi066 ceph-mon[85781]: pgmap v880: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:47.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:46 smithi001 ceph-mon[112940]: pgmap v880: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:49.009 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:48 smithi066 ceph-mon[85781]: pgmap v881: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:49.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:48 smithi001 ceph-mon[112940]: pgmap v881: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:51.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:50 smithi066 ceph-mon[85781]: pgmap v882: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:51.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:50 smithi001 ceph-mon[112940]: pgmap v882: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:53.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:52 smithi066 ceph-mon[85781]: pgmap v883: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:53.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:52 smithi001 ceph-mon[112940]: pgmap v883: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:55.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:54 smithi066 ceph-mon[85781]: pgmap v884: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:55.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:54 smithi001 ceph-mon[112940]: pgmap v884: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:57.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:56 smithi066 ceph-mon[85781]: pgmap v885: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:57.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:03:57.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:56 smithi001 ceph-mon[112940]: pgmap v885: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:57.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:03:59.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:03:58 smithi066 ceph-mon[85781]: pgmap v886: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:03:59.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:03:58 smithi001 ceph-mon[112940]: pgmap v886: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:01.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:00 smithi066 ceph-mon[85781]: pgmap v887: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:01.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:00 smithi001 ceph-mon[112940]: pgmap v887: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:03.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:02 smithi066 ceph-mon[85781]: pgmap v888: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:02 smithi001 ceph-mon[112940]: pgmap v888: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.148+0000 7efdfa400700 1 -- 172.21.15.1:0/2992720095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 msgr2=0x7efdf4072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.148+0000 7efdfa400700 1 --2- 172.21.15.1:0/2992720095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 0x7efdf4072890 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7efde4004940 tx=0x7efde4009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.148+0000 7efdfa400700 1 -- 172.21.15.1:0/2992720095 shutdown_connections 2024-02-28T17:04:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.148+0000 7efdfa400700 1 --2- 172.21.15.1:0/2992720095 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efdf4072dd0 0x7efdf410d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.148+0000 7efdfa400700 1 --2- 172.21.15.1:0/2992720095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 0x7efdf4072890 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.148+0000 7efdfa400700 1 -- 172.21.15.1:0/2992720095 >> 172.21.15.1:0/2992720095 conn(0x7efdf40face0 msgr2=0x7efdf40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.148+0000 7efdfa400700 1 -- 172.21.15.1:0/2992720095 shutdown_connections 2024-02-28T17:04:03.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.148+0000 7efdfa400700 1 -- 172.21.15.1:0/2992720095 wait complete. 2024-02-28T17:04:03.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.149+0000 7efdfa400700 1 Processor -- start 2024-02-28T17:04:03.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.149+0000 7efdfa400700 1 -- start start 2024-02-28T17:04:03.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.150+0000 7efdfa400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 0x7efdf411a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.150+0000 7efdfa400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efdf4072dd0 0x7efdf411a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.150+0000 7efdfa400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf411aba0 con 0x7efdf40724c0 2024-02-28T17:04:03.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.150+0000 7efdfa400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4113eb0 con 0x7efdf4072dd0 2024-02-28T17:04:03.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.150+0000 7efdf3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 0x7efdf411a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.150+0000 7efdf3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 0x7efdf411a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38678/0 (socket says 172.21.15.1:38678) 2024-02-28T17:04:03.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.150+0000 7efdf3400700 1 -- 172.21.15.1:0/3290675728 learned_addr learned my addr 172.21.15.1:0/3290675728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdf2a00700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efdf4072dd0 0x7efdf411a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdf3400700 1 -- 172.21.15.1:0/3290675728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efdf4072dd0 msgr2=0x7efdf411a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdf3400700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efdf4072dd0 0x7efdf411a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdf3400700 1 -- 172.21.15.1:0/3290675728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde4005040 con 0x7efdf40724c0 2024-02-28T17:04:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdf3400700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 0x7efdf411a010 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7efde4009180 tx=0x7efde4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdeb400700 1 -- 172.21.15.1:0/3290675728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efde400e040 con 0x7efdf40724c0 2024-02-28T17:04:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdeb400700 1 -- 172.21.15.1:0/3290675728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efde400bbf0 con 0x7efdf40724c0 2024-02-28T17:04:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf4114130 con 0x7efdf40724c0 2024-02-28T17:04:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdeb400700 1 -- 172.21.15.1:0/3290675728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efde401c630 con 0x7efdf40724c0 2024-02-28T17:04:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.151+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf4114670 con 0x7efdf40724c0 2024-02-28T17:04:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.153+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf410aff0 con 0x7efdf40724c0 2024-02-28T17:04:03.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.154+0000 7efdeb400700 1 -- 172.21.15.1:0/3290675728 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7efde400bd60 con 0x7efdf40724c0 2024-02-28T17:04:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.155+0000 7efdeb400700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efdd806d7f0 0x7efdd806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.155+0000 7efdeb400700 1 -- 172.21.15.1:0/3290675728 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efde408e360 con 0x7efdf40724c0 2024-02-28T17:04:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.155+0000 7efdf2a00700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efdd806d7f0 0x7efdd806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.156+0000 7efdf2a00700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efdd806d7f0 0x7efdd806fca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efdf41158b0 tx=0x7efde000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:03.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.160+0000 7efdeb400700 1 -- 172.21.15.1:0/3290675728 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efde405b6c0 con 0x7efdf40724c0 2024-02-28T17:04:03.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.398+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efdf404bf40 con 0x7efdd806d7f0 2024-02-28T17:04:03.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.401+0000 7efdeb400700 1 -- 172.21.15.1:0/3290675728 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efdf404bf40 con 0x7efdd806d7f0 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efdd806d7f0 msgr2=0x7efdd806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efdd806d7f0 0x7efdd806fca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efdf41158b0 tx=0x7efde000b040 comp rx=0 tx=0).stop 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 msgr2=0x7efdf411a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 0x7efdf411a010 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7efde4009180 tx=0x7efde4009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 shutdown_connections 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efdd806d7f0 0x7efdd806fca0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdf40724c0 0x7efdf411a010 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 --2- 172.21.15.1:0/3290675728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efdf4072dd0 0x7efdf411a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 >> 172.21.15.1:0/3290675728 conn(0x7efdf40face0 msgr2=0x7efdf40fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 shutdown_connections 2024-02-28T17:04:03.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.404+0000 7efdfa400700 1 -- 172.21.15.1:0/3290675728 wait complete. 2024-02-28T17:04:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:04:03.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.561+0000 7fc7cbe00700 1 -- 172.21.15.1:0/4143121797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc103730 msgr2=0x7fc7cc0fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.561+0000 7fc7cbe00700 1 --2- 172.21.15.1:0/4143121797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc103730 0x7fc7cc0fcda0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8004940 tx=0x7fc7b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:03.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.561+0000 7fc7cbe00700 1 -- 172.21.15.1:0/4143121797 shutdown_connections 2024-02-28T17:04:03.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.561+0000 7fc7cbe00700 1 --2- 172.21.15.1:0/4143121797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc103730 0x7fc7cc0fcda0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.561+0000 7fc7cbe00700 1 --2- 172.21.15.1:0/4143121797 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7cc102d90 0x7fc7cc103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.561+0000 7fc7cbe00700 1 -- 172.21.15.1:0/4143121797 >> 172.21.15.1:0/4143121797 conn(0x7fc7cc0f8940 msgr2=0x7fc7cc0fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.561+0000 7fc7cbe00700 1 -- 172.21.15.1:0/4143121797 shutdown_connections 2024-02-28T17:04:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.561+0000 7fc7cbe00700 1 -- 172.21.15.1:0/4143121797 wait complete. 2024-02-28T17:04:03.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.562+0000 7fc7cbe00700 1 Processor -- start 2024-02-28T17:04:03.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.562+0000 7fc7cbe00700 1 -- start start 2024-02-28T17:04:03.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.562+0000 7fc7cbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc102d90 0x7fc7cc1138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.562+0000 7fc7cbe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7cc103730 0x7fc7cc113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.562+0000 7fc7cbe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc114500 con 0x7fc7cc102d90 2024-02-28T17:04:03.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.562+0000 7fc7cbe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc10d7a0 con 0x7fc7cc103730 2024-02-28T17:04:03.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.562+0000 7fc7caa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc102d90 0x7fc7cc1138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.563+0000 7fc7caa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc102d90 0x7fc7cc1138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38696/0 (socket says 172.21.15.1:38696) 2024-02-28T17:04:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.563+0000 7fc7caa00700 1 -- 172.21.15.1:0/3117137929 learned_addr learned my addr 172.21.15.1:0/3117137929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.563+0000 7fc7ca000700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7cc103730 0x7fc7cc113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.563+0000 7fc7caa00700 1 -- 172.21.15.1:0/3117137929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7cc103730 msgr2=0x7fc7cc113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.563+0000 7fc7caa00700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7cc103730 0x7fc7cc113e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.563+0000 7fc7caa00700 1 -- 172.21.15.1:0/3117137929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8005040 con 0x7fc7cc102d90 2024-02-28T17:04:03.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.563+0000 7fc7caa00700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc102d90 0x7fc7cc1138e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00b790 tx=0x7fc7bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:03.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.564+0000 7fc7c2a00700 1 -- 172.21.15.1:0/3117137929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7bc00f770 con 0x7fc7cc102d90 2024-02-28T17:04:03.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.564+0000 7fc7c2a00700 1 -- 172.21.15.1:0/3117137929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7bc00fda0 con 0x7fc7cc102d90 2024-02-28T17:04:03.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.564+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc10da80 con 0x7fc7cc102d90 2024-02-28T17:04:03.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.564+0000 7fc7c2a00700 1 -- 172.21.15.1:0/3117137929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7bc00d310 con 0x7fc7cc102d90 2024-02-28T17:04:03.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.564+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc10dfc0 con 0x7fc7cc102d90 2024-02-28T17:04:03.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.565+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7cc05f5c0 con 0x7fc7cc102d90 2024-02-28T17:04:03.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.566+0000 7fc7c2a00700 1 -- 172.21.15.1:0/3117137929 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc7bc010420 con 0x7fc7cc102d90 2024-02-28T17:04:03.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.567+0000 7fc7c2a00700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7b006d7f0 0x7fc7b006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.567+0000 7fc7c2a00700 1 -- 172.21.15.1:0/3117137929 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc7bc08cf70 con 0x7fc7cc102d90 2024-02-28T17:04:03.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.567+0000 7fc7ca000700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7b006d7f0 0x7fc7b006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.568+0000 7fc7ca000700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7b006d7f0 0x7fc7b006fca0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc10f110 tx=0x7fc7b8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:03.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.572+0000 7fc7c2a00700 1 -- 172.21.15.1:0/3117137929 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc7bc05a2d0 con 0x7fc7cc102d90 2024-02-28T17:04:03.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.816+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7cc10e240 con 0x7fc7b006d7f0 2024-02-28T17:04:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.819+0000 7fc7c2a00700 1 -- 172.21.15.1:0/3117137929 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc7cc10e240 con 0x7fc7b006d7f0 2024-02-28T17:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.821+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7b006d7f0 msgr2=0x7fc7b006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.821+0000 7fc7cbe00700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7b006d7f0 0x7fc7b006fca0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc10f110 tx=0x7fc7b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.821+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc102d90 msgr2=0x7fc7cc1138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.821+0000 7fc7cbe00700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc102d90 0x7fc7cc1138e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00b790 tx=0x7fc7bc009250 comp rx=0 tx=0).stop 2024-02-28T17:04:03.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.821+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 shutdown_connections 2024-02-28T17:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.821+0000 7fc7cbe00700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7b006d7f0 0x7fc7b006fca0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.821+0000 7fc7cbe00700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7cc102d90 0x7fc7cc1138e0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.821+0000 7fc7cbe00700 1 --2- 172.21.15.1:0/3117137929 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7cc103730 0x7fc7cc113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.822+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 >> 172.21.15.1:0/3117137929 conn(0x7fc7cc0f8940 msgr2=0x7fc7cc0fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.822+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 shutdown_connections 2024-02-28T17:04:03.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.822+0000 7fc7cbe00700 1 -- 172.21.15.1:0/3117137929 wait complete. 2024-02-28T17:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.968+0000 7ff3cbe00700 1 -- 172.21.15.1:0/3157366565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 msgr2=0x7ff3cc106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.968+0000 7ff3cbe00700 1 --2- 172.21.15.1:0/3157366565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 0x7ff3cc106e00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4004940 tx=0x7ff3b4009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:03.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.968+0000 7ff3cbe00700 1 -- 172.21.15.1:0/3157366565 shutdown_connections 2024-02-28T17:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.968+0000 7ff3cbe00700 1 --2- 172.21.15.1:0/3157366565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 0x7ff3cc106e00 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.968+0000 7ff3cbe00700 1 --2- 172.21.15.1:0/3157366565 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3cc0fdd90 0x7ff3cc0fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.968+0000 7ff3cbe00700 1 -- 172.21.15.1:0/3157366565 >> 172.21.15.1:0/3157366565 conn(0x7ff3cc073ab0 msgr2=0x7ff3cc073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.968+0000 7ff3cbe00700 1 -- 172.21.15.1:0/3157366565 shutdown_connections 2024-02-28T17:04:03.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.968+0000 7ff3cbe00700 1 -- 172.21.15.1:0/3157366565 wait complete. 2024-02-28T17:04:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.969+0000 7ff3cbe00700 1 Processor -- start 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.969+0000 7ff3cbe00700 1 -- start start 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.969+0000 7ff3cbe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3cc0fdd90 0x7ff3cc0fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.969+0000 7ff3cbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 0x7ff3cc0fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.969+0000 7ff3cbe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0fc790 con 0x7ff3cc0fe730 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.969+0000 7ff3cbe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0fc900 con 0x7ff3cc0fdd90 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.969+0000 7ff3c2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 0x7ff3cc0fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 0x7ff3cc0fc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38718/0 (socket says 172.21.15.1:38718) 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c2000700 1 -- 172.21.15.1:0/2268120286 learned_addr learned my addr 172.21.15.1:0/2268120286 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3caa00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3cc0fdd90 0x7ff3cc0fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c2000700 1 -- 172.21.15.1:0/2268120286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3cc0fdd90 msgr2=0x7ff3cc0fbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c2000700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3cc0fdd90 0x7ff3cc0fbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c2000700 1 -- 172.21.15.1:0/2268120286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4005040 con 0x7ff3cc0fe730 2024-02-28T17:04:03.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3caa00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3cc0fdd90 0x7ff3cc0fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c2000700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 0x7ff3cc0fc190 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009040 tx=0x7ff3b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c3400700 1 -- 172.21.15.1:0/2268120286 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b400e040 con 0x7ff3cc0fe730 2024-02-28T17:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c3400700 1 -- 172.21.15.1:0/2268120286 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3b400bb90 con 0x7ff3cc0fe730 2024-02-28T17:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3c3400700 1 -- 172.21.15.1:0/2268120286 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b401c3f0 con 0x7ff3cc0fe730 2024-02-28T17:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.970+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc119b30 con 0x7ff3cc0fe730 2024-02-28T17:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.971+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc119f90 con 0x7ff3cc0fe730 2024-02-28T17:04:03.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.972+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc1044c0 con 0x7ff3cc0fe730 2024-02-28T17:04:03.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.973+0000 7ff3c3400700 1 -- 172.21.15.1:0/2268120286 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff3b400bd00 con 0x7ff3cc0fe730 2024-02-28T17:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.974+0000 7ff3c3400700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3b006d7f0 0x7ff3b006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.974+0000 7ff3c3400700 1 -- 172.21.15.1:0/2268120286 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff3b408e1b0 con 0x7ff3cc0fe730 2024-02-28T17:04:03.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.974+0000 7ff3caa00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3b006d7f0 0x7ff3b006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:03.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.975+0000 7ff3caa00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3b006d7f0 0x7ff3b006fca0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009fd0 tx=0x7ff3bc00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:03.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:03.979+0000 7ff3c3400700 1 -- 172.21.15.1:0/2268120286 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff3b4057a80 con 0x7ff3cc0fe730 2024-02-28T17:04:04.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.214+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3cc0fd390 con 0x7ff3b006d7f0 2024-02-28T17:04:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.231+0000 7ff3c3400700 1 -- 172.21.15.1:0/2268120286 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff3cc0fd390 con 0x7ff3b006d7f0 2024-02-28T17:04:04.232 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:04:04.232 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (30m) 6m ago 44m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:04:04.232 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (26m) 6m ago 44m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:04:04.232 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (26m) 6m ago 43m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:04:04.232 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (29m) 6m ago 44m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (22m) 6m ago 37m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (22m) 6m ago 37m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (37m) 6m ago 37m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (7m) 6m ago 37m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (32m) 6m ago 46m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (31m) 6m ago 43m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (28m) 6m ago 46m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (27m) 6m ago 43m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (31m) 6m ago 44m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (31m) 6m ago 43m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (26m) 6m ago 42m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (25m) 6m ago 41m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (25m) 6m ago 40m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (24m) 6m ago 40m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:04:04.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (23m) 6m ago 39m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:04:04.234 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (23m) 6m ago 39m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:04:04.234 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (30m) 6m ago 43m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:04:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.234+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3b006d7f0 msgr2=0x7ff3b006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.234+0000 7ff3cbe00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3b006d7f0 0x7ff3b006fca0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009fd0 tx=0x7ff3bc00b040 comp rx=0 tx=0).stop 2024-02-28T17:04:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.234+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 msgr2=0x7ff3cc0fc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.234+0000 7ff3cbe00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 0x7ff3cc0fc190 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009040 tx=0x7ff3b4009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.235+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 shutdown_connections 2024-02-28T17:04:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.235+0000 7ff3cbe00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3b006d7f0 0x7ff3b006fca0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.235+0000 7ff3cbe00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3cc0fdd90 0x7ff3cc0fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.235+0000 7ff3cbe00700 1 --2- 172.21.15.1:0/2268120286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc0fe730 0x7ff3cc0fc190 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.235+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 >> 172.21.15.1:0/2268120286 conn(0x7ff3cc073ab0 msgr2=0x7ff3cc101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:04.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.235+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 shutdown_connections 2024-02-28T17:04:04.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.235+0000 7ff3cbe00700 1 -- 172.21.15.1:0/2268120286 wait complete. 2024-02-28T17:04:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.390+0000 7fdd16200700 1 -- 172.21.15.1:0/2014271044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 msgr2=0x7fdd10104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.390+0000 7fdd16200700 1 --2- 172.21.15.1:0/2014271044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 0x7fdd10104210 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc004940 tx=0x7fdcfc009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.390+0000 7fdd16200700 1 -- 172.21.15.1:0/2014271044 shutdown_connections 2024-02-28T17:04:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.390+0000 7fdd16200700 1 --2- 172.21.15.1:0/2014271044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 0x7fdd10104210 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.390+0000 7fdd16200700 1 --2- 172.21.15.1:0/2014271044 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd101095f0 0x7fdd101099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.390+0000 7fdd16200700 1 -- 172.21.15.1:0/2014271044 >> 172.21.15.1:0/2014271044 conn(0x7fdd10072d50 msgr2=0x7fdd10075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.391+0000 7fdd16200700 1 -- 172.21.15.1:0/2014271044 shutdown_connections 2024-02-28T17:04:04.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.391+0000 7fdd16200700 1 -- 172.21.15.1:0/2014271044 wait complete. 2024-02-28T17:04:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.391+0000 7fdd16200700 1 Processor -- start 2024-02-28T17:04:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.392+0000 7fdd16200700 1 -- start start 2024-02-28T17:04:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.392+0000 7fdd16200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 0x7fdd1011a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.392+0000 7fdd16200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd101095f0 0x7fdd1011a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.392+0000 7fdd16200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1011ac50 con 0x7fdd10103dc0 2024-02-28T17:04:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.392+0000 7fdd16200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10113ed0 con 0x7fdd101095f0 2024-02-28T17:04:04.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.392+0000 7fdd0f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 0x7fdd1011a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.392+0000 7fdd0f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 0x7fdd1011a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38746/0 (socket says 172.21.15.1:38746) 2024-02-28T17:04:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.392+0000 7fdd0f400700 1 -- 172.21.15.1:0/4081311358 learned_addr learned my addr 172.21.15.1:0/4081311358 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.393+0000 7fdd0ea00700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd101095f0 0x7fdd1011a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.393+0000 7fdd0f400700 1 -- 172.21.15.1:0/4081311358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd101095f0 msgr2=0x7fdd1011a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.393+0000 7fdd0f400700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd101095f0 0x7fdd1011a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.393+0000 7fdd0f400700 1 -- 172.21.15.1:0/4081311358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc005040 con 0x7fdd10103dc0 2024-02-28T17:04:04.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.393+0000 7fdd0f400700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 0x7fdd1011a030 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000b790 tx=0x7fdd00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:04.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.394+0000 7fdd07400700 1 -- 172.21.15.1:0/4081311358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd0000f770 con 0x7fdd10103dc0 2024-02-28T17:04:04.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.394+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd10114150 con 0x7fdd10103dc0 2024-02-28T17:04:04.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.394+0000 7fdd07400700 1 -- 172.21.15.1:0/4081311358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd0000fda0 con 0x7fdd10103dc0 2024-02-28T17:04:04.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.394+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd10114690 con 0x7fdd10103dc0 2024-02-28T17:04:04.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.394+0000 7fdd07400700 1 -- 172.21.15.1:0/4081311358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd0001a640 con 0x7fdd10103dc0 2024-02-28T17:04:04.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.395+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd10075870 con 0x7fdd10103dc0 2024-02-28T17:04:04.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.396+0000 7fdd07400700 1 -- 172.21.15.1:0/4081311358 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdd00010420 con 0x7fdd10103dc0 2024-02-28T17:04:04.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.397+0000 7fdd07400700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdcf406d7f0 0x7fdcf406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:04.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.397+0000 7fdd07400700 1 -- 172.21.15.1:0/4081311358 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdd0008cf20 con 0x7fdd10103dc0 2024-02-28T17:04:04.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.397+0000 7fdd0ea00700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdcf406d7f0 0x7fdcf406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:04.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.398+0000 7fdd0ea00700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdcf406d7f0 0x7fdcf406fca0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00b020 tx=0x7fdcfc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:04.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.401+0000 7fdd07400700 1 -- 172.21.15.1:0/4081311358 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdd0005a280 con 0x7fdd10103dc0 2024-02-28T17:04:04.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.716+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd101106a0 con 0x7fdd10103dc0 2024-02-28T17:04:04.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.717+0000 7fdd07400700 1 -- 172.21.15.1:0/4081311358 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fdd00059e10 con 0x7fdd10103dc0 2024-02-28T17:04:04.717 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:04:04.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:04:04.719 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:04:04.719 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:04:04.719 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:04:04.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:04:04.719 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:04:04.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdcf406d7f0 msgr2=0x7fdcf406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:04.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdcf406d7f0 0x7fdcf406fca0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00b020 tx=0x7fdcfc009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:04.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 msgr2=0x7fdd1011a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:04.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 0x7fdd1011a030 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000b790 tx=0x7fdd00009250 comp rx=0 tx=0).stop 2024-02-28T17:04:04.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 shutdown_connections 2024-02-28T17:04:04.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdcf406d7f0 0x7fdcf406fca0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd10103dc0 0x7fdd1011a030 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 --2- 172.21.15.1:0/4081311358 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd101095f0 0x7fdd1011a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 >> 172.21.15.1:0/4081311358 conn(0x7fdd10072d50 msgr2=0x7fdd100746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 shutdown_connections 2024-02-28T17:04:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.720+0000 7fdd16200700 1 -- 172.21.15.1:0/4081311358 wait complete. 2024-02-28T17:04:04.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.875+0000 7f10d4600700 1 -- 172.21.15.1:0/2456729563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 msgr2=0x7f10cc0ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:04.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.875+0000 7f10d4600700 1 --2- 172.21.15.1:0/2456729563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 0x7f10cc0ff180 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f10b8004940 tx=0x7f10b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:04.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.876+0000 7f10d4600700 1 -- 172.21.15.1:0/2456729563 shutdown_connections 2024-02-28T17:04:04.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.876+0000 7f10d4600700 1 --2- 172.21.15.1:0/2456729563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 0x7f10cc0ff180 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.876+0000 7f10d4600700 1 --2- 172.21.15.1:0/2456729563 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10cc105170 0x7f10cc105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.876+0000 7f10d4600700 1 -- 172.21.15.1:0/2456729563 >> 172.21.15.1:0/2456729563 conn(0x7f10cc0face0 msgr2=0x7f10cc0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:04.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.876+0000 7f10d4600700 1 -- 172.21.15.1:0/2456729563 shutdown_connections 2024-02-28T17:04:04.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.876+0000 7f10d4600700 1 -- 172.21.15.1:0/2456729563 wait complete. 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d4600700 1 Processor -- start 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d4600700 1 -- start start 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d4600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10cc105170 0x7f10cc115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d4600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 0x7f10cc116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d4600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc116870 con 0x7f10cc105b10 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d4600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc10fb80 con 0x7f10cc105170 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 0x7f10cc116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 0x7f10cc116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38756/0 (socket says 172.21.15.1:38756) 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.877+0000 7f10d1200700 1 -- 172.21.15.1:0/946234909 learned_addr learned my addr 172.21.15.1:0/946234909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.878+0000 7f10d1c00700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10cc105170 0x7f10cc115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.878+0000 7f10d1200700 1 -- 172.21.15.1:0/946234909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10cc105170 msgr2=0x7f10cc115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.878+0000 7f10d1200700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10cc105170 0x7f10cc115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.878+0000 7f10d1200700 1 -- 172.21.15.1:0/946234909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b8005040 con 0x7f10cc105b10 2024-02-28T17:04:04.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.878+0000 7f10d1200700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 0x7f10cc116220 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009030 tx=0x7f10b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:04.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.878+0000 7f10c9600700 1 -- 172.21.15.1:0/946234909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10b800e040 con 0x7f10cc105b10 2024-02-28T17:04:04.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.879+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc10fe00 con 0x7f10cc105b10 2024-02-28T17:04:04.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.879+0000 7f10c9600700 1 -- 172.21.15.1:0/946234909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10b800bbf0 con 0x7f10cc105b10 2024-02-28T17:04:04.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.879+0000 7f10c9600700 1 -- 172.21.15.1:0/946234909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10b801c630 con 0x7f10cc105b10 2024-02-28T17:04:04.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.879+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc110340 con 0x7f10cc105b10 2024-02-28T17:04:04.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.880+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc107640 con 0x7f10cc105b10 2024-02-28T17:04:04.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.881+0000 7f10c9600700 1 -- 172.21.15.1:0/946234909 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f10b800bd60 con 0x7f10cc105b10 2024-02-28T17:04:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.882+0000 7f10c9600700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10b406d7f0 0x7f10b406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.882+0000 7f10c9600700 1 -- 172.21.15.1:0/946234909 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f10b808e2c0 con 0x7f10cc105b10 2024-02-28T17:04:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.882+0000 7f10d1c00700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10b406d7f0 0x7f10b406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.883+0000 7f10d1c00700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10b406d7f0 0x7f10b406fca0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f10c0004990 tx=0x7f10c0009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:04.887+0000 7f10c9600700 1 -- 172.21.15.1:0/946234909 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f10b805b620 con 0x7f10cc105b10 2024-02-28T17:04:05.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:04 smithi066 ceph-mon[85781]: pgmap v889: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:05.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:04 smithi066 ceph-mon[85781]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:05.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:04 smithi066 ceph-mon[85781]: from='client.35602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:05.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:04 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4081311358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:04:05.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:04 smithi001 ceph-mon[112940]: pgmap v889: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:05.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:04 smithi001 ceph-mon[112940]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:05.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:04 smithi001 ceph-mon[112940]: from='client.35602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:05.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:04 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4081311358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:04:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.150+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f10cc0fe900 con 0x7f10cc105b10 2024-02-28T17:04:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.151+0000 7f10c9600700 1 -- 172.21.15.1:0/946234909 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f10b805b1b0 con 0x7f10cc105b10 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:04:05.152 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:04:05.153 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:04:05.154 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:04:05.155 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:04:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.153+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10b406d7f0 msgr2=0x7f10b406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.153+0000 7f10d4600700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10b406d7f0 0x7f10b406fca0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f10c0004990 tx=0x7f10c0009ee0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.153+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 msgr2=0x7f10cc116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.153+0000 7f10d4600700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 0x7f10cc116220 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009030 tx=0x7f10b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.154+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 shutdown_connections 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.154+0000 7f10d4600700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10b406d7f0 0x7f10b406fca0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.154+0000 7f10d4600700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10cc105170 0x7f10cc115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.154+0000 7f10d4600700 1 --2- 172.21.15.1:0/946234909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10cc105b10 0x7f10cc116220 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.154+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 >> 172.21.15.1:0/946234909 conn(0x7f10cc0face0 msgr2=0x7f10cc0fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.154+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 shutdown_connections 2024-02-28T17:04:05.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.154+0000 7f10d4600700 1 -- 172.21.15.1:0/946234909 wait complete. 2024-02-28T17:04:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:04:05.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.310+0000 7f5edfe00700 1 -- 172.21.15.1:0/1730783255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed81095f0 msgr2=0x7f5ed81099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:05.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.310+0000 7f5edfe00700 1 --2- 172.21.15.1:0/1730783255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed81095f0 0x7f5ed81099c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc004940 tx=0x7f5ecc009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:05.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.310+0000 7f5edfe00700 1 -- 172.21.15.1:0/1730783255 shutdown_connections 2024-02-28T17:04:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.310+0000 7f5edfe00700 1 --2- 172.21.15.1:0/1730783255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ed8103dc0 0x7f5ed8104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.310+0000 7f5edfe00700 1 --2- 172.21.15.1:0/1730783255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed81095f0 0x7f5ed81099c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.310+0000 7f5edfe00700 1 -- 172.21.15.1:0/1730783255 >> 172.21.15.1:0/1730783255 conn(0x7f5ed8072d50 msgr2=0x7f5ed8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.310+0000 7f5edfe00700 1 -- 172.21.15.1:0/1730783255 shutdown_connections 2024-02-28T17:04:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.310+0000 7f5edfe00700 1 -- 172.21.15.1:0/1730783255 wait complete. 2024-02-28T17:04:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.311+0000 7f5edfe00700 1 Processor -- start 2024-02-28T17:04:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.311+0000 7f5edfe00700 1 -- start start 2024-02-28T17:04:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.311+0000 7f5edfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed8103dc0 0x7f5ed811a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.311+0000 7f5edfe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ed81095f0 0x7f5ed811a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.311+0000 7f5edfe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed811ac70 con 0x7f5ed8103dc0 2024-02-28T17:04:05.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.311+0000 7f5edfe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed8113f80 con 0x7f5ed81095f0 2024-02-28T17:04:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.312+0000 7f5edd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed8103dc0 0x7f5ed811a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.312+0000 7f5edd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed8103dc0 0x7f5ed811a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38766/0 (socket says 172.21.15.1:38766) 2024-02-28T17:04:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.312+0000 7f5edd400700 1 -- 172.21.15.1:0/3427859274 learned_addr learned my addr 172.21.15.1:0/3427859274 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.312+0000 7f5edca00700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ed81095f0 0x7f5ed811a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.312+0000 7f5edd400700 1 -- 172.21.15.1:0/3427859274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ed81095f0 msgr2=0x7f5ed811a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.312+0000 7f5edd400700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ed81095f0 0x7f5ed811a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.312+0000 7f5edd400700 1 -- 172.21.15.1:0/3427859274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc005040 con 0x7f5ed8103dc0 2024-02-28T17:04:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.312+0000 7f5edd400700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed8103dc0 0x7f5ed811a0e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009150 tx=0x7f5ecc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.313+0000 7f5ed5600700 1 -- 172.21.15.1:0/3427859274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ecc00e040 con 0x7f5ed8103dc0 2024-02-28T17:04:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.313+0000 7f5ed5600700 1 -- 172.21.15.1:0/3427859274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ecc00bbf0 con 0x7f5ed8103dc0 2024-02-28T17:04:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.313+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed8114200 con 0x7f5ed8103dc0 2024-02-28T17:04:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.313+0000 7f5ed5600700 1 -- 172.21.15.1:0/3427859274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ecc01c630 con 0x7f5ed8103dc0 2024-02-28T17:04:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.313+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed8114740 con 0x7f5ed8103dc0 2024-02-28T17:04:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.314+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed8106850 con 0x7f5ed8103dc0 2024-02-28T17:04:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.315+0000 7f5ed5600700 1 -- 172.21.15.1:0/3427859274 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5ecc00bd60 con 0x7f5ed8103dc0 2024-02-28T17:04:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.316+0000 7f5ed5600700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ec006d7f0 0x7f5ec006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.316+0000 7f5ed5600700 1 -- 172.21.15.1:0/3427859274 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5ecc08e1e0 con 0x7f5ed8103dc0 2024-02-28T17:04:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.316+0000 7f5edca00700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ec006d7f0 0x7f5ec006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.317+0000 7f5edca00700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ec006d7f0 0x7f5ec006fca0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8115980 tx=0x7f5ec4009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:05.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.321+0000 7f5ed5600700 1 -- 172.21.15.1:0/3427859274 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5ecc05b680 con 0x7f5ed8103dc0 2024-02-28T17:04:05.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.567+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ed804bf40 con 0x7f5ec006d7f0 2024-02-28T17:04:05.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.569+0000 7f5ed5600700 1 -- 172.21.15.1:0/3427859274 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5ed804bf40 con 0x7f5ec006d7f0 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:04:05.570 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:04:05.571 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:04:05.571 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ec006d7f0 msgr2=0x7f5ec006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ec006d7f0 0x7f5ec006fca0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8115980 tx=0x7f5ec4009ee0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed8103dc0 msgr2=0x7f5ed811a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed8103dc0 0x7f5ed811a0e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009150 tx=0x7f5ecc009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 shutdown_connections 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ec006d7f0 0x7f5ec006fca0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ed8103dc0 0x7f5ed811a0e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 --2- 172.21.15.1:0/3427859274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ed81095f0 0x7f5ed811a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.572+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 >> 172.21.15.1:0/3427859274 conn(0x7f5ed8072d50 msgr2=0x7f5ed8074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.573+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 shutdown_connections 2024-02-28T17:04:05.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.573+0000 7f5edfe00700 1 -- 172.21.15.1:0/3427859274 wait complete. 2024-02-28T17:04:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.726+0000 7fbaae200700 1 -- 172.21.15.1:0/3421116181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 msgr2=0x7fbaa80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.726+0000 7fbaae200700 1 --2- 172.21.15.1:0/3421116181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 0x7fbaa80fe3b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fba98009d10 tx=0x7fba98004d10 comp rx=0 tx=0).stop 2024-02-28T17:04:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.727+0000 7fbaae200700 1 -- 172.21.15.1:0/3421116181 shutdown_connections 2024-02-28T17:04:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.727+0000 7fbaae200700 1 --2- 172.21.15.1:0/3421116181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa80fe8f0 0x7fbaa8107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.727+0000 7fbaae200700 1 --2- 172.21.15.1:0/3421116181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 0x7fbaa80fe3b0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.727+0000 7fbaae200700 1 -- 172.21.15.1:0/3421116181 >> 172.21.15.1:0/3421116181 conn(0x7fbaa8071d90 msgr2=0x7fbaa8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.727+0000 7fbaae200700 1 -- 172.21.15.1:0/3421116181 shutdown_connections 2024-02-28T17:04:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.727+0000 7fbaae200700 1 -- 172.21.15.1:0/3421116181 wait complete. 2024-02-28T17:04:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.728+0000 7fbaae200700 1 Processor -- start 2024-02-28T17:04:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.728+0000 7fbaae200700 1 -- start start 2024-02-28T17:04:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.728+0000 7fbaae200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 0x7fbaa8115e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.728+0000 7fbaae200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa80fe8f0 0x7fbaa8116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.728+0000 7fbaae200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa81169b0 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.728+0000 7fbaae200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa810fcc0 con 0x7fbaa80fe8f0 2024-02-28T17:04:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.729+0000 7fbaa7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 0x7fbaa8115e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.729+0000 7fbaa7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 0x7fbaa8115e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38782/0 (socket says 172.21.15.1:38782) 2024-02-28T17:04:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.729+0000 7fbaa7400700 1 -- 172.21.15.1:0/2027934627 learned_addr learned my addr 172.21.15.1:0/2027934627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.729+0000 7fbaa6a00700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa80fe8f0 0x7fbaa8116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.729+0000 7fbaa7400700 1 -- 172.21.15.1:0/2027934627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa80fe8f0 msgr2=0x7fbaa8116360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.729+0000 7fbaa7400700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa80fe8f0 0x7fbaa8116360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.729+0000 7fbaa7400700 1 -- 172.21.15.1:0/2027934627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba98009940 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.730+0000 7fbaa7400700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 0x7fbaa8115e20 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fba98008dd0 tx=0x7fba98004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.730+0000 7fba9f400700 1 -- 172.21.15.1:0/2027934627 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba9800e040 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.730+0000 7fba9f400700 1 -- 172.21.15.1:0/2027934627 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba980128e0 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.730+0000 7fba9f400700 1 -- 172.21.15.1:0/2027934627 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba9801c880 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.730+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa810ff40 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.730+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa8110480 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.732+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa8106b20 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.733+0000 7fba9f400700 1 -- 172.21.15.1:0/2027934627 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fba98012410 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.734+0000 7fba9f400700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8c06d8b0 0x7fba8c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.734+0000 7fba9f400700 1 -- 172.21.15.1:0/2027934627 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fba9808ec60 con 0x7fbaa80fdfe0 2024-02-28T17:04:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.734+0000 7fbaa6a00700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8c06d8b0 0x7fba8c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.735+0000 7fbaa6a00700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8c06d8b0 0x7fba8c06fd60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbaa81116c0 tx=0x7fba94009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:05.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:05.739+0000 7fba9f400700 1 -- 172.21.15.1:0/2027934627 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba9805bfc0 con 0x7fbaa80fdfe0 2024-02-28T17:04:06.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:05 smithi066 ceph-mon[85781]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:06.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:05 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/946234909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:04:06.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:05 smithi001 ceph-mon[112940]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:06.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:05 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/946234909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:04:06.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.052+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbaa804bed0 con 0x7fbaa80fdfe0 2024-02-28T17:04:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.053+0000 7fba9f400700 1 -- 172.21.15.1:0/2027934627 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fba98004e10 con 0x7fbaa80fdfe0 2024-02-28T17:04:06.054 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:04:06.054 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:04:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:04:06.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:04:06.054 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:04:06.054 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:04:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:04:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:04:06.055 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:04:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:04:06.056 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:04:06.056 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:04:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8c06d8b0 msgr2=0x7fba8c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8c06d8b0 0x7fba8c06fd60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbaa81116c0 tx=0x7fba94009ee0 comp rx=0 tx=0).stop 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 msgr2=0x7fbaa8115e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 0x7fbaa8115e20 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fba98008dd0 tx=0x7fba98004d10 comp rx=0 tx=0).stop 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 shutdown_connections 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8c06d8b0 0x7fba8c06fd60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa80fdfe0 0x7fbaa8115e20 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 --2- 172.21.15.1:0/2027934627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa80fe8f0 0x7fbaa8116360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 >> 172.21.15.1:0/2027934627 conn(0x7fbaa8071d90 msgr2=0x7fbaa80fba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 shutdown_connections 2024-02-28T17:04:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:06.056+0000 7fbaae200700 1 -- 172.21.15.1:0/2027934627 wait complete. 2024-02-28T17:04:07.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:06 smithi066 ceph-mon[85781]: pgmap v890: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:07.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:06 smithi066 ceph-mon[85781]: from='client.35618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:07.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:06 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2027934627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:04:07.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:06 smithi001 ceph-mon[112940]: pgmap v890: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:07.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:06 smithi001 ceph-mon[112940]: from='client.35618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:07.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:06 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2027934627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:04:09.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:08 smithi066 ceph-mon[85781]: pgmap v891: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:09.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:08 smithi001 ceph-mon[112940]: pgmap v891: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:11.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:10 smithi001 ceph-mon[112940]: pgmap v892: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:11.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:10 smithi066 ceph-mon[85781]: pgmap v892: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:12.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:04:12.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:04:13.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:12 smithi001 ceph-mon[112940]: pgmap v893: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:13.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:12 smithi066 ceph-mon[85781]: pgmap v893: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:15.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:14 smithi066 ceph-mon[85781]: pgmap v894: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:15.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:14 smithi001 ceph-mon[112940]: pgmap v894: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:17.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:16 smithi066 ceph-mon[85781]: pgmap v895: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:17.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:16 smithi001 ceph-mon[112940]: pgmap v895: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:19.084 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:18 smithi001 ceph-mon[112940]: pgmap v896: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:19.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:18 smithi066 ceph-mon[85781]: pgmap v896: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:21.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:20 smithi066 ceph-mon[85781]: pgmap v897: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:21.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:20 smithi001 ceph-mon[112940]: pgmap v897: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:23.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:22 smithi066 ceph-mon[85781]: pgmap v898: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:23.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:22 smithi001 ceph-mon[112940]: pgmap v898: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:25.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:24 smithi066 ceph-mon[85781]: pgmap v899: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:25.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:24 smithi001 ceph-mon[112940]: pgmap v899: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:26.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:04:26.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:04:27.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:26 smithi066 ceph-mon[85781]: pgmap v900: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:27.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:26 smithi001 ceph-mon[112940]: pgmap v900: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:29.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:28 smithi066 ceph-mon[85781]: pgmap v901: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:29.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:28 smithi001 ceph-mon[112940]: pgmap v901: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:31.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:30 smithi066 ceph-mon[85781]: pgmap v902: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:31.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:30 smithi001 ceph-mon[112940]: pgmap v902: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:32.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:31 smithi066 ceph-mon[85781]: pgmap v903: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:32.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:31 smithi001 ceph-mon[112940]: pgmap v903: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:34.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:34 smithi066 ceph-mon[85781]: pgmap v904: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:34.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:34 smithi001 ceph-mon[112940]: pgmap v904: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:36.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.229+0000 7f4cf7200700 1 -- 172.21.15.1:0/1692300247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 msgr2=0x7f4cf0106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:36.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.229+0000 7f4cf7200700 1 --2- 172.21.15.1:0/1692300247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 0x7f4cf0106e10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0004940 tx=0x7f4ce0009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:36.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.229+0000 7f4cf7200700 1 -- 172.21.15.1:0/1692300247 shutdown_connections 2024-02-28T17:04:36.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.229+0000 7f4cf7200700 1 --2- 172.21.15.1:0/1692300247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 0x7f4cf0106e10 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.229+0000 7f4cf7200700 1 --2- 172.21.15.1:0/1692300247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cf00fde30 0x7f4cf00fe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.229+0000 7f4cf7200700 1 -- 172.21.15.1:0/1692300247 >> 172.21.15.1:0/1692300247 conn(0x7f4cf00f78a0 msgr2=0x7f4cf00f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:36.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.229+0000 7f4cf7200700 1 -- 172.21.15.1:0/1692300247 shutdown_connections 2024-02-28T17:04:36.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.229+0000 7f4cf7200700 1 -- 172.21.15.1:0/1692300247 wait complete. 2024-02-28T17:04:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.230+0000 7f4cf7200700 1 Processor -- start 2024-02-28T17:04:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.231+0000 7f4cf7200700 1 -- start start 2024-02-28T17:04:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.231+0000 7f4cf7200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cf00fde30 0x7f4cf0115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.231+0000 7f4cf7200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 0x7f4cf0115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.231+0000 7f4cf7200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf0116620 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.231+0000 7f4cf7200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf010f950 con 0x7f4cf00fde30 2024-02-28T17:04:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.231+0000 7f4cf5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 0x7f4cf0115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.231+0000 7f4cf5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 0x7f4cf0115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36758/0 (socket says 172.21.15.1:36758) 2024-02-28T17:04:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.232+0000 7f4cf5400700 1 -- 172.21.15.1:0/1780513260 learned_addr learned my addr 172.21.15.1:0/1780513260 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.232+0000 7f4cf5e00700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cf00fde30 0x7f4cf0115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.232+0000 7f4cf5400700 1 -- 172.21.15.1:0/1780513260 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cf00fde30 msgr2=0x7f4cf0115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.232+0000 7f4cf5400700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cf00fde30 0x7f4cf0115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.232+0000 7f4cf5400700 1 -- 172.21.15.1:0/1780513260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce0005040 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.232+0000 7f4cf5400700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 0x7f4cf0115fd0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009100 tx=0x7f4ce0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:36.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.233+0000 7f4cea000700 1 -- 172.21.15.1:0/1780513260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ce000e040 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.233+0000 7f4cea000700 1 -- 172.21.15.1:0/1780513260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ce0004290 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.233+0000 7f4cea000700 1 -- 172.21.15.1:0/1780513260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ce001c630 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.233+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf010fbd0 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.233+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf0110110 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.234+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf0106880 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.235+0000 7f4cea000700 1 -- 172.21.15.1:0/1780513260 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4ce000bbf0 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.236+0000 7f4cea000700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4cd806d7e0 0x7f4cd806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:36.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.236+0000 7f4cea000700 1 -- 172.21.15.1:0/1780513260 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4ce008e200 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.237+0000 7f4cf5e00700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4cd806d7e0 0x7f4cd806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:36.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.237+0000 7f4cf5e00700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4cd806d7e0 0x7f4cd806fc90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4004c40 tx=0x7f4ce4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:36.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.241+0000 7f4cea000700 1 -- 172.21.15.1:0/1780513260 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ce005b5e0 con 0x7f4cf00fe7d0 2024-02-28T17:04:36.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.488+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cf01110e0 con 0x7f4cd806d7e0 2024-02-28T17:04:36.490 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:36 smithi001 ceph-mon[112940]: pgmap v905: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:36.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.491+0000 7f4cea000700 1 -- 172.21.15.1:0/1780513260 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4cf01110e0 con 0x7f4cd806d7e0 2024-02-28T17:04:36.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4cd806d7e0 msgr2=0x7f4cd806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:36.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4cd806d7e0 0x7f4cd806fc90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4004c40 tx=0x7f4ce4009040 comp rx=0 tx=0).stop 2024-02-28T17:04:36.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 msgr2=0x7f4cf0115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:36.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 0x7f4cf0115fd0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009100 tx=0x7f4ce0009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:36.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 shutdown_connections 2024-02-28T17:04:36.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4cd806d7e0 0x7f4cd806fc90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4cf00fde30 0x7f4cf0115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 --2- 172.21.15.1:0/1780513260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cf00fe7d0 0x7f4cf0115fd0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.494+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 >> 172.21.15.1:0/1780513260 conn(0x7f4cf00f78a0 msgr2=0x7f4cf00fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.495+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 shutdown_connections 2024-02-28T17:04:36.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.495+0000 7f4cf7200700 1 -- 172.21.15.1:0/1780513260 wait complete. 2024-02-28T17:04:36.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:36 smithi066 ceph-mon[85781]: pgmap v905: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:36.512 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:04:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.658+0000 7fcd56a00700 1 -- 172.21.15.1:0/3719755463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 msgr2=0x7fcd50103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.658+0000 7fcd56a00700 1 --2- 172.21.15.1:0/3719755463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 0x7fcd50103140 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fcd44004940 tx=0x7fcd44009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.658+0000 7fcd56a00700 1 -- 172.21.15.1:0/3719755463 shutdown_connections 2024-02-28T17:04:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.658+0000 7fcd56a00700 1 --2- 172.21.15.1:0/3719755463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd50103710 0x7fcd500fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.658+0000 7fcd56a00700 1 --2- 172.21.15.1:0/3719755463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 0x7fcd50103140 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.658+0000 7fcd56a00700 1 -- 172.21.15.1:0/3719755463 >> 172.21.15.1:0/3719755463 conn(0x7fcd500f88e0 msgr2=0x7fcd500facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.658+0000 7fcd56a00700 1 -- 172.21.15.1:0/3719755463 shutdown_connections 2024-02-28T17:04:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.658+0000 7fcd56a00700 1 -- 172.21.15.1:0/3719755463 wait complete. 2024-02-28T17:04:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.659+0000 7fcd56a00700 1 Processor -- start 2024-02-28T17:04:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.659+0000 7fcd56a00700 1 -- start start 2024-02-28T17:04:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.659+0000 7fcd56a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 0x7fcd50070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd56a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd50103710 0x7fcd500707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd56a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd50070dd0 con 0x7fcd50102d70 2024-02-28T17:04:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd56a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd50070f40 con 0x7fcd50103710 2024-02-28T17:04:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd55600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 0x7fcd50070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd55600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 0x7fcd50070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36770/0 (socket says 172.21.15.1:36770) 2024-02-28T17:04:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd55600700 1 -- 172.21.15.1:0/1803959352 learned_addr learned my addr 172.21.15.1:0/1803959352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd55600700 1 -- 172.21.15.1:0/1803959352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd50103710 msgr2=0x7fcd500707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd55600700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd50103710 0x7fcd500707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.660+0000 7fcd55600700 1 -- 172.21.15.1:0/1803959352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44005040 con 0x7fcd50102d70 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.661+0000 7fcd55600700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 0x7fcd50070290 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009150 tx=0x7fcd44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.661+0000 7fcd4d600700 1 -- 172.21.15.1:0/1803959352 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd4400e040 con 0x7fcd50102d70 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.661+0000 7fcd4d600700 1 -- 172.21.15.1:0/1803959352 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd4400bbf0 con 0x7fcd50102d70 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.661+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd50119a80 con 0x7fcd50102d70 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.661+0000 7fcd4d600700 1 -- 172.21.15.1:0/1803959352 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd4401c630 con 0x7fcd50102d70 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.661+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd50119f60 con 0x7fcd50102d70 2024-02-28T17:04:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.662+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5005f5c0 con 0x7fcd50102d70 2024-02-28T17:04:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.664+0000 7fcd4d600700 1 -- 172.21.15.1:0/1803959352 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcd4400bd60 con 0x7fcd50102d70 2024-02-28T17:04:36.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.665+0000 7fcd4d600700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd3806d7f0 0x7fcd3806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:36.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.665+0000 7fcd4d600700 1 -- 172.21.15.1:0/1803959352 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcd4408e2c0 con 0x7fcd50102d70 2024-02-28T17:04:36.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.665+0000 7fcd54c00700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd3806d7f0 0x7fcd3806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:36.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.666+0000 7fcd54c00700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd3806d7f0 0x7fcd3806fca0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcd50071c90 tx=0x7fcd3c009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.670+0000 7fcd4d600700 1 -- 172.21.15.1:0/1803959352 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcd44057b70 con 0x7fcd50102d70 2024-02-28T17:04:36.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.915+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd500faa20 con 0x7fcd3806d7f0 2024-02-28T17:04:36.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.918+0000 7fcd4d600700 1 -- 172.21.15.1:0/1803959352 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcd500faa20 con 0x7fcd3806d7f0 2024-02-28T17:04:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd3806d7f0 msgr2=0x7fcd3806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd3806d7f0 0x7fcd3806fca0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcd50071c90 tx=0x7fcd3c009ce0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 msgr2=0x7fcd50070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 0x7fcd50070290 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009150 tx=0x7fcd44009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 shutdown_connections 2024-02-28T17:04:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd3806d7f0 0x7fcd3806fca0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50102d70 0x7fcd50070290 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 --2- 172.21.15.1:0/1803959352 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd50103710 0x7fcd500707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.920+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 >> 172.21.15.1:0/1803959352 conn(0x7fcd500f88e0 msgr2=0x7fcd500fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.921+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 shutdown_connections 2024-02-28T17:04:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:36.921+0000 7fcd56a00700 1 -- 172.21.15.1:0/1803959352 wait complete. 2024-02-28T17:04:37.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.075+0000 7f3a9b200700 1 -- 172.21.15.1:0/2574796302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a94102f50 msgr2=0x7f3a94103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.075+0000 7f3a9b200700 1 --2- 172.21.15.1:0/2574796302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a94102f50 0x7f3a94103320 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3a84004940 tx=0x7f3a84009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:37.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.075+0000 7f3a9b200700 1 -- 172.21.15.1:0/2574796302 shutdown_connections 2024-02-28T17:04:37.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.075+0000 7f3a9b200700 1 --2- 172.21.15.1:0/2574796302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a941038f0 0x7f3a940fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.075+0000 7f3a9b200700 1 --2- 172.21.15.1:0/2574796302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a94102f50 0x7f3a94103320 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.075+0000 7f3a9b200700 1 -- 172.21.15.1:0/2574796302 >> 172.21.15.1:0/2574796302 conn(0x7f3a940f8ae0 msgr2=0x7f3a940faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:37.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.075+0000 7f3a9b200700 1 -- 172.21.15.1:0/2574796302 shutdown_connections 2024-02-28T17:04:37.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.075+0000 7f3a9b200700 1 -- 172.21.15.1:0/2574796302 wait complete. 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.076+0000 7f3a9b200700 1 Processor -- start 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.076+0000 7f3a9b200700 1 -- start start 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.077+0000 7f3a9b200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a94102f50 0x7f3a94115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.077+0000 7f3a9b200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941038f0 0x7f3a941161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.077+0000 7f3a9b200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a94116810 con 0x7f3a941038f0 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.077+0000 7f3a9b200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9410fb40 con 0x7f3a94102f50 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.077+0000 7f3a93400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941038f0 0x7f3a941161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.077+0000 7f3a93400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941038f0 0x7f3a941161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36788/0 (socket says 172.21.15.1:36788) 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.077+0000 7f3a93400700 1 -- 172.21.15.1:0/627121521 learned_addr learned my addr 172.21.15.1:0/627121521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.077+0000 7f3a93e00700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a94102f50 0x7f3a94115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:37.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a93400700 1 -- 172.21.15.1:0/627121521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a94102f50 msgr2=0x7f3a94115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a93400700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a94102f50 0x7f3a94115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a93400700 1 -- 172.21.15.1:0/627121521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a84005040 con 0x7f3a941038f0 2024-02-28T17:04:37.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a93400700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941038f0 0x7f3a941161c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3a80002a50 tx=0x7f3a80004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:37.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a8be00700 1 -- 172.21.15.1:0/627121521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a80007cc0 con 0x7f3a941038f0 2024-02-28T17:04:37.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9410fe20 con 0x7f3a941038f0 2024-02-28T17:04:37.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a8be00700 1 -- 172.21.15.1:0/627121521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a8000f040 con 0x7f3a941038f0 2024-02-28T17:04:37.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a8be00700 1 -- 172.21.15.1:0/627121521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a8000daf0 con 0x7f3a941038f0 2024-02-28T17:04:37.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.078+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a94110360 con 0x7f3a941038f0 2024-02-28T17:04:37.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.080+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a940fc6e0 con 0x7f3a941038f0 2024-02-28T17:04:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.081+0000 7f3a8be00700 1 -- 172.21.15.1:0/627121521 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3a80009810 con 0x7f3a941038f0 2024-02-28T17:04:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.082+0000 7f3a8be00700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3a7806d7f0 0x7f3a7806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.082+0000 7f3a8be00700 1 -- 172.21.15.1:0/627121521 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3a8008ccf0 con 0x7f3a941038f0 2024-02-28T17:04:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.082+0000 7f3a93e00700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3a7806d7f0 0x7f3a7806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:37.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.083+0000 7f3a93e00700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3a7806d7f0 0x7f3a7806fca0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400b020 tx=0x7f3a84009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.087+0000 7f3a8be00700 1 -- 172.21.15.1:0/627121521 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3a80059fd0 con 0x7f3a941038f0 2024-02-28T17:04:37.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:37 smithi001 ceph-mon[112940]: from='client.35626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:37.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.324+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a94111260 con 0x7f3a7806d7f0 2024-02-28T17:04:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.351+0000 7f3a8be00700 1 -- 172.21.15.1:0/627121521 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3a94111260 con 0x7f3a7806d7f0 2024-02-28T17:04:37.353 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (30m) 7m ago 44m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (27m) 7m ago 45m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (27m) 7m ago 43m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (30m) 7m ago 44m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (23m) 7m ago 37m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (22m) 7m ago 37m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (37m) 7m ago 37m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (7m) 7m ago 37m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (33m) 7m ago 46m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (32m) 7m ago 43m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (29m) 7m ago 46m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (28m) 7m ago 43m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:04:37.354 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (31m) 7m ago 45m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:04:37.355 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (31m) 7m ago 43m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:04:37.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (26m) 7m ago 42m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:04:37.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (26m) 7m ago 42m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:04:37.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (25m) 7m ago 41m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:04:37.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (25m) 7m ago 40m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:04:37.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (24m) 7m ago 40m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:04:37.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (23m) 7m ago 39m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:04:37.355 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (31m) 7m ago 44m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:04:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3a7806d7f0 msgr2=0x7f3a7806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3a7806d7f0 0x7f3a7806fca0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400b020 tx=0x7f3a84009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941038f0 msgr2=0x7f3a941161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941038f0 0x7f3a941161c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3a80002a50 tx=0x7f3a80004600 comp rx=0 tx=0).stop 2024-02-28T17:04:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 shutdown_connections 2024-02-28T17:04:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3a7806d7f0 0x7f3a7806fca0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3a94102f50 0x7f3a94115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 --2- 172.21.15.1:0/627121521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941038f0 0x7f3a941161c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 >> 172.21.15.1:0/627121521 conn(0x7f3a940f8ae0 msgr2=0x7f3a940fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 shutdown_connections 2024-02-28T17:04:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.354+0000 7f3a9b200700 1 -- 172.21.15.1:0/627121521 wait complete. 2024-02-28T17:04:37.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.507+0000 7f5b4ec00700 1 -- 172.21.15.1:0/2464112985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105150 msgr2=0x7f5b48105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.507+0000 7f5b4ec00700 1 --2- 172.21.15.1:0/2464112985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105150 0x7f5b48105520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5b38004940 tx=0x7f5b38009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:37.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.507+0000 7f5b4ec00700 1 -- 172.21.15.1:0/2464112985 shutdown_connections 2024-02-28T17:04:37.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.507+0000 7f5b4ec00700 1 --2- 172.21.15.1:0/2464112985 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b48105af0 0x7f5b480ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.507+0000 7f5b4ec00700 1 --2- 172.21.15.1:0/2464112985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105150 0x7f5b48105520 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.507+0000 7f5b4ec00700 1 -- 172.21.15.1:0/2464112985 >> 172.21.15.1:0/2464112985 conn(0x7f5b480face0 msgr2=0x7f5b480fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:37.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.507+0000 7f5b4ec00700 1 -- 172.21.15.1:0/2464112985 shutdown_connections 2024-02-28T17:04:37.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.507+0000 7f5b4ec00700 1 -- 172.21.15.1:0/2464112985 wait complete. 2024-02-28T17:04:37.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:37 smithi066 ceph-mon[85781]: from='client.35626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:37.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.508+0000 7f5b4ec00700 1 Processor -- start 2024-02-28T17:04:37.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.508+0000 7f5b4ec00700 1 -- start start 2024-02-28T17:04:37.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.508+0000 7f5b4ec00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b48105150 0x7f5b48115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.508+0000 7f5b4ec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105af0 0x7f5b481161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.508+0000 7f5b4ec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b48116800 con 0x7f5b48105af0 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.508+0000 7f5b4ec00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4810fb30 con 0x7f5b48105150 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.508+0000 7f5b47400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105af0 0x7f5b481161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b47400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105af0 0x7f5b481161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36802/0 (socket says 172.21.15.1:36802) 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b47400700 1 -- 172.21.15.1:0/906792466 learned_addr learned my addr 172.21.15.1:0/906792466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b47e00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b48105150 0x7f5b48115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b47400700 1 -- 172.21.15.1:0/906792466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b48105150 msgr2=0x7f5b48115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b47400700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b48105150 0x7f5b48115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b47400700 1 -- 172.21.15.1:0/906792466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b38005040 con 0x7f5b48105af0 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b47400700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105af0 0x7f5b481161b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5b34002a50 tx=0x7f5b34004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:37.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b3fe00700 1 -- 172.21.15.1:0/906792466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b34007cc0 con 0x7f5b48105af0 2024-02-28T17:04:37.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b3fe00700 1 -- 172.21.15.1:0/906792466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b3400f040 con 0x7f5b48105af0 2024-02-28T17:04:37.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4810fe10 con 0x7f5b48105af0 2024-02-28T17:04:37.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.509+0000 7f5b3fe00700 1 -- 172.21.15.1:0/906792466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b3400daf0 con 0x7f5b48105af0 2024-02-28T17:04:37.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.510+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b48110350 con 0x7f5b48105af0 2024-02-28T17:04:37.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.511+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b480fe8e0 con 0x7f5b48105af0 2024-02-28T17:04:37.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.512+0000 7f5b3fe00700 1 -- 172.21.15.1:0/906792466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5b34009810 con 0x7f5b48105af0 2024-02-28T17:04:37.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.512+0000 7f5b3fe00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b2c06d7f0 0x7f5b2c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.512+0000 7f5b3fe00700 1 -- 172.21.15.1:0/906792466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5b3408cc20 con 0x7f5b48105af0 2024-02-28T17:04:37.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.513+0000 7f5b47e00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b2c06d7f0 0x7f5b2c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:37.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.513+0000 7f5b47e00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b2c06d7f0 0x7f5b2c06fca0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800b020 tx=0x7f5b38009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:37.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.517+0000 7f5b3fe00700 1 -- 172.21.15.1:0/906792466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5b34059f80 con 0x7f5b48105af0 2024-02-28T17:04:37.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.816+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5b481111e0 con 0x7f5b48105af0 2024-02-28T17:04:37.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.817+0000 7f5b3fe00700 1 -- 172.21.15.1:0/906792466 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f5b34059b10 con 0x7f5b48105af0 2024-02-28T17:04:37.819 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:04:37.819 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:04:37.819 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:04:37.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:04:37.819 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:04:37.820 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:04:37.821 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:04:37.821 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:04:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.820+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b2c06d7f0 msgr2=0x7f5b2c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.820+0000 7f5b4ec00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b2c06d7f0 0x7f5b2c06fca0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800b020 tx=0x7f5b38009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.820+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105af0 msgr2=0x7f5b481161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.820+0000 7f5b4ec00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105af0 0x7f5b481161b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5b34002a50 tx=0x7f5b34004600 comp rx=0 tx=0).stop 2024-02-28T17:04:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.821+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 shutdown_connections 2024-02-28T17:04:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.821+0000 7f5b4ec00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b2c06d7f0 0x7f5b2c06fca0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.821+0000 7f5b4ec00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b48105150 0x7f5b48115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.821+0000 7f5b4ec00700 1 --2- 172.21.15.1:0/906792466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b48105af0 0x7f5b481161b0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.821+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 >> 172.21.15.1:0/906792466 conn(0x7f5b480face0 msgr2=0x7f5b480fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.821+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 shutdown_connections 2024-02-28T17:04:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.821+0000 7f5b4ec00700 1 -- 172.21.15.1:0/906792466 wait complete. 2024-02-28T17:04:37.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.976+0000 7f7284c00700 1 -- 172.21.15.1:0/2528359490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 msgr2=0x7f72800fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.976+0000 7f7284c00700 1 --2- 172.21.15.1:0/2528359490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 0x7f72800fe3b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7270004970 tx=0x7f7270009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.976+0000 7f7284c00700 1 -- 172.21.15.1:0/2528359490 shutdown_connections 2024-02-28T17:04:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.976+0000 7f7284c00700 1 --2- 172.21.15.1:0/2528359490 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72800fe8f0 0x7f7280107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.976+0000 7f7284c00700 1 --2- 172.21.15.1:0/2528359490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 0x7f72800fe3b0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.976+0000 7f7284c00700 1 -- 172.21.15.1:0/2528359490 >> 172.21.15.1:0/2528359490 conn(0x7f7280071d90 msgr2=0x7f7280072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.976+0000 7f7284c00700 1 -- 172.21.15.1:0/2528359490 shutdown_connections 2024-02-28T17:04:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.976+0000 7f7284c00700 1 -- 172.21.15.1:0/2528359490 wait complete. 2024-02-28T17:04:37.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.977+0000 7f7284c00700 1 Processor -- start 2024-02-28T17:04:37.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.977+0000 7f7284c00700 1 -- start start 2024-02-28T17:04:37.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.978+0000 7f7284c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 0x7f7280115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.978+0000 7f7284c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72800fe8f0 0x7f7280116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.978+0000 7f7284c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7280116900 con 0x7f72800fdfe0 2024-02-28T17:04:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.978+0000 7f7284c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728010fb80 con 0x7f72800fe8f0 2024-02-28T17:04:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.978+0000 7f727de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 0x7f7280115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.978+0000 7f727de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 0x7f7280115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36822/0 (socket says 172.21.15.1:36822) 2024-02-28T17:04:37.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.978+0000 7f727de00700 1 -- 172.21.15.1:0/3195123633 learned_addr learned my addr 172.21.15.1:0/3195123633 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.979+0000 7f727de00700 1 -- 172.21.15.1:0/3195123633 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72800fe8f0 msgr2=0x7f7280116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.979+0000 7f727de00700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72800fe8f0 0x7f7280116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.979+0000 7f727de00700 1 -- 172.21.15.1:0/3195123633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7270005040 con 0x7f72800fdfe0 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.979+0000 7f727de00700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 0x7f7280115ce0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7270009150 tx=0x7f7270009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.979+0000 7f7276000700 1 -- 172.21.15.1:0/3195123633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f727000e040 con 0x7f72800fdfe0 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.979+0000 7f7276000700 1 -- 172.21.15.1:0/3195123633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7270004290 con 0x7f72800fdfe0 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.979+0000 7f7276000700 1 -- 172.21.15.1:0/3195123633 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f727001c630 con 0x7f72800fdfe0 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.979+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728010fda0 con 0x7f72800fdfe0 2024-02-28T17:04:37.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.980+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72801102e0 con 0x7f72800fdfe0 2024-02-28T17:04:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.981+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7280106b20 con 0x7f72800fdfe0 2024-02-28T17:04:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.982+0000 7f7276000700 1 -- 172.21.15.1:0/3195123633 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7270013980 con 0x7f72800fdfe0 2024-02-28T17:04:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.984+0000 7f7276000700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f726406d7f0 0x7f726406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.984+0000 7f7276000700 1 -- 172.21.15.1:0/3195123633 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f727008e180 con 0x7f72800fdfe0 2024-02-28T17:04:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.984+0000 7f727d400700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f726406d7f0 0x7f726406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.985+0000 7f727d400700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f726406d7f0 0x7f726406fca0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f726c006fd0 tx=0x7f726c006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:37.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:37.989+0000 7f7276000700 1 -- 172.21.15.1:0/3195123633 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f727005b560 con 0x7f72800fdfe0 2024-02-28T17:04:38.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.262+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f72801112c0 con 0x7f72800fdfe0 2024-02-28T17:04:38.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.263+0000 7f7276000700 1 -- 172.21.15.1:0/3195123633 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f727005b0f0 con 0x7f72800fdfe0 2024-02-28T17:04:38.265 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:04:38.265 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:04:38.265 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:04:38.266 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:04:38.266 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:04:38.266 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:04:38.266 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:04:38.266 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:04:38.266 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:04:38.266 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:04:38.266 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:04:38.267 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:04:38.267 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:04:38.267 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:04:38.267 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:04:38.267 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:04:38.268 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:04:38.268 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:04:38.268 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:04:38.268 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:04:38.268 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:04:38.268 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:04:38.269 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:04:38.269 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:04:38.269 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:04:38.269 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:04:38.269 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:04:38.269 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:04:38.270 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:04:38.270 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:04:38.270 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:04:38.270 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:04:38.270 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:04:38.270 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:04:38.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:04:38.271 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:04:38.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:04:38.271 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:04:38.271 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:04:38.271 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:04:38.271 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:04:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f726406d7f0 msgr2=0x7f726406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f726406d7f0 0x7f726406fca0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f726c006fd0 tx=0x7f726c006db0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 msgr2=0x7f7280115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 0x7f7280115ce0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7270009150 tx=0x7f7270009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 shutdown_connections 2024-02-28T17:04:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f726406d7f0 0x7f726406fca0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72800fdfe0 0x7f7280115ce0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 --2- 172.21.15.1:0/3195123633 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72800fe8f0 0x7f7280116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.266+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 >> 172.21.15.1:0/3195123633 conn(0x7f7280071d90 msgr2=0x7f72800fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.267+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 shutdown_connections 2024-02-28T17:04:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.267+0000 7f7284c00700 1 -- 172.21.15.1:0/3195123633 wait complete. 2024-02-28T17:04:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:04:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.423+0000 7fbe7d400700 1 -- 172.21.15.1:0/2332077343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78070870 msgr2=0x7fbe78070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.423+0000 7fbe7d400700 1 --2- 172.21.15.1:0/2332077343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78070870 0x7fbe78070ce0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fbe64004970 tx=0x7fbe64009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.423+0000 7fbe7d400700 1 -- 172.21.15.1:0/2332077343 shutdown_connections 2024-02-28T17:04:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.423+0000 7fbe7d400700 1 --2- 172.21.15.1:0/2332077343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78070870 0x7fbe78070ce0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.423+0000 7fbe7d400700 1 --2- 172.21.15.1:0/2332077343 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe78071f60 0x7fbe78070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.423+0000 7fbe7d400700 1 -- 172.21.15.1:0/2332077343 >> 172.21.15.1:0/2332077343 conn(0x7fbe780f8ae0 msgr2=0x7fbe780faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.424+0000 7fbe7d400700 1 -- 172.21.15.1:0/2332077343 shutdown_connections 2024-02-28T17:04:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.424+0000 7fbe7d400700 1 -- 172.21.15.1:0/2332077343 wait complete. 2024-02-28T17:04:38.426 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:38 smithi001 ceph-mon[112940]: from='client.35630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:38.426 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:38 smithi001 ceph-mon[112940]: pgmap v906: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:38.426 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:38 smithi001 ceph-mon[112940]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:38.426 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:38 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/906792466' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:04:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.424+0000 7fbe7d400700 1 Processor -- start 2024-02-28T17:04:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.425+0000 7fbe7d400700 1 -- start start 2024-02-28T17:04:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.425+0000 7fbe7d400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe78070870 0x7fbe7811a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.425+0000 7fbe7d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78071f60 0x7fbe7811a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.425+0000 7fbe7d400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7811ad50 con 0x7fbe78071f60 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.425+0000 7fbe7d400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe78113fd0 con 0x7fbe78070870 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.425+0000 7fbe75e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78071f60 0x7fbe7811a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.425+0000 7fbe75e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78071f60 0x7fbe7811a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36850/0 (socket says 172.21.15.1:36850) 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.425+0000 7fbe75e00700 1 -- 172.21.15.1:0/3929022077 learned_addr learned my addr 172.21.15.1:0/3929022077 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.426+0000 7fbe76800700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe78070870 0x7fbe7811a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.426+0000 7fbe75e00700 1 -- 172.21.15.1:0/3929022077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe78070870 msgr2=0x7fbe7811a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.426+0000 7fbe75e00700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe78070870 0x7fbe7811a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.426+0000 7fbe75e00700 1 -- 172.21.15.1:0/3929022077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64005040 con 0x7fbe78071f60 2024-02-28T17:04:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.426+0000 7fbe75e00700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78071f60 0x7fbe7811a670 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fbe64004f50 tx=0x7fbe64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:38.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.426+0000 7fbe6ea00700 1 -- 172.21.15.1:0/3929022077 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe6400e040 con 0x7fbe78071f60 2024-02-28T17:04:38.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.427+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe781141f0 con 0x7fbe78071f60 2024-02-28T17:04:38.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.427+0000 7fbe6ea00700 1 -- 172.21.15.1:0/3929022077 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe6400bc10 con 0x7fbe78071f60 2024-02-28T17:04:38.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.427+0000 7fbe6ea00700 1 -- 172.21.15.1:0/3929022077 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe6401c630 con 0x7fbe78071f60 2024-02-28T17:04:38.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.427+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe78114730 con 0x7fbe78071f60 2024-02-28T17:04:38.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.428+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe780fe400 con 0x7fbe78071f60 2024-02-28T17:04:38.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.429+0000 7fbe6ea00700 1 -- 172.21.15.1:0/3929022077 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbe6400bd80 con 0x7fbe78071f60 2024-02-28T17:04:38.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.430+0000 7fbe6ea00700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe5c06d7f0 0x7fbe5c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:38.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.430+0000 7fbe6ea00700 1 -- 172.21.15.1:0/3929022077 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbe6408e200 con 0x7fbe78071f60 2024-02-28T17:04:38.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.431+0000 7fbe76800700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe5c06d7f0 0x7fbe5c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:38.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.431+0000 7fbe76800700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe5c06d7f0 0x7fbe5c06fca0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbe68006fd0 tx=0x7fbe68008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.435+0000 7fbe6ea00700 1 -- 172.21.15.1:0/3929022077 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe6405b5e0 con 0x7fbe78071f60 2024-02-28T17:04:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:38 smithi066 ceph-mon[85781]: from='client.35630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:38 smithi066 ceph-mon[85781]: pgmap v906: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:38 smithi066 ceph-mon[85781]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:38 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/906792466' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:04:38.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.680+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe78115710 con 0x7fbe5c06d7f0 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.682+0000 7fbe6ea00700 1 -- 172.21.15.1:0/3929022077 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbe78115710 con 0x7fbe5c06d7f0 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:04:38.684 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:04:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:04:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:04:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:04:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:04:38.685 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:04:38.685 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:04:38.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe5c06d7f0 msgr2=0x7fbe5c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:38.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe5c06d7f0 0x7fbe5c06fca0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbe68006fd0 tx=0x7fbe68008040 comp rx=0 tx=0).stop 2024-02-28T17:04:38.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78071f60 msgr2=0x7fbe7811a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:38.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78071f60 0x7fbe7811a670 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fbe64004f50 tx=0x7fbe64009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:38.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 shutdown_connections 2024-02-28T17:04:38.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbe5c06d7f0 0x7fbe5c06fca0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbe78070870 0x7fbe7811a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 --2- 172.21.15.1:0/3929022077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe78071f60 0x7fbe7811a670 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.685+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 >> 172.21.15.1:0/3929022077 conn(0x7fbe780f8ae0 msgr2=0x7fbe780fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:38.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.686+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 shutdown_connections 2024-02-28T17:04:38.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.686+0000 7fbe7d400700 1 -- 172.21.15.1:0/3929022077 wait complete. 2024-02-28T17:04:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.842+0000 7fe33f000700 1 -- 172.21.15.1:0/4079052724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105af0 msgr2=0x7fe3380ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.842+0000 7fe33f000700 1 --2- 172.21.15.1:0/4079052724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105af0 0x7fe3380ff160 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe324004940 tx=0x7fe324009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.842+0000 7fe33f000700 1 -- 172.21.15.1:0/4079052724 shutdown_connections 2024-02-28T17:04:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.842+0000 7fe33f000700 1 --2- 172.21.15.1:0/4079052724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105af0 0x7fe3380ff160 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.842+0000 7fe33f000700 1 --2- 172.21.15.1:0/4079052724 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe338105150 0x7fe338105520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.842+0000 7fe33f000700 1 -- 172.21.15.1:0/4079052724 >> 172.21.15.1:0/4079052724 conn(0x7fe3380face0 msgr2=0x7fe3380fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.842+0000 7fe33f000700 1 -- 172.21.15.1:0/4079052724 shutdown_connections 2024-02-28T17:04:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.842+0000 7fe33f000700 1 -- 172.21.15.1:0/4079052724 wait complete. 2024-02-28T17:04:38.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.843+0000 7fe33f000700 1 Processor -- start 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.843+0000 7fe33f000700 1 -- start start 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.843+0000 7fe33f000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105150 0x7fe338115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.843+0000 7fe33f000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe338105af0 0x7fe338116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.843+0000 7fe33f000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe338116880 con 0x7fe338105150 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.843+0000 7fe33f000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33810fb90 con 0x7fe338105af0 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.844+0000 7fe337e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105150 0x7fe338115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.844+0000 7fe337e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105150 0x7fe338115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36870/0 (socket says 172.21.15.1:36870) 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.844+0000 7fe337e00700 1 -- 172.21.15.1:0/3653479753 learned_addr learned my addr 172.21.15.1:0/3653479753 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.844+0000 7fe337e00700 1 -- 172.21.15.1:0/3653479753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe338105af0 msgr2=0x7fe338116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.844+0000 7fe337e00700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe338105af0 0x7fe338116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.844+0000 7fe337e00700 1 -- 172.21.15.1:0/3653479753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe324005040 con 0x7fe338105150 2024-02-28T17:04:38.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.844+0000 7fe337e00700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105150 0x7fe338115cf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe32800c8c0 tx=0x7fe3280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:38.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.845+0000 7fe32fe00700 1 -- 172.21.15.1:0/3653479753 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe32800a980 con 0x7fe338105150 2024-02-28T17:04:38.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.845+0000 7fe32fe00700 1 -- 172.21.15.1:0/3653479753 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe32800ce60 con 0x7fe338105150 2024-02-28T17:04:38.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.845+0000 7fe32fe00700 1 -- 172.21.15.1:0/3653479753 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3280115e0 con 0x7fe338105150 2024-02-28T17:04:38.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.845+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33810fe70 con 0x7fe338105150 2024-02-28T17:04:38.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.845+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3381103b0 con 0x7fe338105150 2024-02-28T17:04:38.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.846+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe338107620 con 0x7fe338105150 2024-02-28T17:04:38.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.847+0000 7fe32fe00700 1 -- 172.21.15.1:0/3653479753 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe328012400 con 0x7fe338105150 2024-02-28T17:04:38.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.848+0000 7fe32fe00700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31c06d7f0 0x7fe31c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:04:38.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.848+0000 7fe32fe00700 1 -- 172.21.15.1:0/3653479753 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe32808cfe0 con 0x7fe338105150 2024-02-28T17:04:38.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.848+0000 7fe337400700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31c06d7f0 0x7fe31c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:04:38.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.849+0000 7fe337400700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31c06d7f0 0x7fe31c06fca0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe324004f90 tx=0x7fe324009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:04:38.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:38.852+0000 7fe32fe00700 1 -- 172.21.15.1:0/3653479753 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe32805a3a0 con 0x7fe338105150 2024-02-28T17:04:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.158+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe3380fe8e0 con 0x7fe338105150 2024-02-28T17:04:39.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.159+0000 7fe32fe00700 1 -- 172.21.15.1:0/3653479753 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fe3380fe8e0 con 0x7fe338105150 2024-02-28T17:04:39.160 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:04:39.161 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:04:39.162 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:04:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.161+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31c06d7f0 msgr2=0x7fe31c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:39.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.161+0000 7fe33f000700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31c06d7f0 0x7fe31c06fca0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe324004f90 tx=0x7fe324009e30 comp rx=0 tx=0).stop 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.161+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105150 msgr2=0x7fe338115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.161+0000 7fe33f000700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105150 0x7fe338115cf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe32800c8c0 tx=0x7fe3280044c0 comp rx=0 tx=0).stop 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.161+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 shutdown_connections 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.161+0000 7fe33f000700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31c06d7f0 0x7fe31c06fca0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.161+0000 7fe33f000700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe338105150 0x7fe338115cf0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.161+0000 7fe33f000700 1 --2- 172.21.15.1:0/3653479753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe338105af0 0x7fe338116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.162+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 >> 172.21.15.1:0/3653479753 conn(0x7fe3380face0 msgr2=0x7fe3380fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.162+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 shutdown_connections 2024-02-28T17:04:39.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:04:39.162+0000 7fe33f000700 1 -- 172.21.15.1:0/3653479753 wait complete. 2024-02-28T17:04:39.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:39 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3195123633' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:04:39.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:39 smithi066 ceph-mon[85781]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:39.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:39 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3195123633' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:04:39.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:39 smithi001 ceph-mon[112940]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:04:40.412 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:40 smithi066 ceph-mon[85781]: pgmap v907: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:40.412 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3653479753' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:04:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:40 smithi001 ceph-mon[112940]: pgmap v907: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3653479753' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:04:41.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:04:41.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:04:42.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:42 smithi066 ceph-mon[85781]: pgmap v908: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:42.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:42 smithi001 ceph-mon[112940]: pgmap v908: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:44.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:44 smithi066 ceph-mon[85781]: pgmap v909: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:44.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:44 smithi001 ceph-mon[112940]: pgmap v909: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:46.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:46 smithi066 ceph-mon[85781]: pgmap v910: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:46.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:46 smithi001 ceph-mon[112940]: pgmap v910: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:48.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:48 smithi066 ceph-mon[85781]: pgmap v911: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:48.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:48 smithi001 ceph-mon[112940]: pgmap v911: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:50.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:50 smithi066 ceph-mon[85781]: pgmap v912: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:50.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:50 smithi001 ceph-mon[112940]: pgmap v912: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:52.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:52 smithi066 ceph-mon[85781]: pgmap v913: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:52.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:52 smithi001 ceph-mon[112940]: pgmap v913: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:54.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:54 smithi066 ceph-mon[85781]: pgmap v914: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:54.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:54 smithi001 ceph-mon[112940]: pgmap v914: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:56.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:56 smithi066 ceph-mon[85781]: pgmap v915: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:56.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:04:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:56 smithi001 ceph-mon[112940]: pgmap v915: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:04:58.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:04:58 smithi066 ceph-mon[85781]: pgmap v916: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:04:58.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:04:58 smithi001 ceph-mon[112940]: pgmap v916: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:00.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:00 smithi066 ceph-mon[85781]: pgmap v917: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:00.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:00 smithi001 ceph-mon[112940]: pgmap v917: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:02.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:02 smithi066 ceph-mon[85781]: pgmap v918: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:02.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:02 smithi001 ceph-mon[112940]: pgmap v918: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:04.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:04 smithi066 ceph-mon[85781]: pgmap v919: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:04.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:04 smithi001 ceph-mon[112940]: pgmap v919: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:06.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:06 smithi066 ceph-mon[85781]: pgmap v920: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:06.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:06 smithi001 ceph-mon[112940]: pgmap v920: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:08.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:08 smithi066 ceph-mon[85781]: pgmap v921: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:08.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:08 smithi001 ceph-mon[112940]: pgmap v921: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:09.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.334+0000 7fda92c00700 1 -- 172.21.15.1:0/913211095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 msgr2=0x7fda8c105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:09.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.334+0000 7fda92c00700 1 --2- 172.21.15.1:0/913211095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 0x7fda8c105530 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fda7c004940 tx=0x7fda7c009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:09.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.334+0000 7fda92c00700 1 -- 172.21.15.1:0/913211095 shutdown_connections 2024-02-28T17:05:09.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.334+0000 7fda92c00700 1 --2- 172.21.15.1:0/913211095 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda8c105b00 0x7fda8c0ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.334+0000 7fda92c00700 1 --2- 172.21.15.1:0/913211095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 0x7fda8c105530 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.334+0000 7fda92c00700 1 -- 172.21.15.1:0/913211095 >> 172.21.15.1:0/913211095 conn(0x7fda8c0facb0 msgr2=0x7fda8c0fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:09.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.334+0000 7fda92c00700 1 -- 172.21.15.1:0/913211095 shutdown_connections 2024-02-28T17:05:09.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.334+0000 7fda92c00700 1 -- 172.21.15.1:0/913211095 wait complete. 2024-02-28T17:05:09.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.335+0000 7fda92c00700 1 Processor -- start 2024-02-28T17:05:09.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.336+0000 7fda92c00700 1 -- start start 2024-02-28T17:05:09.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.336+0000 7fda92c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 0x7fda8c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:09.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.336+0000 7fda92c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda8c105b00 0x7fda8c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:09.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.336+0000 7fda92c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c116800 con 0x7fda8c105160 2024-02-28T17:05:09.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.336+0000 7fda92c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c10fb30 con 0x7fda8c105b00 2024-02-28T17:05:09.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.336+0000 7fda8be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 0x7fda8c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:09.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.336+0000 7fda8be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 0x7fda8c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56124/0 (socket says 172.21.15.1:56124) 2024-02-28T17:05:09.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.336+0000 7fda8be00700 1 -- 172.21.15.1:0/411796202 learned_addr learned my addr 172.21.15.1:0/411796202 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:09.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda8b400700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda8c105b00 0x7fda8c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:09.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda8be00700 1 -- 172.21.15.1:0/411796202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda8c105b00 msgr2=0x7fda8c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:09.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda8be00700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda8c105b00 0x7fda8c1161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda8be00700 1 -- 172.21.15.1:0/411796202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c005040 con 0x7fda8c105160 2024-02-28T17:05:09.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda8be00700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 0x7fda8c115c70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009150 tx=0x7fda7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:09.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda83e00700 1 -- 172.21.15.1:0/411796202 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda7c00e040 con 0x7fda8c105160 2024-02-28T17:05:09.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda83e00700 1 -- 172.21.15.1:0/411796202 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda7c00bbf0 con 0x7fda8c105160 2024-02-28T17:05:09.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c10fdb0 con 0x7fda8c105160 2024-02-28T17:05:09.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda83e00700 1 -- 172.21.15.1:0/411796202 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda7c01c630 con 0x7fda8c105160 2024-02-28T17:05:09.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.337+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c1102f0 con 0x7fda8c105160 2024-02-28T17:05:09.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.339+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c0fe8f0 con 0x7fda8c105160 2024-02-28T17:05:09.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.339+0000 7fda83e00700 1 -- 172.21.15.1:0/411796202 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fda7c00bd60 con 0x7fda8c105160 2024-02-28T17:05:09.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.341+0000 7fda83e00700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fda7006d7f0 0x7fda7006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:09.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.341+0000 7fda83e00700 1 -- 172.21.15.1:0/411796202 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fda7c08e380 con 0x7fda8c105160 2024-02-28T17:05:09.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.341+0000 7fda8b400700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fda7006d7f0 0x7fda7006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:09.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.342+0000 7fda8b400700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fda7006d7f0 0x7fda7006fca0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fda8c1114a0 tx=0x7fda7800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.346+0000 7fda83e00700 1 -- 172.21.15.1:0/411796202 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fda7c05b6e0 con 0x7fda8c105160 2024-02-28T17:05:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.590+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda8c1112c0 con 0x7fda7006d7f0 2024-02-28T17:05:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.592+0000 7fda83e00700 1 -- 172.21.15.1:0/411796202 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fda8c1112c0 con 0x7fda7006d7f0 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fda7006d7f0 msgr2=0x7fda7006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fda7006d7f0 0x7fda7006fca0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fda8c1114a0 tx=0x7fda7800b040 comp rx=0 tx=0).stop 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 msgr2=0x7fda8c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 0x7fda8c115c70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009150 tx=0x7fda7c009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 shutdown_connections 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fda7006d7f0 0x7fda7006fca0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8c105160 0x7fda8c115c70 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 --2- 172.21.15.1:0/411796202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fda8c105b00 0x7fda8c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 >> 172.21.15.1:0/411796202 conn(0x7fda8c0facb0 msgr2=0x7fda8c0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 shutdown_connections 2024-02-28T17:05:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.595+0000 7fda92c00700 1 -- 172.21.15.1:0/411796202 wait complete. 2024-02-28T17:05:09.612 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:05:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.757+0000 7f83b9200700 1 -- 172.21.15.1:0/358580713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fdfe0 msgr2=0x7f83b40fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.757+0000 7f83b9200700 1 --2- 172.21.15.1:0/358580713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fdfe0 0x7f83b40fe3b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009d10 tx=0x7f83a4004d10 comp rx=0 tx=0).stop 2024-02-28T17:05:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.757+0000 7f83b9200700 1 -- 172.21.15.1:0/358580713 shutdown_connections 2024-02-28T17:05:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.757+0000 7f83b9200700 1 --2- 172.21.15.1:0/358580713 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83b40fe8f0 0x7f83b4107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.757+0000 7f83b9200700 1 --2- 172.21.15.1:0/358580713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fdfe0 0x7f83b40fe3b0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.757+0000 7f83b9200700 1 -- 172.21.15.1:0/358580713 >> 172.21.15.1:0/358580713 conn(0x7f83b4071d90 msgr2=0x7f83b4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.757+0000 7f83b9200700 1 -- 172.21.15.1:0/358580713 shutdown_connections 2024-02-28T17:05:09.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.757+0000 7f83b9200700 1 -- 172.21.15.1:0/358580713 wait complete. 2024-02-28T17:05:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.758+0000 7f83b9200700 1 Processor -- start 2024-02-28T17:05:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.759+0000 7f83b9200700 1 -- start start 2024-02-28T17:05:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.759+0000 7f83b9200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83b40fdfe0 0x7f83b4115ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.759+0000 7f83b9200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fe8f0 0x7f83b41163e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.759+0000 7f83b9200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4116a30 con 0x7f83b40fe8f0 2024-02-28T17:05:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.759+0000 7f83b9200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b410fd40 con 0x7f83b40fdfe0 2024-02-28T17:05:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.759+0000 7f83b1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fe8f0 0x7f83b41163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.759+0000 7f83b1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fe8f0 0x7f83b41163e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56142/0 (socket says 172.21.15.1:56142) 2024-02-28T17:05:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.759+0000 7f83b1400700 1 -- 172.21.15.1:0/2996930672 learned_addr learned my addr 172.21.15.1:0/2996930672 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.760+0000 7f83b1e00700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83b40fdfe0 0x7f83b4115ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.760+0000 7f83b1400700 1 -- 172.21.15.1:0/2996930672 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83b40fdfe0 msgr2=0x7f83b4115ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.760+0000 7f83b1400700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83b40fdfe0 0x7f83b4115ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.760+0000 7f83b1400700 1 -- 172.21.15.1:0/2996930672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4009940 con 0x7f83b40fe8f0 2024-02-28T17:05:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.760+0000 7f83b1400700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fe8f0 0x7f83b41163e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f83a0002a50 tx=0x7f83a00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.761+0000 7f83aa000700 1 -- 172.21.15.1:0/2996930672 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a0007cc0 con 0x7f83b40fe8f0 2024-02-28T17:05:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.761+0000 7f83aa000700 1 -- 172.21.15.1:0/2996930672 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83a000f040 con 0x7f83b40fe8f0 2024-02-28T17:05:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.761+0000 7f83aa000700 1 -- 172.21.15.1:0/2996930672 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a0009640 con 0x7f83b40fe8f0 2024-02-28T17:05:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.761+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b4110020 con 0x7f83b40fe8f0 2024-02-28T17:05:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.761+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b4110560 con 0x7f83b40fe8f0 2024-02-28T17:05:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.762+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b4106ac0 con 0x7f83b40fe8f0 2024-02-28T17:05:09.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.763+0000 7f83aa000700 1 -- 172.21.15.1:0/2996930672 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f83a00097a0 con 0x7f83b40fe8f0 2024-02-28T17:05:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.764+0000 7f83aa000700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f839806d7f0 0x7f839806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.764+0000 7f83aa000700 1 -- 172.21.15.1:0/2996930672 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f83a008cbc0 con 0x7f83b40fe8f0 2024-02-28T17:05:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.765+0000 7f83b1e00700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f839806d7f0 0x7f839806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.765+0000 7f83b1e00700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f839806d7f0 0x7f839806fca0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f83a4015040 tx=0x7f83a4004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:09.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:09.769+0000 7f83aa000700 1 -- 172.21.15.1:0/2996930672 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f83a0059fc0 con 0x7f83b40fe8f0 2024-02-28T17:05:10.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.011+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f83b404bf40 con 0x7f839806d7f0 2024-02-28T17:05:10.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.013+0000 7f83aa000700 1 -- 172.21.15.1:0/2996930672 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f83b404bf40 con 0x7f839806d7f0 2024-02-28T17:05:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f839806d7f0 msgr2=0x7f839806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f839806d7f0 0x7f839806fca0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f83a4015040 tx=0x7f83a4004d10 comp rx=0 tx=0).stop 2024-02-28T17:05:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fe8f0 msgr2=0x7f83b41163e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fe8f0 0x7f83b41163e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f83a0002a50 tx=0x7f83a00046b0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 shutdown_connections 2024-02-28T17:05:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f839806d7f0 0x7f839806fca0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83b40fdfe0 0x7f83b4115ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 --2- 172.21.15.1:0/2996930672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83b40fe8f0 0x7f83b41163e0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 >> 172.21.15.1:0/2996930672 conn(0x7f83b4071d90 msgr2=0x7f83b4103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 shutdown_connections 2024-02-28T17:05:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.016+0000 7f83b9200700 1 -- 172.21.15.1:0/2996930672 wait complete. 2024-02-28T17:05:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.171+0000 7f1335800700 1 -- 172.21.15.1:0/108976128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 msgr2=0x7f1330106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.171+0000 7f1335800700 1 --2- 172.21.15.1:0/108976128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 0x7f1330106f40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f131c004940 tx=0x7f131c009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.171+0000 7f1335800700 1 -- 172.21.15.1:0/108976128 shutdown_connections 2024-02-28T17:05:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.171+0000 7f1335800700 1 --2- 172.21.15.1:0/108976128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 0x7f1330106f40 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.171+0000 7f1335800700 1 --2- 172.21.15.1:0/108976128 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13300fdf60 0x7f13300fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.171+0000 7f1335800700 1 -- 172.21.15.1:0/108976128 >> 172.21.15.1:0/108976128 conn(0x7f13300f7a10 msgr2=0x7f13300f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.171+0000 7f1335800700 1 -- 172.21.15.1:0/108976128 shutdown_connections 2024-02-28T17:05:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.171+0000 7f1335800700 1 -- 172.21.15.1:0/108976128 wait complete. 2024-02-28T17:05:10.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.172+0000 7f1335800700 1 Processor -- start 2024-02-28T17:05:10.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.172+0000 7f1335800700 1 -- start start 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.172+0000 7f1335800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13300fdf60 0x7f13300fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.172+0000 7f1335800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 0x7f13300fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.172+0000 7f1335800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13300fc960 con 0x7f13300fe900 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.172+0000 7f1335800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13300fcad0 con 0x7f13300fdf60 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.173+0000 7f132e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 0x7f13300fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.173+0000 7f132e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 0x7f13300fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56172/0 (socket says 172.21.15.1:56172) 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.173+0000 7f132e000700 1 -- 172.21.15.1:0/1568806979 learned_addr learned my addr 172.21.15.1:0/1568806979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.173+0000 7f132e000700 1 -- 172.21.15.1:0/1568806979 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13300fdf60 msgr2=0x7f13300fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.173+0000 7f132e000700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13300fdf60 0x7f13300fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.173+0000 7f132e000700 1 -- 172.21.15.1:0/1568806979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c005040 con 0x7f13300fe900 2024-02-28T17:05:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.173+0000 7f132e000700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 0x7f13300fc2d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f131c009180 tx=0x7f131c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.174+0000 7f1326a00700 1 -- 172.21.15.1:0/1568806979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f131c00e040 con 0x7f13300fe900 2024-02-28T17:05:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.174+0000 7f1326a00700 1 -- 172.21.15.1:0/1568806979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f131c00bbf0 con 0x7f13300fe900 2024-02-28T17:05:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.174+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1330119d50 con 0x7f13300fe900 2024-02-28T17:05:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.174+0000 7f1326a00700 1 -- 172.21.15.1:0/1568806979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f131c01c630 con 0x7f13300fe900 2024-02-28T17:05:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.174+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133011a240 con 0x7f13300fe900 2024-02-28T17:05:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.175+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1330104550 con 0x7f13300fe900 2024-02-28T17:05:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.176+0000 7f1326a00700 1 -- 172.21.15.1:0/1568806979 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f131c00bd60 con 0x7f13300fe900 2024-02-28T17:05:10.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.178+0000 7f1326a00700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131406d7f0 0x7f131406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:10.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.178+0000 7f1326a00700 1 -- 172.21.15.1:0/1568806979 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f131c08e330 con 0x7f13300fe900 2024-02-28T17:05:10.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.178+0000 7f132ea00700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131406d7f0 0x7f131406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.179+0000 7f132ea00700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131406d7f0 0x7f131406fca0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1320006fd0 tx=0x7f1320006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:10.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.182+0000 7f1326a00700 1 -- 172.21.15.1:0/1568806979 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f131c057b60 con 0x7f13300fe900 2024-02-28T17:05:10.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:10 smithi001 ceph-mon[112940]: pgmap v922: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:10.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:10 smithi001 ceph-mon[112940]: from='client.35654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:10.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.418+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f13300fd4d0 con 0x7f131406d7f0 2024-02-28T17:05:10.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.430+0000 7f1326a00700 1 -- 172.21.15.1:0/1568806979 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f13300fd4d0 con 0x7f131406d7f0 2024-02-28T17:05:10.431 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:05:10.431 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (31m) 8m ago 45m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:05:10.431 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (28m) 8m ago 45m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:05:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (27m) 8m ago 44m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:05:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (30m) 8m ago 45m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:05:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (23m) 8m ago 38m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:05:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (23m) 8m ago 38m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:05:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (38m) 8m ago 38m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:05:10.432 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (8m) 8m ago 38m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:05:10.433 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (33m) 8m ago 47m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:05:10.433 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (32m) 8m ago 44m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:05:10.433 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (29m) 8m ago 47m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:05:10.433 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (29m) 8m ago 44m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:05:10.433 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (32m) 8m ago 45m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:05:10.434 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (32m) 8m ago 44m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:05:10.434 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (27m) 8m ago 43m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:05:10.434 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (26m) 8m ago 42m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:05:10.434 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (26m) 8m ago 42m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:05:10.434 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (25m) 8m ago 41m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:05:10.434 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (25m) 8m ago 40m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:05:10.435 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (24m) 8m ago 40m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:05:10.435 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (31m) 8m ago 44m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:05:10.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131406d7f0 msgr2=0x7f131406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131406d7f0 0x7f131406fca0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1320006fd0 tx=0x7f1320006db0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 msgr2=0x7f13300fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 0x7f13300fc2d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f131c009180 tx=0x7f131c009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:10.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 shutdown_connections 2024-02-28T17:05:10.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131406d7f0 0x7f131406fca0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f13300fdf60 0x7f13300fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 --2- 172.21.15.1:0/1568806979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13300fe900 0x7f13300fc2d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.432+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 >> 172.21.15.1:0/1568806979 conn(0x7f13300f7a10 msgr2=0x7f13301016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:10.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.433+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 shutdown_connections 2024-02-28T17:05:10.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.433+0000 7f1335800700 1 -- 172.21.15.1:0/1568806979 wait complete. 2024-02-28T17:05:10.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:10 smithi066 ceph-mon[85781]: pgmap v922: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:10.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:10 smithi066 ceph-mon[85781]: from='client.35654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:10.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.584+0000 7ffa98c00700 1 -- 172.21.15.1:0/146851014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 msgr2=0x7ffa94070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.584+0000 7ffa98c00700 1 --2- 172.21.15.1:0/146851014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 0x7ffa94070330 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ffa84004970 tx=0x7ffa84009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:10.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.584+0000 7ffa98c00700 1 -- 172.21.15.1:0/146851014 shutdown_connections 2024-02-28T17:05:10.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.584+0000 7ffa98c00700 1 --2- 172.21.15.1:0/146851014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa94070870 0x7ffa94070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.584+0000 7ffa98c00700 1 --2- 172.21.15.1:0/146851014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 0x7ffa94070330 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.584+0000 7ffa98c00700 1 -- 172.21.15.1:0/146851014 >> 172.21.15.1:0/146851014 conn(0x7ffa940f8ac0 msgr2=0x7ffa940faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:10.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.585+0000 7ffa98c00700 1 -- 172.21.15.1:0/146851014 shutdown_connections 2024-02-28T17:05:10.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.585+0000 7ffa98c00700 1 -- 172.21.15.1:0/146851014 wait complete. 2024-02-28T17:05:10.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.585+0000 7ffa98c00700 1 Processor -- start 2024-02-28T17:05:10.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.586+0000 7ffa98c00700 1 -- start start 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.586+0000 7ffa98c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa94070870 0x7ffa9411a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.586+0000 7ffa98c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 0x7ffa9411a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.586+0000 7ffa98c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9411ac30 con 0x7ffa94071f60 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.586+0000 7ffa98c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa94113f80 con 0x7ffa94070870 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.586+0000 7ffa91400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 0x7ffa9411a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.586+0000 7ffa91400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 0x7ffa9411a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56188/0 (socket says 172.21.15.1:56188) 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.586+0000 7ffa91400700 1 -- 172.21.15.1:0/2656279844 learned_addr learned my addr 172.21.15.1:0/2656279844 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa91e00700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa94070870 0x7ffa9411a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa91400700 1 -- 172.21.15.1:0/2656279844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa94070870 msgr2=0x7ffa9411a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa91400700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa94070870 0x7ffa9411a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa91400700 1 -- 172.21.15.1:0/2656279844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa84005040 con 0x7ffa94071f60 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa91400700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 0x7ffa9411a5e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ffa80002a50 tx=0x7ffa800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa8a000700 1 -- 172.21.15.1:0/2656279844 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa80007cc0 con 0x7ffa94071f60 2024-02-28T17:05:10.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa8a000700 1 -- 172.21.15.1:0/2656279844 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa8000f040 con 0x7ffa94071f60 2024-02-28T17:05:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa94114260 con 0x7ffa94071f60 2024-02-28T17:05:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa8a000700 1 -- 172.21.15.1:0/2656279844 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa80009640 con 0x7ffa94071f60 2024-02-28T17:05:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.587+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa941147a0 con 0x7ffa94071f60 2024-02-28T17:05:10.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.589+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa940fe400 con 0x7ffa94071f60 2024-02-28T17:05:10.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.590+0000 7ffa8a000700 1 -- 172.21.15.1:0/2656279844 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffa800097a0 con 0x7ffa94071f60 2024-02-28T17:05:10.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.591+0000 7ffa8a000700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa7806d7f0 0x7ffa7806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:10.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.591+0000 7ffa8a000700 1 -- 172.21.15.1:0/2656279844 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffa8008cbc0 con 0x7ffa94071f60 2024-02-28T17:05:10.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.591+0000 7ffa91e00700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa7806d7f0 0x7ffa7806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:10.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.592+0000 7ffa91e00700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa7806d7f0 0x7ffa7806fca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffa84000c00 tx=0x7ffa84009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.596+0000 7ffa8a000700 1 -- 172.21.15.1:0/2656279844 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffa80056490 con 0x7ffa94071f60 2024-02-28T17:05:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.921+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffa9404bed0 con 0x7ffa94071f60 2024-02-28T17:05:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.922+0000 7ffa8a000700 1 -- 172.21.15.1:0/2656279844 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7ffa80059b50 con 0x7ffa94071f60 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:05:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:05:10.924 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:05:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa7806d7f0 msgr2=0x7ffa7806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa7806d7f0 0x7ffa7806fca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffa84000c00 tx=0x7ffa84009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 msgr2=0x7ffa9411a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 0x7ffa9411a5e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ffa80002a50 tx=0x7ffa800046b0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 shutdown_connections 2024-02-28T17:05:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa7806d7f0 0x7ffa7806fca0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa94070870 0x7ffa9411a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 --2- 172.21.15.1:0/2656279844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa94071f60 0x7ffa9411a5e0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:10.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 >> 172.21.15.1:0/2656279844 conn(0x7ffa940f8ac0 msgr2=0x7ffa940fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:10.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.925+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 shutdown_connections 2024-02-28T17:05:10.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:10.926+0000 7ffa98c00700 1 -- 172.21.15.1:0/2656279844 wait complete. 2024-02-28T17:05:11.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.077+0000 7ff189400700 1 -- 172.21.15.1:0/56404683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 msgr2=0x7ff184105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.077+0000 7ff189400700 1 --2- 172.21.15.1:0/56404683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 0x7ff184105340 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff174004940 tx=0x7ff174009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:11.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.077+0000 7ff189400700 1 -- 172.21.15.1:0/56404683 shutdown_connections 2024-02-28T17:05:11.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.077+0000 7ff189400700 1 --2- 172.21.15.1:0/56404683 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff184105910 0x7ff1840fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.077+0000 7ff189400700 1 --2- 172.21.15.1:0/56404683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 0x7ff184105340 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.077+0000 7ff189400700 1 -- 172.21.15.1:0/56404683 >> 172.21.15.1:0/56404683 conn(0x7ff1840faae0 msgr2=0x7ff1840fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:11.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.077+0000 7ff189400700 1 -- 172.21.15.1:0/56404683 shutdown_connections 2024-02-28T17:05:11.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.077+0000 7ff189400700 1 -- 172.21.15.1:0/56404683 wait complete. 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.078+0000 7ff189400700 1 Processor -- start 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.078+0000 7ff189400700 1 -- start start 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff189400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 0x7ff184115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff189400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff184105910 0x7ff184115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff189400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841165b0 con 0x7ff184104f70 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff189400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18410f8e0 con 0x7ff184105910 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff183e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 0x7ff184115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff183e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 0x7ff184115a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56218/0 (socket says 172.21.15.1:56218) 2024-02-28T17:05:11.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff183e00700 1 -- 172.21.15.1:0/247970993 learned_addr learned my addr 172.21.15.1:0/247970993 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:11.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff183e00700 1 -- 172.21.15.1:0/247970993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff184105910 msgr2=0x7ff184115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff183e00700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff184105910 0x7ff184115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.079+0000 7ff183e00700 1 -- 172.21.15.1:0/247970993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff174005040 con 0x7ff184104f70 2024-02-28T17:05:11.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.080+0000 7ff183e00700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 0x7ff184115a20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff174009fd0 tx=0x7ff174009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:11.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.080+0000 7ff17be00700 1 -- 172.21.15.1:0/247970993 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff17400e040 con 0x7ff184104f70 2024-02-28T17:05:11.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.080+0000 7ff17be00700 1 -- 172.21.15.1:0/247970993 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff17400bbf0 con 0x7ff184104f70 2024-02-28T17:05:11.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.080+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18410fb60 con 0x7ff184104f70 2024-02-28T17:05:11.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.080+0000 7ff17be00700 1 -- 172.21.15.1:0/247970993 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff17401c3f0 con 0x7ff184104f70 2024-02-28T17:05:11.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.080+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff184110120 con 0x7ff184104f70 2024-02-28T17:05:11.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.081+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18405f5c0 con 0x7ff184104f70 2024-02-28T17:05:11.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.082+0000 7ff17be00700 1 -- 172.21.15.1:0/247970993 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff17400bd60 con 0x7ff184104f70 2024-02-28T17:05:11.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.083+0000 7ff17be00700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff16806d7f0 0x7ff16806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.083+0000 7ff17be00700 1 -- 172.21.15.1:0/247970993 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff17408e1c0 con 0x7ff184104f70 2024-02-28T17:05:11.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.083+0000 7ff183400700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff16806d7f0 0x7ff16806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:11.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.084+0000 7ff183400700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff16806d7f0 0x7ff16806fca0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff184111250 tx=0x7ff170009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:11.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.088+0000 7ff17be00700 1 -- 172.21.15.1:0/247970993 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff17405b5a0 con 0x7ff184104f70 2024-02-28T17:05:11.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.355+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff184110390 con 0x7ff184104f70 2024-02-28T17:05:11.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.356+0000 7ff17be00700 1 -- 172.21.15.1:0/247970993 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7ff17405b130 con 0x7ff184104f70 2024-02-28T17:05:11.358 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:05:11.358 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:05:11.358 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:05:11.358 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:05:11.358 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:05:11.358 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:05:11.358 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:05:11.359 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:05:11.359 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:05:11.359 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:05:11.359 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:05:11.359 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:05:11.359 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:05:11.359 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:05:11.359 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:05:11.360 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:05:11.361 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:05:11.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.360+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff16806d7f0 msgr2=0x7ff16806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.360+0000 7ff189400700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff16806d7f0 0x7ff16806fca0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff184111250 tx=0x7ff170009ce0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.360+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 msgr2=0x7ff184115a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.360+0000 7ff189400700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 0x7ff184115a20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff174009fd0 tx=0x7ff174009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:11.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.361+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 shutdown_connections 2024-02-28T17:05:11.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.361+0000 7ff189400700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff16806d7f0 0x7ff16806fca0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.361+0000 7ff189400700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff184104f70 0x7ff184115a20 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.361+0000 7ff189400700 1 --2- 172.21.15.1:0/247970993 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff184105910 0x7ff184115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.361+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 >> 172.21.15.1:0/247970993 conn(0x7ff1840faae0 msgr2=0x7ff1840fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:11.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.361+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 shutdown_connections 2024-02-28T17:05:11.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.361+0000 7ff189400700 1 -- 172.21.15.1:0/247970993 wait complete. 2024-02-28T17:05:11.363 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:05:11.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.517+0000 7f1338600700 1 -- 172.21.15.1:0/4120720040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105b10 msgr2=0x7f13300ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.517+0000 7f1338600700 1 --2- 172.21.15.1:0/4120720040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105b10 0x7f13300ff180 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f131c004940 tx=0x7f131c009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:11.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.517+0000 7f1338600700 1 -- 172.21.15.1:0/4120720040 shutdown_connections 2024-02-28T17:05:11.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.517+0000 7f1338600700 1 --2- 172.21.15.1:0/4120720040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105b10 0x7f13300ff180 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.517+0000 7f1338600700 1 --2- 172.21.15.1:0/4120720040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1330105170 0x7f1330105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.517+0000 7f1338600700 1 -- 172.21.15.1:0/4120720040 >> 172.21.15.1:0/4120720040 conn(0x7f13300face0 msgr2=0x7f13300fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:11.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.517+0000 7f1338600700 1 -- 172.21.15.1:0/4120720040 shutdown_connections 2024-02-28T17:05:11.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.517+0000 7f1338600700 1 -- 172.21.15.1:0/4120720040 wait complete. 2024-02-28T17:05:11.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.518+0000 7f1338600700 1 Processor -- start 2024-02-28T17:05:11.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.519+0000 7f1338600700 1 -- start start 2024-02-28T17:05:11.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:11 smithi001 ceph-mon[112940]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:11.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:11 smithi001 ceph-mon[112940]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:11.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:05:11.520 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:11 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2656279844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:05:11.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.519+0000 7f1338600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105170 0x7f1330115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.519+0000 7f1338600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1330105b10 0x7f13301161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.519+0000 7f1338600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1330116800 con 0x7f1330105170 2024-02-28T17:05:11.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.519+0000 7f1338600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133010fb50 con 0x7f1330105b10 2024-02-28T17:05:11.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.519+0000 7f1335c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105170 0x7f1330115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.520+0000 7f1335c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105170 0x7f1330115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56234/0 (socket says 172.21.15.1:56234) 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.520+0000 7f1335c00700 1 -- 172.21.15.1:0/4012046819 learned_addr learned my addr 172.21.15.1:0/4012046819 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.520+0000 7f1335200700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1330105b10 0x7f13301161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.520+0000 7f1335c00700 1 -- 172.21.15.1:0/4012046819 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1330105b10 msgr2=0x7f13301161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.520+0000 7f1335c00700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1330105b10 0x7f13301161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.520+0000 7f1335c00700 1 -- 172.21.15.1:0/4012046819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c005040 con 0x7f1330105170 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.520+0000 7f1335c00700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105170 0x7f1330115c70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1324002a50 tx=0x7f13240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.521+0000 7f132d600700 1 -- 172.21.15.1:0/4012046819 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1324007cc0 con 0x7f1330105170 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.521+0000 7f132d600700 1 -- 172.21.15.1:0/4012046819 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f132400f040 con 0x7f1330105170 2024-02-28T17:05:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.521+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133010fe30 con 0x7f1330105170 2024-02-28T17:05:11.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.521+0000 7f132d600700 1 -- 172.21.15.1:0/4012046819 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1324009640 con 0x7f1330105170 2024-02-28T17:05:11.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.521+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1330110370 con 0x7f1330105170 2024-02-28T17:05:11.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.522+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1330107640 con 0x7f1330105170 2024-02-28T17:05:11.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.523+0000 7f132d600700 1 -- 172.21.15.1:0/4012046819 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f13240097a0 con 0x7f1330105170 2024-02-28T17:05:11.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.524+0000 7f132d600700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131806d7f0 0x7f131806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.524+0000 7f132d600700 1 -- 172.21.15.1:0/4012046819 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f132408cad0 con 0x7f1330105170 2024-02-28T17:05:11.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.525+0000 7f1335200700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131806d7f0 0x7f131806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.525+0000 7f1335200700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131806d7f0 0x7f131806fca0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f131c00b020 tx=0x7f131c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:11.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.530+0000 7f132d600700 1 -- 172.21.15.1:0/4012046819 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1324056280 con 0x7f1330105170 2024-02-28T17:05:11.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:11 smithi066 ceph-mon[85781]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:11.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:11 smithi066 ceph-mon[85781]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:11.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:05:11.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:11 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2656279844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:05:11.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.777+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f133004bf40 con 0x7f131806d7f0 2024-02-28T17:05:11.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.780+0000 7f132d600700 1 -- 172.21.15.1:0/4012046819 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f133004bf40 con 0x7f131806d7f0 2024-02-28T17:05:11.780 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:05:11.780 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:05:11.780 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:05:11.780 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:05:11.780 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:05:11.781 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131806d7f0 msgr2=0x7f131806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131806d7f0 0x7f131806fca0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f131c00b020 tx=0x7f131c009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105170 msgr2=0x7f1330115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105170 0x7f1330115c70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1324002a50 tx=0x7f13240046b0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 shutdown_connections 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f131806d7f0 0x7f131806fca0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330105170 0x7f1330115c70 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 --2- 172.21.15.1:0/4012046819 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1330105b10 0x7f13301161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 >> 172.21.15.1:0/4012046819 conn(0x7f13300face0 msgr2=0x7f13300fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.782+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 shutdown_connections 2024-02-28T17:05:11.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.783+0000 7f1338600700 1 -- 172.21.15.1:0/4012046819 wait complete. 2024-02-28T17:05:11.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.932+0000 7f0bdb200700 1 -- 172.21.15.1:0/1962019054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 msgr2=0x7f0bdc105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.932+0000 7f0bdb200700 1 --2- 172.21.15.1:0/1962019054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 0x7f0bdc105340 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc004940 tx=0x7f0bcc009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:11.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.932+0000 7f0bdb200700 1 -- 172.21.15.1:0/1962019054 shutdown_connections 2024-02-28T17:05:11.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.932+0000 7f0bdb200700 1 --2- 172.21.15.1:0/1962019054 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0bdc105910 0x7f0bdc0fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.932+0000 7f0bdb200700 1 --2- 172.21.15.1:0/1962019054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 0x7f0bdc105340 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.932+0000 7f0bdb200700 1 -- 172.21.15.1:0/1962019054 >> 172.21.15.1:0/1962019054 conn(0x7f0bdc0faae0 msgr2=0x7f0bdc0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:11.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.933+0000 7f0bdb200700 1 -- 172.21.15.1:0/1962019054 shutdown_connections 2024-02-28T17:05:11.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.933+0000 7f0bdb200700 1 -- 172.21.15.1:0/1962019054 wait complete. 2024-02-28T17:05:11.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.933+0000 7f0bdb200700 1 Processor -- start 2024-02-28T17:05:11.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.934+0000 7f0bdb200700 1 -- start start 2024-02-28T17:05:11.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.934+0000 7f0bdb200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 0x7f0bdc115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.934+0000 7f0bdb200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0bdc105910 0x7f0bdc115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.934+0000 7f0bdb200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1165b0 con 0x7f0bdc104f70 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.934+0000 7f0bdb200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc10f8e0 con 0x7f0bdc105910 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.934+0000 7f0bd9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 0x7f0bdc115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 0x7f0bdc115a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55240/0 (socket says 172.21.15.1:55240) 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd9e00700 1 -- 172.21.15.1:0/249933744 learned_addr learned my addr 172.21.15.1:0/249933744 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd9e00700 1 -- 172.21.15.1:0/249933744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0bdc105910 msgr2=0x7f0bdc115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd9e00700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0bdc105910 0x7f0bdc115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd9e00700 1 -- 172.21.15.1:0/249933744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc005040 con 0x7f0bdc104f70 2024-02-28T17:05:11.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd9e00700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 0x7f0bdc115a20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009150 tx=0x7f0bcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:11.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd2000700 1 -- 172.21.15.1:0/249933744 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bcc00e040 con 0x7f0bdc104f70 2024-02-28T17:05:11.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd2000700 1 -- 172.21.15.1:0/249933744 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bcc004290 con 0x7f0bdc104f70 2024-02-28T17:05:11.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.935+0000 7f0bd2000700 1 -- 172.21.15.1:0/249933744 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bcc01c630 con 0x7f0bdc104f70 2024-02-28T17:05:11.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.936+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc10fb60 con 0x7f0bdc104f70 2024-02-28T17:05:11.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.936+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc110120 con 0x7f0bdc104f70 2024-02-28T17:05:11.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.937+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc05f5c0 con 0x7f0bdc104f70 2024-02-28T17:05:11.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.938+0000 7f0bd2000700 1 -- 172.21.15.1:0/249933744 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0bcc00bbf0 con 0x7f0bdc104f70 2024-02-28T17:05:11.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.939+0000 7f0bd2000700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc006d7f0 0x7f0bc006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:11.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.939+0000 7f0bd2000700 1 -- 172.21.15.1:0/249933744 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0bcc08e260 con 0x7f0bdc104f70 2024-02-28T17:05:11.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.939+0000 7f0bd3e00700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc006d7f0 0x7f0bc006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:11.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.940+0000 7f0bd3e00700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc006d7f0 0x7f0bc006fca0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc111250 tx=0x7f0bc4009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:11.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:11.943+0000 7f0bd2000700 1 -- 172.21.15.1:0/249933744 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0bcc05b640 con 0x7f0bdc104f70 2024-02-28T17:05:12.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.253+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0bdc04bed0 con 0x7f0bdc104f70 2024-02-28T17:05:12.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.254+0000 7f0bd2000700 1 -- 172.21.15.1:0/249933744 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f0bcc05b1d0 con 0x7f0bdc104f70 2024-02-28T17:05:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:05:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:05:12.255 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:05:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:05:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:05:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:05:12.255 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:05:12.256 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:05:12.257 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:05:12.257 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:05:12.257 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:05:12.257 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:05:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc006d7f0 msgr2=0x7f0bc006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc006d7f0 0x7f0bc006fca0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc111250 tx=0x7f0bc4009ce0 comp rx=0 tx=0).stop 2024-02-28T17:05:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 msgr2=0x7f0bdc115a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 0x7f0bdc115a20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009150 tx=0x7f0bcc009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:12.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 shutdown_connections 2024-02-28T17:05:12.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0bc006d7f0 0x7f0bc006fca0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:12.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc104f70 0x7f0bdc115a20 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:12.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 --2- 172.21.15.1:0/249933744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0bdc105910 0x7f0bdc115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:12.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 >> 172.21.15.1:0/249933744 conn(0x7f0bdc0faae0 msgr2=0x7f0bdc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:12.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 shutdown_connections 2024-02-28T17:05:12.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:12.257+0000 7f0bdb200700 1 -- 172.21.15.1:0/249933744 wait complete. 2024-02-28T17:05:12.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:12 smithi001 ceph-mon[112940]: pgmap v923: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:12.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:12 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/247970993' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:05:12.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:12 smithi001 ceph-mon[112940]: from='client.35674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:12.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:12 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/249933744' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:05:12.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:12 smithi066 ceph-mon[85781]: pgmap v923: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:12.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:12 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/247970993' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:05:12.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:12 smithi066 ceph-mon[85781]: from='client.35674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:12.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:12 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/249933744' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:05:14.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:14 smithi001 ceph-mon[112940]: pgmap v924: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:14.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:14 smithi066 ceph-mon[85781]: pgmap v924: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:16.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:16 smithi066 ceph-mon[85781]: pgmap v925: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:16.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:16 smithi001 ceph-mon[112940]: pgmap v925: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:18.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:18 smithi066 ceph-mon[85781]: pgmap v926: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:18.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:18 smithi001 ceph-mon[112940]: pgmap v926: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:20.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:20 smithi066 ceph-mon[85781]: pgmap v927: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:20.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:20 smithi001 ceph-mon[112940]: pgmap v927: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:22.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:22 smithi066 ceph-mon[85781]: pgmap v928: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:22.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:22 smithi001 ceph-mon[112940]: pgmap v928: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:24.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:24 smithi066 ceph-mon[85781]: pgmap v929: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:24.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:24 smithi001 ceph-mon[112940]: pgmap v929: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:26.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:26 smithi066 ceph-mon[85781]: pgmap v930: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:26.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:05:26.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:26 smithi001 ceph-mon[112940]: pgmap v930: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:26.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:05:28.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:28 smithi066 ceph-mon[85781]: pgmap v931: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:28.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:28 smithi001 ceph-mon[112940]: pgmap v931: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:30.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:30 smithi066 ceph-mon[85781]: pgmap v932: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:30.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:30 smithi001 ceph-mon[112940]: pgmap v932: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:32.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:32 smithi066 ceph-mon[85781]: pgmap v933: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:32.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:32 smithi001 ceph-mon[112940]: pgmap v933: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:34.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:34 smithi066 ceph-mon[85781]: pgmap v934: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:34.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:34 smithi001 ceph-mon[112940]: pgmap v934: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:36.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:36 smithi066 ceph-mon[85781]: pgmap v935: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:36.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:36 smithi001 ceph-mon[112940]: pgmap v935: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:38.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:38 smithi066 ceph-mon[85781]: pgmap v936: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:38.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:38 smithi001 ceph-mon[112940]: pgmap v936: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:40.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:40 smithi066 ceph-mon[85781]: pgmap v937: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:40.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:40 smithi001 ceph-mon[112940]: pgmap v937: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:41.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:05:41.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:05:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.427+0000 7fe534800700 1 -- 172.21.15.1:0/2156733324 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c103dc0 msgr2=0x7fe52c104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.427+0000 7fe534800700 1 --2- 172.21.15.1:0/2156733324 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c103dc0 0x7fe52c104230 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe51c004940 tx=0x7fe51c009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.428+0000 7fe534800700 1 -- 172.21.15.1:0/2156733324 shutdown_connections 2024-02-28T17:05:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.428+0000 7fe534800700 1 --2- 172.21.15.1:0/2156733324 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c103dc0 0x7fe52c104230 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.428+0000 7fe534800700 1 --2- 172.21.15.1:0/2156733324 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe52c1095b0 0x7fe52c109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.428+0000 7fe534800700 1 -- 172.21.15.1:0/2156733324 >> 172.21.15.1:0/2156733324 conn(0x7fe52c072d70 msgr2=0x7fe52c075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.428+0000 7fe534800700 1 -- 172.21.15.1:0/2156733324 shutdown_connections 2024-02-28T17:05:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.428+0000 7fe534800700 1 -- 172.21.15.1:0/2156733324 wait complete. 2024-02-28T17:05:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.429+0000 7fe534800700 1 Processor -- start 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.429+0000 7fe534800700 1 -- start start 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.429+0000 7fe534800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe52c103dc0 0x7fe52c11a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.429+0000 7fe534800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c1095b0 0x7fe52c11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.429+0000 7fe534800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c11ac00 con 0x7fe52c1095b0 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.429+0000 7fe534800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c113f10 con 0x7fe52c103dc0 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.430+0000 7fe531400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c1095b0 0x7fe52c11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.430+0000 7fe531400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c1095b0 0x7fe52c11a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42306/0 (socket says 172.21.15.1:42306) 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.430+0000 7fe531400700 1 -- 172.21.15.1:0/1366128285 learned_addr learned my addr 172.21.15.1:0/1366128285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.430+0000 7fe531400700 1 -- 172.21.15.1:0/1366128285 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe52c103dc0 msgr2=0x7fe52c11a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.430+0000 7fe531400700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe52c103dc0 0x7fe52c11a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.430+0000 7fe531400700 1 -- 172.21.15.1:0/1366128285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c005040 con 0x7fe52c1095b0 2024-02-28T17:05:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.430+0000 7fe531400700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c1095b0 0x7fe52c11a5b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009fd0 tx=0x7fe51c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:42.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.431+0000 7fe526000700 1 -- 172.21.15.1:0/1366128285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe51c00e040 con 0x7fe52c1095b0 2024-02-28T17:05:42.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.431+0000 7fe526000700 1 -- 172.21.15.1:0/1366128285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe51c00bbf0 con 0x7fe52c1095b0 2024-02-28T17:05:42.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.431+0000 7fe526000700 1 -- 172.21.15.1:0/1366128285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe51c01c3f0 con 0x7fe52c1095b0 2024-02-28T17:05:42.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.431+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c114190 con 0x7fe52c1095b0 2024-02-28T17:05:42.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.431+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c1146d0 con 0x7fe52c1095b0 2024-02-28T17:05:42.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.432+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c10b990 con 0x7fe52c1095b0 2024-02-28T17:05:42.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.433+0000 7fe526000700 1 -- 172.21.15.1:0/1366128285 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe51c00bd60 con 0x7fe52c1095b0 2024-02-28T17:05:42.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.434+0000 7fe526000700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe51406d7f0 0x7fe51406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:42.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.434+0000 7fe526000700 1 -- 172.21.15.1:0/1366128285 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe51c08e1c0 con 0x7fe52c1095b0 2024-02-28T17:05:42.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.434+0000 7fe531e00700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe51406d7f0 0x7fe51406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:42.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.435+0000 7fe531e00700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe51406d7f0 0x7fe51406fca0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe52c076220 tx=0x7fe520007d60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:42.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.439+0000 7fe526000700 1 -- 172.21.15.1:0/1366128285 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe51c05b5a0 con 0x7fe52c1095b0 2024-02-28T17:05:42.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.682+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe52c04bf40 con 0x7fe51406d7f0 2024-02-28T17:05:42.685 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:42 smithi001 ceph-mon[112940]: pgmap v938: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:42.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.687+0000 7fe526000700 1 -- 172.21.15.1:0/1366128285 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe52c04bf40 con 0x7fe51406d7f0 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe51406d7f0 msgr2=0x7fe51406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe51406d7f0 0x7fe51406fca0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe52c076220 tx=0x7fe520007d60 comp rx=0 tx=0).stop 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c1095b0 msgr2=0x7fe52c11a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c1095b0 0x7fe52c11a5b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009fd0 tx=0x7fe51c009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 shutdown_connections 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe51406d7f0 0x7fe51406fca0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe52c103dc0 0x7fe52c11a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 --2- 172.21.15.1:0/1366128285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe52c1095b0 0x7fe52c11a5b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 >> 172.21.15.1:0/1366128285 conn(0x7fe52c072d70 msgr2=0x7fe52c074960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 shutdown_connections 2024-02-28T17:05:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.690+0000 7fe534800700 1 -- 172.21.15.1:0/1366128285 wait complete. 2024-02-28T17:05:42.708 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:05:42.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:42 smithi066 ceph-mon[85781]: pgmap v938: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.855+0000 7f5500400700 1 -- 172.21.15.1:0/1797476962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f8103dc0 msgr2=0x7f54f8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.855+0000 7f5500400700 1 --2- 172.21.15.1:0/1797476962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f8103dc0 0x7f54f8104210 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f54e4004940 tx=0x7f54e4009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.855+0000 7f5500400700 1 -- 172.21.15.1:0/1797476962 shutdown_connections 2024-02-28T17:05:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.855+0000 7f5500400700 1 --2- 172.21.15.1:0/1797476962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f8103dc0 0x7f54f8104210 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.855+0000 7f5500400700 1 --2- 172.21.15.1:0/1797476962 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54f81095f0 0x7f54f81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.855+0000 7f5500400700 1 -- 172.21.15.1:0/1797476962 >> 172.21.15.1:0/1797476962 conn(0x7f54f8072d50 msgr2=0x7f54f8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.855+0000 7f5500400700 1 -- 172.21.15.1:0/1797476962 shutdown_connections 2024-02-28T17:05:42.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.855+0000 7f5500400700 1 -- 172.21.15.1:0/1797476962 wait complete. 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.856+0000 7f5500400700 1 Processor -- start 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.856+0000 7f5500400700 1 -- start start 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.857+0000 7f5500400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54f8103dc0 0x7f54f811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.857+0000 7f5500400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f81095f0 0x7f54f811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.857+0000 7f5500400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f811abd0 con 0x7f54f81095f0 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.857+0000 7f5500400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f8113ee0 con 0x7f54f8103dc0 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.857+0000 7f54fd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f81095f0 0x7f54f811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.857+0000 7f54fd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f81095f0 0x7f54f811a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42324/0 (socket says 172.21.15.1:42324) 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.857+0000 7f54fd000700 1 -- 172.21.15.1:0/3074520107 learned_addr learned my addr 172.21.15.1:0/3074520107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:42.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.857+0000 7f54fda00700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54f8103dc0 0x7f54f811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f54fd000700 1 -- 172.21.15.1:0/3074520107 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54f8103dc0 msgr2=0x7f54f811a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f54fd000700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54f8103dc0 0x7f54f811a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f54fd000700 1 -- 172.21.15.1:0/3074520107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e4005040 con 0x7f54f81095f0 2024-02-28T17:05:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f54fda00700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54f8103dc0 0x7f54f811a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T17:05:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f54fd000700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f81095f0 0x7f54f811a580 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f54e4009150 tx=0x7f54e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f54f5600700 1 -- 172.21.15.1:0/3074520107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e400e040 con 0x7f54f81095f0 2024-02-28T17:05:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f54f5600700 1 -- 172.21.15.1:0/3074520107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54e400bbf0 con 0x7f54f81095f0 2024-02-28T17:05:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f54f5600700 1 -- 172.21.15.1:0/3074520107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e401c630 con 0x7f54f81095f0 2024-02-28T17:05:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.858+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f8114160 con 0x7f54f81095f0 2024-02-28T17:05:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.859+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f81146a0 con 0x7f54f81095f0 2024-02-28T17:05:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.860+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f810b9c0 con 0x7f54f81095f0 2024-02-28T17:05:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.861+0000 7f54f5600700 1 -- 172.21.15.1:0/3074520107 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f54e400bd60 con 0x7f54f81095f0 2024-02-28T17:05:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.862+0000 7f54f5600700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f54e006d7f0 0x7f54e006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.862+0000 7f54f5600700 1 -- 172.21.15.1:0/3074520107 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f54e408e390 con 0x7f54f81095f0 2024-02-28T17:05:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.862+0000 7f54fda00700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f54e006d7f0 0x7f54e006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:42.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.863+0000 7f54fda00700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f54e006d7f0 0x7f54e006fca0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009fd0 tx=0x7f54ec009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:42.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:42.867+0000 7f54f5600700 1 -- 172.21.15.1:0/3074520107 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f54e405b6f0 con 0x7f54f81095f0 2024-02-28T17:05:43.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.114+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f54f804bf40 con 0x7f54e006d7f0 2024-02-28T17:05:43.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.116+0000 7f54f5600700 1 -- 172.21.15.1:0/3074520107 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f54f804bf40 con 0x7f54e006d7f0 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f54e006d7f0 msgr2=0x7f54e006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f54e006d7f0 0x7f54e006fca0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009fd0 tx=0x7f54ec009ee0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f81095f0 msgr2=0x7f54f811a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f81095f0 0x7f54f811a580 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f54e4009150 tx=0x7f54e4009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 shutdown_connections 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f54e006d7f0 0x7f54e006fca0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f54f8103dc0 0x7f54f811a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 --2- 172.21.15.1:0/3074520107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54f81095f0 0x7f54f811a580 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 >> 172.21.15.1:0/3074520107 conn(0x7f54f8072d50 msgr2=0x7f54f8074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 shutdown_connections 2024-02-28T17:05:43.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.119+0000 7f5500400700 1 -- 172.21.15.1:0/3074520107 wait complete. 2024-02-28T17:05:43.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.275+0000 7f552e400700 1 -- 172.21.15.1:0/1654599262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528070870 msgr2=0x7f5528070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:43.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.275+0000 7f552e400700 1 --2- 172.21.15.1:0/1654599262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528070870 0x7f5528070ce0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5514004970 tx=0x7f5514009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:43.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.275+0000 7f552e400700 1 -- 172.21.15.1:0/1654599262 shutdown_connections 2024-02-28T17:05:43.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.275+0000 7f552e400700 1 --2- 172.21.15.1:0/1654599262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528070870 0x7f5528070ce0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.275+0000 7f552e400700 1 --2- 172.21.15.1:0/1654599262 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5528071f60 0x7f5528070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.275+0000 7f552e400700 1 -- 172.21.15.1:0/1654599262 >> 172.21.15.1:0/1654599262 conn(0x7f55280f8ae0 msgr2=0x7f55280faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:43.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.275+0000 7f552e400700 1 -- 172.21.15.1:0/1654599262 shutdown_connections 2024-02-28T17:05:43.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.275+0000 7f552e400700 1 -- 172.21.15.1:0/1654599262 wait complete. 2024-02-28T17:05:43.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.276+0000 7f552e400700 1 Processor -- start 2024-02-28T17:05:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.276+0000 7f552e400700 1 -- start start 2024-02-28T17:05:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.277+0000 7f552e400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5528070870 0x7f552811a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.277+0000 7f552e400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528071f60 0x7f552811a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.277+0000 7f552e400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552811aba0 con 0x7f5528071f60 2024-02-28T17:05:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.277+0000 7f552e400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528113eb0 con 0x7f5528070870 2024-02-28T17:05:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.277+0000 7f5526a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528071f60 0x7f552811a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.277+0000 7f5526a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528071f60 0x7f552811a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42348/0 (socket says 172.21.15.1:42348) 2024-02-28T17:05:43.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.277+0000 7f5526a00700 1 -- 172.21.15.1:0/1908468051 learned_addr learned my addr 172.21.15.1:0/1908468051 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.277+0000 7f5527400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5528070870 0x7f552811a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f5526a00700 1 -- 172.21.15.1:0/1908468051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5528070870 msgr2=0x7f552811a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f5526a00700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5528070870 0x7f552811a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f5526a00700 1 -- 172.21.15.1:0/1908468051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5514005040 con 0x7f5528071f60 2024-02-28T17:05:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f5526a00700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528071f60 0x7f552811a550 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f5514004f50 tx=0x7f5514009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:43.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f551f400700 1 -- 172.21.15.1:0/1908468051 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f551400e040 con 0x7f5528071f60 2024-02-28T17:05:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f551f400700 1 -- 172.21.15.1:0/1908468051 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f551400bc10 con 0x7f5528071f60 2024-02-28T17:05:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f551f400700 1 -- 172.21.15.1:0/1908468051 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f551401c630 con 0x7f5528071f60 2024-02-28T17:05:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5528114130 con 0x7f5528071f60 2024-02-28T17:05:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.278+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5528114670 con 0x7f5528071f60 2024-02-28T17:05:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.279+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55280fe400 con 0x7f5528071f60 2024-02-28T17:05:43.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.280+0000 7f551f400700 1 -- 172.21.15.1:0/1908468051 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f551400bd80 con 0x7f5528071f60 2024-02-28T17:05:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.282+0000 7f551f400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f550c06d7f0 0x7f550c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.282+0000 7f551f400700 1 -- 172.21.15.1:0/1908468051 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f551408e2a0 con 0x7f5528071f60 2024-02-28T17:05:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.282+0000 7f5527400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f550c06d7f0 0x7f550c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.283+0000 7f5527400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f550c06d7f0 0x7f550c06fca0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5518007fd0 tx=0x7f5518009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.287+0000 7f551f400700 1 -- 172.21.15.1:0/1908468051 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f551405b680 con 0x7f5528071f60 2024-02-28T17:05:43.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.514+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f552804bed0 con 0x7f550c06d7f0 2024-02-28T17:05:43.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.543+0000 7f551f400700 1 -- 172.21.15.1:0/1908468051 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f552804bed0 con 0x7f550c06d7f0 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (31m) 8m ago 46m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (28m) 8m ago 46m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (28m) 8m ago 44m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (31m) 8m ago 45m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (24m) 8m ago 38m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (24m) 8m ago 38m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (38m) 8m ago 38m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (8m) 8m ago 38m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (34m) 8m ago 47m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (33m) 8m ago 44m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:05:43.546 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (30m) 8m ago 48m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (29m) 8m ago 44m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (32m) 8m ago 46m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (32m) 8m ago 44m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (27m) 8m ago 43m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (27m) 8m ago 43m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (26m) 8m ago 42m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (26m) 8m ago 42m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (25m) 8m ago 41m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (24m) 8m ago 40m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:05:43.547 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (32m) 8m ago 45m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f550c06d7f0 msgr2=0x7f550c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f550c06d7f0 0x7f550c06fca0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5518007fd0 tx=0x7f5518009040 comp rx=0 tx=0).stop 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528071f60 msgr2=0x7f552811a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528071f60 0x7f552811a550 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f5514004f50 tx=0x7f5514009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 shutdown_connections 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f550c06d7f0 0x7f550c06fca0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5528070870 0x7f552811a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 --2- 172.21.15.1:0/1908468051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5528071f60 0x7f552811a550 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 >> 172.21.15.1:0/1908468051 conn(0x7f55280f8ae0 msgr2=0x7f55280fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:43.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 shutdown_connections 2024-02-28T17:05:43.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.546+0000 7f552e400700 1 -- 172.21.15.1:0/1908468051 wait complete. 2024-02-28T17:05:43.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.700+0000 7ffbcbe00700 1 -- 172.21.15.1:0/2093441473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 msgr2=0x7ffbc4105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.700+0000 7ffbcbe00700 1 --2- 172.21.15.1:0/2093441473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 0x7ffbc4105540 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8004940 tx=0x7ffbb8009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.701+0000 7ffbcbe00700 1 -- 172.21.15.1:0/2093441473 shutdown_connections 2024-02-28T17:05:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.701+0000 7ffbcbe00700 1 --2- 172.21.15.1:0/2093441473 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbc4105b10 0x7ffbc40ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.701+0000 7ffbcbe00700 1 --2- 172.21.15.1:0/2093441473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 0x7ffbc4105540 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.701+0000 7ffbcbe00700 1 -- 172.21.15.1:0/2093441473 >> 172.21.15.1:0/2093441473 conn(0x7ffbc40face0 msgr2=0x7ffbc40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:43.703 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:43 smithi001 ceph-mon[112940]: from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.701+0000 7ffbcbe00700 1 -- 172.21.15.1:0/2093441473 shutdown_connections 2024-02-28T17:05:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.701+0000 7ffbcbe00700 1 -- 172.21.15.1:0/2093441473 wait complete. 2024-02-28T17:05:43.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.702+0000 7ffbcbe00700 1 Processor -- start 2024-02-28T17:05:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.702+0000 7ffbcbe00700 1 -- start start 2024-02-28T17:05:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.702+0000 7ffbcbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 0x7ffbc4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.702+0000 7ffbcbe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbc4105b10 0x7ffbc41161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.702+0000 7ffbcbe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc41167f0 con 0x7ffbc4105170 2024-02-28T17:05:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.702+0000 7ffbcbe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc410fb40 con 0x7ffbc4105b10 2024-02-28T17:05:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.703+0000 7ffbc9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 0x7ffbc4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.703+0000 7ffbc9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 0x7ffbc4115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42362/0 (socket says 172.21.15.1:42362) 2024-02-28T17:05:43.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.703+0000 7ffbc9400700 1 -- 172.21.15.1:0/266578840 learned_addr learned my addr 172.21.15.1:0/266578840 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.703+0000 7ffbc9400700 1 -- 172.21.15.1:0/266578840 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbc4105b10 msgr2=0x7ffbc41161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.703+0000 7ffbc9400700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbc4105b10 0x7ffbc41161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.703+0000 7ffbc9400700 1 -- 172.21.15.1:0/266578840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb8005040 con 0x7ffbc4105170 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.703+0000 7ffbc9400700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 0x7ffbc4115c60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009fd0 tx=0x7ffbb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.704+0000 7ffbc1600700 1 -- 172.21.15.1:0/266578840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbb800e040 con 0x7ffbc4105170 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.704+0000 7ffbc1600700 1 -- 172.21.15.1:0/266578840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbb800bbf0 con 0x7ffbc4105170 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.704+0000 7ffbc1600700 1 -- 172.21.15.1:0/266578840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbb801c3f0 con 0x7ffbc4105170 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.704+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc410fdc0 con 0x7ffbc4105170 2024-02-28T17:05:43.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.704+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc4110300 con 0x7ffbc4105170 2024-02-28T17:05:43.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.705+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc4102480 con 0x7ffbc4105170 2024-02-28T17:05:43.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.707+0000 7ffbc1600700 1 -- 172.21.15.1:0/266578840 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffbb800bd60 con 0x7ffbc4105170 2024-02-28T17:05:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.707+0000 7ffbc1600700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbac06d7f0 0x7ffbac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:43.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.708+0000 7ffbc1600700 1 -- 172.21.15.1:0/266578840 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffbb808e1c0 con 0x7ffbc4105170 2024-02-28T17:05:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.708+0000 7ffbc8a00700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbac06d7f0 0x7ffbac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:43.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.709+0000 7ffbc8a00700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbac06d7f0 0x7ffbac06fca0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4111540 tx=0x7ffbb0009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:43.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:43.712+0000 7ffbc1600700 1 -- 172.21.15.1:0/266578840 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffbb8057a70 con 0x7ffbc4105170 2024-02-28T17:05:43.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:43 smithi066 ceph-mon[85781]: from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:44.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.025+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffbc4111280 con 0x7ffbc4105170 2024-02-28T17:05:44.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.026+0000 7ffbc1600700 1 -- 172.21.15.1:0/266578840 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7ffbb805b130 con 0x7ffbc4105170 2024-02-28T17:05:44.029 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:05:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:05:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:05:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:05:44.029 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:05:44.030 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:05:44.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:05:44.030 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:05:44.030 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:05:44.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:05:44.031 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:05:44.031 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:05:44.031 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:05:44.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:05:44.031 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:05:44.031 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:05:44.032 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:05:44.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:05:44.032 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:05:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbac06d7f0 msgr2=0x7ffbac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbac06d7f0 0x7ffbac06fca0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4111540 tx=0x7ffbb0009ce0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 msgr2=0x7ffbc4115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 0x7ffbc4115c60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009fd0 tx=0x7ffbb8009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 shutdown_connections 2024-02-28T17:05:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbac06d7f0 0x7ffbac06fca0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc4105170 0x7ffbc4115c60 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 --2- 172.21.15.1:0/266578840 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbc4105b10 0x7ffbc41161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.029+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 >> 172.21.15.1:0/266578840 conn(0x7ffbc40face0 msgr2=0x7ffbc40fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.030+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 shutdown_connections 2024-02-28T17:05:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.030+0000 7ffbcbe00700 1 -- 172.21.15.1:0/266578840 wait complete. 2024-02-28T17:05:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.183+0000 7fa9cec00700 1 -- 172.21.15.1:0/1535858333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 msgr2=0x7fa9c8107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.183+0000 7fa9cec00700 1 --2- 172.21.15.1:0/1535858333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 0x7fa9c8107050 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009d10 tx=0x7fa9b0004d10 comp rx=0 tx=0).stop 2024-02-28T17:05:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.184+0000 7fa9cec00700 1 -- 172.21.15.1:0/1535858333 shutdown_connections 2024-02-28T17:05:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.184+0000 7fa9cec00700 1 --2- 172.21.15.1:0/1535858333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 0x7fa9c8107050 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.184+0000 7fa9cec00700 1 --2- 172.21.15.1:0/1535858333 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9c80fdfe0 0x7fa9c80fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.184+0000 7fa9cec00700 1 -- 172.21.15.1:0/1535858333 >> 172.21.15.1:0/1535858333 conn(0x7fa9c8071d90 msgr2=0x7fa9c8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.184+0000 7fa9cec00700 1 -- 172.21.15.1:0/1535858333 shutdown_connections 2024-02-28T17:05:44.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.184+0000 7fa9cec00700 1 -- 172.21.15.1:0/1535858333 wait complete. 2024-02-28T17:05:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.185+0000 7fa9cec00700 1 Processor -- start 2024-02-28T17:05:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.185+0000 7fa9cec00700 1 -- start start 2024-02-28T17:05:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.185+0000 7fa9cec00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9c80fdfe0 0x7fa9c8115e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.185+0000 7fa9cec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 0x7fa9c81163d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:44.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.185+0000 7fa9cec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c8116a20 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.185+0000 7fa9cec00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c810fd30 con 0x7fa9c80fdfe0 2024-02-28T17:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9bf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 0x7fa9c81163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9bf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 0x7fa9c81163d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42388/0 (socket says 172.21.15.1:42388) 2024-02-28T17:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9bf400700 1 -- 172.21.15.1:0/468821027 learned_addr learned my addr 172.21.15.1:0/468821027 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9c7e00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9c80fdfe0 0x7fa9c8115e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9bf400700 1 -- 172.21.15.1:0/468821027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9c80fdfe0 msgr2=0x7fa9c8115e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9bf400700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9c80fdfe0 0x7fa9c8115e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9bf400700 1 -- 172.21.15.1:0/468821027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b0009940 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9c7e00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9c80fdfe0 0x7fa9c8115e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.186+0000 7fa9bf400700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 0x7fa9c81163d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0008dd0 tx=0x7fa9b0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.187+0000 7fa9c4c00700 1 -- 172.21.15.1:0/468821027 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9b000e040 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.187+0000 7fa9c4c00700 1 -- 172.21.15.1:0/468821027 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9b00128e0 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.187+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c810ffb0 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.187+0000 7fa9c4c00700 1 -- 172.21.15.1:0/468821027 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9b001c8f0 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.187+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c81104f0 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.188+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c8106b20 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.189+0000 7fa9c4c00700 1 -- 172.21.15.1:0/468821027 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa9b0012a50 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.190+0000 7fa9c4c00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9ac06d8c0 0x7fa9ac06fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.190+0000 7fa9c4c00700 1 -- 172.21.15.1:0/468821027 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa9b008ebb0 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.190+0000 7fa9c7e00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9ac06d8c0 0x7fa9ac06fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.191+0000 7fa9c7e00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9ac06d8c0 0x7fa9ac06fd70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009fd0 tx=0x7fa9b800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:44.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.195+0000 7fa9c4c00700 1 -- 172.21.15.1:0/468821027 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa9b005bf10 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.457+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa9c80738b0 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.458+0000 7fa9c4c00700 1 -- 172.21.15.1:0/468821027 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fa9b0004e10 con 0x7fa9c80fe8f0 2024-02-28T17:05:44.460 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:05:44.460 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:05:44.460 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:05:44.461 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:05:44.462 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:05:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:05:44.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9ac06d8c0 msgr2=0x7fa9ac06fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9ac06d8c0 0x7fa9ac06fd70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009fd0 tx=0x7fa9b800b040 comp rx=0 tx=0).stop 2024-02-28T17:05:44.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 msgr2=0x7fa9c81163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 0x7fa9c81163d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0008dd0 tx=0x7fa9b0004d10 comp rx=0 tx=0).stop 2024-02-28T17:05:44.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 shutdown_connections 2024-02-28T17:05:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9ac06d8c0 0x7fa9ac06fd70 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9c80fdfe0 0x7fa9c8115e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 --2- 172.21.15.1:0/468821027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9c80fe8f0 0x7fa9c81163d0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 >> 172.21.15.1:0/468821027 conn(0x7fa9c8071d90 msgr2=0x7fa9c80fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 shutdown_connections 2024-02-28T17:05:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.461+0000 7fa9cec00700 1 -- 172.21.15.1:0/468821027 wait complete. 2024-02-28T17:05:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:05:44.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.609+0000 7f5daba00700 1 -- 172.21.15.1:0/79897006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 msgr2=0x7f5da40fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.609+0000 7f5daba00700 1 --2- 172.21.15.1:0/79897006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 0x7f5da40fcf60 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5d90004940 tx=0x7f5d90009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.609+0000 7f5daba00700 1 -- 172.21.15.1:0/79897006 shutdown_connections 2024-02-28T17:05:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.609+0000 7f5daba00700 1 --2- 172.21.15.1:0/79897006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 0x7f5da40fcf60 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.609+0000 7f5daba00700 1 --2- 172.21.15.1:0/79897006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da4102f50 0x7f5da4103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.609+0000 7f5daba00700 1 -- 172.21.15.1:0/79897006 >> 172.21.15.1:0/79897006 conn(0x7f5da40f8ae0 msgr2=0x7f5da40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.610+0000 7f5daba00700 1 -- 172.21.15.1:0/79897006 shutdown_connections 2024-02-28T17:05:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.610+0000 7f5daba00700 1 -- 172.21.15.1:0/79897006 wait complete. 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.610+0000 7f5daba00700 1 Processor -- start 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5daba00700 1 -- start start 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5daba00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da4102f50 0x7f5da4115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5daba00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 0x7f5da41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5daba00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4116810 con 0x7f5da41038f0 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5daba00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da410fb40 con 0x7f5da4102f50 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5da3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 0x7f5da41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5da3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 0x7f5da41161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42410/0 (socket says 172.21.15.1:42410) 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5da3e00700 1 -- 172.21.15.1:0/2077733562 learned_addr learned my addr 172.21.15.1:0/2077733562 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.611+0000 7f5da9000700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da4102f50 0x7f5da4115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5da3e00700 1 -- 172.21.15.1:0/2077733562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da4102f50 msgr2=0x7f5da4115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5da3e00700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da4102f50 0x7f5da4115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5da3e00700 1 -- 172.21.15.1:0/2077733562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d90005040 con 0x7f5da41038f0 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5da3e00700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 0x7f5da41161c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009030 tx=0x7f5d90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5da0c00700 1 -- 172.21.15.1:0/2077733562 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d9000e040 con 0x7f5da41038f0 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5da0c00700 1 -- 172.21.15.1:0/2077733562 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d9000bbf0 con 0x7f5da41038f0 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5da0c00700 1 -- 172.21.15.1:0/2077733562 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d9001c630 con 0x7f5da41038f0 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da410fdc0 con 0x7f5da41038f0 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.612+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da4110300 con 0x7f5da41038f0 2024-02-28T17:05:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.613+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da40fc6e0 con 0x7f5da41038f0 2024-02-28T17:05:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.614+0000 7f5da0c00700 1 -- 172.21.15.1:0/2077733562 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5d9000bd60 con 0x7f5da41038f0 2024-02-28T17:05:44.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.615+0000 7f5da0c00700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7f0 0x7f5d8c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:44.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.615+0000 7f5da0c00700 1 -- 172.21.15.1:0/2077733562 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5d9008e2c0 con 0x7f5da41038f0 2024-02-28T17:05:44.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.616+0000 7f5da9000700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7f0 0x7f5d8c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:44.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.616+0000 7f5da9000700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7f0 0x7f5d8c06fca0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5d98004990 tx=0x7f5d98009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:44.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.620+0000 7f5da0c00700 1 -- 172.21.15.1:0/2077733562 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5d9005b620 con 0x7f5da41038f0 2024-02-28T17:05:44.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:44 smithi066 ceph-mon[85781]: pgmap v939: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:44.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:44 smithi066 ceph-mon[85781]: from='client.35686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:44.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:44 smithi066 ceph-mon[85781]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:44.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:44 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/266578840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:05:44.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:44 smithi001 ceph-mon[112940]: pgmap v939: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:44.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:44 smithi001 ceph-mon[112940]: from='client.35686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:44.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:44 smithi001 ceph-mon[112940]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:44.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:44 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/266578840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:05:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.858+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5da41111f0 con 0x7f5d8c06d7f0 2024-02-28T17:05:44.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.860+0000 7f5da0c00700 1 -- 172.21.15.1:0/2077733562 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5da41111f0 con 0x7f5d8c06d7f0 2024-02-28T17:05:44.862 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:05:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:05:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:05:44.862 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:05:44.863 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:05:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7f0 msgr2=0x7f5d8c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7f0 0x7f5d8c06fca0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5d98004990 tx=0x7f5d98009ee0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 msgr2=0x7f5da41161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 0x7f5da41161c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009030 tx=0x7f5d90009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 shutdown_connections 2024-02-28T17:05:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d8c06d7f0 0x7f5d8c06fca0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5da4102f50 0x7f5da4115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 --2- 172.21.15.1:0/2077733562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5da41038f0 0x7f5da41161c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 >> 172.21.15.1:0/2077733562 conn(0x7f5da40f8ae0 msgr2=0x7f5da40fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 shutdown_connections 2024-02-28T17:05:44.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:44.863+0000 7f5daba00700 1 -- 172.21.15.1:0/2077733562 wait complete. 2024-02-28T17:05:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.018+0000 7fc7ad200700 1 -- 172.21.15.1:0/1550684511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 msgr2=0x7fc7a81092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.018+0000 7fc7ad200700 1 --2- 172.21.15.1:0/1550684511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 0x7fc7a81092c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc794004970 tx=0x7fc794009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.018+0000 7fc7ad200700 1 -- 172.21.15.1:0/1550684511 shutdown_connections 2024-02-28T17:05:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.018+0000 7fc7ad200700 1 --2- 172.21.15.1:0/1550684511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 0x7fc7a81092c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.018+0000 7fc7ad200700 1 --2- 172.21.15.1:0/1550684511 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a80702d0 0x7fc7a80706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.018+0000 7fc7ad200700 1 -- 172.21.15.1:0/1550684511 >> 172.21.15.1:0/1550684511 conn(0x7fc7a80f8ae0 msgr2=0x7fc7a80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.018+0000 7fc7ad200700 1 -- 172.21.15.1:0/1550684511 shutdown_connections 2024-02-28T17:05:45.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.018+0000 7fc7ad200700 1 -- 172.21.15.1:0/1550684511 wait complete. 2024-02-28T17:05:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.019+0000 7fc7ad200700 1 Processor -- start 2024-02-28T17:05:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.019+0000 7fc7ad200700 1 -- start start 2024-02-28T17:05:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7ad200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a80702d0 0x7fc7a8115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7ad200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 0x7fc7a8116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7ad200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a81167a0 con 0x7fc7a8070be0 2024-02-28T17:05:45.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7ad200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a810faf0 con 0x7fc7a80702d0 2024-02-28T17:05:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7a5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 0x7fc7a8116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7a5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 0x7fc7a8116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42422/0 (socket says 172.21.15.1:42422) 2024-02-28T17:05:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7a5400700 1 -- 172.21.15.1:0/1212403159 learned_addr learned my addr 172.21.15.1:0/1212403159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:05:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7a5e00700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a80702d0 0x7fc7a8115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7a5400700 1 -- 172.21.15.1:0/1212403159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a80702d0 msgr2=0x7fc7a8115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7a5400700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a80702d0 0x7fc7a8115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.020+0000 7fc7a5400700 1 -- 172.21.15.1:0/1212403159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc794005040 con 0x7fc7a8070be0 2024-02-28T17:05:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.021+0000 7fc7a5400700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 0x7fc7a8116150 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc794009150 tx=0x7fc794009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.021+0000 7fc79e000700 1 -- 172.21.15.1:0/1212403159 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc79400e040 con 0x7fc7a8070be0 2024-02-28T17:05:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.021+0000 7fc79e000700 1 -- 172.21.15.1:0/1212403159 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc794004290 con 0x7fc7a8070be0 2024-02-28T17:05:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.021+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a810fd70 con 0x7fc7a8070be0 2024-02-28T17:05:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.021+0000 7fc79e000700 1 -- 172.21.15.1:0/1212403159 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc79401c630 con 0x7fc7a8070be0 2024-02-28T17:05:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.021+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a81102b0 con 0x7fc7a8070be0 2024-02-28T17:05:45.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.022+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a8106980 con 0x7fc7a8070be0 2024-02-28T17:05:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.024+0000 7fc79e000700 1 -- 172.21.15.1:0/1212403159 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc794013980 con 0x7fc7a8070be0 2024-02-28T17:05:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.024+0000 7fc79e000700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78c06d7f0 0x7fc78c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:05:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.025+0000 7fc79e000700 1 -- 172.21.15.1:0/1212403159 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc79408e210 con 0x7fc7a8070be0 2024-02-28T17:05:45.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.025+0000 7fc7a5e00700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78c06d7f0 0x7fc78c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:05:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.025+0000 7fc7a5e00700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78c06d7f0 0x7fc78c06fca0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc798007fd0 tx=0x7fc798009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:05:45.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.029+0000 7fc79e000700 1 -- 172.21.15.1:0/1212403159 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc794057ac0 con 0x7fc7a8070be0 2024-02-28T17:05:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.329+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc7a804bed0 con 0x7fc7a8070be0 2024-02-28T17:05:45.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.330+0000 7fc79e000700 1 -- 172.21.15.1:0/1212403159 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fc79405b180 con 0x7fc7a8070be0 2024-02-28T17:05:45.332 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:05:45.332 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:05:45.332 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:05:45.332 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:05:45.332 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:05:45.332 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:05:45.332 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:05:45.332 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:05:45.333 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:05:45.333 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:05:45.333 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:05:45.333 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:05:45.333 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:05:45.334 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:05:45.334 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:05:45.334 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:05:45.334 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:05:45.334 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:05:45.334 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:05:45.334 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:05:45.334 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:05:45.335 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:05:45.335 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:05:45.335 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:05:45.335 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78c06d7f0 msgr2=0x7fc78c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78c06d7f0 0x7fc78c06fca0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc798007fd0 tx=0x7fc798009040 comp rx=0 tx=0).stop 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 msgr2=0x7fc7a8116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 0x7fc7a8116150 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc794009150 tx=0x7fc794009e30 comp rx=0 tx=0).stop 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 shutdown_connections 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc78c06d7f0 0x7fc78c06fca0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7a80702d0 0x7fc7a8115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 --2- 172.21.15.1:0/1212403159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a8070be0 0x7fc7a8116150 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.332+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 >> 172.21.15.1:0/1212403159 conn(0x7fc7a80f8ae0 msgr2=0x7fc7a8103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.333+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 shutdown_connections 2024-02-28T17:05:45.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:05:45.333+0000 7fc7ad200700 1 -- 172.21.15.1:0/1212403159 wait complete. 2024-02-28T17:05:45.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:45 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/468821027' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:05:45.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:45 smithi066 ceph-mon[85781]: from='client.35702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:45.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:45 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1212403159' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:05:45.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:45 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/468821027' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:05:45.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:45 smithi001 ceph-mon[112940]: from='client.35702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:05:45.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:45 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1212403159' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:05:46.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:46 smithi066 ceph-mon[85781]: pgmap v940: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:46.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:46 smithi001 ceph-mon[112940]: pgmap v940: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:48.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:48 smithi066 ceph-mon[85781]: pgmap v941: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:48.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:48 smithi001 ceph-mon[112940]: pgmap v941: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:50.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:50 smithi066 ceph-mon[85781]: pgmap v942: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:50.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:50 smithi001 ceph-mon[112940]: pgmap v942: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:52.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:52 smithi066 ceph-mon[85781]: pgmap v943: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:52.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:52 smithi001 ceph-mon[112940]: pgmap v943: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:54.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:54 smithi066 ceph-mon[85781]: pgmap v944: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:54.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:54 smithi001 ceph-mon[112940]: pgmap v944: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:56.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:56 smithi066 ceph-mon[85781]: pgmap v945: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:56.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:05:56.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:56 smithi001 ceph-mon[112940]: pgmap v945: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:56.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:05:58.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:05:58 smithi066 ceph-mon[85781]: pgmap v946: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:05:58.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:05:58 smithi001 ceph-mon[112940]: pgmap v946: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:00.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:00 smithi066 ceph-mon[85781]: pgmap v947: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:00.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:00 smithi001 ceph-mon[112940]: pgmap v947: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:02.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:02 smithi066 ceph-mon[85781]: pgmap v948: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:02.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:02 smithi001 ceph-mon[112940]: pgmap v948: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:04.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:04 smithi066 ceph-mon[85781]: pgmap v949: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:04.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:04 smithi001 ceph-mon[112940]: pgmap v949: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:06.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:06 smithi066 ceph-mon[85781]: pgmap v950: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:06.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:06 smithi001 ceph-mon[112940]: pgmap v950: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:08.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:08 smithi066 ceph-mon[85781]: pgmap v951: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:08.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:08 smithi001 ceph-mon[112940]: pgmap v951: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:10.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:10 smithi066 ceph-mon[85781]: pgmap v952: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:10.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:10 smithi001 ceph-mon[112940]: pgmap v952: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:11.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:06:11.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:06:12.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:12 smithi066 ceph-mon[85781]: pgmap v953: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:12.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:12 smithi001 ceph-mon[112940]: pgmap v953: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:14.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:14 smithi066 ceph-mon[85781]: pgmap v954: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:14.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:14 smithi001 ceph-mon[112940]: pgmap v954: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.504+0000 7fd4ad600700 1 -- 172.21.15.1:0/1398796554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 msgr2=0x7fd4a8109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.504+0000 7fd4ad600700 1 --2- 172.21.15.1:0/1398796554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 0x7fd4a8109260 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd494004970 tx=0x7fd494009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.505+0000 7fd4ad600700 1 -- 172.21.15.1:0/1398796554 shutdown_connections 2024-02-28T17:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.505+0000 7fd4ad600700 1 --2- 172.21.15.1:0/1398796554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 0x7fd4a8109260 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.505+0000 7fd4ad600700 1 --2- 172.21.15.1:0/1398796554 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4a8070330 0x7fd4a8070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.505+0000 7fd4ad600700 1 -- 172.21.15.1:0/1398796554 >> 172.21.15.1:0/1398796554 conn(0x7fd4a80f8ae0 msgr2=0x7fd4a80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.505+0000 7fd4ad600700 1 -- 172.21.15.1:0/1398796554 shutdown_connections 2024-02-28T17:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.505+0000 7fd4ad600700 1 -- 172.21.15.1:0/1398796554 wait complete. 2024-02-28T17:06:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.505+0000 7fd4ad600700 1 Processor -- start 2024-02-28T17:06:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.506+0000 7fd4ad600700 1 -- start start 2024-02-28T17:06:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.506+0000 7fd4ad600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4a8070330 0x7fd4a8115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.506+0000 7fd4ad600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 0x7fd4a8116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:15.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.506+0000 7fd4ad600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a8116790 con 0x7fd4a8070c40 2024-02-28T17:06:15.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.506+0000 7fd4ad600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a810fae0 con 0x7fd4a8070330 2024-02-28T17:06:15.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.506+0000 7fd4a6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 0x7fd4a8116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:15.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.507+0000 7fd4a6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 0x7fd4a8116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55340/0 (socket says 172.21.15.1:55340) 2024-02-28T17:06:15.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.507+0000 7fd4a6000700 1 -- 172.21.15.1:0/627060255 learned_addr learned my addr 172.21.15.1:0/627060255 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:15.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.507+0000 7fd4a6a00700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4a8070330 0x7fd4a8115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.507+0000 7fd4a6000700 1 -- 172.21.15.1:0/627060255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4a8070330 msgr2=0x7fd4a8115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.507+0000 7fd4a6000700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4a8070330 0x7fd4a8115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.507+0000 7fd4a6000700 1 -- 172.21.15.1:0/627060255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494005040 con 0x7fd4a8070c40 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.507+0000 7fd4a6000700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 0x7fd4a8116140 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd494009150 tx=0x7fd494009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.508+0000 7fd49ea00700 1 -- 172.21.15.1:0/627060255 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd49400e040 con 0x7fd4a8070c40 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.508+0000 7fd49ea00700 1 -- 172.21.15.1:0/627060255 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd494004290 con 0x7fd4a8070c40 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.508+0000 7fd49ea00700 1 -- 172.21.15.1:0/627060255 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd49401c630 con 0x7fd4a8070c40 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.508+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a810fd60 con 0x7fd4a8070c40 2024-02-28T17:06:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.508+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a8110250 con 0x7fd4a8070c40 2024-02-28T17:06:15.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.509+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a8106920 con 0x7fd4a8070c40 2024-02-28T17:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.510+0000 7fd49ea00700 1 -- 172.21.15.1:0/627060255 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd494013420 con 0x7fd4a8070c40 2024-02-28T17:06:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.512+0000 7fd49ea00700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd48c06d7f0 0x7fd48c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.512+0000 7fd49ea00700 1 -- 172.21.15.1:0/627060255 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd49408e2b0 con 0x7fd4a8070c40 2024-02-28T17:06:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.512+0000 7fd4a6a00700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd48c06d7f0 0x7fd48c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.513+0000 7fd4a6a00700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd48c06d7f0 0x7fd48c06fca0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd498004990 tx=0x7fd498009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.517+0000 7fd49ea00700 1 -- 172.21.15.1:0/627060255 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd494057ae0 con 0x7fd4a8070c40 2024-02-28T17:06:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.764+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4a804bed0 con 0x7fd48c06d7f0 2024-02-28T17:06:15.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.766+0000 7fd49ea00700 1 -- 172.21.15.1:0/627060255 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd4a804bed0 con 0x7fd48c06d7f0 2024-02-28T17:06:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd48c06d7f0 msgr2=0x7fd48c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd48c06d7f0 0x7fd48c06fca0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd498004990 tx=0x7fd498009ee0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 msgr2=0x7fd4a8116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 0x7fd4a8116140 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd494009150 tx=0x7fd494009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 shutdown_connections 2024-02-28T17:06:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd48c06d7f0 0x7fd48c06fca0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd4a8070330 0x7fd4a8115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 --2- 172.21.15.1:0/627060255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4a8070c40 0x7fd4a8116140 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.769+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 >> 172.21.15.1:0/627060255 conn(0x7fd4a80f8ae0 msgr2=0x7fd4a8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.770+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 shutdown_connections 2024-02-28T17:06:15.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.770+0000 7fd4ad600700 1 -- 172.21.15.1:0/627060255 wait complete. 2024-02-28T17:06:15.786 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:06:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.927+0000 7f6143200700 1 -- 172.21.15.1:0/2766281375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 msgr2=0x7f613c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.927+0000 7f6143200700 1 --2- 172.21.15.1:0/2766281375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 0x7f613c0fcf60 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6124004940 tx=0x7f6124009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.927+0000 7f6143200700 1 -- 172.21.15.1:0/2766281375 shutdown_connections 2024-02-28T17:06:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.927+0000 7f6143200700 1 --2- 172.21.15.1:0/2766281375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 0x7f613c0fcf60 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.927+0000 7f6143200700 1 --2- 172.21.15.1:0/2766281375 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f613c102f50 0x7f613c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.927+0000 7f6143200700 1 -- 172.21.15.1:0/2766281375 >> 172.21.15.1:0/2766281375 conn(0x7f613c0f8ae0 msgr2=0x7f613c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.928+0000 7f6143200700 1 -- 172.21.15.1:0/2766281375 shutdown_connections 2024-02-28T17:06:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.928+0000 7f6143200700 1 -- 172.21.15.1:0/2766281375 wait complete. 2024-02-28T17:06:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.928+0000 7f6143200700 1 Processor -- start 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.929+0000 7f6143200700 1 -- start start 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.929+0000 7f6143200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f613c102f50 0x7f613c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.929+0000 7f6143200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 0x7f613c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.929+0000 7f6143200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c1168f0 con 0x7f613c1038f0 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.929+0000 7f6143200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c10fb70 con 0x7f613c102f50 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.929+0000 7f6133e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 0x7f613c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.929+0000 7f6133e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 0x7f613c116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55346/0 (socket says 172.21.15.1:55346) 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.929+0000 7f6133e00700 1 -- 172.21.15.1:0/1179632917 learned_addr learned my addr 172.21.15.1:0/1179632917 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.930+0000 7f613be00700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f613c102f50 0x7f613c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.930+0000 7f6133e00700 1 -- 172.21.15.1:0/1179632917 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f613c102f50 msgr2=0x7f613c115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.930+0000 7f6133e00700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f613c102f50 0x7f613c115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.930+0000 7f6133e00700 1 -- 172.21.15.1:0/1179632917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124005040 con 0x7f613c1038f0 2024-02-28T17:06:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.930+0000 7f6133e00700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 0x7f613c116210 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6124009030 tx=0x7f6124009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.930+0000 7f6138c00700 1 -- 172.21.15.1:0/1179632917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f612400e040 con 0x7f613c1038f0 2024-02-28T17:06:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.931+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c10fd90 con 0x7f613c1038f0 2024-02-28T17:06:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.931+0000 7f6138c00700 1 -- 172.21.15.1:0/1179632917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f612400bbf0 con 0x7f613c1038f0 2024-02-28T17:06:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.931+0000 7f6138c00700 1 -- 172.21.15.1:0/1179632917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f612401c630 con 0x7f613c1038f0 2024-02-28T17:06:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.931+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c1102d0 con 0x7f613c1038f0 2024-02-28T17:06:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.932+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613c0fc6e0 con 0x7f613c1038f0 2024-02-28T17:06:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.933+0000 7f6138c00700 1 -- 172.21.15.1:0/1179632917 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f612400bd60 con 0x7f613c1038f0 2024-02-28T17:06:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.934+0000 7f6138c00700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f612006d7f0 0x7f612006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.934+0000 7f6138c00700 1 -- 172.21.15.1:0/1179632917 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f612408e2c0 con 0x7f613c1038f0 2024-02-28T17:06:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.934+0000 7f613be00700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f612006d7f0 0x7f612006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.935+0000 7f613be00700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f612006d7f0 0x7f612006fca0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f612c006fd0 tx=0x7f612c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:15.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:15.939+0000 7f6138c00700 1 -- 172.21.15.1:0/1179632917 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f612405b620 con 0x7f613c1038f0 2024-02-28T17:06:16.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.183+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f613c1113c0 con 0x7f612006d7f0 2024-02-28T17:06:16.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.187+0000 7f6138c00700 1 -- 172.21.15.1:0/1179632917 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f613c1113c0 con 0x7f612006d7f0 2024-02-28T17:06:16.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f612006d7f0 msgr2=0x7f612006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:16.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f612006d7f0 0x7f612006fca0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f612c006fd0 tx=0x7f612c008040 comp rx=0 tx=0).stop 2024-02-28T17:06:16.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 msgr2=0x7f613c116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 0x7f613c116210 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6124009030 tx=0x7f6124009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 shutdown_connections 2024-02-28T17:06:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f612006d7f0 0x7f612006fca0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f613c102f50 0x7f613c115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 --2- 172.21.15.1:0/1179632917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f613c1038f0 0x7f613c116210 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 >> 172.21.15.1:0/1179632917 conn(0x7f613c0f8ae0 msgr2=0x7f613c0fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:16.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 shutdown_connections 2024-02-28T17:06:16.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.190+0000 7f6143200700 1 -- 172.21.15.1:0/1179632917 wait complete. 2024-02-28T17:06:16.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.341+0000 7ff726800700 1 -- 172.21.15.1:0/4197525884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 msgr2=0x7ff7200fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:16.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.341+0000 7ff726800700 1 --2- 172.21.15.1:0/4197525884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 0x7ff7200fcf60 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff708004940 tx=0x7ff708009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:16.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.341+0000 7ff726800700 1 -- 172.21.15.1:0/4197525884 shutdown_connections 2024-02-28T17:06:16.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.341+0000 7ff726800700 1 --2- 172.21.15.1:0/4197525884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 0x7ff7200fcf60 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.341+0000 7ff726800700 1 --2- 172.21.15.1:0/4197525884 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff720102f50 0x7ff720103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.341+0000 7ff726800700 1 -- 172.21.15.1:0/4197525884 >> 172.21.15.1:0/4197525884 conn(0x7ff7200f8ae0 msgr2=0x7ff7200faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:16.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.341+0000 7ff726800700 1 -- 172.21.15.1:0/4197525884 shutdown_connections 2024-02-28T17:06:16.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.341+0000 7ff726800700 1 -- 172.21.15.1:0/4197525884 wait complete. 2024-02-28T17:06:16.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.342+0000 7ff726800700 1 Processor -- start 2024-02-28T17:06:16.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.342+0000 7ff726800700 1 -- start start 2024-02-28T17:06:16.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.343+0000 7ff726800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff720102f50 0x7ff720070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:16.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.343+0000 7ff726800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 0x7ff7200707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:16.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.343+0000 7ff726800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720070e60 con 0x7ff7201038f0 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.343+0000 7ff726800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720070fd0 con 0x7ff720102f50 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.343+0000 7ff717e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 0x7ff7200707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.343+0000 7ff717e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 0x7ff7200707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55372/0 (socket says 172.21.15.1:55372) 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.343+0000 7ff717e00700 1 -- 172.21.15.1:0/4250708477 learned_addr learned my addr 172.21.15.1:0/4250708477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.343+0000 7ff71f400700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff720102f50 0x7ff720070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff717e00700 1 -- 172.21.15.1:0/4250708477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff720102f50 msgr2=0x7ff720070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff717e00700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff720102f50 0x7ff720070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff717e00700 1 -- 172.21.15.1:0/4250708477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708005040 con 0x7ff7201038f0 2024-02-28T17:06:16.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff71f400700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff720102f50 0x7ff720070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:06:16.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff717e00700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 0x7ff7200707d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff708009030 tx=0x7ff708009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:16.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff717400700 1 -- 172.21.15.1:0/4250708477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff70800e040 con 0x7ff7201038f0 2024-02-28T17:06:16.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff717400700 1 -- 172.21.15.1:0/4250708477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff70800bbf0 con 0x7ff7201038f0 2024-02-28T17:06:16.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff720119d00 con 0x7ff7201038f0 2024-02-28T17:06:16.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff717400700 1 -- 172.21.15.1:0/4250708477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff70801c630 con 0x7ff7201038f0 2024-02-28T17:06:16.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.344+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72011a1f0 con 0x7ff7201038f0 2024-02-28T17:06:16.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.345+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7200fc6e0 con 0x7ff7201038f0 2024-02-28T17:06:16.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.346+0000 7ff717400700 1 -- 172.21.15.1:0/4250708477 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff70800bd60 con 0x7ff7201038f0 2024-02-28T17:06:16.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.347+0000 7ff717400700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff70406d7f0 0x7ff70406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:16.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.347+0000 7ff717400700 1 -- 172.21.15.1:0/4250708477 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff70808e1f0 con 0x7ff7201038f0 2024-02-28T17:06:16.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.348+0000 7ff71f400700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff70406d7f0 0x7ff70406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:16.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.348+0000 7ff71f400700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff70406d7f0 0x7ff70406fca0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff710006fd0 tx=0x7ff710008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:16.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.352+0000 7ff717400700 1 -- 172.21.15.1:0/4250708477 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff708057aa0 con 0x7ff7201038f0 2024-02-28T17:06:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.584+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff720071ae0 con 0x7ff70406d7f0 2024-02-28T17:06:16.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.614+0000 7ff717400700 1 -- 172.21.15.1:0/4250708477 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff720071ae0 con 0x7ff70406d7f0 2024-02-28T17:06:16.615 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:06:16.615 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (32m) 9m ago 46m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (29m) 9m ago 46m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (28m) 9m ago 45m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (31m) 9m ago 46m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (25m) 9m ago 39m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (24m) 9m ago 39m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (39m) 9m ago 39m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (9m) 9m ago 39m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (34m) 9m ago 48m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (33m) 9m ago 45m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (30m) 9m ago 48m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (30m) 9m ago 45m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (33m) 9m ago 46m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:06:16.616 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (33m) 9m ago 45m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:06:16.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (28m) 9m ago 44m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:06:16.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (27m) 9m ago 43m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:06:16.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (27m) 9m ago 43m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:06:16.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (26m) 9m ago 42m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:06:16.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (26m) 9m ago 41m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:06:16.617 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (25m) 9m ago 41m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:06:16.617 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (32m) 9m ago 46m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.616+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff70406d7f0 msgr2=0x7ff70406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.616+0000 7ff726800700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff70406d7f0 0x7ff70406fca0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff710006fd0 tx=0x7ff710008040 comp rx=0 tx=0).stop 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 msgr2=0x7ff7200707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 0x7ff7200707d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff708009030 tx=0x7ff708009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 shutdown_connections 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff70406d7f0 0x7ff70406fca0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff720102f50 0x7ff720070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 --2- 172.21.15.1:0/4250708477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7201038f0 0x7ff7200707d0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 >> 172.21.15.1:0/4250708477 conn(0x7ff7200f8ae0 msgr2=0x7ff7200fa480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 shutdown_connections 2024-02-28T17:06:16.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.617+0000 7ff726800700 1 -- 172.21.15.1:0/4250708477 wait complete. 2024-02-28T17:06:16.666 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:16 smithi001 ceph-mon[112940]: pgmap v955: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:16.667 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:16 smithi001 ceph-mon[112940]: from='client.35710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:16.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:16 smithi066 ceph-mon[85781]: pgmap v955: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:16.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:16 smithi066 ceph-mon[85781]: from='client.35710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:16.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.761+0000 7f51d0800700 1 -- 172.21.15.1:0/3456301951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 msgr2=0x7f51c8109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:16.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.761+0000 7f51d0800700 1 --2- 172.21.15.1:0/3456301951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 0x7f51c8109260 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f51b8004970 tx=0x7f51b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:16.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.761+0000 7f51d0800700 1 -- 172.21.15.1:0/3456301951 shutdown_connections 2024-02-28T17:06:16.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.761+0000 7f51d0800700 1 --2- 172.21.15.1:0/3456301951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 0x7f51c8109260 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.761+0000 7f51d0800700 1 --2- 172.21.15.1:0/3456301951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51c8070330 0x7f51c8070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.761+0000 7f51d0800700 1 -- 172.21.15.1:0/3456301951 >> 172.21.15.1:0/3456301951 conn(0x7f51c80f8ae0 msgr2=0x7f51c80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:16.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.762+0000 7f51d0800700 1 -- 172.21.15.1:0/3456301951 shutdown_connections 2024-02-28T17:06:16.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.762+0000 7f51d0800700 1 -- 172.21.15.1:0/3456301951 wait complete. 2024-02-28T17:06:16.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.762+0000 7f51d0800700 1 Processor -- start 2024-02-28T17:06:16.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51d0800700 1 -- start start 2024-02-28T17:06:16.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51d0800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51c8070330 0x7f51c8115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:16.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51d0800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 0x7f51c8116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:16.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51d0800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c8116790 con 0x7f51c8070c40 2024-02-28T17:06:16.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51d0800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c810fae0 con 0x7f51c8070330 2024-02-28T17:06:16.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51cd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 0x7f51c8116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:16.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51cd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 0x7f51c8116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55394/0 (socket says 172.21.15.1:55394) 2024-02-28T17:06:16.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51cd400700 1 -- 172.21.15.1:0/475480981 learned_addr learned my addr 172.21.15.1:0/475480981 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:16.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.763+0000 7f51cde00700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51c8070330 0x7f51c8115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:16.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.764+0000 7f51cd400700 1 -- 172.21.15.1:0/475480981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51c8070330 msgr2=0x7f51c8115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:16.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.764+0000 7f51cd400700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51c8070330 0x7f51c8115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:16.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.764+0000 7f51cd400700 1 -- 172.21.15.1:0/475480981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b8005040 con 0x7f51c8070c40 2024-02-28T17:06:16.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.764+0000 7f51cd400700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 0x7f51c8116140 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f51b8004f50 tx=0x7f51b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:16.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.765+0000 7f51c2000700 1 -- 172.21.15.1:0/475480981 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51b800e040 con 0x7f51c8070c40 2024-02-28T17:06:16.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.765+0000 7f51c2000700 1 -- 172.21.15.1:0/475480981 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51b8004290 con 0x7f51c8070c40 2024-02-28T17:06:16.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.765+0000 7f51c2000700 1 -- 172.21.15.1:0/475480981 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51b801c3f0 con 0x7f51c8070c40 2024-02-28T17:06:16.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.765+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c810fd60 con 0x7f51c8070c40 2024-02-28T17:06:16.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.765+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c81102a0 con 0x7f51c8070c40 2024-02-28T17:06:16.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.766+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c8106920 con 0x7f51c8070c40 2024-02-28T17:06:16.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.767+0000 7f51c2000700 1 -- 172.21.15.1:0/475480981 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f51b800bc10 con 0x7f51c8070c40 2024-02-28T17:06:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.768+0000 7f51c2000700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f51b006d7f0 0x7f51b006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.768+0000 7f51c2000700 1 -- 172.21.15.1:0/475480981 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f51b808e130 con 0x7f51c8070c40 2024-02-28T17:06:16.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.769+0000 7f51cde00700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f51b006d7f0 0x7f51b006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:16.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.769+0000 7f51cde00700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f51b006d7f0 0x7f51b006fca0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0041d0 tx=0x7f51bc0043c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:16.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:16.773+0000 7f51c2000700 1 -- 172.21.15.1:0/475480981 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f51b80579e0 con 0x7f51c8070c40 2024-02-28T17:06:17.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.086+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f51c804bed0 con 0x7f51c8070c40 2024-02-28T17:06:17.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.087+0000 7f51c2000700 1 -- 172.21.15.1:0/475480981 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f51b805b0a0 con 0x7f51c8070c40 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:06:17.089 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:06:17.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:06:17.090 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:06:17.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:06:17.090 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:06:17.090 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:06:17.090 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f51b006d7f0 msgr2=0x7f51b006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f51b006d7f0 0x7f51b006fca0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0041d0 tx=0x7f51bc0043c0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 msgr2=0x7f51c8116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 0x7f51c8116140 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f51b8004f50 tx=0x7f51b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 shutdown_connections 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f51b006d7f0 0x7f51b006fca0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f51c8070330 0x7f51c8115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 --2- 172.21.15.1:0/475480981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c8070c40 0x7f51c8116140 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 >> 172.21.15.1:0/475480981 conn(0x7f51c80f8ae0 msgr2=0x7f51c8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:17.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 shutdown_connections 2024-02-28T17:06:17.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.091+0000 7f51d0800700 1 -- 172.21.15.1:0/475480981 wait complete. 2024-02-28T17:06:17.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.240+0000 7f8eb9400700 1 -- 172.21.15.1:0/2402175481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb41095b0 msgr2=0x7f8eb4109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.240+0000 7f8eb9400700 1 --2- 172.21.15.1:0/2402175481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb41095b0 0x7f8eb4109980 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4004940 tx=0x7f8ea4009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:17.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.240+0000 7f8eb9400700 1 -- 172.21.15.1:0/2402175481 shutdown_connections 2024-02-28T17:06:17.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.240+0000 7f8eb9400700 1 --2- 172.21.15.1:0/2402175481 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb4103dc0 0x7f8eb4104230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.240+0000 7f8eb9400700 1 --2- 172.21.15.1:0/2402175481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb41095b0 0x7f8eb4109980 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.240+0000 7f8eb9400700 1 -- 172.21.15.1:0/2402175481 >> 172.21.15.1:0/2402175481 conn(0x7f8eb4072d70 msgr2=0x7f8eb4075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.241+0000 7f8eb9400700 1 -- 172.21.15.1:0/2402175481 shutdown_connections 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.241+0000 7f8eb9400700 1 -- 172.21.15.1:0/2402175481 wait complete. 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.241+0000 7f8eb9400700 1 Processor -- start 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.242+0000 7f8eb9400700 1 -- start start 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.242+0000 7f8eb9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb4103dc0 0x7f8eb411a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.242+0000 7f8eb9400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb41095b0 0x7f8eb411a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.242+0000 7f8eb9400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb411acb0 con 0x7f8eb4103dc0 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.242+0000 7f8eb9400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb4113f30 con 0x7f8eb41095b0 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.242+0000 7f8eb2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb4103dc0 0x7f8eb411a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.242+0000 7f8eb2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb4103dc0 0x7f8eb411a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55414/0 (socket says 172.21.15.1:55414) 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.242+0000 7f8eb2800700 1 -- 172.21.15.1:0/3771801606 learned_addr learned my addr 172.21.15.1:0/3771801606 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.243+0000 7f8eb1e00700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb41095b0 0x7f8eb411a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.243+0000 7f8eb2800700 1 -- 172.21.15.1:0/3771801606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb41095b0 msgr2=0x7f8eb411a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.243+0000 7f8eb2800700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb41095b0 0x7f8eb411a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.243+0000 7f8eb2800700 1 -- 172.21.15.1:0/3771801606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4005040 con 0x7f8eb4103dc0 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.243+0000 7f8eb2800700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb4103dc0 0x7f8eb411a090 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009030 tx=0x7f8ea4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.243+0000 7f8eaaa00700 1 -- 172.21.15.1:0/3771801606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea400e040 con 0x7f8eb4103dc0 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.244+0000 7f8eaaa00700 1 -- 172.21.15.1:0/3771801606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ea400bbf0 con 0x7f8eb4103dc0 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.244+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb4114150 con 0x7f8eb4103dc0 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.244+0000 7f8eaaa00700 1 -- 172.21.15.1:0/3771801606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea401c630 con 0x7f8eb4103dc0 2024-02-28T17:06:17.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.244+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb4114690 con 0x7f8eb4103dc0 2024-02-28T17:06:17.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.245+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb4075890 con 0x7f8eb4103dc0 2024-02-28T17:06:17.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.246+0000 7f8eaaa00700 1 -- 172.21.15.1:0/3771801606 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8ea400bd60 con 0x7f8eb4103dc0 2024-02-28T17:06:17.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.247+0000 7f8eaaa00700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e9806d7f0 0x7f8e9806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:17.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.247+0000 7f8eaaa00700 1 -- 172.21.15.1:0/3771801606 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8ea408e130 con 0x7f8eb4103dc0 2024-02-28T17:06:17.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.247+0000 7f8eb1e00700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e9806d7f0 0x7f8e9806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:17.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.248+0000 7f8eb1e00700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e9806d7f0 0x7f8e9806fca0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0006fd0 tx=0x7f8ea0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:17.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.252+0000 7f8eaaa00700 1 -- 172.21.15.1:0/3771801606 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8ea405b5d0 con 0x7f8eb4103dc0 2024-02-28T17:06:17.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.518+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8eb4115710 con 0x7f8eb4103dc0 2024-02-28T17:06:17.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.520+0000 7f8eaaa00700 1 -- 172.21.15.1:0/3771801606 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f8ea405b160 con 0x7f8eb4103dc0 2024-02-28T17:06:17.521 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:06:17.521 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:06:17.521 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:06:17.521 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:06:17.521 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:06:17.521 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:06:17.522 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:06:17.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:06:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:06:17.524 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:06:17.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:06:17.524 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:06:17.524 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:06:17.524 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:06:17.524 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:06:17.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e9806d7f0 msgr2=0x7f8e9806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e9806d7f0 0x7f8e9806fca0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0006fd0 tx=0x7f8ea0008040 comp rx=0 tx=0).stop 2024-02-28T17:06:17.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb4103dc0 msgr2=0x7f8eb411a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb4103dc0 0x7f8eb411a090 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009030 tx=0x7f8ea4009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:17.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 shutdown_connections 2024-02-28T17:06:17.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e9806d7f0 0x7f8e9806fca0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8eb4103dc0 0x7f8eb411a090 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 --2- 172.21.15.1:0/3771801606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8eb41095b0 0x7f8eb411a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 >> 172.21.15.1:0/3771801606 conn(0x7f8eb4072d70 msgr2=0x7f8eb4074680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:17.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 shutdown_connections 2024-02-28T17:06:17.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.523+0000 7f8eb9400700 1 -- 172.21.15.1:0/3771801606 wait complete. 2024-02-28T17:06:17.526 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:06:17.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.672+0000 7f2664000700 1 -- 172.21.15.1:0/3415118339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 msgr2=0x7f265c0fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.672+0000 7f2664000700 1 --2- 172.21.15.1:0/3415118339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 0x7f265c0fe320 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2650004940 tx=0x7f2650009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:17.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.672+0000 7f2664000700 1 -- 172.21.15.1:0/3415118339 shutdown_connections 2024-02-28T17:06:17.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.672+0000 7f2664000700 1 --2- 172.21.15.1:0/3415118339 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f265c0fe8f0 0x7f265c106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.672+0000 7f2664000700 1 --2- 172.21.15.1:0/3415118339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 0x7f265c0fe320 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.672+0000 7f2664000700 1 -- 172.21.15.1:0/3415118339 >> 172.21.15.1:0/3415118339 conn(0x7f265c0f7a10 msgr2=0x7f265c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:17.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.672+0000 7f2664000700 1 -- 172.21.15.1:0/3415118339 shutdown_connections 2024-02-28T17:06:17.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.672+0000 7f2664000700 1 -- 172.21.15.1:0/3415118339 wait complete. 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.673+0000 7f2664000700 1 Processor -- start 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.673+0000 7f2664000700 1 -- start start 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2664000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 0x7f265c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2664000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f265c0fe8f0 0x7f265c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2664000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c116800 con 0x7f265c0fdf50 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2664000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c10fb30 con 0x7f265c0fe8f0 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2661600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 0x7f265c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2661600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 0x7f265c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55436/0 (socket says 172.21.15.1:55436) 2024-02-28T17:06:17.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2661600700 1 -- 172.21.15.1:0/2782681820 learned_addr learned my addr 172.21.15.1:0/2782681820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:17.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2661600700 1 -- 172.21.15.1:0/2782681820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f265c0fe8f0 msgr2=0x7f265c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.674+0000 7f2661600700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f265c0fe8f0 0x7f265c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.675+0000 7f2661600700 1 -- 172.21.15.1:0/2782681820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2650005040 con 0x7f265c0fdf50 2024-02-28T17:06:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.675+0000 7f2661600700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 0x7f265c115c70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2650009180 tx=0x7f2650009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.675+0000 7f2659600700 1 -- 172.21.15.1:0/2782681820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f265000e040 con 0x7f265c0fdf50 2024-02-28T17:06:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.675+0000 7f2659600700 1 -- 172.21.15.1:0/2782681820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2650004290 con 0x7f265c0fdf50 2024-02-28T17:06:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.675+0000 7f2659600700 1 -- 172.21.15.1:0/2782681820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f265001c630 con 0x7f265c0fdf50 2024-02-28T17:06:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.675+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c10fdb0 con 0x7f265c0fdf50 2024-02-28T17:06:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.675+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c1102f0 con 0x7f265c0fdf50 2024-02-28T17:06:17.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.676+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c106a20 con 0x7f265c0fdf50 2024-02-28T17:06:17.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.677+0000 7f2659600700 1 -- 172.21.15.1:0/2782681820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f26500139f0 con 0x7f265c0fdf50 2024-02-28T17:06:17.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.678+0000 7f2659600700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f264406d7f0 0x7f264406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:17.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.678+0000 7f2659600700 1 -- 172.21.15.1:0/2782681820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f265008e2f0 con 0x7f265c0fdf50 2024-02-28T17:06:17.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.679+0000 7f2660c00700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f264406d7f0 0x7f264406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:17.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.679+0000 7f2660c00700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f264406d7f0 0x7f264406fca0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f265c1114a0 tx=0x7f2648007d40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:17.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.683+0000 7f2659600700 1 -- 172.21.15.1:0/2782681820 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f265005b6d0 con 0x7f265c0fdf50 2024-02-28T17:06:17.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:17 smithi001 ceph-mon[112940]: from='client.35714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:17.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:17 smithi001 ceph-mon[112940]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:17.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:17 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/475480981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:06:17.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.926+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f265c04bf40 con 0x7f264406d7f0 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.928+0000 7f2659600700 1 -- 172.21.15.1:0/2782681820 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f265c04bf40 con 0x7f264406d7f0 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:06:17.929 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:06:17.930 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:06:17.930 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:06:17.930 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:06:17.930 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:06:17.930 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:06:17.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f264406d7f0 msgr2=0x7f264406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f264406d7f0 0x7f264406fca0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f265c1114a0 tx=0x7f2648007d40 comp rx=0 tx=0).stop 2024-02-28T17:06:17.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 msgr2=0x7f265c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:17.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 0x7f265c115c70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2650009180 tx=0x7f2650009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:17.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 shutdown_connections 2024-02-28T17:06:17.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f264406d7f0 0x7f264406fca0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f265c0fdf50 0x7f265c115c70 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 --2- 172.21.15.1:0/2782681820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f265c0fe8f0 0x7f265c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 >> 172.21.15.1:0/2782681820 conn(0x7f265c0f7a10 msgr2=0x7f265c103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 shutdown_connections 2024-02-28T17:06:17.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:17.931+0000 7f2664000700 1 -- 172.21.15.1:0/2782681820 wait complete. 2024-02-28T17:06:18.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:17 smithi066 ceph-mon[85781]: from='client.35714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:18.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:17 smithi066 ceph-mon[85781]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:18.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:17 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/475480981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:06:18.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.086+0000 7f6dd3600700 1 -- 172.21.15.1:0/2643224106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 msgr2=0x7f6dcc0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:18.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.086+0000 7f6dd3600700 1 --2- 172.21.15.1:0/2643224106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 0x7f6dcc0fe3b0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0004970 tx=0x7f6dc0009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:18.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.086+0000 7f6dd3600700 1 -- 172.21.15.1:0/2643224106 shutdown_connections 2024-02-28T17:06:18.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.086+0000 7f6dd3600700 1 --2- 172.21.15.1:0/2643224106 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dcc0fe8f0 0x7f6dcc107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:18.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.086+0000 7f6dd3600700 1 --2- 172.21.15.1:0/2643224106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 0x7f6dcc0fe3b0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:18.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.086+0000 7f6dd3600700 1 -- 172.21.15.1:0/2643224106 >> 172.21.15.1:0/2643224106 conn(0x7f6dcc071d90 msgr2=0x7f6dcc072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:18.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.086+0000 7f6dd3600700 1 -- 172.21.15.1:0/2643224106 shutdown_connections 2024-02-28T17:06:18.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.086+0000 7f6dd3600700 1 -- 172.21.15.1:0/2643224106 wait complete. 2024-02-28T17:06:18.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.087+0000 7f6dd3600700 1 Processor -- start 2024-02-28T17:06:18.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.087+0000 7f6dd3600700 1 -- start start 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd3600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 0x7f6dcc115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd3600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dcc0fe8f0 0x7f6dcc116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd3600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc1167a0 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd3600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc10faf0 con 0x7f6dcc0fe8f0 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 0x7f6dcc115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 0x7f6dcc115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55460/0 (socket says 172.21.15.1:55460) 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd0c00700 1 -- 172.21.15.1:0/4109701376 learned_addr learned my addr 172.21.15.1:0/4109701376 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd0c00700 1 -- 172.21.15.1:0/4109701376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dcc0fe8f0 msgr2=0x7f6dcc116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd0c00700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dcc0fe8f0 0x7f6dcc116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.088+0000 7f6dd0c00700 1 -- 172.21.15.1:0/4109701376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc0005040 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.089+0000 7f6dd0c00700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 0x7f6dcc115c10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009150 tx=0x7f6dc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:18.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.089+0000 7f6dc8c00700 1 -- 172.21.15.1:0/4109701376 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc000e040 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.089+0000 7f6dc8c00700 1 -- 172.21.15.1:0/4109701376 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dc0004290 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.089+0000 7f6dc8c00700 1 -- 172.21.15.1:0/4109701376 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc001c630 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.089+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc10fd70 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.089+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc1102b0 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.090+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dcc106ac0 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.091+0000 7f6dc8c00700 1 -- 172.21.15.1:0/4109701376 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6dc0013980 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.093+0000 7f6dc8c00700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6db406d7f0 0x7f6db406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:18.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.093+0000 7f6dc8c00700 1 -- 172.21.15.1:0/4109701376 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6dc008e1f0 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.093+0000 7f6dcbe00700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6db406d7f0 0x7f6db406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:18.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.094+0000 7f6dcbe00700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6db406d7f0 0x7f6db406fca0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc1114b0 tx=0x7f6db8009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:18.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.097+0000 7f6dc8c00700 1 -- 172.21.15.1:0/4109701376 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6dc0057aa0 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.397+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6dcc04bed0 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.398+0000 7f6dc8c00700 1 -- 172.21.15.1:0/4109701376 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f6dc005b160 con 0x7f6dcc0fdfe0 2024-02-28T17:06:18.400 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:06:18.400 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:06:18.400 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:06:18.400 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:06:18.400 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:06:18.400 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:06:18.400 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:06:18.400 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:06:18.401 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:06:18.401 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:06:18.401 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:06:18.401 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:06:18.401 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:06:18.401 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:06:18.401 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:06:18.402 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:06:18.402 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:06:18.402 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:06:18.402 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:06:18.402 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:06:18.402 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:06:18.402 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:06:18.403 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:06:18.403 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:06:18.403 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:06:18.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6db406d7f0 msgr2=0x7f6db406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:18.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6db406d7f0 0x7f6db406fca0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc1114b0 tx=0x7f6db8009ce0 comp rx=0 tx=0).stop 2024-02-28T17:06:18.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 msgr2=0x7f6dcc115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:18.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 0x7f6dcc115c10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009150 tx=0x7f6dc0009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:18.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 shutdown_connections 2024-02-28T17:06:18.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6db406d7f0 0x7f6db406fca0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:18.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dcc0fdfe0 0x7f6dcc115c10 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:18.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 --2- 172.21.15.1:0/4109701376 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dcc0fe8f0 0x7f6dcc116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:18.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.400+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 >> 172.21.15.1:0/4109701376 conn(0x7f6dcc071d90 msgr2=0x7f6dcc103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:18.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.401+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 shutdown_connections 2024-02-28T17:06:18.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:18.401+0000 7f6dd3600700 1 -- 172.21.15.1:0/4109701376 wait complete. 2024-02-28T17:06:18.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:18 smithi001 ceph-mon[112940]: pgmap v956: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:18.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:18 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3771801606' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:06:18.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:18 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4109701376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:06:19.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:18 smithi066 ceph-mon[85781]: pgmap v956: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:19.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:18 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3771801606' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:06:19.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:18 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4109701376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:06:19.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:19 smithi001 ceph-mon[112940]: from='client.35730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:20.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:19 smithi066 ceph-mon[85781]: from='client.35730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:20.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:20 smithi001 ceph-mon[112940]: pgmap v957: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:21.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:20 smithi066 ceph-mon[85781]: pgmap v957: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:23.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:22 smithi066 ceph-mon[85781]: pgmap v958: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:23.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:22 smithi001 ceph-mon[112940]: pgmap v958: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:25.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:24 smithi066 ceph-mon[85781]: pgmap v959: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:25.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:24 smithi001 ceph-mon[112940]: pgmap v959: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:27.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:26 smithi066 ceph-mon[85781]: pgmap v960: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:27.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:06:27.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:26 smithi001 ceph-mon[112940]: pgmap v960: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:27.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:06:29.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:28 smithi066 ceph-mon[85781]: pgmap v961: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:29.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:28 smithi001 ceph-mon[112940]: pgmap v961: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:31.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:30 smithi066 ceph-mon[85781]: pgmap v962: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:31.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:30 smithi001 ceph-mon[112940]: pgmap v962: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:33.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:32 smithi066 ceph-mon[85781]: pgmap v963: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:33.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:32 smithi001 ceph-mon[112940]: pgmap v963: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:35.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:34 smithi066 ceph-mon[85781]: pgmap v964: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:35.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:34 smithi001 ceph-mon[112940]: pgmap v964: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:36.833 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:36 smithi066 ceph-mon[85781]: pgmap v965: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:37.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:36 smithi001 ceph-mon[112940]: pgmap v965: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:39.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:38 smithi066 ceph-mon[85781]: pgmap v966: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:39.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:38 smithi001 ceph-mon[112940]: pgmap v966: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:41.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:40 smithi066 ceph-mon[85781]: pgmap v967: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:41.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:40 smithi001 ceph-mon[112940]: pgmap v967: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:42.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:06:42.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:06:43.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:42 smithi066 ceph-mon[85781]: pgmap v968: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:43.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:42 smithi001 ceph-mon[112940]: pgmap v968: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:45.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:44 smithi066 ceph-mon[85781]: pgmap v969: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:45.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:44 smithi001 ceph-mon[112940]: pgmap v969: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:47.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:46 smithi066 ceph-mon[85781]: pgmap v970: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:47.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:46 smithi001 ceph-mon[112940]: pgmap v970: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:48.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.571+0000 7f7f5f000700 1 -- 172.21.15.1:0/284514754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105170 msgr2=0x7f7f58105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:48.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.571+0000 7f7f5f000700 1 --2- 172.21.15.1:0/284514754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105170 0x7f7f58105540 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004940 tx=0x7f7f48009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:48.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.571+0000 7f7f5f000700 1 -- 172.21.15.1:0/284514754 shutdown_connections 2024-02-28T17:06:48.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.571+0000 7f7f5f000700 1 --2- 172.21.15.1:0/284514754 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f58105b10 0x7f7f580ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:48.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.571+0000 7f7f5f000700 1 --2- 172.21.15.1:0/284514754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105170 0x7f7f58105540 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:48.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.571+0000 7f7f5f000700 1 -- 172.21.15.1:0/284514754 >> 172.21.15.1:0/284514754 conn(0x7f7f580face0 msgr2=0x7f7f580fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:48.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.571+0000 7f7f5f000700 1 -- 172.21.15.1:0/284514754 shutdown_connections 2024-02-28T17:06:48.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.571+0000 7f7f5f000700 1 -- 172.21.15.1:0/284514754 wait complete. 2024-02-28T17:06:48.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.572+0000 7f7f5f000700 1 Processor -- start 2024-02-28T17:06:48.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.573+0000 7f7f5f000700 1 -- start start 2024-02-28T17:06:48.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.573+0000 7f7f5f000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f58105170 0x7f7f58115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:48.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.573+0000 7f7f5f000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105b10 0x7f7f581161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:48.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.573+0000 7f7f5f000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f581167f0 con 0x7f7f58105b10 2024-02-28T17:06:48.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.573+0000 7f7f5f000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5810fb40 con 0x7f7f58105170 2024-02-28T17:06:48.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.574+0000 7f7f57400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105b10 0x7f7f581161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:48.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.574+0000 7f7f57400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105b10 0x7f7f581161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44852/0 (socket says 172.21.15.1:44852) 2024-02-28T17:06:48.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.574+0000 7f7f57400700 1 -- 172.21.15.1:0/3933714776 learned_addr learned my addr 172.21.15.1:0/3933714776 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:48.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.574+0000 7f7f57e00700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f58105170 0x7f7f58115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:48.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.574+0000 7f7f57400700 1 -- 172.21.15.1:0/3933714776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f58105170 msgr2=0x7f7f58115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:48.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.574+0000 7f7f57400700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f58105170 0x7f7f58115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:48.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.574+0000 7f7f57400700 1 -- 172.21.15.1:0/3933714776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f48005040 con 0x7f7f58105b10 2024-02-28T17:06:48.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.575+0000 7f7f57400700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105b10 0x7f7f581161a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7f44002a50 tx=0x7f7f44004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:48.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.575+0000 7f7f4fe00700 1 -- 172.21.15.1:0/3933714776 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f44007cc0 con 0x7f7f58105b10 2024-02-28T17:06:48.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.575+0000 7f7f4fe00700 1 -- 172.21.15.1:0/3933714776 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f4400f040 con 0x7f7f58105b10 2024-02-28T17:06:48.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.575+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5810fe20 con 0x7f7f58105b10 2024-02-28T17:06:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.575+0000 7f7f4fe00700 1 -- 172.21.15.1:0/3933714776 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f4400daf0 con 0x7f7f58105b10 2024-02-28T17:06:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.575+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f58110360 con 0x7f7f58105b10 2024-02-28T17:06:48.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.576+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f58102480 con 0x7f7f58105b10 2024-02-28T17:06:48.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.577+0000 7f7f4fe00700 1 -- 172.21.15.1:0/3933714776 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7f44009810 con 0x7f7f58105b10 2024-02-28T17:06:48.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.579+0000 7f7f4fe00700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f3c06d7e0 0x7f7f3c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:48.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.579+0000 7f7f4fe00700 1 -- 172.21.15.1:0/3933714776 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7f4408cc20 con 0x7f7f58105b10 2024-02-28T17:06:48.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.579+0000 7f7f57e00700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f3c06d7e0 0x7f7f3c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:48.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.580+0000 7f7f57e00700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f3c06d7e0 0x7f7f3c06fc90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004f90 tx=0x7f7f48009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:48.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.584+0000 7f7f4fe00700 1 -- 172.21.15.1:0/3933714776 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7f44056450 con 0x7f7f58105b10 2024-02-28T17:06:48.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.831+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f58111280 con 0x7f7f3c06d7e0 2024-02-28T17:06:48.833 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:48 smithi001 ceph-mon[112940]: pgmap v971: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.833+0000 7f7f4fe00700 1 -- 172.21.15.1:0/3933714776 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7f58111280 con 0x7f7f3c06d7e0 2024-02-28T17:06:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f3c06d7e0 msgr2=0x7f7f3c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f3c06d7e0 0x7f7f3c06fc90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004f90 tx=0x7f7f48009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105b10 msgr2=0x7f7f581161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105b10 0x7f7f581161a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7f44002a50 tx=0x7f7f44004600 comp rx=0 tx=0).stop 2024-02-28T17:06:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 shutdown_connections 2024-02-28T17:06:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f3c06d7e0 0x7f7f3c06fc90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f58105170 0x7f7f58115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 --2- 172.21.15.1:0/3933714776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f58105b10 0x7f7f581161a0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 >> 172.21.15.1:0/3933714776 conn(0x7f7f580face0 msgr2=0x7f7f580fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 shutdown_connections 2024-02-28T17:06:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.837+0000 7f7f5f000700 1 -- 172.21.15.1:0/3933714776 wait complete. 2024-02-28T17:06:48.856 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:06:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.999+0000 7fa9d1e00700 1 -- 172.21.15.1:0/3842994243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 msgr2=0x7fa9cc0ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.999+0000 7fa9d1e00700 1 --2- 172.21.15.1:0/3842994243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 0x7fa9cc0ff170 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8004940 tx=0x7fa9b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.999+0000 7fa9d1e00700 1 -- 172.21.15.1:0/3842994243 shutdown_connections 2024-02-28T17:06:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.999+0000 7fa9d1e00700 1 --2- 172.21.15.1:0/3842994243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 0x7fa9cc0ff170 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.999+0000 7fa9d1e00700 1 --2- 172.21.15.1:0/3842994243 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9cc105160 0x7fa9cc105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.999+0000 7fa9d1e00700 1 -- 172.21.15.1:0/3842994243 >> 172.21.15.1:0/3842994243 conn(0x7fa9cc0facb0 msgr2=0x7fa9cc0fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.999+0000 7fa9d1e00700 1 -- 172.21.15.1:0/3842994243 shutdown_connections 2024-02-28T17:06:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:48.999+0000 7fa9d1e00700 1 -- 172.21.15.1:0/3842994243 wait complete. 2024-02-28T17:06:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.000+0000 7fa9d1e00700 1 Processor -- start 2024-02-28T17:06:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.000+0000 7fa9d1e00700 1 -- start start 2024-02-28T17:06:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.000+0000 7fa9d1e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9cc105160 0x7fa9cc0724a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.000+0000 7fa9d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 0x7fa9cc0729e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.000+0000 7fa9d1e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc072f20 con 0x7fa9cc105b00 2024-02-28T17:06:49.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.000+0000 7fa9d1e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc073060 con 0x7fa9cc105160 2024-02-28T17:06:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.001+0000 7fa9ca000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 0x7fa9cc0729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.001+0000 7fa9ca000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 0x7fa9cc0729e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44870/0 (socket says 172.21.15.1:44870) 2024-02-28T17:06:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.001+0000 7fa9ca000700 1 -- 172.21.15.1:0/1282473606 learned_addr learned my addr 172.21.15.1:0/1282473606 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.001+0000 7fa9caa00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9cc105160 0x7fa9cc0724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.001+0000 7fa9ca000700 1 -- 172.21.15.1:0/1282473606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9cc105160 msgr2=0x7fa9cc0724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.001+0000 7fa9ca000700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9cc105160 0x7fa9cc0724a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.001+0000 7fa9ca000700 1 -- 172.21.15.1:0/1282473606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b8005040 con 0x7fa9cc105b00 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.001+0000 7fa9caa00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9cc105160 0x7fa9cc0724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.002+0000 7fa9ca000700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 0x7fa9cc0729e0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009030 tx=0x7fa9b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.002+0000 7fa9c2a00700 1 -- 172.21.15.1:0/1282473606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9b800e040 con 0x7fa9cc105b00 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.002+0000 7fa9c2a00700 1 -- 172.21.15.1:0/1282473606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9b800bbf0 con 0x7fa9cc105b00 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.002+0000 7fa9c2a00700 1 -- 172.21.15.1:0/1282473606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9b801c630 con 0x7fa9cc105b00 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.002+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc0732e0 con 0x7fa9cc105b00 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.002+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9cc11e5a0 con 0x7fa9cc105b00 2024-02-28T17:06:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.003+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9cc107630 con 0x7fa9cc105b00 2024-02-28T17:06:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.004+0000 7fa9c2a00700 1 -- 172.21.15.1:0/1282473606 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa9b800bd60 con 0x7fa9cc105b00 2024-02-28T17:06:49.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.005+0000 7fa9c2a00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9b006d7f0 0x7fa9b006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.005+0000 7fa9c2a00700 1 -- 172.21.15.1:0/1282473606 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa9b808e0e0 con 0x7fa9cc105b00 2024-02-28T17:06:49.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.006+0000 7fa9caa00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9b006d7f0 0x7fa9b006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.006+0000 7fa9caa00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9b006d7f0 0x7fa9b006fca0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc007fd0 tx=0x7fa9bc009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:49.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:48 smithi066 ceph-mon[85781]: pgmap v971: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:49.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.010+0000 7fa9c2a00700 1 -- 172.21.15.1:0/1282473606 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa9b8057a50 con 0x7fa9cc105b00 2024-02-28T17:06:49.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.250+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9cc04bf40 con 0x7fa9b006d7f0 2024-02-28T17:06:49.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.252+0000 7fa9c2a00700 1 -- 172.21.15.1:0/1282473606 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa9cc04bf40 con 0x7fa9b006d7f0 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.254+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9b006d7f0 msgr2=0x7fa9b006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.254+0000 7fa9d1e00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9b006d7f0 0x7fa9b006fca0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc007fd0 tx=0x7fa9bc009040 comp rx=0 tx=0).stop 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.254+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 msgr2=0x7fa9cc0729e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.254+0000 7fa9d1e00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 0x7fa9cc0729e0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009030 tx=0x7fa9b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.254+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 shutdown_connections 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.254+0000 7fa9d1e00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa9b006d7f0 0x7fa9b006fca0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.254+0000 7fa9d1e00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa9cc105160 0x7fa9cc0724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.254+0000 7fa9d1e00700 1 --2- 172.21.15.1:0/1282473606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9cc105b00 0x7fa9cc0729e0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.255+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 >> 172.21.15.1:0/1282473606 conn(0x7fa9cc0facb0 msgr2=0x7fa9cc0fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.255+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 shutdown_connections 2024-02-28T17:06:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.255+0000 7fa9d1e00700 1 -- 172.21.15.1:0/1282473606 wait complete. 2024-02-28T17:06:49.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.398+0000 7f2cb6000700 1 -- 172.21.15.1:0/3485422668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fdf40 msgr2=0x7f2cb00fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.398+0000 7f2cb6000700 1 --2- 172.21.15.1:0/3485422668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fdf40 0x7f2cb00fe310 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009ce0 tx=0x7f2ca0004d10 comp rx=0 tx=0).stop 2024-02-28T17:06:49.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.398+0000 7f2cb6000700 1 -- 172.21.15.1:0/3485422668 shutdown_connections 2024-02-28T17:06:49.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.398+0000 7f2cb6000700 1 --2- 172.21.15.1:0/3485422668 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2cb00fe8e0 0x7f2cb0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.398+0000 7f2cb6000700 1 --2- 172.21.15.1:0/3485422668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fdf40 0x7f2cb00fe310 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.399+0000 7f2cb6000700 1 -- 172.21.15.1:0/3485422668 >> 172.21.15.1:0/3485422668 conn(0x7f2cb00f79d0 msgr2=0x7f2cb00f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:49.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.399+0000 7f2cb6000700 1 -- 172.21.15.1:0/3485422668 shutdown_connections 2024-02-28T17:06:49.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.399+0000 7f2cb6000700 1 -- 172.21.15.1:0/3485422668 wait complete. 2024-02-28T17:06:49.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.399+0000 7f2cb6000700 1 Processor -- start 2024-02-28T17:06:49.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.400+0000 7f2cb6000700 1 -- start start 2024-02-28T17:06:49.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.400+0000 7f2cb6000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2cb00fdf40 0x7f2cb0113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.400+0000 7f2cb6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fe8e0 0x7f2cb0114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.400+0000 7f2cb6000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0114670 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.400+0000 7f2cb6000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb010d9a0 con 0x7f2cb00fdf40 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.400+0000 7f2ca7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fe8e0 0x7f2cb0114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.400+0000 7f2ca7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fe8e0 0x7f2cb0114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44892/0 (socket says 172.21.15.1:44892) 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.400+0000 7f2ca7e00700 1 -- 172.21.15.1:0/2137110766 learned_addr learned my addr 172.21.15.1:0/2137110766 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2caf400700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2cb00fdf40 0x7f2cb0113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2ca7e00700 1 -- 172.21.15.1:0/2137110766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2cb00fdf40 msgr2=0x7f2cb0113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2ca7e00700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2cb00fdf40 0x7f2cb0113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2ca7e00700 1 -- 172.21.15.1:0/2137110766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0009940 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2ca7e00700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fe8e0 0x7f2cb0114020 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f2c98002a50 tx=0x7f2c980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2ca7400700 1 -- 172.21.15.1:0/2137110766 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c98007cc0 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2ca7400700 1 -- 172.21.15.1:0/2137110766 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c9800f040 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2ca7400700 1 -- 172.21.15.1:0/2137110766 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c98009640 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb010dc80 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.401+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb010e1c0 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.402+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb0104530 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.403+0000 7f2ca7400700 1 -- 172.21.15.1:0/2137110766 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2c980097a0 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.404+0000 7f2ca7400700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c9406d7f0 0x7f2c9406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.404+0000 7f2ca7400700 1 -- 172.21.15.1:0/2137110766 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2c9808cc60 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.405+0000 7f2caf400700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c9406d7f0 0x7f2c9406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.405+0000 7f2caf400700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c9406d7f0 0x7f2c9406fca0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0005010 tx=0x7f2ca0004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:49.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.409+0000 7f2ca7400700 1 -- 172.21.15.1:0/2137110766 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c98059fc0 con 0x7f2cb00fe8e0 2024-02-28T17:06:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.635+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cb010a040 con 0x7f2c9406d7f0 2024-02-28T17:06:49.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.652+0000 7f2ca7400700 1 -- 172.21.15.1:0/2137110766 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2cb010a040 con 0x7f2c9406d7f0 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (32m) 9m ago 47m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (29m) 9m ago 47m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (29m) 9m ago 46m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (32m) 9m ago 46m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (25m) 9m ago 40m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (25m) 9m ago 39m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (39m) 9m ago 39m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (9m) 9m ago 40m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (35m) 9m ago 48m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (34m) 9m ago 45m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (31m) 9m ago 49m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (30m) 9m ago 45m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (34m) 9m ago 47m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:06:49.655 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (33m) 9m ago 45m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:06:49.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (29m) 9m ago 44m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:06:49.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (28m) 9m ago 44m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:06:49.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (27m) 9m ago 43m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:06:49.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (27m) 9m ago 43m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:06:49.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (26m) 9m ago 42m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:06:49.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (25m) 9m ago 41m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:06:49.656 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (33m) 9m ago 46m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:06:49.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c9406d7f0 msgr2=0x7f2c9406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c9406d7f0 0x7f2c9406fca0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0005010 tx=0x7f2ca0004d10 comp rx=0 tx=0).stop 2024-02-28T17:06:49.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fe8e0 msgr2=0x7f2cb0114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fe8e0 0x7f2cb0114020 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f2c98002a50 tx=0x7f2c980046b0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 shutdown_connections 2024-02-28T17:06:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c9406d7f0 0x7f2c9406fca0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2cb00fdf40 0x7f2cb0113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 --2- 172.21.15.1:0/2137110766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb00fe8e0 0x7f2cb0114020 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 >> 172.21.15.1:0/2137110766 conn(0x7f2cb00f79d0 msgr2=0x7f2cb01016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 shutdown_connections 2024-02-28T17:06:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.655+0000 7f2cb6000700 1 -- 172.21.15.1:0/2137110766 wait complete. 2024-02-28T17:06:49.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:49 smithi001 ceph-mon[112940]: from='client.35738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.796+0000 7f4795600700 1 -- 172.21.15.1:0/3919643410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fe8e0 msgr2=0x7f4790106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.796+0000 7f4795600700 1 --2- 172.21.15.1:0/3919643410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fe8e0 0x7f4790106fb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f477c004940 tx=0x7f477c009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.797+0000 7f4795600700 1 -- 172.21.15.1:0/3919643410 shutdown_connections 2024-02-28T17:06:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.797+0000 7f4795600700 1 --2- 172.21.15.1:0/3919643410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fe8e0 0x7f4790106fb0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.797+0000 7f4795600700 1 --2- 172.21.15.1:0/3919643410 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47900fdf40 0x7f47900fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.797+0000 7f4795600700 1 -- 172.21.15.1:0/3919643410 >> 172.21.15.1:0/3919643410 conn(0x7f47900f79e0 msgr2=0x7f47900f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.797+0000 7f4795600700 1 -- 172.21.15.1:0/3919643410 shutdown_connections 2024-02-28T17:06:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.797+0000 7f4795600700 1 -- 172.21.15.1:0/3919643410 wait complete. 2024-02-28T17:06:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.797+0000 7f4795600700 1 Processor -- start 2024-02-28T17:06:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.798+0000 7f4795600700 1 -- start start 2024-02-28T17:06:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.798+0000 7f4795600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fdf40 0x7f4790115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.798+0000 7f4795600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47900fe8e0 0x7f47901161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.798+0000 7f4795600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4790116800 con 0x7f47900fdf40 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.798+0000 7f4795600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479010fb30 con 0x7f47900fe8e0 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.798+0000 7f478ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fdf40 0x7f4790115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.798+0000 7f478ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fdf40 0x7f4790115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44900/0 (socket says 172.21.15.1:44900) 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.798+0000 7f478ea00700 1 -- 172.21.15.1:0/887880272 learned_addr learned my addr 172.21.15.1:0/887880272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f478e000700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47900fe8e0 0x7f47901161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f478ea00700 1 -- 172.21.15.1:0/887880272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47900fe8e0 msgr2=0x7f47901161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f478ea00700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47900fe8e0 0x7f47901161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f478ea00700 1 -- 172.21.15.1:0/887880272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c005040 con 0x7f47900fdf40 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f478ea00700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fdf40 0x7f4790115c70 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4780002a50 tx=0x7f47800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f4786a00700 1 -- 172.21.15.1:0/887880272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4780007cc0 con 0x7f47900fdf40 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f4786a00700 1 -- 172.21.15.1:0/887880272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f478000f040 con 0x7f47900fdf40 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f4786a00700 1 -- 172.21.15.1:0/887880272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4780009640 con 0x7f47900fdf40 2024-02-28T17:06:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479010fe10 con 0x7f47900fdf40 2024-02-28T17:06:49.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.799+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4790110350 con 0x7f47900fdf40 2024-02-28T17:06:49.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.800+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4790106a70 con 0x7f47900fdf40 2024-02-28T17:06:49.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.801+0000 7f4786a00700 1 -- 172.21.15.1:0/887880272 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f47800097a0 con 0x7f47900fdf40 2024-02-28T17:06:49.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.802+0000 7f4786a00700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f477406d7f0 0x7f477406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:49.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.802+0000 7f4786a00700 1 -- 172.21.15.1:0/887880272 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f478008cd20 con 0x7f47900fdf40 2024-02-28T17:06:49.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.803+0000 7f478e000700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f477406d7f0 0x7f477406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:49.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.803+0000 7f478e000700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f477406d7f0 0x7f477406fca0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f477c000c00 tx=0x7f477c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:49.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:49.807+0000 7f4786a00700 1 -- 172.21.15.1:0/887880272 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f478005a000 con 0x7f47900fdf40 2024-02-28T17:06:50.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:49 smithi066 ceph-mon[85781]: from='client.35738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.113+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f479004bed0 con 0x7f47900fdf40 2024-02-28T17:06:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.114+0000 7f4786a00700 1 -- 172.21.15.1:0/887880272 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f4780059b90 con 0x7f47900fdf40 2024-02-28T17:06:50.116 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:06:50.116 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:06:50.116 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:06:50.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:06:50.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:06:50.118 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:06:50.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f477406d7f0 msgr2=0x7f477406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f477406d7f0 0x7f477406fca0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f477c000c00 tx=0x7f477c009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:50.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fdf40 msgr2=0x7f4790115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fdf40 0x7f4790115c70 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4780002a50 tx=0x7f47800046b0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 shutdown_connections 2024-02-28T17:06:50.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f477406d7f0 0x7f477406fca0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47900fdf40 0x7f4790115c70 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 --2- 172.21.15.1:0/887880272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47900fe8e0 0x7f47901161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 >> 172.21.15.1:0/887880272 conn(0x7f47900f79e0 msgr2=0x7f47900fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:50.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 shutdown_connections 2024-02-28T17:06:50.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.117+0000 7f4795600700 1 -- 172.21.15.1:0/887880272 wait complete. 2024-02-28T17:06:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.266+0000 7fe882400700 1 -- 172.21.15.1:0/3866745577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 msgr2=0x7fe87c103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.266+0000 7fe882400700 1 --2- 172.21.15.1:0/3866745577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 0x7fe87c103310 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fe86c004940 tx=0x7fe86c009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.267+0000 7fe882400700 1 -- 172.21.15.1:0/3866745577 shutdown_connections 2024-02-28T17:06:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.267+0000 7fe882400700 1 --2- 172.21.15.1:0/3866745577 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe87c1038e0 0x7fe87c0fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.267+0000 7fe882400700 1 --2- 172.21.15.1:0/3866745577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 0x7fe87c103310 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.267+0000 7fe882400700 1 -- 172.21.15.1:0/3866745577 >> 172.21.15.1:0/3866745577 conn(0x7fe87c0f8af0 msgr2=0x7fe87c0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.267+0000 7fe882400700 1 -- 172.21.15.1:0/3866745577 shutdown_connections 2024-02-28T17:06:50.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.267+0000 7fe882400700 1 -- 172.21.15.1:0/3866745577 wait complete. 2024-02-28T17:06:50.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.268+0000 7fe882400700 1 Processor -- start 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.268+0000 7fe882400700 1 -- start start 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.268+0000 7fe882400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 0x7fe87c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.268+0000 7fe882400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe87c1038e0 0x7fe87c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.268+0000 7fe882400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c116800 con 0x7fe87c102f40 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.268+0000 7fe882400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c10fb30 con 0x7fe87c1038e0 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.268+0000 7fe87b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 0x7fe87c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe87b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 0x7fe87c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44916/0 (socket says 172.21.15.1:44916) 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe87b400700 1 -- 172.21.15.1:0/619524766 learned_addr learned my addr 172.21.15.1:0/619524766 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe87b400700 1 -- 172.21.15.1:0/619524766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe87c1038e0 msgr2=0x7fe87c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe87b400700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe87c1038e0 0x7fe87c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe87b400700 1 -- 172.21.15.1:0/619524766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c005040 con 0x7fe87c102f40 2024-02-28T17:06:50.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe87b400700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 0x7fe87c115c70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009fd0 tx=0x7fe86c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:50.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe873400700 1 -- 172.21.15.1:0/619524766 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe86c00e040 con 0x7fe87c102f40 2024-02-28T17:06:50.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe873400700 1 -- 172.21.15.1:0/619524766 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe86c00bbf0 con 0x7fe87c102f40 2024-02-28T17:06:50.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c10fdb0 con 0x7fe87c102f40 2024-02-28T17:06:50.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.269+0000 7fe873400700 1 -- 172.21.15.1:0/619524766 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe86c01c3f0 con 0x7fe87c102f40 2024-02-28T17:06:50.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.270+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c1102f0 con 0x7fe87c102f40 2024-02-28T17:06:50.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.271+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c0fc6f0 con 0x7fe87c102f40 2024-02-28T17:06:50.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.272+0000 7fe873400700 1 -- 172.21.15.1:0/619524766 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe86c00bd60 con 0x7fe87c102f40 2024-02-28T17:06:50.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.272+0000 7fe873400700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe86006d7f0 0x7fe86006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.273+0000 7fe873400700 1 -- 172.21.15.1:0/619524766 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe86c08e210 con 0x7fe87c102f40 2024-02-28T17:06:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.273+0000 7fe873e00700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe86006d7f0 0x7fe86006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.273+0000 7fe873e00700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe86006d7f0 0x7fe86006fca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe87c1114a0 tx=0x7fe864009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:50.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.277+0000 7fe873400700 1 -- 172.21.15.1:0/619524766 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe86c05b610 con 0x7fe87c102f40 2024-02-28T17:06:50.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.537+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe87c1111e0 con 0x7fe87c102f40 2024-02-28T17:06:50.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.538+0000 7fe873400700 1 -- 172.21.15.1:0/619524766 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fe86c05b1a0 con 0x7fe87c102f40 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:06:50.541 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:06:50.542 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:06:50.543 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:06:50.544 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:06:50.544 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:06:50.544 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:06:50.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.540+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe86006d7f0 msgr2=0x7fe86006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.540+0000 7fe882400700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe86006d7f0 0x7fe86006fca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe87c1114a0 tx=0x7fe864009ce0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.540+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 msgr2=0x7fe87c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.540+0000 7fe882400700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 0x7fe87c115c70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009fd0 tx=0x7fe86c009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.541+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 shutdown_connections 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.541+0000 7fe882400700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe86006d7f0 0x7fe86006fca0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.541+0000 7fe882400700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c102f40 0x7fe87c115c70 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.541+0000 7fe882400700 1 --2- 172.21.15.1:0/619524766 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe87c1038e0 0x7fe87c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.541+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 >> 172.21.15.1:0/619524766 conn(0x7fe87c0f8af0 msgr2=0x7fe87c0fa3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.541+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 shutdown_connections 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.541+0000 7fe882400700 1 -- 172.21.15.1:0/619524766 wait complete. 2024-02-28T17:06:50.546 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:06:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.686+0000 7faab0200700 1 -- 172.21.15.1:0/94035111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 msgr2=0x7faaa8105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.686+0000 7faab0200700 1 --2- 172.21.15.1:0/94035111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 0x7faaa8105540 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009ce0 tx=0x7faa9c004d10 comp rx=0 tx=0).stop 2024-02-28T17:06:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.687+0000 7faab0200700 1 -- 172.21.15.1:0/94035111 shutdown_connections 2024-02-28T17:06:50.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.687+0000 7faab0200700 1 --2- 172.21.15.1:0/94035111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faaa8105b10 0x7faaa80ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.687+0000 7faab0200700 1 --2- 172.21.15.1:0/94035111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 0x7faaa8105540 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.687+0000 7faab0200700 1 -- 172.21.15.1:0/94035111 >> 172.21.15.1:0/94035111 conn(0x7faaa80face0 msgr2=0x7faaa80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.687+0000 7faab0200700 1 -- 172.21.15.1:0/94035111 shutdown_connections 2024-02-28T17:06:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.687+0000 7faab0200700 1 -- 172.21.15.1:0/94035111 wait complete. 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.687+0000 7faab0200700 1 Processor -- start 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.688+0000 7faab0200700 1 -- start start 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.688+0000 7faab0200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 0x7faaa8115e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.688+0000 7faab0200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faaa8105b10 0x7faaa81163b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.688+0000 7faab0200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa8116a00 con 0x7faaa8105170 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.688+0000 7faab0200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa810fd10 con 0x7faaa8105b10 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.688+0000 7faaad800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 0x7faaa8115e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.688+0000 7faaad800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 0x7faaa8115e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44932/0 (socket says 172.21.15.1:44932) 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.688+0000 7faaad800700 1 -- 172.21.15.1:0/1265287472 learned_addr learned my addr 172.21.15.1:0/1265287472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.689+0000 7faaad800700 1 -- 172.21.15.1:0/1265287472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faaa8105b10 msgr2=0x7faaa81163b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.689+0000 7faaad800700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faaa8105b10 0x7faaa81163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.689+0000 7faaad800700 1 -- 172.21.15.1:0/1265287472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c009940 con 0x7faaa8105170 2024-02-28T17:06:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.689+0000 7faaad800700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 0x7faaa8115e70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7faa9c004b70 tx=0x7faa9c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.689+0000 7faaa5600700 1 -- 172.21.15.1:0/1265287472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa9c00e040 con 0x7faaa8105170 2024-02-28T17:06:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.689+0000 7faaa5600700 1 -- 172.21.15.1:0/1265287472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa9c004610 con 0x7faaa8105170 2024-02-28T17:06:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.689+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa810ff90 con 0x7faaa8105170 2024-02-28T17:06:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.689+0000 7faaa5600700 1 -- 172.21.15.1:0/1265287472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa9c01c810 con 0x7faaa8105170 2024-02-28T17:06:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.690+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa81104d0 con 0x7faaa8105170 2024-02-28T17:06:50.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.691+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa8102480 con 0x7faaa8105170 2024-02-28T17:06:50.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.692+0000 7faaa5600700 1 -- 172.21.15.1:0/1265287472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7faa9c012c00 con 0x7faaa8105170 2024-02-28T17:06:50.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.693+0000 7faaa5600700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa9006d7f0 0x7faa9006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:50.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.693+0000 7faaa5600700 1 -- 172.21.15.1:0/1265287472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7faa9c08eae0 con 0x7faaa8105170 2024-02-28T17:06:50.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.693+0000 7faaace00700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa9006d7f0 0x7faa9006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:50.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.694+0000 7faaace00700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa9006d7f0 0x7faa9006fca0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7faaa8111710 tx=0x7faa94007d40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:50.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.698+0000 7faaa5600700 1 -- 172.21.15.1:0/1265287472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faa9c05be60 con 0x7faaa8105170 2024-02-28T17:06:50.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.931+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faaa804bf40 con 0x7faa9006d7f0 2024-02-28T17:06:50.933 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:50 smithi001 ceph-mon[112940]: pgmap v972: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:50.933 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:50 smithi001 ceph-mon[112940]: from='client.35742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:50.933 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:50 smithi001 ceph-mon[112940]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:50.933 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:50 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/887880272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:06:50.933 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:50 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/619524766' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:06:50.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.933+0000 7faaa5600700 1 -- 172.21.15.1:0/1265287472 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faaa804bf40 con 0x7faa9006d7f0 2024-02-28T17:06:50.935 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:06:50.935 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:06:50.936 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:06:50.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.935+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa9006d7f0 msgr2=0x7faa9006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.935+0000 7faab0200700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa9006d7f0 0x7faa9006fca0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7faaa8111710 tx=0x7faa94007d40 comp rx=0 tx=0).stop 2024-02-28T17:06:50.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.935+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 msgr2=0x7faaa8115e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:50.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.935+0000 7faab0200700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 0x7faaa8115e70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7faa9c004b70 tx=0x7faa9c004d10 comp rx=0 tx=0).stop 2024-02-28T17:06:50.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.936+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 shutdown_connections 2024-02-28T17:06:50.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.936+0000 7faab0200700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faa9006d7f0 0x7faa9006fca0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.936+0000 7faab0200700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaa8105170 0x7faaa8115e70 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.936+0000 7faab0200700 1 --2- 172.21.15.1:0/1265287472 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faaa8105b10 0x7faaa81163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:50.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.936+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 >> 172.21.15.1:0/1265287472 conn(0x7faaa80face0 msgr2=0x7faaa80fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:50.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.936+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 shutdown_connections 2024-02-28T17:06:50.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:50.936+0000 7faab0200700 1 -- 172.21.15.1:0/1265287472 wait complete. 2024-02-28T17:06:51.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:50 smithi066 ceph-mon[85781]: pgmap v972: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:51.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:50 smithi066 ceph-mon[85781]: from='client.35742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:51.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:50 smithi066 ceph-mon[85781]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:51.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:50 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/887880272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:06:51.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:50 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/619524766' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:06:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.086+0000 7fc8f7e00700 1 -- 172.21.15.1:0/2384541611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 msgr2=0x7fc8f81091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.086+0000 7fc8f7e00700 1 --2- 172.21.15.1:0/2384541611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 0x7fc8f81091f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4004940 tx=0x7fc8e4009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.086+0000 7fc8f7e00700 1 -- 172.21.15.1:0/2384541611 shutdown_connections 2024-02-28T17:06:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.086+0000 7fc8f7e00700 1 --2- 172.21.15.1:0/2384541611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 0x7fc8f81091f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.086+0000 7fc8f7e00700 1 --2- 172.21.15.1:0/2384541611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8f80fbb80 0x7fc8f80fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.086+0000 7fc8f7e00700 1 -- 172.21.15.1:0/2384541611 >> 172.21.15.1:0/2384541611 conn(0x7fc8f8073ab0 msgr2=0x7fc8f8073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.087+0000 7fc8f7e00700 1 -- 172.21.15.1:0/2384541611 shutdown_connections 2024-02-28T17:06:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.087+0000 7fc8f7e00700 1 -- 172.21.15.1:0/2384541611 wait complete. 2024-02-28T17:06:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.088+0000 7fc8f7e00700 1 Processor -- start 2024-02-28T17:06:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.088+0000 7fc8f7e00700 1 -- start start 2024-02-28T17:06:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.088+0000 7fc8f7e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8f80fbb80 0x7fc8f8115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.088+0000 7fc8f7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 0x7fc8f8115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.088+0000 7fc8f7e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f81165d0 con 0x7fc8f80fc490 2024-02-28T17:06:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.088+0000 7fc8f7e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f810f900 con 0x7fc8f80fbb80 2024-02-28T17:06:51.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8f6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 0x7fc8f8115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8f6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 0x7fc8f8115f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44962/0 (socket says 172.21.15.1:44962) 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8f6000700 1 -- 172.21.15.1:0/4247467828 learned_addr learned my addr 172.21.15.1:0/4247467828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8f6a00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8f80fbb80 0x7fc8f8115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8f6000700 1 -- 172.21.15.1:0/4247467828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8f80fbb80 msgr2=0x7fc8f8115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8f6000700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8f80fbb80 0x7fc8f8115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8f6000700 1 -- 172.21.15.1:0/4247467828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4005040 con 0x7fc8f80fc490 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8f6000700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 0x7fc8f8115f80 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009030 tx=0x7fc8e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.089+0000 7fc8eea00700 1 -- 172.21.15.1:0/4247467828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e400e040 con 0x7fc8f80fc490 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.090+0000 7fc8eea00700 1 -- 172.21.15.1:0/4247467828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8e400bbf0 con 0x7fc8f80fc490 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.090+0000 7fc8eea00700 1 -- 172.21.15.1:0/4247467828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e401c630 con 0x7fc8f80fc490 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.090+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f810fb80 con 0x7fc8f80fc490 2024-02-28T17:06:51.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.090+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f8110140 con 0x7fc8f80fc490 2024-02-28T17:06:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.091+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f8106920 con 0x7fc8f80fc490 2024-02-28T17:06:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.092+0000 7fc8eea00700 1 -- 172.21.15.1:0/4247467828 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc8e400bd60 con 0x7fc8f80fc490 2024-02-28T17:06:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.092+0000 7fc8eea00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc8dc06d7f0 0x7fc8dc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:06:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.093+0000 7fc8eea00700 1 -- 172.21.15.1:0/4247467828 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc8e408e1f0 con 0x7fc8f80fc490 2024-02-28T17:06:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.093+0000 7fc8f6a00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc8dc06d7f0 0x7fc8dc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:06:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.093+0000 7fc8f6a00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc8dc06d7f0 0x7fc8dc06fca0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8004d60 tx=0x7fc8e800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:06:51.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.097+0000 7fc8eea00700 1 -- 172.21.15.1:0/4247467828 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc8e405b5d0 con 0x7fc8f80fc490 2024-02-28T17:06:51.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.403+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc8f810c060 con 0x7fc8f80fc490 2024-02-28T17:06:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.404+0000 7fc8eea00700 1 -- 172.21.15.1:0/4247467828 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fc8e405b160 con 0x7fc8f80fc490 2024-02-28T17:06:51.406 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:06:51.406 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:06:51.407 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:06:51.408 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:06:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc8dc06d7f0 msgr2=0x7fc8dc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc8dc06d7f0 0x7fc8dc06fca0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8004d60 tx=0x7fc8e800b040 comp rx=0 tx=0).stop 2024-02-28T17:06:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 msgr2=0x7fc8f8115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:06:51.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 0x7fc8f8115f80 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009030 tx=0x7fc8e4009e30 comp rx=0 tx=0).stop 2024-02-28T17:06:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 shutdown_connections 2024-02-28T17:06:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc8dc06d7f0 0x7fc8dc06fca0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc8f80fbb80 0x7fc8f8115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 --2- 172.21.15.1:0/4247467828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f80fc490 0x7fc8f8115f80 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:06:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.407+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 >> 172.21.15.1:0/4247467828 conn(0x7fc8f8073ab0 msgr2=0x7fc8f80fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:06:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.408+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 shutdown_connections 2024-02-28T17:06:51.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:06:51.408+0000 7fc8f7e00700 1 -- 172.21.15.1:0/4247467828 wait complete. 2024-02-28T17:06:51.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:51 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4247467828' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:06:52.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:51 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4247467828' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:06:53.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:52 smithi066 ceph-mon[85781]: from='client.35758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:53.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:52 smithi066 ceph-mon[85781]: pgmap v973: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:53.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:52 smithi001 ceph-mon[112940]: from='client.35758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:06:53.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:52 smithi001 ceph-mon[112940]: pgmap v973: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:55.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:54 smithi066 ceph-mon[85781]: pgmap v974: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:55.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:54 smithi001 ceph-mon[112940]: pgmap v974: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:57.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:56 smithi066 ceph-mon[85781]: pgmap v975: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:57.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:06:57.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:56 smithi001 ceph-mon[112940]: pgmap v975: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:06:59.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:06:58 smithi066 ceph-mon[85781]: pgmap v976: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:06:59.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:06:58 smithi001 ceph-mon[112940]: pgmap v976: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:01.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:00 smithi066 ceph-mon[85781]: pgmap v977: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:01.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:00 smithi001 ceph-mon[112940]: pgmap v977: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:03.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:02 smithi066 ceph-mon[85781]: pgmap v978: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:03.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:02 smithi001 ceph-mon[112940]: pgmap v978: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:05.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:04 smithi066 ceph-mon[85781]: pgmap v979: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:05.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:04 smithi001 ceph-mon[112940]: pgmap v979: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:07.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:06 smithi066 ceph-mon[85781]: pgmap v980: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:07.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:06 smithi001 ceph-mon[112940]: pgmap v980: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:09.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:08 smithi066 ceph-mon[85781]: pgmap v981: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:09.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:08 smithi001 ceph-mon[112940]: pgmap v981: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:11.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:10 smithi066 ceph-mon[85781]: pgmap v982: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:11.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:10 smithi001 ceph-mon[112940]: pgmap v982: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:12.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:07:12.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:07:13.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:12 smithi066 ceph-mon[85781]: pgmap v983: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:13.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:12 smithi001 ceph-mon[112940]: pgmap v983: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:15.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:14 smithi066 ceph-mon[85781]: pgmap v984: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:15.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:14 smithi001 ceph-mon[112940]: pgmap v984: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:17.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:16 smithi066 ceph-mon[85781]: pgmap v985: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:17.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:16 smithi001 ceph-mon[112940]: pgmap v985: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:19.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:18 smithi066 ceph-mon[85781]: pgmap v986: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:19.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:18 smithi001 ceph-mon[112940]: pgmap v986: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:21.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:20 smithi066 ceph-mon[85781]: pgmap v987: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:21.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:20 smithi001 ceph-mon[112940]: pgmap v987: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:21.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.577+0000 7f2ddaa00700 1 -- 172.21.15.1:0/969714558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 msgr2=0x7f2dd41093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:21.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.577+0000 7f2ddaa00700 1 --2- 172.21.15.1:0/969714558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 0x7f2dd41093b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0004940 tx=0x7f2dc0009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:21.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.577+0000 7f2ddaa00700 1 -- 172.21.15.1:0/969714558 shutdown_connections 2024-02-28T17:07:21.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.577+0000 7f2ddaa00700 1 --2- 172.21.15.1:0/969714558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 0x7f2dd41093b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:21.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.577+0000 7f2ddaa00700 1 --2- 172.21.15.1:0/969714558 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2dd40fbd40 0x7f2dd40fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:21.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.577+0000 7f2ddaa00700 1 -- 172.21.15.1:0/969714558 >> 172.21.15.1:0/969714558 conn(0x7f2dd40f7a10 msgr2=0x7f2dd40f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:21.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.578+0000 7f2ddaa00700 1 -- 172.21.15.1:0/969714558 shutdown_connections 2024-02-28T17:07:21.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.578+0000 7f2ddaa00700 1 -- 172.21.15.1:0/969714558 wait complete. 2024-02-28T17:07:21.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.578+0000 7f2ddaa00700 1 Processor -- start 2024-02-28T17:07:21.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.579+0000 7f2ddaa00700 1 -- start start 2024-02-28T17:07:21.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.579+0000 7f2ddaa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2dd40fbd40 0x7f2dd4115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:21.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.579+0000 7f2ddaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 0x7f2dd4116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:21.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.579+0000 7f2ddaa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd41168f0 con 0x7f2dd40fc650 2024-02-28T17:07:21.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.579+0000 7f2ddaa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd410fb70 con 0x7f2dd40fbd40 2024-02-28T17:07:21.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.579+0000 7f2dd3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 0x7f2dd4116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.579+0000 7f2dd3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 0x7f2dd4116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49566/0 (socket says 172.21.15.1:49566) 2024-02-28T17:07:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.579+0000 7f2dd3400700 1 -- 172.21.15.1:0/3175593631 learned_addr learned my addr 172.21.15.1:0/3175593631 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.580+0000 7f2dd3e00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2dd40fbd40 0x7f2dd4115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.580+0000 7f2dd3400700 1 -- 172.21.15.1:0/3175593631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2dd40fbd40 msgr2=0x7f2dd4115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.580+0000 7f2dd3400700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2dd40fbd40 0x7f2dd4115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.580+0000 7f2dd3400700 1 -- 172.21.15.1:0/3175593631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc0005040 con 0x7f2dd40fc650 2024-02-28T17:07:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.580+0000 7f2dd3400700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 0x7f2dd4116210 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009040 tx=0x7f2dc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.580+0000 7f2dcbe00700 1 -- 172.21.15.1:0/3175593631 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dc000e040 con 0x7f2dd40fc650 2024-02-28T17:07:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.581+0000 7f2dcbe00700 1 -- 172.21.15.1:0/3175593631 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dc000bb90 con 0x7f2dd40fc650 2024-02-28T17:07:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.581+0000 7f2dcbe00700 1 -- 172.21.15.1:0/3175593631 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dc001c3f0 con 0x7f2dd40fc650 2024-02-28T17:07:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.581+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd410fd90 con 0x7f2dd40fc650 2024-02-28T17:07:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.581+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd41102d0 con 0x7f2dd40fc650 2024-02-28T17:07:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.582+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd4106ae0 con 0x7f2dd40fc650 2024-02-28T17:07:21.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.583+0000 7f2dcbe00700 1 -- 172.21.15.1:0/3175593631 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2dc000bd00 con 0x7f2dd40fc650 2024-02-28T17:07:21.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.584+0000 7f2dcbe00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2db806d7f0 0x7f2db806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:21.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.584+0000 7f2dcbe00700 1 -- 172.21.15.1:0/3175593631 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2dc008e160 con 0x7f2dd40fc650 2024-02-28T17:07:21.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.584+0000 7f2dd3e00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2db806d7f0 0x7f2db806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:21.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.585+0000 7f2dd3e00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2db806d7f0 0x7f2db806fca0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4006fd0 tx=0x7f2dc4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:21.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.590+0000 7f2dcbe00700 1 -- 172.21.15.1:0/3175593631 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2dc005b560 con 0x7f2dd40fc650 2024-02-28T17:07:21.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.832+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dd41112b0 con 0x7f2db806d7f0 2024-02-28T17:07:21.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.835+0000 7f2dcbe00700 1 -- 172.21.15.1:0/3175593631 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2dd41112b0 con 0x7f2db806d7f0 2024-02-28T17:07:21.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.837+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2db806d7f0 msgr2=0x7f2db806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:21.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2db806d7f0 0x7f2db806fca0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4006fd0 tx=0x7f2dc4008040 comp rx=0 tx=0).stop 2024-02-28T17:07:21.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 msgr2=0x7f2dd4116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:21.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 0x7f2dd4116210 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009040 tx=0x7f2dc0009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 shutdown_connections 2024-02-28T17:07:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2db806d7f0 0x7f2db806fca0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2dd40fbd40 0x7f2dd4115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 --2- 172.21.15.1:0/3175593631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2dd40fc650 0x7f2dd4116210 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 >> 172.21.15.1:0/3175593631 conn(0x7f2dd40f7a10 msgr2=0x7f2dd40f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 shutdown_connections 2024-02-28T17:07:21.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.838+0000 7f2ddaa00700 1 -- 172.21.15.1:0/3175593631 wait complete. 2024-02-28T17:07:21.856 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:07:21.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.995+0000 7fc70a400700 1 -- 172.21.15.1:0/1131219966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 msgr2=0x7fc704070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:21.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.995+0000 7fc70a400700 1 --2- 172.21.15.1:0/1131219966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 0x7fc704070700 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009d10 tx=0x7fc6f4004d10 comp rx=0 tx=0).stop 2024-02-28T17:07:21.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.995+0000 7fc70a400700 1 -- 172.21.15.1:0/1131219966 shutdown_connections 2024-02-28T17:07:21.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.995+0000 7fc70a400700 1 --2- 172.21.15.1:0/1131219966 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc704070c40 0x7fc704109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:21.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.995+0000 7fc70a400700 1 --2- 172.21.15.1:0/1131219966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 0x7fc704070700 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:21.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.995+0000 7fc70a400700 1 -- 172.21.15.1:0/1131219966 >> 172.21.15.1:0/1131219966 conn(0x7fc7040f8ae0 msgr2=0x7fc7040faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:21.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.996+0000 7fc70a400700 1 -- 172.21.15.1:0/1131219966 shutdown_connections 2024-02-28T17:07:21.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.996+0000 7fc70a400700 1 -- 172.21.15.1:0/1131219966 wait complete. 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.997+0000 7fc70a400700 1 Processor -- start 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.997+0000 7fc70a400700 1 -- start start 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.997+0000 7fc70a400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 0x7fc704115eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.997+0000 7fc70a400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc704070c40 0x7fc7041163f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.997+0000 7fc70a400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc704116a40 con 0x7fc704070330 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.997+0000 7fc70a400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70410fd50 con 0x7fc704070c40 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.997+0000 7fc703400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 0x7fc704115eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.998+0000 7fc703400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 0x7fc704115eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55544/0 (socket says 172.21.15.1:55544) 2024-02-28T17:07:22.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.998+0000 7fc703400700 1 -- 172.21.15.1:0/273330037 learned_addr learned my addr 172.21.15.1:0/273330037 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:22.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.998+0000 7fc6faa00700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc704070c40 0x7fc7041163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:22.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.998+0000 7fc703400700 1 -- 172.21.15.1:0/273330037 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc704070c40 msgr2=0x7fc7041163f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.998+0000 7fc703400700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc704070c40 0x7fc7041163f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.998+0000 7fc703400700 1 -- 172.21.15.1:0/273330037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f4009940 con 0x7fc704070330 2024-02-28T17:07:22.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.998+0000 7fc6faa00700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc704070c40 0x7fc7041163f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:07:22.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.998+0000 7fc703400700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 0x7fc704115eb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4008dd0 tx=0x7fc6f4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:22.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.999+0000 7fc6fbe00700 1 -- 172.21.15.1:0/273330037 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6f400e040 con 0x7fc704070330 2024-02-28T17:07:22.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.999+0000 7fc6fbe00700 1 -- 172.21.15.1:0/273330037 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6f40128e0 con 0x7fc704070330 2024-02-28T17:07:22.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.999+0000 7fc6fbe00700 1 -- 172.21.15.1:0/273330037 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6f401c8f0 con 0x7fc704070330 2024-02-28T17:07:22.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.999+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70410ffd0 con 0x7fc704070330 2024-02-28T17:07:22.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:21.999+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc704110510 con 0x7fc704070330 2024-02-28T17:07:22.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.000+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc704106920 con 0x7fc704070330 2024-02-28T17:07:22.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.001+0000 7fc6fbe00700 1 -- 172.21.15.1:0/273330037 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc6f4012a50 con 0x7fc704070330 2024-02-28T17:07:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.002+0000 7fc6fbe00700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6e806d7f0 0x7fc6e806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.002+0000 7fc6fbe00700 1 -- 172.21.15.1:0/273330037 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc6f408eb10 con 0x7fc704070330 2024-02-28T17:07:22.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.002+0000 7fc6faa00700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6e806d7f0 0x7fc6e806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:22.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.003+0000 7fc6faa00700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6e806d7f0 0x7fc6e806fca0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec007fd0 tx=0x7fc6ec009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:22.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.007+0000 7fc6fbe00700 1 -- 172.21.15.1:0/273330037 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc6f405bf10 con 0x7fc704070330 2024-02-28T17:07:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.245+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc70404bf40 con 0x7fc6e806d7f0 2024-02-28T17:07:22.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.249+0000 7fc6fbe00700 1 -- 172.21.15.1:0/273330037 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc70404bf40 con 0x7fc6e806d7f0 2024-02-28T17:07:22.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6e806d7f0 msgr2=0x7fc6e806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6e806d7f0 0x7fc6e806fca0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec007fd0 tx=0x7fc6ec009040 comp rx=0 tx=0).stop 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 msgr2=0x7fc704115eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 0x7fc704115eb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4008dd0 tx=0x7fc6f4004d10 comp rx=0 tx=0).stop 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 shutdown_connections 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6e806d7f0 0x7fc6e806fca0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc704070330 0x7fc704115eb0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 --2- 172.21.15.1:0/273330037 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc704070c40 0x7fc7041163f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 >> 172.21.15.1:0/273330037 conn(0x7fc7040f8ae0 msgr2=0x7fc704103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 shutdown_connections 2024-02-28T17:07:22.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.252+0000 7fc70a400700 1 -- 172.21.15.1:0/273330037 wait complete. 2024-02-28T17:07:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.407+0000 7f8451c00700 1 -- 172.21.15.1:0/4047748454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fc3e0 msgr2=0x7f844c109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.407+0000 7f8451c00700 1 --2- 172.21.15.1:0/4047748454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fc3e0 0x7f844c109140 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8438004940 tx=0x7f8438009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.408+0000 7f8451c00700 1 -- 172.21.15.1:0/4047748454 shutdown_connections 2024-02-28T17:07:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.408+0000 7f8451c00700 1 --2- 172.21.15.1:0/4047748454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fc3e0 0x7f844c109140 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.408+0000 7f8451c00700 1 --2- 172.21.15.1:0/4047748454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f844c0fbad0 0x7f844c0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.408+0000 7f8451c00700 1 -- 172.21.15.1:0/4047748454 >> 172.21.15.1:0/4047748454 conn(0x7f844c0f77e0 msgr2=0x7f844c0f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.408+0000 7f8451c00700 1 -- 172.21.15.1:0/4047748454 shutdown_connections 2024-02-28T17:07:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.408+0000 7f8451c00700 1 -- 172.21.15.1:0/4047748454 wait complete. 2024-02-28T17:07:22.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.409+0000 7f8451c00700 1 Processor -- start 2024-02-28T17:07:22.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.409+0000 7f8451c00700 1 -- start start 2024-02-28T17:07:22.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.409+0000 7f8451c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fbad0 0x7f844c1159c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.409+0000 7f8451c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f844c0fc3e0 0x7f844c115f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.409+0000 7f8451c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c1165e0 con 0x7f844c0fbad0 2024-02-28T17:07:22.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.409+0000 7f8451c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c10f880 con 0x7f844c0fc3e0 2024-02-28T17:07:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.409+0000 7f844be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fbad0 0x7f844c1159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.410+0000 7f844be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fbad0 0x7f844c1159c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55560/0 (socket says 172.21.15.1:55560) 2024-02-28T17:07:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.410+0000 7f844be00700 1 -- 172.21.15.1:0/3708052175 learned_addr learned my addr 172.21.15.1:0/3708052175 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.410+0000 7f844be00700 1 -- 172.21.15.1:0/3708052175 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f844c0fc3e0 msgr2=0x7f844c115f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.410+0000 7f844be00700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f844c0fc3e0 0x7f844c115f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.410+0000 7f844be00700 1 -- 172.21.15.1:0/3708052175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438005040 con 0x7f844c0fbad0 2024-02-28T17:07:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.410+0000 7f844be00700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fbad0 0x7f844c1159c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f843c00b790 tx=0x7f843c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.410+0000 7f8443e00700 1 -- 172.21.15.1:0/3708052175 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f843c00f770 con 0x7f844c0fbad0 2024-02-28T17:07:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.411+0000 7f8443e00700 1 -- 172.21.15.1:0/3708052175 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f843c00fda0 con 0x7f844c0fbad0 2024-02-28T17:07:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.411+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c10fb60 con 0x7f844c0fbad0 2024-02-28T17:07:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.411+0000 7f8443e00700 1 -- 172.21.15.1:0/3708052175 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f843c00d310 con 0x7f844c0fbad0 2024-02-28T17:07:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.411+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c110120 con 0x7f844c0fbad0 2024-02-28T17:07:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.412+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c106870 con 0x7f844c0fbad0 2024-02-28T17:07:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.413+0000 7f8443e00700 1 -- 172.21.15.1:0/3708052175 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f843c010420 con 0x7f844c0fbad0 2024-02-28T17:07:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.414+0000 7f8443e00700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f843006d7f0 0x7f843006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.414+0000 7f8443e00700 1 -- 172.21.15.1:0/3708052175 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f843c08cdf0 con 0x7f844c0fbad0 2024-02-28T17:07:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.414+0000 7f844b400700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f843006d7f0 0x7f843006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:22.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.415+0000 7f844b400700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f843006d7f0 0x7f843006fca0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8438000c00 tx=0x7f8438009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:22.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.419+0000 7f8443e00700 1 -- 172.21.15.1:0/3708052175 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f843c05a290 con 0x7f844c0fbad0 2024-02-28T17:07:22.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.651+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f844c04bed0 con 0x7f843006d7f0 2024-02-28T17:07:22.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.681+0000 7f8443e00700 1 -- 172.21.15.1:0/3708052175 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f844c04bed0 con 0x7f843006d7f0 2024-02-28T17:07:22.683 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:07:22.683 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (33m) 10m ago 47m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:07:22.683 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (30m) 10m ago 48m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:07:22.683 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (30m) 10m ago 46m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:07:22.683 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (33m) 10m ago 47m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:07:22.683 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (26m) 10m ago 40m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:07:22.683 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (25m) 10m ago 40m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (40m) 10m ago 40m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (10m) 10m ago 40m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (35m) 10m ago 49m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (34m) 10m ago 46m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (31m) 10m ago 49m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (31m) 10m ago 46m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (34m) 10m ago 47m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (34m) 10m ago 46m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (29m) 10m ago 45m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (29m) 10m ago 44m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:07:22.684 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (28m) 10m ago 44m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:07:22.685 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (28m) 10m ago 43m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:07:22.685 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (27m) 10m ago 43m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:07:22.685 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (26m) 10m ago 42m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:07:22.685 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (33m) 10m ago 47m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:07:22.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f843006d7f0 msgr2=0x7f843006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f843006d7f0 0x7f843006fca0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8438000c00 tx=0x7f8438009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fbad0 msgr2=0x7f844c1159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fbad0 0x7f844c1159c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f843c00b790 tx=0x7f843c009250 comp rx=0 tx=0).stop 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 shutdown_connections 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f843006d7f0 0x7f843006fca0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0fbad0 0x7f844c1159c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 --2- 172.21.15.1:0/3708052175 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f844c0fc3e0 0x7f844c115f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 >> 172.21.15.1:0/3708052175 conn(0x7f844c0f77e0 msgr2=0x7f844c0f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 shutdown_connections 2024-02-28T17:07:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.684+0000 7f8451c00700 1 -- 172.21.15.1:0/3708052175 wait complete. 2024-02-28T17:07:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.843+0000 7f8608a00700 1 -- 172.21.15.1:0/1651843423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 msgr2=0x7f8604106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.843+0000 7f8608a00700 1 --2- 172.21.15.1:0/1651843423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 0x7f8604106d70 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f85ec004940 tx=0x7f85ec009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.843+0000 7f8608a00700 1 -- 172.21.15.1:0/1651843423 shutdown_connections 2024-02-28T17:07:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.843+0000 7f8608a00700 1 --2- 172.21.15.1:0/1651843423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 0x7f8604106d70 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.843+0000 7f8608a00700 1 --2- 172.21.15.1:0/1651843423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86040fdd90 0x7f86040fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.843+0000 7f8608a00700 1 -- 172.21.15.1:0/1651843423 >> 172.21.15.1:0/1651843423 conn(0x7f86040f7840 msgr2=0x7f86040f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.843+0000 7f8608a00700 1 -- 172.21.15.1:0/1651843423 shutdown_connections 2024-02-28T17:07:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.843+0000 7f8608a00700 1 -- 172.21.15.1:0/1651843423 wait complete. 2024-02-28T17:07:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.844+0000 7f8608a00700 1 Processor -- start 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.844+0000 7f8608a00700 1 -- start start 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.844+0000 7f8608a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86040fdd90 0x7f8604115a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.844+0000 7f8608a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 0x7f8604115fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f8608a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86041165f0 con 0x7f86040fe730 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f8608a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860410f920 con 0x7f86040fdd90 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f85fbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 0x7f8604115fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f85fbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 0x7f8604115fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55586/0 (socket says 172.21.15.1:55586) 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f85fbe00700 1 -- 172.21.15.1:0/3434235173 learned_addr learned my addr 172.21.15.1:0/3434235173 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f8602800700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86040fdd90 0x7f8604115a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f85fbe00700 1 -- 172.21.15.1:0/3434235173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86040fdd90 msgr2=0x7f8604115a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f85fbe00700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86040fdd90 0x7f8604115a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f85fbe00700 1 -- 172.21.15.1:0/3434235173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec005040 con 0x7f86040fe730 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.845+0000 7f8602800700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86040fdd90 0x7f8604115a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:07:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.846+0000 7f85fbe00700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 0x7f8604115fa0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009180 tx=0x7f85ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:22.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.846+0000 7f85faa00700 1 -- 172.21.15.1:0/3434235173 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85ec00e040 con 0x7f86040fe730 2024-02-28T17:07:22.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.846+0000 7f85faa00700 1 -- 172.21.15.1:0/3434235173 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85ec004290 con 0x7f86040fe730 2024-02-28T17:07:22.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.846+0000 7f85faa00700 1 -- 172.21.15.1:0/3434235173 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85ec01c630 con 0x7f86040fe730 2024-02-28T17:07:22.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.846+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860410fba0 con 0x7f86040fe730 2024-02-28T17:07:22.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.846+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86041100e0 con 0x7f86040fe730 2024-02-28T17:07:22.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.847+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8604106780 con 0x7f86040fe730 2024-02-28T17:07:22.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.848+0000 7f85faa00700 1 -- 172.21.15.1:0/3434235173 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f85ec0139f0 con 0x7f86040fe730 2024-02-28T17:07:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.849+0000 7f85faa00700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85e806d7f0 0x7f85e806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.849+0000 7f85faa00700 1 -- 172.21.15.1:0/3434235173 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f85ec08e2f0 con 0x7f86040fe730 2024-02-28T17:07:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.849+0000 7f8602800700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85e806d7f0 0x7f85e806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:22.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.850+0000 7f8602800700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85e806d7f0 0x7f85e806fca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009fd0 tx=0x7f85f400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:22.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:22.854+0000 7f85faa00700 1 -- 172.21.15.1:0/3434235173 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f85ec05b6d0 con 0x7f86040fe730 2024-02-28T17:07:23.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:22 smithi001 ceph-mon[112940]: pgmap v988: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:23.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:22 smithi001 ceph-mon[112940]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.154+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f860410fd30 con 0x7f86040fe730 2024-02-28T17:07:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.155+0000 7f85faa00700 1 -- 172.21.15.1:0/3434235173 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f860410fd30 con 0x7f86040fe730 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:07:23.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:07:23.158 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:07:23.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85e806d7f0 msgr2=0x7f85e806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:23.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85e806d7f0 0x7f85e806fca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009fd0 tx=0x7f85f400b040 comp rx=0 tx=0).stop 2024-02-28T17:07:23.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 msgr2=0x7f8604115fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:23.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 0x7f8604115fa0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009180 tx=0x7f85ec009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 shutdown_connections 2024-02-28T17:07:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85e806d7f0 0x7f85e806fca0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f86040fdd90 0x7f8604115a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 --2- 172.21.15.1:0/3434235173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86040fe730 0x7f8604115fa0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 >> 172.21.15.1:0/3434235173 conn(0x7f86040f7840 msgr2=0x7f8604103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 shutdown_connections 2024-02-28T17:07:23.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.158+0000 7f8608a00700 1 -- 172.21.15.1:0/3434235173 wait complete. 2024-02-28T17:07:23.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:22 smithi066 ceph-mon[85781]: pgmap v988: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:23.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:22 smithi066 ceph-mon[85781]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.314+0000 7fcf78200700 1 -- 172.21.15.1:0/4255715483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf701027e0 msgr2=0x7fcf700fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.314+0000 7fcf78200700 1 --2- 172.21.15.1:0/4255715483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf701027e0 0x7fcf700fbea0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c004940 tx=0x7fcf5c009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:23.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.314+0000 7fcf78200700 1 -- 172.21.15.1:0/4255715483 shutdown_connections 2024-02-28T17:07:23.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.314+0000 7fcf78200700 1 --2- 172.21.15.1:0/4255715483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf701027e0 0x7fcf700fbea0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.314+0000 7fcf78200700 1 --2- 172.21.15.1:0/4255715483 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf70101e40 0x7fcf70102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.314+0000 7fcf78200700 1 -- 172.21.15.1:0/4255715483 >> 172.21.15.1:0/4255715483 conn(0x7fcf70071d90 msgr2=0x7fcf70072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:23.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.314+0000 7fcf78200700 1 -- 172.21.15.1:0/4255715483 shutdown_connections 2024-02-28T17:07:23.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.314+0000 7fcf78200700 1 -- 172.21.15.1:0/4255715483 wait complete. 2024-02-28T17:07:23.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.315+0000 7fcf78200700 1 Processor -- start 2024-02-28T17:07:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.315+0000 7fcf78200700 1 -- start start 2024-02-28T17:07:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.316+0000 7fcf78200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf70101e40 0x7fcf70115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.316+0000 7fcf78200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf701027e0 0x7fcf701161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.316+0000 7fcf78200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf701168b0 con 0x7fcf70101e40 2024-02-28T17:07:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.316+0000 7fcf78200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7010fb50 con 0x7fcf701027e0 2024-02-28T17:07:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.316+0000 7fcf75800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf70101e40 0x7fcf70115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.316+0000 7fcf75800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf70101e40 0x7fcf70115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55612/0 (socket says 172.21.15.1:55612) 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.316+0000 7fcf75800700 1 -- 172.21.15.1:0/887703602 learned_addr learned my addr 172.21.15.1:0/887703602 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf75800700 1 -- 172.21.15.1:0/887703602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf701027e0 msgr2=0x7fcf701161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf74e00700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf701027e0 0x7fcf701161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf75800700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf701027e0 0x7fcf701161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf75800700 1 -- 172.21.15.1:0/887703602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c005040 con 0x7fcf70101e40 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf75800700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf70101e40 0x7fcf70115c90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400b790 tx=0x7fcf64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf6d600700 1 -- 172.21.15.1:0/887703602 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf6400f770 con 0x7fcf70101e40 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf6d600700 1 -- 172.21.15.1:0/887703602 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf6400fda0 con 0x7fcf70101e40 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf6d600700 1 -- 172.21.15.1:0/887703602 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf6400d310 con 0x7fcf70101e40 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.317+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7010fe30 con 0x7fcf70101e40 2024-02-28T17:07:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.318+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf70110370 con 0x7fcf70101e40 2024-02-28T17:07:23.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.319+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf700738b0 con 0x7fcf70101e40 2024-02-28T17:07:23.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.320+0000 7fcf6d600700 1 -- 172.21.15.1:0/887703602 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcf64010420 con 0x7fcf70101e40 2024-02-28T17:07:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.321+0000 7fcf6d600700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf5806d7f0 0x7fcf5806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.321+0000 7fcf6d600700 1 -- 172.21.15.1:0/887703602 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcf6408ce30 con 0x7fcf70101e40 2024-02-28T17:07:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.321+0000 7fcf74e00700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf5806d7f0 0x7fcf5806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.322+0000 7fcf74e00700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf5806d7f0 0x7fcf5806fca0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fcf701114c0 tx=0x7fcf5c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:23.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.328+0000 7fcf6d600700 1 -- 172.21.15.1:0/887703602 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf6405a2d0 con 0x7fcf70101e40 2024-02-28T17:07:23.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.598+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcf701112a0 con 0x7fcf70101e40 2024-02-28T17:07:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.598+0000 7fcf6d600700 1 -- 172.21.15.1:0/887703602 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fcf64059e60 con 0x7fcf70101e40 2024-02-28T17:07:23.600 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:07:23.600 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:07:23.600 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:07:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:07:23.602 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:07:23.603 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:07:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf5806d7f0 msgr2=0x7fcf5806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf5806d7f0 0x7fcf5806fca0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fcf701114c0 tx=0x7fcf5c009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf70101e40 msgr2=0x7fcf70115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf70101e40 0x7fcf70115c90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400b790 tx=0x7fcf64009250 comp rx=0 tx=0).stop 2024-02-28T17:07:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 shutdown_connections 2024-02-28T17:07:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf5806d7f0 0x7fcf5806fca0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf70101e40 0x7fcf70115c90 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 --2- 172.21.15.1:0/887703602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf701027e0 0x7fcf701161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.601+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 >> 172.21.15.1:0/887703602 conn(0x7fcf70071d90 msgr2=0x7fcf700656e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:23.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.602+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 shutdown_connections 2024-02-28T17:07:23.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.602+0000 7fcf78200700 1 -- 172.21.15.1:0/887703602 wait complete. 2024-02-28T17:07:23.605 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:07:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.761+0000 7fc03e800700 1 -- 172.21.15.1:0/1745167383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 msgr2=0x7fc0381091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.761+0000 7fc03e800700 1 --2- 172.21.15.1:0/1745167383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 0x7fc0381091c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc024004940 tx=0x7fc024009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.761+0000 7fc03e800700 1 -- 172.21.15.1:0/1745167383 shutdown_connections 2024-02-28T17:07:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.761+0000 7fc03e800700 1 --2- 172.21.15.1:0/1745167383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 0x7fc0381091c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.761+0000 7fc03e800700 1 --2- 172.21.15.1:0/1745167383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0380fbb50 0x7fc0380fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.761+0000 7fc03e800700 1 -- 172.21.15.1:0/1745167383 >> 172.21.15.1:0/1745167383 conn(0x7fc0380f77c0 msgr2=0x7fc0380f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:23.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.762+0000 7fc03e800700 1 -- 172.21.15.1:0/1745167383 shutdown_connections 2024-02-28T17:07:23.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.762+0000 7fc03e800700 1 -- 172.21.15.1:0/1745167383 wait complete. 2024-02-28T17:07:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.762+0000 7fc03e800700 1 Processor -- start 2024-02-28T17:07:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.763+0000 7fc03e800700 1 -- start start 2024-02-28T17:07:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.763+0000 7fc03e800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0380fbb50 0x7fc038115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.763+0000 7fc03e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 0x7fc038115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.763+0000 7fc03e800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0381165e0 con 0x7fc0380fc460 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.763+0000 7fc03e800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03810f910 con 0x7fc0380fbb50 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.763+0000 7fc03ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 0x7fc038115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.763+0000 7fc03ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 0x7fc038115f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55636/0 (socket says 172.21.15.1:55636) 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.763+0000 7fc03ca00700 1 -- 172.21.15.1:0/2490919401 learned_addr learned my addr 172.21.15.1:0/2490919401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.764+0000 7fc03d400700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0380fbb50 0x7fc038115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.764+0000 7fc03ca00700 1 -- 172.21.15.1:0/2490919401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0380fbb50 msgr2=0x7fc038115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.764+0000 7fc03ca00700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0380fbb50 0x7fc038115a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.764+0000 7fc03ca00700 1 -- 172.21.15.1:0/2490919401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024005040 con 0x7fc0380fc460 2024-02-28T17:07:23.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.764+0000 7fc03ca00700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 0x7fc038115f90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc024000c00 tx=0x7fc024009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.766+0000 7fc035600700 1 -- 172.21.15.1:0/2490919401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc02400e040 con 0x7fc0380fc460 2024-02-28T17:07:23.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.766+0000 7fc035600700 1 -- 172.21.15.1:0/2490919401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc02400bb90 con 0x7fc0380fc460 2024-02-28T17:07:23.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.766+0000 7fc035600700 1 -- 172.21.15.1:0/2490919401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc02401c630 con 0x7fc0380fc460 2024-02-28T17:07:23.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.766+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03810fb90 con 0x7fc0380fc460 2024-02-28T17:07:23.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.766+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc038110150 con 0x7fc0380fc460 2024-02-28T17:07:23.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.768+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0381068f0 con 0x7fc0380fc460 2024-02-28T17:07:23.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.768+0000 7fc035600700 1 -- 172.21.15.1:0/2490919401 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc02400bd00 con 0x7fc0380fc460 2024-02-28T17:07:23.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.769+0000 7fc035600700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc02006d7f0 0x7fc02006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:23.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.770+0000 7fc035600700 1 -- 172.21.15.1:0/2490919401 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc02408e230 con 0x7fc0380fc460 2024-02-28T17:07:23.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.770+0000 7fc03d400700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc02006d7f0 0x7fc02006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:23.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.771+0000 7fc03d400700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc02006d7f0 0x7fc02006fca0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc02c004990 tx=0x7fc02c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:23.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:23.774+0000 7fc035600700 1 -- 172.21.15.1:0/2490919401 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc02405b590 con 0x7fc0380fc460 2024-02-28T17:07:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.018+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc03810c010 con 0x7fc02006d7f0 2024-02-28T17:07:24.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:23 smithi001 ceph-mon[112940]: from='client.35770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:24.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:23 smithi001 ceph-mon[112940]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:23 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3434235173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:07:24.021 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:23 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/887703602' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:07:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.020+0000 7fc035600700 1 -- 172.21.15.1:0/2490919401 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc03810c010 con 0x7fc02006d7f0 2024-02-28T17:07:24.021 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:07:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:07:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:07:24.022 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:07:24.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.022+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc02006d7f0 msgr2=0x7fc02006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:24.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.022+0000 7fc03e800700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc02006d7f0 0x7fc02006fca0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc02c004990 tx=0x7fc02c009ee0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.022+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 msgr2=0x7fc038115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.022+0000 7fc03e800700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 0x7fc038115f90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc024000c00 tx=0x7fc024009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.023+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 shutdown_connections 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.023+0000 7fc03e800700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc02006d7f0 0x7fc02006fca0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.023+0000 7fc03e800700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc0380fbb50 0x7fc038115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.023+0000 7fc03e800700 1 --2- 172.21.15.1:0/2490919401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0380fc460 0x7fc038115f90 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.023+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 >> 172.21.15.1:0/2490919401 conn(0x7fc0380f77c0 msgr2=0x7fc0380f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.023+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 shutdown_connections 2024-02-28T17:07:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.023+0000 7fc03e800700 1 -- 172.21.15.1:0/2490919401 wait complete. 2024-02-28T17:07:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.182+0000 7ff739200700 1 -- 172.21.15.1:0/3795641181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 msgr2=0x7ff734104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.182+0000 7ff739200700 1 --2- 172.21.15.1:0/3795641181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 0x7ff734104210 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff71c004940 tx=0x7ff71c009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.183+0000 7ff739200700 1 -- 172.21.15.1:0/3795641181 shutdown_connections 2024-02-28T17:07:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.183+0000 7ff739200700 1 --2- 172.21.15.1:0/3795641181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 0x7ff734104210 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.183+0000 7ff739200700 1 --2- 172.21.15.1:0/3795641181 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7341095f0 0x7ff7341099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.183+0000 7ff739200700 1 -- 172.21.15.1:0/3795641181 >> 172.21.15.1:0/3795641181 conn(0x7ff734072d50 msgr2=0x7ff734075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:24.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.183+0000 7ff739200700 1 -- 172.21.15.1:0/3795641181 shutdown_connections 2024-02-28T17:07:24.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.183+0000 7ff739200700 1 -- 172.21.15.1:0/3795641181 wait complete. 2024-02-28T17:07:24.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.184+0000 7ff739200700 1 Processor -- start 2024-02-28T17:07:24.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.184+0000 7ff739200700 1 -- start start 2024-02-28T17:07:24.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.184+0000 7ff739200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 0x7ff73411a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:24.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.184+0000 7ff739200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7341095f0 0x7ff73411a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:24.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.184+0000 7ff739200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73411ac50 con 0x7ff734103dc0 2024-02-28T17:07:24.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.184+0000 7ff739200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734113ed0 con 0x7ff7341095f0 2024-02-28T17:07:24.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.184+0000 7ff731e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 0x7ff73411a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:24.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.185+0000 7ff731e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 0x7ff73411a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55654/0 (socket says 172.21.15.1:55654) 2024-02-28T17:07:24.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.185+0000 7ff731e00700 1 -- 172.21.15.1:0/3186479502 learned_addr learned my addr 172.21.15.1:0/3186479502 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:24.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.185+0000 7ff72be00700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7341095f0 0x7ff73411a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:24.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.185+0000 7ff731e00700 1 -- 172.21.15.1:0/3186479502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7341095f0 msgr2=0x7ff73411a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:24.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.185+0000 7ff731e00700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7341095f0 0x7ff73411a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.185+0000 7ff731e00700 1 -- 172.21.15.1:0/3186479502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c005040 con 0x7ff734103dc0 2024-02-28T17:07:24.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.185+0000 7ff731e00700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 0x7ff73411a030 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff72400b790 tx=0x7ff724009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:24.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.186+0000 7ff72a000700 1 -- 172.21.15.1:0/3186479502 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff72400f770 con 0x7ff734103dc0 2024-02-28T17:07:24.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.186+0000 7ff72a000700 1 -- 172.21.15.1:0/3186479502 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff72400fda0 con 0x7ff734103dc0 2024-02-28T17:07:24.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.186+0000 7ff72a000700 1 -- 172.21.15.1:0/3186479502 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff72400d310 con 0x7ff734103dc0 2024-02-28T17:07:24.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.186+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff734114150 con 0x7ff734103dc0 2024-02-28T17:07:24.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.186+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff734114690 con 0x7ff734103dc0 2024-02-28T17:07:24.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.187+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff734075870 con 0x7ff734103dc0 2024-02-28T17:07:24.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.188+0000 7ff72a000700 1 -- 172.21.15.1:0/3186479502 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff724010420 con 0x7ff734103dc0 2024-02-28T17:07:24.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.189+0000 7ff72a000700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff71806d7f0 0x7ff71806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:24.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.189+0000 7ff72a000700 1 -- 172.21.15.1:0/3186479502 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff72408ceb0 con 0x7ff734103dc0 2024-02-28T17:07:24.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.189+0000 7ff72be00700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff71806d7f0 0x7ff71806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:24.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.190+0000 7ff72be00700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff71806d7f0 0x7ff71806fca0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00b3b0 tx=0x7ff71c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:24.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.194+0000 7ff72a000700 1 -- 172.21.15.1:0/3186479502 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff72405a2d0 con 0x7ff734103dc0 2024-02-28T17:07:24.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:23 smithi066 ceph-mon[85781]: from='client.35770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:24.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:23 smithi066 ceph-mon[85781]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:24.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:23 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3434235173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:07:24.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:23 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/887703602' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:07:24.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.495+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff7341156b0 con 0x7ff734103dc0 2024-02-28T17:07:24.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.496+0000 7ff72a000700 1 -- 172.21.15.1:0/3186479502 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7ff724059e60 con 0x7ff734103dc0 2024-02-28T17:07:24.498 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:07:24.498 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:07:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:07:24.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:07:24.498 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:07:24.498 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:07:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:07:24.499 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:07:24.500 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:07:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:07:24.500 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:07:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:07:24.500 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:07:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.498+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff71806d7f0 msgr2=0x7ff71806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.498+0000 7ff739200700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff71806d7f0 0x7ff71806fca0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00b3b0 tx=0x7ff71c009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 msgr2=0x7ff73411a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 0x7ff73411a030 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff72400b790 tx=0x7ff724009250 comp rx=0 tx=0).stop 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 shutdown_connections 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff71806d7f0 0x7ff71806fca0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff734103dc0 0x7ff73411a030 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 --2- 172.21.15.1:0/3186479502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff7341095f0 0x7ff73411a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 >> 172.21.15.1:0/3186479502 conn(0x7ff734072d50 msgr2=0x7ff7340746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:24.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 shutdown_connections 2024-02-28T17:07:24.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:24.499+0000 7ff739200700 1 -- 172.21.15.1:0/3186479502 wait complete. 2024-02-28T17:07:25.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:24 smithi001 ceph-mon[112940]: pgmap v989: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:25.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:24 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3186479502' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:07:25.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:24 smithi066 ceph-mon[85781]: pgmap v989: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:25.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:24 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3186479502' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:07:26.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:25 smithi066 ceph-mon[85781]: from='client.35786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:26.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:25 smithi001 ceph-mon[112940]: from='client.35786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:27.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:26 smithi066 ceph-mon[85781]: pgmap v990: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:27.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:07:27.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:26 smithi001 ceph-mon[112940]: pgmap v990: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:27.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:07:29.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:28 smithi066 ceph-mon[85781]: pgmap v991: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:29.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:28 smithi001 ceph-mon[112940]: pgmap v991: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:31.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:30 smithi066 ceph-mon[85781]: pgmap v992: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:31.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:30 smithi001 ceph-mon[112940]: pgmap v992: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:33.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:32 smithi066 ceph-mon[85781]: pgmap v993: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:33.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:32 smithi001 ceph-mon[112940]: pgmap v993: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:35.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:34 smithi066 ceph-mon[85781]: pgmap v994: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:35.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:34 smithi001 ceph-mon[112940]: pgmap v994: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:37.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:36 smithi066 ceph-mon[85781]: pgmap v995: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:37.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:36 smithi001 ceph-mon[112940]: pgmap v995: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:39.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:38 smithi066 ceph-mon[85781]: pgmap v996: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:39.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:38 smithi001 ceph-mon[112940]: pgmap v996: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:41.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:40 smithi066 ceph-mon[85781]: pgmap v997: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:41.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:07:41.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:40 smithi001 ceph-mon[112940]: pgmap v997: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:41.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:07:43.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:42 smithi066 ceph-mon[85781]: pgmap v998: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:43.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:42 smithi001 ceph-mon[112940]: pgmap v998: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:45.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:44 smithi066 ceph-mon[85781]: pgmap v999: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:45.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:44 smithi001 ceph-mon[112940]: pgmap v999: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:47.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:46 smithi066 ceph-mon[85781]: pgmap v1000: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:47.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:46 smithi001 ceph-mon[112940]: pgmap v1000: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:49.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:48 smithi066 ceph-mon[85781]: pgmap v1001: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:48 smithi001 ceph-mon[112940]: pgmap v1001: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:51.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:50 smithi066 ceph-mon[85781]: pgmap v1002: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:51.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:50 smithi001 ceph-mon[112940]: pgmap v1002: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:53.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:52 smithi066 ceph-mon[85781]: pgmap v1003: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:53.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:52 smithi001 ceph-mon[112940]: pgmap v1003: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:54.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.670+0000 7f4ac8200700 1 -- 172.21.15.1:0/2492647493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac01095f0 msgr2=0x7f4ac01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.670+0000 7f4ac8200700 1 --2- 172.21.15.1:0/2492647493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac01095f0 0x7f4ac01099c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4004940 tx=0x7f4ab4009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.670+0000 7f4ac8200700 1 -- 172.21.15.1:0/2492647493 shutdown_connections 2024-02-28T17:07:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.670+0000 7f4ac8200700 1 --2- 172.21.15.1:0/2492647493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ac0103dc0 0x7f4ac0104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.670+0000 7f4ac8200700 1 --2- 172.21.15.1:0/2492647493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac01095f0 0x7f4ac01099c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.670+0000 7f4ac8200700 1 -- 172.21.15.1:0/2492647493 >> 172.21.15.1:0/2492647493 conn(0x7f4ac0072d50 msgr2=0x7f4ac0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.670+0000 7f4ac8200700 1 -- 172.21.15.1:0/2492647493 shutdown_connections 2024-02-28T17:07:54.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.670+0000 7f4ac8200700 1 -- 172.21.15.1:0/2492647493 wait complete. 2024-02-28T17:07:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.671+0000 7f4ac8200700 1 Processor -- start 2024-02-28T17:07:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.671+0000 7f4ac8200700 1 -- start start 2024-02-28T17:07:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.671+0000 7f4ac8200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac0103dc0 0x7f4ac011a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:54.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.671+0000 7f4ac8200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ac01095f0 0x7f4ac011a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.671+0000 7f4ac8200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac011ac70 con 0x7f4ac0103dc0 2024-02-28T17:07:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.671+0000 7f4ac8200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac0113ef0 con 0x7f4ac01095f0 2024-02-28T17:07:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.672+0000 7f4ac5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac0103dc0 0x7f4ac011a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.672+0000 7f4ac5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac0103dc0 0x7f4ac011a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58942/0 (socket says 172.21.15.1:58942) 2024-02-28T17:07:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.672+0000 7f4ac5800700 1 -- 172.21.15.1:0/1718182050 learned_addr learned my addr 172.21.15.1:0/1718182050 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.672+0000 7f4ac5800700 1 -- 172.21.15.1:0/1718182050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ac01095f0 msgr2=0x7f4ac011a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.672+0000 7f4ac5800700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ac01095f0 0x7f4ac011a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:54.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.672+0000 7f4ac5800700 1 -- 172.21.15.1:0/1718182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4005040 con 0x7f4ac0103dc0 2024-02-28T17:07:54.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.672+0000 7f4ac5800700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac0103dc0 0x7f4ac011a050 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009fd0 tx=0x7f4ab4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:54.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.673+0000 7f4abd600700 1 -- 172.21.15.1:0/1718182050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ab400e040 con 0x7f4ac0103dc0 2024-02-28T17:07:54.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.673+0000 7f4abd600700 1 -- 172.21.15.1:0/1718182050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ab400bbf0 con 0x7f4ac0103dc0 2024-02-28T17:07:54.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.673+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac0114110 con 0x7f4ac0103dc0 2024-02-28T17:07:54.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.673+0000 7f4abd600700 1 -- 172.21.15.1:0/1718182050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ab401c3f0 con 0x7f4ac0103dc0 2024-02-28T17:07:54.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.673+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac0114650 con 0x7f4ac0103dc0 2024-02-28T17:07:54.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.674+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac0075870 con 0x7f4ac0103dc0 2024-02-28T17:07:54.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.675+0000 7f4abd600700 1 -- 172.21.15.1:0/1718182050 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4ab400bd60 con 0x7f4ac0103dc0 2024-02-28T17:07:54.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.676+0000 7f4abd600700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4aa806d7f0 0x7f4aa806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:54.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.676+0000 7f4abd600700 1 -- 172.21.15.1:0/1718182050 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4ab408e1c0 con 0x7f4ac0103dc0 2024-02-28T17:07:54.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.676+0000 7f4ac4e00700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4aa806d7f0 0x7f4aa806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:54.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.677+0000 7f4ac4e00700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4aa806d7f0 0x7f4aa806fca0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4aac006fd0 tx=0x7f4aac006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.681+0000 7f4abd600700 1 -- 172.21.15.1:0/1718182050 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ab405b5a0 con 0x7f4ac0103dc0 2024-02-28T17:07:54.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.939+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ac01148c0 con 0x7f4aa806d7f0 2024-02-28T17:07:54.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:54 smithi001 ceph-mon[112940]: pgmap v1004: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:54.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.942+0000 7f4abd600700 1 -- 172.21.15.1:0/1718182050 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4ac01148c0 con 0x7f4aa806d7f0 2024-02-28T17:07:54.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4aa806d7f0 msgr2=0x7f4aa806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4aa806d7f0 0x7f4aa806fca0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4aac006fd0 tx=0x7f4aac006db0 comp rx=0 tx=0).stop 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac0103dc0 msgr2=0x7f4ac011a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac0103dc0 0x7f4ac011a050 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009fd0 tx=0x7f4ab4009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 shutdown_connections 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4aa806d7f0 0x7f4aa806fca0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ac0103dc0 0x7f4ac011a050 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 --2- 172.21.15.1:0/1718182050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ac01095f0 0x7f4ac011a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 >> 172.21.15.1:0/1718182050 conn(0x7f4ac0072d50 msgr2=0x7f4ac00746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 shutdown_connections 2024-02-28T17:07:54.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:54.945+0000 7f4ac8200700 1 -- 172.21.15.1:0/1718182050 wait complete. 2024-02-28T17:07:54.962 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:07:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.103+0000 7f9296c00700 1 -- 172.21.15.1:0/265179849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 msgr2=0x7f9290070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.103+0000 7f9296c00700 1 --2- 172.21.15.1:0/265179849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 0x7f9290070ce0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9278004970 tx=0x7f9278009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.104+0000 7f9296c00700 1 -- 172.21.15.1:0/265179849 shutdown_connections 2024-02-28T17:07:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.104+0000 7f9296c00700 1 --2- 172.21.15.1:0/265179849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 0x7f9290070ce0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.104+0000 7f9296c00700 1 --2- 172.21.15.1:0/265179849 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9290071f60 0x7f9290070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.104+0000 7f9296c00700 1 -- 172.21.15.1:0/265179849 >> 172.21.15.1:0/265179849 conn(0x7f92900f8ae0 msgr2=0x7f92900faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.104+0000 7f9296c00700 1 -- 172.21.15.1:0/265179849 shutdown_connections 2024-02-28T17:07:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.104+0000 7f9296c00700 1 -- 172.21.15.1:0/265179849 wait complete. 2024-02-28T17:07:55.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.105+0000 7f9296c00700 1 Processor -- start 2024-02-28T17:07:55.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.105+0000 7f9296c00700 1 -- start start 2024-02-28T17:07:55.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.105+0000 7f9296c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 0x7f929011a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.105+0000 7f9296c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9290071f60 0x7f929011a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.105+0000 7f9296c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929011aba0 con 0x7f9290070870 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.105+0000 7f9296c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9290113eb0 con 0x7f9290071f60 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.105+0000 7f928fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 0x7f929011a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.106+0000 7f928fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 0x7f929011a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58958/0 (socket says 172.21.15.1:58958) 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.106+0000 7f928fe00700 1 -- 172.21.15.1:0/3623281739 learned_addr learned my addr 172.21.15.1:0/3623281739 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.106+0000 7f9287400700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9290071f60 0x7f929011a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.106+0000 7f928fe00700 1 -- 172.21.15.1:0/3623281739 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9290071f60 msgr2=0x7f929011a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.106+0000 7f928fe00700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9290071f60 0x7f929011a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.106+0000 7f928fe00700 1 -- 172.21.15.1:0/3623281739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9278005040 con 0x7f9290070870 2024-02-28T17:07:55.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.106+0000 7f928fe00700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 0x7f929011a010 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9280002a50 tx=0x7f92800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.107+0000 7f928cc00700 1 -- 172.21.15.1:0/3623281739 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9280007cc0 con 0x7f9290070870 2024-02-28T17:07:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.107+0000 7f928cc00700 1 -- 172.21.15.1:0/3623281739 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f928000f040 con 0x7f9290070870 2024-02-28T17:07:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.107+0000 7f928cc00700 1 -- 172.21.15.1:0/3623281739 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9280009640 con 0x7f9290070870 2024-02-28T17:07:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.107+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9290114190 con 0x7f9290070870 2024-02-28T17:07:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.107+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92901146d0 con 0x7f9290070870 2024-02-28T17:07:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.108+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92900fe400 con 0x7f9290070870 2024-02-28T17:07:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.109+0000 7f928cc00700 1 -- 172.21.15.1:0/3623281739 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f92800097a0 con 0x7f9290070870 2024-02-28T17:07:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.110+0000 7f928cc00700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f927406d7f0 0x7f927406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.110+0000 7f928cc00700 1 -- 172.21.15.1:0/3623281739 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f928008cc60 con 0x7f9290070870 2024-02-28T17:07:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.110+0000 7f9287400700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f927406d7f0 0x7f927406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.111+0000 7f9287400700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f927406d7f0 0x7f927406fca0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9278000c00 tx=0x7f9278009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.115+0000 7f928cc00700 1 -- 172.21.15.1:0/3623281739 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9280059fc0 con 0x7f9290070870 2024-02-28T17:07:55.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:54 smithi066 ceph-mon[85781]: pgmap v1004: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:55.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.357+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f929004bf40 con 0x7f927406d7f0 2024-02-28T17:07:55.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.360+0000 7f928cc00700 1 -- 172.21.15.1:0/3623281739 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f929004bf40 con 0x7f927406d7f0 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f927406d7f0 msgr2=0x7f927406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f927406d7f0 0x7f927406fca0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9278000c00 tx=0x7f9278009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 msgr2=0x7f929011a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 0x7f929011a010 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9280002a50 tx=0x7f92800046b0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 shutdown_connections 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f927406d7f0 0x7f927406fca0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9290070870 0x7f929011a010 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 --2- 172.21.15.1:0/3623281739 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9290071f60 0x7f929011a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 >> 172.21.15.1:0/3623281739 conn(0x7f92900f8ae0 msgr2=0x7f92900fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 shutdown_connections 2024-02-28T17:07:55.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.363+0000 7f9296c00700 1 -- 172.21.15.1:0/3623281739 wait complete. 2024-02-28T17:07:55.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.518+0000 7f3f54600700 1 -- 172.21.15.1:0/4144942235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fc5b0 msgr2=0x7f3f4c109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.518+0000 7f3f54600700 1 --2- 172.21.15.1:0/4144942235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fc5b0 0x7f3f4c109310 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3f38004940 tx=0x7f3f38009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:55.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.518+0000 7f3f54600700 1 -- 172.21.15.1:0/4144942235 shutdown_connections 2024-02-28T17:07:55.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.518+0000 7f3f54600700 1 --2- 172.21.15.1:0/4144942235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fc5b0 0x7f3f4c109310 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.518+0000 7f3f54600700 1 --2- 172.21.15.1:0/4144942235 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c0fbca0 0x7f3f4c0fc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.518+0000 7f3f54600700 1 -- 172.21.15.1:0/4144942235 >> 172.21.15.1:0/4144942235 conn(0x7f3f4c0f79d0 msgr2=0x7f3f4c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:55.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.519+0000 7f3f54600700 1 -- 172.21.15.1:0/4144942235 shutdown_connections 2024-02-28T17:07:55.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.519+0000 7f3f54600700 1 -- 172.21.15.1:0/4144942235 wait complete. 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.519+0000 7f3f54600700 1 Processor -- start 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.520+0000 7f3f54600700 1 -- start start 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.520+0000 7f3f54600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fbca0 0x7f3f4c115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.520+0000 7f3f54600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c0fc5b0 0x7f3f4c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.520+0000 7f3f54600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c1167a0 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.520+0000 7f3f54600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c10faf0 con 0x7f3f4c0fc5b0 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.520+0000 7f3f51c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fbca0 0x7f3f4c115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.520+0000 7f3f51c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fbca0 0x7f3f4c115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58974/0 (socket says 172.21.15.1:58974) 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.520+0000 7f3f51c00700 1 -- 172.21.15.1:0/95781313 learned_addr learned my addr 172.21.15.1:0/95781313 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:55.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.521+0000 7f3f51200700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c0fc5b0 0x7f3f4c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.521+0000 7f3f51c00700 1 -- 172.21.15.1:0/95781313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c0fc5b0 msgr2=0x7f3f4c116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.521+0000 7f3f51c00700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c0fc5b0 0x7f3f4c116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.521+0000 7f3f51c00700 1 -- 172.21.15.1:0/95781313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38005040 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.521+0000 7f3f51c00700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fbca0 0x7f3f4c115c10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000cac0 tx=0x7f3f40007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:55.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.521+0000 7f3f49600700 1 -- 172.21.15.1:0/95781313 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f4000aa50 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.522+0000 7f3f49600700 1 -- 172.21.15.1:0/95781313 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f4000abb0 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.522+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c10fdd0 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.522+0000 7f3f49600700 1 -- 172.21.15.1:0/95781313 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f400115e0 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.522+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c110310 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.523+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c106a40 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.524+0000 7f3f49600700 1 -- 172.21.15.1:0/95781313 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3f4000ad20 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.525+0000 7f3f49600700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f3406d7f0 0x7f3f3406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.525+0000 7f3f49600700 1 -- 172.21.15.1:0/95781313 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3f4008d630 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.525+0000 7f3f51200700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f3406d7f0 0x7f3f3406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.526+0000 7f3f51200700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f3406d7f0 0x7f3f3406fca0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c1114b0 tx=0x7f3f38009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:55.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.529+0000 7f3f49600700 1 -- 172.21.15.1:0/95781313 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f40056e60 con 0x7f3f4c0fbca0 2024-02-28T17:07:55.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.759+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f4c04bf70 con 0x7f3f3406d7f0 2024-02-28T17:07:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.789+0000 7f3f49600700 1 -- 172.21.15.1:0/95781313 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3f3800b6e0 con 0x7f3f3406d7f0 2024-02-28T17:07:55.791 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:07:55.791 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (34m) 10m ago 48m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:07:55.791 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (30m) 10m ago 48m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:07:55.791 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (30m) 10m ago 47m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (33m) 10m ago 47m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (26m) 10m ago 41m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (26m) 10m ago 41m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (40m) 10m ago 40m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (10m) 10m ago 41m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (36m) 10m ago 50m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (35m) 10m ago 47m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (32m) 10m ago 50m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (31m) 10m ago 46m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (35m) 10m ago 48m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (35m) 10m ago 47m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (30m) 10m ago 45m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:07:55.792 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (29m) 10m ago 45m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (29m) 10m ago 44m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (28m) 10m ago 44m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (27m) 10m ago 43m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (27m) 10m ago 43m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (34m) 10m ago 47m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f3406d7f0 msgr2=0x7f3f3406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f3406d7f0 0x7f3f3406fca0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c1114b0 tx=0x7f3f38009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fbca0 msgr2=0x7f3f4c115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fbca0 0x7f3f4c115c10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000cac0 tx=0x7f3f40007ee0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 shutdown_connections 2024-02-28T17:07:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f3406d7f0 0x7f3f3406fca0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0fbca0 0x7f3f4c115c10 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 --2- 172.21.15.1:0/95781313 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f4c0fc5b0 0x7f3f4c116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 >> 172.21.15.1:0/95781313 conn(0x7f3f4c0f79d0 msgr2=0x7f3f4c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:55.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 shutdown_connections 2024-02-28T17:07:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.792+0000 7f3f54600700 1 -- 172.21.15.1:0/95781313 wait complete. 2024-02-28T17:07:55.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.945+0000 7fcbf1200700 1 -- 172.21.15.1:0/342219444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec103dc0 msgr2=0x7fcbec104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.945+0000 7fcbf1200700 1 --2- 172.21.15.1:0/342219444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec103dc0 0x7fcbec104210 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4004940 tx=0x7fcbd4009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:55.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.945+0000 7fcbf1200700 1 -- 172.21.15.1:0/342219444 shutdown_connections 2024-02-28T17:07:55.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.945+0000 7fcbf1200700 1 --2- 172.21.15.1:0/342219444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec103dc0 0x7fcbec104210 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.945+0000 7fcbf1200700 1 --2- 172.21.15.1:0/342219444 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbec1095f0 0x7fcbec1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.945+0000 7fcbf1200700 1 -- 172.21.15.1:0/342219444 >> 172.21.15.1:0/342219444 conn(0x7fcbec072d50 msgr2=0x7fcbec075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:55.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.945+0000 7fcbf1200700 1 -- 172.21.15.1:0/342219444 shutdown_connections 2024-02-28T17:07:55.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.945+0000 7fcbf1200700 1 -- 172.21.15.1:0/342219444 wait complete. 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.946+0000 7fcbf1200700 1 Processor -- start 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.946+0000 7fcbf1200700 1 -- start start 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbf1200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbec103dc0 0x7fcbec11a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbf1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec1095f0 0x7fcbec11a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbf1200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec11ac50 con 0x7fcbec1095f0 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbf1200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec113ed0 con 0x7fcbec103dc0 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbe3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec1095f0 0x7fcbec11a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbe3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec1095f0 0x7fcbec11a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59000/0 (socket says 172.21.15.1:59000) 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbe3e00700 1 -- 172.21.15.1:0/1575854614 learned_addr learned my addr 172.21.15.1:0/1575854614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbe9e00700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbec103dc0 0x7fcbec11a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbe3e00700 1 -- 172.21.15.1:0/1575854614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbec103dc0 msgr2=0x7fcbec11a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbe3e00700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbec103dc0 0x7fcbec11a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.947+0000 7fcbe3e00700 1 -- 172.21.15.1:0/1575854614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd4005040 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.948+0000 7fcbe3e00700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec1095f0 0x7fcbec11a570 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009150 tx=0x7fcbd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.948+0000 7fcbe2000700 1 -- 172.21.15.1:0/1575854614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbd400e040 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.948+0000 7fcbe2000700 1 -- 172.21.15.1:0/1575854614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbd400bbf0 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.948+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec1140f0 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.948+0000 7fcbe2000700 1 -- 172.21.15.1:0/1575854614 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbd401c630 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.948+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec114630 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.949+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec075870 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.950+0000 7fcbe2000700 1 -- 172.21.15.1:0/1575854614 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcbd400bd60 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.951+0000 7fcbe2000700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcbd006d7f0 0x7fcbd006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.951+0000 7fcbe2000700 1 -- 172.21.15.1:0/1575854614 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcbd408e2e0 con 0x7fcbec1095f0 2024-02-28T17:07:55.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.951+0000 7fcbe9e00700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcbd006d7f0 0x7fcbd006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:55.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.952+0000 7fcbe9e00700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcbd006d7f0 0x7fcbd006fca0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc006fd0 tx=0x7fcbdc008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:55.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:55.956+0000 7fcbe2000700 1 -- 172.21.15.1:0/1575854614 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcbd405b6e0 con 0x7fcbec1095f0 2024-02-28T17:07:56.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:07:56.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:07:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.269+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcbec1106a0 con 0x7fcbec1095f0 2024-02-28T17:07:56.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.270+0000 7fcbe2000700 1 -- 172.21.15.1:0/1575854614 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fcbd405b270 con 0x7fcbec1095f0 2024-02-28T17:07:56.272 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:07:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:07:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:07:56.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:07:56.273 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:07:56.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcbd006d7f0 msgr2=0x7fcbd006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:56.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcbd006d7f0 0x7fcbd006fca0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc006fd0 tx=0x7fcbdc008040 comp rx=0 tx=0).stop 2024-02-28T17:07:56.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec1095f0 msgr2=0x7fcbec11a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:56.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec1095f0 0x7fcbec11a570 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009150 tx=0x7fcbd4009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:56.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 shutdown_connections 2024-02-28T17:07:56.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcbd006d7f0 0x7fcbd006fca0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcbec103dc0 0x7fcbec11a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 --2- 172.21.15.1:0/1575854614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbec1095f0 0x7fcbec11a570 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.273+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 >> 172.21.15.1:0/1575854614 conn(0x7fcbec072d50 msgr2=0x7fcbec0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:56.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.274+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 shutdown_connections 2024-02-28T17:07:56.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.274+0000 7fcbf1200700 1 -- 172.21.15.1:0/1575854614 wait complete. 2024-02-28T17:07:56.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.432+0000 7f25cec00700 1 -- 172.21.15.1:0/20986519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8101c70 msgr2=0x7f25c8102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:56.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.432+0000 7f25cec00700 1 --2- 172.21.15.1:0/20986519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8101c70 0x7f25c8102040 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f25bc004940 tx=0x7f25bc009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:56.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.432+0000 7f25cec00700 1 -- 172.21.15.1:0/20986519 shutdown_connections 2024-02-28T17:07:56.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.432+0000 7f25cec00700 1 --2- 172.21.15.1:0/20986519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25c8102610 0x7f25c80fbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.432+0000 7f25cec00700 1 --2- 172.21.15.1:0/20986519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8101c70 0x7f25c8102040 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.432+0000 7f25cec00700 1 -- 172.21.15.1:0/20986519 >> 172.21.15.1:0/20986519 conn(0x7f25c80f77c0 msgr2=0x7f25c80f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:56.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.432+0000 7f25cec00700 1 -- 172.21.15.1:0/20986519 shutdown_connections 2024-02-28T17:07:56.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.432+0000 7f25cec00700 1 -- 172.21.15.1:0/20986519 wait complete. 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.433+0000 7f25cec00700 1 Processor -- start 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.434+0000 7f25cec00700 1 -- start start 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.434+0000 7f25cec00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25c8101c70 0x7f25c8102f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.434+0000 7f25cec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8102610 0x7f25c8103490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.434+0000 7f25cec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c8103ae0 con 0x7f25c8102610 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.434+0000 7f25cec00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c8103c50 con 0x7f25c8101c70 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.434+0000 7f25cce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8102610 0x7f25c8103490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.434+0000 7f25cce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8102610 0x7f25c8103490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59008/0 (socket says 172.21.15.1:59008) 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.434+0000 7f25cce00700 1 -- 172.21.15.1:0/2017639367 learned_addr learned my addr 172.21.15.1:0/2017639367 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:56.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25cd800700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25c8101c70 0x7f25c8102f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25cce00700 1 -- 172.21.15.1:0/2017639367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25c8101c70 msgr2=0x7f25c8102f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25cce00700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25c8101c70 0x7f25c8102f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25cce00700 1 -- 172.21.15.1:0/2017639367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc005040 con 0x7f25c8102610 2024-02-28T17:07:56.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25cce00700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8102610 0x7f25c8103490 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f25b400c890 tx=0x7f25b40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25c5600700 1 -- 172.21.15.1:0/2017639367 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25b400a7d0 con 0x7f25c8102610 2024-02-28T17:07:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25c5600700 1 -- 172.21.15.1:0/2017639367 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25b400ae00 con 0x7f25c8102610 2024-02-28T17:07:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25c5600700 1 -- 172.21.15.1:0/2017639367 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25b40113a0 con 0x7f25c8102610 2024-02-28T17:07:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.435+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c8119ae0 con 0x7f25c8102610 2024-02-28T17:07:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.436+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c811a020 con 0x7f25c8102610 2024-02-28T17:07:56.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.437+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c805f5c0 con 0x7f25c8102610 2024-02-28T17:07:56.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.438+0000 7f25c5600700 1 -- 172.21.15.1:0/2017639367 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f25b4012400 con 0x7f25c8102610 2024-02-28T17:07:56.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.439+0000 7f25c5600700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25b006d7f0 0x7f25b006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:56.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.440+0000 7f25c5600700 1 -- 172.21.15.1:0/2017639367 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f25b408cf30 con 0x7f25c8102610 2024-02-28T17:07:56.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.440+0000 7f25cd800700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25b006d7f0 0x7f25b006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:56.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.440+0000 7f25cd800700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25b006d7f0 0x7f25b006fca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00b020 tx=0x7f25bc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:56.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.444+0000 7f25c5600700 1 -- 172.21.15.1:0/2017639367 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f25b40567e0 con 0x7f25c8102610 2024-02-28T17:07:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.706+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f25c8108d60 con 0x7f25c8102610 2024-02-28T17:07:56.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.708+0000 7f25c5600700 1 -- 172.21.15.1:0/2017639367 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f25b4059ea0 con 0x7f25c8102610 2024-02-28T17:07:56.709 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:07:56.709 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:07:56.709 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:07:56.709 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:07:56.710 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:07:56.711 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:07:56.712 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:07:56.713 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:07:56.713 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:07:56.713 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25b006d7f0 msgr2=0x7f25b006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25b006d7f0 0x7f25b006fca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00b020 tx=0x7f25bc009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8102610 msgr2=0x7f25c8103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8102610 0x7f25c8103490 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f25b400c890 tx=0x7f25b40044c0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 shutdown_connections 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25b006d7f0 0x7f25b006fca0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25c8101c70 0x7f25c8102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 --2- 172.21.15.1:0/2017639367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c8102610 0x7f25c8103490 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.710+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 >> 172.21.15.1:0/2017639367 conn(0x7f25c80f77c0 msgr2=0x7f25c8105110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.711+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 shutdown_connections 2024-02-28T17:07:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.711+0000 7f25cec00700 1 -- 172.21.15.1:0/2017639367 wait complete. 2024-02-28T17:07:56.715 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:07:56.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.862+0000 7f714f600700 1 -- 172.21.15.1:0/2005138905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7148102f50 msgr2=0x7f7148103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:56.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.862+0000 7f714f600700 1 --2- 172.21.15.1:0/2005138905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7148102f50 0x7f7148103320 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f713c004940 tx=0x7f713c009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:56.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.862+0000 7f714f600700 1 -- 172.21.15.1:0/2005138905 shutdown_connections 2024-02-28T17:07:56.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.862+0000 7f714f600700 1 --2- 172.21.15.1:0/2005138905 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f71481038f0 0x7f71480fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.862+0000 7f714f600700 1 --2- 172.21.15.1:0/2005138905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7148102f50 0x7f7148103320 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.862+0000 7f714f600700 1 -- 172.21.15.1:0/2005138905 >> 172.21.15.1:0/2005138905 conn(0x7f71480f8ae0 msgr2=0x7f71480faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:56.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.862+0000 7f714f600700 1 -- 172.21.15.1:0/2005138905 shutdown_connections 2024-02-28T17:07:56.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.862+0000 7f714f600700 1 -- 172.21.15.1:0/2005138905 wait complete. 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.863+0000 7f714f600700 1 Processor -- start 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.863+0000 7f714f600700 1 -- start start 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.863+0000 7f714f600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7148102f50 0x7f7148115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f714f600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71481038f0 0x7f7148116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f714f600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71481167a0 con 0x7f71481038f0 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f714f600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714810faf0 con 0x7f7148102f50 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f7147e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71481038f0 0x7f7148116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f7147e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71481038f0 0x7f7148116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59030/0 (socket says 172.21.15.1:59030) 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f7147e00700 1 -- 172.21.15.1:0/52057780 learned_addr learned my addr 172.21.15.1:0/52057780 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f714cc00700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7148102f50 0x7f7148115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f7147e00700 1 -- 172.21.15.1:0/52057780 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7148102f50 msgr2=0x7f7148115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:56.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f7147e00700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7148102f50 0x7f7148115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:56.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.864+0000 7f7147e00700 1 -- 172.21.15.1:0/52057780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c005040 con 0x7f71481038f0 2024-02-28T17:07:56.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.865+0000 7f7147e00700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71481038f0 0x7f7148116150 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f713400c800 tx=0x7f7134004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:56.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.865+0000 7f7144c00700 1 -- 172.21.15.1:0/52057780 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713400a760 con 0x7f71481038f0 2024-02-28T17:07:56.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.865+0000 7f7144c00700 1 -- 172.21.15.1:0/52057780 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f713400ad90 con 0x7f71481038f0 2024-02-28T17:07:56.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.865+0000 7f7144c00700 1 -- 172.21.15.1:0/52057780 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71340113a0 con 0x7f71481038f0 2024-02-28T17:07:56.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.865+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714810fdd0 con 0x7f71481038f0 2024-02-28T17:07:56.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.865+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7148110310 con 0x7f71481038f0 2024-02-28T17:07:56.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.866+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71480fc6e0 con 0x7f71481038f0 2024-02-28T17:07:56.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.867+0000 7f7144c00700 1 -- 172.21.15.1:0/52057780 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7134012400 con 0x7f71481038f0 2024-02-28T17:07:56.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.869+0000 7f7144c00700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f713006d7f0 0x7f713006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:56.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.869+0000 7f7144c00700 1 -- 172.21.15.1:0/52057780 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f713408ce80 con 0x7f71481038f0 2024-02-28T17:07:56.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.870+0000 7f714cc00700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f713006d7f0 0x7f713006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:56.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.870+0000 7f714cc00700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f713006d7f0 0x7f713006fca0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f713c00b3b0 tx=0x7f713c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:56.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:56.874+0000 7f7144c00700 1 -- 172.21.15.1:0/52057780 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7134056730 con 0x7f71481038f0 2024-02-28T17:07:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:56 smithi001 ceph-mon[112940]: from='client.35794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:56 smithi001 ceph-mon[112940]: pgmap v1005: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:56 smithi001 ceph-mon[112940]: from='client.35798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:56 smithi001 ceph-mon[112940]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:56 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1575854614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:07:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:56 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2017639367' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:07:57.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.116+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71481111f0 con 0x7f713006d7f0 2024-02-28T17:07:57.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.118+0000 7f7144c00700 1 -- 172.21.15.1:0/52057780 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f71481111f0 con 0x7f713006d7f0 2024-02-28T17:07:57.120 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:07:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:07:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:07:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:07:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:07:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:07:57.120 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:07:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:07:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:07:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:07:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:07:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:07:57.121 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:07:57.121 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:07:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.121+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f713006d7f0 msgr2=0x7f713006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.121+0000 7f714f600700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f713006d7f0 0x7f713006fca0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f713c00b3b0 tx=0x7f713c009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71481038f0 msgr2=0x7f7148116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71481038f0 0x7f7148116150 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f713400c800 tx=0x7f7134004470 comp rx=0 tx=0).stop 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 shutdown_connections 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f713006d7f0 0x7f713006fca0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7148102f50 0x7f7148115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 --2- 172.21.15.1:0/52057780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71481038f0 0x7f7148116150 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 >> 172.21.15.1:0/52057780 conn(0x7f71480f8ae0 msgr2=0x7f71480fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 shutdown_connections 2024-02-28T17:07:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.122+0000 7f714f600700 1 -- 172.21.15.1:0/52057780 wait complete. 2024-02-28T17:07:57.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:56 smithi066 ceph-mon[85781]: from='client.35794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:57.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:56 smithi066 ceph-mon[85781]: pgmap v1005: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:57.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:56 smithi066 ceph-mon[85781]: from='client.35798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:57.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:56 smithi066 ceph-mon[85781]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:57.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:56 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1575854614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:07:57.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:56 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2017639367' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:07:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.270+0000 7f188b200700 1 -- 172.21.15.1:0/1219628246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 msgr2=0x7f1884102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.270+0000 7f188b200700 1 --2- 172.21.15.1:0/1219628246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 0x7f1884102210 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f1874004940 tx=0x7f1874009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.270+0000 7f188b200700 1 -- 172.21.15.1:0/1219628246 shutdown_connections 2024-02-28T17:07:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.270+0000 7f188b200700 1 --2- 172.21.15.1:0/1219628246 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18841027e0 0x7f18840fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.270+0000 7f188b200700 1 --2- 172.21.15.1:0/1219628246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 0x7f1884102210 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.270+0000 7f188b200700 1 -- 172.21.15.1:0/1219628246 >> 172.21.15.1:0/1219628246 conn(0x7f1884071d90 msgr2=0x7f1884072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.270+0000 7f188b200700 1 -- 172.21.15.1:0/1219628246 shutdown_connections 2024-02-28T17:07:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.270+0000 7f188b200700 1 -- 172.21.15.1:0/1219628246 wait complete. 2024-02-28T17:07:57.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.271+0000 7f188b200700 1 Processor -- start 2024-02-28T17:07:57.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.272+0000 7f188b200700 1 -- start start 2024-02-28T17:07:57.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.272+0000 7f188b200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 0x7f1884115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:57.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.272+0000 7f188b200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18841027e0 0x7f18841161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:57.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.272+0000 7f188b200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1884116830 con 0x7f1884101e40 2024-02-28T17:07:57.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.272+0000 7f188b200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188410fb40 con 0x7f18841027e0 2024-02-28T17:07:57.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.272+0000 7f1883e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 0x7f1884115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:57.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.272+0000 7f1883e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 0x7f1884115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59056/0 (socket says 172.21.15.1:59056) 2024-02-28T17:07:57.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.272+0000 7f1883e00700 1 -- 172.21.15.1:0/522614451 learned_addr learned my addr 172.21.15.1:0/522614451 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f187be00700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18841027e0 0x7f18841161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f1883e00700 1 -- 172.21.15.1:0/522614451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18841027e0 msgr2=0x7f18841161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f1883e00700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18841027e0 0x7f18841161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f1883e00700 1 -- 172.21.15.1:0/522614451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874005040 con 0x7f1884101e40 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f1883e00700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 0x7f1884115ca0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1874009150 tx=0x7f1874009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f1880c00700 1 -- 172.21.15.1:0/522614451 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187400e040 con 0x7f1884101e40 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f1880c00700 1 -- 172.21.15.1:0/522614451 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f187400bbf0 con 0x7f1884101e40 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188410fdc0 con 0x7f1884101e40 2024-02-28T17:07:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f1880c00700 1 -- 172.21.15.1:0/522614451 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187401c630 con 0x7f1884101e40 2024-02-28T17:07:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.273+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1884110300 con 0x7f1884101e40 2024-02-28T17:07:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.275+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18840738b0 con 0x7f1884101e40 2024-02-28T17:07:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.276+0000 7f1880c00700 1 -- 172.21.15.1:0/522614451 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f187400bd60 con 0x7f1884101e40 2024-02-28T17:07:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.276+0000 7f1880c00700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:07:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.277+0000 7f1880c00700 1 -- 172.21.15.1:0/522614451 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f187408e1d0 con 0x7f1884101e40 2024-02-28T17:07:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.277+0000 7f187be00700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:07:57.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.278+0000 7f187be00700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1884111540 tx=0x7f186c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:07:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.281+0000 7f1880c00700 1 -- 172.21.15.1:0/522614451 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f187405b670 con 0x7f1884101e40 2024-02-28T17:07:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.594+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1884111280 con 0x7f1884101e40 2024-02-28T17:07:57.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.595+0000 7f1880c00700 1 -- 172.21.15.1:0/522614451 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f187405b200 con 0x7f1884101e40 2024-02-28T17:07:57.597 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:07:57.597 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:07:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:07:57.597 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:07:57.597 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:07:57.597 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:07:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:07:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:07:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:07:57.599 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:07:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:07:57.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.598+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 msgr2=0x7f186806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:57.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.598+0000 7f188b200700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1884111540 tx=0x7f186c009ee0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.598+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 msgr2=0x7f1884115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:07:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.598+0000 7f188b200700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 0x7f1884115ca0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1874009150 tx=0x7f1874009e30 comp rx=0 tx=0).stop 2024-02-28T17:07:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.599+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 shutdown_connections 2024-02-28T17:07:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.599+0000 7f188b200700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f186806d7f0 0x7f186806fca0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.599+0000 7f188b200700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1884101e40 0x7f1884115ca0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.599+0000 7f188b200700 1 --2- 172.21.15.1:0/522614451 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18841027e0 0x7f18841161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:07:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.599+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 >> 172.21.15.1:0/522614451 conn(0x7f1884071d90 msgr2=0x7f18840656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:07:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.599+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 shutdown_connections 2024-02-28T17:07:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:07:57.599+0000 7f188b200700 1 -- 172.21.15.1:0/522614451 wait complete. 2024-02-28T17:07:58.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:57 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/522614451' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:07:58.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:57 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/522614451' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:07:59.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:58 smithi066 ceph-mon[85781]: from='client.35814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:59.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:58 smithi066 ceph-mon[85781]: pgmap v1006: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:07:59.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:58 smithi001 ceph-mon[112940]: from='client.35814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:07:59.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:58 smithi001 ceph-mon[112940]: pgmap v1006: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:00.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:07:59 smithi066 ceph-mon[85781]: pgmap v1007: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:00.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:07:59 smithi001 ceph-mon[112940]: pgmap v1007: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:02.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:02 smithi066 ceph-mon[85781]: pgmap v1008: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:02.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:02 smithi001 ceph-mon[112940]: pgmap v1008: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:04.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:04 smithi066 ceph-mon[85781]: pgmap v1009: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:04.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:04 smithi001 ceph-mon[112940]: pgmap v1009: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:06.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:06 smithi066 ceph-mon[85781]: pgmap v1010: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:06.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:06 smithi001 ceph-mon[112940]: pgmap v1010: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:08 smithi066 ceph-mon[85781]: pgmap v1011: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:08.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:08 smithi001 ceph-mon[112940]: pgmap v1011: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:10.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:10 smithi066 ceph-mon[85781]: pgmap v1012: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:10.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:10 smithi001 ceph-mon[112940]: pgmap v1012: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:11.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:08:11.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:08:12.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:12 smithi001 ceph-mon[112940]: pgmap v1013: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:12.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:12 smithi066 ceph-mon[85781]: pgmap v1013: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:14.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:14 smithi001 ceph-mon[112940]: pgmap v1014: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:14.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:14 smithi066 ceph-mon[85781]: pgmap v1014: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:16.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:16 smithi001 ceph-mon[112940]: pgmap v1015: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:16.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:16 smithi066 ceph-mon[85781]: pgmap v1015: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:18.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:18 smithi066 ceph-mon[85781]: pgmap v1016: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:18.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:18 smithi001 ceph-mon[112940]: pgmap v1016: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:20.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:20 smithi066 ceph-mon[85781]: pgmap v1017: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:20.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:20 smithi001 ceph-mon[112940]: pgmap v1017: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:22.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:22 smithi066 ceph-mon[85781]: pgmap v1018: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:22.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:22 smithi001 ceph-mon[112940]: pgmap v1018: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:24.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:24 smithi066 ceph-mon[85781]: pgmap v1019: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:24.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:24 smithi001 ceph-mon[112940]: pgmap v1019: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:26.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:26 smithi066 ceph-mon[85781]: pgmap v1020: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:26.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:08:26.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:26 smithi001 ceph-mon[112940]: pgmap v1020: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:26.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:08:27.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.770+0000 7f853d400700 1 -- 172.21.15.1:0/3528209800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 msgr2=0x7f8538070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:27.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.770+0000 7f853d400700 1 --2- 172.21.15.1:0/3528209800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 0x7f8538070ce0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8524004970 tx=0x7f8524009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:27.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.770+0000 7f853d400700 1 -- 172.21.15.1:0/3528209800 shutdown_connections 2024-02-28T17:08:27.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.770+0000 7f853d400700 1 --2- 172.21.15.1:0/3528209800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 0x7f8538070ce0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:27.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.770+0000 7f853d400700 1 --2- 172.21.15.1:0/3528209800 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8538071f60 0x7f8538070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:27.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.770+0000 7f853d400700 1 -- 172.21.15.1:0/3528209800 >> 172.21.15.1:0/3528209800 conn(0x7f85380f8ae0 msgr2=0x7f85380faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:27.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.770+0000 7f853d400700 1 -- 172.21.15.1:0/3528209800 shutdown_connections 2024-02-28T17:08:27.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.771+0000 7f853d400700 1 -- 172.21.15.1:0/3528209800 wait complete. 2024-02-28T17:08:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.771+0000 7f853d400700 1 Processor -- start 2024-02-28T17:08:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.772+0000 7f853d400700 1 -- start start 2024-02-28T17:08:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.772+0000 7f853d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 0x7f853811a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.772+0000 7f853d400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8538071f60 0x7f853811a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.772+0000 7f853d400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853811abc0 con 0x7f8538070870 2024-02-28T17:08:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.772+0000 7f853d400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8538113ed0 con 0x7f8538071f60 2024-02-28T17:08:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.772+0000 7f8536800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 0x7f853811a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:27.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.772+0000 7f8536800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 0x7f853811a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51316/0 (socket says 172.21.15.1:51316) 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.772+0000 7f8536800700 1 -- 172.21.15.1:0/2501890443 learned_addr learned my addr 172.21.15.1:0/2501890443 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.773+0000 7f8535e00700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8538071f60 0x7f853811a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.773+0000 7f8536800700 1 -- 172.21.15.1:0/2501890443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8538071f60 msgr2=0x7f853811a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.773+0000 7f8536800700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8538071f60 0x7f853811a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.773+0000 7f8536800700 1 -- 172.21.15.1:0/2501890443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524005040 con 0x7f8538070870 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.773+0000 7f8536800700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 0x7f853811a030 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8528002a50 tx=0x7f85280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.773+0000 7f852ea00700 1 -- 172.21.15.1:0/2501890443 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8528007cc0 con 0x7f8538070870 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.774+0000 7f852ea00700 1 -- 172.21.15.1:0/2501890443 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f852800f040 con 0x7f8538070870 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.774+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85381141b0 con 0x7f8538070870 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.774+0000 7f852ea00700 1 -- 172.21.15.1:0/2501890443 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8528009640 con 0x7f8538070870 2024-02-28T17:08:27.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.774+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85381146f0 con 0x7f8538070870 2024-02-28T17:08:27.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.775+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85380fe400 con 0x7f8538070870 2024-02-28T17:08:27.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.776+0000 7f852ea00700 1 -- 172.21.15.1:0/2501890443 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f85280097a0 con 0x7f8538070870 2024-02-28T17:08:27.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.777+0000 7f852ea00700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f851c06d7e0 0x7f851c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:27.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.777+0000 7f852ea00700 1 -- 172.21.15.1:0/2501890443 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f852808cbc0 con 0x7f8538070870 2024-02-28T17:08:27.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.777+0000 7f8535e00700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f851c06d7e0 0x7f851c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:27.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.778+0000 7f8535e00700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f851c06d7e0 0x7f851c06fc90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f85381158d0 tx=0x7f8524009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:27.782+0000 7f852ea00700 1 -- 172.21.15.1:0/2501890443 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8528059fc0 con 0x7f8538070870 2024-02-28T17:08:28.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.026+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f853804bf40 con 0x7f851c06d7e0 2024-02-28T17:08:28.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.028+0000 7f852ea00700 1 -- 172.21.15.1:0/2501890443 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f853804bf40 con 0x7f851c06d7e0 2024-02-28T17:08:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.031+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f851c06d7e0 msgr2=0x7f851c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.031+0000 7f853d400700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f851c06d7e0 0x7f851c06fc90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f85381158d0 tx=0x7f8524009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.031+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 msgr2=0x7f853811a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.031+0000 7f853d400700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 0x7f853811a030 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8528002a50 tx=0x7f85280046b0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.032+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 shutdown_connections 2024-02-28T17:08:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.032+0000 7f853d400700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f851c06d7e0 0x7f851c06fc90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.032+0000 7f853d400700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8538070870 0x7f853811a030 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.032+0000 7f853d400700 1 --2- 172.21.15.1:0/2501890443 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8538071f60 0x7f853811a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.032+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 >> 172.21.15.1:0/2501890443 conn(0x7f85380f8ae0 msgr2=0x7f85380fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:28.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.032+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 shutdown_connections 2024-02-28T17:08:28.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.032+0000 7f853d400700 1 -- 172.21.15.1:0/2501890443 wait complete. 2024-02-28T17:08:28.051 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:08:28.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.195+0000 7ff3a7800700 1 -- 172.21.15.1:0/1877177173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 msgr2=0x7ff3a0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.195+0000 7ff3a7800700 1 --2- 172.21.15.1:0/1877177173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 0x7ff3a0103320 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ff394004940 tx=0x7ff394009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:28.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.195+0000 7ff3a7800700 1 -- 172.21.15.1:0/1877177173 shutdown_connections 2024-02-28T17:08:28.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.195+0000 7ff3a7800700 1 --2- 172.21.15.1:0/1877177173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3a01038f0 0x7ff3a00fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.195+0000 7ff3a7800700 1 --2- 172.21.15.1:0/1877177173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 0x7ff3a0103320 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.195+0000 7ff3a7800700 1 -- 172.21.15.1:0/1877177173 >> 172.21.15.1:0/1877177173 conn(0x7ff3a00f8ae0 msgr2=0x7ff3a00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:28.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.196+0000 7ff3a7800700 1 -- 172.21.15.1:0/1877177173 shutdown_connections 2024-02-28T17:08:28.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.196+0000 7ff3a7800700 1 -- 172.21.15.1:0/1877177173 wait complete. 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.197+0000 7ff3a7800700 1 Processor -- start 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.197+0000 7ff3a7800700 1 -- start start 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.197+0000 7ff3a7800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 0x7ff3a0115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.197+0000 7ff3a7800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3a01038f0 0x7ff3a0116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.197+0000 7ff3a7800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a01167b0 con 0x7ff3a0102f50 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.197+0000 7ff3a7800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a010fb00 con 0x7ff3a01038f0 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.197+0000 7ff3a4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 0x7ff3a0115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.198+0000 7ff3a4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 0x7ff3a0115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51348/0 (socket says 172.21.15.1:51348) 2024-02-28T17:08:28.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.198+0000 7ff3a4e00700 1 -- 172.21.15.1:0/189396675 learned_addr learned my addr 172.21.15.1:0/189396675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:08:28.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.198+0000 7ff39fe00700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3a01038f0 0x7ff3a0116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:28.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.198+0000 7ff3a4e00700 1 -- 172.21.15.1:0/189396675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3a01038f0 msgr2=0x7ff3a0116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.198+0000 7ff3a4e00700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3a01038f0 0x7ff3a0116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.198+0000 7ff3a4e00700 1 -- 172.21.15.1:0/189396675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff394005040 con 0x7ff3a0102f50 2024-02-28T17:08:28.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.198+0000 7ff3a4e00700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 0x7ff3a0115c20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff39400b3b0 tx=0x7ff394009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:28.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.199+0000 7ff39cc00700 1 -- 172.21.15.1:0/189396675 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff39400e040 con 0x7ff3a0102f50 2024-02-28T17:08:28.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.199+0000 7ff39cc00700 1 -- 172.21.15.1:0/189396675 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff39400bbf0 con 0x7ff3a0102f50 2024-02-28T17:08:28.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.199+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a010fd80 con 0x7ff3a0102f50 2024-02-28T17:08:28.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.199+0000 7ff39cc00700 1 -- 172.21.15.1:0/189396675 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff39401c630 con 0x7ff3a0102f50 2024-02-28T17:08:28.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.199+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a01102c0 con 0x7ff3a0102f50 2024-02-28T17:08:28.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.200+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a00fc6e0 con 0x7ff3a0102f50 2024-02-28T17:08:28.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.201+0000 7ff39cc00700 1 -- 172.21.15.1:0/189396675 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff39400bd60 con 0x7ff3a0102f50 2024-02-28T17:08:28.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.202+0000 7ff39cc00700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff38806d7e0 0x7ff38806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:28.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.202+0000 7ff39cc00700 1 -- 172.21.15.1:0/189396675 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff39408e380 con 0x7ff3a0102f50 2024-02-28T17:08:28.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.203+0000 7ff39fe00700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff38806d7e0 0x7ff38806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:28.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.203+0000 7ff39fe00700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff38806d7e0 0x7ff38806fc90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff3a01114c0 tx=0x7ff38c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:28.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.207+0000 7ff39cc00700 1 -- 172.21.15.1:0/189396675 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff394057bb0 con 0x7ff3a0102f50 2024-02-28T17:08:28.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.452+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3a0111310 con 0x7ff38806d7e0 2024-02-28T17:08:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.455+0000 7ff39cc00700 1 -- 172.21.15.1:0/189396675 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff3a0111310 con 0x7ff38806d7e0 2024-02-28T17:08:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff38806d7e0 msgr2=0x7ff38806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff38806d7e0 0x7ff38806fc90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff3a01114c0 tx=0x7ff38c00b040 comp rx=0 tx=0).stop 2024-02-28T17:08:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 msgr2=0x7ff3a0115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 0x7ff3a0115c20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff39400b3b0 tx=0x7ff394009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 shutdown_connections 2024-02-28T17:08:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff38806d7e0 0x7ff38806fc90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3a0102f50 0x7ff3a0115c20 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 --2- 172.21.15.1:0/189396675 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3a01038f0 0x7ff3a0116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 >> 172.21.15.1:0/189396675 conn(0x7ff3a00f8ae0 msgr2=0x7ff3a00fa3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 shutdown_connections 2024-02-28T17:08:28.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.458+0000 7ff3a7800700 1 -- 172.21.15.1:0/189396675 wait complete. 2024-02-28T17:08:28.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.612+0000 7f968a600700 1 -- 172.21.15.1:0/2423744023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 msgr2=0x7f96840fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.612+0000 7f968a600700 1 --2- 172.21.15.1:0/2423744023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 0x7f96840fcf60 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9670004940 tx=0x7f9670009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:28.616 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:28 smithi001 ceph-mon[112940]: pgmap v1021: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.613+0000 7f968a600700 1 -- 172.21.15.1:0/2423744023 shutdown_connections 2024-02-28T17:08:28.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.613+0000 7f968a600700 1 --2- 172.21.15.1:0/2423744023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 0x7f96840fcf60 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.613+0000 7f968a600700 1 --2- 172.21.15.1:0/2423744023 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9684102f50 0x7f9684103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.613+0000 7f968a600700 1 -- 172.21.15.1:0/2423744023 >> 172.21.15.1:0/2423744023 conn(0x7f96840f8ae0 msgr2=0x7f96840faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.613+0000 7f968a600700 1 -- 172.21.15.1:0/2423744023 shutdown_connections 2024-02-28T17:08:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.613+0000 7f968a600700 1 -- 172.21.15.1:0/2423744023 wait complete. 2024-02-28T17:08:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.614+0000 7f968a600700 1 Processor -- start 2024-02-28T17:08:28.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.614+0000 7f968a600700 1 -- start start 2024-02-28T17:08:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.614+0000 7f968a600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9684102f50 0x7f9684115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.614+0000 7f968a600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 0x7f96841161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.614+0000 7f968a600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9684116830 con 0x7f96841038f0 2024-02-28T17:08:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.614+0000 7f968a600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968410fb40 con 0x7f9684102f50 2024-02-28T17:08:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.615+0000 7f9682a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 0x7f96841161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.615+0000 7f9682a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 0x7f96841161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51370/0 (socket says 172.21.15.1:51370) 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.615+0000 7f9682a00700 1 -- 172.21.15.1:0/2995266682 learned_addr learned my addr 172.21.15.1:0/2995266682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.615+0000 7f9683400700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9684102f50 0x7f9684115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.615+0000 7f9682a00700 1 -- 172.21.15.1:0/2995266682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9684102f50 msgr2=0x7f9684115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.615+0000 7f9682a00700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9684102f50 0x7f9684115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.615+0000 7f9682a00700 1 -- 172.21.15.1:0/2995266682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9670005040 con 0x7f96841038f0 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.615+0000 7f9683400700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9684102f50 0x7f9684115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.616+0000 7f9682a00700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 0x7f96841161e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9670009030 tx=0x7f9670009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.616+0000 7f967b400700 1 -- 172.21.15.1:0/2995266682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f967000e040 con 0x7f96841038f0 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.616+0000 7f967b400700 1 -- 172.21.15.1:0/2995266682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f967000bbf0 con 0x7f96841038f0 2024-02-28T17:08:28.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.616+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968410fdc0 con 0x7f96841038f0 2024-02-28T17:08:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.616+0000 7f967b400700 1 -- 172.21.15.1:0/2995266682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f967001c630 con 0x7f96841038f0 2024-02-28T17:08:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.616+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9684110300 con 0x7f96841038f0 2024-02-28T17:08:28.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.617+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96840fc6e0 con 0x7f96841038f0 2024-02-28T17:08:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.618+0000 7f967b400700 1 -- 172.21.15.1:0/2995266682 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f967000bd60 con 0x7f96841038f0 2024-02-28T17:08:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.619+0000 7f967b400700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f966806d7f0 0x7f966806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.620+0000 7f967b400700 1 -- 172.21.15.1:0/2995266682 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f967008e130 con 0x7f96841038f0 2024-02-28T17:08:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.620+0000 7f9683400700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f966806d7f0 0x7f966806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.620+0000 7f9683400700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f966806d7f0 0x7f966806fca0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9674009fd0 tx=0x7f967400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:28.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.624+0000 7f967b400700 1 -- 172.21.15.1:0/2995266682 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f967005b5d0 con 0x7f96841038f0 2024-02-28T17:08:28.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:28 smithi066 ceph-mon[85781]: pgmap v1021: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.869+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9684111280 con 0x7f966806d7f0 2024-02-28T17:08:28.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.892+0000 7f967b400700 1 -- 172.21.15.1:0/2995266682 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9684111280 con 0x7f966806d7f0 2024-02-28T17:08:28.895 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:08:28.895 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (34m) 11m ago 48m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:08:28.895 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (31m) 11m ago 49m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:08:28.895 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (31m) 11m ago 47m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:08:28.895 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (34m) 11m ago 48m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (27m) 11m ago 41m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (26m) 11m ago 41m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (41m) 11m ago 41m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (11m) 11m ago 41m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (37m) 11m ago 50m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (36m) 11m ago 47m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (32m) 11m ago 50m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (32m) 11m ago 47m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (35m) 11m ago 49m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:08:28.896 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (35m) 11m ago 47m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (30m) 11m ago 46m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (30m) 11m ago 45m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (29m) 11m ago 45m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (29m) 11m ago 44m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (28m) 11m ago 44m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (27m) 11m ago 43m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (35m) 11m ago 48m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f966806d7f0 msgr2=0x7f966806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f966806d7f0 0x7f966806fca0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9674009fd0 tx=0x7f967400b040 comp rx=0 tx=0).stop 2024-02-28T17:08:28.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 msgr2=0x7f96841161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:28.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 0x7f96841161e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9670009030 tx=0x7f9670009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:28.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 shutdown_connections 2024-02-28T17:08:28.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f966806d7f0 0x7f966806fca0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9684102f50 0x7f9684115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 --2- 172.21.15.1:0/2995266682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96841038f0 0x7f96841161e0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:28.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 >> 172.21.15.1:0/2995266682 conn(0x7f96840f8ae0 msgr2=0x7f96840fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:28.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 shutdown_connections 2024-02-28T17:08:28.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:28.895+0000 7f968a600700 1 -- 172.21.15.1:0/2995266682 wait complete. 2024-02-28T17:08:29.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.055+0000 7fea30c00700 1 -- 172.21.15.1:0/282244972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 msgr2=0x7fea2c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.055+0000 7fea30c00700 1 --2- 172.21.15.1:0/282244972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 0x7fea2c0fcf60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fea18004940 tx=0x7fea18009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:29.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.055+0000 7fea30c00700 1 -- 172.21.15.1:0/282244972 shutdown_connections 2024-02-28T17:08:29.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.055+0000 7fea30c00700 1 --2- 172.21.15.1:0/282244972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 0x7fea2c0fcf60 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.055+0000 7fea30c00700 1 --2- 172.21.15.1:0/282244972 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea2c102f50 0x7fea2c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.055+0000 7fea30c00700 1 -- 172.21.15.1:0/282244972 >> 172.21.15.1:0/282244972 conn(0x7fea2c0f8ae0 msgr2=0x7fea2c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:29.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.055+0000 7fea30c00700 1 -- 172.21.15.1:0/282244972 shutdown_connections 2024-02-28T17:08:29.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.056+0000 7fea30c00700 1 -- 172.21.15.1:0/282244972 wait complete. 2024-02-28T17:08:29.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.056+0000 7fea30c00700 1 Processor -- start 2024-02-28T17:08:29.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.056+0000 7fea30c00700 1 -- start start 2024-02-28T17:08:29.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.057+0000 7fea30c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea2c102f50 0x7fea2c070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.057+0000 7fea30c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 0x7fea2c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.057+0000 7fea30c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c070dd0 con 0x7fea2c1038f0 2024-02-28T17:08:29.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.057+0000 7fea30c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c070f40 con 0x7fea2c102f50 2024-02-28T17:08:29.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.057+0000 7fea29400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 0x7fea2c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.057+0000 7fea29400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 0x7fea2c0707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51390/0 (socket says 172.21.15.1:51390) 2024-02-28T17:08:29.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.057+0000 7fea29400700 1 -- 172.21.15.1:0/1014377217 learned_addr learned my addr 172.21.15.1:0/1014377217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea29e00700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea2c102f50 0x7fea2c070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea29400700 1 -- 172.21.15.1:0/1014377217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea2c102f50 msgr2=0x7fea2c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea29400700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea2c102f50 0x7fea2c070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea29400700 1 -- 172.21.15.1:0/1014377217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18005040 con 0x7fea2c1038f0 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea29400700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 0x7fea2c0707d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fea180090d0 tx=0x7fea18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea22000700 1 -- 172.21.15.1:0/1014377217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea1800e040 con 0x7fea2c1038f0 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea22000700 1 -- 172.21.15.1:0/1014377217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea1800bbf0 con 0x7fea2c1038f0 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2c119cb0 con 0x7fea2c1038f0 2024-02-28T17:08:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.058+0000 7fea22000700 1 -- 172.21.15.1:0/1014377217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea1801c630 con 0x7fea2c1038f0 2024-02-28T17:08:29.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.059+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2c11a110 con 0x7fea2c1038f0 2024-02-28T17:08:29.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.060+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c105420 con 0x7fea2c1038f0 2024-02-28T17:08:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.061+0000 7fea22000700 1 -- 172.21.15.1:0/1014377217 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fea1800bd60 con 0x7fea2c1038f0 2024-02-28T17:08:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.062+0000 7fea22000700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea1006d7e0 0x7fea1006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.062+0000 7fea22000700 1 -- 172.21.15.1:0/1014377217 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fea1808e260 con 0x7fea2c1038f0 2024-02-28T17:08:29.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.062+0000 7fea29e00700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea1006d7e0 0x7fea1006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.063+0000 7fea29e00700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea1006d7e0 0x7fea1006fc90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fea1c004c40 tx=0x7fea1c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:29.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.067+0000 7fea22000700 1 -- 172.21.15.1:0/1014377217 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fea18057a90 con 0x7fea2c1038f0 2024-02-28T17:08:29.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.382+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fea2c0719d0 con 0x7fea2c1038f0 2024-02-28T17:08:29.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.384+0000 7fea22000700 1 -- 172.21.15.1:0/1014377217 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fea18013e60 con 0x7fea2c1038f0 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:08:29.387 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:08:29.388 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:08:29.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:08:29.388 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:08:29.388 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:08:29.389 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:08:29.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:08:29.389 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:08:29.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea1006d7e0 msgr2=0x7fea1006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea1006d7e0 0x7fea1006fc90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fea1c004c40 tx=0x7fea1c009040 comp rx=0 tx=0).stop 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 msgr2=0x7fea2c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 0x7fea2c0707d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fea180090d0 tx=0x7fea18009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 shutdown_connections 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fea1006d7e0 0x7fea1006fc90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fea2c102f50 0x7fea2c070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 --2- 172.21.15.1:0/1014377217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea2c1038f0 0x7fea2c0707d0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 >> 172.21.15.1:0/1014377217 conn(0x7fea2c0f8ae0 msgr2=0x7fea2c0fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 shutdown_connections 2024-02-28T17:08:29.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.387+0000 7fea30c00700 1 -- 172.21.15.1:0/1014377217 wait complete. 2024-02-28T17:08:29.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.542+0000 7f4d27000700 1 -- 172.21.15.1:0/463316981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20071f60 msgr2=0x7f4d20070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.542+0000 7f4d27000700 1 --2- 172.21.15.1:0/463316981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20071f60 0x7f4d20070330 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f4d10004970 tx=0x7f4d10009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.543+0000 7f4d27000700 1 -- 172.21.15.1:0/463316981 shutdown_connections 2024-02-28T17:08:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.543+0000 7f4d27000700 1 --2- 172.21.15.1:0/463316981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d20070870 0x7f4d20070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.543+0000 7f4d27000700 1 --2- 172.21.15.1:0/463316981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20071f60 0x7f4d20070330 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.543+0000 7f4d27000700 1 -- 172.21.15.1:0/463316981 >> 172.21.15.1:0/463316981 conn(0x7f4d200f8ae0 msgr2=0x7f4d200faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.543+0000 7f4d27000700 1 -- 172.21.15.1:0/463316981 shutdown_connections 2024-02-28T17:08:29.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.543+0000 7f4d27000700 1 -- 172.21.15.1:0/463316981 wait complete. 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d27000700 1 Processor -- start 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d27000700 1 -- start start 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d27000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20070870 0x7f4d2011a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d27000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d20071f60 0x7f4d2011a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d27000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2011ad10 con 0x7f4d20070870 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d27000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20113f90 con 0x7f4d20071f60 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d1fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20070870 0x7f4d2011a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d1fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20070870 0x7f4d2011a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51414/0 (socket says 172.21.15.1:51414) 2024-02-28T17:08:29.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.544+0000 7f4d1fe00700 1 -- 172.21.15.1:0/2616947981 learned_addr learned my addr 172.21.15.1:0/2616947981 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.545+0000 7f4d17400700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d20071f60 0x7f4d2011a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.545+0000 7f4d1fe00700 1 -- 172.21.15.1:0/2616947981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d20071f60 msgr2=0x7f4d2011a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.545+0000 7f4d1fe00700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d20071f60 0x7f4d2011a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.545+0000 7f4d1fe00700 1 -- 172.21.15.1:0/2616947981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d10005040 con 0x7f4d20070870 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.545+0000 7f4d17400700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d20071f60 0x7f4d2011a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.545+0000 7f4d1fe00700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20070870 0x7f4d2011a0f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4d10000c00 tx=0x7f4d10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.545+0000 7f4d1cc00700 1 -- 172.21.15.1:0/2616947981 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1000e040 con 0x7f4d20070870 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.546+0000 7f4d1cc00700 1 -- 172.21.15.1:0/2616947981 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d1000bc10 con 0x7f4d20070870 2024-02-28T17:08:29.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.546+0000 7f4d1cc00700 1 -- 172.21.15.1:0/2616947981 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1001c610 con 0x7f4d20070870 2024-02-28T17:08:29.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.546+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d201141b0 con 0x7f4d20070870 2024-02-28T17:08:29.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.546+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d201146f0 con 0x7f4d20070870 2024-02-28T17:08:29.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.547+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d200fe400 con 0x7f4d20070870 2024-02-28T17:08:29.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.548+0000 7f4d1cc00700 1 -- 172.21.15.1:0/2616947981 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4d10013a80 con 0x7f4d20070870 2024-02-28T17:08:29.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.549+0000 7f4d1cc00700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0406d7e0 0x7f4d0406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.549+0000 7f4d1cc00700 1 -- 172.21.15.1:0/2616947981 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4d1008e150 con 0x7f4d20070870 2024-02-28T17:08:29.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.549+0000 7f4d17400700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0406d7e0 0x7f4d0406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.550+0000 7f4d17400700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0406d7e0 0x7f4d0406fc90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4d08006fd0 tx=0x7f4d08008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:29.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.554+0000 7f4d1cc00700 1 -- 172.21.15.1:0/2616947981 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4d1005b5f0 con 0x7f4d20070870 2024-02-28T17:08:29.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:29 smithi066 ceph-mon[85781]: from='client.35822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:08:29.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:29 smithi066 ceph-mon[85781]: from='client.35826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:08:29.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:29 smithi066 ceph-mon[85781]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:08:29.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:29 smithi001 ceph-mon[112940]: from='client.35822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:08:29.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:29 smithi001 ceph-mon[112940]: from='client.35826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:08:29.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:29 smithi001 ceph-mon[112940]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:08:29.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.826+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4d20114340 con 0x7f4d20070870 2024-02-28T17:08:29.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.827+0000 7f4d1cc00700 1 -- 172.21.15.1:0/2616947981 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f4d20114340 con 0x7f4d20070870 2024-02-28T17:08:29.830 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:08:29.830 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:08:29.830 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:08:29.830 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:08:29.831 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:08:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.831+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0406d7e0 msgr2=0x7f4d0406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.831+0000 7f4d27000700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0406d7e0 0x7f4d0406fc90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4d08006fd0 tx=0x7f4d08008040 comp rx=0 tx=0).stop 2024-02-28T17:08:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.831+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20070870 msgr2=0x7f4d2011a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.831+0000 7f4d27000700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20070870 0x7f4d2011a0f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4d10000c00 tx=0x7f4d10009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.832+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 shutdown_connections 2024-02-28T17:08:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.832+0000 7f4d27000700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4d0406d7e0 0x7f4d0406fc90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.832+0000 7f4d27000700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d20070870 0x7f4d2011a0f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.832+0000 7f4d27000700 1 --2- 172.21.15.1:0/2616947981 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4d20071f60 0x7f4d2011a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.832+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 >> 172.21.15.1:0/2616947981 conn(0x7f4d200f8ae0 msgr2=0x7f4d200fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.832+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 shutdown_connections 2024-02-28T17:08:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.832+0000 7f4d27000700 1 -- 172.21.15.1:0/2616947981 wait complete. 2024-02-28T17:08:29.836 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:08:29.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.983+0000 7f6653a00700 1 -- 172.21.15.1:0/891290078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 msgr2=0x7f664c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.983+0000 7f6653a00700 1 --2- 172.21.15.1:0/891290078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 0x7f664c0fcf60 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6638004940 tx=0x7f6638009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:29.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.984+0000 7f6653a00700 1 -- 172.21.15.1:0/891290078 shutdown_connections 2024-02-28T17:08:29.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.984+0000 7f6653a00700 1 --2- 172.21.15.1:0/891290078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 0x7f664c0fcf60 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.984+0000 7f6653a00700 1 --2- 172.21.15.1:0/891290078 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f664c102f50 0x7f664c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.984+0000 7f6653a00700 1 -- 172.21.15.1:0/891290078 >> 172.21.15.1:0/891290078 conn(0x7f664c0f8ae0 msgr2=0x7f664c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:29.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.984+0000 7f6653a00700 1 -- 172.21.15.1:0/891290078 shutdown_connections 2024-02-28T17:08:29.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.984+0000 7f6653a00700 1 -- 172.21.15.1:0/891290078 wait complete. 2024-02-28T17:08:29.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.985+0000 7f6653a00700 1 Processor -- start 2024-02-28T17:08:29.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.985+0000 7f6653a00700 1 -- start start 2024-02-28T17:08:29.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f6653a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f664c102f50 0x7f664c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f6653a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 0x7f664c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f6653a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c116810 con 0x7f664c1038f0 2024-02-28T17:08:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f6653a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c10fb40 con 0x7f664c102f50 2024-02-28T17:08:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f664be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 0x7f664c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f664be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 0x7f664c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51434/0 (socket says 172.21.15.1:51434) 2024-02-28T17:08:29.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f664be00700 1 -- 172.21.15.1:0/1623451251 learned_addr learned my addr 172.21.15.1:0/1623451251 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:08:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f6651000700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f664c102f50 0x7f664c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f664be00700 1 -- 172.21.15.1:0/1623451251 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f664c102f50 msgr2=0x7f664c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f664be00700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f664c102f50 0x7f664c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.986+0000 7f664be00700 1 -- 172.21.15.1:0/1623451251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6638005040 con 0x7f664c1038f0 2024-02-28T17:08:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.987+0000 7f664be00700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 0x7f664c1161c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6638009030 tx=0x7f6638009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.987+0000 7f6648c00700 1 -- 172.21.15.1:0/1623451251 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663800e040 con 0x7f664c1038f0 2024-02-28T17:08:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.987+0000 7f6648c00700 1 -- 172.21.15.1:0/1623451251 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f663800bbf0 con 0x7f664c1038f0 2024-02-28T17:08:29.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.987+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c10fdc0 con 0x7f664c1038f0 2024-02-28T17:08:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.987+0000 7f6648c00700 1 -- 172.21.15.1:0/1623451251 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663801c630 con 0x7f664c1038f0 2024-02-28T17:08:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.987+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c110300 con 0x7f664c1038f0 2024-02-28T17:08:29.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.989+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c0fc6e0 con 0x7f664c1038f0 2024-02-28T17:08:29.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.989+0000 7f6648c00700 1 -- 172.21.15.1:0/1623451251 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f663800bd60 con 0x7f664c1038f0 2024-02-28T17:08:29.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.990+0000 7f6648c00700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663406d7f0 0x7f663406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:29.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.990+0000 7f6648c00700 1 -- 172.21.15.1:0/1623451251 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f663808e280 con 0x7f664c1038f0 2024-02-28T17:08:29.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.991+0000 7f6651000700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663406d7f0 0x7f663406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:29.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.991+0000 7f6651000700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663406d7f0 0x7f663406fca0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6640007fd0 tx=0x7f6640009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:29.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:29.995+0000 7f6648c00700 1 -- 172.21.15.1:0/1623451251 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f663805b620 con 0x7f664c1038f0 2024-02-28T17:08:30.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.230+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f664c1111f0 con 0x7f663406d7f0 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.232+0000 7f6648c00700 1 -- 172.21.15.1:0/1623451251 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f664c1111f0 con 0x7f663406d7f0 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:08:30.235 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:08:30.236 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:08:30.236 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:08:30.236 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:08:30.236 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:08:30.236 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:08:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663406d7f0 msgr2=0x7f663406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663406d7f0 0x7f663406fca0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6640007fd0 tx=0x7f6640009040 comp rx=0 tx=0).stop 2024-02-28T17:08:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 msgr2=0x7f664c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 0x7f664c1161c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6638009030 tx=0x7f6638009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 shutdown_connections 2024-02-28T17:08:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f663406d7f0 0x7f663406fca0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f664c102f50 0x7f664c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 --2- 172.21.15.1:0/1623451251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f664c1038f0 0x7f664c1161c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.235+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 >> 172.21.15.1:0/1623451251 conn(0x7f664c0f8ae0 msgr2=0x7f664c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.236+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 shutdown_connections 2024-02-28T17:08:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.236+0000 7f6653a00700 1 -- 172.21.15.1:0/1623451251 wait complete. 2024-02-28T17:08:30.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.391+0000 7fa6a5c00700 1 -- 172.21.15.1:0/726676205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 msgr2=0x7fa6a00fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:30.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.391+0000 7fa6a5c00700 1 --2- 172.21.15.1:0/726676205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 0x7fa6a00fe160 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa690004940 tx=0x7fa690009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:30.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.392+0000 7fa6a5c00700 1 -- 172.21.15.1:0/726676205 shutdown_connections 2024-02-28T17:08:30.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.392+0000 7fa6a5c00700 1 --2- 172.21.15.1:0/726676205 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a00fe730 0x7fa6a0106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.392+0000 7fa6a5c00700 1 --2- 172.21.15.1:0/726676205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 0x7fa6a00fe160 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.392+0000 7fa6a5c00700 1 -- 172.21.15.1:0/726676205 >> 172.21.15.1:0/726676205 conn(0x7fa6a0073ab0 msgr2=0x7fa6a0073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:30.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.392+0000 7fa6a5c00700 1 -- 172.21.15.1:0/726676205 shutdown_connections 2024-02-28T17:08:30.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.392+0000 7fa6a5c00700 1 -- 172.21.15.1:0/726676205 wait complete. 2024-02-28T17:08:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa6a5c00700 1 Processor -- start 2024-02-28T17:08:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa6a5c00700 1 -- start start 2024-02-28T17:08:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa6a5c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 0x7fa6a0115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa6a5c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a00fe730 0x7fa6a0115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa6a5c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a0116600 con 0x7fa6a00fdd90 2024-02-28T17:08:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa6a5c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a010f930 con 0x7fa6a00fe730 2024-02-28T17:08:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa69fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 0x7fa6a0115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa69fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 0x7fa6a0115a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51458/0 (socket says 172.21.15.1:51458) 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.393+0000 7fa69fe00700 1 -- 172.21.15.1:0/3171526447 learned_addr learned my addr 172.21.15.1:0/3171526447 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.394+0000 7fa69f400700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a00fe730 0x7fa6a0115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.394+0000 7fa69fe00700 1 -- 172.21.15.1:0/3171526447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a00fe730 msgr2=0x7fa6a0115fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.394+0000 7fa69fe00700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a00fe730 0x7fa6a0115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.394+0000 7fa69fe00700 1 -- 172.21.15.1:0/3171526447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa690005040 con 0x7fa6a00fdd90 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.394+0000 7fa69fe00700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 0x7fa6a0115a70 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa690000c00 tx=0x7fa690009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.394+0000 7fa697e00700 1 -- 172.21.15.1:0/3171526447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa69000e040 con 0x7fa6a00fdd90 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.394+0000 7fa697e00700 1 -- 172.21.15.1:0/3171526447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa69000bb90 con 0x7fa6a00fdd90 2024-02-28T17:08:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.394+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a010fbb0 con 0x7fa6a00fdd90 2024-02-28T17:08:30.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.395+0000 7fa697e00700 1 -- 172.21.15.1:0/3171526447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa69001c630 con 0x7fa6a00fdd90 2024-02-28T17:08:30.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.395+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a01100f0 con 0x7fa6a00fdd90 2024-02-28T17:08:30.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.396+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a01068c0 con 0x7fa6a00fdd90 2024-02-28T17:08:30.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.397+0000 7fa697e00700 1 -- 172.21.15.1:0/3171526447 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa69000bd00 con 0x7fa6a00fdd90 2024-02-28T17:08:30.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.398+0000 7fa697e00700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68406d7f0 0x7fa68406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:08:30.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.398+0000 7fa697e00700 1 -- 172.21.15.1:0/3171526447 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa69008e300 con 0x7fa6a00fdd90 2024-02-28T17:08:30.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.398+0000 7fa69f400700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68406d7f0 0x7fa68406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:08:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.399+0000 7fa69f400700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68406d7f0 0x7fa68406fca0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa6a01112a0 tx=0x7fa68c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:08:30.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.402+0000 7fa697e00700 1 -- 172.21.15.1:0/3171526447 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa69005b5e0 con 0x7fa6a00fdd90 2024-02-28T17:08:30.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:30 smithi001 ceph-mon[112940]: pgmap v1022: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:30.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1014377217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:08:30.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2616947981' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:08:30.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.702+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa6a004bed0 con 0x7fa6a00fdd90 2024-02-28T17:08:30.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.703+0000 7fa697e00700 1 -- 172.21.15.1:0/3171526447 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fa69005b170 con 0x7fa6a00fdd90 2024-02-28T17:08:30.706 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:08:30.707 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:08:30.708 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68406d7f0 msgr2=0x7fa68406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68406d7f0 0x7fa68406fca0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa6a01112a0 tx=0x7fa68c00b040 comp rx=0 tx=0).stop 2024-02-28T17:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 msgr2=0x7fa6a0115a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 0x7fa6a0115a70 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa690000c00 tx=0x7fa690009e30 comp rx=0 tx=0).stop 2024-02-28T17:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 shutdown_connections 2024-02-28T17:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa68406d7f0 0x7fa68406fca0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6a00fdd90 0x7fa6a0115a70 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 --2- 172.21.15.1:0/3171526447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6a00fe730 0x7fa6a0115fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:08:30.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 >> 172.21.15.1:0/3171526447 conn(0x7fa6a0073ab0 msgr2=0x7fa6a0103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:08:30.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.706+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 shutdown_connections 2024-02-28T17:08:30.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:08:30.707+0000 7fa6a5c00700 1 -- 172.21.15.1:0/3171526447 wait complete. 2024-02-28T17:08:30.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:30 smithi066 ceph-mon[85781]: pgmap v1022: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:30.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1014377217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:08:30.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2616947981' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:08:31.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:31 smithi066 ceph-mon[85781]: from='client.35842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:08:31.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:31 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3171526447' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:08:31.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:31 smithi001 ceph-mon[112940]: from='client.35842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:08:31.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:31 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3171526447' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:08:32.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:32 smithi066 ceph-mon[85781]: pgmap v1023: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:32.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:32 smithi001 ceph-mon[112940]: pgmap v1023: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:34.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:34 smithi066 ceph-mon[85781]: pgmap v1024: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:34.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:34 smithi001 ceph-mon[112940]: pgmap v1024: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:36.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:36 smithi066 ceph-mon[85781]: pgmap v1025: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:36.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:36 smithi001 ceph-mon[112940]: pgmap v1025: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:38.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:38 smithi066 ceph-mon[85781]: pgmap v1026: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:38.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:38 smithi001 ceph-mon[112940]: pgmap v1026: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:40.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:40 smithi066 ceph-mon[85781]: pgmap v1027: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:40.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:40 smithi001 ceph-mon[112940]: pgmap v1027: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:41.663 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:08:41.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:08:42.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:42 smithi066 ceph-mon[85781]: pgmap v1028: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:42.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:42 smithi001 ceph-mon[112940]: pgmap v1028: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:44.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:44 smithi066 ceph-mon[85781]: pgmap v1029: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:44.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:44 smithi001 ceph-mon[112940]: pgmap v1029: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:46.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:46 smithi066 ceph-mon[85781]: pgmap v1030: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:46.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:46 smithi001 ceph-mon[112940]: pgmap v1030: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:48.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:48 smithi066 ceph-mon[85781]: pgmap v1031: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:48.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:48 smithi001 ceph-mon[112940]: pgmap v1031: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:50.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:50 smithi066 ceph-mon[85781]: pgmap v1032: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:50.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:50 smithi001 ceph-mon[112940]: pgmap v1032: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:52.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:52 smithi066 ceph-mon[85781]: pgmap v1033: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:52.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:52 smithi001 ceph-mon[112940]: pgmap v1033: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:54.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:54 smithi066 ceph-mon[85781]: pgmap v1034: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:54.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:54 smithi001 ceph-mon[112940]: pgmap v1034: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:56.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:56 smithi066 ceph-mon[85781]: pgmap v1035: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:56.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:08:56.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:56 smithi001 ceph-mon[112940]: pgmap v1035: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:56.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:08:58.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:08:58 smithi066 ceph-mon[85781]: pgmap v1036: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:08:58.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:08:58 smithi001 ceph-mon[112940]: pgmap v1036: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:00.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:00 smithi066 ceph-mon[85781]: pgmap v1037: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:00.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:00 smithi001 ceph-mon[112940]: pgmap v1037: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:00.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.879+0000 7feac7e00700 1 -- 172.21.15.1:0/1003505756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0065760 msgr2=0x7feac0065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:00.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.879+0000 7feac7e00700 1 --2- 172.21.15.1:0/1003505756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0065760 0x7feac0065bd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7feaac004940 tx=0x7feaac009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:00.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.879+0000 7feac7e00700 1 -- 172.21.15.1:0/1003505756 shutdown_connections 2024-02-28T17:09:00.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.879+0000 7feac7e00700 1 --2- 172.21.15.1:0/1003505756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0065760 0x7feac0065bd0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:00.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.879+0000 7feac7e00700 1 --2- 172.21.15.1:0/1003505756 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feac0102f30 0x7feac0103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:00.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.879+0000 7feac7e00700 1 -- 172.21.15.1:0/1003505756 >> 172.21.15.1:0/1003505756 conn(0x7feac0071d50 msgr2=0x7feac0072150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:00.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.880+0000 7feac7e00700 1 -- 172.21.15.1:0/1003505756 shutdown_connections 2024-02-28T17:09:00.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.880+0000 7feac7e00700 1 -- 172.21.15.1:0/1003505756 wait complete. 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.880+0000 7feac7e00700 1 Processor -- start 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.880+0000 7feac7e00700 1 -- start start 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.881+0000 7feac7e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feac0065760 0x7feac0115b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.881+0000 7feac7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0102f30 0x7feac0116080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.881+0000 7feac7e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0116760 con 0x7feac0102f30 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.881+0000 7feac7e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac010fa00 con 0x7feac0065760 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.881+0000 7feac4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0102f30 0x7feac0116080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.881+0000 7feac4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0102f30 0x7feac0116080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57338/0 (socket says 172.21.15.1:57338) 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.881+0000 7feac4a00700 1 -- 172.21.15.1:0/3483131933 learned_addr learned my addr 172.21.15.1:0/3483131933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:00.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.881+0000 7feac5400700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feac0065760 0x7feac0115b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:00.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.882+0000 7feac4a00700 1 -- 172.21.15.1:0/3483131933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feac0065760 msgr2=0x7feac0115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:00.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.882+0000 7feac4a00700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feac0065760 0x7feac0115b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.882+0000 7feac4a00700 1 -- 172.21.15.1:0/3483131933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac005040 con 0x7feac0102f30 2024-02-28T17:09:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.882+0000 7feac4a00700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0102f30 0x7feac0116080 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7feaac009150 tx=0x7feaac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.882+0000 7feabd600700 1 -- 172.21.15.1:0/3483131933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaac00e040 con 0x7feac0102f30 2024-02-28T17:09:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.882+0000 7feabd600700 1 -- 172.21.15.1:0/3483131933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feaac00bbf0 con 0x7feac0102f30 2024-02-28T17:09:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.882+0000 7feabd600700 1 -- 172.21.15.1:0/3483131933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaac01c630 con 0x7feac0102f30 2024-02-28T17:09:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.882+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac010fc80 con 0x7feac0102f30 2024-02-28T17:09:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.883+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac0110240 con 0x7feac0102f30 2024-02-28T17:09:00.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.884+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac0075ca0 con 0x7feac0102f30 2024-02-28T17:09:00.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.885+0000 7feabd600700 1 -- 172.21.15.1:0/3483131933 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7feaac00bd60 con 0x7feac0102f30 2024-02-28T17:09:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.886+0000 7feabd600700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feaa806d7f0 0x7feaa806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:00.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.886+0000 7feabd600700 1 -- 172.21.15.1:0/3483131933 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7feaac08e480 con 0x7feac0102f30 2024-02-28T17:09:00.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.886+0000 7feac5400700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feaa806d7f0 0x7feaa806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:00.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.887+0000 7feac5400700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feaa806d7f0 0x7feaa806fca0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feab4006fd0 tx=0x7feab4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:00.890+0000 7feabd600700 1 -- 172.21.15.1:0/3483131933 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feaac05b860 con 0x7feac0102f30 2024-02-28T17:09:01.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.135+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feac0115810 con 0x7feaa806d7f0 2024-02-28T17:09:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.137+0000 7feabd600700 1 -- 172.21.15.1:0/3483131933 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7feac0115810 con 0x7feaa806d7f0 2024-02-28T17:09:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feaa806d7f0 msgr2=0x7feaa806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feaa806d7f0 0x7feaa806fca0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feab4006fd0 tx=0x7feab4008040 comp rx=0 tx=0).stop 2024-02-28T17:09:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0102f30 msgr2=0x7feac0116080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0102f30 0x7feac0116080 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7feaac009150 tx=0x7feaac009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 shutdown_connections 2024-02-28T17:09:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feaa806d7f0 0x7feaa806fca0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feac0065760 0x7feac0115b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 --2- 172.21.15.1:0/3483131933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feac0102f30 0x7feac0116080 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.140+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 >> 172.21.15.1:0/3483131933 conn(0x7feac0071d50 msgr2=0x7feac00fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.141+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 shutdown_connections 2024-02-28T17:09:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.141+0000 7feac7e00700 1 -- 172.21.15.1:0/3483131933 wait complete. 2024-02-28T17:09:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:09:01.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.297+0000 7f28d6200700 1 -- 172.21.15.1:0/3356543574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0070870 msgr2=0x7f28d0070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.297+0000 7f28d6200700 1 --2- 172.21.15.1:0/3356543574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0070870 0x7f28d0070ce0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f28bc004970 tx=0x7f28bc009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:01.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.298+0000 7f28d6200700 1 -- 172.21.15.1:0/3356543574 shutdown_connections 2024-02-28T17:09:01.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.298+0000 7f28d6200700 1 --2- 172.21.15.1:0/3356543574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0070870 0x7f28d0070ce0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.298+0000 7f28d6200700 1 --2- 172.21.15.1:0/3356543574 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28d0071f60 0x7f28d0070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.298+0000 7f28d6200700 1 -- 172.21.15.1:0/3356543574 >> 172.21.15.1:0/3356543574 conn(0x7f28d00f8af0 msgr2=0x7f28d00faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:01.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.298+0000 7f28d6200700 1 -- 172.21.15.1:0/3356543574 shutdown_connections 2024-02-28T17:09:01.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.298+0000 7f28d6200700 1 -- 172.21.15.1:0/3356543574 wait complete. 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.299+0000 7f28d6200700 1 Processor -- start 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.299+0000 7f28d6200700 1 -- start start 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.299+0000 7f28d6200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28d0070870 0x7f28d011a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.299+0000 7f28d6200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0071f60 0x7f28d011a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.299+0000 7f28d6200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d011ac20 con 0x7f28d0071f60 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.299+0000 7f28d6200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d0113f30 con 0x7f28d0070870 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.300+0000 7f28cea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0071f60 0x7f28d011a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.300+0000 7f28cea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0071f60 0x7f28d011a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57352/0 (socket says 172.21.15.1:57352) 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.300+0000 7f28cea00700 1 -- 172.21.15.1:0/820939653 learned_addr learned my addr 172.21.15.1:0/820939653 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.300+0000 7f28cf400700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28d0070870 0x7f28d011a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.300+0000 7f28cea00700 1 -- 172.21.15.1:0/820939653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28d0070870 msgr2=0x7f28d011a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.300+0000 7f28cea00700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28d0070870 0x7f28d011a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.300+0000 7f28cea00700 1 -- 172.21.15.1:0/820939653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc005040 con 0x7f28d0071f60 2024-02-28T17:09:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.300+0000 7f28cea00700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0071f60 0x7f28d011a5d0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009150 tx=0x7f28bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.301+0000 7f28c7400700 1 -- 172.21.15.1:0/820939653 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28bc00e040 con 0x7f28d0071f60 2024-02-28T17:09:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.301+0000 7f28c7400700 1 -- 172.21.15.1:0/820939653 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28bc00bc10 con 0x7f28d0071f60 2024-02-28T17:09:01.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.301+0000 7f28c7400700 1 -- 172.21.15.1:0/820939653 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28bc01c630 con 0x7f28d0071f60 2024-02-28T17:09:01.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.301+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d01141b0 con 0x7f28d0071f60 2024-02-28T17:09:01.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.301+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d01146f0 con 0x7f28d0071f60 2024-02-28T17:09:01.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.302+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d00fe410 con 0x7f28d0071f60 2024-02-28T17:09:01.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.303+0000 7f28c7400700 1 -- 172.21.15.1:0/820939653 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f28bc00bd80 con 0x7f28d0071f60 2024-02-28T17:09:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.304+0000 7f28c7400700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28b406d7f0 0x7f28b406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.305+0000 7f28c7400700 1 -- 172.21.15.1:0/820939653 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f28bc08e2e0 con 0x7f28d0071f60 2024-02-28T17:09:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.305+0000 7f28cf400700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28b406d7f0 0x7f28b406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:01.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.306+0000 7f28cf400700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28b406d7f0 0x7f28b406fca0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f28c0004d60 tx=0x7f28c000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:01.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.310+0000 7f28c7400700 1 -- 172.21.15.1:0/820939653 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f28bc05b6c0 con 0x7f28d0071f60 2024-02-28T17:09:01.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.559+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28d004bed0 con 0x7f28b406d7f0 2024-02-28T17:09:01.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.561+0000 7f28c7400700 1 -- 172.21.15.1:0/820939653 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f28d004bed0 con 0x7f28b406d7f0 2024-02-28T17:09:01.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28b406d7f0 msgr2=0x7f28b406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28b406d7f0 0x7f28b406fca0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f28c0004d60 tx=0x7f28c000b040 comp rx=0 tx=0).stop 2024-02-28T17:09:01.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0071f60 msgr2=0x7f28d011a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0071f60 0x7f28d011a5d0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009150 tx=0x7f28bc009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:01.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 shutdown_connections 2024-02-28T17:09:01.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f28b406d7f0 0x7f28b406fca0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f28d0070870 0x7f28d011a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 --2- 172.21.15.1:0/820939653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d0071f60 0x7f28d011a5d0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 >> 172.21.15.1:0/820939653 conn(0x7f28d00f8af0 msgr2=0x7f28d00fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:01.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 shutdown_connections 2024-02-28T17:09:01.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.564+0000 7f28d6200700 1 -- 172.21.15.1:0/820939653 wait complete. 2024-02-28T17:09:01.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.725+0000 7f2631400700 1 -- 172.21.15.1:0/2816244453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 msgr2=0x7f262c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.725+0000 7f2631400700 1 --2- 172.21.15.1:0/2816244453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 0x7f262c103320 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f261c004940 tx=0x7f261c009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:01.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.725+0000 7f2631400700 1 -- 172.21.15.1:0/2816244453 shutdown_connections 2024-02-28T17:09:01.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.725+0000 7f2631400700 1 --2- 172.21.15.1:0/2816244453 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f262c1038f0 0x7f262c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.725+0000 7f2631400700 1 --2- 172.21.15.1:0/2816244453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 0x7f262c103320 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.725+0000 7f2631400700 1 -- 172.21.15.1:0/2816244453 >> 172.21.15.1:0/2816244453 conn(0x7f262c0f8ae0 msgr2=0x7f262c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:01.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.725+0000 7f2631400700 1 -- 172.21.15.1:0/2816244453 shutdown_connections 2024-02-28T17:09:01.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.725+0000 7f2631400700 1 -- 172.21.15.1:0/2816244453 wait complete. 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.726+0000 7f2631400700 1 Processor -- start 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.726+0000 7f2631400700 1 -- start start 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.727+0000 7f2631400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 0x7f262c113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.727+0000 7f2631400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f262c1038f0 0x7f262c113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.727+0000 7f2631400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c1145f0 con 0x7f262c102f50 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.727+0000 7f2631400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c10d920 con 0x7f262c1038f0 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.727+0000 7f262a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 0x7f262c113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.727+0000 7f262a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 0x7f262c113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57372/0 (socket says 172.21.15.1:57372) 2024-02-28T17:09:01.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.727+0000 7f262a800700 1 -- 172.21.15.1:0/1614674001 learned_addr learned my addr 172.21.15.1:0/1614674001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f2629e00700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f262c1038f0 0x7f262c113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f262a800700 1 -- 172.21.15.1:0/1614674001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f262c1038f0 msgr2=0x7f262c113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f262a800700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f262c1038f0 0x7f262c113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f262a800700 1 -- 172.21.15.1:0/1614674001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c005040 con 0x7f262c102f50 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f2629e00700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f262c1038f0 0x7f262c113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f262a800700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 0x7f262c113a60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f261c009fd0 tx=0x7f261c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f2622a00700 1 -- 172.21.15.1:0/1614674001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f261c00e040 con 0x7f262c102f50 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f2622a00700 1 -- 172.21.15.1:0/1614674001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f261c00bbf0 con 0x7f262c102f50 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c10dba0 con 0x7f262c102f50 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f2622a00700 1 -- 172.21.15.1:0/1614674001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f261c01c3f0 con 0x7f262c102f50 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.728+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c10e0e0 con 0x7f262c102f50 2024-02-28T17:09:01.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.730+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c100260 con 0x7f262c102f50 2024-02-28T17:09:01.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.731+0000 7f2622a00700 1 -- 172.21.15.1:0/1614674001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f261c00bd60 con 0x7f262c102f50 2024-02-28T17:09:01.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.732+0000 7f2622a00700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f261006d7f0 0x7f261006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:01.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.732+0000 7f2622a00700 1 -- 172.21.15.1:0/1614674001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f261c08e210 con 0x7f262c102f50 2024-02-28T17:09:01.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.732+0000 7f2629e00700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f261006d7f0 0x7f261006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:01.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.733+0000 7f2629e00700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f261006d7f0 0x7f261006fca0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2618009fd0 tx=0x7f261800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:01.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.736+0000 7f2622a00700 1 -- 172.21.15.1:0/1614674001 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f261c05b610 con 0x7f262c102f50 2024-02-28T17:09:01.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.969+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f262c04bf40 con 0x7f261006d7f0 2024-02-28T17:09:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.993+0000 7f2622a00700 1 -- 172.21.15.1:0/1614674001 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f262c04bf40 con 0x7f261006d7f0 2024-02-28T17:09:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:09:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (35m) 11m ago 49m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:09:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (31m) 11m ago 49m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:09:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (31m) 11m ago 48m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:09:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (34m) 11m ago 49m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:09:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (27m) 11m ago 42m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (27m) 11m ago 42m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (42m) 11m ago 42m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (12m) 11m ago 42m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (37m) 11m ago 51m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (36m) 11m ago 48m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (33m) 11m ago 51m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (32m) 11m ago 47m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (36m) 11m ago 49m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (36m) 11m ago 48m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (31m) 11m ago 47m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (30m) 11m ago 46m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:09:01.996 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (30m) 11m ago 45m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:09:01.997 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (29m) 11m ago 45m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:09:01.997 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (28m) 11m ago 44m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:09:01.997 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (28m) 11m ago 44m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:09:01.997 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (35m) 11m ago 48m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:09:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f261006d7f0 msgr2=0x7f261006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f261006d7f0 0x7f261006fca0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2618009fd0 tx=0x7f261800b040 comp rx=0 tx=0).stop 2024-02-28T17:09:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 msgr2=0x7f262c113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 0x7f262c113a60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f261c009fd0 tx=0x7f261c009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 shutdown_connections 2024-02-28T17:09:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f261006d7f0 0x7f261006fca0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f262c102f50 0x7f262c113a60 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 --2- 172.21.15.1:0/1614674001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f262c1038f0 0x7f262c113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:01.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 >> 172.21.15.1:0/1614674001 conn(0x7f262c0f8ae0 msgr2=0x7f262c0fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:01.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 shutdown_connections 2024-02-28T17:09:01.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:01.996+0000 7f2631400700 1 -- 172.21.15.1:0/1614674001 wait complete. 2024-02-28T17:09:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.146+0000 7f1689600700 1 -- 172.21.15.1:0/2139300622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070330 msgr2=0x7f1684070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.146+0000 7f1689600700 1 --2- 172.21.15.1:0/2139300622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070330 0x7f1684070700 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1674004970 tx=0x7f1674009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.146+0000 7f1689600700 1 -- 172.21.15.1:0/2139300622 shutdown_connections 2024-02-28T17:09:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.146+0000 7f1689600700 1 --2- 172.21.15.1:0/2139300622 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684070c40 0x7f1684109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.146+0000 7f1689600700 1 --2- 172.21.15.1:0/2139300622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070330 0x7f1684070700 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.146+0000 7f1689600700 1 -- 172.21.15.1:0/2139300622 >> 172.21.15.1:0/2139300622 conn(0x7f16840f8ae0 msgr2=0x7f16840faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.146+0000 7f1689600700 1 -- 172.21.15.1:0/2139300622 shutdown_connections 2024-02-28T17:09:02.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.146+0000 7f1689600700 1 -- 172.21.15.1:0/2139300622 wait complete. 2024-02-28T17:09:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.147+0000 7f1689600700 1 Processor -- start 2024-02-28T17:09:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.147+0000 7f1689600700 1 -- start start 2024-02-28T17:09:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.148+0000 7f1689600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684070330 0x7f1684115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.148+0000 7f1689600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070c40 0x7f1684116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:02.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.148+0000 7f1689600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16841167b0 con 0x7f1684070c40 2024-02-28T17:09:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.148+0000 7f1689600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168410fae0 con 0x7f1684070330 2024-02-28T17:09:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.148+0000 7f1682000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070c40 0x7f1684116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.148+0000 7f1682000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070c40 0x7f1684116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40558/0 (socket says 172.21.15.1:40558) 2024-02-28T17:09:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.148+0000 7f1682000700 1 -- 172.21.15.1:0/3203034194 learned_addr learned my addr 172.21.15.1:0/3203034194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.148+0000 7f1682a00700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684070330 0x7f1684115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.149+0000 7f1682000700 1 -- 172.21.15.1:0/3203034194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684070330 msgr2=0x7f1684115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.149+0000 7f1682000700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684070330 0x7f1684115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.149+0000 7f1682000700 1 -- 172.21.15.1:0/3203034194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1674005040 con 0x7f1684070c40 2024-02-28T17:09:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.149+0000 7f1682000700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070c40 0x7f1684116160 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1670002a50 tx=0x7f16700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.149+0000 7f167aa00700 1 -- 172.21.15.1:0/3203034194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1670007cc0 con 0x7f1684070c40 2024-02-28T17:09:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.149+0000 7f167aa00700 1 -- 172.21.15.1:0/3203034194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f167000f040 con 0x7f1684070c40 2024-02-28T17:09:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.149+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168410fdc0 con 0x7f1684070c40 2024-02-28T17:09:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.149+0000 7f167aa00700 1 -- 172.21.15.1:0/3203034194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1670009640 con 0x7f1684070c40 2024-02-28T17:09:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.150+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1684110300 con 0x7f1684070c40 2024-02-28T17:09:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.151+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1684106920 con 0x7f1684070c40 2024-02-28T17:09:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.152+0000 7f167aa00700 1 -- 172.21.15.1:0/3203034194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f16700097a0 con 0x7f1684070c40 2024-02-28T17:09:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.153+0000 7f167aa00700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.153+0000 7f167aa00700 1 -- 172.21.15.1:0/3203034194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f167008cd20 con 0x7f1684070c40 2024-02-28T17:09:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.153+0000 7f1682a00700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.154+0000 7f1682a00700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1674004f50 tx=0x7f1674009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.158+0000 7f167aa00700 1 -- 172.21.15.1:0/3203034194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f167005a000 con 0x7f1684070c40 2024-02-28T17:09:02.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.466+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f168410c240 con 0x7f1684070c40 2024-02-28T17:09:02.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.467+0000 7f167aa00700 1 -- 172.21.15.1:0/3203034194 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f1670059b90 con 0x7f1684070c40 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:09:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:09:02.470 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:09:02.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 msgr2=0x7f166806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1674004f50 tx=0x7f1674009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070c40 msgr2=0x7f1684116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070c40 0x7f1684116160 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1670002a50 tx=0x7f16700046b0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 shutdown_connections 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f166806d7f0 0x7f166806fca0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1684070330 0x7f1684115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 --2- 172.21.15.1:0/3203034194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1684070c40 0x7f1684116160 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 >> 172.21.15.1:0/3203034194 conn(0x7f16840f8ae0 msgr2=0x7f1684103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 shutdown_connections 2024-02-28T17:09:02.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.470+0000 7f1689600700 1 -- 172.21.15.1:0/3203034194 wait complete. 2024-02-28T17:09:02.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.623+0000 7fcdb9200700 1 -- 172.21.15.1:0/2470826207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105170 msgr2=0x7fcdb4105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.623+0000 7fcdb9200700 1 --2- 172.21.15.1:0/2470826207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105170 0x7fcdb4105540 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fcda4004940 tx=0x7fcda4009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.624+0000 7fcdb9200700 1 -- 172.21.15.1:0/2470826207 shutdown_connections 2024-02-28T17:09:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.624+0000 7fcdb9200700 1 --2- 172.21.15.1:0/2470826207 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcdb4105b10 0x7fcdb40ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.624+0000 7fcdb9200700 1 --2- 172.21.15.1:0/2470826207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105170 0x7fcdb4105540 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.624+0000 7fcdb9200700 1 -- 172.21.15.1:0/2470826207 >> 172.21.15.1:0/2470826207 conn(0x7fcdb40face0 msgr2=0x7fcdb40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.624+0000 7fcdb9200700 1 -- 172.21.15.1:0/2470826207 shutdown_connections 2024-02-28T17:09:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.624+0000 7fcdb9200700 1 -- 172.21.15.1:0/2470826207 wait complete. 2024-02-28T17:09:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.625+0000 7fcdb9200700 1 Processor -- start 2024-02-28T17:09:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.625+0000 7fcdb9200700 1 -- start start 2024-02-28T17:09:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.626+0000 7fcdb9200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcdb4105170 0x7fcdb4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.626+0000 7fcdb9200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105b10 0x7fcdb41161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.626+0000 7fcdb9200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb41167f0 con 0x7fcdb4105b10 2024-02-28T17:09:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.626+0000 7fcdb9200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb410fb40 con 0x7fcdb4105170 2024-02-28T17:09:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.626+0000 7fcdb1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105b10 0x7fcdb41161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.626+0000 7fcdb1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105b10 0x7fcdb41161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40578/0 (socket says 172.21.15.1:40578) 2024-02-28T17:09:02.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.626+0000 7fcdb1400700 1 -- 172.21.15.1:0/3687135405 learned_addr learned my addr 172.21.15.1:0/3687135405 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.626+0000 7fcdb1e00700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcdb4105170 0x7fcdb4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.627+0000 7fcdb1400700 1 -- 172.21.15.1:0/3687135405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcdb4105170 msgr2=0x7fcdb4115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.627+0000 7fcdb1400700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcdb4105170 0x7fcdb4115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.627+0000 7fcdb1400700 1 -- 172.21.15.1:0/3687135405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4005040 con 0x7fcdb4105b10 2024-02-28T17:09:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.627+0000 7fcdb1400700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105b10 0x7fcdb41161a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fcda0002a50 tx=0x7fcda0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.627+0000 7fcdaa000700 1 -- 172.21.15.1:0/3687135405 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda0007cc0 con 0x7fcdb4105b10 2024-02-28T17:09:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.627+0000 7fcdaa000700 1 -- 172.21.15.1:0/3687135405 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcda000f040 con 0x7fcdb4105b10 2024-02-28T17:09:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.627+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb410fe20 con 0x7fcdb4105b10 2024-02-28T17:09:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.627+0000 7fcdaa000700 1 -- 172.21.15.1:0/3687135405 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda000daf0 con 0x7fcdb4105b10 2024-02-28T17:09:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.628+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb4110360 con 0x7fcdb4105b10 2024-02-28T17:09:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.629+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb4102480 con 0x7fcdb4105b10 2024-02-28T17:09:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.630+0000 7fcdaa000700 1 -- 172.21.15.1:0/3687135405 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcda0009810 con 0x7fcdb4105b10 2024-02-28T17:09:02.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.631+0000 7fcdaa000700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd9806d7f0 0x7fcd9806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:02.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.631+0000 7fcdaa000700 1 -- 172.21.15.1:0/3687135405 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcda008cb80 con 0x7fcdb4105b10 2024-02-28T17:09:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.631+0000 7fcdb1e00700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd9806d7f0 0x7fcd9806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.632+0000 7fcdb1e00700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd9806d7f0 0x7fcd9806fca0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcda400b020 tx=0x7fcda4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:02.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.638+0000 7fcdaa000700 1 -- 172.21.15.1:0/3687135405 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcda0056450 con 0x7fcdb4105b10 2024-02-28T17:09:02.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:02 smithi066 ceph-mon[85781]: from='client.35850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:02.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:02 smithi066 ceph-mon[85781]: pgmap v1038: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:02.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:02 smithi066 ceph-mon[85781]: from='client.35854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:02.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:02 smithi001 ceph-mon[112940]: from='client.35850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:02.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:02 smithi001 ceph-mon[112940]: pgmap v1038: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:02.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:02 smithi001 ceph-mon[112940]: from='client.35854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:02.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.900+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcdb404bed0 con 0x7fcdb4105b10 2024-02-28T17:09:02.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.900+0000 7fcdaa000700 1 -- 172.21.15.1:0/3687135405 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fcda0059b10 con 0x7fcdb4105b10 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:09:02.903 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:09:02.904 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:09:02.905 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:09:02.906 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:09:02.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd9806d7f0 msgr2=0x7fcd9806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:02.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd9806d7f0 0x7fcd9806fca0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcda400b020 tx=0x7fcda4009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:02.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105b10 msgr2=0x7fcdb41161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105b10 0x7fcdb41161a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fcda0002a50 tx=0x7fcda0004600 comp rx=0 tx=0).stop 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 shutdown_connections 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd9806d7f0 0x7fcd9806fca0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcdb4105170 0x7fcdb4115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 --2- 172.21.15.1:0/3687135405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb4105b10 0x7fcdb41161a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.903+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 >> 172.21.15.1:0/3687135405 conn(0x7fcdb40face0 msgr2=0x7fcdb40fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.904+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 shutdown_connections 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:02.904+0000 7fcdb9200700 1 -- 172.21.15.1:0/3687135405 wait complete. 2024-02-28T17:09:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:09:03.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.051+0000 7f88a4e00700 1 -- 172.21.15.1:0/2961842641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 msgr2=0x7f88a00fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:03.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.051+0000 7f88a4e00700 1 --2- 172.21.15.1:0/2961842641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 0x7f88a00fe160 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8890004940 tx=0x7f8890009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:03.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.051+0000 7f88a4e00700 1 -- 172.21.15.1:0/2961842641 shutdown_connections 2024-02-28T17:09:03.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.051+0000 7f88a4e00700 1 --2- 172.21.15.1:0/2961842641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88a00fe730 0x7f88a0106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.051+0000 7f88a4e00700 1 --2- 172.21.15.1:0/2961842641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 0x7f88a00fe160 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.051+0000 7f88a4e00700 1 -- 172.21.15.1:0/2961842641 >> 172.21.15.1:0/2961842641 conn(0x7f88a0073ab0 msgr2=0x7f88a0073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:03.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.051+0000 7f88a4e00700 1 -- 172.21.15.1:0/2961842641 shutdown_connections 2024-02-28T17:09:03.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.051+0000 7f88a4e00700 1 -- 172.21.15.1:0/2961842641 wait complete. 2024-02-28T17:09:03.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.052+0000 7f88a4e00700 1 Processor -- start 2024-02-28T17:09:03.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.053+0000 7f88a4e00700 1 -- start start 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.053+0000 7f88a4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 0x7f88a0113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.053+0000 7f88a4e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88a00fe730 0x7f88a0113dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.053+0000 7f88a4e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0114410 con 0x7f88a00fdd90 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.053+0000 7f88a4e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a010d740 con 0x7f88a00fe730 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.053+0000 7f889f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 0x7f88a0113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.053+0000 7f889f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 0x7f88a0113880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40596/0 (socket says 172.21.15.1:40596) 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.053+0000 7f889f400700 1 -- 172.21.15.1:0/3719730710 learned_addr learned my addr 172.21.15.1:0/3719730710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f889f400700 1 -- 172.21.15.1:0/3719730710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88a00fe730 msgr2=0x7f88a0113dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f889f400700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88a00fe730 0x7f88a0113dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f889f400700 1 -- 172.21.15.1:0/3719730710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890005040 con 0x7f88a00fdd90 2024-02-28T17:09:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f889f400700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 0x7f88a0113880 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8890009040 tx=0x7f8890009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f8897400700 1 -- 172.21.15.1:0/3719730710 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f889000e040 con 0x7f88a00fdd90 2024-02-28T17:09:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f8897400700 1 -- 172.21.15.1:0/3719730710 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f889000bb90 con 0x7f88a00fdd90 2024-02-28T17:09:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a010d9c0 con 0x7f88a00fdd90 2024-02-28T17:09:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f8897400700 1 -- 172.21.15.1:0/3719730710 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f889001c3f0 con 0x7f88a00fdd90 2024-02-28T17:09:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.054+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a010df00 con 0x7f88a00fdd90 2024-02-28T17:09:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.056+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a01044c0 con 0x7f88a00fdd90 2024-02-28T17:09:03.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.056+0000 7f8897400700 1 -- 172.21.15.1:0/3719730710 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f889000bd00 con 0x7f88a00fdd90 2024-02-28T17:09:03.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.057+0000 7f8897400700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f888406d7f0 0x7f888406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:03.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.057+0000 7f8897400700 1 -- 172.21.15.1:0/3719730710 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f889008e2d0 con 0x7f88a00fdd90 2024-02-28T17:09:03.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.058+0000 7f8897e00700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f888406d7f0 0x7f888406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:03.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.058+0000 7f8897e00700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f888406d7f0 0x7f888406fca0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f88a010f0b0 tx=0x7f8888009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:03.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.062+0000 7f8897400700 1 -- 172.21.15.1:0/3719730710 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f889005b5b0 con 0x7f88a00fdd90 2024-02-28T17:09:03.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.297+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f88a0109e70 con 0x7f888406d7f0 2024-02-28T17:09:03.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.300+0000 7f8897400700 1 -- 172.21.15.1:0/3719730710 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f88a0109e70 con 0x7f888406d7f0 2024-02-28T17:09:03.302 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:09:03.302 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:09:03.302 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:09:03.303 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:09:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f888406d7f0 msgr2=0x7f888406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f888406d7f0 0x7f888406fca0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f88a010f0b0 tx=0x7f8888009ce0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 msgr2=0x7f88a0113880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 0x7f88a0113880 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8890009040 tx=0x7f8890009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 shutdown_connections 2024-02-28T17:09:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f888406d7f0 0x7f888406fca0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a00fdd90 0x7f88a0113880 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 --2- 172.21.15.1:0/3719730710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f88a00fe730 0x7f88a0113dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.303+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 >> 172.21.15.1:0/3719730710 conn(0x7f88a0073ab0 msgr2=0x7f88a0101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.304+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 shutdown_connections 2024-02-28T17:09:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.304+0000 7f88a4e00700 1 -- 172.21.15.1:0/3719730710 wait complete. 2024-02-28T17:09:03.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.459+0000 7efd43e00700 1 -- 172.21.15.1:0/193623035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44104f80 msgr2=0x7efd44105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.459+0000 7efd43e00700 1 --2- 172.21.15.1:0/193623035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44104f80 0x7efd44105350 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7efd34004940 tx=0x7efd34009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.460+0000 7efd43e00700 1 -- 172.21.15.1:0/193623035 shutdown_connections 2024-02-28T17:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.460+0000 7efd43e00700 1 --2- 172.21.15.1:0/193623035 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd44105920 0x7efd440fef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.460+0000 7efd43e00700 1 --2- 172.21.15.1:0/193623035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44104f80 0x7efd44105350 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.460+0000 7efd43e00700 1 -- 172.21.15.1:0/193623035 >> 172.21.15.1:0/193623035 conn(0x7efd440fab10 msgr2=0x7efd440fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.460+0000 7efd43e00700 1 -- 172.21.15.1:0/193623035 shutdown_connections 2024-02-28T17:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.460+0000 7efd43e00700 1 -- 172.21.15.1:0/193623035 wait complete. 2024-02-28T17:09:03.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.461+0000 7efd43e00700 1 Processor -- start 2024-02-28T17:09:03.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.461+0000 7efd43e00700 1 -- start start 2024-02-28T17:09:03.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.461+0000 7efd43e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd44104f80 0x7efd44115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:03.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.461+0000 7efd43e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44105920 0x7efd44115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:03.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.461+0000 7efd43e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd441165b0 con 0x7efd44105920 2024-02-28T17:09:03.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.461+0000 7efd43e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4410f8e0 con 0x7efd44104f80 2024-02-28T17:09:03.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.462+0000 7efd42000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44105920 0x7efd44115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:03.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.462+0000 7efd42000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44105920 0x7efd44115f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40618/0 (socket says 172.21.15.1:40618) 2024-02-28T17:09:03.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.462+0000 7efd42000700 1 -- 172.21.15.1:0/2077751820 learned_addr learned my addr 172.21.15.1:0/2077751820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:03.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.462+0000 7efd42a00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd44104f80 0x7efd44115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.462+0000 7efd42000700 1 -- 172.21.15.1:0/2077751820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd44104f80 msgr2=0x7efd44115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.462+0000 7efd42000700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd44104f80 0x7efd44115a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.462+0000 7efd42000700 1 -- 172.21.15.1:0/2077751820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd34005040 con 0x7efd44105920 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.463+0000 7efd42000700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44105920 0x7efd44115f60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7efd30002a50 tx=0x7efd30004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.463+0000 7efd3aa00700 1 -- 172.21.15.1:0/2077751820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd30007cc0 con 0x7efd44105920 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.463+0000 7efd3aa00700 1 -- 172.21.15.1:0/2077751820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd3000f040 con 0x7efd44105920 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.463+0000 7efd3aa00700 1 -- 172.21.15.1:0/2077751820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3000daf0 con 0x7efd44105920 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.463+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4410fbc0 con 0x7efd44105920 2024-02-28T17:09:03.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.463+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd44110180 con 0x7efd44105920 2024-02-28T17:09:03.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.464+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd4405f5c0 con 0x7efd44105920 2024-02-28T17:09:03.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.465+0000 7efd3aa00700 1 -- 172.21.15.1:0/2077751820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7efd30009810 con 0x7efd44105920 2024-02-28T17:09:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.466+0000 7efd3aa00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efd2806d7f0 0x7efd2806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.466+0000 7efd3aa00700 1 -- 172.21.15.1:0/2077751820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efd3008cc20 con 0x7efd44105920 2024-02-28T17:09:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.466+0000 7efd42a00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efd2806d7f0 0x7efd2806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.467+0000 7efd42a00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efd2806d7f0 0x7efd2806fca0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b020 tx=0x7efd34009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.470+0000 7efd3aa00700 1 -- 172.21.15.1:0/2077751820 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd30059f80 con 0x7efd44105920 2024-02-28T17:09:03.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:03 smithi066 ceph-mon[85781]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:03.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:03 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3203034194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:09:03.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:03 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3687135405' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:09:03.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:03 smithi001 ceph-mon[112940]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:03.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:03 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3203034194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:09:03.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:03 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3687135405' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:09:03.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.784+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efd4404bed0 con 0x7efd44105920 2024-02-28T17:09:03.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.785+0000 7efd3aa00700 1 -- 172.21.15.1:0/2077751820 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7efd30059b10 con 0x7efd44105920 2024-02-28T17:09:03.787 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:09:03.787 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:09:03.787 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:09:03.788 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:09:03.789 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efd2806d7f0 msgr2=0x7efd2806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efd2806d7f0 0x7efd2806fca0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b020 tx=0x7efd34009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44105920 msgr2=0x7efd44115f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44105920 0x7efd44115f60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7efd30002a50 tx=0x7efd30004600 comp rx=0 tx=0).stop 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 shutdown_connections 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efd2806d7f0 0x7efd2806fca0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efd44104f80 0x7efd44115a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 --2- 172.21.15.1:0/2077751820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd44105920 0x7efd44115f60 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.788+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 >> 172.21.15.1:0/2077751820 conn(0x7efd440fab10 msgr2=0x7efd440fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:03.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.789+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 shutdown_connections 2024-02-28T17:09:03.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:03.789+0000 7efd43e00700 1 -- 172.21.15.1:0/2077751820 wait complete. 2024-02-28T17:09:04.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:04 smithi066 ceph-mon[85781]: pgmap v1039: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:04.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:04 smithi066 ceph-mon[85781]: from='client.35870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:04.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:04 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2077751820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:09:04.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:04 smithi001 ceph-mon[112940]: pgmap v1039: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:04.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:04 smithi001 ceph-mon[112940]: from='client.35870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:04.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:04 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2077751820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:09:06.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:06 smithi066 ceph-mon[85781]: pgmap v1040: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:06.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:06 smithi001 ceph-mon[112940]: pgmap v1040: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:08.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:08 smithi066 ceph-mon[85781]: pgmap v1041: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:08.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:08 smithi001 ceph-mon[112940]: pgmap v1041: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:10.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:10 smithi066 ceph-mon[85781]: pgmap v1042: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:10.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:10 smithi001 ceph-mon[112940]: pgmap v1042: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:11.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:09:11.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:09:12.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:12 smithi066 ceph-mon[85781]: pgmap v1043: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:12.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:12 smithi001 ceph-mon[112940]: pgmap v1043: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:14.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:14 smithi066 ceph-mon[85781]: pgmap v1044: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:14.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:14 smithi001 ceph-mon[112940]: pgmap v1044: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:16.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:16 smithi066 ceph-mon[85781]: pgmap v1045: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:16.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:16 smithi001 ceph-mon[112940]: pgmap v1045: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:18.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:18 smithi001 ceph-mon[112940]: pgmap v1046: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:19.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:18 smithi066 ceph-mon[85781]: pgmap v1046: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:20.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:20 smithi001 ceph-mon[112940]: pgmap v1047: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:21.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:20 smithi066 ceph-mon[85781]: pgmap v1047: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:22.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:22 smithi001 ceph-mon[112940]: pgmap v1048: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:23.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:22 smithi066 ceph-mon[85781]: pgmap v1048: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:24.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:24 smithi001 ceph-mon[112940]: pgmap v1049: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:25.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:24 smithi066 ceph-mon[85781]: pgmap v1049: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:27.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:26 smithi066 ceph-mon[85781]: pgmap v1050: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:27.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:09:27.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:26 smithi001 ceph-mon[112940]: pgmap v1050: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:27.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:09:29.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:28 smithi066 ceph-mon[85781]: pgmap v1051: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:29.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:28 smithi001 ceph-mon[112940]: pgmap v1051: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:31.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:30 smithi066 ceph-mon[85781]: pgmap v1052: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:31.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:30 smithi001 ceph-mon[112940]: pgmap v1052: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:33.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:32 smithi066 ceph-mon[85781]: pgmap v1053: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:33.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:32 smithi001 ceph-mon[112940]: pgmap v1053: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:33.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.958+0000 7f0786200700 1 -- 172.21.15.1:0/3663376230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fdf60 msgr2=0x7f07800fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:33.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.958+0000 7f0786200700 1 --2- 172.21.15.1:0/3663376230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fdf60 0x7f07800fe330 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0770004940 tx=0x7f0770009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:33.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.959+0000 7f0786200700 1 -- 172.21.15.1:0/3663376230 shutdown_connections 2024-02-28T17:09:33.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.959+0000 7f0786200700 1 --2- 172.21.15.1:0/3663376230 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07800fe900 0x7f0780106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:33.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.959+0000 7f0786200700 1 --2- 172.21.15.1:0/3663376230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fdf60 0x7f07800fe330 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:33.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.959+0000 7f0786200700 1 -- 172.21.15.1:0/3663376230 >> 172.21.15.1:0/3663376230 conn(0x7f07800f7a10 msgr2=0x7f07800f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:33.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.959+0000 7f0786200700 1 -- 172.21.15.1:0/3663376230 shutdown_connections 2024-02-28T17:09:33.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.959+0000 7f0786200700 1 -- 172.21.15.1:0/3663376230 wait complete. 2024-02-28T17:09:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.960+0000 7f0786200700 1 Processor -- start 2024-02-28T17:09:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.960+0000 7f0786200700 1 -- start start 2024-02-28T17:09:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.960+0000 7f0786200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07800fdf60 0x7f0780115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.960+0000 7f0786200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fe900 0x7f07801161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.960+0000 7f0786200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0780116810 con 0x7f07800fe900 2024-02-28T17:09:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.960+0000 7f0786200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078010fb40 con 0x7f07800fdf60 2024-02-28T17:09:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.960+0000 7f0777e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fe900 0x7f07801161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:33.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.961+0000 7f0777e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fe900 0x7f07801161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60696/0 (socket says 172.21.15.1:60696) 2024-02-28T17:09:33.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.961+0000 7f0777e00700 1 -- 172.21.15.1:0/695699194 learned_addr learned my addr 172.21.15.1:0/695699194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:33.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.961+0000 7f077f400700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07800fdf60 0x7f0780115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.961+0000 7f0777e00700 1 -- 172.21.15.1:0/695699194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07800fdf60 msgr2=0x7f0780115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.961+0000 7f0777e00700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07800fdf60 0x7f0780115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.961+0000 7f0777e00700 1 -- 172.21.15.1:0/695699194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0770005040 con 0x7f07800fe900 2024-02-28T17:09:33.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.961+0000 7f0777e00700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fe900 0x7f07801161c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f076800c830 tx=0x7f0768004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:33.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.962+0000 7f0777400700 1 -- 172.21.15.1:0/695699194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f076800a760 con 0x7f07800fe900 2024-02-28T17:09:33.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.962+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078010fe20 con 0x7f07800fe900 2024-02-28T17:09:33.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.962+0000 7f0777400700 1 -- 172.21.15.1:0/695699194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f076800ad90 con 0x7f07800fe900 2024-02-28T17:09:33.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.962+0000 7f0777400700 1 -- 172.21.15.1:0/695699194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07680113a0 con 0x7f07800fe900 2024-02-28T17:09:33.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.962+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0780110360 con 0x7f07800fe900 2024-02-28T17:09:33.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.963+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07801069b0 con 0x7f07800fe900 2024-02-28T17:09:33.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.964+0000 7f0777400700 1 -- 172.21.15.1:0/695699194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0768012400 con 0x7f07800fe900 2024-02-28T17:09:33.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.965+0000 7f0777400700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f076406d7e0 0x7f076406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:33.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.965+0000 7f0777400700 1 -- 172.21.15.1:0/695699194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f076808cf00 con 0x7f07800fe900 2024-02-28T17:09:33.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.966+0000 7f077f400700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f076406d7e0 0x7f076406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:33.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.966+0000 7f077f400700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f076406d7e0 0x7f076406fc90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0770004f90 tx=0x7f0770009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:33.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:33.970+0000 7f0777400700 1 -- 172.21.15.1:0/695699194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f076805a2e0 con 0x7f07800fe900 2024-02-28T17:09:34.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.210+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f078004bf40 con 0x7f076406d7e0 2024-02-28T17:09:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.213+0000 7f0777400700 1 -- 172.21.15.1:0/695699194 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f078004bf40 con 0x7f076406d7e0 2024-02-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.216+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f076406d7e0 msgr2=0x7f076406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.216+0000 7f0786200700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f076406d7e0 0x7f076406fc90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0770004f90 tx=0x7f0770009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.216+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fe900 msgr2=0x7f07801161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.216+0000 7f0786200700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fe900 0x7f07801161c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f076800c830 tx=0x7f0768004470 comp rx=0 tx=0).stop 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.217+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 shutdown_connections 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.217+0000 7f0786200700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f076406d7e0 0x7f076406fc90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.217+0000 7f0786200700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07800fdf60 0x7f0780115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.217+0000 7f0786200700 1 --2- 172.21.15.1:0/695699194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07800fe900 0x7f07801161c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.217+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 >> 172.21.15.1:0/695699194 conn(0x7f07800f7a10 msgr2=0x7f07800fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.217+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 shutdown_connections 2024-02-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.217+0000 7f0786200700 1 -- 172.21.15.1:0/695699194 wait complete. 2024-02-28T17:09:34.236 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:09:34.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.375+0000 7f275c400700 1 -- 172.21.15.1:0/695562170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 msgr2=0x7f27540fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.375+0000 7f275c400700 1 --2- 172.21.15.1:0/695562170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 0x7f27540fcf60 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2740004940 tx=0x7f2740009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.376+0000 7f275c400700 1 -- 172.21.15.1:0/695562170 shutdown_connections 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.376+0000 7f275c400700 1 --2- 172.21.15.1:0/695562170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 0x7f27540fcf60 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.376+0000 7f275c400700 1 --2- 172.21.15.1:0/695562170 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2754102f50 0x7f2754103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.376+0000 7f275c400700 1 -- 172.21.15.1:0/695562170 >> 172.21.15.1:0/695562170 conn(0x7f27540f8ae0 msgr2=0x7f27540faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.376+0000 7f275c400700 1 -- 172.21.15.1:0/695562170 shutdown_connections 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.376+0000 7f275c400700 1 -- 172.21.15.1:0/695562170 wait complete. 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.377+0000 7f275c400700 1 Processor -- start 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.377+0000 7f275c400700 1 -- start start 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.377+0000 7f275c400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2754102f50 0x7f2754115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.377+0000 7f275c400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 0x7f2754116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:34.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.377+0000 7f275c400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754116780 con 0x7f27541038f0 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.377+0000 7f275c400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275410fad0 con 0x7f2754102f50 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.378+0000 7f2759000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 0x7f2754116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.378+0000 7f2759000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 0x7f2754116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60712/0 (socket says 172.21.15.1:60712) 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.378+0000 7f2759000700 1 -- 172.21.15.1:0/2176417239 learned_addr learned my addr 172.21.15.1:0/2176417239 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.378+0000 7f2759a00700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2754102f50 0x7f2754115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.378+0000 7f2759000700 1 -- 172.21.15.1:0/2176417239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2754102f50 msgr2=0x7f2754115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.378+0000 7f2759000700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2754102f50 0x7f2754115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.378+0000 7f2759000700 1 -- 172.21.15.1:0/2176417239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2740005040 con 0x7f27541038f0 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.378+0000 7f2759000700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 0x7f2754116130 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2740009fd0 tx=0x7f2740009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.379+0000 7f2751600700 1 -- 172.21.15.1:0/2176417239 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2740021070 con 0x7f27541038f0 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.379+0000 7f2751600700 1 -- 172.21.15.1:0/2176417239 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2740013d00 con 0x7f27541038f0 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.379+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275410fd50 con 0x7f27541038f0 2024-02-28T17:09:34.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.379+0000 7f2751600700 1 -- 172.21.15.1:0/2176417239 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f274000e070 con 0x7f27541038f0 2024-02-28T17:09:34.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.379+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2754110230 con 0x7f27541038f0 2024-02-28T17:09:34.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.380+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27540fc6e0 con 0x7f27541038f0 2024-02-28T17:09:34.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.381+0000 7f2751600700 1 -- 172.21.15.1:0/2176417239 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f274001c400 con 0x7f27541038f0 2024-02-28T17:09:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.383+0000 7f2751600700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f273c06d7f0 0x7f273c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.383+0000 7f2751600700 1 -- 172.21.15.1:0/2176417239 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f274008e440 con 0x7f27541038f0 2024-02-28T17:09:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.383+0000 7f2759a00700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f273c06d7f0 0x7f273c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.384+0000 7f2759a00700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f273c06d7f0 0x7f273c06fca0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2748004a10 tx=0x7f274800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:34.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.387+0000 7f2751600700 1 -- 172.21.15.1:0/2176417239 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2740057cf0 con 0x7f27541038f0 2024-02-28T17:09:34.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.633+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27541111d0 con 0x7f273c06d7f0 2024-02-28T17:09:34.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.636+0000 7f2751600700 1 -- 172.21.15.1:0/2176417239 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f27541111d0 con 0x7f273c06d7f0 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.638+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f273c06d7f0 msgr2=0x7f273c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.638+0000 7f275c400700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f273c06d7f0 0x7f273c06fca0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2748004a10 tx=0x7f274800b040 comp rx=0 tx=0).stop 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.638+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 msgr2=0x7f2754116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.638+0000 7f275c400700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 0x7f2754116130 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2740009fd0 tx=0x7f2740009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.639+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 shutdown_connections 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.639+0000 7f275c400700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f273c06d7f0 0x7f273c06fca0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.639+0000 7f275c400700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2754102f50 0x7f2754115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.639+0000 7f275c400700 1 --2- 172.21.15.1:0/2176417239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27541038f0 0x7f2754116130 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.639+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 >> 172.21.15.1:0/2176417239 conn(0x7f27540f8ae0 msgr2=0x7f2754070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.639+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 shutdown_connections 2024-02-28T17:09:34.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.639+0000 7f275c400700 1 -- 172.21.15.1:0/2176417239 wait complete. 2024-02-28T17:09:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.795+0000 7f5ab7e00700 1 -- 172.21.15.1:0/1719059447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 msgr2=0x7f5ab80fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.795+0000 7f5ab7e00700 1 --2- 172.21.15.1:0/1719059447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 0x7f5ab80fcda0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0004940 tx=0x7f5aa0009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.795+0000 7f5ab7e00700 1 -- 172.21.15.1:0/1719059447 shutdown_connections 2024-02-28T17:09:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.795+0000 7f5ab7e00700 1 --2- 172.21.15.1:0/1719059447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 0x7f5ab80fcda0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.795+0000 7f5ab7e00700 1 --2- 172.21.15.1:0/1719059447 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ab8102d90 0x7f5ab8103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.795+0000 7f5ab7e00700 1 -- 172.21.15.1:0/1719059447 >> 172.21.15.1:0/1719059447 conn(0x7f5ab80f8940 msgr2=0x7f5ab80fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.795+0000 7f5ab7e00700 1 -- 172.21.15.1:0/1719059447 shutdown_connections 2024-02-28T17:09:34.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.795+0000 7f5ab7e00700 1 -- 172.21.15.1:0/1719059447 wait complete. 2024-02-28T17:09:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.796+0000 7f5ab7e00700 1 Processor -- start 2024-02-28T17:09:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.796+0000 7f5ab7e00700 1 -- start start 2024-02-28T17:09:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.796+0000 7f5ab7e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ab8102d90 0x7f5ab8115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.796+0000 7f5ab7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 0x7f5ab8115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.796+0000 7f5ab7e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab8116620 con 0x7f5ab8103730 2024-02-28T17:09:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.796+0000 7f5ab7e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab810f950 con 0x7f5ab8102d90 2024-02-28T17:09:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.797+0000 7f5aafe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 0x7f5ab8115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:34.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.797+0000 7f5aafe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 0x7f5ab8115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60732/0 (socket says 172.21.15.1:60732) 2024-02-28T17:09:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.797+0000 7f5aafe00700 1 -- 172.21.15.1:0/3890467289 learned_addr learned my addr 172.21.15.1:0/3890467289 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.797+0000 7f5aafe00700 1 -- 172.21.15.1:0/3890467289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ab8102d90 msgr2=0x7f5ab8115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.797+0000 7f5ab6a00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ab8102d90 0x7f5ab8115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.797+0000 7f5aafe00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ab8102d90 0x7f5ab8115a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.797+0000 7f5aafe00700 1 -- 172.21.15.1:0/3890467289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa0005040 con 0x7f5ab8103730 2024-02-28T17:09:34.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.797+0000 7f5aafe00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 0x7f5ab8115fd0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009030 tx=0x7f5aa0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.798+0000 7f5aaea00700 1 -- 172.21.15.1:0/3890467289 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa000e040 con 0x7f5ab8103730 2024-02-28T17:09:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.798+0000 7f5aaea00700 1 -- 172.21.15.1:0/3890467289 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aa000bbf0 con 0x7f5ab8103730 2024-02-28T17:09:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.798+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab810fbd0 con 0x7f5ab8103730 2024-02-28T17:09:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.798+0000 7f5aaea00700 1 -- 172.21.15.1:0/3890467289 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa001c630 con 0x7f5ab8103730 2024-02-28T17:09:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.798+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab8110190 con 0x7f5ab8103730 2024-02-28T17:09:34.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.799+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab805f5c0 con 0x7f5ab8103730 2024-02-28T17:09:34.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.800+0000 7f5aaea00700 1 -- 172.21.15.1:0/3890467289 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5aa000bd60 con 0x7f5ab8103730 2024-02-28T17:09:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.801+0000 7f5aaea00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a9c06d7f0 0x7f5a9c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.801+0000 7f5aaea00700 1 -- 172.21.15.1:0/3890467289 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5aa008e1f0 con 0x7f5ab8103730 2024-02-28T17:09:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.802+0000 7f5ab6a00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a9c06d7f0 0x7f5a9c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:34.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.802+0000 7f5ab6a00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a9c06d7f0 0x7f5a9c06fca0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8004d60 tx=0x7f5aa800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:34.806+0000 7f5aaea00700 1 -- 172.21.15.1:0/3890467289 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5aa005b5d0 con 0x7f5ab8103730 2024-02-28T17:09:35.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:34 smithi066 ceph-mon[85781]: pgmap v1054: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:35.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:34 smithi001 ceph-mon[112940]: pgmap v1054: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.050+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ab8115760 con 0x7f5a9c06d7f0 2024-02-28T17:09:35.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.069+0000 7f5aaea00700 1 -- 172.21.15.1:0/3890467289 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5ab8115760 con 0x7f5a9c06d7f0 2024-02-28T17:09:35.072 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:09:35.072 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (35m) 12m ago 49m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:09:35.072 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (32m) 12m ago 50m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:09:35.072 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (32m) 12m ago 48m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:09:35.072 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (35m) 12m ago 49m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:09:35.072 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (28m) 12m ago 42m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:09:35.072 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (27m) 12m ago 42m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (42m) 12m ago 42m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (12m) 12m ago 42m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (38m) 12m ago 51m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (37m) 12m ago 48m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (34m) 12m ago 51m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (33m) 12m ago 48m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (36m) 12m ago 50m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (36m) 12m ago 48m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (31m) 12m ago 47m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (31m) 12m ago 47m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (30m) 12m ago 46m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (30m) 12m ago 45m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (29m) 12m ago 45m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:09:35.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (28m) 12m ago 44m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:09:35.074 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (36m) 12m ago 49m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:09:35.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a9c06d7f0 msgr2=0x7f5a9c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a9c06d7f0 0x7f5a9c06fca0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8004d60 tx=0x7f5aa800b040 comp rx=0 tx=0).stop 2024-02-28T17:09:35.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 msgr2=0x7f5ab8115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 0x7f5ab8115fd0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009030 tx=0x7f5aa0009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:35.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 shutdown_connections 2024-02-28T17:09:35.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a9c06d7f0 0x7f5a9c06fca0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5ab8102d90 0x7f5ab8115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 --2- 172.21.15.1:0/3890467289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ab8103730 0x7f5ab8115fd0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 >> 172.21.15.1:0/3890467289 conn(0x7f5ab80f8940 msgr2=0x7f5ab80fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:35.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 shutdown_connections 2024-02-28T17:09:35.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.072+0000 7f5ab7e00700 1 -- 172.21.15.1:0/3890467289 wait complete. 2024-02-28T17:09:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.227+0000 7fb3a9400700 1 -- 172.21.15.1:0/3181580299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fdfe0 msgr2=0x7fb3a40fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.227+0000 7fb3a9400700 1 --2- 172.21.15.1:0/3181580299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fdfe0 0x7fb3a40fe3b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb394004970 tx=0x7fb394009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.228+0000 7fb3a9400700 1 -- 172.21.15.1:0/3181580299 shutdown_connections 2024-02-28T17:09:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.228+0000 7fb3a9400700 1 --2- 172.21.15.1:0/3181580299 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3a40fe8f0 0x7fb3a4107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.228+0000 7fb3a9400700 1 --2- 172.21.15.1:0/3181580299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fdfe0 0x7fb3a40fe3b0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.228+0000 7fb3a9400700 1 -- 172.21.15.1:0/3181580299 >> 172.21.15.1:0/3181580299 conn(0x7fb3a4071d90 msgr2=0x7fb3a4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.228+0000 7fb3a9400700 1 -- 172.21.15.1:0/3181580299 shutdown_connections 2024-02-28T17:09:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.228+0000 7fb3a9400700 1 -- 172.21.15.1:0/3181580299 wait complete. 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.229+0000 7fb3a9400700 1 Processor -- start 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.229+0000 7fb3a9400700 1 -- start start 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.229+0000 7fb3a9400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3a40fdfe0 0x7fb3a40fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.229+0000 7fb3a9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fe8f0 0x7fb3a40fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.229+0000 7fb3a9400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40fc9e0 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.229+0000 7fb3a9400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40fcb50 con 0x7fb3a40fdfe0 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.230+0000 7fb399e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fe8f0 0x7fb3a40fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.230+0000 7fb399e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fe8f0 0x7fb3a40fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60750/0 (socket says 172.21.15.1:60750) 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.230+0000 7fb399e00700 1 -- 172.21.15.1:0/1392534034 learned_addr learned my addr 172.21.15.1:0/1392534034 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.230+0000 7fb3a2800700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3a40fdfe0 0x7fb3a40fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.230+0000 7fb399e00700 1 -- 172.21.15.1:0/1392534034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3a40fdfe0 msgr2=0x7fb3a40fbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.230+0000 7fb399e00700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3a40fdfe0 0x7fb3a40fbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.230+0000 7fb399e00700 1 -- 172.21.15.1:0/1392534034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394005040 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.231+0000 7fb399e00700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fe8f0 0x7fb3a40fc3e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00c830 tx=0x7fb38c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:35.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.231+0000 7fb39b400700 1 -- 172.21.15.1:0/1392534034 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb38c00a760 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.231+0000 7fb39b400700 1 -- 172.21.15.1:0/1392534034 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb38c00ad90 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.231+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a4119c90 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.231+0000 7fb39b400700 1 -- 172.21.15.1:0/1392534034 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb38c0113a0 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.231+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a411a170 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.232+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a41046c0 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.233+0000 7fb39b400700 1 -- 172.21.15.1:0/1392534034 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb38c012400 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.234+0000 7fb39b400700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb38806d7f0 0x7fb38806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.234+0000 7fb39b400700 1 -- 172.21.15.1:0/1392534034 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb38c08cef0 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.234+0000 7fb3a2800700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb38806d7f0 0x7fb38806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.235+0000 7fb3a2800700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb38806d7f0 0x7fb38806fca0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb394004ee0 tx=0x7fb394009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:35.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.238+0000 7fb39b400700 1 -- 172.21.15.1:0/1392534034 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb38c0567a0 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.538+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb3a40738b0 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.539+0000 7fb39b400700 1 -- 172.21.15.1:0/1392534034 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fb38c059e60 con 0x7fb3a40fe8f0 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:09:35.542 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:09:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:09:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:09:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:09:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:09:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:09:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:09:35.543 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:09:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.541+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb38806d7f0 msgr2=0x7fb38806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.541+0000 7fb3a9400700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb38806d7f0 0x7fb38806fca0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb394004ee0 tx=0x7fb394009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.541+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fe8f0 msgr2=0x7fb3a40fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.541+0000 7fb3a9400700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fe8f0 0x7fb3a40fc3e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00c830 tx=0x7fb38c004470 comp rx=0 tx=0).stop 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.541+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 shutdown_connections 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.542+0000 7fb3a9400700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb38806d7f0 0x7fb38806fca0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.542+0000 7fb3a9400700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb3a40fdfe0 0x7fb3a40fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.542+0000 7fb3a9400700 1 --2- 172.21.15.1:0/1392534034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40fe8f0 0x7fb3a40fc3e0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.542+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 >> 172.21.15.1:0/1392534034 conn(0x7fb3a4071d90 msgr2=0x7fb3a4101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.542+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 shutdown_connections 2024-02-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.542+0000 7fb3a9400700 1 -- 172.21.15.1:0/1392534034 wait complete. 2024-02-28T17:09:35.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.692+0000 7f744a600700 1 -- 172.21.15.1:0/418236559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444104f70 msgr2=0x7f7444105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.692+0000 7f744a600700 1 --2- 172.21.15.1:0/418236559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444104f70 0x7f7444105340 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7438004940 tx=0x7f7438009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:35.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.692+0000 7f744a600700 1 -- 172.21.15.1:0/418236559 shutdown_connections 2024-02-28T17:09:35.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.692+0000 7f744a600700 1 --2- 172.21.15.1:0/418236559 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7444105910 0x7f74440fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.692+0000 7f744a600700 1 --2- 172.21.15.1:0/418236559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444104f70 0x7f7444105340 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.692+0000 7f744a600700 1 -- 172.21.15.1:0/418236559 >> 172.21.15.1:0/418236559 conn(0x7f74440faae0 msgr2=0x7f74440fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.692+0000 7f744a600700 1 -- 172.21.15.1:0/418236559 shutdown_connections 2024-02-28T17:09:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.692+0000 7f744a600700 1 -- 172.21.15.1:0/418236559 wait complete. 2024-02-28T17:09:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.693+0000 7f744a600700 1 Processor -- start 2024-02-28T17:09:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.693+0000 7f744a600700 1 -- start start 2024-02-28T17:09:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f744a600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7444104f70 0x7f7444115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f744a600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444105910 0x7f7444115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f744a600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74441165b0 con 0x7f7444105910 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f744a600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744410f8e0 con 0x7f7444104f70 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f7443e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444105910 0x7f7444115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f7443e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444105910 0x7f7444115f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60770/0 (socket says 172.21.15.1:60770) 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f7443e00700 1 -- 172.21.15.1:0/2874326369 learned_addr learned my addr 172.21.15.1:0/2874326369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f7449200700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7444104f70 0x7f7444115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f7443e00700 1 -- 172.21.15.1:0/2874326369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7444104f70 msgr2=0x7f7444115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f7443e00700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7444104f70 0x7f7444115a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.694+0000 7f7443e00700 1 -- 172.21.15.1:0/2874326369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7438005040 con 0x7f7444105910 2024-02-28T17:09:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.695+0000 7f7443e00700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444105910 0x7f7444115f60 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f743000c890 tx=0x7f74300044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.695+0000 7f7440c00700 1 -- 172.21.15.1:0/2874326369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f743000a7d0 con 0x7f7444105910 2024-02-28T17:09:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.695+0000 7f7440c00700 1 -- 172.21.15.1:0/2874326369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f743000ae00 con 0x7f7444105910 2024-02-28T17:09:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.695+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744410fbc0 con 0x7f7444105910 2024-02-28T17:09:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.695+0000 7f7440c00700 1 -- 172.21.15.1:0/2874326369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74300113a0 con 0x7f7444105910 2024-02-28T17:09:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.695+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7444110180 con 0x7f7444105910 2024-02-28T17:09:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.696+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744405f5c0 con 0x7f7444105910 2024-02-28T17:09:35.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.697+0000 7f7440c00700 1 -- 172.21.15.1:0/2874326369 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7430012400 con 0x7f7444105910 2024-02-28T17:09:35.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.699+0000 7f7440c00700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f742c06d7f0 0x7f742c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:35.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.699+0000 7f7440c00700 1 -- 172.21.15.1:0/2874326369 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f743008cfe0 con 0x7f7444105910 2024-02-28T17:09:35.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.699+0000 7f7449200700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f742c06d7f0 0x7f742c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:35.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.700+0000 7f7449200700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f742c06d7f0 0x7f742c06fca0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f743800b020 tx=0x7f7438009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:35.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.703+0000 7f7440c00700 1 -- 172.21.15.1:0/2874326369 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f743005a380 con 0x7f7444105910 2024-02-28T17:09:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.966+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f744404bed0 con 0x7f7444105910 2024-02-28T17:09:35.970 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:35 smithi001 ceph-mon[112940]: from='client.35878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:35.970 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:35 smithi001 ceph-mon[112940]: from='client.35882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:35.970 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:35 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1392534034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.967+0000 7f7440c00700 1 -- 172.21.15.1:0/2874326369 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f7430059f10 con 0x7f7444105910 2024-02-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:09:35.971 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:09:35.972 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f742c06d7f0 msgr2=0x7f742c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f742c06d7f0 0x7f742c06fca0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f743800b020 tx=0x7f7438009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444105910 msgr2=0x7f7444115f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444105910 0x7f7444115f60 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f743000c890 tx=0x7f74300044c0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 shutdown_connections 2024-02-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f742c06d7f0 0x7f742c06fca0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7444104f70 0x7f7444115a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 --2- 172.21.15.1:0/2874326369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7444105910 0x7f7444115f60 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 >> 172.21.15.1:0/2874326369 conn(0x7f74440faae0 msgr2=0x7f74440fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.971+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 shutdown_connections 2024-02-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:35.972+0000 7f744a600700 1 -- 172.21.15.1:0/2874326369 wait complete. 2024-02-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:09:36.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:35 smithi066 ceph-mon[85781]: from='client.35878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:36.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:35 smithi066 ceph-mon[85781]: from='client.35882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:36.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:35 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1392534034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:09:36.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.124+0000 7f4061000700 1 -- 172.21.15.1:0/3827123983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 msgr2=0x7f405c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:36.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.124+0000 7f4061000700 1 --2- 172.21.15.1:0/3827123983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 0x7f405c1099c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f404c004940 tx=0x7f404c009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:36.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.124+0000 7f4061000700 1 -- 172.21.15.1:0/3827123983 shutdown_connections 2024-02-28T17:09:36.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.124+0000 7f4061000700 1 --2- 172.21.15.1:0/3827123983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f405c103dc0 0x7f405c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.124+0000 7f4061000700 1 --2- 172.21.15.1:0/3827123983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 0x7f405c1099c0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.124+0000 7f4061000700 1 -- 172.21.15.1:0/3827123983 >> 172.21.15.1:0/3827123983 conn(0x7f405c072d50 msgr2=0x7f405c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:36.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.124+0000 7f4061000700 1 -- 172.21.15.1:0/3827123983 shutdown_connections 2024-02-28T17:09:36.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.124+0000 7f4061000700 1 -- 172.21.15.1:0/3827123983 wait complete. 2024-02-28T17:09:36.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.125+0000 7f4061000700 1 Processor -- start 2024-02-28T17:09:36.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.125+0000 7f4061000700 1 -- start start 2024-02-28T17:09:36.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.125+0000 7f4061000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f405c103dc0 0x7f405c11a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.125+0000 7f4061000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 0x7f405c11a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.125+0000 7f4061000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c11ac70 con 0x7f405c1095f0 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.125+0000 7f4061000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c113ef0 con 0x7f405c103dc0 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.126+0000 7f4059400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 0x7f405c11a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.126+0000 7f4059400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 0x7f405c11a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60792/0 (socket says 172.21.15.1:60792) 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.126+0000 7f4059400700 1 -- 172.21.15.1:0/903994273 learned_addr learned my addr 172.21.15.1:0/903994273 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.126+0000 7f4059e00700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f405c103dc0 0x7f405c11a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.126+0000 7f4059400700 1 -- 172.21.15.1:0/903994273 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f405c103dc0 msgr2=0x7f405c11a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.126+0000 7f4059400700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f405c103dc0 0x7f405c11a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.126+0000 7f4059400700 1 -- 172.21.15.1:0/903994273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c005040 con 0x7f405c1095f0 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.126+0000 7f4059400700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 0x7f405c11a590 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f404800b790 tx=0x7f4048009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.127+0000 7f4052000700 1 -- 172.21.15.1:0/903994273 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f404800f770 con 0x7f405c1095f0 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.127+0000 7f4052000700 1 -- 172.21.15.1:0/903994273 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f404800fda0 con 0x7f405c1095f0 2024-02-28T17:09:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.127+0000 7f4052000700 1 -- 172.21.15.1:0/903994273 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f404800d310 con 0x7f405c1095f0 2024-02-28T17:09:36.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.127+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c114170 con 0x7f405c1095f0 2024-02-28T17:09:36.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.127+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c1146b0 con 0x7f405c1095f0 2024-02-28T17:09:36.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.128+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c075870 con 0x7f405c1095f0 2024-02-28T17:09:36.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.129+0000 7f4052000700 1 -- 172.21.15.1:0/903994273 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4048010420 con 0x7f405c1095f0 2024-02-28T17:09:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.130+0000 7f4052000700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f404006d7f0 0x7f404006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.130+0000 7f4052000700 1 -- 172.21.15.1:0/903994273 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f404808cef0 con 0x7f405c1095f0 2024-02-28T17:09:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.130+0000 7f4059e00700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f404006d7f0 0x7f404006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.131+0000 7f4059e00700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f404006d7f0 0x7f404006fca0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f404c00b3b0 tx=0x7f404c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:36.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.134+0000 7f4052000700 1 -- 172.21.15.1:0/903994273 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f404805a2d0 con 0x7f405c1095f0 2024-02-28T17:09:36.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.376+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f405c115710 con 0x7f404006d7f0 2024-02-28T17:09:36.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.378+0000 7f4052000700 1 -- 172.21.15.1:0/903994273 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f405c115710 con 0x7f404006d7f0 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:09:36.381 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:09:36.382 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:09:36.382 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f404006d7f0 msgr2=0x7f404006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f404006d7f0 0x7f404006fca0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f404c00b3b0 tx=0x7f404c009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:36.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 msgr2=0x7f405c11a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 0x7f405c11a590 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f404800b790 tx=0x7f4048009250 comp rx=0 tx=0).stop 2024-02-28T17:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 shutdown_connections 2024-02-28T17:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f404006d7f0 0x7f404006fca0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f405c103dc0 0x7f405c11a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 --2- 172.21.15.1:0/903994273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f405c1095f0 0x7f405c11a590 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.380+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 >> 172.21.15.1:0/903994273 conn(0x7f405c072d50 msgr2=0x7f405c0746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.381+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 shutdown_connections 2024-02-28T17:09:36.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.381+0000 7f4061000700 1 -- 172.21.15.1:0/903994273 wait complete. 2024-02-28T17:09:36.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.530+0000 7f7aa3200700 1 -- 172.21.15.1:0/2904051360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109d20 msgr2=0x7f7aa41033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:36.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.530+0000 7f7aa3200700 1 --2- 172.21.15.1:0/2904051360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109d20 0x7f7aa41033a0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f7a90004940 tx=0x7f7a90009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:36.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.530+0000 7f7aa3200700 1 -- 172.21.15.1:0/2904051360 shutdown_connections 2024-02-28T17:09:36.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.530+0000 7f7aa3200700 1 --2- 172.21.15.1:0/2904051360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109d20 0x7f7aa41033a0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.530+0000 7f7aa3200700 1 --2- 172.21.15.1:0/2904051360 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7aa4109380 0x7f7aa4109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.530+0000 7f7aa3200700 1 -- 172.21.15.1:0/2904051360 >> 172.21.15.1:0/2904051360 conn(0x7f7aa4072b50 msgr2=0x7f7aa4074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:36.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.530+0000 7f7aa3200700 1 -- 172.21.15.1:0/2904051360 shutdown_connections 2024-02-28T17:09:36.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.530+0000 7f7aa3200700 1 -- 172.21.15.1:0/2904051360 wait complete. 2024-02-28T17:09:36.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.531+0000 7f7aa3200700 1 Processor -- start 2024-02-28T17:09:36.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.531+0000 7f7aa3200700 1 -- start start 2024-02-28T17:09:36.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa3200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109380 0x7f7aa4119de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:36.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa3200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7aa4109d20 0x7f7aa411a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:36.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa3200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa411aa00 con 0x7f7aa4109380 2024-02-28T17:09:36.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa3200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa4113cc0 con 0x7f7aa4109d20 2024-02-28T17:09:36.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109380 0x7f7aa4119de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:36.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109380 0x7f7aa4119de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60812/0 (socket says 172.21.15.1:60812) 2024-02-28T17:09:36.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa1e00700 1 -- 172.21.15.1:0/2693990704 learned_addr learned my addr 172.21.15.1:0/2693990704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:09:36.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa1e00700 1 -- 172.21.15.1:0/2693990704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7aa4109d20 msgr2=0x7f7aa411a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:36.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa1e00700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7aa4109d20 0x7f7aa411a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.532+0000 7f7aa1e00700 1 -- 172.21.15.1:0/2693990704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a90005040 con 0x7f7aa4109380 2024-02-28T17:09:36.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.533+0000 7f7aa1e00700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109380 0x7f7aa4119de0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b790 tx=0x7f7a94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:36.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.533+0000 7f7a9a000700 1 -- 172.21.15.1:0/2693990704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a9400f770 con 0x7f7aa4109380 2024-02-28T17:09:36.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.533+0000 7f7a9a000700 1 -- 172.21.15.1:0/2693990704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a9400fda0 con 0x7f7aa4109380 2024-02-28T17:09:36.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.533+0000 7f7a9a000700 1 -- 172.21.15.1:0/2693990704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a9400d310 con 0x7f7aa4109380 2024-02-28T17:09:36.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.533+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa4113fa0 con 0x7f7aa4109380 2024-02-28T17:09:36.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.533+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa4114560 con 0x7f7aa4109380 2024-02-28T17:09:36.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.534+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa405f5c0 con 0x7f7aa4109380 2024-02-28T17:09:36.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.535+0000 7f7a9a000700 1 -- 172.21.15.1:0/2693990704 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7a94010420 con 0x7f7aa4109380 2024-02-28T17:09:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.536+0000 7f7a9a000700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7a8806d7f0 0x7f7a8806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:09:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.536+0000 7f7a9a000700 1 -- 172.21.15.1:0/2693990704 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7a9408cef0 con 0x7f7aa4109380 2024-02-28T17:09:36.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.536+0000 7f7aa1400700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7a8806d7f0 0x7f7a8806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:09:36.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.537+0000 7f7aa1400700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7a8806d7f0 0x7f7a8806fca0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7a90004f90 tx=0x7f7a90009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:09:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.542+0000 7f7a9a000700 1 -- 172.21.15.1:0/2693990704 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7a940567a0 con 0x7f7aa4109380 2024-02-28T17:09:36.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:36 smithi001 ceph-mon[112940]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:36.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:36 smithi001 ceph-mon[112940]: pgmap v1055: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:36.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2874326369' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:09:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.848+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7aa41104d0 con 0x7f7aa4109380 2024-02-28T17:09:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.848+0000 7f7a9a000700 1 -- 172.21.15.1:0/2693990704 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f7a94059e60 con 0x7f7aa4109380 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:09:36.852 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:09:36.853 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:09:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7a8806d7f0 msgr2=0x7f7a8806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7a8806d7f0 0x7f7a8806fca0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7a90004f90 tx=0x7f7a90009e30 comp rx=0 tx=0).stop 2024-02-28T17:09:36.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109380 msgr2=0x7f7aa4119de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:09:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109380 0x7f7aa4119de0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b790 tx=0x7f7a94009250 comp rx=0 tx=0).stop 2024-02-28T17:09:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 shutdown_connections 2024-02-28T17:09:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7a8806d7f0 0x7f7a8806fca0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7aa4109380 0x7f7aa4119de0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 --2- 172.21.15.1:0/2693990704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7aa4109d20 0x7f7aa411a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:09:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.851+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 >> 172.21.15.1:0/2693990704 conn(0x7f7aa4072b50 msgr2=0x7f7aa4075580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:09:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.852+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 shutdown_connections 2024-02-28T17:09:36.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:09:36.852+0000 7f7aa3200700 1 -- 172.21.15.1:0/2693990704 wait complete. 2024-02-28T17:09:37.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:36 smithi066 ceph-mon[85781]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:37.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:36 smithi066 ceph-mon[85781]: pgmap v1055: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:37.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2874326369' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:09:38.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:37 smithi066 ceph-mon[85781]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:38.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:37 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2693990704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:09:38.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:37 smithi001 ceph-mon[112940]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:09:38.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:37 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2693990704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:09:39.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:38 smithi066 ceph-mon[85781]: pgmap v1056: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:39.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:38 smithi001 ceph-mon[112940]: pgmap v1056: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:41.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:40 smithi066 ceph-mon[85781]: pgmap v1057: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:41.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:40 smithi001 ceph-mon[112940]: pgmap v1057: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:41.913 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:09:42.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:09:43.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:42 smithi066 ceph-mon[85781]: pgmap v1058: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:43.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:42 smithi001 ceph-mon[112940]: pgmap v1058: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:45.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:44 smithi066 ceph-mon[85781]: pgmap v1059: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:45.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:44 smithi001 ceph-mon[112940]: pgmap v1059: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:47.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:46 smithi066 ceph-mon[85781]: pgmap v1060: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:47.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:46 smithi001 ceph-mon[112940]: pgmap v1060: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:49.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:48 smithi066 ceph-mon[85781]: pgmap v1061: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:49.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:48 smithi001 ceph-mon[112940]: pgmap v1061: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:51.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:50 smithi066 ceph-mon[85781]: pgmap v1062: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:51.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:50 smithi001 ceph-mon[112940]: pgmap v1062: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:53.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:52 smithi066 ceph-mon[85781]: pgmap v1063: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:53.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:52 smithi001 ceph-mon[112940]: pgmap v1063: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:55.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:54 smithi066 ceph-mon[85781]: pgmap v1064: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:55.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:54 smithi001 ceph-mon[112940]: pgmap v1064: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:57.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:56 smithi066 ceph-mon[85781]: pgmap v1065: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:57.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:09:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:56 smithi001 ceph-mon[112940]: pgmap v1065: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:57.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:09:59.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:09:58 smithi066 ceph-mon[85781]: pgmap v1066: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:09:59.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:09:58 smithi001 ceph-mon[112940]: pgmap v1066: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:00.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T17:09:59.999+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:10:00.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:00 smithi066 ceph-mon[85781]: pgmap v1067: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:00.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:00 smithi066 ceph-mon[85781]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:10:01.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:00 smithi001 ceph-mon[112940]: pgmap v1067: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:01.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:00 smithi001 ceph-mon[112940]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:10:03.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:02 smithi066 ceph-mon[85781]: pgmap v1068: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:02 smithi001 ceph-mon[112940]: pgmap v1068: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:05.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:04 smithi066 ceph-mon[85781]: pgmap v1069: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:05.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:04 smithi001 ceph-mon[112940]: pgmap v1069: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:07.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:06 smithi066 ceph-mon[85781]: pgmap v1070: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:07.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:06 smithi001 ceph-mon[112940]: pgmap v1070: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:07.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.024+0000 7f4613400700 1 -- 172.21.15.1:0/2320230236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c071f60 msgr2=0x7f460c070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.024+0000 7f4613400700 1 --2- 172.21.15.1:0/2320230236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c071f60 0x7f460c070330 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4600009d10 tx=0x7f4600004d10 comp rx=0 tx=0).stop 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.025+0000 7f4613400700 1 -- 172.21.15.1:0/2320230236 shutdown_connections 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.025+0000 7f4613400700 1 --2- 172.21.15.1:0/2320230236 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f460c070870 0x7f460c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.025+0000 7f4613400700 1 --2- 172.21.15.1:0/2320230236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c071f60 0x7f460c070330 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.025+0000 7f4613400700 1 -- 172.21.15.1:0/2320230236 >> 172.21.15.1:0/2320230236 conn(0x7f460c0f8ae0 msgr2=0x7f460c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.025+0000 7f4613400700 1 -- 172.21.15.1:0/2320230236 shutdown_connections 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.025+0000 7f4613400700 1 -- 172.21.15.1:0/2320230236 wait complete. 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.026+0000 7f4613400700 1 Processor -- start 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.026+0000 7f4613400700 1 -- start start 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.026+0000 7f4613400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c070870 0x7f460c11a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.026+0000 7f4613400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f460c071f60 0x7f460c11a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.026+0000 7f4613400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c11ae40 con 0x7f460c070870 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.026+0000 7f4613400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c114150 con 0x7f460c071f60 2024-02-28T17:10:07.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.027+0000 7f4610a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c070870 0x7f460c11a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.027+0000 7f4610a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c070870 0x7f460c11a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57130/0 (socket says 172.21.15.1:57130) 2024-02-28T17:10:07.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.027+0000 7f4610a00700 1 -- 172.21.15.1:0/3806676807 learned_addr learned my addr 172.21.15.1:0/3806676807 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.027+0000 7f460be00700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f460c071f60 0x7f460c11a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.027+0000 7f4610a00700 1 -- 172.21.15.1:0/3806676807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f460c071f60 msgr2=0x7f460c11a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.027+0000 7f4610a00700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f460c071f60 0x7f460c11a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.027+0000 7f4610a00700 1 -- 172.21.15.1:0/3806676807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4600009940 con 0x7f460c070870 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.028+0000 7f4610a00700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c070870 0x7f460c11a2b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4600008dd0 tx=0x7f4600004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.028+0000 7f4608c00700 1 -- 172.21.15.1:0/3806676807 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f460000e040 con 0x7f460c070870 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.028+0000 7f4608c00700 1 -- 172.21.15.1:0/3806676807 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46000128e0 con 0x7f460c070870 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.028+0000 7f4608c00700 1 -- 172.21.15.1:0/3806676807 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f460001c980 con 0x7f460c070870 2024-02-28T17:10:07.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.028+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c1143d0 con 0x7f460c070870 2024-02-28T17:10:07.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.028+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c114910 con 0x7f460c070870 2024-02-28T17:10:07.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.029+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460c0fe400 con 0x7f460c070870 2024-02-28T17:10:07.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.031+0000 7f4608c00700 1 -- 172.21.15.1:0/3806676807 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4600012a50 con 0x7f460c070870 2024-02-28T17:10:07.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.031+0000 7f4608c00700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45f406d7f0 0x7f45f406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.032+0000 7f4608c00700 1 -- 172.21.15.1:0/3806676807 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f460008eb10 con 0x7f460c070870 2024-02-28T17:10:07.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.032+0000 7f460be00700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45f406d7f0 0x7f45f406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.033+0000 7f460be00700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45f406d7f0 0x7f45f406fca0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f45f8007fd0 tx=0x7f45f8009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.039+0000 7f4608c00700 1 -- 172.21.15.1:0/3806676807 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f460005bf10 con 0x7f460c070870 2024-02-28T17:10:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.286+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f460c04bed0 con 0x7f45f406d7f0 2024-02-28T17:10:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.290+0000 7f4608c00700 1 -- 172.21.15.1:0/3806676807 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f460c04bed0 con 0x7f45f406d7f0 2024-02-28T17:10:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45f406d7f0 msgr2=0x7f45f406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45f406d7f0 0x7f45f406fca0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f45f8007fd0 tx=0x7f45f8009040 comp rx=0 tx=0).stop 2024-02-28T17:10:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c070870 msgr2=0x7f460c11a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c070870 0x7f460c11a2b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4600008dd0 tx=0x7f4600004d10 comp rx=0 tx=0).stop 2024-02-28T17:10:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 shutdown_connections 2024-02-28T17:10:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45f406d7f0 0x7f45f406fca0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460c070870 0x7f460c11a2b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 --2- 172.21.15.1:0/3806676807 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f460c071f60 0x7f460c11a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 >> 172.21.15.1:0/3806676807 conn(0x7f460c0f8ae0 msgr2=0x7f460c10c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:07.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 shutdown_connections 2024-02-28T17:10:07.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.293+0000 7f4613400700 1 -- 172.21.15.1:0/3806676807 wait complete. 2024-02-28T17:10:07.313 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:10:07.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.456+0000 7f44d9000700 1 -- 172.21.15.1:0/2514141775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fe8e0 msgr2=0x7f44d4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.456+0000 7f44d9000700 1 --2- 172.21.15.1:0/2514141775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fe8e0 0x7f44d4106f20 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f44c0004940 tx=0x7f44c0009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:07.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.456+0000 7f44d9000700 1 -- 172.21.15.1:0/2514141775 shutdown_connections 2024-02-28T17:10:07.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.456+0000 7f44d9000700 1 --2- 172.21.15.1:0/2514141775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fe8e0 0x7f44d4106f20 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.456+0000 7f44d9000700 1 --2- 172.21.15.1:0/2514141775 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d40fdf40 0x7f44d40fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.456+0000 7f44d9000700 1 -- 172.21.15.1:0/2514141775 >> 172.21.15.1:0/2514141775 conn(0x7f44d40f79d0 msgr2=0x7f44d40f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:07.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.457+0000 7f44d9000700 1 -- 172.21.15.1:0/2514141775 shutdown_connections 2024-02-28T17:10:07.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.457+0000 7f44d9000700 1 -- 172.21.15.1:0/2514141775 wait complete. 2024-02-28T17:10:07.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.457+0000 7f44d9000700 1 Processor -- start 2024-02-28T17:10:07.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.457+0000 7f44d9000700 1 -- start start 2024-02-28T17:10:07.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.458+0000 7f44d9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fdf40 0x7f44d40fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.458+0000 7f44d9000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d40fe8e0 0x7f44d40fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.458+0000 7f44d9000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40fc8b0 con 0x7f44d40fdf40 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.458+0000 7f44d9000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40fca20 con 0x7f44d40fe8e0 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.458+0000 7f44d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fdf40 0x7f44d40fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.458+0000 7f44d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fdf40 0x7f44d40fbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57148/0 (socket says 172.21.15.1:57148) 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.458+0000 7f44d1e00700 1 -- 172.21.15.1:0/1880345000 learned_addr learned my addr 172.21.15.1:0/1880345000 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44d1400700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d40fe8e0 0x7f44d40fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44d1e00700 1 -- 172.21.15.1:0/1880345000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d40fe8e0 msgr2=0x7f44d40fc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44d1e00700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d40fe8e0 0x7f44d40fc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44d1e00700 1 -- 172.21.15.1:0/1880345000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0005040 con 0x7f44d40fdf40 2024-02-28T17:10:07.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44d1400700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d40fe8e0 0x7f44d40fc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:10:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44d1e00700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fdf40 0x7f44d40fbd70 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f44c400cb10 tx=0x7f44c4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44ca000700 1 -- 172.21.15.1:0/1880345000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44c4004200 con 0x7f44d40fdf40 2024-02-28T17:10:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44ca000700 1 -- 172.21.15.1:0/1880345000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44c400acc0 con 0x7f44d40fdf40 2024-02-28T17:10:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d4119cc0 con 0x7f44d40fdf40 2024-02-28T17:10:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44ca000700 1 -- 172.21.15.1:0/1880345000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44c4003d90 con 0x7f44d40fdf40 2024-02-28T17:10:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.459+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d411a1a0 con 0x7f44d40fdf40 2024-02-28T17:10:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.461+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d4104530 con 0x7f44d40fdf40 2024-02-28T17:10:07.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.462+0000 7f44ca000700 1 -- 172.21.15.1:0/1880345000 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f44c4003430 con 0x7f44d40fdf40 2024-02-28T17:10:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.463+0000 7f44ca000700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b806d7f0 0x7f44b806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.463+0000 7f44ca000700 1 -- 172.21.15.1:0/1880345000 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f44c408d520 con 0x7f44d40fdf40 2024-02-28T17:10:07.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.463+0000 7f44d1400700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b806d7f0 0x7f44b806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.464+0000 7f44d1400700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b806d7f0 0x7f44b806fca0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f44c0004f90 tx=0x7f44c0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.468+0000 7f44ca000700 1 -- 172.21.15.1:0/1880345000 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44c4056cd0 con 0x7f44d40fdf40 2024-02-28T17:10:07.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.716+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44d40fd4b0 con 0x7f44b806d7f0 2024-02-28T17:10:07.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.718+0000 7f44ca000700 1 -- 172.21.15.1:0/1880345000 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f44d40fd4b0 con 0x7f44b806d7f0 2024-02-28T17:10:07.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b806d7f0 msgr2=0x7f44b806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b806d7f0 0x7f44b806fca0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f44c0004f90 tx=0x7f44c0009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:07.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fdf40 msgr2=0x7f44d40fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fdf40 0x7f44d40fbd70 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f44c400cb10 tx=0x7f44c4007f70 comp rx=0 tx=0).stop 2024-02-28T17:10:07.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 shutdown_connections 2024-02-28T17:10:07.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b806d7f0 0x7f44b806fca0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d40fdf40 0x7f44d40fbd70 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 --2- 172.21.15.1:0/1880345000 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d40fe8e0 0x7f44d40fc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.721+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 >> 172.21.15.1:0/1880345000 conn(0x7f44d40f79d0 msgr2=0x7f44d41016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:07.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.722+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 shutdown_connections 2024-02-28T17:10:07.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.722+0000 7f44d9000700 1 -- 172.21.15.1:0/1880345000 wait complete. 2024-02-28T17:10:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.885+0000 7f6ef5200700 1 -- 172.21.15.1:0/405284315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0070870 msgr2=0x7f6ef0070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.885+0000 7f6ef5200700 1 --2- 172.21.15.1:0/405284315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0070870 0x7f6ef0070ce0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f6edc004970 tx=0x7f6edc009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.885+0000 7f6ef5200700 1 -- 172.21.15.1:0/405284315 shutdown_connections 2024-02-28T17:10:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.885+0000 7f6ef5200700 1 --2- 172.21.15.1:0/405284315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0070870 0x7f6ef0070ce0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.885+0000 7f6ef5200700 1 --2- 172.21.15.1:0/405284315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ef0071f60 0x7f6ef0070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.885+0000 7f6ef5200700 1 -- 172.21.15.1:0/405284315 >> 172.21.15.1:0/405284315 conn(0x7f6ef00f8af0 msgr2=0x7f6ef00faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:07.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.885+0000 7f6ef5200700 1 -- 172.21.15.1:0/405284315 shutdown_connections 2024-02-28T17:10:07.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.885+0000 7f6ef5200700 1 -- 172.21.15.1:0/405284315 wait complete. 2024-02-28T17:10:07.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.886+0000 7f6ef5200700 1 Processor -- start 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.886+0000 7f6ef5200700 1 -- start start 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.886+0000 7f6ef5200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ef0070870 0x7f6ef011a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.886+0000 7f6ef5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0071f60 0x7f6ef011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.886+0000 7f6ef5200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef011ace0 con 0x7f6ef0071f60 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.886+0000 7f6ef5200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0113f60 con 0x7f6ef0070870 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eed400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0071f60 0x7f6ef011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eed400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0071f60 0x7f6ef011a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57174/0 (socket says 172.21.15.1:57174) 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eed400700 1 -- 172.21.15.1:0/1370398926 learned_addr learned my addr 172.21.15.1:0/1370398926 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:07.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eede00700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ef0070870 0x7f6ef011a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eed400700 1 -- 172.21.15.1:0/1370398926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ef0070870 msgr2=0x7f6ef011a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:07.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eed400700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ef0070870 0x7f6ef011a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:07.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eed400700 1 -- 172.21.15.1:0/1370398926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc005040 con 0x7f6ef0071f60 2024-02-28T17:10:07.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eede00700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ef0070870 0x7f6ef011a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:10:07.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.887+0000 7f6eed400700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0071f60 0x7f6ef011a600 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f6edc004f50 tx=0x7f6edc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:07.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.888+0000 7f6ee6000700 1 -- 172.21.15.1:0/1370398926 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6edc00e040 con 0x7f6ef0071f60 2024-02-28T17:10:07.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.888+0000 7f6ee6000700 1 -- 172.21.15.1:0/1370398926 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6edc00bc10 con 0x7f6ef0071f60 2024-02-28T17:10:07.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.888+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef0114180 con 0x7f6ef0071f60 2024-02-28T17:10:07.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.888+0000 7f6ee6000700 1 -- 172.21.15.1:0/1370398926 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6edc01c630 con 0x7f6ef0071f60 2024-02-28T17:10:07.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.888+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef01146c0 con 0x7f6ef0071f60 2024-02-28T17:10:07.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.889+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef00fe410 con 0x7f6ef0071f60 2024-02-28T17:10:07.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.890+0000 7f6ee6000700 1 -- 172.21.15.1:0/1370398926 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6edc00bd80 con 0x7f6ef0071f60 2024-02-28T17:10:07.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.891+0000 7f6ee6000700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ed406d7f0 0x7f6ed406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.892+0000 7f6ee6000700 1 -- 172.21.15.1:0/1370398926 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6edc08e200 con 0x7f6ef0071f60 2024-02-28T17:10:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.892+0000 7f6eede00700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ed406d7f0 0x7f6ed406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.893+0000 7f6eede00700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ed406d7f0 0x7f6ed406fca0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0006fd0 tx=0x7f6ee0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:07.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:07.897+0000 7f6ee6000700 1 -- 172.21.15.1:0/1370398926 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6edc05b5e0 con 0x7f6ef0071f60 2024-02-28T17:10:08.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.134+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ef004bf40 con 0x7f6ed406d7f0 2024-02-28T17:10:08.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.164+0000 7f6ee6000700 1 -- 172.21.15.1:0/1370398926 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6ef004bf40 con 0x7f6ed406d7f0 2024-02-28T17:10:08.166 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:10:08.166 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (36m) 13m ago 50m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:10:08.166 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (32m) 13m ago 50m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (32m) 13m ago 49m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (35m) 13m ago 50m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (28m) 13m ago 43m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (28m) 13m ago 43m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (43m) 13m ago 43m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (13m) 13m ago 43m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (38m) 13m ago 52m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (37m) 13m ago 49m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (34m) 13m ago 52m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (34m) 13m ago 49m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (37m) 13m ago 50m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (37m) 13m ago 49m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (32m) 13m ago 48m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (31m) 13m ago 47m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:10:08.168 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (31m) 13m ago 47m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:10:08.168 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (30m) 13m ago 46m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:10:08.168 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (30m) 13m ago 45m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:10:08.168 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (29m) 13m ago 45m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:10:08.168 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (36m) 13m ago 49m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:10:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ed406d7f0 msgr2=0x7f6ed406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ed406d7f0 0x7f6ed406fca0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0006fd0 tx=0x7f6ee0008040 comp rx=0 tx=0).stop 2024-02-28T17:10:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0071f60 msgr2=0x7f6ef011a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0071f60 0x7f6ef011a600 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f6edc004f50 tx=0x7f6edc009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 shutdown_connections 2024-02-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6ed406d7f0 0x7f6ed406fca0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ef0070870 0x7f6ef011a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 --2- 172.21.15.1:0/1370398926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ef0071f60 0x7f6ef011a600 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 >> 172.21.15.1:0/1370398926 conn(0x7f6ef00f8af0 msgr2=0x7f6ef00fa760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 shutdown_connections 2024-02-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.167+0000 7f6ef5200700 1 -- 172.21.15.1:0/1370398926 wait complete. 2024-02-28T17:10:08.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.327+0000 7fd6d7800700 1 -- 172.21.15.1:0/3460357474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105170 msgr2=0x7fd6d0105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:08.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.327+0000 7fd6d7800700 1 --2- 172.21.15.1:0/3460357474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105170 0x7fd6d0105540 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4004940 tx=0x7fd6c4009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:08.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.327+0000 7fd6d7800700 1 -- 172.21.15.1:0/3460357474 shutdown_connections 2024-02-28T17:10:08.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.327+0000 7fd6d7800700 1 --2- 172.21.15.1:0/3460357474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6d0105b10 0x7fd6d00ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.327+0000 7fd6d7800700 1 --2- 172.21.15.1:0/3460357474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105170 0x7fd6d0105540 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.327+0000 7fd6d7800700 1 -- 172.21.15.1:0/3460357474 >> 172.21.15.1:0/3460357474 conn(0x7fd6d00face0 msgr2=0x7fd6d00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.327+0000 7fd6d7800700 1 -- 172.21.15.1:0/3460357474 shutdown_connections 2024-02-28T17:10:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.327+0000 7fd6d7800700 1 -- 172.21.15.1:0/3460357474 wait complete. 2024-02-28T17:10:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.328+0000 7fd6d7800700 1 Processor -- start 2024-02-28T17:10:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.329+0000 7fd6d7800700 1 -- start start 2024-02-28T17:10:08.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.329+0000 7fd6d7800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6d0105170 0x7fd6d0115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:08.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.329+0000 7fd6d7800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105b10 0x7fd6d01161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:08.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.329+0000 7fd6d7800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d01167f0 con 0x7fd6d0105b10 2024-02-28T17:10:08.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.329+0000 7fd6d7800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d010fb40 con 0x7fd6d0105170 2024-02-28T17:10:08.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.330+0000 7fd6cfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105b10 0x7fd6d01161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:08.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.330+0000 7fd6cfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105b10 0x7fd6d01161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57196/0 (socket says 172.21.15.1:57196) 2024-02-28T17:10:08.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.330+0000 7fd6cfe00700 1 -- 172.21.15.1:0/2066042819 learned_addr learned my addr 172.21.15.1:0/2066042819 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:08.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.330+0000 7fd6d4e00700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6d0105170 0x7fd6d0115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:08.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.330+0000 7fd6cfe00700 1 -- 172.21.15.1:0/2066042819 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6d0105170 msgr2=0x7fd6d0115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:08.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.330+0000 7fd6cfe00700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6d0105170 0x7fd6d0115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.330+0000 7fd6cfe00700 1 -- 172.21.15.1:0/2066042819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c4005040 con 0x7fd6d0105b10 2024-02-28T17:10:08.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.330+0000 7fd6cfe00700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105b10 0x7fd6d01161a0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc002a50 tx=0x7fd6bc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:08.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.331+0000 7fd6ccc00700 1 -- 172.21.15.1:0/2066042819 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6bc007cc0 con 0x7fd6d0105b10 2024-02-28T17:10:08.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.331+0000 7fd6ccc00700 1 -- 172.21.15.1:0/2066042819 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6bc00f040 con 0x7fd6d0105b10 2024-02-28T17:10:08.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.331+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d010fe20 con 0x7fd6d0105b10 2024-02-28T17:10:08.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.331+0000 7fd6ccc00700 1 -- 172.21.15.1:0/2066042819 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6bc00da40 con 0x7fd6d0105b10 2024-02-28T17:10:08.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.331+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d0110360 con 0x7fd6d0105b10 2024-02-28T17:10:08.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.332+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d0102480 con 0x7fd6d0105b10 2024-02-28T17:10:08.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.334+0000 7fd6ccc00700 1 -- 172.21.15.1:0/2066042819 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd6bc0097c0 con 0x7fd6d0105b10 2024-02-28T17:10:08.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.335+0000 7fd6ccc00700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6b806d7f0 0x7fd6b806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:08.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.335+0000 7fd6ccc00700 1 -- 172.21.15.1:0/2066042819 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd6bc08cc00 con 0x7fd6d0105b10 2024-02-28T17:10:08.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.335+0000 7fd6d4e00700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6b806d7f0 0x7fd6b806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:08.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.336+0000 7fd6d4e00700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6b806d7f0 0x7fd6b806fca0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400b020 tx=0x7fd6c4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:08.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.340+0000 7fd6ccc00700 1 -- 172.21.15.1:0/2066042819 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd6bc092020 con 0x7fd6d0105b10 2024-02-28T17:10:08.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.644+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd6d0111280 con 0x7fd6d0105b10 2024-02-28T17:10:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.645+0000 7fd6ccc00700 1 -- 172.21.15.1:0/2066042819 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fd6bc05a000 con 0x7fd6d0105b10 2024-02-28T17:10:08.647 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:10:08.648 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:10:08.649 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:10:08.649 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:10:08.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:10:08.649 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:10:08.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.648+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6b806d7f0 msgr2=0x7fd6b806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:08.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.648+0000 7fd6d7800700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6b806d7f0 0x7fd6b806fca0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400b020 tx=0x7fd6c4009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:08.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.648+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105b10 msgr2=0x7fd6d01161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:08.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.648+0000 7fd6d7800700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105b10 0x7fd6d01161a0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc002a50 tx=0x7fd6bc004600 comp rx=0 tx=0).stop 2024-02-28T17:10:08.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.649+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 shutdown_connections 2024-02-28T17:10:08.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.649+0000 7fd6d7800700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd6b806d7f0 0x7fd6b806fca0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.649+0000 7fd6d7800700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd6d0105170 0x7fd6d0115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.649+0000 7fd6d7800700 1 --2- 172.21.15.1:0/2066042819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6d0105b10 0x7fd6d01161a0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.649+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 >> 172.21.15.1:0/2066042819 conn(0x7fd6d00face0 msgr2=0x7fd6d00fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.649+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 shutdown_connections 2024-02-28T17:10:08.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.649+0000 7fd6d7800700 1 -- 172.21.15.1:0/2066042819 wait complete. 2024-02-28T17:10:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.803+0000 7fba97600700 1 -- 172.21.15.1:0/3534388734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba90103dc0 msgr2=0x7fba90104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.803+0000 7fba97600700 1 --2- 172.21.15.1:0/3534388734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba90103dc0 0x7fba90104210 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fba7c004940 tx=0x7fba7c009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.803+0000 7fba97600700 1 -- 172.21.15.1:0/3534388734 shutdown_connections 2024-02-28T17:10:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.803+0000 7fba97600700 1 --2- 172.21.15.1:0/3534388734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba90103dc0 0x7fba90104210 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.803+0000 7fba97600700 1 --2- 172.21.15.1:0/3534388734 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba901095f0 0x7fba901099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.803+0000 7fba97600700 1 -- 172.21.15.1:0/3534388734 >> 172.21.15.1:0/3534388734 conn(0x7fba90072d50 msgr2=0x7fba90075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.803+0000 7fba97600700 1 -- 172.21.15.1:0/3534388734 shutdown_connections 2024-02-28T17:10:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.803+0000 7fba97600700 1 -- 172.21.15.1:0/3534388734 wait complete. 2024-02-28T17:10:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.804+0000 7fba97600700 1 Processor -- start 2024-02-28T17:10:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.804+0000 7fba97600700 1 -- start start 2024-02-28T17:10:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.804+0000 7fba97600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba90103dc0 0x7fba9011a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.804+0000 7fba97600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba901095f0 0x7fba9011a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.804+0000 7fba97600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9011ac50 con 0x7fba901095f0 2024-02-28T17:10:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.804+0000 7fba97600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90113ed0 con 0x7fba90103dc0 2024-02-28T17:10:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.805+0000 7fba8fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba901095f0 0x7fba9011a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.805+0000 7fba8fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba901095f0 0x7fba9011a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57218/0 (socket says 172.21.15.1:57218) 2024-02-28T17:10:08.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.805+0000 7fba8fe00700 1 -- 172.21.15.1:0/3876008371 learned_addr learned my addr 172.21.15.1:0/3876008371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:08.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.805+0000 7fba94c00700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba90103dc0 0x7fba9011a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:08.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.805+0000 7fba8fe00700 1 -- 172.21.15.1:0/3876008371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba90103dc0 msgr2=0x7fba9011a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:08.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.805+0000 7fba8fe00700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba90103dc0 0x7fba9011a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:08.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.805+0000 7fba8fe00700 1 -- 172.21.15.1:0/3876008371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c005040 con 0x7fba901095f0 2024-02-28T17:10:08.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.805+0000 7fba8fe00700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba901095f0 0x7fba9011a570 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009150 tx=0x7fba7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.806+0000 7fba8cc00700 1 -- 172.21.15.1:0/3876008371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba7c00e040 con 0x7fba901095f0 2024-02-28T17:10:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.806+0000 7fba8cc00700 1 -- 172.21.15.1:0/3876008371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba7c00bbf0 con 0x7fba901095f0 2024-02-28T17:10:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.806+0000 7fba8cc00700 1 -- 172.21.15.1:0/3876008371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba7c01c630 con 0x7fba901095f0 2024-02-28T17:10:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.806+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba901140f0 con 0x7fba901095f0 2024-02-28T17:10:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.806+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba90114630 con 0x7fba901095f0 2024-02-28T17:10:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.807+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba90075870 con 0x7fba901095f0 2024-02-28T17:10:08.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.808+0000 7fba8cc00700 1 -- 172.21.15.1:0/3876008371 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fba7c00bd60 con 0x7fba901095f0 2024-02-28T17:10:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.809+0000 7fba8cc00700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba7806d7f0 0x7fba7806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.810+0000 7fba8cc00700 1 -- 172.21.15.1:0/3876008371 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fba7c08e290 con 0x7fba901095f0 2024-02-28T17:10:08.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.810+0000 7fba94c00700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba7806d7f0 0x7fba7806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:08.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.810+0000 7fba94c00700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba7806d7f0 0x7fba7806fca0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fba84006fd0 tx=0x7fba84008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:08.814+0000 7fba8cc00700 1 -- 172.21.15.1:0/3876008371 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba7c05b670 con 0x7fba901095f0 2024-02-28T17:10:09.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:08 smithi066 ceph-mon[85781]: pgmap v1071: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:09.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:08 smithi066 ceph-mon[85781]: from='client.35906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:09.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:08 smithi066 ceph-mon[85781]: from='client.35910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:09.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:08 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2066042819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:10:09.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:08 smithi001 ceph-mon[112940]: pgmap v1071: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:09.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:08 smithi001 ceph-mon[112940]: from='client.35906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:09.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:08 smithi001 ceph-mon[112940]: from='client.35910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:09.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:08 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2066042819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:10:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.080+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fba901156e0 con 0x7fba901095f0 2024-02-28T17:10:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.081+0000 7fba8cc00700 1 -- 172.21.15.1:0/3876008371 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fba7c05b200 con 0x7fba901095f0 2024-02-28T17:10:09.083 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:10:09.083 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:10:09.083 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:10:09.084 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:10:09.085 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:10:09.086 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:10:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.084+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba7806d7f0 msgr2=0x7fba7806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.084+0000 7fba97600700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba7806d7f0 0x7fba7806fca0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fba84006fd0 tx=0x7fba84008040 comp rx=0 tx=0).stop 2024-02-28T17:10:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.084+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba901095f0 msgr2=0x7fba9011a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.084+0000 7fba97600700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba901095f0 0x7fba9011a570 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009150 tx=0x7fba7c009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.085+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 shutdown_connections 2024-02-28T17:10:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.085+0000 7fba97600700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba7806d7f0 0x7fba7806fca0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.085+0000 7fba97600700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba90103dc0 0x7fba9011a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.085+0000 7fba97600700 1 --2- 172.21.15.1:0/3876008371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba901095f0 0x7fba9011a570 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.085+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 >> 172.21.15.1:0/3876008371 conn(0x7fba90072d50 msgr2=0x7fba900746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.085+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 shutdown_connections 2024-02-28T17:10:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.085+0000 7fba97600700 1 -- 172.21.15.1:0/3876008371 wait complete. 2024-02-28T17:10:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:10:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.235+0000 7f8157800700 1 -- 172.21.15.1:0/3931820123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 msgr2=0x7f81500ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.235+0000 7f8157800700 1 --2- 172.21.15.1:0/3931820123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 0x7f81500ff160 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f813c004940 tx=0x7f813c009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.235+0000 7f8157800700 1 -- 172.21.15.1:0/3931820123 shutdown_connections 2024-02-28T17:10:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.235+0000 7f8157800700 1 --2- 172.21.15.1:0/3931820123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 0x7f81500ff160 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.235+0000 7f8157800700 1 --2- 172.21.15.1:0/3931820123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8150105150 0x7f8150105520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.235+0000 7f8157800700 1 -- 172.21.15.1:0/3931820123 >> 172.21.15.1:0/3931820123 conn(0x7f81500face0 msgr2=0x7f81500fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.235+0000 7f8157800700 1 -- 172.21.15.1:0/3931820123 shutdown_connections 2024-02-28T17:10:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.236+0000 7f8157800700 1 -- 172.21.15.1:0/3931820123 wait complete. 2024-02-28T17:10:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.236+0000 7f8157800700 1 Processor -- start 2024-02-28T17:10:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.237+0000 7f8157800700 1 -- start start 2024-02-28T17:10:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.237+0000 7f8157800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8150105150 0x7f8150115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.237+0000 7f8157800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 0x7f8150116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.237+0000 7f8157800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8150116880 con 0x7f8150105af0 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.237+0000 7f8157800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815010fb90 con 0x7f8150105150 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.237+0000 7f814fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 0x7f8150116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.237+0000 7f814fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 0x7f8150116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57242/0 (socket says 172.21.15.1:57242) 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.237+0000 7f814fe00700 1 -- 172.21.15.1:0/1257382187 learned_addr learned my addr 172.21.15.1:0/1257382187 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.238+0000 7f8154e00700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8150105150 0x7f8150115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.238+0000 7f814fe00700 1 -- 172.21.15.1:0/1257382187 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8150105150 msgr2=0x7f8150115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.238+0000 7f814fe00700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8150105150 0x7f8150115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.238+0000 7f814fe00700 1 -- 172.21.15.1:0/1257382187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c005040 con 0x7f8150105af0 2024-02-28T17:10:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.238+0000 7f814fe00700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 0x7f8150116230 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f813c009030 tx=0x7f813c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.238+0000 7f814cc00700 1 -- 172.21.15.1:0/1257382187 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f813c00e040 con 0x7f8150105af0 2024-02-28T17:10:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.238+0000 7f814cc00700 1 -- 172.21.15.1:0/1257382187 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f813c00bbf0 con 0x7f8150105af0 2024-02-28T17:10:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.238+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815010fe10 con 0x7f8150105af0 2024-02-28T17:10:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.239+0000 7f814cc00700 1 -- 172.21.15.1:0/1257382187 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f813c01c630 con 0x7f8150105af0 2024-02-28T17:10:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.239+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8150110350 con 0x7f8150105af0 2024-02-28T17:10:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.240+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8150107620 con 0x7f8150105af0 2024-02-28T17:10:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.241+0000 7f814cc00700 1 -- 172.21.15.1:0/1257382187 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f813c00bd60 con 0x7f8150105af0 2024-02-28T17:10:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.242+0000 7f814cc00700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7f0 0x7f813806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:09.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.242+0000 7f814cc00700 1 -- 172.21.15.1:0/1257382187 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f813c08e2c0 con 0x7f8150105af0 2024-02-28T17:10:09.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.242+0000 7f8154e00700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7f0 0x7f813806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:09.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.243+0000 7f8154e00700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7f0 0x7f813806fca0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8144007fd0 tx=0x7f8144009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.246+0000 7f814cc00700 1 -- 172.21.15.1:0/1257382187 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f813c05b620 con 0x7f8150105af0 2024-02-28T17:10:09.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.480+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f81501112d0 con 0x7f813806d7f0 2024-02-28T17:10:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.483+0000 7f814cc00700 1 -- 172.21.15.1:0/1257382187 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f81501112d0 con 0x7f813806d7f0 2024-02-28T17:10:09.485 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:10:09.485 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:10:09.486 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:10:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7f0 msgr2=0x7f813806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7f0 0x7f813806fca0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8144007fd0 tx=0x7f8144009040 comp rx=0 tx=0).stop 2024-02-28T17:10:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 msgr2=0x7f8150116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 0x7f8150116230 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f813c009030 tx=0x7f813c009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 shutdown_connections 2024-02-28T17:10:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7f0 0x7f813806fca0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8150105150 0x7f8150115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 --2- 172.21.15.1:0/1257382187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150105af0 0x7f8150116230 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 >> 172.21.15.1:0/1257382187 conn(0x7f81500face0 msgr2=0x7f81500fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 shutdown_connections 2024-02-28T17:10:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.486+0000 7f8157800700 1 -- 172.21.15.1:0/1257382187 wait complete. 2024-02-28T17:10:09.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.630+0000 7ff3e0000700 1 -- 172.21.15.1:0/3612365831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 msgr2=0x7ff3d8109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.630+0000 7ff3e0000700 1 --2- 172.21.15.1:0/3612365831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 0x7ff3d8109260 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4004970 tx=0x7ff3c4009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:09.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.630+0000 7ff3e0000700 1 -- 172.21.15.1:0/3612365831 shutdown_connections 2024-02-28T17:10:09.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.630+0000 7ff3e0000700 1 --2- 172.21.15.1:0/3612365831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 0x7ff3d8109260 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.630+0000 7ff3e0000700 1 --2- 172.21.15.1:0/3612365831 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3d8070330 0x7ff3d8070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.630+0000 7ff3e0000700 1 -- 172.21.15.1:0/3612365831 >> 172.21.15.1:0/3612365831 conn(0x7ff3d80f8ae0 msgr2=0x7ff3d80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:09.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.630+0000 7ff3e0000700 1 -- 172.21.15.1:0/3612365831 shutdown_connections 2024-02-28T17:10:09.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.630+0000 7ff3e0000700 1 -- 172.21.15.1:0/3612365831 wait complete. 2024-02-28T17:10:09.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.631+0000 7ff3e0000700 1 Processor -- start 2024-02-28T17:10:09.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.631+0000 7ff3e0000700 1 -- start start 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.632+0000 7ff3e0000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3d8070330 0x7ff3d8115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.632+0000 7ff3e0000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 0x7ff3d8116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.632+0000 7ff3e0000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8116790 con 0x7ff3d8070c40 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.632+0000 7ff3e0000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d810fae0 con 0x7ff3d8070330 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.632+0000 7ff3dcc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 0x7ff3d8116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.632+0000 7ff3dcc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 0x7ff3d8116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57270/0 (socket says 172.21.15.1:57270) 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.632+0000 7ff3dcc00700 1 -- 172.21.15.1:0/1136875252 learned_addr learned my addr 172.21.15.1:0/1136875252 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.632+0000 7ff3dd600700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3d8070330 0x7ff3d8115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3dcc00700 1 -- 172.21.15.1:0/1136875252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3d8070330 msgr2=0x7ff3d8115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3dcc00700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3d8070330 0x7ff3d8115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3dcc00700 1 -- 172.21.15.1:0/1136875252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c4005040 con 0x7ff3d8070c40 2024-02-28T17:10:09.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3dcc00700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 0x7ff3d8116140 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4004f50 tx=0x7ff3c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3d5600700 1 -- 172.21.15.1:0/1136875252 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3c400e040 con 0x7ff3d8070c40 2024-02-28T17:10:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3d5600700 1 -- 172.21.15.1:0/1136875252 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3c4004290 con 0x7ff3d8070c40 2024-02-28T17:10:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3d5600700 1 -- 172.21.15.1:0/1136875252 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3c401c630 con 0x7ff3d8070c40 2024-02-28T17:10:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d810fd60 con 0x7ff3d8070c40 2024-02-28T17:10:09.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.633+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d81102a0 con 0x7ff3d8070c40 2024-02-28T17:10:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.635+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d8106920 con 0x7ff3d8070c40 2024-02-28T17:10:09.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.636+0000 7ff3d5600700 1 -- 172.21.15.1:0/1136875252 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff3c400bc10 con 0x7ff3d8070c40 2024-02-28T17:10:09.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.636+0000 7ff3d5600700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3c006d7f0 0x7ff3c006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:09.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.637+0000 7ff3d5600700 1 -- 172.21.15.1:0/1136875252 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff3c408e1f0 con 0x7ff3d8070c40 2024-02-28T17:10:09.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.637+0000 7ff3dd600700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3c006d7f0 0x7ff3c006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:09.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.637+0000 7ff3dd600700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3c006d7f0 0x7ff3c006fca0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc007fd0 tx=0x7ff3cc009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:09.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.641+0000 7ff3d5600700 1 -- 172.21.15.1:0/1136875252 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff3c4057b60 con 0x7ff3d8070c40 2024-02-28T17:10:09.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:09 smithi001 ceph-mon[112940]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:09.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:09 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3876008371' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:10:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.953+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff3d804bed0 con 0x7ff3d8070c40 2024-02-28T17:10:09.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.954+0000 7ff3d5600700 1 -- 172.21.15.1:0/1136875252 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7ff3c4013e60 con 0x7ff3d8070c40 2024-02-28T17:10:09.957 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:10:09.957 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:10:09.957 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:10:09.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:10:09.957 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:10:09.957 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:10:09.958 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:10:09.959 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:10:09.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3c006d7f0 msgr2=0x7ff3c006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3c006d7f0 0x7ff3c006fca0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc007fd0 tx=0x7ff3cc009040 comp rx=0 tx=0).stop 2024-02-28T17:10:09.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 msgr2=0x7ff3d8116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:09.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 0x7ff3d8116140 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4004f50 tx=0x7ff3c4009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:09.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 shutdown_connections 2024-02-28T17:10:09.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff3c006d7f0 0x7ff3c006fca0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3d8070330 0x7ff3d8115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 --2- 172.21.15.1:0/1136875252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3d8070c40 0x7ff3d8116140 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:09.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.957+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 >> 172.21.15.1:0/1136875252 conn(0x7ff3d80f8ae0 msgr2=0x7ff3d8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:09.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.958+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 shutdown_connections 2024-02-28T17:10:09.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:09.958+0000 7ff3e0000700 1 -- 172.21.15.1:0/1136875252 wait complete. 2024-02-28T17:10:10.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:09 smithi066 ceph-mon[85781]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:10.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:09 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3876008371' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:10:11.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:10 smithi066 ceph-mon[85781]: pgmap v1072: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:11.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:10 smithi066 ceph-mon[85781]: from='client.35926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:11.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:10 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1136875252' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:10:11.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:10 smithi001 ceph-mon[112940]: pgmap v1072: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:11.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:10 smithi001 ceph-mon[112940]: from='client.35926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:11.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:10 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1136875252' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:10:12.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:10:12.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:10:13.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:12 smithi066 ceph-mon[85781]: pgmap v1073: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:13.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:12 smithi001 ceph-mon[112940]: pgmap v1073: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:15.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:14 smithi066 ceph-mon[85781]: pgmap v1074: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:15.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:14 smithi001 ceph-mon[112940]: pgmap v1074: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:17.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:16 smithi066 ceph-mon[85781]: pgmap v1075: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:17.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:16 smithi001 ceph-mon[112940]: pgmap v1075: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:19.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:18 smithi066 ceph-mon[85781]: pgmap v1076: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:19.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:18 smithi001 ceph-mon[112940]: pgmap v1076: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:21.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:20 smithi066 ceph-mon[85781]: pgmap v1077: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:21.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:20 smithi001 ceph-mon[112940]: pgmap v1077: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:23.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:22 smithi066 ceph-mon[85781]: pgmap v1078: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:23.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:22 smithi001 ceph-mon[112940]: pgmap v1078: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:25.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:24 smithi066 ceph-mon[85781]: pgmap v1079: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:25.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:24 smithi001 ceph-mon[112940]: pgmap v1079: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:27.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:26 smithi001 ceph-mon[112940]: pgmap v1080: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:27.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:10:27.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:26 smithi066 ceph-mon[85781]: pgmap v1080: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:27.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:10:29.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:28 smithi001 ceph-mon[112940]: pgmap v1081: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:29.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:28 smithi066 ceph-mon[85781]: pgmap v1081: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:31.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:30 smithi001 ceph-mon[112940]: pgmap v1082: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:31.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:30 smithi066 ceph-mon[85781]: pgmap v1082: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:33.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:32 smithi001 ceph-mon[112940]: pgmap v1083: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:33.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:32 smithi066 ceph-mon[85781]: pgmap v1083: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:35.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:34 smithi066 ceph-mon[85781]: pgmap v1084: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:35.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:34 smithi001 ceph-mon[112940]: pgmap v1084: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:37.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:36 smithi066 ceph-mon[85781]: pgmap v1085: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:37.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:36 smithi001 ceph-mon[112940]: pgmap v1085: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:39.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:38 smithi066 ceph-mon[85781]: pgmap v1086: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:39.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:38 smithi001 ceph-mon[112940]: pgmap v1086: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.125+0000 7fadc4a00700 1 -- 172.21.15.1:0/2508319422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105b00 msgr2=0x7fadc00ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.125+0000 7fadc4a00700 1 --2- 172.21.15.1:0/2508319422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105b00 0x7fadc00ff170 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fadac004940 tx=0x7fadac009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.125+0000 7fadc4a00700 1 -- 172.21.15.1:0/2508319422 shutdown_connections 2024-02-28T17:10:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.125+0000 7fadc4a00700 1 --2- 172.21.15.1:0/2508319422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105b00 0x7fadc00ff170 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.125+0000 7fadc4a00700 1 --2- 172.21.15.1:0/2508319422 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadc0105160 0x7fadc0105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.125+0000 7fadc4a00700 1 -- 172.21.15.1:0/2508319422 >> 172.21.15.1:0/2508319422 conn(0x7fadc00fad30 msgr2=0x7fadc00fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.125+0000 7fadc4a00700 1 -- 172.21.15.1:0/2508319422 shutdown_connections 2024-02-28T17:10:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.125+0000 7fadc4a00700 1 -- 172.21.15.1:0/2508319422 wait complete. 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.126+0000 7fadc4a00700 1 Processor -- start 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.126+0000 7fadc4a00700 1 -- start start 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.127+0000 7fadc4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105160 0x7fadc0115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.127+0000 7fadc4a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadc0105b00 0x7fadc0116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.127+0000 7fadc4a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0116860 con 0x7fadc0105160 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.127+0000 7fadc4a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc010fbb0 con 0x7fadc0105b00 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.127+0000 7fadbde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105160 0x7fadc0115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.127+0000 7fadbde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105160 0x7fadc0115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60034/0 (socket says 172.21.15.1:60034) 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.127+0000 7fadbde00700 1 -- 172.21.15.1:0/3782907456 learned_addr learned my addr 172.21.15.1:0/3782907456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadbd400700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadc0105b00 0x7fadc0116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadbde00700 1 -- 172.21.15.1:0/3782907456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadc0105b00 msgr2=0x7fadc0116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadbde00700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadc0105b00 0x7fadc0116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadbde00700 1 -- 172.21.15.1:0/3782907456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac005040 con 0x7fadc0105160 2024-02-28T17:10:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadbd400700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadc0105b00 0x7fadc0116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:10:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadbde00700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105160 0x7fadc0115cd0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fadb000c8c0 tx=0x7fadb00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadb6000700 1 -- 172.21.15.1:0/3782907456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadb000a980 con 0x7fadc0105160 2024-02-28T17:10:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadb6000700 1 -- 172.21.15.1:0/3782907456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadb000ce60 con 0x7fadc0105160 2024-02-28T17:10:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc010fe90 con 0x7fadc0105160 2024-02-28T17:10:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.128+0000 7fadb6000700 1 -- 172.21.15.1:0/3782907456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadb00115e0 con 0x7fadc0105160 2024-02-28T17:10:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.129+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc01103d0 con 0x7fadc0105160 2024-02-28T17:10:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.130+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc0107630 con 0x7fadc0105160 2024-02-28T17:10:40.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.131+0000 7fadb6000700 1 -- 172.21.15.1:0/3782907456 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fadb0012400 con 0x7fadc0105160 2024-02-28T17:10:40.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.132+0000 7fadb6000700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fada406d7e0 0x7fada406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.132+0000 7fadb6000700 1 -- 172.21.15.1:0/3782907456 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fadb008ceb0 con 0x7fadc0105160 2024-02-28T17:10:40.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.132+0000 7fadbd400700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fada406d7e0 0x7fada406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.133+0000 7fadbd400700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fada406d7e0 0x7fada406fc90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fadac009fd0 tx=0x7fadac009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:40.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.137+0000 7fadb6000700 1 -- 172.21.15.1:0/3782907456 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fadb0056820 con 0x7fadc0105160 2024-02-28T17:10:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.381+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fadc004bed0 con 0x7fada406d7e0 2024-02-28T17:10:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.384+0000 7fadb6000700 1 -- 172.21.15.1:0/3782907456 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fadc004bed0 con 0x7fada406d7e0 2024-02-28T17:10:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fada406d7e0 msgr2=0x7fada406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fada406d7e0 0x7fada406fc90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fadac009fd0 tx=0x7fadac009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:40.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105160 msgr2=0x7fadc0115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105160 0x7fadc0115cd0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fadb000c8c0 tx=0x7fadb00044c0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 shutdown_connections 2024-02-28T17:10:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fada406d7e0 0x7fada406fc90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc0105160 0x7fadc0115cd0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 --2- 172.21.15.1:0/3782907456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadc0105b00 0x7fadc0116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 >> 172.21.15.1:0/3782907456 conn(0x7fadc00fad30 msgr2=0x7fadc00fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 shutdown_connections 2024-02-28T17:10:40.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.387+0000 7fadc4a00700 1 -- 172.21.15.1:0/3782907456 wait complete. 2024-02-28T17:10:40.406 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:10:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.546+0000 7f35e7400700 1 -- 172.21.15.1:0/534185788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105b10 msgr2=0x7f35e00ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.546+0000 7f35e7400700 1 --2- 172.21.15.1:0/534185788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105b10 0x7f35e00ff180 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f35cc004940 tx=0x7f35cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.546+0000 7f35e7400700 1 -- 172.21.15.1:0/534185788 shutdown_connections 2024-02-28T17:10:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.546+0000 7f35e7400700 1 --2- 172.21.15.1:0/534185788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105b10 0x7f35e00ff180 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.546+0000 7f35e7400700 1 --2- 172.21.15.1:0/534185788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35e0105170 0x7f35e0105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.546+0000 7f35e7400700 1 -- 172.21.15.1:0/534185788 >> 172.21.15.1:0/534185788 conn(0x7f35e00face0 msgr2=0x7f35e00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.546+0000 7f35e7400700 1 -- 172.21.15.1:0/534185788 shutdown_connections 2024-02-28T17:10:40.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.546+0000 7f35e7400700 1 -- 172.21.15.1:0/534185788 wait complete. 2024-02-28T17:10:40.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.547+0000 7f35e7400700 1 Processor -- start 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.547+0000 7f35e7400700 1 -- start start 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105170 0x7f35e0115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e7400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35e0105b10 0x7f35e0116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e7400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e0116870 con 0x7f35e0105170 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e7400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e010fb80 con 0x7f35e0105b10 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105170 0x7f35e0115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105170 0x7f35e0115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60060/0 (socket says 172.21.15.1:60060) 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e4a00700 1 -- 172.21.15.1:0/2024082958 learned_addr learned my addr 172.21.15.1:0/2024082958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35dfe00700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35e0105b10 0x7f35e0116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e4a00700 1 -- 172.21.15.1:0/2024082958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35e0105b10 msgr2=0x7f35e0116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e4a00700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35e0105b10 0x7f35e0116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35e4a00700 1 -- 172.21.15.1:0/2024082958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc005040 con 0x7f35e0105170 2024-02-28T17:10:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.548+0000 7f35dfe00700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35e0105b10 0x7f35e0116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:10:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.549+0000 7f35e4a00700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105170 0x7f35e0115ce0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f35d4002a50 tx=0x7f35d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.549+0000 7f35dcc00700 1 -- 172.21.15.1:0/2024082958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35d4007cc0 con 0x7f35e0105170 2024-02-28T17:10:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.549+0000 7f35dcc00700 1 -- 172.21.15.1:0/2024082958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35d400f040 con 0x7f35e0105170 2024-02-28T17:10:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.549+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e010fe60 con 0x7f35e0105170 2024-02-28T17:10:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.549+0000 7f35dcc00700 1 -- 172.21.15.1:0/2024082958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35d4009640 con 0x7f35e0105170 2024-02-28T17:10:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.549+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e01103a0 con 0x7f35e0105170 2024-02-28T17:10:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.550+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0107640 con 0x7f35e0105170 2024-02-28T17:10:40.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.552+0000 7f35dcc00700 1 -- 172.21.15.1:0/2024082958 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f35d40097a0 con 0x7f35e0105170 2024-02-28T17:10:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.552+0000 7f35dcc00700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35c806d7f0 0x7f35c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.553+0000 7f35dcc00700 1 -- 172.21.15.1:0/2024082958 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f35d408cc60 con 0x7f35e0105170 2024-02-28T17:10:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.553+0000 7f35dfe00700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35c806d7f0 0x7f35c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.554+0000 7f35dfe00700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35c806d7f0 0x7f35c806fca0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00b3b0 tx=0x7f35cc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:40.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.557+0000 7f35dcc00700 1 -- 172.21.15.1:0/2024082958 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f35d4059fc0 con 0x7f35e0105170 2024-02-28T17:10:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.802+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f35e01112c0 con 0x7f35c806d7f0 2024-02-28T17:10:40.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.804+0000 7f35dcc00700 1 -- 172.21.15.1:0/2024082958 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f35e01112c0 con 0x7f35c806d7f0 2024-02-28T17:10:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35c806d7f0 msgr2=0x7f35c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35c806d7f0 0x7f35c806fca0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00b3b0 tx=0x7f35cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105170 msgr2=0x7f35e0115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105170 0x7f35e0115ce0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f35d4002a50 tx=0x7f35d40046b0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 shutdown_connections 2024-02-28T17:10:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35c806d7f0 0x7f35c806fca0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35e0105170 0x7f35e0115ce0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 --2- 172.21.15.1:0/2024082958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35e0105b10 0x7f35e0116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 >> 172.21.15.1:0/2024082958 conn(0x7f35e00face0 msgr2=0x7f35e00fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:40.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 shutdown_connections 2024-02-28T17:10:40.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.807+0000 7f35e7400700 1 -- 172.21.15.1:0/2024082958 wait complete. 2024-02-28T17:10:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.964+0000 7f0907a00700 1 -- 172.21.15.1:0/3541997614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 msgr2=0x7f09001099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.964+0000 7f0907a00700 1 --2- 172.21.15.1:0/3541997614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 0x7f09001099c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f08f4004940 tx=0x7f08f4009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.965+0000 7f0907a00700 1 -- 172.21.15.1:0/3541997614 shutdown_connections 2024-02-28T17:10:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.965+0000 7f0907a00700 1 --2- 172.21.15.1:0/3541997614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0900103dc0 0x7f0900104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.965+0000 7f0907a00700 1 --2- 172.21.15.1:0/3541997614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 0x7f09001099c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.965+0000 7f0907a00700 1 -- 172.21.15.1:0/3541997614 >> 172.21.15.1:0/3541997614 conn(0x7f0900072d50 msgr2=0x7f0900075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.965+0000 7f0907a00700 1 -- 172.21.15.1:0/3541997614 shutdown_connections 2024-02-28T17:10:40.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.965+0000 7f0907a00700 1 -- 172.21.15.1:0/3541997614 wait complete. 2024-02-28T17:10:40.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f0907a00700 1 Processor -- start 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f0907a00700 1 -- start start 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f0907a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0900103dc0 0x7f090011a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f0907a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 0x7f090011a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f0907a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090011ac70 con 0x7f09001095f0 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f0907a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0900113f80 con 0x7f0900103dc0 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f08ffe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 0x7f090011a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f08ffe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 0x7f090011a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60070/0 (socket says 172.21.15.1:60070) 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.966+0000 7f08ffe00700 1 -- 172.21.15.1:0/634938679 learned_addr learned my addr 172.21.15.1:0/634938679 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f0905000700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0900103dc0 0x7f090011a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f08ffe00700 1 -- 172.21.15.1:0/634938679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0900103dc0 msgr2=0x7f090011a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f08ffe00700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0900103dc0 0x7f090011a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:40.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f08ffe00700 1 -- 172.21.15.1:0/634938679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f4005040 con 0x7f09001095f0 2024-02-28T17:10:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f08ffe00700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 0x7f090011a620 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f08ec002a50 tx=0x7f08ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f08fcc00700 1 -- 172.21.15.1:0/634938679 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08ec007cc0 con 0x7f09001095f0 2024-02-28T17:10:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f08fcc00700 1 -- 172.21.15.1:0/634938679 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08ec00f040 con 0x7f09001095f0 2024-02-28T17:10:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0900114260 con 0x7f09001095f0 2024-02-28T17:10:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.967+0000 7f08fcc00700 1 -- 172.21.15.1:0/634938679 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08ec009640 con 0x7f09001095f0 2024-02-28T17:10:40.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.968+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09001147a0 con 0x7f09001095f0 2024-02-28T17:10:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.969+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0900106850 con 0x7f09001095f0 2024-02-28T17:10:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.970+0000 7f08fcc00700 1 -- 172.21.15.1:0/634938679 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f08ec0097a0 con 0x7f09001095f0 2024-02-28T17:10:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.971+0000 7f08fcc00700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08e806d7f0 0x7f08e806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.971+0000 7f08fcc00700 1 -- 172.21.15.1:0/634938679 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f08ec08cd20 con 0x7f09001095f0 2024-02-28T17:10:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.971+0000 7f0905000700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08e806d7f0 0x7f08e806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.972+0000 7f0905000700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08e806d7f0 0x7f08e806fca0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f08f400b3b0 tx=0x7f08f4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:40.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:40.975+0000 7f08fcc00700 1 -- 172.21.15.1:0/634938679 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f08ec05a000 con 0x7f09001095f0 2024-02-28T17:10:41.064 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:40 smithi066 ceph-mon[85781]: pgmap v1087: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.206+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f09001156c0 con 0x7f08e806d7f0 2024-02-28T17:10:41.209 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:40 smithi001 ceph-mon[112940]: pgmap v1087: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:41.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.238+0000 7f08fcc00700 1 -- 172.21.15.1:0/634938679 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f09001156c0 con 0x7f08e806d7f0 2024-02-28T17:10:41.241 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:10:41.241 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (36m) 13m ago 51m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (33m) 13m ago 51m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (33m) 13m ago 49m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (36m) 13m ago 50m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (29m) 13m ago 43m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (28m) 13m ago 43m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (43m) 13m ago 43m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (13m) 13m ago 43m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (39m) 13m ago 52m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (38m) 13m ago 49m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (35m) 13m ago 52m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (34m) 13m ago 49m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (37m) 13m ago 51m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (37m) 13m ago 49m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:10:41.242 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (32m) 13m ago 48m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:10:41.243 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (32m) 13m ago 48m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:10:41.243 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (31m) 13m ago 47m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:10:41.243 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (31m) 13m ago 46m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:10:41.243 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (30m) 13m ago 46m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:10:41.243 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (29m) 13m ago 45m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:10:41.243 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (37m) 13m ago 50m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:10:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08e806d7f0 msgr2=0x7f08e806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08e806d7f0 0x7f08e806fca0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f08f400b3b0 tx=0x7f08f4009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 msgr2=0x7f090011a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:41.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 0x7f090011a620 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f08ec002a50 tx=0x7f08ec0046b0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 shutdown_connections 2024-02-28T17:10:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08e806d7f0 0x7f08e806fca0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0900103dc0 0x7f090011a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 --2- 172.21.15.1:0/634938679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09001095f0 0x7f090011a620 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.241+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 >> 172.21.15.1:0/634938679 conn(0x7f0900072d50 msgr2=0x7f0900074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.242+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 shutdown_connections 2024-02-28T17:10:41.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.242+0000 7f0907a00700 1 -- 172.21.15.1:0/634938679 wait complete. 2024-02-28T17:10:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.397+0000 7fcfb6800700 1 -- 172.21.15.1:0/2657858787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb01095f0 msgr2=0x7fcfb01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:41.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.397+0000 7fcfb6800700 1 --2- 172.21.15.1:0/2657858787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb01095f0 0x7fcfb01099c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0004940 tx=0x7fcfa0009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:41.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.397+0000 7fcfb6800700 1 -- 172.21.15.1:0/2657858787 shutdown_connections 2024-02-28T17:10:41.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.397+0000 7fcfb6800700 1 --2- 172.21.15.1:0/2657858787 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfb0103dc0 0x7fcfb0104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.397+0000 7fcfb6800700 1 --2- 172.21.15.1:0/2657858787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb01095f0 0x7fcfb01099c0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.397+0000 7fcfb6800700 1 -- 172.21.15.1:0/2657858787 >> 172.21.15.1:0/2657858787 conn(0x7fcfb0072d50 msgr2=0x7fcfb0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:41.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.397+0000 7fcfb6800700 1 -- 172.21.15.1:0/2657858787 shutdown_connections 2024-02-28T17:10:41.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.397+0000 7fcfb6800700 1 -- 172.21.15.1:0/2657858787 wait complete. 2024-02-28T17:10:41.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.398+0000 7fcfb6800700 1 Processor -- start 2024-02-28T17:10:41.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.398+0000 7fcfb6800700 1 -- start start 2024-02-28T17:10:41.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.398+0000 7fcfb6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb0103dc0 0x7fcfb011a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:41.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.398+0000 7fcfb6800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfb01095f0 0x7fcfb011a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:41.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.398+0000 7fcfb6800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb011ac70 con 0x7fcfb0103dc0 2024-02-28T17:10:41.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.398+0000 7fcfb6800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb0113ef0 con 0x7fcfb01095f0 2024-02-28T17:10:41.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfaf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb0103dc0 0x7fcfb011a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:41.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfaf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb0103dc0 0x7fcfb011a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60086/0 (socket says 172.21.15.1:60086) 2024-02-28T17:10:41.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfaf400700 1 -- 172.21.15.1:0/3829709372 learned_addr learned my addr 172.21.15.1:0/3829709372 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:41.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfa7e00700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfb01095f0 0x7fcfb011a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:41.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfaf400700 1 -- 172.21.15.1:0/3829709372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfb01095f0 msgr2=0x7fcfb011a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:41.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfaf400700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfb01095f0 0x7fcfb011a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfaf400700 1 -- 172.21.15.1:0/3829709372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa0005040 con 0x7fcfb0103dc0 2024-02-28T17:10:41.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfa7e00700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfb01095f0 0x7fcfb011a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:10:41.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.399+0000 7fcfaf400700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb0103dc0 0x7fcfb011a050 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009150 tx=0x7fcfa0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:41.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.400+0000 7fcfa7400700 1 -- 172.21.15.1:0/3829709372 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfa000e040 con 0x7fcfb0103dc0 2024-02-28T17:10:41.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.400+0000 7fcfa7400700 1 -- 172.21.15.1:0/3829709372 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfa000bbf0 con 0x7fcfb0103dc0 2024-02-28T17:10:41.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.400+0000 7fcfa7400700 1 -- 172.21.15.1:0/3829709372 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfa001c630 con 0x7fcfb0103dc0 2024-02-28T17:10:41.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.400+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb0114110 con 0x7fcfb0103dc0 2024-02-28T17:10:41.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.400+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb0114650 con 0x7fcfb0103dc0 2024-02-28T17:10:41.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.401+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb0075870 con 0x7fcfb0103dc0 2024-02-28T17:10:41.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.402+0000 7fcfa7400700 1 -- 172.21.15.1:0/3829709372 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcfa000bd60 con 0x7fcfb0103dc0 2024-02-28T17:10:41.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.403+0000 7fcfa7400700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf9406d7f0 0x7fcf9406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:41.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.403+0000 7fcfa7400700 1 -- 172.21.15.1:0/3829709372 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcfa008e290 con 0x7fcfb0103dc0 2024-02-28T17:10:41.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.403+0000 7fcfa7e00700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf9406d7f0 0x7fcf9406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:41.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.404+0000 7fcfa7e00700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf9406d7f0 0x7fcf9406fca0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcf98006fd0 tx=0x7fcf98008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:41.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.408+0000 7fcfa7400700 1 -- 172.21.15.1:0/3829709372 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcfa005b670 con 0x7fcfb0103dc0 2024-02-28T17:10:41.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.717+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcfb01156d0 con 0x7fcfb0103dc0 2024-02-28T17:10:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.719+0000 7fcfa7400700 1 -- 172.21.15.1:0/3829709372 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fcfa005b200 con 0x7fcfb0103dc0 2024-02-28T17:10:41.722 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:10:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:10:41.723 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:10:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:10:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:10:41.724 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:10:41.724 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:10:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.721+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf9406d7f0 msgr2=0x7fcf9406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.721+0000 7fcfb6800700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf9406d7f0 0x7fcf9406fca0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcf98006fd0 tx=0x7fcf98008040 comp rx=0 tx=0).stop 2024-02-28T17:10:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.721+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb0103dc0 msgr2=0x7fcfb011a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.721+0000 7fcfb6800700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb0103dc0 0x7fcfb011a050 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009150 tx=0x7fcfa0009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.722+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 shutdown_connections 2024-02-28T17:10:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.722+0000 7fcfb6800700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf9406d7f0 0x7fcf9406fca0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.722+0000 7fcfb6800700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb0103dc0 0x7fcfb011a050 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.722+0000 7fcfb6800700 1 --2- 172.21.15.1:0/3829709372 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcfb01095f0 0x7fcfb011a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.722+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 >> 172.21.15.1:0/3829709372 conn(0x7fcfb0072d50 msgr2=0x7fcfb00746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.722+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 shutdown_connections 2024-02-28T17:10:41.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.722+0000 7fcfb6800700 1 -- 172.21.15.1:0/3829709372 wait complete. 2024-02-28T17:10:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.883+0000 7fc600400700 1 -- 172.21.15.1:0/562792292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f8102f50 msgr2=0x7fc5f8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.883+0000 7fc600400700 1 --2- 172.21.15.1:0/562792292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f8102f50 0x7fc5f8103320 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec004940 tx=0x7fc5ec009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.883+0000 7fc600400700 1 -- 172.21.15.1:0/562792292 shutdown_connections 2024-02-28T17:10:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.883+0000 7fc600400700 1 --2- 172.21.15.1:0/562792292 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5f81038f0 0x7fc5f80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.883+0000 7fc600400700 1 --2- 172.21.15.1:0/562792292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f8102f50 0x7fc5f8103320 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.883+0000 7fc600400700 1 -- 172.21.15.1:0/562792292 >> 172.21.15.1:0/562792292 conn(0x7fc5f80f8ae0 msgr2=0x7fc5f80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.883+0000 7fc600400700 1 -- 172.21.15.1:0/562792292 shutdown_connections 2024-02-28T17:10:41.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.884+0000 7fc600400700 1 -- 172.21.15.1:0/562792292 wait complete. 2024-02-28T17:10:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.885+0000 7fc600400700 1 Processor -- start 2024-02-28T17:10:41.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.885+0000 7fc600400700 1 -- start start 2024-02-28T17:10:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.885+0000 7fc600400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5f8102f50 0x7fc5f8113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.885+0000 7fc600400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f81038f0 0x7fc5f8113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.885+0000 7fc600400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8114670 con 0x7fc5f81038f0 2024-02-28T17:10:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.886+0000 7fc600400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f810d910 con 0x7fc5f8102f50 2024-02-28T17:10:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.886+0000 7fc5fd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f81038f0 0x7fc5f8113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.886+0000 7fc5fd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f81038f0 0x7fc5f8113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60822/0 (socket says 172.21.15.1:60822) 2024-02-28T17:10:41.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.886+0000 7fc5fd000700 1 -- 172.21.15.1:0/2797466898 learned_addr learned my addr 172.21.15.1:0/2797466898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.886+0000 7fc5fda00700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5f8102f50 0x7fc5f8113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.886+0000 7fc5fd000700 1 -- 172.21.15.1:0/2797466898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5f8102f50 msgr2=0x7fc5f8113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.886+0000 7fc5fd000700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5f8102f50 0x7fc5f8113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.886+0000 7fc5fd000700 1 -- 172.21.15.1:0/2797466898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec005040 con 0x7fc5f81038f0 2024-02-28T17:10:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.887+0000 7fc5fd000700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f81038f0 0x7fc5f8113f90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800b790 tx=0x7fc5e8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.887+0000 7fc5f5600700 1 -- 172.21.15.1:0/2797466898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e800f6f0 con 0x7fc5f81038f0 2024-02-28T17:10:41.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.887+0000 7fc5f5600700 1 -- 172.21.15.1:0/2797466898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5e800fd20 con 0x7fc5f81038f0 2024-02-28T17:10:41.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.887+0000 7fc5f5600700 1 -- 172.21.15.1:0/2797466898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e80103a0 con 0x7fc5f81038f0 2024-02-28T17:10:41.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.887+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f810dbf0 con 0x7fc5f81038f0 2024-02-28T17:10:41.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.887+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f810e130 con 0x7fc5f81038f0 2024-02-28T17:10:41.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.888+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f80fc6e0 con 0x7fc5f81038f0 2024-02-28T17:10:41.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.889+0000 7fc5f5600700 1 -- 172.21.15.1:0/2797466898 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc5e8011400 con 0x7fc5f81038f0 2024-02-28T17:10:41.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.890+0000 7fc5f5600700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc5e006d7f0 0x7fc5e006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:41.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.891+0000 7fc5f5600700 1 -- 172.21.15.1:0/2797466898 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc5e808cd30 con 0x7fc5f81038f0 2024-02-28T17:10:41.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.891+0000 7fc5fda00700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc5e006d7f0 0x7fc5e006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:41.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.891+0000 7fc5fda00700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc5e006d7f0 0x7fc5e006fca0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00b020 tx=0x7fc5ec009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:41.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:41.895+0000 7fc5f5600700 1 -- 172.21.15.1:0/2797466898 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc5e805a090 con 0x7fc5f81038f0 2024-02-28T17:10:42.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:41 smithi001 ceph-mon[112940]: from='client.35934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:42.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:41 smithi001 ceph-mon[112940]: from='client.35938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:42.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:10:42.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:41 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3829709372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:10:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.168+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc5f810f030 con 0x7fc5f81038f0 2024-02-28T17:10:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.168+0000 7fc5f5600700 1 -- 172.21.15.1:0/2797466898 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fc5e8059c20 con 0x7fc5f81038f0 2024-02-28T17:10:42.172 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:10:42.172 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:10:42.172 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:10:42.172 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:10:42.173 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:10:42.174 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:10:42.175 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:10:42.175 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:10:42.175 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:10:42.175 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:10:42.175 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:10:42.175 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:10:42.175 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:10:42.175 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:10:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc5e006d7f0 msgr2=0x7fc5e006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc5e006d7f0 0x7fc5e006fca0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00b020 tx=0x7fc5ec009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f81038f0 msgr2=0x7fc5f8113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f81038f0 0x7fc5f8113f90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800b790 tx=0x7fc5e8009220 comp rx=0 tx=0).stop 2024-02-28T17:10:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 shutdown_connections 2024-02-28T17:10:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc5e006d7f0 0x7fc5e006fca0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5f8102f50 0x7fc5f8113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 --2- 172.21.15.1:0/2797466898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f81038f0 0x7fc5f8113f90 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 >> 172.21.15.1:0/2797466898 conn(0x7fc5f80f8ae0 msgr2=0x7fc5f80fa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 shutdown_connections 2024-02-28T17:10:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.173+0000 7fc600400700 1 -- 172.21.15.1:0/2797466898 wait complete. 2024-02-28T17:10:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:10:42.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:41 smithi066 ceph-mon[85781]: from='client.35934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:42.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:41 smithi066 ceph-mon[85781]: from='client.35938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:42.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:10:42.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:41 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3829709372' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:10:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.335+0000 7f6fef000700 1 -- 172.21.15.1:0/1251857405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105160 msgr2=0x7f6fe8105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.335+0000 7f6fef000700 1 --2- 172.21.15.1:0/1251857405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105160 0x7f6fe8105530 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8004940 tx=0x7f6fd8009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.336+0000 7f6fef000700 1 -- 172.21.15.1:0/1251857405 shutdown_connections 2024-02-28T17:10:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.336+0000 7f6fef000700 1 --2- 172.21.15.1:0/1251857405 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6fe8105b00 0x7f6fe80ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.336+0000 7f6fef000700 1 --2- 172.21.15.1:0/1251857405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105160 0x7f6fe8105530 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.336+0000 7f6fef000700 1 -- 172.21.15.1:0/1251857405 >> 172.21.15.1:0/1251857405 conn(0x7f6fe80fad30 msgr2=0x7f6fe80fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:42.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.336+0000 7f6fef000700 1 -- 172.21.15.1:0/1251857405 shutdown_connections 2024-02-28T17:10:42.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.336+0000 7f6fef000700 1 -- 172.21.15.1:0/1251857405 wait complete. 2024-02-28T17:10:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.337+0000 7f6fef000700 1 Processor -- start 2024-02-28T17:10:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.337+0000 7f6fef000700 1 -- start start 2024-02-28T17:10:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.337+0000 7f6fef000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6fe8105160 0x7f6fe8115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.337+0000 7f6fef000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105b00 0x7f6fe81161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.337+0000 7f6fef000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe81167f0 con 0x7f6fe8105b00 2024-02-28T17:10:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.337+0000 7f6fef000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe810fb40 con 0x7f6fe8105160 2024-02-28T17:10:42.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.338+0000 7f6fe7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105b00 0x7f6fe81161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.338+0000 7f6fe7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105b00 0x7f6fe81161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60842/0 (socket says 172.21.15.1:60842) 2024-02-28T17:10:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.338+0000 7f6fe7400700 1 -- 172.21.15.1:0/1357168630 learned_addr learned my addr 172.21.15.1:0/1357168630 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.338+0000 7f6fe7e00700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6fe8105160 0x7f6fe8115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.338+0000 7f6fe7400700 1 -- 172.21.15.1:0/1357168630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6fe8105160 msgr2=0x7f6fe8115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.338+0000 7f6fe7400700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6fe8105160 0x7f6fe8115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.338+0000 7f6fe7400700 1 -- 172.21.15.1:0/1357168630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd8005040 con 0x7f6fe8105b00 2024-02-28T17:10:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.339+0000 7f6fe7400700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105b00 0x7f6fe81161a0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c800 tx=0x7f6fd4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.339+0000 7f6fdfe00700 1 -- 172.21.15.1:0/1357168630 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd400a760 con 0x7f6fe8105b00 2024-02-28T17:10:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.339+0000 7f6fdfe00700 1 -- 172.21.15.1:0/1357168630 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fd400ad90 con 0x7f6fe8105b00 2024-02-28T17:10:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.339+0000 7f6fdfe00700 1 -- 172.21.15.1:0/1357168630 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd40113a0 con 0x7f6fe8105b00 2024-02-28T17:10:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.339+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe810fe20 con 0x7f6fe8105b00 2024-02-28T17:10:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.339+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe8110360 con 0x7f6fe8105b00 2024-02-28T17:10:42.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.340+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe8102470 con 0x7f6fe8105b00 2024-02-28T17:10:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.342+0000 7f6fdfe00700 1 -- 172.21.15.1:0/1357168630 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6fd4012400 con 0x7f6fe8105b00 2024-02-28T17:10:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.343+0000 7f6fdfe00700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6fcc06d7f0 0x7f6fcc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.343+0000 7f6fdfe00700 1 -- 172.21.15.1:0/1357168630 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6fd408ce80 con 0x7f6fe8105b00 2024-02-28T17:10:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.343+0000 7f6fe7e00700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6fcc06d7f0 0x7f6fcc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:42.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.344+0000 7f6fe7e00700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6fcc06d7f0 0x7f6fcc06fca0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800b020 tx=0x7f6fd8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:42.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.348+0000 7f6fdfe00700 1 -- 172.21.15.1:0/1357168630 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6fd4056730 con 0x7f6fe8105b00 2024-02-28T17:10:42.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.595+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fe8111280 con 0x7f6fcc06d7f0 2024-02-28T17:10:42.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.597+0000 7f6fdfe00700 1 -- 172.21.15.1:0/1357168630 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6fe8111280 con 0x7f6fcc06d7f0 2024-02-28T17:10:42.600 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:10:42.600 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:10:42.600 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:10:42.600 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:10:42.600 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:10:42.601 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:10:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6fcc06d7f0 msgr2=0x7f6fcc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6fcc06d7f0 0x7f6fcc06fca0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800b020 tx=0x7f6fd8009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105b00 msgr2=0x7f6fe81161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105b00 0x7f6fe81161a0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c800 tx=0x7f6fd4004470 comp rx=0 tx=0).stop 2024-02-28T17:10:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 shutdown_connections 2024-02-28T17:10:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6fcc06d7f0 0x7f6fcc06fca0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6fe8105160 0x7f6fe8115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 --2- 172.21.15.1:0/1357168630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe8105b00 0x7f6fe81161a0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.600+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 >> 172.21.15.1:0/1357168630 conn(0x7f6fe80fad30 msgr2=0x7f6fe80fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.601+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 shutdown_connections 2024-02-28T17:10:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.601+0000 7f6fef000700 1 -- 172.21.15.1:0/1357168630 wait complete. 2024-02-28T17:10:42.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.760+0000 7f6119600700 1 -- 172.21.15.1:0/4190304476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105b10 msgr2=0x7f61140ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:42.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.760+0000 7f6119600700 1 --2- 172.21.15.1:0/4190304476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105b10 0x7f61140ff180 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6100004940 tx=0x7f6100009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:42.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.761+0000 7f6119600700 1 -- 172.21.15.1:0/4190304476 shutdown_connections 2024-02-28T17:10:42.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.761+0000 7f6119600700 1 --2- 172.21.15.1:0/4190304476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105b10 0x7f61140ff180 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.761+0000 7f6119600700 1 --2- 172.21.15.1:0/4190304476 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6114105170 0x7f6114105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.761+0000 7f6119600700 1 -- 172.21.15.1:0/4190304476 >> 172.21.15.1:0/4190304476 conn(0x7f61140face0 msgr2=0x7f61140fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:42.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.761+0000 7f6119600700 1 -- 172.21.15.1:0/4190304476 shutdown_connections 2024-02-28T17:10:42.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.761+0000 7f6119600700 1 -- 172.21.15.1:0/4190304476 wait complete. 2024-02-28T17:10:42.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.762+0000 7f6119600700 1 Processor -- start 2024-02-28T17:10:42.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.762+0000 7f6119600700 1 -- start start 2024-02-28T17:10:42.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.763+0000 7f6119600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105170 0x7f6114115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:42.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.763+0000 7f6119600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6114105b10 0x7f61141161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:42.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.763+0000 7f6119600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6114116800 con 0x7f6114105170 2024-02-28T17:10:42.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.763+0000 7f6119600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611410fb50 con 0x7f6114105b10 2024-02-28T17:10:42.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.763+0000 7f6112a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105170 0x7f6114115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:42.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.764+0000 7f6112a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105170 0x7f6114115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60858/0 (socket says 172.21.15.1:60858) 2024-02-28T17:10:42.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.764+0000 7f6112a00700 1 -- 172.21.15.1:0/1371012188 learned_addr learned my addr 172.21.15.1:0/1371012188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:10:42.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.764+0000 7f6112000700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6114105b10 0x7f61141161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:42.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.764+0000 7f6112a00700 1 -- 172.21.15.1:0/1371012188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6114105b10 msgr2=0x7f61141161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:42.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.764+0000 7f6112a00700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6114105b10 0x7f61141161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:42.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.764+0000 7f6112a00700 1 -- 172.21.15.1:0/1371012188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6100005040 con 0x7f6114105170 2024-02-28T17:10:42.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.764+0000 7f6112a00700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105170 0x7f6114115c70 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6104002a50 tx=0x7f61040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:42.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.765+0000 7f610aa00700 1 -- 172.21.15.1:0/1371012188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6104007cc0 con 0x7f6114105170 2024-02-28T17:10:42.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.765+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611410fe30 con 0x7f6114105170 2024-02-28T17:10:42.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.765+0000 7f610aa00700 1 -- 172.21.15.1:0/1371012188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f610400f040 con 0x7f6114105170 2024-02-28T17:10:42.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.765+0000 7f610aa00700 1 -- 172.21.15.1:0/1371012188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6104009640 con 0x7f6114105170 2024-02-28T17:10:42.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.765+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6114110370 con 0x7f6114105170 2024-02-28T17:10:42.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.766+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6114107640 con 0x7f6114105170 2024-02-28T17:10:42.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.767+0000 7f610aa00700 1 -- 172.21.15.1:0/1371012188 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f61040097a0 con 0x7f6114105170 2024-02-28T17:10:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.769+0000 7f610aa00700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60f806d7f0 0x7f60f806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:10:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.769+0000 7f610aa00700 1 -- 172.21.15.1:0/1371012188 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f610408cc60 con 0x7f6114105170 2024-02-28T17:10:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.769+0000 7f6112000700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60f806d7f0 0x7f60f806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:10:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.770+0000 7f6112000700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60f806d7f0 0x7f60f806fca0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f610000b020 tx=0x7f6100009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:10:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:42.775+0000 7f610aa00700 1 -- 172.21.15.1:0/1371012188 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6104056490 con 0x7f6114105170 2024-02-28T17:10:43.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:42 smithi066 ceph-mon[85781]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:43.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:42 smithi066 ceph-mon[85781]: pgmap v1088: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:43.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:42 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2797466898' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:10:43.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:42 smithi001 ceph-mon[112940]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:43.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:42 smithi001 ceph-mon[112940]: pgmap v1088: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:43.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:42 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2797466898' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:10:43.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.077+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f61140fe900 con 0x7f6114105170 2024-02-28T17:10:43.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.078+0000 7f610aa00700 1 -- 172.21.15.1:0/1371012188 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f6104059b50 con 0x7f6114105170 2024-02-28T17:10:43.081 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:10:43.082 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:10:43.082 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:10:43.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:10:43.082 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:10:43.082 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:10:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:10:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:10:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:10:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:10:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:10:43.083 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:10:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:10:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:10:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:10:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:10:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:10:43.084 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:10:43.085 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:10:43.085 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:10:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:10:43.085 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:10:43.085 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:10:43.085 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:10:43.086 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:10:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60f806d7f0 msgr2=0x7f60f806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60f806d7f0 0x7f60f806fca0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f610000b020 tx=0x7f6100009e30 comp rx=0 tx=0).stop 2024-02-28T17:10:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105170 msgr2=0x7f6114115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:10:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105170 0x7f6114115c70 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6104002a50 tx=0x7f61040046b0 comp rx=0 tx=0).stop 2024-02-28T17:10:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 shutdown_connections 2024-02-28T17:10:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f60f806d7f0 0x7f60f806fca0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6114105170 0x7f6114115c70 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 --2- 172.21.15.1:0/1371012188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6114105b10 0x7f61141161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:10:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.080+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 >> 172.21.15.1:0/1371012188 conn(0x7f61140face0 msgr2=0x7f61140fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:10:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.081+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 shutdown_connections 2024-02-28T17:10:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:10:43.081+0000 7f6119600700 1 -- 172.21.15.1:0/1371012188 wait complete. 2024-02-28T17:10:44.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:43 smithi066 ceph-mon[85781]: from='client.35954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:44.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:43 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1371012188' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:10:44.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:43 smithi001 ceph-mon[112940]: from='client.35954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:10:44.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:43 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1371012188' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:10:45.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:44 smithi066 ceph-mon[85781]: pgmap v1089: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:45.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:44 smithi001 ceph-mon[112940]: pgmap v1089: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:47.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:46 smithi066 ceph-mon[85781]: pgmap v1090: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:47.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:46 smithi001 ceph-mon[112940]: pgmap v1090: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:49.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:48 smithi066 ceph-mon[85781]: pgmap v1091: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:48 smithi001 ceph-mon[112940]: pgmap v1091: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:51.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:50 smithi066 ceph-mon[85781]: pgmap v1092: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:51.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:50 smithi001 ceph-mon[112940]: pgmap v1092: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:53.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:52 smithi066 ceph-mon[85781]: pgmap v1093: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:53.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:52 smithi001 ceph-mon[112940]: pgmap v1093: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:54 smithi066 ceph-mon[85781]: pgmap v1094: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:55.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:54 smithi001 ceph-mon[112940]: pgmap v1094: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:56.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:10:56.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:10:57.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:56 smithi066 ceph-mon[85781]: pgmap v1095: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:57.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:56 smithi001 ceph-mon[112940]: pgmap v1095: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:59.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:10:58 smithi066 ceph-mon[85781]: pgmap v1096: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:10:59.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:10:58 smithi001 ceph-mon[112940]: pgmap v1096: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:01.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:00 smithi066 ceph-mon[85781]: pgmap v1097: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:01.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:00 smithi001 ceph-mon[112940]: pgmap v1097: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:02 smithi066 ceph-mon[85781]: pgmap v1098: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:03.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:02 smithi001 ceph-mon[112940]: pgmap v1098: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:05.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:04 smithi066 ceph-mon[85781]: pgmap v1099: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:05.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:04 smithi001 ceph-mon[112940]: pgmap v1099: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:07.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:06 smithi066 ceph-mon[85781]: pgmap v1100: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:07.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:06 smithi001 ceph-mon[112940]: pgmap v1100: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:08 smithi066 ceph-mon[85781]: pgmap v1101: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:09.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:08 smithi001 ceph-mon[112940]: pgmap v1101: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:11.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:10 smithi066 ceph-mon[85781]: pgmap v1102: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:11.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:11:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:10 smithi001 ceph-mon[112940]: pgmap v1102: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:11:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.254+0000 7f7833000700 1 -- 172.21.15.1:0/4038989715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 msgr2=0x7f782c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.254+0000 7f7833000700 1 --2- 172.21.15.1:0/4038989715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 0x7f782c1092c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f7818004970 tx=0x7f7818009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:13.257 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:12 smithi001 ceph-mon[112940]: pgmap v1103: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.254+0000 7f7833000700 1 -- 172.21.15.1:0/4038989715 shutdown_connections 2024-02-28T17:11:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.254+0000 7f7833000700 1 --2- 172.21.15.1:0/4038989715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 0x7f782c1092c0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.254+0000 7f7833000700 1 --2- 172.21.15.1:0/4038989715 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f782c0702d0 0x7f782c0706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.254+0000 7f7833000700 1 -- 172.21.15.1:0/4038989715 >> 172.21.15.1:0/4038989715 conn(0x7f782c0f8ae0 msgr2=0x7f782c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.254+0000 7f7833000700 1 -- 172.21.15.1:0/4038989715 shutdown_connections 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.254+0000 7f7833000700 1 -- 172.21.15.1:0/4038989715 wait complete. 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.255+0000 7f7833000700 1 Processor -- start 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.255+0000 7f7833000700 1 -- start start 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.256+0000 7f7833000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f782c0702d0 0x7f782c115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.256+0000 7f7833000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 0x7f782c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.256+0000 7f7833000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c1167a0 con 0x7f782c070be0 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.256+0000 7f7833000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c10faf0 con 0x7f782c0702d0 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.256+0000 7f782b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 0x7f782c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.256+0000 7f782b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 0x7f782c116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42198/0 (socket says 172.21.15.1:42198) 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.256+0000 7f782b400700 1 -- 172.21.15.1:0/3507574999 learned_addr learned my addr 172.21.15.1:0/3507574999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.256+0000 7f782be00700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f782c0702d0 0x7f782c115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f782b400700 1 -- 172.21.15.1:0/3507574999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f782c0702d0 msgr2=0x7f782c115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f782b400700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f782c0702d0 0x7f782c115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f782b400700 1 -- 172.21.15.1:0/3507574999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818005040 con 0x7f782c070be0 2024-02-28T17:11:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f782b400700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 0x7f782c116150 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f7818004f50 tx=0x7f7818009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f7823e00700 1 -- 172.21.15.1:0/3507574999 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f781800e040 con 0x7f782c070be0 2024-02-28T17:11:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f7823e00700 1 -- 172.21.15.1:0/3507574999 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7818004290 con 0x7f782c070be0 2024-02-28T17:11:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f7823e00700 1 -- 172.21.15.1:0/3507574999 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f781801c630 con 0x7f782c070be0 2024-02-28T17:11:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c10fd70 con 0x7f782c070be0 2024-02-28T17:11:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.257+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c1102b0 con 0x7f782c070be0 2024-02-28T17:11:13.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:12 smithi066 ceph-mon[85781]: pgmap v1103: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.259+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c106980 con 0x7f782c070be0 2024-02-28T17:11:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.260+0000 7f7823e00700 1 -- 172.21.15.1:0/3507574999 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f781800bc10 con 0x7f782c070be0 2024-02-28T17:11:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.261+0000 7f7823e00700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f781006d7f0 0x7f781006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.261+0000 7f7823e00700 1 -- 172.21.15.1:0/3507574999 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f781808e280 con 0x7f782c070be0 2024-02-28T17:11:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.261+0000 7f782be00700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f781006d7f0 0x7f781006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.262+0000 7f782be00700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f781006d7f0 0x7f781006fca0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f781c004fa0 tx=0x7f781c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:13.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.266+0000 7f7823e00700 1 -- 172.21.15.1:0/3507574999 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7818057b30 con 0x7f782c070be0 2024-02-28T17:11:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.514+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f782c04bed0 con 0x7f781006d7f0 2024-02-28T17:11:13.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.516+0000 7f7823e00700 1 -- 172.21.15.1:0/3507574999 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f782c04bed0 con 0x7f781006d7f0 2024-02-28T17:11:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f781006d7f0 msgr2=0x7f781006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f781006d7f0 0x7f781006fca0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f781c004fa0 tx=0x7f781c00b040 comp rx=0 tx=0).stop 2024-02-28T17:11:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 msgr2=0x7f782c116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:13.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 0x7f782c116150 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f7818004f50 tx=0x7f7818009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 shutdown_connections 2024-02-28T17:11:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f781006d7f0 0x7f781006fca0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f782c0702d0 0x7f782c115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 --2- 172.21.15.1:0/3507574999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f782c070be0 0x7f782c116150 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.519+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 >> 172.21.15.1:0/3507574999 conn(0x7f782c0f8ae0 msgr2=0x7f782c103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.520+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 shutdown_connections 2024-02-28T17:11:13.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.520+0000 7f7833000700 1 -- 172.21.15.1:0/3507574999 wait complete. 2024-02-28T17:11:13.538 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:11:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.673+0000 7fae12200700 1 -- 172.21.15.1:0/2250853784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 msgr2=0x7fae0c103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.673+0000 7fae12200700 1 --2- 172.21.15.1:0/2250853784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 0x7fae0c103310 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fadfc004940 tx=0x7fadfc009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:13.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.673+0000 7fae12200700 1 -- 172.21.15.1:0/2250853784 shutdown_connections 2024-02-28T17:11:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.673+0000 7fae12200700 1 --2- 172.21.15.1:0/2250853784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae0c1038e0 0x7fae0c0fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.673+0000 7fae12200700 1 --2- 172.21.15.1:0/2250853784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 0x7fae0c103310 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.673+0000 7fae12200700 1 -- 172.21.15.1:0/2250853784 >> 172.21.15.1:0/2250853784 conn(0x7fae0c0f8af0 msgr2=0x7fae0c0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.674+0000 7fae12200700 1 -- 172.21.15.1:0/2250853784 shutdown_connections 2024-02-28T17:11:13.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.674+0000 7fae12200700 1 -- 172.21.15.1:0/2250853784 wait complete. 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.675+0000 7fae12200700 1 Processor -- start 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.675+0000 7fae12200700 1 -- start start 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.675+0000 7fae12200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 0x7fae0c070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.675+0000 7fae12200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae0c1038e0 0x7fae0c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.675+0000 7fae12200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c070dd0 con 0x7fae0c102f40 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.675+0000 7fae12200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c070f40 con 0x7fae0c1038e0 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.676+0000 7fae0b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 0x7fae0c070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.676+0000 7fae0b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 0x7fae0c070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42216/0 (socket says 172.21.15.1:42216) 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.676+0000 7fae0b400700 1 -- 172.21.15.1:0/347933333 learned_addr learned my addr 172.21.15.1:0/347933333 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:13.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.676+0000 7fae0aa00700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae0c1038e0 0x7fae0c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.676+0000 7fae0b400700 1 -- 172.21.15.1:0/347933333 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae0c1038e0 msgr2=0x7fae0c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.676+0000 7fae0b400700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae0c1038e0 0x7fae0c0707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.676+0000 7fae0b400700 1 -- 172.21.15.1:0/347933333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc005040 con 0x7fae0c102f40 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.676+0000 7fae0b400700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 0x7fae0c070290 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009100 tx=0x7fadfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.677+0000 7fae03400700 1 -- 172.21.15.1:0/347933333 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadfc00e040 con 0x7fae0c102f40 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.677+0000 7fae03400700 1 -- 172.21.15.1:0/347933333 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadfc004290 con 0x7fae0c102f40 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.677+0000 7fae03400700 1 -- 172.21.15.1:0/347933333 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadfc01c630 con 0x7fae0c102f40 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.677+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c119c80 con 0x7fae0c102f40 2024-02-28T17:11:13.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.677+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c11a0e0 con 0x7fae0c102f40 2024-02-28T17:11:13.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.678+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c0fc6f0 con 0x7fae0c102f40 2024-02-28T17:11:13.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.679+0000 7fae03400700 1 -- 172.21.15.1:0/347933333 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fadfc00bbf0 con 0x7fae0c102f40 2024-02-28T17:11:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.680+0000 7fae03400700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadf006d7e0 0x7fadf006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:13.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.680+0000 7fae03400700 1 -- 172.21.15.1:0/347933333 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fadfc08e230 con 0x7fae0c102f40 2024-02-28T17:11:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.680+0000 7fae0aa00700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadf006d7e0 0x7fadf006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:13.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.681+0000 7fae0aa00700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadf006d7e0 0x7fadf006fc90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fae0c071c90 tx=0x7fadf800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:13.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.685+0000 7fae03400700 1 -- 172.21.15.1:0/347933333 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fadfc057ae0 con 0x7fae0c102f40 2024-02-28T17:11:13.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.927+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae0c0fac70 con 0x7fadf006d7e0 2024-02-28T17:11:13.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.930+0000 7fae03400700 1 -- 172.21.15.1:0/347933333 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fae0c0fac70 con 0x7fadf006d7e0 2024-02-28T17:11:13.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.932+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadf006d7e0 msgr2=0x7fadf006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:13.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadf006d7e0 0x7fadf006fc90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fae0c071c90 tx=0x7fadf800b040 comp rx=0 tx=0).stop 2024-02-28T17:11:13.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 msgr2=0x7fae0c070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:13.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 0x7fae0c070290 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009100 tx=0x7fadfc009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:13.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 shutdown_connections 2024-02-28T17:11:13.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadf006d7e0 0x7fadf006fc90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae0c102f40 0x7fae0c070290 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 --2- 172.21.15.1:0/347933333 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae0c1038e0 0x7fae0c0707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:13.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 >> 172.21.15.1:0/347933333 conn(0x7fae0c0f8af0 msgr2=0x7fae0c0fa560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:13.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 shutdown_connections 2024-02-28T17:11:13.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:13.933+0000 7fae12200700 1 -- 172.21.15.1:0/347933333 wait complete. 2024-02-28T17:11:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.083+0000 7fd85be00700 1 -- 172.21.15.1:0/243511883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fe730 msgr2=0x7fd85c106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.083+0000 7fd85be00700 1 --2- 172.21.15.1:0/243511883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fe730 0x7fd85c106d70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd848004940 tx=0x7fd848009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.083+0000 7fd85be00700 1 -- 172.21.15.1:0/243511883 shutdown_connections 2024-02-28T17:11:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.083+0000 7fd85be00700 1 --2- 172.21.15.1:0/243511883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fe730 0x7fd85c106d70 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.083+0000 7fd85be00700 1 --2- 172.21.15.1:0/243511883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd85c0fdd90 0x7fd85c0fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.083+0000 7fd85be00700 1 -- 172.21.15.1:0/243511883 >> 172.21.15.1:0/243511883 conn(0x7fd85c0f7820 msgr2=0x7fd85c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.083+0000 7fd85be00700 1 -- 172.21.15.1:0/243511883 shutdown_connections 2024-02-28T17:11:14.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.083+0000 7fd85be00700 1 -- 172.21.15.1:0/243511883 wait complete. 2024-02-28T17:11:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.084+0000 7fd85be00700 1 Processor -- start 2024-02-28T17:11:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.084+0000 7fd85be00700 1 -- start start 2024-02-28T17:11:14.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fdd90 0x7fd85c115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85be00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd85c0fe730 0x7fd85c115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c116630 con 0x7fd85c0fdd90 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85be00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c10f8d0 con 0x7fd85c0fe730 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fdd90 0x7fd85c115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fdd90 0x7fd85c115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42224/0 (socket says 172.21.15.1:42224) 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85aa00700 1 -- 172.21.15.1:0/1255512709 learned_addr learned my addr 172.21.15.1:0/1255512709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85a000700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd85c0fe730 0x7fd85c115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85aa00700 1 -- 172.21.15.1:0/1255512709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd85c0fe730 msgr2=0x7fd85c115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85aa00700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd85c0fe730 0x7fd85c115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.085+0000 7fd85aa00700 1 -- 172.21.15.1:0/1255512709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd848005040 con 0x7fd85c0fdd90 2024-02-28T17:11:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.086+0000 7fd85aa00700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fdd90 0x7fd85c115a10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fd84c00b790 tx=0x7fd84c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.086+0000 7fd852a00700 1 -- 172.21.15.1:0/1255512709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd84c00f770 con 0x7fd85c0fdd90 2024-02-28T17:11:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.086+0000 7fd852a00700 1 -- 172.21.15.1:0/1255512709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd84c00fda0 con 0x7fd85c0fdd90 2024-02-28T17:11:14.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.086+0000 7fd852a00700 1 -- 172.21.15.1:0/1255512709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd84c00d310 con 0x7fd85c0fdd90 2024-02-28T17:11:14.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.086+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85c10fbb0 con 0x7fd85c0fdd90 2024-02-28T17:11:14.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.086+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85c1100f0 con 0x7fd85c0fdd90 2024-02-28T17:11:14.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.087+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c1067e0 con 0x7fd85c0fdd90 2024-02-28T17:11:14.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.088+0000 7fd852a00700 1 -- 172.21.15.1:0/1255512709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd84c010420 con 0x7fd85c0fdd90 2024-02-28T17:11:14.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.089+0000 7fd852a00700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd84006d7e0 0x7fd84006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.089+0000 7fd852a00700 1 -- 172.21.15.1:0/1255512709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd84c08ce30 con 0x7fd85c0fdd90 2024-02-28T17:11:14.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.090+0000 7fd85a000700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd84006d7e0 0x7fd84006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.090+0000 7fd85a000700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd84006d7e0 0x7fd84006fc90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd84800b3b0 tx=0x7fd848009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:14.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.094+0000 7fd852a00700 1 -- 172.21.15.1:0/1255512709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd84c05a2d0 con 0x7fd85c0fdd90 2024-02-28T17:11:14.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.324+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd85c110ff0 con 0x7fd84006d7e0 2024-02-28T17:11:14.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.354+0000 7fd852a00700 1 -- 172.21.15.1:0/1255512709 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd85c110ff0 con 0x7fd84006d7e0 2024-02-28T17:11:14.357 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:11:14.357 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (37m) 14m ago 51m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:11:14.357 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (34m) 14m ago 51m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:11:14.357 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (33m) 14m ago 50m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (36m) 14m ago 51m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (29m) 14m ago 44m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (29m) 14m ago 44m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (44m) 14m ago 44m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (14m) 14m ago 44m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (39m) 14m ago 53m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (38m) 14m ago 50m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (35m) 14m ago 53m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (35m) 14m ago 50m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (38m) 14m ago 51m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (38m) 14m ago 50m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (33m) 14m ago 49m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (32m) 14m ago 48m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:11:14.358 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (32m) 14m ago 48m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:11:14.359 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (31m) 14m ago 47m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:11:14.359 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (31m) 14m ago 46m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:11:14.359 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (30m) 14m ago 46m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:11:14.359 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (37m) 14m ago 50m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:11:14.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.357+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd84006d7e0 msgr2=0x7fd84006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.357+0000 7fd85be00700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd84006d7e0 0x7fd84006fc90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd84800b3b0 tx=0x7fd848009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:14.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.357+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fdd90 msgr2=0x7fd85c115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.357+0000 7fd85be00700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fdd90 0x7fd85c115a10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fd84c00b790 tx=0x7fd84c009250 comp rx=0 tx=0).stop 2024-02-28T17:11:14.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.358+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 shutdown_connections 2024-02-28T17:11:14.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.358+0000 7fd85be00700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd84006d7e0 0x7fd84006fc90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.358+0000 7fd85be00700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd85c0fdd90 0x7fd85c115a10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.358+0000 7fd85be00700 1 --2- 172.21.15.1:0/1255512709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd85c0fe730 0x7fd85c115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.358+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 >> 172.21.15.1:0/1255512709 conn(0x7fd85c0f7820 msgr2=0x7fd85c0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:14.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.358+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 shutdown_connections 2024-02-28T17:11:14.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.358+0000 7fd85be00700 1 -- 172.21.15.1:0/1255512709 wait complete. 2024-02-28T17:11:14.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.499+0000 7fbaa6600700 1 -- 172.21.15.1:0/1166955279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 msgr2=0x7fbaa00ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.499+0000 7fbaa6600700 1 --2- 172.21.15.1:0/1166955279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 0x7fbaa00ff170 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fba88009ce0 tx=0x7fba88004d10 comp rx=0 tx=0).stop 2024-02-28T17:11:14.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.499+0000 7fbaa6600700 1 -- 172.21.15.1:0/1166955279 shutdown_connections 2024-02-28T17:11:14.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.499+0000 7fbaa6600700 1 --2- 172.21.15.1:0/1166955279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 0x7fbaa00ff170 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.499+0000 7fbaa6600700 1 --2- 172.21.15.1:0/1166955279 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa0105160 0x7fbaa0105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.499+0000 7fbaa6600700 1 -- 172.21.15.1:0/1166955279 >> 172.21.15.1:0/1166955279 conn(0x7fbaa00fad30 msgr2=0x7fbaa00fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:14.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.499+0000 7fbaa6600700 1 -- 172.21.15.1:0/1166955279 shutdown_connections 2024-02-28T17:11:14.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.499+0000 7fbaa6600700 1 -- 172.21.15.1:0/1166955279 wait complete. 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.500+0000 7fbaa6600700 1 Processor -- start 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.500+0000 7fbaa6600700 1 -- start start 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fbaa6600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa0105160 0x7fbaa0115e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fbaa6600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 0x7fbaa01163d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fbaa6600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa0116ab0 con 0x7fbaa0105b00 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fbaa6600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa010fd30 con 0x7fbaa0105160 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fba96a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 0x7fbaa01163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fba96a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 0x7fbaa01163d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42240/0 (socket says 172.21.15.1:42240) 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fba96a00700 1 -- 172.21.15.1:0/2238446686 learned_addr learned my addr 172.21.15.1:0/2238446686 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fba9f400700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa0105160 0x7fbaa0115e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fba96a00700 1 -- 172.21.15.1:0/2238446686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa0105160 msgr2=0x7fbaa0115e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fba96a00700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa0105160 0x7fbaa0115e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.501+0000 7fba96a00700 1 -- 172.21.15.1:0/2238446686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88009940 con 0x7fbaa0105b00 2024-02-28T17:11:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.502+0000 7fba96a00700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 0x7fbaa01163d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fba8800bec0 tx=0x7fba88004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.502+0000 7fba97e00700 1 -- 172.21.15.1:0/2238446686 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8800e040 con 0x7fbaa0105b00 2024-02-28T17:11:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.502+0000 7fba97e00700 1 -- 172.21.15.1:0/2238446686 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba880128e0 con 0x7fbaa0105b00 2024-02-28T17:11:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.502+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa010ff50 con 0x7fbaa0105b00 2024-02-28T17:11:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.502+0000 7fba97e00700 1 -- 172.21.15.1:0/2238446686 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8801c880 con 0x7fbaa0105b00 2024-02-28T17:11:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.502+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa0110490 con 0x7fbaa0105b00 2024-02-28T17:11:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.503+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa0102470 con 0x7fbaa0105b00 2024-02-28T17:11:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.504+0000 7fba97e00700 1 -- 172.21.15.1:0/2238446686 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fba88012a50 con 0x7fbaa0105b00 2024-02-28T17:11:14.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.505+0000 7fba97e00700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8406d7f0 0x7fba8406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.505+0000 7fba97e00700 1 -- 172.21.15.1:0/2238446686 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fba8808ebd0 con 0x7fbaa0105b00 2024-02-28T17:11:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.506+0000 7fba9f400700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8406d7f0 0x7fba8406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.506+0000 7fba9f400700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8406d7f0 0x7fba8406fca0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fba90006fd0 tx=0x7fba90008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:14.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.510+0000 7fba97e00700 1 -- 172.21.15.1:0/2238446686 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba8805beb0 con 0x7fbaa0105b00 2024-02-28T17:11:14.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.807+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbaa00fe930 con 0x7fbaa0105b00 2024-02-28T17:11:14.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.808+0000 7fba97e00700 1 -- 172.21.15.1:0/2238446686 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fba8805ba40 con 0x7fbaa0105b00 2024-02-28T17:11:14.810 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:11:14.810 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:11:14.811 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:11:14.812 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:11:14.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:11:14.812 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:11:14.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8406d7f0 msgr2=0x7fba8406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8406d7f0 0x7fba8406fca0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fba90006fd0 tx=0x7fba90008040 comp rx=0 tx=0).stop 2024-02-28T17:11:14.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 msgr2=0x7fbaa01163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 0x7fbaa01163d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fba8800bec0 tx=0x7fba88004d10 comp rx=0 tx=0).stop 2024-02-28T17:11:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 shutdown_connections 2024-02-28T17:11:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8406d7f0 0x7fba8406fca0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbaa0105160 0x7fbaa0115e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 --2- 172.21.15.1:0/2238446686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaa0105b00 0x7fbaa01163d0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 >> 172.21.15.1:0/2238446686 conn(0x7fbaa00fad30 msgr2=0x7fbaa00fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 shutdown_connections 2024-02-28T17:11:14.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.811+0000 7fbaa6600700 1 -- 172.21.15.1:0/2238446686 wait complete. 2024-02-28T17:11:14.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.964+0000 7fbc33e00700 1 -- 172.21.15.1:0/4183973050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109380 msgr2=0x7fbc34109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.964+0000 7fbc33e00700 1 --2- 172.21.15.1:0/4183973050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109380 0x7fbc34109750 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009ce0 tx=0x7fbc24004d10 comp rx=0 tx=0).stop 2024-02-28T17:11:14.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.965+0000 7fbc33e00700 1 -- 172.21.15.1:0/4183973050 shutdown_connections 2024-02-28T17:11:14.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.965+0000 7fbc33e00700 1 --2- 172.21.15.1:0/4183973050 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc34109d20 0x7fbc341033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.965+0000 7fbc33e00700 1 --2- 172.21.15.1:0/4183973050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109380 0x7fbc34109750 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.965+0000 7fbc33e00700 1 -- 172.21.15.1:0/4183973050 >> 172.21.15.1:0/4183973050 conn(0x7fbc34072b50 msgr2=0x7fbc34074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.965+0000 7fbc33e00700 1 -- 172.21.15.1:0/4183973050 shutdown_connections 2024-02-28T17:11:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.965+0000 7fbc33e00700 1 -- 172.21.15.1:0/4183973050 wait complete. 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.966+0000 7fbc33e00700 1 Processor -- start 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.966+0000 7fbc33e00700 1 -- start start 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.966+0000 7fbc33e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc34109380 0x7fbc34119fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.966+0000 7fbc33e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109d20 0x7fbc3411a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.966+0000 7fbc33e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3411abe0 con 0x7fbc34109d20 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.966+0000 7fbc33e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34113e60 con 0x7fbc34109380 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.967+0000 7fbc2a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109d20 0x7fbc3411a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.967+0000 7fbc2a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109d20 0x7fbc3411a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42254/0 (socket says 172.21.15.1:42254) 2024-02-28T17:11:14.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.967+0000 7fbc2a000700 1 -- 172.21.15.1:0/502017051 learned_addr learned my addr 172.21.15.1:0/502017051 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:14.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.967+0000 7fbc32a00700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc34109380 0x7fbc34119fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.967+0000 7fbc2a000700 1 -- 172.21.15.1:0/502017051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc34109380 msgr2=0x7fbc34119fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:14.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.967+0000 7fbc2a000700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc34109380 0x7fbc34119fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:14.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.967+0000 7fbc2a000700 1 -- 172.21.15.1:0/502017051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc24009940 con 0x7fbc34109d20 2024-02-28T17:11:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.967+0000 7fbc2a000700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109d20 0x7fbc3411a500 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00b790 tx=0x7fbc1c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.968+0000 7fbc2b400700 1 -- 172.21.15.1:0/502017051 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc1c00f6f0 con 0x7fbc34109d20 2024-02-28T17:11:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.968+0000 7fbc2b400700 1 -- 172.21.15.1:0/502017051 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc1c00fd20 con 0x7fbc34109d20 2024-02-28T17:11:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.968+0000 7fbc2b400700 1 -- 172.21.15.1:0/502017051 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc1c0103a0 con 0x7fbc34109d20 2024-02-28T17:11:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.968+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc341140e0 con 0x7fbc34109d20 2024-02-28T17:11:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.968+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc341146a0 con 0x7fbc34109d20 2024-02-28T17:11:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.969+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3405f5c0 con 0x7fbc34109d20 2024-02-28T17:11:14.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.970+0000 7fbc2b400700 1 -- 172.21.15.1:0/502017051 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbc1c011400 con 0x7fbc34109d20 2024-02-28T17:11:14.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.971+0000 7fbc2b400700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbc1806d7e0 0x7fbc1806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:14.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.971+0000 7fbc2b400700 1 -- 172.21.15.1:0/502017051 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbc1c08cde0 con 0x7fbc34109d20 2024-02-28T17:11:14.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.971+0000 7fbc32a00700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbc1806d7e0 0x7fbc1806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:14.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.972+0000 7fbc32a00700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbc1806d7e0 0x7fbc1806fc90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbc24015040 tx=0x7fbc24004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:14.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:14.976+0000 7fbc2b400700 1 -- 172.21.15.1:0/502017051 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbc1c05a0c0 con 0x7fbc34109d20 2024-02-28T17:11:15.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.234+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbc341104f0 con 0x7fbc34109d20 2024-02-28T17:11:15.237 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:14 smithi001 ceph-mon[112940]: pgmap v1104: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:15.237 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:14 smithi001 ceph-mon[112940]: from='client.35962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:15.237 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:14 smithi001 ceph-mon[112940]: from='client.35966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:15.237 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:14 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2238446686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:11:15.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.235+0000 7fbc2b400700 1 -- 172.21.15.1:0/502017051 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fbc1c054e60 con 0x7fbc34109d20 2024-02-28T17:11:15.238 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:11:15.239 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:11:15.240 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:11:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:11:15.244 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:11:15.244 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:11:15.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.238+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbc1806d7e0 msgr2=0x7fbc1806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:15.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.238+0000 7fbc33e00700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbc1806d7e0 0x7fbc1806fc90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbc24015040 tx=0x7fbc24004d10 comp rx=0 tx=0).stop 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.238+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109d20 msgr2=0x7fbc3411a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.238+0000 7fbc33e00700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109d20 0x7fbc3411a500 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00b790 tx=0x7fbc1c009220 comp rx=0 tx=0).stop 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.239+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 shutdown_connections 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.239+0000 7fbc33e00700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbc1806d7e0 0x7fbc1806fc90 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.239+0000 7fbc33e00700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbc34109380 0x7fbc34119fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.239+0000 7fbc33e00700 1 --2- 172.21.15.1:0/502017051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc34109d20 0x7fbc3411a500 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.239+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 >> 172.21.15.1:0/502017051 conn(0x7fbc34072b50 msgr2=0x7fbc34075630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.239+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 shutdown_connections 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.239+0000 7fbc33e00700 1 -- 172.21.15.1:0/502017051 wait complete. 2024-02-28T17:11:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:11:15.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:14 smithi066 ceph-mon[85781]: pgmap v1104: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:15.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:14 smithi066 ceph-mon[85781]: from='client.35962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:15.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:14 smithi066 ceph-mon[85781]: from='client.35966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:15.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:14 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2238446686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:11:15.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.388+0000 7fad44000700 1 -- 172.21.15.1:0/1434312779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 msgr2=0x7fad3c070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:15.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.388+0000 7fad44000700 1 --2- 172.21.15.1:0/1434312779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 0x7fad3c070680 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fad30004940 tx=0x7fad30009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:15.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.389+0000 7fad44000700 1 -- 172.21.15.1:0/1434312779 shutdown_connections 2024-02-28T17:11:15.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.389+0000 7fad44000700 1 --2- 172.21.15.1:0/1434312779 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad3c070bc0 0x7fad3c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.389+0000 7fad44000700 1 --2- 172.21.15.1:0/1434312779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 0x7fad3c070680 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.389+0000 7fad44000700 1 -- 172.21.15.1:0/1434312779 >> 172.21.15.1:0/1434312779 conn(0x7fad3c0f8ae0 msgr2=0x7fad3c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:15.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.389+0000 7fad44000700 1 -- 172.21.15.1:0/1434312779 shutdown_connections 2024-02-28T17:11:15.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.389+0000 7fad44000700 1 -- 172.21.15.1:0/1434312779 wait complete. 2024-02-28T17:11:15.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.389+0000 7fad44000700 1 Processor -- start 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.390+0000 7fad44000700 1 -- start start 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.390+0000 7fad44000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 0x7fad3c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.390+0000 7fad44000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad3c070bc0 0x7fad3c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.390+0000 7fad44000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c116810 con 0x7fad3c0702b0 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.390+0000 7fad44000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c10fb40 con 0x7fad3c070bc0 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.390+0000 7fad41600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 0x7fad3c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.390+0000 7fad41600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 0x7fad3c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42274/0 (socket says 172.21.15.1:42274) 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.390+0000 7fad41600700 1 -- 172.21.15.1:0/2271814513 learned_addr learned my addr 172.21.15.1:0/2271814513 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad40c00700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad3c070bc0 0x7fad3c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad41600700 1 -- 172.21.15.1:0/2271814513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad3c070bc0 msgr2=0x7fad3c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad41600700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad3c070bc0 0x7fad3c1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad41600700 1 -- 172.21.15.1:0/2271814513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad30005040 con 0x7fad3c0702b0 2024-02-28T17:11:15.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad41600700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 0x7fad3c115c80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fad30009150 tx=0x7fad30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:15.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad39600700 1 -- 172.21.15.1:0/2271814513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad3000e040 con 0x7fad3c0702b0 2024-02-28T17:11:15.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad39600700 1 -- 172.21.15.1:0/2271814513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad3000bbf0 con 0x7fad3c0702b0 2024-02-28T17:11:15.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad39600700 1 -- 172.21.15.1:0/2271814513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad3001c630 con 0x7fad3c0702b0 2024-02-28T17:11:15.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.391+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c10fdc0 con 0x7fad3c0702b0 2024-02-28T17:11:15.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.392+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c110300 con 0x7fad3c0702b0 2024-02-28T17:11:15.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.393+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c1069a0 con 0x7fad3c0702b0 2024-02-28T17:11:15.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.394+0000 7fad39600700 1 -- 172.21.15.1:0/2271814513 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fad3000bd60 con 0x7fad3c0702b0 2024-02-28T17:11:15.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.395+0000 7fad39600700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fad2406d7f0 0x7fad2406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:15.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.395+0000 7fad39600700 1 -- 172.21.15.1:0/2271814513 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fad3008e290 con 0x7fad3c0702b0 2024-02-28T17:11:15.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.395+0000 7fad40c00700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fad2406d7f0 0x7fad2406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:15.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.396+0000 7fad40c00700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fad2406d7f0 0x7fad2406fca0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fad28007fd0 tx=0x7fad28009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:15.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.399+0000 7fad39600700 1 -- 172.21.15.1:0/2271814513 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fad3005b670 con 0x7fad3c0702b0 2024-02-28T17:11:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.644+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad3c04bf40 con 0x7fad2406d7f0 2024-02-28T17:11:15.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.647+0000 7fad39600700 1 -- 172.21.15.1:0/2271814513 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fad3c04bf40 con 0x7fad2406d7f0 2024-02-28T17:11:15.649 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:11:15.650 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:11:15.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fad2406d7f0 msgr2=0x7fad2406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:15.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fad2406d7f0 0x7fad2406fca0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fad28007fd0 tx=0x7fad28009040 comp rx=0 tx=0).stop 2024-02-28T17:11:15.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 msgr2=0x7fad3c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:15.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 0x7fad3c115c80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fad30009150 tx=0x7fad30009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:15.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 shutdown_connections 2024-02-28T17:11:15.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fad2406d7f0 0x7fad2406fca0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad3c0702b0 0x7fad3c115c80 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 --2- 172.21.15.1:0/2271814513 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fad3c070bc0 0x7fad3c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 >> 172.21.15.1:0/2271814513 conn(0x7fad3c0f8ae0 msgr2=0x7fad3c103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:15.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 shutdown_connections 2024-02-28T17:11:15.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.650+0000 7fad44000700 1 -- 172.21.15.1:0/2271814513 wait complete. 2024-02-28T17:11:15.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.806+0000 7f0529200700 1 -- 172.21.15.1:0/3550346987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 msgr2=0x7f0524103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:15.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.806+0000 7f0529200700 1 --2- 172.21.15.1:0/3550346987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 0x7f0524103320 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0514004940 tx=0x7f0514009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:15.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.806+0000 7f0529200700 1 -- 172.21.15.1:0/3550346987 shutdown_connections 2024-02-28T17:11:15.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.806+0000 7f0529200700 1 --2- 172.21.15.1:0/3550346987 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05241038f0 0x7f05240fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.806+0000 7f0529200700 1 --2- 172.21.15.1:0/3550346987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 0x7f0524103320 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.806+0000 7f0529200700 1 -- 172.21.15.1:0/3550346987 >> 172.21.15.1:0/3550346987 conn(0x7f05240f8ae0 msgr2=0x7f05240faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:15.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.807+0000 7f0529200700 1 -- 172.21.15.1:0/3550346987 shutdown_connections 2024-02-28T17:11:15.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.807+0000 7f0529200700 1 -- 172.21.15.1:0/3550346987 wait complete. 2024-02-28T17:11:15.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.807+0000 7f0529200700 1 Processor -- start 2024-02-28T17:11:15.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f0529200700 1 -- start start 2024-02-28T17:11:15.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f0529200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 0x7f0524115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:15.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f0529200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05241038f0 0x7f0524116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f0529200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0524116790 con 0x7f0524102f50 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f0529200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052410fae0 con 0x7f05241038f0 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f0521e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 0x7f0524115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f0521e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 0x7f0524115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42292/0 (socket says 172.21.15.1:42292) 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f0521e00700 1 -- 172.21.15.1:0/1117350684 learned_addr learned my addr 172.21.15.1:0/1117350684 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.808+0000 7f051be00700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05241038f0 0x7f0524116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f0521e00700 1 -- 172.21.15.1:0/1117350684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05241038f0 msgr2=0x7f0524116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f0521e00700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05241038f0 0x7f0524116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f0521e00700 1 -- 172.21.15.1:0/1117350684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514005040 con 0x7f0524102f50 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f0521e00700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 0x7f0524115c00 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0514009fd0 tx=0x7f0514009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f051a000700 1 -- 172.21.15.1:0/1117350684 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f051400e040 con 0x7f0524102f50 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f051a000700 1 -- 172.21.15.1:0/1117350684 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f051400bbf0 con 0x7f0524102f50 2024-02-28T17:11:15.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052410fd60 con 0x7f0524102f50 2024-02-28T17:11:15.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f051a000700 1 -- 172.21.15.1:0/1117350684 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f051401c3f0 con 0x7f0524102f50 2024-02-28T17:11:15.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.809+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05241102a0 con 0x7f0524102f50 2024-02-28T17:11:15.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.810+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05240fc6e0 con 0x7f0524102f50 2024-02-28T17:11:15.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.811+0000 7f051a000700 1 -- 172.21.15.1:0/1117350684 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f051400bd60 con 0x7f0524102f50 2024-02-28T17:11:15.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.812+0000 7f051a000700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f050806d7f0 0x7f050806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:15.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.812+0000 7f051a000700 1 -- 172.21.15.1:0/1117350684 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f051408e2b0 con 0x7f0524102f50 2024-02-28T17:11:15.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.813+0000 7f051be00700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f050806d7f0 0x7f050806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:15.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.813+0000 7f051be00700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f050806d7f0 0x7f050806fca0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f05241114a0 tx=0x7f050c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:15.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:15.817+0000 7f051a000700 1 -- 172.21.15.1:0/1117350684 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0514057ae0 con 0x7f0524102f50 2024-02-28T17:11:16.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.118+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f05241111e0 con 0x7f0524102f50 2024-02-28T17:11:16.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.118+0000 7f051a000700 1 -- 172.21.15.1:0/1117350684 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f051405b1a0 con 0x7f0524102f50 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:11:16.226 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:11:16.227 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f050806d7f0 msgr2=0x7f050806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f050806d7f0 0x7f050806fca0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f05241114a0 tx=0x7f050c00b040 comp rx=0 tx=0).stop 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 msgr2=0x7f0524115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 0x7f0524115c00 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0514009fd0 tx=0x7f0514009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 shutdown_connections 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f050806d7f0 0x7f050806fca0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0524102f50 0x7f0524115c00 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 --2- 172.21.15.1:0/1117350684 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f05241038f0 0x7f0524116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 >> 172.21.15.1:0/1117350684 conn(0x7f05240f8ae0 msgr2=0x7f05240fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 shutdown_connections 2024-02-28T17:11:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:16.121+0000 7f0529200700 1 -- 172.21.15.1:0/1117350684 wait complete. 2024-02-28T17:11:16.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:15 smithi066 ceph-mon[85781]: from='client.35970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:16.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:15 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/502017051' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:11:16.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:15 smithi001 ceph-mon[112940]: from='client.35970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:16.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:15 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/502017051' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:11:17.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:16 smithi066 ceph-mon[85781]: pgmap v1105: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:17.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:16 smithi066 ceph-mon[85781]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:17.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:16 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1117350684' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:11:17.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:16 smithi001 ceph-mon[112940]: pgmap v1105: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:17.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:16 smithi001 ceph-mon[112940]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:17.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:16 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1117350684' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:11:18.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:17 smithi066 ceph-mon[85781]: pgmap v1106: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:18.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:17 smithi001 ceph-mon[112940]: pgmap v1106: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:20.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:20 smithi066 ceph-mon[85781]: pgmap v1107: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:20.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:20 smithi001 ceph-mon[112940]: pgmap v1107: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:22.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:22 smithi066 ceph-mon[85781]: pgmap v1108: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:22.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:22 smithi001 ceph-mon[112940]: pgmap v1108: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:24.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:24 smithi066 ceph-mon[85781]: pgmap v1109: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:24.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:24 smithi001 ceph-mon[112940]: pgmap v1109: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:26.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:26 smithi066 ceph-mon[85781]: pgmap v1110: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:26.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:11:26.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:26 smithi001 ceph-mon[112940]: pgmap v1110: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:26.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:11:28.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:28 smithi066 ceph-mon[85781]: pgmap v1111: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:28.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:28 smithi001 ceph-mon[112940]: pgmap v1111: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:30.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:30 smithi066 ceph-mon[85781]: pgmap v1112: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:30.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:30 smithi001 ceph-mon[112940]: pgmap v1112: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:32.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:32 smithi066 ceph-mon[85781]: pgmap v1113: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:32.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:32 smithi001 ceph-mon[112940]: pgmap v1113: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:34.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:34 smithi066 ceph-mon[85781]: pgmap v1114: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:34.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:34 smithi001 ceph-mon[112940]: pgmap v1114: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:36.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:36 smithi066 ceph-mon[85781]: pgmap v1115: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:36.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:36 smithi001 ceph-mon[112940]: pgmap v1115: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:38.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:38 smithi066 ceph-mon[85781]: pgmap v1116: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:38.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:38 smithi001 ceph-mon[112940]: pgmap v1116: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:40.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:40 smithi066 ceph-mon[85781]: pgmap v1117: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:40.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:40 smithi001 ceph-mon[112940]: pgmap v1117: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:41.664 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:11:41.866 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:11:42.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:42 smithi066 ceph-mon[85781]: pgmap v1118: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:42.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:42 smithi001 ceph-mon[112940]: pgmap v1118: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:44.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:44 smithi066 ceph-mon[85781]: pgmap v1119: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:44.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:44 smithi001 ceph-mon[112940]: pgmap v1119: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:46.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.291+0000 7f3e06000700 1 -- 172.21.15.1:0/3188430108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 msgr2=0x7f3e00103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:46.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.291+0000 7f3e06000700 1 --2- 172.21.15.1:0/3188430108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 0x7f3e00103320 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3df0004940 tx=0x7f3df0009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:46.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.291+0000 7f3e06000700 1 -- 172.21.15.1:0/3188430108 shutdown_connections 2024-02-28T17:11:46.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.291+0000 7f3e06000700 1 --2- 172.21.15.1:0/3188430108 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e001038f0 0x7f3e000fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.291+0000 7f3e06000700 1 --2- 172.21.15.1:0/3188430108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 0x7f3e00103320 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.291+0000 7f3e06000700 1 -- 172.21.15.1:0/3188430108 >> 172.21.15.1:0/3188430108 conn(0x7f3e000f8ae0 msgr2=0x7f3e000faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:46.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.291+0000 7f3e06000700 1 -- 172.21.15.1:0/3188430108 shutdown_connections 2024-02-28T17:11:46.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.291+0000 7f3e06000700 1 -- 172.21.15.1:0/3188430108 wait complete. 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.292+0000 7f3e06000700 1 Processor -- start 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.292+0000 7f3e06000700 1 -- start start 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3e06000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 0x7f3e00115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3e06000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e001038f0 0x7f3e00116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3e06000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00116790 con 0x7f3e00102f50 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3e06000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0010fae0 con 0x7f3e001038f0 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3dff400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 0x7f3e00115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3dff400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 0x7f3e00115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39334/0 (socket says 172.21.15.1:39334) 2024-02-28T17:11:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3dff400700 1 -- 172.21.15.1:0/3784826837 learned_addr learned my addr 172.21.15.1:0/3784826837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:46.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3df7e00700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e001038f0 0x7f3e00116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:46.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3dff400700 1 -- 172.21.15.1:0/3784826837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e001038f0 msgr2=0x7f3e00116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:46.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3dff400700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e001038f0 0x7f3e00116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.293+0000 7f3dff400700 1 -- 172.21.15.1:0/3784826837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0005040 con 0x7f3e00102f50 2024-02-28T17:11:46.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.294+0000 7f3dff400700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 0x7f3e00115c00 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009150 tx=0x7f3df0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:46.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.294+0000 7f3df7400700 1 -- 172.21.15.1:0/3784826837 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df000e040 con 0x7f3e00102f50 2024-02-28T17:11:46.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.294+0000 7f3df7400700 1 -- 172.21.15.1:0/3784826837 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3df000bbf0 con 0x7f3e00102f50 2024-02-28T17:11:46.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.294+0000 7f3df7400700 1 -- 172.21.15.1:0/3784826837 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df001c630 con 0x7f3e00102f50 2024-02-28T17:11:46.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.294+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0010fd60 con 0x7f3e00102f50 2024-02-28T17:11:46.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.294+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e001102a0 con 0x7f3e00102f50 2024-02-28T17:11:46.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.296+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e000fc6e0 con 0x7f3e00102f50 2024-02-28T17:11:46.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.297+0000 7f3df7400700 1 -- 172.21.15.1:0/3784826837 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3df000bd60 con 0x7f3e00102f50 2024-02-28T17:11:46.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.298+0000 7f3df7400700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de406d7f0 0x7f3de406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:46.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.298+0000 7f3df7400700 1 -- 172.21.15.1:0/3784826837 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3df008e2a0 con 0x7f3e00102f50 2024-02-28T17:11:46.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.298+0000 7f3df7e00700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de406d7f0 0x7f3de406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:46.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.299+0000 7f3df7e00700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de406d7f0 0x7f3de406fca0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3e001114a0 tx=0x7f3de800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:46.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.303+0000 7f3df7400700 1 -- 172.21.15.1:0/3784826837 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3df0057b50 con 0x7f3e00102f50 2024-02-28T17:11:46.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.550+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e001112f0 con 0x7f3de406d7f0 2024-02-28T17:11:46.554 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:46 smithi001 ceph-mon[112940]: pgmap v1120: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:46.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.552+0000 7f3df7400700 1 -- 172.21.15.1:0/3784826837 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3e001112f0 con 0x7f3de406d7f0 2024-02-28T17:11:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de406d7f0 msgr2=0x7f3de406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de406d7f0 0x7f3de406fca0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3e001114a0 tx=0x7f3de800b040 comp rx=0 tx=0).stop 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 msgr2=0x7f3e00115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 0x7f3e00115c00 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009150 tx=0x7f3df0009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 shutdown_connections 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3de406d7f0 0x7f3de406fca0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e00102f50 0x7f3e00115c00 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 --2- 172.21.15.1:0/3784826837 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3e001038f0 0x7f3e00116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 >> 172.21.15.1:0/3784826837 conn(0x7f3e000f8ae0 msgr2=0x7f3e000fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 shutdown_connections 2024-02-28T17:11:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.555+0000 7f3e06000700 1 -- 172.21.15.1:0/3784826837 wait complete. 2024-02-28T17:11:46.576 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:11:46.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.713+0000 7f45c7400700 1 -- 172.21.15.1:0/1044496026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 msgr2=0x7f45c00706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:46.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.713+0000 7f45c7400700 1 --2- 172.21.15.1:0/1044496026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 0x7f45c00706a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f45b4004970 tx=0x7f45b4009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:46.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.713+0000 7f45c7400700 1 -- 172.21.15.1:0/1044496026 shutdown_connections 2024-02-28T17:11:46.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.713+0000 7f45c7400700 1 --2- 172.21.15.1:0/1044496026 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45c0070be0 0x7f45c01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.713+0000 7f45c7400700 1 --2- 172.21.15.1:0/1044496026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 0x7f45c00706a0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.713+0000 7f45c7400700 1 -- 172.21.15.1:0/1044496026 >> 172.21.15.1:0/1044496026 conn(0x7f45c00f8ae0 msgr2=0x7f45c00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:46.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.713+0000 7f45c7400700 1 -- 172.21.15.1:0/1044496026 shutdown_connections 2024-02-28T17:11:46.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.713+0000 7f45c7400700 1 -- 172.21.15.1:0/1044496026 wait complete. 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.714+0000 7f45c7400700 1 Processor -- start 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.714+0000 7f45c7400700 1 -- start start 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 0x7f45c0115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c7400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45c0070be0 0x7f45c0116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c7400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c0116790 con 0x7f45c00702d0 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c7400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c010fae0 con 0x7f45c0070be0 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 0x7f45c0115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 0x7f45c0115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39356/0 (socket says 172.21.15.1:39356) 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c4a00700 1 -- 172.21.15.1:0/4274235150 learned_addr learned my addr 172.21.15.1:0/4274235150 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c4a00700 1 -- 172.21.15.1:0/4274235150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45c0070be0 msgr2=0x7f45c0116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45bfe00700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45c0070be0 0x7f45c0116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c4a00700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45c0070be0 0x7f45c0116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.715+0000 7f45c4a00700 1 -- 172.21.15.1:0/4274235150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b4005040 con 0x7f45c00702d0 2024-02-28T17:11:46.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.716+0000 7f45c4a00700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 0x7f45c0115c00 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005010 tx=0x7f45b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:46.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.716+0000 7f45bcc00700 1 -- 172.21.15.1:0/4274235150 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45b400e040 con 0x7f45c00702d0 2024-02-28T17:11:46.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.716+0000 7f45bcc00700 1 -- 172.21.15.1:0/4274235150 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45b4004290 con 0x7f45c00702d0 2024-02-28T17:11:46.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.716+0000 7f45bcc00700 1 -- 172.21.15.1:0/4274235150 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45b401c3f0 con 0x7f45c00702d0 2024-02-28T17:11:46.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.716+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c010fd60 con 0x7f45c00702d0 2024-02-28T17:11:46.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.716+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c01102a0 con 0x7f45c00702d0 2024-02-28T17:11:46.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.717+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c0106980 con 0x7f45c00702d0 2024-02-28T17:11:46.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.718+0000 7f45bcc00700 1 -- 172.21.15.1:0/4274235150 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f45b4013820 con 0x7f45c00702d0 2024-02-28T17:11:46.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.719+0000 7f45bcc00700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45a806d7f0 0x7f45a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:46.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.719+0000 7f45bcc00700 1 -- 172.21.15.1:0/4274235150 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f45b408e090 con 0x7f45c00702d0 2024-02-28T17:11:46.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.720+0000 7f45bfe00700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45a806d7f0 0x7f45a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:46.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.720+0000 7f45bfe00700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45a806d7f0 0x7f45a806fca0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f45c01114a0 tx=0x7f45ac00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:46.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.725+0000 7f45bcc00700 1 -- 172.21.15.1:0/4274235150 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f45b4057a00 con 0x7f45c00702d0 2024-02-28T17:11:46.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:46 smithi066 ceph-mon[85781]: pgmap v1120: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.963+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45c004bed0 con 0x7f45a806d7f0 2024-02-28T17:11:46.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.966+0000 7f45bcc00700 1 -- 172.21.15.1:0/4274235150 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f45c004bed0 con 0x7f45a806d7f0 2024-02-28T17:11:46.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45a806d7f0 msgr2=0x7f45a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45a806d7f0 0x7f45a806fca0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f45c01114a0 tx=0x7f45ac00b040 comp rx=0 tx=0).stop 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 msgr2=0x7f45c0115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 0x7f45c0115c00 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005010 tx=0x7f45b4009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 shutdown_connections 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f45a806d7f0 0x7f45a806fca0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45c00702d0 0x7f45c0115c00 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 --2- 172.21.15.1:0/4274235150 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45c0070be0 0x7f45c0116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 >> 172.21.15.1:0/4274235150 conn(0x7f45c00f8ae0 msgr2=0x7f45c0103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 shutdown_connections 2024-02-28T17:11:46.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:46.969+0000 7f45c7400700 1 -- 172.21.15.1:0/4274235150 wait complete. 2024-02-28T17:11:47.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.122+0000 7fb41ac00700 1 -- 172.21.15.1:0/2864660921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414071f60 msgr2=0x7fb414070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:47.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.122+0000 7fb41ac00700 1 --2- 172.21.15.1:0/2864660921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414071f60 0x7fb414070330 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb408004970 tx=0x7fb408009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:47.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.122+0000 7fb41ac00700 1 -- 172.21.15.1:0/2864660921 shutdown_connections 2024-02-28T17:11:47.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.122+0000 7fb41ac00700 1 --2- 172.21.15.1:0/2864660921 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb414070870 0x7fb414070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.123+0000 7fb41ac00700 1 --2- 172.21.15.1:0/2864660921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414071f60 0x7fb414070330 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.123+0000 7fb41ac00700 1 -- 172.21.15.1:0/2864660921 >> 172.21.15.1:0/2864660921 conn(0x7fb4140f88e0 msgr2=0x7fb4140facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:47.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.123+0000 7fb41ac00700 1 -- 172.21.15.1:0/2864660921 shutdown_connections 2024-02-28T17:11:47.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.123+0000 7fb41ac00700 1 -- 172.21.15.1:0/2864660921 wait complete. 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.123+0000 7fb41ac00700 1 Processor -- start 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.124+0000 7fb41ac00700 1 -- start start 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.124+0000 7fb41ac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414070870 0x7fb414119ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.124+0000 7fb41ac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb414071f60 0x7fb41411a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.124+0000 7fb41ac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41411aa30 con 0x7fb414070870 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.124+0000 7fb41ac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb414113d40 con 0x7fb414071f60 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.124+0000 7fb419800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414070870 0x7fb414119ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.124+0000 7fb419800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414070870 0x7fb414119ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39380/0 (socket says 172.21.15.1:39380) 2024-02-28T17:11:47.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.124+0000 7fb419800700 1 -- 172.21.15.1:0/2462912602 learned_addr learned my addr 172.21.15.1:0/2462912602 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:47.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.125+0000 7fb419800700 1 -- 172.21.15.1:0/2462912602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb414071f60 msgr2=0x7fb41411a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:47.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.125+0000 7fb418e00700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb414071f60 0x7fb41411a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:47.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.125+0000 7fb419800700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb414071f60 0x7fb41411a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.125+0000 7fb419800700 1 -- 172.21.15.1:0/2462912602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb408005040 con 0x7fb414070870 2024-02-28T17:11:47.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.125+0000 7fb419800700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414070870 0x7fb414119ea0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb408000c00 tx=0x7fb408009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:47.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.125+0000 7fb411600700 1 -- 172.21.15.1:0/2462912602 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb40800e040 con 0x7fb414070870 2024-02-28T17:11:47.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.125+0000 7fb411600700 1 -- 172.21.15.1:0/2462912602 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb40800bc10 con 0x7fb414070870 2024-02-28T17:11:47.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.126+0000 7fb411600700 1 -- 172.21.15.1:0/2462912602 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb40801c610 con 0x7fb414070870 2024-02-28T17:11:47.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.126+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb414113fc0 con 0x7fb414070870 2024-02-28T17:11:47.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.126+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb414114500 con 0x7fb414070870 2024-02-28T17:11:47.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.127+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4140fe220 con 0x7fb414070870 2024-02-28T17:11:47.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.128+0000 7fb411600700 1 -- 172.21.15.1:0/2462912602 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb408013a80 con 0x7fb414070870 2024-02-28T17:11:47.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.129+0000 7fb411600700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3fc06d7e0 0x7fb3fc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:47.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.129+0000 7fb411600700 1 -- 172.21.15.1:0/2462912602 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb40808e380 con 0x7fb414070870 2024-02-28T17:11:47.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.129+0000 7fb418e00700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3fc06d7e0 0x7fb3fc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:47.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.130+0000 7fb418e00700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3fc06d7e0 0x7fb3fc06fc90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb414115740 tx=0x7fb404009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:47.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.134+0000 7fb411600700 1 -- 172.21.15.1:0/2462912602 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb40805b6e0 con 0x7fb414070870 2024-02-28T17:11:47.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.364+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb41404bf40 con 0x7fb3fc06d7e0 2024-02-28T17:11:47.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.386+0000 7fb411600700 1 -- 172.21.15.1:0/2462912602 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb41404bf40 con 0x7fb3fc06d7e0 2024-02-28T17:11:47.390 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:11:47.390 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (37m) 14m ago 52m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:11:47.390 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (34m) 14m ago 52m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:11:47.390 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (34m) 14m ago 51m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:11:47.390 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (37m) 14m ago 51m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:11:47.390 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (30m) 14m ago 45m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:11:47.390 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (30m) 14m ago 44m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (44m) 14m ago 44m 19.4M - 17.2.7-535-gc0f02e60 1c37ed371348 b36b539fc180 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (14m) 14m ago 44m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (40m) 14m ago 53m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (39m) 14m ago 50m 457M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (36m) 14m ago 54m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (35m) 14m ago 50m 48.2M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (38m) 14m ago 52m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (38m) 14m ago 50m 8242k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (33m) 14m ago 49m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (33m) 14m ago 49m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (32m) 14m ago 48m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (32m) 14m ago 48m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (31m) 14m ago 47m 87.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:11:47.391 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (30m) 14m ago 46m 85.9M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:11:47.392 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (38m) 14m ago 51m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:11:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.389+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3fc06d7e0 msgr2=0x7fb3fc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.389+0000 7fb41ac00700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3fc06d7e0 0x7fb3fc06fc90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb414115740 tx=0x7fb404009040 comp rx=0 tx=0).stop 2024-02-28T17:11:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.389+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414070870 msgr2=0x7fb414119ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.389+0000 7fb41ac00700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414070870 0x7fb414119ea0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb408000c00 tx=0x7fb408009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.390+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 shutdown_connections 2024-02-28T17:11:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.390+0000 7fb41ac00700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb3fc06d7e0 0x7fb3fc06fc90 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.390+0000 7fb41ac00700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb414070870 0x7fb414119ea0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.390+0000 7fb41ac00700 1 --2- 172.21.15.1:0/2462912602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb414071f60 0x7fb41411a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.390+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 >> 172.21.15.1:0/2462912602 conn(0x7fb4140f88e0 msgr2=0x7fb4140fa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.390+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 shutdown_connections 2024-02-28T17:11:47.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.390+0000 7fb41ac00700 1 -- 172.21.15.1:0/2462912602 wait complete. 2024-02-28T17:11:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.534+0000 7f7b1d800700 1 -- 172.21.15.1:0/2069801274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 msgr2=0x7f7b18106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.534+0000 7f7b1d800700 1 --2- 172.21.15.1:0/2069801274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 0x7f7b18106f40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7b04004940 tx=0x7f7b04009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.535+0000 7f7b1d800700 1 -- 172.21.15.1:0/2069801274 shutdown_connections 2024-02-28T17:11:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.535+0000 7f7b1d800700 1 --2- 172.21.15.1:0/2069801274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 0x7f7b18106f40 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.535+0000 7f7b1d800700 1 --2- 172.21.15.1:0/2069801274 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b180fdf60 0x7f7b180fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.535+0000 7f7b1d800700 1 -- 172.21.15.1:0/2069801274 >> 172.21.15.1:0/2069801274 conn(0x7f7b180f7a10 msgr2=0x7f7b180f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:47.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.535+0000 7f7b1d800700 1 -- 172.21.15.1:0/2069801274 shutdown_connections 2024-02-28T17:11:47.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.535+0000 7f7b1d800700 1 -- 172.21.15.1:0/2069801274 wait complete. 2024-02-28T17:11:47.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.535+0000 7f7b1d800700 1 Processor -- start 2024-02-28T17:11:47.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.536+0000 7f7b1d800700 1 -- start start 2024-02-28T17:11:47.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.536+0000 7f7b1d800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b180fdf60 0x7f7b18115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:47.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.536+0000 7f7b1d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 0x7f7b18116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:47.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.536+0000 7f7b1d800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b18116840 con 0x7f7b180fe900 2024-02-28T17:11:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.536+0000 7f7b1d800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1810fb00 con 0x7f7b180fdf60 2024-02-28T17:11:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.536+0000 7f7b16000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 0x7f7b18116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.536+0000 7f7b16000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 0x7f7b18116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39408/0 (socket says 172.21.15.1:39408) 2024-02-28T17:11:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.536+0000 7f7b16000700 1 -- 172.21.15.1:0/19863378 learned_addr learned my addr 172.21.15.1:0/19863378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b16a00700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b180fdf60 0x7f7b18115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b16000700 1 -- 172.21.15.1:0/19863378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b180fdf60 msgr2=0x7f7b18115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b16000700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b180fdf60 0x7f7b18115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b16000700 1 -- 172.21.15.1:0/19863378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b04005040 con 0x7f7b180fe900 2024-02-28T17:11:47.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b16000700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 0x7f7b18116160 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f7b04009180 tx=0x7f7b04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:47.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b0ea00700 1 -- 172.21.15.1:0/19863378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0400e040 con 0x7f7b180fe900 2024-02-28T17:11:47.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b0ea00700 1 -- 172.21.15.1:0/19863378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b0400bbf0 con 0x7f7b180fe900 2024-02-28T17:11:47.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1810fd80 con 0x7f7b180fe900 2024-02-28T17:11:47.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.537+0000 7f7b0ea00700 1 -- 172.21.15.1:0/19863378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0401c630 con 0x7f7b180fe900 2024-02-28T17:11:47.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.538+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b181102c0 con 0x7f7b180fe900 2024-02-28T17:11:47.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.539+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b181069b0 con 0x7f7b180fe900 2024-02-28T17:11:47.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.540+0000 7f7b0ea00700 1 -- 172.21.15.1:0/19863378 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7b0400bd60 con 0x7f7b180fe900 2024-02-28T17:11:47.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.540+0000 7f7b0ea00700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7afc06d7f0 0x7f7afc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:47.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.541+0000 7f7b0ea00700 1 -- 172.21.15.1:0/19863378 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7b0408e360 con 0x7f7b180fe900 2024-02-28T17:11:47.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.541+0000 7f7b16a00700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7afc06d7f0 0x7f7afc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:47.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.541+0000 7f7b16a00700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7afc06d7f0 0x7f7afc06fca0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7b08006fd0 tx=0x7f7b08008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:47.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.545+0000 7f7b0ea00700 1 -- 172.21.15.1:0/19863378 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7b04057b90 con 0x7f7b180fe900 2024-02-28T17:11:47.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:47 smithi066 ceph-mon[85781]: from='client.35990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:47.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:47 smithi001 ceph-mon[112940]: from='client.35990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.857+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b1810ff10 con 0x7f7b180fe900 2024-02-28T17:11:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.858+0000 7f7b0ea00700 1 -- 172.21.15.1:0/19863378 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f7b1810ff10 con 0x7f7b180fe900 2024-02-28T17:11:47.861 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:11:47.861 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 3 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 17.2.7-535-gc0f02e60 (c0f02e608ed955f887728ffa27c686f52f9fd400) quincy (stable)": 1, 2024-02-28T17:11:47.862 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 13 2024-02-28T17:11:47.863 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:11:47.863 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:11:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7afc06d7f0 msgr2=0x7f7afc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7afc06d7f0 0x7f7afc06fca0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7b08006fd0 tx=0x7f7b08008040 comp rx=0 tx=0).stop 2024-02-28T17:11:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 msgr2=0x7f7b18116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 0x7f7b18116160 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f7b04009180 tx=0x7f7b04009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 shutdown_connections 2024-02-28T17:11:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7afc06d7f0 0x7f7afc06fca0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7b180fdf60 0x7f7b18115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 --2- 172.21.15.1:0/19863378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b180fe900 0x7f7b18116160 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.860+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 >> 172.21.15.1:0/19863378 conn(0x7f7b180f7a10 msgr2=0x7f7b180fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.861+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 shutdown_connections 2024-02-28T17:11:47.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:47.861+0000 7f7b1d800700 1 -- 172.21.15.1:0/19863378 wait complete. 2024-02-28T17:11:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.017+0000 7fe98b000700 1 -- 172.21.15.1:0/242049123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe984072dd0 msgr2=0x7fe98410d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.017+0000 7fe98b000700 1 --2- 172.21.15.1:0/242049123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe984072dd0 0x7fe98410d6f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fe970004940 tx=0x7fe970009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.017+0000 7fe98b000700 1 -- 172.21.15.1:0/242049123 shutdown_connections 2024-02-28T17:11:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.017+0000 7fe98b000700 1 --2- 172.21.15.1:0/242049123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe984072dd0 0x7fe98410d6f0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.017+0000 7fe98b000700 1 --2- 172.21.15.1:0/242049123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe9840724c0 0x7fe984072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.017+0000 7fe98b000700 1 -- 172.21.15.1:0/242049123 >> 172.21.15.1:0/242049123 conn(0x7fe9840fab10 msgr2=0x7fe9840fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.017+0000 7fe98b000700 1 -- 172.21.15.1:0/242049123 shutdown_connections 2024-02-28T17:11:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.017+0000 7fe98b000700 1 -- 172.21.15.1:0/242049123 wait complete. 2024-02-28T17:11:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.018+0000 7fe98b000700 1 Processor -- start 2024-02-28T17:11:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.019+0000 7fe98b000700 1 -- start start 2024-02-28T17:11:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.019+0000 7fe98b000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9840724c0 0x7fe984119dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.019+0000 7fe98b000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe984072dd0 0x7fe98411a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.019+0000 7fe98b000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98411a950 con 0x7fe9840724c0 2024-02-28T17:11:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.019+0000 7fe98b000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe984113c60 con 0x7fe984072dd0 2024-02-28T17:11:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.019+0000 7fe989c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9840724c0 0x7fe984119dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.019+0000 7fe989c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9840724c0 0x7fe984119dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39420/0 (socket says 172.21.15.1:39420) 2024-02-28T17:11:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.019+0000 7fe989c00700 1 -- 172.21.15.1:0/1703842770 learned_addr learned my addr 172.21.15.1:0/1703842770 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.020+0000 7fe989200700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe984072dd0 0x7fe98411a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.020+0000 7fe989c00700 1 -- 172.21.15.1:0/1703842770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe984072dd0 msgr2=0x7fe98411a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.020+0000 7fe989c00700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe984072dd0 0x7fe98411a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.020+0000 7fe989c00700 1 -- 172.21.15.1:0/1703842770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe970005040 con 0x7fe9840724c0 2024-02-28T17:11:48.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.020+0000 7fe989c00700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9840724c0 0x7fe984119dc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fe978002a50 tx=0x7fe9780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:48.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.020+0000 7fe981600700 1 -- 172.21.15.1:0/1703842770 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe978007cc0 con 0x7fe9840724c0 2024-02-28T17:11:48.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.020+0000 7fe981600700 1 -- 172.21.15.1:0/1703842770 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe97800f040 con 0x7fe9840724c0 2024-02-28T17:11:48.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.020+0000 7fe981600700 1 -- 172.21.15.1:0/1703842770 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe978009640 con 0x7fe9840724c0 2024-02-28T17:11:48.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.021+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe984113f40 con 0x7fe9840724c0 2024-02-28T17:11:48.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.021+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe984114500 con 0x7fe9840724c0 2024-02-28T17:11:48.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.022+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98410ae20 con 0x7fe9840724c0 2024-02-28T17:11:48.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.023+0000 7fe981600700 1 -- 172.21.15.1:0/1703842770 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe9780097a0 con 0x7fe9840724c0 2024-02-28T17:11:48.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.024+0000 7fe981600700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe96c06d7f0 0x7fe96c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.024+0000 7fe981600700 1 -- 172.21.15.1:0/1703842770 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe97808cbc0 con 0x7fe9840724c0 2024-02-28T17:11:48.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.024+0000 7fe989200700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe96c06d7f0 0x7fe96c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.025+0000 7fe989200700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe96c06d7f0 0x7fe96c06fca0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe970000c00 tx=0x7fe970009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.028+0000 7fe981600700 1 -- 172.21.15.1:0/1703842770 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe978059fc0 con 0x7fe9840724c0 2024-02-28T17:11:48.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.309+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe9841153a0 con 0x7fe9840724c0 2024-02-28T17:11:48.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.310+0000 7fe981600700 1 -- 172.21.15.1:0/1703842770 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fe978059b50 con 0x7fe9840724c0 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout:e26 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:11:48.314 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:11:48.315 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:11:48.316 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:24269} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.66:6826/1841260704,v1:172.21.15.66:6827/1841260704] compat {c=[1],r=[1],i=[7ff]}] 2024-02-28T17:11:48.317 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:11:48.317 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:11:48.317 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:11:48.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe96c06d7f0 msgr2=0x7fe96c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe96c06d7f0 0x7fe96c06fca0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe970000c00 tx=0x7fe970009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:48.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9840724c0 msgr2=0x7fe984119dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9840724c0 0x7fe984119dc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fe978002a50 tx=0x7fe9780046b0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 shutdown_connections 2024-02-28T17:11:48.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe96c06d7f0 0x7fe96c06fca0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9840724c0 0x7fe984119dc0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 --2- 172.21.15.1:0/1703842770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe984072dd0 0x7fe98411a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 >> 172.21.15.1:0/1703842770 conn(0x7fe9840fab10 msgr2=0x7fe9840fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:48.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 shutdown_connections 2024-02-28T17:11:48.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.314+0000 7fe98b000700 1 -- 172.21.15.1:0/1703842770 wait complete. 2024-02-28T17:11:48.319 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:11:48.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.475+0000 7f11b1a00700 1 -- 172.21.15.1:0/889319224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 msgr2=0x7f11ac109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.475+0000 7f11b1a00700 1 --2- 172.21.15.1:0/889319224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 0x7f11ac109250 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f1198004970 tx=0x7f1198009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:48.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.475+0000 7f11b1a00700 1 -- 172.21.15.1:0/889319224 shutdown_connections 2024-02-28T17:11:48.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.475+0000 7f11b1a00700 1 --2- 172.21.15.1:0/889319224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 0x7f11ac109250 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.475+0000 7f11b1a00700 1 --2- 172.21.15.1:0/889319224 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11ac070330 0x7f11ac070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.475+0000 7f11b1a00700 1 -- 172.21.15.1:0/889319224 >> 172.21.15.1:0/889319224 conn(0x7f11ac0f8af0 msgr2=0x7f11ac0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:48.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.475+0000 7f11b1a00700 1 -- 172.21.15.1:0/889319224 shutdown_connections 2024-02-28T17:11:48.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.475+0000 7f11b1a00700 1 -- 172.21.15.1:0/889319224 wait complete. 2024-02-28T17:11:48.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.476+0000 7f11b1a00700 1 Processor -- start 2024-02-28T17:11:48.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.477+0000 7f11b1a00700 1 -- start start 2024-02-28T17:11:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.477+0000 7f11b1a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11ac070330 0x7f11ac115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.477+0000 7f11b1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 0x7f11ac116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.477+0000 7f11b1a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac116780 con 0x7f11ac070c40 2024-02-28T17:11:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.477+0000 7f11b1a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac10fad0 con 0x7f11ac070330 2024-02-28T17:11:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.477+0000 7f11aa000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 0x7f11ac116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.477+0000 7f11aa000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 0x7f11ac116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39436/0 (socket says 172.21.15.1:39436) 2024-02-28T17:11:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.477+0000 7f11aa000700 1 -- 172.21.15.1:0/2113787545 learned_addr learned my addr 172.21.15.1:0/2113787545 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:48.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.478+0000 7f11aaa00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11ac070330 0x7f11ac115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.478+0000 7f11aa000700 1 -- 172.21.15.1:0/2113787545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11ac070330 msgr2=0x7f11ac115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.478+0000 7f11aa000700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11ac070330 0x7f11ac115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.478+0000 7f11aa000700 1 -- 172.21.15.1:0/2113787545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1198005040 con 0x7f11ac070c40 2024-02-28T17:11:48.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.478+0000 7f11aa000700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 0x7f11ac116130 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f1198004f50 tx=0x7f1198009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:48.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.478+0000 7f11a2a00700 1 -- 172.21.15.1:0/2113787545 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f119800e040 con 0x7f11ac070c40 2024-02-28T17:11:48.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.478+0000 7f11a2a00700 1 -- 172.21.15.1:0/2113787545 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1198004290 con 0x7f11ac070c40 2024-02-28T17:11:48.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.478+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac10fd50 con 0x7f11ac070c40 2024-02-28T17:11:48.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.479+0000 7f11a2a00700 1 -- 172.21.15.1:0/2113787545 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f119801c630 con 0x7f11ac070c40 2024-02-28T17:11:48.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.479+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac110290 con 0x7f11ac070c40 2024-02-28T17:11:48.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.480+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac106910 con 0x7f11ac070c40 2024-02-28T17:11:48.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.481+0000 7f11a2a00700 1 -- 172.21.15.1:0/2113787545 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f119800bc10 con 0x7f11ac070c40 2024-02-28T17:11:48.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.482+0000 7f11a2a00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f119006d7f0 0x7f119006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.482+0000 7f11a2a00700 1 -- 172.21.15.1:0/2113787545 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f119808e280 con 0x7f11ac070c40 2024-02-28T17:11:48.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.482+0000 7f11aaa00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f119006d7f0 0x7f119006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.483+0000 7f11aaa00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f119006d7f0 0x7f119006fca0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f119c007fd0 tx=0x7f119c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:48.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.487+0000 7f11a2a00700 1 -- 172.21.15.1:0/2113787545 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1198057b50 con 0x7f11ac070c40 2024-02-28T17:11:48.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.729+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11ac10c2a0 con 0x7f119006d7f0 2024-02-28T17:11:48.733 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:48 smithi001 ceph-mon[112940]: from='client.35994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:48.733 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:48 smithi001 ceph-mon[112940]: pgmap v1121: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:48.733 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:48 smithi001 ceph-mon[112940]: from='client.35998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:48.733 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:48 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/19863378' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:11:48.733 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:48 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1703842770' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:11:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.732+0000 7f11a2a00700 1 -- 172.21.15.1:0/2113787545 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f11ac10c2a0 con 0x7f119006d7f0 2024-02-28T17:11:48.736 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:11:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:11:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "osd" 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "15/21 daemons upgraded", 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:11:48.737 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:11:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.736+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f119006d7f0 msgr2=0x7f119006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.736+0000 7f11b1a00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f119006d7f0 0x7f119006fca0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f119c007fd0 tx=0x7f119c009040 comp rx=0 tx=0).stop 2024-02-28T17:11:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.736+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 msgr2=0x7f11ac116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.736+0000 7f11b1a00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 0x7f11ac116130 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f1198004f50 tx=0x7f1198009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.737+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 shutdown_connections 2024-02-28T17:11:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.737+0000 7f11b1a00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f119006d7f0 0x7f119006fca0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.737+0000 7f11b1a00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f11ac070330 0x7f11ac115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.737+0000 7f11b1a00700 1 --2- 172.21.15.1:0/2113787545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac070c40 0x7f11ac116130 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.737+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 >> 172.21.15.1:0/2113787545 conn(0x7f11ac0f8af0 msgr2=0x7f11ac103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.737+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 shutdown_connections 2024-02-28T17:11:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.737+0000 7f11b1a00700 1 -- 172.21.15.1:0/2113787545 wait complete. 2024-02-28T17:11:48.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:48 smithi066 ceph-mon[85781]: from='client.35994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:48.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:48 smithi066 ceph-mon[85781]: pgmap v1121: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:48.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:48 smithi066 ceph-mon[85781]: from='client.35998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:48.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:48 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/19863378' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:11:48.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:48 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1703842770' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:11:48.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.900+0000 7f6e04800700 1 -- 172.21.15.1:0/2047291496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 msgr2=0x7f6dfc106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.900+0000 7f6e04800700 1 --2- 172.21.15.1:0/2047291496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 0x7f6dfc106f20 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6dec004940 tx=0x7f6dec009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.900+0000 7f6e04800700 1 -- 172.21.15.1:0/2047291496 shutdown_connections 2024-02-28T17:11:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.900+0000 7f6e04800700 1 --2- 172.21.15.1:0/2047291496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 0x7f6dfc106f20 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.900+0000 7f6e04800700 1 --2- 172.21.15.1:0/2047291496 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dfc0fdf40 0x7f6dfc0fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.900+0000 7f6e04800700 1 -- 172.21.15.1:0/2047291496 >> 172.21.15.1:0/2047291496 conn(0x7f6dfc0f79d0 msgr2=0x7f6dfc0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.900+0000 7f6e04800700 1 -- 172.21.15.1:0/2047291496 shutdown_connections 2024-02-28T17:11:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.900+0000 7f6e04800700 1 -- 172.21.15.1:0/2047291496 wait complete. 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.901+0000 7f6e04800700 1 Processor -- start 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.901+0000 7f6e04800700 1 -- start start 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e04800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dfc0fdf40 0x7f6dfc115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e04800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 0x7f6dfc1161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e04800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc116820 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e04800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc10fb50 con 0x7f6dfc0fdf40 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e01400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 0x7f6dfc1161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e01400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 0x7f6dfc1161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39446/0 (socket says 172.21.15.1:39446) 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e01400700 1 -- 172.21.15.1:0/275203790 learned_addr learned my addr 172.21.15.1:0/275203790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e01400700 1 -- 172.21.15.1:0/275203790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dfc0fdf40 msgr2=0x7f6dfc115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e01e00700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dfc0fdf40 0x7f6dfc115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e01400700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dfc0fdf40 0x7f6dfc115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.902+0000 7f6e01400700 1 -- 172.21.15.1:0/275203790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec005040 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.903+0000 7f6e01400700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 0x7f6dfc1161d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009180 tx=0x7f6dec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.903+0000 7f6df6000700 1 -- 172.21.15.1:0/275203790 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dec00e040 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.903+0000 7f6df6000700 1 -- 172.21.15.1:0/275203790 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dec004290 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.903+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc10fdd0 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.903+0000 7f6df6000700 1 -- 172.21.15.1:0/275203790 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dec01c630 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.903+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc110310 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.904+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc106990 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.905+0000 7f6df6000700 1 -- 172.21.15.1:0/275203790 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6dec0139f0 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.906+0000 7f6df6000700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6de406d7f0 0x7f6de406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:11:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.906+0000 7f6df6000700 1 -- 172.21.15.1:0/275203790 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6dec08e2f0 con 0x7f6dfc0fe8e0 2024-02-28T17:11:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.907+0000 7f6e01e00700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6de406d7f0 0x7f6de406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:11:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.907+0000 7f6e01e00700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6de406d7f0 0x7f6de406fca0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6df0004c40 tx=0x7f6df0009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:11:48.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:48.911+0000 7f6df6000700 1 -- 172.21.15.1:0/275203790 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6dec05b6d0 con 0x7f6dfc0fe8e0 2024-02-28T17:11:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.222+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6dfc04bed0 con 0x7f6dfc0fe8e0 2024-02-28T17:11:49.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.223+0000 7f6df6000700 1 -- 172.21.15.1:0/275203790 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f6dec05b260 con 0x7f6dfc0fe8e0 2024-02-28T17:11:49.226 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi001 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:11:49.227 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi001 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsid9jymo:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:11:49.228 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6de406d7f0 msgr2=0x7f6de406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6de406d7f0 0x7f6de406fca0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6df0004c40 tx=0x7f6df0009040 comp rx=0 tx=0).stop 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 msgr2=0x7f6dfc1161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 0x7f6dfc1161d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009180 tx=0x7f6dec009e30 comp rx=0 tx=0).stop 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 shutdown_connections 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6de406d7f0 0x7f6de406fca0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6dfc0fdf40 0x7f6dfc115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 --2- 172.21.15.1:0/275203790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dfc0fe8e0 0x7f6dfc1161d0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:11:49.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.226+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 >> 172.21.15.1:0/275203790 conn(0x7f6dfc0f79d0 msgr2=0x7f6dfc0fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:11:49.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.227+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 shutdown_connections 2024-02-28T17:11:49.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:11:49.227+0000 7f6e04800700 1 -- 172.21.15.1:0/275203790 wait complete. 2024-02-28T17:11:49.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:49 smithi066 ceph-mon[85781]: from='client.36010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:49.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:49 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/275203790' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:11:49.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:49 smithi001 ceph-mon[112940]: from='client.36010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:11:49.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:49 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/275203790' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:11:50.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:50 smithi066 ceph-mon[85781]: pgmap v1122: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:50.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:50 smithi001 ceph-mon[112940]: pgmap v1122: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:52.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:52 smithi066 ceph-mon[85781]: pgmap v1123: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:52.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:52 smithi001 ceph-mon[112940]: pgmap v1123: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:54.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:54 smithi066 ceph-mon[85781]: pgmap v1124: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:54.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:54 smithi001 ceph-mon[112940]: pgmap v1124: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:56.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:56 smithi066 ceph-mon[85781]: pgmap v1125: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:56.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:11:56.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:56 smithi001 ceph-mon[112940]: pgmap v1125: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:56.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:11:58.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:11:58 smithi066 ceph-mon[85781]: pgmap v1126: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:11:58.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:11:58 smithi001 ceph-mon[112940]: pgmap v1126: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:00.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:00 smithi066 ceph-mon[85781]: pgmap v1127: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:00.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:00 smithi001 ceph-mon[112940]: pgmap v1127: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:02.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:02 smithi066 ceph-mon[85781]: pgmap v1128: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:02.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:02 smithi001 ceph-mon[112940]: pgmap v1128: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:04.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:04 smithi066 ceph-mon[85781]: pgmap v1129: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:04.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:04 smithi001 ceph-mon[112940]: pgmap v1129: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:05.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:12:05.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:12:05.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:05 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:05.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:12:05.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:12:05.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:05 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:06.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:06 smithi066 ceph-mon[85781]: pgmap v1130: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:06.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:06 smithi066 ceph-mon[85781]: pgmap v1131: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:06.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:06 smithi066 ceph-mon[85781]: pgmap v1132: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:06.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:12:06.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:06.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:06.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:06.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:06 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:06 smithi001 ceph-mon[112940]: pgmap v1130: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:06 smithi001 ceph-mon[112940]: pgmap v1131: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:06 smithi001 ceph-mon[112940]: pgmap v1132: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:12:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:06 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:08.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:08 smithi066 ceph-mon[85781]: pgmap v1133: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:08.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:08.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.pivimo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T17:12:08.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:08 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:12:08.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:08 smithi001 ceph-mon[112940]: pgmap v1133: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:08.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:08.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi066.pivimo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-28T17:12:08.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:08 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:12:09.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:09 smithi001 ceph-mon[112940]: Upgrade: Updating mds.cephfs.smithi066.pivimo 2024-02-28T17:12:09.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:09 smithi001 ceph-mon[112940]: Deploying daemon mds.cephfs.smithi066.pivimo on smithi066 2024-02-28T17:12:09.880 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:09 smithi066 ceph-mon[85781]: Upgrade: Updating mds.cephfs.smithi066.pivimo 2024-02-28T17:12:09.880 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:09 smithi066 ceph-mon[85781]: Deploying daemon mds.cephfs.smithi066.pivimo on smithi066 2024-02-28T17:12:10.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:10 smithi066 ceph-mon[85781]: pgmap v1134: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:10.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:10 smithi001 ceph-mon[112940]: pgmap v1134: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:12.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:12:12.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:12:12.900 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:12 smithi066 ceph-mon[85781]: pgmap v1135: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:12.900 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:12 smithi066 ceph-mon[85781]: osdmap e84: 6 total, 6 up, 6 in 2024-02-28T17:12:12.900 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:12 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-02-28T17:12:12.900 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:12 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T17:12:13.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:12 smithi001 ceph-mon[112940]: pgmap v1135: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:13.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:12 smithi001 ceph-mon[112940]: osdmap e84: 6 total, 6 up, 6 in 2024-02-28T17:12:13.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:12 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-02-28T17:12:13.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:12 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T17:12:14.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:14 smithi066 ceph-mon[85781]: pgmap v1137: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:15.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:14 smithi001 ceph-mon[112940]: pgmap v1137: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:16.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: pgmap v1138: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:16.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:12:16.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] up:boot 2024-02-28T17:12:16.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T17:12:16.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T17:12:16.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:16 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: pgmap v1138: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:16.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:12:16.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] up:boot 2024-02-28T17:12:16.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T17:12:16.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T17:12:16.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:16.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:16 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:17.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:17 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:17.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:17 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:18.764 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:18 smithi066 ceph-mon[85781]: pgmap v1139: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:18.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:18 smithi001 ceph-mon[112940]: pgmap v1139: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:19.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.407+0000 7f7fe2000700 1 -- 172.21.15.1:0/602645098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc102f50 msgr2=0x7f7fdc103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:19.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.407+0000 7f7fe2000700 1 --2- 172.21.15.1:0/602645098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc102f50 0x7f7fdc103320 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc004940 tx=0x7f7fcc009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.407+0000 7f7fe2000700 1 -- 172.21.15.1:0/602645098 shutdown_connections 2024-02-28T17:12:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.407+0000 7f7fe2000700 1 --2- 172.21.15.1:0/602645098 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc1038f0 0x7f7fdc0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.407+0000 7f7fe2000700 1 --2- 172.21.15.1:0/602645098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc102f50 0x7f7fdc103320 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.407+0000 7f7fe2000700 1 -- 172.21.15.1:0/602645098 >> 172.21.15.1:0/602645098 conn(0x7f7fdc0f8ae0 msgr2=0x7f7fdc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.407+0000 7f7fe2000700 1 -- 172.21.15.1:0/602645098 shutdown_connections 2024-02-28T17:12:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.407+0000 7f7fe2000700 1 -- 172.21.15.1:0/602645098 wait complete. 2024-02-28T17:12:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.408+0000 7f7fe2000700 1 Processor -- start 2024-02-28T17:12:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.409+0000 7f7fe2000700 1 -- start start 2024-02-28T17:12:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.409+0000 7f7fe2000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc102f50 0x7f7fdc115be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.409+0000 7f7fe2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1038f0 0x7f7fdc116120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.409+0000 7f7fe2000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc116770 con 0x7f7fdc1038f0 2024-02-28T17:12:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.409+0000 7f7fe2000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc10faa0 con 0x7f7fdc102f50 2024-02-28T17:12:19.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.410+0000 7f7fdaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1038f0 0x7f7fdc116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.410+0000 7f7fdaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1038f0 0x7f7fdc116120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45970/0 (socket says 172.21.15.1:45970) 2024-02-28T17:12:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.410+0000 7f7fdaa00700 1 -- 172.21.15.1:0/1954028545 learned_addr learned my addr 172.21.15.1:0/1954028545 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.410+0000 7f7fdb400700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc102f50 0x7f7fdc115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.410+0000 7f7fdaa00700 1 -- 172.21.15.1:0/1954028545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc102f50 msgr2=0x7f7fdc115be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.410+0000 7f7fdaa00700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc102f50 0x7f7fdc115be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.410+0000 7f7fdaa00700 1 -- 172.21.15.1:0/1954028545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc8005040 con 0x7f7fdc1038f0 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.411+0000 7f7fdaa00700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1038f0 0x7f7fdc116120 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009700 tx=0x7f7fc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.411+0000 7f7fd3400700 1 -- 172.21.15.1:0/1954028545 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fc800e7e0 con 0x7f7fdc1038f0 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.411+0000 7f7fd3400700 1 -- 172.21.15.1:0/1954028545 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fc800f040 con 0x7f7fdc1038f0 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.411+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc005040 con 0x7f7fdc1038f0 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.411+0000 7f7fd3400700 1 -- 172.21.15.1:0/1954028545 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fc8013d40 con 0x7f7fdc1038f0 2024-02-28T17:12:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.411+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc1100d0 con 0x7f7fdc1038f0 2024-02-28T17:12:19.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.412+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc0fc6e0 con 0x7f7fdc1038f0 2024-02-28T17:12:19.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.415+0000 7f7fd3400700 1 -- 172.21.15.1:0/1954028545 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7fc800e940 con 0x7f7fdc1038f0 2024-02-28T17:12:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.416+0000 7f7fd3400700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7fc006d7e0 0x7f7fc006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.416+0000 7f7fd3400700 1 -- 172.21.15.1:0/1954028545 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7fc808cc20 con 0x7f7fdc1038f0 2024-02-28T17:12:19.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.416+0000 7f7fdb400700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7fc006d7e0 0x7f7fc006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:19.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.417+0000 7f7fdb400700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7fc006d7e0 0x7f7fc006fc90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009fd0 tx=0x7f7fcc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.426+0000 7f7fd3400700 1 -- 172.21.15.1:0/1954028545 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7fc8059fa0 con 0x7f7fdc1038f0 2024-02-28T17:12:19.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.682+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fdc1102f0 con 0x7f7fc006d7e0 2024-02-28T17:12:19.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.683+0000 7f7fd3400700 1 -- 172.21.15.1:0/1954028545 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7fdc1102f0 con 0x7f7fc006d7e0 2024-02-28T17:12:19.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7fc006d7e0 msgr2=0x7f7fc006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:19.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7fc006d7e0 0x7f7fc006fc90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009fd0 tx=0x7f7fcc009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:19.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1038f0 msgr2=0x7f7fdc116120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:19.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1038f0 0x7f7fdc116120 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009700 tx=0x7f7fc8009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:19.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 shutdown_connections 2024-02-28T17:12:19.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7fc006d7e0 0x7f7fc006fc90 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7fdc102f50 0x7f7fdc115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 --2- 172.21.15.1:0/1954028545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fdc1038f0 0x7f7fdc116120 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.686+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 >> 172.21.15.1:0/1954028545 conn(0x7f7fdc0f8ae0 msgr2=0x7f7fdc0fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:19.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.687+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 shutdown_connections 2024-02-28T17:12:19.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.687+0000 7f7fe2000700 1 -- 172.21.15.1:0/1954028545 wait complete. 2024-02-28T17:12:19.705 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:12:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.853+0000 7f99b1e00700 1 -- 172.21.15.1:0/2864534595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac1027e0 msgr2=0x7f99ac0fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.853+0000 7f99b1e00700 1 --2- 172.21.15.1:0/2864534595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac1027e0 0x7f99ac0fbea0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f9998004940 tx=0x7f9998009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.853+0000 7f99b1e00700 1 -- 172.21.15.1:0/2864534595 shutdown_connections 2024-02-28T17:12:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.853+0000 7f99b1e00700 1 --2- 172.21.15.1:0/2864534595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac1027e0 0x7f99ac0fbea0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.853+0000 7f99b1e00700 1 --2- 172.21.15.1:0/2864534595 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ac101e40 0x7f99ac102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.853+0000 7f99b1e00700 1 -- 172.21.15.1:0/2864534595 >> 172.21.15.1:0/2864534595 conn(0x7f99ac071d90 msgr2=0x7f99ac072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.853+0000 7f99b1e00700 1 -- 172.21.15.1:0/2864534595 shutdown_connections 2024-02-28T17:12:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.853+0000 7f99b1e00700 1 -- 172.21.15.1:0/2864534595 wait complete. 2024-02-28T17:12:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.854+0000 7f99b1e00700 1 Processor -- start 2024-02-28T17:12:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.854+0000 7f99b1e00700 1 -- start start 2024-02-28T17:12:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.854+0000 7f99b1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac101e40 0x7f99ac10c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.854+0000 7f99b1e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ac1027e0 0x7f99ac10cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.854+0000 7f99b1e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac10d3e0 con 0x7f99ac101e40 2024-02-28T17:12:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.854+0000 7f99b1e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac10d550 con 0x7f99ac1027e0 2024-02-28T17:12:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.855+0000 7f99aaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac101e40 0x7f99ac10c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.855+0000 7f99aaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac101e40 0x7f99ac10c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45978/0 (socket says 172.21.15.1:45978) 2024-02-28T17:12:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.855+0000 7f99aaa00700 1 -- 172.21.15.1:0/702180024 learned_addr learned my addr 172.21.15.1:0/702180024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.855+0000 7f99aa000700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ac1027e0 0x7f99ac10cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.855+0000 7f99aaa00700 1 -- 172.21.15.1:0/702180024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ac1027e0 msgr2=0x7f99ac10cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.855+0000 7f99aaa00700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ac1027e0 0x7f99ac10cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.855+0000 7f99aaa00700 1 -- 172.21.15.1:0/702180024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9998005040 con 0x7f99ac101e40 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.856+0000 7f99aaa00700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac101e40 0x7f99ac10c8a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f999c00c8c0 tx=0x7f999c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.856+0000 7f99a2a00700 1 -- 172.21.15.1:0/702180024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f999c00a920 con 0x7f99ac101e40 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.856+0000 7f99a2a00700 1 -- 172.21.15.1:0/702180024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f999c00ce60 con 0x7f99ac101e40 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.856+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ac111880 con 0x7f99ac101e40 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.856+0000 7f99a2a00700 1 -- 172.21.15.1:0/702180024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f999c0113a0 con 0x7f99ac101e40 2024-02-28T17:12:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.856+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ac111dc0 con 0x7f99ac101e40 2024-02-28T17:12:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.858+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ac0738b0 con 0x7f99ac101e40 2024-02-28T17:12:19.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.859+0000 7f99a2a00700 1 -- 172.21.15.1:0/702180024 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f999c012400 con 0x7f99ac101e40 2024-02-28T17:12:19.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.860+0000 7f99a2a00700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f999006d7f0 0x7f999006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:19.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.860+0000 7f99a2a00700 1 -- 172.21.15.1:0/702180024 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f999c08cfe0 con 0x7f99ac101e40 2024-02-28T17:12:19.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.860+0000 7f99aa000700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f999006d7f0 0x7f999006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.861+0000 7f99aa000700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f999006d7f0 0x7f999006fca0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0661e0 tx=0x7f9998009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:19.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:19.865+0000 7f99a2a00700 1 -- 172.21.15.1:0/702180024 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f999c0568b0 con 0x7f99ac101e40 2024-02-28T17:12:20.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.114+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99ac065f90 con 0x7f999006d7f0 2024-02-28T17:12:20.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.117+0000 7f99a2a00700 1 -- 172.21.15.1:0/702180024 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f99ac065f90 con 0x7f999006d7f0 2024-02-28T17:12:20.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f999006d7f0 msgr2=0x7f999006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:20.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f999006d7f0 0x7f999006fca0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0661e0 tx=0x7f9998009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:20.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac101e40 msgr2=0x7f99ac10c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac101e40 0x7f99ac10c8a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f999c00c8c0 tx=0x7f999c0044c0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 shutdown_connections 2024-02-28T17:12:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f999006d7f0 0x7f999006fca0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99ac101e40 0x7f99ac10c8a0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 --2- 172.21.15.1:0/702180024 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99ac1027e0 0x7f99ac10cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 >> 172.21.15.1:0/702180024 conn(0x7f99ac071d90 msgr2=0x7f99ac0fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 shutdown_connections 2024-02-28T17:12:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.120+0000 7f99b1e00700 1 -- 172.21.15.1:0/702180024 wait complete. 2024-02-28T17:12:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.271+0000 7f31cb600700 1 -- 172.21.15.1:0/1597662109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 msgr2=0x7f31c4070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.271+0000 7f31cb600700 1 --2- 172.21.15.1:0/1597662109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 0x7f31c4070330 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f31b8004970 tx=0x7f31b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.271+0000 7f31cb600700 1 -- 172.21.15.1:0/1597662109 shutdown_connections 2024-02-28T17:12:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.271+0000 7f31cb600700 1 --2- 172.21.15.1:0/1597662109 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31c4070870 0x7f31c4070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.271+0000 7f31cb600700 1 --2- 172.21.15.1:0/1597662109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 0x7f31c4070330 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.271+0000 7f31cb600700 1 -- 172.21.15.1:0/1597662109 >> 172.21.15.1:0/1597662109 conn(0x7f31c40f8ae0 msgr2=0x7f31c40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.272+0000 7f31cb600700 1 -- 172.21.15.1:0/1597662109 shutdown_connections 2024-02-28T17:12:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.272+0000 7f31cb600700 1 -- 172.21.15.1:0/1597662109 wait complete. 2024-02-28T17:12:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.273+0000 7f31cb600700 1 Processor -- start 2024-02-28T17:12:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.273+0000 7f31cb600700 1 -- start start 2024-02-28T17:12:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.273+0000 7f31cb600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31c4070870 0x7f31c411a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.273+0000 7f31cb600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 0x7f31c411a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.273+0000 7f31cb600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c411ac00 con 0x7f31c4071f60 2024-02-28T17:12:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.273+0000 7f31cb600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4113f10 con 0x7f31c4070870 2024-02-28T17:12:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.274+0000 7f31c3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 0x7f31c411a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.274+0000 7f31c3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 0x7f31c411a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45990/0 (socket says 172.21.15.1:45990) 2024-02-28T17:12:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.274+0000 7f31c3e00700 1 -- 172.21.15.1:0/2926859861 learned_addr learned my addr 172.21.15.1:0/2926859861 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.274+0000 7f31c8c00700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31c4070870 0x7f31c411a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.274+0000 7f31c3e00700 1 -- 172.21.15.1:0/2926859861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31c4070870 msgr2=0x7f31c411a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.274+0000 7f31c3e00700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31c4070870 0x7f31c411a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.274+0000 7f31c3e00700 1 -- 172.21.15.1:0/2926859861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b8005040 con 0x7f31c4071f60 2024-02-28T17:12:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.275+0000 7f31c3e00700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 0x7f31c411a5b0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f31b0002a50 tx=0x7f31b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.275+0000 7f31c0c00700 1 -- 172.21.15.1:0/2926859861 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b0007cc0 con 0x7f31c4071f60 2024-02-28T17:12:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.275+0000 7f31c0c00700 1 -- 172.21.15.1:0/2926859861 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31b000f040 con 0x7f31c4071f60 2024-02-28T17:12:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.275+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c41141f0 con 0x7f31c4071f60 2024-02-28T17:12:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.275+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c4114730 con 0x7f31c4071f60 2024-02-28T17:12:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.275+0000 7f31c0c00700 1 -- 172.21.15.1:0/2926859861 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b000da30 con 0x7f31c4071f60 2024-02-28T17:12:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.277+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a40052e0 con 0x7f31c4071f60 2024-02-28T17:12:20.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.279+0000 7f31c0c00700 1 -- 172.21.15.1:0/2926859861 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f31b000db90 con 0x7f31c4071f60 2024-02-28T17:12:20.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.280+0000 7f31c0c00700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31ac06d7e0 0x7f31ac06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:20.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.280+0000 7f31c0c00700 1 -- 172.21.15.1:0/2926859861 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f31b008d200 con 0x7f31c4071f60 2024-02-28T17:12:20.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.280+0000 7f31c8c00700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31ac06d7e0 0x7f31ac06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:20.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.281+0000 7f31c8c00700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31ac06d7e0 0x7f31ac06fc90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f31b8000c00 tx=0x7f31b8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:20.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.286+0000 7f31c0c00700 1 -- 172.21.15.1:0/2926859861 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f31b005a690 con 0x7f31c4071f60 2024-02-28T17:12:20.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.524+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a4000bf0 con 0x7f31ac06d7e0 2024-02-28T17:12:20.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:20 smithi001 ceph-mon[112940]: pgmap v1140: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:20.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:20 smithi001 ceph-mon[112940]: from='client.36020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:20.527 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:20.528 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:20 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.536+0000 7f31c0c00700 1 -- 172.21.15.1:0/2926859861 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f31a4000bf0 con 0x7f31ac06d7e0 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (38m) 15m ago 52m 14.3M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (35m) 15m ago 53m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (35m) 0s ago 51m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (38m) 15m ago 52m 76.3M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (31m) 15m ago 45m 16.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (30m) 15m ago 45m 16.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (5s) 0s ago 45m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (15m) 0s ago 45m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:12:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (40m) 15m ago 54m 510M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (39m) 0s ago 51m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (36m) 15m ago 54m 67.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (36m) 0s ago 51m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (39m) 15m ago 52m 8321k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (39m) 0s ago 51m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (34m) 15m ago 50m 90.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (34m) 15m ago 49m 96.3M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (33m) 15m ago 49m 85.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (32m) 0s ago 48m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (32m) 0s ago 48m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (31m) 0s ago 47m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:12:20.540 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (38m) 15m ago 52m 61.8M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:12:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.538+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31ac06d7e0 msgr2=0x7f31ac06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.538+0000 7f31cb600700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31ac06d7e0 0x7f31ac06fc90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f31b8000c00 tx=0x7f31b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.538+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 msgr2=0x7f31c411a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.539+0000 7f31cb600700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 0x7f31c411a5b0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f31b0002a50 tx=0x7f31b00046b0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.539+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 shutdown_connections 2024-02-28T17:12:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.539+0000 7f31cb600700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31ac06d7e0 0x7f31ac06fc90 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.539+0000 7f31cb600700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31c4070870 0x7f31c411a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.539+0000 7f31cb600700 1 --2- 172.21.15.1:0/2926859861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31c4071f60 0x7f31c411a5b0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.539+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 >> 172.21.15.1:0/2926859861 conn(0x7f31c40f8ae0 msgr2=0x7f31c40fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.539+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 shutdown_connections 2024-02-28T17:12:20.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.539+0000 7f31cb600700 1 -- 172.21.15.1:0/2926859861 wait complete. 2024-02-28T17:12:20.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.692+0000 7f4743e00700 1 -- 172.21.15.1:0/3755068782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fc490 msgr2=0x7f47441091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:20.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.692+0000 7f4743e00700 1 --2- 172.21.15.1:0/3755068782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fc490 0x7f47441091f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4730004940 tx=0x7f4730009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:20.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.692+0000 7f4743e00700 1 -- 172.21.15.1:0/3755068782 shutdown_connections 2024-02-28T17:12:20.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.692+0000 7f4743e00700 1 --2- 172.21.15.1:0/3755068782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fc490 0x7f47441091f0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.692+0000 7f4743e00700 1 --2- 172.21.15.1:0/3755068782 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47440fbb80 0x7f47440fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.692+0000 7f4743e00700 1 -- 172.21.15.1:0/3755068782 >> 172.21.15.1:0/3755068782 conn(0x7f4744073ab0 msgr2=0x7f4744073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:20.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.692+0000 7f4743e00700 1 -- 172.21.15.1:0/3755068782 shutdown_connections 2024-02-28T17:12:20.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.692+0000 7f4743e00700 1 -- 172.21.15.1:0/3755068782 wait complete. 2024-02-28T17:12:20.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.693+0000 7f4743e00700 1 Processor -- start 2024-02-28T17:12:20.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.693+0000 7f4743e00700 1 -- start start 2024-02-28T17:12:20.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.693+0000 7f4743e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fbb80 0x7f4744115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:20.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.693+0000 7f4743e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47440fc490 0x7f4744115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:20.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.693+0000 7f4743e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47441165d0 con 0x7f47440fbb80 2024-02-28T17:12:20.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.693+0000 7f4743e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474410f900 con 0x7f47440fc490 2024-02-28T17:12:20.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.694+0000 7f4742a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fbb80 0x7f4744115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:20.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.694+0000 7f4742a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fbb80 0x7f4744115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46010/0 (socket says 172.21.15.1:46010) 2024-02-28T17:12:20.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.694+0000 7f4742a00700 1 -- 172.21.15.1:0/1236869933 learned_addr learned my addr 172.21.15.1:0/1236869933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:20.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.694+0000 7f4742000700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47440fc490 0x7f4744115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:20.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.694+0000 7f4742a00700 1 -- 172.21.15.1:0/1236869933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47440fc490 msgr2=0x7f4744115f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:20.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.694+0000 7f4742a00700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47440fc490 0x7f4744115f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:20.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.694+0000 7f4742a00700 1 -- 172.21.15.1:0/1236869933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4730005040 con 0x7f47440fbb80 2024-02-28T17:12:20.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.695+0000 7f4742a00700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fbb80 0x7f4744115a40 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f4734002a50 tx=0x7f47340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:20.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.696+0000 7f473aa00700 1 -- 172.21.15.1:0/1236869933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4734007cc0 con 0x7f47440fbb80 2024-02-28T17:12:20.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.696+0000 7f473aa00700 1 -- 172.21.15.1:0/1236869933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f473400f040 con 0x7f47440fbb80 2024-02-28T17:12:20.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.696+0000 7f473aa00700 1 -- 172.21.15.1:0/1236869933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f473400da30 con 0x7f47440fbb80 2024-02-28T17:12:20.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.696+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474410fbe0 con 0x7f47440fbb80 2024-02-28T17:12:20.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.696+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4744110120 con 0x7f47440fbb80 2024-02-28T17:12:20.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.697+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4744106920 con 0x7f47440fbb80 2024-02-28T17:12:20.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.698+0000 7f473aa00700 1 -- 172.21.15.1:0/1236869933 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f473400dbb0 con 0x7f47440fbb80 2024-02-28T17:12:20.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.699+0000 7f473aa00700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f472806d7f0 0x7f472806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.699+0000 7f473aa00700 1 -- 172.21.15.1:0/1236869933 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f473408cd00 con 0x7f47440fbb80 2024-02-28T17:12:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.700+0000 7f4742000700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f472806d7f0 0x7f472806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.700+0000 7f4742000700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f472806d7f0 0x7f472806fca0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4744110c30 tx=0x7f4730009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:20.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:20.704+0000 7f473aa00700 1 -- 172.21.15.1:0/1236869933 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4734059fe0 con 0x7f47440fbb80 2024-02-28T17:12:20.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:20 smithi066 ceph-mon[85781]: pgmap v1140: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:20.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:20 smithi066 ceph-mon[85781]: from='client.36020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:20.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:20.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:20 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:21.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.018+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f474404bed0 con 0x7f47440fbb80 2024-02-28T17:12:21.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.020+0000 7f473aa00700 1 -- 172.21.15.1:0/1236869933 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4734059b70 con 0x7f47440fbb80 2024-02-28T17:12:21.024 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:12:21.024 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:12:21.024 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:12:21.025 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:12:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.022+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f472806d7f0 msgr2=0x7f472806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.022+0000 7f4743e00700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f472806d7f0 0x7f472806fca0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4744110c30 tx=0x7f4730009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.022+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fbb80 msgr2=0x7f4744115a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.022+0000 7f4743e00700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fbb80 0x7f4744115a40 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f4734002a50 tx=0x7f47340046b0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.023+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 shutdown_connections 2024-02-28T17:12:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.023+0000 7f4743e00700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f472806d7f0 0x7f472806fca0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.023+0000 7f4743e00700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47440fbb80 0x7f4744115a40 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.023+0000 7f4743e00700 1 --2- 172.21.15.1:0/1236869933 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47440fc490 0x7f4744115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.023+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 >> 172.21.15.1:0/1236869933 conn(0x7f4744073ab0 msgr2=0x7f47440fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.023+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 shutdown_connections 2024-02-28T17:12:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.023+0000 7f4743e00700 1 -- 172.21.15.1:0/1236869933 wait complete. 2024-02-28T17:12:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.165+0000 7f5a28600700 1 -- 172.21.15.1:0/449859732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 msgr2=0x7f5a20105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.165+0000 7f5a28600700 1 --2- 172.21.15.1:0/449859732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 0x7f5a20105530 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5a14004940 tx=0x7f5a14009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.166+0000 7f5a28600700 1 -- 172.21.15.1:0/449859732 shutdown_connections 2024-02-28T17:12:21.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.166+0000 7f5a28600700 1 --2- 172.21.15.1:0/449859732 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a20105b00 0x7f5a200ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.166+0000 7f5a28600700 1 --2- 172.21.15.1:0/449859732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 0x7f5a20105530 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.166+0000 7f5a28600700 1 -- 172.21.15.1:0/449859732 >> 172.21.15.1:0/449859732 conn(0x7f5a200fad30 msgr2=0x7f5a200fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.166+0000 7f5a28600700 1 -- 172.21.15.1:0/449859732 shutdown_connections 2024-02-28T17:12:21.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.166+0000 7f5a28600700 1 -- 172.21.15.1:0/449859732 wait complete. 2024-02-28T17:12:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.167+0000 7f5a28600700 1 Processor -- start 2024-02-28T17:12:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.167+0000 7f5a28600700 1 -- start start 2024-02-28T17:12:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.168+0000 7f5a28600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 0x7f5a20115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.168+0000 7f5a28600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a20105b00 0x7f5a20116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.168+0000 7f5a28600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a20116830 con 0x7f5a20105160 2024-02-28T17:12:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.168+0000 7f5a28600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2010faf0 con 0x7f5a20105b00 2024-02-28T17:12:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.168+0000 7f5a25c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 0x7f5a20115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.168+0000 7f5a25200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a20105b00 0x7f5a20116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.168+0000 7f5a25c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 0x7f5a20115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46030/0 (socket says 172.21.15.1:46030) 2024-02-28T17:12:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.168+0000 7f5a25c00700 1 -- 172.21.15.1:0/3068266129 learned_addr learned my addr 172.21.15.1:0/3068266129 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.169+0000 7f5a25c00700 1 -- 172.21.15.1:0/3068266129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a20105b00 msgr2=0x7f5a20116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.169+0000 7f5a25c00700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a20105b00 0x7f5a20116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.169+0000 7f5a25c00700 1 -- 172.21.15.1:0/3068266129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a14005040 con 0x7f5a20105160 2024-02-28T17:12:21.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.169+0000 7f5a25c00700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 0x7f5a20115c10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009150 tx=0x7f5a14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:21.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.170+0000 7f5a1d600700 1 -- 172.21.15.1:0/3068266129 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a1400e040 con 0x7f5a20105160 2024-02-28T17:12:21.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.170+0000 7f5a1d600700 1 -- 172.21.15.1:0/3068266129 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a1400bbf0 con 0x7f5a20105160 2024-02-28T17:12:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.170+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2010fd70 con 0x7f5a20105160 2024-02-28T17:12:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.170+0000 7f5a1d600700 1 -- 172.21.15.1:0/3068266129 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a1401c3f0 con 0x7f5a20105160 2024-02-28T17:12:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.171+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a201102b0 con 0x7f5a20105160 2024-02-28T17:12:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.172+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a200fe930 con 0x7f5a20105160 2024-02-28T17:12:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.173+0000 7f5a1d600700 1 -- 172.21.15.1:0/3068266129 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5a1400bd60 con 0x7f5a20105160 2024-02-28T17:12:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.174+0000 7f5a1d600700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a0806d7f0 0x7f5a0806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.174+0000 7f5a1d600700 1 -- 172.21.15.1:0/3068266129 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5a1408e180 con 0x7f5a20105160 2024-02-28T17:12:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.174+0000 7f5a25200700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a0806d7f0 0x7f5a0806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.175+0000 7f5a25200700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a0806d7f0 0x7f5a0806fca0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5a201114b0 tx=0x7f5a0c0091a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:21.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.179+0000 7f5a1d600700 1 -- 172.21.15.1:0/3068266129 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a14057a50 con 0x7f5a20105160 2024-02-28T17:12:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.500+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5a2010c2d0 con 0x7f5a20105160 2024-02-28T17:12:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.501+0000 7f5a1d600700 1 -- 172.21.15.1:0/3068266129 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f5a1405b110 con 0x7f5a20105160 2024-02-28T17:12:21.503 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:12:21.503 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:12:21.503 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:12:21.504 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:12:21.505 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:12:21.506 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:12:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.503+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a0806d7f0 msgr2=0x7f5a0806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.503+0000 7f5a28600700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a0806d7f0 0x7f5a0806fca0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5a201114b0 tx=0x7f5a0c0091a0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.503+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 msgr2=0x7f5a20115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.503+0000 7f5a28600700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 0x7f5a20115c10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009150 tx=0x7f5a14009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.504+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 shutdown_connections 2024-02-28T17:12:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.504+0000 7f5a28600700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5a0806d7f0 0x7f5a0806fca0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.504+0000 7f5a28600700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a20105160 0x7f5a20115c10 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.504+0000 7f5a28600700 1 --2- 172.21.15.1:0/3068266129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5a20105b00 0x7f5a20116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.504+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 >> 172.21.15.1:0/3068266129 conn(0x7f5a200fad30 msgr2=0x7f5a200fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.504+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 shutdown_connections 2024-02-28T17:12:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.504+0000 7f5a28600700 1 -- 172.21.15.1:0/3068266129 wait complete. 2024-02-28T17:12:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:12:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.653+0000 7f6a6cc00700 1 -- 172.21.15.1:0/1436559526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 msgr2=0x7f6a68070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.653+0000 7f6a6cc00700 1 --2- 172.21.15.1:0/1436559526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 0x7f6a68070330 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6a58004970 tx=0x7f6a58009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.653+0000 7f6a6cc00700 1 -- 172.21.15.1:0/1436559526 shutdown_connections 2024-02-28T17:12:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.653+0000 7f6a6cc00700 1 --2- 172.21.15.1:0/1436559526 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a68070870 0x7f6a68070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.653+0000 7f6a6cc00700 1 --2- 172.21.15.1:0/1436559526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 0x7f6a68070330 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.653+0000 7f6a6cc00700 1 -- 172.21.15.1:0/1436559526 >> 172.21.15.1:0/1436559526 conn(0x7f6a680f8ae0 msgr2=0x7f6a680faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.653+0000 7f6a6cc00700 1 -- 172.21.15.1:0/1436559526 shutdown_connections 2024-02-28T17:12:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.653+0000 7f6a6cc00700 1 -- 172.21.15.1:0/1436559526 wait complete. 2024-02-28T17:12:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.655+0000 7f6a6cc00700 1 Processor -- start 2024-02-28T17:12:21.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.655+0000 7f6a6cc00700 1 -- start start 2024-02-28T17:12:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.655+0000 7f6a6cc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a68070870 0x7f6a6811a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.656+0000 7f6a6cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 0x7f6a6811a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.656+0000 7f6a6cc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6811ad10 con 0x7f6a68071f60 2024-02-28T17:12:21.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.656+0000 7f6a6cc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68113f90 con 0x7f6a68070870 2024-02-28T17:12:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.656+0000 7f6a65400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 0x7f6a6811a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.656+0000 7f6a65400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 0x7f6a6811a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46036/0 (socket says 172.21.15.1:46036) 2024-02-28T17:12:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.656+0000 7f6a65400700 1 -- 172.21.15.1:0/2494290202 learned_addr learned my addr 172.21.15.1:0/2494290202 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.656+0000 7f6a65e00700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a68070870 0x7f6a6811a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.657+0000 7f6a65400700 1 -- 172.21.15.1:0/2494290202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a68070870 msgr2=0x7f6a6811a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.657+0000 7f6a65400700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a68070870 0x7f6a6811a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.657+0000 7f6a65400700 1 -- 172.21.15.1:0/2494290202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58005040 con 0x7f6a68071f60 2024-02-28T17:12:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.657+0000 7f6a65400700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 0x7f6a6811a630 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400b790 tx=0x7f6a54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.657+0000 7f6a5e000700 1 -- 172.21.15.1:0/2494290202 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5400f6f0 con 0x7f6a68071f60 2024-02-28T17:12:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.657+0000 7f6a5e000700 1 -- 172.21.15.1:0/2494290202 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a5400fd20 con 0x7f6a68071f60 2024-02-28T17:12:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.657+0000 7f6a5e000700 1 -- 172.21.15.1:0/2494290202 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a540103a0 con 0x7f6a68071f60 2024-02-28T17:12:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.657+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a68114210 con 0x7f6a68071f60 2024-02-28T17:12:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.658+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a68114750 con 0x7f6a68071f60 2024-02-28T17:12:21.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.659+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a680fe400 con 0x7f6a68071f60 2024-02-28T17:12:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.660+0000 7f6a5e000700 1 -- 172.21.15.1:0/2494290202 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6a54011400 con 0x7f6a68071f60 2024-02-28T17:12:21.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.661+0000 7f6a5e000700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a4c06d7f0 0x7f6a4c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:21.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.661+0000 7f6a5e000700 1 -- 172.21.15.1:0/2494290202 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6a5408cbb0 con 0x7f6a68071f60 2024-02-28T17:12:21.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.661+0000 7f6a65e00700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a4c06d7f0 0x7f6a4c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:21.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.662+0000 7f6a65e00700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a4c06d7f0 0x7f6a4c06fca0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6a58004940 tx=0x7f6a58009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:21.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.666+0000 7f6a5e000700 1 -- 172.21.15.1:0/2494290202 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a54093080 con 0x7f6a68071f60 2024-02-28T17:12:21.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:21 smithi001 ceph-mon[112940]: from='client.36024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:21.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:21 smithi001 ceph-mon[112940]: from='client.36028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:21.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:21 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1236869933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:21.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:21 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3068266129' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:12:21.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.917+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a6804bf70 con 0x7f6a4c06d7f0 2024-02-28T17:12:21.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.919+0000 7f6a5e000700 1 -- 172.21.15.1:0/2494290202 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6a58009710 con 0x7f6a4c06d7f0 2024-02-28T17:12:21.922 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:12:21.922 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:12:21.922 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:12:21.922 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:12:21.922 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:12:21.922 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:12:21.923 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:12:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.922+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a4c06d7f0 msgr2=0x7f6a4c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.922+0000 7f6a6cc00700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a4c06d7f0 0x7f6a4c06fca0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6a58004940 tx=0x7f6a58009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.922+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 msgr2=0x7f6a6811a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.922+0000 7f6a6cc00700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 0x7f6a6811a630 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400b790 tx=0x7f6a54009250 comp rx=0 tx=0).stop 2024-02-28T17:12:21.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.923+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 shutdown_connections 2024-02-28T17:12:21.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.923+0000 7f6a6cc00700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a4c06d7f0 0x7f6a4c06fca0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.923+0000 7f6a6cc00700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a68070870 0x7f6a6811a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.923+0000 7f6a6cc00700 1 --2- 172.21.15.1:0/2494290202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a68071f60 0x7f6a6811a630 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:21.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.923+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 >> 172.21.15.1:0/2494290202 conn(0x7f6a680f8ae0 msgr2=0x7f6a680fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:21.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.923+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 shutdown_connections 2024-02-28T17:12:21.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:21.923+0000 7f6a6cc00700 1 -- 172.21.15.1:0/2494290202 wait complete. 2024-02-28T17:12:22.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:21 smithi066 ceph-mon[85781]: from='client.36024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:22.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:21 smithi066 ceph-mon[85781]: from='client.36028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:22.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:21 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1236869933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:22.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:21 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3068266129' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:12:22.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.073+0000 7f2e00800700 1 -- 172.21.15.1:0/1629801502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fdf30 msgr2=0x7f2df80fe300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:22.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.073+0000 7f2e00800700 1 --2- 172.21.15.1:0/1629801502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fdf30 0x7f2df80fe300 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f2dec004940 tx=0x7f2dec009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:22.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.073+0000 7f2e00800700 1 -- 172.21.15.1:0/1629801502 shutdown_connections 2024-02-28T17:12:22.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.073+0000 7f2e00800700 1 --2- 172.21.15.1:0/1629801502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80fe8d0 0x7f2df8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:22.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.073+0000 7f2e00800700 1 --2- 172.21.15.1:0/1629801502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fdf30 0x7f2df80fe300 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:22.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.073+0000 7f2e00800700 1 -- 172.21.15.1:0/1629801502 >> 172.21.15.1:0/1629801502 conn(0x7f2df80f79d0 msgr2=0x7f2df80f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:22.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.074+0000 7f2e00800700 1 -- 172.21.15.1:0/1629801502 shutdown_connections 2024-02-28T17:12:22.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.074+0000 7f2e00800700 1 -- 172.21.15.1:0/1629801502 wait complete. 2024-02-28T17:12:22.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.074+0000 7f2e00800700 1 Processor -- start 2024-02-28T17:12:22.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2e00800700 1 -- start start 2024-02-28T17:12:22.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2e00800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80fdf30 0x7f2df8115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:22.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2e00800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fe8d0 0x7f2df8116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:22.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2e00800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df81167a0 con 0x7f2df80fe8d0 2024-02-28T17:12:22.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2e00800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df810fad0 con 0x7f2df80fdf30 2024-02-28T17:12:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2dfd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fe8d0 0x7f2df8116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:22.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2dfde00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80fdf30 0x7f2df8115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:22.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2dfd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fe8d0 0x7f2df8116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57142/0 (socket says 172.21.15.1:57142) 2024-02-28T17:12:22.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2dfde00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80fdf30 0x7f2df8115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:54658/0 (socket says 172.21.15.1:54658) 2024-02-28T17:12:22.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.075+0000 7f2dfd400700 1 -- 172.21.15.1:0/1596486245 learned_addr learned my addr 172.21.15.1:0/1596486245 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:22.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.076+0000 7f2dfd400700 1 -- 172.21.15.1:0/1596486245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80fdf30 msgr2=0x7f2df8115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:22.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.076+0000 7f2dfd400700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80fdf30 0x7f2df8115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.076+0000 7f2dfd400700 1 -- 172.21.15.1:0/1596486245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec005040 con 0x7f2df80fe8d0 2024-02-28T17:12:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.076+0000 7f2dfde00700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80fdf30 0x7f2df8115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-28T17:12:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.076+0000 7f2dfd400700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fe8d0 0x7f2df8116150 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2de800eb70 tx=0x7f2de800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.076+0000 7f2df2000700 1 -- 172.21.15.1:0/1596486245 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2de8007cc0 con 0x7f2df80fe8d0 2024-02-28T17:12:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.077+0000 7f2df2000700 1 -- 172.21.15.1:0/1596486245 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2de800f040 con 0x7f2df80fe8d0 2024-02-28T17:12:22.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.077+0000 7f2df2000700 1 -- 172.21.15.1:0/1596486245 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2de800ca90 con 0x7f2df80fe8d0 2024-02-28T17:12:22.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.077+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df810fd50 con 0x7f2df80fe8d0 2024-02-28T17:12:22.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.077+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df8110290 con 0x7f2df80fe8d0 2024-02-28T17:12:22.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.078+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df8106a00 con 0x7f2df80fe8d0 2024-02-28T17:12:22.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.080+0000 7f2df2000700 1 -- 172.21.15.1:0/1596486245 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2de800cbf0 con 0x7f2df80fe8d0 2024-02-28T17:12:22.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.081+0000 7f2df2000700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2de006d7f0 0x7f2de006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:22.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.081+0000 7f2df2000700 1 -- 172.21.15.1:0/1596486245 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2de808c8b0 con 0x7f2df80fe8d0 2024-02-28T17:12:22.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.081+0000 7f2dfde00700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2de006d7f0 0x7f2de006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:22.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.082+0000 7f2dfde00700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2de006d7f0 0x7f2de006fca0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2dec007f50 tx=0x7f2dec009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:22.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.086+0000 7f2df2000700 1 -- 172.21.15.1:0/1596486245 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2de8059c10 con 0x7f2df80fe8d0 2024-02-28T17:12:22.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.414+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2df810a030 con 0x7f2df80fe8d0 2024-02-28T17:12:22.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.415+0000 7f2df2000700 1 -- 172.21.15.1:0/1596486245 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f2de805d1a0 con 0x7f2df80fe8d0 2024-02-28T17:12:22.417 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:12:22.417 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:12:22.417 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:12:22.418 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:12:22.419 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:12:22.419 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:12:22.419 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:12:22.419 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:12:22.419 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:12:22.419 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:12:22.419 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:12:22.420 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:12:22.420 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:12:22.420 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:12:22.420 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:12:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2de006d7f0 msgr2=0x7f2de006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2de006d7f0 0x7f2de006fca0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2dec007f50 tx=0x7f2dec009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fe8d0 msgr2=0x7f2df8116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fe8d0 0x7f2df8116150 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2de800eb70 tx=0x7f2de800b040 comp rx=0 tx=0).stop 2024-02-28T17:12:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 shutdown_connections 2024-02-28T17:12:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2de006d7f0 0x7f2de006fca0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2df80fdf30 0x7f2df8115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 --2- 172.21.15.1:0/1596486245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2df80fe8d0 0x7f2df8116150 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 >> 172.21.15.1:0/1596486245 conn(0x7f2df80f79d0 msgr2=0x7f2df8103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 shutdown_connections 2024-02-28T17:12:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:22.417+0000 7f2e00800700 1 -- 172.21.15.1:0/1596486245 wait complete. 2024-02-28T17:12:22.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:22 smithi001 ceph-mon[112940]: pgmap v1141: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:22.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:22 smithi001 ceph-mon[112940]: from='client.36040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:22.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:22 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:22.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:22 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:22.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:22 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1596486245' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:12:23.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:22 smithi066 ceph-mon[85781]: pgmap v1141: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:23.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:22 smithi066 ceph-mon[85781]: from='client.36040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:23.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:22 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:23.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:22 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:23.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:22 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1596486245' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:12:23.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:23.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:24.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:24.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:24.981 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:24 smithi066 ceph-mon[85781]: pgmap v1142: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:25.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:24 smithi001 ceph-mon[112940]: pgmap v1142: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:26.150 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:26.151 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:26.151 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:12:26.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:26.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:12:26.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:12:27.155 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:26 smithi066 ceph-mon[85781]: pgmap v1143: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:27.211 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:26 smithi001 ceph-mon[112940]: pgmap v1143: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:29.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:28 smithi001 ceph-mon[112940]: pgmap v1144: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:29.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:28 smithi066 ceph-mon[85781]: pgmap v1144: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:31.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:30 smithi066 ceph-mon[85781]: pgmap v1145: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:31.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:30 smithi001 ceph-mon[112940]: pgmap v1145: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:33.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:32 smithi066 ceph-mon[85781]: pgmap v1146: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:33.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:32 smithi001 ceph-mon[112940]: pgmap v1146: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:35.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:34 smithi066 ceph-mon[85781]: pgmap v1147: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:35.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:34 smithi001 ceph-mon[112940]: pgmap v1147: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:37.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:36 smithi066 ceph-mon[85781]: pgmap v1148: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:37.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:36 smithi001 ceph-mon[112940]: pgmap v1148: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:39.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:38 smithi066 ceph-mon[85781]: pgmap v1149: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:39.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:38 smithi001 ceph-mon[112940]: pgmap v1149: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:41.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:40 smithi066 ceph-mon[85781]: pgmap v1150: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:41.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:12:41.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:40 smithi001 ceph-mon[112940]: pgmap v1150: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:41.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:12:43.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:42 smithi066 ceph-mon[85781]: pgmap v1151: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:43.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:42 smithi001 ceph-mon[112940]: pgmap v1151: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:45.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:44 smithi066 ceph-mon[85781]: pgmap v1152: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:45.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:44 smithi001 ceph-mon[112940]: pgmap v1152: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:47.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:46 smithi066 ceph-mon[85781]: pgmap v1153: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:47.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:46 smithi001 ceph-mon[112940]: pgmap v1153: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:49.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:48 smithi066 ceph-mon[85781]: pgmap v1154: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:48 smithi001 ceph-mon[112940]: pgmap v1154: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:51.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:50 smithi066 ceph-mon[85781]: pgmap v1155: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:51.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:50 smithi001 ceph-mon[112940]: pgmap v1155: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.583+0000 7fb671800700 1 -- 172.21.15.1:0/3382239800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 msgr2=0x7fb66c103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.583+0000 7fb671800700 1 --2- 172.21.15.1:0/3382239800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 0x7fb66c103160 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fb65c004940 tx=0x7fb65c009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.584+0000 7fb671800700 1 -- 172.21.15.1:0/3382239800 shutdown_connections 2024-02-28T17:12:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.584+0000 7fb671800700 1 --2- 172.21.15.1:0/3382239800 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb66c103730 0x7fb66c0fcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.584+0000 7fb671800700 1 --2- 172.21.15.1:0/3382239800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 0x7fb66c103160 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.584+0000 7fb671800700 1 -- 172.21.15.1:0/3382239800 >> 172.21.15.1:0/3382239800 conn(0x7fb66c0f8940 msgr2=0x7fb66c0fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.584+0000 7fb671800700 1 -- 172.21.15.1:0/3382239800 shutdown_connections 2024-02-28T17:12:52.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.584+0000 7fb671800700 1 -- 172.21.15.1:0/3382239800 wait complete. 2024-02-28T17:12:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.585+0000 7fb671800700 1 Processor -- start 2024-02-28T17:12:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.585+0000 7fb671800700 1 -- start start 2024-02-28T17:12:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.585+0000 7fb671800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 0x7fb66c115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.585+0000 7fb671800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb66c103730 0x7fb66c115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.585+0000 7fb671800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1165d0 con 0x7fb66c102d90 2024-02-28T17:12:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.585+0000 7fb671800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c10f950 con 0x7fb66c103730 2024-02-28T17:12:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.586+0000 7fb66be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 0x7fb66c115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:52.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.586+0000 7fb66be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 0x7fb66c115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34368/0 (socket says 172.21.15.1:34368) 2024-02-28T17:12:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.586+0000 7fb66be00700 1 -- 172.21.15.1:0/3286997515 learned_addr learned my addr 172.21.15.1:0/3286997515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.586+0000 7fb663e00700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb66c103730 0x7fb66c115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.586+0000 7fb66be00700 1 -- 172.21.15.1:0/3286997515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb66c103730 msgr2=0x7fb66c115f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.586+0000 7fb66be00700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb66c103730 0x7fb66c115f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.586+0000 7fb66be00700 1 -- 172.21.15.1:0/3286997515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c005040 con 0x7fb66c102d90 2024-02-28T17:12:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.586+0000 7fb663e00700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb66c103730 0x7fb66c115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:12:52.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.587+0000 7fb66be00700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 0x7fb66c115a40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009fd0 tx=0x7fb65c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.587+0000 7fb668c00700 1 -- 172.21.15.1:0/3286997515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb65c00e040 con 0x7fb66c102d90 2024-02-28T17:12:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.587+0000 7fb668c00700 1 -- 172.21.15.1:0/3286997515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb65c00bbf0 con 0x7fb66c102d90 2024-02-28T17:12:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.587+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66c10fbd0 con 0x7fb66c102d90 2024-02-28T17:12:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.587+0000 7fb668c00700 1 -- 172.21.15.1:0/3286997515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb65c01c3f0 con 0x7fb66c102d90 2024-02-28T17:12:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.587+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c110190 con 0x7fb66c102d90 2024-02-28T17:12:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.588+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66c05f5c0 con 0x7fb66c102d90 2024-02-28T17:12:52.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.589+0000 7fb668c00700 1 -- 172.21.15.1:0/3286997515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb65c00bd60 con 0x7fb66c102d90 2024-02-28T17:12:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.591+0000 7fb668c00700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65006d7f0 0x7fb65006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.591+0000 7fb668c00700 1 -- 172.21.15.1:0/3286997515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb65c08e1c0 con 0x7fb66c102d90 2024-02-28T17:12:52.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.591+0000 7fb663e00700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65006d7f0 0x7fb65006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:52.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.592+0000 7fb663e00700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65006d7f0 0x7fb65006fca0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb654009fd0 tx=0x7fb65400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:52.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.596+0000 7fb668c00700 1 -- 172.21.15.1:0/3286997515 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb65c05b5a0 con 0x7fb66c102d90 2024-02-28T17:12:52.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.840+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb66c111020 con 0x7fb65006d7f0 2024-02-28T17:12:52.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.843+0000 7fb668c00700 1 -- 172.21.15.1:0/3286997515 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb66c111020 con 0x7fb65006d7f0 2024-02-28T17:12:52.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65006d7f0 msgr2=0x7fb65006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:52.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65006d7f0 0x7fb65006fca0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb654009fd0 tx=0x7fb65400b040 comp rx=0 tx=0).stop 2024-02-28T17:12:52.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 msgr2=0x7fb66c115a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:52.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 0x7fb66c115a40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009fd0 tx=0x7fb65c009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:52.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 shutdown_connections 2024-02-28T17:12:52.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65006d7f0 0x7fb65006fca0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:52.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb66c102d90 0x7fb66c115a40 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:52.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 --2- 172.21.15.1:0/3286997515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb66c103730 0x7fb66c115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:52.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 >> 172.21.15.1:0/3286997515 conn(0x7fb66c0f8940 msgr2=0x7fb66c0fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:52.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 shutdown_connections 2024-02-28T17:12:52.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.846+0000 7fb671800700 1 -- 172.21.15.1:0/3286997515 wait complete. 2024-02-28T17:12:52.864 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:12:52.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.992+0000 7f02e6000700 1 -- 172.21.15.1:0/110513606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0105930 msgr2=0x7f02e00fefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:52.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.992+0000 7f02e6000700 1 --2- 172.21.15.1:0/110513606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0105930 0x7f02e00fefa0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f02cc004940 tx=0x7f02cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:52.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.992+0000 7f02e6000700 1 -- 172.21.15.1:0/110513606 shutdown_connections 2024-02-28T17:12:52.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.992+0000 7f02e6000700 1 --2- 172.21.15.1:0/110513606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0105930 0x7f02e00fefa0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.992+0000 7f02e6000700 1 --2- 172.21.15.1:0/110513606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e0104f90 0x7f02e0105360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.992+0000 7f02e6000700 1 -- 172.21.15.1:0/110513606 >> 172.21.15.1:0/110513606 conn(0x7f02e00fab40 msgr2=0x7f02e00fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.992+0000 7f02e6000700 1 -- 172.21.15.1:0/110513606 shutdown_connections 2024-02-28T17:12:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.993+0000 7f02e6000700 1 -- 172.21.15.1:0/110513606 wait complete. 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.993+0000 7f02e6000700 1 Processor -- start 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.993+0000 7f02e6000700 1 -- start start 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e6000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0104f90 0x7f02e0115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e6000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e0105930 0x7f02e0115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e6000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e01165e0 con 0x7f02e0104f90 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e6000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e010f910 con 0x7f02e0105930 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0104f90 0x7f02e0115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0104f90 0x7f02e0115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34382/0 (socket says 172.21.15.1:34382) 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e4c00700 1 -- 172.21.15.1:0/2375837623 learned_addr learned my addr 172.21.15.1:0/2375837623 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02dfe00700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e0105930 0x7f02e0115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e4c00700 1 -- 172.21.15.1:0/2375837623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e0105930 msgr2=0x7f02e0115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:52.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e4c00700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e0105930 0x7f02e0115f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.994+0000 7f02e4c00700 1 -- 172.21.15.1:0/2375837623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc005040 con 0x7f02e0104f90 2024-02-28T17:12:53.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.995+0000 7f02dfe00700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e0105930 0x7f02e0115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:12:53.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.995+0000 7f02e4c00700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0104f90 0x7f02e0115a50 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c8c0 tx=0x7f02d40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:53.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.995+0000 7f02dcc00700 1 -- 172.21.15.1:0/2375837623 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d400a980 con 0x7f02e0104f90 2024-02-28T17:12:53.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.995+0000 7f02dcc00700 1 -- 172.21.15.1:0/2375837623 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02d400ce60 con 0x7f02e0104f90 2024-02-28T17:12:53.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.995+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e010fbf0 con 0x7f02e0104f90 2024-02-28T17:12:53.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.995+0000 7f02dcc00700 1 -- 172.21.15.1:0/2375837623 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02d40115e0 con 0x7f02e0104f90 2024-02-28T17:12:53.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.995+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e01101b0 con 0x7f02e0104f90 2024-02-28T17:12:53.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.996+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e0107460 con 0x7f02e0104f90 2024-02-28T17:12:53.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.997+0000 7f02dcc00700 1 -- 172.21.15.1:0/2375837623 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f02d4012400 con 0x7f02e0104f90 2024-02-28T17:12:53.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.998+0000 7f02dcc00700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02c806d7e0 0x7f02c806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:53.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.998+0000 7f02dcc00700 1 -- 172.21.15.1:0/2375837623 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f02d408cf70 con 0x7f02e0104f90 2024-02-28T17:12:53.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.998+0000 7f02dfe00700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02c806d7e0 0x7f02c806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:53.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:52.999+0000 7f02dfe00700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02c806d7e0 0x7f02c806fc90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00b3b0 tx=0x7f02cc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:53.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.003+0000 7f02dcc00700 1 -- 172.21.15.1:0/2375837623 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f02d405a350 con 0x7f02e0104f90 2024-02-28T17:12:53.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.236+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f02e004bf40 con 0x7f02c806d7e0 2024-02-28T17:12:53.240 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:52 smithi001 ceph-mon[112940]: pgmap v1156: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:53.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.238+0000 7f02dcc00700 1 -- 172.21.15.1:0/2375837623 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f02e004bf40 con 0x7f02c806d7e0 2024-02-28T17:12:53.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.240+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02c806d7e0 msgr2=0x7f02c806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.240+0000 7f02e6000700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02c806d7e0 0x7f02c806fc90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00b3b0 tx=0x7f02cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0104f90 msgr2=0x7f02e0115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0104f90 0x7f02e0115a50 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c8c0 tx=0x7f02d40044c0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 shutdown_connections 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f02c806d7e0 0x7f02c806fc90 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02e0104f90 0x7f02e0115a50 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 --2- 172.21.15.1:0/2375837623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02e0105930 0x7f02e0115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 >> 172.21.15.1:0/2375837623 conn(0x7f02e00fab40 msgr2=0x7f02e00fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 shutdown_connections 2024-02-28T17:12:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.241+0000 7f02e6000700 1 -- 172.21.15.1:0/2375837623 wait complete. 2024-02-28T17:12:53.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:52 smithi066 ceph-mon[85781]: pgmap v1156: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.399+0000 7fec57200700 1 -- 172.21.15.1:0/696201686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec50070bc0 msgr2=0x7fec501092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.399+0000 7fec57200700 1 --2- 172.21.15.1:0/696201686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec50070bc0 0x7fec501092e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fec38004940 tx=0x7fec38009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.399+0000 7fec57200700 1 -- 172.21.15.1:0/696201686 shutdown_connections 2024-02-28T17:12:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.399+0000 7fec57200700 1 --2- 172.21.15.1:0/696201686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec50070bc0 0x7fec501092e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.399+0000 7fec57200700 1 --2- 172.21.15.1:0/696201686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec500702b0 0x7fec50070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.399+0000 7fec57200700 1 -- 172.21.15.1:0/696201686 >> 172.21.15.1:0/696201686 conn(0x7fec500f8ae0 msgr2=0x7fec500faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.399+0000 7fec57200700 1 -- 172.21.15.1:0/696201686 shutdown_connections 2024-02-28T17:12:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.399+0000 7fec57200700 1 -- 172.21.15.1:0/696201686 wait complete. 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.400+0000 7fec57200700 1 Processor -- start 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.400+0000 7fec57200700 1 -- start start 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.400+0000 7fec57200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec500702b0 0x7fec50115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.400+0000 7fec57200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec50070bc0 0x7fec50116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.400+0000 7fec57200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec501168e0 con 0x7fec500702b0 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.400+0000 7fec57200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5010fbf0 con 0x7fec50070bc0 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.401+0000 7fec4fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec500702b0 0x7fec50115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.401+0000 7fec4fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec500702b0 0x7fec50115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34406/0 (socket says 172.21.15.1:34406) 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.401+0000 7fec4fe00700 1 -- 172.21.15.1:0/1089096400 learned_addr learned my addr 172.21.15.1:0/1089096400 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.401+0000 7fec47e00700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec50070bc0 0x7fec50116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.401+0000 7fec4fe00700 1 -- 172.21.15.1:0/1089096400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec50070bc0 msgr2=0x7fec50116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.401+0000 7fec4fe00700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec50070bc0 0x7fec50116290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.401+0000 7fec4fe00700 1 -- 172.21.15.1:0/1089096400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec38005040 con 0x7fec500702b0 2024-02-28T17:12:53.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.402+0000 7fec4fe00700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec500702b0 0x7fec50115d50 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fec40002a50 tx=0x7fec400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:53.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.402+0000 7fec4cc00700 1 -- 172.21.15.1:0/1089096400 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec40007cc0 con 0x7fec500702b0 2024-02-28T17:12:53.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.402+0000 7fec4cc00700 1 -- 172.21.15.1:0/1089096400 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec4000f040 con 0x7fec500702b0 2024-02-28T17:12:53.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.402+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5010fed0 con 0x7fec500702b0 2024-02-28T17:12:53.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.402+0000 7fec4cc00700 1 -- 172.21.15.1:0/1089096400 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec4000db70 con 0x7fec500702b0 2024-02-28T17:12:53.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.402+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec50110410 con 0x7fec500702b0 2024-02-28T17:12:53.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.403+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec501069a0 con 0x7fec500702b0 2024-02-28T17:12:53.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.404+0000 7fec4cc00700 1 -- 172.21.15.1:0/1089096400 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fec4000dd10 con 0x7fec500702b0 2024-02-28T17:12:53.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.405+0000 7fec4cc00700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec3406d7e0 0x7fec3406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:53.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.405+0000 7fec4cc00700 1 -- 172.21.15.1:0/1089096400 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fec4008c5d0 con 0x7fec500702b0 2024-02-28T17:12:53.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.406+0000 7fec47e00700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec3406d7e0 0x7fec3406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:53.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.406+0000 7fec47e00700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec3406d7e0 0x7fec3406fc90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fec38004f90 tx=0x7fec38009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:53.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.410+0000 7fec4cc00700 1 -- 172.21.15.1:0/1089096400 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec40059950 con 0x7fec500702b0 2024-02-28T17:12:53.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.645+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fec5004bf40 con 0x7fec3406d7e0 2024-02-28T17:12:53.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.656+0000 7fec4cc00700 1 -- 172.21.15.1:0/1089096400 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fec5004bf40 con 0x7fec3406d7e0 2024-02-28T17:12:53.660 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:12:53.660 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (39m) 31s ago 53m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:12:53.660 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (35m) 31s ago 53m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:12:53.660 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (35m) 33s ago 52m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:12:53.660 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (38m) 31s ago 52m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:12:53.660 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (31m) 31s ago 46m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:12:53.660 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (31m) 31s ago 46m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:12:53.660 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (38s) 33s ago 45m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (15m) 33s ago 46m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (41m) 31s ago 55m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (40m) 33s ago 51m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (37m) 31s ago 55m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (36m) 33s ago 51m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (40m) 31s ago 53m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (39m) 33s ago 52m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (35m) 31s ago 50m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (34m) 31s ago 50m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (34m) 31s ago 49m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (33m) 33s ago 49m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (32m) 33s ago 48m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (32m) 33s ago 47m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:12:53.661 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (39m) 31s ago 52m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:12:53.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec3406d7e0 msgr2=0x7fec3406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec3406d7e0 0x7fec3406fc90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fec38004f90 tx=0x7fec38009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec500702b0 msgr2=0x7fec50115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec500702b0 0x7fec50115d50 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fec40002a50 tx=0x7fec400046b0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 shutdown_connections 2024-02-28T17:12:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec3406d7e0 0x7fec3406fc90 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec500702b0 0x7fec50115d50 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 --2- 172.21.15.1:0/1089096400 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec50070bc0 0x7fec50116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 >> 172.21.15.1:0/1089096400 conn(0x7fec500f8ae0 msgr2=0x7fec50103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:53.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 shutdown_connections 2024-02-28T17:12:53.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.659+0000 7fec57200700 1 -- 172.21.15.1:0/1089096400 wait complete. 2024-02-28T17:12:53.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.814+0000 7fe333600700 1 -- 172.21.15.1:0/3055796580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c1038f0 msgr2=0x7fe32c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:53.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.814+0000 7fe333600700 1 --2- 172.21.15.1:0/3055796580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c1038f0 0x7fe32c0fcf60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe318004940 tx=0x7fe318009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:53.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.814+0000 7fe333600700 1 -- 172.21.15.1:0/3055796580 shutdown_connections 2024-02-28T17:12:53.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.814+0000 7fe333600700 1 --2- 172.21.15.1:0/3055796580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c1038f0 0x7fe32c0fcf60 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.814+0000 7fe333600700 1 --2- 172.21.15.1:0/3055796580 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe32c102f50 0x7fe32c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.814+0000 7fe333600700 1 -- 172.21.15.1:0/3055796580 >> 172.21.15.1:0/3055796580 conn(0x7fe32c0f8ae0 msgr2=0x7fe32c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:53.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.814+0000 7fe333600700 1 -- 172.21.15.1:0/3055796580 shutdown_connections 2024-02-28T17:12:53.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.814+0000 7fe333600700 1 -- 172.21.15.1:0/3055796580 wait complete. 2024-02-28T17:12:53.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.815+0000 7fe333600700 1 Processor -- start 2024-02-28T17:12:53.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.815+0000 7fe333600700 1 -- start start 2024-02-28T17:12:53.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe333600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c102f50 0x7fe32c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:53.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe333600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe32c1038f0 0x7fe32c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:53.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe333600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c116790 con 0x7fe32c102f50 2024-02-28T17:12:53.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe333600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c10fae0 con 0x7fe32c1038f0 2024-02-28T17:12:53.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe330c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c102f50 0x7fe32c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:53.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe330c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c102f50 0x7fe32c115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34422/0 (socket says 172.21.15.1:34422) 2024-02-28T17:12:53.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe330c00700 1 -- 172.21.15.1:0/2419279191 learned_addr learned my addr 172.21.15.1:0/2419279191 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe330c00700 1 -- 172.21.15.1:0/2419279191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe32c1038f0 msgr2=0x7fe32c116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe330c00700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe32c1038f0 0x7fe32c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.816+0000 7fe330c00700 1 -- 172.21.15.1:0/2419279191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318005040 con 0x7fe32c102f50 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.817+0000 7fe330c00700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c102f50 0x7fe32c115c00 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe32000eb40 tx=0x7fe32000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.817+0000 7fe328c00700 1 -- 172.21.15.1:0/2419279191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe320007cc0 con 0x7fe32c102f50 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.817+0000 7fe328c00700 1 -- 172.21.15.1:0/2419279191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe32000f040 con 0x7fe32c102f50 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.817+0000 7fe328c00700 1 -- 172.21.15.1:0/2419279191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe320009620 con 0x7fe32c102f50 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.817+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c10fd60 con 0x7fe32c102f50 2024-02-28T17:12:53.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.817+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c1102a0 con 0x7fe32c102f50 2024-02-28T17:12:53.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.818+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c0fc6e0 con 0x7fe32c102f50 2024-02-28T17:12:53.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.819+0000 7fe328c00700 1 -- 172.21.15.1:0/2419279191 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe320009780 con 0x7fe32c102f50 2024-02-28T17:12:53.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.820+0000 7fe328c00700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31406d7e0 0x7fe31406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:53.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.820+0000 7fe328c00700 1 -- 172.21.15.1:0/2419279191 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe32008cb60 con 0x7fe32c102f50 2024-02-28T17:12:53.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.821+0000 7fe32be00700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31406d7e0 0x7fe31406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:53.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.821+0000 7fe32be00700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31406d7e0 0x7fe31406fc90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe318009fd0 tx=0x7fe318009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:53.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:53.825+0000 7fe328c00700 1 -- 172.21.15.1:0/2419279191 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe320056460 con 0x7fe32c102f50 2024-02-28T17:12:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.134+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe32c1111e0 con 0x7fe32c102f50 2024-02-28T17:12:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.135+0000 7fe328c00700 1 -- 172.21.15.1:0/2419279191 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe320059b20 con 0x7fe32c102f50 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:12:54.139 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:12:54.140 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:12:54.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:12:54.140 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:12:54.140 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:12:54.140 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:12:54.140 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:12:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31406d7e0 msgr2=0x7fe31406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31406d7e0 0x7fe31406fc90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe318009fd0 tx=0x7fe318009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c102f50 msgr2=0x7fe32c115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c102f50 0x7fe32c115c00 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe32000eb40 tx=0x7fe32000b040 comp rx=0 tx=0).stop 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 shutdown_connections 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe31406d7e0 0x7fe31406fc90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe32c102f50 0x7fe32c115c00 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 --2- 172.21.15.1:0/2419279191 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe32c1038f0 0x7fe32c116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 >> 172.21.15.1:0/2419279191 conn(0x7fe32c0f8ae0 msgr2=0x7fe32c0fa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 shutdown_connections 2024-02-28T17:12:54.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.138+0000 7fe333600700 1 -- 172.21.15.1:0/2419279191 wait complete. 2024-02-28T17:12:54.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:53 smithi066 ceph-mon[85781]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:54.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:53 smithi001 ceph-mon[112940]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.282+0000 7f7bab200700 1 -- 172.21.15.1:0/4259997041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac100f70 msgr2=0x7f7bac101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.282+0000 7f7bab200700 1 --2- 172.21.15.1:0/4259997041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac100f70 0x7f7bac101340 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c004940 tx=0x7f7b9c009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.282+0000 7f7bab200700 1 -- 172.21.15.1:0/4259997041 shutdown_connections 2024-02-28T17:12:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.282+0000 7f7bab200700 1 --2- 172.21.15.1:0/4259997041 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bac101910 0x7f7bac0faf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.282+0000 7f7bab200700 1 --2- 172.21.15.1:0/4259997041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac100f70 0x7f7bac101340 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.283+0000 7f7bab200700 1 -- 172.21.15.1:0/4259997041 >> 172.21.15.1:0/4259997041 conn(0x7f7bac0f6aa0 msgr2=0x7f7bac0f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:54.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.283+0000 7f7bab200700 1 -- 172.21.15.1:0/4259997041 shutdown_connections 2024-02-28T17:12:54.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.283+0000 7f7bab200700 1 -- 172.21.15.1:0/4259997041 wait complete. 2024-02-28T17:12:54.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.284+0000 7f7bab200700 1 Processor -- start 2024-02-28T17:12:54.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.284+0000 7f7bab200700 1 -- start start 2024-02-28T17:12:54.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.284+0000 7f7bab200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bac100f70 0x7f7bac115a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:54.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.284+0000 7f7bab200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac101910 0x7f7bac115f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:54.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.284+0000 7f7bab200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac1165c0 con 0x7f7bac101910 2024-02-28T17:12:54.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.284+0000 7f7bab200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac10f8f0 con 0x7f7bac100f70 2024-02-28T17:12:54.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.284+0000 7f7ba9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac101910 0x7f7bac115f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:54.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.284+0000 7f7ba9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac101910 0x7f7bac115f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34442/0 (socket says 172.21.15.1:34442) 2024-02-28T17:12:54.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba9400700 1 -- 172.21.15.1:0/3061143288 learned_addr learned my addr 172.21.15.1:0/3061143288 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:54.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba9e00700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bac100f70 0x7f7bac115a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:54.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba9400700 1 -- 172.21.15.1:0/3061143288 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bac100f70 msgr2=0x7f7bac115a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba9400700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bac100f70 0x7f7bac115a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba9400700 1 -- 172.21.15.1:0/3061143288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c005040 con 0x7f7bac101910 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba9400700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac101910 0x7f7bac115f70 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f7b98002a80 tx=0x7f7b98004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba2000700 1 -- 172.21.15.1:0/3061143288 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b98007cc0 con 0x7f7bac101910 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba2000700 1 -- 172.21.15.1:0/3061143288 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b9800f040 con 0x7f7bac101910 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.285+0000 7f7ba2000700 1 -- 172.21.15.1:0/3061143288 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9800daf0 con 0x7f7bac101910 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.286+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bac10fbd0 con 0x7f7bac101910 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.286+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bac110190 con 0x7f7bac101910 2024-02-28T17:12:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.287+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bac05f5c0 con 0x7f7bac101910 2024-02-28T17:12:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.288+0000 7f7ba2000700 1 -- 172.21.15.1:0/3061143288 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7b9800dc90 con 0x7f7bac101910 2024-02-28T17:12:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.288+0000 7f7ba2000700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9006d7f0 0x7f7b9006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.289+0000 7f7ba2000700 1 -- 172.21.15.1:0/3061143288 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7b9808c5b0 con 0x7f7bac101910 2024-02-28T17:12:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.289+0000 7f7ba9e00700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9006d7f0 0x7f7b9006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.289+0000 7f7ba9e00700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9006d7f0 0x7f7b9006fca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c00b020 tx=0x7f7b9c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:54.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.293+0000 7f7ba2000700 1 -- 172.21.15.1:0/3061143288 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7b98059910 con 0x7f7bac101910 2024-02-28T17:12:54.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.565+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7bac04bf00 con 0x7f7bac101910 2024-02-28T17:12:54.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.565+0000 7f7ba2000700 1 -- 172.21.15.1:0/3061143288 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f7b980594a0 con 0x7f7bac101910 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:12:54.570 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:12:54.571 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:12:54.572 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:12:54.573 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:12:54.573 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:12:54.573 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:12:54.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9006d7f0 msgr2=0x7f7b9006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9006d7f0 0x7f7b9006fca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c00b020 tx=0x7f7b9c009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:54.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac101910 msgr2=0x7f7bac115f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac101910 0x7f7bac115f70 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f7b98002a80 tx=0x7f7b98004630 comp rx=0 tx=0).stop 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 shutdown_connections 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9006d7f0 0x7f7b9006fca0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bac100f70 0x7f7bac115a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 --2- 172.21.15.1:0/3061143288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bac101910 0x7f7bac115f70 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.568+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 >> 172.21.15.1:0/3061143288 conn(0x7f7bac0f6aa0 msgr2=0x7f7bac0f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.569+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 shutdown_connections 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.569+0000 7f7bab200700 1 -- 172.21.15.1:0/3061143288 wait complete. 2024-02-28T17:12:54.574 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:12:54.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.715+0000 7fc175200700 1 -- 172.21.15.1:0/1452575762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 msgr2=0x7fc170070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.715+0000 7fc175200700 1 --2- 172.21.15.1:0/1452575762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 0x7fc170070700 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc160004970 tx=0x7fc160009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:54.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.715+0000 7fc175200700 1 -- 172.21.15.1:0/1452575762 shutdown_connections 2024-02-28T17:12:54.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.715+0000 7fc175200700 1 --2- 172.21.15.1:0/1452575762 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc170070c40 0x7fc170109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.715+0000 7fc175200700 1 --2- 172.21.15.1:0/1452575762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 0x7fc170070700 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.715+0000 7fc175200700 1 -- 172.21.15.1:0/1452575762 >> 172.21.15.1:0/1452575762 conn(0x7fc1700f8ae0 msgr2=0x7fc1700faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.715+0000 7fc175200700 1 -- 172.21.15.1:0/1452575762 shutdown_connections 2024-02-28T17:12:54.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.715+0000 7fc175200700 1 -- 172.21.15.1:0/1452575762 wait complete. 2024-02-28T17:12:54.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.716+0000 7fc175200700 1 Processor -- start 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.716+0000 7fc175200700 1 -- start start 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc175200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 0x7fc170115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc175200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc170070c40 0x7fc170116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc175200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1701167b0 con 0x7fc170070330 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc175200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17010fae0 con 0x7fc170070c40 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc16de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 0x7fc170115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc16de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 0x7fc170115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34456/0 (socket says 172.21.15.1:34456) 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc16de00700 1 -- 172.21.15.1:0/745278529 learned_addr learned my addr 172.21.15.1:0/745278529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc16de00700 1 -- 172.21.15.1:0/745278529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc170070c40 msgr2=0x7fc170116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc16de00700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc170070c40 0x7fc170116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.717+0000 7fc16de00700 1 -- 172.21.15.1:0/745278529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc160005040 con 0x7fc170070330 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.718+0000 7fc16de00700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 0x7fc170115c20 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc160004f50 tx=0x7fc160009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:54.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.718+0000 7fc166000700 1 -- 172.21.15.1:0/745278529 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16000e040 con 0x7fc170070330 2024-02-28T17:12:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.718+0000 7fc166000700 1 -- 172.21.15.1:0/745278529 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc160004290 con 0x7fc170070330 2024-02-28T17:12:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.718+0000 7fc166000700 1 -- 172.21.15.1:0/745278529 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16001c630 con 0x7fc170070330 2024-02-28T17:12:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.718+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17010fd60 con 0x7fc170070330 2024-02-28T17:12:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.718+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1701102a0 con 0x7fc170070330 2024-02-28T17:12:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.719+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc170106920 con 0x7fc170070330 2024-02-28T17:12:54.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.720+0000 7fc166000700 1 -- 172.21.15.1:0/745278529 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc16000bc10 con 0x7fc170070330 2024-02-28T17:12:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.721+0000 7fc166000700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15406d7f0 0x7fc15406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:54.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.721+0000 7fc166000700 1 -- 172.21.15.1:0/745278529 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc16008e240 con 0x7fc170070330 2024-02-28T17:12:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.721+0000 7fc16d400700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15406d7f0 0x7fc15406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:54.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.722+0000 7fc16d400700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15406d7f0 0x7fc15406fca0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc170111450 tx=0x7fc15c009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:54.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.726+0000 7fc166000700 1 -- 172.21.15.1:0/745278529 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc16005b660 con 0x7fc170070330 2024-02-28T17:12:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.970+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc17010c240 con 0x7fc15406d7f0 2024-02-28T17:12:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.972+0000 7fc166000700 1 -- 172.21.15.1:0/745278529 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc17010c240 con 0x7fc15406d7f0 2024-02-28T17:12:54.975 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:12:54.976 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:12:54.977 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:12:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.974+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15406d7f0 msgr2=0x7fc15406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.974+0000 7fc175200700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15406d7f0 0x7fc15406fca0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc170111450 tx=0x7fc15c009ce0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.974+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 msgr2=0x7fc170115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.974+0000 7fc175200700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 0x7fc170115c20 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc160004f50 tx=0x7fc160009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.975+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 shutdown_connections 2024-02-28T17:12:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.975+0000 7fc175200700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc15406d7f0 0x7fc15406fca0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.975+0000 7fc175200700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc170070330 0x7fc170115c20 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.975+0000 7fc175200700 1 --2- 172.21.15.1:0/745278529 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc170070c40 0x7fc170116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.975+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 >> 172.21.15.1:0/745278529 conn(0x7fc1700f8ae0 msgr2=0x7fc170103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.975+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 shutdown_connections 2024-02-28T17:12:54.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:54.975+0000 7fc175200700 1 -- 172.21.15.1:0/745278529 wait complete. 2024-02-28T17:12:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.127+0000 7f9959600700 1 -- 172.21.15.1:0/2206052446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105170 msgr2=0x7f9954105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.127+0000 7f9959600700 1 --2- 172.21.15.1:0/2206052446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105170 0x7f9954105540 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f9944004940 tx=0x7f9944009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.127+0000 7f9959600700 1 -- 172.21.15.1:0/2206052446 shutdown_connections 2024-02-28T17:12:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.127+0000 7f9959600700 1 --2- 172.21.15.1:0/2206052446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9954105b10 0x7f99540ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.127+0000 7f9959600700 1 --2- 172.21.15.1:0/2206052446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105170 0x7f9954105540 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.127+0000 7f9959600700 1 -- 172.21.15.1:0/2206052446 >> 172.21.15.1:0/2206052446 conn(0x7f99540face0 msgr2=0x7f99540fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.127+0000 7f9959600700 1 -- 172.21.15.1:0/2206052446 shutdown_connections 2024-02-28T17:12:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.127+0000 7f9959600700 1 -- 172.21.15.1:0/2206052446 wait complete. 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.128+0000 7f9959600700 1 Processor -- start 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.128+0000 7f9959600700 1 -- start start 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.128+0000 7f9959600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9954105170 0x7f9954115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.128+0000 7f9959600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105b10 0x7f99541161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.128+0000 7f9959600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99541168a0 con 0x7f9954105b10 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.128+0000 7f9959600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995410fb40 con 0x7f9954105170 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.129+0000 7f9952000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105b10 0x7f99541161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.129+0000 7f9952000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105b10 0x7f99541161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34470/0 (socket says 172.21.15.1:34470) 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.129+0000 7f9952000700 1 -- 172.21.15.1:0/2039537996 learned_addr learned my addr 172.21.15.1:0/2039537996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.129+0000 7f9952000700 1 -- 172.21.15.1:0/2039537996 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9954105170 msgr2=0x7f9954115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.129+0000 7f9952000700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9954105170 0x7f9954115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:55.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.129+0000 7f9952000700 1 -- 172.21.15.1:0/2039537996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9944005040 con 0x7f9954105b10 2024-02-28T17:12:55.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.129+0000 7f9952000700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105b10 0x7f99541161c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f994000b790 tx=0x7f9940009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:55.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.130+0000 7f994aa00700 1 -- 172.21.15.1:0/2039537996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994000f6f0 con 0x7f9954105b10 2024-02-28T17:12:55.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.130+0000 7f994aa00700 1 -- 172.21.15.1:0/2039537996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f994000fd20 con 0x7f9954105b10 2024-02-28T17:12:55.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.130+0000 7f994aa00700 1 -- 172.21.15.1:0/2039537996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99400103a0 con 0x7f9954105b10 2024-02-28T17:12:55.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.130+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995410fe20 con 0x7f9954105b10 2024-02-28T17:12:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.130+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9954110360 con 0x7f9954105b10 2024-02-28T17:12:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.131+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99540fe900 con 0x7f9954105b10 2024-02-28T17:12:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.132+0000 7f994aa00700 1 -- 172.21.15.1:0/2039537996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9940011400 con 0x7f9954105b10 2024-02-28T17:12:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.133+0000 7f994aa00700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993806d7f0 0x7f993806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:12:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.133+0000 7f994aa00700 1 -- 172.21.15.1:0/2039537996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f994008cc90 con 0x7f9954105b10 2024-02-28T17:12:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.133+0000 7f9952a00700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993806d7f0 0x7f993806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:12:55.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.134+0000 7f9952a00700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993806d7f0 0x7f993806fca0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9944004f90 tx=0x7f9944009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:12:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.138+0000 7f994aa00700 1 -- 172.21.15.1:0/2039537996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f994005a090 con 0x7f9954105b10 2024-02-28T17:12:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:54 smithi066 ceph-mon[85781]: from='client.36052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:54 smithi066 ceph-mon[85781]: pgmap v1157: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:54 smithi066 ceph-mon[85781]: from='client.36056 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:54 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2419279191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:54 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3061143288' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:12:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:54 smithi001 ceph-mon[112940]: from='client.36052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:54 smithi001 ceph-mon[112940]: pgmap v1157: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:54 smithi001 ceph-mon[112940]: from='client.36056 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:54 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2419279191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:12:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:54 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3061143288' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:12:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.446+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9954110630 con 0x7f9954105b10 2024-02-28T17:12:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.447+0000 7f994aa00700 1 -- 172.21.15.1:0/2039537996 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f9940059c20 con 0x7f9954105b10 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:12:55.451 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:12:55.452 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:12:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.450+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993806d7f0 msgr2=0x7f993806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.450+0000 7f9959600700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993806d7f0 0x7f993806fca0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9944004f90 tx=0x7f9944009e30 comp rx=0 tx=0).stop 2024-02-28T17:12:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.450+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105b10 msgr2=0x7f99541161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:12:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.451+0000 7f9959600700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105b10 0x7f99541161c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f994000b790 tx=0x7f9940009220 comp rx=0 tx=0).stop 2024-02-28T17:12:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.451+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 shutdown_connections 2024-02-28T17:12:55.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.451+0000 7f9959600700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993806d7f0 0x7f993806fca0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:55.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.451+0000 7f9959600700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9954105170 0x7f9954115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:55.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.451+0000 7f9959600700 1 --2- 172.21.15.1:0/2039537996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9954105b10 0x7f99541161c0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:12:55.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.451+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 >> 172.21.15.1:0/2039537996 conn(0x7f99540face0 msgr2=0x7f99540fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:12:55.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.451+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 shutdown_connections 2024-02-28T17:12:55.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:12:55.451+0000 7f9959600700 1 -- 172.21.15.1:0/2039537996 wait complete. 2024-02-28T17:12:56.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:55 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2039537996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:12:56.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:12:56.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:55 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2039537996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:12:56.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:12:57.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:56 smithi066 ceph-mon[85781]: from='client.36068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:57.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:56 smithi066 ceph-mon[85781]: pgmap v1158: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:57.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:56 smithi001 ceph-mon[112940]: from='client.36068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:12:57.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:56 smithi001 ceph-mon[112940]: pgmap v1158: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:58.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:12:57 smithi066 ceph-mon[85781]: pgmap v1159: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:12:58.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:12:57 smithi001 ceph-mon[112940]: pgmap v1159: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:00.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:00 smithi066 ceph-mon[85781]: pgmap v1160: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:00.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:00 smithi001 ceph-mon[112940]: pgmap v1160: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:02.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:02 smithi066 ceph-mon[85781]: pgmap v1161: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:02.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:02 smithi001 ceph-mon[112940]: pgmap v1161: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:04.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:04 smithi066 ceph-mon[85781]: pgmap v1162: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:04.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:04 smithi001 ceph-mon[112940]: pgmap v1162: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:06.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:06 smithi066 ceph-mon[85781]: pgmap v1163: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:06.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:06 smithi001 ceph-mon[112940]: pgmap v1163: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:08.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:08 smithi066 ceph-mon[85781]: pgmap v1164: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:08.962 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:08 smithi001 ceph-mon[112940]: pgmap v1164: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:10.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:10 smithi066 ceph-mon[85781]: pgmap v1165: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:10.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:10 smithi001 ceph-mon[112940]: pgmap v1165: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:11.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:13:11.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:13:12.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:12 smithi066 ceph-mon[85781]: pgmap v1166: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:12.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:12 smithi001 ceph-mon[112940]: pgmap v1166: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:14.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:14 smithi066 ceph-mon[85781]: pgmap v1167: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:14.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:14 smithi001 ceph-mon[112940]: pgmap v1167: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:16.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:16 smithi066 ceph-mon[85781]: pgmap v1168: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:16.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:16 smithi001 ceph-mon[112940]: pgmap v1168: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:18.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:18 smithi066 ceph-mon[85781]: pgmap v1169: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:18.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:18 smithi001 ceph-mon[112940]: pgmap v1169: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:20.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:20 smithi066 ceph-mon[85781]: pgmap v1170: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:20.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:20 smithi001 ceph-mon[112940]: pgmap v1170: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:22.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:22 smithi066 ceph-mon[85781]: pgmap v1171: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:22.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:22 smithi001 ceph-mon[112940]: pgmap v1171: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:24.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:24 smithi066 ceph-mon[85781]: pgmap v1172: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:24.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:24 smithi001 ceph-mon[112940]: pgmap v1172: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:25.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.624+0000 7f081a800700 1 -- 172.21.15.1:0/57061042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105b10 msgr2=0x7f08140ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:25.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.624+0000 7f081a800700 1 --2- 172.21.15.1:0/57061042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105b10 0x7f08140ff180 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0800004940 tx=0x7f0800009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:25.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.624+0000 7f081a800700 1 -- 172.21.15.1:0/57061042 shutdown_connections 2024-02-28T17:13:25.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.624+0000 7f081a800700 1 --2- 172.21.15.1:0/57061042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105b10 0x7f08140ff180 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:25.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.624+0000 7f081a800700 1 --2- 172.21.15.1:0/57061042 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0814105170 0x7f0814105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:25.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.624+0000 7f081a800700 1 -- 172.21.15.1:0/57061042 >> 172.21.15.1:0/57061042 conn(0x7f08140face0 msgr2=0x7f08140fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:25.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.625+0000 7f081a800700 1 -- 172.21.15.1:0/57061042 shutdown_connections 2024-02-28T17:13:25.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.625+0000 7f081a800700 1 -- 172.21.15.1:0/57061042 wait complete. 2024-02-28T17:13:25.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.625+0000 7f081a800700 1 Processor -- start 2024-02-28T17:13:25.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.626+0000 7f081a800700 1 -- start start 2024-02-28T17:13:25.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.626+0000 7f081a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105170 0x7f0814115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:25.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.626+0000 7f081a800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0814105b10 0x7f0814116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:25.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.626+0000 7f081a800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814116870 con 0x7f0814105170 2024-02-28T17:13:25.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.626+0000 7f081a800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081410fb80 con 0x7f0814105b10 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.626+0000 7f0813400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105170 0x7f0814115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.626+0000 7f0813400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105170 0x7f0814115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56154/0 (socket says 172.21.15.1:56154) 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.626+0000 7f0813400700 1 -- 172.21.15.1:0/1055622199 learned_addr learned my addr 172.21.15.1:0/1055622199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f0813400700 1 -- 172.21.15.1:0/1055622199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0814105b10 msgr2=0x7f0814116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f0813400700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0814105b10 0x7f0814116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f0813400700 1 -- 172.21.15.1:0/1055622199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0800005040 con 0x7f0814105170 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f0813400700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105170 0x7f0814115ce0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0804002a50 tx=0x7f08040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f080b400700 1 -- 172.21.15.1:0/1055622199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0804007cc0 con 0x7f0814105170 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f080b400700 1 -- 172.21.15.1:0/1055622199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f080400f040 con 0x7f0814105170 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f080b400700 1 -- 172.21.15.1:0/1055622199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0804009640 con 0x7f0814105170 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081410fe60 con 0x7f0814105170 2024-02-28T17:13:25.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.627+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08141103a0 con 0x7f0814105170 2024-02-28T17:13:25.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.629+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0814107640 con 0x7f0814105170 2024-02-28T17:13:25.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.630+0000 7f080b400700 1 -- 172.21.15.1:0/1055622199 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f08040097a0 con 0x7f0814105170 2024-02-28T17:13:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.630+0000 7f080b400700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07f806d7f0 0x7f07f806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.631+0000 7f080b400700 1 -- 172.21.15.1:0/1055622199 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f080408cc60 con 0x7f0814105170 2024-02-28T17:13:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.631+0000 7f0812a00700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07f806d7f0 0x7f07f806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.631+0000 7f0812a00700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07f806d7f0 0x7f07f806fca0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0800004f90 tx=0x7f0800009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.636+0000 7f080b400700 1 -- 172.21.15.1:0/1055622199 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0804059fc0 con 0x7f0814105170 2024-02-28T17:13:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.882+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f081404bf40 con 0x7f07f806d7f0 2024-02-28T17:13:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.884+0000 7f080b400700 1 -- 172.21.15.1:0/1055622199 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f081404bf40 con 0x7f07f806d7f0 2024-02-28T17:13:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07f806d7f0 msgr2=0x7f07f806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07f806d7f0 0x7f07f806fca0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0800004f90 tx=0x7f0800009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105170 msgr2=0x7f0814115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105170 0x7f0814115ce0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0804002a50 tx=0x7f08040046b0 comp rx=0 tx=0).stop 2024-02-28T17:13:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 shutdown_connections 2024-02-28T17:13:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07f806d7f0 0x7f07f806fca0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0814105170 0x7f0814115ce0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 --2- 172.21.15.1:0/1055622199 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0814105b10 0x7f0814116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.887+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 >> 172.21.15.1:0/1055622199 conn(0x7f08140face0 msgr2=0x7f08140fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.888+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 shutdown_connections 2024-02-28T17:13:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:25.888+0000 7f081a800700 1 -- 172.21.15.1:0/1055622199 wait complete. 2024-02-28T17:13:25.908 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:13:26.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.048+0000 7fd325c00700 1 -- 172.21.15.1:0/290361479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3201095f0 msgr2=0x7fd3201099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.048+0000 7fd325c00700 1 --2- 172.21.15.1:0/290361479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3201095f0 0x7fd3201099c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd310004940 tx=0x7fd310009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:26.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.048+0000 7fd325c00700 1 -- 172.21.15.1:0/290361479 shutdown_connections 2024-02-28T17:13:26.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.048+0000 7fd325c00700 1 --2- 172.21.15.1:0/290361479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd320103dc0 0x7fd320104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.048+0000 7fd325c00700 1 --2- 172.21.15.1:0/290361479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3201095f0 0x7fd3201099c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.049+0000 7fd325c00700 1 -- 172.21.15.1:0/290361479 >> 172.21.15.1:0/290361479 conn(0x7fd320072d50 msgr2=0x7fd320075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:26.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.049+0000 7fd325c00700 1 -- 172.21.15.1:0/290361479 shutdown_connections 2024-02-28T17:13:26.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.049+0000 7fd325c00700 1 -- 172.21.15.1:0/290361479 wait complete. 2024-02-28T17:13:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.050+0000 7fd325c00700 1 Processor -- start 2024-02-28T17:13:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.050+0000 7fd325c00700 1 -- start start 2024-02-28T17:13:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.050+0000 7fd325c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd320103dc0 0x7fd32011a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.050+0000 7fd325c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3201095f0 0x7fd32011a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.050+0000 7fd325c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32011ac70 con 0x7fd320103dc0 2024-02-28T17:13:26.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.050+0000 7fd325c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd320113f80 con 0x7fd3201095f0 2024-02-28T17:13:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.051+0000 7fd31ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd320103dc0 0x7fd32011a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.051+0000 7fd31ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd320103dc0 0x7fd32011a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56162/0 (socket says 172.21.15.1:56162) 2024-02-28T17:13:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.051+0000 7fd31ea00700 1 -- 172.21.15.1:0/723204115 learned_addr learned my addr 172.21.15.1:0/723204115 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.051+0000 7fd31ea00700 1 -- 172.21.15.1:0/723204115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3201095f0 msgr2=0x7fd32011a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.051+0000 7fd31ea00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3201095f0 0x7fd32011a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.051+0000 7fd31ea00700 1 -- 172.21.15.1:0/723204115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310005040 con 0x7fd320103dc0 2024-02-28T17:13:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.051+0000 7fd31ea00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd320103dc0 0x7fd32011a0e0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd310009150 tx=0x7fd310009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:26.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.052+0000 7fd316a00700 1 -- 172.21.15.1:0/723204115 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd31000e040 con 0x7fd320103dc0 2024-02-28T17:13:26.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.052+0000 7fd316a00700 1 -- 172.21.15.1:0/723204115 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd31000bbf0 con 0x7fd320103dc0 2024-02-28T17:13:26.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.052+0000 7fd316a00700 1 -- 172.21.15.1:0/723204115 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd31001c630 con 0x7fd320103dc0 2024-02-28T17:13:26.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.052+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd320114200 con 0x7fd320103dc0 2024-02-28T17:13:26.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.052+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd320114740 con 0x7fd320103dc0 2024-02-28T17:13:26.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.053+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd320106850 con 0x7fd320103dc0 2024-02-28T17:13:26.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.054+0000 7fd316a00700 1 -- 172.21.15.1:0/723204115 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd31000bd60 con 0x7fd320103dc0 2024-02-28T17:13:26.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.055+0000 7fd316a00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd30406d7f0 0x7fd30406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.055+0000 7fd316a00700 1 -- 172.21.15.1:0/723204115 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd31008e2a0 con 0x7fd320103dc0 2024-02-28T17:13:26.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.055+0000 7fd317e00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd30406d7f0 0x7fd30406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:26.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.056+0000 7fd317e00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd30406d7f0 0x7fd30406fca0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd320115980 tx=0x7fd308009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:26.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.060+0000 7fd316a00700 1 -- 172.21.15.1:0/723204115 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd31005b680 con 0x7fd320103dc0 2024-02-28T17:13:26.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.302+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd32004bf40 con 0x7fd30406d7f0 2024-02-28T17:13:26.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.304+0000 7fd316a00700 1 -- 172.21.15.1:0/723204115 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd32004bf40 con 0x7fd30406d7f0 2024-02-28T17:13:26.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd30406d7f0 msgr2=0x7fd30406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd30406d7f0 0x7fd30406fca0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd320115980 tx=0x7fd308009ce0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd320103dc0 msgr2=0x7fd32011a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd320103dc0 0x7fd32011a0e0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd310009150 tx=0x7fd310009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:26.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 shutdown_connections 2024-02-28T17:13:26.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd30406d7f0 0x7fd30406fca0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd320103dc0 0x7fd32011a0e0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 --2- 172.21.15.1:0/723204115 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3201095f0 0x7fd32011a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 >> 172.21.15.1:0/723204115 conn(0x7fd320072d50 msgr2=0x7fd320074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:26.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.306+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 shutdown_connections 2024-02-28T17:13:26.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.307+0000 7fd325c00700 1 -- 172.21.15.1:0/723204115 wait complete. 2024-02-28T17:13:26.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.465+0000 7f0b41000700 1 -- 172.21.15.1:0/401568805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c070870 msgr2=0x7f0b3c070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.465+0000 7f0b41000700 1 --2- 172.21.15.1:0/401568805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c070870 0x7f0b3c070ce0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0b24004970 tx=0x7f0b24009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:26.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.465+0000 7f0b41000700 1 -- 172.21.15.1:0/401568805 shutdown_connections 2024-02-28T17:13:26.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.465+0000 7f0b41000700 1 --2- 172.21.15.1:0/401568805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c070870 0x7f0b3c070ce0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.465+0000 7f0b41000700 1 --2- 172.21.15.1:0/401568805 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b3c071f60 0x7f0b3c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.465+0000 7f0b41000700 1 -- 172.21.15.1:0/401568805 >> 172.21.15.1:0/401568805 conn(0x7f0b3c0f8ae0 msgr2=0x7f0b3c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:26.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.465+0000 7f0b41000700 1 -- 172.21.15.1:0/401568805 shutdown_connections 2024-02-28T17:13:26.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.465+0000 7f0b41000700 1 -- 172.21.15.1:0/401568805 wait complete. 2024-02-28T17:13:26.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.466+0000 7f0b41000700 1 Processor -- start 2024-02-28T17:13:26.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.467+0000 7f0b41000700 1 -- start start 2024-02-28T17:13:26.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.467+0000 7f0b41000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b3c070870 0x7f0b3c11a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.467+0000 7f0b41000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c071f60 0x7f0b3c11a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.467+0000 7f0b41000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c11aba0 con 0x7f0b3c071f60 2024-02-28T17:13:26.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.467+0000 7f0b41000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c113eb0 con 0x7f0b3c070870 2024-02-28T17:13:26.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.467+0000 7f0b31400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c071f60 0x7f0b3c11a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:26.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.467+0000 7f0b31400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c071f60 0x7f0b3c11a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56176/0 (socket says 172.21.15.1:56176) 2024-02-28T17:13:26.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.467+0000 7f0b31400700 1 -- 172.21.15.1:0/1518324544 learned_addr learned my addr 172.21.15.1:0/1518324544 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:26.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.468+0000 7f0b39e00700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b3c070870 0x7f0b3c11a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.468+0000 7f0b31400700 1 -- 172.21.15.1:0/1518324544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b3c070870 msgr2=0x7f0b3c11a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.468+0000 7f0b31400700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b3c070870 0x7f0b3c11a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.468+0000 7f0b31400700 1 -- 172.21.15.1:0/1518324544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24005040 con 0x7f0b3c071f60 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.468+0000 7f0b31400700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c071f60 0x7f0b3c11a550 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009150 tx=0x7f0b24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.469+0000 7f0b32a00700 1 -- 172.21.15.1:0/1518324544 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b2400e040 con 0x7f0b3c071f60 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.469+0000 7f0b32a00700 1 -- 172.21.15.1:0/1518324544 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b2400bc10 con 0x7f0b3c071f60 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.469+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c114130 con 0x7f0b3c071f60 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.469+0000 7f0b32a00700 1 -- 172.21.15.1:0/1518324544 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b2401c630 con 0x7f0b3c071f60 2024-02-28T17:13:26.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.469+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c114670 con 0x7f0b3c071f60 2024-02-28T17:13:26.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.470+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c0fe400 con 0x7f0b3c071f60 2024-02-28T17:13:26.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.471+0000 7f0b32a00700 1 -- 172.21.15.1:0/1518324544 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0b2400bd80 con 0x7f0b3c071f60 2024-02-28T17:13:26.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.472+0000 7f0b32a00700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b2006d7f0 0x7f0b2006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.473+0000 7f0b32a00700 1 -- 172.21.15.1:0/1518324544 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0b2408e2e0 con 0x7f0b3c071f60 2024-02-28T17:13:26.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.473+0000 7f0b39e00700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b2006d7f0 0x7f0b2006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:26.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.474+0000 7f0b39e00700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b2006d7f0 0x7f0b2006fca0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c004fa0 tx=0x7f0b2c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:26.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.478+0000 7f0b32a00700 1 -- 172.21.15.1:0/1518324544 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0b2405b6c0 con 0x7f0b3c071f60 2024-02-28T17:13:26.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.711+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b3c04bed0 con 0x7f0b2006d7f0 2024-02-28T17:13:26.714 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:26 smithi001 ceph-mon[112940]: pgmap v1173: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:26.714 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:13:26.714 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:26 smithi001 ceph-mon[112940]: from='client.36076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:13:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.741+0000 7f0b32a00700 1 -- 172.21.15.1:0/1518324544 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0b3c04bed0 con 0x7f0b2006d7f0 2024-02-28T17:13:26.744 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (39m) 64s ago 53m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (36m) 64s ago 54m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (36m) 66s ago 52m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (39m) 64s ago 53m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (32m) 64s ago 46m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (31m) 64s ago 46m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (71s) 66s ago 46m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (16m) 66s ago 46m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (42m) 64s ago 55m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (41m) 66s ago 52m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (37m) 64s ago 55m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (37m) 66s ago 52m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (40m) 64s ago 54m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:13:26.745 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (40m) 66s ago 52m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:13:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (35m) 64s ago 51m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:13:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (35m) 64s ago 50m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:13:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (34m) 64s ago 50m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:13:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (34m) 66s ago 49m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:13:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (33m) 66s ago 49m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:13:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (32m) 66s ago 48m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:13:26.746 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (40m) 64s ago 53m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b2006d7f0 msgr2=0x7f0b2006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b2006d7f0 0x7f0b2006fca0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c004fa0 tx=0x7f0b2c00b040 comp rx=0 tx=0).stop 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c071f60 msgr2=0x7f0b3c11a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c071f60 0x7f0b3c11a550 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009150 tx=0x7f0b24009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 shutdown_connections 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b2006d7f0 0x7f0b2006fca0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b3c070870 0x7f0b3c11a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 --2- 172.21.15.1:0/1518324544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b3c071f60 0x7f0b3c11a550 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 >> 172.21.15.1:0/1518324544 conn(0x7f0b3c0f8ae0 msgr2=0x7f0b3c0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 shutdown_connections 2024-02-28T17:13:26.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.744+0000 7f0b41000700 1 -- 172.21.15.1:0/1518324544 wait complete. 2024-02-28T17:13:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.892+0000 7fdbf5400700 1 -- 172.21.15.1:0/2885350284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf010d9b0 msgr2=0x7fdbf010dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.892+0000 7fdbf5400700 1 --2- 172.21.15.1:0/2885350284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf010d9b0 0x7fdbf010dd80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0004940 tx=0x7fdbe0009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.892+0000 7fdbf5400700 1 -- 172.21.15.1:0/2885350284 shutdown_connections 2024-02-28T17:13:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.892+0000 7fdbf5400700 1 --2- 172.21.15.1:0/2885350284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbf0107fb0 0x7fdbf0108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.892+0000 7fdbf5400700 1 --2- 172.21.15.1:0/2885350284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf010d9b0 0x7fdbf010dd80 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.892+0000 7fdbf5400700 1 -- 172.21.15.1:0/2885350284 >> 172.21.15.1:0/2885350284 conn(0x7fdbf0067b10 msgr2=0x7fdbf0067f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.892+0000 7fdbf5400700 1 -- 172.21.15.1:0/2885350284 shutdown_connections 2024-02-28T17:13:26.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.892+0000 7fdbf5400700 1 -- 172.21.15.1:0/2885350284 wait complete. 2024-02-28T17:13:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.893+0000 7fdbf5400700 1 Processor -- start 2024-02-28T17:13:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.893+0000 7fdbf5400700 1 -- start start 2024-02-28T17:13:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbf5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf0107fb0 0x7fdbf011e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbf5400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbf010d9b0 0x7fdbf011ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbf5400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf011f050 con 0x7fdbf0107fb0 2024-02-28T17:13:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbf5400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf0118380 con 0x7fdbf010d9b0 2024-02-28T17:13:26.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbee800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf0107fb0 0x7fdbf011e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbee800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf0107fb0 0x7fdbf011e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56192/0 (socket says 172.21.15.1:56192) 2024-02-28T17:13:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbee800700 1 -- 172.21.15.1:0/1022502825 learned_addr learned my addr 172.21.15.1:0/1022502825 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbede00700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbf010d9b0 0x7fdbf011ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbee800700 1 -- 172.21.15.1:0/1022502825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbf010d9b0 msgr2=0x7fdbf011ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbee800700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbf010d9b0 0x7fdbf011ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbee800700 1 -- 172.21.15.1:0/1022502825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0005040 con 0x7fdbf0107fb0 2024-02-28T17:13:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.894+0000 7fdbede00700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbf010d9b0 0x7fdbf011ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:13:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.895+0000 7fdbee800700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf0107fb0 0x7fdbf011e4c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009100 tx=0x7fdbe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.895+0000 7fdbe6a00700 1 -- 172.21.15.1:0/1022502825 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbe000e040 con 0x7fdbf0107fb0 2024-02-28T17:13:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.895+0000 7fdbe6a00700 1 -- 172.21.15.1:0/1022502825 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbe0004290 con 0x7fdbf0107fb0 2024-02-28T17:13:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.895+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf0118600 con 0x7fdbf0107fb0 2024-02-28T17:13:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.895+0000 7fdbe6a00700 1 -- 172.21.15.1:0/1022502825 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbe001c3f0 con 0x7fdbf0107fb0 2024-02-28T17:13:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.895+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf0118b40 con 0x7fdbf0107fb0 2024-02-28T17:13:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.896+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf010ac50 con 0x7fdbf0107fb0 2024-02-28T17:13:26.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.897+0000 7fdbe6a00700 1 -- 172.21.15.1:0/1022502825 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdbe000bbf0 con 0x7fdbf0107fb0 2024-02-28T17:13:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.898+0000 7fdbe6a00700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdbd406d7f0 0x7fdbd406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.899+0000 7fdbe6a00700 1 -- 172.21.15.1:0/1022502825 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdbe008e1c0 con 0x7fdbf0107fb0 2024-02-28T17:13:26.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.899+0000 7fdbede00700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdbd406d7f0 0x7fdbd406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:26.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.900+0000 7fdbede00700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdbd406d7f0 0x7fdbd406fca0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc007fd0 tx=0x7fdbdc009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:26.903+0000 7fdbe6a00700 1 -- 172.21.15.1:0/1022502825 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdbe005b5a0 con 0x7fdbf0107fb0 2024-02-28T17:13:27.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:26 smithi066 ceph-mon[85781]: pgmap v1173: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:27.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:13:27.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:26 smithi066 ceph-mon[85781]: from='client.36076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:13:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.213+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdbf004bed0 con 0x7fdbf0107fb0 2024-02-28T17:13:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.214+0000 7fdbe6a00700 1 -- 172.21.15.1:0/1022502825 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fdbe005b130 con 0x7fdbf0107fb0 2024-02-28T17:13:27.217 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:13:27.218 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:13:27.219 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:13:27.219 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:13:27.219 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdbd406d7f0 msgr2=0x7fdbd406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdbd406d7f0 0x7fdbd406fca0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc007fd0 tx=0x7fdbdc009040 comp rx=0 tx=0).stop 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf0107fb0 msgr2=0x7fdbf011e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf0107fb0 0x7fdbf011e4c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009100 tx=0x7fdbe0009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 shutdown_connections 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdbd406d7f0 0x7fdbd406fca0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbf0107fb0 0x7fdbf011e4c0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 --2- 172.21.15.1:0/1022502825 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdbf010d9b0 0x7fdbf011ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 >> 172.21.15.1:0/1022502825 conn(0x7fdbf0067b10 msgr2=0x7fdbf0106ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:27.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 shutdown_connections 2024-02-28T17:13:27.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.217+0000 7fdbf5400700 1 -- 172.21.15.1:0/1022502825 wait complete. 2024-02-28T17:13:27.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.370+0000 7f7660400700 1 -- 172.21.15.1:0/120312533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76581027e0 msgr2=0x7f76580fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:27.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.370+0000 7f7660400700 1 --2- 172.21.15.1:0/120312533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76581027e0 0x7f76580fbea0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f7644004940 tx=0x7f7644009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:27.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.370+0000 7f7660400700 1 -- 172.21.15.1:0/120312533 shutdown_connections 2024-02-28T17:13:27.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.370+0000 7f7660400700 1 --2- 172.21.15.1:0/120312533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76581027e0 0x7f76580fbea0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.370+0000 7f7660400700 1 --2- 172.21.15.1:0/120312533 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7658101e40 0x7f7658102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.370+0000 7f7660400700 1 -- 172.21.15.1:0/120312533 >> 172.21.15.1:0/120312533 conn(0x7f7658071d90 msgr2=0x7f7658072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:27.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.370+0000 7f7660400700 1 -- 172.21.15.1:0/120312533 shutdown_connections 2024-02-28T17:13:27.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.370+0000 7f7660400700 1 -- 172.21.15.1:0/120312533 wait complete. 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.371+0000 7f7660400700 1 Processor -- start 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.371+0000 7f7660400700 1 -- start start 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f7660400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658101e40 0x7f765810eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f7660400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f76581027e0 0x7f765810f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f7660400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765810f600 con 0x7f7658101e40 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f7660400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765810f770 con 0x7f76581027e0 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f765da00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658101e40 0x7f765810eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f765da00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658101e40 0x7f765810eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56210/0 (socket says 172.21.15.1:56210) 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f765da00700 1 -- 172.21.15.1:0/3758858590 learned_addr learned my addr 172.21.15.1:0/3758858590 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f765da00700 1 -- 172.21.15.1:0/3758858590 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f76581027e0 msgr2=0x7f765810f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f765da00700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f76581027e0 0x7f765810f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.372+0000 7f765da00700 1 -- 172.21.15.1:0/3758858590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7644005040 con 0x7f7658101e40 2024-02-28T17:13:27.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.373+0000 7f765da00700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658101e40 0x7f765810eb10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f764c002a50 tx=0x7f764c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.373+0000 7f7655600700 1 -- 172.21.15.1:0/3758858590 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f764c007cc0 con 0x7f7658101e40 2024-02-28T17:13:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.373+0000 7f7655600700 1 -- 172.21.15.1:0/3758858590 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f764c00f040 con 0x7f7658101e40 2024-02-28T17:13:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.373+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7658117ae0 con 0x7f7658101e40 2024-02-28T17:13:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.373+0000 7f7655600700 1 -- 172.21.15.1:0/3758858590 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f764c009640 con 0x7f7658101e40 2024-02-28T17:13:27.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.373+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7658117ff0 con 0x7f7658101e40 2024-02-28T17:13:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.374+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7658104310 con 0x7f7658101e40 2024-02-28T17:13:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.375+0000 7f7655600700 1 -- 172.21.15.1:0/3758858590 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f764c0097a0 con 0x7f7658101e40 2024-02-28T17:13:27.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.376+0000 7f7655600700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764006d7f0 0x7f764006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:27.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.376+0000 7f7655600700 1 -- 172.21.15.1:0/3758858590 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f764c08cc20 con 0x7f7658101e40 2024-02-28T17:13:27.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.377+0000 7f765d000700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764006d7f0 0x7f764006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:27.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.377+0000 7f765d000700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764006d7f0 0x7f764006fca0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7644004f90 tx=0x7f7644009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:27.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.381+0000 7f7655600700 1 -- 172.21.15.1:0/3758858590 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f764c056450 con 0x7f7658101e40 2024-02-28T17:13:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.658+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7658065ec0 con 0x7f7658101e40 2024-02-28T17:13:27.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.659+0000 7f7655600700 1 -- 172.21.15.1:0/3758858590 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f764c059b10 con 0x7f7658101e40 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:13:27.662 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:13:27.663 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:13:27.664 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:13:27.665 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:13:27.665 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.661+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764006d7f0 msgr2=0x7f764006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.661+0000 7f7660400700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764006d7f0 0x7f764006fca0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7644004f90 tx=0x7f7644009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.661+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658101e40 msgr2=0x7f765810eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.661+0000 7f7660400700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658101e40 0x7f765810eb10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f764c002a50 tx=0x7f764c0046b0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.662+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 shutdown_connections 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.662+0000 7f7660400700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f764006d7f0 0x7f764006fca0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.662+0000 7f7660400700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658101e40 0x7f765810eb10 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.662+0000 7f7660400700 1 --2- 172.21.15.1:0/3758858590 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f76581027e0 0x7f765810f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.662+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 >> 172.21.15.1:0/3758858590 conn(0x7f7658071d90 msgr2=0x7f76580fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.662+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 shutdown_connections 2024-02-28T17:13:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.662+0000 7f7660400700 1 -- 172.21.15.1:0/3758858590 wait complete. 2024-02-28T17:13:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:13:27.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:27 smithi001 ceph-mon[112940]: from='client.36080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:13:27.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:27 smithi001 ceph-mon[112940]: from='client.36084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:13:27.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:27 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1022502825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:13:27.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.819+0000 7f4587800700 1 -- 172.21.15.1:0/2124703853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 msgr2=0x7f4580103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.819+0000 7f4587800700 1 --2- 172.21.15.1:0/2124703853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 0x7f4580103320 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4574004940 tx=0x7f4574009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.819+0000 7f4587800700 1 -- 172.21.15.1:0/2124703853 shutdown_connections 2024-02-28T17:13:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.819+0000 7f4587800700 1 --2- 172.21.15.1:0/2124703853 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45801038f0 0x7f45800fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.819+0000 7f4587800700 1 --2- 172.21.15.1:0/2124703853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 0x7f4580103320 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.819+0000 7f4587800700 1 -- 172.21.15.1:0/2124703853 >> 172.21.15.1:0/2124703853 conn(0x7f45800f8ae0 msgr2=0x7f45800faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.820+0000 7f4587800700 1 -- 172.21.15.1:0/2124703853 shutdown_connections 2024-02-28T17:13:27.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.820+0000 7f4587800700 1 -- 172.21.15.1:0/2124703853 wait complete. 2024-02-28T17:13:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.820+0000 7f4587800700 1 Processor -- start 2024-02-28T17:13:27.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.821+0000 7f4587800700 1 -- start start 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.821+0000 7f4587800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 0x7f4580115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.821+0000 7f4587800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45801038f0 0x7f4580116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.821+0000 7f4587800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4580116780 con 0x7f4580102f50 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.821+0000 7f4587800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458010fad0 con 0x7f45801038f0 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.821+0000 7f4584e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 0x7f4580115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.821+0000 7f4584e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 0x7f4580115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56232/0 (socket says 172.21.15.1:56232) 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.821+0000 7f4584e00700 1 -- 172.21.15.1:0/4048432463 learned_addr learned my addr 172.21.15.1:0/4048432463 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f457fe00700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45801038f0 0x7f4580116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f4584e00700 1 -- 172.21.15.1:0/4048432463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45801038f0 msgr2=0x7f4580116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f4584e00700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45801038f0 0x7f4580116130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f4584e00700 1 -- 172.21.15.1:0/4048432463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4574005040 con 0x7f4580102f50 2024-02-28T17:13:27.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f4584e00700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 0x7f4580115bf0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4574009fd0 tx=0x7f4574009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f457cc00700 1 -- 172.21.15.1:0/4048432463 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f457400e040 con 0x7f4580102f50 2024-02-28T17:13:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f457cc00700 1 -- 172.21.15.1:0/4048432463 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f457400bbf0 con 0x7f4580102f50 2024-02-28T17:13:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f457cc00700 1 -- 172.21.15.1:0/4048432463 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f457401c3f0 con 0x7f4580102f50 2024-02-28T17:13:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.822+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458010fd50 con 0x7f4580102f50 2024-02-28T17:13:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.823+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4580110290 con 0x7f4580102f50 2024-02-28T17:13:27.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.824+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45800fc6e0 con 0x7f4580102f50 2024-02-28T17:13:27.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.825+0000 7f457cc00700 1 -- 172.21.15.1:0/4048432463 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f457400bd60 con 0x7f4580102f50 2024-02-28T17:13:27.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.826+0000 7f457cc00700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f456806d7f0 0x7f456806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:27.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.826+0000 7f457cc00700 1 -- 172.21.15.1:0/4048432463 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f457408e1c0 con 0x7f4580102f50 2024-02-28T17:13:27.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.826+0000 7f457fe00700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f456806d7f0 0x7f456806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:27.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.827+0000 7f457fe00700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f456806d7f0 0x7f456806fca0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4580111490 tx=0x7f456c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:27.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:27.831+0000 7f457cc00700 1 -- 172.21.15.1:0/4048432463 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4574057a70 con 0x7f4580102f50 2024-02-28T17:13:28.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:27 smithi066 ceph-mon[85781]: from='client.36080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:13:28.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:27 smithi066 ceph-mon[85781]: from='client.36084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:13:28.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:27 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1022502825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:13:28.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.073+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45801112e0 con 0x7f456806d7f0 2024-02-28T17:13:28.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.076+0000 7f457cc00700 1 -- 172.21.15.1:0/4048432463 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f45801112e0 con 0x7f456806d7f0 2024-02-28T17:13:28.079 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:13:28.079 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:13:28.080 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:13:28.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.078+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f456806d7f0 msgr2=0x7f456806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:28.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.078+0000 7f4587800700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f456806d7f0 0x7f456806fca0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4580111490 tx=0x7f456c009ee0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.078+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 msgr2=0x7f4580115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:28.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.078+0000 7f4587800700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 0x7f4580115bf0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4574009fd0 tx=0x7f4574009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:28.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.079+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 shutdown_connections 2024-02-28T17:13:28.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.079+0000 7f4587800700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f456806d7f0 0x7f456806fca0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.079+0000 7f4587800700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4580102f50 0x7f4580115bf0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.079+0000 7f4587800700 1 --2- 172.21.15.1:0/4048432463 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f45801038f0 0x7f4580116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.079+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 >> 172.21.15.1:0/4048432463 conn(0x7f45800f8ae0 msgr2=0x7f45800fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:28.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.079+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 shutdown_connections 2024-02-28T17:13:28.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.079+0000 7f4587800700 1 -- 172.21.15.1:0/4048432463 wait complete. 2024-02-28T17:13:28.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.233+0000 7ff616600700 1 -- 172.21.15.1:0/415282108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6101038f0 msgr2=0x7ff6100fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:28.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.233+0000 7ff616600700 1 --2- 172.21.15.1:0/415282108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6101038f0 0x7ff6100fcf60 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc004940 tx=0x7ff5fc009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:28.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.233+0000 7ff616600700 1 -- 172.21.15.1:0/415282108 shutdown_connections 2024-02-28T17:13:28.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.233+0000 7ff616600700 1 --2- 172.21.15.1:0/415282108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6101038f0 0x7ff6100fcf60 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.233+0000 7ff616600700 1 --2- 172.21.15.1:0/415282108 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff610102f50 0x7ff610103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.233+0000 7ff616600700 1 -- 172.21.15.1:0/415282108 >> 172.21.15.1:0/415282108 conn(0x7ff6100f8ae0 msgr2=0x7ff6100faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:28.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.233+0000 7ff616600700 1 -- 172.21.15.1:0/415282108 shutdown_connections 2024-02-28T17:13:28.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.233+0000 7ff616600700 1 -- 172.21.15.1:0/415282108 wait complete. 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.234+0000 7ff616600700 1 Processor -- start 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.234+0000 7ff616600700 1 -- start start 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.235+0000 7ff616600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff610102f50 0x7ff610113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.235+0000 7ff616600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6101038f0 0x7ff610113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.235+0000 7ff616600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff610114590 con 0x7ff610102f50 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.235+0000 7ff616600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61010d930 con 0x7ff6101038f0 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.235+0000 7ff60f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff610102f50 0x7ff610113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.235+0000 7ff60f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff610102f50 0x7ff610113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56248/0 (socket says 172.21.15.1:56248) 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.235+0000 7ff60f400700 1 -- 172.21.15.1:0/492329255 learned_addr learned my addr 172.21.15.1:0/492329255 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:28.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.235+0000 7ff60ea00700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6101038f0 0x7ff610113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:28.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff60f400700 1 -- 172.21.15.1:0/492329255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6101038f0 msgr2=0x7ff610113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:28.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff60f400700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6101038f0 0x7ff610113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff60f400700 1 -- 172.21.15.1:0/492329255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc005040 con 0x7ff610102f50 2024-02-28T17:13:28.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff60f400700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff610102f50 0x7ff610113a00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff60000c8c0 tx=0x7ff6000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:28.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff607400700 1 -- 172.21.15.1:0/492329255 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff60000a980 con 0x7ff610102f50 2024-02-28T17:13:28.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff607400700 1 -- 172.21.15.1:0/492329255 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff60000ce60 con 0x7ff610102f50 2024-02-28T17:13:28.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61010dc10 con 0x7ff610102f50 2024-02-28T17:13:28.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff607400700 1 -- 172.21.15.1:0/492329255 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6000115e0 con 0x7ff610102f50 2024-02-28T17:13:28.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.236+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61010e150 con 0x7ff610102f50 2024-02-28T17:13:28.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.237+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff610105420 con 0x7ff610102f50 2024-02-28T17:13:28.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.239+0000 7ff607400700 1 -- 172.21.15.1:0/492329255 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff600012400 con 0x7ff610102f50 2024-02-28T17:13:28.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.239+0000 7ff607400700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5f406d7f0 0x7ff5f406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:28.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.240+0000 7ff607400700 1 -- 172.21.15.1:0/492329255 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff60008cf70 con 0x7ff610102f50 2024-02-28T17:13:28.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.240+0000 7ff60ea00700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5f406d7f0 0x7ff5f406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:28.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.241+0000 7ff60ea00700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5f406d7f0 0x7ff5f406fca0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff61010eb40 tx=0x7ff5fc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:28.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.244+0000 7ff607400700 1 -- 172.21.15.1:0/492329255 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff60005a350 con 0x7ff610102f50 2024-02-28T17:13:28.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.557+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff61010a030 con 0x7ff610102f50 2024-02-28T17:13:28.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.558+0000 7ff607400700 1 -- 172.21.15.1:0/492329255 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7ff6000126b0 con 0x7ff610102f50 2024-02-28T17:13:28.561 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:13:28.561 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:13:28.561 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:13:28.562 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:13:28.563 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:13:28.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.562+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5f406d7f0 msgr2=0x7ff5f406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:28.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.562+0000 7ff616600700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5f406d7f0 0x7ff5f406fca0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff61010eb40 tx=0x7ff5fc009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:28.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.562+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff610102f50 msgr2=0x7ff610113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:28.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.562+0000 7ff616600700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff610102f50 0x7ff610113a00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff60000c8c0 tx=0x7ff6000044c0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.563+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 shutdown_connections 2024-02-28T17:13:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.563+0000 7ff616600700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff5f406d7f0 0x7ff5f406fca0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.563+0000 7ff616600700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff610102f50 0x7ff610113a00 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.563+0000 7ff616600700 1 --2- 172.21.15.1:0/492329255 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6101038f0 0x7ff610113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.563+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 >> 172.21.15.1:0/492329255 conn(0x7ff6100f8ae0 msgr2=0x7ff6100fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.563+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 shutdown_connections 2024-02-28T17:13:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:28.563+0000 7ff616600700 1 -- 172.21.15.1:0/492329255 wait complete. 2024-02-28T17:13:28.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:28 smithi001 ceph-mon[112940]: pgmap v1174: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:28.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:28 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3758858590' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:13:29.015 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:28 smithi066 ceph-mon[85781]: pgmap v1174: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:29.015 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:28 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3758858590' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:13:29.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:29 smithi001 ceph-mon[112940]: from='client.36096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:13:29.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:29 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/492329255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:13:30.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:29 smithi066 ceph-mon[85781]: from='client.36096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:13:30.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:29 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/492329255' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:13:31.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:30 smithi066 ceph-mon[85781]: pgmap v1175: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:31.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:30 smithi001 ceph-mon[112940]: pgmap v1175: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:33.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:32 smithi066 ceph-mon[85781]: pgmap v1176: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:33.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:32 smithi001 ceph-mon[112940]: pgmap v1176: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:35.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:34 smithi066 ceph-mon[85781]: pgmap v1177: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:35.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:34 smithi001 ceph-mon[112940]: pgmap v1177: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:37.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:36 smithi066 ceph-mon[85781]: pgmap v1178: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:37.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:36 smithi001 ceph-mon[112940]: pgmap v1178: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:39.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:38 smithi066 ceph-mon[85781]: pgmap v1179: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:39.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:38 smithi001 ceph-mon[112940]: pgmap v1179: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:41.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:40 smithi066 ceph-mon[85781]: pgmap v1180: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:41.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:40 smithi001 ceph-mon[112940]: pgmap v1180: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:41.914 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:13:42.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:13:43.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:42 smithi066 ceph-mon[85781]: pgmap v1181: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:43.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:42 smithi001 ceph-mon[112940]: pgmap v1181: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:45.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:44 smithi066 ceph-mon[85781]: pgmap v1182: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:45.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:44 smithi001 ceph-mon[112940]: pgmap v1182: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:47.015 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:46 smithi066 ceph-mon[85781]: pgmap v1183: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:47.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:46 smithi001 ceph-mon[112940]: pgmap v1183: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:49.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:48 smithi066 ceph-mon[85781]: pgmap v1184: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:49.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:48 smithi001 ceph-mon[112940]: pgmap v1184: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:51.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:50 smithi066 ceph-mon[85781]: pgmap v1185: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:51.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:50 smithi001 ceph-mon[112940]: pgmap v1185: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:53.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:52 smithi066 ceph-mon[85781]: pgmap v1186: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:53.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:52 smithi001 ceph-mon[112940]: pgmap v1186: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:55.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:54 smithi066 ceph-mon[85781]: pgmap v1187: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:55.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:54 smithi001 ceph-mon[112940]: pgmap v1187: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:57.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:56 smithi066 ceph-mon[85781]: pgmap v1188: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:57.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:13:57.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:56 smithi001 ceph-mon[112940]: pgmap v1188: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:57.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:13:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.733+0000 7f00ec400700 1 -- 172.21.15.1:0/1140537744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070330 msgr2=0x7f00e4070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.733+0000 7f00ec400700 1 --2- 172.21.15.1:0/1140537744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070330 0x7f00e4070700 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009d10 tx=0x7f00d8004d10 comp rx=0 tx=0).stop 2024-02-28T17:13:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.733+0000 7f00ec400700 1 -- 172.21.15.1:0/1140537744 shutdown_connections 2024-02-28T17:13:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.733+0000 7f00ec400700 1 --2- 172.21.15.1:0/1140537744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00e4070c40 0x7f00e4109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.733+0000 7f00ec400700 1 --2- 172.21.15.1:0/1140537744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070330 0x7f00e4070700 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.733+0000 7f00ec400700 1 -- 172.21.15.1:0/1140537744 >> 172.21.15.1:0/1140537744 conn(0x7f00e40f8ae0 msgr2=0x7f00e40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.734+0000 7f00ec400700 1 -- 172.21.15.1:0/1140537744 shutdown_connections 2024-02-28T17:13:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.734+0000 7f00ec400700 1 -- 172.21.15.1:0/1140537744 wait complete. 2024-02-28T17:13:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.734+0000 7f00ec400700 1 Processor -- start 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.735+0000 7f00ec400700 1 -- start start 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.735+0000 7f00ec400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00e4070330 0x7f00e4115e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.735+0000 7f00ec400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070c40 0x7f00e4116370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.735+0000 7f00ec400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e41169c0 con 0x7f00e4070c40 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.735+0000 7f00ec400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e410fcd0 con 0x7f00e4070330 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.735+0000 7f00e9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070c40 0x7f00e4116370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.735+0000 7f00e9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070c40 0x7f00e4116370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46304/0 (socket says 172.21.15.1:46304) 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.735+0000 7f00e9000700 1 -- 172.21.15.1:0/566721709 learned_addr learned my addr 172.21.15.1:0/566721709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e9a00700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00e4070330 0x7f00e4115e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e9000700 1 -- 172.21.15.1:0/566721709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00e4070330 msgr2=0x7f00e4115e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e9000700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00e4070330 0x7f00e4115e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e9000700 1 -- 172.21.15.1:0/566721709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d8009940 con 0x7f00e4070c40 2024-02-28T17:13:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e9a00700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00e4070330 0x7f00e4115e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:13:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e9000700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070c40 0x7f00e4116370 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f00d0002a50 tx=0x7f00d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e1600700 1 -- 172.21.15.1:0/566721709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00d0007cc0 con 0x7f00e4070c40 2024-02-28T17:13:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e1600700 1 -- 172.21.15.1:0/566721709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00d000f040 con 0x7f00e4070c40 2024-02-28T17:13:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e410ffb0 con 0x7f00e4070c40 2024-02-28T17:13:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.736+0000 7f00e1600700 1 -- 172.21.15.1:0/566721709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00d0009640 con 0x7f00e4070c40 2024-02-28T17:13:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.737+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e41104f0 con 0x7f00e4070c40 2024-02-28T17:13:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.738+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e4106920 con 0x7f00e4070c40 2024-02-28T17:13:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.739+0000 7f00e1600700 1 -- 172.21.15.1:0/566721709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f00d00097a0 con 0x7f00e4070c40 2024-02-28T17:13:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.741+0000 7f00e1600700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f00cc06d7e0 0x7f00cc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.741+0000 7f00e1600700 1 -- 172.21.15.1:0/566721709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f00d008cc60 con 0x7f00e4070c40 2024-02-28T17:13:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.741+0000 7f00e9a00700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f00cc06d7e0 0x7f00cc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.742+0000 7f00e9a00700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f00cc06d7e0 0x7f00cc06fc90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f00d8003400 tx=0x7f00d8004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:58.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.746+0000 7f00e1600700 1 -- 172.21.15.1:0/566721709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f00d0059fc0 con 0x7f00e4070c40 2024-02-28T17:13:59.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.996+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f00e404bf40 con 0x7f00cc06d7e0 2024-02-28T17:13:59.000 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:13:58 smithi001 ceph-mon[112940]: pgmap v1189: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:58.999+0000 7f00e1600700 1 -- 172.21.15.1:0/566721709 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f00e404bf40 con 0x7f00cc06d7e0 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.002+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f00cc06d7e0 msgr2=0x7f00cc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.002+0000 7f00ec400700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f00cc06d7e0 0x7f00cc06fc90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f00d8003400 tx=0x7f00d8004d10 comp rx=0 tx=0).stop 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.002+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070c40 msgr2=0x7f00e4116370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.002+0000 7f00ec400700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070c40 0x7f00e4116370 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f00d0002a50 tx=0x7f00d00046b0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.003+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 shutdown_connections 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.003+0000 7f00ec400700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f00cc06d7e0 0x7f00cc06fc90 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.003+0000 7f00ec400700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00e4070330 0x7f00e4115e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.003+0000 7f00ec400700 1 --2- 172.21.15.1:0/566721709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e4070c40 0x7f00e4116370 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.003+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 >> 172.21.15.1:0/566721709 conn(0x7f00e40f8ae0 msgr2=0x7f00e4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.003+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 shutdown_connections 2024-02-28T17:13:59.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.003+0000 7f00ec400700 1 -- 172.21.15.1:0/566721709 wait complete. 2024-02-28T17:13:59.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:13:58 smithi066 ceph-mon[85781]: pgmap v1189: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:13:59.023 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:13:59.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.166+0000 7fb2c8a00700 1 -- 172.21.15.1:0/538645137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 msgr2=0x7fb2c40ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.166+0000 7fb2c8a00700 1 --2- 172.21.15.1:0/538645137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 0x7fb2c40ff180 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac004940 tx=0x7fb2ac009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:59.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.166+0000 7fb2c8a00700 1 -- 172.21.15.1:0/538645137 shutdown_connections 2024-02-28T17:13:59.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.166+0000 7fb2c8a00700 1 --2- 172.21.15.1:0/538645137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 0x7fb2c40ff180 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.166+0000 7fb2c8a00700 1 --2- 172.21.15.1:0/538645137 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2c4105170 0x7fb2c4105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.166+0000 7fb2c8a00700 1 -- 172.21.15.1:0/538645137 >> 172.21.15.1:0/538645137 conn(0x7fb2c40face0 msgr2=0x7fb2c40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:59.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.166+0000 7fb2c8a00700 1 -- 172.21.15.1:0/538645137 shutdown_connections 2024-02-28T17:13:59.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.166+0000 7fb2c8a00700 1 -- 172.21.15.1:0/538645137 wait complete. 2024-02-28T17:13:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.167+0000 7fb2c8a00700 1 Processor -- start 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.167+0000 7fb2c8a00700 1 -- start start 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.168+0000 7fb2c8a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2c4105170 0x7fb2c4115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.168+0000 7fb2c8a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 0x7fb2c4116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.168+0000 7fb2c8a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c4116890 con 0x7fb2c4105b10 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.168+0000 7fb2c8a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c410fba0 con 0x7fb2c4105170 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.168+0000 7fb2bbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 0x7fb2c4116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.168+0000 7fb2bbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 0x7fb2c4116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46328/0 (socket says 172.21.15.1:46328) 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.168+0000 7fb2bbe00700 1 -- 172.21.15.1:0/3775364056 learned_addr learned my addr 172.21.15.1:0/3775364056 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.168+0000 7fb2c1e00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2c4105170 0x7fb2c4115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.169+0000 7fb2bbe00700 1 -- 172.21.15.1:0/3775364056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2c4105170 msgr2=0x7fb2c4115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.169+0000 7fb2bbe00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2c4105170 0x7fb2c4115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.169+0000 7fb2bbe00700 1 -- 172.21.15.1:0/3775364056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac005040 con 0x7fb2c4105b10 2024-02-28T17:13:59.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.169+0000 7fb2bbe00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 0x7fb2c4116240 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009030 tx=0x7fb2ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:59.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.169+0000 7fb2ba000700 1 -- 172.21.15.1:0/3775364056 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2ac00e040 con 0x7fb2c4105b10 2024-02-28T17:13:59.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.169+0000 7fb2ba000700 1 -- 172.21.15.1:0/3775364056 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2ac00bbf0 con 0x7fb2c4105b10 2024-02-28T17:13:59.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.169+0000 7fb2ba000700 1 -- 172.21.15.1:0/3775364056 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2ac01c630 con 0x7fb2c4105b10 2024-02-28T17:13:59.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.169+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c410fe20 con 0x7fb2c4105b10 2024-02-28T17:13:59.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.170+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c4110310 con 0x7fb2c4105b10 2024-02-28T17:13:59.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.171+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c4107640 con 0x7fb2c4105b10 2024-02-28T17:13:59.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.172+0000 7fb2ba000700 1 -- 172.21.15.1:0/3775364056 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb2ac00bd60 con 0x7fb2c4105b10 2024-02-28T17:13:59.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.173+0000 7fb2ba000700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2a806d7f0 0x7fb2a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:59.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.173+0000 7fb2ba000700 1 -- 172.21.15.1:0/3775364056 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb2ac08e2c0 con 0x7fb2c4105b10 2024-02-28T17:13:59.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.174+0000 7fb2c1e00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2a806d7f0 0x7fb2a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:59.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.174+0000 7fb2c1e00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2a806d7f0 0x7fb2a806fca0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009fd0 tx=0x7fb2b4009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:59.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.178+0000 7fb2ba000700 1 -- 172.21.15.1:0/3775364056 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb2ac05b620 con 0x7fb2c4105b10 2024-02-28T17:13:59.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.424+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2c404bed0 con 0x7fb2a806d7f0 2024-02-28T17:13:59.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.427+0000 7fb2ba000700 1 -- 172.21.15.1:0/3775364056 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb2c404bed0 con 0x7fb2a806d7f0 2024-02-28T17:13:59.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2a806d7f0 msgr2=0x7fb2a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2a806d7f0 0x7fb2a806fca0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009fd0 tx=0x7fb2b4009f20 comp rx=0 tx=0).stop 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 msgr2=0x7fb2c4116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 0x7fb2c4116240 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009030 tx=0x7fb2ac009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 shutdown_connections 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2a806d7f0 0x7fb2a806fca0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2c4105170 0x7fb2c4115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 --2- 172.21.15.1:0/3775364056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c4105b10 0x7fb2c4116240 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 >> 172.21.15.1:0/3775364056 conn(0x7fb2c40face0 msgr2=0x7fb2c40fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 shutdown_connections 2024-02-28T17:13:59.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.430+0000 7fb2c8a00700 1 -- 172.21.15.1:0/3775364056 wait complete. 2024-02-28T17:13:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.591+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1107445129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb4102f50 msgr2=0x7f7bb4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.591+0000 7f7bb8e00700 1 --2- 172.21.15.1:0/1107445129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb4102f50 0x7f7bb4103320 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4004940 tx=0x7f7ba4009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.591+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1107445129 shutdown_connections 2024-02-28T17:13:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.591+0000 7f7bb8e00700 1 --2- 172.21.15.1:0/1107445129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb41038f0 0x7f7bb40fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.591+0000 7f7bb8e00700 1 --2- 172.21.15.1:0/1107445129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb4102f50 0x7f7bb4103320 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.591+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1107445129 >> 172.21.15.1:0/1107445129 conn(0x7f7bb40f8ae0 msgr2=0x7f7bb40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.591+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1107445129 shutdown_connections 2024-02-28T17:13:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.591+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1107445129 wait complete. 2024-02-28T17:13:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.592+0000 7f7bb8e00700 1 Processor -- start 2024-02-28T17:13:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.592+0000 7f7bb8e00700 1 -- start start 2024-02-28T17:13:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.593+0000 7f7bb8e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb4102f50 0x7f7bb4115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.593+0000 7f7bb8e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb41038f0 0x7f7bb41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.593+0000 7f7bb8e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb4116810 con 0x7f7bb41038f0 2024-02-28T17:13:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.593+0000 7f7bb8e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb410fb40 con 0x7f7bb4102f50 2024-02-28T17:13:59.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.593+0000 7f7bb1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb41038f0 0x7f7bb41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:59.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.594+0000 7f7bb1e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb4102f50 0x7f7bb4115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:59.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.594+0000 7f7bb1e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb4102f50 0x7f7bb4115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.66:3300/0 says I am v2:172.21.15.1:42474/0 (socket says 172.21.15.1:42474) 2024-02-28T17:13:59.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.594+0000 7f7bb1e00700 1 -- 172.21.15.1:0/1574832491 learned_addr learned my addr 172.21.15.1:0/1574832491 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:13:59.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.594+0000 7f7bb1400700 1 -- 172.21.15.1:0/1574832491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb4102f50 msgr2=0x7f7bb4115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.594+0000 7f7bb1400700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb4102f50 0x7f7bb4115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.594+0000 7f7bb1400700 1 -- 172.21.15.1:0/1574832491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba4005040 con 0x7f7bb41038f0 2024-02-28T17:13:59.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.594+0000 7f7bb1400700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb41038f0 0x7f7bb41161c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0004fd0 tx=0x7f7ba0004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:59.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.595+0000 7f7baa000700 1 -- 172.21.15.1:0/1574832491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba0007cc0 con 0x7f7bb41038f0 2024-02-28T17:13:59.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.595+0000 7f7baa000700 1 -- 172.21.15.1:0/1574832491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ba000f040 con 0x7f7bb41038f0 2024-02-28T17:13:59.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.595+0000 7f7baa000700 1 -- 172.21.15.1:0/1574832491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba0013860 con 0x7f7bb41038f0 2024-02-28T17:13:59.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.595+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb410fe20 con 0x7f7bb41038f0 2024-02-28T17:13:59.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.595+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb4110360 con 0x7f7bb41038f0 2024-02-28T17:13:59.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.596+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb40fc6e0 con 0x7f7bb41038f0 2024-02-28T17:13:59.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.597+0000 7f7baa000700 1 -- 172.21.15.1:0/1574832491 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7ba001c400 con 0x7f7bb41038f0 2024-02-28T17:13:59.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.598+0000 7f7baa000700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9806d7f0 0x7f7b9806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:13:59.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.598+0000 7f7baa000700 1 -- 172.21.15.1:0/1574832491 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7ba008cae0 con 0x7f7bb41038f0 2024-02-28T17:13:59.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.599+0000 7f7bb1e00700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9806d7f0 0x7f7b9806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:13:59.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.599+0000 7f7bb1e00700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9806d7f0 0x7f7b9806fca0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009fd0 tx=0x7f7ba4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:13:59.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.603+0000 7f7baa000700 1 -- 172.21.15.1:0/1574832491 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7ba0059e40 con 0x7f7bb41038f0 2024-02-28T17:13:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.837+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bb4111260 con 0x7f7b9806d7f0 2024-02-28T17:13:59.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.854+0000 7f7baa000700 1 -- 172.21.15.1:0/1574832491 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f7bb4111260 con 0x7f7b9806d7f0 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (40m) 97s ago 54m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (36m) 97s ago 54m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (36m) 99s ago 53m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (39m) 97s ago 54m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (32m) 97s ago 47m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (32m) 97s ago 47m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (105s) 99s ago 47m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (17m) 99s ago 47m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (42m) 97s ago 56m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:13:59.859 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (41m) 99s ago 53m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (38m) 97s ago 56m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (37m) 99s ago 52m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (41m) 97s ago 54m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (41m) 99s ago 53m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (36m) 97s ago 52m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (35m) 97s ago 51m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (35m) 97s ago 50m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (34m) 99s ago 50m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (33m) 99s ago 49m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (33m) 99s ago 49m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:13:59.860 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (40m) 97s ago 53m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:13:59.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9806d7f0 msgr2=0x7f7b9806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9806d7f0 0x7f7b9806fca0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009fd0 tx=0x7f7ba4009e30 comp rx=0 tx=0).stop 2024-02-28T17:13:59.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb41038f0 msgr2=0x7f7bb41161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:13:59.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb41038f0 0x7f7bb41161c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0004fd0 tx=0x7f7ba0004570 comp rx=0 tx=0).stop 2024-02-28T17:13:59.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 shutdown_connections 2024-02-28T17:13:59.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7b9806d7f0 0x7f7b9806fca0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7bb4102f50 0x7f7bb4115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 --2- 172.21.15.1:0/1574832491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb41038f0 0x7f7bb41161c0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:13:59.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 >> 172.21.15.1:0/1574832491 conn(0x7f7bb40f8ae0 msgr2=0x7f7bb40fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:13:59.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 shutdown_connections 2024-02-28T17:13:59.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:13:59.857+0000 7f7bb8e00700 1 -- 172.21.15.1:0/1574832491 wait complete. 2024-02-28T17:14:00.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.015+0000 7fc6ce600700 1 -- 172.21.15.1:0/3511613978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 msgr2=0x7fc6c8105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.015+0000 7fc6ce600700 1 --2- 172.21.15.1:0/3511613978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 0x7fc6c8105520 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8004940 tx=0x7fc6b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:00.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.015+0000 7fc6ce600700 1 -- 172.21.15.1:0/3511613978 shutdown_connections 2024-02-28T17:14:00.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.015+0000 7fc6ce600700 1 --2- 172.21.15.1:0/3511613978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c8105af0 0x7fc6c80ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.015+0000 7fc6ce600700 1 --2- 172.21.15.1:0/3511613978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 0x7fc6c8105520 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.015+0000 7fc6ce600700 1 -- 172.21.15.1:0/3511613978 >> 172.21.15.1:0/3511613978 conn(0x7fc6c80face0 msgr2=0x7fc6c80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:00.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.016+0000 7fc6ce600700 1 -- 172.21.15.1:0/3511613978 shutdown_connections 2024-02-28T17:14:00.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.016+0000 7fc6ce600700 1 -- 172.21.15.1:0/3511613978 wait complete. 2024-02-28T17:14:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.016+0000 7fc6ce600700 1 Processor -- start 2024-02-28T17:14:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.016+0000 7fc6ce600700 1 -- start start 2024-02-28T17:14:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.017+0000 7fc6ce600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 0x7fc6c8115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.017+0000 7fc6ce600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c8105af0 0x7fc6c8116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.017+0000 7fc6ce600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c8116850 con 0x7fc6c8105150 2024-02-28T17:14:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.017+0000 7fc6ce600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c810fb60 con 0x7fc6c8105af0 2024-02-28T17:14:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.017+0000 7fc6c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 0x7fc6c8115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.017+0000 7fc6c7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 0x7fc6c8115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46356/0 (socket says 172.21.15.1:46356) 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.017+0000 7fc6c7400700 1 -- 172.21.15.1:0/2060754904 learned_addr learned my addr 172.21.15.1:0/2060754904 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.017+0000 7fc6c6a00700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c8105af0 0x7fc6c8116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6c7400700 1 -- 172.21.15.1:0/2060754904 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c8105af0 msgr2=0x7fc6c8116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6c7400700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c8105af0 0x7fc6c8116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6c7400700 1 -- 172.21.15.1:0/2060754904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b8005040 con 0x7fc6c8105150 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6c7400700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 0x7fc6c8115cc0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009fd0 tx=0x7fc6b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6bf400700 1 -- 172.21.15.1:0/2060754904 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6b800e040 con 0x7fc6c8105150 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6bf400700 1 -- 172.21.15.1:0/2060754904 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6b800bbf0 con 0x7fc6c8105150 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c810fde0 con 0x7fc6c8105150 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6bf400700 1 -- 172.21.15.1:0/2060754904 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6b801c3f0 con 0x7fc6c8105150 2024-02-28T17:14:00.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.018+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c8110320 con 0x7fc6c8105150 2024-02-28T17:14:00.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.020+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c8102460 con 0x7fc6c8105150 2024-02-28T17:14:00.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.021+0000 7fc6bf400700 1 -- 172.21.15.1:0/2060754904 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc6b800bd60 con 0x7fc6c8105150 2024-02-28T17:14:00.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.021+0000 7fc6bf400700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6ac06d7f0 0x7fc6ac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.022+0000 7fc6bf400700 1 -- 172.21.15.1:0/2060754904 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc6b808e1e0 con 0x7fc6c8105150 2024-02-28T17:14:00.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.022+0000 7fc6c6a00700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6ac06d7f0 0x7fc6ac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.022+0000 7fc6c6a00700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6ac06d7f0 0x7fc6ac06fca0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8111560 tx=0x7fc6b400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:00.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.026+0000 7fc6bf400700 1 -- 172.21.15.1:0/2060754904 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc6b805b5c0 con 0x7fc6c8105150 2024-02-28T17:14:00.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.339+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc6c81112a0 con 0x7fc6c8105150 2024-02-28T17:14:00.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.340+0000 7fc6bf400700 1 -- 172.21.15.1:0/2060754904 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fc6b805b150 con 0x7fc6c8105150 2024-02-28T17:14:00.344 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:14:00.344 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:14:00.344 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6ac06d7f0 msgr2=0x7fc6ac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6ac06d7f0 0x7fc6ac06fca0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8111560 tx=0x7fc6b400b040 comp rx=0 tx=0).stop 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 msgr2=0x7fc6c8115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 0x7fc6c8115cc0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009fd0 tx=0x7fc6b8009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 shutdown_connections 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc6ac06d7f0 0x7fc6ac06fca0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6c8105150 0x7fc6c8115cc0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 --2- 172.21.15.1:0/2060754904 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc6c8105af0 0x7fc6c8116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 >> 172.21.15.1:0/2060754904 conn(0x7fc6c80face0 msgr2=0x7fc6c80fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:00.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 shutdown_connections 2024-02-28T17:14:00.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.343+0000 7fc6ce600700 1 -- 172.21.15.1:0/2060754904 wait complete. 2024-02-28T17:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.499+0000 7f53ece00700 1 -- 172.21.15.1:0/3282878711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 msgr2=0x7f53e8070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.499+0000 7f53ece00700 1 --2- 172.21.15.1:0/3282878711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 0x7f53e8070700 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009d10 tx=0x7f53d8004d10 comp rx=0 tx=0).stop 2024-02-28T17:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.499+0000 7f53ece00700 1 -- 172.21.15.1:0/3282878711 shutdown_connections 2024-02-28T17:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.499+0000 7f53ece00700 1 --2- 172.21.15.1:0/3282878711 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53e8070c40 0x7f53e8109050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.499+0000 7f53ece00700 1 --2- 172.21.15.1:0/3282878711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 0x7f53e8070700 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.499+0000 7f53ece00700 1 -- 172.21.15.1:0/3282878711 >> 172.21.15.1:0/3282878711 conn(0x7f53e80f88b0 msgr2=0x7f53e80facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.500+0000 7f53ece00700 1 -- 172.21.15.1:0/3282878711 shutdown_connections 2024-02-28T17:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.500+0000 7f53ece00700 1 -- 172.21.15.1:0/3282878711 wait complete. 2024-02-28T17:14:00.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.500+0000 7f53ece00700 1 Processor -- start 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.500+0000 7f53ece00700 1 -- start start 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.501+0000 7f53ece00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 0x7f53e80fccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.501+0000 7f53ece00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53e8070c40 0x7f53e80fd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.501+0000 7f53ece00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e80fd730 con 0x7f53e8070330 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.501+0000 7f53ece00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e80fd870 con 0x7f53e8070c40 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.501+0000 7f53e7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 0x7f53e80fccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.501+0000 7f53e7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 0x7f53e80fccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46374/0 (socket says 172.21.15.1:46374) 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.501+0000 7f53e7400700 1 -- 172.21.15.1:0/230914188 learned_addr learned my addr 172.21.15.1:0/230914188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53e6a00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53e8070c40 0x7f53e80fd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53e7400700 1 -- 172.21.15.1:0/230914188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53e8070c40 msgr2=0x7f53e80fd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53e7400700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53e8070c40 0x7f53e80fd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53e7400700 1 -- 172.21.15.1:0/230914188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d8009940 con 0x7f53e8070330 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53e6a00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53e8070c40 0x7f53e80fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53e7400700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 0x7f53e80fccb0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f53d8008dd0 tx=0x7f53d8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:00.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53df400700 1 -- 172.21.15.1:0/230914188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53d800e040 con 0x7f53e8070330 2024-02-28T17:14:00.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53df400700 1 -- 172.21.15.1:0/230914188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53d80128e0 con 0x7f53e8070330 2024-02-28T17:14:00.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53df400700 1 -- 172.21.15.1:0/230914188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53d801c8f0 con 0x7f53e8070330 2024-02-28T17:14:00.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.502+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e80fdaf0 con 0x7f53e8070330 2024-02-28T17:14:00.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.503+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e811e510 con 0x7f53e8070330 2024-02-28T17:14:00.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.504+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e8106710 con 0x7f53e8070330 2024-02-28T17:14:00.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.505+0000 7f53df400700 1 -- 172.21.15.1:0/230914188 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f53d8012a50 con 0x7f53e8070330 2024-02-28T17:14:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.505+0000 7f53df400700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f53cc06d7f0 0x7f53cc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.505+0000 7f53df400700 1 -- 172.21.15.1:0/230914188 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f53d808eb50 con 0x7f53e8070330 2024-02-28T17:14:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.506+0000 7f53e6a00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f53cc06d7f0 0x7f53cc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.506+0000 7f53e6a00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f53cc06d7f0 0x7f53cc06fca0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009fd0 tx=0x7f53d400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:00.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.510+0000 7f53df400700 1 -- 172.21.15.1:0/230914188 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f53d80583a0 con 0x7f53e8070330 2024-02-28T17:14:00.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.783+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f53e80fe3f0 con 0x7f53e8070330 2024-02-28T17:14:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.784+0000 7f53df400700 1 -- 172.21.15.1:0/230914188 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f53d8062020 con 0x7f53e8070330 2024-02-28T17:14:00.788 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:14:00.789 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:14:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:14:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:14:00.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f53cc06d7f0 msgr2=0x7f53cc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f53cc06d7f0 0x7f53cc06fca0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009fd0 tx=0x7f53d400b040 comp rx=0 tx=0).stop 2024-02-28T17:14:00.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 msgr2=0x7f53e80fccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 0x7f53e80fccb0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f53d8008dd0 tx=0x7f53d8004d10 comp rx=0 tx=0).stop 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 shutdown_connections 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f53cc06d7f0 0x7f53cc06fca0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53e8070330 0x7f53e80fccb0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 --2- 172.21.15.1:0/230914188 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f53e8070c40 0x7f53e80fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.787+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 >> 172.21.15.1:0/230914188 conn(0x7f53e80f88b0 msgr2=0x7f53e8103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.788+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 shutdown_connections 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.788+0000 7f53ece00700 1 -- 172.21.15.1:0/230914188 wait complete. 2024-02-28T17:14:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:14:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.938+0000 7f8fbbe00700 1 -- 172.21.15.1:0/1167550983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 msgr2=0x7f8fb4105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.938+0000 7f8fbbe00700 1 --2- 172.21.15.1:0/1167550983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 0x7f8fb4105540 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8004940 tx=0x7f8fa8009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:00.943 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:00 smithi001 ceph-mon[112940]: from='client.36104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:00.943 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:00 smithi001 ceph-mon[112940]: pgmap v1190: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:00.944 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:00 smithi001 ceph-mon[112940]: from='client.36108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:00.944 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:00 smithi001 ceph-mon[112940]: from='client.36112 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:00.944 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:00 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2060754904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.939+0000 7f8fbbe00700 1 -- 172.21.15.1:0/1167550983 shutdown_connections 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.939+0000 7f8fbbe00700 1 --2- 172.21.15.1:0/1167550983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8fb4105b10 0x7f8fb40ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.939+0000 7f8fbbe00700 1 --2- 172.21.15.1:0/1167550983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 0x7f8fb4105540 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.939+0000 7f8fbbe00700 1 -- 172.21.15.1:0/1167550983 >> 172.21.15.1:0/1167550983 conn(0x7f8fb40face0 msgr2=0x7f8fb40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.939+0000 7f8fbbe00700 1 -- 172.21.15.1:0/1167550983 shutdown_connections 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.939+0000 7f8fbbe00700 1 -- 172.21.15.1:0/1167550983 wait complete. 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.940+0000 7f8fbbe00700 1 Processor -- start 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.940+0000 7f8fbbe00700 1 -- start start 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fbbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 0x7f8fb4115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fbbe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8fb4105b10 0x7f8fb4116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fbbe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb4116830 con 0x7f8fb4105170 2024-02-28T17:14:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fbbe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb410fb40 con 0x7f8fb4105b10 2024-02-28T17:14:00.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fb9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 0x7f8fb4115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fb9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 0x7f8fb4115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46390/0 (socket says 172.21.15.1:46390) 2024-02-28T17:14:00.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fb9400700 1 -- 172.21.15.1:0/2909331238 learned_addr learned my addr 172.21.15.1:0/2909331238 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:00.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fb9400700 1 -- 172.21.15.1:0/2909331238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8fb4105b10 msgr2=0x7f8fb4116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:00.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fb8a00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8fb4105b10 0x7f8fb4116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fb9400700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8fb4105b10 0x7f8fb4116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fb9400700 1 -- 172.21.15.1:0/2909331238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa8005040 con 0x7f8fb4105170 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.941+0000 7f8fb8a00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8fb4105b10 0x7f8fb4116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.942+0000 7f8fb9400700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 0x7f8fb4115c10 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009fd0 tx=0x7f8fa8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.942+0000 7f8fb1600700 1 -- 172.21.15.1:0/2909331238 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fa800e040 con 0x7f8fb4105170 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.942+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb410fdc0 con 0x7f8fb4105170 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.942+0000 7f8fb1600700 1 -- 172.21.15.1:0/2909331238 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fa800bbf0 con 0x7f8fb4105170 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.942+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb4110300 con 0x7f8fb4105170 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.942+0000 7f8fb1600700 1 -- 172.21.15.1:0/2909331238 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fa80255e0 con 0x7f8fb4105170 2024-02-28T17:14:00.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.944+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb40fe900 con 0x7f8fb4105170 2024-02-28T17:14:00.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.944+0000 7f8fb1600700 1 -- 172.21.15.1:0/2909331238 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8fa800bd60 con 0x7f8fb4105170 2024-02-28T17:14:00.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.946+0000 7f8fb1600700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8f9c06d7f0 0x7f8f9c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:00.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.946+0000 7f8fb1600700 1 -- 172.21.15.1:0/2909331238 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8fa808e2b0 con 0x7f8fb4105170 2024-02-28T17:14:00.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.946+0000 7f8fb8a00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8f9c06d7f0 0x7f8f9c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:00.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.947+0000 7f8fb8a00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8f9c06d7f0 0x7f8f9c06fca0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0006fd0 tx=0x7f8fa0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:00.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:00.951+0000 7f8fb1600700 1 -- 172.21.15.1:0/2909331238 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8fa805b610 con 0x7f8fb4105170 2024-02-28T17:14:01.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:00 smithi066 ceph-mon[85781]: from='client.36104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:01.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:00 smithi066 ceph-mon[85781]: pgmap v1190: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:01.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:00 smithi066 ceph-mon[85781]: from='client.36108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:01.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:00 smithi066 ceph-mon[85781]: from='client.36112 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:01.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:00 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2060754904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:14:01.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.183+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fb4111260 con 0x7f8f9c06d7f0 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.185+0000 7f8fb1600700 1 -- 172.21.15.1:0/2909331238 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8fb4111260 con 0x7f8f9c06d7f0 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:14:01.189 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:14:01.190 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:14:01.190 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:14:01.190 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:14:01.190 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:14:01.190 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8f9c06d7f0 msgr2=0x7f8f9c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8f9c06d7f0 0x7f8f9c06fca0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0006fd0 tx=0x7f8fa0008040 comp rx=0 tx=0).stop 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 msgr2=0x7f8fb4115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 0x7f8fb4115c10 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009fd0 tx=0x7f8fa8009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 shutdown_connections 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8f9c06d7f0 0x7f8f9c06fca0 secure :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0006fd0 tx=0x7f8fa0008040 comp rx=0 tx=0).stop 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fb4105170 0x7f8fb4115c10 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 --2- 172.21.15.1:0/2909331238 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8fb4105b10 0x7f8fb4116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.187+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 >> 172.21.15.1:0/2909331238 conn(0x7f8fb40face0 msgr2=0x7f8fb40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:01.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.188+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 shutdown_connections 2024-02-28T17:14:01.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.188+0000 7f8fbbe00700 1 -- 172.21.15.1:0/2909331238 wait complete. 2024-02-28T17:14:01.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.342+0000 7fa2ca200700 1 -- 172.21.15.1:0/2154041992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 msgr2=0x7fa2c4105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:01.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.342+0000 7fa2ca200700 1 --2- 172.21.15.1:0/2154041992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 0x7fa2c4105540 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4004940 tx=0x7fa2b4009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:01.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.342+0000 7fa2ca200700 1 -- 172.21.15.1:0/2154041992 shutdown_connections 2024-02-28T17:14:01.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.342+0000 7fa2ca200700 1 --2- 172.21.15.1:0/2154041992 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2c4105b10 0x7fa2c40ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.342+0000 7fa2ca200700 1 --2- 172.21.15.1:0/2154041992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 0x7fa2c4105540 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.342+0000 7fa2ca200700 1 -- 172.21.15.1:0/2154041992 >> 172.21.15.1:0/2154041992 conn(0x7fa2c40face0 msgr2=0x7fa2c40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:01.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.342+0000 7fa2ca200700 1 -- 172.21.15.1:0/2154041992 shutdown_connections 2024-02-28T17:14:01.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.343+0000 7fa2ca200700 1 -- 172.21.15.1:0/2154041992 wait complete. 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.343+0000 7fa2ca200700 1 Processor -- start 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.343+0000 7fa2ca200700 1 -- start start 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2ca200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 0x7fa2c4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2ca200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2c4105b10 0x7fa2c41161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2ca200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c41167f0 con 0x7fa2c4105170 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2ca200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c410fb40 con 0x7fa2c4105b10 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2c3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 0x7fa2c4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2c3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 0x7fa2c4115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46406/0 (socket says 172.21.15.1:46406) 2024-02-28T17:14:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2c3400700 1 -- 172.21.15.1:0/257644826 learned_addr learned my addr 172.21.15.1:0/257644826 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:01.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2c3400700 1 -- 172.21.15.1:0/257644826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2c4105b10 msgr2=0x7fa2c41161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:01.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2c3400700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2c4105b10 0x7fa2c41161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.344+0000 7fa2c3400700 1 -- 172.21.15.1:0/257644826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b4005040 con 0x7fa2c4105170 2024-02-28T17:14:01.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.345+0000 7fa2c3400700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 0x7fa2c4115c60 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009fd0 tx=0x7fa2b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:01.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.345+0000 7fa2bb400700 1 -- 172.21.15.1:0/257644826 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2b400e040 con 0x7fa2c4105170 2024-02-28T17:14:01.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.345+0000 7fa2bb400700 1 -- 172.21.15.1:0/257644826 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2b400bbf0 con 0x7fa2c4105170 2024-02-28T17:14:01.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.345+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c410fdc0 con 0x7fa2c4105170 2024-02-28T17:14:01.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.345+0000 7fa2bb400700 1 -- 172.21.15.1:0/257644826 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2b401c3f0 con 0x7fa2c4105170 2024-02-28T17:14:01.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.345+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c4110300 con 0x7fa2c4105170 2024-02-28T17:14:01.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.346+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c4102480 con 0x7fa2c4105170 2024-02-28T17:14:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.347+0000 7fa2bb400700 1 -- 172.21.15.1:0/257644826 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa2b400bd60 con 0x7fa2c4105170 2024-02-28T17:14:01.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.348+0000 7fa2bb400700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa2a806d7f0 0x7fa2a806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:01.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.349+0000 7fa2bb400700 1 -- 172.21.15.1:0/257644826 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa2b408e1c0 con 0x7fa2c4105170 2024-02-28T17:14:01.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.349+0000 7fa2c2a00700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa2a806d7f0 0x7fa2a806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:01.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.349+0000 7fa2c2a00700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa2a806d7f0 0x7fa2a806fca0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4111540 tx=0x7fa2b0009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.353+0000 7fa2bb400700 1 -- 172.21.15.1:0/257644826 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa2b4057a70 con 0x7fa2c4105170 2024-02-28T17:14:01.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.652+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa2c404bf00 con 0x7fa2c4105170 2024-02-28T17:14:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.657+0000 7fa2bb400700 1 -- 172.21.15.1:0/257644826 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fa2b405b130 con 0x7fa2c4105170 2024-02-28T17:14:01.661 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:14:01.661 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:14:01.661 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:14:01.661 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:14:01.662 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa2a806d7f0 msgr2=0x7fa2a806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:01.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa2a806d7f0 0x7fa2a806fca0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4111540 tx=0x7fa2b0009ce0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 msgr2=0x7fa2c4115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:01.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 0x7fa2c4115c60 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009fd0 tx=0x7fa2b4009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:01.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 shutdown_connections 2024-02-28T17:14:01.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa2a806d7f0 0x7fa2a806fca0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2c4105170 0x7fa2c4115c60 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 --2- 172.21.15.1:0/257644826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa2c4105b10 0x7fa2c41161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:01.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 >> 172.21.15.1:0/257644826 conn(0x7fa2c40face0 msgr2=0x7fa2c40fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:01.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 shutdown_connections 2024-02-28T17:14:01.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:01.660+0000 7fa2ca200700 1 -- 172.21.15.1:0/257644826 wait complete. 2024-02-28T17:14:02.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:01 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/230914188' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:14:02.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:01 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/230914188' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:14:03.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:02 smithi066 ceph-mon[85781]: from='client.36124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:03.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:02 smithi066 ceph-mon[85781]: pgmap v1191: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:03.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:02 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/257644826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:14:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:02 smithi001 ceph-mon[112940]: from='client.36124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:02 smithi001 ceph-mon[112940]: pgmap v1191: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:02 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/257644826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:14:05.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:04 smithi066 ceph-mon[85781]: pgmap v1192: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:05.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:04 smithi001 ceph-mon[112940]: pgmap v1192: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:07.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:06 smithi066 ceph-mon[85781]: pgmap v1193: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:07.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:06 smithi001 ceph-mon[112940]: pgmap v1193: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:09.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:08 smithi066 ceph-mon[85781]: pgmap v1194: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:09.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:08 smithi001 ceph-mon[112940]: pgmap v1194: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:11.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:10 smithi066 ceph-mon[85781]: pgmap v1195: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:11.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:10 smithi001 ceph-mon[112940]: pgmap v1195: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:12.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:14:12.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:14:13.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:12 smithi066 ceph-mon[85781]: pgmap v1196: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:13.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:12 smithi001 ceph-mon[112940]: pgmap v1196: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:15.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:14 smithi066 ceph-mon[85781]: pgmap v1197: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:15.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:14 smithi001 ceph-mon[112940]: pgmap v1197: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:17.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:16 smithi066 ceph-mon[85781]: pgmap v1198: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:17.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:16 smithi001 ceph-mon[112940]: pgmap v1198: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:19.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:18 smithi066 ceph-mon[85781]: pgmap v1199: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:19.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:18 smithi001 ceph-mon[112940]: pgmap v1199: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:21.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:20 smithi066 ceph-mon[85781]: pgmap v1200: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:21.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:20 smithi001 ceph-mon[112940]: pgmap v1200: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:23.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:22 smithi066 ceph-mon[85781]: pgmap v1201: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:23.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:22 smithi001 ceph-mon[112940]: pgmap v1201: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:25.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:24 smithi066 ceph-mon[85781]: pgmap v1202: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:25.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:24 smithi001 ceph-mon[112940]: pgmap v1202: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:27.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:26 smithi066 ceph-mon[85781]: pgmap v1203: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:27.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:14:27.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:26 smithi001 ceph-mon[112940]: pgmap v1203: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:27.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:14:29.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:28 smithi066 ceph-mon[85781]: pgmap v1204: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:29.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:28 smithi001 ceph-mon[112940]: pgmap v1204: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:31.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:30 smithi001 ceph-mon[112940]: pgmap v1205: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:31.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:30 smithi066 ceph-mon[85781]: pgmap v1205: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:31.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.838+0000 7fae3be00700 1 -- 172.21.15.1:0/4162991080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 msgr2=0x7fae3c109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.838+0000 7fae3be00700 1 --2- 172.21.15.1:0/4162991080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 0x7fae3c109750 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fae2c004940 tx=0x7fae2c009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.838+0000 7fae3be00700 1 -- 172.21.15.1:0/4162991080 shutdown_connections 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.838+0000 7fae3be00700 1 --2- 172.21.15.1:0/4162991080 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c109d20 0x7fae3c1033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.838+0000 7fae3be00700 1 --2- 172.21.15.1:0/4162991080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 0x7fae3c109750 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.838+0000 7fae3be00700 1 -- 172.21.15.1:0/4162991080 >> 172.21.15.1:0/4162991080 conn(0x7fae3c072b50 msgr2=0x7fae3c074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.838+0000 7fae3be00700 1 -- 172.21.15.1:0/4162991080 shutdown_connections 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.838+0000 7fae3be00700 1 -- 172.21.15.1:0/4162991080 wait complete. 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.839+0000 7fae3be00700 1 Processor -- start 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.839+0000 7fae3be00700 1 -- start start 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 0x7fae3c119ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:31.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3be00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c109d20 0x7fae3c11a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c11aa30 con 0x7fae3c109380 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3be00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c113d40 con 0x7fae3c109d20 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 0x7fae3c119ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 0x7fae3c119ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52028/0 (socket says 172.21.15.1:52028) 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3aa00700 1 -- 172.21.15.1:0/19782817 learned_addr learned my addr 172.21.15.1:0/19782817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3a000700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c109d20 0x7fae3c11a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3aa00700 1 -- 172.21.15.1:0/19782817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c109d20 msgr2=0x7fae3c11a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3aa00700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c109d20 0x7fae3c11a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.840+0000 7fae3aa00700 1 -- 172.21.15.1:0/19782817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c005040 con 0x7fae3c109380 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.841+0000 7fae3aa00700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 0x7fae3c119ea0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009fd0 tx=0x7fae2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.841+0000 7fae32a00700 1 -- 172.21.15.1:0/19782817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2c00e040 con 0x7fae3c109380 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.841+0000 7fae32a00700 1 -- 172.21.15.1:0/19782817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae2c00bbf0 con 0x7fae3c109380 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.841+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c113fc0 con 0x7fae3c109380 2024-02-28T17:14:31.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.841+0000 7fae32a00700 1 -- 172.21.15.1:0/19782817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2c01c3f0 con 0x7fae3c109380 2024-02-28T17:14:31.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.841+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c114500 con 0x7fae3c109380 2024-02-28T17:14:31.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.843+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c106690 con 0x7fae3c109380 2024-02-28T17:14:31.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.844+0000 7fae32a00700 1 -- 172.21.15.1:0/19782817 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fae2c00bd60 con 0x7fae3c109380 2024-02-28T17:14:31.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.845+0000 7fae32a00700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:31.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.845+0000 7fae32a00700 1 -- 172.21.15.1:0/19782817 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fae2c08e210 con 0x7fae3c109380 2024-02-28T17:14:31.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.845+0000 7fae3a000700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:31.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.846+0000 7fae3a000700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fae3c115740 tx=0x7fae28009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:31.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:31.851+0000 7fae32a00700 1 -- 172.21.15.1:0/19782817 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae2c05b610 con 0x7fae3c109380 2024-02-28T17:14:32.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.101+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae3c04bed0 con 0x7fae2006d7f0 2024-02-28T17:14:32.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.103+0000 7fae32a00700 1 -- 172.21.15.1:0/19782817 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fae3c04bed0 con 0x7fae2006d7f0 2024-02-28T17:14:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 msgr2=0x7fae2006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fae3c115740 tx=0x7fae28009ee0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 msgr2=0x7fae3c119ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 0x7fae3c119ea0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009fd0 tx=0x7fae2c009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:32.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 shutdown_connections 2024-02-28T17:14:32.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c109380 0x7fae3c119ea0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 --2- 172.21.15.1:0/19782817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c109d20 0x7fae3c11a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.106+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 >> 172.21.15.1:0/19782817 conn(0x7fae3c072b50 msgr2=0x7fae3c075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:32.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.107+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 shutdown_connections 2024-02-28T17:14:32.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.107+0000 7fae3be00700 1 -- 172.21.15.1:0/19782817 wait complete. 2024-02-28T17:14:32.126 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:14:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.265+0000 7fa1df400700 1 -- 172.21.15.1:0/817524581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 msgr2=0x7fa1d80fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.265+0000 7fa1df400700 1 --2- 172.21.15.1:0/817524581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 0x7fa1d80fcf50 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4004940 tx=0x7fa1c4009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.265+0000 7fa1df400700 1 -- 172.21.15.1:0/817524581 shutdown_connections 2024-02-28T17:14:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.265+0000 7fa1df400700 1 --2- 172.21.15.1:0/817524581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 0x7fa1d80fcf50 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.265+0000 7fa1df400700 1 --2- 172.21.15.1:0/817524581 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1d8102f40 0x7fa1d8103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.265+0000 7fa1df400700 1 -- 172.21.15.1:0/817524581 >> 172.21.15.1:0/817524581 conn(0x7fa1d80f8af0 msgr2=0x7fa1d80faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:32.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.265+0000 7fa1df400700 1 -- 172.21.15.1:0/817524581 shutdown_connections 2024-02-28T17:14:32.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.265+0000 7fa1df400700 1 -- 172.21.15.1:0/817524581 wait complete. 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.266+0000 7fa1df400700 1 Processor -- start 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.266+0000 7fa1df400700 1 -- start start 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.266+0000 7fa1df400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1d8102f40 0x7fa1d8115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.266+0000 7fa1df400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 0x7fa1d8116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.266+0000 7fa1df400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d81167b0 con 0x7fa1d81038e0 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.266+0000 7fa1df400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d810fae0 con 0x7fa1d8102f40 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.267+0000 7fa1d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 0x7fa1d8116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.267+0000 7fa1d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 0x7fa1d8116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44594/0 (socket says 172.21.15.1:44594) 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.267+0000 7fa1d7e00700 1 -- 172.21.15.1:0/251697971 learned_addr learned my addr 172.21.15.1:0/251697971 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.267+0000 7fa1dca00700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1d8102f40 0x7fa1d8115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.267+0000 7fa1d7e00700 1 -- 172.21.15.1:0/251697971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1d8102f40 msgr2=0x7fa1d8115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.267+0000 7fa1d7e00700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1d8102f40 0x7fa1d8115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.267+0000 7fa1d7e00700 1 -- 172.21.15.1:0/251697971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c4005040 con 0x7fa1d81038e0 2024-02-28T17:14:32.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.268+0000 7fa1d7e00700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 0x7fa1d8116160 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4000c00 tx=0x7fa1c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:32.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.268+0000 7fa1d4c00700 1 -- 172.21.15.1:0/251697971 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1c4021070 con 0x7fa1d81038e0 2024-02-28T17:14:32.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.268+0000 7fa1d4c00700 1 -- 172.21.15.1:0/251697971 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1c4013d00 con 0x7fa1d81038e0 2024-02-28T17:14:32.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.268+0000 7fa1d4c00700 1 -- 172.21.15.1:0/251697971 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1c400e070 con 0x7fa1d81038e0 2024-02-28T17:14:32.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.268+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d810fd60 con 0x7fa1d81038e0 2024-02-28T17:14:32.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.268+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d8110240 con 0x7fa1d81038e0 2024-02-28T17:14:32.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.269+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d80fc6f0 con 0x7fa1d81038e0 2024-02-28T17:14:32.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.270+0000 7fa1d4c00700 1 -- 172.21.15.1:0/251697971 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa1c401cb60 con 0x7fa1d81038e0 2024-02-28T17:14:32.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.271+0000 7fa1d4c00700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1c006d7f0 0x7fa1c006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:32.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.271+0000 7fa1d4c00700 1 -- 172.21.15.1:0/251697971 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa1c408e360 con 0x7fa1d81038e0 2024-02-28T17:14:32.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.271+0000 7fa1dca00700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1c006d7f0 0x7fa1c006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:32.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.272+0000 7fa1dca00700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1c006d7f0 0x7fa1c006fca0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009f10 tx=0x7fa1cc00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:32.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.276+0000 7fa1d4c00700 1 -- 172.21.15.1:0/251697971 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa1c405b740 con 0x7fa1d81038e0 2024-02-28T17:14:32.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.521+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1d81158f0 con 0x7fa1c006d7f0 2024-02-28T17:14:32.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.524+0000 7fa1d4c00700 1 -- 172.21.15.1:0/251697971 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa1d81158f0 con 0x7fa1c006d7f0 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1c006d7f0 msgr2=0x7fa1c006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1c006d7f0 0x7fa1c006fca0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009f10 tx=0x7fa1cc00b040 comp rx=0 tx=0).stop 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 msgr2=0x7fa1d8116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 0x7fa1d8116160 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4000c00 tx=0x7fa1c4009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 shutdown_connections 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1c006d7f0 0x7fa1c006fca0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1d8102f40 0x7fa1d8115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 --2- 172.21.15.1:0/251697971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1d81038e0 0x7fa1d8116160 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 >> 172.21.15.1:0/251697971 conn(0x7fa1d80f8af0 msgr2=0x7fa1d8070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 shutdown_connections 2024-02-28T17:14:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.527+0000 7fa1df400700 1 -- 172.21.15.1:0/251697971 wait complete. 2024-02-28T17:14:32.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.686+0000 7fee62a00700 1 -- 172.21.15.1:0/3556479610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c070870 msgr2=0x7fee5c070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.686+0000 7fee62a00700 1 --2- 172.21.15.1:0/3556479610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c070870 0x7fee5c070ce0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fee48004970 tx=0x7fee48009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:32.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.687+0000 7fee62a00700 1 -- 172.21.15.1:0/3556479610 shutdown_connections 2024-02-28T17:14:32.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.687+0000 7fee62a00700 1 --2- 172.21.15.1:0/3556479610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c070870 0x7fee5c070ce0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.687+0000 7fee62a00700 1 --2- 172.21.15.1:0/3556479610 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee5c071f60 0x7fee5c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.687+0000 7fee62a00700 1 -- 172.21.15.1:0/3556479610 >> 172.21.15.1:0/3556479610 conn(0x7fee5c0f8af0 msgr2=0x7fee5c0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:32.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.687+0000 7fee62a00700 1 -- 172.21.15.1:0/3556479610 shutdown_connections 2024-02-28T17:14:32.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.687+0000 7fee62a00700 1 -- 172.21.15.1:0/3556479610 wait complete. 2024-02-28T17:14:32.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.688+0000 7fee62a00700 1 Processor -- start 2024-02-28T17:14:32.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.688+0000 7fee62a00700 1 -- start start 2024-02-28T17:14:32.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.689+0000 7fee62a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee5c070870 0x7fee5c11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:32.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.689+0000 7fee62a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c071f60 0x7fee5c11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:32.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.689+0000 7fee62a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c11abd0 con 0x7fee5c071f60 2024-02-28T17:14:32.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.689+0000 7fee62a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c113ee0 con 0x7fee5c070870 2024-02-28T17:14:32.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.689+0000 7fee5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c071f60 0x7fee5c11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:32.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.689+0000 7fee5b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c071f60 0x7fee5c11a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44612/0 (socket says 172.21.15.1:44612) 2024-02-28T17:14:32.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.689+0000 7fee5b400700 1 -- 172.21.15.1:0/1591071584 learned_addr learned my addr 172.21.15.1:0/1591071584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:32.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.689+0000 7fee5be00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee5c070870 0x7fee5c11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:32.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee5b400700 1 -- 172.21.15.1:0/1591071584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee5c070870 msgr2=0x7fee5c11a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee5b400700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee5c070870 0x7fee5c11a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee5b400700 1 -- 172.21.15.1:0/1591071584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee48005040 con 0x7fee5c071f60 2024-02-28T17:14:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee5be00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee5c070870 0x7fee5c11a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T17:14:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee5b400700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c071f60 0x7fee5c11a580 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fee48004f50 tx=0x7fee48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee53e00700 1 -- 172.21.15.1:0/1591071584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee4800e040 con 0x7fee5c071f60 2024-02-28T17:14:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee53e00700 1 -- 172.21.15.1:0/1591071584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee4800bc10 con 0x7fee5c071f60 2024-02-28T17:14:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c114160 con 0x7fee5c071f60 2024-02-28T17:14:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.690+0000 7fee53e00700 1 -- 172.21.15.1:0/1591071584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee4801c630 con 0x7fee5c071f60 2024-02-28T17:14:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.691+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c1146a0 con 0x7fee5c071f60 2024-02-28T17:14:32.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.692+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c0fe410 con 0x7fee5c071f60 2024-02-28T17:14:32.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.693+0000 7fee53e00700 1 -- 172.21.15.1:0/1591071584 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fee4800bd80 con 0x7fee5c071f60 2024-02-28T17:14:32.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.694+0000 7fee53e00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fee4006d7f0 0x7fee4006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:32.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.694+0000 7fee53e00700 1 -- 172.21.15.1:0/1591071584 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fee4808e390 con 0x7fee5c071f60 2024-02-28T17:14:32.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.694+0000 7fee5be00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fee4006d7f0 0x7fee4006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:32.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.695+0000 7fee5be00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fee4006d7f0 0x7fee4006fca0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0041d0 tx=0x7fee4c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.699+0000 7fee53e00700 1 -- 172.21.15.1:0/1591071584 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fee4805b6f0 con 0x7fee5c071f60 2024-02-28T17:14:32.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:32 smithi001 ceph-mon[112940]: pgmap v1206: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:32.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.942+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fee5c04bed0 con 0x7fee4006d7f0 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.953+0000 7fee53e00700 1 -- 172.21.15.1:0/1591071584 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fee5c04bed0 con 0x7fee4006d7f0 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (40m) 2m ago 54m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (37m) 2m ago 55m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (37m) 2m ago 53m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (40m) 2m ago 54m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (33m) 2m ago 47m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (32m) 2m ago 47m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:14:32.957 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (2m) 2m ago 47m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (17m) 2m ago 47m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (43m) 2m ago 56m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (42m) 2m ago 53m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (39m) 2m ago 56m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (38m) 2m ago 53m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (41m) 2m ago 55m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (41m) 2m ago 53m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (36m) 2m ago 52m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (36m) 2m ago 52m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (35m) 2m ago 51m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (35m) 2m ago 50m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (34m) 2m ago 50m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (33m) 2m ago 49m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:14:32.958 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (41m) 2m ago 54m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.956+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fee4006d7f0 msgr2=0x7fee4006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fee4006d7f0 0x7fee4006fca0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0041d0 tx=0x7fee4c009040 comp rx=0 tx=0).stop 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c071f60 msgr2=0x7fee5c11a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c071f60 0x7fee5c11a580 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fee48004f50 tx=0x7fee48009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 shutdown_connections 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fee4006d7f0 0x7fee4006fca0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fee5c070870 0x7fee5c11a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 --2- 172.21.15.1:0/1591071584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee5c071f60 0x7fee5c11a580 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 >> 172.21.15.1:0/1591071584 conn(0x7fee5c0f8af0 msgr2=0x7fee5c0fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:32.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 shutdown_connections 2024-02-28T17:14:32.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:32.957+0000 7fee62a00700 1 -- 172.21.15.1:0/1591071584 wait complete. 2024-02-28T17:14:33.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.115+0000 7f2c1be00700 1 -- 172.21.15.1:0/2870020528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 msgr2=0x7f2c1c0fefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:33.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.115+0000 7f2c1be00700 1 --2- 172.21.15.1:0/2870020528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 0x7f2c1c0fefa0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f2c08004940 tx=0x7f2c08009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:33.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.115+0000 7f2c1be00700 1 -- 172.21.15.1:0/2870020528 shutdown_connections 2024-02-28T17:14:33.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.115+0000 7f2c1be00700 1 --2- 172.21.15.1:0/2870020528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 0x7f2c1c0fefa0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.115+0000 7f2c1be00700 1 --2- 172.21.15.1:0/2870020528 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c1c104f90 0x7f2c1c105360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.115+0000 7f2c1be00700 1 -- 172.21.15.1:0/2870020528 >> 172.21.15.1:0/2870020528 conn(0x7f2c1c0fab40 msgr2=0x7f2c1c0fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:33.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.115+0000 7f2c1be00700 1 -- 172.21.15.1:0/2870020528 shutdown_connections 2024-02-28T17:14:33.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.115+0000 7f2c1be00700 1 -- 172.21.15.1:0/2870020528 wait complete. 2024-02-28T17:14:33.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.116+0000 7f2c1be00700 1 Processor -- start 2024-02-28T17:14:33.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.116+0000 7f2c1be00700 1 -- start start 2024-02-28T17:14:33.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.116+0000 7f2c1be00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c1c104f90 0x7f2c1c115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:33.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.116+0000 7f2c1be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 0x7f2c1c115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:33.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.116+0000 7f2c1be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c1165e0 con 0x7f2c1c105930 2024-02-28T17:14:33.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.116+0000 7f2c1be00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c10f910 con 0x7f2c1c104f90 2024-02-28T17:14:33.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.117+0000 7f2c1a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 0x7f2c1c115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:33.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.117+0000 7f2c1a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 0x7f2c1c115f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44636/0 (socket says 172.21.15.1:44636) 2024-02-28T17:14:33.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.117+0000 7f2c1a000700 1 -- 172.21.15.1:0/3074594321 learned_addr learned my addr 172.21.15.1:0/3074594321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.117+0000 7f2c1aa00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c1c104f90 0x7f2c1c115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.117+0000 7f2c1a000700 1 -- 172.21.15.1:0/3074594321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c1c104f90 msgr2=0x7f2c1c115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.117+0000 7f2c1a000700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c1c104f90 0x7f2c1c115a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.117+0000 7f2c1a000700 1 -- 172.21.15.1:0/3074594321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c08005040 con 0x7f2c1c105930 2024-02-28T17:14:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.118+0000 7f2c1a000700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 0x7f2c1c115f90 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009f80 tx=0x7f2c08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.118+0000 7f2c12a00700 1 -- 172.21.15.1:0/3074594321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c0800e040 con 0x7f2c1c105930 2024-02-28T17:14:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.118+0000 7f2c12a00700 1 -- 172.21.15.1:0/3074594321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c0800bbf0 con 0x7f2c1c105930 2024-02-28T17:14:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.118+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c10fb90 con 0x7f2c1c105930 2024-02-28T17:14:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.118+0000 7f2c12a00700 1 -- 172.21.15.1:0/3074594321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c0801c3f0 con 0x7f2c1c105930 2024-02-28T17:14:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.118+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c110150 con 0x7f2c1c105930 2024-02-28T17:14:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.119+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c107460 con 0x7f2c1c105930 2024-02-28T17:14:33.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.120+0000 7f2c12a00700 1 -- 172.21.15.1:0/3074594321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2c0800bd60 con 0x7f2c1c105930 2024-02-28T17:14:33.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.121+0000 7f2c12a00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c0006d7f0 0x7f2c0006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:33.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.121+0000 7f2c12a00700 1 -- 172.21.15.1:0/3074594321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2c0808e310 con 0x7f2c1c105930 2024-02-28T17:14:33.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.122+0000 7f2c1aa00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c0006d7f0 0x7f2c0006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:33.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.122+0000 7f2c1aa00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c0006d7f0 0x7f2c0006fca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c004990 tx=0x7f2c0c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:33.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.126+0000 7f2c12a00700 1 -- 172.21.15.1:0/3074594321 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c0805b670 con 0x7f2c1c105930 2024-02-28T17:14:33.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:32 smithi066 ceph-mon[85781]: pgmap v1206: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.437+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2c1c05f5c0 con 0x7f2c1c105930 2024-02-28T17:14:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.439+0000 7f2c12a00700 1 -- 172.21.15.1:0/3074594321 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2c0805b200 con 0x7f2c1c105930 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:14:33.442 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:14:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:14:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:14:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:14:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:14:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:14:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:14:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:14:33.443 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:14:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c0006d7f0 msgr2=0x7f2c0006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c0006d7f0 0x7f2c0006fca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c004990 tx=0x7f2c0c009ee0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 msgr2=0x7f2c1c115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 0x7f2c1c115f90 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009f80 tx=0x7f2c08009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 shutdown_connections 2024-02-28T17:14:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c0006d7f0 0x7f2c0006fca0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c1c104f90 0x7f2c1c115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 --2- 172.21.15.1:0/3074594321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c1c105930 0x7f2c1c115f90 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 >> 172.21.15.1:0/3074594321 conn(0x7f2c1c0fab40 msgr2=0x7f2c1c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 shutdown_connections 2024-02-28T17:14:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.442+0000 7f2c1be00700 1 -- 172.21.15.1:0/3074594321 wait complete. 2024-02-28T17:14:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.601+0000 7f04a5000700 1 -- 172.21.15.1:0/2209600272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 msgr2=0x7f04a00fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.601+0000 7f04a5000700 1 --2- 172.21.15.1:0/2209600272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 0x7f04a00fcf60 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0490004940 tx=0x7f0490009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.601+0000 7f04a5000700 1 -- 172.21.15.1:0/2209600272 shutdown_connections 2024-02-28T17:14:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.601+0000 7f04a5000700 1 --2- 172.21.15.1:0/2209600272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 0x7f04a00fcf60 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.601+0000 7f04a5000700 1 --2- 172.21.15.1:0/2209600272 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f04a0102f50 0x7f04a0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.601+0000 7f04a5000700 1 -- 172.21.15.1:0/2209600272 >> 172.21.15.1:0/2209600272 conn(0x7f04a00f8ae0 msgr2=0x7f04a00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.601+0000 7f04a5000700 1 -- 172.21.15.1:0/2209600272 shutdown_connections 2024-02-28T17:14:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.601+0000 7f04a5000700 1 -- 172.21.15.1:0/2209600272 wait complete. 2024-02-28T17:14:33.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.602+0000 7f04a5000700 1 Processor -- start 2024-02-28T17:14:33.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.603+0000 7f04a5000700 1 -- start start 2024-02-28T17:14:33.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.603+0000 7f04a5000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f04a0102f50 0x7f04a0115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:33.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.603+0000 7f04a5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 0x7f04a0116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:33.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.603+0000 7f04a5000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a01167a0 con 0x7f04a01038f0 2024-02-28T17:14:33.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.603+0000 7f04a5000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a010faf0 con 0x7f04a0102f50 2024-02-28T17:14:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.603+0000 7f0495400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 0x7f04a0116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.603+0000 7f0495400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 0x7f04a0116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44650/0 (socket says 172.21.15.1:44650) 2024-02-28T17:14:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.603+0000 7f0495400700 1 -- 172.21.15.1:0/167492798 learned_addr learned my addr 172.21.15.1:0/167492798 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.604+0000 7f049de00700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f04a0102f50 0x7f04a0115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.604+0000 7f0495400700 1 -- 172.21.15.1:0/167492798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f04a0102f50 msgr2=0x7f04a0115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.604+0000 7f0495400700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f04a0102f50 0x7f04a0115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.604+0000 7f0495400700 1 -- 172.21.15.1:0/167492798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0490005040 con 0x7f04a01038f0 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.604+0000 7f0495400700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 0x7f04a0116150 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0490009fd0 tx=0x7f0490009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.604+0000 7f0496a00700 1 -- 172.21.15.1:0/167492798 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0490021070 con 0x7f04a01038f0 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.604+0000 7f0496a00700 1 -- 172.21.15.1:0/167492798 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0490013d00 con 0x7f04a01038f0 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.605+0000 7f0496a00700 1 -- 172.21.15.1:0/167492798 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f049000e070 con 0x7f04a01038f0 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.605+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a010fd70 con 0x7f04a01038f0 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.605+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a0110200 con 0x7f04a01038f0 2024-02-28T17:14:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.606+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a00fc6e0 con 0x7f04a01038f0 2024-02-28T17:14:33.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.607+0000 7f0496a00700 1 -- 172.21.15.1:0/167492798 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f049001c400 con 0x7f04a01038f0 2024-02-28T17:14:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.608+0000 7f0496a00700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f048406d7f0 0x7f048406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.608+0000 7f0496a00700 1 -- 172.21.15.1:0/167492798 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f049008e4c0 con 0x7f04a01038f0 2024-02-28T17:14:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.608+0000 7f049de00700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f048406d7f0 0x7f048406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.609+0000 7f049de00700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f048406d7f0 0x7f048406fca0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0488009fd0 tx=0x7f048800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:33.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.613+0000 7f0496a00700 1 -- 172.21.15.1:0/167492798 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0490057cf0 con 0x7f04a01038f0 2024-02-28T17:14:33.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.884+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f04a01111f0 con 0x7f04a01038f0 2024-02-28T17:14:33.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.885+0000 7f0496a00700 1 -- 172.21.15.1:0/167492798 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f049005b3b0 con 0x7f04a01038f0 2024-02-28T17:14:33.888 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:14:33.888 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:14:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:14:33.890 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:14:33.891 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f048406d7f0 msgr2=0x7f048406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f048406d7f0 0x7f048406fca0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0488009fd0 tx=0x7f048800b040 comp rx=0 tx=0).stop 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 msgr2=0x7f04a0116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 0x7f04a0116150 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0490009fd0 tx=0x7f0490009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 shutdown_connections 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f048406d7f0 0x7f048406fca0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f04a0102f50 0x7f04a0115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 --2- 172.21.15.1:0/167492798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04a01038f0 0x7f04a0116150 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 >> 172.21.15.1:0/167492798 conn(0x7f04a00f8ae0 msgr2=0x7f04a00fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 shutdown_connections 2024-02-28T17:14:33.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:33.888+0000 7f04a5000700 1 -- 172.21.15.1:0/167492798 wait complete. 2024-02-28T17:14:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:14:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:33 smithi001 ceph-mon[112940]: from='client.36132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:33 smithi001 ceph-mon[112940]: from='client.36136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:33 smithi001 ceph-mon[112940]: from='client.36140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:33 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3074594321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:14:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.047+0000 7f7f6e200700 1 -- 172.21.15.1:0/130232152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fde30 msgr2=0x7f7f680fe200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.047+0000 7f7f6e200700 1 --2- 172.21.15.1:0/130232152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fde30 0x7f7f680fe200 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c004940 tx=0x7f7f5c009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.048+0000 7f7f6e200700 1 -- 172.21.15.1:0/130232152 shutdown_connections 2024-02-28T17:14:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.048+0000 7f7f6e200700 1 --2- 172.21.15.1:0/130232152 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f680fe7d0 0x7f7f68106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.048+0000 7f7f6e200700 1 --2- 172.21.15.1:0/130232152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fde30 0x7f7f680fe200 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.048+0000 7f7f6e200700 1 -- 172.21.15.1:0/130232152 >> 172.21.15.1:0/130232152 conn(0x7f7f680f78a0 msgr2=0x7f7f680f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.048+0000 7f7f6e200700 1 -- 172.21.15.1:0/130232152 shutdown_connections 2024-02-28T17:14:34.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.048+0000 7f7f6e200700 1 -- 172.21.15.1:0/130232152 wait complete. 2024-02-28T17:14:34.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.049+0000 7f7f6e200700 1 Processor -- start 2024-02-28T17:14:34.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.049+0000 7f7f6e200700 1 -- start start 2024-02-28T17:14:34.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.049+0000 7f7f6e200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f680fde30 0x7f7f68113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.049+0000 7f7f6e200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fe7d0 0x7f7f68113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.049+0000 7f7f6e200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f68114400 con 0x7f7f680fe7d0 2024-02-28T17:14:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.049+0000 7f7f6e200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6810d730 con 0x7f7f680fde30 2024-02-28T17:14:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.050+0000 7f7f67e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fe7d0 0x7f7f68113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.050+0000 7f7f67e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fe7d0 0x7f7f68113db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44674/0 (socket says 172.21.15.1:44674) 2024-02-28T17:14:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.050+0000 7f7f67e00700 1 -- 172.21.15.1:0/2897478043 learned_addr learned my addr 172.21.15.1:0/2897478043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.050+0000 7f7f6ce00700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f680fde30 0x7f7f68113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.050+0000 7f7f67e00700 1 -- 172.21.15.1:0/2897478043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f680fde30 msgr2=0x7f7f68113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.050+0000 7f7f67e00700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f680fde30 0x7f7f68113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.050+0000 7f7f67e00700 1 -- 172.21.15.1:0/2897478043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c005040 con 0x7f7f680fe7d0 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.050+0000 7f7f6ce00700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f680fde30 0x7f7f68113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.051+0000 7f7f67e00700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fe7d0 0x7f7f68113db0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c830 tx=0x7f7f54004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.051+0000 7f7f64c00700 1 -- 172.21.15.1:0/2897478043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5400a760 con 0x7f7f680fe7d0 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.051+0000 7f7f64c00700 1 -- 172.21.15.1:0/2897478043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f5400ad90 con 0x7f7f680fe7d0 2024-02-28T17:14:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.051+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6810da10 con 0x7f7f680fe7d0 2024-02-28T17:14:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.051+0000 7f7f64c00700 1 -- 172.21.15.1:0/2897478043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f540113a0 con 0x7f7f680fe7d0 2024-02-28T17:14:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.051+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6810df50 con 0x7f7f680fe7d0 2024-02-28T17:14:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.052+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f68104420 con 0x7f7f680fe7d0 2024-02-28T17:14:34.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.053+0000 7f7f64c00700 1 -- 172.21.15.1:0/2897478043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7f54012400 con 0x7f7f680fe7d0 2024-02-28T17:14:34.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.054+0000 7f7f64c00700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f5006d7f0 0x7f7f5006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:34.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.055+0000 7f7f64c00700 1 -- 172.21.15.1:0/2897478043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7f5408cf00 con 0x7f7f680fe7d0 2024-02-28T17:14:34.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.055+0000 7f7f6ce00700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f5006d7f0 0x7f7f5006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:34.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.055+0000 7f7f6ce00700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f5006d7f0 0x7f7f5006fca0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00b3b0 tx=0x7f7f5c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:34.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.059+0000 7f7f64c00700 1 -- 172.21.15.1:0/2897478043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7f5405a2e0 con 0x7f7f680fe7d0 2024-02-28T17:14:34.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:33 smithi066 ceph-mon[85781]: from='client.36132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:34.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:33 smithi066 ceph-mon[85781]: from='client.36136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:34.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:33 smithi066 ceph-mon[85781]: from='client.36140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:34.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:33 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3074594321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:14:34.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.302+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f68109f90 con 0x7f7f5006d7f0 2024-02-28T17:14:34.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.305+0000 7f7f64c00700 1 -- 172.21.15.1:0/2897478043 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7f68109f90 con 0x7f7f5006d7f0 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:14:34.309 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:14:34.310 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:14:34.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.308+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f5006d7f0 msgr2=0x7f7f5006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:34.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.308+0000 7f7f6e200700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f5006d7f0 0x7f7f5006fca0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00b3b0 tx=0x7f7f5c009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:34.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fe7d0 msgr2=0x7f7f68113db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:34.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fe7d0 0x7f7f68113db0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c830 tx=0x7f7f54004470 comp rx=0 tx=0).stop 2024-02-28T17:14:34.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 shutdown_connections 2024-02-28T17:14:34.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7f5006d7f0 0x7f7f5006fca0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7f680fde30 0x7f7f68113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 --2- 172.21.15.1:0/2897478043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f680fe7d0 0x7f7f68113db0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 >> 172.21.15.1:0/2897478043 conn(0x7f7f680f78a0 msgr2=0x7f7f681015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:34.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 shutdown_connections 2024-02-28T17:14:34.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.309+0000 7f7f6e200700 1 -- 172.21.15.1:0/2897478043 wait complete. 2024-02-28T17:14:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.463+0000 7f0b1e800700 1 -- 172.21.15.1:0/200682750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070330 msgr2=0x7f0b18070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.463+0000 7f0b1e800700 1 --2- 172.21.15.1:0/200682750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070330 0x7f0b18070700 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c004970 tx=0x7f0b0c009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.463+0000 7f0b1e800700 1 -- 172.21.15.1:0/200682750 shutdown_connections 2024-02-28T17:14:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.463+0000 7f0b1e800700 1 --2- 172.21.15.1:0/200682750 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18070c40 0x7f0b18109050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.463+0000 7f0b1e800700 1 --2- 172.21.15.1:0/200682750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070330 0x7f0b18070700 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.463+0000 7f0b1e800700 1 -- 172.21.15.1:0/200682750 >> 172.21.15.1:0/200682750 conn(0x7f0b180f88b0 msgr2=0x7f0b180facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.463+0000 7f0b1e800700 1 -- 172.21.15.1:0/200682750 shutdown_connections 2024-02-28T17:14:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.463+0000 7f0b1e800700 1 -- 172.21.15.1:0/200682750 wait complete. 2024-02-28T17:14:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.464+0000 7f0b1e800700 1 Processor -- start 2024-02-28T17:14:34.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.464+0000 7f0b1e800700 1 -- start start 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.464+0000 7f0b1e800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18070330 0x7f0b181159b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.464+0000 7f0b1e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070c40 0x7f0b18115ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.464+0000 7f0b1e800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b18116540 con 0x7f0b18070c40 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.464+0000 7f0b1e800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1810f870 con 0x7f0b18070330 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.465+0000 7f0b1ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070c40 0x7f0b18115ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.465+0000 7f0b1ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070c40 0x7f0b18115ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44688/0 (socket says 172.21.15.1:44688) 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.465+0000 7f0b1ca00700 1 -- 172.21.15.1:0/3138600311 learned_addr learned my addr 172.21.15.1:0/3138600311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.465+0000 7f0b1d400700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18070330 0x7f0b181159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.465+0000 7f0b1ca00700 1 -- 172.21.15.1:0/3138600311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18070330 msgr2=0x7f0b181159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.465+0000 7f0b1ca00700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18070330 0x7f0b181159b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.465+0000 7f0b1ca00700 1 -- 172.21.15.1:0/3138600311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c005040 con 0x7f0b18070c40 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.465+0000 7f0b1ca00700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070c40 0x7f0b18115ef0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400c8c0 tx=0x7f0b040044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:34.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.466+0000 7f0b15600700 1 -- 172.21.15.1:0/3138600311 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b0400a980 con 0x7f0b18070c40 2024-02-28T17:14:34.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.466+0000 7f0b15600700 1 -- 172.21.15.1:0/3138600311 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b0400ce60 con 0x7f0b18070c40 2024-02-28T17:14:34.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.466+0000 7f0b15600700 1 -- 172.21.15.1:0/3138600311 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b040115e0 con 0x7f0b18070c40 2024-02-28T17:14:34.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.466+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1810fb50 con 0x7f0b18070c40 2024-02-28T17:14:34.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.466+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b18110090 con 0x7f0b18070c40 2024-02-28T17:14:34.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.467+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b18106710 con 0x7f0b18070c40 2024-02-28T17:14:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.468+0000 7f0b15600700 1 -- 172.21.15.1:0/3138600311 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0b04012400 con 0x7f0b18070c40 2024-02-28T17:14:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.469+0000 7f0b15600700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7e0 0x7f0b0006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:14:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.469+0000 7f0b15600700 1 -- 172.21.15.1:0/3138600311 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0b0408cf70 con 0x7f0b18070c40 2024-02-28T17:14:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.470+0000 7f0b1d400700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7e0 0x7f0b0006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:14:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.470+0000 7f0b1d400700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7e0 0x7f0b0006fc90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c000c00 tx=0x7f0b0c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:14:34.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.474+0000 7f0b15600700 1 -- 172.21.15.1:0/3138600311 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0b0405a350 con 0x7f0b18070c40 2024-02-28T17:14:34.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.784+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0b1810c030 con 0x7f0b18070c40 2024-02-28T17:14:34.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.791+0000 7f0b15600700 1 -- 172.21.15.1:0/3138600311 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f0b040126b0 con 0x7f0b18070c40 2024-02-28T17:14:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:14:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:14:34.795 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:14:34.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:14:34.796 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:14:34.797 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:14:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:14:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:14:34.797 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:14:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:14:34.797 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:14:34.797 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:14:34.797 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.794+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7e0 msgr2=0x7f0b0006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.794+0000 7f0b1e800700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7e0 0x7f0b0006fc90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c000c00 tx=0x7f0b0c009e30 comp rx=0 tx=0).stop 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.794+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070c40 msgr2=0x7f0b18115ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.794+0000 7f0b1e800700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070c40 0x7f0b18115ef0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400c8c0 tx=0x7f0b040044c0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.795+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 shutdown_connections 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.795+0000 7f0b1e800700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7e0 0x7f0b0006fc90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.795+0000 7f0b1e800700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18070330 0x7f0b181159b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.795+0000 7f0b1e800700 1 --2- 172.21.15.1:0/3138600311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18070c40 0x7f0b18115ef0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.795+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 >> 172.21.15.1:0/3138600311 conn(0x7f0b180f88b0 msgr2=0x7f0b18103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.795+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 shutdown_connections 2024-02-28T17:14:34.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:14:34.795+0000 7f0b1e800700 1 -- 172.21.15.1:0/3138600311 wait complete. 2024-02-28T17:14:35.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:34 smithi066 ceph-mon[85781]: pgmap v1207: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:35.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:34 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/167492798' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:14:35.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:34 smithi001 ceph-mon[112940]: pgmap v1207: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:35.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:34 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/167492798' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:14:36.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:35 smithi066 ceph-mon[85781]: from='client.36152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:36.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:35 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3138600311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:14:36.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:35 smithi001 ceph-mon[112940]: from='client.36152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:14:36.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:35 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3138600311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:14:37.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:36 smithi066 ceph-mon[85781]: pgmap v1208: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:37.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:36 smithi001 ceph-mon[112940]: pgmap v1208: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:39.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:38 smithi066 ceph-mon[85781]: pgmap v1209: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:39.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:38 smithi001 ceph-mon[112940]: pgmap v1209: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:41.065 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:40 smithi066 ceph-mon[85781]: pgmap v1210: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:41.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:40 smithi001 ceph-mon[112940]: pgmap v1210: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:42.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:14:42.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:14:43.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:42 smithi066 ceph-mon[85781]: pgmap v1211: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:43.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:42 smithi001 ceph-mon[112940]: pgmap v1211: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:45.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:44 smithi066 ceph-mon[85781]: pgmap v1212: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:45.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:44 smithi001 ceph-mon[112940]: pgmap v1212: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:47.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:46 smithi066 ceph-mon[85781]: pgmap v1213: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:47.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:46 smithi001 ceph-mon[112940]: pgmap v1213: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:49.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:48 smithi066 ceph-mon[85781]: pgmap v1214: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:49.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:48 smithi001 ceph-mon[112940]: pgmap v1214: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:51.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:50 smithi066 ceph-mon[85781]: pgmap v1215: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:51.288 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:50 smithi001 ceph-mon[112940]: pgmap v1215: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:53.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:52 smithi066 ceph-mon[85781]: pgmap v1216: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:53.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:52 smithi001 ceph-mon[112940]: pgmap v1216: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:54 smithi066 ceph-mon[85781]: pgmap v1217: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:54 smithi001 ceph-mon[112940]: pgmap v1217: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:56.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:14:56.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:14:57.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:56 smithi066 ceph-mon[85781]: pgmap v1218: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:57.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:56 smithi001 ceph-mon[112940]: pgmap v1218: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:59.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:14:58 smithi066 ceph-mon[85781]: pgmap v1219: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:14:59.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:14:58 smithi001 ceph-mon[112940]: pgmap v1219: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:01.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:00 smithi066 ceph-mon[85781]: pgmap v1220: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:01.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:00 smithi001 ceph-mon[112940]: pgmap v1220: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:02 smithi066 ceph-mon[85781]: pgmap v1221: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:03.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:02 smithi001 ceph-mon[112940]: pgmap v1221: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.966+0000 7f4e48600700 1 -- 172.21.15.1:0/2775313444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 msgr2=0x7f4e400ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.966+0000 7f4e48600700 1 --2- 172.21.15.1:0/2775313444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 0x7f4e400ff180 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c004940 tx=0x7f4e2c009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.967+0000 7f4e48600700 1 -- 172.21.15.1:0/2775313444 shutdown_connections 2024-02-28T17:15:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.967+0000 7f4e48600700 1 --2- 172.21.15.1:0/2775313444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 0x7f4e400ff180 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.967+0000 7f4e48600700 1 --2- 172.21.15.1:0/2775313444 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e40105170 0x7f4e40105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.967+0000 7f4e48600700 1 -- 172.21.15.1:0/2775313444 >> 172.21.15.1:0/2775313444 conn(0x7f4e400face0 msgr2=0x7f4e400fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.967+0000 7f4e48600700 1 -- 172.21.15.1:0/2775313444 shutdown_connections 2024-02-28T17:15:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.967+0000 7f4e48600700 1 -- 172.21.15.1:0/2775313444 wait complete. 2024-02-28T17:15:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.968+0000 7f4e48600700 1 Processor -- start 2024-02-28T17:15:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.968+0000 7f4e48600700 1 -- start start 2024-02-28T17:15:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.968+0000 7f4e48600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e40105170 0x7f4e40115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.968+0000 7f4e48600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 0x7f4e40116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.968+0000 7f4e48600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e40116a00 con 0x7f4e40105b10 2024-02-28T17:15:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.968+0000 7f4e48600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4010fc80 con 0x7f4e40105170 2024-02-28T17:15:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.969+0000 7f4e45200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 0x7f4e40116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.969+0000 7f4e45200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 0x7f4e40116320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34846/0 (socket says 172.21.15.1:34846) 2024-02-28T17:15:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.969+0000 7f4e45200700 1 -- 172.21.15.1:0/3820397769 learned_addr learned my addr 172.21.15.1:0/3820397769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.969+0000 7f4e45200700 1 -- 172.21.15.1:0/3820397769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e40105170 msgr2=0x7f4e40115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.969+0000 7f4e45200700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e40105170 0x7f4e40115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.969+0000 7f4e45200700 1 -- 172.21.15.1:0/3820397769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c005040 con 0x7f4e40105b10 2024-02-28T17:15:04.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.969+0000 7f4e45200700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 0x7f4e40116320 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009100 tx=0x7f4e2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.970+0000 7f4e3d600700 1 -- 172.21.15.1:0/3820397769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e2c00e040 con 0x7f4e40105b10 2024-02-28T17:15:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.970+0000 7f4e3d600700 1 -- 172.21.15.1:0/3820397769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e2c004290 con 0x7f4e40105b10 2024-02-28T17:15:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.970+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4010fea0 con 0x7f4e40105b10 2024-02-28T17:15:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.970+0000 7f4e3d600700 1 -- 172.21.15.1:0/3820397769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e2c01c3f0 con 0x7f4e40105b10 2024-02-28T17:15:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.970+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e401103e0 con 0x7f4e40105b10 2024-02-28T17:15:04.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.971+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e400fe900 con 0x7f4e40105b10 2024-02-28T17:15:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.972+0000 7f4e3d600700 1 -- 172.21.15.1:0/3820397769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4e2c00bbf0 con 0x7f4e40105b10 2024-02-28T17:15:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.973+0000 7f4e3d600700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e2806d7f0 0x7f4e2806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.973+0000 7f4e3d600700 1 -- 172.21.15.1:0/3820397769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4e2c08e2a0 con 0x7f4e40105b10 2024-02-28T17:15:04.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.974+0000 7f4e45c00700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e2806d7f0 0x7f4e2806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:04.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.974+0000 7f4e45c00700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e2806d7f0 0x7f4e2806fca0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4e34006fd0 tx=0x7f4e34006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:04.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:04.978+0000 7f4e3d600700 1 -- 172.21.15.1:0/3820397769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4e2c05b600 con 0x7f4e40105b10 2024-02-28T17:15:05.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.227+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e40111430 con 0x7f4e2806d7f0 2024-02-28T17:15:05.232 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:04 smithi001 ceph-mon[112940]: pgmap v1222: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.230+0000 7f4e3d600700 1 -- 172.21.15.1:0/3820397769 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4e40111430 con 0x7f4e2806d7f0 2024-02-28T17:15:05.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e2806d7f0 msgr2=0x7f4e2806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:05.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e2806d7f0 0x7f4e2806fca0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4e34006fd0 tx=0x7f4e34006db0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 msgr2=0x7f4e40116320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 0x7f4e40116320 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009100 tx=0x7f4e2c009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 shutdown_connections 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4e2806d7f0 0x7f4e2806fca0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4e40105170 0x7f4e40115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 --2- 172.21.15.1:0/3820397769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e40105b10 0x7f4e40116320 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 >> 172.21.15.1:0/3820397769 conn(0x7f4e400face0 msgr2=0x7f4e400fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.233+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 shutdown_connections 2024-02-28T17:15:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.234+0000 7f4e48600700 1 -- 172.21.15.1:0/3820397769 wait complete. 2024-02-28T17:15:05.254 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:15:05.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:04 smithi066 ceph-mon[85781]: pgmap v1222: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:05.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.396+0000 7fabe2c00700 1 -- 172.21.15.1:0/2665916866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 msgr2=0x7fabdc070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:05.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.396+0000 7fabe2c00700 1 --2- 172.21.15.1:0/2665916866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 0x7fabdc070ce0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fabc8004970 tx=0x7fabc8009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:05.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.397+0000 7fabe2c00700 1 -- 172.21.15.1:0/2665916866 shutdown_connections 2024-02-28T17:15:05.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.397+0000 7fabe2c00700 1 --2- 172.21.15.1:0/2665916866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 0x7fabdc070ce0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.397+0000 7fabe2c00700 1 --2- 172.21.15.1:0/2665916866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabdc071f60 0x7fabdc070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.397+0000 7fabe2c00700 1 -- 172.21.15.1:0/2665916866 >> 172.21.15.1:0/2665916866 conn(0x7fabdc0f8ae0 msgr2=0x7fabdc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:05.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.397+0000 7fabe2c00700 1 -- 172.21.15.1:0/2665916866 shutdown_connections 2024-02-28T17:15:05.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.397+0000 7fabe2c00700 1 -- 172.21.15.1:0/2665916866 wait complete. 2024-02-28T17:15:05.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.398+0000 7fabe2c00700 1 Processor -- start 2024-02-28T17:15:05.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.398+0000 7fabe2c00700 1 -- start start 2024-02-28T17:15:05.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.398+0000 7fabe2c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 0x7fabdc11a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:05.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.398+0000 7fabe2c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabdc071f60 0x7fabdc11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:05.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.398+0000 7fabe2c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc11ac00 con 0x7fabdc070870 2024-02-28T17:15:05.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.398+0000 7fabe2c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc113f10 con 0x7fabdc071f60 2024-02-28T17:15:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.399+0000 7fabdbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 0x7fabdc11a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.399+0000 7fabdbe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 0x7fabdc11a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34864/0 (socket says 172.21.15.1:34864) 2024-02-28T17:15:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.399+0000 7fabdbe00700 1 -- 172.21.15.1:0/300556361 learned_addr learned my addr 172.21.15.1:0/300556361 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.399+0000 7fabdb400700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabdc071f60 0x7fabdc11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.399+0000 7fabdbe00700 1 -- 172.21.15.1:0/300556361 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabdc071f60 msgr2=0x7fabdc11a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.399+0000 7fabdbe00700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabdc071f60 0x7fabdc11a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.399+0000 7fabdbe00700 1 -- 172.21.15.1:0/300556361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc8005040 con 0x7fabdc070870 2024-02-28T17:15:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.399+0000 7fabdbe00700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 0x7fabdc11a070 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00eb40 tx=0x7fabcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:05.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.400+0000 7fabd3e00700 1 -- 172.21.15.1:0/300556361 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabcc007cc0 con 0x7fabdc070870 2024-02-28T17:15:05.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.400+0000 7fabd3e00700 1 -- 172.21.15.1:0/300556361 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabcc00f040 con 0x7fabdc070870 2024-02-28T17:15:05.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.400+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc114190 con 0x7fabdc070870 2024-02-28T17:15:05.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.400+0000 7fabd3e00700 1 -- 172.21.15.1:0/300556361 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabcc009620 con 0x7fabdc070870 2024-02-28T17:15:05.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.400+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc1146d0 con 0x7fabdc070870 2024-02-28T17:15:05.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.401+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc0fe400 con 0x7fabdc070870 2024-02-28T17:15:05.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.402+0000 7fabd3e00700 1 -- 172.21.15.1:0/300556361 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fabcc009780 con 0x7fabdc070870 2024-02-28T17:15:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.403+0000 7fabd3e00700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabc006d7f0 0x7fabc006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.403+0000 7fabd3e00700 1 -- 172.21.15.1:0/300556361 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fabcc08cb40 con 0x7fabdc070870 2024-02-28T17:15:05.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.404+0000 7fabdb400700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabc006d7f0 0x7fabc006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.404+0000 7fabdb400700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabc006d7f0 0x7fabc006fca0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fabc800b370 tx=0x7fabc8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.408+0000 7fabd3e00700 1 -- 172.21.15.1:0/300556361 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fabcc059f50 con 0x7fabdc070870 2024-02-28T17:15:05.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.656+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fabdc04bf40 con 0x7fabc006d7f0 2024-02-28T17:15:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.658+0000 7fabd3e00700 1 -- 172.21.15.1:0/300556361 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fabdc04bf40 con 0x7fabc006d7f0 2024-02-28T17:15:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabc006d7f0 msgr2=0x7fabc006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabc006d7f0 0x7fabc006fca0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fabc800b370 tx=0x7fabc8009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 msgr2=0x7fabdc11a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 0x7fabdc11a070 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00eb40 tx=0x7fabcc00b040 comp rx=0 tx=0).stop 2024-02-28T17:15:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 shutdown_connections 2024-02-28T17:15:05.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fabc006d7f0 0x7fabc006fca0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabdc070870 0x7fabdc11a070 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 --2- 172.21.15.1:0/300556361 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fabdc071f60 0x7fabdc11a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 >> 172.21.15.1:0/300556361 conn(0x7fabdc0f8ae0 msgr2=0x7fabdc0fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:05.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 shutdown_connections 2024-02-28T17:15:05.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.661+0000 7fabe2c00700 1 -- 172.21.15.1:0/300556361 wait complete. 2024-02-28T17:15:05.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.816+0000 7f3421a00700 1 -- 172.21.15.1:0/2739705805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c0724c0 msgr2=0x7f341c072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:05.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.816+0000 7f3421a00700 1 --2- 172.21.15.1:0/2739705805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c0724c0 0x7f341c072890 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f340c004940 tx=0x7f340c009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:05.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.816+0000 7f3421a00700 1 -- 172.21.15.1:0/2739705805 shutdown_connections 2024-02-28T17:15:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.816+0000 7f3421a00700 1 --2- 172.21.15.1:0/2739705805 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f341c072dd0 0x7f341c10d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.816+0000 7f3421a00700 1 --2- 172.21.15.1:0/2739705805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c0724c0 0x7f341c072890 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.816+0000 7f3421a00700 1 -- 172.21.15.1:0/2739705805 >> 172.21.15.1:0/2739705805 conn(0x7f341c0fad30 msgr2=0x7f341c0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.816+0000 7f3421a00700 1 -- 172.21.15.1:0/2739705805 shutdown_connections 2024-02-28T17:15:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.816+0000 7f3421a00700 1 -- 172.21.15.1:0/2739705805 wait complete. 2024-02-28T17:15:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.817+0000 7f3421a00700 1 Processor -- start 2024-02-28T17:15:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.817+0000 7f3421a00700 1 -- start start 2024-02-28T17:15:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.818+0000 7f3421a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f341c0724c0 0x7f341c11a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.818+0000 7f3421a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c072dd0 0x7f341c11a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.818+0000 7f3421a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c11ac70 con 0x7f341c072dd0 2024-02-28T17:15:05.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.818+0000 7f3421a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c113ef0 con 0x7f341c0724c0 2024-02-28T17:15:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.818+0000 7f341a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c072dd0 0x7f341c11a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.818+0000 7f341a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c072dd0 0x7f341c11a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34882/0 (socket says 172.21.15.1:34882) 2024-02-28T17:15:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.818+0000 7f341a000700 1 -- 172.21.15.1:0/1555338263 learned_addr learned my addr 172.21.15.1:0/1555338263 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.819+0000 7f341aa00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f341c0724c0 0x7f341c11a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.819+0000 7f341a000700 1 -- 172.21.15.1:0/1555338263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f341c0724c0 msgr2=0x7f341c11a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.819+0000 7f341a000700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f341c0724c0 0x7f341c11a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.819+0000 7f341a000700 1 -- 172.21.15.1:0/1555338263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c005040 con 0x7f341c072dd0 2024-02-28T17:15:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.819+0000 7f341a000700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c072dd0 0x7f341c11a590 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f340800b790 tx=0x7f3408009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.819+0000 7f3412a00700 1 -- 172.21.15.1:0/1555338263 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340800f770 con 0x7f341c072dd0 2024-02-28T17:15:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.820+0000 7f3412a00700 1 -- 172.21.15.1:0/1555338263 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f340800fda0 con 0x7f341c072dd0 2024-02-28T17:15:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.820+0000 7f3412a00700 1 -- 172.21.15.1:0/1555338263 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340800d310 con 0x7f341c072dd0 2024-02-28T17:15:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.820+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c114170 con 0x7f341c072dd0 2024-02-28T17:15:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.820+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c114660 con 0x7f341c072dd0 2024-02-28T17:15:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.821+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c10b000 con 0x7f341c072dd0 2024-02-28T17:15:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.822+0000 7f3412a00700 1 -- 172.21.15.1:0/1555338263 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3408010420 con 0x7f341c072dd0 2024-02-28T17:15:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.823+0000 7f3412a00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f340006d7f0 0x7f340006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.823+0000 7f3412a00700 1 -- 172.21.15.1:0/1555338263 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f340808cfb0 con 0x7f341c072dd0 2024-02-28T17:15:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.823+0000 7f341aa00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f340006d7f0 0x7f340006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.824+0000 7f341aa00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f340006d7f0 0x7f340006fca0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f340c00b3b0 tx=0x7f340c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:05.828+0000 7f3412a00700 1 -- 172.21.15.1:0/1555338263 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f34080c0d20 con 0x7f341c072dd0 2024-02-28T17:15:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.062+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f341c115630 con 0x7f340006d7f0 2024-02-28T17:15:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.079+0000 7f3412a00700 1 -- 172.21.15.1:0/1555338263 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f341c115630 con 0x7f340006d7f0 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (41m) 2m ago 55m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (37m) 2m ago 55m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (37m) 2m ago 54m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (40m) 2m ago 55m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (33m) 2m ago 48m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (33m) 2m ago 48m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (2m) 2m ago 48m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (18m) 2m ago 48m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (43m) 2m ago 57m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (42m) 2m ago 54m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:15:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (39m) 2m ago 57m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (38m) 2m ago 54m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (42m) 2m ago 55m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (42m) 2m ago 54m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (37m) 2m ago 53m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (36m) 2m ago 52m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (36m) 2m ago 51m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (35m) 2m ago 51m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (35m) 2m ago 50m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (34m) 2m ago 50m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:15:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (41m) 2m ago 54m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:15:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f340006d7f0 msgr2=0x7f340006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f340006d7f0 0x7f340006fca0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f340c00b3b0 tx=0x7f340c009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c072dd0 msgr2=0x7f341c11a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c072dd0 0x7f341c11a590 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f340800b790 tx=0x7f3408009250 comp rx=0 tx=0).stop 2024-02-28T17:15:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 shutdown_connections 2024-02-28T17:15:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f340006d7f0 0x7f340006fca0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f341c0724c0 0x7f341c11a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 --2- 172.21.15.1:0/1555338263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f341c072dd0 0x7f341c11a590 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 >> 172.21.15.1:0/1555338263 conn(0x7f341c0fad30 msgr2=0x7f341c0fb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 shutdown_connections 2024-02-28T17:15:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.082+0000 7f3421a00700 1 -- 172.21.15.1:0/1555338263 wait complete. 2024-02-28T17:15:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.236+0000 7f9e82e00700 1 -- 172.21.15.1:0/890080731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 msgr2=0x7f9e7c109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.236+0000 7f9e82e00700 1 --2- 172.21.15.1:0/890080731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 0x7f9e7c109260 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f9e68004970 tx=0x7f9e68009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.236+0000 7f9e82e00700 1 -- 172.21.15.1:0/890080731 shutdown_connections 2024-02-28T17:15:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.236+0000 7f9e82e00700 1 --2- 172.21.15.1:0/890080731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 0x7f9e7c109260 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.236+0000 7f9e82e00700 1 --2- 172.21.15.1:0/890080731 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e7c070330 0x7f9e7c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.236+0000 7f9e82e00700 1 -- 172.21.15.1:0/890080731 >> 172.21.15.1:0/890080731 conn(0x7f9e7c0f8ae0 msgr2=0x7f9e7c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.237+0000 7f9e82e00700 1 -- 172.21.15.1:0/890080731 shutdown_connections 2024-02-28T17:15:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.237+0000 7f9e82e00700 1 -- 172.21.15.1:0/890080731 wait complete. 2024-02-28T17:15:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.237+0000 7f9e82e00700 1 Processor -- start 2024-02-28T17:15:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.238+0000 7f9e82e00700 1 -- start start 2024-02-28T17:15:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.238+0000 7f9e82e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e7c070330 0x7f9e7c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.238+0000 7f9e82e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 0x7f9e7c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.238+0000 7f9e82e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c116790 con 0x7f9e7c070c40 2024-02-28T17:15:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.238+0000 7f9e82e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c10fae0 con 0x7f9e7c070330 2024-02-28T17:15:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.238+0000 7f9e7b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 0x7f9e7c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.238+0000 7f9e7b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 0x7f9e7c116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34898/0 (socket says 172.21.15.1:34898) 2024-02-28T17:15:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.238+0000 7f9e7b400700 1 -- 172.21.15.1:0/3867621826 learned_addr learned my addr 172.21.15.1:0/3867621826 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.239+0000 7f9e7be00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e7c070330 0x7f9e7c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.239+0000 7f9e7b400700 1 -- 172.21.15.1:0/3867621826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e7c070330 msgr2=0x7f9e7c115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.239+0000 7f9e7b400700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e7c070330 0x7f9e7c115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.239+0000 7f9e7b400700 1 -- 172.21.15.1:0/3867621826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e68005040 con 0x7f9e7c070c40 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.239+0000 7f9e7b400700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 0x7f9e7c116140 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009150 tx=0x7f9e68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.239+0000 7f9e73e00700 1 -- 172.21.15.1:0/3867621826 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e6800e040 con 0x7f9e7c070c40 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.239+0000 7f9e73e00700 1 -- 172.21.15.1:0/3867621826 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e68004290 con 0x7f9e7c070c40 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.240+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c10fd60 con 0x7f9e7c070c40 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.240+0000 7f9e73e00700 1 -- 172.21.15.1:0/3867621826 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e6801c630 con 0x7f9e7c070c40 2024-02-28T17:15:06.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.240+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c1102a0 con 0x7f9e7c070c40 2024-02-28T17:15:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.241+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c106920 con 0x7f9e7c070c40 2024-02-28T17:15:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.242+0000 7f9e73e00700 1 -- 172.21.15.1:0/3867621826 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9e68013980 con 0x7f9e7c070c40 2024-02-28T17:15:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.243+0000 7f9e73e00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9e6006d7f0 0x7f9e6006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.243+0000 7f9e73e00700 1 -- 172.21.15.1:0/3867621826 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9e6808e2c0 con 0x7f9e7c070c40 2024-02-28T17:15:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.243+0000 7f9e7be00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9e6006d7f0 0x7f9e6006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:06.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.244+0000 7f9e7be00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9e6006d7f0 0x7f9e6006fca0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c004d60 tx=0x7f9e6c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:06.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.247+0000 7f9e73e00700 1 -- 172.21.15.1:0/3867621826 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9e68057af0 con 0x7f9e7c070c40 2024-02-28T17:15:06.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.561+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9e7c04bed0 con 0x7f9e7c070c40 2024-02-28T17:15:06.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.562+0000 7f9e73e00700 1 -- 172.21.15.1:0/3867621826 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9e6805b1b0 con 0x7f9e7c070c40 2024-02-28T17:15:06.567 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:15:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:15:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:15:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:15:06.567 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:15:06.568 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:15:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:15:06.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:15:06.569 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:15:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9e6006d7f0 msgr2=0x7f9e6006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9e6006d7f0 0x7f9e6006fca0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c004d60 tx=0x7f9e6c00b040 comp rx=0 tx=0).stop 2024-02-28T17:15:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 msgr2=0x7f9e7c116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 0x7f9e7c116140 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009150 tx=0x7f9e68009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 shutdown_connections 2024-02-28T17:15:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9e6006d7f0 0x7f9e6006fca0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9e7c070330 0x7f9e7c115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 --2- 172.21.15.1:0/3867621826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e7c070c40 0x7f9e7c116140 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 >> 172.21.15.1:0/3867621826 conn(0x7f9e7c0f8ae0 msgr2=0x7f9e7c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 shutdown_connections 2024-02-28T17:15:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.565+0000 7f9e82e00700 1 -- 172.21.15.1:0/3867621826 wait complete. 2024-02-28T17:15:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.718+0000 7f29dfe00700 1 -- 172.21.15.1:0/3578604587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 msgr2=0x7f29e00fefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.718+0000 7f29dfe00700 1 --2- 172.21.15.1:0/3578604587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 0x7f29e00fefe0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f29cc004940 tx=0x7f29cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.718+0000 7f29dfe00700 1 -- 172.21.15.1:0/3578604587 shutdown_connections 2024-02-28T17:15:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.718+0000 7f29dfe00700 1 --2- 172.21.15.1:0/3578604587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 0x7f29e00fefe0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.718+0000 7f29dfe00700 1 --2- 172.21.15.1:0/3578604587 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29e0104f80 0x7f29e0105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.718+0000 7f29dfe00700 1 -- 172.21.15.1:0/3578604587 >> 172.21.15.1:0/3578604587 conn(0x7f29e00faae0 msgr2=0x7f29e00fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.718+0000 7f29dfe00700 1 -- 172.21.15.1:0/3578604587 shutdown_connections 2024-02-28T17:15:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.718+0000 7f29dfe00700 1 -- 172.21.15.1:0/3578604587 wait complete. 2024-02-28T17:15:06.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.719+0000 7f29dfe00700 1 Processor -- start 2024-02-28T17:15:06.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.719+0000 7f29dfe00700 1 -- start start 2024-02-28T17:15:06.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.719+0000 7f29dfe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29e0104f80 0x7f29e0115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:06.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.719+0000 7f29dfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 0x7f29e0116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:06.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.719+0000 7f29dfe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0116650 con 0x7f29e0105920 2024-02-28T17:15:06.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.719+0000 7f29dfe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e010f980 con 0x7f29e0104f80 2024-02-28T17:15:06.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.720+0000 7f29de000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 0x7f29e0116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:06.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.720+0000 7f29de000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 0x7f29e0116000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34916/0 (socket says 172.21.15.1:34916) 2024-02-28T17:15:06.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.720+0000 7f29de000700 1 -- 172.21.15.1:0/3606745092 learned_addr learned my addr 172.21.15.1:0/3606745092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.720+0000 7f29dea00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29e0104f80 0x7f29e0115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.720+0000 7f29de000700 1 -- 172.21.15.1:0/3606745092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29e0104f80 msgr2=0x7f29e0115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.720+0000 7f29de000700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29e0104f80 0x7f29e0115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.720+0000 7f29de000700 1 -- 172.21.15.1:0/3606745092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc005040 con 0x7f29e0105920 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.720+0000 7f29de000700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 0x7f29e0116000 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009150 tx=0x7f29cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.721+0000 7f29d6a00700 1 -- 172.21.15.1:0/3606745092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29cc00e040 con 0x7f29e0105920 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.721+0000 7f29d6a00700 1 -- 172.21.15.1:0/3606745092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29cc00bbf0 con 0x7f29e0105920 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.721+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e010fc00 con 0x7f29e0105920 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.721+0000 7f29d6a00700 1 -- 172.21.15.1:0/3606745092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29cc01c630 con 0x7f29e0105920 2024-02-28T17:15:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.721+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e0110140 con 0x7f29e0105920 2024-02-28T17:15:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.722+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e0107450 con 0x7f29e0105920 2024-02-28T17:15:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.723+0000 7f29d6a00700 1 -- 172.21.15.1:0/3606745092 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f29cc00bd60 con 0x7f29e0105920 2024-02-28T17:15:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.724+0000 7f29d6a00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f29c406d7e0 0x7f29c406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.724+0000 7f29d6a00700 1 -- 172.21.15.1:0/3606745092 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f29cc08e270 con 0x7f29e0105920 2024-02-28T17:15:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.724+0000 7f29dea00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f29c406d7e0 0x7f29c406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.725+0000 7f29dea00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f29c406d7e0 0x7f29c406fc90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f29d0004f30 tx=0x7f29d000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:06.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.729+0000 7f29d6a00700 1 -- 172.21.15.1:0/3606745092 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f29cc05b650 con 0x7f29e0105920 2024-02-28T17:15:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.994+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f29e010c060 con 0x7f29e0105920 2024-02-28T17:15:06.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.995+0000 7f29d6a00700 1 -- 172.21.15.1:0/3606745092 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f29cc05b1e0 con 0x7f29e0105920 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:15:07.001 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:15:07.002 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:15:07.003 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:15:07.004 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:15:07.004 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:15:07.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f29c406d7e0 msgr2=0x7f29c406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f29c406d7e0 0x7f29c406fc90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f29d0004f30 tx=0x7f29d000b040 comp rx=0 tx=0).stop 2024-02-28T17:15:07.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 msgr2=0x7f29e0116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 0x7f29e0116000 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009150 tx=0x7f29cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:07.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 shutdown_connections 2024-02-28T17:15:07.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f29c406d7e0 0x7f29c406fc90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f29e0104f80 0x7f29e0115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 --2- 172.21.15.1:0/3606745092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0105920 0x7f29e0116000 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:06.999+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 >> 172.21.15.1:0/3606745092 conn(0x7f29e00faae0 msgr2=0x7f29e00fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:07.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.000+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 shutdown_connections 2024-02-28T17:15:07.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.000+0000 7f29dfe00700 1 -- 172.21.15.1:0/3606745092 wait complete. 2024-02-28T17:15:07.007 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:15:07.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.144+0000 7f1462600700 1 -- 172.21.15.1:0/1393552473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105b10 msgr2=0x7f145c0ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.144+0000 7f1462600700 1 --2- 172.21.15.1:0/1393552473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105b10 0x7f145c0ff180 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1448004940 tx=0x7f1448009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:07.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.144+0000 7f1462600700 1 -- 172.21.15.1:0/1393552473 shutdown_connections 2024-02-28T17:15:07.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.144+0000 7f1462600700 1 --2- 172.21.15.1:0/1393552473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105b10 0x7f145c0ff180 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.144+0000 7f1462600700 1 --2- 172.21.15.1:0/1393552473 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f145c105170 0x7f145c105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.144+0000 7f1462600700 1 -- 172.21.15.1:0/1393552473 >> 172.21.15.1:0/1393552473 conn(0x7f145c0face0 msgr2=0x7f145c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:07.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.144+0000 7f1462600700 1 -- 172.21.15.1:0/1393552473 shutdown_connections 2024-02-28T17:15:07.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.144+0000 7f1462600700 1 -- 172.21.15.1:0/1393552473 wait complete. 2024-02-28T17:15:07.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.145+0000 7f1462600700 1 Processor -- start 2024-02-28T17:15:07.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.145+0000 7f1462600700 1 -- start start 2024-02-28T17:15:07.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.145+0000 7f1462600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105170 0x7f145c115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:07.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.145+0000 7f1462600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f145c105b10 0x7f145c116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:07.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.145+0000 7f1462600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c116870 con 0x7f145c105170 2024-02-28T17:15:07.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.145+0000 7f1462600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c10fb80 con 0x7f145c105b10 2024-02-28T17:15:07.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.146+0000 7f145b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105170 0x7f145c115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:07.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.146+0000 7f145b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105170 0x7f145c115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34934/0 (socket says 172.21.15.1:34934) 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.146+0000 7f145b400700 1 -- 172.21.15.1:0/3390498160 learned_addr learned my addr 172.21.15.1:0/3390498160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.146+0000 7f145b400700 1 -- 172.21.15.1:0/3390498160 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f145c105b10 msgr2=0x7f145c116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.146+0000 7f145b400700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f145c105b10 0x7f145c116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.146+0000 7f145b400700 1 -- 172.21.15.1:0/3390498160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1448005040 con 0x7f145c105170 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.146+0000 7f145b400700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105170 0x7f145c115ce0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f144c002a50 tx=0x7f144c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.147+0000 7f1453400700 1 -- 172.21.15.1:0/3390498160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f144c007cc0 con 0x7f145c105170 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.147+0000 7f1453400700 1 -- 172.21.15.1:0/3390498160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f144c00f040 con 0x7f145c105170 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.147+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145c10fe60 con 0x7f145c105170 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.147+0000 7f1453400700 1 -- 172.21.15.1:0/3390498160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f144c009640 con 0x7f145c105170 2024-02-28T17:15:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.147+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f145c1103a0 con 0x7f145c105170 2024-02-28T17:15:07.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.148+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145c107640 con 0x7f145c105170 2024-02-28T17:15:07.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.149+0000 7f1453400700 1 -- 172.21.15.1:0/3390498160 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f144c0097a0 con 0x7f145c105170 2024-02-28T17:15:07.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.150+0000 7f1453400700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f144006d7f0 0x7f144006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:07.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.150+0000 7f1453400700 1 -- 172.21.15.1:0/3390498160 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f144c08cc60 con 0x7f145c105170 2024-02-28T17:15:07.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.150+0000 7f145aa00700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f144006d7f0 0x7f144006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:07.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.151+0000 7f145aa00700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f144006d7f0 0x7f144006fca0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1448004f90 tx=0x7f1448009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:07.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.154+0000 7f1453400700 1 -- 172.21.15.1:0/3390498160 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f144c059fc0 con 0x7f145c105170 2024-02-28T17:15:07.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:06 smithi066 ceph-mon[85781]: from='client.36160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:07.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:06 smithi066 ceph-mon[85781]: pgmap v1223: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:07.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:06 smithi066 ceph-mon[85781]: from='client.36164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:07.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:06 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3867621826' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:15:07.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:06 smithi001 ceph-mon[112940]: from='client.36160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:07.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:06 smithi001 ceph-mon[112940]: pgmap v1223: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:07.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:06 smithi001 ceph-mon[112940]: from='client.36164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:07.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:06 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3867621826' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:15:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.389+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f145c04bf40 con 0x7f144006d7f0 2024-02-28T17:15:07.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.392+0000 7f1453400700 1 -- 172.21.15.1:0/3390498160 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f145c04bf40 con 0x7f144006d7f0 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:15:07.397 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:15:07.398 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:15:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f144006d7f0 msgr2=0x7f144006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f144006d7f0 0x7f144006fca0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1448004f90 tx=0x7f1448009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105170 msgr2=0x7f145c115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105170 0x7f145c115ce0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f144c002a50 tx=0x7f144c0046b0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 shutdown_connections 2024-02-28T17:15:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f144006d7f0 0x7f144006fca0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f145c105170 0x7f145c115ce0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 --2- 172.21.15.1:0/3390498160 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f145c105b10 0x7f145c116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.394+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 >> 172.21.15.1:0/3390498160 conn(0x7f145c0face0 msgr2=0x7f145c0fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:07.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.395+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 shutdown_connections 2024-02-28T17:15:07.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.395+0000 7f1462600700 1 -- 172.21.15.1:0/3390498160 wait complete. 2024-02-28T17:15:07.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.550+0000 7f4ee9600700 1 -- 172.21.15.1:0/3459101286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee41095f0 msgr2=0x7f4ee41099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.550+0000 7f4ee9600700 1 --2- 172.21.15.1:0/3459101286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee41095f0 0x7f4ee41099c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4004940 tx=0x7f4ed4009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:07.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.551+0000 7f4ee9600700 1 -- 172.21.15.1:0/3459101286 shutdown_connections 2024-02-28T17:15:07.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.551+0000 7f4ee9600700 1 --2- 172.21.15.1:0/3459101286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ee4103dc0 0x7f4ee4104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.551+0000 7f4ee9600700 1 --2- 172.21.15.1:0/3459101286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee41095f0 0x7f4ee41099c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.551+0000 7f4ee9600700 1 -- 172.21.15.1:0/3459101286 >> 172.21.15.1:0/3459101286 conn(0x7f4ee4072d50 msgr2=0x7f4ee4075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:07.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.551+0000 7f4ee9600700 1 -- 172.21.15.1:0/3459101286 shutdown_connections 2024-02-28T17:15:07.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.551+0000 7f4ee9600700 1 -- 172.21.15.1:0/3459101286 wait complete. 2024-02-28T17:15:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.552+0000 7f4ee9600700 1 Processor -- start 2024-02-28T17:15:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.552+0000 7f4ee9600700 1 -- start start 2024-02-28T17:15:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.552+0000 7f4ee9600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee4103dc0 0x7f4ee411a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.552+0000 7f4ee9600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ee41095f0 0x7f4ee411a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.552+0000 7f4ee9600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee411ac70 con 0x7f4ee4103dc0 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.552+0000 7f4ee9600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee4113f80 con 0x7f4ee41095f0 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.553+0000 7f4ee2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee4103dc0 0x7f4ee411a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.553+0000 7f4ee2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee4103dc0 0x7f4ee411a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34952/0 (socket says 172.21.15.1:34952) 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.553+0000 7f4ee2a00700 1 -- 172.21.15.1:0/1387317203 learned_addr learned my addr 172.21.15.1:0/1387317203 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.553+0000 7f4ee2000700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ee41095f0 0x7f4ee411a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.553+0000 7f4ee2a00700 1 -- 172.21.15.1:0/1387317203 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ee41095f0 msgr2=0x7f4ee411a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.553+0000 7f4ee2a00700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ee41095f0 0x7f4ee411a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.553+0000 7f4ee2a00700 1 -- 172.21.15.1:0/1387317203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed4005040 con 0x7f4ee4103dc0 2024-02-28T17:15:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.554+0000 7f4ee2a00700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee4103dc0 0x7f4ee411a0e0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4009fd0 tx=0x7f4ed4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.554+0000 7f4edaa00700 1 -- 172.21.15.1:0/1387317203 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ed400e040 con 0x7f4ee4103dc0 2024-02-28T17:15:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.554+0000 7f4edaa00700 1 -- 172.21.15.1:0/1387317203 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ed400bbf0 con 0x7f4ee4103dc0 2024-02-28T17:15:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.554+0000 7f4edaa00700 1 -- 172.21.15.1:0/1387317203 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ed401c3f0 con 0x7f4ee4103dc0 2024-02-28T17:15:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.554+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee4114200 con 0x7f4ee4103dc0 2024-02-28T17:15:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.554+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee41146f0 con 0x7f4ee4103dc0 2024-02-28T17:15:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.555+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee4106850 con 0x7f4ee4103dc0 2024-02-28T17:15:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.556+0000 7f4edaa00700 1 -- 172.21.15.1:0/1387317203 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4ed400bd60 con 0x7f4ee4103dc0 2024-02-28T17:15:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.558+0000 7f4edaa00700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4ec806d7f0 0x7f4ec806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.558+0000 7f4edaa00700 1 -- 172.21.15.1:0/1387317203 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4ed408e2b0 con 0x7f4ee4103dc0 2024-02-28T17:15:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.558+0000 7f4ee2000700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4ec806d7f0 0x7f4ec806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.559+0000 7f4ee2000700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4ec806d7f0 0x7f4ec806fca0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4115980 tx=0x7f4ed0009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.563+0000 7f4edaa00700 1 -- 172.21.15.1:0/1387317203 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ed405b610 con 0x7f4ee4103dc0 2024-02-28T17:15:07.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.877+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4ee404bed0 con 0x7f4ee4103dc0 2024-02-28T17:15:07.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.878+0000 7f4edaa00700 1 -- 172.21.15.1:0/1387317203 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f4ed405b1a0 con 0x7f4ee4103dc0 2024-02-28T17:15:07.883 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:15:07.883 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:15:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:15:07.883 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:15:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:15:07.885 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:15:07.885 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:15:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:15:07.885 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:15:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:15:07.885 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:15:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:15:07.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4ec806d7f0 msgr2=0x7f4ec806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4ec806d7f0 0x7f4ec806fca0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4115980 tx=0x7f4ed0009ee0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee4103dc0 msgr2=0x7f4ee411a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:07.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee4103dc0 0x7f4ee411a0e0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4009fd0 tx=0x7f4ed4009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:07.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 shutdown_connections 2024-02-28T17:15:07.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4ec806d7f0 0x7f4ec806fca0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ee4103dc0 0x7f4ee411a0e0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 --2- 172.21.15.1:0/1387317203 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4ee41095f0 0x7f4ee411a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 >> 172.21.15.1:0/1387317203 conn(0x7f4ee4072d50 msgr2=0x7f4ee4074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.881+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 shutdown_connections 2024-02-28T17:15:07.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:07.882+0000 7f4ee9600700 1 -- 172.21.15.1:0/1387317203 wait complete. 2024-02-28T17:15:08.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:07 smithi066 ceph-mon[85781]: from='client.36168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:08.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:07 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3606745092' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:15:08.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:07 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1387317203' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:15:08.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:07 smithi001 ceph-mon[112940]: from='client.36168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:08.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:07 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3606745092' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:15:08.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:07 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1387317203' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:15:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:08 smithi066 ceph-mon[85781]: from='client.36180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:09.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:08 smithi066 ceph-mon[85781]: pgmap v1224: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:09.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:08 smithi001 ceph-mon[112940]: from='client.36180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:09.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:08 smithi001 ceph-mon[112940]: pgmap v1224: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:11.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:10 smithi066 ceph-mon[85781]: pgmap v1225: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:11.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:15:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:10 smithi001 ceph-mon[112940]: pgmap v1225: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:15:13.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:12 smithi066 ceph-mon[85781]: pgmap v1226: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:13.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:12 smithi001 ceph-mon[112940]: pgmap v1226: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:15.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:14 smithi066 ceph-mon[85781]: pgmap v1227: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:15.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:14 smithi001 ceph-mon[112940]: pgmap v1227: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:17.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:16 smithi066 ceph-mon[85781]: pgmap v1228: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:17.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:16 smithi001 ceph-mon[112940]: pgmap v1228: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:19.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:18 smithi066 ceph-mon[85781]: pgmap v1229: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:19.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:18 smithi001 ceph-mon[112940]: pgmap v1229: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:21.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:20 smithi066 ceph-mon[85781]: pgmap v1230: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:21.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:20 smithi001 ceph-mon[112940]: pgmap v1230: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:23.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:22 smithi066 ceph-mon[85781]: pgmap v1231: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:23.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:22 smithi001 ceph-mon[112940]: pgmap v1231: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:25.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:24 smithi066 ceph-mon[85781]: pgmap v1232: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:25.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:24 smithi001 ceph-mon[112940]: pgmap v1232: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:26.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:15:26.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:15:27.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:26 smithi066 ceph-mon[85781]: pgmap v1233: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:27.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:26 smithi001 ceph-mon[112940]: pgmap v1233: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:28.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:28 smithi066 ceph-mon[85781]: pgmap v1234: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:28.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:28 smithi001 ceph-mon[112940]: pgmap v1234: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:30.763 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:30 smithi066 ceph-mon[85781]: pgmap v1235: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:30.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:30 smithi001 ceph-mon[112940]: pgmap v1235: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:32.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:32 smithi066 ceph-mon[85781]: pgmap v1236: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:32.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:32 smithi001 ceph-mon[112940]: pgmap v1236: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:34.762 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:34 smithi066 ceph-mon[85781]: pgmap v1237: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:34.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:34 smithi001 ceph-mon[112940]: pgmap v1237: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:36.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:36 smithi001 ceph-mon[112940]: pgmap v1238: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:37.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:36 smithi066 ceph-mon[85781]: pgmap v1238: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:38.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.052+0000 7ff10c200700 1 -- 172.21.15.1:0/54533518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 msgr2=0x7ff104070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.052+0000 7ff10c200700 1 --2- 172.21.15.1:0/54533518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 0x7ff104070330 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8004970 tx=0x7ff0f8009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:38.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.052+0000 7ff10c200700 1 -- 172.21.15.1:0/54533518 shutdown_connections 2024-02-28T17:15:38.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.052+0000 7ff10c200700 1 --2- 172.21.15.1:0/54533518 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff104070870 0x7ff104070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.052+0000 7ff10c200700 1 --2- 172.21.15.1:0/54533518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 0x7ff104070330 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.052+0000 7ff10c200700 1 -- 172.21.15.1:0/54533518 >> 172.21.15.1:0/54533518 conn(0x7ff1040f8a80 msgr2=0x7ff1040fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:38.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.053+0000 7ff10c200700 1 -- 172.21.15.1:0/54533518 shutdown_connections 2024-02-28T17:15:38.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.053+0000 7ff10c200700 1 -- 172.21.15.1:0/54533518 wait complete. 2024-02-28T17:15:38.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.053+0000 7ff10c200700 1 Processor -- start 2024-02-28T17:15:38.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.054+0000 7ff10c200700 1 -- start start 2024-02-28T17:15:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.054+0000 7ff10c200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff104070870 0x7ff104119f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.054+0000 7ff10c200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 0x7ff10411a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.054+0000 7ff10c200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10411aac0 con 0x7ff104071f60 2024-02-28T17:15:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.054+0000 7ff10c200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff104113dd0 con 0x7ff104070870 2024-02-28T17:15:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.054+0000 7ff108e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 0x7ff10411a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.054+0000 7ff108e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 0x7ff10411a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59600/0 (socket says 172.21.15.1:59600) 2024-02-28T17:15:38.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.054+0000 7ff108e00700 1 -- 172.21.15.1:0/4158219131 learned_addr learned my addr 172.21.15.1:0/4158219131 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.055+0000 7ff109800700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff104070870 0x7ff104119f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.055+0000 7ff108e00700 1 -- 172.21.15.1:0/4158219131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff104070870 msgr2=0x7ff104119f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.055+0000 7ff108e00700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff104070870 0x7ff104119f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.055+0000 7ff108e00700 1 -- 172.21.15.1:0/4158219131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f8005040 con 0x7ff104071f60 2024-02-28T17:15:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.055+0000 7ff109800700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff104070870 0x7ff104119f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:15:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.055+0000 7ff108e00700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 0x7ff10411a470 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0002a50 tx=0x7ff0f00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.055+0000 7ff101600700 1 -- 172.21.15.1:0/4158219131 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f0007cc0 con 0x7ff104071f60 2024-02-28T17:15:38.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.056+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1041140b0 con 0x7ff104071f60 2024-02-28T17:15:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.056+0000 7ff101600700 1 -- 172.21.15.1:0/4158219131 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0f000f040 con 0x7ff104071f60 2024-02-28T17:15:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.056+0000 7ff101600700 1 -- 172.21.15.1:0/4158219131 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f0009640 con 0x7ff104071f60 2024-02-28T17:15:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.056+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1041145f0 con 0x7ff104071f60 2024-02-28T17:15:38.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.057+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1040fe3a0 con 0x7ff104071f60 2024-02-28T17:15:38.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.058+0000 7ff101600700 1 -- 172.21.15.1:0/4158219131 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff0f00097a0 con 0x7ff104071f60 2024-02-28T17:15:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.060+0000 7ff101600700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff0ec06d7e0 0x7ff0ec06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.060+0000 7ff101600700 1 -- 172.21.15.1:0/4158219131 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff0f008d250 con 0x7ff104071f60 2024-02-28T17:15:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.060+0000 7ff109800700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff0ec06d7e0 0x7ff0ec06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.061+0000 7ff109800700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff0ec06d7e0 0x7ff0ec06fc90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009fd0 tx=0x7ff0f8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:38.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.065+0000 7ff101600700 1 -- 172.21.15.1:0/4158219131 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff0f005a630 con 0x7ff104071f60 2024-02-28T17:15:38.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.312+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff104115510 con 0x7ff0ec06d7e0 2024-02-28T17:15:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.315+0000 7ff101600700 1 -- 172.21.15.1:0/4158219131 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff104115510 con 0x7ff0ec06d7e0 2024-02-28T17:15:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff0ec06d7e0 msgr2=0x7ff0ec06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff0ec06d7e0 0x7ff0ec06fc90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009fd0 tx=0x7ff0f8009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 msgr2=0x7ff10411a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 0x7ff10411a470 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0002a50 tx=0x7ff0f00046b0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 shutdown_connections 2024-02-28T17:15:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff0ec06d7e0 0x7ff0ec06fc90 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff104070870 0x7ff104119f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 --2- 172.21.15.1:0/4158219131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104071f60 0x7ff10411a470 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 >> 172.21.15.1:0/4158219131 conn(0x7ff1040f8a80 msgr2=0x7ff1040fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 shutdown_connections 2024-02-28T17:15:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.318+0000 7ff10c200700 1 -- 172.21.15.1:0/4158219131 wait complete. 2024-02-28T17:15:38.338 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:15:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.477+0000 7f504a200700 1 -- 172.21.15.1:0/1958849955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 msgr2=0x7f5044103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.477+0000 7f504a200700 1 --2- 172.21.15.1:0/1958849955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 0x7f5044103320 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5034004940 tx=0x7f5034009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.477+0000 7f504a200700 1 -- 172.21.15.1:0/1958849955 shutdown_connections 2024-02-28T17:15:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.478+0000 7f504a200700 1 --2- 172.21.15.1:0/1958849955 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50441038f0 0x7f50440fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.478+0000 7f504a200700 1 --2- 172.21.15.1:0/1958849955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 0x7f5044103320 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.478+0000 7f504a200700 1 -- 172.21.15.1:0/1958849955 >> 172.21.15.1:0/1958849955 conn(0x7f50440f8ae0 msgr2=0x7f50440faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.478+0000 7f504a200700 1 -- 172.21.15.1:0/1958849955 shutdown_connections 2024-02-28T17:15:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.478+0000 7f504a200700 1 -- 172.21.15.1:0/1958849955 wait complete. 2024-02-28T17:15:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.479+0000 7f504a200700 1 Processor -- start 2024-02-28T17:15:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.479+0000 7f504a200700 1 -- start start 2024-02-28T17:15:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.480+0000 7f504a200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 0x7f5044115be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.480+0000 7f504a200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50441038f0 0x7f5044116120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.480+0000 7f504a200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044116770 con 0x7f5044102f50 2024-02-28T17:15:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.480+0000 7f504a200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504410faa0 con 0x7f50441038f0 2024-02-28T17:15:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.480+0000 7f5043400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 0x7f5044115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.480+0000 7f5043400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 0x7f5044115be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59606/0 (socket says 172.21.15.1:59606) 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.480+0000 7f5043400700 1 -- 172.21.15.1:0/145691292 learned_addr learned my addr 172.21.15.1:0/145691292 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.480+0000 7f5042a00700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50441038f0 0x7f5044116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.481+0000 7f5043400700 1 -- 172.21.15.1:0/145691292 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50441038f0 msgr2=0x7f5044116120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.481+0000 7f5043400700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50441038f0 0x7f5044116120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.481+0000 7f5043400700 1 -- 172.21.15.1:0/145691292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5034005040 con 0x7f5044102f50 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.481+0000 7f5043400700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 0x7f5044115be0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f5034009b30 tx=0x7f5034009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.481+0000 7f503b400700 1 -- 172.21.15.1:0/145691292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5034021070 con 0x7f5044102f50 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.481+0000 7f503b400700 1 -- 172.21.15.1:0/145691292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5034013e50 con 0x7f5044102f50 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.481+0000 7f503b400700 1 -- 172.21.15.1:0/145691292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f503400e070 con 0x7f5044102f50 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.481+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504410fd20 con 0x7f5044102f50 2024-02-28T17:15:38.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.482+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5044110200 con 0x7f5044102f50 2024-02-28T17:15:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.483+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50440fc6e0 con 0x7f5044102f50 2024-02-28T17:15:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.484+0000 7f503b400700 1 -- 172.21.15.1:0/145691292 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f503401cc40 con 0x7f5044102f50 2024-02-28T17:15:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.485+0000 7f503b400700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f502806d7e0 0x7f502806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.485+0000 7f503b400700 1 -- 172.21.15.1:0/145691292 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f503408e390 con 0x7f5044102f50 2024-02-28T17:15:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.485+0000 7f5042a00700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f502806d7e0 0x7f502806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.486+0000 7f5042a00700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f502806d7e0 0x7f502806fc90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5030004600 tx=0x7f503000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:38.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.490+0000 7f503b400700 1 -- 172.21.15.1:0/145691292 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f503405b770 con 0x7f5044102f50 2024-02-28T17:15:38.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.735+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5044111230 con 0x7f502806d7e0 2024-02-28T17:15:38.739 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:38 smithi001 ceph-mon[112940]: pgmap v1239: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.738+0000 7f503b400700 1 -- 172.21.15.1:0/145691292 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5044111230 con 0x7f502806d7e0 2024-02-28T17:15:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f502806d7e0 msgr2=0x7f502806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f502806d7e0 0x7f502806fc90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5030004600 tx=0x7f503000b040 comp rx=0 tx=0).stop 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 msgr2=0x7f5044115be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 0x7f5044115be0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f5034009b30 tx=0x7f5034009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 shutdown_connections 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f502806d7e0 0x7f502806fc90 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5044102f50 0x7f5044115be0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 --2- 172.21.15.1:0/145691292 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50441038f0 0x7f5044116120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.740+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 >> 172.21.15.1:0/145691292 conn(0x7f50440f8ae0 msgr2=0x7f50440fa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.741+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 shutdown_connections 2024-02-28T17:15:38.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.741+0000 7f504a200700 1 -- 172.21.15.1:0/145691292 wait complete. 2024-02-28T17:15:38.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.890+0000 7fd3f9200700 1 -- 172.21.15.1:0/1409602965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f4102f40 msgr2=0x7fd3f4103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.890+0000 7fd3f9200700 1 --2- 172.21.15.1:0/1409602965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f4102f40 0x7fd3f4103310 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4004940 tx=0x7fd3e4009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:38.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.890+0000 7fd3f9200700 1 -- 172.21.15.1:0/1409602965 shutdown_connections 2024-02-28T17:15:38.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.890+0000 7fd3f9200700 1 --2- 172.21.15.1:0/1409602965 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3f41038e0 0x7fd3f40fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.890+0000 7fd3f9200700 1 --2- 172.21.15.1:0/1409602965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f4102f40 0x7fd3f4103310 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.890+0000 7fd3f9200700 1 -- 172.21.15.1:0/1409602965 >> 172.21.15.1:0/1409602965 conn(0x7fd3f40f8af0 msgr2=0x7fd3f40faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.890+0000 7fd3f9200700 1 -- 172.21.15.1:0/1409602965 shutdown_connections 2024-02-28T17:15:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.890+0000 7fd3f9200700 1 -- 172.21.15.1:0/1409602965 wait complete. 2024-02-28T17:15:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.891+0000 7fd3f9200700 1 Processor -- start 2024-02-28T17:15:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.891+0000 7fd3f9200700 1 -- start start 2024-02-28T17:15:38.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f9200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3f4102f40 0x7fd3f4115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f9200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f41038e0 0x7fd3f4116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f9200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41167a0 con 0x7fd3f41038e0 2024-02-28T17:15:38.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f9200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f410faf0 con 0x7fd3f4102f40 2024-02-28T17:15:38.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f41038e0 0x7fd3f4116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f41038e0 0x7fd3f4116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59632/0 (socket says 172.21.15.1:59632) 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f1400700 1 -- 172.21.15.1:0/2822758242 learned_addr learned my addr 172.21.15.1:0/2822758242 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f1e00700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3f4102f40 0x7fd3f4115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f1400700 1 -- 172.21.15.1:0/2822758242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3f4102f40 msgr2=0x7fd3f4115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f1400700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3f4102f40 0x7fd3f4115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.892+0000 7fd3f1400700 1 -- 172.21.15.1:0/2822758242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e4005040 con 0x7fd3f41038e0 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.893+0000 7fd3f1400700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f41038e0 0x7fd3f4116150 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000c800 tx=0x7fd3e0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.893+0000 7fd3ea000700 1 -- 172.21.15.1:0/2822758242 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e000a760 con 0x7fd3f41038e0 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.893+0000 7fd3ea000700 1 -- 172.21.15.1:0/2822758242 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3e000ad90 con 0x7fd3f41038e0 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.893+0000 7fd3ea000700 1 -- 172.21.15.1:0/2822758242 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e00113a0 con 0x7fd3f41038e0 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.893+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f410fdd0 con 0x7fd3f41038e0 2024-02-28T17:15:38.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.893+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f4110310 con 0x7fd3f41038e0 2024-02-28T17:15:38.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.894+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f40fc6f0 con 0x7fd3f41038e0 2024-02-28T17:15:38.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.895+0000 7fd3ea000700 1 -- 172.21.15.1:0/2822758242 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd3e0012400 con 0x7fd3f41038e0 2024-02-28T17:15:38.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.896+0000 7fd3ea000700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3d806d7f0 0x7fd3d806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:38.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.896+0000 7fd3ea000700 1 -- 172.21.15.1:0/2822758242 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd3e008cf60 con 0x7fd3f41038e0 2024-02-28T17:15:38.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.896+0000 7fd3f1e00700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3d806d7f0 0x7fd3d806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:38.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.897+0000 7fd3f1e00700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3d806d7f0 0x7fd3d806fca0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400b3b0 tx=0x7fd3e4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:38.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:38.901+0000 7fd3ea000700 1 -- 172.21.15.1:0/2822758242 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd3e0056790 con 0x7fd3f41038e0 2024-02-28T17:15:39.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:38 smithi066 ceph-mon[85781]: pgmap v1239: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.130+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3f4111260 con 0x7fd3d806d7f0 2024-02-28T17:15:39.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.148+0000 7fd3ea000700 1 -- 172.21.15.1:0/2822758242 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd3f4111260 con 0x7fd3d806d7f0 2024-02-28T17:15:39.152 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:15:39.152 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (41m) 3m ago 56m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:15:39.152 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (38m) 3m ago 56m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:15:39.152 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (38m) 3m ago 54m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (41m) 3m ago 55m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (34m) 3m ago 48m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (33m) 3m ago 48m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (3m) 3m ago 48m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (18m) 3m ago 48m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (44m) 3m ago 57m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (43m) 3m ago 54m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (40m) 3m ago 57m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (39m) 3m ago 54m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (42m) 3m ago 56m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (42m) 3m ago 54m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (37m) 3m ago 53m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (37m) 3m ago 53m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:15:39.153 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (36m) 3m ago 52m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:15:39.154 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (36m) 3m ago 51m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:15:39.154 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (35m) 3m ago 51m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:15:39.154 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (34m) 3m ago 50m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:15:39.154 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (42m) 3m ago 55m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.151+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3d806d7f0 msgr2=0x7fd3d806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.151+0000 7fd3f9200700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3d806d7f0 0x7fd3d806fca0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400b3b0 tx=0x7fd3e4009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.151+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f41038e0 msgr2=0x7fd3f4116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.151+0000 7fd3f9200700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f41038e0 0x7fd3f4116150 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000c800 tx=0x7fd3e0004470 comp rx=0 tx=0).stop 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.152+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 shutdown_connections 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.152+0000 7fd3f9200700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd3d806d7f0 0x7fd3d806fca0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.152+0000 7fd3f9200700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd3f4102f40 0x7fd3f4115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.152+0000 7fd3f9200700 1 --2- 172.21.15.1:0/2822758242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f41038e0 0x7fd3f4116150 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.152+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 >> 172.21.15.1:0/2822758242 conn(0x7fd3f40f8af0 msgr2=0x7fd3f40fa460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.152+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 shutdown_connections 2024-02-28T17:15:39.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.152+0000 7fd3f9200700 1 -- 172.21.15.1:0/2822758242 wait complete. 2024-02-28T17:15:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.306+0000 7f3f18200700 1 -- 172.21.15.1:0/3032994006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 msgr2=0x7f3f10070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.306+0000 7f3f18200700 1 --2- 172.21.15.1:0/3032994006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 0x7f3f10070ce0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3efc004970 tx=0x7f3efc009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.306+0000 7f3f18200700 1 -- 172.21.15.1:0/3032994006 shutdown_connections 2024-02-28T17:15:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.306+0000 7f3f18200700 1 --2- 172.21.15.1:0/3032994006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 0x7f3f10070ce0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.306+0000 7f3f18200700 1 --2- 172.21.15.1:0/3032994006 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f10071f60 0x7f3f10070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.306+0000 7f3f18200700 1 -- 172.21.15.1:0/3032994006 >> 172.21.15.1:0/3032994006 conn(0x7f3f100f8af0 msgr2=0x7f3f100faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.307+0000 7f3f18200700 1 -- 172.21.15.1:0/3032994006 shutdown_connections 2024-02-28T17:15:39.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.307+0000 7f3f18200700 1 -- 172.21.15.1:0/3032994006 wait complete. 2024-02-28T17:15:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.307+0000 7f3f18200700 1 Processor -- start 2024-02-28T17:15:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.308+0000 7f3f18200700 1 -- start start 2024-02-28T17:15:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.308+0000 7f3f18200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 0x7f3f1011a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.308+0000 7f3f18200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f10071f60 0x7f3f1011a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.308+0000 7f3f18200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1011abc0 con 0x7f3f10070870 2024-02-28T17:15:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.308+0000 7f3f18200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f10113ed0 con 0x7f3f10071f60 2024-02-28T17:15:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.308+0000 7f3f15800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 0x7f3f1011a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.308+0000 7f3f15800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 0x7f3f1011a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59648/0 (socket says 172.21.15.1:59648) 2024-02-28T17:15:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.308+0000 7f3f15800700 1 -- 172.21.15.1:0/63456824 learned_addr learned my addr 172.21.15.1:0/63456824 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.309+0000 7f3f15800700 1 -- 172.21.15.1:0/63456824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f10071f60 msgr2=0x7f3f1011a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.309+0000 7f3f14e00700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f10071f60 0x7f3f1011a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.309+0000 7f3f15800700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f10071f60 0x7f3f1011a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.309+0000 7f3f15800700 1 -- 172.21.15.1:0/63456824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc005040 con 0x7f3f10070870 2024-02-28T17:15:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.309+0000 7f3f14e00700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f10071f60 0x7f3f1011a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:15:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.309+0000 7f3f15800700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 0x7f3f1011a030 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f3f04002a50 tx=0x7f3f040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:39.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.310+0000 7f3f0d600700 1 -- 172.21.15.1:0/63456824 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f04007cc0 con 0x7f3f10070870 2024-02-28T17:15:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.310+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f101141b0 con 0x7f3f10070870 2024-02-28T17:15:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.310+0000 7f3f0d600700 1 -- 172.21.15.1:0/63456824 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f0400f040 con 0x7f3f10070870 2024-02-28T17:15:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.310+0000 7f3f0d600700 1 -- 172.21.15.1:0/63456824 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f04009640 con 0x7f3f10070870 2024-02-28T17:15:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.310+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f101146f0 con 0x7f3f10070870 2024-02-28T17:15:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.311+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f100fe410 con 0x7f3f10070870 2024-02-28T17:15:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.312+0000 7f3f0d600700 1 -- 172.21.15.1:0/63456824 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3f040097a0 con 0x7f3f10070870 2024-02-28T17:15:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.313+0000 7f3f0d600700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ef806d7f0 0x7f3ef806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.313+0000 7f3f0d600700 1 -- 172.21.15.1:0/63456824 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3f0408cd20 con 0x7f3f10070870 2024-02-28T17:15:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.313+0000 7f3f14e00700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ef806d7f0 0x7f3ef806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:39.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.314+0000 7f3f14e00700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ef806d7f0 0x7f3ef806fca0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3efc000c00 tx=0x7f3efc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.318+0000 7f3f0d600700 1 -- 172.21.15.1:0/63456824 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f0405a000 con 0x7f3f10070870 2024-02-28T17:15:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.628+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3f1004bed0 con 0x7f3f10070870 2024-02-28T17:15:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.629+0000 7f3f0d600700 1 -- 172.21.15.1:0/63456824 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f3f04059b90 con 0x7f3f10070870 2024-02-28T17:15:39.632 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:15:39.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:15:39.632 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:15:39.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:15:39.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:15:39.633 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:15:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.631+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ef806d7f0 msgr2=0x7f3ef806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.631+0000 7f3f18200700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ef806d7f0 0x7f3ef806fca0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3efc000c00 tx=0x7f3efc009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.631+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 msgr2=0x7f3f1011a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.631+0000 7f3f18200700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 0x7f3f1011a030 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f3f04002a50 tx=0x7f3f040046b0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.632+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 shutdown_connections 2024-02-28T17:15:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.632+0000 7f3f18200700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ef806d7f0 0x7f3ef806fca0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.632+0000 7f3f18200700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f10070870 0x7f3f1011a030 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.632+0000 7f3f18200700 1 --2- 172.21.15.1:0/63456824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f10071f60 0x7f3f1011a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.632+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 >> 172.21.15.1:0/63456824 conn(0x7f3f100f8af0 msgr2=0x7f3f100fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.632+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 shutdown_connections 2024-02-28T17:15:39.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.632+0000 7f3f18200700 1 -- 172.21.15.1:0/63456824 wait complete. 2024-02-28T17:15:39.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:39 smithi001 ceph-mon[112940]: from='client.36188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:39.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:39 smithi001 ceph-mon[112940]: from='client.36192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.788+0000 7f3614800700 1 -- 172.21.15.1:0/640770867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 msgr2=0x7f360c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.788+0000 7f3614800700 1 --2- 172.21.15.1:0/640770867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 0x7f360c103320 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f3600004940 tx=0x7f3600009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.788+0000 7f3614800700 1 -- 172.21.15.1:0/640770867 shutdown_connections 2024-02-28T17:15:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.788+0000 7f3614800700 1 --2- 172.21.15.1:0/640770867 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f360c1038f0 0x7f360c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.788+0000 7f3614800700 1 --2- 172.21.15.1:0/640770867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 0x7f360c103320 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.788+0000 7f3614800700 1 -- 172.21.15.1:0/640770867 >> 172.21.15.1:0/640770867 conn(0x7f360c0f8ae0 msgr2=0x7f360c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.788+0000 7f3614800700 1 -- 172.21.15.1:0/640770867 shutdown_connections 2024-02-28T17:15:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.788+0000 7f3614800700 1 -- 172.21.15.1:0/640770867 wait complete. 2024-02-28T17:15:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3614800700 1 Processor -- start 2024-02-28T17:15:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3614800700 1 -- start start 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3614800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 0x7f360c115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3614800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f360c1038f0 0x7f360c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3614800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1167a0 con 0x7f360c102f50 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3614800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c10faf0 con 0x7f360c1038f0 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3611e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 0x7f360c115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3611e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 0x7f360c115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59662/0 (socket says 172.21.15.1:59662) 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.789+0000 7f3611e00700 1 -- 172.21.15.1:0/1935989030 learned_addr learned my addr 172.21.15.1:0/1935989030 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.790+0000 7f3611400700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f360c1038f0 0x7f360c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.790+0000 7f3611e00700 1 -- 172.21.15.1:0/1935989030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f360c1038f0 msgr2=0x7f360c116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.790+0000 7f3611e00700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f360c1038f0 0x7f360c116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.790+0000 7f3611e00700 1 -- 172.21.15.1:0/1935989030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3600005040 con 0x7f360c102f50 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.790+0000 7f3611e00700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 0x7f360c115c10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f3600009100 tx=0x7f3600009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.790+0000 7f3606000700 1 -- 172.21.15.1:0/1935989030 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360000e040 con 0x7f360c102f50 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.791+0000 7f3606000700 1 -- 172.21.15.1:0/1935989030 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3600004290 con 0x7f360c102f50 2024-02-28T17:15:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.791+0000 7f3606000700 1 -- 172.21.15.1:0/1935989030 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360001c630 con 0x7f360c102f50 2024-02-28T17:15:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.791+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360c10fd70 con 0x7f360c102f50 2024-02-28T17:15:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.791+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360c1102b0 con 0x7f360c102f50 2024-02-28T17:15:39.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.792+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f360c0fc6e0 con 0x7f360c102f50 2024-02-28T17:15:39.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.793+0000 7f3606000700 1 -- 172.21.15.1:0/1935989030 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f360000bbf0 con 0x7f360c102f50 2024-02-28T17:15:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.794+0000 7f3606000700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35f406d7e0 0x7f35f406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.794+0000 7f3606000700 1 -- 172.21.15.1:0/1935989030 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f360008e1e0 con 0x7f360c102f50 2024-02-28T17:15:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.794+0000 7f3611400700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35f406d7e0 0x7f35f406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.795+0000 7f3611400700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35f406d7e0 0x7f35f406fc90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009fd0 tx=0x7f35fc009e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:39.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:39.799+0000 7f3606000700 1 -- 172.21.15.1:0/1935989030 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3600057a90 con 0x7f360c102f50 2024-02-28T17:15:40.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:39 smithi066 ceph-mon[85781]: from='client.36188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:40.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:39 smithi066 ceph-mon[85781]: from='client.36192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:40.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.066+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f360c111290 con 0x7f360c102f50 2024-02-28T17:15:40.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.067+0000 7f3606000700 1 -- 172.21.15.1:0/1935989030 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f360005b150 con 0x7f360c102f50 2024-02-28T17:15:40.071 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:15:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:15:40.073 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:15:40.074 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:15:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35f406d7e0 msgr2=0x7f35f406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35f406d7e0 0x7f35f406fc90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009fd0 tx=0x7f35fc009e50 comp rx=0 tx=0).stop 2024-02-28T17:15:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 msgr2=0x7f360c115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 0x7f360c115c10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f3600009100 tx=0x7f3600009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 shutdown_connections 2024-02-28T17:15:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f35f406d7e0 0x7f35f406fc90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f360c102f50 0x7f360c115c10 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 --2- 172.21.15.1:0/1935989030 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f360c1038f0 0x7f360c116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.069+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 >> 172.21.15.1:0/1935989030 conn(0x7f360c0f8ae0 msgr2=0x7f360c0fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.070+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 shutdown_connections 2024-02-28T17:15:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.070+0000 7f3614800700 1 -- 172.21.15.1:0/1935989030 wait complete. 2024-02-28T17:15:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:15:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.217+0000 7f1609400700 1 -- 172.21.15.1:0/1995272217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 msgr2=0x7f16041099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.217+0000 7f1609400700 1 --2- 172.21.15.1:0/1995272217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 0x7f16041099c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f15f4004940 tx=0x7f15f4009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.217+0000 7f1609400700 1 -- 172.21.15.1:0/1995272217 shutdown_connections 2024-02-28T17:15:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.217+0000 7f1609400700 1 --2- 172.21.15.1:0/1995272217 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1604103dc0 0x7f1604104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.217+0000 7f1609400700 1 --2- 172.21.15.1:0/1995272217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 0x7f16041099c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.217+0000 7f1609400700 1 -- 172.21.15.1:0/1995272217 >> 172.21.15.1:0/1995272217 conn(0x7f1604072d50 msgr2=0x7f1604075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.218+0000 7f1609400700 1 -- 172.21.15.1:0/1995272217 shutdown_connections 2024-02-28T17:15:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.218+0000 7f1609400700 1 -- 172.21.15.1:0/1995272217 wait complete. 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.218+0000 7f1609400700 1 Processor -- start 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.219+0000 7f1609400700 1 -- start start 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.219+0000 7f1609400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1604103dc0 0x7f160411a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.219+0000 7f1609400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 0x7f160411a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.219+0000 7f1609400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160411ac70 con 0x7f16041095f0 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.219+0000 7f1609400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604113ef0 con 0x7f1604103dc0 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.219+0000 7f1601e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 0x7f160411a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.219+0000 7f1601e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 0x7f160411a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59686/0 (socket says 172.21.15.1:59686) 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.219+0000 7f1601e00700 1 -- 172.21.15.1:0/2370491233 learned_addr learned my addr 172.21.15.1:0/2370491233 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f1602800700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1604103dc0 0x7f160411a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f1601e00700 1 -- 172.21.15.1:0/2370491233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1604103dc0 msgr2=0x7f160411a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f1601e00700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1604103dc0 0x7f160411a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f1601e00700 1 -- 172.21.15.1:0/2370491233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f4005040 con 0x7f16041095f0 2024-02-28T17:15:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f1601e00700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 0x7f160411a590 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f15f000b790 tx=0x7f15f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f15faa00700 1 -- 172.21.15.1:0/2370491233 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15f000f770 con 0x7f16041095f0 2024-02-28T17:15:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f15faa00700 1 -- 172.21.15.1:0/2370491233 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15f000fda0 con 0x7f16041095f0 2024-02-28T17:15:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1604114170 con 0x7f16041095f0 2024-02-28T17:15:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.220+0000 7f15faa00700 1 -- 172.21.15.1:0/2370491233 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15f000d310 con 0x7f16041095f0 2024-02-28T17:15:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.221+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16041146b0 con 0x7f16041095f0 2024-02-28T17:15:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.222+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1604075870 con 0x7f16041095f0 2024-02-28T17:15:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.223+0000 7f15faa00700 1 -- 172.21.15.1:0/2370491233 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f15f0010420 con 0x7f16041095f0 2024-02-28T17:15:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.224+0000 7f15faa00700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15e806d7f0 0x7f15e806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.224+0000 7f15faa00700 1 -- 172.21.15.1:0/2370491233 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f15f008ceb0 con 0x7f16041095f0 2024-02-28T17:15:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.224+0000 7f1602800700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15e806d7f0 0x7f15e806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:40.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.225+0000 7f1602800700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15e806d7f0 0x7f15e806fca0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f15f400b3b0 tx=0x7f15f4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:40.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.229+0000 7f15faa00700 1 -- 172.21.15.1:0/2370491233 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f15f005a290 con 0x7f16041095f0 2024-02-28T17:15:40.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.458+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1604115740 con 0x7f15e806d7f0 2024-02-28T17:15:40.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.460+0000 7f15faa00700 1 -- 172.21.15.1:0/2370491233 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1604115740 con 0x7f15e806d7f0 2024-02-28T17:15:40.463 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:15:40.463 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:15:40.463 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:15:40.464 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:15:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.462+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15e806d7f0 msgr2=0x7f15e806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.462+0000 7f1609400700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15e806d7f0 0x7f15e806fca0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f15f400b3b0 tx=0x7f15f4009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 msgr2=0x7f160411a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 0x7f160411a590 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f15f000b790 tx=0x7f15f0009250 comp rx=0 tx=0).stop 2024-02-28T17:15:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 shutdown_connections 2024-02-28T17:15:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f15e806d7f0 0x7f15e806fca0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1604103dc0 0x7f160411a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 --2- 172.21.15.1:0/2370491233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16041095f0 0x7f160411a590 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 >> 172.21.15.1:0/2370491233 conn(0x7f1604072d50 msgr2=0x7f16040746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 shutdown_connections 2024-02-28T17:15:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.463+0000 7f1609400700 1 -- 172.21.15.1:0/2370491233 wait complete. 2024-02-28T17:15:40.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.619+0000 7fb209e00700 1 -- 172.21.15.1:0/374192155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105b00 msgr2=0x7fb2040ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.619+0000 7fb209e00700 1 --2- 172.21.15.1:0/374192155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105b00 0x7fb2040ff170 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0004940 tx=0x7fb1f0009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:40.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.619+0000 7fb209e00700 1 -- 172.21.15.1:0/374192155 shutdown_connections 2024-02-28T17:15:40.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.619+0000 7fb209e00700 1 --2- 172.21.15.1:0/374192155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105b00 0x7fb2040ff170 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.619+0000 7fb209e00700 1 --2- 172.21.15.1:0/374192155 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb204105160 0x7fb204105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.619+0000 7fb209e00700 1 -- 172.21.15.1:0/374192155 >> 172.21.15.1:0/374192155 conn(0x7fb2040facb0 msgr2=0x7fb2040fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:40.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.619+0000 7fb209e00700 1 -- 172.21.15.1:0/374192155 shutdown_connections 2024-02-28T17:15:40.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.620+0000 7fb209e00700 1 -- 172.21.15.1:0/374192155 wait complete. 2024-02-28T17:15:40.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.620+0000 7fb209e00700 1 Processor -- start 2024-02-28T17:15:40.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.620+0000 7fb209e00700 1 -- start start 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.621+0000 7fb209e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105160 0x7fb204115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.621+0000 7fb209e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb204105b00 0x7fb2041161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.621+0000 7fb209e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb204116820 con 0x7fb204105160 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.621+0000 7fb209e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20410fb50 con 0x7fb204105b00 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.621+0000 7fb202a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105160 0x7fb204115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.621+0000 7fb202a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105160 0x7fb204115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59716/0 (socket says 172.21.15.1:59716) 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.621+0000 7fb202a00700 1 -- 172.21.15.1:0/2332217690 learned_addr learned my addr 172.21.15.1:0/2332217690 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb202a00700 1 -- 172.21.15.1:0/2332217690 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb204105b00 msgr2=0x7fb2041161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb202a00700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb204105b00 0x7fb2041161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb202a00700 1 -- 172.21.15.1:0/2332217690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f0005040 con 0x7fb204105160 2024-02-28T17:15:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb202a00700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105160 0x7fb204115c90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4002a50 tx=0x7fb1f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb1faa00700 1 -- 172.21.15.1:0/2332217690 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f4007cc0 con 0x7fb204105160 2024-02-28T17:15:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb1faa00700 1 -- 172.21.15.1:0/2332217690 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1f400f040 con 0x7fb204105160 2024-02-28T17:15:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb1faa00700 1 -- 172.21.15.1:0/2332217690 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f4009640 con 0x7fb204105160 2024-02-28T17:15:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20410fe30 con 0x7fb204105160 2024-02-28T17:15:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.622+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb204110370 con 0x7fb204105160 2024-02-28T17:15:40.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.623+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb204107630 con 0x7fb204105160 2024-02-28T17:15:40.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.624+0000 7fb1faa00700 1 -- 172.21.15.1:0/2332217690 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb1f40097c0 con 0x7fb204105160 2024-02-28T17:15:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.625+0000 7fb1faa00700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb1e806d7f0 0x7fb1e806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:15:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.625+0000 7fb1faa00700 1 -- 172.21.15.1:0/2332217690 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb1f408cc20 con 0x7fb204105160 2024-02-28T17:15:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.626+0000 7fb202000700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb1e806d7f0 0x7fb1e806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:15:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.626+0000 7fb202000700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb1e806d7f0 0x7fb1e806fca0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0004f90 tx=0x7fb1f0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:15:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.630+0000 7fb1faa00700 1 -- 172.21.15.1:0/2332217690 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb1f4059f80 con 0x7fb204105160 2024-02-28T17:15:40.784 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:40 smithi001 ceph-mon[112940]: from='client.36196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:40.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:40 smithi001 ceph-mon[112940]: pgmap v1240: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:40.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/63456824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:15:40.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1935989030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:15:40.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.937+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb20410ffc0 con 0x7fb204105160 2024-02-28T17:15:40.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.937+0000 7fb1faa00700 1 -- 172.21.15.1:0/2332217690 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fb20410ffc0 con 0x7fb204105160 2024-02-28T17:15:40.941 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:15:40.941 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:15:40.941 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:15:40.941 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:15:40.941 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:15:40.941 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:15:40.941 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:15:40.942 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:15:40.943 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:15:40.943 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:15:40.943 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:15:40.943 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:15:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb1e806d7f0 msgr2=0x7fb1e806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb1e806d7f0 0x7fb1e806fca0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0004f90 tx=0x7fb1f0009e30 comp rx=0 tx=0).stop 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105160 msgr2=0x7fb204115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105160 0x7fb204115c90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4002a50 tx=0x7fb1f40046b0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 shutdown_connections 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb1e806d7f0 0x7fb1e806fca0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb204105160 0x7fb204115c90 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 --2- 172.21.15.1:0/2332217690 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb204105b00 0x7fb2041161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.940+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 >> 172.21.15.1:0/2332217690 conn(0x7fb2040facb0 msgr2=0x7fb2040fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.941+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 shutdown_connections 2024-02-28T17:15:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:15:40.941+0000 7fb209e00700 1 -- 172.21.15.1:0/2332217690 wait complete. 2024-02-28T17:15:41.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:40 smithi066 ceph-mon[85781]: from='client.36196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:41.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:40 smithi066 ceph-mon[85781]: pgmap v1240: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:41.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/63456824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:15:41.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1935989030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:15:41.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:41 smithi066 ceph-mon[85781]: from='client.36208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:41.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:15:41.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:41 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2332217690' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:15:42.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:41 smithi001 ceph-mon[112940]: from='client.36208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:15:42.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:15:42.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:41 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2332217690' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:15:43.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:42 smithi066 ceph-mon[85781]: pgmap v1241: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:43.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:42 smithi001 ceph-mon[112940]: pgmap v1241: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:45.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:44 smithi066 ceph-mon[85781]: pgmap v1242: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:45.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:44 smithi001 ceph-mon[112940]: pgmap v1242: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:47.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:46 smithi066 ceph-mon[85781]: pgmap v1243: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:47.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:46 smithi001 ceph-mon[112940]: pgmap v1243: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:49.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:48 smithi066 ceph-mon[85781]: pgmap v1244: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:49.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:48 smithi001 ceph-mon[112940]: pgmap v1244: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:51.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:50 smithi066 ceph-mon[85781]: pgmap v1245: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:51.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:50 smithi001 ceph-mon[112940]: pgmap v1245: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:53.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:52 smithi066 ceph-mon[85781]: pgmap v1246: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:53.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:52 smithi001 ceph-mon[112940]: pgmap v1246: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:55.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:54 smithi066 ceph-mon[85781]: pgmap v1247: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:55.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:54 smithi001 ceph-mon[112940]: pgmap v1247: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:57.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:56 smithi066 ceph-mon[85781]: pgmap v1248: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:57.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:15:57.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:56 smithi001 ceph-mon[112940]: pgmap v1248: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:57.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:15:59.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:15:58 smithi066 ceph-mon[85781]: pgmap v1249: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:15:59.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:15:58 smithi001 ceph-mon[112940]: pgmap v1249: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:01.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:00 smithi066 ceph-mon[85781]: pgmap v1250: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:01.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:00 smithi001 ceph-mon[112940]: pgmap v1250: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:03.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:02 smithi066 ceph-mon[85781]: pgmap v1251: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:02 smithi001 ceph-mon[112940]: pgmap v1251: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:05.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:04 smithi066 ceph-mon[85781]: pgmap v1252: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:05.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:04 smithi001 ceph-mon[112940]: pgmap v1252: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:07.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:06 smithi066 ceph-mon[85781]: pgmap v1253: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:07.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:06 smithi001 ceph-mon[112940]: pgmap v1253: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:09.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:08 smithi066 ceph-mon[85781]: pgmap v1254: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:09.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:08 smithi001 ceph-mon[112940]: pgmap v1254: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:11.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:10 smithi066 ceph-mon[85781]: pgmap v1255: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:11.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:10 smithi001 ceph-mon[112940]: pgmap v1255: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:11.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.107+0000 7f0b1f600700 1 -- 172.21.15.1:0/652976860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18103dc0 msgr2=0x7f0b18104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.107+0000 7f0b1f600700 1 --2- 172.21.15.1:0/652976860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18103dc0 0x7f0b18104210 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0b04004940 tx=0x7f0b04009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:11.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.107+0000 7f0b1f600700 1 -- 172.21.15.1:0/652976860 shutdown_connections 2024-02-28T17:16:11.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.107+0000 7f0b1f600700 1 --2- 172.21.15.1:0/652976860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b18103dc0 0x7f0b18104210 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.107+0000 7f0b1f600700 1 --2- 172.21.15.1:0/652976860 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b181095f0 0x7f0b181099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.107+0000 7f0b1f600700 1 -- 172.21.15.1:0/652976860 >> 172.21.15.1:0/652976860 conn(0x7f0b18072d50 msgr2=0x7f0b18075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:11.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.107+0000 7f0b1f600700 1 -- 172.21.15.1:0/652976860 shutdown_connections 2024-02-28T17:16:11.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.108+0000 7f0b1f600700 1 -- 172.21.15.1:0/652976860 wait complete. 2024-02-28T17:16:11.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.108+0000 7f0b1f600700 1 Processor -- start 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.108+0000 7f0b1f600700 1 -- start start 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.109+0000 7f0b1f600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18103dc0 0x7f0b1811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.109+0000 7f0b1f600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b181095f0 0x7f0b1811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.109+0000 7f0b1f600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1811abd0 con 0x7f0b181095f0 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.109+0000 7f0b1f600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b18113ee0 con 0x7f0b18103dc0 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.109+0000 7f0b17e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b181095f0 0x7f0b1811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.109+0000 7f0b17e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b181095f0 0x7f0b1811a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40856/0 (socket says 172.21.15.1:40856) 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.109+0000 7f0b17e00700 1 -- 172.21.15.1:0/231341111 learned_addr learned my addr 172.21.15.1:0/231341111 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.109+0000 7f0b1cc00700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18103dc0 0x7f0b1811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.110+0000 7f0b17e00700 1 -- 172.21.15.1:0/231341111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18103dc0 msgr2=0x7f0b1811a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.110+0000 7f0b17e00700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18103dc0 0x7f0b1811a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.110+0000 7f0b17e00700 1 -- 172.21.15.1:0/231341111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04005040 con 0x7f0b181095f0 2024-02-28T17:16:11.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.110+0000 7f0b17e00700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b181095f0 0x7f0b1811a580 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009150 tx=0x7f0b04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:11.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.110+0000 7f0b14c00700 1 -- 172.21.15.1:0/231341111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b0400e040 con 0x7f0b181095f0 2024-02-28T17:16:11.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.110+0000 7f0b14c00700 1 -- 172.21.15.1:0/231341111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b0400bbf0 con 0x7f0b181095f0 2024-02-28T17:16:11.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.110+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b18114160 con 0x7f0b181095f0 2024-02-28T17:16:11.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.110+0000 7f0b14c00700 1 -- 172.21.15.1:0/231341111 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b0401c630 con 0x7f0b181095f0 2024-02-28T17:16:11.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.111+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b181146a0 con 0x7f0b181095f0 2024-02-28T17:16:11.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.112+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1810b9c0 con 0x7f0b181095f0 2024-02-28T17:16:11.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.113+0000 7f0b14c00700 1 -- 172.21.15.1:0/231341111 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0b0400bd60 con 0x7f0b181095f0 2024-02-28T17:16:11.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.114+0000 7f0b14c00700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7f0 0x7f0b0006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.114+0000 7f0b14c00700 1 -- 172.21.15.1:0/231341111 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0b0408e290 con 0x7f0b181095f0 2024-02-28T17:16:11.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.114+0000 7f0b1cc00700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7f0 0x7f0b0006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.115+0000 7f0b1cc00700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7f0 0x7f0b0006fca0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c004990 tx=0x7f0b0c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:11.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.119+0000 7f0b14c00700 1 -- 172.21.15.1:0/231341111 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0b0405b670 con 0x7f0b181095f0 2024-02-28T17:16:11.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.366+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b1804bf40 con 0x7f0b0006d7f0 2024-02-28T17:16:11.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.368+0000 7f0b14c00700 1 -- 172.21.15.1:0/231341111 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0b1804bf40 con 0x7f0b0006d7f0 2024-02-28T17:16:11.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.371+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7f0 msgr2=0x7f0b0006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.371+0000 7f0b1f600700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7f0 0x7f0b0006fca0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c004990 tx=0x7f0b0c009ee0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.371+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b181095f0 msgr2=0x7f0b1811a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.371+0000 7f0b1f600700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b181095f0 0x7f0b1811a580 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009150 tx=0x7f0b04009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:11.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.372+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 shutdown_connections 2024-02-28T17:16:11.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.372+0000 7f0b1f600700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b0006d7f0 0x7f0b0006fca0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.372+0000 7f0b1f600700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0b18103dc0 0x7f0b1811a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.372+0000 7f0b1f600700 1 --2- 172.21.15.1:0/231341111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b181095f0 0x7f0b1811a580 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.372+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 >> 172.21.15.1:0/231341111 conn(0x7f0b18072d50 msgr2=0x7f0b18074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.372+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 shutdown_connections 2024-02-28T17:16:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.372+0000 7f0b1f600700 1 -- 172.21.15.1:0/231341111 wait complete. 2024-02-28T17:16:11.393 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:16:11.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.533+0000 7f4674200700 1 -- 172.21.15.1:0/699480036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105170 msgr2=0x7f466c105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.533+0000 7f4674200700 1 --2- 172.21.15.1:0/699480036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105170 0x7f466c105540 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4660004940 tx=0x7f4660009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:11.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.534+0000 7f4674200700 1 -- 172.21.15.1:0/699480036 shutdown_connections 2024-02-28T17:16:11.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.534+0000 7f4674200700 1 --2- 172.21.15.1:0/699480036 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f466c105b10 0x7f466c0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.534+0000 7f4674200700 1 --2- 172.21.15.1:0/699480036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105170 0x7f466c105540 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.534+0000 7f4674200700 1 -- 172.21.15.1:0/699480036 >> 172.21.15.1:0/699480036 conn(0x7f466c0face0 msgr2=0x7f466c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:11.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.534+0000 7f4674200700 1 -- 172.21.15.1:0/699480036 shutdown_connections 2024-02-28T17:16:11.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.534+0000 7f4674200700 1 -- 172.21.15.1:0/699480036 wait complete. 2024-02-28T17:16:11.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.535+0000 7f4674200700 1 Processor -- start 2024-02-28T17:16:11.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.535+0000 7f4674200700 1 -- start start 2024-02-28T17:16:11.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.535+0000 7f4674200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f466c105170 0x7f466c115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.535+0000 7f4674200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105b10 0x7f466c116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.535+0000 7f4674200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c1168e0 con 0x7f466c105b10 2024-02-28T17:16:11.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.536+0000 7f4674200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c10fbf0 con 0x7f466c105170 2024-02-28T17:16:11.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.536+0000 7f4670e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105b10 0x7f466c116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.536+0000 7f4670e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105b10 0x7f466c116290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40874/0 (socket says 172.21.15.1:40874) 2024-02-28T17:16:11.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.536+0000 7f4670e00700 1 -- 172.21.15.1:0/1415438477 learned_addr learned my addr 172.21.15.1:0/1415438477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:11.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.536+0000 7f4671800700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f466c105170 0x7f466c115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.536+0000 7f4670e00700 1 -- 172.21.15.1:0/1415438477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f466c105170 msgr2=0x7f466c115d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.536+0000 7f4670e00700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f466c105170 0x7f466c115d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.536+0000 7f4670e00700 1 -- 172.21.15.1:0/1415438477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4660005040 con 0x7f466c105b10 2024-02-28T17:16:11.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.537+0000 7f4670e00700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105b10 0x7f466c116290 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4658002a80 tx=0x7f4658004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:11.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.537+0000 7f4669600700 1 -- 172.21.15.1:0/1415438477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4658007cc0 con 0x7f466c105b10 2024-02-28T17:16:11.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.537+0000 7f4669600700 1 -- 172.21.15.1:0/1415438477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f465800f040 con 0x7f466c105b10 2024-02-28T17:16:11.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.537+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c10fed0 con 0x7f466c105b10 2024-02-28T17:16:11.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.537+0000 7f4669600700 1 -- 172.21.15.1:0/1415438477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f465800daf0 con 0x7f466c105b10 2024-02-28T17:16:11.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.537+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c110410 con 0x7f466c105b10 2024-02-28T17:16:11.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.538+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466c102480 con 0x7f466c105b10 2024-02-28T17:16:11.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.539+0000 7f4669600700 1 -- 172.21.15.1:0/1415438477 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f465800dc90 con 0x7f466c105b10 2024-02-28T17:16:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.542+0000 7f4669600700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f465406d7f0 0x7f465406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.542+0000 7f4669600700 1 -- 172.21.15.1:0/1415438477 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f465808c660 con 0x7f466c105b10 2024-02-28T17:16:11.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.542+0000 7f4671800700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f465406d7f0 0x7f465406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.543+0000 7f4671800700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f465406d7f0 0x7f465406fca0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f466000b3b0 tx=0x7f4660009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:11.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.547+0000 7f4669600700 1 -- 172.21.15.1:0/1415438477 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4658059940 con 0x7f466c105b10 2024-02-28T17:16:11.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.799+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f466c111330 con 0x7f465406d7f0 2024-02-28T17:16:11.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.802+0000 7f4669600700 1 -- 172.21.15.1:0/1415438477 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f466c111330 con 0x7f465406d7f0 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f465406d7f0 msgr2=0x7f465406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f465406d7f0 0x7f465406fca0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f466000b3b0 tx=0x7f4660009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105b10 msgr2=0x7f466c116290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105b10 0x7f466c116290 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4658002a80 tx=0x7f4658004630 comp rx=0 tx=0).stop 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 shutdown_connections 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f465406d7f0 0x7f465406fca0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f466c105170 0x7f466c115d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 --2- 172.21.15.1:0/1415438477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f466c105b10 0x7f466c116290 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 >> 172.21.15.1:0/1415438477 conn(0x7f466c0face0 msgr2=0x7f466c0fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:11.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 shutdown_connections 2024-02-28T17:16:11.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.806+0000 7f4674200700 1 -- 172.21.15.1:0/1415438477 wait complete. 2024-02-28T17:16:11.914 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:16:11.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.968+0000 7fc28aa00700 1 -- 172.21.15.1:0/1487432727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 msgr2=0x7fc2840fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.968+0000 7fc28aa00700 1 --2- 172.21.15.1:0/1487432727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 0x7fc2840fe350 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc274004940 tx=0x7fc274009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:11.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.968+0000 7fc28aa00700 1 -- 172.21.15.1:0/1487432727 shutdown_connections 2024-02-28T17:16:11.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.968+0000 7fc28aa00700 1 --2- 172.21.15.1:0/1487432727 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2840fe920 0x7fc284106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.968+0000 7fc28aa00700 1 --2- 172.21.15.1:0/1487432727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 0x7fc2840fe350 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.968+0000 7fc28aa00700 1 -- 172.21.15.1:0/1487432727 >> 172.21.15.1:0/1487432727 conn(0x7fc2840f79e0 msgr2=0x7fc2840f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:11.973 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:16:11.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.968+0000 7fc28aa00700 1 -- 172.21.15.1:0/1487432727 shutdown_connections 2024-02-28T17:16:11.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.968+0000 7fc28aa00700 1 -- 172.21.15.1:0/1487432727 wait complete. 2024-02-28T17:16:11.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.969+0000 7fc28aa00700 1 Processor -- start 2024-02-28T17:16:11.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.969+0000 7fc28aa00700 1 -- start start 2024-02-28T17:16:11.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.970+0000 7fc28aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 0x7fc2840fbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.970+0000 7fc28aa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2840fe920 0x7fc2840fc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.970+0000 7fc28aa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc284111950 con 0x7fc2840fdf80 2024-02-28T17:16:11.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.970+0000 7fc28aa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc284111ac0 con 0x7fc2840fe920 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.970+0000 7fc283e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 0x7fc2840fbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.970+0000 7fc283e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 0x7fc2840fbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59262/0 (socket says 172.21.15.1:59262) 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.970+0000 7fc283e00700 1 -- 172.21.15.1:0/751184173 learned_addr learned my addr 172.21.15.1:0/751184173 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.970+0000 7fc27be00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2840fe920 0x7fc2840fc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc283e00700 1 -- 172.21.15.1:0/751184173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2840fe920 msgr2=0x7fc2840fc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc283e00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2840fe920 0x7fc2840fc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc283e00700 1 -- 172.21.15.1:0/751184173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc274005040 con 0x7fc2840fdf80 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc27be00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2840fe920 0x7fc2840fc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc283e00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 0x7fc2840fbe40 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc274000c00 tx=0x7fc274009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc280c00700 1 -- 172.21.15.1:0/751184173 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc27400e040 con 0x7fc2840fdf80 2024-02-28T17:16:11.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc280c00700 1 -- 172.21.15.1:0/751184173 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc27400bb90 con 0x7fc2840fdf80 2024-02-28T17:16:11.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc280c00700 1 -- 172.21.15.1:0/751184173 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc274013ea0 con 0x7fc2840fdf80 2024-02-28T17:16:11.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2840fc9b0 con 0x7fc2840fdf80 2024-02-28T17:16:11.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.971+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28411e650 con 0x7fc2840fdf80 2024-02-28T17:16:11.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.973+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2841046b0 con 0x7fc2840fdf80 2024-02-28T17:16:11.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.974+0000 7fc280c00700 1 -- 172.21.15.1:0/751184173 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc27400bd00 con 0x7fc2840fdf80 2024-02-28T17:16:11.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.975+0000 7fc280c00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc26806d7f0 0x7fc26806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:11.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.975+0000 7fc280c00700 1 -- 172.21.15.1:0/751184173 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc27408e2c0 con 0x7fc2840fdf80 2024-02-28T17:16:11.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.975+0000 7fc27be00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc26806d7f0 0x7fc26806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:11.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.976+0000 7fc27be00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc26806d7f0 0x7fc26806fca0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc26c007fd0 tx=0x7fc26c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:11.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:11.980+0000 7fc280c00700 1 -- 172.21.15.1:0/751184173 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc274057af0 con 0x7fc2840fdf80 2024-02-28T17:16:12.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.220+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2840fd580 con 0x7fc26806d7f0 2024-02-28T17:16:12.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.249+0000 7fc280c00700 1 -- 172.21.15.1:0/751184173 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc2840fd580 con 0x7fc26806d7f0 2024-02-28T17:16:12.253 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (42m) 3m ago 56m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (39m) 3m ago 56m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (38m) 3m ago 55m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (41m) 3m ago 56m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (34m) 3m ago 49m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (34m) 3m ago 49m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (3m) 3m ago 49m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (19m) 3m ago 49m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (44m) 3m ago 58m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (43m) 3m ago 55m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (40m) 3m ago 58m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (40m) 3m ago 55m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (43m) 3m ago 56m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:16:12.254 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (43m) 3m ago 55m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:16:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (38m) 3m ago 54m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:16:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (37m) 3m ago 53m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:16:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (37m) 3m ago 53m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:16:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (36m) 3m ago 52m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:16:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (36m) 3m ago 51m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:16:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (35m) 3m ago 51m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:16:12.255 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (42m) 3m ago 55m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:16:12.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.251+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc26806d7f0 msgr2=0x7fc26806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:12.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc26806d7f0 0x7fc26806fca0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc26c007fd0 tx=0x7fc26c009040 comp rx=0 tx=0).stop 2024-02-28T17:16:12.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 msgr2=0x7fc2840fbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:12.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 0x7fc2840fbe40 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc274000c00 tx=0x7fc274009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 shutdown_connections 2024-02-28T17:16:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc26806d7f0 0x7fc26806fca0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2840fdf80 0x7fc2840fbe40 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 --2- 172.21.15.1:0/751184173 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc2840fe920 0x7fc2840fc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 >> 172.21.15.1:0/751184173 conn(0x7fc2840f79e0 msgr2=0x7fc284101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 shutdown_connections 2024-02-28T17:16:12.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.252+0000 7fc28aa00700 1 -- 172.21.15.1:0/751184173 wait complete. 2024-02-28T17:16:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.414+0000 7fc3abe00700 1 -- 172.21.15.1:0/4282316567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fdd70 msgr2=0x7fc3ac0fe140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.414+0000 7fc3abe00700 1 --2- 172.21.15.1:0/4282316567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fdd70 0x7fc3ac0fe140 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc39c004940 tx=0x7fc39c009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.414+0000 7fc3abe00700 1 -- 172.21.15.1:0/4282316567 shutdown_connections 2024-02-28T17:16:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.414+0000 7fc3abe00700 1 --2- 172.21.15.1:0/4282316567 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3ac0fe710 0x7fc3ac106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.414+0000 7fc3abe00700 1 --2- 172.21.15.1:0/4282316567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fdd70 0x7fc3ac0fe140 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.414+0000 7fc3abe00700 1 -- 172.21.15.1:0/4282316567 >> 172.21.15.1:0/4282316567 conn(0x7fc3ac0f77c0 msgr2=0x7fc3ac0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.414+0000 7fc3abe00700 1 -- 172.21.15.1:0/4282316567 shutdown_connections 2024-02-28T17:16:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.414+0000 7fc3abe00700 1 -- 172.21.15.1:0/4282316567 wait complete. 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.415+0000 7fc3abe00700 1 Processor -- start 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.415+0000 7fc3abe00700 1 -- start start 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.416+0000 7fc3abe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3ac0fdd70 0x7fc3ac115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.416+0000 7fc3abe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fe710 0x7fc3ac115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.416+0000 7fc3abe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac116600 con 0x7fc3ac0fe710 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.416+0000 7fc3abe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac10f930 con 0x7fc3ac0fdd70 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.416+0000 7fc3aa000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fe710 0x7fc3ac115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.416+0000 7fc3aa000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fe710 0x7fc3ac115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59284/0 (socket says 172.21.15.1:59284) 2024-02-28T17:16:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.416+0000 7fc3aa000700 1 -- 172.21.15.1:0/32392144 learned_addr learned my addr 172.21.15.1:0/32392144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.416+0000 7fc3aaa00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3ac0fdd70 0x7fc3ac115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.417+0000 7fc3aa000700 1 -- 172.21.15.1:0/32392144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3ac0fdd70 msgr2=0x7fc3ac115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.417+0000 7fc3aa000700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3ac0fdd70 0x7fc3ac115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.417+0000 7fc3aa000700 1 -- 172.21.15.1:0/32392144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c005040 con 0x7fc3ac0fe710 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.417+0000 7fc3aa000700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fe710 0x7fc3ac115fb0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc398002a50 tx=0x7fc3980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.417+0000 7fc3a2a00700 1 -- 172.21.15.1:0/32392144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc398007cc0 con 0x7fc3ac0fe710 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.417+0000 7fc3a2a00700 1 -- 172.21.15.1:0/32392144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc39800f040 con 0x7fc3ac0fe710 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.417+0000 7fc3a2a00700 1 -- 172.21.15.1:0/32392144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc398009640 con 0x7fc3ac0fe710 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.417+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ac10fc10 con 0x7fc3ac0fe710 2024-02-28T17:16:12.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.418+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac110150 con 0x7fc3ac0fe710 2024-02-28T17:16:12.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.419+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ac106760 con 0x7fc3ac0fe710 2024-02-28T17:16:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.420+0000 7fc3a2a00700 1 -- 172.21.15.1:0/32392144 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc3980097a0 con 0x7fc3ac0fe710 2024-02-28T17:16:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.421+0000 7fc3a2a00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc39006d7e0 0x7fc39006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.421+0000 7fc3a2a00700 1 -- 172.21.15.1:0/32392144 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc39808cbc0 con 0x7fc3ac0fe710 2024-02-28T17:16:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.421+0000 7fc3aaa00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc39006d7e0 0x7fc39006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:12.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.422+0000 7fc3aaa00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc39006d7e0 0x7fc39006fc90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc39c004f90 tx=0x7fc39c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:12.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.426+0000 7fc3a2a00700 1 -- 172.21.15.1:0/32392144 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc398059fc0 con 0x7fc3ac0fe710 2024-02-28T17:16:12.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.745+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc3ac05f5c0 con 0x7fc3ac0fe710 2024-02-28T17:16:12.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.746+0000 7fc3a2a00700 1 -- 172.21.15.1:0/32392144 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fc398059b50 con 0x7fc3ac0fe710 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:16:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:16:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:16:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:16:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:16:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:16:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:16:12.752 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc39006d7e0 msgr2=0x7fc39006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc39006d7e0 0x7fc39006fc90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc39c004f90 tx=0x7fc39c009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fe710 msgr2=0x7fc3ac115fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fe710 0x7fc3ac115fb0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc398002a50 tx=0x7fc3980046b0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 shutdown_connections 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc39006d7e0 0x7fc39006fc90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc3ac0fdd70 0x7fc3ac115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 --2- 172.21.15.1:0/32392144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3ac0fe710 0x7fc3ac115fb0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.749+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 >> 172.21.15.1:0/32392144 conn(0x7fc3ac0f77c0 msgr2=0x7fc3ac1038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.750+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 shutdown_connections 2024-02-28T17:16:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.750+0000 7fc3abe00700 1 -- 172.21.15.1:0/32392144 wait complete. 2024-02-28T17:16:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.912+0000 7fd230400700 1 -- 172.21.15.1:0/3396554392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228071f60 msgr2=0x7fd228070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.912+0000 7fd230400700 1 --2- 172.21.15.1:0/3396554392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228071f60 0x7fd228070330 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fd21c004970 tx=0x7fd21c009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.912+0000 7fd230400700 1 -- 172.21.15.1:0/3396554392 shutdown_connections 2024-02-28T17:16:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.912+0000 7fd230400700 1 --2- 172.21.15.1:0/3396554392 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd228070870 0x7fd228070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.912+0000 7fd230400700 1 --2- 172.21.15.1:0/3396554392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228071f60 0x7fd228070330 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.912+0000 7fd230400700 1 -- 172.21.15.1:0/3396554392 >> 172.21.15.1:0/3396554392 conn(0x7fd2280f8ae0 msgr2=0x7fd2280faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.912+0000 7fd230400700 1 -- 172.21.15.1:0/3396554392 shutdown_connections 2024-02-28T17:16:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.912+0000 7fd230400700 1 -- 172.21.15.1:0/3396554392 wait complete. 2024-02-28T17:16:12.917 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:12 smithi001 ceph-mon[112940]: from='client.36216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:12.918 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:12 smithi001 ceph-mon[112940]: pgmap v1256: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:12.918 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:12 smithi001 ceph-mon[112940]: from='client.36220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.913+0000 7fd230400700 1 Processor -- start 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.913+0000 7fd230400700 1 -- start start 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.914+0000 7fd230400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228070870 0x7fd22811a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.914+0000 7fd230400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd228071f60 0x7fd22811a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.914+0000 7fd230400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22811ac00 con 0x7fd228070870 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.914+0000 7fd230400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd228113f10 con 0x7fd228071f60 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.914+0000 7fd22da00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228070870 0x7fd22811a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.914+0000 7fd22da00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228070870 0x7fd22811a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59306/0 (socket says 172.21.15.1:59306) 2024-02-28T17:16:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.914+0000 7fd22da00700 1 -- 172.21.15.1:0/1353838196 learned_addr learned my addr 172.21.15.1:0/1353838196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd22d000700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd228071f60 0x7fd22811a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd22da00700 1 -- 172.21.15.1:0/1353838196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd228071f60 msgr2=0x7fd22811a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd22da00700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd228071f60 0x7fd22811a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd22da00700 1 -- 172.21.15.1:0/1353838196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c005040 con 0x7fd228070870 2024-02-28T17:16:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd22da00700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228070870 0x7fd22811a070 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd21c004f50 tx=0x7fd21c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd225600700 1 -- 172.21.15.1:0/1353838196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd21c00e040 con 0x7fd228070870 2024-02-28T17:16:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd225600700 1 -- 172.21.15.1:0/1353838196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd21c00bc10 con 0x7fd228070870 2024-02-28T17:16:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd228114190 con 0x7fd228070870 2024-02-28T17:16:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.915+0000 7fd225600700 1 -- 172.21.15.1:0/1353838196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd21c01c630 con 0x7fd228070870 2024-02-28T17:16:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.916+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2281146d0 con 0x7fd228070870 2024-02-28T17:16:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.917+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2280fe400 con 0x7fd228070870 2024-02-28T17:16:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.918+0000 7fd225600700 1 -- 172.21.15.1:0/1353838196 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd21c00bd80 con 0x7fd228070870 2024-02-28T17:16:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.919+0000 7fd225600700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd21006d7f0 0x7fd21006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.919+0000 7fd225600700 1 -- 172.21.15.1:0/1353838196 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd21c08e2a0 con 0x7fd228070870 2024-02-28T17:16:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.920+0000 7fd22d000700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd21006d7f0 0x7fd21006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.921+0000 7fd22d000700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd21006d7f0 0x7fd21006fca0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd228115910 tx=0x7fd214009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:12.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:12.925+0000 7fd225600700 1 -- 172.21.15.1:0/1353838196 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd21c05b680 con 0x7fd228070870 2024-02-28T17:16:13.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:12 smithi066 ceph-mon[85781]: from='client.36216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:13.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:12 smithi066 ceph-mon[85781]: pgmap v1256: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:13.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:12 smithi066 ceph-mon[85781]: from='client.36220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.197+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd22804bed0 con 0x7fd228070870 2024-02-28T17:16:13.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.198+0000 7fd225600700 1 -- 172.21.15.1:0/1353838196 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fd21c05b210 con 0x7fd228070870 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:16:13.203 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:16:13.204 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:16:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:16:13.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:16:13.206 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:16:13.206 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:16:13.206 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:16:13.206 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd21006d7f0 msgr2=0x7fd21006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd21006d7f0 0x7fd21006fca0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd228115910 tx=0x7fd214009ee0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228070870 msgr2=0x7fd22811a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228070870 0x7fd22811a070 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd21c004f50 tx=0x7fd21c009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 shutdown_connections 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd21006d7f0 0x7fd21006fca0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd228070870 0x7fd22811a070 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 --2- 172.21.15.1:0/1353838196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd228071f60 0x7fd22811a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 >> 172.21.15.1:0/1353838196 conn(0x7fd2280f8ae0 msgr2=0x7fd2280fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 shutdown_connections 2024-02-28T17:16:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.201+0000 7fd230400700 1 -- 172.21.15.1:0/1353838196 wait complete. 2024-02-28T17:16:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:16:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.364+0000 7f5246400700 1 -- 172.21.15.1:0/2325495433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52400724c0 msgr2=0x7f5240072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.364+0000 7f5246400700 1 --2- 172.21.15.1:0/2325495433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52400724c0 0x7f5240072890 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5234004940 tx=0x7f5234009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.364+0000 7f5246400700 1 -- 172.21.15.1:0/2325495433 shutdown_connections 2024-02-28T17:16:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.364+0000 7f5246400700 1 --2- 172.21.15.1:0/2325495433 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5240072dd0 0x7f524010d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.364+0000 7f5246400700 1 --2- 172.21.15.1:0/2325495433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52400724c0 0x7f5240072890 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.364+0000 7f5246400700 1 -- 172.21.15.1:0/2325495433 >> 172.21.15.1:0/2325495433 conn(0x7f52400fab10 msgr2=0x7f52400fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.364+0000 7f5246400700 1 -- 172.21.15.1:0/2325495433 shutdown_connections 2024-02-28T17:16:13.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.364+0000 7f5246400700 1 -- 172.21.15.1:0/2325495433 wait complete. 2024-02-28T17:16:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.365+0000 7f5246400700 1 Processor -- start 2024-02-28T17:16:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.365+0000 7f5246400700 1 -- start start 2024-02-28T17:16:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.366+0000 7f5246400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52400724c0 0x7f5240119d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.366+0000 7f5246400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5240072dd0 0x7f524011a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.366+0000 7f5246400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524011a920 con 0x7f5240072dd0 2024-02-28T17:16:13.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.366+0000 7f5246400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5240113c30 con 0x7f52400724c0 2024-02-28T17:16:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.366+0000 7f523fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5240072dd0 0x7f524011a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.366+0000 7f523fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5240072dd0 0x7f524011a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59326/0 (socket says 172.21.15.1:59326) 2024-02-28T17:16:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.366+0000 7f523fe00700 1 -- 172.21.15.1:0/3985138134 learned_addr learned my addr 172.21.15.1:0/3985138134 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:13.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.367+0000 7f5245000700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52400724c0 0x7f5240119d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.367+0000 7f523fe00700 1 -- 172.21.15.1:0/3985138134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52400724c0 msgr2=0x7f5240119d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.367+0000 7f523fe00700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52400724c0 0x7f5240119d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.367+0000 7f523fe00700 1 -- 172.21.15.1:0/3985138134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5234005040 con 0x7f5240072dd0 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.367+0000 7f523fe00700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5240072dd0 0x7f524011a2d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5230002a50 tx=0x7f52300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.367+0000 7f523cc00700 1 -- 172.21.15.1:0/3985138134 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5230007cc0 con 0x7f5240072dd0 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.368+0000 7f523cc00700 1 -- 172.21.15.1:0/3985138134 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f523000f040 con 0x7f5240072dd0 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.368+0000 7f523cc00700 1 -- 172.21.15.1:0/3985138134 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5230009640 con 0x7f5240072dd0 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.368+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5240113f10 con 0x7f5240072dd0 2024-02-28T17:16:13.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.368+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52401144d0 con 0x7f5240072dd0 2024-02-28T17:16:13.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.369+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524010ae20 con 0x7f5240072dd0 2024-02-28T17:16:13.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.370+0000 7f523cc00700 1 -- 172.21.15.1:0/3985138134 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f52300097a0 con 0x7f5240072dd0 2024-02-28T17:16:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.371+0000 7f523cc00700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f522806d7e0 0x7f522806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:13.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.371+0000 7f523cc00700 1 -- 172.21.15.1:0/3985138134 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f523008cd20 con 0x7f5240072dd0 2024-02-28T17:16:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.372+0000 7f5245000700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f522806d7e0 0x7f522806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:13.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.372+0000 7f5245000700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f522806d7e0 0x7f522806fc90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5234004f90 tx=0x7f5234009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.377+0000 7f523cc00700 1 -- 172.21.15.1:0/3985138134 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f523005a000 con 0x7f5240072dd0 2024-02-28T17:16:13.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.628+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5240115370 con 0x7f522806d7e0 2024-02-28T17:16:13.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.630+0000 7f523cc00700 1 -- 172.21.15.1:0/3985138134 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5240115370 con 0x7f522806d7e0 2024-02-28T17:16:13.634 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:16:13.634 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:16:13.634 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:16:13.634 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:16:13.634 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:16:13.635 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:16:13.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.632+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f522806d7e0 msgr2=0x7f522806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:13.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.632+0000 7f5246400700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f522806d7e0 0x7f522806fc90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5234004f90 tx=0x7f5234009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:13.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.632+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5240072dd0 msgr2=0x7f524011a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:13.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.632+0000 7f5246400700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5240072dd0 0x7f524011a2d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5230002a50 tx=0x7f52300046b0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.633+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 shutdown_connections 2024-02-28T17:16:13.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.633+0000 7f5246400700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f522806d7e0 0x7f522806fc90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.633+0000 7f5246400700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f52400724c0 0x7f5240119d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.633+0000 7f5246400700 1 --2- 172.21.15.1:0/3985138134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5240072dd0 0x7f524011a2d0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.633+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 >> 172.21.15.1:0/3985138134 conn(0x7f52400fab10 msgr2=0x7f52400fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:13.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.633+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 shutdown_connections 2024-02-28T17:16:13.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.633+0000 7f5246400700 1 -- 172.21.15.1:0/3985138134 wait complete. 2024-02-28T17:16:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.793+0000 7f527e200700 1 -- 172.21.15.1:0/417433312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 msgr2=0x7f52780fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.793+0000 7f527e200700 1 --2- 172.21.15.1:0/417433312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 0x7f52780fcf60 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5260004940 tx=0x7f5260009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.793+0000 7f527e200700 1 -- 172.21.15.1:0/417433312 shutdown_connections 2024-02-28T17:16:13.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.793+0000 7f527e200700 1 --2- 172.21.15.1:0/417433312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 0x7f52780fcf60 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.793+0000 7f527e200700 1 --2- 172.21.15.1:0/417433312 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5278102f50 0x7f5278103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.793+0000 7f527e200700 1 -- 172.21.15.1:0/417433312 >> 172.21.15.1:0/417433312 conn(0x7f52780f8ae0 msgr2=0x7f52780faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:13.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.793+0000 7f527e200700 1 -- 172.21.15.1:0/417433312 shutdown_connections 2024-02-28T17:16:13.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.794+0000 7f527e200700 1 -- 172.21.15.1:0/417433312 wait complete. 2024-02-28T17:16:13.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.794+0000 7f527e200700 1 Processor -- start 2024-02-28T17:16:13.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.794+0000 7f527e200700 1 -- start start 2024-02-28T17:16:13.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.795+0000 7f527e200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5278102f50 0x7f5278115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:13.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.795+0000 7f527e200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 0x7f52781161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:13.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.795+0000 7f527e200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5278116810 con 0x7f52781038f0 2024-02-28T17:16:13.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.795+0000 7f527e200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527810fb40 con 0x7f5278102f50 2024-02-28T17:16:13.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.795+0000 7f526ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 0x7f52781161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:13.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.795+0000 7f526ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 0x7f52781161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59336/0 (socket says 172.21.15.1:59336) 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.795+0000 7f526ea00700 1 -- 172.21.15.1:0/3700025788 learned_addr learned my addr 172.21.15.1:0/3700025788 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f5277400700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5278102f50 0x7f5278115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f526ea00700 1 -- 172.21.15.1:0/3700025788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5278102f50 msgr2=0x7f5278115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f526ea00700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5278102f50 0x7f5278115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f526ea00700 1 -- 172.21.15.1:0/3700025788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5260005040 con 0x7f52781038f0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f5277400700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5278102f50 0x7f5278115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f526ea00700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 0x7f52781161c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f5260009030 tx=0x7f5260009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f526fe00700 1 -- 172.21.15.1:0/3700025788 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f526000e040 con 0x7f52781038f0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f526fe00700 1 -- 172.21.15.1:0/3700025788 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f526000bbf0 con 0x7f52781038f0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f526fe00700 1 -- 172.21.15.1:0/3700025788 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f526001c630 con 0x7f52781038f0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.796+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527810fdc0 con 0x7f52781038f0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.797+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5278110300 con 0x7f52781038f0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.798+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52780fc6e0 con 0x7f52781038f0 2024-02-28T17:16:13.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.799+0000 7f526fe00700 1 -- 172.21.15.1:0/3700025788 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f526000bd60 con 0x7f52781038f0 2024-02-28T17:16:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.800+0000 7f526fe00700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f525c06d7f0 0x7f525c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.800+0000 7f526fe00700 1 -- 172.21.15.1:0/3700025788 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f526008e130 con 0x7f52781038f0 2024-02-28T17:16:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.800+0000 7f5277400700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f525c06d7f0 0x7f525c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:13.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.801+0000 7f5277400700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f525c06d7f0 0x7f525c06fca0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5268004550 tx=0x7f5268009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:13.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:13.805+0000 7f526fe00700 1 -- 172.21.15.1:0/3700025788 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f526005b5d0 con 0x7f52781038f0 2024-02-28T17:16:14.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:13 smithi066 ceph-mon[85781]: from='client.36224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:14.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:13 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/32392144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:16:14.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:13 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1353838196' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:16:14.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:13 smithi001 ceph-mon[112940]: from='client.36224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:14.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:13 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/32392144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:16:14.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:13 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1353838196' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:16:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.126+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f527810a030 con 0x7f52781038f0 2024-02-28T17:16:14.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.127+0000 7f526fe00700 1 -- 172.21.15.1:0/3700025788 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f526005b160 con 0x7f52781038f0 2024-02-28T17:16:14.133 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:16:14.133 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:16:14.133 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:16:14.133 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:16:14.133 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:16:14.133 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:16:14.133 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:16:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:16:14.135 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:16:14.135 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:16:14.135 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:16:14.135 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:16:14.135 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:16:14.135 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:16:14.135 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:16:14.135 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:16:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.130+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f525c06d7f0 msgr2=0x7f525c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.130+0000 7f527e200700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f525c06d7f0 0x7f525c06fca0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5268004550 tx=0x7f5268009040 comp rx=0 tx=0).stop 2024-02-28T17:16:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 msgr2=0x7f52781161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 0x7f52781161c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f5260009030 tx=0x7f5260009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:14.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 shutdown_connections 2024-02-28T17:16:14.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f525c06d7f0 0x7f525c06fca0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:14.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5278102f50 0x7f5278115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:14.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 --2- 172.21.15.1:0/3700025788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52781038f0 0x7f52781161c0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:14.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 >> 172.21.15.1:0/3700025788 conn(0x7f52780f8ae0 msgr2=0x7f52780fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:14.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 shutdown_connections 2024-02-28T17:16:14.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:14.131+0000 7f527e200700 1 -- 172.21.15.1:0/3700025788 wait complete. 2024-02-28T17:16:15.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:14 smithi066 ceph-mon[85781]: pgmap v1257: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:15.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:14 smithi066 ceph-mon[85781]: from='client.36236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:15.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:14 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3700025788' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:16:15.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:14 smithi001 ceph-mon[112940]: pgmap v1257: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:15.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:14 smithi001 ceph-mon[112940]: from='client.36236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:15.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:14 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3700025788' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:16:17.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:16 smithi066 ceph-mon[85781]: pgmap v1258: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:17.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:16 smithi001 ceph-mon[112940]: pgmap v1258: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:19.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:18 smithi066 ceph-mon[85781]: pgmap v1259: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:19.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:18 smithi001 ceph-mon[112940]: pgmap v1259: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:21.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:20 smithi066 ceph-mon[85781]: pgmap v1260: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:21.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:20 smithi001 ceph-mon[112940]: pgmap v1260: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:23.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:22 smithi066 ceph-mon[85781]: pgmap v1261: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:23.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:22 smithi001 ceph-mon[112940]: pgmap v1261: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:25.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:24 smithi066 ceph-mon[85781]: pgmap v1262: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:25.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:24 smithi001 ceph-mon[112940]: pgmap v1262: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:27.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:26 smithi066 ceph-mon[85781]: pgmap v1263: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:27.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:16:27.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:26 smithi001 ceph-mon[112940]: pgmap v1263: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:27.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:16:29.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:28 smithi066 ceph-mon[85781]: pgmap v1264: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:29.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:28 smithi001 ceph-mon[112940]: pgmap v1264: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:31.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:30 smithi066 ceph-mon[85781]: pgmap v1265: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:30 smithi001 ceph-mon[112940]: pgmap v1265: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:33.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:32 smithi066 ceph-mon[85781]: pgmap v1266: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:33.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:32 smithi001 ceph-mon[112940]: pgmap v1266: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:35.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:34 smithi066 ceph-mon[85781]: pgmap v1267: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:35.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:34 smithi001 ceph-mon[112940]: pgmap v1267: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:37.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:36 smithi066 ceph-mon[85781]: pgmap v1268: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:37.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:36 smithi001 ceph-mon[112940]: pgmap v1268: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:39.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:38 smithi066 ceph-mon[85781]: pgmap v1269: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:39.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:38 smithi001 ceph-mon[112940]: pgmap v1269: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:41.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:40 smithi066 ceph-mon[85781]: pgmap v1270: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:41.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:40 smithi001 ceph-mon[112940]: pgmap v1270: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:42.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:16:42.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:16:43.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:42 smithi001 ceph-mon[112940]: pgmap v1271: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:43.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:42 smithi066 ceph-mon[85781]: pgmap v1271: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:44.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.304+0000 7f1babe00700 1 -- 172.21.15.1:0/2140432881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac1036e0 msgr2=0x7f1bac0fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.304+0000 7f1babe00700 1 --2- 172.21.15.1:0/2140432881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac1036e0 0x7f1bac0fcd50 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f1b98004940 tx=0x7f1b98009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.305+0000 7f1babe00700 1 -- 172.21.15.1:0/2140432881 shutdown_connections 2024-02-28T17:16:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.305+0000 7f1babe00700 1 --2- 172.21.15.1:0/2140432881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac1036e0 0x7f1bac0fcd50 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.305+0000 7f1babe00700 1 --2- 172.21.15.1:0/2140432881 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bac102d40 0x7f1bac103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.305+0000 7f1babe00700 1 -- 172.21.15.1:0/2140432881 >> 172.21.15.1:0/2140432881 conn(0x7f1bac0f88b0 msgr2=0x7f1bac0facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.305+0000 7f1babe00700 1 -- 172.21.15.1:0/2140432881 shutdown_connections 2024-02-28T17:16:44.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.305+0000 7f1babe00700 1 -- 172.21.15.1:0/2140432881 wait complete. 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.306+0000 7f1babe00700 1 Processor -- start 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.306+0000 7f1babe00700 1 -- start start 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.306+0000 7f1babe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac102d40 0x7f1bac115ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.306+0000 7f1babe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bac1036e0 0x7f1bac116010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.306+0000 7f1babe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac116660 con 0x7f1bac102d40 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.306+0000 7f1babe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac10f990 con 0x7f1bac1036e0 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.307+0000 7f1baaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac102d40 0x7f1bac115ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.307+0000 7f1baaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac102d40 0x7f1bac115ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60060/0 (socket says 172.21.15.1:60060) 2024-02-28T17:16:44.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.307+0000 7f1baaa00700 1 -- 172.21.15.1:0/3326168659 learned_addr learned my addr 172.21.15.1:0/3326168659 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.307+0000 7f1baa000700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bac1036e0 0x7f1bac116010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.307+0000 7f1baaa00700 1 -- 172.21.15.1:0/3326168659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bac1036e0 msgr2=0x7f1bac116010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.307+0000 7f1baaa00700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bac1036e0 0x7f1bac116010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.307+0000 7f1baaa00700 1 -- 172.21.15.1:0/3326168659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98005040 con 0x7f1bac102d40 2024-02-28T17:16:44.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.307+0000 7f1baaa00700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac102d40 0x7f1bac115ad0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00c830 tx=0x7f1b9c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.308+0000 7f1ba2a00700 1 -- 172.21.15.1:0/3326168659 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b9c00a760 con 0x7f1bac102d40 2024-02-28T17:16:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.308+0000 7f1ba2a00700 1 -- 172.21.15.1:0/3326168659 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b9c00ad90 con 0x7f1bac102d40 2024-02-28T17:16:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.308+0000 7f1ba2a00700 1 -- 172.21.15.1:0/3326168659 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b9c0113a0 con 0x7f1bac102d40 2024-02-28T17:16:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.308+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac10fc70 con 0x7f1bac102d40 2024-02-28T17:16:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.308+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bac1101b0 con 0x7f1bac102d40 2024-02-28T17:16:44.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.309+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac05f5c0 con 0x7f1bac102d40 2024-02-28T17:16:44.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.310+0000 7f1ba2a00700 1 -- 172.21.15.1:0/3326168659 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1b9c012400 con 0x7f1bac102d40 2024-02-28T17:16:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.311+0000 7f1ba2a00700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b9006d7f0 0x7f1b9006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:44.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.311+0000 7f1ba2a00700 1 -- 172.21.15.1:0/3326168659 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1b9c08cf00 con 0x7f1bac102d40 2024-02-28T17:16:44.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.311+0000 7f1baa000700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b9006d7f0 0x7f1b9006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:44.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.312+0000 7f1baa000700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b9006d7f0 0x7f1b9006fca0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1bac111300 tx=0x7f1b98009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:44.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.316+0000 7f1ba2a00700 1 -- 172.21.15.1:0/3326168659 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b9c05a2e0 con 0x7f1bac102d40 2024-02-28T17:16:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.561+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bac111120 con 0x7f1b9006d7f0 2024-02-28T17:16:44.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.564+0000 7f1ba2a00700 1 -- 172.21.15.1:0/3326168659 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1bac111120 con 0x7f1b9006d7f0 2024-02-28T17:16:44.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b9006d7f0 msgr2=0x7f1b9006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b9006d7f0 0x7f1b9006fca0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1bac111300 tx=0x7f1b98009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac102d40 msgr2=0x7f1bac115ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac102d40 0x7f1bac115ad0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00c830 tx=0x7f1b9c004470 comp rx=0 tx=0).stop 2024-02-28T17:16:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 shutdown_connections 2024-02-28T17:16:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1b9006d7f0 0x7f1b9006fca0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1bac102d40 0x7f1bac115ad0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 --2- 172.21.15.1:0/3326168659 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1bac1036e0 0x7f1bac116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.567+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 >> 172.21.15.1:0/3326168659 conn(0x7f1bac0f88b0 msgr2=0x7f1bac0fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:44.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.568+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 shutdown_connections 2024-02-28T17:16:44.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.568+0000 7f1babe00700 1 -- 172.21.15.1:0/3326168659 wait complete. 2024-02-28T17:16:44.588 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:16:44.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.731+0000 7f8bccc00700 1 -- 172.21.15.1:0/2118068614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc80702b0 msgr2=0x7f8bc8070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:44.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.731+0000 7f8bccc00700 1 --2- 172.21.15.1:0/2118068614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc80702b0 0x7f8bc8070680 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8004940 tx=0x7f8bb8009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:44.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.731+0000 7f8bccc00700 1 -- 172.21.15.1:0/2118068614 shutdown_connections 2024-02-28T17:16:44.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.732+0000 7f8bccc00700 1 --2- 172.21.15.1:0/2118068614 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc8070bc0 0x7f8bc8109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.732+0000 7f8bccc00700 1 --2- 172.21.15.1:0/2118068614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc80702b0 0x7f8bc8070680 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.732+0000 7f8bccc00700 1 -- 172.21.15.1:0/2118068614 >> 172.21.15.1:0/2118068614 conn(0x7f8bc80f8960 msgr2=0x7f8bc80fad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:44.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.732+0000 7f8bccc00700 1 -- 172.21.15.1:0/2118068614 shutdown_connections 2024-02-28T17:16:44.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.732+0000 7f8bccc00700 1 -- 172.21.15.1:0/2118068614 wait complete. 2024-02-28T17:16:44.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.733+0000 7f8bccc00700 1 Processor -- start 2024-02-28T17:16:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.733+0000 7f8bccc00700 1 -- start start 2024-02-28T17:16:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.733+0000 7f8bccc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc80702b0 0x7f8bc8115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.733+0000 7f8bccc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc8070bc0 0x7f8bc8115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.733+0000 7f8bccc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8116630 con 0x7f8bc8070bc0 2024-02-28T17:16:44.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.733+0000 7f8bccc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc810f960 con 0x7f8bc80702b0 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.734+0000 7f8bc6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc8070bc0 0x7f8bc8115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.734+0000 7f8bc6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc8070bc0 0x7f8bc8115fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60086/0 (socket says 172.21.15.1:60086) 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.734+0000 7f8bc6a00700 1 -- 172.21.15.1:0/1142247622 learned_addr learned my addr 172.21.15.1:0/1142247622 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.734+0000 7f8bc7400700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc80702b0 0x7f8bc8115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.734+0000 7f8bc6a00700 1 -- 172.21.15.1:0/1142247622 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc80702b0 msgr2=0x7f8bc8115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.734+0000 7f8bc6a00700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc80702b0 0x7f8bc8115aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.734+0000 7f8bc6a00700 1 -- 172.21.15.1:0/1142247622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8005040 con 0x7f8bc8070bc0 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.734+0000 7f8bc6a00700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc8070bc0 0x7f8bc8115fe0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4002a50 tx=0x7f8bb40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:44.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.735+0000 7f8bbf400700 1 -- 172.21.15.1:0/1142247622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb4007cc0 con 0x7f8bc8070bc0 2024-02-28T17:16:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.735+0000 7f8bbf400700 1 -- 172.21.15.1:0/1142247622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bb400f040 con 0x7f8bc8070bc0 2024-02-28T17:16:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.735+0000 7f8bbf400700 1 -- 172.21.15.1:0/1142247622 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb400db70 con 0x7f8bc8070bc0 2024-02-28T17:16:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.735+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc810fc40 con 0x7f8bc8070bc0 2024-02-28T17:16:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.735+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc8110180 con 0x7f8bc8070bc0 2024-02-28T17:16:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.736+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc81067e0 con 0x7f8bc8070bc0 2024-02-28T17:16:44.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.738+0000 7f8bbf400700 1 -- 172.21.15.1:0/1142247622 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8bb400dd10 con 0x7f8bc8070bc0 2024-02-28T17:16:44.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.739+0000 7f8bbf400700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8bac06d7f0 0x7f8bac06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.739+0000 7f8bbf400700 1 -- 172.21.15.1:0/1142247622 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8bb408c590 con 0x7f8bc8070bc0 2024-02-28T17:16:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.740+0000 7f8bc7400700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8bac06d7f0 0x7f8bac06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:44.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.740+0000 7f8bc7400700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8bac06d7f0 0x7f8bac06fca0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000c00 tx=0x7f8bb8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:44.746+0000 7f8bbf400700 1 -- 172.21.15.1:0/1142247622 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8bb40598f0 con 0x7f8bc8070bc0 2024-02-28T17:16:45.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.002+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bc810c0e0 con 0x7f8bac06d7f0 2024-02-28T17:16:45.006 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:44 smithi001 ceph-mon[112940]: pgmap v1272: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.005+0000 7f8bbf400700 1 -- 172.21.15.1:0/1142247622 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8bc810c0e0 con 0x7f8bac06d7f0 2024-02-28T17:16:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8bac06d7f0 msgr2=0x7f8bac06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8bac06d7f0 0x7f8bac06fca0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000c00 tx=0x7f8bb8009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc8070bc0 msgr2=0x7f8bc8115fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc8070bc0 0x7f8bc8115fe0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4002a50 tx=0x7f8bb40046b0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 shutdown_connections 2024-02-28T17:16:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8bac06d7f0 0x7f8bac06fca0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8bc80702b0 0x7f8bc8115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 --2- 172.21.15.1:0/1142247622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bc8070bc0 0x7f8bc8115fe0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 >> 172.21.15.1:0/1142247622 conn(0x7f8bc80f8960 msgr2=0x7f8bc8103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 shutdown_connections 2024-02-28T17:16:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.008+0000 7f8bccc00700 1 -- 172.21.15.1:0/1142247622 wait complete. 2024-02-28T17:16:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.169+0000 7f6455400700 1 -- 172.21.15.1:0/2562880229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 msgr2=0x7f6450070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.169+0000 7f6455400700 1 --2- 172.21.15.1:0/2562880229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 0x7f6450070700 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6440004970 tx=0x7f6440009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.169+0000 7f6455400700 1 -- 172.21.15.1:0/2562880229 shutdown_connections 2024-02-28T17:16:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.169+0000 7f6455400700 1 --2- 172.21.15.1:0/2562880229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6450070c40 0x7f6450109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.169+0000 7f6455400700 1 --2- 172.21.15.1:0/2562880229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 0x7f6450070700 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.169+0000 7f6455400700 1 -- 172.21.15.1:0/2562880229 >> 172.21.15.1:0/2562880229 conn(0x7f64500f8ae0 msgr2=0x7f64500faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.169+0000 7f6455400700 1 -- 172.21.15.1:0/2562880229 shutdown_connections 2024-02-28T17:16:45.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.169+0000 7f6455400700 1 -- 172.21.15.1:0/2562880229 wait complete. 2024-02-28T17:16:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.170+0000 7f6455400700 1 Processor -- start 2024-02-28T17:16:45.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.170+0000 7f6455400700 1 -- start start 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.170+0000 7f6455400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 0x7f6450115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.170+0000 7f6455400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6450070c40 0x7f6450116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.170+0000 7f6455400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64501167b0 con 0x7f6450070330 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.170+0000 7f6455400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645010fae0 con 0x7f6450070c40 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.171+0000 7f644e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 0x7f6450115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.171+0000 7f644e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 0x7f6450115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60108/0 (socket says 172.21.15.1:60108) 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.171+0000 7f644e800700 1 -- 172.21.15.1:0/1866521924 learned_addr learned my addr 172.21.15.1:0/1866521924 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.171+0000 7f644e800700 1 -- 172.21.15.1:0/1866521924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6450070c40 msgr2=0x7f6450116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.171+0000 7f644e800700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6450070c40 0x7f6450116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.171+0000 7f644e800700 1 -- 172.21.15.1:0/1866521924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6440005040 con 0x7f6450070330 2024-02-28T17:16:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.171+0000 7f644e800700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 0x7f6450115c20 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6440009150 tx=0x7f6440009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.172+0000 7f6447400700 1 -- 172.21.15.1:0/1866521924 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f644000e040 con 0x7f6450070330 2024-02-28T17:16:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.172+0000 7f6447400700 1 -- 172.21.15.1:0/1866521924 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6440004290 con 0x7f6450070330 2024-02-28T17:16:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.172+0000 7f6447400700 1 -- 172.21.15.1:0/1866521924 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f644001c630 con 0x7f6450070330 2024-02-28T17:16:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.172+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645010fd60 con 0x7f6450070330 2024-02-28T17:16:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.172+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64501102a0 con 0x7f6450070330 2024-02-28T17:16:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.173+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6450106920 con 0x7f6450070330 2024-02-28T17:16:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.174+0000 7f6447400700 1 -- 172.21.15.1:0/1866521924 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6440013980 con 0x7f6450070330 2024-02-28T17:16:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.175+0000 7f6447400700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f643406d7f0 0x7f643406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.175+0000 7f6447400700 1 -- 172.21.15.1:0/1866521924 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f644008e210 con 0x7f6450070330 2024-02-28T17:16:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.176+0000 7f6445e00700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f643406d7f0 0x7f643406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.176+0000 7f6445e00700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f643406d7f0 0x7f643406fca0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6450111450 tx=0x7f6438009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.180+0000 7f6447400700 1 -- 172.21.15.1:0/1866521924 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f644005b5f0 con 0x7f6450070330 2024-02-28T17:16:45.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:44 smithi066 ceph-mon[85781]: pgmap v1272: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.419+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f645010c240 con 0x7f643406d7f0 2024-02-28T17:16:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.434+0000 7f6447400700 1 -- 172.21.15.1:0/1866521924 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f645010c240 con 0x7f643406d7f0 2024-02-28T17:16:45.438 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:16:45.438 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (42m) 4m ago 57m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:16:45.438 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (39m) 4m ago 57m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (39m) 4m ago 56m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (42m) 4m ago 56m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (35m) 4m ago 50m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (35m) 4m ago 49m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (4m) 4m ago 49m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (19m) 4m ago 49m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (45m) 4m ago 58m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (44m) 4m ago 55m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (41m) 4m ago 59m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (40m) 4m ago 55m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (43m) 4m ago 57m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (43m) 4m ago 55m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (38m) 4m ago 54m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:16:45.439 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (38m) 4m ago 54m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:16:45.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (37m) 4m ago 53m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:16:45.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (37m) 4m ago 53m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:16:45.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (36m) 4m ago 52m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:16:45.440 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (35m) 4m ago 51m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:16:45.440 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (43m) 4m ago 56m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:16:45.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.437+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f643406d7f0 msgr2=0x7f643406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.437+0000 7f6455400700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f643406d7f0 0x7f643406fca0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6450111450 tx=0x7f6438009ce0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.437+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 msgr2=0x7f6450115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.437+0000 7f6455400700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 0x7f6450115c20 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6440009150 tx=0x7f6440009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:45.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.438+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 shutdown_connections 2024-02-28T17:16:45.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.438+0000 7f6455400700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f643406d7f0 0x7f643406fca0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.438+0000 7f6455400700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6450070330 0x7f6450115c20 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.438+0000 7f6455400700 1 --2- 172.21.15.1:0/1866521924 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6450070c40 0x7f6450116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.438+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 >> 172.21.15.1:0/1866521924 conn(0x7f64500f8ae0 msgr2=0x7f6450103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:45.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.438+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 shutdown_connections 2024-02-28T17:16:45.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.438+0000 7f6455400700 1 -- 172.21.15.1:0/1866521924 wait complete. 2024-02-28T17:16:45.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.592+0000 7faf71a00700 1 -- 172.21.15.1:0/949707381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 msgr2=0x7faf6c070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.592+0000 7faf71a00700 1 --2- 172.21.15.1:0/949707381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 0x7faf6c070700 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009d10 tx=0x7faf5c004d10 comp rx=0 tx=0).stop 2024-02-28T17:16:45.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.592+0000 7faf71a00700 1 -- 172.21.15.1:0/949707381 shutdown_connections 2024-02-28T17:16:45.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.592+0000 7faf71a00700 1 --2- 172.21.15.1:0/949707381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf6c070c40 0x7faf6c109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.592+0000 7faf71a00700 1 --2- 172.21.15.1:0/949707381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 0x7faf6c070700 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.592+0000 7faf71a00700 1 -- 172.21.15.1:0/949707381 >> 172.21.15.1:0/949707381 conn(0x7faf6c0f8ae0 msgr2=0x7faf6c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:45.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.592+0000 7faf71a00700 1 -- 172.21.15.1:0/949707381 shutdown_connections 2024-02-28T17:16:45.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.592+0000 7faf71a00700 1 -- 172.21.15.1:0/949707381 wait complete. 2024-02-28T17:16:45.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.593+0000 7faf71a00700 1 Processor -- start 2024-02-28T17:16:45.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.594+0000 7faf71a00700 1 -- start start 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.594+0000 7faf71a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 0x7faf6c115e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.594+0000 7faf71a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf6c070c40 0x7faf6c116370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.594+0000 7faf71a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c1169c0 con 0x7faf6c070330 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.594+0000 7faf71a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c10fcd0 con 0x7faf6c070c40 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.594+0000 7faf6aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 0x7faf6c115e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.594+0000 7faf6aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 0x7faf6c115e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60132/0 (socket says 172.21.15.1:60132) 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.594+0000 7faf6aa00700 1 -- 172.21.15.1:0/3889218038 learned_addr learned my addr 172.21.15.1:0/3889218038 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.595+0000 7faf6aa00700 1 -- 172.21.15.1:0/3889218038 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf6c070c40 msgr2=0x7faf6c116370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.595+0000 7faf6aa00700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf6c070c40 0x7faf6c116370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.595+0000 7faf6aa00700 1 -- 172.21.15.1:0/3889218038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c009940 con 0x7faf6c070330 2024-02-28T17:16:45.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.595+0000 7faf6aa00700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 0x7faf6c115e30 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7faf5c008dd0 tx=0x7faf5c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:45.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.595+0000 7faf62a00700 1 -- 172.21.15.1:0/3889218038 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf5c00e040 con 0x7faf6c070330 2024-02-28T17:16:45.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.595+0000 7faf62a00700 1 -- 172.21.15.1:0/3889218038 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf5c0128e0 con 0x7faf6c070330 2024-02-28T17:16:45.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.595+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c10ff50 con 0x7faf6c070330 2024-02-28T17:16:45.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.595+0000 7faf62a00700 1 -- 172.21.15.1:0/3889218038 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf5c01c880 con 0x7faf6c070330 2024-02-28T17:16:45.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.596+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c110490 con 0x7faf6c070330 2024-02-28T17:16:45.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.597+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c106920 con 0x7faf6c070330 2024-02-28T17:16:45.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.598+0000 7faf62a00700 1 -- 172.21.15.1:0/3889218038 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7faf5c012410 con 0x7faf6c070330 2024-02-28T17:16:45.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.598+0000 7faf62a00700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf5006d7e0 0x7faf5006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:45.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.599+0000 7faf62a00700 1 -- 172.21.15.1:0/3889218038 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7faf5c08ec60 con 0x7faf6c070330 2024-02-28T17:16:45.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.599+0000 7faf6a000700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf5006d7e0 0x7faf5006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:45.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.599+0000 7faf6a000700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf5006d7e0 0x7faf5006fc90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faf6c1116d0 tx=0x7faf58009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:45.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.603+0000 7faf62a00700 1 -- 172.21.15.1:0/3889218038 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faf5c05bfc0 con 0x7faf6c070330 2024-02-28T17:16:45.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.917+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faf6c04bed0 con 0x7faf6c070330 2024-02-28T17:16:45.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.918+0000 7faf62a00700 1 -- 172.21.15.1:0/3889218038 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7faf5c004e10 con 0x7faf6c070330 2024-02-28T17:16:45.922 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:16:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:16:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:16:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:16:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:16:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:16:45.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:16:45.923 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:16:45.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.921+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf5006d7e0 msgr2=0x7faf5006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.921+0000 7faf71a00700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf5006d7e0 0x7faf5006fc90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faf6c1116d0 tx=0x7faf58009ce0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.921+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 msgr2=0x7faf6c115e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:45.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.921+0000 7faf71a00700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 0x7faf6c115e30 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7faf5c008dd0 tx=0x7faf5c004d10 comp rx=0 tx=0).stop 2024-02-28T17:16:45.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.921+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 shutdown_connections 2024-02-28T17:16:45.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.921+0000 7faf71a00700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf5006d7e0 0x7faf5006fc90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.922+0000 7faf71a00700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf6c070330 0x7faf6c115e30 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.922+0000 7faf71a00700 1 --2- 172.21.15.1:0/3889218038 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf6c070c40 0x7faf6c116370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:45.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.922+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 >> 172.21.15.1:0/3889218038 conn(0x7faf6c0f8ae0 msgr2=0x7faf6c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:45.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.922+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 shutdown_connections 2024-02-28T17:16:45.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:45.922+0000 7faf71a00700 1 -- 172.21.15.1:0/3889218038 wait complete. 2024-02-28T17:16:46.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:45 smithi001 ceph-mon[112940]: from='client.36244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:46.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:45 smithi001 ceph-mon[112940]: from='client.36248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.080+0000 7fe74e200700 1 -- 172.21.15.1:0/1996785427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 msgr2=0x7fe748107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.080+0000 7fe74e200700 1 --2- 172.21.15.1:0/1996785427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 0x7fe748107050 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe734004970 tx=0x7fe734009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.080+0000 7fe74e200700 1 -- 172.21.15.1:0/1996785427 shutdown_connections 2024-02-28T17:16:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.080+0000 7fe74e200700 1 --2- 172.21.15.1:0/1996785427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 0x7fe748107050 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.080+0000 7fe74e200700 1 --2- 172.21.15.1:0/1996785427 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7480fdfe0 0x7fe7480fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.080+0000 7fe74e200700 1 -- 172.21.15.1:0/1996785427 >> 172.21.15.1:0/1996785427 conn(0x7fe748071d90 msgr2=0x7fe748072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.080+0000 7fe74e200700 1 -- 172.21.15.1:0/1996785427 shutdown_connections 2024-02-28T17:16:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.081+0000 7fe74e200700 1 -- 172.21.15.1:0/1996785427 wait complete. 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.081+0000 7fe74e200700 1 Processor -- start 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.082+0000 7fe74e200700 1 -- start start 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.082+0000 7fe74e200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7480fdfe0 0x7fe7481139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.082+0000 7fe74e200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 0x7fe748113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.082+0000 7fe74e200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe748114580 con 0x7fe7480fe8f0 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.082+0000 7fe74e200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74810d8b0 con 0x7fe7480fdfe0 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.082+0000 7fe746a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 0x7fe748113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.082+0000 7fe746a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 0x7fe748113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60144/0 (socket says 172.21.15.1:60144) 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.082+0000 7fe746a00700 1 -- 172.21.15.1:0/2982962314 learned_addr learned my addr 172.21.15.1:0/2982962314 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.083+0000 7fe747400700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7480fdfe0 0x7fe7481139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:46.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.083+0000 7fe746a00700 1 -- 172.21.15.1:0/2982962314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7480fdfe0 msgr2=0x7fe7481139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.083+0000 7fe746a00700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7480fdfe0 0x7fe7481139f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.083+0000 7fe746a00700 1 -- 172.21.15.1:0/2982962314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe734005040 con 0x7fe7480fe8f0 2024-02-28T17:16:46.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.083+0000 7fe746a00700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 0x7fe748113f30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe734004f50 tx=0x7fe734009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:46.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.084+0000 7fe73f400700 1 -- 172.21.15.1:0/2982962314 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe73400e040 con 0x7fe7480fe8f0 2024-02-28T17:16:46.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.084+0000 7fe73f400700 1 -- 172.21.15.1:0/2982962314 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe734004290 con 0x7fe7480fe8f0 2024-02-28T17:16:46.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.084+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74810db30 con 0x7fe7480fe8f0 2024-02-28T17:16:46.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.084+0000 7fe73f400700 1 -- 172.21.15.1:0/2982962314 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe73401c630 con 0x7fe7480fe8f0 2024-02-28T17:16:46.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.084+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74810e070 con 0x7fe7480fe8f0 2024-02-28T17:16:46.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.085+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7481046c0 con 0x7fe7480fe8f0 2024-02-28T17:16:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.086+0000 7fe73f400700 1 -- 172.21.15.1:0/2982962314 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe73400bc10 con 0x7fe7480fe8f0 2024-02-28T17:16:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.087+0000 7fe73f400700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe72c06d7f0 0x7fe72c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.087+0000 7fe73f400700 1 -- 172.21.15.1:0/2982962314 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe73408e280 con 0x7fe7480fe8f0 2024-02-28T17:16:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.087+0000 7fe747400700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe72c06d7f0 0x7fe72c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.088+0000 7fe747400700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe72c06d7f0 0x7fe72c06fca0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe738004990 tx=0x7fe738009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:46.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.091+0000 7fe73f400700 1 -- 172.21.15.1:0/2982962314 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe73405b660 con 0x7fe7480fe8f0 2024-02-28T17:16:46.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:45 smithi066 ceph-mon[85781]: from='client.36244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:46.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:45 smithi066 ceph-mon[85781]: from='client.36248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.363+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe74810a030 con 0x7fe7480fe8f0 2024-02-28T17:16:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.364+0000 7fe73f400700 1 -- 172.21.15.1:0/2982962314 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fe734013e60 con 0x7fe7480fe8f0 2024-02-28T17:16:46.368 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:16:46.368 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:16:46.368 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:16:46.368 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:16:46.369 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:16:46.370 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:16:46.371 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:16:46.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.368+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe72c06d7f0 msgr2=0x7fe72c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.368+0000 7fe74e200700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe72c06d7f0 0x7fe72c06fca0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe738004990 tx=0x7fe738009ee0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.368+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 msgr2=0x7fe748113f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.368+0000 7fe74e200700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 0x7fe748113f30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe734004f50 tx=0x7fe734009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:46.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.368+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 shutdown_connections 2024-02-28T17:16:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.368+0000 7fe74e200700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe72c06d7f0 0x7fe72c06fca0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.368+0000 7fe74e200700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe7480fdfe0 0x7fe7481139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.368+0000 7fe74e200700 1 --2- 172.21.15.1:0/2982962314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7480fe8f0 0x7fe748113f30 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.369+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 >> 172.21.15.1:0/2982962314 conn(0x7fe748071d90 msgr2=0x7fe748101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.369+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 shutdown_connections 2024-02-28T17:16:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.369+0000 7fe74e200700 1 -- 172.21.15.1:0/2982962314 wait complete. 2024-02-28T17:16:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:16:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.522+0000 7f6329a00700 1 -- 172.21.15.1:0/2690929841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6324103dc0 msgr2=0x7f6324104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.522+0000 7f6329a00700 1 --2- 172.21.15.1:0/2690929841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6324103dc0 0x7f6324104210 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f6310004940 tx=0x7f6310009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.523+0000 7f6329a00700 1 -- 172.21.15.1:0/2690929841 shutdown_connections 2024-02-28T17:16:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.523+0000 7f6329a00700 1 --2- 172.21.15.1:0/2690929841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6324103dc0 0x7f6324104210 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.523+0000 7f6329a00700 1 --2- 172.21.15.1:0/2690929841 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f63241095f0 0x7f63241099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.523+0000 7f6329a00700 1 -- 172.21.15.1:0/2690929841 >> 172.21.15.1:0/2690929841 conn(0x7f6324072d50 msgr2=0x7f6324075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.523+0000 7f6329a00700 1 -- 172.21.15.1:0/2690929841 shutdown_connections 2024-02-28T17:16:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.523+0000 7f6329a00700 1 -- 172.21.15.1:0/2690929841 wait complete. 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.524+0000 7f6329a00700 1 Processor -- start 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.524+0000 7f6329a00700 1 -- start start 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.524+0000 7f6329a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6324103dc0 0x7f632411a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.524+0000 7f6329a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63241095f0 0x7f632411a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.524+0000 7f6329a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632411ac50 con 0x7f63241095f0 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.524+0000 7f6329a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6324113ed0 con 0x7f6324103dc0 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.525+0000 7f6322000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63241095f0 0x7f632411a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.525+0000 7f6322000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63241095f0 0x7f632411a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60166/0 (socket says 172.21.15.1:60166) 2024-02-28T17:16:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.525+0000 7f6322000700 1 -- 172.21.15.1:0/384302302 learned_addr learned my addr 172.21.15.1:0/384302302 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:46.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.525+0000 7f6322000700 1 -- 172.21.15.1:0/384302302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6324103dc0 msgr2=0x7f632411a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.525+0000 7f6322000700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6324103dc0 0x7f632411a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.525+0000 7f6322000700 1 -- 172.21.15.1:0/384302302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6310005040 con 0x7f63241095f0 2024-02-28T17:16:46.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.525+0000 7f6322000700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63241095f0 0x7f632411a570 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f6310009150 tx=0x7f6310009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:46.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.526+0000 7f631aa00700 1 -- 172.21.15.1:0/384302302 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f631000e040 con 0x7f63241095f0 2024-02-28T17:16:46.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.526+0000 7f631aa00700 1 -- 172.21.15.1:0/384302302 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f631000bbf0 con 0x7f63241095f0 2024-02-28T17:16:46.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.526+0000 7f631aa00700 1 -- 172.21.15.1:0/384302302 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f631001c630 con 0x7f63241095f0 2024-02-28T17:16:46.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.526+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63241140f0 con 0x7f63241095f0 2024-02-28T17:16:46.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.526+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6324114630 con 0x7f63241095f0 2024-02-28T17:16:46.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.527+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6324075870 con 0x7f63241095f0 2024-02-28T17:16:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.528+0000 7f631aa00700 1 -- 172.21.15.1:0/384302302 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f631000bd60 con 0x7f63241095f0 2024-02-28T17:16:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.529+0000 7f631aa00700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f630806d7f0 0x7f630806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.529+0000 7f631aa00700 1 -- 172.21.15.1:0/384302302 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f631008e1d0 con 0x7f63241095f0 2024-02-28T17:16:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.529+0000 7f6322a00700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f630806d7f0 0x7f630806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.530+0000 7f6322a00700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f630806d7f0 0x7f630806fca0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f631400acf0 tx=0x7f6314009370 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:46.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.534+0000 7f631aa00700 1 -- 172.21.15.1:0/384302302 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f631005b670 con 0x7f63241095f0 2024-02-28T17:16:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.773+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6324115720 con 0x7f630806d7f0 2024-02-28T17:16:46.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.775+0000 7f631aa00700 1 -- 172.21.15.1:0/384302302 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6324115720 con 0x7f630806d7f0 2024-02-28T17:16:46.779 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:16:46.779 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:16:46.779 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:16:46.779 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:16:46.779 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:16:46.779 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:16:46.779 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:16:46.780 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:16:46.780 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:16:46.780 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:16:46.780 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:16:46.780 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:16:46.780 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:16:46.780 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:16:46.780 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:16:46.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.777+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f630806d7f0 msgr2=0x7f630806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.777+0000 7f6329a00700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f630806d7f0 0x7f630806fca0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f631400acf0 tx=0x7f6314009370 comp rx=0 tx=0).stop 2024-02-28T17:16:46.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63241095f0 msgr2=0x7f632411a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63241095f0 0x7f632411a570 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f6310009150 tx=0x7f6310009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:46.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 shutdown_connections 2024-02-28T17:16:46.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f630806d7f0 0x7f630806fca0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6324103dc0 0x7f632411a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 --2- 172.21.15.1:0/384302302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63241095f0 0x7f632411a570 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 >> 172.21.15.1:0/384302302 conn(0x7f6324072d50 msgr2=0x7f63240746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:46.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 shutdown_connections 2024-02-28T17:16:46.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.778+0000 7f6329a00700 1 -- 172.21.15.1:0/384302302 wait complete. 2024-02-28T17:16:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.936+0000 7fce04a00700 1 -- 172.21.15.1:0/3668468326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 msgr2=0x7fce001099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.936+0000 7fce04a00700 1 --2- 172.21.15.1:0/3668468326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 0x7fce001099c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0004940 tx=0x7fcdf0009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.936+0000 7fce04a00700 1 -- 172.21.15.1:0/3668468326 shutdown_connections 2024-02-28T17:16:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.936+0000 7fce04a00700 1 --2- 172.21.15.1:0/3668468326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce00103dc0 0x7fce00104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.936+0000 7fce04a00700 1 --2- 172.21.15.1:0/3668468326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 0x7fce001099c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.936+0000 7fce04a00700 1 -- 172.21.15.1:0/3668468326 >> 172.21.15.1:0/3668468326 conn(0x7fce00072d50 msgr2=0x7fce00075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.936+0000 7fce04a00700 1 -- 172.21.15.1:0/3668468326 shutdown_connections 2024-02-28T17:16:46.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.937+0000 7fce04a00700 1 -- 172.21.15.1:0/3668468326 wait complete. 2024-02-28T17:16:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.937+0000 7fce04a00700 1 Processor -- start 2024-02-28T17:16:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.938+0000 7fce04a00700 1 -- start start 2024-02-28T17:16:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.938+0000 7fce04a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce00103dc0 0x7fce0011a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.938+0000 7fce04a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 0x7fce0011a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.938+0000 7fce04a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0011abf0 con 0x7fce001095f0 2024-02-28T17:16:46.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.938+0000 7fce04a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00113f00 con 0x7fce00103dc0 2024-02-28T17:16:46.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.938+0000 7fcdfd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 0x7fce0011a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:46.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.938+0000 7fcdfd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 0x7fce0011a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60184/0 (socket says 172.21.15.1:60184) 2024-02-28T17:16:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.938+0000 7fcdfd400700 1 -- 172.21.15.1:0/1586892741 learned_addr learned my addr 172.21.15.1:0/1586892741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:16:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.939+0000 7fcdfde00700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce00103dc0 0x7fce0011a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.939+0000 7fcdfd400700 1 -- 172.21.15.1:0/1586892741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce00103dc0 msgr2=0x7fce0011a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.939+0000 7fcdfd400700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce00103dc0 0x7fce0011a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.939+0000 7fcdfd400700 1 -- 172.21.15.1:0/1586892741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf0005040 con 0x7fce001095f0 2024-02-28T17:16:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.939+0000 7fcdfd400700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 0x7fce0011a5a0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c810 tx=0x7fcdec004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.940+0000 7fcdf6000700 1 -- 172.21.15.1:0/1586892741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdec012810 con 0x7fce001095f0 2024-02-28T17:16:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.940+0000 7fcdf6000700 1 -- 172.21.15.1:0/1586892741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdec012e40 con 0x7fce001095f0 2024-02-28T17:16:46.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.940+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce001141e0 con 0x7fce001095f0 2024-02-28T17:16:46.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.940+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce00114720 con 0x7fce001095f0 2024-02-28T17:16:46.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.940+0000 7fcdf6000700 1 -- 172.21.15.1:0/1586892741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdec00a370 con 0x7fce001095f0 2024-02-28T17:16:46.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.941+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce00075870 con 0x7fce001095f0 2024-02-28T17:16:46.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.942+0000 7fcdf6000700 1 -- 172.21.15.1:0/1586892741 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcdec011810 con 0x7fce001095f0 2024-02-28T17:16:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.943+0000 7fcdf6000700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcde406d7f0 0x7fcde406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:16:46.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.943+0000 7fcdf6000700 1 -- 172.21.15.1:0/1586892741 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcdec08d280 con 0x7fce001095f0 2024-02-28T17:16:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.943+0000 7fcdfde00700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcde406d7f0 0x7fcde406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:16:46.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.944+0000 7fcdfde00700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcde406d7f0 0x7fcde406fca0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000b3b0 tx=0x7fcdf0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:16:46.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:46.948+0000 7fcdf6000700 1 -- 172.21.15.1:0/1586892741 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcdec05a680 con 0x7fce001095f0 2024-02-28T17:16:47.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:46 smithi001 ceph-mon[112940]: from='client.36252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:47.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:46 smithi001 ceph-mon[112940]: pgmap v1273: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:47.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:46 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3889218038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:16:47.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:46 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2982962314' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:16:47.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.244+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fce001156e0 con 0x7fce001095f0 2024-02-28T17:16:47.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.245+0000 7fcdf6000700 1 -- 172.21.15.1:0/1586892741 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fcdec05a210 con 0x7fce001095f0 2024-02-28T17:16:47.249 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:16:47.249 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:16:47.249 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:16:47.250 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:16:47.251 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:16:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcde406d7f0 msgr2=0x7fcde406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcde406d7f0 0x7fcde406fca0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000b3b0 tx=0x7fcdf0009e30 comp rx=0 tx=0).stop 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 msgr2=0x7fce0011a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 0x7fce0011a5a0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c810 tx=0x7fcdec004470 comp rx=0 tx=0).stop 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 shutdown_connections 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcde406d7f0 0x7fcde406fca0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fce00103dc0 0x7fce0011a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 --2- 172.21.15.1:0/1586892741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce001095f0 0x7fce0011a5a0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 >> 172.21.15.1:0/1586892741 conn(0x7fce00072d50 msgr2=0x7fce00074760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 shutdown_connections 2024-02-28T17:16:47.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:16:47.249+0000 7fce04a00700 1 -- 172.21.15.1:0/1586892741 wait complete. 2024-02-28T17:16:47.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:46 smithi066 ceph-mon[85781]: from='client.36252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:47.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:46 smithi066 ceph-mon[85781]: pgmap v1273: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:47.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:46 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3889218038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:16:47.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:46 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2982962314' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:16:48.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:47 smithi066 ceph-mon[85781]: from='client.36264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:48.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:47 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1586892741' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:16:48.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:47 smithi001 ceph-mon[112940]: from='client.36264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:16:48.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:47 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1586892741' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:16:49.187 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:48 smithi001 ceph-mon[112940]: pgmap v1274: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:49.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:48 smithi066 ceph-mon[85781]: pgmap v1274: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:51.059 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:50 smithi066 ceph-mon[85781]: pgmap v1275: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:51.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:50 smithi001 ceph-mon[112940]: pgmap v1275: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:53.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:52 smithi066 ceph-mon[85781]: pgmap v1276: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:53.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:52 smithi001 ceph-mon[112940]: pgmap v1276: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:55.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:54 smithi066 ceph-mon[85781]: pgmap v1277: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:54 smithi001 ceph-mon[112940]: pgmap v1277: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:57.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:56 smithi066 ceph-mon[85781]: pgmap v1278: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:57.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:16:57.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:56 smithi001 ceph-mon[112940]: pgmap v1278: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:57.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:16:59.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:16:58 smithi066 ceph-mon[85781]: pgmap v1279: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:16:59.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:16:58 smithi001 ceph-mon[112940]: pgmap v1279: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:01.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:00 smithi066 ceph-mon[85781]: pgmap v1280: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:01.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:00 smithi001 ceph-mon[112940]: pgmap v1280: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:03.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:02 smithi066 ceph-mon[85781]: pgmap v1281: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:03.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:02 smithi001 ceph-mon[112940]: pgmap v1281: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:05.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:04 smithi066 ceph-mon[85781]: pgmap v1282: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:05.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:04 smithi001 ceph-mon[112940]: pgmap v1282: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:07.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:06 smithi066 ceph-mon[85781]: pgmap v1283: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:07.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:06 smithi001 ceph-mon[112940]: pgmap v1283: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:09.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:08 smithi066 ceph-mon[85781]: pgmap v1284: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:09.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:08 smithi001 ceph-mon[112940]: pgmap v1284: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:11.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:10 smithi066 ceph-mon[85781]: pgmap v1285: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:11.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:17:11.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:10 smithi001 ceph-mon[112940]: pgmap v1285: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:11.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:17:13.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:12 smithi066 ceph-mon[85781]: pgmap v1286: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:13.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:12 smithi001 ceph-mon[112940]: pgmap v1286: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:15.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:14 smithi066 ceph-mon[85781]: pgmap v1287: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:15.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:14 smithi001 ceph-mon[112940]: pgmap v1287: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:17.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:16 smithi066 ceph-mon[85781]: pgmap v1288: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:17.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:16 smithi001 ceph-mon[112940]: pgmap v1288: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.418+0000 7fc627c00700 1 -- 172.21.15.1:0/1288851130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 msgr2=0x7fc620070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.418+0000 7fc627c00700 1 --2- 172.21.15.1:0/1288851130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 0x7fc620070700 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc614004970 tx=0x7fc614009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.418+0000 7fc627c00700 1 -- 172.21.15.1:0/1288851130 shutdown_connections 2024-02-28T17:17:17.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.418+0000 7fc627c00700 1 --2- 172.21.15.1:0/1288851130 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc620070c40 0x7fc620109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.418+0000 7fc627c00700 1 --2- 172.21.15.1:0/1288851130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 0x7fc620070700 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.418+0000 7fc627c00700 1 -- 172.21.15.1:0/1288851130 >> 172.21.15.1:0/1288851130 conn(0x7fc6200f8af0 msgr2=0x7fc6200faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:17.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.418+0000 7fc627c00700 1 -- 172.21.15.1:0/1288851130 shutdown_connections 2024-02-28T17:17:17.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.419+0000 7fc627c00700 1 -- 172.21.15.1:0/1288851130 wait complete. 2024-02-28T17:17:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.419+0000 7fc627c00700 1 Processor -- start 2024-02-28T17:17:17.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.420+0000 7fc627c00700 1 -- start start 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.420+0000 7fc627c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 0x7fc620115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.420+0000 7fc627c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc620070c40 0x7fc620116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.420+0000 7fc627c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc620116780 con 0x7fc620070330 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.420+0000 7fc627c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62010fad0 con 0x7fc620070c40 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.420+0000 7fc625200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 0x7fc620115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.420+0000 7fc625200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 0x7fc620115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45024/0 (socket says 172.21.15.1:45024) 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.420+0000 7fc625200700 1 -- 172.21.15.1:0/1265892936 learned_addr learned my addr 172.21.15.1:0/1265892936 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.421+0000 7fc61fe00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc620070c40 0x7fc620116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.421+0000 7fc625200700 1 -- 172.21.15.1:0/1265892936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc620070c40 msgr2=0x7fc620116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.421+0000 7fc625200700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc620070c40 0x7fc620116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.421+0000 7fc625200700 1 -- 172.21.15.1:0/1265892936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614005040 con 0x7fc620070330 2024-02-28T17:17:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.421+0000 7fc625200700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 0x7fc620115bf0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc614009150 tx=0x7fc614009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.421+0000 7fc61cc00700 1 -- 172.21.15.1:0/1265892936 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc61400e040 con 0x7fc620070330 2024-02-28T17:17:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.422+0000 7fc61cc00700 1 -- 172.21.15.1:0/1265892936 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc614004290 con 0x7fc620070330 2024-02-28T17:17:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.422+0000 7fc61cc00700 1 -- 172.21.15.1:0/1265892936 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc61401c630 con 0x7fc620070330 2024-02-28T17:17:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.422+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62010fd50 con 0x7fc620070330 2024-02-28T17:17:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.422+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc620110290 con 0x7fc620070330 2024-02-28T17:17:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.423+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc620106910 con 0x7fc620070330 2024-02-28T17:17:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.424+0000 7fc61cc00700 1 -- 172.21.15.1:0/1265892936 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc614013980 con 0x7fc620070330 2024-02-28T17:17:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.425+0000 7fc61cc00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc60806d7f0 0x7fc60806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.425+0000 7fc61cc00700 1 -- 172.21.15.1:0/1265892936 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc61408e1f0 con 0x7fc620070330 2024-02-28T17:17:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.426+0000 7fc61fe00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc60806d7f0 0x7fc60806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.426+0000 7fc61fe00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc60806d7f0 0x7fc60806fca0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc620111490 tx=0x7fc60c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:17.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.431+0000 7fc61cc00700 1 -- 172.21.15.1:0/1265892936 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc614057aa0 con 0x7fc620070330 2024-02-28T17:17:17.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.676+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc62004bf40 con 0x7fc60806d7f0 2024-02-28T17:17:17.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.678+0000 7fc61cc00700 1 -- 172.21.15.1:0/1265892936 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc62004bf40 con 0x7fc60806d7f0 2024-02-28T17:17:17.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc60806d7f0 msgr2=0x7fc60806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:17.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc60806d7f0 0x7fc60806fca0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc620111490 tx=0x7fc60c00b040 comp rx=0 tx=0).stop 2024-02-28T17:17:17.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 msgr2=0x7fc620115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:17.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 0x7fc620115bf0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc614009150 tx=0x7fc614009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 shutdown_connections 2024-02-28T17:17:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc60806d7f0 0x7fc60806fca0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc620070330 0x7fc620115bf0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 --2- 172.21.15.1:0/1265892936 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc620070c40 0x7fc620116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.681+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 >> 172.21.15.1:0/1265892936 conn(0x7fc6200f8af0 msgr2=0x7fc620103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:17.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.682+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 shutdown_connections 2024-02-28T17:17:17.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.682+0000 7fc627c00700 1 -- 172.21.15.1:0/1265892936 wait complete. 2024-02-28T17:17:17.702 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:17:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.836+0000 7f4c31200700 1 -- 172.21.15.1:0/3260836730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 msgr2=0x7f4c2c0ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:17.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.836+0000 7f4c31200700 1 --2- 172.21.15.1:0/3260836730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 0x7f4c2c0ff180 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4c18004940 tx=0x7f4c18009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:17.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.836+0000 7f4c31200700 1 -- 172.21.15.1:0/3260836730 shutdown_connections 2024-02-28T17:17:17.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.836+0000 7f4c31200700 1 --2- 172.21.15.1:0/3260836730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 0x7f4c2c0ff180 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.836+0000 7f4c31200700 1 --2- 172.21.15.1:0/3260836730 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c2c105170 0x7f4c2c105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.836+0000 7f4c31200700 1 -- 172.21.15.1:0/3260836730 >> 172.21.15.1:0/3260836730 conn(0x7f4c2c0face0 msgr2=0x7f4c2c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:17.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.836+0000 7f4c31200700 1 -- 172.21.15.1:0/3260836730 shutdown_connections 2024-02-28T17:17:17.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.836+0000 7f4c31200700 1 -- 172.21.15.1:0/3260836730 wait complete. 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c31200700 1 Processor -- start 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c31200700 1 -- start start 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c31200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c2c105170 0x7f4c2c115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c31200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 0x7f4c2c116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c31200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c116870 con 0x7f4c2c105b10 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c31200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c10fb80 con 0x7f4c2c105170 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c29400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 0x7f4c2c116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c29400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 0x7f4c2c116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45044/0 (socket says 172.21.15.1:45044) 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.837+0000 7f4c29400700 1 -- 172.21.15.1:0/931561094 learned_addr learned my addr 172.21.15.1:0/931561094 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.838+0000 7f4c29e00700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c2c105170 0x7f4c2c115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.838+0000 7f4c29400700 1 -- 172.21.15.1:0/931561094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c2c105170 msgr2=0x7f4c2c115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.838+0000 7f4c29400700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c2c105170 0x7f4c2c115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.838+0000 7f4c29400700 1 -- 172.21.15.1:0/931561094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c18005040 con 0x7f4c2c105b10 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.838+0000 7f4c29e00700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c2c105170 0x7f4c2c115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.838+0000 7f4c29400700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 0x7f4c2c116220 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009030 tx=0x7f4c18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.838+0000 7f4c22000700 1 -- 172.21.15.1:0/931561094 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c1800e040 con 0x7f4c2c105b10 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.839+0000 7f4c22000700 1 -- 172.21.15.1:0/931561094 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c1800bbf0 con 0x7f4c2c105b10 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.839+0000 7f4c22000700 1 -- 172.21.15.1:0/931561094 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c1801c630 con 0x7f4c2c105b10 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.839+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c10fe00 con 0x7f4c2c105b10 2024-02-28T17:17:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.839+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c110340 con 0x7f4c2c105b10 2024-02-28T17:17:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.840+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c107640 con 0x7f4c2c105b10 2024-02-28T17:17:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.841+0000 7f4c22000700 1 -- 172.21.15.1:0/931561094 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4c1800bd60 con 0x7f4c2c105b10 2024-02-28T17:17:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.842+0000 7f4c22000700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c1006d7f0 0x7f4c1006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.842+0000 7f4c22000700 1 -- 172.21.15.1:0/931561094 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4c1808e1f0 con 0x7f4c2c105b10 2024-02-28T17:17:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.843+0000 7f4c29e00700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c1006d7f0 0x7f4c1006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.843+0000 7f4c29e00700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c1006d7f0 0x7f4c1006fca0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009fd0 tx=0x7f4c1c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:17.847+0000 7f4c22000700 1 -- 172.21.15.1:0/931561094 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c1805b5d0 con 0x7f4c2c105b10 2024-02-28T17:17:18.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.081+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c2c1112c0 con 0x7f4c1006d7f0 2024-02-28T17:17:18.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.084+0000 7f4c22000700 1 -- 172.21.15.1:0/931561094 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4c2c1112c0 con 0x7f4c1006d7f0 2024-02-28T17:17:18.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c1006d7f0 msgr2=0x7f4c1006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c1006d7f0 0x7f4c1006fca0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009fd0 tx=0x7f4c1c00b040 comp rx=0 tx=0).stop 2024-02-28T17:17:18.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 msgr2=0x7f4c2c116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 0x7f4c2c116220 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009030 tx=0x7f4c18009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:18.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 shutdown_connections 2024-02-28T17:17:18.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4c1006d7f0 0x7f4c1006fca0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4c2c105170 0x7f4c2c115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 --2- 172.21.15.1:0/931561094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c2c105b10 0x7f4c2c116220 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 >> 172.21.15.1:0/931561094 conn(0x7f4c2c0face0 msgr2=0x7f4c2c0fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:18.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 shutdown_connections 2024-02-28T17:17:18.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.087+0000 7f4c31200700 1 -- 172.21.15.1:0/931561094 wait complete. 2024-02-28T17:17:18.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.235+0000 7efc31400700 1 -- 172.21.15.1:0/3534449219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 msgr2=0x7efc2c109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.235+0000 7efc31400700 1 --2- 172.21.15.1:0/3534449219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 0x7efc2c109250 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7efc14004970 tx=0x7efc14009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:18.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.235+0000 7efc31400700 1 -- 172.21.15.1:0/3534449219 shutdown_connections 2024-02-28T17:17:18.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.235+0000 7efc31400700 1 --2- 172.21.15.1:0/3534449219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 0x7efc2c109250 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.235+0000 7efc31400700 1 --2- 172.21.15.1:0/3534449219 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc2c070330 0x7efc2c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.235+0000 7efc31400700 1 -- 172.21.15.1:0/3534449219 >> 172.21.15.1:0/3534449219 conn(0x7efc2c0f8af0 msgr2=0x7efc2c0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:18.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.235+0000 7efc31400700 1 -- 172.21.15.1:0/3534449219 shutdown_connections 2024-02-28T17:17:18.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.235+0000 7efc31400700 1 -- 172.21.15.1:0/3534449219 wait complete. 2024-02-28T17:17:18.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.236+0000 7efc31400700 1 Processor -- start 2024-02-28T17:17:18.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.236+0000 7efc31400700 1 -- start start 2024-02-28T17:17:18.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.236+0000 7efc31400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc2c070330 0x7efc2c115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:18.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.236+0000 7efc31400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 0x7efc2c1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:18.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.236+0000 7efc31400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c1168d0 con 0x7efc2c070c40 2024-02-28T17:17:18.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.236+0000 7efc31400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c10fb50 con 0x7efc2c070330 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.237+0000 7efc23e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 0x7efc2c1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.237+0000 7efc23e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 0x7efc2c1161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45056/0 (socket says 172.21.15.1:45056) 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.237+0000 7efc23e00700 1 -- 172.21.15.1:0/1321220641 learned_addr learned my addr 172.21.15.1:0/1321220641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.237+0000 7efc23e00700 1 -- 172.21.15.1:0/1321220641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc2c070330 msgr2=0x7efc2c115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.237+0000 7efc2a800700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc2c070330 0x7efc2c115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.237+0000 7efc23e00700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc2c070330 0x7efc2c115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.237+0000 7efc23e00700 1 -- 172.21.15.1:0/1321220641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc14005040 con 0x7efc2c070c40 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.237+0000 7efc23e00700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 0x7efc2c1161f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7efc14009150 tx=0x7efc14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.238+0000 7efc22a00700 1 -- 172.21.15.1:0/1321220641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc1400e040 con 0x7efc2c070c40 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.238+0000 7efc22a00700 1 -- 172.21.15.1:0/1321220641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc14004290 con 0x7efc2c070c40 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.238+0000 7efc22a00700 1 -- 172.21.15.1:0/1321220641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc1401c630 con 0x7efc2c070c40 2024-02-28T17:17:18.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.238+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2c10fd70 con 0x7efc2c070c40 2024-02-28T17:17:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.238+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2c1102b0 con 0x7efc2c070c40 2024-02-28T17:17:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.239+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2c106910 con 0x7efc2c070c40 2024-02-28T17:17:18.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.240+0000 7efc22a00700 1 -- 172.21.15.1:0/1321220641 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7efc14013980 con 0x7efc2c070c40 2024-02-28T17:17:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.241+0000 7efc22a00700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efc1006d7f0 0x7efc1006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:18.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.241+0000 7efc22a00700 1 -- 172.21.15.1:0/1321220641 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efc1408e0c0 con 0x7efc2c070c40 2024-02-28T17:17:18.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.241+0000 7efc2a800700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efc1006d7f0 0x7efc1006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:18.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.242+0000 7efc2a800700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efc1006d7f0 0x7efc1006fca0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efc1c006fd0 tx=0x7efc1c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:18.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.246+0000 7efc22a00700 1 -- 172.21.15.1:0/1321220641 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc1405b560 con 0x7efc2c070c40 2024-02-28T17:17:18.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.471+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efc2c111290 con 0x7efc1006d7f0 2024-02-28T17:17:18.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.501+0000 7efc22a00700 1 -- 172.21.15.1:0/1321220641 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7efc2c111290 con 0x7efc1006d7f0 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (43m) 4m ago 57m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (40m) 4m ago 57m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (40m) 4m ago 56m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (43m) 4m ago 57m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (36m) 4m ago 50m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (35m) 4m ago 50m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (5m) 4m ago 50m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (20m) 4m ago 50m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (45m) 4m ago 59m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:17:18.506 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (44m) 4m ago 56m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (41m) 4m ago 59m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (41m) 4m ago 56m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (44m) 4m ago 57m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (44m) 4m ago 56m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (39m) 4m ago 55m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (38m) 4m ago 54m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (38m) 4m ago 54m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (37m) 4m ago 53m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (37m) 4m ago 53m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (36m) 4m ago 52m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:17:18.507 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (43m) 4m ago 57m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efc1006d7f0 msgr2=0x7efc1006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efc1006d7f0 0x7efc1006fca0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efc1c006fd0 tx=0x7efc1c008040 comp rx=0 tx=0).stop 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 msgr2=0x7efc2c1161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 0x7efc2c1161f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7efc14009150 tx=0x7efc14009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 shutdown_connections 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efc1006d7f0 0x7efc1006fca0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc2c070330 0x7efc2c115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 --2- 172.21.15.1:0/1321220641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2c070c40 0x7efc2c1161f0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.504+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 >> 172.21.15.1:0/1321220641 conn(0x7efc2c0f8af0 msgr2=0x7efc2c103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.505+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 shutdown_connections 2024-02-28T17:17:18.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.505+0000 7efc31400700 1 -- 172.21.15.1:0/1321220641 wait complete. 2024-02-28T17:17:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.657+0000 7f0a74c00700 1 -- 172.21.15.1:0/3584879229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 msgr2=0x7f0a700ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.657+0000 7f0a74c00700 1 --2- 172.21.15.1:0/3584879229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 0x7f0a700ff180 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c004940 tx=0x7f0a5c009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.657+0000 7f0a74c00700 1 -- 172.21.15.1:0/3584879229 shutdown_connections 2024-02-28T17:17:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.657+0000 7f0a74c00700 1 --2- 172.21.15.1:0/3584879229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 0x7f0a700ff180 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.657+0000 7f0a74c00700 1 --2- 172.21.15.1:0/3584879229 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a70105170 0x7f0a70105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.657+0000 7f0a74c00700 1 -- 172.21.15.1:0/3584879229 >> 172.21.15.1:0/3584879229 conn(0x7f0a700face0 msgr2=0x7f0a700fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.658+0000 7f0a74c00700 1 -- 172.21.15.1:0/3584879229 shutdown_connections 2024-02-28T17:17:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.658+0000 7f0a74c00700 1 -- 172.21.15.1:0/3584879229 wait complete. 2024-02-28T17:17:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.659+0000 7f0a74c00700 1 Processor -- start 2024-02-28T17:17:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.659+0000 7f0a74c00700 1 -- start start 2024-02-28T17:17:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.659+0000 7f0a74c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a70105170 0x7f0a70115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.659+0000 7f0a74c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 0x7f0a70116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.659+0000 7f0a74c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70116870 con 0x7f0a70105b10 2024-02-28T17:17:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.659+0000 7f0a74c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7010fb80 con 0x7f0a70105170 2024-02-28T17:17:18.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.660+0000 7f0a6d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 0x7f0a70116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.660+0000 7f0a6d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 0x7f0a70116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45076/0 (socket says 172.21.15.1:45076) 2024-02-28T17:17:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.660+0000 7f0a6d400700 1 -- 172.21.15.1:0/1175035973 learned_addr learned my addr 172.21.15.1:0/1175035973 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.660+0000 7f0a6de00700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a70105170 0x7f0a70115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.660+0000 7f0a6d400700 1 -- 172.21.15.1:0/1175035973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a70105170 msgr2=0x7f0a70115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.660+0000 7f0a6d400700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a70105170 0x7f0a70115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.660+0000 7f0a6d400700 1 -- 172.21.15.1:0/1175035973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c005040 con 0x7f0a70105b10 2024-02-28T17:17:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.660+0000 7f0a6d400700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 0x7f0a70116220 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009030 tx=0x7f0a5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.661+0000 7f0a66000700 1 -- 172.21.15.1:0/1175035973 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a5c00e040 con 0x7f0a70105b10 2024-02-28T17:17:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.661+0000 7f0a66000700 1 -- 172.21.15.1:0/1175035973 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a5c00bbf0 con 0x7f0a70105b10 2024-02-28T17:17:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.661+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7010fe00 con 0x7f0a70105b10 2024-02-28T17:17:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.661+0000 7f0a66000700 1 -- 172.21.15.1:0/1175035973 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a5c01c630 con 0x7f0a70105b10 2024-02-28T17:17:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.661+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a70110340 con 0x7f0a70105b10 2024-02-28T17:17:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.662+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a70107640 con 0x7f0a70105b10 2024-02-28T17:17:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.663+0000 7f0a66000700 1 -- 172.21.15.1:0/1175035973 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0a5c00bd60 con 0x7f0a70105b10 2024-02-28T17:17:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.664+0000 7f0a66000700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a5406d7f0 0x7f0a5406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.664+0000 7f0a66000700 1 -- 172.21.15.1:0/1175035973 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0a5c08e2c0 con 0x7f0a70105b10 2024-02-28T17:17:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.664+0000 7f0a6de00700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a5406d7f0 0x7f0a5406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.665+0000 7f0a6de00700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a5406d7f0 0x7f0a5406fca0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0a60004d60 tx=0x7f0a6000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:18.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.669+0000 7f0a66000700 1 -- 172.21.15.1:0/1175035973 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0a5c05b620 con 0x7f0a70105b10 2024-02-28T17:17:18.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.988+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0a700fe900 con 0x7f0a70105b10 2024-02-28T17:17:18.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.989+0000 7f0a66000700 1 -- 172.21.15.1:0/1175035973 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0a5c05b1b0 con 0x7f0a70105b10 2024-02-28T17:17:18.993 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:17:18.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:17:18.993 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:17:18.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:17:18.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:17:18.994 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:17:18.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a5406d7f0 msgr2=0x7f0a5406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a5406d7f0 0x7f0a5406fca0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0a60004d60 tx=0x7f0a6000b040 comp rx=0 tx=0).stop 2024-02-28T17:17:18.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 msgr2=0x7f0a70116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:18.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 0x7f0a70116220 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009030 tx=0x7f0a5c009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:18.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 shutdown_connections 2024-02-28T17:17:18.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0a5406d7f0 0x7f0a5406fca0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0a70105170 0x7f0a70115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 --2- 172.21.15.1:0/1175035973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a70105b10 0x7f0a70116220 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:18.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.991+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 >> 172.21.15.1:0/1175035973 conn(0x7f0a700face0 msgr2=0x7f0a700fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:18.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.992+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 shutdown_connections 2024-02-28T17:17:18.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:18.992+0000 7f0a74c00700 1 -- 172.21.15.1:0/1175035973 wait complete. 2024-02-28T17:17:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.146+0000 7f1579a00700 1 -- 172.21.15.1:0/2927518780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 msgr2=0x7f1574070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.146+0000 7f1579a00700 1 --2- 172.21.15.1:0/2927518780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 0x7f1574070ce0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f1560004970 tx=0x7f1560009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.146+0000 7f1579a00700 1 -- 172.21.15.1:0/2927518780 shutdown_connections 2024-02-28T17:17:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.146+0000 7f1579a00700 1 --2- 172.21.15.1:0/2927518780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 0x7f1574070ce0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.146+0000 7f1579a00700 1 --2- 172.21.15.1:0/2927518780 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574071f60 0x7f1574070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.146+0000 7f1579a00700 1 -- 172.21.15.1:0/2927518780 >> 172.21.15.1:0/2927518780 conn(0x7f15740f8af0 msgr2=0x7f15740faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:19.152 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:18 smithi001 ceph-mon[112940]: pgmap v1289: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:19.152 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:18 smithi001 ceph-mon[112940]: from='client.36272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:19.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.147+0000 7f1579a00700 1 -- 172.21.15.1:0/2927518780 shutdown_connections 2024-02-28T17:17:19.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.147+0000 7f1579a00700 1 -- 172.21.15.1:0/2927518780 wait complete. 2024-02-28T17:17:19.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.148+0000 7f1579a00700 1 Processor -- start 2024-02-28T17:17:19.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.148+0000 7f1579a00700 1 -- start start 2024-02-28T17:17:19.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.149+0000 7f1579a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 0x7f157411a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.149+0000 7f1579a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574071f60 0x7f157411a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.149+0000 7f1579a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157411ad00 con 0x7f1574070870 2024-02-28T17:17:19.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.149+0000 7f1579a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1574113f80 con 0x7f1574071f60 2024-02-28T17:17:19.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.149+0000 7f1572a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 0x7f157411a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.149+0000 7f1572a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 0x7f157411a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45102/0 (socket says 172.21.15.1:45102) 2024-02-28T17:17:19.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.149+0000 7f1572a00700 1 -- 172.21.15.1:0/1513274484 learned_addr learned my addr 172.21.15.1:0/1513274484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:19.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.150+0000 7f1572000700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574071f60 0x7f157411a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.150+0000 7f1572a00700 1 -- 172.21.15.1:0/1513274484 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574071f60 msgr2=0x7f157411a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.150+0000 7f1572a00700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574071f60 0x7f157411a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.150+0000 7f1572a00700 1 -- 172.21.15.1:0/1513274484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560005040 con 0x7f1574070870 2024-02-28T17:17:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.150+0000 7f1572a00700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 0x7f157411a0e0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f156400b790 tx=0x7f1564009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.150+0000 7f156aa00700 1 -- 172.21.15.1:0/1513274484 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f156400f770 con 0x7f1574070870 2024-02-28T17:17:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.150+0000 7f156aa00700 1 -- 172.21.15.1:0/1513274484 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f156400fda0 con 0x7f1574070870 2024-02-28T17:17:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.151+0000 7f156aa00700 1 -- 172.21.15.1:0/1513274484 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f156400d310 con 0x7f1574070870 2024-02-28T17:17:19.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.151+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1574114200 con 0x7f1574070870 2024-02-28T17:17:19.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.151+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1574114710 con 0x7f1574070870 2024-02-28T17:17:19.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.152+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15740fe410 con 0x7f1574070870 2024-02-28T17:17:19.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.153+0000 7f156aa00700 1 -- 172.21.15.1:0/1513274484 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1564010420 con 0x7f1574070870 2024-02-28T17:17:19.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.154+0000 7f156aa00700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155806d7f0 0x7f155806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.154+0000 7f156aa00700 1 -- 172.21.15.1:0/1513274484 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f156408cec0 con 0x7f1574070870 2024-02-28T17:17:19.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.154+0000 7f1572000700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155806d7f0 0x7f155806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.155+0000 7f1572000700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155806d7f0 0x7f155806fca0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1560009fd0 tx=0x7f1560009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:19.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.159+0000 7f156aa00700 1 -- 172.21.15.1:0/1513274484 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f156405a2a0 con 0x7f1574070870 2024-02-28T17:17:19.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:18 smithi066 ceph-mon[85781]: pgmap v1289: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:19.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:18 smithi066 ceph-mon[85781]: from='client.36272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:19.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.416+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f15741156c0 con 0x7f1574070870 2024-02-28T17:17:19.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.417+0000 7f156aa00700 1 -- 172.21.15.1:0/1513274484 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f1564059e30 con 0x7f1574070870 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:17:19.422 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:17:19.423 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:17:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:17:19.425 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:17:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155806d7f0 msgr2=0x7f155806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155806d7f0 0x7f155806fca0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1560009fd0 tx=0x7f1560009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 msgr2=0x7f157411a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 0x7f157411a0e0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f156400b790 tx=0x7f1564009250 comp rx=0 tx=0).stop 2024-02-28T17:17:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 shutdown_connections 2024-02-28T17:17:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f155806d7f0 0x7f155806fca0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1574070870 0x7f157411a0e0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 --2- 172.21.15.1:0/1513274484 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1574071f60 0x7f157411a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 >> 172.21.15.1:0/1513274484 conn(0x7f15740f8af0 msgr2=0x7f15740fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 shutdown_connections 2024-02-28T17:17:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.421+0000 7f1579a00700 1 -- 172.21.15.1:0/1513274484 wait complete. 2024-02-28T17:17:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:17:19.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.573+0000 7fba03400700 1 -- 172.21.15.1:0/4030185068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 msgr2=0x7fb9fc070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.573+0000 7fba03400700 1 --2- 172.21.15.1:0/4030185068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 0x7fb9fc070700 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0004970 tx=0x7fb9f0009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:19.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.573+0000 7fba03400700 1 -- 172.21.15.1:0/4030185068 shutdown_connections 2024-02-28T17:17:19.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.573+0000 7fba03400700 1 --2- 172.21.15.1:0/4030185068 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9fc070c40 0x7fb9fc109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.573+0000 7fba03400700 1 --2- 172.21.15.1:0/4030185068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 0x7fb9fc070700 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.573+0000 7fba03400700 1 -- 172.21.15.1:0/4030185068 >> 172.21.15.1:0/4030185068 conn(0x7fb9fc0f8ae0 msgr2=0x7fb9fc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:19.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.574+0000 7fba03400700 1 -- 172.21.15.1:0/4030185068 shutdown_connections 2024-02-28T17:17:19.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.574+0000 7fba03400700 1 -- 172.21.15.1:0/4030185068 wait complete. 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.574+0000 7fba03400700 1 Processor -- start 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.575+0000 7fba03400700 1 -- start start 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.575+0000 7fba03400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 0x7fb9fc115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.575+0000 7fba03400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9fc070c40 0x7fb9fc116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.575+0000 7fba03400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc1167b0 con 0x7fb9fc070330 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.575+0000 7fba03400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc10fae0 con 0x7fb9fc070c40 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.575+0000 7fba00a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 0x7fb9fc115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.575+0000 7fba00a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 0x7fb9fc115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45120/0 (socket says 172.21.15.1:45120) 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.575+0000 7fba00a00700 1 -- 172.21.15.1:0/1550626524 learned_addr learned my addr 172.21.15.1:0/1550626524 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fb9fbe00700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9fc070c40 0x7fb9fc116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fba00a00700 1 -- 172.21.15.1:0/1550626524 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9fc070c40 msgr2=0x7fb9fc116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fba00a00700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9fc070c40 0x7fb9fc116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fba00a00700 1 -- 172.21.15.1:0/1550626524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f0005040 con 0x7fb9fc070330 2024-02-28T17:17:19.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fb9fbe00700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9fc070c40 0x7fb9fc116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:17:19.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fba00a00700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 0x7fb9fc115c20 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0009150 tx=0x7fb9f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:19.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fb9f8c00700 1 -- 172.21.15.1:0/1550626524 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f000e040 con 0x7fb9fc070330 2024-02-28T17:17:19.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fb9f8c00700 1 -- 172.21.15.1:0/1550626524 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9f0004290 con 0x7fb9fc070330 2024-02-28T17:17:19.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fb9f8c00700 1 -- 172.21.15.1:0/1550626524 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f001c630 con 0x7fb9fc070330 2024-02-28T17:17:19.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9fc10fd60 con 0x7fb9fc070330 2024-02-28T17:17:19.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.576+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc1102a0 con 0x7fb9fc070330 2024-02-28T17:17:19.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.577+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9fc106920 con 0x7fb9fc070330 2024-02-28T17:17:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.578+0000 7fb9f8c00700 1 -- 172.21.15.1:0/1550626524 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb9f0013980 con 0x7fb9fc070330 2024-02-28T17:17:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.579+0000 7fb9f8c00700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb9e406d7f0 0x7fb9e406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.579+0000 7fb9f8c00700 1 -- 172.21.15.1:0/1550626524 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb9f008e1f0 con 0x7fb9fc070330 2024-02-28T17:17:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.580+0000 7fb9fbe00700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb9e406d7f0 0x7fb9e406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.580+0000 7fb9fbe00700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb9e406d7f0 0x7fb9e406fca0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009fd0 tx=0x7fb9e800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:19.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.584+0000 7fb9f8c00700 1 -- 172.21.15.1:0/1550626524 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb9f005b5d0 con 0x7fb9fc070330 2024-02-28T17:17:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.814+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9fc04bed0 con 0x7fb9e406d7f0 2024-02-28T17:17:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.816+0000 7fb9f8c00700 1 -- 172.21.15.1:0/1550626524 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb9fc04bed0 con 0x7fb9e406d7f0 2024-02-28T17:17:19.820 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:17:19.820 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:17:19.820 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:17:19.821 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:17:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.818+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb9e406d7f0 msgr2=0x7fb9e406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.818+0000 7fba03400700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb9e406d7f0 0x7fb9e406fca0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009fd0 tx=0x7fb9e800b040 comp rx=0 tx=0).stop 2024-02-28T17:17:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.818+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 msgr2=0x7fb9fc115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.818+0000 7fba03400700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 0x7fb9fc115c20 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0009150 tx=0x7fb9f0009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.819+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 shutdown_connections 2024-02-28T17:17:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.819+0000 7fba03400700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb9e406d7f0 0x7fb9e406fca0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.819+0000 7fba03400700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9fc070330 0x7fb9fc115c20 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.819+0000 7fba03400700 1 --2- 172.21.15.1:0/1550626524 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb9fc070c40 0x7fb9fc116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.819+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 >> 172.21.15.1:0/1550626524 conn(0x7fb9fc0f8ae0 msgr2=0x7fb9fc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.819+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 shutdown_connections 2024-02-28T17:17:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.819+0000 7fba03400700 1 -- 172.21.15.1:0/1550626524 wait complete. 2024-02-28T17:17:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.978+0000 7f492da00700 1 -- 172.21.15.1:0/3074883203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 msgr2=0x7f4928104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.978+0000 7f492da00700 1 --2- 172.21.15.1:0/3074883203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 0x7f4928104210 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4914004940 tx=0x7f4914009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.978+0000 7f492da00700 1 -- 172.21.15.1:0/3074883203 shutdown_connections 2024-02-28T17:17:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.978+0000 7f492da00700 1 --2- 172.21.15.1:0/3074883203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 0x7f4928104210 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.978+0000 7f492da00700 1 --2- 172.21.15.1:0/3074883203 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49281095f0 0x7f49281099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.978+0000 7f492da00700 1 -- 172.21.15.1:0/3074883203 >> 172.21.15.1:0/3074883203 conn(0x7f4928072d50 msgr2=0x7f4928075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.978+0000 7f492da00700 1 -- 172.21.15.1:0/3074883203 shutdown_connections 2024-02-28T17:17:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.978+0000 7f492da00700 1 -- 172.21.15.1:0/3074883203 wait complete. 2024-02-28T17:17:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.979+0000 7f492da00700 1 Processor -- start 2024-02-28T17:17:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.980+0000 7f492da00700 1 -- start start 2024-02-28T17:17:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.980+0000 7f492da00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 0x7f492811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.980+0000 7f492da00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49281095f0 0x7f492811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.980+0000 7f492da00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492811abd0 con 0x7f4928103dc0 2024-02-28T17:17:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.980+0000 7f492da00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4928113ee0 con 0x7f49281095f0 2024-02-28T17:17:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.980+0000 7f4926a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 0x7f492811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.981+0000 7f4926a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 0x7f492811a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45152/0 (socket says 172.21.15.1:45152) 2024-02-28T17:17:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.981+0000 7f4926a00700 1 -- 172.21.15.1:0/579967020 learned_addr learned my addr 172.21.15.1:0/579967020 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.981+0000 7f4926000700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49281095f0 0x7f492811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.981+0000 7f4926a00700 1 -- 172.21.15.1:0/579967020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49281095f0 msgr2=0x7f492811a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.981+0000 7f4926a00700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49281095f0 0x7f492811a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.981+0000 7f4926a00700 1 -- 172.21.15.1:0/579967020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4914005040 con 0x7f4928103dc0 2024-02-28T17:17:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.982+0000 7f4926a00700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 0x7f492811a040 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4918002a50 tx=0x7f49180046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.982+0000 7f491ea00700 1 -- 172.21.15.1:0/579967020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4918007cc0 con 0x7f4928103dc0 2024-02-28T17:17:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.982+0000 7f491ea00700 1 -- 172.21.15.1:0/579967020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f491800f040 con 0x7f4928103dc0 2024-02-28T17:17:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.982+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49281141c0 con 0x7f4928103dc0 2024-02-28T17:17:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.982+0000 7f491ea00700 1 -- 172.21.15.1:0/579967020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4918009640 con 0x7f4928103dc0 2024-02-28T17:17:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.982+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4928114700 con 0x7f4928103dc0 2024-02-28T17:17:19.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.983+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492810b9c0 con 0x7f4928103dc0 2024-02-28T17:17:19.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.985+0000 7f491ea00700 1 -- 172.21.15.1:0/579967020 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f49180097a0 con 0x7f4928103dc0 2024-02-28T17:17:19.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.986+0000 7f491ea00700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f490c06d7f0 0x7f490c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.986+0000 7f491ea00700 1 -- 172.21.15.1:0/579967020 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f491808cc80 con 0x7f4928103dc0 2024-02-28T17:17:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.986+0000 7f4926000700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f490c06d7f0 0x7f490c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:19.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.987+0000 7f4926000700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f490c06d7f0 0x7f490c06fca0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f49281158e0 tx=0x7f4914009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:19.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:19.991+0000 7f491ea00700 1 -- 172.21.15.1:0/579967020 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f491805a000 con 0x7f4928103dc0 2024-02-28T17:17:20.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:19 smithi066 ceph-mon[85781]: from='client.36276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:20.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:19 smithi066 ceph-mon[85781]: from='client.36280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:20.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:19 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1175035973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:17:20.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:19 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1513274484' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:17:20.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:19 smithi001 ceph-mon[112940]: from='client.36276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:20.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:19 smithi001 ceph-mon[112940]: from='client.36280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:20.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:19 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1175035973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:17:20.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:19 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1513274484' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:17:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.314+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4928075870 con 0x7f4928103dc0 2024-02-28T17:17:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.315+0000 7f491ea00700 1 -- 172.21.15.1:0/579967020 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f4918059b90 con 0x7f4928103dc0 2024-02-28T17:17:20.320 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:17:20.320 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:17:20.320 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:17:20.321 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:17:20.322 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:17:20.322 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:17:20.322 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:17:20.322 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:17:20.322 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:17:20.322 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:17:20.322 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:17:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f490c06d7f0 msgr2=0x7f490c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f490c06d7f0 0x7f490c06fca0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f49281158e0 tx=0x7f4914009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 msgr2=0x7f492811a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 0x7f492811a040 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4918002a50 tx=0x7f49180046b0 comp rx=0 tx=0).stop 2024-02-28T17:17:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 shutdown_connections 2024-02-28T17:17:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f490c06d7f0 0x7f490c06fca0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4928103dc0 0x7f492811a040 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 --2- 172.21.15.1:0/579967020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f49281095f0 0x7f492811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.319+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 >> 172.21.15.1:0/579967020 conn(0x7f4928072d50 msgr2=0x7f4928074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.320+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 shutdown_connections 2024-02-28T17:17:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:20.320+0000 7f492da00700 1 -- 172.21.15.1:0/579967020 wait complete. 2024-02-28T17:17:21.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:20 smithi066 ceph-mon[85781]: pgmap v1290: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:21.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:20 smithi066 ceph-mon[85781]: from='client.36292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:21.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:20 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/579967020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:17:21.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:20 smithi001 ceph-mon[112940]: pgmap v1290: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:21.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:20 smithi001 ceph-mon[112940]: from='client.36292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:21.286 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:20 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/579967020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:17:23.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:22 smithi066 ceph-mon[85781]: pgmap v1291: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:23.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:22 smithi001 ceph-mon[112940]: pgmap v1291: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:25.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:24 smithi066 ceph-mon[85781]: pgmap v1292: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:25.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:24 smithi001 ceph-mon[112940]: pgmap v1292: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:26.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:17:26.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:17:27.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:26 smithi066 ceph-mon[85781]: pgmap v1293: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:27.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:26 smithi001 ceph-mon[112940]: pgmap v1293: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:29.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:28 smithi066 ceph-mon[85781]: pgmap v1294: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:29.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:28 smithi001 ceph-mon[112940]: pgmap v1294: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:31.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:30 smithi066 ceph-mon[85781]: pgmap v1295: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:31.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:30 smithi001 ceph-mon[112940]: pgmap v1295: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:33.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:32 smithi066 ceph-mon[85781]: pgmap v1296: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:33.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:32 smithi001 ceph-mon[112940]: pgmap v1296: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:35.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:34 smithi066 ceph-mon[85781]: pgmap v1297: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:35.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:34 smithi001 ceph-mon[112940]: pgmap v1297: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:37.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:36 smithi066 ceph-mon[85781]: pgmap v1298: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:37.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:36 smithi001 ceph-mon[112940]: pgmap v1298: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:39.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:38 smithi066 ceph-mon[85781]: pgmap v1299: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:39.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:38 smithi001 ceph-mon[112940]: pgmap v1299: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:40 smithi066 ceph-mon[85781]: pgmap v1300: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:17:41.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:40 smithi001 ceph-mon[112940]: pgmap v1300: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:41.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:17:43.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:42 smithi066 ceph-mon[85781]: pgmap v1301: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:43.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:42 smithi001 ceph-mon[112940]: pgmap v1301: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:45.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:44 smithi066 ceph-mon[85781]: pgmap v1302: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:45.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:44 smithi001 ceph-mon[112940]: pgmap v1302: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:47.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:46 smithi066 ceph-mon[85781]: pgmap v1303: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:47.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:46 smithi001 ceph-mon[112940]: pgmap v1303: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:49.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:49 smithi066 ceph-mon[85781]: pgmap v1304: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:49 smithi001 ceph-mon[112940]: pgmap v1304: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:50.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.491+0000 7fb120e00700 1 -- 172.21.15.1:0/3081080267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105b00 msgr2=0x7fb11c0ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:50.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.491+0000 7fb120e00700 1 --2- 172.21.15.1:0/3081080267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105b00 0x7fb11c0ff170 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb108004940 tx=0x7fb108009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:50.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.491+0000 7fb120e00700 1 -- 172.21.15.1:0/3081080267 shutdown_connections 2024-02-28T17:17:50.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.491+0000 7fb120e00700 1 --2- 172.21.15.1:0/3081080267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105b00 0x7fb11c0ff170 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.491+0000 7fb120e00700 1 --2- 172.21.15.1:0/3081080267 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb11c105160 0x7fb11c105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.491+0000 7fb120e00700 1 -- 172.21.15.1:0/3081080267 >> 172.21.15.1:0/3081080267 conn(0x7fb11c0fad30 msgr2=0x7fb11c0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:50.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.491+0000 7fb120e00700 1 -- 172.21.15.1:0/3081080267 shutdown_connections 2024-02-28T17:17:50.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.492+0000 7fb120e00700 1 -- 172.21.15.1:0/3081080267 wait complete. 2024-02-28T17:17:50.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.492+0000 7fb120e00700 1 Processor -- start 2024-02-28T17:17:50.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.492+0000 7fb120e00700 1 -- start start 2024-02-28T17:17:50.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.493+0000 7fb120e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105160 0x7fb11c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:50.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.493+0000 7fb120e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb11c105b00 0x7fb11c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:50.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.493+0000 7fb120e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c116860 con 0x7fb11c105160 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.493+0000 7fb120e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c10fbb0 con 0x7fb11c105b00 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.493+0000 7fb119e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105160 0x7fb11c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.493+0000 7fb119e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105160 0x7fb11c115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49870/0 (socket says 172.21.15.1:49870) 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.493+0000 7fb119e00700 1 -- 172.21.15.1:0/1263746971 learned_addr learned my addr 172.21.15.1:0/1263746971 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.493+0000 7fb119400700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb11c105b00 0x7fb11c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.494+0000 7fb119e00700 1 -- 172.21.15.1:0/1263746971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb11c105b00 msgr2=0x7fb11c116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.494+0000 7fb119e00700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb11c105b00 0x7fb11c116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.494+0000 7fb119e00700 1 -- 172.21.15.1:0/1263746971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb108005040 con 0x7fb11c105160 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.494+0000 7fb119e00700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105160 0x7fb11c115cd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fb10c002a50 tx=0x7fb10c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.494+0000 7fb112000700 1 -- 172.21.15.1:0/1263746971 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb10c007cc0 con 0x7fb11c105160 2024-02-28T17:17:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.494+0000 7fb112000700 1 -- 172.21.15.1:0/1263746971 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb10c00f040 con 0x7fb11c105160 2024-02-28T17:17:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.494+0000 7fb112000700 1 -- 172.21.15.1:0/1263746971 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb10c009640 con 0x7fb11c105160 2024-02-28T17:17:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.494+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c10fe90 con 0x7fb11c105160 2024-02-28T17:17:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.495+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c1103d0 con 0x7fb11c105160 2024-02-28T17:17:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.496+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c107630 con 0x7fb11c105160 2024-02-28T17:17:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.497+0000 7fb112000700 1 -- 172.21.15.1:0/1263746971 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb10c0097a0 con 0x7fb11c105160 2024-02-28T17:17:50.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.498+0000 7fb112000700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10006d7f0 0x7fb10006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.498+0000 7fb112000700 1 -- 172.21.15.1:0/1263746971 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb10c08cc60 con 0x7fb11c105160 2024-02-28T17:17:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.498+0000 7fb119400700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10006d7f0 0x7fb10006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.499+0000 7fb119400700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10006d7f0 0x7fb10006fca0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb10800b020 tx=0x7fb108009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:50.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.503+0000 7fb112000700 1 -- 172.21.15.1:0/1263746971 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb10c056490 con 0x7fb11c105160 2024-02-28T17:17:50.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.745+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb11c04bf40 con 0x7fb10006d7f0 2024-02-28T17:17:50.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.747+0000 7fb112000700 1 -- 172.21.15.1:0/1263746971 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb11c04bf40 con 0x7fb10006d7f0 2024-02-28T17:17:50.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10006d7f0 msgr2=0x7fb10006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:50.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10006d7f0 0x7fb10006fca0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb10800b020 tx=0x7fb108009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:50.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105160 msgr2=0x7fb11c115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:50.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105160 0x7fb11c115cd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fb10c002a50 tx=0x7fb10c0046b0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 shutdown_connections 2024-02-28T17:17:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10006d7f0 0x7fb10006fca0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb11c105160 0x7fb11c115cd0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 --2- 172.21.15.1:0/1263746971 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb11c105b00 0x7fb11c116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 >> 172.21.15.1:0/1263746971 conn(0x7fb11c0fad30 msgr2=0x7fb11c0fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 shutdown_connections 2024-02-28T17:17:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.751+0000 7fb120e00700 1 -- 172.21.15.1:0/1263746971 wait complete. 2024-02-28T17:17:50.770 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:17:50.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.911+0000 7ffbf3800700 1 -- 172.21.15.1:0/3227509583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 msgr2=0x7ffbec105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:50.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.911+0000 7ffbf3800700 1 --2- 172.21.15.1:0/3227509583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 0x7ffbec105540 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0004940 tx=0x7ffbe0009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:50.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.912+0000 7ffbf3800700 1 -- 172.21.15.1:0/3227509583 shutdown_connections 2024-02-28T17:17:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.912+0000 7ffbf3800700 1 --2- 172.21.15.1:0/3227509583 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbec105b10 0x7ffbec0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.912+0000 7ffbf3800700 1 --2- 172.21.15.1:0/3227509583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 0x7ffbec105540 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.912+0000 7ffbf3800700 1 -- 172.21.15.1:0/3227509583 >> 172.21.15.1:0/3227509583 conn(0x7ffbec0face0 msgr2=0x7ffbec0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.912+0000 7ffbf3800700 1 -- 172.21.15.1:0/3227509583 shutdown_connections 2024-02-28T17:17:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.912+0000 7ffbf3800700 1 -- 172.21.15.1:0/3227509583 wait complete. 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.913+0000 7ffbf3800700 1 Processor -- start 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.913+0000 7ffbf3800700 1 -- start start 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.913+0000 7ffbf3800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 0x7ffbec115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.913+0000 7ffbf3800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbec105b10 0x7ffbec1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.913+0000 7ffbf3800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1167f0 con 0x7ffbec105170 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.913+0000 7ffbf3800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec10fb40 con 0x7ffbec105b10 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.913+0000 7ffbf0e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 0x7ffbec115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.913+0000 7ffbf0e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 0x7ffbec115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49896/0 (socket says 172.21.15.1:49896) 2024-02-28T17:17:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.914+0000 7ffbf0e00700 1 -- 172.21.15.1:0/34773145 learned_addr learned my addr 172.21.15.1:0/34773145 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.914+0000 7ffbebe00700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbec105b10 0x7ffbec1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.914+0000 7ffbf0e00700 1 -- 172.21.15.1:0/34773145 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbec105b10 msgr2=0x7ffbec1161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.914+0000 7ffbf0e00700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbec105b10 0x7ffbec1161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.914+0000 7ffbf0e00700 1 -- 172.21.15.1:0/34773145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe0005040 con 0x7ffbec105170 2024-02-28T17:17:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.914+0000 7ffbf0e00700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 0x7ffbec115c60 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009150 tx=0x7ffbe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.915+0000 7ffbe8c00700 1 -- 172.21.15.1:0/34773145 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbe000e040 con 0x7ffbec105170 2024-02-28T17:17:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.915+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec10fdc0 con 0x7ffbec105170 2024-02-28T17:17:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.915+0000 7ffbe8c00700 1 -- 172.21.15.1:0/34773145 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbe000bbf0 con 0x7ffbec105170 2024-02-28T17:17:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.915+0000 7ffbe8c00700 1 -- 172.21.15.1:0/34773145 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbe001c630 con 0x7ffbec105170 2024-02-28T17:17:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.915+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec110300 con 0x7ffbec105170 2024-02-28T17:17:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.916+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec102480 con 0x7ffbec105170 2024-02-28T17:17:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.917+0000 7ffbe8c00700 1 -- 172.21.15.1:0/34773145 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffbe000bd60 con 0x7ffbec105170 2024-02-28T17:17:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.918+0000 7ffbe8c00700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbd406d7f0 0x7ffbd406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:50.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.918+0000 7ffbe8c00700 1 -- 172.21.15.1:0/34773145 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffbe008e2a0 con 0x7ffbec105170 2024-02-28T17:17:50.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.918+0000 7ffbebe00700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbd406d7f0 0x7ffbd406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:50.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.919+0000 7ffbebe00700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbd406d7f0 0x7ffbd406fca0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8007fd0 tx=0x7ffbd8009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:50.923+0000 7ffbe8c00700 1 -- 172.21.15.1:0/34773145 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffbe0057b50 con 0x7ffbec105170 2024-02-28T17:17:51.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.176+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbec111280 con 0x7ffbd406d7f0 2024-02-28T17:17:51.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.178+0000 7ffbe8c00700 1 -- 172.21.15.1:0/34773145 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffbec111280 con 0x7ffbd406d7f0 2024-02-28T17:17:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbd406d7f0 msgr2=0x7ffbd406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbd406d7f0 0x7ffbd406fca0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8007fd0 tx=0x7ffbd8009040 comp rx=0 tx=0).stop 2024-02-28T17:17:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 msgr2=0x7ffbec115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 0x7ffbec115c60 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009150 tx=0x7ffbe0009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 shutdown_connections 2024-02-28T17:17:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffbd406d7f0 0x7ffbd406fca0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbec105170 0x7ffbec115c60 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 --2- 172.21.15.1:0/34773145 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffbec105b10 0x7ffbec1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.181+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 >> 172.21.15.1:0/34773145 conn(0x7ffbec0face0 msgr2=0x7ffbec0fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.182+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 shutdown_connections 2024-02-28T17:17:51.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.182+0000 7ffbf3800700 1 -- 172.21.15.1:0/34773145 wait complete. 2024-02-28T17:17:51.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:51 smithi001 ceph-mon[112940]: pgmap v1305: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:51.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.343+0000 7f5075200700 1 -- 172.21.15.1:0/1595396320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50701038f0 msgr2=0x7f50700fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:51.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.343+0000 7f5075200700 1 --2- 172.21.15.1:0/1595396320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50701038f0 0x7f50700fcf60 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f505c004940 tx=0x7f505c009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:51.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.343+0000 7f5075200700 1 -- 172.21.15.1:0/1595396320 shutdown_connections 2024-02-28T17:17:51.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.343+0000 7f5075200700 1 --2- 172.21.15.1:0/1595396320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50701038f0 0x7f50700fcf60 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.343+0000 7f5075200700 1 --2- 172.21.15.1:0/1595396320 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5070102f50 0x7f5070103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.343+0000 7f5075200700 1 -- 172.21.15.1:0/1595396320 >> 172.21.15.1:0/1595396320 conn(0x7f50700f8ae0 msgr2=0x7f50700faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:51.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.343+0000 7f5075200700 1 -- 172.21.15.1:0/1595396320 shutdown_connections 2024-02-28T17:17:51.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.344+0000 7f5075200700 1 -- 172.21.15.1:0/1595396320 wait complete. 2024-02-28T17:17:51.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.344+0000 7f5075200700 1 Processor -- start 2024-02-28T17:17:51.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.345+0000 7f5075200700 1 -- start start 2024-02-28T17:17:51.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.345+0000 7f5075200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5070102f50 0x7f5070115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:51.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.345+0000 7f5075200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50701038f0 0x7f5070116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:51.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.345+0000 7f5075200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5070116870 con 0x7f5070102f50 2024-02-28T17:17:51.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.345+0000 7f5075200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507010fb80 con 0x7f50701038f0 2024-02-28T17:17:51.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.345+0000 7f506de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5070102f50 0x7f5070115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:51.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.345+0000 7f506de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5070102f50 0x7f5070115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49910/0 (socket says 172.21.15.1:49910) 2024-02-28T17:17:51.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.345+0000 7f506de00700 1 -- 172.21.15.1:0/4061234515 learned_addr learned my addr 172.21.15.1:0/4061234515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:51.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f506d400700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50701038f0 0x7f5070116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:51.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f506de00700 1 -- 172.21.15.1:0/4061234515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50701038f0 msgr2=0x7f5070116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:51.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f506de00700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50701038f0 0x7f5070116220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f506de00700 1 -- 172.21.15.1:0/4061234515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c005040 con 0x7f5070102f50 2024-02-28T17:17:51.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f506d400700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50701038f0 0x7f5070116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:17:51.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f506de00700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5070102f50 0x7f5070115ce0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5060002a50 tx=0x7f50600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:51.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f5066000700 1 -- 172.21.15.1:0/4061234515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5060007cc0 con 0x7f5070102f50 2024-02-28T17:17:51.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f5066000700 1 -- 172.21.15.1:0/4061234515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f506000f040 con 0x7f5070102f50 2024-02-28T17:17:51.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507010fe60 con 0x7f5070102f50 2024-02-28T17:17:51.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.346+0000 7f5066000700 1 -- 172.21.15.1:0/4061234515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5060009640 con 0x7f5070102f50 2024-02-28T17:17:51.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.347+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50701103a0 con 0x7f5070102f50 2024-02-28T17:17:51.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.348+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50700fc6e0 con 0x7f5070102f50 2024-02-28T17:17:51.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.349+0000 7f5066000700 1 -- 172.21.15.1:0/4061234515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f50600097a0 con 0x7f5070102f50 2024-02-28T17:17:51.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.350+0000 7f5066000700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505406d7e0 0x7f505406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:51.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.350+0000 7f5066000700 1 -- 172.21.15.1:0/4061234515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f506008cc60 con 0x7f5070102f50 2024-02-28T17:17:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.350+0000 7f506d400700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505406d7e0 0x7f505406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.351+0000 7f506d400700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505406d7e0 0x7f505406fc90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f505c009fd0 tx=0x7f505c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:51.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.355+0000 7f5066000700 1 -- 172.21.15.1:0/4061234515 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5060059fc0 con 0x7f5070102f50 2024-02-28T17:17:51.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:51 smithi066 ceph-mon[85781]: pgmap v1305: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:51.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.590+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5070111330 con 0x7f505406d7e0 2024-02-28T17:17:51.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.607+0000 7f5066000700 1 -- 172.21.15.1:0/4061234515 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5070111330 con 0x7f505406d7e0 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (43m) 5m ago 58m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (40m) 5m ago 58m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (40m) 5m ago 57m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (43m) 5m ago 57m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (36m) 5m ago 51m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (36m) 5m ago 50m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (5m) 5m ago 50m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:17:51.611 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (20m) 5m ago 51m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (46m) 5m ago 59m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (45m) 5m ago 56m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (42m) 5m ago 60m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (41m) 5m ago 56m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (45m) 5m ago 58m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (44m) 5m ago 57m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (40m) 5m ago 55m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (39m) 5m ago 55m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (39m) 5m ago 54m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (38m) 5m ago 54m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (37m) 5m ago 53m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (37m) 5m ago 52m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:17:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (44m) 5m ago 57m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505406d7e0 msgr2=0x7f505406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505406d7e0 0x7f505406fc90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f505c009fd0 tx=0x7f505c009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5070102f50 msgr2=0x7f5070115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5070102f50 0x7f5070115ce0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5060002a50 tx=0x7f50600046b0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 shutdown_connections 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f505406d7e0 0x7f505406fc90 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5070102f50 0x7f5070115ce0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 --2- 172.21.15.1:0/4061234515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f50701038f0 0x7f5070116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 >> 172.21.15.1:0/4061234515 conn(0x7f50700f8ae0 msgr2=0x7f50700fa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 shutdown_connections 2024-02-28T17:17:51.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.610+0000 7f5075200700 1 -- 172.21.15.1:0/4061234515 wait complete. 2024-02-28T17:17:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.768+0000 7f0cd4800700 1 -- 172.21.15.1:0/3052763787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fdfe0 msgr2=0x7f0ccc0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.768+0000 7f0cd4800700 1 --2- 172.21.15.1:0/3052763787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fdfe0 0x7f0ccc0fe3b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0004970 tx=0x7f0cc0009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.768+0000 7f0cd4800700 1 -- 172.21.15.1:0/3052763787 shutdown_connections 2024-02-28T17:17:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.768+0000 7f0cd4800700 1 --2- 172.21.15.1:0/3052763787 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ccc0fe8f0 0x7f0ccc107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.768+0000 7f0cd4800700 1 --2- 172.21.15.1:0/3052763787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fdfe0 0x7f0ccc0fe3b0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.768+0000 7f0cd4800700 1 -- 172.21.15.1:0/3052763787 >> 172.21.15.1:0/3052763787 conn(0x7f0ccc071d90 msgr2=0x7f0ccc072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.768+0000 7f0cd4800700 1 -- 172.21.15.1:0/3052763787 shutdown_connections 2024-02-28T17:17:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.768+0000 7f0cd4800700 1 -- 172.21.15.1:0/3052763787 wait complete. 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.769+0000 7f0cd4800700 1 Processor -- start 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.769+0000 7f0cd4800700 1 -- start start 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.769+0000 7f0cd4800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ccc0fdfe0 0x7f0ccc0fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.769+0000 7f0cd4800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fe8f0 0x7f0ccc0fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.769+0000 7f0cd4800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0fc9e0 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.769+0000 7f0cd4800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0fcb50 con 0x7f0ccc0fdfe0 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.770+0000 7f0cd1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fe8f0 0x7f0ccc0fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.770+0000 7f0cd1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fe8f0 0x7f0ccc0fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49930/0 (socket says 172.21.15.1:49930) 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.770+0000 7f0cd1400700 1 -- 172.21.15.1:0/443170436 learned_addr learned my addr 172.21.15.1:0/443170436 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.770+0000 7f0cd1e00700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ccc0fdfe0 0x7f0ccc0fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.770+0000 7f0cd1400700 1 -- 172.21.15.1:0/443170436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ccc0fdfe0 msgr2=0x7f0ccc0fbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.770+0000 7f0cd1400700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ccc0fdfe0 0x7f0ccc0fbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.770+0000 7f0cd1400700 1 -- 172.21.15.1:0/443170436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc0005040 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.770+0000 7f0cd1e00700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ccc0fdfe0 0x7f0ccc0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:17:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.771+0000 7f0cd1400700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fe8f0 0x7f0ccc0fc3e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c8c0 tx=0x7f0cbc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.771+0000 7f0cc6000700 1 -- 172.21.15.1:0/443170436 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cbc00a980 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.771+0000 7f0cc6000700 1 -- 172.21.15.1:0/443170436 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cbc00ce60 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.771+0000 7f0cc6000700 1 -- 172.21.15.1:0/443170436 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cbc0115e0 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.771+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc119c90 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.771+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc11a170 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.772+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc1046c0 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.773+0000 7f0cc6000700 1 -- 172.21.15.1:0/443170436 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0cbc012400 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.774+0000 7f0cc6000700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0cb406d7f0 0x7f0cb406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.774+0000 7f0cc6000700 1 -- 172.21.15.1:0/443170436 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0cbc08cf40 con 0x7f0ccc0fe8f0 2024-02-28T17:17:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.775+0000 7f0cd1e00700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0cb406d7f0 0x7f0cb406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.775+0000 7f0cd1e00700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0cb406d7f0 0x7f0cb406fca0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0000c00 tx=0x7f0cc0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:51.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:51.779+0000 7f0cc6000700 1 -- 172.21.15.1:0/443170436 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0cbc0567f0 con 0x7f0ccc0fe8f0 2024-02-28T17:17:52.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.100+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0ccc04bed0 con 0x7f0ccc0fe8f0 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.101+0000 7f0cc6000700 1 -- 172.21.15.1:0/443170436 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0cbc0126b0 con 0x7f0ccc0fe8f0 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:17:52.105 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:17:52.106 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:17:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.104+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0cb406d7f0 msgr2=0x7f0cb406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.104+0000 7f0cd4800700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0cb406d7f0 0x7f0cb406fca0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0000c00 tx=0x7f0cc0009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fe8f0 msgr2=0x7f0ccc0fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fe8f0 0x7f0ccc0fc3e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c8c0 tx=0x7f0cbc0044c0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 shutdown_connections 2024-02-28T17:17:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0cb406d7f0 0x7f0cb406fca0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ccc0fdfe0 0x7f0ccc0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 --2- 172.21.15.1:0/443170436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ccc0fe8f0 0x7f0ccc0fc3e0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 >> 172.21.15.1:0/443170436 conn(0x7f0ccc071d90 msgr2=0x7f0ccc101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 shutdown_connections 2024-02-28T17:17:52.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.105+0000 7f0cd4800700 1 -- 172.21.15.1:0/443170436 wait complete. 2024-02-28T17:17:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.264+0000 7f4042600700 1 -- 172.21.15.1:0/3939322034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 msgr2=0x7f403c070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.264+0000 7f4042600700 1 --2- 172.21.15.1:0/3939322034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 0x7f403c070330 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f402c004970 tx=0x7f402c009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.265+0000 7f4042600700 1 -- 172.21.15.1:0/3939322034 shutdown_connections 2024-02-28T17:17:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.265+0000 7f4042600700 1 --2- 172.21.15.1:0/3939322034 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f403c070870 0x7f403c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.265+0000 7f4042600700 1 --2- 172.21.15.1:0/3939322034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 0x7f403c070330 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.265+0000 7f4042600700 1 -- 172.21.15.1:0/3939322034 >> 172.21.15.1:0/3939322034 conn(0x7f403c0f8ae0 msgr2=0x7f403c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.265+0000 7f4042600700 1 -- 172.21.15.1:0/3939322034 shutdown_connections 2024-02-28T17:17:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.265+0000 7f4042600700 1 -- 172.21.15.1:0/3939322034 wait complete. 2024-02-28T17:17:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.266+0000 7f4042600700 1 Processor -- start 2024-02-28T17:17:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.266+0000 7f4042600700 1 -- start start 2024-02-28T17:17:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.266+0000 7f4042600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f403c070870 0x7f403c11a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.266+0000 7f4042600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 0x7f403c11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.266+0000 7f4042600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c11ac00 con 0x7f403c071f60 2024-02-28T17:17:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.266+0000 7f4042600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c113f10 con 0x7f403c070870 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.267+0000 7f403aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 0x7f403c11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.267+0000 7f403aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 0x7f403c11a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32806/0 (socket says 172.21.15.1:32806) 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.267+0000 7f403aa00700 1 -- 172.21.15.1:0/995890552 learned_addr learned my addr 172.21.15.1:0/995890552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.267+0000 7f403b400700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f403c070870 0x7f403c11a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.267+0000 7f403aa00700 1 -- 172.21.15.1:0/995890552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f403c070870 msgr2=0x7f403c11a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.267+0000 7f403aa00700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f403c070870 0x7f403c11a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.268+0000 7f403aa00700 1 -- 172.21.15.1:0/995890552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c005040 con 0x7f403c071f60 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.268+0000 7f403b400700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f403c070870 0x7f403c11a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:17:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.268+0000 7f403aa00700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 0x7f403c11a5b0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f402800c8c0 tx=0x7f40280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.268+0000 7f4033400700 1 -- 172.21.15.1:0/995890552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f402800a980 con 0x7f403c071f60 2024-02-28T17:17:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.268+0000 7f4033400700 1 -- 172.21.15.1:0/995890552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f402800ce60 con 0x7f403c071f60 2024-02-28T17:17:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.268+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c1141f0 con 0x7f403c071f60 2024-02-28T17:17:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.268+0000 7f4033400700 1 -- 172.21.15.1:0/995890552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40280115e0 con 0x7f403c071f60 2024-02-28T17:17:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.268+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c114730 con 0x7f403c071f60 2024-02-28T17:17:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.269+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c0fe400 con 0x7f403c071f60 2024-02-28T17:17:52.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.271+0000 7f4033400700 1 -- 172.21.15.1:0/995890552 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4028012400 con 0x7f403c071f60 2024-02-28T17:17:52.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:52 smithi001 ceph-mon[112940]: from='client.36300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.272+0000 7f4033400700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f402006d7f0 0x7f402006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.272+0000 7f4033400700 1 -- 172.21.15.1:0/995890552 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f402808cf70 con 0x7f403c071f60 2024-02-28T17:17:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.272+0000 7f403b400700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f402006d7f0 0x7f402006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.273+0000 7f403b400700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f402006d7f0 0x7f402006fca0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f402c000c00 tx=0x7f402c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.277+0000 7f4033400700 1 -- 172.21.15.1:0/995890552 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f402805a350 con 0x7f403c071f60 2024-02-28T17:17:52.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:52 smithi066 ceph-mon[85781]: from='client.36300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:52.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.551+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f403c0fc6e0 con 0x7f403c071f60 2024-02-28T17:17:52.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.553+0000 7f4033400700 1 -- 172.21.15.1:0/995890552 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f40280126b0 con 0x7f403c071f60 2024-02-28T17:17:52.558 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:17:52.559 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:17:52.560 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:17:52.561 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:17:52.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f402006d7f0 msgr2=0x7f402006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f402006d7f0 0x7f402006fca0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f402c000c00 tx=0x7f402c009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:52.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 msgr2=0x7f403c11a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 0x7f403c11a5b0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f402800c8c0 tx=0x7f40280044c0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 shutdown_connections 2024-02-28T17:17:52.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f402006d7f0 0x7f402006fca0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f403c070870 0x7f403c11a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 --2- 172.21.15.1:0/995890552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f403c071f60 0x7f403c11a5b0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.555+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 >> 172.21.15.1:0/995890552 conn(0x7f403c0f8ae0 msgr2=0x7f403c0fa790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:52.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.556+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 shutdown_connections 2024-02-28T17:17:52.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.556+0000 7f4042600700 1 -- 172.21.15.1:0/995890552 wait complete. 2024-02-28T17:17:52.563 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:17:52.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.717+0000 7f12efe00700 1 -- 172.21.15.1:0/1210338733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fddc0 msgr2=0x7f12f00fe190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.717+0000 7f12efe00700 1 --2- 172.21.15.1:0/1210338733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fddc0 0x7f12f00fe190 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f12e0004940 tx=0x7f12e0009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:52.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.718+0000 7f12efe00700 1 -- 172.21.15.1:0/1210338733 shutdown_connections 2024-02-28T17:17:52.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.718+0000 7f12efe00700 1 --2- 172.21.15.1:0/1210338733 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12f00fe760 0x7f12f0106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.718+0000 7f12efe00700 1 --2- 172.21.15.1:0/1210338733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fddc0 0x7f12f00fe190 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.718+0000 7f12efe00700 1 -- 172.21.15.1:0/1210338733 >> 172.21.15.1:0/1210338733 conn(0x7f12f00f7830 msgr2=0x7f12f00f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:52.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.718+0000 7f12efe00700 1 -- 172.21.15.1:0/1210338733 shutdown_connections 2024-02-28T17:17:52.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.718+0000 7f12efe00700 1 -- 172.21.15.1:0/1210338733 wait complete. 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.719+0000 7f12efe00700 1 Processor -- start 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.719+0000 7f12efe00700 1 -- start start 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.719+0000 7f12efe00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12f00fddc0 0x7f12f01159f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.719+0000 7f12efe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fe760 0x7f12f0115f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.719+0000 7f12efe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0116580 con 0x7f12f00fe760 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.719+0000 7f12efe00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f010f8b0 con 0x7f12f00fddc0 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.719+0000 7f12ee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fe760 0x7f12f0115f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.720+0000 7f12ee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fe760 0x7f12f0115f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32820/0 (socket says 172.21.15.1:32820) 2024-02-28T17:17:52.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.720+0000 7f12ee000700 1 -- 172.21.15.1:0/1723103632 learned_addr learned my addr 172.21.15.1:0/1723103632 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:52.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.720+0000 7f12eea00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12f00fddc0 0x7f12f01159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.720+0000 7f12ee000700 1 -- 172.21.15.1:0/1723103632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12f00fddc0 msgr2=0x7f12f01159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.720+0000 7f12ee000700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12f00fddc0 0x7f12f01159f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.720+0000 7f12ee000700 1 -- 172.21.15.1:0/1723103632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e0005040 con 0x7f12f00fe760 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.720+0000 7f12ee000700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fe760 0x7f12f0115f30 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00c8c0 tx=0x7f12dc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.720+0000 7f12e6a00700 1 -- 172.21.15.1:0/1723103632 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12dc00a980 con 0x7f12f00fe760 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.721+0000 7f12e6a00700 1 -- 172.21.15.1:0/1723103632 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12dc00ce60 con 0x7f12f00fe760 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.721+0000 7f12e6a00700 1 -- 172.21.15.1:0/1723103632 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12dc0115e0 con 0x7f12f00fe760 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.721+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f010fb90 con 0x7f12f00fe760 2024-02-28T17:17:52.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.721+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f01100d0 con 0x7f12f00fe760 2024-02-28T17:17:52.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.722+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f0106810 con 0x7f12f00fe760 2024-02-28T17:17:52.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.723+0000 7f12e6a00700 1 -- 172.21.15.1:0/1723103632 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f12dc012400 con 0x7f12f00fe760 2024-02-28T17:17:52.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.724+0000 7f12e6a00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f12d406d7f0 0x7f12d406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:52.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.724+0000 7f12e6a00700 1 -- 172.21.15.1:0/1723103632 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f12dc08ceb0 con 0x7f12f00fe760 2024-02-28T17:17:52.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.724+0000 7f12eea00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f12d406d7f0 0x7f12d406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:52.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.725+0000 7f12eea00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f12d406d7f0 0x7f12d406fca0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000c00 tx=0x7f12e0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:52.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.729+0000 7f12e6a00700 1 -- 172.21.15.1:0/1723103632 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f12dc05a350 con 0x7f12f00fe760 2024-02-28T17:17:52.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.973+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f12f0109e50 con 0x7f12d406d7f0 2024-02-28T17:17:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.976+0000 7f12e6a00700 1 -- 172.21.15.1:0/1723103632 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f12f0109e50 con 0x7f12d406d7f0 2024-02-28T17:17:52.980 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:17:52.980 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:17:52.980 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:17:52.980 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:17:52.980 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:17:52.981 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:17:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f12d406d7f0 msgr2=0x7f12d406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f12d406d7f0 0x7f12d406fca0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000c00 tx=0x7f12e0009e30 comp rx=0 tx=0).stop 2024-02-28T17:17:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fe760 msgr2=0x7f12f0115f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fe760 0x7f12f0115f30 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00c8c0 tx=0x7f12dc0044c0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 shutdown_connections 2024-02-28T17:17:52.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f12d406d7f0 0x7f12d406fca0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f12f00fddc0 0x7f12f01159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 --2- 172.21.15.1:0/1723103632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12f00fe760 0x7f12f0115f30 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:52.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.979+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 >> 172.21.15.1:0/1723103632 conn(0x7f12f00f7830 msgr2=0x7f12f00fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:52.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.980+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 shutdown_connections 2024-02-28T17:17:52.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:52.980+0000 7f12efe00700 1 -- 172.21.15.1:0/1723103632 wait complete. 2024-02-28T17:17:53.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.133+0000 7f9489a00700 1 -- 172.21.15.1:0/3942616101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9484102f50 msgr2=0x7f9484103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:53.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.133+0000 7f9489a00700 1 --2- 172.21.15.1:0/3942616101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9484102f50 0x7f9484103320 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f9474009ce0 tx=0x7f9474004d10 comp rx=0 tx=0).stop 2024-02-28T17:17:53.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.133+0000 7f9489a00700 1 -- 172.21.15.1:0/3942616101 shutdown_connections 2024-02-28T17:17:53.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.133+0000 7f9489a00700 1 --2- 172.21.15.1:0/3942616101 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f94841038f0 0x7f94840fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:53.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.133+0000 7f9489a00700 1 --2- 172.21.15.1:0/3942616101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9484102f50 0x7f9484103320 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:53.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.133+0000 7f9489a00700 1 -- 172.21.15.1:0/3942616101 >> 172.21.15.1:0/3942616101 conn(0x7f94840f8ae0 msgr2=0x7f94840faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:53.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.133+0000 7f9489a00700 1 -- 172.21.15.1:0/3942616101 shutdown_connections 2024-02-28T17:17:53.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.133+0000 7f9489a00700 1 -- 172.21.15.1:0/3942616101 wait complete. 2024-02-28T17:17:53.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.134+0000 7f9489a00700 1 Processor -- start 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.134+0000 7f9489a00700 1 -- start start 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.134+0000 7f9489a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9484102f50 0x7f9484113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.134+0000 7f9489a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94841038f0 0x7f9484113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.134+0000 7f9489a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9484114640 con 0x7f94841038f0 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.134+0000 7f9489a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948410d970 con 0x7f9484102f50 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f947a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94841038f0 0x7f9484113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f947a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94841038f0 0x7f9484113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32842/0 (socket says 172.21.15.1:32842) 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f947a000700 1 -- 172.21.15.1:0/4194788978 learned_addr learned my addr 172.21.15.1:0/4194788978 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f9482a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9484102f50 0x7f9484113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f947a000700 1 -- 172.21.15.1:0/4194788978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9484102f50 msgr2=0x7f9484113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f947a000700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9484102f50 0x7f9484113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f947a000700 1 -- 172.21.15.1:0/4194788978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9474009940 con 0x7f94841038f0 2024-02-28T17:17:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f9482a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9484102f50 0x7f9484113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:17:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.135+0000 7f947a000700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94841038f0 0x7f9484113ff0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f946c00eb40 tx=0x7f946c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.136+0000 7f947b400700 1 -- 172.21.15.1:0/4194788978 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f946c007cc0 con 0x7f94841038f0 2024-02-28T17:17:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.136+0000 7f947b400700 1 -- 172.21.15.1:0/4194788978 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f946c00f040 con 0x7f94841038f0 2024-02-28T17:17:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.136+0000 7f947b400700 1 -- 172.21.15.1:0/4194788978 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f946c0095e0 con 0x7f94841038f0 2024-02-28T17:17:53.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.136+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948410dbf0 con 0x7f94841038f0 2024-02-28T17:17:53.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.136+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948410e130 con 0x7f94841038f0 2024-02-28T17:17:53.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.137+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9484100260 con 0x7f94841038f0 2024-02-28T17:17:53.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.138+0000 7f947b400700 1 -- 172.21.15.1:0/4194788978 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f946c009740 con 0x7f94841038f0 2024-02-28T17:17:53.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.139+0000 7f947b400700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f946806d7f0 0x7f946806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:17:53.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.139+0000 7f947b400700 1 -- 172.21.15.1:0/4194788978 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f946c08c8c0 con 0x7f94841038f0 2024-02-28T17:17:53.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.139+0000 7f9482a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f946806d7f0 0x7f946806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:17:53.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.140+0000 7f9482a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f946806d7f0 0x7f946806fca0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9474007ce0 tx=0x7f9474004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:17:53.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.144+0000 7f947b400700 1 -- 172.21.15.1:0/4194788978 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f946c059d70 con 0x7f94841038f0 2024-02-28T17:17:53.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:53 smithi001 ceph-mon[112940]: from='client.36304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:53.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:53 smithi001 ceph-mon[112940]: pgmap v1306: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:53.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:53 smithi001 ceph-mon[112940]: from='client.36308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:53.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:53 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/443170436' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:17:53.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:53 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/995890552' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:17:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.445+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f948404bed0 con 0x7f94841038f0 2024-02-28T17:17:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.446+0000 7f947b400700 1 -- 172.21.15.1:0/4194788978 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f946c059900 con 0x7f94841038f0 2024-02-28T17:17:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:17:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:17:53.451 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:17:53.452 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:17:53.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f946806d7f0 msgr2=0x7f946806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f946806d7f0 0x7f946806fca0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9474007ce0 tx=0x7f9474004d10 comp rx=0 tx=0).stop 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94841038f0 msgr2=0x7f9484113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94841038f0 0x7f9484113ff0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f946c00eb40 tx=0x7f946c00b040 comp rx=0 tx=0).stop 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 shutdown_connections 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f946806d7f0 0x7f946806fca0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9484102f50 0x7f9484113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 --2- 172.21.15.1:0/4194788978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94841038f0 0x7f9484113ff0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 >> 172.21.15.1:0/4194788978 conn(0x7f94840f8ae0 msgr2=0x7f94840fa5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.449+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 shutdown_connections 2024-02-28T17:17:53.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:17:53.450+0000 7f9489a00700 1 -- 172.21.15.1:0/4194788978 wait complete. 2024-02-28T17:17:53.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:53 smithi066 ceph-mon[85781]: from='client.36304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:53.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:53 smithi066 ceph-mon[85781]: pgmap v1306: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:53.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:53 smithi066 ceph-mon[85781]: from='client.36308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:53.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:53 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/443170436' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:17:53.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:53 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/995890552' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:17:54.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:54 smithi066 ceph-mon[85781]: from='client.36320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:54.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:54 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4194788978' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:17:54.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:54 smithi001 ceph-mon[112940]: from='client.36320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:17:54.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:54 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4194788978' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:17:55.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:55 smithi066 ceph-mon[85781]: pgmap v1307: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:55.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:55 smithi001 ceph-mon[112940]: pgmap v1307: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:56.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:56 smithi066 ceph-mon[85781]: pgmap v1308: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:56.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:17:56.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:56 smithi001 ceph-mon[112940]: pgmap v1308: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:56.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:17:59.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:17:58 smithi066 ceph-mon[85781]: pgmap v1309: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:17:59.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:17:58 smithi001 ceph-mon[112940]: pgmap v1309: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:01.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:00 smithi066 ceph-mon[85781]: pgmap v1310: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:01.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:00 smithi001 ceph-mon[112940]: pgmap v1310: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:03.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:02 smithi066 ceph-mon[85781]: pgmap v1311: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:02 smithi001 ceph-mon[112940]: pgmap v1311: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:05.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:04 smithi066 ceph-mon[85781]: pgmap v1312: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:05.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:04 smithi001 ceph-mon[112940]: pgmap v1312: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:07.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:06 smithi066 ceph-mon[85781]: pgmap v1313: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:07.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:06 smithi001 ceph-mon[112940]: pgmap v1313: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:09.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:08 smithi066 ceph-mon[85781]: pgmap v1314: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:09.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:08 smithi001 ceph-mon[112940]: pgmap v1314: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:11.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:10 smithi066 ceph-mon[85781]: pgmap v1315: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:11.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:10 smithi001 ceph-mon[112940]: pgmap v1315: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:11.915 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:18:12.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:18:13.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:12 smithi066 ceph-mon[85781]: pgmap v1316: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:13.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:12 smithi001 ceph-mon[112940]: pgmap v1316: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:15.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:14 smithi066 ceph-mon[85781]: pgmap v1317: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:15.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:14 smithi001 ceph-mon[112940]: pgmap v1317: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:17.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:16 smithi066 ceph-mon[85781]: pgmap v1318: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:17.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:16 smithi001 ceph-mon[112940]: pgmap v1318: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:19.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:18 smithi066 ceph-mon[85781]: pgmap v1319: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:19.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:18 smithi001 ceph-mon[112940]: pgmap v1319: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:21.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:20 smithi066 ceph-mon[85781]: pgmap v1320: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:21.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:20 smithi001 ceph-mon[112940]: pgmap v1320: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:23.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:22 smithi066 ceph-mon[85781]: pgmap v1321: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:23.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:22 smithi001 ceph-mon[112940]: pgmap v1321: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.623+0000 7fe828c00700 1 -- 172.21.15.1:0/261640651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 msgr2=0x7fe824109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.623+0000 7fe828c00700 1 --2- 172.21.15.1:0/261640651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 0x7fe824109750 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fe814004940 tx=0x7fe814009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.624+0000 7fe828c00700 1 -- 172.21.15.1:0/261640651 shutdown_connections 2024-02-28T17:18:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.624+0000 7fe828c00700 1 --2- 172.21.15.1:0/261640651 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe824109d20 0x7fe8241033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.624+0000 7fe828c00700 1 --2- 172.21.15.1:0/261640651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 0x7fe824109750 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.624+0000 7fe828c00700 1 -- 172.21.15.1:0/261640651 >> 172.21.15.1:0/261640651 conn(0x7fe824072b50 msgr2=0x7fe824074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.624+0000 7fe828c00700 1 -- 172.21.15.1:0/261640651 shutdown_connections 2024-02-28T17:18:23.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.624+0000 7fe828c00700 1 -- 172.21.15.1:0/261640651 wait complete. 2024-02-28T17:18:23.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.625+0000 7fe828c00700 1 Processor -- start 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.625+0000 7fe828c00700 1 -- start start 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.625+0000 7fe828c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 0x7fe824119d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.625+0000 7fe828c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe824109d20 0x7fe82411a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.625+0000 7fe828c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82411a900 con 0x7fe824109380 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.625+0000 7fe828c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824113c10 con 0x7fe824109d20 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.626+0000 7fe823400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 0x7fe824119d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.626+0000 7fe823400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 0x7fe824119d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54708/0 (socket says 172.21.15.1:54708) 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.626+0000 7fe823400700 1 -- 172.21.15.1:0/1331754753 learned_addr learned my addr 172.21.15.1:0/1331754753 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.626+0000 7fe822a00700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe824109d20 0x7fe82411a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.626+0000 7fe823400700 1 -- 172.21.15.1:0/1331754753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe824109d20 msgr2=0x7fe82411a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.626+0000 7fe823400700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe824109d20 0x7fe82411a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:23.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.626+0000 7fe823400700 1 -- 172.21.15.1:0/1331754753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe814005040 con 0x7fe824109380 2024-02-28T17:18:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.626+0000 7fe823400700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 0x7fe824119d70 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe814009b30 tx=0x7fe814009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.627+0000 7fe81b400700 1 -- 172.21.15.1:0/1331754753 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe814021070 con 0x7fe824109380 2024-02-28T17:18:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.627+0000 7fe81b400700 1 -- 172.21.15.1:0/1331754753 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe814013e50 con 0x7fe824109380 2024-02-28T17:18:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.627+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe824113e90 con 0x7fe824109380 2024-02-28T17:18:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.627+0000 7fe81b400700 1 -- 172.21.15.1:0/1331754753 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe81400e070 con 0x7fe824109380 2024-02-28T17:18:23.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.627+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe824114370 con 0x7fe824109380 2024-02-28T17:18:23.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.628+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82405f5c0 con 0x7fe824109380 2024-02-28T17:18:23.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.629+0000 7fe81b400700 1 -- 172.21.15.1:0/1331754753 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe81401cc40 con 0x7fe824109380 2024-02-28T17:18:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.630+0000 7fe81b400700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe80806d7e0 0x7fe80806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.631+0000 7fe81b400700 1 -- 172.21.15.1:0/1331754753 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe81408e510 con 0x7fe824109380 2024-02-28T17:18:23.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.631+0000 7fe822a00700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe80806d7e0 0x7fe80806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:23.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.631+0000 7fe822a00700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe80806d7e0 0x7fe80806fc90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe810004600 tx=0x7fe81000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:23.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.636+0000 7fe81b400700 1 -- 172.21.15.1:0/1331754753 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe81405b870 con 0x7fe824109380 2024-02-28T17:18:23.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.881+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe824115460 con 0x7fe80806d7e0 2024-02-28T17:18:23.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.884+0000 7fe81b400700 1 -- 172.21.15.1:0/1331754753 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe824115460 con 0x7fe80806d7e0 2024-02-28T17:18:23.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.886+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe80806d7e0 msgr2=0x7fe80806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:23.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.886+0000 7fe828c00700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe80806d7e0 0x7fe80806fc90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe810004600 tx=0x7fe81000b040 comp rx=0 tx=0).stop 2024-02-28T17:18:23.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.886+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 msgr2=0x7fe824119d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:23.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.886+0000 7fe828c00700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 0x7fe824119d70 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe814009b30 tx=0x7fe814009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.887+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 shutdown_connections 2024-02-28T17:18:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.887+0000 7fe828c00700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe80806d7e0 0x7fe80806fc90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.887+0000 7fe828c00700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824109380 0x7fe824119d70 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.887+0000 7fe828c00700 1 --2- 172.21.15.1:0/1331754753 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe824109d20 0x7fe82411a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.887+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 >> 172.21.15.1:0/1331754753 conn(0x7fe824072b50 msgr2=0x7fe824075540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.887+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 shutdown_connections 2024-02-28T17:18:23.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:23.887+0000 7fe828c00700 1 -- 172.21.15.1:0/1331754753 wait complete. 2024-02-28T17:18:23.906 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:18:24.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.046+0000 7fcf4cc00700 1 -- 172.21.15.1:0/2405461446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf481095f0 msgr2=0x7fcf481099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.046+0000 7fcf4cc00700 1 --2- 172.21.15.1:0/2405461446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf481095f0 0x7fcf481099c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fcf38004940 tx=0x7fcf38009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:24.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.047+0000 7fcf4cc00700 1 -- 172.21.15.1:0/2405461446 shutdown_connections 2024-02-28T17:18:24.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.047+0000 7fcf4cc00700 1 --2- 172.21.15.1:0/2405461446 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf48103dc0 0x7fcf48104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.047+0000 7fcf4cc00700 1 --2- 172.21.15.1:0/2405461446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf481095f0 0x7fcf481099c0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.047+0000 7fcf4cc00700 1 -- 172.21.15.1:0/2405461446 >> 172.21.15.1:0/2405461446 conn(0x7fcf48072d50 msgr2=0x7fcf48075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:24.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.047+0000 7fcf4cc00700 1 -- 172.21.15.1:0/2405461446 shutdown_connections 2024-02-28T17:18:24.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.047+0000 7fcf4cc00700 1 -- 172.21.15.1:0/2405461446 wait complete. 2024-02-28T17:18:24.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf4cc00700 1 Processor -- start 2024-02-28T17:18:24.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf4cc00700 1 -- start start 2024-02-28T17:18:24.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf4cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48103dc0 0x7fcf4811a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf4cc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf481095f0 0x7fcf4811a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf4cc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4811abf0 con 0x7fcf48103dc0 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf4cc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf48113f00 con 0x7fcf481095f0 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf45e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48103dc0 0x7fcf4811a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf45e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48103dc0 0x7fcf4811a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54734/0 (socket says 172.21.15.1:54734) 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.048+0000 7fcf45e00700 1 -- 172.21.15.1:0/1381526201 learned_addr learned my addr 172.21.15.1:0/1381526201 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.049+0000 7fcf45400700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf481095f0 0x7fcf4811a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.049+0000 7fcf45e00700 1 -- 172.21.15.1:0/1381526201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf481095f0 msgr2=0x7fcf4811a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.049+0000 7fcf45e00700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf481095f0 0x7fcf4811a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.049+0000 7fcf45e00700 1 -- 172.21.15.1:0/1381526201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38005040 con 0x7fcf48103dc0 2024-02-28T17:18:24.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.049+0000 7fcf45e00700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48103dc0 0x7fcf4811a060 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fcf38009150 tx=0x7fcf38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:24.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.049+0000 7fcf3e000700 1 -- 172.21.15.1:0/1381526201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf3800e040 con 0x7fcf48103dc0 2024-02-28T17:18:24.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.049+0000 7fcf3e000700 1 -- 172.21.15.1:0/1381526201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf3800bbf0 con 0x7fcf48103dc0 2024-02-28T17:18:24.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.049+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf48114180 con 0x7fcf48103dc0 2024-02-28T17:18:24.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.050+0000 7fcf3e000700 1 -- 172.21.15.1:0/1381526201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf3801c630 con 0x7fcf48103dc0 2024-02-28T17:18:24.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.050+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf481146c0 con 0x7fcf48103dc0 2024-02-28T17:18:24.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.051+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf48075870 con 0x7fcf48103dc0 2024-02-28T17:18:24.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.052+0000 7fcf3e000700 1 -- 172.21.15.1:0/1381526201 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcf3800bd60 con 0x7fcf48103dc0 2024-02-28T17:18:24.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.053+0000 7fcf3e000700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf2c06d7f0 0x7fcf2c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.053+0000 7fcf3e000700 1 -- 172.21.15.1:0/1381526201 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcf3808e290 con 0x7fcf48103dc0 2024-02-28T17:18:24.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.054+0000 7fcf45400700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf2c06d7f0 0x7fcf2c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:24.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.054+0000 7fcf45400700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf2c06d7f0 0x7fcf2c06fca0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009fd0 tx=0x7fcf34009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:24.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.058+0000 7fcf3e000700 1 -- 172.21.15.1:0/1381526201 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf3805b670 con 0x7fcf48103dc0 2024-02-28T17:18:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.304+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf48115640 con 0x7fcf2c06d7f0 2024-02-28T17:18:24.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.306+0000 7fcf3e000700 1 -- 172.21.15.1:0/1381526201 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcf48115640 con 0x7fcf2c06d7f0 2024-02-28T17:18:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf2c06d7f0 msgr2=0x7fcf2c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf2c06d7f0 0x7fcf2c06fca0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009fd0 tx=0x7fcf34009f20 comp rx=0 tx=0).stop 2024-02-28T17:18:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48103dc0 msgr2=0x7fcf4811a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48103dc0 0x7fcf4811a060 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fcf38009150 tx=0x7fcf38009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 shutdown_connections 2024-02-28T17:18:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcf2c06d7f0 0x7fcf2c06fca0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48103dc0 0x7fcf4811a060 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 --2- 172.21.15.1:0/1381526201 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcf481095f0 0x7fcf4811a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 >> 172.21.15.1:0/1381526201 conn(0x7fcf48072d50 msgr2=0x7fcf48074760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 shutdown_connections 2024-02-28T17:18:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.309+0000 7fcf4cc00700 1 -- 172.21.15.1:0/1381526201 wait complete. 2024-02-28T17:18:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.464+0000 7fbcfb200700 1 -- 172.21.15.1:0/133520887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc102610 msgr2=0x7fbcfc0fbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.464+0000 7fbcfb200700 1 --2- 172.21.15.1:0/133520887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc102610 0x7fbcfc0fbc80 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fbce4004940 tx=0x7fbce4009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.464+0000 7fbcfb200700 1 -- 172.21.15.1:0/133520887 shutdown_connections 2024-02-28T17:18:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.464+0000 7fbcfb200700 1 --2- 172.21.15.1:0/133520887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc102610 0x7fbcfc0fbc80 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.464+0000 7fbcfb200700 1 --2- 172.21.15.1:0/133520887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbcfc101c70 0x7fbcfc102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.464+0000 7fbcfb200700 1 -- 172.21.15.1:0/133520887 >> 172.21.15.1:0/133520887 conn(0x7fbcfc0f77c0 msgr2=0x7fbcfc0f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.464+0000 7fbcfb200700 1 -- 172.21.15.1:0/133520887 shutdown_connections 2024-02-28T17:18:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.464+0000 7fbcfb200700 1 -- 172.21.15.1:0/133520887 wait complete. 2024-02-28T17:18:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.465+0000 7fbcfb200700 1 Processor -- start 2024-02-28T17:18:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.465+0000 7fbcfb200700 1 -- start start 2024-02-28T17:18:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.465+0000 7fbcfb200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc101c70 0x7fbcfc115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.465+0000 7fbcfb200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbcfc102610 0x7fbcfc115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.465+0000 7fbcfb200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc1165d0 con 0x7fbcfc101c70 2024-02-28T17:18:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.465+0000 7fbcfb200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc10f900 con 0x7fbcfc102610 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.466+0000 7fbcf9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc101c70 0x7fbcfc115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.466+0000 7fbcf9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc101c70 0x7fbcfc115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54744/0 (socket says 172.21.15.1:54744) 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.466+0000 7fbcf9e00700 1 -- 172.21.15.1:0/2712115332 learned_addr learned my addr 172.21.15.1:0/2712115332 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.466+0000 7fbcf9e00700 1 -- 172.21.15.1:0/2712115332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbcfc102610 msgr2=0x7fbcfc115f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.466+0000 7fbcf9e00700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbcfc102610 0x7fbcfc115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.466+0000 7fbcf9e00700 1 -- 172.21.15.1:0/2712115332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce4005040 con 0x7fbcfc101c70 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.466+0000 7fbcf9e00700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc101c70 0x7fbcfc115a40 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00c8c0 tx=0x7fbcec0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.466+0000 7fbcf2000700 1 -- 172.21.15.1:0/2712115332 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcec00a980 con 0x7fbcfc101c70 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.467+0000 7fbcf2000700 1 -- 172.21.15.1:0/2712115332 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcec00ce60 con 0x7fbcfc101c70 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.467+0000 7fbcf2000700 1 -- 172.21.15.1:0/2712115332 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcec0115e0 con 0x7fbcfc101c70 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.467+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc10fbe0 con 0x7fbcfc101c70 2024-02-28T17:18:24.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.467+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc1101a0 con 0x7fbcfc101c70 2024-02-28T17:18:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.468+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcfc05f5c0 con 0x7fbcfc101c70 2024-02-28T17:18:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.469+0000 7fbcf2000700 1 -- 172.21.15.1:0/2712115332 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbcec012400 con 0x7fbcfc101c70 2024-02-28T17:18:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.470+0000 7fbcf2000700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbce006d7f0 0x7fbce006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.470+0000 7fbcf2000700 1 -- 172.21.15.1:0/2712115332 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbcec08cf60 con 0x7fbcfc101c70 2024-02-28T17:18:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.470+0000 7fbcf3e00700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbce006d7f0 0x7fbce006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.471+0000 7fbcf3e00700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbce006d7f0 0x7fbce006fca0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbce4004f90 tx=0x7fbce4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:24.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.475+0000 7fbcf2000700 1 -- 172.21.15.1:0/2712115332 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbcec05a340 con 0x7fbcfc101c70 2024-02-28T17:18:24.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.705+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcfc04bed0 con 0x7fbce006d7f0 2024-02-28T17:18:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.717+0000 7fbcf2000700 1 -- 172.21.15.1:0/2712115332 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbcfc04bed0 con 0x7fbce006d7f0 2024-02-28T17:18:24.721 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:18:24.721 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (44m) 6m ago 58m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:18:24.721 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (41m) 6m ago 59m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (41m) 6m ago 57m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (44m) 6m ago 58m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (37m) 6m ago 51m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (36m) 6m ago 51m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (6m) 6m ago 51m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (21m) 6m ago 51m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (46m) 6m ago 60m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (45m) 6m ago 57m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (42m) 6m ago 60m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (42m) 6m ago 57m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (45m) 6m ago 58m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (45m) 6m ago 57m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:18:24.722 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (40m) 6m ago 56m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:18:24.723 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (40m) 6m ago 55m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:18:24.723 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (39m) 6m ago 55m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:18:24.723 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (39m) 6m ago 54m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:18:24.723 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (38m) 6m ago 54m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:18:24.723 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (37m) 6m ago 53m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:18:24.723 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (45m) 6m ago 58m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:18:24.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbce006d7f0 msgr2=0x7fbce006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbce006d7f0 0x7fbce006fca0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbce4004f90 tx=0x7fbce4009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc101c70 msgr2=0x7fbcfc115a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc101c70 0x7fbcfc115a40 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00c8c0 tx=0x7fbcec0044c0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 shutdown_connections 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbce006d7f0 0x7fbce006fca0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcfc101c70 0x7fbcfc115a40 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 --2- 172.21.15.1:0/2712115332 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbcfc102610 0x7fbcfc115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.719+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 >> 172.21.15.1:0/2712115332 conn(0x7fbcfc0f77c0 msgr2=0x7fbcfc065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.720+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 shutdown_connections 2024-02-28T17:18:24.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.720+0000 7fbcfb200700 1 -- 172.21.15.1:0/2712115332 wait complete. 2024-02-28T17:18:24.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.870+0000 7f005ce00700 1 -- 172.21.15.1:0/632954729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 msgr2=0x7f0058104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.870+0000 7f005ce00700 1 --2- 172.21.15.1:0/632954729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 0x7f0058104210 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f0044004940 tx=0x7f0044009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:24.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.870+0000 7f005ce00700 1 -- 172.21.15.1:0/632954729 shutdown_connections 2024-02-28T17:18:24.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.870+0000 7f005ce00700 1 --2- 172.21.15.1:0/632954729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 0x7f0058104210 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.870+0000 7f005ce00700 1 --2- 172.21.15.1:0/632954729 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00581095f0 0x7f00581099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.870+0000 7f005ce00700 1 -- 172.21.15.1:0/632954729 >> 172.21.15.1:0/632954729 conn(0x7f0058072d50 msgr2=0x7f0058075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:24.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.870+0000 7f005ce00700 1 -- 172.21.15.1:0/632954729 shutdown_connections 2024-02-28T17:18:24.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.870+0000 7f005ce00700 1 -- 172.21.15.1:0/632954729 wait complete. 2024-02-28T17:18:24.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.871+0000 7f005ce00700 1 Processor -- start 2024-02-28T17:18:24.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.871+0000 7f005ce00700 1 -- start start 2024-02-28T17:18:24.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.872+0000 7f005ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 0x7f005811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.872+0000 7f005ce00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00581095f0 0x7f005811a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.872+0000 7f005ce00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005811abe0 con 0x7f0058103dc0 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.872+0000 7f005ce00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0058113ef0 con 0x7f00581095f0 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.872+0000 7f0055e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 0x7f005811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.872+0000 7f0055e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 0x7f005811a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54758/0 (socket says 172.21.15.1:54758) 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.872+0000 7f0055e00700 1 -- 172.21.15.1:0/3070266604 learned_addr learned my addr 172.21.15.1:0/3070266604 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.872+0000 7f0055400700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00581095f0 0x7f005811a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f0055e00700 1 -- 172.21.15.1:0/3070266604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00581095f0 msgr2=0x7f005811a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f0055e00700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00581095f0 0x7f005811a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f0055e00700 1 -- 172.21.15.1:0/3070266604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0044005040 con 0x7f0058103dc0 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f0055400700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00581095f0 0x7f005811a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:18:24.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f0055e00700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 0x7f005811a050 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f0048002a50 tx=0x7f00480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:24.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f004e000700 1 -- 172.21.15.1:0/3070266604 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0048007cc0 con 0x7f0058103dc0 2024-02-28T17:18:24.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f004e000700 1 -- 172.21.15.1:0/3070266604 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f004800f040 con 0x7f0058103dc0 2024-02-28T17:18:24.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00581141d0 con 0x7f0058103dc0 2024-02-28T17:18:24.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f004e000700 1 -- 172.21.15.1:0/3070266604 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f004800db70 con 0x7f0058103dc0 2024-02-28T17:18:24.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.873+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0058114710 con 0x7f0058103dc0 2024-02-28T17:18:24.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.874+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005810b9c0 con 0x7f0058103dc0 2024-02-28T17:18:24.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.875+0000 7f004e000700 1 -- 172.21.15.1:0/3070266604 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f004800dd10 con 0x7f0058103dc0 2024-02-28T17:18:24.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.876+0000 7f004e000700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f003c06d7f0 0x7f003c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:24.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.876+0000 7f004e000700 1 -- 172.21.15.1:0/3070266604 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f004808c590 con 0x7f0058103dc0 2024-02-28T17:18:24.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.876+0000 7f0055400700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f003c06d7f0 0x7f003c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:24.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.877+0000 7f0055400700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f003c06d7f0 0x7f003c06fca0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f004400b3b0 tx=0x7f0044009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:24.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:24.881+0000 7f004e000700 1 -- 172.21.15.1:0/3070266604 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f00480598f0 con 0x7f0058103dc0 2024-02-28T17:18:25.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:24 smithi066 ceph-mon[85781]: pgmap v1322: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:25.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:24 smithi066 ceph-mon[85781]: from='client.36328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:25.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:24 smithi001 ceph-mon[112940]: pgmap v1322: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:25.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:24 smithi001 ceph-mon[112940]: from='client.36328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.184+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f005804bed0 con 0x7f0058103dc0 2024-02-28T17:18:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.186+0000 7f004e000700 1 -- 172.21.15.1:0/3070266604 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0048059480 con 0x7f0058103dc0 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:18:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:18:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:18:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:18:25.191 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:18:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f003c06d7f0 msgr2=0x7f003c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f003c06d7f0 0x7f003c06fca0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f004400b3b0 tx=0x7f0044009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 msgr2=0x7f005811a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 0x7f005811a050 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f0048002a50 tx=0x7f00480046b0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 shutdown_connections 2024-02-28T17:18:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f003c06d7f0 0x7f003c06fca0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0058103dc0 0x7f005811a050 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 --2- 172.21.15.1:0/3070266604 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f00581095f0 0x7f005811a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.188+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 >> 172.21.15.1:0/3070266604 conn(0x7f0058072d50 msgr2=0x7f0058074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.189+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 shutdown_connections 2024-02-28T17:18:25.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.189+0000 7f005ce00700 1 -- 172.21.15.1:0/3070266604 wait complete. 2024-02-28T17:18:25.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.345+0000 7fe768600700 1 -- 172.21.15.1:0/2417045498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 msgr2=0x7fe760105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:25.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.345+0000 7fe768600700 1 --2- 172.21.15.1:0/2417045498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 0x7fe760105500 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fe754004940 tx=0x7fe754009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.345+0000 7fe768600700 1 -- 172.21.15.1:0/2417045498 shutdown_connections 2024-02-28T17:18:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.345+0000 7fe768600700 1 --2- 172.21.15.1:0/2417045498 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe760105ad0 0x7fe7600ff140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.345+0000 7fe768600700 1 --2- 172.21.15.1:0/2417045498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 0x7fe760105500 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.345+0000 7fe768600700 1 -- 172.21.15.1:0/2417045498 >> 172.21.15.1:0/2417045498 conn(0x7fe7600fad00 msgr2=0x7fe7600fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.346+0000 7fe768600700 1 -- 172.21.15.1:0/2417045498 shutdown_connections 2024-02-28T17:18:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.346+0000 7fe768600700 1 -- 172.21.15.1:0/2417045498 wait complete. 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.346+0000 7fe768600700 1 Processor -- start 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.347+0000 7fe768600700 1 -- start start 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.347+0000 7fe768600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 0x7fe760115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.347+0000 7fe768600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe760105ad0 0x7fe760116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.347+0000 7fe768600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760116850 con 0x7fe760105130 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.347+0000 7fe768600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76010fb60 con 0x7fe760105ad0 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.347+0000 7fe765c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 0x7fe760115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.347+0000 7fe765c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 0x7fe760115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54774/0 (socket says 172.21.15.1:54774) 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.347+0000 7fe765c00700 1 -- 172.21.15.1:0/732954237 learned_addr learned my addr 172.21.15.1:0/732954237 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.348+0000 7fe765200700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe760105ad0 0x7fe760116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.348+0000 7fe765c00700 1 -- 172.21.15.1:0/732954237 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe760105ad0 msgr2=0x7fe760116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.348+0000 7fe765c00700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe760105ad0 0x7fe760116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.348+0000 7fe765c00700 1 -- 172.21.15.1:0/732954237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe754005040 con 0x7fe760105130 2024-02-28T17:18:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.348+0000 7fe765200700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe760105ad0 0x7fe760116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:18:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.348+0000 7fe765c00700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 0x7fe760115cc0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe754009fd0 tx=0x7fe754009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.348+0000 7fe75d600700 1 -- 172.21.15.1:0/732954237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe75400e040 con 0x7fe760105130 2024-02-28T17:18:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.349+0000 7fe75d600700 1 -- 172.21.15.1:0/732954237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe75400bbf0 con 0x7fe760105130 2024-02-28T17:18:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.349+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76010fde0 con 0x7fe760105130 2024-02-28T17:18:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.349+0000 7fe75d600700 1 -- 172.21.15.1:0/732954237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe75401c3f0 con 0x7fe760105130 2024-02-28T17:18:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.349+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe760110320 con 0x7fe760105130 2024-02-28T17:18:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.350+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe760102440 con 0x7fe760105130 2024-02-28T17:18:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.351+0000 7fe75d600700 1 -- 172.21.15.1:0/732954237 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe75400bd60 con 0x7fe760105130 2024-02-28T17:18:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.352+0000 7fe75d600700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe74806d7f0 0x7fe74806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.352+0000 7fe75d600700 1 -- 172.21.15.1:0/732954237 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe75408e1c0 con 0x7fe760105130 2024-02-28T17:18:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.352+0000 7fe765200700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe74806d7f0 0x7fe74806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.353+0000 7fe765200700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe74806d7f0 0x7fe74806fca0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009fd0 tx=0x7fe74c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.356+0000 7fe75d600700 1 -- 172.21.15.1:0/732954237 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe75405b5a0 con 0x7fe760105130 2024-02-28T17:18:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.631+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe7600fe900 con 0x7fe760105130 2024-02-28T17:18:25.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.632+0000 7fe75d600700 1 -- 172.21.15.1:0/732954237 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fe75405b130 con 0x7fe760105130 2024-02-28T17:18:25.636 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:18:25.636 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:18:25.636 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:18:25.636 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:18:25.637 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:18:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:18:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:18:25.639 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:18:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:18:25.639 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:18:25.639 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:18:25.639 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:18:25.639 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe74806d7f0 msgr2=0x7fe74806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe74806d7f0 0x7fe74806fca0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009fd0 tx=0x7fe74c00b040 comp rx=0 tx=0).stop 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 msgr2=0x7fe760115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 0x7fe760115cc0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe754009fd0 tx=0x7fe754009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 shutdown_connections 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe74806d7f0 0x7fe74806fca0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe760105130 0x7fe760115cc0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 --2- 172.21.15.1:0/732954237 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe760105ad0 0x7fe760116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 >> 172.21.15.1:0/732954237 conn(0x7fe7600fad00 msgr2=0x7fe7600fc840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 shutdown_connections 2024-02-28T17:18:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.635+0000 7fe768600700 1 -- 172.21.15.1:0/732954237 wait complete. 2024-02-28T17:18:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:18:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.792+0000 7f0dbb200700 1 -- 172.21.15.1:0/2416868338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 msgr2=0x7f0db40fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.792+0000 7f0dbb200700 1 --2- 172.21.15.1:0/2416868338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 0x7f0db40fcf60 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c004940 tx=0x7f0d9c009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.792+0000 7f0dbb200700 1 -- 172.21.15.1:0/2416868338 shutdown_connections 2024-02-28T17:18:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.792+0000 7f0dbb200700 1 --2- 172.21.15.1:0/2416868338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 0x7f0db40fcf60 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.792+0000 7f0dbb200700 1 --2- 172.21.15.1:0/2416868338 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db4102f50 0x7f0db4103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.792+0000 7f0dbb200700 1 -- 172.21.15.1:0/2416868338 >> 172.21.15.1:0/2416868338 conn(0x7f0db40f8ae0 msgr2=0x7f0db40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.792+0000 7f0dbb200700 1 -- 172.21.15.1:0/2416868338 shutdown_connections 2024-02-28T17:18:25.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.793+0000 7f0dbb200700 1 -- 172.21.15.1:0/2416868338 wait complete. 2024-02-28T17:18:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.793+0000 7f0dbb200700 1 Processor -- start 2024-02-28T17:18:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.793+0000 7f0dbb200700 1 -- start start 2024-02-28T17:18:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.794+0000 7f0dbb200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db4102f50 0x7f0db4115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.794+0000 7f0dbb200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 0x7f0db41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.794+0000 7f0dbb200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db4116810 con 0x7f0db41038f0 2024-02-28T17:18:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.794+0000 7f0dbb200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db410fb40 con 0x7f0db4102f50 2024-02-28T17:18:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.794+0000 7f0dab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 0x7f0db41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.794+0000 7f0dab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 0x7f0db41161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54796/0 (socket says 172.21.15.1:54796) 2024-02-28T17:18:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.794+0000 7f0dab400700 1 -- 172.21.15.1:0/1201874379 learned_addr learned my addr 172.21.15.1:0/1201874379 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.794+0000 7f0db3e00700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db4102f50 0x7f0db4115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.795+0000 7f0dab400700 1 -- 172.21.15.1:0/1201874379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db4102f50 msgr2=0x7f0db4115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.795+0000 7f0dab400700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db4102f50 0x7f0db4115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.795+0000 7f0dab400700 1 -- 172.21.15.1:0/1201874379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c005040 con 0x7f0db41038f0 2024-02-28T17:18:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.795+0000 7f0dab400700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 0x7f0db41161c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009030 tx=0x7f0d9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.795+0000 7f0db0c00700 1 -- 172.21.15.1:0/1201874379 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d9c00e040 con 0x7f0db41038f0 2024-02-28T17:18:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.795+0000 7f0db0c00700 1 -- 172.21.15.1:0/1201874379 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d9c00bbf0 con 0x7f0db41038f0 2024-02-28T17:18:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.795+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db410fdc0 con 0x7f0db41038f0 2024-02-28T17:18:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.795+0000 7f0db0c00700 1 -- 172.21.15.1:0/1201874379 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d9c01c630 con 0x7f0db41038f0 2024-02-28T17:18:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.796+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db4110300 con 0x7f0db41038f0 2024-02-28T17:18:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.797+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db40fc6e0 con 0x7f0db41038f0 2024-02-28T17:18:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.798+0000 7f0db0c00700 1 -- 172.21.15.1:0/1201874379 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0d9c00bd60 con 0x7f0db41038f0 2024-02-28T17:18:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.798+0000 7f0db0c00700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.799+0000 7f0db0c00700 1 -- 172.21.15.1:0/1201874379 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0d9c08e2c0 con 0x7f0db41038f0 2024-02-28T17:18:25.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.799+0000 7f0db3e00700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.799+0000 7f0db3e00700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0da4007fd0 tx=0x7f0da4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:25.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:25.803+0000 7f0db0c00700 1 -- 172.21.15.1:0/1201874379 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0d9c05b620 con 0x7f0db41038f0 2024-02-28T17:18:26.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:25 smithi066 ceph-mon[85781]: from='client.36332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:26.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:25 smithi066 ceph-mon[85781]: from='client.36336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:26.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:25 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3070266604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:18:26.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:25 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/732954237' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:18:26.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:25 smithi001 ceph-mon[112940]: from='client.36332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:26.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:25 smithi001 ceph-mon[112940]: from='client.36336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:26.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:25 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3070266604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:18:26.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:25 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/732954237' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:18:26.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.039+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0db41111f0 con 0x7f0d9806d7f0 2024-02-28T17:18:26.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.041+0000 7f0db0c00700 1 -- 172.21.15.1:0/1201874379 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0db41111f0 con 0x7f0d9806d7f0 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:18:26.048 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:18:26.049 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:18:26.049 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:18:26.049 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:18:26.049 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:18:26.049 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:18:26.049 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 msgr2=0x7f0d9806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0da4007fd0 tx=0x7f0da4009040 comp rx=0 tx=0).stop 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 msgr2=0x7f0db41161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 0x7f0db41161c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009030 tx=0x7f0d9c009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 shutdown_connections 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0d9806d7f0 0x7f0d9806fca0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0db4102f50 0x7f0db4115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 --2- 172.21.15.1:0/1201874379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0db41038f0 0x7f0db41161c0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 >> 172.21.15.1:0/1201874379 conn(0x7f0db40f8ae0 msgr2=0x7f0db40fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 shutdown_connections 2024-02-28T17:18:26.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.044+0000 7f0dbb200700 1 -- 172.21.15.1:0/1201874379 wait complete. 2024-02-28T17:18:26.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.198+0000 7f2ff2600700 1 -- 172.21.15.1:0/3593865667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105170 msgr2=0x7f2fec105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.198+0000 7f2ff2600700 1 --2- 172.21.15.1:0/3593865667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105170 0x7f2fec105540 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc004940 tx=0x7f2fdc009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.199+0000 7f2ff2600700 1 -- 172.21.15.1:0/3593865667 shutdown_connections 2024-02-28T17:18:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.199+0000 7f2ff2600700 1 --2- 172.21.15.1:0/3593865667 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fec105b10 0x7f2fec0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.199+0000 7f2ff2600700 1 --2- 172.21.15.1:0/3593865667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105170 0x7f2fec105540 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.199+0000 7f2ff2600700 1 -- 172.21.15.1:0/3593865667 >> 172.21.15.1:0/3593865667 conn(0x7f2fec0face0 msgr2=0x7f2fec0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.199+0000 7f2ff2600700 1 -- 172.21.15.1:0/3593865667 shutdown_connections 2024-02-28T17:18:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.199+0000 7f2ff2600700 1 -- 172.21.15.1:0/3593865667 wait complete. 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.200+0000 7f2ff2600700 1 Processor -- start 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.200+0000 7f2ff2600700 1 -- start start 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.200+0000 7f2ff2600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fec105170 0x7f2fec115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.200+0000 7f2ff2600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105b10 0x7f2fec116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.200+0000 7f2ff2600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec116780 con 0x7f2fec105b10 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.200+0000 7f2ff2600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec10fad0 con 0x7f2fec105170 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.201+0000 7f2fe3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105b10 0x7f2fec116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.201+0000 7f2fe3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105b10 0x7f2fec116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54812/0 (socket says 172.21.15.1:54812) 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.201+0000 7f2fe3e00700 1 -- 172.21.15.1:0/1991886546 learned_addr learned my addr 172.21.15.1:0/1991886546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.201+0000 7f2feb400700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fec105170 0x7f2fec115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.201+0000 7f2fe3e00700 1 -- 172.21.15.1:0/1991886546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fec105170 msgr2=0x7f2fec115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.201+0000 7f2fe3e00700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fec105170 0x7f2fec115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.201+0000 7f2fe3e00700 1 -- 172.21.15.1:0/1991886546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc005040 con 0x7f2fec105b10 2024-02-28T17:18:26.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.201+0000 7f2fe3e00700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105b10 0x7f2fec116130 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400eb40 tx=0x7f2fd400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:26.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.202+0000 7f2fe3400700 1 -- 172.21.15.1:0/1991886546 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fd4007cc0 con 0x7f2fec105b10 2024-02-28T17:18:26.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.202+0000 7f2fe3400700 1 -- 172.21.15.1:0/1991886546 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fd400f040 con 0x7f2fec105b10 2024-02-28T17:18:26.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.202+0000 7f2fe3400700 1 -- 172.21.15.1:0/1991886546 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fd40095e0 con 0x7f2fec105b10 2024-02-28T17:18:26.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.202+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec10fd50 con 0x7f2fec105b10 2024-02-28T17:18:26.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.202+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec110290 con 0x7f2fec105b10 2024-02-28T17:18:26.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.203+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec0fe900 con 0x7f2fec105b10 2024-02-28T17:18:26.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.204+0000 7f2fe3400700 1 -- 172.21.15.1:0/1991886546 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2fd4009740 con 0x7f2fec105b10 2024-02-28T17:18:26.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.205+0000 7f2fe3400700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2fd006d7f0 0x7f2fd006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:26.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.205+0000 7f2fe3400700 1 -- 172.21.15.1:0/1991886546 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2fd408c960 con 0x7f2fec105b10 2024-02-28T17:18:26.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.205+0000 7f2feb400700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2fd006d7f0 0x7f2fd006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:26.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.206+0000 7f2feb400700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2fd006d7f0 0x7f2fd006fca0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00b0d0 tx=0x7f2fdc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.210+0000 7f2fe3400700 1 -- 172.21.15.1:0/1991886546 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2fd4056240 con 0x7f2fec105b10 2024-02-28T17:18:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.522+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2fec111270 con 0x7f2fec105b10 2024-02-28T17:18:26.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.523+0000 7f2fe3400700 1 -- 172.21.15.1:0/1991886546 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f2fd4059900 con 0x7f2fec105b10 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:18:26.529 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:18:26.530 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:18:26.531 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:18:26.531 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:18:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2fd006d7f0 msgr2=0x7f2fd006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2fd006d7f0 0x7f2fd006fca0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00b0d0 tx=0x7f2fdc009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105b10 msgr2=0x7f2fec116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105b10 0x7f2fec116130 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400eb40 tx=0x7f2fd400b040 comp rx=0 tx=0).stop 2024-02-28T17:18:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 shutdown_connections 2024-02-28T17:18:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2fd006d7f0 0x7f2fd006fca0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2fec105170 0x7f2fec115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 --2- 172.21.15.1:0/1991886546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fec105b10 0x7f2fec116130 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 >> 172.21.15.1:0/1991886546 conn(0x7f2fec0face0 msgr2=0x7f2fec0fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.527+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 shutdown_connections 2024-02-28T17:18:26.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:26.528+0000 7f2ff2600700 1 -- 172.21.15.1:0/1991886546 wait complete. 2024-02-28T17:18:26.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:26 smithi001 ceph-mon[112940]: pgmap v1323: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:26.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:18:26.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:26 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1991886546' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:18:27.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:26 smithi066 ceph-mon[85781]: pgmap v1323: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:27.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:18:27.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:26 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1991886546' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:18:28.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:27 smithi066 ceph-mon[85781]: from='client.36348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:28.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:27 smithi001 ceph-mon[112940]: from='client.36348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:29.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:28 smithi066 ceph-mon[85781]: pgmap v1324: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:29.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:28 smithi001 ceph-mon[112940]: pgmap v1324: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:31.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:30 smithi066 ceph-mon[85781]: pgmap v1325: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:30 smithi001 ceph-mon[112940]: pgmap v1325: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:33.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:32 smithi066 ceph-mon[85781]: pgmap v1326: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:33.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:32 smithi001 ceph-mon[112940]: pgmap v1326: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:35.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:34 smithi066 ceph-mon[85781]: pgmap v1327: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:35.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:34 smithi001 ceph-mon[112940]: pgmap v1327: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:37.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:36 smithi066 ceph-mon[85781]: pgmap v1328: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:37.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:36 smithi001 ceph-mon[112940]: pgmap v1328: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:39.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:38 smithi066 ceph-mon[85781]: pgmap v1329: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:39.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:38 smithi001 ceph-mon[112940]: pgmap v1329: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:41.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:40 smithi066 ceph-mon[85781]: pgmap v1330: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:41.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:40 smithi001 ceph-mon[112940]: pgmap v1330: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:42.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:18:42.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:18:43.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:42 smithi066 ceph-mon[85781]: pgmap v1331: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:43.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:42 smithi001 ceph-mon[112940]: pgmap v1331: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:45.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:44 smithi066 ceph-mon[85781]: pgmap v1332: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:45.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:44 smithi001 ceph-mon[112940]: pgmap v1332: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:47.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:46 smithi001 ceph-mon[112940]: pgmap v1333: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:47.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:46 smithi066 ceph-mon[85781]: pgmap v1333: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:49.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:48 smithi001 ceph-mon[112940]: pgmap v1334: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:49.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:48 smithi066 ceph-mon[85781]: pgmap v1334: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:51.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:50 smithi001 ceph-mon[112940]: pgmap v1335: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:51.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:50 smithi066 ceph-mon[85781]: pgmap v1335: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:53.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:52 smithi066 ceph-mon[85781]: pgmap v1336: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:53.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:52 smithi001 ceph-mon[112940]: pgmap v1336: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:55.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:54 smithi066 ceph-mon[85781]: pgmap v1337: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:55.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:54 smithi001 ceph-mon[112940]: pgmap v1337: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:56.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.699+0000 7f44d5600700 1 -- 172.21.15.1:0/1276134119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d01095f0 msgr2=0x7f44d01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:56.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.699+0000 7f44d5600700 1 --2- 172.21.15.1:0/1276134119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d01095f0 0x7f44d01099c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f44c0004940 tx=0x7f44c0009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:56.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.700+0000 7f44d5600700 1 -- 172.21.15.1:0/1276134119 shutdown_connections 2024-02-28T17:18:56.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.700+0000 7f44d5600700 1 --2- 172.21.15.1:0/1276134119 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d0103dc0 0x7f44d0104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:56.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.700+0000 7f44d5600700 1 --2- 172.21.15.1:0/1276134119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d01095f0 0x7f44d01099c0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:56.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.700+0000 7f44d5600700 1 -- 172.21.15.1:0/1276134119 >> 172.21.15.1:0/1276134119 conn(0x7f44d0072d50 msgr2=0x7f44d0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:56.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.700+0000 7f44d5600700 1 -- 172.21.15.1:0/1276134119 shutdown_connections 2024-02-28T17:18:56.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.700+0000 7f44d5600700 1 -- 172.21.15.1:0/1276134119 wait complete. 2024-02-28T17:18:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.701+0000 7f44d5600700 1 Processor -- start 2024-02-28T17:18:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.701+0000 7f44d5600700 1 -- start start 2024-02-28T17:18:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.701+0000 7f44d5600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d0103dc0 0x7f44d011a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.701+0000 7f44d5600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d011a740 0x7f44d01140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.701+0000 7f44d5600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d011ad50 con 0x7f44d011a740 2024-02-28T17:18:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.701+0000 7f44d5600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01145e0 con 0x7f44d0103dc0 2024-02-28T17:18:56.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.702+0000 7f44ce000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d011a740 0x7f44d01140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.702+0000 7f44ce000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d011a740 0x7f44d01140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39890/0 (socket says 172.21.15.1:39890) 2024-02-28T17:18:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.702+0000 7f44ce000700 1 -- 172.21.15.1:0/1478115329 learned_addr learned my addr 172.21.15.1:0/1478115329 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.702+0000 7f44cea00700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d0103dc0 0x7f44d011a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:56.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.702+0000 7f44ce000700 1 -- 172.21.15.1:0/1478115329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d0103dc0 msgr2=0x7f44d011a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.702+0000 7f44ce000700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d0103dc0 0x7f44d011a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.702+0000 7f44ce000700 1 -- 172.21.15.1:0/1478115329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0005040 con 0x7f44d011a740 2024-02-28T17:18:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.702+0000 7f44ce000700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d011a740 0x7f44d01140a0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00b790 tx=0x7f44bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.703+0000 7f44c6a00700 1 -- 172.21.15.1:0/1478115329 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44bc00f770 con 0x7f44d011a740 2024-02-28T17:18:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.703+0000 7f44c6a00700 1 -- 172.21.15.1:0/1478115329 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44bc00fda0 con 0x7f44d011a740 2024-02-28T17:18:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.703+0000 7f44c6a00700 1 -- 172.21.15.1:0/1478115329 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44bc00d310 con 0x7f44d011a740 2024-02-28T17:18:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.703+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d01148c0 con 0x7f44d011a740 2024-02-28T17:18:56.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.703+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d0114db0 con 0x7f44d011a740 2024-02-28T17:18:56.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.704+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0106850 con 0x7f44d011a740 2024-02-28T17:18:56.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.705+0000 7f44c6a00700 1 -- 172.21.15.1:0/1478115329 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f44bc010420 con 0x7f44d011a740 2024-02-28T17:18:56.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.707+0000 7f44c6a00700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b406d7f0 0x7f44b406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:56.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.707+0000 7f44c6a00700 1 -- 172.21.15.1:0/1478115329 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f44bc08cfb0 con 0x7f44d011a740 2024-02-28T17:18:56.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.707+0000 7f44cea00700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b406d7f0 0x7f44b406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:56.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.708+0000 7f44cea00700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b406d7f0 0x7f44b406fca0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f44c0000c00 tx=0x7f44c0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:56.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.712+0000 7f44c6a00700 1 -- 172.21.15.1:0/1478115329 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44bc0c0d20 con 0x7f44d011a740 2024-02-28T17:18:56.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.959+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44d004bf40 con 0x7f44b406d7f0 2024-02-28T17:18:56.964 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:56 smithi001 ceph-mon[112940]: pgmap v1338: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:56.964 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:18:56.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.961+0000 7f44c6a00700 1 -- 172.21.15.1:0/1478115329 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f44d004bf40 con 0x7f44b406d7f0 2024-02-28T17:18:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.963+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b406d7f0 msgr2=0x7f44b406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b406d7f0 0x7f44b406fca0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f44c0000c00 tx=0x7f44c0009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d011a740 msgr2=0x7f44d01140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:56.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d011a740 0x7f44d01140a0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00b790 tx=0x7f44bc009250 comp rx=0 tx=0).stop 2024-02-28T17:18:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 shutdown_connections 2024-02-28T17:18:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f44b406d7f0 0x7f44b406fca0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f44d0103dc0 0x7f44d011a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 --2- 172.21.15.1:0/1478115329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44d011a740 0x7f44d01140a0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 >> 172.21.15.1:0/1478115329 conn(0x7f44d0072d50 msgr2=0x7f44d0074a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 shutdown_connections 2024-02-28T17:18:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:56.964+0000 7f44d5600700 1 -- 172.21.15.1:0/1478115329 wait complete. 2024-02-28T17:18:56.986 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:18:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.126+0000 7f9936800700 1 -- 172.21.15.1:0/4276254602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 msgr2=0x7f99300fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.126+0000 7f9936800700 1 --2- 172.21.15.1:0/4276254602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 0x7f99300fcf60 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f991c004940 tx=0x7f991c009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:57.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.126+0000 7f9936800700 1 -- 172.21.15.1:0/4276254602 shutdown_connections 2024-02-28T17:18:57.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.126+0000 7f9936800700 1 --2- 172.21.15.1:0/4276254602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 0x7f99300fcf60 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.126+0000 7f9936800700 1 --2- 172.21.15.1:0/4276254602 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9930102f50 0x7f9930103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.126+0000 7f9936800700 1 -- 172.21.15.1:0/4276254602 >> 172.21.15.1:0/4276254602 conn(0x7f99300f8ae0 msgr2=0x7f99300faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:57.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.126+0000 7f9936800700 1 -- 172.21.15.1:0/4276254602 shutdown_connections 2024-02-28T17:18:57.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.126+0000 7f9936800700 1 -- 172.21.15.1:0/4276254602 wait complete. 2024-02-28T17:18:57.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.127+0000 7f9936800700 1 Processor -- start 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.128+0000 7f9936800700 1 -- start start 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.128+0000 7f9936800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9930102f50 0x7f9930070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.128+0000 7f9936800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 0x7f99300707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.128+0000 7f9936800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9930070dd0 con 0x7f99301038f0 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.128+0000 7f9936800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9930070f40 con 0x7f9930102f50 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.128+0000 7f992ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 0x7f99300707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.128+0000 7f992ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 0x7f99300707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39894/0 (socket says 172.21.15.1:39894) 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.128+0000 7f992ea00700 1 -- 172.21.15.1:0/44120141 learned_addr learned my addr 172.21.15.1:0/44120141 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.129+0000 7f992f400700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9930102f50 0x7f9930070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.129+0000 7f992ea00700 1 -- 172.21.15.1:0/44120141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9930102f50 msgr2=0x7f9930070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.129+0000 7f992ea00700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9930102f50 0x7f9930070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.129+0000 7f992ea00700 1 -- 172.21.15.1:0/44120141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c005040 con 0x7f99301038f0 2024-02-28T17:18:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.129+0000 7f992ea00700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 0x7f99300707d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f991c009030 tx=0x7f991c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.129+0000 7f9927400700 1 -- 172.21.15.1:0/44120141 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c00e040 con 0x7f99301038f0 2024-02-28T17:18:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.130+0000 7f9927400700 1 -- 172.21.15.1:0/44120141 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f991c00bbf0 con 0x7f99301038f0 2024-02-28T17:18:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.130+0000 7f9927400700 1 -- 172.21.15.1:0/44120141 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c01c630 con 0x7f99301038f0 2024-02-28T17:18:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.130+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9930119cb0 con 0x7f99301038f0 2024-02-28T17:18:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.130+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f993011a110 con 0x7f99301038f0 2024-02-28T17:18:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.131+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9930105420 con 0x7f99301038f0 2024-02-28T17:18:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.132+0000 7f9927400700 1 -- 172.21.15.1:0/44120141 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f991c00bd60 con 0x7f99301038f0 2024-02-28T17:18:57.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.133+0000 7f9927400700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f991406d7f0 0x7f991406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:57.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.133+0000 7f9927400700 1 -- 172.21.15.1:0/44120141 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f991c08e1f0 con 0x7f99301038f0 2024-02-28T17:18:57.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.133+0000 7f992f400700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f991406d7f0 0x7f991406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:57.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.134+0000 7f992f400700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f991406d7f0 0x7f991406fca0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9920007fd0 tx=0x7f9920009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.138+0000 7f9927400700 1 -- 172.21.15.1:0/44120141 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f991c057aa0 con 0x7f99301038f0 2024-02-28T17:18:57.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:56 smithi066 ceph-mon[85781]: pgmap v1338: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:57.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:18:57.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.388+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99300719d0 con 0x7f991406d7f0 2024-02-28T17:18:57.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.390+0000 7f9927400700 1 -- 172.21.15.1:0/44120141 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f99300719d0 con 0x7f991406d7f0 2024-02-28T17:18:57.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f991406d7f0 msgr2=0x7f991406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f991406d7f0 0x7f991406fca0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9920007fd0 tx=0x7f9920009040 comp rx=0 tx=0).stop 2024-02-28T17:18:57.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 msgr2=0x7f99300707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 0x7f99300707d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f991c009030 tx=0x7f991c009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:57.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 shutdown_connections 2024-02-28T17:18:57.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f991406d7f0 0x7f991406fca0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9930102f50 0x7f9930070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 --2- 172.21.15.1:0/44120141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99301038f0 0x7f99300707d0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 >> 172.21.15.1:0/44120141 conn(0x7f99300f8ae0 msgr2=0x7f99300fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:57.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 shutdown_connections 2024-02-28T17:18:57.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.393+0000 7f9936800700 1 -- 172.21.15.1:0/44120141 wait complete. 2024-02-28T17:18:57.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.553+0000 7f4690e00700 1 -- 172.21.15.1:0/1128532397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c1095f0 msgr2=0x7f468c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.553+0000 7f4690e00700 1 --2- 172.21.15.1:0/1128532397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c1095f0 0x7f468c1099c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f467c004940 tx=0x7f467c009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:57.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.554+0000 7f4690e00700 1 -- 172.21.15.1:0/1128532397 shutdown_connections 2024-02-28T17:18:57.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.554+0000 7f4690e00700 1 --2- 172.21.15.1:0/1128532397 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f468c103dc0 0x7f468c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.554+0000 7f4690e00700 1 --2- 172.21.15.1:0/1128532397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c1095f0 0x7f468c1099c0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.554+0000 7f4690e00700 1 -- 172.21.15.1:0/1128532397 >> 172.21.15.1:0/1128532397 conn(0x7f468c072d50 msgr2=0x7f468c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:57.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.554+0000 7f4690e00700 1 -- 172.21.15.1:0/1128532397 shutdown_connections 2024-02-28T17:18:57.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.554+0000 7f4690e00700 1 -- 172.21.15.1:0/1128532397 wait complete. 2024-02-28T17:18:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.555+0000 7f4690e00700 1 Processor -- start 2024-02-28T17:18:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.555+0000 7f4690e00700 1 -- start start 2024-02-28T17:18:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.555+0000 7f4690e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c103dc0 0x7f468c11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.555+0000 7f4690e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f468c1095f0 0x7f468c11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.555+0000 7f4690e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c11ac70 con 0x7f468c103dc0 2024-02-28T17:18:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.555+0000 7f4690e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c113f80 con 0x7f468c1095f0 2024-02-28T17:18:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.556+0000 7f4689e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c103dc0 0x7f468c11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:57.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.556+0000 7f4689e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c103dc0 0x7f468c11a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39922/0 (socket says 172.21.15.1:39922) 2024-02-28T17:18:57.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.556+0000 7f4689e00700 1 -- 172.21.15.1:0/233299018 learned_addr learned my addr 172.21.15.1:0/233299018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:57.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.556+0000 7f4689e00700 1 -- 172.21.15.1:0/233299018 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f468c1095f0 msgr2=0x7f468c11a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.556+0000 7f4689e00700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f468c1095f0 0x7f468c11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.556+0000 7f4689e00700 1 -- 172.21.15.1:0/233299018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c005040 con 0x7f468c103dc0 2024-02-28T17:18:57.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.557+0000 7f4689e00700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c103dc0 0x7f468c11a0e0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f467c009150 tx=0x7f467c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:57.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.557+0000 7f4682000700 1 -- 172.21.15.1:0/233299018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f467c00e040 con 0x7f468c103dc0 2024-02-28T17:18:57.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.557+0000 7f4682000700 1 -- 172.21.15.1:0/233299018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f467c004290 con 0x7f468c103dc0 2024-02-28T17:18:57.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.557+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468c114200 con 0x7f468c103dc0 2024-02-28T17:18:57.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.557+0000 7f4682000700 1 -- 172.21.15.1:0/233299018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f467c01c630 con 0x7f468c103dc0 2024-02-28T17:18:57.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.557+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c114740 con 0x7f468c103dc0 2024-02-28T17:18:57.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.558+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468c106850 con 0x7f468c103dc0 2024-02-28T17:18:57.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.559+0000 7f4682000700 1 -- 172.21.15.1:0/233299018 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f467c00bbf0 con 0x7f468c103dc0 2024-02-28T17:18:57.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.561+0000 7f4682000700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f467006d7f0 0x7f467006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:57.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.561+0000 7f4682000700 1 -- 172.21.15.1:0/233299018 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f467c08e260 con 0x7f468c103dc0 2024-02-28T17:18:57.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.561+0000 7f4689400700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f467006d7f0 0x7f467006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:57.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.562+0000 7f4689400700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f467006d7f0 0x7f467006fca0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f468c115980 tx=0x7f4678009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:57.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.566+0000 7f4682000700 1 -- 172.21.15.1:0/233299018 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f467c05b640 con 0x7f468c103dc0 2024-02-28T17:18:57.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.803+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f468c04bf70 con 0x7f467006d7f0 2024-02-28T17:18:57.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.834+0000 7f4682000700 1 -- 172.21.15.1:0/233299018 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f467800f040 con 0x7f467006d7f0 2024-02-28T17:18:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:18:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (45m) 6m ago 59m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:18:57.839 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (41m) 6m ago 59m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (41m) 6m ago 58m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (44m) 6m ago 58m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (37m) 6m ago 52m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (37m) 6m ago 52m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (6m) 6m ago 52m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (21m) 6m ago 52m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (47m) 6m ago 61m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (46m) 6m ago 58m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (43m) 6m ago 61m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (42m) 6m ago 57m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (46m) 6m ago 59m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (46m) 6m ago 58m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:18:57.840 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (41m) 6m ago 57m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:18:57.841 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (40m) 6m ago 56m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:18:57.841 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (40m) 6m ago 55m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:18:57.841 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (39m) 6m ago 55m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:18:57.841 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (38m) 6m ago 54m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:18:57.841 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (38m) 6m ago 54m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:18:57.841 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (45m) 6m ago 58m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:18:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f467006d7f0 msgr2=0x7f467006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f467006d7f0 0x7f467006fca0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f468c115980 tx=0x7f4678009ce0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c103dc0 msgr2=0x7f468c11a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c103dc0 0x7f468c11a0e0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f467c009150 tx=0x7f467c009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:57.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 shutdown_connections 2024-02-28T17:18:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f467006d7f0 0x7f467006fca0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c103dc0 0x7f468c11a0e0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 --2- 172.21.15.1:0/233299018 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f468c1095f0 0x7f468c11a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 >> 172.21.15.1:0/233299018 conn(0x7f468c072d50 msgr2=0x7f468c074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.837+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 shutdown_connections 2024-02-28T17:18:57.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.838+0000 7f4690e00700 1 -- 172.21.15.1:0/233299018 wait complete. 2024-02-28T17:18:57.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.993+0000 7f040da00700 1 -- 172.21.15.1:0/4050964640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105ae0 msgr2=0x7f04080ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:57.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.993+0000 7f040da00700 1 --2- 172.21.15.1:0/4050964640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105ae0 0x7f04080ff150 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f03f4004940 tx=0x7f03f4009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:57.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.993+0000 7f040da00700 1 -- 172.21.15.1:0/4050964640 shutdown_connections 2024-02-28T17:18:57.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.993+0000 7f040da00700 1 --2- 172.21.15.1:0/4050964640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105ae0 0x7f04080ff150 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.993+0000 7f040da00700 1 --2- 172.21.15.1:0/4050964640 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105140 0x7f0408105510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:57.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.993+0000 7f040da00700 1 -- 172.21.15.1:0/4050964640 >> 172.21.15.1:0/4050964640 conn(0x7f04080facb0 msgr2=0x7f04080fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:57.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.994+0000 7f040da00700 1 -- 172.21.15.1:0/4050964640 shutdown_connections 2024-02-28T17:18:57.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.994+0000 7f040da00700 1 -- 172.21.15.1:0/4050964640 wait complete. 2024-02-28T17:18:57.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.994+0000 7f040da00700 1 Processor -- start 2024-02-28T17:18:57.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.995+0000 7f040da00700 1 -- start start 2024-02-28T17:18:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.995+0000 7f040da00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105140 0x7f0408115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.995+0000 7f040da00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105ae0 0x7f04081161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.995+0000 7f040da00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408116800 con 0x7f0408105140 2024-02-28T17:18:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.995+0000 7f040da00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040810fb30 con 0x7f0408105ae0 2024-02-28T17:18:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.995+0000 7f0406a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105140 0x7f0408115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.995+0000 7f0406a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105140 0x7f0408115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39934/0 (socket says 172.21.15.1:39934) 2024-02-28T17:18:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.995+0000 7f0406a00700 1 -- 172.21.15.1:0/3642622425 learned_addr learned my addr 172.21.15.1:0/3642622425 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.996+0000 7f0406000700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105ae0 0x7f04081161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.996+0000 7f0406a00700 1 -- 172.21.15.1:0/3642622425 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105ae0 msgr2=0x7f04081161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.996+0000 7f0406a00700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105ae0 0x7f04081161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.996+0000 7f0406a00700 1 -- 172.21.15.1:0/3642622425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f4005040 con 0x7f0408105140 2024-02-28T17:18:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.996+0000 7f0406a00700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105140 0x7f0408115c70 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f03f8002a50 tx=0x7f03f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.996+0000 7f03fea00700 1 -- 172.21.15.1:0/3642622425 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03f8007cc0 con 0x7f0408105140 2024-02-28T17:18:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.996+0000 7f03fea00700 1 -- 172.21.15.1:0/3642622425 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03f800f040 con 0x7f0408105140 2024-02-28T17:18:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.997+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040810fe10 con 0x7f0408105140 2024-02-28T17:18:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.997+0000 7f03fea00700 1 -- 172.21.15.1:0/3642622425 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03f8009640 con 0x7f0408105140 2024-02-28T17:18:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.997+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0408110350 con 0x7f0408105140 2024-02-28T17:18:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.998+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0408107610 con 0x7f0408105140 2024-02-28T17:18:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:57.999+0000 7f03fea00700 1 -- 172.21.15.1:0/3642622425 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f03f80097a0 con 0x7f0408105140 2024-02-28T17:18:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.000+0000 7f03fea00700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03ec06d7f0 0x7f03ec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.000+0000 7f03fea00700 1 -- 172.21.15.1:0/3642622425 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f03f808cd20 con 0x7f0408105140 2024-02-28T17:18:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.000+0000 7f0406000700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03ec06d7f0 0x7f03ec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.001+0000 7f0406000700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03ec06d7f0 0x7f03ec06fca0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f03f400b3b0 tx=0x7f03f4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.006+0000 7f03fea00700 1 -- 172.21.15.1:0/3642622425 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f03f805a000 con 0x7f0408105140 2024-02-28T17:18:58.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:57 smithi066 ceph-mon[85781]: from='client.36356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:58.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:57 smithi001 ceph-mon[112940]: from='client.36356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:58.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.310+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f04081111e0 con 0x7f0408105140 2024-02-28T17:18:58.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.312+0000 7f03fea00700 1 -- 172.21.15.1:0/3642622425 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f03f8059b90 con 0x7f0408105140 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:18:58.317 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:18:58.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:18:58.318 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:18:58.318 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:18:58.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:18:58.318 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:18:58.318 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:18:58.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:18:58.318 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:18:58.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.314+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03ec06d7f0 msgr2=0x7f03ec06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.314+0000 7f040da00700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03ec06d7f0 0x7f03ec06fca0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f03f400b3b0 tx=0x7f03f4009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.314+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105140 msgr2=0x7f0408115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.314+0000 7f040da00700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105140 0x7f0408115c70 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f03f8002a50 tx=0x7f03f80046b0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.315+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 shutdown_connections 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.315+0000 7f040da00700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03ec06d7f0 0x7f03ec06fca0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.315+0000 7f040da00700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0408105140 0x7f0408115c70 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.315+0000 7f040da00700 1 --2- 172.21.15.1:0/3642622425 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0408105ae0 0x7f04081161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.315+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 >> 172.21.15.1:0/3642622425 conn(0x7f04080facb0 msgr2=0x7f04080fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.315+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 shutdown_connections 2024-02-28T17:18:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.315+0000 7f040da00700 1 -- 172.21.15.1:0/3642622425 wait complete. 2024-02-28T17:18:58.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.471+0000 7f771de00700 1 -- 172.21.15.1:0/2755907136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7718102f50 msgr2=0x7f7718103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.471+0000 7f771de00700 1 --2- 172.21.15.1:0/2755907136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7718102f50 0x7f7718103320 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f7708004940 tx=0x7f7708009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.472+0000 7f771de00700 1 -- 172.21.15.1:0/2755907136 shutdown_connections 2024-02-28T17:18:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.472+0000 7f771de00700 1 --2- 172.21.15.1:0/2755907136 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f77181038f0 0x7f77180fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.472+0000 7f771de00700 1 --2- 172.21.15.1:0/2755907136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7718102f50 0x7f7718103320 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.472+0000 7f771de00700 1 -- 172.21.15.1:0/2755907136 >> 172.21.15.1:0/2755907136 conn(0x7f77180f8ae0 msgr2=0x7f77180faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.472+0000 7f771de00700 1 -- 172.21.15.1:0/2755907136 shutdown_connections 2024-02-28T17:18:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.472+0000 7f771de00700 1 -- 172.21.15.1:0/2755907136 wait complete. 2024-02-28T17:18:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.473+0000 7f771de00700 1 Processor -- start 2024-02-28T17:18:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.473+0000 7f771de00700 1 -- start start 2024-02-28T17:18:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.473+0000 7f771de00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7718102f50 0x7f7718115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.473+0000 7f771de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77181038f0 0x7f77181161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.473+0000 7f771de00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7718116810 con 0x7f77181038f0 2024-02-28T17:18:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.473+0000 7f771de00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771810fb40 con 0x7f7718102f50 2024-02-28T17:18:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.474+0000 7f770fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77181038f0 0x7f77181161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.474+0000 7f770fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77181038f0 0x7f77181161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39964/0 (socket says 172.21.15.1:39964) 2024-02-28T17:18:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.474+0000 7f770fe00700 1 -- 172.21.15.1:0/4135324677 learned_addr learned my addr 172.21.15.1:0/4135324677 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.474+0000 7f7716a00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7718102f50 0x7f7718115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.474+0000 7f770fe00700 1 -- 172.21.15.1:0/4135324677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7718102f50 msgr2=0x7f7718115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.474+0000 7f770fe00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7718102f50 0x7f7718115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.474+0000 7f770fe00700 1 -- 172.21.15.1:0/4135324677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7708005040 con 0x7f77181038f0 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.475+0000 7f770fe00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77181038f0 0x7f77181161c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f770000c800 tx=0x7f7700004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.475+0000 7f770ea00700 1 -- 172.21.15.1:0/4135324677 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f770000a760 con 0x7f77181038f0 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.475+0000 7f770ea00700 1 -- 172.21.15.1:0/4135324677 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f770000ad90 con 0x7f77181038f0 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.475+0000 7f770ea00700 1 -- 172.21.15.1:0/4135324677 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77000113a0 con 0x7f77181038f0 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.475+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771810fe20 con 0x7f77181038f0 2024-02-28T17:18:58.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.475+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7718110360 con 0x7f77181038f0 2024-02-28T17:18:58.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.476+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77180fc6e0 con 0x7f77181038f0 2024-02-28T17:18:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.477+0000 7f770ea00700 1 -- 172.21.15.1:0/4135324677 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7700012400 con 0x7f77181038f0 2024-02-28T17:18:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.479+0000 7f770ea00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f76fc06d7e0 0x7f76fc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.479+0000 7f770ea00700 1 -- 172.21.15.1:0/4135324677 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f770008ce80 con 0x7f77181038f0 2024-02-28T17:18:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.479+0000 7f7716a00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f76fc06d7e0 0x7f76fc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:58.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.480+0000 7f7716a00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f76fc06d7e0 0x7f76fc06fc90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7708004f90 tx=0x7f7708009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:58.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.483+0000 7f770ea00700 1 -- 172.21.15.1:0/4135324677 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f770005a260 con 0x7f77181038f0 2024-02-28T17:18:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.755+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f77181112b0 con 0x7f77181038f0 2024-02-28T17:18:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.756+0000 7f770ea00700 1 -- 172.21.15.1:0/4135324677 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f7700059df0 con 0x7f77181038f0 2024-02-28T17:18:58.761 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:18:58.761 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:18:58.761 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:18:58.762 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:18:58.763 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:18:58.764 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f76fc06d7e0 msgr2=0x7f76fc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f76fc06d7e0 0x7f76fc06fc90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7708004f90 tx=0x7f7708009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77181038f0 msgr2=0x7f77181161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77181038f0 0x7f77181161c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f770000c800 tx=0x7f7700004470 comp rx=0 tx=0).stop 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 shutdown_connections 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f76fc06d7e0 0x7f76fc06fc90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7718102f50 0x7f7718115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 --2- 172.21.15.1:0/4135324677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77181038f0 0x7f77181161c0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.760+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 >> 172.21.15.1:0/4135324677 conn(0x7f77180f8ae0 msgr2=0x7f77180fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:58.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.761+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 shutdown_connections 2024-02-28T17:18:58.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.761+0000 7f771de00700 1 -- 172.21.15.1:0/4135324677 wait complete. 2024-02-28T17:18:58.767 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:18:58.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.924+0000 7fb798e00700 1 -- 172.21.15.1:0/2160617784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fc6e0 msgr2=0x7fb794109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.924+0000 7fb798e00700 1 --2- 172.21.15.1:0/2160617784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fc6e0 0x7fb794109440 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb77c004940 tx=0x7fb77c009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:58.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.924+0000 7fb798e00700 1 -- 172.21.15.1:0/2160617784 shutdown_connections 2024-02-28T17:18:58.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.924+0000 7fb798e00700 1 --2- 172.21.15.1:0/2160617784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fc6e0 0x7fb794109440 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.924+0000 7fb798e00700 1 --2- 172.21.15.1:0/2160617784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7940fbdd0 0x7fb7940fc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.924+0000 7fb798e00700 1 -- 172.21.15.1:0/2160617784 >> 172.21.15.1:0/2160617784 conn(0x7fb794071d90 msgr2=0x7fb794072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:58.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.925+0000 7fb798e00700 1 -- 172.21.15.1:0/2160617784 shutdown_connections 2024-02-28T17:18:58.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.925+0000 7fb798e00700 1 -- 172.21.15.1:0/2160617784 wait complete. 2024-02-28T17:18:58.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.926+0000 7fb798e00700 1 Processor -- start 2024-02-28T17:18:58.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.926+0000 7fb798e00700 1 -- start start 2024-02-28T17:18:58.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.926+0000 7fb798e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fbdd0 0x7fb794115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.926+0000 7fb798e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7940fc6e0 0x7fb794116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.926+0000 7fb798e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7941168d0 con 0x7fb7940fbdd0 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.926+0000 7fb798e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79410fbe0 con 0x7fb7940fc6e0 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.927+0000 7fb791e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fbdd0 0x7fb794115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.927+0000 7fb791e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fbdd0 0x7fb794115d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39982/0 (socket says 172.21.15.1:39982) 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.927+0000 7fb791e00700 1 -- 172.21.15.1:0/517896177 learned_addr learned my addr 172.21.15.1:0/517896177 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.927+0000 7fb791e00700 1 -- 172.21.15.1:0/517896177 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7940fc6e0 msgr2=0x7fb794116280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.927+0000 7fb791e00700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7940fc6e0 0x7fb794116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:58.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.927+0000 7fb791e00700 1 -- 172.21.15.1:0/517896177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c005040 con 0x7fb7940fbdd0 2024-02-28T17:18:58.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.927+0000 7fb791e00700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fbdd0 0x7fb794115d40 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb78400c830 tx=0x7fb784004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:58.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.928+0000 7fb78aa00700 1 -- 172.21.15.1:0/517896177 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78400a760 con 0x7fb7940fbdd0 2024-02-28T17:18:58.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.928+0000 7fb78aa00700 1 -- 172.21.15.1:0/517896177 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb78400ad90 con 0x7fb7940fbdd0 2024-02-28T17:18:58.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.928+0000 7fb78aa00700 1 -- 172.21.15.1:0/517896177 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7840113a0 con 0x7fb7940fbdd0 2024-02-28T17:18:58.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.928+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79410fec0 con 0x7fb7940fbdd0 2024-02-28T17:18:58.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.928+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb794110400 con 0x7fb7940fbdd0 2024-02-28T17:18:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.929+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb794106b70 con 0x7fb7940fbdd0 2024-02-28T17:18:58.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.930+0000 7fb78aa00700 1 -- 172.21.15.1:0/517896177 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb784012400 con 0x7fb7940fbdd0 2024-02-28T17:18:58.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.931+0000 7fb78aa00700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb77806d7f0 0x7fb77806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:58.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.931+0000 7fb78aa00700 1 -- 172.21.15.1:0/517896177 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb78408cf00 con 0x7fb7940fbdd0 2024-02-28T17:18:58.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.931+0000 7fb789400700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb77806d7f0 0x7fb77806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:58.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.932+0000 7fb789400700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb77806d7f0 0x7fb77806fca0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb77c004f90 tx=0x7fb77c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:58.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:58.936+0000 7fb78aa00700 1 -- 172.21.15.1:0/517896177 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb78405a2e0 con 0x7fb7940fbdd0 2024-02-28T17:18:59.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:58 smithi001 ceph-mon[112940]: from='client.36360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:59.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:58 smithi001 ceph-mon[112940]: pgmap v1339: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:59.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:58 smithi001 ceph-mon[112940]: from='client.36364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:59.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:58 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3642622425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:18:59.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:58 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4135324677' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:18:59.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.170+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb794111320 con 0x7fb77806d7f0 2024-02-28T17:18:59.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.172+0000 7fb78aa00700 1 -- 172.21.15.1:0/517896177 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb794111320 con 0x7fb77806d7f0 2024-02-28T17:18:59.177 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:18:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:18:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:18:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:18:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:18:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:18:59.178 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:18:59.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb77806d7f0 msgr2=0x7fb77806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:59.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb77806d7f0 0x7fb77806fca0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb77c004f90 tx=0x7fb77c009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:59.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fbdd0 msgr2=0x7fb794115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:59.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fbdd0 0x7fb794115d40 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb78400c830 tx=0x7fb784004470 comp rx=0 tx=0).stop 2024-02-28T17:18:59.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 shutdown_connections 2024-02-28T17:18:59.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb77806d7f0 0x7fb77806fca0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7940fbdd0 0x7fb794115d40 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 --2- 172.21.15.1:0/517896177 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb7940fc6e0 0x7fb794116280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 >> 172.21.15.1:0/517896177 conn(0x7fb794071d90 msgr2=0x7fb7940fa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:59.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 shutdown_connections 2024-02-28T17:18:59.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.175+0000 7fb798e00700 1 -- 172.21.15.1:0/517896177 wait complete. 2024-02-28T17:18:59.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:58 smithi066 ceph-mon[85781]: from='client.36360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:59.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:58 smithi066 ceph-mon[85781]: pgmap v1339: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:18:59.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:58 smithi066 ceph-mon[85781]: from='client.36364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:18:59.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:58 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3642622425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:18:59.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:58 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4135324677' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:18:59.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.334+0000 7fc11ec00700 1 -- 172.21.15.1:0/3450022946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105170 msgr2=0x7fc118105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:59.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.334+0000 7fc11ec00700 1 --2- 172.21.15.1:0/3450022946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105170 0x7fc118105540 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fc108004940 tx=0x7fc108009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:59.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.334+0000 7fc11ec00700 1 -- 172.21.15.1:0/3450022946 shutdown_connections 2024-02-28T17:18:59.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.334+0000 7fc11ec00700 1 --2- 172.21.15.1:0/3450022946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc118105b10 0x7fc1180ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.334+0000 7fc11ec00700 1 --2- 172.21.15.1:0/3450022946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105170 0x7fc118105540 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.334+0000 7fc11ec00700 1 -- 172.21.15.1:0/3450022946 >> 172.21.15.1:0/3450022946 conn(0x7fc1180face0 msgr2=0x7fc1180fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:59.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.334+0000 7fc11ec00700 1 -- 172.21.15.1:0/3450022946 shutdown_connections 2024-02-28T17:18:59.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.334+0000 7fc11ec00700 1 -- 172.21.15.1:0/3450022946 wait complete. 2024-02-28T17:18:59.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.335+0000 7fc11ec00700 1 Processor -- start 2024-02-28T17:18:59.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.336+0000 7fc11ec00700 1 -- start start 2024-02-28T17:18:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.336+0000 7fc11ec00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc118105170 0x7fc118115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.336+0000 7fc11ec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105b10 0x7fc1181161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.336+0000 7fc11ec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1181167f0 con 0x7fc118105b10 2024-02-28T17:18:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.336+0000 7fc11ec00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11810fb40 con 0x7fc118105170 2024-02-28T17:18:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.336+0000 7fc10fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105b10 0x7fc1181161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.336+0000 7fc10fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105b10 0x7fc1181161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39992/0 (socket says 172.21.15.1:39992) 2024-02-28T17:18:59.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.336+0000 7fc10fe00700 1 -- 172.21.15.1:0/818039192 learned_addr learned my addr 172.21.15.1:0/818039192 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:18:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.337+0000 7fc10fe00700 1 -- 172.21.15.1:0/818039192 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc118105170 msgr2=0x7fc118115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.337+0000 7fc10fe00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc118105170 0x7fc118115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.337+0000 7fc10fe00700 1 -- 172.21.15.1:0/818039192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc108005040 con 0x7fc118105b10 2024-02-28T17:18:59.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.337+0000 7fc10fe00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105b10 0x7fc1181161a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fc100002a50 tx=0x7fc100004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.337+0000 7fc114c00700 1 -- 172.21.15.1:0/818039192 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc100007cc0 con 0x7fc118105b10 2024-02-28T17:18:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.337+0000 7fc114c00700 1 -- 172.21.15.1:0/818039192 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc10000f040 con 0x7fc118105b10 2024-02-28T17:18:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.337+0000 7fc114c00700 1 -- 172.21.15.1:0/818039192 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc10000daf0 con 0x7fc118105b10 2024-02-28T17:18:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.338+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11810fe20 con 0x7fc118105b10 2024-02-28T17:18:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.338+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc118110360 con 0x7fc118105b10 2024-02-28T17:18:59.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.339+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc118102480 con 0x7fc118105b10 2024-02-28T17:18:59.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.340+0000 7fc114c00700 1 -- 172.21.15.1:0/818039192 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc100009810 con 0x7fc118105b10 2024-02-28T17:18:59.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.341+0000 7fc114c00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0fc06d7f0 0x7fc0fc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:18:59.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.341+0000 7fc114c00700 1 -- 172.21.15.1:0/818039192 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc10008cc20 con 0x7fc118105b10 2024-02-28T17:18:59.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.341+0000 7fc117e00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0fc06d7f0 0x7fc0fc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:18:59.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.342+0000 7fc117e00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0fc06d7f0 0x7fc0fc06fca0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc108004f90 tx=0x7fc108009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:18:59.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.346+0000 7fc114c00700 1 -- 172.21.15.1:0/818039192 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc100056450 con 0x7fc118105b10 2024-02-28T17:18:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.644+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc11804bed0 con 0x7fc118105b10 2024-02-28T17:18:59.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.645+0000 7fc114c00700 1 -- 172.21.15.1:0/818039192 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fc100059b10 con 0x7fc118105b10 2024-02-28T17:18:59.650 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:18:59.651 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:18:59.652 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.648+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0fc06d7f0 msgr2=0x7fc0fc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.648+0000 7fc11ec00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0fc06d7f0 0x7fc0fc06fca0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc108004f90 tx=0x7fc108009e30 comp rx=0 tx=0).stop 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.648+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105b10 msgr2=0x7fc1181161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.648+0000 7fc11ec00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105b10 0x7fc1181161a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fc100002a50 tx=0x7fc100004600 comp rx=0 tx=0).stop 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.649+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 shutdown_connections 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.649+0000 7fc11ec00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc0fc06d7f0 0x7fc0fc06fca0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.649+0000 7fc11ec00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc118105170 0x7fc118115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.649+0000 7fc11ec00700 1 --2- 172.21.15.1:0/818039192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc118105b10 0x7fc1181161a0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.649+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 >> 172.21.15.1:0/818039192 conn(0x7fc1180face0 msgr2=0x7fc1180fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.649+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 shutdown_connections 2024-02-28T17:18:59.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:18:59.649+0000 7fc11ec00700 1 -- 172.21.15.1:0/818039192 wait complete. 2024-02-28T17:19:00.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:18:59 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/818039192' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:19:00.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:18:59 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/818039192' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:19:01.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:00 smithi066 ceph-mon[85781]: from='client.36376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:01.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:00 smithi066 ceph-mon[85781]: pgmap v1340: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:01.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:00 smithi001 ceph-mon[112940]: from='client.36376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:01.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:00 smithi001 ceph-mon[112940]: pgmap v1340: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:03.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:02 smithi066 ceph-mon[85781]: pgmap v1341: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:03.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:02 smithi001 ceph-mon[112940]: pgmap v1341: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:05.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:04 smithi066 ceph-mon[85781]: pgmap v1342: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:05.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:04 smithi001 ceph-mon[112940]: pgmap v1342: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:07.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:06 smithi066 ceph-mon[85781]: pgmap v1343: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:07.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:06 smithi001 ceph-mon[112940]: pgmap v1343: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:09.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:08 smithi066 ceph-mon[85781]: pgmap v1344: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:09.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:08 smithi001 ceph-mon[112940]: pgmap v1344: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:11.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:10 smithi066 ceph-mon[85781]: pgmap v1345: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:11.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:19:11.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:10 smithi001 ceph-mon[112940]: pgmap v1345: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:11.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:19:13.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:12 smithi066 ceph-mon[85781]: pgmap v1346: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:13.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:12 smithi001 ceph-mon[112940]: pgmap v1346: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:15.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:14 smithi066 ceph-mon[85781]: pgmap v1347: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:15.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:14 smithi001 ceph-mon[112940]: pgmap v1347: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:17.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:16 smithi066 ceph-mon[85781]: pgmap v1348: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:17.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:16 smithi001 ceph-mon[112940]: pgmap v1348: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:19.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:18 smithi066 ceph-mon[85781]: pgmap v1349: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:19.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:18 smithi001 ceph-mon[112940]: pgmap v1349: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:21.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:20 smithi066 ceph-mon[85781]: pgmap v1350: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:21.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:20 smithi001 ceph-mon[112940]: pgmap v1350: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:23.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:22 smithi066 ceph-mon[85781]: pgmap v1351: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:23.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:22 smithi001 ceph-mon[112940]: pgmap v1351: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:25.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:24 smithi066 ceph-mon[85781]: pgmap v1352: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:25.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:24 smithi001 ceph-mon[112940]: pgmap v1352: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:26.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:19:26.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:19:27.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:26 smithi066 ceph-mon[85781]: pgmap v1353: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:27.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:26 smithi001 ceph-mon[112940]: pgmap v1353: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:29.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:28 smithi066 ceph-mon[85781]: pgmap v1354: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:29.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:28 smithi001 ceph-mon[112940]: pgmap v1354: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:29.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.819+0000 7f31ea000700 1 -- 172.21.15.1:0/38548357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 msgr2=0x7f31e4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:29.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.819+0000 7f31ea000700 1 --2- 172.21.15.1:0/38548357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 0x7f31e4103320 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f31d4004940 tx=0x7f31d4009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:29.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.819+0000 7f31ea000700 1 -- 172.21.15.1:0/38548357 shutdown_connections 2024-02-28T17:19:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.819+0000 7f31ea000700 1 --2- 172.21.15.1:0/38548357 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31e41038f0 0x7f31e40fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.819+0000 7f31ea000700 1 --2- 172.21.15.1:0/38548357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 0x7f31e4103320 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.819+0000 7f31ea000700 1 -- 172.21.15.1:0/38548357 >> 172.21.15.1:0/38548357 conn(0x7f31e40f8ae0 msgr2=0x7f31e40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.819+0000 7f31ea000700 1 -- 172.21.15.1:0/38548357 shutdown_connections 2024-02-28T17:19:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.819+0000 7f31ea000700 1 -- 172.21.15.1:0/38548357 wait complete. 2024-02-28T17:19:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.820+0000 7f31ea000700 1 Processor -- start 2024-02-28T17:19:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.820+0000 7f31ea000700 1 -- start start 2024-02-28T17:19:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.821+0000 7f31ea000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 0x7f31e4115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.821+0000 7f31ea000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31e41038f0 0x7f31e4116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.821+0000 7f31ea000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e4116780 con 0x7f31e4102f50 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.821+0000 7f31ea000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e410fad0 con 0x7f31e41038f0 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.821+0000 7f31e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 0x7f31e4115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.821+0000 7f31e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 0x7f31e4115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55684/0 (socket says 172.21.15.1:55684) 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.821+0000 7f31e3400700 1 -- 172.21.15.1:0/4103038670 learned_addr learned my addr 172.21.15.1:0/4103038670 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.821+0000 7f31dbe00700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31e41038f0 0x7f31e4116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31e3400700 1 -- 172.21.15.1:0/4103038670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31e41038f0 msgr2=0x7f31e4116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31e3400700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31e41038f0 0x7f31e4116130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31e3400700 1 -- 172.21.15.1:0/4103038670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4005040 con 0x7f31e4102f50 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31dbe00700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31e41038f0 0x7f31e4116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:19:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31e3400700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 0x7f31e4115bf0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009150 tx=0x7f31d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31db400700 1 -- 172.21.15.1:0/4103038670 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31d400e040 con 0x7f31e4102f50 2024-02-28T17:19:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31db400700 1 -- 172.21.15.1:0/4103038670 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31d400bbf0 con 0x7f31e4102f50 2024-02-28T17:19:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31db400700 1 -- 172.21.15.1:0/4103038670 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31d401c630 con 0x7f31e4102f50 2024-02-28T17:19:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e410fd50 con 0x7f31e4102f50 2024-02-28T17:19:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.822+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e4110290 con 0x7f31e4102f50 2024-02-28T17:19:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.824+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e40fc6e0 con 0x7f31e4102f50 2024-02-28T17:19:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.824+0000 7f31db400700 1 -- 172.21.15.1:0/4103038670 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f31d400bd60 con 0x7f31e4102f50 2024-02-28T17:19:29.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.825+0000 7f31db400700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31c806d7f0 0x7f31c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:29.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.825+0000 7f31db400700 1 -- 172.21.15.1:0/4103038670 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f31d408e2a0 con 0x7f31e4102f50 2024-02-28T17:19:29.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.826+0000 7f31dbe00700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31c806d7f0 0x7f31c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:29.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.826+0000 7f31dbe00700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31c806d7f0 0x7f31c806fca0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f31cc007fd0 tx=0x7f31cc009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:29.830+0000 7f31db400700 1 -- 172.21.15.1:0/4103038670 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f31d4057b50 con 0x7f31e4102f50 2024-02-28T17:19:30.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.079+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f31e41111d0 con 0x7f31c806d7f0 2024-02-28T17:19:30.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.082+0000 7f31db400700 1 -- 172.21.15.1:0/4103038670 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f31e41111d0 con 0x7f31c806d7f0 2024-02-28T17:19:30.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31c806d7f0 msgr2=0x7f31c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31c806d7f0 0x7f31c806fca0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f31cc007fd0 tx=0x7f31cc009040 comp rx=0 tx=0).stop 2024-02-28T17:19:30.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 msgr2=0x7f31e4115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 0x7f31e4115bf0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009150 tx=0x7f31d4009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:30.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 shutdown_connections 2024-02-28T17:19:30.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f31c806d7f0 0x7f31c806fca0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31e4102f50 0x7f31e4115bf0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 --2- 172.21.15.1:0/4103038670 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f31e41038f0 0x7f31e4116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 >> 172.21.15.1:0/4103038670 conn(0x7f31e40f8ae0 msgr2=0x7f31e40fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:30.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 shutdown_connections 2024-02-28T17:19:30.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.085+0000 7f31ea000700 1 -- 172.21.15.1:0/4103038670 wait complete. 2024-02-28T17:19:30.105 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:19:30.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.246+0000 7fdf9a400700 1 -- 172.21.15.1:0/129311640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070c40 msgr2=0x7fdf94109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.246+0000 7fdf9a400700 1 --2- 172.21.15.1:0/129311640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070c40 0x7fdf94109250 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fdf80004970 tx=0x7fdf80009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:30.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.247+0000 7fdf9a400700 1 -- 172.21.15.1:0/129311640 shutdown_connections 2024-02-28T17:19:30.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.247+0000 7fdf9a400700 1 --2- 172.21.15.1:0/129311640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070c40 0x7fdf94109250 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.247+0000 7fdf9a400700 1 --2- 172.21.15.1:0/129311640 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf94070330 0x7fdf94070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.247+0000 7fdf9a400700 1 -- 172.21.15.1:0/129311640 >> 172.21.15.1:0/129311640 conn(0x7fdf940f8af0 msgr2=0x7fdf940faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:30.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.247+0000 7fdf9a400700 1 -- 172.21.15.1:0/129311640 shutdown_connections 2024-02-28T17:19:30.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.247+0000 7fdf9a400700 1 -- 172.21.15.1:0/129311640 wait complete. 2024-02-28T17:19:30.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.248+0000 7fdf9a400700 1 Processor -- start 2024-02-28T17:19:30.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.248+0000 7fdf9a400700 1 -- start start 2024-02-28T17:19:30.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.248+0000 7fdf9a400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070330 0x7fdf94115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:30.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.248+0000 7fdf9a400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf94070c40 0x7fdf94116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:30.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.248+0000 7fdf9a400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf94116780 con 0x7fdf94070330 2024-02-28T17:19:30.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.248+0000 7fdf9a400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9410fad0 con 0x7fdf94070c40 2024-02-28T17:19:30.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.248+0000 7fdf93400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070330 0x7fdf94115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:30.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.249+0000 7fdf93400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070330 0x7fdf94115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55710/0 (socket says 172.21.15.1:55710) 2024-02-28T17:19:30.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.249+0000 7fdf93400700 1 -- 172.21.15.1:0/1251765995 learned_addr learned my addr 172.21.15.1:0/1251765995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.249+0000 7fdf93400700 1 -- 172.21.15.1:0/1251765995 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf94070c40 msgr2=0x7fdf94116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.249+0000 7fdf93400700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf94070c40 0x7fdf94116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.249+0000 7fdf93400700 1 -- 172.21.15.1:0/1251765995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80005040 con 0x7fdf94070330 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.249+0000 7fdf93400700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070330 0x7fdf94115bf0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400c830 tx=0x7fdf84004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.250+0000 7fdf8b400700 1 -- 172.21.15.1:0/1251765995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf8400a760 con 0x7fdf94070330 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.250+0000 7fdf8b400700 1 -- 172.21.15.1:0/1251765995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf8400ad90 con 0x7fdf94070330 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.250+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9410fdb0 con 0x7fdf94070330 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.250+0000 7fdf8b400700 1 -- 172.21.15.1:0/1251765995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf840113a0 con 0x7fdf94070330 2024-02-28T17:19:30.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.250+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf941102f0 con 0x7fdf94070330 2024-02-28T17:19:30.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.251+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf94106910 con 0x7fdf94070330 2024-02-28T17:19:30.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.252+0000 7fdf8b400700 1 -- 172.21.15.1:0/1251765995 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdf84012400 con 0x7fdf94070330 2024-02-28T17:19:30.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.253+0000 7fdf8b400700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf7806d7f0 0x7fdf7806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:30.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.253+0000 7fdf8b400700 1 -- 172.21.15.1:0/1251765995 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdf8408cfa0 con 0x7fdf94070330 2024-02-28T17:19:30.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.253+0000 7fdf92a00700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf7806d7f0 0x7fdf7806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:30.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.254+0000 7fdf92a00700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf7806d7f0 0x7fdf7806fca0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b370 tx=0x7fdf80009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:30.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.258+0000 7fdf8b400700 1 -- 172.21.15.1:0/1251765995 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdf840567d0 con 0x7fdf94070330 2024-02-28T17:19:30.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.502+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf941111d0 con 0x7fdf7806d7f0 2024-02-28T17:19:30.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.505+0000 7fdf8b400700 1 -- 172.21.15.1:0/1251765995 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdf941111d0 con 0x7fdf7806d7f0 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf7806d7f0 msgr2=0x7fdf7806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf7806d7f0 0x7fdf7806fca0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b370 tx=0x7fdf80009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070330 msgr2=0x7fdf94115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070330 0x7fdf94115bf0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400c830 tx=0x7fdf84004470 comp rx=0 tx=0).stop 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 shutdown_connections 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf7806d7f0 0x7fdf7806fca0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf94070330 0x7fdf94115bf0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 --2- 172.21.15.1:0/1251765995 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf94070c40 0x7fdf94116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 >> 172.21.15.1:0/1251765995 conn(0x7fdf940f8af0 msgr2=0x7fdf94103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:30.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 shutdown_connections 2024-02-28T17:19:30.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.508+0000 7fdf9a400700 1 -- 172.21.15.1:0/1251765995 wait complete. 2024-02-28T17:19:30.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.670+0000 7f3817c00700 1 -- 172.21.15.1:0/4211119951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 msgr2=0x7f381010d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.670+0000 7f3817c00700 1 --2- 172.21.15.1:0/4211119951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 0x7f381010d8c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f37fc004940 tx=0x7f37fc009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:30.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.670+0000 7f3817c00700 1 -- 172.21.15.1:0/4211119951 shutdown_connections 2024-02-28T17:19:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.670+0000 7f3817c00700 1 --2- 172.21.15.1:0/4211119951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 0x7f381010d8c0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.670+0000 7f3817c00700 1 --2- 172.21.15.1:0/4211119951 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38100724c0 0x7f3810072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.670+0000 7f3817c00700 1 -- 172.21.15.1:0/4211119951 >> 172.21.15.1:0/4211119951 conn(0x7f38100face0 msgr2=0x7f38100fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.670+0000 7f3817c00700 1 -- 172.21.15.1:0/4211119951 shutdown_connections 2024-02-28T17:19:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.670+0000 7f3817c00700 1 -- 172.21.15.1:0/4211119951 wait complete. 2024-02-28T17:19:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.671+0000 7f3817c00700 1 Processor -- start 2024-02-28T17:19:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.671+0000 7f3817c00700 1 -- start start 2024-02-28T17:19:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.672+0000 7f3817c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38100724c0 0x7f381011a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.672+0000 7f3817c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 0x7f381011a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.672+0000 7f3817c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381011ac10 con 0x7f3810072dd0 2024-02-28T17:19:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.672+0000 7f3817c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3810113f60 con 0x7f38100724c0 2024-02-28T17:19:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.672+0000 7f380fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 0x7f381011a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.672+0000 7f380fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 0x7f381011a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55726/0 (socket says 172.21.15.1:55726) 2024-02-28T17:19:30.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.672+0000 7f380fe00700 1 -- 172.21.15.1:0/189086552 learned_addr learned my addr 172.21.15.1:0/189086552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:19:30.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.672+0000 7f3815200700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38100724c0 0x7f381011a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:30.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f380fe00700 1 -- 172.21.15.1:0/189086552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38100724c0 msgr2=0x7f381011a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f380fe00700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38100724c0 0x7f381011a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f380fe00700 1 -- 172.21.15.1:0/189086552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc005040 con 0x7f3810072dd0 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f380fe00700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 0x7f381011a5c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009390 tx=0x7f37fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f380cc00700 1 -- 172.21.15.1:0/189086552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37fc00e040 con 0x7f3810072dd0 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f380cc00700 1 -- 172.21.15.1:0/189086552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37fc004290 con 0x7f3810072dd0 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f380cc00700 1 -- 172.21.15.1:0/189086552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37fc01c630 con 0x7f3810072dd0 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38101141e0 con 0x7f3810072dd0 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.673+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3810114720 con 0x7f3810072dd0 2024-02-28T17:19:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.675+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381010aff0 con 0x7f3810072dd0 2024-02-28T17:19:30.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.676+0000 7f380cc00700 1 -- 172.21.15.1:0/189086552 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f37fc013420 con 0x7f3810072dd0 2024-02-28T17:19:30.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.677+0000 7f380cc00700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37f806d7f0 0x7f37f806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:30.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.677+0000 7f380cc00700 1 -- 172.21.15.1:0/189086552 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f37fc08e3a0 con 0x7f3810072dd0 2024-02-28T17:19:30.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.677+0000 7f3815200700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37f806d7f0 0x7f37f806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:30.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.678+0000 7f3815200700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37f806d7f0 0x7f37f806fca0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3804007fd0 tx=0x7f3804009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:30.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.682+0000 7f380cc00700 1 -- 172.21.15.1:0/189086552 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f37fc057d00 con 0x7f3810072dd0 2024-02-28T17:19:30.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.912+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3810110630 con 0x7f37f806d7f0 2024-02-28T17:19:30.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.947+0000 7f380cc00700 1 -- 172.21.15.1:0/189086552 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3810110630 con 0x7f37f806d7f0 2024-02-28T17:19:30.951 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:19:30.951 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (45m) 7m ago 59m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (42m) 7m ago 60m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (42m) 7m ago 58m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (45m) 7m ago 59m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (38m) 7m ago 52m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (37m) 7m ago 52m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (7m) 7m ago 52m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (22m) 7m ago 52m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (48m) 7m ago 61m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (47m) 7m ago 58m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (44m) 7m ago 61m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (43m) 7m ago 58m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (46m) 7m ago 60m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:19:30.952 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (46m) 7m ago 58m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:19:30.953 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (41m) 7m ago 57m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:19:30.953 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (41m) 7m ago 56m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:19:30.953 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (40m) 7m ago 56m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:19:30.953 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (40m) 7m ago 55m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:19:30.953 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (39m) 7m ago 55m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:19:30.953 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (38m) 7m ago 54m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:19:30.953 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (46m) 7m ago 59m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37f806d7f0 msgr2=0x7f37f806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37f806d7f0 0x7f37f806fca0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3804007fd0 tx=0x7f3804009040 comp rx=0 tx=0).stop 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 msgr2=0x7f381011a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 0x7f381011a5c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009390 tx=0x7f37fc009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 shutdown_connections 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f37f806d7f0 0x7f37f806fca0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f38100724c0 0x7f381011a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 --2- 172.21.15.1:0/189086552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3810072dd0 0x7f381011a5c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 >> 172.21.15.1:0/189086552 conn(0x7f38100face0 msgr2=0x7f38100fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 shutdown_connections 2024-02-28T17:19:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:30.950+0000 7f3817c00700 1 -- 172.21.15.1:0/189086552 wait complete. 2024-02-28T17:19:31.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.093+0000 7f23da800700 1 -- 172.21.15.1:0/3839554627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 msgr2=0x7f23d4105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:31.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.093+0000 7f23da800700 1 --2- 172.21.15.1:0/3839554627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 0x7f23d4105340 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f23c8004940 tx=0x7f23c8009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:31.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.093+0000 7f23da800700 1 -- 172.21.15.1:0/3839554627 shutdown_connections 2024-02-28T17:19:31.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.093+0000 7f23da800700 1 --2- 172.21.15.1:0/3839554627 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23d4105910 0x7f23d40fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.093+0000 7f23da800700 1 --2- 172.21.15.1:0/3839554627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 0x7f23d4105340 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.093+0000 7f23da800700 1 -- 172.21.15.1:0/3839554627 >> 172.21.15.1:0/3839554627 conn(0x7f23d40faae0 msgr2=0x7f23d40fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:31.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.093+0000 7f23da800700 1 -- 172.21.15.1:0/3839554627 shutdown_connections 2024-02-28T17:19:31.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.093+0000 7f23da800700 1 -- 172.21.15.1:0/3839554627 wait complete. 2024-02-28T17:19:31.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.094+0000 7f23da800700 1 Processor -- start 2024-02-28T17:19:31.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.094+0000 7f23da800700 1 -- start start 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.094+0000 7f23da800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 0x7f23d4115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.094+0000 7f23da800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23d4105910 0x7f23d4115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.094+0000 7f23da800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d4116660 con 0x7f23d4104f70 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.094+0000 7f23da800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d410f900 con 0x7f23d4105910 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.095+0000 7f23d9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 0x7f23d4115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.095+0000 7f23d9400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 0x7f23d4115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55756/0 (socket says 172.21.15.1:55756) 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.095+0000 7f23d9400700 1 -- 172.21.15.1:0/1092113515 learned_addr learned my addr 172.21.15.1:0/1092113515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.095+0000 7f23d9400700 1 -- 172.21.15.1:0/1092113515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23d4105910 msgr2=0x7f23d4115f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.095+0000 7f23d9400700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23d4105910 0x7f23d4115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.095+0000 7f23d9400700 1 -- 172.21.15.1:0/1092113515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8005040 con 0x7f23d4104f70 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.095+0000 7f23d9400700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 0x7f23d4115a40 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009150 tx=0x7f23c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.096+0000 7f23d1600700 1 -- 172.21.15.1:0/1092113515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23c800e040 con 0x7f23d4104f70 2024-02-28T17:19:31.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.096+0000 7f23d1600700 1 -- 172.21.15.1:0/1092113515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23c800bbf0 con 0x7f23d4104f70 2024-02-28T17:19:31.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.096+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d410fb80 con 0x7f23d4104f70 2024-02-28T17:19:31.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.096+0000 7f23d1600700 1 -- 172.21.15.1:0/1092113515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23c801c630 con 0x7f23d4104f70 2024-02-28T17:19:31.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.096+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d4110140 con 0x7f23d4104f70 2024-02-28T17:19:31.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.097+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d405f5c0 con 0x7f23d4104f70 2024-02-28T17:19:31.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.098+0000 7f23d1600700 1 -- 172.21.15.1:0/1092113515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f23c800bd60 con 0x7f23d4104f70 2024-02-28T17:19:31.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.099+0000 7f23d1600700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23bc06d7f0 0x7f23bc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:31.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.099+0000 7f23d1600700 1 -- 172.21.15.1:0/1092113515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f23c808e290 con 0x7f23d4104f70 2024-02-28T17:19:31.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.099+0000 7f23d8a00700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23bc06d7f0 0x7f23bc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:31.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.100+0000 7f23d8a00700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23bc06d7f0 0x7f23bc06fca0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f23c0006fd0 tx=0x7f23c0006db0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:31.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.104+0000 7f23d1600700 1 -- 172.21.15.1:0/1092113515 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f23c805b670 con 0x7f23d4104f70 2024-02-28T17:19:31.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:30 smithi066 ceph-mon[85781]: pgmap v1355: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:31.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:30 smithi001 ceph-mon[112940]: pgmap v1355: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.403+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f23d404bed0 con 0x7f23d4104f70 2024-02-28T17:19:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.404+0000 7f23d1600700 1 -- 172.21.15.1:0/1092113515 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f23c805b200 con 0x7f23d4104f70 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:19:31.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:19:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:19:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:19:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:19:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:19:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:19:31.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:19:31.410 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.407+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23bc06d7f0 msgr2=0x7f23bc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.407+0000 7f23da800700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23bc06d7f0 0x7f23bc06fca0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f23c0006fd0 tx=0x7f23c0006db0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.407+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 msgr2=0x7f23d4115a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.407+0000 7f23da800700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 0x7f23d4115a40 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009150 tx=0x7f23c8009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.408+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 shutdown_connections 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.408+0000 7f23da800700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f23bc06d7f0 0x7f23bc06fca0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.408+0000 7f23da800700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d4104f70 0x7f23d4115a40 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.408+0000 7f23da800700 1 --2- 172.21.15.1:0/1092113515 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f23d4105910 0x7f23d4115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.408+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 >> 172.21.15.1:0/1092113515 conn(0x7f23d40faae0 msgr2=0x7f23d40fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:31.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.408+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 shutdown_connections 2024-02-28T17:19:31.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.408+0000 7f23da800700 1 -- 172.21.15.1:0/1092113515 wait complete. 2024-02-28T17:19:31.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.564+0000 7f08b1e00700 1 -- 172.21.15.1:0/3891459388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 msgr2=0x7f08ac0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:31.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.564+0000 7f08b1e00700 1 --2- 172.21.15.1:0/3891459388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 0x7f08ac0fcf60 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0894004940 tx=0x7f0894009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:31.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.565+0000 7f08b1e00700 1 -- 172.21.15.1:0/3891459388 shutdown_connections 2024-02-28T17:19:31.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.565+0000 7f08b1e00700 1 --2- 172.21.15.1:0/3891459388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 0x7f08ac0fcf60 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.565+0000 7f08b1e00700 1 --2- 172.21.15.1:0/3891459388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08ac102f50 0x7f08ac103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.565+0000 7f08b1e00700 1 -- 172.21.15.1:0/3891459388 >> 172.21.15.1:0/3891459388 conn(0x7f08ac0f8ae0 msgr2=0x7f08ac0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:31.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.565+0000 7f08b1e00700 1 -- 172.21.15.1:0/3891459388 shutdown_connections 2024-02-28T17:19:31.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.565+0000 7f08b1e00700 1 -- 172.21.15.1:0/3891459388 wait complete. 2024-02-28T17:19:31.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.565+0000 7f08b1e00700 1 Processor -- start 2024-02-28T17:19:31.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.566+0000 7f08b1e00700 1 -- start start 2024-02-28T17:19:31.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.566+0000 7f08b1e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08ac102f50 0x7f08ac115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.566+0000 7f08b1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 0x7f08ac1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.566+0000 7f08b1e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac116810 con 0x7f08ac1038f0 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.566+0000 7f08b1e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac10fb40 con 0x7f08ac102f50 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.566+0000 7f08a3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 0x7f08ac1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.566+0000 7f08a3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 0x7f08ac1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55774/0 (socket says 172.21.15.1:55774) 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.566+0000 7f08a3e00700 1 -- 172.21.15.1:0/1969328574 learned_addr learned my addr 172.21.15.1:0/1969328574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.567+0000 7f08aaa00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08ac102f50 0x7f08ac115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.567+0000 7f08a3e00700 1 -- 172.21.15.1:0/1969328574 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08ac102f50 msgr2=0x7f08ac115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.567+0000 7f08a3e00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08ac102f50 0x7f08ac115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.567+0000 7f08a3e00700 1 -- 172.21.15.1:0/1969328574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894005040 con 0x7f08ac1038f0 2024-02-28T17:19:31.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.567+0000 7f08a3e00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 0x7f08ac1161c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f0894009100 tx=0x7f0894009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:31.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.567+0000 7f08a2a00700 1 -- 172.21.15.1:0/1969328574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f089400e040 con 0x7f08ac1038f0 2024-02-28T17:19:31.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.568+0000 7f08a2a00700 1 -- 172.21.15.1:0/1969328574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0894004290 con 0x7f08ac1038f0 2024-02-28T17:19:31.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.568+0000 7f08a2a00700 1 -- 172.21.15.1:0/1969328574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f089401c3f0 con 0x7f08ac1038f0 2024-02-28T17:19:31.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.568+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac10fdc0 con 0x7f08ac1038f0 2024-02-28T17:19:31.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.568+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac110300 con 0x7f08ac1038f0 2024-02-28T17:19:31.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.569+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ac0fc6e0 con 0x7f08ac1038f0 2024-02-28T17:19:31.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.570+0000 7f08a2a00700 1 -- 172.21.15.1:0/1969328574 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f089400bbf0 con 0x7f08ac1038f0 2024-02-28T17:19:31.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.571+0000 7f08a2a00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f089006d7f0 0x7f089006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:31.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.571+0000 7f08a2a00700 1 -- 172.21.15.1:0/1969328574 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f089408e1c0 con 0x7f08ac1038f0 2024-02-28T17:19:31.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.571+0000 7f08aaa00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f089006d7f0 0x7f089006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:31.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.572+0000 7f08aaa00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f089006d7f0 0x7f089006fca0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f089c004990 tx=0x7f089c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:31.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.576+0000 7f08a2a00700 1 -- 172.21.15.1:0/1969328574 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f089405b5a0 con 0x7f08ac1038f0 2024-02-28T17:19:31.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.850+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f08ac1111f0 con 0x7f08ac1038f0 2024-02-28T17:19:31.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.852+0000 7f08a2a00700 1 -- 172.21.15.1:0/1969328574 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f089405b130 con 0x7f08ac1038f0 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:19:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:19:31.857 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:19:31.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:19:31.859 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:19:31.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:19:31.859 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:19:31.859 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:19:31.859 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:19:31.859 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.854+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f089006d7f0 msgr2=0x7f089006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.854+0000 7f08b1e00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f089006d7f0 0x7f089006fca0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f089c004990 tx=0x7f089c009ee0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.854+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 msgr2=0x7f08ac1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.854+0000 7f08b1e00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 0x7f08ac1161c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f0894009100 tx=0x7f0894009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.855+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 shutdown_connections 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.855+0000 7f08b1e00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f089006d7f0 0x7f089006fca0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.855+0000 7f08b1e00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08ac102f50 0x7f08ac115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.855+0000 7f08b1e00700 1 --2- 172.21.15.1:0/1969328574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1038f0 0x7f08ac1161c0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.855+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 >> 172.21.15.1:0/1969328574 conn(0x7f08ac0f8ae0 msgr2=0x7f08ac0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.855+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 shutdown_connections 2024-02-28T17:19:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:31.855+0000 7f08b1e00700 1 -- 172.21.15.1:0/1969328574 wait complete. 2024-02-28T17:19:31.861 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:19:32.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.004+0000 7f8167000700 1 -- 172.21.15.1:0/452002775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 msgr2=0x7f81601099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:32.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.004+0000 7f8167000700 1 --2- 172.21.15.1:0/452002775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 0x7f81601099c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8150004940 tx=0x7f8150009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:32.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.004+0000 7f8167000700 1 -- 172.21.15.1:0/452002775 shutdown_connections 2024-02-28T17:19:32.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.004+0000 7f8167000700 1 --2- 172.21.15.1:0/452002775 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8160103dc0 0x7f8160104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.004+0000 7f8167000700 1 --2- 172.21.15.1:0/452002775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 0x7f81601099c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.004+0000 7f8167000700 1 -- 172.21.15.1:0/452002775 >> 172.21.15.1:0/452002775 conn(0x7f8160072d50 msgr2=0x7f8160075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:32.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.005+0000 7f8167000700 1 -- 172.21.15.1:0/452002775 shutdown_connections 2024-02-28T17:19:32.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.005+0000 7f8167000700 1 -- 172.21.15.1:0/452002775 wait complete. 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.005+0000 7f8167000700 1 Processor -- start 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.006+0000 7f8167000700 1 -- start start 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.006+0000 7f8167000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8160103dc0 0x7f816011a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.006+0000 7f8167000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 0x7f816011a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.006+0000 7f8167000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816011abf0 con 0x7f81601095f0 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.006+0000 7f8167000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8160113f00 con 0x7f8160103dc0 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.006+0000 7f815f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 0x7f816011a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.006+0000 7f815f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 0x7f816011a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38920/0 (socket says 172.21.15.1:38920) 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.006+0000 7f815f400700 1 -- 172.21.15.1:0/4035003009 learned_addr learned my addr 172.21.15.1:0/4035003009 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f815fe00700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8160103dc0 0x7f816011a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:32.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f815f400700 1 -- 172.21.15.1:0/4035003009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8160103dc0 msgr2=0x7f816011a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:32.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f815f400700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8160103dc0 0x7f816011a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f815f400700 1 -- 172.21.15.1:0/4035003009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8150005040 con 0x7f81601095f0 2024-02-28T17:19:32.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f815f400700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 0x7f816011a5a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f814c002a50 tx=0x7f814c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:32.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f8157e00700 1 -- 172.21.15.1:0/4035003009 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f814c007cc0 con 0x7f81601095f0 2024-02-28T17:19:32.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f8157e00700 1 -- 172.21.15.1:0/4035003009 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f814c00f040 con 0x7f81601095f0 2024-02-28T17:19:32.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f8157e00700 1 -- 172.21.15.1:0/4035003009 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f814c00db70 con 0x7f81601095f0 2024-02-28T17:19:32.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.007+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81601141e0 con 0x7f81601095f0 2024-02-28T17:19:32.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.008+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8160114720 con 0x7f81601095f0 2024-02-28T17:19:32.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.009+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8160075870 con 0x7f81601095f0 2024-02-28T17:19:32.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.010+0000 7f8157e00700 1 -- 172.21.15.1:0/4035003009 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f814c00dd10 con 0x7f81601095f0 2024-02-28T17:19:32.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.011+0000 7f8157e00700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f814406d7f0 0x7f814406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:32.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.011+0000 7f8157e00700 1 -- 172.21.15.1:0/4035003009 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f814c08c590 con 0x7f81601095f0 2024-02-28T17:19:32.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.011+0000 7f815fe00700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f814406d7f0 0x7f814406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:32.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.012+0000 7f815fe00700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f814406d7f0 0x7f814406fca0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f815000b3b0 tx=0x7f8150009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:32.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.015+0000 7f8157e00700 1 -- 172.21.15.1:0/4035003009 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f814c0598f0 con 0x7f81601095f0 2024-02-28T17:19:32.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.256+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8160115750 con 0x7f814406d7f0 2024-02-28T17:19:32.261 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:31 smithi001 ceph-mon[112940]: from='client.36384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:32.261 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:31 smithi001 ceph-mon[112940]: from='client.36388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:32.261 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:31 smithi001 ceph-mon[112940]: from='client.36392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:32.261 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:31 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1092113515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:19:32.261 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:31 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1969328574' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.258+0000 7f8157e00700 1 -- 172.21.15.1:0/4035003009 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8160115750 con 0x7f814406d7f0 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:19:32.262 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:19:32.263 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:19:32.263 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:19:32.263 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:19:32.263 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:19:32.263 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:19:32.263 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:19:32.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:31 smithi066 ceph-mon[85781]: from='client.36384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:32.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:31 smithi066 ceph-mon[85781]: from='client.36388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:32.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:31 smithi066 ceph-mon[85781]: from='client.36392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:32.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:31 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1092113515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:19:32.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:31 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1969328574' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:19:32.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.260+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f814406d7f0 msgr2=0x7f814406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:32.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.260+0000 7f8167000700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f814406d7f0 0x7f814406fca0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f815000b3b0 tx=0x7f8150009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:32.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.260+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 msgr2=0x7f816011a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:32.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.260+0000 7f8167000700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 0x7f816011a5a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f814c002a50 tx=0x7f814c0046b0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.261+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 shutdown_connections 2024-02-28T17:19:32.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.261+0000 7f8167000700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f814406d7f0 0x7f814406fca0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.261+0000 7f8167000700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8160103dc0 0x7f816011a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.261+0000 7f8167000700 1 --2- 172.21.15.1:0/4035003009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81601095f0 0x7f816011a5a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.261+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 >> 172.21.15.1:0/4035003009 conn(0x7f8160072d50 msgr2=0x7f8160074760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:32.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.261+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 shutdown_connections 2024-02-28T17:19:32.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.261+0000 7f8167000700 1 -- 172.21.15.1:0/4035003009 wait complete. 2024-02-28T17:19:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.412+0000 7f66b5e00700 1 -- 172.21.15.1:0/1062217356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 msgr2=0x7f66b0103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.412+0000 7f66b5e00700 1 --2- 172.21.15.1:0/1062217356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 0x7f66b0103310 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f66a0004940 tx=0x7f66a0009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.412+0000 7f66b5e00700 1 -- 172.21.15.1:0/1062217356 shutdown_connections 2024-02-28T17:19:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.412+0000 7f66b5e00700 1 --2- 172.21.15.1:0/1062217356 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66b01038e0 0x7f66b00fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.412+0000 7f66b5e00700 1 --2- 172.21.15.1:0/1062217356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 0x7f66b0103310 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.412+0000 7f66b5e00700 1 -- 172.21.15.1:0/1062217356 >> 172.21.15.1:0/1062217356 conn(0x7f66b00f8af0 msgr2=0x7f66b00faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.412+0000 7f66b5e00700 1 -- 172.21.15.1:0/1062217356 shutdown_connections 2024-02-28T17:19:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.412+0000 7f66b5e00700 1 -- 172.21.15.1:0/1062217356 wait complete. 2024-02-28T17:19:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.413+0000 7f66b5e00700 1 Processor -- start 2024-02-28T17:19:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.413+0000 7f66b5e00700 1 -- start start 2024-02-28T17:19:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66b5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 0x7f66b0115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66b5e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66b01038e0 0x7f66b0116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66b5e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b0116900 con 0x7f66b0102f40 2024-02-28T17:19:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66b5e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b010fbc0 con 0x7f66b01038e0 2024-02-28T17:19:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66aea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 0x7f66b0115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66aea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 0x7f66b0115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38926/0 (socket says 172.21.15.1:38926) 2024-02-28T17:19:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66aea00700 1 -- 172.21.15.1:0/2694491423 learned_addr learned my addr 172.21.15.1:0/2694491423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66ae000700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66b01038e0 0x7f66b0116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66aea00700 1 -- 172.21.15.1:0/2694491423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66b01038e0 msgr2=0x7f66b0116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66aea00700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66b01038e0 0x7f66b0116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.414+0000 7f66aea00700 1 -- 172.21.15.1:0/2694491423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a0005040 con 0x7f66b0102f40 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.415+0000 7f66aea00700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 0x7f66b0115ce0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009fd0 tx=0x7f66a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.415+0000 7f66a6a00700 1 -- 172.21.15.1:0/2694491423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a000e040 con 0x7f66b0102f40 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.415+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b010fe40 con 0x7f66b0102f40 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.415+0000 7f66a6a00700 1 -- 172.21.15.1:0/2694491423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66a000bbf0 con 0x7f66b0102f40 2024-02-28T17:19:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.415+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b0110380 con 0x7f66b0102f40 2024-02-28T17:19:32.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.416+0000 7f66a6a00700 1 -- 172.21.15.1:0/2694491423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a00255e0 con 0x7f66b0102f40 2024-02-28T17:19:32.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.416+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b00fc6f0 con 0x7f66b0102f40 2024-02-28T17:19:32.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.417+0000 7f66a6a00700 1 -- 172.21.15.1:0/2694491423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f66a000bd60 con 0x7f66b0102f40 2024-02-28T17:19:32.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.419+0000 7f66a6a00700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f669406d7f0 0x7f669406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:19:32.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.419+0000 7f66a6a00700 1 -- 172.21.15.1:0/2694491423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f66a008e240 con 0x7f66b0102f40 2024-02-28T17:19:32.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.419+0000 7f66ae000700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f669406d7f0 0x7f669406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:19:32.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.420+0000 7f66ae000700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f669406d7f0 0x7f669406fca0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f669c006fd0 tx=0x7f669c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:19:32.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.423+0000 7f66a6a00700 1 -- 172.21.15.1:0/2694491423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f66a0057a70 con 0x7f66b0102f40 2024-02-28T17:19:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.722+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f66b010a0b0 con 0x7f66b0102f40 2024-02-28T17:19:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.723+0000 7f66a6a00700 1 -- 172.21.15.1:0/2694491423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f66a005b130 con 0x7f66b0102f40 2024-02-28T17:19:32.727 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:19:32.727 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:19:32.727 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:19:32.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:19:32.727 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:19:32.727 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:19:32.727 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:19:32.728 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:19:32.729 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:19:32.729 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:19:32.729 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:19:32.729 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:19:32.729 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.725+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f669406d7f0 msgr2=0x7f669406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.725+0000 7f66b5e00700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f669406d7f0 0x7f669406fca0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f669c006fd0 tx=0x7f669c008040 comp rx=0 tx=0).stop 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.725+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 msgr2=0x7f66b0115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.725+0000 7f66b5e00700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 0x7f66b0115ce0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009fd0 tx=0x7f66a0009e30 comp rx=0 tx=0).stop 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.726+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 shutdown_connections 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.726+0000 7f66b5e00700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f669406d7f0 0x7f669406fca0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.726+0000 7f66b5e00700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b0102f40 0x7f66b0115ce0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.726+0000 7f66b5e00700 1 --2- 172.21.15.1:0/2694491423 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f66b01038e0 0x7f66b0116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.726+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 >> 172.21.15.1:0/2694491423 conn(0x7f66b00f8af0 msgr2=0x7f66b00fa340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.726+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 shutdown_connections 2024-02-28T17:19:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:19:32.726+0000 7f66b5e00700 1 -- 172.21.15.1:0/2694491423 wait complete. 2024-02-28T17:19:33.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:32 smithi001 ceph-mon[112940]: pgmap v1356: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:33.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:32 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2694491423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:19:33.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:32 smithi066 ceph-mon[85781]: pgmap v1356: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:33.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:32 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2694491423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:19:34.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:33 smithi066 ceph-mon[85781]: from='client.36404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:34.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:33 smithi001 ceph-mon[112940]: from='client.36404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:19:35.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:34 smithi066 ceph-mon[85781]: pgmap v1357: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:35.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:34 smithi001 ceph-mon[112940]: pgmap v1357: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:37.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:36 smithi066 ceph-mon[85781]: pgmap v1358: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:37.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:36 smithi001 ceph-mon[112940]: pgmap v1358: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:39.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:38 smithi066 ceph-mon[85781]: pgmap v1359: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:39.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:38 smithi001 ceph-mon[112940]: pgmap v1359: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:40 smithi066 ceph-mon[85781]: pgmap v1360: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:19:41.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:40 smithi001 ceph-mon[112940]: pgmap v1360: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:41.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:19:43.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:42 smithi066 ceph-mon[85781]: pgmap v1361: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:43.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:42 smithi001 ceph-mon[112940]: pgmap v1361: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:45.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:44 smithi066 ceph-mon[85781]: pgmap v1362: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:45.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:44 smithi001 ceph-mon[112940]: pgmap v1362: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:47.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:46 smithi066 ceph-mon[85781]: pgmap v1363: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:47.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:46 smithi001 ceph-mon[112940]: pgmap v1363: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:49.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:49 smithi066 ceph-mon[85781]: pgmap v1364: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:49.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:49 smithi001 ceph-mon[112940]: pgmap v1364: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:51.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:51 smithi001 ceph-mon[112940]: pgmap v1365: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:51.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:51 smithi066 ceph-mon[85781]: pgmap v1365: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:53.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:53 smithi001 ceph-mon[112940]: pgmap v1366: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:53.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:53 smithi066 ceph-mon[85781]: pgmap v1366: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:55.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:55 smithi001 ceph-mon[112940]: pgmap v1367: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:55.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:55 smithi066 ceph-mon[85781]: pgmap v1367: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:56.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:19:56.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:19:57.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:57 smithi066 ceph-mon[85781]: pgmap v1368: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:57.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:57 smithi001 ceph-mon[112940]: pgmap v1368: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:59.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:19:59 smithi066 ceph-mon[85781]: pgmap v1369: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:19:59.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:19:59 smithi001 ceph-mon[112940]: pgmap v1369: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:00.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:00 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T17:19:59.998+0000 7f7e40a00700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:20:00.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:00 smithi001 ceph-mon[112940]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:20:00.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:00 smithi066 ceph-mon[85781]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:20:01.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:01 smithi066 ceph-mon[85781]: pgmap v1370: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:01.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:01 smithi001 ceph-mon[112940]: pgmap v1370: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:02.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:02 smithi066 ceph-mon[85781]: pgmap v1371: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:02.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:02 smithi001 ceph-mon[112940]: pgmap v1371: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:02.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.896+0000 7f7971e00700 1 -- 172.21.15.1:0/1036969822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c0fb390 msgr2=0x7f796c0fb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:02.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.896+0000 7f7971e00700 1 --2- 172.21.15.1:0/1036969822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c0fb390 0x7f796c0fb7e0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f7958004940 tx=0x7f7958009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:02.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.896+0000 7f7971e00700 1 -- 172.21.15.1:0/1036969822 shutdown_connections 2024-02-28T17:20:02.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.896+0000 7f7971e00700 1 --2- 172.21.15.1:0/1036969822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c0fb390 0x7f796c0fb7e0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:02.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.896+0000 7f7971e00700 1 --2- 172.21.15.1:0/1036969822 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f796c102dc0 0x7f796c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:02.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.896+0000 7f7971e00700 1 -- 172.21.15.1:0/1036969822 >> 172.21.15.1:0/1036969822 conn(0x7f796c072270 msgr2=0x7f796c072670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:02.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.896+0000 7f7971e00700 1 -- 172.21.15.1:0/1036969822 shutdown_connections 2024-02-28T17:20:02.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.896+0000 7f7971e00700 1 -- 172.21.15.1:0/1036969822 wait complete. 2024-02-28T17:20:02.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.897+0000 7f7971e00700 1 Processor -- start 2024-02-28T17:20:02.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.897+0000 7f7971e00700 1 -- start start 2024-02-28T17:20:02.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.898+0000 7f7971e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f796c0fb390 0x7f796c115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:02.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.898+0000 7f7971e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c102dc0 0x7f796c115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:02.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.898+0000 7f7971e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c1165d0 con 0x7f796c102dc0 2024-02-28T17:20:02.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.898+0000 7f7971e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c10f900 con 0x7f796c0fb390 2024-02-28T17:20:02.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.898+0000 7f796be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c102dc0 0x7f796c115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:02.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.898+0000 7f796be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c102dc0 0x7f796c115f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50994/0 (socket says 172.21.15.1:50994) 2024-02-28T17:20:02.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.898+0000 7f796be00700 1 -- 172.21.15.1:0/1986861153 learned_addr learned my addr 172.21.15.1:0/1986861153 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:02.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.898+0000 7f7970a00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f796c0fb390 0x7f796c115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f796be00700 1 -- 172.21.15.1:0/1986861153 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f796c0fb390 msgr2=0x7f796c115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f796be00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f796c0fb390 0x7f796c115a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f796be00700 1 -- 172.21.15.1:0/1986861153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958005040 con 0x7f796c102dc0 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f796be00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c102dc0 0x7f796c115f80 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7958009fd0 tx=0x7f7958009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f7968c00700 1 -- 172.21.15.1:0/1986861153 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f795800e040 con 0x7f796c102dc0 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f7968c00700 1 -- 172.21.15.1:0/1986861153 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f795800bbf0 con 0x7f796c102dc0 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f7968c00700 1 -- 172.21.15.1:0/1986861153 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f795801c3f0 con 0x7f796c102dc0 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c10fb80 con 0x7f796c102dc0 2024-02-28T17:20:02.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.899+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c110140 con 0x7f796c102dc0 2024-02-28T17:20:02.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.901+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c05f5c0 con 0x7f796c102dc0 2024-02-28T17:20:02.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.902+0000 7f7968c00700 1 -- 172.21.15.1:0/1986861153 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f795800bd60 con 0x7f796c102dc0 2024-02-28T17:20:02.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.903+0000 7f7968c00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f795406d7f0 0x7f795406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:02.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.903+0000 7f7968c00700 1 -- 172.21.15.1:0/1986861153 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f795808e1c0 con 0x7f796c102dc0 2024-02-28T17:20:02.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.904+0000 7f7970a00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f795406d7f0 0x7f795406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:02.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.904+0000 7f7970a00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f795406d7f0 0x7f795406fca0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7960004990 tx=0x7f7960009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:02.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:02.908+0000 7f7968c00700 1 -- 172.21.15.1:0/1986861153 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f795805b5a0 con 0x7f796c102dc0 2024-02-28T17:20:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.152+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f796c04bed0 con 0x7f795406d7f0 2024-02-28T17:20:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.154+0000 7f7968c00700 1 -- 172.21.15.1:0/1986861153 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f796c04bed0 con 0x7f795406d7f0 2024-02-28T17:20:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f795406d7f0 msgr2=0x7f795406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f795406d7f0 0x7f795406fca0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7960004990 tx=0x7f7960009ee0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c102dc0 msgr2=0x7f796c115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c102dc0 0x7f796c115f80 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7958009fd0 tx=0x7f7958009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 shutdown_connections 2024-02-28T17:20:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f795406d7f0 0x7f795406fca0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f796c0fb390 0x7f796c115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 --2- 172.21.15.1:0/1986861153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f796c102dc0 0x7f796c115f80 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 >> 172.21.15.1:0/1986861153 conn(0x7f796c072270 msgr2=0x7f796c0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.157+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 shutdown_connections 2024-02-28T17:20:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.158+0000 7f7971e00700 1 -- 172.21.15.1:0/1986861153 wait complete. 2024-02-28T17:20:03.179 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:20:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.317+0000 7f3548000700 1 -- 172.21.15.1:0/74971741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 msgr2=0x7f3540103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.317+0000 7f3548000700 1 --2- 172.21.15.1:0/74971741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 0x7f3540103320 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3530004940 tx=0x7f3530009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.317+0000 7f3548000700 1 -- 172.21.15.1:0/74971741 shutdown_connections 2024-02-28T17:20:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.317+0000 7f3548000700 1 --2- 172.21.15.1:0/74971741 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35401038f0 0x7f35400fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.317+0000 7f3548000700 1 --2- 172.21.15.1:0/74971741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 0x7f3540103320 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.317+0000 7f3548000700 1 -- 172.21.15.1:0/74971741 >> 172.21.15.1:0/74971741 conn(0x7f35400f8ae0 msgr2=0x7f35400faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.317+0000 7f3548000700 1 -- 172.21.15.1:0/74971741 shutdown_connections 2024-02-28T17:20:03.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.317+0000 7f3548000700 1 -- 172.21.15.1:0/74971741 wait complete. 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.318+0000 7f3548000700 1 Processor -- start 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.318+0000 7f3548000700 1 -- start start 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.318+0000 7f3548000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 0x7f3540115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.318+0000 7f3548000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35401038f0 0x7f3540116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.318+0000 7f3548000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35401167a0 con 0x7f3540102f50 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.318+0000 7f3548000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354010faf0 con 0x7f35401038f0 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.319+0000 7f3545600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 0x7f3540115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.319+0000 7f3545600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 0x7f3540115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51006/0 (socket says 172.21.15.1:51006) 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.319+0000 7f3545600700 1 -- 172.21.15.1:0/1628269906 learned_addr learned my addr 172.21.15.1:0/1628269906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:03.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.319+0000 7f3544c00700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35401038f0 0x7f3540116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.319+0000 7f3545600700 1 -- 172.21.15.1:0/1628269906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35401038f0 msgr2=0x7f3540116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.319+0000 7f3545600700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35401038f0 0x7f3540116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.319+0000 7f3545600700 1 -- 172.21.15.1:0/1628269906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3530005040 con 0x7f3540102f50 2024-02-28T17:20:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.320+0000 7f3545600700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 0x7f3540115c10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f3530009fd0 tx=0x7f3530009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.320+0000 7f353d600700 1 -- 172.21.15.1:0/1628269906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f353000e040 con 0x7f3540102f50 2024-02-28T17:20:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.320+0000 7f353d600700 1 -- 172.21.15.1:0/1628269906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f353000bbf0 con 0x7f3540102f50 2024-02-28T17:20:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.320+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354010fd70 con 0x7f3540102f50 2024-02-28T17:20:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.320+0000 7f353d600700 1 -- 172.21.15.1:0/1628269906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f353001c3f0 con 0x7f3540102f50 2024-02-28T17:20:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.320+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35401102b0 con 0x7f3540102f50 2024-02-28T17:20:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.322+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35400fc6e0 con 0x7f3540102f50 2024-02-28T17:20:03.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.322+0000 7f353d600700 1 -- 172.21.15.1:0/1628269906 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f353000bd60 con 0x7f3540102f50 2024-02-28T17:20:03.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.323+0000 7f353d600700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f352806d7f0 0x7f352806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:03.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.323+0000 7f353d600700 1 -- 172.21.15.1:0/1628269906 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f353008e210 con 0x7f3540102f50 2024-02-28T17:20:03.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.324+0000 7f3544c00700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f352806d7f0 0x7f352806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:03.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.324+0000 7f3544c00700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f352806d7f0 0x7f352806fca0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f35401114b0 tx=0x7f3534009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:03.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.328+0000 7f353d600700 1 -- 172.21.15.1:0/1628269906 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3530057ae0 con 0x7f3540102f50 2024-02-28T17:20:03.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.571+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3540111290 con 0x7f352806d7f0 2024-02-28T17:20:03.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.574+0000 7f353d600700 1 -- 172.21.15.1:0/1628269906 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3540111290 con 0x7f352806d7f0 2024-02-28T17:20:03.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f352806d7f0 msgr2=0x7f352806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:03.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f352806d7f0 0x7f352806fca0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f35401114b0 tx=0x7f3534009ee0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 msgr2=0x7f3540115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:03.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 0x7f3540115c10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f3530009fd0 tx=0x7f3530009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:03.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 shutdown_connections 2024-02-28T17:20:03.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f352806d7f0 0x7f352806fca0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3540102f50 0x7f3540115c10 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 --2- 172.21.15.1:0/1628269906 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f35401038f0 0x7f3540116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 >> 172.21.15.1:0/1628269906 conn(0x7f35400f8ae0 msgr2=0x7f35400fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:03.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 shutdown_connections 2024-02-28T17:20:03.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.577+0000 7f3548000700 1 -- 172.21.15.1:0/1628269906 wait complete. 2024-02-28T17:20:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.722+0000 7fc750000700 1 -- 172.21.15.1:0/737719959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 msgr2=0x7fc7480fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.722+0000 7fc750000700 1 --2- 172.21.15.1:0/737719959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 0x7fc7480fcf60 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fc734004940 tx=0x7fc734009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.722+0000 7fc750000700 1 -- 172.21.15.1:0/737719959 shutdown_connections 2024-02-28T17:20:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.722+0000 7fc750000700 1 --2- 172.21.15.1:0/737719959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 0x7fc7480fcf60 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.722+0000 7fc750000700 1 --2- 172.21.15.1:0/737719959 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc748102f50 0x7fc748103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.722+0000 7fc750000700 1 -- 172.21.15.1:0/737719959 >> 172.21.15.1:0/737719959 conn(0x7fc7480f8ae0 msgr2=0x7fc7480faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.722+0000 7fc750000700 1 -- 172.21.15.1:0/737719959 shutdown_connections 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.722+0000 7fc750000700 1 -- 172.21.15.1:0/737719959 wait complete. 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.723+0000 7fc750000700 1 Processor -- start 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.723+0000 7fc750000700 1 -- start start 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.723+0000 7fc750000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc748102f50 0x7fc748115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.723+0000 7fc750000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 0x7fc7481161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.723+0000 7fc750000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc748116830 con 0x7fc7481038f0 2024-02-28T17:20:03.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.723+0000 7fc750000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74810fb40 con 0x7fc748102f50 2024-02-28T17:20:03.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.724+0000 7fc74cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 0x7fc7481161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.724+0000 7fc74cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 0x7fc7481161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51032/0 (socket says 172.21.15.1:51032) 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.724+0000 7fc74cc00700 1 -- 172.21.15.1:0/3748096861 learned_addr learned my addr 172.21.15.1:0/3748096861 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.724+0000 7fc74d600700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc748102f50 0x7fc748115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.724+0000 7fc74cc00700 1 -- 172.21.15.1:0/3748096861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc748102f50 msgr2=0x7fc748115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.724+0000 7fc74cc00700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc748102f50 0x7fc748115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.724+0000 7fc74cc00700 1 -- 172.21.15.1:0/3748096861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc734005040 con 0x7fc7481038f0 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.725+0000 7fc74cc00700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 0x7fc7481161e0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc734009030 tx=0x7fc734009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.725+0000 7fc745600700 1 -- 172.21.15.1:0/3748096861 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc73400e040 con 0x7fc7481038f0 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.725+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74810fdc0 con 0x7fc7481038f0 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.725+0000 7fc745600700 1 -- 172.21.15.1:0/3748096861 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc73400bbf0 con 0x7fc7481038f0 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.725+0000 7fc745600700 1 -- 172.21.15.1:0/3748096861 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc73401c630 con 0x7fc7481038f0 2024-02-28T17:20:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.725+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc748110300 con 0x7fc7481038f0 2024-02-28T17:20:03.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.726+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7480fc6e0 con 0x7fc7481038f0 2024-02-28T17:20:03.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.727+0000 7fc745600700 1 -- 172.21.15.1:0/3748096861 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc73400bd60 con 0x7fc7481038f0 2024-02-28T17:20:03.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.728+0000 7fc745600700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc73006d7f0 0x7fc73006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:03.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.729+0000 7fc745600700 1 -- 172.21.15.1:0/3748096861 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc73408e1f0 con 0x7fc7481038f0 2024-02-28T17:20:03.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.729+0000 7fc74d600700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc73006d7f0 0x7fc73006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:03.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.729+0000 7fc74d600700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc73006d7f0 0x7fc73006fca0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc73c004d60 tx=0x7fc73c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:03.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.733+0000 7fc745600700 1 -- 172.21.15.1:0/3748096861 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc73405b5d0 con 0x7fc7481038f0 2024-02-28T17:20:03.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:03.965+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7481112f0 con 0x7fc73006d7f0 2024-02-28T17:20:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.000+0000 7fc745600700 1 -- 172.21.15.1:0/3748096861 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc7481112f0 con 0x7fc73006d7f0 2024-02-28T17:20:04.005 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:20:04.005 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (46m) 7m ago 60m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:20:04.005 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (42m) 7m ago 60m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:20:04.005 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (42m) 7m ago 59m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (45m) 7m ago 60m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (38m) 7m ago 53m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (38m) 7m ago 53m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (7m) 7m ago 53m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (23m) 7m ago 53m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (48m) 7m ago 62m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (47m) 7m ago 59m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (44m) 7m ago 62m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (43m) 7m ago 59m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (47m) 7m ago 60m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (47m) 7m ago 59m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (42m) 7m ago 58m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (41m) 7m ago 57m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (41m) 7m ago 56m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (40m) 7m ago 56m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:20:04.006 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (39m) 7m ago 55m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:20:04.007 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (39m) 7m ago 55m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:20:04.007 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (46m) 7m ago 59m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:20:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc73006d7f0 msgr2=0x7fc73006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc73006d7f0 0x7fc73006fca0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc73c004d60 tx=0x7fc73c00b040 comp rx=0 tx=0).stop 2024-02-28T17:20:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 msgr2=0x7fc7481161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 0x7fc7481161e0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc734009030 tx=0x7fc734009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 shutdown_connections 2024-02-28T17:20:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc73006d7f0 0x7fc73006fca0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc748102f50 0x7fc748115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 --2- 172.21.15.1:0/3748096861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7481038f0 0x7fc7481161e0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 >> 172.21.15.1:0/3748096861 conn(0x7fc7480f8ae0 msgr2=0x7fc7480fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 shutdown_connections 2024-02-28T17:20:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.003+0000 7fc750000700 1 -- 172.21.15.1:0/3748096861 wait complete. 2024-02-28T17:20:04.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.163+0000 7f8e5e600700 1 -- 172.21.15.1:0/1000718013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 msgr2=0x7f8e580fe140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.163+0000 7f8e5e600700 1 --2- 172.21.15.1:0/1000718013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 0x7f8e580fe140 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c004940 tx=0x7f8e4c009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:04.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.163+0000 7f8e5e600700 1 -- 172.21.15.1:0/1000718013 shutdown_connections 2024-02-28T17:20:04.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.163+0000 7f8e5e600700 1 --2- 172.21.15.1:0/1000718013 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e580fe710 0x7f8e58106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.163+0000 7f8e5e600700 1 --2- 172.21.15.1:0/1000718013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 0x7f8e580fe140 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.163+0000 7f8e5e600700 1 -- 172.21.15.1:0/1000718013 >> 172.21.15.1:0/1000718013 conn(0x7f8e580f7820 msgr2=0x7f8e580f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:04.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.163+0000 7f8e5e600700 1 -- 172.21.15.1:0/1000718013 shutdown_connections 2024-02-28T17:20:04.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.163+0000 7f8e5e600700 1 -- 172.21.15.1:0/1000718013 wait complete. 2024-02-28T17:20:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.164+0000 7f8e5e600700 1 Processor -- start 2024-02-28T17:20:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.164+0000 7f8e5e600700 1 -- start start 2024-02-28T17:20:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5e600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 0x7f8e581159e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5e600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e580fe710 0x7f8e58115f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5e600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58116570 con 0x7f8e580fdd70 2024-02-28T17:20:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5e600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5810f8a0 con 0x7f8e580fe710 2024-02-28T17:20:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5d200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 0x7f8e581159e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:04.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5d200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 0x7f8e581159e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51044/0 (socket says 172.21.15.1:51044) 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5d200700 1 -- 172.21.15.1:0/178638926 learned_addr learned my addr 172.21.15.1:0/178638926 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5d200700 1 -- 172.21.15.1:0/178638926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e580fe710 msgr2=0x7f8e58115f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.165+0000 7f8e5d200700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e580fe710 0x7f8e58115f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.166+0000 7f8e5d200700 1 -- 172.21.15.1:0/178638926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c005040 con 0x7f8e580fdd70 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.166+0000 7f8e5d200700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 0x7f8e581159e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009180 tx=0x7f8e4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.166+0000 7f8e54c00700 1 -- 172.21.15.1:0/178638926 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e4c00e040 con 0x7f8e580fdd70 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.166+0000 7f8e54c00700 1 -- 172.21.15.1:0/178638926 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e4c004290 con 0x7f8e580fdd70 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.166+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5810fb20 con 0x7f8e580fdd70 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.166+0000 7f8e54c00700 1 -- 172.21.15.1:0/178638926 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e4c01c630 con 0x7f8e580fdd70 2024-02-28T17:20:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.166+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e58110060 con 0x7f8e580fdd70 2024-02-28T17:20:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.167+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e581067c0 con 0x7f8e580fdd70 2024-02-28T17:20:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.168+0000 7f8e54c00700 1 -- 172.21.15.1:0/178638926 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8e4c0139f0 con 0x7f8e580fdd70 2024-02-28T17:20:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.169+0000 7f8e54c00700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4006d7f0 0x7f8e4006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.169+0000 7f8e54c00700 1 -- 172.21.15.1:0/178638926 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8e4c08e2f0 con 0x7f8e580fdd70 2024-02-28T17:20:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.169+0000 7f8e57e00700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4006d7f0 0x7f8e4006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.170+0000 7f8e57e00700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4006d7f0 0x7f8e4006fca0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8e58111210 tx=0x7f8e44009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.174+0000 7f8e54c00700 1 -- 172.21.15.1:0/178638926 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8e4c05b6d0 con 0x7f8e580fdd70 2024-02-28T17:20:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.486+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8e5804bed0 con 0x7f8e580fdd70 2024-02-28T17:20:04.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.487+0000 7f8e54c00700 1 -- 172.21.15.1:0/178638926 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8e4c05b260 con 0x7f8e580fdd70 2024-02-28T17:20:04.492 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:20:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:20:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:20:04.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:20:04.493 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:20:04.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:20:04.494 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.490+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4006d7f0 msgr2=0x7f8e4006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.490+0000 7f8e5e600700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4006d7f0 0x7f8e4006fca0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8e58111210 tx=0x7f8e44009ce0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.490+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 msgr2=0x7f8e581159e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.490+0000 7f8e5e600700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 0x7f8e581159e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009180 tx=0x7f8e4c009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.491+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 shutdown_connections 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.491+0000 7f8e5e600700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8e4006d7f0 0x7f8e4006fca0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.491+0000 7f8e5e600700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e580fdd70 0x7f8e581159e0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.491+0000 7f8e5e600700 1 --2- 172.21.15.1:0/178638926 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8e580fe710 0x7f8e58115f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.491+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 >> 172.21.15.1:0/178638926 conn(0x7f8e580f7820 msgr2=0x7f8e580fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.491+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 shutdown_connections 2024-02-28T17:20:04.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.491+0000 7f8e5e600700 1 -- 172.21.15.1:0/178638926 wait complete. 2024-02-28T17:20:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.645+0000 7fcc68c00700 1 -- 172.21.15.1:0/2285417846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc64103dc0 msgr2=0x7fcc64104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.645+0000 7fcc68c00700 1 --2- 172.21.15.1:0/2285417846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc64103dc0 0x7fcc64104210 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fcc50004940 tx=0x7fcc50009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.645+0000 7fcc68c00700 1 -- 172.21.15.1:0/2285417846 shutdown_connections 2024-02-28T17:20:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.645+0000 7fcc68c00700 1 --2- 172.21.15.1:0/2285417846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc64103dc0 0x7fcc64104210 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.645+0000 7fcc68c00700 1 --2- 172.21.15.1:0/2285417846 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc641095f0 0x7fcc641099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.645+0000 7fcc68c00700 1 -- 172.21.15.1:0/2285417846 >> 172.21.15.1:0/2285417846 conn(0x7fcc64072d50 msgr2=0x7fcc64075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.645+0000 7fcc68c00700 1 -- 172.21.15.1:0/2285417846 shutdown_connections 2024-02-28T17:20:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.645+0000 7fcc68c00700 1 -- 172.21.15.1:0/2285417846 wait complete. 2024-02-28T17:20:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.646+0000 7fcc68c00700 1 Processor -- start 2024-02-28T17:20:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.646+0000 7fcc68c00700 1 -- start start 2024-02-28T17:20:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.646+0000 7fcc68c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc64103dc0 0x7fcc6411a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.646+0000 7fcc68c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc641095f0 0x7fcc6411a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:04.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.646+0000 7fcc68c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6411abd0 con 0x7fcc641095f0 2024-02-28T17:20:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.646+0000 7fcc68c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64113ee0 con 0x7fcc64103dc0 2024-02-28T17:20:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.647+0000 7fcc61400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc641095f0 0x7fcc6411a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.647+0000 7fcc61400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc641095f0 0x7fcc6411a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51066/0 (socket says 172.21.15.1:51066) 2024-02-28T17:20:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.647+0000 7fcc61400700 1 -- 172.21.15.1:0/1262444598 learned_addr learned my addr 172.21.15.1:0/1262444598 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.647+0000 7fcc61400700 1 -- 172.21.15.1:0/1262444598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc64103dc0 msgr2=0x7fcc6411a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.647+0000 7fcc61400700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc64103dc0 0x7fcc6411a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.647+0000 7fcc61400700 1 -- 172.21.15.1:0/1262444598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc50005040 con 0x7fcc641095f0 2024-02-28T17:20:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.647+0000 7fcc61400700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc641095f0 0x7fcc6411a580 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009150 tx=0x7fcc50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.648+0000 7fcc5a000700 1 -- 172.21.15.1:0/1262444598 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc5000e040 con 0x7fcc641095f0 2024-02-28T17:20:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.648+0000 7fcc5a000700 1 -- 172.21.15.1:0/1262444598 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc5000bbf0 con 0x7fcc641095f0 2024-02-28T17:20:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.648+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc64114160 con 0x7fcc641095f0 2024-02-28T17:20:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.648+0000 7fcc5a000700 1 -- 172.21.15.1:0/1262444598 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc5001c630 con 0x7fcc641095f0 2024-02-28T17:20:04.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.648+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc641146a0 con 0x7fcc641095f0 2024-02-28T17:20:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.649+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6410b9c0 con 0x7fcc641095f0 2024-02-28T17:20:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.650+0000 7fcc5a000700 1 -- 172.21.15.1:0/1262444598 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcc5000bd60 con 0x7fcc641095f0 2024-02-28T17:20:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.651+0000 7fcc5a000700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc4806d7f0 0x7fcc4806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.651+0000 7fcc5a000700 1 -- 172.21.15.1:0/1262444598 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcc5008e290 con 0x7fcc641095f0 2024-02-28T17:20:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.651+0000 7fcc61e00700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc4806d7f0 0x7fcc4806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:04.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.652+0000 7fcc61e00700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc4806d7f0 0x7fcc4806fca0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcc54004d00 tx=0x7fcc54009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:04.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.656+0000 7fcc5a000700 1 -- 172.21.15.1:0/1262444598 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcc5005b670 con 0x7fcc641095f0 2024-02-28T17:20:04.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:04 smithi001 ceph-mon[112940]: from='client.36412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:04.786 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:04 smithi001 ceph-mon[112940]: from='client.36416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:04.786 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:04 smithi001 ceph-mon[112940]: pgmap v1372: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:04.786 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:04 smithi001 ceph-mon[112940]: from='client.36420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:04.786 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:04 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/178638926' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:20:04.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.926+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcc64115620 con 0x7fcc641095f0 2024-02-28T17:20:04.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.927+0000 7fcc5a000700 1 -- 172.21.15.1:0/1262444598 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fcc5005b200 con 0x7fcc641095f0 2024-02-28T17:20:04.932 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:20:04.932 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:20:04.932 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:20:04.932 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:20:04.933 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:20:04.934 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:20:04.935 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:20:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc4806d7f0 msgr2=0x7fcc4806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc4806d7f0 0x7fcc4806fca0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcc54004d00 tx=0x7fcc54009ce0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc641095f0 msgr2=0x7fcc6411a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc641095f0 0x7fcc6411a580 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009150 tx=0x7fcc50009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 shutdown_connections 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc4806d7f0 0x7fcc4806fca0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc64103dc0 0x7fcc6411a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 --2- 172.21.15.1:0/1262444598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc641095f0 0x7fcc6411a580 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 >> 172.21.15.1:0/1262444598 conn(0x7fcc64072d50 msgr2=0x7fcc64074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 shutdown_connections 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:04.930+0000 7fcc68c00700 1 -- 172.21.15.1:0/1262444598 wait complete. 2024-02-28T17:20:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:20:05.014 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:04 smithi066 ceph-mon[85781]: from='client.36412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:05.014 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:04 smithi066 ceph-mon[85781]: from='client.36416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:05.014 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:04 smithi066 ceph-mon[85781]: pgmap v1372: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:05.014 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:04 smithi066 ceph-mon[85781]: from='client.36420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:05.014 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:04 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/178638926' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:20:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.090+0000 7fec36c00700 1 -- 172.21.15.1:0/4167337880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 msgr2=0x7fec30104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.090+0000 7fec36c00700 1 --2- 172.21.15.1:0/4167337880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 0x7fec30104210 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fec1c004940 tx=0x7fec1c009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.090+0000 7fec36c00700 1 -- 172.21.15.1:0/4167337880 shutdown_connections 2024-02-28T17:20:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.090+0000 7fec36c00700 1 --2- 172.21.15.1:0/4167337880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 0x7fec30104210 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.090+0000 7fec36c00700 1 --2- 172.21.15.1:0/4167337880 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec301095f0 0x7fec301099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.090+0000 7fec36c00700 1 -- 172.21.15.1:0/4167337880 >> 172.21.15.1:0/4167337880 conn(0x7fec30072d50 msgr2=0x7fec30075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.090+0000 7fec36c00700 1 -- 172.21.15.1:0/4167337880 shutdown_connections 2024-02-28T17:20:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.090+0000 7fec36c00700 1 -- 172.21.15.1:0/4167337880 wait complete. 2024-02-28T17:20:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.091+0000 7fec36c00700 1 Processor -- start 2024-02-28T17:20:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.091+0000 7fec36c00700 1 -- start start 2024-02-28T17:20:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.092+0000 7fec36c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 0x7fec3011a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.092+0000 7fec36c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec301095f0 0x7fec3011a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.092+0000 7fec36c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3011abd0 con 0x7fec30103dc0 2024-02-28T17:20:05.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.092+0000 7fec36c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec30113ee0 con 0x7fec301095f0 2024-02-28T17:20:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.092+0000 7fec2fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 0x7fec3011a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.092+0000 7fec2fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 0x7fec3011a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51092/0 (socket says 172.21.15.1:51092) 2024-02-28T17:20:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.092+0000 7fec2fe00700 1 -- 172.21.15.1:0/3056473883 learned_addr learned my addr 172.21.15.1:0/3056473883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.093+0000 7fec2f400700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec301095f0 0x7fec3011a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.093+0000 7fec2fe00700 1 -- 172.21.15.1:0/3056473883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec301095f0 msgr2=0x7fec3011a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.093+0000 7fec2fe00700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec301095f0 0x7fec3011a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.093+0000 7fec2fe00700 1 -- 172.21.15.1:0/3056473883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c005040 con 0x7fec30103dc0 2024-02-28T17:20:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.093+0000 7fec2fe00700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 0x7fec3011a040 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fec2000c8c0 tx=0x7fec200044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:05.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.093+0000 7fec27e00700 1 -- 172.21.15.1:0/3056473883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec2000a980 con 0x7fec30103dc0 2024-02-28T17:20:05.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.093+0000 7fec27e00700 1 -- 172.21.15.1:0/3056473883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec2000ce60 con 0x7fec30103dc0 2024-02-28T17:20:05.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.093+0000 7fec27e00700 1 -- 172.21.15.1:0/3056473883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec200115e0 con 0x7fec30103dc0 2024-02-28T17:20:05.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.094+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec301141c0 con 0x7fec30103dc0 2024-02-28T17:20:05.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.094+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec301146b0 con 0x7fec30103dc0 2024-02-28T17:20:05.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.095+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3010b9c0 con 0x7fec30103dc0 2024-02-28T17:20:05.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.096+0000 7fec27e00700 1 -- 172.21.15.1:0/3056473883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fec20012400 con 0x7fec30103dc0 2024-02-28T17:20:05.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.097+0000 7fec27e00700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec1406d7f0 0x7fec1406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:05.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.097+0000 7fec27e00700 1 -- 172.21.15.1:0/3056473883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fec2008cfe0 con 0x7fec30103dc0 2024-02-28T17:20:05.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.097+0000 7fec2f400700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec1406d7f0 0x7fec1406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:05.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.098+0000 7fec2f400700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec1406d7f0 0x7fec1406fca0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fec301158e0 tx=0x7fec1c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:05.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.102+0000 7fec27e00700 1 -- 172.21.15.1:0/3056473883 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec2005a3a0 con 0x7fec30103dc0 2024-02-28T17:20:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.335+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fec3004bf40 con 0x7fec1406d7f0 2024-02-28T17:20:05.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.338+0000 7fec27e00700 1 -- 172.21.15.1:0/3056473883 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fec3004bf40 con 0x7fec1406d7f0 2024-02-28T17:20:05.343 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:20:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:20:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:20:05.343 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:20:05.344 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec1406d7f0 msgr2=0x7fec1406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec1406d7f0 0x7fec1406fca0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fec301158e0 tx=0x7fec1c009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 msgr2=0x7fec3011a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 0x7fec3011a040 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fec2000c8c0 tx=0x7fec200044c0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 shutdown_connections 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec1406d7f0 0x7fec1406fca0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec30103dc0 0x7fec3011a040 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 --2- 172.21.15.1:0/3056473883 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec301095f0 0x7fec3011a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 >> 172.21.15.1:0/3056473883 conn(0x7fec30072d50 msgr2=0x7fec30074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 shutdown_connections 2024-02-28T17:20:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.341+0000 7fec36c00700 1 -- 172.21.15.1:0/3056473883 wait complete. 2024-02-28T17:20:05.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.493+0000 7f305a600700 1 -- 172.21.15.1:0/1524991610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30540fbd40 msgr2=0x7f30540fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:05.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.493+0000 7f305a600700 1 --2- 172.21.15.1:0/1524991610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30540fbd40 0x7f30540fc110 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3044004940 tx=0x7f3044009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:05.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.493+0000 7f305a600700 1 -- 172.21.15.1:0/1524991610 shutdown_connections 2024-02-28T17:20:05.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.493+0000 7f305a600700 1 --2- 172.21.15.1:0/1524991610 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30540fc6e0 0x7f3054100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.493+0000 7f305a600700 1 --2- 172.21.15.1:0/1524991610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30540fbd40 0x7f30540fc110 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.493+0000 7f305a600700 1 -- 172.21.15.1:0/1524991610 >> 172.21.15.1:0/1524991610 conn(0x7f30540f79e0 msgr2=0x7f30540f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:05.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.494+0000 7f305a600700 1 -- 172.21.15.1:0/1524991610 shutdown_connections 2024-02-28T17:20:05.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.494+0000 7f305a600700 1 -- 172.21.15.1:0/1524991610 wait complete. 2024-02-28T17:20:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.494+0000 7f305a600700 1 Processor -- start 2024-02-28T17:20:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.495+0000 7f305a600700 1 -- start start 2024-02-28T17:20:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.495+0000 7f305a600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30540fc6e0 0x7f305411a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.495+0000 7f305a600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305411a820 0x7f3054114180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.495+0000 7f305a600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305411ada0 con 0x7f305411a820 2024-02-28T17:20:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.495+0000 7f305a600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305411af10 con 0x7f30540fc6e0 2024-02-28T17:20:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.495+0000 7f304be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305411a820 0x7f3054114180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:05.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.495+0000 7f304be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305411a820 0x7f3054114180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51120/0 (socket says 172.21.15.1:51120) 2024-02-28T17:20:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.495+0000 7f304be00700 1 -- 172.21.15.1:0/1137168196 learned_addr learned my addr 172.21.15.1:0/1137168196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.496+0000 7f3053400700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30540fc6e0 0x7f305411a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.496+0000 7f304be00700 1 -- 172.21.15.1:0/1137168196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30540fc6e0 msgr2=0x7f305411a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.496+0000 7f304be00700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30540fc6e0 0x7f305411a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.496+0000 7f304be00700 1 -- 172.21.15.1:0/1137168196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3044005040 con 0x7f305411a820 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.496+0000 7f304be00700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305411a820 0x7f3054114180 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f303c002a50 tx=0x7f303c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.496+0000 7f304b400700 1 -- 172.21.15.1:0/1137168196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303c007cc0 con 0x7f305411a820 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.496+0000 7f304b400700 1 -- 172.21.15.1:0/1137168196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f303c00f040 con 0x7f305411a820 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.496+0000 7f304b400700 1 -- 172.21.15.1:0/1137168196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303c009640 con 0x7f305411a820 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.497+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3054114780 con 0x7f305411a820 2024-02-28T17:20:05.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.497+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3054114cc0 con 0x7f305411a820 2024-02-28T17:20:05.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.498+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30540fe420 con 0x7f305411a820 2024-02-28T17:20:05.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.499+0000 7f304b400700 1 -- 172.21.15.1:0/1137168196 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f303c009800 con 0x7f305411a820 2024-02-28T17:20:05.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.500+0000 7f304b400700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f303806d7f0 0x7f303806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:05.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.500+0000 7f304b400700 1 -- 172.21.15.1:0/1137168196 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f303c08ca60 con 0x7f305411a820 2024-02-28T17:20:05.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.500+0000 7f3053400700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f303806d7f0 0x7f303806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:05.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.501+0000 7f3053400700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f303806d7f0 0x7f303806fca0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3044000c00 tx=0x7f3044009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:05.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.505+0000 7f304b400700 1 -- 172.21.15.1:0/1137168196 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f303c059dc0 con 0x7f305411a820 2024-02-28T17:20:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.821+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3054114f90 con 0x7f305411a820 2024-02-28T17:20:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.822+0000 7f304b400700 1 -- 172.21.15.1:0/1137168196 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f303c059950 con 0x7f305411a820 2024-02-28T17:20:05.827 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:20:05.827 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:20:05.827 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:20:05.827 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:20:05.827 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:20:05.827 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:20:05.827 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:20:05.828 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:20:05.829 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:20:05.829 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:20:05.829 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:20:05.829 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:20:05.829 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:20:05.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f303806d7f0 msgr2=0x7f303806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f303806d7f0 0x7f303806fca0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3044000c00 tx=0x7f3044009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305411a820 msgr2=0x7f3054114180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305411a820 0x7f3054114180 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f303c002a50 tx=0x7f303c0046b0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 shutdown_connections 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f303806d7f0 0x7f303806fca0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30540fc6e0 0x7f305411a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 --2- 172.21.15.1:0/1137168196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f305411a820 0x7f3054114180 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 >> 172.21.15.1:0/1137168196 conn(0x7f30540f79e0 msgr2=0x7f305410e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 shutdown_connections 2024-02-28T17:20:05.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:05.825+0000 7f305a600700 1 -- 172.21.15.1:0/1137168196 wait complete. 2024-02-28T17:20:06.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:05 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1262444598' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:20:06.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:05 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1262444598' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:20:07.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:06 smithi066 ceph-mon[85781]: from='client.36432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:07.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:06 smithi066 ceph-mon[85781]: pgmap v1373: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:07.014 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:06 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1137168196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:20:07.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:06 smithi001 ceph-mon[112940]: from='client.36432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:07.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:06 smithi001 ceph-mon[112940]: pgmap v1373: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:07.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:06 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1137168196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:20:09.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:08 smithi066 ceph-mon[85781]: pgmap v1374: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:09.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:08 smithi001 ceph-mon[112940]: pgmap v1374: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:11.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:10 smithi066 ceph-mon[85781]: pgmap v1375: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:11.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:10 smithi001 ceph-mon[112940]: pgmap v1375: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:12.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:20:12.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:20:13.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:12 smithi066 ceph-mon[85781]: pgmap v1376: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:13.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:12 smithi001 ceph-mon[112940]: pgmap v1376: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:15.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:14 smithi066 ceph-mon[85781]: pgmap v1377: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:15.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:14 smithi001 ceph-mon[112940]: pgmap v1377: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:17.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:16 smithi066 ceph-mon[85781]: pgmap v1378: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:17.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:16 smithi001 ceph-mon[112940]: pgmap v1378: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:19.014 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:18 smithi066 ceph-mon[85781]: pgmap v1379: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:19.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:18 smithi001 ceph-mon[112940]: pgmap v1379: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:21.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:20 smithi066 ceph-mon[85781]: pgmap v1380: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:21.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:20 smithi001 ceph-mon[112940]: pgmap v1380: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:23.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:22 smithi066 ceph-mon[85781]: pgmap v1381: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:23.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:22 smithi001 ceph-mon[112940]: pgmap v1381: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:25.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:24 smithi066 ceph-mon[85781]: pgmap v1382: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:25.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:24 smithi001 ceph-mon[112940]: pgmap v1382: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:27.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:26 smithi066 ceph-mon[85781]: pgmap v1383: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:27.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:20:27.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:26 smithi001 ceph-mon[112940]: pgmap v1383: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:27.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:20:29.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:28 smithi066 ceph-mon[85781]: pgmap v1384: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:29.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:28 smithi001 ceph-mon[112940]: pgmap v1384: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:31.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:30 smithi066 ceph-mon[85781]: pgmap v1385: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:31.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:30 smithi001 ceph-mon[112940]: pgmap v1385: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:33.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:32 smithi066 ceph-mon[85781]: pgmap v1386: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:33.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:32 smithi001 ceph-mon[112940]: pgmap v1386: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:35.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:34 smithi066 ceph-mon[85781]: pgmap v1387: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:35.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:34 smithi001 ceph-mon[112940]: pgmap v1387: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:36.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.997+0000 7f03b8600700 1 -- 172.21.15.1:0/669284013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 msgr2=0x7f03b00fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.997+0000 7f03b8600700 1 --2- 172.21.15.1:0/669284013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 0x7f03b00fe320 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f03a4004940 tx=0x7f03a4009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.997+0000 7f03b8600700 1 -- 172.21.15.1:0/669284013 shutdown_connections 2024-02-28T17:20:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.997+0000 7f03b8600700 1 --2- 172.21.15.1:0/669284013 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03b00fe8f0 0x7f03b0106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.997+0000 7f03b8600700 1 --2- 172.21.15.1:0/669284013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 0x7f03b00fe320 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.997+0000 7f03b8600700 1 -- 172.21.15.1:0/669284013 >> 172.21.15.1:0/669284013 conn(0x7f03b00f79e0 msgr2=0x7f03b00f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.997+0000 7f03b8600700 1 -- 172.21.15.1:0/669284013 shutdown_connections 2024-02-28T17:20:36.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.997+0000 7f03b8600700 1 -- 172.21.15.1:0/669284013 wait complete. 2024-02-28T17:20:36.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.998+0000 7f03b8600700 1 Processor -- start 2024-02-28T17:20:36.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.998+0000 7f03b8600700 1 -- start start 2024-02-28T17:20:36.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b8600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 0x7f03b0115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b8600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03b00fe8f0 0x7f03b01161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b8600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0116800 con 0x7f03b00fdf50 2024-02-28T17:20:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b8600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b010fb30 con 0x7f03b00fe8f0 2024-02-28T17:20:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b5c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 0x7f03b0115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:36.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b5c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 0x7f03b0115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40306/0 (socket says 172.21.15.1:40306) 2024-02-28T17:20:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b5c00700 1 -- 172.21.15.1:0/4184951655 learned_addr learned my addr 172.21.15.1:0/4184951655 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b5c00700 1 -- 172.21.15.1:0/4184951655 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03b00fe8f0 msgr2=0x7f03b01161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b5c00700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03b00fe8f0 0x7f03b01161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:35.999+0000 7f03b5c00700 1 -- 172.21.15.1:0/4184951655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a4005040 con 0x7f03b00fdf50 2024-02-28T17:20:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.000+0000 7f03b5c00700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 0x7f03b0115c70 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009180 tx=0x7f03a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.000+0000 7f03ad600700 1 -- 172.21.15.1:0/4184951655 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a400e040 con 0x7f03b00fdf50 2024-02-28T17:20:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.000+0000 7f03ad600700 1 -- 172.21.15.1:0/4184951655 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03a4004290 con 0x7f03b00fdf50 2024-02-28T17:20:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.000+0000 7f03ad600700 1 -- 172.21.15.1:0/4184951655 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a401c630 con 0x7f03b00fdf50 2024-02-28T17:20:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.000+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b010fdb0 con 0x7f03b00fdf50 2024-02-28T17:20:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.000+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b01102f0 con 0x7f03b00fdf50 2024-02-28T17:20:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.001+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b01069a0 con 0x7f03b00fdf50 2024-02-28T17:20:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.002+0000 7f03ad600700 1 -- 172.21.15.1:0/4184951655 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f03a40139f0 con 0x7f03b00fdf50 2024-02-28T17:20:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.003+0000 7f03ad600700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f039806d7f0 0x7f039806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.003+0000 7f03ad600700 1 -- 172.21.15.1:0/4184951655 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f03a408e2f0 con 0x7f03b00fdf50 2024-02-28T17:20:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.004+0000 7f03b5200700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f039806d7f0 0x7f039806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.004+0000 7f03b5200700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f039806d7f0 0x7f039806fca0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f03b01114a0 tx=0x7f039c007cc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:36.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.008+0000 7f03ad600700 1 -- 172.21.15.1:0/4184951655 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f03a405b6d0 con 0x7f03b00fdf50 2024-02-28T17:20:36.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.252+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03b004bed0 con 0x7f039806d7f0 2024-02-28T17:20:36.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.254+0000 7f03ad600700 1 -- 172.21.15.1:0/4184951655 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f03b004bed0 con 0x7f039806d7f0 2024-02-28T17:20:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f039806d7f0 msgr2=0x7f039806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f039806d7f0 0x7f039806fca0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f03b01114a0 tx=0x7f039c007cc0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 msgr2=0x7f03b0115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 0x7f03b0115c70 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009180 tx=0x7f03a4009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:36.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 shutdown_connections 2024-02-28T17:20:36.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f039806d7f0 0x7f039806fca0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b00fdf50 0x7f03b0115c70 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 --2- 172.21.15.1:0/4184951655 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03b00fe8f0 0x7f03b01161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 >> 172.21.15.1:0/4184951655 conn(0x7f03b00f79e0 msgr2=0x7f03b00fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:36.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 shutdown_connections 2024-02-28T17:20:36.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.257+0000 7f03b8600700 1 -- 172.21.15.1:0/4184951655 wait complete. 2024-02-28T17:20:36.277 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:20:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.414+0000 7f341e800700 1 -- 172.21.15.1:0/3607582817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105170 msgr2=0x7f3418105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.414+0000 7f341e800700 1 --2- 172.21.15.1:0/3607582817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105170 0x7f3418105540 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3408004940 tx=0x7f3408009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.414+0000 7f341e800700 1 -- 172.21.15.1:0/3607582817 shutdown_connections 2024-02-28T17:20:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.414+0000 7f341e800700 1 --2- 172.21.15.1:0/3607582817 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3418105b10 0x7f34180ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.414+0000 7f341e800700 1 --2- 172.21.15.1:0/3607582817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105170 0x7f3418105540 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.414+0000 7f341e800700 1 -- 172.21.15.1:0/3607582817 >> 172.21.15.1:0/3607582817 conn(0x7f34180face0 msgr2=0x7f34180fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:36.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.414+0000 7f341e800700 1 -- 172.21.15.1:0/3607582817 shutdown_connections 2024-02-28T17:20:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.414+0000 7f341e800700 1 -- 172.21.15.1:0/3607582817 wait complete. 2024-02-28T17:20:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.415+0000 7f341e800700 1 Processor -- start 2024-02-28T17:20:36.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.415+0000 7f341e800700 1 -- start start 2024-02-28T17:20:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.415+0000 7f341e800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3418105170 0x7f3418115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.415+0000 7f341e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105b10 0x7f34181161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.415+0000 7f341e800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34181167f0 con 0x7f3418105b10 2024-02-28T17:20:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.415+0000 7f341e800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341810fb40 con 0x7f3418105170 2024-02-28T17:20:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.416+0000 7f3416a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105b10 0x7f34181161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.416+0000 7f3416a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105b10 0x7f34181161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40322/0 (socket says 172.21.15.1:40322) 2024-02-28T17:20:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.416+0000 7f3416a00700 1 -- 172.21.15.1:0/3259310293 learned_addr learned my addr 172.21.15.1:0/3259310293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.416+0000 7f3417400700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3418105170 0x7f3418115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.416+0000 7f3416a00700 1 -- 172.21.15.1:0/3259310293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3418105170 msgr2=0x7f3418115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.416+0000 7f3416a00700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3418105170 0x7f3418115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.416+0000 7f3416a00700 1 -- 172.21.15.1:0/3259310293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3408005040 con 0x7f3418105b10 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.417+0000 7f3416a00700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105b10 0x7f34181161a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f340400c800 tx=0x7f3404004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.417+0000 7f340f400700 1 -- 172.21.15.1:0/3259310293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340400a760 con 0x7f3418105b10 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.417+0000 7f340f400700 1 -- 172.21.15.1:0/3259310293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f340400ad90 con 0x7f3418105b10 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.417+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341810fe20 con 0x7f3418105b10 2024-02-28T17:20:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.417+0000 7f340f400700 1 -- 172.21.15.1:0/3259310293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34040113a0 con 0x7f3418105b10 2024-02-28T17:20:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.417+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3418110360 con 0x7f3418105b10 2024-02-28T17:20:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.418+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3418102480 con 0x7f3418105b10 2024-02-28T17:20:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.419+0000 7f340f400700 1 -- 172.21.15.1:0/3259310293 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3404012400 con 0x7f3418105b10 2024-02-28T17:20:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.420+0000 7f340f400700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f33fc06d7f0 0x7f33fc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.420+0000 7f340f400700 1 -- 172.21.15.1:0/3259310293 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f340408cf60 con 0x7f3418105b10 2024-02-28T17:20:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.421+0000 7f3417400700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f33fc06d7f0 0x7f33fc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.421+0000 7f3417400700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f33fc06d7f0 0x7f33fc06fca0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f340800b020 tx=0x7f3408009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:36.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.425+0000 7f340f400700 1 -- 172.21.15.1:0/3259310293 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3404056790 con 0x7f3418105b10 2024-02-28T17:20:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.661+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f341804bf40 con 0x7f33fc06d7f0 2024-02-28T17:20:36.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.663+0000 7f340f400700 1 -- 172.21.15.1:0/3259310293 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f341804bf40 con 0x7f33fc06d7f0 2024-02-28T17:20:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f33fc06d7f0 msgr2=0x7f33fc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f33fc06d7f0 0x7f33fc06fca0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f340800b020 tx=0x7f3408009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105b10 msgr2=0x7f34181161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105b10 0x7f34181161a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f340400c800 tx=0x7f3404004470 comp rx=0 tx=0).stop 2024-02-28T17:20:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 shutdown_connections 2024-02-28T17:20:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f33fc06d7f0 0x7f33fc06fca0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3418105170 0x7f3418115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 --2- 172.21.15.1:0/3259310293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3418105b10 0x7f34181161a0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 >> 172.21.15.1:0/3259310293 conn(0x7f34180face0 msgr2=0x7f34180fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.665+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 shutdown_connections 2024-02-28T17:20:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.666+0000 7f341e800700 1 -- 172.21.15.1:0/3259310293 wait complete. 2024-02-28T17:20:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.822+0000 7fd01b600700 1 -- 172.21.15.1:0/1808977404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fdfe0 msgr2=0x7fd0140fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.822+0000 7fd01b600700 1 --2- 172.21.15.1:0/1808977404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fdfe0 0x7fd0140fe3b0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd008004970 tx=0x7fd008009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.822+0000 7fd01b600700 1 -- 172.21.15.1:0/1808977404 shutdown_connections 2024-02-28T17:20:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.822+0000 7fd01b600700 1 --2- 172.21.15.1:0/1808977404 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0140fe8f0 0x7fd014107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.822+0000 7fd01b600700 1 --2- 172.21.15.1:0/1808977404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fdfe0 0x7fd0140fe3b0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.822+0000 7fd01b600700 1 -- 172.21.15.1:0/1808977404 >> 172.21.15.1:0/1808977404 conn(0x7fd014071d90 msgr2=0x7fd014072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.823+0000 7fd01b600700 1 -- 172.21.15.1:0/1808977404 shutdown_connections 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.823+0000 7fd01b600700 1 -- 172.21.15.1:0/1808977404 wait complete. 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.823+0000 7fd01b600700 1 Processor -- start 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.824+0000 7fd01b600700 1 -- start start 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.824+0000 7fd01b600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0140fdfe0 0x7fd014115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.824+0000 7fd01b600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fe8f0 0x7fd0141161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.824+0000 7fd01b600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd014116800 con 0x7fd0140fe8f0 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.824+0000 7fd01b600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01410fb30 con 0x7fd0140fdfe0 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.824+0000 7fd013e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fe8f0 0x7fd0141161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.824+0000 7fd013e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fe8f0 0x7fd0141161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40336/0 (socket says 172.21.15.1:40336) 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.824+0000 7fd013e00700 1 -- 172.21.15.1:0/1319586354 learned_addr learned my addr 172.21.15.1:0/1319586354 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.825+0000 7fd018c00700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0140fdfe0 0x7fd014115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.825+0000 7fd013e00700 1 -- 172.21.15.1:0/1319586354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0140fdfe0 msgr2=0x7fd014115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.825+0000 7fd013e00700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0140fdfe0 0x7fd014115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.825+0000 7fd013e00700 1 -- 172.21.15.1:0/1319586354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd008005040 con 0x7fd0140fe8f0 2024-02-28T17:20:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.825+0000 7fd013e00700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fe8f0 0x7fd0141161b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd000002a50 tx=0x7fd0000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.825+0000 7fd010c00700 1 -- 172.21.15.1:0/1319586354 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd000007cc0 con 0x7fd0140fe8f0 2024-02-28T17:20:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.825+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01410fe10 con 0x7fd0140fe8f0 2024-02-28T17:20:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.826+0000 7fd010c00700 1 -- 172.21.15.1:0/1319586354 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd00000f040 con 0x7fd0140fe8f0 2024-02-28T17:20:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.826+0000 7fd010c00700 1 -- 172.21.15.1:0/1319586354 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd00000db70 con 0x7fd0140fe8f0 2024-02-28T17:20:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.826+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd014110350 con 0x7fd0140fe8f0 2024-02-28T17:20:36.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.827+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd014106ac0 con 0x7fd0140fe8f0 2024-02-28T17:20:36.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.828+0000 7fd010c00700 1 -- 172.21.15.1:0/1319586354 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd00000dd10 con 0x7fd0140fe8f0 2024-02-28T17:20:36.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.829+0000 7fd010c00700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcffc06d7e0 0x7fcffc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:36.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.829+0000 7fd010c00700 1 -- 172.21.15.1:0/1319586354 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd00008c670 con 0x7fd0140fe8f0 2024-02-28T17:20:36.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.829+0000 7fd018c00700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcffc06d7e0 0x7fcffc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:36.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.830+0000 7fd018c00700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcffc06d7e0 0x7fcffc06fc90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd008000c00 tx=0x7fd008009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:36.834+0000 7fd010c00700 1 -- 172.21.15.1:0/1319586354 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd000059950 con 0x7fd0140fe8f0 2024-02-28T17:20:37.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:36 smithi066 ceph-mon[85781]: pgmap v1388: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:37.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:36 smithi001 ceph-mon[112940]: pgmap v1388: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:37.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.068+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd01404bed0 con 0x7fcffc06d7e0 2024-02-28T17:20:37.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.097+0000 7fd010c00700 1 -- 172.21.15.1:0/1319586354 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd01404bed0 con 0x7fcffc06d7e0 2024-02-28T17:20:37.102 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:20:37.102 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (46m) 8m ago 60m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:20:37.102 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (43m) 8m ago 61m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:20:37.102 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (43m) 8m ago 59m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:20:37.102 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (46m) 8m ago 60m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:20:37.102 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (39m) 8m ago 53m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:20:37.102 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (38m) 8m ago 53m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (8m) 8m ago 53m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (23m) 8m ago 53m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (49m) 8m ago 62m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (48m) 8m ago 59m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (45m) 8m ago 62m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (44m) 8m ago 59m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (47m) 8m ago 61m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (47m) 8m ago 59m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (42m) 8m ago 58m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (42m) 8m ago 58m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (41m) 8m ago 57m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (41m) 8m ago 56m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (40m) 8m ago 56m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:20:37.103 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (39m) 8m ago 55m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:20:37.104 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (47m) 8m ago 60m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:20:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.100+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcffc06d7e0 msgr2=0x7fcffc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.100+0000 7fd01b600700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcffc06d7e0 0x7fcffc06fc90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd008000c00 tx=0x7fd008009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.100+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fe8f0 msgr2=0x7fd0141161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:37.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.100+0000 7fd01b600700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fe8f0 0x7fd0141161b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd000002a50 tx=0x7fd0000046b0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.101+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 shutdown_connections 2024-02-28T17:20:37.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.101+0000 7fd01b600700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcffc06d7e0 0x7fcffc06fc90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.101+0000 7fd01b600700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd0140fdfe0 0x7fd014115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.101+0000 7fd01b600700 1 --2- 172.21.15.1:0/1319586354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0140fe8f0 0x7fd0141161b0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.101+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 >> 172.21.15.1:0/1319586354 conn(0x7fd014071d90 msgr2=0x7fd014103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:37.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.101+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 shutdown_connections 2024-02-28T17:20:37.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.101+0000 7fd01b600700 1 -- 172.21.15.1:0/1319586354 wait complete. 2024-02-28T17:20:37.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.252+0000 7fb69b000700 1 -- 172.21.15.1:0/2142308537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb694102f50 msgr2=0x7fb694103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:37.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.252+0000 7fb69b000700 1 --2- 172.21.15.1:0/2142308537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb694102f50 0x7fb694103320 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fb684004940 tx=0x7fb684009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:37.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.252+0000 7fb69b000700 1 -- 172.21.15.1:0/2142308537 shutdown_connections 2024-02-28T17:20:37.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.252+0000 7fb69b000700 1 --2- 172.21.15.1:0/2142308537 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb6941038f0 0x7fb6940fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.252+0000 7fb69b000700 1 --2- 172.21.15.1:0/2142308537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb694102f50 0x7fb694103320 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.252+0000 7fb69b000700 1 -- 172.21.15.1:0/2142308537 >> 172.21.15.1:0/2142308537 conn(0x7fb6940f8ae0 msgr2=0x7fb6940faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:37.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.252+0000 7fb69b000700 1 -- 172.21.15.1:0/2142308537 shutdown_connections 2024-02-28T17:20:37.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.252+0000 7fb69b000700 1 -- 172.21.15.1:0/2142308537 wait complete. 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.253+0000 7fb69b000700 1 Processor -- start 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.253+0000 7fb69b000700 1 -- start start 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb69b000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb694102f50 0x7fb694115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb69b000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6941038f0 0x7fb6941161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb69b000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb694116810 con 0x7fb6941038f0 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb69b000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69410fb40 con 0x7fb694102f50 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb68be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6941038f0 0x7fb6941161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb68be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6941038f0 0x7fb6941161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40344/0 (socket says 172.21.15.1:40344) 2024-02-28T17:20:37.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb68be00700 1 -- 172.21.15.1:0/2306911321 learned_addr learned my addr 172.21.15.1:0/2306911321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb693e00700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb694102f50 0x7fb694115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb68be00700 1 -- 172.21.15.1:0/2306911321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb694102f50 msgr2=0x7fb694115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb68be00700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb694102f50 0x7fb694115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.254+0000 7fb68be00700 1 -- 172.21.15.1:0/2306911321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb684005040 con 0x7fb6941038f0 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.255+0000 7fb68be00700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6941038f0 0x7fb6941161c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fb67c002a50 tx=0x7fb67c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.255+0000 7fb690c00700 1 -- 172.21.15.1:0/2306911321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb67c007cc0 con 0x7fb6941038f0 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.255+0000 7fb690c00700 1 -- 172.21.15.1:0/2306911321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb67c00f040 con 0x7fb6941038f0 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.255+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69410fe20 con 0x7fb6941038f0 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.255+0000 7fb690c00700 1 -- 172.21.15.1:0/2306911321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb67c00da40 con 0x7fb6941038f0 2024-02-28T17:20:37.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.255+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb694110360 con 0x7fb6941038f0 2024-02-28T17:20:37.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.256+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6940fc6e0 con 0x7fb6941038f0 2024-02-28T17:20:37.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.257+0000 7fb690c00700 1 -- 172.21.15.1:0/2306911321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb67c0097c0 con 0x7fb6941038f0 2024-02-28T17:20:37.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.258+0000 7fb690c00700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb67806d7f0 0x7fb67806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:37.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.258+0000 7fb690c00700 1 -- 172.21.15.1:0/2306911321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb67c08cd50 con 0x7fb6941038f0 2024-02-28T17:20:37.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.259+0000 7fb693e00700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb67806d7f0 0x7fb67806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:37.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.259+0000 7fb693e00700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb67806d7f0 0x7fb67806fca0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb68400b3b0 tx=0x7fb684009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:37.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.263+0000 7fb690c00700 1 -- 172.21.15.1:0/2306911321 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb67c059da0 con 0x7fb6941038f0 2024-02-28T17:20:37.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.574+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb6941111f0 con 0x7fb6941038f0 2024-02-28T17:20:37.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.575+0000 7fb690c00700 1 -- 172.21.15.1:0/2306911321 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fb67c059bc0 con 0x7fb6941038f0 2024-02-28T17:20:37.580 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:20:37.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:20:37.580 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:20:37.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:20:37.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:20:37.581 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:20:37.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:20:37.581 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:20:37.581 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:20:37.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:20:37.581 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:20:37.582 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:20:37.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:20:37.582 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:20:37.582 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:20:37.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:20:37.582 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:20:37.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb67806d7f0 msgr2=0x7fb67806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:37.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb67806d7f0 0x7fb67806fca0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb68400b3b0 tx=0x7fb684009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:37.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6941038f0 msgr2=0x7fb6941161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:37.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6941038f0 0x7fb6941161c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fb67c002a50 tx=0x7fb67c004600 comp rx=0 tx=0).stop 2024-02-28T17:20:37.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 shutdown_connections 2024-02-28T17:20:37.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb67806d7f0 0x7fb67806fca0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb694102f50 0x7fb694115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 --2- 172.21.15.1:0/2306911321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6941038f0 0x7fb6941161c0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 >> 172.21.15.1:0/2306911321 conn(0x7fb6940f8ae0 msgr2=0x7fb6940fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:37.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 shutdown_connections 2024-02-28T17:20:37.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.579+0000 7fb69b000700 1 -- 172.21.15.1:0/2306911321 wait complete. 2024-02-28T17:20:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.727+0000 7f03cf400700 1 -- 172.21.15.1:0/822479672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fdf50 msgr2=0x7f03c80fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.727+0000 7f03cf400700 1 --2- 172.21.15.1:0/822479672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fdf50 0x7f03c80fe320 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f03bc004940 tx=0x7f03bc009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.727+0000 7f03cf400700 1 -- 172.21.15.1:0/822479672 shutdown_connections 2024-02-28T17:20:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.727+0000 7f03cf400700 1 --2- 172.21.15.1:0/822479672 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c80fe8f0 0x7f03c8106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.727+0000 7f03cf400700 1 --2- 172.21.15.1:0/822479672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fdf50 0x7f03c80fe320 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.727+0000 7f03cf400700 1 -- 172.21.15.1:0/822479672 >> 172.21.15.1:0/822479672 conn(0x7f03c80f79e0 msgr2=0x7f03c80f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.728+0000 7f03cf400700 1 -- 172.21.15.1:0/822479672 shutdown_connections 2024-02-28T17:20:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.728+0000 7f03cf400700 1 -- 172.21.15.1:0/822479672 wait complete. 2024-02-28T17:20:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03cf400700 1 Processor -- start 2024-02-28T17:20:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03cf400700 1 -- start start 2024-02-28T17:20:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03cf400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c80fdf50 0x7f03c8115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03cf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fe8f0 0x7f03c8116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03cf400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c81167b0 con 0x7f03c80fe8f0 2024-02-28T17:20:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03cf400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c810fb00 con 0x7f03c80fdf50 2024-02-28T17:20:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03c7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fe8f0 0x7f03c8116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03c7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fe8f0 0x7f03c8116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40364/0 (socket says 172.21.15.1:40364) 2024-02-28T17:20:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.729+0000 7f03c7e00700 1 -- 172.21.15.1:0/3875821290 learned_addr learned my addr 172.21.15.1:0/3875821290 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03cca00700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c80fdf50 0x7f03c8115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03c7e00700 1 -- 172.21.15.1:0/3875821290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c80fdf50 msgr2=0x7f03c8115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03c7e00700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c80fdf50 0x7f03c8115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03c7e00700 1 -- 172.21.15.1:0/3875821290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc005040 con 0x7f03c80fe8f0 2024-02-28T17:20:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03cca00700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c80fdf50 0x7f03c8115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:20:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03c7e00700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fe8f0 0x7f03c8116160 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f03b4002a50 tx=0x7f03b40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03c4c00700 1 -- 172.21.15.1:0/3875821290 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03b4007cc0 con 0x7f03c80fe8f0 2024-02-28T17:20:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03c4c00700 1 -- 172.21.15.1:0/3875821290 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03b400f040 con 0x7f03c80fe8f0 2024-02-28T17:20:37.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03c4c00700 1 -- 172.21.15.1:0/3875821290 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03b4009640 con 0x7f03c80fe8f0 2024-02-28T17:20:37.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.730+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c810fde0 con 0x7f03c80fe8f0 2024-02-28T17:20:37.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.731+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c8110320 con 0x7f03c80fe8f0 2024-02-28T17:20:37.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.732+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c8106940 con 0x7f03c80fe8f0 2024-02-28T17:20:37.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.733+0000 7f03c4c00700 1 -- 172.21.15.1:0/3875821290 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f03b40097a0 con 0x7f03c80fe8f0 2024-02-28T17:20:37.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.734+0000 7f03c4c00700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03b006d7f0 0x7f03b006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:37.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.734+0000 7f03c4c00700 1 -- 172.21.15.1:0/3875821290 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f03b408c9c0 con 0x7f03c80fe8f0 2024-02-28T17:20:37.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.734+0000 7f03cca00700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03b006d7f0 0x7f03b006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:37.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.735+0000 7f03cca00700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03b006d7f0 0x7f03b006fca0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f03bc004f90 tx=0x7f03bc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:37.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:37.738+0000 7f03c4c00700 1 -- 172.21.15.1:0/3875821290 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f03b4056290 con 0x7f03c80fe8f0 2024-02-28T17:20:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.008+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f03c8064b00 con 0x7f03c80fe8f0 2024-02-28T17:20:38.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:37 smithi001 ceph-mon[112940]: from='client.36440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:38.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:37 smithi001 ceph-mon[112940]: from='client.36444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:38.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:37 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2306911321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:20:38.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:37 smithi066 ceph-mon[85781]: from='client.36440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:38.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:37 smithi066 ceph-mon[85781]: from='client.36444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:38.014 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:37 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2306911321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:20:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.009+0000 7f03c4c00700 1 -- 172.21.15.1:0/3875821290 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f03b4059950 con 0x7f03c80fe8f0 2024-02-28T17:20:38.014 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:20:38.014 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:20:38.014 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:20:38.014 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:20:38.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:20:38.015 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:20:38.016 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:20:38.017 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:20:38.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03b006d7f0 msgr2=0x7f03b006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03b006d7f0 0x7f03b006fca0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f03bc004f90 tx=0x7f03bc009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:38.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fe8f0 msgr2=0x7f03c8116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fe8f0 0x7f03c8116160 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f03b4002a50 tx=0x7f03b40046b0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 shutdown_connections 2024-02-28T17:20:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f03b006d7f0 0x7f03b006fca0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f03c80fdf50 0x7f03c8115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 --2- 172.21.15.1:0/3875821290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c80fe8f0 0x7f03c8116160 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 >> 172.21.15.1:0/3875821290 conn(0x7f03c80f79e0 msgr2=0x7f03c8103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 shutdown_connections 2024-02-28T17:20:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.012+0000 7f03cf400700 1 -- 172.21.15.1:0/3875821290 wait complete. 2024-02-28T17:20:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:20:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.165+0000 7f4a2be00700 1 -- 172.21.15.1:0/1949529742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 msgr2=0x7f4a241099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.165+0000 7f4a2be00700 1 --2- 172.21.15.1:0/1949529742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 0x7f4a241099c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f4a18004940 tx=0x7f4a18009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.165+0000 7f4a2be00700 1 -- 172.21.15.1:0/1949529742 shutdown_connections 2024-02-28T17:20:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.165+0000 7f4a2be00700 1 --2- 172.21.15.1:0/1949529742 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a24103dc0 0x7f4a24104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.165+0000 7f4a2be00700 1 --2- 172.21.15.1:0/1949529742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 0x7f4a241099c0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.165+0000 7f4a2be00700 1 -- 172.21.15.1:0/1949529742 >> 172.21.15.1:0/1949529742 conn(0x7f4a24072d50 msgr2=0x7f4a24075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.165+0000 7f4a2be00700 1 -- 172.21.15.1:0/1949529742 shutdown_connections 2024-02-28T17:20:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.165+0000 7f4a2be00700 1 -- 172.21.15.1:0/1949529742 wait complete. 2024-02-28T17:20:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.166+0000 7f4a2be00700 1 Processor -- start 2024-02-28T17:20:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.166+0000 7f4a2be00700 1 -- start start 2024-02-28T17:20:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.166+0000 7f4a2be00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a24103dc0 0x7f4a2411a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.166+0000 7f4a2be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 0x7f4a2411a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.166+0000 7f4a2be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2411ac70 con 0x7f4a241095f0 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.166+0000 7f4a2be00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a24113f80 con 0x7f4a24103dc0 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.167+0000 7f4a28a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 0x7f4a2411a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.167+0000 7f4a28a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 0x7f4a2411a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40392/0 (socket says 172.21.15.1:40392) 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.167+0000 7f4a28a00700 1 -- 172.21.15.1:0/3314582489 learned_addr learned my addr 172.21.15.1:0/3314582489 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.167+0000 7f4a29400700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a24103dc0 0x7f4a2411a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.167+0000 7f4a28a00700 1 -- 172.21.15.1:0/3314582489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a24103dc0 msgr2=0x7f4a2411a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.167+0000 7f4a28a00700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a24103dc0 0x7f4a2411a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.167+0000 7f4a28a00700 1 -- 172.21.15.1:0/3314582489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a18005040 con 0x7f4a241095f0 2024-02-28T17:20:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.168+0000 7f4a28a00700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 0x7f4a2411a620 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f4a10002a50 tx=0x7f4a100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.168+0000 7f4a21600700 1 -- 172.21.15.1:0/3314582489 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a10007cc0 con 0x7f4a241095f0 2024-02-28T17:20:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.168+0000 7f4a21600700 1 -- 172.21.15.1:0/3314582489 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a1000f040 con 0x7f4a241095f0 2024-02-28T17:20:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.168+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a24114260 con 0x7f4a241095f0 2024-02-28T17:20:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.168+0000 7f4a21600700 1 -- 172.21.15.1:0/3314582489 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a10009640 con 0x7f4a241095f0 2024-02-28T17:20:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.168+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a241147a0 con 0x7f4a241095f0 2024-02-28T17:20:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.169+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a24106850 con 0x7f4a241095f0 2024-02-28T17:20:38.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.170+0000 7f4a21600700 1 -- 172.21.15.1:0/3314582489 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f4a100097a0 con 0x7f4a241095f0 2024-02-28T17:20:38.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.171+0000 7f4a21600700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:38.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.171+0000 7f4a21600700 1 -- 172.21.15.1:0/3314582489 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4a1008cd20 con 0x7f4a241095f0 2024-02-28T17:20:38.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.172+0000 7f4a29400700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:38.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.172+0000 7f4a29400700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800b3b0 tx=0x7f4a18009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:38.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.176+0000 7f4a21600700 1 -- 172.21.15.1:0/3314582489 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a1005a000 con 0x7f4a241095f0 2024-02-28T17:20:38.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.408+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a241156c0 con 0x7f4a0c06d7f0 2024-02-28T17:20:38.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.411+0000 7f4a21600700 1 -- 172.21.15.1:0/3314582489 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4a241156c0 con 0x7f4a0c06d7f0 2024-02-28T17:20:38.415 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:20:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:20:38.417 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:20:38.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 msgr2=0x7f4a0c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800b3b0 tx=0x7f4a18009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:38.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 msgr2=0x7f4a2411a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 0x7f4a2411a620 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f4a10002a50 tx=0x7f4a100046b0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 shutdown_connections 2024-02-28T17:20:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f4a0c06d7f0 0x7f4a0c06fca0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f4a24103dc0 0x7f4a2411a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 --2- 172.21.15.1:0/3314582489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a241095f0 0x7f4a2411a620 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 >> 172.21.15.1:0/3314582489 conn(0x7f4a24072d50 msgr2=0x7f4a24074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 shutdown_connections 2024-02-28T17:20:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.414+0000 7f4a2be00700 1 -- 172.21.15.1:0/3314582489 wait complete. 2024-02-28T17:20:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.566+0000 7fc360e00700 1 -- 172.21.15.1:0/2854631286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 msgr2=0x7fc35c0fef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.566+0000 7fc360e00700 1 --2- 172.21.15.1:0/2854631286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 0x7fc35c0fef80 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fc348004940 tx=0x7fc348009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:38.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.566+0000 7fc360e00700 1 -- 172.21.15.1:0/2854631286 shutdown_connections 2024-02-28T17:20:38.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.566+0000 7fc360e00700 1 --2- 172.21.15.1:0/2854631286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 0x7fc35c0fef80 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.566+0000 7fc360e00700 1 --2- 172.21.15.1:0/2854631286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc35c104f70 0x7fc35c105340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.566+0000 7fc360e00700 1 -- 172.21.15.1:0/2854631286 >> 172.21.15.1:0/2854631286 conn(0x7fc35c0faae0 msgr2=0x7fc35c0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:38.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.566+0000 7fc360e00700 1 -- 172.21.15.1:0/2854631286 shutdown_connections 2024-02-28T17:20:38.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.566+0000 7fc360e00700 1 -- 172.21.15.1:0/2854631286 wait complete. 2024-02-28T17:20:38.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.567+0000 7fc360e00700 1 Processor -- start 2024-02-28T17:20:38.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.567+0000 7fc360e00700 1 -- start start 2024-02-28T17:20:38.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.568+0000 7fc360e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc35c104f70 0x7fc35c115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.568+0000 7fc360e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 0x7fc35c115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.568+0000 7fc360e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1165a0 con 0x7fc35c105910 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.568+0000 7fc360e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c10f8d0 con 0x7fc35c104f70 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.568+0000 7fc35aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 0x7fc35c115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.568+0000 7fc35aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 0x7fc35c115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40406/0 (socket says 172.21.15.1:40406) 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.568+0000 7fc35aa00700 1 -- 172.21.15.1:0/680354592 learned_addr learned my addr 172.21.15.1:0/680354592 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.568+0000 7fc35b400700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc35c104f70 0x7fc35c115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.569+0000 7fc35aa00700 1 -- 172.21.15.1:0/680354592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc35c104f70 msgr2=0x7fc35c115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.569+0000 7fc35aa00700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc35c104f70 0x7fc35c115a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.569+0000 7fc35aa00700 1 -- 172.21.15.1:0/680354592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc348005040 con 0x7fc35c105910 2024-02-28T17:20:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.569+0000 7fc35aa00700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 0x7fc35c115f50 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fc348009030 tx=0x7fc348009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.569+0000 7fc353400700 1 -- 172.21.15.1:0/680354592 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc34800e040 con 0x7fc35c105910 2024-02-28T17:20:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.569+0000 7fc353400700 1 -- 172.21.15.1:0/680354592 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc34800bbf0 con 0x7fc35c105910 2024-02-28T17:20:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.569+0000 7fc353400700 1 -- 172.21.15.1:0/680354592 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc34801c630 con 0x7fc35c105910 2024-02-28T17:20:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.569+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c10fb50 con 0x7fc35c105910 2024-02-28T17:20:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.570+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c110110 con 0x7fc35c105910 2024-02-28T17:20:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.571+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c05f5c0 con 0x7fc35c105910 2024-02-28T17:20:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.572+0000 7fc353400700 1 -- 172.21.15.1:0/680354592 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc34800bd60 con 0x7fc35c105910 2024-02-28T17:20:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.573+0000 7fc353400700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc34006d7f0 0x7fc34006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:20:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.573+0000 7fc353400700 1 -- 172.21.15.1:0/680354592 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc34808e2c0 con 0x7fc35c105910 2024-02-28T17:20:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.573+0000 7fc35b400700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc34006d7f0 0x7fc34006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:20:38.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.574+0000 7fc35b400700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc34006d7f0 0x7fc34006fca0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc34c007fd0 tx=0x7fc34c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:20:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.577+0000 7fc353400700 1 -- 172.21.15.1:0/680354592 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc34805b620 con 0x7fc35c105910 2024-02-28T17:20:38.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.885+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc35c04bed0 con 0x7fc35c105910 2024-02-28T17:20:38.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.886+0000 7fc353400700 1 -- 172.21.15.1:0/680354592 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fc34805b1b0 con 0x7fc35c105910 2024-02-28T17:20:38.890 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:20:38.891 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:20:38.891 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:20:38.891 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:20:38.891 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:20:38.891 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:20:38.891 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:20:38.891 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:20:38.892 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:20:38.893 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:20:38.893 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:20:38.893 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:20:38.893 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:20:38.893 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:20:38.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc34006d7f0 msgr2=0x7fc34006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc34006d7f0 0x7fc34006fca0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc34c007fd0 tx=0x7fc34c009040 comp rx=0 tx=0).stop 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 msgr2=0x7fc35c115f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 0x7fc35c115f50 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fc348009030 tx=0x7fc348009e30 comp rx=0 tx=0).stop 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 shutdown_connections 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc34006d7f0 0x7fc34006fca0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc35c104f70 0x7fc35c115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 --2- 172.21.15.1:0/680354592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35c105910 0x7fc35c115f50 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.889+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 >> 172.21.15.1:0/680354592 conn(0x7fc35c0faae0 msgr2=0x7fc35c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.890+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 shutdown_connections 2024-02-28T17:20:38.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:20:38.890+0000 7fc360e00700 1 -- 172.21.15.1:0/680354592 wait complete. 2024-02-28T17:20:39.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:38 smithi001 ceph-mon[112940]: from='client.36448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:39.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:38 smithi001 ceph-mon[112940]: pgmap v1389: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:39.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:38 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3875821290' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:20:39.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:38 smithi066 ceph-mon[85781]: from='client.36448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:39.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:38 smithi066 ceph-mon[85781]: pgmap v1389: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:39.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:38 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3875821290' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:20:39.785 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:39 smithi001 ceph-mon[112940]: from='client.36460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:39.786 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:39 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/680354592' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:20:40.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:39 smithi066 ceph-mon[85781]: from='client.36460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:20:40.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:39 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/680354592' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:20:41.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:40 smithi001 ceph-mon[112940]: pgmap v1390: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:41.165 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:40 smithi066 ceph-mon[85781]: pgmap v1390: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:42.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:20:42.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:20:43.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:42 smithi066 ceph-mon[85781]: pgmap v1391: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:43.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:42 smithi001 ceph-mon[112940]: pgmap v1391: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:45.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:44 smithi066 ceph-mon[85781]: pgmap v1392: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:45.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:44 smithi001 ceph-mon[112940]: pgmap v1392: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:47.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:46 smithi066 ceph-mon[85781]: pgmap v1393: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:47.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:46 smithi001 ceph-mon[112940]: pgmap v1393: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:49.087 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:48 smithi001 ceph-mon[112940]: pgmap v1394: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:49.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:48 smithi066 ceph-mon[85781]: pgmap v1394: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:51.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:50 smithi066 ceph-mon[85781]: pgmap v1395: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:51.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:50 smithi001 ceph-mon[112940]: pgmap v1395: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:53.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:52 smithi066 ceph-mon[85781]: pgmap v1396: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:53.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:52 smithi001 ceph-mon[112940]: pgmap v1396: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:55.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:54 smithi066 ceph-mon[85781]: pgmap v1397: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:55.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:54 smithi001 ceph-mon[112940]: pgmap v1397: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:56.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:20:56.287 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:20:57.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:56 smithi066 ceph-mon[85781]: pgmap v1398: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:57.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:56 smithi001 ceph-mon[112940]: pgmap v1398: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:59.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:20:58 smithi066 ceph-mon[85781]: pgmap v1399: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:20:59.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:20:58 smithi001 ceph-mon[112940]: pgmap v1399: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:01.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:00 smithi066 ceph-mon[85781]: pgmap v1400: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:01.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:00 smithi001 ceph-mon[112940]: pgmap v1400: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:03.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:02 smithi066 ceph-mon[85781]: pgmap v1401: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:03.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:02 smithi001 ceph-mon[112940]: pgmap v1401: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:05.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:04 smithi066 ceph-mon[85781]: pgmap v1402: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:05.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:04 smithi001 ceph-mon[112940]: pgmap v1402: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:07.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:06 smithi066 ceph-mon[85781]: pgmap v1403: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:07.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:06 smithi001 ceph-mon[112940]: pgmap v1403: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:09.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.059+0000 7f9dc5000700 1 -- 172.21.15.1:0/3742015498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc0103dc0 msgr2=0x7f9dc0104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.059+0000 7f9dc5000700 1 --2- 172.21.15.1:0/3742015498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc0103dc0 0x7f9dc0104210 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f9dac004940 tx=0x7f9dac009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:09.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.059+0000 7f9dc5000700 1 -- 172.21.15.1:0/3742015498 shutdown_connections 2024-02-28T17:21:09.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.059+0000 7f9dc5000700 1 --2- 172.21.15.1:0/3742015498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc0103dc0 0x7f9dc0104210 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.059+0000 7f9dc5000700 1 --2- 172.21.15.1:0/3742015498 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc01095f0 0x7f9dc01099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.059+0000 7f9dc5000700 1 -- 172.21.15.1:0/3742015498 >> 172.21.15.1:0/3742015498 conn(0x7f9dc0072d50 msgr2=0x7f9dc0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:09.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.059+0000 7f9dc5000700 1 -- 172.21.15.1:0/3742015498 shutdown_connections 2024-02-28T17:21:09.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.059+0000 7f9dc5000700 1 -- 172.21.15.1:0/3742015498 wait complete. 2024-02-28T17:21:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.060+0000 7f9dc5000700 1 Processor -- start 2024-02-28T17:21:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.060+0000 7f9dc5000700 1 -- start start 2024-02-28T17:21:09.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.060+0000 7f9dc5000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc0103dc0 0x7f9dc011a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.060+0000 7f9dc5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc01095f0 0x7f9dc011a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.060+0000 7f9dc5000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc011abe0 con 0x7f9dc01095f0 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.060+0000 7f9dc5000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0113ef0 con 0x7f9dc0103dc0 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.061+0000 7f9dbd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc01095f0 0x7f9dc011a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.061+0000 7f9dbd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc01095f0 0x7f9dc011a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50284/0 (socket says 172.21.15.1:50284) 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.061+0000 7f9dbd400700 1 -- 172.21.15.1:0/3679582615 learned_addr learned my addr 172.21.15.1:0/3679582615 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.061+0000 7f9dbde00700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc0103dc0 0x7f9dc011a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.061+0000 7f9dbd400700 1 -- 172.21.15.1:0/3679582615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc0103dc0 msgr2=0x7f9dc011a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.061+0000 7f9dbd400700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc0103dc0 0x7f9dc011a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.061+0000 7f9dbd400700 1 -- 172.21.15.1:0/3679582615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac005040 con 0x7f9dc01095f0 2024-02-28T17:21:09.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.062+0000 7f9dbd400700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc01095f0 0x7f9dc011a590 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009150 tx=0x7f9dac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.062+0000 7f9db6000700 1 -- 172.21.15.1:0/3679582615 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dac00e040 con 0x7f9dc01095f0 2024-02-28T17:21:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.062+0000 7f9db6000700 1 -- 172.21.15.1:0/3679582615 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dac00bbf0 con 0x7f9dc01095f0 2024-02-28T17:21:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.062+0000 7f9db6000700 1 -- 172.21.15.1:0/3679582615 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dac01c630 con 0x7f9dc01095f0 2024-02-28T17:21:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.062+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc0114170 con 0x7f9dc01095f0 2024-02-28T17:21:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.062+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc01146b0 con 0x7f9dc01095f0 2024-02-28T17:21:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.063+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc010b9c0 con 0x7f9dc01095f0 2024-02-28T17:21:09.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.064+0000 7f9db6000700 1 -- 172.21.15.1:0/3679582615 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9dac00bd60 con 0x7f9dc01095f0 2024-02-28T17:21:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.066+0000 7f9db6000700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da406d7f0 0x7f9da406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.066+0000 7f9db6000700 1 -- 172.21.15.1:0/3679582615 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9dac08e290 con 0x7f9dc01095f0 2024-02-28T17:21:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.066+0000 7f9dbde00700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da406d7f0 0x7f9da406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.067+0000 7f9dbde00700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da406d7f0 0x7f9da406fca0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009fd0 tx=0x7f9db0009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:09.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.071+0000 7f9db6000700 1 -- 172.21.15.1:0/3679582615 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9dac05b670 con 0x7f9dc01095f0 2024-02-28T17:21:09.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:08 smithi066 ceph-mon[85781]: pgmap v1404: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:09.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:08 smithi001 ceph-mon[112940]: pgmap v1404: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.315+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dc004bed0 con 0x7f9da406d7f0 2024-02-28T17:21:09.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.318+0000 7f9db6000700 1 -- 172.21.15.1:0/3679582615 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9dc004bed0 con 0x7f9da406d7f0 2024-02-28T17:21:09.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.320+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da406d7f0 msgr2=0x7f9da406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.320+0000 7f9dc5000700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da406d7f0 0x7f9da406fca0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009fd0 tx=0x7f9db0009f20 comp rx=0 tx=0).stop 2024-02-28T17:21:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.320+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc01095f0 msgr2=0x7f9dc011a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.320+0000 7f9dc5000700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc01095f0 0x7f9dc011a590 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009150 tx=0x7f9dac009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.321+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 shutdown_connections 2024-02-28T17:21:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.321+0000 7f9dc5000700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da406d7f0 0x7f9da406fca0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.321+0000 7f9dc5000700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc0103dc0 0x7f9dc011a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.321+0000 7f9dc5000700 1 --2- 172.21.15.1:0/3679582615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc01095f0 0x7f9dc011a590 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.321+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 >> 172.21.15.1:0/3679582615 conn(0x7f9dc0072d50 msgr2=0x7f9dc0074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.321+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 shutdown_connections 2024-02-28T17:21:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.321+0000 7f9dc5000700 1 -- 172.21.15.1:0/3679582615 wait complete. 2024-02-28T17:21:09.342 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:21:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.481+0000 7f2c3f200700 1 -- 172.21.15.1:0/2580967198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 msgr2=0x7f2c38109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.481+0000 7f2c3f200700 1 --2- 172.21.15.1:0/2580967198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 0x7f2c38109250 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2c24004970 tx=0x7f2c24009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.481+0000 7f2c3f200700 1 -- 172.21.15.1:0/2580967198 shutdown_connections 2024-02-28T17:21:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.481+0000 7f2c3f200700 1 --2- 172.21.15.1:0/2580967198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 0x7f2c38109250 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.481+0000 7f2c3f200700 1 --2- 172.21.15.1:0/2580967198 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c38070330 0x7f2c38070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.481+0000 7f2c3f200700 1 -- 172.21.15.1:0/2580967198 >> 172.21.15.1:0/2580967198 conn(0x7f2c380f8af0 msgr2=0x7f2c380faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.481+0000 7f2c3f200700 1 -- 172.21.15.1:0/2580967198 shutdown_connections 2024-02-28T17:21:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.482+0000 7f2c3f200700 1 -- 172.21.15.1:0/2580967198 wait complete. 2024-02-28T17:21:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.482+0000 7f2c3f200700 1 Processor -- start 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.483+0000 7f2c3f200700 1 -- start start 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.483+0000 7f2c3f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c38070330 0x7f2c38115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.483+0000 7f2c3f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 0x7f2c38116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.483+0000 7f2c3f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c381167a0 con 0x7f2c38070c40 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.483+0000 7f2c3f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3810faf0 con 0x7f2c38070330 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.483+0000 7f2c37400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 0x7f2c38116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.483+0000 7f2c37400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 0x7f2c38116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50306/0 (socket says 172.21.15.1:50306) 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.483+0000 7f2c37400700 1 -- 172.21.15.1:0/3524294706 learned_addr learned my addr 172.21.15.1:0/3524294706 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.484+0000 7f2c37e00700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c38070330 0x7f2c38115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.484+0000 7f2c37400700 1 -- 172.21.15.1:0/3524294706 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c38070330 msgr2=0x7f2c38115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.484+0000 7f2c37400700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c38070330 0x7f2c38115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.484+0000 7f2c37400700 1 -- 172.21.15.1:0/3524294706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c24005040 con 0x7f2c38070c40 2024-02-28T17:21:09.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.484+0000 7f2c37400700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 0x7f2c38116150 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f2c24004f50 tx=0x7f2c24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:09.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.484+0000 7f2c2fe00700 1 -- 172.21.15.1:0/3524294706 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c2400e040 con 0x7f2c38070c40 2024-02-28T17:21:09.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.485+0000 7f2c2fe00700 1 -- 172.21.15.1:0/3524294706 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c24004290 con 0x7f2c38070c40 2024-02-28T17:21:09.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.485+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3810fd70 con 0x7f2c38070c40 2024-02-28T17:21:09.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.485+0000 7f2c2fe00700 1 -- 172.21.15.1:0/3524294706 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c2401c630 con 0x7f2c38070c40 2024-02-28T17:21:09.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.485+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c381102b0 con 0x7f2c38070c40 2024-02-28T17:21:09.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.486+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c38106910 con 0x7f2c38070c40 2024-02-28T17:21:09.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.487+0000 7f2c2fe00700 1 -- 172.21.15.1:0/3524294706 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2c2400bc10 con 0x7f2c38070c40 2024-02-28T17:21:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.488+0000 7f2c2fe00700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c1c06d7f0 0x7f2c1c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.488+0000 7f2c2fe00700 1 -- 172.21.15.1:0/3524294706 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2c2408e320 con 0x7f2c38070c40 2024-02-28T17:21:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.488+0000 7f2c37e00700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c1c06d7f0 0x7f2c1c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.489+0000 7f2c37e00700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c1c06d7f0 0x7f2c1c06fca0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009fd0 tx=0x7f2c28009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.493+0000 7f2c2fe00700 1 -- 172.21.15.1:0/3524294706 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c24057b50 con 0x7f2c38070c40 2024-02-28T17:21:09.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.741+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c3804bed0 con 0x7f2c1c06d7f0 2024-02-28T17:21:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.744+0000 7f2c2fe00700 1 -- 172.21.15.1:0/3524294706 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2c3804bed0 con 0x7f2c1c06d7f0 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.747+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c1c06d7f0 msgr2=0x7f2c1c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.747+0000 7f2c3f200700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c1c06d7f0 0x7f2c1c06fca0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009fd0 tx=0x7f2c28009f20 comp rx=0 tx=0).stop 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.747+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 msgr2=0x7f2c38116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.747+0000 7f2c3f200700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 0x7f2c38116150 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f2c24004f50 tx=0x7f2c24009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.748+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 shutdown_connections 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.748+0000 7f2c3f200700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2c1c06d7f0 0x7f2c1c06fca0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.748+0000 7f2c3f200700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2c38070330 0x7f2c38115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.748+0000 7f2c3f200700 1 --2- 172.21.15.1:0/3524294706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c38070c40 0x7f2c38116150 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.748+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 >> 172.21.15.1:0/3524294706 conn(0x7f2c380f8af0 msgr2=0x7f2c38103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.748+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 shutdown_connections 2024-02-28T17:21:09.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.748+0000 7f2c3f200700 1 -- 172.21.15.1:0/3524294706 wait complete. 2024-02-28T17:21:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.906+0000 7f0447000700 1 -- 172.21.15.1:0/2555003225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105ad0 msgr2=0x7f04400ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.906+0000 7f0447000700 1 --2- 172.21.15.1:0/2555003225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105ad0 0x7f04400ff140 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f042c004940 tx=0x7f042c009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.907+0000 7f0447000700 1 -- 172.21.15.1:0/2555003225 shutdown_connections 2024-02-28T17:21:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.907+0000 7f0447000700 1 --2- 172.21.15.1:0/2555003225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105ad0 0x7f04400ff140 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.907+0000 7f0447000700 1 --2- 172.21.15.1:0/2555003225 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0440105130 0x7f0440105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.907+0000 7f0447000700 1 -- 172.21.15.1:0/2555003225 >> 172.21.15.1:0/2555003225 conn(0x7f04400fad00 msgr2=0x7f04400fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.907+0000 7f0447000700 1 -- 172.21.15.1:0/2555003225 shutdown_connections 2024-02-28T17:21:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.907+0000 7f0447000700 1 -- 172.21.15.1:0/2555003225 wait complete. 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.908+0000 7f0447000700 1 Processor -- start 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.908+0000 7f0447000700 1 -- start start 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.908+0000 7f0447000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105130 0x7f0440115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.908+0000 7f0447000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0440105ad0 0x7f04401161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.908+0000 7f0447000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0440116840 con 0x7f0440105130 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.908+0000 7f0447000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044010fb50 con 0x7f0440105ad0 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.908+0000 7f043fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105130 0x7f0440115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.909+0000 7f043fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105130 0x7f0440115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50318/0 (socket says 172.21.15.1:50318) 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.909+0000 7f043fe00700 1 -- 172.21.15.1:0/1506232456 learned_addr learned my addr 172.21.15.1:0/1506232456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.909+0000 7f043f400700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0440105ad0 0x7f04401161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.909+0000 7f043fe00700 1 -- 172.21.15.1:0/1506232456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0440105ad0 msgr2=0x7f04401161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:09.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.909+0000 7f043fe00700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0440105ad0 0x7f04401161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:09.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.909+0000 7f043fe00700 1 -- 172.21.15.1:0/1506232456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c005040 con 0x7f0440105130 2024-02-28T17:21:09.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.909+0000 7f043f400700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0440105ad0 0x7f04401161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:21:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.909+0000 7f043fe00700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105130 0x7f0440115cb0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f0430002a50 tx=0x7f04300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.910+0000 7f0437e00700 1 -- 172.21.15.1:0/1506232456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0430007cc0 con 0x7f0440105130 2024-02-28T17:21:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.910+0000 7f0437e00700 1 -- 172.21.15.1:0/1506232456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f043000f040 con 0x7f0440105130 2024-02-28T17:21:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.910+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044010fe30 con 0x7f0440105130 2024-02-28T17:21:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.910+0000 7f0437e00700 1 -- 172.21.15.1:0/1506232456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0430009640 con 0x7f0440105130 2024-02-28T17:21:09.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.910+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0440110370 con 0x7f0440105130 2024-02-28T17:21:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.911+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04400fe900 con 0x7f0440105130 2024-02-28T17:21:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.912+0000 7f0437e00700 1 -- 172.21.15.1:0/1506232456 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f04300097a0 con 0x7f0440105130 2024-02-28T17:21:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.913+0000 7f0437e00700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f042406d7e0 0x7f042406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.913+0000 7f0437e00700 1 -- 172.21.15.1:0/1506232456 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f043008cc60 con 0x7f0440105130 2024-02-28T17:21:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.913+0000 7f043f400700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f042406d7e0 0x7f042406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:09.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.914+0000 7f043f400700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f042406d7e0 0x7f042406fc90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f042c000c00 tx=0x7f042c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:09.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:09.918+0000 7f0437e00700 1 -- 172.21.15.1:0/1506232456 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0430059fc0 con 0x7f0440105130 2024-02-28T17:21:10.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.151+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f04401113a0 con 0x7f042406d7e0 2024-02-28T17:21:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.163+0000 7f0437e00700 1 -- 172.21.15.1:0/1506232456 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f04401113a0 con 0x7f042406d7e0 2024-02-28T17:21:10.168 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:21:10.168 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (47m) 8m ago 61m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:21:10.168 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (44m) 8m ago 61m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:21:10.168 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (43m) 8m ago 60m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:21:10.168 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (46m) 8m ago 61m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (39m) 8m ago 54m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (39m) 8m ago 54m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (8m) 8m ago 54m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (24m) 8m ago 54m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (49m) 8m ago 63m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (48m) 8m ago 60m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (45m) 8m ago 63m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (45m) 8m ago 60m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (48m) 8m ago 61m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (48m) 8m ago 60m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (43m) 8m ago 59m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:21:10.169 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (42m) 8m ago 58m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:21:10.170 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (42m) 8m ago 58m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:21:10.170 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (41m) 8m ago 57m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:21:10.170 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (41m) 8m ago 56m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:21:10.170 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (40m) 8m ago 56m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:21:10.170 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (47m) 8m ago 60m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:21:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.165+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f042406d7e0 msgr2=0x7f042406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.165+0000 7f0447000700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f042406d7e0 0x7f042406fc90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f042c000c00 tx=0x7f042c009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.165+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105130 msgr2=0x7f0440115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.165+0000 7f0447000700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105130 0x7f0440115cb0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f0430002a50 tx=0x7f04300046b0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.166+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 shutdown_connections 2024-02-28T17:21:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.166+0000 7f0447000700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f042406d7e0 0x7f042406fc90 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.166+0000 7f0447000700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0440105130 0x7f0440115cb0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.166+0000 7f0447000700 1 --2- 172.21.15.1:0/1506232456 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0440105ad0 0x7f04401161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.166+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 >> 172.21.15.1:0/1506232456 conn(0x7f04400fad00 msgr2=0x7f04400fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.166+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 shutdown_connections 2024-02-28T17:21:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.166+0000 7f0447000700 1 -- 172.21.15.1:0/1506232456 wait complete. 2024-02-28T17:21:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.320+0000 7f6ed4200700 1 -- 172.21.15.1:0/3507450263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc1038f0 msgr2=0x7f6ecc0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.320+0000 7f6ed4200700 1 --2- 172.21.15.1:0/3507450263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc1038f0 0x7f6ecc0fcf60 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8004940 tx=0x7f6eb8009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.320+0000 7f6ed4200700 1 -- 172.21.15.1:0/3507450263 shutdown_connections 2024-02-28T17:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.320+0000 7f6ed4200700 1 --2- 172.21.15.1:0/3507450263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc1038f0 0x7f6ecc0fcf60 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.320+0000 7f6ed4200700 1 --2- 172.21.15.1:0/3507450263 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ecc102f50 0x7f6ecc103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.320+0000 7f6ed4200700 1 -- 172.21.15.1:0/3507450263 >> 172.21.15.1:0/3507450263 conn(0x7f6ecc0f8ae0 msgr2=0x7f6ecc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.320+0000 7f6ed4200700 1 -- 172.21.15.1:0/3507450263 shutdown_connections 2024-02-28T17:21:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.320+0000 7f6ed4200700 1 -- 172.21.15.1:0/3507450263 wait complete. 2024-02-28T17:21:10.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.321+0000 7f6ed4200700 1 Processor -- start 2024-02-28T17:21:10.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.321+0000 7f6ed4200700 1 -- start start 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.322+0000 7f6ed4200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc102f50 0x7f6ecc113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.322+0000 7f6ed4200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ecc1038f0 0x7f6ecc113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.322+0000 7f6ed4200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc114600 con 0x7f6ecc102f50 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.322+0000 7f6ed4200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc10d930 con 0x7f6ecc1038f0 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.322+0000 7f6ed1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc102f50 0x7f6ecc113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.322+0000 7f6ed1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc102f50 0x7f6ecc113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50328/0 (socket says 172.21.15.1:50328) 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.322+0000 7f6ed1800700 1 -- 172.21.15.1:0/1172436707 learned_addr learned my addr 172.21.15.1:0/1172436707 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ed0e00700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ecc1038f0 0x7f6ecc113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ed1800700 1 -- 172.21.15.1:0/1172436707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ecc1038f0 msgr2=0x7f6ecc113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ed1800700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ecc1038f0 0x7f6ecc113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ed1800700 1 -- 172.21.15.1:0/1172436707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb8005040 con 0x7f6ecc102f50 2024-02-28T17:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ed1800700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc102f50 0x7f6ecc113a70 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0002a50 tx=0x7f6ec00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ec9600700 1 -- 172.21.15.1:0/1172436707 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ec0007cc0 con 0x7f6ecc102f50 2024-02-28T17:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ec9600700 1 -- 172.21.15.1:0/1172436707 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ec000f040 con 0x7f6ecc102f50 2024-02-28T17:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc10dc10 con 0x7f6ecc102f50 2024-02-28T17:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.323+0000 7f6ec9600700 1 -- 172.21.15.1:0/1172436707 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ec0009640 con 0x7f6ecc102f50 2024-02-28T17:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.324+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc10e150 con 0x7f6ecc102f50 2024-02-28T17:21:10.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.325+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ecc105420 con 0x7f6ecc102f50 2024-02-28T17:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.326+0000 7f6ec9600700 1 -- 172.21.15.1:0/1172436707 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6ec00097a0 con 0x7f6ecc102f50 2024-02-28T17:21:10.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.326+0000 7f6ec9600700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6eb406d7f0 0x7f6eb406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.327+0000 7f6ec9600700 1 -- 172.21.15.1:0/1172436707 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6ec008cc60 con 0x7f6ecc102f50 2024-02-28T17:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.327+0000 7f6ed0e00700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6eb406d7f0 0x7f6eb406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:10.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.327+0000 7f6ed0e00700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6eb406d7f0 0x7f6eb406fca0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc10f2a0 tx=0x7f6eb8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:10.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.331+0000 7f6ec9600700 1 -- 172.21.15.1:0/1172436707 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6ec0059fc0 con 0x7f6ecc102f50 2024-02-28T17:21:10.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.642+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6ecc04bed0 con 0x7f6ecc102f50 2024-02-28T17:21:10.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.643+0000 7f6ec9600700 1 -- 172.21.15.1:0/1172436707 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6ec0059b50 con 0x7f6ecc102f50 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:21:10.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:21:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:21:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:21:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:21:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:21:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:21:10.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:21:10.650 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6eb406d7f0 msgr2=0x7f6eb406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6eb406d7f0 0x7f6eb406fca0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc10f2a0 tx=0x7f6eb8009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc102f50 msgr2=0x7f6ecc113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc102f50 0x7f6ecc113a70 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0002a50 tx=0x7f6ec00046b0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 shutdown_connections 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6eb406d7f0 0x7f6eb406fca0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ecc102f50 0x7f6ecc113a70 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 --2- 172.21.15.1:0/1172436707 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6ecc1038f0 0x7f6ecc113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 >> 172.21.15.1:0/1172436707 conn(0x7f6ecc0f8ae0 msgr2=0x7f6ecc0fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 shutdown_connections 2024-02-28T17:21:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.646+0000 7f6ed4200700 1 -- 172.21.15.1:0/1172436707 wait complete. 2024-02-28T17:21:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.801+0000 7f8ab9c00700 1 -- 172.21.15.1:0/2176758758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 msgr2=0x7f8ab40fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.801+0000 7f8ab9c00700 1 --2- 172.21.15.1:0/2176758758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 0x7f8ab40fe3b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4004970 tx=0x7f8aa4009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.801+0000 7f8ab9c00700 1 -- 172.21.15.1:0/2176758758 shutdown_connections 2024-02-28T17:21:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.801+0000 7f8ab9c00700 1 --2- 172.21.15.1:0/2176758758 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ab40fe8f0 0x7f8ab4107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.801+0000 7f8ab9c00700 1 --2- 172.21.15.1:0/2176758758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 0x7f8ab40fe3b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.801+0000 7f8ab9c00700 1 -- 172.21.15.1:0/2176758758 >> 172.21.15.1:0/2176758758 conn(0x7f8ab4071d90 msgr2=0x7f8ab4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.801+0000 7f8ab9c00700 1 -- 172.21.15.1:0/2176758758 shutdown_connections 2024-02-28T17:21:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.801+0000 7f8ab9c00700 1 -- 172.21.15.1:0/2176758758 wait complete. 2024-02-28T17:21:10.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.802+0000 7f8ab9c00700 1 Processor -- start 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.802+0000 7f8ab9c00700 1 -- start start 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.802+0000 7f8ab9c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 0x7f8ab40fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.802+0000 7f8ab9c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ab40fe8f0 0x7f8ab40fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.802+0000 7f8ab9c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40fc9e0 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.802+0000 7f8ab9c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40fcb50 con 0x7f8ab40fe8f0 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.802+0000 7f8ab2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 0x7f8ab40fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.803+0000 7f8ab2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 0x7f8ab40fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50350/0 (socket says 172.21.15.1:50350) 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.803+0000 7f8ab2a00700 1 -- 172.21.15.1:0/886125629 learned_addr learned my addr 172.21.15.1:0/886125629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.803+0000 7f8ab2a00700 1 -- 172.21.15.1:0/886125629 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ab40fe8f0 msgr2=0x7f8ab40fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.803+0000 7f8ab2000700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ab40fe8f0 0x7f8ab40fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.803+0000 7f8ab2a00700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ab40fe8f0 0x7f8ab40fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.803+0000 7f8ab2a00700 1 -- 172.21.15.1:0/886125629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa4005040 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.803+0000 7f8ab2a00700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 0x7f8ab40fbea0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4005010 tx=0x7f8aa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:10.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.804+0000 7f8aaaa00700 1 -- 172.21.15.1:0/886125629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8aa400e040 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.804+0000 7f8aaaa00700 1 -- 172.21.15.1:0/886125629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8aa4013420 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.804+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab4119c90 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.804+0000 7f8aaaa00700 1 -- 172.21.15.1:0/886125629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8aa401c610 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.804+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab411a0f0 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.805+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab41046c0 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.806+0000 7f8aaaa00700 1 -- 172.21.15.1:0/886125629 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8aa4013a90 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.807+0000 7f8aaaa00700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a9806d7e0 0x7f8a9806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:10.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.808+0000 7f8aaaa00700 1 -- 172.21.15.1:0/886125629 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8aa408e240 con 0x7f8ab40fdfe0 2024-02-28T17:21:10.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.808+0000 7f8ab2000700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a9806d7e0 0x7f8a9806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:10.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.808+0000 7f8ab2000700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a9806d7e0 0x7f8a9806fc90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40fd8a0 tx=0x7f8aa000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:10.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:10.812+0000 7f8aaaa00700 1 -- 172.21.15.1:0/886125629 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8aa4057a70 con 0x7f8ab40fdfe0 2024-02-28T17:21:11.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:10 smithi001 ceph-mon[112940]: from='client.36468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:11.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:10 smithi001 ceph-mon[112940]: pgmap v1405: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:11.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:10 smithi001 ceph-mon[112940]: from='client.36472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:11.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:10 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1172436707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:21:11.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:21:11.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.081+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8ab404bf00 con 0x7f8ab40fdfe0 2024-02-28T17:21:11.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.082+0000 7f8aaaa00700 1 -- 172.21.15.1:0/886125629 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f8aa405b130 con 0x7f8ab40fdfe0 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:21:11.089 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:21:11.090 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:21:11.091 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:21:11.092 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:21:11.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a9806d7e0 msgr2=0x7f8a9806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a9806d7e0 0x7f8a9806fc90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40fd8a0 tx=0x7f8aa000b040 comp rx=0 tx=0).stop 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 msgr2=0x7f8ab40fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 0x7f8ab40fbea0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4005010 tx=0x7f8aa4009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 shutdown_connections 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8a9806d7e0 0x7f8a9806fc90 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab40fdfe0 0x7f8ab40fbea0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 --2- 172.21.15.1:0/886125629 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8ab40fe8f0 0x7f8ab40fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 >> 172.21.15.1:0/886125629 conn(0x7f8ab4071d90 msgr2=0x7f8ab4101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 shutdown_connections 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.085+0000 7f8ab9c00700 1 -- 172.21.15.1:0/886125629 wait complete. 2024-02-28T17:21:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:21:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.229+0000 7fae40c00700 1 -- 172.21.15.1:0/2051740625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 msgr2=0x7fae3c105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.229+0000 7fae40c00700 1 --2- 172.21.15.1:0/2051740625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 0x7fae3c105540 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fae2c004940 tx=0x7fae2c009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.230+0000 7fae40c00700 1 -- 172.21.15.1:0/2051740625 shutdown_connections 2024-02-28T17:21:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.230+0000 7fae40c00700 1 --2- 172.21.15.1:0/2051740625 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c105b10 0x7fae3c0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.230+0000 7fae40c00700 1 --2- 172.21.15.1:0/2051740625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 0x7fae3c105540 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.230+0000 7fae40c00700 1 -- 172.21.15.1:0/2051740625 >> 172.21.15.1:0/2051740625 conn(0x7fae3c0face0 msgr2=0x7fae3c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.230+0000 7fae40c00700 1 -- 172.21.15.1:0/2051740625 shutdown_connections 2024-02-28T17:21:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.230+0000 7fae40c00700 1 -- 172.21.15.1:0/2051740625 wait complete. 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.231+0000 7fae40c00700 1 Processor -- start 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.231+0000 7fae40c00700 1 -- start start 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.231+0000 7fae40c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 0x7fae3c115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.231+0000 7fae40c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c105b10 0x7fae3c1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.231+0000 7fae40c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c1167f0 con 0x7fae3c105170 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.231+0000 7fae40c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c10fb40 con 0x7fae3c105b10 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.232+0000 7fae39e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 0x7fae3c115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.232+0000 7fae39e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 0x7fae3c115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50380/0 (socket says 172.21.15.1:50380) 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.232+0000 7fae39e00700 1 -- 172.21.15.1:0/148747060 learned_addr learned my addr 172.21.15.1:0/148747060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.232+0000 7fae39400700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c105b10 0x7fae3c1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.232+0000 7fae39e00700 1 -- 172.21.15.1:0/148747060 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c105b10 msgr2=0x7fae3c1161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.232+0000 7fae39e00700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c105b10 0x7fae3c1161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.232+0000 7fae39e00700 1 -- 172.21.15.1:0/148747060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c005040 con 0x7fae3c105170 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.232+0000 7fae39e00700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 0x7fae3c115c60 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009fd0 tx=0x7fae2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.233+0000 7fae32000700 1 -- 172.21.15.1:0/148747060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2c00e040 con 0x7fae3c105170 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.233+0000 7fae32000700 1 -- 172.21.15.1:0/148747060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae2c00bbf0 con 0x7fae3c105170 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.233+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c10fdc0 con 0x7fae3c105170 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.233+0000 7fae32000700 1 -- 172.21.15.1:0/148747060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2c01c3f0 con 0x7fae3c105170 2024-02-28T17:21:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.233+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c110300 con 0x7fae3c105170 2024-02-28T17:21:11.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.234+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c102480 con 0x7fae3c105170 2024-02-28T17:21:11.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.235+0000 7fae32000700 1 -- 172.21.15.1:0/148747060 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fae2c00bd60 con 0x7fae3c105170 2024-02-28T17:21:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.236+0000 7fae32000700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.236+0000 7fae32000700 1 -- 172.21.15.1:0/148747060 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fae2c08e100 con 0x7fae3c105170 2024-02-28T17:21:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.236+0000 7fae39400700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.237+0000 7fae39400700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fae3c111540 tx=0x7fae2800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.241+0000 7fae32000700 1 -- 172.21.15.1:0/148747060 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae2c057a70 con 0x7fae3c105170 2024-02-28T17:21:11.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:10 smithi066 ceph-mon[85781]: from='client.36468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:11.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:10 smithi066 ceph-mon[85781]: pgmap v1405: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:11.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:10 smithi066 ceph-mon[85781]: from='client.36472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:11.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:10 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1172436707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:21:11.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:21:11.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.486+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae3c04bf70 con 0x7fae2006d7f0 2024-02-28T17:21:11.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.489+0000 7fae32000700 1 -- 172.21.15.1:0/148747060 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fae2800f040 con 0x7fae2006d7f0 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:21:11.495 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:21:11.495 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:21:11.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.492+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 msgr2=0x7fae2006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.492+0000 7fae40c00700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fae3c111540 tx=0x7fae2800b040 comp rx=0 tx=0).stop 2024-02-28T17:21:11.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.492+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 msgr2=0x7fae3c115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.492+0000 7fae40c00700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 0x7fae3c115c60 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009fd0 tx=0x7fae2c009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.492+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 shutdown_connections 2024-02-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.492+0000 7fae40c00700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fae2006d7f0 0x7fae2006fca0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.492+0000 7fae40c00700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae3c105170 0x7fae3c115c60 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.492+0000 7fae40c00700 1 --2- 172.21.15.1:0/148747060 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fae3c105b10 0x7fae3c1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.493+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 >> 172.21.15.1:0/148747060 conn(0x7fae3c0face0 msgr2=0x7fae3c0fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.493+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 shutdown_connections 2024-02-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.493+0000 7fae40c00700 1 -- 172.21.15.1:0/148747060 wait complete. 2024-02-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.636+0000 7faae0400700 1 -- 172.21.15.1:0/2944825038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 msgr2=0x7faad8070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.636+0000 7faae0400700 1 --2- 172.21.15.1:0/2944825038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 0x7faad8070330 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7faacc004970 tx=0x7faacc009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.636+0000 7faae0400700 1 -- 172.21.15.1:0/2944825038 shutdown_connections 2024-02-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.636+0000 7faae0400700 1 --2- 172.21.15.1:0/2944825038 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faad8070870 0x7faad8070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.636+0000 7faae0400700 1 --2- 172.21.15.1:0/2944825038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 0x7faad8070330 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.636+0000 7faae0400700 1 -- 172.21.15.1:0/2944825038 >> 172.21.15.1:0/2944825038 conn(0x7faad80f8ae0 msgr2=0x7faad80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.637+0000 7faae0400700 1 -- 172.21.15.1:0/2944825038 shutdown_connections 2024-02-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.637+0000 7faae0400700 1 -- 172.21.15.1:0/2944825038 wait complete. 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.637+0000 7faae0400700 1 Processor -- start 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.638+0000 7faae0400700 1 -- start start 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.638+0000 7faae0400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faad8070870 0x7faad811a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.638+0000 7faae0400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 0x7faad811a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.638+0000 7faae0400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad811ac00 con 0x7faad8071f60 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.638+0000 7faae0400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad8113f10 con 0x7faad8070870 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.638+0000 7faadd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 0x7faad811a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.638+0000 7faadd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 0x7faad811a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50402/0 (socket says 172.21.15.1:50402) 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.638+0000 7faadd000700 1 -- 172.21.15.1:0/1339105696 learned_addr learned my addr 172.21.15.1:0/1339105696 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faadda00700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faad8070870 0x7faad811a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faadd000700 1 -- 172.21.15.1:0/1339105696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faad8070870 msgr2=0x7faad811a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faadd000700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faad8070870 0x7faad811a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faadd000700 1 -- 172.21.15.1:0/1339105696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc005040 con 0x7faad8071f60 2024-02-28T17:21:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faadd000700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 0x7faad811a5b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7faac400c830 tx=0x7faac4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faad5600700 1 -- 172.21.15.1:0/1339105696 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac400a760 con 0x7faad8071f60 2024-02-28T17:21:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faad5600700 1 -- 172.21.15.1:0/1339105696 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faac400ad90 con 0x7faad8071f60 2024-02-28T17:21:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faad5600700 1 -- 172.21.15.1:0/1339105696 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac40113a0 con 0x7faad8071f60 2024-02-28T17:21:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad81141f0 con 0x7faad8071f60 2024-02-28T17:21:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.639+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad8114730 con 0x7faad8071f60 2024-02-28T17:21:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.640+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad80fe400 con 0x7faad8071f60 2024-02-28T17:21:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.642+0000 7faad5600700 1 -- 172.21.15.1:0/1339105696 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7faac4012400 con 0x7faad8071f60 2024-02-28T17:21:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.643+0000 7faad5600700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faac006d7f0 0x7faac006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.643+0000 7faad5600700 1 -- 172.21.15.1:0/1339105696 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7faac408cf00 con 0x7faad8071f60 2024-02-28T17:21:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.643+0000 7faadda00700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faac006d7f0 0x7faac006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.644+0000 7faadda00700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faac006d7f0 0x7faac006fca0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7faacc004ee0 tx=0x7faacc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:11.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.648+0000 7faad5600700 1 -- 172.21.15.1:0/1339105696 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faac405a2e0 con 0x7faad8071f60 2024-02-28T17:21:11.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.948+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faad80fc6e0 con 0x7faad8071f60 2024-02-28T17:21:11.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.949+0000 7faad5600700 1 -- 172.21.15.1:0/1339105696 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7faac4059e70 con 0x7faad8071f60 2024-02-28T17:21:11.954 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:21:11.954 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:21:11.954 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:21:11.954 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:21:11.954 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:21:11.954 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:21:11.955 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:21:11.956 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:21:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:21:11.956 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:21:11.956 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.951+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faac006d7f0 msgr2=0x7faac006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.951+0000 7faae0400700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faac006d7f0 0x7faac006fca0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7faacc004ee0 tx=0x7faacc009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.951+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 msgr2=0x7faad811a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.951+0000 7faae0400700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 0x7faad811a5b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7faac400c830 tx=0x7faac4004470 comp rx=0 tx=0).stop 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.952+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 shutdown_connections 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.952+0000 7faae0400700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faac006d7f0 0x7faac006fca0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.952+0000 7faae0400700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faad8070870 0x7faad811a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.952+0000 7faae0400700 1 --2- 172.21.15.1:0/1339105696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faad8071f60 0x7faad811a5b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.952+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 >> 172.21.15.1:0/1339105696 conn(0x7faad80f8ae0 msgr2=0x7faad80fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.952+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 shutdown_connections 2024-02-28T17:21:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:11.952+0000 7faae0400700 1 -- 172.21.15.1:0/1339105696 wait complete. 2024-02-28T17:21:12.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:11 smithi066 ceph-mon[85781]: from='client.36476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:12.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:11 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/886125629' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:21:12.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:11 smithi001 ceph-mon[112940]: from='client.36476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:12.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:11 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/886125629' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:21:13.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:12 smithi066 ceph-mon[85781]: from='client.36488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:13.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:12 smithi066 ceph-mon[85781]: pgmap v1406: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:13.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:12 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1339105696' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:21:13.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:12 smithi001 ceph-mon[112940]: from='client.36488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:13.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:12 smithi001 ceph-mon[112940]: pgmap v1406: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:13.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:12 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1339105696' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:21:15.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:14 smithi066 ceph-mon[85781]: pgmap v1407: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:15.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:14 smithi001 ceph-mon[112940]: pgmap v1407: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:17.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:16 smithi066 ceph-mon[85781]: pgmap v1408: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:17.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:16 smithi001 ceph-mon[112940]: pgmap v1408: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:19.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:18 smithi066 ceph-mon[85781]: pgmap v1409: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:19.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:18 smithi001 ceph-mon[112940]: pgmap v1409: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:21.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:20 smithi066 ceph-mon[85781]: pgmap v1410: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:21.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:20 smithi001 ceph-mon[112940]: pgmap v1410: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:23.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:22 smithi066 ceph-mon[85781]: pgmap v1411: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:23.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:22 smithi001 ceph-mon[112940]: pgmap v1411: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:25.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:24 smithi066 ceph-mon[85781]: pgmap v1412: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:25.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:24 smithi001 ceph-mon[112940]: pgmap v1412: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:26.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:21:26.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:21:27.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:26 smithi066 ceph-mon[85781]: pgmap v1413: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:27.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:26 smithi001 ceph-mon[112940]: pgmap v1413: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:29.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:28 smithi066 ceph-mon[85781]: pgmap v1414: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:29.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:28 smithi001 ceph-mon[112940]: pgmap v1414: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:31.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:30 smithi066 ceph-mon[85781]: pgmap v1415: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:31.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:30 smithi001 ceph-mon[112940]: pgmap v1415: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:33.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:32 smithi066 ceph-mon[85781]: pgmap v1416: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:33.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:32 smithi001 ceph-mon[112940]: pgmap v1416: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:35.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:34 smithi066 ceph-mon[85781]: pgmap v1417: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:35.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:34 smithi001 ceph-mon[112940]: pgmap v1417: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:37.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:36 smithi066 ceph-mon[85781]: pgmap v1418: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:37.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:36 smithi001 ceph-mon[112940]: pgmap v1418: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:39.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:38 smithi066 ceph-mon[85781]: pgmap v1419: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:39.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:38 smithi001 ceph-mon[112940]: pgmap v1419: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:40 smithi066 ceph-mon[85781]: pgmap v1420: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:21:41.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:40 smithi001 ceph-mon[112940]: pgmap v1420: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:41.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:21:42.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.124+0000 7fb266600700 1 -- 172.21.15.1:0/3158922557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 msgr2=0x7fb2600706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.124+0000 7fb266600700 1 --2- 172.21.15.1:0/3158922557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 0x7fb2600706a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fb250004970 tx=0x7fb250009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:42.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.124+0000 7fb266600700 1 -- 172.21.15.1:0/3158922557 shutdown_connections 2024-02-28T17:21:42.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.124+0000 7fb266600700 1 --2- 172.21.15.1:0/3158922557 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070be0 0x7fb2601092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.124+0000 7fb266600700 1 --2- 172.21.15.1:0/3158922557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 0x7fb2600706a0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.124+0000 7fb266600700 1 -- 172.21.15.1:0/3158922557 >> 172.21.15.1:0/3158922557 conn(0x7fb2600f8ae0 msgr2=0x7fb2600faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:42.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.124+0000 7fb266600700 1 -- 172.21.15.1:0/3158922557 shutdown_connections 2024-02-28T17:21:42.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.124+0000 7fb266600700 1 -- 172.21.15.1:0/3158922557 wait complete. 2024-02-28T17:21:42.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.125+0000 7fb266600700 1 Processor -- start 2024-02-28T17:21:42.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.126+0000 7fb266600700 1 -- start start 2024-02-28T17:21:42.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.126+0000 7fb266600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 0x7fb260115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.126+0000 7fb266600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070be0 0x7fb2601161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.126+0000 7fb266600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2601168c0 con 0x7fb2600702d0 2024-02-28T17:21:42.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.126+0000 7fb266600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26010fb40 con 0x7fb260070be0 2024-02-28T17:21:42.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.127+0000 7fb25f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 0x7fb260115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:42.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.127+0000 7fb25f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 0x7fb260115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34394/0 (socket says 172.21.15.1:34394) 2024-02-28T17:21:42.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.127+0000 7fb25f400700 1 -- 172.21.15.1:0/2538123723 learned_addr learned my addr 172.21.15.1:0/2538123723 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:42.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.127+0000 7fb257e00700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070be0 0x7fb2601161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.127+0000 7fb25f400700 1 -- 172.21.15.1:0/2538123723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070be0 msgr2=0x7fb2601161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.127+0000 7fb25f400700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070be0 0x7fb2601161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.127+0000 7fb25f400700 1 -- 172.21.15.1:0/2538123723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250005040 con 0x7fb2600702d0 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.128+0000 7fb25f400700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 0x7fb260115ca0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fb250004f50 tx=0x7fb250009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.128+0000 7fb257400700 1 -- 172.21.15.1:0/2538123723 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb25000e040 con 0x7fb2600702d0 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.128+0000 7fb257400700 1 -- 172.21.15.1:0/2538123723 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb250004290 con 0x7fb2600702d0 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.128+0000 7fb257400700 1 -- 172.21.15.1:0/2538123723 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb25001c630 con 0x7fb2600702d0 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.128+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26010fd60 con 0x7fb2600702d0 2024-02-28T17:21:42.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.128+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb260110250 con 0x7fb2600702d0 2024-02-28T17:21:42.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.129+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb260106980 con 0x7fb2600702d0 2024-02-28T17:21:42.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.130+0000 7fb257400700 1 -- 172.21.15.1:0/2538123723 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb25000bc10 con 0x7fb2600702d0 2024-02-28T17:21:42.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.131+0000 7fb257400700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.131+0000 7fb257400700 1 -- 172.21.15.1:0/2538123723 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb25008e2c0 con 0x7fb2600702d0 2024-02-28T17:21:42.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.132+0000 7fb257e00700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:42.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.132+0000 7fb257e00700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb248006fd0 tx=0x7fb248008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:42.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.137+0000 7fb257400700 1 -- 172.21.15.1:0/2538123723 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb25005b620 con 0x7fb2600702d0 2024-02-28T17:21:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.383+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb260111280 con 0x7fb24406d7f0 2024-02-28T17:21:42.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.385+0000 7fb257400700 1 -- 172.21.15.1:0/2538123723 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb260111280 con 0x7fb24406d7f0 2024-02-28T17:21:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 msgr2=0x7fb24406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb248006fd0 tx=0x7fb248008040 comp rx=0 tx=0).stop 2024-02-28T17:21:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 msgr2=0x7fb260115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 0x7fb260115ca0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fb250004f50 tx=0x7fb250009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 shutdown_connections 2024-02-28T17:21:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb24406d7f0 0x7fb24406fca0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2600702d0 0x7fb260115ca0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 --2- 172.21.15.1:0/2538123723 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb260070be0 0x7fb2601161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 >> 172.21.15.1:0/2538123723 conn(0x7fb2600f8ae0 msgr2=0x7fb260103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 shutdown_connections 2024-02-28T17:21:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.388+0000 7fb266600700 1 -- 172.21.15.1:0/2538123723 wait complete. 2024-02-28T17:21:42.408 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:21:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.548+0000 7f8d49000700 1 -- 172.21.15.1:0/1670054638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 msgr2=0x7f8d44070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.548+0000 7f8d49000700 1 --2- 172.21.15.1:0/1670054638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 0x7f8d44070330 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f8d34004970 tx=0x7f8d34009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.549+0000 7f8d49000700 1 -- 172.21.15.1:0/1670054638 shutdown_connections 2024-02-28T17:21:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.549+0000 7f8d49000700 1 --2- 172.21.15.1:0/1670054638 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d44070870 0x7f8d44070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.549+0000 7f8d49000700 1 --2- 172.21.15.1:0/1670054638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 0x7f8d44070330 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.549+0000 7f8d49000700 1 -- 172.21.15.1:0/1670054638 >> 172.21.15.1:0/1670054638 conn(0x7f8d440f8940 msgr2=0x7f8d440fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.549+0000 7f8d49000700 1 -- 172.21.15.1:0/1670054638 shutdown_connections 2024-02-28T17:21:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.549+0000 7f8d49000700 1 -- 172.21.15.1:0/1670054638 wait complete. 2024-02-28T17:21:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.550+0000 7f8d49000700 1 Processor -- start 2024-02-28T17:21:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.550+0000 7f8d49000700 1 -- start start 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.550+0000 7f8d49000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d44070870 0x7f8d44119e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.550+0000 7f8d49000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 0x7f8d4411a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.550+0000 7f8d49000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4411aa20 con 0x7f8d44071f60 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.550+0000 7f8d49000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d44113d30 con 0x7f8d44070870 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.551+0000 7f8d42a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 0x7f8d4411a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.551+0000 7f8d42a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 0x7f8d4411a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34418/0 (socket says 172.21.15.1:34418) 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.551+0000 7f8d42a00700 1 -- 172.21.15.1:0/634941449 learned_addr learned my addr 172.21.15.1:0/634941449 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.551+0000 7f8d43400700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d44070870 0x7f8d44119e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.551+0000 7f8d42a00700 1 -- 172.21.15.1:0/634941449 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d44070870 msgr2=0x7f8d44119e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.551+0000 7f8d42a00700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d44070870 0x7f8d44119e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.551+0000 7f8d42a00700 1 -- 172.21.15.1:0/634941449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d34005040 con 0x7f8d44071f60 2024-02-28T17:21:42.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.551+0000 7f8d42a00700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 0x7f8d4411a3d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8d30002a50 tx=0x7f8d300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:42.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.552+0000 7f8d3b400700 1 -- 172.21.15.1:0/634941449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d30007cc0 con 0x7f8d44071f60 2024-02-28T17:21:42.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.552+0000 7f8d3b400700 1 -- 172.21.15.1:0/634941449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d3000f040 con 0x7f8d44071f60 2024-02-28T17:21:42.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.552+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d44114010 con 0x7f8d44071f60 2024-02-28T17:21:42.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.552+0000 7f8d3b400700 1 -- 172.21.15.1:0/634941449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d30009640 con 0x7f8d44071f60 2024-02-28T17:21:42.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.552+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d44114550 con 0x7f8d44071f60 2024-02-28T17:21:42.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.553+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d440fe260 con 0x7f8d44071f60 2024-02-28T17:21:42.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.554+0000 7f8d3b400700 1 -- 172.21.15.1:0/634941449 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8d300097a0 con 0x7f8d44071f60 2024-02-28T17:21:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.556+0000 7f8d3b400700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8d2806d7f0 0x7f8d2806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.556+0000 7f8d3b400700 1 -- 172.21.15.1:0/634941449 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8d3008cc60 con 0x7f8d44071f60 2024-02-28T17:21:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.556+0000 7f8d43400700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8d2806d7f0 0x7f8d2806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.557+0000 7f8d43400700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8d2806d7f0 0x7f8d2806fca0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8d34000c00 tx=0x7f8d34009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:42.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.560+0000 7f8d3b400700 1 -- 172.21.15.1:0/634941449 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8d30059fc0 con 0x7f8d44071f60 2024-02-28T17:21:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.799+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d4404bf40 con 0x7f8d2806d7f0 2024-02-28T17:21:42.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.801+0000 7f8d3b400700 1 -- 172.21.15.1:0/634941449 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8d4404bf40 con 0x7f8d2806d7f0 2024-02-28T17:21:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8d2806d7f0 msgr2=0x7f8d2806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8d2806d7f0 0x7f8d2806fca0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8d34000c00 tx=0x7f8d34009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 msgr2=0x7f8d4411a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 0x7f8d4411a3d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8d30002a50 tx=0x7f8d300046b0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 shutdown_connections 2024-02-28T17:21:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f8d2806d7f0 0x7f8d2806fca0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8d44070870 0x7f8d44119e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 --2- 172.21.15.1:0/634941449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d44071f60 0x7f8d4411a3d0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 >> 172.21.15.1:0/634941449 conn(0x7f8d440f8940 msgr2=0x7f8d440fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 shutdown_connections 2024-02-28T17:21:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.804+0000 7f8d49000700 1 -- 172.21.15.1:0/634941449 wait complete. 2024-02-28T17:21:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.948+0000 7fb2f3000700 1 -- 172.21.15.1:0/19939244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec103dc0 msgr2=0x7fb2ec104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.948+0000 7fb2f3000700 1 --2- 172.21.15.1:0/19939244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec103dc0 0x7fb2ec104230 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4004940 tx=0x7fb2d4009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.948+0000 7fb2f3000700 1 -- 172.21.15.1:0/19939244 shutdown_connections 2024-02-28T17:21:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.948+0000 7fb2f3000700 1 --2- 172.21.15.1:0/19939244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec103dc0 0x7fb2ec104230 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.948+0000 7fb2f3000700 1 --2- 172.21.15.1:0/19939244 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2ec1095b0 0x7fb2ec109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.948+0000 7fb2f3000700 1 -- 172.21.15.1:0/19939244 >> 172.21.15.1:0/19939244 conn(0x7fb2ec072d70 msgr2=0x7fb2ec075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.948+0000 7fb2f3000700 1 -- 172.21.15.1:0/19939244 shutdown_connections 2024-02-28T17:21:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.948+0000 7fb2f3000700 1 -- 172.21.15.1:0/19939244 wait complete. 2024-02-28T17:21:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.949+0000 7fb2f3000700 1 Processor -- start 2024-02-28T17:21:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.949+0000 7fb2f3000700 1 -- start start 2024-02-28T17:21:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.949+0000 7fb2f3000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2ec103dc0 0x7fb2ec11a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.949+0000 7fb2f3000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec1095b0 0x7fb2ec11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.949+0000 7fb2f3000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec11ac00 con 0x7fb2ec1095b0 2024-02-28T17:21:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.949+0000 7fb2f3000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec113f10 con 0x7fb2ec103dc0 2024-02-28T17:21:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.950+0000 7fb2e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec1095b0 0x7fb2ec11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:42.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.950+0000 7fb2e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec1095b0 0x7fb2ec11a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34424/0 (socket says 172.21.15.1:34424) 2024-02-28T17:21:42.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.950+0000 7fb2e3e00700 1 -- 172.21.15.1:0/2906218519 learned_addr learned my addr 172.21.15.1:0/2906218519 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:42.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.950+0000 7fb2e3e00700 1 -- 172.21.15.1:0/2906218519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2ec103dc0 msgr2=0x7fb2ec11a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:42.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.950+0000 7fb2e3e00700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2ec103dc0 0x7fb2ec11a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.950+0000 7fb2e3e00700 1 -- 172.21.15.1:0/2906218519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4005040 con 0x7fb2ec1095b0 2024-02-28T17:21:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.950+0000 7fb2e3e00700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec1095b0 0x7fb2ec11a5b0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009fd0 tx=0x7fb2d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.951+0000 7fb2e8c00700 1 -- 172.21.15.1:0/2906218519 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2d400e040 con 0x7fb2ec1095b0 2024-02-28T17:21:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.951+0000 7fb2e8c00700 1 -- 172.21.15.1:0/2906218519 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2d400bbf0 con 0x7fb2ec1095b0 2024-02-28T17:21:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.951+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec114190 con 0x7fb2ec1095b0 2024-02-28T17:21:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.951+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec1146d0 con 0x7fb2ec1095b0 2024-02-28T17:21:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.951+0000 7fb2e8c00700 1 -- 172.21.15.1:0/2906218519 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2d401c3f0 con 0x7fb2ec1095b0 2024-02-28T17:21:42.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.952+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c80052e0 con 0x7fb2ec1095b0 2024-02-28T17:21:42.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.953+0000 7fb2e8c00700 1 -- 172.21.15.1:0/2906218519 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb2d400bd60 con 0x7fb2ec1095b0 2024-02-28T17:21:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.954+0000 7fb2e8c00700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2d006d8a0 0x7fb2d006fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.955+0000 7fb2e8c00700 1 -- 172.21.15.1:0/2906218519 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb2d408e0c0 con 0x7fb2ec1095b0 2024-02-28T17:21:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.955+0000 7fb2ebe00700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2d006d8a0 0x7fb2d006fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:42.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.956+0000 7fb2ebe00700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2d006d8a0 0x7fb2d006fd50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec076220 tx=0x7fb2dc007d60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:42.960+0000 7fb2e8c00700 1 -- 172.21.15.1:0/2906218519 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb2d405b560 con 0x7fb2ec1095b0 2024-02-28T17:21:43.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.190+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2c8000bf0 con 0x7fb2d006d8a0 2024-02-28T17:21:43.195 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:43 smithi001 ceph-mon[112940]: pgmap v1421: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.228+0000 7fb2e8c00700 1 -- 172.21.15.1:0/2906218519 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb2c8000bf0 con 0x7fb2d006d8a0 2024-02-28T17:21:43.232 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:21:43.232 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (47m) 9m ago 62m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:21:43.232 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (44m) 9m ago 62m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:21:43.232 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (44m) 9m ago 60m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:21:43.232 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (47m) 9m ago 61m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (40m) 9m ago 54m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (40m) 9m ago 54m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (9m) 9m ago 54m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (24m) 9m ago 54m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (50m) 9m ago 63m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (49m) 9m ago 60m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (46m) 9m ago 64m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (45m) 9m ago 60m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (48m) 9m ago 62m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (48m) 9m ago 60m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (43m) 9m ago 59m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (43m) 9m ago 59m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (42m) 9m ago 58m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (42m) 9m ago 58m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:21:43.233 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (41m) 9m ago 57m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:21:43.234 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (40m) 9m ago 56m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:21:43.234 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (48m) 9m ago 61m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2d006d8a0 msgr2=0x7fb2d006fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2d006d8a0 0x7fb2d006fd50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec076220 tx=0x7fb2dc007d60 comp rx=0 tx=0).stop 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec1095b0 msgr2=0x7fb2ec11a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec1095b0 0x7fb2ec11a5b0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009fd0 tx=0x7fb2d4009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 shutdown_connections 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb2d006d8a0 0x7fb2d006fd50 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb2ec103dc0 0x7fb2ec11a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 --2- 172.21.15.1:0/2906218519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ec1095b0 0x7fb2ec11a5b0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 >> 172.21.15.1:0/2906218519 conn(0x7fb2ec072d70 msgr2=0x7fb2ec074960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.231+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 shutdown_connections 2024-02-28T17:21:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.232+0000 7fb2f3000700 1 -- 172.21.15.1:0/2906218519 wait complete. 2024-02-28T17:21:43.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:43 smithi066 ceph-mon[85781]: pgmap v1421: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.388+0000 7fe91ac00700 1 -- 172.21.15.1:0/4001851983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914102d90 msgr2=0x7fe914103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.388+0000 7fe91ac00700 1 --2- 172.21.15.1:0/4001851983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914102d90 0x7fe914103160 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fe908004940 tx=0x7fe908009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.388+0000 7fe91ac00700 1 -- 172.21.15.1:0/4001851983 shutdown_connections 2024-02-28T17:21:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.388+0000 7fe91ac00700 1 --2- 172.21.15.1:0/4001851983 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe914103730 0x7fe9140fcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.388+0000 7fe91ac00700 1 --2- 172.21.15.1:0/4001851983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914102d90 0x7fe914103160 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.388+0000 7fe91ac00700 1 -- 172.21.15.1:0/4001851983 >> 172.21.15.1:0/4001851983 conn(0x7fe9140f8940 msgr2=0x7fe9140fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.388+0000 7fe91ac00700 1 -- 172.21.15.1:0/4001851983 shutdown_connections 2024-02-28T17:21:43.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.388+0000 7fe91ac00700 1 -- 172.21.15.1:0/4001851983 wait complete. 2024-02-28T17:21:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.389+0000 7fe91ac00700 1 Processor -- start 2024-02-28T17:21:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.389+0000 7fe91ac00700 1 -- start start 2024-02-28T17:21:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.389+0000 7fe91ac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe914102d90 0x7fe914115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.389+0000 7fe91ac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914103730 0x7fe914115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.389+0000 7fe91ac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9141165e0 con 0x7fe914103730 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.389+0000 7fe91ac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91410f910 con 0x7fe914102d90 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.390+0000 7fe918e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914103730 0x7fe914115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.390+0000 7fe918e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914103730 0x7fe914115f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34448/0 (socket says 172.21.15.1:34448) 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.390+0000 7fe918e00700 1 -- 172.21.15.1:0/3655683701 learned_addr learned my addr 172.21.15.1:0/3655683701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.390+0000 7fe919800700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe914102d90 0x7fe914115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.390+0000 7fe918e00700 1 -- 172.21.15.1:0/3655683701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe914102d90 msgr2=0x7fe914115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.390+0000 7fe918e00700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe914102d90 0x7fe914115a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.390+0000 7fe918e00700 1 -- 172.21.15.1:0/3655683701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe908005040 con 0x7fe914103730 2024-02-28T17:21:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.391+0000 7fe918e00700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914103730 0x7fe914115f90 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fe90000c800 tx=0x7fe900004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.391+0000 7fe911600700 1 -- 172.21.15.1:0/3655683701 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe90000a760 con 0x7fe914103730 2024-02-28T17:21:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.391+0000 7fe911600700 1 -- 172.21.15.1:0/3655683701 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe90000ad90 con 0x7fe914103730 2024-02-28T17:21:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.391+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91410fbf0 con 0x7fe914103730 2024-02-28T17:21:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.391+0000 7fe911600700 1 -- 172.21.15.1:0/3655683701 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9000113a0 con 0x7fe914103730 2024-02-28T17:21:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.391+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9141101b0 con 0x7fe914103730 2024-02-28T17:21:43.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.392+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91405f5c0 con 0x7fe914103730 2024-02-28T17:21:43.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.393+0000 7fe911600700 1 -- 172.21.15.1:0/3655683701 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fe900012400 con 0x7fe914103730 2024-02-28T17:21:43.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.395+0000 7fe911600700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8fc06d7f0 0x7fe8fc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:43.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.395+0000 7fe911600700 1 -- 172.21.15.1:0/3655683701 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe90008ce80 con 0x7fe914103730 2024-02-28T17:21:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.395+0000 7fe919800700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8fc06d7f0 0x7fe8fc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.396+0000 7fe919800700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8fc06d7f0 0x7fe8fc06fca0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe90800b020 tx=0x7fe908009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:43.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.399+0000 7fe911600700 1 -- 172.21.15.1:0/3655683701 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe90005a260 con 0x7fe914103730 2024-02-28T17:21:43.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.708+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe914111030 con 0x7fe914103730 2024-02-28T17:21:43.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.709+0000 7fe911600700 1 -- 172.21.15.1:0/3655683701 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe900059df0 con 0x7fe914103730 2024-02-28T17:21:43.713 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:21:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:21:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:21:43.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:21:43.714 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:21:43.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8fc06d7f0 msgr2=0x7fe8fc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:43.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8fc06d7f0 0x7fe8fc06fca0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe90800b020 tx=0x7fe908009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:43.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914103730 msgr2=0x7fe914115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:43.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914103730 0x7fe914115f90 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fe90000c800 tx=0x7fe900004470 comp rx=0 tx=0).stop 2024-02-28T17:21:43.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 shutdown_connections 2024-02-28T17:21:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fe8fc06d7f0 0x7fe8fc06fca0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fe914102d90 0x7fe914115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 --2- 172.21.15.1:0/3655683701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe914103730 0x7fe914115f90 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.712+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 >> 172.21.15.1:0/3655683701 conn(0x7fe9140f8940 msgr2=0x7fe9140fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.713+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 shutdown_connections 2024-02-28T17:21:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.713+0000 7fe91ac00700 1 -- 172.21.15.1:0/3655683701 wait complete. 2024-02-28T17:21:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.874+0000 7fa1bf800700 1 -- 172.21.15.1:0/286143775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fdf60 msgr2=0x7fa1b80fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.874+0000 7fa1bf800700 1 --2- 172.21.15.1:0/286143775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fdf60 0x7fa1b80fe330 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac004940 tx=0x7fa1ac009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.874+0000 7fa1bf800700 1 -- 172.21.15.1:0/286143775 shutdown_connections 2024-02-28T17:21:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.874+0000 7fa1bf800700 1 --2- 172.21.15.1:0/286143775 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1b80fe900 0x7fa1b8106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.874+0000 7fa1bf800700 1 --2- 172.21.15.1:0/286143775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fdf60 0x7fa1b80fe330 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.874+0000 7fa1bf800700 1 -- 172.21.15.1:0/286143775 >> 172.21.15.1:0/286143775 conn(0x7fa1b80f7a10 msgr2=0x7fa1b80f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:43.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.874+0000 7fa1bf800700 1 -- 172.21.15.1:0/286143775 shutdown_connections 2024-02-28T17:21:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.874+0000 7fa1bf800700 1 -- 172.21.15.1:0/286143775 wait complete. 2024-02-28T17:21:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.875+0000 7fa1bf800700 1 Processor -- start 2024-02-28T17:21:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.875+0000 7fa1bf800700 1 -- start start 2024-02-28T17:21:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.876+0000 7fa1bf800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1b80fdf60 0x7fa1b80fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.876+0000 7fa1bf800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fe900 0x7fa1b80fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.876+0000 7fa1bf800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80fc8d0 con 0x7fa1b80fe900 2024-02-28T17:21:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.876+0000 7fa1bf800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80fca40 con 0x7fa1b80fdf60 2024-02-28T17:21:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.876+0000 7fa1b7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fe900 0x7fa1b80fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.876+0000 7fa1b7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fe900 0x7fa1b80fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34468/0 (socket says 172.21.15.1:34468) 2024-02-28T17:21:43.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.876+0000 7fa1b7e00700 1 -- 172.21.15.1:0/1079175383 learned_addr learned my addr 172.21.15.1:0/1079175383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.876+0000 7fa1bce00700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1b80fdf60 0x7fa1b80fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1b7e00700 1 -- 172.21.15.1:0/1079175383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1b80fdf60 msgr2=0x7fa1b80fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1b7e00700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1b80fdf60 0x7fa1b80fbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:43.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1b7e00700 1 -- 172.21.15.1:0/1079175383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac005040 con 0x7fa1b80fe900 2024-02-28T17:21:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1bce00700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1b80fdf60 0x7fa1b80fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:21:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1b7e00700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fe900 0x7fa1b80fc2d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400c8c0 tx=0x7fa1a40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1b4c00700 1 -- 172.21.15.1:0/1079175383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1a400a980 con 0x7fa1b80fe900 2024-02-28T17:21:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1b4c00700 1 -- 172.21.15.1:0/1079175383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1a400ce60 con 0x7fa1b80fe900 2024-02-28T17:21:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1b4c00700 1 -- 172.21.15.1:0/1079175383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1a40115e0 con 0x7fa1b80fe900 2024-02-28T17:21:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b8119ce0 con 0x7fa1b80fe900 2024-02-28T17:21:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.877+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b811a1c0 con 0x7fa1b80fe900 2024-02-28T17:21:43.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.878+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b8104550 con 0x7fa1b80fe900 2024-02-28T17:21:43.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.879+0000 7fa1b4c00700 1 -- 172.21.15.1:0/1079175383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa1a4012400 con 0x7fa1b80fe900 2024-02-28T17:21:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.880+0000 7fa1b4c00700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1a006d7f0 0x7fa1a006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.880+0000 7fa1b4c00700 1 -- 172.21.15.1:0/1079175383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa1a408cf40 con 0x7fa1b80fe900 2024-02-28T17:21:43.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.881+0000 7fa1bce00700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1a006d7f0 0x7fa1a006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:43.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.881+0000 7fa1bce00700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1a006d7f0 0x7fa1a006fca0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac004f90 tx=0x7fa1ac009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:43.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:43.885+0000 7fa1b4c00700 1 -- 172.21.15.1:0/1079175383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa1a40567f0 con 0x7fa1b80fe900 2024-02-28T17:21:44.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.160+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa1b804bed0 con 0x7fa1b80fe900 2024-02-28T17:21:44.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.161+0000 7fa1b4c00700 1 -- 172.21.15.1:0/1079175383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fa1a40126b0 con 0x7fa1b80fe900 2024-02-28T17:21:44.165 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:21:44.165 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:21:44.165 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:21:44.165 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:21:44.165 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:21:44.165 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:21:44.165 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:21:44.166 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:21:44.167 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:21:44.168 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:21:44.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.163+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1a006d7f0 msgr2=0x7fa1a006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:44.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.163+0000 7fa1bf800700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1a006d7f0 0x7fa1a006fca0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac004f90 tx=0x7fa1ac009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:44.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fe900 msgr2=0x7fa1b80fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:44.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fe900 0x7fa1b80fc2d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400c8c0 tx=0x7fa1a40044c0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 shutdown_connections 2024-02-28T17:21:44.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa1a006d7f0 0x7fa1a006fca0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa1b80fdf60 0x7fa1b80fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 --2- 172.21.15.1:0/1079175383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b80fe900 0x7fa1b80fc2d0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 >> 172.21.15.1:0/1079175383 conn(0x7fa1b80f7a10 msgr2=0x7fa1b81016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:44.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 shutdown_connections 2024-02-28T17:21:44.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.164+0000 7fa1bf800700 1 -- 172.21.15.1:0/1079175383 wait complete. 2024-02-28T17:21:44.170 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:21:44.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:44 smithi001 ceph-mon[112940]: from='client.36496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:44.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:44 smithi001 ceph-mon[112940]: from='client.36500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:44.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:44 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3655683701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:21:44.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.320+0000 7f47a5a00700 1 -- 172.21.15.1:0/271795186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105170 msgr2=0x7f47a0105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:44.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.320+0000 7f47a5a00700 1 --2- 172.21.15.1:0/271795186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105170 0x7f47a0105540 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4790004940 tx=0x7f4790009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:44.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.320+0000 7f47a5a00700 1 -- 172.21.15.1:0/271795186 shutdown_connections 2024-02-28T17:21:44.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.320+0000 7f47a5a00700 1 --2- 172.21.15.1:0/271795186 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47a0105b10 0x7f47a00ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.320+0000 7f47a5a00700 1 --2- 172.21.15.1:0/271795186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105170 0x7f47a0105540 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.320+0000 7f47a5a00700 1 -- 172.21.15.1:0/271795186 >> 172.21.15.1:0/271795186 conn(0x7f47a00face0 msgr2=0x7f47a00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:44.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.320+0000 7f47a5a00700 1 -- 172.21.15.1:0/271795186 shutdown_connections 2024-02-28T17:21:44.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.320+0000 7f47a5a00700 1 -- 172.21.15.1:0/271795186 wait complete. 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.321+0000 7f47a5a00700 1 Processor -- start 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.321+0000 7f47a5a00700 1 -- start start 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.321+0000 7f47a5a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47a0105170 0x7f47a0115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.321+0000 7f47a5a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105b10 0x7f47a01161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.321+0000 7f47a5a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01167f0 con 0x7f47a0105b10 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.321+0000 7f47a5a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a010fb40 con 0x7f47a0105170 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.322+0000 7f479e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105b10 0x7f47a01161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.322+0000 7f479e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105b10 0x7f47a01161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34486/0 (socket says 172.21.15.1:34486) 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.322+0000 7f479e000700 1 -- 172.21.15.1:0/990360635 learned_addr learned my addr 172.21.15.1:0/990360635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.322+0000 7f479ea00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47a0105170 0x7f47a0115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.322+0000 7f479e000700 1 -- 172.21.15.1:0/990360635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47a0105170 msgr2=0x7f47a0115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.322+0000 7f479e000700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47a0105170 0x7f47a0115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.322+0000 7f479e000700 1 -- 172.21.15.1:0/990360635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4790005040 con 0x7f47a0105b10 2024-02-28T17:21:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.323+0000 7f479e000700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105b10 0x7f47a01161a0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f478c00c800 tx=0x7f478c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.323+0000 7f4796a00700 1 -- 172.21.15.1:0/990360635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f478c00a760 con 0x7f47a0105b10 2024-02-28T17:21:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.323+0000 7f4796a00700 1 -- 172.21.15.1:0/990360635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f478c00ad90 con 0x7f47a0105b10 2024-02-28T17:21:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.323+0000 7f4796a00700 1 -- 172.21.15.1:0/990360635 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f478c0113a0 con 0x7f47a0105b10 2024-02-28T17:21:44.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.323+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a010fe20 con 0x7f47a0105b10 2024-02-28T17:21:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.323+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a0110360 con 0x7f47a0105b10 2024-02-28T17:21:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.324+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a0102480 con 0x7f47a0105b10 2024-02-28T17:21:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.325+0000 7f4796a00700 1 -- 172.21.15.1:0/990360635 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f478c012400 con 0x7f47a0105b10 2024-02-28T17:21:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.326+0000 7f4796a00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f478406d7f0 0x7f478406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.326+0000 7f4796a00700 1 -- 172.21.15.1:0/990360635 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f478c08ce80 con 0x7f47a0105b10 2024-02-28T17:21:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.326+0000 7f479ea00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f478406d7f0 0x7f478406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:44.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.327+0000 7f479ea00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f478406d7f0 0x7f478406fca0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f479000b020 tx=0x7f4790009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.331+0000 7f4796a00700 1 -- 172.21.15.1:0/990360635 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f478c056730 con 0x7f47a0105b10 2024-02-28T17:21:44.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:44 smithi066 ceph-mon[85781]: from='client.36496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:44.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:44 smithi066 ceph-mon[85781]: from='client.36500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:44.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:44 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3655683701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:21:44.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.574+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f47a004bf40 con 0x7f478406d7f0 2024-02-28T17:21:44.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.577+0000 7f4796a00700 1 -- 172.21.15.1:0/990360635 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f47a004bf40 con 0x7f478406d7f0 2024-02-28T17:21:44.581 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:21:44.582 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:21:44.583 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:21:44.583 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:21:44.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f478406d7f0 msgr2=0x7f478406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:44.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f478406d7f0 0x7f478406fca0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f479000b020 tx=0x7f4790009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:44.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105b10 msgr2=0x7f47a01161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:44.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105b10 0x7f47a01161a0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f478c00c800 tx=0x7f478c004470 comp rx=0 tx=0).stop 2024-02-28T17:21:44.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 shutdown_connections 2024-02-28T17:21:44.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f478406d7f0 0x7f478406fca0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f47a0105170 0x7f47a0115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 --2- 172.21.15.1:0/990360635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47a0105b10 0x7f47a01161a0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.580+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 >> 172.21.15.1:0/990360635 conn(0x7f47a00face0 msgr2=0x7f47a00fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:44.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.581+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 shutdown_connections 2024-02-28T17:21:44.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.581+0000 7f47a5a00700 1 -- 172.21.15.1:0/990360635 wait complete. 2024-02-28T17:21:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.744+0000 7fd17e600700 1 -- 172.21.15.1:0/2736476584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 msgr2=0x7fd1780fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.744+0000 7fd17e600700 1 --2- 172.21.15.1:0/2736476584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 0x7fd1780fcf60 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fd164004940 tx=0x7fd164009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:44.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.744+0000 7fd17e600700 1 -- 172.21.15.1:0/2736476584 shutdown_connections 2024-02-28T17:21:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.744+0000 7fd17e600700 1 --2- 172.21.15.1:0/2736476584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 0x7fd1780fcf60 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.744+0000 7fd17e600700 1 --2- 172.21.15.1:0/2736476584 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd178102f50 0x7fd178103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.744+0000 7fd17e600700 1 -- 172.21.15.1:0/2736476584 >> 172.21.15.1:0/2736476584 conn(0x7fd1780f8ae0 msgr2=0x7fd1780faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.744+0000 7fd17e600700 1 -- 172.21.15.1:0/2736476584 shutdown_connections 2024-02-28T17:21:44.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.744+0000 7fd17e600700 1 -- 172.21.15.1:0/2736476584 wait complete. 2024-02-28T17:21:44.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.745+0000 7fd17e600700 1 Processor -- start 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.746+0000 7fd17e600700 1 -- start start 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.746+0000 7fd17e600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd178102f50 0x7fd178115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.746+0000 7fd17e600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 0x7fd1781161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.746+0000 7fd17e600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd178116830 con 0x7fd1781038f0 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.746+0000 7fd17e600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17810fb40 con 0x7fd178102f50 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.746+0000 7fd176a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 0x7fd1781161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.746+0000 7fd176a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 0x7fd1781161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34498/0 (socket says 172.21.15.1:34498) 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.746+0000 7fd176a00700 1 -- 172.21.15.1:0/4101828242 learned_addr learned my addr 172.21.15.1:0/4101828242 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.747+0000 7fd177400700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd178102f50 0x7fd178115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.747+0000 7fd176a00700 1 -- 172.21.15.1:0/4101828242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd178102f50 msgr2=0x7fd178115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.747+0000 7fd176a00700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd178102f50 0x7fd178115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:44.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.747+0000 7fd176a00700 1 -- 172.21.15.1:0/4101828242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164005040 con 0x7fd1781038f0 2024-02-28T17:21:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.748+0000 7fd176a00700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 0x7fd1781161e0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fd164009100 tx=0x7fd164009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.748+0000 7fd16f400700 1 -- 172.21.15.1:0/4101828242 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd16400e040 con 0x7fd1781038f0 2024-02-28T17:21:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.748+0000 7fd16f400700 1 -- 172.21.15.1:0/4101828242 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd164004290 con 0x7fd1781038f0 2024-02-28T17:21:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.748+0000 7fd16f400700 1 -- 172.21.15.1:0/4101828242 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd16401c3f0 con 0x7fd1781038f0 2024-02-28T17:21:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.748+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17810fdc0 con 0x7fd1781038f0 2024-02-28T17:21:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.748+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd178110300 con 0x7fd1781038f0 2024-02-28T17:21:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.750+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1780fc6e0 con 0x7fd1781038f0 2024-02-28T17:21:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.752+0000 7fd16f400700 1 -- 172.21.15.1:0/4101828242 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd16400bbf0 con 0x7fd1781038f0 2024-02-28T17:21:44.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.753+0000 7fd16f400700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd15c06d7f0 0x7fd15c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:21:44.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.753+0000 7fd16f400700 1 -- 172.21.15.1:0/4101828242 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd16408ec00 con 0x7fd1781038f0 2024-02-28T17:21:44.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.753+0000 7fd177400700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd15c06d7f0 0x7fd15c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:21:44.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.754+0000 7fd177400700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd15c06d7f0 0x7fd15c06fca0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd168009fd0 tx=0x7fd168009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:21:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:44.757+0000 7fd16f400700 1 -- 172.21.15.1:0/4101828242 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd16405c0a0 con 0x7fd1781038f0 2024-02-28T17:21:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.067+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd17810a050 con 0x7fd1781038f0 2024-02-28T17:21:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.068+0000 7fd16f400700 1 -- 172.21.15.1:0/4101828242 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fd16405bc30 con 0x7fd1781038f0 2024-02-28T17:21:45.072 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:21:45.072 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:21:45.072 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:21:45.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:21:45.072 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:21:45.073 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:21:45.073 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:21:45.073 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:21:45.073 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:21:45.073 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:21:45.073 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:21:45.074 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:21:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:21:45.075 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:21:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:21:45.075 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:21:45.075 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:21:45.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.070+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd15c06d7f0 msgr2=0x7fd15c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:45.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.070+0000 7fd17e600700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd15c06d7f0 0x7fd15c06fca0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd168009fd0 tx=0x7fd168009f20 comp rx=0 tx=0).stop 2024-02-28T17:21:45.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.070+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 msgr2=0x7fd1781161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:21:45.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.070+0000 7fd17e600700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 0x7fd1781161e0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fd164009100 tx=0x7fd164009e30 comp rx=0 tx=0).stop 2024-02-28T17:21:45.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.071+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 shutdown_connections 2024-02-28T17:21:45.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.071+0000 7fd17e600700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd15c06d7f0 0x7fd15c06fca0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:45.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.071+0000 7fd17e600700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd178102f50 0x7fd178115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:45.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.071+0000 7fd17e600700 1 --2- 172.21.15.1:0/4101828242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1781038f0 0x7fd1781161e0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:21:45.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.071+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 >> 172.21.15.1:0/4101828242 conn(0x7fd1780f8ae0 msgr2=0x7fd1780fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:21:45.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.071+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 shutdown_connections 2024-02-28T17:21:45.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:21:45.071+0000 7fd17e600700 1 -- 172.21.15.1:0/4101828242 wait complete. 2024-02-28T17:21:45.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:45 smithi001 ceph-mon[112940]: from='client.36504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:45.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:45 smithi001 ceph-mon[112940]: pgmap v1422: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:45.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:45 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1079175383' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:21:45.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:45 smithi066 ceph-mon[85781]: from='client.36504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:45.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:45 smithi066 ceph-mon[85781]: pgmap v1422: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:45.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:45 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1079175383' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:21:46.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:46 smithi001 ceph-mon[112940]: from='client.36516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:46.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:46 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4101828242' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:21:46.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:46 smithi066 ceph-mon[85781]: from='client.36516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:21:46.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:46 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4101828242' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:21:47.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:47 smithi001 ceph-mon[112940]: pgmap v1423: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:47.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:47 smithi066 ceph-mon[85781]: pgmap v1423: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:49.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:49 smithi066 ceph-mon[85781]: pgmap v1424: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:49.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:49 smithi001 ceph-mon[112940]: pgmap v1424: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:51.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:51 smithi066 ceph-mon[85781]: pgmap v1425: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:51.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:51 smithi001 ceph-mon[112940]: pgmap v1425: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:53.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:53 smithi066 ceph-mon[85781]: pgmap v1426: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:53.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:53 smithi001 ceph-mon[112940]: pgmap v1426: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:55.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:55 smithi066 ceph-mon[85781]: pgmap v1427: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:55.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:55 smithi001 ceph-mon[112940]: pgmap v1427: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:56.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:21:56.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:21:57.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:57 smithi066 ceph-mon[85781]: pgmap v1428: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:57.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:57 smithi001 ceph-mon[112940]: pgmap v1428: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:59.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:21:59 smithi066 ceph-mon[85781]: pgmap v1429: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:21:59.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:21:59 smithi001 ceph-mon[112940]: pgmap v1429: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:00.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:00 smithi066 ceph-mon[85781]: pgmap v1430: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:00.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:00 smithi001 ceph-mon[112940]: pgmap v1430: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:03.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:02 smithi066 ceph-mon[85781]: pgmap v1431: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:02 smithi001 ceph-mon[112940]: pgmap v1431: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:05.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:04 smithi066 ceph-mon[85781]: pgmap v1432: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:05.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:04 smithi001 ceph-mon[112940]: pgmap v1432: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:07.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:06 smithi066 ceph-mon[85781]: pgmap v1433: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:07.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:06 smithi001 ceph-mon[112940]: pgmap v1433: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:09.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:08 smithi066 ceph-mon[85781]: pgmap v1434: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:09.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:08 smithi001 ceph-mon[112940]: pgmap v1434: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:11.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:10 smithi066 ceph-mon[85781]: pgmap v1435: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:11.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:10 smithi001 ceph-mon[112940]: pgmap v1435: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:12.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:22:12.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:22:13.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:12 smithi066 ceph-mon[85781]: pgmap v1436: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:13.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:12 smithi001 ceph-mon[112940]: pgmap v1436: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:15.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:14 smithi066 ceph-mon[85781]: pgmap v1437: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:15.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:14 smithi001 ceph-mon[112940]: pgmap v1437: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:15.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.239+0000 7f22b2200700 1 -- 172.21.15.1:0/2446762346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac1095f0 msgr2=0x7f22ac1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:15.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.239+0000 7f22b2200700 1 --2- 172.21.15.1:0/2446762346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac1095f0 0x7f22ac1099c0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f229c004940 tx=0x7f229c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.240+0000 7f22b2200700 1 -- 172.21.15.1:0/2446762346 shutdown_connections 2024-02-28T17:22:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.240+0000 7f22b2200700 1 --2- 172.21.15.1:0/2446762346 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22ac103dc0 0x7f22ac104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.240+0000 7f22b2200700 1 --2- 172.21.15.1:0/2446762346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac1095f0 0x7f22ac1099c0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.240+0000 7f22b2200700 1 -- 172.21.15.1:0/2446762346 >> 172.21.15.1:0/2446762346 conn(0x7f22ac072d50 msgr2=0x7f22ac075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.240+0000 7f22b2200700 1 -- 172.21.15.1:0/2446762346 shutdown_connections 2024-02-28T17:22:15.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.240+0000 7f22b2200700 1 -- 172.21.15.1:0/2446762346 wait complete. 2024-02-28T17:22:15.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.241+0000 7f22b2200700 1 Processor -- start 2024-02-28T17:22:15.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.241+0000 7f22b2200700 1 -- start start 2024-02-28T17:22:15.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.241+0000 7f22b2200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac103dc0 0x7f22ac11a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.241+0000 7f22b2200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22ac11a740 0x7f22ac1140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.241+0000 7f22b2200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac11ad50 con 0x7f22ac103dc0 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.241+0000 7f22b2200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac1145e0 con 0x7f22ac11a740 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.241+0000 7f22ab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac103dc0 0x7f22ac11a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.242+0000 7f22ab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac103dc0 0x7f22ac11a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48402/0 (socket says 172.21.15.1:48402) 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.242+0000 7f22ab400700 1 -- 172.21.15.1:0/2046911560 learned_addr learned my addr 172.21.15.1:0/2046911560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.242+0000 7f22ab400700 1 -- 172.21.15.1:0/2046911560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22ac11a740 msgr2=0x7f22ac1140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.242+0000 7f22aaa00700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22ac11a740 0x7f22ac1140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.242+0000 7f22ab400700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22ac11a740 0x7f22ac1140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.242+0000 7f22ab400700 1 -- 172.21.15.1:0/2046911560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c005040 con 0x7f22ac103dc0 2024-02-28T17:22:15.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.243+0000 7f22ab400700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac103dc0 0x7f22ac11a200 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f229c000c00 tx=0x7f229c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:15.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.243+0000 7f22a3400700 1 -- 172.21.15.1:0/2046911560 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f229c00e040 con 0x7f22ac103dc0 2024-02-28T17:22:15.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.243+0000 7f22a3400700 1 -- 172.21.15.1:0/2046911560 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f229c009430 con 0x7f22ac103dc0 2024-02-28T17:22:15.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.243+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ac114860 con 0x7f22ac103dc0 2024-02-28T17:22:15.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.243+0000 7f22a3400700 1 -- 172.21.15.1:0/2046911560 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f229c01c630 con 0x7f22ac103dc0 2024-02-28T17:22:15.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.243+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ac114da0 con 0x7f22ac103dc0 2024-02-28T17:22:15.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.244+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ac106850 con 0x7f22ac103dc0 2024-02-28T17:22:15.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.245+0000 7f22a3400700 1 -- 172.21.15.1:0/2046911560 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f229c00bbf0 con 0x7f22ac103dc0 2024-02-28T17:22:15.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.246+0000 7f22a3400700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f229006d7f0 0x7f229006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:15.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.246+0000 7f22a3400700 1 -- 172.21.15.1:0/2046911560 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f229c08e2b0 con 0x7f22ac103dc0 2024-02-28T17:22:15.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.247+0000 7f22aaa00700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f229006d7f0 0x7f229006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:15.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.247+0000 7f22aaa00700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f229006d7f0 0x7f229006fca0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2298006fd0 tx=0x7f2298008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:15.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.251+0000 7f22a3400700 1 -- 172.21.15.1:0/2046911560 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f229c05b690 con 0x7f22ac103dc0 2024-02-28T17:22:15.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.498+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f22ac110710 con 0x7f229006d7f0 2024-02-28T17:22:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.501+0000 7f22a3400700 1 -- 172.21.15.1:0/2046911560 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f22ac110710 con 0x7f229006d7f0 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f229006d7f0 msgr2=0x7f229006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f229006d7f0 0x7f229006fca0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2298006fd0 tx=0x7f2298008040 comp rx=0 tx=0).stop 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac103dc0 msgr2=0x7f22ac11a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac103dc0 0x7f22ac11a200 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f229c000c00 tx=0x7f229c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 shutdown_connections 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f229006d7f0 0x7f229006fca0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac103dc0 0x7f22ac11a200 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 --2- 172.21.15.1:0/2046911560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f22ac11a740 0x7f22ac1140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 >> 172.21.15.1:0/2046911560 conn(0x7f22ac072d50 msgr2=0x7f22ac074a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 shutdown_connections 2024-02-28T17:22:15.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.504+0000 7f22b2200700 1 -- 172.21.15.1:0/2046911560 wait complete. 2024-02-28T17:22:15.524 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:22:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.658+0000 7f85fde00700 1 -- 172.21.15.1:0/2477347381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f8102f50 msgr2=0x7f85f8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.658+0000 7f85fde00700 1 --2- 172.21.15.1:0/2477347381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f8102f50 0x7f85f8103320 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f85e8004940 tx=0x7f85e8009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.658+0000 7f85fde00700 1 -- 172.21.15.1:0/2477347381 shutdown_connections 2024-02-28T17:22:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.658+0000 7f85fde00700 1 --2- 172.21.15.1:0/2477347381 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f85f81038f0 0x7f85f80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.658+0000 7f85fde00700 1 --2- 172.21.15.1:0/2477347381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f8102f50 0x7f85f8103320 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.658+0000 7f85fde00700 1 -- 172.21.15.1:0/2477347381 >> 172.21.15.1:0/2477347381 conn(0x7f85f80f8ae0 msgr2=0x7f85f80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.658+0000 7f85fde00700 1 -- 172.21.15.1:0/2477347381 shutdown_connections 2024-02-28T17:22:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.658+0000 7f85fde00700 1 -- 172.21.15.1:0/2477347381 wait complete. 2024-02-28T17:22:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.659+0000 7f85fde00700 1 Processor -- start 2024-02-28T17:22:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.659+0000 7f85fde00700 1 -- start start 2024-02-28T17:22:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85fde00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f85f8102f50 0x7f85f8113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85fde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f81038f0 0x7f85f8113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85fde00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f81145e0 con 0x7f85f81038f0 2024-02-28T17:22:15.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85fde00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f810d910 con 0x7f85f8102f50 2024-02-28T17:22:15.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85ee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f81038f0 0x7f85f8113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:15.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85ee000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f81038f0 0x7f85f8113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48422/0 (socket says 172.21.15.1:48422) 2024-02-28T17:22:15.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85ee000700 1 -- 172.21.15.1:0/3742442887 learned_addr learned my addr 172.21.15.1:0/3742442887 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85f6a00700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f85f8102f50 0x7f85f8113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85ee000700 1 -- 172.21.15.1:0/3742442887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f85f8102f50 msgr2=0x7f85f8113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85ee000700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f85f8102f50 0x7f85f8113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.660+0000 7f85ee000700 1 -- 172.21.15.1:0/3742442887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8005040 con 0x7f85f81038f0 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.661+0000 7f85ee000700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f81038f0 0x7f85f8113f90 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f85e000c800 tx=0x7f85e0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.661+0000 7f85ef400700 1 -- 172.21.15.1:0/3742442887 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85e000a760 con 0x7f85f81038f0 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.661+0000 7f85ef400700 1 -- 172.21.15.1:0/3742442887 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85e000ad90 con 0x7f85f81038f0 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.661+0000 7f85ef400700 1 -- 172.21.15.1:0/3742442887 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85e00113a0 con 0x7f85f81038f0 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.661+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f810dbf0 con 0x7f85f81038f0 2024-02-28T17:22:15.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.661+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f810e130 con 0x7f85f81038f0 2024-02-28T17:22:15.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.662+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f80fc6e0 con 0x7f85f81038f0 2024-02-28T17:22:15.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.663+0000 7f85ef400700 1 -- 172.21.15.1:0/3742442887 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f85e0012400 con 0x7f85f81038f0 2024-02-28T17:22:15.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.664+0000 7f85ef400700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85dc06d7e0 0x7f85dc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:15.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.664+0000 7f85ef400700 1 -- 172.21.15.1:0/3742442887 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f85e008ce80 con 0x7f85f81038f0 2024-02-28T17:22:15.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.664+0000 7f85f6a00700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85dc06d7e0 0x7f85dc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:15.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.665+0000 7f85f6a00700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85dc06d7e0 0x7f85dc06fc90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f85e8004f90 tx=0x7f85e8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:15.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.669+0000 7f85ef400700 1 -- 172.21.15.1:0/3742442887 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f85e005a260 con 0x7f85f81038f0 2024-02-28T17:22:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.916+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f85f8113720 con 0x7f85dc06d7e0 2024-02-28T17:22:15.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.919+0000 7f85ef400700 1 -- 172.21.15.1:0/3742442887 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f85f8113720 con 0x7f85dc06d7e0 2024-02-28T17:22:15.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.921+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85dc06d7e0 msgr2=0x7f85dc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:15.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.921+0000 7f85fde00700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85dc06d7e0 0x7f85dc06fc90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f85e8004f90 tx=0x7f85e8009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:15.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.921+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f81038f0 msgr2=0x7f85f8113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:15.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.921+0000 7f85fde00700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f81038f0 0x7f85f8113f90 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f85e000c800 tx=0x7f85e0004470 comp rx=0 tx=0).stop 2024-02-28T17:22:15.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.922+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 shutdown_connections 2024-02-28T17:22:15.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.922+0000 7f85fde00700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f85dc06d7e0 0x7f85dc06fc90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.922+0000 7f85fde00700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f85f8102f50 0x7f85f8113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.922+0000 7f85fde00700 1 --2- 172.21.15.1:0/3742442887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85f81038f0 0x7f85f8113f90 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:15.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.922+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 >> 172.21.15.1:0/3742442887 conn(0x7f85f80f8ae0 msgr2=0x7f85f80fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:15.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.922+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 shutdown_connections 2024-02-28T17:22:15.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:15.922+0000 7f85fde00700 1 -- 172.21.15.1:0/3742442887 wait complete. 2024-02-28T17:22:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.076+0000 7efc0b000700 1 -- 172.21.15.1:0/592238616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105b10 msgr2=0x7efc040ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.076+0000 7efc0b000700 1 --2- 172.21.15.1:0/592238616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105b10 0x7efc040ff180 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7efbf0004940 tx=0x7efbf0009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.076+0000 7efc0b000700 1 -- 172.21.15.1:0/592238616 shutdown_connections 2024-02-28T17:22:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.076+0000 7efc0b000700 1 --2- 172.21.15.1:0/592238616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105b10 0x7efc040ff180 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.076+0000 7efc0b000700 1 --2- 172.21.15.1:0/592238616 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc04105170 0x7efc04105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.076+0000 7efc0b000700 1 -- 172.21.15.1:0/592238616 >> 172.21.15.1:0/592238616 conn(0x7efc040face0 msgr2=0x7efc040fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.076+0000 7efc0b000700 1 -- 172.21.15.1:0/592238616 shutdown_connections 2024-02-28T17:22:16.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.076+0000 7efc0b000700 1 -- 172.21.15.1:0/592238616 wait complete. 2024-02-28T17:22:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.077+0000 7efc0b000700 1 Processor -- start 2024-02-28T17:22:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.077+0000 7efc0b000700 1 -- start start 2024-02-28T17:22:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.077+0000 7efc0b000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105170 0x7efc04115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.077+0000 7efc0b000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc04105b10 0x7efc04116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.077+0000 7efc0b000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc04116870 con 0x7efc04105170 2024-02-28T17:22:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.077+0000 7efc0b000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0410fb80 con 0x7efc04105b10 2024-02-28T17:22:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.078+0000 7efc03e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105170 0x7efc04115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.078+0000 7efc03e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105170 0x7efc04115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48438/0 (socket says 172.21.15.1:48438) 2024-02-28T17:22:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.078+0000 7efc03e00700 1 -- 172.21.15.1:0/1461461519 learned_addr learned my addr 172.21.15.1:0/1461461519 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.078+0000 7efc03400700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc04105b10 0x7efc04116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.078+0000 7efc03e00700 1 -- 172.21.15.1:0/1461461519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc04105b10 msgr2=0x7efc04116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.078+0000 7efc03e00700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc04105b10 0x7efc04116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.078+0000 7efc03e00700 1 -- 172.21.15.1:0/1461461519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0005040 con 0x7efc04105170 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.079+0000 7efc03e00700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105170 0x7efc04115ce0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c830 tx=0x7efbf4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.079+0000 7efbfbe00700 1 -- 172.21.15.1:0/1461461519 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbf400a760 con 0x7efc04105170 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.079+0000 7efbfbe00700 1 -- 172.21.15.1:0/1461461519 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbf400ad90 con 0x7efc04105170 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.079+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0410fe60 con 0x7efc04105170 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.079+0000 7efbfbe00700 1 -- 172.21.15.1:0/1461461519 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbf40113a0 con 0x7efc04105170 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.079+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc041103a0 con 0x7efc04105170 2024-02-28T17:22:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.080+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc04107640 con 0x7efc04105170 2024-02-28T17:22:16.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.081+0000 7efbfbe00700 1 -- 172.21.15.1:0/1461461519 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7efbf4012400 con 0x7efc04105170 2024-02-28T17:22:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.082+0000 7efbfbe00700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efbe806d7f0 0x7efbe806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.082+0000 7efbfbe00700 1 -- 172.21.15.1:0/1461461519 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efbf408cf00 con 0x7efc04105170 2024-02-28T17:22:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.083+0000 7efc03400700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efbe806d7f0 0x7efbe806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.083+0000 7efc03400700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efbe806d7f0 0x7efbe806fca0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7efc04111580 tx=0x7efbf0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.087+0000 7efbfbe00700 1 -- 172.21.15.1:0/1461461519 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efbf405a2e0 con 0x7efc04105170 2024-02-28T17:22:16.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.318+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efc041112c0 con 0x7efbe806d7f0 2024-02-28T17:22:16.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.348+0000 7efbfbe00700 1 -- 172.21.15.1:0/1461461519 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7efc041112c0 con 0x7efbe806d7f0 2024-02-28T17:22:16.353 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:22:16.353 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (48m) 9m ago 62m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (45m) 9m ago 62m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (44m) 9m ago 61m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (47m) 9m ago 62m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (41m) 9m ago 55m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (40m) 9m ago 55m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (10m) 9m ago 55m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (25m) 9m ago 55m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (50m) 9m ago 64m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (49m) 9m ago 61m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (46m) 9m ago 64m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (46m) 9m ago 61m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (49m) 9m ago 62m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (49m) 9m ago 61m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:22:16.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (44m) 9m ago 60m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:22:16.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (43m) 9m ago 59m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:22:16.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (43m) 9m ago 59m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:22:16.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (42m) 9m ago 58m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:22:16.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (42m) 9m ago 57m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:22:16.355 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (41m) 9m ago 57m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:22:16.355 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (48m) 9m ago 61m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:22:16.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.352+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efbe806d7f0 msgr2=0x7efbe806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.352+0000 7efc0b000700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efbe806d7f0 0x7efbe806fca0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7efc04111580 tx=0x7efbf0009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:16.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.352+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105170 msgr2=0x7efc04115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.352+0000 7efc0b000700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105170 0x7efc04115ce0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c830 tx=0x7efbf4004470 comp rx=0 tx=0).stop 2024-02-28T17:22:16.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.353+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 shutdown_connections 2024-02-28T17:22:16.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.353+0000 7efc0b000700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7efbe806d7f0 0x7efbe806fca0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.353+0000 7efc0b000700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc04105170 0x7efc04115ce0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.353+0000 7efc0b000700 1 --2- 172.21.15.1:0/1461461519 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7efc04105b10 0x7efc04116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.353+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 >> 172.21.15.1:0/1461461519 conn(0x7efc040face0 msgr2=0x7efc040fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:16.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.353+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 shutdown_connections 2024-02-28T17:22:16.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.353+0000 7efc0b000700 1 -- 172.21.15.1:0/1461461519 wait complete. 2024-02-28T17:22:16.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.503+0000 7f114f200700 1 -- 172.21.15.1:0/2584938333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 msgr2=0x7f11501033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.503+0000 7f114f200700 1 --2- 172.21.15.1:0/2584938333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 0x7f11501033a0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f1138004940 tx=0x7f1138009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:16.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.503+0000 7f114f200700 1 -- 172.21.15.1:0/2584938333 shutdown_connections 2024-02-28T17:22:16.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.503+0000 7f114f200700 1 --2- 172.21.15.1:0/2584938333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 0x7f11501033a0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.503+0000 7f114f200700 1 --2- 172.21.15.1:0/2584938333 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1150109380 0x7f1150109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.503+0000 7f114f200700 1 -- 172.21.15.1:0/2584938333 >> 172.21.15.1:0/2584938333 conn(0x7f1150072b50 msgr2=0x7f1150074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:16.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.503+0000 7f114f200700 1 -- 172.21.15.1:0/2584938333 shutdown_connections 2024-02-28T17:22:16.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.503+0000 7f114f200700 1 -- 172.21.15.1:0/2584938333 wait complete. 2024-02-28T17:22:16.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.504+0000 7f114f200700 1 Processor -- start 2024-02-28T17:22:16.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.504+0000 7f114f200700 1 -- start start 2024-02-28T17:22:16.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.504+0000 7f114f200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1150109380 0x7f1150119de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.504+0000 7f114f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 0x7f115011a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.504+0000 7f114f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115011aa00 con 0x7f1150109d20 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.504+0000 7f114f200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150113cc0 con 0x7f1150109380 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.505+0000 7f1147e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 0x7f115011a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.505+0000 7f1147e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 0x7f115011a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48458/0 (socket says 172.21.15.1:48458) 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.505+0000 7f1147e00700 1 -- 172.21.15.1:0/3016578599 learned_addr learned my addr 172.21.15.1:0/3016578599 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.505+0000 7f114de00700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1150109380 0x7f1150119de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.505+0000 7f1147e00700 1 -- 172.21.15.1:0/3016578599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1150109380 msgr2=0x7f1150119de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.505+0000 7f1147e00700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1150109380 0x7f1150119de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.505+0000 7f1147e00700 1 -- 172.21.15.1:0/3016578599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1138005040 con 0x7f1150109d20 2024-02-28T17:22:16.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.506+0000 7f1147e00700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 0x7f115011a320 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f1138009030 tx=0x7f1138009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:16.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.506+0000 7f1146000700 1 -- 172.21.15.1:0/3016578599 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f113800e040 con 0x7f1150109d20 2024-02-28T17:22:16.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.506+0000 7f1146000700 1 -- 172.21.15.1:0/3016578599 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f113800bbf0 con 0x7f1150109d20 2024-02-28T17:22:16.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.506+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1150113f40 con 0x7f1150109d20 2024-02-28T17:22:16.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.506+0000 7f1146000700 1 -- 172.21.15.1:0/3016578599 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f113801c630 con 0x7f1150109d20 2024-02-28T17:22:16.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.506+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1150114500 con 0x7f1150109d20 2024-02-28T17:22:16.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.507+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115005f5c0 con 0x7f1150109d20 2024-02-28T17:22:16.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.508+0000 7f1146000700 1 -- 172.21.15.1:0/3016578599 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f113800bd60 con 0x7f1150109d20 2024-02-28T17:22:16.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.509+0000 7f1146000700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f113406d7f0 0x7f113406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.509+0000 7f1146000700 1 -- 172.21.15.1:0/3016578599 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f113808e130 con 0x7f1150109d20 2024-02-28T17:22:16.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.509+0000 7f114de00700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f113406d7f0 0x7f113406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.510+0000 7f114de00700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f113406d7f0 0x7f113406fca0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1140006fd0 tx=0x7f1140008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:16.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.514+0000 7f1146000700 1 -- 172.21.15.1:0/3016578599 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1138057aa0 con 0x7f1150109d20 2024-02-28T17:22:16.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.815+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1150115400 con 0x7f1150109d20 2024-02-28T17:22:16.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.816+0000 7f1146000700 1 -- 172.21.15.1:0/3016578599 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f113805b160 con 0x7f1150109d20 2024-02-28T17:22:16.821 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:22:16.821 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:22:16.821 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:22:16.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:22:16.821 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:22:16.821 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:22:16.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:22:16.822 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:22:16.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f113406d7f0 msgr2=0x7f113406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f113406d7f0 0x7f113406fca0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1140006fd0 tx=0x7f1140008040 comp rx=0 tx=0).stop 2024-02-28T17:22:16.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 msgr2=0x7f115011a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 0x7f115011a320 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f1138009030 tx=0x7f1138009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:16.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 shutdown_connections 2024-02-28T17:22:16.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f113406d7f0 0x7f113406fca0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1150109380 0x7f1150119de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 --2- 172.21.15.1:0/3016578599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150109d20 0x7f115011a320 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 >> 172.21.15.1:0/3016578599 conn(0x7f1150072b50 msgr2=0x7f1150075580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:16.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.819+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 shutdown_connections 2024-02-28T17:22:16.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.820+0000 7f114f200700 1 -- 172.21.15.1:0/3016578599 wait complete. 2024-02-28T17:22:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.980+0000 7f2280c00700 1 -- 172.21.15.1:0/1625309534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070330 msgr2=0x7f227c070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.980+0000 7f2280c00700 1 --2- 172.21.15.1:0/1625309534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070330 0x7f227c070700 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f226c004970 tx=0x7f226c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.980+0000 7f2280c00700 1 -- 172.21.15.1:0/1625309534 shutdown_connections 2024-02-28T17:22:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.980+0000 7f2280c00700 1 --2- 172.21.15.1:0/1625309534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f227c070c40 0x7f227c109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.980+0000 7f2280c00700 1 --2- 172.21.15.1:0/1625309534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070330 0x7f227c070700 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.980+0000 7f2280c00700 1 -- 172.21.15.1:0/1625309534 >> 172.21.15.1:0/1625309534 conn(0x7f227c0f8ac0 msgr2=0x7f227c0faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.980+0000 7f2280c00700 1 -- 172.21.15.1:0/1625309534 shutdown_connections 2024-02-28T17:22:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.980+0000 7f2280c00700 1 -- 172.21.15.1:0/1625309534 wait complete. 2024-02-28T17:22:16.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.981+0000 7f2280c00700 1 Processor -- start 2024-02-28T17:22:16.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.982+0000 7f2280c00700 1 -- start start 2024-02-28T17:22:16.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.982+0000 7f2280c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f227c070330 0x7f227c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.982+0000 7f2280c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070c40 0x7f227c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.982+0000 7f2280c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c116790 con 0x7f227c070c40 2024-02-28T17:22:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.982+0000 7f2280c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c10fae0 con 0x7f227c070330 2024-02-28T17:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.983+0000 7f2279400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070c40 0x7f227c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.983+0000 7f2279400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070c40 0x7f227c116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48482/0 (socket says 172.21.15.1:48482) 2024-02-28T17:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.983+0000 7f2279400700 1 -- 172.21.15.1:0/1799405209 learned_addr learned my addr 172.21.15.1:0/1799405209 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.983+0000 7f2279e00700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f227c070330 0x7f227c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.983+0000 7f2279400700 1 -- 172.21.15.1:0/1799405209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f227c070330 msgr2=0x7f227c115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.983+0000 7f2279400700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f227c070330 0x7f227c115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.983+0000 7f2279400700 1 -- 172.21.15.1:0/1799405209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c005040 con 0x7f227c070c40 2024-02-28T17:22:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.983+0000 7f2279400700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070c40 0x7f227c116140 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f2268002a50 tx=0x7f22680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:16.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.984+0000 7f2272000700 1 -- 172.21.15.1:0/1799405209 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2268007cc0 con 0x7f227c070c40 2024-02-28T17:22:16.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.984+0000 7f2272000700 1 -- 172.21.15.1:0/1799405209 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f226800f040 con 0x7f227c070c40 2024-02-28T17:22:16.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.984+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227c10fdc0 con 0x7f227c070c40 2024-02-28T17:22:16.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.984+0000 7f2272000700 1 -- 172.21.15.1:0/1799405209 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2268009640 con 0x7f227c070c40 2024-02-28T17:22:16.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.984+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227c110300 con 0x7f227c070c40 2024-02-28T17:22:16.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.985+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f227c106920 con 0x7f227c070c40 2024-02-28T17:22:16.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.986+0000 7f2272000700 1 -- 172.21.15.1:0/1799405209 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f22680097a0 con 0x7f227c070c40 2024-02-28T17:22:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.987+0000 7f2272000700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f226006d7f0 0x7f226006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.987+0000 7f2272000700 1 -- 172.21.15.1:0/1799405209 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f226808cc60 con 0x7f227c070c40 2024-02-28T17:22:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.988+0000 7f2279e00700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f226006d7f0 0x7f226006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.988+0000 7f2279e00700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f226006d7f0 0x7f226006fca0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f226c000c00 tx=0x7f226c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:16.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:16.992+0000 7f2272000700 1 -- 172.21.15.1:0/1799405209 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2268056490 con 0x7f227c070c40 2024-02-28T17:22:17.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:16 smithi066 ceph-mon[85781]: from='client.36524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:17.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:16 smithi066 ceph-mon[85781]: pgmap v1438: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:17.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:16 smithi066 ceph-mon[85781]: from='client.36528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:17.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:16 smithi001 ceph-mon[112940]: from='client.36524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:17.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:16 smithi001 ceph-mon[112940]: pgmap v1438: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:17.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:16 smithi001 ceph-mon[112940]: from='client.36528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.264+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f227c04bed0 con 0x7f227c070c40 2024-02-28T17:22:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.265+0000 7f2272000700 1 -- 172.21.15.1:0/1799405209 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f2268059b50 con 0x7f227c070c40 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:22:17.272 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:22:17.273 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:22:17.274 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:22:17.275 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:22:17.275 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:22:17.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f226006d7f0 msgr2=0x7f226006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:17.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f226006d7f0 0x7f226006fca0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f226c000c00 tx=0x7f226c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070c40 msgr2=0x7f227c116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070c40 0x7f227c116140 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f2268002a50 tx=0x7f22680046b0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 shutdown_connections 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f226006d7f0 0x7f226006fca0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f227c070330 0x7f227c115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 --2- 172.21.15.1:0/1799405209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227c070c40 0x7f227c116140 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.267+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 >> 172.21.15.1:0/1799405209 conn(0x7f227c0f8ac0 msgr2=0x7f227c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.268+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 shutdown_connections 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.268+0000 7f2280c00700 1 -- 172.21.15.1:0/1799405209 wait complete. 2024-02-28T17:22:17.276 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.421+0000 7faddb600700 1 -- 172.21.15.1:0/1542112349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 msgr2=0x7fadd4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.421+0000 7faddb600700 1 --2- 172.21.15.1:0/1542112349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 0x7fadd4103320 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fadc8004940 tx=0x7fadc8009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.422+0000 7faddb600700 1 -- 172.21.15.1:0/1542112349 shutdown_connections 2024-02-28T17:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.422+0000 7faddb600700 1 --2- 172.21.15.1:0/1542112349 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadd41038f0 0x7fadd40fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.422+0000 7faddb600700 1 --2- 172.21.15.1:0/1542112349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 0x7fadd4103320 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.422+0000 7faddb600700 1 -- 172.21.15.1:0/1542112349 >> 172.21.15.1:0/1542112349 conn(0x7fadd40f8ae0 msgr2=0x7fadd40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.422+0000 7faddb600700 1 -- 172.21.15.1:0/1542112349 shutdown_connections 2024-02-28T17:22:17.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.422+0000 7faddb600700 1 -- 172.21.15.1:0/1542112349 wait complete. 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7faddb600700 1 Processor -- start 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7faddb600700 1 -- start start 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7faddb600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 0x7fadd4115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7faddb600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadd41038f0 0x7fadd4116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7faddb600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd4116780 con 0x7fadd4102f50 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7faddb600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd410fad0 con 0x7fadd41038f0 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7fadd8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 0x7fadd4115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7fadd8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 0x7fadd4115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48498/0 (socket says 172.21.15.1:48498) 2024-02-28T17:22:17.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.423+0000 7fadd8c00700 1 -- 172.21.15.1:0/2774730918 learned_addr learned my addr 172.21.15.1:0/2774730918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7fadd8c00700 1 -- 172.21.15.1:0/2774730918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadd41038f0 msgr2=0x7fadd4116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7fadd3e00700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadd41038f0 0x7fadd4116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7fadd8c00700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadd41038f0 0x7fadd4116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7fadd8c00700 1 -- 172.21.15.1:0/2774730918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc8005040 con 0x7fadd4102f50 2024-02-28T17:22:17.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7fadd8c00700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 0x7fadd4115bf0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009fd0 tx=0x7fadc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7fadd0c00700 1 -- 172.21.15.1:0/2774730918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc800e040 con 0x7fadd4102f50 2024-02-28T17:22:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7fadd0c00700 1 -- 172.21.15.1:0/2774730918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadc800bbf0 con 0x7fadd4102f50 2024-02-28T17:22:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd410fd50 con 0x7fadd4102f50 2024-02-28T17:22:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.424+0000 7fadd0c00700 1 -- 172.21.15.1:0/2774730918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc801c3f0 con 0x7fadd4102f50 2024-02-28T17:22:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.425+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd4110290 con 0x7fadd4102f50 2024-02-28T17:22:17.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.426+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd40fc6e0 con 0x7fadd4102f50 2024-02-28T17:22:17.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.427+0000 7fadd0c00700 1 -- 172.21.15.1:0/2774730918 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fadc800bd60 con 0x7fadd4102f50 2024-02-28T17:22:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.428+0000 7fadd0c00700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadbc06d7f0 0x7fadbc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.428+0000 7fadd0c00700 1 -- 172.21.15.1:0/2774730918 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fadc808e1c0 con 0x7fadd4102f50 2024-02-28T17:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.428+0000 7fadd3e00700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadbc06d7f0 0x7fadbc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:17.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.429+0000 7fadd3e00700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadbc06d7f0 0x7fadbc06fca0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fadd4111490 tx=0x7fadc000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:17.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.433+0000 7fadd0c00700 1 -- 172.21.15.1:0/2774730918 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fadc8057a70 con 0x7fadd4102f50 2024-02-28T17:22:17.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.676+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fadd4110500 con 0x7fadbc06d7f0 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.678+0000 7fadd0c00700 1 -- 172.21.15.1:0/2774730918 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fadd4110500 con 0x7fadbc06d7f0 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:22:17.683 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:22:17.684 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:22:17.684 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:22:17.684 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:22:17.684 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:22:17.684 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:22:17.684 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:22:17.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadbc06d7f0 msgr2=0x7fadbc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:17.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadbc06d7f0 0x7fadbc06fca0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fadd4111490 tx=0x7fadc000b040 comp rx=0 tx=0).stop 2024-02-28T17:22:17.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 msgr2=0x7fadd4115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 0x7fadd4115bf0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009fd0 tx=0x7fadc8009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 shutdown_connections 2024-02-28T17:22:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fadbc06d7f0 0x7fadbc06fca0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadd4102f50 0x7fadd4115bf0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 --2- 172.21.15.1:0/2774730918 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fadd41038f0 0x7fadd4116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.681+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 >> 172.21.15.1:0/2774730918 conn(0x7fadd40f8ae0 msgr2=0x7fadd40fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:17.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.682+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 shutdown_connections 2024-02-28T17:22:17.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.682+0000 7faddb600700 1 -- 172.21.15.1:0/2774730918 wait complete. 2024-02-28T17:22:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.841+0000 7f8045c00700 1 -- 172.21.15.1:0/2469034756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8040070be0 msgr2=0x7f80401092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.841+0000 7f8045c00700 1 --2- 172.21.15.1:0/2469034756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8040070be0 0x7f80401092b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f802c004970 tx=0x7f802c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.841+0000 7f8045c00700 1 -- 172.21.15.1:0/2469034756 shutdown_connections 2024-02-28T17:22:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.841+0000 7f8045c00700 1 --2- 172.21.15.1:0/2469034756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8040070be0 0x7f80401092b0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.841+0000 7f8045c00700 1 --2- 172.21.15.1:0/2469034756 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f80400702d0 0x7f80400706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.841+0000 7f8045c00700 1 -- 172.21.15.1:0/2469034756 >> 172.21.15.1:0/2469034756 conn(0x7f80400f8af0 msgr2=0x7f80400faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.841+0000 7f8045c00700 1 -- 172.21.15.1:0/2469034756 shutdown_connections 2024-02-28T17:22:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.841+0000 7f8045c00700 1 -- 172.21.15.1:0/2469034756 wait complete. 2024-02-28T17:22:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.842+0000 7f8045c00700 1 Processor -- start 2024-02-28T17:22:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.842+0000 7f8045c00700 1 -- start start 2024-02-28T17:22:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.842+0000 7f8045c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80400702d0 0x7f8040115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f8045c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8040070be0 0x7f8040116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f8045c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8040116780 con 0x7f80400702d0 2024-02-28T17:22:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f8045c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804010fad0 con 0x7f8040070be0 2024-02-28T17:22:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f803ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80400702d0 0x7f8040115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f803ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80400702d0 0x7f8040115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48518/0 (socket says 172.21.15.1:48518) 2024-02-28T17:22:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f803ea00700 1 -- 172.21.15.1:0/1636515743 learned_addr learned my addr 172.21.15.1:0/1636515743 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f803ea00700 1 -- 172.21.15.1:0/1636515743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8040070be0 msgr2=0x7f8040116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f803ea00700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8040070be0 0x7f8040116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.843+0000 7f803ea00700 1 -- 172.21.15.1:0/1636515743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c005040 con 0x7f80400702d0 2024-02-28T17:22:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.844+0000 7f803ea00700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80400702d0 0x7f8040115bf0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f803000c810 tx=0x7f8030004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.844+0000 7f8036a00700 1 -- 172.21.15.1:0/1636515743 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8030012810 con 0x7f80400702d0 2024-02-28T17:22:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.844+0000 7f8036a00700 1 -- 172.21.15.1:0/1636515743 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8030012e40 con 0x7f80400702d0 2024-02-28T17:22:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.844+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804010fdb0 con 0x7f80400702d0 2024-02-28T17:22:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.844+0000 7f8036a00700 1 -- 172.21.15.1:0/1636515743 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f803000a370 con 0x7f80400702d0 2024-02-28T17:22:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.844+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80401102f0 con 0x7f80400702d0 2024-02-28T17:22:17.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.845+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8040106970 con 0x7f80400702d0 2024-02-28T17:22:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.847+0000 7f8036a00700 1 -- 172.21.15.1:0/1636515743 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8030011810 con 0x7f80400702d0 2024-02-28T17:22:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.847+0000 7f8036a00700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f802406d7f0 0x7f802406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.848+0000 7f8036a00700 1 -- 172.21.15.1:0/1636515743 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f803008d3f0 con 0x7f80400702d0 2024-02-28T17:22:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.848+0000 7f803e000700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f802406d7f0 0x7f802406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.849+0000 7f803e000700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f802406d7f0 0x7f802406fca0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f802c00b370 tx=0x7f802c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:17.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:17.852+0000 7f8036a00700 1 -- 172.21.15.1:0/1636515743 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8030056ba0 con 0x7f80400702d0 2024-02-28T17:22:18.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:17 smithi066 ceph-mon[85781]: from='client.36532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:18.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:17 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3016578599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:22:18.013 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:17 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1799405209' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:22:18.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:17 smithi001 ceph-mon[112940]: from='client.36532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:18.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:17 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3016578599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:22:18.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:17 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1799405209' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:22:18.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.170+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f80400fc6f0 con 0x7f80400702d0 2024-02-28T17:22:18.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.170+0000 7f8036a00700 1 -- 172.21.15.1:0/1636515743 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f803005a260 con 0x7f80400702d0 2024-02-28T17:22:18.175 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:22:18.175 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:22:18.176 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:22:18.177 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:22:18.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.173+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f802406d7f0 msgr2=0x7f802406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:18.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.173+0000 7f8045c00700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f802406d7f0 0x7f802406fca0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f802c00b370 tx=0x7f802c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:18.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.173+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80400702d0 msgr2=0x7f8040115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:18.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.173+0000 7f8045c00700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80400702d0 0x7f8040115bf0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f803000c810 tx=0x7f8030004470 comp rx=0 tx=0).stop 2024-02-28T17:22:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.174+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 shutdown_connections 2024-02-28T17:22:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.174+0000 7f8045c00700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f802406d7f0 0x7f802406fca0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.174+0000 7f8045c00700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80400702d0 0x7f8040115bf0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.174+0000 7f8045c00700 1 --2- 172.21.15.1:0/1636515743 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f8040070be0 0x7f8040116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.174+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 >> 172.21.15.1:0/1636515743 conn(0x7f80400f8af0 msgr2=0x7f8040103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.174+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 shutdown_connections 2024-02-28T17:22:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:18.174+0000 7f8045c00700 1 -- 172.21.15.1:0/1636515743 wait complete. 2024-02-28T17:22:19.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:18 smithi001 ceph-mon[112940]: pgmap v1439: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:19.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:18 smithi001 ceph-mon[112940]: from='client.36544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:19.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:18 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1636515743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:22:19.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:18 smithi066 ceph-mon[85781]: pgmap v1439: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:19.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:18 smithi066 ceph-mon[85781]: from='client.36544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:19.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:18 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1636515743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:22:21.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:20 smithi001 ceph-mon[112940]: pgmap v1440: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:21.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:20 smithi066 ceph-mon[85781]: pgmap v1440: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:23.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:22 smithi001 ceph-mon[112940]: pgmap v1441: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:23.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:22 smithi066 ceph-mon[85781]: pgmap v1441: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:25.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:24 smithi001 ceph-mon[112940]: pgmap v1442: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:25.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:24 smithi066 ceph-mon[85781]: pgmap v1442: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:27.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:26 smithi066 ceph-mon[85781]: pgmap v1443: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:27.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:22:27.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:26 smithi001 ceph-mon[112940]: pgmap v1443: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:27.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:22:29.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:28 smithi066 ceph-mon[85781]: pgmap v1444: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:29.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:28 smithi001 ceph-mon[112940]: pgmap v1444: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:31.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:30 smithi066 ceph-mon[85781]: pgmap v1445: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:31.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:30 smithi001 ceph-mon[112940]: pgmap v1445: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:33.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:32 smithi066 ceph-mon[85781]: pgmap v1446: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:33.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:32 smithi001 ceph-mon[112940]: pgmap v1446: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:35.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:34 smithi066 ceph-mon[85781]: pgmap v1447: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:35.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:34 smithi001 ceph-mon[112940]: pgmap v1447: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:37.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:36 smithi066 ceph-mon[85781]: pgmap v1448: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:37.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:36 smithi001 ceph-mon[112940]: pgmap v1448: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:39.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:38 smithi066 ceph-mon[85781]: pgmap v1449: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:39.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:38 smithi001 ceph-mon[112940]: pgmap v1449: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:40 smithi066 ceph-mon[85781]: pgmap v1450: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:22:41.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:40 smithi001 ceph-mon[112940]: pgmap v1450: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:41.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:22:43.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:42 smithi066 ceph-mon[85781]: pgmap v1451: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:43.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:42 smithi001 ceph-mon[112940]: pgmap v1451: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:45.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:44 smithi066 ceph-mon[85781]: pgmap v1452: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:45.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:44 smithi001 ceph-mon[112940]: pgmap v1452: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:47.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:46 smithi066 ceph-mon[85781]: pgmap v1453: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:47.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:46 smithi001 ceph-mon[112940]: pgmap v1453: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.347+0000 7fbaf0200700 1 -- 172.21.15.1:0/3351835436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8070870 msgr2=0x7fbae8070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.347+0000 7fbaf0200700 1 --2- 172.21.15.1:0/3351835436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8070870 0x7fbae8070ce0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fbad4004970 tx=0x7fbad4009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.348+0000 7fbaf0200700 1 -- 172.21.15.1:0/3351835436 shutdown_connections 2024-02-28T17:22:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.348+0000 7fbaf0200700 1 --2- 172.21.15.1:0/3351835436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8070870 0x7fbae8070ce0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.348+0000 7fbaf0200700 1 --2- 172.21.15.1:0/3351835436 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae8071f60 0x7fbae8070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.348+0000 7fbaf0200700 1 -- 172.21.15.1:0/3351835436 >> 172.21.15.1:0/3351835436 conn(0x7fbae80f8ae0 msgr2=0x7fbae80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.348+0000 7fbaf0200700 1 -- 172.21.15.1:0/3351835436 shutdown_connections 2024-02-28T17:22:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.348+0000 7fbaf0200700 1 -- 172.21.15.1:0/3351835436 wait complete. 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.349+0000 7fbaf0200700 1 Processor -- start 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.349+0000 7fbaf0200700 1 -- start start 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.349+0000 7fbaf0200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8071f60 0x7fbae811a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.349+0000 7fbaf0200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae811a710 0x7fbae8114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.349+0000 7fbaf0200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae811ac90 con 0x7fbae8071f60 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.349+0000 7fbaf0200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae811ae00 con 0x7fbae811a710 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.350+0000 7fbaed800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8071f60 0x7fbae811a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.350+0000 7fbaed800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8071f60 0x7fbae811a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56612/0 (socket says 172.21.15.1:56612) 2024-02-28T17:22:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.350+0000 7fbaed800700 1 -- 172.21.15.1:0/1065679760 learned_addr learned my addr 172.21.15.1:0/1065679760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.350+0000 7fbaece00700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae811a710 0x7fbae8114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.350+0000 7fbaed800700 1 -- 172.21.15.1:0/1065679760 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae811a710 msgr2=0x7fbae8114070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.350+0000 7fbaed800700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae811a710 0x7fbae8114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.350+0000 7fbaed800700 1 -- 172.21.15.1:0/1065679760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad4005040 con 0x7fbae8071f60 2024-02-28T17:22:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.351+0000 7fbaed800700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8071f60 0x7fbae811a1d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00cb10 tx=0x7fbadc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.351+0000 7fbae5600700 1 -- 172.21.15.1:0/1065679760 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbadc004200 con 0x7fbae8071f60 2024-02-28T17:22:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.351+0000 7fbae5600700 1 -- 172.21.15.1:0/1065679760 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbadc00acc0 con 0x7fbae8071f60 2024-02-28T17:22:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.351+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae8114670 con 0x7fbae8071f60 2024-02-28T17:22:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.351+0000 7fbae5600700 1 -- 172.21.15.1:0/1065679760 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbadc003d90 con 0x7fbae8071f60 2024-02-28T17:22:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.351+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae8114bb0 con 0x7fbae8071f60 2024-02-28T17:22:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.352+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae8114e90 con 0x7fbae8071f60 2024-02-28T17:22:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.353+0000 7fbae5600700 1 -- 172.21.15.1:0/1065679760 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbadc003430 con 0x7fbae8071f60 2024-02-28T17:22:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.354+0000 7fbae5600700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbad006d7f0 0x7fbad006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.354+0000 7fbae5600700 1 -- 172.21.15.1:0/1065679760 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbadc08d550 con 0x7fbae8071f60 2024-02-28T17:22:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.354+0000 7fbaece00700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbad006d7f0 0x7fbad006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.355+0000 7fbaece00700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbad006d7f0 0x7fbad006fca0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009040 tx=0x7fbad4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.359+0000 7fbae5600700 1 -- 172.21.15.1:0/1065679760 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbadc05a830 con 0x7fbae8071f60 2024-02-28T17:22:48.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.603+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbae8110670 con 0x7fbad006d7f0 2024-02-28T17:22:48.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.605+0000 7fbae5600700 1 -- 172.21.15.1:0/1065679760 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbae8110670 con 0x7fbad006d7f0 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.608+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbad006d7f0 msgr2=0x7fbad006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.608+0000 7fbaf0200700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbad006d7f0 0x7fbad006fca0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009040 tx=0x7fbad4009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.608+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8071f60 msgr2=0x7fbae811a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.608+0000 7fbaf0200700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8071f60 0x7fbae811a1d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00cb10 tx=0x7fbadc007f90 comp rx=0 tx=0).stop 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.609+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 shutdown_connections 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.609+0000 7fbaf0200700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbad006d7f0 0x7fbad006fca0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.609+0000 7fbaf0200700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbae8071f60 0x7fbae811a1d0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.609+0000 7fbaf0200700 1 --2- 172.21.15.1:0/1065679760 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbae811a710 0x7fbae8114070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.609+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 >> 172.21.15.1:0/1065679760 conn(0x7fbae80f8ae0 msgr2=0x7fbae810c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.609+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 shutdown_connections 2024-02-28T17:22:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.609+0000 7fbaf0200700 1 -- 172.21.15.1:0/1065679760 wait complete. 2024-02-28T17:22:48.627 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:22:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.758+0000 7fa868200700 1 -- 172.21.15.1:0/2711806585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fe8f0 msgr2=0x7fa860107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.758+0000 7fa868200700 1 --2- 172.21.15.1:0/2711806585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fe8f0 0x7fa860107050 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fa84c004970 tx=0x7fa84c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.759+0000 7fa868200700 1 -- 172.21.15.1:0/2711806585 shutdown_connections 2024-02-28T17:22:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.759+0000 7fa868200700 1 --2- 172.21.15.1:0/2711806585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fe8f0 0x7fa860107050 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.759+0000 7fa868200700 1 --2- 172.21.15.1:0/2711806585 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8600fdfe0 0x7fa8600fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.759+0000 7fa868200700 1 -- 172.21.15.1:0/2711806585 >> 172.21.15.1:0/2711806585 conn(0x7fa860071d90 msgr2=0x7fa860072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.759+0000 7fa868200700 1 -- 172.21.15.1:0/2711806585 shutdown_connections 2024-02-28T17:22:48.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.759+0000 7fa868200700 1 -- 172.21.15.1:0/2711806585 wait complete. 2024-02-28T17:22:48.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.759+0000 7fa868200700 1 Processor -- start 2024-02-28T17:22:48.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.760+0000 7fa868200700 1 -- start start 2024-02-28T17:22:48.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.760+0000 7fa868200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fdfe0 0x7fa860115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:48.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.760+0000 7fa868200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8600fe8f0 0x7fa860116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.760+0000 7fa868200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8601167a0 con 0x7fa8600fdfe0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.760+0000 7fa868200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86010fad0 con 0x7fa8600fe8f0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.760+0000 7fa865800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fdfe0 0x7fa860115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.760+0000 7fa865800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fdfe0 0x7fa860115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56634/0 (socket says 172.21.15.1:56634) 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.760+0000 7fa865800700 1 -- 172.21.15.1:0/2762796545 learned_addr learned my addr 172.21.15.1:0/2762796545 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa865800700 1 -- 172.21.15.1:0/2762796545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8600fe8f0 msgr2=0x7fa860116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa864e00700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8600fe8f0 0x7fa860116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa865800700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8600fe8f0 0x7fa860116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa865800700 1 -- 172.21.15.1:0/2762796545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c005040 con 0x7fa8600fdfe0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa865800700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fdfe0 0x7fa860115c10 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fa854001d00 tx=0x7fa854009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa85d600700 1 -- 172.21.15.1:0/2762796545 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa854004020 con 0x7fa8600fdfe0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa85d600700 1 -- 172.21.15.1:0/2762796545 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa85400f040 con 0x7fa8600fdfe0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa85d600700 1 -- 172.21.15.1:0/2762796545 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa854009310 con 0x7fa8600fdfe0 2024-02-28T17:22:48.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.761+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86010fdb0 con 0x7fa8600fdfe0 2024-02-28T17:22:48.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.762+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8601102f0 con 0x7fa8600fdfe0 2024-02-28T17:22:48.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.763+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa860106ac0 con 0x7fa8600fdfe0 2024-02-28T17:22:48.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.764+0000 7fa85d600700 1 -- 172.21.15.1:0/2762796545 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa854009490 con 0x7fa8600fdfe0 2024-02-28T17:22:48.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.765+0000 7fa85d600700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa84806d7f0 0x7fa84806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.765+0000 7fa85d600700 1 -- 172.21.15.1:0/2762796545 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa85408c880 con 0x7fa8600fdfe0 2024-02-28T17:22:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.765+0000 7fa864e00700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa84806d7f0 0x7fa84806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:48.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.766+0000 7fa864e00700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa84806d7f0 0x7fa84806fca0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa84c000c00 tx=0x7fa84c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:48.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:48.769+0000 7fa85d600700 1 -- 172.21.15.1:0/2762796545 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa854059be0 con 0x7fa8600fdfe0 2024-02-28T17:22:49.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.014+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa86004bed0 con 0x7fa84806d7f0 2024-02-28T17:22:49.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.016+0000 7fa85d600700 1 -- 172.21.15.1:0/2762796545 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa86004bed0 con 0x7fa84806d7f0 2024-02-28T17:22:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa84806d7f0 msgr2=0x7fa84806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa84806d7f0 0x7fa84806fca0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa84c000c00 tx=0x7fa84c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:49.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fdfe0 msgr2=0x7fa860115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fdfe0 0x7fa860115c10 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fa854001d00 tx=0x7fa854009f30 comp rx=0 tx=0).stop 2024-02-28T17:22:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 shutdown_connections 2024-02-28T17:22:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa84806d7f0 0x7fa84806fca0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8600fdfe0 0x7fa860115c10 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 --2- 172.21.15.1:0/2762796545 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa8600fe8f0 0x7fa860116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 >> 172.21.15.1:0/2762796545 conn(0x7fa860071d90 msgr2=0x7fa860103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 shutdown_connections 2024-02-28T17:22:49.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.019+0000 7fa868200700 1 -- 172.21.15.1:0/2762796545 wait complete. 2024-02-28T17:22:49.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.172+0000 7effc5c00700 1 -- 172.21.15.1:0/719183389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc01095f0 msgr2=0x7effc01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.172+0000 7effc5c00700 1 --2- 172.21.15.1:0/719183389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc01095f0 0x7effc01099c0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7effb0004940 tx=0x7effb0009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:49.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.172+0000 7effc5c00700 1 -- 172.21.15.1:0/719183389 shutdown_connections 2024-02-28T17:22:49.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.172+0000 7effc5c00700 1 --2- 172.21.15.1:0/719183389 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effc0103dc0 0x7effc0104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.172+0000 7effc5c00700 1 --2- 172.21.15.1:0/719183389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc01095f0 0x7effc01099c0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.172+0000 7effc5c00700 1 -- 172.21.15.1:0/719183389 >> 172.21.15.1:0/719183389 conn(0x7effc0072d50 msgr2=0x7effc0075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:49.176 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:48 smithi001 ceph-mon[112940]: pgmap v1454: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.173+0000 7effc5c00700 1 -- 172.21.15.1:0/719183389 shutdown_connections 2024-02-28T17:22:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.173+0000 7effc5c00700 1 -- 172.21.15.1:0/719183389 wait complete. 2024-02-28T17:22:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.174+0000 7effc5c00700 1 Processor -- start 2024-02-28T17:22:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.174+0000 7effc5c00700 1 -- start start 2024-02-28T17:22:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.174+0000 7effc5c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc0103dc0 0x7effc011a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.174+0000 7effc5c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effc01095f0 0x7effc011a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effc5c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc011ac70 con 0x7effc0103dc0 2024-02-28T17:22:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effc5c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0113f80 con 0x7effc01095f0 2024-02-28T17:22:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effbea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc0103dc0 0x7effc011a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effbea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc0103dc0 0x7effc011a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56640/0 (socket says 172.21.15.1:56640) 2024-02-28T17:22:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effbea00700 1 -- 172.21.15.1:0/611425055 learned_addr learned my addr 172.21.15.1:0/611425055 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effbe000700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effc01095f0 0x7effc011a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effbea00700 1 -- 172.21.15.1:0/611425055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effc01095f0 msgr2=0x7effc011a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effbea00700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effc01095f0 0x7effc011a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effbea00700 1 -- 172.21.15.1:0/611425055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb0005040 con 0x7effc0103dc0 2024-02-28T17:22:49.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.175+0000 7effbe000700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effc01095f0 0x7effc011a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:22:49.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.176+0000 7effbea00700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc0103dc0 0x7effc011a0e0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7effb0009150 tx=0x7effb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:49.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.176+0000 7effb6a00700 1 -- 172.21.15.1:0/611425055 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effb000e040 con 0x7effc0103dc0 2024-02-28T17:22:49.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.176+0000 7effb6a00700 1 -- 172.21.15.1:0/611425055 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effb000bbf0 con 0x7effc0103dc0 2024-02-28T17:22:49.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.176+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc0114200 con 0x7effc0103dc0 2024-02-28T17:22:49.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.176+0000 7effb6a00700 1 -- 172.21.15.1:0/611425055 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effb001c630 con 0x7effc0103dc0 2024-02-28T17:22:49.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.176+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc0114740 con 0x7effc0103dc0 2024-02-28T17:22:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.178+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc0106850 con 0x7effc0103dc0 2024-02-28T17:22:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.178+0000 7effb6a00700 1 -- 172.21.15.1:0/611425055 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7effb000bd60 con 0x7effc0103dc0 2024-02-28T17:22:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.180+0000 7effb6a00700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effa406d8a0 0x7effa406fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.180+0000 7effb6a00700 1 -- 172.21.15.1:0/611425055 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7effb008e240 con 0x7effc0103dc0 2024-02-28T17:22:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.180+0000 7effbe000700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effa406d8a0 0x7effa406fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.181+0000 7effbe000700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effa406d8a0 0x7effa406fd50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7effac009fd0 tx=0x7effac00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.184+0000 7effb6a00700 1 -- 172.21.15.1:0/611425055 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7effb005b620 con 0x7effc0103dc0 2024-02-28T17:22:49.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:48 smithi066 ceph-mon[85781]: pgmap v1454: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.418+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7effc004bed0 con 0x7effa406d8a0 2024-02-28T17:22:49.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.449+0000 7effb6a00700 1 -- 172.21.15.1:0/611425055 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7effc004bed0 con 0x7effa406d8a0 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (48m) 10m ago 63m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (45m) 10m ago 63m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (45m) 10m ago 62m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (48m) 10m ago 62m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (41m) 10m ago 56m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (41m) 10m ago 55m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (10m) 10m ago 55m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (25m) 10m ago 56m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (51m) 10m ago 64m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (50m) 10m ago 61m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:22:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (47m) 10m ago 65m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (46m) 10m ago 61m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (50m) 10m ago 63m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (49m) 10m ago 61m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (45m) 10m ago 60m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (44m) 10m ago 60m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (43m) 10m ago 59m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (43m) 10m ago 59m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (42m) 10m ago 58m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (41m) 10m ago 57m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:22:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (49m) 10m ago 62m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:22:49.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effa406d8a0 msgr2=0x7effa406fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effa406d8a0 0x7effa406fd50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7effac009fd0 tx=0x7effac00b040 comp rx=0 tx=0).stop 2024-02-28T17:22:49.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc0103dc0 msgr2=0x7effc011a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc0103dc0 0x7effc011a0e0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7effb0009150 tx=0x7effb0009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 shutdown_connections 2024-02-28T17:22:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7effa406d8a0 0x7effa406fd50 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effc0103dc0 0x7effc011a0e0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 --2- 172.21.15.1:0/611425055 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effc01095f0 0x7effc011a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.452+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 >> 172.21.15.1:0/611425055 conn(0x7effc0072d50 msgr2=0x7effc0074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.453+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 shutdown_connections 2024-02-28T17:22:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.453+0000 7effc5c00700 1 -- 172.21.15.1:0/611425055 wait complete. 2024-02-28T17:22:49.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.601+0000 7fdd8aa00700 1 -- 172.21.15.1:0/4152537087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105b10 msgr2=0x7fdd840ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.601+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/4152537087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105b10 0x7fdd840ff180 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c004940 tx=0x7fdd6c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.601+0000 7fdd8aa00700 1 -- 172.21.15.1:0/4152537087 shutdown_connections 2024-02-28T17:22:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.601+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/4152537087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105b10 0x7fdd840ff180 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.601+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/4152537087 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd84105170 0x7fdd84105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.601+0000 7fdd8aa00700 1 -- 172.21.15.1:0/4152537087 >> 172.21.15.1:0/4152537087 conn(0x7fdd840face0 msgr2=0x7fdd840fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.601+0000 7fdd8aa00700 1 -- 172.21.15.1:0/4152537087 shutdown_connections 2024-02-28T17:22:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.601+0000 7fdd8aa00700 1 -- 172.21.15.1:0/4152537087 wait complete. 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.602+0000 7fdd8aa00700 1 Processor -- start 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.602+0000 7fdd8aa00700 1 -- start start 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.602+0000 7fdd8aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105170 0x7fdd84115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.602+0000 7fdd8aa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd84105b10 0x7fdd84116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.602+0000 7fdd8aa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd84116870 con 0x7fdd84105170 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.602+0000 7fdd8aa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8410fb80 con 0x7fdd84105b10 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.603+0000 7fdd83e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105170 0x7fdd84115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.603+0000 7fdd83e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105170 0x7fdd84115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56654/0 (socket says 172.21.15.1:56654) 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.603+0000 7fdd83e00700 1 -- 172.21.15.1:0/3182991371 learned_addr learned my addr 172.21.15.1:0/3182991371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.603+0000 7fdd83e00700 1 -- 172.21.15.1:0/3182991371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd84105b10 msgr2=0x7fdd84116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.603+0000 7fdd7be00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd84105b10 0x7fdd84116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.603+0000 7fdd83e00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd84105b10 0x7fdd84116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.603+0000 7fdd83e00700 1 -- 172.21.15.1:0/3182991371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c005040 con 0x7fdd84105170 2024-02-28T17:22:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.603+0000 7fdd83e00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105170 0x7fdd84115ce0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fdd74002a50 tx=0x7fdd740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.604+0000 7fdd80c00700 1 -- 172.21.15.1:0/3182991371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd74007cc0 con 0x7fdd84105170 2024-02-28T17:22:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.604+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8410fe60 con 0x7fdd84105170 2024-02-28T17:22:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.604+0000 7fdd80c00700 1 -- 172.21.15.1:0/3182991371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd7400f040 con 0x7fdd84105170 2024-02-28T17:22:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.604+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd841103a0 con 0x7fdd84105170 2024-02-28T17:22:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.604+0000 7fdd80c00700 1 -- 172.21.15.1:0/3182991371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd74009640 con 0x7fdd84105170 2024-02-28T17:22:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.605+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd84107640 con 0x7fdd84105170 2024-02-28T17:22:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.606+0000 7fdd80c00700 1 -- 172.21.15.1:0/3182991371 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdd740097a0 con 0x7fdd84105170 2024-02-28T17:22:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.607+0000 7fdd80c00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd6806d7f0 0x7fdd6806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.607+0000 7fdd80c00700 1 -- 172.21.15.1:0/3182991371 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdd7408cc60 con 0x7fdd84105170 2024-02-28T17:22:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.607+0000 7fdd7be00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd6806d7f0 0x7fdd6806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.608+0000 7fdd7be00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd6806d7f0 0x7fdd6806fca0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdd84111580 tx=0x7fdd6c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:49.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.612+0000 7fdd80c00700 1 -- 172.21.15.1:0/3182991371 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdd74059fc0 con 0x7fdd84105170 2024-02-28T17:22:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.912+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd8404bed0 con 0x7fdd84105170 2024-02-28T17:22:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.913+0000 7fdd80c00700 1 -- 172.21.15.1:0/3182991371 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fdd74059b50 con 0x7fdd84105170 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:22:49.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:22:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:22:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:22:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:22:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:22:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:22:49.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:22:49.918 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:22:49.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd6806d7f0 msgr2=0x7fdd6806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd6806d7f0 0x7fdd6806fca0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdd84111580 tx=0x7fdd6c009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105170 msgr2=0x7fdd84115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:49.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105170 0x7fdd84115ce0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fdd74002a50 tx=0x7fdd740046b0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 shutdown_connections 2024-02-28T17:22:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdd6806d7f0 0x7fdd6806fca0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd84105170 0x7fdd84115ce0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 --2- 172.21.15.1:0/3182991371 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdd84105b10 0x7fdd84116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 >> 172.21.15.1:0/3182991371 conn(0x7fdd840face0 msgr2=0x7fdd840fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:49.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 shutdown_connections 2024-02-28T17:22:49.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:49.916+0000 7fdd8aa00700 1 -- 172.21.15.1:0/3182991371 wait complete. 2024-02-28T17:22:50.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.056+0000 7f0592200700 1 -- 172.21.15.1:0/248407142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105b00 msgr2=0x7f058c0ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.056+0000 7f0592200700 1 --2- 172.21.15.1:0/248407142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105b00 0x7f058c0ff170 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f0574004940 tx=0x7f0574009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.057+0000 7f0592200700 1 -- 172.21.15.1:0/248407142 shutdown_connections 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.057+0000 7f0592200700 1 --2- 172.21.15.1:0/248407142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105b00 0x7f058c0ff170 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.057+0000 7f0592200700 1 --2- 172.21.15.1:0/248407142 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f058c105160 0x7f058c105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.057+0000 7f0592200700 1 -- 172.21.15.1:0/248407142 >> 172.21.15.1:0/248407142 conn(0x7f058c0facb0 msgr2=0x7f058c0fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.057+0000 7f0592200700 1 -- 172.21.15.1:0/248407142 shutdown_connections 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.057+0000 7f0592200700 1 -- 172.21.15.1:0/248407142 wait complete. 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.057+0000 7f0592200700 1 Processor -- start 2024-02-28T17:22:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.058+0000 7f0592200700 1 -- start start 2024-02-28T17:22:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.058+0000 7f0592200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105160 0x7f058c115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.058+0000 7f0592200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f058c105b00 0x7f058c1161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.058+0000 7f0592200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c116820 con 0x7f058c105160 2024-02-28T17:22:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.058+0000 7f0592200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c10fb50 con 0x7f058c105b00 2024-02-28T17:22:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.058+0000 7f058b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105160 0x7f058c115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.058+0000 7f058b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105160 0x7f058c115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56680/0 (socket says 172.21.15.1:56680) 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.058+0000 7f058b400700 1 -- 172.21.15.1:0/566989794 learned_addr learned my addr 172.21.15.1:0/566989794 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f058b400700 1 -- 172.21.15.1:0/566989794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f058c105b00 msgr2=0x7f058c1161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f0582a00700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f058c105b00 0x7f058c1161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f058b400700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f058c105b00 0x7f058c1161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f058b400700 1 -- 172.21.15.1:0/566989794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0574005040 con 0x7f058c105160 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f058b400700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105160 0x7f058c115c90 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f057c00c830 tx=0x7f057c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f0583e00700 1 -- 172.21.15.1:0/566989794 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f057c00a760 con 0x7f058c105160 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f0583e00700 1 -- 172.21.15.1:0/566989794 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f057c00ad90 con 0x7f058c105160 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c10fe30 con 0x7f058c105160 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.059+0000 7f0583e00700 1 -- 172.21.15.1:0/566989794 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f057c0113a0 con 0x7f058c105160 2024-02-28T17:22:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.060+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c110370 con 0x7f058c105160 2024-02-28T17:22:50.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.061+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c107630 con 0x7f058c105160 2024-02-28T17:22:50.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.062+0000 7f0583e00700 1 -- 172.21.15.1:0/566989794 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f057c012400 con 0x7f058c105160 2024-02-28T17:22:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.062+0000 7f0583e00700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f057006d7f0 0x7f057006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.063+0000 7f0583e00700 1 -- 172.21.15.1:0/566989794 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f057c08cfa0 con 0x7f058c105160 2024-02-28T17:22:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.063+0000 7f0582a00700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f057006d7f0 0x7f057006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.063+0000 7f0582a00700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f057006d7f0 0x7f057006fca0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f058c1114c0 tx=0x7f0574009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:50.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.067+0000 7f0583e00700 1 -- 172.21.15.1:0/566989794 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f057c05a300 con 0x7f058c105160 2024-02-28T17:22:50.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:49 smithi066 ceph-mon[85781]: from='client.36552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:50.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:49 smithi066 ceph-mon[85781]: from='client.36556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:50.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:49 smithi001 ceph-mon[112940]: from='client.36552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:50.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:49 smithi001 ceph-mon[112940]: from='client.36556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.334+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f058c111200 con 0x7f058c105160 2024-02-28T17:22:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.336+0000 7f0583e00700 1 -- 172.21.15.1:0/566989794 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f057c059e90 con 0x7f058c105160 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:22:50.340 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:22:50.341 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:22:50.342 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:22:50.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:22:50.343 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:22:50.343 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:22:50.343 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:22:50.343 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:22:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.338+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f057006d7f0 msgr2=0x7f057006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.338+0000 7f0592200700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f057006d7f0 0x7f057006fca0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f058c1114c0 tx=0x7f0574009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.338+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105160 msgr2=0x7f058c115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.338+0000 7f0592200700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105160 0x7f058c115c90 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f057c00c830 tx=0x7f057c004470 comp rx=0 tx=0).stop 2024-02-28T17:22:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.338+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 shutdown_connections 2024-02-28T17:22:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.338+0000 7f0592200700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f057006d7f0 0x7f057006fca0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.339+0000 7f0592200700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f058c105160 0x7f058c115c90 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.339+0000 7f0592200700 1 --2- 172.21.15.1:0/566989794 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f058c105b00 0x7f058c1161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.339+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 >> 172.21.15.1:0/566989794 conn(0x7f058c0facb0 msgr2=0x7f058c0fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.339+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 shutdown_connections 2024-02-28T17:22:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.339+0000 7f0592200700 1 -- 172.21.15.1:0/566989794 wait complete. 2024-02-28T17:22:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:22:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.496+0000 7f517dc00700 1 -- 172.21.15.1:0/1825698363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 msgr2=0x7f5178109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.496+0000 7f517dc00700 1 --2- 172.21.15.1:0/1825698363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 0x7f5178109980 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5168009ce0 tx=0x7f5168004d10 comp rx=0 tx=0).stop 2024-02-28T17:22:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.496+0000 7f517dc00700 1 -- 172.21.15.1:0/1825698363 shutdown_connections 2024-02-28T17:22:50.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.496+0000 7f517dc00700 1 --2- 172.21.15.1:0/1825698363 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5178103dc0 0x7f5178104230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.496+0000 7f517dc00700 1 --2- 172.21.15.1:0/1825698363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 0x7f5178109980 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.496+0000 7f517dc00700 1 -- 172.21.15.1:0/1825698363 >> 172.21.15.1:0/1825698363 conn(0x7f5178072d70 msgr2=0x7f5178075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.496+0000 7f517dc00700 1 -- 172.21.15.1:0/1825698363 shutdown_connections 2024-02-28T17:22:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.496+0000 7f517dc00700 1 -- 172.21.15.1:0/1825698363 wait complete. 2024-02-28T17:22:50.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.497+0000 7f517dc00700 1 Processor -- start 2024-02-28T17:22:50.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.498+0000 7f517dc00700 1 -- start start 2024-02-28T17:22:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.498+0000 7f517dc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5178103dc0 0x7f517811a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.498+0000 7f517dc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 0x7f517811a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.498+0000 7f517dc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517811ae30 con 0x7f51781095b0 2024-02-28T17:22:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.498+0000 7f517dc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51781140b0 con 0x7f5178103dc0 2024-02-28T17:22:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.499+0000 7f516fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 0x7f517811a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.499+0000 7f516fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 0x7f517811a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56686/0 (socket says 172.21.15.1:56686) 2024-02-28T17:22:50.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.499+0000 7f516fe00700 1 -- 172.21.15.1:0/2110311878 learned_addr learned my addr 172.21.15.1:0/2110311878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:50.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.499+0000 7f5176a00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5178103dc0 0x7f517811a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.499+0000 7f516fe00700 1 -- 172.21.15.1:0/2110311878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5178103dc0 msgr2=0x7f517811a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.499+0000 7f516fe00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5178103dc0 0x7f517811a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.499+0000 7f516fe00700 1 -- 172.21.15.1:0/2110311878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5168009940 con 0x7f51781095b0 2024-02-28T17:22:50.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.500+0000 7f516fe00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 0x7f517811a750 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f516000b790 tx=0x7f5160009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:50.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.500+0000 7f516ea00700 1 -- 172.21.15.1:0/2110311878 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f516000f770 con 0x7f51781095b0 2024-02-28T17:22:50.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.500+0000 7f516ea00700 1 -- 172.21.15.1:0/2110311878 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f516000fda0 con 0x7f51781095b0 2024-02-28T17:22:50.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.500+0000 7f516ea00700 1 -- 172.21.15.1:0/2110311878 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f516000d310 con 0x7f51781095b0 2024-02-28T17:22:50.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.500+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5178114330 con 0x7f51781095b0 2024-02-28T17:22:50.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.500+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5178114870 con 0x7f51781095b0 2024-02-28T17:22:50.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.501+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5178075890 con 0x7f51781095b0 2024-02-28T17:22:50.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.502+0000 7f516ea00700 1 -- 172.21.15.1:0/2110311878 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5160010420 con 0x7f51781095b0 2024-02-28T17:22:50.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.503+0000 7f516ea00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f515c06d7e0 0x7f515c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.503+0000 7f516ea00700 1 -- 172.21.15.1:0/2110311878 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f516008cfb0 con 0x7f51781095b0 2024-02-28T17:22:50.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.504+0000 7f5176a00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f515c06d7e0 0x7f515c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.505+0000 7f5176a00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f515c06d7e0 0x7f515c06fc90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5168015040 tx=0x7f5168004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:50.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.508+0000 7f516ea00700 1 -- 172.21.15.1:0/2110311878 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f51600c0d20 con 0x7f51781095b0 2024-02-28T17:22:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.752+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5178115900 con 0x7f515c06d7e0 2024-02-28T17:22:50.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.756+0000 7f516ea00700 1 -- 172.21.15.1:0/2110311878 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5178115900 con 0x7f515c06d7e0 2024-02-28T17:22:50.759 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:22:50.759 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:22:50.760 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:22:50.761 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:22:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.758+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f515c06d7e0 msgr2=0x7f515c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.758+0000 7f517dc00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f515c06d7e0 0x7f515c06fc90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5168015040 tx=0x7f5168004d10 comp rx=0 tx=0).stop 2024-02-28T17:22:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.758+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 msgr2=0x7f517811a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.758+0000 7f517dc00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 0x7f517811a750 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f516000b790 tx=0x7f5160009250 comp rx=0 tx=0).stop 2024-02-28T17:22:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.759+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 shutdown_connections 2024-02-28T17:22:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.759+0000 7f517dc00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f515c06d7e0 0x7f515c06fc90 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.759+0000 7f517dc00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5178103dc0 0x7f517811a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.759+0000 7f517dc00700 1 --2- 172.21.15.1:0/2110311878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51781095b0 0x7f517811a750 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.759+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 >> 172.21.15.1:0/2110311878 conn(0x7f5178072d70 msgr2=0x7f5178074680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.759+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 shutdown_connections 2024-02-28T17:22:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.759+0000 7f517dc00700 1 -- 172.21.15.1:0/2110311878 wait complete. 2024-02-28T17:22:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.913+0000 7fdde4c00700 1 -- 172.21.15.1:0/1162703531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 msgr2=0x7fdde00ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.913+0000 7fdde4c00700 1 --2- 172.21.15.1:0/1162703531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 0x7fdde00ff180 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fddc8004940 tx=0x7fddc8009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.914+0000 7fdde4c00700 1 -- 172.21.15.1:0/1162703531 shutdown_connections 2024-02-28T17:22:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.914+0000 7fdde4c00700 1 --2- 172.21.15.1:0/1162703531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 0x7fdde00ff180 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.914+0000 7fdde4c00700 1 --2- 172.21.15.1:0/1162703531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdde0105170 0x7fdde0105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.914+0000 7fdde4c00700 1 -- 172.21.15.1:0/1162703531 >> 172.21.15.1:0/1162703531 conn(0x7fdde00face0 msgr2=0x7fdde00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.914+0000 7fdde4c00700 1 -- 172.21.15.1:0/1162703531 shutdown_connections 2024-02-28T17:22:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.914+0000 7fdde4c00700 1 -- 172.21.15.1:0/1162703531 wait complete. 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.914+0000 7fdde4c00700 1 Processor -- start 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.915+0000 7fdde4c00700 1 -- start start 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.915+0000 7fdde4c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdde0105170 0x7fdde0115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.915+0000 7fdde4c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 0x7fdde0116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.915+0000 7fdde4c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01166e0 con 0x7fdde0105b10 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.915+0000 7fdde4c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde010fa10 con 0x7fdde0105170 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.915+0000 7fddd5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 0x7fdde0116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.915+0000 7fddd5400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 0x7fdde0116090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56712/0 (socket says 172.21.15.1:56712) 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.915+0000 7fddd5400700 1 -- 172.21.15.1:0/1293868739 learned_addr learned my addr 172.21.15.1:0/1293868739 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:22:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.916+0000 7fdddde00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdde0105170 0x7fdde0115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.916+0000 7fddd5400700 1 -- 172.21.15.1:0/1293868739 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdde0105170 msgr2=0x7fdde0115b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.916+0000 7fddd5400700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdde0105170 0x7fdde0115b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.916+0000 7fddd5400700 1 -- 172.21.15.1:0/1293868739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc8005040 con 0x7fdde0105b10 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.916+0000 7fddd5400700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 0x7fdde0116090 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fddc800ba40 tx=0x7fddc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.916+0000 7fddd6a00700 1 -- 172.21.15.1:0/1293868739 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc800e040 con 0x7fdde0105b10 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.917+0000 7fddd6a00700 1 -- 172.21.15.1:0/1293868739 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddc8009400 con 0x7fdde0105b10 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.917+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde010fc90 con 0x7fdde0105b10 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.917+0000 7fddd6a00700 1 -- 172.21.15.1:0/1293868739 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc801c930 con 0x7fdde0105b10 2024-02-28T17:22:50.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.917+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde01101d0 con 0x7fdde0105b10 2024-02-28T17:22:50.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.918+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde00fe900 con 0x7fdde0105b10 2024-02-28T17:22:50.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.919+0000 7fddd6a00700 1 -- 172.21.15.1:0/1293868739 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fddc800ba70 con 0x7fdde0105b10 2024-02-28T17:22:50.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.920+0000 7fddd6a00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fddc006d830 0x7fddc006fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:22:50.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.920+0000 7fddd6a00700 1 -- 172.21.15.1:0/1293868739 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fddc808e3c0 con 0x7fdde0105b10 2024-02-28T17:22:50.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.920+0000 7fdddde00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fddc006d830 0x7fddc006fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:22:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.921+0000 7fdddde00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fddc006d830 0x7fddc006fce0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fddd0004990 tx=0x7fddd0009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:22:50.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:50.925+0000 7fddd6a00700 1 -- 172.21.15.1:0/1293868739 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fddc805b7d0 con 0x7fdde0105b10 2024-02-28T17:22:51.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:50 smithi001 ceph-mon[112940]: from='client.36560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:51.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:50 smithi001 ceph-mon[112940]: pgmap v1455: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:51.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:50 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3182991371' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:22:51.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:50 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/566989794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:22:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.224+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdde0111130 con 0x7fdde0105b10 2024-02-28T17:22:51.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.225+0000 7fddd6a00700 1 -- 172.21.15.1:0/1293868739 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fddc805b360 con 0x7fdde0105b10 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:22:51.229 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:22:51.230 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:22:51.231 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:22:51.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fddc006d830 msgr2=0x7fddc006fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:51.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fddc006d830 0x7fddc006fce0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fddd0004990 tx=0x7fddd0009ee0 comp rx=0 tx=0).stop 2024-02-28T17:22:51.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 msgr2=0x7fdde0116090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:22:51.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 0x7fdde0116090 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fddc800ba40 tx=0x7fddc8009e30 comp rx=0 tx=0).stop 2024-02-28T17:22:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 shutdown_connections 2024-02-28T17:22:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fddc006d830 0x7fddc006fce0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdde0105170 0x7fdde0115b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 --2- 172.21.15.1:0/1293868739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde0105b10 0x7fdde0116090 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:22:51.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.228+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 >> 172.21.15.1:0/1293868739 conn(0x7fdde00face0 msgr2=0x7fdde00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:22:51.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.229+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 shutdown_connections 2024-02-28T17:22:51.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:22:51.229+0000 7fdde4c00700 1 -- 172.21.15.1:0/1293868739 wait complete. 2024-02-28T17:22:51.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:50 smithi066 ceph-mon[85781]: from='client.36560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:51.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:50 smithi066 ceph-mon[85781]: pgmap v1455: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:51.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:50 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3182991371' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:22:51.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:50 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/566989794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:22:52.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:51 smithi066 ceph-mon[85781]: from='client.36572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:52.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:51 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1293868739' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:22:52.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:51 smithi001 ceph-mon[112940]: from='client.36572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:22:52.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:51 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1293868739' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:22:53.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:52 smithi066 ceph-mon[85781]: pgmap v1456: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:53.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:52 smithi001 ceph-mon[112940]: pgmap v1456: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:54 smithi066 ceph-mon[85781]: pgmap v1457: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:54 smithi001 ceph-mon[112940]: pgmap v1457: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:56.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:22:56.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:22:57.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:56 smithi066 ceph-mon[85781]: pgmap v1458: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:57.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:56 smithi001 ceph-mon[112940]: pgmap v1458: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:59.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:22:58 smithi066 ceph-mon[85781]: pgmap v1459: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:22:59.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:22:58 smithi001 ceph-mon[112940]: pgmap v1459: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:01.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:00 smithi066 ceph-mon[85781]: pgmap v1460: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:01.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:00 smithi001 ceph-mon[112940]: pgmap v1460: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:02 smithi066 ceph-mon[85781]: pgmap v1461: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:03.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:02 smithi001 ceph-mon[112940]: pgmap v1461: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:05.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:04 smithi066 ceph-mon[85781]: pgmap v1462: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:05.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:04 smithi001 ceph-mon[112940]: pgmap v1462: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:07.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:06 smithi066 ceph-mon[85781]: pgmap v1463: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:07.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:06 smithi001 ceph-mon[112940]: pgmap v1463: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:08 smithi066 ceph-mon[85781]: pgmap v1464: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:09.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:08 smithi001 ceph-mon[112940]: pgmap v1464: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:11.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:10 smithi066 ceph-mon[85781]: pgmap v1465: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:11.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:23:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:10 smithi001 ceph-mon[112940]: pgmap v1465: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:23:13.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:12 smithi066 ceph-mon[85781]: pgmap v1466: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:13.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:12 smithi001 ceph-mon[112940]: pgmap v1466: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:15.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:14 smithi066 ceph-mon[85781]: pgmap v1467: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:15.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:14 smithi001 ceph-mon[112940]: pgmap v1467: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:17.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:16 smithi066 ceph-mon[85781]: pgmap v1468: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:17.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:16 smithi001 ceph-mon[112940]: pgmap v1468: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:19.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:18 smithi066 ceph-mon[85781]: pgmap v1469: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:19.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:18 smithi001 ceph-mon[112940]: pgmap v1469: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:21.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:20 smithi066 ceph-mon[85781]: pgmap v1470: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:21.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:20 smithi001 ceph-mon[112940]: pgmap v1470: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.399+0000 7ff4e6800700 1 -- 172.21.15.1:0/512561424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 msgr2=0x7ff4e0105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.399+0000 7ff4e6800700 1 --2- 172.21.15.1:0/512561424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 0x7ff4e0105540 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0004940 tx=0x7ff4d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.399+0000 7ff4e6800700 1 -- 172.21.15.1:0/512561424 shutdown_connections 2024-02-28T17:23:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.399+0000 7ff4e6800700 1 --2- 172.21.15.1:0/512561424 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4e0105b10 0x7ff4e00ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.399+0000 7ff4e6800700 1 --2- 172.21.15.1:0/512561424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 0x7ff4e0105540 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.399+0000 7ff4e6800700 1 -- 172.21.15.1:0/512561424 >> 172.21.15.1:0/512561424 conn(0x7ff4e00face0 msgr2=0x7ff4e00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.399+0000 7ff4e6800700 1 -- 172.21.15.1:0/512561424 shutdown_connections 2024-02-28T17:23:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.399+0000 7ff4e6800700 1 -- 172.21.15.1:0/512561424 wait complete. 2024-02-28T17:23:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.400+0000 7ff4e6800700 1 Processor -- start 2024-02-28T17:23:21.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.400+0000 7ff4e6800700 1 -- start start 2024-02-28T17:23:21.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.401+0000 7ff4e6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 0x7ff4e0115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:21.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.401+0000 7ff4e6800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4e0105b10 0x7ff4e01161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:21.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.401+0000 7ff4e6800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e01167f0 con 0x7ff4e0105170 2024-02-28T17:23:21.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.401+0000 7ff4e6800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e010fb40 con 0x7ff4e0105b10 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.401+0000 7ff4df400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 0x7ff4e0115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.401+0000 7ff4df400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 0x7ff4e0115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42064/0 (socket says 172.21.15.1:42064) 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.401+0000 7ff4df400700 1 -- 172.21.15.1:0/843173183 learned_addr learned my addr 172.21.15.1:0/843173183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.401+0000 7ff4dea00700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4e0105b10 0x7ff4e01161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4df400700 1 -- 172.21.15.1:0/843173183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4e0105b10 msgr2=0x7ff4e01161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4df400700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4e0105b10 0x7ff4e01161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4df400700 1 -- 172.21.15.1:0/843173183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d0005040 con 0x7ff4e0105170 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4df400700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 0x7ff4e0115c60 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009fd0 tx=0x7ff4d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4d7400700 1 -- 172.21.15.1:0/843173183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d000e040 con 0x7ff4e0105170 2024-02-28T17:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4d7400700 1 -- 172.21.15.1:0/843173183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4d000bbf0 con 0x7ff4e0105170 2024-02-28T17:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4d7400700 1 -- 172.21.15.1:0/843173183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d001c3f0 con 0x7ff4e0105170 2024-02-28T17:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e010fdc0 con 0x7ff4e0105170 2024-02-28T17:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.402+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e0110300 con 0x7ff4e0105170 2024-02-28T17:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.404+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e0102480 con 0x7ff4e0105170 2024-02-28T17:23:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.405+0000 7ff4d7400700 1 -- 172.21.15.1:0/843173183 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff4d000bd60 con 0x7ff4e0105170 2024-02-28T17:23:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.406+0000 7ff4d7400700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff4c406d7f0 0x7ff4c406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.406+0000 7ff4d7400700 1 -- 172.21.15.1:0/843173183 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff4d008e1c0 con 0x7ff4e0105170 2024-02-28T17:23:21.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.406+0000 7ff4dea00700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff4c406d7f0 0x7ff4c406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.407+0000 7ff4dea00700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff4c406d7f0 0x7ff4c406fca0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0111540 tx=0x7ff4cc009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.411+0000 7ff4d7400700 1 -- 172.21.15.1:0/843173183 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff4d0057a70 con 0x7ff4e0105170 2024-02-28T17:23:21.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.655+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4e004bf40 con 0x7ff4c406d7f0 2024-02-28T17:23:21.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.658+0000 7ff4d7400700 1 -- 172.21.15.1:0/843173183 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff4e004bf40 con 0x7ff4c406d7f0 2024-02-28T17:23:21.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff4c406d7f0 msgr2=0x7ff4c406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:21.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff4c406d7f0 0x7ff4c406fca0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0111540 tx=0x7ff4cc009ee0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 msgr2=0x7ff4e0115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:21.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 0x7ff4e0115c60 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009fd0 tx=0x7ff4d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 shutdown_connections 2024-02-28T17:23:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff4c406d7f0 0x7ff4c406fca0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e0105170 0x7ff4e0115c60 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 --2- 172.21.15.1:0/843173183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4e0105b10 0x7ff4e01161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.661+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 >> 172.21.15.1:0/843173183 conn(0x7ff4e00face0 msgr2=0x7ff4e00fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.662+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 shutdown_connections 2024-02-28T17:23:21.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.662+0000 7ff4e6800700 1 -- 172.21.15.1:0/843173183 wait complete. 2024-02-28T17:23:21.682 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:23:21.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.824+0000 7ffb84800700 1 -- 172.21.15.1:0/1710944310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 msgr2=0x7ffb7c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:21.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.824+0000 7ffb84800700 1 --2- 172.21.15.1:0/1710944310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 0x7ffb7c106f40 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c004940 tx=0x7ffb6c009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:21.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.825+0000 7ffb84800700 1 -- 172.21.15.1:0/1710944310 shutdown_connections 2024-02-28T17:23:21.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.825+0000 7ffb84800700 1 --2- 172.21.15.1:0/1710944310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 0x7ffb7c106f40 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.825+0000 7ffb84800700 1 --2- 172.21.15.1:0/1710944310 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c0fdf60 0x7ffb7c0fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.825+0000 7ffb84800700 1 -- 172.21.15.1:0/1710944310 >> 172.21.15.1:0/1710944310 conn(0x7ffb7c0f7a10 msgr2=0x7ffb7c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:21.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.825+0000 7ffb84800700 1 -- 172.21.15.1:0/1710944310 shutdown_connections 2024-02-28T17:23:21.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.825+0000 7ffb84800700 1 -- 172.21.15.1:0/1710944310 wait complete. 2024-02-28T17:23:21.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.826+0000 7ffb84800700 1 Processor -- start 2024-02-28T17:23:21.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.826+0000 7ffb84800700 1 -- start start 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.826+0000 7ffb84800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c0fdf60 0x7ffb7c113b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.826+0000 7ffb84800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 0x7ffb7c114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.826+0000 7ffb84800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1146c0 con 0x7ffb7c0fe900 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.826+0000 7ffb84800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c10d9f0 con 0x7ffb7c0fdf60 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.827+0000 7ffb81400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 0x7ffb7c114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.827+0000 7ffb81400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 0x7ffb7c114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42080/0 (socket says 172.21.15.1:42080) 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.827+0000 7ffb81400700 1 -- 172.21.15.1:0/2322883408 learned_addr learned my addr 172.21.15.1:0/2322883408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.827+0000 7ffb81e00700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c0fdf60 0x7ffb7c113b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.827+0000 7ffb81400700 1 -- 172.21.15.1:0/2322883408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c0fdf60 msgr2=0x7ffb7c113b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.827+0000 7ffb81400700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c0fdf60 0x7ffb7c113b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.827+0000 7ffb81400700 1 -- 172.21.15.1:0/2322883408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c005040 con 0x7ffb7c0fe900 2024-02-28T17:23:21.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.828+0000 7ffb81400700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 0x7ffb7c114070 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009180 tx=0x7ffb6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:21.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.828+0000 7ffb76000700 1 -- 172.21.15.1:0/2322883408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb6c00e040 con 0x7ffb7c0fe900 2024-02-28T17:23:21.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.828+0000 7ffb76000700 1 -- 172.21.15.1:0/2322883408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb6c004290 con 0x7ffb7c0fe900 2024-02-28T17:23:21.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.828+0000 7ffb76000700 1 -- 172.21.15.1:0/2322883408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb6c01c630 con 0x7ffb7c0fe900 2024-02-28T17:23:21.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.828+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb7c10dc70 con 0x7ffb7c0fe900 2024-02-28T17:23:21.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.828+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb7c10e1b0 con 0x7ffb7c0fe900 2024-02-28T17:23:21.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.829+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb7c104550 con 0x7ffb7c0fe900 2024-02-28T17:23:21.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.830+0000 7ffb76000700 1 -- 172.21.15.1:0/2322883408 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffb6c0139f0 con 0x7ffb7c0fe900 2024-02-28T17:23:21.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.831+0000 7ffb76000700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb6406d7f0 0x7ffb6406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:21.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.832+0000 7ffb76000700 1 -- 172.21.15.1:0/2322883408 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffb6c08e3e0 con 0x7ffb7c0fe900 2024-02-28T17:23:21.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.832+0000 7ffb81e00700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb6406d7f0 0x7ffb6406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:21.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.832+0000 7ffb81e00700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb6406d7f0 0x7ffb6406fca0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ffb70004990 tx=0x7ffb70009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:21.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:21.837+0000 7ffb76000700 1 -- 172.21.15.1:0/2322883408 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffb6c05b740 con 0x7ffb7c0fe900 2024-02-28T17:23:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.080+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb7c109ff0 con 0x7ffb6406d7f0 2024-02-28T17:23:22.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.082+0000 7ffb76000700 1 -- 172.21.15.1:0/2322883408 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffb7c109ff0 con 0x7ffb6406d7f0 2024-02-28T17:23:22.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb6406d7f0 msgr2=0x7ffb6406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb6406d7f0 0x7ffb6406fca0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ffb70004990 tx=0x7ffb70009ee0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 msgr2=0x7ffb7c114070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 0x7ffb7c114070 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009180 tx=0x7ffb6c009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:22.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 shutdown_connections 2024-02-28T17:23:22.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb6406d7f0 0x7ffb6406fca0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb7c0fdf60 0x7ffb7c113b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 --2- 172.21.15.1:0/2322883408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb7c0fe900 0x7ffb7c114070 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 >> 172.21.15.1:0/2322883408 conn(0x7ffb7c0f7a10 msgr2=0x7ffb7c1016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:22.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 shutdown_connections 2024-02-28T17:23:22.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.085+0000 7ffb84800700 1 -- 172.21.15.1:0/2322883408 wait complete. 2024-02-28T17:23:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.241+0000 7fbf1ac00700 1 -- 172.21.15.1:0/2552523871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 msgr2=0x7fbf14109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.242+0000 7fbf1ac00700 1 --2- 172.21.15.1:0/2552523871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 0x7fbf14109260 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fbefc004970 tx=0x7fbefc009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.242+0000 7fbf1ac00700 1 -- 172.21.15.1:0/2552523871 shutdown_connections 2024-02-28T17:23:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.242+0000 7fbf1ac00700 1 --2- 172.21.15.1:0/2552523871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 0x7fbf14109260 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.242+0000 7fbf1ac00700 1 --2- 172.21.15.1:0/2552523871 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf14070330 0x7fbf14070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.242+0000 7fbf1ac00700 1 -- 172.21.15.1:0/2552523871 >> 172.21.15.1:0/2552523871 conn(0x7fbf140f8ae0 msgr2=0x7fbf140faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.242+0000 7fbf1ac00700 1 -- 172.21.15.1:0/2552523871 shutdown_connections 2024-02-28T17:23:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.242+0000 7fbf1ac00700 1 -- 172.21.15.1:0/2552523871 wait complete. 2024-02-28T17:23:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.243+0000 7fbf1ac00700 1 Processor -- start 2024-02-28T17:23:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.243+0000 7fbf1ac00700 1 -- start start 2024-02-28T17:23:22.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.243+0000 7fbf1ac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf14070330 0x7fbf14115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:22.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.243+0000 7fbf1ac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 0x7fbf14116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:22.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.243+0000 7fbf1ac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf14116790 con 0x7fbf14070c40 2024-02-28T17:23:22.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.243+0000 7fbf1ac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1410fae0 con 0x7fbf14070330 2024-02-28T17:23:22.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.244+0000 7fbf0be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 0x7fbf14116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:22.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.244+0000 7fbf0be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 0x7fbf14116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44794/0 (socket says 172.21.15.1:44794) 2024-02-28T17:23:22.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.244+0000 7fbf0be00700 1 -- 172.21.15.1:0/1435015534 learned_addr learned my addr 172.21.15.1:0/1435015534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:22.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.244+0000 7fbf13e00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf14070330 0x7fbf14115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:22.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.244+0000 7fbf0be00700 1 -- 172.21.15.1:0/1435015534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf14070330 msgr2=0x7fbf14115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.244+0000 7fbf0be00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf14070330 0x7fbf14115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.244+0000 7fbf0be00700 1 -- 172.21.15.1:0/1435015534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc005040 con 0x7fbf14070c40 2024-02-28T17:23:22.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.244+0000 7fbf13e00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf14070330 0x7fbf14115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:23:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.245+0000 7fbf0be00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 0x7fbf14116140 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fbefc004f50 tx=0x7fbefc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.245+0000 7fbf10c00700 1 -- 172.21.15.1:0/1435015534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbefc00e040 con 0x7fbf14070c40 2024-02-28T17:23:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.245+0000 7fbf10c00700 1 -- 172.21.15.1:0/1435015534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbefc004290 con 0x7fbf14070c40 2024-02-28T17:23:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.245+0000 7fbf10c00700 1 -- 172.21.15.1:0/1435015534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbefc01c630 con 0x7fbf14070c40 2024-02-28T17:23:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.245+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1410fd60 con 0x7fbf14070c40 2024-02-28T17:23:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.245+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf141102a0 con 0x7fbf14070c40 2024-02-28T17:23:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.246+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf14106920 con 0x7fbf14070c40 2024-02-28T17:23:22.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.247+0000 7fbf10c00700 1 -- 172.21.15.1:0/1435015534 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbefc00bc10 con 0x7fbf14070c40 2024-02-28T17:23:22.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.248+0000 7fbf10c00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbef806d7f0 0x7fbef806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:22.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.248+0000 7fbf10c00700 1 -- 172.21.15.1:0/1435015534 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbefc08e280 con 0x7fbf14070c40 2024-02-28T17:23:22.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.249+0000 7fbf13e00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbef806d7f0 0x7fbef806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:22.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.249+0000 7fbf13e00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbef806d7f0 0x7fbef806fca0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009fd0 tx=0x7fbf0400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:22.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.253+0000 7fbf10c00700 1 -- 172.21.15.1:0/1435015534 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbefc057b30 con 0x7fbf14070c40 2024-02-28T17:23:22.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.478+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf1404bed0 con 0x7fbef806d7f0 2024-02-28T17:23:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.508+0000 7fbf10c00700 1 -- 172.21.15.1:0/1435015534 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbf1404bed0 con 0x7fbef806d7f0 2024-02-28T17:23:22.513 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:23:22.513 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (49m) 10m ago 63m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:23:22.513 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (46m) 10m ago 64m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:23:22.513 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (46m) 11m ago 62m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (49m) 10m ago 63m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (42m) 10m ago 56m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (41m) 10m ago 56m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (11m) 11m ago 56m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (26m) 11m ago 56m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (51m) 10m ago 65m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (50m) 11m ago 62m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (47m) 10m ago 65m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (47m) 11m ago 62m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (50m) 10m ago 63m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (50m) 11m ago 62m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:23:22.514 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (45m) 10m ago 61m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:23:22.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (45m) 10m ago 60m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:23:22.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (44m) 10m ago 60m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:23:22.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (44m) 11m ago 59m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:23:22.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (43m) 11m ago 59m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:23:22.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (42m) 11m ago 58m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:23:22.515 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (49m) 10m ago 63m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:23:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbef806d7f0 msgr2=0x7fbef806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbef806d7f0 0x7fbef806fca0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009fd0 tx=0x7fbf0400b040 comp rx=0 tx=0).stop 2024-02-28T17:23:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 msgr2=0x7fbf14116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 0x7fbf14116140 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fbefc004f50 tx=0x7fbefc009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 shutdown_connections 2024-02-28T17:23:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbef806d7f0 0x7fbef806fca0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbf14070330 0x7fbf14115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 --2- 172.21.15.1:0/1435015534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf14070c40 0x7fbf14116140 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 >> 172.21.15.1:0/1435015534 conn(0x7fbf140f8ae0 msgr2=0x7fbf14103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 shutdown_connections 2024-02-28T17:23:22.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.512+0000 7fbf1ac00700 1 -- 172.21.15.1:0/1435015534 wait complete. 2024-02-28T17:23:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.655+0000 7effb2400700 1 -- 172.21.15.1:0/3532398335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 msgr2=0x7effac104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.655+0000 7effb2400700 1 --2- 172.21.15.1:0/3532398335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 0x7effac104210 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7eff98004940 tx=0x7eff98009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.655+0000 7effb2400700 1 -- 172.21.15.1:0/3532398335 shutdown_connections 2024-02-28T17:23:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.655+0000 7effb2400700 1 --2- 172.21.15.1:0/3532398335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 0x7effac104210 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.655+0000 7effb2400700 1 --2- 172.21.15.1:0/3532398335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effac1095f0 0x7effac1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.655+0000 7effb2400700 1 -- 172.21.15.1:0/3532398335 >> 172.21.15.1:0/3532398335 conn(0x7effac072d50 msgr2=0x7effac075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.656+0000 7effb2400700 1 -- 172.21.15.1:0/3532398335 shutdown_connections 2024-02-28T17:23:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.656+0000 7effb2400700 1 -- 172.21.15.1:0/3532398335 wait complete. 2024-02-28T17:23:22.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.656+0000 7effb2400700 1 Processor -- start 2024-02-28T17:23:22.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.656+0000 7effb2400700 1 -- start start 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.657+0000 7effb2400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 0x7effac11a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.657+0000 7effb2400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effac1095f0 0x7effac11a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.657+0000 7effb2400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac11ab90 con 0x7effac103dc0 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.657+0000 7effb2400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac113ee0 con 0x7effac1095f0 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.657+0000 7effab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 0x7effac11a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.657+0000 7effab400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 0x7effac11a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44824/0 (socket says 172.21.15.1:44824) 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.657+0000 7effab400700 1 -- 172.21.15.1:0/3757316123 learned_addr learned my addr 172.21.15.1:0/3757316123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effab400700 1 -- 172.21.15.1:0/3757316123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effac1095f0 msgr2=0x7effac11a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effaaa00700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effac1095f0 0x7effac11a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effab400700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effac1095f0 0x7effac11a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effab400700 1 -- 172.21.15.1:0/3757316123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98005040 con 0x7effac103dc0 2024-02-28T17:23:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effab400700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 0x7effac11a000 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00eb40 tx=0x7eff9c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effa3400700 1 -- 172.21.15.1:0/3757316123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9c007cc0 con 0x7effac103dc0 2024-02-28T17:23:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effac114160 con 0x7effac103dc0 2024-02-28T17:23:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effa3400700 1 -- 172.21.15.1:0/3757316123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff9c00f040 con 0x7effac103dc0 2024-02-28T17:23:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effa3400700 1 -- 172.21.15.1:0/3757316123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9c009620 con 0x7effac103dc0 2024-02-28T17:23:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.658+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effac1146a0 con 0x7effac103dc0 2024-02-28T17:23:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.659+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effac075870 con 0x7effac103dc0 2024-02-28T17:23:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.660+0000 7effa3400700 1 -- 172.21.15.1:0/3757316123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7eff9c009780 con 0x7effac103dc0 2024-02-28T17:23:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.661+0000 7effa3400700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7eff9006d7f0 0x7eff9006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.661+0000 7effa3400700 1 -- 172.21.15.1:0/3757316123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7eff9c08cb40 con 0x7effac103dc0 2024-02-28T17:23:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.662+0000 7effaaa00700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7eff9006d7f0 0x7eff9006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:22.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.662+0000 7effaaa00700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7eff9006d7f0 0x7eff9006fca0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7eff98000c00 tx=0x7eff98009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:22.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.666+0000 7effa3400700 1 -- 172.21.15.1:0/3757316123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7eff9c056420 con 0x7effac103dc0 2024-02-28T17:23:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.973+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7effac115620 con 0x7effac103dc0 2024-02-28T17:23:22.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.974+0000 7effa3400700 1 -- 172.21.15.1:0/3757316123 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7eff9c059ae0 con 0x7effac103dc0 2024-02-28T17:23:22.978 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:23:22.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:23:22.978 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:23:22.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:23:22.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:23:22.978 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:23:22.979 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.976+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7eff9006d7f0 msgr2=0x7eff9006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.976+0000 7effb2400700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7eff9006d7f0 0x7eff9006fca0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7eff98000c00 tx=0x7eff98009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.976+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 msgr2=0x7effac11a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.976+0000 7effb2400700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 0x7effac11a000 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00eb40 tx=0x7eff9c00b040 comp rx=0 tx=0).stop 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.977+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 shutdown_connections 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.977+0000 7effb2400700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7eff9006d7f0 0x7eff9006fca0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.977+0000 7effb2400700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effac103dc0 0x7effac11a000 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.977+0000 7effb2400700 1 --2- 172.21.15.1:0/3757316123 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7effac1095f0 0x7effac11a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.977+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 >> 172.21.15.1:0/3757316123 conn(0x7effac072d50 msgr2=0x7effac074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:22.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.977+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 shutdown_connections 2024-02-28T17:23:22.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:22.977+0000 7effb2400700 1 -- 172.21.15.1:0/3757316123 wait complete. 2024-02-28T17:23:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.131+0000 7f8783600700 1 -- 172.21.15.1:0/3809052163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 msgr2=0x7f877c0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.131+0000 7f8783600700 1 --2- 172.21.15.1:0/3809052163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 0x7f877c0fe3b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f8760004970 tx=0x7f8760009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.131+0000 7f8783600700 1 -- 172.21.15.1:0/3809052163 shutdown_connections 2024-02-28T17:23:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.131+0000 7f8783600700 1 --2- 172.21.15.1:0/3809052163 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f877c0fe8f0 0x7f877c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.131+0000 7f8783600700 1 --2- 172.21.15.1:0/3809052163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 0x7f877c0fe3b0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.131+0000 7f8783600700 1 -- 172.21.15.1:0/3809052163 >> 172.21.15.1:0/3809052163 conn(0x7f877c071d90 msgr2=0x7f877c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.131+0000 7f8783600700 1 -- 172.21.15.1:0/3809052163 shutdown_connections 2024-02-28T17:23:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.131+0000 7f8783600700 1 -- 172.21.15.1:0/3809052163 wait complete. 2024-02-28T17:23:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.133+0000 7f8783600700 1 Processor -- start 2024-02-28T17:23:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.133+0000 7f8783600700 1 -- start start 2024-02-28T17:23:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.133+0000 7f8783600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 0x7f877c0ff070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.133+0000 7f8783600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f877c0fe8f0 0x7f877c0ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.133+0000 7f8783600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c0ffaf0 con 0x7f877c0fdfe0 2024-02-28T17:23:23.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.133+0000 7f8783600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c0ffc30 con 0x7f877c0fe8f0 2024-02-28T17:23:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.134+0000 7f8780c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 0x7f877c0ff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.134+0000 7f8780c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 0x7f877c0ff070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44840/0 (socket says 172.21.15.1:44840) 2024-02-28T17:23:23.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.134+0000 7f8780c00700 1 -- 172.21.15.1:0/949308001 learned_addr learned my addr 172.21.15.1:0/949308001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.134+0000 7f8780c00700 1 -- 172.21.15.1:0/949308001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f877c0fe8f0 msgr2=0x7f877c0ff5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.134+0000 7f8780c00700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f877c0fe8f0 0x7f877c0ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.134+0000 7f8780c00700 1 -- 172.21.15.1:0/949308001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8760005040 con 0x7f877c0fdfe0 2024-02-28T17:23:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.134+0000 7f8780c00700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 0x7f877c0ff070 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f8760004f50 tx=0x7f8760009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.135+0000 7f8779600700 1 -- 172.21.15.1:0/949308001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f876000e040 con 0x7f877c0fdfe0 2024-02-28T17:23:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.135+0000 7f8779600700 1 -- 172.21.15.1:0/949308001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8760004290 con 0x7f877c0fdfe0 2024-02-28T17:23:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.135+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c0ffeb0 con 0x7f877c0fdfe0 2024-02-28T17:23:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.135+0000 7f8779600700 1 -- 172.21.15.1:0/949308001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f876001c630 con 0x7f877c0fdfe0 2024-02-28T17:23:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.135+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c11e570 con 0x7f877c0fdfe0 2024-02-28T17:23:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.136+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c106b20 con 0x7f877c0fdfe0 2024-02-28T17:23:23.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.137+0000 7f8779600700 1 -- 172.21.15.1:0/949308001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f876000bc10 con 0x7f877c0fdfe0 2024-02-28T17:23:23.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.138+0000 7f8779600700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875806d7f0 0x7f875806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.139+0000 7f8779600700 1 -- 172.21.15.1:0/949308001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f876008e330 con 0x7f877c0fdfe0 2024-02-28T17:23:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.139+0000 7f8773e00700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875806d7f0 0x7f875806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.140+0000 7f8773e00700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875806d7f0 0x7f875806fca0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f877c100a70 tx=0x7f8768007d40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.144+0000 7f8779600700 1 -- 172.21.15.1:0/949308001 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8760057b60 con 0x7f877c0fdfe0 2024-02-28T17:23:23.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:22 smithi066 ceph-mon[85781]: from='client.36580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:23.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:22 smithi066 ceph-mon[85781]: pgmap v1471: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:23.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:22 smithi066 ceph-mon[85781]: from='client.36584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:23.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:22 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3757316123' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:23:23.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:22 smithi001 ceph-mon[112940]: from='client.36580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:23.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:22 smithi001 ceph-mon[112940]: pgmap v1471: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:23.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:22 smithi001 ceph-mon[112940]: from='client.36584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:23.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:22 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3757316123' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:23:23.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.409+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f877c1007b0 con 0x7f877c0fdfe0 2024-02-28T17:23:23.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.410+0000 7f8779600700 1 -- 172.21.15.1:0/949308001 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f8760013e60 con 0x7f877c0fdfe0 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:23:23.415 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:23:23.416 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:23:23.417 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:23:23.418 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:23:23.418 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:23:23.418 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875806d7f0 msgr2=0x7f875806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875806d7f0 0x7f875806fca0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f877c100a70 tx=0x7f8768007d40 comp rx=0 tx=0).stop 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 msgr2=0x7f877c0ff070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 0x7f877c0ff070 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f8760004f50 tx=0x7f8760009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 shutdown_connections 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f875806d7f0 0x7f875806fca0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f877c0fdfe0 0x7f877c0ff070 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 --2- 172.21.15.1:0/949308001 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f877c0fe8f0 0x7f877c0ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 >> 172.21.15.1:0/949308001 conn(0x7f877c071d90 msgr2=0x7f877c0fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.413+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 shutdown_connections 2024-02-28T17:23:23.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.414+0000 7f8783600700 1 -- 172.21.15.1:0/949308001 wait complete. 2024-02-28T17:23:23.420 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:23:23.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.571+0000 7f3835a00700 1 -- 172.21.15.1:0/3971186301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 msgr2=0x7f38300ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:23.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.571+0000 7f3835a00700 1 --2- 172.21.15.1:0/3971186301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 0x7f38300ff180 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f381c004940 tx=0x7f381c009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:23.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.571+0000 7f3835a00700 1 -- 172.21.15.1:0/3971186301 shutdown_connections 2024-02-28T17:23:23.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.571+0000 7f3835a00700 1 --2- 172.21.15.1:0/3971186301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 0x7f38300ff180 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.571+0000 7f3835a00700 1 --2- 172.21.15.1:0/3971186301 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3830105170 0x7f3830105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.571+0000 7f3835a00700 1 -- 172.21.15.1:0/3971186301 >> 172.21.15.1:0/3971186301 conn(0x7f38300face0 msgr2=0x7f38300fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:23.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.571+0000 7f3835a00700 1 -- 172.21.15.1:0/3971186301 shutdown_connections 2024-02-28T17:23:23.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.571+0000 7f3835a00700 1 -- 172.21.15.1:0/3971186301 wait complete. 2024-02-28T17:23:23.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.572+0000 7f3835a00700 1 Processor -- start 2024-02-28T17:23:23.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.573+0000 7f3835a00700 1 -- start start 2024-02-28T17:23:23.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.573+0000 7f3835a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3830105170 0x7f3830115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:23.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.573+0000 7f3835a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 0x7f3830116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:23.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.573+0000 7f3835a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3830116870 con 0x7f3830105b10 2024-02-28T17:23:23.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.573+0000 7f3835a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383010fb80 con 0x7f3830105170 2024-02-28T17:23:23.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.573+0000 7f382e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 0x7f3830116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:23.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.573+0000 7f382e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 0x7f3830116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44846/0 (socket says 172.21.15.1:44846) 2024-02-28T17:23:23.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.573+0000 7f382e000700 1 -- 172.21.15.1:0/2473790919 learned_addr learned my addr 172.21.15.1:0/2473790919 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:23.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.574+0000 7f382ea00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3830105170 0x7f3830115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:23.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.574+0000 7f382e000700 1 -- 172.21.15.1:0/2473790919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3830105170 msgr2=0x7f3830115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.574+0000 7f382e000700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3830105170 0x7f3830115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.574+0000 7f382e000700 1 -- 172.21.15.1:0/2473790919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c005040 con 0x7f3830105b10 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.574+0000 7f382e000700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 0x7f3830116220 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f381c0090d0 tx=0x7f381c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.574+0000 7f3826a00700 1 -- 172.21.15.1:0/2473790919 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f381c00e040 con 0x7f3830105b10 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.575+0000 7f3826a00700 1 -- 172.21.15.1:0/2473790919 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f381c00bbf0 con 0x7f3830105b10 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.575+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383010fe00 con 0x7f3830105b10 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.575+0000 7f3826a00700 1 -- 172.21.15.1:0/2473790919 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f381c01c630 con 0x7f3830105b10 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.575+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3830110340 con 0x7f3830105b10 2024-02-28T17:23:23.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.576+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3830107640 con 0x7f3830105b10 2024-02-28T17:23:23.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.577+0000 7f3826a00700 1 -- 172.21.15.1:0/2473790919 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f381c00bd60 con 0x7f3830105b10 2024-02-28T17:23:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.578+0000 7f3826a00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f381406d7e0 0x7f381406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.578+0000 7f3826a00700 1 -- 172.21.15.1:0/2473790919 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f381c08e2e0 con 0x7f3830105b10 2024-02-28T17:23:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.578+0000 7f382ea00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f381406d7e0 0x7f381406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:23.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.579+0000 7f382ea00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f381406d7e0 0x7f381406fc90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3820009fd0 tx=0x7f3820009e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:23.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.583+0000 7f3826a00700 1 -- 172.21.15.1:0/2473790919 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f381c05b640 con 0x7f3830105b10 2024-02-28T17:23:23.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.836+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f383004bed0 con 0x7f381406d7e0 2024-02-28T17:23:23.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.838+0000 7f3826a00700 1 -- 172.21.15.1:0/2473790919 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f383004bed0 con 0x7f381406d7e0 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:23:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:23:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:23:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:23:23.844 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f381406d7e0 msgr2=0x7f381406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f381406d7e0 0x7f381406fc90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3820009fd0 tx=0x7f3820009e50 comp rx=0 tx=0).stop 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 msgr2=0x7f3830116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 0x7f3830116220 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f381c0090d0 tx=0x7f381c009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 shutdown_connections 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f381406d7e0 0x7f381406fc90 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3830105170 0x7f3830115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 --2- 172.21.15.1:0/2473790919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3830105b10 0x7f3830116220 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.841+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 >> 172.21.15.1:0/2473790919 conn(0x7f38300face0 msgr2=0x7f38300fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.842+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 shutdown_connections 2024-02-28T17:23:23.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:23.842+0000 7f3835a00700 1 -- 172.21.15.1:0/2473790919 wait complete. 2024-02-28T17:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.002+0000 7f6483600700 1 -- 172.21.15.1:0/1025363408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c102f50 msgr2=0x7f647c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.002+0000 7f6483600700 1 --2- 172.21.15.1:0/1025363408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c102f50 0x7f647c103320 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f6470004940 tx=0x7f6470009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.002+0000 7f6483600700 1 -- 172.21.15.1:0/1025363408 shutdown_connections 2024-02-28T17:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.002+0000 7f6483600700 1 --2- 172.21.15.1:0/1025363408 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f647c1038f0 0x7f647c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.002+0000 7f6483600700 1 --2- 172.21.15.1:0/1025363408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c102f50 0x7f647c103320 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.002+0000 7f6483600700 1 -- 172.21.15.1:0/1025363408 >> 172.21.15.1:0/1025363408 conn(0x7f647c0f8ae0 msgr2=0x7f647c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.003+0000 7f6483600700 1 -- 172.21.15.1:0/1025363408 shutdown_connections 2024-02-28T17:23:24.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.003+0000 7f6483600700 1 -- 172.21.15.1:0/1025363408 wait complete. 2024-02-28T17:23:24.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.003+0000 7f6483600700 1 Processor -- start 2024-02-28T17:23:24.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.004+0000 7f6483600700 1 -- start start 2024-02-28T17:23:24.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.004+0000 7f6483600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f647c102f50 0x7f647c115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:24.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.004+0000 7f6483600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c1038f0 0x7f647c116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:24.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.004+0000 7f6483600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c1167b0 con 0x7f647c1038f0 2024-02-28T17:23:24.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.004+0000 7f6483600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c10fb00 con 0x7f647c102f50 2024-02-28T17:23:24.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.004+0000 7f647be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c1038f0 0x7f647c116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:24.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.004+0000 7f647be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c1038f0 0x7f647c116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44866/0 (socket says 172.21.15.1:44866) 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.004+0000 7f647be00700 1 -- 172.21.15.1:0/1626990891 learned_addr learned my addr 172.21.15.1:0/1626990891 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.005+0000 7f6480c00700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f647c102f50 0x7f647c115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.005+0000 7f647be00700 1 -- 172.21.15.1:0/1626990891 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f647c102f50 msgr2=0x7f647c115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.005+0000 7f647be00700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f647c102f50 0x7f647c115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.005+0000 7f647be00700 1 -- 172.21.15.1:0/1626990891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470005040 con 0x7f647c1038f0 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.005+0000 7f647be00700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c1038f0 0x7f647c116160 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f6468002a50 tx=0x7f6468004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.005+0000 7f6478c00700 1 -- 172.21.15.1:0/1626990891 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6468007cc0 con 0x7f647c1038f0 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.005+0000 7f6478c00700 1 -- 172.21.15.1:0/1626990891 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f646800f040 con 0x7f647c1038f0 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.006+0000 7f6478c00700 1 -- 172.21.15.1:0/1626990891 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f646800daf0 con 0x7f647c1038f0 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.006+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c10fde0 con 0x7f647c1038f0 2024-02-28T17:23:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.006+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c110320 con 0x7f647c1038f0 2024-02-28T17:23:24.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.007+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c0fc6e0 con 0x7f647c1038f0 2024-02-28T17:23:24.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.008+0000 7f6478c00700 1 -- 172.21.15.1:0/1626990891 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6468009810 con 0x7f647c1038f0 2024-02-28T17:23:24.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.009+0000 7f6478c00700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f646406d7f0 0x7f646406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:24.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.009+0000 7f6478c00700 1 -- 172.21.15.1:0/1626990891 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f646808cc50 con 0x7f647c1038f0 2024-02-28T17:23:24.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.010+0000 7f6480c00700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f646406d7f0 0x7f646406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:24.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.010+0000 7f6480c00700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f646406d7f0 0x7f646406fca0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f647000b3b0 tx=0x7f6470009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:24.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.013+0000 7f6478c00700 1 -- 172.21.15.1:0/1626990891 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f64680564a0 con 0x7f647c1038f0 2024-02-28T17:23:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:23 smithi066 ceph-mon[85781]: from='client.36588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:23 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/949308001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:23:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:23 smithi001 ceph-mon[112940]: from='client.36588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:23 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/949308001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:23:24.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.333+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f647c111200 con 0x7f647c1038f0 2024-02-28T17:23:24.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.334+0000 7f6478c00700 1 -- 172.21.15.1:0/1626990891 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f6468059b60 con 0x7f647c1038f0 2024-02-28T17:23:24.338 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:23:24.338 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:23:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:23:24.338 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:23:24.339 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:23:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:23:24.340 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:23:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:23:24.340 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:23:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f646406d7f0 msgr2=0x7f646406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f646406d7f0 0x7f646406fca0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f647000b3b0 tx=0x7f6470009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c1038f0 msgr2=0x7f647c116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c1038f0 0x7f647c116160 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f6468002a50 tx=0x7f6468004600 comp rx=0 tx=0).stop 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 shutdown_connections 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f646406d7f0 0x7f646406fca0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f647c102f50 0x7f647c115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 --2- 172.21.15.1:0/1626990891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f647c1038f0 0x7f647c116160 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.337+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 >> 172.21.15.1:0/1626990891 conn(0x7f647c0f8ae0 msgr2=0x7f647c0fa3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.338+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 shutdown_connections 2024-02-28T17:23:24.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:24.338+0000 7f6483600700 1 -- 172.21.15.1:0/1626990891 wait complete. 2024-02-28T17:23:25.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:25 smithi066 ceph-mon[85781]: pgmap v1472: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:25.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:25 smithi066 ceph-mon[85781]: from='client.36600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:25.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:25 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1626990891' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:23:25.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:25 smithi001 ceph-mon[112940]: pgmap v1472: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:25.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:25 smithi001 ceph-mon[112940]: from='client.36600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:25.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:25 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1626990891' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:23:26.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:23:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:23:27.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:27 smithi001 ceph-mon[112940]: pgmap v1473: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:27.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:27 smithi066 ceph-mon[85781]: pgmap v1473: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:29.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:29 smithi001 ceph-mon[112940]: pgmap v1474: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:29.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:29 smithi066 ceph-mon[85781]: pgmap v1474: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:31.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:31 smithi001 ceph-mon[112940]: pgmap v1475: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:31.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:31 smithi066 ceph-mon[85781]: pgmap v1475: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:33.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:33 smithi066 ceph-mon[85781]: pgmap v1476: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:33.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:33 smithi001 ceph-mon[112940]: pgmap v1476: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:35.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:35 smithi066 ceph-mon[85781]: pgmap v1477: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:35.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:35 smithi001 ceph-mon[112940]: pgmap v1477: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:37.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:37 smithi066 ceph-mon[85781]: pgmap v1478: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:37.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:37 smithi001 ceph-mon[112940]: pgmap v1478: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:39.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:39 smithi066 ceph-mon[85781]: pgmap v1479: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:39.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:39 smithi001 ceph-mon[112940]: pgmap v1479: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:41.315 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:41 smithi066 ceph-mon[85781]: pgmap v1480: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:41.315 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:23:41.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:41 smithi001 ceph-mon[112940]: pgmap v1480: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:41.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:23:43.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:43 smithi066 ceph-mon[85781]: pgmap v1481: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:43.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:43 smithi001 ceph-mon[112940]: pgmap v1481: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:45.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:45 smithi066 ceph-mon[85781]: pgmap v1482: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:45.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:45 smithi001 ceph-mon[112940]: pgmap v1482: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:47.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:47 smithi066 ceph-mon[85781]: pgmap v1483: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:47.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:47 smithi001 ceph-mon[112940]: pgmap v1483: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:49.339 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:49 smithi001 ceph-mon[112940]: pgmap v1484: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:49.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:49 smithi066 ceph-mon[85781]: pgmap v1484: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:51.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:51 smithi066 ceph-mon[85781]: pgmap v1485: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:51.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:51 smithi001 ceph-mon[112940]: pgmap v1485: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:53.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:53 smithi066 ceph-mon[85781]: pgmap v1486: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:53.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:53 smithi001 ceph-mon[112940]: pgmap v1486: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:54.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.510+0000 7f97eaa00700 1 -- 172.21.15.1:0/2924516898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e4103dc0 msgr2=0x7f97e4104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:54.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.510+0000 7f97eaa00700 1 --2- 172.21.15.1:0/2924516898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e4103dc0 0x7f97e4104210 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f97d0004940 tx=0x7f97d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:54.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.510+0000 7f97eaa00700 1 -- 172.21.15.1:0/2924516898 shutdown_connections 2024-02-28T17:23:54.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.510+0000 7f97eaa00700 1 --2- 172.21.15.1:0/2924516898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e4103dc0 0x7f97e4104210 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.510+0000 7f97eaa00700 1 --2- 172.21.15.1:0/2924516898 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97e41095f0 0x7f97e41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.510+0000 7f97eaa00700 1 -- 172.21.15.1:0/2924516898 >> 172.21.15.1:0/2924516898 conn(0x7f97e4072d50 msgr2=0x7f97e4075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:54.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.510+0000 7f97eaa00700 1 -- 172.21.15.1:0/2924516898 shutdown_connections 2024-02-28T17:23:54.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.510+0000 7f97eaa00700 1 -- 172.21.15.1:0/2924516898 wait complete. 2024-02-28T17:23:54.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.511+0000 7f97eaa00700 1 Processor -- start 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.512+0000 7f97eaa00700 1 -- start start 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.512+0000 7f97eaa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97e4103dc0 0x7f97e411a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.512+0000 7f97eaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e41095f0 0x7f97e411a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.512+0000 7f97eaa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e411abe0 con 0x7f97e41095f0 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.512+0000 7f97eaa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e4113ef0 con 0x7f97e4103dc0 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.512+0000 7f97e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e41095f0 0x7f97e411a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.512+0000 7f97e3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e41095f0 0x7f97e411a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38026/0 (socket says 172.21.15.1:38026) 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.512+0000 7f97e3400700 1 -- 172.21.15.1:0/256538040 learned_addr learned my addr 172.21.15.1:0/256538040 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.513+0000 7f97e3e00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97e4103dc0 0x7f97e411a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.513+0000 7f97e3400700 1 -- 172.21.15.1:0/256538040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97e4103dc0 msgr2=0x7f97e411a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.513+0000 7f97e3400700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97e4103dc0 0x7f97e411a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.513+0000 7f97e3400700 1 -- 172.21.15.1:0/256538040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d0005040 con 0x7f97e41095f0 2024-02-28T17:23:54.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.513+0000 7f97e3400700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e41095f0 0x7f97e411a590 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009150 tx=0x7f97d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:54.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.513+0000 7f97dbe00700 1 -- 172.21.15.1:0/256538040 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97d000e040 con 0x7f97e41095f0 2024-02-28T17:23:54.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.514+0000 7f97dbe00700 1 -- 172.21.15.1:0/256538040 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97d000bbf0 con 0x7f97e41095f0 2024-02-28T17:23:54.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.514+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e4114170 con 0x7f97e41095f0 2024-02-28T17:23:54.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.514+0000 7f97dbe00700 1 -- 172.21.15.1:0/256538040 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97d001c630 con 0x7f97e41095f0 2024-02-28T17:23:54.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.514+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e41146b0 con 0x7f97e41095f0 2024-02-28T17:23:54.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.515+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e410b9c0 con 0x7f97e41095f0 2024-02-28T17:23:54.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.516+0000 7f97dbe00700 1 -- 172.21.15.1:0/256538040 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f97d000bd60 con 0x7f97e41095f0 2024-02-28T17:23:54.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.517+0000 7f97dbe00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97c806d7f0 0x7f97c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:54.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.517+0000 7f97dbe00700 1 -- 172.21.15.1:0/256538040 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f97d008e390 con 0x7f97e41095f0 2024-02-28T17:23:54.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.517+0000 7f97e3e00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97c806d7f0 0x7f97c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:54.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.518+0000 7f97e3e00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97c806d7f0 0x7f97c806fca0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009fd0 tx=0x7f97d4009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:54.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.522+0000 7f97dbe00700 1 -- 172.21.15.1:0/256538040 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f97d005b6f0 con 0x7f97e41095f0 2024-02-28T17:23:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.799+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97e404bed0 con 0x7f97c806d7f0 2024-02-28T17:23:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.801+0000 7f97dbe00700 1 -- 172.21.15.1:0/256538040 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f97e404bed0 con 0x7f97c806d7f0 2024-02-28T17:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97c806d7f0 msgr2=0x7f97c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97c806d7f0 0x7f97c806fca0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009fd0 tx=0x7f97d4009f20 comp rx=0 tx=0).stop 2024-02-28T17:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e41095f0 msgr2=0x7f97e411a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:54.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e41095f0 0x7f97e411a590 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009150 tx=0x7f97d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 shutdown_connections 2024-02-28T17:23:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97c806d7f0 0x7f97c806fca0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97e4103dc0 0x7f97e411a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 --2- 172.21.15.1:0/256538040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97e41095f0 0x7f97e411a590 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.804+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 >> 172.21.15.1:0/256538040 conn(0x7f97e4072d50 msgr2=0x7f97e4074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.805+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 shutdown_connections 2024-02-28T17:23:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.805+0000 7f97eaa00700 1 -- 172.21.15.1:0/256538040 wait complete. 2024-02-28T17:23:54.823 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:23:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.967+0000 7f37b2000700 1 -- 172.21.15.1:0/2013798804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac1095f0 msgr2=0x7f37ac1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.967+0000 7f37b2000700 1 --2- 172.21.15.1:0/2013798804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac1095f0 0x7f37ac1099c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f379c004940 tx=0x7f379c009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:54.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.967+0000 7f37b2000700 1 -- 172.21.15.1:0/2013798804 shutdown_connections 2024-02-28T17:23:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.967+0000 7f37b2000700 1 --2- 172.21.15.1:0/2013798804 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37ac103dc0 0x7f37ac104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.967+0000 7f37b2000700 1 --2- 172.21.15.1:0/2013798804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac1095f0 0x7f37ac1099c0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.967+0000 7f37b2000700 1 -- 172.21.15.1:0/2013798804 >> 172.21.15.1:0/2013798804 conn(0x7f37ac072d50 msgr2=0x7f37ac075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.967+0000 7f37b2000700 1 -- 172.21.15.1:0/2013798804 shutdown_connections 2024-02-28T17:23:54.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.967+0000 7f37b2000700 1 -- 172.21.15.1:0/2013798804 wait complete. 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.968+0000 7f37b2000700 1 Processor -- start 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.969+0000 7f37b2000700 1 -- start start 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.969+0000 7f37b2000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37ac103dc0 0x7f37ac11a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.969+0000 7f37b2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac11a720 0x7f37ac114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.969+0000 7f37b2000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac11aca0 con 0x7f37ac11a720 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.969+0000 7f37b2000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac11ae10 con 0x7f37ac103dc0 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.969+0000 7f37aaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac11a720 0x7f37ac114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.969+0000 7f37aaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac11a720 0x7f37ac114080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38042/0 (socket says 172.21.15.1:38042) 2024-02-28T17:23:54.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.969+0000 7f37aaa00700 1 -- 172.21.15.1:0/3958145903 learned_addr learned my addr 172.21.15.1:0/3958145903 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37ab400700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37ac103dc0 0x7f37ac11a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:54.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37aaa00700 1 -- 172.21.15.1:0/3958145903 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37ac103dc0 msgr2=0x7f37ac11a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37aaa00700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37ac103dc0 0x7f37ac11a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37aaa00700 1 -- 172.21.15.1:0/3958145903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c005040 con 0x7f37ac11a720 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37aaa00700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac11a720 0x7f37ac114080 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f3798002a50 tx=0x7f37980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37a3400700 1 -- 172.21.15.1:0/3958145903 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3798007cc0 con 0x7f37ac11a720 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37a3400700 1 -- 172.21.15.1:0/3958145903 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f379800f040 con 0x7f37ac11a720 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37a3400700 1 -- 172.21.15.1:0/3958145903 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3798009640 con 0x7f37ac11a720 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.970+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac114680 con 0x7f37ac11a720 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.971+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac114bc0 con 0x7f37ac11a720 2024-02-28T17:23:54.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.972+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac106850 con 0x7f37ac11a720 2024-02-28T17:23:54.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.973+0000 7f37a3400700 1 -- 172.21.15.1:0/3958145903 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3798009800 con 0x7f37ac11a720 2024-02-28T17:23:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.974+0000 7f37a3400700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f379006d7f0 0x7f379006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.974+0000 7f37a3400700 1 -- 172.21.15.1:0/3958145903 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f379808cd10 con 0x7f37ac11a720 2024-02-28T17:23:54.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.975+0000 7f37ab400700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f379006d7f0 0x7f379006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:54.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.975+0000 7f37ab400700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f379006d7f0 0x7f379006fca0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f379c000c00 tx=0x7f379c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:54.979+0000 7f37a3400700 1 -- 172.21.15.1:0/3958145903 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3798059ff0 con 0x7f37ac11a720 2024-02-28T17:23:55.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.224+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37ac04bf70 con 0x7f379006d7f0 2024-02-28T17:23:55.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.227+0000 7f37a3400700 1 -- 172.21.15.1:0/3958145903 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f379c007cc0 con 0x7f379006d7f0 2024-02-28T17:23:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f379006d7f0 msgr2=0x7f379006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f379006d7f0 0x7f379006fca0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f379c000c00 tx=0x7f379c009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac11a720 msgr2=0x7f37ac114080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac11a720 0x7f37ac114080 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f3798002a50 tx=0x7f37980046b0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 shutdown_connections 2024-02-28T17:23:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f379006d7f0 0x7f379006fca0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f37ac103dc0 0x7f37ac11a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 --2- 172.21.15.1:0/3958145903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37ac11a720 0x7f37ac114080 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 >> 172.21.15.1:0/3958145903 conn(0x7f37ac072d50 msgr2=0x7f37ac074af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 shutdown_connections 2024-02-28T17:23:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.230+0000 7f37b2000700 1 -- 172.21.15.1:0/3958145903 wait complete. 2024-02-28T17:23:55.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.378+0000 7f0881600700 1 -- 172.21.15.1:0/1972029579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 msgr2=0x7f087c105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:55.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.378+0000 7f0881600700 1 --2- 172.21.15.1:0/1972029579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 0x7f087c105540 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f086c004940 tx=0x7f086c009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:55.381 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:55 smithi001 ceph-mon[112940]: pgmap v1487: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.378+0000 7f0881600700 1 -- 172.21.15.1:0/1972029579 shutdown_connections 2024-02-28T17:23:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.378+0000 7f0881600700 1 --2- 172.21.15.1:0/1972029579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f087c105b10 0x7f087c0ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.378+0000 7f0881600700 1 --2- 172.21.15.1:0/1972029579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 0x7f087c105540 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.378+0000 7f0881600700 1 -- 172.21.15.1:0/1972029579 >> 172.21.15.1:0/1972029579 conn(0x7f087c0face0 msgr2=0x7f087c0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.378+0000 7f0881600700 1 -- 172.21.15.1:0/1972029579 shutdown_connections 2024-02-28T17:23:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.378+0000 7f0881600700 1 -- 172.21.15.1:0/1972029579 wait complete. 2024-02-28T17:23:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.379+0000 7f0881600700 1 Processor -- start 2024-02-28T17:23:55.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.379+0000 7f0881600700 1 -- start start 2024-02-28T17:23:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f0881600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 0x7f087c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:55.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f0881600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f087c105b10 0x7f087c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f0881600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c1168a0 con 0x7f087c105170 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f0881600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c10fb40 con 0x7f087c105b10 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f087aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 0x7f087c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f087aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 0x7f087c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38062/0 (socket says 172.21.15.1:38062) 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f087aa00700 1 -- 172.21.15.1:0/2328625835 learned_addr learned my addr 172.21.15.1:0/2328625835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f087a000700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f087c105b10 0x7f087c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f087aa00700 1 -- 172.21.15.1:0/2328625835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f087c105b10 msgr2=0x7f087c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f087aa00700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f087c105b10 0x7f087c1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f087aa00700 1 -- 172.21.15.1:0/2328625835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c005040 con 0x7f087c105170 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.380+0000 7f087a000700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f087c105b10 0x7f087c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.381+0000 7f087aa00700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 0x7f087c115c80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f086c009150 tx=0x7f086c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:55.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.381+0000 7f0872a00700 1 -- 172.21.15.1:0/2328625835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f086c00e040 con 0x7f087c105170 2024-02-28T17:23:55.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.381+0000 7f0872a00700 1 -- 172.21.15.1:0/2328625835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f086c00bbf0 con 0x7f087c105170 2024-02-28T17:23:55.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.381+0000 7f0872a00700 1 -- 172.21.15.1:0/2328625835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f086c01c630 con 0x7f087c105170 2024-02-28T17:23:55.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.381+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c10fdc0 con 0x7f087c105170 2024-02-28T17:23:55.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.381+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c110300 con 0x7f087c105170 2024-02-28T17:23:55.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.382+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c0fe900 con 0x7f087c105170 2024-02-28T17:23:55.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.383+0000 7f0872a00700 1 -- 172.21.15.1:0/2328625835 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f086c00bd60 con 0x7f087c105170 2024-02-28T17:23:55.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.385+0000 7f0872a00700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f086006d7f0 0x7f086006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.385+0000 7f0872a00700 1 -- 172.21.15.1:0/2328625835 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f086c08e290 con 0x7f087c105170 2024-02-28T17:23:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.385+0000 7f087a000700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f086006d7f0 0x7f086006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:55.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.386+0000 7f087a000700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f086006d7f0 0x7f086006fca0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0864006fd0 tx=0x7f0864008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:55.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.390+0000 7f0872a00700 1 -- 172.21.15.1:0/2328625835 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f086c05b670 con 0x7f087c105170 2024-02-28T17:23:55.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:55 smithi066 ceph-mon[85781]: pgmap v1487: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.615+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f087c1112d0 con 0x7f086006d7f0 2024-02-28T17:23:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.632+0000 7f0872a00700 1 -- 172.21.15.1:0/2328625835 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f087c1112d0 con 0x7f086006d7f0 2024-02-28T17:23:55.635 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:23:55.635 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (50m) 11m ago 64m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:23:55.635 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (46m) 11m ago 64m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (46m) 11m ago 63m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (49m) 11m ago 63m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (42m) 11m ago 57m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (42m) 11m ago 57m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (11m) 11m ago 56m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (26m) 11m ago 57m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (52m) 11m ago 66m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (51m) 11m ago 63m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (48m) 11m ago 66m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (47m) 11m ago 62m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (51m) 11m ago 64m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (50m) 11m ago 63m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:23:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (46m) 11m ago 61m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:23:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (45m) 11m ago 61m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:23:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (45m) 11m ago 60m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:23:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (44m) 11m ago 60m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:23:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (43m) 11m ago 59m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:23:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (43m) 11m ago 59m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:23:55.637 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (50m) 11m ago 63m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:23:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f086006d7f0 msgr2=0x7f086006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f086006d7f0 0x7f086006fca0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0864006fd0 tx=0x7f0864008040 comp rx=0 tx=0).stop 2024-02-28T17:23:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 msgr2=0x7f087c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 0x7f087c115c80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f086c009150 tx=0x7f086c009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 shutdown_connections 2024-02-28T17:23:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f086006d7f0 0x7f086006fca0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f087c105170 0x7f087c115c80 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 --2- 172.21.15.1:0/2328625835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f087c105b10 0x7f087c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 >> 172.21.15.1:0/2328625835 conn(0x7f087c0face0 msgr2=0x7f087c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 shutdown_connections 2024-02-28T17:23:55.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.636+0000 7f0881600700 1 -- 172.21.15.1:0/2328625835 wait complete. 2024-02-28T17:23:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.794+0000 7fca8e800700 1 -- 172.21.15.1:0/3116743925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca88102f50 msgr2=0x7fca88103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.794+0000 7fca8e800700 1 --2- 172.21.15.1:0/3116743925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca88102f50 0x7fca88103320 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fca78004940 tx=0x7fca78009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.794+0000 7fca8e800700 1 -- 172.21.15.1:0/3116743925 shutdown_connections 2024-02-28T17:23:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.794+0000 7fca8e800700 1 --2- 172.21.15.1:0/3116743925 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca881038f0 0x7fca880fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.794+0000 7fca8e800700 1 --2- 172.21.15.1:0/3116743925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca88102f50 0x7fca88103320 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.794+0000 7fca8e800700 1 -- 172.21.15.1:0/3116743925 >> 172.21.15.1:0/3116743925 conn(0x7fca880f8ae0 msgr2=0x7fca880faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.794+0000 7fca8e800700 1 -- 172.21.15.1:0/3116743925 shutdown_connections 2024-02-28T17:23:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.794+0000 7fca8e800700 1 -- 172.21.15.1:0/3116743925 wait complete. 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.795+0000 7fca8e800700 1 Processor -- start 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.795+0000 7fca8e800700 1 -- start start 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca8e800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca88102f50 0x7fca88115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca8e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca881038f0 0x7fca88116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca8e800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca881167b0 con 0x7fca881038f0 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca8e800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8810fb50 con 0x7fca88102f50 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca86a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca881038f0 0x7fca88116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca86a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca881038f0 0x7fca88116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38076/0 (socket says 172.21.15.1:38076) 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca86a00700 1 -- 172.21.15.1:0/1297498673 learned_addr learned my addr 172.21.15.1:0/1297498673 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:55.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca87400700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca88102f50 0x7fca88115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca86a00700 1 -- 172.21.15.1:0/1297498673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca88102f50 msgr2=0x7fca88115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca86a00700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca88102f50 0x7fca88115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.796+0000 7fca86a00700 1 -- 172.21.15.1:0/1297498673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca78005040 con 0x7fca881038f0 2024-02-28T17:23:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.797+0000 7fca86a00700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca881038f0 0x7fca88116160 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fca74002a80 tx=0x7fca74004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:55.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.797+0000 7fca7f400700 1 -- 172.21.15.1:0/1297498673 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca74007cc0 con 0x7fca881038f0 2024-02-28T17:23:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.797+0000 7fca7f400700 1 -- 172.21.15.1:0/1297498673 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca7400f040 con 0x7fca881038f0 2024-02-28T17:23:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.797+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8810fe30 con 0x7fca881038f0 2024-02-28T17:23:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.797+0000 7fca7f400700 1 -- 172.21.15.1:0/1297498673 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca7400daf0 con 0x7fca881038f0 2024-02-28T17:23:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.797+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca88110370 con 0x7fca881038f0 2024-02-28T17:23:55.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.798+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca880fc6e0 con 0x7fca881038f0 2024-02-28T17:23:55.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.799+0000 7fca7f400700 1 -- 172.21.15.1:0/1297498673 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fca7400dc90 con 0x7fca881038f0 2024-02-28T17:23:55.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.800+0000 7fca7f400700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca6c06d7f0 0x7fca6c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:55.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.800+0000 7fca7f400700 1 -- 172.21.15.1:0/1297498673 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fca7408c660 con 0x7fca881038f0 2024-02-28T17:23:55.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.801+0000 7fca87400700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca6c06d7f0 0x7fca6c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:55.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.801+0000 7fca87400700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca6c06d7f0 0x7fca6c06fca0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fca7800b020 tx=0x7fca78009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:55.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:55.805+0000 7fca7f400700 1 -- 172.21.15.1:0/1297498673 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fca74059940 con 0x7fca881038f0 2024-02-28T17:23:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.112+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fca8804bed0 con 0x7fca881038f0 2024-02-28T17:23:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.113+0000 7fca7f400700 1 -- 172.21.15.1:0/1297498673 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fca740594d0 con 0x7fca881038f0 2024-02-28T17:23:56.116 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:23:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:23:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:23:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:23:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:23:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:23:56.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:23:56.117 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:23:56.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca6c06d7f0 msgr2=0x7fca6c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca6c06d7f0 0x7fca6c06fca0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fca7800b020 tx=0x7fca78009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca881038f0 msgr2=0x7fca88116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca881038f0 0x7fca88116160 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fca74002a80 tx=0x7fca74004630 comp rx=0 tx=0).stop 2024-02-28T17:23:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 shutdown_connections 2024-02-28T17:23:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fca6c06d7f0 0x7fca6c06fca0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fca88102f50 0x7fca88115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 --2- 172.21.15.1:0/1297498673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca881038f0 0x7fca88116160 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 >> 172.21.15.1:0/1297498673 conn(0x7fca880f8ae0 msgr2=0x7fca880fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.116+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 shutdown_connections 2024-02-28T17:23:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.117+0000 7fca8e800700 1 -- 172.21.15.1:0/1297498673 wait complete. 2024-02-28T17:23:56.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.275+0000 7fdf7d800700 1 -- 172.21.15.1:0/3111017493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78103710 msgr2=0x7fdf780fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.275+0000 7fdf7d800700 1 --2- 172.21.15.1:0/3111017493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78103710 0x7fdf780fcd80 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fdf64004940 tx=0x7fdf64009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:56.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.275+0000 7fdf7d800700 1 -- 172.21.15.1:0/3111017493 shutdown_connections 2024-02-28T17:23:56.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.275+0000 7fdf7d800700 1 --2- 172.21.15.1:0/3111017493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78103710 0x7fdf780fcd80 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.275+0000 7fdf7d800700 1 --2- 172.21.15.1:0/3111017493 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf78102d70 0x7fdf78103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.275+0000 7fdf7d800700 1 -- 172.21.15.1:0/3111017493 >> 172.21.15.1:0/3111017493 conn(0x7fdf780f88e0 msgr2=0x7fdf780facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:56.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.275+0000 7fdf7d800700 1 -- 172.21.15.1:0/3111017493 shutdown_connections 2024-02-28T17:23:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.275+0000 7fdf7d800700 1 -- 172.21.15.1:0/3111017493 wait complete. 2024-02-28T17:23:56.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.276+0000 7fdf7d800700 1 Processor -- start 2024-02-28T17:23:56.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.276+0000 7fdf7d800700 1 -- start start 2024-02-28T17:23:56.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.276+0000 7fdf7d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78102d70 0x7fdf78115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.276+0000 7fdf7d800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf78103710 0x7fdf78115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.276+0000 7fdf7d800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78116670 con 0x7fdf78102d70 2024-02-28T17:23:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.276+0000 7fdf7d800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7810f910 con 0x7fdf78103710 2024-02-28T17:23:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.277+0000 7fdf77e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78102d70 0x7fdf78115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.277+0000 7fdf77e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78102d70 0x7fdf78115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38096/0 (socket says 172.21.15.1:38096) 2024-02-28T17:23:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.277+0000 7fdf77e00700 1 -- 172.21.15.1:0/4001647385 learned_addr learned my addr 172.21.15.1:0/4001647385 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.277+0000 7fdf77400700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf78103710 0x7fdf78115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.277+0000 7fdf77e00700 1 -- 172.21.15.1:0/4001647385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf78103710 msgr2=0x7fdf78115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.277+0000 7fdf77e00700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf78103710 0x7fdf78115f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.277+0000 7fdf77e00700 1 -- 172.21.15.1:0/4001647385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf64005040 con 0x7fdf78102d70 2024-02-28T17:23:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.277+0000 7fdf77400700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf78103710 0x7fdf78115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:23:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.278+0000 7fdf77e00700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78102d70 0x7fdf78115a50 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800b790 tx=0x7fdf68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.278+0000 7fdf6fe00700 1 -- 172.21.15.1:0/4001647385 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf6800f770 con 0x7fdf78102d70 2024-02-28T17:23:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.278+0000 7fdf6fe00700 1 -- 172.21.15.1:0/4001647385 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf6800fda0 con 0x7fdf78102d70 2024-02-28T17:23:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.278+0000 7fdf6fe00700 1 -- 172.21.15.1:0/4001647385 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf6800d310 con 0x7fdf78102d70 2024-02-28T17:23:56.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.278+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7810fbf0 con 0x7fdf78102d70 2024-02-28T17:23:56.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.278+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf781101b0 con 0x7fdf78102d70 2024-02-28T17:23:56.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.279+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7805f5c0 con 0x7fdf78102d70 2024-02-28T17:23:56.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.281+0000 7fdf6fe00700 1 -- 172.21.15.1:0/4001647385 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdf68010420 con 0x7fdf78102d70 2024-02-28T17:23:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.281+0000 7fdf6fe00700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf5c06d7f0 0x7fdf5c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.282+0000 7fdf6fe00700 1 -- 172.21.15.1:0/4001647385 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdf6808cef0 con 0x7fdf78102d70 2024-02-28T17:23:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.282+0000 7fdf77400700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf5c06d7f0 0x7fdf5c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.283+0000 7fdf77400700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf5c06d7f0 0x7fdf5c06fca0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400b3b0 tx=0x7fdf64009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.287+0000 7fdf6fe00700 1 -- 172.21.15.1:0/4001647385 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdf6805a2d0 con 0x7fdf78102d70 2024-02-28T17:23:56.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:56 smithi066 ceph-mon[85781]: from='client.36608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:56.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:56 smithi066 ceph-mon[85781]: from='client.36612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:56.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:56 smithi066 ceph-mon[85781]: from='client.36616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:56.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:56 smithi066 ceph-mon[85781]: pgmap v1488: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:56.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:23:56.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:56 smithi001 ceph-mon[112940]: from='client.36608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:56.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:56 smithi001 ceph-mon[112940]: from='client.36612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:56.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:56 smithi001 ceph-mon[112940]: from='client.36616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:56.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:56 smithi001 ceph-mon[112940]: pgmap v1488: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:56.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:23:56.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.559+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdf78109f40 con 0x7fdf78102d70 2024-02-28T17:23:56.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.559+0000 7fdf6fe00700 1 -- 172.21.15.1:0/4001647385 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fdf78109f40 con 0x7fdf78102d70 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:23:56.563 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:23:56.564 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:23:56.565 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:23:56.566 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:23:56.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.562+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf5c06d7f0 msgr2=0x7fdf5c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.562+0000 7fdf7d800700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf5c06d7f0 0x7fdf5c06fca0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400b3b0 tx=0x7fdf64009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:56.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.562+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78102d70 msgr2=0x7fdf78115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.562+0000 7fdf7d800700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78102d70 0x7fdf78115a50 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800b790 tx=0x7fdf68009250 comp rx=0 tx=0).stop 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.563+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 shutdown_connections 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.563+0000 7fdf7d800700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdf5c06d7f0 0x7fdf5c06fca0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.563+0000 7fdf7d800700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf78102d70 0x7fdf78115a50 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.563+0000 7fdf7d800700 1 --2- 172.21.15.1:0/4001647385 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdf78103710 0x7fdf78115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.563+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 >> 172.21.15.1:0/4001647385 conn(0x7fdf780f88e0 msgr2=0x7fdf78070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.563+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 shutdown_connections 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.563+0000 7fdf7d800700 1 -- 172.21.15.1:0/4001647385 wait complete. 2024-02-28T17:23:56.568 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:23:56.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.720+0000 7fb05d400700 1 -- 172.21.15.1:0/1270191940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fdfe0 msgr2=0x7fb0580fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.720+0000 7fb05d400700 1 --2- 172.21.15.1:0/1270191940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fdfe0 0x7fb0580fe3b0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fb048004970 tx=0x7fb048009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:56.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.720+0000 7fb05d400700 1 -- 172.21.15.1:0/1270191940 shutdown_connections 2024-02-28T17:23:56.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.720+0000 7fb05d400700 1 --2- 172.21.15.1:0/1270191940 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0580fe8f0 0x7fb058107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.720+0000 7fb05d400700 1 --2- 172.21.15.1:0/1270191940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fdfe0 0x7fb0580fe3b0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.720+0000 7fb05d400700 1 -- 172.21.15.1:0/1270191940 >> 172.21.15.1:0/1270191940 conn(0x7fb058071d90 msgr2=0x7fb058072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:56.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.721+0000 7fb05d400700 1 -- 172.21.15.1:0/1270191940 shutdown_connections 2024-02-28T17:23:56.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.721+0000 7fb05d400700 1 -- 172.21.15.1:0/1270191940 wait complete. 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.721+0000 7fb05d400700 1 Processor -- start 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.722+0000 7fb05d400700 1 -- start start 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.722+0000 7fb05d400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0580fdfe0 0x7fb058115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.722+0000 7fb05d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fe8f0 0x7fb058116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.722+0000 7fb05d400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb058116790 con 0x7fb0580fe8f0 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.722+0000 7fb05d400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05810fae0 con 0x7fb0580fdfe0 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.722+0000 7fb055e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fe8f0 0x7fb058116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.722+0000 7fb055e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fe8f0 0x7fb058116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38110/0 (socket says 172.21.15.1:38110) 2024-02-28T17:23:56.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.722+0000 7fb055e00700 1 -- 172.21.15.1:0/1158787293 learned_addr learned my addr 172.21.15.1:0/1158787293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:56.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb056800700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0580fdfe0 0x7fb058115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:56.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb055e00700 1 -- 172.21.15.1:0/1158787293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0580fdfe0 msgr2=0x7fb058115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb055e00700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0580fdfe0 0x7fb058115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb055e00700 1 -- 172.21.15.1:0/1158787293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb048005040 con 0x7fb0580fe8f0 2024-02-28T17:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb055e00700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fe8f0 0x7fb058116140 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fb044002a50 tx=0x7fb0440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb04ea00700 1 -- 172.21.15.1:0/1158787293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb044007cc0 con 0x7fb0580fe8f0 2024-02-28T17:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb04ea00700 1 -- 172.21.15.1:0/1158787293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb04400f040 con 0x7fb0580fe8f0 2024-02-28T17:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05810fdc0 con 0x7fb0580fe8f0 2024-02-28T17:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.723+0000 7fb04ea00700 1 -- 172.21.15.1:0/1158787293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb044009640 con 0x7fb0580fe8f0 2024-02-28T17:23:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.724+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb058110300 con 0x7fb0580fe8f0 2024-02-28T17:23:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.725+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb058106b20 con 0x7fb0580fe8f0 2024-02-28T17:23:56.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.726+0000 7fb04ea00700 1 -- 172.21.15.1:0/1158787293 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb0440097a0 con 0x7fb0580fe8f0 2024-02-28T17:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.727+0000 7fb04ea00700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb03c06d7e0 0x7fb03c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.727+0000 7fb04ea00700 1 -- 172.21.15.1:0/1158787293 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb04408cc60 con 0x7fb0580fe8f0 2024-02-28T17:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.727+0000 7fb056800700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb03c06d7e0 0x7fb03c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:56.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.728+0000 7fb056800700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb03c06d7e0 0x7fb03c06fc90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb048000c00 tx=0x7fb048009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.732+0000 7fb04ea00700 1 -- 172.21.15.1:0/1158787293 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb044056490 con 0x7fb0580fe8f0 2024-02-28T17:23:56.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.969+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb05804bed0 con 0x7fb03c06d7e0 2024-02-28T17:23:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.972+0000 7fb04ea00700 1 -- 172.21.15.1:0/1158787293 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb05804bed0 con 0x7fb03c06d7e0 2024-02-28T17:23:56.975 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:23:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:23:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:23:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:23:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:23:56.975 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:23:56.976 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:23:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb03c06d7e0 msgr2=0x7fb03c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb03c06d7e0 0x7fb03c06fc90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb048000c00 tx=0x7fb048009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fe8f0 msgr2=0x7fb058116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fe8f0 0x7fb058116140 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fb044002a50 tx=0x7fb0440046b0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 shutdown_connections 2024-02-28T17:23:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb03c06d7e0 0x7fb03c06fc90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb0580fdfe0 0x7fb058115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 --2- 172.21.15.1:0/1158787293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0580fe8f0 0x7fb058116140 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 >> 172.21.15.1:0/1158787293 conn(0x7fb058071d90 msgr2=0x7fb0580fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:56.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 shutdown_connections 2024-02-28T17:23:56.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:56.976+0000 7fb05d400700 1 -- 172.21.15.1:0/1158787293 wait complete. 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.127+0000 7f0263200700 1 -- 172.21.15.1:0/1957324517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 msgr2=0x7f0264106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.127+0000 7f0263200700 1 --2- 172.21.15.1:0/1957324517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 0x7f0264106e00 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f0250004940 tx=0x7f0250009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.127+0000 7f0263200700 1 -- 172.21.15.1:0/1957324517 shutdown_connections 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.127+0000 7f0263200700 1 --2- 172.21.15.1:0/1957324517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 0x7f0264106e00 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.127+0000 7f0263200700 1 --2- 172.21.15.1:0/1957324517 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02640fdd90 0x7f02640fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.127+0000 7f0263200700 1 -- 172.21.15.1:0/1957324517 >> 172.21.15.1:0/1957324517 conn(0x7f0264073ab0 msgr2=0x7f0264073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.127+0000 7f0263200700 1 -- 172.21.15.1:0/1957324517 shutdown_connections 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.127+0000 7f0263200700 1 -- 172.21.15.1:0/1957324517 wait complete. 2024-02-28T17:23:57.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.128+0000 7f0263200700 1 Processor -- start 2024-02-28T17:23:57.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.128+0000 7f0263200700 1 -- start start 2024-02-28T17:23:57.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0263200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02640fdd90 0x7f0264115ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:57.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0263200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 0x7f0264116010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0263200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264116660 con 0x7f02640fe730 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0263200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026410f990 con 0x7f02640fdd90 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0261400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 0x7f0264116010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0261400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 0x7f0264116010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38128/0 (socket says 172.21.15.1:38128) 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0261400700 1 -- 172.21.15.1:0/3824150491 learned_addr learned my addr 172.21.15.1:0/3824150491 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0261e00700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02640fdd90 0x7f0264115ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0261400700 1 -- 172.21.15.1:0/3824150491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02640fdd90 msgr2=0x7f0264115ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0261400700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02640fdd90 0x7f0264115ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.129+0000 7f0261400700 1 -- 172.21.15.1:0/3824150491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0250005040 con 0x7f02640fe730 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.130+0000 7f0261e00700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02640fdd90 0x7f0264115ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.130+0000 7f0261400700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 0x7f0264116010 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0250000c00 tx=0x7f0250009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.130+0000 7f025a000700 1 -- 172.21.15.1:0/3824150491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f025000e040 con 0x7f02640fe730 2024-02-28T17:23:57.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.130+0000 7f025a000700 1 -- 172.21.15.1:0/3824150491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f025000bb90 con 0x7f02640fe730 2024-02-28T17:23:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.130+0000 7f025a000700 1 -- 172.21.15.1:0/3824150491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f025001c630 con 0x7f02640fe730 2024-02-28T17:23:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.130+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026410fc10 con 0x7f02640fe730 2024-02-28T17:23:57.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.130+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0264110150 con 0x7f02640fe730 2024-02-28T17:23:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.131+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0264106920 con 0x7f02640fe730 2024-02-28T17:23:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.132+0000 7f025a000700 1 -- 172.21.15.1:0/3824150491 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f025000bd00 con 0x7f02640fe730 2024-02-28T17:23:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.133+0000 7f025a000700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f024806d7f0 0x7f024806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:23:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.133+0000 7f025a000700 1 -- 172.21.15.1:0/3824150491 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f025008e230 con 0x7f02640fe730 2024-02-28T17:23:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.134+0000 7f0261e00700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f024806d7f0 0x7f024806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:23:57.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.134+0000 7f0261e00700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f024806d7f0 0x7f024806fca0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f0254009fd0 tx=0x7f025400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:23:57.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.138+0000 7f025a000700 1 -- 172.21.15.1:0/3824150491 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f025005b590 con 0x7f02640fe730 2024-02-28T17:23:57.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:57 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1297498673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:23:57.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:57 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4001647385' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:23:57.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:57 smithi001 ceph-mon[112940]: from='client.36628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:57.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.433+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0264109e10 con 0x7f02640fe730 2024-02-28T17:23:57.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.434+0000 7f025a000700 1 -- 172.21.15.1:0/3824150491 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f025005b120 con 0x7f02640fe730 2024-02-28T17:23:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:23:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:23:57.437 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:23:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:23:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:23:57.437 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:23:57.437 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:23:57.438 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:23:57.439 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:23:57.439 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:23:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f024806d7f0 msgr2=0x7f024806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f024806d7f0 0x7f024806fca0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f0254009fd0 tx=0x7f025400b040 comp rx=0 tx=0).stop 2024-02-28T17:23:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 msgr2=0x7f0264116010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:23:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 0x7f0264116010 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0250000c00 tx=0x7f0250009e30 comp rx=0 tx=0).stop 2024-02-28T17:23:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 shutdown_connections 2024-02-28T17:23:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f024806d7f0 0x7f024806fca0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02640fdd90 0x7f0264115ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 --2- 172.21.15.1:0/3824150491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02640fe730 0x7f0264116010 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:23:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 >> 172.21.15.1:0/3824150491 conn(0x7f0264073ab0 msgr2=0x7f02640fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:23:57.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 shutdown_connections 2024-02-28T17:23:57.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:23:57.437+0000 7f0263200700 1 -- 172.21.15.1:0/3824150491 wait complete. 2024-02-28T17:23:57.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:57 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1297498673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:23:57.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:57 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4001647385' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:23:57.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:57 smithi066 ceph-mon[85781]: from='client.36628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:23:58.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:58 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3824150491' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:23:58.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:23:58 smithi066 ceph-mon[85781]: pgmap v1489: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:23:58.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:58 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3824150491' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:23:58.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:23:58 smithi001 ceph-mon[112940]: pgmap v1489: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:01.012 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:00 smithi066 ceph-mon[85781]: pgmap v1490: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:01.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:00 smithi001 ceph-mon[112940]: pgmap v1490: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:02 smithi001 ceph-mon[112940]: pgmap v1491: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:02 smithi066 ceph-mon[85781]: pgmap v1491: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:05.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:04 smithi001 ceph-mon[112940]: pgmap v1492: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:05.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:04 smithi066 ceph-mon[85781]: pgmap v1492: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:07.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:06 smithi001 ceph-mon[112940]: pgmap v1493: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:07.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:06 smithi066 ceph-mon[85781]: pgmap v1493: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:09.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:08 smithi001 ceph-mon[112940]: pgmap v1494: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:08 smithi066 ceph-mon[85781]: pgmap v1494: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:11.065 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:10 smithi066 ceph-mon[85781]: pgmap v1495: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:10 smithi001 ceph-mon[112940]: pgmap v1495: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:12.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:24:12.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:24:13.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:12 smithi066 ceph-mon[85781]: pgmap v1496: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:13.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:12 smithi001 ceph-mon[112940]: pgmap v1496: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:15.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:14 smithi066 ceph-mon[85781]: pgmap v1497: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:15.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:14 smithi001 ceph-mon[112940]: pgmap v1497: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:17.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:16 smithi066 ceph-mon[85781]: pgmap v1498: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:17.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:16 smithi001 ceph-mon[112940]: pgmap v1498: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:19.086 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:18 smithi001 ceph-mon[112940]: pgmap v1499: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:19.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:18 smithi066 ceph-mon[85781]: pgmap v1499: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:21.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:20 smithi066 ceph-mon[85781]: pgmap v1500: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:21.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:20 smithi001 ceph-mon[112940]: pgmap v1500: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:23.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:22 smithi066 ceph-mon[85781]: pgmap v1501: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:23.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:22 smithi001 ceph-mon[112940]: pgmap v1501: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:25.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:24 smithi066 ceph-mon[85781]: pgmap v1502: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:25.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:24 smithi001 ceph-mon[112940]: pgmap v1502: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:26.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:24:26.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:24:27.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:26 smithi066 ceph-mon[85781]: pgmap v1503: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:27.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:26 smithi001 ceph-mon[112940]: pgmap v1503: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:27.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.607+0000 7f7eef400700 1 -- 172.21.15.1:0/3521730541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee8103dc0 msgr2=0x7f7ee8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:27.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.607+0000 7f7eef400700 1 --2- 172.21.15.1:0/3521730541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee8103dc0 0x7f7ee8104210 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4004940 tx=0x7f7ed4009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:27.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.607+0000 7f7eef400700 1 -- 172.21.15.1:0/3521730541 shutdown_connections 2024-02-28T17:24:27.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.607+0000 7f7eef400700 1 --2- 172.21.15.1:0/3521730541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee8103dc0 0x7f7ee8104210 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:27.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.607+0000 7f7eef400700 1 --2- 172.21.15.1:0/3521730541 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ee81095f0 0x7f7ee81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:27.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.607+0000 7f7eef400700 1 -- 172.21.15.1:0/3521730541 >> 172.21.15.1:0/3521730541 conn(0x7f7ee8072d50 msgr2=0x7f7ee8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:27.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.607+0000 7f7eef400700 1 -- 172.21.15.1:0/3521730541 shutdown_connections 2024-02-28T17:24:27.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.607+0000 7f7eef400700 1 -- 172.21.15.1:0/3521730541 wait complete. 2024-02-28T17:24:27.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.608+0000 7f7eef400700 1 Processor -- start 2024-02-28T17:24:27.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.608+0000 7f7eef400700 1 -- start start 2024-02-28T17:24:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.609+0000 7f7eef400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ee8103dc0 0x7f7ee811a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.609+0000 7f7eef400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee81095f0 0x7f7ee811a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.609+0000 7f7eef400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee811abe0 con 0x7f7ee81095f0 2024-02-28T17:24:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.609+0000 7f7eef400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee8113ef0 con 0x7f7ee8103dc0 2024-02-28T17:24:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.609+0000 7f7ee7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee81095f0 0x7f7ee811a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.609+0000 7f7ee7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee81095f0 0x7f7ee811a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40396/0 (socket says 172.21.15.1:40396) 2024-02-28T17:24:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.609+0000 7f7ee7e00700 1 -- 172.21.15.1:0/2349287824 learned_addr learned my addr 172.21.15.1:0/2349287824 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:24:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.609+0000 7f7eeca00700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ee8103dc0 0x7f7ee811a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7ee7e00700 1 -- 172.21.15.1:0/2349287824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ee8103dc0 msgr2=0x7f7ee811a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7ee7e00700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ee8103dc0 0x7f7ee811a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7ee7e00700 1 -- 172.21.15.1:0/2349287824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed4005040 con 0x7f7ee81095f0 2024-02-28T17:24:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7eeca00700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ee8103dc0 0x7f7ee811a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:24:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7ee7e00700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee81095f0 0x7f7ee811a590 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009fd0 tx=0x7f7ed4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7ee4c00700 1 -- 172.21.15.1:0/2349287824 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed400e040 con 0x7f7ee81095f0 2024-02-28T17:24:27.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7ee4c00700 1 -- 172.21.15.1:0/2349287824 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ed400bbf0 con 0x7f7ee81095f0 2024-02-28T17:24:27.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7ee4c00700 1 -- 172.21.15.1:0/2349287824 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed401c3f0 con 0x7f7ee81095f0 2024-02-28T17:24:27.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee8114170 con 0x7f7ee81095f0 2024-02-28T17:24:27.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.610+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee81146b0 con 0x7f7ee81095f0 2024-02-28T17:24:27.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.612+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee810b9c0 con 0x7f7ee81095f0 2024-02-28T17:24:27.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.612+0000 7f7ee4c00700 1 -- 172.21.15.1:0/2349287824 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7ed400bd60 con 0x7f7ee81095f0 2024-02-28T17:24:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.613+0000 7f7ee4c00700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7ed006d7f0 0x7f7ed006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.613+0000 7f7ee4c00700 1 -- 172.21.15.1:0/2349287824 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7ed408e1c0 con 0x7f7ee81095f0 2024-02-28T17:24:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.614+0000 7f7eeca00700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7ed006d7f0 0x7f7ed006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.614+0000 7f7eeca00700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7ed006d7f0 0x7f7ed006fca0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009fd0 tx=0x7f7edc00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:27.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.618+0000 7f7ee4c00700 1 -- 172.21.15.1:0/2349287824 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7ed405b5a0 con 0x7f7ee81095f0 2024-02-28T17:24:27.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.864+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ee804bf40 con 0x7f7ed006d7f0 2024-02-28T17:24:27.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.866+0000 7f7ee4c00700 1 -- 172.21.15.1:0/2349287824 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ee804bf40 con 0x7f7ed006d7f0 2024-02-28T17:24:27.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.868+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7ed006d7f0 msgr2=0x7f7ed006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:27.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.868+0000 7f7eef400700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7ed006d7f0 0x7f7ed006fca0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009fd0 tx=0x7f7edc00b040 comp rx=0 tx=0).stop 2024-02-28T17:24:27.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.868+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee81095f0 msgr2=0x7f7ee811a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:27.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.868+0000 7f7eef400700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee81095f0 0x7f7ee811a590 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009fd0 tx=0x7f7ed4009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.869+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 shutdown_connections 2024-02-28T17:24:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.869+0000 7f7eef400700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f7ed006d7f0 0x7f7ed006fca0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.869+0000 7f7eef400700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7ee8103dc0 0x7f7ee811a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.869+0000 7f7eef400700 1 --2- 172.21.15.1:0/2349287824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ee81095f0 0x7f7ee811a590 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.869+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 >> 172.21.15.1:0/2349287824 conn(0x7f7ee8072d50 msgr2=0x7f7ee8074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.869+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 shutdown_connections 2024-02-28T17:24:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:27.869+0000 7f7eef400700 1 -- 172.21.15.1:0/2349287824 wait complete. 2024-02-28T17:24:27.890 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:24:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.030+0000 7facf2800700 1 -- 172.21.15.1:0/2642056964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 msgr2=0x7facec103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.030+0000 7facf2800700 1 --2- 172.21.15.1:0/2642056964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 0x7facec103320 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7facdc004940 tx=0x7facdc009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.030+0000 7facf2800700 1 -- 172.21.15.1:0/2642056964 shutdown_connections 2024-02-28T17:24:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.030+0000 7facf2800700 1 --2- 172.21.15.1:0/2642056964 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7facec1038f0 0x7facec0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.030+0000 7facf2800700 1 --2- 172.21.15.1:0/2642056964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 0x7facec103320 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.030+0000 7facf2800700 1 -- 172.21.15.1:0/2642056964 >> 172.21.15.1:0/2642056964 conn(0x7facec0f8ae0 msgr2=0x7facec0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.030+0000 7facf2800700 1 -- 172.21.15.1:0/2642056964 shutdown_connections 2024-02-28T17:24:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.030+0000 7facf2800700 1 -- 172.21.15.1:0/2642056964 wait complete. 2024-02-28T17:24:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.031+0000 7facf2800700 1 Processor -- start 2024-02-28T17:24:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.031+0000 7facf2800700 1 -- start start 2024-02-28T17:24:28.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.032+0000 7facf2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 0x7facec115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.032+0000 7facf2800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7facec1038f0 0x7facec1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.032+0000 7facf2800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec116800 con 0x7facec102f50 2024-02-28T17:24:28.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.032+0000 7facf2800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec10fb30 con 0x7facec1038f0 2024-02-28T17:24:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.032+0000 7faceb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 0x7facec115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.032+0000 7faceb400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 0x7facec115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40416/0 (socket says 172.21.15.1:40416) 2024-02-28T17:24:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.032+0000 7faceb400700 1 -- 172.21.15.1:0/142791278 learned_addr learned my addr 172.21.15.1:0/142791278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:24:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.032+0000 7faceaa00700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7facec1038f0 0x7facec1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.033+0000 7faceb400700 1 -- 172.21.15.1:0/142791278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7facec1038f0 msgr2=0x7facec1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.033+0000 7faceb400700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7facec1038f0 0x7facec1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.033+0000 7faceb400700 1 -- 172.21.15.1:0/142791278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc005040 con 0x7facec102f50 2024-02-28T17:24:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.033+0000 7faceb400700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 0x7facec115c70 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7facdc009fd0 tx=0x7facdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.033+0000 7face3400700 1 -- 172.21.15.1:0/142791278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facdc00e040 con 0x7facec102f50 2024-02-28T17:24:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.034+0000 7face3400700 1 -- 172.21.15.1:0/142791278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facdc00bbf0 con 0x7facec102f50 2024-02-28T17:24:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.034+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec10fdb0 con 0x7facec102f50 2024-02-28T17:24:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.034+0000 7face3400700 1 -- 172.21.15.1:0/142791278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facdc01c630 con 0x7facec102f50 2024-02-28T17:24:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.034+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec1102f0 con 0x7facec102f50 2024-02-28T17:24:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.035+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec0fc6e0 con 0x7facec102f50 2024-02-28T17:24:28.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.036+0000 7face3400700 1 -- 172.21.15.1:0/142791278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7facdc00bd60 con 0x7facec102f50 2024-02-28T17:24:28.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.037+0000 7face3400700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7f0 0x7facd006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.037+0000 7face3400700 1 -- 172.21.15.1:0/142791278 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7facdc08e1d0 con 0x7facec102f50 2024-02-28T17:24:28.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.037+0000 7faceaa00700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7f0 0x7facd006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.038+0000 7faceaa00700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7f0 0x7facd006fca0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7facec1114a0 tx=0x7facd800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:28.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.042+0000 7face3400700 1 -- 172.21.15.1:0/142791278 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7facdc05b530 con 0x7facec102f50 2024-02-28T17:24:28.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.284+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7facec111250 con 0x7facd006d7f0 2024-02-28T17:24:28.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.288+0000 7face3400700 1 -- 172.21.15.1:0/142791278 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7facec111250 con 0x7facd006d7f0 2024-02-28T17:24:28.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.291+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7f0 msgr2=0x7facd006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.291+0000 7facf2800700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7f0 0x7facd006fca0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7facec1114a0 tx=0x7facd800b040 comp rx=0 tx=0).stop 2024-02-28T17:24:28.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.291+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 msgr2=0x7facec115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.291+0000 7facf2800700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 0x7facec115c70 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7facdc009fd0 tx=0x7facdc009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:28.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.292+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 shutdown_connections 2024-02-28T17:24:28.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.292+0000 7facf2800700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7f0 0x7facd006fca0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.292+0000 7facf2800700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facec102f50 0x7facec115c70 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.292+0000 7facf2800700 1 --2- 172.21.15.1:0/142791278 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7facec1038f0 0x7facec1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.292+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 >> 172.21.15.1:0/142791278 conn(0x7facec0f8ae0 msgr2=0x7facec070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:28.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.292+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 shutdown_connections 2024-02-28T17:24:28.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.292+0000 7facf2800700 1 -- 172.21.15.1:0/142791278 wait complete. 2024-02-28T17:24:28.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.441+0000 7fbb19600700 1 -- 172.21.15.1:0/2266141308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105160 msgr2=0x7fbb14105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.441+0000 7fbb19600700 1 --2- 172.21.15.1:0/2266141308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105160 0x7fbb14105530 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fbb04004940 tx=0x7fbb04009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:28.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.441+0000 7fbb19600700 1 -- 172.21.15.1:0/2266141308 shutdown_connections 2024-02-28T17:24:28.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.441+0000 7fbb19600700 1 --2- 172.21.15.1:0/2266141308 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb14105b00 0x7fbb140ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.441+0000 7fbb19600700 1 --2- 172.21.15.1:0/2266141308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105160 0x7fbb14105530 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.441+0000 7fbb19600700 1 -- 172.21.15.1:0/2266141308 >> 172.21.15.1:0/2266141308 conn(0x7fbb140facb0 msgr2=0x7fbb140fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.442+0000 7fbb19600700 1 -- 172.21.15.1:0/2266141308 shutdown_connections 2024-02-28T17:24:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.442+0000 7fbb19600700 1 -- 172.21.15.1:0/2266141308 wait complete. 2024-02-28T17:24:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.443+0000 7fbb19600700 1 Processor -- start 2024-02-28T17:24:28.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.443+0000 7fbb19600700 1 -- start start 2024-02-28T17:24:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.443+0000 7fbb19600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb14105160 0x7fbb14115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb19600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105b00 0x7fbb141161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb19600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb14116800 con 0x7fbb14105b00 2024-02-28T17:24:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb19600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1410fb30 con 0x7fbb14105160 2024-02-28T17:24:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb12000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105b00 0x7fbb141161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb12000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105b00 0x7fbb141161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40448/0 (socket says 172.21.15.1:40448) 2024-02-28T17:24:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb12000700 1 -- 172.21.15.1:0/1320188785 learned_addr learned my addr 172.21.15.1:0/1320188785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb12a00700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb14105160 0x7fbb14115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb12000700 1 -- 172.21.15.1:0/1320188785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb14105160 msgr2=0x7fbb14115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb12000700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb14105160 0x7fbb14115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.444+0000 7fbb12000700 1 -- 172.21.15.1:0/1320188785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb04005040 con 0x7fbb14105b00 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.445+0000 7fbb12000700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105b00 0x7fbb141161b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000c890 tx=0x7fbb000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.445+0000 7fbb0aa00700 1 -- 172.21.15.1:0/1320188785 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb0000a7d0 con 0x7fbb14105b00 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.445+0000 7fbb0aa00700 1 -- 172.21.15.1:0/1320188785 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb0000ae00 con 0x7fbb14105b00 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.445+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1410fe10 con 0x7fbb14105b00 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.445+0000 7fbb0aa00700 1 -- 172.21.15.1:0/1320188785 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb000113a0 con 0x7fbb14105b00 2024-02-28T17:24:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.445+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb14110350 con 0x7fbb14105b00 2024-02-28T17:24:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.447+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb140fe8f0 con 0x7fbb14105b00 2024-02-28T17:24:28.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.448+0000 7fbb0aa00700 1 -- 172.21.15.1:0/1320188785 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbb00012400 con 0x7fbb14105b00 2024-02-28T17:24:28.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.448+0000 7fbb0aa00700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbaf806d7f0 0x7fbaf806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.449+0000 7fbb0aa00700 1 -- 172.21.15.1:0/1320188785 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbb0008cf70 con 0x7fbb14105b00 2024-02-28T17:24:28.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.449+0000 7fbb12a00700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbaf806d7f0 0x7fbaf806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.449+0000 7fbb12a00700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbaf806d7f0 0x7fbaf806fca0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400b3b0 tx=0x7fbb04009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:28.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.453+0000 7fbb0aa00700 1 -- 172.21.15.1:0/1320188785 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbb0005a350 con 0x7fbb14105b00 2024-02-28T17:24:28.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.678+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb14111250 con 0x7fbaf806d7f0 2024-02-28T17:24:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.691+0000 7fbb0aa00700 1 -- 172.21.15.1:0/1320188785 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbb14111250 con 0x7fbaf806d7f0 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (50m) 12m ago 64m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (47m) 12m ago 65m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (47m) 12m ago 63m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (50m) 12m ago 64m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (43m) 12m ago 57m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (42m) 12m ago 57m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (12m) 12m ago 57m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (27m) 12m ago 57m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:24:28.695 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (53m) 12m ago 66m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (52m) 12m ago 63m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (48m) 12m ago 66m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (48m) 12m ago 63m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (51m) 12m ago 65m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (51m) 12m ago 63m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (46m) 12m ago 62m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (46m) 12m ago 61m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (45m) 12m ago 61m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (45m) 12m ago 60m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (44m) 12m ago 60m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (43m) 12m ago 59m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:24:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (51m) 12m ago 64m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:24:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbaf806d7f0 msgr2=0x7fbaf806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbaf806d7f0 0x7fbaf806fca0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400b3b0 tx=0x7fbb04009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105b00 msgr2=0x7fbb141161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105b00 0x7fbb141161b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000c890 tx=0x7fbb000044c0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 shutdown_connections 2024-02-28T17:24:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbaf806d7f0 0x7fbaf806fca0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbb14105160 0x7fbb14115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 --2- 172.21.15.1:0/1320188785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb14105b00 0x7fbb141161b0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.693+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 >> 172.21.15.1:0/1320188785 conn(0x7fbb140facb0 msgr2=0x7fbb140fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.694+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 shutdown_connections 2024-02-28T17:24:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.694+0000 7fbb19600700 1 -- 172.21.15.1:0/1320188785 wait complete. 2024-02-28T17:24:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.852+0000 7f5c8b000700 1 -- 172.21.15.1:0/79554831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 msgr2=0x7f5c84109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.852+0000 7f5c8b000700 1 --2- 172.21.15.1:0/79554831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 0x7f5c84109260 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f5c70004970 tx=0x7f5c70009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.852+0000 7f5c8b000700 1 -- 172.21.15.1:0/79554831 shutdown_connections 2024-02-28T17:24:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.852+0000 7f5c8b000700 1 --2- 172.21.15.1:0/79554831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 0x7f5c84109260 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.852+0000 7f5c8b000700 1 --2- 172.21.15.1:0/79554831 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c84070330 0x7f5c84070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.852+0000 7f5c8b000700 1 -- 172.21.15.1:0/79554831 >> 172.21.15.1:0/79554831 conn(0x7f5c840f8ae0 msgr2=0x7f5c840faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.852+0000 7f5c8b000700 1 -- 172.21.15.1:0/79554831 shutdown_connections 2024-02-28T17:24:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.852+0000 7f5c8b000700 1 -- 172.21.15.1:0/79554831 wait complete. 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.853+0000 7f5c8b000700 1 Processor -- start 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.853+0000 7f5c8b000700 1 -- start start 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c8b000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c84070330 0x7f5c84115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c8b000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 0x7f5c84116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c8b000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c84116790 con 0x7f5c84070c40 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c8b000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8410fae0 con 0x7f5c84070330 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c83400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 0x7f5c84116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c83400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 0x7f5c84116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40456/0 (socket says 172.21.15.1:40456) 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c83400700 1 -- 172.21.15.1:0/2526073183 learned_addr learned my addr 172.21.15.1:0/2526073183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c83e00700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c84070330 0x7f5c84115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c83400700 1 -- 172.21.15.1:0/2526073183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c84070330 msgr2=0x7f5c84115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c83400700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c84070330 0x7f5c84115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.854+0000 7f5c83400700 1 -- 172.21.15.1:0/2526073183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c70005040 con 0x7f5c84070c40 2024-02-28T17:24:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.855+0000 7f5c83400700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 0x7f5c84116140 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f5c70004f50 tx=0x7f5c70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.855+0000 7f5c7be00700 1 -- 172.21.15.1:0/2526073183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c7000e040 con 0x7f5c84070c40 2024-02-28T17:24:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.855+0000 7f5c7be00700 1 -- 172.21.15.1:0/2526073183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c70004290 con 0x7f5c84070c40 2024-02-28T17:24:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.855+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8410fd60 con 0x7f5c84070c40 2024-02-28T17:24:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.855+0000 7f5c7be00700 1 -- 172.21.15.1:0/2526073183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c7001c630 con 0x7f5c84070c40 2024-02-28T17:24:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.855+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c841102a0 con 0x7f5c84070c40 2024-02-28T17:24:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.856+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c84106920 con 0x7f5c84070c40 2024-02-28T17:24:28.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.857+0000 7f5c7be00700 1 -- 172.21.15.1:0/2526073183 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5c7000bc10 con 0x7f5c84070c40 2024-02-28T17:24:28.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.858+0000 7f5c7be00700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c6806d7f0 0x7f5c6806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:28.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.858+0000 7f5c7be00700 1 -- 172.21.15.1:0/2526073183 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5c7008e310 con 0x7f5c84070c40 2024-02-28T17:24:28.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.859+0000 7f5c83e00700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c6806d7f0 0x7f5c6806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:28.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.859+0000 7f5c83e00700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c6806d7f0 0x7f5c6806fca0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5c74004550 tx=0x7f5c74009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:28.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:28.865+0000 7f5c7be00700 1 -- 172.21.15.1:0/2526073183 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5c70057b40 con 0x7f5c84070c40 2024-02-28T17:24:29.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:28 smithi001 ceph-mon[112940]: pgmap v1504: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:29.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:28 smithi001 ceph-mon[112940]: from='client.36636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:24:29.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.167+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5c8404bed0 con 0x7f5c84070c40 2024-02-28T17:24:29.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.168+0000 7f5c7be00700 1 -- 172.21.15.1:0/2526073183 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5c70013e60 con 0x7f5c84070c40 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:24:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:24:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:24:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:24:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:24:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:24:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:24:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:24:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:24:29.173 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:24:29.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.170+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c6806d7f0 msgr2=0x7f5c6806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.170+0000 7f5c8b000700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c6806d7f0 0x7f5c6806fca0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5c74004550 tx=0x7f5c74009040 comp rx=0 tx=0).stop 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.170+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 msgr2=0x7f5c84116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.170+0000 7f5c8b000700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 0x7f5c84116140 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f5c70004f50 tx=0x7f5c70009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.171+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 shutdown_connections 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.171+0000 7f5c8b000700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5c6806d7f0 0x7f5c6806fca0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.171+0000 7f5c8b000700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5c84070330 0x7f5c84115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.171+0000 7f5c8b000700 1 --2- 172.21.15.1:0/2526073183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c84070c40 0x7f5c84116140 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.171+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 >> 172.21.15.1:0/2526073183 conn(0x7f5c840f8ae0 msgr2=0x7f5c84103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.171+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 shutdown_connections 2024-02-28T17:24:29.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.171+0000 7f5c8b000700 1 -- 172.21.15.1:0/2526073183 wait complete. 2024-02-28T17:24:29.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:28 smithi066 ceph-mon[85781]: pgmap v1504: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:29.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:28 smithi066 ceph-mon[85781]: from='client.36636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:24:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.324+0000 7fcc51e00700 1 -- 172.21.15.1:0/3347779275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 msgr2=0x7fcc4c0fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.324+0000 7fcc51e00700 1 --2- 172.21.15.1:0/3347779275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 0x7fcc4c0fe330 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c004940 tx=0x7fcc3c009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.324+0000 7fcc51e00700 1 -- 172.21.15.1:0/3347779275 shutdown_connections 2024-02-28T17:24:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.324+0000 7fcc51e00700 1 --2- 172.21.15.1:0/3347779275 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc4c0fe900 0x7fcc4c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.324+0000 7fcc51e00700 1 --2- 172.21.15.1:0/3347779275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 0x7fcc4c0fe330 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.324+0000 7fcc51e00700 1 -- 172.21.15.1:0/3347779275 >> 172.21.15.1:0/3347779275 conn(0x7fcc4c0f7a10 msgr2=0x7fcc4c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.325+0000 7fcc51e00700 1 -- 172.21.15.1:0/3347779275 shutdown_connections 2024-02-28T17:24:29.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.325+0000 7fcc51e00700 1 -- 172.21.15.1:0/3347779275 wait complete. 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.325+0000 7fcc51e00700 1 Processor -- start 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.326+0000 7fcc51e00700 1 -- start start 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.326+0000 7fcc51e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 0x7fcc4c115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.326+0000 7fcc51e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc4c0fe900 0x7fcc4c1161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.326+0000 7fcc51e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c116820 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.326+0000 7fcc51e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c10fb50 con 0x7fcc4c0fe900 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.326+0000 7fcc4aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 0x7fcc4c115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.326+0000 7fcc4aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 0x7fcc4c115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40480/0 (socket says 172.21.15.1:40480) 2024-02-28T17:24:29.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.326+0000 7fcc4aa00700 1 -- 172.21.15.1:0/3654605835 learned_addr learned my addr 172.21.15.1:0/3654605835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:24:29.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc43e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc4c0fe900 0x7fcc4c1161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:29.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc4aa00700 1 -- 172.21.15.1:0/3654605835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc4c0fe900 msgr2=0x7fcc4c1161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:29.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc4aa00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc4c0fe900 0x7fcc4c1161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc4aa00700 1 -- 172.21.15.1:0/3654605835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c005040 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc43e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc4c0fe900 0x7fcc4c1161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:24:29.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc4aa00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 0x7fcc4c115c90 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009180 tx=0x7fcc3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:29.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc42a00700 1 -- 172.21.15.1:0/3654605835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc3c00e040 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc42a00700 1 -- 172.21.15.1:0/3654605835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc3c00bbf0 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc42a00700 1 -- 172.21.15.1:0/3654605835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc3c01c630 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.327+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c10fdd0 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.328+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c110310 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.329+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c106950 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.330+0000 7fcc42a00700 1 -- 172.21.15.1:0/3654605835 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcc3c00bd60 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.331+0000 7fcc42a00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc3006d7f0 0x7fcc3006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:29.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.331+0000 7fcc42a00700 1 -- 172.21.15.1:0/3654605835 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcc3c08e2a0 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.331+0000 7fcc43e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc3006d7f0 0x7fcc3006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:29.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.332+0000 7fcc43e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc3006d7f0 0x7fcc3006fca0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009fd0 tx=0x7fcc3400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:29.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.336+0000 7fcc42a00700 1 -- 172.21.15.1:0/3654605835 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcc3c05b680 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.608+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcc4c0fb5e0 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.608+0000 7fcc42a00700 1 -- 172.21.15.1:0/3654605835 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fcc3c05b210 con 0x7fcc4c0fdf60 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:24:29.613 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:24:29.614 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:24:29.615 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:24:29.616 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:24:29.616 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.611+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc3006d7f0 msgr2=0x7fcc3006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.611+0000 7fcc51e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc3006d7f0 0x7fcc3006fca0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009fd0 tx=0x7fcc3400b040 comp rx=0 tx=0).stop 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.611+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 msgr2=0x7fcc4c115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.611+0000 7fcc51e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 0x7fcc4c115c90 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009180 tx=0x7fcc3c009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.612+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 shutdown_connections 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.612+0000 7fcc51e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcc3006d7f0 0x7fcc3006fca0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.612+0000 7fcc51e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc4c0fdf60 0x7fcc4c115c90 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.612+0000 7fcc51e00700 1 --2- 172.21.15.1:0/3654605835 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcc4c0fe900 0x7fcc4c1161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.612+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 >> 172.21.15.1:0/3654605835 conn(0x7fcc4c0f7a10 msgr2=0x7fcc4c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.612+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 shutdown_connections 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.612+0000 7fcc51e00700 1 -- 172.21.15.1:0/3654605835 wait complete. 2024-02-28T17:24:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:24:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.764+0000 7fc7de000700 1 -- 172.21.15.1:0/1938632387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d8102f50 msgr2=0x7fc7d8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.764+0000 7fc7de000700 1 --2- 172.21.15.1:0/1938632387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d8102f50 0x7fc7d8103320 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8004940 tx=0x7fc7c8009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.764+0000 7fc7de000700 1 -- 172.21.15.1:0/1938632387 shutdown_connections 2024-02-28T17:24:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.764+0000 7fc7de000700 1 --2- 172.21.15.1:0/1938632387 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7d81038f0 0x7fc7d80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.764+0000 7fc7de000700 1 --2- 172.21.15.1:0/1938632387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d8102f50 0x7fc7d8103320 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.764+0000 7fc7de000700 1 -- 172.21.15.1:0/1938632387 >> 172.21.15.1:0/1938632387 conn(0x7fc7d80f8ae0 msgr2=0x7fc7d80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.765+0000 7fc7de000700 1 -- 172.21.15.1:0/1938632387 shutdown_connections 2024-02-28T17:24:29.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.765+0000 7fc7de000700 1 -- 172.21.15.1:0/1938632387 wait complete. 2024-02-28T17:24:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.765+0000 7fc7de000700 1 Processor -- start 2024-02-28T17:24:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.766+0000 7fc7de000700 1 -- start start 2024-02-28T17:24:29.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.766+0000 7fc7de000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d81038f0 0x7fc7d8115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.766+0000 7fc7de000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7d81162e0 0x7fc7d810fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.766+0000 7fc7de000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d8116860 con 0x7fc7d81038f0 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.766+0000 7fc7de000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d81169d0 con 0x7fc7d81162e0 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.766+0000 7fc7d7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d81038f0 0x7fc7d8115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.766+0000 7fc7d7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d81038f0 0x7fc7d8115da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40498/0 (socket says 172.21.15.1:40498) 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.766+0000 7fc7d7400700 1 -- 172.21.15.1:0/1829821315 learned_addr learned my addr 172.21.15.1:0/1829821315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.767+0000 7fc7d6a00700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7d81162e0 0x7fc7d810fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.767+0000 7fc7d7400700 1 -- 172.21.15.1:0/1829821315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7d81162e0 msgr2=0x7fc7d810fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.767+0000 7fc7d7400700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7d81162e0 0x7fc7d810fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.767+0000 7fc7d7400700 1 -- 172.21.15.1:0/1829821315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8005040 con 0x7fc7d81038f0 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.767+0000 7fc7d6a00700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7d81162e0 0x7fc7d810fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.767+0000 7fc7d7400700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d81038f0 0x7fc7d8115da0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8000c00 tx=0x7fc7c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.767+0000 7fc7cf400700 1 -- 172.21.15.1:0/1829821315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7c800e040 con 0x7fc7d81038f0 2024-02-28T17:24:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.768+0000 7fc7cf400700 1 -- 172.21.15.1:0/1829821315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7c8009430 con 0x7fc7d81038f0 2024-02-28T17:24:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.768+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d81102b0 con 0x7fc7d81038f0 2024-02-28T17:24:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.768+0000 7fc7cf400700 1 -- 172.21.15.1:0/1829821315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7c801c630 con 0x7fc7d81038f0 2024-02-28T17:24:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.768+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d81107f0 con 0x7fc7d81038f0 2024-02-28T17:24:29.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.769+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d8110b20 con 0x7fc7d81038f0 2024-02-28T17:24:29.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.770+0000 7fc7cf400700 1 -- 172.21.15.1:0/1829821315 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc7c800bbf0 con 0x7fc7d81038f0 2024-02-28T17:24:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.771+0000 7fc7cf400700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7bc06d7f0 0x7fc7bc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.771+0000 7fc7cf400700 1 -- 172.21.15.1:0/1829821315 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc7c808e2b0 con 0x7fc7d81038f0 2024-02-28T17:24:29.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.772+0000 7fc7d6a00700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7bc06d7f0 0x7fc7bc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:29.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.772+0000 7fc7d6a00700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7bc06d7f0 0x7fc7bc06fca0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4007fd0 tx=0x7fc7c4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:29.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:29.776+0000 7fc7cf400700 1 -- 172.21.15.1:0/1829821315 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc7d8110b20 con 0x7fc7d81038f0 2024-02-28T17:24:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.022+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7d8110b20 con 0x7fc7bc06d7f0 2024-02-28T17:24:30.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.024+0000 7fc7cf400700 1 -- 172.21.15.1:0/1829821315 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc7d8110b20 con 0x7fc7bc06d7f0 2024-02-28T17:24:30.027 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:24:30.027 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:24:30.027 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:24:30.028 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:24:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7bc06d7f0 msgr2=0x7fc7bc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7bc06d7f0 0x7fc7bc06fca0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4007fd0 tx=0x7fc7c4009040 comp rx=0 tx=0).stop 2024-02-28T17:24:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d81038f0 msgr2=0x7fc7d8115da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d81038f0 0x7fc7d8115da0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8000c00 tx=0x7fc7c8009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 shutdown_connections 2024-02-28T17:24:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc7bc06d7f0 0x7fc7bc06fca0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7d81038f0 0x7fc7d8115da0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 --2- 172.21.15.1:0/1829821315 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc7d81162e0 0x7fc7d810fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 >> 172.21.15.1:0/1829821315 conn(0x7fc7d80f8ae0 msgr2=0x7fc7d80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 shutdown_connections 2024-02-28T17:24:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.027+0000 7fc7de000700 1 -- 172.21.15.1:0/1829821315 wait complete. 2024-02-28T17:24:30.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.180+0000 7f8157400700 1 -- 172.21.15.1:0/3570840635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 msgr2=0x7f8150103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:30.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.180+0000 7f8157400700 1 --2- 172.21.15.1:0/3570840635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 0x7f8150103310 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f8144004940 tx=0x7f8144009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:30.185 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:29 smithi001 ceph-mon[112940]: from='client.36640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:24:30.185 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:29 smithi001 ceph-mon[112940]: from='client.36644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:24:30.185 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:29 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2526073183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:24:30.185 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:29 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3654605835' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:24:30.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.181+0000 7f8157400700 1 -- 172.21.15.1:0/3570840635 shutdown_connections 2024-02-28T17:24:30.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.181+0000 7f8157400700 1 --2- 172.21.15.1:0/3570840635 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f81501038e0 0x7f81500fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.181+0000 7f8157400700 1 --2- 172.21.15.1:0/3570840635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 0x7f8150103310 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.181+0000 7f8157400700 1 -- 172.21.15.1:0/3570840635 >> 172.21.15.1:0/3570840635 conn(0x7f81500f8af0 msgr2=0x7f81500faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:30.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.181+0000 7f8157400700 1 -- 172.21.15.1:0/3570840635 shutdown_connections 2024-02-28T17:24:30.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.181+0000 7f8157400700 1 -- 172.21.15.1:0/3570840635 wait complete. 2024-02-28T17:24:30.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.182+0000 7f8157400700 1 Processor -- start 2024-02-28T17:24:30.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.182+0000 7f8157400700 1 -- start start 2024-02-28T17:24:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.183+0000 7f8157400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 0x7f8150113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.183+0000 7f8157400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f81501038e0 0x7f8150113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.183+0000 7f8157400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81501145e0 con 0x7f8150102f40 2024-02-28T17:24:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.183+0000 7f8157400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815010d910 con 0x7f81501038e0 2024-02-28T17:24:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.183+0000 7f8154a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 0x7f8150113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.183+0000 7f8154a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 0x7f8150113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40508/0 (socket says 172.21.15.1:40508) 2024-02-28T17:24:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.183+0000 7f8154a00700 1 -- 172.21.15.1:0/2062097176 learned_addr learned my addr 172.21.15.1:0/2062097176 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:24:30.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.183+0000 7f814fe00700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f81501038e0 0x7f8150113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.184+0000 7f8154a00700 1 -- 172.21.15.1:0/2062097176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f81501038e0 msgr2=0x7f8150113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.184+0000 7f8154a00700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f81501038e0 0x7f8150113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.184+0000 7f8154a00700 1 -- 172.21.15.1:0/2062097176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8144005040 con 0x7f8150102f40 2024-02-28T17:24:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.184+0000 7f8154a00700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 0x7f8150113a50 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f814400b3b0 tx=0x7f8144009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:30.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.184+0000 7f814cc00700 1 -- 172.21.15.1:0/2062097176 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f814400e040 con 0x7f8150102f40 2024-02-28T17:24:30.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.184+0000 7f814cc00700 1 -- 172.21.15.1:0/2062097176 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f814400bbf0 con 0x7f8150102f40 2024-02-28T17:24:30.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.184+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815010db90 con 0x7f8150102f40 2024-02-28T17:24:30.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.184+0000 7f814cc00700 1 -- 172.21.15.1:0/2062097176 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f814401c630 con 0x7f8150102f40 2024-02-28T17:24:30.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.185+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815010e0d0 con 0x7f8150102f40 2024-02-28T17:24:30.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.186+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8150100250 con 0x7f8150102f40 2024-02-28T17:24:30.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.187+0000 7f814cc00700 1 -- 172.21.15.1:0/2062097176 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f814400bd60 con 0x7f8150102f40 2024-02-28T17:24:30.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.187+0000 7f814cc00700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7e0 0x7f813806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:24:30.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.188+0000 7f814cc00700 1 -- 172.21.15.1:0/2062097176 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f814408e380 con 0x7f8150102f40 2024-02-28T17:24:30.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.188+0000 7f814fe00700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7e0 0x7f813806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:24:30.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.188+0000 7f814fe00700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7e0 0x7f813806fc90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f815010f280 tx=0x7f813c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:24:30.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.192+0000 7f814cc00700 1 -- 172.21.15.1:0/2062097176 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f814405b6e0 con 0x7f8150102f40 2024-02-28T17:24:30.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:29 smithi066 ceph-mon[85781]: from='client.36640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:24:30.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:29 smithi066 ceph-mon[85781]: from='client.36644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:24:30.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:29 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2526073183' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:24:30.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:29 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3654605835' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:24:30.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.501+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f815004bed0 con 0x7f8150102f40 2024-02-28T17:24:30.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.502+0000 7f814cc00700 1 -- 172.21.15.1:0/2062097176 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f814405b270 con 0x7f8150102f40 2024-02-28T17:24:30.507 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:24:30.507 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:24:30.507 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:24:30.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:24:30.507 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:24:30.507 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:24:30.508 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:24:30.509 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:24:30.509 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:24:30.509 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:24:30.509 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:24:30.509 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:24:30.509 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.505+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7e0 msgr2=0x7f813806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.505+0000 7f8157400700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7e0 0x7f813806fc90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f815010f280 tx=0x7f813c00b040 comp rx=0 tx=0).stop 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.505+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 msgr2=0x7f8150113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.505+0000 7f8157400700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 0x7f8150113a50 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f814400b3b0 tx=0x7f8144009e30 comp rx=0 tx=0).stop 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.506+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 shutdown_connections 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.506+0000 7f8157400700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f813806d7e0 0x7f813806fc90 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.506+0000 7f8157400700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8150102f40 0x7f8150113a50 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.506+0000 7f8157400700 1 --2- 172.21.15.1:0/2062097176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f81501038e0 0x7f8150113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.506+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 >> 172.21.15.1:0/2062097176 conn(0x7f81500f8af0 msgr2=0x7f81500fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.506+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 shutdown_connections 2024-02-28T17:24:30.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:24:30.506+0000 7f8157400700 1 -- 172.21.15.1:0/2062097176 wait complete. 2024-02-28T17:24:31.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:30 smithi066 ceph-mon[85781]: pgmap v1505: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:31.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:30 smithi066 ceph-mon[85781]: from='client.36656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:24:31.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2062097176' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:24:31.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:30 smithi001 ceph-mon[112940]: pgmap v1505: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:31.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:30 smithi001 ceph-mon[112940]: from='client.36656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:24:31.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2062097176' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:24:33.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:32 smithi066 ceph-mon[85781]: pgmap v1506: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:33.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:32 smithi001 ceph-mon[112940]: pgmap v1506: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:35.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:34 smithi066 ceph-mon[85781]: pgmap v1507: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:35.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:34 smithi001 ceph-mon[112940]: pgmap v1507: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:37.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:36 smithi066 ceph-mon[85781]: pgmap v1508: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:37.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:36 smithi001 ceph-mon[112940]: pgmap v1508: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:39.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:38 smithi066 ceph-mon[85781]: pgmap v1509: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:39.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:38 smithi001 ceph-mon[112940]: pgmap v1509: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:41.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:40 smithi066 ceph-mon[85781]: pgmap v1510: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:41.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:40 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:24:41.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:40 smithi001 ceph-mon[112940]: pgmap v1510: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:41.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:40 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:24:43.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:42 smithi066 ceph-mon[85781]: pgmap v1511: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:43.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:42 smithi001 ceph-mon[112940]: pgmap v1511: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:45.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:44 smithi066 ceph-mon[85781]: pgmap v1512: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:45.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:44 smithi001 ceph-mon[112940]: pgmap v1512: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:47.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:46 smithi066 ceph-mon[85781]: pgmap v1513: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:47.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:46 smithi001 ceph-mon[112940]: pgmap v1513: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:49.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:48 smithi066 ceph-mon[85781]: pgmap v1514: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:48 smithi001 ceph-mon[112940]: pgmap v1514: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:51.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:50 smithi066 ceph-mon[85781]: pgmap v1515: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:51.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:50 smithi001 ceph-mon[112940]: pgmap v1515: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:53.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:52 smithi066 ceph-mon[85781]: pgmap v1516: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:53.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:52 smithi001 ceph-mon[112940]: pgmap v1516: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:55.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:54 smithi066 ceph-mon[85781]: pgmap v1517: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:55.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:54 smithi001 ceph-mon[112940]: pgmap v1517: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:56.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:24:56.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:24:57.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:56 smithi066 ceph-mon[85781]: pgmap v1518: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:57.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:56 smithi001 ceph-mon[112940]: pgmap v1518: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:59.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:24:58 smithi066 ceph-mon[85781]: pgmap v1519: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:24:59.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:24:58 smithi001 ceph-mon[112940]: pgmap v1519: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.678+0000 7f07da400700 1 -- 172.21.15.1:0/4164407335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105b00 msgr2=0x7f07d40ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.678+0000 7f07da400700 1 --2- 172.21.15.1:0/4164407335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105b00 0x7f07d40ff170 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f07c0004940 tx=0x7f07c0009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.678+0000 7f07da400700 1 -- 172.21.15.1:0/4164407335 shutdown_connections 2024-02-28T17:25:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.678+0000 7f07da400700 1 --2- 172.21.15.1:0/4164407335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105b00 0x7f07d40ff170 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.678+0000 7f07da400700 1 --2- 172.21.15.1:0/4164407335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07d4105160 0x7f07d4105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.678+0000 7f07da400700 1 -- 172.21.15.1:0/4164407335 >> 172.21.15.1:0/4164407335 conn(0x7f07d40facb0 msgr2=0x7f07d40fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.678+0000 7f07da400700 1 -- 172.21.15.1:0/4164407335 shutdown_connections 2024-02-28T17:25:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.678+0000 7f07da400700 1 -- 172.21.15.1:0/4164407335 wait complete. 2024-02-28T17:25:00.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.679+0000 7f07da400700 1 Processor -- start 2024-02-28T17:25:00.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.680+0000 7f07da400700 1 -- start start 2024-02-28T17:25:00.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.680+0000 7f07da400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105160 0x7f07d4115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:00.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.680+0000 7f07da400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07d4105b00 0x7f07d41161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.680+0000 7f07da400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4116820 con 0x7f07d4105160 2024-02-28T17:25:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.680+0000 7f07da400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d410fb50 con 0x7f07d4105b00 2024-02-28T17:25:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.680+0000 7f07d3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105160 0x7f07d4115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.680+0000 7f07d3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105160 0x7f07d4115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53348/0 (socket says 172.21.15.1:53348) 2024-02-28T17:25:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.680+0000 7f07d3400700 1 -- 172.21.15.1:0/1771238978 learned_addr learned my addr 172.21.15.1:0/1771238978 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.681+0000 7f07d2a00700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07d4105b00 0x7f07d41161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.681+0000 7f07d3400700 1 -- 172.21.15.1:0/1771238978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07d4105b00 msgr2=0x7f07d41161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.681+0000 7f07d3400700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07d4105b00 0x7f07d41161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.681+0000 7f07d3400700 1 -- 172.21.15.1:0/1771238978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c0005040 con 0x7f07d4105160 2024-02-28T17:25:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.681+0000 7f07d3400700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105160 0x7f07d4115c90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f07c400c8c0 tx=0x7f07c40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.681+0000 7f07cb400700 1 -- 172.21.15.1:0/1771238978 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07c400a980 con 0x7f07d4105160 2024-02-28T17:25:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.681+0000 7f07cb400700 1 -- 172.21.15.1:0/1771238978 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07c400ce60 con 0x7f07d4105160 2024-02-28T17:25:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.681+0000 7f07cb400700 1 -- 172.21.15.1:0/1771238978 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07c40115e0 con 0x7f07d4105160 2024-02-28T17:25:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.682+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d410fe30 con 0x7f07d4105160 2024-02-28T17:25:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.682+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d4110370 con 0x7f07d4105160 2024-02-28T17:25:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.683+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d4107630 con 0x7f07d4105160 2024-02-28T17:25:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.684+0000 7f07cb400700 1 -- 172.21.15.1:0/1771238978 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f07c4012400 con 0x7f07d4105160 2024-02-28T17:25:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.685+0000 7f07cb400700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07b806d7f0 0x7f07b806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.685+0000 7f07cb400700 1 -- 172.21.15.1:0/1771238978 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f07c408cf60 con 0x7f07d4105160 2024-02-28T17:25:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.685+0000 7f07d2a00700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07b806d7f0 0x7f07b806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.686+0000 7f07d2a00700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07b806d7f0 0x7f07b806fca0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f07d41114c0 tx=0x7f07c0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.690+0000 7f07cb400700 1 -- 172.21.15.1:0/1771238978 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f07c405a340 con 0x7f07d4105160 2024-02-28T17:25:00.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.932+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07d4111200 con 0x7f07b806d7f0 2024-02-28T17:25:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.934+0000 7f07cb400700 1 -- 172.21.15.1:0/1771238978 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f07d4111200 con 0x7f07b806d7f0 2024-02-28T17:25:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07b806d7f0 msgr2=0x7f07b806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07b806d7f0 0x7f07b806fca0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f07d41114c0 tx=0x7f07c0009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105160 msgr2=0x7f07d4115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105160 0x7f07d4115c90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f07c400c8c0 tx=0x7f07c40044c0 comp rx=0 tx=0).stop 2024-02-28T17:25:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 shutdown_connections 2024-02-28T17:25:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f07b806d7f0 0x7f07b806fca0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07d4105160 0x7f07d4115c90 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 --2- 172.21.15.1:0/1771238978 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f07d4105b00 0x7f07d41161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.936+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 >> 172.21.15.1:0/1771238978 conn(0x7f07d40facb0 msgr2=0x7f07d40fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.937+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 shutdown_connections 2024-02-28T17:25:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:00.937+0000 7f07da400700 1 -- 172.21.15.1:0/1771238978 wait complete. 2024-02-28T17:25:00.954 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:25:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.095+0000 7feaea800700 1 -- 172.21.15.1:0/1578285797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 msgr2=0x7feae40fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.095+0000 7feaea800700 1 --2- 172.21.15.1:0/1578285797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 0x7feae40fcf60 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7feacc009ce0 tx=0x7feacc004d10 comp rx=0 tx=0).stop 2024-02-28T17:25:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.095+0000 7feaea800700 1 -- 172.21.15.1:0/1578285797 shutdown_connections 2024-02-28T17:25:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.095+0000 7feaea800700 1 --2- 172.21.15.1:0/1578285797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 0x7feae40fcf60 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.095+0000 7feaea800700 1 --2- 172.21.15.1:0/1578285797 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feae4102f50 0x7feae4103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.095+0000 7feaea800700 1 -- 172.21.15.1:0/1578285797 >> 172.21.15.1:0/1578285797 conn(0x7feae40f8ae0 msgr2=0x7feae40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.095+0000 7feaea800700 1 -- 172.21.15.1:0/1578285797 shutdown_connections 2024-02-28T17:25:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.095+0000 7feaea800700 1 -- 172.21.15.1:0/1578285797 wait complete. 2024-02-28T17:25:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.096+0000 7feaea800700 1 Processor -- start 2024-02-28T17:25:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.096+0000 7feaea800700 1 -- start start 2024-02-28T17:25:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.097+0000 7feaea800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feae4102f50 0x7feae4113c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.097+0000 7feaea800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 0x7feae4114150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.097+0000 7feaea800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae41147a0 con 0x7feae41038f0 2024-02-28T17:25:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.097+0000 7feaea800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae410dad0 con 0x7feae4102f50 2024-02-28T17:25:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.097+0000 7feadaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 0x7feae4114150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.097+0000 7feadaa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 0x7feae4114150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53354/0 (socket says 172.21.15.1:53354) 2024-02-28T17:25:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.097+0000 7feadaa00700 1 -- 172.21.15.1:0/169970291 learned_addr learned my addr 172.21.15.1:0/169970291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.097+0000 7feae3400700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feae4102f50 0x7feae4113c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feadaa00700 1 -- 172.21.15.1:0/169970291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feae4102f50 msgr2=0x7feae4113c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feadaa00700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feae4102f50 0x7feae4113c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feadaa00700 1 -- 172.21.15.1:0/169970291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc009940 con 0x7feae41038f0 2024-02-28T17:25:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feadaa00700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 0x7feae4114150 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7feacc00be40 tx=0x7feacc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feadbe00700 1 -- 172.21.15.1:0/169970291 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feacc00e040 con 0x7feae41038f0 2024-02-28T17:25:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feadbe00700 1 -- 172.21.15.1:0/169970291 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feacc008ea0 con 0x7feae41038f0 2024-02-28T17:25:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae410dd50 con 0x7feae41038f0 2024-02-28T17:25:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feadbe00700 1 -- 172.21.15.1:0/169970291 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feacc01c740 con 0x7feae41038f0 2024-02-28T17:25:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.098+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae410e290 con 0x7feae41038f0 2024-02-28T17:25:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.100+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae4105420 con 0x7feae41038f0 2024-02-28T17:25:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.100+0000 7feadbe00700 1 -- 172.21.15.1:0/169970291 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7feacc012b70 con 0x7feae41038f0 2024-02-28T17:25:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.101+0000 7feadbe00700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feac806d7f0 0x7feac806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.102+0000 7feadbe00700 1 -- 172.21.15.1:0/169970291 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7feacc08e9b0 con 0x7feae41038f0 2024-02-28T17:25:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.102+0000 7feae3400700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feac806d7f0 0x7feac806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.102+0000 7feae3400700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feac806d7f0 0x7feac806fca0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fead4009fd0 tx=0x7fead4009f20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:01.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.106+0000 7feadbe00700 1 -- 172.21.15.1:0/169970291 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feacc05bd10 con 0x7feae41038f0 2024-02-28T17:25:01.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:00 smithi066 ceph-mon[85781]: pgmap v1520: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:01.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:00 smithi001 ceph-mon[112940]: pgmap v1520: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:01.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.350+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feae410a050 con 0x7feac806d7f0 2024-02-28T17:25:01.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.351+0000 7feadbe00700 1 -- 172.21.15.1:0/169970291 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feae410a050 con 0x7feac806d7f0 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feac806d7f0 msgr2=0x7feac806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feac806d7f0 0x7feac806fca0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fead4009fd0 tx=0x7fead4009f20 comp rx=0 tx=0).stop 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 msgr2=0x7feae4114150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 0x7feae4114150 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7feacc00be40 tx=0x7feacc004d10 comp rx=0 tx=0).stop 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 shutdown_connections 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7feac806d7f0 0x7feac806fca0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7feae4102f50 0x7feae4113c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 --2- 172.21.15.1:0/169970291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feae41038f0 0x7feae4114150 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 >> 172.21.15.1:0/169970291 conn(0x7feae40f8ae0 msgr2=0x7feae40fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.354+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 shutdown_connections 2024-02-28T17:25:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.355+0000 7feaea800700 1 -- 172.21.15.1:0/169970291 wait complete. 2024-02-28T17:25:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.511+0000 7ff157200700 1 -- 172.21.15.1:0/32750843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff150102f50 msgr2=0x7ff150103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.511+0000 7ff157200700 1 --2- 172.21.15.1:0/32750843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff150102f50 0x7ff150103320 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff140004940 tx=0x7ff140009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.511+0000 7ff157200700 1 -- 172.21.15.1:0/32750843 shutdown_connections 2024-02-28T17:25:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.511+0000 7ff157200700 1 --2- 172.21.15.1:0/32750843 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff1501038f0 0x7ff1500fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.511+0000 7ff157200700 1 --2- 172.21.15.1:0/32750843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff150102f50 0x7ff150103320 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.511+0000 7ff157200700 1 -- 172.21.15.1:0/32750843 >> 172.21.15.1:0/32750843 conn(0x7ff1500f8ae0 msgr2=0x7ff1500faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:01.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.511+0000 7ff157200700 1 -- 172.21.15.1:0/32750843 shutdown_connections 2024-02-28T17:25:01.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.512+0000 7ff157200700 1 -- 172.21.15.1:0/32750843 wait complete. 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.512+0000 7ff157200700 1 Processor -- start 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.513+0000 7ff157200700 1 -- start start 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.513+0000 7ff157200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff150102f50 0x7ff150115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.513+0000 7ff157200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1501038f0 0x7ff150116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.513+0000 7ff157200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1501167b0 con 0x7ff1501038f0 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.513+0000 7ff157200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15010fb00 con 0x7ff150102f50 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.513+0000 7ff14f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1501038f0 0x7ff150116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.514+0000 7ff14f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1501038f0 0x7ff150116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53386/0 (socket says 172.21.15.1:53386) 2024-02-28T17:25:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.514+0000 7ff14f400700 1 -- 172.21.15.1:0/3792377325 learned_addr learned my addr 172.21.15.1:0/3792377325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.514+0000 7ff14fe00700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff150102f50 0x7ff150115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.514+0000 7ff14f400700 1 -- 172.21.15.1:0/3792377325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff150102f50 msgr2=0x7ff150115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.514+0000 7ff14f400700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff150102f50 0x7ff150115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.514+0000 7ff14f400700 1 -- 172.21.15.1:0/3792377325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff140005040 con 0x7ff1501038f0 2024-02-28T17:25:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.514+0000 7ff14f400700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1501038f0 0x7ff150116160 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00eb40 tx=0x7ff13c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.514+0000 7ff147e00700 1 -- 172.21.15.1:0/3792377325 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff13c007cc0 con 0x7ff1501038f0 2024-02-28T17:25:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.515+0000 7ff147e00700 1 -- 172.21.15.1:0/3792377325 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff13c00f040 con 0x7ff1501038f0 2024-02-28T17:25:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.515+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15010fd80 con 0x7ff1501038f0 2024-02-28T17:25:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.515+0000 7ff147e00700 1 -- 172.21.15.1:0/3792377325 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff13c0095e0 con 0x7ff1501038f0 2024-02-28T17:25:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.515+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1501102c0 con 0x7ff1501038f0 2024-02-28T17:25:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.516+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1500fc6e0 con 0x7ff1501038f0 2024-02-28T17:25:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.517+0000 7ff147e00700 1 -- 172.21.15.1:0/3792377325 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff13c009740 con 0x7ff1501038f0 2024-02-28T17:25:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.518+0000 7ff147e00700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff13406d7f0 0x7ff13406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.518+0000 7ff147e00700 1 -- 172.21.15.1:0/3792377325 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff13c08c960 con 0x7ff1501038f0 2024-02-28T17:25:01.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.519+0000 7ff14fe00700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff13406d7f0 0x7ff13406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.519+0000 7ff14fe00700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff13406d7f0 0x7ff13406fca0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff140008fe0 tx=0x7ff140009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:01.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.523+0000 7ff147e00700 1 -- 172.21.15.1:0/3792377325 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff13c056240 con 0x7ff1501038f0 2024-02-28T17:25:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.749+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1501112c0 con 0x7ff13406d7f0 2024-02-28T17:25:01.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.779+0000 7ff147e00700 1 -- 172.21.15.1:0/3792377325 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff1501112c0 con 0x7ff13406d7f0 2024-02-28T17:25:01.782 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:25:01.782 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (51m) 12m ago 65m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:25:01.782 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (47m) 12m ago 65m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (47m) 12m ago 64m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (50m) 12m ago 65m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (43m) 12m ago 58m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (43m) 12m ago 58m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (12m) 12m ago 58m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (28m) 12m ago 58m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (53m) 12m ago 67m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (52m) 12m ago 64m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (49m) 12m ago 67m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (48m) 12m ago 63m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (52m) 12m ago 65m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:25:01.783 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (52m) 12m ago 64m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:25:01.784 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (47m) 12m ago 63m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:25:01.784 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (46m) 12m ago 62m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:25:01.784 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (46m) 12m ago 61m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:25:01.784 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (45m) 12m ago 61m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:25:01.784 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (44m) 12m ago 60m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:25:01.784 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (44m) 12m ago 60m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:25:01.784 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (51m) 12m ago 64m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.781+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff13406d7f0 msgr2=0x7ff13406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.781+0000 7ff157200700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff13406d7f0 0x7ff13406fca0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff140008fe0 tx=0x7ff140009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.781+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1501038f0 msgr2=0x7ff150116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.781+0000 7ff157200700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1501038f0 0x7ff150116160 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00eb40 tx=0x7ff13c00b040 comp rx=0 tx=0).stop 2024-02-28T17:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.782+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 shutdown_connections 2024-02-28T17:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.782+0000 7ff157200700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff13406d7f0 0x7ff13406fca0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.782+0000 7ff157200700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff150102f50 0x7ff150115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.782+0000 7ff157200700 1 --2- 172.21.15.1:0/3792377325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1501038f0 0x7ff150116160 secure :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00eb40 tx=0x7ff13c00b040 comp rx=0 tx=0).stop 2024-02-28T17:25:01.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.782+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 >> 172.21.15.1:0/3792377325 conn(0x7ff1500f8ae0 msgr2=0x7ff1500fa340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:01.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.782+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 shutdown_connections 2024-02-28T17:25:01.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.782+0000 7ff157200700 1 -- 172.21.15.1:0/3792377325 wait complete. 2024-02-28T17:25:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.933+0000 7f707a200700 1 -- 172.21.15.1:0/229542129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 msgr2=0x7f70740fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.933+0000 7f707a200700 1 --2- 172.21.15.1:0/229542129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 0x7f70740fe330 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f7064009ce0 tx=0x7f7064004d10 comp rx=0 tx=0).stop 2024-02-28T17:25:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.933+0000 7f707a200700 1 -- 172.21.15.1:0/229542129 shutdown_connections 2024-02-28T17:25:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.933+0000 7f707a200700 1 --2- 172.21.15.1:0/229542129 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70740fe900 0x7f7074106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.933+0000 7f707a200700 1 --2- 172.21.15.1:0/229542129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 0x7f70740fe330 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.933+0000 7f707a200700 1 -- 172.21.15.1:0/229542129 >> 172.21.15.1:0/229542129 conn(0x7f70740f7a10 msgr2=0x7f70740f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.933+0000 7f707a200700 1 -- 172.21.15.1:0/229542129 shutdown_connections 2024-02-28T17:25:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.933+0000 7f707a200700 1 -- 172.21.15.1:0/229542129 wait complete. 2024-02-28T17:25:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.934+0000 7f707a200700 1 Processor -- start 2024-02-28T17:25:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.934+0000 7f707a200700 1 -- start start 2024-02-28T17:25:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f707a200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 0x7f70740fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f707a200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70740fe900 0x7f70740fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f707a200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70740fc8d0 con 0x7f70740fdf60 2024-02-28T17:25:01.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f707a200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70740fca40 con 0x7f70740fe900 2024-02-28T17:25:01.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f7073400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 0x7f70740fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f7073400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 0x7f70740fbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33412/0 (socket says 172.21.15.1:33412) 2024-02-28T17:25:01.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f7073400700 1 -- 172.21.15.1:0/1636177184 learned_addr learned my addr 172.21.15.1:0/1636177184 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:01.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f706aa00700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70740fe900 0x7f70740fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f7073400700 1 -- 172.21.15.1:0/1636177184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70740fe900 msgr2=0x7f70740fc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:01.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.935+0000 7f7073400700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70740fe900 0x7f70740fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:01.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.936+0000 7f7073400700 1 -- 172.21.15.1:0/1636177184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064009940 con 0x7f70740fdf60 2024-02-28T17:25:01.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.936+0000 7f7073400700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 0x7f70740fbd90 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f706400ba80 tx=0x7f7064004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:01.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.936+0000 7f706be00700 1 -- 172.21.15.1:0/1636177184 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f706400e040 con 0x7f70740fdf60 2024-02-28T17:25:01.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.936+0000 7f706be00700 1 -- 172.21.15.1:0/1636177184 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f706401cd00 con 0x7f70740fdf60 2024-02-28T17:25:01.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.936+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7074119d90 con 0x7f70740fdf60 2024-02-28T17:25:01.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.936+0000 7f706be00700 1 -- 172.21.15.1:0/1636177184 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7064012be0 con 0x7f70740fdf60 2024-02-28T17:25:01.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.936+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707411a1f0 con 0x7f70740fdf60 2024-02-28T17:25:01.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.938+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7074104550 con 0x7f70740fdf60 2024-02-28T17:25:01.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.939+0000 7f706be00700 1 -- 172.21.15.1:0/1636177184 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f7064012410 con 0x7f70740fdf60 2024-02-28T17:25:01.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.940+0000 7f706be00700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f705806d7e0 0x7f705806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:01.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.940+0000 7f706be00700 1 -- 172.21.15.1:0/1636177184 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f706408f200 con 0x7f70740fdf60 2024-02-28T17:25:01.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.940+0000 7f706aa00700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f705806d7e0 0x7f705806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:01.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.941+0000 7f706aa00700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f705806d7e0 0x7f705806fc90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f705c004600 tx=0x7f705c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:01.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:01.945+0000 7f706be00700 1 -- 172.21.15.1:0/1636177184 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7064058a50 con 0x7f70740fdf60 2024-02-28T17:25:02.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:01 smithi001 ceph-mon[112940]: from='client.36664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.257+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f70740fb610 con 0x7f70740fdf60 2024-02-28T17:25:02.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.258+0000 7f706be00700 1 -- 172.21.15.1:0/1636177184 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f706405c110 con 0x7f70740fdf60 2024-02-28T17:25:02.261 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:25:02.261 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:25:02.261 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:25:02.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:25:02.261 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:25:02.262 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:25:02.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:25:02.262 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:25:02.262 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:25:02.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:25:02.262 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:25:02.262 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:25:02.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:25:02.263 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:25:02.263 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:25:02.263 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:25:02.263 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:25:02.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:01 smithi066 ceph-mon[85781]: from='client.36664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.261+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f705806d7e0 msgr2=0x7f705806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.261+0000 7f707a200700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f705806d7e0 0x7f705806fc90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f705c004600 tx=0x7f705c00b040 comp rx=0 tx=0).stop 2024-02-28T17:25:02.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.261+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 msgr2=0x7f70740fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.261+0000 7f707a200700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 0x7f70740fbd90 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f706400ba80 tx=0x7f7064004d10 comp rx=0 tx=0).stop 2024-02-28T17:25:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.262+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 shutdown_connections 2024-02-28T17:25:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.262+0000 7f707a200700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f705806d7e0 0x7f705806fc90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.262+0000 7f707a200700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70740fdf60 0x7f70740fbd90 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.262+0000 7f707a200700 1 --2- 172.21.15.1:0/1636177184 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70740fe900 0x7f70740fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.262+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 >> 172.21.15.1:0/1636177184 conn(0x7f70740f7a10 msgr2=0x7f70741016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.262+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 shutdown_connections 2024-02-28T17:25:02.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.262+0000 7f707a200700 1 -- 172.21.15.1:0/1636177184 wait complete. 2024-02-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.411+0000 7f5813a00700 1 -- 172.21.15.1:0/3956609907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 msgr2=0x7f580c109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.411+0000 7f5813a00700 1 --2- 172.21.15.1:0/3956609907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 0x7f580c109260 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f57f4004970 tx=0x7f57f4009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:02.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.411+0000 7f5813a00700 1 -- 172.21.15.1:0/3956609907 shutdown_connections 2024-02-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.411+0000 7f5813a00700 1 --2- 172.21.15.1:0/3956609907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 0x7f580c109260 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.411+0000 7f5813a00700 1 --2- 172.21.15.1:0/3956609907 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f580c070330 0x7f580c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.411+0000 7f5813a00700 1 -- 172.21.15.1:0/3956609907 >> 172.21.15.1:0/3956609907 conn(0x7f580c0f8ae0 msgr2=0x7f580c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.411+0000 7f5813a00700 1 -- 172.21.15.1:0/3956609907 shutdown_connections 2024-02-28T17:25:02.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.412+0000 7f5813a00700 1 -- 172.21.15.1:0/3956609907 wait complete. 2024-02-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.412+0000 7f5813a00700 1 Processor -- start 2024-02-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.413+0000 7f5813a00700 1 -- start start 2024-02-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.413+0000 7f5813a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f580c070330 0x7f580c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.413+0000 7f5813a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 0x7f580c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.413+0000 7f5813a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c1168c0 con 0x7f580c070c40 2024-02-28T17:25:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.413+0000 7f5813a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c10fb40 con 0x7f580c070330 2024-02-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.413+0000 7f5803e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 0x7f580c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.413+0000 7f5803e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 0x7f580c1161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33432/0 (socket says 172.21.15.1:33432) 2024-02-28T17:25:02.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.413+0000 7f5803e00700 1 -- 172.21.15.1:0/2817454502 learned_addr learned my addr 172.21.15.1:0/2817454502 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5803e00700 1 -- 172.21.15.1:0/2817454502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f580c070330 msgr2=0x7f580c115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5811000700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f580c070330 0x7f580c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5803e00700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f580c070330 0x7f580c115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5803e00700 1 -- 172.21.15.1:0/2817454502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f4005040 con 0x7f580c070c40 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5811000700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f580c070330 0x7f580c115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5803e00700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 0x7f580c1161e0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f57f4000c00 tx=0x7f57f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5809600700 1 -- 172.21.15.1:0/2817454502 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57f400e040 con 0x7f580c070c40 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5809600700 1 -- 172.21.15.1:0/2817454502 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57f4004290 con 0x7f580c070c40 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c10fd60 con 0x7f580c070c40 2024-02-28T17:25:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5809600700 1 -- 172.21.15.1:0/2817454502 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57f401c630 con 0x7f580c070c40 2024-02-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.414+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c1102a0 con 0x7f580c070c40 2024-02-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.415+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c106920 con 0x7f580c070c40 2024-02-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.416+0000 7f5809600700 1 -- 172.21.15.1:0/2817454502 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f57f4013980 con 0x7f580c070c40 2024-02-28T17:25:02.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.417+0000 7f5809600700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57f006d7e0 0x7f57f006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.417+0000 7f5809600700 1 -- 172.21.15.1:0/2817454502 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f57f408e160 con 0x7f580c070c40 2024-02-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.418+0000 7f5811000700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57f006d7e0 0x7f57f006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:02.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.418+0000 7f5811000700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57f006d7e0 0x7f57f006fc90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f57fc006fd0 tx=0x7f57fc008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:02.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.422+0000 7f5809600700 1 -- 172.21.15.1:0/2817454502 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f57f405b540 con 0x7f580c070c40 2024-02-28T17:25:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.687+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f580c111280 con 0x7f580c070c40 2024-02-28T17:25:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.688+0000 7f5809600700 1 -- 172.21.15.1:0/2817454502 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f57f405b0d0 con 0x7f580c070c40 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:25:02.691 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:25:02.692 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:25:02.693 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:25:02.694 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:25:02.694 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:25:02.694 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:25:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57f006d7e0 msgr2=0x7f57f006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57f006d7e0 0x7f57f006fc90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f57fc006fd0 tx=0x7f57fc008040 comp rx=0 tx=0).stop 2024-02-28T17:25:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 msgr2=0x7f580c1161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 0x7f580c1161e0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f57f4000c00 tx=0x7f57f4009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 shutdown_connections 2024-02-28T17:25:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f57f006d7e0 0x7f57f006fc90 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f580c070330 0x7f580c115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 --2- 172.21.15.1:0/2817454502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c070c40 0x7f580c1161e0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.692+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 >> 172.21.15.1:0/2817454502 conn(0x7f580c0f8ae0 msgr2=0x7f580c103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.693+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 shutdown_connections 2024-02-28T17:25:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.693+0000 7f5813a00700 1 -- 172.21.15.1:0/2817454502 wait complete. 2024-02-28T17:25:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.845+0000 7f7536200700 1 -- 172.21.15.1:0/935112284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 msgr2=0x7f75301099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.845+0000 7f7536200700 1 --2- 172.21.15.1:0/935112284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 0x7f75301099c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7520004940 tx=0x7f7520009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.846+0000 7f7536200700 1 -- 172.21.15.1:0/935112284 shutdown_connections 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.846+0000 7f7536200700 1 --2- 172.21.15.1:0/935112284 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7530103dc0 0x7f7530104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.846+0000 7f7536200700 1 --2- 172.21.15.1:0/935112284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 0x7f75301099c0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.846+0000 7f7536200700 1 -- 172.21.15.1:0/935112284 >> 172.21.15.1:0/935112284 conn(0x7f7530072d50 msgr2=0x7f7530075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.846+0000 7f7536200700 1 -- 172.21.15.1:0/935112284 shutdown_connections 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.846+0000 7f7536200700 1 -- 172.21.15.1:0/935112284 wait complete. 2024-02-28T17:25:02.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.846+0000 7f7536200700 1 Processor -- start 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.847+0000 7f7536200700 1 -- start start 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.847+0000 7f7536200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7530103dc0 0x7f753011a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.847+0000 7f7536200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 0x7f753011a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.847+0000 7f7536200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753011ac70 con 0x7f75301095f0 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.847+0000 7f7536200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530113f80 con 0x7f7530103dc0 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.847+0000 7f752ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 0x7f753011a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.847+0000 7f752ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 0x7f753011a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33444/0 (socket says 172.21.15.1:33444) 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.847+0000 7f752ea00700 1 -- 172.21.15.1:0/2328516947 learned_addr learned my addr 172.21.15.1:0/2328516947 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f752f400700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7530103dc0 0x7f753011a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:02.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f752ea00700 1 -- 172.21.15.1:0/2328516947 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7530103dc0 msgr2=0x7f753011a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f752ea00700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7530103dc0 0x7f753011a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f752ea00700 1 -- 172.21.15.1:0/2328516947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7520005040 con 0x7f75301095f0 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f752ea00700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 0x7f753011a620 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f751c00c830 tx=0x7f751c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f7527400700 1 -- 172.21.15.1:0/2328516947 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f751c00a760 con 0x7f75301095f0 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f7527400700 1 -- 172.21.15.1:0/2328516947 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f751c00ad90 con 0x7f75301095f0 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f7527400700 1 -- 172.21.15.1:0/2328516947 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f751c0113a0 con 0x7f75301095f0 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.848+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7530114260 con 0x7f75301095f0 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.849+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75301147a0 con 0x7f75301095f0 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.850+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7530106850 con 0x7f75301095f0 2024-02-28T17:25:02.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.851+0000 7f7527400700 1 -- 172.21.15.1:0/2328516947 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f751c012400 con 0x7f75301095f0 2024-02-28T17:25:02.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.852+0000 7f7527400700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f751406d7f0 0x7f751406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:02.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.852+0000 7f7527400700 1 -- 172.21.15.1:0/2328516947 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f751c08cf00 con 0x7f75301095f0 2024-02-28T17:25:02.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.852+0000 7f752f400700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f751406d7f0 0x7f751406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:02.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.852+0000 7f752f400700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f751406d7f0 0x7f751406fca0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f752000b020 tx=0x7f7520009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:02.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:02.856+0000 7f7527400700 1 -- 172.21.15.1:0/2328516947 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f751c05a2e0 con 0x7f75301095f0 2024-02-28T17:25:03.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.102+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f753004bf40 con 0x7f751406d7f0 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.104+0000 7f7527400700 1 -- 172.21.15.1:0/2328516947 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f753004bf40 con 0x7f751406d7f0 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:25:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:25:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:25:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:25:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:25:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:25:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:25:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:25:03.108 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:25:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f751406d7f0 msgr2=0x7f751406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f751406d7f0 0x7f751406fca0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f752000b020 tx=0x7f7520009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 msgr2=0x7f753011a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 0x7f753011a620 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f751c00c830 tx=0x7f751c004470 comp rx=0 tx=0).stop 2024-02-28T17:25:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 shutdown_connections 2024-02-28T17:25:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f751406d7f0 0x7f751406fca0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7530103dc0 0x7f753011a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 --2- 172.21.15.1:0/2328516947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75301095f0 0x7f753011a620 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.107+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 >> 172.21.15.1:0/2328516947 conn(0x7f7530072d50 msgr2=0x7f7530074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.108+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 shutdown_connections 2024-02-28T17:25:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.108+0000 7f7536200700 1 -- 172.21.15.1:0/2328516947 wait complete. 2024-02-28T17:25:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:02 smithi066 ceph-mon[85781]: from='client.36668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:02 smithi066 ceph-mon[85781]: pgmap v1521: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:02 smithi066 ceph-mon[85781]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:02 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1636177184' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:25:03.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:02 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2817454502' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:25:03.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.269+0000 7fa45aa00700 1 -- 172.21.15.1:0/4012630297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 msgr2=0x7fa454105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:03.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.269+0000 7fa45aa00700 1 --2- 172.21.15.1:0/4012630297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 0x7fa454105520 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fa444004940 tx=0x7fa444009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:03.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.269+0000 7fa45aa00700 1 -- 172.21.15.1:0/4012630297 shutdown_connections 2024-02-28T17:25:03.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.269+0000 7fa45aa00700 1 --2- 172.21.15.1:0/4012630297 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa454105af0 0x7fa4540ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.269+0000 7fa45aa00700 1 --2- 172.21.15.1:0/4012630297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 0x7fa454105520 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.269+0000 7fa45aa00700 1 -- 172.21.15.1:0/4012630297 >> 172.21.15.1:0/4012630297 conn(0x7fa4540face0 msgr2=0x7fa4540fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:03.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:02 smithi001 ceph-mon[112940]: from='client.36668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:03.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:02 smithi001 ceph-mon[112940]: pgmap v1521: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:03.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:02 smithi001 ceph-mon[112940]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:03.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:02 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1636177184' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:25:03.272 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:02 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2817454502' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:25:03.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.269+0000 7fa45aa00700 1 -- 172.21.15.1:0/4012630297 shutdown_connections 2024-02-28T17:25:03.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.269+0000 7fa45aa00700 1 -- 172.21.15.1:0/4012630297 wait complete. 2024-02-28T17:25:03.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.270+0000 7fa45aa00700 1 Processor -- start 2024-02-28T17:25:03.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.271+0000 7fa45aa00700 1 -- start start 2024-02-28T17:25:03.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.271+0000 7fa45aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 0x7fa454115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:03.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.271+0000 7fa45aa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa454105af0 0x7fa454116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:03.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.271+0000 7fa45aa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa454116850 con 0x7fa454105150 2024-02-28T17:25:03.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.271+0000 7fa45aa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45410fb60 con 0x7fa454105af0 2024-02-28T17:25:03.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.271+0000 7fa453e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 0x7fa454115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:03.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.271+0000 7fa453e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 0x7fa454115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33456/0 (socket says 172.21.15.1:33456) 2024-02-28T17:25:03.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.271+0000 7fa453e00700 1 -- 172.21.15.1:0/4086717704 learned_addr learned my addr 172.21.15.1:0/4086717704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:03.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.272+0000 7fa44be00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa454105af0 0x7fa454116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:03.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.272+0000 7fa453e00700 1 -- 172.21.15.1:0/4086717704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa454105af0 msgr2=0x7fa454116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:03.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.272+0000 7fa453e00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa454105af0 0x7fa454116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.272+0000 7fa453e00700 1 -- 172.21.15.1:0/4086717704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa444005040 con 0x7fa454105150 2024-02-28T17:25:03.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.272+0000 7fa453e00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 0x7fa454115cc0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fa444009fd0 tx=0x7fa444009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:03.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.273+0000 7fa450c00700 1 -- 172.21.15.1:0/4086717704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa44400e040 con 0x7fa454105150 2024-02-28T17:25:03.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.273+0000 7fa450c00700 1 -- 172.21.15.1:0/4086717704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa44400bbf0 con 0x7fa454105150 2024-02-28T17:25:03.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.273+0000 7fa450c00700 1 -- 172.21.15.1:0/4086717704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa44401c3f0 con 0x7fa454105150 2024-02-28T17:25:03.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.273+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45410fde0 con 0x7fa454105150 2024-02-28T17:25:03.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.273+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa454110320 con 0x7fa454105150 2024-02-28T17:25:03.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.274+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa454102460 con 0x7fa454105150 2024-02-28T17:25:03.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.278+0000 7fa450c00700 1 -- 172.21.15.1:0/4086717704 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa44400bd60 con 0x7fa454105150 2024-02-28T17:25:03.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.279+0000 7fa450c00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa43806d7f0 0x7fa43806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:03.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.279+0000 7fa450c00700 1 -- 172.21.15.1:0/4086717704 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa44408ed10 con 0x7fa454105150 2024-02-28T17:25:03.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.279+0000 7fa44be00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa43806d7f0 0x7fa43806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:03.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.280+0000 7fa44be00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa43806d7f0 0x7fa43806fca0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa454111560 tx=0x7fa43c009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:03.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.286+0000 7fa450c00700 1 -- 172.21.15.1:0/4086717704 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa44405c0f0 con 0x7fa454105150 2024-02-28T17:25:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.600+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa4540fe8e0 con 0x7fa454105150 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.601+0000 7fa450c00700 1 -- 172.21.15.1:0/4086717704 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fa44405bc80 con 0x7fa454105150 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:25:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:25:03.605 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:25:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:25:03.606 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:25:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:25:03.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa43806d7f0 msgr2=0x7fa43806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:03.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa43806d7f0 0x7fa43806fca0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa454111560 tx=0x7fa43c009ee0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 msgr2=0x7fa454115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 0x7fa454115cc0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fa444009fd0 tx=0x7fa444009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 shutdown_connections 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa43806d7f0 0x7fa43806fca0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454105150 0x7fa454115cc0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 --2- 172.21.15.1:0/4086717704 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa454105af0 0x7fa454116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 >> 172.21.15.1:0/4086717704 conn(0x7fa4540face0 msgr2=0x7fa4540fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 shutdown_connections 2024-02-28T17:25:03.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:03.605+0000 7fa45aa00700 1 -- 172.21.15.1:0/4086717704 wait complete. 2024-02-28T17:25:04.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:04 smithi066 ceph-mon[85781]: from='client.36684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:04.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:04 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4086717704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:25:04.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:04 smithi001 ceph-mon[112940]: from='client.36684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:04.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:04 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4086717704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:25:05.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:05 smithi066 ceph-mon[85781]: pgmap v1522: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:05.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:05 smithi001 ceph-mon[112940]: pgmap v1522: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:07.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:07 smithi001 ceph-mon[112940]: pgmap v1523: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:07.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:07 smithi066 ceph-mon[85781]: pgmap v1523: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:09.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:09 smithi001 ceph-mon[112940]: pgmap v1524: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:09.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:09 smithi066 ceph-mon[85781]: pgmap v1524: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:11 smithi001 ceph-mon[112940]: pgmap v1525: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:11.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:25:11.314 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:11 smithi066 ceph-mon[85781]: pgmap v1525: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:11.315 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:25:13.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:13 smithi066 ceph-mon[85781]: pgmap v1526: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:13.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:13 smithi001 ceph-mon[112940]: pgmap v1526: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:15.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:15 smithi066 ceph-mon[85781]: pgmap v1527: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:15.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:15 smithi001 ceph-mon[112940]: pgmap v1527: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:17.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:17 smithi066 ceph-mon[85781]: pgmap v1528: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:17.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:17 smithi001 ceph-mon[112940]: pgmap v1528: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:19.336 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:19 smithi001 ceph-mon[112940]: pgmap v1529: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:19.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:19 smithi066 ceph-mon[85781]: pgmap v1529: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:21.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:21 smithi066 ceph-mon[85781]: pgmap v1530: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:21.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:21 smithi001 ceph-mon[112940]: pgmap v1530: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:23.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:23 smithi066 ceph-mon[85781]: pgmap v1531: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:23.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:23 smithi001 ceph-mon[112940]: pgmap v1531: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:25.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:25 smithi066 ceph-mon[85781]: pgmap v1532: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:25.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:25 smithi001 ceph-mon[112940]: pgmap v1532: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:25:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:25:27.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:27 smithi066 ceph-mon[85781]: pgmap v1533: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:27.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:27 smithi001 ceph-mon[112940]: pgmap v1533: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:29.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:29 smithi066 ceph-mon[85781]: pgmap v1534: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:29.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:29 smithi001 ceph-mon[112940]: pgmap v1534: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:31.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:31 smithi066 ceph-mon[85781]: pgmap v1535: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:31.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:31 smithi001 ceph-mon[112940]: pgmap v1535: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:33.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:33 smithi066 ceph-mon[85781]: pgmap v1536: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:33.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:33 smithi001 ceph-mon[112940]: pgmap v1536: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.776+0000 7f72bac00700 1 -- 172.21.15.1:0/3289115846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 msgr2=0x7f72b4105530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.776+0000 7f72bac00700 1 --2- 172.21.15.1:0/3289115846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 0x7f72b4105530 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f72a4004940 tx=0x7f72a4009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.776+0000 7f72bac00700 1 -- 172.21.15.1:0/3289115846 shutdown_connections 2024-02-28T17:25:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.776+0000 7f72bac00700 1 --2- 172.21.15.1:0/3289115846 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72b4105b00 0x7f72b40ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.776+0000 7f72bac00700 1 --2- 172.21.15.1:0/3289115846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 0x7f72b4105530 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.776+0000 7f72bac00700 1 -- 172.21.15.1:0/3289115846 >> 172.21.15.1:0/3289115846 conn(0x7f72b40facb0 msgr2=0x7f72b40fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.776+0000 7f72bac00700 1 -- 172.21.15.1:0/3289115846 shutdown_connections 2024-02-28T17:25:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.776+0000 7f72bac00700 1 -- 172.21.15.1:0/3289115846 wait complete. 2024-02-28T17:25:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.777+0000 7f72bac00700 1 Processor -- start 2024-02-28T17:25:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.777+0000 7f72bac00700 1 -- start start 2024-02-28T17:25:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.777+0000 7f72bac00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 0x7f72b4115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.777+0000 7f72bac00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72b4105b00 0x7f72b41161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.777+0000 7f72bac00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41168c0 con 0x7f72b4105160 2024-02-28T17:25:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.777+0000 7f72bac00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b410fb40 con 0x7f72b4105b00 2024-02-28T17:25:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.778+0000 7f72b3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 0x7f72b4115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.778+0000 7f72b3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 0x7f72b4115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52700/0 (socket says 172.21.15.1:52700) 2024-02-28T17:25:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.778+0000 7f72b3e00700 1 -- 172.21.15.1:0/567756103 learned_addr learned my addr 172.21.15.1:0/567756103 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.778+0000 7f72b3400700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72b4105b00 0x7f72b41161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.778+0000 7f72b3e00700 1 -- 172.21.15.1:0/567756103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72b4105b00 msgr2=0x7f72b41161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.778+0000 7f72b3e00700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72b4105b00 0x7f72b41161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.778+0000 7f72b3e00700 1 -- 172.21.15.1:0/567756103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4005040 con 0x7f72b4105160 2024-02-28T17:25:33.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.778+0000 7f72b3e00700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 0x7f72b4115ca0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009150 tx=0x7f72a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:33.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.779+0000 7f72abe00700 1 -- 172.21.15.1:0/567756103 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72a400e040 con 0x7f72b4105160 2024-02-28T17:25:33.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.779+0000 7f72abe00700 1 -- 172.21.15.1:0/567756103 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72a400bbf0 con 0x7f72b4105160 2024-02-28T17:25:33.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.779+0000 7f72abe00700 1 -- 172.21.15.1:0/567756103 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72a401c630 con 0x7f72b4105160 2024-02-28T17:25:33.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.779+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b410fd60 con 0x7f72b4105160 2024-02-28T17:25:33.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.779+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b41102a0 con 0x7f72b4105160 2024-02-28T17:25:33.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.780+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b40fe8f0 con 0x7f72b4105160 2024-02-28T17:25:33.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.781+0000 7f72abe00700 1 -- 172.21.15.1:0/567756103 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f72a400bd60 con 0x7f72b4105160 2024-02-28T17:25:33.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.782+0000 7f72abe00700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f729806d7f0 0x7f729806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:33.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.782+0000 7f72abe00700 1 -- 172.21.15.1:0/567756103 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f72a408e290 con 0x7f72b4105160 2024-02-28T17:25:33.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.782+0000 7f72b3400700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f729806d7f0 0x7f729806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:33.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.783+0000 7f72b3400700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f729806d7f0 0x7f729806fca0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f72a0006fd0 tx=0x7f72a0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:33.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:33.787+0000 7f72abe00700 1 -- 172.21.15.1:0/567756103 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f72a405b670 con 0x7f72b4105160 2024-02-28T17:25:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.031+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f72b41105d0 con 0x7f729806d7f0 2024-02-28T17:25:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.033+0000 7f72abe00700 1 -- 172.21.15.1:0/567756103 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f72b41105d0 con 0x7f729806d7f0 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f729806d7f0 msgr2=0x7f729806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f729806d7f0 0x7f729806fca0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f72a0006fd0 tx=0x7f72a0008040 comp rx=0 tx=0).stop 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 msgr2=0x7f72b4115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 0x7f72b4115ca0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009150 tx=0x7f72a4009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 shutdown_connections 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f729806d7f0 0x7f729806fca0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72b4105160 0x7f72b4115ca0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 --2- 172.21.15.1:0/567756103 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f72b4105b00 0x7f72b41161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.036+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 >> 172.21.15.1:0/567756103 conn(0x7f72b40facb0 msgr2=0x7f72b40fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:34.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.037+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 shutdown_connections 2024-02-28T17:25:34.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.037+0000 7f72bac00700 1 -- 172.21.15.1:0/567756103 wait complete. 2024-02-28T17:25:34.056 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:25:34.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.199+0000 7f08e4a00700 1 -- 172.21.15.1:0/291888804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fe960 msgr2=0x7f08e0106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.199+0000 7f08e4a00700 1 --2- 172.21.15.1:0/291888804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fe960 0x7f08e0106fa0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f08cc004940 tx=0x7f08cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.199+0000 7f08e4a00700 1 -- 172.21.15.1:0/291888804 shutdown_connections 2024-02-28T17:25:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.199+0000 7f08e4a00700 1 --2- 172.21.15.1:0/291888804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fe960 0x7f08e0106fa0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.199+0000 7f08e4a00700 1 --2- 172.21.15.1:0/291888804 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08e00fdfc0 0x7f08e00fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.199+0000 7f08e4a00700 1 -- 172.21.15.1:0/291888804 >> 172.21.15.1:0/291888804 conn(0x7f08e00f7a10 msgr2=0x7f08e00f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.199+0000 7f08e4a00700 1 -- 172.21.15.1:0/291888804 shutdown_connections 2024-02-28T17:25:34.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.199+0000 7f08e4a00700 1 -- 172.21.15.1:0/291888804 wait complete. 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.200+0000 7f08e4a00700 1 Processor -- start 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.200+0000 7f08e4a00700 1 -- start start 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.200+0000 7f08e4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fdfc0 0x7f08e0115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.200+0000 7f08e4a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08e00fe960 0x7f08e0116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.200+0000 7f08e4a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e01167b0 con 0x7f08e00fdfc0 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.200+0000 7f08e4a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e010fb00 con 0x7f08e00fe960 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.201+0000 7f08dde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fdfc0 0x7f08e0115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.201+0000 7f08dde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fdfc0 0x7f08e0115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52714/0 (socket says 172.21.15.1:52714) 2024-02-28T17:25:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.201+0000 7f08dde00700 1 -- 172.21.15.1:0/457560892 learned_addr learned my addr 172.21.15.1:0/457560892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.201+0000 7f08dd400700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08e00fe960 0x7f08e0116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.201+0000 7f08dde00700 1 -- 172.21.15.1:0/457560892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08e00fe960 msgr2=0x7f08e0116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.201+0000 7f08dde00700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08e00fe960 0x7f08e0116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.201+0000 7f08dde00700 1 -- 172.21.15.1:0/457560892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc005040 con 0x7f08e00fdfc0 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.202+0000 7f08dde00700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fdfc0 0x7f08e0115c20 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f08d0002a50 tx=0x7f08d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.202+0000 7f08d6000700 1 -- 172.21.15.1:0/457560892 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08d0007cc0 con 0x7f08e00fdfc0 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.202+0000 7f08d6000700 1 -- 172.21.15.1:0/457560892 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08d000f040 con 0x7f08e00fdfc0 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.202+0000 7f08d6000700 1 -- 172.21.15.1:0/457560892 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08d0009640 con 0x7f08e00fdfc0 2024-02-28T17:25:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.202+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e010fde0 con 0x7f08e00fdfc0 2024-02-28T17:25:34.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.202+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e0110320 con 0x7f08e00fdfc0 2024-02-28T17:25:34.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.203+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e0106a10 con 0x7f08e00fdfc0 2024-02-28T17:25:34.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.204+0000 7f08d6000700 1 -- 172.21.15.1:0/457560892 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f08d00097a0 con 0x7f08e00fdfc0 2024-02-28T17:25:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.205+0000 7f08d6000700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08c406d7f0 0x7f08c406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.205+0000 7f08d6000700 1 -- 172.21.15.1:0/457560892 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f08d008cc60 con 0x7f08e00fdfc0 2024-02-28T17:25:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.205+0000 7f08dd400700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08c406d7f0 0x7f08c406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.206+0000 7f08dd400700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08c406d7f0 0x7f08c406fca0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f08e01114c0 tx=0x7f08cc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:34.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.210+0000 7f08d6000700 1 -- 172.21.15.1:0/457560892 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f08d0056490 con 0x7f08e00fdfc0 2024-02-28T17:25:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.457+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f08e004bed0 con 0x7f08c406d7f0 2024-02-28T17:25:34.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.460+0000 7f08d6000700 1 -- 172.21.15.1:0/457560892 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f08e004bed0 con 0x7f08c406d7f0 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08c406d7f0 msgr2=0x7f08c406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08c406d7f0 0x7f08c406fca0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f08e01114c0 tx=0x7f08cc009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fdfc0 msgr2=0x7f08e0115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fdfc0 0x7f08e0115c20 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f08d0002a50 tx=0x7f08d00046b0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 shutdown_connections 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f08c406d7f0 0x7f08c406fca0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08e00fdfc0 0x7f08e0115c20 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 --2- 172.21.15.1:0/457560892 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f08e00fe960 0x7f08e0116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.463+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 >> 172.21.15.1:0/457560892 conn(0x7f08e00f7a10 msgr2=0x7f08e00fc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:34.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.464+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 shutdown_connections 2024-02-28T17:25:34.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.464+0000 7f08e4a00700 1 -- 172.21.15.1:0/457560892 wait complete. 2024-02-28T17:25:34.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.625+0000 7f9732c00700 1 -- 172.21.15.1:0/2186857611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c1095f0 msgr2=0x7f972c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.625+0000 7f9732c00700 1 --2- 172.21.15.1:0/2186857611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c1095f0 0x7f972c1099c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f971c004940 tx=0x7f971c009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:34.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.625+0000 7f9732c00700 1 -- 172.21.15.1:0/2186857611 shutdown_connections 2024-02-28T17:25:34.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.625+0000 7f9732c00700 1 --2- 172.21.15.1:0/2186857611 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f972c103dc0 0x7f972c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.625+0000 7f9732c00700 1 --2- 172.21.15.1:0/2186857611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c1095f0 0x7f972c1099c0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.625+0000 7f9732c00700 1 -- 172.21.15.1:0/2186857611 >> 172.21.15.1:0/2186857611 conn(0x7f972c072d50 msgr2=0x7f972c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:34.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.625+0000 7f9732c00700 1 -- 172.21.15.1:0/2186857611 shutdown_connections 2024-02-28T17:25:34.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.625+0000 7f9732c00700 1 -- 172.21.15.1:0/2186857611 wait complete. 2024-02-28T17:25:34.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.626+0000 7f9732c00700 1 Processor -- start 2024-02-28T17:25:34.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.626+0000 7f9732c00700 1 -- start start 2024-02-28T17:25:34.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f9732c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c103dc0 0x7f972c11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:34.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f9732c00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f972c1095f0 0x7f972c11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:34.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f9732c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c11ac70 con 0x7f972c103dc0 2024-02-28T17:25:34.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f9732c00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c113f80 con 0x7f972c1095f0 2024-02-28T17:25:34.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f972be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c103dc0 0x7f972c11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:34.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f972be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c103dc0 0x7f972c11a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52734/0 (socket says 172.21.15.1:52734) 2024-02-28T17:25:34.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f972be00700 1 -- 172.21.15.1:0/902449536 learned_addr learned my addr 172.21.15.1:0/902449536 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:34.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f9723e00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f972c1095f0 0x7f972c11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:34.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f972be00700 1 -- 172.21.15.1:0/902449536 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f972c1095f0 msgr2=0x7f972c11a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.627+0000 7f972be00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f972c1095f0 0x7f972c11a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.628+0000 7f972be00700 1 -- 172.21.15.1:0/902449536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c005040 con 0x7f972c103dc0 2024-02-28T17:25:34.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.628+0000 7f972be00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c103dc0 0x7f972c11a0e0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f971c00b3b0 tx=0x7f971c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:34.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.628+0000 7f9728c00700 1 -- 172.21.15.1:0/902449536 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f971c00e040 con 0x7f972c103dc0 2024-02-28T17:25:34.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.628+0000 7f9728c00700 1 -- 172.21.15.1:0/902449536 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f971c00bbf0 con 0x7f972c103dc0 2024-02-28T17:25:34.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.628+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c114200 con 0x7f972c103dc0 2024-02-28T17:25:34.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.628+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c114740 con 0x7f972c103dc0 2024-02-28T17:25:34.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.628+0000 7f9728c00700 1 -- 172.21.15.1:0/902449536 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f971c01c630 con 0x7f972c103dc0 2024-02-28T17:25:34.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.629+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c106850 con 0x7f972c103dc0 2024-02-28T17:25:34.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.630+0000 7f9728c00700 1 -- 172.21.15.1:0/902449536 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f971c00bd60 con 0x7f972c103dc0 2024-02-28T17:25:34.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.631+0000 7f9728c00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f971006d7e0 0x7f971006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:34.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.631+0000 7f9728c00700 1 -- 172.21.15.1:0/902449536 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f971c08e360 con 0x7f972c103dc0 2024-02-28T17:25:34.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.632+0000 7f9723e00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f971006d7e0 0x7f971006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:34.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.632+0000 7f9723e00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f971006d7e0 0x7f971006fc90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f972c115980 tx=0x7f971400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:34.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.636+0000 7f9728c00700 1 -- 172.21.15.1:0/902449536 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f971c05b6c0 con 0x7f972c103dc0 2024-02-28T17:25:34.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.898+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f972c04bed0 con 0x7f971006d7e0 2024-02-28T17:25:34.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.916+0000 7f9728c00700 1 -- 172.21.15.1:0/902449536 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f972c04bed0 con 0x7f971006d7e0 2024-02-28T17:25:34.920 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:25:34.920 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (51m) 13m ago 65m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (48m) 13m ago 66m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (48m) 13m ago 64m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (51m) 13m ago 65m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (44m) 13m ago 58m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (43m) 13m ago 58m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (13m) 13m ago 58m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (28m) 13m ago 58m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (54m) 13m ago 67m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (53m) 13m ago 64m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (50m) 13m ago 67m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (49m) 13m ago 64m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (52m) 13m ago 66m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:25:34.921 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (52m) 13m ago 64m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:25:34.922 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (47m) 13m ago 63m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:25:34.922 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (47m) 13m ago 63m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:25:34.922 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (46m) 13m ago 62m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:25:34.922 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (46m) 13m ago 61m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:25:34.922 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (45m) 13m ago 61m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:25:34.922 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (44m) 13m ago 60m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:25:34.922 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (52m) 13m ago 65m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:25:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f971006d7e0 msgr2=0x7f971006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f971006d7e0 0x7f971006fc90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f972c115980 tx=0x7f971400b040 comp rx=0 tx=0).stop 2024-02-28T17:25:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c103dc0 msgr2=0x7f972c11a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c103dc0 0x7f972c11a0e0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f971c00b3b0 tx=0x7f971c009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 shutdown_connections 2024-02-28T17:25:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f971006d7e0 0x7f971006fc90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f972c103dc0 0x7f972c11a0e0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 --2- 172.21.15.1:0/902449536 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f972c1095f0 0x7f972c11a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.920+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 >> 172.21.15.1:0/902449536 conn(0x7f972c072d50 msgr2=0x7f972c074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.921+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 shutdown_connections 2024-02-28T17:25:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:34.921+0000 7f9732c00700 1 -- 172.21.15.1:0/902449536 wait complete. 2024-02-28T17:25:35.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.082+0000 7ff6b5800700 1 -- 172.21.15.1:0/3092747844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 msgr2=0x7ff6b00ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:35.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.082+0000 7ff6b5800700 1 --2- 172.21.15.1:0/3092747844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 0x7ff6b00ff180 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff69c004940 tx=0x7ff69c009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:35.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.082+0000 7ff6b5800700 1 -- 172.21.15.1:0/3092747844 shutdown_connections 2024-02-28T17:25:35.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.082+0000 7ff6b5800700 1 --2- 172.21.15.1:0/3092747844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 0x7ff6b00ff180 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.082+0000 7ff6b5800700 1 --2- 172.21.15.1:0/3092747844 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6b0105170 0x7ff6b0105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.082+0000 7ff6b5800700 1 -- 172.21.15.1:0/3092747844 >> 172.21.15.1:0/3092747844 conn(0x7ff6b00face0 msgr2=0x7ff6b00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:35.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.082+0000 7ff6b5800700 1 -- 172.21.15.1:0/3092747844 shutdown_connections 2024-02-28T17:25:35.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.082+0000 7ff6b5800700 1 -- 172.21.15.1:0/3092747844 wait complete. 2024-02-28T17:25:35.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.083+0000 7ff6b5800700 1 Processor -- start 2024-02-28T17:25:35.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.083+0000 7ff6b5800700 1 -- start start 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6b5800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6b0105170 0x7ff6b0115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6b5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 0x7ff6b0116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6b5800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0116870 con 0x7ff6b0105b10 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6b5800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b010fb80 con 0x7ff6b0105170 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6ae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 0x7ff6b0116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6ae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 0x7ff6b0116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52744/0 (socket says 172.21.15.1:52744) 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6ae000700 1 -- 172.21.15.1:0/862389003 learned_addr learned my addr 172.21.15.1:0/862389003 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6aea00700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6b0105170 0x7ff6b0115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6ae000700 1 -- 172.21.15.1:0/862389003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6b0105170 msgr2=0x7ff6b0115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.084+0000 7ff6ae000700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6b0105170 0x7ff6b0115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.085+0000 7ff6ae000700 1 -- 172.21.15.1:0/862389003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c005040 con 0x7ff6b0105b10 2024-02-28T17:25:35.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.085+0000 7ff6ae000700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 0x7ff6b0116220 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009030 tx=0x7ff69c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:35.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.085+0000 7ff6a6a00700 1 -- 172.21.15.1:0/862389003 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff69c00e040 con 0x7ff6b0105b10 2024-02-28T17:25:35.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.085+0000 7ff6a6a00700 1 -- 172.21.15.1:0/862389003 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff69c00bbf0 con 0x7ff6b0105b10 2024-02-28T17:25:35.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.085+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b010fe00 con 0x7ff6b0105b10 2024-02-28T17:25:35.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.085+0000 7ff6a6a00700 1 -- 172.21.15.1:0/862389003 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff69c01c630 con 0x7ff6b0105b10 2024-02-28T17:25:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.085+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b0110340 con 0x7ff6b0105b10 2024-02-28T17:25:35.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.087+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b0107640 con 0x7ff6b0105b10 2024-02-28T17:25:35.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.088+0000 7ff6a6a00700 1 -- 172.21.15.1:0/862389003 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff69c00bd60 con 0x7ff6b0105b10 2024-02-28T17:25:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.089+0000 7ff6a6a00700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff69406d7f0 0x7ff69406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.089+0000 7ff6a6a00700 1 -- 172.21.15.1:0/862389003 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff69c08e2c0 con 0x7ff6b0105b10 2024-02-28T17:25:35.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.089+0000 7ff6aea00700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff69406d7f0 0x7ff69406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:35.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.090+0000 7ff6aea00700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff69406d7f0 0x7ff69406fca0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0004d60 tx=0x7ff6a000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:35.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.094+0000 7ff6a6a00700 1 -- 172.21.15.1:0/862389003 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff69c05b620 con 0x7ff6b0105b10 2024-02-28T17:25:35.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:35 smithi001 ceph-mon[112940]: pgmap v1537: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:35.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:35 smithi001 ceph-mon[112940]: from='client.36692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:35.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.412+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff6b00fe900 con 0x7ff6b0105b10 2024-02-28T17:25:35.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.413+0000 7ff6a6a00700 1 -- 172.21.15.1:0/862389003 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff69c05b1b0 con 0x7ff6b0105b10 2024-02-28T17:25:35.416 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:25:35.417 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:25:35.418 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:25:35.418 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:25:35.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff69406d7f0 msgr2=0x7ff69406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:35.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff69406d7f0 0x7ff69406fca0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0004d60 tx=0x7ff6a000b040 comp rx=0 tx=0).stop 2024-02-28T17:25:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 msgr2=0x7ff6b0116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 0x7ff6b0116220 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009030 tx=0x7ff69c009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 shutdown_connections 2024-02-28T17:25:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff69406d7f0 0x7ff69406fca0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff6b0105170 0x7ff6b0115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 --2- 172.21.15.1:0/862389003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b0105b10 0x7ff6b0116220 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.416+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 >> 172.21.15.1:0/862389003 conn(0x7ff6b00face0 msgr2=0x7ff6b00fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.417+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 shutdown_connections 2024-02-28T17:25:35.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.417+0000 7ff6b5800700 1 -- 172.21.15.1:0/862389003 wait complete. 2024-02-28T17:25:35.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:35 smithi066 ceph-mon[85781]: pgmap v1537: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:35.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:35 smithi066 ceph-mon[85781]: from='client.36692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.574+0000 7fcd74e00700 1 -- 172.21.15.1:0/2662869708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 msgr2=0x7fcd700fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.574+0000 7fcd74e00700 1 --2- 172.21.15.1:0/2662869708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 0x7fcd700fe310 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fcd60004940 tx=0x7fcd60009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.574+0000 7fcd74e00700 1 -- 172.21.15.1:0/2662869708 shutdown_connections 2024-02-28T17:25:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.574+0000 7fcd74e00700 1 --2- 172.21.15.1:0/2662869708 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd700fe8e0 0x7fcd70106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.574+0000 7fcd74e00700 1 --2- 172.21.15.1:0/2662869708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 0x7fcd700fe310 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.574+0000 7fcd74e00700 1 -- 172.21.15.1:0/2662869708 >> 172.21.15.1:0/2662869708 conn(0x7fcd700f79d0 msgr2=0x7fcd700f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.574+0000 7fcd74e00700 1 -- 172.21.15.1:0/2662869708 shutdown_connections 2024-02-28T17:25:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.574+0000 7fcd74e00700 1 -- 172.21.15.1:0/2662869708 wait complete. 2024-02-28T17:25:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.575+0000 7fcd74e00700 1 Processor -- start 2024-02-28T17:25:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.576+0000 7fcd74e00700 1 -- start start 2024-02-28T17:25:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.576+0000 7fcd74e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 0x7fcd70115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.576+0000 7fcd74e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd700fe8e0 0x7fcd70116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.576+0000 7fcd74e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd701167b0 con 0x7fcd700fdf40 2024-02-28T17:25:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.576+0000 7fcd74e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7010fae0 con 0x7fcd700fe8e0 2024-02-28T17:25:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.577+0000 7fcd6de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 0x7fcd70115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.577+0000 7fcd6de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 0x7fcd70115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52774/0 (socket says 172.21.15.1:52774) 2024-02-28T17:25:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.577+0000 7fcd6de00700 1 -- 172.21.15.1:0/2454793111 learned_addr learned my addr 172.21.15.1:0/2454793111 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:35.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.577+0000 7fcd6d400700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd700fe8e0 0x7fcd70116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:35.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.577+0000 7fcd6de00700 1 -- 172.21.15.1:0/2454793111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd700fe8e0 msgr2=0x7fcd70116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:35.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.577+0000 7fcd6de00700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd700fe8e0 0x7fcd70116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.577+0000 7fcd6de00700 1 -- 172.21.15.1:0/2454793111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd60005040 con 0x7fcd700fdf40 2024-02-28T17:25:35.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.577+0000 7fcd6d400700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd700fe8e0 0x7fcd70116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:25:35.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.578+0000 7fcd6de00700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 0x7fcd70115c20 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009180 tx=0x7fcd60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:35.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.578+0000 7fcd66000700 1 -- 172.21.15.1:0/2454793111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd6000e040 con 0x7fcd700fdf40 2024-02-28T17:25:35.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.578+0000 7fcd66000700 1 -- 172.21.15.1:0/2454793111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd60004290 con 0x7fcd700fdf40 2024-02-28T17:25:35.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.578+0000 7fcd66000700 1 -- 172.21.15.1:0/2454793111 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd6001c630 con 0x7fcd700fdf40 2024-02-28T17:25:35.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.578+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7010fd60 con 0x7fcd700fdf40 2024-02-28T17:25:35.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.578+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd701102a0 con 0x7fcd700fdf40 2024-02-28T17:25:35.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.579+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd70106990 con 0x7fcd700fdf40 2024-02-28T17:25:35.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.580+0000 7fcd66000700 1 -- 172.21.15.1:0/2454793111 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcd600139f0 con 0x7fcd700fdf40 2024-02-28T17:25:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.581+0000 7fcd66000700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd5406d7f0 0x7fcd5406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.582+0000 7fcd66000700 1 -- 172.21.15.1:0/2454793111 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcd6008e2f0 con 0x7fcd700fdf40 2024-02-28T17:25:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.582+0000 7fcd6d400700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd5406d7f0 0x7fcd5406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.583+0000 7fcd6d400700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd5406d7f0 0x7fcd5406fca0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c007fd0 tx=0x7fcd5c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:35.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.586+0000 7fcd66000700 1 -- 172.21.15.1:0/2454793111 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcd6005b6d0 con 0x7fcd700fdf40 2024-02-28T17:25:35.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.860+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcd7004bf20 con 0x7fcd700fdf40 2024-02-28T17:25:35.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.861+0000 7fcd66000700 1 -- 172.21.15.1:0/2454793111 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fcd6005b260 con 0x7fcd700fdf40 2024-02-28T17:25:35.865 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:25:35.865 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:25:35.865 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:25:35.865 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:25:35.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:25:35.865 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:25:35.866 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:25:35.867 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:25:35.868 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:25:35.868 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:25:35.868 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:25:35.868 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:25:35.868 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:25:35.868 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd5406d7f0 msgr2=0x7fcd5406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd5406d7f0 0x7fcd5406fca0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c007fd0 tx=0x7fcd5c009040 comp rx=0 tx=0).stop 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 msgr2=0x7fcd70115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 0x7fcd70115c20 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009180 tx=0x7fcd60009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 shutdown_connections 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fcd5406d7f0 0x7fcd5406fca0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd700fdf40 0x7fcd70115c20 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 --2- 172.21.15.1:0/2454793111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fcd700fe8e0 0x7fcd70116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 >> 172.21.15.1:0/2454793111 conn(0x7fcd700f79d0 msgr2=0x7fcd700fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.864+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 shutdown_connections 2024-02-28T17:25:35.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:35.865+0000 7fcd74e00700 1 -- 172.21.15.1:0/2454793111 wait complete. 2024-02-28T17:25:35.870 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:25:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.022+0000 7f0bab000700 1 -- 172.21.15.1:0/1299208466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 msgr2=0x7f0ba4109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.022+0000 7f0bab000700 1 --2- 172.21.15.1:0/1299208466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 0x7f0ba4109260 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f0b90004970 tx=0x7f0b90009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.023+0000 7f0bab000700 1 -- 172.21.15.1:0/1299208466 shutdown_connections 2024-02-28T17:25:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.023+0000 7f0bab000700 1 --2- 172.21.15.1:0/1299208466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 0x7f0ba4109260 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.023+0000 7f0bab000700 1 --2- 172.21.15.1:0/1299208466 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ba4070330 0x7f0ba4070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.023+0000 7f0bab000700 1 -- 172.21.15.1:0/1299208466 >> 172.21.15.1:0/1299208466 conn(0x7f0ba40f8ae0 msgr2=0x7f0ba40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.023+0000 7f0bab000700 1 -- 172.21.15.1:0/1299208466 shutdown_connections 2024-02-28T17:25:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.023+0000 7f0bab000700 1 -- 172.21.15.1:0/1299208466 wait complete. 2024-02-28T17:25:36.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.024+0000 7f0bab000700 1 Processor -- start 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.024+0000 7f0bab000700 1 -- start start 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.024+0000 7f0bab000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ba4070330 0x7f0ba4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.024+0000 7f0bab000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 0x7f0ba4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.024+0000 7f0bab000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba4116790 con 0x7f0ba4070c40 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.024+0000 7f0bab000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba410fae0 con 0x7f0ba4070330 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.025+0000 7f0ba3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 0x7f0ba4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.025+0000 7f0ba3400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 0x7f0ba4116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52790/0 (socket says 172.21.15.1:52790) 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.025+0000 7f0ba3400700 1 -- 172.21.15.1:0/464640384 learned_addr learned my addr 172.21.15.1:0/464640384 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.025+0000 7f0ba3e00700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ba4070330 0x7f0ba4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.025+0000 7f0ba3400700 1 -- 172.21.15.1:0/464640384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ba4070330 msgr2=0x7f0ba4115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.025+0000 7f0ba3400700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ba4070330 0x7f0ba4115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.025+0000 7f0ba3400700 1 -- 172.21.15.1:0/464640384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b90005040 con 0x7f0ba4070c40 2024-02-28T17:25:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.026+0000 7f0ba3400700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 0x7f0ba4116140 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f0b90004f50 tx=0x7f0b90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.026+0000 7f0b9be00700 1 -- 172.21.15.1:0/464640384 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b9000e040 con 0x7f0ba4070c40 2024-02-28T17:25:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.026+0000 7f0b9be00700 1 -- 172.21.15.1:0/464640384 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b90004290 con 0x7f0ba4070c40 2024-02-28T17:25:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.026+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba410fd60 con 0x7f0ba4070c40 2024-02-28T17:25:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.026+0000 7f0b9be00700 1 -- 172.21.15.1:0/464640384 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b9001c630 con 0x7f0ba4070c40 2024-02-28T17:25:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.026+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba41102a0 con 0x7f0ba4070c40 2024-02-28T17:25:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.027+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba4106920 con 0x7f0ba4070c40 2024-02-28T17:25:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.028+0000 7f0b9be00700 1 -- 172.21.15.1:0/464640384 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f0b9000bc10 con 0x7f0ba4070c40 2024-02-28T17:25:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.029+0000 7f0b9be00700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b8806d7f0 0x7f0b8806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.029+0000 7f0b9be00700 1 -- 172.21.15.1:0/464640384 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f0b9008e280 con 0x7f0ba4070c40 2024-02-28T17:25:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.030+0000 7f0ba3e00700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b8806d7f0 0x7f0b8806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.030+0000 7f0ba3e00700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b8806d7f0 0x7f0b8806fca0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0b94004d60 tx=0x7f0b9400b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:36.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.034+0000 7f0b9be00700 1 -- 172.21.15.1:0/464640384 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0b90057b30 con 0x7f0ba4070c40 2024-02-28T17:25:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.274+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ba404bed0 con 0x7f0b8806d7f0 2024-02-28T17:25:36.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.277+0000 7f0b9be00700 1 -- 172.21.15.1:0/464640384 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0ba404bed0 con 0x7f0b8806d7f0 2024-02-28T17:25:36.281 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:25:36.281 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:25:36.281 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:25:36.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b8806d7f0 msgr2=0x7f0b8806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:36.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b8806d7f0 0x7f0b8806fca0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0b94004d60 tx=0x7f0b9400b040 comp rx=0 tx=0).stop 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 msgr2=0x7f0ba4116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 0x7f0ba4116140 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f0b90004f50 tx=0x7f0b90009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 shutdown_connections 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f0b8806d7f0 0x7f0b8806fca0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f0ba4070330 0x7f0ba4115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 --2- 172.21.15.1:0/464640384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba4070c40 0x7f0ba4116140 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 >> 172.21.15.1:0/464640384 conn(0x7f0ba40f8ae0 msgr2=0x7f0ba4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 shutdown_connections 2024-02-28T17:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.280+0000 7f0bab000700 1 -- 172.21.15.1:0/464640384 wait complete. 2024-02-28T17:25:36.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.434+0000 7fbaa2000700 1 -- 172.21.15.1:0/1699126549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 msgr2=0x7fba9c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:36.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.434+0000 7fbaa2000700 1 --2- 172.21.15.1:0/1699126549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 0x7fba9c103320 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fba8c004940 tx=0x7fba8c009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:36.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.435+0000 7fbaa2000700 1 -- 172.21.15.1:0/1699126549 shutdown_connections 2024-02-28T17:25:36.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.435+0000 7fbaa2000700 1 --2- 172.21.15.1:0/1699126549 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba9c1038f0 0x7fba9c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.435+0000 7fbaa2000700 1 --2- 172.21.15.1:0/1699126549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 0x7fba9c103320 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.435+0000 7fbaa2000700 1 -- 172.21.15.1:0/1699126549 >> 172.21.15.1:0/1699126549 conn(0x7fba9c0f8ae0 msgr2=0x7fba9c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:36.439 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:36 smithi001 ceph-mon[112940]: from='client.36696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:36.439 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:36 smithi001 ceph-mon[112940]: from='client.36700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:36.439 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/862389003' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:25:36.439 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2454793111' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:25:36.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.435+0000 7fbaa2000700 1 -- 172.21.15.1:0/1699126549 shutdown_connections 2024-02-28T17:25:36.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.435+0000 7fbaa2000700 1 -- 172.21.15.1:0/1699126549 wait complete. 2024-02-28T17:25:36.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.436+0000 7fbaa2000700 1 Processor -- start 2024-02-28T17:25:36.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.436+0000 7fbaa2000700 1 -- start start 2024-02-28T17:25:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.437+0000 7fbaa2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 0x7fba9c115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.437+0000 7fbaa2000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba9c1038f0 0x7fba9c116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.437+0000 7fbaa2000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c1167b0 con 0x7fba9c102f50 2024-02-28T17:25:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.437+0000 7fbaa2000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c10fb00 con 0x7fba9c1038f0 2024-02-28T17:25:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.437+0000 7fba9b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 0x7fba9c115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:36.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.437+0000 7fba9b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 0x7fba9c115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52798/0 (socket says 172.21.15.1:52798) 2024-02-28T17:25:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.437+0000 7fba9b400700 1 -- 172.21.15.1:0/516392401 learned_addr learned my addr 172.21.15.1:0/516392401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:25:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.438+0000 7fba93e00700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba9c1038f0 0x7fba9c116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.438+0000 7fba9b400700 1 -- 172.21.15.1:0/516392401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba9c1038f0 msgr2=0x7fba9c116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.438+0000 7fba9b400700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba9c1038f0 0x7fba9c116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.438+0000 7fba9b400700 1 -- 172.21.15.1:0/516392401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c005040 con 0x7fba9c102f50 2024-02-28T17:25:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.438+0000 7fba9b400700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 0x7fba9c115c20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00b3b0 tx=0x7fba8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.439+0000 7fba93400700 1 -- 172.21.15.1:0/516392401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8c00e040 con 0x7fba9c102f50 2024-02-28T17:25:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.439+0000 7fba93400700 1 -- 172.21.15.1:0/516392401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba8c00bbf0 con 0x7fba9c102f50 2024-02-28T17:25:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.439+0000 7fba93400700 1 -- 172.21.15.1:0/516392401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8c01c630 con 0x7fba9c102f50 2024-02-28T17:25:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.439+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c10fd80 con 0x7fba9c102f50 2024-02-28T17:25:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.439+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c1102c0 con 0x7fba9c102f50 2024-02-28T17:25:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.440+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c0fc6e0 con 0x7fba9c102f50 2024-02-28T17:25:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.441+0000 7fba93400700 1 -- 172.21.15.1:0/516392401 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fba8c00bd60 con 0x7fba9c102f50 2024-02-28T17:25:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.442+0000 7fba93400700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8006d7e0 0x7fba8006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:25:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.442+0000 7fba93400700 1 -- 172.21.15.1:0/516392401 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fba8c08e260 con 0x7fba9c102f50 2024-02-28T17:25:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.442+0000 7fba93e00700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8006d7e0 0x7fba8006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:25:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.443+0000 7fba93e00700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8006d7e0 0x7fba8006fc90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fba9c1114c0 tx=0x7fba84009ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:25:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.447+0000 7fba93400700 1 -- 172.21.15.1:0/516392401 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba8c057b10 con 0x7fba9c102f50 2024-02-28T17:25:36.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:36 smithi066 ceph-mon[85781]: from='client.36696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:36.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:36 smithi066 ceph-mon[85781]: from='client.36700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:36.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/862389003' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:25:36.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2454793111' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:25:36.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.745+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fba9c111200 con 0x7fba9c102f50 2024-02-28T17:25:36.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.746+0000 7fba93400700 1 -- 172.21.15.1:0/516392401 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fba8c05b1d0 con 0x7fba9c102f50 2024-02-28T17:25:36.749 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:25:36.749 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:25:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:25:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:25:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:25:36.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8006d7e0 msgr2=0x7fba8006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:36.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8006d7e0 0x7fba8006fc90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fba9c1114c0 tx=0x7fba84009ee0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 msgr2=0x7fba9c115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:25:36.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 0x7fba9c115c20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00b3b0 tx=0x7fba8c009e30 comp rx=0 tx=0).stop 2024-02-28T17:25:36.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 shutdown_connections 2024-02-28T17:25:36.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fba8006d7e0 0x7fba8006fc90 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba9c102f50 0x7fba9c115c20 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 --2- 172.21.15.1:0/516392401 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fba9c1038f0 0x7fba9c116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:25:36.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.749+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 >> 172.21.15.1:0/516392401 conn(0x7fba9c0f8ae0 msgr2=0x7fba9c0fa3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:25:36.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.750+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 shutdown_connections 2024-02-28T17:25:36.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:25:36.750+0000 7fbaa2000700 1 -- 172.21.15.1:0/516392401 wait complete. 2024-02-28T17:25:37.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:37 smithi066 ceph-mon[85781]: pgmap v1538: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:37.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:37 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/516392401' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:25:37.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:37 smithi001 ceph-mon[112940]: pgmap v1538: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:37.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:37 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/516392401' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:25:38.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:38 smithi066 ceph-mon[85781]: from='client.36712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:38.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:38 smithi001 ceph-mon[112940]: from='client.36712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:25:39.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:39 smithi066 ceph-mon[85781]: pgmap v1539: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:39.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:39 smithi001 ceph-mon[112940]: pgmap v1539: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:40.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:40 smithi066 ceph-mon[85781]: pgmap v1540: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:40.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:40 smithi001 ceph-mon[112940]: pgmap v1540: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:41.413 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:25:41.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:25:43.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:42 smithi066 ceph-mon[85781]: pgmap v1541: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:43.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:42 smithi001 ceph-mon[112940]: pgmap v1541: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:45.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:44 smithi066 ceph-mon[85781]: pgmap v1542: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:45.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:44 smithi001 ceph-mon[112940]: pgmap v1542: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:47.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:46 smithi066 ceph-mon[85781]: pgmap v1543: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:47.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:46 smithi001 ceph-mon[112940]: pgmap v1543: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:49.086 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:48 smithi001 ceph-mon[112940]: pgmap v1544: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:49.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:48 smithi066 ceph-mon[85781]: pgmap v1544: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:51.264 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:50 smithi066 ceph-mon[85781]: pgmap v1545: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:51.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:50 smithi001 ceph-mon[112940]: pgmap v1545: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:53.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:52 smithi066 ceph-mon[85781]: pgmap v1546: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:53.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:52 smithi001 ceph-mon[112940]: pgmap v1546: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:55.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:54 smithi066 ceph-mon[85781]: pgmap v1547: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:55.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:54 smithi001 ceph-mon[112940]: pgmap v1547: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:56.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:25:56.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:25:57.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:56 smithi066 ceph-mon[85781]: pgmap v1548: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:57.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:56 smithi001 ceph-mon[112940]: pgmap v1548: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:59.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:25:58 smithi066 ceph-mon[85781]: pgmap v1549: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:25:59.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:25:58 smithi001 ceph-mon[112940]: pgmap v1549: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:01.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:00 smithi066 ceph-mon[85781]: pgmap v1550: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:01.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:00 smithi001 ceph-mon[112940]: pgmap v1550: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:03.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:02 smithi066 ceph-mon[85781]: pgmap v1551: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:03.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:02 smithi001 ceph-mon[112940]: pgmap v1551: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:05.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:04 smithi066 ceph-mon[85781]: pgmap v1552: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:05.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:04 smithi001 ceph-mon[112940]: pgmap v1552: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:06.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.921+0000 7facefa00700 1 -- 172.21.15.1:0/101446396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 msgr2=0x7face8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:06.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.921+0000 7facefa00700 1 --2- 172.21.15.1:0/101446396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 0x7face8104210 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7facd4004940 tx=0x7facd4009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:06.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.921+0000 7facefa00700 1 -- 172.21.15.1:0/101446396 shutdown_connections 2024-02-28T17:26:06.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.921+0000 7facefa00700 1 --2- 172.21.15.1:0/101446396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 0x7face8104210 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:06.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.921+0000 7facefa00700 1 --2- 172.21.15.1:0/101446396 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7face81095f0 0x7face81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:06.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.921+0000 7facefa00700 1 -- 172.21.15.1:0/101446396 >> 172.21.15.1:0/101446396 conn(0x7face8072d50 msgr2=0x7face8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:06.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.921+0000 7facefa00700 1 -- 172.21.15.1:0/101446396 shutdown_connections 2024-02-28T17:26:06.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.921+0000 7facefa00700 1 -- 172.21.15.1:0/101446396 wait complete. 2024-02-28T17:26:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.922+0000 7facefa00700 1 Processor -- start 2024-02-28T17:26:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.922+0000 7facefa00700 1 -- start start 2024-02-28T17:26:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.922+0000 7facefa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 0x7face811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.922+0000 7facefa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7face81095f0 0x7face811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.922+0000 7facefa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face811abd0 con 0x7face8103dc0 2024-02-28T17:26:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.922+0000 7facefa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face8113ee0 con 0x7face81095f0 2024-02-28T17:26:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.923+0000 7faced000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 0x7face811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.923+0000 7faced000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 0x7face811a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41584/0 (socket says 172.21.15.1:41584) 2024-02-28T17:26:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.923+0000 7faced000700 1 -- 172.21.15.1:0/480809326 learned_addr learned my addr 172.21.15.1:0/480809326 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.923+0000 7face7e00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7face81095f0 0x7face811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.923+0000 7faced000700 1 -- 172.21.15.1:0/480809326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7face81095f0 msgr2=0x7face811a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:06.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.923+0000 7faced000700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7face81095f0 0x7face811a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:06.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.923+0000 7faced000700 1 -- 172.21.15.1:0/480809326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4005040 con 0x7face8103dc0 2024-02-28T17:26:06.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.924+0000 7faced000700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 0x7face811a040 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c830 tx=0x7facdc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:06.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.924+0000 7face4c00700 1 -- 172.21.15.1:0/480809326 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facdc00a760 con 0x7face8103dc0 2024-02-28T17:26:06.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.924+0000 7face4c00700 1 -- 172.21.15.1:0/480809326 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facdc00ad90 con 0x7face8103dc0 2024-02-28T17:26:06.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.924+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face81141c0 con 0x7face8103dc0 2024-02-28T17:26:06.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.924+0000 7face4c00700 1 -- 172.21.15.1:0/480809326 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facdc0113a0 con 0x7face8103dc0 2024-02-28T17:26:06.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.924+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face8114700 con 0x7face8103dc0 2024-02-28T17:26:06.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.925+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face810b9c0 con 0x7face8103dc0 2024-02-28T17:26:06.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.926+0000 7face4c00700 1 -- 172.21.15.1:0/480809326 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7facdc012400 con 0x7face8103dc0 2024-02-28T17:26:06.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.927+0000 7face4c00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7e0 0x7facd006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:06.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.927+0000 7face4c00700 1 -- 172.21.15.1:0/480809326 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7facdc08cfa0 con 0x7face8103dc0 2024-02-28T17:26:06.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.928+0000 7face7e00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7e0 0x7facd006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:06.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.928+0000 7face7e00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7e0 0x7facd006fc90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7facd4004f90 tx=0x7facd4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:06.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:06.933+0000 7face4c00700 1 -- 172.21.15.1:0/480809326 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7facdc05a300 con 0x7face8103dc0 2024-02-28T17:26:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.182+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7face8115620 con 0x7facd006d7e0 2024-02-28T17:26:07.185 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:06 smithi001 ceph-mon[112940]: pgmap v1553: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.184+0000 7face4c00700 1 -- 172.21.15.1:0/480809326 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7face8115620 con 0x7facd006d7e0 2024-02-28T17:26:07.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7e0 msgr2=0x7facd006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:07.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7e0 0x7facd006fc90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7facd4004f90 tx=0x7facd4009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:07.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 msgr2=0x7face811a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:07.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 0x7face811a040 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c830 tx=0x7facdc004470 comp rx=0 tx=0).stop 2024-02-28T17:26:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 shutdown_connections 2024-02-28T17:26:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7facd006d7e0 0x7facd006fc90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7face8103dc0 0x7face811a040 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 --2- 172.21.15.1:0/480809326 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7face81095f0 0x7face811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.187+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 >> 172.21.15.1:0/480809326 conn(0x7face8072d50 msgr2=0x7face8074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.188+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 shutdown_connections 2024-02-28T17:26:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.188+0000 7facefa00700 1 -- 172.21.15.1:0/480809326 wait complete. 2024-02-28T17:26:07.206 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:26:07.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:06 smithi066 ceph-mon[85781]: pgmap v1553: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:07.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.351+0000 7f25ff800700 1 -- 172.21.15.1:0/3859134689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f81038f0 msgr2=0x7f25f80fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:07.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.351+0000 7f25ff800700 1 --2- 172.21.15.1:0/3859134689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f81038f0 0x7f25f80fcf60 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f25e4004940 tx=0x7f25e4009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:07.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.351+0000 7f25ff800700 1 -- 172.21.15.1:0/3859134689 shutdown_connections 2024-02-28T17:26:07.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.351+0000 7f25ff800700 1 --2- 172.21.15.1:0/3859134689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f81038f0 0x7f25f80fcf60 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.351+0000 7f25ff800700 1 --2- 172.21.15.1:0/3859134689 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25f8102f50 0x7f25f8103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.351+0000 7f25ff800700 1 -- 172.21.15.1:0/3859134689 >> 172.21.15.1:0/3859134689 conn(0x7f25f80f8ae0 msgr2=0x7f25f80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:07.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.351+0000 7f25ff800700 1 -- 172.21.15.1:0/3859134689 shutdown_connections 2024-02-28T17:26:07.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.351+0000 7f25ff800700 1 -- 172.21.15.1:0/3859134689 wait complete. 2024-02-28T17:26:07.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.352+0000 7f25ff800700 1 Processor -- start 2024-02-28T17:26:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.352+0000 7f25ff800700 1 -- start start 2024-02-28T17:26:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.353+0000 7f25ff800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8102f50 0x7f25f8070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.353+0000 7f25ff800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25f81038f0 0x7f25f80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.353+0000 7f25ff800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8111980 con 0x7f25f8102f50 2024-02-28T17:26:07.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.353+0000 7f25ff800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8111af0 con 0x7f25f81038f0 2024-02-28T17:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.353+0000 7f25fce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8102f50 0x7f25f8070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.353+0000 7f25fce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8102f50 0x7f25f8070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41598/0 (socket says 172.21.15.1:41598) 2024-02-28T17:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.353+0000 7f25fce00700 1 -- 172.21.15.1:0/4142349677 learned_addr learned my addr 172.21.15.1:0/4142349677 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.353+0000 7f25f7e00700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25f81038f0 0x7f25f80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25fce00700 1 -- 172.21.15.1:0/4142349677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25f81038f0 msgr2=0x7f25f80707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25fce00700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25f81038f0 0x7f25f80707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25fce00700 1 -- 172.21.15.1:0/4142349677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e4005040 con 0x7f25f8102f50 2024-02-28T17:26:07.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25fce00700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8102f50 0x7f25f8070290 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f25ec002a50 tx=0x7f25ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:07.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25f4c00700 1 -- 172.21.15.1:0/4142349677 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25ec007cc0 con 0x7f25f8102f50 2024-02-28T17:26:07.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25f4c00700 1 -- 172.21.15.1:0/4142349677 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25ec00f040 con 0x7f25f8102f50 2024-02-28T17:26:07.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f8070e60 con 0x7f25f8102f50 2024-02-28T17:26:07.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25f4c00700 1 -- 172.21.15.1:0/4142349677 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25ec00db10 con 0x7f25f8102f50 2024-02-28T17:26:07.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.354+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f811e680 con 0x7f25f8102f50 2024-02-28T17:26:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.356+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f8105420 con 0x7f25f8102f50 2024-02-28T17:26:07.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.357+0000 7f25f4c00700 1 -- 172.21.15.1:0/4142349677 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f25ec00dcb0 con 0x7f25f8102f50 2024-02-28T17:26:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.357+0000 7f25f4c00700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25e006d7f0 0x7f25e006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:07.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.358+0000 7f25f4c00700 1 -- 172.21.15.1:0/4142349677 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f25ec08c5a0 con 0x7f25f8102f50 2024-02-28T17:26:07.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.358+0000 7f25f7e00700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25e006d7f0 0x7f25e006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:07.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.359+0000 7f25f7e00700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25e006d7f0 0x7f25e006fca0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f25f80723a0 tx=0x7f25e4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:07.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.362+0000 7f25f4c00700 1 -- 172.21.15.1:0/4142349677 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f25ec055dd0 con 0x7f25f8102f50 2024-02-28T17:26:07.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.608+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25f80719d0 con 0x7f25e006d7f0 2024-02-28T17:26:07.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.611+0000 7f25f4c00700 1 -- 172.21.15.1:0/4142349677 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f25f80719d0 con 0x7f25e006d7f0 2024-02-28T17:26:07.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.613+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25e006d7f0 msgr2=0x7f25e006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:07.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.613+0000 7f25ff800700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25e006d7f0 0x7f25e006fca0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f25f80723a0 tx=0x7f25e4009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:07.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.613+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8102f50 msgr2=0x7f25f8070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:07.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.614+0000 7f25ff800700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8102f50 0x7f25f8070290 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f25ec002a50 tx=0x7f25ec0046b0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.614+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 shutdown_connections 2024-02-28T17:26:07.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.614+0000 7f25ff800700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f25e006d7f0 0x7f25e006fca0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.614+0000 7f25ff800700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8102f50 0x7f25f8070290 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.614+0000 7f25ff800700 1 --2- 172.21.15.1:0/4142349677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f25f81038f0 0x7f25f80707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.614+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 >> 172.21.15.1:0/4142349677 conn(0x7f25f80f8ae0 msgr2=0x7f25f80fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:07.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.614+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 shutdown_connections 2024-02-28T17:26:07.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.614+0000 7f25ff800700 1 -- 172.21.15.1:0/4142349677 wait complete. 2024-02-28T17:26:07.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.772+0000 7f8404000700 1 -- 172.21.15.1:0/4080300152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 msgr2=0x7f83fc0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:07.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.772+0000 7f8404000700 1 --2- 172.21.15.1:0/4080300152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 0x7f83fc0fcf60 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f83e8004940 tx=0x7f83e8009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:07.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.772+0000 7f8404000700 1 -- 172.21.15.1:0/4080300152 shutdown_connections 2024-02-28T17:26:07.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.772+0000 7f8404000700 1 --2- 172.21.15.1:0/4080300152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 0x7f83fc0fcf60 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.772+0000 7f8404000700 1 --2- 172.21.15.1:0/4080300152 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83fc102f50 0x7f83fc103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.772+0000 7f8404000700 1 -- 172.21.15.1:0/4080300152 >> 172.21.15.1:0/4080300152 conn(0x7f83fc0f8ae0 msgr2=0x7f83fc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:07.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.772+0000 7f8404000700 1 -- 172.21.15.1:0/4080300152 shutdown_connections 2024-02-28T17:26:07.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.772+0000 7f8404000700 1 -- 172.21.15.1:0/4080300152 wait complete. 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.773+0000 7f8404000700 1 Processor -- start 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.773+0000 7f8404000700 1 -- start start 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.773+0000 7f8404000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83fc102f50 0x7f83fc115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.773+0000 7f8404000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 0x7f83fc1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.773+0000 7f8404000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc116810 con 0x7f83fc1038f0 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.773+0000 7f8404000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc10fb40 con 0x7f83fc102f50 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8400c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 0x7f83fc1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8400c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 0x7f83fc1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41602/0 (socket says 172.21.15.1:41602) 2024-02-28T17:26:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8400c00700 1 -- 172.21.15.1:0/1439508157 learned_addr learned my addr 172.21.15.1:0/1439508157 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:07.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8401600700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83fc102f50 0x7f83fc115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:07.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8400c00700 1 -- 172.21.15.1:0/1439508157 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83fc102f50 msgr2=0x7f83fc115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:07.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8400c00700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83fc102f50 0x7f83fc115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:07.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8400c00700 1 -- 172.21.15.1:0/1439508157 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8005040 con 0x7f83fc1038f0 2024-02-28T17:26:07.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8401600700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83fc102f50 0x7f83fc115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:26:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.774+0000 7f8400c00700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 0x7f83fc1161c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009030 tx=0x7f83e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.775+0000 7f83f9600700 1 -- 172.21.15.1:0/1439508157 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83e800e040 con 0x7f83fc1038f0 2024-02-28T17:26:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.775+0000 7f83f9600700 1 -- 172.21.15.1:0/1439508157 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83e800bbf0 con 0x7f83fc1038f0 2024-02-28T17:26:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.775+0000 7f83f9600700 1 -- 172.21.15.1:0/1439508157 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83e801c630 con 0x7f83fc1038f0 2024-02-28T17:26:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.775+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83fc10fdc0 con 0x7f83fc1038f0 2024-02-28T17:26:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.775+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83fc110300 con 0x7f83fc1038f0 2024-02-28T17:26:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.776+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83fc0fc6e0 con 0x7f83fc1038f0 2024-02-28T17:26:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.777+0000 7f83f9600700 1 -- 172.21.15.1:0/1439508157 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f83e800bd60 con 0x7f83fc1038f0 2024-02-28T17:26:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.779+0000 7f83f9600700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83e406d7f0 0x7f83e406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.779+0000 7f83f9600700 1 -- 172.21.15.1:0/1439508157 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f83e808e130 con 0x7f83fc1038f0 2024-02-28T17:26:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.779+0000 7f8401600700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83e406d7f0 0x7f83e406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.780+0000 7f8401600700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83e406d7f0 0x7f83e406fca0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f83f0009fd0 tx=0x7f83f000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:07.783+0000 7f83f9600700 1 -- 172.21.15.1:0/1439508157 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f83e805b5d0 con 0x7f83fc1038f0 2024-02-28T17:26:08.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.015+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f83fc1111f0 con 0x7f83e406d7f0 2024-02-28T17:26:08.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.044+0000 7f83f9600700 1 -- 172.21.15.1:0/1439508157 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f83fc1111f0 con 0x7f83e406d7f0 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (52m) 13m ago 66m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (48m) 13m ago 66m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (48m) 13m ago 65m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (51m) 13m ago 66m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (44m) 13m ago 59m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (44m) 13m ago 59m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (13m) 13m ago 59m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (29m) 13m ago 59m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (54m) 13m ago 68m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (53m) 13m ago 65m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:26:08.048 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (50m) 13m ago 68m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (50m) 13m ago 65m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (53m) 13m ago 66m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (53m) 13m ago 65m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (48m) 13m ago 64m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (47m) 13m ago 63m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (47m) 13m ago 63m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (46m) 13m ago 62m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (46m) 13m ago 61m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (45m) 13m ago 61m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:26:08.049 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (52m) 13m ago 65m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:26:08.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83e406d7f0 msgr2=0x7f83e406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83e406d7f0 0x7f83e406fca0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f83f0009fd0 tx=0x7f83f000b040 comp rx=0 tx=0).stop 2024-02-28T17:26:08.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 msgr2=0x7f83fc1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 0x7f83fc1161c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009030 tx=0x7f83e8009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:08.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 shutdown_connections 2024-02-28T17:26:08.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f83e406d7f0 0x7f83e406fca0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f83fc102f50 0x7f83fc115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 --2- 172.21.15.1:0/1439508157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83fc1038f0 0x7f83fc1161c0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 >> 172.21.15.1:0/1439508157 conn(0x7f83fc0f8ae0 msgr2=0x7f83fc0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:08.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 shutdown_connections 2024-02-28T17:26:08.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.047+0000 7f8404000700 1 -- 172.21.15.1:0/1439508157 wait complete. 2024-02-28T17:26:08.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.199+0000 7ff307000700 1 -- 172.21.15.1:0/1113178988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff300070be0 msgr2=0x7ff3001092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.199+0000 7ff307000700 1 --2- 172.21.15.1:0/1113178988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff300070be0 0x7ff3001092c0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec004970 tx=0x7ff2ec009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:08.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.199+0000 7ff307000700 1 -- 172.21.15.1:0/1113178988 shutdown_connections 2024-02-28T17:26:08.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.199+0000 7ff307000700 1 --2- 172.21.15.1:0/1113178988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff300070be0 0x7ff3001092c0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.199+0000 7ff307000700 1 --2- 172.21.15.1:0/1113178988 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff3000702d0 0x7ff3000706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.199+0000 7ff307000700 1 -- 172.21.15.1:0/1113178988 >> 172.21.15.1:0/1113178988 conn(0x7ff3000f8ae0 msgr2=0x7ff3000faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:08.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.199+0000 7ff307000700 1 -- 172.21.15.1:0/1113178988 shutdown_connections 2024-02-28T17:26:08.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.199+0000 7ff307000700 1 -- 172.21.15.1:0/1113178988 wait complete. 2024-02-28T17:26:08.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.200+0000 7ff307000700 1 Processor -- start 2024-02-28T17:26:08.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.200+0000 7ff307000700 1 -- start start 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.200+0000 7ff307000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3000702d0 0x7ff300115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.200+0000 7ff307000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff300070be0 0x7ff300116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.200+0000 7ff307000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3001167a0 con 0x7ff3000702d0 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.200+0000 7ff307000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30010faf0 con 0x7ff300070be0 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.201+0000 7ff2ffe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3000702d0 0x7ff300115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.201+0000 7ff2ffe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3000702d0 0x7ff300115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41624/0 (socket says 172.21.15.1:41624) 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.201+0000 7ff2ffe00700 1 -- 172.21.15.1:0/140900350 learned_addr learned my addr 172.21.15.1:0/140900350 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.201+0000 7ff2ff400700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff300070be0 0x7ff300116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.201+0000 7ff2ffe00700 1 -- 172.21.15.1:0/140900350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff300070be0 msgr2=0x7ff300116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.201+0000 7ff2ffe00700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff300070be0 0x7ff300116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.201+0000 7ff2ffe00700 1 -- 172.21.15.1:0/140900350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec005040 con 0x7ff3000702d0 2024-02-28T17:26:08.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.202+0000 7ff2ffe00700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3000702d0 0x7ff300115c10 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000c830 tx=0x7ff2f0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:08.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.202+0000 7ff2f7e00700 1 -- 172.21.15.1:0/140900350 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f000a760 con 0x7ff3000702d0 2024-02-28T17:26:08.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.202+0000 7ff2f7e00700 1 -- 172.21.15.1:0/140900350 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2f000ad90 con 0x7ff3000702d0 2024-02-28T17:26:08.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.202+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30010fdd0 con 0x7ff3000702d0 2024-02-28T17:26:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.202+0000 7ff2f7e00700 1 -- 172.21.15.1:0/140900350 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f00113a0 con 0x7ff3000702d0 2024-02-28T17:26:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.202+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff300110310 con 0x7ff3000702d0 2024-02-28T17:26:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.203+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff300106980 con 0x7ff3000702d0 2024-02-28T17:26:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.204+0000 7ff2f7e00700 1 -- 172.21.15.1:0/140900350 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff2f0012400 con 0x7ff3000702d0 2024-02-28T17:26:08.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.206+0000 7ff2f7e00700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff2e406d7e0 0x7ff2e406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:08.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.206+0000 7ff2f7e00700 1 -- 172.21.15.1:0/140900350 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff2f008cec0 con 0x7ff3000702d0 2024-02-28T17:26:08.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.206+0000 7ff2ff400700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff2e406d7e0 0x7ff2e406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:08.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.207+0000 7ff2ff400700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff2e406d7e0 0x7ff2e406fc90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff3001114b0 tx=0x7ff2ec009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.211+0000 7ff2f7e00700 1 -- 172.21.15.1:0/140900350 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff2f0056770 con 0x7ff3000702d0 2024-02-28T17:26:08.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.514+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff3001111f0 con 0x7ff3000702d0 2024-02-28T17:26:08.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.515+0000 7ff2f7e00700 1 -- 172.21.15.1:0/140900350 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff2f0059e30 con 0x7ff3000702d0 2024-02-28T17:26:08.518 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:26:08.518 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:26:08.518 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:26:08.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:26:08.518 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:26:08.519 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:26:08.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:26:08.519 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:26:08.519 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:26:08.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:26:08.520 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:26:08.520 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:26:08.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:26:08.520 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:26:08.520 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:26:08.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:26:08.521 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:26:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff2e406d7e0 msgr2=0x7ff2e406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff2e406d7e0 0x7ff2e406fc90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff3001114b0 tx=0x7ff2ec009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3000702d0 msgr2=0x7ff300115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3000702d0 0x7ff300115c10 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000c830 tx=0x7ff2f0004470 comp rx=0 tx=0).stop 2024-02-28T17:26:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 shutdown_connections 2024-02-28T17:26:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff2e406d7e0 0x7ff2e406fc90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3000702d0 0x7ff300115c10 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 --2- 172.21.15.1:0/140900350 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff300070be0 0x7ff300116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 >> 172.21.15.1:0/140900350 conn(0x7ff3000f8ae0 msgr2=0x7ff300103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 shutdown_connections 2024-02-28T17:26:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.518+0000 7ff307000700 1 -- 172.21.15.1:0/140900350 wait complete. 2024-02-28T17:26:08.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.674+0000 7fc5a4e00700 1 -- 172.21.15.1:0/1520835569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105170 msgr2=0x7fc5a0105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.674+0000 7fc5a4e00700 1 --2- 172.21.15.1:0/1520835569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105170 0x7fc5a0105540 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc590004940 tx=0x7fc590009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:08.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.674+0000 7fc5a4e00700 1 -- 172.21.15.1:0/1520835569 shutdown_connections 2024-02-28T17:26:08.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.674+0000 7fc5a4e00700 1 --2- 172.21.15.1:0/1520835569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a0105b10 0x7fc5a00ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.674+0000 7fc5a4e00700 1 --2- 172.21.15.1:0/1520835569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105170 0x7fc5a0105540 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.674+0000 7fc5a4e00700 1 -- 172.21.15.1:0/1520835569 >> 172.21.15.1:0/1520835569 conn(0x7fc5a00face0 msgr2=0x7fc5a00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:08.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.675+0000 7fc5a4e00700 1 -- 172.21.15.1:0/1520835569 shutdown_connections 2024-02-28T17:26:08.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.675+0000 7fc5a4e00700 1 -- 172.21.15.1:0/1520835569 wait complete. 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.675+0000 7fc5a4e00700 1 Processor -- start 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.676+0000 7fc5a4e00700 1 -- start start 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.676+0000 7fc5a4e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a0105170 0x7fc5a0115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.676+0000 7fc5a4e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105b10 0x7fc5a01161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.676+0000 7fc5a4e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a01167f0 con 0x7fc5a0105b10 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.676+0000 7fc5a4e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a010fb40 con 0x7fc5a0105170 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.676+0000 7fc59d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105b10 0x7fc5a01161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.676+0000 7fc59d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105b10 0x7fc5a01161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41630/0 (socket says 172.21.15.1:41630) 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.676+0000 7fc59d400700 1 -- 172.21.15.1:0/2567430820 learned_addr learned my addr 172.21.15.1:0/2567430820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.677+0000 7fc59de00700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a0105170 0x7fc5a0115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.677+0000 7fc59d400700 1 -- 172.21.15.1:0/2567430820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a0105170 msgr2=0x7fc5a0115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.677+0000 7fc59d400700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a0105170 0x7fc5a0115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.677+0000 7fc59d400700 1 -- 172.21.15.1:0/2567430820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc590005040 con 0x7fc5a0105b10 2024-02-28T17:26:08.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.677+0000 7fc59d400700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105b10 0x7fc5a01161a0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fc58c002a50 tx=0x7fc58c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:08.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.677+0000 7fc596000700 1 -- 172.21.15.1:0/2567430820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc58c007cc0 con 0x7fc5a0105b10 2024-02-28T17:26:08.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.678+0000 7fc596000700 1 -- 172.21.15.1:0/2567430820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc58c00f040 con 0x7fc5a0105b10 2024-02-28T17:26:08.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.678+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a010fe20 con 0x7fc5a0105b10 2024-02-28T17:26:08.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.678+0000 7fc596000700 1 -- 172.21.15.1:0/2567430820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc58c00daf0 con 0x7fc5a0105b10 2024-02-28T17:26:08.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.678+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a0110360 con 0x7fc5a0105b10 2024-02-28T17:26:08.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.679+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a0102480 con 0x7fc5a0105b10 2024-02-28T17:26:08.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.680+0000 7fc596000700 1 -- 172.21.15.1:0/2567430820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc58c009810 con 0x7fc5a0105b10 2024-02-28T17:26:08.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.681+0000 7fc596000700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58406d7f0 0x7fc58406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:08.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.681+0000 7fc596000700 1 -- 172.21.15.1:0/2567430820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc58c08ccf0 con 0x7fc5a0105b10 2024-02-28T17:26:08.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.681+0000 7fc59de00700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58406d7f0 0x7fc58406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:08.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.682+0000 7fc59de00700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58406d7f0 0x7fc58406fca0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc590000c00 tx=0x7fc590009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:08.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.686+0000 7fc596000700 1 -- 172.21.15.1:0/2567430820 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc58c0564a0 con 0x7fc5a0105b10 2024-02-28T17:26:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.958+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc5a004bed0 con 0x7fc5a0105b10 2024-02-28T17:26:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.959+0000 7fc596000700 1 -- 172.21.15.1:0/2567430820 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fc58c059b60 con 0x7fc5a0105b10 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:26:08.962 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:26:08.963 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:26:08.964 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:26:08.965 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:26:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.961+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58406d7f0 msgr2=0x7fc58406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.961+0000 7fc5a4e00700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58406d7f0 0x7fc58406fca0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc590000c00 tx=0x7fc590009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.961+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105b10 msgr2=0x7fc5a01161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.961+0000 7fc5a4e00700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105b10 0x7fc5a01161a0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fc58c002a50 tx=0x7fc58c004600 comp rx=0 tx=0).stop 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.961+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 shutdown_connections 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.961+0000 7fc5a4e00700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc58406d7f0 0x7fc58406fca0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.961+0000 7fc5a4e00700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc5a0105170 0x7fc5a0115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.962+0000 7fc5a4e00700 1 --2- 172.21.15.1:0/2567430820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5a0105b10 0x7fc5a01161a0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.962+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 >> 172.21.15.1:0/2567430820 conn(0x7fc5a00face0 msgr2=0x7fc5a00fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.962+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 shutdown_connections 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:08.962+0000 7fc5a4e00700 1 -- 172.21.15.1:0/2567430820 wait complete. 2024-02-28T17:26:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:26:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.118+0000 7f703d200700 1 -- 172.21.15.1:0/1993300198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70381095b0 msgr2=0x7f7038109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.118+0000 7f703d200700 1 --2- 172.21.15.1:0/1993300198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70381095b0 0x7f7038109980 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f7028004940 tx=0x7f7028009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.118+0000 7f703d200700 1 -- 172.21.15.1:0/1993300198 shutdown_connections 2024-02-28T17:26:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.118+0000 7f703d200700 1 --2- 172.21.15.1:0/1993300198 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7038103dc0 0x7f7038104230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.118+0000 7f703d200700 1 --2- 172.21.15.1:0/1993300198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70381095b0 0x7f7038109980 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.119+0000 7f703d200700 1 -- 172.21.15.1:0/1993300198 >> 172.21.15.1:0/1993300198 conn(0x7f7038072d70 msgr2=0x7f7038075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.119+0000 7f703d200700 1 -- 172.21.15.1:0/1993300198 shutdown_connections 2024-02-28T17:26:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.119+0000 7f703d200700 1 -- 172.21.15.1:0/1993300198 wait complete. 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.120+0000 7f703d200700 1 Processor -- start 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.120+0000 7f703d200700 1 -- start start 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.120+0000 7f703d200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7038103dc0 0x7f703811a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.120+0000 7f703d200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70381095b0 0x7f703811a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.120+0000 7f703d200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703811ac30 con 0x7f7038103dc0 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.120+0000 7f703d200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7038113f40 con 0x7f70381095b0 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.121+0000 7f7035e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7038103dc0 0x7f703811a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.121+0000 7f7035e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7038103dc0 0x7f703811a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41638/0 (socket says 172.21.15.1:41638) 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.121+0000 7f7035e00700 1 -- 172.21.15.1:0/1686118920 learned_addr learned my addr 172.21.15.1:0/1686118920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.121+0000 7f7035e00700 1 -- 172.21.15.1:0/1686118920 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70381095b0 msgr2=0x7f703811a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.121+0000 7f7035e00700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70381095b0 0x7f703811a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.121+0000 7f7035e00700 1 -- 172.21.15.1:0/1686118920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7028005040 con 0x7f7038103dc0 2024-02-28T17:26:09.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.121+0000 7f7035e00700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7038103dc0 0x7f703811a0a0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7028009100 tx=0x7f7028009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:09.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.121+0000 7f702e000700 1 -- 172.21.15.1:0/1686118920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702800e040 con 0x7f7038103dc0 2024-02-28T17:26:09.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.122+0000 7f702e000700 1 -- 172.21.15.1:0/1686118920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7028004290 con 0x7f7038103dc0 2024-02-28T17:26:09.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.122+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70381141c0 con 0x7f7038103dc0 2024-02-28T17:26:09.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.122+0000 7f702e000700 1 -- 172.21.15.1:0/1686118920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702801c3f0 con 0x7f7038103dc0 2024-02-28T17:26:09.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.122+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7038114700 con 0x7f7038103dc0 2024-02-28T17:26:09.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.123+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7038106850 con 0x7f7038103dc0 2024-02-28T17:26:09.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.124+0000 7f702e000700 1 -- 172.21.15.1:0/1686118920 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f702800bbf0 con 0x7f7038103dc0 2024-02-28T17:26:09.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.125+0000 7f702e000700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f701c06d7f0 0x7f701c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:09.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.125+0000 7f702e000700 1 -- 172.21.15.1:0/1686118920 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f702808e200 con 0x7f7038103dc0 2024-02-28T17:26:09.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.125+0000 7f7035400700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f701c06d7f0 0x7f701c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:09.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.126+0000 7f7035400700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f701c06d7f0 0x7f701c06fca0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7038115940 tx=0x7f7024009ce0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:09.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.130+0000 7f702e000700 1 -- 172.21.15.1:0/1686118920 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f702805b600 con 0x7f7038103dc0 2024-02-28T17:26:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:08 smithi066 ceph-mon[85781]: from='client.36720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:08 smithi066 ceph-mon[85781]: from='client.36724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:08 smithi066 ceph-mon[85781]: pgmap v1554: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:08 smithi066 ceph-mon[85781]: from='client.36728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:09.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:08 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/140900350' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:26:09.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:08 smithi001 ceph-mon[112940]: from='client.36720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:09.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:08 smithi001 ceph-mon[112940]: from='client.36724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:09.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:08 smithi001 ceph-mon[112940]: pgmap v1554: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:09.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:08 smithi001 ceph-mon[112940]: from='client.36728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:09.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:08 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/140900350' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:26:09.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.371+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f703804bf40 con 0x7f701c06d7f0 2024-02-28T17:26:09.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.373+0000 7f702e000700 1 -- 172.21.15.1:0/1686118920 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f703804bf40 con 0x7f701c06d7f0 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:26:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:26:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:26:09.377 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:26:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.376+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f701c06d7f0 msgr2=0x7f701c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.376+0000 7f703d200700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f701c06d7f0 0x7f701c06fca0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7038115940 tx=0x7f7024009ce0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.376+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7038103dc0 msgr2=0x7f703811a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:09.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.376+0000 7f703d200700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7038103dc0 0x7f703811a0a0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7028009100 tx=0x7f7028009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:09.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.377+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 shutdown_connections 2024-02-28T17:26:09.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.377+0000 7f703d200700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f701c06d7f0 0x7f701c06fca0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.377+0000 7f703d200700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7038103dc0 0x7f703811a0a0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.377+0000 7f703d200700 1 --2- 172.21.15.1:0/1686118920 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f70381095b0 0x7f703811a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.377+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 >> 172.21.15.1:0/1686118920 conn(0x7f7038072d70 msgr2=0x7f7038074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.377+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 shutdown_connections 2024-02-28T17:26:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.377+0000 7f703d200700 1 -- 172.21.15.1:0/1686118920 wait complete. 2024-02-28T17:26:09.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.530+0000 7fdca1a00700 1 -- 172.21.15.1:0/681140677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c1038f0 msgr2=0x7fdc9c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:09.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.530+0000 7fdca1a00700 1 --2- 172.21.15.1:0/681140677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c1038f0 0x7fdc9c0fcf60 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fdc88004940 tx=0x7fdc88009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:09.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.530+0000 7fdca1a00700 1 -- 172.21.15.1:0/681140677 shutdown_connections 2024-02-28T17:26:09.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.530+0000 7fdca1a00700 1 --2- 172.21.15.1:0/681140677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c1038f0 0x7fdc9c0fcf60 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.530+0000 7fdca1a00700 1 --2- 172.21.15.1:0/681140677 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc9c102f50 0x7fdc9c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.530+0000 7fdca1a00700 1 -- 172.21.15.1:0/681140677 >> 172.21.15.1:0/681140677 conn(0x7fdc9c0f8ae0 msgr2=0x7fdc9c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:09.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.530+0000 7fdca1a00700 1 -- 172.21.15.1:0/681140677 shutdown_connections 2024-02-28T17:26:09.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.530+0000 7fdca1a00700 1 -- 172.21.15.1:0/681140677 wait complete. 2024-02-28T17:26:09.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.531+0000 7fdca1a00700 1 Processor -- start 2024-02-28T17:26:09.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.531+0000 7fdca1a00700 1 -- start start 2024-02-28T17:26:09.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.531+0000 7fdca1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c102f50 0x7fdc9c113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:09.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdca1a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc9c1038f0 0x7fdc9c113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdca1a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c114590 con 0x7fdc9c102f50 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdca1a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c10d8c0 con 0x7fdc9c1038f0 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdc9aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c102f50 0x7fdc9c113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdc9aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c102f50 0x7fdc9c113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41650/0 (socket says 172.21.15.1:41650) 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdc9aa00700 1 -- 172.21.15.1:0/2563059143 learned_addr learned my addr 172.21.15.1:0/2563059143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdc9aa00700 1 -- 172.21.15.1:0/2563059143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc9c1038f0 msgr2=0x7fdc9c113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdc9aa00700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc9c1038f0 0x7fdc9c113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdc9aa00700 1 -- 172.21.15.1:0/2563059143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc88005040 con 0x7fdc9c102f50 2024-02-28T17:26:09.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.532+0000 7fdc9aa00700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c102f50 0x7fdc9c113a00 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00eb40 tx=0x7fdc8c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:09.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.533+0000 7fdc92a00700 1 -- 172.21.15.1:0/2563059143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc8c007cc0 con 0x7fdc9c102f50 2024-02-28T17:26:09.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.533+0000 7fdc92a00700 1 -- 172.21.15.1:0/2563059143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc8c00f040 con 0x7fdc9c102f50 2024-02-28T17:26:09.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.533+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c10db40 con 0x7fdc9c102f50 2024-02-28T17:26:09.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.533+0000 7fdc92a00700 1 -- 172.21.15.1:0/2563059143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc8c009620 con 0x7fdc9c102f50 2024-02-28T17:26:09.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.533+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c10e080 con 0x7fdc9c102f50 2024-02-28T17:26:09.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.534+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c105420 con 0x7fdc9c102f50 2024-02-28T17:26:09.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.535+0000 7fdc92a00700 1 -- 172.21.15.1:0/2563059143 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fdc8c009780 con 0x7fdc9c102f50 2024-02-28T17:26:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.536+0000 7fdc92a00700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc8006d7f0 0x7fdc8006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.536+0000 7fdc92a00700 1 -- 172.21.15.1:0/2563059143 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fdc8c08cb40 con 0x7fdc9c102f50 2024-02-28T17:26:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.536+0000 7fdc9a000700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc8006d7f0 0x7fdc8006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:09.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.537+0000 7fdc9a000700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc8006d7f0 0x7fdc8006fca0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800b0d0 tx=0x7fdc88009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:09.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.540+0000 7fdc92a00700 1 -- 172.21.15.1:0/2563059143 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdc8c059f50 con 0x7fdc9c102f50 2024-02-28T17:26:09.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.841+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdc9c04bed0 con 0x7fdc9c102f50 2024-02-28T17:26:09.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.842+0000 7fdc92a00700 1 -- 172.21.15.1:0/2563059143 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fdc8c059ae0 con 0x7fdc9c102f50 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:26:09.845 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:26:09.846 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:26:09.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc8006d7f0 msgr2=0x7fdc8006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:09.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc8006d7f0 0x7fdc8006fca0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800b0d0 tx=0x7fdc88009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:09.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c102f50 msgr2=0x7fdc9c113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:09.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c102f50 0x7fdc9c113a00 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00eb40 tx=0x7fdc8c00b040 comp rx=0 tx=0).stop 2024-02-28T17:26:09.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 shutdown_connections 2024-02-28T17:26:09.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fdc8006d7f0 0x7fdc8006fca0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c102f50 0x7fdc9c113a00 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 --2- 172.21.15.1:0/2563059143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fdc9c1038f0 0x7fdc9c113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:09.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 >> 172.21.15.1:0/2563059143 conn(0x7fdc9c0f8ae0 msgr2=0x7fdc9c0fa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:09.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 shutdown_connections 2024-02-28T17:26:09.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:09.845+0000 7fdca1a00700 1 -- 172.21.15.1:0/2563059143 wait complete. 2024-02-28T17:26:10.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:09 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2567430820' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:26:10.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:09 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2563059143' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:26:10.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:09 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2567430820' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:26:10.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:09 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2563059143' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:26:11.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:10 smithi066 ceph-mon[85781]: from='client.36740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:11.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:10 smithi066 ceph-mon[85781]: pgmap v1555: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:11.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:10 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:26:11.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:10 smithi001 ceph-mon[112940]: from='client.36740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:11.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:10 smithi001 ceph-mon[112940]: pgmap v1555: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:11.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:10 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:26:13.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:12 smithi066 ceph-mon[85781]: pgmap v1556: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:13.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:12 smithi001 ceph-mon[112940]: pgmap v1556: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:15.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:14 smithi066 ceph-mon[85781]: pgmap v1557: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:15.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:14 smithi001 ceph-mon[112940]: pgmap v1557: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:17.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:16 smithi066 ceph-mon[85781]: pgmap v1558: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:17.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:16 smithi001 ceph-mon[112940]: pgmap v1558: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:19.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:18 smithi066 ceph-mon[85781]: pgmap v1559: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:19.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:18 smithi001 ceph-mon[112940]: pgmap v1559: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:21.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:20 smithi066 ceph-mon[85781]: pgmap v1560: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:21.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:20 smithi001 ceph-mon[112940]: pgmap v1560: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:23.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:22 smithi066 ceph-mon[85781]: pgmap v1561: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:23.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:22 smithi001 ceph-mon[112940]: pgmap v1561: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:25.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:24 smithi066 ceph-mon[85781]: pgmap v1562: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:25.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:24 smithi001 ceph-mon[112940]: pgmap v1562: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:26.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:26:26.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:26:27.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:26 smithi066 ceph-mon[85781]: pgmap v1563: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:27.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:26 smithi001 ceph-mon[112940]: pgmap v1563: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:29.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:28 smithi066 ceph-mon[85781]: pgmap v1564: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:29.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:28 smithi001 ceph-mon[112940]: pgmap v1564: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:31.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:30 smithi066 ceph-mon[85781]: pgmap v1565: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:31.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:30 smithi001 ceph-mon[112940]: pgmap v1565: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:33.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:32 smithi066 ceph-mon[85781]: pgmap v1566: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:33.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:32 smithi001 ceph-mon[112940]: pgmap v1566: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:35.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:34 smithi066 ceph-mon[85781]: pgmap v1567: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:35.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:34 smithi001 ceph-mon[112940]: pgmap v1567: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:37.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:37 smithi066 ceph-mon[85781]: pgmap v1568: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:37.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:37 smithi001 ceph-mon[112940]: pgmap v1568: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:39.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:39 smithi001 ceph-mon[112940]: pgmap v1569: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:39.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:39 smithi066 ceph-mon[85781]: pgmap v1569: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:40.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.015+0000 7f757fa00700 1 -- 172.21.15.1:0/584144363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 msgr2=0x7f75780ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.015+0000 7f757fa00700 1 --2- 172.21.15.1:0/584144363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 0x7f75780ff160 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f7564004940 tx=0x7f7564009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:40.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.015+0000 7f757fa00700 1 -- 172.21.15.1:0/584144363 shutdown_connections 2024-02-28T17:26:40.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.016+0000 7f757fa00700 1 --2- 172.21.15.1:0/584144363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 0x7f75780ff160 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.016+0000 7f757fa00700 1 --2- 172.21.15.1:0/584144363 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7578105150 0x7f7578105520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.016+0000 7f757fa00700 1 -- 172.21.15.1:0/584144363 >> 172.21.15.1:0/584144363 conn(0x7f75780face0 msgr2=0x7f75780fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:40.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.016+0000 7f757fa00700 1 -- 172.21.15.1:0/584144363 shutdown_connections 2024-02-28T17:26:40.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.016+0000 7f757fa00700 1 -- 172.21.15.1:0/584144363 wait complete. 2024-02-28T17:26:40.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.017+0000 7f757fa00700 1 Processor -- start 2024-02-28T17:26:40.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.017+0000 7f757fa00700 1 -- start start 2024-02-28T17:26:40.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.017+0000 7f757fa00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7578105150 0x7f7578115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.017+0000 7f757fa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 0x7f75781161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.017+0000 7f757fa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75781167f0 con 0x7f7578105af0 2024-02-28T17:26:40.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.017+0000 7f757fa00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757810fb40 con 0x7f7578105150 2024-02-28T17:26:40.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.018+0000 7f7577e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 0x7f75781161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.018+0000 7f7577e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 0x7f75781161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44520/0 (socket says 172.21.15.1:44520) 2024-02-28T17:26:40.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.018+0000 7f7577e00700 1 -- 172.21.15.1:0/2366133579 learned_addr learned my addr 172.21.15.1:0/2366133579 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.018+0000 7f757d000700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7578105150 0x7f7578115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.018+0000 7f7577e00700 1 -- 172.21.15.1:0/2366133579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7578105150 msgr2=0x7f7578115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.018+0000 7f7577e00700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7578105150 0x7f7578115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.018+0000 7f7577e00700 1 -- 172.21.15.1:0/2366133579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7564005040 con 0x7f7578105af0 2024-02-28T17:26:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.019+0000 7f7577e00700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 0x7f75781161a0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f7564009030 tx=0x7f7564009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.019+0000 7f7574c00700 1 -- 172.21.15.1:0/2366133579 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f756400e040 con 0x7f7578105af0 2024-02-28T17:26:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.019+0000 7f7574c00700 1 -- 172.21.15.1:0/2366133579 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f756400bbf0 con 0x7f7578105af0 2024-02-28T17:26:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.019+0000 7f7574c00700 1 -- 172.21.15.1:0/2366133579 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f756401c630 con 0x7f7578105af0 2024-02-28T17:26:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.019+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757810fdc0 con 0x7f7578105af0 2024-02-28T17:26:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.019+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7578110300 con 0x7f7578105af0 2024-02-28T17:26:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.021+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7578107620 con 0x7f7578105af0 2024-02-28T17:26:40.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.021+0000 7f7574c00700 1 -- 172.21.15.1:0/2366133579 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f756400bd60 con 0x7f7578105af0 2024-02-28T17:26:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.022+0000 7f7574c00700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f756006d7f0 0x7f756006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.022+0000 7f7574c00700 1 -- 172.21.15.1:0/2366133579 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f756408e1f0 con 0x7f7578105af0 2024-02-28T17:26:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.023+0000 7f757d000700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f756006d7f0 0x7f756006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.024+0000 7f757d000700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f756006d7f0 0x7f756006fca0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f756c004d60 tx=0x7f756c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.027+0000 7f7574c00700 1 -- 172.21.15.1:0/2366133579 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7564057aa0 con 0x7f7578105af0 2024-02-28T17:26:40.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.278+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f757804bed0 con 0x7f756006d7f0 2024-02-28T17:26:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.282+0000 7f7574c00700 1 -- 172.21.15.1:0/2366133579 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f757804bed0 con 0x7f756006d7f0 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.284+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f756006d7f0 msgr2=0x7f756006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.284+0000 7f757fa00700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f756006d7f0 0x7f756006fca0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f756c004d60 tx=0x7f756c00b040 comp rx=0 tx=0).stop 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.284+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 msgr2=0x7f75781161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.284+0000 7f757fa00700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 0x7f75781161a0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f7564009030 tx=0x7f7564009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.285+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 shutdown_connections 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.285+0000 7f757fa00700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f756006d7f0 0x7f756006fca0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.285+0000 7f757fa00700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7578105150 0x7f7578115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.285+0000 7f757fa00700 1 --2- 172.21.15.1:0/2366133579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7578105af0 0x7f75781161a0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.285+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 >> 172.21.15.1:0/2366133579 conn(0x7f75780face0 msgr2=0x7f75780fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:40.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.285+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 shutdown_connections 2024-02-28T17:26:40.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.285+0000 7f757fa00700 1 -- 172.21.15.1:0/2366133579 wait complete. 2024-02-28T17:26:40.304 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:26:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.439+0000 7f1fdcc00700 1 -- 172.21.15.1:0/589446222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 msgr2=0x7f1fd8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.439+0000 7f1fdcc00700 1 --2- 172.21.15.1:0/589446222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 0x7f1fd8104210 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4004940 tx=0x7f1fc4009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.439+0000 7f1fdcc00700 1 -- 172.21.15.1:0/589446222 shutdown_connections 2024-02-28T17:26:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.439+0000 7f1fdcc00700 1 --2- 172.21.15.1:0/589446222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 0x7f1fd8104210 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.439+0000 7f1fdcc00700 1 --2- 172.21.15.1:0/589446222 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fd81095f0 0x7f1fd81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.439+0000 7f1fdcc00700 1 -- 172.21.15.1:0/589446222 >> 172.21.15.1:0/589446222 conn(0x7f1fd8072d50 msgr2=0x7f1fd8075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.439+0000 7f1fdcc00700 1 -- 172.21.15.1:0/589446222 shutdown_connections 2024-02-28T17:26:40.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.439+0000 7f1fdcc00700 1 -- 172.21.15.1:0/589446222 wait complete. 2024-02-28T17:26:40.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.440+0000 7f1fdcc00700 1 Processor -- start 2024-02-28T17:26:40.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.440+0000 7f1fdcc00700 1 -- start start 2024-02-28T17:26:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.440+0000 7f1fdcc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 0x7f1fd811a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.440+0000 7f1fdcc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fd81095f0 0x7f1fd811a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.440+0000 7f1fdcc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd811abd0 con 0x7f1fd8103dc0 2024-02-28T17:26:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.440+0000 7f1fdcc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8113ee0 con 0x7f1fd81095f0 2024-02-28T17:26:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.441+0000 7f1fd5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 0x7f1fd811a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.441+0000 7f1fd5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 0x7f1fd811a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44552/0 (socket says 172.21.15.1:44552) 2024-02-28T17:26:40.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.441+0000 7f1fd5e00700 1 -- 172.21.15.1:0/1579151600 learned_addr learned my addr 172.21.15.1:0/1579151600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:40.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.441+0000 7f1fd5400700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fd81095f0 0x7f1fd811a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.441+0000 7f1fd5e00700 1 -- 172.21.15.1:0/1579151600 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fd81095f0 msgr2=0x7f1fd811a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.441+0000 7f1fd5e00700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fd81095f0 0x7f1fd811a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.441+0000 7f1fd5e00700 1 -- 172.21.15.1:0/1579151600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4005040 con 0x7f1fd8103dc0 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.442+0000 7f1fd5e00700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 0x7f1fd811a040 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800c8c0 tx=0x7f1fc80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.442+0000 7f1fce000700 1 -- 172.21.15.1:0/1579151600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fc800a980 con 0x7f1fd8103dc0 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.442+0000 7f1fce000700 1 -- 172.21.15.1:0/1579151600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fc800ce60 con 0x7f1fd8103dc0 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.442+0000 7f1fce000700 1 -- 172.21.15.1:0/1579151600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fc80115e0 con 0x7f1fd8103dc0 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.442+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd81141c0 con 0x7f1fd8103dc0 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.442+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd8114700 con 0x7f1fd8103dc0 2024-02-28T17:26:40.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.444+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd810b9c0 con 0x7f1fd8103dc0 2024-02-28T17:26:40.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.445+0000 7f1fce000700 1 -- 172.21.15.1:0/1579151600 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1fc8012400 con 0x7f1fd8103dc0 2024-02-28T17:26:40.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.446+0000 7f1fce000700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fbc06d7f0 0x7f1fbc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.446+0000 7f1fce000700 1 -- 172.21.15.1:0/1579151600 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1fc808cf70 con 0x7f1fd8103dc0 2024-02-28T17:26:40.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.446+0000 7f1fd5400700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fbc06d7f0 0x7f1fbc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.447+0000 7f1fd5400700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fbc06d7f0 0x7f1fbc06fca0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400b3b0 tx=0x7f1fc4009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:40.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.450+0000 7f1fce000700 1 -- 172.21.15.1:0/1579151600 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1fc805a350 con 0x7f1fd8103dc0 2024-02-28T17:26:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.695+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fd804bf40 con 0x7f1fbc06d7f0 2024-02-28T17:26:40.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.697+0000 7f1fce000700 1 -- 172.21.15.1:0/1579151600 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1fd804bf40 con 0x7f1fbc06d7f0 2024-02-28T17:26:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.700+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fbc06d7f0 msgr2=0x7f1fbc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.700+0000 7f1fdcc00700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fbc06d7f0 0x7f1fbc06fca0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400b3b0 tx=0x7f1fc4009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 msgr2=0x7f1fd811a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 0x7f1fd811a040 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800c8c0 tx=0x7f1fc80044c0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 shutdown_connections 2024-02-28T17:26:40.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1fbc06d7f0 0x7f1fbc06fca0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fd8103dc0 0x7f1fd811a040 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 --2- 172.21.15.1:0/1579151600 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1fd81095f0 0x7f1fd811a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 >> 172.21.15.1:0/1579151600 conn(0x7f1fd8072d50 msgr2=0x7f1fd8074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 shutdown_connections 2024-02-28T17:26:40.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.701+0000 7f1fdcc00700 1 -- 172.21.15.1:0/1579151600 wait complete. 2024-02-28T17:26:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.861+0000 7f8181000700 1 -- 172.21.15.1:0/2404987579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fdfa0 msgr2=0x7f817c0fe370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.861+0000 7f8181000700 1 --2- 172.21.15.1:0/2404987579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fdfa0 0x7f817c0fe370 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f816c004940 tx=0x7f816c009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.861+0000 7f8181000700 1 -- 172.21.15.1:0/2404987579 shutdown_connections 2024-02-28T17:26:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.861+0000 7f8181000700 1 --2- 172.21.15.1:0/2404987579 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f817c0fe940 0x7f817c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.861+0000 7f8181000700 1 --2- 172.21.15.1:0/2404987579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fdfa0 0x7f817c0fe370 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.861+0000 7f8181000700 1 -- 172.21.15.1:0/2404987579 >> 172.21.15.1:0/2404987579 conn(0x7f817c0f79d0 msgr2=0x7f817c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.862+0000 7f8181000700 1 -- 172.21.15.1:0/2404987579 shutdown_connections 2024-02-28T17:26:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.862+0000 7f8181000700 1 -- 172.21.15.1:0/2404987579 wait complete. 2024-02-28T17:26:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.862+0000 7f8181000700 1 Processor -- start 2024-02-28T17:26:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.863+0000 7f8181000700 1 -- start start 2024-02-28T17:26:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.863+0000 7f8181000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f817c0fdfa0 0x7f817c115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.863+0000 7f8181000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fe940 0x7f817c1160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.863+0000 7f8181000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c116740 con 0x7f817c0fe940 2024-02-28T17:26:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.863+0000 7f8181000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c10fa70 con 0x7f817c0fdfa0 2024-02-28T17:26:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.863+0000 7f8179400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fe940 0x7f817c1160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.863+0000 7f8179400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fe940 0x7f817c1160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44572/0 (socket says 172.21.15.1:44572) 2024-02-28T17:26:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.863+0000 7f8179400700 1 -- 172.21.15.1:0/464094143 learned_addr learned my addr 172.21.15.1:0/464094143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.864+0000 7f8179e00700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f817c0fdfa0 0x7f817c115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.864+0000 7f8179400700 1 -- 172.21.15.1:0/464094143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f817c0fdfa0 msgr2=0x7f817c115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.864+0000 7f8179400700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f817c0fdfa0 0x7f817c115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.864+0000 7f8179400700 1 -- 172.21.15.1:0/464094143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c005040 con 0x7f817c0fe940 2024-02-28T17:26:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.864+0000 7f8179400700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fe940 0x7f817c1160f0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f816800c830 tx=0x7f8168004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.865+0000 7f8172000700 1 -- 172.21.15.1:0/464094143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f816800a760 con 0x7f817c0fe940 2024-02-28T17:26:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.865+0000 7f8172000700 1 -- 172.21.15.1:0/464094143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f816800ad90 con 0x7f817c0fe940 2024-02-28T17:26:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.865+0000 7f8172000700 1 -- 172.21.15.1:0/464094143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81680113a0 con 0x7f817c0fe940 2024-02-28T17:26:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.865+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c10fd50 con 0x7f817c0fe940 2024-02-28T17:26:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.865+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c110290 con 0x7f817c0fe940 2024-02-28T17:26:40.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.866+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c1069f0 con 0x7f817c0fe940 2024-02-28T17:26:40.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.867+0000 7f8172000700 1 -- 172.21.15.1:0/464094143 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f8168012400 con 0x7f817c0fe940 2024-02-28T17:26:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.868+0000 7f8172000700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f816006d7f0 0x7f816006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.868+0000 7f8172000700 1 -- 172.21.15.1:0/464094143 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f816808cf00 con 0x7f817c0fe940 2024-02-28T17:26:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.868+0000 7f8179e00700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f816006d7f0 0x7f816006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:40.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.869+0000 7f8179e00700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f816006d7f0 0x7f816006fca0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f816c00b3b0 tx=0x7f816c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:40.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:40.873+0000 7f8172000700 1 -- 172.21.15.1:0/464094143 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f816805a2e0 con 0x7f817c0fe940 2024-02-28T17:26:41.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.100+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f817c109f60 con 0x7f816006d7f0 2024-02-28T17:26:41.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.119+0000 7f8172000700 1 -- 172.21.15.1:0/464094143 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f817c109f60 con 0x7f816006d7f0 2024-02-28T17:26:41.122 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:26:41.122 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (52m) 14m ago 67m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:26:41.122 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (49m) 14m ago 67m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:26:41.122 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (49m) 14m ago 65m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:26:41.122 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (52m) 14m ago 66m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:26:41.122 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (45m) 14m ago 59m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:26:41.122 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (44m) 14m ago 59m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (14m) 14m ago 59m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (29m) 14m ago 59m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (55m) 14m ago 68m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (54m) 14m ago 65m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (51m) 14m ago 68m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (50m) 14m ago 65m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (53m) 14m ago 67m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (53m) 14m ago 65m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (48m) 14m ago 64m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (48m) 14m ago 64m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (47m) 14m ago 63m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (47m) 14m ago 62m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (46m) 14m ago 62m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:26:41.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (45m) 14m ago 61m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:26:41.124 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (53m) 14m ago 66m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:26:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f816006d7f0 msgr2=0x7f816006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f816006d7f0 0x7f816006fca0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f816c00b3b0 tx=0x7f816c009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fe940 msgr2=0x7f817c1160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:41.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fe940 0x7f817c1160f0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f816800c830 tx=0x7f8168004470 comp rx=0 tx=0).stop 2024-02-28T17:26:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 shutdown_connections 2024-02-28T17:26:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f816006d7f0 0x7f816006fca0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f817c0fdfa0 0x7f817c115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 --2- 172.21.15.1:0/464094143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c0fe940 0x7f817c1160f0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 >> 172.21.15.1:0/464094143 conn(0x7f817c0f79d0 msgr2=0x7f817c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.122+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 shutdown_connections 2024-02-28T17:26:41.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.123+0000 7f8181000700 1 -- 172.21.15.1:0/464094143 wait complete. 2024-02-28T17:26:41.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.273+0000 7fd83d800700 1 -- 172.21.15.1:0/3143424403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 msgr2=0x7fd8380fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:41.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.273+0000 7fd83d800700 1 --2- 172.21.15.1:0/3143424403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 0x7fd8380fcf60 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd820004940 tx=0x7fd820009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:41.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:41 smithi001 ceph-mon[112940]: pgmap v1570: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:41.277 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.274+0000 7fd83d800700 1 -- 172.21.15.1:0/3143424403 shutdown_connections 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.274+0000 7fd83d800700 1 --2- 172.21.15.1:0/3143424403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 0x7fd8380fcf60 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.274+0000 7fd83d800700 1 --2- 172.21.15.1:0/3143424403 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd838102f50 0x7fd838103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.274+0000 7fd83d800700 1 -- 172.21.15.1:0/3143424403 >> 172.21.15.1:0/3143424403 conn(0x7fd8380f8ae0 msgr2=0x7fd8380faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.274+0000 7fd83d800700 1 -- 172.21.15.1:0/3143424403 shutdown_connections 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.274+0000 7fd83d800700 1 -- 172.21.15.1:0/3143424403 wait complete. 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.275+0000 7fd83d800700 1 Processor -- start 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.275+0000 7fd83d800700 1 -- start start 2024-02-28T17:26:41.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.275+0000 7fd83d800700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd838102f50 0x7fd838115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.275+0000 7fd83d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 0x7fd8381161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.275+0000 7fd83d800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838116810 con 0x7fd8381038f0 2024-02-28T17:26:41.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.275+0000 7fd83d800700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83810fb40 con 0x7fd838102f50 2024-02-28T17:26:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.276+0000 7fd82e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 0x7fd8381161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.276+0000 7fd82e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 0x7fd8381161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44598/0 (socket says 172.21.15.1:44598) 2024-02-28T17:26:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.276+0000 7fd82e000700 1 -- 172.21.15.1:0/3345197569 learned_addr learned my addr 172.21.15.1:0/3345197569 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.276+0000 7fd836a00700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd838102f50 0x7fd838115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.276+0000 7fd82e000700 1 -- 172.21.15.1:0/3345197569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd838102f50 msgr2=0x7fd838115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:41.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.276+0000 7fd82e000700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd838102f50 0x7fd838115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.276+0000 7fd82e000700 1 -- 172.21.15.1:0/3345197569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd820005040 con 0x7fd8381038f0 2024-02-28T17:26:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.276+0000 7fd82e000700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 0x7fd8381161c0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd820009030 tx=0x7fd820009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.277+0000 7fd82f400700 1 -- 172.21.15.1:0/3345197569 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd82000e040 con 0x7fd8381038f0 2024-02-28T17:26:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.277+0000 7fd82f400700 1 -- 172.21.15.1:0/3345197569 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd82000bbf0 con 0x7fd8381038f0 2024-02-28T17:26:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.277+0000 7fd82f400700 1 -- 172.21.15.1:0/3345197569 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd82001c630 con 0x7fd8381038f0 2024-02-28T17:26:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.277+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83810fdc0 con 0x7fd8381038f0 2024-02-28T17:26:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.277+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd838110300 con 0x7fd8381038f0 2024-02-28T17:26:41.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.278+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8380fc6e0 con 0x7fd8381038f0 2024-02-28T17:26:41.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.279+0000 7fd82f400700 1 -- 172.21.15.1:0/3345197569 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd82000bd60 con 0x7fd8381038f0 2024-02-28T17:26:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.280+0000 7fd82f400700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd81806d7f0 0x7fd81806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.280+0000 7fd82f400700 1 -- 172.21.15.1:0/3345197569 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd82008e1f0 con 0x7fd8381038f0 2024-02-28T17:26:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.280+0000 7fd836a00700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd81806d7f0 0x7fd81806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:41.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.281+0000 7fd836a00700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd81806d7f0 0x7fd81806fca0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd828007fd0 tx=0x7fd828009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:41.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.285+0000 7fd82f400700 1 -- 172.21.15.1:0/3345197569 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd82005b5d0 con 0x7fd8381038f0 2024-02-28T17:26:41.314 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:41 smithi066 ceph-mon[85781]: pgmap v1570: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:41.314 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:26:41.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.585+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd8381111f0 con 0x7fd8381038f0 2024-02-28T17:26:41.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.586+0000 7fd82f400700 1 -- 172.21.15.1:0/3345197569 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd82005b160 con 0x7fd8381038f0 2024-02-28T17:26:41.589 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:26:41.589 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:26:41.589 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:26:41.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:26:41.590 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:26:41.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.589+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd81806d7f0 msgr2=0x7fd81806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:41.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.589+0000 7fd83d800700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd81806d7f0 0x7fd81806fca0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd828007fd0 tx=0x7fd828009040 comp rx=0 tx=0).stop 2024-02-28T17:26:41.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.589+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 msgr2=0x7fd8381161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:41.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.589+0000 7fd83d800700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 0x7fd8381161c0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd820009030 tx=0x7fd820009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:41.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.590+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 shutdown_connections 2024-02-28T17:26:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.590+0000 7fd83d800700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd81806d7f0 0x7fd81806fca0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.590+0000 7fd83d800700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd838102f50 0x7fd838115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.590+0000 7fd83d800700 1 --2- 172.21.15.1:0/3345197569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8381038f0 0x7fd8381161c0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.590+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 >> 172.21.15.1:0/3345197569 conn(0x7fd8380f8ae0 msgr2=0x7fd8380fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.590+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 shutdown_connections 2024-02-28T17:26:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.590+0000 7fd83d800700 1 -- 172.21.15.1:0/3345197569 wait complete. 2024-02-28T17:26:41.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.746+0000 7fccf4400700 1 -- 172.21.15.1:0/394517599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec1038f0 msgr2=0x7fccec0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:41.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.746+0000 7fccf4400700 1 --2- 172.21.15.1:0/394517599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec1038f0 0x7fccec0fcf60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fccd8004940 tx=0x7fccd8009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:41.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.746+0000 7fccf4400700 1 -- 172.21.15.1:0/394517599 shutdown_connections 2024-02-28T17:26:41.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.746+0000 7fccf4400700 1 --2- 172.21.15.1:0/394517599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec1038f0 0x7fccec0fcf60 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.746+0000 7fccf4400700 1 --2- 172.21.15.1:0/394517599 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccec102f50 0x7fccec103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.746+0000 7fccf4400700 1 -- 172.21.15.1:0/394517599 >> 172.21.15.1:0/394517599 conn(0x7fccec0f8ae0 msgr2=0x7fccec0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:41.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.746+0000 7fccf4400700 1 -- 172.21.15.1:0/394517599 shutdown_connections 2024-02-28T17:26:41.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.746+0000 7fccf4400700 1 -- 172.21.15.1:0/394517599 wait complete. 2024-02-28T17:26:41.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.747+0000 7fccf4400700 1 Processor -- start 2024-02-28T17:26:41.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.748+0000 7fccf4400700 1 -- start start 2024-02-28T17:26:41.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.748+0000 7fccf4400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec102f50 0x7fccec115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:41.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.748+0000 7fccf4400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccec1038f0 0x7fccec1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:41.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.748+0000 7fccf4400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec116830 con 0x7fccec102f50 2024-02-28T17:26:41.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.748+0000 7fccf4400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec10fb40 con 0x7fccec1038f0 2024-02-28T17:26:41.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.748+0000 7fccf1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec102f50 0x7fccec115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:41.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.748+0000 7fccf1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec102f50 0x7fccec115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44610/0 (socket says 172.21.15.1:44610) 2024-02-28T17:26:41.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.748+0000 7fccf1a00700 1 -- 172.21.15.1:0/2132011623 learned_addr learned my addr 172.21.15.1:0/2132011623 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:41.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.749+0000 7fccf1000700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccec1038f0 0x7fccec1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.749+0000 7fccf1a00700 1 -- 172.21.15.1:0/2132011623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccec1038f0 msgr2=0x7fccec1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.749+0000 7fccf1a00700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccec1038f0 0x7fccec1161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.749+0000 7fccf1a00700 1 -- 172.21.15.1:0/2132011623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8005040 con 0x7fccec102f50 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.749+0000 7fccf1a00700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec102f50 0x7fccec115ca0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcce0002a50 tx=0x7fcce00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.749+0000 7fcce9600700 1 -- 172.21.15.1:0/2132011623 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce0007cc0 con 0x7fccec102f50 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.749+0000 7fcce9600700 1 -- 172.21.15.1:0/2132011623 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcce000f040 con 0x7fccec102f50 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.749+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec10fe20 con 0x7fccec102f50 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.750+0000 7fcce9600700 1 -- 172.21.15.1:0/2132011623 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce0009640 con 0x7fccec102f50 2024-02-28T17:26:41.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.750+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec110360 con 0x7fccec102f50 2024-02-28T17:26:41.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.751+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec0fc6e0 con 0x7fccec102f50 2024-02-28T17:26:41.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.752+0000 7fcce9600700 1 -- 172.21.15.1:0/2132011623 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fcce00097a0 con 0x7fccec102f50 2024-02-28T17:26:41.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.753+0000 7fcce9600700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fccd406d7f0 0x7fccd406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:41.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.753+0000 7fcce9600700 1 -- 172.21.15.1:0/2132011623 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcce008cc60 con 0x7fccec102f50 2024-02-28T17:26:41.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.753+0000 7fccf1000700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fccd406d7f0 0x7fccd406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:41.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.754+0000 7fccf1000700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fccd406d7f0 0x7fccd406fca0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fccec111540 tx=0x7fccd8009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:41.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:41.758+0000 7fcce9600700 1 -- 172.21.15.1:0/2132011623 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcce0059fc0 con 0x7fccec102f50 2024-02-28T17:26:42.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.030+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fccec111280 con 0x7fccec102f50 2024-02-28T17:26:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.032+0000 7fcce9600700 1 -- 172.21.15.1:0/2132011623 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7fcce0059b50 con 0x7fccec102f50 2024-02-28T17:26:42.035 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:26:42.035 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:26:42.035 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:26:42.036 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:26:42.037 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:26:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:26:42.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fccd406d7f0 msgr2=0x7fccd406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fccd406d7f0 0x7fccd406fca0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fccec111540 tx=0x7fccd8009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:42.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec102f50 msgr2=0x7fccec115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec102f50 0x7fccec115ca0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcce0002a50 tx=0x7fcce00046b0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 shutdown_connections 2024-02-28T17:26:42.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fccd406d7f0 0x7fccd406fca0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccec102f50 0x7fccec115ca0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 --2- 172.21.15.1:0/2132011623 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fccec1038f0 0x7fccec1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.035+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 >> 172.21.15.1:0/2132011623 conn(0x7fccec0f8ae0 msgr2=0x7fccec0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.036+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 shutdown_connections 2024-02-28T17:26:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.036+0000 7fccf4400700 1 -- 172.21.15.1:0/2132011623 wait complete. 2024-02-28T17:26:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:26:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.198+0000 7ffac5200700 1 -- 172.21.15.1:0/3405499956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fdfe0 msgr2=0x7ffac00fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.198+0000 7ffac5200700 1 --2- 172.21.15.1:0/3405499956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fdfe0 0x7ffac00fe3b0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ffab0004970 tx=0x7ffab0009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.198+0000 7ffac5200700 1 -- 172.21.15.1:0/3405499956 shutdown_connections 2024-02-28T17:26:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.198+0000 7ffac5200700 1 --2- 172.21.15.1:0/3405499956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffac00fe8f0 0x7ffac0107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.198+0000 7ffac5200700 1 --2- 172.21.15.1:0/3405499956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fdfe0 0x7ffac00fe3b0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.198+0000 7ffac5200700 1 -- 172.21.15.1:0/3405499956 >> 172.21.15.1:0/3405499956 conn(0x7ffac0071d90 msgr2=0x7ffac0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.199+0000 7ffac5200700 1 -- 172.21.15.1:0/3405499956 shutdown_connections 2024-02-28T17:26:42.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.199+0000 7ffac5200700 1 -- 172.21.15.1:0/3405499956 wait complete. 2024-02-28T17:26:42.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.199+0000 7ffac5200700 1 Processor -- start 2024-02-28T17:26:42.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.199+0000 7ffac5200700 1 -- start start 2024-02-28T17:26:42.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.200+0000 7ffac5200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffac00fdfe0 0x7ffac0115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:42.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.200+0000 7ffac5200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fe8f0 0x7ffac0116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:42.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.200+0000 7ffac5200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac01167a0 con 0x7ffac00fe8f0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.200+0000 7ffac5200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac010faf0 con 0x7ffac00fdfe0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.200+0000 7ffabd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fe8f0 0x7ffac0116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.200+0000 7ffabd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fe8f0 0x7ffac0116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53266/0 (socket says 172.21.15.1:53266) 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.200+0000 7ffabd400700 1 -- 172.21.15.1:0/476173388 learned_addr learned my addr 172.21.15.1:0/476173388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffabde00700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffac00fdfe0 0x7ffac0115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffabd400700 1 -- 172.21.15.1:0/476173388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffac00fdfe0 msgr2=0x7ffac0115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffabd400700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffac00fdfe0 0x7ffac0115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffabd400700 1 -- 172.21.15.1:0/476173388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab0005040 con 0x7ffac00fe8f0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffabd400700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fe8f0 0x7ffac0116150 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00c8c0 tx=0x7ffaac0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffab6000700 1 -- 172.21.15.1:0/476173388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaac00a980 con 0x7ffac00fe8f0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffab6000700 1 -- 172.21.15.1:0/476173388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffaac00ce60 con 0x7ffac00fe8f0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffab6000700 1 -- 172.21.15.1:0/476173388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaac0115e0 con 0x7ffac00fe8f0 2024-02-28T17:26:42.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac010fdd0 con 0x7ffac00fe8f0 2024-02-28T17:26:42.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.201+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac0110310 con 0x7ffac00fe8f0 2024-02-28T17:26:42.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.202+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac0106ac0 con 0x7ffac00fe8f0 2024-02-28T17:26:42.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.204+0000 7ffab6000700 1 -- 172.21.15.1:0/476173388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffaac012400 con 0x7ffac00fe8f0 2024-02-28T17:26:42.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.205+0000 7ffab6000700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffaa406d7f0 0x7ffaa406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:42.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.205+0000 7ffab6000700 1 -- 172.21.15.1:0/476173388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffaac08cf70 con 0x7ffac00fe8f0 2024-02-28T17:26:42.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.205+0000 7ffabde00700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffaa406d7f0 0x7ffaa406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:42.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.206+0000 7ffabde00700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffaa406d7f0 0x7ffaa406fca0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ffab0004ee0 tx=0x7ffab0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:42.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.210+0000 7ffab6000700 1 -- 172.21.15.1:0/476173388 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffaac056820 con 0x7ffac00fe8f0 2024-02-28T17:26:42.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:42 smithi001 ceph-mon[112940]: from='client.36748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:42.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:42 smithi001 ceph-mon[112940]: from='client.36752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:42.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:42 smithi001 ceph-mon[112940]: from='client.36756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:42.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:42 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3345197569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:26:42.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.444+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffac004bed0 con 0x7ffaa406d7f0 2024-02-28T17:26:42.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.447+0000 7ffab6000700 1 -- 172.21.15.1:0/476173388 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffac004bed0 con 0x7ffaa406d7f0 2024-02-28T17:26:42.450 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:26:42.450 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:26:42.451 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.449+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffaa406d7f0 msgr2=0x7ffaa406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.449+0000 7ffac5200700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffaa406d7f0 0x7ffaa406fca0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ffab0004ee0 tx=0x7ffab0009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.449+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fe8f0 msgr2=0x7ffac0116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.449+0000 7ffac5200700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fe8f0 0x7ffac0116150 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00c8c0 tx=0x7ffaac0044c0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.449+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 shutdown_connections 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.450+0000 7ffac5200700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffaa406d7f0 0x7ffaa406fca0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.450+0000 7ffac5200700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffac00fdfe0 0x7ffac0115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.450+0000 7ffac5200700 1 --2- 172.21.15.1:0/476173388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac00fe8f0 0x7ffac0116150 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.450+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 >> 172.21.15.1:0/476173388 conn(0x7ffac0071d90 msgr2=0x7ffac0103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.450+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 shutdown_connections 2024-02-28T17:26:42.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.450+0000 7ffac5200700 1 -- 172.21.15.1:0/476173388 wait complete. 2024-02-28T17:26:42.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:42 smithi066 ceph-mon[85781]: from='client.36748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:42.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:42 smithi066 ceph-mon[85781]: from='client.36752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:42.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:42 smithi066 ceph-mon[85781]: from='client.36756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:42.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:42 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3345197569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:26:42.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.591+0000 7f9915400700 1 -- 172.21.15.1:0/779995699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fe710 msgr2=0x7f9910106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.591+0000 7f9915400700 1 --2- 172.21.15.1:0/779995699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fe710 0x7f9910106d50 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f98fc004940 tx=0x7f98fc009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:42.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.592+0000 7f9915400700 1 -- 172.21.15.1:0/779995699 shutdown_connections 2024-02-28T17:26:42.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.592+0000 7f9915400700 1 --2- 172.21.15.1:0/779995699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fe710 0x7f9910106d50 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.592+0000 7f9915400700 1 --2- 172.21.15.1:0/779995699 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99100fdd70 0x7f99100fe140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.592+0000 7f9915400700 1 -- 172.21.15.1:0/779995699 >> 172.21.15.1:0/779995699 conn(0x7f99100f77e0 msgr2=0x7f99100f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:42.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.592+0000 7f9915400700 1 -- 172.21.15.1:0/779995699 shutdown_connections 2024-02-28T17:26:42.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.592+0000 7f9915400700 1 -- 172.21.15.1:0/779995699 wait complete. 2024-02-28T17:26:42.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.593+0000 7f9915400700 1 Processor -- start 2024-02-28T17:26:42.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.593+0000 7f9915400700 1 -- start start 2024-02-28T17:26:42.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f9915400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fdd70 0x7f9910115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:42.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f9915400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99100fe710 0x7f9910115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:42.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f9915400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99101165a0 con 0x7f99100fdd70 2024-02-28T17:26:42.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f9915400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991010f8d0 con 0x7f99100fe710 2024-02-28T17:26:42.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f990fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fdd70 0x7f9910115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:42.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f990fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fdd70 0x7f9910115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53276/0 (socket says 172.21.15.1:53276) 2024-02-28T17:26:42.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f990fe00700 1 -- 172.21.15.1:0/4075026180 learned_addr learned my addr 172.21.15.1:0/4075026180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:26:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f990f400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99100fe710 0x7f9910115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.594+0000 7f990fe00700 1 -- 172.21.15.1:0/4075026180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99100fe710 msgr2=0x7f9910115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f990fe00700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99100fe710 0x7f9910115f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f990fe00700 1 -- 172.21.15.1:0/4075026180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc005040 con 0x7f99100fdd70 2024-02-28T17:26:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f990f400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99100fe710 0x7f9910115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:26:42.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f990fe00700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fdd70 0x7f9910115a10 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f990000c830 tx=0x7f9900004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f9907e00700 1 -- 172.21.15.1:0/4075026180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f990000a760 con 0x7f99100fdd70 2024-02-28T17:26:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f9907e00700 1 -- 172.21.15.1:0/4075026180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f990000ad90 con 0x7f99100fdd70 2024-02-28T17:26:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f9907e00700 1 -- 172.21.15.1:0/4075026180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99000113a0 con 0x7f99100fdd70 2024-02-28T17:26:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991010fbb0 con 0x7f99100fdd70 2024-02-28T17:26:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.595+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99101100f0 con 0x7f99100fdd70 2024-02-28T17:26:42.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.596+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99101067c0 con 0x7f99100fdd70 2024-02-28T17:26:42.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.598+0000 7f9907e00700 1 -- 172.21.15.1:0/4075026180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9900012400 con 0x7f99100fdd70 2024-02-28T17:26:42.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.598+0000 7f9907e00700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98f406d7e0 0x7f98f406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:26:42.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.599+0000 7f9907e00700 1 -- 172.21.15.1:0/4075026180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f990008cf00 con 0x7f99100fdd70 2024-02-28T17:26:42.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.599+0000 7f990f400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98f406d7e0 0x7f98f406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:26:42.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.599+0000 7f990f400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98f406d7e0 0x7f98f406fc90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f98fc004560 tx=0x7f98fc009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:26:42.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.603+0000 7f9907e00700 1 -- 172.21.15.1:0/4075026180 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f990005a2e0 con 0x7f99100fdd70 2024-02-28T17:26:42.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.903+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f991004bed0 con 0x7f99100fdd70 2024-02-28T17:26:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.904+0000 7f9907e00700 1 -- 172.21.15.1:0/4075026180 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f9900059e70 con 0x7f99100fdd70 2024-02-28T17:26:42.907 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:26:42.907 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:26:42.908 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:26:42.909 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.906+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98f406d7e0 msgr2=0x7f98f406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.906+0000 7f9915400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98f406d7e0 0x7f98f406fc90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f98fc004560 tx=0x7f98fc009e30 comp rx=0 tx=0).stop 2024-02-28T17:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.906+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fdd70 msgr2=0x7f9910115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.907+0000 7f9915400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fdd70 0x7f9910115a10 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f990000c830 tx=0x7f9900004470 comp rx=0 tx=0).stop 2024-02-28T17:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.907+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 shutdown_connections 2024-02-28T17:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.907+0000 7f9915400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f98f406d7e0 0x7f98f406fc90 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.907+0000 7f9915400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99100fdd70 0x7f9910115a10 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.907+0000 7f9915400700 1 --2- 172.21.15.1:0/4075026180 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99100fe710 0x7f9910115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:26:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.907+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 >> 172.21.15.1:0/4075026180 conn(0x7f99100f77e0 msgr2=0x7f99100fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:26:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.907+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 shutdown_connections 2024-02-28T17:26:42.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:26:42.907+0000 7f9915400700 1 -- 172.21.15.1:0/4075026180 wait complete. 2024-02-28T17:26:43.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:43 smithi001 ceph-mon[112940]: pgmap v1571: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:43.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:43 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2132011623' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:26:43.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:43 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4075026180' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:26:43.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:43 smithi066 ceph-mon[85781]: pgmap v1571: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:43.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:43 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2132011623' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:26:43.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:43 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4075026180' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:26:44.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:44 smithi066 ceph-mon[85781]: from='client.36768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:44.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:44 smithi001 ceph-mon[112940]: from='client.36768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:26:45.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:45 smithi066 ceph-mon[85781]: pgmap v1572: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:45.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:45 smithi001 ceph-mon[112940]: pgmap v1572: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:47.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:47 smithi066 ceph-mon[85781]: pgmap v1573: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:47.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:47 smithi001 ceph-mon[112940]: pgmap v1573: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:49.336 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:49 smithi001 ceph-mon[112940]: pgmap v1574: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:49.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:49 smithi066 ceph-mon[85781]: pgmap v1574: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:51.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:51 smithi066 ceph-mon[85781]: pgmap v1575: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:51.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:51 smithi001 ceph-mon[112940]: pgmap v1575: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:53.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:53 smithi066 ceph-mon[85781]: pgmap v1576: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:53.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:53 smithi001 ceph-mon[112940]: pgmap v1576: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:55.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:55 smithi066 ceph-mon[85781]: pgmap v1577: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:55.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:55 smithi001 ceph-mon[112940]: pgmap v1577: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:26:56.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:26:57.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:57 smithi066 ceph-mon[85781]: pgmap v1578: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:57.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:57 smithi001 ceph-mon[112940]: pgmap v1578: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:59.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:26:59 smithi066 ceph-mon[85781]: pgmap v1579: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:26:59.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:26:59 smithi001 ceph-mon[112940]: pgmap v1579: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:01.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:01 smithi066 ceph-mon[85781]: pgmap v1580: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:01.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:01 smithi001 ceph-mon[112940]: pgmap v1580: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:03.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:03 smithi066 ceph-mon[85781]: pgmap v1581: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:03.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:03 smithi001 ceph-mon[112940]: pgmap v1581: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:05.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:05 smithi066 ceph-mon[85781]: pgmap v1582: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:05.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:05 smithi001 ceph-mon[112940]: pgmap v1582: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:07.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:07 smithi066 ceph-mon[85781]: pgmap v1583: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:07.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:07 smithi001 ceph-mon[112940]: pgmap v1583: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:09.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:09 smithi066 ceph-mon[85781]: pgmap v1584: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:09.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:09 smithi001 ceph-mon[112940]: pgmap v1584: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:11.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:11 smithi066 ceph-mon[85781]: pgmap v1585: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:11.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:27:11.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:11 smithi001 ceph-mon[112940]: pgmap v1585: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:11.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:27:13.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.077+0000 7ff894a00700 1 -- 172.21.15.1:0/1329521669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 msgr2=0x7ff890070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.077+0000 7ff894a00700 1 --2- 172.21.15.1:0/1329521669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 0x7ff890070330 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7ff880009d10 tx=0x7ff880004d10 comp rx=0 tx=0).stop 2024-02-28T17:27:13.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.078+0000 7ff894a00700 1 -- 172.21.15.1:0/1329521669 shutdown_connections 2024-02-28T17:27:13.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.078+0000 7ff894a00700 1 --2- 172.21.15.1:0/1329521669 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff890070870 0x7ff890070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.078+0000 7ff894a00700 1 --2- 172.21.15.1:0/1329521669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 0x7ff890070330 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.078+0000 7ff894a00700 1 -- 172.21.15.1:0/1329521669 >> 172.21.15.1:0/1329521669 conn(0x7ff8900f8ae0 msgr2=0x7ff8900faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:13.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.078+0000 7ff894a00700 1 -- 172.21.15.1:0/1329521669 shutdown_connections 2024-02-28T17:27:13.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.078+0000 7ff894a00700 1 -- 172.21.15.1:0/1329521669 wait complete. 2024-02-28T17:27:13.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.079+0000 7ff894a00700 1 Processor -- start 2024-02-28T17:27:13.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.079+0000 7ff894a00700 1 -- start start 2024-02-28T17:27:13.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.079+0000 7ff894a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff890070870 0x7ff89011a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.079+0000 7ff894a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 0x7ff89011a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.079+0000 7ff894a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89011aeb0 con 0x7ff890071f60 2024-02-28T17:27:13.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.079+0000 7ff894a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890114130 con 0x7ff890070870 2024-02-28T17:27:13.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.080+0000 7ff88d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 0x7ff89011a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.080+0000 7ff88d400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 0x7ff89011a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47398/0 (socket says 172.21.15.1:47398) 2024-02-28T17:27:13.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.080+0000 7ff88d400700 1 -- 172.21.15.1:0/3169993051 learned_addr learned my addr 172.21.15.1:0/3169993051 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:13.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.080+0000 7ff88de00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff890070870 0x7ff89011a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.080+0000 7ff88d400700 1 -- 172.21.15.1:0/3169993051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff890070870 msgr2=0x7ff89011a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.080+0000 7ff88d400700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff890070870 0x7ff89011a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.081+0000 7ff88d400700 1 -- 172.21.15.1:0/3169993051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880009940 con 0x7ff890071f60 2024-02-28T17:27:13.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.081+0000 7ff88de00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff890070870 0x7ff89011a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:27:13.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.081+0000 7ff88d400700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 0x7ff89011a7d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00b790 tx=0x7ff87c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:13.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.081+0000 7ff886000700 1 -- 172.21.15.1:0/3169993051 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff87c00f770 con 0x7ff890071f60 2024-02-28T17:27:13.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.081+0000 7ff886000700 1 -- 172.21.15.1:0/3169993051 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff87c00fda0 con 0x7ff890071f60 2024-02-28T17:27:13.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.081+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8901143b0 con 0x7ff890071f60 2024-02-28T17:27:13.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.081+0000 7ff886000700 1 -- 172.21.15.1:0/3169993051 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff87c00d310 con 0x7ff890071f60 2024-02-28T17:27:13.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.081+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8901148f0 con 0x7ff890071f60 2024-02-28T17:27:13.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.083+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8900fe400 con 0x7ff890071f60 2024-02-28T17:27:13.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.084+0000 7ff886000700 1 -- 172.21.15.1:0/3169993051 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff87c010420 con 0x7ff890071f60 2024-02-28T17:27:13.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.084+0000 7ff886000700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff87406d7f0 0x7ff87406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.085+0000 7ff886000700 1 -- 172.21.15.1:0/3169993051 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff87c08cef0 con 0x7ff890071f60 2024-02-28T17:27:13.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.085+0000 7ff88de00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff87406d7f0 0x7ff87406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.086+0000 7ff88de00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff87406d7f0 0x7ff87406fca0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff880009ef0 tx=0x7ff880004d10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:13.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.090+0000 7ff886000700 1 -- 172.21.15.1:0/3169993051 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff87c05a2d0 con 0x7ff890071f60 2024-02-28T17:27:13.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.333+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff890115870 con 0x7ff87406d7f0 2024-02-28T17:27:13.336 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:13 smithi001 ceph-mon[112940]: pgmap v1586: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.336+0000 7ff886000700 1 -- 172.21.15.1:0/3169993051 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff890115870 con 0x7ff87406d7f0 2024-02-28T17:27:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff87406d7f0 msgr2=0x7ff87406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff87406d7f0 0x7ff87406fca0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff880009ef0 tx=0x7ff880004d10 comp rx=0 tx=0).stop 2024-02-28T17:27:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 msgr2=0x7ff89011a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 0x7ff89011a7d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00b790 tx=0x7ff87c009250 comp rx=0 tx=0).stop 2024-02-28T17:27:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 shutdown_connections 2024-02-28T17:27:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff87406d7f0 0x7ff87406fca0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff890070870 0x7ff89011a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 --2- 172.21.15.1:0/3169993051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff890071f60 0x7ff89011a7d0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.339+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 >> 172.21.15.1:0/3169993051 conn(0x7ff8900f8ae0 msgr2=0x7ff8900fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.340+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 shutdown_connections 2024-02-28T17:27:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.340+0000 7ff894a00700 1 -- 172.21.15.1:0/3169993051 wait complete. 2024-02-28T17:27:13.356 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.485+0000 7f6430000700 1 -- 172.21.15.1:0/3436617440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fe8f0 msgr2=0x7f6428107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.485+0000 7f6430000700 1 --2- 172.21.15.1:0/3436617440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fe8f0 0x7f6428107050 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f6414004970 tx=0x7f6414009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.485+0000 7f6430000700 1 -- 172.21.15.1:0/3436617440 shutdown_connections 2024-02-28T17:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.485+0000 7f6430000700 1 --2- 172.21.15.1:0/3436617440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fe8f0 0x7f6428107050 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.485+0000 7f6430000700 1 --2- 172.21.15.1:0/3436617440 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f64280fdfe0 0x7f64280fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.486+0000 7f6430000700 1 -- 172.21.15.1:0/3436617440 >> 172.21.15.1:0/3436617440 conn(0x7f6428071d90 msgr2=0x7f6428072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.486+0000 7f6430000700 1 -- 172.21.15.1:0/3436617440 shutdown_connections 2024-02-28T17:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.486+0000 7f6430000700 1 -- 172.21.15.1:0/3436617440 wait complete. 2024-02-28T17:27:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.486+0000 7f6430000700 1 Processor -- start 2024-02-28T17:27:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.487+0000 7f6430000700 1 -- start start 2024-02-28T17:27:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.487+0000 7f6430000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fdfe0 0x7f6428115dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.487+0000 7f6430000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6428116310 0x7f642810fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.487+0000 7f6430000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6428116890 con 0x7f64280fdfe0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.487+0000 7f6430000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6428116a00 con 0x7f6428116310 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.487+0000 7f642d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fdfe0 0x7f6428115dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.487+0000 7f642d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fdfe0 0x7f6428115dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47412/0 (socket says 172.21.15.1:47412) 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.487+0000 7f642d600700 1 -- 172.21.15.1:0/1765039479 learned_addr learned my addr 172.21.15.1:0/1765039479 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f642cc00700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6428116310 0x7f642810fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f642d600700 1 -- 172.21.15.1:0/1765039479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6428116310 msgr2=0x7f642810fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f642d600700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6428116310 0x7f642810fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f642d600700 1 -- 172.21.15.1:0/1765039479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414005040 con 0x7f64280fdfe0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f642d600700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fdfe0 0x7f6428115dd0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f641c00c830 tx=0x7f641c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f6425600700 1 -- 172.21.15.1:0/1765039479 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641c00a760 con 0x7f64280fdfe0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6428110270 con 0x7f64280fdfe0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f6425600700 1 -- 172.21.15.1:0/1765039479 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f641c00ad90 con 0x7f64280fdfe0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.488+0000 7f6425600700 1 -- 172.21.15.1:0/1765039479 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641c0113a0 con 0x7f64280fdfe0 2024-02-28T17:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.489+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64281107b0 con 0x7f64280fdfe0 2024-02-28T17:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.490+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6428110a90 con 0x7f64280fdfe0 2024-02-28T17:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.490+0000 7f6425600700 1 -- 172.21.15.1:0/1765039479 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f641c012400 con 0x7f64280fdfe0 2024-02-28T17:27:13.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.491+0000 7f6425600700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f641006d7f0 0x7f641006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.492+0000 7f6425600700 1 -- 172.21.15.1:0/1765039479 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f641c08cfd0 con 0x7f64280fdfe0 2024-02-28T17:27:13.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.492+0000 7f642cc00700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f641006d7f0 0x7f641006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.492+0000 7f642cc00700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f641006d7f0 0x7f641006fca0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f641400b370 tx=0x7f6414009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:13.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.496+0000 7f6425600700 1 -- 172.21.15.1:0/1765039479 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f641c0c0d20 con 0x7f64280fdfe0 2024-02-28T17:27:13.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:13 smithi066 ceph-mon[85781]: pgmap v1586: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:13.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.727+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f642810a030 con 0x7f641006d7f0 2024-02-28T17:27:13.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.731+0000 7f6425600700 1 -- 172.21.15.1:0/1765039479 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f642810a030 con 0x7f641006d7f0 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.733+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f641006d7f0 msgr2=0x7f641006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.733+0000 7f6430000700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f641006d7f0 0x7f641006fca0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f641400b370 tx=0x7f6414009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.733+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fdfe0 msgr2=0x7f6428115dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.733+0000 7f6430000700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fdfe0 0x7f6428115dd0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f641c00c830 tx=0x7f641c004470 comp rx=0 tx=0).stop 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.734+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 shutdown_connections 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.734+0000 7f6430000700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f641006d7f0 0x7f641006fca0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.734+0000 7f6430000700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64280fdfe0 0x7f6428115dd0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.734+0000 7f6430000700 1 --2- 172.21.15.1:0/1765039479 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6428116310 0x7f642810fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.734+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 >> 172.21.15.1:0/1765039479 conn(0x7f6428071d90 msgr2=0x7f6428103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:13.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.734+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 shutdown_connections 2024-02-28T17:27:13.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.734+0000 7f6430000700 1 -- 172.21.15.1:0/1765039479 wait complete. 2024-02-28T17:27:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.880+0000 7f5d1fc00700 1 -- 172.21.15.1:0/1197791266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fc6e0 msgr2=0x7f5d18109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.880+0000 7f5d1fc00700 1 --2- 172.21.15.1:0/1197791266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fc6e0 0x7f5d18109440 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f5d04004940 tx=0x7f5d04009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.880+0000 7f5d1fc00700 1 -- 172.21.15.1:0/1197791266 shutdown_connections 2024-02-28T17:27:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.880+0000 7f5d1fc00700 1 --2- 172.21.15.1:0/1197791266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fc6e0 0x7f5d18109440 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.880+0000 7f5d1fc00700 1 --2- 172.21.15.1:0/1197791266 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d180fbdd0 0x7f5d180fc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.880+0000 7f5d1fc00700 1 -- 172.21.15.1:0/1197791266 >> 172.21.15.1:0/1197791266 conn(0x7f5d18071d90 msgr2=0x7f5d18072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:13.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.880+0000 7f5d1fc00700 1 -- 172.21.15.1:0/1197791266 shutdown_connections 2024-02-28T17:27:13.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.880+0000 7f5d1fc00700 1 -- 172.21.15.1:0/1197791266 wait complete. 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.881+0000 7f5d1fc00700 1 Processor -- start 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.881+0000 7f5d1fc00700 1 -- start start 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.881+0000 7f5d1fc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fbdd0 0x7f5d18115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.881+0000 7f5d1fc00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d180fc6e0 0x7f5d18116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.881+0000 7f5d1fc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d181168d0 con 0x7f5d180fbdd0 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.881+0000 7f5d1fc00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1810fbe0 con 0x7f5d180fc6e0 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.882+0000 7f5d1d200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fbdd0 0x7f5d18115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.882+0000 7f5d1d200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fbdd0 0x7f5d18115d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47440/0 (socket says 172.21.15.1:47440) 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.882+0000 7f5d1d200700 1 -- 172.21.15.1:0/4058426783 learned_addr learned my addr 172.21.15.1:0/4058426783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.882+0000 7f5d17e00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d180fc6e0 0x7f5d18116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.882+0000 7f5d1d200700 1 -- 172.21.15.1:0/4058426783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d180fc6e0 msgr2=0x7f5d18116280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.882+0000 7f5d1d200700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d180fc6e0 0x7f5d18116280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:13.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.882+0000 7f5d1d200700 1 -- 172.21.15.1:0/4058426783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d04005040 con 0x7f5d180fbdd0 2024-02-28T17:27:13.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.883+0000 7f5d1d200700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fbdd0 0x7f5d18115d40 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c002a50 tx=0x7f5d0c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:13.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.883+0000 7f5d14c00700 1 -- 172.21.15.1:0/4058426783 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d0c007cc0 con 0x7f5d180fbdd0 2024-02-28T17:27:13.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.883+0000 7f5d14c00700 1 -- 172.21.15.1:0/4058426783 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d0c00f040 con 0x7f5d180fbdd0 2024-02-28T17:27:13.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.883+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1810fec0 con 0x7f5d180fbdd0 2024-02-28T17:27:13.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.883+0000 7f5d14c00700 1 -- 172.21.15.1:0/4058426783 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d0c009640 con 0x7f5d180fbdd0 2024-02-28T17:27:13.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.883+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d18110400 con 0x7f5d180fbdd0 2024-02-28T17:27:13.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.884+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d18106b70 con 0x7f5d180fbdd0 2024-02-28T17:27:13.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.885+0000 7f5d14c00700 1 -- 172.21.15.1:0/4058426783 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5d0c0097a0 con 0x7f5d180fbdd0 2024-02-28T17:27:13.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.886+0000 7f5d14c00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d0006d7f0 0x7f5d0006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:13.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.886+0000 7f5d14c00700 1 -- 172.21.15.1:0/4058426783 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5d0c08cbc0 con 0x7f5d180fbdd0 2024-02-28T17:27:13.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.886+0000 7f5d17e00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d0006d7f0 0x7f5d0006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:13.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.887+0000 7f5d17e00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d0006d7f0 0x7f5d0006fca0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5d04004890 tx=0x7f5d04009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:13.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:13.891+0000 7f5d14c00700 1 -- 172.21.15.1:0/4058426783 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5d0c059fc0 con 0x7f5d180fbdd0 2024-02-28T17:27:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.129+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d1804bf40 con 0x7f5d0006d7f0 2024-02-28T17:27:14.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.141+0000 7f5d14c00700 1 -- 172.21.15.1:0/4058426783 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5d1804bf40 con 0x7f5d0006d7f0 2024-02-28T17:27:14.143 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:27:14.143 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (53m) 14m ago 67m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:27:14.143 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (50m) 14m ago 67m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:27:14.143 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (49m) 14m ago 66m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:27:14.143 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (52m) 14m ago 67m 77.6M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (45m) 14m ago 60m 21.9M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (45m) 14m ago 60m 21.7M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (14m) 14m ago 60m 10.8M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (30m) 14m ago 60m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (55m) 14m ago 69m 515M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (54m) 14m ago 66m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (51m) 14m ago 69m 82.9M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (51m) 14m ago 66m 55.1M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (54m) 14m ago 67m 8460k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (54m) 14m ago 66m 8460k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:27:14.144 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (49m) 14m ago 65m 90.9M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:27:14.145 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (48m) 14m ago 64m 96.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:27:14.145 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (48m) 14m ago 64m 86.4M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:27:14.145 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (47m) 14m ago 63m 106M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:27:14.145 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (47m) 14m ago 62m 87.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:27:14.145 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (46m) 14m ago 62m 86.2M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:27:14.145 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (53m) 14m ago 66m 61.5M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.143+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d0006d7f0 msgr2=0x7f5d0006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d0006d7f0 0x7f5d0006fca0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5d04004890 tx=0x7f5d04009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fbdd0 msgr2=0x7f5d18115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fbdd0 0x7f5d18115d40 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c002a50 tx=0x7f5d0c0046b0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 shutdown_connections 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5d0006d7f0 0x7f5d0006fca0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d180fbdd0 0x7f5d18115d40 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 --2- 172.21.15.1:0/4058426783 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5d180fc6e0 0x7f5d18116280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 >> 172.21.15.1:0/4058426783 conn(0x7f5d18071d90 msgr2=0x7f5d180fa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 shutdown_connections 2024-02-28T17:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.144+0000 7f5d1fc00700 1 -- 172.21.15.1:0/4058426783 wait complete. 2024-02-28T17:27:14.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.299+0000 7fd27de00700 1 -- 172.21.15.1:0/3283754247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 msgr2=0x7fd2780fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:14.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.299+0000 7fd27de00700 1 --2- 172.21.15.1:0/3283754247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 0x7fd2780fe3b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd268004970 tx=0x7fd268009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:14.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.299+0000 7fd27de00700 1 -- 172.21.15.1:0/3283754247 shutdown_connections 2024-02-28T17:27:14.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.299+0000 7fd27de00700 1 --2- 172.21.15.1:0/3283754247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2780fe8f0 0x7fd278107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.299+0000 7fd27de00700 1 --2- 172.21.15.1:0/3283754247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 0x7fd2780fe3b0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.299+0000 7fd27de00700 1 -- 172.21.15.1:0/3283754247 >> 172.21.15.1:0/3283754247 conn(0x7fd278071d90 msgr2=0x7fd278072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:14.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.299+0000 7fd27de00700 1 -- 172.21.15.1:0/3283754247 shutdown_connections 2024-02-28T17:27:14.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.299+0000 7fd27de00700 1 -- 172.21.15.1:0/3283754247 wait complete. 2024-02-28T17:27:14.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.300+0000 7fd27de00700 1 Processor -- start 2024-02-28T17:27:14.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.300+0000 7fd27de00700 1 -- start start 2024-02-28T17:27:14.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.300+0000 7fd27de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 0x7fd2780fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:14.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd27de00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2780fe8f0 0x7fd2780fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:14.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd27de00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780fc9e0 con 0x7fd2780fdfe0 2024-02-28T17:27:14.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd27de00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780fcb50 con 0x7fd2780fe8f0 2024-02-28T17:27:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd276a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 0x7fd2780fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd276a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 0x7fd2780fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47458/0 (socket says 172.21.15.1:47458) 2024-02-28T17:27:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd276a00700 1 -- 172.21.15.1:0/1603053286 learned_addr learned my addr 172.21.15.1:0/1603053286 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd276000700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2780fe8f0 0x7fd2780fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd276a00700 1 -- 172.21.15.1:0/1603053286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2780fe8f0 msgr2=0x7fd2780fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd276a00700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2780fe8f0 0x7fd2780fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.301+0000 7fd276a00700 1 -- 172.21.15.1:0/1603053286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268005040 con 0x7fd2780fdfe0 2024-02-28T17:27:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.302+0000 7fd276a00700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 0x7fd2780fbea0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd268000c00 tx=0x7fd268009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:14.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.302+0000 7fd26ea00700 1 -- 172.21.15.1:0/1603053286 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd26800e040 con 0x7fd2780fdfe0 2024-02-28T17:27:14.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.302+0000 7fd26ea00700 1 -- 172.21.15.1:0/1603053286 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd268004290 con 0x7fd2780fdfe0 2024-02-28T17:27:14.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.302+0000 7fd26ea00700 1 -- 172.21.15.1:0/1603053286 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd26801c630 con 0x7fd2780fdfe0 2024-02-28T17:27:14.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.302+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd278119c90 con 0x7fd2780fdfe0 2024-02-28T17:27:14.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.302+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27811a0f0 con 0x7fd2780fdfe0 2024-02-28T17:27:14.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.304+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2781046c0 con 0x7fd2780fdfe0 2024-02-28T17:27:14.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.304+0000 7fd26ea00700 1 -- 172.21.15.1:0/1603053286 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd268013980 con 0x7fd2780fdfe0 2024-02-28T17:27:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.305+0000 7fd26ea00700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25c06d7e0 0x7fd25c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.305+0000 7fd26ea00700 1 -- 172.21.15.1:0/1603053286 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd26808e1d0 con 0x7fd2780fdfe0 2024-02-28T17:27:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.306+0000 7fd276000700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25c06d7e0 0x7fd25c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.306+0000 7fd276000700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25c06d7e0 0x7fd25c06fc90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd2780fd8a0 tx=0x7fd264009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.310+0000 7fd26ea00700 1 -- 172.21.15.1:0/1603053286 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd268057a80 con 0x7fd2780fdfe0 2024-02-28T17:27:14.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.623+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd2780fd5e0 con 0x7fd2780fdfe0 2024-02-28T17:27:14.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.624+0000 7fd26ea00700 1 -- 172.21.15.1:0/1603053286 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd26805b140 con 0x7fd2780fdfe0 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:27:14.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:27:14.628 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:27:14.628 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:27:14.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:27:14.628 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25c06d7e0 msgr2=0x7fd25c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25c06d7e0 0x7fd25c06fc90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd2780fd8a0 tx=0x7fd264009040 comp rx=0 tx=0).stop 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 msgr2=0x7fd2780fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 0x7fd2780fbea0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd268000c00 tx=0x7fd268009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 shutdown_connections 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd25c06d7e0 0x7fd25c06fc90 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2780fdfe0 0x7fd2780fbea0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 --2- 172.21.15.1:0/1603053286 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd2780fe8f0 0x7fd2780fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 >> 172.21.15.1:0/1603053286 conn(0x7fd278071d90 msgr2=0x7fd278101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 shutdown_connections 2024-02-28T17:27:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.627+0000 7fd27de00700 1 -- 172.21.15.1:0/1603053286 wait complete. 2024-02-28T17:27:14.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.773+0000 7f97fd000700 1 -- 172.21.15.1:0/646911956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 msgr2=0x7f97f80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.773+0000 7f97fd000700 1 --2- 172.21.15.1:0/646911956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 0x7f97f80fe3b0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f97e8004970 tx=0x7f97e8009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.773+0000 7f97fd000700 1 -- 172.21.15.1:0/646911956 shutdown_connections 2024-02-28T17:27:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.773+0000 7f97fd000700 1 --2- 172.21.15.1:0/646911956 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f80fe8f0 0x7f97f8107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.773+0000 7f97fd000700 1 --2- 172.21.15.1:0/646911956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 0x7f97f80fe3b0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.773+0000 7f97fd000700 1 -- 172.21.15.1:0/646911956 >> 172.21.15.1:0/646911956 conn(0x7f97f8071d90 msgr2=0x7f97f8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.773+0000 7f97fd000700 1 -- 172.21.15.1:0/646911956 shutdown_connections 2024-02-28T17:27:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.773+0000 7f97fd000700 1 -- 172.21.15.1:0/646911956 wait complete. 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.774+0000 7f97fd000700 1 Processor -- start 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.774+0000 7f97fd000700 1 -- start start 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97fd000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 0x7f97f8115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97fd000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f80fe8f0 0x7f97f8116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97fd000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8116790 con 0x7f97f80fdfe0 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97fd000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f810fae0 con 0x7f97f80fe8f0 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97f5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 0x7f97f8115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97f5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 0x7f97f8115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47472/0 (socket says 172.21.15.1:47472) 2024-02-28T17:27:14.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97f5e00700 1 -- 172.21.15.1:0/407149664 learned_addr learned my addr 172.21.15.1:0/407149664 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97efe00700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f80fe8f0 0x7f97f8116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97f5e00700 1 -- 172.21.15.1:0/407149664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f80fe8f0 msgr2=0x7f97f8116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97f5e00700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f80fe8f0 0x7f97f8116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.775+0000 7f97f5e00700 1 -- 172.21.15.1:0/407149664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e8005040 con 0x7f97f80fdfe0 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.776+0000 7f97efe00700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f80fe8f0 0x7f97f8116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.776+0000 7f97f5e00700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 0x7f97f8115c00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f97e8000c00 tx=0x7f97e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.776+0000 7f97ee000700 1 -- 172.21.15.1:0/407149664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e800e040 con 0x7f97f80fdfe0 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.776+0000 7f97ee000700 1 -- 172.21.15.1:0/407149664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97e800bce0 con 0x7f97f80fdfe0 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.776+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f810fd60 con 0x7f97f80fdfe0 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.776+0000 7f97ee000700 1 -- 172.21.15.1:0/407149664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e801c630 con 0x7f97f80fdfe0 2024-02-28T17:27:14.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.776+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f81102a0 con 0x7f97f80fdfe0 2024-02-28T17:27:14.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.777+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f8106b20 con 0x7f97f80fdfe0 2024-02-28T17:27:14.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.778+0000 7f97ee000700 1 -- 172.21.15.1:0/407149664 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f97e80139f0 con 0x7f97f80fdfe0 2024-02-28T17:27:14.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.780+0000 7f97ee000700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97dc06d7e0 0x7f97dc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:14.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.780+0000 7f97ee000700 1 -- 172.21.15.1:0/407149664 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f97e808e340 con 0x7f97f80fdfe0 2024-02-28T17:27:14.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.780+0000 7f97efe00700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97dc06d7e0 0x7f97dc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:14.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.781+0000 7f97efe00700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97dc06d7e0 0x7f97dc06fc90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009fd0 tx=0x7f97e000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:14.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:14.784+0000 7f97ee000700 1 -- 172.21.15.1:0/407149664 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f97e8057b70 con 0x7f97f80fdfe0 2024-02-28T17:27:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.046+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f97f804bed0 con 0x7f97f80fdfe0 2024-02-28T17:27:15.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.047+0000 7f97ee000700 1 -- 172.21.15.1:0/407149664 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f97e8004de0 con 0x7f97f80fdfe0 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:e28 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 19 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:27:15.050 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T16:41:05.748872+0000 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:27:15.051 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:up {} 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:failed 0,1 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:27:15.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:27:15.053 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{-1:34438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:27:15.053 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{-1:34470} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:27:15.053 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:27:15.053 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97dc06d7e0 msgr2=0x7f97dc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97dc06d7e0 0x7f97dc06fc90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009fd0 tx=0x7f97e000b040 comp rx=0 tx=0).stop 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 msgr2=0x7f97f8115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 0x7f97f8115c00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f97e8000c00 tx=0x7f97e8009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 shutdown_connections 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f97dc06d7e0 0x7f97dc06fc90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97f80fdfe0 0x7f97f8115c00 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 --2- 172.21.15.1:0/407149664 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f97f80fe8f0 0x7f97f8116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 >> 172.21.15.1:0/407149664 conn(0x7f97f8071d90 msgr2=0x7f97f80fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 shutdown_connections 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.050+0000 7f97fd000700 1 -- 172.21.15.1:0/407149664 wait complete. 2024-02-28T17:27:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:27:15.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.198+0000 7f32e3e00700 1 -- 172.21.15.1:0/1532990028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e40fad30 msgr2=0x7f32e40fb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.198+0000 7f32e3e00700 1 --2- 172.21.15.1:0/1532990028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e40fad30 0x7f32e40fb1a0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f32d0004940 tx=0x7f32d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.198+0000 7f32e3e00700 1 -- 172.21.15.1:0/1532990028 shutdown_connections 2024-02-28T17:27:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.198+0000 7f32e3e00700 1 --2- 172.21.15.1:0/1532990028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e40fad30 0x7f32e40fb1a0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.198+0000 7f32e3e00700 1 --2- 172.21.15.1:0/1532990028 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32e4102e80 0x7f32e4103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.198+0000 7f32e3e00700 1 -- 172.21.15.1:0/1532990028 >> 172.21.15.1:0/1532990028 conn(0x7f32e40f6aa0 msgr2=0x7f32e40f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.198+0000 7f32e3e00700 1 -- 172.21.15.1:0/1532990028 shutdown_connections 2024-02-28T17:27:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.198+0000 7f32e3e00700 1 -- 172.21.15.1:0/1532990028 wait complete. 2024-02-28T17:27:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.199+0000 7f32e3e00700 1 Processor -- start 2024-02-28T17:27:15.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.199+0000 7f32e3e00700 1 -- start start 2024-02-28T17:27:15.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.199+0000 7f32e3e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32e40fad30 0x7f32e40fc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.199+0000 7f32e3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e4102e80 0x7f32e40fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.199+0000 7f32e3e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e40fd370 con 0x7f32e4102e80 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.199+0000 7f32e3e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e40fd4e0 con 0x7f32e40fad30 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.200+0000 7f32e2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e4102e80 0x7f32e40fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.200+0000 7f32e2000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e4102e80 0x7f32e40fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47490/0 (socket says 172.21.15.1:47490) 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.200+0000 7f32e2000700 1 -- 172.21.15.1:0/3639197043 learned_addr learned my addr 172.21.15.1:0/3639197043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.200+0000 7f32e2a00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32e40fad30 0x7f32e40fc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.200+0000 7f32e2000700 1 -- 172.21.15.1:0/3639197043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32e40fad30 msgr2=0x7f32e40fc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.200+0000 7f32e2000700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32e40fad30 0x7f32e40fc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.200+0000 7f32e2000700 1 -- 172.21.15.1:0/3639197043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d0005040 con 0x7f32e4102e80 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.200+0000 7f32e2000700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e4102e80 0x7f32e40fcce0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009150 tx=0x7f32d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.201+0000 7f32daa00700 1 -- 172.21.15.1:0/3639197043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32d000e040 con 0x7f32e4102e80 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.201+0000 7f32daa00700 1 -- 172.21.15.1:0/3639197043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32d000bbf0 con 0x7f32e4102e80 2024-02-28T17:27:15.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.201+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e4119ba0 con 0x7f32e4102e80 2024-02-28T17:27:15.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.201+0000 7f32daa00700 1 -- 172.21.15.1:0/3639197043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32d001c630 con 0x7f32e4102e80 2024-02-28T17:27:15.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.201+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e411a090 con 0x7f32e4102e80 2024-02-28T17:27:15.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.202+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e41000e0 con 0x7f32e4102e80 2024-02-28T17:27:15.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.203+0000 7f32daa00700 1 -- 172.21.15.1:0/3639197043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f32d000bd60 con 0x7f32e4102e80 2024-02-28T17:27:15.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.204+0000 7f32daa00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f32c806d7f0 0x7f32c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.204+0000 7f32daa00700 1 -- 172.21.15.1:0/3639197043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f32d008e290 con 0x7f32e4102e80 2024-02-28T17:27:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.204+0000 7f32e2a00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f32c806d7f0 0x7f32c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.205+0000 7f32e2a00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f32c806d7f0 0x7f32c806fca0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f32d4006fd0 tx=0x7f32d4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:15.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.209+0000 7f32daa00700 1 -- 172.21.15.1:0/3639197043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f32d0057b40 con 0x7f32e4102e80 2024-02-28T17:27:15.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.449+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32e404bf40 con 0x7f32c806d7f0 2024-02-28T17:27:15.452 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:15 smithi001 ceph-mon[112940]: from='client.36776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:15.452 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:15 smithi001 ceph-mon[112940]: from='client.36780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:15.452 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:15 smithi001 ceph-mon[112940]: pgmap v1587: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:15.452 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:15 smithi001 ceph-mon[112940]: from='client.36784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:15.452 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:15 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1603053286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:15.452 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:15 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/407149664' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:27:15.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.451+0000 7f32daa00700 1 -- 172.21.15.1:0/3639197043 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f32e404bf40 con 0x7f32c806d7f0 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc", 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": true, 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [ 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osd", 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "mon", 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr", 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "mds" 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": "16/21 daemons upgraded", 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "Currently upgrading mds daemons", 2024-02-28T17:27:15.454 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:27:15.455 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:27:15.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f32c806d7f0 msgr2=0x7f32c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f32c806d7f0 0x7f32c806fca0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f32d4006fd0 tx=0x7f32d4008040 comp rx=0 tx=0).stop 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e4102e80 msgr2=0x7f32e40fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e4102e80 0x7f32e40fcce0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009150 tx=0x7f32d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 shutdown_connections 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f32c806d7f0 0x7f32c806fca0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f32e40fad30 0x7f32e40fc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 --2- 172.21.15.1:0/3639197043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e4102e80 0x7f32e40fcce0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 >> 172.21.15.1:0/3639197043 conn(0x7f32e40f6aa0 msgr2=0x7f32e40f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 shutdown_connections 2024-02-28T17:27:15.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.454+0000 7f32e3e00700 1 -- 172.21.15.1:0/3639197043 wait complete. 2024-02-28T17:27:15.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:15 smithi066 ceph-mon[85781]: from='client.36776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:15.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:15 smithi066 ceph-mon[85781]: from='client.36780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:15.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:15 smithi066 ceph-mon[85781]: pgmap v1587: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:15.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:15 smithi066 ceph-mon[85781]: from='client.36784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:15.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:15 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1603053286' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:15.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:15 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/407149664' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:27:15.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.607+0000 7ff48c600700 1 -- 172.21.15.1:0/1281650476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fbdd0 msgr2=0x7ff4840fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.607+0000 7ff48c600700 1 --2- 172.21.15.1:0/1281650476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fbdd0 0x7ff4840fc1a0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff478004940 tx=0x7ff478009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:15.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.607+0000 7ff48c600700 1 -- 172.21.15.1:0/1281650476 shutdown_connections 2024-02-28T17:27:15.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.607+0000 7ff48c600700 1 --2- 172.21.15.1:0/1281650476 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4840fc6e0 0x7ff484100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.607+0000 7ff48c600700 1 --2- 172.21.15.1:0/1281650476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fbdd0 0x7ff4840fc1a0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.607+0000 7ff48c600700 1 -- 172.21.15.1:0/1281650476 >> 172.21.15.1:0/1281650476 conn(0x7ff484071d90 msgr2=0x7ff484072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:15.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.607+0000 7ff48c600700 1 -- 172.21.15.1:0/1281650476 shutdown_connections 2024-02-28T17:27:15.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.607+0000 7ff48c600700 1 -- 172.21.15.1:0/1281650476 wait complete. 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.608+0000 7ff48c600700 1 Processor -- start 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.608+0000 7ff48c600700 1 -- start start 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff48c600700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4840fbdd0 0x7ff48411a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff48c600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fc6e0 0x7ff48411a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff48c600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48411abc0 con 0x7ff4840fc6e0 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff48c600700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff484113f40 con 0x7ff4840fbdd0 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff489200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fc6e0 0x7ff48411a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff489200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fc6e0 0x7ff48411a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47512/0 (socket says 172.21.15.1:47512) 2024-02-28T17:27:15.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff489200700 1 -- 172.21.15.1:0/2078271045 learned_addr learned my addr 172.21.15.1:0/2078271045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff489c00700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4840fbdd0 0x7ff48411a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff489200700 1 -- 172.21.15.1:0/2078271045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4840fbdd0 msgr2=0x7ff48411a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.609+0000 7ff489200700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4840fbdd0 0x7ff48411a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.610+0000 7ff489200700 1 -- 172.21.15.1:0/2078271045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478005040 con 0x7ff4840fc6e0 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.610+0000 7ff489200700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fc6e0 0x7ff48411a570 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7ff470002a50 tx=0x7ff4700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.610+0000 7ff481600700 1 -- 172.21.15.1:0/2078271045 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff470007cc0 con 0x7ff4840fc6e0 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.610+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff484114220 con 0x7ff4840fc6e0 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.610+0000 7ff481600700 1 -- 172.21.15.1:0/2078271045 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff47000f040 con 0x7ff4840fc6e0 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.610+0000 7ff481600700 1 -- 172.21.15.1:0/2078271045 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff470009640 con 0x7ff4840fc6e0 2024-02-28T17:27:15.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.610+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff484114760 con 0x7ff4840fc6e0 2024-02-28T17:27:15.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.611+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4840fe540 con 0x7ff4840fc6e0 2024-02-28T17:27:15.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.612+0000 7ff481600700 1 -- 172.21.15.1:0/2078271045 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff4700097c0 con 0x7ff4840fc6e0 2024-02-28T17:27:15.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.614+0000 7ff481600700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff46c06d7f0 0x7ff46c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:15.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.614+0000 7ff481600700 1 -- 172.21.15.1:0/2078271045 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff47008cce0 con 0x7ff4840fc6e0 2024-02-28T17:27:15.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.614+0000 7ff489c00700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff46c06d7f0 0x7ff46c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:15.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.615+0000 7ff489c00700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff46c06d7f0 0x7ff46c06fca0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff47800b020 tx=0x7ff478009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:15.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.618+0000 7ff481600700 1 -- 172.21.15.1:0/2078271045 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff470059fc0 con 0x7ff4840fc6e0 2024-02-28T17:27:15.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.924+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff4840738b0 con 0x7ff4840fc6e0 2024-02-28T17:27:15.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.925+0000 7ff481600700 1 -- 172.21.15.1:0/2078271045 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7ff470059b50 con 0x7ff4840fc6e0 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: host smithi066 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi066/config 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/podman: timeout after 895 seconds 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout:Traceback (most recent call last): 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: return _run_code(code, main_globals, None, 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: exec(code, run_globals) 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5520, in 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 5508, in main 2024-02-28T17:27:15.928 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 406, in _infer_config 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 321, in _infer_fsid 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 434, in _infer_image 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 308, in _validate_fsid 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/__main__.py", line 3301, in command_ceph_volume 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout: File "/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/cephadm.8e42bbf4765eafad2b4b094655392168f9104bcdebfde51be0935be4834d689e/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc -e NODE_NAME=smithi066 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/run/ceph:z -v /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptor5qy2p:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc inventory --format=json-pretty --filter-for-batch 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is degraded 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-02-28T17:27:15.929 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs is offline because no MDS is active for it. 2024-02-28T17:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff46c06d7f0 msgr2=0x7ff46c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff46c06d7f0 0x7ff46c06fca0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff47800b020 tx=0x7ff478009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fc6e0 msgr2=0x7ff48411a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fc6e0 0x7ff48411a570 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7ff470002a50 tx=0x7ff4700046b0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 shutdown_connections 2024-02-28T17:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff46c06d7f0 0x7ff46c06fca0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff4840fbdd0 0x7ff48411a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 --2- 172.21.15.1:0/2078271045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4840fc6e0 0x7ff48411a570 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 >> 172.21.15.1:0/2078271045 conn(0x7ff484071d90 msgr2=0x7ff48410d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 shutdown_connections 2024-02-28T17:27:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:15.928+0000 7ff48c600700 1 -- 172.21.15.1:0/2078271045 wait complete. 2024-02-28T17:27:16.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:16 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2078271045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:27:16.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:16 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2078271045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:27:17.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:17 smithi066 ceph-mon[85781]: from='client.36796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:17.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:17 smithi066 ceph-mon[85781]: pgmap v1588: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:17.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:17 smithi001 ceph-mon[112940]: from='client.36796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:17.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:17 smithi001 ceph-mon[112940]: pgmap v1588: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:18.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:18 smithi066 ceph-mon[85781]: pgmap v1589: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:18.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:18 smithi001 ceph-mon[112940]: pgmap v1589: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:21.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:20 smithi066 ceph-mon[85781]: pgmap v1590: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:21.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:20 smithi001 ceph-mon[112940]: pgmap v1590: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:23.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:22 smithi066 ceph-mon[85781]: pgmap v1591: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:23.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:22 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:27:23.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:22 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:27:23.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:22 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:23.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:22 smithi001 ceph-mon[112940]: pgmap v1591: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:23.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:22 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:27:23.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:22 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:27:23.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:22 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: pgmap v1592: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: pgmap v1593: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all mds 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi001.awxxdo"}]': finished 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi001.rtdwhc"}]': finished 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi066.pivimo"}]': finished 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi066.wfbdor"}]': finished 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: Upgrade: Setting filesystem cephfs Joinable 2024-02-28T17:27:24.263 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:23 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-02-28T17:27:24.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: pgmap v1592: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: pgmap v1593: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all mds 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi001.awxxdo"}]: dispatch 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi001.awxxdo"}]': finished 2024-02-28T17:27:24.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi001.rtdwhc"}]: dispatch 2024-02-28T17:27:24.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi001.rtdwhc"}]': finished 2024-02-28T17:27:24.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi066.pivimo"}]: dispatch 2024-02-28T17:27:24.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi066.pivimo"}]': finished 2024-02-28T17:27:24.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi066.wfbdor"}]: dispatch 2024-02-28T17:27:24.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi066.wfbdor"}]': finished 2024-02-28T17:27:24.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: Upgrade: Setting filesystem cephfs Joinable 2024-02-28T17:27:24.285 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:23 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-02-28T17:27:25.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: Standby daemon mds.cephfs.smithi001.awxxdo assigned to filesystem cephfs as rank 0 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: Standby daemon mds.cephfs.smithi001.rtdwhc assigned to filesystem cephfs as rank 1 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: fsmap cephfs:2/2 {0=cephfs.smithi001.awxxdo=up:replay,1=cephfs.smithi001.rtdwhc=up:replay} 2 up:standby 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all rgw 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.513 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all rbd-mirror 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all ceph-exporter 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all iscsi 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all nfs 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.514 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.515 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-28T17:27:25.516 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-28T17:27:25.517 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-28T17:27:25.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-28T17:27:25.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.518 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-28T17:27:25.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-28T17:27:25.519 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:25 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: Standby daemon mds.cephfs.smithi001.awxxdo assigned to filesystem cephfs as rank 0 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: Standby daemon mds.cephfs.smithi001.rtdwhc assigned to filesystem cephfs as rank 1 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: fsmap cephfs:2/2 {0=cephfs.smithi001.awxxdo=up:replay,1=cephfs.smithi001.rtdwhc=up:replay} 2 up:standby 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all rgw 2024-02-28T17:27:25.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all rbd-mirror 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all ceph-exporter 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all iscsi 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all nfs 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-28T17:27:25.536 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-28T17:27:25.537 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-28T17:27:25.538 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-28T17:27:25.539 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-28T17:27:25.539 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:25 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:27:26.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: Upgrade: Setting container_image for all nvmeof 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: Upgrade: Finalizing container_image settings 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: Upgrade: Complete! 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: pgmap v1594: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4.1 MiB/s rd, 1 op/s 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:27:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: Upgrade: Setting container_image for all nvmeof 2024-02-28T17:27:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: Upgrade: Finalizing container_image settings 2024-02-28T17:27:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: Upgrade: Complete! 2024-02-28T17:27:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: pgmap v1594: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4.1 MiB/s rd, 1 op/s 2024-02-28T17:27:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:26.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:27:27.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:27 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:27.761 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:27 smithi066 ceph-mon[85781]: pgmap v1595: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 5.4 MiB/s rd, 2 op/s 2024-02-28T17:27:27.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:27 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:27.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:27 smithi001 ceph-mon[112940]: pgmap v1595: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 5.4 MiB/s rd, 2 op/s 2024-02-28T17:27:29.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:29 smithi001 ceph-mon[112940]: pgmap v1596: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 7 op/s 2024-02-28T17:27:29.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:29 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:29.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:29 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:30.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:29 smithi066 ceph-mon[85781]: pgmap v1596: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 7 op/s 2024-02-28T17:27:30.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:29 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:30.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:29 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:32.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:31 smithi066 ceph-mon[85781]: pgmap v1597: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 31 MiB/s rd, 10 op/s 2024-02-28T17:27:32.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:31 smithi001 ceph-mon[112940]: pgmap v1597: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 31 MiB/s rd, 10 op/s 2024-02-28T17:27:32.835 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:32 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:resolve 2024-02-28T17:27:32.835 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:32 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:resolve 2024-02-28T17:27:32.835 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:32 smithi066 ceph-mon[85781]: fsmap cephfs:2/2 {0=cephfs.smithi001.awxxdo=up:resolve,1=cephfs.smithi001.rtdwhc=up:resolve} 2 up:standby 2024-02-28T17:27:32.835 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:32.835 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:32 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:33.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:32 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:resolve 2024-02-28T17:27:33.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:32 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:resolve 2024-02-28T17:27:33.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:32 smithi001 ceph-mon[112940]: fsmap cephfs:2/2 {0=cephfs.smithi001.awxxdo=up:resolve,1=cephfs.smithi001.rtdwhc=up:resolve} 2 up:standby 2024-02-28T17:27:33.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:33.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:32 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:33 smithi001 ceph-mon[112940]: pgmap v1598: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 31 MiB/s rd, 10 op/s 2024-02-28T17:27:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:33 smithi001 ceph-mon[112940]: reconnect by client.14524 192.168.0.1:0/612538778 after 0.00199997 2024-02-28T17:27:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:33 smithi001 ceph-mon[112940]: reconnect by client.24299 192.168.0.1:0/790488749 after 0.00299996 2024-02-28T17:27:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:33 smithi001 ceph-mon[112940]: reconnect by client.14524 192.168.0.1:0/612538778 after 0.00399995 2024-02-28T17:27:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:33 smithi001 ceph-mon[112940]: reconnect by client.24299 192.168.0.1:0/790488749 after 0.00299996 2024-02-28T17:27:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:33 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:reconnect 2024-02-28T17:27:34.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:33 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:reconnect 2024-02-28T17:27:34.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:33 smithi001 ceph-mon[112940]: fsmap cephfs:2/2 {0=cephfs.smithi001.awxxdo=up:reconnect,1=cephfs.smithi001.rtdwhc=up:reconnect} 2 up:standby 2024-02-28T17:27:34.202 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:33 smithi066 ceph-mon[85781]: pgmap v1598: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 31 MiB/s rd, 10 op/s 2024-02-28T17:27:34.202 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:33 smithi066 ceph-mon[85781]: reconnect by client.14524 192.168.0.1:0/612538778 after 0.00199997 2024-02-28T17:27:34.202 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:33 smithi066 ceph-mon[85781]: reconnect by client.24299 192.168.0.1:0/790488749 after 0.00299996 2024-02-28T17:27:34.203 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:33 smithi066 ceph-mon[85781]: reconnect by client.14524 192.168.0.1:0/612538778 after 0.00399995 2024-02-28T17:27:34.203 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:33 smithi066 ceph-mon[85781]: reconnect by client.24299 192.168.0.1:0/790488749 after 0.00299996 2024-02-28T17:27:34.203 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:33 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:reconnect 2024-02-28T17:27:34.203 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:33 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:reconnect 2024-02-28T17:27:34.203 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:33 smithi066 ceph-mon[85781]: fsmap cephfs:2/2 {0=cephfs.smithi001.awxxdo=up:reconnect,1=cephfs.smithi001.rtdwhc=up:reconnect} 2 up:standby 2024-02-28T17:27:34.806 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:34 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:rejoin 2024-02-28T17:27:34.806 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:34 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:rejoin 2024-02-28T17:27:34.806 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:34 smithi066 ceph-mon[85781]: fsmap cephfs:2/2 {0=cephfs.smithi001.awxxdo=up:rejoin,1=cephfs.smithi001.rtdwhc=up:rejoin} 2 up:standby 2024-02-28T17:27:34.806 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:34 smithi066 ceph-mon[85781]: daemon mds.cephfs.smithi001.rtdwhc is now active in filesystem cephfs as rank 1 2024-02-28T17:27:34.806 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:34 smithi066 ceph-mon[85781]: daemon mds.cephfs.smithi001.awxxdo is now active in filesystem cephfs as rank 0 2024-02-28T17:27:35.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:34 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:rejoin 2024-02-28T17:27:35.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:34 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:rejoin 2024-02-28T17:27:35.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:34 smithi001 ceph-mon[112940]: fsmap cephfs:2/2 {0=cephfs.smithi001.awxxdo=up:rejoin,1=cephfs.smithi001.rtdwhc=up:rejoin} 2 up:standby 2024-02-28T17:27:35.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:34 smithi001 ceph-mon[112940]: daemon mds.cephfs.smithi001.rtdwhc is now active in filesystem cephfs as rank 1 2024-02-28T17:27:35.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:34 smithi001 ceph-mon[112940]: daemon mds.cephfs.smithi001.awxxdo is now active in filesystem cephfs as rank 0 2024-02-28T17:27:36.163 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:35 smithi066 ceph-mon[85781]: pgmap v1599: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 170 B/s wr, 8 op/s 2024-02-28T17:27:36.163 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:35 smithi066 ceph-mon[85781]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-28T17:27:36.163 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:35 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:active 2024-02-28T17:27:36.163 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:35 smithi066 ceph-mon[85781]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:active 2024-02-28T17:27:36.163 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:35 smithi066 ceph-mon[85781]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi001.rtdwhc=up:active} 2 up:standby 2024-02-28T17:27:36.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:35 smithi001 ceph-mon[112940]: pgmap v1599: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 170 B/s wr, 8 op/s 2024-02-28T17:27:36.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:35 smithi001 ceph-mon[112940]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-28T17:27:36.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:35 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] up:active 2024-02-28T17:27:36.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:35 smithi001 ceph-mon[112940]: mds.? [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] up:active 2024-02-28T17:27:36.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:35 smithi001 ceph-mon[112940]: fsmap cephfs:2 {0=cephfs.smithi001.awxxdo=up:active,1=cephfs.smithi001.rtdwhc=up:active} 2 up:standby 2024-02-28T17:27:38.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:37 smithi066 ceph-mon[85781]: pgmap v1600: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 170 B/s wr, 8 op/s 2024-02-28T17:27:38.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:37 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:38.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:37 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:38.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:37 smithi001 ceph-mon[112940]: pgmap v1600: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 170 B/s wr, 8 op/s 2024-02-28T17:27:38.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:37 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:38.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:37 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:39.783 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:39 smithi001 ceph-mon[112940]: pgmap v1601: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 341 B/s wr, 11 op/s 2024-02-28T17:27:40.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:39 smithi066 ceph-mon[85781]: pgmap v1601: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 341 B/s wr, 11 op/s 2024-02-28T17:27:42.176 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:41 smithi001 ceph-mon[112940]: pgmap v1602: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 13 MiB/s rd, 4.5 KiB/s wr, 11 op/s 2024-02-28T17:27:42.176 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:42.176 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:27:42.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:41 smithi066 ceph-mon[85781]: pgmap v1602: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 13 MiB/s rd, 4.5 KiB/s wr, 11 op/s 2024-02-28T17:27:42.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:42.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:27:44.137 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:43 smithi001 ceph-mon[112940]: pgmap v1603: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.8 MiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-02-28T17:27:44.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:43 smithi066 ceph-mon[85781]: pgmap v1603: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.8 MiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-02-28T17:27:46.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.100+0000 7fec25400700 1 -- 172.21.15.1:0/2677469122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 msgr2=0x7fec20104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:46.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.100+0000 7fec25400700 1 --2- 172.21.15.1:0/2677469122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 0x7fec20104210 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fec0c004940 tx=0x7fec0c009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:46.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.101+0000 7fec25400700 1 -- 172.21.15.1:0/2677469122 shutdown_connections 2024-02-28T17:27:46.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.101+0000 7fec25400700 1 --2- 172.21.15.1:0/2677469122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 0x7fec20104210 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:46.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.101+0000 7fec25400700 1 --2- 172.21.15.1:0/2677469122 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec201095f0 0x7fec201099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:46.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.101+0000 7fec25400700 1 -- 172.21.15.1:0/2677469122 >> 172.21.15.1:0/2677469122 conn(0x7fec20072d50 msgr2=0x7fec20075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:46.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.101+0000 7fec25400700 1 -- 172.21.15.1:0/2677469122 shutdown_connections 2024-02-28T17:27:46.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.101+0000 7fec25400700 1 -- 172.21.15.1:0/2677469122 wait complete. 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.102+0000 7fec25400700 1 Processor -- start 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.102+0000 7fec25400700 1 -- start start 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.102+0000 7fec25400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 0x7fec2011a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.102+0000 7fec25400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec201095f0 0x7fec2011a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.102+0000 7fec25400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2011abd0 con 0x7fec20103dc0 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.102+0000 7fec25400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec20113ee0 con 0x7fec201095f0 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.103+0000 7fec1e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 0x7fec2011a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.103+0000 7fec1e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 0x7fec2011a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50740/0 (socket says 172.21.15.1:50740) 2024-02-28T17:27:46.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.103+0000 7fec1e800700 1 -- 172.21.15.1:0/658902686 learned_addr learned my addr 172.21.15.1:0/658902686 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:46.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.103+0000 7fec1e800700 1 -- 172.21.15.1:0/658902686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec201095f0 msgr2=0x7fec2011a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:46.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.103+0000 7fec1de00700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec201095f0 0x7fec2011a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:46.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.103+0000 7fec1e800700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec201095f0 0x7fec2011a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:46.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.103+0000 7fec1e800700 1 -- 172.21.15.1:0/658902686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c005040 con 0x7fec20103dc0 2024-02-28T17:27:46.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.103+0000 7fec1e800700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 0x7fec2011a040 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fec1000c830 tx=0x7fec10004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:46.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.104+0000 7fec16a00700 1 -- 172.21.15.1:0/658902686 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec1000a760 con 0x7fec20103dc0 2024-02-28T17:27:46.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.104+0000 7fec16a00700 1 -- 172.21.15.1:0/658902686 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec1000ad90 con 0x7fec20103dc0 2024-02-28T17:27:46.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.104+0000 7fec16a00700 1 -- 172.21.15.1:0/658902686 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec100113a0 con 0x7fec20103dc0 2024-02-28T17:27:46.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.104+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec201141c0 con 0x7fec20103dc0 2024-02-28T17:27:46.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.104+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec20114700 con 0x7fec20103dc0 2024-02-28T17:27:46.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.105+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2010b9c0 con 0x7fec20103dc0 2024-02-28T17:27:46.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.106+0000 7fec16a00700 1 -- 172.21.15.1:0/658902686 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fec10012400 con 0x7fec20103dc0 2024-02-28T17:27:46.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.107+0000 7fec16a00700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec0406d7f0 0x7fec0406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:46.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.108+0000 7fec16a00700 1 -- 172.21.15.1:0/658902686 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fec1008cf00 con 0x7fec20103dc0 2024-02-28T17:27:46.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.108+0000 7fec1de00700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec0406d7f0 0x7fec0406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:46.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.109+0000 7fec1de00700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec0406d7f0 0x7fec0406fca0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fec201158e0 tx=0x7fec0c009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:46.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.113+0000 7fec16a00700 1 -- 172.21.15.1:0/658902686 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec1005a2e0 con 0x7fec20103dc0 2024-02-28T17:27:46.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: pgmap v1604: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 867 KiB/s rd, 8.4 KiB/s wr, 10 op/s 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: pgmap v1605: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7.2 KiB/s rd, 9.6 KiB/s wr, 11 op/s 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:27:46.262 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:45 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:46.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: pgmap v1604: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 867 KiB/s rd, 8.4 KiB/s wr, 10 op/s 2024-02-28T17:27:46.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:46.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:46.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:27:46.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:27:46.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: pgmap v1605: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7.2 KiB/s rd, 9.6 KiB/s wr, 11 op/s 2024-02-28T17:27:46.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:46.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:27:46.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:27:46.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:27:46.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:45 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:46.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.361+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fec2004bf70 con 0x7fec0406d7f0 2024-02-28T17:27:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.363+0000 7fec16a00700 1 -- 172.21.15.1:0/658902686 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fec0c009710 con 0x7fec0406d7f0 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec0406d7f0 msgr2=0x7fec0406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec0406d7f0 0x7fec0406fca0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fec201158e0 tx=0x7fec0c009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 msgr2=0x7fec2011a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 0x7fec2011a040 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fec1000c830 tx=0x7fec10004470 comp rx=0 tx=0).stop 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 shutdown_connections 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fec0406d7f0 0x7fec0406fca0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec20103dc0 0x7fec2011a040 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 --2- 172.21.15.1:0/658902686 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fec201095f0 0x7fec2011a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 >> 172.21.15.1:0/658902686 conn(0x7fec20072d50 msgr2=0x7fec20074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 shutdown_connections 2024-02-28T17:27:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:46.366+0000 7fec25400700 1 -- 172.21.15.1:0/658902686 wait complete. 2024-02-28T17:27:47.134 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph orch ps' 2024-02-28T17:27:47.228 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:46 smithi001 ceph-mon[112940]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-02-28T17:27:47.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:46 smithi066 ceph-mon[85781]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-02-28T17:27:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:27:48.112 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:47 smithi001 ceph-mon[112940]: from='client.36804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:48.112 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:47 smithi001 ceph-mon[112940]: pgmap v1606: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 9.6 KiB/s wr, 10 op/s 2024-02-28T17:27:48.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:47 smithi066 ceph-mon[85781]: from='client.36804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:48.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:47 smithi066 ceph-mon[85781]: pgmap v1606: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 9.6 KiB/s wr, 10 op/s 2024-02-28T17:27:49.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.674+0000 7f5bcb200700 1 -- 172.21.15.1:0/2819983701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 msgr2=0x7f5bc4070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:49.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.674+0000 7f5bcb200700 1 --2- 172.21.15.1:0/2819983701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 0x7f5bc4070ce0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f5bac004970 tx=0x7f5bac009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:49.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.674+0000 7f5bcb200700 1 -- 172.21.15.1:0/2819983701 shutdown_connections 2024-02-28T17:27:49.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.674+0000 7f5bcb200700 1 --2- 172.21.15.1:0/2819983701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 0x7f5bc4070ce0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:49.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.674+0000 7f5bcb200700 1 --2- 172.21.15.1:0/2819983701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5bc4071f60 0x7f5bc4070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:49.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.674+0000 7f5bcb200700 1 -- 172.21.15.1:0/2819983701 >> 172.21.15.1:0/2819983701 conn(0x7f5bc40f8ae0 msgr2=0x7f5bc40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:49.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.674+0000 7f5bcb200700 1 -- 172.21.15.1:0/2819983701 shutdown_connections 2024-02-28T17:27:49.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.674+0000 7f5bcb200700 1 -- 172.21.15.1:0/2819983701 wait complete. 2024-02-28T17:27:49.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.675+0000 7f5bcb200700 1 Processor -- start 2024-02-28T17:27:49.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.675+0000 7f5bcb200700 1 -- start start 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.675+0000 7f5bcb200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 0x7f5bc411a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.675+0000 7f5bcb200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5bc4071f60 0x7f5bc411a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.675+0000 7f5bcb200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc411aba0 con 0x7f5bc4070870 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.675+0000 7f5bcb200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc4113eb0 con 0x7f5bc4071f60 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.675+0000 7f5bc3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 0x7f5bc411a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.676+0000 7f5bc3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 0x7f5bc411a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50764/0 (socket says 172.21.15.1:50764) 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.676+0000 7f5bc3e00700 1 -- 172.21.15.1:0/1480926176 learned_addr learned my addr 172.21.15.1:0/1480926176 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.676+0000 7f5bbb400700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5bc4071f60 0x7f5bc411a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.676+0000 7f5bc3e00700 1 -- 172.21.15.1:0/1480926176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5bc4071f60 msgr2=0x7f5bc411a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.676+0000 7f5bc3e00700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5bc4071f60 0x7f5bc411a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.676+0000 7f5bc3e00700 1 -- 172.21.15.1:0/1480926176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bac005040 con 0x7f5bc4070870 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.676+0000 7f5bc3e00700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 0x7f5bc411a010 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400cac0 tx=0x7f5bb4007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:49.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.676+0000 7f5bc0c00700 1 -- 172.21.15.1:0/1480926176 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bb400aa50 con 0x7f5bc4070870 2024-02-28T17:27:49.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.677+0000 7f5bc0c00700 1 -- 172.21.15.1:0/1480926176 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bb400abb0 con 0x7f5bc4070870 2024-02-28T17:27:49.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.677+0000 7f5bc0c00700 1 -- 172.21.15.1:0/1480926176 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bb40115e0 con 0x7f5bc4070870 2024-02-28T17:27:49.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.677+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc4114190 con 0x7f5bc4070870 2024-02-28T17:27:49.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.677+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc41146d0 con 0x7f5bc4070870 2024-02-28T17:27:49.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.678+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc40fe400 con 0x7f5bc4070870 2024-02-28T17:27:49.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.679+0000 7f5bc0c00700 1 -- 172.21.15.1:0/1480926176 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5bb400ad20 con 0x7f5bc4070870 2024-02-28T17:27:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.680+0000 7f5bc0c00700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ba806d7e0 0x7f5ba806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.680+0000 7f5bc0c00700 1 -- 172.21.15.1:0/1480926176 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5bb408d640 con 0x7f5bc4070870 2024-02-28T17:27:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.680+0000 7f5bbb400700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ba806d7e0 0x7f5ba806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.681+0000 7f5bbb400700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ba806d7e0 0x7f5ba806fc90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5bac000c00 tx=0x7f5bac009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:49.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.685+0000 7f5bc0c00700 1 -- 172.21.15.1:0/1480926176 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5bb405a9a0 con 0x7f5bc4070870 2024-02-28T17:27:49.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.921+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bc404bed0 con 0x7f5ba806d7e0 2024-02-28T17:27:49.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.938+0000 7f5bc0c00700 1 -- 172.21.15.1:0/1480926176 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5bc404bed0 con 0x7f5ba806d7e0 2024-02-28T17:27:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:27:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (53m) 17s ago 68m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:27:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (50m) 17s ago 68m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:27:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (50m) 21s ago 67m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:27:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (53m) 17s ago 67m 77.4M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:27:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (46m) 17s ago 61m 160M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (46m) 17s ago 60m 162M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (15m) 21s ago 60m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (30m) 21s ago 61m 21.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (56m) 17s ago 69m 519M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (55m) 21s ago 66m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (52m) 17s ago 70m 99.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (51m) 21s ago 66m 64.0M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (55m) 17s ago 68m 8820k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (54m) 21s ago 66m 8619k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (50m) 17s ago 65m 95.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (49m) 17s ago 65m 102M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (48m) 17s ago 64m 90.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:27:49.941 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (48m) 21s ago 64m 111M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:27:49.942 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (47m) 21s ago 63m 92.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:27:49.942 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (46m) 21s ago 62m 91.3M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:27:49.942 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (54m) 17s ago 67m 65.0M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ba806d7e0 msgr2=0x7f5ba806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ba806d7e0 0x7f5ba806fc90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5bac000c00 tx=0x7f5bac009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:49.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 msgr2=0x7f5bc411a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 0x7f5bc411a010 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400cac0 tx=0x7f5bb4007ee0 comp rx=0 tx=0).stop 2024-02-28T17:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 shutdown_connections 2024-02-28T17:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5ba806d7e0 0x7f5ba806fc90 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc4070870 0x7f5bc411a010 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 --2- 172.21.15.1:0/1480926176 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5bc4071f60 0x7f5bc411a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 >> 172.21.15.1:0/1480926176 conn(0x7f5bc40f8ae0 msgr2=0x7f5bc40fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 shutdown_connections 2024-02-28T17:27:49.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:49.941+0000 7f5bcb200700 1 -- 172.21.15.1:0/1480926176 wait complete. 2024-02-28T17:27:50.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:50 smithi001 ceph-mon[112940]: pgmap v1607: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 9.4 KiB/s wr, 6 op/s 2024-02-28T17:27:50.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:50 smithi066 ceph-mon[85781]: pgmap v1607: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 9.4 KiB/s wr, 6 op/s 2024-02-28T17:27:50.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph orch upgrade status' 2024-02-28T17:27:51.227 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:27:51.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:51 smithi001 ceph-mon[112940]: from='client.36808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:51.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:51 smithi066 ceph-mon[85781]: from='client.36808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:52.281 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:52 smithi001 ceph-mon[112940]: pgmap v1608: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4.5 KiB/s wr, 2 op/s 2024-02-28T17:27:52.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:52 smithi066 ceph-mon[85781]: pgmap v1608: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4.5 KiB/s wr, 2 op/s 2024-02-28T17:27:53.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.375+0000 7f7957a00700 1 -- 172.21.15.1:0/3024058384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105150 msgr2=0x7f7950105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:53.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.375+0000 7f7957a00700 1 --2- 172.21.15.1:0/3024058384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105150 0x7f7950105520 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f7940004940 tx=0x7f7940009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:53.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.375+0000 7f7957a00700 1 -- 172.21.15.1:0/3024058384 shutdown_connections 2024-02-28T17:27:53.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.375+0000 7f7957a00700 1 --2- 172.21.15.1:0/3024058384 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7950105af0 0x7f79500ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:53.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.375+0000 7f7957a00700 1 --2- 172.21.15.1:0/3024058384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105150 0x7f7950105520 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:53.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.375+0000 7f7957a00700 1 -- 172.21.15.1:0/3024058384 >> 172.21.15.1:0/3024058384 conn(0x7f79500face0 msgr2=0x7f79500fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:53.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.375+0000 7f7957a00700 1 -- 172.21.15.1:0/3024058384 shutdown_connections 2024-02-28T17:27:53.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.375+0000 7f7957a00700 1 -- 172.21.15.1:0/3024058384 wait complete. 2024-02-28T17:27:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.376+0000 7f7957a00700 1 Processor -- start 2024-02-28T17:27:53.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.376+0000 7f7957a00700 1 -- start start 2024-02-28T17:27:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.377+0000 7f7957a00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7950105150 0x7f7950115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.377+0000 7f7957a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105af0 0x7f79501161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.377+0000 7f7957a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7950116800 con 0x7f7950105af0 2024-02-28T17:27:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.377+0000 7f7957a00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795010fb30 con 0x7f7950105150 2024-02-28T17:27:53.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.377+0000 7f7947e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105af0 0x7f79501161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.377+0000 7f7947e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105af0 0x7f79501161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53840/0 (socket says 172.21.15.1:53840) 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.377+0000 7f7947e00700 1 -- 172.21.15.1:0/2721405040 learned_addr learned my addr 172.21.15.1:0/2721405040 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f7955000700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7950105150 0x7f7950115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f7947e00700 1 -- 172.21.15.1:0/2721405040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7950105150 msgr2=0x7f7950115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f7947e00700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7950105150 0x7f7950115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f7947e00700 1 -- 172.21.15.1:0/2721405040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940005040 con 0x7f7950105af0 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f7955000700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7950105150 0x7f7950115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f7947e00700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105af0 0x7f79501161b0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f7938002a50 tx=0x7f7938004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f794d600700 1 -- 172.21.15.1:0/2721405040 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7938007cc0 con 0x7f7950105af0 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f794d600700 1 -- 172.21.15.1:0/2721405040 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f793800f040 con 0x7f7950105af0 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f794d600700 1 -- 172.21.15.1:0/2721405040 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f793800da40 con 0x7f7950105af0 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795010fe10 con 0x7f7950105af0 2024-02-28T17:27:53.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.378+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7950110350 con 0x7f7950105af0 2024-02-28T17:27:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.380+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79500fe8e0 con 0x7f7950105af0 2024-02-28T17:27:53.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.381+0000 7f794d600700 1 -- 172.21.15.1:0/2721405040 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f79380097c0 con 0x7f7950105af0 2024-02-28T17:27:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.382+0000 7f794d600700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f793406d7f0 0x7f793406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:53.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.382+0000 7f794d600700 1 -- 172.21.15.1:0/2721405040 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f793808cca0 con 0x7f7950105af0 2024-02-28T17:27:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.382+0000 7f7955000700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f793406d7f0 0x7f793406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:53.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.383+0000 7f7955000700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f793406d7f0 0x7f793406fca0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f794000b3b0 tx=0x7f7940009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:53.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.387+0000 7f794d600700 1 -- 172.21.15.1:0/2721405040 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7938059d70 con 0x7f7950105af0 2024-02-28T17:27:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.632+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7950111250 con 0x7f793406d7f0 2024-02-28T17:27:53.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.634+0000 7f794d600700 1 -- 172.21.15.1:0/2721405040 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f7950111250 con 0x7f793406d7f0 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "target_image": null, 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "in_progress": false, 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "which": "", 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "services_complete": [], 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "progress": null, 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "message": "", 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "is_paused": false 2024-02-28T17:27:53.637 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f793406d7f0 msgr2=0x7f793406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f793406d7f0 0x7f793406fca0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f794000b3b0 tx=0x7f7940009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105af0 msgr2=0x7f79501161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105af0 0x7f79501161b0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f7938002a50 tx=0x7f7938004600 comp rx=0 tx=0).stop 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 shutdown_connections 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f793406d7f0 0x7f793406fca0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7950105150 0x7f7950115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 --2- 172.21.15.1:0/2721405040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7950105af0 0x7f79501161b0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 >> 172.21.15.1:0/2721405040 conn(0x7f79500face0 msgr2=0x7f79500fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 shutdown_connections 2024-02-28T17:27:53.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:53.637+0000 7f7957a00700 1 -- 172.21.15.1:0/2721405040 wait complete. 2024-02-28T17:27:54.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:54 smithi001 ceph-mon[112940]: pgmap v1609: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4.5 KiB/s wr, 2 op/s 2024-02-28T17:27:54.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph health detail' 2024-02-28T17:27:54.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:54 smithi066 ceph-mon[85781]: pgmap v1609: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4.5 KiB/s wr, 2 op/s 2024-02-28T17:27:54.884 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:27:55.373 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:55 smithi001 ceph-mon[112940]: from='client.36812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:55.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:55 smithi066 ceph-mon[85781]: from='client.36812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:27:56.168 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:56 smithi001 ceph-mon[112940]: pgmap v1610: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:56.168 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:56.168 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:27:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:56 smithi066 ceph-mon[85781]: pgmap v1610: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:27:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:27:56.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.970+0000 7f01e5e00700 1 -- 172.21.15.1:0/2289646744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e01095b0 msgr2=0x7f01e0109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:56.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.970+0000 7f01e5e00700 1 --2- 172.21.15.1:0/2289646744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e01095b0 0x7f01e0109980 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f01d0004940 tx=0x7f01d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:56.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.970+0000 7f01e5e00700 1 -- 172.21.15.1:0/2289646744 shutdown_connections 2024-02-28T17:27:56.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.970+0000 7f01e5e00700 1 --2- 172.21.15.1:0/2289646744 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f01e0103dc0 0x7f01e0104230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:56.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.970+0000 7f01e5e00700 1 --2- 172.21.15.1:0/2289646744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e01095b0 0x7f01e0109980 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:56.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.970+0000 7f01e5e00700 1 -- 172.21.15.1:0/2289646744 >> 172.21.15.1:0/2289646744 conn(0x7f01e0072d70 msgr2=0x7f01e0075180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:56.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.970+0000 7f01e5e00700 1 -- 172.21.15.1:0/2289646744 shutdown_connections 2024-02-28T17:27:56.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.970+0000 7f01e5e00700 1 -- 172.21.15.1:0/2289646744 wait complete. 2024-02-28T17:27:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.971+0000 7f01e5e00700 1 Processor -- start 2024-02-28T17:27:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.971+0000 7f01e5e00700 1 -- start start 2024-02-28T17:27:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.972+0000 7f01e5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e0103dc0 0x7f01e011a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.972+0000 7f01e5e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f01e01095b0 0x7f01e011a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.972+0000 7f01e5e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e011ac30 con 0x7f01e0103dc0 2024-02-28T17:27:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.972+0000 7f01e5e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0113f40 con 0x7f01e01095b0 2024-02-28T17:27:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.972+0000 7f01dea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e0103dc0 0x7f01e011a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.972+0000 7f01dea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e0103dc0 0x7f01e011a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53870/0 (socket says 172.21.15.1:53870) 2024-02-28T17:27:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.972+0000 7f01dea00700 1 -- 172.21.15.1:0/2580670020 learned_addr learned my addr 172.21.15.1:0/2580670020 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:27:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.972+0000 7f01de000700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f01e01095b0 0x7f01e011a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.973+0000 7f01dea00700 1 -- 172.21.15.1:0/2580670020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f01e01095b0 msgr2=0x7f01e011a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.973+0000 7f01dea00700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f01e01095b0 0x7f01e011a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.973+0000 7f01dea00700 1 -- 172.21.15.1:0/2580670020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0005040 con 0x7f01e0103dc0 2024-02-28T17:27:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.973+0000 7f01de000700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f01e01095b0 0x7f01e011a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-28T17:27:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.973+0000 7f01dea00700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e0103dc0 0x7f01e011a0a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f01d00090d0 tx=0x7f01d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.973+0000 7f01d6a00700 1 -- 172.21.15.1:0/2580670020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01d000e040 con 0x7f01e0103dc0 2024-02-28T17:27:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.974+0000 7f01d6a00700 1 -- 172.21.15.1:0/2580670020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01d000bbf0 con 0x7f01e0103dc0 2024-02-28T17:27:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.974+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e01141c0 con 0x7f01e0103dc0 2024-02-28T17:27:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.974+0000 7f01d6a00700 1 -- 172.21.15.1:0/2580670020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01d001c630 con 0x7f01e0103dc0 2024-02-28T17:27:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.974+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e0114700 con 0x7f01e0103dc0 2024-02-28T17:27:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.975+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e0106850 con 0x7f01e0103dc0 2024-02-28T17:27:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.977+0000 7f01d6a00700 1 -- 172.21.15.1:0/2580670020 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f01d000bd60 con 0x7f01e0103dc0 2024-02-28T17:27:56.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.978+0000 7f01d6a00700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f01c406d7e0 0x7f01c406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:27:56.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.978+0000 7f01d6a00700 1 -- 172.21.15.1:0/2580670020 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f01d008e2e0 con 0x7f01e0103dc0 2024-02-28T17:27:56.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.978+0000 7f01de000700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f01c406d7e0 0x7f01c406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:27:56.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.979+0000 7f01de000700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f01c406d7e0 0x7f01c406fc90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009fd0 tx=0x7f01cc009e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:27:56.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:56.984+0000 7f01d6a00700 1 -- 172.21.15.1:0/2580670020 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f01d005b640 con 0x7f01e0103dc0 2024-02-28T17:27:57.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.300+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f01e004bed0 con 0x7f01e0103dc0 2024-02-28T17:27:57.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.301+0000 7f01d6a00700 1 -- 172.21.15.1:0/2580670020 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f01d0013e60 con 0x7f01e0103dc0 2024-02-28T17:27:57.304 INFO:teuthology.orchestra.run.smithi001.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T17:27:57.304 INFO:teuthology.orchestra.run.smithi001.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:27:57.304 INFO:teuthology.orchestra.run.smithi001.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:27:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.304+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f01c406d7e0 msgr2=0x7f01c406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.304+0000 7f01e5e00700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f01c406d7e0 0x7f01c406fc90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009fd0 tx=0x7f01cc009e50 comp rx=0 tx=0).stop 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.304+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e0103dc0 msgr2=0x7f01e011a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.304+0000 7f01e5e00700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e0103dc0 0x7f01e011a0a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f01d00090d0 tx=0x7f01d0009e30 comp rx=0 tx=0).stop 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.305+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 shutdown_connections 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.305+0000 7f01e5e00700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f01c406d7e0 0x7f01c406fc90 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.305+0000 7f01e5e00700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e0103dc0 0x7f01e011a0a0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.305+0000 7f01e5e00700 1 --2- 172.21.15.1:0/2580670020 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f01e01095b0 0x7f01e011a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.305+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 >> 172.21.15.1:0/2580670020 conn(0x7f01e0072d70 msgr2=0x7f01e0074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.305+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 shutdown_connections 2024-02-28T17:27:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:27:57.305+0000 7f01e5e00700 1 -- 172.21.15.1:0/2580670020 wait complete. 2024-02-28T17:27:58.012 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph versions' 2024-02-28T17:27:58.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:58 smithi001 ceph-mon[112940]: pgmap v1611: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:58.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:27:58 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2580670020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:27:58.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:58 smithi066 ceph-mon[85781]: pgmap v1611: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:27:58.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:27:58 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2580670020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-28T17:27:58.517 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:28:00.153 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:00 smithi001 ceph-mon[112940]: pgmap v1612: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:00.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:00 smithi066 ceph-mon[85781]: pgmap v1612: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.601+0000 7fafe8e00700 1 -- 172.21.15.1:0/2229092812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 msgr2=0x7fafe4104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:28:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.601+0000 7fafe8e00700 1 --2- 172.21.15.1:0/2229092812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 0x7fafe4104210 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fafd0004940 tx=0x7fafd0009e30 comp rx=0 tx=0).stop 2024-02-28T17:28:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.602+0000 7fafe8e00700 1 -- 172.21.15.1:0/2229092812 shutdown_connections 2024-02-28T17:28:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.602+0000 7fafe8e00700 1 --2- 172.21.15.1:0/2229092812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 0x7fafe4104210 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:28:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.602+0000 7fafe8e00700 1 --2- 172.21.15.1:0/2229092812 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe41095f0 0x7fafe41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:28:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.602+0000 7fafe8e00700 1 -- 172.21.15.1:0/2229092812 >> 172.21.15.1:0/2229092812 conn(0x7fafe4072d50 msgr2=0x7fafe4075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:28:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.602+0000 7fafe8e00700 1 -- 172.21.15.1:0/2229092812 shutdown_connections 2024-02-28T17:28:00.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.602+0000 7fafe8e00700 1 -- 172.21.15.1:0/2229092812 wait complete. 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.603+0000 7fafe8e00700 1 Processor -- start 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.603+0000 7fafe8e00700 1 -- start start 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.603+0000 7fafe8e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 0x7fafe411a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.603+0000 7fafe8e00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe41095f0 0x7fafe411a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.603+0000 7fafe8e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe411ac50 con 0x7fafe4103dc0 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.603+0000 7fafe8e00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe4113ed0 con 0x7fafe41095f0 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.604+0000 7fafe1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 0x7fafe411a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.604+0000 7fafe1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 0x7fafe411a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53888/0 (socket says 172.21.15.1:53888) 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.604+0000 7fafe1e00700 1 -- 172.21.15.1:0/315844728 learned_addr learned my addr 172.21.15.1:0/315844728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:28:00.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.604+0000 7fafe1400700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe41095f0 0x7fafe411a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:28:00.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.604+0000 7fafe1e00700 1 -- 172.21.15.1:0/315844728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe41095f0 msgr2=0x7fafe411a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:28:00.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.604+0000 7fafe1e00700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe41095f0 0x7fafe411a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:28:00.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.604+0000 7fafe1e00700 1 -- 172.21.15.1:0/315844728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd0005040 con 0x7fafe4103dc0 2024-02-28T17:28:00.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.604+0000 7fafe1e00700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 0x7fafe411a030 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fafd400b790 tx=0x7fafd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:28:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.605+0000 7fafda000700 1 -- 172.21.15.1:0/315844728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd400f770 con 0x7fafe4103dc0 2024-02-28T17:28:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.605+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe4114150 con 0x7fafe4103dc0 2024-02-28T17:28:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.605+0000 7fafda000700 1 -- 172.21.15.1:0/315844728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafd400fda0 con 0x7fafe4103dc0 2024-02-28T17:28:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.605+0000 7fafda000700 1 -- 172.21.15.1:0/315844728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd400d310 con 0x7fafe4103dc0 2024-02-28T17:28:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.605+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe4114690 con 0x7fafe4103dc0 2024-02-28T17:28:00.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.606+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe4075870 con 0x7fafe4103dc0 2024-02-28T17:28:00.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.607+0000 7fafda000700 1 -- 172.21.15.1:0/315844728 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fafd4010420 con 0x7fafe4103dc0 2024-02-28T17:28:00.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.608+0000 7fafda000700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fafc806d7f0 0x7fafc806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:28:00.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.608+0000 7fafda000700 1 -- 172.21.15.1:0/315844728 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fafd408cfb0 con 0x7fafe4103dc0 2024-02-28T17:28:00.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.609+0000 7fafe1400700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fafc806d7f0 0x7fafc806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:28:00.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.609+0000 7fafe1400700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fafc806d7f0 0x7fafc806fca0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fafd000b020 tx=0x7fafd0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:28:00.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.613+0000 7fafda000700 1 -- 172.21.15.1:0/315844728 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fafd40c0d20 con 0x7fafe4103dc0 2024-02-28T17:28:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.933+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fafe4115710 con 0x7fafe4103dc0 2024-02-28T17:28:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.934+0000 7fafda000700 1 -- 172.21.15.1:0/315844728 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fafd405a310 con 0x7fafe4103dc0 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:28:00.938 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:28:00.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:28:00.939 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:28:00.939 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:28:00.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:28:00.939 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fafc806d7f0 msgr2=0x7fafc806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fafc806d7f0 0x7fafc806fca0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fafd000b020 tx=0x7fafd0009e30 comp rx=0 tx=0).stop 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 msgr2=0x7fafe411a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 0x7fafe411a030 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fafd400b790 tx=0x7fafd4009250 comp rx=0 tx=0).stop 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 shutdown_connections 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fafc806d7f0 0x7fafc806fca0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe4103dc0 0x7fafe411a030 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 --2- 172.21.15.1:0/315844728 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fafe41095f0 0x7fafe411a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 >> 172.21.15.1:0/315844728 conn(0x7fafe4072d50 msgr2=0x7fafe40746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:28:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 shutdown_connections 2024-02-28T17:28:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:28:00.938+0000 7fafe8e00700 1 -- 172.21.15.1:0/315844728 wait complete. 2024-02-28T17:28:01.346 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:01 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/315844728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:28:01.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:01 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/315844728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:28:01.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-28T17:28:02.238 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:28:02.389 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:02 smithi001 ceph-mon[112940]: pgmap v1613: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:02.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:02 smithi066 ceph-mon[85781]: pgmap v1613: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:04.284 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:04 smithi001 ceph-mon[112940]: pgmap v1614: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:04.399 INFO:teuthology.orchestra.run.smithi001.stdout:wait for servicemap items w/ changing names to refresh 2024-02-28T17:28:04.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:04 smithi066 ceph-mon[85781]: pgmap v1614: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:05.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'sleep 60' 2024-02-28T17:28:05.641 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:28:06.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:06 smithi066 ceph-mon[85781]: pgmap v1615: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:06.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:06 smithi001 ceph-mon[112940]: pgmap v1615: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:08 smithi066 ceph-mon[85781]: pgmap v1616: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:08.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:08 smithi001 ceph-mon[112940]: pgmap v1616: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:10.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:10 smithi066 ceph-mon[85781]: pgmap v1617: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:10.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:10 smithi001 ceph-mon[112940]: pgmap v1617: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:11.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:28:11.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:28:12.413 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:12 smithi066 ceph-mon[85781]: pgmap v1618: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:12.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:12 smithi001 ceph-mon[112940]: pgmap v1618: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:14.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:14 smithi066 ceph-mon[85781]: pgmap v1619: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:14.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:14 smithi001 ceph-mon[112940]: pgmap v1619: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:16.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:16 smithi066 ceph-mon[85781]: pgmap v1620: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:16.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:16 smithi001 ceph-mon[112940]: pgmap v1620: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:18.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:18 smithi066 ceph-mon[85781]: pgmap v1621: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:18.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:18 smithi001 ceph-mon[112940]: pgmap v1621: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:20.506 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:20 smithi001 ceph-mon[112940]: pgmap v1622: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:20.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:20 smithi066 ceph-mon[85781]: pgmap v1622: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:22.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:22 smithi066 ceph-mon[85781]: pgmap v1623: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:22.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:22 smithi001 ceph-mon[112940]: pgmap v1623: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:24.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:24 smithi066 ceph-mon[85781]: pgmap v1624: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:24.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:24 smithi001 ceph-mon[112940]: pgmap v1624: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:26.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:26 smithi066 ceph-mon[85781]: pgmap v1625: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:26.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:28:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:26 smithi001 ceph-mon[112940]: pgmap v1625: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:28:28.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:28 smithi066 ceph-mon[85781]: pgmap v1626: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:28.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:28 smithi001 ceph-mon[112940]: pgmap v1626: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:29.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:29 smithi066 ceph-mon[85781]: pgmap v1627: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:29.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:29 smithi001 ceph-mon[112940]: pgmap v1627: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:32.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:32 smithi066 ceph-mon[85781]: pgmap v1628: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:32.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:32 smithi001 ceph-mon[112940]: pgmap v1628: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:34.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:34 smithi066 ceph-mon[85781]: pgmap v1629: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:34.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:34 smithi001 ceph-mon[112940]: pgmap v1629: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:36.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:36 smithi066 ceph-mon[85781]: pgmap v1630: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:36.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:36 smithi001 ceph-mon[112940]: pgmap v1630: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:38.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:38 smithi066 ceph-mon[85781]: pgmap v1631: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:38.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:38 smithi001 ceph-mon[112940]: pgmap v1631: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:40.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:40 smithi066 ceph-mon[85781]: pgmap v1632: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:40.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:40 smithi001 ceph-mon[112940]: pgmap v1632: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:41.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:28:41.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:28:42.413 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:42 smithi066 ceph-mon[85781]: pgmap v1633: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:42.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:42 smithi001 ceph-mon[112940]: pgmap v1633: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:44.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:44 smithi066 ceph-mon[85781]: pgmap v1634: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:44.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:44 smithi001 ceph-mon[112940]: pgmap v1634: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:46.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:46 smithi066 ceph-mon[85781]: pgmap v1635: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:46.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:46 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:28:46.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:46 smithi001 ceph-mon[112940]: pgmap v1635: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:46.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:46 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:28:48.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:48 smithi066 ceph-mon[85781]: pgmap v1636: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:48.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:48 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:28:48.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:48 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:28:48.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:48 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:28:48.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:48 smithi001 ceph-mon[112940]: pgmap v1636: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:48.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:28:48.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:28:48.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:28:50.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:50 smithi066 ceph-mon[85781]: pgmap v1637: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:50.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:50 smithi001 ceph-mon[112940]: pgmap v1637: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:52.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:52 smithi066 ceph-mon[85781]: pgmap v1638: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:52.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:52 smithi001 ceph-mon[112940]: pgmap v1638: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:54.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:54 smithi066 ceph-mon[85781]: pgmap v1639: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:54.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:54 smithi001 ceph-mon[112940]: pgmap v1639: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:56 smithi066 ceph-mon[85781]: pgmap v1640: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:28:56.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:56 smithi001 ceph-mon[112940]: pgmap v1640: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:56.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:28:58.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:28:58 smithi066 ceph-mon[85781]: pgmap v1641: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:28:58.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:28:58 smithi001 ceph-mon[112940]: pgmap v1641: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:00.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:00 smithi066 ceph-mon[85781]: pgmap v1642: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:00.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:00 smithi001 ceph-mon[112940]: pgmap v1642: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:02.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:02 smithi066 ceph-mon[85781]: pgmap v1643: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:02.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:02 smithi001 ceph-mon[112940]: pgmap v1643: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:04.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:04 smithi066 ceph-mon[85781]: pgmap v1644: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:04.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:04 smithi001 ceph-mon[112940]: pgmap v1644: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:06.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:06 smithi066 ceph-mon[85781]: pgmap v1645: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:06.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:06 smithi001 ceph-mon[112940]: pgmap v1645: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:08.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph orch ps' 2024-02-28T17:29:08.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:08 smithi001 ceph-mon[112940]: pgmap v1646: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:08 smithi066 ceph-mon[85781]: pgmap v1646: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:08.878 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:29:10.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:10 smithi001 ceph-mon[112940]: pgmap v1647: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:10.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:10 smithi066 ceph-mon[85781]: pgmap v1647: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.848+0000 7f3ee5000700 1 -- 172.21.15.1:0/2804698142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee0070be0 msgr2=0x7f3ee01092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.848+0000 7f3ee5000700 1 --2- 172.21.15.1:0/2804698142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee0070be0 0x7f3ee01092c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0004970 tx=0x7f3ed0009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.848+0000 7f3ee5000700 1 -- 172.21.15.1:0/2804698142 shutdown_connections 2024-02-28T17:29:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.848+0000 7f3ee5000700 1 --2- 172.21.15.1:0/2804698142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee0070be0 0x7f3ee01092c0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.848+0000 7f3ee5000700 1 --2- 172.21.15.1:0/2804698142 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ee00702d0 0x7f3ee00706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.848+0000 7f3ee5000700 1 -- 172.21.15.1:0/2804698142 >> 172.21.15.1:0/2804698142 conn(0x7f3ee00f8ae0 msgr2=0x7f3ee00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.848+0000 7f3ee5000700 1 -- 172.21.15.1:0/2804698142 shutdown_connections 2024-02-28T17:29:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.848+0000 7f3ee5000700 1 -- 172.21.15.1:0/2804698142 wait complete. 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.850+0000 7f3ee5000700 1 Processor -- start 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.850+0000 7f3ee5000700 1 -- start start 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.850+0000 7f3ee5000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee00702d0 0x7f3ee0115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.850+0000 7f3ee5000700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ee0070be0 0x7f3ee0116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.850+0000 7f3ee5000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee01167a0 con 0x7f3ee00702d0 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.850+0000 7f3ee5000700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee010faf0 con 0x7f3ee0070be0 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.851+0000 7f3edde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee00702d0 0x7f3ee0115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.851+0000 7f3edde00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee00702d0 0x7f3ee0115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57388/0 (socket says 172.21.15.1:57388) 2024-02-28T17:29:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.851+0000 7f3edde00700 1 -- 172.21.15.1:0/3724832752 learned_addr learned my addr 172.21.15.1:0/3724832752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.851+0000 7f3edd400700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ee0070be0 0x7f3ee0116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.851+0000 7f3edde00700 1 -- 172.21.15.1:0/3724832752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ee0070be0 msgr2=0x7f3ee0116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.851+0000 7f3edde00700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ee0070be0 0x7f3ee0116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.851+0000 7f3edde00700 1 -- 172.21.15.1:0/3724832752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed0005040 con 0x7f3ee00702d0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.852+0000 7f3edde00700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee00702d0 0x7f3ee0115c10 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00c810 tx=0x7f3ecc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.852+0000 7f3ed6000700 1 -- 172.21.15.1:0/3724832752 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ecc012810 con 0x7f3ee00702d0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.852+0000 7f3ed6000700 1 -- 172.21.15.1:0/3724832752 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ecc012e40 con 0x7f3ee00702d0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.852+0000 7f3ed6000700 1 -- 172.21.15.1:0/3724832752 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ecc00a370 con 0x7f3ee00702d0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.852+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee010fdd0 con 0x7f3ee00702d0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.852+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee01102e0 con 0x7f3ee00702d0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.853+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee0106980 con 0x7f3ee00702d0 2024-02-28T17:29:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.855+0000 7f3ed6000700 1 -- 172.21.15.1:0/3724832752 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3ecc011810 con 0x7f3ee00702d0 2024-02-28T17:29:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.856+0000 7f3ed6000700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ec406d7f0 0x7f3ec406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.856+0000 7f3ed6000700 1 -- 172.21.15.1:0/3724832752 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3ecc08d3f0 con 0x7f3ee00702d0 2024-02-28T17:29:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.857+0000 7f3edd400700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ec406d7f0 0x7f3ec406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:10.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.857+0000 7f3edd400700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ec406d7f0 0x7f3ec406fca0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009fd0 tx=0x7f3ed0009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:10.861+0000 7f3ed6000700 1 -- 172.21.15.1:0/3724832752 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3ecc05a6d0 con 0x7f3ee00702d0 2024-02-28T17:29:11.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.091+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 --> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ee01111f0 con 0x7f3ec406d7f0 2024-02-28T17:29:11.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.121+0000 7f3ed6000700 1 -- 172.21.15.1:0/3724832752 <== mgr.34102 v2:172.21.15.1:6800/3120269352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3ee01111f0 con 0x7f3ec406d7f0 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.smithi001 smithi001 *:9093,9094 running (55m) 98s ago 69m 14.5M - 0.25.0 c8568f914cd2 536b3ee80f90 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi001 smithi001 running (52m) 98s ago 69m 6656k - 19.0.0-1670-g1fb5d106 2272b2c42f9b ae06062a3f89 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:crash.smithi066 smithi066 running (51m) 102s ago 68m 6665k - 19.0.0-1670-g1fb5d106 2272b2c42f9b d3ed4604b498 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:grafana.smithi001 smithi001 *:3000 running (54m) 98s ago 69m 77.4M - 9.4.12 a0ea5e3c46d9 a9ca9eef7cd7 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.awxxdo smithi001 running (47m) 98s ago 62m 160M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 88205beb7946 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi001.rtdwhc smithi001 running (47m) 98s ago 62m 162M - 19.0.0-1670-g1fb5d106 2272b2c42f9b 015a9d56553f 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.pivimo smithi066 running (16m) 102s ago 62m 16.5M - 19.0.0-1670-g1fb5d106 2272b2c42f9b e43fe0274d56 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:mds.cephfs.smithi066.wfbdor smithi066 running (32m) 102s ago 62m 21.6M - 19.0.0-1670-g1fb5d106 2272b2c42f9b a2a2e76b9e0c 2024-02-28T17:29:11.124 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi001.rmthtx smithi001 *:8443,9283,8765 running (57m) 98s ago 71m 519M - 19.0.0-1670-g1fb5d106 2272b2c42f9b bb6864aebb8f 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.smithi066.ztraxe smithi066 *:8443,9283,8765 running (56m) 102s ago 68m 458M - 19.0.0-1670-g1fb5d106 2272b2c42f9b d75f84aa1c41 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi001 smithi001 running (53m) 98s ago 71m 99.8M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d04080cee273 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:mon.smithi066 smithi066 running (53m) 102s ago 68m 64.0M 2048M 19.0.0-1670-g1fb5d106 2272b2c42f9b d17ce055575e 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi001 smithi001 *:9100 running (56m) 98s ago 69m 8820k - 1.5.0 0da6a335fe13 4846eba7571c 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter.smithi066 smithi066 *:9100 running (56m) 102s ago 68m 8619k - 1.5.0 0da6a335fe13 18c744b36641 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (51m) 98s ago 67m 95.5M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cea5750b27a 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi001 running (50m) 98s ago 66m 102M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 2ff850335691 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi001 running (50m) 98s ago 66m 90.1M 1559M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4cd5368c3ac6 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi066 running (49m) 102s ago 65m 111M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 4953ba021406 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi066 running (49m) 102s ago 64m 92.7M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 1e0b6d867560 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi066 running (48m) 102s ago 64m 91.3M 2582M 19.0.0-1670-g1fb5d106 2272b2c42f9b 7cf5aef03cba 2024-02-28T17:29:11.125 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.smithi001 smithi001 *:9095 running (55m) 98s ago 68m 65.0M - 2.43.0 a07b618ecd1d 590c7eaf4130 2024-02-28T17:29:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.123+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ec406d7f0 msgr2=0x7f3ec406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ec406d7f0 0x7f3ec406fca0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009fd0 tx=0x7f3ed0009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee00702d0 msgr2=0x7f3ee0115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee00702d0 0x7f3ee0115c10 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00c810 tx=0x7f3ecc004470 comp rx=0 tx=0).stop 2024-02-28T17:29:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 shutdown_connections 2024-02-28T17:29:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3ec406d7f0 0x7f3ec406fca0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ee00702d0 0x7f3ee0115c10 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 --2- 172.21.15.1:0/3724832752 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3ee0070be0 0x7f3ee0116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 >> 172.21.15.1:0/3724832752 conn(0x7f3ee00f8ae0 msgr2=0x7f3ee0103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 shutdown_connections 2024-02-28T17:29:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:11.124+0000 7f3ee5000700 1 -- 172.21.15.1:0/3724832752 wait complete. 2024-02-28T17:29:11.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:29:11.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:29:12.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph versions' 2024-02-28T17:29:12.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:12 smithi001 ceph-mon[112940]: pgmap v1648: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:12.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:12 smithi001 ceph-mon[112940]: from='client.36824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:29:12.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:12 smithi066 ceph-mon[85781]: pgmap v1648: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:12.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:12 smithi066 ceph-mon[85781]: from='client.36824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-28T17:29:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:29:13.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:13 smithi001 ceph-mon[112940]: pgmap v1649: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:13.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:13 smithi066 ceph-mon[85781]: pgmap v1649: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:14.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.703+0000 7fa05ee00700 1 -- 172.21.15.1:0/999939901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fe8f0 msgr2=0x7fa058107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:14.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.703+0000 7fa05ee00700 1 --2- 172.21.15.1:0/999939901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fe8f0 0x7fa058107050 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fa040004970 tx=0x7fa040009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:14.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.704+0000 7fa05ee00700 1 -- 172.21.15.1:0/999939901 shutdown_connections 2024-02-28T17:29:14.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.704+0000 7fa05ee00700 1 --2- 172.21.15.1:0/999939901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fe8f0 0x7fa058107050 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.704+0000 7fa05ee00700 1 --2- 172.21.15.1:0/999939901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0580fdfe0 0x7fa0580fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.704+0000 7fa05ee00700 1 -- 172.21.15.1:0/999939901 >> 172.21.15.1:0/999939901 conn(0x7fa058071d90 msgr2=0x7fa058072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.704+0000 7fa05ee00700 1 -- 172.21.15.1:0/999939901 shutdown_connections 2024-02-28T17:29:14.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.704+0000 7fa05ee00700 1 -- 172.21.15.1:0/999939901 wait complete. 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.705+0000 7fa05ee00700 1 Processor -- start 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.706+0000 7fa05ee00700 1 -- start start 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.706+0000 7fa05ee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fdfe0 0x7fa0581139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.706+0000 7fa05ee00700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0580fe8f0 0x7fa058113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.706+0000 7fa05ee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa058114580 con 0x7fa0580fdfe0 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.706+0000 7fa05ee00700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05810d8b0 con 0x7fa0580fe8f0 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.706+0000 7fa057e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fdfe0 0x7fa0581139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.706+0000 7fa057e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fdfe0 0x7fa0581139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45686/0 (socket says 172.21.15.1:45686) 2024-02-28T17:29:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.706+0000 7fa057e00700 1 -- 172.21.15.1:0/1201654507 learned_addr learned my addr 172.21.15.1:0/1201654507 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa04f400700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0580fe8f0 0x7fa058113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa057e00700 1 -- 172.21.15.1:0/1201654507 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0580fe8f0 msgr2=0x7fa058113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa057e00700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0580fe8f0 0x7fa058113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa057e00700 1 -- 172.21.15.1:0/1201654507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa040005040 con 0x7fa0580fdfe0 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa057e00700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fdfe0 0x7fa0581139f0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fa04800eb40 tx=0x7fa04800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa054c00700 1 -- 172.21.15.1:0/1201654507 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa048007cc0 con 0x7fa0580fdfe0 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa054c00700 1 -- 172.21.15.1:0/1201654507 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa04800f040 con 0x7fa0580fdfe0 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa054c00700 1 -- 172.21.15.1:0/1201654507 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa048009620 con 0x7fa0580fdfe0 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.707+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05810db30 con 0x7fa0580fdfe0 2024-02-28T17:29:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.708+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05810e070 con 0x7fa0580fdfe0 2024-02-28T17:29:14.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.709+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0581046c0 con 0x7fa0580fdfe0 2024-02-28T17:29:14.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.710+0000 7fa054c00700 1 -- 172.21.15.1:0/1201654507 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa048009780 con 0x7fa0580fdfe0 2024-02-28T17:29:14.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.712+0000 7fa054c00700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa03c06d7f0 0x7fa03c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:14.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.712+0000 7fa054c00700 1 -- 172.21.15.1:0/1201654507 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa04808cd80 con 0x7fa0580fdfe0 2024-02-28T17:29:14.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.712+0000 7fa04f400700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa03c06d7f0 0x7fa03c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:14.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.713+0000 7fa04f400700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa03c06d7f0 0x7fa03c06fca0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa04000b370 tx=0x7fa040009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:14.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:14.716+0000 7fa054c00700 1 -- 172.21.15.1:0/1201654507 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa04805a230 con 0x7fa0580fdfe0 2024-02-28T17:29:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.034+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa0580738b0 con 0x7fa0580fdfe0 2024-02-28T17:29:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.035+0000 7fa054c00700 1 -- 172.21.15.1:0/1201654507 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa048059dc0 con 0x7fa0580fdfe0 2024-02-28T17:29:15.038 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-02-28T17:29:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": { 2024-02-28T17:29:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:29:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:29:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": { 2024-02-28T17:29:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 2 2024-02-28T17:29:15.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": { 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 6 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 4 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "overall": { 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)": 14 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-02-28T17:29:15.039 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-02-28T17:29:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.038+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa03c06d7f0 msgr2=0x7fa03c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.038+0000 7fa05ee00700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa03c06d7f0 0x7fa03c06fca0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa04000b370 tx=0x7fa040009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.038+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fdfe0 msgr2=0x7fa0581139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.038+0000 7fa05ee00700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fdfe0 0x7fa0581139f0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fa04800eb40 tx=0x7fa04800b040 comp rx=0 tx=0).stop 2024-02-28T17:29:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.039+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 shutdown_connections 2024-02-28T17:29:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.039+0000 7fa05ee00700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa03c06d7f0 0x7fa03c06fca0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.039+0000 7fa05ee00700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0580fdfe0 0x7fa0581139f0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.039+0000 7fa05ee00700 1 --2- 172.21.15.1:0/1201654507 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa0580fe8f0 0x7fa058113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.039+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 >> 172.21.15.1:0/1201654507 conn(0x7fa058071d90 msgr2=0x7fa058101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.039+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 shutdown_connections 2024-02-28T17:29:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:15.039+0000 7fa05ee00700 1 -- 172.21.15.1:0/1201654507 wait complete. 2024-02-28T17:29:15.383 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:15 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1201654507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:29:15.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:15 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1201654507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:29:15.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-28T17:29:16.301 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:29:16.459 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:16 smithi001 ceph-mon[112940]: pgmap v1650: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:16.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:16 smithi066 ceph-mon[85781]: pgmap v1650: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:18.248 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:18 smithi001 ceph-mon[112940]: pgmap v1651: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.413+0000 7f3081400700 1 -- 172.21.15.1:0/3703840710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1095f0 msgr2=0x7f307c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.413+0000 7f3081400700 1 --2- 172.21.15.1:0/3703840710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1095f0 0x7f307c1099c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f306c004940 tx=0x7f306c009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.414+0000 7f3081400700 1 -- 172.21.15.1:0/3703840710 shutdown_connections 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.414+0000 7f3081400700 1 --2- 172.21.15.1:0/3703840710 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f307c103dc0 0x7f307c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.414+0000 7f3081400700 1 --2- 172.21.15.1:0/3703840710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c1095f0 0x7f307c1099c0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.414+0000 7f3081400700 1 -- 172.21.15.1:0/3703840710 >> 172.21.15.1:0/3703840710 conn(0x7f307c072d50 msgr2=0x7f307c075160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.414+0000 7f3081400700 1 -- 172.21.15.1:0/3703840710 shutdown_connections 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.414+0000 7f3081400700 1 -- 172.21.15.1:0/3703840710 wait complete. 2024-02-28T17:29:18.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.415+0000 7f3081400700 1 Processor -- start 2024-02-28T17:29:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.415+0000 7f3081400700 1 -- start start 2024-02-28T17:29:18.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.415+0000 7f3081400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c103dc0 0x7f307c11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.415+0000 7f3081400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f307c1095f0 0x7f307c11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.415+0000 7f3081400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c11ac70 con 0x7f307c103dc0 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.415+0000 7f3081400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c113f80 con 0x7f307c1095f0 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f307a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c103dc0 0x7f307c11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f307a800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c103dc0 0x7f307c11a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45710/0 (socket says 172.21.15.1:45710) 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f307a800700 1 -- 172.21.15.1:0/2085994247 learned_addr learned my addr 172.21.15.1:0/2085994247 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f3079e00700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f307c1095f0 0x7f307c11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f307a800700 1 -- 172.21.15.1:0/2085994247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f307c1095f0 msgr2=0x7f307c11a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f307a800700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f307c1095f0 0x7f307c11a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f307a800700 1 -- 172.21.15.1:0/2085994247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c005040 con 0x7f307c103dc0 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f3079e00700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f307c1095f0 0x7f307c11a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:18.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.416+0000 7f307a800700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c103dc0 0x7f307c11a0e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f306c009150 tx=0x7f306c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:18.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.417+0000 7f3072a00700 1 -- 172.21.15.1:0/2085994247 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f306c00e040 con 0x7f307c103dc0 2024-02-28T17:29:18.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.417+0000 7f3072a00700 1 -- 172.21.15.1:0/2085994247 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f306c00bbf0 con 0x7f307c103dc0 2024-02-28T17:29:18.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.417+0000 7f3072a00700 1 -- 172.21.15.1:0/2085994247 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f306c01c630 con 0x7f307c103dc0 2024-02-28T17:29:18.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.417+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c114200 con 0x7f307c103dc0 2024-02-28T17:29:18.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.417+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c114740 con 0x7f307c103dc0 2024-02-28T17:29:18.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.418+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c106850 con 0x7f307c103dc0 2024-02-28T17:29:18.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.419+0000 7f3072a00700 1 -- 172.21.15.1:0/2085994247 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f306c00bd60 con 0x7f307c103dc0 2024-02-28T17:29:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.420+0000 7f3072a00700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f306006d7f0 0x7f306006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.420+0000 7f3072a00700 1 -- 172.21.15.1:0/2085994247 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f306c08e290 con 0x7f307c103dc0 2024-02-28T17:29:18.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.420+0000 7f3079e00700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f306006d7f0 0x7f306006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:18.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.421+0000 7f3079e00700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f306006d7f0 0x7f306006fca0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3068009fd0 tx=0x7f306800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:18.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.425+0000 7f3072a00700 1 -- 172.21.15.1:0/2085994247 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f306c05b670 con 0x7f307c103dc0 2024-02-28T17:29:18.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:18 smithi066 ceph-mon[85781]: pgmap v1651: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:18.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.747+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f307c04bed0 con 0x7f307c103dc0 2024-02-28T17:29:18.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.749+0000 7f3072a00700 1 -- 172.21.15.1:0/2085994247 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f306c05b200 con 0x7f307c103dc0 2024-02-28T17:29:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f306006d7f0 msgr2=0x7f306006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f306006d7f0 0x7f306006fca0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3068009fd0 tx=0x7f306800b040 comp rx=0 tx=0).stop 2024-02-28T17:29:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c103dc0 msgr2=0x7f307c11a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c103dc0 0x7f307c11a0e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f306c009150 tx=0x7f306c009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 shutdown_connections 2024-02-28T17:29:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f306006d7f0 0x7f306006fca0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f307c103dc0 0x7f307c11a0e0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 --2- 172.21.15.1:0/2085994247 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f307c1095f0 0x7f307c11a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 >> 172.21.15.1:0/2085994247 conn(0x7f307c072d50 msgr2=0x7f307c074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 shutdown_connections 2024-02-28T17:29:18.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:18.752+0000 7f3081400700 1 -- 172.21.15.1:0/2085994247 wait complete. 2024-02-28T17:29:18.771 INFO:teuthology.orchestra.run.smithi001.stdout:true 2024-02-28T17:29:19.434 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:19 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2085994247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:29:19.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:19 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2085994247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:29:19.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -e sha1=1fb5d10608dccee73edd6ef8a2c7b804a097c324 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-28T17:29:20.060 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:29:20.367 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:20 smithi001 ceph-mon[112940]: pgmap v1652: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:20.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:20 smithi066 ceph-mon[85781]: pgmap v1652: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.078+0000 7f5cfd200700 1 -- 172.21.15.1:0/2988722808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 msgr2=0x7f5cf8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:22.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.078+0000 7f5cfd200700 1 --2- 172.21.15.1:0/2988722808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 0x7f5cf8103320 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8004940 tx=0x7f5ce8009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.078+0000 7f5cfd200700 1 -- 172.21.15.1:0/2988722808 shutdown_connections 2024-02-28T17:29:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.078+0000 7f5cfd200700 1 --2- 172.21.15.1:0/2988722808 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5cf81038f0 0x7f5cf80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.078+0000 7f5cfd200700 1 --2- 172.21.15.1:0/2988722808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 0x7f5cf8103320 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.078+0000 7f5cfd200700 1 -- 172.21.15.1:0/2988722808 >> 172.21.15.1:0/2988722808 conn(0x7f5cf80f8ae0 msgr2=0x7f5cf80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.078+0000 7f5cfd200700 1 -- 172.21.15.1:0/2988722808 shutdown_connections 2024-02-28T17:29:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.078+0000 7f5cfd200700 1 -- 172.21.15.1:0/2988722808 wait complete. 2024-02-28T17:29:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.079+0000 7f5cfd200700 1 Processor -- start 2024-02-28T17:29:22.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.079+0000 7f5cfd200700 1 -- start start 2024-02-28T17:29:22.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.079+0000 7f5cfd200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 0x7f5cf8115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:22.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.079+0000 7f5cfd200700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5cf81038f0 0x7f5cf81161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.079+0000 7f5cfd200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8116820 con 0x7f5cf8102f50 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.079+0000 7f5cfd200700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf810fb50 con 0x7f5cf81038f0 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.080+0000 7f5cf5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 0x7f5cf8115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.080+0000 7f5cf5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 0x7f5cf8115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36940/0 (socket says 172.21.15.1:36940) 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.080+0000 7f5cf5e00700 1 -- 172.21.15.1:0/1577179012 learned_addr learned my addr 172.21.15.1:0/1577179012 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.080+0000 7f5cf5e00700 1 -- 172.21.15.1:0/1577179012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5cf81038f0 msgr2=0x7f5cf81161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.080+0000 7f5cefe00700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5cf81038f0 0x7f5cf81161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.080+0000 7f5cf5e00700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5cf81038f0 0x7f5cf81161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.080+0000 7f5cf5e00700 1 -- 172.21.15.1:0/1577179012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce8005040 con 0x7f5cf8102f50 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.080+0000 7f5cf5e00700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 0x7f5cf8115c90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009100 tx=0x7f5ce8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.081+0000 7f5cee000700 1 -- 172.21.15.1:0/1577179012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ce800e040 con 0x7f5cf8102f50 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.081+0000 7f5cee000700 1 -- 172.21.15.1:0/1577179012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ce8004290 con 0x7f5cf8102f50 2024-02-28T17:29:22.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.081+0000 7f5cee000700 1 -- 172.21.15.1:0/1577179012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ce801c630 con 0x7f5cf8102f50 2024-02-28T17:29:22.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.081+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf810fdd0 con 0x7f5cf8102f50 2024-02-28T17:29:22.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.081+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf8110310 con 0x7f5cf8102f50 2024-02-28T17:29:22.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.082+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf80fc6e0 con 0x7f5cf8102f50 2024-02-28T17:29:22.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.083+0000 7f5cee000700 1 -- 172.21.15.1:0/1577179012 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5ce800bbf0 con 0x7f5cf8102f50 2024-02-28T17:29:22.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.084+0000 7f5cee000700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cdc06d7e0 0x7f5cdc06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:22.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.084+0000 7f5cee000700 1 -- 172.21.15.1:0/1577179012 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5ce808e1d0 con 0x7f5cf8102f50 2024-02-28T17:29:22.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.084+0000 7f5cefe00700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cdc06d7e0 0x7f5cdc06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:22.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.085+0000 7f5cefe00700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cdc06d7e0 0x7f5cdc06fc90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5cf81114c0 tx=0x7f5ce000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:22.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.089+0000 7f5cee000700 1 -- 172.21.15.1:0/1577179012 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5ce805b5b0 con 0x7f5cf8102f50 2024-02-28T17:29:22.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:22 smithi001 ceph-mon[112940]: pgmap v1653: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.405+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5cf8111230 con 0x7f5cf8102f50 2024-02-28T17:29:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.406+0000 7f5cee000700 1 -- 172.21.15.1:0/1577179012 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5ce805b140 con 0x7f5cf8102f50 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cdc06d7e0 msgr2=0x7f5cdc06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cdc06d7e0 0x7f5cdc06fc90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5cf81114c0 tx=0x7f5ce000b040 comp rx=0 tx=0).stop 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 msgr2=0x7f5cf8115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 0x7f5cf8115c90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009100 tx=0x7f5ce8009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 shutdown_connections 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5cdc06d7e0 0x7f5cdc06fc90 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf8102f50 0x7f5cf8115c90 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 --2- 172.21.15.1:0/1577179012 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5cf81038f0 0x7f5cf81161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:22.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.409+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 >> 172.21.15.1:0/1577179012 conn(0x7f5cf80f8ae0 msgr2=0x7f5cf80fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.410+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 shutdown_connections 2024-02-28T17:29:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:22.410+0000 7f5cfd200700 1 -- 172.21.15.1:0/1577179012 wait complete. 2024-02-28T17:29:22.428 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph version 19.0.0-1670-g1fb5d106 (1fb5d10608dccee73edd6ef8a2c7b804a097c324) squid (dev)" 2024-02-28T17:29:22.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:22 smithi066 ceph-mon[85781]: pgmap v1653: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:23.323 DEBUG:teuthology.parallel:result is None 2024-02-28T17:29:23.323 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-28T17:29:23.335 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T17:29:23.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0118718-d654-11ee-95c3-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-28T17:29:23.429 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:23 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1577179012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:29:23.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:23 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1577179012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-28T17:29:23.859 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/mon.smithi001/config 2024-02-28T17:29:24.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:24 smithi001 ceph-mon[112940]: pgmap v1654: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:24.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:24 smithi066 ceph-mon[85781]: pgmap v1654: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.883+0000 7f1534400700 1 -- 172.21.15.1:0/1448824522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c070be0 msgr2=0x7f152c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:25.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.883+0000 7f1534400700 1 --2- 172.21.15.1:0/1448824522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c070be0 0x7f152c1092c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f1518004970 tx=0x7f1518009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.883+0000 7f1534400700 1 -- 172.21.15.1:0/1448824522 shutdown_connections 2024-02-28T17:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.883+0000 7f1534400700 1 --2- 172.21.15.1:0/1448824522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c070be0 0x7f152c1092c0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.883+0000 7f1534400700 1 --2- 172.21.15.1:0/1448824522 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f152c0702d0 0x7f152c0706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.884+0000 7f1534400700 1 -- 172.21.15.1:0/1448824522 >> 172.21.15.1:0/1448824522 conn(0x7f152c0f8ae0 msgr2=0x7f152c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.884+0000 7f1534400700 1 -- 172.21.15.1:0/1448824522 shutdown_connections 2024-02-28T17:29:25.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.884+0000 7f1534400700 1 -- 172.21.15.1:0/1448824522 wait complete. 2024-02-28T17:29:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.885+0000 7f1534400700 1 Processor -- start 2024-02-28T17:29:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.885+0000 7f1534400700 1 -- start start 2024-02-28T17:29:25.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.885+0000 7f1534400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c0702d0 0x7f152c115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.885+0000 7f1534400700 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f152c070be0 0x7f152c116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.885+0000 7f1534400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c116780 con 0x7f152c0702d0 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.885+0000 7f1534400700 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c10fad0 con 0x7f152c070be0 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.885+0000 7f1531a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c0702d0 0x7f152c115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.886+0000 7f1531a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c0702d0 0x7f152c115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36956/0 (socket says 172.21.15.1:36956) 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.886+0000 7f1531a00700 1 -- 172.21.15.1:0/1037361394 learned_addr learned my addr 172.21.15.1:0/1037361394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.886+0000 7f1531a00700 1 -- 172.21.15.1:0/1037361394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f152c070be0 msgr2=0x7f152c116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.886+0000 7f1531a00700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f152c070be0 0x7f152c116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.886+0000 7f1531a00700 1 -- 172.21.15.1:0/1037361394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1518005040 con 0x7f152c0702d0 2024-02-28T17:29:25.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.886+0000 7f1531a00700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c0702d0 0x7f152c115bf0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f1520002a50 tx=0x7f15200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.886+0000 7f1529600700 1 -- 172.21.15.1:0/1037361394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1520007cc0 con 0x7f152c0702d0 2024-02-28T17:29:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.887+0000 7f1529600700 1 -- 172.21.15.1:0/1037361394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f152000f040 con 0x7f152c0702d0 2024-02-28T17:29:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.887+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152c10fdb0 con 0x7f152c0702d0 2024-02-28T17:29:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.887+0000 7f1529600700 1 -- 172.21.15.1:0/1037361394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1520009640 con 0x7f152c0702d0 2024-02-28T17:29:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.887+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152c1102f0 con 0x7f152c0702d0 2024-02-28T17:29:25.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.888+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152c106980 con 0x7f152c0702d0 2024-02-28T17:29:25.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.889+0000 7f1529600700 1 -- 172.21.15.1:0/1037361394 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f15200097a0 con 0x7f152c0702d0 2024-02-28T17:29:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.890+0000 7f1529600700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f151406d7f0 0x7f151406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.890+0000 7f1529600700 1 -- 172.21.15.1:0/1037361394 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f152008cd20 con 0x7f152c0702d0 2024-02-28T17:29:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.890+0000 7f1531000700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f151406d7f0 0x7f151406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.891+0000 7f1531000700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f151406d7f0 0x7f151406fca0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1518000c00 tx=0x7f1518009e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:25.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:25.895+0000 7f1529600700 1 -- 172.21.15.1:0/1037361394 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f15200564d0 con 0x7f152c0702d0 2024-02-28T17:29:26.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.166+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f152c1111d0 con 0x7f152c0702d0 2024-02-28T17:29:26.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.167+0000 7f1529600700 1 -- 172.21.15.1:0/1037361394 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f1520059b90 con 0x7f152c0702d0 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:e34 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 34 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-02-28T16:26:38.901002+0000 2024-02-28T17:29:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-02-28T17:27:33.852703+0000 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 60 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 80 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 2 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=34438,1=34470} 2024-02-28T17:29:26.171 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3] 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data enabled 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 1 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.awxxdo{0:34438} state up:active seq 698 join_fscid=1 addr [v2:172.21.15.1:6826/3510634910,v1:172.21.15.1:6827/3510634910] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi001.rtdwhc{1:34470} state up:active seq 692 join_fscid=1 addr [v2:172.21.15.1:6828/1993293367,v1:172.21.15.1:6829/1993293367] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-02-28T17:29:26.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:26.173 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.wfbdor{-1:44619} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6824/3536246532,v1:172.21.15.66:6825/3536246532] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:29:26.173 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.cephfs.smithi066.pivimo{-1:44939} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.66:6826/3265306782,v1:172.21.15.66:6827/3265306782] compat {c=[1],r=[1],i=[fff]}] 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.170+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f151406d7f0 msgr2=0x7f151406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.170+0000 7f1534400700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f151406d7f0 0x7f151406fca0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1518000c00 tx=0x7f1518009e30 comp rx=0 tx=0).stop 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c0702d0 msgr2=0x7f152c115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c0702d0 0x7f152c115bf0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f1520002a50 tx=0x7f15200046b0 comp rx=0 tx=0).stop 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 shutdown_connections 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f151406d7f0 0x7f151406fca0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f152c0702d0 0x7f152c115bf0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 --2- 172.21.15.1:0/1037361394 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f152c070be0 0x7f152c116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 >> 172.21.15.1:0/1037361394 conn(0x7f152c0f8ae0 msgr2=0x7f152c103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 shutdown_connections 2024-02-28T17:29:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:26.171+0000 7f1534400700 1 -- 172.21.15.1:0/1037361394 wait complete. 2024-02-28T17:29:26.175 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 34 2024-02-28T17:29:26.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:26 smithi066 ceph-mon[85781]: pgmap v1655: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:26.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:29:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:26 smithi001 ceph-mon[112940]: pgmap v1655: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:29:27.084 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-02-28T17:29:27.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-28T17:29:27.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:27 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1037361394' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:29:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.323+0000 7f1654a00640 1 -- 172.21.15.1:0/3023414147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102b30 msgr2=0x7f16500fc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.323+0000 7f1654a00640 1 --2- 172.21.15.1:0/3023414147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102b30 0x7f16500fc1c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f163c0048d0 tx=0x7f163c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.323+0000 7f1654a00640 1 -- 172.21.15.1:0/3023414147 shutdown_connections 2024-02-28T17:29:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.323+0000 7f1654a00640 1 --2- 172.21.15.1:0/3023414147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102b30 0x7f16500fc1c0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.323+0000 7f1654a00640 1 --2- 172.21.15.1:0/3023414147 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1650102190 0x7f1650102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.323+0000 7f1654a00640 1 -- 172.21.15.1:0/3023414147 >> 172.21.15.1:0/3023414147 conn(0x7f16500f7d60 msgr2=0x7f16500fa170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.323+0000 7f1654a00640 1 -- 172.21.15.1:0/3023414147 shutdown_connections 2024-02-28T17:29:27.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.323+0000 7f1654a00640 1 -- 172.21.15.1:0/3023414147 wait complete. 2024-02-28T17:29:27.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.324+0000 7f1654a00640 1 Processor -- start 2024-02-28T17:29:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.324+0000 7f1654a00640 1 -- start start 2024-02-28T17:29:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.324+0000 7f1654a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102190 0x7f1650114ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:27.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f1654a00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1650102b30 0x7f16501151e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f1654a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16501157e0 con 0x7f1650102190 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f1654a00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165010ed70 con 0x7f1650102b30 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f164e800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102190 0x7f1650114ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f164e800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102190 0x7f1650114ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36978/0 (socket says 172.21.15.1:36978) 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f164e800640 1 -- 172.21.15.1:0/2875215866 learned_addr learned my addr 172.21.15.1:0/2875215866 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f164de00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1650102b30 0x7f16501151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f164e800640 1 -- 172.21.15.1:0/2875215866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1650102b30 msgr2=0x7f16501151e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f164e800640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1650102b30 0x7f16501151e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f164e800640 1 -- 172.21.15.1:0/2875215866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c005040 con 0x7f1650102190 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.325+0000 7f164de00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1650102b30 0x7f16501151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.326+0000 7f164e800640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102190 0x7f1650114ca0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f1640002a50 tx=0x7f1640004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.326+0000 7f1646a00640 1 -- 172.21.15.1:0/2875215866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1640007c30 con 0x7f1650102190 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.326+0000 7f1646a00640 1 -- 172.21.15.1:0/2875215866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f164000f040 con 0x7f1650102190 2024-02-28T17:29:27.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.326+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165010f050 con 0x7f1650102190 2024-02-28T17:29:27.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.326+0000 7f1646a00640 1 -- 172.21.15.1:0/2875215866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164000db50 con 0x7f1650102190 2024-02-28T17:29:27.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.326+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165010f570 con 0x7f1650102190 2024-02-28T17:29:27.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.327+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165004bcc0 con 0x7f1650102190 2024-02-28T17:29:27.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.328+0000 7f1646a00640 1 -- 172.21.15.1:0/2875215866 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f164001c400 con 0x7f1650102190 2024-02-28T17:29:27.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.329+0000 7f1646a00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f163406d760 0x7f163406fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:27.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.329+0000 7f1646a00640 1 -- 172.21.15.1:0/2875215866 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f164008c840 con 0x7f1650102190 2024-02-28T17:29:27.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.329+0000 7f164de00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f163406d760 0x7f163406fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:27.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.330+0000 7f164de00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f163406d760 0x7f163406fc10 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f163c02bda0 tx=0x7f163c009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:27.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.334+0000 7f1646a00640 1 -- 172.21.15.1:0/2875215866 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1640055ff0 con 0x7f1650102190 2024-02-28T17:29:27.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:27 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1037361394' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-28T17:29:27.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.540+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f165010f790 con 0x7f1650102190 2024-02-28T17:29:27.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.543+0000 7f1646a00640 1 -- 172.21.15.1:0/2875215866 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16420 (secure 0 0 0) 0x7f16400596b0 con 0x7f1650102190 2024-02-28T17:29:27.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:27.546 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":84,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","created":"2024-02-28T16:17:42.450467+0000","modified":"2024-02-28T17:12:10.940975+0000","last_up_change":"2024-02-28T16:40:59.596149+0000","last_in_change":"2024-02-28T16:24:36.954989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-28T16:23:15.048529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-28T16:26:37.820888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-28T16:26:38.430385+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac92369f-8144-4695-87fe-f9cd73fb5f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6803","nonce":524238602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6805","nonce":524238602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6809","nonce":524238602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6807","nonce":524238602}]},"public_addr":"172.21.15.1:6803/524238602","cluster_addr":"172.21.15.1:6805/524238602","heartbeat_back_addr":"172.21.15.1:6809/524238602","heartbeat_front_addr":"172.21.15.1:6807/524238602","state":["exists","up"]},{"osd":1,"uuid":"bfd6ea17-fd9b-46ae-b3ff-547d736cd232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6811","nonce":2672566113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6813","nonce":2672566113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6817","nonce":2672566113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6815","nonce":2672566113}]},"public_addr":"172.21.15.1:6811/2672566113","cluster_addr":"172.21.15.1:6813/2672566113","heartbeat_back_addr":"172.21.15.1:6817/2672566113","heartbeat_front_addr":"172.21.15.1:6815/2672566113","state":["exists","up"]},{"osd":2,"uuid":"b713f513-6604-4e88-8cc1-0f963d9f8599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6819","nonce":4011428371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6821","nonce":4011428371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6825","nonce":4011428371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6823","nonce":4011428371}]},"public_addr":"172.21.15.1:6819/4011428371","cluster_addr":"172.21.15.1:6821/4011428371","heartbeat_back_addr":"172.21.15.1:6825/4011428371","heartbeat_front_addr":"172.21.15.1:6823/4011428371","state":["exists","up"]},{"osd":3,"uuid":"4f425b10-672f-4265-a314-5a9a15237257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6801","nonce":2991662728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6803","nonce":2991662728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6807","nonce":2991662728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6805","nonce":2991662728}]},"public_addr":"172.21.15.66:6801/2991662728","cluster_addr":"172.21.15.66:6803/2991662728","heartbeat_back_addr":"172.21.15.66:6807/2991662728","heartbeat_front_addr":"172.21.15.66:6805/2991662728","state":["exists","up"]},{"osd":4,"uuid":"c1f2651b-47f3-444e-9325-22a0a962c3b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6809","nonce":2169044357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6811","nonce":2169044357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6815","nonce":2169044357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6813","nonce":2169044357}]},"public_addr":"172.21.15.66:6809/2169044357","cluster_addr":"172.21.15.66:6811/2169044357","heartbeat_back_addr":"172.21.15.66:6815/2169044357","heartbeat_front_addr":"172.21.15.66:6813/2169044357","state":["exists","up"]},{"osd":5,"uuid":"e83df14f-7d81-4bb7-9617-024ad94de1a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6817","nonce":2636222889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6819","nonce":2636222889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6823","nonce":2636222889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6821","nonce":2636222889}]},"public_addr":"172.21.15.66:6817/2636222889","cluster_addr":"172.21.15.66:6819/2636222889","heartbeat_back_addr":"172.21.15.66:6823/2636222889","heartbeat_front_addr":"172.21.15.66:6821/2636222889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:01.831777+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:36.505002+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:13.091284+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:47.260219+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:24:23.366141+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:25:00.532840+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.66:6825/1481832428":"2024-02-29T16:41:05.748852+0000","172.21.15.1:6800/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/2061955559":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/4223980442":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/3865282714":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/833315191":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/298653290":"2024-02-29T16:20:26.347228+0000","172.21.15.1:6801/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.66:0/3927136860":"2024-02-29T16:31:44.722257+0000","172.21.15.66:0/2885505378":"2024-02-29T16:31:44.722257+0000","172.21.15.66:6828/2196519723":"2024-02-29T16:31:44.722257+0000","172.21.15.1:0/2780166226":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/179187285":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6801/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:6801/4056372531":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6801/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:6800/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1224289276":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/762266793":"2024-02-29T16:29:38.122257+0000","172.21.15.1:0/2866890740":"2024-02-29T16:29:38.122257+0000","172.21.15.1:6801/762266793":"2024-02-29T16:29:38.122257+0000","172.21.15.1:0/202779563":"2024-02-29T16:29:38.122257+0000","172.21.15.66:0/1412047282":"2024-02-29T16:31:44.722257+0000","172.21.15.1:6826/1940162712":"2024-02-29T16:41:05.748809+0000","172.21.15.1:0/1339161018":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/4090914359":"2024-02-29T16:35:40.611415+0000","172.21.15.66:6829/2196519723":"2024-02-29T16:31:44.722257+0000","172.21.15.1:0/968413644":"2024-02-29T16:29:38.122257+0000","172.21.15.66:0/1854919128":"2024-02-29T16:31:44.722257+0000","172.21.15.1:6800/4056372531":"2024-02-29T16:35:40.611415+0000","172.21.15.1:0/4195777758":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1492016386":"2024-02-29T16:29:38.122257+0000","172.21.15.66:6824/1481832428":"2024-02-29T16:41:05.748852+0000","172.21.15.1:0/2160247963":"2024-02-29T16:35:40.611415+0000","172.21.15.1:0/1575378374":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6827/1940162712":"2024-02-29T16:41:05.748809+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-28T17:29:27.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.546+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f163406d760 msgr2=0x7f163406fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:27.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.546+0000 7f1654a00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f163406d760 0x7f163406fc10 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f163c02bda0 tx=0x7f163c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.546+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102190 msgr2=0x7f1650114ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:27.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.546+0000 7f1654a00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102190 0x7f1650114ca0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f1640002a50 tx=0x7f1640004600 comp rx=0 tx=0).stop 2024-02-28T17:29:27.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.547+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 shutdown_connections 2024-02-28T17:29:27.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.547+0000 7f1654a00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f163406d760 0x7f163406fc10 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.547+0000 7f1654a00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1650102b30 0x7f16501151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.547+0000 7f1654a00640 1 --2- 172.21.15.1:0/2875215866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1650102190 0x7f1650114ca0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.547+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 >> 172.21.15.1:0/2875215866 conn(0x7f16500f7d60 msgr2=0x7f16500f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:27.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.547+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 shutdown_connections 2024-02-28T17:29:27.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.547+0000 7f1654a00640 1 -- 172.21.15.1:0/2875215866 wait complete. 2024-02-28T17:29:27.570 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-28T16:23:15.048529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-28T16:26:37.820888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-28T16:26:38.430385+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-28T17:29:27.571 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-28T17:29:27.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.797+0000 7f788de00640 1 -- 172.21.15.1:0/1329119946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78880fe9c0 msgr2=0x7f78880fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:27.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.797+0000 7f788de00640 1 --2- 172.21.15.1:0/1329119946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78880fe9c0 0x7f78880fed90 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f78780048d0 tx=0x7f7878009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.797+0000 7f788de00640 1 -- 172.21.15.1:0/1329119946 shutdown_connections 2024-02-28T17:29:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.797+0000 7f788de00640 1 --2- 172.21.15.1:0/1329119946 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f7888100110 0x7f7888100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.797+0000 7f788de00640 1 --2- 172.21.15.1:0/1329119946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78880fe9c0 0x7f78880fed90 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.797+0000 7f788de00640 1 -- 172.21.15.1:0/1329119946 >> 172.21.15.1:0/1329119946 conn(0x7f78880fa160 msgr2=0x7f78880fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.797+0000 7f788de00640 1 -- 172.21.15.1:0/1329119946 shutdown_connections 2024-02-28T17:29:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.797+0000 7f788de00640 1 -- 172.21.15.1:0/1329119946 wait complete. 2024-02-28T17:29:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.798+0000 7f788de00640 1 Processor -- start 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.798+0000 7f788de00640 1 -- start start 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f788de00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78880fe9c0 0x7f7888114db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f788de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7888100110 0x7f78881152f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f788de00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78881158f0 con 0x7f7888100110 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f788de00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810ee80 con 0x7f78880fe9c0 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f7886000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7888100110 0x7f78881152f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f7886000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7888100110 0x7f78881152f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37010/0 (socket says 172.21.15.1:37010) 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f7886000640 1 -- 172.21.15.1:0/2341746901 learned_addr learned my addr 172.21.15.1:0/2341746901 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f7886000640 1 -- 172.21.15.1:0/2341746901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78880fe9c0 msgr2=0x7f7888114db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f7886000640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78880fe9c0 0x7f7888114db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.799+0000 7f7886000640 1 -- 172.21.15.1:0/2341746901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878005040 con 0x7f7888100110 2024-02-28T17:29:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.800+0000 7f7886000640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7888100110 0x7f78881152f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f7874002a50 tx=0x7f7874004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.800+0000 7f787ea00640 1 -- 172.21.15.1:0/2341746901 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7874007c30 con 0x7f7888100110 2024-02-28T17:29:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.800+0000 7f787ea00640 1 -- 172.21.15.1:0/2341746901 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f787400f040 con 0x7f7888100110 2024-02-28T17:29:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.800+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788810f160 con 0x7f7888100110 2024-02-28T17:29:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.800+0000 7f787ea00640 1 -- 172.21.15.1:0/2341746901 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f787400db50 con 0x7f7888100110 2024-02-28T17:29:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.800+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788810f680 con 0x7f7888100110 2024-02-28T17:29:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.801+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888064030 con 0x7f7888100110 2024-02-28T17:29:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.802+0000 7f787ea00640 1 -- 172.21.15.1:0/2341746901 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f787400dcf0 con 0x7f7888100110 2024-02-28T17:29:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.803+0000 7f787ea00640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f786c06d760 0x7f786c06fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.804+0000 7f787ea00640 1 -- 172.21.15.1:0/2341746901 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f787408c7c0 con 0x7f7888100110 2024-02-28T17:29:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.804+0000 7f7886a00640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f786c06d760 0x7f786c06fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:27.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.805+0000 7f7886a00640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f786c06d760 0x7f786c06fc10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7878034040 tx=0x7f7878009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:27.808+0000 7f787ea00640 1 -- 172.21.15.1:0/2341746901 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7874055ff0 con 0x7f7888100110 2024-02-28T17:29:28.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.008+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78881027f0 con 0x7f7888100110 2024-02-28T17:29:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.009+0000 7f787ea00640 1 -- 172.21.15.1:0/2341746901 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f78740596b0 con 0x7f7888100110 2024-02-28T17:29:28.011 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-02-28T17:29:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f786c06d760 msgr2=0x7f786c06fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f786c06d760 0x7f786c06fc10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7878034040 tx=0x7f7878009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7888100110 msgr2=0x7f78881152f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7888100110 0x7f78881152f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f7874002a50 tx=0x7f7874004600 comp rx=0 tx=0).stop 2024-02-28T17:29:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 shutdown_connections 2024-02-28T17:29:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f786c06d760 0x7f786c06fc10 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7888100110 0x7f78881152f0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 --2- 172.21.15.1:0/2341746901 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f78880fe9c0 0x7f7888114db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 >> 172.21.15.1:0/2341746901 conn(0x7f78880fa160 msgr2=0x7f78880fbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 shutdown_connections 2024-02-28T17:29:28.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.012+0000 7f788de00640 1 -- 172.21.15.1:0/2341746901 wait complete. 2024-02-28T17:29:28.033 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-28T17:29:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.250+0000 7f1e81e00640 1 -- 172.21.15.1:0/1732556239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 msgr2=0x7f1e7c104950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.250+0000 7f1e81e00640 1 --2- 172.21.15.1:0/1732556239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 0x7f1e7c104950 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0048d0 tx=0x7f1e6c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.250+0000 7f1e81e00640 1 -- 172.21.15.1:0/1732556239 shutdown_connections 2024-02-28T17:29:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.250+0000 7f1e81e00640 1 --2- 172.21.15.1:0/1732556239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1e7c104e90 0x7f1e7c0fe5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.250+0000 7f1e81e00640 1 --2- 172.21.15.1:0/1732556239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 0x7f1e7c104950 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.250+0000 7f1e81e00640 1 -- 172.21.15.1:0/1732556239 >> 172.21.15.1:0/1732556239 conn(0x7f1e7c0fa160 msgr2=0x7f1e7c0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.250+0000 7f1e81e00640 1 -- 172.21.15.1:0/1732556239 shutdown_connections 2024-02-28T17:29:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.250+0000 7f1e81e00640 1 -- 172.21.15.1:0/1732556239 wait complete. 2024-02-28T17:29:28.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.251+0000 7f1e81e00640 1 Processor -- start 2024-02-28T17:29:28.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.251+0000 7f1e81e00640 1 -- start start 2024-02-28T17:29:28.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.251+0000 7f1e81e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 0x7f1e7c114df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:28.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.251+0000 7f1e81e00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1e7c104e90 0x7f1e7c115330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:28.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.251+0000 7f1e81e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c115930 con 0x7f1e7c104580 2024-02-28T17:29:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.251+0000 7f1e81e00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c10eec0 con 0x7f1e7c104e90 2024-02-28T17:29:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.252+0000 7f1e7aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 0x7f1e7c114df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.252+0000 7f1e7aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 0x7f1e7c114df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37018/0 (socket says 172.21.15.1:37018) 2024-02-28T17:29:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.252+0000 7f1e7aa00640 1 -- 172.21.15.1:0/4035017520 learned_addr learned my addr 172.21.15.1:0/4035017520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.252+0000 7f1e7aa00640 1 -- 172.21.15.1:0/4035017520 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1e7c104e90 msgr2=0x7f1e7c115330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.252+0000 7f1e7aa00640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1e7c104e90 0x7f1e7c115330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.252+0000 7f1e7aa00640 1 -- 172.21.15.1:0/4035017520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c005040 con 0x7f1e7c104580 2024-02-28T17:29:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.253+0000 7f1e7aa00640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 0x7f1e7c114df0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c031720 tx=0x7f1e6c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.253+0000 7f1e72a00640 1 -- 172.21.15.1:0/4035017520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e6c0094c0 con 0x7f1e7c104580 2024-02-28T17:29:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.253+0000 7f1e72a00640 1 -- 172.21.15.1:0/4035017520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e6c009620 con 0x7f1e7c104580 2024-02-28T17:29:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.253+0000 7f1e72a00640 1 -- 172.21.15.1:0/4035017520 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e6c03b640 con 0x7f1e7c104580 2024-02-28T17:29:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.253+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7c10f140 con 0x7f1e7c104580 2024-02-28T17:29:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.253+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c10f610 con 0x7f1e7c104580 2024-02-28T17:29:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.254+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7c064480 con 0x7f1e7c104580 2024-02-28T17:29:28.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.255+0000 7f1e72a00640 1 -- 172.21.15.1:0/4035017520 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1e6c02f070 con 0x7f1e7c104580 2024-02-28T17:29:28.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.257+0000 7f1e72a00640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1e6006d750 0x7f1e6006fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:28.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.257+0000 7f1e72a00640 1 -- 172.21.15.1:0/4035017520 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1e6c0af560 con 0x7f1e7c104580 2024-02-28T17:29:28.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.257+0000 7f1e7a000640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1e6006d750 0x7f1e6006fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:28.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.258+0000 7f1e7a000640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1e6006d750 0x7f1e6006fc00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c1106b0 tx=0x7f1e68007cb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:28.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.262+0000 7f1e72a00640 1 -- 172.21.15.1:0/4035017520 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1e6c078d10 con 0x7f1e7c104580 2024-02-28T17:29:28.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:28 smithi001 ceph-mon[112940]: pgmap v1656: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:28.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:28 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2875215866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T17:29:28.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:28 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2341746901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T17:29:28.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.462+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f1e7c10f7f0 con 0x7f1e7c104580 2024-02-28T17:29:28.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.462+0000 7f1e72a00640 1 -- 172.21.15.1:0/4035017520 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f1e6c07c3d0 con 0x7f1e7c104580 2024-02-28T17:29:28.467 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-02-28T17:29:28.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1e6006d750 msgr2=0x7f1e6006fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1e6006d750 0x7f1e6006fc00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c1106b0 tx=0x7f1e68007cb0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 msgr2=0x7f1e7c114df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 0x7f1e7c114df0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c031720 tx=0x7f1e6c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 shutdown_connections 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1e6006d750 0x7f1e6006fc00 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1e7c104e90 0x7f1e7c115330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 --2- 172.21.15.1:0/4035017520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7c104580 0x7f1e7c114df0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.465+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 >> 172.21.15.1:0/4035017520 conn(0x7f1e7c0fa160 msgr2=0x7f1e7c0fba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.466+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 shutdown_connections 2024-02-28T17:29:28.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.466+0000 7f1e81e00640 1 -- 172.21.15.1:0/4035017520 wait complete. 2024-02-28T17:29:28.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-28T17:29:28.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:28 smithi066 ceph-mon[85781]: pgmap v1656: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:28.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:28 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2875215866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T17:29:28.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:28 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2341746901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T17:29:28.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.697+0000 7fb127200640 1 -- 172.21.15.1:0/1594801606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 msgr2=0x7fb120104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.697+0000 7fb127200640 1 --2- 172.21.15.1:0/1594801606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 0x7fb120104910 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fb1100048d0 tx=0x7fb110009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.697+0000 7fb127200640 1 -- 172.21.15.1:0/1594801606 shutdown_connections 2024-02-28T17:29:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.697+0000 7fb127200640 1 --2- 172.21.15.1:0/1594801606 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb120104ee0 0x7fb1200fe5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.697+0000 7fb127200640 1 --2- 172.21.15.1:0/1594801606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 0x7fb120104910 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.697+0000 7fb127200640 1 -- 172.21.15.1:0/1594801606 >> 172.21.15.1:0/1594801606 conn(0x7fb1200fa120 msgr2=0x7fb1200fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.697+0000 7fb127200640 1 -- 172.21.15.1:0/1594801606 shutdown_connections 2024-02-28T17:29:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.697+0000 7fb127200640 1 -- 172.21.15.1:0/1594801606 wait complete. 2024-02-28T17:29:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.698+0000 7fb127200640 1 Processor -- start 2024-02-28T17:29:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.698+0000 7fb127200640 1 -- start start 2024-02-28T17:29:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.698+0000 7fb127200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 0x7fb120114e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb127200640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb120104ee0 0x7fb120115390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb127200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb120115a20 con 0x7fb120104540 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb127200640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010ef20 con 0x7fb120104ee0 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb11fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 0x7fb120114e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb11fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 0x7fb120114e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37042/0 (socket says 172.21.15.1:37042) 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb11fe00640 1 -- 172.21.15.1:0/3828390431 learned_addr learned my addr 172.21.15.1:0/3828390431 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb11fe00640 1 -- 172.21.15.1:0/3828390431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb120104ee0 msgr2=0x7fb120115390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb11fe00640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb120104ee0 0x7fb120115390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.699+0000 7fb11fe00640 1 -- 172.21.15.1:0/3828390431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb110005040 con 0x7fb120104540 2024-02-28T17:29:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.700+0000 7fb11fe00640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 0x7fb120114e50 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fb11002bdc0 tx=0x7fb110009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.700+0000 7fb117e00640 1 -- 172.21.15.1:0/3828390431 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb11003d450 con 0x7fb120104540 2024-02-28T17:29:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.700+0000 7fb117e00640 1 -- 172.21.15.1:0/3828390431 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb11003da80 con 0x7fb120104540 2024-02-28T17:29:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.700+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12010f1a0 con 0x7fb120104540 2024-02-28T17:29:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.700+0000 7fb117e00640 1 -- 172.21.15.1:0/3828390431 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb11003ba70 con 0x7fb120104540 2024-02-28T17:29:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.700+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12010f6c0 con 0x7fb120104540 2024-02-28T17:29:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.701+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb120064480 con 0x7fb120104540 2024-02-28T17:29:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.702+0000 7fb117e00640 1 -- 172.21.15.1:0/3828390431 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb11003d5b0 con 0x7fb120104540 2024-02-28T17:29:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.703+0000 7fb117e00640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10406d750 0x7fb10406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.703+0000 7fb117e00640 1 -- 172.21.15.1:0/3828390431 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb11002d040 con 0x7fb120104540 2024-02-28T17:29:28.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.703+0000 7fb11f400640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10406d750 0x7fb10406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.704+0000 7fb11f400640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10406d750 0x7fb10406fc00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb120110710 tx=0x7fb10c006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:28.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.708+0000 7fb117e00640 1 -- 172.21.15.1:0/3828390431 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb110079480 con 0x7fb120104540 2024-02-28T17:29:28.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.912+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb12010b740 con 0x7fb120104540 2024-02-28T17:29:28.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.913+0000 7fb117e00640 1 -- 172.21.15.1:0/3828390431 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb11007cb40 con 0x7fb120104540 2024-02-28T17:29:28.915 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-02-28T17:29:28.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.915+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10406d750 msgr2=0x7fb10406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10406d750 0x7fb10406fc00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb120110710 tx=0x7fb10c006d20 comp rx=0 tx=0).stop 2024-02-28T17:29:28.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 msgr2=0x7fb120114e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:28.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 0x7fb120114e50 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fb11002bdc0 tx=0x7fb110009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 shutdown_connections 2024-02-28T17:29:28.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb10406d750 0x7fb10406fc00 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb120104ee0 0x7fb120115390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 --2- 172.21.15.1:0/3828390431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb120104540 0x7fb120114e50 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:28.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 >> 172.21.15.1:0/3828390431 conn(0x7fb1200fa120 msgr2=0x7fb1200fba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:28.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 shutdown_connections 2024-02-28T17:29:28.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:28.916+0000 7fb127200640 1 -- 172.21.15.1:0/3828390431 wait complete. 2024-02-28T17:29:28.939 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-28T17:29:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.155+0000 7faf7fe00640 1 -- 172.21.15.1:0/2925281977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf80104d80 msgr2=0x7faf800fe410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.155+0000 7faf7fe00640 1 --2- 172.21.15.1:0/2925281977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf80104d80 0x7faf800fe410 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0048d0 tx=0x7faf6c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.156+0000 7faf7fe00640 1 -- 172.21.15.1:0/2925281977 shutdown_connections 2024-02-28T17:29:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.156+0000 7faf7fe00640 1 --2- 172.21.15.1:0/2925281977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf80104d80 0x7faf800fe410 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.156+0000 7faf7fe00640 1 --2- 172.21.15.1:0/2925281977 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf801043e0 0x7faf801047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.156+0000 7faf7fe00640 1 -- 172.21.15.1:0/2925281977 >> 172.21.15.1:0/2925281977 conn(0x7faf800f9ff0 msgr2=0x7faf800fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.156+0000 7faf7fe00640 1 -- 172.21.15.1:0/2925281977 shutdown_connections 2024-02-28T17:29:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.156+0000 7faf7fe00640 1 -- 172.21.15.1:0/2925281977 wait complete. 2024-02-28T17:29:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.157+0000 7faf7fe00640 1 Processor -- start 2024-02-28T17:29:29.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.157+0000 7faf7fe00640 1 -- start start 2024-02-28T17:29:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf801043e0 0x7faf80114c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7fe00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf80104d80 0x7faf801151c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7fe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf801157c0 con 0x7faf801043e0 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7fe00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8010ed50 con 0x7faf80104d80 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf801043e0 0x7faf80114c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf801043e0 0x7faf80114c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37050/0 (socket says 172.21.15.1:37050) 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7ea00640 1 -- 172.21.15.1:0/2739675553 learned_addr learned my addr 172.21.15.1:0/2739675553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7e000640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf80104d80 0x7faf801151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7ea00640 1 -- 172.21.15.1:0/2739675553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf80104d80 msgr2=0x7faf801151c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7ea00640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf80104d80 0x7faf801151c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.158+0000 7faf7ea00640 1 -- 172.21.15.1:0/2739675553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c005040 con 0x7faf801043e0 2024-02-28T17:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.159+0000 7faf7e000640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf80104d80 0x7faf801151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:29.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.159+0000 7faf7ea00640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf801043e0 0x7faf80114c80 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7faf70002a50 tx=0x7faf70004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:29.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.160+0000 7faf76a00640 1 -- 172.21.15.1:0/2739675553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf70007c30 con 0x7faf801043e0 2024-02-28T17:29:29.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.160+0000 7faf76a00640 1 -- 172.21.15.1:0/2739675553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf7000f040 con 0x7faf801043e0 2024-02-28T17:29:29.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.160+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8010f030 con 0x7faf801043e0 2024-02-28T17:29:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.160+0000 7faf76a00640 1 -- 172.21.15.1:0/2739675553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf7000db50 con 0x7faf801043e0 2024-02-28T17:29:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.160+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8010f550 con 0x7faf801043e0 2024-02-28T17:29:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.161+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8004bcc0 con 0x7faf801043e0 2024-02-28T17:29:29.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.163+0000 7faf76a00640 1 -- 172.21.15.1:0/2739675553 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7faf7001c400 con 0x7faf801043e0 2024-02-28T17:29:29.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.164+0000 7faf76a00640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf6406d760 0x7faf6406fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:29.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.164+0000 7faf76a00640 1 -- 172.21.15.1:0/2739675553 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7faf7008c870 con 0x7faf801043e0 2024-02-28T17:29:29.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.164+0000 7faf7e000640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf6406d760 0x7faf6406fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:29.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.165+0000 7faf7e000640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf6406d760 0x7faf6406fc10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7faf6c02bda0 tx=0x7faf6c009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.168+0000 7faf76a00640 1 -- 172.21.15.1:0/2739675553 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faf70056020 con 0x7faf801043e0 2024-02-28T17:29:29.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:29 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4035017520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-28T17:29:29.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:29 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3828390431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-28T17:29:29.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.410+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf80110350 con 0x7faf801043e0 2024-02-28T17:29:29.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.412+0000 7faf76a00640 1 -- 172.21.15.1:0/2739675553 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5009 (secure 0 0 0) 0x7faf700596e0 con 0x7faf801043e0 2024-02-28T17:29:29.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:29.414 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":44939,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6827/3265306782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":3265306782},{"type":"v1","addr":"172.21.15.66:6827","nonce":3265306782}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T17:27:33.852703+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34438,"mds_1":34470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34438":{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":30,"state":"up:active","state_seq":698,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34470":{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":1,"incarnation":30,"state":"up:active","state_seq":692,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:29.415 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 34 2024-02-28T17:29:29.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf6406d760 msgr2=0x7faf6406fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf6406d760 0x7faf6406fc10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7faf6c02bda0 tx=0x7faf6c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf801043e0 msgr2=0x7faf80114c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf801043e0 0x7faf80114c80 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7faf70002a50 tx=0x7faf70004600 comp rx=0 tx=0).stop 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 shutdown_connections 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7faf6406d760 0x7faf6406fc10 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7faf80104d80 0x7faf801151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 --2- 172.21.15.1:0/2739675553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf801043e0 0x7faf80114c80 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 >> 172.21.15.1:0/2739675553 conn(0x7faf800f9ff0 msgr2=0x7faf800fb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 shutdown_connections 2024-02-28T17:29:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.415+0000 7faf7fe00640 1 -- 172.21.15.1:0/2739675553 wait complete. 2024-02-28T17:29:29.435 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 13, 'max_mds': 2, 'flags': 18} 2024-02-28T17:29:29.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 14 2024-02-28T17:29:29.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:29 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4035017520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-28T17:29:29.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:29 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3828390431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-28T17:29:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.646+0000 7f99f0000640 1 -- 172.21.15.1:0/1877093693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104ee0 msgr2=0x7f99e80fe5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.646+0000 7f99f0000640 1 --2- 172.21.15.1:0/1877093693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104ee0 0x7f99e80fe5a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f99d40048d0 tx=0x7f99d4009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.646+0000 7f99f0000640 1 -- 172.21.15.1:0/1877093693 shutdown_connections 2024-02-28T17:29:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.646+0000 7f99f0000640 1 --2- 172.21.15.1:0/1877093693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104ee0 0x7f99e80fe5a0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.646+0000 7f99f0000640 1 --2- 172.21.15.1:0/1877093693 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99e8104540 0x7f99e8104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.646+0000 7f99f0000640 1 -- 172.21.15.1:0/1877093693 >> 172.21.15.1:0/1877093693 conn(0x7f99e80fa120 msgr2=0x7f99e80fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.646+0000 7f99f0000640 1 -- 172.21.15.1:0/1877093693 shutdown_connections 2024-02-28T17:29:29.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.646+0000 7f99f0000640 1 -- 172.21.15.1:0/1877093693 wait complete. 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.647+0000 7f99f0000640 1 Processor -- start 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.647+0000 7f99f0000640 1 -- start start 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.648+0000 7f99f0000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104540 0x7f99e8114df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.648+0000 7f99f0000640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99e8104ee0 0x7f99e8115330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.648+0000 7f99f0000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8115930 con 0x7f99e8104540 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.648+0000 7f99f0000640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e810eec0 con 0x7f99e8104ee0 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.648+0000 7f99ed600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104540 0x7f99e8114df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.648+0000 7f99ed600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104540 0x7f99e8114df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37058/0 (socket says 172.21.15.1:37058) 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.648+0000 7f99ed600640 1 -- 172.21.15.1:0/3985180148 learned_addr learned my addr 172.21.15.1:0/3985180148 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99ecc00640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99e8104ee0 0x7f99e8115330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99ed600640 1 -- 172.21.15.1:0/3985180148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99e8104ee0 msgr2=0x7f99e8115330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99ed600640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99e8104ee0 0x7f99e8115330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99ed600640 1 -- 172.21.15.1:0/3985180148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4005040 con 0x7f99e8104540 2024-02-28T17:29:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99ecc00640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99e8104ee0 0x7f99e8115330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99ed600640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104540 0x7f99e8114df0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f99d8002a50 tx=0x7f99d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99e5600640 1 -- 172.21.15.1:0/3985180148 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d8007c30 con 0x7f99e8104540 2024-02-28T17:29:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99e5600640 1 -- 172.21.15.1:0/3985180148 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99d800f040 con 0x7f99e8104540 2024-02-28T17:29:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e810f1a0 con 0x7f99e8104540 2024-02-28T17:29:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.649+0000 7f99e5600640 1 -- 172.21.15.1:0/3985180148 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d800dbe0 con 0x7f99e8104540 2024-02-28T17:29:29.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.650+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e810f6c0 con 0x7f99e8104540 2024-02-28T17:29:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.651+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e8064480 con 0x7f99e8104540 2024-02-28T17:29:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.652+0000 7f99e5600640 1 -- 172.21.15.1:0/3985180148 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f99d800dd40 con 0x7f99e8104540 2024-02-28T17:29:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.653+0000 7f99e5600640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99cc06d750 0x7f99cc06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.653+0000 7f99e5600640 1 -- 172.21.15.1:0/3985180148 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f99d808cd90 con 0x7f99e8104540 2024-02-28T17:29:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.653+0000 7f99ecc00640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99cc06d750 0x7f99cc06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.654+0000 7f99ecc00640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99cc06d750 0x7f99cc06fc00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f99d4000c00 tx=0x7f99d4009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:29.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.658+0000 7f99e5600640 1 -- 172.21.15.1:0/3985180148 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99d8056660 con 0x7f99e8104540 2024-02-28T17:29:29.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.897+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 14, "format": "json"} v 0) v1 -- 0x7f99e810b6a0 con 0x7f99e8104540 2024-02-28T17:29:29.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.898+0000 7f99e5600640 1 -- 172.21.15.1:0/3985180148 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 14, "format": "json"}]=0 dumped fsmap epoch 14 v34) v1 ==== 107+0+4832 (secure 0 0 0) 0x7f99d8059d20 con 0x7f99e8104540 2024-02-28T17:29:29.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:29.901 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:27:00.455017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14464,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14464":{"gid":14464,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6827/1940162712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1940162712},{"type":"v1","addr":"172.21.15.1:6827","nonce":1940162712}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi066.wfbdor","rank":1,"incarnation":7,"state":"up:active","state_seq":5,"addr":"172.21.15.66:6825/1481832428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1481832428},{"type":"v1","addr":"172.21.15.66:6825","nonce":1481832428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:29.901 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 14 2024-02-28T17:29:29.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99cc06d750 msgr2=0x7f99cc06fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:29.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99cc06d750 0x7f99cc06fc00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f99d4000c00 tx=0x7f99d4009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104540 msgr2=0x7f99e8114df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:29.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104540 0x7f99e8114df0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f99d8002a50 tx=0x7f99d8004600 comp rx=0 tx=0).stop 2024-02-28T17:29:29.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 shutdown_connections 2024-02-28T17:29:29.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f99cc06d750 0x7f99cc06fc00 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f99e8104ee0 0x7f99e8115330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 --2- 172.21.15.1:0/3985180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99e8104540 0x7f99e8114df0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:29.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 >> 172.21.15.1:0/3985180148 conn(0x7f99e80fa120 msgr2=0x7f99e80fba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:29.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 shutdown_connections 2024-02-28T17:29:29.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:29.901+0000 7f99f0000640 1 -- 172.21.15.1:0/3985180148 wait complete. 2024-02-28T17:29:29.922 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 15 2024-02-28T17:29:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.139+0000 7f2b8e000640 1 -- 172.21.15.1:0/2692633543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 msgr2=0x7f2b880648e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.139+0000 7f2b8e000640 1 --2- 172.21.15.1:0/2692633543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 0x7f2b880648e0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f2b700048d0 tx=0x7f2b70009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.139+0000 7f2b8e000640 1 -- 172.21.15.1:0/2692633543 shutdown_connections 2024-02-28T17:29:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.139+0000 7f2b8e000640 1 --2- 172.21.15.1:0/2692633543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 0x7f2b880648e0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.140+0000 7f2b8e000640 1 --2- 172.21.15.1:0/2692633543 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b881023e0 0x7f2b881027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.140+0000 7f2b8e000640 1 -- 172.21.15.1:0/2692633543 >> 172.21.15.1:0/2692633543 conn(0x7f2b88071230 msgr2=0x7f2b88071630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.140+0000 7f2b8e000640 1 -- 172.21.15.1:0/2692633543 shutdown_connections 2024-02-28T17:29:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.140+0000 7f2b8e000640 1 -- 172.21.15.1:0/2692633543 wait complete. 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.141+0000 7f2b8e000640 1 Processor -- start 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.141+0000 7f2b8e000640 1 -- start start 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.141+0000 7f2b8e000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 0x7f2b88114dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.141+0000 7f2b8e000640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b881023e0 0x7f2b88115310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.141+0000 7f2b8e000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b88115910 con 0x7f2b880644b0 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.141+0000 7f2b8e000640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b88115a80 con 0x7f2b881023e0 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.142+0000 7f2b87400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 0x7f2b88114dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.142+0000 7f2b87400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 0x7f2b88114dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37072/0 (socket says 172.21.15.1:37072) 2024-02-28T17:29:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.142+0000 7f2b87400640 1 -- 172.21.15.1:0/2403455645 learned_addr learned my addr 172.21.15.1:0/2403455645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.142+0000 7f2b7ea00640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b881023e0 0x7f2b88115310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.142+0000 7f2b87400640 1 -- 172.21.15.1:0/2403455645 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b881023e0 msgr2=0x7f2b88115310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.142+0000 7f2b87400640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b881023e0 0x7f2b88115310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.142+0000 7f2b87400640 1 -- 172.21.15.1:0/2403455645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b70005040 con 0x7f2b880644b0 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.142+0000 7f2b7ea00640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b881023e0 0x7f2b88115310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.143+0000 7f2b87400640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 0x7f2b88114dd0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800db00 tx=0x7f2b78004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.143+0000 7f2b7fe00640 1 -- 172.21.15.1:0/2403455645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b78017070 con 0x7f2b880644b0 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.143+0000 7f2b7fe00640 1 -- 172.21.15.1:0/2403455645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b7800f040 con 0x7f2b880644b0 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.143+0000 7f2b7fe00640 1 -- 172.21.15.1:0/2403455645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b78003ae0 con 0x7f2b880644b0 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.143+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8810f050 con 0x7f2b880644b0 2024-02-28T17:29:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.143+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8810f570 con 0x7f2b880644b0 2024-02-28T17:29:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.145+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b88064030 con 0x7f2b880644b0 2024-02-28T17:29:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.145+0000 7f2b7fe00640 1 -- 172.21.15.1:0/2403455645 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2b78003c40 con 0x7f2b880644b0 2024-02-28T17:29:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.146+0000 7f2b7fe00640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b6c06d750 0x7f2b6c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.147+0000 7f2b7fe00640 1 -- 172.21.15.1:0/2403455645 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2b7808cfe0 con 0x7f2b880644b0 2024-02-28T17:29:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.147+0000 7f2b7ea00640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b6c06d750 0x7f2b6c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:30.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.148+0000 7f2b7ea00640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b6c06d750 0x7f2b6c06fc00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2b7002bda0 tx=0x7f2b70009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.151+0000 7f2b7fe00640 1 -- 172.21.15.1:0/2403455645 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b78056850 con 0x7f2b880644b0 2024-02-28T17:29:30.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:30 smithi001 ceph-mon[112940]: pgmap v1657: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:30.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2739675553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-28T17:29:30.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:30 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3985180148' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 14, "format": "json"}]: dispatch 2024-02-28T17:29:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.395+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 15, "format": "json"} v 0) v1 -- 0x7f2b881094b0 con 0x7f2b880644b0 2024-02-28T17:29:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.396+0000 7f2b7fe00640 1 -- 172.21.15.1:0/2403455645 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 15, "format": "json"}]=0 dumped fsmap epoch 15 v34) v1 ==== 107+0+4833 (secure 0 0 0) 0x7f2b78059f10 con 0x7f2b880644b0 2024-02-28T17:29:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:30.398 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:29:24.592374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14464,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14464":{"gid":14464,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6827/1940162712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1940162712},{"type":"v1","addr":"172.21.15.1:6827","nonce":1940162712}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi066.wfbdor","rank":1,"incarnation":7,"state":"up:active","state_seq":5,"addr":"172.21.15.66:6825/1481832428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1481832428},{"type":"v1","addr":"172.21.15.66:6825","nonce":1481832428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 15 2024-02-28T17:29:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.398+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b6c06d750 msgr2=0x7f2b6c06fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.398+0000 7f2b8e000640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b6c06d750 0x7f2b6c06fc00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2b7002bda0 tx=0x7f2b70009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 msgr2=0x7f2b88114dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 0x7f2b88114dd0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800db00 tx=0x7f2b78004e60 comp rx=0 tx=0).stop 2024-02-28T17:29:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 shutdown_connections 2024-02-28T17:29:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2b6c06d750 0x7f2b6c06fc00 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2b881023e0 0x7f2b88115310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 --2- 172.21.15.1:0/2403455645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b880644b0 0x7f2b88114dd0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 >> 172.21.15.1:0/2403455645 conn(0x7f2b88071230 msgr2=0x7f2b880f9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:30.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 shutdown_connections 2024-02-28T17:29:30.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.399+0000 7f2b8e000640 1 -- 172.21.15.1:0/2403455645 wait complete. 2024-02-28T17:29:30.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 16 2024-02-28T17:29:30.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:30 smithi066 ceph-mon[85781]: pgmap v1657: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:30.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2739675553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-28T17:29:30.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:30 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3985180148' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 14, "format": "json"}]: dispatch 2024-02-28T17:29:30.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.640+0000 7ff751800640 1 -- 172.21.15.1:0/6528637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 msgr2=0x7ff74c0fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:30.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.640+0000 7ff751800640 1 --2- 172.21.15.1:0/6528637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 0x7ff74c0fc150 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7ff7380048d0 tx=0x7ff738009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.640+0000 7ff751800640 1 -- 172.21.15.1:0/6528637 shutdown_connections 2024-02-28T17:29:30.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.640+0000 7ff751800640 1 --2- 172.21.15.1:0/6528637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 0x7ff74c0fc150 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.640+0000 7ff751800640 1 --2- 172.21.15.1:0/6528637 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff74c102120 0x7ff74c1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.640+0000 7ff751800640 1 -- 172.21.15.1:0/6528637 >> 172.21.15.1:0/6528637 conn(0x7ff74c0f7cf0 msgr2=0x7ff74c0fa100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:30.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.640+0000 7ff751800640 1 -- 172.21.15.1:0/6528637 shutdown_connections 2024-02-28T17:29:30.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.640+0000 7ff751800640 1 -- 172.21.15.1:0/6528637 wait complete. 2024-02-28T17:29:30.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.641+0000 7ff751800640 1 Processor -- start 2024-02-28T17:29:30.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.641+0000 7ff751800640 1 -- start start 2024-02-28T17:29:30.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.641+0000 7ff751800640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff74c102120 0x7ff74c114c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:30.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.641+0000 7ff751800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 0x7ff74c1151c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:30.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.641+0000 7ff751800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c1157c0 con 0x7ff74c102ac0 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.641+0000 7ff751800640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c10ed50 con 0x7ff74c102120 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.642+0000 7ff74b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 0x7ff74c1151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.642+0000 7ff74b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 0x7ff74c1151c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37098/0 (socket says 172.21.15.1:37098) 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.642+0000 7ff74b400640 1 -- 172.21.15.1:0/88068796 learned_addr learned my addr 172.21.15.1:0/88068796 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.642+0000 7ff74be00640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff74c102120 0x7ff74c114c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.642+0000 7ff74b400640 1 -- 172.21.15.1:0/88068796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff74c102120 msgr2=0x7ff74c114c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.642+0000 7ff74b400640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff74c102120 0x7ff74c114c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.642+0000 7ff74b400640 1 -- 172.21.15.1:0/88068796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff738005040 con 0x7ff74c102ac0 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.643+0000 7ff74b400640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 0x7ff74c1151c0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7ff73802bda0 tx=0x7ff738009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.643+0000 7ff743e00640 1 -- 172.21.15.1:0/88068796 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7380094c0 con 0x7ff74c102ac0 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.643+0000 7ff743e00640 1 -- 172.21.15.1:0/88068796 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff738032e50 con 0x7ff74c102ac0 2024-02-28T17:29:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.643+0000 7ff743e00640 1 -- 172.21.15.1:0/88068796 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7380435f0 con 0x7ff74c102ac0 2024-02-28T17:29:30.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.643+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74c10efd0 con 0x7ff74c102ac0 2024-02-28T17:29:30.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.643+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74c10f490 con 0x7ff74c102ac0 2024-02-28T17:29:30.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.644+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74c04bcc0 con 0x7ff74c102ac0 2024-02-28T17:29:30.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.645+0000 7ff743e00640 1 -- 172.21.15.1:0/88068796 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ff73802f070 con 0x7ff74c102ac0 2024-02-28T17:29:30.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.646+0000 7ff743e00640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff73006d760 0x7ff73006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:30.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.646+0000 7ff743e00640 1 -- 172.21.15.1:0/88068796 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff7380aedb0 con 0x7ff74c102ac0 2024-02-28T17:29:30.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.647+0000 7ff74be00640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff73006d760 0x7ff73006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:30.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.647+0000 7ff74be00640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff73006d760 0x7ff73006fc10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff73c009fd0 tx=0x7ff73c009e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:30.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.651+0000 7ff743e00640 1 -- 172.21.15.1:0/88068796 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff738078560 con 0x7ff74c102ac0 2024-02-28T17:29:30.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.895+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 16, "format": "json"} v 0) v1 -- 0x7ff74c10f700 con 0x7ff74c102ac0 2024-02-28T17:29:30.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.897+0000 7ff743e00640 1 -- 172.21.15.1:0/88068796 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 16, "format": "json"}]=0 dumped fsmap epoch 16 v34) v1 ==== 107+0+4834 (secure 0 0 0) 0x7ff73807bc20 con 0x7ff74c102ac0 2024-02-28T17:29:30.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:30.899 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:30:03.631200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14464,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14464":{"gid":14464,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6827/1940162712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1940162712},{"type":"v1","addr":"172.21.15.1:6827","nonce":1940162712}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi066.wfbdor","rank":1,"incarnation":7,"state":"up:active","state_seq":5,"addr":"172.21.15.66:6825/1481832428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1481832428},{"type":"v1","addr":"172.21.15.66:6825","nonce":1481832428}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:30.900 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-02-28T17:29:30.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff73006d760 msgr2=0x7ff73006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:30.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff73006d760 0x7ff73006fc10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff73c009fd0 tx=0x7ff73c009e50 comp rx=0 tx=0).stop 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 msgr2=0x7ff74c1151c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 0x7ff74c1151c0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7ff73802bda0 tx=0x7ff738009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 shutdown_connections 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ff73006d760 0x7ff73006fc10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff74c102ac0 0x7ff74c1151c0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 --2- 172.21.15.1:0/88068796 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ff74c102120 0x7ff74c114c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 >> 172.21.15.1:0/88068796 conn(0x7ff74c0f7cf0 msgr2=0x7ff74c06fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 shutdown_connections 2024-02-28T17:29:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:30.900+0000 7ff751800640 1 -- 172.21.15.1:0/88068796 wait complete. 2024-02-28T17:29:30.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 17 2024-02-28T17:29:31.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.119+0000 7f10e1200640 1 -- 172.21.15.1:0/1898712117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc102740 msgr2=0x7f10dc102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:31.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.119+0000 7f10e1200640 1 --2- 172.21.15.1:0/1898712117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc102740 0x7f10dc102bb0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f10c80048d0 tx=0x7f10c8009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.119+0000 7f10e1200640 1 -- 172.21.15.1:0/1898712117 shutdown_connections 2024-02-28T17:29:31.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.119+0000 7f10e1200640 1 --2- 172.21.15.1:0/1898712117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc102740 0x7f10dc102bb0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.119+0000 7f10e1200640 1 --2- 172.21.15.1:0/1898712117 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10dc108760 0x7f10dc108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.119+0000 7f10e1200640 1 -- 172.21.15.1:0/1898712117 >> 172.21.15.1:0/1898712117 conn(0x7f10dc071fc0 msgr2=0x7f10dc0743d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:31.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.119+0000 7f10e1200640 1 -- 172.21.15.1:0/1898712117 shutdown_connections 2024-02-28T17:29:31.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.119+0000 7f10e1200640 1 -- 172.21.15.1:0/1898712117 wait complete. 2024-02-28T17:29:31.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10e1200640 1 Processor -- start 2024-02-28T17:29:31.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10e1200640 1 -- start start 2024-02-28T17:29:31.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10e1200640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10dc102740 0x7f10dc1190b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:31.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10e1200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc108760 0x7f10dc1195f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10e1200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc119bf0 con 0x7f10dc108760 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10e1200640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc113180 con 0x7f10dc102740 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10daa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc108760 0x7f10dc1195f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10daa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc108760 0x7f10dc1195f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37108/0 (socket says 172.21.15.1:37108) 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.120+0000 7f10daa00640 1 -- 172.21.15.1:0/3456890505 learned_addr learned my addr 172.21.15.1:0/3456890505 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.121+0000 7f10db400640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10dc102740 0x7f10dc1190b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.121+0000 7f10daa00640 1 -- 172.21.15.1:0/3456890505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10dc102740 msgr2=0x7f10dc1190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.121+0000 7f10daa00640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10dc102740 0x7f10dc1190b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.121+0000 7f10daa00640 1 -- 172.21.15.1:0/3456890505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8005040 con 0x7f10dc108760 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.121+0000 7f10db400640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10dc102740 0x7f10dc1190b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:31.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.121+0000 7f10daa00640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc108760 0x7f10dc1195f0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f10c802bda0 tx=0x7f10c8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:31.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.122+0000 7f10d3400640 1 -- 172.21.15.1:0/3456890505 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10c80094c0 con 0x7f10dc108760 2024-02-28T17:29:31.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.122+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc113400 con 0x7f10dc108760 2024-02-28T17:29:31.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.122+0000 7f10d3400640 1 -- 172.21.15.1:0/3456890505 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10c8009620 con 0x7f10dc108760 2024-02-28T17:29:31.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.122+0000 7f10d3400640 1 -- 172.21.15.1:0/3456890505 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10c80435f0 con 0x7f10dc108760 2024-02-28T17:29:31.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.122+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc1138c0 con 0x7f10dc108760 2024-02-28T17:29:31.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.123+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc04bcc0 con 0x7f10dc108760 2024-02-28T17:29:31.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.124+0000 7f10d3400640 1 -- 172.21.15.1:0/3456890505 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f10c802f070 con 0x7f10dc108760 2024-02-28T17:29:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.126+0000 7f10d3400640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10c006d760 0x7f10c006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.126+0000 7f10d3400640 1 -- 172.21.15.1:0/3456890505 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f10c80aeea0 con 0x7f10dc108760 2024-02-28T17:29:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.126+0000 7f10db400640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10c006d760 0x7f10c006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.127+0000 7f10db400640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10c006d760 0x7f10c006fc10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f10cc004c80 tx=0x7f10cc009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.131+0000 7f10d3400640 1 -- 172.21.15.1:0/3456890505 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f10c8078650 con 0x7f10dc108760 2024-02-28T17:29:31.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:31 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2403455645' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-02-28T17:29:31.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:31 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/88068796' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-02-28T17:29:31.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.376+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 17, "format": "json"} v 0) v1 -- 0x7f10dc105950 con 0x7f10dc108760 2024-02-28T17:29:31.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.377+0000 7f10d3400640 1 -- 172.21.15.1:0/3456890505 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 17, "format": "json"}]=0 dumped fsmap epoch 17 v34) v1 ==== 107+0+4833 (secure 0 0 0) 0x7f10c807bd10 con 0x7f10dc108760 2024-02-28T17:29:31.379 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:31.379 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:35:33.958603+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14464,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14464":{"gid":14464,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6827/1940162712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1940162712},{"type":"v1","addr":"172.21.15.1:6827","nonce":1940162712}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi066.wfbdor","rank":1,"incarnation":7,"state":"up:active","state_seq":5,"addr":"172.21.15.66:6825/1481832428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1481832428},{"type":"v1","addr":"172.21.15.66:6825","nonce":1481832428}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:31.379 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-02-28T17:29:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10c006d760 msgr2=0x7f10c006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10c006d760 0x7f10c006fc10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f10cc004c80 tx=0x7f10cc009040 comp rx=0 tx=0).stop 2024-02-28T17:29:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc108760 msgr2=0x7f10dc1195f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc108760 0x7f10dc1195f0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f10c802bda0 tx=0x7f10c8009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 shutdown_connections 2024-02-28T17:29:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f10c006d760 0x7f10c006fc10 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10dc108760 0x7f10dc1195f0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 --2- 172.21.15.1:0/3456890505 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f10dc102740 0x7f10dc1190b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 >> 172.21.15.1:0/3456890505 conn(0x7f10dc071fc0 msgr2=0x7f10dc074d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 shutdown_connections 2024-02-28T17:29:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.380+0000 7f10e1200640 1 -- 172.21.15.1:0/3456890505 wait complete. 2024-02-28T17:29:31.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-02-28T17:29:31.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:31 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2403455645' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-02-28T17:29:31.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:31 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/88068796' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-02-28T17:29:31.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.602+0000 7ffb3c000640 1 -- 172.21.15.1:0/2050847531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 msgr2=0x7ffb3410cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.602+0000 7ffb3c000640 1 --2- 172.21.15.1:0/2050847531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 0x7ffb3410cd40 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7ffb200048d0 tx=0x7ffb20009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.602+0000 7ffb3c000640 1 -- 172.21.15.1:0/2050847531 shutdown_connections 2024-02-28T17:29:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.602+0000 7ffb3c000640 1 --2- 172.21.15.1:0/2050847531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 0x7ffb3410cd40 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.602+0000 7ffb3c000640 1 --2- 172.21.15.1:0/2050847531 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb34073570 0x7ffb34073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.602+0000 7ffb3c000640 1 -- 172.21.15.1:0/2050847531 >> 172.21.15.1:0/2050847531 conn(0x7ffb340fa160 msgr2=0x7ffb340fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.602+0000 7ffb3c000640 1 -- 172.21.15.1:0/2050847531 shutdown_connections 2024-02-28T17:29:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.602+0000 7ffb3c000640 1 -- 172.21.15.1:0/2050847531 wait complete. 2024-02-28T17:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.603+0000 7ffb3c000640 1 Processor -- start 2024-02-28T17:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.603+0000 7ffb3c000640 1 -- start start 2024-02-28T17:29:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.603+0000 7ffb3c000640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb34073570 0x7ffb34119260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.603+0000 7ffb3c000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 0x7ffb341197a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.603+0000 7ffb3c000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb34119da0 con 0x7ffb34073e80 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.603+0000 7ffb3c000640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb34113330 con 0x7ffb34073570 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.604+0000 7ffb38c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 0x7ffb341197a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.604+0000 7ffb38c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 0x7ffb341197a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37130/0 (socket says 172.21.15.1:37130) 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.604+0000 7ffb38c00640 1 -- 172.21.15.1:0/263034228 learned_addr learned my addr 172.21.15.1:0/263034228 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.604+0000 7ffb38c00640 1 -- 172.21.15.1:0/263034228 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb34073570 msgr2=0x7ffb34119260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.604+0000 7ffb38c00640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb34073570 0x7ffb34119260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.604+0000 7ffb38c00640 1 -- 172.21.15.1:0/263034228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb20005040 con 0x7ffb34073e80 2024-02-28T17:29:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.604+0000 7ffb38c00640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 0x7ffb341197a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7ffb200047f0 tx=0x7ffb20009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.605+0000 7ffb31600640 1 -- 172.21.15.1:0/263034228 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb200094c0 con 0x7ffb34073e80 2024-02-28T17:29:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.605+0000 7ffb31600640 1 -- 172.21.15.1:0/263034228 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb20009620 con 0x7ffb34073e80 2024-02-28T17:29:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.605+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb341135b0 con 0x7ffb34073e80 2024-02-28T17:29:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.605+0000 7ffb31600640 1 -- 172.21.15.1:0/263034228 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb200435f0 con 0x7ffb34073e80 2024-02-28T17:29:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.605+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb34113a70 con 0x7ffb34073e80 2024-02-28T17:29:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.606+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb34064030 con 0x7ffb34073e80 2024-02-28T17:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.607+0000 7ffb31600640 1 -- 172.21.15.1:0/263034228 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffb2002f070 con 0x7ffb34073e80 2024-02-28T17:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.608+0000 7ffb31600640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb1c06d760 0x7ffb1c06fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.608+0000 7ffb31600640 1 -- 172.21.15.1:0/263034228 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffb200aee50 con 0x7ffb34073e80 2024-02-28T17:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.608+0000 7ffb39600640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb1c06d760 0x7ffb1c06fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:31.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.609+0000 7ffb39600640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb1c06d760 0x7ffb1c06fc10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ffb28004a50 tx=0x7ffb28007cb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:31.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.613+0000 7ffb31600640 1 -- 172.21.15.1:0/263034228 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffb200786a0 con 0x7ffb34073e80 2024-02-28T17:29:31.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.853+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7ffb3410a470 con 0x7ffb34073e80 2024-02-28T17:29:31.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.854+0000 7ffb31600640 1 -- 172.21.15.1:0/263034228 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v34) v1 ==== 107+0+4832 (secure 0 0 0) 0x7ffb2007bd60 con 0x7ffb34073e80 2024-02-28T17:29:31.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:31.856 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:35:47.667892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14464,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14464":{"gid":14464,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6827/1940162712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1940162712},{"type":"v1","addr":"172.21.15.1:6827","nonce":1940162712}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi066.wfbdor","rank":1,"incarnation":7,"state":"up:active","state_seq":5,"addr":"172.21.15.66:6825/1481832428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1481832428},{"type":"v1","addr":"172.21.15.66:6825","nonce":1481832428}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:31.857 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 18 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb1c06d760 msgr2=0x7ffb1c06fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb1c06d760 0x7ffb1c06fc10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ffb28004a50 tx=0x7ffb28007cb0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 msgr2=0x7ffb341197a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 0x7ffb341197a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7ffb200047f0 tx=0x7ffb20009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 shutdown_connections 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffb1c06d760 0x7ffb1c06fc10 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb34073e80 0x7ffb341197a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 --2- 172.21.15.1:0/263034228 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffb34073570 0x7ffb34119260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:31.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 >> 172.21.15.1:0/263034228 conn(0x7ffb340fa160 msgr2=0x7ffb340fbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 shutdown_connections 2024-02-28T17:29:31.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:31.857+0000 7ffb3c000640 1 -- 172.21.15.1:0/263034228 wait complete. 2024-02-28T17:29:31.878 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-02-28T17:29:32.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.095+0000 7ffa65200640 1 -- 172.21.15.1:0/2477598859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 msgr2=0x7ffa6006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:32.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.095+0000 7ffa65200640 1 --2- 172.21.15.1:0/2477598859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 0x7ffa6006fc10 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7ffa500048d0 tx=0x7ffa50009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.095+0000 7ffa65200640 1 -- 172.21.15.1:0/2477598859 shutdown_connections 2024-02-28T17:29:32.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.095+0000 7ffa65200640 1 --2- 172.21.15.1:0/2477598859 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa60070150 0x7ffa601086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.095+0000 7ffa65200640 1 --2- 172.21.15.1:0/2477598859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 0x7ffa6006fc10 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.095+0000 7ffa65200640 1 -- 172.21.15.1:0/2477598859 >> 172.21.15.1:0/2477598859 conn(0x7ffa600f7fa0 msgr2=0x7ffa600fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:32.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.095+0000 7ffa65200640 1 -- 172.21.15.1:0/2477598859 shutdown_connections 2024-02-28T17:29:32.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.095+0000 7ffa65200640 1 -- 172.21.15.1:0/2477598859 wait complete. 2024-02-28T17:29:32.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.096+0000 7ffa65200640 1 Processor -- start 2024-02-28T17:29:32.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.096+0000 7ffa65200640 1 -- start start 2024-02-28T17:29:32.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.096+0000 7ffa65200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 0x7ffa60114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.096+0000 7ffa65200640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa60070150 0x7ffa601153c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.096+0000 7ffa65200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa601159c0 con 0x7ffa6006f840 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.096+0000 7ffa65200640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6010ef50 con 0x7ffa60070150 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.097+0000 7ffa5de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 0x7ffa60114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.097+0000 7ffa5de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 0x7ffa60114e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57202/0 (socket says 172.21.15.1:57202) 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.097+0000 7ffa5de00640 1 -- 172.21.15.1:0/1794232144 learned_addr learned my addr 172.21.15.1:0/1794232144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.097+0000 7ffa5de00640 1 -- 172.21.15.1:0/1794232144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa60070150 msgr2=0x7ffa601153c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.097+0000 7ffa5de00640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa60070150 0x7ffa601153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.097+0000 7ffa5de00640 1 -- 172.21.15.1:0/1794232144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa50005040 con 0x7ffa6006f840 2024-02-28T17:29:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.097+0000 7ffa5de00640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 0x7ffa60114e80 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7ffa500316b0 tx=0x7ffa50009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.098+0000 7ffa56000640 1 -- 172.21.15.1:0/1794232144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa500094c0 con 0x7ffa6006f840 2024-02-28T17:29:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.098+0000 7ffa56000640 1 -- 172.21.15.1:0/1794232144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa50009620 con 0x7ffa6006f840 2024-02-28T17:29:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.098+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6010f1d0 con 0x7ffa6006f840 2024-02-28T17:29:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.098+0000 7ffa56000640 1 -- 172.21.15.1:0/1794232144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa5003b640 con 0x7ffa6006f840 2024-02-28T17:29:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.098+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6010f6f0 con 0x7ffa6006f840 2024-02-28T17:29:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.099+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa60064030 con 0x7ffa6006f840 2024-02-28T17:29:32.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.100+0000 7ffa56000640 1 -- 172.21.15.1:0/1794232144 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7ffa5002f070 con 0x7ffa6006f840 2024-02-28T17:29:32.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.101+0000 7ffa56000640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa4406d750 0x7ffa4406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:32.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.101+0000 7ffa56000640 1 -- 172.21.15.1:0/1794232144 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ffa500af3e0 con 0x7ffa6006f840 2024-02-28T17:29:32.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.102+0000 7ffa5d400640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa4406d750 0x7ffa4406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:32.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.102+0000 7ffa5d400640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa4406d750 0x7ffa4406fc00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffa60110740 tx=0x7ffa4c009c50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.106+0000 7ffa56000640 1 -- 172.21.15.1:0/1794232144 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffa50078b90 con 0x7ffa6006f840 2024-02-28T17:29:32.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.350+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7ffa6010f360 con 0x7ffa6006f840 2024-02-28T17:29:32.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.352+0000 7ffa56000640 1 -- 172.21.15.1:0/1794232144 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v34) v1 ==== 107+0+3258 (secure 0 0 0) 0x7ffa50041cd0 con 0x7ffa6006f840 2024-02-28T17:29:32.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:32.354 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:32.355 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 19 2024-02-28T17:29:32.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.354+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa4406d750 msgr2=0x7ffa4406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:32.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.354+0000 7ffa65200640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa4406d750 0x7ffa4406fc00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffa60110740 tx=0x7ffa4c009c50 comp rx=0 tx=0).stop 2024-02-28T17:29:32.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.354+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 msgr2=0x7ffa60114e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:32.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.354+0000 7ffa65200640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 0x7ffa60114e80 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7ffa500316b0 tx=0x7ffa50009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.355+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 shutdown_connections 2024-02-28T17:29:32.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.355+0000 7ffa65200640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7ffa4406d750 0x7ffa4406fc00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.355+0000 7ffa65200640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7ffa60070150 0x7ffa601153c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.355+0000 7ffa65200640 1 --2- 172.21.15.1:0/1794232144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6006f840 0x7ffa60114e80 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.355+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 >> 172.21.15.1:0/1794232144 conn(0x7ffa600f7fa0 msgr2=0x7ffa60102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:32.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.355+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 shutdown_connections 2024-02-28T17:29:32.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.355+0000 7ffa65200640 1 -- 172.21.15.1:0/1794232144 wait complete. 2024-02-28T17:29:32.379 DEBUG:tasks.fs:max_mds not reduced in epoch 19 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:32.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-02-28T17:29:32.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:32 smithi001 ceph-mon[112940]: pgmap v1658: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:32.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:32 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3456890505' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-02-28T17:29:32.462 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:32 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/263034228' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-02-28T17:29:32.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:32 smithi066 ceph-mon[85781]: pgmap v1658: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:32.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:32 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3456890505' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-02-28T17:29:32.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:32 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/263034228' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-02-28T17:29:32.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.598+0000 7f7773000640 1 -- 172.21.15.1:0/3358711239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0719d0 msgr2=0x7f776c071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.598+0000 7f7773000640 1 --2- 172.21.15.1:0/3358711239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0719d0 0x7f776c071da0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f775c0048d0 tx=0x7f775c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.598+0000 7f7773000640 1 -- 172.21.15.1:0/3358711239 shutdown_connections 2024-02-28T17:29:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.598+0000 7f7773000640 1 --2- 172.21.15.1:0/3358711239 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0722e0 0x7f776c10cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.598+0000 7f7773000640 1 --2- 172.21.15.1:0/3358711239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0719d0 0x7f776c071da0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.598+0000 7f7773000640 1 -- 172.21.15.1:0/3358711239 >> 172.21.15.1:0/3358711239 conn(0x7f776c0fa120 msgr2=0x7f776c0fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.598+0000 7f7773000640 1 -- 172.21.15.1:0/3358711239 shutdown_connections 2024-02-28T17:29:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.598+0000 7f7773000640 1 -- 172.21.15.1:0/3358711239 wait complete. 2024-02-28T17:29:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.599+0000 7f7773000640 1 Processor -- start 2024-02-28T17:29:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.599+0000 7f7773000640 1 -- start start 2024-02-28T17:29:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.599+0000 7f7773000640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0719d0 0x7f776c119190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.599+0000 7f7773000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0722e0 0x7f776c1196d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.599+0000 7f7773000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c119cd0 con 0x7f776c0722e0 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.599+0000 7f7773000640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c113260 con 0x7f776c0719d0 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.600+0000 7f7763e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0722e0 0x7f776c1196d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.600+0000 7f7763e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0722e0 0x7f776c1196d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57212/0 (socket says 172.21.15.1:57212) 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.600+0000 7f7763e00640 1 -- 172.21.15.1:0/2944290958 learned_addr learned my addr 172.21.15.1:0/2944290958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.600+0000 7f776be00640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0719d0 0x7f776c119190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.600+0000 7f7763e00640 1 -- 172.21.15.1:0/2944290958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0719d0 msgr2=0x7f776c119190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.600+0000 7f7763e00640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0719d0 0x7f776c119190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.600+0000 7f7763e00640 1 -- 172.21.15.1:0/2944290958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c005040 con 0x7f776c0722e0 2024-02-28T17:29:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.600+0000 7f7763e00640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0722e0 0x7f776c1196d0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f7754002a50 tx=0x7f7754004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.601+0000 7f7768c00640 1 -- 172.21.15.1:0/2944290958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7754007c30 con 0x7f776c0722e0 2024-02-28T17:29:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.601+0000 7f7768c00640 1 -- 172.21.15.1:0/2944290958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f775400f040 con 0x7f776c0722e0 2024-02-28T17:29:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.601+0000 7f7768c00640 1 -- 172.21.15.1:0/2944290958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f775400dad0 con 0x7f776c0722e0 2024-02-28T17:29:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.601+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776c113540 con 0x7f776c0722e0 2024-02-28T17:29:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.601+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c113a60 con 0x7f776c0722e0 2024-02-28T17:29:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.602+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776c064480 con 0x7f776c0722e0 2024-02-28T17:29:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.603+0000 7f7768c00640 1 -- 172.21.15.1:0/2944290958 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f775400dc70 con 0x7f776c0722e0 2024-02-28T17:29:32.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.604+0000 7f7768c00640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775006d760 0x7f775006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:32.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.604+0000 7f7768c00640 1 -- 172.21.15.1:0/2944290958 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f775408c6e0 con 0x7f776c0722e0 2024-02-28T17:29:32.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.604+0000 7f776be00640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775006d760 0x7f775006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:32.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.605+0000 7f776be00640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775006d760 0x7f775006fc10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f775c034040 tx=0x7f775c009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.609+0000 7f7768c00640 1 -- 172.21.15.1:0/2944290958 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7754055f10 con 0x7f776c0722e0 2024-02-28T17:29:32.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.860+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f776c114790 con 0x7f776c0722e0 2024-02-28T17:29:32.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.861+0000 7f7768c00640 1 -- 172.21.15.1:0/2944290958 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v34) v1 ==== 107+0+4810 (secure 0 0 0) 0x7f77540595d0 con 0x7f776c0722e0 2024-02-28T17:29:32.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:32.864 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34434,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/996956384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":996956384},{"type":"v1","addr":"172.21.15.1:6827","nonce":996956384}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44267,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/1243103917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1243103917},{"type":"v1","addr":"172.21.15.66:6825","nonce":1243103917}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:32.864 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 20 2024-02-28T17:29:32.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775006d760 msgr2=0x7f775006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775006d760 0x7f775006fc10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f775c034040 tx=0x7f775c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0722e0 msgr2=0x7f776c1196d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0722e0 0x7f776c1196d0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f7754002a50 tx=0x7f7754004600 comp rx=0 tx=0).stop 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 shutdown_connections 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f775006d760 0x7f775006fc10 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f776c0722e0 0x7f776c1196d0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 --2- 172.21.15.1:0/2944290958 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f776c0719d0 0x7f776c119190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 >> 172.21.15.1:0/2944290958 conn(0x7f776c0fa120 msgr2=0x7f776c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.864+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 shutdown_connections 2024-02-28T17:29:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:32.865+0000 7f7773000640 1 -- 172.21.15.1:0/2944290958 wait complete. 2024-02-28T17:29:32.888 DEBUG:tasks.fs:max_mds not reduced in epoch 20 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:32.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-02-28T17:29:33.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.101+0000 7fc81b200640 1 -- 172.21.15.1:0/3946163865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104410 msgr2=0x7fc81c1047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.101+0000 7fc81b200640 1 --2- 172.21.15.1:0/3946163865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104410 0x7fc81c1047e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0048d0 tx=0x7fc80c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.101+0000 7fc81b200640 1 -- 172.21.15.1:0/3946163865 shutdown_connections 2024-02-28T17:29:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.101+0000 7fc81b200640 1 --2- 172.21.15.1:0/3946163865 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc81c104db0 0x7fc81c0fe440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.101+0000 7fc81b200640 1 --2- 172.21.15.1:0/3946163865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104410 0x7fc81c1047e0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.101+0000 7fc81b200640 1 -- 172.21.15.1:0/3946163865 >> 172.21.15.1:0/3946163865 conn(0x7fc81c0fa020 msgr2=0x7fc81c0fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.101+0000 7fc81b200640 1 -- 172.21.15.1:0/3946163865 shutdown_connections 2024-02-28T17:29:33.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.101+0000 7fc81b200640 1 -- 172.21.15.1:0/3946163865 wait complete. 2024-02-28T17:29:33.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.102+0000 7fc81b200640 1 Processor -- start 2024-02-28T17:29:33.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.102+0000 7fc81b200640 1 -- start start 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.102+0000 7fc81b200640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc81c104410 0x7fc81c114c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.103+0000 7fc81b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104db0 0x7fc81c1151a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.103+0000 7fc81b200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c1157a0 con 0x7fc81c104db0 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.103+0000 7fc81b200640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c10ed30 con 0x7fc81c104410 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.103+0000 7fc813e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104db0 0x7fc81c1151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.103+0000 7fc813e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104db0 0x7fc81c1151a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57224/0 (socket says 172.21.15.1:57224) 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.103+0000 7fc813e00640 1 -- 172.21.15.1:0/1198376305 learned_addr learned my addr 172.21.15.1:0/1198376305 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.103+0000 7fc819e00640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc81c104410 0x7fc81c114c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc813e00640 1 -- 172.21.15.1:0/1198376305 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc81c104410 msgr2=0x7fc81c114c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc813e00640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc81c104410 0x7fc81c114c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc813e00640 1 -- 172.21.15.1:0/1198376305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80c005040 con 0x7fc81c104db0 2024-02-28T17:29:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc813e00640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104db0 0x7fc81c1151a0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fc804002870 tx=0x7fc804004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc812000640 1 -- 172.21.15.1:0/1198376305 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc804007c30 con 0x7fc81c104db0 2024-02-28T17:29:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc812000640 1 -- 172.21.15.1:0/1198376305 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc80400f040 con 0x7fc81c104db0 2024-02-28T17:29:33.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c10f010 con 0x7fc81c104db0 2024-02-28T17:29:33.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc812000640 1 -- 172.21.15.1:0/1198376305 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc80400da90 con 0x7fc81c104db0 2024-02-28T17:29:33.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.104+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c10f530 con 0x7fc81c104db0 2024-02-28T17:29:33.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.105+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c04bcc0 con 0x7fc81c104db0 2024-02-28T17:29:33.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.107+0000 7fc812000640 1 -- 172.21.15.1:0/1198376305 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc80400dd00 con 0x7fc81c104db0 2024-02-28T17:29:33.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.107+0000 7fc812000640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc80006d760 0x7fc80006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:33.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.108+0000 7fc812000640 1 -- 172.21.15.1:0/1198376305 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc80408c630 con 0x7fc81c104db0 2024-02-28T17:29:33.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.108+0000 7fc819e00640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc80006d760 0x7fc80006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:33.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.109+0000 7fc819e00640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc80006d760 0x7fc80006fc10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc80c034040 tx=0x7fc80c009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:33.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.113+0000 7fc812000640 1 -- 172.21.15.1:0/1198376305 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc804055e60 con 0x7fc81c104db0 2024-02-28T17:29:33.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.355+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7fc81c10b5b0 con 0x7fc81c104db0 2024-02-28T17:29:33.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.356+0000 7fc812000640 1 -- 172.21.15.1:0/1198376305 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v34) v1 ==== 107+0+4037 (secure 0 0 0) 0x7fc804059520 con 0x7fc81c104db0 2024-02-28T17:29:33.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:33.359 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":44267,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/1243103917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1243103917},{"type":"v1","addr":"172.21.15.66:6825","nonce":1243103917}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:33.359 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 21 2024-02-28T17:29:33.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc80006d760 msgr2=0x7fc80006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:33.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc80006d760 0x7fc80006fc10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc80c034040 tx=0x7fc80c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104db0 msgr2=0x7fc81c1151a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104db0 0x7fc81c1151a0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fc804002870 tx=0x7fc804004600 comp rx=0 tx=0).stop 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 shutdown_connections 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc80006d760 0x7fc80006fc10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c104db0 0x7fc81c1151a0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 --2- 172.21.15.1:0/1198376305 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc81c104410 0x7fc81c114c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 >> 172.21.15.1:0/1198376305 conn(0x7fc81c0fa020 msgr2=0x7fc81c0fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 shutdown_connections 2024-02-28T17:29:33.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.359+0000 7fc81b200640 1 -- 172.21.15.1:0/1198376305 wait complete. 2024-02-28T17:29:33.381 DEBUG:tasks.fs:max_mds not reduced in epoch 21 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:33.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-02-28T17:29:33.471 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:33 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1794232144' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-02-28T17:29:33.471 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:33 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2944290958' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-02-28T17:29:33.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:33 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1794232144' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-02-28T17:29:33.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:33 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2944290958' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-02-28T17:29:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.605+0000 7f9dcac00640 1 -- 172.21.15.1:0/3297273082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc406fd80 msgr2=0x7f9dc40701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.605+0000 7f9dcac00640 1 --2- 172.21.15.1:0/3297273082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc406fd80 0x7f9dc40701f0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0048d0 tx=0x7f9dac009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.605+0000 7f9dcac00640 1 -- 172.21.15.1:0/3297273082 shutdown_connections 2024-02-28T17:29:33.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.605+0000 7f9dcac00640 1 --2- 172.21.15.1:0/3297273082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc406fd80 0x7f9dc40701f0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.605+0000 7f9dcac00640 1 --2- 172.21.15.1:0/3297273082 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc4071470 0x7f9dc406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.605+0000 7f9dcac00640 1 -- 172.21.15.1:0/3297273082 >> 172.21.15.1:0/3297273082 conn(0x7f9dc40f7ef0 msgr2=0x7f9dc40fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:33.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.605+0000 7f9dcac00640 1 -- 172.21.15.1:0/3297273082 shutdown_connections 2024-02-28T17:29:33.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.605+0000 7f9dcac00640 1 -- 172.21.15.1:0/3297273082 wait complete. 2024-02-28T17:29:33.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.606+0000 7f9dcac00640 1 Processor -- start 2024-02-28T17:29:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.606+0000 7f9dcac00640 1 -- start start 2024-02-28T17:29:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dcac00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc406fd80 0x7f9dc41191d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dcac00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc4071470 0x7f9dc4119710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dcac00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4119d10 con 0x7f9dc4071470 2024-02-28T17:29:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dcac00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc41132a0 con 0x7f9dc406fd80 2024-02-28T17:29:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dc3400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc4071470 0x7f9dc4119710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dc3400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc4071470 0x7f9dc4119710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57234/0 (socket says 172.21.15.1:57234) 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dc3400640 1 -- 172.21.15.1:0/955152715 learned_addr learned my addr 172.21.15.1:0/955152715 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dc3e00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc406fd80 0x7f9dc41191d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dc3400640 1 -- 172.21.15.1:0/955152715 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc406fd80 msgr2=0x7f9dc41191d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.607+0000 7f9dc3400640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc406fd80 0x7f9dc41191d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.608+0000 7f9dc3400640 1 -- 172.21.15.1:0/955152715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac005040 con 0x7f9dc4071470 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.608+0000 7f9dc3e00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc406fd80 0x7f9dc41191d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.608+0000 7f9dc3400640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc4071470 0x7f9dc4119710 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00b020 tx=0x7f9dac009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.608+0000 7f9dbbe00640 1 -- 172.21.15.1:0/955152715 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dac0092e0 con 0x7f9dc4071470 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.608+0000 7f9dbbe00640 1 -- 172.21.15.1:0/955152715 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dac009440 con 0x7f9dc4071470 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.608+0000 7f9dbbe00640 1 -- 172.21.15.1:0/955152715 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dac03a5f0 con 0x7f9dc4071470 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.608+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc4113520 con 0x7f9dc4071470 2024-02-28T17:29:33.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.608+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc4113a40 con 0x7f9dc4071470 2024-02-28T17:29:33.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.609+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc4064480 con 0x7f9dc4071470 2024-02-28T17:29:33.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.611+0000 7f9dbbe00640 1 -- 172.21.15.1:0/955152715 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f9dac03b420 con 0x7f9dc4071470 2024-02-28T17:29:33.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.612+0000 7f9dbbe00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da806d750 0x7f9da806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:33.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.612+0000 7f9dbbe00640 1 -- 172.21.15.1:0/955152715 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9dac0afc70 con 0x7f9dc4071470 2024-02-28T17:29:33.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.612+0000 7f9dc3e00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da806d750 0x7f9da806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:33.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.613+0000 7f9dc3e00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da806d750 0x7f9da806fc00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9db4004e80 tx=0x7f9db4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:33.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.617+0000 7f9dbbe00640 1 -- 172.21.15.1:0/955152715 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9dac0795e0 con 0x7f9dc4071470 2024-02-28T17:29:33.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.860+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f9dc410fa60 con 0x7f9dc4071470 2024-02-28T17:29:33.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.862+0000 7f9dbbe00640 1 -- 172.21.15.1:0/955152715 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v34) v1 ==== 107+0+4847 (secure 0 0 0) 0x7f9dac07cca0 con 0x7f9dc4071470 2024-02-28T17:29:33.864 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:33.864 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.1:6829/1371327686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1371327686},{"type":"v1","addr":"172.21.15.1:6829","nonce":1371327686}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":44267,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/1243103917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1243103917},{"type":"v1","addr":"172.21.15.66:6825","nonce":1243103917}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:33.865 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 22 2024-02-28T17:29:33.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da806d750 msgr2=0x7f9da806fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:33.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da806d750 0x7f9da806fc00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9db4004e80 tx=0x7f9db4009040 comp rx=0 tx=0).stop 2024-02-28T17:29:33.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc4071470 msgr2=0x7f9dc4119710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:33.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc4071470 0x7f9dc4119710 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00b020 tx=0x7f9dac009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 shutdown_connections 2024-02-28T17:29:33.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f9da806d750 0x7f9da806fc00 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9dc4071470 0x7f9dc4119710 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 --2- 172.21.15.1:0/955152715 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9dc406fd80 0x7f9dc41191d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:33.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 >> 172.21.15.1:0/955152715 conn(0x7f9dc40f7ef0 msgr2=0x7f9dc410b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:33.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 shutdown_connections 2024-02-28T17:29:33.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:33.865+0000 7f9dcac00640 1 -- 172.21.15.1:0/955152715 wait complete. 2024-02-28T17:29:33.885 DEBUG:tasks.fs:max_mds not reduced in epoch 22 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:33.885 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-02-28T17:29:34.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.099+0000 7f90c1a00640 1 -- 172.21.15.1:0/2094878225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073570 msgr2=0x7f90bc073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.099+0000 7f90c1a00640 1 --2- 172.21.15.1:0/2094878225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073570 0x7f90bc073940 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0048d0 tx=0x7f90ac009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.099+0000 7f90c1a00640 1 -- 172.21.15.1:0/2094878225 shutdown_connections 2024-02-28T17:29:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.099+0000 7f90c1a00640 1 --2- 172.21.15.1:0/2094878225 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90bc073e80 0x7f90bc10cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.099+0000 7f90c1a00640 1 --2- 172.21.15.1:0/2094878225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073570 0x7f90bc073940 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.099+0000 7f90c1a00640 1 -- 172.21.15.1:0/2094878225 >> 172.21.15.1:0/2094878225 conn(0x7f90bc0fa160 msgr2=0x7f90bc0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.099+0000 7f90c1a00640 1 -- 172.21.15.1:0/2094878225 shutdown_connections 2024-02-28T17:29:34.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.099+0000 7f90c1a00640 1 -- 172.21.15.1:0/2094878225 wait complete. 2024-02-28T17:29:34.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.100+0000 7f90c1a00640 1 Processor -- start 2024-02-28T17:29:34.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.100+0000 7f90c1a00640 1 -- start start 2024-02-28T17:29:34.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.101+0000 7f90c1a00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90bc073570 0x7f90bc119280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:34.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.101+0000 7f90c1a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073e80 0x7f90bc1197c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:34.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.101+0000 7f90c1a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc119dc0 con 0x7f90bc073e80 2024-02-28T17:29:34.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.101+0000 7f90c1a00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc113350 con 0x7f90bc073570 2024-02-28T17:29:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.101+0000 7f90ba000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073e80 0x7f90bc1197c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.101+0000 7f90ba000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073e80 0x7f90bc1197c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57252/0 (socket says 172.21.15.1:57252) 2024-02-28T17:29:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.101+0000 7f90ba000640 1 -- 172.21.15.1:0/1249856535 learned_addr learned my addr 172.21.15.1:0/1249856535 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90baa00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90bc073570 0x7f90bc119280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90ba000640 1 -- 172.21.15.1:0/1249856535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90bc073570 msgr2=0x7f90bc119280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90ba000640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90bc073570 0x7f90bc119280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90ba000640 1 -- 172.21.15.1:0/1249856535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac005040 con 0x7f90bc073e80 2024-02-28T17:29:34.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90ba000640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073e80 0x7f90bc1197c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f90a8002a50 tx=0x7f90a8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:34.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90b2a00640 1 -- 172.21.15.1:0/1249856535 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a8007c30 con 0x7f90bc073e80 2024-02-28T17:29:34.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc113630 con 0x7f90bc073e80 2024-02-28T17:29:34.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90b2a00640 1 -- 172.21.15.1:0/1249856535 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90a800f040 con 0x7f90bc073e80 2024-02-28T17:29:34.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.102+0000 7f90b2a00640 1 -- 172.21.15.1:0/1249856535 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a800dad0 con 0x7f90bc073e80 2024-02-28T17:29:34.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.103+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc113b50 con 0x7f90bc073e80 2024-02-28T17:29:34.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.104+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc064030 con 0x7f90bc073e80 2024-02-28T17:29:34.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.105+0000 7f90b2a00640 1 -- 172.21.15.1:0/1249856535 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f90a800dc70 con 0x7f90bc073e80 2024-02-28T17:29:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.107+0000 7f90b2a00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f90a006d760 0x7f90a006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.107+0000 7f90b2a00640 1 -- 172.21.15.1:0/1249856535 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f90a808c6e0 con 0x7f90bc073e80 2024-02-28T17:29:34.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.107+0000 7f90baa00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f90a006d760 0x7f90a006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:34.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.108+0000 7f90baa00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f90a006d760 0x7f90a006fc10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0047f0 tx=0x7f90ac009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:34.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.112+0000 7f90b2a00640 1 -- 172.21.15.1:0/1249856535 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f90a8055f10 con 0x7f90bc073e80 2024-02-28T17:29:34.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.360+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f90bc114880 con 0x7f90bc073e80 2024-02-28T17:29:34.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.361+0000 7f90b2a00640 1 -- 172.21.15.1:0/1249856535 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v34) v1 ==== 107+0+4071 (secure 0 0 0) 0x7f90a80595d0 con 0x7f90bc073e80 2024-02-28T17:29:34.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:34.364 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":44267,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/1243103917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1243103917},{"type":"v1","addr":"172.21.15.66:6825","nonce":1243103917}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:34.364 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 23 2024-02-28T17:29:34.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f90a006d760 msgr2=0x7f90a006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f90a006d760 0x7f90a006fc10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0047f0 tx=0x7f90ac009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073e80 msgr2=0x7f90bc1197c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073e80 0x7f90bc1197c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f90a8002a50 tx=0x7f90a8004600 comp rx=0 tx=0).stop 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 shutdown_connections 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f90a006d760 0x7f90a006fc10 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90bc073e80 0x7f90bc1197c0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 --2- 172.21.15.1:0/1249856535 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90bc073570 0x7f90bc119280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 >> 172.21.15.1:0/1249856535 conn(0x7f90bc0fa160 msgr2=0x7f90bc0fbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 shutdown_connections 2024-02-28T17:29:34.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.364+0000 7f90c1a00640 1 -- 172.21.15.1:0/1249856535 wait complete. 2024-02-28T17:29:34.384 DEBUG:tasks.fs:max_mds not reduced in epoch 23 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:34.384 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-02-28T17:29:34.468 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:34 smithi001 ceph-mon[112940]: pgmap v1659: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:34.468 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:34 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1198376305' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-02-28T17:29:34.468 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:34 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/955152715' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-02-28T17:29:34.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:34 smithi066 ceph-mon[85781]: pgmap v1659: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:34.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:34 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1198376305' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-02-28T17:29:34.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:34 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/955152715' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-02-28T17:29:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.598+0000 7f9088a00640 1 -- 172.21.15.1:0/2537189501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 msgr2=0x7f9084100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.598+0000 7f9088a00640 1 --2- 172.21.15.1:0/2537189501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 0x7f9084100560 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f90700048d0 tx=0x7f9070009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.599+0000 7f9088a00640 1 -- 172.21.15.1:0/2537189501 shutdown_connections 2024-02-28T17:29:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.599+0000 7f9088a00640 1 --2- 172.21.15.1:0/2537189501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 0x7f9084100560 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.599+0000 7f9088a00640 1 --2- 172.21.15.1:0/2537189501 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90840fe9c0 0x7f90840fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.599+0000 7f9088a00640 1 -- 172.21.15.1:0/2537189501 >> 172.21.15.1:0/2537189501 conn(0x7f90840fa160 msgr2=0x7f90840fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.599+0000 7f9088a00640 1 -- 172.21.15.1:0/2537189501 shutdown_connections 2024-02-28T17:29:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.599+0000 7f9088a00640 1 -- 172.21.15.1:0/2537189501 wait complete. 2024-02-28T17:29:34.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.599+0000 7f9088a00640 1 Processor -- start 2024-02-28T17:29:34.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.600+0000 7f9088a00640 1 -- start start 2024-02-28T17:29:34.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.600+0000 7f9088a00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90840fe9c0 0x7f9084114de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:34.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.600+0000 7f9088a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 0x7f9084115320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:34.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.600+0000 7f9088a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9084115920 con 0x7f9084100110 2024-02-28T17:29:34.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.600+0000 7f9088a00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908410eeb0 con 0x7f90840fe9c0 2024-02-28T17:29:34.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.600+0000 7f9081400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 0x7f9084115320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:34.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.600+0000 7f9081400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 0x7f9084115320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57274/0 (socket says 172.21.15.1:57274) 2024-02-28T17:29:34.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.600+0000 7f9081400640 1 -- 172.21.15.1:0/1675442770 learned_addr learned my addr 172.21.15.1:0/1675442770 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:34.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.601+0000 7f9081e00640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90840fe9c0 0x7f9084114de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:34.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.601+0000 7f9081400640 1 -- 172.21.15.1:0/1675442770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90840fe9c0 msgr2=0x7f9084114de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:34.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.601+0000 7f9081400640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90840fe9c0 0x7f9084114de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.601+0000 7f9081400640 1 -- 172.21.15.1:0/1675442770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9070005040 con 0x7f9084100110 2024-02-28T17:29:34.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.601+0000 7f9081400640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 0x7f9084115320 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f907002bda0 tx=0x7f9070009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:34.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.601+0000 7f907a000640 1 -- 172.21.15.1:0/1675442770 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90700094c0 con 0x7f9084100110 2024-02-28T17:29:34.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.602+0000 7f907a000640 1 -- 172.21.15.1:0/1675442770 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9070032e50 con 0x7f9084100110 2024-02-28T17:29:34.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.602+0000 7f907a000640 1 -- 172.21.15.1:0/1675442770 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90700435f0 con 0x7f9084100110 2024-02-28T17:29:34.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.602+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908410f130 con 0x7f9084100110 2024-02-28T17:29:34.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.602+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908410f5f0 con 0x7f9084100110 2024-02-28T17:29:34.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.603+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9084064030 con 0x7f9084100110 2024-02-28T17:29:34.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.604+0000 7f907a000640 1 -- 172.21.15.1:0/1675442770 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f907002f070 con 0x7f9084100110 2024-02-28T17:29:34.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.605+0000 7f907a000640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f906806d760 0x7f906806fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:34.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.605+0000 7f907a000640 1 -- 172.21.15.1:0/1675442770 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f90700aee20 con 0x7f9084100110 2024-02-28T17:29:34.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.605+0000 7f9081e00640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f906806d760 0x7f906806fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:34.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.606+0000 7f9081e00640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f906806d760 0x7f906806fc10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9074007fd0 tx=0x7f9074009e90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:34.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.610+0000 7f907a000640 1 -- 172.21.15.1:0/1675442770 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f90700785d0 con 0x7f9084100110 2024-02-28T17:29:34.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.854+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f908410f860 con 0x7f9084100110 2024-02-28T17:29:34.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.855+0000 7f907a000640 1 -- 172.21.15.1:0/1675442770 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v34) v1 ==== 107+0+4881 (secure 0 0 0) 0x7f907007bc90 con 0x7f9084100110 2024-02-28T17:29:34.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:34.858 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":44267,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/1243103917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":1243103917},{"type":"v1","addr":"172.21.15.66:6825","nonce":1243103917}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:34.858 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 24 2024-02-28T17:29:34.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f906806d760 msgr2=0x7f906806fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:34.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f906806d760 0x7f906806fc10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9074007fd0 tx=0x7f9074009e90 comp rx=0 tx=0).stop 2024-02-28T17:29:34.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 msgr2=0x7f9084115320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:34.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 0x7f9084115320 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f907002bda0 tx=0x7f9070009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 shutdown_connections 2024-02-28T17:29:34.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f906806d760 0x7f906806fc10 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9084100110 0x7f9084115320 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 --2- 172.21.15.1:0/1675442770 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f90840fe9c0 0x7f9084114de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:34.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.858+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 >> 172.21.15.1:0/1675442770 conn(0x7f90840fa160 msgr2=0x7f90840fbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:34.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.859+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 shutdown_connections 2024-02-28T17:29:34.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:34.859+0000 7f9088a00640 1 -- 172.21.15.1:0/1675442770 wait complete. 2024-02-28T17:29:34.879 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:34.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-02-28T17:29:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.100+0000 7f3f9d200640 1 -- 172.21.15.1:0/991045321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 msgr2=0x7f3f981086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.100+0000 7f3f9d200640 1 --2- 172.21.15.1:0/991045321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 0x7f3f981086e0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f3f840048d0 tx=0x7f3f84009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.100+0000 7f3f9d200640 1 -- 172.21.15.1:0/991045321 shutdown_connections 2024-02-28T17:29:35.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.100+0000 7f3f9d200640 1 --2- 172.21.15.1:0/991045321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 0x7f3f981086e0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.100+0000 7f3f9d200640 1 --2- 172.21.15.1:0/991045321 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f9806f7e0 0x7f3f9806fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.100+0000 7f3f9d200640 1 -- 172.21.15.1:0/991045321 >> 172.21.15.1:0/991045321 conn(0x7f3f980f7ee0 msgr2=0x7f3f980fa2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:35.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.100+0000 7f3f9d200640 1 -- 172.21.15.1:0/991045321 shutdown_connections 2024-02-28T17:29:35.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.100+0000 7f3f9d200640 1 -- 172.21.15.1:0/991045321 wait complete. 2024-02-28T17:29:35.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f9d200640 1 Processor -- start 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f9d200640 1 -- start start 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f9d200640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f9806f7e0 0x7f3f98114e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f9d200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 0x7f3f98115370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f9d200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f98115970 con 0x7f3f980700f0 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f9d200640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9810ef00 con 0x7f3f9806f7e0 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f95400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 0x7f3f98115370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f95400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 0x7f3f98115370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57294/0 (socket says 172.21.15.1:57294) 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.101+0000 7f3f95400640 1 -- 172.21.15.1:0/1056033851 learned_addr learned my addr 172.21.15.1:0/1056033851 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f95e00640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f9806f7e0 0x7f3f98114e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f95400640 1 -- 172.21.15.1:0/1056033851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f9806f7e0 msgr2=0x7f3f98114e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f95400640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f9806f7e0 0x7f3f98114e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f95400640 1 -- 172.21.15.1:0/1056033851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f84005040 con 0x7f3f980700f0 2024-02-28T17:29:35.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f95400640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 0x7f3f98115370 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f3f8402bda0 tx=0x7f3f84009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f8e000640 1 -- 172.21.15.1:0/1056033851 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f840094c0 con 0x7f3f980700f0 2024-02-28T17:29:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f8e000640 1 -- 172.21.15.1:0/1056033851 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f84032e50 con 0x7f3f980700f0 2024-02-28T17:29:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f8e000640 1 -- 172.21.15.1:0/1056033851 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f840435f0 con 0x7f3f980700f0 2024-02-28T17:29:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.102+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9810f180 con 0x7f3f980700f0 2024-02-28T17:29:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.103+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9810f640 con 0x7f3f980700f0 2024-02-28T17:29:35.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.104+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f98064480 con 0x7f3f980700f0 2024-02-28T17:29:35.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.105+0000 7f3f8e000640 1 -- 172.21.15.1:0/1056033851 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f3f8402f070 con 0x7f3f980700f0 2024-02-28T17:29:35.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.106+0000 7f3f8e000640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f7c06d760 0x7f3f7c06fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:35.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.106+0000 7f3f8e000640 1 -- 172.21.15.1:0/1056033851 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3f840aedb0 con 0x7f3f980700f0 2024-02-28T17:29:35.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.107+0000 7f3f95e00640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f7c06d760 0x7f3f7c06fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:35.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.107+0000 7f3f95e00640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f7c06d760 0x7f3f7c06fc10 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3f88004ce0 tx=0x7f3f8800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:35.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.111+0000 7f3f8e000640 1 -- 172.21.15.1:0/1056033851 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f84078600 con 0x7f3f980700f0 2024-02-28T17:29:35.193 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:35 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1249856535' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-02-28T17:29:35.193 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:35 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1675442770' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-02-28T17:29:35.193 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:35 smithi001 ceph-mon[112940]: pgmap v1660: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:35.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.353+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f3f98110430 con 0x7f3f980700f0 2024-02-28T17:29:35.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.355+0000 7f3f8e000640 1 -- 172.21.15.1:0/1056033851 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v34) v1 ==== 107+0+4102 (secure 0 0 0) 0x7f3f8407bcc0 con 0x7f3f980700f0 2024-02-28T17:29:35.357 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:35.357 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:35.357 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.357+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f7c06d760 msgr2=0x7f3f7c06fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.357+0000 7f3f9d200640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f7c06d760 0x7f3f7c06fc10 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3f88004ce0 tx=0x7f3f8800b040 comp rx=0 tx=0).stop 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.357+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 msgr2=0x7f3f98115370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.357+0000 7f3f9d200640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 0x7f3f98115370 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f3f8402bda0 tx=0x7f3f84009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.358+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 shutdown_connections 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.358+0000 7f3f9d200640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f3f7c06d760 0x7f3f7c06fc10 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.358+0000 7f3f9d200640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f980700f0 0x7f3f98115370 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.358+0000 7f3f9d200640 1 --2- 172.21.15.1:0/1056033851 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f3f9806f7e0 0x7f3f98114e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.358+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 >> 172.21.15.1:0/1056033851 conn(0x7f3f980f7ee0 msgr2=0x7f3f98102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:35.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.358+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 shutdown_connections 2024-02-28T17:29:35.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.358+0000 7f3f9d200640 1 -- 172.21.15.1:0/1056033851 wait complete. 2024-02-28T17:29:35.378 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:35.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-02-28T17:29:35.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:35 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1249856535' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-02-28T17:29:35.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:35 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1675442770' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-02-28T17:29:35.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:35 smithi066 ceph-mon[85781]: pgmap v1660: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.585+0000 7f9954a00640 1 -- 172.21.15.1:0/3824839897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950108760 msgr2=0x7f9950108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.585+0000 7f9954a00640 1 --2- 172.21.15.1:0/3824839897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950108760 0x7f9950108b30 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f99400048d0 tx=0x7f9940009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.585+0000 7f9954a00640 1 -- 172.21.15.1:0/3824839897 shutdown_connections 2024-02-28T17:29:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.585+0000 7f9954a00640 1 --2- 172.21.15.1:0/3824839897 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9950102740 0x7f9950102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.585+0000 7f9954a00640 1 --2- 172.21.15.1:0/3824839897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950108760 0x7f9950108b30 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.585+0000 7f9954a00640 1 -- 172.21.15.1:0/3824839897 >> 172.21.15.1:0/3824839897 conn(0x7f9950071fc0 msgr2=0x7f99500743d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.586+0000 7f9954a00640 1 -- 172.21.15.1:0/3824839897 shutdown_connections 2024-02-28T17:29:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.586+0000 7f9954a00640 1 -- 172.21.15.1:0/3824839897 wait complete. 2024-02-28T17:29:35.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.586+0000 7f9954a00640 1 Processor -- start 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.586+0000 7f9954a00640 1 -- start start 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f9954a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950102740 0x7f9950119180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f9954a00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9950108760 0x7f99501196c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f9954a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9950119cc0 con 0x7f9950102740 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f9954a00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9950113250 con 0x7f9950108760 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f994e800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950102740 0x7f9950119180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f994e800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950102740 0x7f9950119180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57322/0 (socket says 172.21.15.1:57322) 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f994e800640 1 -- 172.21.15.1:0/3066317701 learned_addr learned my addr 172.21.15.1:0/3066317701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f9945e00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9950108760 0x7f99501196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f994e800640 1 -- 172.21.15.1:0/3066317701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9950108760 msgr2=0x7f99501196c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:35.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f994e800640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9950108760 0x7f99501196c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f994e800640 1 -- 172.21.15.1:0/3066317701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9940005040 con 0x7f9950102740 2024-02-28T17:29:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.587+0000 7f9945e00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9950108760 0x7f99501196c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.588+0000 7f994e800640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950102740 0x7f9950119180 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f9940031720 tx=0x7f9940009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.588+0000 7f9947400640 1 -- 172.21.15.1:0/3066317701 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99400094c0 con 0x7f9950102740 2024-02-28T17:29:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.588+0000 7f9947400640 1 -- 172.21.15.1:0/3066317701 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9940009620 con 0x7f9950102740 2024-02-28T17:29:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.588+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99501134d0 con 0x7f9950102740 2024-02-28T17:29:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.588+0000 7f9947400640 1 -- 172.21.15.1:0/3066317701 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994003b640 con 0x7f9950102740 2024-02-28T17:29:35.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.588+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99501139f0 con 0x7f9950102740 2024-02-28T17:29:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.590+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995004bcc0 con 0x7f9950102740 2024-02-28T17:29:35.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.590+0000 7f9947400640 1 -- 172.21.15.1:0/3066317701 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f994002f070 con 0x7f9950102740 2024-02-28T17:29:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.591+0000 7f9947400640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993406d750 0x7f993406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.592+0000 7f9947400640 1 -- 172.21.15.1:0/3066317701 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f99400af3b0 con 0x7f9950102740 2024-02-28T17:29:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.592+0000 7f9945e00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993406d750 0x7f993406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:35.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.592+0000 7f9945e00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993406d750 0x7f993406fc00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9950114a40 tx=0x7f993800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:35.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.596+0000 7f9947400640 1 -- 172.21.15.1:0/3066317701 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9940078c00 con 0x7f9950102740 2024-02-28T17:29:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.827+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f9950105a20 con 0x7f9950102740 2024-02-28T17:29:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.829+0000 7f9947400640 1 -- 172.21.15.1:0/3066317701 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v34) v1 ==== 107+0+4915 (secure 0 0 0) 0x7f9950105a20 con 0x7f9950102740 2024-02-28T17:29:35.831 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:35.831 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24269,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.66:6827/1841260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":1841260704},{"type":"v1","addr":"172.21.15.66:6827","nonce":1841260704}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 26 2024-02-28T17:29:35.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993406d750 msgr2=0x7f993406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:35.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993406d750 0x7f993406fc00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9950114a40 tx=0x7f993800b040 comp rx=0 tx=0).stop 2024-02-28T17:29:35.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950102740 msgr2=0x7f9950119180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:35.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950102740 0x7f9950119180 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f9940031720 tx=0x7f9940009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 shutdown_connections 2024-02-28T17:29:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f993406d750 0x7f993406fc00 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f9950108760 0x7f99501196c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 --2- 172.21.15.1:0/3066317701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9950102740 0x7f9950119180 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 >> 172.21.15.1:0/3066317701 conn(0x7f9950071fc0 msgr2=0x7f9950074e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 shutdown_connections 2024-02-28T17:29:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:35.831+0000 7f9954a00640 1 -- 172.21.15.1:0/3066317701 wait complete. 2024-02-28T17:29:35.850 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:35.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-02-28T17:29:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.065+0000 7f93d7200640 1 -- 172.21.15.1:0/1640956157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 msgr2=0x7f93d80fc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.065+0000 7f93d7200640 1 --2- 172.21.15.1:0/1640956157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 0x7f93d80fc170 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f93c00048d0 tx=0x7f93c0009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.065+0000 7f93d7200640 1 -- 172.21.15.1:0/1640956157 shutdown_connections 2024-02-28T17:29:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.065+0000 7f93d7200640 1 --2- 172.21.15.1:0/1640956157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 0x7f93d80fc170 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.065+0000 7f93d7200640 1 --2- 172.21.15.1:0/1640956157 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f93d8102140 0x7f93d8102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.065+0000 7f93d7200640 1 -- 172.21.15.1:0/1640956157 >> 172.21.15.1:0/1640956157 conn(0x7f93d80f7d90 msgr2=0x7f93d80fa1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.065+0000 7f93d7200640 1 -- 172.21.15.1:0/1640956157 shutdown_connections 2024-02-28T17:29:36.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.065+0000 7f93d7200640 1 -- 172.21.15.1:0/1640956157 wait complete. 2024-02-28T17:29:36.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.066+0000 7f93d7200640 1 Processor -- start 2024-02-28T17:29:36.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.066+0000 7f93d7200640 1 -- start start 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93d7200640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f93d8102140 0x7f93d8114c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93d7200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 0x7f93d8115140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93d7200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d8115740 con 0x7f93d8102ae0 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93d7200640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d810ecd0 con 0x7f93d8102140 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93cd400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 0x7f93d8115140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93cd400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 0x7f93d8115140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57332/0 (socket says 172.21.15.1:57332) 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93cd400640 1 -- 172.21.15.1:0/443880336 learned_addr learned my addr 172.21.15.1:0/443880336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93d5e00640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f93d8102140 0x7f93d8114c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93cd400640 1 -- 172.21.15.1:0/443880336 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f93d8102140 msgr2=0x7f93d8114c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93cd400640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f93d8102140 0x7f93d8114c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.067+0000 7f93cd400640 1 -- 172.21.15.1:0/443880336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c0005040 con 0x7f93d8102ae0 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.068+0000 7f93cd400640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 0x7f93d8115140 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f93c002bda0 tx=0x7f93c0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:36.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.068+0000 7f93cea00640 1 -- 172.21.15.1:0/443880336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c00094c0 con 0x7f93d8102ae0 2024-02-28T17:29:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.068+0000 7f93cea00640 1 -- 172.21.15.1:0/443880336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93c0032e50 con 0x7f93d8102ae0 2024-02-28T17:29:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.068+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d810ef50 con 0x7f93d8102ae0 2024-02-28T17:29:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.068+0000 7f93cea00640 1 -- 172.21.15.1:0/443880336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c00435f0 con 0x7f93d8102ae0 2024-02-28T17:29:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.068+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d810f410 con 0x7f93d8102ae0 2024-02-28T17:29:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.069+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d804bcc0 con 0x7f93d8102ae0 2024-02-28T17:29:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.070+0000 7f93cea00640 1 -- 172.21.15.1:0/443880336 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f93c002f070 con 0x7f93d8102ae0 2024-02-28T17:29:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.071+0000 7f93cea00640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f93bc06d760 0x7f93bc06fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:36.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.071+0000 7f93cea00640 1 -- 172.21.15.1:0/443880336 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f93c00aee50 con 0x7f93d8102ae0 2024-02-28T17:29:36.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.071+0000 7f93d5e00640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f93bc06d760 0x7f93bc06fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:36.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.072+0000 7f93d5e00640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f93bc06d760 0x7f93bc06fc10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f93c8004ce0 tx=0x7f93c800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:36.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.076+0000 7f93cea00640 1 -- 172.21.15.1:0/443880336 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f93c0078600 con 0x7f93d8102ae0 2024-02-28T17:29:36.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.318+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f93d810f630 con 0x7f93d8102ae0 2024-02-28T17:29:36.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.319+0000 7f93cea00640 1 -- 172.21.15.1:0/443880336 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v34) v1 ==== 107+0+4136 (secure 0 0 0) 0x7f93c007bcc0 con 0x7f93d8102ae0 2024-02-28T17:29:36.322 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:36.322 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:36.322 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 27 2024-02-28T17:29:36.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f93bc06d760 msgr2=0x7f93bc06fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f93bc06d760 0x7f93bc06fc10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f93c8004ce0 tx=0x7f93c800b040 comp rx=0 tx=0).stop 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 msgr2=0x7f93d8115140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 0x7f93d8115140 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f93c002bda0 tx=0x7f93c0009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 shutdown_connections 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f93bc06d760 0x7f93bc06fc10 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d8102ae0 0x7f93d8115140 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 --2- 172.21.15.1:0/443880336 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f93d8102140 0x7f93d8114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.322+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 >> 172.21.15.1:0/443880336 conn(0x7f93d80f7d90 msgr2=0x7f93d80f9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.323+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 shutdown_connections 2024-02-28T17:29:36.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.323+0000 7f93d7200640 1 -- 172.21.15.1:0/443880336 wait complete. 2024-02-28T17:29:36.344 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:36.344 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-02-28T17:29:36.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1056033851' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-02-28T17:29:36.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:36 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3066317701' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-02-28T17:29:36.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1056033851' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-02-28T17:29:36.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:36 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3066317701' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-02-28T17:29:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.550+0000 7fa6c7000640 1 -- 172.21.15.1:0/1517538056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104cc0 msgr2=0x7fa6c00fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.550+0000 7fa6c7000640 1 --2- 172.21.15.1:0/1517538056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104cc0 0x7fa6c00fe350 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac0048d0 tx=0x7fa6ac009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.550+0000 7fa6c7000640 1 -- 172.21.15.1:0/1517538056 shutdown_connections 2024-02-28T17:29:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.550+0000 7fa6c7000640 1 --2- 172.21.15.1:0/1517538056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104cc0 0x7fa6c00fe350 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.550+0000 7fa6c7000640 1 --2- 172.21.15.1:0/1517538056 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6c0104320 0x7fa6c01046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.550+0000 7fa6c7000640 1 -- 172.21.15.1:0/1517538056 >> 172.21.15.1:0/1517538056 conn(0x7fa6c00f9f90 msgr2=0x7fa6c00fc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.550+0000 7fa6c7000640 1 -- 172.21.15.1:0/1517538056 shutdown_connections 2024-02-28T17:29:36.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.550+0000 7fa6c7000640 1 -- 172.21.15.1:0/1517538056 wait complete. 2024-02-28T17:29:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.551+0000 7fa6c7000640 1 Processor -- start 2024-02-28T17:29:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.551+0000 7fa6c7000640 1 -- start start 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.551+0000 7fa6c7000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104320 0x7fa6c0114b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c7000640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6c0104cc0 0x7fa6c01150b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c7000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c01156b0 con 0x7fa6c0104320 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c7000640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c010ec40 con 0x7fa6c0104cc0 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c5c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104320 0x7fa6c0114b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c5c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104320 0x7fa6c0114b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57344/0 (socket says 172.21.15.1:57344) 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c5c00640 1 -- 172.21.15.1:0/3034447111 learned_addr learned my addr 172.21.15.1:0/3034447111 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c5c00640 1 -- 172.21.15.1:0/3034447111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6c0104cc0 msgr2=0x7fa6c01150b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c5c00640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6c0104cc0 0x7fa6c01150b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c5c00640 1 -- 172.21.15.1:0/3034447111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac005040 con 0x7fa6c0104320 2024-02-28T17:29:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.552+0000 7fa6c5c00640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104320 0x7fa6c0114b70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4002a50 tx=0x7fa6b4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:36.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.553+0000 7fa6bd600640 1 -- 172.21.15.1:0/3034447111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b4007c30 con 0x7fa6c0104320 2024-02-28T17:29:36.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.553+0000 7fa6bd600640 1 -- 172.21.15.1:0/3034447111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6b400f040 con 0x7fa6c0104320 2024-02-28T17:29:36.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.553+0000 7fa6bd600640 1 -- 172.21.15.1:0/3034447111 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b400db50 con 0x7fa6c0104320 2024-02-28T17:29:36.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.553+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c010ef20 con 0x7fa6c0104320 2024-02-28T17:29:36.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.553+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c010f440 con 0x7fa6c0104320 2024-02-28T17:29:36.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.554+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c004bcc0 con 0x7fa6c0104320 2024-02-28T17:29:36.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.555+0000 7fa6bd600640 1 -- 172.21.15.1:0/3034447111 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fa6b400dcf0 con 0x7fa6c0104320 2024-02-28T17:29:36.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.556+0000 7fa6bd600640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa6a806d750 0x7fa6a806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:36.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.556+0000 7fa6bd600640 1 -- 172.21.15.1:0/3034447111 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa6b408c760 con 0x7fa6c0104320 2024-02-28T17:29:36.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.556+0000 7fa6c5200640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa6a806d750 0x7fa6a806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:36.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.557+0000 7fa6c5200640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa6a806d750 0x7fa6a806fc00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac034040 tx=0x7fa6ac009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:36.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.561+0000 7fa6bd600640 1 -- 172.21.15.1:0/3034447111 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa6b4055f90 con 0x7fa6c0104320 2024-02-28T17:29:36.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.793+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fa6c0110280 con 0x7fa6c0104320 2024-02-28T17:29:36.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.794+0000 7fa6bd600640 1 -- 172.21.15.1:0/3034447111 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v34) v1 ==== 107+0+4949 (secure 0 0 0) 0x7fa6b4059650 con 0x7fa6c0104320 2024-02-28T17:29:36.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:36.797 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":44939,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6827/3265306782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":3265306782},{"type":"v1","addr":"172.21.15.66:6827","nonce":3265306782}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T16:41:05.748872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 28 2024-02-28T17:29:36.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa6a806d750 msgr2=0x7fa6a806fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:36.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa6a806d750 0x7fa6a806fc00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac034040 tx=0x7fa6ac009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104320 msgr2=0x7fa6c0114b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:36.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104320 0x7fa6c0114b70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4002a50 tx=0x7fa6b4004600 comp rx=0 tx=0).stop 2024-02-28T17:29:36.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 shutdown_connections 2024-02-28T17:29:36.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fa6a806d750 0x7fa6a806fc00 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fa6c0104cc0 0x7fa6c01150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 --2- 172.21.15.1:0/3034447111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6c0104320 0x7fa6c0114b70 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:36.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 >> 172.21.15.1:0/3034447111 conn(0x7fa6c00f9f90 msgr2=0x7fa6c00fb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:36.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 shutdown_connections 2024-02-28T17:29:36.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:36.797+0000 7fa6c7000640 1 -- 172.21.15.1:0/3034447111 wait complete. 2024-02-28T17:29:36.816 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-02-28T17:29:36.816 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-02-28T17:29:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.026+0000 7f6a4fe00640 1 -- 172.21.15.1:0/11315608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 msgr2=0x7f6a50064480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.026+0000 7f6a4fe00640 1 --2- 172.21.15.1:0/11315608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 0x7f6a50064480 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009c50 tx=0x7f6a40004d10 comp rx=0 tx=0).stop 2024-02-28T17:29:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.027+0000 7f6a4fe00640 1 -- 172.21.15.1:0/11315608 shutdown_connections 2024-02-28T17:29:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.027+0000 7f6a4fe00640 1 --2- 172.21.15.1:0/11315608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 0x7f6a50064480 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.027+0000 7f6a4fe00640 1 --2- 172.21.15.1:0/11315608 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a50102100 0x7f6a501024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.027+0000 7f6a4fe00640 1 -- 172.21.15.1:0/11315608 >> 172.21.15.1:0/11315608 conn(0x7f6a500710f0 msgr2=0x7f6a500714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.027+0000 7f6a4fe00640 1 -- 172.21.15.1:0/11315608 shutdown_connections 2024-02-28T17:29:37.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.027+0000 7f6a4fe00640 1 -- 172.21.15.1:0/11315608 wait complete. 2024-02-28T17:29:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.027+0000 7f6a4fe00640 1 Processor -- start 2024-02-28T17:29:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.028+0000 7f6a4fe00640 1 -- start start 2024-02-28T17:29:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.028+0000 7f6a4fe00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a50102100 0x7f6a50114c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.028+0000 7f6a4fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 0x7f6a501151a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.028+0000 7f6a4fe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501157a0 con 0x7f6a50102aa0 2024-02-28T17:29:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.028+0000 7f6a4fe00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5010ed30 con 0x7f6a50102100 2024-02-28T17:29:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.028+0000 7f6a4e000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 0x7f6a501151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:37.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.028+0000 7f6a4e000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 0x7f6a501151a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57364/0 (socket says 172.21.15.1:57364) 2024-02-28T17:29:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.028+0000 7f6a4e000640 1 -- 172.21.15.1:0/2255283430 learned_addr learned my addr 172.21.15.1:0/2255283430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.029+0000 7f6a4ea00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a50102100 0x7f6a50114c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.029+0000 7f6a4e000640 1 -- 172.21.15.1:0/2255283430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a50102100 msgr2=0x7f6a50114c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.029+0000 7f6a4e000640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a50102100 0x7f6a50114c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.029+0000 7f6a4e000640 1 -- 172.21.15.1:0/2255283430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400098b0 con 0x7f6a50102aa0 2024-02-28T17:29:37.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.029+0000 7f6a4e000640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 0x7f6a501151a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6a40000f80 tx=0x7f6a40004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.029+0000 7f6a46a00640 1 -- 172.21.15.1:0/2255283430 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a4003a070 con 0x7f6a50102aa0 2024-02-28T17:29:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.029+0000 7f6a46a00640 1 -- 172.21.15.1:0/2255283430 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a40035b20 con 0x7f6a50102aa0 2024-02-28T17:29:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.029+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5010efb0 con 0x7f6a50102aa0 2024-02-28T17:29:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.030+0000 7f6a46a00640 1 -- 172.21.15.1:0/2255283430 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a4003eae0 con 0x7f6a50102aa0 2024-02-28T17:29:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.030+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5010f4d0 con 0x7f6a50102aa0 2024-02-28T17:29:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.031+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5004bcc0 con 0x7f6a50102aa0 2024-02-28T17:29:37.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.032+0000 7f6a46a00640 1 -- 172.21.15.1:0/2255283430 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f6a40032070 con 0x7f6a50102aa0 2024-02-28T17:29:37.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.032+0000 7f6a46a00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a3406d760 0x7f6a3406fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:37.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.033+0000 7f6a46a00640 1 -- 172.21.15.1:0/2255283430 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f6a400af870 con 0x7f6a50102aa0 2024-02-28T17:29:37.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.033+0000 7f6a4ea00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a3406d760 0x7f6a3406fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:37.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.033+0000 7f6a4ea00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a3406d760 0x7f6a3406fc10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6a38004c40 tx=0x7f6a38009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:37.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.038+0000 7f6a46a00640 1 -- 172.21.15.1:0/2255283430 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a400790c0 con 0x7f6a50102aa0 2024-02-28T17:29:37.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.276+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f6a5010f6f0 con 0x7f6a50102aa0 2024-02-28T17:29:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.278+0000 7f6a46a00640 1 -- 172.21.15.1:0/2255283430 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v34) v1 ==== 107+0+4948 (secure 0 0 0) 0x7f6a4007c780 con 0x7f6a50102aa0 2024-02-28T17:29:37.280 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:37.280 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":44939,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6827/3265306782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":3265306782},{"type":"v1","addr":"172.21.15.66:6827","nonce":3265306782}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T17:27:23.070106+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 29 2024-02-28T17:29:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a3406d760 msgr2=0x7f6a3406fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a3406d760 0x7f6a3406fc10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6a38004c40 tx=0x7f6a38009040 comp rx=0 tx=0).stop 2024-02-28T17:29:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 msgr2=0x7f6a501151a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 0x7f6a501151a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6a40000f80 tx=0x7f6a40004d10 comp rx=0 tx=0).stop 2024-02-28T17:29:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 shutdown_connections 2024-02-28T17:29:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f6a3406d760 0x7f6a3406fc10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a50102aa0 0x7f6a501151a0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 --2- 172.21.15.1:0/2255283430 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f6a50102100 0x7f6a50114c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.280+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 >> 172.21.15.1:0/2255283430 conn(0x7f6a500710f0 msgr2=0x7f6a500fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.281+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 shutdown_connections 2024-02-28T17:29:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.281+0000 7f6a4fe00640 1 -- 172.21.15.1:0/2255283430 wait complete. 2024-02-28T17:29:37.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-02-28T17:29:37.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.501+0000 7f30b1000640 1 -- 172.21.15.1:0/2067901588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac102740 msgr2=0x7f30ac102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.501+0000 7f30b1000640 1 --2- 172.21.15.1:0/2067901588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac102740 0x7f30ac102bb0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f30980048d0 tx=0x7f3098009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.501+0000 7f30b1000640 1 -- 172.21.15.1:0/2067901588 shutdown_connections 2024-02-28T17:29:37.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.501+0000 7f30b1000640 1 --2- 172.21.15.1:0/2067901588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac102740 0x7f30ac102bb0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.501+0000 7f30b1000640 1 --2- 172.21.15.1:0/2067901588 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30ac108760 0x7f30ac108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.501+0000 7f30b1000640 1 -- 172.21.15.1:0/2067901588 >> 172.21.15.1:0/2067901588 conn(0x7f30ac071fc0 msgr2=0x7f30ac0743d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:37.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.501+0000 7f30b1000640 1 -- 172.21.15.1:0/2067901588 shutdown_connections 2024-02-28T17:29:37.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.501+0000 7f30b1000640 1 -- 172.21.15.1:0/2067901588 wait complete. 2024-02-28T17:29:37.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.502+0000 7f30b1000640 1 Processor -- start 2024-02-28T17:29:37.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.502+0000 7f30b1000640 1 -- start start 2024-02-28T17:29:37.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.502+0000 7f30b1000640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30ac102740 0x7f30ac118fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.502+0000 7f30b1000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac108760 0x7f30ac119520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.502+0000 7f30b1000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac119bb0 con 0x7f30ac108760 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.502+0000 7f30b1000640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac1130b0 con 0x7f30ac102740 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.503+0000 7f30aaa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac108760 0x7f30ac119520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.503+0000 7f30aaa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac108760 0x7f30ac119520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57380/0 (socket says 172.21.15.1:57380) 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.503+0000 7f30aaa00640 1 -- 172.21.15.1:0/2340513114 learned_addr learned my addr 172.21.15.1:0/2340513114 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.503+0000 7f30ab400640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30ac102740 0x7f30ac118fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.503+0000 7f30aaa00640 1 -- 172.21.15.1:0/2340513114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30ac102740 msgr2=0x7f30ac118fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.503+0000 7f30aaa00640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30ac102740 0x7f30ac118fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.503+0000 7f30aaa00640 1 -- 172.21.15.1:0/2340513114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098005040 con 0x7f30ac108760 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.503+0000 7f30aaa00640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac108760 0x7f30ac119520 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f309802bda0 tx=0x7f3098009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.504+0000 7f30a3400640 1 -- 172.21.15.1:0/2340513114 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30980094c0 con 0x7f30ac108760 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.504+0000 7f30a3400640 1 -- 172.21.15.1:0/2340513114 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3098009620 con 0x7f30ac108760 2024-02-28T17:29:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.504+0000 7f30a3400640 1 -- 172.21.15.1:0/2340513114 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30980435f0 con 0x7f30ac108760 2024-02-28T17:29:37.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.504+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac113330 con 0x7f30ac108760 2024-02-28T17:29:37.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.504+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac1137f0 con 0x7f30ac108760 2024-02-28T17:29:37.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.505+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac04bcc0 con 0x7f30ac108760 2024-02-28T17:29:37.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.507+0000 7f30a3400640 1 -- 172.21.15.1:0/2340513114 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f309802f070 con 0x7f30ac108760 2024-02-28T17:29:37.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.508+0000 7f30a3400640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f309006d760 0x7f309006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:37.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.508+0000 7f30a3400640 1 -- 172.21.15.1:0/2340513114 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f30980aed90 con 0x7f30ac108760 2024-02-28T17:29:37.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.508+0000 7f30ab400640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f309006d760 0x7f309006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:37.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:37 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/443880336' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-02-28T17:29:37.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:37 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3034447111' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-02-28T17:29:37.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:37 smithi066 ceph-mon[85781]: pgmap v1661: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:37.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.509+0000 7f30ab400640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f309006d760 0x7f309006fc10 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f309c00a900 tx=0x7f309c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:37.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.513+0000 7f30a3400640 1 -- 172.21.15.1:0/2340513114 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f30980785e0 con 0x7f30ac108760 2024-02-28T17:29:37.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:37 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/443880336' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-02-28T17:29:37.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:37 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3034447111' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-02-28T17:29:37.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:37 smithi001 ceph-mon[112940]: pgmap v1661: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:37.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.756+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f30ac10f8c0 con 0x7f30ac108760 2024-02-28T17:29:37.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.758+0000 7f30a3400640 1 -- 172.21.15.1:0/2340513114 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v34) v1 ==== 107+0+5005 (secure 0 0 0) 0x7f309807bca0 con 0x7f30ac108760 2024-02-28T17:29:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:37.761 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":44939,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6827/3265306782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":3265306782},{"type":"v1","addr":"172.21.15.66:6827","nonce":3265306782}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T17:27:24.069214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34438,"mds_1":34470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34438":{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34470":{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":1,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:37.761 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 30 2024-02-28T17:29:37.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f309006d760 msgr2=0x7f309006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f309006d760 0x7f309006fc10 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f309c00a900 tx=0x7f309c008040 comp rx=0 tx=0).stop 2024-02-28T17:29:37.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac108760 msgr2=0x7f30ac119520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac108760 0x7f30ac119520 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f309802bda0 tx=0x7f3098009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 shutdown_connections 2024-02-28T17:29:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f309006d760 0x7f309006fc10 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30ac108760 0x7f30ac119520 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 --2- 172.21.15.1:0/2340513114 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f30ac102740 0x7f30ac118fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 >> 172.21.15.1:0/2340513114 conn(0x7f30ac071fc0 msgr2=0x7f30ac0748e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 shutdown_connections 2024-02-28T17:29:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.761+0000 7f30b1000640 1 -- 172.21.15.1:0/2340513114 wait complete. 2024-02-28T17:29:37.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-02-28T17:29:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.993+0000 7fb67c600640 1 -- 172.21.15.1:0/2717646306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104540 msgr2=0x7fb674104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.993+0000 7fb67c600640 1 --2- 172.21.15.1:0/2717646306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104540 0x7fb674104910 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb668009c50 tx=0x7fb668004d10 comp rx=0 tx=0).stop 2024-02-28T17:29:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.993+0000 7fb67c600640 1 -- 172.21.15.1:0/2717646306 shutdown_connections 2024-02-28T17:29:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.993+0000 7fb67c600640 1 --2- 172.21.15.1:0/2717646306 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb674104ee0 0x7fb6740fe5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.993+0000 7fb67c600640 1 --2- 172.21.15.1:0/2717646306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104540 0x7fb674104910 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.993+0000 7fb67c600640 1 -- 172.21.15.1:0/2717646306 >> 172.21.15.1:0/2717646306 conn(0x7fb6740fa120 msgr2=0x7fb6740fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.994+0000 7fb67c600640 1 -- 172.21.15.1:0/2717646306 shutdown_connections 2024-02-28T17:29:37.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.994+0000 7fb67c600640 1 -- 172.21.15.1:0/2717646306 wait complete. 2024-02-28T17:29:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.994+0000 7fb67c600640 1 Processor -- start 2024-02-28T17:29:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.995+0000 7fb67c600640 1 -- start start 2024-02-28T17:29:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.995+0000 7fb67c600640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb674104540 0x7fb674114e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.995+0000 7fb67c600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104ee0 0x7fb6741153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.995+0000 7fb67c600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741159b0 con 0x7fb674104ee0 2024-02-28T17:29:37.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.995+0000 7fb67c600640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67410ef40 con 0x7fb674104540 2024-02-28T17:29:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.995+0000 7fb679200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104ee0 0x7fb6741153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb679200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104ee0 0x7fb6741153b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57400/0 (socket says 172.21.15.1:57400) 2024-02-28T17:29:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb679c00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb674104540 0x7fb674114e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:37.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb679200640 1 -- 172.21.15.1:0/2461147725 learned_addr learned my addr 172.21.15.1:0/2461147725 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb679200640 1 -- 172.21.15.1:0/2461147725 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb674104540 msgr2=0x7fb674114e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb679200640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb674104540 0x7fb674114e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb679200640 1 -- 172.21.15.1:0/2461147725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6680098b0 con 0x7fb674104ee0 2024-02-28T17:29:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb679200640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104ee0 0x7fb6741153b0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb660002840 tx=0x7fb660004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb671600640 1 -- 172.21.15.1:0/2461147725 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb660007c30 con 0x7fb674104ee0 2024-02-28T17:29:38.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb671600640 1 -- 172.21.15.1:0/2461147725 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb66000f040 con 0x7fb674104ee0 2024-02-28T17:29:38.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67410f220 con 0x7fb674104ee0 2024-02-28T17:29:38.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb671600640 1 -- 172.21.15.1:0/2461147725 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb660009570 con 0x7fb674104ee0 2024-02-28T17:29:38.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.997+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67410f740 con 0x7fb674104ee0 2024-02-28T17:29:38.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:37.998+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb674064480 con 0x7fb674104ee0 2024-02-28T17:29:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.000+0000 7fb671600640 1 -- 172.21.15.1:0/2461147725 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fb66000dc80 con 0x7fb674104ee0 2024-02-28T17:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.001+0000 7fb671600640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65c06d750 0x7fb65c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.001+0000 7fb671600640 1 -- 172.21.15.1:0/2461147725 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fb66008c770 con 0x7fb674104ee0 2024-02-28T17:29:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.001+0000 7fb679c00640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65c06d750 0x7fb65c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.002+0000 7fb679c00640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65c06d750 0x7fb65c06fc00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb668004750 tx=0x7fb668037040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.007+0000 7fb671600640 1 -- 172.21.15.1:0/2461147725 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb660055fd0 con 0x7fb674104ee0 2024-02-28T17:29:38.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.236+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7fb67410f960 con 0x7fb674104ee0 2024-02-28T17:29:38.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.237+0000 7fb671600640 1 -- 172.21.15.1:0/2461147725 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v34) v1 ==== 107+0+5011 (secure 0 0 0) 0x7fb660059690 con 0x7fb674104ee0 2024-02-28T17:29:38.240 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:38.240 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":44939,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6827/3265306782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":3265306782},{"type":"v1","addr":"172.21.15.66:6827","nonce":3265306782}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T17:27:31.211778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34438,"mds_1":34470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34438":{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":30,"state":"up:resolve","state_seq":695,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34470":{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":1,"incarnation":30,"state":"up:resolve","state_seq":689,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 31 2024-02-28T17:29:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65c06d750 msgr2=0x7fb65c06fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65c06d750 0x7fb65c06fc00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb668004750 tx=0x7fb668037040 comp rx=0 tx=0).stop 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104ee0 msgr2=0x7fb6741153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104ee0 0x7fb6741153b0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb660002840 tx=0x7fb660004570 comp rx=0 tx=0).stop 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 shutdown_connections 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fb65c06d750 0x7fb65c06fc00 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb674104ee0 0x7fb6741153b0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 --2- 172.21.15.1:0/2461147725 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fb674104540 0x7fb674114e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 >> 172.21.15.1:0/2461147725 conn(0x7fb6740fa120 msgr2=0x7fb6740fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 shutdown_connections 2024-02-28T17:29:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.240+0000 7fb67c600640 1 -- 172.21.15.1:0/2461147725 wait complete. 2024-02-28T17:29:38.261 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-02-28T17:29:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.474+0000 7f82ed800640 1 -- 172.21.15.1:0/1393654679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 msgr2=0x7f82e806fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.474+0000 7f82ed800640 1 --2- 172.21.15.1:0/1393654679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 0x7f82e806fc10 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f82d80048d0 tx=0x7f82d8009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.474+0000 7f82ed800640 1 -- 172.21.15.1:0/1393654679 shutdown_connections 2024-02-28T17:29:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.474+0000 7f82ed800640 1 --2- 172.21.15.1:0/1393654679 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8070150 0x7f82e8108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.474+0000 7f82ed800640 1 --2- 172.21.15.1:0/1393654679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 0x7f82e806fc10 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.474+0000 7f82ed800640 1 -- 172.21.15.1:0/1393654679 >> 172.21.15.1:0/1393654679 conn(0x7f82e80f7ef0 msgr2=0x7f82e80fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:38.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.474+0000 7f82ed800640 1 -- 172.21.15.1:0/1393654679 shutdown_connections 2024-02-28T17:29:38.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.475+0000 7f82ed800640 1 -- 172.21.15.1:0/1393654679 wait complete. 2024-02-28T17:29:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.475+0000 7f82ed800640 1 Processor -- start 2024-02-28T17:29:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.475+0000 7f82ed800640 1 -- start start 2024-02-28T17:29:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.476+0000 7f82ed800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 0x7f82e8114e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.476+0000 7f82ed800640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8070150 0x7f82e81153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.476+0000 7f82ed800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e81159b0 con 0x7f82e806f840 2024-02-28T17:29:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.476+0000 7f82ed800640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e810ef40 con 0x7f82e8070150 2024-02-28T17:29:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.476+0000 7f82e6a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 0x7f82e8114e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.476+0000 7f82e6a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 0x7f82e8114e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57418/0 (socket says 172.21.15.1:57418) 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.476+0000 7f82e6a00640 1 -- 172.21.15.1:0/3146435828 learned_addr learned my addr 172.21.15.1:0/3146435828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82e6a00640 1 -- 172.21.15.1:0/3146435828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8070150 msgr2=0x7f82e81153b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82e6000640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8070150 0x7f82e81153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82e6a00640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8070150 0x7f82e81153b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82e6a00640 1 -- 172.21.15.1:0/3146435828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8005040 con 0x7f82e806f840 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82e6a00640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 0x7f82e8114e70 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f82d80048a0 tx=0x7f82d8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82dea00640 1 -- 172.21.15.1:0/3146435828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82d80094c0 con 0x7f82e806f840 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82dea00640 1 -- 172.21.15.1:0/3146435828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82d8009620 con 0x7f82e806f840 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82dea00640 1 -- 172.21.15.1:0/3146435828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82d80435f0 con 0x7f82e806f840 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e810f1c0 con 0x7f82e806f840 2024-02-28T17:29:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.477+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e810f680 con 0x7f82e806f840 2024-02-28T17:29:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.479+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e8064480 con 0x7f82e806f840 2024-02-28T17:29:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.481+0000 7f82dea00640 1 -- 172.21.15.1:0/3146435828 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f82d802f070 con 0x7f82e806f840 2024-02-28T17:29:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.482+0000 7f82dea00640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82cc06d760 0x7f82cc06fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.482+0000 7f82dea00640 1 -- 172.21.15.1:0/3146435828 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f82d80aee40 con 0x7f82e806f840 2024-02-28T17:29:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.482+0000 7f82e6000640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82cc06d760 0x7f82cc06fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.482+0000 7f82e6000640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82cc06d760 0x7f82cc06fc10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f82d4007fd0 tx=0x7f82d4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.486+0000 7f82dea00640 1 -- 172.21.15.1:0/3146435828 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82d8078670 con 0x7f82e806f840 2024-02-28T17:29:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:38 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2255283430' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-02-28T17:29:38.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:38 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2340513114' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-02-28T17:29:38.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:38 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2255283430' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-02-28T17:29:38.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:38 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2340513114' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-02-28T17:29:38.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.728+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f82e8105d90 con 0x7f82e806f840 2024-02-28T17:29:38.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.729+0000 7f82dea00640 1 -- 172.21.15.1:0/3146435828 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v34) v1 ==== 107+0+5015 (secure 0 0 0) 0x7f82e8105d90 con 0x7f82e806f840 2024-02-28T17:29:38.732 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:38.732 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":44939,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6827/3265306782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":3265306782},{"type":"v1","addr":"172.21.15.66:6827","nonce":3265306782}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T17:27:31.804792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34438,"mds_1":34470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34438":{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":30,"state":"up:reconnect","state_seq":696,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34470":{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":1,"incarnation":30,"state":"up:reconnect","state_seq":690,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:38.732 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 32 2024-02-28T17:29:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82cc06d760 msgr2=0x7f82cc06fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82cc06d760 0x7f82cc06fc10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f82d4007fd0 tx=0x7f82d4009040 comp rx=0 tx=0).stop 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 msgr2=0x7f82e8114e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 0x7f82e8114e70 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f82d80048a0 tx=0x7f82d8009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 shutdown_connections 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f82cc06d760 0x7f82cc06fc10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f82e8070150 0x7f82e81153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 --2- 172.21.15.1:0/3146435828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e806f840 0x7f82e8114e70 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 >> 172.21.15.1:0/3146435828 conn(0x7f82e80f7ef0 msgr2=0x7f82e8102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 shutdown_connections 2024-02-28T17:29:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.732+0000 7f82ed800640 1 -- 172.21.15.1:0/3146435828 wait complete. 2024-02-28T17:29:38.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-02-28T17:29:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.975+0000 7fc63ea00640 1 -- 172.21.15.1:0/1216797600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638102740 msgr2=0x7fc638102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:38.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.975+0000 7fc63ea00640 1 --2- 172.21.15.1:0/1216797600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638102740 0x7fc638102bb0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fc6240048d0 tx=0x7fc624009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.975+0000 7fc63ea00640 1 -- 172.21.15.1:0/1216797600 shutdown_connections 2024-02-28T17:29:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.975+0000 7fc63ea00640 1 --2- 172.21.15.1:0/1216797600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638102740 0x7fc638102bb0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.975+0000 7fc63ea00640 1 --2- 172.21.15.1:0/1216797600 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc638108760 0x7fc638108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.975+0000 7fc63ea00640 1 -- 172.21.15.1:0/1216797600 >> 172.21.15.1:0/1216797600 conn(0x7fc638071fc0 msgr2=0x7fc6380743d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.975+0000 7fc63ea00640 1 -- 172.21.15.1:0/1216797600 shutdown_connections 2024-02-28T17:29:38.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.975+0000 7fc63ea00640 1 -- 172.21.15.1:0/1216797600 wait complete. 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.976+0000 7fc63ea00640 1 Processor -- start 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.976+0000 7fc63ea00640 1 -- start start 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.976+0000 7fc63ea00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc638102740 0x7fc6381190d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.976+0000 7fc63ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638108760 0x7fc638119610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.976+0000 7fc63ea00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc638119c10 con 0x7fc638108760 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.976+0000 7fc63ea00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381131a0 con 0x7fc638102740 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.977+0000 7fc63cc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638108760 0x7fc638119610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.977+0000 7fc63cc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638108760 0x7fc638119610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57434/0 (socket says 172.21.15.1:57434) 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.977+0000 7fc63cc00640 1 -- 172.21.15.1:0/2850413454 learned_addr learned my addr 172.21.15.1:0/2850413454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.977+0000 7fc63d600640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc638102740 0x7fc6381190d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:38.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.977+0000 7fc63cc00640 1 -- 172.21.15.1:0/2850413454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc638102740 msgr2=0x7fc6381190d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.977+0000 7fc63cc00640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc638102740 0x7fc6381190d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.977+0000 7fc63cc00640 1 -- 172.21.15.1:0/2850413454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc624005040 con 0x7fc638108760 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.977+0000 7fc63cc00640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638108760 0x7fc638119610 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fc62402bda0 tx=0x7fc624009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.978+0000 7fc635600640 1 -- 172.21.15.1:0/2850413454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6240094c0 con 0x7fc638108760 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.978+0000 7fc635600640 1 -- 172.21.15.1:0/2850413454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc624032e50 con 0x7fc638108760 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.978+0000 7fc635600640 1 -- 172.21.15.1:0/2850413454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6240435f0 con 0x7fc638108760 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.978+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc638113420 con 0x7fc638108760 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.978+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6381138e0 con 0x7fc638108760 2024-02-28T17:29:38.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.979+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63804bcc0 con 0x7fc638108760 2024-02-28T17:29:38.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.980+0000 7fc635600640 1 -- 172.21.15.1:0/2850413454 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fc62402f070 con 0x7fc638108760 2024-02-28T17:29:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.981+0000 7fc635600640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc62006d760 0x7fc62006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.981+0000 7fc635600640 1 -- 172.21.15.1:0/2850413454 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc6240aeee0 con 0x7fc638108760 2024-02-28T17:29:38.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.981+0000 7fc63d600640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc62006d760 0x7fc62006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:38.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.982+0000 7fc63d600640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc62006d760 0x7fc62006fc10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc62c004ce0 tx=0x7fc62c00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:38.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:38.985+0000 7fc635600640 1 -- 172.21.15.1:0/2850413454 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc624078690 con 0x7fc638108760 2024-02-28T17:29:39.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.229+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7fc6381059f0 con 0x7fc638108760 2024-02-28T17:29:39.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.231+0000 7fc635600640 1 -- 172.21.15.1:0/2850413454 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v34) v1 ==== 107+0+5009 (secure 0 0 0) 0x7fc62407bd50 con 0x7fc638108760 2024-02-28T17:29:39.233 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:39.233 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44619,"name":"cephfs.smithi066.wfbdor","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6825/3536246532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":3536246532},{"type":"v1","addr":"172.21.15.66:6825","nonce":3536246532}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":44939,"name":"cephfs.smithi066.pivimo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.66:6827/3265306782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6826","nonce":3265306782},{"type":"v1","addr":"172.21.15.66:6827","nonce":3265306782}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-28T16:26:38.901002+0000","modified":"2024-02-28T17:27:32.770801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34438,"mds_1":34470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34438":{"gid":34438,"name":"cephfs.smithi001.awxxdo","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":697,"addr":"172.21.15.1:6827/3510634910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":3510634910},{"type":"v1","addr":"172.21.15.1:6827","nonce":3510634910}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34470":{"gid":34470,"name":"cephfs.smithi001.rtdwhc","rank":1,"incarnation":30,"state":"up:rejoin","state_seq":691,"addr":"172.21.15.1:6829/1993293367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1993293367},{"type":"v1","addr":"172.21.15.1:6829","nonce":1993293367}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-28T17:29:39.233 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 33 2024-02-28T17:29:39.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc62006d760 msgr2=0x7fc62006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:39.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc62006d760 0x7fc62006fc10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc62c004ce0 tx=0x7fc62c00b040 comp rx=0 tx=0).stop 2024-02-28T17:29:39.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638108760 msgr2=0x7fc638119610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:39.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638108760 0x7fc638119610 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fc62402bda0 tx=0x7fc624009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 shutdown_connections 2024-02-28T17:29:39.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fc62006d760 0x7fc62006fc10 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc638108760 0x7fc638119610 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 --2- 172.21.15.1:0/2850413454 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fc638102740 0x7fc6381190d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 >> 172.21.15.1:0/2850413454 conn(0x7fc638071fc0 msgr2=0x7fc638074d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:39.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 shutdown_connections 2024-02-28T17:29:39.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.234+0000 7fc63ea00640 1 -- 172.21.15.1:0/2850413454 wait complete. 2024-02-28T17:29:39.253 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-02-28T17:29:39.266 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-02-28T17:29:39.266 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T17:29:39.266 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-28T17:29:39.320 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T17:29:39.320 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-28T17:29:39.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-28T17:29:39.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:39 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2461147725' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-02-28T17:29:39.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:39 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3146435828' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-02-28T17:29:39.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:39 smithi066 ceph-mon[85781]: pgmap v1662: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:39.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:39 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2461147725' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-02-28T17:29:39.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:39 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3146435828' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-02-28T17:29:39.530 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:39 smithi001 ceph-mon[112940]: pgmap v1662: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:39.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.588+0000 7f5b3ba00640 1 -- 172.21.15.1:0/3595992292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 msgr2=0x7f5b341028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:39.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.588+0000 7f5b3ba00640 1 --2- 172.21.15.1:0/3595992292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 0x7f5b341028f0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f5b280048d0 tx=0x7f5b28009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.589+0000 7f5b3ba00640 1 -- 172.21.15.1:0/3595992292 shutdown_connections 2024-02-28T17:29:39.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.589+0000 7f5b3ba00640 1 --2- 172.21.15.1:0/3595992292 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b340fc4e0 0x7f5b340fc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.589+0000 7f5b3ba00640 1 --2- 172.21.15.1:0/3595992292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 0x7f5b341028f0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.589+0000 7f5b3ba00640 1 -- 172.21.15.1:0/3595992292 >> 172.21.15.1:0/3595992292 conn(0x7f5b340f8060 msgr2=0x7f5b340fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:39.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.589+0000 7f5b3ba00640 1 -- 172.21.15.1:0/3595992292 shutdown_connections 2024-02-28T17:29:39.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.589+0000 7f5b3ba00640 1 -- 172.21.15.1:0/3595992292 wait complete. 2024-02-28T17:29:39.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.590+0000 7f5b3ba00640 1 Processor -- start 2024-02-28T17:29:39.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.590+0000 7f5b3ba00640 1 -- start start 2024-02-28T17:29:39.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.590+0000 7f5b3ba00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b340fc4e0 0x7f5b34112d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:39.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.590+0000 7f5b3ba00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 0x7f5b34113250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:39.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.590+0000 7f5b3ba00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b34113850 con 0x7f5b34102520 2024-02-28T17:29:39.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.590+0000 7f5b3ba00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3410cde0 con 0x7f5b340fc4e0 2024-02-28T17:29:39.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.591+0000 7f5b33e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 0x7f5b34113250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:39.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.591+0000 7f5b33e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 0x7f5b34113250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57458/0 (socket says 172.21.15.1:57458) 2024-02-28T17:29:39.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.591+0000 7f5b33e00640 1 -- 172.21.15.1:0/682536560 learned_addr learned my addr 172.21.15.1:0/682536560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.591+0000 7f5b39000640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b340fc4e0 0x7f5b34112d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b33e00640 1 -- 172.21.15.1:0/682536560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b340fc4e0 msgr2=0x7f5b34112d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b33e00640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b340fc4e0 0x7f5b34112d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b33e00640 1 -- 172.21.15.1:0/682536560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b28005040 con 0x7f5b34102520 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b33e00640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 0x7f5b34113250 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5b20002a50 tx=0x7f5b20004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b30c00640 1 -- 172.21.15.1:0/682536560 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b20007c30 con 0x7f5b34102520 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b30c00640 1 -- 172.21.15.1:0/682536560 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b2000f040 con 0x7f5b34102520 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b30c00640 1 -- 172.21.15.1:0/682536560 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b2000dad0 con 0x7f5b34102520 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3410d0c0 con 0x7f5b34102520 2024-02-28T17:29:39.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.592+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3410d5e0 con 0x7f5b34102520 2024-02-28T17:29:39.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.594+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b34064480 con 0x7f5b34102520 2024-02-28T17:29:39.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.594+0000 7f5b30c00640 1 -- 172.21.15.1:0/682536560 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f5b2000dc70 con 0x7f5b34102520 2024-02-28T17:29:39.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.595+0000 7f5b30c00640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b1c06d760 0x7f5b1c06fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:39.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.595+0000 7f5b30c00640 1 -- 172.21.15.1:0/682536560 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5b2008c590 con 0x7f5b34102520 2024-02-28T17:29:39.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.596+0000 7f5b39000640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b1c06d760 0x7f5b1c06fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:39.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.597+0000 7f5b39000640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b1c06d760 0x7f5b1c06fc10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5b28034040 tx=0x7f5b28009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:39.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.600+0000 7f5b30c00640 1 -- 172.21.15.1:0/682536560 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5b20055f10 con 0x7f5b34102520 2024-02-28T17:29:39.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.801+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b3410e310 con 0x7f5b34102520 2024-02-28T17:29:39.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.802+0000 7f5b30c00640 1 -- 172.21.15.1:0/682536560 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16420 (secure 0 0 0) 0x7f5b200595d0 con 0x7f5b34102520 2024-02-28T17:29:39.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:39.804 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":84,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","created":"2024-02-28T16:17:42.450467+0000","modified":"2024-02-28T17:12:10.940975+0000","last_up_change":"2024-02-28T16:40:59.596149+0000","last_in_change":"2024-02-28T16:24:36.954989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-28T16:23:15.048529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-28T16:26:37.820888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-28T16:26:38.430385+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac92369f-8144-4695-87fe-f9cd73fb5f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6803","nonce":524238602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6805","nonce":524238602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6809","nonce":524238602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6807","nonce":524238602}]},"public_addr":"172.21.15.1:6803/524238602","cluster_addr":"172.21.15.1:6805/524238602","heartbeat_back_addr":"172.21.15.1:6809/524238602","heartbeat_front_addr":"172.21.15.1:6807/524238602","state":["exists","up"]},{"osd":1,"uuid":"bfd6ea17-fd9b-46ae-b3ff-547d736cd232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6811","nonce":2672566113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6813","nonce":2672566113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6817","nonce":2672566113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6815","nonce":2672566113}]},"public_addr":"172.21.15.1:6811/2672566113","cluster_addr":"172.21.15.1:6813/2672566113","heartbeat_back_addr":"172.21.15.1:6817/2672566113","heartbeat_front_addr":"172.21.15.1:6815/2672566113","state":["exists","up"]},{"osd":2,"uuid":"b713f513-6604-4e88-8cc1-0f963d9f8599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6819","nonce":4011428371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6821","nonce":4011428371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6825","nonce":4011428371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6823","nonce":4011428371}]},"public_addr":"172.21.15.1:6819/4011428371","cluster_addr":"172.21.15.1:6821/4011428371","heartbeat_back_addr":"172.21.15.1:6825/4011428371","heartbeat_front_addr":"172.21.15.1:6823/4011428371","state":["exists","up"]},{"osd":3,"uuid":"4f425b10-672f-4265-a314-5a9a15237257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6801","nonce":2991662728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6803","nonce":2991662728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6807","nonce":2991662728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6805","nonce":2991662728}]},"public_addr":"172.21.15.66:6801/2991662728","cluster_addr":"172.21.15.66:6803/2991662728","heartbeat_back_addr":"172.21.15.66:6807/2991662728","heartbeat_front_addr":"172.21.15.66:6805/2991662728","state":["exists","up"]},{"osd":4,"uuid":"c1f2651b-47f3-444e-9325-22a0a962c3b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6809","nonce":2169044357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6811","nonce":2169044357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6815","nonce":2169044357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6813","nonce":2169044357}]},"public_addr":"172.21.15.66:6809/2169044357","cluster_addr":"172.21.15.66:6811/2169044357","heartbeat_back_addr":"172.21.15.66:6815/2169044357","heartbeat_front_addr":"172.21.15.66:6813/2169044357","state":["exists","up"]},{"osd":5,"uuid":"e83df14f-7d81-4bb7-9617-024ad94de1a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6817","nonce":2636222889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6819","nonce":2636222889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6823","nonce":2636222889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6821","nonce":2636222889}]},"public_addr":"172.21.15.66:6817/2636222889","cluster_addr":"172.21.15.66:6819/2636222889","heartbeat_back_addr":"172.21.15.66:6823/2636222889","heartbeat_front_addr":"172.21.15.66:6821/2636222889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:01.831777+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:36.505002+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:13.091284+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:47.260219+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:24:23.366141+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:25:00.532840+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.66:6825/1481832428":"2024-02-29T16:41:05.748852+0000","172.21.15.1:6800/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/2061955559":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/4223980442":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/3865282714":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/833315191":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/298653290":"2024-02-29T16:20:26.347228+0000","172.21.15.1:6801/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.66:0/3927136860":"2024-02-29T16:31:44.722257+0000","172.21.15.66:0/2885505378":"2024-02-29T16:31:44.722257+0000","172.21.15.66:6828/2196519723":"2024-02-29T16:31:44.722257+0000","172.21.15.1:0/2780166226":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/179187285":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6801/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:6801/4056372531":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6801/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:6800/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1224289276":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/762266793":"2024-02-29T16:29:38.122257+0000","172.21.15.1:0/2866890740":"2024-02-29T16:29:38.122257+0000","172.21.15.1:6801/762266793":"2024-02-29T16:29:38.122257+0000","172.21.15.1:0/202779563":"2024-02-29T16:29:38.122257+0000","172.21.15.66:0/1412047282":"2024-02-29T16:31:44.722257+0000","172.21.15.1:6826/1940162712":"2024-02-29T16:41:05.748809+0000","172.21.15.1:0/1339161018":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/4090914359":"2024-02-29T16:35:40.611415+0000","172.21.15.66:6829/2196519723":"2024-02-29T16:31:44.722257+0000","172.21.15.1:0/968413644":"2024-02-29T16:29:38.122257+0000","172.21.15.66:0/1854919128":"2024-02-29T16:31:44.722257+0000","172.21.15.1:6800/4056372531":"2024-02-29T16:35:40.611415+0000","172.21.15.1:0/4195777758":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1492016386":"2024-02-29T16:29:38.122257+0000","172.21.15.66:6824/1481832428":"2024-02-29T16:41:05.748852+0000","172.21.15.1:0/2160247963":"2024-02-29T16:35:40.611415+0000","172.21.15.1:0/1575378374":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6827/1940162712":"2024-02-29T16:41:05.748809+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-28T17:29:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b1c06d760 msgr2=0x7f5b1c06fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b1c06d760 0x7f5b1c06fc10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5b28034040 tx=0x7f5b28009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 msgr2=0x7f5b34113250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 0x7f5b34113250 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5b20002a50 tx=0x7f5b20004600 comp rx=0 tx=0).stop 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 shutdown_connections 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f5b1c06d760 0x7f5b1c06fc10 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b34102520 0x7f5b34113250 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 --2- 172.21.15.1:0/682536560 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5b340fc4e0 0x7f5b34112d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 >> 172.21.15.1:0/682536560 conn(0x7f5b340f8060 msgr2=0x7f5b340f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 shutdown_connections 2024-02-28T17:29:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:39.805+0000 7f5b3ba00640 1 -- 172.21.15.1:0/682536560 wait complete. 2024-02-28T17:29:39.826 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-28T16:23:15.048529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-28T16:26:37.820888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-28T16:26:38.430385+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-28T17:29:39.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-28T17:29:40.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.042+0000 7f1487200640 1 -- 172.21.15.1:0/4174628593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 msgr2=0x7f148010cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.042+0000 7f1487200640 1 --2- 172.21.15.1:0/4174628593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 0x7f148010cab0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f14700048d0 tx=0x7f1470009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.042+0000 7f1487200640 1 -- 172.21.15.1:0/4174628593 shutdown_connections 2024-02-28T17:29:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.042+0000 7f1487200640 1 --2- 172.21.15.1:0/4174628593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 0x7f148010cab0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.042+0000 7f1487200640 1 --2- 172.21.15.1:0/4174628593 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f14800719d0 0x7f1480071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.042+0000 7f1487200640 1 -- 172.21.15.1:0/4174628593 >> 172.21.15.1:0/4174628593 conn(0x7f14800f9f90 msgr2=0x7f14800fc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.042+0000 7f1487200640 1 -- 172.21.15.1:0/4174628593 shutdown_connections 2024-02-28T17:29:40.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.042+0000 7f1487200640 1 -- 172.21.15.1:0/4174628593 wait complete. 2024-02-28T17:29:40.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.043+0000 7f1487200640 1 Processor -- start 2024-02-28T17:29:40.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.043+0000 7f1487200640 1 -- start start 2024-02-28T17:29:40.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.044+0000 7f1487200640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f14800719d0 0x7f14801190d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.044+0000 7f1487200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 0x7f1480119610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.044+0000 7f1487200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1480119c10 con 0x7f14800722e0 2024-02-28T17:29:40.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.044+0000 7f1487200640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14801131a0 con 0x7f14800719d0 2024-02-28T17:29:40.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.044+0000 7f1485400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 0x7f1480119610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.044+0000 7f1485400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 0x7f1480119610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57480/0 (socket says 172.21.15.1:57480) 2024-02-28T17:29:40.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.044+0000 7f1485400640 1 -- 172.21.15.1:0/1651685474 learned_addr learned my addr 172.21.15.1:0/1651685474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:40.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.045+0000 7f1485e00640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f14800719d0 0x7f14801190d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.045+0000 7f1485400640 1 -- 172.21.15.1:0/1651685474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f14800719d0 msgr2=0x7f14801190d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.045+0000 7f1485400640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f14800719d0 0x7f14801190d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.045+0000 7f1485400640 1 -- 172.21.15.1:0/1651685474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470005040 con 0x7f14800722e0 2024-02-28T17:29:40.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.045+0000 7f1485400640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 0x7f1480119610 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f147002bda0 tx=0x7f1470009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:40.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.045+0000 7f147a000640 1 -- 172.21.15.1:0/1651685474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14700094c0 con 0x7f14800722e0 2024-02-28T17:29:40.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.045+0000 7f147a000640 1 -- 172.21.15.1:0/1651685474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1470032e50 con 0x7f14800722e0 2024-02-28T17:29:40.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.046+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1480113420 con 0x7f14800722e0 2024-02-28T17:29:40.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.046+0000 7f147a000640 1 -- 172.21.15.1:0/1651685474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14700433b0 con 0x7f14800722e0 2024-02-28T17:29:40.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.046+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14801138e0 con 0x7f14800722e0 2024-02-28T17:29:40.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.047+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148004bcc0 con 0x7f14800722e0 2024-02-28T17:29:40.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.048+0000 7f147a000640 1 -- 172.21.15.1:0/1651685474 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f147002f070 con 0x7f14800722e0 2024-02-28T17:29:40.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.049+0000 7f147a000640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f146806d760 0x7f146806fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.049+0000 7f147a000640 1 -- 172.21.15.1:0/1651685474 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f14700aeda0 con 0x7f14800722e0 2024-02-28T17:29:40.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.049+0000 7f1485e00640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f146806d760 0x7f146806fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.050+0000 7f1485e00640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f146806d760 0x7f146806fc10 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1474007fd0 tx=0x7f1474009e90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:40.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.054+0000 7f147a000640 1 -- 172.21.15.1:0/1651685474 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1470078550 con 0x7f14800722e0 2024-02-28T17:29:40.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.260+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f148010a220 con 0x7f14800722e0 2024-02-28T17:29:40.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.261+0000 7f147a000640 1 -- 172.21.15.1:0/1651685474 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f148010a220 con 0x7f14800722e0 2024-02-28T17:29:40.262 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-02-28T17:29:40.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f146806d760 msgr2=0x7f146806fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f146806d760 0x7f146806fc10 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1474007fd0 tx=0x7f1474009e90 comp rx=0 tx=0).stop 2024-02-28T17:29:40.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 msgr2=0x7f1480119610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 0x7f1480119610 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f147002bda0 tx=0x7f1470009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 shutdown_connections 2024-02-28T17:29:40.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f146806d760 0x7f146806fc10 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14800722e0 0x7f1480119610 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 --2- 172.21.15.1:0/1651685474 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f14800719d0 0x7f14801190d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 >> 172.21.15.1:0/1651685474 conn(0x7f14800f9f90 msgr2=0x7f14800fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:40.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 shutdown_connections 2024-02-28T17:29:40.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.264+0000 7f1487200640 1 -- 172.21.15.1:0/1651685474 wait complete. 2024-02-28T17:29:40.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-28T17:29:40.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.508+0000 7f1901200640 1 -- 172.21.15.1:0/2999894769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 msgr2=0x7f18fc071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.508+0000 7f1901200640 1 --2- 172.21.15.1:0/2999894769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 0x7f18fc071da0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0048d0 tx=0x7f18ec009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.508+0000 7f1901200640 1 -- 172.21.15.1:0/2999894769 shutdown_connections 2024-02-28T17:29:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.508+0000 7f1901200640 1 --2- 172.21.15.1:0/2999894769 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc0722e0 0x7f18fc10ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.508+0000 7f1901200640 1 --2- 172.21.15.1:0/2999894769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 0x7f18fc071da0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.508+0000 7f1901200640 1 -- 172.21.15.1:0/2999894769 >> 172.21.15.1:0/2999894769 conn(0x7f18fc0f9f90 msgr2=0x7f18fc0fc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.508+0000 7f1901200640 1 -- 172.21.15.1:0/2999894769 shutdown_connections 2024-02-28T17:29:40.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.508+0000 7f1901200640 1 -- 172.21.15.1:0/2999894769 wait complete. 2024-02-28T17:29:40.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/2850413454' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-02-28T17:29:40.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:40 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/682536560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.509+0000 7f1901200640 1 Processor -- start 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.509+0000 7f1901200640 1 -- start start 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.510+0000 7f1901200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 0x7f18fc119020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.510+0000 7f1901200640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc0722e0 0x7f18fc119560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.510+0000 7f1901200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc119bf0 con 0x7f18fc0719d0 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.510+0000 7f1901200640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc1130f0 con 0x7f18fc0722e0 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.510+0000 7f18fb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 0x7f18fc119020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.510+0000 7f18fb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 0x7f18fc119020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57496/0 (socket says 172.21.15.1:57496) 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.510+0000 7f18fb400640 1 -- 172.21.15.1:0/4134736143 learned_addr learned my addr 172.21.15.1:0/4134736143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.510+0000 7f18faa00640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc0722e0 0x7f18fc119560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f18fb400640 1 -- 172.21.15.1:0/4134736143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc0722e0 msgr2=0x7f18fc119560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f18fb400640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc0722e0 0x7f18fc119560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f18fb400640 1 -- 172.21.15.1:0/4134736143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec005040 con 0x7f18fc0719d0 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f18faa00640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc0722e0 0x7f18fc119560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f18fb400640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 0x7f18fc119020 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f18ec02bda0 tx=0x7f18ec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f18f3400640 1 -- 172.21.15.1:0/4134736143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18ec0094c0 con 0x7f18fc0719d0 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f18f3400640 1 -- 172.21.15.1:0/4134736143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18ec009620 con 0x7f18fc0719d0 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f18f3400640 1 -- 172.21.15.1:0/4134736143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18ec0435f0 con 0x7f18fc0719d0 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18fc113370 con 0x7f18fc0719d0 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.511+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc113830 con 0x7f18fc0719d0 2024-02-28T17:29:40.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.513+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18fc04bcc0 con 0x7f18fc0719d0 2024-02-28T17:29:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.514+0000 7f18f3400640 1 -- 172.21.15.1:0/4134736143 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f18ec02f070 con 0x7f18fc0719d0 2024-02-28T17:29:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.515+0000 7f18f3400640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f18e006d760 0x7f18e006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.515+0000 7f18f3400640 1 -- 172.21.15.1:0/4134736143 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f18ec0aee70 con 0x7f18fc0719d0 2024-02-28T17:29:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.515+0000 7f18faa00640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f18e006d760 0x7f18e006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.516+0000 7f18faa00640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f18e006d760 0x7f18e006fc10 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f18e8006fd0 tx=0x7f18e8008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:40.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.521+0000 7f18f3400640 1 -- 172.21.15.1:0/4134736143 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f18ec078620 con 0x7f18fc0719d0 2024-02-28T17:29:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/2850413454' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-02-28T17:29:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:40 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/682536560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T17:29:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.721+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f18fc10f8b0 con 0x7f18fc0719d0 2024-02-28T17:29:40.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.721+0000 7f18f3400640 1 -- 172.21.15.1:0/4134736143 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f18ec07bce0 con 0x7f18fc0719d0 2024-02-28T17:29:40.723 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-02-28T17:29:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f18e006d760 msgr2=0x7f18e006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f18e006d760 0x7f18e006fc10 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f18e8006fd0 tx=0x7f18e8008040 comp rx=0 tx=0).stop 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 msgr2=0x7f18fc119020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 0x7f18fc119020 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f18ec02bda0 tx=0x7f18ec009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 shutdown_connections 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f18e006d760 0x7f18e006fc10 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f18fc0722e0 0x7f18fc119560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 --2- 172.21.15.1:0/4134736143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18fc0719d0 0x7f18fc119020 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 >> 172.21.15.1:0/4134736143 conn(0x7f18fc0f9f90 msgr2=0x7f18fc0fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.724+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 shutdown_connections 2024-02-28T17:29:40.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.725+0000 7f1901200640 1 -- 172.21.15.1:0/4134736143 wait complete. 2024-02-28T17:29:40.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-28T17:29:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.970+0000 7f0289e00640 1 -- 172.21.15.1:0/3217625718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 msgr2=0x7f0284106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.970+0000 7f0289e00640 1 --2- 172.21.15.1:0/3217625718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 0x7f0284106370 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f026c0048d0 tx=0x7f026c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.970+0000 7f0289e00640 1 -- 172.21.15.1:0/3217625718 shutdown_connections 2024-02-28T17:29:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.970+0000 7f0289e00640 1 --2- 172.21.15.1:0/3217625718 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 0x7f0284106370 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.970+0000 7f0289e00640 1 --2- 172.21.15.1:0/3217625718 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02840fd390 0x7f02840fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.971+0000 7f0289e00640 1 -- 172.21.15.1:0/3217625718 >> 172.21.15.1:0/3217625718 conn(0x7f02840f6de0 msgr2=0x7f02840f91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.971+0000 7f0289e00640 1 -- 172.21.15.1:0/3217625718 shutdown_connections 2024-02-28T17:29:40.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.971+0000 7f0289e00640 1 -- 172.21.15.1:0/3217625718 wait complete. 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.972+0000 7f0289e00640 1 Processor -- start 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.972+0000 7f0289e00640 1 -- start start 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.972+0000 7f0289e00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02840fd390 0x7f0284112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.972+0000 7f0289e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 0x7f0284113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.972+0000 7f0289e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0284113810 con 0x7f02840fdd30 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.972+0000 7f0289e00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028410cda0 con 0x7f02840fd390 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.973+0000 7f027a000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 0x7f0284113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.973+0000 7f027a000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 0x7f0284113210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57512/0 (socket says 172.21.15.1:57512) 2024-02-28T17:29:40.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.973+0000 7f027a000640 1 -- 172.21.15.1:0/660507014 learned_addr learned my addr 172.21.15.1:0/660507014 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.973+0000 7f0282a00640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02840fd390 0x7f0284112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.973+0000 7f027a000640 1 -- 172.21.15.1:0/660507014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02840fd390 msgr2=0x7f0284112cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.973+0000 7f027a000640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02840fd390 0x7f0284112cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:40.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.973+0000 7f027a000640 1 -- 172.21.15.1:0/660507014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c005040 con 0x7f02840fdd30 2024-02-28T17:29:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.973+0000 7f027a000640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 0x7f0284113210 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f026c02bda0 tx=0x7f026c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.974+0000 7f027b400640 1 -- 172.21.15.1:0/660507014 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f026c0094c0 con 0x7f02840fdd30 2024-02-28T17:29:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.974+0000 7f027b400640 1 -- 172.21.15.1:0/660507014 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f026c009620 con 0x7f02840fdd30 2024-02-28T17:29:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.974+0000 7f027b400640 1 -- 172.21.15.1:0/660507014 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f026c0435f0 con 0x7f02840fdd30 2024-02-28T17:29:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.974+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028410d020 con 0x7f02840fdd30 2024-02-28T17:29:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.974+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028410d4e0 con 0x7f02840fdd30 2024-02-28T17:29:40.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.975+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0284064480 con 0x7f02840fdd30 2024-02-28T17:29:40.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.976+0000 7f027b400640 1 -- 172.21.15.1:0/660507014 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f026c02f070 con 0x7f02840fdd30 2024-02-28T17:29:40.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.977+0000 7f027b400640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f026806d760 0x7f026806fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:40.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.977+0000 7f027b400640 1 -- 172.21.15.1:0/660507014 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f026c0aee30 con 0x7f02840fdd30 2024-02-28T17:29:40.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.977+0000 7f0282a00640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f026806d760 0x7f026806fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:40.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.978+0000 7f0282a00640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f026806d760 0x7f026806fc10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f0274004550 tx=0x7f0274009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:40.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:40.982+0000 7f027b400640 1 -- 172.21.15.1:0/660507014 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f026c0785e0 con 0x7f02840fdd30 2024-02-28T17:29:41.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.184+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f0284103a70 con 0x7f02840fdd30 2024-02-28T17:29:41.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.185+0000 7f027b400640 1 -- 172.21.15.1:0/660507014 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0284103a70 con 0x7f02840fdd30 2024-02-28T17:29:41.186 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.187+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f026806d760 msgr2=0x7f026806fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.187+0000 7f0289e00640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f026806d760 0x7f026806fc10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f0274004550 tx=0x7f0274009040 comp rx=0 tx=0).stop 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.187+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 msgr2=0x7f0284113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.187+0000 7f0289e00640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 0x7f0284113210 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f026c02bda0 tx=0x7f026c009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.188+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 shutdown_connections 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.188+0000 7f0289e00640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f026806d760 0x7f026806fc10 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.188+0000 7f0289e00640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02840fdd30 0x7f0284113210 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.188+0000 7f0289e00640 1 --2- 172.21.15.1:0/660507014 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f02840fd390 0x7f0284112cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.188+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 >> 172.21.15.1:0/660507014 conn(0x7f02840f6de0 msgr2=0x7f0284100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:41.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.188+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 shutdown_connections 2024-02-28T17:29:41.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.188+0000 7f0289e00640 1 -- 172.21.15.1:0/660507014 wait complete. 2024-02-28T17:29:41.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-28T17:29:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.430+0000 7f5709e00640 1 -- 172.21.15.1:0/1180164035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704102740 msgr2=0x7f5704102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.430+0000 7f5709e00640 1 --2- 172.21.15.1:0/1180164035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704102740 0x7f5704102bb0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f56f00048d0 tx=0x7f56f0009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.430+0000 7f5709e00640 1 -- 172.21.15.1:0/1180164035 shutdown_connections 2024-02-28T17:29:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.430+0000 7f5709e00640 1 --2- 172.21.15.1:0/1180164035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704102740 0x7f5704102bb0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.430+0000 7f5709e00640 1 --2- 172.21.15.1:0/1180164035 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5704108760 0x7f5704108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.430+0000 7f5709e00640 1 -- 172.21.15.1:0/1180164035 >> 172.21.15.1:0/1180164035 conn(0x7f5704071fc0 msgr2=0x7f57040743d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.430+0000 7f5709e00640 1 -- 172.21.15.1:0/1180164035 shutdown_connections 2024-02-28T17:29:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.430+0000 7f5709e00640 1 -- 172.21.15.1:0/1180164035 wait complete. 2024-02-28T17:29:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.431+0000 7f5709e00640 1 Processor -- start 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.431+0000 7f5709e00640 1 -- start start 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.431+0000 7f5709e00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5704102740 0x7f57041190d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.431+0000 7f5709e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704108760 0x7f5704119610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.431+0000 7f5709e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5704119c10 con 0x7f5704108760 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.431+0000 7f5709e00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57041131a0 con 0x7f5704102740 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.431+0000 7f5703e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704108760 0x7f5704119610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.432+0000 7f5703e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704108760 0x7f5704119610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57540/0 (socket says 172.21.15.1:57540) 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.432+0000 7f5703e00640 1 -- 172.21.15.1:0/758302784 learned_addr learned my addr 172.21.15.1:0/758302784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.432+0000 7f5708a00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5704102740 0x7f57041190d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.432+0000 7f5703e00640 1 -- 172.21.15.1:0/758302784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5704102740 msgr2=0x7f57041190d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.432+0000 7f5703e00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5704102740 0x7f57041190d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.432+0000 7f5703e00640 1 -- 172.21.15.1:0/758302784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f0005040 con 0x7f5704108760 2024-02-28T17:29:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.432+0000 7f5703e00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704108760 0x7f5704119610 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f56f002bda0 tx=0x7f56f0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.433+0000 7f5700c00640 1 -- 172.21.15.1:0/758302784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56f00092b0 con 0x7f5704108760 2024-02-28T17:29:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.433+0000 7f5700c00640 1 -- 172.21.15.1:0/758302784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56f003b400 con 0x7f5704108760 2024-02-28T17:29:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.433+0000 7f5700c00640 1 -- 172.21.15.1:0/758302784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56f00435d0 con 0x7f5704108760 2024-02-28T17:29:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.433+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5704113420 con 0x7f5704108760 2024-02-28T17:29:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.433+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57041138e0 con 0x7f5704108760 2024-02-28T17:29:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.434+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570404bcc0 con 0x7f5704108760 2024-02-28T17:29:41.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.435+0000 7f5700c00640 1 -- 172.21.15.1:0/758302784 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f56f002f070 con 0x7f5704108760 2024-02-28T17:29:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.436+0000 7f5700c00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f56ec06d760 0x7f56ec06fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.436+0000 7f5700c00640 1 -- 172.21.15.1:0/758302784 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f56f00aed20 con 0x7f5704108760 2024-02-28T17:29:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.437+0000 7f5708a00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f56ec06d760 0x7f56ec06fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.437+0000 7f5708a00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f56ec06d760 0x7f56ec06fc10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f56f8004a70 tx=0x7f56f800b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.441+0000 7f5700c00640 1 -- 172.21.15.1:0/758302784 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f56f0078550 con 0x7f5704108760 2024-02-28T17:29:41.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:41 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1651685474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T17:29:41.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:41 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/4134736143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-28T17:29:41.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:29:41.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:41 smithi066 ceph-mon[85781]: pgmap v1663: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:41.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:41 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/660507014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-28T17:29:41.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:41 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1651685474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T17:29:41.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:41 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/4134736143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-28T17:29:41.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:29:41.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:41 smithi001 ceph-mon[112940]: pgmap v1663: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:41.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:41 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/660507014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-28T17:29:41.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.662+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f57041059b0 con 0x7f5704108760 2024-02-28T17:29:41.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.663+0000 7f5700c00640 1 -- 172.21.15.1:0/758302784 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 37 entries v84) v1 ==== 81+0+2156 (secure 0 0 0) 0x7f56f007bc10 con 0x7f5704108760 2024-02-28T17:29:41.665 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:6825/1481832428 2024-02-29T16:41:05.748852+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/701199259 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2061955559 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4223980442 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3865282714 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/833315191 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/298653290 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/3844925093 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:0/3927136860 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:0/2885505378 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:6828/2196519723 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2780166226 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/3844925093 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:41.666 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/179187285 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/3320028482 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/4056372531 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/701199259 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/3320028482 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1224289276 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/762266793 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2866890740 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/762266793 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/202779563 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:0/1412047282 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6826/1940162712 2024-02-29T16:41:05.748809+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1339161018 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4090914359 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:6829/2196519723 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/968413644 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:0/1854919128 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/4056372531 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4195777758 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1492016386 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:6824/1481832428 2024-02-29T16:41:05.748852+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2160247963 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1575378374 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:41.668 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6827/1940162712 2024-02-29T16:41:05.748809+0000 2024-02-28T17:29:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:listed 37 entries 2024-02-28T17:29:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.665+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f56ec06d760 msgr2=0x7f56ec06fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.665+0000 7f5709e00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f56ec06d760 0x7f56ec06fc10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f56f8004a70 tx=0x7f56f800b040 comp rx=0 tx=0).stop 2024-02-28T17:29:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.665+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704108760 msgr2=0x7f5704119610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.665+0000 7f5709e00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704108760 0x7f5704119610 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f56f002bda0 tx=0x7f56f0009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.666+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 shutdown_connections 2024-02-28T17:29:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.666+0000 7f5709e00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f56ec06d760 0x7f56ec06fc10 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.666+0000 7f5709e00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5704108760 0x7f5704119610 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.666+0000 7f5709e00640 1 --2- 172.21.15.1:0/758302784 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f5704102740 0x7f57041190d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.666+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 >> 172.21.15.1:0/758302784 conn(0x7f5704071fc0 msgr2=0x7f5704074d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.666+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 shutdown_connections 2024-02-28T17:29:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.666+0000 7f5709e00640 1 -- 172.21.15.1:0/758302784 wait complete. 2024-02-28T17:29:41.686 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-02-28T17:29:41.686 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-28T17:29:41.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-28T17:29:41.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.959+0000 7f2a77e00640 1 -- 172.21.15.1:0/616585740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a781043e0 msgr2=0x7f2a781047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:41.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.959+0000 7f2a77e00640 1 --2- 172.21.15.1:0/616585740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a781043e0 0x7f2a781047b0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f2a680048d0 tx=0x7f2a68009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.959+0000 7f2a77e00640 1 -- 172.21.15.1:0/616585740 shutdown_connections 2024-02-28T17:29:41.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.959+0000 7f2a77e00640 1 --2- 172.21.15.1:0/616585740 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2a78104d80 0x7f2a780fe410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.959+0000 7f2a77e00640 1 --2- 172.21.15.1:0/616585740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a781043e0 0x7f2a781047b0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.959+0000 7f2a77e00640 1 -- 172.21.15.1:0/616585740 >> 172.21.15.1:0/616585740 conn(0x7f2a780f9ff0 msgr2=0x7f2a780fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:41.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.959+0000 7f2a77e00640 1 -- 172.21.15.1:0/616585740 shutdown_connections 2024-02-28T17:29:41.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.959+0000 7f2a77e00640 1 -- 172.21.15.1:0/616585740 wait complete. 2024-02-28T17:29:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a77e00640 1 Processor -- start 2024-02-28T17:29:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a77e00640 1 -- start start 2024-02-28T17:29:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a77e00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2a781043e0 0x7f2a78114c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a77e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a78104d80 0x7f2a78115170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a77e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a78115770 con 0x7f2a78104d80 2024-02-28T17:29:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a77e00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7810ed00 con 0x7f2a781043e0 2024-02-28T17:29:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a76000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a78104d80 0x7f2a78115170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a76000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a78104d80 0x7f2a78115170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60776/0 (socket says 172.21.15.1:60776) 2024-02-28T17:29:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.960+0000 7f2a76000640 1 -- 172.21.15.1:0/381199318 learned_addr learned my addr 172.21.15.1:0/381199318 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.961+0000 7f2a76a00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2a781043e0 0x7f2a78114c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.961+0000 7f2a76000640 1 -- 172.21.15.1:0/381199318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2a781043e0 msgr2=0x7f2a78114c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.961+0000 7f2a76000640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2a781043e0 0x7f2a78114c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.961+0000 7f2a76000640 1 -- 172.21.15.1:0/381199318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a68005040 con 0x7f2a78104d80 2024-02-28T17:29:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.961+0000 7f2a76000640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a78104d80 0x7f2a78115170 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f2a64002a50 tx=0x7f2a64004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:41.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.962+0000 7f2a6ea00640 1 -- 172.21.15.1:0/381199318 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a64007c30 con 0x7f2a78104d80 2024-02-28T17:29:41.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.962+0000 7f2a6ea00640 1 -- 172.21.15.1:0/381199318 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a6400f040 con 0x7f2a78104d80 2024-02-28T17:29:41.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.962+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7810efe0 con 0x7f2a78104d80 2024-02-28T17:29:41.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.962+0000 7f2a6ea00640 1 -- 172.21.15.1:0/381199318 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a6400db50 con 0x7f2a78104d80 2024-02-28T17:29:41.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.962+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7810f500 con 0x7f2a78104d80 2024-02-28T17:29:41.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.963+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7804bcc0 con 0x7f2a78104d80 2024-02-28T17:29:41.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.964+0000 7f2a6ea00640 1 -- 172.21.15.1:0/381199318 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f2a6400dcf0 con 0x7f2a78104d80 2024-02-28T17:29:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.965+0000 7f2a6ea00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2a5c06d750 0x7f2a5c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.965+0000 7f2a6ea00640 1 -- 172.21.15.1:0/381199318 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f2a6408c720 con 0x7f2a78104d80 2024-02-28T17:29:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.966+0000 7f2a76a00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2a5c06d750 0x7f2a5c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:41.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.966+0000 7f2a76a00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2a5c06d750 0x7f2a5c06fc00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f2a68004540 tx=0x7f2a68009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:41.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:41.970+0000 7f2a6ea00640 1 -- 172.21.15.1:0/381199318 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2a64055ff0 con 0x7f2a78104d80 2024-02-28T17:29:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.171+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a7810f770 con 0x7f2a78104d80 2024-02-28T17:29:42.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.173+0000 7f2a6ea00640 1 -- 172.21.15.1:0/381199318 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16420 (secure 0 0 0) 0x7f2a640596b0 con 0x7f2a78104d80 2024-02-28T17:29:42.175 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-02-28T17:29:42.175 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":84,"fsid":"c0118718-d654-11ee-95c3-87774f69a715","created":"2024-02-28T16:17:42.450467+0000","modified":"2024-02-28T17:12:10.940975+0000","last_up_change":"2024-02-28T16:40:59.596149+0000","last_in_change":"2024-02-28T16:24:36.954989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-28T16:23:15.048529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-28T16:26:37.820888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-28T16:26:38.430385+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac92369f-8144-4695-87fe-f9cd73fb5f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6803","nonce":524238602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6805","nonce":524238602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6809","nonce":524238602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":524238602},{"type":"v1","addr":"172.21.15.1:6807","nonce":524238602}]},"public_addr":"172.21.15.1:6803/524238602","cluster_addr":"172.21.15.1:6805/524238602","heartbeat_back_addr":"172.21.15.1:6809/524238602","heartbeat_front_addr":"172.21.15.1:6807/524238602","state":["exists","up"]},{"osd":1,"uuid":"bfd6ea17-fd9b-46ae-b3ff-547d736cd232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6811","nonce":2672566113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6813","nonce":2672566113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6817","nonce":2672566113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2672566113},{"type":"v1","addr":"172.21.15.1:6815","nonce":2672566113}]},"public_addr":"172.21.15.1:6811/2672566113","cluster_addr":"172.21.15.1:6813/2672566113","heartbeat_back_addr":"172.21.15.1:6817/2672566113","heartbeat_front_addr":"172.21.15.1:6815/2672566113","state":["exists","up"]},{"osd":2,"uuid":"b713f513-6604-4e88-8cc1-0f963d9f8599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6819","nonce":4011428371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6821","nonce":4011428371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6825","nonce":4011428371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":4011428371},{"type":"v1","addr":"172.21.15.1:6823","nonce":4011428371}]},"public_addr":"172.21.15.1:6819/4011428371","cluster_addr":"172.21.15.1:6821/4011428371","heartbeat_back_addr":"172.21.15.1:6825/4011428371","heartbeat_front_addr":"172.21.15.1:6823/4011428371","state":["exists","up"]},{"osd":3,"uuid":"4f425b10-672f-4265-a314-5a9a15237257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6801","nonce":2991662728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6803","nonce":2991662728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6807","nonce":2991662728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2991662728},{"type":"v1","addr":"172.21.15.66:6805","nonce":2991662728}]},"public_addr":"172.21.15.66:6801/2991662728","cluster_addr":"172.21.15.66:6803/2991662728","heartbeat_back_addr":"172.21.15.66:6807/2991662728","heartbeat_front_addr":"172.21.15.66:6805/2991662728","state":["exists","up"]},{"osd":4,"uuid":"c1f2651b-47f3-444e-9325-22a0a962c3b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6809","nonce":2169044357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6810","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6811","nonce":2169044357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6815","nonce":2169044357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":2169044357},{"type":"v1","addr":"172.21.15.66:6813","nonce":2169044357}]},"public_addr":"172.21.15.66:6809/2169044357","cluster_addr":"172.21.15.66:6811/2169044357","heartbeat_back_addr":"172.21.15.66:6815/2169044357","heartbeat_front_addr":"172.21.15.66:6813/2169044357","state":["exists","up"]},{"osd":5,"uuid":"e83df14f-7d81-4bb7-9617-024ad94de1a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6817","nonce":2636222889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6819","nonce":2636222889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6823","nonce":2636222889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":2636222889},{"type":"v1","addr":"172.21.15.66:6821","nonce":2636222889}]},"public_addr":"172.21.15.66:6817/2636222889","cluster_addr":"172.21.15.66:6819/2636222889","heartbeat_back_addr":"172.21.15.66:6823/2636222889","heartbeat_front_addr":"172.21.15.66:6821/2636222889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:01.831777+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:22:36.505002+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:13.091284+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:23:47.260219+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:24:23.366141+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-28T16:25:00.532840+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.66:6825/1481832428":"2024-02-29T16:41:05.748852+0000","172.21.15.1:6800/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/2061955559":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/4223980442":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/3865282714":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/833315191":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/298653290":"2024-02-29T16:20:26.347228+0000","172.21.15.1:6801/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.66:0/3927136860":"2024-02-29T16:31:44.722257+0000","172.21.15.66:0/2885505378":"2024-02-29T16:31:44.722257+0000","172.21.15.66:6828/2196519723":"2024-02-29T16:31:44.722257+0000","172.21.15.1:0/2780166226":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/3844925093":"2024-02-29T16:18:15.275613+0000","172.21.15.1:0/179187285":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6801/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:6801/4056372531":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6801/701199259":"2024-02-29T16:20:26.347228+0000","172.21.15.1:6800/3320028482":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1224289276":"2024-02-29T16:18:15.275613+0000","172.21.15.1:6800/762266793":"2024-02-29T16:29:38.122257+0000","172.21.15.1:0/2866890740":"2024-02-29T16:29:38.122257+0000","172.21.15.1:6801/762266793":"2024-02-29T16:29:38.122257+0000","172.21.15.1:0/202779563":"2024-02-29T16:29:38.122257+0000","172.21.15.66:0/1412047282":"2024-02-29T16:31:44.722257+0000","172.21.15.1:6826/1940162712":"2024-02-29T16:41:05.748809+0000","172.21.15.1:0/1339161018":"2024-02-29T16:20:26.347228+0000","172.21.15.1:0/4090914359":"2024-02-29T16:35:40.611415+0000","172.21.15.66:6829/2196519723":"2024-02-29T16:31:44.722257+0000","172.21.15.1:0/968413644":"2024-02-29T16:29:38.122257+0000","172.21.15.66:0/1854919128":"2024-02-29T16:31:44.722257+0000","172.21.15.1:6800/4056372531":"2024-02-29T16:35:40.611415+0000","172.21.15.1:0/4195777758":"2024-02-29T16:18:55.593679+0000","172.21.15.1:0/1492016386":"2024-02-29T16:29:38.122257+0000","172.21.15.66:6824/1481832428":"2024-02-29T16:41:05.748852+0000","172.21.15.1:0/2160247963":"2024-02-29T16:35:40.611415+0000","172.21.15.1:0/1575378374":"2024-02-29T16:35:40.611415+0000","172.21.15.1:6827/1940162712":"2024-02-29T16:41:05.748809+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-28T17:29:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2a5c06d750 msgr2=0x7f2a5c06fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2a5c06d750 0x7f2a5c06fc00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f2a68004540 tx=0x7f2a68009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a78104d80 msgr2=0x7f2a78115170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a78104d80 0x7f2a78115170 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f2a64002a50 tx=0x7f2a64004600 comp rx=0 tx=0).stop 2024-02-28T17:29:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 shutdown_connections 2024-02-28T17:29:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f2a5c06d750 0x7f2a5c06fc00 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a78104d80 0x7f2a78115170 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 --2- 172.21.15.1:0/381199318 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f2a781043e0 0x7f2a78114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 >> 172.21.15.1:0/381199318 conn(0x7f2a780f9ff0 msgr2=0x7f2a780fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 shutdown_connections 2024-02-28T17:29:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.176+0000 7f2a77e00640 1 -- 172.21.15.1:0/381199318 wait complete. 2024-02-28T17:29:42.197 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-28T16:23:15.048529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-28T16:26:37.820888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-28T16:26:38.430385+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-28T17:29:42.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-28T17:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.419+0000 7f9171800640 1 -- 172.21.15.1:0/2134841317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 msgr2=0x7f916c0fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.419+0000 7f9171800640 1 --2- 172.21.15.1:0/2134841317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 0x7f916c0fc150 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f91580048d0 tx=0x7f9158009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.419+0000 7f9171800640 1 -- 172.21.15.1:0/2134841317 shutdown_connections 2024-02-28T17:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.419+0000 7f9171800640 1 --2- 172.21.15.1:0/2134841317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 0x7f916c0fc150 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.419+0000 7f9171800640 1 --2- 172.21.15.1:0/2134841317 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f916c102120 0x7f916c1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.419+0000 7f9171800640 1 -- 172.21.15.1:0/2134841317 >> 172.21.15.1:0/2134841317 conn(0x7f916c0f7cf0 msgr2=0x7f916c0fa100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.419+0000 7f9171800640 1 -- 172.21.15.1:0/2134841317 shutdown_connections 2024-02-28T17:29:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.419+0000 7f9171800640 1 -- 172.21.15.1:0/2134841317 wait complete. 2024-02-28T17:29:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.420+0000 7f9171800640 1 Processor -- start 2024-02-28T17:29:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.420+0000 7f9171800640 1 -- start start 2024-02-28T17:29:42.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f9171800640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f916c102120 0x7f916c114c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:42.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f9171800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 0x7f916c115170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:42.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f9171800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c115770 con 0x7f916c102ac0 2024-02-28T17:29:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f9171800640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c10ed00 con 0x7f916c102120 2024-02-28T17:29:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f916b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 0x7f916c115170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f916b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 0x7f916c115170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60806/0 (socket says 172.21.15.1:60806) 2024-02-28T17:29:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f916b400640 1 -- 172.21.15.1:0/1312114335 learned_addr learned my addr 172.21.15.1:0/1312114335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f916be00640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f916c102120 0x7f916c114c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.421+0000 7f916b400640 1 -- 172.21.15.1:0/1312114335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f916c102120 msgr2=0x7f916c114c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:42.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.422+0000 7f916b400640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f916c102120 0x7f916c114c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.422+0000 7f916b400640 1 -- 172.21.15.1:0/1312114335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9158005040 con 0x7f916c102ac0 2024-02-28T17:29:42.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.422+0000 7f916b400640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 0x7f916c115170 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f915802bda0 tx=0x7f9158009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:42.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.422+0000 7f9163e00640 1 -- 172.21.15.1:0/1312114335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91580094c0 con 0x7f916c102ac0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.422+0000 7f9163e00640 1 -- 172.21.15.1:0/1312114335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9158009620 con 0x7f916c102ac0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.422+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916c10ef80 con 0x7f916c102ac0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.422+0000 7f9163e00640 1 -- 172.21.15.1:0/1312114335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91580435f0 con 0x7f916c102ac0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.422+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c10f440 con 0x7f916c102ac0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.424+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916c04bcc0 con 0x7f916c102ac0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.425+0000 7f9163e00640 1 -- 172.21.15.1:0/1312114335 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f915802f070 con 0x7f916c102ac0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.426+0000 7f9163e00640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f915006d760 0x7f915006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.426+0000 7f9163e00640 1 -- 172.21.15.1:0/1312114335 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f91580aef70 con 0x7f916c102ac0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.426+0000 7f916be00640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f915006d760 0x7f915006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:42.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.426+0000 7f916be00640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f915006d760 0x7f915006fc10 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f915c0041d0 tx=0x7f915c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:42.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.430+0000 7f9163e00640 1 -- 172.21.15.1:0/1312114335 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9158078720 con 0x7f916c102ac0 2024-02-28T17:29:42.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:42 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/758302784' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-28T17:29:42.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:42 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/381199318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T17:29:42.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:42 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/758302784' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-28T17:29:42.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:42 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/381199318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-28T17:29:42.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.630+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f916c10f720 con 0x7f916c102ac0 2024-02-28T17:29:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.630+0000 7f9163e00640 1 -- 172.21.15.1:0/1312114335 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f915807bde0 con 0x7f916c102ac0 2024-02-28T17:29:42.632 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-02-28T17:29:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.633+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f915006d760 msgr2=0x7f915006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f915006d760 0x7f915006fc10 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f915c0041d0 tx=0x7f915c009040 comp rx=0 tx=0).stop 2024-02-28T17:29:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 msgr2=0x7f916c115170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 0x7f916c115170 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f915802bda0 tx=0x7f9158009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 shutdown_connections 2024-02-28T17:29:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f915006d760 0x7f915006fc10 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c102ac0 0x7f916c115170 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 --2- 172.21.15.1:0/1312114335 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f916c102120 0x7f916c114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 >> 172.21.15.1:0/1312114335 conn(0x7f916c0f7cf0 msgr2=0x7f916c0f96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 shutdown_connections 2024-02-28T17:29:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.634+0000 7f9171800640 1 -- 172.21.15.1:0/1312114335 wait complete. 2024-02-28T17:29:42.655 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-28T17:29:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.870+0000 7fbbfde00640 1 -- 172.21.15.1:0/4241048182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf806fd80 msgr2=0x7fbbf80701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.870+0000 7fbbfde00640 1 --2- 172.21.15.1:0/4241048182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf806fd80 0x7fbbf80701f0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80048d0 tx=0x7fbbe8009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.870+0000 7fbbfde00640 1 -- 172.21.15.1:0/4241048182 shutdown_connections 2024-02-28T17:29:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.870+0000 7fbbfde00640 1 --2- 172.21.15.1:0/4241048182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf806fd80 0x7fbbf80701f0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.870+0000 7fbbfde00640 1 --2- 172.21.15.1:0/4241048182 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbf8071470 0x7fbbf806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.870+0000 7fbbfde00640 1 -- 172.21.15.1:0/4241048182 >> 172.21.15.1:0/4241048182 conn(0x7fbbf80f7cf0 msgr2=0x7fbbf80fa100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.870+0000 7fbbfde00640 1 -- 172.21.15.1:0/4241048182 shutdown_connections 2024-02-28T17:29:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.870+0000 7fbbfde00640 1 -- 172.21.15.1:0/4241048182 wait complete. 2024-02-28T17:29:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.871+0000 7fbbfde00640 1 Processor -- start 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.871+0000 7fbbfde00640 1 -- start start 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbfde00640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbf806fd80 0x7fbbf8118fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbfde00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf8071470 0x7fbbf81194f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbfde00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf8119af0 con 0x7fbbf8071470 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbfde00640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf8113080 con 0x7fbbf806fd80 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf8071470 0x7fbbf81194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf8071470 0x7fbbf81194f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60822/0 (socket says 172.21.15.1:60822) 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbf7e00640 1 -- 172.21.15.1:0/1488445351 learned_addr learned my addr 172.21.15.1:0/1488445351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbfca00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbf806fd80 0x7fbbf8118fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbf7e00640 1 -- 172.21.15.1:0/1488445351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbf806fd80 msgr2=0x7fbbf8118fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbf7e00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbf806fd80 0x7fbbf8118fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:42.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbf7e00640 1 -- 172.21.15.1:0/1488445351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8005040 con 0x7fbbf8071470 2024-02-28T17:29:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.872+0000 7fbbfca00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbf806fd80 0x7fbbf8118fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.873+0000 7fbbf7e00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf8071470 0x7fbbf81194f0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8030640 tx=0x7fbbe8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.873+0000 7fbbf4c00640 1 -- 172.21.15.1:0/1488445351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe80092e0 con 0x7fbbf8071470 2024-02-28T17:29:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.873+0000 7fbbf4c00640 1 -- 172.21.15.1:0/1488445351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbe8009440 con 0x7fbbf8071470 2024-02-28T17:29:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.873+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf8113300 con 0x7fbbf8071470 2024-02-28T17:29:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.873+0000 7fbbf4c00640 1 -- 172.21.15.1:0/1488445351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe8043640 con 0x7fbbf8071470 2024-02-28T17:29:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.873+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf8113820 con 0x7fbbf8071470 2024-02-28T17:29:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.874+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf804bcc0 con 0x7fbbf8071470 2024-02-28T17:29:42.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.875+0000 7fbbf4c00640 1 -- 172.21.15.1:0/1488445351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fbbe803b420 con 0x7fbbf8071470 2024-02-28T17:29:42.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.876+0000 7fbbf4c00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbbe006d750 0x7fbbe006fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:42.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.876+0000 7fbbf4c00640 1 -- 172.21.15.1:0/1488445351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbbe80afa90 con 0x7fbbf8071470 2024-02-28T17:29:42.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.877+0000 7fbbfca00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbbe006d750 0x7fbbe006fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:42.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.878+0000 7fbbfca00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbbe006d750 0x7fbbe006fc00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbbec004d60 tx=0x7fbbec00b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:42.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:42.881+0000 7fbbf4c00640 1 -- 172.21.15.1:0/1488445351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbbe80792c0 con 0x7fbbf8071470 2024-02-28T17:29:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.084+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fbbf81145b0 con 0x7fbbf8071470 2024-02-28T17:29:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.085+0000 7fbbf4c00640 1 -- 172.21.15.1:0/1488445351 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbbe807c980 con 0x7fbbf8071470 2024-02-28T17:29:43.087 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-02-28T17:29:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.087+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbbe006d750 msgr2=0x7fbbe006fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.087+0000 7fbbfde00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbbe006d750 0x7fbbe006fc00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbbec004d60 tx=0x7fbbec00b040 comp rx=0 tx=0).stop 2024-02-28T17:29:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf8071470 msgr2=0x7fbbf81194f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf8071470 0x7fbbf81194f0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8030640 tx=0x7fbbe8009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 shutdown_connections 2024-02-28T17:29:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fbbe006d750 0x7fbbe006fc00 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf8071470 0x7fbbf81194f0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 --2- 172.21.15.1:0/1488445351 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fbbf806fd80 0x7fbbf8118fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 >> 172.21.15.1:0/1488445351 conn(0x7fbbf80f7cf0 msgr2=0x7fbbf810b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 shutdown_connections 2024-02-28T17:29:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.088+0000 7fbbfde00640 1 -- 172.21.15.1:0/1488445351 wait complete. 2024-02-28T17:29:43.107 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-28T17:29:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.314+0000 7fd5d9600640 1 -- 172.21.15.1:0/2032324460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073e80 msgr2=0x7fd5d410cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.314+0000 7fd5d9600640 1 --2- 172.21.15.1:0/2032324460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073e80 0x7fd5d410cd40 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fd5c00048d0 tx=0x7fd5c0009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.315+0000 7fd5d9600640 1 -- 172.21.15.1:0/2032324460 shutdown_connections 2024-02-28T17:29:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.315+0000 7fd5d9600640 1 --2- 172.21.15.1:0/2032324460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073e80 0x7fd5d410cd40 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.315+0000 7fd5d9600640 1 --2- 172.21.15.1:0/2032324460 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5d4073570 0x7fd5d4073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.315+0000 7fd5d9600640 1 -- 172.21.15.1:0/2032324460 >> 172.21.15.1:0/2032324460 conn(0x7fd5d40fa160 msgr2=0x7fd5d40fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:43.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.315+0000 7fd5d9600640 1 -- 172.21.15.1:0/2032324460 shutdown_connections 2024-02-28T17:29:43.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.315+0000 7fd5d9600640 1 -- 172.21.15.1:0/2032324460 wait complete. 2024-02-28T17:29:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.316+0000 7fd5d9600640 1 Processor -- start 2024-02-28T17:29:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.316+0000 7fd5d9600640 1 -- start start 2024-02-28T17:29:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.316+0000 7fd5d9600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073570 0x7fd5d41192b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.316+0000 7fd5d9600640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5d4073e80 0x7fd5d41197f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:43.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.316+0000 7fd5d9600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4119df0 con 0x7fd5d4073570 2024-02-28T17:29:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.316+0000 7fd5d9600640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4113380 con 0x7fd5d4073e80 2024-02-28T17:29:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.317+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073570 0x7fd5d41192b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.317+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073570 0x7fd5d41192b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60840/0 (socket says 172.21.15.1:60840) 2024-02-28T17:29:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.317+0000 7fd5d2a00640 1 -- 172.21.15.1:0/196469800 learned_addr learned my addr 172.21.15.1:0/196469800 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.317+0000 7fd5d2a00640 1 -- 172.21.15.1:0/196469800 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5d4073e80 msgr2=0x7fd5d41197f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.317+0000 7fd5d2a00640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5d4073e80 0x7fd5d41197f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.317+0000 7fd5d2a00640 1 -- 172.21.15.1:0/196469800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c0005040 con 0x7fd5d4073570 2024-02-28T17:29:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.317+0000 7fd5d2a00640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073570 0x7fd5d41192b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4002a50 tx=0x7fd5c4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.318+0000 7fd5caa00640 1 -- 172.21.15.1:0/196469800 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5c4007c30 con 0x7fd5d4073570 2024-02-28T17:29:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.318+0000 7fd5caa00640 1 -- 172.21.15.1:0/196469800 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5c400f040 con 0x7fd5d4073570 2024-02-28T17:29:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.318+0000 7fd5caa00640 1 -- 172.21.15.1:0/196469800 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5c400db50 con 0x7fd5d4073570 2024-02-28T17:29:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.318+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d4113660 con 0x7fd5d4073570 2024-02-28T17:29:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.318+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d4113b80 con 0x7fd5d4073570 2024-02-28T17:29:43.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.319+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d4064030 con 0x7fd5d4073570 2024-02-28T17:29:43.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.321+0000 7fd5caa00640 1 -- 172.21.15.1:0/196469800 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7fd5c400dcf0 con 0x7fd5d4073570 2024-02-28T17:29:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.322+0000 7fd5caa00640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd5b806d750 0x7fd5b806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.322+0000 7fd5caa00640 1 -- 172.21.15.1:0/196469800 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd5c408c7c0 con 0x7fd5d4073570 2024-02-28T17:29:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.322+0000 7fd5d2000640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd5b806d750 0x7fd5b806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:43.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.323+0000 7fd5d2000640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd5b806d750 0x7fd5b806fc00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0034040 tx=0x7fd5c0009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.326+0000 7fd5caa00640 1 -- 172.21.15.1:0/196469800 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd5c4055ff0 con 0x7fd5d4073570 2024-02-28T17:29:43.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:43 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1312114335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T17:29:43.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:43 smithi066 ceph-mon[85781]: pgmap v1664: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:43.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:43 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/1488445351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-28T17:29:43.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.526+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fd5d41148b0 con 0x7fd5d4073570 2024-02-28T17:29:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.527+0000 7fd5caa00640 1 -- 172.21.15.1:0/196469800 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd5c40596b0 con 0x7fd5d4073570 2024-02-28T17:29:43.528 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-02-28T17:29:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd5b806d750 msgr2=0x7fd5b806fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd5b806d750 0x7fd5b806fc00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0034040 tx=0x7fd5c0009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073570 msgr2=0x7fd5d41192b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073570 0x7fd5d41192b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4002a50 tx=0x7fd5c4004600 comp rx=0 tx=0).stop 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 shutdown_connections 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7fd5b806d750 0x7fd5b806fc00 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7fd5d4073e80 0x7fd5d41197f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 --2- 172.21.15.1:0/196469800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5d4073570 0x7fd5d41192b0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 >> 172.21.15.1:0/196469800 conn(0x7fd5d40fa160 msgr2=0x7fd5d40fbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 shutdown_connections 2024-02-28T17:29:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.530+0000 7fd5d9600640 1 -- 172.21.15.1:0/196469800 wait complete. 2024-02-28T17:29:43.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:43 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1312114335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-28T17:29:43.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:43 smithi001 ceph-mon[112940]: pgmap v1664: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:43.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:43 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/1488445351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-28T17:29:43.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-28T17:29:43.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.763+0000 7f1be8600640 1 -- 172.21.15.1:0/1949449138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be0100110 msgr2=0x7f1be0100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.763+0000 7f1be8600640 1 --2- 172.21.15.1:0/1949449138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be0100110 0x7f1be0100560 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0048d0 tx=0x7f1bcc009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.763+0000 7f1be8600640 1 -- 172.21.15.1:0/1949449138 shutdown_connections 2024-02-28T17:29:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.763+0000 7f1be8600640 1 --2- 172.21.15.1:0/1949449138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be0100110 0x7f1be0100560 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.763+0000 7f1be8600640 1 --2- 172.21.15.1:0/1949449138 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1be00fe9c0 0x7f1be00fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.763+0000 7f1be8600640 1 -- 172.21.15.1:0/1949449138 >> 172.21.15.1:0/1949449138 conn(0x7f1be00fa160 msgr2=0x7f1be00fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.763+0000 7f1be8600640 1 -- 172.21.15.1:0/1949449138 shutdown_connections 2024-02-28T17:29:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.763+0000 7f1be8600640 1 -- 172.21.15.1:0/1949449138 wait complete. 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.764+0000 7f1be8600640 1 Processor -- start 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.764+0000 7f1be8600640 1 -- start start 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.764+0000 7f1be8600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be00fe9c0 0x7f1be0114dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.764+0000 7f1be8600640 1 --2- >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1be0100110 0x7f1be0115300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.764+0000 7f1be8600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0115900 con 0x7f1be00fe9c0 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.764+0000 7f1be8600640 1 -- --> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be010ee90 con 0x7f1be0100110 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.765+0000 7f1be5c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be00fe9c0 0x7f1be0114dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.765+0000 7f1be5c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be00fe9c0 0x7f1be0114dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60856/0 (socket says 172.21.15.1:60856) 2024-02-28T17:29:43.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.765+0000 7f1be5c00640 1 -- 172.21.15.1:0/3146636007 learned_addr learned my addr 172.21.15.1:0/3146636007 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.765+0000 7f1be5200640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1be0100110 0x7f1be0115300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.765+0000 7f1be5c00640 1 -- 172.21.15.1:0/3146636007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1be0100110 msgr2=0x7f1be0115300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.765+0000 7f1be5c00640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1be0100110 0x7f1be0115300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.765+0000 7f1be5c00640 1 -- 172.21.15.1:0/3146636007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc005040 con 0x7f1be00fe9c0 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.765+0000 7f1be5200640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1be0100110 0x7f1be0115300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.766+0000 7f1be5c00640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be00fe9c0 0x7f1be0114dc0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400db50 tx=0x7f1bd4004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.766+0000 7f1bdd600640 1 -- 172.21.15.1:0/3146636007 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bd400b9d0 con 0x7f1be00fe9c0 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.766+0000 7f1bdd600640 1 -- 172.21.15.1:0/3146636007 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bd400f040 con 0x7f1be00fe9c0 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.766+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be010f170 con 0x7f1be00fe9c0 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.766+0000 7f1bdd600640 1 -- 172.21.15.1:0/3146636007 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bd40125d0 con 0x7f1be00fe9c0 2024-02-28T17:29:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.766+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be010f690 con 0x7f1be00fe9c0 2024-02-28T17:29:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.767+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be0064030 con 0x7f1be00fe9c0 2024-02-28T17:29:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.768+0000 7f1bdd600640 1 -- 172.21.15.1:0/3146636007 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 91483+0+0 (secure 0 0 0) 0x7f1bd4013430 con 0x7f1be00fe9c0 2024-02-28T17:29:43.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.769+0000 7f1bdd600640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1bc806d760 0x7f1bc806fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-28T17:29:43.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.769+0000 7f1bdd600640 1 -- 172.21.15.1:0/3146636007 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1bd408d360 con 0x7f1be00fe9c0 2024-02-28T17:29:43.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.770+0000 7f1be5200640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1bc806d760 0x7f1bc806fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-28T17:29:43.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.770+0000 7f1be5200640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1bc806d760 0x7f1bc806fc10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc02bda0 tx=0x7f1bcc009da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-28T17:29:43.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.774+0000 7f1bdd600640 1 -- 172.21.15.1:0/3146636007 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1bd4056c30 con 0x7f1be00fe9c0 2024-02-28T17:29:43.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.979+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1be01103c0 con 0x7f1be00fe9c0 2024-02-28T17:29:43.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.980+0000 7f1bdd600640 1 -- 172.21.15.1:0/3146636007 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 37 entries v84) v1 ==== 81+0+2156 (secure 0 0 0) 0x7f1bd405a2f0 con 0x7f1be00fe9c0 2024-02-28T17:29:43.981 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:6825/1481832428 2024-02-29T16:41:05.748852+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/701199259 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2061955559 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4223980442 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3865282714 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/833315191 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/298653290 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/3844925093 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:0/3927136860 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:0/2885505378 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:6828/2196519723 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2780166226 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/3844925093 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:43.982 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/179187285 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/3320028482 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/4056372531 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/701199259 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/3320028482 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1224289276 2024-02-29T16:18:15.275613+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/762266793 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2866890740 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/762266793 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/202779563 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:0/1412047282 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6826/1940162712 2024-02-29T16:41:05.748809+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1339161018 2024-02-29T16:20:26.347228+0000 2024-02-28T17:29:43.983 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4090914359 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:6829/2196519723 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/968413644 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:0/1854919128 2024-02-29T16:31:44.722257+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/4056372531 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4195777758 2024-02-29T16:18:55.593679+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1492016386 2024-02-29T16:29:38.122257+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.66:6824/1481832428 2024-02-29T16:41:05.748852+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2160247963 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1575378374 2024-02-29T16:35:40.611415+0000 2024-02-28T17:29:43.984 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6827/1940162712 2024-02-29T16:41:05.748809+0000 2024-02-28T17:29:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:listed 37 entries 2024-02-28T17:29:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.982+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1bc806d760 msgr2=0x7f1bc806fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.982+0000 7f1be8600640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1bc806d760 0x7f1bc806fc10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc02bda0 tx=0x7f1bcc009da0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.982+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be00fe9c0 msgr2=0x7f1be0114dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-28T17:29:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.982+0000 7f1be8600640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be00fe9c0 0x7f1be0114dc0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400db50 tx=0x7f1bd4004ee0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.983+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 shutdown_connections 2024-02-28T17:29:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.983+0000 7f1be8600640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:6800/3120269352,v1:172.21.15.1:6801/3120269352] conn(0x7f1bc806d760 0x7f1bc806fc10 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.983+0000 7f1be8600640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] conn(0x7f1be0100110 0x7f1be0115300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.983+0000 7f1be8600640 1 --2- 172.21.15.1:0/3146636007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1be00fe9c0 0x7f1be0114dc0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-28T17:29:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.983+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 >> 172.21.15.1:0/3146636007 conn(0x7f1be00fa160 msgr2=0x7f1be00fbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-28T17:29:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.983+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 shutdown_connections 2024-02-28T17:29:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-02-28T17:29:43.983+0000 7f1be8600640 1 -- 172.21.15.1:0/3146636007 wait complete. 2024-02-28T17:29:44.002 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-02-28T17:29:44.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-28T17:29:44.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-02-28T17:29:44.097 INFO:teuthology.orchestra.run:waiting for 300 2024-02-28T17:29:44.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:44 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/196469800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-28T17:29:44.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:44 smithi066 ceph-mon[85781]: from='client.? 172.21.15.1:0/3146636007' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-28T17:29:44.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:44 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/196469800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-28T17:29:44.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:44 smithi001 ceph-mon[112940]: from='client.? 172.21.15.1:0/3146636007' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-28T17:29:45.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:45 smithi001 ceph-mon[112940]: pgmap v1665: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:45.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:45 smithi066 ceph-mon[85781]: pgmap v1665: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:48.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:47 smithi066 ceph-mon[85781]: pgmap v1666: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:48.011 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:47 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:29:48.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:47 smithi001 ceph-mon[112940]: pgmap v1666: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:48.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:47 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:29:49.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:48 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:29:49.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:48 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:29:49.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:48 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:29:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:29:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:29:49.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:48 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:29:50.261 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:49 smithi066 ceph-mon[85781]: pgmap v1667: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:50.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:49 smithi001 ceph-mon[112940]: pgmap v1667: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:52.283 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:52 smithi001 ceph-mon[112940]: pgmap v1668: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:52.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:52 smithi066 ceph-mon[85781]: pgmap v1668: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:54.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:54 smithi066 ceph-mon[85781]: pgmap v1669: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:54.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:54 smithi001 ceph-mon[112940]: pgmap v1669: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:56 smithi066 ceph-mon[85781]: pgmap v1670: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:29:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:56 smithi001 ceph-mon[112940]: pgmap v1670: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:56.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:29:58.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:29:58 smithi066 ceph-mon[85781]: pgmap v1671: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:29:58.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:29:58 smithi001 ceph-mon[112940]: pgmap v1671: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:00.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:00 smithi066 ceph-mon[85781]: pgmap v1672: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:00.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:00 smithi066 ceph-mon[85781]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T17:30:00.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:00 smithi066 ceph-mon[85781]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:30:00.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:00 smithi066 ceph-mon[85781]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:30:00.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:00 smithi001 ceph-mon[112940]: pgmap v1672: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:00.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:00 smithi001 ceph-mon[112940]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T17:30:00.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:00 smithi001 ceph-mon[112940]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-28T17:30:00.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:00 smithi001 ceph-mon[112940]: fs cephfs has deprecated feature inline_data enabled. 2024-02-28T17:30:02.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:02 smithi066 ceph-mon[85781]: pgmap v1673: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:02.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:02 smithi001 ceph-mon[112940]: pgmap v1673: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:04.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:04 smithi066 ceph-mon[85781]: pgmap v1674: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:04.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:04 smithi001 ceph-mon[112940]: pgmap v1674: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:06.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:06 smithi066 ceph-mon[85781]: pgmap v1675: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:06.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:06 smithi001 ceph-mon[112940]: pgmap v1675: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:08.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:08 smithi066 ceph-mon[85781]: pgmap v1676: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:08.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:08 smithi001 ceph-mon[112940]: pgmap v1676: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:10.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:10 smithi066 ceph-mon[85781]: pgmap v1677: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:10.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:10 smithi001 ceph-mon[112940]: pgmap v1677: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:11.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:30:11.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:30:12.412 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:12 smithi066 ceph-mon[85781]: pgmap v1678: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:12.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:12 smithi001 ceph-mon[112940]: pgmap v1678: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:14.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:14 smithi066 ceph-mon[85781]: pgmap v1679: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:14.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:14 smithi001 ceph-mon[112940]: pgmap v1679: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:16.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:16 smithi066 ceph-mon[85781]: pgmap v1680: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:16.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:16 smithi001 ceph-mon[112940]: pgmap v1680: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:18.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:18 smithi066 ceph-mon[85781]: pgmap v1681: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:18.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:18 smithi001 ceph-mon[112940]: pgmap v1681: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:20.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:20 smithi066 ceph-mon[85781]: pgmap v1682: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:20.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:20 smithi001 ceph-mon[112940]: pgmap v1682: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:22.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:22 smithi066 ceph-mon[85781]: pgmap v1683: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:22.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:22 smithi001 ceph-mon[112940]: pgmap v1683: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:24.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:24 smithi066 ceph-mon[85781]: pgmap v1684: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:24.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:24 smithi001 ceph-mon[112940]: pgmap v1684: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:26.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:26 smithi066 ceph-mon[85781]: pgmap v1685: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:26.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:30:26.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:26 smithi001 ceph-mon[112940]: pgmap v1685: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:26.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:30:28.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:28 smithi066 ceph-mon[85781]: pgmap v1686: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:28.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:28 smithi001 ceph-mon[112940]: pgmap v1686: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:30.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:30 smithi066 ceph-mon[85781]: pgmap v1687: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:30.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:30 smithi001 ceph-mon[112940]: pgmap v1687: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:31.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:31 smithi066 ceph-mon[85781]: pgmap v1688: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:31.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:31 smithi001 ceph-mon[112940]: pgmap v1688: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:34.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:34 smithi066 ceph-mon[85781]: pgmap v1689: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:34.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:34 smithi001 ceph-mon[112940]: pgmap v1689: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:36.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:36 smithi066 ceph-mon[85781]: pgmap v1690: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:36.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:36 smithi001 ceph-mon[112940]: pgmap v1690: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:38.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:38 smithi066 ceph-mon[85781]: pgmap v1691: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:38.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:38 smithi001 ceph-mon[112940]: pgmap v1691: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:40.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:40 smithi066 ceph-mon[85781]: pgmap v1692: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:40 smithi001 ceph-mon[112940]: pgmap v1692: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:41.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:30:41.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:30:42.412 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:42 smithi066 ceph-mon[85781]: pgmap v1693: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:42.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:42 smithi001 ceph-mon[112940]: pgmap v1693: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:44.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:44 smithi066 ceph-mon[85781]: pgmap v1694: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:44.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:44 smithi001 ceph-mon[112940]: pgmap v1694: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:46.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:46 smithi066 ceph-mon[85781]: pgmap v1695: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:46.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:46 smithi001 ceph-mon[112940]: pgmap v1695: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:48.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:48 smithi066 ceph-mon[85781]: pgmap v1696: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:48.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:48 smithi001 ceph-mon[112940]: pgmap v1696: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:50.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:50 smithi066 ceph-mon[85781]: pgmap v1697: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:50.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:50 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:30:50.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:50 smithi001 ceph-mon[112940]: pgmap v1697: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:50.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:50 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:30:51.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:51 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:30:51.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:51 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:30:51.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:51 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:30:51.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:51 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:30:51.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:51 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:30:51.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:51 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:30:52.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:52 smithi066 ceph-mon[85781]: pgmap v1698: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:52.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:52 smithi001 ceph-mon[112940]: pgmap v1698: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:54.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:54 smithi066 ceph-mon[85781]: pgmap v1699: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:54.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:54 smithi001 ceph-mon[112940]: pgmap v1699: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:56 smithi066 ceph-mon[85781]: pgmap v1700: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:30:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:56 smithi001 ceph-mon[112940]: pgmap v1700: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:56.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:30:58.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:30:58 smithi066 ceph-mon[85781]: pgmap v1701: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:30:58.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:30:58 smithi001 ceph-mon[112940]: pgmap v1701: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:00.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:00 smithi066 ceph-mon[85781]: pgmap v1702: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:00.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:00 smithi001 ceph-mon[112940]: pgmap v1702: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:02.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:02 smithi066 ceph-mon[85781]: pgmap v1703: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:02.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:02 smithi001 ceph-mon[112940]: pgmap v1703: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:04.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:04 smithi066 ceph-mon[85781]: pgmap v1704: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:04.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:04 smithi001 ceph-mon[112940]: pgmap v1704: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:06.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:06 smithi066 ceph-mon[85781]: pgmap v1705: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:06.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:06 smithi001 ceph-mon[112940]: pgmap v1705: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:08.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:08 smithi066 ceph-mon[85781]: pgmap v1706: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:08.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:08 smithi001 ceph-mon[112940]: pgmap v1706: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:10.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:10 smithi066 ceph-mon[85781]: pgmap v1707: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:10.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:10 smithi001 ceph-mon[112940]: pgmap v1707: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:11.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:31:11.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:11 smithi066 ceph-mon[85781]: pgmap v1708: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:11.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:31:11.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:11 smithi001 ceph-mon[112940]: pgmap v1708: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:14.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:14 smithi066 ceph-mon[85781]: pgmap v1709: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:14.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:14 smithi001 ceph-mon[112940]: pgmap v1709: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:16.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:16 smithi066 ceph-mon[85781]: pgmap v1710: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:16.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:16 smithi001 ceph-mon[112940]: pgmap v1710: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:18.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:18 smithi066 ceph-mon[85781]: pgmap v1711: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:18.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:18 smithi001 ceph-mon[112940]: pgmap v1711: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:20.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:20 smithi066 ceph-mon[85781]: pgmap v1712: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:20.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:20 smithi001 ceph-mon[112940]: pgmap v1712: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:22.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:22 smithi066 ceph-mon[85781]: pgmap v1713: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:22.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:22 smithi001 ceph-mon[112940]: pgmap v1713: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:24.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:24 smithi066 ceph-mon[85781]: pgmap v1714: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:24.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:24 smithi001 ceph-mon[112940]: pgmap v1714: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:26.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:26 smithi066 ceph-mon[85781]: pgmap v1715: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:26.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:31:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:26 smithi001 ceph-mon[112940]: pgmap v1715: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:26.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:31:28.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:28 smithi066 ceph-mon[85781]: pgmap v1716: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:28.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:28 smithi001 ceph-mon[112940]: pgmap v1716: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:30.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:30 smithi066 ceph-mon[85781]: pgmap v1717: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:30.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:30 smithi001 ceph-mon[112940]: pgmap v1717: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:32.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:32 smithi066 ceph-mon[85781]: pgmap v1718: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:32.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:32 smithi001 ceph-mon[112940]: pgmap v1718: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:34.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:34 smithi066 ceph-mon[85781]: pgmap v1719: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:34.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:34 smithi001 ceph-mon[112940]: pgmap v1719: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:36.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:36 smithi066 ceph-mon[85781]: pgmap v1720: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:36.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:36 smithi001 ceph-mon[112940]: pgmap v1720: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:38 smithi066 ceph-mon[85781]: pgmap v1721: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:38.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:38 smithi001 ceph-mon[112940]: pgmap v1721: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:40.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:40 smithi066 ceph-mon[85781]: pgmap v1722: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:40 smithi001 ceph-mon[112940]: pgmap v1722: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:41.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:31:41.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:31:42.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:42 smithi066 ceph-mon[85781]: pgmap v1723: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:42.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:42 smithi001 ceph-mon[112940]: pgmap v1723: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:44.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:44 smithi066 ceph-mon[85781]: pgmap v1724: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:44.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:44 smithi001 ceph-mon[112940]: pgmap v1724: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:46.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:46 smithi066 ceph-mon[85781]: pgmap v1725: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:46.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:46 smithi001 ceph-mon[112940]: pgmap v1725: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:47.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:47 smithi001 ceph-mon[112940]: pgmap v1726: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:47.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:47 smithi066 ceph-mon[85781]: pgmap v1726: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:50.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:50 smithi066 ceph-mon[85781]: pgmap v1727: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:50.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:50 smithi001 ceph-mon[112940]: pgmap v1727: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:51.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:51 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:31:51.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:51 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:31:52.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:52 smithi066 ceph-mon[85781]: pgmap v1728: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:52.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:52 smithi001 ceph-mon[112940]: pgmap v1728: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:53.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:53 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:31:53.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:53 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:31:53.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:53 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:31:53.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:53 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:31:53.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:53 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:31:53.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:53 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:31:54.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:54 smithi066 ceph-mon[85781]: pgmap v1729: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:54.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:54 smithi001 ceph-mon[112940]: pgmap v1729: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:56.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:56 smithi066 ceph-mon[85781]: pgmap v1730: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:31:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:56 smithi001 ceph-mon[112940]: pgmap v1730: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:31:58.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:31:58 smithi066 ceph-mon[85781]: pgmap v1731: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:31:58.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:31:58 smithi001 ceph-mon[112940]: pgmap v1731: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:00.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:00 smithi066 ceph-mon[85781]: pgmap v1732: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:00.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:00 smithi001 ceph-mon[112940]: pgmap v1732: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:02.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:02 smithi066 ceph-mon[85781]: pgmap v1733: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:02.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:02 smithi001 ceph-mon[112940]: pgmap v1733: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:04.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:04 smithi066 ceph-mon[85781]: pgmap v1734: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:04.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:04 smithi001 ceph-mon[112940]: pgmap v1734: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:06.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:06 smithi066 ceph-mon[85781]: pgmap v1735: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:06.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:06 smithi001 ceph-mon[112940]: pgmap v1735: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:08.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:08 smithi066 ceph-mon[85781]: pgmap v1736: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:08.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:08 smithi001 ceph-mon[112940]: pgmap v1736: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:10.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:10 smithi066 ceph-mon[85781]: pgmap v1737: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:10.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:10 smithi001 ceph-mon[112940]: pgmap v1737: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:11.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:32:11.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:32:12.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:12 smithi066 ceph-mon[85781]: pgmap v1738: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:12.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:12 smithi001 ceph-mon[112940]: pgmap v1738: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:14.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:14 smithi066 ceph-mon[85781]: pgmap v1739: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:14.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:14 smithi001 ceph-mon[112940]: pgmap v1739: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:16.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:16 smithi066 ceph-mon[85781]: pgmap v1740: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:16.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:16 smithi001 ceph-mon[112940]: pgmap v1740: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:18.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:18 smithi066 ceph-mon[85781]: pgmap v1741: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:18.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:18 smithi001 ceph-mon[112940]: pgmap v1741: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:19.529 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:19 smithi001 ceph-mon[112940]: pgmap v1742: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:19.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:19 smithi066 ceph-mon[85781]: pgmap v1742: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:22.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:22 smithi066 ceph-mon[85781]: pgmap v1743: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:22.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:22 smithi001 ceph-mon[112940]: pgmap v1743: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:24.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:24 smithi066 ceph-mon[85781]: pgmap v1744: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:24.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:24 smithi001 ceph-mon[112940]: pgmap v1744: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:26.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:26 smithi066 ceph-mon[85781]: pgmap v1745: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:26.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:32:26.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:26 smithi001 ceph-mon[112940]: pgmap v1745: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:26.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:32:28.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:28 smithi066 ceph-mon[85781]: pgmap v1746: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:28.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:28 smithi001 ceph-mon[112940]: pgmap v1746: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:30.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:30 smithi066 ceph-mon[85781]: pgmap v1747: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:30.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:30 smithi001 ceph-mon[112940]: pgmap v1747: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:32.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:32 smithi066 ceph-mon[85781]: pgmap v1748: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:32.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:32 smithi001 ceph-mon[112940]: pgmap v1748: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:34.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:34 smithi066 ceph-mon[85781]: pgmap v1749: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:34.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:34 smithi001 ceph-mon[112940]: pgmap v1749: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:36.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:36 smithi066 ceph-mon[85781]: pgmap v1750: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:36.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:36 smithi001 ceph-mon[112940]: pgmap v1750: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:38 smithi066 ceph-mon[85781]: pgmap v1751: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:38.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:38 smithi001 ceph-mon[112940]: pgmap v1751: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:40.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:40 smithi066 ceph-mon[85781]: pgmap v1752: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:40 smithi001 ceph-mon[112940]: pgmap v1752: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:41.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:32:41.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:32:42.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:42 smithi066 ceph-mon[85781]: pgmap v1753: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:42.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:42 smithi001 ceph-mon[112940]: pgmap v1753: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:44.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:44 smithi066 ceph-mon[85781]: pgmap v1754: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:44.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:44 smithi001 ceph-mon[112940]: pgmap v1754: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:46.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:46 smithi066 ceph-mon[85781]: pgmap v1755: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:46.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:46 smithi001 ceph-mon[112940]: pgmap v1755: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:48.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:47 smithi066 ceph-mon[85781]: pgmap v1756: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:48.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:47 smithi001 ceph-mon[112940]: pgmap v1756: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:50.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:50 smithi066 ceph-mon[85781]: pgmap v1757: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:50.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:50 smithi001 ceph-mon[112940]: pgmap v1757: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:52.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:52 smithi066 ceph-mon[85781]: pgmap v1758: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:52.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:52 smithi001 ceph-mon[112940]: pgmap v1758: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:53.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:53 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:32:53.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:53 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:32:54.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:54 smithi066 ceph-mon[85781]: pgmap v1759: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:54.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:54 smithi001 ceph-mon[112940]: pgmap v1759: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:55.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:32:55.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:32:55.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:32:55.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:32:55.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:32:55.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:32:56.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:56 smithi066 ceph-mon[85781]: pgmap v1760: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:32:56.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:56 smithi001 ceph-mon[112940]: pgmap v1760: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:56.533 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:32:58.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:32:58 smithi066 ceph-mon[85781]: pgmap v1761: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:32:58.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:32:58 smithi001 ceph-mon[112940]: pgmap v1761: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:00.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:00 smithi066 ceph-mon[85781]: pgmap v1762: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:00.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:00 smithi001 ceph-mon[112940]: pgmap v1762: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:02.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:02 smithi066 ceph-mon[85781]: pgmap v1763: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:02.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:02 smithi001 ceph-mon[112940]: pgmap v1763: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:04.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:04 smithi066 ceph-mon[85781]: pgmap v1764: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:04.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:04 smithi001 ceph-mon[112940]: pgmap v1764: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:06.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:06 smithi066 ceph-mon[85781]: pgmap v1765: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:06.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:06 smithi001 ceph-mon[112940]: pgmap v1765: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:08.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:08 smithi066 ceph-mon[85781]: pgmap v1766: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:08.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:08 smithi001 ceph-mon[112940]: pgmap v1766: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:10.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:10 smithi066 ceph-mon[85781]: pgmap v1767: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:10.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:10 smithi001 ceph-mon[112940]: pgmap v1767: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:11.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:33:11.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:33:12.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:12 smithi066 ceph-mon[85781]: pgmap v1768: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:12.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:12 smithi001 ceph-mon[112940]: pgmap v1768: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:14.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:14 smithi066 ceph-mon[85781]: pgmap v1769: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:14.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:14 smithi001 ceph-mon[112940]: pgmap v1769: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:15.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:15 smithi001 ceph-mon[112940]: pgmap v1770: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:15.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:15 smithi066 ceph-mon[85781]: pgmap v1770: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:18.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:18 smithi066 ceph-mon[85781]: pgmap v1771: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:18.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:18 smithi001 ceph-mon[112940]: pgmap v1771: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:20.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:20 smithi066 ceph-mon[85781]: pgmap v1772: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:20.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:20 smithi001 ceph-mon[112940]: pgmap v1772: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:22.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:22 smithi066 ceph-mon[85781]: pgmap v1773: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:22.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:22 smithi001 ceph-mon[112940]: pgmap v1773: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:24.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:24 smithi066 ceph-mon[85781]: pgmap v1774: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:24.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:24 smithi001 ceph-mon[112940]: pgmap v1774: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:26.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:26 smithi066 ceph-mon[85781]: pgmap v1775: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:26.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:33:26.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:26 smithi001 ceph-mon[112940]: pgmap v1775: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:26.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:33:28.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:28 smithi066 ceph-mon[85781]: pgmap v1776: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:28.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:28 smithi001 ceph-mon[112940]: pgmap v1776: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:30.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:30 smithi066 ceph-mon[85781]: pgmap v1777: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:30.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:30 smithi001 ceph-mon[112940]: pgmap v1777: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:32.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:32 smithi066 ceph-mon[85781]: pgmap v1778: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:32.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:32 smithi001 ceph-mon[112940]: pgmap v1778: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:34.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:34 smithi066 ceph-mon[85781]: pgmap v1779: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:34.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:34 smithi001 ceph-mon[112940]: pgmap v1779: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:36.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:36 smithi066 ceph-mon[85781]: pgmap v1780: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:36.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:36 smithi001 ceph-mon[112940]: pgmap v1780: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:38 smithi066 ceph-mon[85781]: pgmap v1781: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:38.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:38 smithi001 ceph-mon[112940]: pgmap v1781: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:40.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:40 smithi066 ceph-mon[85781]: pgmap v1782: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:40 smithi001 ceph-mon[112940]: pgmap v1782: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:41.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:33:41.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:33:42.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:42 smithi066 ceph-mon[85781]: pgmap v1783: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:42.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:42 smithi001 ceph-mon[112940]: pgmap v1783: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:43.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:43 smithi066 ceph-mon[85781]: pgmap v1784: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:43.782 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:43 smithi001 ceph-mon[112940]: pgmap v1784: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:46.464 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:46 smithi066 ceph-mon[85781]: pgmap v1785: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:46.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:46 smithi001 ceph-mon[112940]: pgmap v1785: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:48.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:47 smithi001 ceph-mon[112940]: pgmap v1786: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:48.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:47 smithi066 ceph-mon[85781]: pgmap v1786: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:50.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:50 smithi066 ceph-mon[85781]: pgmap v1787: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:50.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:50 smithi001 ceph-mon[112940]: pgmap v1787: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:52.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:52 smithi066 ceph-mon[85781]: pgmap v1788: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:52.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:52 smithi001 ceph-mon[112940]: pgmap v1788: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:54.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:54 smithi066 ceph-mon[85781]: pgmap v1789: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:54.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:54 smithi001 ceph-mon[112940]: pgmap v1789: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:55.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:55 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:33:55.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:55 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-28T17:33:56.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:56 smithi066 ceph-mon[85781]: pgmap v1790: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:56.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:33:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:33:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:33:56.511 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:56 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:33:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:56 smithi001 ceph-mon[112940]: pgmap v1790: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:33:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-28T17:33:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-28T17:33:56.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:56 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' 2024-02-28T17:33:58.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:33:58 smithi066 ceph-mon[85781]: pgmap v1791: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:33:58.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:33:58 smithi001 ceph-mon[112940]: pgmap v1791: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:00.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:00 smithi066 ceph-mon[85781]: pgmap v1792: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:00.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:00 smithi001 ceph-mon[112940]: pgmap v1792: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:02.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:02 smithi066 ceph-mon[85781]: pgmap v1793: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:02.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:02 smithi001 ceph-mon[112940]: pgmap v1793: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:04.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:04 smithi066 ceph-mon[85781]: pgmap v1794: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:04.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:04 smithi001 ceph-mon[112940]: pgmap v1794: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:06.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:06 smithi066 ceph-mon[85781]: pgmap v1795: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:06.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:06 smithi001 ceph-mon[112940]: pgmap v1795: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:08.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:08 smithi066 ceph-mon[85781]: pgmap v1796: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:08.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:08 smithi001 ceph-mon[112940]: pgmap v1796: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:10.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:10 smithi066 ceph-mon[85781]: pgmap v1797: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:10.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:10 smithi001 ceph-mon[112940]: pgmap v1797: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:11.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:11 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:34:11.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:11 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:34:12.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:12 smithi066 ceph-mon[85781]: pgmap v1798: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:12.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:12 smithi001 ceph-mon[112940]: pgmap v1798: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:13.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:13 smithi066 ceph-mon[85781]: pgmap v1799: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:13.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:13 smithi001 ceph-mon[112940]: pgmap v1799: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:16.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:16 smithi066 ceph-mon[85781]: pgmap v1800: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:16.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:16 smithi001 ceph-mon[112940]: pgmap v1800: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:18.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:18 smithi066 ceph-mon[85781]: pgmap v1801: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:18.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:18 smithi001 ceph-mon[112940]: pgmap v1801: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:20.512 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:20 smithi066 ceph-mon[85781]: pgmap v1802: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:20.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:20 smithi001 ceph-mon[112940]: pgmap v1802: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:22.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:22 smithi066 ceph-mon[85781]: pgmap v1803: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:22.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:22 smithi001 ceph-mon[112940]: pgmap v1803: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:24.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:24 smithi066 ceph-mon[85781]: pgmap v1804: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:24.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:24 smithi001 ceph-mon[112940]: pgmap v1804: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:26.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:26 smithi066 ceph-mon[85781]: pgmap v1805: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:26.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:26 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:34:26.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:26 smithi001 ceph-mon[112940]: pgmap v1805: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:26.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:26 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:34:28.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:28 smithi066 ceph-mon[85781]: pgmap v1806: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:28.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:28 smithi001 ceph-mon[112940]: pgmap v1806: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:30.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:30 smithi066 ceph-mon[85781]: pgmap v1807: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:30.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:30 smithi001 ceph-mon[112940]: pgmap v1807: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:32.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:32 smithi066 ceph-mon[85781]: pgmap v1808: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:32.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:32 smithi001 ceph-mon[112940]: pgmap v1808: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:34.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:34 smithi066 ceph-mon[85781]: pgmap v1809: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:34.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:34 smithi001 ceph-mon[112940]: pgmap v1809: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:36.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:36 smithi066 ceph-mon[85781]: pgmap v1810: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:36.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:36 smithi001 ceph-mon[112940]: pgmap v1810: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:38.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:38 smithi066 ceph-mon[85781]: pgmap v1811: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:38.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:38 smithi001 ceph-mon[112940]: pgmap v1811: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:40.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:40 smithi066 ceph-mon[85781]: pgmap v1812: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:40.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:40 smithi001 ceph-mon[112940]: pgmap v1812: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:41.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:41 smithi066 ceph-mon[85781]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:34:41.510 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:41 smithi066 ceph-mon[85781]: pgmap v1813: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:41.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:41 smithi001 ceph-mon[112940]: from='mgr.34102 172.21.15.1:0/372766' entity='mgr.smithi001.rmthtx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-28T17:34:41.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:41 smithi001 ceph-mon[112940]: pgmap v1813: 65 pgs: 65 active+clean; 324 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-02-28T17:34:43.168 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-02-28T17:34:43.168 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-28T17:34:43.170 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-28T17:34:43.185 INFO:tasks.cephadm:Teardown begin 2024-02-28T17:34:43.185 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-28T17:34:43.187 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-28T17:34:43.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-28T17:34:43.417 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-28T17:34:43.417 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-28T17:34:43.477 DEBUG:teuthology.orchestra.run.smithi066:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-28T17:34:43.538 INFO:tasks.cephadm:Stopping all daemons... 2024-02-28T17:34:43.539 INFO:tasks.cephadm.mon.smithi001:Stopping mon.smithi001... 2024-02-28T17:34:43.539 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001 2024-02-28T17:34:44.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:43 smithi001 systemd[1]: Stopping Ceph mon.smithi001 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T17:34:44.418 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:44 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T17:34:44.138+0000 7f7e44600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi001 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T17:34:44.418 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:44 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001[112936]: 2024-02-28T17:34:44.138+0000 7f7e44600700 -1 mon.smithi001@0(leader) e3 *** Got Signal Terminated *** 2024-02-28T17:34:44.714 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:44 smithi001 podman[175917]: 2024-02-28 17:34:44.417493438 +0000 UTC m=+0.409016378 container died d04080cee273b377ffa020a7a6d8cb90785749c8b688b39b513c71cf1becd2a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, io.buildah.version=1.33.5, RELEASE=main-1fb5d10, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-28T17:34:45.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:44 smithi001 podman[175917]: 2024-02-28 17:34:44.928716086 +0000 UTC m=+0.920239027 container cleanup d04080cee273b377ffa020a7a6d8cb90785749c8b688b39b513c71cf1becd2a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, org.label-schema.build-date=20240222) 2024-02-28T17:34:45.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:44 smithi001 bash[175917]: ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001 2024-02-28T17:34:45.532 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Feb 28 17:34:45 smithi001 podman[175942]: 2024-02-28 17:34:45.15497654 +0000 UTC m=+0.720281979 container remove d04080cee273b377ffa020a7a6d8cb90785749c8b688b39b513c71cf1becd2a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi001, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=main-1fb5d10, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-28T17:34:45.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi001.service' 2024-02-28T17:34:45.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-28T17:34:45.796 INFO:tasks.cephadm.mon.smithi001:Stopped mon.smithi001 2024-02-28T17:34:45.796 INFO:tasks.cephadm.mon.smithi066:Stopping mon.smithi066... 2024-02-28T17:34:45.796 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi066 2024-02-28T17:34:46.260 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:45 smithi066 systemd[1]: Stopping Ceph mon.smithi066 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T17:34:46.706 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:46 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066[85777]: 2024-02-28T17:34:46.415+0000 7fd08c400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi066 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T17:34:46.706 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:46 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066[85777]: 2024-02-28T17:34:46.415+0000 7fd08c400700 -1 mon.smithi066@1(peon) e3 *** Got Signal Terminated *** 2024-02-28T17:34:47.010 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:46 smithi066 podman[122301]: 2024-02-28 17:34:46.705607227 +0000 UTC m=+0.408352149 container died d17ce055575ed9becef55c650eb004794d1ea9bccaf9614b63cd819af5be0782 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-28T17:34:47.469 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:47 smithi066 podman[122301]: 2024-02-28 17:34:47.250474447 +0000 UTC m=+0.953219386 container cleanup d17ce055575ed9becef55c650eb004794d1ea9bccaf9614b63cd819af5be0782 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-02-28T17:34:47.469 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:47 smithi066 bash[122301]: ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066 2024-02-28T17:34:47.760 INFO:journalctl@ceph.mon.smithi066.smithi066.stdout:Feb 28 17:34:47 smithi066 podman[122316]: 2024-02-28 17:34:47.46854522 +0000 UTC m=+0.738671743 container remove d17ce055575ed9becef55c650eb004794d1ea9bccaf9614b63cd819af5be0782 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-mon-smithi066, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-02-28T17:34:48.034 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@mon.smithi066.service' 2024-02-28T17:34:48.191 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-28T17:34:48.192 INFO:tasks.cephadm.mon.smithi066:Stopped mon.smithi066 2024-02-28T17:34:48.192 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-28T17:34:48.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.0 2024-02-28T17:34:48.760 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:48 smithi001 systemd[1]: Stopping Ceph osd.0 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T17:34:48.760 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:48 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[124605]: 2024-02-28T17:34:48.756+0000 7fa4bfe00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T17:34:48.760 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:48 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[124605]: 2024-02-28T17:34:48.756+0000 7fa4bfe00700 -1 osd.0 84 *** Got signal Terminated *** 2024-02-28T17:34:48.761 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:48 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0[124605]: 2024-02-28T17:34:48.756+0000 7fa4bfe00700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T17:34:54.533 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:54 smithi001 podman[176061]: 2024-02-28 17:34:54.230914823 +0000 UTC m=+5.619507823 container died 7cea5750b27a0a8d645276d26dbd51de3df9e4f3adce54760fa5d5553508eace (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-02-28T17:34:55.032 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:54 smithi001 podman[176061]: 2024-02-28 17:34:54.766833677 +0000 UTC m=+6.155426722 container cleanup 7cea5750b27a0a8d645276d26dbd51de3df9e4f3adce54760fa5d5553508eace (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-28T17:34:55.032 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:54 smithi001 bash[176061]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0 2024-02-28T17:34:55.532 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:55 smithi001 podman[176078]: 2024-02-28 17:34:55.034928513 +0000 UTC m=+0.790268082 container remove 7cea5750b27a0a8d645276d26dbd51de3df9e4f3adce54760fa5d5553508eace (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=) 2024-02-28T17:34:56.233 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:55 smithi001 podman[176168]: 2024-02-28 17:34:55.80713116 +0000 UTC m=+0.035390905 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T17:34:56.233 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:55 smithi001 podman[176168]: 2024-02-28 17:34:55.952620712 +0000 UTC m=+0.180880560 container create b738e15c4d8fc8e830cc022ea59ba88508d1d26cfb017e25734f47a20b093846 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T17:34:56.234 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:56 smithi001 podman[176168]: 2024-02-28 17:34:56.23188744 +0000 UTC m=+0.460147215 container init b738e15c4d8fc8e830cc022ea59ba88508d1d26cfb017e25734f47a20b093846 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-02-28T17:34:56.532 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:56 smithi001 podman[176168]: 2024-02-28 17:34:56.245129589 +0000 UTC m=+0.473389327 container start b738e15c4d8fc8e830cc022ea59ba88508d1d26cfb017e25734f47a20b093846 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-02-28T17:34:56.532 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:56 smithi001 podman[176168]: 2024-02-28 17:34:56.330045632 +0000 UTC m=+0.558305374 container attach b738e15c4d8fc8e830cc022ea59ba88508d1d26cfb017e25734f47a20b093846 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-28T17:34:57.014 INFO:journalctl@ceph.osd.0.smithi001.stdout:Feb 28 17:34:56 smithi001 podman[176168]: 2024-02-28 17:34:56.694340092 +0000 UTC m=+0.922599903 container died b738e15c4d8fc8e830cc022ea59ba88508d1d26cfb017e25734f47a20b093846 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T17:34:57.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.0.service' 2024-02-28T17:34:57.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-28T17:34:57.531 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-28T17:34:57.531 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-28T17:34:57.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.1 2024-02-28T17:34:57.888 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:34:57 smithi001 systemd[1]: Stopping Ceph osd.1 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T17:34:58.282 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:34:58 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[130116]: 2024-02-28T17:34:58.093+0000 7f33f7600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T17:34:58.283 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:34:58 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[130116]: 2024-02-28T17:34:58.093+0000 7f33f7600700 -1 osd.1 84 *** Got signal Terminated *** 2024-02-28T17:34:58.283 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:34:58 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1[130116]: 2024-02-28T17:34:58.093+0000 7f33f7600700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T17:35:03.961 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:03 smithi001 podman[176274]: 2024-02-28 17:35:03.652065603 +0000 UTC m=+5.702172207 container stop 2ff850335691a3c4d4b4fe8af676c24de1d16801882275888a27ed8f5f7f5abc (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0) 2024-02-28T17:35:03.961 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:03 smithi001 podman[176274]: 2024-02-28 17:35:03.678499088 +0000 UTC m=+5.728605672 container died 2ff850335691a3c4d4b4fe8af676c24de1d16801882275888a27ed8f5f7f5abc (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-02-28T17:35:04.282 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:04 smithi001 podman[176274]: 2024-02-28 17:35:04.096155288 +0000 UTC m=+6.146261918 container cleanup 2ff850335691a3c4d4b4fe8af676c24de1d16801882275888a27ed8f5f7f5abc (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-02-28T17:35:04.283 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:04 smithi001 bash[176274]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1 2024-02-28T17:35:04.782 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:04 smithi001 podman[176296]: 2024-02-28 17:35:04.472272146 +0000 UTC m=+0.798842103 container remove 2ff850335691a3c4d4b4fe8af676c24de1d16801882275888a27ed8f5f7f5abc (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-28T17:35:05.532 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:05 smithi001 podman[176381]: 2024-02-28 17:35:05.188257077 +0000 UTC m=+0.038892633 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T17:35:05.820 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:05 smithi001 podman[176381]: 2024-02-28 17:35:05.539841569 +0000 UTC m=+0.390477156 container create cfd9d152f35b57b913e7cacc5093dd13ae4b68f2ed55a81eeca4ca3fe20d16ca (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=main-1fb5d10, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-28T17:35:05.820 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:05 smithi001 podman[176381]: 2024-02-28 17:35:05.818957962 +0000 UTC m=+0.669593556 container init cfd9d152f35b57b913e7cacc5093dd13ae4b68f2ed55a81eeca4ca3fe20d16ca (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, org.label-schema.license=GPLv2) 2024-02-28T17:35:06.281 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:05 smithi001 podman[176381]: 2024-02-28 17:35:05.8307069 +0000 UTC m=+0.681342479 container start cfd9d152f35b57b913e7cacc5093dd13ae4b68f2ed55a81eeca4ca3fe20d16ca (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-1fb5d10, GIT_CLEAN=True, ceph=True) 2024-02-28T17:35:06.281 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:05 smithi001 podman[176381]: 2024-02-28 17:35:05.909242974 +0000 UTC m=+0.759878601 container attach cfd9d152f35b57b913e7cacc5093dd13ae4b68f2ed55a81eeca4ca3fe20d16ca (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T17:35:06.532 INFO:journalctl@ceph.osd.1.smithi001.stdout:Feb 28 17:35:06 smithi001 podman[176381]: 2024-02-28 17:35:06.280344154 +0000 UTC m=+1.130979781 container died cfd9d152f35b57b913e7cacc5093dd13ae4b68f2ed55a81eeca4ca3fe20d16ca (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-28T17:35:07.075 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.1.service' 2024-02-28T17:35:07.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-28T17:35:07.201 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-28T17:35:07.201 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-28T17:35:07.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.2 2024-02-28T17:35:07.783 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:07 smithi001 systemd[1]: Stopping Ceph osd.2 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T17:35:08.282 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:07 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[135641]: 2024-02-28T17:35:07.856+0000 7fb3f4400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T17:35:08.283 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:07 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[135641]: 2024-02-28T17:35:07.856+0000 7fb3f4400700 -1 osd.2 84 *** Got signal Terminated *** 2024-02-28T17:35:08.283 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:07 smithi001 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2[135641]: 2024-02-28T17:35:07.856+0000 7fb3f4400700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T17:35:13.525 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:13 smithi001 podman[176485]: 2024-02-28 17:35:13.251470284 +0000 UTC m=+5.537704124 container died 4cd5368c3ac635a883f4dbd0194e698e6513feb56723585c140a03aaa40e802b (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-28T17:35:13.783 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:13 smithi001 podman[176485]: 2024-02-28 17:35:13.632724983 +0000 UTC m=+5.918958852 container cleanup 4cd5368c3ac635a883f4dbd0194e698e6513feb56723585c140a03aaa40e802b (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T17:35:13.866 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:13 smithi001 bash[176485]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2 2024-02-28T17:35:14.282 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:13 smithi001 podman[176505]: 2024-02-28 17:35:13.842143671 +0000 UTC m=+0.576948389 container remove 4cd5368c3ac635a883f4dbd0194e698e6513feb56723585c140a03aaa40e802b (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T17:35:15.033 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:14 smithi001 podman[176593]: 2024-02-28 17:35:14.668130063 +0000 UTC m=+0.041098603 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T17:35:15.033 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:14 smithi001 podman[176593]: 2024-02-28 17:35:14.768059323 +0000 UTC m=+0.141027898 container create ecf78d4e00807f546099df7ee2718880d1d2434e1b66f8d070af47b96ae99c70 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T17:35:15.496 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:15 smithi001 podman[176593]: 2024-02-28 17:35:15.041175265 +0000 UTC m=+0.414143842 container init ecf78d4e00807f546099df7ee2718880d1d2434e1b66f8d070af47b96ae99c70 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-02-28T17:35:15.496 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:15 smithi001 podman[176593]: 2024-02-28 17:35:15.052706476 +0000 UTC m=+0.425675058 container start ecf78d4e00807f546099df7ee2718880d1d2434e1b66f8d070af47b96ae99c70 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-02-28T17:35:15.496 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:15 smithi001 podman[176593]: 2024-02-28 17:35:15.128863055 +0000 UTC m=+0.501831670 container attach ecf78d4e00807f546099df7ee2718880d1d2434e1b66f8d070af47b96ae99c70 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-28T17:35:15.782 INFO:journalctl@ceph.osd.2.smithi001.stdout:Feb 28 17:35:15 smithi001 podman[176593]: 2024-02-28 17:35:15.496112302 +0000 UTC m=+0.869080884 container died ecf78d4e00807f546099df7ee2718880d1d2434e1b66f8d070af47b96ae99c70 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-1fb5d10) 2024-02-28T17:35:16.010 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:15 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:15.552+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:16.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.2.service' 2024-02-28T17:35:16.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-28T17:35:16.318 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-28T17:35:16.318 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-28T17:35:16.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.3 2024-02-28T17:35:16.699 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:16 smithi066 systemd[1]: Stopping Ceph osd.3 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T17:35:16.700 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:16 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:16.511+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:17.010 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:16 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T17:35:16.875+0000 7f1261200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T17:35:17.011 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:16 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T17:35:16.875+0000 7f1261200700 -1 osd.3 84 *** Got signal Terminated *** 2024-02-28T17:35:17.011 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:16 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T17:35:16.875+0000 7f1261200700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T17:35:17.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:17 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:17.486+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:18.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:18 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:18.482+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:19.478 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:19 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T17:35:19.404+0000 7f1258400700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.585725+0000 front 2024-02-28T17:34:53.586295+0000 (oldest deadline 2024-02-28T17:35:18.885833+0000) 2024-02-28T17:35:19.478 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:19 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:19.203+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:19.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:19 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:19.476+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:20.510 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T17:35:20.363+0000 7f1258400700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.585725+0000 front 2024-02-28T17:34:53.586295+0000 (oldest deadline 2024-02-28T17:35:18.885833+0000) 2024-02-28T17:35:20.511 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:20.217+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:20.511 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:20 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:20.461+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:21.483 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:21 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3[95613]: 2024-02-28T17:35:21.381+0000 7f1258400700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.585725+0000 front 2024-02-28T17:34:53.586295+0000 (oldest deadline 2024-02-28T17:35:18.885833+0000) 2024-02-28T17:35:21.483 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:21 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:21.202+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:21.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:21 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:21.481+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:22.510 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:22 smithi066 podman[122431]: 2024-02-28 17:35:22.344159056 +0000 UTC m=+5.622203711 container died 4953ba0214066081e61c1b229adf3c4ce493878ad1d546f502fe9c8fbad7a3b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T17:35:22.511 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:22 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:22.253+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:22.511 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:22 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:22.499+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:22.947 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:22 smithi066 podman[122431]: 2024-02-28 17:35:22.728202043 +0000 UTC m=+6.006246738 container cleanup 4953ba0214066081e61c1b229adf3c4ce493878ad1d546f502fe9c8fbad7a3b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=main-1fb5d10, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-28T17:35:22.947 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:22 smithi066 bash[122431]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3 2024-02-28T17:35:23.260 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:22 smithi066 podman[122454]: 2024-02-28 17:35:22.945964349 +0000 UTC m=+0.589616880 container remove 4953ba0214066081e61c1b229adf3c4ce493878ad1d546f502fe9c8fbad7a3b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-28T17:35:23.760 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:23 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:23.292+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:23.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:23 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:23.468+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:24.102 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:23 smithi066 podman[122542]: 2024-02-28 17:35:23.715155869 +0000 UTC m=+0.036922446 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T17:35:24.102 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:23 smithi066 podman[122542]: 2024-02-28 17:35:23.838622875 +0000 UTC m=+0.160389476 container create e78d9fa38aed91c78c535a4d0c8a27585e198018987ec13d0c11a339dfa079a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T17:35:24.102 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:24 smithi066 podman[122542]: 2024-02-28 17:35:24.100882967 +0000 UTC m=+0.422649575 container init e78d9fa38aed91c78c535a4d0c8a27585e198018987ec13d0c11a339dfa079a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-28T17:35:24.439 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:24 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:24.314+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:24.440 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:24 smithi066 podman[122542]: 2024-02-28 17:35:24.113180692 +0000 UTC m=+0.434947300 container start e78d9fa38aed91c78c535a4d0c8a27585e198018987ec13d0c11a339dfa079a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222) 2024-02-28T17:35:24.440 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:24 smithi066 podman[122542]: 2024-02-28 17:35:24.349474597 +0000 UTC m=+0.671241191 container attach e78d9fa38aed91c78c535a4d0c8a27585e198018987ec13d0c11a339dfa079a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-28T17:35:24.760 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:24 smithi066 podman[122542]: 2024-02-28 17:35:24.579292276 +0000 UTC m=+0.901058947 container died e78d9fa38aed91c78c535a4d0c8a27585e198018987ec13d0c11a339dfa079a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10) 2024-02-28T17:35:24.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:24 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:24.437+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:24.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:24 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:24.437+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:25.453 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.3.service' 2024-02-28T17:35:25.548 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:25 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:25.278+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:25.549 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:25 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:25.278+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:34:58.438839+0000 front 2024-02-28T17:34:58.439013+0000 (oldest deadline 2024-02-28T17:35:24.338511+0000) 2024-02-28T17:35:25.549 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:25 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:25.454+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:25.549 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:25 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:25.454+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:25.549 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:25 smithi066 podman[122568]: 2024-02-28 17:35:25.395172759 +0000 UTC m=+0.799517753 container remove e78d9fa38aed91c78c535a4d0c8a27585e198018987ec13d0c11a339dfa079a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-3-deactivate, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-28T17:35:25.549 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:25 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-28T17:35:25.549 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:25 smithi066 systemd[1]: Stopped Ceph osd.3 for c0118718-d654-11ee-95c3-87774f69a715. 2024-02-28T17:35:25.550 INFO:journalctl@ceph.osd.3.smithi066.stdout:Feb 28 17:35:25 smithi066 systemd[1]: ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.3.service: Consumed 1min 1.894s CPU time. 2024-02-28T17:35:25.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-28T17:35:25.592 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-28T17:35:25.592 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-28T17:35:25.592 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.4 2024-02-28T17:35:26.011 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:25 smithi066 systemd[1]: Stopping Ceph osd.4 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T17:35:26.510 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:26 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:26.466+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:26.510 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:26 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:26.466+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:26.511 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:26 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:26.255+0000 7f6f2fe00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T17:35:26.511 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:26 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:26.255+0000 7f6f2fe00700 -1 osd.4 84 *** Got signal Terminated *** 2024-02-28T17:35:26.511 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:26 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:26.255+0000 7f6f2fe00700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T17:35:26.511 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:26 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:26.259+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:26.511 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:26 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:26.259+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:34:58.438839+0000 front 2024-02-28T17:34:58.439013+0000 (oldest deadline 2024-02-28T17:35:24.338511+0000) 2024-02-28T17:35:27.507 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:27 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:27.256+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:27.507 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:27 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:27.256+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:34:58.438839+0000 front 2024-02-28T17:34:58.439013+0000 (oldest deadline 2024-02-28T17:35:24.338511+0000) 2024-02-28T17:35:27.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:27 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:27.505+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:27.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:27 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:27.505+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:28.510 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:28 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:28.252+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:28.510 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:28 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:28.252+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:34:58.438839+0000 front 2024-02-28T17:34:58.439013+0000 (oldest deadline 2024-02-28T17:35:24.338511+0000) 2024-02-28T17:35:29.010 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:28 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:28.533+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:29.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:28 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:28.533+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:29.548 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:29 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:29.260+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:29.549 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:29 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:29.260+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:34:58.438839+0000 front 2024-02-28T17:34:58.439013+0000 (oldest deadline 2024-02-28T17:35:24.338511+0000) 2024-02-28T17:35:30.010 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:29 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:29.547+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:30.010 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:29 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:29.547+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:30.590 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:30 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:30.300+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.737648+0000 front 2024-02-28T17:34:53.738096+0000 (oldest deadline 2024-02-28T17:35:18.437645+0000) 2024-02-28T17:35:30.590 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:30 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4[100488]: 2024-02-28T17:35:30.300+0000 7f6f2c000700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:34:58.438839+0000 front 2024-02-28T17:34:58.439013+0000 (oldest deadline 2024-02-28T17:35:24.338511+0000) 2024-02-28T17:35:31.010 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:30 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:30.587+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:31.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:30 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:30.587+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:31.818 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:31 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:31.543+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:31.818 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:31 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:31.543+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:32.096 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:31 smithi066 podman[122646]: 2024-02-28 17:35:31.817695242 +0000 UTC m=+5.717673393 container died 1e0b6d867560cd5b9e87bf5d0a2c4c343c1b336d999d6618856d8feaf485c270 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-02-28T17:35:32.400 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:32 smithi066 podman[122646]: 2024-02-28 17:35:32.215142037 +0000 UTC m=+6.115120230 container cleanup 1e0b6d867560cd5b9e87bf5d0a2c4c343c1b336d999d6618856d8feaf485c270 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, io.buildah.version=1.33.5, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T17:35:32.401 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:32 smithi066 bash[122646]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4 2024-02-28T17:35:32.760 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:32 smithi066 podman[122668]: 2024-02-28 17:35:32.399426613 +0000 UTC m=+0.566565020 container remove 1e0b6d867560cd5b9e87bf5d0a2c4c343c1b336d999d6618856d8feaf485c270 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-28T17:35:32.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:32 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:32.590+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:32.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:32 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:32.590+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:33.582 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:33 smithi066 podman[122757]: 2024-02-28 17:35:33.18535356 +0000 UTC m=+0.037665670 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T17:35:33.582 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:33 smithi066 podman[122757]: 2024-02-28 17:35:33.30859323 +0000 UTC m=+0.160905364 container create 17639fe9dc5237c4d42f10558deb42aa6876b69a7aec623797527aa6dfaf7137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-1fb5d10, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-28T17:35:34.011 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:33 smithi066 podman[122757]: 2024-02-28 17:35:33.596689334 +0000 UTC m=+0.449001503 container init 17639fe9dc5237c4d42f10558deb42aa6876b69a7aec623797527aa6dfaf7137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-28T17:35:34.011 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:33 smithi066 podman[122757]: 2024-02-28 17:35:33.609727294 +0000 UTC m=+0.462039443 container start 17639fe9dc5237c4d42f10558deb42aa6876b69a7aec623797527aa6dfaf7137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-1fb5d10, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-02-28T17:35:34.011 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:33 smithi066 podman[122757]: 2024-02-28 17:35:33.685882909 +0000 UTC m=+0.538195105 container attach 17639fe9dc5237c4d42f10558deb42aa6876b69a7aec623797527aa6dfaf7137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-28T17:35:34.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:33 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:33.579+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:34.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:33 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:33.579+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:34.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:33 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:33.579+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-02-28T17:35:11.165128+0000 front 2024-02-28T17:35:11.165323+0000 (oldest deadline 2024-02-28T17:35:33.464746+0000) 2024-02-28T17:35:34.419 INFO:journalctl@ceph.osd.4.smithi066.stdout:Feb 28 17:35:34 smithi066 podman[122757]: 2024-02-28 17:35:34.067898201 +0000 UTC m=+0.920210316 container died 17639fe9dc5237c4d42f10558deb42aa6876b69a7aec623797527aa6dfaf7137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-4-deactivate, ceph=True, RELEASE=main-1fb5d10, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-28T17:35:34.740 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:34 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:34.544+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:34.740 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:34 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:34.544+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:34.741 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:34 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:34.544+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-02-28T17:35:11.165128+0000 front 2024-02-28T17:35:11.165323+0000 (oldest deadline 2024-02-28T17:35:33.464746+0000) 2024-02-28T17:35:34.784 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.4.service' 2024-02-28T17:35:34.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-28T17:35:34.915 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-28T17:35:34.916 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-28T17:35:34.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.5 2024-02-28T17:35:35.260 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:35 smithi066 systemd[1]: Stopping Ceph osd.5 for c0118718-d654-11ee-95c3-87774f69a715... 2024-02-28T17:35:36.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:35 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:35.529+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:36.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:35 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:35.529+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:36.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:35 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:35.529+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-02-28T17:35:11.165128+0000 front 2024-02-28T17:35:11.165323+0000 (oldest deadline 2024-02-28T17:35:33.464746+0000) 2024-02-28T17:35:36.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:35 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:35.566+0000 7f8218000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-28T17:35:36.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:35 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:35.566+0000 7f8218000700 -1 osd.5 84 *** Got signal Terminated *** 2024-02-28T17:35:36.011 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:35 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:35.566+0000 7f8218000700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-28T17:35:36.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:36 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:36.497+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:36.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:36 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:36.497+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:36.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:36 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:36.497+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-02-28T17:35:11.165128+0000 front 2024-02-28T17:35:11.165323+0000 (oldest deadline 2024-02-28T17:35:33.464746+0000) 2024-02-28T17:35:37.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:37 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:37.448+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:37.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:37 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:37.448+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:37.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:37 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:37.448+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-02-28T17:35:11.165128+0000 front 2024-02-28T17:35:11.165323+0000 (oldest deadline 2024-02-28T17:35:33.464746+0000) 2024-02-28T17:35:38.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:38 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:38.451+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:38.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:38 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:38.451+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:38.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:38 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:38.451+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-02-28T17:35:11.165128+0000 front 2024-02-28T17:35:11.165323+0000 (oldest deadline 2024-02-28T17:35:33.464746+0000) 2024-02-28T17:35:39.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:39 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:39.426+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:39.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:39 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:39.426+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:39.761 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:39 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:39.426+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-02-28T17:35:11.165128+0000 front 2024-02-28T17:35:11.165323+0000 (oldest deadline 2024-02-28T17:35:33.464746+0000) 2024-02-28T17:35:40.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:40 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:40.455+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-02-28T17:34:53.558658+0000 front 2024-02-28T17:34:53.558701+0000 (oldest deadline 2024-02-28T17:35:15.258063+0000) 2024-02-28T17:35:40.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:40 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:40.455+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-02-28T17:35:02.160971+0000 front 2024-02-28T17:35:02.160905+0000 (oldest deadline 2024-02-28T17:35:23.860626+0000) 2024-02-28T17:35:40.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:40 smithi066 ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5[105343]: 2024-02-28T17:35:40.455+0000 7f820f200700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-02-28T17:35:11.165128+0000 front 2024-02-28T17:35:11.165323+0000 (oldest deadline 2024-02-28T17:35:33.464746+0000) 2024-02-28T17:35:41.356 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:41 smithi066 podman[122863]: 2024-02-28 17:35:41.091147488 +0000 UTC m=+5.658776840 container died 7cf5aef03cba4182eb8263f5a2f5a77c251a9d8d1b51af801137d6c16d56236a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-28T17:35:41.677 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:41 smithi066 podman[122863]: 2024-02-28 17:35:41.492070072 +0000 UTC m=+6.059699416 container cleanup 7cf5aef03cba4182eb8263f5a2f5a77c251a9d8d1b51af801137d6c16d56236a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, RELEASE=main-1fb5d10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-28T17:35:41.677 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:41 smithi066 bash[122863]: ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5 2024-02-28T17:35:42.010 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:41 smithi066 podman[122886]: 2024-02-28 17:35:41.676351631 +0000 UTC m=+0.569488111 container remove 7cf5aef03cba4182eb8263f5a2f5a77c251a9d8d1b51af801137d6c16d56236a (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-28T17:35:42.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:42 smithi066 podman[122972]: 2024-02-28 17:35:42.408400154 +0000 UTC m=+0.033484520 image pull 2272b2c42f9babde922baaabf6efc9b50088cc9d3a128524b9d525dcb1eddc6e quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc 2024-02-28T17:35:42.760 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:42 smithi066 podman[122972]: 2024-02-28 17:35:42.518875666 +0000 UTC m=+0.143960072 container create c799b0af714d9c6e01c712699b82cfbaabede099079931d1d973b85262583197 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, RELEASE=main-1fb5d10, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-02-28T17:35:43.235 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:42 smithi066 podman[122972]: 2024-02-28 17:35:42.7676581 +0000 UTC m=+0.392742509 container init c799b0af714d9c6e01c712699b82cfbaabede099079931d1d973b85262583197 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-02-28T17:35:43.236 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:42 smithi066 podman[122972]: 2024-02-28 17:35:42.780016079 +0000 UTC m=+0.405100486 container start c799b0af714d9c6e01c712699b82cfbaabede099079931d1d973b85262583197 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-28T17:35:43.236 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:42 smithi066 podman[122972]: 2024-02-28 17:35:42.880306256 +0000 UTC m=+0.505390705 container attach c799b0af714d9c6e01c712699b82cfbaabede099079931d1d973b85262583197 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-1fb5d10, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-28T17:35:43.510 INFO:journalctl@ceph.osd.5.smithi066.stdout:Feb 28 17:35:43 smithi066 podman[122972]: 2024-02-28 17:35:43.235882353 +0000 UTC m=+0.860966763 container died c799b0af714d9c6e01c712699b82cfbaabede099079931d1d973b85262583197 (image=quay.ceph.io/ceph-ci/ceph@sha256:7051beed95d8dfc8c2b1de37206abb8f79d596c447bea39002f2bbaf581330cc, name=ceph-c0118718-d654-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-1fb5d10, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) 2024-02-28T17:35:43.956 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c0118718-d654-11ee-95c3-87774f69a715@osd.5.service' 2024-02-28T17:35:44.085 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-28T17:35:44.086 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-28T17:35:44.086 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c0118718-d654-11ee-95c3-87774f69a715 --force --keep-logs 2024-02-28T17:35:49.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[76738]: fuse finished with error 0 and tester_r 0 2024-02-28T17:36:13.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c0118718-d654-11ee-95c3-87774f69a715 --force --keep-logs 2024-02-28T17:36:29.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-28T17:36:30.028 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-28T17:36:30.143 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-28T17:36:30.145 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589/remote/smithi001/crash 2024-02-28T17:36:30.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash -- . 2024-02-28T17:36:30.245 INFO:teuthology.orchestra.run.smithi001.stderr:tar: /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash: Cannot open: No such file or directory 2024-02-28T17:36:30.246 INFO:teuthology.orchestra.run.smithi001.stderr:tar: Error is not recoverable: exiting now 2024-02-28T17:36:30.250 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589/remote/smithi066/crash 2024-02-28T17:36:30.251 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash -- . 2024-02-28T17:36:30.360 INFO:teuthology.orchestra.run.smithi066.stderr:tar: /var/lib/ceph/c0118718-d654-11ee-95c3-87774f69a715/crash: Cannot open: No such file or directory 2024-02-28T17:36:30.360 INFO:teuthology.orchestra.run.smithi066.stderr:tar: Error is not recoverable: exiting now 2024-02-28T17:36:30.365 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-28T17:36:30.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-28T17:36:30.474 INFO:teuthology.orchestra.run.smithi001.stdout:1709137800.0001647 mon.smithi001 (mon.0) 848 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-28T17:36:30.476 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-28T17:36:30.476 INFO:tasks.cephadm:Compressing logs... 2024-02-28T17:36:30.476 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-28T17:36:30.481 DEBUG:teuthology.orchestra.run.smithi066:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-28T17:36:30.545 INFO:teuthology.orchestra.run.smithi001.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-28T17:36:30.551 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-28T17:36:30.552 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-02-28T17:36:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mgr.smithi001.rmthtx.log 2024-02-28T17:36:30.554 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.audit.log 2024-02-28T17:36:30.555 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mgr.smithi001.rmthtx.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.1.log 2024-02-28T17:36:30.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi001.awxxdo.log 2024-02-28T17:36:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.0.log 2024-02-28T17:36:30.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi001.awxxdo.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.cephadm.log 2024-02-28T17:36:30.565 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.0.log: /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.cephadm.log: 85.0%gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.log 2024-02-28T17:36:30.566 INFO:teuthology.orchestra.run.smithi001.stderr: 90.8% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.audit.log.gz 2024-02-28T17:36:30.566 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.cephadm.log.gz 2024-02-28T17:36:30.566 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.2.log 2024-02-28T17:36:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.log: /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-volume.log 2024-02-28T17:36:30.571 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mon.smithi001.log 2024-02-28T17:36:30.573 INFO:teuthology.orchestra.run.smithi001.stderr: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-28T17:36:30.575 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi001.rtdwhc.log 2024-02-28T17:36:30.577 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.log.gz 2024-02-28T17:36:30.584 INFO:teuthology.orchestra.run.smithi066.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-28T17:36:30.584 INFO:teuthology.orchestra.run.smithi066.stderr:: No such file or directory 2024-02-28T17:36:30.585 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-02-28T17:36:30.586 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi066.wfbdor.log 2024-02-28T17:36:30.587 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.1.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.audit.log 2024-02-28T17:36:30.588 INFO:teuthology.orchestra.run.smithi066.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-28T17:36:30.589 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi066.wfbdor.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.4.log 2024-02-28T17:36:30.590 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi066.pivimo.log 2024-02-28T17:36:30.592 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mgr.smithi066.ztraxe.log 2024-02-28T17:36:30.594 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi066.pivimo.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.5.log 2024-02-28T17:36:30.596 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mgr.smithi066.ztraxe.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.3.log 2024-02-28T17:36:30.598 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.5.log: /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.cephadm.log 2024-02-28T17:36:30.599 INFO:teuthology.orchestra.run.smithi066.stderr: 90.9% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.audit.log.gz 2024-02-28T17:36:30.602 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.log 2024-02-28T17:36:30.606 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.log: /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.cephadm.log: 85.0% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.cephadm.log.gz 2024-02-28T17:36:30.606 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-volume.log 2024-02-28T17:36:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mon.smithi001.log: /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi001.rtdwhc.log: 90.9% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-volume.log.gz 2024-02-28T17:36:30.607 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mon.smithi066.log 2024-02-28T17:36:30.614 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-volume.log: /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mon.smithi066.log: 94.5% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi066.pivimo.log.gz 2024-02-28T17:36:30.619 INFO:teuthology.orchestra.run.smithi066.stderr: 89.6% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph.log.gz 2024-02-28T17:36:30.641 INFO:teuthology.orchestra.run.smithi066.stderr: 90.8% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-volume.log.gz 2024-02-28T17:36:30.680 INFO:teuthology.orchestra.run.smithi066.stderr: 89.9% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mgr.smithi066.ztraxe.log.gz 2024-02-28T17:36:31.408 INFO:teuthology.orchestra.run.smithi066.stderr: 92.5% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mon.smithi066.log.gz 2024-02-28T17:36:31.603 INFO:teuthology.orchestra.run.smithi001.stderr: 89.8% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mgr.smithi001.rmthtx.log.gz 2024-02-28T17:36:33.260 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mon.smithi001.log.gz 2024-02-28T17:36:35.545 INFO:teuthology.orchestra.run.smithi066.stderr: 93.4% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.4.log.gz 2024-02-28T17:36:36.454 INFO:teuthology.orchestra.run.smithi066.stderr: 93.6% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.5.log.gz 2024-02-28T17:36:36.580 INFO:teuthology.orchestra.run.smithi001.stderr: 95.1% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi001.rtdwhc.log.gz 2024-02-28T17:36:36.930 INFO:teuthology.orchestra.run.smithi001.stderr: 93.3% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.2.log.gz 2024-02-28T17:36:36.942 INFO:teuthology.orchestra.run.smithi001.stderr: 93.4% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.0.log.gz 2024-02-28T17:36:37.271 INFO:teuthology.orchestra.run.smithi001.stderr: 93.5% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.1.log.gz 2024-02-28T17:36:38.838 INFO:teuthology.orchestra.run.smithi066.stderr: 93.5% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-osd.3.log.gz 2024-02-28T17:36:45.725 INFO:teuthology.orchestra.run.smithi066.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-02-28T17:36:45.725 INFO:teuthology.orchestra.run.smithi066.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-02-28T17:36:48.485 INFO:teuthology.orchestra.run.smithi001.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-02-28T17:37:18.835 INFO:teuthology.orchestra.run.smithi066.stderr: 93.3% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi066.wfbdor.log.gz 2024-02-28T17:37:18.839 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-02-28T17:37:18.839 INFO:teuthology.orchestra.run.smithi066.stderr:real 0m48.302s 2024-02-28T17:37:18.839 INFO:teuthology.orchestra.run.smithi066.stderr:user 1m18.106s 2024-02-28T17:37:18.839 INFO:teuthology.orchestra.run.smithi066.stderr:sys 0m5.261s 2024-02-28T17:37:24.948 INFO:teuthology.orchestra.run.smithi001.stderr: 93.4% -- replaced with /var/log/ceph/c0118718-d654-11ee-95c3-87774f69a715/ceph-mds.cephfs.smithi001.awxxdo.log.gz 2024-02-28T17:37:24.951 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-02-28T17:37:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m54.438s 2024-02-28T17:37:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:user 1m33.987s 2024-02-28T17:37:24.951 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m6.937s 2024-02-28T17:37:24.953 INFO:tasks.cephadm:Archiving logs... 2024-02-28T17:37:24.953 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589/remote/smithi001/log 2024-02-28T17:37:24.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-28T17:37:37.425 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589/remote/smithi066/log 2024-02-28T17:37:37.426 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-28T17:37:48.729 INFO:tasks.cephadm:Removing cluster... 2024-02-28T17:37:48.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c0118718-d654-11ee-95c3-87774f69a715 --force 2024-02-28T17:37:50.036 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c0118718-d654-11ee-95c3-87774f69a715 --force 2024-02-28T17:37:51.363 INFO:tasks.cephadm:Removing cephadm ... 2024-02-28T17:37:51.364 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-28T17:37:51.425 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-28T17:37:51.480 INFO:tasks.cephadm:Teardown complete 2024-02-28T17:37:51.480 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-28T17:37:51.492 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-28T17:37:51.493 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-28T17:37:51.493 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-28T17:37:51.501 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-28T17:37:51.646 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-28T17:37:51.657 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-28T17:37:51.657 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-28T17:37:51.665 DEBUG:teuthology.orchestra.run.smithi066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-28T17:37:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-02-28T17:37:51.717 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-02-28T17:37:51.724 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-28T17:37:51.724 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-02-28T17:37:51.724 INFO:teuthology.orchestra.run.smithi001.stdout:^- hv01.front.sepia.ceph.com 2 6 377 58 +1360us[+1360us] +/- 140ms 2024-02-28T17:37:51.724 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv02.front.sepia.ceph.com 3 7 377 119 -45us[ -23us] +/- 59ms 2024-02-28T17:37:51.724 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +62us[ +62us] +/- 76ms 2024-02-28T17:37:51.724 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T17:37:51.727 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-28T17:37:51.727 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-02-28T17:37:51.727 INFO:teuthology.orchestra.run.smithi066.stdout:^- hv01.front.sepia.ceph.com 2 7 377 116 +1433us[+1497us] +/- 139ms 2024-02-28T17:37:51.728 INFO:teuthology.orchestra.run.smithi066.stdout:^* hv02.front.sepia.ceph.com 3 7 377 116 -16us[ +47us] +/- 59ms 2024-02-28T17:37:51.728 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +92us[ +92us] +/- 76ms 2024-02-28T17:37:51.728 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-28T17:37:51.729 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-28T17:37:51.740 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-28T17:37:51.741 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-28T17:37:51.753 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-28T17:37:52.015 DEBUG:teuthology.orchestra.run.smithi066:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-28T17:37:52.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-28T17:37:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T17:37:52.337 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-28T17:37:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T17:37:52.440 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-28T17:37:52.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-28T17:37:52.469 INFO:teuthology.task.internal:Duration was 5498.614586 seconds 2024-02-28T17:37:52.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-28T17:37:52.489 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-28T17:37:52.489 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-28T17:37:52.497 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-28T17:37:52.662 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-28T17:37:52.670 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-28T17:37:53.126 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-28T17:37:53.126 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-02-28T17:37:53.126 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-28T17:37:53.191 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi066.front.sepia.ceph.com 2024-02-28T17:37:53.191 DEBUG:teuthology.orchestra.run.smithi066:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-28T17:37:53.270 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-28T17:37:53.270 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-28T17:37:53.277 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-28T17:37:53.374 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-28T17:37:53.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-28T17:37:53.381 DEBUG:teuthology.orchestra.run.smithi066:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-28T17:37:54.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-28T17:37:55.002 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-28T17:37:55.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-28T17:37:55.088 DEBUG:teuthology.orchestra.run.smithi066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-28T17:37:55.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-28T17:37:55.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-28T17:37:55.224 DEBUG:teuthology.orchestra.run.smithi066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-28T17:37:55.331 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-02-28T17:37:55.346 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = core 2024-02-28T17:37:55.432 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-28T17:37:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T17:37:55.490 DEBUG:teuthology.orchestra.run.smithi066:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-28T17:37:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T17:37:55.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-28T17:37:55.559 INFO:teuthology.task.internal:Transferring archived files... 2024-02-28T17:37:55.560 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589/remote/smithi001 2024-02-28T17:37:55.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-28T17:37:55.706 DEBUG:teuthology.misc:Transferring archived files from smithi066:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589/remote/smithi066 2024-02-28T17:37:55.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-28T17:37:55.833 INFO:teuthology.task.internal:Removing archive directory... 2024-02-28T17:37:55.833 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-28T17:37:55.841 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-28T17:37:55.899 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-28T17:37:55.913 INFO:teuthology.task.internal:Not uploading archives. 2024-02-28T17:37:55.913 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-28T17:37:55.928 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-28T17:37:55.928 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-28T17:37:55.936 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-28T17:37:55.989 INFO:teuthology.orchestra.run.smithi001.stdout: 393229 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 28 17:37 /home/ubuntu/cephtest 2024-02-28T17:37:55.990 INFO:teuthology.orchestra.run.smithi001.stdout: 786768 4 d--------- 2 ubuntu ubuntu 4096 Feb 28 16:27 /home/ubuntu/cephtest/mnt.0 2024-02-28T17:37:55.990 INFO:teuthology.orchestra.run.smithi001.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-02-28T17:37:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-28T17:37:55.993 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-28T17:37:55.993 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-28T17:37:55.994 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-28T17:37:56.009 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-28T17:37:56.025 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-02-28T17:37:56.025 INFO:teuthology.nuke:Checking targets against current locks 2024-02-28T17:37:56.052 DEBUG:teuthology.nuke:shortname: smithi001 2024-02-28T17:37:56.052 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-28T17:37:56.076 DEBUG:teuthology.nuke:shortname: smithi066 2024-02-28T17:37:56.077 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-28T17:37:56.080 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-28 15:48:21.484039', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJhQDYXyDZaLzE/d7Nsps3cOMWi1ygHuaOzm+e+/sACBuqlTUUchOEbsP1h+nN4ONrGGRgTxPgmmnbAC232Nco='} 2024-02-28T17:37:56.102 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-02-28_15:21:56-fs:upgrade:mds_upgrade_sequence-main-testing-default-smithi/7575589', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-28 15:48:21.485960', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCcFfzZm+EyMI205iYFlk8i1qEFqGB7M/9JUZv1Pn6faQKtShms8EsSV8ZQC4+J5ZAE2rGAB2+DeVQ0bP+XIIBY='} 2024-02-28T17:37:56.130 INFO:teuthology.orchestra.console.smithi001:Power off 2024-02-28T17:37:56.130 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-28T17:37:56.154 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-02-28T17:37:56.154 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-28T17:37:56.171 INFO:teuthology.orchestra.console.smithi066:Power off 2024-02-28T17:37:56.172 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-28T17:37:56.183 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-02-28T17:37:56.196 DEBUG:teuthology.orchestra.console.smithi066:power off output: Chassis Power Control: Down/Off 2024-02-28T17:37:56.196 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-28T17:37:56.219 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-02-28T17:38:00.185 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-28T17:38:00.220 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-28T17:38:00.295 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-02-28T17:38:00.330 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-02-28T17:38:04.297 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-28T17:38:04.332 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-28T17:38:04.409 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-02-28T17:38:04.444 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off 2024-02-28T17:38:04.511 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-02-28T17:38:04.545 INFO:teuthology.orchestra.console.smithi066:Power off completed 2024-02-28T17:38:04.650 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 5498.614586353302 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default host.a-kernel-sha1: bd7d00fb724897dda560070d01bc6e5c80881f15 host.b-kernel-sha1: bd7d00fb724897dda560070d01bc6e5c80881f15 owner: scheduled_vshankar@teuthology status: fail success: false 2024-02-28T17:38:04.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-28T17:38:04.739 INFO:teuthology.run:FAIL